2024-04-24T23:46:59.997 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-24T23:46:59.998 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672224 --name teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi --verbose --owner scheduled_teuthology@teuthology --description fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} -- /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672224/orig.config.yaml 2024-04-24T23:47:00.050 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-24T23:47:00.150 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672224 branch: squid description: fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7672224' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: squid ceph: cephfs: fs: - name: a - name: b session_timeout: 300 conf: client: client mount timeout: 600 client_shutdown_timeout: 120 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 filestore flush min: 0 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - Replacing daemon mds - SLOW_OPS - slow request sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: cephfs_name: a client.1: cephfs_name: b install: ceph: flavor: default sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 kclient: client.0: cephfs_name: a client.1: cephfs_name: b thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 seed: 8528 sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 sleep_before_teardown: 0 subset: 122/512 suite: fs suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi116.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBZUErwp5x/iXFsF5BkmYc2BOevi/pGO8L/YzJPjxT/XGX/GymjVee/7ER30H/8Spn1Vf3u3LChuWJYK5jVPz4g= smithi174.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOELdnCNwpalxyvSIzsFkB1x0eynd3fo9oN+EekJ9w0Ohu/oWPwHdJAwN+hH7oCyaqXhva52JR/KPNxC0lso5I8= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - sequential: - mgrmodules - ceph-fuse: null - mds_thrash: null - workunit: clients: all: - suites/ffsb.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-24_21:24:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-24T23:47:00.150 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa; will attempt to use it 2024-04-24T23:47:00.151 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa/tasks 2024-04-24T23:47:00.151 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-24T23:47:00.152 INFO:teuthology.task.internal:Checking packages... 2024-04-24T23:47:00.175 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'd2c45d977142e024a59b68a8f1dce8f0b9840551' 2024-04-24T23:47:00.175 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-24T23:47:00.175 INFO:teuthology.packaging:ref: None 2024-04-24T23:47:00.175 INFO:teuthology.packaging:tag: None 2024-04-24T23:47:00.175 INFO:teuthology.packaging:branch: squid 2024-04-24T23:47:00.175 INFO:teuthology.packaging:sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-24T23:47:00.176 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=squid 2024-04-24T23:47:00.491 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2445-gd2c45d97-1jammy 2024-04-24T23:47:00.493 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-24T23:47:00.500 INFO:teuthology.task.internal:no buildpackages task found 2024-04-24T23:47:00.500 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-24T23:47:00.507 INFO:teuthology.task.internal:Saving configuration 2024-04-24T23:47:00.524 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-24T23:47:00.531 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-24T23:47:00.553 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi116.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672224', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-24 23:41:21.770918', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJvUtCgKbpIRThYcqU1W9Hlx4EL6Nt0LelZbDNIwYD4u'} 2024-04-24T23:47:00.573 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi174.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672224', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-24 23:41:21.769167', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINgMZD5jkt5bUC6JlbDMLb0HizdfuN6aWlgno3YBiJfC'} 2024-04-24T23:47:00.573 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-24T23:47:00.579 INFO:teuthology.task.internal:roles: ubuntu@smithi116.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-04-24T23:47:00.579 INFO:teuthology.task.internal:roles: ubuntu@smithi174.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2024-04-24T23:47:00.579 INFO:teuthology.run_tasks:Running task console_log... 2024-04-24T23:47:00.646 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f39963e1040>, signals=[15]) 2024-04-24T23:47:00.647 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-24T23:47:00.653 INFO:teuthology.task.internal:Opening connections... 2024-04-24T23:47:00.653 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi116.front.sepia.ceph.com 2024-04-24T23:47:00.655 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T23:47:00.735 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi174.front.sepia.ceph.com 2024-04-24T23:47:00.736 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T23:47:00.818 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-24T23:47:00.825 DEBUG:teuthology.orchestra.run.smithi116:> uname -m 2024-04-24T23:47:00.830 INFO:teuthology.orchestra.run.smithi116.stdout:x86_64 2024-04-24T23:47:00.830 DEBUG:teuthology.orchestra.run.smithi116:> cat /etc/os-release 2024-04-24T23:47:00.881 INFO:teuthology.orchestra.run.smithi116.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-24T23:47:00.881 INFO:teuthology.orchestra.run.smithi116.stdout:NAME="Ubuntu" 2024-04-24T23:47:00.881 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION_ID="22.04" 2024-04-24T23:47:00.881 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-24T23:47:00.881 INFO:teuthology.orchestra.run.smithi116.stdout:VERSION_CODENAME=jammy 2024-04-24T23:47:00.881 INFO:teuthology.orchestra.run.smithi116.stdout:ID=ubuntu 2024-04-24T23:47:00.882 INFO:teuthology.orchestra.run.smithi116.stdout:ID_LIKE=debian 2024-04-24T23:47:00.882 INFO:teuthology.orchestra.run.smithi116.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-24T23:47:00.882 INFO:teuthology.orchestra.run.smithi116.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-24T23:47:00.882 INFO:teuthology.orchestra.run.smithi116.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-24T23:47:00.882 INFO:teuthology.orchestra.run.smithi116.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-24T23:47:00.882 INFO:teuthology.orchestra.run.smithi116.stdout:UBUNTU_CODENAME=jammy 2024-04-24T23:47:00.882 INFO:teuthology.lock.ops:Updating smithi116.front.sepia.ceph.com on lock server 2024-04-24T23:47:00.903 DEBUG:teuthology.orchestra.run.smithi174:> uname -m 2024-04-24T23:47:00.908 INFO:teuthology.orchestra.run.smithi174.stdout:x86_64 2024-04-24T23:47:00.908 DEBUG:teuthology.orchestra.run.smithi174:> cat /etc/os-release 2024-04-24T23:47:00.955 INFO:teuthology.orchestra.run.smithi174.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-24T23:47:00.955 INFO:teuthology.orchestra.run.smithi174.stdout:NAME="Ubuntu" 2024-04-24T23:47:00.955 INFO:teuthology.orchestra.run.smithi174.stdout:VERSION_ID="22.04" 2024-04-24T23:47:00.955 INFO:teuthology.orchestra.run.smithi174.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-24T23:47:00.955 INFO:teuthology.orchestra.run.smithi174.stdout:VERSION_CODENAME=jammy 2024-04-24T23:47:00.955 INFO:teuthology.orchestra.run.smithi174.stdout:ID=ubuntu 2024-04-24T23:47:00.955 INFO:teuthology.orchestra.run.smithi174.stdout:ID_LIKE=debian 2024-04-24T23:47:00.955 INFO:teuthology.orchestra.run.smithi174.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-24T23:47:00.955 INFO:teuthology.orchestra.run.smithi174.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-24T23:47:00.955 INFO:teuthology.orchestra.run.smithi174.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-24T23:47:00.955 INFO:teuthology.orchestra.run.smithi174.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-24T23:47:00.956 INFO:teuthology.orchestra.run.smithi174.stdout:UBUNTU_CODENAME=jammy 2024-04-24T23:47:00.956 INFO:teuthology.lock.ops:Updating smithi174.front.sepia.ceph.com on lock server 2024-04-24T23:47:00.970 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-24T23:47:00.979 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-24T23:47:00.985 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-24T23:47:00.985 DEBUG:teuthology.orchestra.run.smithi116:> test '!' -e /home/ubuntu/cephtest 2024-04-24T23:47:00.988 DEBUG:teuthology.orchestra.run.smithi174:> test '!' -e /home/ubuntu/cephtest 2024-04-24T23:47:01.002 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-24T23:47:01.008 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-24T23:47:01.008 DEBUG:teuthology.orchestra.run.smithi116:> test -z $(ls -A /var/lib/ceph) 2024-04-24T23:47:01.035 DEBUG:teuthology.orchestra.run.smithi174:> test -z $(ls -A /var/lib/ceph) 2024-04-24T23:47:01.059 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-24T23:47:01.101 INFO:teuthology.run_tasks:Running task kernel... 2024-04-24T23:47:01.116 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-24T23:47:01.116 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-24T23:47:01.116 DEBUG:teuthology.orchestra.run.smithi116:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-24T23:47:01.117 DEBUG:teuthology.orchestra.run.smithi174:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-24T23:47:01.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T23:47:01.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T23:47:01.122 DEBUG:teuthology.orchestra.run.smithi116:> uname -r 2024-04-24T23:47:01.123 DEBUG:teuthology.orchestra.run.smithi174:> uname -r 2024-04-24T23:47:01.170 INFO:teuthology.orchestra.run.smithi116.stdout:5.15.0-56-generic 2024-04-24T23:47:01.170 INFO:teuthology.task.kernel:Running kernel on smithi116: 5.15.0-56-generic 2024-04-24T23:47:01.170 DEBUG:teuthology.orchestra.run.smithi116:> sudo apt-get clean 2024-04-24T23:47:01.171 INFO:teuthology.orchestra.run.smithi174.stdout:5.15.0-56-generic 2024-04-24T23:47:01.173 INFO:teuthology.task.kernel:Running kernel on smithi174: 5.15.0-56-generic 2024-04-24T23:47:01.173 DEBUG:teuthology.orchestra.run.smithi174:> sudo apt-get clean 2024-04-24T23:47:01.302 DEBUG:teuthology.orchestra.run.smithi116:> sudo apt-get update 2024-04-24T23:47:01.304 DEBUG:teuthology.orchestra.run.smithi174:> sudo apt-get update 2024-04-24T23:47:01.456 INFO:teuthology.orchestra.run.smithi174.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-24T23:47:01.463 INFO:teuthology.orchestra.run.smithi174.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-24T23:47:01.467 INFO:teuthology.orchestra.run.smithi174.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-24T23:47:01.467 INFO:teuthology.orchestra.run.smithi116.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-24T23:47:01.484 INFO:teuthology.orchestra.run.smithi116.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-24T23:47:01.558 INFO:teuthology.orchestra.run.smithi116.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-24T23:47:01.558 INFO:teuthology.orchestra.run.smithi174.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-24T23:47:01.568 INFO:teuthology.orchestra.run.smithi116.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-24T23:47:01.683 INFO:teuthology.orchestra.run.smithi174.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,394 kB] 2024-04-24T23:47:01.701 INFO:teuthology.orchestra.run.smithi116.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [618 kB] 2024-04-24T23:47:01.761 INFO:teuthology.orchestra.run.smithi174.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,610 kB] 2024-04-24T23:47:01.798 INFO:teuthology.orchestra.run.smithi174.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [452 kB] 2024-04-24T23:47:01.799 INFO:teuthology.orchestra.run.smithi116.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,610 kB] 2024-04-24T23:47:01.803 INFO:teuthology.orchestra.run.smithi174.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [243 kB] 2024-04-24T23:47:01.808 INFO:teuthology.orchestra.run.smithi174.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-24T23:47:01.808 INFO:teuthology.orchestra.run.smithi174.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-04-24T23:47:01.808 INFO:teuthology.orchestra.run.smithi174.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,773 kB] 2024-04-24T23:47:01.836 INFO:teuthology.orchestra.run.smithi174.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [300 kB] 2024-04-24T23:47:01.841 INFO:teuthology.orchestra.run.smithi174.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-24T23:47:01.841 INFO:teuthology.orchestra.run.smithi174.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [601 kB] 2024-04-24T23:47:01.850 INFO:teuthology.orchestra.run.smithi174.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-04-24T23:47:01.855 INFO:teuthology.orchestra.run.smithi116.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [303 kB] 2024-04-24T23:47:01.862 INFO:teuthology.orchestra.run.smithi174.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-04-24T23:47:01.864 INFO:teuthology.orchestra.run.smithi174.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-24T23:47:01.867 INFO:teuthology.orchestra.run.smithi116.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-24T23:47:01.867 INFO:teuthology.orchestra.run.smithi116.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-24T23:47:01.867 INFO:teuthology.orchestra.run.smithi116.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,830 kB] 2024-04-24T23:47:01.886 INFO:teuthology.orchestra.run.smithi174.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-24T23:47:01.887 INFO:teuthology.orchestra.run.smithi174.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-24T23:47:01.887 INFO:teuthology.orchestra.run.smithi174.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-24T23:47:01.887 INFO:teuthology.orchestra.run.smithi116.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [311 kB] 2024-04-24T23:47:01.889 INFO:teuthology.orchestra.run.smithi174.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [618 kB] 2024-04-24T23:47:01.889 INFO:teuthology.orchestra.run.smithi174.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-24T23:47:01.891 INFO:teuthology.orchestra.run.smithi116.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-24T23:47:01.902 INFO:teuthology.orchestra.run.smithi116.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,070 kB] 2024-04-24T23:47:01.906 INFO:teuthology.orchestra.run.smithi174.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [303 kB] 2024-04-24T23:47:01.909 INFO:teuthology.orchestra.run.smithi116.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [700 kB] 2024-04-24T23:47:01.916 INFO:teuthology.orchestra.run.smithi174.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-24T23:47:01.917 INFO:teuthology.orchestra.run.smithi174.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-24T23:47:01.917 INFO:teuthology.orchestra.run.smithi174.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,830 kB] 2024-04-24T23:47:01.917 INFO:teuthology.orchestra.run.smithi116.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB] 2024-04-24T23:47:01.919 INFO:teuthology.orchestra.run.smithi116.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-24T23:47:01.919 INFO:teuthology.orchestra.run.smithi116.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-24T23:47:01.920 INFO:teuthology.orchestra.run.smithi116.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-24T23:47:01.920 INFO:teuthology.orchestra.run.smithi116.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-24T23:47:01.937 INFO:teuthology.orchestra.run.smithi116.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-24T23:47:01.937 INFO:teuthology.orchestra.run.smithi116.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [90.8 kB] 2024-04-24T23:47:01.937 INFO:teuthology.orchestra.run.smithi116.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [82.9 kB] 2024-04-24T23:47:01.938 INFO:teuthology.orchestra.run.smithi116.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-24T23:47:01.946 INFO:teuthology.orchestra.run.smithi174.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [311 kB] 2024-04-24T23:47:01.949 INFO:teuthology.orchestra.run.smithi174.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-24T23:47:01.949 INFO:teuthology.orchestra.run.smithi174.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [700 kB] 2024-04-24T23:47:01.956 INFO:teuthology.orchestra.run.smithi174.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,070 kB] 2024-04-24T23:47:01.957 INFO:teuthology.orchestra.run.smithi116.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-24T23:47:01.958 INFO:teuthology.orchestra.run.smithi116.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [18.5 kB] 2024-04-24T23:47:01.958 INFO:teuthology.orchestra.run.smithi116.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [29.8 kB] 2024-04-24T23:47:01.958 INFO:teuthology.orchestra.run.smithi116.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-24T23:47:01.958 INFO:teuthology.orchestra.run.smithi116.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-24T23:47:01.968 INFO:teuthology.orchestra.run.smithi174.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB] 2024-04-24T23:47:01.970 INFO:teuthology.orchestra.run.smithi174.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-24T23:47:01.970 INFO:teuthology.orchestra.run.smithi174.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-24T23:47:01.971 INFO:teuthology.orchestra.run.smithi174.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-24T23:47:01.971 INFO:teuthology.orchestra.run.smithi174.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-24T23:47:01.973 INFO:teuthology.orchestra.run.smithi174.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-24T23:47:01.973 INFO:teuthology.orchestra.run.smithi174.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [90.8 kB] 2024-04-24T23:47:01.974 INFO:teuthology.orchestra.run.smithi174.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [82.9 kB] 2024-04-24T23:47:01.994 INFO:teuthology.orchestra.run.smithi174.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-24T23:47:01.994 INFO:teuthology.orchestra.run.smithi174.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-24T23:47:01.994 INFO:teuthology.orchestra.run.smithi174.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [18.5 kB] 2024-04-24T23:47:01.994 INFO:teuthology.orchestra.run.smithi174.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [29.8 kB] 2024-04-24T23:47:01.995 INFO:teuthology.orchestra.run.smithi174.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-24T23:47:01.995 INFO:teuthology.orchestra.run.smithi174.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-24T23:47:01.999 INFO:teuthology.orchestra.run.smithi116.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [452 kB] 2024-04-24T23:47:02.283 INFO:teuthology.orchestra.run.smithi116.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,394 kB] 2024-04-24T23:47:02.451 INFO:teuthology.orchestra.run.smithi116.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [243 kB] 2024-04-24T23:47:02.464 INFO:teuthology.orchestra.run.smithi116.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-24T23:47:02.464 INFO:teuthology.orchestra.run.smithi116.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,773 kB] 2024-04-24T23:47:02.547 INFO:teuthology.orchestra.run.smithi116.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-04-24T23:47:02.548 INFO:teuthology.orchestra.run.smithi116.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [300 kB] 2024-04-24T23:47:02.556 INFO:teuthology.orchestra.run.smithi116.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-24T23:47:02.556 INFO:teuthology.orchestra.run.smithi116.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-04-24T23:47:02.577 INFO:teuthology.orchestra.run.smithi116.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [601 kB] 2024-04-24T23:47:02.755 INFO:teuthology.orchestra.run.smithi116.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-04-24T23:47:03.191 INFO:teuthology.orchestra.run.smithi116.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-24T23:47:03.199 INFO:teuthology.orchestra.run.smithi116.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-24T23:47:03.200 INFO:teuthology.orchestra.run.smithi116.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-24T23:47:03.217 INFO:teuthology.orchestra.run.smithi116.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-24T23:47:03.225 INFO:teuthology.orchestra.run.smithi116.stdout:Get:44 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-24T23:47:07.441 INFO:teuthology.orchestra.run.smithi174.stdout:Fetched 13.3 MB in 2s (7,125 kB/s) 2024-04-24T23:47:07.557 INFO:teuthology.orchestra.run.smithi116.stdout:Fetched 13.3 MB in 2s (6,959 kB/s) 2024-04-24T23:47:08.815 INFO:teuthology.orchestra.run.smithi174.stdout:Reading package lists... 2024-04-24T23:47:08.831 DEBUG:teuthology.orchestra.run.smithi174:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-24T23:47:08.882 INFO:teuthology.orchestra.run.smithi174.stdout:Reading package lists... 2024-04-24T23:47:08.928 INFO:teuthology.orchestra.run.smithi116.stdout:Reading package lists... 2024-04-24T23:47:08.944 DEBUG:teuthology.orchestra.run.smithi116:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-24T23:47:08.995 INFO:teuthology.orchestra.run.smithi116.stdout:Reading package lists... 2024-04-24T23:47:09.059 INFO:teuthology.orchestra.run.smithi174.stdout:Building dependency tree... 2024-04-24T23:47:09.060 INFO:teuthology.orchestra.run.smithi174.stdout:Reading state information... 2024-04-24T23:47:09.173 INFO:teuthology.orchestra.run.smithi116.stdout:Building dependency tree... 2024-04-24T23:47:09.173 INFO:teuthology.orchestra.run.smithi116.stdout:Reading state information... 2024-04-24T23:47:09.214 INFO:teuthology.orchestra.run.smithi174.stdout:The following packages were automatically installed and are no longer required: 2024-04-24T23:47:09.214 INFO:teuthology.orchestra.run.smithi174.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-24T23:47:09.214 INFO:teuthology.orchestra.run.smithi174.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-24T23:47:09.215 INFO:teuthology.orchestra.run.smithi174.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-24T23:47:09.215 INFO:teuthology.orchestra.run.smithi174.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-24T23:47:09.215 INFO:teuthology.orchestra.run.smithi174.stdout: python2.7-minimal 2024-04-24T23:47:09.215 INFO:teuthology.orchestra.run.smithi174.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-24T23:47:09.216 INFO:teuthology.orchestra.run.smithi174.stdout:The following additional packages will be installed: 2024-04-24T23:47:09.216 INFO:teuthology.orchestra.run.smithi174.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-24T23:47:09.216 INFO:teuthology.orchestra.run.smithi174.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-04-24T23:47:09.217 INFO:teuthology.orchestra.run.smithi174.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-04-24T23:47:09.217 INFO:teuthology.orchestra.run.smithi174.stdout:Suggested packages: 2024-04-24T23:47:09.218 INFO:teuthology.orchestra.run.smithi174.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-24T23:47:09.218 INFO:teuthology.orchestra.run.smithi174.stdout:Recommended packages: 2024-04-24T23:47:09.218 INFO:teuthology.orchestra.run.smithi174.stdout: thermald 2024-04-24T23:47:09.269 INFO:teuthology.orchestra.run.smithi174.stdout:The following NEW packages will be installed: 2024-04-24T23:47:09.269 INFO:teuthology.orchestra.run.smithi174.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-24T23:47:09.269 INFO:teuthology.orchestra.run.smithi174.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-04-24T23:47:09.270 INFO:teuthology.orchestra.run.smithi174.stdout: linux-modules-extra-5.15.0-105-generic 2024-04-24T23:47:09.270 INFO:teuthology.orchestra.run.smithi174.stdout:The following packages will be upgraded: 2024-04-24T23:47:09.271 INFO:teuthology.orchestra.run.smithi174.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-24T23:47:09.325 INFO:teuthology.orchestra.run.smithi174.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-24T23:47:09.325 INFO:teuthology.orchestra.run.smithi174.stdout:Need to get 113 MB of archives. 2024-04-24T23:47:09.325 INFO:teuthology.orchestra.run.smithi174.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-24T23:47:09.325 INFO:teuthology.orchestra.run.smithi174.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-04-24T23:47:09.326 INFO:teuthology.orchestra.run.smithi116.stdout:The following packages were automatically installed and are no longer required: 2024-04-24T23:47:09.326 INFO:teuthology.orchestra.run.smithi116.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-24T23:47:09.326 INFO:teuthology.orchestra.run.smithi116.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-24T23:47:09.327 INFO:teuthology.orchestra.run.smithi116.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-24T23:47:09.327 INFO:teuthology.orchestra.run.smithi116.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-24T23:47:09.327 INFO:teuthology.orchestra.run.smithi116.stdout: python2.7-minimal 2024-04-24T23:47:09.327 INFO:teuthology.orchestra.run.smithi116.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-24T23:47:09.328 INFO:teuthology.orchestra.run.smithi116.stdout:The following additional packages will be installed: 2024-04-24T23:47:09.328 INFO:teuthology.orchestra.run.smithi116.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-24T23:47:09.329 INFO:teuthology.orchestra.run.smithi116.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-04-24T23:47:09.329 INFO:teuthology.orchestra.run.smithi116.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-04-24T23:47:09.330 INFO:teuthology.orchestra.run.smithi116.stdout:Suggested packages: 2024-04-24T23:47:09.330 INFO:teuthology.orchestra.run.smithi116.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-24T23:47:09.330 INFO:teuthology.orchestra.run.smithi116.stdout:Recommended packages: 2024-04-24T23:47:09.330 INFO:teuthology.orchestra.run.smithi116.stdout: thermald 2024-04-24T23:47:09.377 INFO:teuthology.orchestra.run.smithi116.stdout:The following NEW packages will be installed: 2024-04-24T23:47:09.378 INFO:teuthology.orchestra.run.smithi116.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-24T23:47:09.378 INFO:teuthology.orchestra.run.smithi116.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-04-24T23:47:09.378 INFO:teuthology.orchestra.run.smithi116.stdout: linux-modules-extra-5.15.0-105-generic 2024-04-24T23:47:09.379 INFO:teuthology.orchestra.run.smithi116.stdout:The following packages will be upgraded: 2024-04-24T23:47:09.379 INFO:teuthology.orchestra.run.smithi116.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-24T23:47:09.433 INFO:teuthology.orchestra.run.smithi116.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-24T23:47:09.433 INFO:teuthology.orchestra.run.smithi116.stdout:Need to get 113 MB of archives. 2024-04-24T23:47:09.434 INFO:teuthology.orchestra.run.smithi116.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-24T23:47:09.434 INFO:teuthology.orchestra.run.smithi116.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-04-24T23:47:09.909 INFO:teuthology.orchestra.run.smithi174.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-04-24T23:47:10.090 INFO:teuthology.orchestra.run.smithi174.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-04-24T23:47:10.174 INFO:teuthology.orchestra.run.smithi116.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-04-24T23:47:10.388 INFO:teuthology.orchestra.run.smithi116.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-04-24T23:47:11.434 INFO:teuthology.orchestra.run.smithi174.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-04-24T23:47:11.434 INFO:teuthology.orchestra.run.smithi174.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-04-24T23:47:11.434 INFO:teuthology.orchestra.run.smithi174.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-04-24T23:47:11.590 INFO:teuthology.orchestra.run.smithi174.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-04-24T23:47:11.620 INFO:teuthology.orchestra.run.smithi174.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-04-24T23:47:11.689 INFO:teuthology.orchestra.run.smithi116.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-04-24T23:47:11.689 INFO:teuthology.orchestra.run.smithi116.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-04-24T23:47:11.689 INFO:teuthology.orchestra.run.smithi116.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-04-24T23:47:11.898 INFO:teuthology.orchestra.run.smithi116.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-04-24T23:47:11.931 INFO:teuthology.orchestra.run.smithi116.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-04-24T23:47:12.034 INFO:teuthology.orchestra.run.smithi174.stdout:Fetched 113 MB in 2s (48.4 MB/s) 2024-04-24T23:47:12.226 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-04-24T23:47:12.322 INFO:teuthology.orchestra.run.smithi116.stdout:Fetched 113 MB in 3s (44.5 MB/s) 2024-04-24T23:47:12.469 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-04-24T23:47:14.178 INFO:teuthology.orchestra.run.smithi174.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-24T23:47:14.183 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-24T23:47:14.312 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T23:47:14.439 INFO:teuthology.orchestra.run.smithi116.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-24T23:47:14.443 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-24T23:47:14.632 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T23:47:16.750 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-04-24T23:47:16.753 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-04-24T23:47:16.771 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-24T23:47:16.774 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-24T23:47:16.855 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T23:47:16.856 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T23:47:17.289 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-04-24T23:47:17.310 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-24T23:47:17.320 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-04-24T23:47:17.340 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-24T23:47:17.347 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T23:47:17.378 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T23:47:23.204 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-04-24T23:47:23.312 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-24T23:47:23.644 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-04-24T23:47:23.773 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-24T23:47:24.026 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-04-24T23:47:24.048 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-04-24T23:47:24.084 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-24T23:47:24.136 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-04-24T23:47:24.261 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-24T23:47:24.576 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-04-24T23:47:24.698 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-24T23:47:25.001 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-04-24T23:47:25.023 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-04-24T23:47:25.067 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-24T23:47:29.412 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-04-24T23:47:29.431 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-24T23:47:29.471 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T23:47:31.078 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-04-24T23:47:31.118 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-24T23:47:31.153 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T23:47:32.590 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-04-24T23:47:32.712 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-24T23:47:33.054 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-24T23:47:33.184 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T23:47:33.310 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-04-24T23:47:33.438 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T23:47:33.765 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-04-24T23:47:33.905 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-24T23:47:34.246 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-24T23:47:34.376 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T23:47:34.502 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-04-24T23:47:34.614 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T23:47:34.754 INFO:teuthology.orchestra.run.smithi174.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-24T23:47:34.754 INFO:teuthology.orchestra.run.smithi174.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-24T23:47:34.754 INFO:teuthology.orchestra.run.smithi174.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-04-24T23:47:34.754 INFO:teuthology.orchestra.run.smithi174.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-04-24T23:47:34.905 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T23:47:35.916 INFO:teuthology.orchestra.run.smithi116.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-24T23:47:35.916 INFO:teuthology.orchestra.run.smithi116.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-24T23:47:35.917 INFO:teuthology.orchestra.run.smithi116.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-04-24T23:47:35.917 INFO:teuthology.orchestra.run.smithi116.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-04-24T23:47:36.074 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T23:47:36.308 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T23:47:37.735 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T23:47:37.892 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-04-24T23:47:38.017 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-04-24T23:47:38.144 INFO:teuthology.orchestra.run.smithi174.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T23:47:38.211 INFO:teuthology.orchestra.run.smithi174.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-24T23:47:38.211 INFO:teuthology.orchestra.run.smithi174.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-04-24T23:47:39.126 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-04-24T23:47:39.244 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-04-24T23:47:39.395 INFO:teuthology.orchestra.run.smithi116.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-24T23:47:39.466 INFO:teuthology.orchestra.run.smithi116.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-24T23:47:39.466 INFO:teuthology.orchestra.run.smithi116.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-04-24T23:47:52.680 INFO:teuthology.orchestra.run.smithi174.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-24T23:47:52.680 INFO:teuthology.orchestra.run.smithi174.stdout:Sourcing file `/etc/default/grub' 2024-04-24T23:47:52.696 INFO:teuthology.orchestra.run.smithi174.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-24T23:47:52.715 INFO:teuthology.orchestra.run.smithi174.stdout:Generating grub configuration file ... 2024-04-24T23:47:52.999 INFO:teuthology.orchestra.run.smithi174.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-24T23:47:53.024 INFO:teuthology.orchestra.run.smithi174.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-24T23:47:53.341 INFO:teuthology.orchestra.run.smithi174.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-24T23:47:53.346 INFO:teuthology.orchestra.run.smithi174.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-24T23:47:53.418 INFO:teuthology.orchestra.run.smithi174.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-24T23:47:53.424 INFO:teuthology.orchestra.run.smithi174.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-24T23:47:53.593 INFO:teuthology.orchestra.run.smithi174.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-24T23:47:53.593 INFO:teuthology.orchestra.run.smithi174.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-24T23:47:53.593 INFO:teuthology.orchestra.run.smithi174.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-24T23:47:53.617 INFO:teuthology.orchestra.run.smithi116.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-24T23:47:53.617 INFO:teuthology.orchestra.run.smithi116.stdout:Sourcing file `/etc/default/grub' 2024-04-24T23:47:53.628 INFO:teuthology.orchestra.run.smithi116.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-24T23:47:53.636 INFO:teuthology.orchestra.run.smithi174.stdout:done 2024-04-24T23:47:53.647 INFO:teuthology.orchestra.run.smithi116.stdout:Generating grub configuration file ... 2024-04-24T23:47:53.903 INFO:teuthology.orchestra.run.smithi116.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-24T23:47:53.927 INFO:teuthology.orchestra.run.smithi116.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-24T23:47:54.081 DEBUG:teuthology.orchestra.run.smithi174:> dpkg -s linux-image-generic 2024-04-24T23:47:54.103 INFO:teuthology.orchestra.run.smithi174.stdout:Package: linux-image-generic 2024-04-24T23:47:54.103 INFO:teuthology.orchestra.run.smithi174.stdout:Status: install ok installed 2024-04-24T23:47:54.103 INFO:teuthology.orchestra.run.smithi174.stdout:Priority: optional 2024-04-24T23:47:54.103 INFO:teuthology.orchestra.run.smithi174.stdout:Section: kernel 2024-04-24T23:47:54.103 INFO:teuthology.orchestra.run.smithi174.stdout:Installed-Size: 21 2024-04-24T23:47:54.103 INFO:teuthology.orchestra.run.smithi174.stdout:Maintainer: Ubuntu Kernel Team 2024-04-24T23:47:54.103 INFO:teuthology.orchestra.run.smithi174.stdout:Architecture: amd64 2024-04-24T23:47:54.103 INFO:teuthology.orchestra.run.smithi174.stdout:Source: linux-meta 2024-04-24T23:47:54.103 INFO:teuthology.orchestra.run.smithi174.stdout:Version: 5.15.0.105.102 2024-04-24T23:47:54.103 INFO:teuthology.orchestra.run.smithi174.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-24T23:47:54.103 INFO:teuthology.orchestra.run.smithi174.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-24T23:47:54.104 INFO:teuthology.orchestra.run.smithi174.stdout:Recommends: thermald 2024-04-24T23:47:54.104 INFO:teuthology.orchestra.run.smithi174.stdout:Description: Generic Linux kernel image 2024-04-24T23:47:54.104 INFO:teuthology.orchestra.run.smithi174.stdout: This package will always depend on the latest generic kernel image 2024-04-24T23:47:54.104 INFO:teuthology.orchestra.run.smithi174.stdout: available. 2024-04-24T23:47:54.104 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-04-24T23:47:54.104 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-24T23:47:54.104 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-24T23:47:54.104 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-04-24T23:47:54.104 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi174.front.sepia.ceph.com, path=None, version=distro) 2024-04-24T23:47:54.104 DEBUG:teuthology.orchestra.run.smithi174:> sudo apt-get clean 2024-04-24T23:47:54.212 INFO:teuthology.orchestra.run.smithi116.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-24T23:47:54.217 INFO:teuthology.orchestra.run.smithi116.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-24T23:47:54.272 DEBUG:teuthology.orchestra.run.smithi174:> sudo apt-get update 2024-04-24T23:47:54.284 INFO:teuthology.orchestra.run.smithi116.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-24T23:47:54.290 INFO:teuthology.orchestra.run.smithi116.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-24T23:47:54.424 INFO:teuthology.orchestra.run.smithi174.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-24T23:47:54.424 INFO:teuthology.orchestra.run.smithi174.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-24T23:47:54.426 INFO:teuthology.orchestra.run.smithi174.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-24T23:47:54.459 INFO:teuthology.orchestra.run.smithi116.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-24T23:47:54.459 INFO:teuthology.orchestra.run.smithi116.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-24T23:47:54.460 INFO:teuthology.orchestra.run.smithi116.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-24T23:47:54.497 INFO:teuthology.orchestra.run.smithi116.stdout:done 2024-04-24T23:47:54.509 INFO:teuthology.orchestra.run.smithi174.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-24T23:47:54.963 DEBUG:teuthology.orchestra.run.smithi116:> dpkg -s linux-image-generic 2024-04-24T23:47:54.981 INFO:teuthology.orchestra.run.smithi116.stdout:Package: linux-image-generic 2024-04-24T23:47:54.982 INFO:teuthology.orchestra.run.smithi116.stdout:Status: install ok installed 2024-04-24T23:47:54.982 INFO:teuthology.orchestra.run.smithi116.stdout:Priority: optional 2024-04-24T23:47:54.982 INFO:teuthology.orchestra.run.smithi116.stdout:Section: kernel 2024-04-24T23:47:54.982 INFO:teuthology.orchestra.run.smithi116.stdout:Installed-Size: 21 2024-04-24T23:47:54.982 INFO:teuthology.orchestra.run.smithi116.stdout:Maintainer: Ubuntu Kernel Team 2024-04-24T23:47:54.982 INFO:teuthology.orchestra.run.smithi116.stdout:Architecture: amd64 2024-04-24T23:47:54.982 INFO:teuthology.orchestra.run.smithi116.stdout:Source: linux-meta 2024-04-24T23:47:54.982 INFO:teuthology.orchestra.run.smithi116.stdout:Version: 5.15.0.105.102 2024-04-24T23:47:54.982 INFO:teuthology.orchestra.run.smithi116.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-24T23:47:54.982 INFO:teuthology.orchestra.run.smithi116.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-24T23:47:54.982 INFO:teuthology.orchestra.run.smithi116.stdout:Recommends: thermald 2024-04-24T23:47:54.982 INFO:teuthology.orchestra.run.smithi116.stdout:Description: Generic Linux kernel image 2024-04-24T23:47:54.982 INFO:teuthology.orchestra.run.smithi116.stdout: This package will always depend on the latest generic kernel image 2024-04-24T23:47:54.983 INFO:teuthology.orchestra.run.smithi116.stdout: available. 2024-04-24T23:47:54.983 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-04-24T23:47:54.983 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-24T23:47:54.983 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-24T23:47:54.983 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-24T23:47:54.983 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi116.front.sepia.ceph.com, path=None, version=distro) 2024-04-24T23:47:54.983 DEBUG:teuthology.orchestra.run.smithi116:> sudo apt-get clean 2024-04-24T23:47:55.151 DEBUG:teuthology.orchestra.run.smithi116:> sudo apt-get update 2024-04-24T23:47:55.285 INFO:teuthology.orchestra.run.smithi116.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-24T23:47:55.410 INFO:teuthology.orchestra.run.smithi116.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-24T23:47:55.482 INFO:teuthology.orchestra.run.smithi116.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-24T23:47:55.564 INFO:teuthology.orchestra.run.smithi116.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-24T23:47:55.772 INFO:teuthology.orchestra.run.smithi174.stdout:Reading package lists... 2024-04-24T23:47:55.789 DEBUG:teuthology.orchestra.run.smithi174:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-24T23:47:55.853 INFO:teuthology.orchestra.run.smithi174.stdout:Reading package lists... 2024-04-24T23:47:56.053 INFO:teuthology.orchestra.run.smithi174.stdout:Building dependency tree... 2024-04-24T23:47:56.054 INFO:teuthology.orchestra.run.smithi174.stdout:Reading state information... 2024-04-24T23:47:56.230 INFO:teuthology.orchestra.run.smithi174.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-04-24T23:47:56.230 INFO:teuthology.orchestra.run.smithi174.stdout:The following packages were automatically installed and are no longer required: 2024-04-24T23:47:56.231 INFO:teuthology.orchestra.run.smithi174.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-24T23:47:56.231 INFO:teuthology.orchestra.run.smithi174.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-24T23:47:56.231 INFO:teuthology.orchestra.run.smithi174.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-24T23:47:56.231 INFO:teuthology.orchestra.run.smithi174.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-24T23:47:56.231 INFO:teuthology.orchestra.run.smithi174.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-24T23:47:56.231 INFO:teuthology.orchestra.run.smithi174.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-24T23:47:56.272 INFO:teuthology.orchestra.run.smithi174.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-24T23:47:56.274 DEBUG:teuthology.orchestra.run.smithi174:> dpkg -s linux-image-generic 2024-04-24T23:47:56.290 INFO:teuthology.orchestra.run.smithi174.stdout:Package: linux-image-generic 2024-04-24T23:47:56.290 INFO:teuthology.orchestra.run.smithi174.stdout:Status: install ok installed 2024-04-24T23:47:56.290 INFO:teuthology.orchestra.run.smithi174.stdout:Priority: optional 2024-04-24T23:47:56.290 INFO:teuthology.orchestra.run.smithi174.stdout:Section: kernel 2024-04-24T23:47:56.291 INFO:teuthology.orchestra.run.smithi174.stdout:Installed-Size: 21 2024-04-24T23:47:56.291 INFO:teuthology.orchestra.run.smithi174.stdout:Maintainer: Ubuntu Kernel Team 2024-04-24T23:47:56.291 INFO:teuthology.orchestra.run.smithi174.stdout:Architecture: amd64 2024-04-24T23:47:56.291 INFO:teuthology.orchestra.run.smithi174.stdout:Source: linux-meta 2024-04-24T23:47:56.291 INFO:teuthology.orchestra.run.smithi174.stdout:Version: 5.15.0.105.102 2024-04-24T23:47:56.291 INFO:teuthology.orchestra.run.smithi174.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-24T23:47:56.292 INFO:teuthology.orchestra.run.smithi174.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-24T23:47:56.292 INFO:teuthology.orchestra.run.smithi174.stdout:Recommends: thermald 2024-04-24T23:47:56.292 INFO:teuthology.orchestra.run.smithi174.stdout:Description: Generic Linux kernel image 2024-04-24T23:47:56.292 INFO:teuthology.orchestra.run.smithi174.stdout: This package will always depend on the latest generic kernel image 2024-04-24T23:47:56.292 INFO:teuthology.orchestra.run.smithi174.stdout: available. 2024-04-24T23:47:56.293 DEBUG:teuthology.orchestra.run.smithi174:> mktemp 2024-04-24T23:47:56.338 INFO:teuthology.orchestra.run.smithi174.stdout:/tmp/tmp.abNX3D0AyL 2024-04-24T23:47:56.338 DEBUG:teuthology.orchestra.run.smithi174:> sudo cp /boot/grub/grub.cfg /tmp/tmp.abNX3D0AyL 2024-04-24T23:47:56.396 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 0666 /tmp/tmp.abNX3D0AyL 2024-04-24T23:47:56.516 DEBUG:teuthology.orchestra.remote:smithi174:/tmp/tmp.abNX3D0AyL is 10KB 2024-04-24T23:47:56.529 DEBUG:teuthology.orchestra.run.smithi174:> rm -fr /tmp/tmp.abNX3D0AyL 2024-04-24T23:47:56.534 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-24T23:47:56.586 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-04-24T23:47:56.587 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-24T23:47:56.587 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-24T23:47:56.654 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-04-24T23:47:56.654 DEBUG:teuthology.orchestra.run.smithi174:> sudo update-grub 2024-04-24T23:47:56.709 INFO:teuthology.orchestra.run.smithi116.stdout:Reading package lists... 2024-04-24T23:47:56.726 DEBUG:teuthology.orchestra.run.smithi116:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-24T23:47:56.777 INFO:teuthology.orchestra.run.smithi116.stdout:Reading package lists... 2024-04-24T23:47:56.966 INFO:teuthology.orchestra.run.smithi116.stdout:Building dependency tree... 2024-04-24T23:47:56.966 INFO:teuthology.orchestra.run.smithi116.stdout:Reading state information... 2024-04-24T23:47:57.120 INFO:teuthology.orchestra.run.smithi116.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-04-24T23:47:57.120 INFO:teuthology.orchestra.run.smithi116.stdout:The following packages were automatically installed and are no longer required: 2024-04-24T23:47:57.120 INFO:teuthology.orchestra.run.smithi116.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-24T23:47:57.120 INFO:teuthology.orchestra.run.smithi116.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-24T23:47:57.120 INFO:teuthology.orchestra.run.smithi116.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-24T23:47:57.120 INFO:teuthology.orchestra.run.smithi116.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-24T23:47:57.120 INFO:teuthology.orchestra.run.smithi116.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-24T23:47:57.120 INFO:teuthology.orchestra.run.smithi116.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-24T23:47:57.158 INFO:teuthology.orchestra.run.smithi116.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-24T23:47:57.160 DEBUG:teuthology.orchestra.run.smithi116:> dpkg -s linux-image-generic 2024-04-24T23:47:57.215 INFO:teuthology.orchestra.run.smithi116.stdout:Package: linux-image-generic 2024-04-24T23:47:57.216 INFO:teuthology.orchestra.run.smithi116.stdout:Status: install ok installed 2024-04-24T23:47:57.216 INFO:teuthology.orchestra.run.smithi116.stdout:Priority: optional 2024-04-24T23:47:57.216 INFO:teuthology.orchestra.run.smithi116.stdout:Section: kernel 2024-04-24T23:47:57.216 INFO:teuthology.orchestra.run.smithi116.stdout:Installed-Size: 21 2024-04-24T23:47:57.216 INFO:teuthology.orchestra.run.smithi116.stdout:Maintainer: Ubuntu Kernel Team 2024-04-24T23:47:57.216 INFO:teuthology.orchestra.run.smithi116.stdout:Architecture: amd64 2024-04-24T23:47:57.216 INFO:teuthology.orchestra.run.smithi116.stdout:Source: linux-meta 2024-04-24T23:47:57.216 INFO:teuthology.orchestra.run.smithi116.stdout:Version: 5.15.0.105.102 2024-04-24T23:47:57.216 INFO:teuthology.orchestra.run.smithi116.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-24T23:47:57.216 INFO:teuthology.orchestra.run.smithi116.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-24T23:47:57.216 INFO:teuthology.orchestra.run.smithi116.stdout:Recommends: thermald 2024-04-24T23:47:57.216 INFO:teuthology.orchestra.run.smithi116.stdout:Description: Generic Linux kernel image 2024-04-24T23:47:57.216 INFO:teuthology.orchestra.run.smithi116.stdout: This package will always depend on the latest generic kernel image 2024-04-24T23:47:57.217 INFO:teuthology.orchestra.run.smithi116.stdout: available. 2024-04-24T23:47:57.217 DEBUG:teuthology.orchestra.run.smithi116:> mktemp 2024-04-24T23:47:57.261 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/tmp.GVARZwrHIM 2024-04-24T23:47:57.261 DEBUG:teuthology.orchestra.run.smithi116:> sudo cp /boot/grub/grub.cfg /tmp/tmp.GVARZwrHIM 2024-04-24T23:47:57.316 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0666 /tmp/tmp.GVARZwrHIM 2024-04-24T23:47:57.425 DEBUG:teuthology.orchestra.remote:smithi116:/tmp/tmp.GVARZwrHIM is 10KB 2024-04-24T23:47:57.438 DEBUG:teuthology.orchestra.run.smithi116:> rm -fr /tmp/tmp.GVARZwrHIM 2024-04-24T23:47:57.443 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-24T23:47:57.496 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-04-24T23:47:57.497 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-24T23:47:57.497 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-24T23:47:57.574 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-04-24T23:47:57.574 DEBUG:teuthology.orchestra.run.smithi116:> sudo update-grub 2024-04-24T23:47:57.865 INFO:teuthology.orchestra.run.smithi174.stderr:Sourcing file `/etc/default/grub' 2024-04-24T23:47:57.867 INFO:teuthology.orchestra.run.smithi174.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-24T23:47:57.868 INFO:teuthology.orchestra.run.smithi174.stderr:Generating grub configuration file ... 2024-04-24T23:47:58.048 INFO:teuthology.orchestra.run.smithi174.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-24T23:47:58.054 INFO:teuthology.orchestra.run.smithi174.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-24T23:47:58.286 INFO:teuthology.orchestra.run.smithi174.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-24T23:47:58.291 INFO:teuthology.orchestra.run.smithi174.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-24T23:47:58.332 INFO:teuthology.orchestra.run.smithi174.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-24T23:47:58.337 INFO:teuthology.orchestra.run.smithi174.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-24T23:47:58.452 INFO:teuthology.orchestra.run.smithi174.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-24T23:47:58.453 INFO:teuthology.orchestra.run.smithi174.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-24T23:47:58.453 INFO:teuthology.orchestra.run.smithi174.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-24T23:47:58.474 INFO:teuthology.orchestra.run.smithi174.stderr:done 2024-04-24T23:47:58.488 DEBUG:teuthology.orchestra.run.smithi174:> sudo shutdown -r now 2024-04-24T23:47:58.788 INFO:teuthology.orchestra.run.smithi116.stderr:Sourcing file `/etc/default/grub' 2024-04-24T23:47:58.789 INFO:teuthology.orchestra.run.smithi116.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-24T23:47:58.792 INFO:teuthology.orchestra.run.smithi116.stderr:Generating grub configuration file ... 2024-04-24T23:47:58.960 INFO:teuthology.orchestra.run.smithi116.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-24T23:47:58.966 INFO:teuthology.orchestra.run.smithi116.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-24T23:47:59.189 INFO:teuthology.orchestra.run.smithi116.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-24T23:47:59.205 INFO:teuthology.orchestra.run.smithi116.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-24T23:47:59.236 INFO:teuthology.orchestra.run.smithi116.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-24T23:47:59.241 INFO:teuthology.orchestra.run.smithi116.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-24T23:47:59.367 INFO:teuthology.orchestra.run.smithi116.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-24T23:47:59.367 INFO:teuthology.orchestra.run.smithi116.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-24T23:47:59.367 INFO:teuthology.orchestra.run.smithi116.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-24T23:47:59.391 INFO:teuthology.orchestra.run.smithi116.stderr:done 2024-04-24T23:47:59.402 DEBUG:teuthology.orchestra.run.smithi116:> sudo shutdown -r now 2024-04-24T23:48:28.519 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-04-24T23:48:28.520 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-04-24T23:48:28.521 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T23:48:29.412 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-24T23:48:29.412 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-04-24T23:48:29.413 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T23:48:47.007 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-04-24T23:48:56.014 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-04-24T23:48:56.015 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T23:48:59.074 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.174 2024-04-24T23:49:03.902 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.116 2024-04-24T23:49:11.080 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-04-24T23:49:11.081 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T23:49:12.905 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-04-24T23:49:12.905 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T23:49:31.295 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.116 2024-04-24T23:49:42.888 DEBUG:teuthology.orchestra.run.smithi174:> true 2024-04-24T23:49:43.295 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-04-24T23:49:43.296 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T23:49:43.409 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-04-24T23:49:43.409 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-105-generic"... 2024-04-24T23:49:43.409 DEBUG:teuthology.orchestra.run.smithi174:> uname -r 2024-04-24T23:49:43.457 INFO:teuthology.orchestra.run.smithi174.stdout:5.15.0-105-generic 2024-04-24T23:49:43.458 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-04-24T23:49:43.458 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-24T23:49:43.458 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-24T23:49:43.841 DEBUG:teuthology.orchestra.run.smithi116:> true 2024-04-24T23:49:44.459 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-24T23:49:44.459 DEBUG:teuthology.orchestra.run.smithi174:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-24T23:49:44.524 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-04-24T23:49:44.524 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-105-generic"... 2024-04-24T23:49:44.524 DEBUG:teuthology.orchestra.run.smithi116:> uname -r 2024-04-24T23:49:44.546 INFO:teuthology.orchestra.run.smithi174.stdout:ttyS1 2024-04-24T23:49:44.577 INFO:teuthology.orchestra.run.smithi116.stdout:5.15.0-105-generic 2024-04-24T23:49:44.578 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-04-24T23:49:44.578 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-24T23:49:44.578 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-24T23:49:44.579 DEBUG:teuthology.parallel:result is None 2024-04-24T23:49:45.579 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-24T23:49:45.580 DEBUG:teuthology.orchestra.run.smithi116:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-24T23:49:45.684 INFO:teuthology.orchestra.run.smithi116.stdout:ttyS1 2024-04-24T23:49:45.718 DEBUG:teuthology.parallel:result is None 2024-04-24T23:49:45.718 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-24T23:49:45.734 INFO:teuthology.task.internal:Creating test directory... 2024-04-24T23:49:45.734 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-24T23:49:45.740 DEBUG:teuthology.orchestra.run.smithi174:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-24T23:49:45.746 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-24T23:49:45.771 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-24T23:49:45.813 INFO:teuthology.task.internal:Creating archive directory... 2024-04-24T23:49:45.814 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-24T23:49:45.816 DEBUG:teuthology.orchestra.run.smithi174:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-24T23:49:45.913 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-24T23:49:45.922 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-24T23:49:45.922 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-24T23:49:45.925 DEBUG:teuthology.orchestra.run.smithi174:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-24T23:49:45.942 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T23:49:45.949 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T23:49:45.949 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T23:49:45.958 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T23:49:45.960 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-24T23:49:45.970 INFO:teuthology.task.internal:Configuring sudo... 2024-04-24T23:49:45.970 DEBUG:teuthology.orchestra.run.smithi116:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-24T23:49:45.997 DEBUG:teuthology.orchestra.run.smithi174:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-24T23:49:46.031 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-24T23:49:46.046 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-24T23:49:46.046 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-24T23:49:46.057 DEBUG:teuthology.orchestra.run.smithi174:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-24T23:49:46.087 DEBUG:teuthology.orchestra.run.smithi116:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T23:49:46.110 DEBUG:teuthology.orchestra.run.smithi116:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T23:49:46.156 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-04-24T23:49:46.156 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-24T23:49:46.233 DEBUG:teuthology.orchestra.run.smithi174:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T23:49:46.240 DEBUG:teuthology.orchestra.run.smithi174:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T23:49:46.290 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-04-24T23:49:46.290 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-24T23:49:46.366 DEBUG:teuthology.orchestra.run.smithi116:> sudo service rsyslog restart 2024-04-24T23:49:46.372 DEBUG:teuthology.orchestra.run.smithi174:> sudo service rsyslog restart 2024-04-24T23:49:46.443 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-24T23:49:46.457 INFO:teuthology.task.internal:Starting timer... 2024-04-24T23:49:46.458 INFO:teuthology.run_tasks:Running task pcp... 2024-04-24T23:49:46.475 INFO:teuthology.run_tasks:Running task selinux... 2024-04-24T23:49:46.495 DEBUG:teuthology.task.selinux:Excluding smithi116: OS 'ubuntu' does not support SELinux 2024-04-24T23:49:46.495 DEBUG:teuthology.task.selinux:Excluding smithi174: OS 'ubuntu' does not support SELinux 2024-04-24T23:49:46.495 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-24T23:49:46.495 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-04-24T23:49:46.495 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-24T23:49:46.495 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-24T23:49:46.525 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-24T23:49:46.530 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-24T23:49:46.683 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-24T23:49:46.702 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-24T23:49:46.703 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi116.front.sepia.ceph.com,smithi174.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-24T23:56:48.719 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi116.front.sepia.ceph.com'), Remote(name='ubuntu@smithi174.front.sepia.ceph.com')] 2024-04-24T23:56:48.748 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-04-24T23:56:48.749 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi116.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T23:56:48.829 DEBUG:teuthology.orchestra.run.smithi116:> true 2024-04-24T23:56:48.911 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi116.front.sepia.ceph.com' 2024-04-24T23:56:48.911 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-04-24T23:56:48.911 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T23:56:49.029 DEBUG:teuthology.orchestra.run.smithi174:> true 2024-04-24T23:56:49.106 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-04-24T23:56:49.106 INFO:teuthology.run_tasks:Running task clock... 2024-04-24T23:56:49.122 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-24T23:56:49.123 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-24T23:56:49.124 DEBUG:teuthology.orchestra.run.smithi116:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T23:56:49.127 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-24T23:56:49.127 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T23:56:49.153 INFO:teuthology.orchestra.run.smithi116.stdout:24 Apr 23:56:49 ntpd[17211]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-24T23:56:49.153 INFO:teuthology.orchestra.run.smithi116.stdout:24 Apr 23:56:49 ntpd[17211]: Command line: ntpd -gq 2024-04-24T23:56:49.153 INFO:teuthology.orchestra.run.smithi116.stdout:24 Apr 23:56:49 ntpd[17211]: ---------------------------------------------------- 2024-04-24T23:56:49.153 INFO:teuthology.orchestra.run.smithi116.stdout:24 Apr 23:56:49 ntpd[17211]: ntp-4 is maintained by Network Time Foundation, 2024-04-24T23:56:49.154 INFO:teuthology.orchestra.run.smithi116.stdout:24 Apr 23:56:49 ntpd[17211]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-24T23:56:49.154 INFO:teuthology.orchestra.run.smithi116.stdout:24 Apr 23:56:49 ntpd[17211]: corporation. Support and training for ntp-4 are 2024-04-24T23:56:49.154 INFO:teuthology.orchestra.run.smithi116.stdout:24 Apr 23:56:49 ntpd[17211]: available at https://www.nwtime.org/support 2024-04-24T23:56:49.154 INFO:teuthology.orchestra.run.smithi116.stdout:24 Apr 23:56:49 ntpd[17211]: ---------------------------------------------------- 2024-04-24T23:56:49.155 INFO:teuthology.orchestra.run.smithi116.stdout:24 Apr 23:56:49 ntpd[17211]: proto: precision = 0.045 usec (-24) 2024-04-24T23:56:49.156 INFO:teuthology.orchestra.run.smithi116.stdout:24 Apr 23:56:49 ntpd[17211]: basedate set to 2022-02-04 2024-04-24T23:56:49.156 INFO:teuthology.orchestra.run.smithi116.stdout:24 Apr 23:56:49 ntpd[17211]: gps base set to 2022-02-06 (week 2196) 2024-04-24T23:56:49.156 INFO:teuthology.orchestra.run.smithi116.stdout:24 Apr 23:56:49 ntpd[17211]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-24T23:56:49.156 INFO:teuthology.orchestra.run.smithi116.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-24T23:56:49.157 INFO:teuthology.orchestra.run.smithi116.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-24T23:56:49.158 INFO:teuthology.orchestra.run.smithi116.stdout:24 Apr 23:56:49 ntpd[17211]: restrict ::: KOD does nothing without LIMITED. 2024-04-24T23:56:49.159 INFO:teuthology.orchestra.run.smithi116.stdout:24 Apr 23:56:49 ntpd[17211]: Listen and drop on 0 v6wildcard [::]:123 2024-04-24T23:56:49.159 INFO:teuthology.orchestra.run.smithi116.stdout:24 Apr 23:56:49 ntpd[17211]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-24T23:56:49.159 INFO:teuthology.orchestra.run.smithi116.stdout:24 Apr 23:56:49 ntpd[17211]: Listen normally on 2 lo 127.0.0.1:123 2024-04-24T23:56:49.159 INFO:teuthology.orchestra.run.smithi116.stdout:24 Apr 23:56:49 ntpd[17211]: Listen normally on 3 enp3s0f1 172.21.15.116:123 2024-04-24T23:56:49.159 INFO:teuthology.orchestra.run.smithi116.stdout:24 Apr 23:56:49 ntpd[17211]: Listen normally on 4 lo [::1]:123 2024-04-24T23:56:49.159 INFO:teuthology.orchestra.run.smithi116.stdout:24 Apr 23:56:49 ntpd[17211]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:3f67%5]:123 2024-04-24T23:56:49.159 INFO:teuthology.orchestra.run.smithi116.stdout:24 Apr 23:56:49 ntpd[17211]: Listening on routing socket on fd #22 for interface updates 2024-04-24T23:56:49.174 INFO:teuthology.orchestra.run.smithi174.stdout:24 Apr 23:56:49 ntpd[17239]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-24T23:56:49.174 INFO:teuthology.orchestra.run.smithi174.stdout:24 Apr 23:56:49 ntpd[17239]: Command line: ntpd -gq 2024-04-24T23:56:49.174 INFO:teuthology.orchestra.run.smithi174.stdout:24 Apr 23:56:49 ntpd[17239]: ---------------------------------------------------- 2024-04-24T23:56:49.175 INFO:teuthology.orchestra.run.smithi174.stdout:24 Apr 23:56:49 ntpd[17239]: ntp-4 is maintained by Network Time Foundation, 2024-04-24T23:56:49.175 INFO:teuthology.orchestra.run.smithi174.stdout:24 Apr 23:56:49 ntpd[17239]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-24T23:56:49.175 INFO:teuthology.orchestra.run.smithi174.stdout:24 Apr 23:56:49 ntpd[17239]: corporation. Support and training for ntp-4 are 2024-04-24T23:56:49.175 INFO:teuthology.orchestra.run.smithi174.stdout:24 Apr 23:56:49 ntpd[17239]: available at https://www.nwtime.org/support 2024-04-24T23:56:49.175 INFO:teuthology.orchestra.run.smithi174.stdout:24 Apr 23:56:49 ntpd[17239]: ---------------------------------------------------- 2024-04-24T23:56:49.175 INFO:teuthology.orchestra.run.smithi174.stdout:24 Apr 23:56:49 ntpd[17239]: proto: precision = 0.043 usec (-24) 2024-04-24T23:56:49.176 INFO:teuthology.orchestra.run.smithi174.stdout:24 Apr 23:56:49 ntpd[17239]: basedate set to 2022-02-04 2024-04-24T23:56:49.176 INFO:teuthology.orchestra.run.smithi174.stdout:24 Apr 23:56:49 ntpd[17239]: gps base set to 2022-02-06 (week 2196) 2024-04-24T23:56:49.176 INFO:teuthology.orchestra.run.smithi174.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-24T23:56:49.176 INFO:teuthology.orchestra.run.smithi174.stdout:24 Apr 23:56:49 ntpd[17239]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-24T23:56:49.176 INFO:teuthology.orchestra.run.smithi174.stdout:24 Apr 23:56:49 ntpd[17239]: restrict ::: KOD does nothing without LIMITED. 2024-04-24T23:56:49.177 INFO:teuthology.orchestra.run.smithi174.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-24T23:56:49.178 INFO:teuthology.orchestra.run.smithi174.stdout:24 Apr 23:56:49 ntpd[17239]: Listen and drop on 0 v6wildcard [::]:123 2024-04-24T23:56:49.178 INFO:teuthology.orchestra.run.smithi174.stdout:24 Apr 23:56:49 ntpd[17239]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-24T23:56:49.178 INFO:teuthology.orchestra.run.smithi174.stdout:24 Apr 23:56:49 ntpd[17239]: Listen normally on 2 lo 127.0.0.1:123 2024-04-24T23:56:49.178 INFO:teuthology.orchestra.run.smithi174.stdout:24 Apr 23:56:49 ntpd[17239]: Listen normally on 3 enp3s0f1 172.21.15.174:123 2024-04-24T23:56:49.178 INFO:teuthology.orchestra.run.smithi174.stdout:24 Apr 23:56:49 ntpd[17239]: Listen normally on 4 lo [::1]:123 2024-04-24T23:56:49.178 INFO:teuthology.orchestra.run.smithi174.stdout:24 Apr 23:56:49 ntpd[17239]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:771b%5]:123 2024-04-24T23:56:49.178 INFO:teuthology.orchestra.run.smithi174.stdout:24 Apr 23:56:49 ntpd[17239]: Listening on routing socket on fd #22 for interface updates 2024-04-24T23:56:50.154 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:50 ntpd[17211]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T23:56:50.154 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:50 ntpd[17211]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T23:56:50.154 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:50 ntpd[17211]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T23:56:50.154 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:50 ntpd[17211]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T23:56:50.154 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:50 ntpd[17211]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T23:56:50.154 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:50 ntpd[17211]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T23:56:50.154 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:50 ntpd[17211]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T23:56:50.154 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:50 ntpd[17211]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T23:56:50.174 INFO:teuthology.orchestra.run.smithi174.stderr:24 Apr 23:56:50 ntpd[17239]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T23:56:50.175 INFO:teuthology.orchestra.run.smithi174.stderr:24 Apr 23:56:50 ntpd[17239]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T23:56:50.175 INFO:teuthology.orchestra.run.smithi174.stderr:24 Apr 23:56:50 ntpd[17239]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T23:56:50.175 INFO:teuthology.orchestra.run.smithi174.stderr:24 Apr 23:56:50 ntpd[17239]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T23:56:50.175 INFO:teuthology.orchestra.run.smithi174.stderr:24 Apr 23:56:50 ntpd[17239]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T23:56:50.175 INFO:teuthology.orchestra.run.smithi174.stderr:24 Apr 23:56:50 ntpd[17239]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T23:56:52.154 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:52 ntpd[17211]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T23:56:52.154 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:52 ntpd[17211]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T23:56:52.154 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:52 ntpd[17211]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T23:56:52.154 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:52 ntpd[17211]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T23:56:52.154 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:52 ntpd[17211]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T23:56:52.154 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:52 ntpd[17211]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T23:56:52.175 INFO:teuthology.orchestra.run.smithi174.stderr:24 Apr 23:56:52 ntpd[17239]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T23:56:52.175 INFO:teuthology.orchestra.run.smithi174.stderr:24 Apr 23:56:52 ntpd[17239]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T23:56:52.175 INFO:teuthology.orchestra.run.smithi174.stderr:24 Apr 23:56:52 ntpd[17239]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T23:56:52.175 INFO:teuthology.orchestra.run.smithi174.stderr:24 Apr 23:56:52 ntpd[17239]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T23:56:52.176 INFO:teuthology.orchestra.run.smithi174.stderr:24 Apr 23:56:52 ntpd[17239]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T23:56:52.176 INFO:teuthology.orchestra.run.smithi174.stderr:24 Apr 23:56:52 ntpd[17239]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T23:56:52.176 INFO:teuthology.orchestra.run.smithi174.stderr:24 Apr 23:56:52 ntpd[17239]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T23:56:52.176 INFO:teuthology.orchestra.run.smithi174.stderr:24 Apr 23:56:52 ntpd[17239]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T23:56:52.176 INFO:teuthology.orchestra.run.smithi174.stderr:24 Apr 23:56:52 ntpd[17239]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T23:56:52.176 INFO:teuthology.orchestra.run.smithi174.stderr:24 Apr 23:56:52 ntpd[17239]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T23:56:52.176 INFO:teuthology.orchestra.run.smithi174.stderr:24 Apr 23:56:52 ntpd[17239]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T23:56:52.176 INFO:teuthology.orchestra.run.smithi174.stderr:24 Apr 23:56:52 ntpd[17239]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T23:56:53.154 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:53 ntpd[17211]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T23:56:53.154 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:53 ntpd[17211]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T23:56:53.154 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:53 ntpd[17211]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T23:56:53.154 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:53 ntpd[17211]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T23:56:53.154 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:53 ntpd[17211]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T23:56:53.154 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:53 ntpd[17211]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T23:56:53.175 INFO:teuthology.orchestra.run.smithi174.stderr:24 Apr 23:56:53 ntpd[17239]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T23:56:53.175 INFO:teuthology.orchestra.run.smithi174.stderr:24 Apr 23:56:53 ntpd[17239]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T23:56:53.175 INFO:teuthology.orchestra.run.smithi174.stderr:24 Apr 23:56:53 ntpd[17239]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T23:56:53.175 INFO:teuthology.orchestra.run.smithi174.stderr:24 Apr 23:56:53 ntpd[17239]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T23:56:53.175 INFO:teuthology.orchestra.run.smithi174.stderr:24 Apr 23:56:53 ntpd[17239]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T23:56:53.175 INFO:teuthology.orchestra.run.smithi174.stderr:24 Apr 23:56:53 ntpd[17239]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T23:56:54.154 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:54 ntpd[17211]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T23:56:54.155 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:54 ntpd[17211]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T23:56:54.155 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:54 ntpd[17211]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T23:56:54.155 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:54 ntpd[17211]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T23:56:54.155 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:54 ntpd[17211]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T23:56:54.155 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:54 ntpd[17211]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T23:56:54.155 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:54 ntpd[17211]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T23:56:54.155 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:54 ntpd[17211]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T23:56:54.155 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:54 ntpd[17211]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T23:56:54.155 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:54 ntpd[17211]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T23:56:54.155 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:54 ntpd[17211]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T23:56:54.155 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:54 ntpd[17211]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T23:56:54.175 INFO:teuthology.orchestra.run.smithi174.stderr:24 Apr 23:56:54 ntpd[17239]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T23:56:54.176 INFO:teuthology.orchestra.run.smithi174.stderr:24 Apr 23:56:54 ntpd[17239]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T23:56:54.176 INFO:teuthology.orchestra.run.smithi174.stderr:24 Apr 23:56:54 ntpd[17239]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T23:56:54.176 INFO:teuthology.orchestra.run.smithi174.stderr:24 Apr 23:56:54 ntpd[17239]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T23:56:54.176 INFO:teuthology.orchestra.run.smithi174.stderr:24 Apr 23:56:54 ntpd[17239]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T23:56:54.176 INFO:teuthology.orchestra.run.smithi174.stderr:24 Apr 23:56:54 ntpd[17239]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T23:56:54.176 INFO:teuthology.orchestra.run.smithi174.stderr:24 Apr 23:56:54 ntpd[17239]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T23:56:54.176 INFO:teuthology.orchestra.run.smithi174.stderr:24 Apr 23:56:54 ntpd[17239]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T23:56:54.176 INFO:teuthology.orchestra.run.smithi174.stderr:24 Apr 23:56:54 ntpd[17239]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T23:56:54.176 INFO:teuthology.orchestra.run.smithi174.stderr:24 Apr 23:56:54 ntpd[17239]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T23:56:54.176 INFO:teuthology.orchestra.run.smithi174.stderr:24 Apr 23:56:54 ntpd[17239]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T23:56:54.176 INFO:teuthology.orchestra.run.smithi174.stderr:24 Apr 23:56:54 ntpd[17239]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T23:56:54.176 INFO:teuthology.orchestra.run.smithi174.stderr:24 Apr 23:56:54 ntpd[17239]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T23:56:54.176 INFO:teuthology.orchestra.run.smithi174.stderr:24 Apr 23:56:54 ntpd[17239]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T23:56:55.154 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:55 ntpd[17211]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T23:56:55.154 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:55 ntpd[17211]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T23:56:55.154 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:55 ntpd[17211]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T23:56:55.155 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:55 ntpd[17211]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T23:56:55.155 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:55 ntpd[17211]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T23:56:55.155 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:55 ntpd[17211]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T23:56:55.175 INFO:teuthology.orchestra.run.smithi174.stderr:24 Apr 23:56:55 ntpd[17239]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T23:56:55.175 INFO:teuthology.orchestra.run.smithi174.stderr:24 Apr 23:56:55 ntpd[17239]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T23:56:55.175 INFO:teuthology.orchestra.run.smithi174.stderr:24 Apr 23:56:55 ntpd[17239]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T23:56:55.175 INFO:teuthology.orchestra.run.smithi174.stderr:24 Apr 23:56:55 ntpd[17239]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T23:56:55.175 INFO:teuthology.orchestra.run.smithi174.stderr:24 Apr 23:56:55 ntpd[17239]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T23:56:55.175 INFO:teuthology.orchestra.run.smithi174.stderr:24 Apr 23:56:55 ntpd[17239]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T23:56:56.154 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:56 ntpd[17211]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T23:56:56.154 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:56 ntpd[17211]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T23:56:56.154 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:56 ntpd[17211]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T23:56:56.154 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:56 ntpd[17211]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T23:56:56.155 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:56 ntpd[17211]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T23:56:56.155 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:56 ntpd[17211]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T23:56:56.155 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:56 ntpd[17211]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T23:56:56.155 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:56 ntpd[17211]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T23:56:56.155 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:56 ntpd[17211]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T23:56:56.155 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:56 ntpd[17211]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T23:56:56.155 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:56 ntpd[17211]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T23:56:56.155 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:56 ntpd[17211]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T23:56:56.175 INFO:teuthology.orchestra.run.smithi174.stderr:24 Apr 23:56:56 ntpd[17239]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T23:56:56.176 INFO:teuthology.orchestra.run.smithi174.stderr:24 Apr 23:56:56 ntpd[17239]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T23:56:56.176 INFO:teuthology.orchestra.run.smithi174.stderr:24 Apr 23:56:56 ntpd[17239]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T23:56:56.176 INFO:teuthology.orchestra.run.smithi174.stderr:24 Apr 23:56:56 ntpd[17239]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T23:56:56.176 INFO:teuthology.orchestra.run.smithi174.stderr:24 Apr 23:56:56 ntpd[17239]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T23:56:56.176 INFO:teuthology.orchestra.run.smithi174.stderr:24 Apr 23:56:56 ntpd[17239]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T23:56:56.176 INFO:teuthology.orchestra.run.smithi174.stderr:24 Apr 23:56:56 ntpd[17239]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T23:56:56.176 INFO:teuthology.orchestra.run.smithi174.stderr:24 Apr 23:56:56 ntpd[17239]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T23:56:56.176 INFO:teuthology.orchestra.run.smithi174.stderr:24 Apr 23:56:56 ntpd[17239]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T23:56:56.176 INFO:teuthology.orchestra.run.smithi174.stderr:24 Apr 23:56:56 ntpd[17239]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T23:56:56.176 INFO:teuthology.orchestra.run.smithi174.stderr:24 Apr 23:56:56 ntpd[17239]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T23:56:56.176 INFO:teuthology.orchestra.run.smithi174.stderr:24 Apr 23:56:56 ntpd[17239]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T23:56:56.176 INFO:teuthology.orchestra.run.smithi174.stdout:24 Apr 23:56:56 ntpd[17239]: ntpd: time slew -0.000511 s 2024-04-24T23:56:56.177 INFO:teuthology.orchestra.run.smithi174.stdout:ntpd: time slew -0.000511s 2024-04-24T23:56:56.177 INFO:teuthology.orchestra.run.smithi174.stderr:24 Apr 23:56:56 ntpd[17239]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-24T23:56:56.177 INFO:teuthology.orchestra.run.smithi174.stderr:24 Apr 23:56:56 ntpd[17239]: can't open /var/log/ntpstats/loopstats.20240424: Permission denied 2024-04-24T23:56:56.519 INFO:teuthology.orchestra.run.smithi174.stdout: remote refid st t when poll reach delay offset jitter 2024-04-24T23:56:56.519 INFO:teuthology.orchestra.run.smithi174.stdout:============================================================================== 2024-04-24T23:56:56.519 INFO:teuthology.orchestra.run.smithi174.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-24T23:56:56.519 INFO:teuthology.orchestra.run.smithi174.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-24T23:56:56.519 INFO:teuthology.orchestra.run.smithi174.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-24T23:56:56.519 INFO:teuthology.orchestra.run.smithi174.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-24T23:56:57.153 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:57 ntpd[17211]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T23:56:57.154 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:57 ntpd[17211]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T23:56:57.154 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:57 ntpd[17211]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T23:56:57.154 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:57 ntpd[17211]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T23:56:57.154 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:57 ntpd[17211]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T23:56:57.154 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:57 ntpd[17211]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T23:56:58.154 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:58 ntpd[17211]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T23:56:58.155 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:58 ntpd[17211]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T23:56:58.155 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:58 ntpd[17211]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T23:56:58.155 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:58 ntpd[17211]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T23:56:58.155 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:58 ntpd[17211]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T23:56:58.155 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:58 ntpd[17211]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T23:56:58.155 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:58 ntpd[17211]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T23:56:58.155 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:58 ntpd[17211]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T23:56:58.155 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:58 ntpd[17211]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-24T23:56:58.155 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:58 ntpd[17211]: can't open /var/log/ntpstats/rawstats.20240424: Permission denied 2024-04-24T23:56:58.155 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:58 ntpd[17211]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-24T23:56:58.155 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:58 ntpd[17211]: can't open /var/log/ntpstats/peerstats.20240424: Permission denied 2024-04-24T23:56:58.156 INFO:teuthology.orchestra.run.smithi116.stdout:24 Apr 23:56:58 ntpd[17211]: ntpd: time slew -0.001932 s 2024-04-24T23:56:58.156 INFO:teuthology.orchestra.run.smithi116.stdout:ntpd: time slew -0.001932s 2024-04-24T23:56:58.156 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:58 ntpd[17211]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-24T23:56:58.156 INFO:teuthology.orchestra.run.smithi116.stderr:24 Apr 23:56:58 ntpd[17211]: can't open /var/log/ntpstats/loopstats.20240424: Permission denied 2024-04-24T23:56:58.205 INFO:teuthology.orchestra.run.smithi116.stdout: remote refid st t when poll reach delay offset jitter 2024-04-24T23:56:58.205 INFO:teuthology.orchestra.run.smithi116.stdout:============================================================================== 2024-04-24T23:56:58.205 INFO:teuthology.orchestra.run.smithi116.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-24T23:56:58.205 INFO:teuthology.orchestra.run.smithi116.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-24T23:56:58.205 INFO:teuthology.orchestra.run.smithi116.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-24T23:56:58.205 INFO:teuthology.orchestra.run.smithi116.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-24T23:56:58.205 INFO:teuthology.run_tasks:Running task install... 2024-04-24T23:56:58.214 DEBUG:teuthology.task.install:project ceph 2024-04-24T23:56:58.214 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'}} 2024-04-24T23:56:58.215 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'} 2024-04-24T23:56:58.215 INFO:teuthology.task.install:Using flavor: default 2024-04-24T23:56:58.220 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-24T23:56:58.220 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-24T23:56:58.221 DEBUG:teuthology.orchestra.run.smithi116:> sudo apt-key list | grep Ceph 2024-04-24T23:56:58.221 DEBUG:teuthology.orchestra.run.smithi174:> sudo apt-key list | grep Ceph 2024-04-24T23:56:58.289 INFO:teuthology.orchestra.run.smithi174.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-24T23:56:58.307 INFO:teuthology.orchestra.run.smithi116.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-24T23:56:58.319 INFO:teuthology.orchestra.run.smithi174.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-24T23:56:58.320 INFO:teuthology.orchestra.run.smithi174.stdout:uid [ unknown] Ceph.com (release key) 2024-04-24T23:56:58.320 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2024-04-24T23:56:58.321 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2024-04-24T23:56:58.321 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-24T23:56:58.332 INFO:teuthology.orchestra.run.smithi116.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-24T23:56:58.332 INFO:teuthology.orchestra.run.smithi116.stdout:uid [ unknown] Ceph.com (release key) 2024-04-24T23:56:58.333 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2024-04-24T23:56:58.333 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2024-04-24T23:56:58.334 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-24T23:56:58.537 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default/ 2024-04-24T23:56:58.537 INFO:teuthology.task.install.deb:Package version is 19.0.0-2445-gd2c45d97-1jammy 2024-04-24T23:56:58.548 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default/ 2024-04-24T23:56:58.548 INFO:teuthology.task.install.deb:Package version is 19.0.0-2445-gd2c45d97-1jammy 2024-04-24T23:56:58.704 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-04-24T23:56:58.704 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-24T23:56:58.711 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-04-24T23:56:58.712 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-24T23:56:58.718 DEBUG:teuthology.orchestra.run.smithi174:> sudo apt-get update 2024-04-24T23:56:58.725 DEBUG:teuthology.orchestra.run.smithi116:> sudo apt-get update 2024-04-24T23:56:58.937 INFO:teuthology.orchestra.run.smithi174.stdout:Ign:1 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy InRelease 2024-04-24T23:56:58.945 INFO:teuthology.orchestra.run.smithi116.stdout:Ign:1 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy InRelease 2024-04-24T23:56:58.961 INFO:teuthology.orchestra.run.smithi174.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-24T23:56:58.966 INFO:teuthology.orchestra.run.smithi116.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-24T23:56:58.984 INFO:teuthology.orchestra.run.smithi174.stdout:Ign:3 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-24T23:56:58.989 INFO:teuthology.orchestra.run.smithi116.stdout:Ign:3 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-24T23:56:59.010 INFO:teuthology.orchestra.run.smithi116.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,410 B] 2024-04-24T23:56:59.015 INFO:teuthology.orchestra.run.smithi174.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-24T23:56:59.015 INFO:teuthology.orchestra.run.smithi174.stdout:Get:5 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.8 kB] 2024-04-24T23:56:59.025 INFO:teuthology.orchestra.run.smithi174.stdout:Hit:6 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-24T23:56:59.028 INFO:teuthology.orchestra.run.smithi116.stdout:Hit:5 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-24T23:56:59.028 INFO:teuthology.orchestra.run.smithi116.stdout:Hit:6 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-24T23:56:59.044 INFO:teuthology.orchestra.run.smithi174.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,410 B] 2024-04-24T23:56:59.049 INFO:teuthology.orchestra.run.smithi116.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.8 kB] 2024-04-24T23:56:59.086 INFO:teuthology.orchestra.run.smithi174.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-24T23:56:59.094 INFO:teuthology.orchestra.run.smithi116.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-24T23:56:59.170 INFO:teuthology.orchestra.run.smithi174.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-24T23:56:59.175 INFO:teuthology.orchestra.run.smithi116.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-24T23:56:59.380 INFO:teuthology.orchestra.run.smithi174.stdout:Fetched 28.9 kB in 0s (59.6 kB/s) 2024-04-24T23:56:59.396 INFO:teuthology.orchestra.run.smithi116.stdout:Fetched 28.9 kB in 0s (59.0 kB/s) 2024-04-24T23:57:00.345 INFO:teuthology.orchestra.run.smithi174.stdout:Reading package lists... 2024-04-24T23:57:00.361 DEBUG:teuthology.orchestra.run.smithi174:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2445-gd2c45d97-1jammy cephadm=19.0.0-2445-gd2c45d97-1jammy ceph-mds=19.0.0-2445-gd2c45d97-1jammy ceph-mgr=19.0.0-2445-gd2c45d97-1jammy ceph-common=19.0.0-2445-gd2c45d97-1jammy ceph-fuse=19.0.0-2445-gd2c45d97-1jammy ceph-test=19.0.0-2445-gd2c45d97-1jammy ceph-volume=19.0.0-2445-gd2c45d97-1jammy radosgw=19.0.0-2445-gd2c45d97-1jammy python3-rados=19.0.0-2445-gd2c45d97-1jammy python3-rgw=19.0.0-2445-gd2c45d97-1jammy python3-cephfs=19.0.0-2445-gd2c45d97-1jammy python3-rbd=19.0.0-2445-gd2c45d97-1jammy libcephfs2=19.0.0-2445-gd2c45d97-1jammy libcephfs-dev=19.0.0-2445-gd2c45d97-1jammy librados2=19.0.0-2445-gd2c45d97-1jammy librbd1=19.0.0-2445-gd2c45d97-1jammy rbd-fuse=19.0.0-2445-gd2c45d97-1jammy python3-cephfs=19.0.0-2445-gd2c45d97-1jammy cephfs-shell=19.0.0-2445-gd2c45d97-1jammy cephfs-top=19.0.0-2445-gd2c45d97-1jammy cephfs-mirror=19.0.0-2445-gd2c45d97-1jammy 2024-04-24T23:57:00.363 INFO:teuthology.orchestra.run.smithi116.stdout:Reading package lists... 2024-04-24T23:57:00.379 DEBUG:teuthology.orchestra.run.smithi116:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2445-gd2c45d97-1jammy cephadm=19.0.0-2445-gd2c45d97-1jammy ceph-mds=19.0.0-2445-gd2c45d97-1jammy ceph-mgr=19.0.0-2445-gd2c45d97-1jammy ceph-common=19.0.0-2445-gd2c45d97-1jammy ceph-fuse=19.0.0-2445-gd2c45d97-1jammy ceph-test=19.0.0-2445-gd2c45d97-1jammy ceph-volume=19.0.0-2445-gd2c45d97-1jammy radosgw=19.0.0-2445-gd2c45d97-1jammy python3-rados=19.0.0-2445-gd2c45d97-1jammy python3-rgw=19.0.0-2445-gd2c45d97-1jammy python3-cephfs=19.0.0-2445-gd2c45d97-1jammy python3-rbd=19.0.0-2445-gd2c45d97-1jammy libcephfs2=19.0.0-2445-gd2c45d97-1jammy libcephfs-dev=19.0.0-2445-gd2c45d97-1jammy librados2=19.0.0-2445-gd2c45d97-1jammy librbd1=19.0.0-2445-gd2c45d97-1jammy rbd-fuse=19.0.0-2445-gd2c45d97-1jammy python3-cephfs=19.0.0-2445-gd2c45d97-1jammy cephfs-shell=19.0.0-2445-gd2c45d97-1jammy cephfs-top=19.0.0-2445-gd2c45d97-1jammy cephfs-mirror=19.0.0-2445-gd2c45d97-1jammy 2024-04-24T23:57:00.423 INFO:teuthology.orchestra.run.smithi174.stdout:Reading package lists... 2024-04-24T23:57:00.440 INFO:teuthology.orchestra.run.smithi116.stdout:Reading package lists... 2024-04-24T23:57:00.623 INFO:teuthology.orchestra.run.smithi174.stdout:Building dependency tree... 2024-04-24T23:57:00.623 INFO:teuthology.orchestra.run.smithi174.stdout:Reading state information... 2024-04-24T23:57:00.639 INFO:teuthology.orchestra.run.smithi116.stdout:Building dependency tree... 2024-04-24T23:57:00.640 INFO:teuthology.orchestra.run.smithi116.stdout:Reading state information... 2024-04-24T23:57:00.818 INFO:teuthology.orchestra.run.smithi174.stdout:The following packages were automatically installed and are no longer required: 2024-04-24T23:57:00.818 INFO:teuthology.orchestra.run.smithi174.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-24T23:57:00.818 INFO:teuthology.orchestra.run.smithi174.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-24T23:57:00.818 INFO:teuthology.orchestra.run.smithi174.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-24T23:57:00.819 INFO:teuthology.orchestra.run.smithi174.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-24T23:57:00.819 INFO:teuthology.orchestra.run.smithi174.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-24T23:57:00.819 INFO:teuthology.orchestra.run.smithi174.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-24T23:57:00.819 INFO:teuthology.orchestra.run.smithi174.stdout:The following additional packages will be installed: 2024-04-24T23:57:00.819 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-24T23:57:00.821 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-24T23:57:00.821 INFO:teuthology.orchestra.run.smithi174.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-24T23:57:00.821 INFO:teuthology.orchestra.run.smithi174.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-24T23:57:00.821 INFO:teuthology.orchestra.run.smithi174.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-24T23:57:00.821 INFO:teuthology.orchestra.run.smithi174.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-24T23:57:00.821 INFO:teuthology.orchestra.run.smithi174.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-24T23:57:00.821 INFO:teuthology.orchestra.run.smithi174.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-24T23:57:00.821 INFO:teuthology.orchestra.run.smithi174.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-24T23:57:00.821 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2024-04-24T23:57:00.821 INFO:teuthology.orchestra.run.smithi174.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-04-24T23:57:00.821 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-04-24T23:57:00.821 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-04-24T23:57:00.821 INFO:teuthology.orchestra.run.smithi174.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-04-24T23:57:00.821 INFO:teuthology.orchestra.run.smithi174.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-04-24T23:57:00.822 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-04-24T23:57:00.822 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-04-24T23:57:00.822 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2024-04-24T23:57:00.822 INFO:teuthology.orchestra.run.smithi174.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2024-04-24T23:57:00.822 INFO:teuthology.orchestra.run.smithi174.stdout: python3-simplejson python3-singledispatch python3-sklearn 2024-04-24T23:57:00.822 INFO:teuthology.orchestra.run.smithi174.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2024-04-24T23:57:00.822 INFO:teuthology.orchestra.run.smithi174.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2024-04-24T23:57:00.822 INFO:teuthology.orchestra.run.smithi174.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2024-04-24T23:57:00.822 INFO:teuthology.orchestra.run.smithi174.stdout: qttranslations5-l10n socat unzip xmlstarlet zip 2024-04-24T23:57:00.822 INFO:teuthology.orchestra.run.smithi174.stdout:Suggested packages: 2024-04-24T23:57:00.822 INFO:teuthology.orchestra.run.smithi174.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-24T23:57:00.823 INFO:teuthology.orchestra.run.smithi174.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-24T23:57:00.823 INFO:teuthology.orchestra.run.smithi174.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-24T23:57:00.823 INFO:teuthology.orchestra.run.smithi174.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-24T23:57:00.823 INFO:teuthology.orchestra.run.smithi174.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-24T23:57:00.823 INFO:teuthology.orchestra.run.smithi174.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-24T23:57:00.823 INFO:teuthology.orchestra.run.smithi174.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-24T23:57:00.823 INFO:teuthology.orchestra.run.smithi174.stdout:Recommended packages: 2024-04-24T23:57:00.823 INFO:teuthology.orchestra.run.smithi174.stdout: btrfs-tools 2024-04-24T23:57:00.831 INFO:teuthology.orchestra.run.smithi116.stdout:The following packages were automatically installed and are no longer required: 2024-04-24T23:57:00.831 INFO:teuthology.orchestra.run.smithi116.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-24T23:57:00.831 INFO:teuthology.orchestra.run.smithi116.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-24T23:57:00.831 INFO:teuthology.orchestra.run.smithi116.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-24T23:57:00.832 INFO:teuthology.orchestra.run.smithi116.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-24T23:57:00.832 INFO:teuthology.orchestra.run.smithi116.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-24T23:57:00.832 INFO:teuthology.orchestra.run.smithi116.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-24T23:57:00.833 INFO:teuthology.orchestra.run.smithi116.stdout:The following additional packages will be installed: 2024-04-24T23:57:00.833 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-24T23:57:00.833 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-24T23:57:00.833 INFO:teuthology.orchestra.run.smithi116.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-24T23:57:00.833 INFO:teuthology.orchestra.run.smithi116.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-24T23:57:00.834 INFO:teuthology.orchestra.run.smithi116.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-24T23:57:00.834 INFO:teuthology.orchestra.run.smithi116.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-24T23:57:00.834 INFO:teuthology.orchestra.run.smithi116.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-24T23:57:00.834 INFO:teuthology.orchestra.run.smithi116.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-24T23:57:00.834 INFO:teuthology.orchestra.run.smithi116.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-24T23:57:00.834 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2024-04-24T23:57:00.834 INFO:teuthology.orchestra.run.smithi116.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-04-24T23:57:00.835 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-04-24T23:57:00.835 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-04-24T23:57:00.835 INFO:teuthology.orchestra.run.smithi116.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-04-24T23:57:00.835 INFO:teuthology.orchestra.run.smithi116.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-04-24T23:57:00.835 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-04-24T23:57:00.835 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-04-24T23:57:00.835 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2024-04-24T23:57:00.835 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2024-04-24T23:57:00.835 INFO:teuthology.orchestra.run.smithi116.stdout: python3-simplejson python3-singledispatch python3-sklearn 2024-04-24T23:57:00.835 INFO:teuthology.orchestra.run.smithi116.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2024-04-24T23:57:00.835 INFO:teuthology.orchestra.run.smithi116.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2024-04-24T23:57:00.835 INFO:teuthology.orchestra.run.smithi116.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2024-04-24T23:57:00.835 INFO:teuthology.orchestra.run.smithi116.stdout: qttranslations5-l10n socat unzip xmlstarlet zip 2024-04-24T23:57:00.836 INFO:teuthology.orchestra.run.smithi116.stdout:Suggested packages: 2024-04-24T23:57:00.836 INFO:teuthology.orchestra.run.smithi116.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-24T23:57:00.836 INFO:teuthology.orchestra.run.smithi116.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-24T23:57:00.836 INFO:teuthology.orchestra.run.smithi116.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-24T23:57:00.836 INFO:teuthology.orchestra.run.smithi116.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-24T23:57:00.836 INFO:teuthology.orchestra.run.smithi116.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-24T23:57:00.836 INFO:teuthology.orchestra.run.smithi116.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-24T23:57:00.836 INFO:teuthology.orchestra.run.smithi116.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-24T23:57:00.836 INFO:teuthology.orchestra.run.smithi116.stdout:Recommended packages: 2024-04-24T23:57:00.836 INFO:teuthology.orchestra.run.smithi116.stdout: btrfs-tools 2024-04-24T23:57:00.929 INFO:teuthology.orchestra.run.smithi174.stdout:The following NEW packages will be installed: 2024-04-24T23:57:00.929 INFO:teuthology.orchestra.run.smithi174.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-24T23:57:00.929 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-24T23:57:00.930 INFO:teuthology.orchestra.run.smithi174.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-04-24T23:57:00.930 INFO:teuthology.orchestra.run.smithi174.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2024-04-24T23:57:00.930 INFO:teuthology.orchestra.run.smithi174.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-24T23:57:00.930 INFO:teuthology.orchestra.run.smithi174.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2024-04-24T23:57:00.930 INFO:teuthology.orchestra.run.smithi174.stdout: libradosstriper1 librbd1 librdkafka1 libreadline-dev librgw2 2024-04-24T23:57:00.931 INFO:teuthology.orchestra.run.smithi174.stdout: libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 2024-04-24T23:57:00.931 INFO:teuthology.orchestra.run.smithi174.stdout: luarocks nvme-cli pkg-config python-asyncssh-doc python-babel-localedata 2024-04-24T23:57:00.931 INFO:teuthology.orchestra.run.smithi174.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-04-24T23:57:00.931 INFO:teuthology.orchestra.run.smithi174.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-04-24T23:57:00.931 INFO:teuthology.orchestra.run.smithi174.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2024-04-24T23:57:00.931 INFO:teuthology.orchestra.run.smithi174.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2024-04-24T23:57:00.931 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-24T23:57:00.932 INFO:teuthology.orchestra.run.smithi174.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-24T23:57:00.932 INFO:teuthology.orchestra.run.smithi174.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-24T23:57:00.932 INFO:teuthology.orchestra.run.smithi174.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-24T23:57:00.932 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-24T23:57:00.932 INFO:teuthology.orchestra.run.smithi174.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-24T23:57:00.932 INFO:teuthology.orchestra.run.smithi174.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2024-04-24T23:57:00.932 INFO:teuthology.orchestra.run.smithi174.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2024-04-24T23:57:00.932 INFO:teuthology.orchestra.run.smithi174.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-04-24T23:57:00.932 INFO:teuthology.orchestra.run.smithi174.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-24T23:57:00.932 INFO:teuthology.orchestra.run.smithi174.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-24T23:57:00.932 INFO:teuthology.orchestra.run.smithi174.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-24T23:57:00.932 INFO:teuthology.orchestra.run.smithi174.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-24T23:57:00.932 INFO:teuthology.orchestra.run.smithi174.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2024-04-24T23:57:00.932 INFO:teuthology.orchestra.run.smithi174.stdout: socat unzip xmlstarlet zip 2024-04-24T23:57:00.940 INFO:teuthology.orchestra.run.smithi116.stdout:The following NEW packages will be installed: 2024-04-24T23:57:00.940 INFO:teuthology.orchestra.run.smithi116.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-24T23:57:00.940 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-24T23:57:00.940 INFO:teuthology.orchestra.run.smithi116.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-04-24T23:57:00.940 INFO:teuthology.orchestra.run.smithi116.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2024-04-24T23:57:00.942 INFO:teuthology.orchestra.run.smithi116.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-24T23:57:00.942 INFO:teuthology.orchestra.run.smithi116.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2024-04-24T23:57:00.942 INFO:teuthology.orchestra.run.smithi116.stdout: libradosstriper1 librbd1 librdkafka1 libreadline-dev librgw2 2024-04-24T23:57:00.942 INFO:teuthology.orchestra.run.smithi116.stdout: libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 2024-04-24T23:57:00.942 INFO:teuthology.orchestra.run.smithi116.stdout: luarocks nvme-cli pkg-config python-asyncssh-doc python-babel-localedata 2024-04-24T23:57:00.942 INFO:teuthology.orchestra.run.smithi116.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-04-24T23:57:00.942 INFO:teuthology.orchestra.run.smithi116.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-04-24T23:57:00.942 INFO:teuthology.orchestra.run.smithi116.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2024-04-24T23:57:00.942 INFO:teuthology.orchestra.run.smithi116.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2024-04-24T23:57:00.942 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-24T23:57:00.942 INFO:teuthology.orchestra.run.smithi116.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-24T23:57:00.942 INFO:teuthology.orchestra.run.smithi116.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-24T23:57:00.942 INFO:teuthology.orchestra.run.smithi116.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-24T23:57:00.943 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-24T23:57:00.943 INFO:teuthology.orchestra.run.smithi116.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-24T23:57:00.943 INFO:teuthology.orchestra.run.smithi116.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2024-04-24T23:57:00.943 INFO:teuthology.orchestra.run.smithi116.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2024-04-24T23:57:00.943 INFO:teuthology.orchestra.run.smithi116.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-04-24T23:57:00.943 INFO:teuthology.orchestra.run.smithi116.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-24T23:57:00.943 INFO:teuthology.orchestra.run.smithi116.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-24T23:57:00.943 INFO:teuthology.orchestra.run.smithi116.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-24T23:57:00.943 INFO:teuthology.orchestra.run.smithi116.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-24T23:57:00.943 INFO:teuthology.orchestra.run.smithi116.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2024-04-24T23:57:00.943 INFO:teuthology.orchestra.run.smithi116.stdout: socat unzip xmlstarlet zip 2024-04-24T23:57:00.997 INFO:teuthology.orchestra.run.smithi174.stdout:0 upgraded, 124 newly installed, 0 to remove and 302 not upgraded. 2024-04-24T23:57:00.997 INFO:teuthology.orchestra.run.smithi174.stdout:Need to get 180 MB of archives. 2024-04-24T23:57:00.997 INFO:teuthology.orchestra.run.smithi174.stdout:After this operation, 823 MB of additional disk space will be used. 2024-04-24T23:57:00.997 INFO:teuthology.orchestra.run.smithi174.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2024-04-24T23:57:01.014 INFO:teuthology.orchestra.run.smithi116.stdout:0 upgraded, 124 newly installed, 0 to remove and 302 not upgraded. 2024-04-24T23:57:01.014 INFO:teuthology.orchestra.run.smithi116.stdout:Need to get 180 MB of archives. 2024-04-24T23:57:01.014 INFO:teuthology.orchestra.run.smithi116.stdout:After this operation, 823 MB of additional disk space will be used. 2024-04-24T23:57:01.014 INFO:teuthology.orchestra.run.smithi116.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2024-04-24T23:57:01.018 INFO:teuthology.orchestra.run.smithi174.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-24T23:57:01.029 INFO:teuthology.orchestra.run.smithi174.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2024-04-24T23:57:01.035 INFO:teuthology.orchestra.run.smithi116.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-24T23:57:01.047 INFO:teuthology.orchestra.run.smithi116.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2024-04-24T23:57:01.075 INFO:teuthology.orchestra.run.smithi174.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2445-gd2c45d97-1jammy [3,540 kB] 2024-04-24T23:57:01.078 INFO:teuthology.orchestra.run.smithi174.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-24T23:57:01.096 INFO:teuthology.orchestra.run.smithi116.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-24T23:57:01.103 INFO:teuthology.orchestra.run.smithi174.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-24T23:57:01.107 INFO:teuthology.orchestra.run.smithi174.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-24T23:57:01.119 INFO:teuthology.orchestra.run.smithi174.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-24T23:57:01.121 INFO:teuthology.orchestra.run.smithi116.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-24T23:57:01.125 INFO:teuthology.orchestra.run.smithi116.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-24T23:57:01.138 INFO:teuthology.orchestra.run.smithi116.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-24T23:57:01.157 INFO:teuthology.orchestra.run.smithi116.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2445-gd2c45d97-1jammy [3,540 kB] 2024-04-24T23:57:01.174 INFO:teuthology.orchestra.run.smithi174.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-24T23:57:01.178 INFO:teuthology.orchestra.run.smithi174.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-24T23:57:01.190 INFO:teuthology.orchestra.run.smithi174.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-24T23:57:01.192 INFO:teuthology.orchestra.run.smithi116.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-24T23:57:01.194 INFO:teuthology.orchestra.run.smithi174.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-24T23:57:01.195 INFO:teuthology.orchestra.run.smithi174.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-24T23:57:01.197 INFO:teuthology.orchestra.run.smithi116.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-24T23:57:01.202 INFO:teuthology.orchestra.run.smithi174.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-24T23:57:01.204 INFO:teuthology.orchestra.run.smithi174.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-24T23:57:01.206 INFO:teuthology.orchestra.run.smithi174.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-24T23:57:01.208 INFO:teuthology.orchestra.run.smithi174.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-24T23:57:01.208 INFO:teuthology.orchestra.run.smithi174.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-24T23:57:01.210 INFO:teuthology.orchestra.run.smithi116.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-24T23:57:01.215 INFO:teuthology.orchestra.run.smithi116.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-24T23:57:01.216 INFO:teuthology.orchestra.run.smithi116.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-24T23:57:01.226 INFO:teuthology.orchestra.run.smithi116.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-24T23:57:01.227 INFO:teuthology.orchestra.run.smithi174.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-24T23:57:01.228 INFO:teuthology.orchestra.run.smithi116.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-24T23:57:01.230 INFO:teuthology.orchestra.run.smithi174.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-24T23:57:01.231 INFO:teuthology.orchestra.run.smithi116.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-24T23:57:01.231 INFO:teuthology.orchestra.run.smithi174.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-24T23:57:01.232 INFO:teuthology.orchestra.run.smithi174.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-24T23:57:01.232 INFO:teuthology.orchestra.run.smithi174.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-24T23:57:01.233 INFO:teuthology.orchestra.run.smithi116.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-24T23:57:01.233 INFO:teuthology.orchestra.run.smithi116.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-24T23:57:01.233 INFO:teuthology.orchestra.run.smithi174.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-24T23:57:01.238 INFO:teuthology.orchestra.run.smithi116.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-24T23:57:01.241 INFO:teuthology.orchestra.run.smithi116.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-24T23:57:01.248 INFO:teuthology.orchestra.run.smithi174.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-24T23:57:01.248 INFO:teuthology.orchestra.run.smithi174.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-24T23:57:01.248 INFO:teuthology.orchestra.run.smithi174.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-24T23:57:01.249 INFO:teuthology.orchestra.run.smithi174.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-24T23:57:01.256 INFO:teuthology.orchestra.run.smithi116.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-24T23:57:01.257 INFO:teuthology.orchestra.run.smithi116.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-24T23:57:01.257 INFO:teuthology.orchestra.run.smithi116.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-24T23:57:01.259 INFO:teuthology.orchestra.run.smithi116.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-24T23:57:01.259 INFO:teuthology.orchestra.run.smithi116.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-24T23:57:01.259 INFO:teuthology.orchestra.run.smithi116.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-24T23:57:01.260 INFO:teuthology.orchestra.run.smithi116.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-24T23:57:01.260 INFO:teuthology.orchestra.run.smithi116.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-24T23:57:01.268 INFO:teuthology.orchestra.run.smithi174.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-24T23:57:01.269 INFO:teuthology.orchestra.run.smithi174.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-24T23:57:01.271 INFO:teuthology.orchestra.run.smithi174.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-24T23:57:01.272 INFO:teuthology.orchestra.run.smithi174.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-24T23:57:01.272 INFO:teuthology.orchestra.run.smithi174.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-24T23:57:01.273 INFO:teuthology.orchestra.run.smithi174.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-24T23:57:01.279 INFO:teuthology.orchestra.run.smithi116.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-24T23:57:01.279 INFO:teuthology.orchestra.run.smithi116.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-24T23:57:01.285 INFO:teuthology.orchestra.run.smithi174.stdout:Get:35 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2445-gd2c45d97-1jammy [3,218 kB] 2024-04-24T23:57:01.289 INFO:teuthology.orchestra.run.smithi174.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-24T23:57:01.289 INFO:teuthology.orchestra.run.smithi174.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-24T23:57:01.290 INFO:teuthology.orchestra.run.smithi174.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-24T23:57:01.291 INFO:teuthology.orchestra.run.smithi174.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-24T23:57:01.300 INFO:teuthology.orchestra.run.smithi116.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-24T23:57:01.300 INFO:teuthology.orchestra.run.smithi116.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-24T23:57:01.301 INFO:teuthology.orchestra.run.smithi116.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-24T23:57:01.301 INFO:teuthology.orchestra.run.smithi116.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-24T23:57:01.302 INFO:teuthology.orchestra.run.smithi116.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-24T23:57:01.303 INFO:teuthology.orchestra.run.smithi116.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-24T23:57:01.303 INFO:teuthology.orchestra.run.smithi116.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-24T23:57:01.304 INFO:teuthology.orchestra.run.smithi116.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-24T23:57:01.309 INFO:teuthology.orchestra.run.smithi174.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-24T23:57:01.310 INFO:teuthology.orchestra.run.smithi174.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-24T23:57:01.315 INFO:teuthology.orchestra.run.smithi174.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-24T23:57:01.315 INFO:teuthology.orchestra.run.smithi174.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-24T23:57:01.316 INFO:teuthology.orchestra.run.smithi174.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-24T23:57:01.316 INFO:teuthology.orchestra.run.smithi174.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-24T23:57:01.320 INFO:teuthology.orchestra.run.smithi116.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-24T23:57:01.320 INFO:teuthology.orchestra.run.smithi116.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-24T23:57:01.330 INFO:teuthology.orchestra.run.smithi174.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-24T23:57:01.332 INFO:teuthology.orchestra.run.smithi174.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-24T23:57:01.333 INFO:teuthology.orchestra.run.smithi174.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-24T23:57:01.334 INFO:teuthology.orchestra.run.smithi174.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-24T23:57:01.340 INFO:teuthology.orchestra.run.smithi116.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-24T23:57:01.341 INFO:teuthology.orchestra.run.smithi116.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-24T23:57:01.341 INFO:teuthology.orchestra.run.smithi116.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-24T23:57:01.342 INFO:teuthology.orchestra.run.smithi116.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-24T23:57:01.344 INFO:teuthology.orchestra.run.smithi116.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-24T23:57:01.346 INFO:teuthology.orchestra.run.smithi116.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-24T23:57:01.346 INFO:teuthology.orchestra.run.smithi116.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-24T23:57:01.348 INFO:teuthology.orchestra.run.smithi116.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-24T23:57:01.350 INFO:teuthology.orchestra.run.smithi174.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-24T23:57:01.353 INFO:teuthology.orchestra.run.smithi174.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-24T23:57:01.353 INFO:teuthology.orchestra.run.smithi174.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-24T23:57:01.354 INFO:teuthology.orchestra.run.smithi174.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-24T23:57:01.356 INFO:teuthology.orchestra.run.smithi174.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-24T23:57:01.370 INFO:teuthology.orchestra.run.smithi116.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-24T23:57:01.373 INFO:teuthology.orchestra.run.smithi116.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-24T23:57:01.389 INFO:teuthology.orchestra.run.smithi174.stdout:Get:55 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2445-gd2c45d97-1jammy [711 kB] 2024-04-24T23:57:01.397 INFO:teuthology.orchestra.run.smithi116.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-24T23:57:01.398 INFO:teuthology.orchestra.run.smithi116.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-24T23:57:01.404 INFO:teuthology.orchestra.run.smithi116.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-24T23:57:01.411 INFO:teuthology.orchestra.run.smithi174.stdout:Get:56 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2445-gd2c45d97-1jammy [356 kB] 2024-04-24T23:57:01.413 INFO:teuthology.orchestra.run.smithi174.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-24T23:57:01.414 INFO:teuthology.orchestra.run.smithi174.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-24T23:57:01.417 INFO:teuthology.orchestra.run.smithi174.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-24T23:57:01.417 INFO:teuthology.orchestra.run.smithi174.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-24T23:57:01.420 INFO:teuthology.orchestra.run.smithi174.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-24T23:57:01.426 INFO:teuthology.orchestra.run.smithi174.stdout:Get:62 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2445-gd2c45d97-1jammy [32.8 kB] 2024-04-24T23:57:01.427 INFO:teuthology.orchestra.run.smithi174.stdout:Get:63 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2445-gd2c45d97-1jammy [180 kB] 2024-04-24T23:57:01.434 INFO:teuthology.orchestra.run.smithi174.stdout:Get:64 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2445-gd2c45d97-1jammy [65.7 kB] 2024-04-24T23:57:01.435 INFO:teuthology.orchestra.run.smithi174.stdout:Get:65 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2445-gd2c45d97-1jammy [330 kB] 2024-04-24T23:57:01.446 INFO:teuthology.orchestra.run.smithi174.stdout:Get:66 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2445-gd2c45d97-1jammy [6,775 kB] 2024-04-24T23:57:01.450 INFO:teuthology.orchestra.run.smithi174.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-24T23:57:01.452 INFO:teuthology.orchestra.run.smithi174.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-24T23:57:01.453 INFO:teuthology.orchestra.run.smithi174.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-24T23:57:01.487 INFO:teuthology.orchestra.run.smithi174.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-24T23:57:01.487 INFO:teuthology.orchestra.run.smithi174.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-24T23:57:01.488 INFO:teuthology.orchestra.run.smithi174.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-24T23:57:01.489 INFO:teuthology.orchestra.run.smithi174.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-24T23:57:01.489 INFO:teuthology.orchestra.run.smithi174.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-24T23:57:01.489 INFO:teuthology.orchestra.run.smithi116.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-24T23:57:01.490 INFO:teuthology.orchestra.run.smithi174.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-24T23:57:01.491 INFO:teuthology.orchestra.run.smithi116.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-24T23:57:01.493 INFO:teuthology.orchestra.run.smithi116.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-24T23:57:01.493 INFO:teuthology.orchestra.run.smithi116.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-24T23:57:01.495 INFO:teuthology.orchestra.run.smithi116.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-24T23:57:01.508 INFO:teuthology.orchestra.run.smithi174.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-24T23:57:01.509 INFO:teuthology.orchestra.run.smithi174.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-24T23:57:01.513 INFO:teuthology.orchestra.run.smithi174.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-24T23:57:01.521 INFO:teuthology.orchestra.run.smithi116.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-24T23:57:01.524 INFO:teuthology.orchestra.run.smithi116.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-24T23:57:01.524 INFO:teuthology.orchestra.run.smithi116.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-24T23:57:01.534 INFO:teuthology.orchestra.run.smithi174.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-24T23:57:01.536 INFO:teuthology.orchestra.run.smithi174.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-24T23:57:01.537 INFO:teuthology.orchestra.run.smithi174.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-24T23:57:01.541 INFO:teuthology.orchestra.run.smithi174.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-24T23:57:01.543 INFO:teuthology.orchestra.run.smithi116.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-24T23:57:01.543 INFO:teuthology.orchestra.run.smithi116.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-24T23:57:01.544 INFO:teuthology.orchestra.run.smithi116.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-24T23:57:01.544 INFO:teuthology.orchestra.run.smithi174.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2024-04-24T23:57:01.545 INFO:teuthology.orchestra.run.smithi174.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-24T23:57:01.545 INFO:teuthology.orchestra.run.smithi116.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-24T23:57:01.545 INFO:teuthology.orchestra.run.smithi116.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-24T23:57:01.546 INFO:teuthology.orchestra.run.smithi116.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-24T23:57:01.546 INFO:teuthology.orchestra.run.smithi174.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-24T23:57:01.554 INFO:teuthology.orchestra.run.smithi174.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-24T23:57:01.561 INFO:teuthology.orchestra.run.smithi174.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-24T23:57:01.561 INFO:teuthology.orchestra.run.smithi174.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-24T23:57:01.565 INFO:teuthology.orchestra.run.smithi116.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-24T23:57:01.565 INFO:teuthology.orchestra.run.smithi116.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-24T23:57:01.569 INFO:teuthology.orchestra.run.smithi116.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-24T23:57:01.571 INFO:teuthology.orchestra.run.smithi116.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-24T23:57:01.572 INFO:teuthology.orchestra.run.smithi116.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-24T23:57:01.572 INFO:teuthology.orchestra.run.smithi116.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-24T23:57:01.575 INFO:teuthology.orchestra.run.smithi174.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-24T23:57:01.575 INFO:teuthology.orchestra.run.smithi174.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-24T23:57:01.576 INFO:teuthology.orchestra.run.smithi174.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-24T23:57:01.577 INFO:teuthology.orchestra.run.smithi174.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-24T23:57:01.579 INFO:teuthology.orchestra.run.smithi174.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-24T23:57:01.580 INFO:teuthology.orchestra.run.smithi174.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-24T23:57:01.585 INFO:teuthology.orchestra.run.smithi116.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-24T23:57:01.588 INFO:teuthology.orchestra.run.smithi116.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2024-04-24T23:57:01.588 INFO:teuthology.orchestra.run.smithi116.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-24T23:57:01.589 INFO:teuthology.orchestra.run.smithi174.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-24T23:57:01.589 INFO:teuthology.orchestra.run.smithi116.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-24T23:57:01.595 INFO:teuthology.orchestra.run.smithi174.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-24T23:57:01.596 INFO:teuthology.orchestra.run.smithi174.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-24T23:57:01.599 INFO:teuthology.orchestra.run.smithi174.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-24T23:57:01.606 INFO:teuthology.orchestra.run.smithi116.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-24T23:57:01.610 INFO:teuthology.orchestra.run.smithi116.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-24T23:57:01.611 INFO:teuthology.orchestra.run.smithi116.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-24T23:57:01.614 INFO:teuthology.orchestra.run.smithi116.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-24T23:57:01.614 INFO:teuthology.orchestra.run.smithi116.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-24T23:57:01.615 INFO:teuthology.orchestra.run.smithi116.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-24T23:57:01.616 INFO:teuthology.orchestra.run.smithi174.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-24T23:57:01.635 INFO:teuthology.orchestra.run.smithi116.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-24T23:57:01.637 INFO:teuthology.orchestra.run.smithi116.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-24T23:57:01.638 INFO:teuthology.orchestra.run.smithi116.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-24T23:57:01.645 INFO:teuthology.orchestra.run.smithi116.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-24T23:57:01.663 INFO:teuthology.orchestra.run.smithi116.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-24T23:57:01.664 INFO:teuthology.orchestra.run.smithi116.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-24T23:57:01.667 INFO:teuthology.orchestra.run.smithi116.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-24T23:57:01.668 INFO:teuthology.orchestra.run.smithi116.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-24T23:57:01.681 INFO:teuthology.orchestra.run.smithi174.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2445-gd2c45d97-1jammy [112 kB] 2024-04-24T23:57:01.684 INFO:teuthology.orchestra.run.smithi174.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2445-gd2c45d97-1jammy [466 kB] 2024-04-24T23:57:01.700 INFO:teuthology.orchestra.run.smithi174.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2445-gd2c45d97-1jammy [25.6 MB] 2024-04-24T23:57:02.694 INFO:teuthology.orchestra.run.smithi174.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2445-gd2c45d97-1jammy [5,084 kB] 2024-04-24T23:57:02.797 INFO:teuthology.orchestra.run.smithi174.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2445-gd2c45d97-1jammy [244 kB] 2024-04-24T23:57:02.804 INFO:teuthology.orchestra.run.smithi174.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2445-gd2c45d97-1jammy [124 kB] 2024-04-24T23:57:02.808 INFO:teuthology.orchestra.run.smithi174.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2445-gd2c45d97-1jammy [1,525 kB] 2024-04-24T23:57:02.836 INFO:teuthology.orchestra.run.smithi116.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2445-gd2c45d97-1jammy [3,218 kB] 2024-04-24T23:57:02.867 INFO:teuthology.orchestra.run.smithi174.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2445-gd2c45d97-1jammy [6,245 kB] 2024-04-24T23:57:03.108 INFO:teuthology.orchestra.run.smithi174.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2445-gd2c45d97-1jammy [22.7 MB] 2024-04-24T23:57:03.538 INFO:teuthology.orchestra.run.smithi116.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2445-gd2c45d97-1jammy [711 kB] 2024-04-24T23:57:03.677 INFO:teuthology.orchestra.run.smithi116.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2445-gd2c45d97-1jammy [356 kB] 2024-04-24T23:57:03.743 INFO:teuthology.orchestra.run.smithi116.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2445-gd2c45d97-1jammy [32.8 kB] 2024-04-24T23:57:03.748 INFO:teuthology.orchestra.run.smithi116.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2445-gd2c45d97-1jammy [180 kB] 2024-04-24T23:57:03.774 INFO:teuthology.orchestra.run.smithi116.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2445-gd2c45d97-1jammy [65.7 kB] 2024-04-24T23:57:03.789 INFO:teuthology.orchestra.run.smithi116.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2445-gd2c45d97-1jammy [330 kB] 2024-04-24T23:57:03.843 INFO:teuthology.orchestra.run.smithi116.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2445-gd2c45d97-1jammy [6,775 kB] 2024-04-24T23:57:04.079 INFO:teuthology.orchestra.run.smithi174.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2445-gd2c45d97-1jammy [14.0 kB] 2024-04-24T23:57:04.080 INFO:teuthology.orchestra.run.smithi174.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2445-gd2c45d97-1jammy [869 kB] 2024-04-24T23:57:04.096 INFO:teuthology.orchestra.run.smithi174.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2445-gd2c45d97-1jammy [2,437 kB] 2024-04-24T23:57:04.144 INFO:teuthology.orchestra.run.smithi174.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2445-gd2c45d97-1jammy [790 kB] 2024-04-24T23:57:04.178 INFO:teuthology.orchestra.run.smithi174.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2445-gd2c45d97-1jammy [147 kB] 2024-04-24T23:57:04.183 INFO:teuthology.orchestra.run.smithi174.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2445-gd2c45d97-1jammy [2,293 kB] 2024-04-24T23:57:04.283 INFO:teuthology.orchestra.run.smithi174.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2445-gd2c45d97-1jammy [8,625 kB] 2024-04-24T23:57:04.660 INFO:teuthology.orchestra.run.smithi174.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2445-gd2c45d97-1jammy [14.2 kB] 2024-04-24T23:57:04.661 INFO:teuthology.orchestra.run.smithi174.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2445-gd2c45d97-1jammy [50.7 MB] 2024-04-24T23:57:04.790 INFO:teuthology.orchestra.run.smithi116.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2445-gd2c45d97-1jammy [112 kB] 2024-04-24T23:57:04.799 INFO:teuthology.orchestra.run.smithi116.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2445-gd2c45d97-1jammy [466 kB] 2024-04-24T23:57:04.845 INFO:teuthology.orchestra.run.smithi116.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2445-gd2c45d97-1jammy [25.6 MB] 2024-04-24T23:57:06.624 INFO:teuthology.orchestra.run.smithi116.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2445-gd2c45d97-1jammy [5,084 kB] 2024-04-24T23:57:06.815 INFO:teuthology.orchestra.run.smithi116.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2445-gd2c45d97-1jammy [244 kB] 2024-04-24T23:57:06.826 INFO:teuthology.orchestra.run.smithi116.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2445-gd2c45d97-1jammy [124 kB] 2024-04-24T23:57:06.831 INFO:teuthology.orchestra.run.smithi116.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2445-gd2c45d97-1jammy [1,525 kB] 2024-04-24T23:57:06.915 INFO:teuthology.orchestra.run.smithi116.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2445-gd2c45d97-1jammy [6,245 kB] 2024-04-24T23:57:07.251 INFO:teuthology.orchestra.run.smithi116.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2445-gd2c45d97-1jammy [22.7 MB] 2024-04-24T23:57:08.548 INFO:teuthology.orchestra.run.smithi116.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2445-gd2c45d97-1jammy [14.0 kB] 2024-04-24T23:57:08.548 INFO:teuthology.orchestra.run.smithi116.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2445-gd2c45d97-1jammy [869 kB] 2024-04-24T23:57:08.566 INFO:teuthology.orchestra.run.smithi116.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2445-gd2c45d97-1jammy [2,437 kB] 2024-04-24T23:57:08.619 INFO:teuthology.orchestra.run.smithi116.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2445-gd2c45d97-1jammy [790 kB] 2024-04-24T23:57:08.645 INFO:teuthology.orchestra.run.smithi116.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2445-gd2c45d97-1jammy [147 kB] 2024-04-24T23:57:08.659 INFO:teuthology.orchestra.run.smithi116.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2445-gd2c45d97-1jammy [2,293 kB] 2024-04-24T23:57:08.753 INFO:teuthology.orchestra.run.smithi116.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2445-gd2c45d97-1jammy [8,625 kB] 2024-04-24T23:57:08.813 INFO:teuthology.orchestra.run.smithi174.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2445-gd2c45d97-1jammy [121 kB] 2024-04-24T23:57:08.815 INFO:teuthology.orchestra.run.smithi174.stdout:Get:119 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 19.0.0-2445-gd2c45d97-1jammy [229 kB] 2024-04-24T23:57:08.820 INFO:teuthology.orchestra.run.smithi174.stdout:Get:120 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 19.0.0-2445-gd2c45d97-1jammy [34.0 kB] 2024-04-24T23:57:08.821 INFO:teuthology.orchestra.run.smithi174.stdout:Get:121 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 19.0.0-2445-gd2c45d97-1jammy [23.4 kB] 2024-04-24T23:57:08.822 INFO:teuthology.orchestra.run.smithi174.stdout:Get:122 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2445-gd2c45d97-1jammy [38.9 kB] 2024-04-24T23:57:08.823 INFO:teuthology.orchestra.run.smithi174.stdout:Get:123 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2445-gd2c45d97-1jammy [13.4 MB] 2024-04-24T23:57:09.149 INFO:teuthology.orchestra.run.smithi116.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2445-gd2c45d97-1jammy [14.2 kB] 2024-04-24T23:57:09.149 INFO:teuthology.orchestra.run.smithi116.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2445-gd2c45d97-1jammy [50.7 MB] 2024-04-24T23:57:09.772 INFO:teuthology.orchestra.run.smithi174.stdout:Get:124 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2445-gd2c45d97-1jammy [91.2 kB] 2024-04-24T23:57:10.393 INFO:teuthology.orchestra.run.smithi174.stdout:Fetched 180 MB in 9s (20.4 MB/s) 2024-04-24T23:57:10.532 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-pyperclip. 2024-04-24T23:57:10.631 INFO:teuthology.orchestra.run.smithi174.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-24T23:57:10.637 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2024-04-24T23:57:10.680 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2024-04-24T23:57:10.982 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-24T23:57:11.009 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-24T23:57:11.049 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-24T23:57:11.286 INFO:teuthology.orchestra.run.smithi116.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2445-gd2c45d97-1jammy [121 kB] 2024-04-24T23:57:11.288 INFO:teuthology.orchestra.run.smithi116.stdout:Get:119 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 19.0.0-2445-gd2c45d97-1jammy [229 kB] 2024-04-24T23:57:11.293 INFO:teuthology.orchestra.run.smithi116.stdout:Get:120 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 19.0.0-2445-gd2c45d97-1jammy [34.0 kB] 2024-04-24T23:57:11.294 INFO:teuthology.orchestra.run.smithi116.stdout:Get:121 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 19.0.0-2445-gd2c45d97-1jammy [23.4 kB] 2024-04-24T23:57:11.294 INFO:teuthology.orchestra.run.smithi116.stdout:Get:122 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2445-gd2c45d97-1jammy [38.9 kB] 2024-04-24T23:57:11.296 INFO:teuthology.orchestra.run.smithi116.stdout:Get:123 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2445-gd2c45d97-1jammy [13.4 MB] 2024-04-24T23:57:11.318 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-cmd2. 2024-04-24T23:57:11.345 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2024-04-24T23:57:11.377 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2024-04-24T23:57:11.646 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-24T23:57:11.647 INFO:teuthology.orchestra.run.smithi116.stdout:Get:124 https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2445-gd2c45d97-1jammy [91.2 kB] 2024-04-24T23:57:11.661 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../003-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-24T23:57:11.738 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-24T23:57:12.075 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-24T23:57:12.101 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../004-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-24T23:57:12.150 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-24T23:57:12.266 INFO:teuthology.orchestra.run.smithi116.stdout:Fetched 180 MB in 11s (16.8 MB/s) 2024-04-24T23:57:12.440 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-pyperclip. 2024-04-24T23:57:12.477 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-24T23:57:12.504 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../005-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-24T23:57:12.541 INFO:teuthology.orchestra.run.smithi116.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-24T23:57:12.544 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-24T23:57:12.546 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2024-04-24T23:57:12.596 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2024-04-24T23:57:12.939 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-24T23:57:12.941 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-24T23:57:12.965 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../006-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-24T23:57:12.967 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-24T23:57:13.014 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-24T23:57:13.016 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-24T23:57:13.392 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-24T23:57:13.393 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-cmd2. 2024-04-24T23:57:13.419 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../007-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-24T23:57:13.420 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2024-04-24T23:57:13.450 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-24T23:57:13.494 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2024-04-24T23:57:13.803 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-24T23:57:13.830 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../008-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-24T23:57:13.870 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-24T23:57:13.913 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-24T23:57:13.939 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../003-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-24T23:57:14.022 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-24T23:57:14.223 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-24T23:57:14.249 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../009-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-24T23:57:14.289 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-24T23:57:14.359 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-24T23:57:14.385 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../004-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-24T23:57:14.425 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-24T23:57:14.617 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package librados2. 2024-04-24T23:57:14.644 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../010-librados2_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-24T23:57:14.684 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking librados2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:14.770 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-24T23:57:14.795 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../005-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-24T23:57:14.844 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-24T23:57:15.095 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package librbd1. 2024-04-24T23:57:15.122 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../011-librbd1_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-24T23:57:15.161 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking librbd1 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:15.373 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-24T23:57:15.399 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../006-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-24T23:57:15.456 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-24T23:57:15.597 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libcephfs2. 2024-04-24T23:57:15.624 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../012-libcephfs2_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-24T23:57:15.664 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libcephfs2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:15.843 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-24T23:57:15.869 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../007-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-24T23:57:15.909 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-24T23:57:15.949 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-rados. 2024-04-24T23:57:15.972 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../013-python3-rados_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-24T23:57:16.016 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-rados (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:16.254 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-24T23:57:16.280 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../008-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-24T23:57:16.319 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-24T23:57:16.320 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-24T23:57:16.345 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../014-python3-ceph-argparse_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-24T23:57:16.385 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-ceph-argparse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:16.579 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-cephfs. 2024-04-24T23:57:16.595 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../015-python3-cephfs_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-24T23:57:16.638 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-cephfs (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:16.690 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-24T23:57:16.716 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../009-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-24T23:57:16.756 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-24T23:57:16.924 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-24T23:57:16.950 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../016-python3-ceph-common_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-24T23:57:16.990 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-ceph-common (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:17.068 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package librados2. 2024-04-24T23:57:17.094 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../010-librados2_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-24T23:57:17.134 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking librados2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:17.327 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-prettytable. 2024-04-24T23:57:17.353 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../017-python3-prettytable_2.5.0-2_all.deb ... 2024-04-24T23:57:17.393 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-24T23:57:17.671 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-rbd. 2024-04-24T23:57:17.686 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../018-python3-rbd_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-24T23:57:17.729 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-rbd (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:17.779 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package librbd1. 2024-04-24T23:57:17.805 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../011-librbd1_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-24T23:57:17.854 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking librbd1 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:18.040 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-24T23:57:18.066 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../019-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-24T23:57:18.098 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-24T23:57:18.368 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-24T23:57:18.373 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libcephfs2. 2024-04-24T23:57:18.383 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../020-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-24T23:57:18.399 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../012-libcephfs2_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-24T23:57:18.426 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-24T23:57:18.448 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libcephfs2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:18.709 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-rados. 2024-04-24T23:57:18.724 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../013-python3-rados_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-24T23:57:18.728 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-24T23:57:18.755 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../021-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-24T23:57:18.767 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-rados (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:18.795 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-24T23:57:19.095 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-24T23:57:19.121 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../014-python3-ceph-argparse_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-24T23:57:19.131 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package lua5.1. 2024-04-24T23:57:19.158 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../022-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-24T23:57:19.161 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-ceph-argparse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:19.198 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-24T23:57:19.456 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-cephfs. 2024-04-24T23:57:19.468 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package lua-any. 2024-04-24T23:57:19.482 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../015-python3-cephfs_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-24T23:57:19.483 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../023-lua-any_27ubuntu1_all.deb ... 2024-04-24T23:57:19.526 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-24T23:57:19.530 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-cephfs (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:19.753 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package zip. 2024-04-24T23:57:19.769 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../024-zip_3.0-12build2_amd64.deb ... 2024-04-24T23:57:19.812 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking zip (3.0-12build2) ... 2024-04-24T23:57:19.950 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-24T23:57:19.976 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../016-python3-ceph-common_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-24T23:57:20.025 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-ceph-common (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:20.131 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package unzip. 2024-04-24T23:57:20.158 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../025-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-24T23:57:20.198 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-24T23:57:20.345 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-prettytable. 2024-04-24T23:57:20.371 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../017-python3-prettytable_2.5.0-2_all.deb ... 2024-04-24T23:57:20.411 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-24T23:57:20.602 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package luarocks. 2024-04-24T23:57:20.628 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../026-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-24T23:57:20.639 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-rbd. 2024-04-24T23:57:20.654 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../018-python3-rbd_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-24T23:57:20.668 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-24T23:57:20.688 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-rbd (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:21.017 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package librgw2. 2024-04-24T23:57:21.034 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-24T23:57:21.044 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../027-librgw2_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-24T23:57:21.062 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../019-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-24T23:57:21.075 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking librgw2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:21.100 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-24T23:57:21.386 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-24T23:57:21.401 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../020-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-24T23:57:21.444 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-24T23:57:21.586 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-rgw. 2024-04-24T23:57:21.601 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../028-python3-rgw_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-24T23:57:21.644 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-rgw (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:21.747 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-24T23:57:21.773 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../021-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-24T23:57:21.813 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-24T23:57:21.997 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-24T23:57:22.024 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../029-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-24T23:57:22.064 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-24T23:57:22.100 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package lua5.1. 2024-04-24T23:57:22.126 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../022-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-24T23:57:22.167 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-24T23:57:22.486 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package lua-any. 2024-04-24T23:57:22.512 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../023-lua-any_27ubuntu1_all.deb ... 2024-04-24T23:57:22.553 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-24T23:57:22.660 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libradosstriper1. 2024-04-24T23:57:22.687 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../030-libradosstriper1_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-24T23:57:22.733 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libradosstriper1 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:22.972 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package zip. 2024-04-24T23:57:22.998 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../024-zip_3.0-12build2_amd64.deb ... 2024-04-24T23:57:23.038 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking zip (3.0-12build2) ... 2024-04-24T23:57:23.045 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-common. 2024-04-24T23:57:23.074 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../031-ceph-common_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-24T23:57:23.111 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-common (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:23.334 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package unzip. 2024-04-24T23:57:23.360 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../025-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-24T23:57:23.392 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-24T23:57:23.771 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package luarocks. 2024-04-24T23:57:23.797 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../026-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-24T23:57:23.837 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-24T23:57:24.211 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package librgw2. 2024-04-24T23:57:24.238 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../027-librgw2_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-24T23:57:24.278 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking librgw2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:24.416 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-base. 2024-04-24T23:57:24.442 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../032-ceph-base_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-24T23:57:24.491 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-base (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:24.814 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-rgw. 2024-04-24T23:57:24.829 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../028-python3-rgw_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-24T23:57:24.872 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-rgw (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:25.095 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-24T23:57:25.122 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../033-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-24T23:57:25.162 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-24T23:57:25.175 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-24T23:57:25.190 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../029-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-24T23:57:25.233 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-24T23:57:25.464 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-cheroot. 2024-04-24T23:57:25.491 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../034-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-24T23:57:25.528 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libradosstriper1. 2024-04-24T23:57:25.531 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-24T23:57:25.554 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../030-libradosstriper1_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-24T23:57:25.594 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libradosstriper1 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:25.825 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-24T23:57:25.840 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../035-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-24T23:57:25.884 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-24T23:57:26.039 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph-common. 2024-04-24T23:57:26.065 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../031-ceph-common_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-24T23:57:26.105 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph-common (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:26.161 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-24T23:57:26.188 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../036-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-24T23:57:26.228 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-24T23:57:26.531 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-24T23:57:26.557 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../037-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-24T23:57:26.597 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-24T23:57:26.917 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-tempora. 2024-04-24T23:57:26.943 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../038-python3-tempora_4.1.2-1_all.deb ... 2024-04-24T23:57:26.983 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-24T23:57:27.286 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-portend. 2024-04-24T23:57:27.313 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../039-python3-portend_3.0.0-1_all.deb ... 2024-04-24T23:57:27.353 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph-base. 2024-04-24T23:57:27.353 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-24T23:57:27.379 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../032-ceph-base_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-24T23:57:27.421 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph-base (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:27.614 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-24T23:57:27.629 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../040-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-24T23:57:27.664 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-24T23:57:28.116 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-24T23:57:28.132 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-24T23:57:28.143 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../041-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-24T23:57:28.158 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../033-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-24T23:57:28.183 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-24T23:57:28.198 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-24T23:57:28.501 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-cheroot. 2024-04-24T23:57:28.512 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-natsort. 2024-04-24T23:57:28.527 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../034-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-24T23:57:28.538 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../042-python3-natsort_8.0.2-1_all.deb ... 2024-04-24T23:57:28.568 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-24T23:57:28.569 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-24T23:57:28.848 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-logutils. 2024-04-24T23:57:28.874 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../043-python3-logutils_0.3.3-8_all.deb ... 2024-04-24T23:57:28.912 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-24T23:57:28.914 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-24T23:57:28.939 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../035-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-24T23:57:28.979 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-24T23:57:29.248 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-24T23:57:29.275 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../036-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-24T23:57:29.300 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-24T23:57:29.306 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-24T23:57:29.327 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../044-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-24T23:57:29.367 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-24T23:57:29.595 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-24T23:57:29.621 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../037-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-24T23:57:29.661 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-24T23:57:29.670 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-mako. 2024-04-24T23:57:29.696 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../045-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-24T23:57:29.736 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-24T23:57:29.939 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-tempora. 2024-04-24T23:57:29.965 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../038-python3-tempora_4.1.2-1_all.deb ... 2024-04-24T23:57:29.997 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-24T23:57:30.064 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-24T23:57:30.091 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../046-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-24T23:57:30.131 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-24T23:57:30.300 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-portend. 2024-04-24T23:57:30.326 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../039-python3-portend_3.0.0-1_all.deb ... 2024-04-24T23:57:30.366 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-24T23:57:30.434 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-24T23:57:30.460 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../047-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-24T23:57:30.492 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-24T23:57:30.677 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-24T23:57:30.704 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../040-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-24T23:57:30.746 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-24T23:57:30.803 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-webob. 2024-04-24T23:57:30.829 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../048-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-24T23:57:30.869 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-24T23:57:31.005 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-24T23:57:31.020 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../041-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-24T23:57:31.055 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-24T23:57:31.214 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-waitress. 2024-04-24T23:57:31.240 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../049-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-24T23:57:31.284 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-24T23:57:31.400 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-natsort. 2024-04-24T23:57:31.426 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../042-python3-natsort_8.0.2-1_all.deb ... 2024-04-24T23:57:31.467 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-24T23:57:31.592 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-tempita. 2024-04-24T23:57:31.618 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../050-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-24T23:57:31.658 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-24T23:57:31.737 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-logutils. 2024-04-24T23:57:31.763 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../043-python3-logutils_0.3.3-8_all.deb ... 2024-04-24T23:57:31.795 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-24T23:57:31.944 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-paste. 2024-04-24T23:57:31.959 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../051-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-24T23:57:31.994 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-24T23:57:32.106 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-24T23:57:32.132 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../044-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-24T23:57:32.172 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-24T23:57:32.407 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-24T23:57:32.434 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../052-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-24T23:57:32.483 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-mako. 2024-04-24T23:57:32.510 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../045-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-24T23:57:32.550 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-24T23:57:32.789 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-24T23:57:32.853 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-24T23:57:32.880 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../046-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-24T23:57:32.920 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-24T23:57:33.198 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-24T23:57:33.224 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../047-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-24T23:57:33.256 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-24T23:57:33.367 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-24T23:57:33.382 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../053-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-24T23:57:33.417 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-24T23:57:33.534 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-webob. 2024-04-24T23:57:33.561 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../048-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-24T23:57:33.600 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-24T23:57:33.687 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-webtest. 2024-04-24T23:57:33.713 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../054-python3-webtest_2.0.35-1_all.deb ... 2024-04-24T23:57:33.753 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-24T23:57:33.962 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-waitress. 2024-04-24T23:57:33.989 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../049-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-24T23:57:34.031 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-24T23:57:34.064 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-pecan. 2024-04-24T23:57:34.091 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../055-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-24T23:57:34.131 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-24T23:57:34.511 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-certifi. 2024-04-24T23:57:34.537 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../056-python3-certifi_2020.6.20-1_all.deb ... 2024-04-24T23:57:34.577 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-24T23:57:34.892 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-idna. 2024-04-24T23:57:34.919 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../057-python3-idna_3.3-1_all.deb ... 2024-04-24T23:57:34.955 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-24T23:57:35.115 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-tempita. 2024-04-24T23:57:35.142 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../050-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-24T23:57:35.181 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-24T23:57:35.207 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-urllib3. 2024-04-24T23:57:35.234 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../058-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-24T23:57:35.265 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-24T23:57:35.459 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-paste. 2024-04-24T23:57:35.486 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../051-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-24T23:57:35.526 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-24T23:57:35.552 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-requests. 2024-04-24T23:57:35.579 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../059-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-24T23:57:35.618 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-24T23:57:35.871 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-24T23:57:35.897 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../052-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-24T23:57:35.930 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-24T23:57:35.937 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-24T23:57:35.956 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../060-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-24T23:57:35.996 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-24T23:57:36.198 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-24T23:57:36.224 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../053-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-24T23:57:36.240 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-24T23:57:36.256 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../061-ceph-mgr-modules-core_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-24T23:57:36.265 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-24T23:57:36.290 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:36.518 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-webtest. 2024-04-24T23:57:36.533 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../054-python3-webtest_2.0.35-1_all.deb ... 2024-04-24T23:57:36.568 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-24T23:57:36.662 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-24T23:57:36.689 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../062-libsqlite3-mod-ceph_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-24T23:57:36.728 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:36.871 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-pecan. 2024-04-24T23:57:36.897 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../055-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-24T23:57:36.937 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-24T23:57:37.023 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-24T23:57:37.049 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../063-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-24T23:57:37.089 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-24T23:57:37.300 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-certifi. 2024-04-24T23:57:37.327 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../056-python3-certifi_2020.6.20-1_all.deb ... 2024-04-24T23:57:37.366 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-24T23:57:37.367 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-openssl. 2024-04-24T23:57:37.393 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../064-python3-openssl_21.0.0-1_all.deb ... 2024-04-24T23:57:37.434 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-24T23:57:37.695 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-idna. 2024-04-24T23:57:37.721 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../057-python3-idna_3.3-1_all.deb ... 2024-04-24T23:57:37.745 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-mgr. 2024-04-24T23:57:37.761 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-24T23:57:37.772 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../065-ceph-mgr_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-24T23:57:37.811 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-mgr (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:38.081 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-urllib3. 2024-04-24T23:57:38.107 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../058-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-24T23:57:38.147 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-24T23:57:38.309 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-mon. 2024-04-24T23:57:38.336 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../066-ceph-mon_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-24T23:57:38.439 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-mon (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:38.625 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-requests. 2024-04-24T23:57:38.652 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../059-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-24T23:57:38.692 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-24T23:57:38.962 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-24T23:57:38.988 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../060-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-24T23:57:39.020 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-24T23:57:39.185 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-osd. 2024-04-24T23:57:39.212 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../067-ceph-osd_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-24T23:57:39.259 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-osd (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:39.331 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-24T23:57:39.347 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../061-ceph-mgr-modules-core_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-24T23:57:39.381 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:39.778 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-24T23:57:39.805 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../062-libsqlite3-mod-ceph_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-24T23:57:39.844 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:40.205 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-24T23:57:40.232 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../063-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-24T23:57:40.280 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-24T23:57:40.346 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph. 2024-04-24T23:57:40.373 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../068-ceph_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-24T23:57:40.412 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:40.649 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-openssl. 2024-04-24T23:57:40.676 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../064-python3-openssl_21.0.0-1_all.deb ... 2024-04-24T23:57:40.724 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-24T23:57:40.742 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-fuse. 2024-04-24T23:57:40.767 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../069-ceph-fuse_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-24T23:57:40.806 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-fuse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:41.069 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph-mgr. 2024-04-24T23:57:41.097 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../065-ceph-mgr_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-24T23:57:41.144 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph-mgr (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:41.168 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-mds. 2024-04-24T23:57:41.194 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../070-ceph-mds_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-24T23:57:41.234 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-mds (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:41.564 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph-mon. 2024-04-24T23:57:41.591 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../066-ceph-mon_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-24T23:57:41.595 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package cephadm. 2024-04-24T23:57:41.622 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../071-cephadm_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-24T23:57:41.653 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking cephadm (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:41.655 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph-mon (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:41.965 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-24T23:57:41.991 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../072-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-24T23:57:42.031 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-24T23:57:42.292 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph-osd. 2024-04-24T23:57:42.309 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-24T23:57:42.318 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../067-ceph-osd_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-24T23:57:42.335 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../073-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-24T23:57:42.358 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph-osd (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:42.376 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-24T23:57:43.416 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-babel. 2024-04-24T23:57:43.428 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph. 2024-04-24T23:57:43.443 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../074-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-24T23:57:43.444 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../068-ceph_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-24T23:57:43.482 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-24T23:57:43.487 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:43.773 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph-fuse. 2024-04-24T23:57:43.799 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../069-ceph-fuse_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-24T23:57:43.831 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph-fuse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:43.835 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-jinja2. 2024-04-24T23:57:43.862 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../075-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-24T23:57:43.901 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-24T23:57:44.192 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph-mds. 2024-04-24T23:57:44.222 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../070-ceph-mds_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-24T23:57:44.259 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph-mds (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:44.263 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-24T23:57:44.290 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../076-ceph-mgr-cephadm_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-24T23:57:44.329 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:44.654 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package cephadm. 2024-04-24T23:57:44.657 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-24T23:57:44.680 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../071-cephadm_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-24T23:57:44.684 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../077-python3-repoze.lru_0.7-2_all.deb ... 2024-04-24T23:57:44.720 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking cephadm (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:44.724 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-24T23:57:45.026 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-routes. 2024-04-24T23:57:45.041 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-24T23:57:45.053 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../078-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-24T23:57:45.066 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../072-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-24T23:57:45.093 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-24T23:57:45.106 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-24T23:57:45.430 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-24T23:57:45.456 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../079-ceph-mgr-dashboard_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-24T23:57:45.467 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-24T23:57:45.482 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../073-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-24T23:57:45.496 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:45.525 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-24T23:57:46.538 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-babel. 2024-04-24T23:57:46.554 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../074-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-24T23:57:46.588 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-24T23:57:46.668 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-24T23:57:46.694 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../080-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-24T23:57:46.725 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-24T23:57:46.866 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-jinja2. 2024-04-24T23:57:46.882 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../075-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-24T23:57:46.907 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-24T23:57:47.196 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-joblib. 2024-04-24T23:57:47.223 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../081-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-24T23:57:47.262 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-24T23:57:47.269 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-24T23:57:47.295 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../076-ceph-mgr-cephadm_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-24T23:57:47.335 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:47.599 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-24T23:57:47.626 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../082-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-24T23:57:47.681 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-24T23:57:47.707 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../077-python3-repoze.lru_0.7-2_all.deb ... 2024-04-24T23:57:47.747 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-24T23:57:47.750 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-24T23:57:48.020 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-sklearn. 2024-04-24T23:57:48.033 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-routes. 2024-04-24T23:57:48.047 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../083-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-24T23:57:48.059 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../078-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-24T23:57:48.095 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-24T23:57:48.099 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-24T23:57:48.436 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-24T23:57:48.452 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../079-ceph-mgr-dashboard_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-24T23:57:48.494 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:48.728 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-24T23:57:48.744 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../084-ceph-mgr-diskprediction-local_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-24T23:57:48.787 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:49.740 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-cachetools. 2024-04-24T23:57:49.767 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../085-python3-cachetools_5.0.0-1_all.deb ... 2024-04-24T23:57:49.806 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-24T23:57:49.941 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-24T23:57:49.957 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../080-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-24T23:57:49.991 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-24T23:57:50.109 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-24T23:57:50.137 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../086-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-24T23:57:50.176 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-24T23:57:50.488 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-24T23:57:50.503 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-joblib. 2024-04-24T23:57:50.515 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../087-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-24T23:57:50.530 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../081-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-24T23:57:50.546 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-24T23:57:50.569 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-24T23:57:50.815 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-rsa. 2024-04-24T23:57:50.843 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../088-python3-rsa_4.8-1_all.deb ... 2024-04-24T23:57:50.873 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-24T23:57:50.923 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-24T23:57:50.950 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../082-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-24T23:57:51.074 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-24T23:57:51.169 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-google-auth. 2024-04-24T23:57:51.196 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../089-python3-google-auth_1.5.1-3_all.deb ... 2024-04-24T23:57:51.235 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-24T23:57:51.386 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-sklearn. 2024-04-24T23:57:51.413 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../083-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-24T23:57:51.461 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-24T23:57:51.555 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-24T23:57:51.582 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../090-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-24T23:57:51.621 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-24T23:57:51.933 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-websocket. 2024-04-24T23:57:51.960 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../091-python3-websocket_1.2.3-1_all.deb ... 2024-04-24T23:57:51.999 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-24T23:57:52.145 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-24T23:57:52.172 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../084-ceph-mgr-diskprediction-local_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-24T23:57:52.203 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:52.311 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-24T23:57:52.338 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../092-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-24T23:57:52.377 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-24T23:57:53.016 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-24T23:57:53.031 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../093-ceph-mgr-k8sevents_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-24T23:57:53.074 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:53.173 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-cachetools. 2024-04-24T23:57:53.200 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../085-python3-cachetools_5.0.0-1_all.deb ... 2024-04-24T23:57:53.239 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-24T23:57:53.402 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-24T23:57:53.429 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../094-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-24T23:57:53.468 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-24T23:57:53.717 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-24T23:57:53.738 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-24T23:57:53.744 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../086-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-24T23:57:53.765 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../095-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-24T23:57:53.783 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-24T23:57:53.796 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-24T23:57:54.128 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-24T23:57:54.155 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../087-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-24T23:57:54.195 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-24T23:57:54.315 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package jq. 2024-04-24T23:57:54.342 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../096-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-24T23:57:54.381 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-24T23:57:54.506 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-rsa. 2024-04-24T23:57:54.533 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../088-python3-rsa_4.8-1_all.deb ... 2024-04-24T23:57:54.573 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-24T23:57:54.634 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package socat. 2024-04-24T23:57:54.650 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../097-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-24T23:57:54.692 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-24T23:57:54.835 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-google-auth. 2024-04-24T23:57:54.862 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../089-python3-google-auth_1.5.1-3_all.deb ... 2024-04-24T23:57:54.901 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-24T23:57:54.987 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package xmlstarlet. 2024-04-24T23:57:55.015 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../098-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-24T23:57:55.054 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-24T23:57:55.246 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-24T23:57:55.274 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../090-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-24T23:57:55.313 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-24T23:57:55.477 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-test. 2024-04-24T23:57:55.502 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../099-ceph-test_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-24T23:57:55.541 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-test (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:55.624 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-websocket. 2024-04-24T23:57:55.651 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../091-python3-websocket_1.2.3-1_all.deb ... 2024-04-24T23:57:55.690 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-24T23:57:56.010 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-24T23:57:56.030 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../092-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-24T23:57:56.068 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-24T23:57:56.774 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-24T23:57:56.790 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../093-ceph-mgr-k8sevents_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-24T23:57:56.824 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:57.110 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-24T23:57:57.137 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../094-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-24T23:57:57.176 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-24T23:57:57.521 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-24T23:57:57.549 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../095-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-24T23:57:57.588 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-24T23:57:57.837 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ceph-volume. 2024-04-24T23:57:57.848 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package jq. 2024-04-24T23:57:57.864 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../100-ceph-volume_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-24T23:57:57.864 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../096-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-24T23:57:57.903 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ceph-volume (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:57.906 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-24T23:57:58.134 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package socat. 2024-04-24T23:57:58.150 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../097-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-24T23:57:58.184 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-24T23:57:58.274 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package cephfs-mirror. 2024-04-24T23:57:58.301 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../101-cephfs-mirror_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-24T23:57:58.340 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking cephfs-mirror (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:58.462 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package xmlstarlet. 2024-04-24T23:57:58.478 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../098-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-24T23:57:58.520 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-24T23:57:58.651 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-colorama. 2024-04-24T23:57:58.679 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../102-python3-colorama_0.4.4-1_all.deb ... 2024-04-24T23:57:58.718 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-colorama (0.4.4-1) ... 2024-04-24T23:57:58.875 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph-test. 2024-04-24T23:57:58.891 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../099-ceph-test_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-24T23:57:58.924 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph-test (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:58.970 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package cephfs-shell. 2024-04-24T23:57:58.986 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../103-cephfs-shell_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-24T23:57:59.020 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking cephfs-shell (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:59.273 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package cephfs-top. 2024-04-24T23:57:59.300 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../104-cephfs-top_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-24T23:57:59.339 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking cephfs-top (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:59.608 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-24T23:57:59.624 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../105-libcephfs-dev_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-24T23:57:59.666 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libcephfs-dev (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:57:59.986 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-24T23:58:00.013 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../106-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-24T23:58:00.052 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-24T23:58:00.373 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-24T23:58:00.401 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../107-lua-sec_1.0.2-1_amd64.deb ... 2024-04-24T23:58:00.440 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-24T23:58:00.727 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package nvme-cli. 2024-04-24T23:58:00.754 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../108-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-24T23:58:00.793 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-24T23:58:01.199 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package pkg-config. 2024-04-24T23:58:01.226 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../109-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-24T23:58:01.265 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-24T23:58:01.279 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ceph-volume. 2024-04-24T23:58:01.307 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../100-ceph-volume_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-24T23:58:01.345 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ceph-volume (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:01.526 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-24T23:58:01.550 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../110-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-24T23:58:01.592 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-24T23:58:01.724 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package cephfs-mirror. 2024-04-24T23:58:01.752 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../101-cephfs-mirror_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-24T23:58:01.791 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking cephfs-mirror (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:01.979 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-24T23:58:02.006 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../111-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-24T23:58:02.045 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-24T23:58:02.077 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-colorama. 2024-04-24T23:58:02.104 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../102-python3-colorama_0.4.4-1_all.deb ... 2024-04-24T23:58:02.135 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-colorama (0.4.4-1) ... 2024-04-24T23:58:02.346 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package cephfs-shell. 2024-04-24T23:58:02.348 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-pastescript. 2024-04-24T23:58:02.362 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../103-cephfs-shell_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-24T23:58:02.375 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../112-python3-pastescript_2.0.2-4_all.deb ... 2024-04-24T23:58:02.404 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking cephfs-shell (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:02.414 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-24T23:58:02.690 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package cephfs-top. 2024-04-24T23:58:02.718 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../104-cephfs-top_19.0.0-2445-gd2c45d97-1jammy_all.deb ... 2024-04-24T23:58:02.734 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-pluggy. 2024-04-24T23:58:02.757 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking cephfs-top (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:02.762 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../113-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-24T23:58:02.800 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-24T23:58:02.992 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-24T23:58:03.020 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../105-libcephfs-dev_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-24T23:58:03.059 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libcephfs-dev (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:03.103 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-psutil. 2024-04-24T23:58:03.131 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../114-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-24T23:58:03.170 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-24T23:58:03.387 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-24T23:58:03.415 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../106-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-24T23:58:03.453 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-24T23:58:03.506 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-py. 2024-04-24T23:58:03.534 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../115-python3-py_1.10.0-1_all.deb ... 2024-04-24T23:58:03.598 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-24T23:58:03.774 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-24T23:58:03.802 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../107-lua-sec_1.0.2-1_amd64.deb ... 2024-04-24T23:58:03.832 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-24T23:58:03.926 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-pygments. 2024-04-24T23:58:03.953 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../116-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-24T23:58:03.992 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-24T23:58:04.211 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package nvme-cli. 2024-04-24T23:58:04.239 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../108-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-24T23:58:04.278 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-24T23:58:04.388 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-24T23:58:04.416 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../117-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-24T23:58:04.454 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-24T23:58:04.699 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package pkg-config. 2024-04-24T23:58:04.727 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../109-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-24T23:58:04.765 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-toml. 2024-04-24T23:58:04.766 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-24T23:58:04.793 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../118-python3-toml_0.10.2-1_all.deb ... 2024-04-24T23:58:04.832 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-24T23:58:05.019 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-24T23:58:05.034 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../110-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-24T23:58:05.077 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-24T23:58:05.135 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-pytest. 2024-04-24T23:58:05.162 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../119-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-24T23:58:05.201 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-24T23:58:05.489 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-24T23:58:05.516 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../111-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-24T23:58:05.546 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package python3-simplejson. 2024-04-24T23:58:05.555 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-24T23:58:05.573 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../120-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-24T23:58:05.629 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-24T23:58:05.849 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-pastescript. 2024-04-24T23:58:05.877 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../112-python3-pastescript_2.0.2-4_all.deb ... 2024-04-24T23:58:05.898 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-24T23:58:05.907 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-24T23:58:05.914 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../121-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-24T23:58:05.956 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-24T23:58:06.169 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-pluggy. 2024-04-24T23:58:06.184 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../113-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-24T23:58:06.219 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-24T23:58:06.505 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-psutil. 2024-04-24T23:58:06.533 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../114-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-24T23:58:06.571 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-24T23:58:06.702 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package radosgw. 2024-04-24T23:58:06.730 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../122-radosgw_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-24T23:58:06.768 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking radosgw (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:06.900 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-py. 2024-04-24T23:58:06.927 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../115-python3-py_1.10.0-1_all.deb ... 2024-04-24T23:58:06.966 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-24T23:58:07.269 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-pygments. 2024-04-24T23:58:07.297 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../116-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-24T23:58:07.335 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-24T23:58:07.563 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package rbd-fuse. 2024-04-24T23:58:07.579 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../123-rbd-fuse_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-24T23:58:07.613 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking rbd-fuse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:07.815 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-24T23:58:07.843 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../117-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-24T23:58:07.881 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-24T23:58:07.926 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-24T23:58:08.167 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-toml. 2024-04-24T23:58:08.176 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-24T23:58:08.195 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../118-python3-toml_0.10.2-1_all.deb ... 2024-04-24T23:58:08.225 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-24T23:58:08.302 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-24T23:58:08.470 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-pytest. 2024-04-24T23:58:08.498 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../119-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-24T23:58:08.536 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-24T23:58:08.567 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-24T23:58:08.892 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package python3-simplejson. 2024-04-24T23:58:08.910 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-24T23:58:08.919 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../120-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-24T23:58:08.958 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-24T23:58:09.228 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-24T23:58:09.245 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../121-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-24T23:58:09.294 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-24T23:58:09.351 INFO:teuthology.orchestra.run.smithi174.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-24T23:58:09.420 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up cephadm (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:10.031 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package radosgw. 2024-04-24T23:58:10.060 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../122-radosgw_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-24T23:58:10.097 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking radosgw (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:10.253 INFO:teuthology.orchestra.run.smithi174.stdout:Adding system user cephadm....done 2024-04-24T23:58:10.376 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-24T23:58:10.644 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-24T23:58:10.893 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package rbd-fuse. 2024-04-24T23:58:10.895 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-24T23:58:10.920 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../123-rbd-fuse_19.0.0-2445-gd2c45d97-1jammy_amd64.deb ... 2024-04-24T23:58:10.959 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking rbd-fuse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:11.021 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-24T23:58:11.247 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-24T23:58:11.272 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-24T23:58:11.523 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-24T23:58:11.650 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-24T23:58:11.706 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-24T23:58:11.848 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-24T23:58:11.976 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-24T23:58:12.113 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-24T23:58:12.336 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-24T23:58:12.449 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-24T23:58:12.588 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-colorama (0.4.4-1) ... 2024-04-24T23:58:12.830 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-24T23:58:12.864 INFO:teuthology.orchestra.run.smithi116.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-24T23:58:12.933 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up cephadm (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:13.090 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-24T23:58:13.333 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-24T23:58:13.382 INFO:teuthology.orchestra.run.smithi116.stdout:Adding system user cephadm....done 2024-04-24T23:58:13.523 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-24T23:58:13.576 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-24T23:58:13.799 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-24T23:58:13.835 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-24T23:58:14.050 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-24T23:58:14.094 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-ceph-argparse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:14.176 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-24T23:58:14.354 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-24T23:58:14.427 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-24T23:58:14.488 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-24T23:58:14.614 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2024-04-24T23:58:14.678 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-24T23:58:14.804 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-24T23:58:14.874 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-24T23:58:14.983 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-24T23:58:15.148 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-24T23:58:15.242 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-24T23:58:15.525 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-24T23:58:15.630 INFO:teuthology.orchestra.run.smithi174.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-24T23:58:15.665 INFO:teuthology.orchestra.run.smithi174.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-24T23:58:15.720 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-24T23:58:15.784 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-colorama (0.4.4-1) ... 2024-04-24T23:58:15.855 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-24T23:58:16.044 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-24T23:58:16.214 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-24T23:58:16.303 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-24T23:58:16.340 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-24T23:58:16.546 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-24T23:58:16.599 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-24T23:58:16.814 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-24T23:58:16.859 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-24T23:58:17.074 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-24T23:58:17.151 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up zip (3.0-12build2) ... 2024-04-24T23:58:17.277 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-24T23:58:17.333 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-ceph-argparse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:17.593 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-24T23:58:17.702 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-24T23:58:17.811 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2024-04-24T23:58:17.930 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-24T23:58:18.079 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-24T23:58:18.189 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-24T23:58:18.205 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-24T23:58:18.298 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-24T23:58:18.424 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-24T23:58:18.456 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-24T23:58:18.717 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-24T23:58:18.813 INFO:teuthology.orchestra.run.smithi116.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-24T23:58:18.845 INFO:teuthology.orchestra.run.smithi116.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-24T23:58:18.918 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-24T23:58:19.069 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-24T23:58:19.101 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-24T23:58:19.444 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-24T23:58:19.453 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-24T23:58:19.563 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-24T23:58:19.814 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-24T23:58:19.829 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-24T23:58:20.074 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-24T23:58:20.113 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2024-04-24T23:58:20.374 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up zip (3.0-12build2) ... 2024-04-24T23:58:20.398 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-24T23:58:20.500 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-24T23:58:20.732 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-24T23:58:20.983 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-24T23:58:21.093 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-ceph-common (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:21.144 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-24T23:58:21.386 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-idna (3.3-1) ... 2024-04-24T23:58:21.386 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-24T23:58:21.513 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-24T23:58:21.639 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-24T23:58:21.695 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-24T23:58:22.065 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-24T23:58:22.450 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-24T23:58:22.701 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-24T23:58:22.792 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-24T23:58:23.060 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-24T23:58:23.069 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-24T23:58:23.195 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-24T23:58:23.370 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2024-04-24T23:58:23.463 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-24T23:58:23.589 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-24T23:58:23.671 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-24T23:58:23.715 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-24T23:58:23.999 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-24T23:58:24.006 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-24T23:58:24.257 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-24T23:58:24.275 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-24T23:58:24.383 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-ceph-common (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:24.526 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-24T23:58:24.652 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-24T23:58:24.676 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-idna (3.3-1) ... 2024-04-24T23:58:24.970 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-24T23:58:25.002 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-24T23:58:25.230 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-24T23:58:25.480 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-24T23:58:25.732 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-24T23:58:25.991 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-24T23:58:26.158 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-24T23:58:26.250 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-24T23:58:26.376 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-24T23:58:26.418 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-24T23:58:26.536 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-24T23:58:26.660 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-24T23:58:26.778 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-24T23:58:26.786 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-24T23:58:26.896 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-24T23:58:27.014 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-24T23:58:27.081 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-24T23:58:27.307 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-24T23:58:27.374 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-24T23:58:27.558 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-24T23:58:27.625 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-24T23:58:27.743 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-24T23:58:27.809 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-24T23:58:27.869 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-24T23:58:27.936 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-24T23:58:27.957 INFO:teuthology.orchestra.run.smithi174.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-24T23:58:28.187 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-24T23:58:28.237 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-24T23:58:28.313 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-24T23:58:28.480 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-24T23:58:28.698 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-24T23:58:28.731 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-24T23:58:28.957 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-24T23:58:28.990 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-24T23:58:29.083 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-24T23:58:29.241 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-24T23:58:29.342 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-24T23:58:29.468 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-24T23:58:29.509 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-24T23:58:29.618 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-24T23:58:29.794 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-24T23:58:29.894 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-24T23:58:30.012 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-24T23:58:30.087 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-24T23:58:30.297 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-24T23:58:30.388 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-24T23:58:30.599 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-24T23:58:30.606 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-24T23:58:30.857 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-24T23:58:30.859 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-24T23:58:30.983 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-24T23:58:31.109 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-24T23:58:31.118 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-24T23:58:31.189 INFO:teuthology.orchestra.run.smithi116.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-24T23:58:31.386 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-24T23:58:31.436 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-24T23:58:31.545 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-24T23:58:31.763 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-24T23:58:31.938 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-24T23:58:32.056 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-24T23:58:32.231 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-24T23:58:32.357 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-24T23:58:32.392 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-24T23:58:32.518 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up librados2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:32.617 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-24T23:58:32.644 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:32.743 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-24T23:58:32.770 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-24T23:58:33.069 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-24T23:58:33.122 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libcephfs2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:33.240 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libradosstriper1 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:33.349 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-24T23:58:33.361 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-24T23:58:33.609 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up librbd1 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:33.629 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-24T23:58:33.735 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-mgr-modules-core (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:33.841 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-24T23:58:33.861 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-fuse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:34.074 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-24T23:58:34.074 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-24T23:58:34.117 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-24T23:58:34.394 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-24T23:58:34.554 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libcephfs-dev (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:34.661 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-24T23:58:34.680 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-rados (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:34.806 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up librgw2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:34.924 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up cephfs-top (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:35.024 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-24T23:58:35.080 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-24T23:58:35.431 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-24T23:58:35.784 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-24T23:58:35.910 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up librados2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:36.036 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:36.153 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-24T23:58:36.236 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-rbd (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:36.379 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up rbd-fuse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:36.489 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libcephfs2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:36.505 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-rgw (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:36.605 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up python3-cephfs (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:36.615 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libradosstriper1 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:36.732 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up cephfs-shell (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:36.741 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-24T23:58:36.858 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-common (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:37.035 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up librbd1 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:37.160 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph-mgr-modules-core (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:37.287 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph-fuse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:37.375 INFO:teuthology.orchestra.run.smithi174.stdout:Adding group ceph....done 2024-04-24T23:58:37.508 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-24T23:58:37.508 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-24T23:58:37.751 INFO:teuthology.orchestra.run.smithi174.stdout:Adding system user ceph....done 2024-04-24T23:58:37.919 INFO:teuthology.orchestra.run.smithi174.stdout:Setting system user ceph properties....done 2024-04-24T23:58:37.932 INFO:teuthology.orchestra.run.smithi174.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-24T23:58:37.996 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libcephfs-dev (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:38.075 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-24T23:58:38.131 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-rados (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:38.265 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up librgw2 (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:38.416 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up cephfs-top (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:38.416 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-24T23:58:38.567 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-24T23:58:38.933 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up cephfs-mirror (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:39.126 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-04-24T23:58:39.126 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-04-24T23:58:39.609 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-test (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:39.744 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up radosgw (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:39.762 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-rbd (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:39.913 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up rbd-fuse (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:40.065 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-rgw (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:40.200 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-24T23:58:40.200 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-24T23:58:40.216 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up python3-cephfs (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:40.342 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up cephfs-shell (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:40.468 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph-common (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:40.696 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-base (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:41.041 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-24T23:58:41.078 INFO:teuthology.orchestra.run.smithi116.stdout:Adding group ceph....done 2024-04-24T23:58:41.462 INFO:teuthology.orchestra.run.smithi116.stdout:Adding system user ceph....done 2024-04-24T23:58:41.497 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-mds (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:41.631 INFO:teuthology.orchestra.run.smithi116.stdout:Setting system user ceph properties....done 2024-04-24T23:58:41.646 INFO:teuthology.orchestra.run.smithi116.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-24T23:58:41.728 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-24T23:58:41.728 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-24T23:58:41.784 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-24T23:58:42.121 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-24T23:58:42.182 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-mgr (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:42.400 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-24T23:58:42.400 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-24T23:58:42.850 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-osd (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:42.873 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up cephfs-mirror (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:43.096 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-04-24T23:58:43.097 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-04-24T23:58:43.124 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-24T23:58:43.124 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-24T23:58:43.558 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph-test (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:43.593 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:43.684 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up radosgw (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:43.719 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:43.870 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-mon (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:44.082 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-24T23:58:44.083 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-24T23:58:44.145 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-24T23:58:44.145 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-24T23:58:44.554 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-mgr-cephadm (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:44.610 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph-base (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:44.689 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:44.798 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-mgr-dashboard (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:44.932 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ceph-volume (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:44.959 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-24T23:58:45.277 INFO:teuthology.orchestra.run.smithi174.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-24T23:58:45.386 INFO:teuthology.orchestra.run.smithi174.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-24T23:58:45.462 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph-mds (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:45.564 INFO:teuthology.orchestra.run.smithi174.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-24T23:58:45.676 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-24T23:58:45.676 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-24T23:58:46.148 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph-mgr (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:46.359 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-24T23:58:46.359 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-24T23:58:46.682 INFO:teuthology.orchestra.run.smithi174.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-24T23:58:46.799 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph-osd (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:47.043 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-24T23:58:47.043 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-24T23:58:47.364 INFO:teuthology.orchestra.run.smithi174.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-24T23:58:47.367 DEBUG:teuthology.orchestra.run.smithi174:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2024-04-24T23:58:47.417 INFO:teuthology.orchestra.run.smithi174.stdout:Reading package lists... 2024-04-24T23:58:47.500 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:47.597 INFO:teuthology.orchestra.run.smithi174.stdout:Building dependency tree... 2024-04-24T23:58:47.598 INFO:teuthology.orchestra.run.smithi174.stdout:Reading state information... 2024-04-24T23:58:47.626 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:47.772 INFO:teuthology.orchestra.run.smithi174.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2024-04-24T23:58:47.772 INFO:teuthology.orchestra.run.smithi174.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2024-04-24T23:58:47.772 INFO:teuthology.orchestra.run.smithi174.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2024-04-24T23:58:47.773 INFO:teuthology.orchestra.run.smithi174.stdout:The following packages were automatically installed and are no longer required: 2024-04-24T23:58:47.773 INFO:teuthology.orchestra.run.smithi174.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-24T23:58:47.773 INFO:teuthology.orchestra.run.smithi174.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-24T23:58:47.773 INFO:teuthology.orchestra.run.smithi174.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-24T23:58:47.773 INFO:teuthology.orchestra.run.smithi174.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-24T23:58:47.773 INFO:teuthology.orchestra.run.smithi174.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-24T23:58:47.773 INFO:teuthology.orchestra.run.smithi174.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-24T23:58:47.774 INFO:teuthology.orchestra.run.smithi174.stdout:The following additional packages will be installed: 2024-04-24T23:58:47.774 INFO:teuthology.orchestra.run.smithi174.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2024-04-24T23:58:47.774 INFO:teuthology.orchestra.run.smithi174.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2024-04-24T23:58:47.774 INFO:teuthology.orchestra.run.smithi174.stdout: libqmi-proxy libssl3 libteamdctl0 libuuid1 modemmanager network-manager-pptp 2024-04-24T23:58:47.774 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2024-04-24T23:58:47.774 INFO:teuthology.orchestra.run.smithi174.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2024-04-24T23:58:47.775 INFO:teuthology.orchestra.run.smithi174.stdout:Suggested packages: 2024-04-24T23:58:47.776 INFO:teuthology.orchestra.run.smithi174.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2024-04-24T23:58:47.776 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial 2024-04-24T23:58:47.776 INFO:teuthology.orchestra.run.smithi174.stdout: util-linux-locales wpagui libengine-pkcs11-openssl 2024-04-24T23:58:47.816 INFO:teuthology.orchestra.run.smithi174.stdout:The following NEW packages will be installed: 2024-04-24T23:58:47.816 INFO:teuthology.orchestra.run.smithi174.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2024-04-24T23:58:47.817 INFO:teuthology.orchestra.run.smithi174.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2024-04-24T23:58:47.817 INFO:teuthology.orchestra.run.smithi174.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2024-04-24T23:58:47.817 INFO:teuthology.orchestra.run.smithi174.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2024-04-24T23:58:47.817 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2024-04-24T23:58:47.817 INFO:teuthology.orchestra.run.smithi174.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2024-04-24T23:58:47.818 INFO:teuthology.orchestra.run.smithi174.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2024-04-24T23:58:47.819 INFO:teuthology.orchestra.run.smithi174.stdout:The following packages will be upgraded: 2024-04-24T23:58:47.819 INFO:teuthology.orchestra.run.smithi174.stdout: libssl3 libuuid1 util-linux uuid-dev 2024-04-24T23:58:47.986 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph-mon (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:48.009 INFO:teuthology.orchestra.run.smithi174.stdout:4 upgraded, 39 newly installed, 0 to remove and 298 not upgraded. 2024-04-24T23:58:48.009 INFO:teuthology.orchestra.run.smithi174.stdout:Need to get 55.7 MB of archives. 2024-04-24T23:58:48.009 INFO:teuthology.orchestra.run.smithi174.stdout:After this operation, 202 MB of additional disk space will be used. 2024-04-24T23:58:48.010 INFO:teuthology.orchestra.run.smithi174.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 util-linux amd64 2.37.2-4ubuntu3.4 [1,063 kB] 2024-04-24T23:58:48.203 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-24T23:58:48.203 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-24T23:58:48.642 INFO:teuthology.orchestra.run.smithi174.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2024-04-24T23:58:48.659 INFO:teuthology.orchestra.run.smithi174.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.15 [1,905 kB] 2024-04-24T23:58:48.704 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph-mgr-cephadm (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:48.784 INFO:teuthology.orchestra.run.smithi174.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 uuid-dev amd64 2.37.2-4ubuntu3.4 [33.1 kB] 2024-04-24T23:58:48.785 INFO:teuthology.orchestra.run.smithi174.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libuuid1 amd64 2.37.2-4ubuntu3.4 [23.8 kB] 2024-04-24T23:58:48.786 INFO:teuthology.orchestra.run.smithi174.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2024-04-24T23:58:48.815 INFO:teuthology.orchestra.run.smithi174.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2024-04-24T23:58:48.816 INFO:teuthology.orchestra.run.smithi174.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2024-04-24T23:58:48.820 INFO:teuthology.orchestra.run.smithi174.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2024-04-24T23:58:48.823 INFO:teuthology.orchestra.run.smithi174.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2024-04-24T23:58:48.824 INFO:teuthology.orchestra.run.smithi174.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libbluetooth3 amd64 5.64-0ubuntu1.1 [87.0 kB] 2024-04-24T23:58:48.855 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:48.981 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph-mgr-dashboard (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:48.990 INFO:teuthology.orchestra.run.smithi174.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2024-04-24T23:58:49.107 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ceph-volume (19.0.0-2445-gd2c45d97-1jammy) ... 2024-04-24T23:58:49.284 INFO:teuthology.orchestra.run.smithi174.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2024-04-24T23:58:49.293 INFO:teuthology.orchestra.run.smithi174.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2024-04-24T23:58:49.299 INFO:teuthology.orchestra.run.smithi174.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2024-04-24T23:58:49.384 INFO:teuthology.orchestra.run.smithi174.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2024-04-24T23:58:49.451 INFO:teuthology.orchestra.run.smithi116.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-24T23:58:49.569 INFO:teuthology.orchestra.run.smithi116.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-24T23:58:49.722 INFO:teuthology.orchestra.run.smithi116.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-24T23:58:50.473 INFO:teuthology.orchestra.run.smithi174.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2024-04-24T23:58:50.478 INFO:teuthology.orchestra.run.smithi174.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2024-04-24T23:58:50.479 INFO:teuthology.orchestra.run.smithi174.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.3 [263 kB] 2024-04-24T23:58:50.485 INFO:teuthology.orchestra.run.smithi174.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2024-04-24T23:58:50.486 INFO:teuthology.orchestra.run.smithi174.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2024-04-24T23:58:50.497 INFO:teuthology.orchestra.run.smithi174.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.11-0ubuntu0.22.04.1 [144 kB] 2024-04-24T23:58:50.665 INFO:teuthology.orchestra.run.smithi174.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2024-04-24T23:58:50.872 INFO:teuthology.orchestra.run.smithi116.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-24T23:58:51.257 INFO:teuthology.orchestra.run.smithi174.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2024-04-24T23:58:51.257 INFO:teuthology.orchestra.run.smithi174.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.15 [2,376 kB] 2024-04-24T23:58:51.440 INFO:teuthology.orchestra.run.smithi174.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2024-04-24T23:58:51.440 INFO:teuthology.orchestra.run.smithi174.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.3 [1,094 kB] 2024-04-24T23:58:51.479 INFO:teuthology.orchestra.run.smithi174.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2024-04-24T23:58:51.538 INFO:teuthology.orchestra.run.smithi174.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2024-04-24T23:58:51.540 INFO:teuthology.orchestra.run.smithi116.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-24T23:58:51.544 DEBUG:teuthology.orchestra.run.smithi116:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2024-04-24T23:58:51.545 INFO:teuthology.orchestra.run.smithi174.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2024-04-24T23:58:51.547 INFO:teuthology.orchestra.run.smithi174.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2024-04-24T23:58:51.547 INFO:teuthology.orchestra.run.smithi174.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2024-04-24T23:58:51.548 INFO:teuthology.orchestra.run.smithi174.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.11-0ubuntu0.22.04.1 [1,222 kB] 2024-04-24T23:58:51.603 INFO:teuthology.orchestra.run.smithi116.stdout:Reading package lists... 2024-04-24T23:58:51.755 INFO:teuthology.orchestra.run.smithi174.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2024-04-24T23:58:51.805 INFO:teuthology.orchestra.run.smithi116.stdout:Building dependency tree... 2024-04-24T23:58:51.806 INFO:teuthology.orchestra.run.smithi116.stdout:Reading state information... 2024-04-24T23:58:51.987 INFO:teuthology.orchestra.run.smithi116.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2024-04-24T23:58:51.987 INFO:teuthology.orchestra.run.smithi116.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2024-04-24T23:58:51.987 INFO:teuthology.orchestra.run.smithi116.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2024-04-24T23:58:51.987 INFO:teuthology.orchestra.run.smithi116.stdout:The following packages were automatically installed and are no longer required: 2024-04-24T23:58:51.987 INFO:teuthology.orchestra.run.smithi116.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-24T23:58:51.987 INFO:teuthology.orchestra.run.smithi116.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-24T23:58:51.987 INFO:teuthology.orchestra.run.smithi116.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-24T23:58:51.988 INFO:teuthology.orchestra.run.smithi116.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-24T23:58:51.988 INFO:teuthology.orchestra.run.smithi116.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-24T23:58:51.988 INFO:teuthology.orchestra.run.smithi116.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-24T23:58:51.988 INFO:teuthology.orchestra.run.smithi116.stdout:The following additional packages will be installed: 2024-04-24T23:58:51.989 INFO:teuthology.orchestra.run.smithi116.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2024-04-24T23:58:51.989 INFO:teuthology.orchestra.run.smithi116.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2024-04-24T23:58:51.989 INFO:teuthology.orchestra.run.smithi116.stdout: libqmi-proxy libssl3 libteamdctl0 libuuid1 modemmanager network-manager-pptp 2024-04-24T23:58:51.989 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2024-04-24T23:58:51.989 INFO:teuthology.orchestra.run.smithi116.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2024-04-24T23:58:51.990 INFO:teuthology.orchestra.run.smithi116.stdout:Suggested packages: 2024-04-24T23:58:51.990 INFO:teuthology.orchestra.run.smithi116.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2024-04-24T23:58:51.990 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial 2024-04-24T23:58:51.990 INFO:teuthology.orchestra.run.smithi116.stdout: util-linux-locales wpagui libengine-pkcs11-openssl 2024-04-24T23:58:52.029 INFO:teuthology.orchestra.run.smithi116.stdout:The following NEW packages will be installed: 2024-04-24T23:58:52.030 INFO:teuthology.orchestra.run.smithi116.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2024-04-24T23:58:52.030 INFO:teuthology.orchestra.run.smithi116.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2024-04-24T23:58:52.030 INFO:teuthology.orchestra.run.smithi116.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2024-04-24T23:58:52.030 INFO:teuthology.orchestra.run.smithi116.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2024-04-24T23:58:52.030 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2024-04-24T23:58:52.031 INFO:teuthology.orchestra.run.smithi116.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2024-04-24T23:58:52.031 INFO:teuthology.orchestra.run.smithi116.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2024-04-24T23:58:52.031 INFO:teuthology.orchestra.run.smithi116.stdout:The following packages will be upgraded: 2024-04-24T23:58:52.032 INFO:teuthology.orchestra.run.smithi116.stdout: libssl3 libuuid1 util-linux uuid-dev 2024-04-24T23:58:52.224 INFO:teuthology.orchestra.run.smithi174.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.11-0ubuntu0.22.04.1 [16.2 MB] 2024-04-24T23:58:52.235 INFO:teuthology.orchestra.run.smithi116.stdout:4 upgraded, 39 newly installed, 0 to remove and 298 not upgraded. 2024-04-24T23:58:52.235 INFO:teuthology.orchestra.run.smithi116.stdout:Need to get 55.7 MB of archives. 2024-04-24T23:58:52.236 INFO:teuthology.orchestra.run.smithi116.stdout:After this operation, 202 MB of additional disk space will be used. 2024-04-24T23:58:52.236 INFO:teuthology.orchestra.run.smithi116.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 util-linux amd64 2.37.2-4ubuntu3.4 [1,063 kB] 2024-04-24T23:58:52.861 INFO:teuthology.orchestra.run.smithi116.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2024-04-24T23:58:52.878 INFO:teuthology.orchestra.run.smithi116.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.15 [1,905 kB] 2024-04-24T23:58:53.002 INFO:teuthology.orchestra.run.smithi116.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 uuid-dev amd64 2.37.2-4ubuntu3.4 [33.1 kB] 2024-04-24T23:58:53.003 INFO:teuthology.orchestra.run.smithi116.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libuuid1 amd64 2.37.2-4ubuntu3.4 [23.8 kB] 2024-04-24T23:58:53.004 INFO:teuthology.orchestra.run.smithi116.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2024-04-24T23:58:53.032 INFO:teuthology.orchestra.run.smithi116.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2024-04-24T23:58:53.034 INFO:teuthology.orchestra.run.smithi116.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2024-04-24T23:58:53.038 INFO:teuthology.orchestra.run.smithi116.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2024-04-24T23:58:53.041 INFO:teuthology.orchestra.run.smithi116.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2024-04-24T23:58:53.042 INFO:teuthology.orchestra.run.smithi116.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libbluetooth3 amd64 5.64-0ubuntu1.1 [87.0 kB] 2024-04-24T23:58:53.207 INFO:teuthology.orchestra.run.smithi116.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2024-04-24T23:58:53.500 INFO:teuthology.orchestra.run.smithi116.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2024-04-24T23:58:53.510 INFO:teuthology.orchestra.run.smithi116.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2024-04-24T23:58:53.515 INFO:teuthology.orchestra.run.smithi116.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2024-04-24T23:58:53.600 INFO:teuthology.orchestra.run.smithi116.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2024-04-24T23:58:54.726 INFO:teuthology.orchestra.run.smithi116.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2024-04-24T23:58:54.731 INFO:teuthology.orchestra.run.smithi116.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2024-04-24T23:58:54.732 INFO:teuthology.orchestra.run.smithi116.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.3 [263 kB] 2024-04-24T23:58:54.738 INFO:teuthology.orchestra.run.smithi116.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2024-04-24T23:58:54.739 INFO:teuthology.orchestra.run.smithi116.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2024-04-24T23:58:54.749 INFO:teuthology.orchestra.run.smithi116.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.11-0ubuntu0.22.04.1 [144 kB] 2024-04-24T23:58:54.916 INFO:teuthology.orchestra.run.smithi116.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2024-04-24T23:58:55.508 INFO:teuthology.orchestra.run.smithi116.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2024-04-24T23:58:55.508 INFO:teuthology.orchestra.run.smithi116.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.15 [2,376 kB] 2024-04-24T23:58:55.688 INFO:teuthology.orchestra.run.smithi116.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2024-04-24T23:58:55.688 INFO:teuthology.orchestra.run.smithi116.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.3 [1,094 kB] 2024-04-24T23:58:55.727 INFO:teuthology.orchestra.run.smithi116.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2024-04-24T23:58:55.788 INFO:teuthology.orchestra.run.smithi116.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2024-04-24T23:58:55.795 INFO:teuthology.orchestra.run.smithi116.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2024-04-24T23:58:55.796 INFO:teuthology.orchestra.run.smithi116.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2024-04-24T23:58:55.797 INFO:teuthology.orchestra.run.smithi116.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2024-04-24T23:58:55.798 INFO:teuthology.orchestra.run.smithi116.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.11-0ubuntu0.22.04.1 [1,222 kB] 2024-04-24T23:58:56.007 INFO:teuthology.orchestra.run.smithi116.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2024-04-24T23:58:56.420 INFO:teuthology.orchestra.run.smithi116.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.11-0ubuntu0.22.04.1 [16.2 MB] 2024-04-24T23:58:57.215 INFO:teuthology.orchestra.run.smithi116.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2024-04-24T23:58:57.215 INFO:teuthology.orchestra.run.smithi116.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2024-04-24T23:58:57.215 INFO:teuthology.orchestra.run.smithi116.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2024-04-24T23:58:57.215 INFO:teuthology.orchestra.run.smithi116.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2024-04-24T23:58:57.249 INFO:teuthology.orchestra.run.smithi116.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2024-04-24T23:58:57.249 INFO:teuthology.orchestra.run.smithi116.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2024-04-24T23:58:57.278 INFO:teuthology.orchestra.run.smithi116.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2024-04-24T23:58:57.278 INFO:teuthology.orchestra.run.smithi116.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2024-04-24T23:58:57.686 INFO:teuthology.orchestra.run.smithi116.stdout:Preconfiguring packages ... 2024-04-24T23:58:57.850 INFO:teuthology.orchestra.run.smithi116.stdout:Fetched 55.7 MB in 5s (10.6 MB/s) 2024-04-24T23:58:58.244 INFO:teuthology.orchestra.run.smithi116.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 173984 files and directories currently installed.) 2024-04-24T23:58:58.249 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../util-linux_2.37.2-4ubuntu3.4_amd64.deb ... 2024-04-24T23:58:58.412 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking util-linux (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-04-24T23:58:59.049 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up util-linux (2.37.2-4ubuntu3.4) ... 2024-04-24T23:58:59.964 INFO:teuthology.orchestra.run.smithi116.stdout:fstrim.service is a disabled or a static unit not running, not starting it. 2024-04-24T23:59:00.241 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package flex. 2024-04-24T23:59:00.334 INFO:teuthology.orchestra.run.smithi116.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 173984 files and directories currently installed.) 2024-04-24T23:59:00.340 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2024-04-24T23:59:00.381 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking flex (2.6.4-8build2) ... 2024-04-24T23:59:00.905 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.15_amd64.deb ... 2024-04-24T23:59:01.112 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.15) over (3.0.2-0ubuntu1.7) ... 2024-04-24T23:59:01.522 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.15) ... 2024-04-24T23:59:02.043 INFO:teuthology.orchestra.run.smithi116.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 174057 files and directories currently installed.) 2024-04-24T23:59:02.048 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../uuid-dev_2.37.2-4ubuntu3.4_amd64.deb ... 2024-04-24T23:59:02.169 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking uuid-dev:amd64 (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-04-24T23:59:02.562 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../libuuid1_2.37.2-4ubuntu3.4_amd64.deb ... 2024-04-24T23:59:02.693 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libuuid1:amd64 (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-04-24T23:59:03.028 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libuuid1:amd64 (2.37.2-4ubuntu3.4) ... 2024-04-24T23:59:03.393 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package bison. 2024-04-24T23:59:03.496 INFO:teuthology.orchestra.run.smithi116.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 174057 files and directories currently installed.) 2024-04-24T23:59:03.501 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2024-04-24T23:59:03.546 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2024-04-24T23:59:03.896 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package liblzo2-2:amd64. 2024-04-24T23:59:03.924 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2024-04-24T23:59:03.954 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2024-04-24T23:59:04.301 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package dump. 2024-04-24T23:59:04.328 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2024-04-24T23:59:04.368 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking dump (0.4b47-3) ... 2024-04-24T23:59:04.730 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package indent. 2024-04-24T23:59:04.746 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2024-04-24T23:59:04.779 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2024-04-24T23:59:05.041 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libaio-dev:amd64. 2024-04-24T23:59:05.068 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2024-04-24T23:59:05.107 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2024-04-24T23:59:05.436 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libbluetooth3:amd64. 2024-04-24T23:59:05.464 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1.1_amd64.deb ... 2024-04-24T23:59:05.502 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-04-24T23:59:05.747 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libelf-dev:amd64. 2024-04-24T23:59:05.763 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2024-04-24T23:59:05.797 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2024-04-24T23:59:06.159 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libfl2:amd64. 2024-04-24T23:59:06.186 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2024-04-24T23:59:06.225 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2024-04-24T23:59:06.478 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libfl-dev:amd64. 2024-04-24T23:59:06.494 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2024-04-24T23:59:06.536 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2024-04-24T23:59:06.780 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libjson-perl. 2024-04-24T23:59:06.797 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2024-04-24T23:59:06.830 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libjson-perl (4.04000-1) ... 2024-04-24T23:59:07.142 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libllvm14:amd64. 2024-04-24T23:59:07.170 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2024-04-24T23:59:07.208 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-04-24T23:59:08.903 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2024-04-24T23:59:08.931 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-04-24T23:59:08.969 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-04-24T23:59:09.222 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libmbim-proxy. 2024-04-24T23:59:09.238 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-04-24T23:59:09.280 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-04-24T23:59:09.583 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libmm-glib0:amd64. 2024-04-24T23:59:09.611 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-04-24T23:59:09.641 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-04-24T23:59:09.969 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libndp0:amd64. 2024-04-24T23:59:09.997 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2024-04-24T23:59:10.036 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2024-04-24T23:59:10.380 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libnm0:amd64. 2024-04-24T23:59:10.408 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2024-04-24T23:59:10.446 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-04-24T23:59:10.783 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libpq5:amd64. 2024-04-24T23:59:10.811 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../16-libpq5_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-04-24T23:59:10.850 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-04-24T23:59:11.169 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2024-04-24T23:59:11.195 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-04-24T23:59:11.236 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-04-24T23:59:11.546 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libqmi-proxy. 2024-04-24T23:59:11.563 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-04-24T23:59:11.605 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-04-24T23:59:11.849 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libssl-dev:amd64. 2024-04-24T23:59:11.865 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.15_amd64.deb ... 2024-04-24T23:59:11.907 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-04-24T23:59:12.411 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package libteamdctl0:amd64. 2024-04-24T23:59:12.439 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2024-04-24T23:59:12.492 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2024-04-24T23:59:12.830 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package modemmanager. 2024-04-24T23:59:12.858 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-04-24T23:59:12.889 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-04-24T23:59:14.021 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package network-manager. 2024-04-24T23:59:14.049 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2024-04-24T23:59:14.096 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2024-04-24T23:59:14.628 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package ppp. 2024-04-24T23:59:14.656 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2024-04-24T23:59:14.935 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2024-04-24T23:59:15.240 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package pptp-linux. 2024-04-24T23:59:15.256 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2024-04-24T23:59:15.290 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2024-04-24T23:59:15.552 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package network-manager-pptp. 2024-04-24T23:59:15.568 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2024-04-24T23:59:15.610 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2024-04-24T23:59:15.912 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package postgresql-client-common. 2024-04-24T23:59:15.940 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2024-04-24T23:59:15.979 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking postgresql-client-common (238) ... 2024-04-24T23:59:16.291 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package postgresql-client-14. 2024-04-24T23:59:16.319 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../27-postgresql-client-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-04-24T23:59:16.374 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-04-24T23:59:16.954 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package postgresql-common. 2024-04-24T23:59:16.982 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2024-04-24T23:59:17.039 INFO:teuthology.orchestra.run.smithi116.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2024-04-24T23:59:17.068 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking postgresql-common (238) ... 2024-04-24T23:59:17.511 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package postgresql-14. 2024-04-24T23:59:17.539 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../29-postgresql-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-04-24T23:59:17.581 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-04-24T23:59:17.723 INFO:teuthology.orchestra.run.smithi174.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2024-04-24T23:59:17.723 INFO:teuthology.orchestra.run.smithi174.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2024-04-24T23:59:17.723 INFO:teuthology.orchestra.run.smithi174.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2024-04-24T23:59:17.723 INFO:teuthology.orchestra.run.smithi174.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2024-04-24T23:59:17.724 INFO:teuthology.orchestra.run.smithi174.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2024-04-24T23:59:17.724 INFO:teuthology.orchestra.run.smithi174.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2024-04-24T23:59:19.039 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package postgresql. 2024-04-24T23:59:19.056 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2024-04-24T23:59:19.097 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking postgresql (14+238) ... 2024-04-24T23:59:19.349 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package postgresql-client. 2024-04-24T23:59:19.377 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2024-04-24T23:59:19.449 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking postgresql-client (14+238) ... 2024-04-24T23:59:19.743 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package postgresql-contrib. 2024-04-24T23:59:19.771 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2024-04-24T23:59:19.818 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking postgresql-contrib (14+238) ... 2024-04-24T23:59:20.013 INFO:teuthology.orchestra.run.smithi174.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2024-04-24T23:59:20.070 INFO:teuthology.orchestra.run.smithi174.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2024-04-24T23:59:20.187 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package tcl8.6. 2024-04-24T23:59:20.215 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2024-04-24T23:59:20.254 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2024-04-24T23:59:20.539 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package tcl. 2024-04-24T23:59:20.567 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2024-04-24T23:59:20.586 INFO:teuthology.orchestra.run.smithi174.stdout:Preconfiguring packages ... 2024-04-24T23:59:20.611 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking tcl (8.6.11+1build2) ... 2024-04-24T23:59:20.724 INFO:teuthology.orchestra.run.smithi174.stdout:Fetched 55.7 MB in 32s (1,723 kB/s) 2024-04-24T23:59:20.967 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package wpasupplicant. 2024-04-24T23:59:20.995 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2024-04-24T23:59:21.034 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2024-04-24T23:59:21.046 INFO:teuthology.orchestra.run.smithi174.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 173984 files and directories currently installed.) 2024-04-24T23:59:21.052 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../util-linux_2.37.2-4ubuntu3.4_amd64.deb ... 2024-04-24T23:59:21.197 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking util-linux (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-04-24T23:59:21.396 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package usb-modeswitch-data. 2024-04-24T23:59:21.412 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2024-04-24T23:59:21.454 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2024-04-24T23:59:21.749 INFO:teuthology.orchestra.run.smithi116.stdout:Selecting previously unselected package usb-modeswitch. 2024-04-24T23:59:21.777 INFO:teuthology.orchestra.run.smithi116.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2024-04-24T23:59:21.816 INFO:teuthology.orchestra.run.smithi116.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2024-04-24T23:59:22.112 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up postgresql-client-common (238) ... 2024-04-24T23:59:22.192 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up util-linux (2.37.2-4ubuntu3.4) ... 2024-04-24T23:59:22.321 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up flex (2.6.4-8build2) ... 2024-04-24T23:59:22.422 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2024-04-24T23:59:22.549 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-04-24T23:59:22.666 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2024-04-24T23:59:22.792 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2024-04-24T23:59:22.918 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-04-24T23:59:23.112 INFO:teuthology.orchestra.run.smithi174.stdout:fstrim.service is a disabled or a static unit not running, not starting it. 2024-04-24T23:59:23.371 INFO:teuthology.orchestra.run.smithi116.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2024-04-24T23:59:23.376 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package flex. 2024-04-24T23:59:23.440 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2024-04-24T23:59:23.476 INFO:teuthology.orchestra.run.smithi174.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 173984 files and directories currently installed.) 2024-04-24T23:59:23.482 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2024-04-24T23:59:23.532 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking flex (2.6.4-8build2) ... 2024-04-24T23:59:23.817 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2024-04-24T23:59:23.817 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2024-04-24T23:59:23.965 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.15_amd64.deb ... 2024-04-24T23:59:24.121 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.15) over (3.0.2-0ubuntu1.7) ... 2024-04-24T23:59:24.531 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.15) ... 2024-04-24T23:59:24.677 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up uuid-dev:amd64 (2.37.2-4ubuntu3.4) ... 2024-04-24T23:59:24.803 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2024-04-24T23:59:24.929 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-04-24T23:59:25.054 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2024-04-24T23:59:25.067 INFO:teuthology.orchestra.run.smithi174.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 174057 files and directories currently installed.) 2024-04-24T23:59:25.072 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../uuid-dev_2.37.2-4ubuntu3.4_amd64.deb ... 2024-04-24T23:59:25.180 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2024-04-24T23:59:25.195 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking uuid-dev:amd64 (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-04-24T23:59:25.281 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2024-04-24T23:59:25.382 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-04-24T23:59:25.508 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2024-04-24T23:59:25.612 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../libuuid1_2.37.2-4ubuntu3.4_amd64.deb ... 2024-04-24T23:59:25.743 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libuuid1:amd64 (2.37.2-4ubuntu3.4) over (2.37.2-4ubuntu3) ... 2024-04-24T23:59:26.078 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libuuid1:amd64 (2.37.2-4ubuntu3.4) ... 2024-04-24T23:59:26.196 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-04-24T23:59:26.368 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package bison. 2024-04-24T23:59:26.379 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up dump (0.4b47-3) ... 2024-04-24T23:59:26.464 INFO:teuthology.orchestra.run.smithi174.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 174057 files and directories currently installed.) 2024-04-24T23:59:26.469 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2024-04-24T23:59:26.521 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2024-04-24T23:59:26.525 INFO:teuthology.orchestra.run.smithi116.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2024-04-24T23:59:26.591 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-04-24T23:59:26.717 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-04-24T23:59:26.835 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up tcl (8.6.11+1build2) ... 2024-04-24T23:59:26.904 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package liblzo2-2:amd64. 2024-04-24T23:59:26.932 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2024-04-24T23:59:26.944 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libjson-perl (4.04000-1) ... 2024-04-24T23:59:26.971 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2024-04-24T23:59:27.053 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2024-04-24T23:59:27.144 INFO:teuthology.orchestra.run.smithi116.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2024-04-24T23:59:27.213 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2024-04-24T23:59:27.299 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package dump. 2024-04-24T23:59:27.326 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2024-04-24T23:59:27.330 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2024-04-24T23:59:27.365 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking dump (0.4b47-3) ... 2024-04-24T23:59:27.482 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2024-04-24T23:59:27.583 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-04-24T23:59:27.669 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package indent. 2024-04-24T23:59:27.685 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2024-04-24T23:59:27.709 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2024-04-24T23:59:27.727 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2024-04-24T23:59:27.877 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up postgresql-client (14+238) ... 2024-04-24T23:59:27.955 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libaio-dev:amd64. 2024-04-24T23:59:27.971 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2024-04-24T23:59:28.003 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2024-04-24T23:59:28.004 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2024-04-24T23:59:28.333 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libbluetooth3:amd64. 2024-04-24T23:59:28.361 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1.1_amd64.deb ... 2024-04-24T23:59:28.400 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-04-24T23:59:28.490 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-24T23:59:28.490 INFO:teuthology.orchestra.run.smithi116.stdout:The following network interfaces were found in /etc/network/interfaces 2024-04-24T23:59:28.490 INFO:teuthology.orchestra.run.smithi116.stdout:which means they are currently configured by ifupdown: 2024-04-24T23:59:28.490 INFO:teuthology.orchestra.run.smithi116.stdout:- enp3s0f1 2024-04-24T23:59:28.490 INFO:teuthology.orchestra.run.smithi116.stdout:If you want to manage those interfaces with NetworkManager instead 2024-04-24T23:59:28.490 INFO:teuthology.orchestra.run.smithi116.stdout:remove their configuration from /etc/network/interfaces. 2024-04-24T23:59:28.490 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-24T23:59:28.614 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2024-04-24T23:59:28.654 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libelf-dev:amd64. 2024-04-24T23:59:28.669 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2024-04-24T23:59:28.711 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2024-04-24T23:59:28.929 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2024-04-24T23:59:29.022 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libfl2:amd64. 2024-04-24T23:59:29.050 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2024-04-24T23:59:29.081 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2024-04-24T23:59:29.270 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2024-04-24T23:59:29.316 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libfl-dev:amd64. 2024-04-24T23:59:29.333 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2024-04-24T23:59:29.374 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2024-04-24T23:59:29.653 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libjson-perl. 2024-04-24T23:59:29.680 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2024-04-24T23:59:29.719 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libjson-perl (4.04000-1) ... 2024-04-24T23:59:29.780 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-04-24T23:59:29.898 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2024-04-24T23:59:29.999 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2024-04-24T23:59:30.005 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libllvm14:amd64. 2024-04-24T23:59:30.034 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2024-04-24T23:59:30.072 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-04-24T23:59:30.108 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up postgresql-common (238) ... 2024-04-24T23:59:31.064 INFO:teuthology.orchestra.run.smithi116.stdout:Adding user postgres to group ssl-cert 2024-04-24T23:59:31.281 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-24T23:59:31.282 INFO:teuthology.orchestra.run.smithi116.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2024-04-24T23:59:31.499 INFO:teuthology.orchestra.run.smithi116.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2024-04-24T23:59:31.500 INFO:teuthology.orchestra.run.smithi116.stdout:Removing obsolete dictionary files: 2024-04-24T23:59:31.691 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2024-04-24T23:59:31.719 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-04-24T23:59:31.758 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-04-24T23:59:32.018 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libmbim-proxy. 2024-04-24T23:59:32.046 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2024-04-24T23:59:32.050 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2024-04-24T23:59:32.085 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-04-24T23:59:32.413 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libmm-glib0:amd64. 2024-04-24T23:59:32.440 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-04-24T23:59:32.443 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-04-24T23:59:32.479 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-04-24T23:59:32.757 INFO:teuthology.orchestra.run.smithi116.stdout:Creating new PostgreSQL cluster 14/main ... 2024-04-24T23:59:32.770 INFO:teuthology.orchestra.run.smithi116.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2024-04-24T23:59:32.807 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libndp0:amd64. 2024-04-24T23:59:32.822 INFO:teuthology.orchestra.run.smithi116.stdout:The files belonging to this database system will be owned by user "postgres". 2024-04-24T23:59:32.823 INFO:teuthology.orchestra.run.smithi116.stdout:This user must also own the server process. 2024-04-24T23:59:32.823 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-24T23:59:32.823 INFO:teuthology.orchestra.run.smithi116.stdout:The database cluster will be initialized with locale "en_US". 2024-04-24T23:59:32.823 INFO:teuthology.orchestra.run.smithi116.stdout:The default database encoding has accordingly been set to "LATIN1". 2024-04-24T23:59:32.823 INFO:teuthology.orchestra.run.smithi116.stdout:The default text search configuration will be set to "english". 2024-04-24T23:59:32.823 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-24T23:59:32.823 INFO:teuthology.orchestra.run.smithi116.stdout:Data page checksums are disabled. 2024-04-24T23:59:32.823 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-24T23:59:32.823 INFO:teuthology.orchestra.run.smithi116.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2024-04-24T23:59:32.823 INFO:teuthology.orchestra.run.smithi116.stdout:creating subdirectories ... ok 2024-04-24T23:59:32.823 INFO:teuthology.orchestra.run.smithi116.stdout:selecting dynamic shared memory implementation ... posix 2024-04-24T23:59:32.835 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2024-04-24T23:59:32.838 INFO:teuthology.orchestra.run.smithi116.stdout:selecting default max_connections ... 100 2024-04-24T23:59:32.859 INFO:teuthology.orchestra.run.smithi116.stdout:selecting default shared_buffers ... 128MB 2024-04-24T23:59:32.869 INFO:teuthology.orchestra.run.smithi116.stdout:selecting default time zone ... Etc/UTC 2024-04-24T23:59:32.870 INFO:teuthology.orchestra.run.smithi116.stdout:creating configuration files ... ok 2024-04-24T23:59:32.874 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2024-04-24T23:59:32.985 INFO:teuthology.orchestra.run.smithi116.stdout:running bootstrap script ... ok 2024-04-24T23:59:33.176 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libnm0:amd64. 2024-04-24T23:59:33.204 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2024-04-24T23:59:33.243 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-04-24T23:59:33.563 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libpq5:amd64. 2024-04-24T23:59:33.590 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../16-libpq5_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-04-24T23:59:33.618 INFO:teuthology.orchestra.run.smithi116.stdout:performing post-bootstrap initialization ... ok 2024-04-24T23:59:33.629 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-04-24T23:59:33.999 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2024-04-24T23:59:34.026 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-04-24T23:59:34.065 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-04-24T23:59:34.368 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libqmi-proxy. 2024-04-24T23:59:34.372 INFO:teuthology.orchestra.run.smithi116.stdout:syncing data to disk ... ok 2024-04-24T23:59:34.395 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2024-04-24T23:59:34.434 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-04-24T23:59:34.645 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libssl-dev:amd64. 2024-04-24T23:59:34.661 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.15_amd64.deb ... 2024-04-24T23:59:34.695 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-04-24T23:59:34.766 INFO:teuthology.orchestra.run.smithi116.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2024-04-24T23:59:35.190 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package libteamdctl0:amd64. 2024-04-24T23:59:35.218 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2024-04-24T23:59:35.257 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2024-04-24T23:59:35.568 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package modemmanager. 2024-04-24T23:59:35.596 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.3_amd64.deb ... 2024-04-24T23:59:35.634 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-04-24T23:59:36.183 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package network-manager. 2024-04-24T23:59:36.211 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2024-04-24T23:59:36.249 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2024-04-24T23:59:36.865 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package ppp. 2024-04-24T23:59:36.893 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2024-04-24T23:59:37.165 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2024-04-24T23:59:37.452 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package pptp-linux. 2024-04-24T23:59:37.468 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2024-04-24T23:59:37.502 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2024-04-24T23:59:37.642 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-04-24T23:59:37.705 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package network-manager-pptp. 2024-04-24T23:59:37.721 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2024-04-24T23:59:37.768 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up postgresql-contrib (14+238) ... 2024-04-24T23:59:37.788 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2024-04-24T23:59:37.919 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up postgresql (14+238) ... 2024-04-24T23:59:38.045 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-04-24T23:59:38.091 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package postgresql-client-common. 2024-04-24T23:59:38.119 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2024-04-24T23:59:38.157 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking postgresql-client-common (238) ... 2024-04-24T23:59:38.171 INFO:teuthology.orchestra.run.smithi116.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-04-24T23:59:38.428 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2024-04-24T23:59:38.428 INFO:teuthology.orchestra.run.smithi116.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2024-04-24T23:59:38.486 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package postgresql-client-14. 2024-04-24T23:59:38.514 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../27-postgresql-client-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-04-24T23:59:38.553 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-04-24T23:59:38.981 INFO:teuthology.orchestra.run.smithi116.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-24T23:59:39.057 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package postgresql-common. 2024-04-24T23:59:39.074 INFO:teuthology.orchestra.run.smithi116.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2024-04-24T23:59:39.085 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2024-04-24T23:59:39.134 INFO:teuthology.orchestra.run.smithi174.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2024-04-24T23:59:39.154 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking postgresql-common (238) ... 2024-04-24T23:59:39.384 INFO:teuthology.orchestra.run.smithi116.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-24T23:59:39.536 INFO:teuthology.orchestra.run.smithi116.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-24T23:59:39.616 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package postgresql-14. 2024-04-24T23:59:39.644 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../29-postgresql-14_14.11-0ubuntu0.22.04.1_amd64.deb ... 2024-04-24T23:59:39.685 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-04-24T23:59:40.901 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package postgresql. 2024-04-24T23:59:40.930 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2024-04-24T23:59:40.959 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking postgresql (14+238) ... 2024-04-24T23:59:41.064 INFO:teuthology.orchestra.run.smithi116.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2024-04-24T23:59:41.147 INFO:teuthology.orchestra.run.smithi116.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-24T23:59:41.186 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package postgresql-client. 2024-04-24T23:59:41.214 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2024-04-24T23:59:41.253 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking postgresql-client (14+238) ... 2024-04-24T23:59:41.497 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package postgresql-contrib. 2024-04-24T23:59:41.525 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2024-04-24T23:59:41.563 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking postgresql-contrib (14+238) ... 2024-04-24T23:59:41.655 INFO:teuthology.orchestra.run.smithi116.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-24T23:59:41.661 DEBUG:teuthology.parallel:result is None 2024-04-24T23:59:41.907 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package tcl8.6. 2024-04-24T23:59:41.936 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2024-04-24T23:59:41.974 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2024-04-24T23:59:42.243 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package tcl. 2024-04-24T23:59:42.259 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2024-04-24T23:59:42.306 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking tcl (8.6.11+1build2) ... 2024-04-24T23:59:42.621 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package wpasupplicant. 2024-04-24T23:59:42.649 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2024-04-24T23:59:42.687 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2024-04-24T23:59:43.050 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package usb-modeswitch-data. 2024-04-24T23:59:43.078 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2024-04-24T23:59:43.116 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2024-04-24T23:59:43.477 INFO:teuthology.orchestra.run.smithi174.stdout:Selecting previously unselected package usb-modeswitch. 2024-04-24T23:59:43.505 INFO:teuthology.orchestra.run.smithi174.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2024-04-24T23:59:43.552 INFO:teuthology.orchestra.run.smithi174.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2024-04-24T23:59:43.840 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up postgresql-client-common (238) ... 2024-04-24T23:59:44.019 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up flex (2.6.4-8build2) ... 2024-04-24T23:59:44.142 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2024-04-24T23:59:44.268 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libpq5:amd64 (14.11-0ubuntu0.22.04.1) ... 2024-04-24T23:59:44.394 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2024-04-24T23:59:44.520 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2024-04-24T23:59:44.645 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up postgresql-client-14 (14.11-0ubuntu0.22.04.1) ... 2024-04-24T23:59:45.098 INFO:teuthology.orchestra.run.smithi174.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2024-04-24T23:59:45.176 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2024-04-24T23:59:45.549 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2024-04-24T23:59:45.550 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2024-04-24T23:59:46.029 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up uuid-dev:amd64 (2.37.2-4ubuntu3.4) ... 2024-04-24T23:59:46.154 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2024-04-24T23:59:46.255 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.15) ... 2024-04-24T23:59:46.381 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2024-04-24T23:59:46.507 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2024-04-24T23:59:46.633 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2024-04-24T23:59:46.759 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2024-04-24T23:59:46.885 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2024-04-24T23:59:47.865 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2024-04-24T23:59:47.991 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up dump (0.4b47-3) ... 2024-04-24T23:59:48.084 INFO:teuthology.orchestra.run.smithi174.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2024-04-24T23:59:48.151 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.3) ... 2024-04-24T23:59:48.277 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1.1) ... 2024-04-24T23:59:48.402 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up tcl (8.6.11+1build2) ... 2024-04-24T23:59:48.528 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libjson-perl (4.04000-1) ... 2024-04-24T23:59:48.654 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2024-04-24T23:59:48.744 INFO:teuthology.orchestra.run.smithi174.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2024-04-24T23:59:48.797 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2024-04-24T23:59:48.915 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2024-04-24T23:59:49.091 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2024-04-24T23:59:49.217 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2024-04-24T23:59:49.343 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2024-04-24T23:59:49.511 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up postgresql-client (14+238) ... 2024-04-24T23:59:49.637 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2024-04-24T23:59:50.105 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-24T23:59:50.106 INFO:teuthology.orchestra.run.smithi174.stdout:The following network interfaces were found in /etc/network/interfaces 2024-04-24T23:59:50.106 INFO:teuthology.orchestra.run.smithi174.stdout:which means they are currently configured by ifupdown: 2024-04-24T23:59:50.106 INFO:teuthology.orchestra.run.smithi174.stdout:- enp3s0f1 2024-04-24T23:59:50.106 INFO:teuthology.orchestra.run.smithi174.stdout:If you want to manage those interfaces with NetworkManager instead 2024-04-24T23:59:50.106 INFO:teuthology.orchestra.run.smithi174.stdout:remove their configuration from /etc/network/interfaces. 2024-04-24T23:59:50.106 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-24T23:59:50.234 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2024-04-24T23:59:50.553 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2024-04-24T23:59:50.871 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2024-04-24T23:59:51.348 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2024-04-24T23:59:51.491 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2024-04-24T23:59:51.617 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2024-04-24T23:59:51.742 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up postgresql-common (238) ... 2024-04-24T23:59:52.617 INFO:teuthology.orchestra.run.smithi174.stdout:Adding user postgres to group ssl-cert 2024-04-24T23:59:52.837 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-24T23:59:52.837 INFO:teuthology.orchestra.run.smithi174.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2024-04-24T23:59:53.014 INFO:teuthology.orchestra.run.smithi174.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2024-04-24T23:59:53.014 INFO:teuthology.orchestra.run.smithi174.stdout:Removing obsolete dictionary files: 2024-04-24T23:59:53.548 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2024-04-24T23:59:53.935 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up postgresql-14 (14.11-0ubuntu0.22.04.1) ... 2024-04-24T23:59:54.275 INFO:teuthology.orchestra.run.smithi174.stdout:Creating new PostgreSQL cluster 14/main ... 2024-04-24T23:59:54.291 INFO:teuthology.orchestra.run.smithi174.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2024-04-24T23:59:54.330 INFO:teuthology.orchestra.run.smithi174.stdout:The files belonging to this database system will be owned by user "postgres". 2024-04-24T23:59:54.330 INFO:teuthology.orchestra.run.smithi174.stdout:This user must also own the server process. 2024-04-24T23:59:54.330 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-24T23:59:54.330 INFO:teuthology.orchestra.run.smithi174.stdout:The database cluster will be initialized with locale "en_US". 2024-04-24T23:59:54.330 INFO:teuthology.orchestra.run.smithi174.stdout:The default database encoding has accordingly been set to "LATIN1". 2024-04-24T23:59:54.330 INFO:teuthology.orchestra.run.smithi174.stdout:The default text search configuration will be set to "english". 2024-04-24T23:59:54.331 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-24T23:59:54.331 INFO:teuthology.orchestra.run.smithi174.stdout:Data page checksums are disabled. 2024-04-24T23:59:54.331 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-24T23:59:54.331 INFO:teuthology.orchestra.run.smithi174.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2024-04-24T23:59:54.331 INFO:teuthology.orchestra.run.smithi174.stdout:creating subdirectories ... ok 2024-04-24T23:59:54.331 INFO:teuthology.orchestra.run.smithi174.stdout:selecting dynamic shared memory implementation ... posix 2024-04-24T23:59:54.343 INFO:teuthology.orchestra.run.smithi174.stdout:selecting default max_connections ... 100 2024-04-24T23:59:54.361 INFO:teuthology.orchestra.run.smithi174.stdout:selecting default shared_buffers ... 128MB 2024-04-24T23:59:54.371 INFO:teuthology.orchestra.run.smithi174.stdout:selecting default time zone ... Etc/UTC 2024-04-24T23:59:54.371 INFO:teuthology.orchestra.run.smithi174.stdout:creating configuration files ... ok 2024-04-24T23:59:54.483 INFO:teuthology.orchestra.run.smithi174.stdout:running bootstrap script ... ok 2024-04-24T23:59:55.065 INFO:teuthology.orchestra.run.smithi174.stdout:performing post-bootstrap initialization ... ok 2024-04-24T23:59:55.690 INFO:teuthology.orchestra.run.smithi174.stdout:syncing data to disk ... ok 2024-04-24T23:59:56.105 INFO:teuthology.orchestra.run.smithi174.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2024-04-24T23:59:58.858 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2024-04-24T23:59:58.984 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up postgresql-contrib (14+238) ... 2024-04-24T23:59:59.110 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up postgresql (14+238) ... 2024-04-24T23:59:59.235 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2024-04-24T23:59:59.361 INFO:teuthology.orchestra.run.smithi174.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.3) ... 2024-04-24T23:59:59.615 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2024-04-24T23:59:59.615 INFO:teuthology.orchestra.run.smithi174.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2024-04-25T00:00:00.189 INFO:teuthology.orchestra.run.smithi174.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-25T00:00:00.349 INFO:teuthology.orchestra.run.smithi174.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2024-04-25T00:00:00.676 INFO:teuthology.orchestra.run.smithi174.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-25T00:00:00.820 INFO:teuthology.orchestra.run.smithi174.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-25T00:00:02.355 INFO:teuthology.orchestra.run.smithi174.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2024-04-25T00:00:02.480 INFO:teuthology.orchestra.run.smithi174.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-25T00:00:02.963 INFO:teuthology.orchestra.run.smithi174.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-25T00:00:02.967 DEBUG:teuthology.parallel:result is None 2024-04-25T00:00:02.968 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T00:00:03.195 DEBUG:teuthology.orchestra.run.smithi116:> dpkg-query -W -f '${Version}' ceph 2024-04-25T00:00:03.214 INFO:teuthology.orchestra.run.smithi116.stdout:19.0.0-2445-gd2c45d97-1jammy 2024-04-25T00:00:03.215 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2445-gd2c45d97-1jammy 2024-04-25T00:00:03.215 INFO:teuthology.task.install:The correct ceph version 19.0.0-2445-gd2c45d97-1jammy is installed. 2024-04-25T00:00:03.217 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T00:00:03.438 DEBUG:teuthology.orchestra.run.smithi174:> dpkg-query -W -f '${Version}' ceph 2024-04-25T00:00:03.458 INFO:teuthology.orchestra.run.smithi174.stdout:19.0.0-2445-gd2c45d97-1jammy 2024-04-25T00:00:03.458 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2445-gd2c45d97-1jammy 2024-04-25T00:00:03.458 INFO:teuthology.task.install:The correct ceph version 19.0.0-2445-gd2c45d97-1jammy is installed. 2024-04-25T00:00:03.460 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-25T00:00:03.460 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-04-25T00:00:03.460 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T00:00:03.475 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-04-25T00:00:03.475 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T00:00:03.512 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-25T00:00:03.513 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-04-25T00:00:03.513 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T00:00:03.527 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T00:00:03.582 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-04-25T00:00:03.583 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T00:00:03.596 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T00:00:03.651 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-25T00:00:03.651 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-04-25T00:00:03.652 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T00:00:03.664 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T00:00:03.716 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-04-25T00:00:03.716 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T00:00:03.729 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T00:00:03.783 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-25T00:00:03.783 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-04-25T00:00:03.784 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T00:00:03.795 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T00:00:03.854 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-04-25T00:00:03.854 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T00:00:03.868 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T00:00:03.924 INFO:teuthology.run_tasks:Running task ceph... 2024-04-25T00:00:04.011 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-25T00:00:04.011 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 777 /var/log/ceph 2024-04-25T00:00:04.014 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 777 /var/log/ceph 2024-04-25T00:00:04.027 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-25T00:00:04.027 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-25T00:00:04.067 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-25T00:00:04.080 INFO:tasks.ceph:Creating extra log directories... 2024-04-25T00:00:04.081 DEBUG:teuthology.orchestra.run.smithi116:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-25T00:00:04.120 DEBUG:teuthology.orchestra.run.smithi174:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-25T00:00:04.142 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-04-25T00:00:04.142 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-04-25T00:00:04.142 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-04-25T00:00:04.142 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-04-25T00:00:04.143 DEBUG:teuthology.orchestra.run.smithi116:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-04-25T00:00:04.178 DEBUG:teuthology.orchestra.run.smithi116:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-04-25T00:00:04.258 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-04-25T00:00:04.258 DEBUG:teuthology.orchestra.run.smithi174:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-04-25T00:00:04.264 DEBUG:teuthology.orchestra.run.smithi174:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-04-25T00:00:04.347 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-25T00:00:04.347 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'filestore flush min': 0, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'Replacing daemon mds', 'SLOW_OPS', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-25T00:00:04.347 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672224', 'branch': 'squid', 'description': 'fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7672224', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'mgrmodules': {'sequential': [{'print': 'Enabling mgr modules'}]}, 'name': 'teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'cephfs': {'fs': [{'name': 'a'}, {'name': 'b'}], 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'filestore flush min': 0, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'Replacing daemon mds', 'SLOW_OPS', 'slow request'], 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'ceph-fuse': {'client.0': {'cephfs_name': 'a'}, 'client.1': {'cephfs_name': 'b'}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'}}, 'kclient': {'client.0': {'cephfs_name': 'a'}, 'client.1': {'cephfs_name': 'b'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']], 'seed': 8528, 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'sleep_before_teardown': 0, 'subset': '122/512', 'suite': 'fs', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi116.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBZUErwp5x/iXFsF5BkmYc2BOevi/pGO8L/YzJPjxT/XGX/GymjVee/7ER30H/8Spn1Vf3u3LChuWJYK5jVPz4g=', 'smithi174.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOELdnCNwpalxyvSIzsFkB1x0eynd3fo9oN+EekJ9w0Ohu/oWPwHdJAwN+hH7oCyaqXhva52JR/KPNxC0lso5I8='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'fs': [{'name': 'a'}, {'name': 'b'}], 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'filestore flush min': 0, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'Replacing daemon mds', 'SLOW_OPS', 'slow request'], 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'cluster': 'ceph'}}, {'sequential': ['mgrmodules']}, {'ceph-fuse': None}, {'mds_thrash': None}, {'workunit': {'clients': {'all': ['suites/ffsb.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-24_21:24:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-25T00:00:04.348 DEBUG:teuthology.orchestra.run.smithi116:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-25T00:00:04.350 DEBUG:teuthology.orchestra.run.smithi174:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-25T00:00:04.389 DEBUG:teuthology.orchestra.run.smithi116:> sudo install -d -m0777 -- /var/run/ceph 2024-04-25T00:00:04.400 DEBUG:teuthology.orchestra.run.smithi174:> sudo install -d -m0777 -- /var/run/ceph 2024-04-25T00:00:04.445 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-04-25T00:00:04.445 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T00:00:04.459 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T00:00:04.459 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_1 2024-04-25T00:00:04.507 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T00:00:04.507 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T00:00:04.508 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 869 Links: 1 2024-04-25T00:00:04.508 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T00:00:04.508 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-04-24 23:53:34.593834273 +0000 2024-04-25T00:00:04.508 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-04-24 23:53:34.345836489 +0000 2024-04-25T00:00:04.508 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-04-24 23:53:34.345836489 +0000 2024-04-25T00:00:04.508 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: - 2024-04-25T00:00:04.509 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T00:00:04.561 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-04-25T00:00:04.561 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-04-25T00:00:04.561 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000386575 s, 1.3 MB/s 2024-04-25T00:00:04.562 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T00:00:04.611 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_2 2024-04-25T00:00:04.658 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T00:00:04.659 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T00:00:04.659 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 892 Links: 1 2024-04-25T00:00:04.659 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T00:00:04.659 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-04-24 23:53:35.077829950 +0000 2024-04-25T00:00:04.659 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-04-24 23:53:34.821832238 +0000 2024-04-25T00:00:04.659 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-04-24 23:53:34.821832238 +0000 2024-04-25T00:00:04.659 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: - 2024-04-25T00:00:04.659 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T00:00:04.713 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-04-25T00:00:04.713 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-04-25T00:00:04.713 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000369426 s, 1.4 MB/s 2024-04-25T00:00:04.714 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T00:00:04.764 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_3 2024-04-25T00:00:04.810 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T00:00:04.810 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T00:00:04.811 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 909 Links: 1 2024-04-25T00:00:04.811 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T00:00:04.811 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-04-24 23:53:35.553825699 +0000 2024-04-25T00:00:04.811 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-04-24 23:53:35.297827985 +0000 2024-04-25T00:00:04.811 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-04-24 23:53:35.297827985 +0000 2024-04-25T00:00:04.811 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: - 2024-04-25T00:00:04.811 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T00:00:04.864 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-04-25T00:00:04.865 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-04-25T00:00:04.865 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.00042432 s, 1.2 MB/s 2024-04-25T00:00:04.866 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T00:00:04.916 DEBUG:teuthology.orchestra.run.smithi116:> stat /dev/vg_nvme/lv_4 2024-04-25T00:00:04.963 INFO:teuthology.orchestra.run.smithi116.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T00:00:04.963 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T00:00:04.963 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 5h/5d Inode: 927 Links: 1 2024-04-25T00:00:04.963 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T00:00:04.963 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 2024-04-24 23:53:36.021821519 +0000 2024-04-25T00:00:04.963 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-04-24 23:53:35.789823591 +0000 2024-04-25T00:00:04.964 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-04-24 23:53:35.789823591 +0000 2024-04-25T00:00:04.964 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: - 2024-04-25T00:00:04.964 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T00:00:05.017 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records in 2024-04-25T00:00:05.018 INFO:teuthology.orchestra.run.smithi116.stderr:1+0 records out 2024-04-25T00:00:05.018 INFO:teuthology.orchestra.run.smithi116.stderr:512 bytes copied, 0.000363115 s, 1.4 MB/s 2024-04-25T00:00:05.019 DEBUG:teuthology.orchestra.run.smithi116:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T00:00:05.068 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-25T00:00:05.068 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-04-25T00:00:05.068 DEBUG:teuthology.orchestra.run.smithi174:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T00:00:05.074 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T00:00:05.074 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_1 2024-04-25T00:00:05.121 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T00:00:05.121 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T00:00:05.122 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 872 Links: 1 2024-04-25T00:00:05.122 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T00:00:05.122 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-04-24 23:53:34.572481904 +0000 2024-04-25T00:00:05.122 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-04-24 23:53:34.324484117 +0000 2024-04-25T00:00:05.122 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-04-24 23:53:34.324484117 +0000 2024-04-25T00:00:05.122 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: - 2024-04-25T00:00:05.122 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T00:00:05.175 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-04-25T00:00:05.176 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-04-25T00:00:05.176 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000449243 s, 1.1 MB/s 2024-04-25T00:00:05.177 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T00:00:05.230 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_2 2024-04-25T00:00:05.281 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T00:00:05.281 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T00:00:05.281 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 888 Links: 1 2024-04-25T00:00:05.281 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T00:00:05.281 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-04-24 23:53:35.064477516 +0000 2024-04-25T00:00:05.281 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-04-24 23:53:34.796479906 +0000 2024-04-25T00:00:05.281 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-04-24 23:53:34.796479906 +0000 2024-04-25T00:00:05.281 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: - 2024-04-25T00:00:05.281 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T00:00:05.335 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-04-25T00:00:05.335 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-04-25T00:00:05.336 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000421047 s, 1.2 MB/s 2024-04-25T00:00:05.336 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T00:00:05.386 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_3 2024-04-25T00:00:05.433 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T00:00:05.433 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T00:00:05.433 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 908 Links: 1 2024-04-25T00:00:05.433 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T00:00:05.433 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-04-24 23:53:35.540473268 +0000 2024-04-25T00:00:05.433 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-04-24 23:53:35.292475481 +0000 2024-04-25T00:00:05.433 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-04-24 23:53:35.292475481 +0000 2024-04-25T00:00:05.433 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: - 2024-04-25T00:00:05.434 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T00:00:05.486 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-04-25T00:00:05.486 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-04-25T00:00:05.487 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000314283 s, 1.6 MB/s 2024-04-25T00:00:05.488 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T00:00:05.538 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_4 2024-04-25T00:00:05.584 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T00:00:05.584 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T00:00:05.585 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 927 Links: 1 2024-04-25T00:00:05.585 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T00:00:05.585 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-04-24 23:53:36.008469094 +0000 2024-04-25T00:00:05.585 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-04-24 23:53:35.764471271 +0000 2024-04-25T00:00:05.585 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-04-24 23:53:35.764471271 +0000 2024-04-25T00:00:05.585 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: - 2024-04-25T00:00:05.585 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T00:00:05.639 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-04-25T00:00:05.639 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-04-25T00:00:05.639 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000378047 s, 1.4 MB/s 2024-04-25T00:00:05.640 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T00:00:05.690 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-25T00:00:05.690 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi116.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi174.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-25T00:00:05.690 INFO:tasks.ceph:Generating config... 2024-04-25T00:00:05.691 INFO:tasks.ceph:[client] client mount timeout = 600 2024-04-25T00:00:05.691 INFO:tasks.ceph:[client] client_shutdown_timeout = 120 2024-04-25T00:00:05.691 INFO:tasks.ceph:[client] debug client = 20 2024-04-25T00:00:05.691 INFO:tasks.ceph:[client] debug ms = 1 2024-04-25T00:00:05.691 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-04-25T00:00:05.691 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-04-25T00:00:05.692 INFO:tasks.ceph:[mds] debug mds = 20 2024-04-25T00:00:05.692 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-04-25T00:00:05.692 INFO:tasks.ceph:[mds] debug ms = 1 2024-04-25T00:00:05.692 INFO:tasks.ceph:[mds] mds bal fragment size max = 10000 2024-04-25T00:00:05.692 INFO:tasks.ceph:[mds] mds bal merge size = 5 2024-04-25T00:00:05.692 INFO:tasks.ceph:[mds] mds bal split bits = 3 2024-04-25T00:00:05.692 INFO:tasks.ceph:[mds] mds bal split size = 100 2024-04-25T00:00:05.692 INFO:tasks.ceph:[mds] mds debug frag = True 2024-04-25T00:00:05.692 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-04-25T00:00:05.692 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-04-25T00:00:05.692 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-04-25T00:00:05.692 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-04-25T00:00:05.692 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-04-25T00:00:05.692 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-04-25T00:00:05.692 INFO:tasks.ceph:[mgr] debug client = 20 2024-04-25T00:00:05.693 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-25T00:00:05.693 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-25T00:00:05.693 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-25T00:00:05.693 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-25T00:00:05.693 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-25T00:00:05.693 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-04-25T00:00:05.693 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-04-25T00:00:05.693 INFO:tasks.ceph:[osd] bdev async discard = True 2024-04-25T00:00:05.693 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-04-25T00:00:05.693 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-04-25T00:00:05.693 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-25T00:00:05.693 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-25T00:00:05.693 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-04-25T00:00:05.693 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-04-25T00:00:05.693 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-25T00:00:05.694 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-25T00:00:05.694 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-04-25T00:00:05.694 INFO:tasks.ceph:[osd] filestore flush min = 0 2024-04-25T00:00:05.694 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-25T00:00:05.694 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-25T00:00:05.694 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-25T00:00:05.694 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-04-25T00:00:05.694 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-25T00:00:05.694 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-04-25T00:00:05.694 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-25T00:00:05.694 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-04-25T00:00:05.694 INFO:tasks.ceph:Setting up mon.a... 2024-04-25T00:00:05.695 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-25T00:00:05.728 INFO:teuthology.orchestra.run.smithi116.stdout:creating /etc/ceph/ceph.keyring 2024-04-25T00:00:05.748 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-25T00:00:05.826 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-25T00:00:05.886 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.116'), ('mon.b', '172.21.15.174'), ('mon.c', '[v2:172.21.15.174:3301,v1:172.21.15.174:6790]')] 2024-04-25T00:00:05.887 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.116,172.21.15.174,[v2:172.21.15.174:3301,v1:172.21.15.174:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'filestore flush min': 0, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-04-25T00:00:05.888 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-04-25T00:00:05.888 DEBUG:teuthology.orchestra.run.smithi116:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-25T00:00:05.935 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.116 --add b 172.21.15.174 --addv c '[v2:172.21.15.174:3301,v1:172.21.15.174:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-04-25T00:00:06.021 INFO:teuthology.orchestra.run.smithi116.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-25T00:00:06.021 INFO:teuthology.orchestra.run.smithi116.stdout:monmaptool: generated fsid 6d6a1aa7-0478-4c83-8057-05917e81f046 2024-04-25T00:00:06.021 INFO:teuthology.orchestra.run.smithi116.stdout:setting min_mon_release = quincy 2024-04-25T00:00:06.021 INFO:teuthology.orchestra.run.smithi116.stdout:epoch 0 2024-04-25T00:00:06.021 INFO:teuthology.orchestra.run.smithi116.stdout:fsid 6d6a1aa7-0478-4c83-8057-05917e81f046 2024-04-25T00:00:06.021 INFO:teuthology.orchestra.run.smithi116.stdout:last_changed 2024-04-25T00:00:06.022698+0000 2024-04-25T00:00:06.021 INFO:teuthology.orchestra.run.smithi116.stdout:created 2024-04-25T00:00:06.022698+0000 2024-04-25T00:00:06.022 INFO:teuthology.orchestra.run.smithi116.stdout:min_mon_release 17 (quincy) 2024-04-25T00:00:06.022 INFO:teuthology.orchestra.run.smithi116.stdout:election_strategy: 1 2024-04-25T00:00:06.022 INFO:teuthology.orchestra.run.smithi116.stdout:0: [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] mon.a 2024-04-25T00:00:06.022 INFO:teuthology.orchestra.run.smithi116.stdout:1: [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon.b 2024-04-25T00:00:06.022 INFO:teuthology.orchestra.run.smithi116.stdout:2: [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] mon.c 2024-04-25T00:00:06.022 INFO:teuthology.orchestra.run.smithi116.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-04-25T00:00:06.023 DEBUG:teuthology.orchestra.run.smithi116:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-25T00:00:06.071 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 6d6a1aa7-0478-4c83-8057-05917e81f046... 2024-04-25T00:00:06.072 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-25T00:00:06.120 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-25T00:00:06.146 INFO:teuthology.orchestra.run.smithi116.stdout:[global] 2024-04-25T00:00:06.146 INFO:teuthology.orchestra.run.smithi116.stdout: chdir = "" 2024-04-25T00:00:06.146 INFO:teuthology.orchestra.run.smithi116.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-25T00:00:06.146 INFO:teuthology.orchestra.run.smithi116.stdout: auth supported = cephx 2024-04-25T00:00:06.146 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:00:06.150 INFO:teuthology.orchestra.run.smithi116.stdout: filestore xattr use omap = true 2024-04-25T00:00:06.150 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:00:06.150 INFO:teuthology.orchestra.run.smithi116.stdout: mon clock drift allowed = 1.000 2024-04-25T00:00:06.150 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:00:06.150 INFO:teuthology.orchestra.run.smithi116.stdout: osd crush chooseleaf type = 0 2024-04-25T00:00:06.151 INFO:teuthology.orchestra.run.smithi116.stdout: auth debug = true 2024-04-25T00:00:06.151 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:00:06.151 INFO:teuthology.orchestra.run.smithi116.stdout: ms die on old message = true 2024-04-25T00:00:06.151 INFO:teuthology.orchestra.run.smithi116.stdout: ms die on bug = true 2024-04-25T00:00:06.151 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:00:06.151 INFO:teuthology.orchestra.run.smithi116.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-25T00:00:06.151 INFO:teuthology.orchestra.run.smithi116.stdout: mon pg warn max object skew = 0 2024-04-25T00:00:06.151 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:00:06.151 INFO:teuthology.orchestra.run.smithi116.stdout: # disable pg_autoscaler by default for new pools 2024-04-25T00:00:06.151 INFO:teuthology.orchestra.run.smithi116.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-25T00:00:06.151 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:00:06.151 INFO:teuthology.orchestra.run.smithi116.stdout: osd pool default size = 2 2024-04-25T00:00:06.151 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:00:06.152 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd allow primary affinity = true 2024-04-25T00:00:06.152 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd allow pg remap = true 2024-04-25T00:00:06.152 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on legacy crush tunables = false 2024-04-25T00:00:06.152 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on crush straw calc version zero = false 2024-04-25T00:00:06.152 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on no sortbitwise = false 2024-04-25T00:00:06.152 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on osd down out interval zero = false 2024-04-25T00:00:06.152 INFO:teuthology.orchestra.run.smithi116.stdout: mon warn on too few osds = false 2024-04-25T00:00:06.152 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-25T00:00:06.152 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-25T00:00:06.152 INFO:teuthology.orchestra.run.smithi116.stdout: mon_allow_pool_size_one = true 2024-04-25T00:00:06.152 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:00:06.152 INFO:teuthology.orchestra.run.smithi116.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-25T00:00:06.152 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:00:06.152 INFO:teuthology.orchestra.run.smithi116.stdout: osd default data pool replay window = 5 2024-04-25T00:00:06.153 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:00:06.153 INFO:teuthology.orchestra.run.smithi116.stdout: mon allow pool delete = true 2024-04-25T00:00:06.153 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:00:06.153 INFO:teuthology.orchestra.run.smithi116.stdout: mon cluster log file level = debug 2024-04-25T00:00:06.153 INFO:teuthology.orchestra.run.smithi116.stdout: debug asserts on shutdown = true 2024-04-25T00:00:06.153 INFO:teuthology.orchestra.run.smithi116.stdout: mon health detail to clog = false 2024-04-25T00:00:06.153 INFO:teuthology.orchestra.run.smithi116.stdout: mon host = "172.21.15.116,172.21.15.174,[v2:172.21.15.174:3301,v1:172.21.15.174:6790]" 2024-04-25T00:00:06.153 INFO:teuthology.orchestra.run.smithi116.stdout: fsid = 6d6a1aa7-0478-4c83-8057-05917e81f046 2024-04-25T00:00:06.153 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:00:06.153 INFO:teuthology.orchestra.run.smithi116.stdout:[osd] 2024-04-25T00:00:06.153 INFO:teuthology.orchestra.run.smithi116.stdout: osd journal size = 100 2024-04-25T00:00:06.153 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:00:06.153 INFO:teuthology.orchestra.run.smithi116.stdout: osd scrub load threshold = 5.0 2024-04-25T00:00:06.154 INFO:teuthology.orchestra.run.smithi116.stdout: osd scrub max interval = 600 2024-04-25T00:00:06.154 INFO:teuthology.orchestra.run.smithi116.stdout: osd mclock profile = high_recovery_ops 2024-04-25T00:00:06.154 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:00:06.154 INFO:teuthology.orchestra.run.smithi116.stdout: osd recover clone overlap = true 2024-04-25T00:00:06.154 INFO:teuthology.orchestra.run.smithi116.stdout: osd recovery max chunk = 1048576 2024-04-25T00:00:06.154 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:00:06.154 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug shutdown = true 2024-04-25T00:00:06.154 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug op order = true 2024-04-25T00:00:06.154 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug verify stray on activate = true 2024-04-25T00:00:06.154 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug trim objects = true 2024-04-25T00:00:06.154 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:00:06.154 INFO:teuthology.orchestra.run.smithi116.stdout: osd open classes on start = true 2024-04-25T00:00:06.154 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug pg log writeout = true 2024-04-25T00:00:06.155 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:00:06.155 INFO:teuthology.orchestra.run.smithi116.stdout: osd deep scrub update digest min age = 30 2024-04-25T00:00:06.155 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:00:06.155 INFO:teuthology.orchestra.run.smithi116.stdout: osd map max advance = 10 2024-04-25T00:00:06.155 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:00:06.155 INFO:teuthology.orchestra.run.smithi116.stdout: journal zero on create = true 2024-04-25T00:00:06.155 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:00:06.155 INFO:teuthology.orchestra.run.smithi116.stdout: filestore ondisk finisher threads = 3 2024-04-25T00:00:06.155 INFO:teuthology.orchestra.run.smithi116.stdout: filestore apply finisher threads = 3 2024-04-25T00:00:06.155 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:00:06.155 INFO:teuthology.orchestra.run.smithi116.stdout: bdev debug aio = true 2024-04-25T00:00:06.155 INFO:teuthology.orchestra.run.smithi116.stdout: osd debug misdirected ops = true 2024-04-25T00:00:06.155 INFO:teuthology.orchestra.run.smithi116.stdout: bdev async discard = True 2024-04-25T00:00:06.156 INFO:teuthology.orchestra.run.smithi116.stdout: bdev enable discard = True 2024-04-25T00:00:06.156 INFO:teuthology.orchestra.run.smithi116.stdout: bluestore allocator = bitmap 2024-04-25T00:00:06.156 INFO:teuthology.orchestra.run.smithi116.stdout: bluestore block size = 96636764160 2024-04-25T00:00:06.156 INFO:teuthology.orchestra.run.smithi116.stdout: bluestore fsck on mount = True 2024-04-25T00:00:06.156 INFO:teuthology.orchestra.run.smithi116.stdout: debug bluefs = 1/20 2024-04-25T00:00:06.156 INFO:teuthology.orchestra.run.smithi116.stdout: debug bluestore = 1/20 2024-04-25T00:00:06.156 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-04-25T00:00:06.156 INFO:teuthology.orchestra.run.smithi116.stdout: debug osd = 20 2024-04-25T00:00:06.156 INFO:teuthology.orchestra.run.smithi116.stdout: debug rocksdb = 4/10 2024-04-25T00:00:06.156 INFO:teuthology.orchestra.run.smithi116.stdout: filestore flush min = 0 2024-04-25T00:00:06.156 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd backfillfull_ratio = 0.85 2024-04-25T00:00:06.156 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd full ratio = 0.9 2024-04-25T00:00:06.156 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd nearfull ratio = 0.8 2024-04-25T00:00:06.157 INFO:teuthology.orchestra.run.smithi116.stdout: osd blocked scrub grace period = 3600 2024-04-25T00:00:06.157 INFO:teuthology.orchestra.run.smithi116.stdout: osd failsafe full ratio = 0.95 2024-04-25T00:00:06.157 INFO:teuthology.orchestra.run.smithi116.stdout: osd max markdown count = 1000 2024-04-25T00:00:06.157 INFO:teuthology.orchestra.run.smithi116.stdout: osd objectstore = bluestore 2024-04-25T00:00:06.157 INFO:teuthology.orchestra.run.smithi116.stdout: osd op complaint time = 180 2024-04-25T00:00:06.157 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:00:06.157 INFO:teuthology.orchestra.run.smithi116.stdout:[mgr] 2024-04-25T00:00:06.157 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-04-25T00:00:06.157 INFO:teuthology.orchestra.run.smithi116.stdout: debug mgr = 20 2024-04-25T00:00:06.157 INFO:teuthology.orchestra.run.smithi116.stdout: debug mon = 20 2024-04-25T00:00:06.157 INFO:teuthology.orchestra.run.smithi116.stdout: debug auth = 20 2024-04-25T00:00:06.157 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min pgs per osd = 4 2024-04-25T00:00:06.157 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min bytes per osd = 10 2024-04-25T00:00:06.157 INFO:teuthology.orchestra.run.smithi116.stdout: mgr/telemetry/nag = false 2024-04-25T00:00:06.157 INFO:teuthology.orchestra.run.smithi116.stdout: debug client = 20 2024-04-25T00:00:06.158 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:00:06.158 INFO:teuthology.orchestra.run.smithi116.stdout:[mon] 2024-04-25T00:00:06.158 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-04-25T00:00:06.158 INFO:teuthology.orchestra.run.smithi116.stdout: debug mon = 20 2024-04-25T00:00:06.158 INFO:teuthology.orchestra.run.smithi116.stdout: debug paxos = 20 2024-04-25T00:00:06.158 INFO:teuthology.orchestra.run.smithi116.stdout: debug auth = 20 2024-04-25T00:00:06.158 INFO:teuthology.orchestra.run.smithi116.stdout: mon data avail warn = 5 2024-04-25T00:00:06.158 INFO:teuthology.orchestra.run.smithi116.stdout: mon mgr mkfs grace = 240 2024-04-25T00:00:06.158 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min pgs per osd = 4 2024-04-25T00:00:06.158 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd reporter subtree level = osd 2024-04-25T00:00:06.158 INFO:teuthology.orchestra.run.smithi116.stdout: mon osd prime pg temp = true 2024-04-25T00:00:06.158 INFO:teuthology.orchestra.run.smithi116.stdout: mon reweight min bytes per osd = 10 2024-04-25T00:00:06.158 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:00:06.159 INFO:teuthology.orchestra.run.smithi116.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-25T00:00:06.159 INFO:teuthology.orchestra.run.smithi116.stdout: auth mon ticket ttl = 660 # 11m 2024-04-25T00:00:06.159 INFO:teuthology.orchestra.run.smithi116.stdout: auth service ticket ttl = 240 # 4m 2024-04-25T00:00:06.159 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:00:06.159 INFO:teuthology.orchestra.run.smithi116.stdout: # don't complain about insecure global_id in the test suite 2024-04-25T00:00:06.159 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-25T00:00:06.159 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-25T00:00:06.159 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:00:06.159 INFO:teuthology.orchestra.run.smithi116.stdout: # 1m isn't quite enough 2024-04-25T00:00:06.159 INFO:teuthology.orchestra.run.smithi116.stdout: mon_down_mkfs_grace = 2m 2024-04-25T00:00:06.159 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:00:06.159 INFO:teuthology.orchestra.run.smithi116.stdout: mon_warn_on_filestore_osds = false 2024-04-25T00:00:06.159 INFO:teuthology.orchestra.run.smithi116.stdout: mon down mkfs grace = 300 2024-04-25T00:00:06.159 INFO:teuthology.orchestra.run.smithi116.stdout: mon op complaint time = 120 2024-04-25T00:00:06.159 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:00:06.160 INFO:teuthology.orchestra.run.smithi116.stdout:[client] 2024-04-25T00:00:06.160 INFO:teuthology.orchestra.run.smithi116.stdout: rgw cache enabled = true 2024-04-25T00:00:06.160 INFO:teuthology.orchestra.run.smithi116.stdout: rgw enable ops log = true 2024-04-25T00:00:06.160 INFO:teuthology.orchestra.run.smithi116.stdout: rgw enable usage log = true 2024-04-25T00:00:06.160 INFO:teuthology.orchestra.run.smithi116.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-25T00:00:06.160 INFO:teuthology.orchestra.run.smithi116.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-25T00:00:06.160 INFO:teuthology.orchestra.run.smithi116.stdout: client mount timeout = 600 2024-04-25T00:00:06.160 INFO:teuthology.orchestra.run.smithi116.stdout: client_shutdown_timeout = 120 2024-04-25T00:00:06.160 INFO:teuthology.orchestra.run.smithi116.stdout: debug client = 20 2024-04-25T00:00:06.160 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-04-25T00:00:06.160 INFO:teuthology.orchestra.run.smithi116.stdout: rados mon op timeout = 900 2024-04-25T00:00:06.160 INFO:teuthology.orchestra.run.smithi116.stdout: rados osd op timeout = 900 2024-04-25T00:00:06.160 INFO:teuthology.orchestra.run.smithi116.stdout:[mon.a] 2024-04-25T00:00:06.160 INFO:teuthology.orchestra.run.smithi116.stdout:[mon.b] 2024-04-25T00:00:06.161 INFO:teuthology.orchestra.run.smithi116.stdout:[mon.c] 2024-04-25T00:00:06.161 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.a] 2024-04-25T00:00:06.161 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.c] 2024-04-25T00:00:06.161 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.b] 2024-04-25T00:00:06.161 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.d] 2024-04-25T00:00:06.161 INFO:teuthology.orchestra.run.smithi116.stdout:[mds] 2024-04-25T00:00:06.161 INFO:teuthology.orchestra.run.smithi116.stdout: debug mds = 20 2024-04-25T00:00:06.161 INFO:teuthology.orchestra.run.smithi116.stdout: debug mds balancer = 20 2024-04-25T00:00:06.161 INFO:teuthology.orchestra.run.smithi116.stdout: debug ms = 1 2024-04-25T00:00:06.161 INFO:teuthology.orchestra.run.smithi116.stdout: mds bal fragment size max = 10000 2024-04-25T00:00:06.161 INFO:teuthology.orchestra.run.smithi116.stdout: mds bal merge size = 5 2024-04-25T00:00:06.162 INFO:teuthology.orchestra.run.smithi116.stdout: mds bal split bits = 3 2024-04-25T00:00:06.162 INFO:teuthology.orchestra.run.smithi116.stdout: mds bal split size = 100 2024-04-25T00:00:06.162 INFO:teuthology.orchestra.run.smithi116.stdout: mds debug frag = True 2024-04-25T00:00:06.162 INFO:teuthology.orchestra.run.smithi116.stdout: mds debug scatterstat = True 2024-04-25T00:00:06.162 INFO:teuthology.orchestra.run.smithi116.stdout: mds op complaint time = 180 2024-04-25T00:00:06.162 INFO:teuthology.orchestra.run.smithi116.stdout: mds verify scatter = True 2024-04-25T00:00:06.162 INFO:teuthology.orchestra.run.smithi116.stdout: osd op complaint time = 180 2024-04-25T00:00:06.162 INFO:teuthology.orchestra.run.smithi116.stdout: rados mon op timeout = 900 2024-04-25T00:00:06.162 INFO:teuthology.orchestra.run.smithi116.stdout: rados osd op timeout = 900 2024-04-25T00:00:06.164 INFO:teuthology.orchestra.run.smithi174.stdout:[global] 2024-04-25T00:00:06.164 INFO:teuthology.orchestra.run.smithi174.stdout: chdir = "" 2024-04-25T00:00:06.164 INFO:teuthology.orchestra.run.smithi174.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-25T00:00:06.164 INFO:teuthology.orchestra.run.smithi174.stdout: auth supported = cephx 2024-04-25T00:00:06.164 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-25T00:00:06.164 INFO:teuthology.orchestra.run.smithi174.stdout: filestore xattr use omap = true 2024-04-25T00:00:06.164 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-25T00:00:06.164 INFO:teuthology.orchestra.run.smithi174.stdout: mon clock drift allowed = 1.000 2024-04-25T00:00:06.164 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-25T00:00:06.164 INFO:teuthology.orchestra.run.smithi174.stdout: osd crush chooseleaf type = 0 2024-04-25T00:00:06.164 INFO:teuthology.orchestra.run.smithi174.stdout: auth debug = true 2024-04-25T00:00:06.164 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-25T00:00:06.165 INFO:teuthology.orchestra.run.smithi174.stdout: ms die on old message = true 2024-04-25T00:00:06.165 INFO:teuthology.orchestra.run.smithi174.stdout: ms die on bug = true 2024-04-25T00:00:06.166 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-25T00:00:06.166 INFO:teuthology.orchestra.run.smithi174.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-25T00:00:06.166 INFO:teuthology.orchestra.run.smithi174.stdout: mon pg warn max object skew = 0 2024-04-25T00:00:06.166 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-25T00:00:06.166 INFO:teuthology.orchestra.run.smithi174.stdout: # disable pg_autoscaler by default for new pools 2024-04-25T00:00:06.166 INFO:teuthology.orchestra.run.smithi174.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-25T00:00:06.166 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-25T00:00:06.166 INFO:teuthology.orchestra.run.smithi174.stdout: osd pool default size = 2 2024-04-25T00:00:06.167 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-25T00:00:06.167 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd allow primary affinity = true 2024-04-25T00:00:06.167 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd allow pg remap = true 2024-04-25T00:00:06.167 INFO:teuthology.orchestra.run.smithi174.stdout: mon warn on legacy crush tunables = false 2024-04-25T00:00:06.167 INFO:teuthology.orchestra.run.smithi174.stdout: mon warn on crush straw calc version zero = false 2024-04-25T00:00:06.167 INFO:teuthology.orchestra.run.smithi174.stdout: mon warn on no sortbitwise = false 2024-04-25T00:00:06.167 INFO:teuthology.orchestra.run.smithi174.stdout: mon warn on osd down out interval zero = false 2024-04-25T00:00:06.167 INFO:teuthology.orchestra.run.smithi174.stdout: mon warn on too few osds = false 2024-04-25T00:00:06.167 INFO:teuthology.orchestra.run.smithi174.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-25T00:00:06.167 INFO:teuthology.orchestra.run.smithi174.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-25T00:00:06.167 INFO:teuthology.orchestra.run.smithi174.stdout: mon_allow_pool_size_one = true 2024-04-25T00:00:06.167 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-25T00:00:06.167 INFO:teuthology.orchestra.run.smithi174.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-25T00:00:06.168 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-25T00:00:06.168 INFO:teuthology.orchestra.run.smithi174.stdout: osd default data pool replay window = 5 2024-04-25T00:00:06.168 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-25T00:00:06.168 INFO:teuthology.orchestra.run.smithi174.stdout: mon allow pool delete = true 2024-04-25T00:00:06.168 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-25T00:00:06.168 INFO:teuthology.orchestra.run.smithi174.stdout: mon cluster log file level = debug 2024-04-25T00:00:06.168 INFO:teuthology.orchestra.run.smithi174.stdout: debug asserts on shutdown = true 2024-04-25T00:00:06.168 INFO:teuthology.orchestra.run.smithi174.stdout: mon health detail to clog = false 2024-04-25T00:00:06.168 INFO:teuthology.orchestra.run.smithi174.stdout: mon host = "172.21.15.116,172.21.15.174,[v2:172.21.15.174:3301,v1:172.21.15.174:6790]" 2024-04-25T00:00:06.168 INFO:teuthology.orchestra.run.smithi174.stdout: fsid = 6d6a1aa7-0478-4c83-8057-05917e81f046 2024-04-25T00:00:06.168 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-25T00:00:06.168 INFO:teuthology.orchestra.run.smithi174.stdout:[osd] 2024-04-25T00:00:06.168 INFO:teuthology.orchestra.run.smithi174.stdout: osd journal size = 100 2024-04-25T00:00:06.168 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-25T00:00:06.169 INFO:teuthology.orchestra.run.smithi174.stdout: osd scrub load threshold = 5.0 2024-04-25T00:00:06.169 INFO:teuthology.orchestra.run.smithi174.stdout: osd scrub max interval = 600 2024-04-25T00:00:06.169 INFO:teuthology.orchestra.run.smithi174.stdout: osd mclock profile = high_recovery_ops 2024-04-25T00:00:06.169 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-25T00:00:06.169 INFO:teuthology.orchestra.run.smithi174.stdout: osd recover clone overlap = true 2024-04-25T00:00:06.169 INFO:teuthology.orchestra.run.smithi174.stdout: osd recovery max chunk = 1048576 2024-04-25T00:00:06.169 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-25T00:00:06.169 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug shutdown = true 2024-04-25T00:00:06.169 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug op order = true 2024-04-25T00:00:06.169 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug verify stray on activate = true 2024-04-25T00:00:06.169 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug trim objects = true 2024-04-25T00:00:06.169 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-25T00:00:06.169 INFO:teuthology.orchestra.run.smithi174.stdout: osd open classes on start = true 2024-04-25T00:00:06.169 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug pg log writeout = true 2024-04-25T00:00:06.169 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-25T00:00:06.170 INFO:teuthology.orchestra.run.smithi174.stdout: osd deep scrub update digest min age = 30 2024-04-25T00:00:06.170 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-25T00:00:06.170 INFO:teuthology.orchestra.run.smithi174.stdout: osd map max advance = 10 2024-04-25T00:00:06.170 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-25T00:00:06.170 INFO:teuthology.orchestra.run.smithi174.stdout: journal zero on create = true 2024-04-25T00:00:06.170 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-25T00:00:06.170 INFO:teuthology.orchestra.run.smithi174.stdout: filestore ondisk finisher threads = 3 2024-04-25T00:00:06.170 INFO:teuthology.orchestra.run.smithi174.stdout: filestore apply finisher threads = 3 2024-04-25T00:00:06.170 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-25T00:00:06.170 INFO:teuthology.orchestra.run.smithi174.stdout: bdev debug aio = true 2024-04-25T00:00:06.170 INFO:teuthology.orchestra.run.smithi174.stdout: osd debug misdirected ops = true 2024-04-25T00:00:06.170 INFO:teuthology.orchestra.run.smithi174.stdout: bdev async discard = True 2024-04-25T00:00:06.170 INFO:teuthology.orchestra.run.smithi174.stdout: bdev enable discard = True 2024-04-25T00:00:06.171 INFO:teuthology.orchestra.run.smithi174.stdout: bluestore allocator = bitmap 2024-04-25T00:00:06.171 INFO:teuthology.orchestra.run.smithi174.stdout: bluestore block size = 96636764160 2024-04-25T00:00:06.171 INFO:teuthology.orchestra.run.smithi174.stdout: bluestore fsck on mount = True 2024-04-25T00:00:06.171 INFO:teuthology.orchestra.run.smithi174.stdout: debug bluefs = 1/20 2024-04-25T00:00:06.171 INFO:teuthology.orchestra.run.smithi174.stdout: debug bluestore = 1/20 2024-04-25T00:00:06.173 INFO:teuthology.orchestra.run.smithi174.stdout: debug ms = 1 2024-04-25T00:00:06.173 INFO:teuthology.orchestra.run.smithi174.stdout: debug osd = 20 2024-04-25T00:00:06.173 INFO:teuthology.orchestra.run.smithi174.stdout: debug rocksdb = 4/10 2024-04-25T00:00:06.173 INFO:teuthology.orchestra.run.smithi174.stdout: filestore flush min = 0 2024-04-25T00:00:06.174 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd backfillfull_ratio = 0.85 2024-04-25T00:00:06.174 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd full ratio = 0.9 2024-04-25T00:00:06.174 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd nearfull ratio = 0.8 2024-04-25T00:00:06.174 INFO:teuthology.orchestra.run.smithi174.stdout: osd blocked scrub grace period = 3600 2024-04-25T00:00:06.174 INFO:teuthology.orchestra.run.smithi174.stdout: osd failsafe full ratio = 0.95 2024-04-25T00:00:06.175 INFO:teuthology.orchestra.run.smithi174.stdout: osd max markdown count = 1000 2024-04-25T00:00:06.176 INFO:teuthology.orchestra.run.smithi174.stdout: osd objectstore = bluestore 2024-04-25T00:00:06.176 INFO:teuthology.orchestra.run.smithi174.stdout: osd op complaint time = 180 2024-04-25T00:00:06.176 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-25T00:00:06.176 INFO:teuthology.orchestra.run.smithi174.stdout:[mgr] 2024-04-25T00:00:06.176 INFO:teuthology.orchestra.run.smithi174.stdout: debug ms = 1 2024-04-25T00:00:06.176 INFO:teuthology.orchestra.run.smithi174.stdout: debug mgr = 20 2024-04-25T00:00:06.176 INFO:teuthology.orchestra.run.smithi174.stdout: debug mon = 20 2024-04-25T00:00:06.176 INFO:teuthology.orchestra.run.smithi174.stdout: debug auth = 20 2024-04-25T00:00:06.176 INFO:teuthology.orchestra.run.smithi174.stdout: mon reweight min pgs per osd = 4 2024-04-25T00:00:06.176 INFO:teuthology.orchestra.run.smithi174.stdout: mon reweight min bytes per osd = 10 2024-04-25T00:00:06.177 INFO:teuthology.orchestra.run.smithi174.stdout: mgr/telemetry/nag = false 2024-04-25T00:00:06.177 INFO:teuthology.orchestra.run.smithi174.stdout: debug client = 20 2024-04-25T00:00:06.177 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-25T00:00:06.177 INFO:teuthology.orchestra.run.smithi174.stdout:[mon] 2024-04-25T00:00:06.177 INFO:teuthology.orchestra.run.smithi174.stdout: debug ms = 1 2024-04-25T00:00:06.177 INFO:teuthology.orchestra.run.smithi174.stdout: debug mon = 20 2024-04-25T00:00:06.177 INFO:teuthology.orchestra.run.smithi174.stdout: debug paxos = 20 2024-04-25T00:00:06.177 INFO:teuthology.orchestra.run.smithi174.stdout: debug auth = 20 2024-04-25T00:00:06.177 INFO:teuthology.orchestra.run.smithi174.stdout: mon data avail warn = 5 2024-04-25T00:00:06.177 INFO:teuthology.orchestra.run.smithi174.stdout: mon mgr mkfs grace = 240 2024-04-25T00:00:06.177 INFO:teuthology.orchestra.run.smithi174.stdout: mon reweight min pgs per osd = 4 2024-04-25T00:00:06.177 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd reporter subtree level = osd 2024-04-25T00:00:06.177 INFO:teuthology.orchestra.run.smithi174.stdout: mon osd prime pg temp = true 2024-04-25T00:00:06.177 INFO:teuthology.orchestra.run.smithi174.stdout: mon reweight min bytes per osd = 10 2024-04-25T00:00:06.178 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-25T00:00:06.178 INFO:teuthology.orchestra.run.smithi174.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-25T00:00:06.178 INFO:teuthology.orchestra.run.smithi174.stdout: auth mon ticket ttl = 660 # 11m 2024-04-25T00:00:06.178 INFO:teuthology.orchestra.run.smithi174.stdout: auth service ticket ttl = 240 # 4m 2024-04-25T00:00:06.178 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-25T00:00:06.178 INFO:teuthology.orchestra.run.smithi174.stdout: # don't complain about insecure global_id in the test suite 2024-04-25T00:00:06.178 INFO:teuthology.orchestra.run.smithi174.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-25T00:00:06.178 INFO:teuthology.orchestra.run.smithi174.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-25T00:00:06.178 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-25T00:00:06.178 INFO:teuthology.orchestra.run.smithi174.stdout: # 1m isn't quite enough 2024-04-25T00:00:06.178 INFO:teuthology.orchestra.run.smithi174.stdout: mon_down_mkfs_grace = 2m 2024-04-25T00:00:06.178 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-25T00:00:06.178 INFO:teuthology.orchestra.run.smithi174.stdout: mon_warn_on_filestore_osds = false 2024-04-25T00:00:06.178 INFO:teuthology.orchestra.run.smithi174.stdout: mon down mkfs grace = 300 2024-04-25T00:00:06.179 INFO:teuthology.orchestra.run.smithi174.stdout: mon op complaint time = 120 2024-04-25T00:00:06.179 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-25T00:00:06.179 INFO:teuthology.orchestra.run.smithi174.stdout:[client] 2024-04-25T00:00:06.179 INFO:teuthology.orchestra.run.smithi174.stdout: rgw cache enabled = true 2024-04-25T00:00:06.179 INFO:teuthology.orchestra.run.smithi174.stdout: rgw enable ops log = true 2024-04-25T00:00:06.179 INFO:teuthology.orchestra.run.smithi174.stdout: rgw enable usage log = true 2024-04-25T00:00:06.179 INFO:teuthology.orchestra.run.smithi174.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-25T00:00:06.179 INFO:teuthology.orchestra.run.smithi174.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-25T00:00:06.179 INFO:teuthology.orchestra.run.smithi174.stdout: client mount timeout = 600 2024-04-25T00:00:06.179 INFO:teuthology.orchestra.run.smithi174.stdout: client_shutdown_timeout = 120 2024-04-25T00:00:06.179 INFO:teuthology.orchestra.run.smithi174.stdout: debug client = 20 2024-04-25T00:00:06.179 INFO:teuthology.orchestra.run.smithi174.stdout: debug ms = 1 2024-04-25T00:00:06.179 INFO:teuthology.orchestra.run.smithi174.stdout: rados mon op timeout = 900 2024-04-25T00:00:06.180 INFO:teuthology.orchestra.run.smithi174.stdout: rados osd op timeout = 900 2024-04-25T00:00:06.180 INFO:teuthology.orchestra.run.smithi174.stdout:[mon.a] 2024-04-25T00:00:06.180 INFO:teuthology.orchestra.run.smithi174.stdout:[mon.b] 2024-04-25T00:00:06.180 INFO:teuthology.orchestra.run.smithi174.stdout:[mon.c] 2024-04-25T00:00:06.180 INFO:teuthology.orchestra.run.smithi174.stdout:[mds.a] 2024-04-25T00:00:06.180 INFO:teuthology.orchestra.run.smithi174.stdout:[mds.c] 2024-04-25T00:00:06.180 INFO:teuthology.orchestra.run.smithi174.stdout:[mds.b] 2024-04-25T00:00:06.180 INFO:teuthology.orchestra.run.smithi174.stdout:[mds.d] 2024-04-25T00:00:06.180 INFO:teuthology.orchestra.run.smithi174.stdout:[mds] 2024-04-25T00:00:06.180 INFO:teuthology.orchestra.run.smithi174.stdout: debug mds = 20 2024-04-25T00:00:06.180 INFO:teuthology.orchestra.run.smithi174.stdout: debug mds balancer = 20 2024-04-25T00:00:06.180 INFO:teuthology.orchestra.run.smithi174.stdout: debug ms = 1 2024-04-25T00:00:06.180 INFO:teuthology.orchestra.run.smithi174.stdout: mds bal fragment size max = 10000 2024-04-25T00:00:06.180 INFO:teuthology.orchestra.run.smithi174.stdout: mds bal merge size = 5 2024-04-25T00:00:06.181 INFO:teuthology.orchestra.run.smithi174.stdout: mds bal split bits = 3 2024-04-25T00:00:06.181 INFO:teuthology.orchestra.run.smithi174.stdout: mds bal split size = 100 2024-04-25T00:00:06.181 INFO:teuthology.orchestra.run.smithi174.stdout: mds debug frag = True 2024-04-25T00:00:06.181 INFO:teuthology.orchestra.run.smithi174.stdout: mds debug scatterstat = True 2024-04-25T00:00:06.181 INFO:teuthology.orchestra.run.smithi174.stdout: mds op complaint time = 180 2024-04-25T00:00:06.181 INFO:teuthology.orchestra.run.smithi174.stdout: mds verify scatter = True 2024-04-25T00:00:06.181 INFO:teuthology.orchestra.run.smithi174.stdout: osd op complaint time = 180 2024-04-25T00:00:06.181 INFO:teuthology.orchestra.run.smithi174.stdout: rados mon op timeout = 900 2024-04-25T00:00:06.181 INFO:teuthology.orchestra.run.smithi174.stdout: rados osd op timeout = 900 2024-04-25T00:00:06.181 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-25T00:00:06.182 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-25T00:00:06.251 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-25T00:00:06.252 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-04-25T00:00:06.252 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-25T00:00:06.302 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-04-25T00:00:06.302 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-25T00:00:06.350 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi116.front.sepia.ceph.com 2024-04-25T00:00:06.350 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-04-25T00:00:06.350 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-25T00:00:06.351 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-25T00:00:06.416 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-04-25T00:00:06.416 DEBUG:teuthology.orchestra.run.smithi116:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-25T00:00:06.463 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi174.front.sepia.ceph.com 2024-04-25T00:00:06.463 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-04-25T00:00:06.463 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-25T00:00:06.463 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-25T00:00:06.490 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-04-25T00:00:06.490 DEBUG:teuthology.orchestra.run.smithi174:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-25T00:00:06.537 INFO:tasks.ceph:Setting up mon nodes... 2024-04-25T00:00:06.537 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-25T00:00:06.537 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-04-25T00:00:06.582 INFO:teuthology.orchestra.run.smithi116.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-04-25T00:00:06.585 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-25T00:00:06.628 INFO:teuthology.orchestra.run.smithi174.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-25T00:00:06.642 INFO:tasks.ceph:Setting up mds nodes... 2024-04-25T00:00:06.642 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-04-25T00:00:06.686 INFO:teuthology.orchestra.run.smithi116.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-04-25T00:00:06.689 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-04-25T00:00:06.702 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-04-25T00:00:06.786 INFO:teuthology.orchestra.run.smithi116.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-04-25T00:00:06.788 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-04-25T00:00:06.841 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-04-25T00:00:06.885 INFO:teuthology.orchestra.run.smithi174.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-04-25T00:00:06.888 DEBUG:teuthology.orchestra.run.smithi174:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-04-25T00:00:06.944 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-04-25T00:00:07.028 INFO:teuthology.orchestra.run.smithi174.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-04-25T00:00:07.031 DEBUG:teuthology.orchestra.run.smithi174:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-04-25T00:00:07.084 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-25T00:00:07.084 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-25T00:00:07.120 INFO:teuthology.orchestra.run.smithi116.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-25T00:00:07.130 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-25T00:00:07.163 INFO:teuthology.orchestra.run.smithi174.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-04-25T00:00:07.172 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-25T00:00:07.172 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi116.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi174.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-25T00:00:07.173 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-25T00:00:07.184 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-25T00:00:07.185 INFO:tasks.ceph:role: osd.0 2024-04-25T00:00:07.185 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi116.front.sepia.ceph.com 2024-04-25T00:00:07.185 DEBUG:teuthology.orchestra.run.smithi116:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-25T00:00:07.245 INFO:teuthology.orchestra.run.smithi116.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T00:00:07.246 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T00:00:07.246 INFO:teuthology.orchestra.run.smithi116.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T00:00:07.246 INFO:teuthology.orchestra.run.smithi116.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-25T00:00:07.246 INFO:teuthology.orchestra.run.smithi116.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T00:00:07.246 INFO:teuthology.orchestra.run.smithi116.stdout: = sunit=0 swidth=0 blks 2024-04-25T00:00:07.246 INFO:teuthology.orchestra.run.smithi116.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T00:00:07.246 INFO:teuthology.orchestra.run.smithi116.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-25T00:00:07.246 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T00:00:07.246 INFO:teuthology.orchestra.run.smithi116.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T00:00:07.722 INFO:teuthology.orchestra.run.smithi116.stdout:Discarding blocks...Done. 2024-04-25T00:00:07.724 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi116.front.sepia.ceph.com -o noatime 2024-04-25T00:00:07.724 DEBUG:teuthology.orchestra.run.smithi116:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-25T00:00:07.866 DEBUG:teuthology.orchestra.run.smithi116:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-25T00:00:07.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T00:00:07.877 INFO:teuthology.orchestra.run.smithi116.stderr:sudo: /sbin/restorecon: command not found 2024-04-25T00:00:07.878 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-25T00:00:07.933 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-25T00:00:07.934 INFO:tasks.ceph:role: osd.1 2024-04-25T00:00:07.934 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi116.front.sepia.ceph.com 2024-04-25T00:00:07.934 DEBUG:teuthology.orchestra.run.smithi116:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-25T00:00:07.997 INFO:teuthology.orchestra.run.smithi116.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T00:00:07.997 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T00:00:07.997 INFO:teuthology.orchestra.run.smithi116.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T00:00:07.998 INFO:teuthology.orchestra.run.smithi116.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-25T00:00:07.998 INFO:teuthology.orchestra.run.smithi116.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T00:00:07.998 INFO:teuthology.orchestra.run.smithi116.stdout: = sunit=0 swidth=0 blks 2024-04-25T00:00:07.998 INFO:teuthology.orchestra.run.smithi116.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T00:00:07.998 INFO:teuthology.orchestra.run.smithi116.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-25T00:00:07.998 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T00:00:07.998 INFO:teuthology.orchestra.run.smithi116.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T00:00:08.427 INFO:teuthology.orchestra.run.smithi116.stdout:Discarding blocks...Done. 2024-04-25T00:00:08.430 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi116.front.sepia.ceph.com -o noatime 2024-04-25T00:00:08.430 DEBUG:teuthology.orchestra.run.smithi116:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-25T00:00:08.498 DEBUG:teuthology.orchestra.run.smithi116:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-25T00:00:08.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T00:00:08.551 INFO:teuthology.orchestra.run.smithi116.stderr:sudo: /sbin/restorecon: command not found 2024-04-25T00:00:08.552 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-25T00:00:08.606 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-25T00:00:08.606 INFO:tasks.ceph:role: osd.2 2024-04-25T00:00:08.606 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi116.front.sepia.ceph.com 2024-04-25T00:00:08.606 DEBUG:teuthology.orchestra.run.smithi116:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-25T00:00:08.669 INFO:teuthology.orchestra.run.smithi116.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T00:00:08.670 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T00:00:08.670 INFO:teuthology.orchestra.run.smithi116.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T00:00:08.670 INFO:teuthology.orchestra.run.smithi116.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-25T00:00:08.671 INFO:teuthology.orchestra.run.smithi116.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T00:00:08.671 INFO:teuthology.orchestra.run.smithi116.stdout: = sunit=0 swidth=0 blks 2024-04-25T00:00:08.671 INFO:teuthology.orchestra.run.smithi116.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T00:00:08.671 INFO:teuthology.orchestra.run.smithi116.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-25T00:00:08.671 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T00:00:08.671 INFO:teuthology.orchestra.run.smithi116.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T00:00:09.125 INFO:teuthology.orchestra.run.smithi116.stdout:Discarding blocks...Done. 2024-04-25T00:00:09.127 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi116.front.sepia.ceph.com -o noatime 2024-04-25T00:00:09.127 DEBUG:teuthology.orchestra.run.smithi116:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-25T00:00:09.218 DEBUG:teuthology.orchestra.run.smithi116:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-25T00:00:09.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T00:00:09.228 INFO:teuthology.orchestra.run.smithi116.stderr:sudo: /sbin/restorecon: command not found 2024-04-25T00:00:09.228 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-25T00:00:09.281 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-25T00:00:09.281 INFO:tasks.ceph:role: osd.3 2024-04-25T00:00:09.281 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi116.front.sepia.ceph.com 2024-04-25T00:00:09.281 DEBUG:teuthology.orchestra.run.smithi116:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-25T00:00:09.343 INFO:teuthology.orchestra.run.smithi116.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T00:00:09.343 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T00:00:09.343 INFO:teuthology.orchestra.run.smithi116.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T00:00:09.343 INFO:teuthology.orchestra.run.smithi116.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-25T00:00:09.344 INFO:teuthology.orchestra.run.smithi116.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T00:00:09.344 INFO:teuthology.orchestra.run.smithi116.stdout: = sunit=0 swidth=0 blks 2024-04-25T00:00:09.344 INFO:teuthology.orchestra.run.smithi116.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T00:00:09.344 INFO:teuthology.orchestra.run.smithi116.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-25T00:00:09.344 INFO:teuthology.orchestra.run.smithi116.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T00:00:09.344 INFO:teuthology.orchestra.run.smithi116.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T00:00:09.807 INFO:teuthology.orchestra.run.smithi116.stdout:Discarding blocks...Done. 2024-04-25T00:00:09.813 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi116.front.sepia.ceph.com -o noatime 2024-04-25T00:00:09.813 DEBUG:teuthology.orchestra.run.smithi116:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-04-25T00:00:09.957 DEBUG:teuthology.orchestra.run.smithi116:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-25T00:00:09.967 INFO:teuthology.orchestra.run.smithi116.stderr:sudo: /sbin/restorecon: command not found 2024-04-25T00:00:09.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T00:00:09.968 DEBUG:teuthology.orchestra.run.smithi116:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T00:00:10.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:10.079+0000 7f27bc1717c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-04-25T00:00:10.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:10.079+0000 7f27bc1717c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-04-25T00:00:10.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:10.079+0000 7f27bc1717c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-25T00:00:10.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:10.079+0000 7f27bc1717c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-25T00:00:10.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:10.079+0000 7f27bc1717c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-25T00:00:10.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:10.087+0000 7f27bc1717c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-04-25T00:00:13.065 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-25T00:00:13.079 DEBUG:teuthology.orchestra.run.smithi116:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T00:00:13.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:13.159+0000 7fdfce66c7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-04-25T00:00:13.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:13.159+0000 7fdfce66c7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-04-25T00:00:13.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:13.159+0000 7fdfce66c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-25T00:00:13.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:13.159+0000 7fdfce66c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-25T00:00:13.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:13.159+0000 7fdfce66c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-25T00:00:13.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:13.159+0000 7fdfce66c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-04-25T00:00:16.146 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-25T00:00:16.204 DEBUG:teuthology.orchestra.run.smithi116:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T00:00:16.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:16.283+0000 7ff8dd64e7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-04-25T00:00:16.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:16.283+0000 7ff8dd64e7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-04-25T00:00:16.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:16.283+0000 7ff8dd64e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-25T00:00:16.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:16.283+0000 7ff8dd64e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-25T00:00:16.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:16.283+0000 7ff8dd64e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-25T00:00:16.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:16.283+0000 7ff8dd64e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-04-25T00:00:19.290 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-25T00:00:19.348 DEBUG:teuthology.orchestra.run.smithi116:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T00:00:19.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:19.427+0000 7f8205e317c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-04-25T00:00:19.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:19.427+0000 7f8205e317c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-04-25T00:00:19.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:19.427+0000 7f8205e317c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-25T00:00:19.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:19.427+0000 7f8205e317c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-25T00:00:19.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:19.427+0000 7f8205e317c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-25T00:00:19.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:19.427+0000 7f8205e317c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-04-25T00:00:22.410 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-25T00:00:22.467 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-04-25T00:00:22.482 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-25T00:00:22.482 INFO:tasks.ceph:role: osd.4 2024-04-25T00:00:22.482 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi174.front.sepia.ceph.com 2024-04-25T00:00:22.482 DEBUG:teuthology.orchestra.run.smithi174:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-25T00:00:22.545 INFO:teuthology.orchestra.run.smithi174.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T00:00:22.545 INFO:teuthology.orchestra.run.smithi174.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T00:00:22.545 INFO:teuthology.orchestra.run.smithi174.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T00:00:22.545 INFO:teuthology.orchestra.run.smithi174.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-25T00:00:22.545 INFO:teuthology.orchestra.run.smithi174.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T00:00:22.545 INFO:teuthology.orchestra.run.smithi174.stdout: = sunit=0 swidth=0 blks 2024-04-25T00:00:22.545 INFO:teuthology.orchestra.run.smithi174.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T00:00:22.545 INFO:teuthology.orchestra.run.smithi174.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-25T00:00:22.545 INFO:teuthology.orchestra.run.smithi174.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T00:00:22.546 INFO:teuthology.orchestra.run.smithi174.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T00:00:22.999 INFO:teuthology.orchestra.run.smithi174.stdout:Discarding blocks...Done. 2024-04-25T00:00:23.001 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi174.front.sepia.ceph.com -o noatime 2024-04-25T00:00:23.001 DEBUG:teuthology.orchestra.run.smithi174:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-04-25T00:00:23.095 DEBUG:teuthology.orchestra.run.smithi174:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-04-25T00:00:23.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T00:00:23.146 INFO:teuthology.orchestra.run.smithi174.stderr:sudo: /sbin/restorecon: command not found 2024-04-25T00:00:23.147 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-04-25T00:00:23.199 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-25T00:00:23.199 INFO:tasks.ceph:role: osd.5 2024-04-25T00:00:23.199 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi174.front.sepia.ceph.com 2024-04-25T00:00:23.200 DEBUG:teuthology.orchestra.run.smithi174:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-25T00:00:23.259 INFO:teuthology.orchestra.run.smithi174.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T00:00:23.260 INFO:teuthology.orchestra.run.smithi174.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T00:00:23.260 INFO:teuthology.orchestra.run.smithi174.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T00:00:23.260 INFO:teuthology.orchestra.run.smithi174.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-25T00:00:23.260 INFO:teuthology.orchestra.run.smithi174.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T00:00:23.260 INFO:teuthology.orchestra.run.smithi174.stdout: = sunit=0 swidth=0 blks 2024-04-25T00:00:23.260 INFO:teuthology.orchestra.run.smithi174.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T00:00:23.260 INFO:teuthology.orchestra.run.smithi174.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-25T00:00:23.260 INFO:teuthology.orchestra.run.smithi174.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T00:00:23.260 INFO:teuthology.orchestra.run.smithi174.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T00:00:23.737 INFO:teuthology.orchestra.run.smithi174.stdout:Discarding blocks...Done. 2024-04-25T00:00:23.739 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi174.front.sepia.ceph.com -o noatime 2024-04-25T00:00:23.739 DEBUG:teuthology.orchestra.run.smithi174:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-04-25T00:00:23.875 DEBUG:teuthology.orchestra.run.smithi174:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-04-25T00:00:23.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T00:00:23.889 INFO:teuthology.orchestra.run.smithi174.stderr:sudo: /sbin/restorecon: command not found 2024-04-25T00:00:23.889 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-04-25T00:00:23.945 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-25T00:00:23.945 INFO:tasks.ceph:role: osd.6 2024-04-25T00:00:23.945 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi174.front.sepia.ceph.com 2024-04-25T00:00:23.945 DEBUG:teuthology.orchestra.run.smithi174:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-25T00:00:24.008 INFO:teuthology.orchestra.run.smithi174.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T00:00:24.008 INFO:teuthology.orchestra.run.smithi174.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T00:00:24.009 INFO:teuthology.orchestra.run.smithi174.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T00:00:24.009 INFO:teuthology.orchestra.run.smithi174.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-25T00:00:24.009 INFO:teuthology.orchestra.run.smithi174.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T00:00:24.009 INFO:teuthology.orchestra.run.smithi174.stdout: = sunit=0 swidth=0 blks 2024-04-25T00:00:24.009 INFO:teuthology.orchestra.run.smithi174.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T00:00:24.009 INFO:teuthology.orchestra.run.smithi174.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-25T00:00:24.009 INFO:teuthology.orchestra.run.smithi174.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T00:00:24.009 INFO:teuthology.orchestra.run.smithi174.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T00:00:24.476 INFO:teuthology.orchestra.run.smithi174.stdout:Discarding blocks...Done. 2024-04-25T00:00:24.477 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi174.front.sepia.ceph.com -o noatime 2024-04-25T00:00:24.477 DEBUG:teuthology.orchestra.run.smithi174:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-04-25T00:00:24.634 DEBUG:teuthology.orchestra.run.smithi174:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-04-25T00:00:24.645 INFO:teuthology.orchestra.run.smithi174.stderr:sudo: /sbin/restorecon: command not found 2024-04-25T00:00:24.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T00:00:24.647 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-04-25T00:00:24.700 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-25T00:00:24.701 INFO:tasks.ceph:role: osd.7 2024-04-25T00:00:24.701 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi174.front.sepia.ceph.com 2024-04-25T00:00:24.701 DEBUG:teuthology.orchestra.run.smithi174:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-25T00:00:24.765 INFO:teuthology.orchestra.run.smithi174.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T00:00:24.765 INFO:teuthology.orchestra.run.smithi174.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T00:00:24.765 INFO:teuthology.orchestra.run.smithi174.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T00:00:24.765 INFO:teuthology.orchestra.run.smithi174.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-04-25T00:00:24.765 INFO:teuthology.orchestra.run.smithi174.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T00:00:24.765 INFO:teuthology.orchestra.run.smithi174.stdout: = sunit=0 swidth=0 blks 2024-04-25T00:00:24.765 INFO:teuthology.orchestra.run.smithi174.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T00:00:24.765 INFO:teuthology.orchestra.run.smithi174.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-04-25T00:00:24.766 INFO:teuthology.orchestra.run.smithi174.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T00:00:24.766 INFO:teuthology.orchestra.run.smithi174.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T00:00:25.216 INFO:teuthology.orchestra.run.smithi174.stdout:Discarding blocks...Done. 2024-04-25T00:00:25.216 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi174.front.sepia.ceph.com -o noatime 2024-04-25T00:00:25.216 DEBUG:teuthology.orchestra.run.smithi174:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-04-25T00:00:25.312 DEBUG:teuthology.orchestra.run.smithi174:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-04-25T00:00:25.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T00:00:25.323 INFO:teuthology.orchestra.run.smithi174.stderr:sudo: /sbin/restorecon: command not found 2024-04-25T00:00:25.323 DEBUG:teuthology.orchestra.run.smithi174:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T00:00:25.437 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:25.431+0000 7fd53f7b97c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-04-25T00:00:25.437 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:25.435+0000 7fd53f7b97c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-04-25T00:00:25.437 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:25.435+0000 7fd53f7b97c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-25T00:00:25.437 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:25.435+0000 7fd53f7b97c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-25T00:00:25.437 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:25.435+0000 7fd53f7b97c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-25T00:00:25.438 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:25.435+0000 7fd53f7b97c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-04-25T00:00:28.408 DEBUG:teuthology.orchestra.run.smithi174:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-25T00:00:28.422 DEBUG:teuthology.orchestra.run.smithi174:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T00:00:28.507 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:28.503+0000 7f5d7a89c7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-04-25T00:00:28.508 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:28.503+0000 7f5d7a89c7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-04-25T00:00:28.508 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:28.503+0000 7f5d7a89c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-25T00:00:28.508 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:28.503+0000 7f5d7a89c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-25T00:00:28.508 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:28.503+0000 7f5d7a89c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-25T00:00:28.508 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:28.507+0000 7f5d7a89c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-04-25T00:00:31.485 DEBUG:teuthology.orchestra.run.smithi174:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-25T00:00:31.541 DEBUG:teuthology.orchestra.run.smithi174:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T00:00:31.623 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:31.619+0000 7f5ba2deb7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-04-25T00:00:31.624 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:31.619+0000 7f5ba2deb7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-04-25T00:00:31.624 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:31.619+0000 7f5ba2deb7c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-25T00:00:31.624 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:31.619+0000 7f5ba2deb7c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-25T00:00:31.624 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:31.619+0000 7f5ba2deb7c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-25T00:00:31.624 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:31.623+0000 7f5ba2deb7c0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-04-25T00:00:34.349 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:00:34.352 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:00:34.608 DEBUG:teuthology.orchestra.run.smithi174:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-25T00:00:34.622 DEBUG:teuthology.orchestra.run.smithi174:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T00:00:34.701 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:34.699+0000 7ffb6dec57c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-04-25T00:00:34.702 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:34.699+0000 7ffb6dec57c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-04-25T00:00:34.702 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:34.699+0000 7ffb6dec57c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-25T00:00:34.702 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:34.699+0000 7ffb6dec57c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-25T00:00:34.702 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:34.699+0000 7ffb6dec57c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-25T00:00:34.702 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:34.699+0000 7ffb6dec57c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-04-25T00:00:37.685 DEBUG:teuthology.orchestra.run.smithi174:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-25T00:00:37.741 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-25T00:00:37.741 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-04-25T00:00:37.741 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-04-25T00:00:37.755 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-04-25T00:00:37.755 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-04-25T00:00:37.810 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-04-25T00:00:37.811 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-04-25T00:00:37.866 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-04-25T00:00:37.866 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-25T00:00:37.921 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-04-25T00:00:37.921 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-25T00:00:37.973 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-04-25T00:00:37.973 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-25T00:00:38.025 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-04-25T00:00:38.025 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-25T00:00:38.077 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-04-25T00:00:38.077 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-25T00:00:38.091 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-04-25T00:00:38.091 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-04-25T00:00:38.144 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-04-25T00:00:38.144 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-04-25T00:00:38.200 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-04-25T00:00:38.200 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-04-25T00:00:38.255 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-04-25T00:00:38.255 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-04-25T00:00:38.308 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-04-25T00:00:38.308 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-04-25T00:00:38.364 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-04-25T00:00:38.365 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-04-25T00:00:38.420 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-04-25T00:00:38.420 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-25T00:00:38.426 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-04-25T00:00:38.426 DEBUG:teuthology.orchestra.run.smithi174:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-04-25T00:00:38.469 INFO:tasks.ceph:Adding keys to all mons... 2024-04-25T00:00:38.469 DEBUG:teuthology.orchestra.run.smithi116:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-25T00:00:38.471 DEBUG:teuthology.orchestra.run.smithi174:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-25T00:00:38.515 INFO:teuthology.orchestra.run.smithi116.stdout:[mgr.y] 2024-04-25T00:00:38.515 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQAGnSlmWCnMIhAArxJRyznF6+96SRdUIQ7sUQ== 2024-04-25T00:00:38.515 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.a] 2024-04-25T00:00:38.515 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQAGnSlmbToMKRAAvN/+75wrWpMh7krRZTVN5A== 2024-04-25T00:00:38.515 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.c] 2024-04-25T00:00:38.515 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQAGnSlmmUfnLhAAsyp002Uq1lGrocU5XHQGhA== 2024-04-25T00:00:38.515 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.0] 2024-04-25T00:00:38.516 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQAKnSlmXP/JBBAAk884h7I/PKt9PeYVEIqwRA== 2024-04-25T00:00:38.516 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.1] 2024-04-25T00:00:38.516 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQANnSlmD+aTCRAAcwjV201nyPK0xxa3v20iRA== 2024-04-25T00:00:38.516 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.2] 2024-04-25T00:00:38.516 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQAQnSlmQqH/EBAA2NoBf5Q557hNHrlBGnSnkA== 2024-04-25T00:00:38.516 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.3] 2024-04-25T00:00:38.516 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQATnSlmBEGWGRAA4SGyRmG2xgKSTl0CUS2G1A== 2024-04-25T00:00:38.516 INFO:teuthology.orchestra.run.smithi116.stdout:[mgr.x] 2024-04-25T00:00:38.516 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQAGnSlmYymMJRAAohPdFNROqDUrQ/GJfIwPWg== 2024-04-25T00:00:38.516 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.b] 2024-04-25T00:00:38.516 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQAGnSlmBPHpNBAA1qrZWXyLzxNP6GI5013GEA== 2024-04-25T00:00:38.516 INFO:teuthology.orchestra.run.smithi116.stdout:[mds.d] 2024-04-25T00:00:38.516 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQAHnSlm8hjLARAAwFsaX5Y/XeMTl37nLssNag== 2024-04-25T00:00:38.517 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.4] 2024-04-25T00:00:38.517 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQAZnSlmYv4UGhAABSMFHU5YnN6W2wJ/coR54w== 2024-04-25T00:00:38.517 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.5] 2024-04-25T00:00:38.517 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQAcnSlmklNWHhAAh9YoBc2VccvB8nYrzMcNmw== 2024-04-25T00:00:38.517 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.6] 2024-04-25T00:00:38.517 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQAfnSlmXmlBJRAAqTWRzjlyciHJhOde6i3o4A== 2024-04-25T00:00:38.517 INFO:teuthology.orchestra.run.smithi116.stdout:[osd.7] 2024-04-25T00:00:38.517 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQAinSlm9GPmKRAAqSo6fs2e00+ENMGqcRf8SQ== 2024-04-25T00:00:38.517 INFO:teuthology.orchestra.run.smithi116.stdout:[client.0] 2024-04-25T00:00:38.517 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQAHnSlmx0tEBxAAGnwgy/o20FiLzXpo6sLVRg== 2024-04-25T00:00:38.517 INFO:teuthology.orchestra.run.smithi116.stdout:[client.1] 2024-04-25T00:00:38.517 INFO:teuthology.orchestra.run.smithi116.stdout: key = AQAHnSlmzrXWCRAAcQxXrxYwlY+DSUs62WjbGw== 2024-04-25T00:00:38.522 INFO:teuthology.orchestra.run.smithi174.stdout:[mgr.y] 2024-04-25T00:00:38.522 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQAGnSlmWCnMIhAArxJRyznF6+96SRdUIQ7sUQ== 2024-04-25T00:00:38.523 INFO:teuthology.orchestra.run.smithi174.stdout:[mds.a] 2024-04-25T00:00:38.523 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQAGnSlmbToMKRAAvN/+75wrWpMh7krRZTVN5A== 2024-04-25T00:00:38.523 INFO:teuthology.orchestra.run.smithi174.stdout:[mds.c] 2024-04-25T00:00:38.523 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQAGnSlmmUfnLhAAsyp002Uq1lGrocU5XHQGhA== 2024-04-25T00:00:38.523 INFO:teuthology.orchestra.run.smithi174.stdout:[osd.0] 2024-04-25T00:00:38.523 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQAKnSlmXP/JBBAAk884h7I/PKt9PeYVEIqwRA== 2024-04-25T00:00:38.523 INFO:teuthology.orchestra.run.smithi174.stdout:[osd.1] 2024-04-25T00:00:38.523 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQANnSlmD+aTCRAAcwjV201nyPK0xxa3v20iRA== 2024-04-25T00:00:38.523 INFO:teuthology.orchestra.run.smithi174.stdout:[osd.2] 2024-04-25T00:00:38.523 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQAQnSlmQqH/EBAA2NoBf5Q557hNHrlBGnSnkA== 2024-04-25T00:00:38.523 INFO:teuthology.orchestra.run.smithi174.stdout:[osd.3] 2024-04-25T00:00:38.523 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQATnSlmBEGWGRAA4SGyRmG2xgKSTl0CUS2G1A== 2024-04-25T00:00:38.523 INFO:teuthology.orchestra.run.smithi174.stdout:[mgr.x] 2024-04-25T00:00:38.524 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQAGnSlmYymMJRAAohPdFNROqDUrQ/GJfIwPWg== 2024-04-25T00:00:38.524 INFO:teuthology.orchestra.run.smithi174.stdout:[mds.b] 2024-04-25T00:00:38.524 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQAGnSlmBPHpNBAA1qrZWXyLzxNP6GI5013GEA== 2024-04-25T00:00:38.524 INFO:teuthology.orchestra.run.smithi174.stdout:[mds.d] 2024-04-25T00:00:38.524 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQAHnSlm8hjLARAAwFsaX5Y/XeMTl37nLssNag== 2024-04-25T00:00:38.524 INFO:teuthology.orchestra.run.smithi174.stdout:[osd.4] 2024-04-25T00:00:38.524 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQAZnSlmYv4UGhAABSMFHU5YnN6W2wJ/coR54w== 2024-04-25T00:00:38.524 INFO:teuthology.orchestra.run.smithi174.stdout:[osd.5] 2024-04-25T00:00:38.524 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQAcnSlmklNWHhAAh9YoBc2VccvB8nYrzMcNmw== 2024-04-25T00:00:38.524 INFO:teuthology.orchestra.run.smithi174.stdout:[osd.6] 2024-04-25T00:00:38.524 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQAfnSlmXmlBJRAAqTWRzjlyciHJhOde6i3o4A== 2024-04-25T00:00:38.524 INFO:teuthology.orchestra.run.smithi174.stdout:[osd.7] 2024-04-25T00:00:38.524 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQAinSlm9GPmKRAAqSo6fs2e00+ENMGqcRf8SQ== 2024-04-25T00:00:38.525 INFO:teuthology.orchestra.run.smithi174.stdout:[client.0] 2024-04-25T00:00:38.525 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQAHnSlmx0tEBxAAGnwgy/o20FiLzXpo6sLVRg== 2024-04-25T00:00:38.525 INFO:teuthology.orchestra.run.smithi174.stdout:[client.1] 2024-04-25T00:00:38.525 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQAHnSlmzrXWCRAAcQxXrxYwlY+DSUs62WjbGw== 2024-04-25T00:00:38.526 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-25T00:00:38.567 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-25T00:00:38.612 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T00:00:38.652 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T00:00:38.706 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T00:00:38.740 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T00:00:38.785 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T00:00:38.787 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T00:00:38.864 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T00:00:38.867 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T00:00:38.948 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T00:00:38.950 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T00:00:39.035 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T00:00:39.038 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T00:00:39.120 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-25T00:00:39.122 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-25T00:00:39.200 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T00:00:39.208 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T00:00:39.287 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T00:00:39.290 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T00:00:39.337 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T00:00:39.375 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T00:00:39.416 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T00:00:39.460 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T00:00:39.500 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T00:00:39.544 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T00:00:39.586 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T00:00:39.632 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T00:00:39.675 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T00:00:39.720 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T00:00:39.768 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T00:00:39.770 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T00:00:39.812 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-25T00:00:39.812 DEBUG:teuthology.orchestra.run.smithi116:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-25T00:00:39.865 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-25T00:00:39.959 DEBUG:teuthology.orchestra.run.smithi116:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-25T00:00:39.970 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-04-25T00:00:39.985 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-25T00:00:40.080 DEBUG:teuthology.orchestra.run.smithi174:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-04-25T00:00:40.091 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-04-25T00:00:40.141 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-25T00:00:40.232 DEBUG:teuthology.orchestra.run.smithi174:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-04-25T00:00:40.242 DEBUG:teuthology.orchestra.run.smithi116:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-25T00:00:40.245 DEBUG:teuthology.orchestra.run.smithi174:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-25T00:00:40.288 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-25T00:00:40.289 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-25T00:00:40.289 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-25T00:00:40.291 INFO:tasks.ceph.mon.a:Started 2024-04-25T00:00:40.291 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-25T00:00:40.292 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-25T00:00:40.334 INFO:tasks.ceph.mon.b:Started 2024-04-25T00:00:40.334 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-25T00:00:40.335 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-25T00:00:40.337 INFO:tasks.ceph.mon.c:Started 2024-04-25T00:00:40.337 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-25T00:00:40.337 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-04-25T00:00:40.337 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-04-25T00:00:40.340 INFO:tasks.ceph.mgr.y:Started 2024-04-25T00:00:40.340 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-25T00:00:40.340 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-25T00:00:40.343 INFO:tasks.ceph.mgr.x:Started 2024-04-25T00:00:40.343 INFO:tasks.ceph:Setting crush tunables to default 2024-04-25T00:00:40.343 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --cluster ceph osd crush tunables default 2024-04-25T00:00:40.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.454+0000 7f6f232b3640 1 Processor -- start 2024-04-25T00:00:40.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.454+0000 7f6f232b3640 1 -- start start 2024-04-25T00:00:40.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.458+0000 7f6f232b3640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f6f1c152da0 0x7f6f1c06da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:40.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.458+0000 7f6f232b3640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6f1c06dfc0 0x7f6f1c06b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:40.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.458+0000 7f6f232b3640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f1c1566c0 con 0x7f6f1c0adcf0 2024-04-25T00:00:40.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.458+0000 7f6f232b3640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f6f1c0c31f0 con 0x7f6f1c152da0 2024-04-25T00:00:40.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.458+0000 7f6f232b3640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6f1c0c28c0 con 0x7f6f1c06dfc0 2024-04-25T00:00:40.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.458+0000 7f6f21850640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f6f1c0adcf0 0x7f6f1c158e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:36658/0 (socket says 172.21.15.116:36658) 2024-04-25T00:00:40.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.458+0000 7f6f21850640 1 -- 172.21.15.116:0/4265697325 learned_addr learned my addr 172.21.15.116:0/4265697325 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:00:40.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.458+0000 7f6f2084e640 1 --2- 172.21.15.116:0/4265697325 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6f1c06dfc0 0x7f6f1c06b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:40.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.458+0000 7f6f2104f640 1 --2- 172.21.15.116:0/4265697325 >> v2:172.21.15.174:3300/0 conn(0x7f6f1c152da0 0x7f6f1c06da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:40.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.458+0000 7f6f2084e640 1 -- 172.21.15.116:0/4265697325 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6f1c06dfc0 msgr2=0x7f6f1c06b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T00:00:40.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.458+0000 7f6f2084e640 1 -- 172.21.15.116:0/4265697325 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6f1c06dfc0 msgr2=0x7f6f1c06b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T00:00:40.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.458+0000 7f6f2104f640 1 -- 172.21.15.116:0/4265697325 >> v2:172.21.15.174:3300/0 conn(0x7f6f1c152da0 msgr2=0x7f6f1c06da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-25T00:00:40.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.458+0000 7f6f2084e640 1 --2- 172.21.15.116:0/4265697325 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6f1c06dfc0 0x7f6f1c06b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T00:00:40.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.458+0000 7f6f2104f640 1 -- 172.21.15.116:0/4265697325 >> v2:172.21.15.174:3300/0 conn(0x7f6f1c152da0 msgr2=0x7f6f1c06da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T00:00:40.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.458+0000 7f6f2104f640 1 --2- 172.21.15.116:0/4265697325 >> v2:172.21.15.174:3300/0 conn(0x7f6f1c152da0 0x7f6f1c06da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T00:00:40.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.458+0000 7f6f2084e640 1 --2- 172.21.15.116:0/4265697325 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6f1c06dfc0 0x7f6f1c06b210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T00:00:40.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.458+0000 7f6f2104f640 1 --2- 172.21.15.116:0/4265697325 >> v2:172.21.15.174:3300/0 conn(0x7f6f1c152da0 0x7f6f1c06da80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T00:00:40.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.662+0000 7f6f2104f640 1 --2- 172.21.15.116:0/4265697325 >> v2:172.21.15.174:3300/0 conn(0x7f6f1c152da0 0x7f6f1c06da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:40.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.662+0000 7f6f2084e640 1 --2- 172.21.15.116:0/4265697325 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6f1c06dfc0 0x7f6f1c06b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:40.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.662+0000 7f6f2104f640 1 -- 172.21.15.116:0/4265697325 >> v2:172.21.15.174:3300/0 conn(0x7f6f1c152da0 msgr2=0x7f6f1c06da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-25T00:00:40.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.662+0000 7f6f2104f640 1 -- 172.21.15.116:0/4265697325 >> v2:172.21.15.174:3300/0 conn(0x7f6f1c152da0 msgr2=0x7f6f1c06da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T00:00:40.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.662+0000 7f6f2104f640 1 --2- 172.21.15.116:0/4265697325 >> v2:172.21.15.174:3300/0 conn(0x7f6f1c152da0 0x7f6f1c06da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T00:00:40.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.662+0000 7f6f2084e640 1 -- 172.21.15.116:0/4265697325 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6f1c06dfc0 msgr2=0x7f6f1c06b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-25T00:00:40.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.662+0000 7f6f2084e640 1 -- 172.21.15.116:0/4265697325 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6f1c06dfc0 msgr2=0x7f6f1c06b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T00:00:40.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.662+0000 7f6f2084e640 1 --2- 172.21.15.116:0/4265697325 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6f1c06dfc0 0x7f6f1c06b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T00:00:40.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.662+0000 7f6f2104f640 1 --2- 172.21.15.116:0/4265697325 >> v2:172.21.15.174:3300/0 conn(0x7f6f1c152da0 0x7f6f1c06da80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T00:00:40.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.662+0000 7f6f2084e640 1 --2- 172.21.15.116:0/4265697325 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6f1c06dfc0 0x7f6f1c06b210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T00:00:40.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.714+0000 7f6f13fff640 1 -- 172.21.15.116:0/4265697325 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 324328103 0 0) 0x7f6f1c1566c0 con 0x7f6f1c0adcf0 2024-04-25T00:00:40.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.714+0000 7f6f13fff640 1 -- 172.21.15.116:0/4265697325 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f08003530 con 0x7f6f1c0adcf0 2024-04-25T00:00:40.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.714+0000 7f6f13fff640 1 -- 172.21.15.116:0/4265697325 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 770664062 0 0) 0x7f6f08003530 con 0x7f6f1c0adcf0 2024-04-25T00:00:40.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.714+0000 7f6f13fff640 1 -- 172.21.15.116:0/4265697325 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6f1c06dfc0 msgr2=0x7f6f1c06b210 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T00:00:40.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.714+0000 7f6f13fff640 1 --2- 172.21.15.116:0/4265697325 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6f1c06dfc0 0x7f6f1c06b210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:40.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.714+0000 7f6f13fff640 1 -- 172.21.15.116:0/4265697325 >> v2:172.21.15.174:3300/0 conn(0x7f6f1c152da0 msgr2=0x7f6f1c06da80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T00:00:40.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.714+0000 7f6f13fff640 1 --2- 172.21.15.116:0/4265697325 >> v2:172.21.15.174:3300/0 conn(0x7f6f1c152da0 0x7f6f1c06da80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:40.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.714+0000 7f6f13fff640 1 -- 172.21.15.116:0/4265697325 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f1c0c2ba0 con 0x7f6f1c0adcf0 2024-04-25T00:00:40.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.714+0000 7f6f13fff640 1 -- 172.21.15.116:0/4265697325 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f6f14001a00 con 0x7f6f1c0adcf0 2024-04-25T00:00:40.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.714+0000 7f6f13fff640 1 -- 172.21.15.116:0/4265697325 >> v1:172.21.15.116:6789/0 conn(0x7f6f1c0adcf0 legacy=0x7f6f1c158e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:40.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.714+0000 7f6f13fff640 1 --2- 172.21.15.116:0/4265697325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f080039c0 0x7f6f08003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:40.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.714+0000 7f6f13fff640 1 --2- 172.21.15.116:0/4265697325 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6f1c06dfc0 0x7f6f08004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:40.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.714+0000 7f6f13fff640 1 --2- 172.21.15.116:0/4265697325 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6f08004d50 0x7f6f080051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:40.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.714+0000 7f6f13fff640 1 -- 172.21.15.116:0/4265697325 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6f1c0c31f0 con 0x7f6f080039c0 2024-04-25T00:00:40.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.714+0000 7f6f13fff640 1 -- 172.21.15.116:0/4265697325 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6f1c0c28c0 con 0x7f6f08004d50 2024-04-25T00:00:40.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.714+0000 7f6f13fff640 1 -- 172.21.15.116:0/4265697325 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6f0800ba80 con 0x7f6f1c06dfc0 2024-04-25T00:00:40.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.714+0000 7f6f2104f640 1 --2- 172.21.15.116:0/4265697325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f080039c0 0x7f6f08003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:40.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.714+0000 7f6f21850640 1 --2- 172.21.15.116:0/4265697325 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6f08004d50 0x7f6f080051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:40.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.714+0000 7f6f2084e640 1 --2- 172.21.15.116:0/4265697325 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6f1c06dfc0 0x7f6f08004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:40.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.718+0000 7f6f2104f640 1 -- 172.21.15.116:0/4265697325 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6f1c06dfc0 msgr2=0x7f6f08004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:00:40.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.718+0000 7f6f2104f640 1 --2- 172.21.15.116:0/4265697325 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6f1c06dfc0 0x7f6f08004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:40.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.718+0000 7f6f2104f640 1 -- 172.21.15.116:0/4265697325 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6f08004d50 msgr2=0x7f6f080051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:00:40.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.718+0000 7f6f2104f640 1 --2- 172.21.15.116:0/4265697325 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6f08004d50 0x7f6f080051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:40.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.718+0000 7f6f2104f640 1 -- 172.21.15.116:0/4265697325 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f08003530 con 0x7f6f080039c0 2024-04-25T00:00:40.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.718+0000 7f6f21850640 1 --2- 172.21.15.116:0/4265697325 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6f08004d50 0x7f6f080051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:00:40.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.722+0000 7f6f2104f640 1 --2- 172.21.15.116:0/4265697325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f080039c0 0x7f6f08003de0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6f1800e910 tx=0x7f6f1801c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=553d4de5829652df server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:00:40.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.722+0000 7f6f13fff640 1 -- 172.21.15.116:0/4265697325 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f1800f040 con 0x7f6f080039c0 2024-04-25T00:00:40.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.722+0000 7f6f13fff640 1 -- 172.21.15.116:0/4265697325 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f180aa630 con 0x7f6f080039c0 2024-04-25T00:00:40.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.722+0000 7f6f13fff640 1 -- 172.21.15.116:0/4265697325 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f180aaa00 con 0x7f6f080039c0 2024-04-25T00:00:40.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.726+0000 7f6f232b3640 1 -- 172.21.15.116:0/4265697325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f080039c0 msgr2=0x7f6f08003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:40.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.726+0000 7f6f232b3640 1 --2- 172.21.15.116:0/4265697325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f080039c0 0x7f6f08003de0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6f1800e910 tx=0x7f6f1801c8b0 comp rx=0 tx=0).stop 2024-04-25T00:00:40.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.726+0000 7f6f21850640 1 -- 172.21.15.116:0/4265697325 reap_dead start 2024-04-25T00:00:40.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.726+0000 7f6f232b3640 1 -- 172.21.15.116:0/4265697325 shutdown_connections 2024-04-25T00:00:40.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.726+0000 7f6f232b3640 1 -- 172.21.15.116:0/4265697325 >> 172.21.15.116:0/4265697325 conn(0x7f6f1c0c49c0 msgr2=0x7f6f1c0c4dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:00:40.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.726+0000 7f6f232b3640 1 -- 172.21.15.116:0/4265697325 shutdown_connections 2024-04-25T00:00:40.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.726+0000 7f6f232b3640 1 -- 172.21.15.116:0/4265697325 wait complete. 2024-04-25T00:00:40.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.726+0000 7f6f232b3640 1 Processor -- start 2024-04-25T00:00:40.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.726+0000 7f6f232b3640 1 -- start start 2024-04-25T00:00:40.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.730+0000 7f6f232b3640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6f080039c0 0x7f6f1c13cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:40.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.730+0000 7f6f232b3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6f1c14d050 0x7f6f1c0c6e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:40.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.730+0000 7f6f232b3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f1c147ec0 0x7f6f1c1f2060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:40.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.730+0000 7f6f232b3640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6f1c06b390 con 0x7f6f1c147ec0 2024-04-25T00:00:40.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.730+0000 7f6f232b3640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6f1c06b210 con 0x7f6f1c14d050 2024-04-25T00:00:40.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.730+0000 7f6f232b3640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6f1c06b510 con 0x7f6f080039c0 2024-04-25T00:00:40.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.730+0000 7f6f21850640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f1c147ec0 0x7f6f1c1f2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:40.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.730+0000 7f6f21850640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f1c147ec0 0x7f6f1c1f2060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43254/0 (socket says 172.21.15.116:43254) 2024-04-25T00:00:40.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.730+0000 7f6f2104f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6f080039c0 0x7f6f1c13cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:40.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.730+0000 7f6f2084e640 1 --2- 172.21.15.116:0/3432266833 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6f1c14d050 0x7f6f1c0c6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:40.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.730+0000 7f6f21850640 1 -- 172.21.15.116:0/3432266833 learned_addr learned my addr 172.21.15.116:0/3432266833 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:00:40.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.730+0000 7f6f21850640 1 -- 172.21.15.116:0/3432266833 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6f080039c0 msgr2=0x7f6f1c13cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:40.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.730+0000 7f6f21850640 1 --2- 172.21.15.116:0/3432266833 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6f080039c0 0x7f6f1c13cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:40.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.730+0000 7f6f21850640 1 -- 172.21.15.116:0/3432266833 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6f1c14d050 msgr2=0x7f6f1c0c6e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:40.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.730+0000 7f6f21850640 1 --2- 172.21.15.116:0/3432266833 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6f1c14d050 0x7f6f1c0c6e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:40.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.730+0000 7f6f21850640 1 -- 172.21.15.116:0/3432266833 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f1c158190 con 0x7f6f1c147ec0 2024-04-25T00:00:40.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.730+0000 7f6f21850640 1 --2- 172.21.15.116:0/3432266833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f1c147ec0 0x7f6f1c1f2060 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c073390 tx=0x7f6f14007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:00:40.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.730+0000 7f6f11ffb640 1 -- 172.21.15.116:0/3432266833 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f1400a4c0 con 0x7f6f1c147ec0 2024-04-25T00:00:40.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.730+0000 7f6f11ffb640 1 -- 172.21.15.116:0/3432266833 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f140133f0 con 0x7f6f1c147ec0 2024-04-25T00:00:40.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.730+0000 7f6f232b3640 1 -- 172.21.15.116:0/3432266833 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f1c1ef3d0 con 0x7f6f1c147ec0 2024-04-25T00:00:40.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.730+0000 7f6f11ffb640 1 -- 172.21.15.116:0/3432266833 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f140137c0 con 0x7f6f1c147ec0 2024-04-25T00:00:40.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.730+0000 7f6f232b3640 1 -- 172.21.15.116:0/3432266833 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f1c1ef0f0 con 0x7f6f1c147ec0 2024-04-25T00:00:40.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.730+0000 7f6f11ffb640 1 -- 172.21.15.116:0/3432266833 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6f14013cf0 con 0x7f6f1c147ec0 2024-04-25T00:00:40.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.730+0000 7f6f11ffb640 1 -- 172.21.15.116:0/3432266833 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f6f1401b860 con 0x7f6f1c147ec0 2024-04-25T00:00:40.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.730+0000 7f6f232b3640 1 -- 172.21.15.116:0/3432266833 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f1c1eeee0 con 0x7f6f1c147ec0 2024-04-25T00:00:40.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.734+0000 7f6f11ffb640 1 -- 172.21.15.116:0/3432266833 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f6f1401ba60 con 0x7f6f1c147ec0 2024-04-25T00:00:40.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.802+0000 7f6f232b3640 1 -- 172.21.15.116:0/3432266833 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f6f1c04f510 con 0x7f6f1c147ec0 2024-04-25T00:00:40.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.802+0000 7f6f11ffb640 1 -- 172.21.15.116:0/3432266833 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7f6f1401bc40 con 0x7f6f1c147ec0 2024-04-25T00:00:40.806 INFO:teuthology.orchestra.run.smithi116.stderr:adjusted tunables profile to default 2024-04-25T00:00:40.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.806+0000 7f6f232b3640 1 -- 172.21.15.116:0/3432266833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f1c147ec0 msgr2=0x7f6f1c1f2060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:40.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.806+0000 7f6f232b3640 1 --2- 172.21.15.116:0/3432266833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f1c147ec0 0x7f6f1c1f2060 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c073390 tx=0x7f6f14007430 comp rx=0 tx=0).stop 2024-04-25T00:00:40.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.806+0000 7f6f232b3640 1 -- 172.21.15.116:0/3432266833 shutdown_connections 2024-04-25T00:00:40.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.806+0000 7f6f232b3640 1 --2- 172.21.15.116:0/3432266833 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f1c147ec0 0x7f6f1c1f2060 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:40.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.806+0000 7f6f232b3640 1 --2- 172.21.15.116:0/3432266833 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6f1c14d050 0x7f6f1c0c6e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:40.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.806+0000 7f6f232b3640 1 --2- 172.21.15.116:0/3432266833 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6f080039c0 0x7f6f1c13cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:40.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.806+0000 7f6f232b3640 1 -- 172.21.15.116:0/3432266833 >> 172.21.15.116:0/3432266833 conn(0x7f6f1c0c49c0 msgr2=0x7f6f1c15adf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:00:40.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.806+0000 7f6f232b3640 1 -- 172.21.15.116:0/3432266833 shutdown_connections 2024-04-25T00:00:40.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:40.806+0000 7f6f232b3640 1 -- 172.21.15.116:0/3432266833 wait complete. 2024-04-25T00:00:40.825 INFO:tasks.ceph:check_enable_crimson: False 2024-04-25T00:00:40.825 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-25T00:00:40.825 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-04-25T00:00:40.825 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-25T00:00:40.836 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-04-25T00:00:40.836 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-25T00:00:40.892 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-04-25T00:00:40.893 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-25T00:00:40.908 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-04-25T00:00:40.903+0000 7fc0f5688180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T00:00:40.938 INFO:tasks.ceph.mgr.y.smithi116.stderr:2024-04-25T00:00:40.934+0000 7f494fb44180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T00:00:40.946 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-04-25T00:00:40.946 DEBUG:teuthology.orchestra.run.smithi116:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-25T00:00:41.003 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-04-25T00:00:41.003 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-04-25T00:00:41.017 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-04-25T00:00:41.017 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-04-25T00:00:41.072 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-04-25T00:00:41.072 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-04-25T00:00:41.129 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-04-25T00:00:41.129 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-04-25T00:00:41.186 DEBUG:teuthology.orchestra.run.smithi174:> sudo ceph --cluster ceph osd new 8b6ddcb2-1ad9-4167-961c-bd6936bb9060 0 2024-04-25T00:00:41.351 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.347+0000 7fa8819d5640 1 Processor -- start 2024-04-25T00:00:41.351 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.347+0000 7fa8819d5640 1 -- start start 2024-04-25T00:00:41.352 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.347+0000 7fa8819d5640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa87c072ff0 0x7fa87c1531c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:41.352 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.347+0000 7fa8819d5640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa87c06b210 0x7fa87c158e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:41.352 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.347+0000 7fa8819d5640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa87c06c930 con 0x7fa87c06e930 2024-04-25T00:00:41.352 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.347+0000 7fa8819d5640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fa87c06b660 con 0x7fa87c072ff0 2024-04-25T00:00:41.352 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.347+0000 7fa8819d5640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa87c071e90 con 0x7fa87c06b210 2024-04-25T00:00:41.352 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.347+0000 7fa87b7fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa87c06b210 0x7fa87c158e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:41.352 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.347+0000 7fa87b7fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa87c06b210 0x7fa87c158e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.174:54628/0 (socket says 172.21.15.174:54628) 2024-04-25T00:00:41.352 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.347+0000 7fa87b7fe640 1 -- 172.21.15.174:0/1334805883 learned_addr learned my addr 172.21.15.174:0/1334805883 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-04-25T00:00:41.352 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.347+0000 7fa87a7fc640 1 --2- 172.21.15.174:0/1334805883 >> v2:172.21.15.116:3300/0 conn(0x7fa87c072ff0 0x7fa87c1531c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:41.353 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.347+0000 7fa87b7fe640 1 -- 172.21.15.174:0/1334805883 >> v2:172.21.15.116:3300/0 conn(0x7fa87c072ff0 msgr2=0x7fa87c1531c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:00:41.353 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.347+0000 7fa87b7fe640 1 --2- 172.21.15.174:0/1334805883 >> v2:172.21.15.116:3300/0 conn(0x7fa87c072ff0 0x7fa87c1531c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:41.353 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.347+0000 7fa87b7fe640 1 -- 172.21.15.174:0/1334805883 >> v1:172.21.15.116:6789/0 conn(0x7fa87c06e930 legacy=0x7fa87c152ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:00:41.353 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.347+0000 7fa87b7fe640 1 -- 172.21.15.174:0/1334805883 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa87c072170 con 0x7fa87c06b210 2024-04-25T00:00:41.355 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.351+0000 7fa87b7fe640 1 --2- 172.21.15.174:0/1334805883 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa87c06b210 0x7fa87c158e30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa87c073480 tx=0x7fa86c01c3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36fb426d779ff7d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:00:41.355 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.351+0000 7fa879ffb640 1 -- 172.21.15.174:0/1334805883 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa86c0b1070 con 0x7fa87c06b210 2024-04-25T00:00:41.355 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.351+0000 7fa879ffb640 1 -- 172.21.15.174:0/1334805883 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa86c0ac410 con 0x7fa87c06b210 2024-04-25T00:00:41.355 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.351+0000 7fa879ffb640 1 -- 172.21.15.174:0/1334805883 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa86c0ac7e0 con 0x7fa87c06b210 2024-04-25T00:00:41.356 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.351+0000 7fa8819d5640 1 -- 172.21.15.174:0/1334805883 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa87c06b210 msgr2=0x7fa87c158e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:41.356 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.351+0000 7fa8819d5640 1 --2- 172.21.15.174:0/1334805883 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa87c06b210 0x7fa87c158e30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa87c073480 tx=0x7fa86c01c3f0 comp rx=0 tx=0).stop 2024-04-25T00:00:41.356 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.351+0000 7fa8819d5640 1 -- 172.21.15.174:0/1334805883 shutdown_connections 2024-04-25T00:00:41.356 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.351+0000 7fa8819d5640 1 --2- 172.21.15.174:0/1334805883 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa87c06b210 0x7fa87c158e30 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:41.356 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.351+0000 7fa8819d5640 1 --2- 172.21.15.174:0/1334805883 >> v2:172.21.15.116:3300/0 conn(0x7fa87c072ff0 0x7fa87c1531c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:41.356 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.351+0000 7fa8819d5640 1 -- 172.21.15.174:0/1334805883 >> 172.21.15.174:0/1334805883 conn(0x7fa87c0c7a90 msgr2=0x7fa87c0c7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:00:41.356 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.351+0000 7fa8819d5640 1 -- 172.21.15.174:0/1334805883 shutdown_connections 2024-04-25T00:00:41.356 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.351+0000 7fa8819d5640 1 -- 172.21.15.174:0/1334805883 wait complete. 2024-04-25T00:00:41.357 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.355+0000 7fa8819d5640 1 Processor -- start 2024-04-25T00:00:41.357 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.355+0000 7fa8819d5640 1 -- start start 2024-04-25T00:00:41.359 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.355+0000 7fa8819d5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa87c06b210 0x7fa87c0ac110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:41.359 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.355+0000 7fa8819d5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa87c06e930 0x7fa87c14de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:41.359 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.355+0000 7fa8819d5640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa87c072ff0 0x7fa87c1f1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:41.360 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.355+0000 7fa8819d5640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa87c06ee90 con 0x7fa87c06e930 2024-04-25T00:00:41.360 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.355+0000 7fa8819d5640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa87c06ed10 con 0x7fa87c06b210 2024-04-25T00:00:41.360 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.355+0000 7fa8819d5640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa87c162170 con 0x7fa87c072ff0 2024-04-25T00:00:41.360 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.355+0000 7fa87affd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa87c06b210 0x7fa87c0ac110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:41.360 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.355+0000 7fa87b7fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa87c072ff0 0x7fa87c1f1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:41.360 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.355+0000 7fa87affd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa87c06b210 0x7fa87c0ac110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:55156/0 (socket says 172.21.15.174:55156) 2024-04-25T00:00:41.360 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.355+0000 7fa87b7fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa87c072ff0 0x7fa87c1f1a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.174:54638/0 (socket says 172.21.15.174:54638) 2024-04-25T00:00:41.360 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.355+0000 7fa87affd640 1 -- 172.21.15.174:0/3776154352 learned_addr learned my addr 172.21.15.174:0/3776154352 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-04-25T00:00:41.360 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.355+0000 7fa87a7fc640 1 --2- 172.21.15.174:0/3776154352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa87c06e930 0x7fa87c14de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:41.360 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.355+0000 7fa87b7fe640 1 -- 172.21.15.174:0/3776154352 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa87c06b210 msgr2=0x7fa87c0ac110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:41.361 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.355+0000 7fa87b7fe640 1 --2- 172.21.15.174:0/3776154352 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa87c06b210 0x7fa87c0ac110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:41.361 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.355+0000 7fa87b7fe640 1 -- 172.21.15.174:0/3776154352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa87c06e930 msgr2=0x7fa87c14de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:41.361 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.355+0000 7fa87b7fe640 1 --2- 172.21.15.174:0/3776154352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa87c06e930 0x7fa87c14de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:41.361 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.355+0000 7fa87b7fe640 1 -- 172.21.15.174:0/3776154352 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa87c0c65e0 con 0x7fa87c072ff0 2024-04-25T00:00:41.361 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.355+0000 7fa87a7fc640 1 --2- 172.21.15.174:0/3776154352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa87c06e930 0x7fa87c14de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:00:41.361 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.355+0000 7fa87b7fe640 1 --2- 172.21.15.174:0/3776154352 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa87c072ff0 0x7fa87c1f1a50 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa86c0ac000 tx=0x7fa86c019190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:00:41.361 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.355+0000 7fa863fff640 1 -- 172.21.15.174:0/3776154352 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa86c0bc070 con 0x7fa87c072ff0 2024-04-25T00:00:41.361 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.355+0000 7fa863fff640 1 -- 172.21.15.174:0/3776154352 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa86c0acd80 con 0x7fa87c072ff0 2024-04-25T00:00:41.361 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.355+0000 7fa8819d5640 1 -- 172.21.15.174:0/3776154352 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa87c0c6dc0 con 0x7fa87c072ff0 2024-04-25T00:00:41.361 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.355+0000 7fa863fff640 1 -- 172.21.15.174:0/3776154352 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa86c0b1040 con 0x7fa87c072ff0 2024-04-25T00:00:41.361 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.355+0000 7fa8819d5640 1 -- 172.21.15.174:0/3776154352 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa87c1f1530 con 0x7fa87c072ff0 2024-04-25T00:00:41.361 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.355+0000 7fa863fff640 1 -- 172.21.15.174:0/3776154352 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa86c0bf070 con 0x7fa87c072ff0 2024-04-25T00:00:41.361 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.355+0000 7fa863fff640 1 -- 172.21.15.174:0/3776154352 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7fa86c0c1440 con 0x7fa87c072ff0 2024-04-25T00:00:41.362 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.355+0000 7fa8819d5640 1 -- 172.21.15.174:0/3776154352 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa87c04f510 con 0x7fa87c072ff0 2024-04-25T00:00:41.366 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-04-25T00:00:41.363+0000 7fc0f5688180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T00:00:41.366 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.363+0000 7fa863fff640 1 -- 172.21.15.174:0/3776154352 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fa87c0c6dc0 con 0x7fa87c072ff0 2024-04-25T00:00:41.418 INFO:tasks.ceph.mgr.y.smithi116.stderr:2024-04-25T00:00:41.414+0000 7f494fb44180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T00:00:41.435 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.431+0000 7fa8819d5640 1 -- 172.21.15.174:0/3776154352 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "osd new", "uuid": "8b6ddcb2-1ad9-4167-961c-bd6936bb9060", "id": 0} v 0) -- 0x7fa87c153390 con 0x7fa87c072ff0 2024-04-25T00:00:41.441 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.439+0000 7fa863fff640 1 -- 172.21.15.174:0/3776154352 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8b6ddcb2-1ad9-4167-961c-bd6936bb9060", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7fa86c0160d0 con 0x7fa87c072ff0 2024-04-25T00:00:41.441 INFO:teuthology.orchestra.run.smithi174.stdout:0 2024-04-25T00:00:41.443 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.439+0000 7fa8819d5640 1 -- 172.21.15.174:0/3776154352 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa87c072ff0 msgr2=0x7fa87c1f1a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:41.443 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.439+0000 7fa8819d5640 1 --2- 172.21.15.174:0/3776154352 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa87c072ff0 0x7fa87c1f1a50 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa86c0ac000 tx=0x7fa86c019190 comp rx=0 tx=0).stop 2024-04-25T00:00:41.443 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.439+0000 7fa8819d5640 1 -- 172.21.15.174:0/3776154352 shutdown_connections 2024-04-25T00:00:41.443 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.439+0000 7fa8819d5640 1 --2- 172.21.15.174:0/3776154352 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa87c072ff0 0x7fa87c1f1a50 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:41.443 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.439+0000 7fa8819d5640 1 --2- 172.21.15.174:0/3776154352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa87c06e930 0x7fa87c14de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:41.443 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.439+0000 7fa8819d5640 1 --2- 172.21.15.174:0/3776154352 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa87c06b210 0x7fa87c0ac110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:41.443 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.439+0000 7fa8819d5640 1 -- 172.21.15.174:0/3776154352 >> 172.21.15.174:0/3776154352 conn(0x7fa87c0c7a90 msgr2=0x7fa87c1558f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:00:41.443 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.439+0000 7fa8819d5640 1 -- 172.21.15.174:0/3776154352 shutdown_connections 2024-04-25T00:00:41.443 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.439+0000 7fa8819d5640 1 -- 172.21.15.174:0/3776154352 wait complete. 2024-04-25T00:00:41.463 DEBUG:teuthology.orchestra.run.smithi174:> sudo ceph --cluster ceph osd new a6431768-994f-41da-8853-886f5687fb91 1 2024-04-25T00:00:41.564 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-04-25T00:00:41.559+0000 7fc0f5688180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T00:00:41.576 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.571+0000 7f6ded81a640 1 Processor -- start 2024-04-25T00:00:41.576 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.571+0000 7f6ded81a640 1 -- start start 2024-04-25T00:00:41.577 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.575+0000 7f6ded81a640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f6de806b210 0x7f6de806b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:41.577 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.575+0000 7f6ded81a640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6de81539e0 con 0x7f6de806c580 2024-04-25T00:00:41.578 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.575+0000 7f6ded81a640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6de8153cc0 con 0x7f6de8150dd0 2024-04-25T00:00:41.578 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.575+0000 7f6ded81a640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f6de8153fa0 con 0x7f6de806b210 2024-04-25T00:00:41.578 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.575+0000 7f6de77fe640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f6de806b210 0x7f6de806b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:41.578 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.575+0000 7f6de6ffd640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f6de8150dd0 0x7f6de80aa740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.174:59140/0 (socket says 172.21.15.174:59140) 2024-04-25T00:00:41.578 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.575+0000 7f6de6ffd640 1 -- 172.21.15.174:0/2252255651 learned_addr learned my addr 172.21.15.174:0/2252255651 (peer_addr_for_me v1:172.21.15.174:0/0) 2024-04-25T00:00:41.578 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.575+0000 7f6de77fe640 1 -- 172.21.15.174:0/2252255651 >> v1:172.21.15.174:6789/0 conn(0x7f6de8150dd0 legacy=0x7f6de80aa740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:41.578 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.575+0000 7f6de77fe640 1 -- 172.21.15.174:0/2252255651 >> v1:172.21.15.116:6789/0 conn(0x7f6de806c580 legacy=0x7f6de806e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:00:41.578 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.575+0000 7f6de67fc640 1 -- 172.21.15.174:0/2252255651 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3571350649 0 0) 0x7f6de8153cc0 con 0x7f6de8150dd0 2024-04-25T00:00:41.578 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.575+0000 7f6de77fe640 1 -- 172.21.15.174:0/2252255651 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6de8153690 con 0x7f6de806b210 2024-04-25T00:00:41.580 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.575+0000 7f6de77fe640 1 --2- 172.21.15.174:0/2252255651 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6de806b210 0x7f6de806b6a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6de0009d90 tx=0x7f6de001c930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56bed375b95fe1b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:00:41.580 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.575+0000 7f6de67fc640 1 -- 172.21.15.174:0/2252255651 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de00b2070 con 0x7f6de806b210 2024-04-25T00:00:41.580 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.575+0000 7f6de67fc640 1 -- 172.21.15.174:0/2252255651 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6de00a5900 con 0x7f6de806b210 2024-04-25T00:00:41.580 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.575+0000 7f6de67fc640 1 -- 172.21.15.174:0/2252255651 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de00a5cd0 con 0x7f6de806b210 2024-04-25T00:00:41.581 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.579+0000 7f6ded81a640 1 -- 172.21.15.174:0/2252255651 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6de806b210 msgr2=0x7f6de806b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:41.581 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.579+0000 7f6ded81a640 1 --2- 172.21.15.174:0/2252255651 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6de806b210 0x7f6de806b6a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6de0009d90 tx=0x7f6de001c930 comp rx=0 tx=0).stop 2024-04-25T00:00:41.581 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.579+0000 7f6ded81a640 1 -- 172.21.15.174:0/2252255651 shutdown_connections 2024-04-25T00:00:41.581 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.579+0000 7f6ded81a640 1 --2- 172.21.15.174:0/2252255651 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6de806b210 0x7f6de806b6a0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:41.581 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.579+0000 7f6ded81a640 1 -- 172.21.15.174:0/2252255651 >> 172.21.15.174:0/2252255651 conn(0x7f6de80c7a90 msgr2=0x7f6de80c7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:00:41.581 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.579+0000 7f6ded81a640 1 -- 172.21.15.174:0/2252255651 shutdown_connections 2024-04-25T00:00:41.581 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.579+0000 7f6ded81a640 1 -- 172.21.15.174:0/2252255651 wait complete. 2024-04-25T00:00:41.582 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.579+0000 7f6ded81a640 1 Processor -- start 2024-04-25T00:00:41.582 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.579+0000 7f6ded81a640 1 -- start start 2024-04-25T00:00:41.584 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.579+0000 7f6ded81a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6de806b210 0x7f6de8155470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:41.584 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.579+0000 7f6ded81a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6de806c580 0x7f6de814b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:41.584 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.579+0000 7f6ded81a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6de8150dd0 0x7f6de814a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:41.584 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.579+0000 7f6ded81a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6de8161bd0 con 0x7f6de806b210 2024-04-25T00:00:41.584 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.579+0000 7f6ded81a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6de8161a50 con 0x7f6de806c580 2024-04-25T00:00:41.584 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.579+0000 7f6ded81a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6de8161d50 con 0x7f6de8150dd0 2024-04-25T00:00:41.585 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.579+0000 7f6dde7fc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6de806c580 0x7f6de814b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:41.585 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.579+0000 7f6dde7fc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6de806c580 0x7f6de814b8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:55170/0 (socket says 172.21.15.174:55170) 2024-04-25T00:00:41.585 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.579+0000 7f6dde7fc640 1 -- 172.21.15.174:0/3232734874 learned_addr learned my addr 172.21.15.174:0/3232734874 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-04-25T00:00:41.585 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.579+0000 7f6de77fe640 1 --2- 172.21.15.174:0/3232734874 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6de8150dd0 0x7f6de814a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:41.585 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.579+0000 7f6de6ffd640 1 --2- 172.21.15.174:0/3232734874 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6de806b210 0x7f6de8155470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:41.585 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.579+0000 7f6dde7fc640 1 -- 172.21.15.174:0/3232734874 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6de8150dd0 msgr2=0x7f6de814a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:41.585 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.579+0000 7f6dde7fc640 1 --2- 172.21.15.174:0/3232734874 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6de8150dd0 0x7f6de814a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:41.585 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.579+0000 7f6dde7fc640 1 -- 172.21.15.174:0/3232734874 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6de806b210 msgr2=0x7f6de8155470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:41.585 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.579+0000 7f6dde7fc640 1 --2- 172.21.15.174:0/3232734874 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6de806b210 0x7f6de8155470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:41.585 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.579+0000 7f6dde7fc640 1 -- 172.21.15.174:0/3232734874 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6de8154b20 con 0x7f6de806c580 2024-04-25T00:00:41.585 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.579+0000 7f6de6ffd640 1 --2- 172.21.15.174:0/3232734874 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6de806b210 0x7f6de8155470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:00:41.585 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.579+0000 7f6de77fe640 1 --2- 172.21.15.174:0/3232734874 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6de8150dd0 0x7f6de814a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:00:41.585 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.579+0000 7f6dde7fc640 1 --2- 172.21.15.174:0/3232734874 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6de806c580 0x7f6de814b8a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6dd000c450 tx=0x7f6dd000c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:00:41.585 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.579+0000 7f6de4ff9640 1 -- 172.21.15.174:0/3232734874 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dd0012830 con 0x7f6de806c580 2024-04-25T00:00:41.586 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.579+0000 7f6de4ff9640 1 -- 172.21.15.174:0/3232734874 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6dd0017070 con 0x7f6de806c580 2024-04-25T00:00:41.586 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.579+0000 7f6de4ff9640 1 -- 172.21.15.174:0/3232734874 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dd0012c30 con 0x7f6de806c580 2024-04-25T00:00:41.586 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.579+0000 7f6ded81a640 1 -- 172.21.15.174:0/3232734874 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6de8154e00 con 0x7f6de806c580 2024-04-25T00:00:41.586 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.579+0000 7f6ded81a640 1 -- 172.21.15.174:0/3232734874 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6de814bde0 con 0x7f6de806c580 2024-04-25T00:00:41.586 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.579+0000 7f6de4ff9640 1 -- 172.21.15.174:0/3232734874 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6dd001c460 con 0x7f6de806c580 2024-04-25T00:00:41.586 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.579+0000 7f6de4ff9640 1 -- 172.21.15.174:0/3232734874 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f6dd001b7c0 con 0x7f6de806c580 2024-04-25T00:00:41.586 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.579+0000 7f6ded81a640 1 -- 172.21.15.174:0/3232734874 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6de804f510 con 0x7f6de806c580 2024-04-25T00:00:41.586 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.583+0000 7f6de4ff9640 1 -- 172.21.15.174:0/3232734874 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f6dd0012dd0 con 0x7f6de806c580 2024-04-25T00:00:41.626 INFO:tasks.ceph.mgr.y.smithi116.stderr:2024-04-25T00:00:41.622+0000 7f494fb44180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T00:00:41.655 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.651+0000 7f6ded81a640 1 -- 172.21.15.174:0/3232734874 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd new", "uuid": "a6431768-994f-41da-8853-886f5687fb91", "id": 1} v 0) -- 0x7f6de8074480 con 0x7f6de806c580 2024-04-25T00:00:41.665 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.663+0000 7f6de4ff9640 1 -- 172.21.15.174:0/3232734874 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a6431768-994f-41da-8853-886f5687fb91", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f6dd001b9c0 con 0x7f6de806c580 2024-04-25T00:00:41.666 INFO:teuthology.orchestra.run.smithi174.stdout:1 2024-04-25T00:00:41.667 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.663+0000 7f6ded81a640 1 -- 172.21.15.174:0/3232734874 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6de806c580 msgr2=0x7f6de814b8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:41.667 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.663+0000 7f6ded81a640 1 --2- 172.21.15.174:0/3232734874 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6de806c580 0x7f6de814b8a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6dd000c450 tx=0x7f6dd000c910 comp rx=0 tx=0).stop 2024-04-25T00:00:41.667 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.663+0000 7f6ded81a640 1 -- 172.21.15.174:0/3232734874 shutdown_connections 2024-04-25T00:00:41.667 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.663+0000 7f6ded81a640 1 --2- 172.21.15.174:0/3232734874 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6de8150dd0 0x7f6de814a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:41.667 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.663+0000 7f6ded81a640 1 --2- 172.21.15.174:0/3232734874 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6de806c580 0x7f6de814b8a0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:41.667 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.663+0000 7f6ded81a640 1 --2- 172.21.15.174:0/3232734874 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6de806b210 0x7f6de8155470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:41.667 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.663+0000 7f6ded81a640 1 -- 172.21.15.174:0/3232734874 >> 172.21.15.174:0/3232734874 conn(0x7f6de80c7a90 msgr2=0x7f6de806e6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:00:41.667 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.663+0000 7f6ded81a640 1 -- 172.21.15.174:0/3232734874 shutdown_connections 2024-04-25T00:00:41.667 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.663+0000 7f6ded81a640 1 -- 172.21.15.174:0/3232734874 wait complete. 2024-04-25T00:00:41.682 DEBUG:teuthology.orchestra.run.smithi174:> sudo ceph --cluster ceph osd new 4f69bd40-5a23-45ff-8693-e1de75e1aa9c 2 2024-04-25T00:00:41.794 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.791+0000 7f0b3f2ef640 1 Processor -- start 2024-04-25T00:00:41.795 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.791+0000 7f0b3f2ef640 1 -- start start 2024-04-25T00:00:41.798 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.791+0000 7f0b3f2ef640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b3806c6a0 0x7f0b380ac330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:41.799 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.791+0000 7f0b3f2ef640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0b380ac870 0x7f0b3806b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:41.799 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.791+0000 7f0b3f2ef640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b380aae50 con 0x7f0b3814db70 2024-04-25T00:00:41.799 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.791+0000 7f0b3f2ef640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f0b38157940 con 0x7f0b380ac870 2024-04-25T00:00:41.799 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.791+0000 7f0b3f2ef640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f0b38156d70 con 0x7f0b3806c6a0 2024-04-25T00:00:41.799 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.791+0000 7f0b3c88a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b3806c6a0 0x7f0b380ac330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:41.799 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.791+0000 7f0b3d08b640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f0b3814db70 0x7f0b380aa740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.174:59142/0 (socket says 172.21.15.174:59142) 2024-04-25T00:00:41.799 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.791+0000 7f0b3d08b640 1 -- 172.21.15.174:0/150021674 learned_addr learned my addr 172.21.15.174:0/150021674 (peer_addr_for_me v1:172.21.15.174:0/0) 2024-04-25T00:00:41.799 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.791+0000 7f0b3c88a640 1 -- 172.21.15.174:0/150021674 >> v2:172.21.15.116:3300/0 conn(0x7f0b380ac870 msgr2=0x7f0b3806b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:00:41.799 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.791+0000 7f0b2ffff640 1 -- 172.21.15.174:0/150021674 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4012693221 0 0) 0x7f0b380aae50 con 0x7f0b3814db70 2024-04-25T00:00:41.799 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.791+0000 7f0b3c88a640 1 --2- 172.21.15.174:0/150021674 >> v2:172.21.15.116:3300/0 conn(0x7f0b380ac870 0x7f0b3806b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:41.799 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.791+0000 7f0b3c88a640 1 -- 172.21.15.174:0/150021674 >> v1:172.21.15.174:6789/0 conn(0x7f0b3814db70 legacy=0x7f0b380aa740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:41.799 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.791+0000 7f0b3c88a640 1 -- 172.21.15.174:0/150021674 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b38157010 con 0x7f0b3806c6a0 2024-04-25T00:00:41.800 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.795+0000 7f0b3c88a640 1 --2- 172.21.15.174:0/150021674 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b3806c6a0 0x7f0b380ac330 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0b280097c0 tx=0x7f0b2801c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11eb1d747e097535 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:00:41.800 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.795+0000 7f0b2ffff640 1 -- 172.21.15.174:0/150021674 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b280b0070 con 0x7f0b3806c6a0 2024-04-25T00:00:41.800 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.795+0000 7f0b2ffff640 1 -- 172.21.15.174:0/150021674 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b280ab600 con 0x7f0b3806c6a0 2024-04-25T00:00:41.800 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.795+0000 7f0b2ffff640 1 -- 172.21.15.174:0/150021674 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b280ab9d0 con 0x7f0b3806c6a0 2024-04-25T00:00:41.801 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.795+0000 7f0b3f2ef640 1 -- 172.21.15.174:0/150021674 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b3806c6a0 msgr2=0x7f0b380ac330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:41.801 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.795+0000 7f0b3f2ef640 1 --2- 172.21.15.174:0/150021674 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b3806c6a0 0x7f0b380ac330 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0b280097c0 tx=0x7f0b2801c3e0 comp rx=0 tx=0).stop 2024-04-25T00:00:41.801 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.795+0000 7f0b3f2ef640 1 -- 172.21.15.174:0/150021674 shutdown_connections 2024-04-25T00:00:41.801 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.795+0000 7f0b3f2ef640 1 --2- 172.21.15.174:0/150021674 >> v2:172.21.15.116:3300/0 conn(0x7f0b380ac870 0x7f0b3806b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:41.801 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.795+0000 7f0b3f2ef640 1 --2- 172.21.15.174:0/150021674 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b3806c6a0 0x7f0b380ac330 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:41.801 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.795+0000 7f0b3f2ef640 1 -- 172.21.15.174:0/150021674 >> 172.21.15.174:0/150021674 conn(0x7f0b380c7a90 msgr2=0x7f0b380c7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:00:41.801 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.795+0000 7f0b3f2ef640 1 -- 172.21.15.174:0/150021674 shutdown_connections 2024-04-25T00:00:41.801 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.795+0000 7f0b3f2ef640 1 -- 172.21.15.174:0/150021674 wait complete. 2024-04-25T00:00:41.801 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.799+0000 7f0b3f2ef640 1 Processor -- start 2024-04-25T00:00:41.801 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.799+0000 7f0b3f2ef640 1 -- start start 2024-04-25T00:00:41.804 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.799+0000 7f0b3f2ef640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b380ac870 0x7f0b381505c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:41.804 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.799+0000 7f0b3f2ef640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b3814db70 0x7f0b3815c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:41.804 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.799+0000 7f0b3f2ef640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b381473c0 0x7f0b38144860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:41.804 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.799+0000 7f0b3f2ef640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0b38160370 con 0x7f0b380ac870 2024-04-25T00:00:41.804 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.799+0000 7f0b3f2ef640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0b381601f0 con 0x7f0b381473c0 2024-04-25T00:00:41.804 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.799+0000 7f0b3f2ef640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f0b381604f0 con 0x7f0b3814db70 2024-04-25T00:00:41.804 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.799+0000 7f0b3c88a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b3814db70 0x7f0b3815c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:41.804 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.799+0000 7f0b3c88a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b3814db70 0x7f0b3815c610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.174:54660/0 (socket says 172.21.15.174:54660) 2024-04-25T00:00:41.804 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.799+0000 7f0b3c88a640 1 -- 172.21.15.174:0/3309232863 learned_addr learned my addr 172.21.15.174:0/3309232863 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-04-25T00:00:41.804 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.799+0000 7f0b3d88c640 1 --2- 172.21.15.174:0/3309232863 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b381473c0 0x7f0b38144860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:41.804 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.799+0000 7f0b3d08b640 1 --2- 172.21.15.174:0/3309232863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b380ac870 0x7f0b381505c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:41.805 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.799+0000 7f0b3c88a640 1 -- 172.21.15.174:0/3309232863 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b381473c0 msgr2=0x7f0b38144860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:41.805 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.799+0000 7f0b3c88a640 1 --2- 172.21.15.174:0/3309232863 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b381473c0 0x7f0b38144860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:41.805 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.799+0000 7f0b3c88a640 1 -- 172.21.15.174:0/3309232863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b380ac870 msgr2=0x7f0b381505c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:41.805 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.799+0000 7f0b3c88a640 1 --2- 172.21.15.174:0/3309232863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b380ac870 0x7f0b381505c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:41.805 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.799+0000 7f0b3c88a640 1 -- 172.21.15.174:0/3309232863 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b381478c0 con 0x7f0b3814db70 2024-04-25T00:00:41.805 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.799+0000 7f0b3c88a640 1 --2- 172.21.15.174:0/3309232863 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b3814db70 0x7f0b3815c610 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0b2801a010 tx=0x7f0b2801ac20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:00:41.805 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.799+0000 7f0b2dffb640 1 -- 172.21.15.174:0/3309232863 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b280b0050 con 0x7f0b3814db70 2024-04-25T00:00:41.805 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.799+0000 7f0b2dffb640 1 -- 172.21.15.174:0/3309232863 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b280bc070 con 0x7f0b3814db70 2024-04-25T00:00:41.805 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.799+0000 7f0b3f2ef640 1 -- 172.21.15.174:0/3309232863 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b38071a30 con 0x7f0b3814db70 2024-04-25T00:00:41.805 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.799+0000 7f0b2dffb640 1 -- 172.21.15.174:0/3309232863 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b280b5a70 con 0x7f0b3814db70 2024-04-25T00:00:41.805 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.799+0000 7f0b3f2ef640 1 -- 172.21.15.174:0/3309232863 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b38071ff0 con 0x7f0b3814db70 2024-04-25T00:00:41.805 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.799+0000 7f0b2dffb640 1 -- 172.21.15.174:0/3309232863 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0b280ab4c0 con 0x7f0b3814db70 2024-04-25T00:00:41.805 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.799+0000 7f0b3f2ef640 1 -- 172.21.15.174:0/3309232863 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b3804f510 con 0x7f0b3814db70 2024-04-25T00:00:41.806 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.799+0000 7f0b2dffb640 1 -- 172.21.15.174:0/3309232863 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f0b280b7450 con 0x7f0b3814db70 2024-04-25T00:00:41.806 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.803+0000 7f0b2dffb640 1 -- 172.21.15.174:0/3309232863 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f0b28016090 con 0x7f0b3814db70 2024-04-25T00:00:41.873 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.871+0000 7f0b3f2ef640 1 -- 172.21.15.174:0/3309232863 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "osd new", "uuid": "4f69bd40-5a23-45ff-8693-e1de75e1aa9c", "id": 2} v 0) -- 0x7f0b38068c00 con 0x7f0b3814db70 2024-04-25T00:00:41.881 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.879+0000 7f0b2dffb640 1 -- 172.21.15.174:0/3309232863 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4f69bd40-5a23-45ff-8693-e1de75e1aa9c", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f0b280c1460 con 0x7f0b3814db70 2024-04-25T00:00:41.882 INFO:teuthology.orchestra.run.smithi174.stdout:2 2024-04-25T00:00:41.883 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.879+0000 7f0b3f2ef640 1 -- 172.21.15.174:0/3309232863 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b3814db70 msgr2=0x7f0b3815c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:41.883 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.879+0000 7f0b3f2ef640 1 --2- 172.21.15.174:0/3309232863 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b3814db70 0x7f0b3815c610 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0b2801a010 tx=0x7f0b2801ac20 comp rx=0 tx=0).stop 2024-04-25T00:00:41.883 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.879+0000 7f0b3f2ef640 1 -- 172.21.15.174:0/3309232863 shutdown_connections 2024-04-25T00:00:41.883 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.879+0000 7f0b3f2ef640 1 --2- 172.21.15.174:0/3309232863 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b381473c0 0x7f0b38144860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:41.883 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.879+0000 7f0b3f2ef640 1 --2- 172.21.15.174:0/3309232863 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b3814db70 0x7f0b3815c610 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:41.883 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.879+0000 7f0b3f2ef640 1 --2- 172.21.15.174:0/3309232863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b380ac870 0x7f0b381505c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:41.883 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.879+0000 7f0b3f2ef640 1 -- 172.21.15.174:0/3309232863 >> 172.21.15.174:0/3309232863 conn(0x7f0b380c7a90 msgr2=0x7f0b381489c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:00:41.883 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.879+0000 7f0b3f2ef640 1 -- 172.21.15.174:0/3309232863 shutdown_connections 2024-04-25T00:00:41.883 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:41.879+0000 7f0b3f2ef640 1 -- 172.21.15.174:0/3309232863 wait complete. 2024-04-25T00:00:41.898 DEBUG:teuthology.orchestra.run.smithi174:> sudo ceph --cluster ceph osd new 6a0d5344-8b6f-40d3-918d-0f0a4be688e0 3 2024-04-25T00:00:42.007 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.003+0000 7fc7ba813640 1 Processor -- start 2024-04-25T00:00:42.007 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.003+0000 7fc7ba813640 1 -- start start 2024-04-25T00:00:42.010 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.003+0000 7fc7ba813640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc7b414c6d0 0x7fc7b40c28c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:42.010 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.003+0000 7fc7ba813640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7b4156650 con 0x7fc7b41559f0 2024-04-25T00:00:42.010 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.003+0000 7fc7ba813640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7b406f860 con 0x7fc7b406da80 2024-04-25T00:00:42.010 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.003+0000 7fc7ba813640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fc7b4156930 con 0x7fc7b414c6d0 2024-04-25T00:00:42.011 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.003+0000 7fc7b8db0640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fc7b406da80 0x7fc7b415cd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.174:59150/0 (socket says 172.21.15.174:59150) 2024-04-25T00:00:42.011 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.003+0000 7fc7b8db0640 1 -- 172.21.15.174:0/3280855263 learned_addr learned my addr 172.21.15.174:0/3280855263 (peer_addr_for_me v1:172.21.15.174:0/0) 2024-04-25T00:00:42.011 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.007+0000 7fc7ab7fe640 1 --2- 172.21.15.174:0/3280855263 >> v2:172.21.15.116:3300/0 conn(0x7fc7b414c6d0 0x7fc7b40c28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:42.011 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.007+0000 7fc7b37fe640 1 -- 172.21.15.174:0/3280855263 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3021752497 0 0) 0x7fc7b406f860 con 0x7fc7b406da80 2024-04-25T00:00:42.011 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.007+0000 7fc7b37fe640 1 -- 172.21.15.174:0/3280855263 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc7a0003530 con 0x7fc7b406da80 2024-04-25T00:00:42.011 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.007+0000 7fc7b37fe640 1 -- 172.21.15.174:0/3280855263 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2696089650 0 0) 0x7fc7a0003530 con 0x7fc7b406da80 2024-04-25T00:00:42.011 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.007+0000 7fc7b37fe640 1 -- 172.21.15.174:0/3280855263 >> v2:172.21.15.116:3300/0 conn(0x7fc7b414c6d0 msgr2=0x7fc7b40c28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:00:42.011 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.007+0000 7fc7b37fe640 1 --2- 172.21.15.174:0/3280855263 >> v2:172.21.15.116:3300/0 conn(0x7fc7b414c6d0 0x7fc7b40c28c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:42.011 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.007+0000 7fc7b37fe640 1 -- 172.21.15.174:0/3280855263 >> v1:172.21.15.116:6789/0 conn(0x7fc7b41559f0 legacy=0x7fc7b4155dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:42.011 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.007+0000 7fc7b37fe640 1 -- 172.21.15.174:0/3280855263 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7b406fb40 con 0x7fc7b406da80 2024-04-25T00:00:42.011 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.007+0000 7fc7b37fe640 1 -- 172.21.15.174:0/3280855263 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7fc7ac001a00 con 0x7fc7b406da80 2024-04-25T00:00:42.011 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.007+0000 7fc7b37fe640 1 -- 172.21.15.174:0/3280855263 >> v1:172.21.15.174:6789/0 conn(0x7fc7b406da80 legacy=0x7fc7b415cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:42.011 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.007+0000 7fc7b37fe640 1 --2- 172.21.15.174:0/3280855263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7a00039c0 0x7fc7a0003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:42.011 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.007+0000 7fc7b37fe640 1 --2- 172.21.15.174:0/3280855263 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7a00043c0 0x7fc7a0004890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:42.012 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.007+0000 7fc7b37fe640 1 --2- 172.21.15.174:0/3280855263 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc7a0005130 0x7fc7a000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:42.012 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.007+0000 7fc7b37fe640 1 -- 172.21.15.174:0/3280855263 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc7b4156930 con 0x7fc7a00039c0 2024-04-25T00:00:42.012 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.007+0000 7fc7b37fe640 1 -- 172.21.15.174:0/3280855263 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc7a000c070 con 0x7fc7a00043c0 2024-04-25T00:00:42.012 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.007+0000 7fc7b37fe640 1 -- 172.21.15.174:0/3280855263 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fc7a000c270 con 0x7fc7a0005130 2024-04-25T00:00:42.012 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.007+0000 7fc7ab7fe640 1 --2- 172.21.15.174:0/3280855263 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7a00043c0 0x7fc7a0004890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:42.012 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.007+0000 7fc7b8db0640 1 --2- 172.21.15.174:0/3280855263 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc7a0005130 0x7fc7a000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:42.012 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.007+0000 7fc7b3fff640 1 --2- 172.21.15.174:0/3280855263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7a00039c0 0x7fc7a0003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:42.012 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.007+0000 7fc7ab7fe640 1 -- 172.21.15.174:0/3280855263 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc7a0005130 msgr2=0x7fc7a000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:00:42.012 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.007+0000 7fc7ab7fe640 1 --2- 172.21.15.174:0/3280855263 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc7a0005130 0x7fc7a000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:42.012 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.007+0000 7fc7ab7fe640 1 -- 172.21.15.174:0/3280855263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7a00039c0 msgr2=0x7fc7a0003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:00:42.012 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.007+0000 7fc7ab7fe640 1 --2- 172.21.15.174:0/3280855263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7a00039c0 0x7fc7a0003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:42.012 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.007+0000 7fc7ab7fe640 1 -- 172.21.15.174:0/3280855263 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7a0003530 con 0x7fc7a00043c0 2024-04-25T00:00:42.012 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.007+0000 7fc7b8db0640 1 -- 172.21.15.174:0/3280855263 reap_dead start 2024-04-25T00:00:42.016 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.007+0000 7fc7ab7fe640 1 --2- 172.21.15.174:0/3280855263 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7a00043c0 0x7fc7a0004890 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc7b40c5620 tx=0x7fc79c01c9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=988f3388e0c668a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:00:42.016 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.007+0000 7fc7b37fe640 1 -- 172.21.15.174:0/3280855263 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc79c009690 con 0x7fc7a00043c0 2024-04-25T00:00:42.016 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.007+0000 7fc7b37fe640 1 -- 172.21.15.174:0/3280855263 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc79c0a64e0 con 0x7fc7a00043c0 2024-04-25T00:00:42.016 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.007+0000 7fc7b37fe640 1 -- 172.21.15.174:0/3280855263 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc79c0a68b0 con 0x7fc7a00043c0 2024-04-25T00:00:42.016 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.007+0000 7fc7ba813640 1 -- 172.21.15.174:0/3280855263 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7a00043c0 msgr2=0x7fc7a0004890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:42.016 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.007+0000 7fc7ba813640 1 --2- 172.21.15.174:0/3280855263 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7a00043c0 0x7fc7a0004890 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc7b40c5620 tx=0x7fc79c01c9f0 comp rx=0 tx=0).stop 2024-04-25T00:00:42.016 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.011+0000 7fc7ba813640 1 -- 172.21.15.174:0/3280855263 shutdown_connections 2024-04-25T00:00:42.016 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.011+0000 7fc7ba813640 1 --2- 172.21.15.174:0/3280855263 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7a00043c0 0x7fc7a0004890 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:42.016 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.011+0000 7fc7ba813640 1 -- 172.21.15.174:0/3280855263 >> 172.21.15.174:0/3280855263 conn(0x7fc7b40c49c0 msgr2=0x7fc7b40c4dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:00:42.016 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.011+0000 7fc7ba813640 1 -- 172.21.15.174:0/3280855263 shutdown_connections 2024-04-25T00:00:42.017 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.011+0000 7fc7ba813640 1 -- 172.21.15.174:0/3280855263 wait complete. 2024-04-25T00:00:42.017 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.011+0000 7fc7ba813640 1 Processor -- start 2024-04-25T00:00:42.017 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.011+0000 7fc7ba813640 1 -- start start 2024-04-25T00:00:42.017 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.011+0000 7fc7ba813640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc7a00043c0 0x7fc7b4155110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:42.017 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.011+0000 7fc7ba813640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7b4154760 0x7fc7b41447c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:42.017 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.011+0000 7fc7ba813640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7b41484e0 0x7fc7b4144d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:42.017 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.011+0000 7fc7ba813640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc7b40c30d0 con 0x7fc7b4154760 2024-04-25T00:00:42.017 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.011+0000 7fc7ba813640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc7b41563f0 con 0x7fc7b41484e0 2024-04-25T00:00:42.017 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.011+0000 7fc7ba813640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fc7b40c3250 con 0x7fc7a00043c0 2024-04-25T00:00:42.017 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.011+0000 7fc7b3fff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc7a00043c0 0x7fc7b4155110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:42.017 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.011+0000 7fc7b8db0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7b41484e0 0x7fc7b4144d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:42.017 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.011+0000 7fc7b3fff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc7a00043c0 0x7fc7b4155110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.174:54676/0 (socket says 172.21.15.174:54676) 2024-04-25T00:00:42.017 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.011+0000 7fc7b3fff640 1 -- 172.21.15.174:0/1774202313 learned_addr learned my addr 172.21.15.174:0/1774202313 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-04-25T00:00:42.017 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.011+0000 7fc7ab7fe640 1 --2- 172.21.15.174:0/1774202313 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7b4154760 0x7fc7b41447c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:42.018 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.011+0000 7fc7b8db0640 1 -- 172.21.15.174:0/1774202313 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc7a00043c0 msgr2=0x7fc7b4155110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:42.018 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.011+0000 7fc7b8db0640 1 --2- 172.21.15.174:0/1774202313 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc7a00043c0 0x7fc7b4155110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:42.018 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.011+0000 7fc7b8db0640 1 -- 172.21.15.174:0/1774202313 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7b4154760 msgr2=0x7fc7b41447c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:42.018 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.011+0000 7fc7b8db0640 1 --2- 172.21.15.174:0/1774202313 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7b4154760 0x7fc7b41447c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:42.018 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.011+0000 7fc7b8db0640 1 -- 172.21.15.174:0/1774202313 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7b4153840 con 0x7fc7b41484e0 2024-04-25T00:00:42.018 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.011+0000 7fc7b3fff640 1 --2- 172.21.15.174:0/1774202313 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc7a00043c0 0x7fc7b4155110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:00:42.018 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.011+0000 7fc7b8db0640 1 --2- 172.21.15.174:0/1774202313 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7b41484e0 0x7fc7b4144d00 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc7b40c4920 tx=0x7fc7ac007bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:00:42.018 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.011+0000 7fc7b17fa640 1 -- 172.21.15.174:0/1774202313 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7ac017020 con 0x7fc7b41484e0 2024-04-25T00:00:42.018 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.011+0000 7fc7b17fa640 1 -- 172.21.15.174:0/1774202313 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7ac00ac20 con 0x7fc7b41484e0 2024-04-25T00:00:42.018 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.011+0000 7fc7ba813640 1 -- 172.21.15.174:0/1774202313 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7b4153560 con 0x7fc7b41484e0 2024-04-25T00:00:42.018 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.011+0000 7fc7b17fa640 1 -- 172.21.15.174:0/1774202313 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7ac013610 con 0x7fc7b41484e0 2024-04-25T00:00:42.018 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.011+0000 7fc7ba813640 1 -- 172.21.15.174:0/1774202313 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7b4145420 con 0x7fc7b41484e0 2024-04-25T00:00:42.018 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.011+0000 7fc7b17fa640 1 -- 172.21.15.174:0/1774202313 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc7ac01a020 con 0x7fc7b41484e0 2024-04-25T00:00:42.018 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.011+0000 7fc7b17fa640 1 -- 172.21.15.174:0/1774202313 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7fc7ac024350 con 0x7fc7b41484e0 2024-04-25T00:00:42.019 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.011+0000 7fc7ba813640 1 -- 172.21.15.174:0/1774202313 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7b404f510 con 0x7fc7b41484e0 2024-04-25T00:00:42.019 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.015+0000 7fc7b17fa640 1 -- 172.21.15.174:0/1774202313 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fc7ac021020 con 0x7fc7b41484e0 2024-04-25T00:00:42.083 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.079+0000 7fc7ba813640 1 -- 172.21.15.174:0/1774202313 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd new", "uuid": "6a0d5344-8b6f-40d3-918d-0f0a4be688e0", "id": 3} v 0) -- 0x7fc7b406e000 con 0x7fc7b41484e0 2024-04-25T00:00:42.090 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.087+0000 7fc7b17fa640 1 -- 172.21.15.174:0/1774202313 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6a0d5344-8b6f-40d3-918d-0f0a4be688e0", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7fc7ac01b540 con 0x7fc7b41484e0 2024-04-25T00:00:42.091 INFO:teuthology.orchestra.run.smithi174.stdout:3 2024-04-25T00:00:42.092 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.087+0000 7fc7ba813640 1 -- 172.21.15.174:0/1774202313 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7b41484e0 msgr2=0x7fc7b4144d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:42.092 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.087+0000 7fc7ba813640 1 --2- 172.21.15.174:0/1774202313 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7b41484e0 0x7fc7b4144d00 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc7b40c4920 tx=0x7fc7ac007bd0 comp rx=0 tx=0).stop 2024-04-25T00:00:42.092 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.087+0000 7fc7ba813640 1 -- 172.21.15.174:0/1774202313 shutdown_connections 2024-04-25T00:00:42.092 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.087+0000 7fc7ba813640 1 --2- 172.21.15.174:0/1774202313 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7b41484e0 0x7fc7b4144d00 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:42.092 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.087+0000 7fc7ba813640 1 --2- 172.21.15.174:0/1774202313 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7b4154760 0x7fc7b41447c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:42.093 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.087+0000 7fc7ba813640 1 --2- 172.21.15.174:0/1774202313 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc7a00043c0 0x7fc7b4155110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:42.093 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.087+0000 7fc7ba813640 1 -- 172.21.15.174:0/1774202313 >> 172.21.15.174:0/1774202313 conn(0x7fc7b40c49c0 msgr2=0x7fc7b40c34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:00:42.093 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.087+0000 7fc7ba813640 1 -- 172.21.15.174:0/1774202313 shutdown_connections 2024-04-25T00:00:42.093 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.087+0000 7fc7ba813640 1 -- 172.21.15.174:0/1774202313 wait complete. 2024-04-25T00:00:42.107 DEBUG:teuthology.orchestra.run.smithi174:> sudo ceph --cluster ceph osd new 132e92e4-b825-441e-965a-ca00533fa86a 4 2024-04-25T00:00:42.129 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-04-25T00:00:42.127+0000 7fc0f5688180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T00:00:42.173 INFO:tasks.ceph.mgr.y.smithi116.stderr:2024-04-25T00:00:42.170+0000 7f494fb44180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T00:00:42.189 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-04-25T00:00:42.187+0000 7fc0f5688180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T00:00:42.219 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.215+0000 7ff142029640 1 Processor -- start 2024-04-25T00:00:42.219 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.215+0000 7ff142029640 1 -- start start 2024-04-25T00:00:42.220 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.215+0000 7ff142029640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff13c0c32a0 0x7ff13c0a94a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:42.220 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.215+0000 7ff142029640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff13c157380 0x7ff13c06fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:42.220 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.215+0000 7ff142029640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff13c0c4030 con 0x7ff13c150d00 2024-04-25T00:00:42.221 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.215+0000 7ff142029640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7ff13c0c3de0 con 0x7ff13c157380 2024-04-25T00:00:42.221 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.215+0000 7ff142029640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff13c0c4310 con 0x7ff13c0c32a0 2024-04-25T00:00:42.221 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.215+0000 7ff133fff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff13c0c32a0 0x7ff13c0a94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:42.221 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.215+0000 7ff13bfff640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff13c157380 0x7ff13c06fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:42.221 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.215+0000 7ff13bfff640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff13c157380 0x7ff13c06fd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:55186/0 (socket says 172.21.15.174:55186) 2024-04-25T00:00:42.221 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.215+0000 7ff133fff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff13c0c32a0 0x7ff13c0a94a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.174:54682/0 (socket says 172.21.15.174:54682) 2024-04-25T00:00:42.221 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.215+0000 7ff13bfff640 1 -- 172.21.15.174:0/3043477449 learned_addr learned my addr 172.21.15.174:0/3043477449 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-04-25T00:00:42.221 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.219+0000 7ff13bfff640 1 -- 172.21.15.174:0/3043477449 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff13c0c32a0 msgr2=0x7ff13c0a94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:00:42.221 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.219+0000 7ff13bfff640 1 --2- 172.21.15.174:0/3043477449 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff13c0c32a0 0x7ff13c0a94a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:42.221 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.219+0000 7ff13bfff640 1 -- 172.21.15.174:0/3043477449 >> v1:172.21.15.116:6789/0 conn(0x7ff13c150d00 legacy=0x7ff13c154000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:42.221 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.219+0000 7ff13bfff640 1 -- 172.21.15.174:0/3043477449 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff13c0c45f0 con 0x7ff13c157380 2024-04-25T00:00:42.221 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.219+0000 7ff133fff640 1 --2- 172.21.15.174:0/3043477449 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff13c0c32a0 0x7ff13c0a94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:00:42.223 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.219+0000 7ff13bfff640 1 --2- 172.21.15.174:0/3043477449 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff13c157380 0x7ff13c06fd90 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff134009e50 tx=0x7ff13401c9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9fc9535235d8c37 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:00:42.223 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.219+0000 7ff13affd640 1 -- 172.21.15.174:0/3043477449 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1340a5660 con 0x7ff13c157380 2024-04-25T00:00:42.223 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.219+0000 7ff13affd640 1 -- 172.21.15.174:0/3043477449 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1340a5800 con 0x7ff13c157380 2024-04-25T00:00:42.223 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.219+0000 7ff13affd640 1 -- 172.21.15.174:0/3043477449 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1340a5bd0 con 0x7ff13c157380 2024-04-25T00:00:42.224 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.219+0000 7ff142029640 1 -- 172.21.15.174:0/3043477449 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff13c157380 msgr2=0x7ff13c06fd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:42.224 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.219+0000 7ff142029640 1 --2- 172.21.15.174:0/3043477449 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff13c157380 0x7ff13c06fd90 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff134009e50 tx=0x7ff13401c9a0 comp rx=0 tx=0).stop 2024-04-25T00:00:42.224 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.219+0000 7ff142029640 1 -- 172.21.15.174:0/3043477449 shutdown_connections 2024-04-25T00:00:42.224 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.219+0000 7ff142029640 1 --2- 172.21.15.174:0/3043477449 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff13c157380 0x7ff13c06fd90 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:42.224 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.219+0000 7ff142029640 1 --2- 172.21.15.174:0/3043477449 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff13c0c32a0 0x7ff13c0a94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:42.224 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.219+0000 7ff142029640 1 -- 172.21.15.174:0/3043477449 >> 172.21.15.174:0/3043477449 conn(0x7ff13c0c49c0 msgr2=0x7ff13c0c4dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:00:42.224 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.219+0000 7ff142029640 1 -- 172.21.15.174:0/3043477449 shutdown_connections 2024-04-25T00:00:42.224 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.219+0000 7ff142029640 1 -- 172.21.15.174:0/3043477449 wait complete. 2024-04-25T00:00:42.225 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.223+0000 7ff142029640 1 Processor -- start 2024-04-25T00:00:42.225 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.223+0000 7ff142029640 1 -- start start 2024-04-25T00:00:42.228 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.223+0000 7ff142029640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff13c0c32a0 0x7ff13c14d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:42.228 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.223+0000 7ff142029640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff13c150d00 0x7ff13c147a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:42.228 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.223+0000 7ff142029640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff13c157380 0x7ff13c147f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:42.228 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.223+0000 7ff142029640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff13c160ac0 con 0x7ff13c157380 2024-04-25T00:00:42.228 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.223+0000 7ff142029640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff13c160940 con 0x7ff13c0c32a0 2024-04-25T00:00:42.228 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.223+0000 7ff142029640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff13c160c40 con 0x7ff13c150d00 2024-04-25T00:00:42.228 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.223+0000 7ff13b7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff13c0c32a0 0x7ff13c14d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:42.228 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.223+0000 7ff133fff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff13c150d00 0x7ff13c147a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:42.228 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.223+0000 7ff13b7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff13c0c32a0 0x7ff13c14d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:55198/0 (socket says 172.21.15.174:55198) 2024-04-25T00:00:42.228 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.223+0000 7ff13b7fe640 1 -- 172.21.15.174:0/3518560870 learned_addr learned my addr 172.21.15.174:0/3518560870 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-04-25T00:00:42.229 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.223+0000 7ff13bfff640 1 --2- 172.21.15.174:0/3518560870 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff13c157380 0x7ff13c147f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:42.229 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.223+0000 7ff133fff640 1 -- 172.21.15.174:0/3518560870 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff13c0c32a0 msgr2=0x7ff13c14d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:42.229 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.223+0000 7ff133fff640 1 --2- 172.21.15.174:0/3518560870 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff13c0c32a0 0x7ff13c14d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:42.229 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.223+0000 7ff133fff640 1 -- 172.21.15.174:0/3518560870 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff13c157380 msgr2=0x7ff13c147f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:42.229 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.223+0000 7ff133fff640 1 --2- 172.21.15.174:0/3518560870 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff13c157380 0x7ff13c147f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:42.229 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.223+0000 7ff133fff640 1 -- 172.21.15.174:0/3518560870 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff13c154e20 con 0x7ff13c150d00 2024-04-25T00:00:42.229 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.223+0000 7ff13bfff640 1 --2- 172.21.15.174:0/3518560870 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff13c157380 0x7ff13c147f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:00:42.229 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.223+0000 7ff133fff640 1 --2- 172.21.15.174:0/3518560870 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff13c150d00 0x7ff13c147a20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff12c010c90 tx=0x7ff12c00b510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:00:42.229 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.223+0000 7ff138ff9640 1 -- 172.21.15.174:0/3518560870 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff12c01e070 con 0x7ff13c150d00 2024-04-25T00:00:42.229 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.223+0000 7ff138ff9640 1 -- 172.21.15.174:0/3518560870 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff12c00be30 con 0x7ff13c150d00 2024-04-25T00:00:42.229 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.223+0000 7ff142029640 1 -- 172.21.15.174:0/3518560870 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff13c1563a0 con 0x7ff13c150d00 2024-04-25T00:00:42.229 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.223+0000 7ff138ff9640 1 -- 172.21.15.174:0/3518560870 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff12c019610 con 0x7ff13c150d00 2024-04-25T00:00:42.229 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.223+0000 7ff142029640 1 -- 172.21.15.174:0/3518560870 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff13c153190 con 0x7ff13c150d00 2024-04-25T00:00:42.229 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.223+0000 7ff138ff9640 1 -- 172.21.15.174:0/3518560870 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff12c021070 con 0x7ff13c150d00 2024-04-25T00:00:42.230 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.223+0000 7ff138ff9640 1 -- 172.21.15.174:0/3518560870 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7ff12c019d50 con 0x7ff13c150d00 2024-04-25T00:00:42.230 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.223+0000 7ff142029640 1 -- 172.21.15.174:0/3518560870 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff13c04f510 con 0x7ff13c150d00 2024-04-25T00:00:42.230 INFO:tasks.ceph.mgr.y.smithi116.stderr:2024-04-25T00:00:42.226+0000 7f494fb44180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T00:00:42.230 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.227+0000 7ff138ff9640 1 -- 172.21.15.174:0/3518560870 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7ff12c028070 con 0x7ff13c150d00 2024-04-25T00:00:42.249 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-04-25T00:00:42.247+0000 7fc0f5688180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T00:00:42.279 INFO:tasks.ceph.mgr.y.smithi116.stderr:2024-04-25T00:00:42.278+0000 7f494fb44180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T00:00:42.294 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.291+0000 7ff142029640 1 -- 172.21.15.174:0/3518560870 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "osd new", "uuid": "132e92e4-b825-441e-965a-ca00533fa86a", "id": 4} v 0) -- 0x7ff13c0c3010 con 0x7ff13c150d00 2024-04-25T00:00:42.300 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.295+0000 7ff138ff9640 1 -- 172.21.15.174:0/3518560870 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "132e92e4-b825-441e-965a-ca00533fa86a", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7ff12c019880 con 0x7ff13c150d00 2024-04-25T00:00:42.300 INFO:teuthology.orchestra.run.smithi174.stdout:4 2024-04-25T00:00:42.301 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.299+0000 7ff142029640 1 -- 172.21.15.174:0/3518560870 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff13c150d00 msgr2=0x7ff13c147a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:42.301 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.299+0000 7ff142029640 1 --2- 172.21.15.174:0/3518560870 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff13c150d00 0x7ff13c147a20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff12c010c90 tx=0x7ff12c00b510 comp rx=0 tx=0).stop 2024-04-25T00:00:42.301 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.299+0000 7ff142029640 1 -- 172.21.15.174:0/3518560870 shutdown_connections 2024-04-25T00:00:42.301 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.299+0000 7ff142029640 1 --2- 172.21.15.174:0/3518560870 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff13c157380 0x7ff13c147f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:42.302 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.299+0000 7ff142029640 1 --2- 172.21.15.174:0/3518560870 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff13c150d00 0x7ff13c147a20 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:42.302 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.299+0000 7ff142029640 1 --2- 172.21.15.174:0/3518560870 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff13c0c32a0 0x7ff13c14d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:42.302 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.299+0000 7ff142029640 1 -- 172.21.15.174:0/3518560870 >> 172.21.15.174:0/3518560870 conn(0x7ff13c0c49c0 msgr2=0x7ff13c15d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:00:42.302 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.299+0000 7ff142029640 1 -- 172.21.15.174:0/3518560870 shutdown_connections 2024-04-25T00:00:42.302 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.299+0000 7ff142029640 1 -- 172.21.15.174:0/3518560870 wait complete. 2024-04-25T00:00:42.316 DEBUG:teuthology.orchestra.run.smithi174:> sudo ceph --cluster ceph osd new a2eeb5b6-1b31-43cc-8306-a0adf3ee15bf 5 2024-04-25T00:00:42.329 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-04-25T00:00:42.327+0000 7fc0f5688180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T00:00:42.356 INFO:tasks.ceph.mgr.y.smithi116.stderr:2024-04-25T00:00:42.354+0000 7f494fb44180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T00:00:42.383 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-04-25T00:00:42.379+0000 7fc0f5688180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T00:00:42.407 INFO:tasks.ceph.mgr.y.smithi116.stderr:2024-04-25T00:00:42.406+0000 7f494fb44180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T00:00:42.449 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.447+0000 7fc1cf3d0640 1 Processor -- start 2024-04-25T00:00:42.449 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.447+0000 7fc1cf3d0640 1 -- start start 2024-04-25T00:00:42.450 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.447+0000 7fc1cf3d0640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fc1c8072ff0 0x7fc1c81531c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:42.450 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.447+0000 7fc1cf3d0640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1c806b210 0x7fc1c8158e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:42.450 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.447+0000 7fc1cf3d0640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1c806c930 con 0x7fc1c806e930 2024-04-25T00:00:42.450 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.447+0000 7fc1cf3d0640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fc1c806b660 con 0x7fc1c8072ff0 2024-04-25T00:00:42.450 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.447+0000 7fc1cf3d0640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fc1c8071e90 con 0x7fc1c806b210 2024-04-25T00:00:42.450 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.447+0000 7fc1cc96b640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fc1c8072ff0 0x7fc1c81531c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:42.450 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.447+0000 7fc1cd96d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1c806b210 0x7fc1c8158e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:42.450 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.447+0000 7fc1cd16c640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fc1c806e930 0x7fc1c8152ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.174:59154/0 (socket says 172.21.15.174:59154) 2024-04-25T00:00:42.450 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.447+0000 7fc1cd16c640 1 -- 172.21.15.174:0/1377217674 learned_addr learned my addr 172.21.15.174:0/1377217674 (peer_addr_for_me v1:172.21.15.174:0/0) 2024-04-25T00:00:42.451 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.447+0000 7fc1cc96b640 1 -- 172.21.15.174:0/1377217674 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1c806b210 msgr2=0x7fc1c8158e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:00:42.451 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.447+0000 7fc1cc96b640 1 --2- 172.21.15.174:0/1377217674 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1c806b210 0x7fc1c8158e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:42.451 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.447+0000 7fc1cc96b640 1 -- 172.21.15.174:0/1377217674 >> v1:172.21.15.174:6789/0 conn(0x7fc1c806e930 legacy=0x7fc1c8152ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:42.451 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.447+0000 7fc1bffff640 1 -- 172.21.15.174:0/1377217674 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2027449242 0 0) 0x7fc1c806c930 con 0x7fc1c806e930 2024-04-25T00:00:42.451 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.447+0000 7fc1cc96b640 1 -- 172.21.15.174:0/1377217674 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1c8072170 con 0x7fc1c8072ff0 2024-04-25T00:00:42.451 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.447+0000 7fc1cd96d640 1 --2- 172.21.15.174:0/1377217674 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1c806b210 0x7fc1c8158e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:00:42.452 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.447+0000 7fc1cc96b640 1 --2- 172.21.15.174:0/1377217674 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1c8072ff0 0x7fc1c81531c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc1b80097c0 tx=0x7fc1b801c3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31ca4d0169e2bd4e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:00:42.452 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.451+0000 7fc1bffff640 1 -- 172.21.15.174:0/1377217674 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1b80b1070 con 0x7fc1c8072ff0 2024-04-25T00:00:42.452 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.451+0000 7fc1bffff640 1 -- 172.21.15.174:0/1377217674 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1b80ac410 con 0x7fc1c8072ff0 2024-04-25T00:00:42.452 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.451+0000 7fc1bffff640 1 -- 172.21.15.174:0/1377217674 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1b80ac7e0 con 0x7fc1c8072ff0 2024-04-25T00:00:42.453 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.451+0000 7fc1cf3d0640 1 -- 172.21.15.174:0/1377217674 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1c8072ff0 msgr2=0x7fc1c81531c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:42.453 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.451+0000 7fc1cf3d0640 1 --2- 172.21.15.174:0/1377217674 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1c8072ff0 0x7fc1c81531c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc1b80097c0 tx=0x7fc1b801c3b0 comp rx=0 tx=0).stop 2024-04-25T00:00:42.453 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.451+0000 7fc1cf3d0640 1 -- 172.21.15.174:0/1377217674 shutdown_connections 2024-04-25T00:00:42.453 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.451+0000 7fc1cf3d0640 1 --2- 172.21.15.174:0/1377217674 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1c806b210 0x7fc1c8158e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:42.454 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.451+0000 7fc1cf3d0640 1 --2- 172.21.15.174:0/1377217674 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1c8072ff0 0x7fc1c81531c0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:42.454 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.451+0000 7fc1cf3d0640 1 -- 172.21.15.174:0/1377217674 >> 172.21.15.174:0/1377217674 conn(0x7fc1c80c7a90 msgr2=0x7fc1c80c7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:00:42.454 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.451+0000 7fc1cf3d0640 1 -- 172.21.15.174:0/1377217674 shutdown_connections 2024-04-25T00:00:42.454 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.451+0000 7fc1cf3d0640 1 -- 172.21.15.174:0/1377217674 wait complete. 2024-04-25T00:00:42.454 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.451+0000 7fc1cf3d0640 1 Processor -- start 2024-04-25T00:00:42.454 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.451+0000 7fc1cf3d0640 1 -- start start 2024-04-25T00:00:42.456 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.451+0000 7fc1cf3d0640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1c806b210 0x7fc1c81479b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:42.457 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.451+0000 7fc1cf3d0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1c806e930 0x7fc1c8146740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:42.457 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.451+0000 7fc1cf3d0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1c8072ff0 0x7fc1c81ed810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:42.457 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.451+0000 7fc1cf3d0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc1c806ee90 con 0x7fc1c8072ff0 2024-04-25T00:00:42.457 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.451+0000 7fc1cf3d0640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc1c806ed10 con 0x7fc1c806e930 2024-04-25T00:00:42.457 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.451+0000 7fc1cf3d0640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fc1c8162170 con 0x7fc1c806b210 2024-04-25T00:00:42.457 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.451+0000 7fc1cd16c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1c806b210 0x7fc1c81479b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:42.457 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.451+0000 7fc1cc96b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1c806e930 0x7fc1c8146740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:42.457 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.451+0000 7fc1cd16c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1c806b210 0x7fc1c81479b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.174:54712/0 (socket says 172.21.15.174:54712) 2024-04-25T00:00:42.457 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.451+0000 7fc1cd16c640 1 -- 172.21.15.174:0/634774803 learned_addr learned my addr 172.21.15.174:0/634774803 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-04-25T00:00:42.457 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.451+0000 7fc1cc96b640 1 -- 172.21.15.174:0/634774803 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1c806b210 msgr2=0x7fc1c81479b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:42.457 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.451+0000 7fc1cc96b640 1 --2- 172.21.15.174:0/634774803 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1c806b210 0x7fc1c81479b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:42.457 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.451+0000 7fc1cc96b640 1 -- 172.21.15.174:0/634774803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1c8072ff0 msgr2=0x7fc1c81ed810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:42.458 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.451+0000 7fc1cc96b640 1 --2- 172.21.15.174:0/634774803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1c8072ff0 0x7fc1c81ed810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:42.458 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.451+0000 7fc1cc96b640 1 -- 172.21.15.174:0/634774803 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1c81f1a10 con 0x7fc1c806e930 2024-04-25T00:00:42.458 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.451+0000 7fc1cd16c640 1 --2- 172.21.15.174:0/634774803 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1c806b210 0x7fc1c81479b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:00:42.458 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.451+0000 7fc1cc96b640 1 --2- 172.21.15.174:0/634774803 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1c806e930 0x7fc1c8146740 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fc1b80ac000 tx=0x7fc1b8019150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:00:42.458 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.451+0000 7fc1bdffb640 1 -- 172.21.15.174:0/634774803 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1b80b1040 con 0x7fc1c806e930 2024-04-25T00:00:42.458 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.451+0000 7fc1bdffb640 1 -- 172.21.15.174:0/634774803 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1b80acdb0 con 0x7fc1c806e930 2024-04-25T00:00:42.458 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.451+0000 7fc1cf3d0640 1 -- 172.21.15.174:0/634774803 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1c80c53b0 con 0x7fc1c806e930 2024-04-25T00:00:42.458 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.451+0000 7fc1bdffb640 1 -- 172.21.15.174:0/634774803 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1b8006e20 con 0x7fc1c806e930 2024-04-25T00:00:42.458 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.451+0000 7fc1cf3d0640 1 -- 172.21.15.174:0/634774803 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1c8146c80 con 0x7fc1c806e930 2024-04-25T00:00:42.459 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.455+0000 7fc1cf3d0640 1 -- 172.21.15.174:0/634774803 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1c81f17a0 con 0x7fc1c806e930 2024-04-25T00:00:42.459 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.455+0000 7fc1bdffb640 1 -- 172.21.15.174:0/634774803 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc1b8003620 con 0x7fc1c806e930 2024-04-25T00:00:42.459 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.455+0000 7fc1bdffb640 1 -- 172.21.15.174:0/634774803 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7fc1b80b5b70 con 0x7fc1c806e930 2024-04-25T00:00:42.459 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.455+0000 7fc1bdffb640 1 -- 172.21.15.174:0/634774803 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fc1b80c2030 con 0x7fc1c806e930 2024-04-25T00:00:42.463 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-04-25T00:00:42.459+0000 7fc0f5688180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T00:00:42.484 INFO:tasks.ceph.mgr.y.smithi116.stderr:2024-04-25T00:00:42.482+0000 7f494fb44180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T00:00:42.517 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-04-25T00:00:42.515+0000 7fc0f5688180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T00:00:42.527 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.523+0000 7fc1cf3d0640 1 -- 172.21.15.174:0/634774803 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd new", "uuid": "a2eeb5b6-1b31-43cc-8306-a0adf3ee15bf", "id": 5} v 0) -- 0x7fc1c804f510 con 0x7fc1c806e930 2024-04-25T00:00:42.533 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.531+0000 7fc1bdffb640 1 -- 172.21.15.174:0/634774803 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a2eeb5b6-1b31-43cc-8306-a0adf3ee15bf", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7fc1b80b9030 con 0x7fc1c806e930 2024-04-25T00:00:42.533 INFO:teuthology.orchestra.run.smithi174.stdout:5 2024-04-25T00:00:42.535 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.531+0000 7fc1cf3d0640 1 -- 172.21.15.174:0/634774803 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1c806e930 msgr2=0x7fc1c8146740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:42.535 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.531+0000 7fc1cf3d0640 1 --2- 172.21.15.174:0/634774803 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1c806e930 0x7fc1c8146740 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fc1b80ac000 tx=0x7fc1b8019150 comp rx=0 tx=0).stop 2024-04-25T00:00:42.535 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.531+0000 7fc1cf3d0640 1 -- 172.21.15.174:0/634774803 shutdown_connections 2024-04-25T00:00:42.535 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.531+0000 7fc1cf3d0640 1 --2- 172.21.15.174:0/634774803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1c8072ff0 0x7fc1c81ed810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:42.535 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.531+0000 7fc1cf3d0640 1 --2- 172.21.15.174:0/634774803 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1c806e930 0x7fc1c8146740 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:42.535 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.531+0000 7fc1cf3d0640 1 --2- 172.21.15.174:0/634774803 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1c806b210 0x7fc1c81479b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:42.536 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.531+0000 7fc1cf3d0640 1 -- 172.21.15.174:0/634774803 >> 172.21.15.174:0/634774803 conn(0x7fc1c80c7a90 msgr2=0x7fc1c80c7e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:00:42.536 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.531+0000 7fc1cf3d0640 1 -- 172.21.15.174:0/634774803 shutdown_connections 2024-04-25T00:00:42.536 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.531+0000 7fc1cf3d0640 1 -- 172.21.15.174:0/634774803 wait complete. 2024-04-25T00:00:42.537 INFO:tasks.ceph.mgr.y.smithi116.stderr:2024-04-25T00:00:42.534+0000 7f494fb44180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T00:00:42.555 DEBUG:teuthology.orchestra.run.smithi174:> sudo ceph --cluster ceph osd new 588e873a-c841-4ba7-90f2-8956c87a9983 6 2024-04-25T00:00:42.672 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.667+0000 7f4770809640 1 Processor -- start 2024-04-25T00:00:42.672 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.667+0000 7f4770809640 1 -- start start 2024-04-25T00:00:42.673 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.671+0000 7f4770809640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f476c151260 0x7f476c157a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:42.673 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.671+0000 7f4770809640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f476c0c8740 0x7f476c0c7040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:42.673 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.671+0000 7f4770809640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f476c153cb0 con 0x7f476c0c7580 2024-04-25T00:00:42.673 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.671+0000 7f4770809640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f476c1548c0 con 0x7f476c0c8740 2024-04-25T00:00:42.673 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.671+0000 7f4770809640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f476c153f90 con 0x7f476c151260 2024-04-25T00:00:42.674 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.671+0000 7f476a59c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f476c151260 0x7f476c157a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:42.674 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.671+0000 7f476ad9d640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f476c0c7580 0x7f476c0bf130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.174:59166/0 (socket says 172.21.15.174:59166) 2024-04-25T00:00:42.674 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.671+0000 7f476ad9d640 1 -- 172.21.15.174:0/3875885412 learned_addr learned my addr 172.21.15.174:0/3875885412 (peer_addr_for_me v1:172.21.15.174:0/0) 2024-04-25T00:00:42.674 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.671+0000 7f4769d9b640 1 --2- 172.21.15.174:0/3875885412 >> v2:172.21.15.174:3300/0 conn(0x7f476c0c8740 0x7f476c0c7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:42.674 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.671+0000 7f4769d9b640 1 -- 172.21.15.174:0/3875885412 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f476c151260 msgr2=0x7f476c157a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:00:42.674 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.671+0000 7f4769d9b640 1 --2- 172.21.15.174:0/3875885412 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f476c151260 0x7f476c157a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:42.674 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.671+0000 7f4769d9b640 1 -- 172.21.15.174:0/3875885412 >> v1:172.21.15.174:6789/0 conn(0x7f476c0c7580 legacy=0x7f476c0bf130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:42.674 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.671+0000 7f4769d9b640 1 -- 172.21.15.174:0/3875885412 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f476c154270 con 0x7f476c0c8740 2024-04-25T00:00:42.674 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.671+0000 7f476a59c640 1 --2- 172.21.15.174:0/3875885412 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f476c151260 0x7f476c157a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:00:42.675 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.671+0000 7f4769d9b640 1 --2- 172.21.15.174:0/3875885412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f476c0c8740 0x7f476c0c7040 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f47600097c0 tx=0x7f476001c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a0315c5990892fe1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:00:42.676 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.671+0000 7f476959a640 1 -- 172.21.15.174:0/3875885412 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47600b0070 con 0x7f476c0c8740 2024-04-25T00:00:42.676 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.671+0000 7f476959a640 1 -- 172.21.15.174:0/3875885412 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47600ab600 con 0x7f476c0c8740 2024-04-25T00:00:42.676 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.671+0000 7f476959a640 1 -- 172.21.15.174:0/3875885412 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47600ab9d0 con 0x7f476c0c8740 2024-04-25T00:00:42.677 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.671+0000 7f4770809640 1 -- 172.21.15.174:0/3875885412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f476c0c8740 msgr2=0x7f476c0c7040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:42.677 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.671+0000 7f4770809640 1 --2- 172.21.15.174:0/3875885412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f476c0c8740 0x7f476c0c7040 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f47600097c0 tx=0x7f476001c4a0 comp rx=0 tx=0).stop 2024-04-25T00:00:42.677 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.671+0000 7f4770809640 1 -- 172.21.15.174:0/3875885412 shutdown_connections 2024-04-25T00:00:42.677 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.671+0000 7f4770809640 1 --2- 172.21.15.174:0/3875885412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f476c0c8740 0x7f476c0c7040 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:42.677 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.671+0000 7f4770809640 1 --2- 172.21.15.174:0/3875885412 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f476c151260 0x7f476c157a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:42.677 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.671+0000 7f4770809640 1 -- 172.21.15.174:0/3875885412 >> 172.21.15.174:0/3875885412 conn(0x7f476c0c8f90 msgr2=0x7f476c0c93a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:00:42.677 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.671+0000 7f4770809640 1 -- 172.21.15.174:0/3875885412 shutdown_connections 2024-04-25T00:00:42.677 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.671+0000 7f4770809640 1 -- 172.21.15.174:0/3875885412 wait complete. 2024-04-25T00:00:42.679 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.675+0000 7f4770809640 1 Processor -- start 2024-04-25T00:00:42.680 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.675+0000 7f4770809640 1 -- start start 2024-04-25T00:00:42.680 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.675+0000 7f4770809640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f476c0c7580 0x7f476c15cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:42.680 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.675+0000 7f4770809640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f476c0c8740 0x7f476c06c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:42.680 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.675+0000 7f4770809640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f476c151260 0x7f476c06cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:42.680 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.675+0000 7f4770809640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f476c0c5cc0 con 0x7f476c151260 2024-04-25T00:00:42.680 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.675+0000 7f4770809640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f476c0c5b40 con 0x7f476c0c8740 2024-04-25T00:00:42.680 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.675+0000 7f4770809640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f476c0c5e40 con 0x7f476c0c7580 2024-04-25T00:00:42.680 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.675+0000 7f476a59c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f476c0c7580 0x7f476c15cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:42.680 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.675+0000 7f4769d9b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f476c0c8740 0x7f476c06c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:42.680 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.675+0000 7f476a59c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f476c0c7580 0x7f476c15cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.174:54716/0 (socket says 172.21.15.174:54716) 2024-04-25T00:00:42.680 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.675+0000 7f476a59c640 1 -- 172.21.15.174:0/3131085362 learned_addr learned my addr 172.21.15.174:0/3131085362 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-04-25T00:00:42.680 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.675+0000 7f4769d9b640 1 -- 172.21.15.174:0/3131085362 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f476c0c7580 msgr2=0x7f476c15cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:42.680 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.675+0000 7f4769d9b640 1 --2- 172.21.15.174:0/3131085362 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f476c0c7580 0x7f476c15cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:42.680 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.675+0000 7f4769d9b640 1 -- 172.21.15.174:0/3131085362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f476c151260 msgr2=0x7f476c06cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:42.681 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.675+0000 7f4769d9b640 1 --2- 172.21.15.174:0/3131085362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f476c151260 0x7f476c06cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:42.681 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.675+0000 7f4769d9b640 1 -- 172.21.15.174:0/3131085362 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f476c146160 con 0x7f476c0c8740 2024-04-25T00:00:42.681 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.675+0000 7f476a59c640 1 --2- 172.21.15.174:0/3131085362 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f476c0c7580 0x7f476c15cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:00:42.681 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.675+0000 7f4769d9b640 1 --2- 172.21.15.174:0/3131085362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f476c0c8740 0x7f476c06c9d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f47600ab000 tx=0x7f4760019240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:00:42.681 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.675+0000 7f475affd640 1 -- 172.21.15.174:0/3131085362 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47600b8020 con 0x7f476c0c8740 2024-04-25T00:00:42.681 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.675+0000 7f475affd640 1 -- 172.21.15.174:0/3131085362 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47600b4410 con 0x7f476c0c8740 2024-04-25T00:00:42.681 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.675+0000 7f4770809640 1 -- 172.21.15.174:0/3131085362 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f476c146400 con 0x7f476c0c8740 2024-04-25T00:00:42.681 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.675+0000 7f475affd640 1 -- 172.21.15.174:0/3131085362 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47600b0040 con 0x7f476c0c8740 2024-04-25T00:00:42.681 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.675+0000 7f4770809640 1 -- 172.21.15.174:0/3131085362 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f476c15c480 con 0x7f476c0c8740 2024-04-25T00:00:42.681 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.675+0000 7f475affd640 1 -- 172.21.15.174:0/3131085362 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f47600bb020 con 0x7f476c0c8740 2024-04-25T00:00:42.681 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.675+0000 7f475affd640 1 -- 172.21.15.174:0/3131085362 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f47600bc3b0 con 0x7f476c0c8740 2024-04-25T00:00:42.681 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.675+0000 7f4770809640 1 -- 172.21.15.174:0/3131085362 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f476c04f510 con 0x7f476c0c8740 2024-04-25T00:00:42.682 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-04-25T00:00:42.679+0000 7fc0f5688180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T00:00:42.683 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.679+0000 7f475affd640 1 -- 172.21.15.174:0/3131085362 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f47600c2030 con 0x7f476c0c8740 2024-04-25T00:00:42.686 INFO:tasks.ceph.mgr.y.smithi116.stderr:2024-04-25T00:00:42.682+0000 7f494fb44180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T00:00:42.746 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-04-25T00:00:42.743+0000 7fc0f5688180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T00:00:42.747 INFO:tasks.ceph.mgr.y.smithi116.stderr:2024-04-25T00:00:42.746+0000 7f494fb44180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T00:00:42.750 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.747+0000 7f4770809640 1 -- 172.21.15.174:0/3131085362 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd new", "uuid": "588e873a-c841-4ba7-90f2-8956c87a9983", "id": 6} v 0) -- 0x7f476c0ad390 con 0x7f476c0c8740 2024-04-25T00:00:42.762 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.759+0000 7f475affd640 1 -- 172.21.15.174:0/3131085362 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "588e873a-c841-4ba7-90f2-8956c87a9983", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7f4760016090 con 0x7f476c0c8740 2024-04-25T00:00:42.762 INFO:teuthology.orchestra.run.smithi174.stdout:6 2024-04-25T00:00:42.764 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.759+0000 7f4770809640 1 -- 172.21.15.174:0/3131085362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f476c0c8740 msgr2=0x7f476c06c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:42.764 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.759+0000 7f4770809640 1 --2- 172.21.15.174:0/3131085362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f476c0c8740 0x7f476c06c9d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f47600ab000 tx=0x7f4760019240 comp rx=0 tx=0).stop 2024-04-25T00:00:42.764 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.759+0000 7f4770809640 1 -- 172.21.15.174:0/3131085362 shutdown_connections 2024-04-25T00:00:42.764 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.759+0000 7f4770809640 1 --2- 172.21.15.174:0/3131085362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f476c151260 0x7f476c06cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:42.764 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.759+0000 7f4770809640 1 --2- 172.21.15.174:0/3131085362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f476c0c8740 0x7f476c06c9d0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:42.764 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.759+0000 7f4770809640 1 --2- 172.21.15.174:0/3131085362 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f476c0c7580 0x7f476c15cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:42.764 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.759+0000 7f4770809640 1 -- 172.21.15.174:0/3131085362 >> 172.21.15.174:0/3131085362 conn(0x7f476c0c8f90 msgr2=0x7f476c152c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:00:42.764 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.759+0000 7f4770809640 1 -- 172.21.15.174:0/3131085362 shutdown_connections 2024-04-25T00:00:42.764 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.759+0000 7f4770809640 1 -- 172.21.15.174:0/3131085362 wait complete. 2024-04-25T00:00:42.779 DEBUG:teuthology.orchestra.run.smithi174:> sudo ceph --cluster ceph osd new 3b90f9b0-8a0e-4883-b088-ca1d4012870c 7 2024-04-25T00:00:42.897 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.895+0000 7fd05dc66640 1 Processor -- start 2024-04-25T00:00:42.897 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.895+0000 7fd05dc66640 1 -- start start 2024-04-25T00:00:42.898 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.895+0000 7fd05dc66640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fd0580c41c0 0x7fd058154020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:42.899 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.895+0000 7fd05dc66640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd0580a9b10 0x7fd0580a9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:42.899 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.895+0000 7fd05dc66640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd058156d50 con 0x7fd05806fff0 2024-04-25T00:00:42.899 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.895+0000 7fd05dc66640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fd058157960 con 0x7fd0580c41c0 2024-04-25T00:00:42.899 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.895+0000 7fd05dc66640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fd058157030 con 0x7fd0580a9b10 2024-04-25T00:00:42.899 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.895+0000 7fd056ffd640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fd0580c41c0 0x7fd058154020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:42.899 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.895+0000 7fd056ffd640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fd0580c41c0 0x7fd058154020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:55236/0 (socket says 172.21.15.174:55236) 2024-04-25T00:00:42.899 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.895+0000 7fd056ffd640 1 -- 172.21.15.174:0/2149194455 learned_addr learned my addr 172.21.15.174:0/2149194455 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-04-25T00:00:42.899 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.895+0000 7fd057fff640 1 --2- 172.21.15.174:0/2149194455 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd0580a9b10 0x7fd0580a9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:42.899 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.895+0000 7fd056ffd640 1 -- 172.21.15.174:0/2149194455 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd0580a9b10 msgr2=0x7fd0580a9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:00:42.899 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.895+0000 7fd056ffd640 1 --2- 172.21.15.174:0/2149194455 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd0580a9b10 0x7fd0580a9fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:42.899 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.895+0000 7fd056ffd640 1 -- 172.21.15.174:0/2149194455 >> v1:172.21.15.116:6789/0 conn(0x7fd05806fff0 legacy=0x7fd058150660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:00:42.899 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.895+0000 7fd056ffd640 1 -- 172.21.15.174:0/2149194455 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd058157310 con 0x7fd0580c41c0 2024-04-25T00:00:42.900 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.899+0000 7fd056ffd640 1 --2- 172.21.15.174:0/2149194455 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0580c41c0 0x7fd058154020 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd040009870 tx=0x7fd04001c520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=632cd2a19abf3528 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:00:42.901 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.899+0000 7fd0567fc640 1 -- 172.21.15.174:0/2149194455 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0400ab5f0 con 0x7fd0580c41c0 2024-04-25T00:00:42.901 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.899+0000 7fd0567fc640 1 -- 172.21.15.174:0/2149194455 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0400b0070 con 0x7fd0580c41c0 2024-04-25T00:00:42.901 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.899+0000 7fd0567fc640 1 -- 172.21.15.174:0/2149194455 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0400ab9b0 con 0x7fd0580c41c0 2024-04-25T00:00:42.901 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.899+0000 7fd05dc66640 1 -- 172.21.15.174:0/2149194455 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0580c41c0 msgr2=0x7fd058154020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:42.902 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.899+0000 7fd05dc66640 1 --2- 172.21.15.174:0/2149194455 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0580c41c0 0x7fd058154020 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd040009870 tx=0x7fd04001c520 comp rx=0 tx=0).stop 2024-04-25T00:00:42.902 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.899+0000 7fd05dc66640 1 -- 172.21.15.174:0/2149194455 shutdown_connections 2024-04-25T00:00:42.902 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.899+0000 7fd05dc66640 1 --2- 172.21.15.174:0/2149194455 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd0580a9b10 0x7fd0580a9fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:42.902 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.899+0000 7fd05dc66640 1 --2- 172.21.15.174:0/2149194455 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0580c41c0 0x7fd058154020 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:42.902 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.899+0000 7fd05dc66640 1 -- 172.21.15.174:0/2149194455 >> 172.21.15.174:0/2149194455 conn(0x7fd0580c49c0 msgr2=0x7fd0580c4dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:00:42.902 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.899+0000 7fd05dc66640 1 -- 172.21.15.174:0/2149194455 shutdown_connections 2024-04-25T00:00:42.902 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.899+0000 7fd05dc66640 1 -- 172.21.15.174:0/2149194455 wait complete. 2024-04-25T00:00:42.905 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.899+0000 7fd05dc66640 1 Processor -- start 2024-04-25T00:00:42.905 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.899+0000 7fd05dc66640 1 -- start start 2024-04-25T00:00:42.905 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.899+0000 7fd05dc66640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd05806fff0 0x7fd0580c1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:42.905 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.899+0000 7fd05dc66640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd0580a9b10 0x7fd0580acf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:42.905 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.899+0000 7fd05dc66640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0580c41c0 0x7fd0580ad460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:42.905 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.899+0000 7fd05dc66640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd058160740 con 0x7fd05806fff0 2024-04-25T00:00:42.905 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.899+0000 7fd05dc66640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd058150f80 con 0x7fd0580c41c0 2024-04-25T00:00:42.905 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.899+0000 7fd05dc66640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fd0581608c0 con 0x7fd0580a9b10 2024-04-25T00:00:42.905 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.899+0000 7fd056ffd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd0580a9b10 0x7fd0580acf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:42.905 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.899+0000 7fd057fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0580c41c0 0x7fd0580ad460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:42.906 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.899+0000 7fd056ffd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd0580a9b10 0x7fd0580acf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.174:54746/0 (socket says 172.21.15.174:54746) 2024-04-25T00:00:42.906 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.899+0000 7fd056ffd640 1 -- 172.21.15.174:0/683748702 learned_addr learned my addr 172.21.15.174:0/683748702 (peer_addr_for_me v2:172.21.15.174:0/0) 2024-04-25T00:00:42.906 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.899+0000 7fd057fff640 1 --2- 172.21.15.174:0/683748702 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0580c41c0 0x7fd0580ad460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.174:55246/0 (socket says 172.21.15.174:55246) 2024-04-25T00:00:42.906 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.899+0000 7fd057fff640 1 -- 172.21.15.174:0/683748702 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd0580a9b10 msgr2=0x7fd0580acf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:42.906 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.899+0000 7fd057fff640 1 --2- 172.21.15.174:0/683748702 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd0580a9b10 0x7fd0580acf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:42.906 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.899+0000 7fd0577fe640 1 --2- 172.21.15.174:0/683748702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd05806fff0 0x7fd0580c1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:42.906 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.899+0000 7fd057fff640 1 -- 172.21.15.174:0/683748702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd05806fff0 msgr2=0x7fd0580c1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:42.906 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.899+0000 7fd057fff640 1 --2- 172.21.15.174:0/683748702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd05806fff0 0x7fd0580c1550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:42.906 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.899+0000 7fd057fff640 1 -- 172.21.15.174:0/683748702 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0580c7b10 con 0x7fd0580c41c0 2024-04-25T00:00:42.906 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.899+0000 7fd056ffd640 1 --2- 172.21.15.174:0/683748702 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd0580a9b10 0x7fd0580acf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:00:42.906 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.899+0000 7fd057fff640 1 --2- 172.21.15.174:0/683748702 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0580c41c0 0x7fd0580ad460 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd05000efc0 tx=0x7fd05000c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:00:42.906 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.899+0000 7fd04ffff640 1 -- 172.21.15.174:0/683748702 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd050018020 con 0x7fd0580c41c0 2024-04-25T00:00:42.906 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.899+0000 7fd04ffff640 1 -- 172.21.15.174:0/683748702 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd050014480 con 0x7fd0580c41c0 2024-04-25T00:00:42.907 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.899+0000 7fd05dc66640 1 -- 172.21.15.174:0/683748702 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0580ad9a0 con 0x7fd0580c41c0 2024-04-25T00:00:42.907 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.899+0000 7fd04ffff640 1 -- 172.21.15.174:0/683748702 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd050014870 con 0x7fd0580c41c0 2024-04-25T00:00:42.907 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.899+0000 7fd05dc66640 1 -- 172.21.15.174:0/683748702 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd058143080 con 0x7fd0580c41c0 2024-04-25T00:00:42.907 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.903+0000 7fd04ffff640 1 -- 172.21.15.174:0/683748702 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd05001b020 con 0x7fd0580c41c0 2024-04-25T00:00:42.907 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.903+0000 7fd04ffff640 1 -- 172.21.15.174:0/683748702 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7fd050013a40 con 0x7fd0580c41c0 2024-04-25T00:00:42.907 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.903+0000 7fd05dc66640 1 -- 172.21.15.174:0/683748702 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd05804f510 con 0x7fd0580c41c0 2024-04-25T00:00:42.907 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.903+0000 7fd04ffff640 1 -- 172.21.15.174:0/683748702 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fd050022030 con 0x7fd0580c41c0 2024-04-25T00:00:42.978 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.975+0000 7fd05dc66640 1 -- 172.21.15.174:0/683748702 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd new", "uuid": "3b90f9b0-8a0e-4883-b088-ca1d4012870c", "id": 7} v 0) -- 0x7fd058153150 con 0x7fd0580c41c0 2024-04-25T00:00:42.989 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.983+0000 7fd04ffff640 1 -- 172.21.15.174:0/683748702 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3b90f9b0-8a0e-4883-b088-ca1d4012870c", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7fd0580ad9a0 con 0x7fd0580c41c0 2024-04-25T00:00:42.989 INFO:teuthology.orchestra.run.smithi174.stdout:7 2024-04-25T00:00:42.991 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.987+0000 7fd05dc66640 1 -- 172.21.15.174:0/683748702 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0580c41c0 msgr2=0x7fd0580ad460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:42.991 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.987+0000 7fd05dc66640 1 --2- 172.21.15.174:0/683748702 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0580c41c0 0x7fd0580ad460 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd05000efc0 tx=0x7fd05000c490 comp rx=0 tx=0).stop 2024-04-25T00:00:42.991 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.987+0000 7fd05dc66640 1 -- 172.21.15.174:0/683748702 shutdown_connections 2024-04-25T00:00:42.992 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.987+0000 7fd05dc66640 1 --2- 172.21.15.174:0/683748702 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0580c41c0 0x7fd0580ad460 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:42.992 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.987+0000 7fd05dc66640 1 --2- 172.21.15.174:0/683748702 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd0580a9b10 0x7fd0580acf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:42.992 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.987+0000 7fd05dc66640 1 --2- 172.21.15.174:0/683748702 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd05806fff0 0x7fd0580c1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:42.992 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.987+0000 7fd05dc66640 1 -- 172.21.15.174:0/683748702 >> 172.21.15.174:0/683748702 conn(0x7fd0580c49c0 msgr2=0x7fd0580c3680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:00:42.992 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.987+0000 7fd05dc66640 1 -- 172.21.15.174:0/683748702 shutdown_connections 2024-04-25T00:00:42.993 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:00:42.987+0000 7fd05dc66640 1 -- 172.21.15.174:0/683748702 wait complete. 2024-04-25T00:00:43.005 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-25T00:00:43.005 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-04-25T00:00:43.007 INFO:tasks.ceph.osd.0:Started 2024-04-25T00:00:43.007 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-25T00:00:43.007 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-25T00:00:43.010 INFO:tasks.ceph.osd.1:Started 2024-04-25T00:00:43.010 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-25T00:00:43.010 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-04-25T00:00:43.012 INFO:tasks.ceph.osd.2:Started 2024-04-25T00:00:43.013 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-25T00:00:43.013 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-04-25T00:00:43.015 INFO:tasks.ceph.osd.3:Started 2024-04-25T00:00:43.015 INFO:tasks.ceph.osd.4:Restarting daemon 2024-04-25T00:00:43.015 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-04-25T00:00:43.017 INFO:tasks.ceph.osd.4:Started 2024-04-25T00:00:43.017 INFO:tasks.ceph.osd.5:Restarting daemon 2024-04-25T00:00:43.017 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-04-25T00:00:43.019 INFO:tasks.ceph.osd.5:Started 2024-04-25T00:00:43.019 INFO:tasks.ceph.osd.6:Restarting daemon 2024-04-25T00:00:43.019 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-04-25T00:00:43.021 INFO:tasks.ceph.osd.6:Started 2024-04-25T00:00:43.021 INFO:tasks.ceph.osd.7:Restarting daemon 2024-04-25T00:00:43.021 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-04-25T00:00:43.023 INFO:tasks.ceph.osd.7:Started 2024-04-25T00:00:43.026 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T00:00:43.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.178+0000 7fa910d77640 1 Processor -- start 2024-04-25T00:00:43.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.178+0000 7fa910d77640 1 -- start start 2024-04-25T00:00:43.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.178+0000 7fa910d77640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fa90c0c5200 0x7fa90c154040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:43.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.178+0000 7fa910d77640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa90c156bb0 0x7fa90c156fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:43.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.178+0000 7fa910d77640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa90c0c2b40 con 0x7fa90c0c4760 2024-04-25T00:00:43.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.178+0000 7fa910d77640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fa90c072b80 con 0x7fa90c156bb0 2024-04-25T00:00:43.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.178+0000 7fa910d77640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fa90c154580 con 0x7fa90c0c5200 2024-04-25T00:00:43.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.178+0000 7fa90ad9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa90c156bb0 0x7fa90c156fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:43.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.178+0000 7fa90ad9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa90c156bb0 0x7fa90c156fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43300/0 (socket says 172.21.15.116:43300) 2024-04-25T00:00:43.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.178+0000 7fa909d9b640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fa90c0c4760 0x7fa90c14d230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:36670/0 (socket says 172.21.15.116:36670) 2024-04-25T00:00:43.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.178+0000 7fa90ad9d640 1 -- 172.21.15.116:0/536912309 learned_addr learned my addr 172.21.15.116:0/536912309 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:00:43.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.178+0000 7fa90a59c640 1 --2- 172.21.15.116:0/536912309 >> v2:172.21.15.174:3300/0 conn(0x7fa90c0c5200 0x7fa90c154040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:43.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.178+0000 7fa90ad9d640 1 -- 172.21.15.116:0/536912309 >> v2:172.21.15.174:3300/0 conn(0x7fa90c0c5200 msgr2=0x7fa90c154040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:00:43.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.178+0000 7fa90ad9d640 1 --2- 172.21.15.116:0/536912309 >> v2:172.21.15.174:3300/0 conn(0x7fa90c0c5200 0x7fa90c154040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:43.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.178+0000 7fa90ad9d640 1 -- 172.21.15.116:0/536912309 >> v1:172.21.15.116:6789/0 conn(0x7fa90c0c4760 legacy=0x7fa90c14d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:43.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.178+0000 7fa90ad9d640 1 -- 172.21.15.116:0/536912309 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa90c0c2db0 con 0x7fa90c156bb0 2024-04-25T00:00:43.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.182+0000 7fa90ad9d640 1 --2- 172.21.15.116:0/536912309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa90c156bb0 0x7fa90c156fe0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa90c0c5a60 tx=0x7fa8fc01c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edef4bd93affbdec server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:00:43.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.182+0000 7fa90959a640 1 -- 172.21.15.116:0/536912309 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8fc0a5710 con 0x7fa90c156bb0 2024-04-25T00:00:43.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.182+0000 7fa90959a640 1 -- 172.21.15.116:0/536912309 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8fc0a58b0 con 0x7fa90c156bb0 2024-04-25T00:00:43.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.182+0000 7fa90959a640 1 -- 172.21.15.116:0/536912309 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8fc0a5c80 con 0x7fa90c156bb0 2024-04-25T00:00:43.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.182+0000 7fa910d77640 1 -- 172.21.15.116:0/536912309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa90c156bb0 msgr2=0x7fa90c156fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:43.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.182+0000 7fa910d77640 1 --2- 172.21.15.116:0/536912309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa90c156bb0 0x7fa90c156fe0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa90c0c5a60 tx=0x7fa8fc01c960 comp rx=0 tx=0).stop 2024-04-25T00:00:43.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.182+0000 7fa910d77640 1 -- 172.21.15.116:0/536912309 shutdown_connections 2024-04-25T00:00:43.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.182+0000 7fa910d77640 1 --2- 172.21.15.116:0/536912309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa90c156bb0 0x7fa90c156fe0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:43.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.182+0000 7fa910d77640 1 --2- 172.21.15.116:0/536912309 >> v2:172.21.15.174:3300/0 conn(0x7fa90c0c5200 0x7fa90c154040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:43.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.182+0000 7fa910d77640 1 -- 172.21.15.116:0/536912309 >> 172.21.15.116:0/536912309 conn(0x7fa90c06c720 msgr2=0x7fa90c06cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:00:43.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.182+0000 7fa910d77640 1 -- 172.21.15.116:0/536912309 shutdown_connections 2024-04-25T00:00:43.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.182+0000 7fa910d77640 1 -- 172.21.15.116:0/536912309 wait complete. 2024-04-25T00:00:43.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.182+0000 7fa910d77640 1 Processor -- start 2024-04-25T00:00:43.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.182+0000 7fa910d77640 1 -- start start 2024-04-25T00:00:43.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.186+0000 7fa910d77640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa90c0c4760 0x7fa90c14ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:43.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.186+0000 7fa910d77640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa90c0c5200 0x7fa90c1ef8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:43.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.186+0000 7fa910d77640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa90c156bb0 0x7fa90c1efe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:43.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.186+0000 7fa910d77640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa90c14dac0 con 0x7fa90c156bb0 2024-04-25T00:00:43.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.186+0000 7fa910d77640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa90c14d940 con 0x7fa90c0c5200 2024-04-25T00:00:43.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.186+0000 7fa910d77640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa90c0c55e0 con 0x7fa90c0c4760 2024-04-25T00:00:43.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.186+0000 7fa90ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa90c156bb0 0x7fa90c1efe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:43.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.186+0000 7fa90ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa90c156bb0 0x7fa90c1efe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43308/0 (socket says 172.21.15.116:43308) 2024-04-25T00:00:43.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.186+0000 7fa90ad9d640 1 -- 172.21.15.116:0/2007215982 learned_addr learned my addr 172.21.15.116:0/2007215982 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:00:43.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.186+0000 7fa90a59c640 1 --2- 172.21.15.116:0/2007215982 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa90c0c4760 0x7fa90c14ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:43.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.186+0000 7fa909d9b640 1 --2- 172.21.15.116:0/2007215982 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa90c0c5200 0x7fa90c1ef8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:43.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.186+0000 7fa90ad9d640 1 -- 172.21.15.116:0/2007215982 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa90c0c4760 msgr2=0x7fa90c14ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:43.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.186+0000 7fa90ad9d640 1 --2- 172.21.15.116:0/2007215982 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa90c0c4760 0x7fa90c14ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:43.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.186+0000 7fa90ad9d640 1 -- 172.21.15.116:0/2007215982 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa90c0c5200 msgr2=0x7fa90c1ef8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:43.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.186+0000 7fa90ad9d640 1 --2- 172.21.15.116:0/2007215982 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa90c0c5200 0x7fa90c1ef8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:43.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.186+0000 7fa90ad9d640 1 -- 172.21.15.116:0/2007215982 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa90c0c5ee0 con 0x7fa90c156bb0 2024-04-25T00:00:43.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.186+0000 7fa909d9b640 1 --2- 172.21.15.116:0/2007215982 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa90c0c5200 0x7fa90c1ef8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:00:43.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.186+0000 7fa90ad9d640 1 --2- 172.21.15.116:0/2007215982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa90c156bb0 0x7fa90c1efe10 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc019570 tx=0x7fa8fc019700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:00:43.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.186+0000 7fa8faffd640 1 -- 172.21.15.116:0/2007215982 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8fc0bc020 con 0x7fa90c156bb0 2024-04-25T00:00:43.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.186+0000 7fa8faffd640 1 -- 172.21.15.116:0/2007215982 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8fc0b8670 con 0x7fa90c156bb0 2024-04-25T00:00:43.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.186+0000 7fa910d77640 1 -- 172.21.15.116:0/2007215982 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa90c0c61c0 con 0x7fa90c156bb0 2024-04-25T00:00:43.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.186+0000 7fa8faffd640 1 -- 172.21.15.116:0/2007215982 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8fc009510 con 0x7fa90c156bb0 2024-04-25T00:00:43.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.186+0000 7fa910d77640 1 -- 172.21.15.116:0/2007215982 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa90c0c7940 con 0x7fa90c156bb0 2024-04-25T00:00:43.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.186+0000 7fa8faffd640 1 -- 172.21.15.116:0/2007215982 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa8fc0bf020 con 0x7fa90c156bb0 2024-04-25T00:00:43.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.186+0000 7fa910d77640 1 -- 172.21.15.116:0/2007215982 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa90c068a60 con 0x7fa90c156bb0 2024-04-25T00:00:43.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.186+0000 7fa8faffd640 1 -- 172.21.15.116:0/2007215982 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7fa8fc0b8dd0 con 0x7fa90c156bb0 2024-04-25T00:00:43.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.190+0000 7fa8faffd640 1 -- 172.21.15.116:0/2007215982 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fa8fc0c6030 con 0x7fa90c156bb0 2024-04-25T00:00:43.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.246+0000 7fa910d77640 1 -- 172.21.15.116:0/2007215982 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa90c14a8b0 con 0x7fa90c156bb0 2024-04-25T00:00:43.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.246+0000 7fa8faffd640 1 -- 172.21.15.116:0/2007215982 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7fa8fc0165e0 con 0x7fa90c156bb0 2024-04-25T00:00:43.248 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:00:43.248 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":10,"fsid":"6d6a1aa7-0478-4c83-8057-05917e81f046","created":"2024-04-25T00:00:40.682493+0000","modified":"2024-04-25T00:00:42.980473+0000","last_up_change":"0.000000","last_in_change":"2024-04-25T00:00:42.980473+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8b6ddcb2-1ad9-4167-961c-bd6936bb9060","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a6431768-994f-41da-8853-886f5687fb91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4f69bd40-5a23-45ff-8693-e1de75e1aa9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6a0d5344-8b6f-40d3-918d-0f0a4be688e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"132e92e4-b825-441e-965a-ca00533fa86a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"a2eeb5b6-1b31-43cc-8306-a0adf3ee15bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"588e873a-c841-4ba7-90f2-8956c87a9983","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3b90f9b0-8a0e-4883-b088-ca1d4012870c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:00:43.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.246+0000 7fa910d77640 1 -- 172.21.15.116:0/2007215982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa90c156bb0 msgr2=0x7fa90c1efe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:43.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.246+0000 7fa910d77640 1 --2- 172.21.15.116:0/2007215982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa90c156bb0 0x7fa90c1efe10 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc019570 tx=0x7fa8fc019700 comp rx=0 tx=0).stop 2024-04-25T00:00:43.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.246+0000 7fa910d77640 1 -- 172.21.15.116:0/2007215982 shutdown_connections 2024-04-25T00:00:43.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.246+0000 7fa910d77640 1 --2- 172.21.15.116:0/2007215982 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa90c156bb0 0x7fa90c1efe10 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:43.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.246+0000 7fa910d77640 1 --2- 172.21.15.116:0/2007215982 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa90c0c5200 0x7fa90c1ef8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:43.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.246+0000 7fa910d77640 1 --2- 172.21.15.116:0/2007215982 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa90c0c4760 0x7fa90c14ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:43.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.246+0000 7fa910d77640 1 -- 172.21.15.116:0/2007215982 >> 172.21.15.116:0/2007215982 conn(0x7fa90c06c720 msgr2=0x7fa90c151f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:00:43.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.246+0000 7fa910d77640 1 -- 172.21.15.116:0/2007215982 shutdown_connections 2024-04-25T00:00:43.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:43.246+0000 7fa910d77640 1 -- 172.21.15.116:0/2007215982 wait complete. 2024-04-25T00:00:43.263 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-25T00:00:43.263 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-25T00:00:43.354 INFO:tasks.ceph.mgr.y.smithi116.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-04-25T00:00:43.354 INFO:tasks.ceph.mgr.y.smithi116.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-04-25T00:00:43.354 INFO:tasks.ceph.mgr.y.smithi116.stderr: from numpy import show_config as show_numpy_config 2024-04-25T00:00:43.377 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-04-25T00:00:43.375+0000 7f1fa98727c0 -1 Falling back to public interface 2024-04-25T00:00:43.382 INFO:tasks.ceph.mgr.x.smithi174.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-04-25T00:00:43.382 INFO:tasks.ceph.mgr.x.smithi174.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-04-25T00:00:43.382 INFO:tasks.ceph.mgr.x.smithi174.stderr: from numpy import show_config as show_numpy_config 2024-04-25T00:00:43.405 INFO:tasks.ceph.mgr.y.smithi116.stderr:2024-04-25T00:00:43.402+0000 7f494fb44180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T00:00:43.417 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-04-25T00:00:43.415+0000 7fef9320a7c0 -1 Falling back to public interface 2024-04-25T00:00:43.419 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-04-25T00:00:43.415+0000 7fc0f5688180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T00:00:43.427 INFO:tasks.ceph.osd.1.smithi116.stderr:2024-04-25T00:00:43.426+0000 7f0efd8047c0 -1 Falling back to public interface 2024-04-25T00:00:43.435 INFO:tasks.ceph.osd.0.smithi116.stderr:2024-04-25T00:00:43.434+0000 7fa0195a87c0 -1 Falling back to public interface 2024-04-25T00:00:43.539 INFO:tasks.ceph.mgr.y.smithi116.stderr:2024-04-25T00:00:43.538+0000 7f494fb44180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T00:00:43.551 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-04-25T00:00:43.547+0000 7fc0f5688180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T00:00:43.642 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-04-25T00:00:43.639+0000 7f2e9c8f47c0 -1 Falling back to public interface 2024-04-25T00:00:43.649 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-04-25T00:00:43.647+0000 7f8da06d37c0 -1 Falling back to public interface 2024-04-25T00:00:43.692 INFO:tasks.ceph.osd.2.smithi116.stderr:2024-04-25T00:00:43.690+0000 7fab4bdbf7c0 -1 Falling back to public interface 2024-04-25T00:00:43.695 INFO:tasks.ceph.mgr.y.smithi116.stderr:2024-04-25T00:00:43.694+0000 7f494fb44180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T00:00:43.700 INFO:tasks.ceph.osd.3.smithi116.stderr:2024-04-25T00:00:43.698+0000 7f0ea4bb87c0 -1 Falling back to public interface 2024-04-25T00:00:43.711 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-04-25T00:00:43.707+0000 7fc0f5688180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T00:00:43.760 INFO:tasks.ceph.mgr.y.smithi116.stderr:2024-04-25T00:00:43.758+0000 7f494fb44180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T00:00:43.775 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-04-25T00:00:43.771+0000 7fc0f5688180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T00:00:43.894 INFO:tasks.ceph.mgr.y.smithi116.stderr:2024-04-25T00:00:43.894+0000 7f494fb44180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T00:00:43.907 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-04-25T00:00:43.903+0000 7fc0f5688180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T00:00:44.069 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-25T00:00:44.182 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.182+0000 7f07b9285640 1 Processor -- start 2024-04-25T00:00:44.182 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.182+0000 7f07b9285640 1 -- start start 2024-04-25T00:00:44.184 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.182+0000 7f07b9285640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f07b4152c70 0x7f07b41530d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:44.184 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.182+0000 7f07b9285640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f07b4155900 0x7f07b4155d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:44.184 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.182+0000 7f07b9285640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07b40c47a0 con 0x7f07b40c2f40 2024-04-25T00:00:44.184 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.182+0000 7f07b9285640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f07b40c3a80 con 0x7f07b4155900 2024-04-25T00:00:44.184 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.182+0000 7f07b9285640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f07b4072a60 con 0x7f07b4152c70 2024-04-25T00:00:44.184 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.182+0000 7f07b37fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f07b4155900 0x7f07b4155d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:44.184 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.182+0000 7f07b37fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f07b4155900 0x7f07b4155d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43320/0 (socket says 172.21.15.116:43320) 2024-04-25T00:00:44.184 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.182+0000 7f07b37fe640 1 -- 172.21.15.116:0/22557886 learned_addr learned my addr 172.21.15.116:0/22557886 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:00:44.184 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.182+0000 7f07b27fc640 1 --2- 172.21.15.116:0/22557886 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f07b4152c70 0x7f07b41530d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:44.185 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.182+0000 7f07b37fe640 1 -- 172.21.15.116:0/22557886 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f07b4152c70 msgr2=0x7f07b41530d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:00:44.185 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.182+0000 7f07b37fe640 1 --2- 172.21.15.116:0/22557886 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f07b4152c70 0x7f07b41530d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:44.185 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.182+0000 7f07b37fe640 1 -- 172.21.15.116:0/22557886 >> v1:172.21.15.174:6789/0 conn(0x7f07b40c2f40 legacy=0x7f07b4071f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:44.185 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.182+0000 7f07b37fe640 1 -- 172.21.15.116:0/22557886 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07b40c3cc0 con 0x7f07b4155900 2024-04-25T00:00:44.185 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.182+0000 7f07b27fc640 1 --2- 172.21.15.116:0/22557886 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f07b4152c70 0x7f07b41530d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:00:44.186 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.182+0000 7f07b37fe640 1 --2- 172.21.15.116:0/22557886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07b4155900 0x7f07b4155d30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f07ac009e50 tx=0x7f07ac01ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa4300b480bd4d83 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:00:44.186 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.186+0000 7f07b1ffb640 1 -- 172.21.15.116:0/22557886 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07ac0a5700 con 0x7f07b4155900 2024-04-25T00:00:44.186 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.186+0000 7f07b1ffb640 1 -- 172.21.15.116:0/22557886 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07ac0a58a0 con 0x7f07b4155900 2024-04-25T00:00:44.186 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.186+0000 7f07b1ffb640 1 -- 172.21.15.116:0/22557886 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07ac0a5c70 con 0x7f07b4155900 2024-04-25T00:00:44.187 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.186+0000 7f07b9285640 1 -- 172.21.15.116:0/22557886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07b4155900 msgr2=0x7f07b4155d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:44.187 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.186+0000 7f07b9285640 1 --2- 172.21.15.116:0/22557886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07b4155900 0x7f07b4155d30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f07ac009e50 tx=0x7f07ac01ca30 comp rx=0 tx=0).stop 2024-04-25T00:00:44.187 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.186+0000 7f07b9285640 1 -- 172.21.15.116:0/22557886 shutdown_connections 2024-04-25T00:00:44.187 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.186+0000 7f07b9285640 1 --2- 172.21.15.116:0/22557886 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07b4155900 0x7f07b4155d30 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:44.187 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.186+0000 7f07b9285640 1 --2- 172.21.15.116:0/22557886 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f07b4152c70 0x7f07b41530d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:44.187 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.186+0000 7f07b9285640 1 -- 172.21.15.116:0/22557886 >> 172.21.15.116:0/22557886 conn(0x7f07b40a98a0 msgr2=0x7f07b40c7310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:00:44.187 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.186+0000 7f07b9285640 1 -- 172.21.15.116:0/22557886 shutdown_connections 2024-04-25T00:00:44.187 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.186+0000 7f07b9285640 1 -- 172.21.15.116:0/22557886 wait complete. 2024-04-25T00:00:44.188 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.186+0000 7f07b9285640 1 Processor -- start 2024-04-25T00:00:44.188 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.186+0000 7f07b9285640 1 -- start start 2024-04-25T00:00:44.190 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.186+0000 7f07b9285640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f07b40c2f40 0x7f07b413c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:44.190 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.186+0000 7f07b9285640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f07b4152c70 0x7f07b41549f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:44.190 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.186+0000 7f07b9285640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07b4155900 0x7f07b41519c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:44.190 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.186+0000 7f07b9285640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f07b40727e0 con 0x7f07b4155900 2024-04-25T00:00:44.190 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.186+0000 7f07b9285640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f07b4072660 con 0x7f07b4152c70 2024-04-25T00:00:44.190 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.186+0000 7f07b9285640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f07b4156270 con 0x7f07b40c2f40 2024-04-25T00:00:44.190 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.186+0000 7f07b37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07b4155900 0x7f07b41519c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:44.191 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.186+0000 7f07b37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07b4155900 0x7f07b41519c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43330/0 (socket says 172.21.15.116:43330) 2024-04-25T00:00:44.191 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.186+0000 7f07b37fe640 1 -- 172.21.15.116:0/4216505776 learned_addr learned my addr 172.21.15.116:0/4216505776 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:00:44.191 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.186+0000 7f07b27fc640 1 --2- 172.21.15.116:0/4216505776 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f07b4152c70 0x7f07b41549f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:44.191 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.186+0000 7f07b2ffd640 1 --2- 172.21.15.116:0/4216505776 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f07b40c2f40 0x7f07b413c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:44.191 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.186+0000 7f07b37fe640 1 -- 172.21.15.116:0/4216505776 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f07b40c2f40 msgr2=0x7f07b413c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:44.191 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.186+0000 7f07b37fe640 1 --2- 172.21.15.116:0/4216505776 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f07b40c2f40 0x7f07b413c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:44.191 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.186+0000 7f07b37fe640 1 -- 172.21.15.116:0/4216505776 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f07b4152c70 msgr2=0x7f07b41549f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:44.192 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.186+0000 7f07b37fe640 1 --2- 172.21.15.116:0/4216505776 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f07b4152c70 0x7f07b41549f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:44.192 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.186+0000 7f07b37fe640 1 -- 172.21.15.116:0/4216505776 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07b41556a0 con 0x7f07b4155900 2024-04-25T00:00:44.192 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.186+0000 7f07b37fe640 1 --2- 172.21.15.116:0/4216505776 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07b4155900 0x7f07b41519c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f07ac0af040 tx=0x7f07ac0197d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:00:44.192 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.186+0000 7f079b7fe640 1 -- 172.21.15.116:0/4216505776 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07ac0bc020 con 0x7f07b4155900 2024-04-25T00:00:44.192 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.186+0000 7f079b7fe640 1 -- 172.21.15.116:0/4216505776 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07ac0b8670 con 0x7f07b4155900 2024-04-25T00:00:44.192 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.186+0000 7f07b9285640 1 -- 172.21.15.116:0/4216505776 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07b413bab0 con 0x7f07b4155900 2024-04-25T00:00:44.192 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.186+0000 7f079b7fe640 1 -- 172.21.15.116:0/4216505776 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07ac0b8a60 con 0x7f07b4155900 2024-04-25T00:00:44.192 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.186+0000 7f07b9285640 1 -- 172.21.15.116:0/4216505776 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07b413c070 con 0x7f07b4155900 2024-04-25T00:00:44.193 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.186+0000 7f079b7fe640 1 -- 172.21.15.116:0/4216505776 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f07ac0bf020 con 0x7f07b4155900 2024-04-25T00:00:44.193 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.186+0000 7f079b7fe640 1 -- 172.21.15.116:0/4216505776 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f07ac0c1c70 con 0x7f07b4155900 2024-04-25T00:00:44.193 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.190+0000 7f07b9285640 1 -- 172.21.15.116:0/4216505776 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07b40758b0 con 0x7f07b4155900 2024-04-25T00:00:44.193 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.190+0000 7f079b7fe640 1 -- 172.21.15.116:0/4216505776 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f07ac0c6030 con 0x7f07b4155900 2024-04-25T00:00:44.250 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.246+0000 7f07b9285640 1 -- 172.21.15.116:0/4216505776 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f07b415fc50 con 0x7f07b4155900 2024-04-25T00:00:44.250 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.250+0000 7f079b7fe640 1 -- 172.21.15.116:0/4216505776 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f07ac016620 con 0x7f07b4155900 2024-04-25T00:00:44.251 INFO:teuthology.misc.health.smithi116.stdout: 2024-04-25T00:00:44.251 INFO:teuthology.misc.health.smithi116.stdout:{"epoch":10,"fsid":"6d6a1aa7-0478-4c83-8057-05917e81f046","created":"2024-04-25T00:00:40.682493+0000","modified":"2024-04-25T00:00:42.980473+0000","last_up_change":"0.000000","last_in_change":"2024-04-25T00:00:42.980473+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8b6ddcb2-1ad9-4167-961c-bd6936bb9060","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a6431768-994f-41da-8853-886f5687fb91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4f69bd40-5a23-45ff-8693-e1de75e1aa9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6a0d5344-8b6f-40d3-918d-0f0a4be688e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"132e92e4-b825-441e-965a-ca00533fa86a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"a2eeb5b6-1b31-43cc-8306-a0adf3ee15bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"588e873a-c841-4ba7-90f2-8956c87a9983","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3b90f9b0-8a0e-4883-b088-ca1d4012870c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:00:44.252 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.250+0000 7f07b9285640 1 -- 172.21.15.116:0/4216505776 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07b4155900 msgr2=0x7f07b41519c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:44.252 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.250+0000 7f07b9285640 1 --2- 172.21.15.116:0/4216505776 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07b4155900 0x7f07b41519c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f07ac0af040 tx=0x7f07ac0197d0 comp rx=0 tx=0).stop 2024-04-25T00:00:44.252 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.250+0000 7f07b9285640 1 -- 172.21.15.116:0/4216505776 shutdown_connections 2024-04-25T00:00:44.252 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.250+0000 7f07b9285640 1 --2- 172.21.15.116:0/4216505776 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07b4155900 0x7f07b41519c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:44.252 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.250+0000 7f07b9285640 1 --2- 172.21.15.116:0/4216505776 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f07b4152c70 0x7f07b41549f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:44.252 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.250+0000 7f07b9285640 1 --2- 172.21.15.116:0/4216505776 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f07b40c2f40 0x7f07b413c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:44.252 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.250+0000 7f07b9285640 1 -- 172.21.15.116:0/4216505776 >> 172.21.15.116:0/4216505776 conn(0x7f07b40a98a0 msgr2=0x7f07b40c0f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:00:44.252 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.250+0000 7f07b9285640 1 -- 172.21.15.116:0/4216505776 shutdown_connections 2024-04-25T00:00:44.252 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:44.250+0000 7f07b9285640 1 -- 172.21.15.116:0/4216505776 wait complete. 2024-04-25T00:00:44.266 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-25T00:00:44.529 INFO:tasks.ceph.mgr.y.smithi116.stderr:2024-04-25T00:00:44.526+0000 7f494fb44180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T00:00:44.533 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-04-25T00:00:44.531+0000 7fc0f5688180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T00:00:44.604 INFO:tasks.ceph.mgr.y.smithi116.stderr:2024-04-25T00:00:44.602+0000 7f494fb44180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T00:00:44.608 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-04-25T00:00:44.603+0000 7fc0f5688180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T00:00:44.787 INFO:tasks.ceph.mgr.y.smithi116.stderr:2024-04-25T00:00:44.786+0000 7f494fb44180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T00:00:44.790 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-04-25T00:00:44.787+0000 7fc0f5688180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T00:00:44.861 INFO:tasks.ceph.mgr.y.smithi116.stderr:2024-04-25T00:00:44.858+0000 7f494fb44180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T00:00:44.866 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-04-25T00:00:44.859+0000 7fc0f5688180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T00:00:45.152 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-04-25T00:00:45.147+0000 7fc0f5688180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T00:00:45.152 INFO:tasks.ceph.mgr.y.smithi116.stderr:2024-04-25T00:00:45.150+0000 7f494fb44180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T00:00:45.213 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-04-25T00:00:45.211+0000 7fc0f5688180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T00:00:45.214 INFO:tasks.ceph.mgr.y.smithi116.stderr:2024-04-25T00:00:45.210+0000 7f494fb44180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T00:00:46.253 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-04-25T00:00:46.251+0000 7fc09cad1640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-25T00:00:46.254 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-04-25T00:00:46.251+0000 7fc09cad1640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-25T00:00:47.227 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-04-25T00:00:47.223+0000 7f1fa98727c0 -1 osd.4 0 log_to_monitors true 2024-04-25T00:00:47.280 INFO:tasks.ceph.osd.0.smithi116.stderr:2024-04-25T00:00:47.278+0000 7fa0195a87c0 -1 osd.0 0 log_to_monitors true 2024-04-25T00:00:47.319 INFO:tasks.ceph.osd.1.smithi116.stderr:2024-04-25T00:00:47.318+0000 7f0efd8047c0 -1 osd.1 0 log_to_monitors true 2024-04-25T00:00:47.340 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-04-25T00:00:47.335+0000 7fef9320a7c0 -1 osd.5 0 log_to_monitors true 2024-04-25T00:00:47.440 INFO:tasks.ceph.osd.2.smithi116.stderr:2024-04-25T00:00:47.438+0000 7fab4bdbf7c0 -1 osd.2 0 log_to_monitors true 2024-04-25T00:00:47.464 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-04-25T00:00:47.459+0000 7f2e9c8f47c0 -1 osd.7 0 log_to_monitors true 2024-04-25T00:00:47.502 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-04-25T00:00:47.499+0000 7f8da06d37c0 -1 osd.6 0 log_to_monitors true 2024-04-25T00:00:47.568 INFO:tasks.ceph.osd.3.smithi116.stderr:2024-04-25T00:00:47.566+0000 7f0ea4bb87c0 -1 osd.3 0 log_to_monitors true 2024-04-25T00:00:49.178 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-04-25T00:00:49.175+0000 7f1fa682e640 -1 osd.4 0 waiting for initial osdmap 2024-04-25T00:00:49.206 INFO:tasks.ceph.osd.4.smithi174.stderr:2024-04-25T00:00:49.203+0000 7f1f9d63f640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T00:00:50.818 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-04-25T00:00:50.815+0000 7f8d9d68f640 -1 osd.6 0 waiting for initial osdmap 2024-04-25T00:00:50.838 INFO:tasks.ceph.osd.6.smithi174.stderr:2024-04-25T00:00:50.835+0000 7f8d944a0640 -1 osd.6 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T00:00:50.953 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-04-25T00:00:50.951+0000 7fef8e990640 -1 osd.5 0 waiting for initial osdmap 2024-04-25T00:00:50.978 INFO:tasks.ceph.osd.5.smithi174.stderr:2024-04-25T00:00:50.975+0000 7fef86fd7640 -1 osd.5 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T00:00:51.048 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-04-25T00:00:51.043+0000 7f2e9807a640 -1 osd.7 0 waiting for initial osdmap 2024-04-25T00:00:51.070 INFO:tasks.ceph.osd.7.smithi174.stderr:2024-04-25T00:00:51.067+0000 7f2e906c1640 -1 osd.7 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T00:00:51.073 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-25T00:00:51.234 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.230+0000 7fb090734640 1 Processor -- start 2024-04-25T00:00:51.235 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.230+0000 7fb090734640 1 -- start start 2024-04-25T00:00:51.236 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.234+0000 7fb090734640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fb0880cbcb0 0x7fb08815a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:51.236 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.234+0000 7fb090734640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb08806c720 0x7fb0881645b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:51.236 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.234+0000 7fb090734640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0880c64c0 con 0x7fb0880cb8d0 2024-04-25T00:00:51.237 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.234+0000 7fb090734640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fb0880cc160 con 0x7fb0880cbcb0 2024-04-25T00:00:51.237 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.234+0000 7fb090734640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fb08815acd0 con 0x7fb08806c720 2024-04-25T00:00:51.237 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.234+0000 7fb08dccf640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fb0880cbcb0 0x7fb08815a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:51.237 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.234+0000 7fb08ecd1640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb08806c720 0x7fb0881645b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:51.237 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.234+0000 7fb08dccf640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fb0880cbcb0 0x7fb08815a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:54260/0 (socket says 172.21.15.116:54260) 2024-04-25T00:00:51.237 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.234+0000 7fb08dccf640 1 -- 172.21.15.116:0/2991889200 learned_addr learned my addr 172.21.15.116:0/2991889200 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:00:51.237 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.234+0000 7fb08dccf640 1 -- 172.21.15.116:0/2991889200 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb08806c720 msgr2=0x7fb0881645b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:00:51.237 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.234+0000 7fb08dccf640 1 --2- 172.21.15.116:0/2991889200 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb08806c720 0x7fb0881645b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:51.237 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.234+0000 7fb08dccf640 1 -- 172.21.15.116:0/2991889200 >> v1:172.21.15.174:6789/0 conn(0x7fb0880cb8d0 legacy=0x7fb0880c90c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:51.237 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.234+0000 7fb08dccf640 1 -- 172.21.15.116:0/2991889200 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0880c5f00 con 0x7fb0880cbcb0 2024-04-25T00:00:51.239 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.238+0000 7fb08dccf640 1 --2- 172.21.15.116:0/2991889200 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb0880cbcb0 0x7fb08815a790 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb084009870 tx=0x7fb08401c460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79fd39d9d7cf0c64 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:00:51.240 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.238+0000 7fb08d4ce640 1 -- 172.21.15.116:0/2991889200 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0840b0070 con 0x7fb0880cbcb0 2024-04-25T00:00:51.240 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.238+0000 7fb08d4ce640 1 -- 172.21.15.116:0/2991889200 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb084004230 con 0x7fb0880cbcb0 2024-04-25T00:00:51.240 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.238+0000 7fb08d4ce640 1 -- 172.21.15.116:0/2991889200 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb084003620 con 0x7fb0880cbcb0 2024-04-25T00:00:51.241 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.238+0000 7fb090734640 1 -- 172.21.15.116:0/2991889200 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb0880cbcb0 msgr2=0x7fb08815a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:51.242 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.238+0000 7fb090734640 1 --2- 172.21.15.116:0/2991889200 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb0880cbcb0 0x7fb08815a790 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb084009870 tx=0x7fb08401c460 comp rx=0 tx=0).stop 2024-04-25T00:00:51.242 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.238+0000 7fb090734640 1 -- 172.21.15.116:0/2991889200 shutdown_connections 2024-04-25T00:00:51.242 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.238+0000 7fb090734640 1 --2- 172.21.15.116:0/2991889200 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb08806c720 0x7fb0881645b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:51.242 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.238+0000 7fb090734640 1 --2- 172.21.15.116:0/2991889200 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb0880cbcb0 0x7fb08815a790 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:51.242 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.238+0000 7fb090734640 1 -- 172.21.15.116:0/2991889200 >> 172.21.15.116:0/2991889200 conn(0x7fb0880ad2d0 msgr2=0x7fb0880cb1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:00:51.242 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.238+0000 7fb090734640 1 -- 172.21.15.116:0/2991889200 shutdown_connections 2024-04-25T00:00:51.242 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.238+0000 7fb090734640 1 -- 172.21.15.116:0/2991889200 wait complete. 2024-04-25T00:00:51.243 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.242+0000 7fb090734640 1 Processor -- start 2024-04-25T00:00:51.243 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.242+0000 7fb090734640 1 -- start start 2024-04-25T00:00:51.244 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.242+0000 7fb090734640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb08806c720 0x7fb0881f3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:51.244 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.242+0000 7fb090734640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb0880cb8d0 0x7fb0881f2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:51.244 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.242+0000 7fb090734640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb0880cbcb0 0x7fb088154720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:51.244 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.242+0000 7fb090734640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb0880c5cc0 con 0x7fb08806c720 2024-04-25T00:00:51.244 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.242+0000 7fb090734640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb0880c5b40 con 0x7fb0880cbcb0 2024-04-25T00:00:51.244 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.242+0000 7fb090734640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fb08806cb30 con 0x7fb0880cb8d0 2024-04-25T00:00:51.244 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.242+0000 7fb08e4d0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb08806c720 0x7fb0881f3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:51.244 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.242+0000 7fb08e4d0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb08806c720 0x7fb0881f3770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34154/0 (socket says 172.21.15.116:34154) 2024-04-25T00:00:51.245 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.242+0000 7fb08e4d0640 1 -- 172.21.15.116:0/2967364963 learned_addr learned my addr 172.21.15.116:0/2967364963 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:00:51.245 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.242+0000 7fb08ecd1640 1 --2- 172.21.15.116:0/2967364963 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb0880cbcb0 0x7fb088154720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:51.245 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.242+0000 7fb08dccf640 1 --2- 172.21.15.116:0/2967364963 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb0880cb8d0 0x7fb0881f2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:51.245 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.242+0000 7fb08e4d0640 1 -- 172.21.15.116:0/2967364963 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb0880cb8d0 msgr2=0x7fb0881f2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:51.245 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.242+0000 7fb08e4d0640 1 --2- 172.21.15.116:0/2967364963 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb0880cb8d0 0x7fb0881f2bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:51.246 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.242+0000 7fb08e4d0640 1 -- 172.21.15.116:0/2967364963 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb0880cbcb0 msgr2=0x7fb088154720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:51.246 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.242+0000 7fb08e4d0640 1 --2- 172.21.15.116:0/2967364963 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb0880cbcb0 0x7fb088154720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:51.246 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.242+0000 7fb08e4d0640 1 -- 172.21.15.116:0/2967364963 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0881f2410 con 0x7fb08806c720 2024-04-25T00:00:51.246 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.242+0000 7fb08e4d0640 1 --2- 172.21.15.116:0/2967364963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb08806c720 0x7fb0881f3770 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb07800c480 tx=0x7fb07800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:00:51.246 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.242+0000 7fb07effd640 1 -- 172.21.15.116:0/2967364963 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0780052d0 con 0x7fb08806c720 2024-04-25T00:00:51.246 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.242+0000 7fb07effd640 1 -- 172.21.15.116:0/2967364963 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb078005490 con 0x7fb08806c720 2024-04-25T00:00:51.246 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.242+0000 7fb090734640 1 -- 172.21.15.116:0/2967364963 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0880c1610 con 0x7fb08806c720 2024-04-25T00:00:51.246 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.242+0000 7fb07effd640 1 -- 172.21.15.116:0/2967364963 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb078012850 con 0x7fb08806c720 2024-04-25T00:00:51.246 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.246+0000 7fb090734640 1 -- 172.21.15.116:0/2967364963 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0880c18f0 con 0x7fb08806c720 2024-04-25T00:00:51.247 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.246+0000 7fb07effd640 1 -- 172.21.15.116:0/2967364963 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 4) ==== 87389+0+0 (secure 0 0 0) 0x7fb0780129f0 con 0x7fb08806c720 2024-04-25T00:00:51.247 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.246+0000 7fb07effd640 1 --2- 172.21.15.116:0/2967364963 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb06c067880 0x7fb06c069d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:51.247 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.246+0000 7fb07effd640 1 -- 172.21.15.116:0/2967364963 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2511+0+0 (secure 0 0 0) 0x7fb078089ba0 con 0x7fb08806c720 2024-04-25T00:00:51.248 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.246+0000 7fb090734640 1 -- 172.21.15.116:0/2967364963 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0880758b0 con 0x7fb08806c720 2024-04-25T00:00:51.248 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.246+0000 7fb08dccf640 1 --2- 172.21.15.116:0/2967364963 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb06c067880 0x7fb06c069d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:51.248 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.246+0000 7fb08dccf640 1 --2- 172.21.15.116:0/2967364963 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb06c067880 0x7fb06c069d40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb0840ad000 tx=0x7fb084008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:00:51.272 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.270+0000 7fb07effd640 1 -- 172.21.15.116:0/2967364963 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fb078055b10 con 0x7fb08806c720 2024-04-25T00:00:51.402 INFO:tasks.ceph.osd.1.smithi116.stderr:2024-04-25T00:00:51.398+0000 7f0ef8f8a640 -1 osd.1 0 waiting for initial osdmap 2024-04-25T00:00:51.406 INFO:tasks.ceph.osd.0.smithi116.stderr:2024-04-25T00:00:51.406+0000 7fa014d2e640 -1 osd.0 0 waiting for initial osdmap 2024-04-25T00:00:51.415 INFO:tasks.ceph.osd.3.smithi116.stderr:2024-04-25T00:00:51.414+0000 7f0ea033e640 -1 osd.3 0 waiting for initial osdmap 2024-04-25T00:00:51.426 INFO:tasks.ceph.osd.1.smithi116.stderr:2024-04-25T00:00:51.426+0000 7f0ef15d1640 -1 osd.1 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T00:00:51.443 INFO:tasks.ceph.osd.0.smithi116.stderr:2024-04-25T00:00:51.442+0000 7fa00d375640 -1 osd.0 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T00:00:51.463 INFO:tasks.ceph.osd.3.smithi116.stderr:2024-04-25T00:00:51.462+0000 7f0e98985640 -1 osd.3 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T00:00:51.503 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.502+0000 7fb090734640 1 -- 172.21.15.116:0/2967364963 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb088157960 con 0x7fb08806c720 2024-04-25T00:00:51.504 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.502+0000 7fb07effd640 1 -- 172.21.15.116:0/2967364963 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) ==== 74+0+7590 (secure 0 0 0) 0x7fb078055410 con 0x7fb08806c720 2024-04-25T00:00:51.505 INFO:teuthology.misc.health.smithi116.stdout: 2024-04-25T00:00:51.505 INFO:teuthology.misc.health.smithi116.stdout:{"epoch":14,"fsid":"6d6a1aa7-0478-4c83-8057-05917e81f046","created":"2024-04-25T00:00:40.682493+0000","modified":"2024-04-25T00:00:50.392687+0000","last_up_change":"2024-04-25T00:00:49.249074+0000","last_in_change":"2024-04-25T00:00:42.980473+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8b6ddcb2-1ad9-4167-961c-bd6936bb9060","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a6431768-994f-41da-8853-886f5687fb91","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4f69bd40-5a23-45ff-8693-e1de75e1aa9c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6a0d5344-8b6f-40d3-918d-0f0a4be688e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"132e92e4-b825-441e-965a-ca00533fa86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6801","nonce":3132444330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6803","nonce":3132444330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6807","nonce":3132444330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6805","nonce":3132444330}]},"public_addr":"172.21.15.174:6801/3132444330","cluster_addr":"172.21.15.174:6803/3132444330","heartbeat_back_addr":"172.21.15.174:6807/3132444330","heartbeat_front_addr":"172.21.15.174:6805/3132444330","state":["exists","up"]},{"osd":5,"uuid":"a2eeb5b6-1b31-43cc-8306-a0adf3ee15bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"588e873a-c841-4ba7-90f2-8956c87a9983","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3b90f9b0-8a0e-4883-b088-ca1d4012870c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.192174+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:00:51.505 INFO:tasks.ceph.osd.2.smithi116.stderr:2024-04-25T00:00:51.502+0000 7fab47545640 -1 osd.2 0 waiting for initial osdmap 2024-04-25T00:00:51.507 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.506+0000 7fb090734640 1 -- 172.21.15.116:0/2967364963 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb06c067880 msgr2=0x7fb06c069d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:51.507 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.506+0000 7fb090734640 1 --2- 172.21.15.116:0/2967364963 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb06c067880 0x7fb06c069d40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb0840ad000 tx=0x7fb084008040 comp rx=0 tx=0).stop 2024-04-25T00:00:51.507 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.506+0000 7fb090734640 1 -- 172.21.15.116:0/2967364963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb08806c720 msgr2=0x7fb0881f3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:51.507 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.506+0000 7fb090734640 1 --2- 172.21.15.116:0/2967364963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb08806c720 0x7fb0881f3770 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb07800c480 tx=0x7fb07800c940 comp rx=0 tx=0).stop 2024-04-25T00:00:51.508 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.506+0000 7fb090734640 1 -- 172.21.15.116:0/2967364963 shutdown_connections 2024-04-25T00:00:51.508 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.506+0000 7fb090734640 1 --2- 172.21.15.116:0/2967364963 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb06c067880 0x7fb06c069d40 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:51.508 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.506+0000 7fb090734640 1 --2- 172.21.15.116:0/2967364963 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb0880cbcb0 0x7fb088154720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:51.508 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.506+0000 7fb090734640 1 --2- 172.21.15.116:0/2967364963 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb0880cb8d0 0x7fb0881f2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:51.508 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.506+0000 7fb090734640 1 --2- 172.21.15.116:0/2967364963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb08806c720 0x7fb0881f3770 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:51.508 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.506+0000 7fb090734640 1 -- 172.21.15.116:0/2967364963 >> 172.21.15.116:0/2967364963 conn(0x7fb0880ad2d0 msgr2=0x7fb088159040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:00:51.508 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.506+0000 7fb090734640 1 -- 172.21.15.116:0/2967364963 shutdown_connections 2024-04-25T00:00:51.509 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:51.506+0000 7fb090734640 1 -- 172.21.15.116:0/2967364963 wait complete. 2024-04-25T00:00:51.521 DEBUG:teuthology.misc:1 of 8 OSDs are up 2024-04-25T00:00:51.543 INFO:tasks.ceph.osd.2.smithi116.stderr:2024-04-25T00:00:51.542+0000 7fab3fb8c640 -1 osd.2 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T00:00:58.328 DEBUG:teuthology.orchestra.run.smithi116:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-25T00:00:58.450 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.446+0000 7f1a5da9f640 1 Processor -- start 2024-04-25T00:00:58.450 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.446+0000 7f1a5da9f640 1 -- start start 2024-04-25T00:00:58.452 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.450+0000 7f1a5da9f640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f1a58154110 0x7f1a5806d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:58.452 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.450+0000 7f1a5da9f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1a5806e2c0 0x7f1a5806e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:58.452 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.450+0000 7f1a5da9f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a5806ec50 con 0x7f1a58150ca0 2024-04-25T00:00:58.452 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.450+0000 7f1a5da9f640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f1a580ab590 con 0x7f1a58154110 2024-04-25T00:00:58.452 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.450+0000 7f1a5da9f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f1a580ab830 con 0x7f1a5806e2c0 2024-04-25T00:00:58.453 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.450+0000 7f1a577fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f1a58150ca0 0x7f1a580a94a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:59432/0 (socket says 172.21.15.116:59432) 2024-04-25T00:00:58.453 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.450+0000 7f1a577fe640 1 -- 172.21.15.116:0/922077240 learned_addr learned my addr 172.21.15.116:0/922077240 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:00:58.453 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.450+0000 7f1a56ffd640 1 --2- 172.21.15.116:0/922077240 >> v2:172.21.15.174:3300/0 conn(0x7f1a58154110 0x7f1a5806d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:58.453 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.450+0000 7f1a567fc640 1 -- 172.21.15.116:0/922077240 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2555855028 0 0) 0x7f1a5806ec50 con 0x7f1a58150ca0 2024-04-25T00:00:58.453 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.450+0000 7f1a567fc640 1 -- 172.21.15.116:0/922077240 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a40003530 con 0x7f1a58150ca0 2024-04-25T00:00:58.453 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.450+0000 7f1a57fff640 1 --2- 172.21.15.116:0/922077240 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1a5806e2c0 0x7f1a5806e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:58.453 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.450+0000 7f1a567fc640 1 -- 172.21.15.116:0/922077240 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3877236263 0 0) 0x7f1a40003530 con 0x7f1a58150ca0 2024-04-25T00:00:58.453 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.450+0000 7f1a567fc640 1 -- 172.21.15.116:0/922077240 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1a5806e2c0 msgr2=0x7f1a5806e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:00:58.453 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.450+0000 7f1a567fc640 1 --2- 172.21.15.116:0/922077240 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1a5806e2c0 0x7f1a5806e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:58.453 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.450+0000 7f1a567fc640 1 -- 172.21.15.116:0/922077240 >> v2:172.21.15.174:3300/0 conn(0x7f1a58154110 msgr2=0x7f1a5806d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:00:58.453 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.450+0000 7f1a567fc640 1 --2- 172.21.15.116:0/922077240 >> v2:172.21.15.174:3300/0 conn(0x7f1a58154110 0x7f1a5806d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:58.453 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.450+0000 7f1a567fc640 1 -- 172.21.15.116:0/922077240 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a580abb10 con 0x7f1a58150ca0 2024-04-25T00:00:58.453 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.450+0000 7f1a567fc640 1 -- 172.21.15.116:0/922077240 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f1a48001680 con 0x7f1a58150ca0 2024-04-25T00:00:58.453 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.450+0000 7f1a567fc640 1 -- 172.21.15.116:0/922077240 >> v1:172.21.15.116:6789/0 conn(0x7f1a58150ca0 legacy=0x7f1a580a94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:58.454 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.450+0000 7f1a567fc640 1 --2- 172.21.15.116:0/922077240 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a400039c0 0x7f1a40003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:58.454 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.450+0000 7f1a567fc640 1 --2- 172.21.15.116:0/922077240 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a40004810 0x7f1a40004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:58.454 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.450+0000 7f1a567fc640 1 --2- 172.21.15.116:0/922077240 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1a5806e2c0 0x7f1a400051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:58.454 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.450+0000 7f1a567fc640 1 -- 172.21.15.116:0/922077240 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1a580ab590 con 0x7f1a400039c0 2024-04-25T00:00:58.454 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.450+0000 7f1a567fc640 1 -- 172.21.15.116:0/922077240 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1a580ab830 con 0x7f1a40004810 2024-04-25T00:00:58.454 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.450+0000 7f1a567fc640 1 -- 172.21.15.116:0/922077240 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f1a4000ba80 con 0x7f1a5806e2c0 2024-04-25T00:00:58.454 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.450+0000 7f1a577fe640 1 --2- 172.21.15.116:0/922077240 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a400039c0 0x7f1a40003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:58.454 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.450+0000 7f1a56ffd640 1 --2- 172.21.15.116:0/922077240 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a40004810 0x7f1a40004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:58.454 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.450+0000 7f1a577fe640 1 -- 172.21.15.116:0/922077240 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1a5806e2c0 msgr2=0x7f1a400051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:00:58.454 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.450+0000 7f1a577fe640 1 --2- 172.21.15.116:0/922077240 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1a5806e2c0 0x7f1a400051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:58.454 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.450+0000 7f1a577fe640 1 -- 172.21.15.116:0/922077240 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a40004810 msgr2=0x7f1a40004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:00:58.454 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.450+0000 7f1a577fe640 1 --2- 172.21.15.116:0/922077240 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a40004810 0x7f1a40004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:58.454 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.450+0000 7f1a577fe640 1 -- 172.21.15.116:0/922077240 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a40003530 con 0x7f1a400039c0 2024-04-25T00:00:58.455 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.454+0000 7f1a577fe640 1 --2- 172.21.15.116:0/922077240 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a400039c0 0x7f1a40003de0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1a4800c210 tx=0x7f1a4801cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4453010539852720 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:00:58.456 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.454+0000 7f1a567fc640 1 -- 172.21.15.116:0/922077240 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a48007d40 con 0x7f1a400039c0 2024-04-25T00:00:58.456 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.454+0000 7f1a567fc640 1 -- 172.21.15.116:0/922077240 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a480a9020 con 0x7f1a400039c0 2024-04-25T00:00:58.456 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.454+0000 7f1a567fc640 1 -- 172.21.15.116:0/922077240 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a48005b00 con 0x7f1a400039c0 2024-04-25T00:00:58.456 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.454+0000 7f1a5da9f640 1 -- 172.21.15.116:0/922077240 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a400039c0 msgr2=0x7f1a40003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:58.456 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.454+0000 7f1a5da9f640 1 --2- 172.21.15.116:0/922077240 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a400039c0 0x7f1a40003de0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1a4800c210 tx=0x7f1a4801cba0 comp rx=0 tx=0).stop 2024-04-25T00:00:58.456 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.454+0000 7f1a57fff640 1 -- 172.21.15.116:0/922077240 reap_dead start 2024-04-25T00:00:58.456 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.454+0000 7f1a5da9f640 1 -- 172.21.15.116:0/922077240 shutdown_connections 2024-04-25T00:00:58.456 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.454+0000 7f1a5da9f640 1 -- 172.21.15.116:0/922077240 >> 172.21.15.116:0/922077240 conn(0x7f1a5806b9a0 msgr2=0x7f1a580c6e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:00:58.456 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.454+0000 7f1a5da9f640 1 -- 172.21.15.116:0/922077240 shutdown_connections 2024-04-25T00:00:58.456 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.454+0000 7f1a5da9f640 1 -- 172.21.15.116:0/922077240 wait complete. 2024-04-25T00:00:58.458 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.454+0000 7f1a5da9f640 1 Processor -- start 2024-04-25T00:00:58.458 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.454+0000 7f1a5da9f640 1 -- start start 2024-04-25T00:00:58.459 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.454+0000 7f1a5da9f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1a400039c0 0x7f1a5814e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:58.459 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.454+0000 7f1a5da9f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a580c6d30 0x7f1a581ee860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:58.459 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.454+0000 7f1a5da9f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a58070d90 0x7f1a581eeda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:58.459 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.454+0000 7f1a5da9f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1a580a9d30 con 0x7f1a58070d90 2024-04-25T00:00:58.459 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.454+0000 7f1a5da9f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1a580a9bb0 con 0x7f1a580c6d30 2024-04-25T00:00:58.459 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.454+0000 7f1a5da9f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f1a5806cda0 con 0x7f1a400039c0 2024-04-25T00:00:58.459 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.454+0000 7f1a57fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a58070d90 0x7f1a581eeda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:58.459 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.454+0000 7f1a57fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a58070d90 0x7f1a581eeda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43038/0 (socket says 172.21.15.116:43038) 2024-04-25T00:00:58.459 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.454+0000 7f1a57fff640 1 -- 172.21.15.116:0/1566953815 learned_addr learned my addr 172.21.15.116:0/1566953815 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:00:58.459 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.454+0000 7f1a56ffd640 1 --2- 172.21.15.116:0/1566953815 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a580c6d30 0x7f1a581ee860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:58.459 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.454+0000 7f1a577fe640 1 --2- 172.21.15.116:0/1566953815 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1a400039c0 0x7f1a5814e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:58.459 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.454+0000 7f1a57fff640 1 -- 172.21.15.116:0/1566953815 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1a400039c0 msgr2=0x7f1a5814e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:58.459 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.454+0000 7f1a57fff640 1 --2- 172.21.15.116:0/1566953815 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1a400039c0 0x7f1a5814e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:58.459 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.454+0000 7f1a57fff640 1 -- 172.21.15.116:0/1566953815 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a580c6d30 msgr2=0x7f1a581ee860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:58.459 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.454+0000 7f1a57fff640 1 --2- 172.21.15.116:0/1566953815 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a580c6d30 0x7f1a581ee860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:58.460 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.454+0000 7f1a57fff640 1 -- 172.21.15.116:0/1566953815 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a581f1950 con 0x7f1a58070d90 2024-04-25T00:00:58.460 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.454+0000 7f1a57fff640 1 --2- 172.21.15.116:0/1566953815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a58070d90 0x7f1a581eeda0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1a580c7940 tx=0x7f1a5000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:00:58.460 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.458+0000 7f1a3ffff640 1 -- 172.21.15.116:0/1566953815 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a5000eb60 con 0x7f1a58070d90 2024-04-25T00:00:58.460 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.458+0000 7f1a3ffff640 1 -- 172.21.15.116:0/1566953815 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a5000ed20 con 0x7f1a58070d90 2024-04-25T00:00:58.460 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.458+0000 7f1a5da9f640 1 -- 172.21.15.116:0/1566953815 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a581ee360 con 0x7f1a58070d90 2024-04-25T00:00:58.460 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.458+0000 7f1a3ffff640 1 -- 172.21.15.116:0/1566953815 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a50003be0 con 0x7f1a58070d90 2024-04-25T00:00:58.460 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.458+0000 7f1a5da9f640 1 -- 172.21.15.116:0/1566953815 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a580c48a0 con 0x7f1a58070d90 2024-04-25T00:00:58.461 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.458+0000 7f1a5da9f640 1 -- 172.21.15.116:0/1566953815 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a580758b0 con 0x7f1a58070d90 2024-04-25T00:00:58.461 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.458+0000 7f1a3ffff640 1 -- 172.21.15.116:0/1566953815 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1a5001b020 con 0x7f1a58070d90 2024-04-25T00:00:58.461 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.458+0000 7f1a3ffff640 1 --2- 172.21.15.116:0/1566953815 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1a440678f0 0x7f1a44069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:58.461 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.458+0000 7f1a3ffff640 1 -- 172.21.15.116:0/1566953815 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 4857+0+0 (secure 0 0 0) 0x7f1a50085580 con 0x7f1a58070d90 2024-04-25T00:00:58.461 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.458+0000 7f1a577fe640 1 --2- 172.21.15.116:0/1566953815 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1a440678f0 0x7f1a44069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:58.461 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.458+0000 7f1a577fe640 1 --2- 172.21.15.116:0/1566953815 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1a440678f0 0x7f1a44069db0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1a480aa000 tx=0x7f1a480b0040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:00:58.465 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.462+0000 7f1a3ffff640 1 -- 172.21.15.116:0/1566953815 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f1a500564a0 con 0x7f1a58070d90 2024-04-25T00:00:58.625 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.622+0000 7f1a5da9f640 1 -- 172.21.15.116:0/1566953815 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1a581ef2e0 con 0x7f1a58070d90 2024-04-25T00:00:58.627 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.626+0000 7f1a3ffff640 1 -- 172.21.15.116:0/1566953815 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) ==== 74+0+12908 (secure 0 0 0) 0x7f1a50055da0 con 0x7f1a58070d90 2024-04-25T00:00:58.627 INFO:teuthology.misc.health.smithi116.stdout: 2024-04-25T00:00:58.627 INFO:teuthology.misc.health.smithi116.stdout:{"epoch":19,"fsid":"6d6a1aa7-0478-4c83-8057-05917e81f046","created":"2024-04-25T00:00:40.682493+0000","modified":"2024-04-25T00:00:56.079560+0000","last_up_change":"2024-04-25T00:00:51.822174+0000","last_in_change":"2024-04-25T00:00:42.980473+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T00:00:53.324376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8b6ddcb2-1ad9-4167-961c-bd6936bb9060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6809","nonce":3751765682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6811","nonce":3751765682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6815","nonce":3751765682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6813","nonce":3751765682}]},"public_addr":"172.21.15.116:6809/3751765682","cluster_addr":"172.21.15.116:6811/3751765682","heartbeat_back_addr":"172.21.15.116:6815/3751765682","heartbeat_front_addr":"172.21.15.116:6813/3751765682","state":["exists","up"]},{"osd":1,"uuid":"a6431768-994f-41da-8853-886f5687fb91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6801","nonce":377879124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6803","nonce":377879124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6807","nonce":377879124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6805","nonce":377879124}]},"public_addr":"172.21.15.116:6801/377879124","cluster_addr":"172.21.15.116:6803/377879124","heartbeat_back_addr":"172.21.15.116:6807/377879124","heartbeat_front_addr":"172.21.15.116:6805/377879124","state":["exists","up"]},{"osd":2,"uuid":"4f69bd40-5a23-45ff-8693-e1de75e1aa9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6817","nonce":1099282267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6819","nonce":1099282267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6823","nonce":1099282267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6821","nonce":1099282267}]},"public_addr":"172.21.15.116:6817/1099282267","cluster_addr":"172.21.15.116:6819/1099282267","heartbeat_back_addr":"172.21.15.116:6823/1099282267","heartbeat_front_addr":"172.21.15.116:6821/1099282267","state":["exists","up"]},{"osd":3,"uuid":"6a0d5344-8b6f-40d3-918d-0f0a4be688e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6825","nonce":1179942661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6827","nonce":1179942661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6831","nonce":1179942661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6829","nonce":1179942661}]},"public_addr":"172.21.15.116:6825/1179942661","cluster_addr":"172.21.15.116:6827/1179942661","heartbeat_back_addr":"172.21.15.116:6831/1179942661","heartbeat_front_addr":"172.21.15.116:6829/1179942661","state":["exists","up"]},{"osd":4,"uuid":"132e92e4-b825-441e-965a-ca00533fa86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6801","nonce":3132444330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6803","nonce":3132444330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6807","nonce":3132444330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6805","nonce":3132444330}]},"public_addr":"172.21.15.174:6801/3132444330","cluster_addr":"172.21.15.174:6803/3132444330","heartbeat_back_addr":"172.21.15.174:6807/3132444330","heartbeat_front_addr":"172.21.15.174:6805/3132444330","state":["exists","up"]},{"osd":5,"uuid":"a2eeb5b6-1b31-43cc-8306-a0adf3ee15bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6809","nonce":881817164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6811","nonce":881817164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6815","nonce":881817164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6813","nonce":881817164}]},"public_addr":"172.21.15.174:6809/881817164","cluster_addr":"172.21.15.174:6811/881817164","heartbeat_back_addr":"172.21.15.174:6815/881817164","heartbeat_front_addr":"172.21.15.174:6813/881817164","state":["exists","up"]},{"osd":6,"uuid":"588e873a-c841-4ba7-90f2-8956c87a9983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6825","nonce":117574251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6827","nonce":117574251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6831","nonce":117574251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6829","nonce":117574251}]},"public_addr":"172.21.15.174:6825/117574251","cluster_addr":"172.21.15.174:6827/117574251","heartbeat_back_addr":"172.21.15.174:6831/117574251","heartbeat_front_addr":"172.21.15.174:6829/117574251","state":["exists","up"]},{"osd":7,"uuid":"3b90f9b0-8a0e-4883-b088-ca1d4012870c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6817","nonce":2543899196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6819","nonce":2543899196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6823","nonce":2543899196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6821","nonce":2543899196}]},"public_addr":"172.21.15.174:6817/2543899196","cluster_addr":"172.21.15.174:6819/2543899196","heartbeat_back_addr":"172.21.15.174:6823/2543899196","heartbeat_front_addr":"172.21.15.174:6821/2543899196","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.293769+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.310034+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.404865+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.549658+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.192174+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.297266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.493649+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.452309+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:00:58.630 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.626+0000 7f1a5da9f640 1 -- 172.21.15.116:0/1566953815 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1a440678f0 msgr2=0x7f1a44069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:58.630 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.626+0000 7f1a5da9f640 1 --2- 172.21.15.116:0/1566953815 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1a440678f0 0x7f1a44069db0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1a480aa000 tx=0x7f1a480b0040 comp rx=0 tx=0).stop 2024-04-25T00:00:58.630 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.626+0000 7f1a5da9f640 1 -- 172.21.15.116:0/1566953815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a58070d90 msgr2=0x7f1a581eeda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:58.630 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.626+0000 7f1a5da9f640 1 --2- 172.21.15.116:0/1566953815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a58070d90 0x7f1a581eeda0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1a580c7940 tx=0x7f1a5000e4b0 comp rx=0 tx=0).stop 2024-04-25T00:00:58.630 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.626+0000 7f1a5da9f640 1 -- 172.21.15.116:0/1566953815 shutdown_connections 2024-04-25T00:00:58.630 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.626+0000 7f1a5da9f640 1 --2- 172.21.15.116:0/1566953815 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1a440678f0 0x7f1a44069db0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:58.630 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.626+0000 7f1a5da9f640 1 --2- 172.21.15.116:0/1566953815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a58070d90 0x7f1a581eeda0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:58.630 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.626+0000 7f1a5da9f640 1 --2- 172.21.15.116:0/1566953815 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a580c6d30 0x7f1a581ee860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:58.630 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.626+0000 7f1a5da9f640 1 --2- 172.21.15.116:0/1566953815 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1a400039c0 0x7f1a5814e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:58.630 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.626+0000 7f1a5da9f640 1 -- 172.21.15.116:0/1566953815 >> 172.21.15.116:0/1566953815 conn(0x7f1a5806b9a0 msgr2=0x7f1a5814f8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:00:58.630 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.626+0000 7f1a5da9f640 1 -- 172.21.15.116:0/1566953815 shutdown_connections 2024-04-25T00:00:58.631 INFO:teuthology.misc.health.smithi116.stderr:2024-04-25T00:00:58.626+0000 7f1a5da9f640 1 -- 172.21.15.116:0/1566953815 wait complete. 2024-04-25T00:00:58.645 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-04-25T00:00:58.645 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-25T00:00:58.646 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-25T00:00:58.646 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-25T00:00:58.648 INFO:tasks.ceph.mds.a:Started 2024-04-25T00:00:58.649 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-25T00:00:58.649 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-25T00:00:58.651 INFO:tasks.ceph.mds.c:Started 2024-04-25T00:00:58.651 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-25T00:00:58.651 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-25T00:00:58.654 INFO:tasks.ceph.mds.b:Started 2024-04-25T00:00:58.654 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-25T00:00:58.654 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-25T00:00:58.657 INFO:tasks.ceph.mds.d:Started 2024-04-25T00:00:58.657 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-04-25T00:00:58.657 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T00:00:58.720 INFO:tasks.ceph.mds.a.smithi116.stdout:starting mds.a at 2024-04-25T00:00:58.721 INFO:tasks.ceph.mds.c.smithi116.stdout:starting mds.c at 2024-04-25T00:00:58.741 INFO:tasks.ceph.mds.d.smithi174.stdout:starting mds.d at 2024-04-25T00:00:58.745 INFO:tasks.ceph.mds.b.smithi174.stdout:starting mds.b at 2024-04-25T00:00:58.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.770+0000 7f43b487a640 1 Processor -- start 2024-04-25T00:00:58.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.770+0000 7f43b487a640 1 -- start start 2024-04-25T00:00:58.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.770+0000 7f43b487a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f43b0150d10 0x7f43b01541f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:58.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.770+0000 7f43b487a640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f43b00c2b10 0x7f43b00a94a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:58.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.770+0000 7f43b487a640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43b00734b0 con 0x7f43b00a99e0 2024-04-25T00:00:58.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.770+0000 7f43b487a640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f43b0153750 con 0x7f43b0150d10 2024-04-25T00:00:58.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.770+0000 7f43b487a640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f43b00731d0 con 0x7f43b00c2b10 2024-04-25T00:00:58.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.770+0000 7f43aed9d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f43b00a99e0 0x7f43b00c3260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:59454/0 (socket says 172.21.15.116:59454) 2024-04-25T00:00:58.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.770+0000 7f43aed9d640 1 -- 172.21.15.116:0/3532946313 learned_addr learned my addr 172.21.15.116:0/3532946313 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:00:58.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.770+0000 7f43ae59c640 1 --2- 172.21.15.116:0/3532946313 >> v2:172.21.15.116:3300/0 conn(0x7f43b0150d10 0x7f43b01541f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:58.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.770+0000 7f43add9b640 1 --2- 172.21.15.116:0/3532946313 >> v2:172.21.15.174:3300/0 conn(0x7f43b00c2b10 0x7f43b00a94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:58.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.770+0000 7f43ad59a640 1 -- 172.21.15.116:0/3532946313 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2284137500 0 0) 0x7f43b00734b0 con 0x7f43b00a99e0 2024-04-25T00:00:58.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.770+0000 7f43ad59a640 1 -- 172.21.15.116:0/3532946313 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4398003530 con 0x7f43b00a99e0 2024-04-25T00:00:58.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.770+0000 7f43ae59c640 1 -- 172.21.15.116:0/3532946313 >> v2:172.21.15.174:3300/0 conn(0x7f43b00c2b10 msgr2=0x7f43b00a94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:00:58.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.770+0000 7f43ae59c640 1 --2- 172.21.15.116:0/3532946313 >> v2:172.21.15.174:3300/0 conn(0x7f43b00c2b10 0x7f43b00a94a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:58.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.770+0000 7f43ae59c640 1 -- 172.21.15.116:0/3532946313 >> v1:172.21.15.116:6789/0 conn(0x7f43b00a99e0 legacy=0x7f43b00c3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:58.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.770+0000 7f43ae59c640 1 -- 172.21.15.116:0/3532946313 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43b01539f0 con 0x7f43b0150d10 2024-04-25T00:00:58.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.774+0000 7f43ae59c640 1 --2- 172.21.15.116:0/3532946313 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43b0150d10 0x7f43b01541f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f43b0073700 tx=0x7f43a401c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc02f5a35c9079ce server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:00:58.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.774+0000 7f43ad59a640 1 -- 172.21.15.116:0/3532946313 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a40b0070 con 0x7f43b0150d10 2024-04-25T00:00:58.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.774+0000 7f43ad59a640 1 -- 172.21.15.116:0/3532946313 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43a40ab600 con 0x7f43b0150d10 2024-04-25T00:00:58.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.774+0000 7f43ad59a640 1 -- 172.21.15.116:0/3532946313 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a40ab9b0 con 0x7f43b0150d10 2024-04-25T00:00:58.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.774+0000 7f43b487a640 1 -- 172.21.15.116:0/3532946313 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43b0150d10 msgr2=0x7f43b01541f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:58.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.774+0000 7f43b487a640 1 --2- 172.21.15.116:0/3532946313 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43b0150d10 0x7f43b01541f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f43b0073700 tx=0x7f43a401c3f0 comp rx=0 tx=0).stop 2024-04-25T00:00:58.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.774+0000 7f43b487a640 1 -- 172.21.15.116:0/3532946313 shutdown_connections 2024-04-25T00:00:58.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.774+0000 7f43b487a640 1 --2- 172.21.15.116:0/3532946313 >> v2:172.21.15.174:3300/0 conn(0x7f43b00c2b10 0x7f43b00a94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:58.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.774+0000 7f43b487a640 1 --2- 172.21.15.116:0/3532946313 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43b0150d10 0x7f43b01541f0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:58.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.774+0000 7f43b487a640 1 -- 172.21.15.116:0/3532946313 >> 172.21.15.116:0/3532946313 conn(0x7f43b00c4c10 msgr2=0x7f43b00c5020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:00:58.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.774+0000 7f43b487a640 1 -- 172.21.15.116:0/3532946313 shutdown_connections 2024-04-25T00:00:58.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.774+0000 7f43b487a640 1 -- 172.21.15.116:0/3532946313 wait complete. 2024-04-25T00:00:58.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.774+0000 7f43b487a640 1 Processor -- start 2024-04-25T00:00:58.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.774+0000 7f43b487a640 1 -- start start 2024-04-25T00:00:58.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.774+0000 7f43b487a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43b00c2b10 0x7f43b0145c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:58.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.774+0000 7f43b487a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43b0150d10 0x7f43b00c7340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:58.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.774+0000 7f43b487a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43b0140630 0x7f43b00c7880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:58.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.774+0000 7f43b487a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f43b0151270 con 0x7f43b0140630 2024-04-25T00:00:58.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.774+0000 7f43b487a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f43b01510f0 con 0x7f43b0150d10 2024-04-25T00:00:58.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.774+0000 7f43b487a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f43b0160750 con 0x7f43b00c2b10 2024-04-25T00:00:58.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.774+0000 7f43aed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43b0140630 0x7f43b00c7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:58.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.774+0000 7f43aed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43b0140630 0x7f43b00c7880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43096/0 (socket says 172.21.15.116:43096) 2024-04-25T00:00:58.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.774+0000 7f43aed9d640 1 -- 172.21.15.116:0/1082798192 learned_addr learned my addr 172.21.15.116:0/1082798192 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:00:58.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.774+0000 7f43add9b640 1 --2- 172.21.15.116:0/1082798192 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43b0150d10 0x7f43b00c7340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:58.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.774+0000 7f43aed9d640 1 -- 172.21.15.116:0/1082798192 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43b00c2b10 msgr2=0x7f43b0145c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:58.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.774+0000 7f43aed9d640 1 --2- 172.21.15.116:0/1082798192 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43b00c2b10 0x7f43b0145c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:58.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.774+0000 7f43aed9d640 1 -- 172.21.15.116:0/1082798192 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43b0150d10 msgr2=0x7f43b00c7340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:58.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.774+0000 7f43aed9d640 1 --2- 172.21.15.116:0/1082798192 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43b0150d10 0x7f43b00c7340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:58.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.774+0000 7f43aed9d640 1 -- 172.21.15.116:0/1082798192 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43b0141510 con 0x7f43b0140630 2024-04-25T00:00:58.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.778+0000 7f43aed9d640 1 --2- 172.21.15.116:0/1082798192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43b0140630 0x7f43b00c7880 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f43a800ca70 tx=0x7f43a800cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:00:58.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.778+0000 7f439effd640 1 -- 172.21.15.116:0/1082798192 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a8017070 con 0x7f43b0140630 2024-04-25T00:00:58.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.778+0000 7f439effd640 1 -- 172.21.15.116:0/1082798192 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43a800abd0 con 0x7f43b0140630 2024-04-25T00:00:58.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.778+0000 7f43b487a640 1 -- 172.21.15.116:0/1082798192 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43b0141ff0 con 0x7f43b0140630 2024-04-25T00:00:58.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.778+0000 7f439effd640 1 -- 172.21.15.116:0/1082798192 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a8012400 con 0x7f43b0140630 2024-04-25T00:00:58.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.778+0000 7f43b487a640 1 -- 172.21.15.116:0/1082798192 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43b01461a0 con 0x7f43b0140630 2024-04-25T00:00:58.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.778+0000 7f43b487a640 1 -- 172.21.15.116:0/1082798192 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43b0068a60 con 0x7f43b0140630 2024-04-25T00:00:58.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.778+0000 7f439effd640 1 -- 172.21.15.116:0/1082798192 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f43a801a070 con 0x7f43b0140630 2024-04-25T00:00:58.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.778+0000 7f439effd640 1 --2- 172.21.15.116:0/1082798192 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f43900678c0 0x7f4390069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:58.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.778+0000 7f439effd640 1 -- 172.21.15.116:0/1082798192 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 4857+0+0 (secure 0 0 0) 0x7f43a80855b0 con 0x7f43b0140630 2024-04-25T00:00:58.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.778+0000 7f43ae59c640 1 --2- 172.21.15.116:0/1082798192 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f43900678c0 0x7f4390069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:58.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.778+0000 7f43ae59c640 1 --2- 172.21.15.116:0/1082798192 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f43900678c0 0x7f4390069d80 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f43a40ad000 tx=0x7f43a4004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:00:58.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.782+0000 7f439effd640 1 -- 172.21.15.116:0/1082798192 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f43a8055de0 con 0x7f43b0140630 2024-04-25T00:00:58.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.946+0000 7f43b487a640 1 -- 172.21.15.116:0/1082798192 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f43b01419f0 con 0x7f43b0140630 2024-04-25T00:00:58.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.946+0000 7f439effd640 1 -- 172.21.15.116:0/1082798192 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) ==== 74+0+12908 (secure 0 0 0) 0x7f43a8085030 con 0x7f43b0140630 2024-04-25T00:00:58.949 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:00:58.949 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":19,"fsid":"6d6a1aa7-0478-4c83-8057-05917e81f046","created":"2024-04-25T00:00:40.682493+0000","modified":"2024-04-25T00:00:56.079560+0000","last_up_change":"2024-04-25T00:00:51.822174+0000","last_in_change":"2024-04-25T00:00:42.980473+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T00:00:53.324376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8b6ddcb2-1ad9-4167-961c-bd6936bb9060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6809","nonce":3751765682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6811","nonce":3751765682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6815","nonce":3751765682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6813","nonce":3751765682}]},"public_addr":"172.21.15.116:6809/3751765682","cluster_addr":"172.21.15.116:6811/3751765682","heartbeat_back_addr":"172.21.15.116:6815/3751765682","heartbeat_front_addr":"172.21.15.116:6813/3751765682","state":["exists","up"]},{"osd":1,"uuid":"a6431768-994f-41da-8853-886f5687fb91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6801","nonce":377879124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6803","nonce":377879124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6807","nonce":377879124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6805","nonce":377879124}]},"public_addr":"172.21.15.116:6801/377879124","cluster_addr":"172.21.15.116:6803/377879124","heartbeat_back_addr":"172.21.15.116:6807/377879124","heartbeat_front_addr":"172.21.15.116:6805/377879124","state":["exists","up"]},{"osd":2,"uuid":"4f69bd40-5a23-45ff-8693-e1de75e1aa9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6817","nonce":1099282267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6819","nonce":1099282267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6823","nonce":1099282267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6821","nonce":1099282267}]},"public_addr":"172.21.15.116:6817/1099282267","cluster_addr":"172.21.15.116:6819/1099282267","heartbeat_back_addr":"172.21.15.116:6823/1099282267","heartbeat_front_addr":"172.21.15.116:6821/1099282267","state":["exists","up"]},{"osd":3,"uuid":"6a0d5344-8b6f-40d3-918d-0f0a4be688e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6825","nonce":1179942661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6827","nonce":1179942661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6831","nonce":1179942661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6829","nonce":1179942661}]},"public_addr":"172.21.15.116:6825/1179942661","cluster_addr":"172.21.15.116:6827/1179942661","heartbeat_back_addr":"172.21.15.116:6831/1179942661","heartbeat_front_addr":"172.21.15.116:6829/1179942661","state":["exists","up"]},{"osd":4,"uuid":"132e92e4-b825-441e-965a-ca00533fa86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6801","nonce":3132444330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6803","nonce":3132444330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6807","nonce":3132444330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6805","nonce":3132444330}]},"public_addr":"172.21.15.174:6801/3132444330","cluster_addr":"172.21.15.174:6803/3132444330","heartbeat_back_addr":"172.21.15.174:6807/3132444330","heartbeat_front_addr":"172.21.15.174:6805/3132444330","state":["exists","up"]},{"osd":5,"uuid":"a2eeb5b6-1b31-43cc-8306-a0adf3ee15bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6809","nonce":881817164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6811","nonce":881817164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6815","nonce":881817164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6813","nonce":881817164}]},"public_addr":"172.21.15.174:6809/881817164","cluster_addr":"172.21.15.174:6811/881817164","heartbeat_back_addr":"172.21.15.174:6815/881817164","heartbeat_front_addr":"172.21.15.174:6813/881817164","state":["exists","up"]},{"osd":6,"uuid":"588e873a-c841-4ba7-90f2-8956c87a9983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6825","nonce":117574251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6827","nonce":117574251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6831","nonce":117574251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6829","nonce":117574251}]},"public_addr":"172.21.15.174:6825/117574251","cluster_addr":"172.21.15.174:6827/117574251","heartbeat_back_addr":"172.21.15.174:6831/117574251","heartbeat_front_addr":"172.21.15.174:6829/117574251","state":["exists","up"]},{"osd":7,"uuid":"3b90f9b0-8a0e-4883-b088-ca1d4012870c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6817","nonce":2543899196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6819","nonce":2543899196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6823","nonce":2543899196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6821","nonce":2543899196}]},"public_addr":"172.21.15.174:6817/2543899196","cluster_addr":"172.21.15.174:6819/2543899196","heartbeat_back_addr":"172.21.15.174:6823/2543899196","heartbeat_front_addr":"172.21.15.174:6821/2543899196","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.293769+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.310034+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.404865+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.549658+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.192174+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.297266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.493649+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.452309+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:00:58.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.950+0000 7f43b487a640 1 -- 172.21.15.116:0/1082798192 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f43900678c0 msgr2=0x7f4390069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:58.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.950+0000 7f43b487a640 1 --2- 172.21.15.116:0/1082798192 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f43900678c0 0x7f4390069d80 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f43a40ad000 tx=0x7f43a4004740 comp rx=0 tx=0).stop 2024-04-25T00:00:58.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.950+0000 7f43b487a640 1 -- 172.21.15.116:0/1082798192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43b0140630 msgr2=0x7f43b00c7880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:58.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.950+0000 7f43b487a640 1 --2- 172.21.15.116:0/1082798192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43b0140630 0x7f43b00c7880 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f43a800ca70 tx=0x7f43a800cf30 comp rx=0 tx=0).stop 2024-04-25T00:00:58.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.950+0000 7f43b487a640 1 -- 172.21.15.116:0/1082798192 shutdown_connections 2024-04-25T00:00:58.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.950+0000 7f43b487a640 1 --2- 172.21.15.116:0/1082798192 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f43900678c0 0x7f4390069d80 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:58.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.950+0000 7f43b487a640 1 --2- 172.21.15.116:0/1082798192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43b0140630 0x7f43b00c7880 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:58.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.950+0000 7f43b487a640 1 --2- 172.21.15.116:0/1082798192 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43b0150d10 0x7f43b00c7340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:58.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.950+0000 7f43b487a640 1 --2- 172.21.15.116:0/1082798192 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43b00c2b10 0x7f43b0145c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:58.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.950+0000 7f43b487a640 1 -- 172.21.15.116:0/1082798192 >> 172.21.15.116:0/1082798192 conn(0x7f43b00c4c10 msgr2=0x7f43b00c2ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:00:58.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.950+0000 7f43b487a640 1 -- 172.21.15.116:0/1082798192 shutdown_connections 2024-04-25T00:00:58.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:58.950+0000 7f43b487a640 1 -- 172.21.15.116:0/1082798192 wait complete. 2024-04-25T00:00:58.967 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T00:00:53.324376+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T00:00:58.967 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T00:00:59.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.078+0000 7feab2f7c640 1 Processor -- start 2024-04-25T00:00:59.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.078+0000 7feab2f7c640 1 -- start start 2024-04-25T00:00:59.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.082+0000 7feab2f7c640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7feaac153130 0x7feaac1562e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:59.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.082+0000 7feab2f7c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7feaac0c9260 0x7feaac0c9710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:59.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.082+0000 7feab2f7c640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feaac156820 con 0x7feaac0ca1b0 2024-04-25T00:00:59.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.082+0000 7feab2f7c640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7feaac0ca820 con 0x7feaac153130 2024-04-25T00:00:59.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.082+0000 7feab2f7c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7feaac155e50 con 0x7feaac0c9260 2024-04-25T00:00:59.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.082+0000 7feaabfff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7feaac0ca1b0 0x7feaac06bdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:59458/0 (socket says 172.21.15.116:59458) 2024-04-25T00:00:59.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.082+0000 7feaabfff640 1 -- 172.21.15.116:0/3809665362 learned_addr learned my addr 172.21.15.116:0/3809665362 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:00:59.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.082+0000 7feab0d18640 1 --2- 172.21.15.116:0/3809665362 >> v2:172.21.15.174:3300/0 conn(0x7feaac153130 0x7feaac1562e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:59.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.082+0000 7feaab7fe640 1 -- 172.21.15.116:0/3809665362 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4284561141 0 0) 0x7feaac156820 con 0x7feaac0ca1b0 2024-04-25T00:00:59.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.082+0000 7feaab7fe640 1 -- 172.21.15.116:0/3809665362 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fea94003530 con 0x7feaac0ca1b0 2024-04-25T00:00:59.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.082+0000 7feaab7fe640 1 -- 172.21.15.116:0/3809665362 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1485473344 0 0) 0x7fea94003530 con 0x7feaac0ca1b0 2024-04-25T00:00:59.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.082+0000 7feaab7fe640 1 -- 172.21.15.116:0/3809665362 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7feaac0c9260 msgr2=0x7feaac0c9710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:00:59.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.082+0000 7feaab7fe640 1 --2- 172.21.15.116:0/3809665362 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7feaac0c9260 0x7feaac0c9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:59.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.082+0000 7feaab7fe640 1 -- 172.21.15.116:0/3809665362 >> v2:172.21.15.174:3300/0 conn(0x7feaac153130 msgr2=0x7feaac1562e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:00:59.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.082+0000 7feaab7fe640 1 --2- 172.21.15.116:0/3809665362 >> v2:172.21.15.174:3300/0 conn(0x7feaac153130 0x7feaac1562e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:59.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.082+0000 7feaab7fe640 1 -- 172.21.15.116:0/3809665362 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feaac0caa70 con 0x7feaac0ca1b0 2024-04-25T00:00:59.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.082+0000 7feaab7fe640 1 -- 172.21.15.116:0/3809665362 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7feaa0001680 con 0x7feaac0ca1b0 2024-04-25T00:00:59.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.082+0000 7feaab7fe640 1 -- 172.21.15.116:0/3809665362 >> v1:172.21.15.116:6789/0 conn(0x7feaac0ca1b0 legacy=0x7feaac06bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:59.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.082+0000 7feaab7fe640 1 --2- 172.21.15.116:0/3809665362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fea940039c0 0x7fea94003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:59.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.082+0000 7feaab7fe640 1 --2- 172.21.15.116:0/3809665362 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7feaac0c9260 0x7fea94004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:59.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.082+0000 7feaab7fe640 1 --2- 172.21.15.116:0/3809665362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea94004da0 0x7fea94005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:59.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.082+0000 7feaab7fe640 1 -- 172.21.15.116:0/3809665362 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7feaac0ca820 con 0x7fea94004da0 2024-04-25T00:00:59.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.082+0000 7feaab7fe640 1 -- 172.21.15.116:0/3809665362 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7feaac155e50 con 0x7fea940039c0 2024-04-25T00:00:59.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.082+0000 7feaab7fe640 1 -- 172.21.15.116:0/3809665362 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fea9400bce0 con 0x7feaac0c9260 2024-04-25T00:00:59.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.082+0000 7feab1519640 1 --2- 172.21.15.116:0/3809665362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea94004da0 0x7fea94005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:59.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.082+0000 7feab0d18640 1 --2- 172.21.15.116:0/3809665362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fea940039c0 0x7fea94003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:59.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.082+0000 7feab1519640 1 -- 172.21.15.116:0/3809665362 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7feaac0c9260 msgr2=0x7fea94004810 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T00:00:59.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.082+0000 7feab1519640 1 --2- 172.21.15.116:0/3809665362 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7feaac0c9260 0x7fea94004810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:59.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.082+0000 7feab1519640 1 -- 172.21.15.116:0/3809665362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fea940039c0 msgr2=0x7fea94003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:00:59.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.082+0000 7feab1519640 1 --2- 172.21.15.116:0/3809665362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fea940039c0 0x7fea94003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:59.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.082+0000 7feab1519640 1 -- 172.21.15.116:0/3809665362 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea94003530 con 0x7fea94004da0 2024-04-25T00:00:59.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.086+0000 7feab1519640 1 --2- 172.21.15.116:0/3809665362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea94004da0 0x7fea94005220 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7feaa4009ca0 tx=0x7feaa401cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28a60517b4992c49 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:00:59.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.086+0000 7feaab7fe640 1 -- 172.21.15.116:0/3809665362 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaa40b2070 con 0x7fea94004da0 2024-04-25T00:00:59.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.086+0000 7feaab7fe640 1 -- 172.21.15.116:0/3809665362 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feaa40a5b60 con 0x7fea94004da0 2024-04-25T00:00:59.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.086+0000 7feaab7fe640 1 -- 172.21.15.116:0/3809665362 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaa4006e60 con 0x7fea94004da0 2024-04-25T00:00:59.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.086+0000 7feab2f7c640 1 -- 172.21.15.116:0/3809665362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea94004da0 msgr2=0x7fea94005220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:59.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.086+0000 7feab2f7c640 1 --2- 172.21.15.116:0/3809665362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fea94004da0 0x7fea94005220 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7feaa4009ca0 tx=0x7feaa401cc00 comp rx=0 tx=0).stop 2024-04-25T00:00:59.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.086+0000 7feab1519640 1 -- 172.21.15.116:0/3809665362 reap_dead start 2024-04-25T00:00:59.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.086+0000 7feab2f7c640 1 -- 172.21.15.116:0/3809665362 shutdown_connections 2024-04-25T00:00:59.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.086+0000 7feab2f7c640 1 -- 172.21.15.116:0/3809665362 >> 172.21.15.116:0/3809665362 conn(0x7feaac0cb310 msgr2=0x7feaac0cb720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:00:59.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.086+0000 7feab2f7c640 1 -- 172.21.15.116:0/3809665362 shutdown_connections 2024-04-25T00:00:59.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.086+0000 7feab2f7c640 1 -- 172.21.15.116:0/3809665362 wait complete. 2024-04-25T00:00:59.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.086+0000 7feab2f7c640 1 Processor -- start 2024-04-25T00:00:59.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.086+0000 7feab2f7c640 1 -- start start 2024-04-25T00:00:59.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.086+0000 7feab2f7c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feaac1ee370 0x7feaac1ee750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:59.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.086+0000 7feab2f7c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7feaac14c840 0x7feaac1496f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:59.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.086+0000 7feab2f7c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feaac1eec90 0x7feaac149c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:59.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.086+0000 7feab2f7c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7feaac1602f0 con 0x7feaac1eec90 2024-04-25T00:00:59.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.086+0000 7feab2f7c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7feaac160170 con 0x7feaac1ee370 2024-04-25T00:00:59.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.086+0000 7feab2f7c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7feaac160470 con 0x7feaac14c840 2024-04-25T00:00:59.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.086+0000 7feab1519640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feaac1eec90 0x7feaac149c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:59.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.086+0000 7feab1519640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feaac1eec90 0x7feaac149c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43102/0 (socket says 172.21.15.116:43102) 2024-04-25T00:00:59.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.086+0000 7feab1519640 1 -- 172.21.15.116:0/4289833415 learned_addr learned my addr 172.21.15.116:0/4289833415 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:00:59.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.086+0000 7feab1519640 1 -- 172.21.15.116:0/4289833415 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7feaac14c840 msgr2=0x7feaac1496f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:59.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.086+0000 7feab1519640 1 --2- 172.21.15.116:0/4289833415 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7feaac14c840 0x7feaac1496f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:59.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.086+0000 7feab1519640 1 -- 172.21.15.116:0/4289833415 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feaac1ee370 msgr2=0x7feaac1ee750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:59.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.086+0000 7feab0d18640 1 --2- 172.21.15.116:0/4289833415 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feaac1ee370 0x7feaac1ee750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:59.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.086+0000 7feab1519640 1 --2- 172.21.15.116:0/4289833415 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feaac1ee370 0x7feaac1ee750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:59.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.086+0000 7feab1519640 1 -- 172.21.15.116:0/4289833415 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feaac14d0c0 con 0x7feaac1eec90 2024-04-25T00:00:59.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.086+0000 7feab1519640 1 --2- 172.21.15.116:0/4289833415 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feaac1eec90 0x7feaac149c30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7feaa401d4a0 tx=0x7fea94005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:00:59.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.086+0000 7feaa97fa640 1 -- 172.21.15.116:0/4289833415 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaa40b2040 con 0x7feaac1eec90 2024-04-25T00:00:59.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.086+0000 7feaa97fa640 1 -- 172.21.15.116:0/4289833415 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feaa40a5940 con 0x7feaac1eec90 2024-04-25T00:00:59.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.086+0000 7feab2f7c640 1 -- 172.21.15.116:0/4289833415 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feaac156c40 con 0x7feaac1eec90 2024-04-25T00:00:59.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.086+0000 7feaa97fa640 1 -- 172.21.15.116:0/4289833415 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaa4006e60 con 0x7feaac1eec90 2024-04-25T00:00:59.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.086+0000 7feab2f7c640 1 -- 172.21.15.116:0/4289833415 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feaac157200 con 0x7feaac1eec90 2024-04-25T00:00:59.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.090+0000 7feab2f7c640 1 -- 172.21.15.116:0/4289833415 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feaac04f510 con 0x7feaac1eec90 2024-04-25T00:00:59.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.090+0000 7feaa97fa640 1 -- 172.21.15.116:0/4289833415 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7feaa40a5d10 con 0x7feaac1eec90 2024-04-25T00:00:59.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.090+0000 7feaa97fa640 1 --2- 172.21.15.116:0/4289833415 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fea980678f0 0x7fea98069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:59.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.090+0000 7feaa97fa640 1 -- 172.21.15.116:0/4289833415 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 4857+0+0 (secure 0 0 0) 0x7feaa411efa0 con 0x7feaac1eec90 2024-04-25T00:00:59.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.090+0000 7feab0d18640 1 --2- 172.21.15.116:0/4289833415 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fea980678f0 0x7fea98069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:59.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.094+0000 7feab0d18640 1 --2- 172.21.15.116:0/4289833415 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fea980678f0 0x7fea98069db0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fea9c00b890 tx=0x7fea9c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:00:59.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.094+0000 7feaa97fa640 1 -- 172.21.15.116:0/4289833415 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7feaa40bd030 con 0x7feaac1eec90 2024-04-25T00:00:59.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.246+0000 7feab2f7c640 1 -- 172.21.15.116:0/4289833415 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7feaac06da40 con 0x7feaac1eec90 2024-04-25T00:00:59.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.246+0000 7feaa97fa640 1 -- 172.21.15.116:0/4289833415 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v19) ==== 93+0+10 (secure 0 0 0) 0x7feaa4008030 con 0x7feaac1eec90 2024-04-25T00:00:59.249 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-04-25T00:00:59.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.250+0000 7feab2f7c640 1 -- 172.21.15.116:0/4289833415 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fea980678f0 msgr2=0x7fea98069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:59.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.250+0000 7feab2f7c640 1 --2- 172.21.15.116:0/4289833415 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fea980678f0 0x7fea98069db0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fea9c00b890 tx=0x7fea9c008040 comp rx=0 tx=0).stop 2024-04-25T00:00:59.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.250+0000 7feab2f7c640 1 -- 172.21.15.116:0/4289833415 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feaac1eec90 msgr2=0x7feaac149c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:59.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.250+0000 7feab2f7c640 1 --2- 172.21.15.116:0/4289833415 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feaac1eec90 0x7feaac149c30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7feaa401d4a0 tx=0x7fea94005840 comp rx=0 tx=0).stop 2024-04-25T00:00:59.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.250+0000 7feab2f7c640 1 -- 172.21.15.116:0/4289833415 shutdown_connections 2024-04-25T00:00:59.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.250+0000 7feab2f7c640 1 --2- 172.21.15.116:0/4289833415 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fea980678f0 0x7fea98069db0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:59.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.250+0000 7feab2f7c640 1 --2- 172.21.15.116:0/4289833415 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feaac1eec90 0x7feaac149c30 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:59.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.250+0000 7feab2f7c640 1 --2- 172.21.15.116:0/4289833415 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7feaac14c840 0x7feaac1496f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:59.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.250+0000 7feab2f7c640 1 --2- 172.21.15.116:0/4289833415 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feaac1ee370 0x7feaac1ee750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:59.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.250+0000 7feab2f7c640 1 -- 172.21.15.116:0/4289833415 >> 172.21.15.116:0/4289833415 conn(0x7feaac0cb310 msgr2=0x7feaac1550d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:00:59.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.250+0000 7feab2f7c640 1 -- 172.21.15.116:0/4289833415 shutdown_connections 2024-04-25T00:00:59.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.250+0000 7feab2f7c640 1 -- 172.21.15.116:0/4289833415 wait complete. 2024-04-25T00:00:59.267 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:00:59.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.378+0000 7f6cd7407640 1 Processor -- start 2024-04-25T00:00:59.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.378+0000 7f6cd7407640 1 -- start start 2024-04-25T00:00:59.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.382+0000 7f6cd7407640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6cd0070dd0 0x7f6cd0071270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:59.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.382+0000 7f6cd7407640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6cd0152c10 con 0x7f6cd006e890 2024-04-25T00:00:59.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.382+0000 7f6cd7407640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6cd0151e50 con 0x7f6cd0148890 2024-04-25T00:00:59.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.382+0000 7f6cd7407640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6cd0152130 con 0x7f6cd0070dd0 2024-04-25T00:00:59.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.382+0000 7f6cd59a4640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f6cd006e890 0x7f6cd00717b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:59470/0 (socket says 172.21.15.116:59470) 2024-04-25T00:00:59.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.382+0000 7f6cd59a4640 1 -- 172.21.15.116:0/441025828 learned_addr learned my addr 172.21.15.116:0/441025828 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:00:59.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.382+0000 7f6cc7fff640 1 -- 172.21.15.116:0/441025828 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2991477350 0 0) 0x7f6cd0152c10 con 0x7f6cd006e890 2024-04-25T00:00:59.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.382+0000 7f6cc7fff640 1 -- 172.21.15.116:0/441025828 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6cbc003450 con 0x7f6cd006e890 2024-04-25T00:00:59.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.382+0000 7f6cd49a2640 1 --2- 172.21.15.116:0/441025828 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6cd0070dd0 0x7f6cd0071270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:59.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.382+0000 7f6cc7fff640 1 -- 172.21.15.116:0/441025828 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2998869286 0 0) 0x7f6cbc003450 con 0x7f6cd006e890 2024-04-25T00:00:59.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.382+0000 7f6cc7fff640 1 -- 172.21.15.116:0/441025828 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6cd0070dd0 msgr2=0x7f6cd0071270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:00:59.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.382+0000 7f6cc7fff640 1 --2- 172.21.15.116:0/441025828 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6cd0070dd0 0x7f6cd0071270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:59.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.382+0000 7f6cc7fff640 1 -- 172.21.15.116:0/441025828 >> v1:172.21.15.174:6789/0 conn(0x7f6cd0148890 legacy=0x7f6cd0072590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:59.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.382+0000 7f6cc7fff640 1 -- 172.21.15.116:0/441025828 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cd0152410 con 0x7f6cd006e890 2024-04-25T00:00:59.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.382+0000 7f6cc7fff640 1 -- 172.21.15.116:0/441025828 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f6cc8001a00 con 0x7f6cd006e890 2024-04-25T00:00:59.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.382+0000 7f6cc7fff640 1 -- 172.21.15.116:0/441025828 >> v1:172.21.15.116:6789/0 conn(0x7f6cd006e890 legacy=0x7f6cd00717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:59.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.382+0000 7f6cc7fff640 1 --2- 172.21.15.116:0/441025828 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6cbc003ae0 0x7f6cbc003f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:59.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.382+0000 7f6cc7fff640 1 --2- 172.21.15.116:0/441025828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cbc0088d0 0x7f6cbc00ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:59.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.382+0000 7f6cc7fff640 1 --2- 172.21.15.116:0/441025828 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6cd0070dd0 0x7f6cbc00d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:59.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.382+0000 7f6cc7fff640 1 -- 172.21.15.116:0/441025828 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6cd0152130 con 0x7f6cbc0088d0 2024-04-25T00:00:59.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.382+0000 7f6cc7fff640 1 -- 172.21.15.116:0/441025828 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6cbc00dd40 con 0x7f6cbc003ae0 2024-04-25T00:00:59.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.382+0000 7f6cc7fff640 1 -- 172.21.15.116:0/441025828 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6cbc00df40 con 0x7f6cd0070dd0 2024-04-25T00:00:59.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.382+0000 7f6cd49a2640 1 --2- 172.21.15.116:0/441025828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cbc0088d0 0x7f6cbc00ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:59.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.382+0000 7f6cd59a4640 1 --2- 172.21.15.116:0/441025828 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6cd0070dd0 0x7f6cbc00d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:59.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.382+0000 7f6cd51a3640 1 --2- 172.21.15.116:0/441025828 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6cbc003ae0 0x7f6cbc003f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:59.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.382+0000 7f6cd49a2640 1 -- 172.21.15.116:0/441025828 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6cd0070dd0 msgr2=0x7f6cbc00d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:00:59.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.382+0000 7f6cd49a2640 1 --2- 172.21.15.116:0/441025828 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6cd0070dd0 0x7f6cbc00d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:59.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.382+0000 7f6cd49a2640 1 -- 172.21.15.116:0/441025828 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6cbc003ae0 msgr2=0x7f6cbc003f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:00:59.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.382+0000 7f6cd49a2640 1 --2- 172.21.15.116:0/441025828 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6cbc003ae0 0x7f6cbc003f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:59.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.382+0000 7f6cd49a2640 1 -- 172.21.15.116:0/441025828 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cbc003450 con 0x7f6cbc0088d0 2024-04-25T00:00:59.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.382+0000 7f6cd49a2640 1 --2- 172.21.15.116:0/441025828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cbc0088d0 0x7f6cbc00ad60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6cc000ba30 tx=0x7f6cc001c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57911ac290a4a424 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:00:59.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.382+0000 7f6cc7fff640 1 -- 172.21.15.116:0/441025828 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc000b600 con 0x7f6cbc0088d0 2024-04-25T00:00:59.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.382+0000 7f6cc7fff640 1 -- 172.21.15.116:0/441025828 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cc00b1070 con 0x7f6cbc0088d0 2024-04-25T00:00:59.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.382+0000 7f6cc7fff640 1 -- 172.21.15.116:0/441025828 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc00ac680 con 0x7f6cbc0088d0 2024-04-25T00:00:59.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.382+0000 7f6cd7407640 1 -- 172.21.15.116:0/441025828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cbc0088d0 msgr2=0x7f6cbc00ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:59.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.382+0000 7f6cd7407640 1 --2- 172.21.15.116:0/441025828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cbc0088d0 0x7f6cbc00ad60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6cc000ba30 tx=0x7f6cc001c6a0 comp rx=0 tx=0).stop 2024-04-25T00:00:59.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.382+0000 7f6cd59a4640 1 -- 172.21.15.116:0/441025828 reap_dead start 2024-04-25T00:00:59.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.382+0000 7f6cd7407640 1 -- 172.21.15.116:0/441025828 shutdown_connections 2024-04-25T00:00:59.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.382+0000 7f6cd7407640 1 -- 172.21.15.116:0/441025828 >> 172.21.15.116:0/441025828 conn(0x7f6cd00c3b00 msgr2=0x7f6cd00c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:00:59.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.386+0000 7f6cd7407640 1 -- 172.21.15.116:0/441025828 shutdown_connections 2024-04-25T00:00:59.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.386+0000 7f6cd7407640 1 -- 172.21.15.116:0/441025828 wait complete. 2024-04-25T00:00:59.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.386+0000 7f6cd7407640 1 Processor -- start 2024-04-25T00:00:59.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.386+0000 7f6cd7407640 1 -- start start 2024-04-25T00:00:59.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.386+0000 7f6cd7407640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6cbc0088d0 0x7f6cd014ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:59.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.386+0000 7f6cd7407640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cd0145ce0 0x7f6cd0153130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:59.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.386+0000 7f6cd7407640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6cd01453d0 0x7f6cd00c4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:59.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.386+0000 7f6cd7407640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6cd0072ca0 con 0x7f6cd0145ce0 2024-04-25T00:00:59.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.386+0000 7f6cd7407640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6cd006ed70 con 0x7f6cd01453d0 2024-04-25T00:00:59.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.386+0000 7f6cd7407640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6cd0072e20 con 0x7f6cbc0088d0 2024-04-25T00:00:59.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.386+0000 7f6cd49a2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cd0145ce0 0x7f6cd0153130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:59.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.386+0000 7f6cd49a2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cd0145ce0 0x7f6cd0153130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43116/0 (socket says 172.21.15.116:43116) 2024-04-25T00:00:59.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.386+0000 7f6cd49a2640 1 -- 172.21.15.116:0/2280248405 learned_addr learned my addr 172.21.15.116:0/2280248405 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:00:59.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.386+0000 7f6cd49a2640 1 -- 172.21.15.116:0/2280248405 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6cbc0088d0 msgr2=0x7f6cd014ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:59.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.386+0000 7f6cd49a2640 1 --2- 172.21.15.116:0/2280248405 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6cbc0088d0 0x7f6cd014ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:59.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.386+0000 7f6cd49a2640 1 -- 172.21.15.116:0/2280248405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6cd01453d0 msgr2=0x7f6cd00c4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:59.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.386+0000 7f6cd49a2640 1 --2- 172.21.15.116:0/2280248405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6cd01453d0 0x7f6cd00c4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:59.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.386+0000 7f6cd49a2640 1 -- 172.21.15.116:0/2280248405 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cd01482a0 con 0x7f6cd0145ce0 2024-04-25T00:00:59.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.386+0000 7f6cd49a2640 1 --2- 172.21.15.116:0/2280248405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cd0145ce0 0x7f6cd0153130 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6cc00ae040 tx=0x7f6cbc003740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:00:59.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.386+0000 7f6cc5ffb640 1 -- 172.21.15.116:0/2280248405 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc00bb070 con 0x7f6cd0145ce0 2024-04-25T00:00:59.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.386+0000 7f6cc5ffb640 1 -- 172.21.15.116:0/2280248405 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cc00acc80 con 0x7f6cd0145ce0 2024-04-25T00:00:59.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.386+0000 7f6cd7407640 1 -- 172.21.15.116:0/2280248405 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cd01507e0 con 0x7f6cd0145ce0 2024-04-25T00:00:59.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.386+0000 7f6cc5ffb640 1 -- 172.21.15.116:0/2280248405 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc0004240 con 0x7f6cd0145ce0 2024-04-25T00:00:59.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.386+0000 7f6cd7407640 1 -- 172.21.15.116:0/2280248405 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cd0152bf0 con 0x7f6cd0145ce0 2024-04-25T00:00:59.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.386+0000 7f6cd7407640 1 -- 172.21.15.116:0/2280248405 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cd0068350 con 0x7f6cd0145ce0 2024-04-25T00:00:59.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.386+0000 7f6cc5ffb640 1 -- 172.21.15.116:0/2280248405 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6cc0003620 con 0x7f6cd0145ce0 2024-04-25T00:00:59.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.390+0000 7f6cc5ffb640 1 --2- 172.21.15.116:0/2280248405 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6cb40678f0 0x7f6cb4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:00:59.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.390+0000 7f6cc5ffb640 1 -- 172.21.15.116:0/2280248405 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 4857+0+0 (secure 0 0 0) 0x7f6cc01200b0 con 0x7f6cd0145ce0 2024-04-25T00:00:59.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.390+0000 7f6cd51a3640 1 --2- 172.21.15.116:0/2280248405 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6cb40678f0 0x7f6cb4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:00:59.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.394+0000 7f6cd51a3640 1 --2- 172.21.15.116:0/2280248405 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6cb40678f0 0x7f6cb4069db0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc007fd0 tx=0x7f6ccc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:00:59.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.394+0000 7f6cc5ffb640 1 -- 172.21.15.116:0/2280248405 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f6cc00163b0 con 0x7f6cd0145ce0 2024-04-25T00:00:59.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.570+0000 7f6cd7407640 1 -- 172.21.15.116:0/2280248405 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6cd006e670 con 0x7f6cd0145ce0 2024-04-25T00:00:59.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.570+0000 7f6cc5ffb640 1 -- 172.21.15.116:0/2280248405 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1392 (secure 0 0 0) 0x7f6cc00f0fd0 con 0x7f6cd0145ce0 2024-04-25T00:00:59.573 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:00:59.573 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/174781672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":174781672},{"type":"v1","addr":"172.21.15.116:6835","nonce":174781672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-25T00:00:59.574 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 2 2024-04-25T00:00:59.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.574+0000 7f6cd7407640 1 -- 172.21.15.116:0/2280248405 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6cb40678f0 msgr2=0x7f6cb4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:59.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.574+0000 7f6cd7407640 1 --2- 172.21.15.116:0/2280248405 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6cb40678f0 0x7f6cb4069db0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc007fd0 tx=0x7f6ccc009040 comp rx=0 tx=0).stop 2024-04-25T00:00:59.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.574+0000 7f6cd7407640 1 -- 172.21.15.116:0/2280248405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cd0145ce0 msgr2=0x7f6cd0153130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:00:59.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.574+0000 7f6cd7407640 1 --2- 172.21.15.116:0/2280248405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cd0145ce0 0x7f6cd0153130 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6cc00ae040 tx=0x7f6cbc003740 comp rx=0 tx=0).stop 2024-04-25T00:00:59.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.574+0000 7f6cd7407640 1 -- 172.21.15.116:0/2280248405 shutdown_connections 2024-04-25T00:00:59.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.574+0000 7f6cd7407640 1 --2- 172.21.15.116:0/2280248405 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6cb40678f0 0x7f6cb4069db0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:59.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.574+0000 7f6cd7407640 1 --2- 172.21.15.116:0/2280248405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6cd01453d0 0x7f6cd00c4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:59.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.574+0000 7f6cd7407640 1 --2- 172.21.15.116:0/2280248405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cd0145ce0 0x7f6cd0153130 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:59.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.574+0000 7f6cd7407640 1 --2- 172.21.15.116:0/2280248405 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6cbc0088d0 0x7f6cd014ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:00:59.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.574+0000 7f6cd7407640 1 -- 172.21.15.116:0/2280248405 >> 172.21.15.116:0/2280248405 conn(0x7f6cd00c3b00 msgr2=0x7f6cd00c20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:00:59.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.574+0000 7f6cd7407640 1 -- 172.21.15.116:0/2280248405 shutdown_connections 2024-04-25T00:00:59.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:00:59.574+0000 7f6cd7407640 1 -- 172.21.15.116:0/2280248405 wait complete. 2024-04-25T00:01:01.592 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:01:01.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.718+0000 7f7f29fdb640 1 Processor -- start 2024-04-25T00:01:01.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.722+0000 7f7f29fdb640 1 -- start start 2024-04-25T00:01:01.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.722+0000 7f7f29fdb640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7f240a94a0 0x7f7f241563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:01.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.722+0000 7f7f29fdb640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7f24071680 0x7f7f24071ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:01.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.722+0000 7f7f29fdb640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f24071220 con 0x7f7f2414c720 2024-04-25T00:01:01.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.722+0000 7f7f29fdb640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f7f24153460 con 0x7f7f24071680 2024-04-25T00:01:01.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.722+0000 7f7f29fdb640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f7f24152420 con 0x7f7f240a94a0 2024-04-25T00:01:01.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.722+0000 7f7f237fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f7f2414c720 0x7f7f240c2a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:59480/0 (socket says 172.21.15.116:59480) 2024-04-25T00:01:01.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.722+0000 7f7f237fe640 1 -- 172.21.15.116:0/2462889987 learned_addr learned my addr 172.21.15.116:0/2462889987 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:01.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.722+0000 7f7f23fff640 1 --2- 172.21.15.116:0/2462889987 >> v2:172.21.15.116:3300/0 conn(0x7f7f24071680 0x7f7f24071ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:01.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.722+0000 7f7f227fc640 1 -- 172.21.15.116:0/2462889987 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3899755329 0 0) 0x7f7f24071220 con 0x7f7f2414c720 2024-04-25T00:01:01.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.722+0000 7f7f227fc640 1 -- 172.21.15.116:0/2462889987 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f0c003530 con 0x7f7f2414c720 2024-04-25T00:01:01.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.722+0000 7f7f23fff640 1 -- 172.21.15.116:0/2462889987 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7f240a94a0 msgr2=0x7f7f241563d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:01.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.722+0000 7f7f22ffd640 1 --2- 172.21.15.116:0/2462889987 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7f240a94a0 0x7f7f241563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:01.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.722+0000 7f7f23fff640 1 --2- 172.21.15.116:0/2462889987 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7f240a94a0 0x7f7f241563d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:01.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.722+0000 7f7f23fff640 1 -- 172.21.15.116:0/2462889987 >> v1:172.21.15.116:6789/0 conn(0x7f7f2414c720 legacy=0x7f7f240c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:01.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.722+0000 7f7f23fff640 1 -- 172.21.15.116:0/2462889987 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f2414cb00 con 0x7f7f24071680 2024-04-25T00:01:01.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.722+0000 7f7f23fff640 1 --2- 172.21.15.116:0/2462889987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f24071680 0x7f7f24071ad0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7f140097c0 tx=0x7f7f1401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fba98a3080bd9e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:01.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.722+0000 7f7f227fc640 1 -- 172.21.15.116:0/2462889987 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f140b1070 con 0x7f7f24071680 2024-04-25T00:01:01.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.726+0000 7f7f227fc640 1 -- 172.21.15.116:0/2462889987 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f140a8050 con 0x7f7f24071680 2024-04-25T00:01:01.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.726+0000 7f7f227fc640 1 -- 172.21.15.116:0/2462889987 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f140ac840 con 0x7f7f24071680 2024-04-25T00:01:01.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.726+0000 7f7f29fdb640 1 -- 172.21.15.116:0/2462889987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f24071680 msgr2=0x7f7f24071ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:01.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.726+0000 7f7f29fdb640 1 --2- 172.21.15.116:0/2462889987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f24071680 0x7f7f24071ad0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7f140097c0 tx=0x7f7f1401c3e0 comp rx=0 tx=0).stop 2024-04-25T00:01:01.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.726+0000 7f7f29fdb640 1 -- 172.21.15.116:0/2462889987 shutdown_connections 2024-04-25T00:01:01.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.726+0000 7f7f29fdb640 1 --2- 172.21.15.116:0/2462889987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f24071680 0x7f7f24071ad0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:01.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.726+0000 7f7f29fdb640 1 --2- 172.21.15.116:0/2462889987 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7f240a94a0 0x7f7f241563d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:01.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.726+0000 7f7f29fdb640 1 -- 172.21.15.116:0/2462889987 >> 172.21.15.116:0/2462889987 conn(0x7f7f240c3b00 msgr2=0x7f7f240c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:01.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.726+0000 7f7f29fdb640 1 -- 172.21.15.116:0/2462889987 shutdown_connections 2024-04-25T00:01:01.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.726+0000 7f7f29fdb640 1 -- 172.21.15.116:0/2462889987 wait complete. 2024-04-25T00:01:01.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.726+0000 7f7f29fdb640 1 Processor -- start 2024-04-25T00:01:01.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.726+0000 7f7f29fdb640 1 -- start start 2024-04-25T00:01:01.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.726+0000 7f7f29fdb640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7f24071680 0x7f7f24148330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:01.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.726+0000 7f7f29fdb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f240a94a0 0x7f7f24147000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:01.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.726+0000 7f7f29fdb640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7f2414c720 0x7f7f24147540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:01.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.726+0000 7f7f29fdb640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7f240c3120 con 0x7f7f240a94a0 2024-04-25T00:01:01.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.726+0000 7f7f29fdb640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7f24152f40 con 0x7f7f24071680 2024-04-25T00:01:01.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.726+0000 7f7f29fdb640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f7f240c32a0 con 0x7f7f2414c720 2024-04-25T00:01:01.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.726+0000 7f7f22ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f240a94a0 0x7f7f24147000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:01.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.726+0000 7f7f22ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f240a94a0 0x7f7f24147000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43142/0 (socket says 172.21.15.116:43142) 2024-04-25T00:01:01.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.726+0000 7f7f22ffd640 1 -- 172.21.15.116:0/1817625120 learned_addr learned my addr 172.21.15.116:0/1817625120 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:01.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.726+0000 7f7f23fff640 1 --2- 172.21.15.116:0/1817625120 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7f2414c720 0x7f7f24147540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:01.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.726+0000 7f7f237fe640 1 --2- 172.21.15.116:0/1817625120 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7f24071680 0x7f7f24148330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:01.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.726+0000 7f7f22ffd640 1 -- 172.21.15.116:0/1817625120 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7f2414c720 msgr2=0x7f7f24147540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:01.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.726+0000 7f7f22ffd640 1 --2- 172.21.15.116:0/1817625120 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7f2414c720 0x7f7f24147540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:01.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.726+0000 7f7f22ffd640 1 -- 172.21.15.116:0/1817625120 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7f24071680 msgr2=0x7f7f24148330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:01.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.726+0000 7f7f22ffd640 1 --2- 172.21.15.116:0/1817625120 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7f24071680 0x7f7f24148330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:01.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.726+0000 7f7f22ffd640 1 -- 172.21.15.116:0/1817625120 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f240ac890 con 0x7f7f240a94a0 2024-04-25T00:01:01.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.726+0000 7f7f22ffd640 1 --2- 172.21.15.116:0/1817625120 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f240a94a0 0x7f7f24147000 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7f1c00ce10 tx=0x7f7f1c00a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:01.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.726+0000 7f7f0bfff640 1 -- 172.21.15.116:0/1817625120 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f1c003ba0 con 0x7f7f240a94a0 2024-04-25T00:01:01.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.726+0000 7f7f0bfff640 1 -- 172.21.15.116:0/1817625120 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f1c003d40 con 0x7f7f240a94a0 2024-04-25T00:01:01.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.726+0000 7f7f29fdb640 1 -- 172.21.15.116:0/1817625120 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f240acb70 con 0x7f7f240a94a0 2024-04-25T00:01:01.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.726+0000 7f7f0bfff640 1 -- 172.21.15.116:0/1817625120 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f1c004240 con 0x7f7f240a94a0 2024-04-25T00:01:01.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.726+0000 7f7f29fdb640 1 -- 172.21.15.116:0/1817625120 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f24153c60 con 0x7f7f240a94a0 2024-04-25T00:01:01.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.730+0000 7f7f29fdb640 1 -- 172.21.15.116:0/1817625120 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f24068350 con 0x7f7f240a94a0 2024-04-25T00:01:01.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.730+0000 7f7f0bfff640 1 -- 172.21.15.116:0/1817625120 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7f1c0043e0 con 0x7f7f240a94a0 2024-04-25T00:01:01.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.730+0000 7f7f0bfff640 1 --2- 172.21.15.116:0/1817625120 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7f100678c0 0x7f7f10069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:01.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.730+0000 7f7f0bfff640 1 -- 172.21.15.116:0/1817625120 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 4857+0+0 (secure 0 0 0) 0x7f7f1c086470 con 0x7f7f240a94a0 2024-04-25T00:01:01.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.730+0000 7f7f237fe640 1 --2- 172.21.15.116:0/1817625120 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7f100678c0 0x7f7f10069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:01.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.734+0000 7f7f237fe640 1 --2- 172.21.15.116:0/1817625120 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7f100678c0 0x7f7f10069d80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7f18005890 tx=0x7f7f18009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:01.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.734+0000 7f7f0bfff640 1 -- 172.21.15.116:0/1817625120 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f7f1c057310 con 0x7f7f240a94a0 2024-04-25T00:01:01.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.930+0000 7f7f29fdb640 1 -- 172.21.15.116:0/1817625120 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7f2406f310 con 0x7f7f240a94a0 2024-04-25T00:01:01.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.930+0000 7f7f0bfff640 1 -- 172.21.15.116:0/1817625120 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+3873 (secure 0 0 0) 0x7f7f1c073e20 con 0x7f7f240a94a0 2024-04-25T00:01:01.932 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:01:01.933 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6834/1459177921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1459177921},{"type":"v1","addr":"172.21.15.116:6834","nonce":1459177921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/174781672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":174781672},{"type":"v1","addr":"172.21.15.116:6835","nonce":174781672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4270,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/1762356055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1762356055},{"type":"v1","addr":"172.21.15.174:6837","nonce":1762356055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-25T00:01:01.933 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 3 2024-04-25T00:01:01.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.934+0000 7f7f29fdb640 1 -- 172.21.15.116:0/1817625120 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7f100678c0 msgr2=0x7f7f10069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:01.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.934+0000 7f7f29fdb640 1 --2- 172.21.15.116:0/1817625120 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7f100678c0 0x7f7f10069d80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7f18005890 tx=0x7f7f18009040 comp rx=0 tx=0).stop 2024-04-25T00:01:01.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.934+0000 7f7f29fdb640 1 -- 172.21.15.116:0/1817625120 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f240a94a0 msgr2=0x7f7f24147000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:01.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.934+0000 7f7f29fdb640 1 --2- 172.21.15.116:0/1817625120 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f240a94a0 0x7f7f24147000 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7f1c00ce10 tx=0x7f7f1c00a660 comp rx=0 tx=0).stop 2024-04-25T00:01:01.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.934+0000 7f7f29fdb640 1 -- 172.21.15.116:0/1817625120 shutdown_connections 2024-04-25T00:01:01.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.934+0000 7f7f29fdb640 1 --2- 172.21.15.116:0/1817625120 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7f100678c0 0x7f7f10069d80 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:01.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.934+0000 7f7f29fdb640 1 --2- 172.21.15.116:0/1817625120 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7f2414c720 0x7f7f24147540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:01.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.934+0000 7f7f29fdb640 1 --2- 172.21.15.116:0/1817625120 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f240a94a0 0x7f7f24147000 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:01.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.934+0000 7f7f29fdb640 1 --2- 172.21.15.116:0/1817625120 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7f24071680 0x7f7f24148330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:01.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.934+0000 7f7f29fdb640 1 -- 172.21.15.116:0/1817625120 >> 172.21.15.116:0/1817625120 conn(0x7f7f240c3b00 msgr2=0x7f7f241492e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:01.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.934+0000 7f7f29fdb640 1 -- 172.21.15.116:0/1817625120 shutdown_connections 2024-04-25T00:01:01.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:01.934+0000 7f7f29fdb640 1 -- 172.21.15.116:0/1817625120 wait complete. 2024-04-25T00:01:01.951 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T00:01:02.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.062+0000 7f5db6ed0640 1 Processor -- start 2024-04-25T00:01:02.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.062+0000 7f5db6ed0640 1 -- start start 2024-04-25T00:01:02.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.066+0000 7f5db6ed0640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5db00a94a0 0x7f5db01538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:02.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.066+0000 7f5db6ed0640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f5db00a99f0 0x7f5db0153e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:02.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.066+0000 7f5db6ed0640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5db00c3910 con 0x7f5db0150d20 2024-04-25T00:01:02.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.066+0000 7f5db6ed0640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f5db00c16e0 con 0x7f5db00a94a0 2024-04-25T00:01:02.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.066+0000 7f5db6ed0640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f5db00c1400 con 0x7f5db00a99f0 2024-04-25T00:01:02.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.066+0000 7f5daffff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5db00a94a0 0x7f5db01538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:02.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.066+0000 7f5db4c6c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f5db0150d20 0x7f5db00c2a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:59492/0 (socket says 172.21.15.116:59492) 2024-04-25T00:01:02.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.066+0000 7f5db4c6c640 1 -- 172.21.15.116:0/3735215618 learned_addr learned my addr 172.21.15.116:0/3735215618 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:02.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.066+0000 7f5daffff640 1 -- 172.21.15.116:0/3735215618 >> v2:172.21.15.174:3300/0 conn(0x7f5db00a99f0 msgr2=0x7f5db0153e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:02.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.066+0000 7f5daffff640 1 --2- 172.21.15.116:0/3735215618 >> v2:172.21.15.174:3300/0 conn(0x7f5db00a99f0 0x7f5db0153e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:02.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.066+0000 7f5daffff640 1 -- 172.21.15.116:0/3735215618 >> v1:172.21.15.116:6789/0 conn(0x7f5db0150d20 legacy=0x7f5db00c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:02.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.066+0000 7f5daffff640 1 -- 172.21.15.116:0/3735215618 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5db006e660 con 0x7f5db00a94a0 2024-04-25T00:01:02.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.066+0000 7f5daffff640 1 --2- 172.21.15.116:0/3735215618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5db00a94a0 0x7f5db01538e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5db0157b40 tx=0x7f5da401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4d8563edaaea670 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:02.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.066+0000 7f5daf7fe640 1 -- 172.21.15.116:0/3735215618 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da40b1070 con 0x7f5db00a94a0 2024-04-25T00:01:02.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.066+0000 7f5daf7fe640 1 -- 172.21.15.116:0/3735215618 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5da40ac5f0 con 0x7f5db00a94a0 2024-04-25T00:01:02.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.066+0000 7f5daf7fe640 1 -- 172.21.15.116:0/3735215618 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da40ac9c0 con 0x7f5db00a94a0 2024-04-25T00:01:02.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.070+0000 7f5db6ed0640 1 -- 172.21.15.116:0/3735215618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5db00a94a0 msgr2=0x7f5db01538e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:02.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.070+0000 7f5db6ed0640 1 --2- 172.21.15.116:0/3735215618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5db00a94a0 0x7f5db01538e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5db0157b40 tx=0x7f5da401c480 comp rx=0 tx=0).stop 2024-04-25T00:01:02.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.070+0000 7f5db6ed0640 1 -- 172.21.15.116:0/3735215618 shutdown_connections 2024-04-25T00:01:02.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.070+0000 7f5db6ed0640 1 --2- 172.21.15.116:0/3735215618 >> v2:172.21.15.174:3300/0 conn(0x7f5db00a99f0 0x7f5db0153e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:02.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.070+0000 7f5db6ed0640 1 --2- 172.21.15.116:0/3735215618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5db00a94a0 0x7f5db01538e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:02.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.070+0000 7f5db6ed0640 1 -- 172.21.15.116:0/3735215618 >> 172.21.15.116:0/3735215618 conn(0x7f5db00c3b00 msgr2=0x7f5db00c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:02.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.070+0000 7f5db6ed0640 1 -- 172.21.15.116:0/3735215618 shutdown_connections 2024-04-25T00:01:02.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.070+0000 7f5db6ed0640 1 -- 172.21.15.116:0/3735215618 wait complete. 2024-04-25T00:01:02.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.070+0000 7f5db6ed0640 1 Processor -- start 2024-04-25T00:01:02.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.070+0000 7f5db6ed0640 1 -- start start 2024-04-25T00:01:02.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.070+0000 7f5db6ed0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5db00a94a0 0x7f5db014c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:02.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.070+0000 7f5db6ed0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5db00a99f0 0x7f5db014d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:02.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.070+0000 7f5db6ed0640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5db0150d20 0x7f5db01408d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:02.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.070+0000 7f5db6ed0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5db01544e0 con 0x7f5db00a94a0 2024-04-25T00:01:02.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.070+0000 7f5db6ed0640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5db0154360 con 0x7f5db00a99f0 2024-04-25T00:01:02.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.070+0000 7f5db6ed0640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f5db0160730 con 0x7f5db0150d20 2024-04-25T00:01:02.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.070+0000 7f5db4c6c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5db00a94a0 0x7f5db014c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:02.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.070+0000 7f5db4c6c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5db00a94a0 0x7f5db014c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43172/0 (socket says 172.21.15.116:43172) 2024-04-25T00:01:02.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.070+0000 7f5db4c6c640 1 -- 172.21.15.116:0/2089884313 learned_addr learned my addr 172.21.15.116:0/2089884313 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:02.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.070+0000 7f5db4c6c640 1 -- 172.21.15.116:0/2089884313 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5db0150d20 msgr2=0x7f5db01408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:02.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.070+0000 7f5db4c6c640 1 --2- 172.21.15.116:0/2089884313 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5db0150d20 0x7f5db01408d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:02.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.070+0000 7f5daffff640 1 --2- 172.21.15.116:0/2089884313 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5db00a99f0 0x7f5db014d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:02.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.070+0000 7f5db4c6c640 1 -- 172.21.15.116:0/2089884313 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5db00a99f0 msgr2=0x7f5db014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:02.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.070+0000 7f5db4c6c640 1 --2- 172.21.15.116:0/2089884313 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5db00a99f0 0x7f5db014d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:02.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.070+0000 7f5db4c6c640 1 -- 172.21.15.116:0/2089884313 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5db014bae0 con 0x7f5db00a94a0 2024-04-25T00:01:02.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.070+0000 7f5db4c6c640 1 --2- 172.21.15.116:0/2089884313 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5db00a94a0 0x7f5db014c910 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5da000c420 tx=0x7f5da000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:02.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.070+0000 7f5dad7fa640 1 -- 172.21.15.116:0/2089884313 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da0017070 con 0x7f5db00a94a0 2024-04-25T00:01:02.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.070+0000 7f5dad7fa640 1 -- 172.21.15.116:0/2089884313 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5da0005cf0 con 0x7f5db00a94a0 2024-04-25T00:01:02.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.070+0000 7f5db6ed0640 1 -- 172.21.15.116:0/2089884313 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5db014bdc0 con 0x7f5db00a94a0 2024-04-25T00:01:02.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.070+0000 7f5dad7fa640 1 -- 172.21.15.116:0/2089884313 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da00072f0 con 0x7f5db00a94a0 2024-04-25T00:01:02.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.070+0000 7f5db6ed0640 1 -- 172.21.15.116:0/2089884313 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5db00c5de0 con 0x7f5db00a94a0 2024-04-25T00:01:02.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.070+0000 7f5db6ed0640 1 -- 172.21.15.116:0/2089884313 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5db0068350 con 0x7f5db00a94a0 2024-04-25T00:01:02.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.074+0000 7f5dad7fa640 1 -- 172.21.15.116:0/2089884313 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5da001a070 con 0x7f5db00a94a0 2024-04-25T00:01:02.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.074+0000 7f5dad7fa640 1 --2- 172.21.15.116:0/2089884313 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5d9c0678c0 0x7f5d9c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:02.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.074+0000 7f5dad7fa640 1 -- 172.21.15.116:0/2089884313 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 4857+0+0 (secure 0 0 0) 0x7f5da0084ca0 con 0x7f5db00a94a0 2024-04-25T00:01:02.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.074+0000 7f5daffff640 1 --2- 172.21.15.116:0/2089884313 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5d9c0678c0 0x7f5d9c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:02.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.074+0000 7f5daffff640 1 --2- 172.21.15.116:0/2089884313 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5d9c0678c0 0x7f5d9c069d80 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5da40ae000 tx=0x7f5da40a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:02.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.078+0000 7f5dad7fa640 1 -- 172.21.15.116:0/2089884313 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f5da0055ba0 con 0x7f5db00a94a0 2024-04-25T00:01:02.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.238+0000 7f5db6ed0640 1 -- 172.21.15.116:0/2089884313 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5db00c34f0 con 0x7f5db00a94a0 2024-04-25T00:01:02.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.242+0000 7f5dad7fa640 1 -- 172.21.15.116:0/2089884313 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) ==== 74+0+12908 (secure 0 0 0) 0x7f5da00554a0 con 0x7f5db00a94a0 2024-04-25T00:01:02.243 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:01:02.243 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":19,"fsid":"6d6a1aa7-0478-4c83-8057-05917e81f046","created":"2024-04-25T00:00:40.682493+0000","modified":"2024-04-25T00:00:56.079560+0000","last_up_change":"2024-04-25T00:00:51.822174+0000","last_in_change":"2024-04-25T00:00:42.980473+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T00:00:53.324376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8b6ddcb2-1ad9-4167-961c-bd6936bb9060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6809","nonce":3751765682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6811","nonce":3751765682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6815","nonce":3751765682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6813","nonce":3751765682}]},"public_addr":"172.21.15.116:6809/3751765682","cluster_addr":"172.21.15.116:6811/3751765682","heartbeat_back_addr":"172.21.15.116:6815/3751765682","heartbeat_front_addr":"172.21.15.116:6813/3751765682","state":["exists","up"]},{"osd":1,"uuid":"a6431768-994f-41da-8853-886f5687fb91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6801","nonce":377879124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6803","nonce":377879124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6807","nonce":377879124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6805","nonce":377879124}]},"public_addr":"172.21.15.116:6801/377879124","cluster_addr":"172.21.15.116:6803/377879124","heartbeat_back_addr":"172.21.15.116:6807/377879124","heartbeat_front_addr":"172.21.15.116:6805/377879124","state":["exists","up"]},{"osd":2,"uuid":"4f69bd40-5a23-45ff-8693-e1de75e1aa9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6817","nonce":1099282267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6819","nonce":1099282267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6823","nonce":1099282267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6821","nonce":1099282267}]},"public_addr":"172.21.15.116:6817/1099282267","cluster_addr":"172.21.15.116:6819/1099282267","heartbeat_back_addr":"172.21.15.116:6823/1099282267","heartbeat_front_addr":"172.21.15.116:6821/1099282267","state":["exists","up"]},{"osd":3,"uuid":"6a0d5344-8b6f-40d3-918d-0f0a4be688e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6825","nonce":1179942661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6827","nonce":1179942661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6831","nonce":1179942661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6829","nonce":1179942661}]},"public_addr":"172.21.15.116:6825/1179942661","cluster_addr":"172.21.15.116:6827/1179942661","heartbeat_back_addr":"172.21.15.116:6831/1179942661","heartbeat_front_addr":"172.21.15.116:6829/1179942661","state":["exists","up"]},{"osd":4,"uuid":"132e92e4-b825-441e-965a-ca00533fa86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6801","nonce":3132444330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6803","nonce":3132444330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6807","nonce":3132444330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6805","nonce":3132444330}]},"public_addr":"172.21.15.174:6801/3132444330","cluster_addr":"172.21.15.174:6803/3132444330","heartbeat_back_addr":"172.21.15.174:6807/3132444330","heartbeat_front_addr":"172.21.15.174:6805/3132444330","state":["exists","up"]},{"osd":5,"uuid":"a2eeb5b6-1b31-43cc-8306-a0adf3ee15bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6809","nonce":881817164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6811","nonce":881817164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6815","nonce":881817164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6813","nonce":881817164}]},"public_addr":"172.21.15.174:6809/881817164","cluster_addr":"172.21.15.174:6811/881817164","heartbeat_back_addr":"172.21.15.174:6815/881817164","heartbeat_front_addr":"172.21.15.174:6813/881817164","state":["exists","up"]},{"osd":6,"uuid":"588e873a-c841-4ba7-90f2-8956c87a9983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6825","nonce":117574251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6827","nonce":117574251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6831","nonce":117574251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6829","nonce":117574251}]},"public_addr":"172.21.15.174:6825/117574251","cluster_addr":"172.21.15.174:6827/117574251","heartbeat_back_addr":"172.21.15.174:6831/117574251","heartbeat_front_addr":"172.21.15.174:6829/117574251","state":["exists","up"]},{"osd":7,"uuid":"3b90f9b0-8a0e-4883-b088-ca1d4012870c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6817","nonce":2543899196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6819","nonce":2543899196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6823","nonce":2543899196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6821","nonce":2543899196}]},"public_addr":"172.21.15.174:6817/2543899196","cluster_addr":"172.21.15.174:6819/2543899196","heartbeat_back_addr":"172.21.15.174:6823/2543899196","heartbeat_front_addr":"172.21.15.174:6821/2543899196","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.293769+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.310034+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.404865+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.549658+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.192174+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.297266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.493649+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.452309+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:01:02.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.242+0000 7f5db6ed0640 1 -- 172.21.15.116:0/2089884313 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5d9c0678c0 msgr2=0x7f5d9c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:02.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.242+0000 7f5db6ed0640 1 --2- 172.21.15.116:0/2089884313 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5d9c0678c0 0x7f5d9c069d80 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5da40ae000 tx=0x7f5da40a7000 comp rx=0 tx=0).stop 2024-04-25T00:01:02.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.242+0000 7f5db6ed0640 1 -- 172.21.15.116:0/2089884313 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5db00a94a0 msgr2=0x7f5db014c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:02.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.242+0000 7f5db6ed0640 1 --2- 172.21.15.116:0/2089884313 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5db00a94a0 0x7f5db014c910 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5da000c420 tx=0x7f5da000c8e0 comp rx=0 tx=0).stop 2024-04-25T00:01:02.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.242+0000 7f5db6ed0640 1 -- 172.21.15.116:0/2089884313 shutdown_connections 2024-04-25T00:01:02.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.242+0000 7f5db6ed0640 1 --2- 172.21.15.116:0/2089884313 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5d9c0678c0 0x7f5d9c069d80 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:02.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.242+0000 7f5db6ed0640 1 --2- 172.21.15.116:0/2089884313 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5db0150d20 0x7f5db01408d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:02.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.242+0000 7f5db6ed0640 1 --2- 172.21.15.116:0/2089884313 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5db00a99f0 0x7f5db014d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:02.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.242+0000 7f5db6ed0640 1 --2- 172.21.15.116:0/2089884313 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5db00a94a0 0x7f5db014c910 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:02.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.242+0000 7f5db6ed0640 1 -- 172.21.15.116:0/2089884313 >> 172.21.15.116:0/2089884313 conn(0x7f5db00c3b00 msgr2=0x7f5db00a9f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:02.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.242+0000 7f5db6ed0640 1 -- 172.21.15.116:0/2089884313 shutdown_connections 2024-04-25T00:01:02.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.242+0000 7f5db6ed0640 1 -- 172.21.15.116:0/2089884313 wait complete. 2024-04-25T00:01:02.265 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T00:00:53.324376+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T00:01:02.266 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T00:01:02.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.390+0000 7f62f42cf640 1 Processor -- start 2024-04-25T00:01:02.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.390+0000 7f62f42cf640 1 -- start start 2024-04-25T00:01:02.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.390+0000 7f62f42cf640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f62ec0ca1b0 0x7f62ec06bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:02.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.390+0000 7f62f42cf640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f62ec0c9260 0x7f62ec0c9710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:02.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.390+0000 7f62f42cf640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62ec06c300 con 0x7f62ec153130 2024-04-25T00:01:02.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.390+0000 7f62f42cf640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f62ec0ca820 con 0x7f62ec0ca1b0 2024-04-25T00:01:02.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.390+0000 7f62f42cf640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f62ec155e50 con 0x7f62ec0c9260 2024-04-25T00:01:02.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.390+0000 7f62f186a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f62ec0ca1b0 0x7f62ec06bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:02.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.390+0000 7f62f206b640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f62ec153130 0x7f62ec1562e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:59504/0 (socket says 172.21.15.116:59504) 2024-04-25T00:01:02.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.390+0000 7f62f206b640 1 -- 172.21.15.116:0/2273638376 learned_addr learned my addr 172.21.15.116:0/2273638376 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:02.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.390+0000 7f62f186a640 1 -- 172.21.15.116:0/2273638376 >> v2:172.21.15.174:3300/0 conn(0x7f62ec0c9260 msgr2=0x7f62ec0c9710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:02.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.390+0000 7f62f186a640 1 --2- 172.21.15.116:0/2273638376 >> v2:172.21.15.174:3300/0 conn(0x7f62ec0c9260 0x7f62ec0c9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:02.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.390+0000 7f62f186a640 1 -- 172.21.15.116:0/2273638376 >> v1:172.21.15.116:6789/0 conn(0x7f62ec153130 legacy=0x7f62ec1562e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:02.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.390+0000 7f62f186a640 1 -- 172.21.15.116:0/2273638376 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62ec0caa70 con 0x7f62ec0ca1b0 2024-04-25T00:01:02.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.390+0000 7f62f1069640 1 -- 172.21.15.116:0/2273638376 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1767425437 0 0) 0x7f62ec06c300 con 0x7f62ec153130 2024-04-25T00:01:02.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.394+0000 7f62f186a640 1 --2- 172.21.15.116:0/2273638376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62ec0ca1b0 0x7f62ec06bdc0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f62ec06b9f0 tx=0x7f62dc01c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f420e97769178a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:02.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.394+0000 7f62f1069640 1 -- 172.21.15.116:0/2273638376 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62dc0b0070 con 0x7f62ec0ca1b0 2024-04-25T00:01:02.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.394+0000 7f62f1069640 1 -- 172.21.15.116:0/2273638376 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62dc0ab600 con 0x7f62ec0ca1b0 2024-04-25T00:01:02.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.394+0000 7f62f1069640 1 -- 172.21.15.116:0/2273638376 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62dc0ab9d0 con 0x7f62ec0ca1b0 2024-04-25T00:01:02.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.394+0000 7f62f42cf640 1 -- 172.21.15.116:0/2273638376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62ec0ca1b0 msgr2=0x7f62ec06bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:02.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.394+0000 7f62f42cf640 1 --2- 172.21.15.116:0/2273638376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62ec0ca1b0 0x7f62ec06bdc0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f62ec06b9f0 tx=0x7f62dc01c360 comp rx=0 tx=0).stop 2024-04-25T00:01:02.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.394+0000 7f62f42cf640 1 -- 172.21.15.116:0/2273638376 shutdown_connections 2024-04-25T00:01:02.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.394+0000 7f62f42cf640 1 --2- 172.21.15.116:0/2273638376 >> v2:172.21.15.174:3300/0 conn(0x7f62ec0c9260 0x7f62ec0c9710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:02.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.394+0000 7f62f42cf640 1 --2- 172.21.15.116:0/2273638376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62ec0ca1b0 0x7f62ec06bdc0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:02.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.394+0000 7f62f42cf640 1 -- 172.21.15.116:0/2273638376 >> 172.21.15.116:0/2273638376 conn(0x7f62ec0cb310 msgr2=0x7f62ec0cb720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:02.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.394+0000 7f62f42cf640 1 -- 172.21.15.116:0/2273638376 shutdown_connections 2024-04-25T00:01:02.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.394+0000 7f62f42cf640 1 -- 172.21.15.116:0/2273638376 wait complete. 2024-04-25T00:01:02.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.394+0000 7f62f42cf640 1 Processor -- start 2024-04-25T00:01:02.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.394+0000 7f62f42cf640 1 -- start start 2024-04-25T00:01:02.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.394+0000 7f62f42cf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62ec0c9260 0x7f62ec0b08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:02.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.394+0000 7f62f42cf640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f62ec0ca1b0 0x7f62ec159270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:02.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.394+0000 7f62f42cf640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f62ec153130 0x7f62ec0c2540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:02.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.394+0000 7f62f42cf640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f62ec0bea70 con 0x7f62ec0c9260 2024-04-25T00:01:02.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.394+0000 7f62f42cf640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f62ec0be8f0 con 0x7f62ec153130 2024-04-25T00:01:02.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.394+0000 7f62f42cf640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f62ec162990 con 0x7f62ec0ca1b0 2024-04-25T00:01:02.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.394+0000 7f62f206b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62ec0c9260 0x7f62ec0b08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:02.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.394+0000 7f62f206b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62ec0c9260 0x7f62ec0b08f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43182/0 (socket says 172.21.15.116:43182) 2024-04-25T00:01:02.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.394+0000 7f62f206b640 1 -- 172.21.15.116:0/838112184 learned_addr learned my addr 172.21.15.116:0/838112184 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:02.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.398+0000 7f62f286c640 1 --2- 172.21.15.116:0/838112184 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f62ec153130 0x7f62ec0c2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:02.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.398+0000 7f62f206b640 1 -- 172.21.15.116:0/838112184 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f62ec0ca1b0 msgr2=0x7f62ec159270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:02.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.398+0000 7f62f206b640 1 --2- 172.21.15.116:0/838112184 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f62ec0ca1b0 0x7f62ec159270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:02.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.398+0000 7f62f206b640 1 -- 172.21.15.116:0/838112184 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f62ec153130 msgr2=0x7f62ec0c2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:02.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.398+0000 7f62f206b640 1 --2- 172.21.15.116:0/838112184 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f62ec153130 0x7f62ec0c2540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:02.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.398+0000 7f62f206b640 1 -- 172.21.15.116:0/838112184 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62ec1f2690 con 0x7f62ec0c9260 2024-04-25T00:01:02.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.398+0000 7f62f286c640 1 --2- 172.21.15.116:0/838112184 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f62ec153130 0x7f62ec0c2540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:02.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.398+0000 7f62f206b640 1 --2- 172.21.15.116:0/838112184 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62ec0c9260 0x7f62ec0b08f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f62e800c480 tx=0x7f62e800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:02.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.398+0000 7f62e2ffd640 1 -- 172.21.15.116:0/838112184 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e8018070 con 0x7f62ec0c9260 2024-04-25T00:01:02.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.398+0000 7f62e2ffd640 1 -- 172.21.15.116:0/838112184 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62e8007500 con 0x7f62ec0c9260 2024-04-25T00:01:02.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.398+0000 7f62f42cf640 1 -- 172.21.15.116:0/838112184 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62ec0afe10 con 0x7f62ec0c9260 2024-04-25T00:01:02.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.398+0000 7f62e2ffd640 1 -- 172.21.15.116:0/838112184 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62e8004a30 con 0x7f62ec0c9260 2024-04-25T00:01:02.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.398+0000 7f62f42cf640 1 -- 172.21.15.116:0/838112184 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62ec0b03d0 con 0x7f62ec0c9260 2024-04-25T00:01:02.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.398+0000 7f62f42cf640 1 -- 172.21.15.116:0/838112184 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62ec04f510 con 0x7f62ec0c9260 2024-04-25T00:01:02.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.398+0000 7f62e2ffd640 1 -- 172.21.15.116:0/838112184 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f62e801b070 con 0x7f62ec0c9260 2024-04-25T00:01:02.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.398+0000 7f62e2ffd640 1 --2- 172.21.15.116:0/838112184 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f62d00678c0 0x7f62d0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:02.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.398+0000 7f62e2ffd640 1 -- 172.21.15.116:0/838112184 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 4857+0+0 (secure 0 0 0) 0x7f62e8084c70 con 0x7f62ec0c9260 2024-04-25T00:01:02.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.398+0000 7f62f186a640 1 --2- 172.21.15.116:0/838112184 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f62d00678c0 0x7f62d0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:02.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.398+0000 7f62f186a640 1 --2- 172.21.15.116:0/838112184 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f62d00678c0 0x7f62d0069d80 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f62dc0ad000 tx=0x7f62dc004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:02.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.402+0000 7f62e2ffd640 1 -- 172.21.15.116:0/838112184 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f62e8055b90 con 0x7f62ec0c9260 2024-04-25T00:01:02.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.562+0000 7f62f42cf640 1 -- 172.21.15.116:0/838112184 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f62ec0b0610 con 0x7f62ec0c9260 2024-04-25T00:01:02.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.562+0000 7f62e2ffd640 1 -- 172.21.15.116:0/838112184 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v19) ==== 93+0+10 (secure 0 0 0) 0x7f62e8055490 con 0x7f62ec0c9260 2024-04-25T00:01:02.565 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-04-25T00:01:02.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.566+0000 7f62f42cf640 1 -- 172.21.15.116:0/838112184 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f62d00678c0 msgr2=0x7f62d0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:02.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.566+0000 7f62f42cf640 1 --2- 172.21.15.116:0/838112184 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f62d00678c0 0x7f62d0069d80 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f62dc0ad000 tx=0x7f62dc004740 comp rx=0 tx=0).stop 2024-04-25T00:01:02.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.566+0000 7f62f42cf640 1 -- 172.21.15.116:0/838112184 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62ec0c9260 msgr2=0x7f62ec0b08f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:02.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.566+0000 7f62f42cf640 1 --2- 172.21.15.116:0/838112184 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62ec0c9260 0x7f62ec0b08f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f62e800c480 tx=0x7f62e800c940 comp rx=0 tx=0).stop 2024-04-25T00:01:02.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.566+0000 7f62f42cf640 1 -- 172.21.15.116:0/838112184 shutdown_connections 2024-04-25T00:01:02.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.566+0000 7f62f42cf640 1 --2- 172.21.15.116:0/838112184 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f62d00678c0 0x7f62d0069d80 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:02.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.566+0000 7f62f42cf640 1 --2- 172.21.15.116:0/838112184 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f62ec153130 0x7f62ec0c2540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:02.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.566+0000 7f62f42cf640 1 --2- 172.21.15.116:0/838112184 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f62ec0ca1b0 0x7f62ec159270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:02.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.566+0000 7f62f42cf640 1 --2- 172.21.15.116:0/838112184 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f62ec0c9260 0x7f62ec0b08f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:02.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.566+0000 7f62f42cf640 1 -- 172.21.15.116:0/838112184 >> 172.21.15.116:0/838112184 conn(0x7f62ec0cb310 msgr2=0x7f62ec0cb6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:02.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.566+0000 7f62f42cf640 1 -- 172.21.15.116:0/838112184 shutdown_connections 2024-04-25T00:01:02.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.566+0000 7f62f42cf640 1 -- 172.21.15.116:0/838112184 wait complete. 2024-04-25T00:01:02.583 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-25T00:01:02.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.694+0000 7f5534c8a640 1 Processor -- start 2024-04-25T00:01:02.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.694+0000 7f5534c8a640 1 -- start start 2024-04-25T00:01:02.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.698+0000 7f5534c8a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f553006be40 0x7f5530155f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:02.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.698+0000 7f5534c8a640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f553013e7a0 con 0x7f553006fb80 2024-04-25T00:01:02.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.698+0000 7f5534c8a640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f553013ea80 con 0x7f553013e210 2024-04-25T00:01:02.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.698+0000 7f5534c8a640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f553013e5f0 con 0x7f553006be40 2024-04-25T00:01:02.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.698+0000 7f552ed9d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f553006fb80 0x7f55301601c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:59506/0 (socket says 172.21.15.116:59506) 2024-04-25T00:01:02.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.698+0000 7f552ed9d640 1 -- 172.21.15.116:0/2405833452 learned_addr learned my addr 172.21.15.116:0/2405833452 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:02.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.698+0000 7f552dd9b640 1 --2- 172.21.15.116:0/2405833452 >> v2:172.21.15.116:3300/0 conn(0x7f553006be40 0x7f5530155f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:02.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.698+0000 7f552d59a640 1 -- 172.21.15.116:0/2405833452 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1717220153 0 0) 0x7f553013e7a0 con 0x7f553006fb80 2024-04-25T00:01:02.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.698+0000 7f552d59a640 1 -- 172.21.15.116:0/2405833452 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5518003530 con 0x7f553006fb80 2024-04-25T00:01:02.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.698+0000 7f552dd9b640 1 -- 172.21.15.116:0/2405833452 >> v1:172.21.15.174:6789/0 conn(0x7f553013e210 legacy=0x7f553006b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:02.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.698+0000 7f552dd9b640 1 -- 172.21.15.116:0/2405833452 >> v1:172.21.15.116:6789/0 conn(0x7f553006fb80 legacy=0x7f55301601c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:02.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.698+0000 7f552dd9b640 1 -- 172.21.15.116:0/2405833452 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f553013ed60 con 0x7f553006be40 2024-04-25T00:01:02.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.702+0000 7f552dd9b640 1 --2- 172.21.15.116:0/2405833452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f553006be40 0x7f5530155f20 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5520009d90 tx=0x7f552001c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d359ff2425b6e57 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:02.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.702+0000 7f552d59a640 1 -- 172.21.15.116:0/2405833452 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55200b2070 con 0x7f553006be40 2024-04-25T00:01:02.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.702+0000 7f552d59a640 1 -- 172.21.15.116:0/2405833452 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55200a58b0 con 0x7f553006be40 2024-04-25T00:01:02.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.702+0000 7f552d59a640 1 -- 172.21.15.116:0/2405833452 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55200a5c60 con 0x7f553006be40 2024-04-25T00:01:02.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.702+0000 7f5534c8a640 1 -- 172.21.15.116:0/2405833452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f553006be40 msgr2=0x7f5530155f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:02.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.702+0000 7f5534c8a640 1 --2- 172.21.15.116:0/2405833452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f553006be40 0x7f5530155f20 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5520009d90 tx=0x7f552001c8e0 comp rx=0 tx=0).stop 2024-04-25T00:01:02.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.702+0000 7f5534c8a640 1 -- 172.21.15.116:0/2405833452 shutdown_connections 2024-04-25T00:01:02.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.702+0000 7f5534c8a640 1 --2- 172.21.15.116:0/2405833452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f553006be40 0x7f5530155f20 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:02.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.702+0000 7f5534c8a640 1 -- 172.21.15.116:0/2405833452 >> 172.21.15.116:0/2405833452 conn(0x7f55300c4aa0 msgr2=0x7f55300c4eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:02.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.702+0000 7f5534c8a640 1 -- 172.21.15.116:0/2405833452 shutdown_connections 2024-04-25T00:01:02.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.702+0000 7f5534c8a640 1 -- 172.21.15.116:0/2405833452 wait complete. 2024-04-25T00:01:02.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.702+0000 7f5534c8a640 1 Processor -- start 2024-04-25T00:01:02.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.702+0000 7f5534c8a640 1 -- start start 2024-04-25T00:01:02.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.702+0000 7f5534c8a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f553006fb80 0x7f55301f1db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:02.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.702+0000 7f5534c8a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f553013e210 0x7f5530151310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:02.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.702+0000 7f5534c8a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f553013a9d0 0x7f5530151850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:02.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.702+0000 7f5534c8a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f553013f040 con 0x7f553013a9d0 2024-04-25T00:01:02.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.702+0000 7f5534c8a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5530164310 con 0x7f553006fb80 2024-04-25T00:01:02.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.702+0000 7f5534c8a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f5530151d90 con 0x7f553013e210 2024-04-25T00:01:02.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.702+0000 7f552ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f553013a9d0 0x7f5530151850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:02.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.702+0000 7f552ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f553013a9d0 0x7f5530151850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43206/0 (socket says 172.21.15.116:43206) 2024-04-25T00:01:02.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.702+0000 7f552ed9d640 1 -- 172.21.15.116:0/2913731054 learned_addr learned my addr 172.21.15.116:0/2913731054 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:02.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.702+0000 7f552e59c640 1 --2- 172.21.15.116:0/2913731054 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f553006fb80 0x7f55301f1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:02.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.702+0000 7f552dd9b640 1 --2- 172.21.15.116:0/2913731054 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f553013e210 0x7f5530151310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:02.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.702+0000 7f552ed9d640 1 -- 172.21.15.116:0/2913731054 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f553013e210 msgr2=0x7f5530151310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:02.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.702+0000 7f552ed9d640 1 --2- 172.21.15.116:0/2913731054 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f553013e210 0x7f5530151310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:02.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.702+0000 7f552ed9d640 1 -- 172.21.15.116:0/2913731054 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f553006fb80 msgr2=0x7f55301f1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:02.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.702+0000 7f552ed9d640 1 --2- 172.21.15.116:0/2913731054 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f553006fb80 0x7f55301f1db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:02.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.702+0000 7f552ed9d640 1 -- 172.21.15.116:0/2913731054 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5530159370 con 0x7f553013a9d0 2024-04-25T00:01:02.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.702+0000 7f552dd9b640 1 --2- 172.21.15.116:0/2913731054 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f553013e210 0x7f5530151310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:02.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.702+0000 7f552e59c640 1 --2- 172.21.15.116:0/2913731054 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f553006fb80 0x7f55301f1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:02.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.706+0000 7f552ed9d640 1 --2- 172.21.15.116:0/2913731054 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f553013a9d0 0x7f5530151850 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f552800c450 tx=0x7f552800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:02.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.706+0000 7f551effd640 1 -- 172.21.15.116:0/2913731054 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5528017070 con 0x7f553013a9d0 2024-04-25T00:01:02.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.706+0000 7f551effd640 1 -- 172.21.15.116:0/2913731054 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5528005cf0 con 0x7f553013a9d0 2024-04-25T00:01:02.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.706+0000 7f5534c8a640 1 -- 172.21.15.116:0/2913731054 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55301f2700 con 0x7f553013a9d0 2024-04-25T00:01:02.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.706+0000 7f551effd640 1 -- 172.21.15.116:0/2913731054 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5528007270 con 0x7f553013a9d0 2024-04-25T00:01:02.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.706+0000 7f5534c8a640 1 -- 172.21.15.116:0/2913731054 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55301f22f0 con 0x7f553013a9d0 2024-04-25T00:01:02.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.706+0000 7f5534c8a640 1 -- 172.21.15.116:0/2913731054 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5530068a60 con 0x7f553013a9d0 2024-04-25T00:01:02.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.706+0000 7f551effd640 1 -- 172.21.15.116:0/2913731054 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f552801a070 con 0x7f553013a9d0 2024-04-25T00:01:02.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.706+0000 7f551effd640 1 --2- 172.21.15.116:0/2913731054 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f55100678c0 0x7f5510069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:02.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.706+0000 7f551effd640 1 -- 172.21.15.116:0/2913731054 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 4857+0+0 (secure 0 0 0) 0x7f5528084bd0 con 0x7f553013a9d0 2024-04-25T00:01:02.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.706+0000 7f552e59c640 1 --2- 172.21.15.116:0/2913731054 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f55100678c0 0x7f5510069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:02.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.706+0000 7f552e59c640 1 --2- 172.21.15.116:0/2913731054 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f55100678c0 0x7f5510069d80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f55240016f0 tx=0x7f5524009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:02.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.710+0000 7f551effd640 1 -- 172.21.15.116:0/2913731054 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f5528055bb0 con 0x7f553013a9d0 2024-04-25T00:01:02.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.866+0000 7f5534c8a640 1 -- 172.21.15.116:0/2913731054 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f55300c2a40 con 0x7f553013a9d0 2024-04-25T00:01:02.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.866+0000 7f551effd640 1 -- 172.21.15.116:0/2913731054 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v19) ==== 84+0+65 (secure 0 0 0) 0x7f55280554b0 con 0x7f553013a9d0 2024-04-25T00:01:02.868 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:01:02.868 INFO:teuthology.orchestra.run.smithi116.stdout:[ 2024-04-25T00:01:02.868 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-25T00:01:02.868 INFO:teuthology.orchestra.run.smithi116.stdout: "poolnum": 1, 2024-04-25T00:01:02.868 INFO:teuthology.orchestra.run.smithi116.stdout: "poolname": ".mgr" 2024-04-25T00:01:02.868 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-25T00:01:02.868 INFO:teuthology.orchestra.run.smithi116.stdout:] 2024-04-25T00:01:02.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.866+0000 7f5534c8a640 1 -- 172.21.15.116:0/2913731054 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f55100678c0 msgr2=0x7f5510069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:02.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.866+0000 7f5534c8a640 1 --2- 172.21.15.116:0/2913731054 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f55100678c0 0x7f5510069d80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f55240016f0 tx=0x7f5524009040 comp rx=0 tx=0).stop 2024-04-25T00:01:02.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.866+0000 7f5534c8a640 1 -- 172.21.15.116:0/2913731054 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f553013a9d0 msgr2=0x7f5530151850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:02.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.866+0000 7f5534c8a640 1 --2- 172.21.15.116:0/2913731054 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f553013a9d0 0x7f5530151850 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f552800c450 tx=0x7f552800c910 comp rx=0 tx=0).stop 2024-04-25T00:01:02.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.866+0000 7f5534c8a640 1 -- 172.21.15.116:0/2913731054 shutdown_connections 2024-04-25T00:01:02.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.866+0000 7f5534c8a640 1 --2- 172.21.15.116:0/2913731054 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f55100678c0 0x7f5510069d80 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:02.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.866+0000 7f5534c8a640 1 --2- 172.21.15.116:0/2913731054 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f553013a9d0 0x7f5530151850 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:02.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.866+0000 7f5534c8a640 1 --2- 172.21.15.116:0/2913731054 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f553013e210 0x7f5530151310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:02.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.866+0000 7f5534c8a640 1 --2- 172.21.15.116:0/2913731054 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f553006fb80 0x7f55301f1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:02.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.866+0000 7f5534c8a640 1 -- 172.21.15.116:0/2913731054 >> 172.21.15.116:0/2913731054 conn(0x7f55300c4aa0 msgr2=0x7f553012f950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:02.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.866+0000 7f5534c8a640 1 -- 172.21.15.116:0/2913731054 shutdown_connections 2024-04-25T00:01:02.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.870+0000 7f5534c8a640 1 -- 172.21.15.116:0/2913731054 wait complete. 2024-04-25T00:01:02.886 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'a' 2024-04-25T00:01:02.886 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create a_metadata --pg_num_min 64 2024-04-25T00:01:03.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.998+0000 7f801d683640 1 Processor -- start 2024-04-25T00:01:03.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.998+0000 7f801d683640 1 -- start start 2024-04-25T00:01:03.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.998+0000 7f801d683640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f8018150c40 0x7f80180c4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:03.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.998+0000 7f801d683640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8018071090 0x7f80181573f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:03.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.998+0000 7f801d683640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80180c8810 con 0x7f80180c5510 2024-04-25T00:01:03.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.998+0000 7f801d683640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f8018157930 con 0x7f8018071090 2024-04-25T00:01:03.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:02.998+0000 7f801d683640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f80180c8530 con 0x7f8018150c40 2024-04-25T00:01:03.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.002+0000 7f80167fc640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f80180c5510 0x7f8018156ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:59516/0 (socket says 172.21.15.116:59516) 2024-04-25T00:01:03.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.002+0000 7f80167fc640 1 -- 172.21.15.116:0/477399200 learned_addr learned my addr 172.21.15.116:0/477399200 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:03.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.002+0000 7f80177fe640 1 --2- 172.21.15.116:0/477399200 >> v2:172.21.15.116:3300/0 conn(0x7f8018071090 0x7f80181573f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:03.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.002+0000 7f8016ffd640 1 --2- 172.21.15.116:0/477399200 >> v2:172.21.15.174:3300/0 conn(0x7f8018150c40 0x7f80180c4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:03.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.002+0000 7f8015ffb640 1 -- 172.21.15.116:0/477399200 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2437834622 0 0) 0x7f80180c8810 con 0x7f80180c5510 2024-04-25T00:01:03.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.002+0000 7f8015ffb640 1 -- 172.21.15.116:0/477399200 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8004003530 con 0x7f80180c5510 2024-04-25T00:01:03.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.002+0000 7f80177fe640 1 -- 172.21.15.116:0/477399200 >> v2:172.21.15.174:3300/0 conn(0x7f8018150c40 msgr2=0x7f80180c4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:03.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.002+0000 7f80177fe640 1 --2- 172.21.15.116:0/477399200 >> v2:172.21.15.174:3300/0 conn(0x7f8018150c40 0x7f80180c4fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:03.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.002+0000 7f80177fe640 1 -- 172.21.15.116:0/477399200 >> v1:172.21.15.116:6789/0 conn(0x7f80180c5510 legacy=0x7f8018156ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:03.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.002+0000 7f80177fe640 1 -- 172.21.15.116:0/477399200 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80180ad830 con 0x7f8018071090 2024-04-25T00:01:03.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.002+0000 7f80177fe640 1 --2- 172.21.15.116:0/477399200 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8018071090 0x7f80181573f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8018072b50 tx=0x7f801001c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d35d76d7ecfbaf4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:03.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.002+0000 7f8015ffb640 1 -- 172.21.15.116:0/477399200 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80100b2070 con 0x7f8018071090 2024-04-25T00:01:03.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.002+0000 7f8015ffb640 1 -- 172.21.15.116:0/477399200 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80100a5850 con 0x7f8018071090 2024-04-25T00:01:03.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.002+0000 7f8015ffb640 1 -- 172.21.15.116:0/477399200 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80100a5c20 con 0x7f8018071090 2024-04-25T00:01:03.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.002+0000 7f801d683640 1 -- 172.21.15.116:0/477399200 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8018071090 msgr2=0x7f80181573f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:03.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.002+0000 7f801d683640 1 --2- 172.21.15.116:0/477399200 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8018071090 0x7f80181573f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8018072b50 tx=0x7f801001c8a0 comp rx=0 tx=0).stop 2024-04-25T00:01:03.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.002+0000 7f801d683640 1 -- 172.21.15.116:0/477399200 shutdown_connections 2024-04-25T00:01:03.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.002+0000 7f801d683640 1 --2- 172.21.15.116:0/477399200 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8018071090 0x7f80181573f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:03.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.002+0000 7f801d683640 1 --2- 172.21.15.116:0/477399200 >> v2:172.21.15.174:3300/0 conn(0x7f8018150c40 0x7f80180c4fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:03.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.002+0000 7f801d683640 1 -- 172.21.15.116:0/477399200 >> 172.21.15.116:0/477399200 conn(0x7f80180aacb0 msgr2=0x7f80180ab0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:03.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.002+0000 7f801d683640 1 -- 172.21.15.116:0/477399200 shutdown_connections 2024-04-25T00:01:03.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.002+0000 7f801d683640 1 -- 172.21.15.116:0/477399200 wait complete. 2024-04-25T00:01:03.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.006+0000 7f801d683640 1 Processor -- start 2024-04-25T00:01:03.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.006+0000 7f801d683640 1 -- start start 2024-04-25T00:01:03.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.006+0000 7f801d683640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8018071090 0x7f8018158140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:03.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.006+0000 7f801d683640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f80180c5510 0x7f80181ee400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:03.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.006+0000 7f801d683640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8018150c40 0x7f801814a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:03.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.006+0000 7f801d683640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8018151020 con 0x7f8018071090 2024-04-25T00:01:03.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.006+0000 7f801d683640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8018071890 con 0x7f80180c5510 2024-04-25T00:01:03.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.006+0000 7f801d683640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f8018158680 con 0x7f8018150c40 2024-04-25T00:01:03.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.006+0000 7f8016ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8018071090 0x7f8018158140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:03.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.006+0000 7f8016ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8018071090 0x7f8018158140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43218/0 (socket says 172.21.15.116:43218) 2024-04-25T00:01:03.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.006+0000 7f8016ffd640 1 -- 172.21.15.116:0/1140750211 learned_addr learned my addr 172.21.15.116:0/1140750211 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:03.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.006+0000 7f80177fe640 1 --2- 172.21.15.116:0/1140750211 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8018150c40 0x7f801814a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:03.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.006+0000 7f80167fc640 1 --2- 172.21.15.116:0/1140750211 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f80180c5510 0x7f80181ee400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:03.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.006+0000 7f8016ffd640 1 -- 172.21.15.116:0/1140750211 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8018150c40 msgr2=0x7f801814a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:03.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.006+0000 7f8016ffd640 1 --2- 172.21.15.116:0/1140750211 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8018150c40 0x7f801814a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:03.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.006+0000 7f8016ffd640 1 -- 172.21.15.116:0/1140750211 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f80180c5510 msgr2=0x7f80181ee400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:03.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.006+0000 7f8016ffd640 1 --2- 172.21.15.116:0/1140750211 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f80180c5510 0x7f80181ee400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:03.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.006+0000 7f8016ffd640 1 -- 172.21.15.116:0/1140750211 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f801815d1f0 con 0x7f8018071090 2024-04-25T00:01:03.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.006+0000 7f80177fe640 1 --2- 172.21.15.116:0/1140750211 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8018150c40 0x7f801814a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:03.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.006+0000 7f8016ffd640 1 --2- 172.21.15.116:0/1140750211 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8018071090 0x7f8018158140 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f800000d5f0 tx=0x7f800000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:03.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.006+0000 7f800f7fe640 1 -- 172.21.15.116:0/1140750211 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80000125f0 con 0x7f8018071090 2024-04-25T00:01:03.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.006+0000 7f800f7fe640 1 -- 172.21.15.116:0/1140750211 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8000019070 con 0x7f8018071090 2024-04-25T00:01:03.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.006+0000 7f801d683640 1 -- 172.21.15.116:0/1140750211 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80181eeee0 con 0x7f8018071090 2024-04-25T00:01:03.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.006+0000 7f800f7fe640 1 -- 172.21.15.116:0/1140750211 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80000129b0 con 0x7f8018071090 2024-04-25T00:01:03.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.006+0000 7f801d683640 1 -- 172.21.15.116:0/1140750211 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f801815cf10 con 0x7f8018071090 2024-04-25T00:01:03.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.006+0000 7f801d683640 1 -- 172.21.15.116:0/1140750211 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8018068a60 con 0x7f8018071090 2024-04-25T00:01:03.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.006+0000 7f800f7fe640 1 -- 172.21.15.116:0/1140750211 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8000012b50 con 0x7f8018071090 2024-04-25T00:01:03.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.006+0000 7f800f7fe640 1 --2- 172.21.15.116:0/1140750211 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7ff80678c0 0x7f7ff8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:03.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.006+0000 7f800f7fe640 1 -- 172.21.15.116:0/1140750211 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 4857+0+0 (secure 0 0 0) 0x7f8000085d10 con 0x7f8018071090 2024-04-25T00:01:03.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.006+0000 7f80167fc640 1 --2- 172.21.15.116:0/1140750211 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7ff80678c0 0x7f7ff8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:03.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.010+0000 7f80167fc640 1 --2- 172.21.15.116:0/1140750211 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7ff80678c0 0x7f7ff8069d80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8008007fd0 tx=0x7f8008009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:03.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.010+0000 7f800f7fe640 1 -- 172.21.15.116:0/1140750211 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f8000056ba0 con 0x7f8018071090 2024-04-25T00:01:03.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.158+0000 7f801d683640 1 -- 172.21.15.116:0/1140750211 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "a_metadata", "pg_num_min": 64} v 0) -- 0x7f801814ab10 con 0x7f8018071090 2024-04-25T00:01:03.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.310+0000 7f800f7fe640 1 -- 172.21.15.116:0/1140750211 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "a_metadata", "pg_num_min": 64}]=0 pool 'a_metadata' created v20) ==== 128+0+0 (secure 0 0 0) 0x7f80000564a0 con 0x7f8018071090 2024-04-25T00:01:03.380 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'a_metadata' created 2024-04-25T00:01:03.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.314+0000 7f801d683640 1 -- 172.21.15.116:0/1140750211 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7ff80678c0 msgr2=0x7f7ff8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:03.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.314+0000 7f801d683640 1 --2- 172.21.15.116:0/1140750211 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7ff80678c0 0x7f7ff8069d80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8008007fd0 tx=0x7f8008009040 comp rx=0 tx=0).stop 2024-04-25T00:01:03.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.314+0000 7f801d683640 1 -- 172.21.15.116:0/1140750211 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8018071090 msgr2=0x7f8018158140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:03.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.314+0000 7f801d683640 1 --2- 172.21.15.116:0/1140750211 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8018071090 0x7f8018158140 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f800000d5f0 tx=0x7f800000dab0 comp rx=0 tx=0).stop 2024-04-25T00:01:03.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.314+0000 7f801d683640 1 -- 172.21.15.116:0/1140750211 shutdown_connections 2024-04-25T00:01:03.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.314+0000 7f801d683640 1 --2- 172.21.15.116:0/1140750211 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7ff80678c0 0x7f7ff8069d80 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:03.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.314+0000 7f801d683640 1 --2- 172.21.15.116:0/1140750211 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8018150c40 0x7f801814a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:03.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.314+0000 7f801d683640 1 --2- 172.21.15.116:0/1140750211 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f80180c5510 0x7f80181ee400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:03.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.314+0000 7f801d683640 1 --2- 172.21.15.116:0/1140750211 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8018071090 0x7f8018158140 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:03.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.314+0000 7f801d683640 1 -- 172.21.15.116:0/1140750211 >> 172.21.15.116:0/1140750211 conn(0x7f80180aacb0 msgr2=0x7f80180c7710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:03.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.314+0000 7f801d683640 1 -- 172.21.15.116:0/1140750211 shutdown_connections 2024-04-25T00:01:03.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.314+0000 7f801d683640 1 -- 172.21.15.116:0/1140750211 wait complete. 2024-04-25T00:01:03.383 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create a_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-25T00:01:03.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.514+0000 7ff673cda640 1 Processor -- start 2024-04-25T00:01:03.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.514+0000 7ff673cda640 1 -- start start 2024-04-25T00:01:03.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.514+0000 7ff673cda640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff66c153130 0x7ff66c1562e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:03.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.514+0000 7ff673cda640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff66c0ca1b0 0x7ff66c06bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:03.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.514+0000 7ff673cda640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff66c156820 con 0x7ff66c0c9260 2024-04-25T00:01:03.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.514+0000 7ff673cda640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7ff66c0ca820 con 0x7ff66c153130 2024-04-25T00:01:03.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.514+0000 7ff673cda640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff66c155e50 con 0x7ff66c0ca1b0 2024-04-25T00:01:03.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.514+0000 7ff671a76640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff66c153130 0x7ff66c1562e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:03.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.514+0000 7ff671a76640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff66c153130 0x7ff66c1562e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43228/0 (socket says 172.21.15.116:43228) 2024-04-25T00:01:03.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.514+0000 7ff671a76640 1 -- 172.21.15.116:0/2963458587 learned_addr learned my addr 172.21.15.116:0/2963458587 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:03.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.514+0000 7ff671275640 1 --2- 172.21.15.116:0/2963458587 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff66c0ca1b0 0x7ff66c06bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:03.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.514+0000 7ff671a76640 1 -- 172.21.15.116:0/2963458587 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff66c0ca1b0 msgr2=0x7ff66c06bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:03.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.514+0000 7ff671a76640 1 --2- 172.21.15.116:0/2963458587 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff66c0ca1b0 0x7ff66c06bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:03.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.514+0000 7ff671a76640 1 -- 172.21.15.116:0/2963458587 >> v1:172.21.15.174:6789/0 conn(0x7ff66c0c9260 legacy=0x7ff66c0c96e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:03.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.514+0000 7ff671a76640 1 -- 172.21.15.116:0/2963458587 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff66c06c300 con 0x7ff66c153130 2024-04-25T00:01:03.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.514+0000 7ff671275640 1 --2- 172.21.15.116:0/2963458587 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff66c0ca1b0 0x7ff66c06bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:03.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.518+0000 7ff671a76640 1 --2- 172.21.15.116:0/2963458587 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff66c153130 0x7ff66c1562e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff668009870 tx=0x7ff66801c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5263156184966dca server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:03.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.518+0000 7ff670a74640 1 -- 172.21.15.116:0/2963458587 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6680b0070 con 0x7ff66c153130 2024-04-25T00:01:03.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.518+0000 7ff670a74640 1 -- 172.21.15.116:0/2963458587 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6680a4e50 con 0x7ff66c153130 2024-04-25T00:01:03.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.518+0000 7ff670a74640 1 -- 172.21.15.116:0/2963458587 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6680ab820 con 0x7ff66c153130 2024-04-25T00:01:03.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.518+0000 7ff673cda640 1 -- 172.21.15.116:0/2963458587 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff66c153130 msgr2=0x7ff66c1562e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:03.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.518+0000 7ff673cda640 1 --2- 172.21.15.116:0/2963458587 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff66c153130 0x7ff66c1562e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff668009870 tx=0x7ff66801c400 comp rx=0 tx=0).stop 2024-04-25T00:01:03.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.518+0000 7ff673cda640 1 -- 172.21.15.116:0/2963458587 shutdown_connections 2024-04-25T00:01:03.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.518+0000 7ff673cda640 1 --2- 172.21.15.116:0/2963458587 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff66c0ca1b0 0x7ff66c06bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:03.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.518+0000 7ff673cda640 1 --2- 172.21.15.116:0/2963458587 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff66c153130 0x7ff66c1562e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:03.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.518+0000 7ff673cda640 1 -- 172.21.15.116:0/2963458587 >> 172.21.15.116:0/2963458587 conn(0x7ff66c0cb310 msgr2=0x7ff66c0cb720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:03.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.518+0000 7ff673cda640 1 -- 172.21.15.116:0/2963458587 shutdown_connections 2024-04-25T00:01:03.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.518+0000 7ff673cda640 1 -- 172.21.15.116:0/2963458587 wait complete. 2024-04-25T00:01:03.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.518+0000 7ff673cda640 1 Processor -- start 2024-04-25T00:01:03.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.518+0000 7ff673cda640 1 -- start start 2024-04-25T00:01:03.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.518+0000 7ff673cda640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff66c0c9260 0x7ff66c148ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:03.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.518+0000 7ff673cda640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff66c0ca1b0 0x7ff66c1f07a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:03.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.518+0000 7ff673cda640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff66c153130 0x7ff66c1f0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:03.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.518+0000 7ff673cda640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff66c0bea70 con 0x7ff66c0ca1b0 2024-04-25T00:01:03.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.518+0000 7ff673cda640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff66c0be8f0 con 0x7ff66c0c9260 2024-04-25T00:01:03.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.518+0000 7ff673cda640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff66c162990 con 0x7ff66c153130 2024-04-25T00:01:03.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.518+0000 7ff671275640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff66c0ca1b0 0x7ff66c1f07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:03.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.518+0000 7ff671275640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff66c0ca1b0 0x7ff66c1f07a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43230/0 (socket says 172.21.15.116:43230) 2024-04-25T00:01:03.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.518+0000 7ff671275640 1 -- 172.21.15.116:0/2523200820 learned_addr learned my addr 172.21.15.116:0/2523200820 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:03.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.518+0000 7ff671a76640 1 --2- 172.21.15.116:0/2523200820 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff66c0c9260 0x7ff66c148ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:03.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.518+0000 7ff672277640 1 --2- 172.21.15.116:0/2523200820 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff66c153130 0x7ff66c1f0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:03.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.522+0000 7ff671275640 1 -- 172.21.15.116:0/2523200820 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff66c153130 msgr2=0x7ff66c1f0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:03.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.522+0000 7ff671275640 1 --2- 172.21.15.116:0/2523200820 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff66c153130 0x7ff66c1f0ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:03.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.522+0000 7ff671275640 1 -- 172.21.15.116:0/2523200820 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff66c0c9260 msgr2=0x7ff66c148ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:03.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.522+0000 7ff671275640 1 --2- 172.21.15.116:0/2523200820 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff66c0c9260 0x7ff66c148ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:03.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.522+0000 7ff671275640 1 -- 172.21.15.116:0/2523200820 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff66c0aae60 con 0x7ff66c0ca1b0 2024-04-25T00:01:03.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.522+0000 7ff671a76640 1 --2- 172.21.15.116:0/2523200820 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff66c0c9260 0x7ff66c148ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:03.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.522+0000 7ff671275640 1 --2- 172.21.15.116:0/2523200820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff66c0ca1b0 0x7ff66c1f07a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff65c00c820 tx=0x7ff65c00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:03.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.522+0000 7ff6627fc640 1 -- 172.21.15.116:0/2523200820 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff65c015070 con 0x7ff66c0ca1b0 2024-04-25T00:01:03.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.522+0000 7ff6627fc640 1 -- 172.21.15.116:0/2523200820 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff65c007a30 con 0x7ff66c0ca1b0 2024-04-25T00:01:03.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.522+0000 7ff673cda640 1 -- 172.21.15.116:0/2523200820 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff66c0aabc0 con 0x7ff66c0ca1b0 2024-04-25T00:01:03.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.522+0000 7ff6627fc640 1 -- 172.21.15.116:0/2523200820 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff65c007e20 con 0x7ff66c0ca1b0 2024-04-25T00:01:03.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.522+0000 7ff673cda640 1 -- 172.21.15.116:0/2523200820 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff66c1effa0 con 0x7ff66c0ca1b0 2024-04-25T00:01:03.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.522+0000 7ff673cda640 1 -- 172.21.15.116:0/2523200820 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff66c04f510 con 0x7ff66c0ca1b0 2024-04-25T00:01:03.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.522+0000 7ff6627fc640 1 -- 172.21.15.116:0/2523200820 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff65c018070 con 0x7ff66c0ca1b0 2024-04-25T00:01:03.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.522+0000 7ff6627fc640 1 --2- 172.21.15.116:0/2523200820 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff6500678f0 0x7ff650069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:03.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.522+0000 7ff6627fc640 1 -- 172.21.15.116:0/2523200820 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5222+0+0 (secure 0 0 0) 0x7ff65c0856d0 con 0x7ff66c0ca1b0 2024-04-25T00:01:03.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.522+0000 7ff671a76640 1 --2- 172.21.15.116:0/2523200820 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff6500678f0 0x7ff650069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:03.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.522+0000 7ff671a76640 1 --2- 172.21.15.116:0/2523200820 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff6500678f0 0x7ff650069db0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff6680ad000 tx=0x7ff668008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:03.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.526+0000 7ff6627fc640 1 -- 172.21.15.116:0/2523200820 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ff65c0563f0 con 0x7ff66c0ca1b0 2024-04-25T00:01:03.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:03.686+0000 7ff673cda640 1 -- 172.21.15.116:0/2523200820 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "a_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7ff66c06da40 con 0x7ff66c0ca1b0 2024-04-25T00:01:04.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.322+0000 7ff6627fc640 1 -- 172.21.15.116:0/2523200820 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "a_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'a_data' created v21) ==== 160+0+0 (secure 0 0 0) 0x7ff65c055cf0 con 0x7ff66c0ca1b0 2024-04-25T00:01:04.326 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'a_data' created 2024-04-25T00:01:04.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.326+0000 7ff673cda640 1 -- 172.21.15.116:0/2523200820 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff6500678f0 msgr2=0x7ff650069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:04.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.326+0000 7ff673cda640 1 --2- 172.21.15.116:0/2523200820 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff6500678f0 0x7ff650069db0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ff6680ad000 tx=0x7ff668008040 comp rx=0 tx=0).stop 2024-04-25T00:01:04.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.326+0000 7ff673cda640 1 -- 172.21.15.116:0/2523200820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff66c0ca1b0 msgr2=0x7ff66c1f07a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:04.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.326+0000 7ff673cda640 1 --2- 172.21.15.116:0/2523200820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff66c0ca1b0 0x7ff66c1f07a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff65c00c820 tx=0x7ff65c00cce0 comp rx=0 tx=0).stop 2024-04-25T00:01:04.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.326+0000 7ff673cda640 1 -- 172.21.15.116:0/2523200820 shutdown_connections 2024-04-25T00:01:04.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.326+0000 7ff673cda640 1 --2- 172.21.15.116:0/2523200820 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff6500678f0 0x7ff650069db0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:04.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.326+0000 7ff673cda640 1 --2- 172.21.15.116:0/2523200820 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff66c153130 0x7ff66c1f0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:04.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.326+0000 7ff673cda640 1 --2- 172.21.15.116:0/2523200820 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff66c0ca1b0 0x7ff66c1f07a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:04.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.326+0000 7ff673cda640 1 --2- 172.21.15.116:0/2523200820 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff66c0c9260 0x7ff66c148ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:04.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.326+0000 7ff673cda640 1 -- 172.21.15.116:0/2523200820 >> 172.21.15.116:0/2523200820 conn(0x7ff66c0cb310 msgr2=0x7ff66c154150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:04.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.326+0000 7ff673cda640 1 -- 172.21.15.116:0/2523200820 shutdown_connections 2024-04-25T00:01:04.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.326+0000 7ff673cda640 1 -- 172.21.15.116:0/2523200820 wait complete. 2024-04-25T00:01:04.361 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new a a_metadata a_data 2024-04-25T00:01:04.408 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:01:04.410 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:01:04.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.478+0000 7f4bd6f42640 1 Processor -- start 2024-04-25T00:01:04.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.478+0000 7f4bd6f42640 1 -- start start 2024-04-25T00:01:04.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.478+0000 7f4bd6f42640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4bd0153130 0x7f4bd01562e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:04.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.478+0000 7f4bd6f42640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4bd00ca1b0 0x7f4bd006bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:04.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.478+0000 7f4bd6f42640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bd0156820 con 0x7f4bd00c9260 2024-04-25T00:01:04.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.478+0000 7f4bd6f42640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f4bd00ca820 con 0x7f4bd0153130 2024-04-25T00:01:04.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.478+0000 7f4bd6f42640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f4bd0155e50 con 0x7f4bd00ca1b0 2024-04-25T00:01:04.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.478+0000 7f4bd4cde640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4bd0153130 0x7f4bd01562e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:04.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.478+0000 7f4bd54df640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f4bd00c9260 0x7f4bd00c96e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:59532/0 (socket says 172.21.15.116:59532) 2024-04-25T00:01:04.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.478+0000 7f4bd54df640 1 -- 172.21.15.116:0/4132489354 learned_addr learned my addr 172.21.15.116:0/4132489354 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:04.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.478+0000 7f4bcf7fe640 1 -- 172.21.15.116:0/4132489354 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 292161203 0 0) 0x7f4bd0156820 con 0x7f4bd00c9260 2024-04-25T00:01:04.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.478+0000 7f4bcf7fe640 1 -- 172.21.15.116:0/4132489354 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4bb8003530 con 0x7f4bd00c9260 2024-04-25T00:01:04.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.478+0000 7f4bcffff640 1 --2- 172.21.15.116:0/4132489354 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4bd00ca1b0 0x7f4bd006bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:04.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.478+0000 7f4bd4cde640 1 -- 172.21.15.116:0/4132489354 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4bd00ca1b0 msgr2=0x7f4bd006bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:04.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.478+0000 7f4bd4cde640 1 --2- 172.21.15.116:0/4132489354 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4bd00ca1b0 0x7f4bd006bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:04.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.478+0000 7f4bd4cde640 1 -- 172.21.15.116:0/4132489354 >> v1:172.21.15.116:6789/0 conn(0x7f4bd00c9260 legacy=0x7f4bd00c96e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:04.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.478+0000 7f4bd4cde640 1 -- 172.21.15.116:0/4132489354 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bd006c300 con 0x7f4bd0153130 2024-04-25T00:01:04.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.478+0000 7f4bcffff640 1 --2- 172.21.15.116:0/4132489354 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4bd00ca1b0 0x7f4bd006bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:04.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.482+0000 7f4bd4cde640 1 --2- 172.21.15.116:0/4132489354 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4bd0153130 0x7f4bd01562e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4bc00097c0 tx=0x7f4bc001c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d26f830d644f32bf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:04.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.482+0000 7f4bcf7fe640 1 -- 172.21.15.116:0/4132489354 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc00b0070 con 0x7f4bd0153130 2024-04-25T00:01:04.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.482+0000 7f4bcf7fe640 1 -- 172.21.15.116:0/4132489354 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bc00ab600 con 0x7f4bd0153130 2024-04-25T00:01:04.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.482+0000 7f4bcf7fe640 1 -- 172.21.15.116:0/4132489354 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc00ab9d0 con 0x7f4bd0153130 2024-04-25T00:01:04.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.482+0000 7f4bd6f42640 1 -- 172.21.15.116:0/4132489354 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4bd0153130 msgr2=0x7f4bd01562e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:04.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.482+0000 7f4bd6f42640 1 --2- 172.21.15.116:0/4132489354 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4bd0153130 0x7f4bd01562e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4bc00097c0 tx=0x7f4bc001c380 comp rx=0 tx=0).stop 2024-04-25T00:01:04.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.482+0000 7f4bd6f42640 1 -- 172.21.15.116:0/4132489354 shutdown_connections 2024-04-25T00:01:04.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.482+0000 7f4bd6f42640 1 --2- 172.21.15.116:0/4132489354 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4bd00ca1b0 0x7f4bd006bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:04.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.482+0000 7f4bd6f42640 1 --2- 172.21.15.116:0/4132489354 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4bd0153130 0x7f4bd01562e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:04.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.482+0000 7f4bd6f42640 1 -- 172.21.15.116:0/4132489354 >> 172.21.15.116:0/4132489354 conn(0x7f4bd00cb310 msgr2=0x7f4bd00cb720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:04.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.482+0000 7f4bd6f42640 1 -- 172.21.15.116:0/4132489354 shutdown_connections 2024-04-25T00:01:04.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.482+0000 7f4bd6f42640 1 -- 172.21.15.116:0/4132489354 wait complete. 2024-04-25T00:01:04.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.482+0000 7f4bd6f42640 1 Processor -- start 2024-04-25T00:01:04.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.482+0000 7f4bd6f42640 1 -- start start 2024-04-25T00:01:04.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.482+0000 7f4bd6f42640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4bd00c9260 0x7f4bd01eeb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:04.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.482+0000 7f4bd6f42640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4bd00ca1b0 0x7f4bd014e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:04.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.482+0000 7f4bd6f42640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4bd0153130 0x7f4bd0156c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:04.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.482+0000 7f4bd6f42640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd00bea70 con 0x7f4bd0153130 2024-04-25T00:01:04.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.482+0000 7f4bd6f42640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd00be8f0 con 0x7f4bd00c9260 2024-04-25T00:01:04.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.482+0000 7f4bd6f42640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f4bd0162990 con 0x7f4bd00ca1b0 2024-04-25T00:01:04.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.482+0000 7f4bd54df640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4bd0153130 0x7f4bd0156c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:04.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.482+0000 7f4bd54df640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4bd0153130 0x7f4bd0156c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43248/0 (socket says 172.21.15.116:43248) 2024-04-25T00:01:04.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.482+0000 7f4bd54df640 1 -- 172.21.15.116:0/3237283226 learned_addr learned my addr 172.21.15.116:0/3237283226 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:04.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.482+0000 7f4bd4cde640 1 --2- 172.21.15.116:0/3237283226 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4bd00c9260 0x7f4bd01eeb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:04.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.482+0000 7f4bcffff640 1 --2- 172.21.15.116:0/3237283226 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4bd00ca1b0 0x7f4bd014e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:04.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.486+0000 7f4bd54df640 1 -- 172.21.15.116:0/3237283226 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4bd00ca1b0 msgr2=0x7f4bd014e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:04.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.486+0000 7f4bd54df640 1 --2- 172.21.15.116:0/3237283226 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4bd00ca1b0 0x7f4bd014e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:04.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.486+0000 7f4bd54df640 1 -- 172.21.15.116:0/3237283226 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4bd00c9260 msgr2=0x7f4bd01eeb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:04.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.486+0000 7f4bd54df640 1 --2- 172.21.15.116:0/3237283226 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4bd00c9260 0x7f4bd01eeb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:04.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.486+0000 7f4bd54df640 1 -- 172.21.15.116:0/3237283226 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bd00c5290 con 0x7f4bd0153130 2024-04-25T00:01:04.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.486+0000 7f4bd54df640 1 --2- 172.21.15.116:0/3237283226 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4bd0153130 0x7f4bd0156c40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4bc800caa0 tx=0x7f4bc800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:04.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.486+0000 7f4bcd7fa640 1 -- 172.21.15.116:0/3237283226 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc8017070 con 0x7f4bd0153130 2024-04-25T00:01:04.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.486+0000 7f4bcd7fa640 1 -- 172.21.15.116:0/3237283226 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bc800ac00 con 0x7f4bd0153130 2024-04-25T00:01:04.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.486+0000 7f4bd6f42640 1 -- 172.21.15.116:0/3237283226 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bd00c4fb0 con 0x7f4bd0153130 2024-04-25T00:01:04.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.486+0000 7f4bd6f42640 1 -- 172.21.15.116:0/3237283226 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bd01ee350 con 0x7f4bd0153130 2024-04-25T00:01:04.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.486+0000 7f4bcd7fa640 1 -- 172.21.15.116:0/3237283226 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bc8012400 con 0x7f4bd0153130 2024-04-25T00:01:04.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.486+0000 7f4bd6f42640 1 -- 172.21.15.116:0/3237283226 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bd004f510 con 0x7f4bd0153130 2024-04-25T00:01:04.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.486+0000 7f4bcd7fa640 1 -- 172.21.15.116:0/3237283226 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4bc801a070 con 0x7f4bd0153130 2024-04-25T00:01:04.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.486+0000 7f4bcd7fa640 1 --2- 172.21.15.116:0/3237283226 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4bbc0678c0 0x7f4bbc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:04.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.486+0000 7f4bcd7fa640 1 -- 172.21.15.116:0/3237283226 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5599+0+0 (secure 0 0 0) 0x7f4bc8085dc0 con 0x7f4bd0153130 2024-04-25T00:01:04.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.486+0000 7f4bd4cde640 1 --2- 172.21.15.116:0/3237283226 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4bbc0678c0 0x7f4bbc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:04.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.490+0000 7f4bd4cde640 1 --2- 172.21.15.116:0/3237283226 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4bbc0678c0 0x7f4bbc069d80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4bc00ad000 tx=0x7f4bc0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:04.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.490+0000 7f4bcd7fa640 1 -- 172.21.15.116:0/3237283226 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f4bc8021020 con 0x7f4bd0153130 2024-04-25T00:01:04.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:04.674+0000 7f4bd6f42640 1 -- 172.21.15.116:0/3237283226 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "a", "metadata": "a_metadata", "data": "a_data"} v 0) -- 0x7f4bd006da40 con 0x7f4bd0153130 2024-04-25T00:01:05.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.382+0000 7f4bcd7fa640 1 -- 172.21.15.116:0/3237283226 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "a", "metadata": "a_metadata", "data": "a_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 157+0+0 (secure 0 0 0) 0x7f4bc8056a50 con 0x7f4bd0153130 2024-04-25T00:01:05.384 INFO:teuthology.orchestra.run.smithi116.stderr:new fs with metadata pool 2 and data pool 3 2024-04-25T00:01:05.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.386+0000 7f4bd6f42640 1 -- 172.21.15.116:0/3237283226 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4bbc0678c0 msgr2=0x7f4bbc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:05.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.386+0000 7f4bd6f42640 1 --2- 172.21.15.116:0/3237283226 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4bbc0678c0 0x7f4bbc069d80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4bc00ad000 tx=0x7f4bc0008040 comp rx=0 tx=0).stop 2024-04-25T00:01:05.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.386+0000 7f4bd6f42640 1 -- 172.21.15.116:0/3237283226 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4bd0153130 msgr2=0x7f4bd0156c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:05.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.386+0000 7f4bd6f42640 1 --2- 172.21.15.116:0/3237283226 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4bd0153130 0x7f4bd0156c40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4bc800caa0 tx=0x7f4bc800cf60 comp rx=0 tx=0).stop 2024-04-25T00:01:05.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.386+0000 7f4bd6f42640 1 -- 172.21.15.116:0/3237283226 shutdown_connections 2024-04-25T00:01:05.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.386+0000 7f4bd6f42640 1 --2- 172.21.15.116:0/3237283226 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4bbc0678c0 0x7f4bbc069d80 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:05.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.386+0000 7f4bd6f42640 1 --2- 172.21.15.116:0/3237283226 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4bd0153130 0x7f4bd0156c40 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:05.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.386+0000 7f4bd6f42640 1 --2- 172.21.15.116:0/3237283226 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4bd00ca1b0 0x7f4bd014e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:05.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.386+0000 7f4bd6f42640 1 --2- 172.21.15.116:0/3237283226 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4bd00c9260 0x7f4bd01eeb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:05.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.386+0000 7f4bd6f42640 1 -- 172.21.15.116:0/3237283226 >> 172.21.15.116:0/3237283226 conn(0x7f4bd00cb310 msgr2=0x7f4bd00cb6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:05.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.386+0000 7f4bd6f42640 1 -- 172.21.15.116:0/3237283226 shutdown_connections 2024-04-25T00:01:05.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.386+0000 7f4bd6f42640 1 -- 172.21.15.116:0/3237283226 wait complete. 2024-04-25T00:01:05.414 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T00:01:05.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.578+0000 7f6421cb6640 1 Processor -- start 2024-04-25T00:01:05.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.578+0000 7f6421cb6640 1 -- start start 2024-04-25T00:01:05.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.578+0000 7f6421cb6640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f641c071d90 0x7f641c1563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:05.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.578+0000 7f6421cb6640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f641c071220 con 0x7f641c14c720 2024-04-25T00:01:05.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.578+0000 7f6421cb6640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f641c152420 con 0x7f641c0a94a0 2024-04-25T00:01:05.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.578+0000 7f6421cb6640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f641c153460 con 0x7f641c071d90 2024-04-25T00:01:05.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.578+0000 7f641b7fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f641c14c720 0x7f641c0c2a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:47842/0 (socket says 172.21.15.116:47842) 2024-04-25T00:01:05.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.578+0000 7f641b7fe640 1 -- 172.21.15.116:0/1970468258 learned_addr learned my addr 172.21.15.116:0/1970468258 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:05.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.578+0000 7f641bfff640 1 --2- 172.21.15.116:0/1970468258 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f641c071d90 0x7f641c1563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:05.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.578+0000 7f641a7fc640 1 -- 172.21.15.116:0/1970468258 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2288472729 0 0) 0x7f641c071220 con 0x7f641c14c720 2024-04-25T00:01:05.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.578+0000 7f641a7fc640 1 -- 172.21.15.116:0/1970468258 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6404003530 con 0x7f641c14c720 2024-04-25T00:01:05.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.578+0000 7f641a7fc640 1 -- 172.21.15.116:0/1970468258 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2207823839 0 0) 0x7f6404003530 con 0x7f641c14c720 2024-04-25T00:01:05.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.578+0000 7f641a7fc640 1 -- 172.21.15.116:0/1970468258 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f641c071d90 msgr2=0x7f641c1563d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:05.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.578+0000 7f641a7fc640 1 --2- 172.21.15.116:0/1970468258 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f641c071d90 0x7f641c1563d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:05.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.578+0000 7f641a7fc640 1 -- 172.21.15.116:0/1970468258 >> v1:172.21.15.174:6789/0 conn(0x7f641c0a94a0 legacy=0x7f641c071680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:05.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.578+0000 7f641a7fc640 1 -- 172.21.15.116:0/1970468258 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f641c0721c0 con 0x7f641c14c720 2024-04-25T00:01:05.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.578+0000 7f641a7fc640 1 -- 172.21.15.116:0/1970468258 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f6410001680 con 0x7f641c14c720 2024-04-25T00:01:05.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.578+0000 7f641a7fc640 1 -- 172.21.15.116:0/1970468258 >> v1:172.21.15.116:6789/0 conn(0x7f641c14c720 legacy=0x7f641c0c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:05.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.578+0000 7f641a7fc640 1 --2- 172.21.15.116:0/1970468258 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64040039f0 0x7f6404003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:05.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.578+0000 7f641a7fc640 1 --2- 172.21.15.116:0/1970468258 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6404004810 0x7f6404004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:05.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.578+0000 7f641a7fc640 1 --2- 172.21.15.116:0/1970468258 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f641c071d90 0x7f6404005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:05.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.578+0000 7f641a7fc640 1 -- 172.21.15.116:0/1970468258 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f641c153460 con 0x7f64040039f0 2024-04-25T00:01:05.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.578+0000 7f641a7fc640 1 -- 172.21.15.116:0/1970468258 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f640400dd40 con 0x7f6404004810 2024-04-25T00:01:05.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.578+0000 7f641a7fc640 1 -- 172.21.15.116:0/1970468258 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f640400df40 con 0x7f641c071d90 2024-04-25T00:01:05.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.578+0000 7f641b7fe640 1 --2- 172.21.15.116:0/1970468258 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64040039f0 0x7f6404003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:05.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.578+0000 7f641b7fe640 1 -- 172.21.15.116:0/1970468258 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f641c071d90 msgr2=0x7f6404005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:05.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.578+0000 7f641affd640 1 --2- 172.21.15.116:0/1970468258 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6404004810 0x7f6404004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:05.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.578+0000 7f641b7fe640 1 --2- 172.21.15.116:0/1970468258 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f641c071d90 0x7f6404005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:05.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.578+0000 7f641b7fe640 1 -- 172.21.15.116:0/1970468258 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6404004810 msgr2=0x7f6404004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:05.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.578+0000 7f641b7fe640 1 --2- 172.21.15.116:0/1970468258 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6404004810 0x7f6404004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:05.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.578+0000 7f641b7fe640 1 -- 172.21.15.116:0/1970468258 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6404003530 con 0x7f64040039f0 2024-04-25T00:01:05.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.578+0000 7f641affd640 1 --2- 172.21.15.116:0/1970468258 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6404004810 0x7f6404004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:05.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.582+0000 7f641b7fe640 1 --2- 172.21.15.116:0/1970468258 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64040039f0 0x7f6404003e10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f641000c210 tx=0x7f641001cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57001835326bbfce server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:05.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.582+0000 7f641a7fc640 1 -- 172.21.15.116:0/1970468258 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6410007d40 con 0x7f64040039f0 2024-04-25T00:01:05.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.582+0000 7f641a7fc640 1 -- 172.21.15.116:0/1970468258 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64100a9070 con 0x7f64040039f0 2024-04-25T00:01:05.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.582+0000 7f641a7fc640 1 -- 172.21.15.116:0/1970468258 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6410005d40 con 0x7f64040039f0 2024-04-25T00:01:05.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.582+0000 7f6421cb6640 1 -- 172.21.15.116:0/1970468258 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64040039f0 msgr2=0x7f6404003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:05.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.582+0000 7f6421cb6640 1 --2- 172.21.15.116:0/1970468258 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64040039f0 0x7f6404003e10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f641000c210 tx=0x7f641001cca0 comp rx=0 tx=0).stop 2024-04-25T00:01:05.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.582+0000 7f641bfff640 1 -- 172.21.15.116:0/1970468258 reap_dead start 2024-04-25T00:01:05.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.582+0000 7f6421cb6640 1 -- 172.21.15.116:0/1970468258 shutdown_connections 2024-04-25T00:01:05.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.582+0000 7f6421cb6640 1 -- 172.21.15.116:0/1970468258 >> 172.21.15.116:0/1970468258 conn(0x7f641c0c3b00 msgr2=0x7f641c0c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:05.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.582+0000 7f6421cb6640 1 -- 172.21.15.116:0/1970468258 shutdown_connections 2024-04-25T00:01:05.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.582+0000 7f6421cb6640 1 -- 172.21.15.116:0/1970468258 wait complete. 2024-04-25T00:01:05.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.582+0000 7f6421cb6640 1 Processor -- start 2024-04-25T00:01:05.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.582+0000 7f6421cb6640 1 -- start start 2024-04-25T00:01:05.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.582+0000 7f6421cb6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64040039f0 0x7f641c14ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:05.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.582+0000 7f6421cb6640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f641c0c5430 0x7f641c151060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:05.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.582+0000 7f6421cb6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f641c1515a0 0x7f641c1ea070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:05.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.582+0000 7f6421cb6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f641c152c60 con 0x7f641c1515a0 2024-04-25T00:01:05.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.582+0000 7f6421cb6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f641c152980 con 0x7f64040039f0 2024-04-25T00:01:05.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.582+0000 7f6421cb6640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f641c152f40 con 0x7f641c0c5430 2024-04-25T00:01:05.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.582+0000 7f641bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f641c1515a0 0x7f641c1ea070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:05.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.582+0000 7f641bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f641c1515a0 0x7f641c1ea070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37474/0 (socket says 172.21.15.116:37474) 2024-04-25T00:01:05.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.582+0000 7f641bfff640 1 -- 172.21.15.116:0/3065627240 learned_addr learned my addr 172.21.15.116:0/3065627240 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:05.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.582+0000 7f641b7fe640 1 --2- 172.21.15.116:0/3065627240 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64040039f0 0x7f641c14ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:05.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.582+0000 7f641affd640 1 --2- 172.21.15.116:0/3065627240 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f641c0c5430 0x7f641c151060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:05.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.582+0000 7f641bfff640 1 -- 172.21.15.116:0/3065627240 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f641c0c5430 msgr2=0x7f641c151060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:05.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.582+0000 7f641bfff640 1 --2- 172.21.15.116:0/3065627240 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f641c0c5430 0x7f641c151060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:05.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.582+0000 7f641bfff640 1 -- 172.21.15.116:0/3065627240 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64040039f0 msgr2=0x7f641c14ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:05.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.582+0000 7f641bfff640 1 --2- 172.21.15.116:0/3065627240 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64040039f0 0x7f641c14ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:05.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.586+0000 7f641bfff640 1 -- 172.21.15.116:0/3065627240 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f641c0c4c90 con 0x7f641c1515a0 2024-04-25T00:01:05.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.586+0000 7f641affd640 1 --2- 172.21.15.116:0/3065627240 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f641c0c5430 0x7f641c151060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:05.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.586+0000 7f641b7fe640 1 --2- 172.21.15.116:0/3065627240 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64040039f0 0x7f641c14ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:01:05.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.586+0000 7f641bfff640 1 --2- 172.21.15.116:0/3065627240 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f641c1515a0 0x7f641c1ea070 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6404004790 tx=0x7f641400b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:05.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.586+0000 7f6403fff640 1 -- 172.21.15.116:0/3065627240 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f641400bab0 con 0x7f641c1515a0 2024-04-25T00:01:05.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.586+0000 7f6403fff640 1 -- 172.21.15.116:0/3065627240 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f641401a070 con 0x7f641c1515a0 2024-04-25T00:01:05.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.586+0000 7f6421cb6640 1 -- 172.21.15.116:0/3065627240 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f641c1ed780 con 0x7f641c1515a0 2024-04-25T00:01:05.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.586+0000 7f6403fff640 1 -- 172.21.15.116:0/3065627240 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6414014400 con 0x7f641c1515a0 2024-04-25T00:01:05.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.586+0000 7f6421cb6640 1 -- 172.21.15.116:0/3065627240 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f641c145680 con 0x7f641c1515a0 2024-04-25T00:01:05.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.586+0000 7f6421cb6640 1 -- 172.21.15.116:0/3065627240 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f641c068350 con 0x7f641c1515a0 2024-04-25T00:01:05.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.586+0000 7f6403fff640 1 -- 172.21.15.116:0/3065627240 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6414013070 con 0x7f641c1515a0 2024-04-25T00:01:05.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.586+0000 7f6403fff640 1 --2- 172.21.15.116:0/3065627240 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f64080678c0 0x7f6408069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:05.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.586+0000 7f6403fff640 1 -- 172.21.15.116:0/3065627240 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5689+0+0 (secure 0 0 0) 0x7f6414085c80 con 0x7f641c1515a0 2024-04-25T00:01:05.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.586+0000 7f641b7fe640 1 --2- 172.21.15.116:0/3065627240 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f64080678c0 0x7f6408069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:05.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.586+0000 7f641b7fe640 1 --2- 172.21.15.116:0/3065627240 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f64080678c0 0x7f6408069d80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f64100ab000 tx=0x7f64100b1040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:05.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.590+0000 7f6403fff640 1 -- 172.21.15.116:0/3065627240 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f64140567d0 con 0x7f641c1515a0 2024-04-25T00:01:05.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.738+0000 7f6421cb6640 1 -- 172.21.15.116:0/3065627240 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f641c0c3640 con 0x7f641c1515a0 2024-04-25T00:01:05.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.742+0000 7f6403fff640 1 -- 172.21.15.116:0/3065627240 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16626 (secure 0 0 0) 0x7f64140560d0 con 0x7f641c1515a0 2024-04-25T00:01:05.747 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:01:05.748 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":22,"fsid":"6d6a1aa7-0478-4c83-8057-05917e81f046","created":"2024-04-25T00:00:40.682493+0000","modified":"2024-04-25T00:01:04.677995+0000","last_up_change":"2024-04-25T00:00:51.822174+0000","last_in_change":"2024-04-25T00:00:42.980473+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T00:00:53.324376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-04-25T00:01:03.197326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-04-25T00:01:03.728618+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8b6ddcb2-1ad9-4167-961c-bd6936bb9060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6809","nonce":3751765682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6811","nonce":3751765682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6815","nonce":3751765682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6813","nonce":3751765682}]},"public_addr":"172.21.15.116:6809/3751765682","cluster_addr":"172.21.15.116:6811/3751765682","heartbeat_back_addr":"172.21.15.116:6815/3751765682","heartbeat_front_addr":"172.21.15.116:6813/3751765682","state":["exists","up"]},{"osd":1,"uuid":"a6431768-994f-41da-8853-886f5687fb91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6801","nonce":377879124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6803","nonce":377879124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6807","nonce":377879124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6805","nonce":377879124}]},"public_addr":"172.21.15.116:6801/377879124","cluster_addr":"172.21.15.116:6803/377879124","heartbeat_back_addr":"172.21.15.116:6807/377879124","heartbeat_front_addr":"172.21.15.116:6805/377879124","state":["exists","up"]},{"osd":2,"uuid":"4f69bd40-5a23-45ff-8693-e1de75e1aa9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6817","nonce":1099282267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6819","nonce":1099282267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6823","nonce":1099282267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6821","nonce":1099282267}]},"public_addr":"172.21.15.116:6817/1099282267","cluster_addr":"172.21.15.116:6819/1099282267","heartbeat_back_addr":"172.21.15.116:6823/1099282267","heartbeat_front_addr":"172.21.15.116:6821/1099282267","state":["exists","up"]},{"osd":3,"uuid":"6a0d5344-8b6f-40d3-918d-0f0a4be688e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6825","nonce":1179942661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6827","nonce":1179942661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6831","nonce":1179942661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6829","nonce":1179942661}]},"public_addr":"172.21.15.116:6825/1179942661","cluster_addr":"172.21.15.116:6827/1179942661","heartbeat_back_addr":"172.21.15.116:6831/1179942661","heartbeat_front_addr":"172.21.15.116:6829/1179942661","state":["exists","up"]},{"osd":4,"uuid":"132e92e4-b825-441e-965a-ca00533fa86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6801","nonce":3132444330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6803","nonce":3132444330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6807","nonce":3132444330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6805","nonce":3132444330}]},"public_addr":"172.21.15.174:6801/3132444330","cluster_addr":"172.21.15.174:6803/3132444330","heartbeat_back_addr":"172.21.15.174:6807/3132444330","heartbeat_front_addr":"172.21.15.174:6805/3132444330","state":["exists","up"]},{"osd":5,"uuid":"a2eeb5b6-1b31-43cc-8306-a0adf3ee15bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6809","nonce":881817164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6811","nonce":881817164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6815","nonce":881817164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6813","nonce":881817164}]},"public_addr":"172.21.15.174:6809/881817164","cluster_addr":"172.21.15.174:6811/881817164","heartbeat_back_addr":"172.21.15.174:6815/881817164","heartbeat_front_addr":"172.21.15.174:6813/881817164","state":["exists","up"]},{"osd":6,"uuid":"588e873a-c841-4ba7-90f2-8956c87a9983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6825","nonce":117574251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6827","nonce":117574251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6831","nonce":117574251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6829","nonce":117574251}]},"public_addr":"172.21.15.174:6825/117574251","cluster_addr":"172.21.15.174:6827/117574251","heartbeat_back_addr":"172.21.15.174:6831/117574251","heartbeat_front_addr":"172.21.15.174:6829/117574251","state":["exists","up"]},{"osd":7,"uuid":"3b90f9b0-8a0e-4883-b088-ca1d4012870c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6817","nonce":2543899196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6819","nonce":2543899196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6823","nonce":2543899196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6821","nonce":2543899196}]},"public_addr":"172.21.15.174:6817/2543899196","cluster_addr":"172.21.15.174:6819/2543899196","heartbeat_back_addr":"172.21.15.174:6823/2543899196","heartbeat_front_addr":"172.21.15.174:6821/2543899196","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.293769+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.310034+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.404865+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.549658+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.192174+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.297266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.493649+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.452309+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:01:05.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.742+0000 7f6421cb6640 1 -- 172.21.15.116:0/3065627240 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f64080678c0 msgr2=0x7f6408069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:05.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.742+0000 7f6421cb6640 1 --2- 172.21.15.116:0/3065627240 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f64080678c0 0x7f6408069d80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f64100ab000 tx=0x7f64100b1040 comp rx=0 tx=0).stop 2024-04-25T00:01:05.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.742+0000 7f6421cb6640 1 -- 172.21.15.116:0/3065627240 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f641c1515a0 msgr2=0x7f641c1ea070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:05.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.742+0000 7f6421cb6640 1 --2- 172.21.15.116:0/3065627240 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f641c1515a0 0x7f641c1ea070 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6404004790 tx=0x7f641400b480 comp rx=0 tx=0).stop 2024-04-25T00:01:05.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.742+0000 7f6421cb6640 1 -- 172.21.15.116:0/3065627240 shutdown_connections 2024-04-25T00:01:05.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.742+0000 7f6421cb6640 1 --2- 172.21.15.116:0/3065627240 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f64080678c0 0x7f6408069d80 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:05.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.742+0000 7f6421cb6640 1 --2- 172.21.15.116:0/3065627240 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f641c1515a0 0x7f641c1ea070 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:05.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.742+0000 7f6421cb6640 1 --2- 172.21.15.116:0/3065627240 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f641c0c5430 0x7f641c151060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:05.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.742+0000 7f6421cb6640 1 --2- 172.21.15.116:0/3065627240 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64040039f0 0x7f641c14ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:05.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.742+0000 7f6421cb6640 1 -- 172.21.15.116:0/3065627240 >> 172.21.15.116:0/3065627240 conn(0x7f641c0c3b00 msgr2=0x7f641c072c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:05.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.742+0000 7f6421cb6640 1 -- 172.21.15.116:0/3065627240 shutdown_connections 2024-04-25T00:01:05.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.742+0000 7f6421cb6640 1 -- 172.21.15.116:0/3065627240 wait complete. 2024-04-25T00:01:05.763 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T00:01:05.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.882+0000 7fe697807640 1 Processor -- start 2024-04-25T00:01:05.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.882+0000 7fe697807640 1 -- start start 2024-04-25T00:01:05.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.882+0000 7fe697807640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe690148890 0x7fe690072590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:05.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.882+0000 7fe697807640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe690070dd0 0x7fe690071270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:05.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.882+0000 7fe697807640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fe69006e890 0x7fe6900717b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:05.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.882+0000 7fe697807640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fe690152c10 con 0x7fe690070dd0 2024-04-25T00:01:05.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.882+0000 7fe697807640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fe690071cf0 con 0x7fe69006e890 2024-04-25T00:01:05.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.882+0000 7fe697807640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe690151e50 con 0x7fe690148890 2024-04-25T00:01:05.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.882+0000 7fe694da2640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe690070dd0 0x7fe690071270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:05.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.882+0000 7fe694da2640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe690070dd0 0x7fe690071270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37478/0 (socket says 172.21.15.116:37478) 2024-04-25T00:01:05.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.882+0000 7fe694da2640 1 -- 172.21.15.116:0/1984401412 learned_addr learned my addr 172.21.15.116:0/1984401412 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:05.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.882+0000 7fe6955a3640 1 --2- 172.21.15.116:0/1984401412 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe690148890 0x7fe690072590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:05.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.882+0000 7fe694da2640 1 -- 172.21.15.116:0/1984401412 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe690148890 msgr2=0x7fe690072590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:05.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.882+0000 7fe694da2640 1 --2- 172.21.15.116:0/1984401412 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe690148890 0x7fe690072590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:05.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.882+0000 7fe694da2640 1 -- 172.21.15.116:0/1984401412 >> v2:172.21.15.174:3300/0 conn(0x7fe69006e890 msgr2=0x7fe6900717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:05.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.882+0000 7fe694da2640 1 --2- 172.21.15.116:0/1984401412 >> v2:172.21.15.174:3300/0 conn(0x7fe69006e890 0x7fe6900717b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:05.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.882+0000 7fe694da2640 1 -- 172.21.15.116:0/1984401412 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe690152130 con 0x7fe690070dd0 2024-04-25T00:01:05.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.886+0000 7fe694da2640 1 --2- 172.21.15.116:0/1984401412 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe690070dd0 0x7fe690071270 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe690070cd0 tx=0x7fe68001d8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc39427dd5971465 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:05.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.886+0000 7fe687fff640 1 -- 172.21.15.116:0/1984401412 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6800b3070 con 0x7fe690070dd0 2024-04-25T00:01:05.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.886+0000 7fe687fff640 1 -- 172.21.15.116:0/1984401412 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6800a6780 con 0x7fe690070dd0 2024-04-25T00:01:05.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.886+0000 7fe687fff640 1 -- 172.21.15.116:0/1984401412 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe68000b040 con 0x7fe690070dd0 2024-04-25T00:01:05.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.886+0000 7fe697807640 1 -- 172.21.15.116:0/1984401412 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe690070dd0 msgr2=0x7fe690071270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:05.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.886+0000 7fe697807640 1 --2- 172.21.15.116:0/1984401412 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe690070dd0 0x7fe690071270 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe690070cd0 tx=0x7fe68001d8f0 comp rx=0 tx=0).stop 2024-04-25T00:01:05.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.886+0000 7fe697807640 1 -- 172.21.15.116:0/1984401412 shutdown_connections 2024-04-25T00:01:05.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.886+0000 7fe697807640 1 --2- 172.21.15.116:0/1984401412 >> v2:172.21.15.174:3300/0 conn(0x7fe69006e890 0x7fe6900717b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:05.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.886+0000 7fe697807640 1 --2- 172.21.15.116:0/1984401412 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe690070dd0 0x7fe690071270 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:05.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.886+0000 7fe697807640 1 --2- 172.21.15.116:0/1984401412 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe690148890 0x7fe690072590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:05.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.886+0000 7fe697807640 1 -- 172.21.15.116:0/1984401412 >> 172.21.15.116:0/1984401412 conn(0x7fe6900c3b00 msgr2=0x7fe6900c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:05.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.886+0000 7fe697807640 1 -- 172.21.15.116:0/1984401412 shutdown_connections 2024-04-25T00:01:05.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.886+0000 7fe697807640 1 -- 172.21.15.116:0/1984401412 wait complete. 2024-04-25T00:01:05.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.890+0000 7fe697807640 1 Processor -- start 2024-04-25T00:01:05.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.890+0000 7fe697807640 1 -- start start 2024-04-25T00:01:05.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.890+0000 7fe697807640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe69006e890 0x7fe690144a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:05.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.890+0000 7fe697807640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe690070dd0 0x7fe690147b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:05.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.890+0000 7fe697807640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe690148890 0x7fe6901effe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:05.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.890+0000 7fe697807640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe6900ab710 con 0x7fe69006e890 2024-04-25T00:01:05.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.890+0000 7fe697807640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe6900ab590 con 0x7fe690070dd0 2024-04-25T00:01:05.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.890+0000 7fe697807640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe6900ab890 con 0x7fe690148890 2024-04-25T00:01:05.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.890+0000 7fe6955a3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe69006e890 0x7fe690144a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:05.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.890+0000 7fe6955a3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe69006e890 0x7fe690144a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37490/0 (socket says 172.21.15.116:37490) 2024-04-25T00:01:05.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.890+0000 7fe6955a3640 1 -- 172.21.15.116:0/3795308483 learned_addr learned my addr 172.21.15.116:0/3795308483 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:05.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.890+0000 7fe694da2640 1 --2- 172.21.15.116:0/3795308483 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe690070dd0 0x7fe690147b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:05.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.890+0000 7fe6955a3640 1 -- 172.21.15.116:0/3795308483 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe690148890 msgr2=0x7fe6901effe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:05.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.890+0000 7fe6955a3640 1 --2- 172.21.15.116:0/3795308483 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe690148890 0x7fe6901effe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:05.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.890+0000 7fe6955a3640 1 -- 172.21.15.116:0/3795308483 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe690070dd0 msgr2=0x7fe690147b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:05.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.890+0000 7fe6955a3640 1 --2- 172.21.15.116:0/3795308483 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe690070dd0 0x7fe690147b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:05.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.890+0000 7fe6955a3640 1 -- 172.21.15.116:0/3795308483 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe690144740 con 0x7fe69006e890 2024-04-25T00:01:05.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.890+0000 7fe6955a3640 1 --2- 172.21.15.116:0/3795308483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe69006e890 0x7fe690144a20 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe68c00c820 tx=0x7fe68c00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:05.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.890+0000 7fe685ffb640 1 -- 172.21.15.116:0/3795308483 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe68c015070 con 0x7fe69006e890 2024-04-25T00:01:05.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.890+0000 7fe685ffb640 1 -- 172.21.15.116:0/3795308483 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe68c007a30 con 0x7fe69006e890 2024-04-25T00:01:05.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.890+0000 7fe697807640 1 -- 172.21.15.116:0/3795308483 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6900bc0c0 con 0x7fe69006e890 2024-04-25T00:01:05.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.890+0000 7fe685ffb640 1 -- 172.21.15.116:0/3795308483 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe68c007e20 con 0x7fe69006e890 2024-04-25T00:01:05.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.890+0000 7fe697807640 1 -- 172.21.15.116:0/3795308483 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6900c0e00 con 0x7fe69006e890 2024-04-25T00:01:05.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.890+0000 7fe697807640 1 -- 172.21.15.116:0/3795308483 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe690068350 con 0x7fe69006e890 2024-04-25T00:01:05.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.890+0000 7fe685ffb640 1 -- 172.21.15.116:0/3795308483 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe68c018070 con 0x7fe69006e890 2024-04-25T00:01:05.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.890+0000 7fe685ffb640 1 --2- 172.21.15.116:0/3795308483 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe6740678c0 0x7fe674069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:05.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.890+0000 7fe685ffb640 1 -- 172.21.15.116:0/3795308483 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5689+0+0 (secure 0 0 0) 0x7fe68c085ab0 con 0x7fe69006e890 2024-04-25T00:01:05.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.890+0000 7fe694da2640 1 --2- 172.21.15.116:0/3795308483 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe6740678c0 0x7fe674069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:05.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.894+0000 7fe694da2640 1 --2- 172.21.15.116:0/3795308483 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe6740678c0 0x7fe674069d80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe6800b0000 tx=0x7fe6800a8070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:05.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:05.898+0000 7fe685ffb640 1 -- 172.21.15.116:0/3795308483 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fe68c056690 con 0x7fe69006e890 2024-04-25T00:01:06.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.058+0000 7fe697807640 1 -- 172.21.15.116:0/3795308483 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe69006e670 con 0x7fe69006e890 2024-04-25T00:01:06.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.062+0000 7fe685ffb640 1 -- 172.21.15.116:0/3795308483 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16626 (secure 0 0 0) 0x7fe68c055f90 con 0x7fe69006e890 2024-04-25T00:01:06.065 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:01:06.065 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":22,"fsid":"6d6a1aa7-0478-4c83-8057-05917e81f046","created":"2024-04-25T00:00:40.682493+0000","modified":"2024-04-25T00:01:04.677995+0000","last_up_change":"2024-04-25T00:00:51.822174+0000","last_in_change":"2024-04-25T00:00:42.980473+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T00:00:53.324376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-04-25T00:01:03.197326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-04-25T00:01:03.728618+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8b6ddcb2-1ad9-4167-961c-bd6936bb9060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6809","nonce":3751765682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6811","nonce":3751765682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6815","nonce":3751765682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6813","nonce":3751765682}]},"public_addr":"172.21.15.116:6809/3751765682","cluster_addr":"172.21.15.116:6811/3751765682","heartbeat_back_addr":"172.21.15.116:6815/3751765682","heartbeat_front_addr":"172.21.15.116:6813/3751765682","state":["exists","up"]},{"osd":1,"uuid":"a6431768-994f-41da-8853-886f5687fb91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6801","nonce":377879124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6803","nonce":377879124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6807","nonce":377879124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6805","nonce":377879124}]},"public_addr":"172.21.15.116:6801/377879124","cluster_addr":"172.21.15.116:6803/377879124","heartbeat_back_addr":"172.21.15.116:6807/377879124","heartbeat_front_addr":"172.21.15.116:6805/377879124","state":["exists","up"]},{"osd":2,"uuid":"4f69bd40-5a23-45ff-8693-e1de75e1aa9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6817","nonce":1099282267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6819","nonce":1099282267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6823","nonce":1099282267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6821","nonce":1099282267}]},"public_addr":"172.21.15.116:6817/1099282267","cluster_addr":"172.21.15.116:6819/1099282267","heartbeat_back_addr":"172.21.15.116:6823/1099282267","heartbeat_front_addr":"172.21.15.116:6821/1099282267","state":["exists","up"]},{"osd":3,"uuid":"6a0d5344-8b6f-40d3-918d-0f0a4be688e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6825","nonce":1179942661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6827","nonce":1179942661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6831","nonce":1179942661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6829","nonce":1179942661}]},"public_addr":"172.21.15.116:6825/1179942661","cluster_addr":"172.21.15.116:6827/1179942661","heartbeat_back_addr":"172.21.15.116:6831/1179942661","heartbeat_front_addr":"172.21.15.116:6829/1179942661","state":["exists","up"]},{"osd":4,"uuid":"132e92e4-b825-441e-965a-ca00533fa86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6801","nonce":3132444330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6803","nonce":3132444330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6807","nonce":3132444330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6805","nonce":3132444330}]},"public_addr":"172.21.15.174:6801/3132444330","cluster_addr":"172.21.15.174:6803/3132444330","heartbeat_back_addr":"172.21.15.174:6807/3132444330","heartbeat_front_addr":"172.21.15.174:6805/3132444330","state":["exists","up"]},{"osd":5,"uuid":"a2eeb5b6-1b31-43cc-8306-a0adf3ee15bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6809","nonce":881817164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6811","nonce":881817164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6815","nonce":881817164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6813","nonce":881817164}]},"public_addr":"172.21.15.174:6809/881817164","cluster_addr":"172.21.15.174:6811/881817164","heartbeat_back_addr":"172.21.15.174:6815/881817164","heartbeat_front_addr":"172.21.15.174:6813/881817164","state":["exists","up"]},{"osd":6,"uuid":"588e873a-c841-4ba7-90f2-8956c87a9983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6825","nonce":117574251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6827","nonce":117574251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6831","nonce":117574251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6829","nonce":117574251}]},"public_addr":"172.21.15.174:6825/117574251","cluster_addr":"172.21.15.174:6827/117574251","heartbeat_back_addr":"172.21.15.174:6831/117574251","heartbeat_front_addr":"172.21.15.174:6829/117574251","state":["exists","up"]},{"osd":7,"uuid":"3b90f9b0-8a0e-4883-b088-ca1d4012870c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6817","nonce":2543899196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6819","nonce":2543899196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6823","nonce":2543899196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6821","nonce":2543899196}]},"public_addr":"172.21.15.174:6817/2543899196","cluster_addr":"172.21.15.174:6819/2543899196","heartbeat_back_addr":"172.21.15.174:6823/2543899196","heartbeat_front_addr":"172.21.15.174:6821/2543899196","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.293769+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.310034+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.404865+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.549658+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.192174+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.297266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.493649+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.452309+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:01:06.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.066+0000 7fe697807640 1 -- 172.21.15.116:0/3795308483 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe6740678c0 msgr2=0x7fe674069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:06.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.066+0000 7fe697807640 1 --2- 172.21.15.116:0/3795308483 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe6740678c0 0x7fe674069d80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe6800b0000 tx=0x7fe6800a8070 comp rx=0 tx=0).stop 2024-04-25T00:01:06.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.066+0000 7fe697807640 1 -- 172.21.15.116:0/3795308483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe69006e890 msgr2=0x7fe690144a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:06.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.066+0000 7fe697807640 1 --2- 172.21.15.116:0/3795308483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe69006e890 0x7fe690144a20 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe68c00c820 tx=0x7fe68c00cce0 comp rx=0 tx=0).stop 2024-04-25T00:01:06.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.066+0000 7fe697807640 1 -- 172.21.15.116:0/3795308483 shutdown_connections 2024-04-25T00:01:06.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.066+0000 7fe697807640 1 --2- 172.21.15.116:0/3795308483 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe6740678c0 0x7fe674069d80 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:06.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.066+0000 7fe697807640 1 --2- 172.21.15.116:0/3795308483 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe690148890 0x7fe6901effe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:06.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.066+0000 7fe697807640 1 --2- 172.21.15.116:0/3795308483 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe690070dd0 0x7fe690147b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:06.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.066+0000 7fe697807640 1 --2- 172.21.15.116:0/3795308483 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe69006e890 0x7fe690144a20 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:06.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.066+0000 7fe697807640 1 -- 172.21.15.116:0/3795308483 >> 172.21.15.116:0/3795308483 conn(0x7fe6900c3b00 msgr2=0x7fe69006d330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:06.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.066+0000 7fe697807640 1 -- 172.21.15.116:0/3795308483 shutdown_connections 2024-04-25T00:01:06.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.066+0000 7fe697807640 1 -- 172.21.15.116:0/3795308483 wait complete. 2024-04-25T00:01:06.083 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set a standby_count_wanted 0 2024-04-25T00:01:06.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.198+0000 7f0054fd6640 1 Processor -- start 2024-04-25T00:01:06.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.198+0000 7f0054fd6640 1 -- start start 2024-04-25T00:01:06.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.202+0000 7f0054fd6640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0050148f20 0x7f00500c2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:06.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.202+0000 7f0054fd6640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0050149300 con 0x7f005006d510 2024-04-25T00:01:06.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.202+0000 7f0054fd6640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00500a95d0 con 0x7f005006d000 2024-04-25T00:01:06.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.202+0000 7f0054fd6640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f0050153350 con 0x7f0050148f20 2024-04-25T00:01:06.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.202+0000 7f004dd9b640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f005006d510 0x7f0050072500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:47850/0 (socket says 172.21.15.116:47850) 2024-04-25T00:01:06.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.202+0000 7f004dd9b640 1 -- 172.21.15.116:0/826311644 learned_addr learned my addr 172.21.15.116:0/826311644 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:06.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.202+0000 7f004e59c640 1 --2- 172.21.15.116:0/826311644 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0050148f20 0x7f00500c2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:06.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.202+0000 7f004d59a640 1 -- 172.21.15.116:0/826311644 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2247790172 0 0) 0x7f0044001120 con 0x7f005006d510 2024-04-25T00:01:06.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.202+0000 7f004d59a640 1 -- 172.21.15.116:0/826311644 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0038003530 con 0x7f005006d510 2024-04-25T00:01:06.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.202+0000 7f004d59a640 1 -- 172.21.15.116:0/826311644 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 802243129 0 0) 0x7f0038003530 con 0x7f005006d510 2024-04-25T00:01:06.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.202+0000 7f004d59a640 1 -- 172.21.15.116:0/826311644 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0050148f20 msgr2=0x7f00500c2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:06.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.202+0000 7f004d59a640 1 --2- 172.21.15.116:0/826311644 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0050148f20 0x7f00500c2930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:06.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.202+0000 7f004d59a640 1 -- 172.21.15.116:0/826311644 >> v1:172.21.15.174:6789/0 conn(0x7f005006d000 legacy=0x7f005015cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:06.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.202+0000 7f004d59a640 1 -- 172.21.15.116:0/826311644 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0050153070 con 0x7f005006d510 2024-04-25T00:01:06.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.202+0000 7f004d59a640 1 -- 172.21.15.116:0/826311644 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f0044001850 con 0x7f005006d510 2024-04-25T00:01:06.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.202+0000 7f004d59a640 1 -- 172.21.15.116:0/826311644 >> v1:172.21.15.116:6789/0 conn(0x7f005006d510 legacy=0x7f0050072500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:06.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.202+0000 7f004d59a640 1 --2- 172.21.15.116:0/826311644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00380039f0 0x7f0038003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:06.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.202+0000 7f004d59a640 1 --2- 172.21.15.116:0/826311644 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0050148f20 0x7f0038004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:06.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.202+0000 7f004d59a640 1 --2- 172.21.15.116:0/826311644 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0038004d50 0x7f00380051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:06.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.202+0000 7f004d59a640 1 -- 172.21.15.116:0/826311644 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0050153350 con 0x7f00380039f0 2024-04-25T00:01:06.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.202+0000 7f004d59a640 1 -- 172.21.15.116:0/826311644 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f003800bb90 con 0x7f0038004d50 2024-04-25T00:01:06.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.202+0000 7f004d59a640 1 -- 172.21.15.116:0/826311644 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f003800bd90 con 0x7f0050148f20 2024-04-25T00:01:06.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.202+0000 7f004e59c640 1 --2- 172.21.15.116:0/826311644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00380039f0 0x7f0038003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:06.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.202+0000 7f004dd9b640 1 --2- 172.21.15.116:0/826311644 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0050148f20 0x7f0038004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:06.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.202+0000 7f004ed9d640 1 --2- 172.21.15.116:0/826311644 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0038004d50 0x7f00380051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:06.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.202+0000 7f004e59c640 1 -- 172.21.15.116:0/826311644 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0050148f20 msgr2=0x7f0038004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:06.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.202+0000 7f004e59c640 1 --2- 172.21.15.116:0/826311644 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0050148f20 0x7f0038004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:06.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.202+0000 7f004e59c640 1 -- 172.21.15.116:0/826311644 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0038004d50 msgr2=0x7f00380051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:06.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.202+0000 7f004e59c640 1 --2- 172.21.15.116:0/826311644 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0038004d50 0x7f00380051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:06.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.202+0000 7f004e59c640 1 -- 172.21.15.116:0/826311644 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0038003530 con 0x7f00380039f0 2024-04-25T00:01:06.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.202+0000 7f004ed9d640 1 --2- 172.21.15.116:0/826311644 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0038004d50 0x7f00380051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:06.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.206+0000 7f004e59c640 1 --2- 172.21.15.116:0/826311644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00380039f0 0x7f0038003e10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0040009ae0 tx=0x7f004001ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad6ca069b77f1a13 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:06.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.206+0000 7f004d59a640 1 -- 172.21.15.116:0/826311644 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00400b2070 con 0x7f00380039f0 2024-04-25T00:01:06.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.206+0000 7f004d59a640 1 -- 172.21.15.116:0/826311644 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00400a5550 con 0x7f00380039f0 2024-04-25T00:01:06.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.206+0000 7f004d59a640 1 -- 172.21.15.116:0/826311644 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00400096b0 con 0x7f00380039f0 2024-04-25T00:01:06.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.206+0000 7f0054fd6640 1 -- 172.21.15.116:0/826311644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00380039f0 msgr2=0x7f0038003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:06.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.206+0000 7f0054fd6640 1 --2- 172.21.15.116:0/826311644 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00380039f0 0x7f0038003e10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0040009ae0 tx=0x7f004001ca70 comp rx=0 tx=0).stop 2024-04-25T00:01:06.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.206+0000 7f004ed9d640 1 -- 172.21.15.116:0/826311644 reap_dead start 2024-04-25T00:01:06.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.206+0000 7f0054fd6640 1 -- 172.21.15.116:0/826311644 shutdown_connections 2024-04-25T00:01:06.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.206+0000 7f0054fd6640 1 -- 172.21.15.116:0/826311644 >> 172.21.15.116:0/826311644 conn(0x7f00500c3a20 msgr2=0x7f00500c3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:06.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.206+0000 7f0054fd6640 1 -- 172.21.15.116:0/826311644 shutdown_connections 2024-04-25T00:01:06.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.206+0000 7f0054fd6640 1 -- 172.21.15.116:0/826311644 wait complete. 2024-04-25T00:01:06.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.206+0000 7f0054fd6640 1 Processor -- start 2024-04-25T00:01:06.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.206+0000 7f0054fd6640 1 -- start start 2024-04-25T00:01:06.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.206+0000 7f0054fd6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00380039f0 0x7f0050151670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:06.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.206+0000 7f0054fd6640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f00501565c0 0x7f00501418c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:06.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.206+0000 7f0054fd6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0050145f10 0x7f005014eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:06.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.206+0000 7f0054fd6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f005015e670 con 0x7f00380039f0 2024-04-25T00:01:06.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.206+0000 7f0054fd6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f005015e4f0 con 0x7f0050145f10 2024-04-25T00:01:06.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.206+0000 7f0054fd6640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f005015e7f0 con 0x7f00501565c0 2024-04-25T00:01:06.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.206+0000 7f004e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00380039f0 0x7f0050151670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:06.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.206+0000 7f004e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00380039f0 0x7f0050151670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37510/0 (socket says 172.21.15.116:37510) 2024-04-25T00:01:06.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.206+0000 7f004e59c640 1 -- 172.21.15.116:0/997496851 learned_addr learned my addr 172.21.15.116:0/997496851 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:06.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.206+0000 7f004dd9b640 1 --2- 172.21.15.116:0/997496851 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f00501565c0 0x7f00501418c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:06.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.206+0000 7f004ed9d640 1 --2- 172.21.15.116:0/997496851 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0050145f10 0x7f005014eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:06.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.206+0000 7f004e59c640 1 -- 172.21.15.116:0/997496851 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f00501565c0 msgr2=0x7f00501418c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:06.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.206+0000 7f004e59c640 1 --2- 172.21.15.116:0/997496851 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f00501565c0 0x7f00501418c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:06.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.206+0000 7f004e59c640 1 -- 172.21.15.116:0/997496851 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0050145f10 msgr2=0x7f005014eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:06.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.206+0000 7f004e59c640 1 --2- 172.21.15.116:0/997496851 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0050145f10 0x7f005014eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:06.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.206+0000 7f004e59c640 1 -- 172.21.15.116:0/997496851 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f005014d650 con 0x7f00380039f0 2024-04-25T00:01:06.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.206+0000 7f004e59c640 1 --2- 172.21.15.116:0/997496851 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00380039f0 0x7f0050151670 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f004001c8a0 tx=0x7f0038005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:06.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.206+0000 7f003effd640 1 -- 172.21.15.116:0/997496851 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00400bb070 con 0x7f00380039f0 2024-04-25T00:01:06.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.206+0000 7f003effd640 1 -- 172.21.15.116:0/997496851 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0040004020 con 0x7f00380039f0 2024-04-25T00:01:06.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.206+0000 7f003effd640 1 -- 172.21.15.116:0/997496851 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00400b2040 con 0x7f00380039f0 2024-04-25T00:01:06.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.206+0000 7f0054fd6640 1 -- 172.21.15.116:0/997496851 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00501447f0 con 0x7f00380039f0 2024-04-25T00:01:06.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.210+0000 7f0054fd6640 1 -- 172.21.15.116:0/997496851 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0050144db0 con 0x7f00380039f0 2024-04-25T00:01:06.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.210+0000 7f0054fd6640 1 -- 172.21.15.116:0/997496851 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f005004f510 con 0x7f00380039f0 2024-04-25T00:01:06.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.210+0000 7f003effd640 1 -- 172.21.15.116:0/997496851 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f00400be070 con 0x7f00380039f0 2024-04-25T00:01:06.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.210+0000 7f003effd640 1 --2- 172.21.15.116:0/997496851 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f00300678f0 0x7f0030069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:06.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.210+0000 7f003effd640 1 -- 172.21.15.116:0/997496851 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5689+0+0 (secure 0 0 0) 0x7f004011f040 con 0x7f00380039f0 2024-04-25T00:01:06.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.210+0000 7f004dd9b640 1 --2- 172.21.15.116:0/997496851 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f00300678f0 0x7f0030069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:06.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.210+0000 7f004dd9b640 1 --2- 172.21.15.116:0/997496851 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f00300678f0 0x7f0030069db0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0050156550 tx=0x7f0044009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:06.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.214+0000 7f003effd640 1 -- 172.21.15.116:0/997496851 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f0040016780 con 0x7f00380039f0 2024-04-25T00:01:06.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:06.406+0000 7f0054fd6640 1 -- 172.21.15.116:0/997496851 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "a", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f00500c34b0 con 0x7f00380039f0 2024-04-25T00:01:07.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.394+0000 7f003effd640 1 -- 172.21.15.116:0/997496851 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "a", "var": "standby_count_wanted", "val": "0"}]=0 v7) ==== 113+0+0 (secure 0 0 0) 0x7f00400ef540 con 0x7f00380039f0 2024-04-25T00:01:07.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.398+0000 7f0054fd6640 1 -- 172.21.15.116:0/997496851 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f00300678f0 msgr2=0x7f0030069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:07.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.398+0000 7f0054fd6640 1 --2- 172.21.15.116:0/997496851 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f00300678f0 0x7f0030069db0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0050156550 tx=0x7f0044009040 comp rx=0 tx=0).stop 2024-04-25T00:01:07.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.398+0000 7f0054fd6640 1 -- 172.21.15.116:0/997496851 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00380039f0 msgr2=0x7f0050151670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:07.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.398+0000 7f0054fd6640 1 --2- 172.21.15.116:0/997496851 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00380039f0 0x7f0050151670 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f004001c8a0 tx=0x7f0038005840 comp rx=0 tx=0).stop 2024-04-25T00:01:07.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.398+0000 7f0054fd6640 1 -- 172.21.15.116:0/997496851 shutdown_connections 2024-04-25T00:01:07.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.398+0000 7f0054fd6640 1 --2- 172.21.15.116:0/997496851 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f00300678f0 0x7f0030069db0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:07.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.398+0000 7f0054fd6640 1 --2- 172.21.15.116:0/997496851 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0050145f10 0x7f005014eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:07.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.398+0000 7f0054fd6640 1 --2- 172.21.15.116:0/997496851 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f00501565c0 0x7f00501418c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:07.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.398+0000 7f0054fd6640 1 --2- 172.21.15.116:0/997496851 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00380039f0 0x7f0050151670 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:07.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.398+0000 7f0054fd6640 1 -- 172.21.15.116:0/997496851 >> 172.21.15.116:0/997496851 conn(0x7f00500c3a20 msgr2=0x7f0050142760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:07.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.398+0000 7f0054fd6640 1 -- 172.21.15.116:0/997496851 shutdown_connections 2024-04-25T00:01:07.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.398+0000 7f0054fd6640 1 -- 172.21.15.116:0/997496851 wait complete. 2024-04-25T00:01:07.420 DEBUG:tasks.cephfs.filesystem:fs_config: {'session_timeout': 300} 2024-04-25T00:01:07.420 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set a allow_standby_replay false 2024-04-25T00:01:07.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.542+0000 7fd361ed2640 1 Processor -- start 2024-04-25T00:01:07.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.542+0000 7fd361ed2640 1 -- start start 2024-04-25T00:01:07.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.542+0000 7fd361ed2640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd35c150d60 0x7fd35c0c2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:07.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.542+0000 7fd361ed2640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd35c0c2e70 0x7fd35c157510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:07.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.542+0000 7fd361ed2640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd35c0a9cb0 con 0x7fd35c0a9630 2024-04-25T00:01:07.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.542+0000 7fd361ed2640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fd35c0a9f90 con 0x7fd35c0c2e70 2024-04-25T00:01:07.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.542+0000 7fd361ed2640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fd35c0aa270 con 0x7fd35c150d60 2024-04-25T00:01:07.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.542+0000 7fd35bfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd35c0c2e70 0x7fd35c157510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:07.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.542+0000 7fd35bfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd35c0c2e70 0x7fd35c157510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37518/0 (socket says 172.21.15.116:37518) 2024-04-25T00:01:07.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.542+0000 7fd35bfff640 1 -- 172.21.15.116:0/2329210792 learned_addr learned my addr 172.21.15.116:0/2329210792 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:07.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.546+0000 7fd35bfff640 1 -- 172.21.15.116:0/2329210792 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd35c150d60 msgr2=0x7fd35c0c2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:07.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.546+0000 7fd35bfff640 1 --2- 172.21.15.116:0/2329210792 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd35c150d60 0x7fd35c0c2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:07.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.546+0000 7fd35bfff640 1 -- 172.21.15.116:0/2329210792 >> v1:172.21.15.174:6789/0 conn(0x7fd35c0a9630 legacy=0x7fd35c156e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:07.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.546+0000 7fd35bfff640 1 -- 172.21.15.116:0/2329210792 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd35c0aa550 con 0x7fd35c0c2e70 2024-04-25T00:01:07.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.546+0000 7fd35bfff640 1 --2- 172.21.15.116:0/2329210792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd35c0c2e70 0x7fd35c157510 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd354009e50 tx=0x7fd35401c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97641e667e81a3f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:07.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.546+0000 7fd35a7fc640 1 -- 172.21.15.116:0/2329210792 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3540a5740 con 0x7fd35c0c2e70 2024-04-25T00:01:07.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.546+0000 7fd35a7fc640 1 -- 172.21.15.116:0/2329210792 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3540a58e0 con 0x7fd35c0c2e70 2024-04-25T00:01:07.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.546+0000 7fd35a7fc640 1 -- 172.21.15.116:0/2329210792 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3540a5cb0 con 0x7fd35c0c2e70 2024-04-25T00:01:07.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.546+0000 7fd361ed2640 1 -- 172.21.15.116:0/2329210792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd35c0c2e70 msgr2=0x7fd35c157510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:07.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.546+0000 7fd361ed2640 1 --2- 172.21.15.116:0/2329210792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd35c0c2e70 0x7fd35c157510 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd354009e50 tx=0x7fd35401c8a0 comp rx=0 tx=0).stop 2024-04-25T00:01:07.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.546+0000 7fd361ed2640 1 -- 172.21.15.116:0/2329210792 shutdown_connections 2024-04-25T00:01:07.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.546+0000 7fd361ed2640 1 --2- 172.21.15.116:0/2329210792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd35c0c2e70 0x7fd35c157510 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:07.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.546+0000 7fd361ed2640 1 --2- 172.21.15.116:0/2329210792 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd35c150d60 0x7fd35c0c2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:07.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.546+0000 7fd361ed2640 1 -- 172.21.15.116:0/2329210792 >> 172.21.15.116:0/2329210792 conn(0x7fd35c0c3a20 msgr2=0x7fd35c0c3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:07.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.546+0000 7fd361ed2640 1 -- 172.21.15.116:0/2329210792 shutdown_connections 2024-04-25T00:01:07.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.546+0000 7fd361ed2640 1 -- 172.21.15.116:0/2329210792 wait complete. 2024-04-25T00:01:07.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.550+0000 7fd361ed2640 1 Processor -- start 2024-04-25T00:01:07.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.550+0000 7fd361ed2640 1 -- start start 2024-04-25T00:01:07.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.550+0000 7fd361ed2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd35c0a9630 0x7fd35c15c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:07.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.550+0000 7fd361ed2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd35c0c2e70 0x7fd35c141840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:07.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.550+0000 7fd361ed2640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd35c150d60 0x7fd35c141d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:07.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.550+0000 7fd361ed2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd35c071fd0 con 0x7fd35c0c2e70 2024-04-25T00:01:07.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.550+0000 7fd361ed2640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd35c0ab7a0 con 0x7fd35c0a9630 2024-04-25T00:01:07.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.550+0000 7fd361ed2640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fd35c072150 con 0x7fd35c150d60 2024-04-25T00:01:07.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.550+0000 7fd35affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd35c0c2e70 0x7fd35c141840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:07.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.550+0000 7fd35affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd35c0c2e70 0x7fd35c141840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37522/0 (socket says 172.21.15.116:37522) 2024-04-25T00:01:07.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.550+0000 7fd35affd640 1 -- 172.21.15.116:0/1846027128 learned_addr learned my addr 172.21.15.116:0/1846027128 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:07.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.550+0000 7fd35bfff640 1 --2- 172.21.15.116:0/1846027128 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd35c150d60 0x7fd35c141d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:07.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.550+0000 7fd35affd640 1 -- 172.21.15.116:0/1846027128 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd35c150d60 msgr2=0x7fd35c141d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:07.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.550+0000 7fd35affd640 1 --2- 172.21.15.116:0/1846027128 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd35c150d60 0x7fd35c141d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:07.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.550+0000 7fd35affd640 1 -- 172.21.15.116:0/1846027128 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd35c0a9630 msgr2=0x7fd35c15c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:07.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.550+0000 7fd35affd640 1 --2- 172.21.15.116:0/1846027128 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd35c0a9630 0x7fd35c15c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:07.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.550+0000 7fd35affd640 1 -- 172.21.15.116:0/1846027128 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd35c1424c0 con 0x7fd35c0c2e70 2024-04-25T00:01:07.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.550+0000 7fd35affd640 1 --2- 172.21.15.116:0/1846027128 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd35c0c2e70 0x7fd35c141840 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd35000b3e0 tx=0x7fd35000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:07.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.550+0000 7fd343fff640 1 -- 172.21.15.116:0/1846027128 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd350015020 con 0x7fd35c0c2e70 2024-04-25T00:01:07.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.550+0000 7fd343fff640 1 -- 172.21.15.116:0/1846027128 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd350011850 con 0x7fd35c0c2e70 2024-04-25T00:01:07.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.550+0000 7fd361ed2640 1 -- 172.21.15.116:0/1846027128 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd35c1ee680 con 0x7fd35c0c2e70 2024-04-25T00:01:07.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.550+0000 7fd343fff640 1 -- 172.21.15.116:0/1846027128 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd350011c40 con 0x7fd35c0c2e70 2024-04-25T00:01:07.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.550+0000 7fd361ed2640 1 -- 172.21.15.116:0/1846027128 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd35c1eec20 con 0x7fd35c0c2e70 2024-04-25T00:01:07.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.550+0000 7fd361ed2640 1 -- 172.21.15.116:0/1846027128 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd35c04f510 con 0x7fd35c0c2e70 2024-04-25T00:01:07.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.550+0000 7fd343fff640 1 -- 172.21.15.116:0/1846027128 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd350018020 con 0x7fd35c0c2e70 2024-04-25T00:01:07.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.550+0000 7fd343fff640 1 --2- 172.21.15.116:0/1846027128 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd3480678c0 0x7fd348069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:07.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.550+0000 7fd343fff640 1 -- 172.21.15.116:0/1846027128 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5689+0+0 (secure 0 0 0) 0x7fd350085b40 con 0x7fd35c0c2e70 2024-04-25T00:01:07.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.550+0000 7fd35b7fe640 1 --2- 172.21.15.116:0/1846027128 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd3480678c0 0x7fd348069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:07.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.554+0000 7fd35b7fe640 1 --2- 172.21.15.116:0/1846027128 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd3480678c0 0x7fd348069d80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd35c142ba0 tx=0x7fd34c006ca0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:07.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.554+0000 7fd343fff640 1 -- 172.21.15.116:0/1846027128 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fd35001f020 con 0x7fd35c0c2e70 2024-04-25T00:01:07.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:07.750+0000 7fd361ed2640 1 -- 172.21.15.116:0/1846027128 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "a", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fd35c14f190 con 0x7fd35c0c2e70 2024-04-25T00:01:08.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.398+0000 7fd343fff640 1 -- 172.21.15.116:0/1846027128 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "a", "var": "allow_standby_replay", "val": "false"}]=0 v8) ==== 117+0+0 (secure 0 0 0) 0x7fd3500172c0 con 0x7fd35c0c2e70 2024-04-25T00:01:08.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.402+0000 7fd361ed2640 1 -- 172.21.15.116:0/1846027128 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd3480678c0 msgr2=0x7fd348069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:08.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.402+0000 7fd361ed2640 1 --2- 172.21.15.116:0/1846027128 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd3480678c0 0x7fd348069d80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd35c142ba0 tx=0x7fd34c006ca0 comp rx=0 tx=0).stop 2024-04-25T00:01:08.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.402+0000 7fd361ed2640 1 -- 172.21.15.116:0/1846027128 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd35c0c2e70 msgr2=0x7fd35c141840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:08.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.402+0000 7fd361ed2640 1 --2- 172.21.15.116:0/1846027128 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd35c0c2e70 0x7fd35c141840 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd35000b3e0 tx=0x7fd35000b8a0 comp rx=0 tx=0).stop 2024-04-25T00:01:08.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.402+0000 7fd361ed2640 1 -- 172.21.15.116:0/1846027128 shutdown_connections 2024-04-25T00:01:08.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.402+0000 7fd361ed2640 1 --2- 172.21.15.116:0/1846027128 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd3480678c0 0x7fd348069d80 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:08.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.402+0000 7fd361ed2640 1 --2- 172.21.15.116:0/1846027128 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd35c150d60 0x7fd35c141d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:08.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.402+0000 7fd361ed2640 1 --2- 172.21.15.116:0/1846027128 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd35c0c2e70 0x7fd35c141840 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:08.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.402+0000 7fd361ed2640 1 --2- 172.21.15.116:0/1846027128 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd35c0a9630 0x7fd35c15c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:08.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.402+0000 7fd361ed2640 1 -- 172.21.15.116:0/1846027128 >> 172.21.15.116:0/1846027128 conn(0x7fd35c0c3a20 msgr2=0x7fd35c14d880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:08.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.402+0000 7fd361ed2640 1 -- 172.21.15.116:0/1846027128 shutdown_connections 2024-04-25T00:01:08.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.402+0000 7fd361ed2640 1 -- 172.21.15.116:0/1846027128 wait complete. 2024-04-25T00:01:08.425 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set a session_timeout 300 2024-04-25T00:01:08.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.546+0000 7fca6e5ea640 1 Processor -- start 2024-04-25T00:01:08.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.546+0000 7fca6e5ea640 1 -- start start 2024-04-25T00:01:08.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.550+0000 7fca6e5ea640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fca6806fd90 0x7fca68070170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:08.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.550+0000 7fca6e5ea640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fca680c45c0 0x7fca68156d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:08.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.550+0000 7fca6e5ea640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca680a9cb0 con 0x7fca680a95c0 2024-04-25T00:01:08.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.550+0000 7fca6e5ea640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fca680a9f90 con 0x7fca6806fd90 2024-04-25T00:01:08.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.550+0000 7fca6e5ea640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fca680aa270 con 0x7fca680c45c0 2024-04-25T00:01:08.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.550+0000 7fca67fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fca6806fd90 0x7fca68070170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:08.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.550+0000 7fca6cb87640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fca680a95c0 0x7fca681572b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:47854/0 (socket says 172.21.15.116:47854) 2024-04-25T00:01:08.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.550+0000 7fca6cb87640 1 -- 172.21.15.116:0/2902846053 learned_addr learned my addr 172.21.15.116:0/2902846053 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:08.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.550+0000 7fca677fe640 1 --2- 172.21.15.116:0/2902846053 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fca680c45c0 0x7fca68156d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:08.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.550+0000 7fca67fff640 1 -- 172.21.15.116:0/2902846053 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fca680c45c0 msgr2=0x7fca68156d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:08.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.550+0000 7fca67fff640 1 --2- 172.21.15.116:0/2902846053 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fca680c45c0 0x7fca68156d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:08.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.550+0000 7fca67fff640 1 -- 172.21.15.116:0/2902846053 >> v1:172.21.15.116:6789/0 conn(0x7fca680a95c0 legacy=0x7fca681572b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:08.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.550+0000 7fca66ffd640 1 -- 172.21.15.116:0/2902846053 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3342778421 0 0) 0x7fca680a9cb0 con 0x7fca680a95c0 2024-04-25T00:01:08.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.550+0000 7fca67fff640 1 -- 172.21.15.116:0/2902846053 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca680aa550 con 0x7fca6806fd90 2024-04-25T00:01:08.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.550+0000 7fca67fff640 1 --2- 172.21.15.116:0/2902846053 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca6806fd90 0x7fca68070170 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fca580097c0 tx=0x7fca5801c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c48e9db0cbee40a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:08.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.550+0000 7fca66ffd640 1 -- 172.21.15.116:0/2902846053 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca580ab610 con 0x7fca6806fd90 2024-04-25T00:01:08.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.550+0000 7fca66ffd640 1 -- 172.21.15.116:0/2902846053 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca580b0070 con 0x7fca6806fd90 2024-04-25T00:01:08.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.550+0000 7fca66ffd640 1 -- 172.21.15.116:0/2902846053 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca580ab9d0 con 0x7fca6806fd90 2024-04-25T00:01:08.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.550+0000 7fca6e5ea640 1 -- 172.21.15.116:0/2902846053 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca6806fd90 msgr2=0x7fca68070170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:08.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.550+0000 7fca6e5ea640 1 --2- 172.21.15.116:0/2902846053 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca6806fd90 0x7fca68070170 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fca580097c0 tx=0x7fca5801c410 comp rx=0 tx=0).stop 2024-04-25T00:01:08.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.550+0000 7fca6e5ea640 1 -- 172.21.15.116:0/2902846053 shutdown_connections 2024-04-25T00:01:08.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.550+0000 7fca6e5ea640 1 --2- 172.21.15.116:0/2902846053 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fca680c45c0 0x7fca68156d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:08.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.550+0000 7fca6e5ea640 1 --2- 172.21.15.116:0/2902846053 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca6806fd90 0x7fca68070170 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:08.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.550+0000 7fca6e5ea640 1 -- 172.21.15.116:0/2902846053 >> 172.21.15.116:0/2902846053 conn(0x7fca680c49c0 msgr2=0x7fca680c4dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:08.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.554+0000 7fca6e5ea640 1 -- 172.21.15.116:0/2902846053 shutdown_connections 2024-04-25T00:01:08.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.554+0000 7fca6e5ea640 1 -- 172.21.15.116:0/2902846053 wait complete. 2024-04-25T00:01:08.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.554+0000 7fca6e5ea640 1 Processor -- start 2024-04-25T00:01:08.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.554+0000 7fca6e5ea640 1 -- start start 2024-04-25T00:01:08.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.554+0000 7fca6e5ea640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fca6806fd90 0x7fca680ad300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:08.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.554+0000 7fca6e5ea640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca680a95c0 0x7fca681427b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:08.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.554+0000 7fca6e5ea640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca680c45c0 0x7fca68142cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:08.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.554+0000 7fca6e5ea640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fca6814db10 con 0x7fca680a95c0 2024-04-25T00:01:08.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.554+0000 7fca6e5ea640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fca6814d990 con 0x7fca680c45c0 2024-04-25T00:01:08.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.554+0000 7fca6e5ea640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fca68160840 con 0x7fca6806fd90 2024-04-25T00:01:08.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.554+0000 7fca677fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca680a95c0 0x7fca681427b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:08.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.554+0000 7fca677fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca680a95c0 0x7fca681427b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37540/0 (socket says 172.21.15.116:37540) 2024-04-25T00:01:08.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.554+0000 7fca677fe640 1 -- 172.21.15.116:0/3791067908 learned_addr learned my addr 172.21.15.116:0/3791067908 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:08.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.554+0000 7fca67fff640 1 --2- 172.21.15.116:0/3791067908 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fca6806fd90 0x7fca680ad300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:08.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.554+0000 7fca677fe640 1 -- 172.21.15.116:0/3791067908 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fca6806fd90 msgr2=0x7fca680ad300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:08.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.554+0000 7fca677fe640 1 --2- 172.21.15.116:0/3791067908 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fca6806fd90 0x7fca680ad300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:08.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.554+0000 7fca677fe640 1 -- 172.21.15.116:0/3791067908 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca680c45c0 msgr2=0x7fca68142cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:08.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.554+0000 7fca677fe640 1 --2- 172.21.15.116:0/3791067908 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca680c45c0 0x7fca68142cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:08.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.554+0000 7fca677fe640 1 -- 172.21.15.116:0/3791067908 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca681487f0 con 0x7fca680a95c0 2024-04-25T00:01:08.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.554+0000 7fca677fe640 1 --2- 172.21.15.116:0/3791067908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca680a95c0 0x7fca681427b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fca5c00c7f0 tx=0x7fca5c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:08.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.554+0000 7fca64ff9640 1 -- 172.21.15.116:0/3791067908 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca5c015070 con 0x7fca680a95c0 2024-04-25T00:01:08.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.554+0000 7fca64ff9640 1 -- 172.21.15.116:0/3791067908 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca5c007820 con 0x7fca680a95c0 2024-04-25T00:01:08.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.554+0000 7fca6e5ea640 1 -- 172.21.15.116:0/3791067908 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca68148ad0 con 0x7fca680a95c0 2024-04-25T00:01:08.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.554+0000 7fca64ff9640 1 -- 172.21.15.116:0/3791067908 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca5c007c10 con 0x7fca680a95c0 2024-04-25T00:01:08.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.554+0000 7fca6e5ea640 1 -- 172.21.15.116:0/3791067908 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca680ad840 con 0x7fca680a95c0 2024-04-25T00:01:08.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.554+0000 7fca6e5ea640 1 -- 172.21.15.116:0/3791067908 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca6804f510 con 0x7fca680a95c0 2024-04-25T00:01:08.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.554+0000 7fca64ff9640 1 -- 172.21.15.116:0/3791067908 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fca5c018070 con 0x7fca680a95c0 2024-04-25T00:01:08.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.558+0000 7fca64ff9640 1 --2- 172.21.15.116:0/3791067908 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fca540678f0 0x7fca54069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:08.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.558+0000 7fca64ff9640 1 -- 172.21.15.116:0/3791067908 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5689+0+0 (secure 0 0 0) 0x7fca5c085d30 con 0x7fca680a95c0 2024-04-25T00:01:08.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.558+0000 7fca67fff640 1 --2- 172.21.15.116:0/3791067908 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fca540678f0 0x7fca54069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:08.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.558+0000 7fca67fff640 1 --2- 172.21.15.116:0/3791067908 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fca540678f0 0x7fca54069db0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fca580af000 tx=0x7fca580a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:08.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.562+0000 7fca64ff9640 1 -- 172.21.15.116:0/3791067908 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fca5c056880 con 0x7fca680a95c0 2024-04-25T00:01:08.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:08.746+0000 7fca6e5ea640 1 -- 172.21.15.116:0/3791067908 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "a", "var": "session_timeout", "val": "300"} v 0) -- 0x7fca68148ca0 con 0x7fca680a95c0 2024-04-25T00:01:09.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.402+0000 7fca64ff9640 1 -- 172.21.15.116:0/3791067908 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "a", "var": "session_timeout", "val": "300"}]=0 v9) ==== 110+0+0 (secure 0 0 0) 0x7fca5c056180 con 0x7fca680a95c0 2024-04-25T00:01:09.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.402+0000 7fca6e5ea640 1 -- 172.21.15.116:0/3791067908 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fca540678f0 msgr2=0x7fca54069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:09.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.402+0000 7fca6e5ea640 1 --2- 172.21.15.116:0/3791067908 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fca540678f0 0x7fca54069db0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fca580af000 tx=0x7fca580a7000 comp rx=0 tx=0).stop 2024-04-25T00:01:09.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.402+0000 7fca6e5ea640 1 -- 172.21.15.116:0/3791067908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca680a95c0 msgr2=0x7fca681427b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:09.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.402+0000 7fca6e5ea640 1 --2- 172.21.15.116:0/3791067908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca680a95c0 0x7fca681427b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fca5c00c7f0 tx=0x7fca5c00ccb0 comp rx=0 tx=0).stop 2024-04-25T00:01:09.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.402+0000 7fca6e5ea640 1 -- 172.21.15.116:0/3791067908 shutdown_connections 2024-04-25T00:01:09.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.402+0000 7fca6e5ea640 1 --2- 172.21.15.116:0/3791067908 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fca540678f0 0x7fca54069db0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:09.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.402+0000 7fca6e5ea640 1 --2- 172.21.15.116:0/3791067908 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca680c45c0 0x7fca68142cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:09.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.402+0000 7fca6e5ea640 1 --2- 172.21.15.116:0/3791067908 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca680a95c0 0x7fca681427b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:09.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.402+0000 7fca6e5ea640 1 --2- 172.21.15.116:0/3791067908 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fca6806fd90 0x7fca680ad300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:09.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.402+0000 7fca6e5ea640 1 -- 172.21.15.116:0/3791067908 >> 172.21.15.116:0/3791067908 conn(0x7fca680c49c0 msgr2=0x7fca68140c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:09.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.402+0000 7fca6e5ea640 1 -- 172.21.15.116:0/3791067908 shutdown_connections 2024-04-25T00:01:09.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.402+0000 7fca6e5ea640 1 -- 172.21.15.116:0/3791067908 wait complete. 2024-04-25T00:01:09.425 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'a' 2024-04-25T00:01:09.425 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:01:09.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.546+0000 7ff305b02640 1 Processor -- start 2024-04-25T00:01:09.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.546+0000 7ff305b02640 1 -- start start 2024-04-25T00:01:09.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.546+0000 7ff305b02640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff3000a94a0 0x7ff3001563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:09.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.546+0000 7ff305b02640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff300071680 0x7ff300071ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:09.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.546+0000 7ff305b02640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff300071220 con 0x7ff30014c720 2024-04-25T00:01:09.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.546+0000 7ff305b02640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7ff300153460 con 0x7ff300071680 2024-04-25T00:01:09.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.546+0000 7ff305b02640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff300152420 con 0x7ff3000a94a0 2024-04-25T00:01:09.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.546+0000 7ff2fffff640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff300071680 0x7ff300071ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:09.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.546+0000 7ff2feffd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff3000a94a0 0x7ff3001563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:09.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.546+0000 7ff2fffff640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff300071680 0x7ff300071ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:45812/0 (socket says 172.21.15.116:45812) 2024-04-25T00:01:09.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.546+0000 7ff2fffff640 1 -- 172.21.15.116:0/4179953449 learned_addr learned my addr 172.21.15.116:0/4179953449 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:09.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.546+0000 7ff2fffff640 1 -- 172.21.15.116:0/4179953449 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff3000a94a0 msgr2=0x7ff3001563d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:09.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.546+0000 7ff2fffff640 1 --2- 172.21.15.116:0/4179953449 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff3000a94a0 0x7ff3001563d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:09.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.546+0000 7ff2fffff640 1 -- 172.21.15.116:0/4179953449 >> v1:172.21.15.174:6789/0 conn(0x7ff30014c720 legacy=0x7ff3000c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:09.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.546+0000 7ff2fffff640 1 -- 172.21.15.116:0/4179953449 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff30014cb00 con 0x7ff300071680 2024-04-25T00:01:09.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.546+0000 7ff2fe7fc640 1 -- 172.21.15.116:0/4179953449 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 496189705 0 0) 0x7ff300071220 con 0x7ff30014c720 2024-04-25T00:01:09.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.546+0000 7ff2feffd640 1 --2- 172.21.15.116:0/4179953449 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff3000a94a0 0x7ff3001563d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:01:09.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.550+0000 7ff2fffff640 1 --2- 172.21.15.116:0/4179953449 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff300071680 0x7ff300071ad0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff2f8009e50 tx=0x7ff2f801c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1cec489b45af66a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:09.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.550+0000 7ff2fe7fc640 1 -- 172.21.15.116:0/4179953449 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2f80a5830 con 0x7ff300071680 2024-04-25T00:01:09.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.550+0000 7ff2fe7fc640 1 -- 172.21.15.116:0/4179953449 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2f80a59d0 con 0x7ff300071680 2024-04-25T00:01:09.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.550+0000 7ff2fe7fc640 1 -- 172.21.15.116:0/4179953449 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2f80a5da0 con 0x7ff300071680 2024-04-25T00:01:09.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.550+0000 7ff305b02640 1 -- 172.21.15.116:0/4179953449 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff300071680 msgr2=0x7ff300071ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:09.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.550+0000 7ff305b02640 1 --2- 172.21.15.116:0/4179953449 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff300071680 0x7ff300071ad0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff2f8009e50 tx=0x7ff2f801c970 comp rx=0 tx=0).stop 2024-04-25T00:01:09.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.550+0000 7ff305b02640 1 -- 172.21.15.116:0/4179953449 shutdown_connections 2024-04-25T00:01:09.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.550+0000 7ff305b02640 1 --2- 172.21.15.116:0/4179953449 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff300071680 0x7ff300071ad0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:09.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.550+0000 7ff305b02640 1 --2- 172.21.15.116:0/4179953449 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff3000a94a0 0x7ff3001563d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:09.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.550+0000 7ff305b02640 1 -- 172.21.15.116:0/4179953449 >> 172.21.15.116:0/4179953449 conn(0x7ff3000c3b00 msgr2=0x7ff3000c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:09.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.550+0000 7ff305b02640 1 -- 172.21.15.116:0/4179953449 shutdown_connections 2024-04-25T00:01:09.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.550+0000 7ff305b02640 1 -- 172.21.15.116:0/4179953449 wait complete. 2024-04-25T00:01:09.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.554+0000 7ff305b02640 1 Processor -- start 2024-04-25T00:01:09.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.554+0000 7ff305b02640 1 -- start start 2024-04-25T00:01:09.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.554+0000 7ff305b02640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff300071680 0x7ff300154dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:09.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.554+0000 7ff305b02640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3000a94a0 0x7ff300145680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:09.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.554+0000 7ff305b02640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff30014c720 0x7ff300145bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:09.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.554+0000 7ff305b02640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff3000c3120 con 0x7ff30014c720 2024-04-25T00:01:09.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.554+0000 7ff305b02640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff300152f40 con 0x7ff3000a94a0 2024-04-25T00:01:09.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.554+0000 7ff305b02640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff3000c32a0 con 0x7ff300071680 2024-04-25T00:01:09.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.554+0000 7ff2fffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff30014c720 0x7ff300145bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:09.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.554+0000 7ff2fffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff30014c720 0x7ff300145bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37552/0 (socket says 172.21.15.116:37552) 2024-04-25T00:01:09.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.554+0000 7ff2fffff640 1 -- 172.21.15.116:0/1126616141 learned_addr learned my addr 172.21.15.116:0/1126616141 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:09.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.554+0000 7ff2feffd640 1 --2- 172.21.15.116:0/1126616141 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3000a94a0 0x7ff300145680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:09.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.554+0000 7ff2ff7fe640 1 --2- 172.21.15.116:0/1126616141 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff300071680 0x7ff300154dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:09.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.554+0000 7ff2fffff640 1 -- 172.21.15.116:0/1126616141 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff300071680 msgr2=0x7ff300154dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:09.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.554+0000 7ff2fffff640 1 --2- 172.21.15.116:0/1126616141 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff300071680 0x7ff300154dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:09.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.554+0000 7ff2fffff640 1 -- 172.21.15.116:0/1126616141 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3000a94a0 msgr2=0x7ff300145680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:09.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.554+0000 7ff2fffff640 1 --2- 172.21.15.116:0/1126616141 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3000a94a0 0x7ff300145680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:09.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.554+0000 7ff2fffff640 1 -- 172.21.15.116:0/1126616141 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3000c6380 con 0x7ff30014c720 2024-04-25T00:01:09.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.554+0000 7ff2fffff640 1 --2- 172.21.15.116:0/1126616141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff30014c720 0x7ff300145bc0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff2f8009e50 tx=0x7ff2f8019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:09.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.554+0000 7ff2e7fff640 1 -- 172.21.15.116:0/1126616141 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2f80bc030 con 0x7ff30014c720 2024-04-25T00:01:09.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.554+0000 7ff2e7fff640 1 -- 172.21.15.116:0/1126616141 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2f80b8470 con 0x7ff30014c720 2024-04-25T00:01:09.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.554+0000 7ff305b02640 1 -- 172.21.15.116:0/1126616141 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3000c6660 con 0x7ff30014c720 2024-04-25T00:01:09.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.554+0000 7ff2e7fff640 1 -- 172.21.15.116:0/1126616141 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2f80b8820 con 0x7ff30014c720 2024-04-25T00:01:09.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.554+0000 7ff305b02640 1 -- 172.21.15.116:0/1126616141 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3000c6c20 con 0x7ff30014c720 2024-04-25T00:01:09.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.554+0000 7ff305b02640 1 -- 172.21.15.116:0/1126616141 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff300068350 con 0x7ff30014c720 2024-04-25T00:01:09.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.554+0000 7ff2e7fff640 1 -- 172.21.15.116:0/1126616141 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff2f80bf070 con 0x7ff30014c720 2024-04-25T00:01:09.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.554+0000 7ff2e7fff640 1 --2- 172.21.15.116:0/1126616141 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff2ec0678c0 0x7ff2ec069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:09.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.558+0000 7ff2e7fff640 1 -- 172.21.15.116:0/1126616141 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5689+0+0 (secure 0 0 0) 0x7ff2f8120830 con 0x7ff30014c720 2024-04-25T00:01:09.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.558+0000 7ff2ff7fe640 1 --2- 172.21.15.116:0/1126616141 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff2ec0678c0 0x7ff2ec069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:09.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.558+0000 7ff2ff7fe640 1 --2- 172.21.15.116:0/1126616141 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff2ec0678c0 0x7ff2ec069d80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff2f4000f70 tx=0x7ff2f4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:09.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.562+0000 7ff2e7fff640 1 -- 172.21.15.116:0/1126616141 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ff2f8016620 con 0x7ff30014c720 2024-04-25T00:01:09.740 INFO:tasks.ceph.mds.b.smithi174.stderr:2024-04-25T00:01:09.734+0000 7f12a8e50640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T00:01:09.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.746+0000 7ff305b02640 1 -- 172.21.15.116:0/1126616141 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff3000c3640 con 0x7ff30014c720 2024-04-25T00:01:09.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.746+0000 7ff2e7fff640 1 -- 172.21.15.116:0/1126616141 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5180 (secure 0 0 0) 0x7ff2f80f14a0 con 0x7ff30014c720 2024-04-25T00:01:09.750 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:01:09.750 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6834/1459177921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1459177921},{"type":"v1","addr":"172.21.15.116:6834","nonce":1459177921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4245,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/174781672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":174781672},{"type":"v1","addr":"172.21.15.116:6835","nonce":174781672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:01:08.748957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6837/1762356055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1762356055},{"type":"v1","addr":"172.21.15.174:6837","nonce":1762356055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270]},"id":1}]} 2024-04-25T00:01:09.750 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 9 2024-04-25T00:01:09.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.750+0000 7ff305b02640 1 -- 172.21.15.116:0/1126616141 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff2ec0678c0 msgr2=0x7ff2ec069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:09.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.750+0000 7ff305b02640 1 --2- 172.21.15.116:0/1126616141 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff2ec0678c0 0x7ff2ec069d80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff2f4000f70 tx=0x7ff2f4009040 comp rx=0 tx=0).stop 2024-04-25T00:01:09.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.750+0000 7ff305b02640 1 -- 172.21.15.116:0/1126616141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff30014c720 msgr2=0x7ff300145bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:09.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.750+0000 7ff305b02640 1 --2- 172.21.15.116:0/1126616141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff30014c720 0x7ff300145bc0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff2f8009e50 tx=0x7ff2f8019710 comp rx=0 tx=0).stop 2024-04-25T00:01:09.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.750+0000 7ff305b02640 1 -- 172.21.15.116:0/1126616141 shutdown_connections 2024-04-25T00:01:09.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.750+0000 7ff305b02640 1 --2- 172.21.15.116:0/1126616141 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff2ec0678c0 0x7ff2ec069d80 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:09.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.750+0000 7ff305b02640 1 --2- 172.21.15.116:0/1126616141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff30014c720 0x7ff300145bc0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:09.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.750+0000 7ff305b02640 1 --2- 172.21.15.116:0/1126616141 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff3000a94a0 0x7ff300145680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:09.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.750+0000 7ff305b02640 1 --2- 172.21.15.116:0/1126616141 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff300071680 0x7ff300154dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:09.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.750+0000 7ff305b02640 1 -- 172.21.15.116:0/1126616141 >> 172.21.15.116:0/1126616141 conn(0x7ff3000c3b00 msgr2=0x7ff30013bdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:09.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.750+0000 7ff305b02640 1 -- 172.21.15.116:0/1126616141 shutdown_connections 2024-04-25T00:01:09.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.750+0000 7ff305b02640 1 -- 172.21.15.116:0/1126616141 wait complete. 2024-04-25T00:01:09.769 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T00:01:09.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.886+0000 7fca52567640 1 Processor -- start 2024-04-25T00:01:09.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.886+0000 7fca52567640 1 -- start start 2024-04-25T00:01:09.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.890+0000 7fca52567640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fca4c0c4760 0x7fca4c14d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:09.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.890+0000 7fca52567640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fca4c14d770 0x7fca4c156bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:09.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.890+0000 7fca52567640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca4c0c2b40 con 0x7fca4c0c5200 2024-04-25T00:01:09.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.890+0000 7fca52567640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fca4c072b80 con 0x7fca4c0c4760 2024-04-25T00:01:09.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.890+0000 7fca52567640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fca4c0c2db0 con 0x7fca4c14d770 2024-04-25T00:01:09.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.890+0000 7fca4b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fca4c0c4760 0x7fca4c14d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:09.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.890+0000 7fca4b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fca4c0c4760 0x7fca4c14d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37556/0 (socket says 172.21.15.116:37556) 2024-04-25T00:01:09.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.890+0000 7fca4b7fe640 1 -- 172.21.15.116:0/453979252 learned_addr learned my addr 172.21.15.116:0/453979252 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:09.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.890+0000 7fca50b04640 1 --2- 172.21.15.116:0/453979252 >> v2:172.21.15.174:3300/0 conn(0x7fca4c14d770 0x7fca4c156bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:09.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.890+0000 7fca4b7fe640 1 -- 172.21.15.116:0/453979252 >> v2:172.21.15.174:3300/0 conn(0x7fca4c14d770 msgr2=0x7fca4c156bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:09.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.890+0000 7fca4b7fe640 1 --2- 172.21.15.116:0/453979252 >> v2:172.21.15.174:3300/0 conn(0x7fca4c14d770 0x7fca4c156bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:09.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.890+0000 7fca4b7fe640 1 -- 172.21.15.116:0/453979252 >> v1:172.21.15.174:6789/0 conn(0x7fca4c0c5200 legacy=0x7fca4c154040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:09.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.890+0000 7fca4b7fe640 1 -- 172.21.15.116:0/453979252 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca4c0c3090 con 0x7fca4c0c4760 2024-04-25T00:01:09.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.890+0000 7fca4b7fe640 1 --2- 172.21.15.116:0/453979252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca4c0c4760 0x7fca4c14d230 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fca4c151190 tx=0x7fca4001ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80d83cf165b3405e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:09.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.890+0000 7fca4affd640 1 -- 172.21.15.116:0/453979252 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca400b2070 con 0x7fca4c0c4760 2024-04-25T00:01:09.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.890+0000 7fca4affd640 1 -- 172.21.15.116:0/453979252 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca400a9070 con 0x7fca4c0c4760 2024-04-25T00:01:09.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.890+0000 7fca4affd640 1 -- 172.21.15.116:0/453979252 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca400ad880 con 0x7fca4c0c4760 2024-04-25T00:01:09.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.894+0000 7fca52567640 1 -- 172.21.15.116:0/453979252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca4c0c4760 msgr2=0x7fca4c14d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:09.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.894+0000 7fca52567640 1 --2- 172.21.15.116:0/453979252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca4c0c4760 0x7fca4c14d230 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fca4c151190 tx=0x7fca4001ceb0 comp rx=0 tx=0).stop 2024-04-25T00:01:09.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.894+0000 7fca52567640 1 -- 172.21.15.116:0/453979252 shutdown_connections 2024-04-25T00:01:09.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.894+0000 7fca52567640 1 --2- 172.21.15.116:0/453979252 >> v2:172.21.15.174:3300/0 conn(0x7fca4c14d770 0x7fca4c156bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:09.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.894+0000 7fca52567640 1 --2- 172.21.15.116:0/453979252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca4c0c4760 0x7fca4c14d230 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:09.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.894+0000 7fca52567640 1 -- 172.21.15.116:0/453979252 >> 172.21.15.116:0/453979252 conn(0x7fca4c06c720 msgr2=0x7fca4c06cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:09.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.894+0000 7fca52567640 1 -- 172.21.15.116:0/453979252 shutdown_connections 2024-04-25T00:01:09.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.894+0000 7fca52567640 1 -- 172.21.15.116:0/453979252 wait complete. 2024-04-25T00:01:09.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.894+0000 7fca52567640 1 Processor -- start 2024-04-25T00:01:09.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.894+0000 7fca52567640 1 -- start start 2024-04-25T00:01:09.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.894+0000 7fca52567640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca4c0c5200 0x7fca4c0c7360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:09.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.894+0000 7fca52567640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca4c14d770 0x7fca4c140640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:09.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.894+0000 7fca52567640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fca4c0c2320 0x7fca4c140b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:09.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.894+0000 7fca52567640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fca4c15ff50 con 0x7fca4c14d770 2024-04-25T00:01:09.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.894+0000 7fca52567640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fca4c0c57f0 con 0x7fca4c0c5200 2024-04-25T00:01:09.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.894+0000 7fca52567640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fca4c1600d0 con 0x7fca4c0c2320 2024-04-25T00:01:09.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.894+0000 7fca4b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca4c14d770 0x7fca4c140640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:09.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.894+0000 7fca4b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca4c14d770 0x7fca4c140640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37564/0 (socket says 172.21.15.116:37564) 2024-04-25T00:01:09.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.894+0000 7fca4b7fe640 1 -- 172.21.15.116:0/331680993 learned_addr learned my addr 172.21.15.116:0/331680993 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:09.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.894+0000 7fca4bfff640 1 --2- 172.21.15.116:0/331680993 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca4c0c5200 0x7fca4c0c7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:09.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.894+0000 7fca50b04640 1 --2- 172.21.15.116:0/331680993 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fca4c0c2320 0x7fca4c140b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:09.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.894+0000 7fca4b7fe640 1 -- 172.21.15.116:0/331680993 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fca4c0c2320 msgr2=0x7fca4c140b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:09.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.894+0000 7fca4b7fe640 1 --2- 172.21.15.116:0/331680993 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fca4c0c2320 0x7fca4c140b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:09.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.894+0000 7fca4b7fe640 1 -- 172.21.15.116:0/331680993 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca4c0c5200 msgr2=0x7fca4c0c7360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:09.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.894+0000 7fca4b7fe640 1 --2- 172.21.15.116:0/331680993 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca4c0c5200 0x7fca4c0c7360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:09.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.894+0000 7fca4b7fe640 1 -- 172.21.15.116:0/331680993 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca4c1410c0 con 0x7fca4c14d770 2024-04-25T00:01:09.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.894+0000 7fca4bfff640 1 --2- 172.21.15.116:0/331680993 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca4c0c5200 0x7fca4c0c7360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:09.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.894+0000 7fca4b7fe640 1 --2- 172.21.15.116:0/331680993 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca4c14d770 0x7fca4c140640 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fca4001ab00 tx=0x7fca400191f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:09.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.894+0000 7fca48ff9640 1 -- 172.21.15.116:0/331680993 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca400bd070 con 0x7fca4c14d770 2024-04-25T00:01:09.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.894+0000 7fca48ff9640 1 -- 172.21.15.116:0/331680993 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca400addb0 con 0x7fca4c14d770 2024-04-25T00:01:09.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.894+0000 7fca52567640 1 -- 172.21.15.116:0/331680993 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca4c142ab0 con 0x7fca4c14d770 2024-04-25T00:01:09.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.894+0000 7fca48ff9640 1 -- 172.21.15.116:0/331680993 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca400b2040 con 0x7fca4c14d770 2024-04-25T00:01:09.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.894+0000 7fca52567640 1 -- 172.21.15.116:0/331680993 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca4c1eff10 con 0x7fca4c14d770 2024-04-25T00:01:09.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.898+0000 7fca52567640 1 -- 172.21.15.116:0/331680993 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca4c1f0bb0 con 0x7fca4c14d770 2024-04-25T00:01:09.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.898+0000 7fca48ff9640 1 -- 172.21.15.116:0/331680993 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fca400c0070 con 0x7fca4c14d770 2024-04-25T00:01:09.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.898+0000 7fca48ff9640 1 --2- 172.21.15.116:0/331680993 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fca380678c0 0x7fca38069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:09.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.898+0000 7fca48ff9640 1 -- 172.21.15.116:0/331680993 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5689+0+0 (secure 0 0 0) 0x7fca4011f350 con 0x7fca4c14d770 2024-04-25T00:01:09.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.898+0000 7fca4bfff640 1 --2- 172.21.15.116:0/331680993 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fca380678c0 0x7fca38069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:09.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.898+0000 7fca4bfff640 1 --2- 172.21.15.116:0/331680993 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fca380678c0 0x7fca38069d80 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fca4c157070 tx=0x7fca3c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:09.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:09.902+0000 7fca48ff9640 1 -- 172.21.15.116:0/331680993 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fca400160d0 con 0x7fca4c14d770 2024-04-25T00:01:10.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.054+0000 7fca52567640 1 -- 172.21.15.116:0/331680993 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fca4c068a60 con 0x7fca4c14d770 2024-04-25T00:01:10.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.058+0000 7fca48ff9640 1 -- 172.21.15.116:0/331680993 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) ==== 74+0+16613 (secure 0 0 0) 0x7fca400eff30 con 0x7fca4c14d770 2024-04-25T00:01:10.059 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:01:10.059 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":23,"fsid":"6d6a1aa7-0478-4c83-8057-05917e81f046","created":"2024-04-25T00:00:40.682493+0000","modified":"2024-04-25T00:01:06.329793+0000","last_up_change":"2024-04-25T00:00:51.822174+0000","last_in_change":"2024-04-25T00:00:42.980473+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T00:00:53.324376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-04-25T00:01:03.197326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-04-25T00:01:03.728618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8b6ddcb2-1ad9-4167-961c-bd6936bb9060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6809","nonce":3751765682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6811","nonce":3751765682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6815","nonce":3751765682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6813","nonce":3751765682}]},"public_addr":"172.21.15.116:6809/3751765682","cluster_addr":"172.21.15.116:6811/3751765682","heartbeat_back_addr":"172.21.15.116:6815/3751765682","heartbeat_front_addr":"172.21.15.116:6813/3751765682","state":["exists","up"]},{"osd":1,"uuid":"a6431768-994f-41da-8853-886f5687fb91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6801","nonce":377879124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6803","nonce":377879124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6807","nonce":377879124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6805","nonce":377879124}]},"public_addr":"172.21.15.116:6801/377879124","cluster_addr":"172.21.15.116:6803/377879124","heartbeat_back_addr":"172.21.15.116:6807/377879124","heartbeat_front_addr":"172.21.15.116:6805/377879124","state":["exists","up"]},{"osd":2,"uuid":"4f69bd40-5a23-45ff-8693-e1de75e1aa9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6817","nonce":1099282267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6819","nonce":1099282267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6823","nonce":1099282267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6821","nonce":1099282267}]},"public_addr":"172.21.15.116:6817/1099282267","cluster_addr":"172.21.15.116:6819/1099282267","heartbeat_back_addr":"172.21.15.116:6823/1099282267","heartbeat_front_addr":"172.21.15.116:6821/1099282267","state":["exists","up"]},{"osd":3,"uuid":"6a0d5344-8b6f-40d3-918d-0f0a4be688e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6825","nonce":1179942661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6827","nonce":1179942661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6831","nonce":1179942661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6829","nonce":1179942661}]},"public_addr":"172.21.15.116:6825/1179942661","cluster_addr":"172.21.15.116:6827/1179942661","heartbeat_back_addr":"172.21.15.116:6831/1179942661","heartbeat_front_addr":"172.21.15.116:6829/1179942661","state":["exists","up"]},{"osd":4,"uuid":"132e92e4-b825-441e-965a-ca00533fa86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6801","nonce":3132444330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6803","nonce":3132444330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6807","nonce":3132444330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6805","nonce":3132444330}]},"public_addr":"172.21.15.174:6801/3132444330","cluster_addr":"172.21.15.174:6803/3132444330","heartbeat_back_addr":"172.21.15.174:6807/3132444330","heartbeat_front_addr":"172.21.15.174:6805/3132444330","state":["exists","up"]},{"osd":5,"uuid":"a2eeb5b6-1b31-43cc-8306-a0adf3ee15bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6809","nonce":881817164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6811","nonce":881817164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6815","nonce":881817164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6813","nonce":881817164}]},"public_addr":"172.21.15.174:6809/881817164","cluster_addr":"172.21.15.174:6811/881817164","heartbeat_back_addr":"172.21.15.174:6815/881817164","heartbeat_front_addr":"172.21.15.174:6813/881817164","state":["exists","up"]},{"osd":6,"uuid":"588e873a-c841-4ba7-90f2-8956c87a9983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6825","nonce":117574251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6827","nonce":117574251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6831","nonce":117574251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6829","nonce":117574251}]},"public_addr":"172.21.15.174:6825/117574251","cluster_addr":"172.21.15.174:6827/117574251","heartbeat_back_addr":"172.21.15.174:6831/117574251","heartbeat_front_addr":"172.21.15.174:6829/117574251","state":["exists","up"]},{"osd":7,"uuid":"3b90f9b0-8a0e-4883-b088-ca1d4012870c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6817","nonce":2543899196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6819","nonce":2543899196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6823","nonce":2543899196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6821","nonce":2543899196}]},"public_addr":"172.21.15.174:6817/2543899196","cluster_addr":"172.21.15.174:6819/2543899196","heartbeat_back_addr":"172.21.15.174:6823/2543899196","heartbeat_front_addr":"172.21.15.174:6821/2543899196","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.293769+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.310034+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.404865+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.549658+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.192174+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.297266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.493649+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.452309+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:01:10.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.058+0000 7fca52567640 1 -- 172.21.15.116:0/331680993 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fca380678c0 msgr2=0x7fca38069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:10.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.058+0000 7fca52567640 1 --2- 172.21.15.116:0/331680993 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fca380678c0 0x7fca38069d80 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fca4c157070 tx=0x7fca3c009040 comp rx=0 tx=0).stop 2024-04-25T00:01:10.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.058+0000 7fca52567640 1 -- 172.21.15.116:0/331680993 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca4c14d770 msgr2=0x7fca4c140640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:10.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.058+0000 7fca52567640 1 --2- 172.21.15.116:0/331680993 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca4c14d770 0x7fca4c140640 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fca4001ab00 tx=0x7fca400191f0 comp rx=0 tx=0).stop 2024-04-25T00:01:10.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.058+0000 7fca52567640 1 -- 172.21.15.116:0/331680993 shutdown_connections 2024-04-25T00:01:10.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.058+0000 7fca52567640 1 --2- 172.21.15.116:0/331680993 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fca380678c0 0x7fca38069d80 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:10.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.058+0000 7fca52567640 1 --2- 172.21.15.116:0/331680993 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fca4c0c2320 0x7fca4c140b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:10.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.058+0000 7fca52567640 1 --2- 172.21.15.116:0/331680993 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fca4c14d770 0x7fca4c140640 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:10.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.058+0000 7fca52567640 1 --2- 172.21.15.116:0/331680993 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fca4c0c5200 0x7fca4c0c7360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:10.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.058+0000 7fca52567640 1 -- 172.21.15.116:0/331680993 >> 172.21.15.116:0/331680993 conn(0x7fca4c06c720 msgr2=0x7fca4c151ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:10.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.058+0000 7fca52567640 1 -- 172.21.15.116:0/331680993 shutdown_connections 2024-04-25T00:01:10.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.058+0000 7fca52567640 1 -- 172.21.15.116:0/331680993 wait complete. 2024-04-25T00:01:10.076 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-25T00:01:10.076 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T00:01:10.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.194+0000 7fbbbbfa5640 1 Processor -- start 2024-04-25T00:01:10.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.194+0000 7fbbbbfa5640 1 -- start start 2024-04-25T00:01:10.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.194+0000 7fbbbbfa5640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbbb4150d20 0x7fbbb40c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:10.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.194+0000 7fbbbbfa5640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbb40c3910 con 0x7fbbb40c2f50 2024-04-25T00:01:10.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.194+0000 7fbbbbfa5640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbb40c16e0 con 0x7fbbb40a94a0 2024-04-25T00:01:10.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.194+0000 7fbbbbfa5640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fbbb40c1400 con 0x7fbbb4150d20 2024-04-25T00:01:10.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.194+0000 7fbbb9d41640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbbb4150d20 0x7fbbb40c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:10.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.198+0000 7fbbb9d41640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbbb4150d20 0x7fbbb40c2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37574/0 (socket says 172.21.15.116:37574) 2024-04-25T00:01:10.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.198+0000 7fbbb9d41640 1 -- 172.21.15.116:0/1270686503 learned_addr learned my addr 172.21.15.116:0/1270686503 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:10.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.198+0000 7fbbb9d41640 1 -- 172.21.15.116:0/1270686503 >> v1:172.21.15.174:6789/0 conn(0x7fbbb40a94a0 legacy=0x7fbbb4153ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:10.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.198+0000 7fbbb9d41640 1 -- 172.21.15.116:0/1270686503 >> v1:172.21.15.116:6789/0 conn(0x7fbbb40c2f50 legacy=0x7fbbb41538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:10.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.198+0000 7fbbb9d41640 1 -- 172.21.15.116:0/1270686503 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbb406e660 con 0x7fbbb4150d20 2024-04-25T00:01:10.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.198+0000 7fbbb9d41640 1 --2- 172.21.15.116:0/1270686503 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbb4150d20 0x7fbbb40c2a10 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbbb0009870 tx=0x7fbbb001c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe0740e9245416b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:10.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.198+0000 7fbbb8d3f640 1 -- 172.21.15.116:0/1270686503 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb00ab5f0 con 0x7fbbb4150d20 2024-04-25T00:01:10.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.198+0000 7fbbb8d3f640 1 -- 172.21.15.116:0/1270686503 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbb00b0070 con 0x7fbbb4150d20 2024-04-25T00:01:10.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.198+0000 7fbbb8d3f640 1 -- 172.21.15.116:0/1270686503 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbb00ab9b0 con 0x7fbbb4150d20 2024-04-25T00:01:10.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.198+0000 7fbbbbfa5640 1 -- 172.21.15.116:0/1270686503 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbb4150d20 msgr2=0x7fbbb40c2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:10.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.198+0000 7fbbbbfa5640 1 --2- 172.21.15.116:0/1270686503 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbb4150d20 0x7fbbb40c2a10 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbbb0009870 tx=0x7fbbb001c4e0 comp rx=0 tx=0).stop 2024-04-25T00:01:10.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.198+0000 7fbbbbfa5640 1 -- 172.21.15.116:0/1270686503 shutdown_connections 2024-04-25T00:01:10.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.198+0000 7fbbbbfa5640 1 --2- 172.21.15.116:0/1270686503 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbb4150d20 0x7fbbb40c2a10 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:10.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.198+0000 7fbbbbfa5640 1 -- 172.21.15.116:0/1270686503 >> 172.21.15.116:0/1270686503 conn(0x7fbbb40c3b00 msgr2=0x7fbbb40c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:10.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.198+0000 7fbbbbfa5640 1 -- 172.21.15.116:0/1270686503 shutdown_connections 2024-04-25T00:01:10.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.198+0000 7fbbbbfa5640 1 -- 172.21.15.116:0/1270686503 wait complete. 2024-04-25T00:01:10.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.202+0000 7fbbbbfa5640 1 Processor -- start 2024-04-25T00:01:10.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.202+0000 7fbbbbfa5640 1 -- start start 2024-04-25T00:01:10.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.202+0000 7fbbbbfa5640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbbb40a94a0 0x7fbbb413f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:10.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.202+0000 7fbbbbfa5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbb40c2f50 0x7fbbb4157030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:10.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.202+0000 7fbbbbfa5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbbb4150d20 0x7fbbb41ede60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:10.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.202+0000 7fbbbbfa5640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbbb4161ad0 con 0x7fbbb40c2f50 2024-04-25T00:01:10.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.202+0000 7fbbbbfa5640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbbb4161950 con 0x7fbbb4150d20 2024-04-25T00:01:10.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.202+0000 7fbbbbfa5640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fbbb4161c50 con 0x7fbbb40a94a0 2024-04-25T00:01:10.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.202+0000 7fbbb9540640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbb40c2f50 0x7fbbb4157030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:10.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.202+0000 7fbbb9540640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbb40c2f50 0x7fbbb4157030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37580/0 (socket says 172.21.15.116:37580) 2024-04-25T00:01:10.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.202+0000 7fbbb9540640 1 -- 172.21.15.116:0/2569542595 learned_addr learned my addr 172.21.15.116:0/2569542595 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:10.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.202+0000 7fbbba542640 1 --2- 172.21.15.116:0/2569542595 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbbb4150d20 0x7fbbb41ede60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:10.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.202+0000 7fbbb9d41640 1 --2- 172.21.15.116:0/2569542595 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbbb40a94a0 0x7fbbb413f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:10.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.202+0000 7fbbb9540640 1 -- 172.21.15.116:0/2569542595 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbbb40a94a0 msgr2=0x7fbbb413f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:10.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.202+0000 7fbbb9540640 1 --2- 172.21.15.116:0/2569542595 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbbb40a94a0 0x7fbbb413f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:10.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.202+0000 7fbbb9540640 1 -- 172.21.15.116:0/2569542595 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbbb4150d20 msgr2=0x7fbbb41ede60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:10.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.202+0000 7fbbb9540640 1 --2- 172.21.15.116:0/2569542595 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbbb4150d20 0x7fbbb41ede60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:10.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.202+0000 7fbbb9540640 1 -- 172.21.15.116:0/2569542595 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbb4147820 con 0x7fbbb40c2f50 2024-04-25T00:01:10.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.202+0000 7fbbb9d41640 1 --2- 172.21.15.116:0/2569542595 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbbb40a94a0 0x7fbbb413f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:10.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.202+0000 7fbbba542640 1 --2- 172.21.15.116:0/2569542595 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbbb4150d20 0x7fbbb41ede60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:10.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.202+0000 7fbbb9540640 1 --2- 172.21.15.116:0/2569542595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbb40c2f50 0x7fbbb4157030 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbba400c480 tx=0x7fbba400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:10.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.202+0000 7fbbaa7fc640 1 -- 172.21.15.116:0/2569542595 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbba4012830 con 0x7fbbb40c2f50 2024-04-25T00:01:10.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.202+0000 7fbbaa7fc640 1 -- 172.21.15.116:0/2569542595 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbba4017070 con 0x7fbbb40c2f50 2024-04-25T00:01:10.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.202+0000 7fbbbbfa5640 1 -- 172.21.15.116:0/2569542595 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbb4155360 con 0x7fbbb40c2f50 2024-04-25T00:01:10.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.202+0000 7fbbaa7fc640 1 -- 172.21.15.116:0/2569542595 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbba4012c30 con 0x7fbbb40c2f50 2024-04-25T00:01:10.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.202+0000 7fbbbbfa5640 1 -- 172.21.15.116:0/2569542595 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbb4155920 con 0x7fbbb40c2f50 2024-04-25T00:01:10.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.202+0000 7fbbbbfa5640 1 -- 172.21.15.116:0/2569542595 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbb4068350 con 0x7fbbb40c2f50 2024-04-25T00:01:10.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.202+0000 7fbbaa7fc640 1 -- 172.21.15.116:0/2569542595 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbba401c460 con 0x7fbbb40c2f50 2024-04-25T00:01:10.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.206+0000 7fbbaa7fc640 1 --2- 172.21.15.116:0/2569542595 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbb980678c0 0x7fbb98069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:10.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.206+0000 7fbbaa7fc640 1 -- 172.21.15.116:0/2569542595 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5689+0+0 (secure 0 0 0) 0x7fbba4085a20 con 0x7fbbb40c2f50 2024-04-25T00:01:10.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.206+0000 7fbbb9d41640 1 --2- 172.21.15.116:0/2569542595 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbb980678c0 0x7fbb98069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:10.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.206+0000 7fbbb9d41640 1 --2- 172.21.15.116:0/2569542595 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbb980678c0 0x7fbb98069d80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbbb00af000 tx=0x7fbbb0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:10.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.210+0000 7fbbaa7fc640 1 -- 172.21.15.116:0/2569542595 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fbba4056600 con 0x7fbbb40c2f50 2024-04-25T00:01:10.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.354+0000 7fbbbbfa5640 1 -- 172.21.15.116:0/2569542595 --> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbbb40c34f0 con 0x7fbb980678c0 2024-04-25T00:01:10.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.374+0000 7fbbaa7fc640 1 -- 172.21.15.116:0/2569542595 <== mgr.4105 v2:172.21.15.174:6832/2004448211 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251355 (secure 0 0 0) 0x7fbbb40c34f0 con 0x7fbb980678c0 2024-04-25T00:01:10.378 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:01:10.379 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-04-25T00:01:10.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.378+0000 7fbbbbfa5640 1 -- 172.21.15.116:0/2569542595 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbb980678c0 msgr2=0x7fbb98069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:10.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.378+0000 7fbbbbfa5640 1 --2- 172.21.15.116:0/2569542595 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbb980678c0 0x7fbb98069d80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbbb00af000 tx=0x7fbbb0008040 comp rx=0 tx=0).stop 2024-04-25T00:01:10.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.378+0000 7fbbbbfa5640 1 -- 172.21.15.116:0/2569542595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbb40c2f50 msgr2=0x7fbbb4157030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:10.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.378+0000 7fbbbbfa5640 1 --2- 172.21.15.116:0/2569542595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbb40c2f50 0x7fbbb4157030 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbba400c480 tx=0x7fbba400c940 comp rx=0 tx=0).stop 2024-04-25T00:01:10.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.378+0000 7fbbbbfa5640 1 -- 172.21.15.116:0/2569542595 shutdown_connections 2024-04-25T00:01:10.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.378+0000 7fbbbbfa5640 1 --2- 172.21.15.116:0/2569542595 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbb980678c0 0x7fbb98069d80 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:10.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.378+0000 7fbbbbfa5640 1 --2- 172.21.15.116:0/2569542595 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbbb4150d20 0x7fbbb41ede60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:10.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.378+0000 7fbbbbfa5640 1 --2- 172.21.15.116:0/2569542595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbbb40c2f50 0x7fbbb4157030 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:10.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.378+0000 7fbbbbfa5640 1 --2- 172.21.15.116:0/2569542595 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbbb40a94a0 0x7fbbb413f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:10.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.378+0000 7fbbbbfa5640 1 -- 172.21.15.116:0/2569542595 >> 172.21.15.116:0/2569542595 conn(0x7fbbb40c3b00 msgr2=0x7fbbb4147030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:10.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.378+0000 7fbbbbfa5640 1 -- 172.21.15.116:0/2569542595 shutdown_connections 2024-04-25T00:01:10.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.378+0000 7fbbbbfa5640 1 -- 172.21.15.116:0/2569542595 wait complete. 2024-04-25T00:01:10.395 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-04-25T00:01:09.251353+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592196,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":184,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141,"ondisk_log_size":141,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":181,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219136,"kb_used_data":2956,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755584,"statfs":{"total":773094113280,"available":772869718016,"internally_reserved":0,"allocated":3026944,"data_stored":1794407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":51,"apply_latency_ms":51,"commit_latency_ns":51000000,"apply_latency_ns":51000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1828,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.935044"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380670+0000","last_change":"2024-04-25T00:01:05.389596+0000","last_active":"2024-04-25T00:01:06.380670+0000","last_peered":"2024-04-25T00:01:06.380670+0000","last_clean":"2024-04-25T00:01:06.380670+0000","last_became_active":"2024-04-25T00:01:05.389127+0000","last_became_peered":"2024-04-25T00:01:05.389127+0000","last_unstale":"2024-04-25T00:01:06.380670+0000","last_undegraded":"2024-04-25T00:01:06.380670+0000","last_fullsized":"2024-04-25T00:01:06.380670+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:09.858617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.340220+0000","last_change":"2024-04-25T00:01:05.368190+0000","last_active":"2024-04-25T00:01:06.340220+0000","last_peered":"2024-04-25T00:01:06.340220+0000","last_clean":"2024-04-25T00:01:06.340220+0000","last_became_active":"2024-04-25T00:01:05.362606+0000","last_became_peered":"2024-04-25T00:01:05.362606+0000","last_unstale":"2024-04-25T00:01:06.340220+0000","last_undegraded":"2024-04-25T00:01:06.340220+0000","last_fullsized":"2024-04-25T00:01:06.340220+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:12:52.846250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.339987+0000","last_change":"2024-04-25T00:01:05.371291+0000","last_active":"2024-04-25T00:01:06.339987+0000","last_peered":"2024-04-25T00:01:06.339987+0000","last_clean":"2024-04-25T00:01:06.339987+0000","last_became_active":"2024-04-25T00:01:05.370522+0000","last_became_peered":"2024-04-25T00:01:05.370522+0000","last_unstale":"2024-04-25T00:01:06.339987+0000","last_undegraded":"2024-04-25T00:01:06.339987+0000","last_fullsized":"2024-04-25T00:01:06.339987+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:11:27.044893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.377499+0000","last_change":"2024-04-25T00:01:05.359716+0000","last_active":"2024-04-25T00:01:06.377499+0000","last_peered":"2024-04-25T00:01:06.377499+0000","last_clean":"2024-04-25T00:01:06.377499+0000","last_became_active":"2024-04-25T00:01:05.358866+0000","last_became_peered":"2024-04-25T00:01:05.358866+0000","last_unstale":"2024-04-25T00:01:06.377499+0000","last_undegraded":"2024-04-25T00:01:06.377499+0000","last_fullsized":"2024-04-25T00:01:06.377499+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:50.489125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380258+0000","last_change":"2024-04-25T00:01:05.386068+0000","last_active":"2024-04-25T00:01:06.380258+0000","last_peered":"2024-04-25T00:01:06.380258+0000","last_clean":"2024-04-25T00:01:06.380258+0000","last_became_active":"2024-04-25T00:01:05.378650+0000","last_became_peered":"2024-04-25T00:01:05.378650+0000","last_unstale":"2024-04-25T00:01:06.380258+0000","last_undegraded":"2024-04-25T00:01:06.380258+0000","last_fullsized":"2024-04-25T00:01:06.380258+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:12:52.846250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.377084+0000","last_change":"2024-04-25T00:01:05.359253+0000","last_active":"2024-04-25T00:01:06.377084+0000","last_peered":"2024-04-25T00:01:06.377084+0000","last_clean":"2024-04-25T00:01:06.377084+0000","last_became_active":"2024-04-25T00:01:05.358477+0000","last_became_peered":"2024-04-25T00:01:05.358477+0000","last_unstale":"2024-04-25T00:01:06.377084+0000","last_undegraded":"2024-04-25T00:01:06.377084+0000","last_fullsized":"2024-04-25T00:01:06.377084+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:22.739214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.379079+0000","last_change":"2024-04-25T00:01:05.374766+0000","last_active":"2024-04-25T00:01:06.379079+0000","last_peered":"2024-04-25T00:01:06.379079+0000","last_clean":"2024-04-25T00:01:06.379079+0000","last_became_active":"2024-04-25T00:01:05.369739+0000","last_became_peered":"2024-04-25T00:01:05.369739+0000","last_unstale":"2024-04-25T00:01:06.379079+0000","last_undegraded":"2024-04-25T00:01:06.379079+0000","last_fullsized":"2024-04-25T00:01:06.379079+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:09:18.315688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.339014+0000","last_change":"2024-04-25T00:01:05.390256+0000","last_active":"2024-04-25T00:01:06.339014+0000","last_peered":"2024-04-25T00:01:06.339014+0000","last_clean":"2024-04-25T00:01:06.339014+0000","last_became_active":"2024-04-25T00:01:05.389724+0000","last_became_peered":"2024-04-25T00:01:05.389724+0000","last_unstale":"2024-04-25T00:01:06.339014+0000","last_undegraded":"2024-04-25T00:01:06.339014+0000","last_fullsized":"2024-04-25T00:01:06.339014+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:09:18.315688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.379000+0000","last_change":"2024-04-25T00:01:05.362303+0000","last_active":"2024-04-25T00:01:06.379000+0000","last_peered":"2024-04-25T00:01:06.379000+0000","last_clean":"2024-04-25T00:01:06.379000+0000","last_became_active":"2024-04-25T00:01:05.361586+0000","last_became_peered":"2024-04-25T00:01:05.361586+0000","last_unstale":"2024-04-25T00:01:06.379000+0000","last_undegraded":"2024-04-25T00:01:06.379000+0000","last_fullsized":"2024-04-25T00:01:06.379000+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:30.228060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380324+0000","last_change":"2024-04-25T00:01:05.368974+0000","last_active":"2024-04-25T00:01:06.380324+0000","last_peered":"2024-04-25T00:01:06.380324+0000","last_clean":"2024-04-25T00:01:06.380324+0000","last_became_active":"2024-04-25T00:01:05.367893+0000","last_became_peered":"2024-04-25T00:01:05.367893+0000","last_unstale":"2024-04-25T00:01:06.380324+0000","last_undegraded":"2024-04-25T00:01:06.380324+0000","last_fullsized":"2024-04-25T00:01:06.380324+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:30:23.557381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380371+0000","last_change":"2024-04-25T00:01:05.389424+0000","last_active":"2024-04-25T00:01:06.380371+0000","last_peered":"2024-04-25T00:01:06.380371+0000","last_clean":"2024-04-25T00:01:06.380371+0000","last_became_active":"2024-04-25T00:01:05.388999+0000","last_became_peered":"2024-04-25T00:01:05.388999+0000","last_unstale":"2024-04-25T00:01:06.380371+0000","last_undegraded":"2024-04-25T00:01:06.380371+0000","last_fullsized":"2024-04-25T00:01:06.380371+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:31.456792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.379890+0000","last_change":"2024-04-25T00:01:05.385853+0000","last_active":"2024-04-25T00:01:06.379890+0000","last_peered":"2024-04-25T00:01:06.379890+0000","last_clean":"2024-04-25T00:01:06.379890+0000","last_became_active":"2024-04-25T00:01:05.384997+0000","last_became_peered":"2024-04-25T00:01:05.384997+0000","last_unstale":"2024-04-25T00:01:06.379890+0000","last_undegraded":"2024-04-25T00:01:06.379890+0000","last_fullsized":"2024-04-25T00:01:06.379890+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:36:55.436661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.339127+0000","last_change":"2024-04-25T00:01:05.381011+0000","last_active":"2024-04-25T00:01:06.339127+0000","last_peered":"2024-04-25T00:01:06.339127+0000","last_clean":"2024-04-25T00:01:06.339127+0000","last_became_active":"2024-04-25T00:01:05.380236+0000","last_became_peered":"2024-04-25T00:01:05.380236+0000","last_unstale":"2024-04-25T00:01:06.339127+0000","last_undegraded":"2024-04-25T00:01:06.339127+0000","last_fullsized":"2024-04-25T00:01:06.339127+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:06:19.307378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.337813+0000","last_change":"2024-04-25T00:01:05.378373+0000","last_active":"2024-04-25T00:01:06.337813+0000","last_peered":"2024-04-25T00:01:06.337813+0000","last_clean":"2024-04-25T00:01:06.337813+0000","last_became_active":"2024-04-25T00:01:05.376570+0000","last_became_peered":"2024-04-25T00:01:05.376570+0000","last_unstale":"2024-04-25T00:01:06.337813+0000","last_undegraded":"2024-04-25T00:01:06.337813+0000","last_fullsized":"2024-04-25T00:01:06.337813+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:30.228060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.337758+0000","last_change":"2024-04-25T00:01:05.390341+0000","last_active":"2024-04-25T00:01:06.337758+0000","last_peered":"2024-04-25T00:01:06.337758+0000","last_clean":"2024-04-25T00:01:06.337758+0000","last_became_active":"2024-04-25T00:01:05.389707+0000","last_became_peered":"2024-04-25T00:01:05.389707+0000","last_unstale":"2024-04-25T00:01:06.337758+0000","last_undegraded":"2024-04-25T00:01:06.337758+0000","last_fullsized":"2024-04-25T00:01:06.337758+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:54:21.138947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.335984+0000","last_change":"2024-04-25T00:01:05.370826+0000","last_active":"2024-04-25T00:01:06.335984+0000","last_peered":"2024-04-25T00:01:06.335984+0000","last_clean":"2024-04-25T00:01:06.335984+0000","last_became_active":"2024-04-25T00:01:05.369848+0000","last_became_peered":"2024-04-25T00:01:05.369848+0000","last_unstale":"2024-04-25T00:01:06.335984+0000","last_undegraded":"2024-04-25T00:01:06.335984+0000","last_fullsized":"2024-04-25T00:01:06.335984+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:23.870121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.337683+0000","last_change":"2024-04-25T00:01:05.379958+0000","last_active":"2024-04-25T00:01:06.337683+0000","last_peered":"2024-04-25T00:01:06.337683+0000","last_clean":"2024-04-25T00:01:06.337683+0000","last_became_active":"2024-04-25T00:01:05.379090+0000","last_became_peered":"2024-04-25T00:01:05.379090+0000","last_unstale":"2024-04-25T00:01:06.337683+0000","last_undegraded":"2024-04-25T00:01:06.337683+0000","last_fullsized":"2024-04-25T00:01:06.337683+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:59:50.832958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.379255+0000","last_change":"2024-04-25T00:01:05.367001+0000","last_active":"2024-04-25T00:01:06.379255+0000","last_peered":"2024-04-25T00:01:06.379255+0000","last_clean":"2024-04-25T00:01:06.379255+0000","last_became_active":"2024-04-25T00:01:05.366264+0000","last_became_peered":"2024-04-25T00:01:05.366264+0000","last_unstale":"2024-04-25T00:01:06.379255+0000","last_undegraded":"2024-04-25T00:01:06.379255+0000","last_fullsized":"2024-04-25T00:01:06.379255+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.337778+0000","last_change":"2024-04-25T00:01:05.370653+0000","last_active":"2024-04-25T00:01:06.337778+0000","last_peered":"2024-04-25T00:01:06.337778+0000","last_clean":"2024-04-25T00:01:06.337778+0000","last_became_active":"2024-04-25T00:01:05.369919+0000","last_became_peered":"2024-04-25T00:01:05.369919+0000","last_unstale":"2024-04-25T00:01:06.337778+0000","last_undegraded":"2024-04-25T00:01:06.337778+0000","last_fullsized":"2024-04-25T00:01:06.337778+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:30:08.311326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380032+0000","last_change":"2024-04-25T00:01:05.385253+0000","last_active":"2024-04-25T00:01:06.380032+0000","last_peered":"2024-04-25T00:01:06.380032+0000","last_clean":"2024-04-25T00:01:06.380032+0000","last_became_active":"2024-04-25T00:01:05.379429+0000","last_became_peered":"2024-04-25T00:01:05.379429+0000","last_unstale":"2024-04-25T00:01:06.380032+0000","last_undegraded":"2024-04-25T00:01:06.380032+0000","last_fullsized":"2024-04-25T00:01:06.380032+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:36:10.377757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":1,"reported_epoch":21,"state":"creating+peering","last_fresh":"2024-04-25T00:01:04.338341+0000","last_change":"2024-04-25T00:01:04.338341+0000","last_active":"2024-04-25T00:01:04.314940+0000","last_peered":"2024-04-25T00:01:04.314940+0000","last_clean":"2024-04-25T00:01:04.314940+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T00:01:04.338341+0000","last_undegraded":"2024-04-25T00:01:04.338341+0000","last_fullsized":"2024-04-25T00:01:04.338341+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378876+0000","last_change":"2024-04-25T00:01:05.387457+0000","last_active":"2024-04-25T00:01:06.378876+0000","last_peered":"2024-04-25T00:01:06.378876+0000","last_clean":"2024-04-25T00:01:06.378876+0000","last_became_active":"2024-04-25T00:01:05.386919+0000","last_became_peered":"2024-04-25T00:01:05.386919+0000","last_unstale":"2024-04-25T00:01:06.378876+0000","last_undegraded":"2024-04-25T00:01:06.378876+0000","last_fullsized":"2024-04-25T00:01:06.378876+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.338263+0000","last_change":"2024-04-25T00:01:05.366340+0000","last_active":"2024-04-25T00:01:06.338263+0000","last_peered":"2024-04-25T00:01:06.338263+0000","last_clean":"2024-04-25T00:01:06.338263+0000","last_became_active":"2024-04-25T00:01:05.361952+0000","last_became_peered":"2024-04-25T00:01:05.361952+0000","last_unstale":"2024-04-25T00:01:06.338263+0000","last_undegraded":"2024-04-25T00:01:06.338263+0000","last_fullsized":"2024-04-25T00:01:06.338263+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:41.948662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.335946+0000","last_change":"2024-04-25T00:01:05.372906+0000","last_active":"2024-04-25T00:01:06.335946+0000","last_peered":"2024-04-25T00:01:06.335946+0000","last_clean":"2024-04-25T00:01:06.335946+0000","last_became_active":"2024-04-25T00:01:05.372148+0000","last_became_peered":"2024-04-25T00:01:05.372148+0000","last_unstale":"2024-04-25T00:01:06.335946+0000","last_undegraded":"2024-04-25T00:01:06.335946+0000","last_fullsized":"2024-04-25T00:01:06.335946+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:43.615895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.340447+0000","last_change":"2024-04-25T00:01:05.373464+0000","last_active":"2024-04-25T00:01:06.340447+0000","last_peered":"2024-04-25T00:01:06.340447+0000","last_clean":"2024-04-25T00:01:06.340447+0000","last_became_active":"2024-04-25T00:01:05.372611+0000","last_became_peered":"2024-04-25T00:01:05.372611+0000","last_unstale":"2024-04-25T00:01:06.340447+0000","last_undegraded":"2024-04-25T00:01:06.340447+0000","last_fullsized":"2024-04-25T00:01:06.340447+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:31.456792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378208+0000","last_change":"2024-04-25T00:01:05.365761+0000","last_active":"2024-04-25T00:01:06.378208+0000","last_peered":"2024-04-25T00:01:06.378208+0000","last_clean":"2024-04-25T00:01:06.378208+0000","last_became_active":"2024-04-25T00:01:05.361119+0000","last_became_peered":"2024-04-25T00:01:05.361119+0000","last_unstale":"2024-04-25T00:01:06.378208+0000","last_undegraded":"2024-04-25T00:01:06.378208+0000","last_fullsized":"2024-04-25T00:01:06.378208+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.341008+0000","last_change":"2024-04-25T00:01:05.367296+0000","last_active":"2024-04-25T00:01:06.341008+0000","last_peered":"2024-04-25T00:01:06.341008+0000","last_clean":"2024-04-25T00:01:06.341008+0000","last_became_active":"2024-04-25T00:01:05.363838+0000","last_became_peered":"2024-04-25T00:01:05.363838+0000","last_unstale":"2024-04-25T00:01:06.341008+0000","last_undegraded":"2024-04-25T00:01:06.341008+0000","last_fullsized":"2024-04-25T00:01:06.341008+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:43:07.053074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.339622+0000","last_change":"2024-04-25T00:01:05.387057+0000","last_active":"2024-04-25T00:01:06.339622+0000","last_peered":"2024-04-25T00:01:06.339622+0000","last_clean":"2024-04-25T00:01:06.339622+0000","last_became_active":"2024-04-25T00:01:05.386724+0000","last_became_peered":"2024-04-25T00:01:05.386724+0000","last_unstale":"2024-04-25T00:01:06.339622+0000","last_undegraded":"2024-04-25T00:01:06.339622+0000","last_fullsized":"2024-04-25T00:01:06.339622+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:55:09.073286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.339053+0000","last_change":"2024-04-25T00:01:05.358690+0000","last_active":"2024-04-25T00:01:06.339053+0000","last_peered":"2024-04-25T00:01:06.339053+0000","last_clean":"2024-04-25T00:01:06.339053+0000","last_became_active":"2024-04-25T00:01:05.357340+0000","last_became_peered":"2024-04-25T00:01:05.357340+0000","last_unstale":"2024-04-25T00:01:06.339053+0000","last_undegraded":"2024-04-25T00:01:06.339053+0000","last_fullsized":"2024-04-25T00:01:06.339053+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:36:55.436661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.336324+0000","last_change":"2024-04-25T00:01:05.362960+0000","last_active":"2024-04-25T00:01:06.336324+0000","last_peered":"2024-04-25T00:01:06.336324+0000","last_clean":"2024-04-25T00:01:06.336324+0000","last_became_active":"2024-04-25T00:01:05.361067+0000","last_became_peered":"2024-04-25T00:01:05.361067+0000","last_unstale":"2024-04-25T00:01:06.336324+0000","last_undegraded":"2024-04-25T00:01:06.336324+0000","last_fullsized":"2024-04-25T00:01:06.336324+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:24.017263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.377980+0000","last_change":"2024-04-25T00:01:05.372927+0000","last_active":"2024-04-25T00:01:06.377980+0000","last_peered":"2024-04-25T00:01:06.377980+0000","last_clean":"2024-04-25T00:01:06.377980+0000","last_became_active":"2024-04-25T00:01:05.372050+0000","last_became_peered":"2024-04-25T00:01:05.372050+0000","last_unstale":"2024-04-25T00:01:06.377980+0000","last_undegraded":"2024-04-25T00:01:06.377980+0000","last_fullsized":"2024-04-25T00:01:06.377980+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:48.202251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.376574+0000","last_change":"2024-04-25T00:01:05.378517+0000","last_active":"2024-04-25T00:01:06.376574+0000","last_peered":"2024-04-25T00:01:06.376574+0000","last_clean":"2024-04-25T00:01:06.376574+0000","last_became_active":"2024-04-25T00:01:05.377742+0000","last_became_peered":"2024-04-25T00:01:05.377742+0000","last_unstale":"2024-04-25T00:01:06.376574+0000","last_undegraded":"2024-04-25T00:01:06.376574+0000","last_fullsized":"2024-04-25T00:01:06.376574+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:06.450369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.379065+0000","last_change":"2024-04-25T00:01:05.375142+0000","last_active":"2024-04-25T00:01:06.379065+0000","last_peered":"2024-04-25T00:01:06.379065+0000","last_clean":"2024-04-25T00:01:06.379065+0000","last_became_active":"2024-04-25T00:01:05.373857+0000","last_became_peered":"2024-04-25T00:01:05.373857+0000","last_unstale":"2024-04-25T00:01:06.379065+0000","last_undegraded":"2024-04-25T00:01:06.379065+0000","last_fullsized":"2024-04-25T00:01:06.379065+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:15.879673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"22'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.339475+0000","last_change":"2024-04-25T00:01:04.365607+0000","last_active":"2024-04-25T00:01:06.339475+0000","last_peered":"2024-04-25T00:01:06.339475+0000","last_clean":"2024-04-25T00:01:06.339475+0000","last_became_active":"2024-04-25T00:01:04.364788+0000","last_became_peered":"2024-04-25T00:01:04.364788+0000","last_unstale":"2024-04-25T00:01:06.339475+0000","last_undegraded":"2024-04-25T00:01:06.339475+0000","last_fullsized":"2024-04-25T00:01:06.339475+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":1,"reported_epoch":21,"state":"creating+peering","last_fresh":"2024-04-25T00:01:04.333513+0000","last_change":"2024-04-25T00:01:04.333513+0000","last_active":"2024-04-25T00:01:04.314940+0000","last_peered":"2024-04-25T00:01:04.314940+0000","last_clean":"2024-04-25T00:01:04.314940+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T00:01:04.333513+0000","last_undegraded":"2024-04-25T00:01:04.333513+0000","last_fullsized":"2024-04-25T00:01:04.333513+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"22'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.336967+0000","last_change":"2024-04-25T00:01:04.360254+0000","last_active":"2024-04-25T00:01:06.336967+0000","last_peered":"2024-04-25T00:01:06.336967+0000","last_clean":"2024-04-25T00:01:06.336967+0000","last_became_active":"2024-04-25T00:01:04.359586+0000","last_became_peered":"2024-04-25T00:01:04.359586+0000","last_unstale":"2024-04-25T00:01:06.336967+0000","last_undegraded":"2024-04-25T00:01:06.336967+0000","last_fullsized":"2024-04-25T00:01:06.336967+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:47.192930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.337866+0000","last_change":"2024-04-25T00:01:04.370896+0000","last_active":"2024-04-25T00:01:06.337866+0000","last_peered":"2024-04-25T00:01:06.337866+0000","last_clean":"2024-04-25T00:01:06.337866+0000","last_became_active":"2024-04-25T00:01:04.370273+0000","last_became_peered":"2024-04-25T00:01:04.370273+0000","last_unstale":"2024-04-25T00:01:06.337866+0000","last_undegraded":"2024-04-25T00:01:06.337866+0000","last_fullsized":"2024-04-25T00:01:06.337866+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:49:22.673421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.339542+0000","last_change":"2024-04-25T00:01:05.378867+0000","last_active":"2024-04-25T00:01:06.339542+0000","last_peered":"2024-04-25T00:01:06.339542+0000","last_clean":"2024-04-25T00:01:06.339542+0000","last_became_active":"2024-04-25T00:01:05.378056+0000","last_became_peered":"2024-04-25T00:01:05.378056+0000","last_unstale":"2024-04-25T00:01:06.339542+0000","last_undegraded":"2024-04-25T00:01:06.339542+0000","last_fullsized":"2024-04-25T00:01:06.339542+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:36:10.377757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.336193+0000","last_change":"2024-04-25T00:01:04.364523+0000","last_active":"2024-04-25T00:01:06.336193+0000","last_peered":"2024-04-25T00:01:06.336193+0000","last_clean":"2024-04-25T00:01:06.336193+0000","last_became_active":"2024-04-25T00:01:04.364050+0000","last_became_peered":"2024-04-25T00:01:04.364050+0000","last_unstale":"2024-04-25T00:01:06.336193+0000","last_undegraded":"2024-04-25T00:01:06.336193+0000","last_fullsized":"2024-04-25T00:01:06.336193+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:49.479804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":1,"reported_epoch":21,"state":"creating+peering","last_fresh":"2024-04-25T00:01:04.336083+0000","last_change":"2024-04-25T00:01:04.336083+0000","last_active":"2024-04-25T00:01:04.314940+0000","last_peered":"2024-04-25T00:01:04.314940+0000","last_clean":"2024-04-25T00:01:04.314940+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T00:01:04.336083+0000","last_undegraded":"2024-04-25T00:01:04.336083+0000","last_fullsized":"2024-04-25T00:01:04.336083+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.379231+0000","last_change":"2024-04-25T00:01:04.360804+0000","last_active":"2024-04-25T00:01:06.379231+0000","last_peered":"2024-04-25T00:01:06.379231+0000","last_clean":"2024-04-25T00:01:06.379231+0000","last_became_active":"2024-04-25T00:01:04.360374+0000","last_became_peered":"2024-04-25T00:01:04.360374+0000","last_unstale":"2024-04-25T00:01:06.379231+0000","last_undegraded":"2024-04-25T00:01:06.379231+0000","last_fullsized":"2024-04-25T00:01:06.379231+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378559+0000","last_change":"2024-04-25T00:01:05.365413+0000","last_active":"2024-04-25T00:01:06.378559+0000","last_peered":"2024-04-25T00:01:06.378559+0000","last_clean":"2024-04-25T00:01:06.378559+0000","last_became_active":"2024-04-25T00:01:05.361865+0000","last_became_peered":"2024-04-25T00:01:05.361865+0000","last_unstale":"2024-04-25T00:01:06.378559+0000","last_undegraded":"2024-04-25T00:01:06.378559+0000","last_fullsized":"2024-04-25T00:01:06.378559+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:30.228060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380688+0000","last_change":"2024-04-25T00:01:04.362129+0000","last_active":"2024-04-25T00:01:06.380688+0000","last_peered":"2024-04-25T00:01:06.380688+0000","last_clean":"2024-04-25T00:01:06.380688+0000","last_became_active":"2024-04-25T00:01:04.361827+0000","last_became_peered":"2024-04-25T00:01:04.361827+0000","last_unstale":"2024-04-25T00:01:06.380688+0000","last_undegraded":"2024-04-25T00:01:06.380688+0000","last_fullsized":"2024-04-25T00:01:06.380688+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:26.243635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.340801+0000","last_change":"2024-04-25T00:01:05.361723+0000","last_active":"2024-04-25T00:01:06.340801+0000","last_peered":"2024-04-25T00:01:06.340801+0000","last_clean":"2024-04-25T00:01:06.340801+0000","last_became_active":"2024-04-25T00:01:05.360956+0000","last_became_peered":"2024-04-25T00:01:05.360956+0000","last_unstale":"2024-04-25T00:01:06.340801+0000","last_undegraded":"2024-04-25T00:01:06.340801+0000","last_fullsized":"2024-04-25T00:01:06.340801+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:23.870121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"22'3","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.379142+0000","last_change":"2024-04-25T00:01:04.366503+0000","last_active":"2024-04-25T00:01:06.379142+0000","last_peered":"2024-04-25T00:01:06.379142+0000","last_clean":"2024-04-25T00:01:06.379142+0000","last_became_active":"2024-04-25T00:01:04.365945+0000","last_became_peered":"2024-04-25T00:01:04.365945+0000","last_unstale":"2024-04-25T00:01:06.379142+0000","last_undegraded":"2024-04-25T00:01:06.379142+0000","last_fullsized":"2024-04-25T00:01:06.379142+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.336525+0000","last_change":"2024-04-25T00:01:05.371061+0000","last_active":"2024-04-25T00:01:06.336525+0000","last_peered":"2024-04-25T00:01:06.336525+0000","last_clean":"2024-04-25T00:01:06.336525+0000","last_became_active":"2024-04-25T00:01:05.369950+0000","last_became_peered":"2024-04-25T00:01:05.369950+0000","last_unstale":"2024-04-25T00:01:06.336525+0000","last_undegraded":"2024-04-25T00:01:06.336525+0000","last_fullsized":"2024-04-25T00:01:06.336525+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:02:43.237902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"22'6","reported_seq":20,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.377439+0000","last_change":"2024-04-25T00:01:04.365185+0000","last_active":"2024-04-25T00:01:06.377439+0000","last_peered":"2024-04-25T00:01:06.377439+0000","last_clean":"2024-04-25T00:01:06.377439+0000","last_became_active":"2024-04-25T00:01:04.364930+0000","last_became_peered":"2024-04-25T00:01:04.364930+0000","last_unstale":"2024-04-25T00:01:06.377439+0000","last_undegraded":"2024-04-25T00:01:06.377439+0000","last_fullsized":"2024-04-25T00:01:06.377439+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:45:03.618340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380617+0000","last_change":"2024-04-25T00:01:05.385827+0000","last_active":"2024-04-25T00:01:06.380617+0000","last_peered":"2024-04-25T00:01:06.380617+0000","last_clean":"2024-04-25T00:01:06.380617+0000","last_became_active":"2024-04-25T00:01:05.385258+0000","last_became_peered":"2024-04-25T00:01:05.385258+0000","last_unstale":"2024-04-25T00:01:06.380617+0000","last_undegraded":"2024-04-25T00:01:06.380617+0000","last_fullsized":"2024-04-25T00:01:06.380617+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:23.870121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"22'2","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378778+0000","last_change":"2024-04-25T00:01:04.370640+0000","last_active":"2024-04-25T00:01:06.378778+0000","last_peered":"2024-04-25T00:01:06.378778+0000","last_clean":"2024-04-25T00:01:06.378778+0000","last_became_active":"2024-04-25T00:01:04.370031+0000","last_became_peered":"2024-04-25T00:01:04.370031+0000","last_unstale":"2024-04-25T00:01:06.378778+0000","last_undegraded":"2024-04-25T00:01:06.378778+0000","last_fullsized":"2024-04-25T00:01:06.378778+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:26.243635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.338234+0000","last_change":"2024-04-25T00:01:05.379102+0000","last_active":"2024-04-25T00:01:06.338234+0000","last_peered":"2024-04-25T00:01:06.338234+0000","last_clean":"2024-04-25T00:01:06.338234+0000","last_became_active":"2024-04-25T00:01:05.377625+0000","last_became_peered":"2024-04-25T00:01:05.377625+0000","last_unstale":"2024-04-25T00:01:06.338234+0000","last_undegraded":"2024-04-25T00:01:06.338234+0000","last_fullsized":"2024-04-25T00:01:06.338234+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:15.879673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.337539+0000","last_change":"2024-04-25T00:01:04.360987+0000","last_active":"2024-04-25T00:01:06.337539+0000","last_peered":"2024-04-25T00:01:06.337539+0000","last_clean":"2024-04-25T00:01:06.337539+0000","last_became_active":"2024-04-25T00:01:04.360664+0000","last_became_peered":"2024-04-25T00:01:04.360664+0000","last_unstale":"2024-04-25T00:01:06.337539+0000","last_undegraded":"2024-04-25T00:01:06.337539+0000","last_fullsized":"2024-04-25T00:01:06.337539+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:55.283076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.336555+0000","last_change":"2024-04-25T00:01:05.379501+0000","last_active":"2024-04-25T00:01:06.336555+0000","last_peered":"2024-04-25T00:01:06.336555+0000","last_clean":"2024-04-25T00:01:06.336555+0000","last_became_active":"2024-04-25T00:01:05.378644+0000","last_became_peered":"2024-04-25T00:01:05.378644+0000","last_unstale":"2024-04-25T00:01:06.336555+0000","last_undegraded":"2024-04-25T00:01:06.336555+0000","last_fullsized":"2024-04-25T00:01:06.336555+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:08:35.505154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.379209+0000","last_change":"2024-04-25T00:01:04.357265+0000","last_active":"2024-04-25T00:01:06.379209+0000","last_peered":"2024-04-25T00:01:06.379209+0000","last_clean":"2024-04-25T00:01:06.379209+0000","last_became_active":"2024-04-25T00:01:04.357004+0000","last_became_peered":"2024-04-25T00:01:04.357004+0000","last_unstale":"2024-04-25T00:01:06.379209+0000","last_undegraded":"2024-04-25T00:01:06.379209+0000","last_fullsized":"2024-04-25T00:01:06.379209+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:47:25.595275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378119+0000","last_change":"2024-04-25T00:01:05.373927+0000","last_active":"2024-04-25T00:01:06.378119+0000","last_peered":"2024-04-25T00:01:06.378119+0000","last_clean":"2024-04-25T00:01:06.378119+0000","last_became_active":"2024-04-25T00:01:05.373077+0000","last_became_peered":"2024-04-25T00:01:05.373077+0000","last_unstale":"2024-04-25T00:01:06.378119+0000","last_undegraded":"2024-04-25T00:01:06.378119+0000","last_fullsized":"2024-04-25T00:01:06.378119+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:50.489125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-25T00:01:04.363964+0000","last_change":"2024-04-25T00:01:04.358365+0000","last_active":"2024-04-25T00:01:04.363964+0000","last_peered":"2024-04-25T00:01:04.363964+0000","last_clean":"2024-04-25T00:01:04.363964+0000","last_became_active":"2024-04-25T00:01:04.357571+0000","last_became_peered":"2024-04-25T00:01:04.357571+0000","last_unstale":"2024-04-25T00:01:04.363964+0000","last_undegraded":"2024-04-25T00:01:04.363964+0000","last_fullsized":"2024-04-25T00:01:04.363964+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:11:06.373584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.335878+0000","last_change":"2024-04-25T00:01:05.364103+0000","last_active":"2024-04-25T00:01:06.335878+0000","last_peered":"2024-04-25T00:01:06.335878+0000","last_clean":"2024-04-25T00:01:06.335878+0000","last_became_active":"2024-04-25T00:01:05.363320+0000","last_became_peered":"2024-04-25T00:01:05.363320+0000","last_unstale":"2024-04-25T00:01:06.335878+0000","last_undegraded":"2024-04-25T00:01:06.335878+0000","last_fullsized":"2024-04-25T00:01:06.335878+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:31.456792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"22'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380600+0000","last_change":"2024-04-25T00:01:04.372376+0000","last_active":"2024-04-25T00:01:06.380600+0000","last_peered":"2024-04-25T00:01:06.380600+0000","last_clean":"2024-04-25T00:01:06.380600+0000","last_became_active":"2024-04-25T00:01:04.372040+0000","last_became_peered":"2024-04-25T00:01:04.372040+0000","last_unstale":"2024-04-25T00:01:06.380600+0000","last_undegraded":"2024-04-25T00:01:06.380600+0000","last_fullsized":"2024-04-25T00:01:06.380600+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:21.729893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378915+0000","last_change":"2024-04-25T00:01:05.372591+0000","last_active":"2024-04-25T00:01:06.378915+0000","last_peered":"2024-04-25T00:01:06.378915+0000","last_clean":"2024-04-25T00:01:06.378915+0000","last_became_active":"2024-04-25T00:01:05.371204+0000","last_became_peered":"2024-04-25T00:01:05.371204+0000","last_unstale":"2024-04-25T00:01:06.378915+0000","last_undegraded":"2024-04-25T00:01:06.378915+0000","last_fullsized":"2024-04-25T00:01:06.378915+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:15.879673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-25T00:01:04.373291+0000","last_change":"2024-04-25T00:01:04.370794+0000","last_active":"2024-04-25T00:01:04.373291+0000","last_peered":"2024-04-25T00:01:04.373291+0000","last_clean":"2024-04-25T00:01:04.373291+0000","last_became_active":"2024-04-25T00:01:04.370099+0000","last_became_peered":"2024-04-25T00:01:04.370099+0000","last_unstale":"2024-04-25T00:01:04.373291+0000","last_undegraded":"2024-04-25T00:01:04.373291+0000","last_fullsized":"2024-04-25T00:01:04.373291+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:27:27.293170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380496+0000","last_change":"2024-04-25T00:01:05.378840+0000","last_active":"2024-04-25T00:01:06.380496+0000","last_peered":"2024-04-25T00:01:06.380496+0000","last_clean":"2024-04-25T00:01:06.380496+0000","last_became_active":"2024-04-25T00:01:05.378112+0000","last_became_peered":"2024-04-25T00:01:05.378112+0000","last_unstale":"2024-04-25T00:01:06.380496+0000","last_undegraded":"2024-04-25T00:01:06.380496+0000","last_fullsized":"2024-04-25T00:01:06.380496+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:43.615895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"22'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.340664+0000","last_change":"2024-04-25T00:01:04.356627+0000","last_active":"2024-04-25T00:01:06.340664+0000","last_peered":"2024-04-25T00:01:06.340664+0000","last_clean":"2024-04-25T00:01:06.340664+0000","last_became_active":"2024-04-25T00:01:04.354419+0000","last_became_peered":"2024-04-25T00:01:04.354419+0000","last_unstale":"2024-04-25T00:01:06.340664+0000","last_undegraded":"2024-04-25T00:01:06.340664+0000","last_fullsized":"2024-04-25T00:01:06.340664+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:26.243635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.381029+0000","last_change":"2024-04-25T00:01:05.385670+0000","last_active":"2024-04-25T00:01:06.381029+0000","last_peered":"2024-04-25T00:01:06.381029+0000","last_clean":"2024-04-25T00:01:06.381029+0000","last_became_active":"2024-04-25T00:01:05.384822+0000","last_became_peered":"2024-04-25T00:01:05.384822+0000","last_unstale":"2024-04-25T00:01:06.381029+0000","last_undegraded":"2024-04-25T00:01:06.381029+0000","last_fullsized":"2024-04-25T00:01:06.381029+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:24.017263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.340457+0000","last_change":"2024-04-25T00:01:04.368954+0000","last_active":"2024-04-25T00:01:06.340457+0000","last_peered":"2024-04-25T00:01:06.340457+0000","last_clean":"2024-04-25T00:01:06.340457+0000","last_became_active":"2024-04-25T00:01:04.367903+0000","last_became_peered":"2024-04-25T00:01:04.367903+0000","last_unstale":"2024-04-25T00:01:06.340457+0000","last_undegraded":"2024-04-25T00:01:06.340457+0000","last_fullsized":"2024-04-25T00:01:06.340457+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.337227+0000","last_change":"2024-04-25T00:01:05.376609+0000","last_active":"2024-04-25T00:01:06.337227+0000","last_peered":"2024-04-25T00:01:06.337227+0000","last_clean":"2024-04-25T00:01:06.337227+0000","last_became_active":"2024-04-25T00:01:05.375881+0000","last_became_peered":"2024-04-25T00:01:05.375881+0000","last_unstale":"2024-04-25T00:01:06.337227+0000","last_undegraded":"2024-04-25T00:01:06.337227+0000","last_fullsized":"2024-04-25T00:01:06.337227+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:46:05.592233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":170,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378246+0000","last_change":"2024-04-25T00:00:55.084492+0000","last_active":"2024-04-25T00:01:06.378246+0000","last_peered":"2024-04-25T00:01:06.378246+0000","last_clean":"2024-04-25T00:01:06.378246+0000","last_became_active":"2024-04-25T00:00:55.083493+0000","last_became_peered":"2024-04-25T00:00:55.083493+0000","last_unstale":"2024-04-25T00:01:06.378246+0000","last_undegraded":"2024-04-25T00:01:06.378246+0000","last_fullsized":"2024-04-25T00:01:06.378246+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:00:54.060578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:00:54.060578+0000","last_clean_scrub_stamp":"2024-04-25T00:00:54.060578+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:14:58.472477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.336858+0000","last_change":"2024-04-25T00:01:04.357028+0000","last_active":"2024-04-25T00:01:06.336858+0000","last_peered":"2024-04-25T00:01:06.336858+0000","last_clean":"2024-04-25T00:01:06.336858+0000","last_became_active":"2024-04-25T00:01:04.356644+0000","last_became_peered":"2024-04-25T00:01:04.356644+0000","last_unstale":"2024-04-25T00:01:06.336858+0000","last_undegraded":"2024-04-25T00:01:06.336858+0000","last_fullsized":"2024-04-25T00:01:06.336858+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:21.729893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1,"reported_epoch":21,"state":"creating+peering","last_fresh":"2024-04-25T00:01:04.333098+0000","last_change":"2024-04-25T00:01:04.333098+0000","last_active":"2024-04-25T00:01:04.314940+0000","last_peered":"2024-04-25T00:01:04.314940+0000","last_clean":"2024-04-25T00:01:04.314940+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T00:01:04.333098+0000","last_undegraded":"2024-04-25T00:01:04.333098+0000","last_fullsized":"2024-04-25T00:01:04.333098+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.336259+0000","last_change":"2024-04-25T00:01:04.370226+0000","last_active":"2024-04-25T00:01:06.336259+0000","last_peered":"2024-04-25T00:01:06.336259+0000","last_clean":"2024-04-25T00:01:06.336259+0000","last_became_active":"2024-04-25T00:01:04.369904+0000","last_became_peered":"2024-04-25T00:01:04.369904+0000","last_unstale":"2024-04-25T00:01:06.336259+0000","last_undegraded":"2024-04-25T00:01:06.336259+0000","last_fullsized":"2024-04-25T00:01:06.336259+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:47:25.595275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.337599+0000","last_change":"2024-04-25T00:01:05.370968+0000","last_active":"2024-04-25T00:01:06.337599+0000","last_peered":"2024-04-25T00:01:06.337599+0000","last_clean":"2024-04-25T00:01:06.337599+0000","last_became_active":"2024-04-25T00:01:05.369987+0000","last_became_peered":"2024-04-25T00:01:05.369987+0000","last_unstale":"2024-04-25T00:01:06.337599+0000","last_undegraded":"2024-04-25T00:01:06.337599+0000","last_fullsized":"2024-04-25T00:01:06.337599+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.377854+0000","last_change":"2024-04-25T00:01:04.372916+0000","last_active":"2024-04-25T00:01:06.377854+0000","last_peered":"2024-04-25T00:01:06.377854+0000","last_clean":"2024-04-25T00:01:06.377854+0000","last_became_active":"2024-04-25T00:01:04.372185+0000","last_became_peered":"2024-04-25T00:01:04.372185+0000","last_unstale":"2024-04-25T00:01:06.377854+0000","last_undegraded":"2024-04-25T00:01:06.377854+0000","last_fullsized":"2024-04-25T00:01:06.377854+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.338998+0000","last_change":"2024-04-25T00:01:05.354557+0000","last_active":"2024-04-25T00:01:06.338998+0000","last_peered":"2024-04-25T00:01:06.338998+0000","last_clean":"2024-04-25T00:01:06.338998+0000","last_became_active":"2024-04-25T00:01:05.353556+0000","last_became_peered":"2024-04-25T00:01:05.353556+0000","last_unstale":"2024-04-25T00:01:06.338998+0000","last_undegraded":"2024-04-25T00:01:06.338998+0000","last_fullsized":"2024-04-25T00:01:06.338998+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:24.964055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.337431+0000","last_change":"2024-04-25T00:01:04.361900+0000","last_active":"2024-04-25T00:01:06.337431+0000","last_peered":"2024-04-25T00:01:06.337431+0000","last_clean":"2024-04-25T00:01:06.337431+0000","last_became_active":"2024-04-25T00:01:04.361572+0000","last_became_peered":"2024-04-25T00:01:04.361572+0000","last_unstale":"2024-04-25T00:01:06.337431+0000","last_undegraded":"2024-04-25T00:01:06.337431+0000","last_fullsized":"2024-04-25T00:01:06.337431+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:29.218739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380225+0000","last_change":"2024-04-25T00:01:05.368769+0000","last_active":"2024-04-25T00:01:06.380225+0000","last_peered":"2024-04-25T00:01:06.380225+0000","last_clean":"2024-04-25T00:01:06.380225+0000","last_became_active":"2024-04-25T00:01:05.366817+0000","last_became_peered":"2024-04-25T00:01:05.366817+0000","last_unstale":"2024-04-25T00:01:06.380225+0000","last_undegraded":"2024-04-25T00:01:06.380225+0000","last_fullsized":"2024-04-25T00:01:06.380225+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:11:27.044893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.339852+0000","last_change":"2024-04-25T00:01:04.366577+0000","last_active":"2024-04-25T00:01:06.339852+0000","last_peered":"2024-04-25T00:01:06.339852+0000","last_clean":"2024-04-25T00:01:06.339852+0000","last_became_active":"2024-04-25T00:01:04.365881+0000","last_became_peered":"2024-04-25T00:01:04.365881+0000","last_unstale":"2024-04-25T00:01:06.339852+0000","last_undegraded":"2024-04-25T00:01:06.339852+0000","last_fullsized":"2024-04-25T00:01:06.339852+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:49:22.673421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.377363+0000","last_change":"2024-04-25T00:01:05.378340+0000","last_active":"2024-04-25T00:01:06.377363+0000","last_peered":"2024-04-25T00:01:06.377363+0000","last_clean":"2024-04-25T00:01:06.377363+0000","last_became_active":"2024-04-25T00:01:05.377593+0000","last_became_peered":"2024-04-25T00:01:05.377593+0000","last_unstale":"2024-04-25T00:01:06.377363+0000","last_undegraded":"2024-04-25T00:01:06.377363+0000","last_fullsized":"2024-04-25T00:01:06.377363+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:48.202251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.337490+0000","last_change":"2024-04-25T00:01:04.365391+0000","last_active":"2024-04-25T00:01:06.337490+0000","last_peered":"2024-04-25T00:01:06.337490+0000","last_clean":"2024-04-25T00:01:06.337490+0000","last_became_active":"2024-04-25T00:01:04.364989+0000","last_became_peered":"2024-04-25T00:01:04.364989+0000","last_unstale":"2024-04-25T00:01:06.337490+0000","last_undegraded":"2024-04-25T00:01:06.337490+0000","last_fullsized":"2024-04-25T00:01:06.337490+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:06:18.298057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.340089+0000","last_change":"2024-04-25T00:01:05.374011+0000","last_active":"2024-04-25T00:01:06.340089+0000","last_peered":"2024-04-25T00:01:06.340089+0000","last_clean":"2024-04-25T00:01:06.340089+0000","last_became_active":"2024-04-25T00:01:05.373276+0000","last_became_peered":"2024-04-25T00:01:05.373276+0000","last_unstale":"2024-04-25T00:01:06.340089+0000","last_undegraded":"2024-04-25T00:01:06.340089+0000","last_fullsized":"2024-04-25T00:01:06.340089+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:40:53.349428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-25T00:01:04.365099+0000","last_change":"2024-04-25T00:01:04.361603+0000","last_active":"2024-04-25T00:01:04.365099+0000","last_peered":"2024-04-25T00:01:04.365099+0000","last_clean":"2024-04-25T00:01:04.365099+0000","last_became_active":"2024-04-25T00:01:04.361046+0000","last_became_peered":"2024-04-25T00:01:04.361046+0000","last_unstale":"2024-04-25T00:01:04.365099+0000","last_undegraded":"2024-04-25T00:01:04.365099+0000","last_fullsized":"2024-04-25T00:01:04.365099+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:23:33.973380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":1,"reported_epoch":21,"state":"creating+peering","last_fresh":"2024-04-25T00:01:04.334917+0000","last_change":"2024-04-25T00:01:04.334917+0000","last_active":"2024-04-25T00:01:04.314940+0000","last_peered":"2024-04-25T00:01:04.314940+0000","last_clean":"2024-04-25T00:01:04.314940+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T00:01:04.334917+0000","last_undegraded":"2024-04-25T00:01:04.334917+0000","last_fullsized":"2024-04-25T00:01:04.334917+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"22'2","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.337655+0000","last_change":"2024-04-25T00:01:04.369917+0000","last_active":"2024-04-25T00:01:06.337655+0000","last_peered":"2024-04-25T00:01:06.337655+0000","last_clean":"2024-04-25T00:01:06.337655+0000","last_became_active":"2024-04-25T00:01:04.369309+0000","last_became_peered":"2024-04-25T00:01:04.369309+0000","last_unstale":"2024-04-25T00:01:06.337655+0000","last_undegraded":"2024-04-25T00:01:06.337655+0000","last_fullsized":"2024-04-25T00:01:06.337655+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:55:08.063965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":1,"reported_epoch":21,"state":"creating+peering","last_fresh":"2024-04-25T00:01:04.334467+0000","last_change":"2024-04-25T00:01:04.334467+0000","last_active":"2024-04-25T00:01:04.314940+0000","last_peered":"2024-04-25T00:01:04.314940+0000","last_clean":"2024-04-25T00:01:04.314940+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T00:01:04.334467+0000","last_undegraded":"2024-04-25T00:01:04.334467+0000","last_fullsized":"2024-04-25T00:01:04.334467+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378173+0000","last_change":"2024-04-25T00:01:04.365515+0000","last_active":"2024-04-25T00:01:06.378173+0000","last_peered":"2024-04-25T00:01:06.378173+0000","last_clean":"2024-04-25T00:01:06.378173+0000","last_became_active":"2024-04-25T00:01:04.364942+0000","last_became_peered":"2024-04-25T00:01:04.364942+0000","last_unstale":"2024-04-25T00:01:06.378173+0000","last_undegraded":"2024-04-25T00:01:06.378173+0000","last_fullsized":"2024-04-25T00:01:06.378173+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.336035+0000","last_change":"2024-04-25T00:01:05.354953+0000","last_active":"2024-04-25T00:01:06.336035+0000","last_peered":"2024-04-25T00:01:06.336035+0000","last_clean":"2024-04-25T00:01:06.336035+0000","last_became_active":"2024-04-25T00:01:05.354118+0000","last_became_peered":"2024-04-25T00:01:05.354118+0000","last_unstale":"2024-04-25T00:01:06.336035+0000","last_undegraded":"2024-04-25T00:01:06.336035+0000","last_fullsized":"2024-04-25T00:01:06.336035+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:09.858617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-25T00:01:04.367419+0000","last_change":"2024-04-25T00:01:04.362268+0000","last_active":"2024-04-25T00:01:04.367419+0000","last_peered":"2024-04-25T00:01:04.367419+0000","last_clean":"2024-04-25T00:01:04.367419+0000","last_became_active":"2024-04-25T00:01:04.361622+0000","last_became_peered":"2024-04-25T00:01:04.361622+0000","last_unstale":"2024-04-25T00:01:04.367419+0000","last_undegraded":"2024-04-25T00:01:04.367419+0000","last_fullsized":"2024-04-25T00:01:04.367419+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:40:09.442761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1,"reported_epoch":21,"state":"creating+peering","last_fresh":"2024-04-25T00:01:04.333620+0000","last_change":"2024-04-25T00:01:04.333620+0000","last_active":"2024-04-25T00:01:04.314940+0000","last_peered":"2024-04-25T00:01:04.314940+0000","last_clean":"2024-04-25T00:01:04.314940+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T00:01:04.333620+0000","last_undegraded":"2024-04-25T00:01:04.333620+0000","last_fullsized":"2024-04-25T00:01:04.333620+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.338480+0000","last_change":"2024-04-25T00:01:04.371935+0000","last_active":"2024-04-25T00:01:06.338480+0000","last_peered":"2024-04-25T00:01:06.338480+0000","last_clean":"2024-04-25T00:01:06.338480+0000","last_became_active":"2024-04-25T00:01:04.371420+0000","last_became_peered":"2024-04-25T00:01:04.371420+0000","last_unstale":"2024-04-25T00:01:06.338480+0000","last_undegraded":"2024-04-25T00:01:06.338480+0000","last_fullsized":"2024-04-25T00:01:06.338480+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378351+0000","last_change":"2024-04-25T00:01:05.362611+0000","last_active":"2024-04-25T00:01:06.378351+0000","last_peered":"2024-04-25T00:01:06.378351+0000","last_clean":"2024-04-25T00:01:06.378351+0000","last_became_active":"2024-04-25T00:01:05.361817+0000","last_became_peered":"2024-04-25T00:01:05.361817+0000","last_unstale":"2024-04-25T00:01:06.378351+0000","last_undegraded":"2024-04-25T00:01:06.378351+0000","last_fullsized":"2024-04-25T00:01:06.378351+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:50.489125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"22'2","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.336722+0000","last_change":"2024-04-25T00:01:04.359262+0000","last_active":"2024-04-25T00:01:06.336722+0000","last_peered":"2024-04-25T00:01:06.336722+0000","last_clean":"2024-04-25T00:01:06.336722+0000","last_became_active":"2024-04-25T00:01:04.357971+0000","last_became_peered":"2024-04-25T00:01:04.357971+0000","last_unstale":"2024-04-25T00:01:06.336722+0000","last_undegraded":"2024-04-25T00:01:06.336722+0000","last_fullsized":"2024-04-25T00:01:06.336722+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:05.441048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.336709+0000","last_change":"2024-04-25T00:01:05.376984+0000","last_active":"2024-04-25T00:01:06.336709+0000","last_peered":"2024-04-25T00:01:06.336709+0000","last_clean":"2024-04-25T00:01:06.336709+0000","last_became_active":"2024-04-25T00:01:05.376259+0000","last_became_peered":"2024-04-25T00:01:05.376259+0000","last_unstale":"2024-04-25T00:01:06.336709+0000","last_undegraded":"2024-04-25T00:01:06.336709+0000","last_fullsized":"2024-04-25T00:01:06.336709+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:13:28.543302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.336976+0000","last_change":"2024-04-25T00:01:05.381120+0000","last_active":"2024-04-25T00:01:06.336976+0000","last_peered":"2024-04-25T00:01:06.336976+0000","last_clean":"2024-04-25T00:01:06.336976+0000","last_became_active":"2024-04-25T00:01:05.378320+0000","last_became_peered":"2024-04-25T00:01:05.378320+0000","last_unstale":"2024-04-25T00:01:06.336976+0000","last_undegraded":"2024-04-25T00:01:06.336976+0000","last_fullsized":"2024-04-25T00:01:06.336976+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:34:06.175918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"22'2","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380832+0000","last_change":"2024-04-25T00:01:04.362832+0000","last_active":"2024-04-25T00:01:06.380832+0000","last_peered":"2024-04-25T00:01:06.380832+0000","last_clean":"2024-04-25T00:01:06.380832+0000","last_became_active":"2024-04-25T00:01:04.362059+0000","last_became_peered":"2024-04-25T00:01:04.362059+0000","last_unstale":"2024-04-25T00:01:06.380832+0000","last_undegraded":"2024-04-25T00:01:06.380832+0000","last_fullsized":"2024-04-25T00:01:06.380832+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:05.441048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.379969+0000","last_change":"2024-04-25T00:01:05.387195+0000","last_active":"2024-04-25T00:01:06.379969+0000","last_peered":"2024-04-25T00:01:06.379969+0000","last_clean":"2024-04-25T00:01:06.379969+0000","last_became_active":"2024-04-25T00:01:05.386853+0000","last_became_peered":"2024-04-25T00:01:05.386853+0000","last_unstale":"2024-04-25T00:01:06.379969+0000","last_undegraded":"2024-04-25T00:01:06.379969+0000","last_fullsized":"2024-04-25T00:01:06.379969+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:06:19.307378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"22'2","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380527+0000","last_change":"2024-04-25T00:01:04.362030+0000","last_active":"2024-04-25T00:01:06.380527+0000","last_peered":"2024-04-25T00:01:06.380527+0000","last_clean":"2024-04-25T00:01:06.380527+0000","last_became_active":"2024-04-25T00:01:04.361708+0000","last_became_peered":"2024-04-25T00:01:04.361708+0000","last_unstale":"2024-04-25T00:01:06.380527+0000","last_undegraded":"2024-04-25T00:01:06.380527+0000","last_fullsized":"2024-04-25T00:01:06.380527+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:38:33.392719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.340575+0000","last_change":"2024-04-25T00:01:05.370818+0000","last_active":"2024-04-25T00:01:06.340575+0000","last_peered":"2024-04-25T00:01:06.340575+0000","last_clean":"2024-04-25T00:01:06.340575+0000","last_became_active":"2024-04-25T00:01:05.369991+0000","last_became_peered":"2024-04-25T00:01:05.369991+0000","last_unstale":"2024-04-25T00:01:06.340575+0000","last_undegraded":"2024-04-25T00:01:06.340575+0000","last_fullsized":"2024-04-25T00:01:06.340575+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:43.615895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"22'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.340582+0000","last_change":"2024-04-25T00:01:04.361809+0000","last_active":"2024-04-25T00:01:06.340582+0000","last_peered":"2024-04-25T00:01:06.340582+0000","last_clean":"2024-04-25T00:01:06.340582+0000","last_became_active":"2024-04-25T00:01:04.361059+0000","last_became_peered":"2024-04-25T00:01:04.361059+0000","last_unstale":"2024-04-25T00:01:06.340582+0000","last_undegraded":"2024-04-25T00:01:06.340582+0000","last_fullsized":"2024-04-25T00:01:06.340582+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.340280+0000","last_change":"2024-04-25T00:01:05.367207+0000","last_active":"2024-04-25T00:01:06.340280+0000","last_peered":"2024-04-25T00:01:06.340280+0000","last_clean":"2024-04-25T00:01:06.340280+0000","last_became_active":"2024-04-25T00:01:05.366471+0000","last_became_peered":"2024-04-25T00:01:05.366471+0000","last_unstale":"2024-04-25T00:01:06.340280+0000","last_undegraded":"2024-04-25T00:01:06.340280+0000","last_fullsized":"2024-04-25T00:01:06.340280+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:30:23.557381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-25T00:01:04.364774+0000","last_change":"2024-04-25T00:01:04.356032+0000","last_active":"2024-04-25T00:01:04.364774+0000","last_peered":"2024-04-25T00:01:04.364774+0000","last_clean":"2024-04-25T00:01:04.364774+0000","last_became_active":"2024-04-25T00:01:04.355293+0000","last_became_peered":"2024-04-25T00:01:04.355293+0000","last_unstale":"2024-04-25T00:01:04.364774+0000","last_undegraded":"2024-04-25T00:01:04.364774+0000","last_fullsized":"2024-04-25T00:01:04.364774+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:34:05.574864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1828,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509446,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":241664,"data_stored":81279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509446,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":233472,"data_stored":81155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509445,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":823296,"data_stored":665340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509446,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":237568,"data_stored":81737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509447,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":233472,"data_stored":76492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509446,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":819200,"data_stored":665318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509446,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":241664,"data_stored":81211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":196608,"data_stored":61875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T00:01:10.399 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T00:01:10.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.554+0000 7f0cd68d8640 1 Processor -- start 2024-04-25T00:01:10.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.554+0000 7f0cd68d8640 1 -- start start 2024-04-25T00:01:10.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.558+0000 7f0cd68d8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0cd00a94a0 0x7f0cd01563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:10.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.558+0000 7f0cd68d8640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0cd0071220 con 0x7f0cd014c720 2024-04-25T00:01:10.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.558+0000 7f0cd68d8640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0cd0152420 con 0x7f0cd0071680 2024-04-25T00:01:10.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.558+0000 7f0cd68d8640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f0cd0153460 con 0x7f0cd00a94a0 2024-04-25T00:01:10.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.558+0000 7f0ccf7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0cd00a94a0 0x7f0cd01563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:10.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.558+0000 7f0ccf7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0cd00a94a0 0x7f0cd01563d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37596/0 (socket says 172.21.15.116:37596) 2024-04-25T00:01:10.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.558+0000 7f0ccffff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f0cd014c720 0x7f0cd00c2a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:47876/0 (socket says 172.21.15.116:47876) 2024-04-25T00:01:10.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.558+0000 7f0ccf7fe640 1 -- 172.21.15.116:0/2278641069 learned_addr learned my addr 172.21.15.116:0/2278641069 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:10.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.558+0000 7f0ccf7fe640 1 -- 172.21.15.116:0/2278641069 >> v1:172.21.15.174:6789/0 conn(0x7f0cd0071680 legacy=0x7f0cd0071ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T00:01:10.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.558+0000 7f0ccf7fe640 1 -- 172.21.15.116:0/2278641069 >> v1:172.21.15.116:6789/0 conn(0x7f0cd014c720 legacy=0x7f0cd00c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:10.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.558+0000 7f0ccf7fe640 1 -- 172.21.15.116:0/2278641069 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cd00721c0 con 0x7f0cd00a94a0 2024-04-25T00:01:10.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.558+0000 7f0ccf7fe640 1 --2- 172.21.15.116:0/2278641069 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cd00a94a0 0x7f0cd01563d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0cc400bb10 tx=0x7f0cc401e980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db5a0c64fc62b5af server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:10.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.558+0000 7f0cceffd640 1 -- 172.21.15.116:0/2278641069 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cc40b4070 con 0x7f0cd00a94a0 2024-04-25T00:01:10.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.558+0000 7f0cceffd640 1 -- 172.21.15.116:0/2278641069 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cc40a7950 con 0x7f0cd00a94a0 2024-04-25T00:01:10.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.558+0000 7f0cceffd640 1 -- 172.21.15.116:0/2278641069 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cc40a7d20 con 0x7f0cd00a94a0 2024-04-25T00:01:10.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.562+0000 7f0cd68d8640 1 -- 172.21.15.116:0/2278641069 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cd00a94a0 msgr2=0x7f0cd01563d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:10.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.562+0000 7f0cd68d8640 1 --2- 172.21.15.116:0/2278641069 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cd00a94a0 0x7f0cd01563d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0cc400bb10 tx=0x7f0cc401e980 comp rx=0 tx=0).stop 2024-04-25T00:01:10.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.562+0000 7f0cd68d8640 1 -- 172.21.15.116:0/2278641069 shutdown_connections 2024-04-25T00:01:10.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.562+0000 7f0cd68d8640 1 --2- 172.21.15.116:0/2278641069 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cd00a94a0 0x7f0cd01563d0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:10.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.562+0000 7f0cd68d8640 1 -- 172.21.15.116:0/2278641069 >> 172.21.15.116:0/2278641069 conn(0x7f0cd00c3b00 msgr2=0x7f0cd00c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:10.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.562+0000 7f0cd68d8640 1 -- 172.21.15.116:0/2278641069 shutdown_connections 2024-04-25T00:01:10.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.562+0000 7f0cd68d8640 1 -- 172.21.15.116:0/2278641069 wait complete. 2024-04-25T00:01:10.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.562+0000 7f0cd68d8640 1 Processor -- start 2024-04-25T00:01:10.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.562+0000 7f0cd68d8640 1 -- start start 2024-04-25T00:01:10.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.562+0000 7f0cd68d8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cd0071680 0x7f0cd0141210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:10.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.562+0000 7f0cd68d8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0cd00a94a0 0x7f0cd0154f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:10.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.562+0000 7f0cd68d8640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0cd014c720 0x7f0cd0153bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:10.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.562+0000 7f0cd68d8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0cd0152f40 con 0x7f0cd0071680 2024-04-25T00:01:10.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.562+0000 7f0cd68d8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0cd0152c60 con 0x7f0cd00a94a0 2024-04-25T00:01:10.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.562+0000 7f0cd68d8640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f0cd00c3120 con 0x7f0cd014c720 2024-04-25T00:01:10.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.562+0000 7f0ccffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cd0071680 0x7f0cd0141210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:10.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.562+0000 7f0ccffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cd0071680 0x7f0cd0141210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37602/0 (socket says 172.21.15.116:37602) 2024-04-25T00:01:10.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.562+0000 7f0ccffff640 1 -- 172.21.15.116:0/3810709975 learned_addr learned my addr 172.21.15.116:0/3810709975 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:10.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.562+0000 7f0ccffff640 1 -- 172.21.15.116:0/3810709975 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0cd014c720 msgr2=0x7f0cd0153bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:10.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.562+0000 7f0ccffff640 1 --2- 172.21.15.116:0/3810709975 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0cd014c720 0x7f0cd0153bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:10.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.562+0000 7f0ccffff640 1 -- 172.21.15.116:0/3810709975 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0cd00a94a0 msgr2=0x7f0cd0154f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:10.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.562+0000 7f0ccffff640 1 --2- 172.21.15.116:0/3810709975 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0cd00a94a0 0x7f0cd0154f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:10.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.562+0000 7f0ccffff640 1 -- 172.21.15.116:0/3810709975 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cd00ab710 con 0x7f0cd0071680 2024-04-25T00:01:10.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.562+0000 7f0ccffff640 1 --2- 172.21.15.116:0/3810709975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cd0071680 0x7f0cd0141210 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0cc000c760 tx=0x7f0cc000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:10.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.562+0000 7f0cccff9640 1 -- 172.21.15.116:0/3810709975 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cc0017070 con 0x7f0cd0071680 2024-04-25T00:01:10.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.562+0000 7f0cccff9640 1 -- 172.21.15.116:0/3810709975 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cc000a640 con 0x7f0cd0071680 2024-04-25T00:01:10.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.562+0000 7f0cccff9640 1 -- 172.21.15.116:0/3810709975 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cc000aa10 con 0x7f0cd0071680 2024-04-25T00:01:10.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.562+0000 7f0cd68d8640 1 -- 172.21.15.116:0/3810709975 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cd00c6970 con 0x7f0cd0071680 2024-04-25T00:01:10.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.562+0000 7f0cd68d8640 1 -- 172.21.15.116:0/3810709975 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cd00ab3a0 con 0x7f0cd0071680 2024-04-25T00:01:10.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.566+0000 7f0cd68d8640 1 -- 172.21.15.116:0/3810709975 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0cd0068350 con 0x7f0cd0071680 2024-04-25T00:01:10.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.566+0000 7f0cccff9640 1 -- 172.21.15.116:0/3810709975 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f0cc001a070 con 0x7f0cd0071680 2024-04-25T00:01:10.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.566+0000 7f0cccff9640 1 --2- 172.21.15.116:0/3810709975 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0cbc0678f0 0x7f0cbc069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:10.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.566+0000 7f0cccff9640 1 -- 172.21.15.116:0/3810709975 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5689+0+0 (secure 0 0 0) 0x7f0cc0085230 con 0x7f0cd0071680 2024-04-25T00:01:10.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.566+0000 7f0ccf7fe640 1 --2- 172.21.15.116:0/3810709975 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0cbc0678f0 0x7f0cbc069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:10.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.570+0000 7f0ccf7fe640 1 --2- 172.21.15.116:0/3810709975 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0cbc0678f0 0x7f0cbc069db0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0cc40b1000 tx=0x7f0cc4004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:10.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.570+0000 7f0cccff9640 1 -- 172.21.15.116:0/3810709975 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f0cc0055e10 con 0x7f0cd0071680 2024-04-25T00:01:10.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.726+0000 7f0cd68d8640 1 -- 172.21.15.116:0/3810709975 --> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0cd00aae70 con 0x7f0cbc0678f0 2024-04-25T00:01:10.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.746+0000 7f0cccff9640 1 -- 172.21.15.116:0/3810709975 <== mgr.4105 v2:172.21.15.174:6832/2004448211 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251355 (secure 0 0 0) 0x7f0cd00aae70 con 0x7f0cbc0678f0 2024-04-25T00:01:10.751 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:01:10.753 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-04-25T00:01:10.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.750+0000 7f0cd68d8640 1 -- 172.21.15.116:0/3810709975 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0cbc0678f0 msgr2=0x7f0cbc069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:10.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.750+0000 7f0cd68d8640 1 --2- 172.21.15.116:0/3810709975 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0cbc0678f0 0x7f0cbc069db0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0cc40b1000 tx=0x7f0cc4004740 comp rx=0 tx=0).stop 2024-04-25T00:01:10.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.750+0000 7f0cd68d8640 1 -- 172.21.15.116:0/3810709975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cd0071680 msgr2=0x7f0cd0141210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:10.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.750+0000 7f0cd68d8640 1 --2- 172.21.15.116:0/3810709975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cd0071680 0x7f0cd0141210 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0cc000c760 tx=0x7f0cc000cc20 comp rx=0 tx=0).stop 2024-04-25T00:01:10.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.750+0000 7f0cd68d8640 1 -- 172.21.15.116:0/3810709975 shutdown_connections 2024-04-25T00:01:10.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.750+0000 7f0cd68d8640 1 --2- 172.21.15.116:0/3810709975 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0cbc0678f0 0x7f0cbc069db0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:10.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.750+0000 7f0cd68d8640 1 --2- 172.21.15.116:0/3810709975 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0cd014c720 0x7f0cd0153bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:10.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.750+0000 7f0cd68d8640 1 --2- 172.21.15.116:0/3810709975 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0cd00a94a0 0x7f0cd0154f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:10.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.750+0000 7f0cd68d8640 1 --2- 172.21.15.116:0/3810709975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0cd0071680 0x7f0cd0141210 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:10.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.750+0000 7f0cd68d8640 1 -- 172.21.15.116:0/3810709975 >> 172.21.15.116:0/3810709975 conn(0x7f0cd00c3b00 msgr2=0x7f0cd00c2690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:10.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.750+0000 7f0cd68d8640 1 -- 172.21.15.116:0/3810709975 shutdown_connections 2024-04-25T00:01:10.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.750+0000 7f0cd68d8640 1 -- 172.21.15.116:0/3810709975 wait complete. 2024-04-25T00:01:10.768 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-04-25T00:01:09.251353+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592196,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":184,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141,"ondisk_log_size":141,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":181,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219136,"kb_used_data":2956,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755584,"statfs":{"total":773094113280,"available":772869718016,"internally_reserved":0,"allocated":3026944,"data_stored":1794407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":51,"apply_latency_ms":51,"commit_latency_ns":51000000,"apply_latency_ns":51000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1828,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.935044"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380670+0000","last_change":"2024-04-25T00:01:05.389596+0000","last_active":"2024-04-25T00:01:06.380670+0000","last_peered":"2024-04-25T00:01:06.380670+0000","last_clean":"2024-04-25T00:01:06.380670+0000","last_became_active":"2024-04-25T00:01:05.389127+0000","last_became_peered":"2024-04-25T00:01:05.389127+0000","last_unstale":"2024-04-25T00:01:06.380670+0000","last_undegraded":"2024-04-25T00:01:06.380670+0000","last_fullsized":"2024-04-25T00:01:06.380670+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:09.858617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.340220+0000","last_change":"2024-04-25T00:01:05.368190+0000","last_active":"2024-04-25T00:01:06.340220+0000","last_peered":"2024-04-25T00:01:06.340220+0000","last_clean":"2024-04-25T00:01:06.340220+0000","last_became_active":"2024-04-25T00:01:05.362606+0000","last_became_peered":"2024-04-25T00:01:05.362606+0000","last_unstale":"2024-04-25T00:01:06.340220+0000","last_undegraded":"2024-04-25T00:01:06.340220+0000","last_fullsized":"2024-04-25T00:01:06.340220+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:12:52.846250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.339987+0000","last_change":"2024-04-25T00:01:05.371291+0000","last_active":"2024-04-25T00:01:06.339987+0000","last_peered":"2024-04-25T00:01:06.339987+0000","last_clean":"2024-04-25T00:01:06.339987+0000","last_became_active":"2024-04-25T00:01:05.370522+0000","last_became_peered":"2024-04-25T00:01:05.370522+0000","last_unstale":"2024-04-25T00:01:06.339987+0000","last_undegraded":"2024-04-25T00:01:06.339987+0000","last_fullsized":"2024-04-25T00:01:06.339987+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:11:27.044893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.377499+0000","last_change":"2024-04-25T00:01:05.359716+0000","last_active":"2024-04-25T00:01:06.377499+0000","last_peered":"2024-04-25T00:01:06.377499+0000","last_clean":"2024-04-25T00:01:06.377499+0000","last_became_active":"2024-04-25T00:01:05.358866+0000","last_became_peered":"2024-04-25T00:01:05.358866+0000","last_unstale":"2024-04-25T00:01:06.377499+0000","last_undegraded":"2024-04-25T00:01:06.377499+0000","last_fullsized":"2024-04-25T00:01:06.377499+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:50.489125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380258+0000","last_change":"2024-04-25T00:01:05.386068+0000","last_active":"2024-04-25T00:01:06.380258+0000","last_peered":"2024-04-25T00:01:06.380258+0000","last_clean":"2024-04-25T00:01:06.380258+0000","last_became_active":"2024-04-25T00:01:05.378650+0000","last_became_peered":"2024-04-25T00:01:05.378650+0000","last_unstale":"2024-04-25T00:01:06.380258+0000","last_undegraded":"2024-04-25T00:01:06.380258+0000","last_fullsized":"2024-04-25T00:01:06.380258+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:12:52.846250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.377084+0000","last_change":"2024-04-25T00:01:05.359253+0000","last_active":"2024-04-25T00:01:06.377084+0000","last_peered":"2024-04-25T00:01:06.377084+0000","last_clean":"2024-04-25T00:01:06.377084+0000","last_became_active":"2024-04-25T00:01:05.358477+0000","last_became_peered":"2024-04-25T00:01:05.358477+0000","last_unstale":"2024-04-25T00:01:06.377084+0000","last_undegraded":"2024-04-25T00:01:06.377084+0000","last_fullsized":"2024-04-25T00:01:06.377084+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:22.739214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.379079+0000","last_change":"2024-04-25T00:01:05.374766+0000","last_active":"2024-04-25T00:01:06.379079+0000","last_peered":"2024-04-25T00:01:06.379079+0000","last_clean":"2024-04-25T00:01:06.379079+0000","last_became_active":"2024-04-25T00:01:05.369739+0000","last_became_peered":"2024-04-25T00:01:05.369739+0000","last_unstale":"2024-04-25T00:01:06.379079+0000","last_undegraded":"2024-04-25T00:01:06.379079+0000","last_fullsized":"2024-04-25T00:01:06.379079+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:09:18.315688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.339014+0000","last_change":"2024-04-25T00:01:05.390256+0000","last_active":"2024-04-25T00:01:06.339014+0000","last_peered":"2024-04-25T00:01:06.339014+0000","last_clean":"2024-04-25T00:01:06.339014+0000","last_became_active":"2024-04-25T00:01:05.389724+0000","last_became_peered":"2024-04-25T00:01:05.389724+0000","last_unstale":"2024-04-25T00:01:06.339014+0000","last_undegraded":"2024-04-25T00:01:06.339014+0000","last_fullsized":"2024-04-25T00:01:06.339014+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:09:18.315688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.379000+0000","last_change":"2024-04-25T00:01:05.362303+0000","last_active":"2024-04-25T00:01:06.379000+0000","last_peered":"2024-04-25T00:01:06.379000+0000","last_clean":"2024-04-25T00:01:06.379000+0000","last_became_active":"2024-04-25T00:01:05.361586+0000","last_became_peered":"2024-04-25T00:01:05.361586+0000","last_unstale":"2024-04-25T00:01:06.379000+0000","last_undegraded":"2024-04-25T00:01:06.379000+0000","last_fullsized":"2024-04-25T00:01:06.379000+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:30.228060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380324+0000","last_change":"2024-04-25T00:01:05.368974+0000","last_active":"2024-04-25T00:01:06.380324+0000","last_peered":"2024-04-25T00:01:06.380324+0000","last_clean":"2024-04-25T00:01:06.380324+0000","last_became_active":"2024-04-25T00:01:05.367893+0000","last_became_peered":"2024-04-25T00:01:05.367893+0000","last_unstale":"2024-04-25T00:01:06.380324+0000","last_undegraded":"2024-04-25T00:01:06.380324+0000","last_fullsized":"2024-04-25T00:01:06.380324+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:30:23.557381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380371+0000","last_change":"2024-04-25T00:01:05.389424+0000","last_active":"2024-04-25T00:01:06.380371+0000","last_peered":"2024-04-25T00:01:06.380371+0000","last_clean":"2024-04-25T00:01:06.380371+0000","last_became_active":"2024-04-25T00:01:05.388999+0000","last_became_peered":"2024-04-25T00:01:05.388999+0000","last_unstale":"2024-04-25T00:01:06.380371+0000","last_undegraded":"2024-04-25T00:01:06.380371+0000","last_fullsized":"2024-04-25T00:01:06.380371+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:31.456792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.379890+0000","last_change":"2024-04-25T00:01:05.385853+0000","last_active":"2024-04-25T00:01:06.379890+0000","last_peered":"2024-04-25T00:01:06.379890+0000","last_clean":"2024-04-25T00:01:06.379890+0000","last_became_active":"2024-04-25T00:01:05.384997+0000","last_became_peered":"2024-04-25T00:01:05.384997+0000","last_unstale":"2024-04-25T00:01:06.379890+0000","last_undegraded":"2024-04-25T00:01:06.379890+0000","last_fullsized":"2024-04-25T00:01:06.379890+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:36:55.436661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.339127+0000","last_change":"2024-04-25T00:01:05.381011+0000","last_active":"2024-04-25T00:01:06.339127+0000","last_peered":"2024-04-25T00:01:06.339127+0000","last_clean":"2024-04-25T00:01:06.339127+0000","last_became_active":"2024-04-25T00:01:05.380236+0000","last_became_peered":"2024-04-25T00:01:05.380236+0000","last_unstale":"2024-04-25T00:01:06.339127+0000","last_undegraded":"2024-04-25T00:01:06.339127+0000","last_fullsized":"2024-04-25T00:01:06.339127+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:06:19.307378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.337813+0000","last_change":"2024-04-25T00:01:05.378373+0000","last_active":"2024-04-25T00:01:06.337813+0000","last_peered":"2024-04-25T00:01:06.337813+0000","last_clean":"2024-04-25T00:01:06.337813+0000","last_became_active":"2024-04-25T00:01:05.376570+0000","last_became_peered":"2024-04-25T00:01:05.376570+0000","last_unstale":"2024-04-25T00:01:06.337813+0000","last_undegraded":"2024-04-25T00:01:06.337813+0000","last_fullsized":"2024-04-25T00:01:06.337813+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:30.228060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.337758+0000","last_change":"2024-04-25T00:01:05.390341+0000","last_active":"2024-04-25T00:01:06.337758+0000","last_peered":"2024-04-25T00:01:06.337758+0000","last_clean":"2024-04-25T00:01:06.337758+0000","last_became_active":"2024-04-25T00:01:05.389707+0000","last_became_peered":"2024-04-25T00:01:05.389707+0000","last_unstale":"2024-04-25T00:01:06.337758+0000","last_undegraded":"2024-04-25T00:01:06.337758+0000","last_fullsized":"2024-04-25T00:01:06.337758+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:54:21.138947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.335984+0000","last_change":"2024-04-25T00:01:05.370826+0000","last_active":"2024-04-25T00:01:06.335984+0000","last_peered":"2024-04-25T00:01:06.335984+0000","last_clean":"2024-04-25T00:01:06.335984+0000","last_became_active":"2024-04-25T00:01:05.369848+0000","last_became_peered":"2024-04-25T00:01:05.369848+0000","last_unstale":"2024-04-25T00:01:06.335984+0000","last_undegraded":"2024-04-25T00:01:06.335984+0000","last_fullsized":"2024-04-25T00:01:06.335984+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:23.870121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.337683+0000","last_change":"2024-04-25T00:01:05.379958+0000","last_active":"2024-04-25T00:01:06.337683+0000","last_peered":"2024-04-25T00:01:06.337683+0000","last_clean":"2024-04-25T00:01:06.337683+0000","last_became_active":"2024-04-25T00:01:05.379090+0000","last_became_peered":"2024-04-25T00:01:05.379090+0000","last_unstale":"2024-04-25T00:01:06.337683+0000","last_undegraded":"2024-04-25T00:01:06.337683+0000","last_fullsized":"2024-04-25T00:01:06.337683+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:59:50.832958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.379255+0000","last_change":"2024-04-25T00:01:05.367001+0000","last_active":"2024-04-25T00:01:06.379255+0000","last_peered":"2024-04-25T00:01:06.379255+0000","last_clean":"2024-04-25T00:01:06.379255+0000","last_became_active":"2024-04-25T00:01:05.366264+0000","last_became_peered":"2024-04-25T00:01:05.366264+0000","last_unstale":"2024-04-25T00:01:06.379255+0000","last_undegraded":"2024-04-25T00:01:06.379255+0000","last_fullsized":"2024-04-25T00:01:06.379255+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.337778+0000","last_change":"2024-04-25T00:01:05.370653+0000","last_active":"2024-04-25T00:01:06.337778+0000","last_peered":"2024-04-25T00:01:06.337778+0000","last_clean":"2024-04-25T00:01:06.337778+0000","last_became_active":"2024-04-25T00:01:05.369919+0000","last_became_peered":"2024-04-25T00:01:05.369919+0000","last_unstale":"2024-04-25T00:01:06.337778+0000","last_undegraded":"2024-04-25T00:01:06.337778+0000","last_fullsized":"2024-04-25T00:01:06.337778+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:30:08.311326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380032+0000","last_change":"2024-04-25T00:01:05.385253+0000","last_active":"2024-04-25T00:01:06.380032+0000","last_peered":"2024-04-25T00:01:06.380032+0000","last_clean":"2024-04-25T00:01:06.380032+0000","last_became_active":"2024-04-25T00:01:05.379429+0000","last_became_peered":"2024-04-25T00:01:05.379429+0000","last_unstale":"2024-04-25T00:01:06.380032+0000","last_undegraded":"2024-04-25T00:01:06.380032+0000","last_fullsized":"2024-04-25T00:01:06.380032+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:36:10.377757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":1,"reported_epoch":21,"state":"creating+peering","last_fresh":"2024-04-25T00:01:04.338341+0000","last_change":"2024-04-25T00:01:04.338341+0000","last_active":"2024-04-25T00:01:04.314940+0000","last_peered":"2024-04-25T00:01:04.314940+0000","last_clean":"2024-04-25T00:01:04.314940+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T00:01:04.338341+0000","last_undegraded":"2024-04-25T00:01:04.338341+0000","last_fullsized":"2024-04-25T00:01:04.338341+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378876+0000","last_change":"2024-04-25T00:01:05.387457+0000","last_active":"2024-04-25T00:01:06.378876+0000","last_peered":"2024-04-25T00:01:06.378876+0000","last_clean":"2024-04-25T00:01:06.378876+0000","last_became_active":"2024-04-25T00:01:05.386919+0000","last_became_peered":"2024-04-25T00:01:05.386919+0000","last_unstale":"2024-04-25T00:01:06.378876+0000","last_undegraded":"2024-04-25T00:01:06.378876+0000","last_fullsized":"2024-04-25T00:01:06.378876+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.338263+0000","last_change":"2024-04-25T00:01:05.366340+0000","last_active":"2024-04-25T00:01:06.338263+0000","last_peered":"2024-04-25T00:01:06.338263+0000","last_clean":"2024-04-25T00:01:06.338263+0000","last_became_active":"2024-04-25T00:01:05.361952+0000","last_became_peered":"2024-04-25T00:01:05.361952+0000","last_unstale":"2024-04-25T00:01:06.338263+0000","last_undegraded":"2024-04-25T00:01:06.338263+0000","last_fullsized":"2024-04-25T00:01:06.338263+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:41.948662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.335946+0000","last_change":"2024-04-25T00:01:05.372906+0000","last_active":"2024-04-25T00:01:06.335946+0000","last_peered":"2024-04-25T00:01:06.335946+0000","last_clean":"2024-04-25T00:01:06.335946+0000","last_became_active":"2024-04-25T00:01:05.372148+0000","last_became_peered":"2024-04-25T00:01:05.372148+0000","last_unstale":"2024-04-25T00:01:06.335946+0000","last_undegraded":"2024-04-25T00:01:06.335946+0000","last_fullsized":"2024-04-25T00:01:06.335946+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:43.615895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.340447+0000","last_change":"2024-04-25T00:01:05.373464+0000","last_active":"2024-04-25T00:01:06.340447+0000","last_peered":"2024-04-25T00:01:06.340447+0000","last_clean":"2024-04-25T00:01:06.340447+0000","last_became_active":"2024-04-25T00:01:05.372611+0000","last_became_peered":"2024-04-25T00:01:05.372611+0000","last_unstale":"2024-04-25T00:01:06.340447+0000","last_undegraded":"2024-04-25T00:01:06.340447+0000","last_fullsized":"2024-04-25T00:01:06.340447+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:31.456792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378208+0000","last_change":"2024-04-25T00:01:05.365761+0000","last_active":"2024-04-25T00:01:06.378208+0000","last_peered":"2024-04-25T00:01:06.378208+0000","last_clean":"2024-04-25T00:01:06.378208+0000","last_became_active":"2024-04-25T00:01:05.361119+0000","last_became_peered":"2024-04-25T00:01:05.361119+0000","last_unstale":"2024-04-25T00:01:06.378208+0000","last_undegraded":"2024-04-25T00:01:06.378208+0000","last_fullsized":"2024-04-25T00:01:06.378208+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.341008+0000","last_change":"2024-04-25T00:01:05.367296+0000","last_active":"2024-04-25T00:01:06.341008+0000","last_peered":"2024-04-25T00:01:06.341008+0000","last_clean":"2024-04-25T00:01:06.341008+0000","last_became_active":"2024-04-25T00:01:05.363838+0000","last_became_peered":"2024-04-25T00:01:05.363838+0000","last_unstale":"2024-04-25T00:01:06.341008+0000","last_undegraded":"2024-04-25T00:01:06.341008+0000","last_fullsized":"2024-04-25T00:01:06.341008+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:43:07.053074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.339622+0000","last_change":"2024-04-25T00:01:05.387057+0000","last_active":"2024-04-25T00:01:06.339622+0000","last_peered":"2024-04-25T00:01:06.339622+0000","last_clean":"2024-04-25T00:01:06.339622+0000","last_became_active":"2024-04-25T00:01:05.386724+0000","last_became_peered":"2024-04-25T00:01:05.386724+0000","last_unstale":"2024-04-25T00:01:06.339622+0000","last_undegraded":"2024-04-25T00:01:06.339622+0000","last_fullsized":"2024-04-25T00:01:06.339622+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:55:09.073286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.339053+0000","last_change":"2024-04-25T00:01:05.358690+0000","last_active":"2024-04-25T00:01:06.339053+0000","last_peered":"2024-04-25T00:01:06.339053+0000","last_clean":"2024-04-25T00:01:06.339053+0000","last_became_active":"2024-04-25T00:01:05.357340+0000","last_became_peered":"2024-04-25T00:01:05.357340+0000","last_unstale":"2024-04-25T00:01:06.339053+0000","last_undegraded":"2024-04-25T00:01:06.339053+0000","last_fullsized":"2024-04-25T00:01:06.339053+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:36:55.436661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.336324+0000","last_change":"2024-04-25T00:01:05.362960+0000","last_active":"2024-04-25T00:01:06.336324+0000","last_peered":"2024-04-25T00:01:06.336324+0000","last_clean":"2024-04-25T00:01:06.336324+0000","last_became_active":"2024-04-25T00:01:05.361067+0000","last_became_peered":"2024-04-25T00:01:05.361067+0000","last_unstale":"2024-04-25T00:01:06.336324+0000","last_undegraded":"2024-04-25T00:01:06.336324+0000","last_fullsized":"2024-04-25T00:01:06.336324+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:24.017263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.377980+0000","last_change":"2024-04-25T00:01:05.372927+0000","last_active":"2024-04-25T00:01:06.377980+0000","last_peered":"2024-04-25T00:01:06.377980+0000","last_clean":"2024-04-25T00:01:06.377980+0000","last_became_active":"2024-04-25T00:01:05.372050+0000","last_became_peered":"2024-04-25T00:01:05.372050+0000","last_unstale":"2024-04-25T00:01:06.377980+0000","last_undegraded":"2024-04-25T00:01:06.377980+0000","last_fullsized":"2024-04-25T00:01:06.377980+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:48.202251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.376574+0000","last_change":"2024-04-25T00:01:05.378517+0000","last_active":"2024-04-25T00:01:06.376574+0000","last_peered":"2024-04-25T00:01:06.376574+0000","last_clean":"2024-04-25T00:01:06.376574+0000","last_became_active":"2024-04-25T00:01:05.377742+0000","last_became_peered":"2024-04-25T00:01:05.377742+0000","last_unstale":"2024-04-25T00:01:06.376574+0000","last_undegraded":"2024-04-25T00:01:06.376574+0000","last_fullsized":"2024-04-25T00:01:06.376574+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:06.450369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.379065+0000","last_change":"2024-04-25T00:01:05.375142+0000","last_active":"2024-04-25T00:01:06.379065+0000","last_peered":"2024-04-25T00:01:06.379065+0000","last_clean":"2024-04-25T00:01:06.379065+0000","last_became_active":"2024-04-25T00:01:05.373857+0000","last_became_peered":"2024-04-25T00:01:05.373857+0000","last_unstale":"2024-04-25T00:01:06.379065+0000","last_undegraded":"2024-04-25T00:01:06.379065+0000","last_fullsized":"2024-04-25T00:01:06.379065+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:15.879673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"22'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.339475+0000","last_change":"2024-04-25T00:01:04.365607+0000","last_active":"2024-04-25T00:01:06.339475+0000","last_peered":"2024-04-25T00:01:06.339475+0000","last_clean":"2024-04-25T00:01:06.339475+0000","last_became_active":"2024-04-25T00:01:04.364788+0000","last_became_peered":"2024-04-25T00:01:04.364788+0000","last_unstale":"2024-04-25T00:01:06.339475+0000","last_undegraded":"2024-04-25T00:01:06.339475+0000","last_fullsized":"2024-04-25T00:01:06.339475+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":1,"reported_epoch":21,"state":"creating+peering","last_fresh":"2024-04-25T00:01:04.333513+0000","last_change":"2024-04-25T00:01:04.333513+0000","last_active":"2024-04-25T00:01:04.314940+0000","last_peered":"2024-04-25T00:01:04.314940+0000","last_clean":"2024-04-25T00:01:04.314940+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T00:01:04.333513+0000","last_undegraded":"2024-04-25T00:01:04.333513+0000","last_fullsized":"2024-04-25T00:01:04.333513+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"22'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.336967+0000","last_change":"2024-04-25T00:01:04.360254+0000","last_active":"2024-04-25T00:01:06.336967+0000","last_peered":"2024-04-25T00:01:06.336967+0000","last_clean":"2024-04-25T00:01:06.336967+0000","last_became_active":"2024-04-25T00:01:04.359586+0000","last_became_peered":"2024-04-25T00:01:04.359586+0000","last_unstale":"2024-04-25T00:01:06.336967+0000","last_undegraded":"2024-04-25T00:01:06.336967+0000","last_fullsized":"2024-04-25T00:01:06.336967+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:47.192930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.337866+0000","last_change":"2024-04-25T00:01:04.370896+0000","last_active":"2024-04-25T00:01:06.337866+0000","last_peered":"2024-04-25T00:01:06.337866+0000","last_clean":"2024-04-25T00:01:06.337866+0000","last_became_active":"2024-04-25T00:01:04.370273+0000","last_became_peered":"2024-04-25T00:01:04.370273+0000","last_unstale":"2024-04-25T00:01:06.337866+0000","last_undegraded":"2024-04-25T00:01:06.337866+0000","last_fullsized":"2024-04-25T00:01:06.337866+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:49:22.673421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.339542+0000","last_change":"2024-04-25T00:01:05.378867+0000","last_active":"2024-04-25T00:01:06.339542+0000","last_peered":"2024-04-25T00:01:06.339542+0000","last_clean":"2024-04-25T00:01:06.339542+0000","last_became_active":"2024-04-25T00:01:05.378056+0000","last_became_peered":"2024-04-25T00:01:05.378056+0000","last_unstale":"2024-04-25T00:01:06.339542+0000","last_undegraded":"2024-04-25T00:01:06.339542+0000","last_fullsized":"2024-04-25T00:01:06.339542+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:36:10.377757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.336193+0000","last_change":"2024-04-25T00:01:04.364523+0000","last_active":"2024-04-25T00:01:06.336193+0000","last_peered":"2024-04-25T00:01:06.336193+0000","last_clean":"2024-04-25T00:01:06.336193+0000","last_became_active":"2024-04-25T00:01:04.364050+0000","last_became_peered":"2024-04-25T00:01:04.364050+0000","last_unstale":"2024-04-25T00:01:06.336193+0000","last_undegraded":"2024-04-25T00:01:06.336193+0000","last_fullsized":"2024-04-25T00:01:06.336193+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:49.479804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":1,"reported_epoch":21,"state":"creating+peering","last_fresh":"2024-04-25T00:01:04.336083+0000","last_change":"2024-04-25T00:01:04.336083+0000","last_active":"2024-04-25T00:01:04.314940+0000","last_peered":"2024-04-25T00:01:04.314940+0000","last_clean":"2024-04-25T00:01:04.314940+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T00:01:04.336083+0000","last_undegraded":"2024-04-25T00:01:04.336083+0000","last_fullsized":"2024-04-25T00:01:04.336083+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.379231+0000","last_change":"2024-04-25T00:01:04.360804+0000","last_active":"2024-04-25T00:01:06.379231+0000","last_peered":"2024-04-25T00:01:06.379231+0000","last_clean":"2024-04-25T00:01:06.379231+0000","last_became_active":"2024-04-25T00:01:04.360374+0000","last_became_peered":"2024-04-25T00:01:04.360374+0000","last_unstale":"2024-04-25T00:01:06.379231+0000","last_undegraded":"2024-04-25T00:01:06.379231+0000","last_fullsized":"2024-04-25T00:01:06.379231+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378559+0000","last_change":"2024-04-25T00:01:05.365413+0000","last_active":"2024-04-25T00:01:06.378559+0000","last_peered":"2024-04-25T00:01:06.378559+0000","last_clean":"2024-04-25T00:01:06.378559+0000","last_became_active":"2024-04-25T00:01:05.361865+0000","last_became_peered":"2024-04-25T00:01:05.361865+0000","last_unstale":"2024-04-25T00:01:06.378559+0000","last_undegraded":"2024-04-25T00:01:06.378559+0000","last_fullsized":"2024-04-25T00:01:06.378559+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:30.228060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380688+0000","last_change":"2024-04-25T00:01:04.362129+0000","last_active":"2024-04-25T00:01:06.380688+0000","last_peered":"2024-04-25T00:01:06.380688+0000","last_clean":"2024-04-25T00:01:06.380688+0000","last_became_active":"2024-04-25T00:01:04.361827+0000","last_became_peered":"2024-04-25T00:01:04.361827+0000","last_unstale":"2024-04-25T00:01:06.380688+0000","last_undegraded":"2024-04-25T00:01:06.380688+0000","last_fullsized":"2024-04-25T00:01:06.380688+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:26.243635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.340801+0000","last_change":"2024-04-25T00:01:05.361723+0000","last_active":"2024-04-25T00:01:06.340801+0000","last_peered":"2024-04-25T00:01:06.340801+0000","last_clean":"2024-04-25T00:01:06.340801+0000","last_became_active":"2024-04-25T00:01:05.360956+0000","last_became_peered":"2024-04-25T00:01:05.360956+0000","last_unstale":"2024-04-25T00:01:06.340801+0000","last_undegraded":"2024-04-25T00:01:06.340801+0000","last_fullsized":"2024-04-25T00:01:06.340801+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:23.870121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"22'3","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.379142+0000","last_change":"2024-04-25T00:01:04.366503+0000","last_active":"2024-04-25T00:01:06.379142+0000","last_peered":"2024-04-25T00:01:06.379142+0000","last_clean":"2024-04-25T00:01:06.379142+0000","last_became_active":"2024-04-25T00:01:04.365945+0000","last_became_peered":"2024-04-25T00:01:04.365945+0000","last_unstale":"2024-04-25T00:01:06.379142+0000","last_undegraded":"2024-04-25T00:01:06.379142+0000","last_fullsized":"2024-04-25T00:01:06.379142+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.336525+0000","last_change":"2024-04-25T00:01:05.371061+0000","last_active":"2024-04-25T00:01:06.336525+0000","last_peered":"2024-04-25T00:01:06.336525+0000","last_clean":"2024-04-25T00:01:06.336525+0000","last_became_active":"2024-04-25T00:01:05.369950+0000","last_became_peered":"2024-04-25T00:01:05.369950+0000","last_unstale":"2024-04-25T00:01:06.336525+0000","last_undegraded":"2024-04-25T00:01:06.336525+0000","last_fullsized":"2024-04-25T00:01:06.336525+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:02:43.237902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"22'6","reported_seq":20,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.377439+0000","last_change":"2024-04-25T00:01:04.365185+0000","last_active":"2024-04-25T00:01:06.377439+0000","last_peered":"2024-04-25T00:01:06.377439+0000","last_clean":"2024-04-25T00:01:06.377439+0000","last_became_active":"2024-04-25T00:01:04.364930+0000","last_became_peered":"2024-04-25T00:01:04.364930+0000","last_unstale":"2024-04-25T00:01:06.377439+0000","last_undegraded":"2024-04-25T00:01:06.377439+0000","last_fullsized":"2024-04-25T00:01:06.377439+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:45:03.618340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380617+0000","last_change":"2024-04-25T00:01:05.385827+0000","last_active":"2024-04-25T00:01:06.380617+0000","last_peered":"2024-04-25T00:01:06.380617+0000","last_clean":"2024-04-25T00:01:06.380617+0000","last_became_active":"2024-04-25T00:01:05.385258+0000","last_became_peered":"2024-04-25T00:01:05.385258+0000","last_unstale":"2024-04-25T00:01:06.380617+0000","last_undegraded":"2024-04-25T00:01:06.380617+0000","last_fullsized":"2024-04-25T00:01:06.380617+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:23.870121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"22'2","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378778+0000","last_change":"2024-04-25T00:01:04.370640+0000","last_active":"2024-04-25T00:01:06.378778+0000","last_peered":"2024-04-25T00:01:06.378778+0000","last_clean":"2024-04-25T00:01:06.378778+0000","last_became_active":"2024-04-25T00:01:04.370031+0000","last_became_peered":"2024-04-25T00:01:04.370031+0000","last_unstale":"2024-04-25T00:01:06.378778+0000","last_undegraded":"2024-04-25T00:01:06.378778+0000","last_fullsized":"2024-04-25T00:01:06.378778+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:26.243635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.338234+0000","last_change":"2024-04-25T00:01:05.379102+0000","last_active":"2024-04-25T00:01:06.338234+0000","last_peered":"2024-04-25T00:01:06.338234+0000","last_clean":"2024-04-25T00:01:06.338234+0000","last_became_active":"2024-04-25T00:01:05.377625+0000","last_became_peered":"2024-04-25T00:01:05.377625+0000","last_unstale":"2024-04-25T00:01:06.338234+0000","last_undegraded":"2024-04-25T00:01:06.338234+0000","last_fullsized":"2024-04-25T00:01:06.338234+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:15.879673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.337539+0000","last_change":"2024-04-25T00:01:04.360987+0000","last_active":"2024-04-25T00:01:06.337539+0000","last_peered":"2024-04-25T00:01:06.337539+0000","last_clean":"2024-04-25T00:01:06.337539+0000","last_became_active":"2024-04-25T00:01:04.360664+0000","last_became_peered":"2024-04-25T00:01:04.360664+0000","last_unstale":"2024-04-25T00:01:06.337539+0000","last_undegraded":"2024-04-25T00:01:06.337539+0000","last_fullsized":"2024-04-25T00:01:06.337539+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:55.283076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.336555+0000","last_change":"2024-04-25T00:01:05.379501+0000","last_active":"2024-04-25T00:01:06.336555+0000","last_peered":"2024-04-25T00:01:06.336555+0000","last_clean":"2024-04-25T00:01:06.336555+0000","last_became_active":"2024-04-25T00:01:05.378644+0000","last_became_peered":"2024-04-25T00:01:05.378644+0000","last_unstale":"2024-04-25T00:01:06.336555+0000","last_undegraded":"2024-04-25T00:01:06.336555+0000","last_fullsized":"2024-04-25T00:01:06.336555+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:08:35.505154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.379209+0000","last_change":"2024-04-25T00:01:04.357265+0000","last_active":"2024-04-25T00:01:06.379209+0000","last_peered":"2024-04-25T00:01:06.379209+0000","last_clean":"2024-04-25T00:01:06.379209+0000","last_became_active":"2024-04-25T00:01:04.357004+0000","last_became_peered":"2024-04-25T00:01:04.357004+0000","last_unstale":"2024-04-25T00:01:06.379209+0000","last_undegraded":"2024-04-25T00:01:06.379209+0000","last_fullsized":"2024-04-25T00:01:06.379209+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:47:25.595275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378119+0000","last_change":"2024-04-25T00:01:05.373927+0000","last_active":"2024-04-25T00:01:06.378119+0000","last_peered":"2024-04-25T00:01:06.378119+0000","last_clean":"2024-04-25T00:01:06.378119+0000","last_became_active":"2024-04-25T00:01:05.373077+0000","last_became_peered":"2024-04-25T00:01:05.373077+0000","last_unstale":"2024-04-25T00:01:06.378119+0000","last_undegraded":"2024-04-25T00:01:06.378119+0000","last_fullsized":"2024-04-25T00:01:06.378119+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:50.489125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-25T00:01:04.363964+0000","last_change":"2024-04-25T00:01:04.358365+0000","last_active":"2024-04-25T00:01:04.363964+0000","last_peered":"2024-04-25T00:01:04.363964+0000","last_clean":"2024-04-25T00:01:04.363964+0000","last_became_active":"2024-04-25T00:01:04.357571+0000","last_became_peered":"2024-04-25T00:01:04.357571+0000","last_unstale":"2024-04-25T00:01:04.363964+0000","last_undegraded":"2024-04-25T00:01:04.363964+0000","last_fullsized":"2024-04-25T00:01:04.363964+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:11:06.373584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.335878+0000","last_change":"2024-04-25T00:01:05.364103+0000","last_active":"2024-04-25T00:01:06.335878+0000","last_peered":"2024-04-25T00:01:06.335878+0000","last_clean":"2024-04-25T00:01:06.335878+0000","last_became_active":"2024-04-25T00:01:05.363320+0000","last_became_peered":"2024-04-25T00:01:05.363320+0000","last_unstale":"2024-04-25T00:01:06.335878+0000","last_undegraded":"2024-04-25T00:01:06.335878+0000","last_fullsized":"2024-04-25T00:01:06.335878+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:31.456792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"22'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380600+0000","last_change":"2024-04-25T00:01:04.372376+0000","last_active":"2024-04-25T00:01:06.380600+0000","last_peered":"2024-04-25T00:01:06.380600+0000","last_clean":"2024-04-25T00:01:06.380600+0000","last_became_active":"2024-04-25T00:01:04.372040+0000","last_became_peered":"2024-04-25T00:01:04.372040+0000","last_unstale":"2024-04-25T00:01:06.380600+0000","last_undegraded":"2024-04-25T00:01:06.380600+0000","last_fullsized":"2024-04-25T00:01:06.380600+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:21.729893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378915+0000","last_change":"2024-04-25T00:01:05.372591+0000","last_active":"2024-04-25T00:01:06.378915+0000","last_peered":"2024-04-25T00:01:06.378915+0000","last_clean":"2024-04-25T00:01:06.378915+0000","last_became_active":"2024-04-25T00:01:05.371204+0000","last_became_peered":"2024-04-25T00:01:05.371204+0000","last_unstale":"2024-04-25T00:01:06.378915+0000","last_undegraded":"2024-04-25T00:01:06.378915+0000","last_fullsized":"2024-04-25T00:01:06.378915+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:15.879673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-25T00:01:04.373291+0000","last_change":"2024-04-25T00:01:04.370794+0000","last_active":"2024-04-25T00:01:04.373291+0000","last_peered":"2024-04-25T00:01:04.373291+0000","last_clean":"2024-04-25T00:01:04.373291+0000","last_became_active":"2024-04-25T00:01:04.370099+0000","last_became_peered":"2024-04-25T00:01:04.370099+0000","last_unstale":"2024-04-25T00:01:04.373291+0000","last_undegraded":"2024-04-25T00:01:04.373291+0000","last_fullsized":"2024-04-25T00:01:04.373291+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:27:27.293170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380496+0000","last_change":"2024-04-25T00:01:05.378840+0000","last_active":"2024-04-25T00:01:06.380496+0000","last_peered":"2024-04-25T00:01:06.380496+0000","last_clean":"2024-04-25T00:01:06.380496+0000","last_became_active":"2024-04-25T00:01:05.378112+0000","last_became_peered":"2024-04-25T00:01:05.378112+0000","last_unstale":"2024-04-25T00:01:06.380496+0000","last_undegraded":"2024-04-25T00:01:06.380496+0000","last_fullsized":"2024-04-25T00:01:06.380496+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:43.615895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"22'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.340664+0000","last_change":"2024-04-25T00:01:04.356627+0000","last_active":"2024-04-25T00:01:06.340664+0000","last_peered":"2024-04-25T00:01:06.340664+0000","last_clean":"2024-04-25T00:01:06.340664+0000","last_became_active":"2024-04-25T00:01:04.354419+0000","last_became_peered":"2024-04-25T00:01:04.354419+0000","last_unstale":"2024-04-25T00:01:06.340664+0000","last_undegraded":"2024-04-25T00:01:06.340664+0000","last_fullsized":"2024-04-25T00:01:06.340664+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:26.243635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.381029+0000","last_change":"2024-04-25T00:01:05.385670+0000","last_active":"2024-04-25T00:01:06.381029+0000","last_peered":"2024-04-25T00:01:06.381029+0000","last_clean":"2024-04-25T00:01:06.381029+0000","last_became_active":"2024-04-25T00:01:05.384822+0000","last_became_peered":"2024-04-25T00:01:05.384822+0000","last_unstale":"2024-04-25T00:01:06.381029+0000","last_undegraded":"2024-04-25T00:01:06.381029+0000","last_fullsized":"2024-04-25T00:01:06.381029+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:24.017263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.340457+0000","last_change":"2024-04-25T00:01:04.368954+0000","last_active":"2024-04-25T00:01:06.340457+0000","last_peered":"2024-04-25T00:01:06.340457+0000","last_clean":"2024-04-25T00:01:06.340457+0000","last_became_active":"2024-04-25T00:01:04.367903+0000","last_became_peered":"2024-04-25T00:01:04.367903+0000","last_unstale":"2024-04-25T00:01:06.340457+0000","last_undegraded":"2024-04-25T00:01:06.340457+0000","last_fullsized":"2024-04-25T00:01:06.340457+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.337227+0000","last_change":"2024-04-25T00:01:05.376609+0000","last_active":"2024-04-25T00:01:06.337227+0000","last_peered":"2024-04-25T00:01:06.337227+0000","last_clean":"2024-04-25T00:01:06.337227+0000","last_became_active":"2024-04-25T00:01:05.375881+0000","last_became_peered":"2024-04-25T00:01:05.375881+0000","last_unstale":"2024-04-25T00:01:06.337227+0000","last_undegraded":"2024-04-25T00:01:06.337227+0000","last_fullsized":"2024-04-25T00:01:06.337227+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:46:05.592233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":170,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378246+0000","last_change":"2024-04-25T00:00:55.084492+0000","last_active":"2024-04-25T00:01:06.378246+0000","last_peered":"2024-04-25T00:01:06.378246+0000","last_clean":"2024-04-25T00:01:06.378246+0000","last_became_active":"2024-04-25T00:00:55.083493+0000","last_became_peered":"2024-04-25T00:00:55.083493+0000","last_unstale":"2024-04-25T00:01:06.378246+0000","last_undegraded":"2024-04-25T00:01:06.378246+0000","last_fullsized":"2024-04-25T00:01:06.378246+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:00:54.060578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:00:54.060578+0000","last_clean_scrub_stamp":"2024-04-25T00:00:54.060578+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:14:58.472477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.336858+0000","last_change":"2024-04-25T00:01:04.357028+0000","last_active":"2024-04-25T00:01:06.336858+0000","last_peered":"2024-04-25T00:01:06.336858+0000","last_clean":"2024-04-25T00:01:06.336858+0000","last_became_active":"2024-04-25T00:01:04.356644+0000","last_became_peered":"2024-04-25T00:01:04.356644+0000","last_unstale":"2024-04-25T00:01:06.336858+0000","last_undegraded":"2024-04-25T00:01:06.336858+0000","last_fullsized":"2024-04-25T00:01:06.336858+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:21.729893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1,"reported_epoch":21,"state":"creating+peering","last_fresh":"2024-04-25T00:01:04.333098+0000","last_change":"2024-04-25T00:01:04.333098+0000","last_active":"2024-04-25T00:01:04.314940+0000","last_peered":"2024-04-25T00:01:04.314940+0000","last_clean":"2024-04-25T00:01:04.314940+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T00:01:04.333098+0000","last_undegraded":"2024-04-25T00:01:04.333098+0000","last_fullsized":"2024-04-25T00:01:04.333098+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.336259+0000","last_change":"2024-04-25T00:01:04.370226+0000","last_active":"2024-04-25T00:01:06.336259+0000","last_peered":"2024-04-25T00:01:06.336259+0000","last_clean":"2024-04-25T00:01:06.336259+0000","last_became_active":"2024-04-25T00:01:04.369904+0000","last_became_peered":"2024-04-25T00:01:04.369904+0000","last_unstale":"2024-04-25T00:01:06.336259+0000","last_undegraded":"2024-04-25T00:01:06.336259+0000","last_fullsized":"2024-04-25T00:01:06.336259+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:47:25.595275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.337599+0000","last_change":"2024-04-25T00:01:05.370968+0000","last_active":"2024-04-25T00:01:06.337599+0000","last_peered":"2024-04-25T00:01:06.337599+0000","last_clean":"2024-04-25T00:01:06.337599+0000","last_became_active":"2024-04-25T00:01:05.369987+0000","last_became_peered":"2024-04-25T00:01:05.369987+0000","last_unstale":"2024-04-25T00:01:06.337599+0000","last_undegraded":"2024-04-25T00:01:06.337599+0000","last_fullsized":"2024-04-25T00:01:06.337599+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.377854+0000","last_change":"2024-04-25T00:01:04.372916+0000","last_active":"2024-04-25T00:01:06.377854+0000","last_peered":"2024-04-25T00:01:06.377854+0000","last_clean":"2024-04-25T00:01:06.377854+0000","last_became_active":"2024-04-25T00:01:04.372185+0000","last_became_peered":"2024-04-25T00:01:04.372185+0000","last_unstale":"2024-04-25T00:01:06.377854+0000","last_undegraded":"2024-04-25T00:01:06.377854+0000","last_fullsized":"2024-04-25T00:01:06.377854+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.338998+0000","last_change":"2024-04-25T00:01:05.354557+0000","last_active":"2024-04-25T00:01:06.338998+0000","last_peered":"2024-04-25T00:01:06.338998+0000","last_clean":"2024-04-25T00:01:06.338998+0000","last_became_active":"2024-04-25T00:01:05.353556+0000","last_became_peered":"2024-04-25T00:01:05.353556+0000","last_unstale":"2024-04-25T00:01:06.338998+0000","last_undegraded":"2024-04-25T00:01:06.338998+0000","last_fullsized":"2024-04-25T00:01:06.338998+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:24.964055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.337431+0000","last_change":"2024-04-25T00:01:04.361900+0000","last_active":"2024-04-25T00:01:06.337431+0000","last_peered":"2024-04-25T00:01:06.337431+0000","last_clean":"2024-04-25T00:01:06.337431+0000","last_became_active":"2024-04-25T00:01:04.361572+0000","last_became_peered":"2024-04-25T00:01:04.361572+0000","last_unstale":"2024-04-25T00:01:06.337431+0000","last_undegraded":"2024-04-25T00:01:06.337431+0000","last_fullsized":"2024-04-25T00:01:06.337431+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:29.218739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380225+0000","last_change":"2024-04-25T00:01:05.368769+0000","last_active":"2024-04-25T00:01:06.380225+0000","last_peered":"2024-04-25T00:01:06.380225+0000","last_clean":"2024-04-25T00:01:06.380225+0000","last_became_active":"2024-04-25T00:01:05.366817+0000","last_became_peered":"2024-04-25T00:01:05.366817+0000","last_unstale":"2024-04-25T00:01:06.380225+0000","last_undegraded":"2024-04-25T00:01:06.380225+0000","last_fullsized":"2024-04-25T00:01:06.380225+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:11:27.044893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.339852+0000","last_change":"2024-04-25T00:01:04.366577+0000","last_active":"2024-04-25T00:01:06.339852+0000","last_peered":"2024-04-25T00:01:06.339852+0000","last_clean":"2024-04-25T00:01:06.339852+0000","last_became_active":"2024-04-25T00:01:04.365881+0000","last_became_peered":"2024-04-25T00:01:04.365881+0000","last_unstale":"2024-04-25T00:01:06.339852+0000","last_undegraded":"2024-04-25T00:01:06.339852+0000","last_fullsized":"2024-04-25T00:01:06.339852+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:49:22.673421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.377363+0000","last_change":"2024-04-25T00:01:05.378340+0000","last_active":"2024-04-25T00:01:06.377363+0000","last_peered":"2024-04-25T00:01:06.377363+0000","last_clean":"2024-04-25T00:01:06.377363+0000","last_became_active":"2024-04-25T00:01:05.377593+0000","last_became_peered":"2024-04-25T00:01:05.377593+0000","last_unstale":"2024-04-25T00:01:06.377363+0000","last_undegraded":"2024-04-25T00:01:06.377363+0000","last_fullsized":"2024-04-25T00:01:06.377363+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:48.202251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.337490+0000","last_change":"2024-04-25T00:01:04.365391+0000","last_active":"2024-04-25T00:01:06.337490+0000","last_peered":"2024-04-25T00:01:06.337490+0000","last_clean":"2024-04-25T00:01:06.337490+0000","last_became_active":"2024-04-25T00:01:04.364989+0000","last_became_peered":"2024-04-25T00:01:04.364989+0000","last_unstale":"2024-04-25T00:01:06.337490+0000","last_undegraded":"2024-04-25T00:01:06.337490+0000","last_fullsized":"2024-04-25T00:01:06.337490+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:06:18.298057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.340089+0000","last_change":"2024-04-25T00:01:05.374011+0000","last_active":"2024-04-25T00:01:06.340089+0000","last_peered":"2024-04-25T00:01:06.340089+0000","last_clean":"2024-04-25T00:01:06.340089+0000","last_became_active":"2024-04-25T00:01:05.373276+0000","last_became_peered":"2024-04-25T00:01:05.373276+0000","last_unstale":"2024-04-25T00:01:06.340089+0000","last_undegraded":"2024-04-25T00:01:06.340089+0000","last_fullsized":"2024-04-25T00:01:06.340089+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:40:53.349428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-25T00:01:04.365099+0000","last_change":"2024-04-25T00:01:04.361603+0000","last_active":"2024-04-25T00:01:04.365099+0000","last_peered":"2024-04-25T00:01:04.365099+0000","last_clean":"2024-04-25T00:01:04.365099+0000","last_became_active":"2024-04-25T00:01:04.361046+0000","last_became_peered":"2024-04-25T00:01:04.361046+0000","last_unstale":"2024-04-25T00:01:04.365099+0000","last_undegraded":"2024-04-25T00:01:04.365099+0000","last_fullsized":"2024-04-25T00:01:04.365099+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:23:33.973380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":1,"reported_epoch":21,"state":"creating+peering","last_fresh":"2024-04-25T00:01:04.334917+0000","last_change":"2024-04-25T00:01:04.334917+0000","last_active":"2024-04-25T00:01:04.314940+0000","last_peered":"2024-04-25T00:01:04.314940+0000","last_clean":"2024-04-25T00:01:04.314940+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T00:01:04.334917+0000","last_undegraded":"2024-04-25T00:01:04.334917+0000","last_fullsized":"2024-04-25T00:01:04.334917+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"22'2","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.337655+0000","last_change":"2024-04-25T00:01:04.369917+0000","last_active":"2024-04-25T00:01:06.337655+0000","last_peered":"2024-04-25T00:01:06.337655+0000","last_clean":"2024-04-25T00:01:06.337655+0000","last_became_active":"2024-04-25T00:01:04.369309+0000","last_became_peered":"2024-04-25T00:01:04.369309+0000","last_unstale":"2024-04-25T00:01:06.337655+0000","last_undegraded":"2024-04-25T00:01:06.337655+0000","last_fullsized":"2024-04-25T00:01:06.337655+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:55:08.063965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":1,"reported_epoch":21,"state":"creating+peering","last_fresh":"2024-04-25T00:01:04.334467+0000","last_change":"2024-04-25T00:01:04.334467+0000","last_active":"2024-04-25T00:01:04.314940+0000","last_peered":"2024-04-25T00:01:04.314940+0000","last_clean":"2024-04-25T00:01:04.314940+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T00:01:04.334467+0000","last_undegraded":"2024-04-25T00:01:04.334467+0000","last_fullsized":"2024-04-25T00:01:04.334467+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378173+0000","last_change":"2024-04-25T00:01:04.365515+0000","last_active":"2024-04-25T00:01:06.378173+0000","last_peered":"2024-04-25T00:01:06.378173+0000","last_clean":"2024-04-25T00:01:06.378173+0000","last_became_active":"2024-04-25T00:01:04.364942+0000","last_became_peered":"2024-04-25T00:01:04.364942+0000","last_unstale":"2024-04-25T00:01:06.378173+0000","last_undegraded":"2024-04-25T00:01:06.378173+0000","last_fullsized":"2024-04-25T00:01:06.378173+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.336035+0000","last_change":"2024-04-25T00:01:05.354953+0000","last_active":"2024-04-25T00:01:06.336035+0000","last_peered":"2024-04-25T00:01:06.336035+0000","last_clean":"2024-04-25T00:01:06.336035+0000","last_became_active":"2024-04-25T00:01:05.354118+0000","last_became_peered":"2024-04-25T00:01:05.354118+0000","last_unstale":"2024-04-25T00:01:06.336035+0000","last_undegraded":"2024-04-25T00:01:06.336035+0000","last_fullsized":"2024-04-25T00:01:06.336035+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:09.858617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-25T00:01:04.367419+0000","last_change":"2024-04-25T00:01:04.362268+0000","last_active":"2024-04-25T00:01:04.367419+0000","last_peered":"2024-04-25T00:01:04.367419+0000","last_clean":"2024-04-25T00:01:04.367419+0000","last_became_active":"2024-04-25T00:01:04.361622+0000","last_became_peered":"2024-04-25T00:01:04.361622+0000","last_unstale":"2024-04-25T00:01:04.367419+0000","last_undegraded":"2024-04-25T00:01:04.367419+0000","last_fullsized":"2024-04-25T00:01:04.367419+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:40:09.442761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1,"reported_epoch":21,"state":"creating+peering","last_fresh":"2024-04-25T00:01:04.333620+0000","last_change":"2024-04-25T00:01:04.333620+0000","last_active":"2024-04-25T00:01:04.314940+0000","last_peered":"2024-04-25T00:01:04.314940+0000","last_clean":"2024-04-25T00:01:04.314940+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T00:01:04.333620+0000","last_undegraded":"2024-04-25T00:01:04.333620+0000","last_fullsized":"2024-04-25T00:01:04.333620+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.338480+0000","last_change":"2024-04-25T00:01:04.371935+0000","last_active":"2024-04-25T00:01:06.338480+0000","last_peered":"2024-04-25T00:01:06.338480+0000","last_clean":"2024-04-25T00:01:06.338480+0000","last_became_active":"2024-04-25T00:01:04.371420+0000","last_became_peered":"2024-04-25T00:01:04.371420+0000","last_unstale":"2024-04-25T00:01:06.338480+0000","last_undegraded":"2024-04-25T00:01:06.338480+0000","last_fullsized":"2024-04-25T00:01:06.338480+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378351+0000","last_change":"2024-04-25T00:01:05.362611+0000","last_active":"2024-04-25T00:01:06.378351+0000","last_peered":"2024-04-25T00:01:06.378351+0000","last_clean":"2024-04-25T00:01:06.378351+0000","last_became_active":"2024-04-25T00:01:05.361817+0000","last_became_peered":"2024-04-25T00:01:05.361817+0000","last_unstale":"2024-04-25T00:01:06.378351+0000","last_undegraded":"2024-04-25T00:01:06.378351+0000","last_fullsized":"2024-04-25T00:01:06.378351+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:50.489125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"22'2","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.336722+0000","last_change":"2024-04-25T00:01:04.359262+0000","last_active":"2024-04-25T00:01:06.336722+0000","last_peered":"2024-04-25T00:01:06.336722+0000","last_clean":"2024-04-25T00:01:06.336722+0000","last_became_active":"2024-04-25T00:01:04.357971+0000","last_became_peered":"2024-04-25T00:01:04.357971+0000","last_unstale":"2024-04-25T00:01:06.336722+0000","last_undegraded":"2024-04-25T00:01:06.336722+0000","last_fullsized":"2024-04-25T00:01:06.336722+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:05.441048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.336709+0000","last_change":"2024-04-25T00:01:05.376984+0000","last_active":"2024-04-25T00:01:06.336709+0000","last_peered":"2024-04-25T00:01:06.336709+0000","last_clean":"2024-04-25T00:01:06.336709+0000","last_became_active":"2024-04-25T00:01:05.376259+0000","last_became_peered":"2024-04-25T00:01:05.376259+0000","last_unstale":"2024-04-25T00:01:06.336709+0000","last_undegraded":"2024-04-25T00:01:06.336709+0000","last_fullsized":"2024-04-25T00:01:06.336709+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:13:28.543302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.336976+0000","last_change":"2024-04-25T00:01:05.381120+0000","last_active":"2024-04-25T00:01:06.336976+0000","last_peered":"2024-04-25T00:01:06.336976+0000","last_clean":"2024-04-25T00:01:06.336976+0000","last_became_active":"2024-04-25T00:01:05.378320+0000","last_became_peered":"2024-04-25T00:01:05.378320+0000","last_unstale":"2024-04-25T00:01:06.336976+0000","last_undegraded":"2024-04-25T00:01:06.336976+0000","last_fullsized":"2024-04-25T00:01:06.336976+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:34:06.175918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"22'2","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380832+0000","last_change":"2024-04-25T00:01:04.362832+0000","last_active":"2024-04-25T00:01:06.380832+0000","last_peered":"2024-04-25T00:01:06.380832+0000","last_clean":"2024-04-25T00:01:06.380832+0000","last_became_active":"2024-04-25T00:01:04.362059+0000","last_became_peered":"2024-04-25T00:01:04.362059+0000","last_unstale":"2024-04-25T00:01:06.380832+0000","last_undegraded":"2024-04-25T00:01:06.380832+0000","last_fullsized":"2024-04-25T00:01:06.380832+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:05.441048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.379969+0000","last_change":"2024-04-25T00:01:05.387195+0000","last_active":"2024-04-25T00:01:06.379969+0000","last_peered":"2024-04-25T00:01:06.379969+0000","last_clean":"2024-04-25T00:01:06.379969+0000","last_became_active":"2024-04-25T00:01:05.386853+0000","last_became_peered":"2024-04-25T00:01:05.386853+0000","last_unstale":"2024-04-25T00:01:06.379969+0000","last_undegraded":"2024-04-25T00:01:06.379969+0000","last_fullsized":"2024-04-25T00:01:06.379969+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:06:19.307378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"22'2","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380527+0000","last_change":"2024-04-25T00:01:04.362030+0000","last_active":"2024-04-25T00:01:06.380527+0000","last_peered":"2024-04-25T00:01:06.380527+0000","last_clean":"2024-04-25T00:01:06.380527+0000","last_became_active":"2024-04-25T00:01:04.361708+0000","last_became_peered":"2024-04-25T00:01:04.361708+0000","last_unstale":"2024-04-25T00:01:06.380527+0000","last_undegraded":"2024-04-25T00:01:06.380527+0000","last_fullsized":"2024-04-25T00:01:06.380527+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:38:33.392719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.340575+0000","last_change":"2024-04-25T00:01:05.370818+0000","last_active":"2024-04-25T00:01:06.340575+0000","last_peered":"2024-04-25T00:01:06.340575+0000","last_clean":"2024-04-25T00:01:06.340575+0000","last_became_active":"2024-04-25T00:01:05.369991+0000","last_became_peered":"2024-04-25T00:01:05.369991+0000","last_unstale":"2024-04-25T00:01:06.340575+0000","last_undegraded":"2024-04-25T00:01:06.340575+0000","last_fullsized":"2024-04-25T00:01:06.340575+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:43.615895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"22'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.340582+0000","last_change":"2024-04-25T00:01:04.361809+0000","last_active":"2024-04-25T00:01:06.340582+0000","last_peered":"2024-04-25T00:01:06.340582+0000","last_clean":"2024-04-25T00:01:06.340582+0000","last_became_active":"2024-04-25T00:01:04.361059+0000","last_became_peered":"2024-04-25T00:01:04.361059+0000","last_unstale":"2024-04-25T00:01:06.340582+0000","last_undegraded":"2024-04-25T00:01:06.340582+0000","last_fullsized":"2024-04-25T00:01:06.340582+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.340280+0000","last_change":"2024-04-25T00:01:05.367207+0000","last_active":"2024-04-25T00:01:06.340280+0000","last_peered":"2024-04-25T00:01:06.340280+0000","last_clean":"2024-04-25T00:01:06.340280+0000","last_became_active":"2024-04-25T00:01:05.366471+0000","last_became_peered":"2024-04-25T00:01:05.366471+0000","last_unstale":"2024-04-25T00:01:06.340280+0000","last_undegraded":"2024-04-25T00:01:06.340280+0000","last_fullsized":"2024-04-25T00:01:06.340280+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:30:23.557381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-25T00:01:04.364774+0000","last_change":"2024-04-25T00:01:04.356032+0000","last_active":"2024-04-25T00:01:04.364774+0000","last_peered":"2024-04-25T00:01:04.364774+0000","last_clean":"2024-04-25T00:01:04.364774+0000","last_became_active":"2024-04-25T00:01:04.355293+0000","last_became_peered":"2024-04-25T00:01:04.355293+0000","last_unstale":"2024-04-25T00:01:04.364774+0000","last_undegraded":"2024-04-25T00:01:04.364774+0000","last_fullsized":"2024-04-25T00:01:04.364774+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:34:05.574864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1828,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509446,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":241664,"data_stored":81279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509446,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":233472,"data_stored":81155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509445,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":823296,"data_stored":665340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509446,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":237568,"data_stored":81737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509447,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":233472,"data_stored":76492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509446,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":819200,"data_stored":665318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509446,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":241664,"data_stored":81211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":196608,"data_stored":61875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T00:01:10.773 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T00:01:10.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.930+0000 7f6e8bb9f640 1 Processor -- start 2024-04-25T00:01:10.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.930+0000 7f6e8bb9f640 1 -- start start 2024-04-25T00:01:10.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.930+0000 7f6e8bb9f640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f6e8414c720 0x7f6e841563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:10.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.930+0000 7f6e8bb9f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6e84071680 0x7f6e84071ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:10.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.930+0000 7f6e8bb9f640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e84071220 con 0x7f6e840a94a0 2024-04-25T00:01:10.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.930+0000 7f6e8bb9f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f6e84153460 con 0x7f6e84071680 2024-04-25T00:01:10.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.930+0000 7f6e8bb9f640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f6e84152420 con 0x7f6e8414c720 2024-04-25T00:01:10.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.930+0000 7f6e8a13c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6e84071680 0x7f6e84071ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:10.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.930+0000 7f6e8a13c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6e84071680 0x7f6e84071ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37606/0 (socket says 172.21.15.116:37606) 2024-04-25T00:01:10.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.930+0000 7f6e8a13c640 1 -- 172.21.15.116:0/2045632266 learned_addr learned my addr 172.21.15.116:0/2045632266 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:10.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.930+0000 7f6e8993b640 1 --2- 172.21.15.116:0/2045632266 >> v2:172.21.15.174:3300/0 conn(0x7f6e8414c720 0x7f6e841563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:10.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.930+0000 7f6e8a13c640 1 -- 172.21.15.116:0/2045632266 >> v2:172.21.15.174:3300/0 conn(0x7f6e8414c720 msgr2=0x7f6e841563d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:10.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.930+0000 7f6e8a13c640 1 --2- 172.21.15.116:0/2045632266 >> v2:172.21.15.174:3300/0 conn(0x7f6e8414c720 0x7f6e841563d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:10.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.930+0000 7f6e8a13c640 1 -- 172.21.15.116:0/2045632266 >> v1:172.21.15.174:6789/0 conn(0x7f6e840a94a0 legacy=0x7f6e840c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:10.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.930+0000 7f6e8a13c640 1 -- 172.21.15.116:0/2045632266 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e8414cb00 con 0x7f6e84071680 2024-04-25T00:01:10.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.930+0000 7f6e8993b640 1 --2- 172.21.15.116:0/2045632266 >> v2:172.21.15.174:3300/0 conn(0x7f6e8414c720 0x7f6e841563d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:10.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.930+0000 7f6e8a13c640 1 --2- 172.21.15.116:0/2045632266 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e84071680 0x7f6e84071ab0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6e84070190 tx=0x7f6e7c01c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b9decaf8bc4f9ae server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:10.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.934+0000 7f6e88939640 1 -- 172.21.15.116:0/2045632266 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e7c0ab620 con 0x7f6e84071680 2024-04-25T00:01:10.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.934+0000 7f6e88939640 1 -- 172.21.15.116:0/2045632266 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e7c0b0070 con 0x7f6e84071680 2024-04-25T00:01:10.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.934+0000 7f6e88939640 1 -- 172.21.15.116:0/2045632266 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e7c0ab9e0 con 0x7f6e84071680 2024-04-25T00:01:10.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.934+0000 7f6e8bb9f640 1 -- 172.21.15.116:0/2045632266 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e84071680 msgr2=0x7f6e84071ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:10.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.934+0000 7f6e8bb9f640 1 --2- 172.21.15.116:0/2045632266 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e84071680 0x7f6e84071ab0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f6e84070190 tx=0x7f6e7c01c420 comp rx=0 tx=0).stop 2024-04-25T00:01:10.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.934+0000 7f6e8bb9f640 1 -- 172.21.15.116:0/2045632266 shutdown_connections 2024-04-25T00:01:10.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.934+0000 7f6e8bb9f640 1 --2- 172.21.15.116:0/2045632266 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e84071680 0x7f6e84071ab0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:10.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.934+0000 7f6e8bb9f640 1 --2- 172.21.15.116:0/2045632266 >> v2:172.21.15.174:3300/0 conn(0x7f6e8414c720 0x7f6e841563d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:10.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.934+0000 7f6e8bb9f640 1 -- 172.21.15.116:0/2045632266 >> 172.21.15.116:0/2045632266 conn(0x7f6e840c3b00 msgr2=0x7f6e840c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:10.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.934+0000 7f6e8bb9f640 1 -- 172.21.15.116:0/2045632266 shutdown_connections 2024-04-25T00:01:10.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.934+0000 7f6e8bb9f640 1 -- 172.21.15.116:0/2045632266 wait complete. 2024-04-25T00:01:10.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.934+0000 7f6e8bb9f640 1 Processor -- start 2024-04-25T00:01:10.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.934+0000 7f6e8bb9f640 1 -- start start 2024-04-25T00:01:10.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.934+0000 7f6e8bb9f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6e84071680 0x7f6e8406e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:10.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.934+0000 7f6e8bb9f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6e840a94a0 0x7f6e840ac9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:10.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.934+0000 7f6e8bb9f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e8414c720 0x7f6e840c5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:10.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.934+0000 7f6e8bb9f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6e840c3120 con 0x7f6e8414c720 2024-04-25T00:01:10.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.934+0000 7f6e8bb9f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6e84152f40 con 0x7f6e84071680 2024-04-25T00:01:10.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.934+0000 7f6e8bb9f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6e840c32a0 con 0x7f6e840a94a0 2024-04-25T00:01:10.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.934+0000 7f6e8a13c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e8414c720 0x7f6e840c5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:10.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.934+0000 7f6e8a13c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e8414c720 0x7f6e840c5140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37616/0 (socket says 172.21.15.116:37616) 2024-04-25T00:01:10.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.934+0000 7f6e8a13c640 1 -- 172.21.15.116:0/234947776 learned_addr learned my addr 172.21.15.116:0/234947776 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:10.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.934+0000 7f6e8993b640 1 --2- 172.21.15.116:0/234947776 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6e84071680 0x7f6e8406e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:10.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.934+0000 7f6e8913a640 1 --2- 172.21.15.116:0/234947776 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6e840a94a0 0x7f6e840ac9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:10.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.934+0000 7f6e8a13c640 1 -- 172.21.15.116:0/234947776 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6e840a94a0 msgr2=0x7f6e840ac9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:10.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.934+0000 7f6e8a13c640 1 --2- 172.21.15.116:0/234947776 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6e840a94a0 0x7f6e840ac9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:10.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.934+0000 7f6e8a13c640 1 -- 172.21.15.116:0/234947776 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6e84071680 msgr2=0x7f6e8406e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:10.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.934+0000 7f6e8a13c640 1 --2- 172.21.15.116:0/234947776 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6e84071680 0x7f6e8406e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:10.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.934+0000 7f6e8a13c640 1 -- 172.21.15.116:0/234947776 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e840c4c90 con 0x7f6e8414c720 2024-04-25T00:01:10.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.934+0000 7f6e8993b640 1 --2- 172.21.15.116:0/234947776 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6e84071680 0x7f6e8406e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:10.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.934+0000 7f6e8913a640 1 --2- 172.21.15.116:0/234947776 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6e840a94a0 0x7f6e840ac9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:10.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.934+0000 7f6e8a13c640 1 --2- 172.21.15.116:0/234947776 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e8414c720 0x7f6e840c5140 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c0ad040 tx=0x7f6e7c01ac60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:10.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.934+0000 7f6e827fc640 1 -- 172.21.15.116:0/234947776 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e7c0abcf0 con 0x7f6e8414c720 2024-04-25T00:01:10.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.934+0000 7f6e827fc640 1 -- 172.21.15.116:0/234947776 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e7c0be3f0 con 0x7f6e8414c720 2024-04-25T00:01:10.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.934+0000 7f6e8bb9f640 1 -- 172.21.15.116:0/234947776 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e84154f70 con 0x7f6e8414c720 2024-04-25T00:01:10.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.934+0000 7f6e827fc640 1 -- 172.21.15.116:0/234947776 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e7c0be7e0 con 0x7f6e8414c720 2024-04-25T00:01:10.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.934+0000 7f6e8bb9f640 1 -- 172.21.15.116:0/234947776 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e84155530 con 0x7f6e8414c720 2024-04-25T00:01:10.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.938+0000 7f6e8bb9f640 1 -- 172.21.15.116:0/234947776 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e84068350 con 0x7f6e8414c720 2024-04-25T00:01:10.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.938+0000 7f6e827fc640 1 -- 172.21.15.116:0/234947776 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6e7c0b43f0 con 0x7f6e8414c720 2024-04-25T00:01:10.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.938+0000 7f6e827fc640 1 --2- 172.21.15.116:0/234947776 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6e680678c0 0x7f6e68069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:10.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.938+0000 7f6e827fc640 1 -- 172.21.15.116:0/234947776 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5689+0+0 (secure 0 0 0) 0x7f6e7c11fda0 con 0x7f6e8414c720 2024-04-25T00:01:10.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.938+0000 7f6e8993b640 1 --2- 172.21.15.116:0/234947776 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6e680678c0 0x7f6e68069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:10.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.938+0000 7f6e8993b640 1 --2- 172.21.15.116:0/234947776 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6e680678c0 0x7f6e68069d80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6e70008890 tx=0x7f6e70008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:10.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:10.942+0000 7f6e827fc640 1 -- 172.21.15.116:0/234947776 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f6e7c0c4060 con 0x7f6e8414c720 2024-04-25T00:01:11.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.094+0000 7f6e8bb9f640 1 -- 172.21.15.116:0/234947776 --> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6e840c3640 con 0x7f6e680678c0 2024-04-25T00:01:11.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.110+0000 7f6e827fc640 1 -- 172.21.15.116:0/234947776 <== mgr.4105 v2:172.21.15.174:6832/2004448211 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251355 (secure 0 0 0) 0x7f6e840c3640 con 0x7f6e680678c0 2024-04-25T00:01:11.114 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:01:11.118 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-04-25T00:01:11.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.118+0000 7f6e8bb9f640 1 -- 172.21.15.116:0/234947776 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6e680678c0 msgr2=0x7f6e68069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:11.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.118+0000 7f6e8bb9f640 1 --2- 172.21.15.116:0/234947776 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6e680678c0 0x7f6e68069d80 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6e70008890 tx=0x7f6e70008040 comp rx=0 tx=0).stop 2024-04-25T00:01:11.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.118+0000 7f6e8bb9f640 1 -- 172.21.15.116:0/234947776 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e8414c720 msgr2=0x7f6e840c5140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:11.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.118+0000 7f6e8bb9f640 1 --2- 172.21.15.116:0/234947776 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e8414c720 0x7f6e840c5140 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c0ad040 tx=0x7f6e7c01ac60 comp rx=0 tx=0).stop 2024-04-25T00:01:11.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.118+0000 7f6e8bb9f640 1 -- 172.21.15.116:0/234947776 shutdown_connections 2024-04-25T00:01:11.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.118+0000 7f6e8bb9f640 1 --2- 172.21.15.116:0/234947776 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6e680678c0 0x7f6e68069d80 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:11.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.118+0000 7f6e8bb9f640 1 --2- 172.21.15.116:0/234947776 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e8414c720 0x7f6e840c5140 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:11.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.118+0000 7f6e8bb9f640 1 --2- 172.21.15.116:0/234947776 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6e840a94a0 0x7f6e840ac9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:11.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.118+0000 7f6e8bb9f640 1 --2- 172.21.15.116:0/234947776 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6e84071680 0x7f6e8406e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:11.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.118+0000 7f6e8bb9f640 1 -- 172.21.15.116:0/234947776 >> 172.21.15.116:0/234947776 conn(0x7f6e840c3b00 msgr2=0x7f6e84145d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:11.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.118+0000 7f6e8bb9f640 1 -- 172.21.15.116:0/234947776 shutdown_connections 2024-04-25T00:01:11.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.118+0000 7f6e8bb9f640 1 -- 172.21.15.116:0/234947776 wait complete. 2024-04-25T00:01:11.134 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-04-25T00:01:09.251353+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592196,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":184,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":141,"ondisk_log_size":141,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":181,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219136,"kb_used_data":2956,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755584,"statfs":{"total":773094113280,"available":772869718016,"internally_reserved":0,"allocated":3026944,"data_stored":1794407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":51,"apply_latency_ms":51,"commit_latency_ns":51000000,"apply_latency_ns":51000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1828,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.935044"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380670+0000","last_change":"2024-04-25T00:01:05.389596+0000","last_active":"2024-04-25T00:01:06.380670+0000","last_peered":"2024-04-25T00:01:06.380670+0000","last_clean":"2024-04-25T00:01:06.380670+0000","last_became_active":"2024-04-25T00:01:05.389127+0000","last_became_peered":"2024-04-25T00:01:05.389127+0000","last_unstale":"2024-04-25T00:01:06.380670+0000","last_undegraded":"2024-04-25T00:01:06.380670+0000","last_fullsized":"2024-04-25T00:01:06.380670+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:09.858617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.340220+0000","last_change":"2024-04-25T00:01:05.368190+0000","last_active":"2024-04-25T00:01:06.340220+0000","last_peered":"2024-04-25T00:01:06.340220+0000","last_clean":"2024-04-25T00:01:06.340220+0000","last_became_active":"2024-04-25T00:01:05.362606+0000","last_became_peered":"2024-04-25T00:01:05.362606+0000","last_unstale":"2024-04-25T00:01:06.340220+0000","last_undegraded":"2024-04-25T00:01:06.340220+0000","last_fullsized":"2024-04-25T00:01:06.340220+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:12:52.846250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.339987+0000","last_change":"2024-04-25T00:01:05.371291+0000","last_active":"2024-04-25T00:01:06.339987+0000","last_peered":"2024-04-25T00:01:06.339987+0000","last_clean":"2024-04-25T00:01:06.339987+0000","last_became_active":"2024-04-25T00:01:05.370522+0000","last_became_peered":"2024-04-25T00:01:05.370522+0000","last_unstale":"2024-04-25T00:01:06.339987+0000","last_undegraded":"2024-04-25T00:01:06.339987+0000","last_fullsized":"2024-04-25T00:01:06.339987+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:11:27.044893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.377499+0000","last_change":"2024-04-25T00:01:05.359716+0000","last_active":"2024-04-25T00:01:06.377499+0000","last_peered":"2024-04-25T00:01:06.377499+0000","last_clean":"2024-04-25T00:01:06.377499+0000","last_became_active":"2024-04-25T00:01:05.358866+0000","last_became_peered":"2024-04-25T00:01:05.358866+0000","last_unstale":"2024-04-25T00:01:06.377499+0000","last_undegraded":"2024-04-25T00:01:06.377499+0000","last_fullsized":"2024-04-25T00:01:06.377499+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:50.489125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380258+0000","last_change":"2024-04-25T00:01:05.386068+0000","last_active":"2024-04-25T00:01:06.380258+0000","last_peered":"2024-04-25T00:01:06.380258+0000","last_clean":"2024-04-25T00:01:06.380258+0000","last_became_active":"2024-04-25T00:01:05.378650+0000","last_became_peered":"2024-04-25T00:01:05.378650+0000","last_unstale":"2024-04-25T00:01:06.380258+0000","last_undegraded":"2024-04-25T00:01:06.380258+0000","last_fullsized":"2024-04-25T00:01:06.380258+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:12:52.846250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.377084+0000","last_change":"2024-04-25T00:01:05.359253+0000","last_active":"2024-04-25T00:01:06.377084+0000","last_peered":"2024-04-25T00:01:06.377084+0000","last_clean":"2024-04-25T00:01:06.377084+0000","last_became_active":"2024-04-25T00:01:05.358477+0000","last_became_peered":"2024-04-25T00:01:05.358477+0000","last_unstale":"2024-04-25T00:01:06.377084+0000","last_undegraded":"2024-04-25T00:01:06.377084+0000","last_fullsized":"2024-04-25T00:01:06.377084+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:22.739214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.379079+0000","last_change":"2024-04-25T00:01:05.374766+0000","last_active":"2024-04-25T00:01:06.379079+0000","last_peered":"2024-04-25T00:01:06.379079+0000","last_clean":"2024-04-25T00:01:06.379079+0000","last_became_active":"2024-04-25T00:01:05.369739+0000","last_became_peered":"2024-04-25T00:01:05.369739+0000","last_unstale":"2024-04-25T00:01:06.379079+0000","last_undegraded":"2024-04-25T00:01:06.379079+0000","last_fullsized":"2024-04-25T00:01:06.379079+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:09:18.315688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.339014+0000","last_change":"2024-04-25T00:01:05.390256+0000","last_active":"2024-04-25T00:01:06.339014+0000","last_peered":"2024-04-25T00:01:06.339014+0000","last_clean":"2024-04-25T00:01:06.339014+0000","last_became_active":"2024-04-25T00:01:05.389724+0000","last_became_peered":"2024-04-25T00:01:05.389724+0000","last_unstale":"2024-04-25T00:01:06.339014+0000","last_undegraded":"2024-04-25T00:01:06.339014+0000","last_fullsized":"2024-04-25T00:01:06.339014+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:09:18.315688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.379000+0000","last_change":"2024-04-25T00:01:05.362303+0000","last_active":"2024-04-25T00:01:06.379000+0000","last_peered":"2024-04-25T00:01:06.379000+0000","last_clean":"2024-04-25T00:01:06.379000+0000","last_became_active":"2024-04-25T00:01:05.361586+0000","last_became_peered":"2024-04-25T00:01:05.361586+0000","last_unstale":"2024-04-25T00:01:06.379000+0000","last_undegraded":"2024-04-25T00:01:06.379000+0000","last_fullsized":"2024-04-25T00:01:06.379000+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:30.228060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380324+0000","last_change":"2024-04-25T00:01:05.368974+0000","last_active":"2024-04-25T00:01:06.380324+0000","last_peered":"2024-04-25T00:01:06.380324+0000","last_clean":"2024-04-25T00:01:06.380324+0000","last_became_active":"2024-04-25T00:01:05.367893+0000","last_became_peered":"2024-04-25T00:01:05.367893+0000","last_unstale":"2024-04-25T00:01:06.380324+0000","last_undegraded":"2024-04-25T00:01:06.380324+0000","last_fullsized":"2024-04-25T00:01:06.380324+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:30:23.557381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380371+0000","last_change":"2024-04-25T00:01:05.389424+0000","last_active":"2024-04-25T00:01:06.380371+0000","last_peered":"2024-04-25T00:01:06.380371+0000","last_clean":"2024-04-25T00:01:06.380371+0000","last_became_active":"2024-04-25T00:01:05.388999+0000","last_became_peered":"2024-04-25T00:01:05.388999+0000","last_unstale":"2024-04-25T00:01:06.380371+0000","last_undegraded":"2024-04-25T00:01:06.380371+0000","last_fullsized":"2024-04-25T00:01:06.380371+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:31.456792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.379890+0000","last_change":"2024-04-25T00:01:05.385853+0000","last_active":"2024-04-25T00:01:06.379890+0000","last_peered":"2024-04-25T00:01:06.379890+0000","last_clean":"2024-04-25T00:01:06.379890+0000","last_became_active":"2024-04-25T00:01:05.384997+0000","last_became_peered":"2024-04-25T00:01:05.384997+0000","last_unstale":"2024-04-25T00:01:06.379890+0000","last_undegraded":"2024-04-25T00:01:06.379890+0000","last_fullsized":"2024-04-25T00:01:06.379890+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:36:55.436661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.339127+0000","last_change":"2024-04-25T00:01:05.381011+0000","last_active":"2024-04-25T00:01:06.339127+0000","last_peered":"2024-04-25T00:01:06.339127+0000","last_clean":"2024-04-25T00:01:06.339127+0000","last_became_active":"2024-04-25T00:01:05.380236+0000","last_became_peered":"2024-04-25T00:01:05.380236+0000","last_unstale":"2024-04-25T00:01:06.339127+0000","last_undegraded":"2024-04-25T00:01:06.339127+0000","last_fullsized":"2024-04-25T00:01:06.339127+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:06:19.307378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.337813+0000","last_change":"2024-04-25T00:01:05.378373+0000","last_active":"2024-04-25T00:01:06.337813+0000","last_peered":"2024-04-25T00:01:06.337813+0000","last_clean":"2024-04-25T00:01:06.337813+0000","last_became_active":"2024-04-25T00:01:05.376570+0000","last_became_peered":"2024-04-25T00:01:05.376570+0000","last_unstale":"2024-04-25T00:01:06.337813+0000","last_undegraded":"2024-04-25T00:01:06.337813+0000","last_fullsized":"2024-04-25T00:01:06.337813+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:30.228060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.337758+0000","last_change":"2024-04-25T00:01:05.390341+0000","last_active":"2024-04-25T00:01:06.337758+0000","last_peered":"2024-04-25T00:01:06.337758+0000","last_clean":"2024-04-25T00:01:06.337758+0000","last_became_active":"2024-04-25T00:01:05.389707+0000","last_became_peered":"2024-04-25T00:01:05.389707+0000","last_unstale":"2024-04-25T00:01:06.337758+0000","last_undegraded":"2024-04-25T00:01:06.337758+0000","last_fullsized":"2024-04-25T00:01:06.337758+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:54:21.138947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.335984+0000","last_change":"2024-04-25T00:01:05.370826+0000","last_active":"2024-04-25T00:01:06.335984+0000","last_peered":"2024-04-25T00:01:06.335984+0000","last_clean":"2024-04-25T00:01:06.335984+0000","last_became_active":"2024-04-25T00:01:05.369848+0000","last_became_peered":"2024-04-25T00:01:05.369848+0000","last_unstale":"2024-04-25T00:01:06.335984+0000","last_undegraded":"2024-04-25T00:01:06.335984+0000","last_fullsized":"2024-04-25T00:01:06.335984+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:23.870121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.337683+0000","last_change":"2024-04-25T00:01:05.379958+0000","last_active":"2024-04-25T00:01:06.337683+0000","last_peered":"2024-04-25T00:01:06.337683+0000","last_clean":"2024-04-25T00:01:06.337683+0000","last_became_active":"2024-04-25T00:01:05.379090+0000","last_became_peered":"2024-04-25T00:01:05.379090+0000","last_unstale":"2024-04-25T00:01:06.337683+0000","last_undegraded":"2024-04-25T00:01:06.337683+0000","last_fullsized":"2024-04-25T00:01:06.337683+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:59:50.832958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.379255+0000","last_change":"2024-04-25T00:01:05.367001+0000","last_active":"2024-04-25T00:01:06.379255+0000","last_peered":"2024-04-25T00:01:06.379255+0000","last_clean":"2024-04-25T00:01:06.379255+0000","last_became_active":"2024-04-25T00:01:05.366264+0000","last_became_peered":"2024-04-25T00:01:05.366264+0000","last_unstale":"2024-04-25T00:01:06.379255+0000","last_undegraded":"2024-04-25T00:01:06.379255+0000","last_fullsized":"2024-04-25T00:01:06.379255+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.337778+0000","last_change":"2024-04-25T00:01:05.370653+0000","last_active":"2024-04-25T00:01:06.337778+0000","last_peered":"2024-04-25T00:01:06.337778+0000","last_clean":"2024-04-25T00:01:06.337778+0000","last_became_active":"2024-04-25T00:01:05.369919+0000","last_became_peered":"2024-04-25T00:01:05.369919+0000","last_unstale":"2024-04-25T00:01:06.337778+0000","last_undegraded":"2024-04-25T00:01:06.337778+0000","last_fullsized":"2024-04-25T00:01:06.337778+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:30:08.311326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380032+0000","last_change":"2024-04-25T00:01:05.385253+0000","last_active":"2024-04-25T00:01:06.380032+0000","last_peered":"2024-04-25T00:01:06.380032+0000","last_clean":"2024-04-25T00:01:06.380032+0000","last_became_active":"2024-04-25T00:01:05.379429+0000","last_became_peered":"2024-04-25T00:01:05.379429+0000","last_unstale":"2024-04-25T00:01:06.380032+0000","last_undegraded":"2024-04-25T00:01:06.380032+0000","last_fullsized":"2024-04-25T00:01:06.380032+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:36:10.377757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":1,"reported_epoch":21,"state":"creating+peering","last_fresh":"2024-04-25T00:01:04.338341+0000","last_change":"2024-04-25T00:01:04.338341+0000","last_active":"2024-04-25T00:01:04.314940+0000","last_peered":"2024-04-25T00:01:04.314940+0000","last_clean":"2024-04-25T00:01:04.314940+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T00:01:04.338341+0000","last_undegraded":"2024-04-25T00:01:04.338341+0000","last_fullsized":"2024-04-25T00:01:04.338341+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378876+0000","last_change":"2024-04-25T00:01:05.387457+0000","last_active":"2024-04-25T00:01:06.378876+0000","last_peered":"2024-04-25T00:01:06.378876+0000","last_clean":"2024-04-25T00:01:06.378876+0000","last_became_active":"2024-04-25T00:01:05.386919+0000","last_became_peered":"2024-04-25T00:01:05.386919+0000","last_unstale":"2024-04-25T00:01:06.378876+0000","last_undegraded":"2024-04-25T00:01:06.378876+0000","last_fullsized":"2024-04-25T00:01:06.378876+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.338263+0000","last_change":"2024-04-25T00:01:05.366340+0000","last_active":"2024-04-25T00:01:06.338263+0000","last_peered":"2024-04-25T00:01:06.338263+0000","last_clean":"2024-04-25T00:01:06.338263+0000","last_became_active":"2024-04-25T00:01:05.361952+0000","last_became_peered":"2024-04-25T00:01:05.361952+0000","last_unstale":"2024-04-25T00:01:06.338263+0000","last_undegraded":"2024-04-25T00:01:06.338263+0000","last_fullsized":"2024-04-25T00:01:06.338263+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:41.948662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.335946+0000","last_change":"2024-04-25T00:01:05.372906+0000","last_active":"2024-04-25T00:01:06.335946+0000","last_peered":"2024-04-25T00:01:06.335946+0000","last_clean":"2024-04-25T00:01:06.335946+0000","last_became_active":"2024-04-25T00:01:05.372148+0000","last_became_peered":"2024-04-25T00:01:05.372148+0000","last_unstale":"2024-04-25T00:01:06.335946+0000","last_undegraded":"2024-04-25T00:01:06.335946+0000","last_fullsized":"2024-04-25T00:01:06.335946+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:43.615895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.340447+0000","last_change":"2024-04-25T00:01:05.373464+0000","last_active":"2024-04-25T00:01:06.340447+0000","last_peered":"2024-04-25T00:01:06.340447+0000","last_clean":"2024-04-25T00:01:06.340447+0000","last_became_active":"2024-04-25T00:01:05.372611+0000","last_became_peered":"2024-04-25T00:01:05.372611+0000","last_unstale":"2024-04-25T00:01:06.340447+0000","last_undegraded":"2024-04-25T00:01:06.340447+0000","last_fullsized":"2024-04-25T00:01:06.340447+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:31.456792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378208+0000","last_change":"2024-04-25T00:01:05.365761+0000","last_active":"2024-04-25T00:01:06.378208+0000","last_peered":"2024-04-25T00:01:06.378208+0000","last_clean":"2024-04-25T00:01:06.378208+0000","last_became_active":"2024-04-25T00:01:05.361119+0000","last_became_peered":"2024-04-25T00:01:05.361119+0000","last_unstale":"2024-04-25T00:01:06.378208+0000","last_undegraded":"2024-04-25T00:01:06.378208+0000","last_fullsized":"2024-04-25T00:01:06.378208+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.341008+0000","last_change":"2024-04-25T00:01:05.367296+0000","last_active":"2024-04-25T00:01:06.341008+0000","last_peered":"2024-04-25T00:01:06.341008+0000","last_clean":"2024-04-25T00:01:06.341008+0000","last_became_active":"2024-04-25T00:01:05.363838+0000","last_became_peered":"2024-04-25T00:01:05.363838+0000","last_unstale":"2024-04-25T00:01:06.341008+0000","last_undegraded":"2024-04-25T00:01:06.341008+0000","last_fullsized":"2024-04-25T00:01:06.341008+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:43:07.053074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.339622+0000","last_change":"2024-04-25T00:01:05.387057+0000","last_active":"2024-04-25T00:01:06.339622+0000","last_peered":"2024-04-25T00:01:06.339622+0000","last_clean":"2024-04-25T00:01:06.339622+0000","last_became_active":"2024-04-25T00:01:05.386724+0000","last_became_peered":"2024-04-25T00:01:05.386724+0000","last_unstale":"2024-04-25T00:01:06.339622+0000","last_undegraded":"2024-04-25T00:01:06.339622+0000","last_fullsized":"2024-04-25T00:01:06.339622+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:55:09.073286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.339053+0000","last_change":"2024-04-25T00:01:05.358690+0000","last_active":"2024-04-25T00:01:06.339053+0000","last_peered":"2024-04-25T00:01:06.339053+0000","last_clean":"2024-04-25T00:01:06.339053+0000","last_became_active":"2024-04-25T00:01:05.357340+0000","last_became_peered":"2024-04-25T00:01:05.357340+0000","last_unstale":"2024-04-25T00:01:06.339053+0000","last_undegraded":"2024-04-25T00:01:06.339053+0000","last_fullsized":"2024-04-25T00:01:06.339053+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:36:55.436661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.336324+0000","last_change":"2024-04-25T00:01:05.362960+0000","last_active":"2024-04-25T00:01:06.336324+0000","last_peered":"2024-04-25T00:01:06.336324+0000","last_clean":"2024-04-25T00:01:06.336324+0000","last_became_active":"2024-04-25T00:01:05.361067+0000","last_became_peered":"2024-04-25T00:01:05.361067+0000","last_unstale":"2024-04-25T00:01:06.336324+0000","last_undegraded":"2024-04-25T00:01:06.336324+0000","last_fullsized":"2024-04-25T00:01:06.336324+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:24.017263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.377980+0000","last_change":"2024-04-25T00:01:05.372927+0000","last_active":"2024-04-25T00:01:06.377980+0000","last_peered":"2024-04-25T00:01:06.377980+0000","last_clean":"2024-04-25T00:01:06.377980+0000","last_became_active":"2024-04-25T00:01:05.372050+0000","last_became_peered":"2024-04-25T00:01:05.372050+0000","last_unstale":"2024-04-25T00:01:06.377980+0000","last_undegraded":"2024-04-25T00:01:06.377980+0000","last_fullsized":"2024-04-25T00:01:06.377980+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:48.202251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.376574+0000","last_change":"2024-04-25T00:01:05.378517+0000","last_active":"2024-04-25T00:01:06.376574+0000","last_peered":"2024-04-25T00:01:06.376574+0000","last_clean":"2024-04-25T00:01:06.376574+0000","last_became_active":"2024-04-25T00:01:05.377742+0000","last_became_peered":"2024-04-25T00:01:05.377742+0000","last_unstale":"2024-04-25T00:01:06.376574+0000","last_undegraded":"2024-04-25T00:01:06.376574+0000","last_fullsized":"2024-04-25T00:01:06.376574+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:06.450369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.379065+0000","last_change":"2024-04-25T00:01:05.375142+0000","last_active":"2024-04-25T00:01:06.379065+0000","last_peered":"2024-04-25T00:01:06.379065+0000","last_clean":"2024-04-25T00:01:06.379065+0000","last_became_active":"2024-04-25T00:01:05.373857+0000","last_became_peered":"2024-04-25T00:01:05.373857+0000","last_unstale":"2024-04-25T00:01:06.379065+0000","last_undegraded":"2024-04-25T00:01:06.379065+0000","last_fullsized":"2024-04-25T00:01:06.379065+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:15.879673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"22'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.339475+0000","last_change":"2024-04-25T00:01:04.365607+0000","last_active":"2024-04-25T00:01:06.339475+0000","last_peered":"2024-04-25T00:01:06.339475+0000","last_clean":"2024-04-25T00:01:06.339475+0000","last_became_active":"2024-04-25T00:01:04.364788+0000","last_became_peered":"2024-04-25T00:01:04.364788+0000","last_unstale":"2024-04-25T00:01:06.339475+0000","last_undegraded":"2024-04-25T00:01:06.339475+0000","last_fullsized":"2024-04-25T00:01:06.339475+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":1,"reported_epoch":21,"state":"creating+peering","last_fresh":"2024-04-25T00:01:04.333513+0000","last_change":"2024-04-25T00:01:04.333513+0000","last_active":"2024-04-25T00:01:04.314940+0000","last_peered":"2024-04-25T00:01:04.314940+0000","last_clean":"2024-04-25T00:01:04.314940+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T00:01:04.333513+0000","last_undegraded":"2024-04-25T00:01:04.333513+0000","last_fullsized":"2024-04-25T00:01:04.333513+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"22'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.336967+0000","last_change":"2024-04-25T00:01:04.360254+0000","last_active":"2024-04-25T00:01:06.336967+0000","last_peered":"2024-04-25T00:01:06.336967+0000","last_clean":"2024-04-25T00:01:06.336967+0000","last_became_active":"2024-04-25T00:01:04.359586+0000","last_became_peered":"2024-04-25T00:01:04.359586+0000","last_unstale":"2024-04-25T00:01:06.336967+0000","last_undegraded":"2024-04-25T00:01:06.336967+0000","last_fullsized":"2024-04-25T00:01:06.336967+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:47.192930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.337866+0000","last_change":"2024-04-25T00:01:04.370896+0000","last_active":"2024-04-25T00:01:06.337866+0000","last_peered":"2024-04-25T00:01:06.337866+0000","last_clean":"2024-04-25T00:01:06.337866+0000","last_became_active":"2024-04-25T00:01:04.370273+0000","last_became_peered":"2024-04-25T00:01:04.370273+0000","last_unstale":"2024-04-25T00:01:06.337866+0000","last_undegraded":"2024-04-25T00:01:06.337866+0000","last_fullsized":"2024-04-25T00:01:06.337866+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:49:22.673421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.339542+0000","last_change":"2024-04-25T00:01:05.378867+0000","last_active":"2024-04-25T00:01:06.339542+0000","last_peered":"2024-04-25T00:01:06.339542+0000","last_clean":"2024-04-25T00:01:06.339542+0000","last_became_active":"2024-04-25T00:01:05.378056+0000","last_became_peered":"2024-04-25T00:01:05.378056+0000","last_unstale":"2024-04-25T00:01:06.339542+0000","last_undegraded":"2024-04-25T00:01:06.339542+0000","last_fullsized":"2024-04-25T00:01:06.339542+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:36:10.377757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.336193+0000","last_change":"2024-04-25T00:01:04.364523+0000","last_active":"2024-04-25T00:01:06.336193+0000","last_peered":"2024-04-25T00:01:06.336193+0000","last_clean":"2024-04-25T00:01:06.336193+0000","last_became_active":"2024-04-25T00:01:04.364050+0000","last_became_peered":"2024-04-25T00:01:04.364050+0000","last_unstale":"2024-04-25T00:01:06.336193+0000","last_undegraded":"2024-04-25T00:01:06.336193+0000","last_fullsized":"2024-04-25T00:01:06.336193+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:49.479804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":1,"reported_epoch":21,"state":"creating+peering","last_fresh":"2024-04-25T00:01:04.336083+0000","last_change":"2024-04-25T00:01:04.336083+0000","last_active":"2024-04-25T00:01:04.314940+0000","last_peered":"2024-04-25T00:01:04.314940+0000","last_clean":"2024-04-25T00:01:04.314940+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T00:01:04.336083+0000","last_undegraded":"2024-04-25T00:01:04.336083+0000","last_fullsized":"2024-04-25T00:01:04.336083+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.379231+0000","last_change":"2024-04-25T00:01:04.360804+0000","last_active":"2024-04-25T00:01:06.379231+0000","last_peered":"2024-04-25T00:01:06.379231+0000","last_clean":"2024-04-25T00:01:06.379231+0000","last_became_active":"2024-04-25T00:01:04.360374+0000","last_became_peered":"2024-04-25T00:01:04.360374+0000","last_unstale":"2024-04-25T00:01:06.379231+0000","last_undegraded":"2024-04-25T00:01:06.379231+0000","last_fullsized":"2024-04-25T00:01:06.379231+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378559+0000","last_change":"2024-04-25T00:01:05.365413+0000","last_active":"2024-04-25T00:01:06.378559+0000","last_peered":"2024-04-25T00:01:06.378559+0000","last_clean":"2024-04-25T00:01:06.378559+0000","last_became_active":"2024-04-25T00:01:05.361865+0000","last_became_peered":"2024-04-25T00:01:05.361865+0000","last_unstale":"2024-04-25T00:01:06.378559+0000","last_undegraded":"2024-04-25T00:01:06.378559+0000","last_fullsized":"2024-04-25T00:01:06.378559+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:30.228060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380688+0000","last_change":"2024-04-25T00:01:04.362129+0000","last_active":"2024-04-25T00:01:06.380688+0000","last_peered":"2024-04-25T00:01:06.380688+0000","last_clean":"2024-04-25T00:01:06.380688+0000","last_became_active":"2024-04-25T00:01:04.361827+0000","last_became_peered":"2024-04-25T00:01:04.361827+0000","last_unstale":"2024-04-25T00:01:06.380688+0000","last_undegraded":"2024-04-25T00:01:06.380688+0000","last_fullsized":"2024-04-25T00:01:06.380688+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:26.243635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.340801+0000","last_change":"2024-04-25T00:01:05.361723+0000","last_active":"2024-04-25T00:01:06.340801+0000","last_peered":"2024-04-25T00:01:06.340801+0000","last_clean":"2024-04-25T00:01:06.340801+0000","last_became_active":"2024-04-25T00:01:05.360956+0000","last_became_peered":"2024-04-25T00:01:05.360956+0000","last_unstale":"2024-04-25T00:01:06.340801+0000","last_undegraded":"2024-04-25T00:01:06.340801+0000","last_fullsized":"2024-04-25T00:01:06.340801+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:23.870121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"22'3","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.379142+0000","last_change":"2024-04-25T00:01:04.366503+0000","last_active":"2024-04-25T00:01:06.379142+0000","last_peered":"2024-04-25T00:01:06.379142+0000","last_clean":"2024-04-25T00:01:06.379142+0000","last_became_active":"2024-04-25T00:01:04.365945+0000","last_became_peered":"2024-04-25T00:01:04.365945+0000","last_unstale":"2024-04-25T00:01:06.379142+0000","last_undegraded":"2024-04-25T00:01:06.379142+0000","last_fullsized":"2024-04-25T00:01:06.379142+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.336525+0000","last_change":"2024-04-25T00:01:05.371061+0000","last_active":"2024-04-25T00:01:06.336525+0000","last_peered":"2024-04-25T00:01:06.336525+0000","last_clean":"2024-04-25T00:01:06.336525+0000","last_became_active":"2024-04-25T00:01:05.369950+0000","last_became_peered":"2024-04-25T00:01:05.369950+0000","last_unstale":"2024-04-25T00:01:06.336525+0000","last_undegraded":"2024-04-25T00:01:06.336525+0000","last_fullsized":"2024-04-25T00:01:06.336525+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:02:43.237902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"22'6","reported_seq":20,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.377439+0000","last_change":"2024-04-25T00:01:04.365185+0000","last_active":"2024-04-25T00:01:06.377439+0000","last_peered":"2024-04-25T00:01:06.377439+0000","last_clean":"2024-04-25T00:01:06.377439+0000","last_became_active":"2024-04-25T00:01:04.364930+0000","last_became_peered":"2024-04-25T00:01:04.364930+0000","last_unstale":"2024-04-25T00:01:06.377439+0000","last_undegraded":"2024-04-25T00:01:06.377439+0000","last_fullsized":"2024-04-25T00:01:06.377439+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:45:03.618340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380617+0000","last_change":"2024-04-25T00:01:05.385827+0000","last_active":"2024-04-25T00:01:06.380617+0000","last_peered":"2024-04-25T00:01:06.380617+0000","last_clean":"2024-04-25T00:01:06.380617+0000","last_became_active":"2024-04-25T00:01:05.385258+0000","last_became_peered":"2024-04-25T00:01:05.385258+0000","last_unstale":"2024-04-25T00:01:06.380617+0000","last_undegraded":"2024-04-25T00:01:06.380617+0000","last_fullsized":"2024-04-25T00:01:06.380617+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:23.870121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"22'2","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378778+0000","last_change":"2024-04-25T00:01:04.370640+0000","last_active":"2024-04-25T00:01:06.378778+0000","last_peered":"2024-04-25T00:01:06.378778+0000","last_clean":"2024-04-25T00:01:06.378778+0000","last_became_active":"2024-04-25T00:01:04.370031+0000","last_became_peered":"2024-04-25T00:01:04.370031+0000","last_unstale":"2024-04-25T00:01:06.378778+0000","last_undegraded":"2024-04-25T00:01:06.378778+0000","last_fullsized":"2024-04-25T00:01:06.378778+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:26.243635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.338234+0000","last_change":"2024-04-25T00:01:05.379102+0000","last_active":"2024-04-25T00:01:06.338234+0000","last_peered":"2024-04-25T00:01:06.338234+0000","last_clean":"2024-04-25T00:01:06.338234+0000","last_became_active":"2024-04-25T00:01:05.377625+0000","last_became_peered":"2024-04-25T00:01:05.377625+0000","last_unstale":"2024-04-25T00:01:06.338234+0000","last_undegraded":"2024-04-25T00:01:06.338234+0000","last_fullsized":"2024-04-25T00:01:06.338234+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:15.879673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.337539+0000","last_change":"2024-04-25T00:01:04.360987+0000","last_active":"2024-04-25T00:01:06.337539+0000","last_peered":"2024-04-25T00:01:06.337539+0000","last_clean":"2024-04-25T00:01:06.337539+0000","last_became_active":"2024-04-25T00:01:04.360664+0000","last_became_peered":"2024-04-25T00:01:04.360664+0000","last_unstale":"2024-04-25T00:01:06.337539+0000","last_undegraded":"2024-04-25T00:01:06.337539+0000","last_fullsized":"2024-04-25T00:01:06.337539+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:55.283076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.336555+0000","last_change":"2024-04-25T00:01:05.379501+0000","last_active":"2024-04-25T00:01:06.336555+0000","last_peered":"2024-04-25T00:01:06.336555+0000","last_clean":"2024-04-25T00:01:06.336555+0000","last_became_active":"2024-04-25T00:01:05.378644+0000","last_became_peered":"2024-04-25T00:01:05.378644+0000","last_unstale":"2024-04-25T00:01:06.336555+0000","last_undegraded":"2024-04-25T00:01:06.336555+0000","last_fullsized":"2024-04-25T00:01:06.336555+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:08:35.505154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.379209+0000","last_change":"2024-04-25T00:01:04.357265+0000","last_active":"2024-04-25T00:01:06.379209+0000","last_peered":"2024-04-25T00:01:06.379209+0000","last_clean":"2024-04-25T00:01:06.379209+0000","last_became_active":"2024-04-25T00:01:04.357004+0000","last_became_peered":"2024-04-25T00:01:04.357004+0000","last_unstale":"2024-04-25T00:01:06.379209+0000","last_undegraded":"2024-04-25T00:01:06.379209+0000","last_fullsized":"2024-04-25T00:01:06.379209+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:47:25.595275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378119+0000","last_change":"2024-04-25T00:01:05.373927+0000","last_active":"2024-04-25T00:01:06.378119+0000","last_peered":"2024-04-25T00:01:06.378119+0000","last_clean":"2024-04-25T00:01:06.378119+0000","last_became_active":"2024-04-25T00:01:05.373077+0000","last_became_peered":"2024-04-25T00:01:05.373077+0000","last_unstale":"2024-04-25T00:01:06.378119+0000","last_undegraded":"2024-04-25T00:01:06.378119+0000","last_fullsized":"2024-04-25T00:01:06.378119+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:50.489125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-25T00:01:04.363964+0000","last_change":"2024-04-25T00:01:04.358365+0000","last_active":"2024-04-25T00:01:04.363964+0000","last_peered":"2024-04-25T00:01:04.363964+0000","last_clean":"2024-04-25T00:01:04.363964+0000","last_became_active":"2024-04-25T00:01:04.357571+0000","last_became_peered":"2024-04-25T00:01:04.357571+0000","last_unstale":"2024-04-25T00:01:04.363964+0000","last_undegraded":"2024-04-25T00:01:04.363964+0000","last_fullsized":"2024-04-25T00:01:04.363964+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:11:06.373584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.335878+0000","last_change":"2024-04-25T00:01:05.364103+0000","last_active":"2024-04-25T00:01:06.335878+0000","last_peered":"2024-04-25T00:01:06.335878+0000","last_clean":"2024-04-25T00:01:06.335878+0000","last_became_active":"2024-04-25T00:01:05.363320+0000","last_became_peered":"2024-04-25T00:01:05.363320+0000","last_unstale":"2024-04-25T00:01:06.335878+0000","last_undegraded":"2024-04-25T00:01:06.335878+0000","last_fullsized":"2024-04-25T00:01:06.335878+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:31.456792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"22'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380600+0000","last_change":"2024-04-25T00:01:04.372376+0000","last_active":"2024-04-25T00:01:06.380600+0000","last_peered":"2024-04-25T00:01:06.380600+0000","last_clean":"2024-04-25T00:01:06.380600+0000","last_became_active":"2024-04-25T00:01:04.372040+0000","last_became_peered":"2024-04-25T00:01:04.372040+0000","last_unstale":"2024-04-25T00:01:06.380600+0000","last_undegraded":"2024-04-25T00:01:06.380600+0000","last_fullsized":"2024-04-25T00:01:06.380600+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:21.729893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378915+0000","last_change":"2024-04-25T00:01:05.372591+0000","last_active":"2024-04-25T00:01:06.378915+0000","last_peered":"2024-04-25T00:01:06.378915+0000","last_clean":"2024-04-25T00:01:06.378915+0000","last_became_active":"2024-04-25T00:01:05.371204+0000","last_became_peered":"2024-04-25T00:01:05.371204+0000","last_unstale":"2024-04-25T00:01:06.378915+0000","last_undegraded":"2024-04-25T00:01:06.378915+0000","last_fullsized":"2024-04-25T00:01:06.378915+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:15.879673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-25T00:01:04.373291+0000","last_change":"2024-04-25T00:01:04.370794+0000","last_active":"2024-04-25T00:01:04.373291+0000","last_peered":"2024-04-25T00:01:04.373291+0000","last_clean":"2024-04-25T00:01:04.373291+0000","last_became_active":"2024-04-25T00:01:04.370099+0000","last_became_peered":"2024-04-25T00:01:04.370099+0000","last_unstale":"2024-04-25T00:01:04.373291+0000","last_undegraded":"2024-04-25T00:01:04.373291+0000","last_fullsized":"2024-04-25T00:01:04.373291+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:27:27.293170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380496+0000","last_change":"2024-04-25T00:01:05.378840+0000","last_active":"2024-04-25T00:01:06.380496+0000","last_peered":"2024-04-25T00:01:06.380496+0000","last_clean":"2024-04-25T00:01:06.380496+0000","last_became_active":"2024-04-25T00:01:05.378112+0000","last_became_peered":"2024-04-25T00:01:05.378112+0000","last_unstale":"2024-04-25T00:01:06.380496+0000","last_undegraded":"2024-04-25T00:01:06.380496+0000","last_fullsized":"2024-04-25T00:01:06.380496+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:43.615895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"22'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.340664+0000","last_change":"2024-04-25T00:01:04.356627+0000","last_active":"2024-04-25T00:01:06.340664+0000","last_peered":"2024-04-25T00:01:06.340664+0000","last_clean":"2024-04-25T00:01:06.340664+0000","last_became_active":"2024-04-25T00:01:04.354419+0000","last_became_peered":"2024-04-25T00:01:04.354419+0000","last_unstale":"2024-04-25T00:01:06.340664+0000","last_undegraded":"2024-04-25T00:01:06.340664+0000","last_fullsized":"2024-04-25T00:01:06.340664+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:26.243635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.381029+0000","last_change":"2024-04-25T00:01:05.385670+0000","last_active":"2024-04-25T00:01:06.381029+0000","last_peered":"2024-04-25T00:01:06.381029+0000","last_clean":"2024-04-25T00:01:06.381029+0000","last_became_active":"2024-04-25T00:01:05.384822+0000","last_became_peered":"2024-04-25T00:01:05.384822+0000","last_unstale":"2024-04-25T00:01:06.381029+0000","last_undegraded":"2024-04-25T00:01:06.381029+0000","last_fullsized":"2024-04-25T00:01:06.381029+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:24.017263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.340457+0000","last_change":"2024-04-25T00:01:04.368954+0000","last_active":"2024-04-25T00:01:06.340457+0000","last_peered":"2024-04-25T00:01:06.340457+0000","last_clean":"2024-04-25T00:01:06.340457+0000","last_became_active":"2024-04-25T00:01:04.367903+0000","last_became_peered":"2024-04-25T00:01:04.367903+0000","last_unstale":"2024-04-25T00:01:06.340457+0000","last_undegraded":"2024-04-25T00:01:06.340457+0000","last_fullsized":"2024-04-25T00:01:06.340457+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.337227+0000","last_change":"2024-04-25T00:01:05.376609+0000","last_active":"2024-04-25T00:01:06.337227+0000","last_peered":"2024-04-25T00:01:06.337227+0000","last_clean":"2024-04-25T00:01:06.337227+0000","last_became_active":"2024-04-25T00:01:05.375881+0000","last_became_peered":"2024-04-25T00:01:05.375881+0000","last_unstale":"2024-04-25T00:01:06.337227+0000","last_undegraded":"2024-04-25T00:01:06.337227+0000","last_fullsized":"2024-04-25T00:01:06.337227+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:46:05.592233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":170,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378246+0000","last_change":"2024-04-25T00:00:55.084492+0000","last_active":"2024-04-25T00:01:06.378246+0000","last_peered":"2024-04-25T00:01:06.378246+0000","last_clean":"2024-04-25T00:01:06.378246+0000","last_became_active":"2024-04-25T00:00:55.083493+0000","last_became_peered":"2024-04-25T00:00:55.083493+0000","last_unstale":"2024-04-25T00:01:06.378246+0000","last_undegraded":"2024-04-25T00:01:06.378246+0000","last_fullsized":"2024-04-25T00:01:06.378246+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:00:54.060578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:00:54.060578+0000","last_clean_scrub_stamp":"2024-04-25T00:00:54.060578+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:14:58.472477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.336858+0000","last_change":"2024-04-25T00:01:04.357028+0000","last_active":"2024-04-25T00:01:06.336858+0000","last_peered":"2024-04-25T00:01:06.336858+0000","last_clean":"2024-04-25T00:01:06.336858+0000","last_became_active":"2024-04-25T00:01:04.356644+0000","last_became_peered":"2024-04-25T00:01:04.356644+0000","last_unstale":"2024-04-25T00:01:06.336858+0000","last_undegraded":"2024-04-25T00:01:06.336858+0000","last_fullsized":"2024-04-25T00:01:06.336858+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:21.729893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1,"reported_epoch":21,"state":"creating+peering","last_fresh":"2024-04-25T00:01:04.333098+0000","last_change":"2024-04-25T00:01:04.333098+0000","last_active":"2024-04-25T00:01:04.314940+0000","last_peered":"2024-04-25T00:01:04.314940+0000","last_clean":"2024-04-25T00:01:04.314940+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T00:01:04.333098+0000","last_undegraded":"2024-04-25T00:01:04.333098+0000","last_fullsized":"2024-04-25T00:01:04.333098+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.336259+0000","last_change":"2024-04-25T00:01:04.370226+0000","last_active":"2024-04-25T00:01:06.336259+0000","last_peered":"2024-04-25T00:01:06.336259+0000","last_clean":"2024-04-25T00:01:06.336259+0000","last_became_active":"2024-04-25T00:01:04.369904+0000","last_became_peered":"2024-04-25T00:01:04.369904+0000","last_unstale":"2024-04-25T00:01:06.336259+0000","last_undegraded":"2024-04-25T00:01:06.336259+0000","last_fullsized":"2024-04-25T00:01:06.336259+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:47:25.595275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.337599+0000","last_change":"2024-04-25T00:01:05.370968+0000","last_active":"2024-04-25T00:01:06.337599+0000","last_peered":"2024-04-25T00:01:06.337599+0000","last_clean":"2024-04-25T00:01:06.337599+0000","last_became_active":"2024-04-25T00:01:05.369987+0000","last_became_peered":"2024-04-25T00:01:05.369987+0000","last_unstale":"2024-04-25T00:01:06.337599+0000","last_undegraded":"2024-04-25T00:01:06.337599+0000","last_fullsized":"2024-04-25T00:01:06.337599+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.377854+0000","last_change":"2024-04-25T00:01:04.372916+0000","last_active":"2024-04-25T00:01:06.377854+0000","last_peered":"2024-04-25T00:01:06.377854+0000","last_clean":"2024-04-25T00:01:06.377854+0000","last_became_active":"2024-04-25T00:01:04.372185+0000","last_became_peered":"2024-04-25T00:01:04.372185+0000","last_unstale":"2024-04-25T00:01:06.377854+0000","last_undegraded":"2024-04-25T00:01:06.377854+0000","last_fullsized":"2024-04-25T00:01:06.377854+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.338998+0000","last_change":"2024-04-25T00:01:05.354557+0000","last_active":"2024-04-25T00:01:06.338998+0000","last_peered":"2024-04-25T00:01:06.338998+0000","last_clean":"2024-04-25T00:01:06.338998+0000","last_became_active":"2024-04-25T00:01:05.353556+0000","last_became_peered":"2024-04-25T00:01:05.353556+0000","last_unstale":"2024-04-25T00:01:06.338998+0000","last_undegraded":"2024-04-25T00:01:06.338998+0000","last_fullsized":"2024-04-25T00:01:06.338998+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:24.964055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.337431+0000","last_change":"2024-04-25T00:01:04.361900+0000","last_active":"2024-04-25T00:01:06.337431+0000","last_peered":"2024-04-25T00:01:06.337431+0000","last_clean":"2024-04-25T00:01:06.337431+0000","last_became_active":"2024-04-25T00:01:04.361572+0000","last_became_peered":"2024-04-25T00:01:04.361572+0000","last_unstale":"2024-04-25T00:01:06.337431+0000","last_undegraded":"2024-04-25T00:01:06.337431+0000","last_fullsized":"2024-04-25T00:01:06.337431+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:29.218739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380225+0000","last_change":"2024-04-25T00:01:05.368769+0000","last_active":"2024-04-25T00:01:06.380225+0000","last_peered":"2024-04-25T00:01:06.380225+0000","last_clean":"2024-04-25T00:01:06.380225+0000","last_became_active":"2024-04-25T00:01:05.366817+0000","last_became_peered":"2024-04-25T00:01:05.366817+0000","last_unstale":"2024-04-25T00:01:06.380225+0000","last_undegraded":"2024-04-25T00:01:06.380225+0000","last_fullsized":"2024-04-25T00:01:06.380225+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:11:27.044893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.339852+0000","last_change":"2024-04-25T00:01:04.366577+0000","last_active":"2024-04-25T00:01:06.339852+0000","last_peered":"2024-04-25T00:01:06.339852+0000","last_clean":"2024-04-25T00:01:06.339852+0000","last_became_active":"2024-04-25T00:01:04.365881+0000","last_became_peered":"2024-04-25T00:01:04.365881+0000","last_unstale":"2024-04-25T00:01:06.339852+0000","last_undegraded":"2024-04-25T00:01:06.339852+0000","last_fullsized":"2024-04-25T00:01:06.339852+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:49:22.673421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.377363+0000","last_change":"2024-04-25T00:01:05.378340+0000","last_active":"2024-04-25T00:01:06.377363+0000","last_peered":"2024-04-25T00:01:06.377363+0000","last_clean":"2024-04-25T00:01:06.377363+0000","last_became_active":"2024-04-25T00:01:05.377593+0000","last_became_peered":"2024-04-25T00:01:05.377593+0000","last_unstale":"2024-04-25T00:01:06.377363+0000","last_undegraded":"2024-04-25T00:01:06.377363+0000","last_fullsized":"2024-04-25T00:01:06.377363+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:48.202251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.337490+0000","last_change":"2024-04-25T00:01:04.365391+0000","last_active":"2024-04-25T00:01:06.337490+0000","last_peered":"2024-04-25T00:01:06.337490+0000","last_clean":"2024-04-25T00:01:06.337490+0000","last_became_active":"2024-04-25T00:01:04.364989+0000","last_became_peered":"2024-04-25T00:01:04.364989+0000","last_unstale":"2024-04-25T00:01:06.337490+0000","last_undegraded":"2024-04-25T00:01:06.337490+0000","last_fullsized":"2024-04-25T00:01:06.337490+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:06:18.298057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.340089+0000","last_change":"2024-04-25T00:01:05.374011+0000","last_active":"2024-04-25T00:01:06.340089+0000","last_peered":"2024-04-25T00:01:06.340089+0000","last_clean":"2024-04-25T00:01:06.340089+0000","last_became_active":"2024-04-25T00:01:05.373276+0000","last_became_peered":"2024-04-25T00:01:05.373276+0000","last_unstale":"2024-04-25T00:01:06.340089+0000","last_undegraded":"2024-04-25T00:01:06.340089+0000","last_fullsized":"2024-04-25T00:01:06.340089+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:40:53.349428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-25T00:01:04.365099+0000","last_change":"2024-04-25T00:01:04.361603+0000","last_active":"2024-04-25T00:01:04.365099+0000","last_peered":"2024-04-25T00:01:04.365099+0000","last_clean":"2024-04-25T00:01:04.365099+0000","last_became_active":"2024-04-25T00:01:04.361046+0000","last_became_peered":"2024-04-25T00:01:04.361046+0000","last_unstale":"2024-04-25T00:01:04.365099+0000","last_undegraded":"2024-04-25T00:01:04.365099+0000","last_fullsized":"2024-04-25T00:01:04.365099+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:23:33.973380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":1,"reported_epoch":21,"state":"creating+peering","last_fresh":"2024-04-25T00:01:04.334917+0000","last_change":"2024-04-25T00:01:04.334917+0000","last_active":"2024-04-25T00:01:04.314940+0000","last_peered":"2024-04-25T00:01:04.314940+0000","last_clean":"2024-04-25T00:01:04.314940+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T00:01:04.334917+0000","last_undegraded":"2024-04-25T00:01:04.334917+0000","last_fullsized":"2024-04-25T00:01:04.334917+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"22'2","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.337655+0000","last_change":"2024-04-25T00:01:04.369917+0000","last_active":"2024-04-25T00:01:06.337655+0000","last_peered":"2024-04-25T00:01:06.337655+0000","last_clean":"2024-04-25T00:01:06.337655+0000","last_became_active":"2024-04-25T00:01:04.369309+0000","last_became_peered":"2024-04-25T00:01:04.369309+0000","last_unstale":"2024-04-25T00:01:06.337655+0000","last_undegraded":"2024-04-25T00:01:06.337655+0000","last_fullsized":"2024-04-25T00:01:06.337655+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:55:08.063965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":1,"reported_epoch":21,"state":"creating+peering","last_fresh":"2024-04-25T00:01:04.334467+0000","last_change":"2024-04-25T00:01:04.334467+0000","last_active":"2024-04-25T00:01:04.314940+0000","last_peered":"2024-04-25T00:01:04.314940+0000","last_clean":"2024-04-25T00:01:04.314940+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T00:01:04.334467+0000","last_undegraded":"2024-04-25T00:01:04.334467+0000","last_fullsized":"2024-04-25T00:01:04.334467+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378173+0000","last_change":"2024-04-25T00:01:04.365515+0000","last_active":"2024-04-25T00:01:06.378173+0000","last_peered":"2024-04-25T00:01:06.378173+0000","last_clean":"2024-04-25T00:01:06.378173+0000","last_became_active":"2024-04-25T00:01:04.364942+0000","last_became_peered":"2024-04-25T00:01:04.364942+0000","last_unstale":"2024-04-25T00:01:06.378173+0000","last_undegraded":"2024-04-25T00:01:06.378173+0000","last_fullsized":"2024-04-25T00:01:06.378173+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.336035+0000","last_change":"2024-04-25T00:01:05.354953+0000","last_active":"2024-04-25T00:01:06.336035+0000","last_peered":"2024-04-25T00:01:06.336035+0000","last_clean":"2024-04-25T00:01:06.336035+0000","last_became_active":"2024-04-25T00:01:05.354118+0000","last_became_peered":"2024-04-25T00:01:05.354118+0000","last_unstale":"2024-04-25T00:01:06.336035+0000","last_undegraded":"2024-04-25T00:01:06.336035+0000","last_fullsized":"2024-04-25T00:01:06.336035+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:09.858617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-25T00:01:04.367419+0000","last_change":"2024-04-25T00:01:04.362268+0000","last_active":"2024-04-25T00:01:04.367419+0000","last_peered":"2024-04-25T00:01:04.367419+0000","last_clean":"2024-04-25T00:01:04.367419+0000","last_became_active":"2024-04-25T00:01:04.361622+0000","last_became_peered":"2024-04-25T00:01:04.361622+0000","last_unstale":"2024-04-25T00:01:04.367419+0000","last_undegraded":"2024-04-25T00:01:04.367419+0000","last_fullsized":"2024-04-25T00:01:04.367419+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:40:09.442761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1,"reported_epoch":21,"state":"creating+peering","last_fresh":"2024-04-25T00:01:04.333620+0000","last_change":"2024-04-25T00:01:04.333620+0000","last_active":"2024-04-25T00:01:04.314940+0000","last_peered":"2024-04-25T00:01:04.314940+0000","last_clean":"2024-04-25T00:01:04.314940+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T00:01:04.333620+0000","last_undegraded":"2024-04-25T00:01:04.333620+0000","last_fullsized":"2024-04-25T00:01:04.333620+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.338480+0000","last_change":"2024-04-25T00:01:04.371935+0000","last_active":"2024-04-25T00:01:06.338480+0000","last_peered":"2024-04-25T00:01:06.338480+0000","last_clean":"2024-04-25T00:01:06.338480+0000","last_became_active":"2024-04-25T00:01:04.371420+0000","last_became_peered":"2024-04-25T00:01:04.371420+0000","last_unstale":"2024-04-25T00:01:06.338480+0000","last_undegraded":"2024-04-25T00:01:06.338480+0000","last_fullsized":"2024-04-25T00:01:06.338480+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378351+0000","last_change":"2024-04-25T00:01:05.362611+0000","last_active":"2024-04-25T00:01:06.378351+0000","last_peered":"2024-04-25T00:01:06.378351+0000","last_clean":"2024-04-25T00:01:06.378351+0000","last_became_active":"2024-04-25T00:01:05.361817+0000","last_became_peered":"2024-04-25T00:01:05.361817+0000","last_unstale":"2024-04-25T00:01:06.378351+0000","last_undegraded":"2024-04-25T00:01:06.378351+0000","last_fullsized":"2024-04-25T00:01:06.378351+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:50.489125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"22'2","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.336722+0000","last_change":"2024-04-25T00:01:04.359262+0000","last_active":"2024-04-25T00:01:06.336722+0000","last_peered":"2024-04-25T00:01:06.336722+0000","last_clean":"2024-04-25T00:01:06.336722+0000","last_became_active":"2024-04-25T00:01:04.357971+0000","last_became_peered":"2024-04-25T00:01:04.357971+0000","last_unstale":"2024-04-25T00:01:06.336722+0000","last_undegraded":"2024-04-25T00:01:06.336722+0000","last_fullsized":"2024-04-25T00:01:06.336722+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:05.441048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.336709+0000","last_change":"2024-04-25T00:01:05.376984+0000","last_active":"2024-04-25T00:01:06.336709+0000","last_peered":"2024-04-25T00:01:06.336709+0000","last_clean":"2024-04-25T00:01:06.336709+0000","last_became_active":"2024-04-25T00:01:05.376259+0000","last_became_peered":"2024-04-25T00:01:05.376259+0000","last_unstale":"2024-04-25T00:01:06.336709+0000","last_undegraded":"2024-04-25T00:01:06.336709+0000","last_fullsized":"2024-04-25T00:01:06.336709+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:13:28.543302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.336976+0000","last_change":"2024-04-25T00:01:05.381120+0000","last_active":"2024-04-25T00:01:06.336976+0000","last_peered":"2024-04-25T00:01:06.336976+0000","last_clean":"2024-04-25T00:01:06.336976+0000","last_became_active":"2024-04-25T00:01:05.378320+0000","last_became_peered":"2024-04-25T00:01:05.378320+0000","last_unstale":"2024-04-25T00:01:06.336976+0000","last_undegraded":"2024-04-25T00:01:06.336976+0000","last_fullsized":"2024-04-25T00:01:06.336976+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:34:06.175918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"22'2","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380832+0000","last_change":"2024-04-25T00:01:04.362832+0000","last_active":"2024-04-25T00:01:06.380832+0000","last_peered":"2024-04-25T00:01:06.380832+0000","last_clean":"2024-04-25T00:01:06.380832+0000","last_became_active":"2024-04-25T00:01:04.362059+0000","last_became_peered":"2024-04-25T00:01:04.362059+0000","last_unstale":"2024-04-25T00:01:06.380832+0000","last_undegraded":"2024-04-25T00:01:06.380832+0000","last_fullsized":"2024-04-25T00:01:06.380832+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:05.441048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.379969+0000","last_change":"2024-04-25T00:01:05.387195+0000","last_active":"2024-04-25T00:01:06.379969+0000","last_peered":"2024-04-25T00:01:06.379969+0000","last_clean":"2024-04-25T00:01:06.379969+0000","last_became_active":"2024-04-25T00:01:05.386853+0000","last_became_peered":"2024-04-25T00:01:05.386853+0000","last_unstale":"2024-04-25T00:01:06.379969+0000","last_undegraded":"2024-04-25T00:01:06.379969+0000","last_fullsized":"2024-04-25T00:01:06.379969+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:06:19.307378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"22'2","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380527+0000","last_change":"2024-04-25T00:01:04.362030+0000","last_active":"2024-04-25T00:01:06.380527+0000","last_peered":"2024-04-25T00:01:06.380527+0000","last_clean":"2024-04-25T00:01:06.380527+0000","last_became_active":"2024-04-25T00:01:04.361708+0000","last_became_peered":"2024-04-25T00:01:04.361708+0000","last_unstale":"2024-04-25T00:01:06.380527+0000","last_undegraded":"2024-04-25T00:01:06.380527+0000","last_fullsized":"2024-04-25T00:01:06.380527+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:38:33.392719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.340575+0000","last_change":"2024-04-25T00:01:05.370818+0000","last_active":"2024-04-25T00:01:06.340575+0000","last_peered":"2024-04-25T00:01:06.340575+0000","last_clean":"2024-04-25T00:01:06.340575+0000","last_became_active":"2024-04-25T00:01:05.369991+0000","last_became_peered":"2024-04-25T00:01:05.369991+0000","last_unstale":"2024-04-25T00:01:06.340575+0000","last_undegraded":"2024-04-25T00:01:06.340575+0000","last_fullsized":"2024-04-25T00:01:06.340575+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:43.615895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"22'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.340582+0000","last_change":"2024-04-25T00:01:04.361809+0000","last_active":"2024-04-25T00:01:06.340582+0000","last_peered":"2024-04-25T00:01:06.340582+0000","last_clean":"2024-04-25T00:01:06.340582+0000","last_became_active":"2024-04-25T00:01:04.361059+0000","last_became_peered":"2024-04-25T00:01:04.361059+0000","last_unstale":"2024-04-25T00:01:06.340582+0000","last_undegraded":"2024-04-25T00:01:06.340582+0000","last_fullsized":"2024-04-25T00:01:06.340582+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.340280+0000","last_change":"2024-04-25T00:01:05.367207+0000","last_active":"2024-04-25T00:01:06.340280+0000","last_peered":"2024-04-25T00:01:06.340280+0000","last_clean":"2024-04-25T00:01:06.340280+0000","last_became_active":"2024-04-25T00:01:05.366471+0000","last_became_peered":"2024-04-25T00:01:05.366471+0000","last_unstale":"2024-04-25T00:01:06.340280+0000","last_undegraded":"2024-04-25T00:01:06.340280+0000","last_fullsized":"2024-04-25T00:01:06.340280+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:30:23.557381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-04-25T00:01:04.364774+0000","last_change":"2024-04-25T00:01:04.356032+0000","last_active":"2024-04-25T00:01:04.364774+0000","last_peered":"2024-04-25T00:01:04.364774+0000","last_clean":"2024-04-25T00:01:04.364774+0000","last_became_active":"2024-04-25T00:01:04.355293+0000","last_became_peered":"2024-04-25T00:01:04.355293+0000","last_unstale":"2024-04-25T00:01:04.364774+0000","last_undegraded":"2024-04-25T00:01:04.364774+0000","last_fullsized":"2024-04-25T00:01:04.364774+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:34:05.574864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1828,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24,"ondisk_log_size":24,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509446,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":241664,"data_stored":81279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509446,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":233472,"data_stored":81155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509445,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":823296,"data_stored":665340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509446,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":237568,"data_stored":81737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509447,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":233472,"data_stored":76492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509446,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":819200,"data_stored":665318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509446,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":241664,"data_stored":81211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":196608,"data_stored":61875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T00:01:11.138 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.2b is not active+clean 2024-04-25T00:01:11.138 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 21, 'state': 'creating+peering', 'last_fresh': '2024-04-25T00:01:04.338341+0000', 'last_change': '2024-04-25T00:01:04.338341+0000', 'last_active': '2024-04-25T00:01:04.314940+0000', 'last_peered': '2024-04-25T00:01:04.314940+0000', 'last_clean': '2024-04-25T00:01:04.314940+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T00:01:04.338341+0000', 'last_undegraded': '2024-04-25T00:01:04.338341+0000', 'last_fullsized': '2024-04-25T00:01:04.338341+0000', 'mapping_epoch': 21, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 21, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T00:01:04.314940+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T00:01:04.314940+0000', 'last_clean_scrub_stamp': '2024-04-25T00:01:04.314940+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-25T00:01:11.139 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.1b is not active+clean 2024-04-25T00:01:11.139 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 21, 'state': 'creating+peering', 'last_fresh': '2024-04-25T00:01:04.333513+0000', 'last_change': '2024-04-25T00:01:04.333513+0000', 'last_active': '2024-04-25T00:01:04.314940+0000', 'last_peered': '2024-04-25T00:01:04.314940+0000', 'last_clean': '2024-04-25T00:01:04.314940+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T00:01:04.333513+0000', 'last_undegraded': '2024-04-25T00:01:04.333513+0000', 'last_fullsized': '2024-04-25T00:01:04.333513+0000', 'mapping_epoch': 21, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 21, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T00:01:04.314940+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T00:01:04.314940+0000', 'last_clean_scrub_stamp': '2024-04-25T00:01:04.314940+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-25T00:01:11.139 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.19 is not active+clean 2024-04-25T00:01:11.139 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 21, 'state': 'creating+peering', 'last_fresh': '2024-04-25T00:01:04.336083+0000', 'last_change': '2024-04-25T00:01:04.336083+0000', 'last_active': '2024-04-25T00:01:04.314940+0000', 'last_peered': '2024-04-25T00:01:04.314940+0000', 'last_clean': '2024-04-25T00:01:04.314940+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T00:01:04.336083+0000', 'last_undegraded': '2024-04-25T00:01:04.336083+0000', 'last_fullsized': '2024-04-25T00:01:04.336083+0000', 'mapping_epoch': 21, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 21, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T00:01:04.314940+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T00:01:04.314940+0000', 'last_clean_scrub_stamp': '2024-04-25T00:01:04.314940+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-25T00:01:11.139 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.2 is not active+clean 2024-04-25T00:01:11.139 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 21, 'state': 'creating+peering', 'last_fresh': '2024-04-25T00:01:04.333098+0000', 'last_change': '2024-04-25T00:01:04.333098+0000', 'last_active': '2024-04-25T00:01:04.314940+0000', 'last_peered': '2024-04-25T00:01:04.314940+0000', 'last_clean': '2024-04-25T00:01:04.314940+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T00:01:04.333098+0000', 'last_undegraded': '2024-04-25T00:01:04.333098+0000', 'last_fullsized': '2024-04-25T00:01:04.333098+0000', 'mapping_epoch': 21, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 21, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T00:01:04.314940+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T00:01:04.314940+0000', 'last_clean_scrub_stamp': '2024-04-25T00:01:04.314940+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-25T00:01:11.139 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.7 is not active+clean 2024-04-25T00:01:11.139 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 21, 'state': 'creating+peering', 'last_fresh': '2024-04-25T00:01:04.334917+0000', 'last_change': '2024-04-25T00:01:04.334917+0000', 'last_active': '2024-04-25T00:01:04.314940+0000', 'last_peered': '2024-04-25T00:01:04.314940+0000', 'last_clean': '2024-04-25T00:01:04.314940+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T00:01:04.334917+0000', 'last_undegraded': '2024-04-25T00:01:04.334917+0000', 'last_fullsized': '2024-04-25T00:01:04.334917+0000', 'mapping_epoch': 21, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 21, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T00:01:04.314940+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T00:01:04.314940+0000', 'last_clean_scrub_stamp': '2024-04-25T00:01:04.314940+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-25T00:01:11.139 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.6 is not active+clean 2024-04-25T00:01:11.139 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 21, 'state': 'creating+peering', 'last_fresh': '2024-04-25T00:01:04.334467+0000', 'last_change': '2024-04-25T00:01:04.334467+0000', 'last_active': '2024-04-25T00:01:04.314940+0000', 'last_peered': '2024-04-25T00:01:04.314940+0000', 'last_clean': '2024-04-25T00:01:04.314940+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T00:01:04.334467+0000', 'last_undegraded': '2024-04-25T00:01:04.334467+0000', 'last_fullsized': '2024-04-25T00:01:04.334467+0000', 'mapping_epoch': 21, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 21, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T00:01:04.314940+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T00:01:04.314940+0000', 'last_clean_scrub_stamp': '2024-04-25T00:01:04.314940+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-25T00:01:11.139 INFO:tasks.cephfs.filesystem.ceph_manager:PG 3.8 is not active+clean 2024-04-25T00:01:11.139 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '3.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 21, 'state': 'creating+peering', 'last_fresh': '2024-04-25T00:01:04.333620+0000', 'last_change': '2024-04-25T00:01:04.333620+0000', 'last_active': '2024-04-25T00:01:04.314940+0000', 'last_peered': '2024-04-25T00:01:04.314940+0000', 'last_clean': '2024-04-25T00:01:04.314940+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T00:01:04.333620+0000', 'last_undegraded': '2024-04-25T00:01:04.333620+0000', 'last_fullsized': '2024-04-25T00:01:04.333620+0000', 'mapping_epoch': 21, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 21, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T00:01:04.314940+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T00:01:04.314940+0000', 'last_clean_scrub_stamp': '2024-04-25T00:01:04.314940+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-25T00:01:11.140 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-25T00:01:11.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.290+0000 7f58674f3640 1 Processor -- start 2024-04-25T00:01:11.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.290+0000 7f58674f3640 1 -- start start 2024-04-25T00:01:11.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.294+0000 7f58674f3640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5860152f80 0x7f58600c4580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:11.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.294+0000 7f58674f3640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f58600a94a0 0x7f58600720f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:11.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.294+0000 7f58674f3640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5860155900 0x7f586006b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:11.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.294+0000 7f58674f3640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f58600a9d20 con 0x7f5860152f80 2024-04-25T00:01:11.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.294+0000 7f58674f3640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f5860153360 con 0x7f58600a94a0 2024-04-25T00:01:11.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.294+0000 7f58674f3640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f5860072f60 con 0x7f5860155900 2024-04-25T00:01:11.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.294+0000 7f586528f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5860152f80 0x7f58600c4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:11.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.294+0000 7f586528f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5860152f80 0x7f58600c4580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37618/0 (socket says 172.21.15.116:37618) 2024-04-25T00:01:11.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.294+0000 7f586528f640 1 -- 172.21.15.116:0/3819701828 learned_addr learned my addr 172.21.15.116:0/3819701828 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:11.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.294+0000 7f586528f640 1 -- 172.21.15.116:0/3819701828 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5860155900 msgr2=0x7f586006b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:11.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.294+0000 7f586528f640 1 --2- 172.21.15.116:0/3819701828 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5860155900 0x7f586006b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:11.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.294+0000 7f5864a8e640 1 --2- 172.21.15.116:0/3819701828 >> v2:172.21.15.174:3300/0 conn(0x7f58600a94a0 0x7f58600720f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:11.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.294+0000 7f586528f640 1 -- 172.21.15.116:0/3819701828 >> v2:172.21.15.174:3300/0 conn(0x7f58600a94a0 msgr2=0x7f58600720f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:11.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.294+0000 7f586528f640 1 --2- 172.21.15.116:0/3819701828 >> v2:172.21.15.174:3300/0 conn(0x7f58600a94a0 0x7f58600720f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:11.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.294+0000 7f586528f640 1 -- 172.21.15.116:0/3819701828 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5860073240 con 0x7f5860152f80 2024-04-25T00:01:11.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.294+0000 7f5864a8e640 1 --2- 172.21.15.116:0/3819701828 >> v2:172.21.15.174:3300/0 conn(0x7f58600a94a0 0x7f58600720f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:11.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.294+0000 7f586528f640 1 --2- 172.21.15.116:0/3819701828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5860152f80 0x7f58600c4580 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f58600c8f50 tx=0x7f585c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68393d066ac7aa88 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:11.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.294+0000 7f5857fff640 1 -- 172.21.15.116:0/3819701828 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f585c0ab5f0 con 0x7f5860152f80 2024-04-25T00:01:11.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.294+0000 7f5857fff640 1 -- 172.21.15.116:0/3819701828 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f585c0b0070 con 0x7f5860152f80 2024-04-25T00:01:11.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.294+0000 7f5857fff640 1 -- 172.21.15.116:0/3819701828 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f585c0ab9b0 con 0x7f5860152f80 2024-04-25T00:01:11.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.294+0000 7f58674f3640 1 -- 172.21.15.116:0/3819701828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5860152f80 msgr2=0x7f58600c4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:11.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.294+0000 7f58674f3640 1 --2- 172.21.15.116:0/3819701828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5860152f80 0x7f58600c4580 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f58600c8f50 tx=0x7f585c01c520 comp rx=0 tx=0).stop 2024-04-25T00:01:11.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.294+0000 7f58674f3640 1 -- 172.21.15.116:0/3819701828 shutdown_connections 2024-04-25T00:01:11.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.294+0000 7f58674f3640 1 --2- 172.21.15.116:0/3819701828 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5860155900 0x7f586006b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:11.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.294+0000 7f58674f3640 1 --2- 172.21.15.116:0/3819701828 >> v2:172.21.15.174:3300/0 conn(0x7f58600a94a0 0x7f58600720f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:11.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.294+0000 7f58674f3640 1 --2- 172.21.15.116:0/3819701828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5860152f80 0x7f58600c4580 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:11.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.294+0000 7f58674f3640 1 -- 172.21.15.116:0/3819701828 >> 172.21.15.116:0/3819701828 conn(0x7f58600c8110 msgr2=0x7f58600c8520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:11.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.294+0000 7f58674f3640 1 -- 172.21.15.116:0/3819701828 shutdown_connections 2024-04-25T00:01:11.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.294+0000 7f58674f3640 1 -- 172.21.15.116:0/3819701828 wait complete. 2024-04-25T00:01:11.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.298+0000 7f58674f3640 1 Processor -- start 2024-04-25T00:01:11.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.298+0000 7f58674f3640 1 -- start start 2024-04-25T00:01:11.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.298+0000 7f58674f3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58600a94a0 0x7f5860072350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:11.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.298+0000 7f58674f3640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5860152f80 0x7f58601f2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:11.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.298+0000 7f58674f3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5860155900 0x7f58601f26c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:11.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.298+0000 7f58674f3640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f58600c20e0 con 0x7f5860155900 2024-04-25T00:01:11.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.298+0000 7f58674f3640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f58600c1f60 con 0x7f58600a94a0 2024-04-25T00:01:11.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.298+0000 7f58674f3640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f58600c2260 con 0x7f5860152f80 2024-04-25T00:01:11.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.298+0000 7f5865a90640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5860155900 0x7f58601f26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:11.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.298+0000 7f5865a90640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5860155900 0x7f58601f26c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37620/0 (socket says 172.21.15.116:37620) 2024-04-25T00:01:11.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.298+0000 7f5865a90640 1 -- 172.21.15.116:0/216504057 learned_addr learned my addr 172.21.15.116:0/216504057 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:11.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.298+0000 7f5864a8e640 1 --2- 172.21.15.116:0/216504057 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5860152f80 0x7f58601f2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:11.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.298+0000 7f586528f640 1 --2- 172.21.15.116:0/216504057 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58600a94a0 0x7f5860072350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:11.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.298+0000 7f5865a90640 1 -- 172.21.15.116:0/216504057 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5860152f80 msgr2=0x7f58601f2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:11.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.298+0000 7f5865a90640 1 --2- 172.21.15.116:0/216504057 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5860152f80 0x7f58601f2180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:11.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.298+0000 7f5865a90640 1 -- 172.21.15.116:0/216504057 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58600a94a0 msgr2=0x7f5860072350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:11.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.298+0000 7f5865a90640 1 --2- 172.21.15.116:0/216504057 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58600a94a0 0x7f5860072350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:11.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.298+0000 7f5865a90640 1 -- 172.21.15.116:0/216504057 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58601590c0 con 0x7f5860155900 2024-04-25T00:01:11.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.298+0000 7f5864a8e640 1 --2- 172.21.15.116:0/216504057 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5860152f80 0x7f58601f2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:11.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.298+0000 7f586528f640 1 --2- 172.21.15.116:0/216504057 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58600a94a0 0x7f5860072350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:11.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.298+0000 7f5865a90640 1 --2- 172.21.15.116:0/216504057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5860155900 0x7f58601f26c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f585800b970 tx=0x7f585800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:11.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.298+0000 7f5855ffb640 1 -- 172.21.15.116:0/216504057 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5858019070 con 0x7f5860155900 2024-04-25T00:01:11.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.298+0000 7f5855ffb640 1 -- 172.21.15.116:0/216504057 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f585800cb60 con 0x7f5860155900 2024-04-25T00:01:11.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.298+0000 7f58674f3640 1 -- 172.21.15.116:0/216504057 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f586014bb40 con 0x7f5860155900 2024-04-25T00:01:11.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.298+0000 7f5855ffb640 1 -- 172.21.15.116:0/216504057 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5858014440 con 0x7f5860155900 2024-04-25T00:01:11.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.298+0000 7f58674f3640 1 -- 172.21.15.116:0/216504057 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5860072890 con 0x7f5860155900 2024-04-25T00:01:11.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.298+0000 7f58674f3640 1 -- 172.21.15.116:0/216504057 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58601593b0 con 0x7f5860155900 2024-04-25T00:01:11.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.302+0000 7f5855ffb640 1 -- 172.21.15.116:0/216504057 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f585801c070 con 0x7f5860155900 2024-04-25T00:01:11.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.302+0000 7f5855ffb640 1 --2- 172.21.15.116:0/216504057 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f58440678f0 0x7f5844069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:11.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.302+0000 7f5855ffb640 1 -- 172.21.15.116:0/216504057 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5689+0+0 (secure 0 0 0) 0x7f5858085930 con 0x7f5860155900 2024-04-25T00:01:11.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.302+0000 7f586528f640 1 --2- 172.21.15.116:0/216504057 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f58440678f0 0x7f5844069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:11.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.302+0000 7f586528f640 1 --2- 172.21.15.116:0/216504057 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f58440678f0 0x7f5844069db0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f585c0af000 tx=0x7f585c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:11.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.306+0000 7f5855ffb640 1 -- 172.21.15.116:0/216504057 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f5858056510 con 0x7f5860155900 2024-04-25T00:01:11.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.510+0000 7f58674f3640 1 -- 172.21.15.116:0/216504057 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f5860068a60 con 0x7f5860155900 2024-04-25T00:01:11.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.510+0000 7f5855ffb640 1 -- 172.21.15.116:0/216504057 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1011 (secure 0 0 0) 0x7f5858055e10 con 0x7f5860155900 2024-04-25T00:01:11.513 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:01:11.513 INFO:teuthology.orchestra.run.smithi116.stdout:{"fsid":"6d6a1aa7-0478-4c83-8057-05917e81f046","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":30,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":23,"num_osds":8,"num_up_osds":8,"osd_up_since":1714003251,"num_in_osds":8,"osd_in_since":1714003242,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":24,"data_bytes":592778,"bytes_used":224460800,"bytes_avail":772869652480,"bytes_total":773094113280,"write_bytes_sec":2070,"read_op_per_sec":0,"write_op_per_sec":6},"fsmap":{"epoch":9,"id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"b","status":"up:active","gid":4270}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2024-04-25T00:00:40.685756+0000","services":{}},"progress_events":{}} 2024-04-25T00:01:11.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.514+0000 7f58674f3640 1 -- 172.21.15.116:0/216504057 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f58440678f0 msgr2=0x7f5844069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:11.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.514+0000 7f58674f3640 1 --2- 172.21.15.116:0/216504057 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f58440678f0 0x7f5844069db0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f585c0af000 tx=0x7f585c008040 comp rx=0 tx=0).stop 2024-04-25T00:01:11.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.514+0000 7f58674f3640 1 -- 172.21.15.116:0/216504057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5860155900 msgr2=0x7f58601f26c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:11.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.514+0000 7f58674f3640 1 --2- 172.21.15.116:0/216504057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5860155900 0x7f58601f26c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f585800b970 tx=0x7f585800be30 comp rx=0 tx=0).stop 2024-04-25T00:01:11.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.514+0000 7f58674f3640 1 -- 172.21.15.116:0/216504057 shutdown_connections 2024-04-25T00:01:11.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.514+0000 7f58674f3640 1 --2- 172.21.15.116:0/216504057 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f58440678f0 0x7f5844069db0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:11.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.514+0000 7f58674f3640 1 --2- 172.21.15.116:0/216504057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5860155900 0x7f58601f26c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:11.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.514+0000 7f58674f3640 1 --2- 172.21.15.116:0/216504057 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5860152f80 0x7f58601f2180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:11.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.514+0000 7f58674f3640 1 --2- 172.21.15.116:0/216504057 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58600a94a0 0x7f5860072350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:11.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.514+0000 7f58674f3640 1 -- 172.21.15.116:0/216504057 >> 172.21.15.116:0/216504057 conn(0x7f58600c8110 msgr2=0x7f58600c5410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:11.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.514+0000 7f58674f3640 1 -- 172.21.15.116:0/216504057 shutdown_connections 2024-04-25T00:01:11.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.514+0000 7f58674f3640 1 -- 172.21.15.116:0/216504057 wait complete. 2024-04-25T00:01:11.530 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-04-25T00:01:11.530 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T00:01:11.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.630+0000 7f05d8c56640 1 Processor -- start 2024-04-25T00:01:11.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.634+0000 7f05d8c56640 1 -- start start 2024-04-25T00:01:11.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.634+0000 7f05d8c56640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f05d40c3ad0 0x7f05d4152f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:11.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.634+0000 7f05d8c56640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f05d40c3300 0x7f05d4071d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:11.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.634+0000 7f05d8c56640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05d40c3f50 con 0x7f05d4148f20 2024-04-25T00:01:11.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.634+0000 7f05d8c56640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f05d40c4790 con 0x7f05d40c3ad0 2024-04-25T00:01:11.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.634+0000 7f05d8c56640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f05d40c2b40 con 0x7f05d40c3300 2024-04-25T00:01:11.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.634+0000 7f05d1d9b640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f05d40c3ad0 0x7f05d4152f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:11.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.634+0000 7f05d2d9d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f05d40c3300 0x7f05d4071d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:11.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.634+0000 7f05d259c640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f05d4148f20 0x7f05d406e480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:59888/0 (socket says 172.21.15.116:59888) 2024-04-25T00:01:11.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.634+0000 7f05d259c640 1 -- 172.21.15.116:0/109539104 learned_addr learned my addr 172.21.15.116:0/109539104 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:11.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.634+0000 7f05d1d9b640 1 -- 172.21.15.116:0/109539104 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f05d40c3300 msgr2=0x7f05d4071d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:11.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.634+0000 7f05d1d9b640 1 --2- 172.21.15.116:0/109539104 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f05d40c3300 0x7f05d4071d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:11.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.634+0000 7f05d1d9b640 1 -- 172.21.15.116:0/109539104 >> v1:172.21.15.174:6789/0 conn(0x7f05d4148f20 legacy=0x7f05d406e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:11.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.634+0000 7f05d1d9b640 1 -- 172.21.15.116:0/109539104 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05d40c41d0 con 0x7f05d40c3ad0 2024-04-25T00:01:11.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.638+0000 7f05d1d9b640 1 --2- 172.21.15.116:0/109539104 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f05d40c3ad0 0x7f05d4152f70 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f05bc0097c0 tx=0x7f05bc01c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=444522bf1cc514d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:11.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.638+0000 7f05d159a640 1 -- 172.21.15.116:0/109539104 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05bc0b2070 con 0x7f05d40c3ad0 2024-04-25T00:01:11.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.638+0000 7f05d159a640 1 -- 172.21.15.116:0/109539104 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05bc0ad640 con 0x7f05d40c3ad0 2024-04-25T00:01:11.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.638+0000 7f05d159a640 1 -- 172.21.15.116:0/109539104 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05bc0ada10 con 0x7f05d40c3ad0 2024-04-25T00:01:11.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.638+0000 7f05d8c56640 1 -- 172.21.15.116:0/109539104 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f05d40c3ad0 msgr2=0x7f05d4152f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:11.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.638+0000 7f05d8c56640 1 --2- 172.21.15.116:0/109539104 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f05d40c3ad0 0x7f05d4152f70 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f05bc0097c0 tx=0x7f05bc01c4a0 comp rx=0 tx=0).stop 2024-04-25T00:01:11.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.638+0000 7f05d8c56640 1 -- 172.21.15.116:0/109539104 shutdown_connections 2024-04-25T00:01:11.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.638+0000 7f05d8c56640 1 --2- 172.21.15.116:0/109539104 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f05d40c3300 0x7f05d4071d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:11.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.638+0000 7f05d8c56640 1 --2- 172.21.15.116:0/109539104 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f05d40c3ad0 0x7f05d4152f70 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:11.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.638+0000 7f05d8c56640 1 -- 172.21.15.116:0/109539104 >> 172.21.15.116:0/109539104 conn(0x7f05d40c7fa0 msgr2=0x7f05d40c83b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:11.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.638+0000 7f05d8c56640 1 -- 172.21.15.116:0/109539104 shutdown_connections 2024-04-25T00:01:11.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.638+0000 7f05d8c56640 1 -- 172.21.15.116:0/109539104 wait complete. 2024-04-25T00:01:11.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.638+0000 7f05d8c56640 1 Processor -- start 2024-04-25T00:01:11.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.638+0000 7f05d8c56640 1 -- start start 2024-04-25T00:01:11.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.638+0000 7f05d8c56640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f05d40c3300 0x7f05d413f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:11.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.638+0000 7f05d8c56640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f05d40c3ad0 0x7f05d4151b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:11.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.638+0000 7f05d8c56640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05d4148f20 0x7f05d4152040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:11.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.638+0000 7f05d8c56640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f05d40c3890 con 0x7f05d4148f20 2024-04-25T00:01:11.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.638+0000 7f05d8c56640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f05d40c3710 con 0x7f05d40c3300 2024-04-25T00:01:11.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.638+0000 7f05d8c56640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f05d415f690 con 0x7f05d40c3ad0 2024-04-25T00:01:11.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.638+0000 7f05d2d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05d4148f20 0x7f05d4152040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:11.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.638+0000 7f05d2d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05d4148f20 0x7f05d4152040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37626/0 (socket says 172.21.15.116:37626) 2024-04-25T00:01:11.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.638+0000 7f05d2d9d640 1 -- 172.21.15.116:0/414139311 learned_addr learned my addr 172.21.15.116:0/414139311 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:11.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.642+0000 7f05d259c640 1 --2- 172.21.15.116:0/414139311 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f05d40c3300 0x7f05d413f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:11.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.642+0000 7f05d1d9b640 1 --2- 172.21.15.116:0/414139311 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f05d40c3ad0 0x7f05d4151b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:11.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.642+0000 7f05d2d9d640 1 -- 172.21.15.116:0/414139311 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f05d40c3ad0 msgr2=0x7f05d4151b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:11.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.642+0000 7f05d2d9d640 1 --2- 172.21.15.116:0/414139311 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f05d40c3ad0 0x7f05d4151b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:11.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.642+0000 7f05d2d9d640 1 -- 172.21.15.116:0/414139311 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f05d40c3300 msgr2=0x7f05d413f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:11.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.642+0000 7f05d2d9d640 1 --2- 172.21.15.116:0/414139311 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f05d40c3300 0x7f05d413f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:11.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.642+0000 7f05d2d9d640 1 -- 172.21.15.116:0/414139311 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05d40c7250 con 0x7f05d4148f20 2024-04-25T00:01:11.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.642+0000 7f05d2d9d640 1 --2- 172.21.15.116:0/414139311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05d4148f20 0x7f05d4152040 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f05cc009d30 tx=0x7f05cc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:11.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.642+0000 7f05caffd640 1 -- 172.21.15.116:0/414139311 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05cc018020 con 0x7f05d4148f20 2024-04-25T00:01:11.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.642+0000 7f05caffd640 1 -- 172.21.15.116:0/414139311 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05cc014480 con 0x7f05d4148f20 2024-04-25T00:01:11.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.642+0000 7f05d8c56640 1 -- 172.21.15.116:0/414139311 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05d40acdd0 con 0x7f05d4148f20 2024-04-25T00:01:11.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.642+0000 7f05caffd640 1 -- 172.21.15.116:0/414139311 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05cc014870 con 0x7f05d4148f20 2024-04-25T00:01:11.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.642+0000 7f05d8c56640 1 -- 172.21.15.116:0/414139311 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05d40c77d0 con 0x7f05d4148f20 2024-04-25T00:01:11.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.642+0000 7f05d8c56640 1 -- 172.21.15.116:0/414139311 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05d4068a60 con 0x7f05d4148f20 2024-04-25T00:01:11.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.642+0000 7f05caffd640 1 -- 172.21.15.116:0/414139311 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f05cc01b020 con 0x7f05d4148f20 2024-04-25T00:01:11.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.642+0000 7f05caffd640 1 --2- 172.21.15.116:0/414139311 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f05b40678c0 0x7f05b4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:11.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.642+0000 7f05caffd640 1 -- 172.21.15.116:0/414139311 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5689+0+0 (secure 0 0 0) 0x7f05cc0866e0 con 0x7f05d4148f20 2024-04-25T00:01:11.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.642+0000 7f05d259c640 1 --2- 172.21.15.116:0/414139311 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f05b40678c0 0x7f05b4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:11.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.642+0000 7f05d259c640 1 --2- 172.21.15.116:0/414139311 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f05b40678c0 0x7f05b4069d80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f05c4000f70 tx=0x7f05c4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:11.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.646+0000 7f05caffd640 1 -- 172.21.15.116:0/414139311 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f05cc022070 con 0x7f05d4148f20 2024-04-25T00:01:11.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.802+0000 7f05d8c56640 1 -- 172.21.15.116:0/414139311 --> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f05d414faa0 con 0x7f05b40678c0 2024-04-25T00:01:11.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.822+0000 7f05caffd640 1 -- 172.21.15.116:0/414139311 <== mgr.4105 v2:172.21.15.174:6832/2004448211 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251944 (secure 0 0 0) 0x7f05d414faa0 con 0x7f05b40678c0 2024-04-25T00:01:11.825 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:01:11.826 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-04-25T00:01:11.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.822+0000 7f05d8c56640 1 -- 172.21.15.116:0/414139311 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f05b40678c0 msgr2=0x7f05b4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:11.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.822+0000 7f05d8c56640 1 --2- 172.21.15.116:0/414139311 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f05b40678c0 0x7f05b4069d80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f05c4000f70 tx=0x7f05c4009040 comp rx=0 tx=0).stop 2024-04-25T00:01:11.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.822+0000 7f05d8c56640 1 -- 172.21.15.116:0/414139311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05d4148f20 msgr2=0x7f05d4152040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:11.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.822+0000 7f05d8c56640 1 --2- 172.21.15.116:0/414139311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05d4148f20 0x7f05d4152040 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f05cc009d30 tx=0x7f05cc00c490 comp rx=0 tx=0).stop 2024-04-25T00:01:11.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.822+0000 7f05d8c56640 1 -- 172.21.15.116:0/414139311 shutdown_connections 2024-04-25T00:01:11.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.822+0000 7f05d8c56640 1 --2- 172.21.15.116:0/414139311 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f05b40678c0 0x7f05b4069d80 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:11.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.822+0000 7f05d8c56640 1 --2- 172.21.15.116:0/414139311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05d4148f20 0x7f05d4152040 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:11.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.822+0000 7f05d8c56640 1 --2- 172.21.15.116:0/414139311 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f05d40c3ad0 0x7f05d4151b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:11.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.822+0000 7f05d8c56640 1 --2- 172.21.15.116:0/414139311 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f05d40c3300 0x7f05d413f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:11.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.822+0000 7f05d8c56640 1 -- 172.21.15.116:0/414139311 >> 172.21.15.116:0/414139311 conn(0x7f05d40c7fa0 msgr2=0x7f05d406ee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:11.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.826+0000 7f05d8c56640 1 -- 172.21.15.116:0/414139311 shutdown_connections 2024-04-25T00:01:11.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:11.826+0000 7f05d8c56640 1 -- 172.21.15.116:0/414139311 wait complete. 2024-04-25T00:01:11.843 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2024-04-25T00:01:11.252180+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":187,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219200,"kb_used_data":3012,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755520,"statfs":{"total":773094113280,"available":772869652480,"internally_reserved":0,"allocated":3084288,"data_stored":1815971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":51,"apply_latency_ms":51,"commit_latency_ns":51000000,"apply_latency_ns":51000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.923247"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380670+0000","last_change":"2024-04-25T00:01:05.389596+0000","last_active":"2024-04-25T00:01:06.380670+0000","last_peered":"2024-04-25T00:01:06.380670+0000","last_clean":"2024-04-25T00:01:06.380670+0000","last_became_active":"2024-04-25T00:01:05.389127+0000","last_became_peered":"2024-04-25T00:01:05.389127+0000","last_unstale":"2024-04-25T00:01:06.380670+0000","last_undegraded":"2024-04-25T00:01:06.380670+0000","last_fullsized":"2024-04-25T00:01:06.380670+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:09.858617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.340220+0000","last_change":"2024-04-25T00:01:05.368190+0000","last_active":"2024-04-25T00:01:06.340220+0000","last_peered":"2024-04-25T00:01:06.340220+0000","last_clean":"2024-04-25T00:01:06.340220+0000","last_became_active":"2024-04-25T00:01:05.362606+0000","last_became_peered":"2024-04-25T00:01:05.362606+0000","last_unstale":"2024-04-25T00:01:06.340220+0000","last_undegraded":"2024-04-25T00:01:06.340220+0000","last_fullsized":"2024-04-25T00:01:06.340220+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:12:52.846250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.339987+0000","last_change":"2024-04-25T00:01:05.371291+0000","last_active":"2024-04-25T00:01:06.339987+0000","last_peered":"2024-04-25T00:01:06.339987+0000","last_clean":"2024-04-25T00:01:06.339987+0000","last_became_active":"2024-04-25T00:01:05.370522+0000","last_became_peered":"2024-04-25T00:01:05.370522+0000","last_unstale":"2024-04-25T00:01:06.339987+0000","last_undegraded":"2024-04-25T00:01:06.339987+0000","last_fullsized":"2024-04-25T00:01:06.339987+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:11:27.044893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.377499+0000","last_change":"2024-04-25T00:01:05.359716+0000","last_active":"2024-04-25T00:01:06.377499+0000","last_peered":"2024-04-25T00:01:06.377499+0000","last_clean":"2024-04-25T00:01:06.377499+0000","last_became_active":"2024-04-25T00:01:05.358866+0000","last_became_peered":"2024-04-25T00:01:05.358866+0000","last_unstale":"2024-04-25T00:01:06.377499+0000","last_undegraded":"2024-04-25T00:01:06.377499+0000","last_fullsized":"2024-04-25T00:01:06.377499+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:50.489125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380258+0000","last_change":"2024-04-25T00:01:05.386068+0000","last_active":"2024-04-25T00:01:06.380258+0000","last_peered":"2024-04-25T00:01:06.380258+0000","last_clean":"2024-04-25T00:01:06.380258+0000","last_became_active":"2024-04-25T00:01:05.378650+0000","last_became_peered":"2024-04-25T00:01:05.378650+0000","last_unstale":"2024-04-25T00:01:06.380258+0000","last_undegraded":"2024-04-25T00:01:06.380258+0000","last_fullsized":"2024-04-25T00:01:06.380258+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:12:52.846250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.377084+0000","last_change":"2024-04-25T00:01:05.359253+0000","last_active":"2024-04-25T00:01:06.377084+0000","last_peered":"2024-04-25T00:01:06.377084+0000","last_clean":"2024-04-25T00:01:06.377084+0000","last_became_active":"2024-04-25T00:01:05.358477+0000","last_became_peered":"2024-04-25T00:01:05.358477+0000","last_unstale":"2024-04-25T00:01:06.377084+0000","last_undegraded":"2024-04-25T00:01:06.377084+0000","last_fullsized":"2024-04-25T00:01:06.377084+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:22.739214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.379079+0000","last_change":"2024-04-25T00:01:05.374766+0000","last_active":"2024-04-25T00:01:06.379079+0000","last_peered":"2024-04-25T00:01:06.379079+0000","last_clean":"2024-04-25T00:01:06.379079+0000","last_became_active":"2024-04-25T00:01:05.369739+0000","last_became_peered":"2024-04-25T00:01:05.369739+0000","last_unstale":"2024-04-25T00:01:06.379079+0000","last_undegraded":"2024-04-25T00:01:06.379079+0000","last_fullsized":"2024-04-25T00:01:06.379079+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:09:18.315688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.339014+0000","last_change":"2024-04-25T00:01:05.390256+0000","last_active":"2024-04-25T00:01:06.339014+0000","last_peered":"2024-04-25T00:01:06.339014+0000","last_clean":"2024-04-25T00:01:06.339014+0000","last_became_active":"2024-04-25T00:01:05.389724+0000","last_became_peered":"2024-04-25T00:01:05.389724+0000","last_unstale":"2024-04-25T00:01:06.339014+0000","last_undegraded":"2024-04-25T00:01:06.339014+0000","last_fullsized":"2024-04-25T00:01:06.339014+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:09:18.315688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.379000+0000","last_change":"2024-04-25T00:01:05.362303+0000","last_active":"2024-04-25T00:01:06.379000+0000","last_peered":"2024-04-25T00:01:06.379000+0000","last_clean":"2024-04-25T00:01:06.379000+0000","last_became_active":"2024-04-25T00:01:05.361586+0000","last_became_peered":"2024-04-25T00:01:05.361586+0000","last_unstale":"2024-04-25T00:01:06.379000+0000","last_undegraded":"2024-04-25T00:01:06.379000+0000","last_fullsized":"2024-04-25T00:01:06.379000+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:30.228060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380324+0000","last_change":"2024-04-25T00:01:05.368974+0000","last_active":"2024-04-25T00:01:06.380324+0000","last_peered":"2024-04-25T00:01:06.380324+0000","last_clean":"2024-04-25T00:01:06.380324+0000","last_became_active":"2024-04-25T00:01:05.367893+0000","last_became_peered":"2024-04-25T00:01:05.367893+0000","last_unstale":"2024-04-25T00:01:06.380324+0000","last_undegraded":"2024-04-25T00:01:06.380324+0000","last_fullsized":"2024-04-25T00:01:06.380324+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:30:23.557381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380371+0000","last_change":"2024-04-25T00:01:05.389424+0000","last_active":"2024-04-25T00:01:06.380371+0000","last_peered":"2024-04-25T00:01:06.380371+0000","last_clean":"2024-04-25T00:01:06.380371+0000","last_became_active":"2024-04-25T00:01:05.388999+0000","last_became_peered":"2024-04-25T00:01:05.388999+0000","last_unstale":"2024-04-25T00:01:06.380371+0000","last_undegraded":"2024-04-25T00:01:06.380371+0000","last_fullsized":"2024-04-25T00:01:06.380371+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:31.456792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.379890+0000","last_change":"2024-04-25T00:01:05.385853+0000","last_active":"2024-04-25T00:01:06.379890+0000","last_peered":"2024-04-25T00:01:06.379890+0000","last_clean":"2024-04-25T00:01:06.379890+0000","last_became_active":"2024-04-25T00:01:05.384997+0000","last_became_peered":"2024-04-25T00:01:05.384997+0000","last_unstale":"2024-04-25T00:01:06.379890+0000","last_undegraded":"2024-04-25T00:01:06.379890+0000","last_fullsized":"2024-04-25T00:01:06.379890+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:36:55.436661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.339127+0000","last_change":"2024-04-25T00:01:05.381011+0000","last_active":"2024-04-25T00:01:06.339127+0000","last_peered":"2024-04-25T00:01:06.339127+0000","last_clean":"2024-04-25T00:01:06.339127+0000","last_became_active":"2024-04-25T00:01:05.380236+0000","last_became_peered":"2024-04-25T00:01:05.380236+0000","last_unstale":"2024-04-25T00:01:06.339127+0000","last_undegraded":"2024-04-25T00:01:06.339127+0000","last_fullsized":"2024-04-25T00:01:06.339127+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:06:19.307378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.337813+0000","last_change":"2024-04-25T00:01:05.378373+0000","last_active":"2024-04-25T00:01:06.337813+0000","last_peered":"2024-04-25T00:01:06.337813+0000","last_clean":"2024-04-25T00:01:06.337813+0000","last_became_active":"2024-04-25T00:01:05.376570+0000","last_became_peered":"2024-04-25T00:01:05.376570+0000","last_unstale":"2024-04-25T00:01:06.337813+0000","last_undegraded":"2024-04-25T00:01:06.337813+0000","last_fullsized":"2024-04-25T00:01:06.337813+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:30.228060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.337758+0000","last_change":"2024-04-25T00:01:05.390341+0000","last_active":"2024-04-25T00:01:06.337758+0000","last_peered":"2024-04-25T00:01:06.337758+0000","last_clean":"2024-04-25T00:01:06.337758+0000","last_became_active":"2024-04-25T00:01:05.389707+0000","last_became_peered":"2024-04-25T00:01:05.389707+0000","last_unstale":"2024-04-25T00:01:06.337758+0000","last_undegraded":"2024-04-25T00:01:06.337758+0000","last_fullsized":"2024-04-25T00:01:06.337758+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:54:21.138947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.335984+0000","last_change":"2024-04-25T00:01:05.370826+0000","last_active":"2024-04-25T00:01:06.335984+0000","last_peered":"2024-04-25T00:01:06.335984+0000","last_clean":"2024-04-25T00:01:06.335984+0000","last_became_active":"2024-04-25T00:01:05.369848+0000","last_became_peered":"2024-04-25T00:01:05.369848+0000","last_unstale":"2024-04-25T00:01:06.335984+0000","last_undegraded":"2024-04-25T00:01:06.335984+0000","last_fullsized":"2024-04-25T00:01:06.335984+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:23.870121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.337683+0000","last_change":"2024-04-25T00:01:05.379958+0000","last_active":"2024-04-25T00:01:06.337683+0000","last_peered":"2024-04-25T00:01:06.337683+0000","last_clean":"2024-04-25T00:01:06.337683+0000","last_became_active":"2024-04-25T00:01:05.379090+0000","last_became_peered":"2024-04-25T00:01:05.379090+0000","last_unstale":"2024-04-25T00:01:06.337683+0000","last_undegraded":"2024-04-25T00:01:06.337683+0000","last_fullsized":"2024-04-25T00:01:06.337683+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:59:50.832958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.379255+0000","last_change":"2024-04-25T00:01:05.367001+0000","last_active":"2024-04-25T00:01:06.379255+0000","last_peered":"2024-04-25T00:01:06.379255+0000","last_clean":"2024-04-25T00:01:06.379255+0000","last_became_active":"2024-04-25T00:01:05.366264+0000","last_became_peered":"2024-04-25T00:01:05.366264+0000","last_unstale":"2024-04-25T00:01:06.379255+0000","last_undegraded":"2024-04-25T00:01:06.379255+0000","last_fullsized":"2024-04-25T00:01:06.379255+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.337778+0000","last_change":"2024-04-25T00:01:05.370653+0000","last_active":"2024-04-25T00:01:06.337778+0000","last_peered":"2024-04-25T00:01:06.337778+0000","last_clean":"2024-04-25T00:01:06.337778+0000","last_became_active":"2024-04-25T00:01:05.369919+0000","last_became_peered":"2024-04-25T00:01:05.369919+0000","last_unstale":"2024-04-25T00:01:06.337778+0000","last_undegraded":"2024-04-25T00:01:06.337778+0000","last_fullsized":"2024-04-25T00:01:06.337778+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:30:08.311326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380032+0000","last_change":"2024-04-25T00:01:05.385253+0000","last_active":"2024-04-25T00:01:06.380032+0000","last_peered":"2024-04-25T00:01:06.380032+0000","last_clean":"2024-04-25T00:01:06.380032+0000","last_became_active":"2024-04-25T00:01:05.379429+0000","last_became_peered":"2024-04-25T00:01:05.379429+0000","last_unstale":"2024-04-25T00:01:06.380032+0000","last_undegraded":"2024-04-25T00:01:06.380032+0000","last_fullsized":"2024-04-25T00:01:06.380032+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:36:10.377757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378157+0000","last_change":"2024-04-25T00:01:05.366249+0000","last_active":"2024-04-25T00:01:06.378157+0000","last_peered":"2024-04-25T00:01:06.378157+0000","last_clean":"2024-04-25T00:01:06.378157+0000","last_became_active":"2024-04-25T00:01:05.365548+0000","last_became_peered":"2024-04-25T00:01:05.365548+0000","last_unstale":"2024-04-25T00:01:06.378157+0000","last_undegraded":"2024-04-25T00:01:06.378157+0000","last_fullsized":"2024-04-25T00:01:06.378157+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:22:21.143639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378876+0000","last_change":"2024-04-25T00:01:05.387457+0000","last_active":"2024-04-25T00:01:06.378876+0000","last_peered":"2024-04-25T00:01:06.378876+0000","last_clean":"2024-04-25T00:01:06.378876+0000","last_became_active":"2024-04-25T00:01:05.386919+0000","last_became_peered":"2024-04-25T00:01:05.386919+0000","last_unstale":"2024-04-25T00:01:06.378876+0000","last_undegraded":"2024-04-25T00:01:06.378876+0000","last_fullsized":"2024-04-25T00:01:06.378876+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.338263+0000","last_change":"2024-04-25T00:01:05.366340+0000","last_active":"2024-04-25T00:01:06.338263+0000","last_peered":"2024-04-25T00:01:06.338263+0000","last_clean":"2024-04-25T00:01:06.338263+0000","last_became_active":"2024-04-25T00:01:05.361952+0000","last_became_peered":"2024-04-25T00:01:05.361952+0000","last_unstale":"2024-04-25T00:01:06.338263+0000","last_undegraded":"2024-04-25T00:01:06.338263+0000","last_fullsized":"2024-04-25T00:01:06.338263+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:41.948662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.335946+0000","last_change":"2024-04-25T00:01:05.372906+0000","last_active":"2024-04-25T00:01:06.335946+0000","last_peered":"2024-04-25T00:01:06.335946+0000","last_clean":"2024-04-25T00:01:06.335946+0000","last_became_active":"2024-04-25T00:01:05.372148+0000","last_became_peered":"2024-04-25T00:01:05.372148+0000","last_unstale":"2024-04-25T00:01:06.335946+0000","last_undegraded":"2024-04-25T00:01:06.335946+0000","last_fullsized":"2024-04-25T00:01:06.335946+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:43.615895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.340447+0000","last_change":"2024-04-25T00:01:05.373464+0000","last_active":"2024-04-25T00:01:06.340447+0000","last_peered":"2024-04-25T00:01:06.340447+0000","last_clean":"2024-04-25T00:01:06.340447+0000","last_became_active":"2024-04-25T00:01:05.372611+0000","last_became_peered":"2024-04-25T00:01:05.372611+0000","last_unstale":"2024-04-25T00:01:06.340447+0000","last_undegraded":"2024-04-25T00:01:06.340447+0000","last_fullsized":"2024-04-25T00:01:06.340447+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:31.456792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378208+0000","last_change":"2024-04-25T00:01:05.365761+0000","last_active":"2024-04-25T00:01:06.378208+0000","last_peered":"2024-04-25T00:01:06.378208+0000","last_clean":"2024-04-25T00:01:06.378208+0000","last_became_active":"2024-04-25T00:01:05.361119+0000","last_became_peered":"2024-04-25T00:01:05.361119+0000","last_unstale":"2024-04-25T00:01:06.378208+0000","last_undegraded":"2024-04-25T00:01:06.378208+0000","last_fullsized":"2024-04-25T00:01:06.378208+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.341008+0000","last_change":"2024-04-25T00:01:05.367296+0000","last_active":"2024-04-25T00:01:06.341008+0000","last_peered":"2024-04-25T00:01:06.341008+0000","last_clean":"2024-04-25T00:01:06.341008+0000","last_became_active":"2024-04-25T00:01:05.363838+0000","last_became_peered":"2024-04-25T00:01:05.363838+0000","last_unstale":"2024-04-25T00:01:06.341008+0000","last_undegraded":"2024-04-25T00:01:06.341008+0000","last_fullsized":"2024-04-25T00:01:06.341008+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:43:07.053074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.339622+0000","last_change":"2024-04-25T00:01:05.387057+0000","last_active":"2024-04-25T00:01:06.339622+0000","last_peered":"2024-04-25T00:01:06.339622+0000","last_clean":"2024-04-25T00:01:06.339622+0000","last_became_active":"2024-04-25T00:01:05.386724+0000","last_became_peered":"2024-04-25T00:01:05.386724+0000","last_unstale":"2024-04-25T00:01:06.339622+0000","last_undegraded":"2024-04-25T00:01:06.339622+0000","last_fullsized":"2024-04-25T00:01:06.339622+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:55:09.073286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.339053+0000","last_change":"2024-04-25T00:01:05.358690+0000","last_active":"2024-04-25T00:01:06.339053+0000","last_peered":"2024-04-25T00:01:06.339053+0000","last_clean":"2024-04-25T00:01:06.339053+0000","last_became_active":"2024-04-25T00:01:05.357340+0000","last_became_peered":"2024-04-25T00:01:05.357340+0000","last_unstale":"2024-04-25T00:01:06.339053+0000","last_undegraded":"2024-04-25T00:01:06.339053+0000","last_fullsized":"2024-04-25T00:01:06.339053+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:36:55.436661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.336324+0000","last_change":"2024-04-25T00:01:05.362960+0000","last_active":"2024-04-25T00:01:06.336324+0000","last_peered":"2024-04-25T00:01:06.336324+0000","last_clean":"2024-04-25T00:01:06.336324+0000","last_became_active":"2024-04-25T00:01:05.361067+0000","last_became_peered":"2024-04-25T00:01:05.361067+0000","last_unstale":"2024-04-25T00:01:06.336324+0000","last_undegraded":"2024-04-25T00:01:06.336324+0000","last_fullsized":"2024-04-25T00:01:06.336324+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:24.017263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.377980+0000","last_change":"2024-04-25T00:01:05.372927+0000","last_active":"2024-04-25T00:01:06.377980+0000","last_peered":"2024-04-25T00:01:06.377980+0000","last_clean":"2024-04-25T00:01:06.377980+0000","last_became_active":"2024-04-25T00:01:05.372050+0000","last_became_peered":"2024-04-25T00:01:05.372050+0000","last_unstale":"2024-04-25T00:01:06.377980+0000","last_undegraded":"2024-04-25T00:01:06.377980+0000","last_fullsized":"2024-04-25T00:01:06.377980+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:48.202251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.376574+0000","last_change":"2024-04-25T00:01:05.378517+0000","last_active":"2024-04-25T00:01:06.376574+0000","last_peered":"2024-04-25T00:01:06.376574+0000","last_clean":"2024-04-25T00:01:06.376574+0000","last_became_active":"2024-04-25T00:01:05.377742+0000","last_became_peered":"2024-04-25T00:01:05.377742+0000","last_unstale":"2024-04-25T00:01:06.376574+0000","last_undegraded":"2024-04-25T00:01:06.376574+0000","last_fullsized":"2024-04-25T00:01:06.376574+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:06.450369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.379065+0000","last_change":"2024-04-25T00:01:05.375142+0000","last_active":"2024-04-25T00:01:06.379065+0000","last_peered":"2024-04-25T00:01:06.379065+0000","last_clean":"2024-04-25T00:01:06.379065+0000","last_became_active":"2024-04-25T00:01:05.373857+0000","last_became_peered":"2024-04-25T00:01:05.373857+0000","last_unstale":"2024-04-25T00:01:06.379065+0000","last_undegraded":"2024-04-25T00:01:06.379065+0000","last_fullsized":"2024-04-25T00:01:06.379065+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:15.879673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"22'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.339475+0000","last_change":"2024-04-25T00:01:04.365607+0000","last_active":"2024-04-25T00:01:06.339475+0000","last_peered":"2024-04-25T00:01:06.339475+0000","last_clean":"2024-04-25T00:01:06.339475+0000","last_became_active":"2024-04-25T00:01:04.364788+0000","last_became_peered":"2024-04-25T00:01:04.364788+0000","last_unstale":"2024-04-25T00:01:06.339475+0000","last_undegraded":"2024-04-25T00:01:06.339475+0000","last_fullsized":"2024-04-25T00:01:06.339475+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378624+0000","last_change":"2024-04-25T00:01:05.370723+0000","last_active":"2024-04-25T00:01:06.378624+0000","last_peered":"2024-04-25T00:01:06.378624+0000","last_clean":"2024-04-25T00:01:06.378624+0000","last_became_active":"2024-04-25T00:01:05.368970+0000","last_became_peered":"2024-04-25T00:01:05.368970+0000","last_unstale":"2024-04-25T00:01:06.378624+0000","last_undegraded":"2024-04-25T00:01:06.378624+0000","last_fullsized":"2024-04-25T00:01:06.378624+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:06:53.743074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"22'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.336967+0000","last_change":"2024-04-25T00:01:04.360254+0000","last_active":"2024-04-25T00:01:06.336967+0000","last_peered":"2024-04-25T00:01:06.336967+0000","last_clean":"2024-04-25T00:01:06.336967+0000","last_became_active":"2024-04-25T00:01:04.359586+0000","last_became_peered":"2024-04-25T00:01:04.359586+0000","last_unstale":"2024-04-25T00:01:06.336967+0000","last_undegraded":"2024-04-25T00:01:06.336967+0000","last_fullsized":"2024-04-25T00:01:06.336967+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:47.192930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.337866+0000","last_change":"2024-04-25T00:01:04.370896+0000","last_active":"2024-04-25T00:01:06.337866+0000","last_peered":"2024-04-25T00:01:06.337866+0000","last_clean":"2024-04-25T00:01:06.337866+0000","last_became_active":"2024-04-25T00:01:04.370273+0000","last_became_peered":"2024-04-25T00:01:04.370273+0000","last_unstale":"2024-04-25T00:01:06.337866+0000","last_undegraded":"2024-04-25T00:01:06.337866+0000","last_fullsized":"2024-04-25T00:01:06.337866+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:49:22.673421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.339542+0000","last_change":"2024-04-25T00:01:05.378867+0000","last_active":"2024-04-25T00:01:06.339542+0000","last_peered":"2024-04-25T00:01:06.339542+0000","last_clean":"2024-04-25T00:01:06.339542+0000","last_became_active":"2024-04-25T00:01:05.378056+0000","last_became_peered":"2024-04-25T00:01:05.378056+0000","last_unstale":"2024-04-25T00:01:06.339542+0000","last_undegraded":"2024-04-25T00:01:06.339542+0000","last_fullsized":"2024-04-25T00:01:06.339542+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:36:10.377757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.336193+0000","last_change":"2024-04-25T00:01:04.364523+0000","last_active":"2024-04-25T00:01:06.336193+0000","last_peered":"2024-04-25T00:01:06.336193+0000","last_clean":"2024-04-25T00:01:06.336193+0000","last_became_active":"2024-04-25T00:01:04.364050+0000","last_became_peered":"2024-04-25T00:01:04.364050+0000","last_unstale":"2024-04-25T00:01:06.336193+0000","last_undegraded":"2024-04-25T00:01:06.336193+0000","last_fullsized":"2024-04-25T00:01:06.336193+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:49.479804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.377824+0000","last_change":"2024-04-25T00:01:05.370043+0000","last_active":"2024-04-25T00:01:06.377824+0000","last_peered":"2024-04-25T00:01:06.377824+0000","last_clean":"2024-04-25T00:01:06.377824+0000","last_became_active":"2024-04-25T00:01:05.368080+0000","last_became_peered":"2024-04-25T00:01:05.368080+0000","last_unstale":"2024-04-25T00:01:06.377824+0000","last_undegraded":"2024-04-25T00:01:06.377824+0000","last_fullsized":"2024-04-25T00:01:06.377824+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:58:07.610923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.379231+0000","last_change":"2024-04-25T00:01:04.360804+0000","last_active":"2024-04-25T00:01:06.379231+0000","last_peered":"2024-04-25T00:01:06.379231+0000","last_clean":"2024-04-25T00:01:06.379231+0000","last_became_active":"2024-04-25T00:01:04.360374+0000","last_became_peered":"2024-04-25T00:01:04.360374+0000","last_unstale":"2024-04-25T00:01:06.379231+0000","last_undegraded":"2024-04-25T00:01:06.379231+0000","last_fullsized":"2024-04-25T00:01:06.379231+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378559+0000","last_change":"2024-04-25T00:01:05.365413+0000","last_active":"2024-04-25T00:01:06.378559+0000","last_peered":"2024-04-25T00:01:06.378559+0000","last_clean":"2024-04-25T00:01:06.378559+0000","last_became_active":"2024-04-25T00:01:05.361865+0000","last_became_peered":"2024-04-25T00:01:05.361865+0000","last_unstale":"2024-04-25T00:01:06.378559+0000","last_undegraded":"2024-04-25T00:01:06.378559+0000","last_fullsized":"2024-04-25T00:01:06.378559+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:30.228060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380688+0000","last_change":"2024-04-25T00:01:04.362129+0000","last_active":"2024-04-25T00:01:06.380688+0000","last_peered":"2024-04-25T00:01:06.380688+0000","last_clean":"2024-04-25T00:01:06.380688+0000","last_became_active":"2024-04-25T00:01:04.361827+0000","last_became_peered":"2024-04-25T00:01:04.361827+0000","last_unstale":"2024-04-25T00:01:06.380688+0000","last_undegraded":"2024-04-25T00:01:06.380688+0000","last_fullsized":"2024-04-25T00:01:06.380688+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:26.243635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.340801+0000","last_change":"2024-04-25T00:01:05.361723+0000","last_active":"2024-04-25T00:01:06.340801+0000","last_peered":"2024-04-25T00:01:06.340801+0000","last_clean":"2024-04-25T00:01:06.340801+0000","last_became_active":"2024-04-25T00:01:05.360956+0000","last_became_peered":"2024-04-25T00:01:05.360956+0000","last_unstale":"2024-04-25T00:01:06.340801+0000","last_undegraded":"2024-04-25T00:01:06.340801+0000","last_fullsized":"2024-04-25T00:01:06.340801+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:23.870121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"22'3","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.379142+0000","last_change":"2024-04-25T00:01:04.366503+0000","last_active":"2024-04-25T00:01:06.379142+0000","last_peered":"2024-04-25T00:01:06.379142+0000","last_clean":"2024-04-25T00:01:06.379142+0000","last_became_active":"2024-04-25T00:01:04.365945+0000","last_became_peered":"2024-04-25T00:01:04.365945+0000","last_unstale":"2024-04-25T00:01:06.379142+0000","last_undegraded":"2024-04-25T00:01:06.379142+0000","last_fullsized":"2024-04-25T00:01:06.379142+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.336525+0000","last_change":"2024-04-25T00:01:05.371061+0000","last_active":"2024-04-25T00:01:06.336525+0000","last_peered":"2024-04-25T00:01:06.336525+0000","last_clean":"2024-04-25T00:01:06.336525+0000","last_became_active":"2024-04-25T00:01:05.369950+0000","last_became_peered":"2024-04-25T00:01:05.369950+0000","last_unstale":"2024-04-25T00:01:06.336525+0000","last_undegraded":"2024-04-25T00:01:06.336525+0000","last_fullsized":"2024-04-25T00:01:06.336525+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:02:43.237902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"22'6","reported_seq":20,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.377439+0000","last_change":"2024-04-25T00:01:04.365185+0000","last_active":"2024-04-25T00:01:06.377439+0000","last_peered":"2024-04-25T00:01:06.377439+0000","last_clean":"2024-04-25T00:01:06.377439+0000","last_became_active":"2024-04-25T00:01:04.364930+0000","last_became_peered":"2024-04-25T00:01:04.364930+0000","last_unstale":"2024-04-25T00:01:06.377439+0000","last_undegraded":"2024-04-25T00:01:06.377439+0000","last_fullsized":"2024-04-25T00:01:06.377439+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:45:03.618340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380617+0000","last_change":"2024-04-25T00:01:05.385827+0000","last_active":"2024-04-25T00:01:06.380617+0000","last_peered":"2024-04-25T00:01:06.380617+0000","last_clean":"2024-04-25T00:01:06.380617+0000","last_became_active":"2024-04-25T00:01:05.385258+0000","last_became_peered":"2024-04-25T00:01:05.385258+0000","last_unstale":"2024-04-25T00:01:06.380617+0000","last_undegraded":"2024-04-25T00:01:06.380617+0000","last_fullsized":"2024-04-25T00:01:06.380617+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:23.870121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"22'2","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378778+0000","last_change":"2024-04-25T00:01:04.370640+0000","last_active":"2024-04-25T00:01:06.378778+0000","last_peered":"2024-04-25T00:01:06.378778+0000","last_clean":"2024-04-25T00:01:06.378778+0000","last_became_active":"2024-04-25T00:01:04.370031+0000","last_became_peered":"2024-04-25T00:01:04.370031+0000","last_unstale":"2024-04-25T00:01:06.378778+0000","last_undegraded":"2024-04-25T00:01:06.378778+0000","last_fullsized":"2024-04-25T00:01:06.378778+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:26.243635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.338234+0000","last_change":"2024-04-25T00:01:05.379102+0000","last_active":"2024-04-25T00:01:06.338234+0000","last_peered":"2024-04-25T00:01:06.338234+0000","last_clean":"2024-04-25T00:01:06.338234+0000","last_became_active":"2024-04-25T00:01:05.377625+0000","last_became_peered":"2024-04-25T00:01:05.377625+0000","last_unstale":"2024-04-25T00:01:06.338234+0000","last_undegraded":"2024-04-25T00:01:06.338234+0000","last_fullsized":"2024-04-25T00:01:06.338234+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:15.879673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.337539+0000","last_change":"2024-04-25T00:01:04.360987+0000","last_active":"2024-04-25T00:01:06.337539+0000","last_peered":"2024-04-25T00:01:06.337539+0000","last_clean":"2024-04-25T00:01:06.337539+0000","last_became_active":"2024-04-25T00:01:04.360664+0000","last_became_peered":"2024-04-25T00:01:04.360664+0000","last_unstale":"2024-04-25T00:01:06.337539+0000","last_undegraded":"2024-04-25T00:01:06.337539+0000","last_fullsized":"2024-04-25T00:01:06.337539+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:55.283076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.336555+0000","last_change":"2024-04-25T00:01:05.379501+0000","last_active":"2024-04-25T00:01:06.336555+0000","last_peered":"2024-04-25T00:01:06.336555+0000","last_clean":"2024-04-25T00:01:06.336555+0000","last_became_active":"2024-04-25T00:01:05.378644+0000","last_became_peered":"2024-04-25T00:01:05.378644+0000","last_unstale":"2024-04-25T00:01:06.336555+0000","last_undegraded":"2024-04-25T00:01:06.336555+0000","last_fullsized":"2024-04-25T00:01:06.336555+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:08:35.505154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.379209+0000","last_change":"2024-04-25T00:01:04.357265+0000","last_active":"2024-04-25T00:01:06.379209+0000","last_peered":"2024-04-25T00:01:06.379209+0000","last_clean":"2024-04-25T00:01:06.379209+0000","last_became_active":"2024-04-25T00:01:04.357004+0000","last_became_peered":"2024-04-25T00:01:04.357004+0000","last_unstale":"2024-04-25T00:01:06.379209+0000","last_undegraded":"2024-04-25T00:01:06.379209+0000","last_fullsized":"2024-04-25T00:01:06.379209+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:47:25.595275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378119+0000","last_change":"2024-04-25T00:01:05.373927+0000","last_active":"2024-04-25T00:01:06.378119+0000","last_peered":"2024-04-25T00:01:06.378119+0000","last_clean":"2024-04-25T00:01:06.378119+0000","last_became_active":"2024-04-25T00:01:05.373077+0000","last_became_peered":"2024-04-25T00:01:05.373077+0000","last_unstale":"2024-04-25T00:01:06.378119+0000","last_undegraded":"2024-04-25T00:01:06.378119+0000","last_fullsized":"2024-04-25T00:01:06.378119+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:50.489125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"22'8","reported_seq":22,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.377425+0000","last_change":"2024-04-25T00:01:04.358365+0000","last_active":"2024-04-25T00:01:06.377425+0000","last_peered":"2024-04-25T00:01:06.377425+0000","last_clean":"2024-04-25T00:01:06.377425+0000","last_became_active":"2024-04-25T00:01:04.357571+0000","last_became_peered":"2024-04-25T00:01:04.357571+0000","last_unstale":"2024-04-25T00:01:06.377425+0000","last_undegraded":"2024-04-25T00:01:06.377425+0000","last_fullsized":"2024-04-25T00:01:06.377425+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:50:23.023114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.335878+0000","last_change":"2024-04-25T00:01:05.364103+0000","last_active":"2024-04-25T00:01:06.335878+0000","last_peered":"2024-04-25T00:01:06.335878+0000","last_clean":"2024-04-25T00:01:06.335878+0000","last_became_active":"2024-04-25T00:01:05.363320+0000","last_became_peered":"2024-04-25T00:01:05.363320+0000","last_unstale":"2024-04-25T00:01:06.335878+0000","last_undegraded":"2024-04-25T00:01:06.335878+0000","last_fullsized":"2024-04-25T00:01:06.335878+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:31.456792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"22'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380600+0000","last_change":"2024-04-25T00:01:04.372376+0000","last_active":"2024-04-25T00:01:06.380600+0000","last_peered":"2024-04-25T00:01:06.380600+0000","last_clean":"2024-04-25T00:01:06.380600+0000","last_became_active":"2024-04-25T00:01:04.372040+0000","last_became_peered":"2024-04-25T00:01:04.372040+0000","last_unstale":"2024-04-25T00:01:06.380600+0000","last_undegraded":"2024-04-25T00:01:06.380600+0000","last_fullsized":"2024-04-25T00:01:06.380600+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:21.729893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378915+0000","last_change":"2024-04-25T00:01:05.372591+0000","last_active":"2024-04-25T00:01:06.378915+0000","last_peered":"2024-04-25T00:01:06.378915+0000","last_clean":"2024-04-25T00:01:06.378915+0000","last_became_active":"2024-04-25T00:01:05.371204+0000","last_became_peered":"2024-04-25T00:01:05.371204+0000","last_unstale":"2024-04-25T00:01:06.378915+0000","last_undegraded":"2024-04-25T00:01:06.378915+0000","last_fullsized":"2024-04-25T00:01:06.378915+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:15.879673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"22'2","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.379212+0000","last_change":"2024-04-25T00:01:04.370794+0000","last_active":"2024-04-25T00:01:06.379212+0000","last_peered":"2024-04-25T00:01:06.379212+0000","last_clean":"2024-04-25T00:01:06.379212+0000","last_became_active":"2024-04-25T00:01:04.370099+0000","last_became_peered":"2024-04-25T00:01:04.370099+0000","last_unstale":"2024-04-25T00:01:06.379212+0000","last_undegraded":"2024-04-25T00:01:06.379212+0000","last_fullsized":"2024-04-25T00:01:06.379212+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:42:03.181036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380496+0000","last_change":"2024-04-25T00:01:05.378840+0000","last_active":"2024-04-25T00:01:06.380496+0000","last_peered":"2024-04-25T00:01:06.380496+0000","last_clean":"2024-04-25T00:01:06.380496+0000","last_became_active":"2024-04-25T00:01:05.378112+0000","last_became_peered":"2024-04-25T00:01:05.378112+0000","last_unstale":"2024-04-25T00:01:06.380496+0000","last_undegraded":"2024-04-25T00:01:06.380496+0000","last_fullsized":"2024-04-25T00:01:06.380496+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:43.615895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"22'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.340664+0000","last_change":"2024-04-25T00:01:04.356627+0000","last_active":"2024-04-25T00:01:06.340664+0000","last_peered":"2024-04-25T00:01:06.340664+0000","last_clean":"2024-04-25T00:01:06.340664+0000","last_became_active":"2024-04-25T00:01:04.354419+0000","last_became_peered":"2024-04-25T00:01:04.354419+0000","last_unstale":"2024-04-25T00:01:06.340664+0000","last_undegraded":"2024-04-25T00:01:06.340664+0000","last_fullsized":"2024-04-25T00:01:06.340664+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:26.243635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.381029+0000","last_change":"2024-04-25T00:01:05.385670+0000","last_active":"2024-04-25T00:01:06.381029+0000","last_peered":"2024-04-25T00:01:06.381029+0000","last_clean":"2024-04-25T00:01:06.381029+0000","last_became_active":"2024-04-25T00:01:05.384822+0000","last_became_peered":"2024-04-25T00:01:05.384822+0000","last_unstale":"2024-04-25T00:01:06.381029+0000","last_undegraded":"2024-04-25T00:01:06.381029+0000","last_fullsized":"2024-04-25T00:01:06.381029+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:24.017263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.340457+0000","last_change":"2024-04-25T00:01:04.368954+0000","last_active":"2024-04-25T00:01:06.340457+0000","last_peered":"2024-04-25T00:01:06.340457+0000","last_clean":"2024-04-25T00:01:06.340457+0000","last_became_active":"2024-04-25T00:01:04.367903+0000","last_became_peered":"2024-04-25T00:01:04.367903+0000","last_unstale":"2024-04-25T00:01:06.340457+0000","last_undegraded":"2024-04-25T00:01:06.340457+0000","last_fullsized":"2024-04-25T00:01:06.340457+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.337227+0000","last_change":"2024-04-25T00:01:05.376609+0000","last_active":"2024-04-25T00:01:06.337227+0000","last_peered":"2024-04-25T00:01:06.337227+0000","last_clean":"2024-04-25T00:01:06.337227+0000","last_became_active":"2024-04-25T00:01:05.375881+0000","last_became_peered":"2024-04-25T00:01:05.375881+0000","last_unstale":"2024-04-25T00:01:06.337227+0000","last_undegraded":"2024-04-25T00:01:06.337227+0000","last_fullsized":"2024-04-25T00:01:06.337227+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:46:05.592233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":170,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378246+0000","last_change":"2024-04-25T00:00:55.084492+0000","last_active":"2024-04-25T00:01:06.378246+0000","last_peered":"2024-04-25T00:01:06.378246+0000","last_clean":"2024-04-25T00:01:06.378246+0000","last_became_active":"2024-04-25T00:00:55.083493+0000","last_became_peered":"2024-04-25T00:00:55.083493+0000","last_unstale":"2024-04-25T00:01:06.378246+0000","last_undegraded":"2024-04-25T00:01:06.378246+0000","last_fullsized":"2024-04-25T00:01:06.378246+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:00:54.060578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:00:54.060578+0000","last_clean_scrub_stamp":"2024-04-25T00:00:54.060578+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:14:58.472477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.336858+0000","last_change":"2024-04-25T00:01:04.357028+0000","last_active":"2024-04-25T00:01:06.336858+0000","last_peered":"2024-04-25T00:01:06.336858+0000","last_clean":"2024-04-25T00:01:06.336858+0000","last_became_active":"2024-04-25T00:01:04.356644+0000","last_became_peered":"2024-04-25T00:01:04.356644+0000","last_unstale":"2024-04-25T00:01:06.336858+0000","last_undegraded":"2024-04-25T00:01:06.336858+0000","last_fullsized":"2024-04-25T00:01:06.336858+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:21.729893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378588+0000","last_change":"2024-04-25T00:01:05.364939+0000","last_active":"2024-04-25T00:01:06.378588+0000","last_peered":"2024-04-25T00:01:06.378588+0000","last_clean":"2024-04-25T00:01:06.378588+0000","last_became_active":"2024-04-25T00:01:05.364272+0000","last_became_peered":"2024-04-25T00:01:05.364272+0000","last_unstale":"2024-04-25T00:01:06.378588+0000","last_undegraded":"2024-04-25T00:01:06.378588+0000","last_fullsized":"2024-04-25T00:01:06.378588+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:24:59.679908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.336259+0000","last_change":"2024-04-25T00:01:04.370226+0000","last_active":"2024-04-25T00:01:06.336259+0000","last_peered":"2024-04-25T00:01:06.336259+0000","last_clean":"2024-04-25T00:01:06.336259+0000","last_became_active":"2024-04-25T00:01:04.369904+0000","last_became_peered":"2024-04-25T00:01:04.369904+0000","last_unstale":"2024-04-25T00:01:06.336259+0000","last_undegraded":"2024-04-25T00:01:06.336259+0000","last_fullsized":"2024-04-25T00:01:06.336259+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:47:25.595275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.337599+0000","last_change":"2024-04-25T00:01:05.370968+0000","last_active":"2024-04-25T00:01:06.337599+0000","last_peered":"2024-04-25T00:01:06.337599+0000","last_clean":"2024-04-25T00:01:06.337599+0000","last_became_active":"2024-04-25T00:01:05.369987+0000","last_became_peered":"2024-04-25T00:01:05.369987+0000","last_unstale":"2024-04-25T00:01:06.337599+0000","last_undegraded":"2024-04-25T00:01:06.337599+0000","last_fullsized":"2024-04-25T00:01:06.337599+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.377854+0000","last_change":"2024-04-25T00:01:04.372916+0000","last_active":"2024-04-25T00:01:06.377854+0000","last_peered":"2024-04-25T00:01:06.377854+0000","last_clean":"2024-04-25T00:01:06.377854+0000","last_became_active":"2024-04-25T00:01:04.372185+0000","last_became_peered":"2024-04-25T00:01:04.372185+0000","last_unstale":"2024-04-25T00:01:06.377854+0000","last_undegraded":"2024-04-25T00:01:06.377854+0000","last_fullsized":"2024-04-25T00:01:06.377854+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.338998+0000","last_change":"2024-04-25T00:01:05.354557+0000","last_active":"2024-04-25T00:01:06.338998+0000","last_peered":"2024-04-25T00:01:06.338998+0000","last_clean":"2024-04-25T00:01:06.338998+0000","last_became_active":"2024-04-25T00:01:05.353556+0000","last_became_peered":"2024-04-25T00:01:05.353556+0000","last_unstale":"2024-04-25T00:01:06.338998+0000","last_undegraded":"2024-04-25T00:01:06.338998+0000","last_fullsized":"2024-04-25T00:01:06.338998+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:24.964055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.337431+0000","last_change":"2024-04-25T00:01:04.361900+0000","last_active":"2024-04-25T00:01:06.337431+0000","last_peered":"2024-04-25T00:01:06.337431+0000","last_clean":"2024-04-25T00:01:06.337431+0000","last_became_active":"2024-04-25T00:01:04.361572+0000","last_became_peered":"2024-04-25T00:01:04.361572+0000","last_unstale":"2024-04-25T00:01:06.337431+0000","last_undegraded":"2024-04-25T00:01:06.337431+0000","last_fullsized":"2024-04-25T00:01:06.337431+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:29.218739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380225+0000","last_change":"2024-04-25T00:01:05.368769+0000","last_active":"2024-04-25T00:01:06.380225+0000","last_peered":"2024-04-25T00:01:06.380225+0000","last_clean":"2024-04-25T00:01:06.380225+0000","last_became_active":"2024-04-25T00:01:05.366817+0000","last_became_peered":"2024-04-25T00:01:05.366817+0000","last_unstale":"2024-04-25T00:01:06.380225+0000","last_undegraded":"2024-04-25T00:01:06.380225+0000","last_fullsized":"2024-04-25T00:01:06.380225+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:11:27.044893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.339852+0000","last_change":"2024-04-25T00:01:04.366577+0000","last_active":"2024-04-25T00:01:06.339852+0000","last_peered":"2024-04-25T00:01:06.339852+0000","last_clean":"2024-04-25T00:01:06.339852+0000","last_became_active":"2024-04-25T00:01:04.365881+0000","last_became_peered":"2024-04-25T00:01:04.365881+0000","last_unstale":"2024-04-25T00:01:06.339852+0000","last_undegraded":"2024-04-25T00:01:06.339852+0000","last_fullsized":"2024-04-25T00:01:06.339852+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:49:22.673421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.377363+0000","last_change":"2024-04-25T00:01:05.378340+0000","last_active":"2024-04-25T00:01:06.377363+0000","last_peered":"2024-04-25T00:01:06.377363+0000","last_clean":"2024-04-25T00:01:06.377363+0000","last_became_active":"2024-04-25T00:01:05.377593+0000","last_became_peered":"2024-04-25T00:01:05.377593+0000","last_unstale":"2024-04-25T00:01:06.377363+0000","last_undegraded":"2024-04-25T00:01:06.377363+0000","last_fullsized":"2024-04-25T00:01:06.377363+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:48.202251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.337490+0000","last_change":"2024-04-25T00:01:04.365391+0000","last_active":"2024-04-25T00:01:06.337490+0000","last_peered":"2024-04-25T00:01:06.337490+0000","last_clean":"2024-04-25T00:01:06.337490+0000","last_became_active":"2024-04-25T00:01:04.364989+0000","last_became_peered":"2024-04-25T00:01:04.364989+0000","last_unstale":"2024-04-25T00:01:06.337490+0000","last_undegraded":"2024-04-25T00:01:06.337490+0000","last_fullsized":"2024-04-25T00:01:06.337490+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:06:18.298057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.340089+0000","last_change":"2024-04-25T00:01:05.374011+0000","last_active":"2024-04-25T00:01:06.340089+0000","last_peered":"2024-04-25T00:01:06.340089+0000","last_clean":"2024-04-25T00:01:06.340089+0000","last_became_active":"2024-04-25T00:01:05.373276+0000","last_became_peered":"2024-04-25T00:01:05.373276+0000","last_unstale":"2024-04-25T00:01:06.340089+0000","last_undegraded":"2024-04-25T00:01:06.340089+0000","last_fullsized":"2024-04-25T00:01:06.340089+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:40:53.349428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"23'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.379136+0000","last_change":"2024-04-25T00:01:04.361603+0000","last_active":"2024-04-25T00:01:06.379136+0000","last_peered":"2024-04-25T00:01:06.379136+0000","last_clean":"2024-04-25T00:01:06.379136+0000","last_became_active":"2024-04-25T00:01:04.361046+0000","last_became_peered":"2024-04-25T00:01:04.361046+0000","last_unstale":"2024-04-25T00:01:06.379136+0000","last_undegraded":"2024-04-25T00:01:06.379136+0000","last_fullsized":"2024-04-25T00:01:06.379136+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:35:18.036843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378489+0000","last_change":"2024-04-25T00:01:05.364829+0000","last_active":"2024-04-25T00:01:06.378489+0000","last_peered":"2024-04-25T00:01:06.378489+0000","last_clean":"2024-04-25T00:01:06.378489+0000","last_became_active":"2024-04-25T00:01:05.364125+0000","last_became_peered":"2024-04-25T00:01:05.364125+0000","last_unstale":"2024-04-25T00:01:06.378489+0000","last_undegraded":"2024-04-25T00:01:06.378489+0000","last_fullsized":"2024-04-25T00:01:06.378489+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:15:45.531760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"22'2","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.337655+0000","last_change":"2024-04-25T00:01:04.369917+0000","last_active":"2024-04-25T00:01:06.337655+0000","last_peered":"2024-04-25T00:01:06.337655+0000","last_clean":"2024-04-25T00:01:06.337655+0000","last_became_active":"2024-04-25T00:01:04.369309+0000","last_became_peered":"2024-04-25T00:01:04.369309+0000","last_unstale":"2024-04-25T00:01:06.337655+0000","last_undegraded":"2024-04-25T00:01:06.337655+0000","last_fullsized":"2024-04-25T00:01:06.337655+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:55:08.063965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378696+0000","last_change":"2024-04-25T00:01:05.358106+0000","last_active":"2024-04-25T00:01:06.378696+0000","last_peered":"2024-04-25T00:01:06.378696+0000","last_clean":"2024-04-25T00:01:06.378696+0000","last_became_active":"2024-04-25T00:01:05.357135+0000","last_became_peered":"2024-04-25T00:01:05.357135+0000","last_unstale":"2024-04-25T00:01:06.378696+0000","last_undegraded":"2024-04-25T00:01:06.378696+0000","last_fullsized":"2024-04-25T00:01:06.378696+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:41:42.228234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378173+0000","last_change":"2024-04-25T00:01:04.365515+0000","last_active":"2024-04-25T00:01:06.378173+0000","last_peered":"2024-04-25T00:01:06.378173+0000","last_clean":"2024-04-25T00:01:06.378173+0000","last_became_active":"2024-04-25T00:01:04.364942+0000","last_became_peered":"2024-04-25T00:01:04.364942+0000","last_unstale":"2024-04-25T00:01:06.378173+0000","last_undegraded":"2024-04-25T00:01:06.378173+0000","last_fullsized":"2024-04-25T00:01:06.378173+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.336035+0000","last_change":"2024-04-25T00:01:05.354953+0000","last_active":"2024-04-25T00:01:06.336035+0000","last_peered":"2024-04-25T00:01:06.336035+0000","last_clean":"2024-04-25T00:01:06.336035+0000","last_became_active":"2024-04-25T00:01:05.354118+0000","last_became_peered":"2024-04-25T00:01:05.354118+0000","last_unstale":"2024-04-25T00:01:06.336035+0000","last_undegraded":"2024-04-25T00:01:06.336035+0000","last_fullsized":"2024-04-25T00:01:06.336035+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:09.858617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378070+0000","last_change":"2024-04-25T00:01:04.362268+0000","last_active":"2024-04-25T00:01:06.378070+0000","last_peered":"2024-04-25T00:01:06.378070+0000","last_clean":"2024-04-25T00:01:06.378070+0000","last_became_active":"2024-04-25T00:01:04.361622+0000","last_became_peered":"2024-04-25T00:01:04.361622+0000","last_unstale":"2024-04-25T00:01:06.378070+0000","last_undegraded":"2024-04-25T00:01:06.378070+0000","last_fullsized":"2024-04-25T00:01:06.378070+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:58:46.021343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.377525+0000","last_change":"2024-04-25T00:01:05.355455+0000","last_active":"2024-04-25T00:01:06.377525+0000","last_peered":"2024-04-25T00:01:06.377525+0000","last_clean":"2024-04-25T00:01:06.377525+0000","last_became_active":"2024-04-25T00:01:05.354699+0000","last_became_peered":"2024-04-25T00:01:05.354699+0000","last_unstale":"2024-04-25T00:01:06.377525+0000","last_undegraded":"2024-04-25T00:01:06.377525+0000","last_fullsized":"2024-04-25T00:01:06.377525+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:43:58.069271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.338480+0000","last_change":"2024-04-25T00:01:04.371935+0000","last_active":"2024-04-25T00:01:06.338480+0000","last_peered":"2024-04-25T00:01:06.338480+0000","last_clean":"2024-04-25T00:01:06.338480+0000","last_became_active":"2024-04-25T00:01:04.371420+0000","last_became_peered":"2024-04-25T00:01:04.371420+0000","last_unstale":"2024-04-25T00:01:06.338480+0000","last_undegraded":"2024-04-25T00:01:06.338480+0000","last_fullsized":"2024-04-25T00:01:06.338480+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378351+0000","last_change":"2024-04-25T00:01:05.362611+0000","last_active":"2024-04-25T00:01:06.378351+0000","last_peered":"2024-04-25T00:01:06.378351+0000","last_clean":"2024-04-25T00:01:06.378351+0000","last_became_active":"2024-04-25T00:01:05.361817+0000","last_became_peered":"2024-04-25T00:01:05.361817+0000","last_unstale":"2024-04-25T00:01:06.378351+0000","last_undegraded":"2024-04-25T00:01:06.378351+0000","last_fullsized":"2024-04-25T00:01:06.378351+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:50.489125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"22'2","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.336722+0000","last_change":"2024-04-25T00:01:04.359262+0000","last_active":"2024-04-25T00:01:06.336722+0000","last_peered":"2024-04-25T00:01:06.336722+0000","last_clean":"2024-04-25T00:01:06.336722+0000","last_became_active":"2024-04-25T00:01:04.357971+0000","last_became_peered":"2024-04-25T00:01:04.357971+0000","last_unstale":"2024-04-25T00:01:06.336722+0000","last_undegraded":"2024-04-25T00:01:06.336722+0000","last_fullsized":"2024-04-25T00:01:06.336722+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:05.441048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.336709+0000","last_change":"2024-04-25T00:01:05.376984+0000","last_active":"2024-04-25T00:01:06.336709+0000","last_peered":"2024-04-25T00:01:06.336709+0000","last_clean":"2024-04-25T00:01:06.336709+0000","last_became_active":"2024-04-25T00:01:05.376259+0000","last_became_peered":"2024-04-25T00:01:05.376259+0000","last_unstale":"2024-04-25T00:01:06.336709+0000","last_undegraded":"2024-04-25T00:01:06.336709+0000","last_fullsized":"2024-04-25T00:01:06.336709+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:13:28.543302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.336976+0000","last_change":"2024-04-25T00:01:05.381120+0000","last_active":"2024-04-25T00:01:06.336976+0000","last_peered":"2024-04-25T00:01:06.336976+0000","last_clean":"2024-04-25T00:01:06.336976+0000","last_became_active":"2024-04-25T00:01:05.378320+0000","last_became_peered":"2024-04-25T00:01:05.378320+0000","last_unstale":"2024-04-25T00:01:06.336976+0000","last_undegraded":"2024-04-25T00:01:06.336976+0000","last_fullsized":"2024-04-25T00:01:06.336976+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:34:06.175918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"22'2","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380832+0000","last_change":"2024-04-25T00:01:04.362832+0000","last_active":"2024-04-25T00:01:06.380832+0000","last_peered":"2024-04-25T00:01:06.380832+0000","last_clean":"2024-04-25T00:01:06.380832+0000","last_became_active":"2024-04-25T00:01:04.362059+0000","last_became_peered":"2024-04-25T00:01:04.362059+0000","last_unstale":"2024-04-25T00:01:06.380832+0000","last_undegraded":"2024-04-25T00:01:06.380832+0000","last_fullsized":"2024-04-25T00:01:06.380832+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:05.441048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.379969+0000","last_change":"2024-04-25T00:01:05.387195+0000","last_active":"2024-04-25T00:01:06.379969+0000","last_peered":"2024-04-25T00:01:06.379969+0000","last_clean":"2024-04-25T00:01:06.379969+0000","last_became_active":"2024-04-25T00:01:05.386853+0000","last_became_peered":"2024-04-25T00:01:05.386853+0000","last_unstale":"2024-04-25T00:01:06.379969+0000","last_undegraded":"2024-04-25T00:01:06.379969+0000","last_fullsized":"2024-04-25T00:01:06.379969+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:06:19.307378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"22'2","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380527+0000","last_change":"2024-04-25T00:01:04.362030+0000","last_active":"2024-04-25T00:01:06.380527+0000","last_peered":"2024-04-25T00:01:06.380527+0000","last_clean":"2024-04-25T00:01:06.380527+0000","last_became_active":"2024-04-25T00:01:04.361708+0000","last_became_peered":"2024-04-25T00:01:04.361708+0000","last_unstale":"2024-04-25T00:01:06.380527+0000","last_undegraded":"2024-04-25T00:01:06.380527+0000","last_fullsized":"2024-04-25T00:01:06.380527+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:38:33.392719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.340575+0000","last_change":"2024-04-25T00:01:05.370818+0000","last_active":"2024-04-25T00:01:06.340575+0000","last_peered":"2024-04-25T00:01:06.340575+0000","last_clean":"2024-04-25T00:01:06.340575+0000","last_became_active":"2024-04-25T00:01:05.369991+0000","last_became_peered":"2024-04-25T00:01:05.369991+0000","last_unstale":"2024-04-25T00:01:06.340575+0000","last_undegraded":"2024-04-25T00:01:06.340575+0000","last_fullsized":"2024-04-25T00:01:06.340575+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:43.615895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"22'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.340582+0000","last_change":"2024-04-25T00:01:04.361809+0000","last_active":"2024-04-25T00:01:06.340582+0000","last_peered":"2024-04-25T00:01:06.340582+0000","last_clean":"2024-04-25T00:01:06.340582+0000","last_became_active":"2024-04-25T00:01:04.361059+0000","last_became_peered":"2024-04-25T00:01:04.361059+0000","last_unstale":"2024-04-25T00:01:06.340582+0000","last_undegraded":"2024-04-25T00:01:06.340582+0000","last_fullsized":"2024-04-25T00:01:06.340582+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.340280+0000","last_change":"2024-04-25T00:01:05.367207+0000","last_active":"2024-04-25T00:01:06.340280+0000","last_peered":"2024-04-25T00:01:06.340280+0000","last_clean":"2024-04-25T00:01:06.340280+0000","last_became_active":"2024-04-25T00:01:05.366471+0000","last_became_peered":"2024-04-25T00:01:05.366471+0000","last_unstale":"2024-04-25T00:01:06.340280+0000","last_undegraded":"2024-04-25T00:01:06.340280+0000","last_fullsized":"2024-04-25T00:01:06.340280+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:30:23.557381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"22'4","reported_seq":18,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378766+0000","last_change":"2024-04-25T00:01:04.356032+0000","last_active":"2024-04-25T00:01:06.378766+0000","last_peered":"2024-04-25T00:01:06.378766+0000","last_clean":"2024-04-25T00:01:06.378766+0000","last_became_active":"2024-04-25T00:01:04.355293+0000","last_became_peered":"2024-04-25T00:01:04.355293+0000","last_unstale":"2024-04-25T00:01:06.378766+0000","last_undegraded":"2024-04-25T00:01:06.378766+0000","last_fullsized":"2024-04-25T00:01:06.378766+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:56:16.678132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509446,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":241664,"data_stored":81279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509446,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":233472,"data_stored":81155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509445,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":823296,"data_stored":665340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509446,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":237568,"data_stored":81737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509447,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":233472,"data_stored":76492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509446,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":819200,"data_stored":665318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509446,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":241664,"data_stored":81211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":253952,"data_stored":83439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T00:01:14.848 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T00:01:14.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.978+0000 7f8d2c3e1640 1 Processor -- start 2024-04-25T00:01:14.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.978+0000 7f8d2c3e1640 1 -- start start 2024-04-25T00:01:14.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.978+0000 7f8d2c3e1640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8d240a94a0 0x7f8d241538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:14.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.978+0000 7f8d2c3e1640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d240c3910 con 0x7f8d240a99f0 2024-04-25T00:01:14.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.978+0000 7f8d2c3e1640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d24154530 con 0x7f8d24150d20 2024-04-25T00:01:14.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.978+0000 7f8d2c3e1640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f8d240c16e0 con 0x7f8d240a94a0 2024-04-25T00:01:14.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.978+0000 7f8d2a97e640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f8d240a99f0 0x7f8d24153e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:60814/0 (socket says 172.21.15.116:60814) 2024-04-25T00:01:14.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.978+0000 7f8d2a97e640 1 -- 172.21.15.116:0/3111522100 learned_addr learned my addr 172.21.15.116:0/3111522100 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:14.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.978+0000 7f8d2997c640 1 --2- 172.21.15.116:0/3111522100 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8d240a94a0 0x7f8d241538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:14.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.978+0000 7f8d2917b640 1 -- 172.21.15.116:0/3111522100 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2902728457 0 0) 0x7f8d240c3910 con 0x7f8d240a99f0 2024-04-25T00:01:14.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.978+0000 7f8d2917b640 1 -- 172.21.15.116:0/3111522100 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8d10003530 con 0x7f8d240a99f0 2024-04-25T00:01:14.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.978+0000 7f8d2917b640 1 -- 172.21.15.116:0/3111522100 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4018629503 0 0) 0x7f8d10003530 con 0x7f8d240a99f0 2024-04-25T00:01:14.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.978+0000 7f8d2917b640 1 -- 172.21.15.116:0/3111522100 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8d240a94a0 msgr2=0x7f8d241538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:14.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.978+0000 7f8d2917b640 1 --2- 172.21.15.116:0/3111522100 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8d240a94a0 0x7f8d241538e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:14.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.978+0000 7f8d2917b640 1 -- 172.21.15.116:0/3111522100 >> v1:172.21.15.174:6789/0 conn(0x7f8d24150d20 legacy=0x7f8d240c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:14.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.978+0000 7f8d2917b640 1 -- 172.21.15.116:0/3111522100 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d240c1400 con 0x7f8d240a99f0 2024-04-25T00:01:14.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.978+0000 7f8d2917b640 1 -- 172.21.15.116:0/3111522100 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f8d1c001a00 con 0x7f8d240a99f0 2024-04-25T00:01:14.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.978+0000 7f8d2917b640 1 -- 172.21.15.116:0/3111522100 >> v1:172.21.15.116:6789/0 conn(0x7f8d240a99f0 legacy=0x7f8d24153e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:14.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.978+0000 7f8d2917b640 1 --2- 172.21.15.116:0/3111522100 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d100039f0 0x7f8d10003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:14.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.978+0000 7f8d2917b640 1 --2- 172.21.15.116:0/3111522100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d10004810 0x7f8d10004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:14.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.978+0000 7f8d2917b640 1 --2- 172.21.15.116:0/3111522100 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8d240a94a0 0x7f8d100051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:14.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.978+0000 7f8d2917b640 1 -- 172.21.15.116:0/3111522100 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8d240c16e0 con 0x7f8d10004810 2024-04-25T00:01:14.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.978+0000 7f8d2917b640 1 -- 172.21.15.116:0/3111522100 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8d1000ba80 con 0x7f8d100039f0 2024-04-25T00:01:14.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.978+0000 7f8d2917b640 1 -- 172.21.15.116:0/3111522100 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f8d1000bc80 con 0x7f8d240a94a0 2024-04-25T00:01:14.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.982+0000 7f8d2997c640 1 --2- 172.21.15.116:0/3111522100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d10004810 0x7f8d10004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:14.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.982+0000 7f8d2a17d640 1 --2- 172.21.15.116:0/3111522100 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d100039f0 0x7f8d10003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:14.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.982+0000 7f8d2a97e640 1 --2- 172.21.15.116:0/3111522100 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8d240a94a0 0x7f8d100051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:14.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.982+0000 7f8d2997c640 1 -- 172.21.15.116:0/3111522100 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8d240a94a0 msgr2=0x7f8d100051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:14.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.982+0000 7f8d2997c640 1 --2- 172.21.15.116:0/3111522100 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8d240a94a0 0x7f8d100051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:14.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.982+0000 7f8d2997c640 1 -- 172.21.15.116:0/3111522100 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d100039f0 msgr2=0x7f8d10003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:14.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.982+0000 7f8d2997c640 1 --2- 172.21.15.116:0/3111522100 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d100039f0 0x7f8d10003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:14.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.982+0000 7f8d2997c640 1 -- 172.21.15.116:0/3111522100 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d10003530 con 0x7f8d10004810 2024-04-25T00:01:14.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.982+0000 7f8d2997c640 1 --2- 172.21.15.116:0/3111522100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d10004810 0x7f8d10004c90 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8d14009ab0 tx=0x7f8d1401ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df1c3783ee940d80 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:14.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.982+0000 7f8d2917b640 1 -- 172.21.15.116:0/3111522100 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d140b2070 con 0x7f8d10004810 2024-04-25T00:01:14.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.982+0000 7f8d2917b640 1 -- 172.21.15.116:0/3111522100 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d140a5620 con 0x7f8d10004810 2024-04-25T00:01:14.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.982+0000 7f8d2917b640 1 -- 172.21.15.116:0/3111522100 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d140096b0 con 0x7f8d10004810 2024-04-25T00:01:14.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.982+0000 7f8d2c3e1640 1 -- 172.21.15.116:0/3111522100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d10004810 msgr2=0x7f8d10004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:14.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.982+0000 7f8d2c3e1640 1 --2- 172.21.15.116:0/3111522100 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d10004810 0x7f8d10004c90 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8d14009ab0 tx=0x7f8d1401ca30 comp rx=0 tx=0).stop 2024-04-25T00:01:14.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.982+0000 7f8d2a97e640 1 -- 172.21.15.116:0/3111522100 reap_dead start 2024-04-25T00:01:14.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.982+0000 7f8d2c3e1640 1 -- 172.21.15.116:0/3111522100 shutdown_connections 2024-04-25T00:01:14.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.982+0000 7f8d2c3e1640 1 -- 172.21.15.116:0/3111522100 >> 172.21.15.116:0/3111522100 conn(0x7f8d240c3b00 msgr2=0x7f8d240c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:14.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.982+0000 7f8d2c3e1640 1 -- 172.21.15.116:0/3111522100 shutdown_connections 2024-04-25T00:01:14.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.982+0000 7f8d2c3e1640 1 -- 172.21.15.116:0/3111522100 wait complete. 2024-04-25T00:01:14.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.986+0000 7f8d2c3e1640 1 Processor -- start 2024-04-25T00:01:14.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.986+0000 7f8d2c3e1640 1 -- start start 2024-04-25T00:01:14.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.986+0000 7f8d2c3e1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d10004810 0x7f8d2413ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:14.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.986+0000 7f8d2c3e1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d24152ca0 0x7f8d240c6630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:14.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.986+0000 7f8d2c3e1640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8d240c6b70 0x7f8d240c0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:14.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.986+0000 7f8d2c3e1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8d2406ec20 con 0x7f8d10004810 2024-04-25T00:01:14.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.986+0000 7f8d2c3e1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8d2406e940 con 0x7f8d24152ca0 2024-04-25T00:01:14.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.986+0000 7f8d2c3e1640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f8d240c3120 con 0x7f8d240c6b70 2024-04-25T00:01:14.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.986+0000 7f8d2a17d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d10004810 0x7f8d2413ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:14.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.986+0000 7f8d2a17d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d10004810 0x7f8d2413ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54216/0 (socket says 172.21.15.116:54216) 2024-04-25T00:01:14.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.986+0000 7f8d2a17d640 1 -- 172.21.15.116:0/1761371528 learned_addr learned my addr 172.21.15.116:0/1761371528 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:14.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.986+0000 7f8d2997c640 1 --2- 172.21.15.116:0/1761371528 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d24152ca0 0x7f8d240c6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:14.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.986+0000 7f8d2a97e640 1 --2- 172.21.15.116:0/1761371528 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8d240c6b70 0x7f8d240c0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:14.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.986+0000 7f8d2a17d640 1 -- 172.21.15.116:0/1761371528 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8d240c6b70 msgr2=0x7f8d240c0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:14.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.986+0000 7f8d2a17d640 1 --2- 172.21.15.116:0/1761371528 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8d240c6b70 0x7f8d240c0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:14.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.986+0000 7f8d2a17d640 1 -- 172.21.15.116:0/1761371528 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d24152ca0 msgr2=0x7f8d240c6630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:14.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.986+0000 7f8d2a17d640 1 --2- 172.21.15.116:0/1761371528 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d24152ca0 0x7f8d240c6630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:14.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.986+0000 7f8d2a17d640 1 -- 172.21.15.116:0/1761371528 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d241408f0 con 0x7f8d10004810 2024-04-25T00:01:14.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.986+0000 7f8d2a97e640 1 --2- 172.21.15.116:0/1761371528 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8d240c6b70 0x7f8d240c0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:14.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.986+0000 7f8d2a17d640 1 --2- 172.21.15.116:0/1761371528 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d10004810 0x7f8d2413ffe0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8d2000e550 tx=0x7f8d2000ea10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:14.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.986+0000 7f8d1affd640 1 -- 172.21.15.116:0/1761371528 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d200052e0 con 0x7f8d10004810 2024-04-25T00:01:14.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.986+0000 7f8d1affd640 1 -- 172.21.15.116:0/1761371528 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d20018070 con 0x7f8d10004810 2024-04-25T00:01:14.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.986+0000 7f8d2c3e1640 1 -- 172.21.15.116:0/1761371528 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d240c4bf0 con 0x7f8d10004810 2024-04-25T00:01:14.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.986+0000 7f8d1affd640 1 -- 172.21.15.116:0/1761371528 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d20003820 con 0x7f8d10004810 2024-04-25T00:01:14.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.986+0000 7f8d2c3e1640 1 -- 172.21.15.116:0/1761371528 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d241ec040 con 0x7f8d10004810 2024-04-25T00:01:14.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.986+0000 7f8d2c3e1640 1 -- 172.21.15.116:0/1761371528 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d24068350 con 0x7f8d10004810 2024-04-25T00:01:14.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.986+0000 7f8d1affd640 1 -- 172.21.15.116:0/1761371528 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8d200039c0 con 0x7f8d10004810 2024-04-25T00:01:14.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.986+0000 7f8d1affd640 1 --2- 172.21.15.116:0/1761371528 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8d080678f0 0x7f8d08069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:14.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.990+0000 7f8d1affd640 1 -- 172.21.15.116:0/1761371528 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5689+0+0 (secure 0 0 0) 0x7f8d20085420 con 0x7f8d10004810 2024-04-25T00:01:14.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.990+0000 7f8d2997c640 1 --2- 172.21.15.116:0/1761371528 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8d080678f0 0x7f8d08069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:14.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.990+0000 7f8d2997c640 1 --2- 172.21.15.116:0/1761371528 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8d080678f0 0x7f8d08069db0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8d140af000 tx=0x7f8d140046d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:14.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:14.994+0000 7f8d1affd640 1 -- 172.21.15.116:0/1761371528 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f8d20056000 con 0x7f8d10004810 2024-04-25T00:01:15.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.150+0000 7f8d2c3e1640 1 -- 172.21.15.116:0/1761371528 --> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8d241406a0 con 0x7f8d080678f0 2024-04-25T00:01:15.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.166+0000 7f8d1affd640 1 -- 172.21.15.116:0/1761371528 <== mgr.4105 v2:172.21.15.174:6832/2004448211 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251856 (secure 0 0 0) 0x7f8d241406a0 con 0x7f8d080678f0 2024-04-25T00:01:15.172 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:01:15.173 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-04-25T00:01:15.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.170+0000 7f8d2c3e1640 1 -- 172.21.15.116:0/1761371528 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8d080678f0 msgr2=0x7f8d08069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:15.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.170+0000 7f8d2c3e1640 1 --2- 172.21.15.116:0/1761371528 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8d080678f0 0x7f8d08069db0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8d140af000 tx=0x7f8d140046d0 comp rx=0 tx=0).stop 2024-04-25T00:01:15.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.170+0000 7f8d2c3e1640 1 -- 172.21.15.116:0/1761371528 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d10004810 msgr2=0x7f8d2413ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:15.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.170+0000 7f8d2c3e1640 1 --2- 172.21.15.116:0/1761371528 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d10004810 0x7f8d2413ffe0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8d2000e550 tx=0x7f8d2000ea10 comp rx=0 tx=0).stop 2024-04-25T00:01:15.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.170+0000 7f8d2c3e1640 1 -- 172.21.15.116:0/1761371528 shutdown_connections 2024-04-25T00:01:15.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.170+0000 7f8d2c3e1640 1 --2- 172.21.15.116:0/1761371528 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8d080678f0 0x7f8d08069db0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:15.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.170+0000 7f8d2c3e1640 1 --2- 172.21.15.116:0/1761371528 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8d240c6b70 0x7f8d240c0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:15.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.170+0000 7f8d2c3e1640 1 --2- 172.21.15.116:0/1761371528 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d24152ca0 0x7f8d240c6630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:15.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.170+0000 7f8d2c3e1640 1 --2- 172.21.15.116:0/1761371528 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d10004810 0x7f8d2413ffe0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:15.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.170+0000 7f8d2c3e1640 1 -- 172.21.15.116:0/1761371528 >> 172.21.15.116:0/1761371528 conn(0x7f8d240c3b00 msgr2=0x7f8d240abeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:15.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.170+0000 7f8d2c3e1640 1 -- 172.21.15.116:0/1761371528 shutdown_connections 2024-04-25T00:01:15.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.170+0000 7f8d2c3e1640 1 -- 172.21.15.116:0/1761371528 wait complete. 2024-04-25T00:01:15.189 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2024-04-25T00:01:13.252822+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219228,"kb_used_data":3048,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755492,"statfs":{"total":773094113280,"available":772869623808,"internally_reserved":0,"allocated":3121152,"data_stored":1834796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002746"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380670+0000","last_change":"2024-04-25T00:01:05.389596+0000","last_active":"2024-04-25T00:01:06.380670+0000","last_peered":"2024-04-25T00:01:06.380670+0000","last_clean":"2024-04-25T00:01:06.380670+0000","last_became_active":"2024-04-25T00:01:05.389127+0000","last_became_peered":"2024-04-25T00:01:05.389127+0000","last_unstale":"2024-04-25T00:01:06.380670+0000","last_undegraded":"2024-04-25T00:01:06.380670+0000","last_fullsized":"2024-04-25T00:01:06.380670+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:09.858617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.340220+0000","last_change":"2024-04-25T00:01:05.368190+0000","last_active":"2024-04-25T00:01:06.340220+0000","last_peered":"2024-04-25T00:01:06.340220+0000","last_clean":"2024-04-25T00:01:06.340220+0000","last_became_active":"2024-04-25T00:01:05.362606+0000","last_became_peered":"2024-04-25T00:01:05.362606+0000","last_unstale":"2024-04-25T00:01:06.340220+0000","last_undegraded":"2024-04-25T00:01:06.340220+0000","last_fullsized":"2024-04-25T00:01:06.340220+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:12:52.846250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.339987+0000","last_change":"2024-04-25T00:01:05.371291+0000","last_active":"2024-04-25T00:01:06.339987+0000","last_peered":"2024-04-25T00:01:06.339987+0000","last_clean":"2024-04-25T00:01:06.339987+0000","last_became_active":"2024-04-25T00:01:05.370522+0000","last_became_peered":"2024-04-25T00:01:05.370522+0000","last_unstale":"2024-04-25T00:01:06.339987+0000","last_undegraded":"2024-04-25T00:01:06.339987+0000","last_fullsized":"2024-04-25T00:01:06.339987+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:11:27.044893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.377499+0000","last_change":"2024-04-25T00:01:05.359716+0000","last_active":"2024-04-25T00:01:06.377499+0000","last_peered":"2024-04-25T00:01:06.377499+0000","last_clean":"2024-04-25T00:01:06.377499+0000","last_became_active":"2024-04-25T00:01:05.358866+0000","last_became_peered":"2024-04-25T00:01:05.358866+0000","last_unstale":"2024-04-25T00:01:06.377499+0000","last_undegraded":"2024-04-25T00:01:06.377499+0000","last_fullsized":"2024-04-25T00:01:06.377499+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:50.489125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380258+0000","last_change":"2024-04-25T00:01:05.386068+0000","last_active":"2024-04-25T00:01:06.380258+0000","last_peered":"2024-04-25T00:01:06.380258+0000","last_clean":"2024-04-25T00:01:06.380258+0000","last_became_active":"2024-04-25T00:01:05.378650+0000","last_became_peered":"2024-04-25T00:01:05.378650+0000","last_unstale":"2024-04-25T00:01:06.380258+0000","last_undegraded":"2024-04-25T00:01:06.380258+0000","last_fullsized":"2024-04-25T00:01:06.380258+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:12:52.846250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.377084+0000","last_change":"2024-04-25T00:01:05.359253+0000","last_active":"2024-04-25T00:01:06.377084+0000","last_peered":"2024-04-25T00:01:06.377084+0000","last_clean":"2024-04-25T00:01:06.377084+0000","last_became_active":"2024-04-25T00:01:05.358477+0000","last_became_peered":"2024-04-25T00:01:05.358477+0000","last_unstale":"2024-04-25T00:01:06.377084+0000","last_undegraded":"2024-04-25T00:01:06.377084+0000","last_fullsized":"2024-04-25T00:01:06.377084+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:22.739214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.379079+0000","last_change":"2024-04-25T00:01:05.374766+0000","last_active":"2024-04-25T00:01:06.379079+0000","last_peered":"2024-04-25T00:01:06.379079+0000","last_clean":"2024-04-25T00:01:06.379079+0000","last_became_active":"2024-04-25T00:01:05.369739+0000","last_became_peered":"2024-04-25T00:01:05.369739+0000","last_unstale":"2024-04-25T00:01:06.379079+0000","last_undegraded":"2024-04-25T00:01:06.379079+0000","last_fullsized":"2024-04-25T00:01:06.379079+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:09:18.315688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.339014+0000","last_change":"2024-04-25T00:01:05.390256+0000","last_active":"2024-04-25T00:01:06.339014+0000","last_peered":"2024-04-25T00:01:06.339014+0000","last_clean":"2024-04-25T00:01:06.339014+0000","last_became_active":"2024-04-25T00:01:05.389724+0000","last_became_peered":"2024-04-25T00:01:05.389724+0000","last_unstale":"2024-04-25T00:01:06.339014+0000","last_undegraded":"2024-04-25T00:01:06.339014+0000","last_fullsized":"2024-04-25T00:01:06.339014+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:09:18.315688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.379000+0000","last_change":"2024-04-25T00:01:05.362303+0000","last_active":"2024-04-25T00:01:06.379000+0000","last_peered":"2024-04-25T00:01:06.379000+0000","last_clean":"2024-04-25T00:01:06.379000+0000","last_became_active":"2024-04-25T00:01:05.361586+0000","last_became_peered":"2024-04-25T00:01:05.361586+0000","last_unstale":"2024-04-25T00:01:06.379000+0000","last_undegraded":"2024-04-25T00:01:06.379000+0000","last_fullsized":"2024-04-25T00:01:06.379000+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:30.228060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380324+0000","last_change":"2024-04-25T00:01:05.368974+0000","last_active":"2024-04-25T00:01:06.380324+0000","last_peered":"2024-04-25T00:01:06.380324+0000","last_clean":"2024-04-25T00:01:06.380324+0000","last_became_active":"2024-04-25T00:01:05.367893+0000","last_became_peered":"2024-04-25T00:01:05.367893+0000","last_unstale":"2024-04-25T00:01:06.380324+0000","last_undegraded":"2024-04-25T00:01:06.380324+0000","last_fullsized":"2024-04-25T00:01:06.380324+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:30:23.557381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380371+0000","last_change":"2024-04-25T00:01:05.389424+0000","last_active":"2024-04-25T00:01:06.380371+0000","last_peered":"2024-04-25T00:01:06.380371+0000","last_clean":"2024-04-25T00:01:06.380371+0000","last_became_active":"2024-04-25T00:01:05.388999+0000","last_became_peered":"2024-04-25T00:01:05.388999+0000","last_unstale":"2024-04-25T00:01:06.380371+0000","last_undegraded":"2024-04-25T00:01:06.380371+0000","last_fullsized":"2024-04-25T00:01:06.380371+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:31.456792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.379890+0000","last_change":"2024-04-25T00:01:05.385853+0000","last_active":"2024-04-25T00:01:06.379890+0000","last_peered":"2024-04-25T00:01:06.379890+0000","last_clean":"2024-04-25T00:01:06.379890+0000","last_became_active":"2024-04-25T00:01:05.384997+0000","last_became_peered":"2024-04-25T00:01:05.384997+0000","last_unstale":"2024-04-25T00:01:06.379890+0000","last_undegraded":"2024-04-25T00:01:06.379890+0000","last_fullsized":"2024-04-25T00:01:06.379890+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:36:55.436661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.339127+0000","last_change":"2024-04-25T00:01:05.381011+0000","last_active":"2024-04-25T00:01:06.339127+0000","last_peered":"2024-04-25T00:01:06.339127+0000","last_clean":"2024-04-25T00:01:06.339127+0000","last_became_active":"2024-04-25T00:01:05.380236+0000","last_became_peered":"2024-04-25T00:01:05.380236+0000","last_unstale":"2024-04-25T00:01:06.339127+0000","last_undegraded":"2024-04-25T00:01:06.339127+0000","last_fullsized":"2024-04-25T00:01:06.339127+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:06:19.307378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.337813+0000","last_change":"2024-04-25T00:01:05.378373+0000","last_active":"2024-04-25T00:01:06.337813+0000","last_peered":"2024-04-25T00:01:06.337813+0000","last_clean":"2024-04-25T00:01:06.337813+0000","last_became_active":"2024-04-25T00:01:05.376570+0000","last_became_peered":"2024-04-25T00:01:05.376570+0000","last_unstale":"2024-04-25T00:01:06.337813+0000","last_undegraded":"2024-04-25T00:01:06.337813+0000","last_fullsized":"2024-04-25T00:01:06.337813+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:30.228060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.337758+0000","last_change":"2024-04-25T00:01:05.390341+0000","last_active":"2024-04-25T00:01:06.337758+0000","last_peered":"2024-04-25T00:01:06.337758+0000","last_clean":"2024-04-25T00:01:06.337758+0000","last_became_active":"2024-04-25T00:01:05.389707+0000","last_became_peered":"2024-04-25T00:01:05.389707+0000","last_unstale":"2024-04-25T00:01:06.337758+0000","last_undegraded":"2024-04-25T00:01:06.337758+0000","last_fullsized":"2024-04-25T00:01:06.337758+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:54:21.138947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.335984+0000","last_change":"2024-04-25T00:01:05.370826+0000","last_active":"2024-04-25T00:01:06.335984+0000","last_peered":"2024-04-25T00:01:06.335984+0000","last_clean":"2024-04-25T00:01:06.335984+0000","last_became_active":"2024-04-25T00:01:05.369848+0000","last_became_peered":"2024-04-25T00:01:05.369848+0000","last_unstale":"2024-04-25T00:01:06.335984+0000","last_undegraded":"2024-04-25T00:01:06.335984+0000","last_fullsized":"2024-04-25T00:01:06.335984+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:23.870121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.337683+0000","last_change":"2024-04-25T00:01:05.379958+0000","last_active":"2024-04-25T00:01:06.337683+0000","last_peered":"2024-04-25T00:01:06.337683+0000","last_clean":"2024-04-25T00:01:06.337683+0000","last_became_active":"2024-04-25T00:01:05.379090+0000","last_became_peered":"2024-04-25T00:01:05.379090+0000","last_unstale":"2024-04-25T00:01:06.337683+0000","last_undegraded":"2024-04-25T00:01:06.337683+0000","last_fullsized":"2024-04-25T00:01:06.337683+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:59:50.832958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.379255+0000","last_change":"2024-04-25T00:01:05.367001+0000","last_active":"2024-04-25T00:01:06.379255+0000","last_peered":"2024-04-25T00:01:06.379255+0000","last_clean":"2024-04-25T00:01:06.379255+0000","last_became_active":"2024-04-25T00:01:05.366264+0000","last_became_peered":"2024-04-25T00:01:05.366264+0000","last_unstale":"2024-04-25T00:01:06.379255+0000","last_undegraded":"2024-04-25T00:01:06.379255+0000","last_fullsized":"2024-04-25T00:01:06.379255+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.337778+0000","last_change":"2024-04-25T00:01:05.370653+0000","last_active":"2024-04-25T00:01:06.337778+0000","last_peered":"2024-04-25T00:01:06.337778+0000","last_clean":"2024-04-25T00:01:06.337778+0000","last_became_active":"2024-04-25T00:01:05.369919+0000","last_became_peered":"2024-04-25T00:01:05.369919+0000","last_unstale":"2024-04-25T00:01:06.337778+0000","last_undegraded":"2024-04-25T00:01:06.337778+0000","last_fullsized":"2024-04-25T00:01:06.337778+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:30:08.311326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380032+0000","last_change":"2024-04-25T00:01:05.385253+0000","last_active":"2024-04-25T00:01:06.380032+0000","last_peered":"2024-04-25T00:01:06.380032+0000","last_clean":"2024-04-25T00:01:06.380032+0000","last_became_active":"2024-04-25T00:01:05.379429+0000","last_became_peered":"2024-04-25T00:01:05.379429+0000","last_unstale":"2024-04-25T00:01:06.380032+0000","last_undegraded":"2024-04-25T00:01:06.380032+0000","last_fullsized":"2024-04-25T00:01:06.380032+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:36:10.377757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378157+0000","last_change":"2024-04-25T00:01:05.366249+0000","last_active":"2024-04-25T00:01:06.378157+0000","last_peered":"2024-04-25T00:01:06.378157+0000","last_clean":"2024-04-25T00:01:06.378157+0000","last_became_active":"2024-04-25T00:01:05.365548+0000","last_became_peered":"2024-04-25T00:01:05.365548+0000","last_unstale":"2024-04-25T00:01:06.378157+0000","last_undegraded":"2024-04-25T00:01:06.378157+0000","last_fullsized":"2024-04-25T00:01:06.378157+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:22:21.143639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378876+0000","last_change":"2024-04-25T00:01:05.387457+0000","last_active":"2024-04-25T00:01:06.378876+0000","last_peered":"2024-04-25T00:01:06.378876+0000","last_clean":"2024-04-25T00:01:06.378876+0000","last_became_active":"2024-04-25T00:01:05.386919+0000","last_became_peered":"2024-04-25T00:01:05.386919+0000","last_unstale":"2024-04-25T00:01:06.378876+0000","last_undegraded":"2024-04-25T00:01:06.378876+0000","last_fullsized":"2024-04-25T00:01:06.378876+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.338263+0000","last_change":"2024-04-25T00:01:05.366340+0000","last_active":"2024-04-25T00:01:06.338263+0000","last_peered":"2024-04-25T00:01:06.338263+0000","last_clean":"2024-04-25T00:01:06.338263+0000","last_became_active":"2024-04-25T00:01:05.361952+0000","last_became_peered":"2024-04-25T00:01:05.361952+0000","last_unstale":"2024-04-25T00:01:06.338263+0000","last_undegraded":"2024-04-25T00:01:06.338263+0000","last_fullsized":"2024-04-25T00:01:06.338263+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:41.948662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.335946+0000","last_change":"2024-04-25T00:01:05.372906+0000","last_active":"2024-04-25T00:01:06.335946+0000","last_peered":"2024-04-25T00:01:06.335946+0000","last_clean":"2024-04-25T00:01:06.335946+0000","last_became_active":"2024-04-25T00:01:05.372148+0000","last_became_peered":"2024-04-25T00:01:05.372148+0000","last_unstale":"2024-04-25T00:01:06.335946+0000","last_undegraded":"2024-04-25T00:01:06.335946+0000","last_fullsized":"2024-04-25T00:01:06.335946+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:43.615895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.340447+0000","last_change":"2024-04-25T00:01:05.373464+0000","last_active":"2024-04-25T00:01:06.340447+0000","last_peered":"2024-04-25T00:01:06.340447+0000","last_clean":"2024-04-25T00:01:06.340447+0000","last_became_active":"2024-04-25T00:01:05.372611+0000","last_became_peered":"2024-04-25T00:01:05.372611+0000","last_unstale":"2024-04-25T00:01:06.340447+0000","last_undegraded":"2024-04-25T00:01:06.340447+0000","last_fullsized":"2024-04-25T00:01:06.340447+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:31.456792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378208+0000","last_change":"2024-04-25T00:01:05.365761+0000","last_active":"2024-04-25T00:01:06.378208+0000","last_peered":"2024-04-25T00:01:06.378208+0000","last_clean":"2024-04-25T00:01:06.378208+0000","last_became_active":"2024-04-25T00:01:05.361119+0000","last_became_peered":"2024-04-25T00:01:05.361119+0000","last_unstale":"2024-04-25T00:01:06.378208+0000","last_undegraded":"2024-04-25T00:01:06.378208+0000","last_fullsized":"2024-04-25T00:01:06.378208+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.341008+0000","last_change":"2024-04-25T00:01:05.367296+0000","last_active":"2024-04-25T00:01:06.341008+0000","last_peered":"2024-04-25T00:01:06.341008+0000","last_clean":"2024-04-25T00:01:06.341008+0000","last_became_active":"2024-04-25T00:01:05.363838+0000","last_became_peered":"2024-04-25T00:01:05.363838+0000","last_unstale":"2024-04-25T00:01:06.341008+0000","last_undegraded":"2024-04-25T00:01:06.341008+0000","last_fullsized":"2024-04-25T00:01:06.341008+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:43:07.053074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.339622+0000","last_change":"2024-04-25T00:01:05.387057+0000","last_active":"2024-04-25T00:01:06.339622+0000","last_peered":"2024-04-25T00:01:06.339622+0000","last_clean":"2024-04-25T00:01:06.339622+0000","last_became_active":"2024-04-25T00:01:05.386724+0000","last_became_peered":"2024-04-25T00:01:05.386724+0000","last_unstale":"2024-04-25T00:01:06.339622+0000","last_undegraded":"2024-04-25T00:01:06.339622+0000","last_fullsized":"2024-04-25T00:01:06.339622+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:55:09.073286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.339053+0000","last_change":"2024-04-25T00:01:05.358690+0000","last_active":"2024-04-25T00:01:06.339053+0000","last_peered":"2024-04-25T00:01:06.339053+0000","last_clean":"2024-04-25T00:01:06.339053+0000","last_became_active":"2024-04-25T00:01:05.357340+0000","last_became_peered":"2024-04-25T00:01:05.357340+0000","last_unstale":"2024-04-25T00:01:06.339053+0000","last_undegraded":"2024-04-25T00:01:06.339053+0000","last_fullsized":"2024-04-25T00:01:06.339053+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:36:55.436661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.336324+0000","last_change":"2024-04-25T00:01:05.362960+0000","last_active":"2024-04-25T00:01:06.336324+0000","last_peered":"2024-04-25T00:01:06.336324+0000","last_clean":"2024-04-25T00:01:06.336324+0000","last_became_active":"2024-04-25T00:01:05.361067+0000","last_became_peered":"2024-04-25T00:01:05.361067+0000","last_unstale":"2024-04-25T00:01:06.336324+0000","last_undegraded":"2024-04-25T00:01:06.336324+0000","last_fullsized":"2024-04-25T00:01:06.336324+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:24.017263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.377980+0000","last_change":"2024-04-25T00:01:05.372927+0000","last_active":"2024-04-25T00:01:06.377980+0000","last_peered":"2024-04-25T00:01:06.377980+0000","last_clean":"2024-04-25T00:01:06.377980+0000","last_became_active":"2024-04-25T00:01:05.372050+0000","last_became_peered":"2024-04-25T00:01:05.372050+0000","last_unstale":"2024-04-25T00:01:06.377980+0000","last_undegraded":"2024-04-25T00:01:06.377980+0000","last_fullsized":"2024-04-25T00:01:06.377980+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:48.202251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.376574+0000","last_change":"2024-04-25T00:01:05.378517+0000","last_active":"2024-04-25T00:01:06.376574+0000","last_peered":"2024-04-25T00:01:06.376574+0000","last_clean":"2024-04-25T00:01:06.376574+0000","last_became_active":"2024-04-25T00:01:05.377742+0000","last_became_peered":"2024-04-25T00:01:05.377742+0000","last_unstale":"2024-04-25T00:01:06.376574+0000","last_undegraded":"2024-04-25T00:01:06.376574+0000","last_fullsized":"2024-04-25T00:01:06.376574+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:06.450369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.379065+0000","last_change":"2024-04-25T00:01:05.375142+0000","last_active":"2024-04-25T00:01:06.379065+0000","last_peered":"2024-04-25T00:01:06.379065+0000","last_clean":"2024-04-25T00:01:06.379065+0000","last_became_active":"2024-04-25T00:01:05.373857+0000","last_became_peered":"2024-04-25T00:01:05.373857+0000","last_unstale":"2024-04-25T00:01:06.379065+0000","last_undegraded":"2024-04-25T00:01:06.379065+0000","last_fullsized":"2024-04-25T00:01:06.379065+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:15.879673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"22'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.339475+0000","last_change":"2024-04-25T00:01:04.365607+0000","last_active":"2024-04-25T00:01:06.339475+0000","last_peered":"2024-04-25T00:01:06.339475+0000","last_clean":"2024-04-25T00:01:06.339475+0000","last_became_active":"2024-04-25T00:01:04.364788+0000","last_became_peered":"2024-04-25T00:01:04.364788+0000","last_unstale":"2024-04-25T00:01:06.339475+0000","last_undegraded":"2024-04-25T00:01:06.339475+0000","last_fullsized":"2024-04-25T00:01:06.339475+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378624+0000","last_change":"2024-04-25T00:01:05.370723+0000","last_active":"2024-04-25T00:01:06.378624+0000","last_peered":"2024-04-25T00:01:06.378624+0000","last_clean":"2024-04-25T00:01:06.378624+0000","last_became_active":"2024-04-25T00:01:05.368970+0000","last_became_peered":"2024-04-25T00:01:05.368970+0000","last_unstale":"2024-04-25T00:01:06.378624+0000","last_undegraded":"2024-04-25T00:01:06.378624+0000","last_fullsized":"2024-04-25T00:01:06.378624+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:06:53.743074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"22'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.336967+0000","last_change":"2024-04-25T00:01:04.360254+0000","last_active":"2024-04-25T00:01:06.336967+0000","last_peered":"2024-04-25T00:01:06.336967+0000","last_clean":"2024-04-25T00:01:06.336967+0000","last_became_active":"2024-04-25T00:01:04.359586+0000","last_became_peered":"2024-04-25T00:01:04.359586+0000","last_unstale":"2024-04-25T00:01:06.336967+0000","last_undegraded":"2024-04-25T00:01:06.336967+0000","last_fullsized":"2024-04-25T00:01:06.336967+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:47.192930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.337866+0000","last_change":"2024-04-25T00:01:04.370896+0000","last_active":"2024-04-25T00:01:06.337866+0000","last_peered":"2024-04-25T00:01:06.337866+0000","last_clean":"2024-04-25T00:01:06.337866+0000","last_became_active":"2024-04-25T00:01:04.370273+0000","last_became_peered":"2024-04-25T00:01:04.370273+0000","last_unstale":"2024-04-25T00:01:06.337866+0000","last_undegraded":"2024-04-25T00:01:06.337866+0000","last_fullsized":"2024-04-25T00:01:06.337866+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:49:22.673421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.339542+0000","last_change":"2024-04-25T00:01:05.378867+0000","last_active":"2024-04-25T00:01:06.339542+0000","last_peered":"2024-04-25T00:01:06.339542+0000","last_clean":"2024-04-25T00:01:06.339542+0000","last_became_active":"2024-04-25T00:01:05.378056+0000","last_became_peered":"2024-04-25T00:01:05.378056+0000","last_unstale":"2024-04-25T00:01:06.339542+0000","last_undegraded":"2024-04-25T00:01:06.339542+0000","last_fullsized":"2024-04-25T00:01:06.339542+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:36:10.377757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.336193+0000","last_change":"2024-04-25T00:01:04.364523+0000","last_active":"2024-04-25T00:01:06.336193+0000","last_peered":"2024-04-25T00:01:06.336193+0000","last_clean":"2024-04-25T00:01:06.336193+0000","last_became_active":"2024-04-25T00:01:04.364050+0000","last_became_peered":"2024-04-25T00:01:04.364050+0000","last_unstale":"2024-04-25T00:01:06.336193+0000","last_undegraded":"2024-04-25T00:01:06.336193+0000","last_fullsized":"2024-04-25T00:01:06.336193+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:49.479804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.377824+0000","last_change":"2024-04-25T00:01:05.370043+0000","last_active":"2024-04-25T00:01:06.377824+0000","last_peered":"2024-04-25T00:01:06.377824+0000","last_clean":"2024-04-25T00:01:06.377824+0000","last_became_active":"2024-04-25T00:01:05.368080+0000","last_became_peered":"2024-04-25T00:01:05.368080+0000","last_unstale":"2024-04-25T00:01:06.377824+0000","last_undegraded":"2024-04-25T00:01:06.377824+0000","last_fullsized":"2024-04-25T00:01:06.377824+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:58:07.610923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.379231+0000","last_change":"2024-04-25T00:01:04.360804+0000","last_active":"2024-04-25T00:01:06.379231+0000","last_peered":"2024-04-25T00:01:06.379231+0000","last_clean":"2024-04-25T00:01:06.379231+0000","last_became_active":"2024-04-25T00:01:04.360374+0000","last_became_peered":"2024-04-25T00:01:04.360374+0000","last_unstale":"2024-04-25T00:01:06.379231+0000","last_undegraded":"2024-04-25T00:01:06.379231+0000","last_fullsized":"2024-04-25T00:01:06.379231+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378559+0000","last_change":"2024-04-25T00:01:05.365413+0000","last_active":"2024-04-25T00:01:06.378559+0000","last_peered":"2024-04-25T00:01:06.378559+0000","last_clean":"2024-04-25T00:01:06.378559+0000","last_became_active":"2024-04-25T00:01:05.361865+0000","last_became_peered":"2024-04-25T00:01:05.361865+0000","last_unstale":"2024-04-25T00:01:06.378559+0000","last_undegraded":"2024-04-25T00:01:06.378559+0000","last_fullsized":"2024-04-25T00:01:06.378559+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:30.228060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380688+0000","last_change":"2024-04-25T00:01:04.362129+0000","last_active":"2024-04-25T00:01:06.380688+0000","last_peered":"2024-04-25T00:01:06.380688+0000","last_clean":"2024-04-25T00:01:06.380688+0000","last_became_active":"2024-04-25T00:01:04.361827+0000","last_became_peered":"2024-04-25T00:01:04.361827+0000","last_unstale":"2024-04-25T00:01:06.380688+0000","last_undegraded":"2024-04-25T00:01:06.380688+0000","last_fullsized":"2024-04-25T00:01:06.380688+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:26.243635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.340801+0000","last_change":"2024-04-25T00:01:05.361723+0000","last_active":"2024-04-25T00:01:06.340801+0000","last_peered":"2024-04-25T00:01:06.340801+0000","last_clean":"2024-04-25T00:01:06.340801+0000","last_became_active":"2024-04-25T00:01:05.360956+0000","last_became_peered":"2024-04-25T00:01:05.360956+0000","last_unstale":"2024-04-25T00:01:06.340801+0000","last_undegraded":"2024-04-25T00:01:06.340801+0000","last_fullsized":"2024-04-25T00:01:06.340801+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:23.870121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"22'3","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.379142+0000","last_change":"2024-04-25T00:01:04.366503+0000","last_active":"2024-04-25T00:01:06.379142+0000","last_peered":"2024-04-25T00:01:06.379142+0000","last_clean":"2024-04-25T00:01:06.379142+0000","last_became_active":"2024-04-25T00:01:04.365945+0000","last_became_peered":"2024-04-25T00:01:04.365945+0000","last_unstale":"2024-04-25T00:01:06.379142+0000","last_undegraded":"2024-04-25T00:01:06.379142+0000","last_fullsized":"2024-04-25T00:01:06.379142+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.336525+0000","last_change":"2024-04-25T00:01:05.371061+0000","last_active":"2024-04-25T00:01:06.336525+0000","last_peered":"2024-04-25T00:01:06.336525+0000","last_clean":"2024-04-25T00:01:06.336525+0000","last_became_active":"2024-04-25T00:01:05.369950+0000","last_became_peered":"2024-04-25T00:01:05.369950+0000","last_unstale":"2024-04-25T00:01:06.336525+0000","last_undegraded":"2024-04-25T00:01:06.336525+0000","last_fullsized":"2024-04-25T00:01:06.336525+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:02:43.237902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"22'6","reported_seq":20,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.377439+0000","last_change":"2024-04-25T00:01:04.365185+0000","last_active":"2024-04-25T00:01:06.377439+0000","last_peered":"2024-04-25T00:01:06.377439+0000","last_clean":"2024-04-25T00:01:06.377439+0000","last_became_active":"2024-04-25T00:01:04.364930+0000","last_became_peered":"2024-04-25T00:01:04.364930+0000","last_unstale":"2024-04-25T00:01:06.377439+0000","last_undegraded":"2024-04-25T00:01:06.377439+0000","last_fullsized":"2024-04-25T00:01:06.377439+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:45:03.618340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380617+0000","last_change":"2024-04-25T00:01:05.385827+0000","last_active":"2024-04-25T00:01:06.380617+0000","last_peered":"2024-04-25T00:01:06.380617+0000","last_clean":"2024-04-25T00:01:06.380617+0000","last_became_active":"2024-04-25T00:01:05.385258+0000","last_became_peered":"2024-04-25T00:01:05.385258+0000","last_unstale":"2024-04-25T00:01:06.380617+0000","last_undegraded":"2024-04-25T00:01:06.380617+0000","last_fullsized":"2024-04-25T00:01:06.380617+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:23.870121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"22'2","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378778+0000","last_change":"2024-04-25T00:01:04.370640+0000","last_active":"2024-04-25T00:01:06.378778+0000","last_peered":"2024-04-25T00:01:06.378778+0000","last_clean":"2024-04-25T00:01:06.378778+0000","last_became_active":"2024-04-25T00:01:04.370031+0000","last_became_peered":"2024-04-25T00:01:04.370031+0000","last_unstale":"2024-04-25T00:01:06.378778+0000","last_undegraded":"2024-04-25T00:01:06.378778+0000","last_fullsized":"2024-04-25T00:01:06.378778+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:26.243635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.338234+0000","last_change":"2024-04-25T00:01:05.379102+0000","last_active":"2024-04-25T00:01:06.338234+0000","last_peered":"2024-04-25T00:01:06.338234+0000","last_clean":"2024-04-25T00:01:06.338234+0000","last_became_active":"2024-04-25T00:01:05.377625+0000","last_became_peered":"2024-04-25T00:01:05.377625+0000","last_unstale":"2024-04-25T00:01:06.338234+0000","last_undegraded":"2024-04-25T00:01:06.338234+0000","last_fullsized":"2024-04-25T00:01:06.338234+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:15.879673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.337539+0000","last_change":"2024-04-25T00:01:04.360987+0000","last_active":"2024-04-25T00:01:06.337539+0000","last_peered":"2024-04-25T00:01:06.337539+0000","last_clean":"2024-04-25T00:01:06.337539+0000","last_became_active":"2024-04-25T00:01:04.360664+0000","last_became_peered":"2024-04-25T00:01:04.360664+0000","last_unstale":"2024-04-25T00:01:06.337539+0000","last_undegraded":"2024-04-25T00:01:06.337539+0000","last_fullsized":"2024-04-25T00:01:06.337539+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:55.283076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.336555+0000","last_change":"2024-04-25T00:01:05.379501+0000","last_active":"2024-04-25T00:01:06.336555+0000","last_peered":"2024-04-25T00:01:06.336555+0000","last_clean":"2024-04-25T00:01:06.336555+0000","last_became_active":"2024-04-25T00:01:05.378644+0000","last_became_peered":"2024-04-25T00:01:05.378644+0000","last_unstale":"2024-04-25T00:01:06.336555+0000","last_undegraded":"2024-04-25T00:01:06.336555+0000","last_fullsized":"2024-04-25T00:01:06.336555+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:08:35.505154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.379209+0000","last_change":"2024-04-25T00:01:04.357265+0000","last_active":"2024-04-25T00:01:06.379209+0000","last_peered":"2024-04-25T00:01:06.379209+0000","last_clean":"2024-04-25T00:01:06.379209+0000","last_became_active":"2024-04-25T00:01:04.357004+0000","last_became_peered":"2024-04-25T00:01:04.357004+0000","last_unstale":"2024-04-25T00:01:06.379209+0000","last_undegraded":"2024-04-25T00:01:06.379209+0000","last_fullsized":"2024-04-25T00:01:06.379209+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:47:25.595275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378119+0000","last_change":"2024-04-25T00:01:05.373927+0000","last_active":"2024-04-25T00:01:06.378119+0000","last_peered":"2024-04-25T00:01:06.378119+0000","last_clean":"2024-04-25T00:01:06.378119+0000","last_became_active":"2024-04-25T00:01:05.373077+0000","last_became_peered":"2024-04-25T00:01:05.373077+0000","last_unstale":"2024-04-25T00:01:06.378119+0000","last_undegraded":"2024-04-25T00:01:06.378119+0000","last_fullsized":"2024-04-25T00:01:06.378119+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:50.489125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"22'8","reported_seq":22,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.377425+0000","last_change":"2024-04-25T00:01:04.358365+0000","last_active":"2024-04-25T00:01:06.377425+0000","last_peered":"2024-04-25T00:01:06.377425+0000","last_clean":"2024-04-25T00:01:06.377425+0000","last_became_active":"2024-04-25T00:01:04.357571+0000","last_became_peered":"2024-04-25T00:01:04.357571+0000","last_unstale":"2024-04-25T00:01:06.377425+0000","last_undegraded":"2024-04-25T00:01:06.377425+0000","last_fullsized":"2024-04-25T00:01:06.377425+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:50:23.023114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.335878+0000","last_change":"2024-04-25T00:01:05.364103+0000","last_active":"2024-04-25T00:01:06.335878+0000","last_peered":"2024-04-25T00:01:06.335878+0000","last_clean":"2024-04-25T00:01:06.335878+0000","last_became_active":"2024-04-25T00:01:05.363320+0000","last_became_peered":"2024-04-25T00:01:05.363320+0000","last_unstale":"2024-04-25T00:01:06.335878+0000","last_undegraded":"2024-04-25T00:01:06.335878+0000","last_fullsized":"2024-04-25T00:01:06.335878+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:31.456792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"22'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380600+0000","last_change":"2024-04-25T00:01:04.372376+0000","last_active":"2024-04-25T00:01:06.380600+0000","last_peered":"2024-04-25T00:01:06.380600+0000","last_clean":"2024-04-25T00:01:06.380600+0000","last_became_active":"2024-04-25T00:01:04.372040+0000","last_became_peered":"2024-04-25T00:01:04.372040+0000","last_unstale":"2024-04-25T00:01:06.380600+0000","last_undegraded":"2024-04-25T00:01:06.380600+0000","last_fullsized":"2024-04-25T00:01:06.380600+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:21.729893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378915+0000","last_change":"2024-04-25T00:01:05.372591+0000","last_active":"2024-04-25T00:01:06.378915+0000","last_peered":"2024-04-25T00:01:06.378915+0000","last_clean":"2024-04-25T00:01:06.378915+0000","last_became_active":"2024-04-25T00:01:05.371204+0000","last_became_peered":"2024-04-25T00:01:05.371204+0000","last_unstale":"2024-04-25T00:01:06.378915+0000","last_undegraded":"2024-04-25T00:01:06.378915+0000","last_fullsized":"2024-04-25T00:01:06.378915+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:15.879673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"22'2","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.379212+0000","last_change":"2024-04-25T00:01:04.370794+0000","last_active":"2024-04-25T00:01:06.379212+0000","last_peered":"2024-04-25T00:01:06.379212+0000","last_clean":"2024-04-25T00:01:06.379212+0000","last_became_active":"2024-04-25T00:01:04.370099+0000","last_became_peered":"2024-04-25T00:01:04.370099+0000","last_unstale":"2024-04-25T00:01:06.379212+0000","last_undegraded":"2024-04-25T00:01:06.379212+0000","last_fullsized":"2024-04-25T00:01:06.379212+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:42:03.181036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380496+0000","last_change":"2024-04-25T00:01:05.378840+0000","last_active":"2024-04-25T00:01:06.380496+0000","last_peered":"2024-04-25T00:01:06.380496+0000","last_clean":"2024-04-25T00:01:06.380496+0000","last_became_active":"2024-04-25T00:01:05.378112+0000","last_became_peered":"2024-04-25T00:01:05.378112+0000","last_unstale":"2024-04-25T00:01:06.380496+0000","last_undegraded":"2024-04-25T00:01:06.380496+0000","last_fullsized":"2024-04-25T00:01:06.380496+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:43.615895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"22'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.340664+0000","last_change":"2024-04-25T00:01:04.356627+0000","last_active":"2024-04-25T00:01:06.340664+0000","last_peered":"2024-04-25T00:01:06.340664+0000","last_clean":"2024-04-25T00:01:06.340664+0000","last_became_active":"2024-04-25T00:01:04.354419+0000","last_became_peered":"2024-04-25T00:01:04.354419+0000","last_unstale":"2024-04-25T00:01:06.340664+0000","last_undegraded":"2024-04-25T00:01:06.340664+0000","last_fullsized":"2024-04-25T00:01:06.340664+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:26.243635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.381029+0000","last_change":"2024-04-25T00:01:05.385670+0000","last_active":"2024-04-25T00:01:06.381029+0000","last_peered":"2024-04-25T00:01:06.381029+0000","last_clean":"2024-04-25T00:01:06.381029+0000","last_became_active":"2024-04-25T00:01:05.384822+0000","last_became_peered":"2024-04-25T00:01:05.384822+0000","last_unstale":"2024-04-25T00:01:06.381029+0000","last_undegraded":"2024-04-25T00:01:06.381029+0000","last_fullsized":"2024-04-25T00:01:06.381029+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:24.017263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.340457+0000","last_change":"2024-04-25T00:01:04.368954+0000","last_active":"2024-04-25T00:01:06.340457+0000","last_peered":"2024-04-25T00:01:06.340457+0000","last_clean":"2024-04-25T00:01:06.340457+0000","last_became_active":"2024-04-25T00:01:04.367903+0000","last_became_peered":"2024-04-25T00:01:04.367903+0000","last_unstale":"2024-04-25T00:01:06.340457+0000","last_undegraded":"2024-04-25T00:01:06.340457+0000","last_fullsized":"2024-04-25T00:01:06.340457+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.337227+0000","last_change":"2024-04-25T00:01:05.376609+0000","last_active":"2024-04-25T00:01:06.337227+0000","last_peered":"2024-04-25T00:01:06.337227+0000","last_clean":"2024-04-25T00:01:06.337227+0000","last_became_active":"2024-04-25T00:01:05.375881+0000","last_became_peered":"2024-04-25T00:01:05.375881+0000","last_unstale":"2024-04-25T00:01:06.337227+0000","last_undegraded":"2024-04-25T00:01:06.337227+0000","last_fullsized":"2024-04-25T00:01:06.337227+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:46:05.592233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":170,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378246+0000","last_change":"2024-04-25T00:00:55.084492+0000","last_active":"2024-04-25T00:01:06.378246+0000","last_peered":"2024-04-25T00:01:06.378246+0000","last_clean":"2024-04-25T00:01:06.378246+0000","last_became_active":"2024-04-25T00:00:55.083493+0000","last_became_peered":"2024-04-25T00:00:55.083493+0000","last_unstale":"2024-04-25T00:01:06.378246+0000","last_undegraded":"2024-04-25T00:01:06.378246+0000","last_fullsized":"2024-04-25T00:01:06.378246+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:00:54.060578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:00:54.060578+0000","last_clean_scrub_stamp":"2024-04-25T00:00:54.060578+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:14:58.472477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.336858+0000","last_change":"2024-04-25T00:01:04.357028+0000","last_active":"2024-04-25T00:01:06.336858+0000","last_peered":"2024-04-25T00:01:06.336858+0000","last_clean":"2024-04-25T00:01:06.336858+0000","last_became_active":"2024-04-25T00:01:04.356644+0000","last_became_peered":"2024-04-25T00:01:04.356644+0000","last_unstale":"2024-04-25T00:01:06.336858+0000","last_undegraded":"2024-04-25T00:01:06.336858+0000","last_fullsized":"2024-04-25T00:01:06.336858+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:21.729893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378588+0000","last_change":"2024-04-25T00:01:05.364939+0000","last_active":"2024-04-25T00:01:06.378588+0000","last_peered":"2024-04-25T00:01:06.378588+0000","last_clean":"2024-04-25T00:01:06.378588+0000","last_became_active":"2024-04-25T00:01:05.364272+0000","last_became_peered":"2024-04-25T00:01:05.364272+0000","last_unstale":"2024-04-25T00:01:06.378588+0000","last_undegraded":"2024-04-25T00:01:06.378588+0000","last_fullsized":"2024-04-25T00:01:06.378588+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:24:59.679908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.336259+0000","last_change":"2024-04-25T00:01:04.370226+0000","last_active":"2024-04-25T00:01:06.336259+0000","last_peered":"2024-04-25T00:01:06.336259+0000","last_clean":"2024-04-25T00:01:06.336259+0000","last_became_active":"2024-04-25T00:01:04.369904+0000","last_became_peered":"2024-04-25T00:01:04.369904+0000","last_unstale":"2024-04-25T00:01:06.336259+0000","last_undegraded":"2024-04-25T00:01:06.336259+0000","last_fullsized":"2024-04-25T00:01:06.336259+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:47:25.595275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.337599+0000","last_change":"2024-04-25T00:01:05.370968+0000","last_active":"2024-04-25T00:01:06.337599+0000","last_peered":"2024-04-25T00:01:06.337599+0000","last_clean":"2024-04-25T00:01:06.337599+0000","last_became_active":"2024-04-25T00:01:05.369987+0000","last_became_peered":"2024-04-25T00:01:05.369987+0000","last_unstale":"2024-04-25T00:01:06.337599+0000","last_undegraded":"2024-04-25T00:01:06.337599+0000","last_fullsized":"2024-04-25T00:01:06.337599+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.377854+0000","last_change":"2024-04-25T00:01:04.372916+0000","last_active":"2024-04-25T00:01:06.377854+0000","last_peered":"2024-04-25T00:01:06.377854+0000","last_clean":"2024-04-25T00:01:06.377854+0000","last_became_active":"2024-04-25T00:01:04.372185+0000","last_became_peered":"2024-04-25T00:01:04.372185+0000","last_unstale":"2024-04-25T00:01:06.377854+0000","last_undegraded":"2024-04-25T00:01:06.377854+0000","last_fullsized":"2024-04-25T00:01:06.377854+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.338998+0000","last_change":"2024-04-25T00:01:05.354557+0000","last_active":"2024-04-25T00:01:06.338998+0000","last_peered":"2024-04-25T00:01:06.338998+0000","last_clean":"2024-04-25T00:01:06.338998+0000","last_became_active":"2024-04-25T00:01:05.353556+0000","last_became_peered":"2024-04-25T00:01:05.353556+0000","last_unstale":"2024-04-25T00:01:06.338998+0000","last_undegraded":"2024-04-25T00:01:06.338998+0000","last_fullsized":"2024-04-25T00:01:06.338998+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:24.964055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.337431+0000","last_change":"2024-04-25T00:01:04.361900+0000","last_active":"2024-04-25T00:01:06.337431+0000","last_peered":"2024-04-25T00:01:06.337431+0000","last_clean":"2024-04-25T00:01:06.337431+0000","last_became_active":"2024-04-25T00:01:04.361572+0000","last_became_peered":"2024-04-25T00:01:04.361572+0000","last_unstale":"2024-04-25T00:01:06.337431+0000","last_undegraded":"2024-04-25T00:01:06.337431+0000","last_fullsized":"2024-04-25T00:01:06.337431+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:29.218739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380225+0000","last_change":"2024-04-25T00:01:05.368769+0000","last_active":"2024-04-25T00:01:06.380225+0000","last_peered":"2024-04-25T00:01:06.380225+0000","last_clean":"2024-04-25T00:01:06.380225+0000","last_became_active":"2024-04-25T00:01:05.366817+0000","last_became_peered":"2024-04-25T00:01:05.366817+0000","last_unstale":"2024-04-25T00:01:06.380225+0000","last_undegraded":"2024-04-25T00:01:06.380225+0000","last_fullsized":"2024-04-25T00:01:06.380225+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:11:27.044893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.339852+0000","last_change":"2024-04-25T00:01:04.366577+0000","last_active":"2024-04-25T00:01:06.339852+0000","last_peered":"2024-04-25T00:01:06.339852+0000","last_clean":"2024-04-25T00:01:06.339852+0000","last_became_active":"2024-04-25T00:01:04.365881+0000","last_became_peered":"2024-04-25T00:01:04.365881+0000","last_unstale":"2024-04-25T00:01:06.339852+0000","last_undegraded":"2024-04-25T00:01:06.339852+0000","last_fullsized":"2024-04-25T00:01:06.339852+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:49:22.673421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.377363+0000","last_change":"2024-04-25T00:01:05.378340+0000","last_active":"2024-04-25T00:01:06.377363+0000","last_peered":"2024-04-25T00:01:06.377363+0000","last_clean":"2024-04-25T00:01:06.377363+0000","last_became_active":"2024-04-25T00:01:05.377593+0000","last_became_peered":"2024-04-25T00:01:05.377593+0000","last_unstale":"2024-04-25T00:01:06.377363+0000","last_undegraded":"2024-04-25T00:01:06.377363+0000","last_fullsized":"2024-04-25T00:01:06.377363+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:48.202251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.337490+0000","last_change":"2024-04-25T00:01:04.365391+0000","last_active":"2024-04-25T00:01:06.337490+0000","last_peered":"2024-04-25T00:01:06.337490+0000","last_clean":"2024-04-25T00:01:06.337490+0000","last_became_active":"2024-04-25T00:01:04.364989+0000","last_became_peered":"2024-04-25T00:01:04.364989+0000","last_unstale":"2024-04-25T00:01:06.337490+0000","last_undegraded":"2024-04-25T00:01:06.337490+0000","last_fullsized":"2024-04-25T00:01:06.337490+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:06:18.298057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.340089+0000","last_change":"2024-04-25T00:01:05.374011+0000","last_active":"2024-04-25T00:01:06.340089+0000","last_peered":"2024-04-25T00:01:06.340089+0000","last_clean":"2024-04-25T00:01:06.340089+0000","last_became_active":"2024-04-25T00:01:05.373276+0000","last_became_peered":"2024-04-25T00:01:05.373276+0000","last_unstale":"2024-04-25T00:01:06.340089+0000","last_undegraded":"2024-04-25T00:01:06.340089+0000","last_fullsized":"2024-04-25T00:01:06.340089+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:40:53.349428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"23'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.379136+0000","last_change":"2024-04-25T00:01:04.361603+0000","last_active":"2024-04-25T00:01:06.379136+0000","last_peered":"2024-04-25T00:01:06.379136+0000","last_clean":"2024-04-25T00:01:06.379136+0000","last_became_active":"2024-04-25T00:01:04.361046+0000","last_became_peered":"2024-04-25T00:01:04.361046+0000","last_unstale":"2024-04-25T00:01:06.379136+0000","last_undegraded":"2024-04-25T00:01:06.379136+0000","last_fullsized":"2024-04-25T00:01:06.379136+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:35:18.036843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378489+0000","last_change":"2024-04-25T00:01:05.364829+0000","last_active":"2024-04-25T00:01:06.378489+0000","last_peered":"2024-04-25T00:01:06.378489+0000","last_clean":"2024-04-25T00:01:06.378489+0000","last_became_active":"2024-04-25T00:01:05.364125+0000","last_became_peered":"2024-04-25T00:01:05.364125+0000","last_unstale":"2024-04-25T00:01:06.378489+0000","last_undegraded":"2024-04-25T00:01:06.378489+0000","last_fullsized":"2024-04-25T00:01:06.378489+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:15:45.531760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"22'2","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.337655+0000","last_change":"2024-04-25T00:01:04.369917+0000","last_active":"2024-04-25T00:01:06.337655+0000","last_peered":"2024-04-25T00:01:06.337655+0000","last_clean":"2024-04-25T00:01:06.337655+0000","last_became_active":"2024-04-25T00:01:04.369309+0000","last_became_peered":"2024-04-25T00:01:04.369309+0000","last_unstale":"2024-04-25T00:01:06.337655+0000","last_undegraded":"2024-04-25T00:01:06.337655+0000","last_fullsized":"2024-04-25T00:01:06.337655+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:55:08.063965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378696+0000","last_change":"2024-04-25T00:01:05.358106+0000","last_active":"2024-04-25T00:01:06.378696+0000","last_peered":"2024-04-25T00:01:06.378696+0000","last_clean":"2024-04-25T00:01:06.378696+0000","last_became_active":"2024-04-25T00:01:05.357135+0000","last_became_peered":"2024-04-25T00:01:05.357135+0000","last_unstale":"2024-04-25T00:01:06.378696+0000","last_undegraded":"2024-04-25T00:01:06.378696+0000","last_fullsized":"2024-04-25T00:01:06.378696+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:41:42.228234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378173+0000","last_change":"2024-04-25T00:01:04.365515+0000","last_active":"2024-04-25T00:01:06.378173+0000","last_peered":"2024-04-25T00:01:06.378173+0000","last_clean":"2024-04-25T00:01:06.378173+0000","last_became_active":"2024-04-25T00:01:04.364942+0000","last_became_peered":"2024-04-25T00:01:04.364942+0000","last_unstale":"2024-04-25T00:01:06.378173+0000","last_undegraded":"2024-04-25T00:01:06.378173+0000","last_fullsized":"2024-04-25T00:01:06.378173+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.336035+0000","last_change":"2024-04-25T00:01:05.354953+0000","last_active":"2024-04-25T00:01:06.336035+0000","last_peered":"2024-04-25T00:01:06.336035+0000","last_clean":"2024-04-25T00:01:06.336035+0000","last_became_active":"2024-04-25T00:01:05.354118+0000","last_became_peered":"2024-04-25T00:01:05.354118+0000","last_unstale":"2024-04-25T00:01:06.336035+0000","last_undegraded":"2024-04-25T00:01:06.336035+0000","last_fullsized":"2024-04-25T00:01:06.336035+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:09.858617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378070+0000","last_change":"2024-04-25T00:01:04.362268+0000","last_active":"2024-04-25T00:01:06.378070+0000","last_peered":"2024-04-25T00:01:06.378070+0000","last_clean":"2024-04-25T00:01:06.378070+0000","last_became_active":"2024-04-25T00:01:04.361622+0000","last_became_peered":"2024-04-25T00:01:04.361622+0000","last_unstale":"2024-04-25T00:01:06.378070+0000","last_undegraded":"2024-04-25T00:01:06.378070+0000","last_fullsized":"2024-04-25T00:01:06.378070+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:58:46.021343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.377525+0000","last_change":"2024-04-25T00:01:05.355455+0000","last_active":"2024-04-25T00:01:06.377525+0000","last_peered":"2024-04-25T00:01:06.377525+0000","last_clean":"2024-04-25T00:01:06.377525+0000","last_became_active":"2024-04-25T00:01:05.354699+0000","last_became_peered":"2024-04-25T00:01:05.354699+0000","last_unstale":"2024-04-25T00:01:06.377525+0000","last_undegraded":"2024-04-25T00:01:06.377525+0000","last_fullsized":"2024-04-25T00:01:06.377525+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:43:58.069271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.338480+0000","last_change":"2024-04-25T00:01:04.371935+0000","last_active":"2024-04-25T00:01:06.338480+0000","last_peered":"2024-04-25T00:01:06.338480+0000","last_clean":"2024-04-25T00:01:06.338480+0000","last_became_active":"2024-04-25T00:01:04.371420+0000","last_became_peered":"2024-04-25T00:01:04.371420+0000","last_unstale":"2024-04-25T00:01:06.338480+0000","last_undegraded":"2024-04-25T00:01:06.338480+0000","last_fullsized":"2024-04-25T00:01:06.338480+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378351+0000","last_change":"2024-04-25T00:01:05.362611+0000","last_active":"2024-04-25T00:01:06.378351+0000","last_peered":"2024-04-25T00:01:06.378351+0000","last_clean":"2024-04-25T00:01:06.378351+0000","last_became_active":"2024-04-25T00:01:05.361817+0000","last_became_peered":"2024-04-25T00:01:05.361817+0000","last_unstale":"2024-04-25T00:01:06.378351+0000","last_undegraded":"2024-04-25T00:01:06.378351+0000","last_fullsized":"2024-04-25T00:01:06.378351+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:50.489125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"22'2","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.336722+0000","last_change":"2024-04-25T00:01:04.359262+0000","last_active":"2024-04-25T00:01:06.336722+0000","last_peered":"2024-04-25T00:01:06.336722+0000","last_clean":"2024-04-25T00:01:06.336722+0000","last_became_active":"2024-04-25T00:01:04.357971+0000","last_became_peered":"2024-04-25T00:01:04.357971+0000","last_unstale":"2024-04-25T00:01:06.336722+0000","last_undegraded":"2024-04-25T00:01:06.336722+0000","last_fullsized":"2024-04-25T00:01:06.336722+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:05.441048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.336709+0000","last_change":"2024-04-25T00:01:05.376984+0000","last_active":"2024-04-25T00:01:06.336709+0000","last_peered":"2024-04-25T00:01:06.336709+0000","last_clean":"2024-04-25T00:01:06.336709+0000","last_became_active":"2024-04-25T00:01:05.376259+0000","last_became_peered":"2024-04-25T00:01:05.376259+0000","last_unstale":"2024-04-25T00:01:06.336709+0000","last_undegraded":"2024-04-25T00:01:06.336709+0000","last_fullsized":"2024-04-25T00:01:06.336709+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:13:28.543302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.336976+0000","last_change":"2024-04-25T00:01:05.381120+0000","last_active":"2024-04-25T00:01:06.336976+0000","last_peered":"2024-04-25T00:01:06.336976+0000","last_clean":"2024-04-25T00:01:06.336976+0000","last_became_active":"2024-04-25T00:01:05.378320+0000","last_became_peered":"2024-04-25T00:01:05.378320+0000","last_unstale":"2024-04-25T00:01:06.336976+0000","last_undegraded":"2024-04-25T00:01:06.336976+0000","last_fullsized":"2024-04-25T00:01:06.336976+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:34:06.175918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"22'2","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380832+0000","last_change":"2024-04-25T00:01:04.362832+0000","last_active":"2024-04-25T00:01:06.380832+0000","last_peered":"2024-04-25T00:01:06.380832+0000","last_clean":"2024-04-25T00:01:06.380832+0000","last_became_active":"2024-04-25T00:01:04.362059+0000","last_became_peered":"2024-04-25T00:01:04.362059+0000","last_unstale":"2024-04-25T00:01:06.380832+0000","last_undegraded":"2024-04-25T00:01:06.380832+0000","last_fullsized":"2024-04-25T00:01:06.380832+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:05.441048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.379969+0000","last_change":"2024-04-25T00:01:05.387195+0000","last_active":"2024-04-25T00:01:06.379969+0000","last_peered":"2024-04-25T00:01:06.379969+0000","last_clean":"2024-04-25T00:01:06.379969+0000","last_became_active":"2024-04-25T00:01:05.386853+0000","last_became_peered":"2024-04-25T00:01:05.386853+0000","last_unstale":"2024-04-25T00:01:06.379969+0000","last_undegraded":"2024-04-25T00:01:06.379969+0000","last_fullsized":"2024-04-25T00:01:06.379969+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:06:19.307378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"22'2","reported_seq":16,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.380527+0000","last_change":"2024-04-25T00:01:04.362030+0000","last_active":"2024-04-25T00:01:06.380527+0000","last_peered":"2024-04-25T00:01:06.380527+0000","last_clean":"2024-04-25T00:01:06.380527+0000","last_became_active":"2024-04-25T00:01:04.361708+0000","last_became_peered":"2024-04-25T00:01:04.361708+0000","last_unstale":"2024-04-25T00:01:06.380527+0000","last_undegraded":"2024-04-25T00:01:06.380527+0000","last_fullsized":"2024-04-25T00:01:06.380527+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:38:33.392719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.340575+0000","last_change":"2024-04-25T00:01:05.370818+0000","last_active":"2024-04-25T00:01:06.340575+0000","last_peered":"2024-04-25T00:01:06.340575+0000","last_clean":"2024-04-25T00:01:06.340575+0000","last_became_active":"2024-04-25T00:01:05.369991+0000","last_became_peered":"2024-04-25T00:01:05.369991+0000","last_unstale":"2024-04-25T00:01:06.340575+0000","last_undegraded":"2024-04-25T00:01:06.340575+0000","last_fullsized":"2024-04-25T00:01:06.340575+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:43.615895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"22'1","reported_seq":15,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.340582+0000","last_change":"2024-04-25T00:01:04.361809+0000","last_active":"2024-04-25T00:01:06.340582+0000","last_peered":"2024-04-25T00:01:06.340582+0000","last_clean":"2024-04-25T00:01:06.340582+0000","last_became_active":"2024-04-25T00:01:04.361059+0000","last_became_peered":"2024-04-25T00:01:04.361059+0000","last_unstale":"2024-04-25T00:01:06.340582+0000","last_undegraded":"2024-04-25T00:01:06.340582+0000","last_fullsized":"2024-04-25T00:01:06.340582+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.340280+0000","last_change":"2024-04-25T00:01:05.367207+0000","last_active":"2024-04-25T00:01:06.340280+0000","last_peered":"2024-04-25T00:01:06.340280+0000","last_clean":"2024-04-25T00:01:06.340280+0000","last_became_active":"2024-04-25T00:01:05.366471+0000","last_became_peered":"2024-04-25T00:01:05.366471+0000","last_unstale":"2024-04-25T00:01:06.340280+0000","last_undegraded":"2024-04-25T00:01:06.340280+0000","last_fullsized":"2024-04-25T00:01:06.340280+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:30:23.557381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"22'4","reported_seq":18,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-04-25T00:01:06.378766+0000","last_change":"2024-04-25T00:01:04.356032+0000","last_active":"2024-04-25T00:01:06.378766+0000","last_peered":"2024-04-25T00:01:06.378766+0000","last_clean":"2024-04-25T00:01:06.378766+0000","last_became_active":"2024-04-25T00:01:04.355293+0000","last_became_peered":"2024-04-25T00:01:04.355293+0000","last_unstale":"2024-04-25T00:01:06.378766+0000","last_undegraded":"2024-04-25T00:01:06.378766+0000","last_fullsized":"2024-04-25T00:01:06.378766+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:56:16.678132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509447,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":241664,"data_stored":81279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509447,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":233472,"data_stored":81155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509446,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27844,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343996,"statfs":{"total":96636764160,"available":96608251904,"internally_reserved":0,"allocated":835584,"data_stored":671615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509447,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":237568,"data_stored":81737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509448,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":245760,"data_stored":82767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509447,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27832,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344008,"statfs":{"total":96636764160,"available":96608264192,"internally_reserved":0,"allocated":831488,"data_stored":671593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509447,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27260,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344580,"statfs":{"total":96636764160,"available":96608849920,"internally_reserved":0,"allocated":241664,"data_stored":81211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":253952,"data_stored":83439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T00:01:15.193 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-25T00:01:15.194 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T00:01:15.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.354+0000 7f49474b6640 1 Processor -- start 2024-04-25T00:01:15.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.354+0000 7f49474b6640 1 -- start start 2024-04-25T00:01:15.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.354+0000 7f49474b6640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4940152d80 0x7f494006bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:15.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.354+0000 7f49474b6640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f494006c290 0x7f4940157390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:15.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.354+0000 7f49474b6640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49400c9150 con 0x7f494006d5f0 2024-04-25T00:01:15.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.354+0000 7f49474b6640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f49400c9d60 con 0x7f494006c290 2024-04-25T00:01:15.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.354+0000 7f49474b6640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f49400c9430 con 0x7f4940152d80 2024-04-25T00:01:15.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.358+0000 7f4945a53640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f494006c290 0x7f4940157390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:15.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.358+0000 7f4945252640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4940152d80 0x7f494006bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:15.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.358+0000 7f4945a53640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f494006c290 0x7f4940157390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:51042/0 (socket says 172.21.15.116:51042) 2024-04-25T00:01:15.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.358+0000 7f4945a53640 1 -- 172.21.15.116:0/2090419450 learned_addr learned my addr 172.21.15.116:0/2090419450 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:15.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.358+0000 7f4944a51640 1 --1- 172.21.15.116:0/2090419450 >> v1:172.21.15.174:6789/0 conn(0x7f494006d5f0 0x7f4940156c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:54530/0 (socket says 172.21.15.116:54530) 2024-04-25T00:01:15.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.358+0000 7f4945a53640 1 -- 172.21.15.116:0/2090419450 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4940152d80 msgr2=0x7f494006bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:15.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.358+0000 7f4945a53640 1 --2- 172.21.15.116:0/2090419450 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4940152d80 0x7f494006bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:15.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.358+0000 7f4945a53640 1 -- 172.21.15.116:0/2090419450 >> v1:172.21.15.174:6789/0 conn(0x7f494006d5f0 legacy=0x7f4940156c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:15.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.358+0000 7f4945a53640 1 -- 172.21.15.116:0/2090419450 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49400c9710 con 0x7f494006c290 2024-04-25T00:01:15.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.358+0000 7f4945a53640 1 --2- 172.21.15.116:0/2090419450 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f494006c290 0x7f4940157390 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4938009e50 tx=0x7f493801c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=830349348d6b589b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:15.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.358+0000 7f4937fff640 1 -- 172.21.15.116:0/2090419450 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49380a5850 con 0x7f494006c290 2024-04-25T00:01:15.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.358+0000 7f4937fff640 1 -- 172.21.15.116:0/2090419450 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49380a59f0 con 0x7f494006c290 2024-04-25T00:01:15.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.358+0000 7f4937fff640 1 -- 172.21.15.116:0/2090419450 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49380a5dc0 con 0x7f494006c290 2024-04-25T00:01:15.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.362+0000 7f49474b6640 1 -- 172.21.15.116:0/2090419450 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f494006c290 msgr2=0x7f4940157390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:15.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.362+0000 7f49474b6640 1 --2- 172.21.15.116:0/2090419450 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f494006c290 0x7f4940157390 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4938009e50 tx=0x7f493801c970 comp rx=0 tx=0).stop 2024-04-25T00:01:15.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.362+0000 7f49474b6640 1 -- 172.21.15.116:0/2090419450 shutdown_connections 2024-04-25T00:01:15.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.362+0000 7f49474b6640 1 --2- 172.21.15.116:0/2090419450 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f494006c290 0x7f4940157390 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:15.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.362+0000 7f49474b6640 1 --2- 172.21.15.116:0/2090419450 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4940152d80 0x7f494006bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:15.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.362+0000 7f49474b6640 1 -- 172.21.15.116:0/2090419450 >> 172.21.15.116:0/2090419450 conn(0x7f49400acdd0 msgr2=0x7f49400ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:15.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.362+0000 7f49474b6640 1 -- 172.21.15.116:0/2090419450 shutdown_connections 2024-04-25T00:01:15.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.362+0000 7f49474b6640 1 -- 172.21.15.116:0/2090419450 wait complete. 2024-04-25T00:01:15.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.362+0000 7f49474b6640 1 Processor -- start 2024-04-25T00:01:15.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.362+0000 7f49474b6640 1 -- start start 2024-04-25T00:01:15.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.362+0000 7f49474b6640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f494006c290 0x7f49400c6e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:15.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.362+0000 7f49474b6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f494006d5f0 0x7f49400ae060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:15.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.362+0000 7f49474b6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4940152d80 0x7f49400ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:15.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.362+0000 7f49474b6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f49400caf00 con 0x7f494006d5f0 2024-04-25T00:01:15.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.362+0000 7f49474b6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f49400cad80 con 0x7f4940152d80 2024-04-25T00:01:15.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.362+0000 7f49474b6640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f49401628d0 con 0x7f494006c290 2024-04-25T00:01:15.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.362+0000 7f4944a51640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f494006d5f0 0x7f49400ae060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:15.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.362+0000 7f4944a51640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f494006d5f0 0x7f49400ae060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54220/0 (socket says 172.21.15.116:54220) 2024-04-25T00:01:15.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.362+0000 7f4944a51640 1 -- 172.21.15.116:0/3759764758 learned_addr learned my addr 172.21.15.116:0/3759764758 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:15.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.362+0000 7f4945a53640 1 --2- 172.21.15.116:0/3759764758 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4940152d80 0x7f49400ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:15.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.362+0000 7f4945252640 1 --2- 172.21.15.116:0/3759764758 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f494006c290 0x7f49400c6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:15.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.362+0000 7f4944a51640 1 -- 172.21.15.116:0/3759764758 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f494006c290 msgr2=0x7f49400c6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:15.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.362+0000 7f4944a51640 1 --2- 172.21.15.116:0/3759764758 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f494006c290 0x7f49400c6e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:15.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.362+0000 7f4944a51640 1 -- 172.21.15.116:0/3759764758 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4940152d80 msgr2=0x7f49400ae5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:15.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.362+0000 7f4944a51640 1 --2- 172.21.15.116:0/3759764758 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4940152d80 0x7f49400ae5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:15.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.362+0000 7f4944a51640 1 -- 172.21.15.116:0/3759764758 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49400c59d0 con 0x7f494006d5f0 2024-04-25T00:01:15.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.362+0000 7f4944a51640 1 --2- 172.21.15.116:0/3759764758 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f494006d5f0 0x7f49400ae060 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f493000c370 tx=0x7f493000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:15.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.362+0000 7f4935ffb640 1 -- 172.21.15.116:0/3759764758 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49300074e0 con 0x7f494006d5f0 2024-04-25T00:01:15.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.362+0000 7f4935ffb640 1 -- 172.21.15.116:0/3759764758 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4930007680 con 0x7f494006d5f0 2024-04-25T00:01:15.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.362+0000 7f49474b6640 1 -- 172.21.15.116:0/3759764758 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49400c5cb0 con 0x7f494006d5f0 2024-04-25T00:01:15.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.362+0000 7f4935ffb640 1 -- 172.21.15.116:0/3759764758 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4930004a30 con 0x7f494006d5f0 2024-04-25T00:01:15.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.362+0000 7f49474b6640 1 -- 172.21.15.116:0/3759764758 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49400c6270 con 0x7f494006d5f0 2024-04-25T00:01:15.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.362+0000 7f49474b6640 1 -- 172.21.15.116:0/3759764758 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4940068350 con 0x7f494006d5f0 2024-04-25T00:01:15.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.366+0000 7f4935ffb640 1 -- 172.21.15.116:0/3759764758 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f493001a020 con 0x7f494006d5f0 2024-04-25T00:01:15.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.366+0000 7f4935ffb640 1 --2- 172.21.15.116:0/3759764758 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f49240678c0 0x7f4924069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:15.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.366+0000 7f4935ffb640 1 -- 172.21.15.116:0/3759764758 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5689+0+0 (secure 0 0 0) 0x7f4930085380 con 0x7f494006d5f0 2024-04-25T00:01:15.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.366+0000 7f4945252640 1 --2- 172.21.15.116:0/3759764758 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f49240678c0 0x7f4924069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:15.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.366+0000 7f4945252640 1 --2- 172.21.15.116:0/3759764758 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f49240678c0 0x7f4924069d80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f493c009af0 tx=0x7f493c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:15.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.370+0000 7f4935ffb640 1 -- 172.21.15.116:0/3759764758 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f4930055f60 con 0x7f494006d5f0 2024-04-25T00:01:15.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.530+0000 7f49474b6640 1 -- 172.21.15.116:0/3759764758 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f49400c6580 con 0x7f494006d5f0 2024-04-25T00:01:15.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.534+0000 7f4935ffb640 1 -- 172.21.15.116:0/3759764758 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) ==== 74+0+16613 (secure 0 0 0) 0x7f4930055860 con 0x7f494006d5f0 2024-04-25T00:01:15.535 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:01:15.535 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":23,"fsid":"6d6a1aa7-0478-4c83-8057-05917e81f046","created":"2024-04-25T00:00:40.682493+0000","modified":"2024-04-25T00:01:06.329793+0000","last_up_change":"2024-04-25T00:00:51.822174+0000","last_in_change":"2024-04-25T00:00:42.980473+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T00:00:53.324376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-04-25T00:01:03.197326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-04-25T00:01:03.728618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8b6ddcb2-1ad9-4167-961c-bd6936bb9060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6809","nonce":3751765682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6811","nonce":3751765682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6815","nonce":3751765682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6813","nonce":3751765682}]},"public_addr":"172.21.15.116:6809/3751765682","cluster_addr":"172.21.15.116:6811/3751765682","heartbeat_back_addr":"172.21.15.116:6815/3751765682","heartbeat_front_addr":"172.21.15.116:6813/3751765682","state":["exists","up"]},{"osd":1,"uuid":"a6431768-994f-41da-8853-886f5687fb91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6801","nonce":377879124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6803","nonce":377879124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6807","nonce":377879124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6805","nonce":377879124}]},"public_addr":"172.21.15.116:6801/377879124","cluster_addr":"172.21.15.116:6803/377879124","heartbeat_back_addr":"172.21.15.116:6807/377879124","heartbeat_front_addr":"172.21.15.116:6805/377879124","state":["exists","up"]},{"osd":2,"uuid":"4f69bd40-5a23-45ff-8693-e1de75e1aa9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6817","nonce":1099282267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6819","nonce":1099282267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6823","nonce":1099282267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6821","nonce":1099282267}]},"public_addr":"172.21.15.116:6817/1099282267","cluster_addr":"172.21.15.116:6819/1099282267","heartbeat_back_addr":"172.21.15.116:6823/1099282267","heartbeat_front_addr":"172.21.15.116:6821/1099282267","state":["exists","up"]},{"osd":3,"uuid":"6a0d5344-8b6f-40d3-918d-0f0a4be688e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6825","nonce":1179942661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6827","nonce":1179942661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6831","nonce":1179942661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6829","nonce":1179942661}]},"public_addr":"172.21.15.116:6825/1179942661","cluster_addr":"172.21.15.116:6827/1179942661","heartbeat_back_addr":"172.21.15.116:6831/1179942661","heartbeat_front_addr":"172.21.15.116:6829/1179942661","state":["exists","up"]},{"osd":4,"uuid":"132e92e4-b825-441e-965a-ca00533fa86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6801","nonce":3132444330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6803","nonce":3132444330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6807","nonce":3132444330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6805","nonce":3132444330}]},"public_addr":"172.21.15.174:6801/3132444330","cluster_addr":"172.21.15.174:6803/3132444330","heartbeat_back_addr":"172.21.15.174:6807/3132444330","heartbeat_front_addr":"172.21.15.174:6805/3132444330","state":["exists","up"]},{"osd":5,"uuid":"a2eeb5b6-1b31-43cc-8306-a0adf3ee15bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6809","nonce":881817164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6811","nonce":881817164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6815","nonce":881817164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6813","nonce":881817164}]},"public_addr":"172.21.15.174:6809/881817164","cluster_addr":"172.21.15.174:6811/881817164","heartbeat_back_addr":"172.21.15.174:6815/881817164","heartbeat_front_addr":"172.21.15.174:6813/881817164","state":["exists","up"]},{"osd":6,"uuid":"588e873a-c841-4ba7-90f2-8956c87a9983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6825","nonce":117574251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6827","nonce":117574251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6831","nonce":117574251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6829","nonce":117574251}]},"public_addr":"172.21.15.174:6825/117574251","cluster_addr":"172.21.15.174:6827/117574251","heartbeat_back_addr":"172.21.15.174:6831/117574251","heartbeat_front_addr":"172.21.15.174:6829/117574251","state":["exists","up"]},{"osd":7,"uuid":"3b90f9b0-8a0e-4883-b088-ca1d4012870c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6817","nonce":2543899196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6819","nonce":2543899196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6823","nonce":2543899196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6821","nonce":2543899196}]},"public_addr":"172.21.15.174:6817/2543899196","cluster_addr":"172.21.15.174:6819/2543899196","heartbeat_back_addr":"172.21.15.174:6823/2543899196","heartbeat_front_addr":"172.21.15.174:6821/2543899196","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.293769+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.310034+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.404865+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.549658+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.192174+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.297266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.493649+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.452309+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:01:15.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.534+0000 7f49474b6640 1 -- 172.21.15.116:0/3759764758 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f49240678c0 msgr2=0x7f4924069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:15.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.534+0000 7f49474b6640 1 --2- 172.21.15.116:0/3759764758 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f49240678c0 0x7f4924069d80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f493c009af0 tx=0x7f493c008040 comp rx=0 tx=0).stop 2024-04-25T00:01:15.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.534+0000 7f49474b6640 1 -- 172.21.15.116:0/3759764758 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f494006d5f0 msgr2=0x7f49400ae060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:15.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.534+0000 7f49474b6640 1 --2- 172.21.15.116:0/3759764758 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f494006d5f0 0x7f49400ae060 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f493000c370 tx=0x7f493000c830 comp rx=0 tx=0).stop 2024-04-25T00:01:15.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.534+0000 7f49474b6640 1 -- 172.21.15.116:0/3759764758 shutdown_connections 2024-04-25T00:01:15.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.534+0000 7f49474b6640 1 --2- 172.21.15.116:0/3759764758 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f49240678c0 0x7f4924069d80 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:15.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.534+0000 7f49474b6640 1 --2- 172.21.15.116:0/3759764758 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4940152d80 0x7f49400ae5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:15.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.534+0000 7f49474b6640 1 --2- 172.21.15.116:0/3759764758 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f494006d5f0 0x7f49400ae060 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:15.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.534+0000 7f49474b6640 1 --2- 172.21.15.116:0/3759764758 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f494006c290 0x7f49400c6e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:15.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.534+0000 7f49474b6640 1 -- 172.21.15.116:0/3759764758 >> 172.21.15.116:0/3759764758 conn(0x7f49400acdd0 msgr2=0x7f494014e310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:15.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.534+0000 7f49474b6640 1 -- 172.21.15.116:0/3759764758 shutdown_connections 2024-04-25T00:01:15.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.534+0000 7f49474b6640 1 -- 172.21.15.116:0/3759764758 wait complete. 2024-04-25T00:01:15.555 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T00:00:53.324376+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'a_metadata', 'create_time': '2024-04-25T00:01:03.197326+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'a'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'a_data', 'create_time': '2024-04-25T00:01:03.728618+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'a'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T00:01:15.556 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T00:01:15.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.674+0000 7f12fe554640 1 Processor -- start 2024-04-25T00:01:15.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.674+0000 7f12fe554640 1 -- start start 2024-04-25T00:01:15.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.678+0000 7f12fe554640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f12f8157510 0x7f12f815ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:15.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.678+0000 7f12fe554640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12f80a9cb0 con 0x7f12f8150d60 2024-04-25T00:01:15.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.678+0000 7f12fe554640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12f80a9f90 con 0x7f12f80a9630 2024-04-25T00:01:15.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.678+0000 7f12fe554640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f12f80aa270 con 0x7f12f8157510 2024-04-25T00:01:15.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.678+0000 7f12f7fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f12f8150d60 0x7f12f80c2930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:60828/0 (socket says 172.21.15.116:60828) 2024-04-25T00:01:15.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.678+0000 7f12f7fff640 1 -- 172.21.15.116:0/4167741699 learned_addr learned my addr 172.21.15.116:0/4167741699 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:15.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.678+0000 7f12f6ffd640 1 -- 172.21.15.116:0/4167741699 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2524058708 0 0) 0x7f12f80a9cb0 con 0x7f12f8150d60 2024-04-25T00:01:15.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.678+0000 7f12f6ffd640 1 -- 172.21.15.116:0/4167741699 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12e4003530 con 0x7f12f8150d60 2024-04-25T00:01:15.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.678+0000 7f12fcaf1640 1 --2- 172.21.15.116:0/4167741699 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f12f8157510 0x7f12f815ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:15.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.678+0000 7f12f6ffd640 1 -- 172.21.15.116:0/4167741699 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3743166541 0 0) 0x7f12e4003530 con 0x7f12f8150d60 2024-04-25T00:01:15.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.678+0000 7f12f6ffd640 1 -- 172.21.15.116:0/4167741699 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f12f8157510 msgr2=0x7f12f815ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:15.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.678+0000 7f12f6ffd640 1 --2- 172.21.15.116:0/4167741699 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f12f8157510 0x7f12f815ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:15.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.678+0000 7f12f6ffd640 1 -- 172.21.15.116:0/4167741699 >> v1:172.21.15.174:6789/0 conn(0x7f12f80a9630 legacy=0x7f12f8156e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:15.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.678+0000 7f12f6ffd640 1 -- 172.21.15.116:0/4167741699 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12f80aa550 con 0x7f12f8150d60 2024-04-25T00:01:15.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.678+0000 7f12f6ffd640 1 -- 172.21.15.116:0/4167741699 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f12f0001a00 con 0x7f12f8150d60 2024-04-25T00:01:15.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.678+0000 7f12f6ffd640 1 -- 172.21.15.116:0/4167741699 >> v1:172.21.15.116:6789/0 conn(0x7f12f8150d60 legacy=0x7f12f80c2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:15.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.678+0000 7f12f6ffd640 1 --2- 172.21.15.116:0/4167741699 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f12e40039f0 0x7f12e4003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:15.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.678+0000 7f12f6ffd640 1 --2- 172.21.15.116:0/4167741699 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12e4004810 0x7f12e4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:15.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.678+0000 7f12f6ffd640 1 --2- 172.21.15.116:0/4167741699 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f12f8157510 0x7f12e400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:15.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.678+0000 7f12f6ffd640 1 -- 172.21.15.116:0/4167741699 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f12f80aa270 con 0x7f12e4004810 2024-04-25T00:01:15.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.678+0000 7f12f6ffd640 1 -- 172.21.15.116:0/4167741699 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f12e4005550 con 0x7f12e40039f0 2024-04-25T00:01:15.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.678+0000 7f12f6ffd640 1 -- 172.21.15.116:0/4167741699 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f12e400bf10 con 0x7f12f8157510 2024-04-25T00:01:15.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.678+0000 7f12f77fe640 1 --2- 172.21.15.116:0/4167741699 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12e4004810 0x7f12e4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:15.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.678+0000 7f12f7fff640 1 --2- 172.21.15.116:0/4167741699 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f12e40039f0 0x7f12e4003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:15.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.678+0000 7f12fcaf1640 1 --2- 172.21.15.116:0/4167741699 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f12f8157510 0x7f12e400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:15.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.678+0000 7f12f77fe640 1 -- 172.21.15.116:0/4167741699 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f12f8157510 msgr2=0x7f12e400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:15.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.678+0000 7f12f77fe640 1 --2- 172.21.15.116:0/4167741699 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f12f8157510 0x7f12e400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:15.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.678+0000 7f12f77fe640 1 -- 172.21.15.116:0/4167741699 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f12e40039f0 msgr2=0x7f12e4003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:15.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.678+0000 7f12f77fe640 1 --2- 172.21.15.116:0/4167741699 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f12e40039f0 0x7f12e4003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:15.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.678+0000 7f12f77fe640 1 -- 172.21.15.116:0/4167741699 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12e4003530 con 0x7f12e4004810 2024-04-25T00:01:15.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.678+0000 7f12f7fff640 1 --2- 172.21.15.116:0/4167741699 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f12e40039f0 0x7f12e4003e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:01:15.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.678+0000 7f12fcaf1640 1 --2- 172.21.15.116:0/4167741699 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f12f8157510 0x7f12e400b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:15.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.678+0000 7f12f77fe640 1 --2- 172.21.15.116:0/4167741699 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12e4004810 0x7f12e4004c90 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f12e0001780 tx=0x7f12e001c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f77d0c7a87a7ccf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:15.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.682+0000 7f12f6ffd640 1 -- 172.21.15.116:0/4167741699 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e00013a0 con 0x7f12e4004810 2024-04-25T00:01:15.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.682+0000 7f12f6ffd640 1 -- 172.21.15.116:0/4167741699 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12e00052f0 con 0x7f12e4004810 2024-04-25T00:01:15.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.682+0000 7f12f6ffd640 1 -- 172.21.15.116:0/4167741699 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e0003840 con 0x7f12e4004810 2024-04-25T00:01:15.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.682+0000 7f12fe554640 1 -- 172.21.15.116:0/4167741699 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12e4004810 msgr2=0x7f12e4004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:15.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.682+0000 7f12fe554640 1 --2- 172.21.15.116:0/4167741699 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12e4004810 0x7f12e4004c90 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f12e0001780 tx=0x7f12e001c3f0 comp rx=0 tx=0).stop 2024-04-25T00:01:15.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.682+0000 7f12fcaf1640 1 -- 172.21.15.116:0/4167741699 reap_dead start 2024-04-25T00:01:15.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.682+0000 7f12fe554640 1 -- 172.21.15.116:0/4167741699 shutdown_connections 2024-04-25T00:01:15.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.682+0000 7f12fe554640 1 -- 172.21.15.116:0/4167741699 >> 172.21.15.116:0/4167741699 conn(0x7f12f80c3a20 msgr2=0x7f12f80c3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:15.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.682+0000 7f12fe554640 1 -- 172.21.15.116:0/4167741699 shutdown_connections 2024-04-25T00:01:15.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.682+0000 7f12fe554640 1 -- 172.21.15.116:0/4167741699 wait complete. 2024-04-25T00:01:15.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.682+0000 7f12fe554640 1 Processor -- start 2024-04-25T00:01:15.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.682+0000 7f12fe554640 1 -- start start 2024-04-25T00:01:15.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.682+0000 7f12fe554640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f12e4004810 0x7f12f814fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:15.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.682+0000 7f12fe554640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12f813f9e0 0x7f12f8142b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:15.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.682+0000 7f12fe554640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f12f81efc40 0x7f12f81f00c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:15.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.682+0000 7f12fe554640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f12f8072150 con 0x7f12f813f9e0 2024-04-25T00:01:15.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.682+0000 7f12fe554640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f12f8071fd0 con 0x7f12f81efc40 2024-04-25T00:01:15.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.682+0000 7f12fe554640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f12f8157920 con 0x7f12e4004810 2024-04-25T00:01:15.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.682+0000 7f12f77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12f813f9e0 0x7f12f8142b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:15.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.682+0000 7f12f77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12f813f9e0 0x7f12f8142b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54234/0 (socket says 172.21.15.116:54234) 2024-04-25T00:01:15.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.682+0000 7f12f77fe640 1 -- 172.21.15.116:0/2119376631 learned_addr learned my addr 172.21.15.116:0/2119376631 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:15.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.682+0000 7f12f7fff640 1 --2- 172.21.15.116:0/2119376631 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f12e4004810 0x7f12f814fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:15.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.682+0000 7f12fcaf1640 1 --2- 172.21.15.116:0/2119376631 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f12f81efc40 0x7f12f81f00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:15.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.682+0000 7f12f77fe640 1 -- 172.21.15.116:0/2119376631 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f12e4004810 msgr2=0x7f12f814fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:15.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.682+0000 7f12f77fe640 1 --2- 172.21.15.116:0/2119376631 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f12e4004810 0x7f12f814fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:15.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.682+0000 7f12f77fe640 1 -- 172.21.15.116:0/2119376631 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f12f81efc40 msgr2=0x7f12f81f00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:15.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.682+0000 7f12f77fe640 1 --2- 172.21.15.116:0/2119376631 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f12f81efc40 0x7f12f81f00c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:15.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.682+0000 7f12f77fe640 1 -- 172.21.15.116:0/2119376631 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12f8145830 con 0x7f12f813f9e0 2024-04-25T00:01:15.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.682+0000 7f12fcaf1640 1 --2- 172.21.15.116:0/2119376631 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f12f81efc40 0x7f12f81f00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:15.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.682+0000 7f12f77fe640 1 --2- 172.21.15.116:0/2119376631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12f813f9e0 0x7f12f8142b10 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f12e001c8d0 tx=0x7f12e4005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:15.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.682+0000 7f12f4ff9640 1 -- 172.21.15.116:0/2119376631 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e00ba020 con 0x7f12f813f9e0 2024-04-25T00:01:15.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.682+0000 7f12f4ff9640 1 -- 172.21.15.116:0/2119376631 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12e0000ee0 con 0x7f12f813f9e0 2024-04-25T00:01:15.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.682+0000 7f12f4ff9640 1 -- 172.21.15.116:0/2119376631 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e0007a40 con 0x7f12f813f9e0 2024-04-25T00:01:15.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.682+0000 7f12fe554640 1 -- 172.21.15.116:0/2119376631 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12f8145b10 con 0x7f12f813f9e0 2024-04-25T00:01:15.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.682+0000 7f12fe554640 1 -- 172.21.15.116:0/2119376631 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12f8142260 con 0x7f12f813f9e0 2024-04-25T00:01:15.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.686+0000 7f12fe554640 1 -- 172.21.15.116:0/2119376631 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12f804f510 con 0x7f12f813f9e0 2024-04-25T00:01:15.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.686+0000 7f12f4ff9640 1 -- 172.21.15.116:0/2119376631 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f12e00bd020 con 0x7f12f813f9e0 2024-04-25T00:01:15.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.686+0000 7f12f4ff9640 1 --2- 172.21.15.116:0/2119376631 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f12d80678c0 0x7f12d8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:15.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.686+0000 7f12f4ff9640 1 -- 172.21.15.116:0/2119376631 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5689+0+0 (secure 0 0 0) 0x7f12e011fd00 con 0x7f12f813f9e0 2024-04-25T00:01:15.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.686+0000 7f12f7fff640 1 --2- 172.21.15.116:0/2119376631 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f12d80678c0 0x7f12d8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:15.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.686+0000 7f12f7fff640 1 --2- 172.21.15.116:0/2119376631 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f12d80678c0 0x7f12d8069d80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f12f0001590 tx=0x7f12f0002160 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:15.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.690+0000 7f12f4ff9640 1 -- 172.21.15.116:0/2119376631 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f12e00c4030 con 0x7f12f813f9e0 2024-04-25T00:01:15.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.850+0000 7f12fe554640 1 -- 172.21.15.116:0/2119376631 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f12f8071db0 con 0x7f12f813f9e0 2024-04-25T00:01:15.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.850+0000 7f12f4ff9640 1 -- 172.21.15.116:0/2119376631 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) ==== 93+0+10 (secure 0 0 0) 0x7f12e0016130 con 0x7f12f813f9e0 2024-04-25T00:01:15.852 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-04-25T00:01:15.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.854+0000 7f12fe554640 1 -- 172.21.15.116:0/2119376631 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f12d80678c0 msgr2=0x7f12d8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:15.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.854+0000 7f12fe554640 1 --2- 172.21.15.116:0/2119376631 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f12d80678c0 0x7f12d8069d80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f12f0001590 tx=0x7f12f0002160 comp rx=0 tx=0).stop 2024-04-25T00:01:15.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.854+0000 7f12fe554640 1 -- 172.21.15.116:0/2119376631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12f813f9e0 msgr2=0x7f12f8142b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:15.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.854+0000 7f12fe554640 1 --2- 172.21.15.116:0/2119376631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12f813f9e0 0x7f12f8142b10 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f12e001c8d0 tx=0x7f12e4005840 comp rx=0 tx=0).stop 2024-04-25T00:01:15.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.854+0000 7f12fe554640 1 -- 172.21.15.116:0/2119376631 shutdown_connections 2024-04-25T00:01:15.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.854+0000 7f12fe554640 1 --2- 172.21.15.116:0/2119376631 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f12d80678c0 0x7f12d8069d80 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:15.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.854+0000 7f12fe554640 1 --2- 172.21.15.116:0/2119376631 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f12f81efc40 0x7f12f81f00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:15.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.854+0000 7f12fe554640 1 --2- 172.21.15.116:0/2119376631 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f12f813f9e0 0x7f12f8142b10 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:15.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.854+0000 7f12fe554640 1 --2- 172.21.15.116:0/2119376631 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f12e4004810 0x7f12f814fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:15.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.854+0000 7f12fe554640 1 -- 172.21.15.116:0/2119376631 >> 172.21.15.116:0/2119376631 conn(0x7f12f80c3a20 msgr2=0x7f12f80c1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:15.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.854+0000 7f12fe554640 1 -- 172.21.15.116:0/2119376631 shutdown_connections 2024-04-25T00:01:15.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.854+0000 7f12fe554640 1 -- 172.21.15.116:0/2119376631 wait complete. 2024-04-25T00:01:15.871 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_metadata pg_num 2024-04-25T00:01:15.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.990+0000 7f66cee94640 1 Processor -- start 2024-04-25T00:01:15.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.990+0000 7f66cee94640 1 -- start start 2024-04-25T00:01:15.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.994+0000 7f66cee94640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f66c8152f60 0x7f66c80c3e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:15.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.994+0000 7f66cee94640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66c8158f40 con 0x7f66c81559a0 2024-04-25T00:01:15.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.994+0000 7f66cee94640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66c8159220 con 0x7f66c80c3900 2024-04-25T00:01:15.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.994+0000 7f66cee94640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f66c8159500 con 0x7f66c8152f60 2024-04-25T00:01:15.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.994+0000 7f66cd431640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f66c81559a0 0x7f66c8155e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:60844/0 (socket says 172.21.15.116:60844) 2024-04-25T00:01:15.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.994+0000 7f66cd431640 1 -- 172.21.15.116:0/2604163841 learned_addr learned my addr 172.21.15.116:0/2604163841 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:15.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.994+0000 7f66c77fe640 1 -- 172.21.15.116:0/2604163841 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1114944559 0 0) 0x7f66c8158f40 con 0x7f66c81559a0 2024-04-25T00:01:15.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.994+0000 7f66ccc30640 1 --2- 172.21.15.116:0/2604163841 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f66c8152f60 0x7f66c80c3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:15.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.994+0000 7f66c77fe640 1 -- 172.21.15.116:0/2604163841 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f66b4003530 con 0x7f66c81559a0 2024-04-25T00:01:15.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.994+0000 7f66c77fe640 1 -- 172.21.15.116:0/2604163841 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3846382940 0 0) 0x7f66b4003530 con 0x7f66c81559a0 2024-04-25T00:01:15.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.994+0000 7f66c77fe640 1 -- 172.21.15.116:0/2604163841 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f66c8152f60 msgr2=0x7f66c80c3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:15.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.994+0000 7f66c77fe640 1 --2- 172.21.15.116:0/2604163841 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f66c8152f60 0x7f66c80c3e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:15.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.994+0000 7f66c77fe640 1 -- 172.21.15.116:0/2604163841 >> v1:172.21.15.174:6789/0 conn(0x7f66c80c3900 legacy=0x7f66c80c4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:15.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.994+0000 7f66c77fe640 1 -- 172.21.15.116:0/2604163841 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66c81597e0 con 0x7f66c81559a0 2024-04-25T00:01:15.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.994+0000 7f66c77fe640 1 -- 172.21.15.116:0/2604163841 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f66c0001a00 con 0x7f66c81559a0 2024-04-25T00:01:15.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.994+0000 7f66ccc30640 1 --2- 172.21.15.116:0/2604163841 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f66c8152f60 0x7f66c80c3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:01:15.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.994+0000 7f66c77fe640 1 -- 172.21.15.116:0/2604163841 >> v1:172.21.15.116:6789/0 conn(0x7f66c81559a0 legacy=0x7f66c8155e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:15.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.994+0000 7f66c77fe640 1 --2- 172.21.15.116:0/2604163841 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66b40039f0 0x7f66b4003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:15.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.994+0000 7f66c77fe640 1 --2- 172.21.15.116:0/2604163841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66b4004810 0x7f66b4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:15.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.994+0000 7f66c77fe640 1 --2- 172.21.15.116:0/2604163841 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f66c8152f60 0x7f66b400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:15.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.994+0000 7f66c77fe640 1 -- 172.21.15.116:0/2604163841 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f66c8159500 con 0x7f66b4004810 2024-04-25T00:01:15.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.994+0000 7f66c77fe640 1 -- 172.21.15.116:0/2604163841 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f66b40055d0 con 0x7f66b40039f0 2024-04-25T00:01:15.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.994+0000 7f66c77fe640 1 -- 172.21.15.116:0/2604163841 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f66b400bf10 con 0x7f66c8152f60 2024-04-25T00:01:15.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.994+0000 7f66c7fff640 1 --2- 172.21.15.116:0/2604163841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66b4004810 0x7f66b4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:15.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.994+0000 7f66ccc30640 1 --2- 172.21.15.116:0/2604163841 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66b40039f0 0x7f66b4003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:15.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.994+0000 7f66cd431640 1 --2- 172.21.15.116:0/2604163841 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f66c8152f60 0x7f66b400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:15.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.994+0000 7f66c7fff640 1 -- 172.21.15.116:0/2604163841 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f66c8152f60 msgr2=0x7f66b400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:15.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.994+0000 7f66c7fff640 1 --2- 172.21.15.116:0/2604163841 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f66c8152f60 0x7f66b400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:15.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.994+0000 7f66c7fff640 1 -- 172.21.15.116:0/2604163841 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66b40039f0 msgr2=0x7f66b4003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:15.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.994+0000 7f66c7fff640 1 --2- 172.21.15.116:0/2604163841 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66b40039f0 0x7f66b4003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:15.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.994+0000 7f66c7fff640 1 -- 172.21.15.116:0/2604163841 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66b4003530 con 0x7f66b4004810 2024-04-25T00:01:15.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.994+0000 7f66cd431640 1 --2- 172.21.15.116:0/2604163841 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f66c8152f60 0x7f66b400b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:16.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.998+0000 7f66c7fff640 1 --2- 172.21.15.116:0/2604163841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66b4004810 0x7f66b4004c90 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f66b8001780 tx=0x7f66b801c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=968e7ec1f010500e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:16.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.998+0000 7f66c77fe640 1 -- 172.21.15.116:0/2604163841 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b80013a0 con 0x7f66b4004810 2024-04-25T00:01:16.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.998+0000 7f66c77fe640 1 -- 172.21.15.116:0/2604163841 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66b80052f0 con 0x7f66b4004810 2024-04-25T00:01:16.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.998+0000 7f66c77fe640 1 -- 172.21.15.116:0/2604163841 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b8003840 con 0x7f66b4004810 2024-04-25T00:01:16.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.998+0000 7f66cee94640 1 -- 172.21.15.116:0/2604163841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66b4004810 msgr2=0x7f66b4004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:16.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.998+0000 7f66cee94640 1 --2- 172.21.15.116:0/2604163841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66b4004810 0x7f66b4004c90 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f66b8001780 tx=0x7f66b801c420 comp rx=0 tx=0).stop 2024-04-25T00:01:16.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.998+0000 7f66cd431640 1 -- 172.21.15.116:0/2604163841 reap_dead start 2024-04-25T00:01:16.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.998+0000 7f66cee94640 1 -- 172.21.15.116:0/2604163841 shutdown_connections 2024-04-25T00:01:16.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.998+0000 7f66cee94640 1 -- 172.21.15.116:0/2604163841 >> 172.21.15.116:0/2604163841 conn(0x7f66c80c8f00 msgr2=0x7f66c80c9310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:16.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.998+0000 7f66cee94640 1 -- 172.21.15.116:0/2604163841 shutdown_connections 2024-04-25T00:01:16.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.998+0000 7f66cee94640 1 -- 172.21.15.116:0/2604163841 wait complete. 2024-04-25T00:01:16.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.998+0000 7f66cee94640 1 Processor -- start 2024-04-25T00:01:16.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.998+0000 7f66cee94640 1 -- start start 2024-04-25T00:01:16.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.998+0000 7f66cee94640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66b4004810 0x7f66c8151170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:16.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.998+0000 7f66cee94640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f66c814efd0 0x7f66c81f1a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:16.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.998+0000 7f66cee94640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66c8160580 0x7f66c81eed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:16.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.998+0000 7f66cee94640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f66c8161ca0 con 0x7f66b4004810 2024-04-25T00:01:16.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.998+0000 7f66cee94640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f66c8161b20 con 0x7f66c8160580 2024-04-25T00:01:16.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.998+0000 7f66cee94640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f66c8161e20 con 0x7f66c814efd0 2024-04-25T00:01:16.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.998+0000 7f66ccc30640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66b4004810 0x7f66c8151170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:16.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.998+0000 7f66ccc30640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66b4004810 0x7f66c8151170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54264/0 (socket says 172.21.15.116:54264) 2024-04-25T00:01:16.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.998+0000 7f66ccc30640 1 -- 172.21.15.116:0/1227087320 learned_addr learned my addr 172.21.15.116:0/1227087320 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:16.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.998+0000 7f66cd431640 1 --2- 172.21.15.116:0/1227087320 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66c8160580 0x7f66c81eed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:16.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.998+0000 7f66c7fff640 1 --2- 172.21.15.116:0/1227087320 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f66c814efd0 0x7f66c81f1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:16.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.998+0000 7f66ccc30640 1 -- 172.21.15.116:0/1227087320 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f66c814efd0 msgr2=0x7f66c81f1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:16.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.998+0000 7f66ccc30640 1 --2- 172.21.15.116:0/1227087320 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f66c814efd0 0x7f66c81f1a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:16.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.998+0000 7f66ccc30640 1 -- 172.21.15.116:0/1227087320 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66c8160580 msgr2=0x7f66c81eed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:16.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.998+0000 7f66ccc30640 1 --2- 172.21.15.116:0/1227087320 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66c8160580 0x7f66c81eed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:16.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.998+0000 7f66ccc30640 1 -- 172.21.15.116:0/1227087320 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66c81508c0 con 0x7f66b4004810 2024-04-25T00:01:16.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.998+0000 7f66cd431640 1 --2- 172.21.15.116:0/1227087320 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66c8160580 0x7f66c81eed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:16.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.998+0000 7f66ccc30640 1 --2- 172.21.15.116:0/1227087320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66b4004810 0x7f66c8151170 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f66b00108c0 tx=0x7f66b0010d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:16.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.998+0000 7f66c57fa640 1 -- 172.21.15.116:0/1227087320 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b001a020 con 0x7f66b4004810 2024-04-25T00:01:16.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.998+0000 7f66c57fa640 1 -- 172.21.15.116:0/1227087320 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66b000cb40 con 0x7f66b4004810 2024-04-25T00:01:16.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.998+0000 7f66cee94640 1 -- 172.21.15.116:0/1227087320 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66c8150b60 con 0x7f66b4004810 2024-04-25T00:01:16.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:15.998+0000 7f66c57fa640 1 -- 172.21.15.116:0/1227087320 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b000e050 con 0x7f66b4004810 2024-04-25T00:01:16.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.002+0000 7f66cee94640 1 -- 172.21.15.116:0/1227087320 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66c814a540 con 0x7f66b4004810 2024-04-25T00:01:16.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.002+0000 7f66cee94640 1 -- 172.21.15.116:0/1227087320 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66c804f510 con 0x7f66b4004810 2024-04-25T00:01:16.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.002+0000 7f66c57fa640 1 -- 172.21.15.116:0/1227087320 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f66b001d020 con 0x7f66b4004810 2024-04-25T00:01:16.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.002+0000 7f66c57fa640 1 --2- 172.21.15.116:0/1227087320 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f66a80678c0 0x7f66a8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:16.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.002+0000 7f66c57fa640 1 -- 172.21.15.116:0/1227087320 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5689+0+0 (secure 0 0 0) 0x7f66b0086350 con 0x7f66b4004810 2024-04-25T00:01:16.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.002+0000 7f66c7fff640 1 --2- 172.21.15.116:0/1227087320 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f66a80678c0 0x7f66a8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:16.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.002+0000 7f66c7fff640 1 --2- 172.21.15.116:0/1227087320 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f66a80678c0 0x7f66a8069d80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f66b8001750 tx=0x7f66b80011a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:16.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.006+0000 7f66c57fa640 1 -- 172.21.15.116:0/1227087320 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f66b0024020 con 0x7f66b4004810 2024-04-25T00:01:16.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.158+0000 7f66cee94640 1 -- 172.21.15.116:0/1227087320 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"} v 0) -- 0x7f66c80c74c0 con 0x7f66b4004810 2024-04-25T00:01:16.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.158+0000 7f66c57fa640 1 -- 172.21.15.116:0/1227087320 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"}]=0 v23) ==== 99+0+11 (secure 0 0 0) 0x7f66b0027350 con 0x7f66b4004810 2024-04-25T00:01:16.160 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-04-25T00:01:16.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.162+0000 7f66cee94640 1 -- 172.21.15.116:0/1227087320 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f66a80678c0 msgr2=0x7f66a8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:16.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.162+0000 7f66cee94640 1 --2- 172.21.15.116:0/1227087320 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f66a80678c0 0x7f66a8069d80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f66b8001750 tx=0x7f66b80011a0 comp rx=0 tx=0).stop 2024-04-25T00:01:16.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.162+0000 7f66cee94640 1 -- 172.21.15.116:0/1227087320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66b4004810 msgr2=0x7f66c8151170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:16.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.162+0000 7f66cee94640 1 --2- 172.21.15.116:0/1227087320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66b4004810 0x7f66c8151170 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f66b00108c0 tx=0x7f66b0010d80 comp rx=0 tx=0).stop 2024-04-25T00:01:16.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.162+0000 7f66cee94640 1 -- 172.21.15.116:0/1227087320 shutdown_connections 2024-04-25T00:01:16.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.162+0000 7f66cee94640 1 --2- 172.21.15.116:0/1227087320 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f66a80678c0 0x7f66a8069d80 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:16.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.162+0000 7f66cee94640 1 --2- 172.21.15.116:0/1227087320 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66c8160580 0x7f66c81eed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:16.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.162+0000 7f66cee94640 1 --2- 172.21.15.116:0/1227087320 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f66c814efd0 0x7f66c81f1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:16.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.162+0000 7f66cee94640 1 --2- 172.21.15.116:0/1227087320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66b4004810 0x7f66c8151170 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:16.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.162+0000 7f66cee94640 1 -- 172.21.15.116:0/1227087320 >> 172.21.15.116:0/1227087320 conn(0x7f66c80c8f00 msgr2=0x7f66c80c9310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:16.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.162+0000 7f66cee94640 1 -- 172.21.15.116:0/1227087320 shutdown_connections 2024-04-25T00:01:16.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.162+0000 7f66cee94640 1 -- 172.21.15.116:0/1227087320 wait complete. 2024-04-25T00:01:16.178 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_data pg_num 2024-04-25T00:01:16.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.298+0000 7fdac2f6b640 1 Processor -- start 2024-04-25T00:01:16.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.298+0000 7fdac2f6b640 1 -- start start 2024-04-25T00:01:16.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.302+0000 7fdac2f6b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdabc0c5a70 0x7fdabc0c9710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:16.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.302+0000 7fdac2f6b640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdabc0c8630 con 0x7fdabc0c7ad0 2024-04-25T00:01:16.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.302+0000 7fdac2f6b640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdabc0c8070 con 0x7fdabc0c5e50 2024-04-25T00:01:16.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.302+0000 7fdac2f6b640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fdabc0c62e0 con 0x7fdabc0c5a70 2024-04-25T00:01:16.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.302+0000 7fdabbfff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fdabc0c7ad0 0x7fdabc06d450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:60856/0 (socket says 172.21.15.116:60856) 2024-04-25T00:01:16.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.302+0000 7fdabbfff640 1 -- 172.21.15.116:0/1378808547 learned_addr learned my addr 172.21.15.116:0/1378808547 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:16.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.302+0000 7fdac0d07640 1 --2- 172.21.15.116:0/1378808547 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdabc0c5a70 0x7fdabc0c9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:16.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.302+0000 7fdabb7fe640 1 -- 172.21.15.116:0/1378808547 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3552899183 0 0) 0x7fdabc0c8630 con 0x7fdabc0c7ad0 2024-04-25T00:01:16.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.302+0000 7fdabb7fe640 1 -- 172.21.15.116:0/1378808547 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdaa8003530 con 0x7fdabc0c7ad0 2024-04-25T00:01:16.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.302+0000 7fdabb7fe640 1 -- 172.21.15.116:0/1378808547 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1772283410 0 0) 0x7fdaa8003530 con 0x7fdabc0c7ad0 2024-04-25T00:01:16.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.302+0000 7fdabb7fe640 1 -- 172.21.15.116:0/1378808547 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdabc0c5a70 msgr2=0x7fdabc0c9710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:16.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.302+0000 7fdabb7fe640 1 --2- 172.21.15.116:0/1378808547 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdabc0c5a70 0x7fdabc0c9710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:16.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.302+0000 7fdabb7fe640 1 -- 172.21.15.116:0/1378808547 >> v1:172.21.15.174:6789/0 conn(0x7fdabc0c5e50 legacy=0x7fdabc1602c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:16.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.302+0000 7fdabb7fe640 1 -- 172.21.15.116:0/1378808547 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdabc0c8350 con 0x7fdabc0c7ad0 2024-04-25T00:01:16.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.302+0000 7fdabb7fe640 1 -- 172.21.15.116:0/1378808547 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7fdaa4001680 con 0x7fdabc0c7ad0 2024-04-25T00:01:16.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.302+0000 7fdabb7fe640 1 -- 172.21.15.116:0/1378808547 >> v1:172.21.15.116:6789/0 conn(0x7fdabc0c7ad0 legacy=0x7fdabc06d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:16.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.302+0000 7fdabb7fe640 1 --2- 172.21.15.116:0/1378808547 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdabc0c5a70 0x7fdaa80039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:16.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.302+0000 7fdabb7fe640 1 --2- 172.21.15.116:0/1378808547 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdaa8003f70 0x7fdaa8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:16.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.302+0000 7fdabb7fe640 1 --2- 172.21.15.116:0/1378808547 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdaa8004d50 0x7fdaa80051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:16.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.302+0000 7fdabb7fe640 1 -- 172.21.15.116:0/1378808547 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdabc0c62e0 con 0x7fdaa8003f70 2024-04-25T00:01:16.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.302+0000 7fdabb7fe640 1 -- 172.21.15.116:0/1378808547 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa800bb60 con 0x7fdaa8004d50 2024-04-25T00:01:16.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.302+0000 7fdabb7fe640 1 -- 172.21.15.116:0/1378808547 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fdaa800bd60 con 0x7fdabc0c5a70 2024-04-25T00:01:16.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.302+0000 7fdabbfff640 1 --2- 172.21.15.116:0/1378808547 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdaa8003f70 0x7fdaa8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:16.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.302+0000 7fdac1508640 1 --2- 172.21.15.116:0/1378808547 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdaa8004d50 0x7fdaa80051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:16.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.302+0000 7fdabbfff640 1 -- 172.21.15.116:0/1378808547 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdabc0c5a70 msgr2=0x7fdaa80039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:16.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.302+0000 7fdabbfff640 1 --2- 172.21.15.116:0/1378808547 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdabc0c5a70 0x7fdaa80039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:16.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.302+0000 7fdabbfff640 1 -- 172.21.15.116:0/1378808547 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdaa8004d50 msgr2=0x7fdaa80051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:16.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.302+0000 7fdabbfff640 1 --2- 172.21.15.116:0/1378808547 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdaa8004d50 0x7fdaa80051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:16.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.302+0000 7fdabbfff640 1 -- 172.21.15.116:0/1378808547 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaa8003530 con 0x7fdaa8003f70 2024-04-25T00:01:16.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.302+0000 7fdac1508640 1 --2- 172.21.15.116:0/1378808547 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdaa8004d50 0x7fdaa80051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:16.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.306+0000 7fdabbfff640 1 --2- 172.21.15.116:0/1378808547 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdaa8003f70 0x7fdaa8004810 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fdaa400c240 tx=0x7fdaa401cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acc73b5be4f9fb1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:16.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.306+0000 7fdabb7fe640 1 -- 172.21.15.116:0/1378808547 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa4007d20 con 0x7fdaa8003f70 2024-04-25T00:01:16.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.306+0000 7fdabb7fe640 1 -- 172.21.15.116:0/1378808547 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdaa40a9070 con 0x7fdaa8003f70 2024-04-25T00:01:16.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.306+0000 7fdabb7fe640 1 -- 172.21.15.116:0/1378808547 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa4005d60 con 0x7fdaa8003f70 2024-04-25T00:01:16.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.306+0000 7fdac2f6b640 1 -- 172.21.15.116:0/1378808547 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdaa8003f70 msgr2=0x7fdaa8004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:16.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.306+0000 7fdac2f6b640 1 --2- 172.21.15.116:0/1378808547 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdaa8003f70 0x7fdaa8004810 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fdaa400c240 tx=0x7fdaa401cc80 comp rx=0 tx=0).stop 2024-04-25T00:01:16.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.306+0000 7fdac1508640 1 -- 172.21.15.116:0/1378808547 reap_dead start 2024-04-25T00:01:16.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.306+0000 7fdac2f6b640 1 -- 172.21.15.116:0/1378808547 shutdown_connections 2024-04-25T00:01:16.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.306+0000 7fdac2f6b640 1 -- 172.21.15.116:0/1378808547 >> 172.21.15.116:0/1378808547 conn(0x7fdabc0cb0f0 msgr2=0x7fdabc0cb500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:16.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.306+0000 7fdac2f6b640 1 -- 172.21.15.116:0/1378808547 shutdown_connections 2024-04-25T00:01:16.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.306+0000 7fdac2f6b640 1 -- 172.21.15.116:0/1378808547 wait complete. 2024-04-25T00:01:16.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.306+0000 7fdac2f6b640 1 Processor -- start 2024-04-25T00:01:16.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.306+0000 7fdac2f6b640 1 -- start start 2024-04-25T00:01:16.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.306+0000 7fdac2f6b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdaa8003f70 0x7fdabc147890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:16.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.306+0000 7fdac2f6b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdabc145530 0x7fdabc1f0310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:16.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.306+0000 7fdac2f6b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdabc145a80 0x7fdabc1f0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:16.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.306+0000 7fdac2f6b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdabc06cfc0 con 0x7fdabc145530 2024-04-25T00:01:16.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.306+0000 7fdac2f6b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fdabc153440 con 0x7fdabc145a80 2024-04-25T00:01:16.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.306+0000 7fdac2f6b640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fdabc06db60 con 0x7fdaa8003f70 2024-04-25T00:01:16.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.306+0000 7fdabbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdabc145530 0x7fdabc1f0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:16.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.306+0000 7fdabbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdabc145530 0x7fdabc1f0310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54288/0 (socket says 172.21.15.116:54288) 2024-04-25T00:01:16.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.306+0000 7fdabbfff640 1 -- 172.21.15.116:0/3962423860 learned_addr learned my addr 172.21.15.116:0/3962423860 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:16.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.306+0000 7fdac0d07640 1 --2- 172.21.15.116:0/3962423860 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdaa8003f70 0x7fdabc147890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:16.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.306+0000 7fdac1508640 1 --2- 172.21.15.116:0/3962423860 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdabc145a80 0x7fdabc1f0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:16.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.306+0000 7fdabbfff640 1 -- 172.21.15.116:0/3962423860 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdaa8003f70 msgr2=0x7fdabc147890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:16.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.306+0000 7fdabbfff640 1 --2- 172.21.15.116:0/3962423860 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdaa8003f70 0x7fdabc147890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:16.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.306+0000 7fdabbfff640 1 -- 172.21.15.116:0/3962423860 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdabc145a80 msgr2=0x7fdabc1f0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:16.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.306+0000 7fdabbfff640 1 --2- 172.21.15.116:0/3962423860 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdabc145a80 0x7fdabc1f0850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:16.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.306+0000 7fdabbfff640 1 -- 172.21.15.116:0/3962423860 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdabc0c2340 con 0x7fdabc145530 2024-04-25T00:01:16.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.306+0000 7fdabbfff640 1 --2- 172.21.15.116:0/3962423860 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdabc145530 0x7fdabc1f0310 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fdaa4001980 tx=0x7fdaa8005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:16.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.306+0000 7fdab97fa640 1 -- 172.21.15.116:0/3962423860 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa4007d20 con 0x7fdabc145530 2024-04-25T00:01:16.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.306+0000 7fdab97fa640 1 -- 172.21.15.116:0/3962423860 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdaa4000ee0 con 0x7fdabc145530 2024-04-25T00:01:16.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.306+0000 7fdab97fa640 1 -- 172.21.15.116:0/3962423860 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa4001640 con 0x7fdabc145530 2024-04-25T00:01:16.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.306+0000 7fdac2f6b640 1 -- 172.21.15.116:0/3962423860 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdabc1f2660 con 0x7fdabc145530 2024-04-25T00:01:16.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.306+0000 7fdac2f6b640 1 -- 172.21.15.116:0/3962423860 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdabc1f2c20 con 0x7fdabc145530 2024-04-25T00:01:16.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.310+0000 7fdac2f6b640 1 -- 172.21.15.116:0/3962423860 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdabc04f510 con 0x7fdabc145530 2024-04-25T00:01:16.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.310+0000 7fdab97fa640 1 -- 172.21.15.116:0/3962423860 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fdaa4007090 con 0x7fdabc145530 2024-04-25T00:01:16.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.310+0000 7fdab97fa640 1 --2- 172.21.15.116:0/3962423860 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fda9c0678c0 0x7fda9c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:16.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.310+0000 7fdab97fa640 1 -- 172.21.15.116:0/3962423860 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5689+0+0 (secure 0 0 0) 0x7fdaa411f4f0 con 0x7fdabc145530 2024-04-25T00:01:16.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.310+0000 7fdac0d07640 1 --2- 172.21.15.116:0/3962423860 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fda9c0678c0 0x7fda9c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:16.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.310+0000 7fdac0d07640 1 --2- 172.21.15.116:0/3962423860 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fda9c0678c0 0x7fda9c069d80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fdaac007520 tx=0x7fdaac00d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:16.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.314+0000 7fdab97fa640 1 -- 172.21.15.116:0/3962423860 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fdaa40bf060 con 0x7fdabc145530 2024-04-25T00:01:16.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.478+0000 7fdac2f6b640 1 -- 172.21.15.116:0/3962423860 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"} v 0) -- 0x7fdabc0c88c0 con 0x7fdabc145530 2024-04-25T00:01:16.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.478+0000 7fdab97fa640 1 -- 172.21.15.116:0/3962423860 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"}]=0 v23) ==== 95+0+11 (secure 0 0 0) 0x7fdaa4016990 con 0x7fdabc145530 2024-04-25T00:01:16.479 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-04-25T00:01:16.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.478+0000 7fdac2f6b640 1 -- 172.21.15.116:0/3962423860 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fda9c0678c0 msgr2=0x7fda9c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:16.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.478+0000 7fdac2f6b640 1 --2- 172.21.15.116:0/3962423860 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fda9c0678c0 0x7fda9c069d80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fdaac007520 tx=0x7fdaac00d040 comp rx=0 tx=0).stop 2024-04-25T00:01:16.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.478+0000 7fdac2f6b640 1 -- 172.21.15.116:0/3962423860 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdabc145530 msgr2=0x7fdabc1f0310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:16.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.478+0000 7fdac2f6b640 1 --2- 172.21.15.116:0/3962423860 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdabc145530 0x7fdabc1f0310 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fdaa4001980 tx=0x7fdaa8005840 comp rx=0 tx=0).stop 2024-04-25T00:01:16.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.478+0000 7fdac2f6b640 1 -- 172.21.15.116:0/3962423860 shutdown_connections 2024-04-25T00:01:16.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.478+0000 7fdac2f6b640 1 --2- 172.21.15.116:0/3962423860 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fda9c0678c0 0x7fda9c069d80 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:16.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.478+0000 7fdac2f6b640 1 --2- 172.21.15.116:0/3962423860 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdabc145a80 0x7fdabc1f0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:16.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.478+0000 7fdac2f6b640 1 --2- 172.21.15.116:0/3962423860 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdabc145530 0x7fdabc1f0310 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:16.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.478+0000 7fdac2f6b640 1 --2- 172.21.15.116:0/3962423860 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdaa8003f70 0x7fdabc147890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:16.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.478+0000 7fdac2f6b640 1 -- 172.21.15.116:0/3962423860 >> 172.21.15.116:0/3962423860 conn(0x7fdabc0cb0f0 msgr2=0x7fdabc0ca570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:16.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.478+0000 7fdac2f6b640 1 -- 172.21.15.116:0/3962423860 shutdown_connections 2024-04-25T00:01:16.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.478+0000 7fdac2f6b640 1 -- 172.21.15.116:0/3962423860 wait complete. 2024-04-25T00:01:16.498 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-25T00:01:16.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.610+0000 7f8d750d0640 1 Processor -- start 2024-04-25T00:01:16.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.610+0000 7f8d750d0640 1 -- start start 2024-04-25T00:01:16.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.610+0000 7f8d750d0640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8d70148890 0x7f8d70072590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:16.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.610+0000 7f8d750d0640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f8d7006e890 0x7f8d70071980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:16.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.610+0000 7f8d750d0640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d70152c10 con 0x7f8d70070dd0 2024-04-25T00:01:16.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.610+0000 7f8d750d0640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f8d70151e50 con 0x7f8d70148890 2024-04-25T00:01:16.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.610+0000 7f8d750d0640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f8d70152130 con 0x7f8d7006e890 2024-04-25T00:01:16.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.610+0000 7f8d6ed9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8d70148890 0x7f8d70072590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:16.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.610+0000 7f8d6e59c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f8d70070dd0 0x7f8d70071270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:60858/0 (socket says 172.21.15.116:60858) 2024-04-25T00:01:16.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.610+0000 7f8d6e59c640 1 -- 172.21.15.116:0/2213705643 learned_addr learned my addr 172.21.15.116:0/2213705643 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:16.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.610+0000 7f8d6f59e640 1 --2- 172.21.15.116:0/2213705643 >> v2:172.21.15.174:3300/0 conn(0x7f8d7006e890 0x7f8d70071980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:16.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.610+0000 7f8d6ed9d640 1 -- 172.21.15.116:0/2213705643 >> v2:172.21.15.174:3300/0 conn(0x7f8d7006e890 msgr2=0x7f8d70071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:16.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.610+0000 7f8d6ed9d640 1 --2- 172.21.15.116:0/2213705643 >> v2:172.21.15.174:3300/0 conn(0x7f8d7006e890 0x7f8d70071980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:16.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.610+0000 7f8d6ed9d640 1 -- 172.21.15.116:0/2213705643 >> v1:172.21.15.116:6789/0 conn(0x7f8d70070dd0 legacy=0x7f8d70071270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:16.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.610+0000 7f8d6dd9b640 1 -- 172.21.15.116:0/2213705643 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2360289781 0 0) 0x7f8d70152c10 con 0x7f8d70070dd0 2024-04-25T00:01:16.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.610+0000 7f8d6ed9d640 1 -- 172.21.15.116:0/2213705643 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d70152410 con 0x7f8d70148890 2024-04-25T00:01:16.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.614+0000 7f8d6ed9d640 1 --2- 172.21.15.116:0/2213705643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d70148890 0x7f8d70072590 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8d600097c0 tx=0x7f8d6001d570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eda07649ccedd546 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:16.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.614+0000 7f8d6dd9b640 1 -- 172.21.15.116:0/2213705643 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d600b3070 con 0x7f8d70148890 2024-04-25T00:01:16.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.614+0000 7f8d6dd9b640 1 -- 172.21.15.116:0/2213705643 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d600ae680 con 0x7f8d70148890 2024-04-25T00:01:16.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.614+0000 7f8d6dd9b640 1 -- 172.21.15.116:0/2213705643 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d600aea50 con 0x7f8d70148890 2024-04-25T00:01:16.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.614+0000 7f8d750d0640 1 -- 172.21.15.116:0/2213705643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d70148890 msgr2=0x7f8d70072590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:16.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.614+0000 7f8d750d0640 1 --2- 172.21.15.116:0/2213705643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d70148890 0x7f8d70072590 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8d600097c0 tx=0x7f8d6001d570 comp rx=0 tx=0).stop 2024-04-25T00:01:16.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.614+0000 7f8d750d0640 1 -- 172.21.15.116:0/2213705643 shutdown_connections 2024-04-25T00:01:16.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.614+0000 7f8d750d0640 1 --2- 172.21.15.116:0/2213705643 >> v2:172.21.15.174:3300/0 conn(0x7f8d7006e890 0x7f8d70071980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:16.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.614+0000 7f8d750d0640 1 --2- 172.21.15.116:0/2213705643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d70148890 0x7f8d70072590 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:16.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.614+0000 7f8d750d0640 1 -- 172.21.15.116:0/2213705643 >> 172.21.15.116:0/2213705643 conn(0x7f8d700c3b00 msgr2=0x7f8d700c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:16.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.614+0000 7f8d750d0640 1 -- 172.21.15.116:0/2213705643 shutdown_connections 2024-04-25T00:01:16.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.614+0000 7f8d750d0640 1 -- 172.21.15.116:0/2213705643 wait complete. 2024-04-25T00:01:16.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.614+0000 7f8d750d0640 1 Processor -- start 2024-04-25T00:01:16.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.614+0000 7f8d750d0640 1 -- start start 2024-04-25T00:01:16.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.618+0000 7f8d750d0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d7006e890 0x7f8d700c0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:16.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.618+0000 7f8d750d0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d70070dd0 0x7f8d701441e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:16.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.618+0000 7f8d750d0640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8d70148890 0x7f8d70144720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:16.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.618+0000 7f8d750d0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8d700ab920 con 0x7f8d7006e890 2024-04-25T00:01:16.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.618+0000 7f8d750d0640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8d700ab7a0 con 0x7f8d70070dd0 2024-04-25T00:01:16.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.618+0000 7f8d750d0640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f8d700abaa0 con 0x7f8d70148890 2024-04-25T00:01:16.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.618+0000 7f8d6ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d7006e890 0x7f8d700c0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:16.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.618+0000 7f8d6ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d7006e890 0x7f8d700c0600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54314/0 (socket says 172.21.15.116:54314) 2024-04-25T00:01:16.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.618+0000 7f8d6ed9d640 1 -- 172.21.15.116:0/2866334362 learned_addr learned my addr 172.21.15.116:0/2866334362 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:16.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.618+0000 7f8d6e59c640 1 --2- 172.21.15.116:0/2866334362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d70070dd0 0x7f8d701441e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:16.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.618+0000 7f8d6ed9d640 1 -- 172.21.15.116:0/2866334362 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8d70148890 msgr2=0x7f8d70144720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:16.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.618+0000 7f8d6f59e640 1 --2- 172.21.15.116:0/2866334362 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8d70148890 0x7f8d70144720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:16.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.618+0000 7f8d6ed9d640 1 --2- 172.21.15.116:0/2866334362 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8d70148890 0x7f8d70144720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:16.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.618+0000 7f8d6ed9d640 1 -- 172.21.15.116:0/2866334362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d70070dd0 msgr2=0x7f8d701441e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:16.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.618+0000 7f8d6ed9d640 1 --2- 172.21.15.116:0/2866334362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d70070dd0 0x7f8d701441e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:16.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.618+0000 7f8d6ed9d640 1 -- 172.21.15.116:0/2866334362 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d701ef1a0 con 0x7f8d7006e890 2024-04-25T00:01:16.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.618+0000 7f8d6e59c640 1 --2- 172.21.15.116:0/2866334362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d70070dd0 0x7f8d701441e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:16.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.618+0000 7f8d6ed9d640 1 --2- 172.21.15.116:0/2866334362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d7006e890 0x7f8d700c0600 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8d6001a180 tx=0x7f8d6001a310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:16.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.618+0000 7f8d577fe640 1 -- 172.21.15.116:0/2866334362 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d600b3070 con 0x7f8d7006e890 2024-04-25T00:01:16.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.618+0000 7f8d577fe640 1 -- 172.21.15.116:0/2866334362 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d600aeda0 con 0x7f8d7006e890 2024-04-25T00:01:16.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.618+0000 7f8d750d0640 1 -- 172.21.15.116:0/2866334362 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d700c5750 con 0x7f8d7006e890 2024-04-25T00:01:16.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.618+0000 7f8d577fe640 1 -- 172.21.15.116:0/2866334362 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d600b9650 con 0x7f8d7006e890 2024-04-25T00:01:16.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.618+0000 7f8d750d0640 1 -- 172.21.15.116:0/2866334362 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d701eee30 con 0x7f8d7006e890 2024-04-25T00:01:16.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.618+0000 7f8d750d0640 1 -- 172.21.15.116:0/2866334362 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d70144c60 con 0x7f8d7006e890 2024-04-25T00:01:16.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.618+0000 7f8d577fe640 1 -- 172.21.15.116:0/2866334362 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8d600c0060 con 0x7f8d7006e890 2024-04-25T00:01:16.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.618+0000 7f8d577fe640 1 --2- 172.21.15.116:0/2866334362 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8d5c0678f0 0x7f8d5c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:16.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.618+0000 7f8d577fe640 1 -- 172.21.15.116:0/2866334362 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5689+0+0 (secure 0 0 0) 0x7f8d6011f460 con 0x7f8d7006e890 2024-04-25T00:01:16.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.618+0000 7f8d6e59c640 1 --2- 172.21.15.116:0/2866334362 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8d5c0678f0 0x7f8d5c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:16.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.622+0000 7f8d6e59c640 1 --2- 172.21.15.116:0/2866334362 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8d5c0678f0 0x7f8d5c069db0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8d70072910 tx=0x7f8d64009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:16.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.622+0000 7f8d577fe640 1 -- 172.21.15.116:0/2866334362 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f8d60017160 con 0x7f8d7006e890 2024-04-25T00:01:16.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.774+0000 7f8d750d0640 1 -- 172.21.15.116:0/2866334362 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f8d7006e670 con 0x7f8d7006e890 2024-04-25T00:01:16.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.774+0000 7f8d577fe640 1 -- 172.21.15.116:0/2866334362 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v23) ==== 84+0+197 (secure 0 0 0) 0x7f8d600effc0 con 0x7f8d7006e890 2024-04-25T00:01:16.776 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:01:16.776 INFO:teuthology.orchestra.run.smithi116.stdout:[ 2024-04-25T00:01:16.776 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-25T00:01:16.776 INFO:teuthology.orchestra.run.smithi116.stdout: "poolnum": 1, 2024-04-25T00:01:16.776 INFO:teuthology.orchestra.run.smithi116.stdout: "poolname": ".mgr" 2024-04-25T00:01:16.776 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-25T00:01:16.776 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-25T00:01:16.777 INFO:teuthology.orchestra.run.smithi116.stdout: "poolnum": 2, 2024-04-25T00:01:16.777 INFO:teuthology.orchestra.run.smithi116.stdout: "poolname": "a_metadata" 2024-04-25T00:01:16.777 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-25T00:01:16.777 INFO:teuthology.orchestra.run.smithi116.stdout: { 2024-04-25T00:01:16.777 INFO:teuthology.orchestra.run.smithi116.stdout: "poolnum": 3, 2024-04-25T00:01:16.777 INFO:teuthology.orchestra.run.smithi116.stdout: "poolname": "a_data" 2024-04-25T00:01:16.777 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-25T00:01:16.777 INFO:teuthology.orchestra.run.smithi116.stdout:] 2024-04-25T00:01:16.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.774+0000 7f8d750d0640 1 -- 172.21.15.116:0/2866334362 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8d5c0678f0 msgr2=0x7f8d5c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:16.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.774+0000 7f8d750d0640 1 --2- 172.21.15.116:0/2866334362 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8d5c0678f0 0x7f8d5c069db0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8d70072910 tx=0x7f8d64009040 comp rx=0 tx=0).stop 2024-04-25T00:01:16.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.774+0000 7f8d750d0640 1 -- 172.21.15.116:0/2866334362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d7006e890 msgr2=0x7f8d700c0600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:16.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.774+0000 7f8d750d0640 1 --2- 172.21.15.116:0/2866334362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d7006e890 0x7f8d700c0600 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8d6001a180 tx=0x7f8d6001a310 comp rx=0 tx=0).stop 2024-04-25T00:01:16.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.778+0000 7f8d750d0640 1 -- 172.21.15.116:0/2866334362 shutdown_connections 2024-04-25T00:01:16.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.778+0000 7f8d750d0640 1 --2- 172.21.15.116:0/2866334362 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8d5c0678f0 0x7f8d5c069db0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:16.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.778+0000 7f8d750d0640 1 --2- 172.21.15.116:0/2866334362 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8d70148890 0x7f8d70144720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:16.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.778+0000 7f8d750d0640 1 --2- 172.21.15.116:0/2866334362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8d70070dd0 0x7f8d701441e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:16.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.778+0000 7f8d750d0640 1 --2- 172.21.15.116:0/2866334362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8d7006e890 0x7f8d700c0600 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:16.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.778+0000 7f8d750d0640 1 -- 172.21.15.116:0/2866334362 >> 172.21.15.116:0/2866334362 conn(0x7f8d700c3b00 msgr2=0x7f8d700714b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:16.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.778+0000 7f8d750d0640 1 -- 172.21.15.116:0/2866334362 shutdown_connections 2024-04-25T00:01:16.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.778+0000 7f8d750d0640 1 -- 172.21.15.116:0/2866334362 wait complete. 2024-04-25T00:01:16.794 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'b' 2024-04-25T00:01:16.794 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create b_metadata --pg_num_min 64 2024-04-25T00:01:16.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.918+0000 7f2411ad1640 1 Processor -- start 2024-04-25T00:01:16.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.918+0000 7f2411ad1640 1 -- start start 2024-04-25T00:01:16.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.918+0000 7f2411ad1640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f240c0c75c0 0x7f240c06c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:16.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.918+0000 7f2411ad1640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f240c156530 con 0x7f240c0c6d70 2024-04-25T00:01:16.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.918+0000 7f2411ad1640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f240c06bf00 con 0x7f240c153210 2024-04-25T00:01:16.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.918+0000 7f2411ad1640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f240c1567b0 con 0x7f240c0c75c0 2024-04-25T00:01:16.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.918+0000 7f240bfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f240c0c75c0 0x7f240c06c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:16.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.918+0000 7f240bfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f240c0c75c0 0x7f240c06c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54326/0 (socket says 172.21.15.116:54326) 2024-04-25T00:01:16.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.918+0000 7f240bfff640 1 -- 172.21.15.116:0/2381818136 learned_addr learned my addr 172.21.15.116:0/2381818136 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:16.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.918+0000 7f240affd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f240c0c6d70 0x7f240c0c5c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:60866/0 (socket says 172.21.15.116:60866) 2024-04-25T00:01:16.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.918+0000 7f240bfff640 1 -- 172.21.15.116:0/2381818136 >> v1:172.21.15.174:6789/0 conn(0x7f240c153210 legacy=0x7f240c0c8fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:16.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.918+0000 7f240bfff640 1 -- 172.21.15.116:0/2381818136 >> v1:172.21.15.116:6789/0 conn(0x7f240c0c6d70 legacy=0x7f240c0c5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:16.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.918+0000 7f240a7fc640 1 -- 172.21.15.116:0/2381818136 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2708224090 0 0) 0x7f2400001120 con 0x7f240c0c6d70 2024-04-25T00:01:16.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.918+0000 7f240bfff640 1 -- 172.21.15.116:0/2381818136 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f240c0c96e0 con 0x7f240c0c75c0 2024-04-25T00:01:16.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.922+0000 7f240bfff640 1 --2- 172.21.15.116:0/2381818136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f240c0c75c0 0x7f240c06c5d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f240c0c9910 tx=0x7f240401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a5a7b87cb606520 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:16.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.922+0000 7f240a7fc640 1 -- 172.21.15.116:0/2381818136 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24040b0070 con 0x7f240c0c75c0 2024-04-25T00:01:16.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.922+0000 7f240a7fc640 1 -- 172.21.15.116:0/2381818136 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24040ab600 con 0x7f240c0c75c0 2024-04-25T00:01:16.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.922+0000 7f240a7fc640 1 -- 172.21.15.116:0/2381818136 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24040ab9d0 con 0x7f240c0c75c0 2024-04-25T00:01:16.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.922+0000 7f2411ad1640 1 -- 172.21.15.116:0/2381818136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f240c0c75c0 msgr2=0x7f240c06c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:16.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.922+0000 7f2411ad1640 1 --2- 172.21.15.116:0/2381818136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f240c0c75c0 0x7f240c06c5d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f240c0c9910 tx=0x7f240401c3e0 comp rx=0 tx=0).stop 2024-04-25T00:01:16.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.922+0000 7f2411ad1640 1 -- 172.21.15.116:0/2381818136 shutdown_connections 2024-04-25T00:01:16.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.922+0000 7f2411ad1640 1 --2- 172.21.15.116:0/2381818136 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f240c0c75c0 0x7f240c06c5d0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:16.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.922+0000 7f2411ad1640 1 -- 172.21.15.116:0/2381818136 >> 172.21.15.116:0/2381818136 conn(0x7f240c0acbc0 msgr2=0x7f240c0acfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:16.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.922+0000 7f2411ad1640 1 -- 172.21.15.116:0/2381818136 shutdown_connections 2024-04-25T00:01:16.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.922+0000 7f2411ad1640 1 -- 172.21.15.116:0/2381818136 wait complete. 2024-04-25T00:01:16.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.922+0000 7f2411ad1640 1 Processor -- start 2024-04-25T00:01:16.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.922+0000 7f2411ad1640 1 -- start start 2024-04-25T00:01:16.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.922+0000 7f2411ad1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f240c0c6d70 0x7f240c14bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:16.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.922+0000 7f2411ad1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f240c0c75c0 0x7f240c14c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:16.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.922+0000 7f2411ad1640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f240c153210 0x7f240c158530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:16.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.922+0000 7f2411ad1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f240c163f40 con 0x7f240c0c6d70 2024-04-25T00:01:16.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.922+0000 7f2411ad1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f240c163dc0 con 0x7f240c0c75c0 2024-04-25T00:01:16.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.926+0000 7f2411ad1640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f240c1640c0 con 0x7f240c153210 2024-04-25T00:01:16.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.926+0000 7f240b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f240c0c6d70 0x7f240c14bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:16.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.926+0000 7f240b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f240c0c6d70 0x7f240c14bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54336/0 (socket says 172.21.15.116:54336) 2024-04-25T00:01:16.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.926+0000 7f240b7fe640 1 -- 172.21.15.116:0/3759373985 learned_addr learned my addr 172.21.15.116:0/3759373985 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:16.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.926+0000 7f240affd640 1 --2- 172.21.15.116:0/3759373985 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f240c0c75c0 0x7f240c14c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:16.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.926+0000 7f240b7fe640 1 -- 172.21.15.116:0/3759373985 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f240c153210 msgr2=0x7f240c158530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:16.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.926+0000 7f240bfff640 1 --2- 172.21.15.116:0/3759373985 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f240c153210 0x7f240c158530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:16.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.926+0000 7f240b7fe640 1 --2- 172.21.15.116:0/3759373985 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f240c153210 0x7f240c158530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:16.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.926+0000 7f240b7fe640 1 -- 172.21.15.116:0/3759373985 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f240c0c75c0 msgr2=0x7f240c14c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:16.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.926+0000 7f240b7fe640 1 --2- 172.21.15.116:0/3759373985 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f240c0c75c0 0x7f240c14c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:16.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.926+0000 7f240b7fe640 1 -- 172.21.15.116:0/3759373985 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f240c150060 con 0x7f240c0c6d70 2024-04-25T00:01:16.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.926+0000 7f240affd640 1 --2- 172.21.15.116:0/3759373985 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f240c0c75c0 0x7f240c14c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:16.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.926+0000 7f240b7fe640 1 --2- 172.21.15.116:0/3759373985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f240c0c6d70 0x7f240c14bdc0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f23fc00c7c0 tx=0x7f23fc00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:16.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.926+0000 7f23f3fff640 1 -- 172.21.15.116:0/3759373985 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23fc016020 con 0x7f240c0c6d70 2024-04-25T00:01:16.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.926+0000 7f23f3fff640 1 -- 172.21.15.116:0/3759373985 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23fc00aa50 con 0x7f240c0c6d70 2024-04-25T00:01:16.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.926+0000 7f2411ad1640 1 -- 172.21.15.116:0/3759373985 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f240c0c2460 con 0x7f240c0c6d70 2024-04-25T00:01:16.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.926+0000 7f23f3fff640 1 -- 172.21.15.116:0/3759373985 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23fc012420 con 0x7f240c0c6d70 2024-04-25T00:01:16.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.926+0000 7f2411ad1640 1 -- 172.21.15.116:0/3759373985 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f240c157d30 con 0x7f240c0c6d70 2024-04-25T00:01:16.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.926+0000 7f2411ad1640 1 -- 172.21.15.116:0/3759373985 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f240c068350 con 0x7f240c0c6d70 2024-04-25T00:01:16.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.926+0000 7f23f3fff640 1 -- 172.21.15.116:0/3759373985 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f23fc019020 con 0x7f240c0c6d70 2024-04-25T00:01:16.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.926+0000 7f23f3fff640 1 --2- 172.21.15.116:0/3759373985 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f23f80678c0 0x7f23f8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:16.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.926+0000 7f23f3fff640 1 -- 172.21.15.116:0/3759373985 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5689+0+0 (secure 0 0 0) 0x7f23fc085f20 con 0x7f240c0c6d70 2024-04-25T00:01:16.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.926+0000 7f240affd640 1 --2- 172.21.15.116:0/3759373985 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f23f80678c0 0x7f23f8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:16.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.926+0000 7f240affd640 1 --2- 172.21.15.116:0/3759373985 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f23f80678c0 0x7f23f8069d80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2400007fd0 tx=0x7f2400009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:16.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:16.930+0000 7f23f3fff640 1 -- 172.21.15.116:0/3759373985 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f23fc020020 con 0x7f240c0c6d70 2024-04-25T00:01:17.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.086+0000 7f2411ad1640 1 -- 172.21.15.116:0/3759373985 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "b_metadata", "pg_num_min": 64} v 0) -- 0x7f240c0cad00 con 0x7f240c0c6d70 2024-04-25T00:01:17.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.446+0000 7f23f3fff640 1 -- 172.21.15.116:0/3759373985 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "b_metadata", "pg_num_min": 64}]=0 pool 'b_metadata' created v24) ==== 128+0+0 (secure 0 0 0) 0x7f23fc056b00 con 0x7f240c0c6d70 2024-04-25T00:01:17.449 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'b_metadata' created 2024-04-25T00:01:17.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.450+0000 7f2411ad1640 1 -- 172.21.15.116:0/3759373985 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f23f80678c0 msgr2=0x7f23f8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:17.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.450+0000 7f2411ad1640 1 --2- 172.21.15.116:0/3759373985 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f23f80678c0 0x7f23f8069d80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2400007fd0 tx=0x7f2400009040 comp rx=0 tx=0).stop 2024-04-25T00:01:17.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.450+0000 7f2411ad1640 1 -- 172.21.15.116:0/3759373985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f240c0c6d70 msgr2=0x7f240c14bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:17.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.450+0000 7f2411ad1640 1 --2- 172.21.15.116:0/3759373985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f240c0c6d70 0x7f240c14bdc0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f23fc00c7c0 tx=0x7f23fc00cc80 comp rx=0 tx=0).stop 2024-04-25T00:01:17.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.450+0000 7f2411ad1640 1 -- 172.21.15.116:0/3759373985 shutdown_connections 2024-04-25T00:01:17.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.450+0000 7f2411ad1640 1 --2- 172.21.15.116:0/3759373985 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f23f80678c0 0x7f23f8069d80 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:17.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.450+0000 7f2411ad1640 1 --2- 172.21.15.116:0/3759373985 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f240c153210 0x7f240c158530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:17.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.450+0000 7f2411ad1640 1 --2- 172.21.15.116:0/3759373985 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f240c0c75c0 0x7f240c14c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:17.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.450+0000 7f2411ad1640 1 --2- 172.21.15.116:0/3759373985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f240c0c6d70 0x7f240c14bdc0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:17.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.450+0000 7f2411ad1640 1 -- 172.21.15.116:0/3759373985 >> 172.21.15.116:0/3759373985 conn(0x7f240c0acbc0 msgr2=0x7f240c151f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:17.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.450+0000 7f2411ad1640 1 -- 172.21.15.116:0/3759373985 shutdown_connections 2024-04-25T00:01:17.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.450+0000 7f2411ad1640 1 -- 172.21.15.116:0/3759373985 wait complete. 2024-04-25T00:01:17.477 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create b_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-25T00:01:17.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.606+0000 7f64e8bde640 1 Processor -- start 2024-04-25T00:01:17.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.606+0000 7f64e8bde640 1 -- start start 2024-04-25T00:01:17.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.606+0000 7f64e8bde640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f64e4153130 0x7f64e41562e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:17.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.606+0000 7f64e8bde640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f64e40ca1b0 0x7f64e406bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:17.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.606+0000 7f64e8bde640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64e4156820 con 0x7f64e40c9260 2024-04-25T00:01:17.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.606+0000 7f64e8bde640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f64e40ca820 con 0x7f64e4153130 2024-04-25T00:01:17.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.606+0000 7f64e8bde640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f64e4155e50 con 0x7f64e40ca1b0 2024-04-25T00:01:17.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.606+0000 7f64e259c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f64e4153130 0x7f64e41562e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:17.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.606+0000 7f64e2d9d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f64e40c9260 0x7f64e40c96e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:60878/0 (socket says 172.21.15.116:60878) 2024-04-25T00:01:17.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.606+0000 7f64e2d9d640 1 -- 172.21.15.116:0/18397870 learned_addr learned my addr 172.21.15.116:0/18397870 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:17.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.606+0000 7f64e259c640 1 -- 172.21.15.116:0/18397870 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f64e40ca1b0 msgr2=0x7f64e406bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:17.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.606+0000 7f64e259c640 1 --2- 172.21.15.116:0/18397870 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f64e40ca1b0 0x7f64e406bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:17.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.606+0000 7f64e259c640 1 -- 172.21.15.116:0/18397870 >> v1:172.21.15.116:6789/0 conn(0x7f64e40c9260 legacy=0x7f64e40c96e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:17.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.606+0000 7f64e259c640 1 -- 172.21.15.116:0/18397870 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64e406c300 con 0x7f64e4153130 2024-04-25T00:01:17.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.610+0000 7f64e259c640 1 --2- 172.21.15.116:0/18397870 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64e4153130 0x7f64e41562e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f64d40097c0 tx=0x7f64d401c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=161326caae636ffa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:17.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.610+0000 7f64e1d9b640 1 -- 172.21.15.116:0/18397870 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64d40b0070 con 0x7f64e4153130 2024-04-25T00:01:17.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.610+0000 7f64e1d9b640 1 -- 172.21.15.116:0/18397870 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64d40ab600 con 0x7f64e4153130 2024-04-25T00:01:17.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.610+0000 7f64e1d9b640 1 -- 172.21.15.116:0/18397870 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64d40ab9d0 con 0x7f64e4153130 2024-04-25T00:01:17.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.610+0000 7f64e8bde640 1 -- 172.21.15.116:0/18397870 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64e4153130 msgr2=0x7f64e41562e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:17.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.610+0000 7f64e8bde640 1 --2- 172.21.15.116:0/18397870 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64e4153130 0x7f64e41562e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f64d40097c0 tx=0x7f64d401c380 comp rx=0 tx=0).stop 2024-04-25T00:01:17.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.610+0000 7f64e8bde640 1 -- 172.21.15.116:0/18397870 shutdown_connections 2024-04-25T00:01:17.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.610+0000 7f64e8bde640 1 --2- 172.21.15.116:0/18397870 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f64e40ca1b0 0x7f64e406bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:17.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.610+0000 7f64e8bde640 1 --2- 172.21.15.116:0/18397870 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64e4153130 0x7f64e41562e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:17.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.610+0000 7f64e8bde640 1 -- 172.21.15.116:0/18397870 >> 172.21.15.116:0/18397870 conn(0x7f64e40cb310 msgr2=0x7f64e40cb720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:17.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.610+0000 7f64e8bde640 1 -- 172.21.15.116:0/18397870 shutdown_connections 2024-04-25T00:01:17.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.610+0000 7f64e8bde640 1 -- 172.21.15.116:0/18397870 wait complete. 2024-04-25T00:01:17.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.614+0000 7f64e8bde640 1 Processor -- start 2024-04-25T00:01:17.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.614+0000 7f64e8bde640 1 -- start start 2024-04-25T00:01:17.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.614+0000 7f64e8bde640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f64e40c9260 0x7f64e41ee0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:17.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.614+0000 7f64e8bde640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64e40ca1b0 0x7f64e414e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:17.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.614+0000 7f64e8bde640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64e4153130 0x7f64e4159040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:17.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.614+0000 7f64e8bde640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f64e40bea70 con 0x7f64e40ca1b0 2024-04-25T00:01:17.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.614+0000 7f64e8bde640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f64e40be8f0 con 0x7f64e4153130 2024-04-25T00:01:17.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.614+0000 7f64e8bde640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f64e4162990 con 0x7f64e40c9260 2024-04-25T00:01:17.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.614+0000 7f64dbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64e40ca1b0 0x7f64e414e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:17.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.614+0000 7f64dbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64e40ca1b0 0x7f64e414e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54348/0 (socket says 172.21.15.116:54348) 2024-04-25T00:01:17.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.614+0000 7f64dbfff640 1 -- 172.21.15.116:0/1430617430 learned_addr learned my addr 172.21.15.116:0/1430617430 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:17.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.614+0000 7f64e259c640 1 --2- 172.21.15.116:0/1430617430 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f64e40c9260 0x7f64e41ee0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:17.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.614+0000 7f64e2d9d640 1 --2- 172.21.15.116:0/1430617430 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64e4153130 0x7f64e4159040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:17.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.614+0000 7f64dbfff640 1 -- 172.21.15.116:0/1430617430 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f64e40c9260 msgr2=0x7f64e41ee0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:17.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.614+0000 7f64dbfff640 1 --2- 172.21.15.116:0/1430617430 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f64e40c9260 0x7f64e41ee0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:17.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.614+0000 7f64dbfff640 1 -- 172.21.15.116:0/1430617430 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64e4153130 msgr2=0x7f64e4159040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:17.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.614+0000 7f64dbfff640 1 --2- 172.21.15.116:0/1430617430 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64e4153130 0x7f64e4159040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:17.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.614+0000 7f64dbfff640 1 -- 172.21.15.116:0/1430617430 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64e40c4d50 con 0x7f64e40ca1b0 2024-04-25T00:01:17.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.614+0000 7f64dbfff640 1 --2- 172.21.15.116:0/1430617430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64e40ca1b0 0x7f64e414e690 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f64cc00b480 tx=0x7f64cc00b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:17.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.614+0000 7f64daffd640 1 -- 172.21.15.116:0/1430617430 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64cc014070 con 0x7f64e40ca1b0 2024-04-25T00:01:17.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.614+0000 7f64daffd640 1 -- 172.21.15.116:0/1430617430 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64cc010890 con 0x7f64e40ca1b0 2024-04-25T00:01:17.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.614+0000 7f64e8bde640 1 -- 172.21.15.116:0/1430617430 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64e414a2b0 con 0x7f64e40ca1b0 2024-04-25T00:01:17.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.614+0000 7f64daffd640 1 -- 172.21.15.116:0/1430617430 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64cc010c80 con 0x7f64e40ca1b0 2024-04-25T00:01:17.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.614+0000 7f64e8bde640 1 -- 172.21.15.116:0/1430617430 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64e414a870 con 0x7f64e40ca1b0 2024-04-25T00:01:17.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.614+0000 7f64e8bde640 1 -- 172.21.15.116:0/1430617430 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64e404f510 con 0x7f64e40ca1b0 2024-04-25T00:01:17.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.614+0000 7f64daffd640 1 -- 172.21.15.116:0/1430617430 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f64cc017070 con 0x7f64e40ca1b0 2024-04-25T00:01:17.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.614+0000 7f64daffd640 1 --2- 172.21.15.116:0/1430617430 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f64c40678c0 0x7f64c4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:17.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.614+0000 7f64daffd640 1 -- 172.21.15.116:0/1430617430 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6054+0+0 (secure 0 0 0) 0x7f64cc085b80 con 0x7f64e40ca1b0 2024-04-25T00:01:17.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.618+0000 7f64e259c640 1 --2- 172.21.15.116:0/1430617430 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f64c40678c0 0x7f64c4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:17.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.618+0000 7f64e259c640 1 --2- 172.21.15.116:0/1430617430 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f64c40678c0 0x7f64c4069d80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f64d40ad000 tx=0x7f64d40a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:17.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.622+0000 7f64daffd640 1 -- 172.21.15.116:0/1430617430 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f64cc0565f0 con 0x7f64e40ca1b0 2024-04-25T00:01:17.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:17.782+0000 7f64e8bde640 1 -- 172.21.15.116:0/1430617430 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "b_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f64e4159cc0 con 0x7f64e40ca1b0 2024-04-25T00:01:18.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.450+0000 7f64daffd640 1 -- 172.21.15.116:0/1430617430 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "b_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'b_data' created v25) ==== 160+0+0 (secure 0 0 0) 0x7f64cc010e20 con 0x7f64e40ca1b0 2024-04-25T00:01:18.453 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'b_data' created 2024-04-25T00:01:18.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.458+0000 7f64e8bde640 1 -- 172.21.15.116:0/1430617430 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f64c40678c0 msgr2=0x7f64c4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:18.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.458+0000 7f64e8bde640 1 --2- 172.21.15.116:0/1430617430 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f64c40678c0 0x7f64c4069d80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f64d40ad000 tx=0x7f64d40a6000 comp rx=0 tx=0).stop 2024-04-25T00:01:18.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.458+0000 7f64e8bde640 1 -- 172.21.15.116:0/1430617430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64e40ca1b0 msgr2=0x7f64e414e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:18.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.458+0000 7f64e8bde640 1 --2- 172.21.15.116:0/1430617430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64e40ca1b0 0x7f64e414e690 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f64cc00b480 tx=0x7f64cc00b940 comp rx=0 tx=0).stop 2024-04-25T00:01:18.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.458+0000 7f64e8bde640 1 -- 172.21.15.116:0/1430617430 shutdown_connections 2024-04-25T00:01:18.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.458+0000 7f64e8bde640 1 --2- 172.21.15.116:0/1430617430 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f64c40678c0 0x7f64c4069d80 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:18.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.458+0000 7f64e8bde640 1 --2- 172.21.15.116:0/1430617430 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64e4153130 0x7f64e4159040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:18.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.458+0000 7f64e8bde640 1 --2- 172.21.15.116:0/1430617430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64e40ca1b0 0x7f64e414e690 secure :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f64cc00b480 tx=0x7f64cc00b940 comp rx=0 tx=0).stop 2024-04-25T00:01:18.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.458+0000 7f64e8bde640 1 --2- 172.21.15.116:0/1430617430 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f64e40c9260 0x7f64e41ee0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:18.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.458+0000 7f64e8bde640 1 -- 172.21.15.116:0/1430617430 >> 172.21.15.116:0/1430617430 conn(0x7f64e40cb310 msgr2=0x7f64e4154c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:18.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.462+0000 7f64e8bde640 1 -- 172.21.15.116:0/1430617430 shutdown_connections 2024-04-25T00:01:18.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.462+0000 7f64e8bde640 1 -- 172.21.15.116:0/1430617430 wait complete. 2024-04-25T00:01:18.496 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new b b_metadata b_data 2024-04-25T00:01:18.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.614+0000 7fcf556d3640 1 Processor -- start 2024-04-25T00:01:18.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.614+0000 7fcf556d3640 1 -- start start 2024-04-25T00:01:18.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.614+0000 7fcf556d3640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fcf50150d60 0x7fcf500c2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:18.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.614+0000 7fcf556d3640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcf500a9630 0x7fcf50156e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:18.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.614+0000 7fcf556d3640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf500a9cb0 con 0x7fcf500c2e70 2024-04-25T00:01:18.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.614+0000 7fcf556d3640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fcf500a9f90 con 0x7fcf500a9630 2024-04-25T00:01:18.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.614+0000 7fcf556d3640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fcf500aa270 con 0x7fcf50150d60 2024-04-25T00:01:18.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.614+0000 7fcf4e7fc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcf500a9630 0x7fcf50156e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:18.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.614+0000 7fcf4e7fc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcf500a9630 0x7fcf50156e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54364/0 (socket says 172.21.15.116:54364) 2024-04-25T00:01:18.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.614+0000 7fcf4e7fc640 1 -- 172.21.15.116:0/1272691206 learned_addr learned my addr 172.21.15.116:0/1272691206 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:18.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.614+0000 7fcf4effd640 1 --2- 172.21.15.116:0/1272691206 >> v2:172.21.15.174:3300/0 conn(0x7fcf50150d60 0x7fcf500c2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:18.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.614+0000 7fcf4e7fc640 1 -- 172.21.15.116:0/1272691206 >> v2:172.21.15.174:3300/0 conn(0x7fcf50150d60 msgr2=0x7fcf500c2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:18.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.614+0000 7fcf4e7fc640 1 --2- 172.21.15.116:0/1272691206 >> v2:172.21.15.174:3300/0 conn(0x7fcf50150d60 0x7fcf500c2930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:18.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.614+0000 7fcf4e7fc640 1 -- 172.21.15.116:0/1272691206 >> v1:172.21.15.174:6789/0 conn(0x7fcf500c2e70 legacy=0x7fcf50157340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:18.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.614+0000 7fcf4e7fc640 1 -- 172.21.15.116:0/1272691206 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf500aa550 con 0x7fcf500a9630 2024-04-25T00:01:18.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.618+0000 7fcf4e7fc640 1 --2- 172.21.15.116:0/1272691206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf500a9630 0x7fcf50156e00 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fcf50068090 tx=0x7fcf4401c390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b4a200511d19727 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:18.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.618+0000 7fcf4dffb640 1 -- 172.21.15.116:0/1272691206 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf440b1070 con 0x7fcf500a9630 2024-04-25T00:01:18.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.618+0000 7fcf4dffb640 1 -- 172.21.15.116:0/1272691206 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf440ac410 con 0x7fcf500a9630 2024-04-25T00:01:18.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.618+0000 7fcf4dffb640 1 -- 172.21.15.116:0/1272691206 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf440ac7e0 con 0x7fcf500a9630 2024-04-25T00:01:18.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.618+0000 7fcf556d3640 1 -- 172.21.15.116:0/1272691206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf500a9630 msgr2=0x7fcf50156e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:18.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.618+0000 7fcf556d3640 1 --2- 172.21.15.116:0/1272691206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf500a9630 0x7fcf50156e00 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fcf50068090 tx=0x7fcf4401c390 comp rx=0 tx=0).stop 2024-04-25T00:01:18.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.618+0000 7fcf556d3640 1 -- 172.21.15.116:0/1272691206 shutdown_connections 2024-04-25T00:01:18.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.618+0000 7fcf556d3640 1 --2- 172.21.15.116:0/1272691206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf500a9630 0x7fcf50156e00 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:18.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.618+0000 7fcf556d3640 1 --2- 172.21.15.116:0/1272691206 >> v2:172.21.15.174:3300/0 conn(0x7fcf50150d60 0x7fcf500c2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:18.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.618+0000 7fcf556d3640 1 -- 172.21.15.116:0/1272691206 >> 172.21.15.116:0/1272691206 conn(0x7fcf500c3a20 msgr2=0x7fcf500c3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:18.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.618+0000 7fcf556d3640 1 -- 172.21.15.116:0/1272691206 shutdown_connections 2024-04-25T00:01:18.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.618+0000 7fcf556d3640 1 -- 172.21.15.116:0/1272691206 wait complete. 2024-04-25T00:01:18.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.618+0000 7fcf556d3640 1 Processor -- start 2024-04-25T00:01:18.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.618+0000 7fcf556d3640 1 -- start start 2024-04-25T00:01:18.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.618+0000 7fcf556d3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf500a9630 0x7fcf50155c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:18.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.618+0000 7fcf556d3640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcf500c2e70 0x7fcf501f1d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:18.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.618+0000 7fcf556d3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf50150d60 0x7fcf50145b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:18.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.618+0000 7fcf556d3640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcf50071fd0 con 0x7fcf500a9630 2024-04-25T00:01:18.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.618+0000 7fcf556d3640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcf500ab7a0 con 0x7fcf50150d60 2024-04-25T00:01:18.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.618+0000 7fcf556d3640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fcf50072150 con 0x7fcf500c2e70 2024-04-25T00:01:18.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.618+0000 7fcf4effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf500a9630 0x7fcf50155c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:18.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.618+0000 7fcf4effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf500a9630 0x7fcf50155c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54374/0 (socket says 172.21.15.116:54374) 2024-04-25T00:01:18.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.618+0000 7fcf4effd640 1 -- 172.21.15.116:0/972287968 learned_addr learned my addr 172.21.15.116:0/972287968 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:18.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.618+0000 7fcf4e7fc640 1 --2- 172.21.15.116:0/972287968 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcf500c2e70 0x7fcf501f1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:18.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.618+0000 7fcf4effd640 1 -- 172.21.15.116:0/972287968 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcf500c2e70 msgr2=0x7fcf501f1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:18.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.618+0000 7fcf4effd640 1 --2- 172.21.15.116:0/972287968 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcf500c2e70 0x7fcf501f1d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:18.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.618+0000 7fcf4f7fe640 1 --2- 172.21.15.116:0/972287968 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf50150d60 0x7fcf50145b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:18.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.618+0000 7fcf4effd640 1 -- 172.21.15.116:0/972287968 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf50150d60 msgr2=0x7fcf50145b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:18.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.618+0000 7fcf4effd640 1 --2- 172.21.15.116:0/972287968 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf50150d60 0x7fcf50145b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:18.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.618+0000 7fcf4effd640 1 -- 172.21.15.116:0/972287968 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf5006d150 con 0x7fcf500a9630 2024-04-25T00:01:18.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.618+0000 7fcf4effd640 1 --2- 172.21.15.116:0/972287968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf500a9630 0x7fcf50155c70 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fcf4000d5f0 tx=0x7fcf4000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:18.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.618+0000 7fcf377fe640 1 -- 172.21.15.116:0/972287968 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf400125f0 con 0x7fcf500a9630 2024-04-25T00:01:18.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.622+0000 7fcf377fe640 1 -- 172.21.15.116:0/972287968 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf40019070 con 0x7fcf500a9630 2024-04-25T00:01:18.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.622+0000 7fcf556d3640 1 -- 172.21.15.116:0/972287968 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf500c5730 con 0x7fcf500a9630 2024-04-25T00:01:18.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.622+0000 7fcf377fe640 1 -- 172.21.15.116:0/972287968 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf400129b0 con 0x7fcf500a9630 2024-04-25T00:01:18.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.622+0000 7fcf556d3640 1 -- 172.21.15.116:0/972287968 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf5014b3f0 con 0x7fcf500a9630 2024-04-25T00:01:18.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.622+0000 7fcf556d3640 1 -- 172.21.15.116:0/972287968 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf500c5520 con 0x7fcf500a9630 2024-04-25T00:01:18.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.622+0000 7fcf377fe640 1 -- 172.21.15.116:0/972287968 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fcf40012b50 con 0x7fcf500a9630 2024-04-25T00:01:18.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.622+0000 7fcf377fe640 1 --2- 172.21.15.116:0/972287968 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcf3c0678c0 0x7fcf3c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:18.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.622+0000 7fcf377fe640 1 -- 172.21.15.116:0/972287968 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6431+0+0 (secure 0 0 0) 0x7fcf40086d10 con 0x7fcf500a9630 2024-04-25T00:01:18.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.622+0000 7fcf4e7fc640 1 --2- 172.21.15.116:0/972287968 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcf3c0678c0 0x7fcf3c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:18.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.622+0000 7fcf4e7fc640 1 --2- 172.21.15.116:0/972287968 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcf3c0678c0 0x7fcf3c069d80 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fcf440ae000 tx=0x7fcf440a6040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:18.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.626+0000 7fcf377fe640 1 -- 172.21.15.116:0/972287968 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fcf40013390 con 0x7fcf500a9630 2024-04-25T00:01:18.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:18.822+0000 7fcf556d3640 1 -- 172.21.15.116:0/972287968 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "b", "metadata": "b_metadata", "data": "b_data"} v 0) -- 0x7fcf5004f510 con 0x7fcf500a9630 2024-04-25T00:01:19.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.502+0000 7fcf377fe640 1 -- 172.21.15.116:0/972287968 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "b", "metadata": "b_metadata", "data": "b_data"}]=0 new fs with metadata pool 4 and data pool 5 v10) ==== 157+0+0 (secure 0 0 0) 0x7fcf400576f0 con 0x7fcf500a9630 2024-04-25T00:01:19.510 INFO:teuthology.orchestra.run.smithi116.stderr:new fs with metadata pool 4 and data pool 5 2024-04-25T00:01:19.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.514+0000 7fcf556d3640 1 -- 172.21.15.116:0/972287968 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcf3c0678c0 msgr2=0x7fcf3c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:19.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.514+0000 7fcf556d3640 1 --2- 172.21.15.116:0/972287968 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcf3c0678c0 0x7fcf3c069d80 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fcf440ae000 tx=0x7fcf440a6040 comp rx=0 tx=0).stop 2024-04-25T00:01:19.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.514+0000 7fcf556d3640 1 -- 172.21.15.116:0/972287968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf500a9630 msgr2=0x7fcf50155c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:19.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.514+0000 7fcf556d3640 1 --2- 172.21.15.116:0/972287968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf500a9630 0x7fcf50155c70 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fcf4000d5f0 tx=0x7fcf4000dab0 comp rx=0 tx=0).stop 2024-04-25T00:01:19.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.514+0000 7fcf556d3640 1 -- 172.21.15.116:0/972287968 shutdown_connections 2024-04-25T00:01:19.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.514+0000 7fcf556d3640 1 --2- 172.21.15.116:0/972287968 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcf3c0678c0 0x7fcf3c069d80 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:19.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.514+0000 7fcf556d3640 1 --2- 172.21.15.116:0/972287968 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf50150d60 0x7fcf50145b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:19.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.514+0000 7fcf556d3640 1 --2- 172.21.15.116:0/972287968 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcf500c2e70 0x7fcf501f1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:19.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.514+0000 7fcf556d3640 1 --2- 172.21.15.116:0/972287968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf500a9630 0x7fcf50155c70 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:19.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.514+0000 7fcf556d3640 1 -- 172.21.15.116:0/972287968 >> 172.21.15.116:0/972287968 conn(0x7fcf500c3a20 msgr2=0x7fcf5014dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:19.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.514+0000 7fcf556d3640 1 -- 172.21.15.116:0/972287968 shutdown_connections 2024-04-25T00:01:19.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.518+0000 7fcf556d3640 1 -- 172.21.15.116:0/972287968 wait complete. 2024-04-25T00:01:19.543 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T00:01:19.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.654+0000 7fcfd66e5640 1 Processor -- start 2024-04-25T00:01:19.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.654+0000 7fcfd66e5640 1 -- start start 2024-04-25T00:01:19.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.658+0000 7fcfd66e5640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcfd013e310 0x7fcfd00c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:19.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.658+0000 7fcfd66e5640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfd006d200 con 0x7fcfd00c2f50 2024-04-25T00:01:19.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.658+0000 7fcfd66e5640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfd006cb50 con 0x7fcfd013ed60 2024-04-25T00:01:19.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.658+0000 7fcfd66e5640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fcfd0070d40 con 0x7fcfd013e310 2024-04-25T00:01:19.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.658+0000 7fcfcffff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcfd013e310 0x7fcfd00c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:19.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.658+0000 7fcfcf7fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fcfd00c2f50 0x7fcfd01559c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:60894/0 (socket says 172.21.15.116:60894) 2024-04-25T00:01:19.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.658+0000 7fcfcf7fe640 1 -- 172.21.15.116:0/3582318847 learned_addr learned my addr 172.21.15.116:0/3582318847 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:19.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.658+0000 7fcfceffd640 1 -- 172.21.15.116:0/3582318847 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2432986204 0 0) 0x7fcfd006d200 con 0x7fcfd00c2f50 2024-04-25T00:01:19.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.658+0000 7fcfceffd640 1 -- 172.21.15.116:0/3582318847 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcfbc003530 con 0x7fcfd00c2f50 2024-04-25T00:01:19.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.658+0000 7fcfcffff640 1 -- 172.21.15.116:0/3582318847 >> v1:172.21.15.174:6789/0 conn(0x7fcfd013ed60 legacy=0x7fcfd01560d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:19.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.658+0000 7fcfcffff640 1 -- 172.21.15.116:0/3582318847 >> v1:172.21.15.116:6789/0 conn(0x7fcfd00c2f50 legacy=0x7fcfd01559c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:19.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.658+0000 7fcfcffff640 1 -- 172.21.15.116:0/3582318847 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfd006d470 con 0x7fcfd013e310 2024-04-25T00:01:19.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.662+0000 7fcfcffff640 1 --2- 172.21.15.116:0/3582318847 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfd013e310 0x7fcfd00c2a10 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fcfc80097c0 tx=0x7fcfc801c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fc782a54788e3c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:19.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.662+0000 7fcfceffd640 1 -- 172.21.15.116:0/3582318847 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc80b0070 con 0x7fcfd013e310 2024-04-25T00:01:19.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.662+0000 7fcfceffd640 1 -- 172.21.15.116:0/3582318847 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfc80ab600 con 0x7fcfd013e310 2024-04-25T00:01:19.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.662+0000 7fcfceffd640 1 -- 172.21.15.116:0/3582318847 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc80ab9d0 con 0x7fcfd013e310 2024-04-25T00:01:19.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.662+0000 7fcfd66e5640 1 -- 172.21.15.116:0/3582318847 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfd013e310 msgr2=0x7fcfd00c2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:19.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.662+0000 7fcfd66e5640 1 --2- 172.21.15.116:0/3582318847 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfd013e310 0x7fcfd00c2a10 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fcfc80097c0 tx=0x7fcfc801c340 comp rx=0 tx=0).stop 2024-04-25T00:01:19.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.662+0000 7fcfd66e5640 1 -- 172.21.15.116:0/3582318847 shutdown_connections 2024-04-25T00:01:19.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.662+0000 7fcfd66e5640 1 --2- 172.21.15.116:0/3582318847 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfd013e310 0x7fcfd00c2a10 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:19.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.662+0000 7fcfd66e5640 1 -- 172.21.15.116:0/3582318847 >> 172.21.15.116:0/3582318847 conn(0x7fcfd00c3b00 msgr2=0x7fcfd00c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:19.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.662+0000 7fcfd66e5640 1 -- 172.21.15.116:0/3582318847 shutdown_connections 2024-04-25T00:01:19.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.662+0000 7fcfd66e5640 1 -- 172.21.15.116:0/3582318847 wait complete. 2024-04-25T00:01:19.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.662+0000 7fcfd66e5640 1 Processor -- start 2024-04-25T00:01:19.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.662+0000 7fcfd66e5640 1 -- start start 2024-04-25T00:01:19.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.662+0000 7fcfd66e5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfd00c2f50 0x7fcfd00c57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:19.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.662+0000 7fcfd66e5640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcfd013e310 0x7fcfd0152a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:19.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.662+0000 7fcfd66e5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfd013ed60 0x7fcfd0152f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:19.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.662+0000 7fcfd66e5640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd0163c10 con 0x7fcfd00c2f50 2024-04-25T00:01:19.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.662+0000 7fcfd66e5640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcfd0071020 con 0x7fcfd013ed60 2024-04-25T00:01:19.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.662+0000 7fcfd66e5640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fcfd0163d90 con 0x7fcfd013e310 2024-04-25T00:01:19.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.666+0000 7fcfcffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfd00c2f50 0x7fcfd00c57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:19.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.666+0000 7fcfcffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfd00c2f50 0x7fcfd00c57e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54394/0 (socket says 172.21.15.116:54394) 2024-04-25T00:01:19.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.666+0000 7fcfcffff640 1 -- 172.21.15.116:0/1128526450 learned_addr learned my addr 172.21.15.116:0/1128526450 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:19.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.666+0000 7fcfcffff640 1 -- 172.21.15.116:0/1128526450 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcfd013e310 msgr2=0x7fcfd0152a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:19.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.666+0000 7fcfcffff640 1 --2- 172.21.15.116:0/1128526450 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcfd013e310 0x7fcfd0152a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:19.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.666+0000 7fcfcffff640 1 -- 172.21.15.116:0/1128526450 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfd013ed60 msgr2=0x7fcfd0152f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:19.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.666+0000 7fcfd4c82640 1 --2- 172.21.15.116:0/1128526450 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfd013ed60 0x7fcfd0152f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:19.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.666+0000 7fcfcffff640 1 --2- 172.21.15.116:0/1128526450 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfd013ed60 0x7fcfd0152f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:19.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.666+0000 7fcfcffff640 1 -- 172.21.15.116:0/1128526450 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfd01f25e0 con 0x7fcfd00c2f50 2024-04-25T00:01:19.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.666+0000 7fcfcffff640 1 --2- 172.21.15.116:0/1128526450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfd00c2f50 0x7fcfd00c57e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fcfc80ae040 tx=0x7fcfc801ab80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:19.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.666+0000 7fcfccff9640 1 -- 172.21.15.116:0/1128526450 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc80b0050 con 0x7fcfd00c2f50 2024-04-25T00:01:19.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.666+0000 7fcfccff9640 1 -- 172.21.15.116:0/1128526450 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfc80bc070 con 0x7fcfd00c2f50 2024-04-25T00:01:19.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.666+0000 7fcfd66e5640 1 -- 172.21.15.116:0/1128526450 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfd01f28c0 con 0x7fcfd00c2f50 2024-04-25T00:01:19.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.666+0000 7fcfccff9640 1 -- 172.21.15.116:0/1128526450 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc80b5a70 con 0x7fcfd00c2f50 2024-04-25T00:01:19.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.666+0000 7fcfd66e5640 1 -- 172.21.15.116:0/1128526450 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfd0139d30 con 0x7fcfd00c2f50 2024-04-25T00:01:19.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.666+0000 7fcfd66e5640 1 -- 172.21.15.116:0/1128526450 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfd0068350 con 0x7fcfd00c2f50 2024-04-25T00:01:19.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.666+0000 7fcfccff9640 1 -- 172.21.15.116:0/1128526450 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fcfc80b5c10 con 0x7fcfd00c2f50 2024-04-25T00:01:19.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.666+0000 7fcfccff9640 1 --2- 172.21.15.116:0/1128526450 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcfb00678c0 0x7fcfb0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:19.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.666+0000 7fcfccff9640 1 -- 172.21.15.116:0/1128526450 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7fcfc8121080 con 0x7fcfd00c2f50 2024-04-25T00:01:19.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.666+0000 7fcfcf7fe640 1 --2- 172.21.15.116:0/1128526450 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcfb00678c0 0x7fcfb0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:19.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.670+0000 7fcfcf7fe640 1 --2- 172.21.15.116:0/1128526450 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcfb00678c0 0x7fcfb0069d80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fcfb8007fd0 tx=0x7fcfb8007940 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:19.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.670+0000 7fcfccff9640 1 -- 172.21.15.116:0/1128526450 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fcfc8016050 con 0x7fcfd00c2f50 2024-04-25T00:01:19.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.834+0000 7fcfd66e5640 1 -- 172.21.15.116:0/1128526450 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcfd00ac760 con 0x7fcfd00c2f50 2024-04-25T00:01:19.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.838+0000 7fcfccff9640 1 -- 172.21.15.116:0/1128526450 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+20324 (secure 0 0 0) 0x7fcfc80f1b20 con 0x7fcfd00c2f50 2024-04-25T00:01:19.839 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:01:19.839 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":26,"fsid":"6d6a1aa7-0478-4c83-8057-05917e81f046","created":"2024-04-25T00:00:40.682493+0000","modified":"2024-04-25T00:01:18.825112+0000","last_up_change":"2024-04-25T00:00:51.822174+0000","last_in_change":"2024-04-25T00:00:42.980473+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T00:00:53.324376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-04-25T00:01:03.197326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-04-25T00:01:03.728618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-04-25T00:01:17.125210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-04-25T00:01:17.821873+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8b6ddcb2-1ad9-4167-961c-bd6936bb9060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6809","nonce":3751765682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6811","nonce":3751765682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6815","nonce":3751765682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6813","nonce":3751765682}]},"public_addr":"172.21.15.116:6809/3751765682","cluster_addr":"172.21.15.116:6811/3751765682","heartbeat_back_addr":"172.21.15.116:6815/3751765682","heartbeat_front_addr":"172.21.15.116:6813/3751765682","state":["exists","up"]},{"osd":1,"uuid":"a6431768-994f-41da-8853-886f5687fb91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6801","nonce":377879124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6803","nonce":377879124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6807","nonce":377879124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6805","nonce":377879124}]},"public_addr":"172.21.15.116:6801/377879124","cluster_addr":"172.21.15.116:6803/377879124","heartbeat_back_addr":"172.21.15.116:6807/377879124","heartbeat_front_addr":"172.21.15.116:6805/377879124","state":["exists","up"]},{"osd":2,"uuid":"4f69bd40-5a23-45ff-8693-e1de75e1aa9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6817","nonce":1099282267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6819","nonce":1099282267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6823","nonce":1099282267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6821","nonce":1099282267}]},"public_addr":"172.21.15.116:6817/1099282267","cluster_addr":"172.21.15.116:6819/1099282267","heartbeat_back_addr":"172.21.15.116:6823/1099282267","heartbeat_front_addr":"172.21.15.116:6821/1099282267","state":["exists","up"]},{"osd":3,"uuid":"6a0d5344-8b6f-40d3-918d-0f0a4be688e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6825","nonce":1179942661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6827","nonce":1179942661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6831","nonce":1179942661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6829","nonce":1179942661}]},"public_addr":"172.21.15.116:6825/1179942661","cluster_addr":"172.21.15.116:6827/1179942661","heartbeat_back_addr":"172.21.15.116:6831/1179942661","heartbeat_front_addr":"172.21.15.116:6829/1179942661","state":["exists","up"]},{"osd":4,"uuid":"132e92e4-b825-441e-965a-ca00533fa86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6801","nonce":3132444330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6803","nonce":3132444330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6807","nonce":3132444330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6805","nonce":3132444330}]},"public_addr":"172.21.15.174:6801/3132444330","cluster_addr":"172.21.15.174:6803/3132444330","heartbeat_back_addr":"172.21.15.174:6807/3132444330","heartbeat_front_addr":"172.21.15.174:6805/3132444330","state":["exists","up"]},{"osd":5,"uuid":"a2eeb5b6-1b31-43cc-8306-a0adf3ee15bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6809","nonce":881817164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6811","nonce":881817164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6815","nonce":881817164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6813","nonce":881817164}]},"public_addr":"172.21.15.174:6809/881817164","cluster_addr":"172.21.15.174:6811/881817164","heartbeat_back_addr":"172.21.15.174:6815/881817164","heartbeat_front_addr":"172.21.15.174:6813/881817164","state":["exists","up"]},{"osd":6,"uuid":"588e873a-c841-4ba7-90f2-8956c87a9983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6825","nonce":117574251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6827","nonce":117574251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6831","nonce":117574251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6829","nonce":117574251}]},"public_addr":"172.21.15.174:6825/117574251","cluster_addr":"172.21.15.174:6827/117574251","heartbeat_back_addr":"172.21.15.174:6831/117574251","heartbeat_front_addr":"172.21.15.174:6829/117574251","state":["exists","up"]},{"osd":7,"uuid":"3b90f9b0-8a0e-4883-b088-ca1d4012870c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6817","nonce":2543899196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6819","nonce":2543899196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6823","nonce":2543899196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6821","nonce":2543899196}]},"public_addr":"172.21.15.174:6817/2543899196","cluster_addr":"172.21.15.174:6819/2543899196","heartbeat_back_addr":"172.21.15.174:6823/2543899196","heartbeat_front_addr":"172.21.15.174:6821/2543899196","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.293769+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.310034+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.404865+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.549658+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.192174+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.297266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.493649+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.452309+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:01:19.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.838+0000 7fcfd66e5640 1 -- 172.21.15.116:0/1128526450 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcfb00678c0 msgr2=0x7fcfb0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:19.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.838+0000 7fcfd66e5640 1 --2- 172.21.15.116:0/1128526450 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcfb00678c0 0x7fcfb0069d80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fcfb8007fd0 tx=0x7fcfb8007940 comp rx=0 tx=0).stop 2024-04-25T00:01:19.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.838+0000 7fcfd66e5640 1 -- 172.21.15.116:0/1128526450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfd00c2f50 msgr2=0x7fcfd00c57e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:19.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.838+0000 7fcfd66e5640 1 --2- 172.21.15.116:0/1128526450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfd00c2f50 0x7fcfd00c57e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fcfc80ae040 tx=0x7fcfc801ab80 comp rx=0 tx=0).stop 2024-04-25T00:01:19.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.838+0000 7fcfd66e5640 1 -- 172.21.15.116:0/1128526450 shutdown_connections 2024-04-25T00:01:19.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.838+0000 7fcfd66e5640 1 --2- 172.21.15.116:0/1128526450 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcfb00678c0 0x7fcfb0069d80 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:19.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.838+0000 7fcfd66e5640 1 --2- 172.21.15.116:0/1128526450 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfd013ed60 0x7fcfd0152f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:19.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.838+0000 7fcfd66e5640 1 --2- 172.21.15.116:0/1128526450 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcfd013e310 0x7fcfd0152a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:19.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.838+0000 7fcfd66e5640 1 --2- 172.21.15.116:0/1128526450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfd00c2f50 0x7fcfd00c57e0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:19.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.838+0000 7fcfd66e5640 1 -- 172.21.15.116:0/1128526450 >> 172.21.15.116:0/1128526450 conn(0x7fcfd00c3b00 msgr2=0x7fcfd0072810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:19.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.838+0000 7fcfd66e5640 1 -- 172.21.15.116:0/1128526450 shutdown_connections 2024-04-25T00:01:19.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.838+0000 7fcfd66e5640 1 -- 172.21.15.116:0/1128526450 wait complete. 2024-04-25T00:01:19.862 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T00:01:19.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.974+0000 7f50df422640 1 Processor -- start 2024-04-25T00:01:19.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.974+0000 7f50df422640 1 -- start start 2024-04-25T00:01:19.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.974+0000 7f50df422640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50d80a99f0 0x7f50d8153ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:19.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.978+0000 7f50df422640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50d80c3910 con 0x7f50d8150d20 2024-04-25T00:01:19.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.978+0000 7f50df422640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50d8154530 con 0x7f50d80a94a0 2024-04-25T00:01:19.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.978+0000 7f50df422640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f50d80c16e0 con 0x7f50d80a99f0 2024-04-25T00:01:19.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.978+0000 7f50dd1be640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f50d8150d20 0x7f50d80c2a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:60904/0 (socket says 172.21.15.116:60904) 2024-04-25T00:01:19.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.978+0000 7f50dd1be640 1 -- 172.21.15.116:0/2158505993 learned_addr learned my addr 172.21.15.116:0/2158505993 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:19.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.978+0000 7f50cffff640 1 -- 172.21.15.116:0/2158505993 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3703719160 0 0) 0x7f50d80c3910 con 0x7f50d8150d20 2024-04-25T00:01:19.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.978+0000 7f50dd9bf640 1 --2- 172.21.15.116:0/2158505993 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50d80a99f0 0x7f50d8153ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:19.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.978+0000 7f50cffff640 1 -- 172.21.15.116:0/2158505993 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f50c4003530 con 0x7f50d8150d20 2024-04-25T00:01:19.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.978+0000 7f50cffff640 1 -- 172.21.15.116:0/2158505993 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2041142969 0 0) 0x7f50c4003530 con 0x7f50d8150d20 2024-04-25T00:01:19.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.978+0000 7f50cffff640 1 -- 172.21.15.116:0/2158505993 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50d80a99f0 msgr2=0x7f50d8153ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:19.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.978+0000 7f50cffff640 1 --2- 172.21.15.116:0/2158505993 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50d80a99f0 0x7f50d8153ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:19.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.978+0000 7f50cffff640 1 -- 172.21.15.116:0/2158505993 >> v1:172.21.15.174:6789/0 conn(0x7f50d80a94a0 legacy=0x7f50d81538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:19.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.978+0000 7f50cffff640 1 -- 172.21.15.116:0/2158505993 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50d80c1400 con 0x7f50d8150d20 2024-04-25T00:01:19.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.978+0000 7f50cffff640 1 -- 172.21.15.116:0/2158505993 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f50d4001680 con 0x7f50d8150d20 2024-04-25T00:01:19.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.978+0000 7f50cffff640 1 -- 172.21.15.116:0/2158505993 >> v1:172.21.15.116:6789/0 conn(0x7f50d8150d20 legacy=0x7f50d80c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:19.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.978+0000 7f50cffff640 1 --2- 172.21.15.116:0/2158505993 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50c40039f0 0x7f50c4003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:19.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.978+0000 7f50cffff640 1 --2- 172.21.15.116:0/2158505993 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50c4004810 0x7f50c4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:19.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.978+0000 7f50cffff640 1 --2- 172.21.15.116:0/2158505993 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50d80a99f0 0x7f50c40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:19.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.978+0000 7f50cffff640 1 -- 172.21.15.116:0/2158505993 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f50d80c16e0 con 0x7f50c4004810 2024-04-25T00:01:19.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.978+0000 7f50cffff640 1 -- 172.21.15.116:0/2158505993 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f50c400ba80 con 0x7f50c40039f0 2024-04-25T00:01:19.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.978+0000 7f50cffff640 1 -- 172.21.15.116:0/2158505993 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f50c400bc80 con 0x7f50d80a99f0 2024-04-25T00:01:19.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.978+0000 7f50dc9bd640 1 --2- 172.21.15.116:0/2158505993 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50c4004810 0x7f50c4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:19.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.978+0000 7f50dd1be640 1 --2- 172.21.15.116:0/2158505993 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50c40039f0 0x7f50c4003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:19.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.978+0000 7f50dc9bd640 1 -- 172.21.15.116:0/2158505993 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50d80a99f0 msgr2=0x7f50c40051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:19.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.978+0000 7f50dc9bd640 1 --2- 172.21.15.116:0/2158505993 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50d80a99f0 0x7f50c40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:19.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.978+0000 7f50dc9bd640 1 -- 172.21.15.116:0/2158505993 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50c40039f0 msgr2=0x7f50c4003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:19.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.978+0000 7f50dc9bd640 1 --2- 172.21.15.116:0/2158505993 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50c40039f0 0x7f50c4003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:19.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.978+0000 7f50dc9bd640 1 -- 172.21.15.116:0/2158505993 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50c4003530 con 0x7f50c4004810 2024-04-25T00:01:19.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.978+0000 7f50dd1be640 1 --2- 172.21.15.116:0/2158505993 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50c40039f0 0x7f50c4003e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:19.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.978+0000 7f50dc9bd640 1 --2- 172.21.15.116:0/2158505993 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50c4004810 0x7f50c4004c90 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f50c8001750 tx=0x7f50c801c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0f8966ddcd88884 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:19.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.978+0000 7f50cffff640 1 -- 172.21.15.116:0/2158505993 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50c80013a0 con 0x7f50c4004810 2024-04-25T00:01:19.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.978+0000 7f50cffff640 1 -- 172.21.15.116:0/2158505993 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50c80052f0 con 0x7f50c4004810 2024-04-25T00:01:19.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.978+0000 7f50cffff640 1 -- 172.21.15.116:0/2158505993 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50c8003840 con 0x7f50c4004810 2024-04-25T00:01:19.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.982+0000 7f50df422640 1 -- 172.21.15.116:0/2158505993 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50c4004810 msgr2=0x7f50c4004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:19.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.982+0000 7f50df422640 1 --2- 172.21.15.116:0/2158505993 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50c4004810 0x7f50c4004c90 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f50c8001750 tx=0x7f50c801c3c0 comp rx=0 tx=0).stop 2024-04-25T00:01:19.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.982+0000 7f50dd9bf640 1 -- 172.21.15.116:0/2158505993 reap_dead start 2024-04-25T00:01:19.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.982+0000 7f50df422640 1 -- 172.21.15.116:0/2158505993 shutdown_connections 2024-04-25T00:01:19.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.982+0000 7f50df422640 1 -- 172.21.15.116:0/2158505993 >> 172.21.15.116:0/2158505993 conn(0x7f50d80c3b00 msgr2=0x7f50d80c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:19.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.982+0000 7f50df422640 1 -- 172.21.15.116:0/2158505993 shutdown_connections 2024-04-25T00:01:19.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.982+0000 7f50df422640 1 -- 172.21.15.116:0/2158505993 wait complete. 2024-04-25T00:01:19.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.982+0000 7f50df422640 1 Processor -- start 2024-04-25T00:01:19.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.982+0000 7f50df422640 1 -- start start 2024-04-25T00:01:19.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.982+0000 7f50df422640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50c4004810 0x7f50d814dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:19.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.982+0000 7f50df422640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50d8072a00 0x7f50d806d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:19.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.982+0000 7f50df422640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50d8071de0 0x7f50d8072260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:19.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.982+0000 7f50df422640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f50d806ec20 con 0x7f50d8071de0 2024-04-25T00:01:19.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.982+0000 7f50df422640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f50d806e940 con 0x7f50c4004810 2024-04-25T00:01:19.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.982+0000 7f50df422640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f50d80c3120 con 0x7f50d8072a00 2024-04-25T00:01:19.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.982+0000 7f50dd9bf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50d8071de0 0x7f50d8072260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:19.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.982+0000 7f50dd9bf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50d8071de0 0x7f50d8072260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54422/0 (socket says 172.21.15.116:54422) 2024-04-25T00:01:19.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.982+0000 7f50dd9bf640 1 -- 172.21.15.116:0/72712025 learned_addr learned my addr 172.21.15.116:0/72712025 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:19.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.982+0000 7f50dd1be640 1 --2- 172.21.15.116:0/72712025 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50c4004810 0x7f50d814dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:19.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.982+0000 7f50dd9bf640 1 -- 172.21.15.116:0/72712025 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50d8072a00 msgr2=0x7f50d806d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:19.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.982+0000 7f50dd9bf640 1 --2- 172.21.15.116:0/72712025 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50d8072a00 0x7f50d806d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:19.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.982+0000 7f50dd9bf640 1 -- 172.21.15.116:0/72712025 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50c4004810 msgr2=0x7f50d814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:19.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.982+0000 7f50dd9bf640 1 --2- 172.21.15.116:0/72712025 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50c4004810 0x7f50d814dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:19.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.982+0000 7f50dd9bf640 1 -- 172.21.15.116:0/72712025 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50d814ca00 con 0x7f50d8071de0 2024-04-25T00:01:19.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.982+0000 7f50dd9bf640 1 --2- 172.21.15.116:0/72712025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50d8071de0 0x7f50d8072260 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f50d80c4aa0 tx=0x7f50d000b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:19.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.982+0000 7f50cdffb640 1 -- 172.21.15.116:0/72712025 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d000bb10 con 0x7f50d8071de0 2024-04-25T00:01:19.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.982+0000 7f50cdffb640 1 -- 172.21.15.116:0/72712025 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50d001a070 con 0x7f50d8071de0 2024-04-25T00:01:19.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.982+0000 7f50df422640 1 -- 172.21.15.116:0/72712025 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50d814b2b0 con 0x7f50d8071de0 2024-04-25T00:01:19.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.982+0000 7f50cdffb640 1 -- 172.21.15.116:0/72712025 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d0014400 con 0x7f50d8071de0 2024-04-25T00:01:19.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.982+0000 7f50df422640 1 -- 172.21.15.116:0/72712025 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50d8071270 con 0x7f50d8071de0 2024-04-25T00:01:19.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.982+0000 7f50df422640 1 -- 172.21.15.116:0/72712025 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50d8068350 con 0x7f50d8071de0 2024-04-25T00:01:19.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.986+0000 7f50cdffb640 1 -- 172.21.15.116:0/72712025 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f50d0013070 con 0x7f50d8071de0 2024-04-25T00:01:19.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.986+0000 7f50cdffb640 1 --2- 172.21.15.116:0/72712025 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f50bc0678c0 0x7f50bc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:19.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.986+0000 7f50cdffb640 1 -- 172.21.15.116:0/72712025 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f50d0086210 con 0x7f50d8071de0 2024-04-25T00:01:19.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.986+0000 7f50dd1be640 1 --2- 172.21.15.116:0/72712025 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f50bc0678c0 0x7f50bc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:19.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.986+0000 7f50dd1be640 1 --2- 172.21.15.116:0/72712025 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f50bc0678c0 0x7f50bc069d80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f50d40017f0 tx=0x7f50d4000f00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:19.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:19.990+0000 7f50cdffb640 1 -- 172.21.15.116:0/72712025 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f50d0024020 con 0x7f50d8071de0 2024-04-25T00:01:20.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.150+0000 7f50df422640 1 -- 172.21.15.116:0/72712025 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f50d80c34f0 con 0x7f50d8071de0 2024-04-25T00:01:20.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.150+0000 7f50cdffb640 1 -- 172.21.15.116:0/72712025 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+20324 (secure 0 0 0) 0x7f50d0056a20 con 0x7f50d8071de0 2024-04-25T00:01:20.153 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:01:20.154 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":26,"fsid":"6d6a1aa7-0478-4c83-8057-05917e81f046","created":"2024-04-25T00:00:40.682493+0000","modified":"2024-04-25T00:01:18.825112+0000","last_up_change":"2024-04-25T00:00:51.822174+0000","last_in_change":"2024-04-25T00:00:42.980473+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T00:00:53.324376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-04-25T00:01:03.197326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-04-25T00:01:03.728618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-04-25T00:01:17.125210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-04-25T00:01:17.821873+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8b6ddcb2-1ad9-4167-961c-bd6936bb9060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6809","nonce":3751765682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6811","nonce":3751765682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6815","nonce":3751765682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6813","nonce":3751765682}]},"public_addr":"172.21.15.116:6809/3751765682","cluster_addr":"172.21.15.116:6811/3751765682","heartbeat_back_addr":"172.21.15.116:6815/3751765682","heartbeat_front_addr":"172.21.15.116:6813/3751765682","state":["exists","up"]},{"osd":1,"uuid":"a6431768-994f-41da-8853-886f5687fb91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6801","nonce":377879124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6803","nonce":377879124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6807","nonce":377879124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6805","nonce":377879124}]},"public_addr":"172.21.15.116:6801/377879124","cluster_addr":"172.21.15.116:6803/377879124","heartbeat_back_addr":"172.21.15.116:6807/377879124","heartbeat_front_addr":"172.21.15.116:6805/377879124","state":["exists","up"]},{"osd":2,"uuid":"4f69bd40-5a23-45ff-8693-e1de75e1aa9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6817","nonce":1099282267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6819","nonce":1099282267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6823","nonce":1099282267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6821","nonce":1099282267}]},"public_addr":"172.21.15.116:6817/1099282267","cluster_addr":"172.21.15.116:6819/1099282267","heartbeat_back_addr":"172.21.15.116:6823/1099282267","heartbeat_front_addr":"172.21.15.116:6821/1099282267","state":["exists","up"]},{"osd":3,"uuid":"6a0d5344-8b6f-40d3-918d-0f0a4be688e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6825","nonce":1179942661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6827","nonce":1179942661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6831","nonce":1179942661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6829","nonce":1179942661}]},"public_addr":"172.21.15.116:6825/1179942661","cluster_addr":"172.21.15.116:6827/1179942661","heartbeat_back_addr":"172.21.15.116:6831/1179942661","heartbeat_front_addr":"172.21.15.116:6829/1179942661","state":["exists","up"]},{"osd":4,"uuid":"132e92e4-b825-441e-965a-ca00533fa86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6801","nonce":3132444330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6803","nonce":3132444330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6807","nonce":3132444330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6805","nonce":3132444330}]},"public_addr":"172.21.15.174:6801/3132444330","cluster_addr":"172.21.15.174:6803/3132444330","heartbeat_back_addr":"172.21.15.174:6807/3132444330","heartbeat_front_addr":"172.21.15.174:6805/3132444330","state":["exists","up"]},{"osd":5,"uuid":"a2eeb5b6-1b31-43cc-8306-a0adf3ee15bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6809","nonce":881817164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6811","nonce":881817164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6815","nonce":881817164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6813","nonce":881817164}]},"public_addr":"172.21.15.174:6809/881817164","cluster_addr":"172.21.15.174:6811/881817164","heartbeat_back_addr":"172.21.15.174:6815/881817164","heartbeat_front_addr":"172.21.15.174:6813/881817164","state":["exists","up"]},{"osd":6,"uuid":"588e873a-c841-4ba7-90f2-8956c87a9983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6825","nonce":117574251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6827","nonce":117574251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6831","nonce":117574251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6829","nonce":117574251}]},"public_addr":"172.21.15.174:6825/117574251","cluster_addr":"172.21.15.174:6827/117574251","heartbeat_back_addr":"172.21.15.174:6831/117574251","heartbeat_front_addr":"172.21.15.174:6829/117574251","state":["exists","up"]},{"osd":7,"uuid":"3b90f9b0-8a0e-4883-b088-ca1d4012870c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6817","nonce":2543899196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6819","nonce":2543899196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6823","nonce":2543899196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6821","nonce":2543899196}]},"public_addr":"172.21.15.174:6817/2543899196","cluster_addr":"172.21.15.174:6819/2543899196","heartbeat_back_addr":"172.21.15.174:6823/2543899196","heartbeat_front_addr":"172.21.15.174:6821/2543899196","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.293769+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.310034+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.404865+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.549658+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.192174+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.297266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.493649+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.452309+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:01:20.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.154+0000 7f50df422640 1 -- 172.21.15.116:0/72712025 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f50bc0678c0 msgr2=0x7f50bc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:20.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.154+0000 7f50df422640 1 --2- 172.21.15.116:0/72712025 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f50bc0678c0 0x7f50bc069d80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f50d40017f0 tx=0x7f50d4000f00 comp rx=0 tx=0).stop 2024-04-25T00:01:20.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.154+0000 7f50df422640 1 -- 172.21.15.116:0/72712025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50d8071de0 msgr2=0x7f50d8072260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:20.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.154+0000 7f50df422640 1 --2- 172.21.15.116:0/72712025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50d8071de0 0x7f50d8072260 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f50d80c4aa0 tx=0x7f50d000b480 comp rx=0 tx=0).stop 2024-04-25T00:01:20.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.154+0000 7f50df422640 1 -- 172.21.15.116:0/72712025 shutdown_connections 2024-04-25T00:01:20.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.154+0000 7f50df422640 1 --2- 172.21.15.116:0/72712025 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f50bc0678c0 0x7f50bc069d80 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:20.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.154+0000 7f50df422640 1 --2- 172.21.15.116:0/72712025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50d8071de0 0x7f50d8072260 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:20.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.154+0000 7f50df422640 1 --2- 172.21.15.116:0/72712025 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50d8072a00 0x7f50d806d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:20.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.154+0000 7f50df422640 1 --2- 172.21.15.116:0/72712025 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50c4004810 0x7f50d814dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:20.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.154+0000 7f50df422640 1 -- 172.21.15.116:0/72712025 >> 172.21.15.116:0/72712025 conn(0x7f50d80c3b00 msgr2=0x7f50d814fbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:20.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.154+0000 7f50df422640 1 -- 172.21.15.116:0/72712025 shutdown_connections 2024-04-25T00:01:20.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.154+0000 7f50df422640 1 -- 172.21.15.116:0/72712025 wait complete. 2024-04-25T00:01:20.172 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set b standby_count_wanted 0 2024-04-25T00:01:20.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.290+0000 7f7cc3e56640 1 Processor -- start 2024-04-25T00:01:20.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.290+0000 7f7cc3e56640 1 -- start start 2024-04-25T00:01:20.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.294+0000 7f7cc3e56640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7cbc06d510 0x7f7cbc072500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:20.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.294+0000 7f7cc3e56640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7cbc06d000 0x7f7cbc072a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:20.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.294+0000 7f7cc3e56640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7cbc149300 con 0x7f7cbc148f20 2024-04-25T00:01:20.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.294+0000 7f7cc3e56640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f7cbc153350 con 0x7f7cbc06d000 2024-04-25T00:01:20.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.294+0000 7f7cc3e56640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f7cbc0a95d0 con 0x7f7cbc06d510 2024-04-25T00:01:20.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.294+0000 7f7cc23f3640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7cbc06d000 0x7f7cbc072a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:20.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.294+0000 7f7cc23f3640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7cbc06d000 0x7f7cbc072a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54434/0 (socket says 172.21.15.116:54434) 2024-04-25T00:01:20.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.294+0000 7f7cc23f3640 1 -- 172.21.15.116:0/3976687635 learned_addr learned my addr 172.21.15.116:0/3976687635 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:20.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.294+0000 7f7cc1bf2640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f7cbc148f20 0x7f7cbc0c2930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:60906/0 (socket says 172.21.15.116:60906) 2024-04-25T00:01:20.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.294+0000 7f7cc13f1640 1 --2- 172.21.15.116:0/3976687635 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7cbc06d510 0x7f7cbc072500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:20.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.294+0000 7f7cc23f3640 1 -- 172.21.15.116:0/3976687635 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7cbc06d510 msgr2=0x7f7cbc072500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:20.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.294+0000 7f7cc23f3640 1 --2- 172.21.15.116:0/3976687635 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7cbc06d510 0x7f7cbc072500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:20.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.294+0000 7f7cc23f3640 1 -- 172.21.15.116:0/3976687635 >> v1:172.21.15.116:6789/0 conn(0x7f7cbc148f20 legacy=0x7f7cbc0c2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:20.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.294+0000 7f7cc23f3640 1 -- 172.21.15.116:0/3976687635 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cbc153070 con 0x7f7cbc06d000 2024-04-25T00:01:20.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.294+0000 7f7cc0bf0640 1 -- 172.21.15.116:0/3976687635 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4129806785 0 0) 0x7f7cb8001120 con 0x7f7cbc148f20 2024-04-25T00:01:20.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.294+0000 7f7cc23f3640 1 --2- 172.21.15.116:0/3976687635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7cbc06d000 0x7f7cbc072a40 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7cb4009e50 tx=0x7f7cb401c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9482b11a4e6a250a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:20.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.294+0000 7f7cc0bf0640 1 -- 172.21.15.116:0/3976687635 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cb40a55d0 con 0x7f7cbc06d000 2024-04-25T00:01:20.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.294+0000 7f7cc0bf0640 1 -- 172.21.15.116:0/3976687635 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cb40a5770 con 0x7f7cbc06d000 2024-04-25T00:01:20.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.294+0000 7f7cc0bf0640 1 -- 172.21.15.116:0/3976687635 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cb40a5b40 con 0x7f7cbc06d000 2024-04-25T00:01:20.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.298+0000 7f7cc3e56640 1 -- 172.21.15.116:0/3976687635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7cbc06d000 msgr2=0x7f7cbc072a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:20.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.298+0000 7f7cc3e56640 1 --2- 172.21.15.116:0/3976687635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7cbc06d000 0x7f7cbc072a40 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7cb4009e50 tx=0x7f7cb401c910 comp rx=0 tx=0).stop 2024-04-25T00:01:20.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.298+0000 7f7cc3e56640 1 -- 172.21.15.116:0/3976687635 shutdown_connections 2024-04-25T00:01:20.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.298+0000 7f7cc3e56640 1 --2- 172.21.15.116:0/3976687635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7cbc06d000 0x7f7cbc072a40 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:20.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.298+0000 7f7cc3e56640 1 --2- 172.21.15.116:0/3976687635 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7cbc06d510 0x7f7cbc072500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:20.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.298+0000 7f7cc3e56640 1 -- 172.21.15.116:0/3976687635 >> 172.21.15.116:0/3976687635 conn(0x7f7cbc0c3a20 msgr2=0x7f7cbc0c3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:20.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.298+0000 7f7cc3e56640 1 -- 172.21.15.116:0/3976687635 shutdown_connections 2024-04-25T00:01:20.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.298+0000 7f7cc3e56640 1 -- 172.21.15.116:0/3976687635 wait complete. 2024-04-25T00:01:20.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.298+0000 7f7cc3e56640 1 Processor -- start 2024-04-25T00:01:20.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.298+0000 7f7cc3e56640 1 -- start start 2024-04-25T00:01:20.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.298+0000 7f7cc3e56640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7cbc06d000 0x7f7cbc154200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:20.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.298+0000 7f7cc3e56640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7cbc06d510 0x7f7cbc0ac0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:20.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.298+0000 7f7cc3e56640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7cbc148f20 0x7f7cbc14e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:20.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.298+0000 7f7cc3e56640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7cbc0c3850 con 0x7f7cbc148f20 2024-04-25T00:01:20.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.298+0000 7f7cc3e56640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7cbc0c36d0 con 0x7f7cbc06d510 2024-04-25T00:01:20.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.298+0000 7f7cc3e56640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f7cbc15d5e0 con 0x7f7cbc06d000 2024-04-25T00:01:20.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.298+0000 7f7cc23f3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7cbc148f20 0x7f7cbc14e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:20.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.298+0000 7f7cc23f3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7cbc148f20 0x7f7cbc14e290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54444/0 (socket says 172.21.15.116:54444) 2024-04-25T00:01:20.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.298+0000 7f7cc23f3640 1 -- 172.21.15.116:0/890567312 learned_addr learned my addr 172.21.15.116:0/890567312 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:20.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.298+0000 7f7cc13f1640 1 --2- 172.21.15.116:0/890567312 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7cbc06d510 0x7f7cbc0ac0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:20.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.298+0000 7f7cc1bf2640 1 --2- 172.21.15.116:0/890567312 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7cbc06d000 0x7f7cbc154200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:20.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.298+0000 7f7cc23f3640 1 -- 172.21.15.116:0/890567312 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7cbc06d000 msgr2=0x7f7cbc154200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:20.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.298+0000 7f7cc23f3640 1 --2- 172.21.15.116:0/890567312 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7cbc06d000 0x7f7cbc154200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:20.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.298+0000 7f7cc23f3640 1 -- 172.21.15.116:0/890567312 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7cbc06d510 msgr2=0x7f7cbc0ac0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:20.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.298+0000 7f7cc23f3640 1 --2- 172.21.15.116:0/890567312 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7cbc06d510 0x7f7cbc0ac0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:20.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.298+0000 7f7cc23f3640 1 -- 172.21.15.116:0/890567312 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cbc156410 con 0x7f7cbc148f20 2024-04-25T00:01:20.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.298+0000 7f7cc1bf2640 1 --2- 172.21.15.116:0/890567312 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7cbc06d000 0x7f7cbc154200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:20.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.298+0000 7f7cc23f3640 1 --2- 172.21.15.116:0/890567312 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7cbc148f20 0x7f7cbc14e290 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7cb4009e50 tx=0x7f7cb40196b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:20.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.298+0000 7f7cb27fc640 1 -- 172.21.15.116:0/890567312 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cb40bc020 con 0x7f7cbc148f20 2024-04-25T00:01:20.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.298+0000 7f7cb27fc640 1 -- 172.21.15.116:0/890567312 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cb40b8670 con 0x7f7cbc148f20 2024-04-25T00:01:20.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.298+0000 7f7cc3e56640 1 -- 172.21.15.116:0/890567312 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cbc1566f0 con 0x7f7cbc148f20 2024-04-25T00:01:20.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.298+0000 7f7cb27fc640 1 -- 172.21.15.116:0/890567312 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cb4009510 con 0x7f7cbc148f20 2024-04-25T00:01:20.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.298+0000 7f7cc3e56640 1 -- 172.21.15.116:0/890567312 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cbc0ac5f0 con 0x7f7cbc148f20 2024-04-25T00:01:20.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.302+0000 7f7cc3e56640 1 -- 172.21.15.116:0/890567312 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cbc14efd0 con 0x7f7cbc148f20 2024-04-25T00:01:20.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.302+0000 7f7cb27fc640 1 -- 172.21.15.116:0/890567312 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7cb40bf020 con 0x7f7cbc148f20 2024-04-25T00:01:20.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.302+0000 7f7cb27fc640 1 --2- 172.21.15.116:0/890567312 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7ca00678f0 0x7f7ca0069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:20.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.302+0000 7f7cb27fc640 1 -- 172.21.15.116:0/890567312 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6521+0+0 (secure 0 0 0) 0x7f7cb41211d0 con 0x7f7cbc148f20 2024-04-25T00:01:20.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.302+0000 7f7cc1bf2640 1 --2- 172.21.15.116:0/890567312 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7ca00678f0 0x7f7ca0069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:20.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.302+0000 7f7cc1bf2640 1 --2- 172.21.15.116:0/890567312 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7ca00678f0 0x7f7ca0069db0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7cb8007fd0 tx=0x7f7cb8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:20.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.306+0000 7f7cb27fc640 1 -- 172.21.15.116:0/890567312 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f7cbc14efd0 con 0x7f7cbc148f20 2024-04-25T00:01:20.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:20.502+0000 7f7cc3e56640 1 -- 172.21.15.116:0/890567312 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "b", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f7cbc04f510 con 0x7f7cbc148f20 2024-04-25T00:01:21.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.514+0000 7f7cb27fc640 1 -- 172.21.15.116:0/890567312 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "b", "var": "standby_count_wanted", "val": "0"}]=0 v12) ==== 113+0+0 (secure 0 0 0) 0x7f7cb40c7020 con 0x7f7cbc148f20 2024-04-25T00:01:21.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.518+0000 7f7cc3e56640 1 -- 172.21.15.116:0/890567312 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7ca00678f0 msgr2=0x7f7ca0069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:21.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.518+0000 7f7cc3e56640 1 --2- 172.21.15.116:0/890567312 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7ca00678f0 0x7f7ca0069db0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7cb8007fd0 tx=0x7f7cb8009040 comp rx=0 tx=0).stop 2024-04-25T00:01:21.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.518+0000 7f7cc3e56640 1 -- 172.21.15.116:0/890567312 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7cbc148f20 msgr2=0x7f7cbc14e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:21.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.518+0000 7f7cc3e56640 1 --2- 172.21.15.116:0/890567312 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7cbc148f20 0x7f7cbc14e290 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7cb4009e50 tx=0x7f7cb40196b0 comp rx=0 tx=0).stop 2024-04-25T00:01:21.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.518+0000 7f7cc3e56640 1 -- 172.21.15.116:0/890567312 shutdown_connections 2024-04-25T00:01:21.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.518+0000 7f7cc3e56640 1 --2- 172.21.15.116:0/890567312 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7ca00678f0 0x7f7ca0069db0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:21.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.518+0000 7f7cc3e56640 1 --2- 172.21.15.116:0/890567312 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7cbc148f20 0x7f7cbc14e290 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:21.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.518+0000 7f7cc3e56640 1 --2- 172.21.15.116:0/890567312 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7cbc06d510 0x7f7cbc0ac0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:21.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.518+0000 7f7cc3e56640 1 --2- 172.21.15.116:0/890567312 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7cbc06d000 0x7f7cbc154200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:21.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.518+0000 7f7cc3e56640 1 -- 172.21.15.116:0/890567312 >> 172.21.15.116:0/890567312 conn(0x7f7cbc0c3a20 msgr2=0x7f7cbc0c28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:21.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.518+0000 7f7cc3e56640 1 -- 172.21.15.116:0/890567312 shutdown_connections 2024-04-25T00:01:21.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.518+0000 7f7cc3e56640 1 -- 172.21.15.116:0/890567312 wait complete. 2024-04-25T00:01:21.538 DEBUG:tasks.cephfs.filesystem:fs_config: {'session_timeout': 300} 2024-04-25T00:01:21.538 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set b allow_standby_replay false 2024-04-25T00:01:21.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.662+0000 7f30e80a6640 1 Processor -- start 2024-04-25T00:01:21.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.662+0000 7f30e80a6640 1 -- start start 2024-04-25T00:01:21.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.666+0000 7f30e80a6640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f30e0150d60 0x7f30e00c2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:21.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.666+0000 7f30e80a6640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30e00a9cb0 con 0x7f30e00a9630 2024-04-25T00:01:21.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.666+0000 7f30e80a6640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30e00a9f90 con 0x7f30e00c2e70 2024-04-25T00:01:21.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.666+0000 7f30e80a6640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f30e00aa270 con 0x7f30e0150d60 2024-04-25T00:01:21.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.666+0000 7f30e5641640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f30e00a9630 0x7f30e0156e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:60910/0 (socket says 172.21.15.116:60910) 2024-04-25T00:01:21.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.666+0000 7f30e5641640 1 -- 172.21.15.116:0/3669923928 learned_addr learned my addr 172.21.15.116:0/3669923928 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:21.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.666+0000 7f30e5e42640 1 --2- 172.21.15.116:0/3669923928 >> v2:172.21.15.174:3300/0 conn(0x7f30e0150d60 0x7f30e00c2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:21.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.666+0000 7f30e4e40640 1 -- 172.21.15.116:0/3669923928 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3864929745 0 0) 0x7f30e00a9cb0 con 0x7f30e00a9630 2024-04-25T00:01:21.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.666+0000 7f30e4e40640 1 -- 172.21.15.116:0/3669923928 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f30cc003530 con 0x7f30e00a9630 2024-04-25T00:01:21.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.666+0000 7f30e4e40640 1 -- 172.21.15.116:0/3669923928 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4970374 0 0) 0x7f30cc003530 con 0x7f30e00a9630 2024-04-25T00:01:21.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.666+0000 7f30e4e40640 1 -- 172.21.15.116:0/3669923928 >> v2:172.21.15.174:3300/0 conn(0x7f30e0150d60 msgr2=0x7f30e00c2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:21.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.666+0000 7f30e4e40640 1 --2- 172.21.15.116:0/3669923928 >> v2:172.21.15.174:3300/0 conn(0x7f30e0150d60 0x7f30e00c2930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:21.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.666+0000 7f30e4e40640 1 -- 172.21.15.116:0/3669923928 >> v1:172.21.15.174:6789/0 conn(0x7f30e00c2e70 legacy=0x7f30e0157510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:21.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.666+0000 7f30e4e40640 1 -- 172.21.15.116:0/3669923928 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30e00aa550 con 0x7f30e00a9630 2024-04-25T00:01:21.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.666+0000 7f30e4e40640 1 -- 172.21.15.116:0/3669923928 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f30d0001680 con 0x7f30e00a9630 2024-04-25T00:01:21.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.666+0000 7f30e4e40640 1 -- 172.21.15.116:0/3669923928 >> v1:172.21.15.116:6789/0 conn(0x7f30e00a9630 legacy=0x7f30e0156e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:21.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.666+0000 7f30e4e40640 1 --2- 172.21.15.116:0/3669923928 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30cc0039c0 0x7f30cc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:21.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.666+0000 7f30e4e40640 1 --2- 172.21.15.116:0/3669923928 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30cc004810 0x7f30cc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:21.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.666+0000 7f30e4e40640 1 --2- 172.21.15.116:0/3669923928 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f30cc0052f0 0x7f30cc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:21.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.666+0000 7f30e4e40640 1 -- 172.21.15.116:0/3669923928 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f30e00aa270 con 0x7f30cc004810 2024-04-25T00:01:21.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.666+0000 7f30e4e40640 1 -- 172.21.15.116:0/3669923928 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f30cc00c1c0 con 0x7f30cc0039c0 2024-04-25T00:01:21.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.666+0000 7f30e4e40640 1 -- 172.21.15.116:0/3669923928 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f30cc00c3c0 con 0x7f30cc0052f0 2024-04-25T00:01:21.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.666+0000 7f30e5641640 1 --2- 172.21.15.116:0/3669923928 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30cc004810 0x7f30cc004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:21.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.666+0000 7f30e5e42640 1 --2- 172.21.15.116:0/3669923928 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30cc0039c0 0x7f30cc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:21.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.666+0000 7f30e5641640 1 -- 172.21.15.116:0/3669923928 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f30cc0052f0 msgr2=0x7f30cc00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:21.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.666+0000 7f30e5641640 1 --2- 172.21.15.116:0/3669923928 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f30cc0052f0 0x7f30cc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:21.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.666+0000 7f30e5641640 1 -- 172.21.15.116:0/3669923928 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30cc0039c0 msgr2=0x7f30cc003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:21.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.666+0000 7f30e5641640 1 --2- 172.21.15.116:0/3669923928 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30cc0039c0 0x7f30cc003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:21.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.666+0000 7f30e5641640 1 -- 172.21.15.116:0/3669923928 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30cc003530 con 0x7f30cc004810 2024-04-25T00:01:21.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.666+0000 7f30e5e42640 1 --2- 172.21.15.116:0/3669923928 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30cc0039c0 0x7f30cc003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:21.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.666+0000 7f30e6643640 1 -- 172.21.15.116:0/3669923928 reap_dead start 2024-04-25T00:01:21.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.670+0000 7f30e5641640 1 --2- 172.21.15.116:0/3669923928 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30cc004810 0x7f30cc004c90 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f30d000c190 tx=0x7f30d001cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96b63ebed2cc4659 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:21.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.670+0000 7f30e4e40640 1 -- 172.21.15.116:0/3669923928 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d0007d00 con 0x7f30cc004810 2024-04-25T00:01:21.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.670+0000 7f30e4e40640 1 -- 172.21.15.116:0/3669923928 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30d00a9020 con 0x7f30cc004810 2024-04-25T00:01:21.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.670+0000 7f30e4e40640 1 -- 172.21.15.116:0/3669923928 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d0005ae0 con 0x7f30cc004810 2024-04-25T00:01:21.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.670+0000 7f30e80a6640 1 -- 172.21.15.116:0/3669923928 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30cc004810 msgr2=0x7f30cc004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:21.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.670+0000 7f30e80a6640 1 --2- 172.21.15.116:0/3669923928 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30cc004810 0x7f30cc004c90 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f30d000c190 tx=0x7f30d001cbf0 comp rx=0 tx=0).stop 2024-04-25T00:01:21.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.670+0000 7f30e80a6640 1 -- 172.21.15.116:0/3669923928 shutdown_connections 2024-04-25T00:01:21.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.670+0000 7f30e80a6640 1 --2- 172.21.15.116:0/3669923928 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30cc004810 0x7f30cc004c90 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:21.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.670+0000 7f30e80a6640 1 -- 172.21.15.116:0/3669923928 >> 172.21.15.116:0/3669923928 conn(0x7f30e00c3a20 msgr2=0x7f30e00c3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:21.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.670+0000 7f30e80a6640 1 -- 172.21.15.116:0/3669923928 shutdown_connections 2024-04-25T00:01:21.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.670+0000 7f30e80a6640 1 -- 172.21.15.116:0/3669923928 wait complete. 2024-04-25T00:01:21.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.670+0000 7f30e80a6640 1 Processor -- start 2024-04-25T00:01:21.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.670+0000 7f30e80a6640 1 -- start start 2024-04-25T00:01:21.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.670+0000 7f30e80a6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30cc004810 0x7f30e0147d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:21.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.670+0000 7f30e80a6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30e01549c0 0x7f30e01477a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:21.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.670+0000 7f30e80a6640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f30e00c1820 0x7f30e00c1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:21.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.670+0000 7f30e80a6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f30e00ab7a0 con 0x7f30e01549c0 2024-04-25T00:01:21.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.670+0000 7f30e80a6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f30e00721e0 con 0x7f30cc004810 2024-04-25T00:01:21.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.670+0000 7f30e80a6640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f30e0157610 con 0x7f30e00c1820 2024-04-25T00:01:21.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.670+0000 7f30e5641640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30e01549c0 0x7f30e01477a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:21.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.670+0000 7f30e5641640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30e01549c0 0x7f30e01477a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54464/0 (socket says 172.21.15.116:54464) 2024-04-25T00:01:21.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.670+0000 7f30e5641640 1 -- 172.21.15.116:0/1378401173 learned_addr learned my addr 172.21.15.116:0/1378401173 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:21.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.670+0000 7f30e5e42640 1 --2- 172.21.15.116:0/1378401173 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30cc004810 0x7f30e0147d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:21.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.670+0000 7f30e6643640 1 --2- 172.21.15.116:0/1378401173 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f30e00c1820 0x7f30e00c1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:21.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.670+0000 7f30e5641640 1 -- 172.21.15.116:0/1378401173 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f30e00c1820 msgr2=0x7f30e00c1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:21.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.670+0000 7f30e5641640 1 --2- 172.21.15.116:0/1378401173 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f30e00c1820 0x7f30e00c1cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:21.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.670+0000 7f30e5641640 1 -- 172.21.15.116:0/1378401173 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30cc004810 msgr2=0x7f30e0147d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:21.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.670+0000 7f30e5641640 1 --2- 172.21.15.116:0/1378401173 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30cc004810 0x7f30e0147d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:21.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.670+0000 7f30e5641640 1 -- 172.21.15.116:0/1378401173 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30e006e9c0 con 0x7f30e01549c0 2024-04-25T00:01:21.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.670+0000 7f30e5641640 1 --2- 172.21.15.116:0/1378401173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30e01549c0 0x7f30e01477a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f30d00ab040 tx=0x7f30cc005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:21.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.670+0000 7f30d67fc640 1 -- 172.21.15.116:0/1378401173 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d0007d00 con 0x7f30e01549c0 2024-04-25T00:01:21.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.670+0000 7f30d67fc640 1 -- 172.21.15.116:0/1378401173 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30d0001480 con 0x7f30e01549c0 2024-04-25T00:01:21.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.670+0000 7f30d67fc640 1 -- 172.21.15.116:0/1378401173 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d00a59e0 con 0x7f30e01549c0 2024-04-25T00:01:21.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.670+0000 7f30e80a6640 1 -- 172.21.15.116:0/1378401173 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30e006d3b0 con 0x7f30e01549c0 2024-04-25T00:01:21.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.670+0000 7f30e80a6640 1 -- 172.21.15.116:0/1378401173 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30e006ec60 con 0x7f30e01549c0 2024-04-25T00:01:21.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.674+0000 7f30e80a6640 1 -- 172.21.15.116:0/1378401173 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30e004f510 con 0x7f30e01549c0 2024-04-25T00:01:21.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.674+0000 7f30d67fc640 1 -- 172.21.15.116:0/1378401173 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f30d00a5b80 con 0x7f30e01549c0 2024-04-25T00:01:21.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.674+0000 7f30d67fc640 1 --2- 172.21.15.116:0/1378401173 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f30c40678f0 0x7f30c4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:21.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.674+0000 7f30d67fc640 1 -- 172.21.15.116:0/1378401173 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7f30d011f670 con 0x7f30e01549c0 2024-04-25T00:01:21.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.674+0000 7f30e5e42640 1 --2- 172.21.15.116:0/1378401173 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f30c40678f0 0x7f30c4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:21.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.674+0000 7f30e5e42640 1 --2- 172.21.15.116:0/1378401173 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f30c40678f0 0x7f30c4069db0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f30e014fa80 tx=0x7f30dc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:21.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.678+0000 7f30d67fc640 1 -- 172.21.15.116:0/1378401173 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f30d0016930 con 0x7f30e01549c0 2024-04-25T00:01:21.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:21.874+0000 7f30e80a6640 1 -- 172.21.15.116:0/1378401173 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "b", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f30e0071db0 con 0x7f30e01549c0 2024-04-25T00:01:22.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.518+0000 7f30d67fc640 1 -- 172.21.15.116:0/1378401173 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "b", "var": "allow_standby_replay", "val": "false"}]=0 v13) ==== 117+0+0 (secure 0 0 0) 0x7f30d00eff10 con 0x7f30e01549c0 2024-04-25T00:01:22.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.522+0000 7f30e80a6640 1 -- 172.21.15.116:0/1378401173 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f30c40678f0 msgr2=0x7f30c4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:22.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.522+0000 7f30e80a6640 1 --2- 172.21.15.116:0/1378401173 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f30c40678f0 0x7f30c4069db0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f30e014fa80 tx=0x7f30dc008040 comp rx=0 tx=0).stop 2024-04-25T00:01:22.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.522+0000 7f30e80a6640 1 -- 172.21.15.116:0/1378401173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30e01549c0 msgr2=0x7f30e01477a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:22.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.522+0000 7f30e80a6640 1 --2- 172.21.15.116:0/1378401173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30e01549c0 0x7f30e01477a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f30d00ab040 tx=0x7f30cc005840 comp rx=0 tx=0).stop 2024-04-25T00:01:22.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.522+0000 7f30e80a6640 1 -- 172.21.15.116:0/1378401173 shutdown_connections 2024-04-25T00:01:22.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.522+0000 7f30e80a6640 1 --2- 172.21.15.116:0/1378401173 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f30c40678f0 0x7f30c4069db0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:22.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.522+0000 7f30e80a6640 1 --2- 172.21.15.116:0/1378401173 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f30e00c1820 0x7f30e00c1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:22.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.522+0000 7f30e80a6640 1 --2- 172.21.15.116:0/1378401173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30e01549c0 0x7f30e01477a0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:22.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.522+0000 7f30e80a6640 1 --2- 172.21.15.116:0/1378401173 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30cc004810 0x7f30e0147d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:22.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.522+0000 7f30e80a6640 1 -- 172.21.15.116:0/1378401173 >> 172.21.15.116:0/1378401173 conn(0x7f30e00c3a20 msgr2=0x7f30e0149b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:22.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.522+0000 7f30e80a6640 1 -- 172.21.15.116:0/1378401173 shutdown_connections 2024-04-25T00:01:22.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.522+0000 7f30e80a6640 1 -- 172.21.15.116:0/1378401173 wait complete. 2024-04-25T00:01:22.546 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set b session_timeout 300 2024-04-25T00:01:22.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.674+0000 7f75b3679640 1 Processor -- start 2024-04-25T00:01:22.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.674+0000 7f75b3679640 1 -- start start 2024-04-25T00:01:22.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.674+0000 7f75b3679640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f75ac152da0 0x7f75ac06da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:22.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.674+0000 7f75b3679640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f75ac06dfc0 0x7f75ac06b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:22.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.674+0000 7f75b3679640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75ac1566c0 con 0x7f75ac0adcf0 2024-04-25T00:01:22.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.674+0000 7f75b3679640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f75ac0c31f0 con 0x7f75ac06dfc0 2024-04-25T00:01:22.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.674+0000 7f75b3679640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f75ac0c28c0 con 0x7f75ac152da0 2024-04-25T00:01:22.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.674+0000 7f75b0c14640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f75ac06dfc0 0x7f75ac06b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:22.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.674+0000 7f75b1c16640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f75ac0adcf0 0x7f75ac158e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:60922/0 (socket says 172.21.15.116:60922) 2024-04-25T00:01:22.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.674+0000 7f75b1c16640 1 -- 172.21.15.116:0/477143005 learned_addr learned my addr 172.21.15.116:0/477143005 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:22.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.674+0000 7f75b1415640 1 --2- 172.21.15.116:0/477143005 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f75ac152da0 0x7f75ac06da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:22.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.674+0000 7f75b0c14640 1 -- 172.21.15.116:0/477143005 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f75ac152da0 msgr2=0x7f75ac06da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:22.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.674+0000 7f75b0c14640 1 --2- 172.21.15.116:0/477143005 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f75ac152da0 0x7f75ac06da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:22.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.674+0000 7f75b0c14640 1 -- 172.21.15.116:0/477143005 >> v1:172.21.15.116:6789/0 conn(0x7f75ac0adcf0 legacy=0x7f75ac158e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:22.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.674+0000 7f75b0c14640 1 -- 172.21.15.116:0/477143005 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75ac0c2ba0 con 0x7f75ac06dfc0 2024-04-25T00:01:22.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.674+0000 7f75abfff640 1 -- 172.21.15.116:0/477143005 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3254283870 0 0) 0x7f75ac1566c0 con 0x7f75ac0adcf0 2024-04-25T00:01:22.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.674+0000 7f75b1415640 1 --2- 172.21.15.116:0/477143005 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f75ac152da0 0x7f75ac06da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:22.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.678+0000 7f75b0c14640 1 --2- 172.21.15.116:0/477143005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75ac06dfc0 0x7f75ac06b210 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f75a00097c0 tx=0x7f75a001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba3e6acbc6b40f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:22.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.678+0000 7f75abfff640 1 -- 172.21.15.116:0/477143005 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a00b0070 con 0x7f75ac06dfc0 2024-04-25T00:01:22.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.678+0000 7f75abfff640 1 -- 172.21.15.116:0/477143005 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75a00ab600 con 0x7f75ac06dfc0 2024-04-25T00:01:22.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.678+0000 7f75abfff640 1 -- 172.21.15.116:0/477143005 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a00ab9d0 con 0x7f75ac06dfc0 2024-04-25T00:01:22.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.678+0000 7f75b3679640 1 -- 172.21.15.116:0/477143005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75ac06dfc0 msgr2=0x7f75ac06b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:22.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.678+0000 7f75b3679640 1 --2- 172.21.15.116:0/477143005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75ac06dfc0 0x7f75ac06b210 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f75a00097c0 tx=0x7f75a001c4a0 comp rx=0 tx=0).stop 2024-04-25T00:01:22.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.678+0000 7f75b3679640 1 -- 172.21.15.116:0/477143005 shutdown_connections 2024-04-25T00:01:22.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.678+0000 7f75b3679640 1 --2- 172.21.15.116:0/477143005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75ac06dfc0 0x7f75ac06b210 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:22.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.678+0000 7f75b3679640 1 --2- 172.21.15.116:0/477143005 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f75ac152da0 0x7f75ac06da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:22.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.678+0000 7f75b3679640 1 -- 172.21.15.116:0/477143005 >> 172.21.15.116:0/477143005 conn(0x7f75ac0c49c0 msgr2=0x7f75ac0c4dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:22.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.678+0000 7f75b3679640 1 -- 172.21.15.116:0/477143005 shutdown_connections 2024-04-25T00:01:22.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.678+0000 7f75b3679640 1 -- 172.21.15.116:0/477143005 wait complete. 2024-04-25T00:01:22.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.678+0000 7f75b3679640 1 Processor -- start 2024-04-25T00:01:22.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.678+0000 7f75b3679640 1 -- start start 2024-04-25T00:01:22.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.678+0000 7f75b3679640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f75ac06dfc0 0x7f75ac1589e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:22.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.678+0000 7f75b3679640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75ac0adcf0 0x7f75ac145fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:22.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.678+0000 7f75b3679640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f75ac152da0 0x7f75ac1f2810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:22.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.678+0000 7f75b3679640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f75ac0c37e0 con 0x7f75ac0adcf0 2024-04-25T00:01:22.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.678+0000 7f75b3679640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f75ac0c3660 con 0x7f75ac06dfc0 2024-04-25T00:01:22.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.678+0000 7f75b3679640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f75ac0c3960 con 0x7f75ac152da0 2024-04-25T00:01:22.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.678+0000 7f75b0c14640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75ac0adcf0 0x7f75ac145fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:22.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.678+0000 7f75b0c14640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75ac0adcf0 0x7f75ac145fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54478/0 (socket says 172.21.15.116:54478) 2024-04-25T00:01:22.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.678+0000 7f75b0c14640 1 -- 172.21.15.116:0/3681802326 learned_addr learned my addr 172.21.15.116:0/3681802326 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:22.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.678+0000 7f75b0c14640 1 -- 172.21.15.116:0/3681802326 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f75ac152da0 msgr2=0x7f75ac1f2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:22.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.678+0000 7f75b0c14640 1 --2- 172.21.15.116:0/3681802326 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f75ac152da0 0x7f75ac1f2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:22.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.678+0000 7f75b0c14640 1 -- 172.21.15.116:0/3681802326 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f75ac06dfc0 msgr2=0x7f75ac1589e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:22.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.678+0000 7f75b0c14640 1 --2- 172.21.15.116:0/3681802326 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f75ac06dfc0 0x7f75ac1589e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:22.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.678+0000 7f75b0c14640 1 -- 172.21.15.116:0/3681802326 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75ac146630 con 0x7f75ac0adcf0 2024-04-25T00:01:22.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.682+0000 7f75b0c14640 1 --2- 172.21.15.116:0/3681802326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75ac0adcf0 0x7f75ac145fc0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f75a001a0d0 tx=0x7f75a001ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:22.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.682+0000 7f75a9ffb640 1 -- 172.21.15.116:0/3681802326 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a00b0050 con 0x7f75ac0adcf0 2024-04-25T00:01:22.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.682+0000 7f75a9ffb640 1 -- 172.21.15.116:0/3681802326 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75a00bc070 con 0x7f75ac0adcf0 2024-04-25T00:01:22.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.682+0000 7f75b3679640 1 -- 172.21.15.116:0/3681802326 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75ac146910 con 0x7f75ac0adcf0 2024-04-25T00:01:22.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.682+0000 7f75a9ffb640 1 -- 172.21.15.116:0/3681802326 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a00b5830 con 0x7f75ac0adcf0 2024-04-25T00:01:22.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.682+0000 7f75b3679640 1 -- 172.21.15.116:0/3681802326 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75ac1f2010 con 0x7f75ac0adcf0 2024-04-25T00:01:22.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.682+0000 7f75b3679640 1 -- 172.21.15.116:0/3681802326 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75ac1f2340 con 0x7f75ac0adcf0 2024-04-25T00:01:22.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.682+0000 7f75a9ffb640 1 -- 172.21.15.116:0/3681802326 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f75a00b59d0 con 0x7f75ac0adcf0 2024-04-25T00:01:22.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.682+0000 7f75a9ffb640 1 --2- 172.21.15.116:0/3681802326 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f75980678c0 0x7f7598069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:22.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.682+0000 7f75a9ffb640 1 -- 172.21.15.116:0/3681802326 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7f75a0120890 con 0x7f75ac0adcf0 2024-04-25T00:01:22.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.682+0000 7f75b1415640 1 --2- 172.21.15.116:0/3681802326 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f75980678c0 0x7f7598069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:22.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.682+0000 7f75b1415640 1 --2- 172.21.15.116:0/3681802326 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f75980678c0 0x7f7598069d80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f75ac068e90 tx=0x7f759c008330 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:22.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.686+0000 7f75a9ffb640 1 -- 172.21.15.116:0/3681802326 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f75a0016090 con 0x7f75ac0adcf0 2024-04-25T00:01:22.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:22.886+0000 7f75b3679640 1 -- 172.21.15.116:0/3681802326 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "b", "var": "session_timeout", "val": "300"} v 0) -- 0x7f75ac04f510 con 0x7f75ac0adcf0 2024-04-25T00:01:23.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.526+0000 7f75a9ffb640 1 -- 172.21.15.116:0/3681802326 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "b", "var": "session_timeout", "val": "300"}]=0 v14) ==== 110+0+0 (secure 0 0 0) 0x7f75a00f1250 con 0x7f75ac0adcf0 2024-04-25T00:01:23.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.530+0000 7f75b3679640 1 -- 172.21.15.116:0/3681802326 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f75980678c0 msgr2=0x7f7598069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:23.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.530+0000 7f75b3679640 1 --2- 172.21.15.116:0/3681802326 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f75980678c0 0x7f7598069d80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f75ac068e90 tx=0x7f759c008330 comp rx=0 tx=0).stop 2024-04-25T00:01:23.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.530+0000 7f75b3679640 1 -- 172.21.15.116:0/3681802326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75ac0adcf0 msgr2=0x7f75ac145fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:23.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.530+0000 7f75b3679640 1 --2- 172.21.15.116:0/3681802326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75ac0adcf0 0x7f75ac145fc0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f75a001a0d0 tx=0x7f75a001ace0 comp rx=0 tx=0).stop 2024-04-25T00:01:23.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.530+0000 7f75b3679640 1 -- 172.21.15.116:0/3681802326 shutdown_connections 2024-04-25T00:01:23.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.530+0000 7f75b3679640 1 --2- 172.21.15.116:0/3681802326 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f75980678c0 0x7f7598069d80 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:23.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.530+0000 7f75b3679640 1 --2- 172.21.15.116:0/3681802326 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f75ac152da0 0x7f75ac1f2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:23.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.530+0000 7f75b3679640 1 --2- 172.21.15.116:0/3681802326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75ac0adcf0 0x7f75ac145fc0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:23.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.530+0000 7f75b3679640 1 --2- 172.21.15.116:0/3681802326 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f75ac06dfc0 0x7f75ac1589e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:23.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.530+0000 7f75b3679640 1 -- 172.21.15.116:0/3681802326 >> 172.21.15.116:0/3681802326 conn(0x7f75ac0c49c0 msgr2=0x7f75ac153cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:23.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.530+0000 7f75b3679640 1 -- 172.21.15.116:0/3681802326 shutdown_connections 2024-04-25T00:01:23.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.530+0000 7f75b3679640 1 -- 172.21.15.116:0/3681802326 wait complete. 2024-04-25T00:01:23.555 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'b' 2024-04-25T00:01:23.555 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:01:23.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.678+0000 7f1721dcc640 1 Processor -- start 2024-04-25T00:01:23.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.678+0000 7f1721dcc640 1 -- start start 2024-04-25T00:01:23.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.682+0000 7f1721dcc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f171c0a99f0 0x7f171c153ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:23.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.682+0000 7f1721dcc640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f171c0c3910 con 0x7f171c150d20 2024-04-25T00:01:23.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.682+0000 7f1721dcc640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f171c154530 con 0x7f171c0a94a0 2024-04-25T00:01:23.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.682+0000 7f1721dcc640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f171c0c16e0 con 0x7f171c0a99f0 2024-04-25T00:01:23.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.682+0000 7f171bfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f171c0a99f0 0x7f171c153ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:23.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.682+0000 7f171b7fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f171c150d20 0x7f171c0c2a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:60936/0 (socket says 172.21.15.116:60936) 2024-04-25T00:01:23.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.682+0000 7f171b7fe640 1 -- 172.21.15.116:0/1198934037 learned_addr learned my addr 172.21.15.116:0/1198934037 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:23.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.682+0000 7f171bfff640 1 -- 172.21.15.116:0/1198934037 >> v1:172.21.15.174:6789/0 conn(0x7f171c0a94a0 legacy=0x7f171c1538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:23.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.682+0000 7f171bfff640 1 -- 172.21.15.116:0/1198934037 >> v1:172.21.15.116:6789/0 conn(0x7f171c150d20 legacy=0x7f171c0c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:23.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.682+0000 7f171bfff640 1 -- 172.21.15.116:0/1198934037 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f171c0c1400 con 0x7f171c0a99f0 2024-04-25T00:01:23.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.682+0000 7f171affd640 1 -- 172.21.15.116:0/1198934037 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 303978505 0 0) 0x7f171c0c3910 con 0x7f171c150d20 2024-04-25T00:01:23.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.682+0000 7f171bfff640 1 --2- 172.21.15.116:0/1198934037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f171c0a99f0 0x7f171c153ff0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1714009d90 tx=0x7f171401c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3d3dfd946ce7b8e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:23.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.682+0000 7f171affd640 1 -- 172.21.15.116:0/1198934037 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17140b2070 con 0x7f171c0a99f0 2024-04-25T00:01:23.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.682+0000 7f171affd640 1 -- 172.21.15.116:0/1198934037 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17140a57f0 con 0x7f171c0a99f0 2024-04-25T00:01:23.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.682+0000 7f171affd640 1 -- 172.21.15.116:0/1198934037 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17140a5bc0 con 0x7f171c0a99f0 2024-04-25T00:01:23.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.682+0000 7f1721dcc640 1 -- 172.21.15.116:0/1198934037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f171c0a99f0 msgr2=0x7f171c153ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:23.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.682+0000 7f1721dcc640 1 --2- 172.21.15.116:0/1198934037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f171c0a99f0 0x7f171c153ff0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1714009d90 tx=0x7f171401c930 comp rx=0 tx=0).stop 2024-04-25T00:01:23.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.682+0000 7f1721dcc640 1 -- 172.21.15.116:0/1198934037 shutdown_connections 2024-04-25T00:01:23.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.682+0000 7f1721dcc640 1 --2- 172.21.15.116:0/1198934037 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f171c0a99f0 0x7f171c153ff0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:23.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.682+0000 7f1721dcc640 1 -- 172.21.15.116:0/1198934037 >> 172.21.15.116:0/1198934037 conn(0x7f171c0c3b00 msgr2=0x7f171c0c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:23.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.686+0000 7f1721dcc640 1 -- 172.21.15.116:0/1198934037 shutdown_connections 2024-04-25T00:01:23.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.686+0000 7f1721dcc640 1 -- 172.21.15.116:0/1198934037 wait complete. 2024-04-25T00:01:23.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.686+0000 7f1721dcc640 1 Processor -- start 2024-04-25T00:01:23.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.686+0000 7f1721dcc640 1 -- start start 2024-04-25T00:01:23.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.686+0000 7f1721dcc640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f171c0a94a0 0x7f171c13f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:23.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.686+0000 7f1721dcc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f171c0a99f0 0x7f171c13cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:23.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.686+0000 7f1721dcc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f171c150d20 0x7f171c14d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:23.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.686+0000 7f1721dcc640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f171c0c32a0 con 0x7f171c150d20 2024-04-25T00:01:23.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.686+0000 7f1721dcc640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f171c0c3120 con 0x7f171c0a99f0 2024-04-25T00:01:23.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.686+0000 7f1721dcc640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f171c161740 con 0x7f171c0a94a0 2024-04-25T00:01:23.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.686+0000 7f171bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f171c150d20 0x7f171c14d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:23.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.686+0000 7f171bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f171c150d20 0x7f171c14d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54498/0 (socket says 172.21.15.116:54498) 2024-04-25T00:01:23.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.686+0000 7f171bfff640 1 -- 172.21.15.116:0/1267240621 learned_addr learned my addr 172.21.15.116:0/1267240621 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:23.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.686+0000 7f171bfff640 1 -- 172.21.15.116:0/1267240621 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f171c0a94a0 msgr2=0x7f171c13f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:23.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.686+0000 7f171bfff640 1 --2- 172.21.15.116:0/1267240621 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f171c0a94a0 0x7f171c13f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:23.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.686+0000 7f171bfff640 1 -- 172.21.15.116:0/1267240621 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f171c0a99f0 msgr2=0x7f171c13cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:23.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.686+0000 7f171bfff640 1 --2- 172.21.15.116:0/1267240621 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f171c0a99f0 0x7f171c13cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:23.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.686+0000 7f171bfff640 1 -- 172.21.15.116:0/1267240621 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f171c142bc0 con 0x7f171c150d20 2024-04-25T00:01:23.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.686+0000 7f171bfff640 1 --2- 172.21.15.116:0/1267240621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f171c150d20 0x7f171c14d6a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f17140b0040 tx=0x7f17140196d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:23.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.686+0000 7f1718ff9640 1 -- 172.21.15.116:0/1267240621 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17140bd020 con 0x7f171c150d20 2024-04-25T00:01:23.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.686+0000 7f1718ff9640 1 -- 172.21.15.116:0/1267240621 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17140b8670 con 0x7f171c150d20 2024-04-25T00:01:23.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.686+0000 7f1721dcc640 1 -- 172.21.15.116:0/1267240621 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f171c14cbc0 con 0x7f171c150d20 2024-04-25T00:01:23.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.686+0000 7f1718ff9640 1 -- 172.21.15.116:0/1267240621 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17140b2040 con 0x7f171c150d20 2024-04-25T00:01:23.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.686+0000 7f1721dcc640 1 -- 172.21.15.116:0/1267240621 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f171c14d180 con 0x7f171c150d20 2024-04-25T00:01:23.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.686+0000 7f1721dcc640 1 -- 172.21.15.116:0/1267240621 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f171c14c9d0 con 0x7f171c150d20 2024-04-25T00:01:23.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.690+0000 7f1718ff9640 1 -- 172.21.15.116:0/1267240621 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f17140c0020 con 0x7f171c150d20 2024-04-25T00:01:23.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.690+0000 7f1718ff9640 1 --2- 172.21.15.116:0/1267240621 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f16fc0678c0 0x7f16fc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:23.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.690+0000 7f1718ff9640 1 -- 172.21.15.116:0/1267240621 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7f1714120fc0 con 0x7f171c150d20 2024-04-25T00:01:23.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.690+0000 7f171b7fe640 1 --2- 172.21.15.116:0/1267240621 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f16fc0678c0 0x7f16fc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:23.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.690+0000 7f171b7fe640 1 --2- 172.21.15.116:0/1267240621 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f16fc0678c0 0x7f16fc069d80 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f170c0015a0 tx=0x7f170c007970 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:23.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.694+0000 7f1718ff9640 1 -- 172.21.15.116:0/1267240621 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f17140c7030 con 0x7f171c150d20 2024-04-25T00:01:23.734 INFO:tasks.ceph.mds.a.smithi116.stderr:2024-04-25T00:01:23.734+0000 7f206cd4a640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T00:01:23.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.890+0000 7f1721dcc640 1 -- 172.21.15.116:0/1267240621 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f171c068350 con 0x7f171c150d20 2024-04-25T00:01:23.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.890+0000 7f1718ff9640 1 -- 172.21.15.116:0/1267240621 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6492 (secure 0 0 0) 0x7f1714016520 con 0x7f171c150d20 2024-04-25T00:01:23.893 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:01:23.893 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6834/1459177921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1459177921},{"type":"v1","addr":"172.21.15.116:6834","nonce":1459177921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:01:08.748957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6837/1762356055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1762356055},{"type":"v1","addr":"172.21.15.174:6837","nonce":1762356055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270]},"id":1},{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:01:22.888641+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6835/174781672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":174781672},{"type":"v1","addr":"172.21.15.116:6835","nonce":174781672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4245,"qdb_cluster":[4245]},"id":2}]} 2024-04-25T00:01:23.893 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 14 2024-04-25T00:01:23.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.894+0000 7f1721dcc640 1 -- 172.21.15.116:0/1267240621 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f16fc0678c0 msgr2=0x7f16fc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:23.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.894+0000 7f1721dcc640 1 --2- 172.21.15.116:0/1267240621 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f16fc0678c0 0x7f16fc069d80 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f170c0015a0 tx=0x7f170c007970 comp rx=0 tx=0).stop 2024-04-25T00:01:23.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.894+0000 7f1721dcc640 1 -- 172.21.15.116:0/1267240621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f171c150d20 msgr2=0x7f171c14d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:23.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.894+0000 7f1721dcc640 1 --2- 172.21.15.116:0/1267240621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f171c150d20 0x7f171c14d6a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f17140b0040 tx=0x7f17140196d0 comp rx=0 tx=0).stop 2024-04-25T00:01:23.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.894+0000 7f1721dcc640 1 -- 172.21.15.116:0/1267240621 shutdown_connections 2024-04-25T00:01:23.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.894+0000 7f1721dcc640 1 --2- 172.21.15.116:0/1267240621 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f16fc0678c0 0x7f16fc069d80 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:23.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.894+0000 7f1721dcc640 1 --2- 172.21.15.116:0/1267240621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f171c150d20 0x7f171c14d6a0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:23.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.894+0000 7f1721dcc640 1 --2- 172.21.15.116:0/1267240621 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f171c0a99f0 0x7f171c13cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:23.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.894+0000 7f1721dcc640 1 --2- 172.21.15.116:0/1267240621 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f171c0a94a0 0x7f171c13f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:23.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.894+0000 7f1721dcc640 1 -- 172.21.15.116:0/1267240621 >> 172.21.15.116:0/1267240621 conn(0x7f171c0c3b00 msgr2=0x7f171c1470e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:23.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.894+0000 7f1721dcc640 1 -- 172.21.15.116:0/1267240621 shutdown_connections 2024-04-25T00:01:23.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:23.894+0000 7f1721dcc640 1 -- 172.21.15.116:0/1267240621 wait complete. 2024-04-25T00:01:23.911 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T00:01:24.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.030+0000 7fe93b938640 1 Processor -- start 2024-04-25T00:01:24.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.030+0000 7fe93b938640 1 -- start start 2024-04-25T00:01:24.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.030+0000 7fe93b938640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe934150d20 0x7fe9340c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:24.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.030+0000 7fe93b938640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe9340c2f50 0x7fe9341538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:24.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.030+0000 7fe93b938640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9340c3910 con 0x7fe9340a94a0 2024-04-25T00:01:24.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.030+0000 7fe93b938640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fe934154530 con 0x7fe9340c2f50 2024-04-25T00:01:24.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.030+0000 7fe93b938640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe9340c16e0 con 0x7fe934150d20 2024-04-25T00:01:24.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.030+0000 7fe938ed3640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe9340c2f50 0x7fe9341538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:24.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.030+0000 7fe938ed3640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe9340c2f50 0x7fe9341538e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54514/0 (socket says 172.21.15.116:54514) 2024-04-25T00:01:24.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.030+0000 7fe938ed3640 1 -- 172.21.15.116:0/622391421 learned_addr learned my addr 172.21.15.116:0/622391421 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:24.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.030+0000 7fe9396d4640 1 --2- 172.21.15.116:0/622391421 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe934150d20 0x7fe9340c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:24.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.030+0000 7fe938ed3640 1 -- 172.21.15.116:0/622391421 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe934150d20 msgr2=0x7fe9340c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:24.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.030+0000 7fe938ed3640 1 --2- 172.21.15.116:0/622391421 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe934150d20 0x7fe9340c2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:24.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.030+0000 7fe938ed3640 1 -- 172.21.15.116:0/622391421 >> v1:172.21.15.174:6789/0 conn(0x7fe9340a94a0 legacy=0x7fe934153e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:24.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.030+0000 7fe938ed3640 1 -- 172.21.15.116:0/622391421 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9340c1400 con 0x7fe9340c2f50 2024-04-25T00:01:24.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.030+0000 7fe9396d4640 1 --2- 172.21.15.116:0/622391421 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe934150d20 0x7fe9340c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:24.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.030+0000 7fe938ed3640 1 --2- 172.21.15.116:0/622391421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9340c2f50 0x7fe9341538e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe930009870 tx=0x7fe93001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4a5488fabeb1e0e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:24.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.030+0000 7fe92bfff640 1 -- 172.21.15.116:0/622391421 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9300ab5f0 con 0x7fe9340c2f50 2024-04-25T00:01:24.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.034+0000 7fe92bfff640 1 -- 172.21.15.116:0/622391421 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9300b0070 con 0x7fe9340c2f50 2024-04-25T00:01:24.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.034+0000 7fe92bfff640 1 -- 172.21.15.116:0/622391421 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9300ab9b0 con 0x7fe9340c2f50 2024-04-25T00:01:24.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.034+0000 7fe93b938640 1 -- 172.21.15.116:0/622391421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9340c2f50 msgr2=0x7fe9341538e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:24.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.034+0000 7fe93b938640 1 --2- 172.21.15.116:0/622391421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9340c2f50 0x7fe9341538e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe930009870 tx=0x7fe93001c520 comp rx=0 tx=0).stop 2024-04-25T00:01:24.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.034+0000 7fe93b938640 1 -- 172.21.15.116:0/622391421 shutdown_connections 2024-04-25T00:01:24.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.034+0000 7fe93b938640 1 --2- 172.21.15.116:0/622391421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9340c2f50 0x7fe9341538e0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:24.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.034+0000 7fe93b938640 1 --2- 172.21.15.116:0/622391421 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe934150d20 0x7fe9340c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:24.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.034+0000 7fe93b938640 1 -- 172.21.15.116:0/622391421 >> 172.21.15.116:0/622391421 conn(0x7fe9340c3b00 msgr2=0x7fe9340c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:24.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.034+0000 7fe93b938640 1 -- 172.21.15.116:0/622391421 shutdown_connections 2024-04-25T00:01:24.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.034+0000 7fe93b938640 1 -- 172.21.15.116:0/622391421 wait complete. 2024-04-25T00:01:24.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.034+0000 7fe93b938640 1 Processor -- start 2024-04-25T00:01:24.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.034+0000 7fe93b938640 1 -- start start 2024-04-25T00:01:24.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.034+0000 7fe93b938640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe9340a94a0 0x7fe934152c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:24.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.034+0000 7fe93b938640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9340c2f50 0x7fe93413f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:24.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.034+0000 7fe93b938640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe934150d20 0x7fe93413f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:24.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.034+0000 7fe93b938640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe9341608b0 con 0x7fe9340c2f50 2024-04-25T00:01:24.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.034+0000 7fe93b938640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe934160730 con 0x7fe934150d20 2024-04-25T00:01:24.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.034+0000 7fe93b938640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe934160a30 con 0x7fe9340a94a0 2024-04-25T00:01:24.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.034+0000 7fe938ed3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9340c2f50 0x7fe93413f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:24.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.034+0000 7fe938ed3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9340c2f50 0x7fe93413f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54522/0 (socket says 172.21.15.116:54522) 2024-04-25T00:01:24.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.034+0000 7fe938ed3640 1 -- 172.21.15.116:0/1374553504 learned_addr learned my addr 172.21.15.116:0/1374553504 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:24.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.034+0000 7fe938ed3640 1 -- 172.21.15.116:0/1374553504 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe9340a94a0 msgr2=0x7fe934152c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:24.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.034+0000 7fe938ed3640 1 --2- 172.21.15.116:0/1374553504 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe9340a94a0 0x7fe934152c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:24.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.034+0000 7fe938ed3640 1 -- 172.21.15.116:0/1374553504 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe934150d20 msgr2=0x7fe93413f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:24.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.034+0000 7fe938ed3640 1 --2- 172.21.15.116:0/1374553504 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe934150d20 0x7fe93413f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:24.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.034+0000 7fe938ed3640 1 -- 172.21.15.116:0/1374553504 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe93414bd70 con 0x7fe9340c2f50 2024-04-25T00:01:24.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.034+0000 7fe938ed3640 1 --2- 172.21.15.116:0/1374553504 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9340c2f50 0x7fe93413f370 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe93001abd0 tx=0x7fe93001ad60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:24.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.034+0000 7fe929ffb640 1 -- 172.21.15.116:0/1374553504 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9300be5f0 con 0x7fe9340c2f50 2024-04-25T00:01:24.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.034+0000 7fe929ffb640 1 -- 172.21.15.116:0/1374553504 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9300be7b0 con 0x7fe9340c2f50 2024-04-25T00:01:24.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.034+0000 7fe93b938640 1 -- 172.21.15.116:0/1374553504 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9340c4e40 con 0x7fe9340c2f50 2024-04-25T00:01:24.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.034+0000 7fe929ffb640 1 -- 172.21.15.116:0/1374553504 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9300beba0 con 0x7fe9340c2f50 2024-04-25T00:01:24.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.034+0000 7fe93b938640 1 -- 172.21.15.116:0/1374553504 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9340c5400 con 0x7fe9340c2f50 2024-04-25T00:01:24.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.038+0000 7fe93b938640 1 -- 172.21.15.116:0/1374553504 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe934068350 con 0x7fe9340c2f50 2024-04-25T00:01:24.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.038+0000 7fe929ffb640 1 -- 172.21.15.116:0/1374553504 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe9300b43f0 con 0x7fe9340c2f50 2024-04-25T00:01:24.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.038+0000 7fe929ffb640 1 --2- 172.21.15.116:0/1374553504 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe9180678c0 0x7fe918069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:24.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.038+0000 7fe929ffb640 1 -- 172.21.15.116:0/1374553504 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7fe9301208c0 con 0x7fe9340c2f50 2024-04-25T00:01:24.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.038+0000 7fe9396d4640 1 --2- 172.21.15.116:0/1374553504 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe9180678c0 0x7fe918069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:24.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.038+0000 7fe9396d4640 1 --2- 172.21.15.116:0/1374553504 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe9180678c0 0x7fe918069d80 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe92400a890 tx=0x7fe924008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:24.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.042+0000 7fe929ffb640 1 -- 172.21.15.116:0/1374553504 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fe9300c4060 con 0x7fe9340c2f50 2024-04-25T00:01:24.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.202+0000 7fe93b938640 1 -- 172.21.15.116:0/1374553504 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe934068350 con 0x7fe9340c2f50 2024-04-25T00:01:24.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.206+0000 7fe929ffb640 1 -- 172.21.15.116:0/1374553504 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+20311 (secure 0 0 0) 0x7fe930016110 con 0x7fe9340c2f50 2024-04-25T00:01:24.207 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:01:24.207 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":27,"fsid":"6d6a1aa7-0478-4c83-8057-05917e81f046","created":"2024-04-25T00:00:40.682493+0000","modified":"2024-04-25T00:01:20.501508+0000","last_up_change":"2024-04-25T00:00:51.822174+0000","last_in_change":"2024-04-25T00:00:42.980473+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T00:00:53.324376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-04-25T00:01:03.197326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-04-25T00:01:03.728618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-04-25T00:01:17.125210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-04-25T00:01:17.821873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8b6ddcb2-1ad9-4167-961c-bd6936bb9060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6809","nonce":3751765682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6811","nonce":3751765682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6815","nonce":3751765682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6813","nonce":3751765682}]},"public_addr":"172.21.15.116:6809/3751765682","cluster_addr":"172.21.15.116:6811/3751765682","heartbeat_back_addr":"172.21.15.116:6815/3751765682","heartbeat_front_addr":"172.21.15.116:6813/3751765682","state":["exists","up"]},{"osd":1,"uuid":"a6431768-994f-41da-8853-886f5687fb91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6801","nonce":377879124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6803","nonce":377879124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6807","nonce":377879124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6805","nonce":377879124}]},"public_addr":"172.21.15.116:6801/377879124","cluster_addr":"172.21.15.116:6803/377879124","heartbeat_back_addr":"172.21.15.116:6807/377879124","heartbeat_front_addr":"172.21.15.116:6805/377879124","state":["exists","up"]},{"osd":2,"uuid":"4f69bd40-5a23-45ff-8693-e1de75e1aa9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6817","nonce":1099282267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6819","nonce":1099282267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6823","nonce":1099282267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6821","nonce":1099282267}]},"public_addr":"172.21.15.116:6817/1099282267","cluster_addr":"172.21.15.116:6819/1099282267","heartbeat_back_addr":"172.21.15.116:6823/1099282267","heartbeat_front_addr":"172.21.15.116:6821/1099282267","state":["exists","up"]},{"osd":3,"uuid":"6a0d5344-8b6f-40d3-918d-0f0a4be688e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6825","nonce":1179942661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6827","nonce":1179942661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6831","nonce":1179942661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6829","nonce":1179942661}]},"public_addr":"172.21.15.116:6825/1179942661","cluster_addr":"172.21.15.116:6827/1179942661","heartbeat_back_addr":"172.21.15.116:6831/1179942661","heartbeat_front_addr":"172.21.15.116:6829/1179942661","state":["exists","up"]},{"osd":4,"uuid":"132e92e4-b825-441e-965a-ca00533fa86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6801","nonce":3132444330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6803","nonce":3132444330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6807","nonce":3132444330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6805","nonce":3132444330}]},"public_addr":"172.21.15.174:6801/3132444330","cluster_addr":"172.21.15.174:6803/3132444330","heartbeat_back_addr":"172.21.15.174:6807/3132444330","heartbeat_front_addr":"172.21.15.174:6805/3132444330","state":["exists","up"]},{"osd":5,"uuid":"a2eeb5b6-1b31-43cc-8306-a0adf3ee15bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6809","nonce":881817164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6811","nonce":881817164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6815","nonce":881817164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6813","nonce":881817164}]},"public_addr":"172.21.15.174:6809/881817164","cluster_addr":"172.21.15.174:6811/881817164","heartbeat_back_addr":"172.21.15.174:6815/881817164","heartbeat_front_addr":"172.21.15.174:6813/881817164","state":["exists","up"]},{"osd":6,"uuid":"588e873a-c841-4ba7-90f2-8956c87a9983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6825","nonce":117574251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6827","nonce":117574251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6831","nonce":117574251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6829","nonce":117574251}]},"public_addr":"172.21.15.174:6825/117574251","cluster_addr":"172.21.15.174:6827/117574251","heartbeat_back_addr":"172.21.15.174:6831/117574251","heartbeat_front_addr":"172.21.15.174:6829/117574251","state":["exists","up"]},{"osd":7,"uuid":"3b90f9b0-8a0e-4883-b088-ca1d4012870c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6817","nonce":2543899196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6819","nonce":2543899196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6823","nonce":2543899196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6821","nonce":2543899196}]},"public_addr":"172.21.15.174:6817/2543899196","cluster_addr":"172.21.15.174:6819/2543899196","heartbeat_back_addr":"172.21.15.174:6823/2543899196","heartbeat_front_addr":"172.21.15.174:6821/2543899196","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.293769+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.310034+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.404865+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.549658+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.192174+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.297266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.493649+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.452309+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:01:24.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.206+0000 7fe93b938640 1 -- 172.21.15.116:0/1374553504 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe9180678c0 msgr2=0x7fe918069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:24.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.206+0000 7fe93b938640 1 --2- 172.21.15.116:0/1374553504 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe9180678c0 0x7fe918069d80 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe92400a890 tx=0x7fe924008040 comp rx=0 tx=0).stop 2024-04-25T00:01:24.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.206+0000 7fe93b938640 1 -- 172.21.15.116:0/1374553504 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9340c2f50 msgr2=0x7fe93413f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:24.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.206+0000 7fe93b938640 1 --2- 172.21.15.116:0/1374553504 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9340c2f50 0x7fe93413f370 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe93001abd0 tx=0x7fe93001ad60 comp rx=0 tx=0).stop 2024-04-25T00:01:24.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.206+0000 7fe93b938640 1 -- 172.21.15.116:0/1374553504 shutdown_connections 2024-04-25T00:01:24.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.206+0000 7fe93b938640 1 --2- 172.21.15.116:0/1374553504 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe9180678c0 0x7fe918069d80 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:24.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.206+0000 7fe93b938640 1 --2- 172.21.15.116:0/1374553504 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe934150d20 0x7fe93413f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:24.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.206+0000 7fe93b938640 1 --2- 172.21.15.116:0/1374553504 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9340c2f50 0x7fe93413f370 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:24.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.206+0000 7fe93b938640 1 --2- 172.21.15.116:0/1374553504 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe9340a94a0 0x7fe934152c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:24.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.206+0000 7fe93b938640 1 -- 172.21.15.116:0/1374553504 >> 172.21.15.116:0/1374553504 conn(0x7fe9340c3b00 msgr2=0x7fe934147070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:24.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.206+0000 7fe93b938640 1 -- 172.21.15.116:0/1374553504 shutdown_connections 2024-04-25T00:01:24.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.206+0000 7fe93b938640 1 -- 172.21.15.116:0/1374553504 wait complete. 2024-04-25T00:01:24.225 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-25T00:01:24.225 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T00:01:24.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.334+0000 7ff7cfb80640 1 Processor -- start 2024-04-25T00:01:24.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.334+0000 7ff7cfb80640 1 -- start start 2024-04-25T00:01:24.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.334+0000 7ff7cfb80640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff7c80a94a0 0x7ff7c81538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:24.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.334+0000 7ff7cfb80640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7c80c3910 con 0x7ff7c80a99f0 2024-04-25T00:01:24.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.334+0000 7ff7cfb80640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7c8154530 con 0x7ff7c8150d20 2024-04-25T00:01:24.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.334+0000 7ff7cfb80640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7ff7c80c16e0 con 0x7ff7c80a94a0 2024-04-25T00:01:24.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.334+0000 7ff7ce11d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7ff7c80a99f0 0x7ff7c8153e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:60938/0 (socket says 172.21.15.116:60938) 2024-04-25T00:01:24.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.334+0000 7ff7ce11d640 1 -- 172.21.15.116:0/3507230835 learned_addr learned my addr 172.21.15.116:0/3507230835 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:24.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.334+0000 7ff7cd11b640 1 --2- 172.21.15.116:0/3507230835 >> v2:172.21.15.174:3300/0 conn(0x7ff7c80a94a0 0x7ff7c81538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:24.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.334+0000 7ff7cc91a640 1 -- 172.21.15.116:0/3507230835 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 974763463 0 0) 0x7ff7c80c3910 con 0x7ff7c80a99f0 2024-04-25T00:01:24.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.334+0000 7ff7cc91a640 1 -- 172.21.15.116:0/3507230835 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff7b4003530 con 0x7ff7c80a99f0 2024-04-25T00:01:24.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.334+0000 7ff7cc91a640 1 -- 172.21.15.116:0/3507230835 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3994222142 0 0) 0x7ff7b4003530 con 0x7ff7c80a99f0 2024-04-25T00:01:24.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.334+0000 7ff7cc91a640 1 -- 172.21.15.116:0/3507230835 >> v2:172.21.15.174:3300/0 conn(0x7ff7c80a94a0 msgr2=0x7ff7c81538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:24.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.334+0000 7ff7cc91a640 1 --2- 172.21.15.116:0/3507230835 >> v2:172.21.15.174:3300/0 conn(0x7ff7c80a94a0 0x7ff7c81538e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:24.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.334+0000 7ff7cc91a640 1 -- 172.21.15.116:0/3507230835 >> v1:172.21.15.174:6789/0 conn(0x7ff7c8150d20 legacy=0x7ff7c80c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:24.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.334+0000 7ff7cc91a640 1 -- 172.21.15.116:0/3507230835 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7c80c1400 con 0x7ff7c80a99f0 2024-04-25T00:01:24.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.334+0000 7ff7cc91a640 1 -- 172.21.15.116:0/3507230835 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7ff7c0001a00 con 0x7ff7c80a99f0 2024-04-25T00:01:24.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.334+0000 7ff7cc91a640 1 -- 172.21.15.116:0/3507230835 >> v1:172.21.15.116:6789/0 conn(0x7ff7c80a99f0 legacy=0x7ff7c8153e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:24.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.334+0000 7ff7cc91a640 1 --2- 172.21.15.116:0/3507230835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7b40039c0 0x7ff7b4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:24.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.334+0000 7ff7cc91a640 1 --2- 172.21.15.116:0/3507230835 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff7b4004810 0x7ff7b4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:24.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.334+0000 7ff7cc91a640 1 --2- 172.21.15.116:0/3507230835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff7b40051d0 0x7ff7b400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:24.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.334+0000 7ff7cc91a640 1 -- 172.21.15.116:0/3507230835 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff7c80c16e0 con 0x7ff7b40039c0 2024-04-25T00:01:24.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.334+0000 7ff7cc91a640 1 -- 172.21.15.116:0/3507230835 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff7b400c070 con 0x7ff7b40051d0 2024-04-25T00:01:24.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.334+0000 7ff7cc91a640 1 -- 172.21.15.116:0/3507230835 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff7b400c270 con 0x7ff7b4004810 2024-04-25T00:01:24.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.334+0000 7ff7cd91c640 1 --2- 172.21.15.116:0/3507230835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7b40039c0 0x7ff7b4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:24.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.338+0000 7ff7ce11d640 1 --2- 172.21.15.116:0/3507230835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff7b40051d0 0x7ff7b400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:24.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.338+0000 7ff7cd11b640 1 --2- 172.21.15.116:0/3507230835 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff7b4004810 0x7ff7b4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:24.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.338+0000 7ff7cd91c640 1 -- 172.21.15.116:0/3507230835 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff7b4004810 msgr2=0x7ff7b4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:24.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.338+0000 7ff7cd91c640 1 --2- 172.21.15.116:0/3507230835 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff7b4004810 0x7ff7b4004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:24.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.338+0000 7ff7cd91c640 1 -- 172.21.15.116:0/3507230835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff7b40051d0 msgr2=0x7ff7b400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:24.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.338+0000 7ff7cd91c640 1 --2- 172.21.15.116:0/3507230835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff7b40051d0 0x7ff7b400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:24.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.338+0000 7ff7ce11d640 1 -- 172.21.15.116:0/3507230835 reap_dead start 2024-04-25T00:01:24.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.338+0000 7ff7cd91c640 1 -- 172.21.15.116:0/3507230835 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7b4003530 con 0x7ff7b40039c0 2024-04-25T00:01:24.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.338+0000 7ff7cd11b640 1 --2- 172.21.15.116:0/3507230835 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff7b4004810 0x7ff7b4004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:24.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.338+0000 7ff7cd91c640 1 --2- 172.21.15.116:0/3507230835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7b40039c0 0x7ff7b4003de0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff7c4001780 tx=0x7ff7c401c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=698e49115286866e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:24.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.338+0000 7ff7cc91a640 1 -- 172.21.15.116:0/3507230835 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c40013a0 con 0x7ff7b40039c0 2024-04-25T00:01:24.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.338+0000 7ff7cc91a640 1 -- 172.21.15.116:0/3507230835 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7c40050d0 con 0x7ff7b40039c0 2024-04-25T00:01:24.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.338+0000 7ff7cc91a640 1 -- 172.21.15.116:0/3507230835 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c4003600 con 0x7ff7b40039c0 2024-04-25T00:01:24.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.338+0000 7ff7cfb80640 1 -- 172.21.15.116:0/3507230835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7b40039c0 msgr2=0x7ff7b4003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:24.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.338+0000 7ff7cfb80640 1 --2- 172.21.15.116:0/3507230835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7b40039c0 0x7ff7b4003de0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff7c4001780 tx=0x7ff7c401c400 comp rx=0 tx=0).stop 2024-04-25T00:01:24.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.338+0000 7ff7cfb80640 1 -- 172.21.15.116:0/3507230835 shutdown_connections 2024-04-25T00:01:24.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.338+0000 7ff7cfb80640 1 --2- 172.21.15.116:0/3507230835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7b40039c0 0x7ff7b4003de0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:24.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.338+0000 7ff7cfb80640 1 -- 172.21.15.116:0/3507230835 >> 172.21.15.116:0/3507230835 conn(0x7ff7c80c3b00 msgr2=0x7ff7c80c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:24.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.338+0000 7ff7cfb80640 1 -- 172.21.15.116:0/3507230835 shutdown_connections 2024-04-25T00:01:24.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.338+0000 7ff7cfb80640 1 -- 172.21.15.116:0/3507230835 wait complete. 2024-04-25T00:01:24.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.342+0000 7ff7cfb80640 1 Processor -- start 2024-04-25T00:01:24.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.342+0000 7ff7cfb80640 1 -- start start 2024-04-25T00:01:24.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.342+0000 7ff7cfb80640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7b40039c0 0x7ff7c81e3270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:24.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.342+0000 7ff7cfb80640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff7c814afd0 0x7ff7c8140760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:24.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.342+0000 7ff7cfb80640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff7c814b410 0x7ff7c8140ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:24.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.342+0000 7ff7cfb80640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff7c8153a60 con 0x7ff7b40039c0 2024-04-25T00:01:24.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.342+0000 7ff7cfb80640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff7c81538e0 con 0x7ff7c814afd0 2024-04-25T00:01:24.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.342+0000 7ff7cfb80640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff7c8153be0 con 0x7ff7c814b410 2024-04-25T00:01:24.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.342+0000 7ff7cd91c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7b40039c0 0x7ff7c81e3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:24.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.342+0000 7ff7cd91c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7b40039c0 0x7ff7c81e3270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54538/0 (socket says 172.21.15.116:54538) 2024-04-25T00:01:24.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.342+0000 7ff7cd91c640 1 -- 172.21.15.116:0/3776243883 learned_addr learned my addr 172.21.15.116:0/3776243883 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:24.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.342+0000 7ff7cd11b640 1 --2- 172.21.15.116:0/3776243883 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff7c814afd0 0x7ff7c8140760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:24.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.342+0000 7ff7ce11d640 1 --2- 172.21.15.116:0/3776243883 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff7c814b410 0x7ff7c8140ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:24.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.342+0000 7ff7cd91c640 1 -- 172.21.15.116:0/3776243883 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff7c814b410 msgr2=0x7ff7c8140ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:24.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.342+0000 7ff7cd91c640 1 --2- 172.21.15.116:0/3776243883 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff7c814b410 0x7ff7c8140ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:24.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.342+0000 7ff7cd91c640 1 -- 172.21.15.116:0/3776243883 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff7c814afd0 msgr2=0x7ff7c8140760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:24.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.342+0000 7ff7cd91c640 1 --2- 172.21.15.116:0/3776243883 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff7c814afd0 0x7ff7c8140760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:24.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.342+0000 7ff7cd91c640 1 -- 172.21.15.116:0/3776243883 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7c80c47e0 con 0x7ff7b40039c0 2024-04-25T00:01:24.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.342+0000 7ff7ce11d640 1 --2- 172.21.15.116:0/3776243883 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff7c814b410 0x7ff7c8140ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:24.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.342+0000 7ff7cd91c640 1 --2- 172.21.15.116:0/3776243883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7b40039c0 0x7ff7c81e3270 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff7c4005040 tx=0x7ff7b4005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:24.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.342+0000 7ff7be7fc640 1 -- 172.21.15.116:0/3776243883 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c40072f0 con 0x7ff7b40039c0 2024-04-25T00:01:24.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.342+0000 7ff7be7fc640 1 -- 172.21.15.116:0/3776243883 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7c40ba070 con 0x7ff7b40039c0 2024-04-25T00:01:24.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.342+0000 7ff7cfb80640 1 -- 172.21.15.116:0/3776243883 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7c81f2130 con 0x7ff7b40039c0 2024-04-25T00:01:24.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.342+0000 7ff7be7fc640 1 -- 172.21.15.116:0/3776243883 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c40076b0 con 0x7ff7b40039c0 2024-04-25T00:01:24.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.342+0000 7ff7cfb80640 1 -- 172.21.15.116:0/3776243883 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7c81f26f0 con 0x7ff7b40039c0 2024-04-25T00:01:24.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.342+0000 7ff7cfb80640 1 -- 172.21.15.116:0/3776243883 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7c8068350 con 0x7ff7b40039c0 2024-04-25T00:01:24.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.342+0000 7ff7be7fc640 1 -- 172.21.15.116:0/3776243883 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff7c40067e0 con 0x7ff7b40039c0 2024-04-25T00:01:24.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.342+0000 7ff7be7fc640 1 --2- 172.21.15.116:0/3776243883 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff7ac0678f0 0x7ff7ac069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:24.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.342+0000 7ff7be7fc640 1 -- 172.21.15.116:0/3776243883 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7ff7c411fd80 con 0x7ff7b40039c0 2024-04-25T00:01:24.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.342+0000 7ff7cd11b640 1 --2- 172.21.15.116:0/3776243883 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff7ac0678f0 0x7ff7ac069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:24.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.346+0000 7ff7cd11b640 1 --2- 172.21.15.116:0/3776243883 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff7ac0678f0 0x7ff7ac069db0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff7c8072400 tx=0x7ff7b8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:24.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.346+0000 7ff7be7fc640 1 -- 172.21.15.116:0/3776243883 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ff7c4016110 con 0x7ff7b40039c0 2024-04-25T00:01:24.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.490+0000 7ff7cfb80640 1 -- 172.21.15.116:0/3776243883 --> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff7c81f1ee0 con 0x7ff7ac0678f0 2024-04-25T00:01:24.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.526+0000 7ff7be7fc640 1 -- 172.21.15.116:0/3776243883 <== mgr.4105 v2:172.21.15.174:6832/2004448211 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+490414 (secure 0 0 0) 0x7ff7c81f1ee0 con 0x7ff7ac0678f0 2024-04-25T00:01:24.534 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:01:24.537 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-04-25T00:01:24.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.534+0000 7ff7cfb80640 1 -- 172.21.15.116:0/3776243883 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff7ac0678f0 msgr2=0x7ff7ac069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:24.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.534+0000 7ff7cfb80640 1 --2- 172.21.15.116:0/3776243883 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff7ac0678f0 0x7ff7ac069db0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff7c8072400 tx=0x7ff7b8008040 comp rx=0 tx=0).stop 2024-04-25T00:01:24.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.534+0000 7ff7cfb80640 1 -- 172.21.15.116:0/3776243883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7b40039c0 msgr2=0x7ff7c81e3270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:24.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.534+0000 7ff7cfb80640 1 --2- 172.21.15.116:0/3776243883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7b40039c0 0x7ff7c81e3270 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff7c4005040 tx=0x7ff7b4005840 comp rx=0 tx=0).stop 2024-04-25T00:01:24.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.534+0000 7ff7cfb80640 1 -- 172.21.15.116:0/3776243883 shutdown_connections 2024-04-25T00:01:24.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.534+0000 7ff7cfb80640 1 --2- 172.21.15.116:0/3776243883 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff7ac0678f0 0x7ff7ac069db0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:24.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.534+0000 7ff7cfb80640 1 --2- 172.21.15.116:0/3776243883 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff7c814b410 0x7ff7c8140ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:24.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.534+0000 7ff7cfb80640 1 --2- 172.21.15.116:0/3776243883 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff7c814afd0 0x7ff7c8140760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:24.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.534+0000 7ff7cfb80640 1 --2- 172.21.15.116:0/3776243883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7b40039c0 0x7ff7c81e3270 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:24.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.534+0000 7ff7cfb80640 1 -- 172.21.15.116:0/3776243883 >> 172.21.15.116:0/3776243883 conn(0x7ff7c80c3b00 msgr2=0x7ff7c8151400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:24.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.534+0000 7ff7cfb80640 1 -- 172.21.15.116:0/3776243883 shutdown_connections 2024-04-25T00:01:24.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.534+0000 7ff7cfb80640 1 -- 172.21.15.116:0/3776243883 wait complete. 2024-04-25T00:01:24.550 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-04-25T00:01:23.256272+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595188,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":249,"num_write_kb":2444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":195,"ondisk_log_size":195,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":372,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219668,"kb_used_data":3456,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755052,"statfs":{"total":773094113280,"available":772869173248,"internally_reserved":0,"allocated":3538944,"data_stored":2050618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":53,"apply_latency_ms":53,"commit_latency_ns":53000000,"apply_latency_ns":53000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.804465"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.577366+0000","last_change":"2024-04-25T00:01:19.508728+0000","last_active":"2024-04-25T00:01:20.577366+0000","last_peered":"2024-04-25T00:01:20.577366+0000","last_clean":"2024-04-25T00:01:20.577366+0000","last_became_active":"2024-04-25T00:01:19.508390+0000","last_became_peered":"2024-04-25T00:01:19.508390+0000","last_unstale":"2024-04-25T00:01:20.577366+0000","last_undegraded":"2024-04-25T00:01:20.577366+0000","last_fullsized":"2024-04-25T00:01:20.577366+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:45.591905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1f","version":"22'4","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T00:01:19.494954+0000","last_change":"2024-04-25T00:01:04.356032+0000","last_active":"2024-04-25T00:01:19.494954+0000","last_peered":"2024-04-25T00:01:19.494954+0000","last_clean":"2024-04-25T00:01:19.494954+0000","last_became_active":"2024-04-25T00:01:04.355293+0000","last_became_peered":"2024-04-25T00:01:04.355293+0000","last_unstale":"2024-04-25T00:01:19.494954+0000","last_undegraded":"2024-04-25T00:01:19.494954+0000","last_fullsized":"2024-04-25T00:01:19.494954+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:56:16.678132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.520751+0000","last_change":"2024-04-25T00:01:05.367207+0000","last_active":"2024-04-25T00:01:20.520751+0000","last_peered":"2024-04-25T00:01:20.520751+0000","last_clean":"2024-04-25T00:01:20.520751+0000","last_became_active":"2024-04-25T00:01:05.366471+0000","last_became_peered":"2024-04-25T00:01:05.366471+0000","last_unstale":"2024-04-25T00:01:20.520751+0000","last_undegraded":"2024-04-25T00:01:20.520751+0000","last_fullsized":"2024-04-25T00:01:20.520751+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:30:23.557381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511325+0000","last_change":"2024-04-25T00:01:18.502364+0000","last_active":"2024-04-25T00:01:20.511325+0000","last_peered":"2024-04-25T00:01:20.511325+0000","last_clean":"2024-04-25T00:01:20.511325+0000","last_became_active":"2024-04-25T00:01:18.501633+0000","last_became_peered":"2024-04-25T00:01:18.501633+0000","last_unstale":"2024-04-25T00:01:20.511325+0000","last_undegraded":"2024-04-25T00:01:20.511325+0000","last_fullsized":"2024-04-25T00:01:20.511325+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:55.074018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T00:01:19.541052+0000","last_change":"2024-04-25T00:01:19.512822+0000","last_active":"2024-04-25T00:01:19.541052+0000","last_peered":"2024-04-25T00:01:19.541052+0000","last_clean":"2024-04-25T00:01:19.541052+0000","last_became_active":"2024-04-25T00:01:19.512105+0000","last_became_peered":"2024-04-25T00:01:19.512105+0000","last_unstale":"2024-04-25T00:01:19.541052+0000","last_undegraded":"2024-04-25T00:01:19.541052+0000","last_fullsized":"2024-04-25T00:01:19.541052+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:07:42.932302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"22'1","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.520903+0000","last_change":"2024-04-25T00:01:04.361809+0000","last_active":"2024-04-25T00:01:20.520903+0000","last_peered":"2024-04-25T00:01:20.520903+0000","last_clean":"2024-04-25T00:01:20.520903+0000","last_became_active":"2024-04-25T00:01:04.361059+0000","last_became_peered":"2024-04-25T00:01:04.361059+0000","last_unstale":"2024-04-25T00:01:20.520903+0000","last_undegraded":"2024-04-25T00:01:20.520903+0000","last_fullsized":"2024-04-25T00:01:20.520903+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.520801+0000","last_change":"2024-04-25T00:01:05.370818+0000","last_active":"2024-04-25T00:01:20.520801+0000","last_peered":"2024-04-25T00:01:20.520801+0000","last_clean":"2024-04-25T00:01:20.520801+0000","last_became_active":"2024-04-25T00:01:05.369991+0000","last_became_peered":"2024-04-25T00:01:05.369991+0000","last_unstale":"2024-04-25T00:01:20.520801+0000","last_undegraded":"2024-04-25T00:01:20.520801+0000","last_fullsized":"2024-04-25T00:01:20.520801+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:43.615895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.510695+0000","last_change":"2024-04-25T00:01:18.497682+0000","last_active":"2024-04-25T00:01:20.510695+0000","last_peered":"2024-04-25T00:01:20.510695+0000","last_clean":"2024-04-25T00:01:20.510695+0000","last_became_active":"2024-04-25T00:01:18.497397+0000","last_became_peered":"2024-04-25T00:01:18.497397+0000","last_unstale":"2024-04-25T00:01:20.510695+0000","last_undegraded":"2024-04-25T00:01:20.510695+0000","last_fullsized":"2024-04-25T00:01:20.510695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:22.983972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578826+0000","last_change":"2024-04-25T00:01:19.510591+0000","last_active":"2024-04-25T00:01:20.578826+0000","last_peered":"2024-04-25T00:01:20.578826+0000","last_clean":"2024-04-25T00:01:20.578826+0000","last_became_active":"2024-04-25T00:01:19.510252+0000","last_became_peered":"2024-04-25T00:01:19.510252+0000","last_unstale":"2024-04-25T00:01:20.578826+0000","last_undegraded":"2024-04-25T00:01:20.578826+0000","last_fullsized":"2024-04-25T00:01:20.578826+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:38.005233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"22'2","reported_seq":24,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582530+0000","last_change":"2024-04-25T00:01:04.362030+0000","last_active":"2024-04-25T00:01:20.582530+0000","last_peered":"2024-04-25T00:01:20.582530+0000","last_clean":"2024-04-25T00:01:20.582530+0000","last_became_active":"2024-04-25T00:01:04.361708+0000","last_became_peered":"2024-04-25T00:01:04.361708+0000","last_unstale":"2024-04-25T00:01:20.582530+0000","last_undegraded":"2024-04-25T00:01:20.582530+0000","last_fullsized":"2024-04-25T00:01:20.582530+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:38:33.392719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578882+0000","last_change":"2024-04-25T00:01:05.387195+0000","last_active":"2024-04-25T00:01:20.578882+0000","last_peered":"2024-04-25T00:01:20.578882+0000","last_clean":"2024-04-25T00:01:20.578882+0000","last_became_active":"2024-04-25T00:01:05.386853+0000","last_became_peered":"2024-04-25T00:01:05.386853+0000","last_unstale":"2024-04-25T00:01:20.578882+0000","last_undegraded":"2024-04-25T00:01:20.578882+0000","last_fullsized":"2024-04-25T00:01:20.578882+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:06:19.307378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"26'1","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511288+0000","last_change":"2024-04-25T00:01:18.487835+0000","last_active":"2024-04-25T00:01:20.511288+0000","last_peered":"2024-04-25T00:01:20.511288+0000","last_clean":"2024-04-25T00:01:20.511288+0000","last_became_active":"2024-04-25T00:01:18.487571+0000","last_became_peered":"2024-04-25T00:01:18.487571+0000","last_unstale":"2024-04-25T00:01:20.511288+0000","last_undegraded":"2024-04-25T00:01:20.511288+0000","last_fullsized":"2024-04-25T00:01:20.511288+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:44.582148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511094+0000","last_change":"2024-04-25T00:01:19.519834+0000","last_active":"2024-04-25T00:01:20.511094+0000","last_peered":"2024-04-25T00:01:20.511094+0000","last_clean":"2024-04-25T00:01:20.511094+0000","last_became_active":"2024-04-25T00:01:19.519470+0000","last_became_peered":"2024-04-25T00:01:19.519470+0000","last_unstale":"2024-04-25T00:01:20.511094+0000","last_undegraded":"2024-04-25T00:01:20.511094+0000","last_fullsized":"2024-04-25T00:01:20.511094+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:47.859626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"22'2","reported_seq":24,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582640+0000","last_change":"2024-04-25T00:01:04.362832+0000","last_active":"2024-04-25T00:01:20.582640+0000","last_peered":"2024-04-25T00:01:20.582640+0000","last_clean":"2024-04-25T00:01:20.582640+0000","last_became_active":"2024-04-25T00:01:04.362059+0000","last_became_peered":"2024-04-25T00:01:04.362059+0000","last_unstale":"2024-04-25T00:01:20.582640+0000","last_undegraded":"2024-04-25T00:01:20.582640+0000","last_fullsized":"2024-04-25T00:01:20.582640+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:05.441048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.583017+0000","last_change":"2024-04-25T00:01:05.381120+0000","last_active":"2024-04-25T00:01:20.583017+0000","last_peered":"2024-04-25T00:01:20.583017+0000","last_clean":"2024-04-25T00:01:20.583017+0000","last_became_active":"2024-04-25T00:01:05.378320+0000","last_became_peered":"2024-04-25T00:01:05.378320+0000","last_unstale":"2024-04-25T00:01:20.583017+0000","last_undegraded":"2024-04-25T00:01:20.583017+0000","last_fullsized":"2024-04-25T00:01:20.583017+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:34:06.175918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"26'1","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.510926+0000","last_change":"2024-04-25T00:01:18.490300+0000","last_active":"2024-04-25T00:01:20.510926+0000","last_peered":"2024-04-25T00:01:20.510926+0000","last_clean":"2024-04-25T00:01:20.510926+0000","last_became_active":"2024-04-25T00:01:18.490050+0000","last_became_peered":"2024-04-25T00:01:18.490050+0000","last_unstale":"2024-04-25T00:01:20.510926+0000","last_undegraded":"2024-04-25T00:01:20.510926+0000","last_fullsized":"2024-04-25T00:01:20.510926+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:13:05.971606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.519505+0000","last_change":"2024-04-25T00:01:19.500026+0000","last_active":"2024-04-25T00:01:20.519505+0000","last_peered":"2024-04-25T00:01:20.519505+0000","last_clean":"2024-04-25T00:01:20.519505+0000","last_became_active":"2024-04-25T00:01:19.499244+0000","last_became_peered":"2024-04-25T00:01:19.499244+0000","last_unstale":"2024-04-25T00:01:20.519505+0000","last_undegraded":"2024-04-25T00:01:20.519505+0000","last_fullsized":"2024-04-25T00:01:20.519505+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:40:36.350832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.580552+0000","last_change":"2024-04-25T00:01:05.376984+0000","last_active":"2024-04-25T00:01:20.580552+0000","last_peered":"2024-04-25T00:01:20.580552+0000","last_clean":"2024-04-25T00:01:20.580552+0000","last_became_active":"2024-04-25T00:01:05.376259+0000","last_became_peered":"2024-04-25T00:01:05.376259+0000","last_unstale":"2024-04-25T00:01:20.580552+0000","last_undegraded":"2024-04-25T00:01:20.580552+0000","last_fullsized":"2024-04-25T00:01:20.580552+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:13:28.543302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"22'2","reported_seq":24,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.581267+0000","last_change":"2024-04-25T00:01:04.359262+0000","last_active":"2024-04-25T00:01:20.581267+0000","last_peered":"2024-04-25T00:01:20.581267+0000","last_clean":"2024-04-25T00:01:20.581267+0000","last_became_active":"2024-04-25T00:01:04.357971+0000","last_became_peered":"2024-04-25T00:01:04.357971+0000","last_unstale":"2024-04-25T00:01:20.581267+0000","last_undegraded":"2024-04-25T00:01:20.581267+0000","last_fullsized":"2024-04-25T00:01:20.581267+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:05.441048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"26'1","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582746+0000","last_change":"2024-04-25T00:01:18.475624+0000","last_active":"2024-04-25T00:01:20.582746+0000","last_peered":"2024-04-25T00:01:20.582746+0000","last_clean":"2024-04-25T00:01:20.582746+0000","last_became_active":"2024-04-25T00:01:18.475149+0000","last_became_peered":"2024-04-25T00:01:18.475149+0000","last_unstale":"2024-04-25T00:01:20.582746+0000","last_undegraded":"2024-04-25T00:01:20.582746+0000","last_fullsized":"2024-04-25T00:01:20.582746+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:40:35.341075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.512956+0000","last_change":"2024-04-25T00:01:19.531113+0000","last_active":"2024-04-25T00:01:20.512956+0000","last_peered":"2024-04-25T00:01:20.512956+0000","last_clean":"2024-04-25T00:01:20.512956+0000","last_became_active":"2024-04-25T00:01:19.520150+0000","last_became_peered":"2024-04-25T00:01:19.520150+0000","last_unstale":"2024-04-25T00:01:20.512956+0000","last_undegraded":"2024-04-25T00:01:20.512956+0000","last_fullsized":"2024-04-25T00:01:20.512956+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:38.152376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511363+0000","last_change":"2024-04-25T00:01:05.362611+0000","last_active":"2024-04-25T00:01:20.511363+0000","last_peered":"2024-04-25T00:01:20.511363+0000","last_clean":"2024-04-25T00:01:20.511363+0000","last_became_active":"2024-04-25T00:01:05.361817+0000","last_became_peered":"2024-04-25T00:01:05.361817+0000","last_unstale":"2024-04-25T00:01:20.511363+0000","last_undegraded":"2024-04-25T00:01:20.511363+0000","last_fullsized":"2024-04-25T00:01:20.511363+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:50.489125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578606+0000","last_change":"2024-04-25T00:01:04.371935+0000","last_active":"2024-04-25T00:01:20.578606+0000","last_peered":"2024-04-25T00:01:20.578606+0000","last_clean":"2024-04-25T00:01:20.578606+0000","last_became_active":"2024-04-25T00:01:04.371420+0000","last_became_peered":"2024-04-25T00:01:04.371420+0000","last_unstale":"2024-04-25T00:01:20.578606+0000","last_undegraded":"2024-04-25T00:01:20.578606+0000","last_fullsized":"2024-04-25T00:01:20.578606+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582229+0000","last_change":"2024-04-25T00:01:18.488623+0000","last_active":"2024-04-25T00:01:20.582229+0000","last_peered":"2024-04-25T00:01:20.582229+0000","last_clean":"2024-04-25T00:01:20.582229+0000","last_became_active":"2024-04-25T00:01:18.488291+0000","last_became_peered":"2024-04-25T00:01:18.488291+0000","last_unstale":"2024-04-25T00:01:20.582229+0000","last_undegraded":"2024-04-25T00:01:20.582229+0000","last_fullsized":"2024-04-25T00:01:20.582229+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:30:21.436681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.584490+0000","last_change":"2024-04-25T00:01:19.516731+0000","last_active":"2024-04-25T00:01:20.584490+0000","last_peered":"2024-04-25T00:01:20.584490+0000","last_clean":"2024-04-25T00:01:20.584490+0000","last_became_active":"2024-04-25T00:01:19.516405+0000","last_became_peered":"2024-04-25T00:01:19.516405+0000","last_unstale":"2024-04-25T00:01:20.584490+0000","last_undegraded":"2024-04-25T00:01:20.584490+0000","last_fullsized":"2024-04-25T00:01:20.584490+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:44:37.936092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T00:01:19.493937+0000","last_change":"2024-04-25T00:01:05.355455+0000","last_active":"2024-04-25T00:01:19.493937+0000","last_peered":"2024-04-25T00:01:19.493937+0000","last_clean":"2024-04-25T00:01:19.493937+0000","last_became_active":"2024-04-25T00:01:05.354699+0000","last_became_peered":"2024-04-25T00:01:05.354699+0000","last_unstale":"2024-04-25T00:01:19.493937+0000","last_undegraded":"2024-04-25T00:01:19.493937+0000","last_fullsized":"2024-04-25T00:01:19.493937+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:43:58.069271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T00:01:19.494923+0000","last_change":"2024-04-25T00:01:04.362268+0000","last_active":"2024-04-25T00:01:19.494923+0000","last_peered":"2024-04-25T00:01:19.494923+0000","last_clean":"2024-04-25T00:01:19.494923+0000","last_became_active":"2024-04-25T00:01:04.361622+0000","last_became_peered":"2024-04-25T00:01:04.361622+0000","last_unstale":"2024-04-25T00:01:19.494923+0000","last_undegraded":"2024-04-25T00:01:19.494923+0000","last_fullsized":"2024-04-25T00:01:19.494923+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:58:46.021343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"26'1","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.514314+0000","last_change":"2024-04-25T00:01:18.494047+0000","last_active":"2024-04-25T00:01:20.514314+0000","last_peered":"2024-04-25T00:01:20.514314+0000","last_clean":"2024-04-25T00:01:20.514314+0000","last_became_active":"2024-04-25T00:01:18.493614+0000","last_became_peered":"2024-04-25T00:01:18.493614+0000","last_unstale":"2024-04-25T00:01:20.514314+0000","last_undegraded":"2024-04-25T00:01:20.514314+0000","last_fullsized":"2024-04-25T00:01:20.514314+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:37:08.562016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582182+0000","last_change":"2024-04-25T00:01:19.505397+0000","last_active":"2024-04-25T00:01:20.582182+0000","last_peered":"2024-04-25T00:01:20.582182+0000","last_clean":"2024-04-25T00:01:20.582182+0000","last_became_active":"2024-04-25T00:01:19.505126+0000","last_became_peered":"2024-04-25T00:01:19.505126+0000","last_unstale":"2024-04-25T00:01:20.582182+0000","last_undegraded":"2024-04-25T00:01:20.582182+0000","last_fullsized":"2024-04-25T00:01:20.582182+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:08:49.640267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.581145+0000","last_change":"2024-04-25T00:01:05.354953+0000","last_active":"2024-04-25T00:01:20.581145+0000","last_peered":"2024-04-25T00:01:20.581145+0000","last_clean":"2024-04-25T00:01:20.581145+0000","last_became_active":"2024-04-25T00:01:05.354118+0000","last_became_peered":"2024-04-25T00:01:05.354118+0000","last_unstale":"2024-04-25T00:01:20.581145+0000","last_undegraded":"2024-04-25T00:01:20.581145+0000","last_fullsized":"2024-04-25T00:01:20.581145+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:09.858617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578494+0000","last_change":"2024-04-25T00:01:04.365515+0000","last_active":"2024-04-25T00:01:20.578494+0000","last_peered":"2024-04-25T00:01:20.578494+0000","last_clean":"2024-04-25T00:01:20.578494+0000","last_became_active":"2024-04-25T00:01:04.364942+0000","last_became_peered":"2024-04-25T00:01:04.364942+0000","last_unstale":"2024-04-25T00:01:20.578494+0000","last_undegraded":"2024-04-25T00:01:20.578494+0000","last_fullsized":"2024-04-25T00:01:20.578494+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"26'2","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511306+0000","last_change":"2024-04-25T00:01:18.476207+0000","last_active":"2024-04-25T00:01:20.511306+0000","last_peered":"2024-04-25T00:01:20.511306+0000","last_clean":"2024-04-25T00:01:20.511306+0000","last_became_active":"2024-04-25T00:01:18.475862+0000","last_became_peered":"2024-04-25T00:01:18.475862+0000","last_unstale":"2024-04-25T00:01:20.511306+0000","last_undegraded":"2024-04-25T00:01:20.511306+0000","last_fullsized":"2024-04-25T00:01:20.511306+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:11:40.170249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T00:01:19.542432+0000","last_change":"2024-04-25T00:01:19.529607+0000","last_active":"2024-04-25T00:01:19.542432+0000","last_peered":"2024-04-25T00:01:19.542432+0000","last_clean":"2024-04-25T00:01:19.542432+0000","last_became_active":"2024-04-25T00:01:19.528509+0000","last_became_peered":"2024-04-25T00:01:19.528509+0000","last_unstale":"2024-04-25T00:01:19.542432+0000","last_undegraded":"2024-04-25T00:01:19.542432+0000","last_fullsized":"2024-04-25T00:01:19.542432+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:47:26.096921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T00:01:19.497197+0000","last_change":"2024-04-25T00:01:05.358106+0000","last_active":"2024-04-25T00:01:19.497197+0000","last_peered":"2024-04-25T00:01:19.497197+0000","last_clean":"2024-04-25T00:01:19.497197+0000","last_became_active":"2024-04-25T00:01:05.357135+0000","last_became_peered":"2024-04-25T00:01:05.357135+0000","last_unstale":"2024-04-25T00:01:19.497197+0000","last_undegraded":"2024-04-25T00:01:19.497197+0000","last_fullsized":"2024-04-25T00:01:19.497197+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:41:42.228234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"22'2","reported_seq":24,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.584846+0000","last_change":"2024-04-25T00:01:04.369917+0000","last_active":"2024-04-25T00:01:20.584846+0000","last_peered":"2024-04-25T00:01:20.584846+0000","last_clean":"2024-04-25T00:01:20.584846+0000","last_became_active":"2024-04-25T00:01:04.369309+0000","last_became_peered":"2024-04-25T00:01:04.369309+0000","last_unstale":"2024-04-25T00:01:20.584846+0000","last_undegraded":"2024-04-25T00:01:20.584846+0000","last_fullsized":"2024-04-25T00:01:20.584846+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:55:08.063965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511314+0000","last_change":"2024-04-25T00:01:18.477189+0000","last_active":"2024-04-25T00:01:20.511314+0000","last_peered":"2024-04-25T00:01:20.511314+0000","last_clean":"2024-04-25T00:01:20.511314+0000","last_became_active":"2024-04-25T00:01:18.476909+0000","last_became_peered":"2024-04-25T00:01:18.476909+0000","last_unstale":"2024-04-25T00:01:20.511314+0000","last_undegraded":"2024-04-25T00:01:20.511314+0000","last_fullsized":"2024-04-25T00:01:20.511314+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:36.995477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.580550+0000","last_change":"2024-04-25T00:01:19.517500+0000","last_active":"2024-04-25T00:01:20.580550+0000","last_peered":"2024-04-25T00:01:20.580550+0000","last_clean":"2024-04-25T00:01:20.580550+0000","last_became_active":"2024-04-25T00:01:19.517182+0000","last_became_peered":"2024-04-25T00:01:19.517182+0000","last_unstale":"2024-04-25T00:01:20.580550+0000","last_undegraded":"2024-04-25T00:01:20.580550+0000","last_fullsized":"2024-04-25T00:01:20.580550+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:58:49.738233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T00:01:19.494701+0000","last_change":"2024-04-25T00:01:05.364829+0000","last_active":"2024-04-25T00:01:19.494701+0000","last_peered":"2024-04-25T00:01:19.494701+0000","last_clean":"2024-04-25T00:01:19.494701+0000","last_became_active":"2024-04-25T00:01:05.364125+0000","last_became_peered":"2024-04-25T00:01:05.364125+0000","last_unstale":"2024-04-25T00:01:19.494701+0000","last_undegraded":"2024-04-25T00:01:19.494701+0000","last_fullsized":"2024-04-25T00:01:19.494701+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:15:45.531760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"23'1","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T00:01:19.498006+0000","last_change":"2024-04-25T00:01:04.361603+0000","last_active":"2024-04-25T00:01:19.498006+0000","last_peered":"2024-04-25T00:01:19.498006+0000","last_clean":"2024-04-25T00:01:19.498006+0000","last_became_active":"2024-04-25T00:01:04.361046+0000","last_became_peered":"2024-04-25T00:01:04.361046+0000","last_unstale":"2024-04-25T00:01:19.498006+0000","last_undegraded":"2024-04-25T00:01:19.498006+0000","last_fullsized":"2024-04-25T00:01:19.498006+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:35:18.036843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T00:01:19.494649+0000","last_change":"2024-04-25T00:01:18.488352+0000","last_active":"2024-04-25T00:01:19.494649+0000","last_peered":"2024-04-25T00:01:19.494649+0000","last_clean":"2024-04-25T00:01:19.494649+0000","last_became_active":"2024-04-25T00:01:18.487321+0000","last_became_peered":"2024-04-25T00:01:18.487321+0000","last_unstale":"2024-04-25T00:01:19.494649+0000","last_undegraded":"2024-04-25T00:01:19.494649+0000","last_fullsized":"2024-04-25T00:01:19.494649+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:37:22.329655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578359+0000","last_change":"2024-04-25T00:01:19.519653+0000","last_active":"2024-04-25T00:01:20.578359+0000","last_peered":"2024-04-25T00:01:20.578359+0000","last_clean":"2024-04-25T00:01:20.578359+0000","last_became_active":"2024-04-25T00:01:19.519052+0000","last_became_peered":"2024-04-25T00:01:19.519052+0000","last_unstale":"2024-04-25T00:01:20.578359+0000","last_undegraded":"2024-04-25T00:01:20.578359+0000","last_fullsized":"2024-04-25T00:01:20.578359+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:40:36.350832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.579276+0000","last_change":"2024-04-25T00:01:05.374011+0000","last_active":"2024-04-25T00:01:20.579276+0000","last_peered":"2024-04-25T00:01:20.579276+0000","last_clean":"2024-04-25T00:01:20.579276+0000","last_became_active":"2024-04-25T00:01:05.373276+0000","last_became_peered":"2024-04-25T00:01:05.373276+0000","last_unstale":"2024-04-25T00:01:20.579276+0000","last_undegraded":"2024-04-25T00:01:20.579276+0000","last_fullsized":"2024-04-25T00:01:20.579276+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:40:53.349428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.583723+0000","last_change":"2024-04-25T00:01:04.365391+0000","last_active":"2024-04-25T00:01:20.583723+0000","last_peered":"2024-04-25T00:01:20.583723+0000","last_clean":"2024-04-25T00:01:20.583723+0000","last_became_active":"2024-04-25T00:01:04.364989+0000","last_became_peered":"2024-04-25T00:01:04.364989+0000","last_unstale":"2024-04-25T00:01:20.583723+0000","last_undegraded":"2024-04-25T00:01:20.583723+0000","last_fullsized":"2024-04-25T00:01:20.583723+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:06:18.298057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578642+0000","last_change":"2024-04-25T00:01:18.491597+0000","last_active":"2024-04-25T00:01:20.578642+0000","last_peered":"2024-04-25T00:01:20.578642+0000","last_clean":"2024-04-25T00:01:20.578642+0000","last_became_active":"2024-04-25T00:01:18.490859+0000","last_became_peered":"2024-04-25T00:01:18.490859+0000","last_unstale":"2024-04-25T00:01:20.578642+0000","last_undegraded":"2024-04-25T00:01:20.578642+0000","last_fullsized":"2024-04-25T00:01:20.578642+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:36.995477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T00:01:19.542507+0000","last_change":"2024-04-25T00:01:19.532926+0000","last_active":"2024-04-25T00:01:19.542507+0000","last_peered":"2024-04-25T00:01:19.542507+0000","last_clean":"2024-04-25T00:01:19.542507+0000","last_became_active":"2024-04-25T00:01:19.529742+0000","last_became_peered":"2024-04-25T00:01:19.529742+0000","last_unstale":"2024-04-25T00:01:19.542507+0000","last_undegraded":"2024-04-25T00:01:19.542507+0000","last_fullsized":"2024-04-25T00:01:19.542507+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:14:12.893663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511333+0000","last_change":"2024-04-25T00:01:05.378340+0000","last_active":"2024-04-25T00:01:20.511333+0000","last_peered":"2024-04-25T00:01:20.511333+0000","last_clean":"2024-04-25T00:01:20.511333+0000","last_became_active":"2024-04-25T00:01:05.377593+0000","last_became_peered":"2024-04-25T00:01:05.377593+0000","last_unstale":"2024-04-25T00:01:20.511333+0000","last_undegraded":"2024-04-25T00:01:20.511333+0000","last_fullsized":"2024-04-25T00:01:20.511333+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:48.202251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.519859+0000","last_change":"2024-04-25T00:01:04.366577+0000","last_active":"2024-04-25T00:01:20.519859+0000","last_peered":"2024-04-25T00:01:20.519859+0000","last_clean":"2024-04-25T00:01:20.519859+0000","last_became_active":"2024-04-25T00:01:04.365881+0000","last_became_peered":"2024-04-25T00:01:04.365881+0000","last_unstale":"2024-04-25T00:01:20.519859+0000","last_undegraded":"2024-04-25T00:01:20.519859+0000","last_fullsized":"2024-04-25T00:01:20.519859+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:49:22.673421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.577736+0000","last_change":"2024-04-25T00:01:18.504341+0000","last_active":"2024-04-25T00:01:20.577736+0000","last_peered":"2024-04-25T00:01:20.577736+0000","last_clean":"2024-04-25T00:01:20.577736+0000","last_became_active":"2024-04-25T00:01:18.491498+0000","last_became_peered":"2024-04-25T00:01:18.491498+0000","last_unstale":"2024-04-25T00:01:20.577736+0000","last_undegraded":"2024-04-25T00:01:20.577736+0000","last_fullsized":"2024-04-25T00:01:20.577736+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:56.741250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.513341+0000","last_change":"2024-04-25T00:01:19.539074+0000","last_active":"2024-04-25T00:01:20.513341+0000","last_peered":"2024-04-25T00:01:20.513341+0000","last_clean":"2024-04-25T00:01:20.513341+0000","last_became_active":"2024-04-25T00:01:19.538447+0000","last_became_peered":"2024-04-25T00:01:19.538447+0000","last_unstale":"2024-04-25T00:01:20.513341+0000","last_undegraded":"2024-04-25T00:01:20.513341+0000","last_fullsized":"2024-04-25T00:01:20.513341+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:08:49.640267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.581346+0000","last_change":"2024-04-25T00:01:05.368769+0000","last_active":"2024-04-25T00:01:20.581346+0000","last_peered":"2024-04-25T00:01:20.581346+0000","last_clean":"2024-04-25T00:01:20.581346+0000","last_became_active":"2024-04-25T00:01:05.366817+0000","last_became_peered":"2024-04-25T00:01:05.366817+0000","last_unstale":"2024-04-25T00:01:20.581346+0000","last_undegraded":"2024-04-25T00:01:20.581346+0000","last_fullsized":"2024-04-25T00:01:20.581346+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:11:27.044893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.581006+0000","last_change":"2024-04-25T00:01:04.361900+0000","last_active":"2024-04-25T00:01:20.581006+0000","last_peered":"2024-04-25T00:01:20.581006+0000","last_clean":"2024-04-25T00:01:20.581006+0000","last_became_active":"2024-04-25T00:01:04.361572+0000","last_became_peered":"2024-04-25T00:01:04.361572+0000","last_unstale":"2024-04-25T00:01:20.581006+0000","last_undegraded":"2024-04-25T00:01:20.581006+0000","last_fullsized":"2024-04-25T00:01:20.581006+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:29.218739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.579029+0000","last_change":"2024-04-25T00:01:18.504139+0000","last_active":"2024-04-25T00:01:20.579029+0000","last_peered":"2024-04-25T00:01:20.579029+0000","last_clean":"2024-04-25T00:01:20.579029+0000","last_became_active":"2024-04-25T00:01:18.503379+0000","last_became_peered":"2024-04-25T00:01:18.503379+0000","last_unstale":"2024-04-25T00:01:20.579029+0000","last_undegraded":"2024-04-25T00:01:20.579029+0000","last_fullsized":"2024-04-25T00:01:20.579029+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:02:56.363258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.514229+0000","last_change":"2024-04-25T00:01:19.531756+0000","last_active":"2024-04-25T00:01:20.514229+0000","last_peered":"2024-04-25T00:01:20.514229+0000","last_clean":"2024-04-25T00:01:20.514229+0000","last_became_active":"2024-04-25T00:01:19.531145+0000","last_became_peered":"2024-04-25T00:01:19.531145+0000","last_unstale":"2024-04-25T00:01:20.514229+0000","last_undegraded":"2024-04-25T00:01:20.514229+0000","last_fullsized":"2024-04-25T00:01:20.514229+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:13:42.678415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.518780+0000","last_change":"2024-04-25T00:01:05.354557+0000","last_active":"2024-04-25T00:01:20.518780+0000","last_peered":"2024-04-25T00:01:20.518780+0000","last_clean":"2024-04-25T00:01:20.518780+0000","last_became_active":"2024-04-25T00:01:05.353556+0000","last_became_peered":"2024-04-25T00:01:05.353556+0000","last_unstale":"2024-04-25T00:01:20.518780+0000","last_undegraded":"2024-04-25T00:01:20.518780+0000","last_fullsized":"2024-04-25T00:01:20.518780+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:24.964055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511447+0000","last_change":"2024-04-25T00:01:04.372916+0000","last_active":"2024-04-25T00:01:20.511447+0000","last_peered":"2024-04-25T00:01:20.511447+0000","last_clean":"2024-04-25T00:01:20.511447+0000","last_became_active":"2024-04-25T00:01:04.372185+0000","last_became_peered":"2024-04-25T00:01:04.372185+0000","last_unstale":"2024-04-25T00:01:20.511447+0000","last_undegraded":"2024-04-25T00:01:20.511447+0000","last_fullsized":"2024-04-25T00:01:20.511447+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"26'2","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.579299+0000","last_change":"2024-04-25T00:01:18.504614+0000","last_active":"2024-04-25T00:01:20.579299+0000","last_peered":"2024-04-25T00:01:20.579299+0000","last_clean":"2024-04-25T00:01:20.579299+0000","last_became_active":"2024-04-25T00:01:18.501448+0000","last_became_peered":"2024-04-25T00:01:18.501448+0000","last_unstale":"2024-04-25T00:01:20.579299+0000","last_undegraded":"2024-04-25T00:01:20.579299+0000","last_fullsized":"2024-04-25T00:01:20.579299+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:22.983972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.584367+0000","last_change":"2024-04-25T00:01:19.517510+0000","last_active":"2024-04-25T00:01:20.584367+0000","last_peered":"2024-04-25T00:01:20.584367+0000","last_clean":"2024-04-25T00:01:20.584367+0000","last_became_active":"2024-04-25T00:01:19.517243+0000","last_became_peered":"2024-04-25T00:01:19.517243+0000","last_unstale":"2024-04-25T00:01:20.584367+0000","last_undegraded":"2024-04-25T00:01:20.584367+0000","last_fullsized":"2024-04-25T00:01:20.584367+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:42:38.149502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.577555+0000","last_change":"2024-04-25T00:01:05.370968+0000","last_active":"2024-04-25T00:01:20.577555+0000","last_peered":"2024-04-25T00:01:20.577555+0000","last_clean":"2024-04-25T00:01:20.577555+0000","last_became_active":"2024-04-25T00:01:05.369987+0000","last_became_peered":"2024-04-25T00:01:05.369987+0000","last_unstale":"2024-04-25T00:01:20.577555+0000","last_undegraded":"2024-04-25T00:01:20.577555+0000","last_fullsized":"2024-04-25T00:01:20.577555+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.580498+0000","last_change":"2024-04-25T00:01:04.370226+0000","last_active":"2024-04-25T00:01:20.580498+0000","last_peered":"2024-04-25T00:01:20.580498+0000","last_clean":"2024-04-25T00:01:20.580498+0000","last_became_active":"2024-04-25T00:01:04.369904+0000","last_became_peered":"2024-04-25T00:01:04.369904+0000","last_unstale":"2024-04-25T00:01:20.580498+0000","last_undegraded":"2024-04-25T00:01:20.580498+0000","last_fullsized":"2024-04-25T00:01:20.580498+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:47:25.595275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"27'1","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.523846+0000","last_change":"2024-04-25T00:01:18.497156+0000","last_active":"2024-04-25T00:01:20.523846+0000","last_peered":"2024-04-25T00:01:20.523846+0000","last_clean":"2024-04-25T00:01:20.523846+0000","last_became_active":"2024-04-25T00:01:18.495573+0000","last_became_peered":"2024-04-25T00:01:18.495573+0000","last_unstale":"2024-04-25T00:01:20.523846+0000","last_undegraded":"2024-04-25T00:01:20.523846+0000","last_fullsized":"2024-04-25T00:01:20.523846+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:29.119299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.581909+0000","last_change":"2024-04-25T00:01:19.519411+0000","last_active":"2024-04-25T00:01:20.581909+0000","last_peered":"2024-04-25T00:01:20.581909+0000","last_clean":"2024-04-25T00:01:20.581909+0000","last_became_active":"2024-04-25T00:01:19.519020+0000","last_became_peered":"2024-04-25T00:01:19.519020+0000","last_unstale":"2024-04-25T00:01:20.581909+0000","last_undegraded":"2024-04-25T00:01:20.581909+0000","last_fullsized":"2024-04-25T00:01:20.581909+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:50.351599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T00:01:19.496766+0000","last_change":"2024-04-25T00:01:05.364939+0000","last_active":"2024-04-25T00:01:19.496766+0000","last_peered":"2024-04-25T00:01:19.496766+0000","last_clean":"2024-04-25T00:01:19.496766+0000","last_became_active":"2024-04-25T00:01:05.364272+0000","last_became_peered":"2024-04-25T00:01:05.364272+0000","last_unstale":"2024-04-25T00:01:19.496766+0000","last_undegraded":"2024-04-25T00:01:19.496766+0000","last_fullsized":"2024-04-25T00:01:19.496766+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:24:59.679908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.581536+0000","last_change":"2024-04-25T00:01:04.357028+0000","last_active":"2024-04-25T00:01:20.581536+0000","last_peered":"2024-04-25T00:01:20.581536+0000","last_clean":"2024-04-25T00:01:20.581536+0000","last_became_active":"2024-04-25T00:01:04.356644+0000","last_became_peered":"2024-04-25T00:01:04.356644+0000","last_unstale":"2024-04-25T00:01:20.581536+0000","last_undegraded":"2024-04-25T00:01:20.581536+0000","last_fullsized":"2024-04-25T00:01:20.581536+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:21.729893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":178,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.577513+0000","last_change":"2024-04-25T00:00:55.084492+0000","last_active":"2024-04-25T00:01:20.577513+0000","last_peered":"2024-04-25T00:01:20.577513+0000","last_clean":"2024-04-25T00:01:20.577513+0000","last_became_active":"2024-04-25T00:00:55.083493+0000","last_became_peered":"2024-04-25T00:00:55.083493+0000","last_unstale":"2024-04-25T00:01:20.577513+0000","last_undegraded":"2024-04-25T00:01:20.577513+0000","last_fullsized":"2024-04-25T00:01:20.577513+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:00:54.060578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:00:54.060578+0000","last_clean_scrub_stamp":"2024-04-25T00:00:54.060578+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:14:58.472477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.583214+0000","last_change":"2024-04-25T00:01:18.476298+0000","last_active":"2024-04-25T00:01:20.583214+0000","last_peered":"2024-04-25T00:01:20.583214+0000","last_clean":"2024-04-25T00:01:20.583214+0000","last_became_active":"2024-04-25T00:01:18.475898+0000","last_became_peered":"2024-04-25T00:01:18.475898+0000","last_unstale":"2024-04-25T00:01:20.583214+0000","last_undegraded":"2024-04-25T00:01:20.583214+0000","last_fullsized":"2024-04-25T00:01:20.583214+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:49.341842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.519562+0000","last_change":"2024-04-25T00:01:19.518102+0000","last_active":"2024-04-25T00:01:20.519562+0000","last_peered":"2024-04-25T00:01:20.519562+0000","last_clean":"2024-04-25T00:01:20.519562+0000","last_became_active":"2024-04-25T00:01:19.517376+0000","last_became_peered":"2024-04-25T00:01:19.517376+0000","last_unstale":"2024-04-25T00:01:20.519562+0000","last_undegraded":"2024-04-25T00:01:20.519562+0000","last_fullsized":"2024-04-25T00:01:20.519562+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:55:46.853565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.583333+0000","last_change":"2024-04-25T00:01:05.376609+0000","last_active":"2024-04-25T00:01:20.583333+0000","last_peered":"2024-04-25T00:01:20.583333+0000","last_clean":"2024-04-25T00:01:20.583333+0000","last_became_active":"2024-04-25T00:01:05.375881+0000","last_became_peered":"2024-04-25T00:01:05.375881+0000","last_unstale":"2024-04-25T00:01:20.583333+0000","last_undegraded":"2024-04-25T00:01:20.583333+0000","last_fullsized":"2024-04-25T00:01:20.583333+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:46:05.592233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.579387+0000","last_change":"2024-04-25T00:01:04.368954+0000","last_active":"2024-04-25T00:01:20.579387+0000","last_peered":"2024-04-25T00:01:20.579387+0000","last_clean":"2024-04-25T00:01:20.579387+0000","last_became_active":"2024-04-25T00:01:04.367903+0000","last_became_peered":"2024-04-25T00:01:04.367903+0000","last_unstale":"2024-04-25T00:01:20.579387+0000","last_undegraded":"2024-04-25T00:01:20.579387+0000","last_fullsized":"2024-04-25T00:01:20.579387+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511154+0000","last_change":"2024-04-25T00:01:18.476525+0000","last_active":"2024-04-25T00:01:20.511154+0000","last_peered":"2024-04-25T00:01:20.511154+0000","last_clean":"2024-04-25T00:01:20.511154+0000","last_became_active":"2024-04-25T00:01:18.476247+0000","last_became_peered":"2024-04-25T00:01:18.476247+0000","last_unstale":"2024-04-25T00:01:20.511154+0000","last_undegraded":"2024-04-25T00:01:20.511154+0000","last_fullsized":"2024-04-25T00:01:20.511154+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:30:36.682736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.577886+0000","last_change":"2024-04-25T00:01:19.514615+0000","last_active":"2024-04-25T00:01:20.577886+0000","last_peered":"2024-04-25T00:01:20.577886+0000","last_clean":"2024-04-25T00:01:20.577886+0000","last_became_active":"2024-04-25T00:01:19.513640+0000","last_became_peered":"2024-04-25T00:01:19.513640+0000","last_unstale":"2024-04-25T00:01:20.577886+0000","last_undegraded":"2024-04-25T00:01:20.577886+0000","last_fullsized":"2024-04-25T00:01:20.577886+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:08.620602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582498+0000","last_change":"2024-04-25T00:01:05.385670+0000","last_active":"2024-04-25T00:01:20.582498+0000","last_peered":"2024-04-25T00:01:20.582498+0000","last_clean":"2024-04-25T00:01:20.582498+0000","last_became_active":"2024-04-25T00:01:05.384822+0000","last_became_peered":"2024-04-25T00:01:05.384822+0000","last_unstale":"2024-04-25T00:01:20.582498+0000","last_undegraded":"2024-04-25T00:01:20.582498+0000","last_fullsized":"2024-04-25T00:01:20.582498+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:24.017263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"22'1","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.520366+0000","last_change":"2024-04-25T00:01:04.356627+0000","last_active":"2024-04-25T00:01:20.520366+0000","last_peered":"2024-04-25T00:01:20.520366+0000","last_clean":"2024-04-25T00:01:20.520366+0000","last_became_active":"2024-04-25T00:01:04.354419+0000","last_became_peered":"2024-04-25T00:01:04.354419+0000","last_unstale":"2024-04-25T00:01:20.520366+0000","last_undegraded":"2024-04-25T00:01:20.520366+0000","last_fullsized":"2024-04-25T00:01:20.520366+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:26.243635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"26'2","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.519762+0000","last_change":"2024-04-25T00:01:18.490511+0000","last_active":"2024-04-25T00:01:20.519762+0000","last_peered":"2024-04-25T00:01:20.519762+0000","last_clean":"2024-04-25T00:01:20.519762+0000","last_became_active":"2024-04-25T00:01:18.489629+0000","last_became_peered":"2024-04-25T00:01:18.489629+0000","last_unstale":"2024-04-25T00:01:20.519762+0000","last_undegraded":"2024-04-25T00:01:20.519762+0000","last_fullsized":"2024-04-25T00:01:20.519762+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:07.610846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.579469+0000","last_change":"2024-04-25T00:01:19.508931+0000","last_active":"2024-04-25T00:01:20.579469+0000","last_peered":"2024-04-25T00:01:20.579469+0000","last_clean":"2024-04-25T00:01:20.579469+0000","last_became_active":"2024-04-25T00:01:19.507867+0000","last_became_peered":"2024-04-25T00:01:19.507867+0000","last_unstale":"2024-04-25T00:01:20.579469+0000","last_undegraded":"2024-04-25T00:01:20.579469+0000","last_fullsized":"2024-04-25T00:01:20.579469+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:38.152376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.583381+0000","last_change":"2024-04-25T00:01:05.378840+0000","last_active":"2024-04-25T00:01:20.583381+0000","last_peered":"2024-04-25T00:01:20.583381+0000","last_clean":"2024-04-25T00:01:20.583381+0000","last_became_active":"2024-04-25T00:01:05.378112+0000","last_became_peered":"2024-04-25T00:01:05.378112+0000","last_unstale":"2024-04-25T00:01:20.583381+0000","last_undegraded":"2024-04-25T00:01:20.583381+0000","last_fullsized":"2024-04-25T00:01:20.583381+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:43.615895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"22'2","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T00:01:19.497702+0000","last_change":"2024-04-25T00:01:04.370794+0000","last_active":"2024-04-25T00:01:19.497702+0000","last_peered":"2024-04-25T00:01:19.497702+0000","last_clean":"2024-04-25T00:01:19.497702+0000","last_became_active":"2024-04-25T00:01:04.370099+0000","last_became_peered":"2024-04-25T00:01:04.370099+0000","last_unstale":"2024-04-25T00:01:19.497702+0000","last_undegraded":"2024-04-25T00:01:19.497702+0000","last_fullsized":"2024-04-25T00:01:19.497702+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:42:03.181036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.579618+0000","last_change":"2024-04-25T00:01:18.475256+0000","last_active":"2024-04-25T00:01:20.579618+0000","last_peered":"2024-04-25T00:01:20.579618+0000","last_clean":"2024-04-25T00:01:20.579618+0000","last_became_active":"2024-04-25T00:01:18.474952+0000","last_became_peered":"2024-04-25T00:01:18.474952+0000","last_unstale":"2024-04-25T00:01:20.579618+0000","last_undegraded":"2024-04-25T00:01:20.579618+0000","last_fullsized":"2024-04-25T00:01:20.579618+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:46.849870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.577592+0000","last_change":"2024-04-25T00:01:19.516833+0000","last_active":"2024-04-25T00:01:20.577592+0000","last_peered":"2024-04-25T00:01:20.577592+0000","last_clean":"2024-04-25T00:01:20.577592+0000","last_became_active":"2024-04-25T00:01:19.515926+0000","last_became_peered":"2024-04-25T00:01:19.515926+0000","last_unstale":"2024-04-25T00:01:20.577592+0000","last_undegraded":"2024-04-25T00:01:20.577592+0000","last_fullsized":"2024-04-25T00:01:20.577592+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:23:48.855381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511807+0000","last_change":"2024-04-25T00:01:05.372591+0000","last_active":"2024-04-25T00:01:20.511807+0000","last_peered":"2024-04-25T00:01:20.511807+0000","last_clean":"2024-04-25T00:01:20.511807+0000","last_became_active":"2024-04-25T00:01:05.371204+0000","last_became_peered":"2024-04-25T00:01:05.371204+0000","last_unstale":"2024-04-25T00:01:20.511807+0000","last_undegraded":"2024-04-25T00:01:20.511807+0000","last_fullsized":"2024-04-25T00:01:20.511807+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:15.879673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"22'1","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.583451+0000","last_change":"2024-04-25T00:01:04.372376+0000","last_active":"2024-04-25T00:01:20.583451+0000","last_peered":"2024-04-25T00:01:20.583451+0000","last_clean":"2024-04-25T00:01:20.583451+0000","last_became_active":"2024-04-25T00:01:04.372040+0000","last_became_peered":"2024-04-25T00:01:04.372040+0000","last_unstale":"2024-04-25T00:01:20.583451+0000","last_undegraded":"2024-04-25T00:01:20.583451+0000","last_fullsized":"2024-04-25T00:01:20.583451+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:21.729893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578134+0000","last_change":"2024-04-25T00:01:18.487548+0000","last_active":"2024-04-25T00:01:20.578134+0000","last_peered":"2024-04-25T00:01:20.578134+0000","last_clean":"2024-04-25T00:01:20.578134+0000","last_became_active":"2024-04-25T00:01:18.487227+0000","last_became_peered":"2024-04-25T00:01:18.487227+0000","last_unstale":"2024-04-25T00:01:20.578134+0000","last_undegraded":"2024-04-25T00:01:20.578134+0000","last_fullsized":"2024-04-25T00:01:20.578134+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:55.074018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T00:01:19.542702+0000","last_change":"2024-04-25T00:01:19.528574+0000","last_active":"2024-04-25T00:01:19.542702+0000","last_peered":"2024-04-25T00:01:19.542702+0000","last_clean":"2024-04-25T00:01:19.542702+0000","last_became_active":"2024-04-25T00:01:19.525531+0000","last_became_peered":"2024-04-25T00:01:19.525531+0000","last_unstale":"2024-04-25T00:01:19.542702+0000","last_undegraded":"2024-04-25T00:01:19.542702+0000","last_fullsized":"2024-04-25T00:01:19.542702+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:20:40.636536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.580931+0000","last_change":"2024-04-25T00:01:05.364103+0000","last_active":"2024-04-25T00:01:20.580931+0000","last_peered":"2024-04-25T00:01:20.580931+0000","last_clean":"2024-04-25T00:01:20.580931+0000","last_became_active":"2024-04-25T00:01:05.363320+0000","last_became_peered":"2024-04-25T00:01:05.363320+0000","last_unstale":"2024-04-25T00:01:20.580931+0000","last_undegraded":"2024-04-25T00:01:20.580931+0000","last_fullsized":"2024-04-25T00:01:20.580931+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:31.456792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"22'8","reported_seq":28,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T00:01:19.494273+0000","last_change":"2024-04-25T00:01:04.358365+0000","last_active":"2024-04-25T00:01:19.494273+0000","last_peered":"2024-04-25T00:01:19.494273+0000","last_clean":"2024-04-25T00:01:19.494273+0000","last_became_active":"2024-04-25T00:01:04.357571+0000","last_became_peered":"2024-04-25T00:01:04.357571+0000","last_unstale":"2024-04-25T00:01:19.494273+0000","last_undegraded":"2024-04-25T00:01:19.494273+0000","last_fullsized":"2024-04-25T00:01:19.494273+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:50:23.023114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.520269+0000","last_change":"2024-04-25T00:01:18.497679+0000","last_active":"2024-04-25T00:01:20.520269+0000","last_peered":"2024-04-25T00:01:20.520269+0000","last_clean":"2024-04-25T00:01:20.520269+0000","last_became_active":"2024-04-25T00:01:18.496489+0000","last_became_peered":"2024-04-25T00:01:18.496489+0000","last_unstale":"2024-04-25T00:01:20.520269+0000","last_undegraded":"2024-04-25T00:01:20.520269+0000","last_fullsized":"2024-04-25T00:01:20.520269+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:23:47.845624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.584174+0000","last_change":"2024-04-25T00:01:19.498082+0000","last_active":"2024-04-25T00:01:20.584174+0000","last_peered":"2024-04-25T00:01:20.584174+0000","last_clean":"2024-04-25T00:01:20.584174+0000","last_became_active":"2024-04-25T00:01:19.496805+0000","last_became_peered":"2024-04-25T00:01:19.496805+0000","last_unstale":"2024-04-25T00:01:20.584174+0000","last_undegraded":"2024-04-25T00:01:20.584174+0000","last_fullsized":"2024-04-25T00:01:20.584174+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:57:36.256250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.577453+0000","last_change":"2024-04-25T00:01:05.373927+0000","last_active":"2024-04-25T00:01:20.577453+0000","last_peered":"2024-04-25T00:01:20.577453+0000","last_clean":"2024-04-25T00:01:20.577453+0000","last_became_active":"2024-04-25T00:01:05.373077+0000","last_became_peered":"2024-04-25T00:01:05.373077+0000","last_unstale":"2024-04-25T00:01:20.577453+0000","last_undegraded":"2024-04-25T00:01:20.577453+0000","last_fullsized":"2024-04-25T00:01:20.577453+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:50.489125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578902+0000","last_change":"2024-04-25T00:01:04.357265+0000","last_active":"2024-04-25T00:01:20.578902+0000","last_peered":"2024-04-25T00:01:20.578902+0000","last_clean":"2024-04-25T00:01:20.578902+0000","last_became_active":"2024-04-25T00:01:04.357004+0000","last_became_peered":"2024-04-25T00:01:04.357004+0000","last_unstale":"2024-04-25T00:01:20.578902+0000","last_undegraded":"2024-04-25T00:01:20.578902+0000","last_fullsized":"2024-04-25T00:01:20.578902+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:47:25.595275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.520612+0000","last_change":"2024-04-25T00:01:18.487276+0000","last_active":"2024-04-25T00:01:20.520612+0000","last_peered":"2024-04-25T00:01:20.520612+0000","last_clean":"2024-04-25T00:01:20.520612+0000","last_became_active":"2024-04-25T00:01:18.486279+0000","last_became_peered":"2024-04-25T00:01:18.486279+0000","last_unstale":"2024-04-25T00:01:20.520612+0000","last_undegraded":"2024-04-25T00:01:20.520612+0000","last_fullsized":"2024-04-25T00:01:20.520612+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:17:37.141843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582117+0000","last_change":"2024-04-25T00:01:19.516827+0000","last_active":"2024-04-25T00:01:20.582117+0000","last_peered":"2024-04-25T00:01:20.582117+0000","last_clean":"2024-04-25T00:01:20.582117+0000","last_became_active":"2024-04-25T00:01:19.515765+0000","last_became_peered":"2024-04-25T00:01:19.515765+0000","last_unstale":"2024-04-25T00:01:20.582117+0000","last_undegraded":"2024-04-25T00:01:20.582117+0000","last_fullsized":"2024-04-25T00:01:20.582117+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:13:34.153117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.579960+0000","last_change":"2024-04-25T00:01:05.379501+0000","last_active":"2024-04-25T00:01:20.579960+0000","last_peered":"2024-04-25T00:01:20.579960+0000","last_clean":"2024-04-25T00:01:20.579960+0000","last_became_active":"2024-04-25T00:01:05.378644+0000","last_became_peered":"2024-04-25T00:01:05.378644+0000","last_unstale":"2024-04-25T00:01:20.579960+0000","last_undegraded":"2024-04-25T00:01:20.579960+0000","last_fullsized":"2024-04-25T00:01:20.579960+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:08:35.505154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.580791+0000","last_change":"2024-04-25T00:01:04.360987+0000","last_active":"2024-04-25T00:01:20.580791+0000","last_peered":"2024-04-25T00:01:20.580791+0000","last_clean":"2024-04-25T00:01:20.580791+0000","last_became_active":"2024-04-25T00:01:04.360664+0000","last_became_peered":"2024-04-25T00:01:04.360664+0000","last_unstale":"2024-04-25T00:01:20.580791+0000","last_undegraded":"2024-04-25T00:01:20.580791+0000","last_fullsized":"2024-04-25T00:01:20.580791+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:55.283076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"26'6","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578827+0000","last_change":"2024-04-25T00:01:18.505577+0000","last_active":"2024-04-25T00:01:20.578827+0000","last_peered":"2024-04-25T00:01:20.578827+0000","last_clean":"2024-04-25T00:01:20.578827+0000","last_became_active":"2024-04-25T00:01:18.504963+0000","last_became_peered":"2024-04-25T00:01:18.504963+0000","last_unstale":"2024-04-25T00:01:20.578827+0000","last_undegraded":"2024-04-25T00:01:20.578827+0000","last_fullsized":"2024-04-25T00:01:20.578827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:37.142619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T00:01:19.541800+0000","last_change":"2024-04-25T00:01:19.524723+0000","last_active":"2024-04-25T00:01:19.541800+0000","last_peered":"2024-04-25T00:01:19.541800+0000","last_clean":"2024-04-25T00:01:19.541800+0000","last_became_active":"2024-04-25T00:01:19.523753+0000","last_became_peered":"2024-04-25T00:01:19.523753+0000","last_unstale":"2024-04-25T00:01:19.541800+0000","last_undegraded":"2024-04-25T00:01:19.541800+0000","last_fullsized":"2024-04-25T00:01:19.541800+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:37:16.164314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578487+0000","last_change":"2024-04-25T00:01:05.379102+0000","last_active":"2024-04-25T00:01:20.578487+0000","last_peered":"2024-04-25T00:01:20.578487+0000","last_clean":"2024-04-25T00:01:20.578487+0000","last_became_active":"2024-04-25T00:01:05.377625+0000","last_became_peered":"2024-04-25T00:01:05.377625+0000","last_unstale":"2024-04-25T00:01:20.578487+0000","last_undegraded":"2024-04-25T00:01:20.578487+0000","last_fullsized":"2024-04-25T00:01:20.578487+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:15.879673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"22'2","reported_seq":24,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511726+0000","last_change":"2024-04-25T00:01:04.370640+0000","last_active":"2024-04-25T00:01:20.511726+0000","last_peered":"2024-04-25T00:01:20.511726+0000","last_clean":"2024-04-25T00:01:20.511726+0000","last_became_active":"2024-04-25T00:01:04.370031+0000","last_became_peered":"2024-04-25T00:01:04.370031+0000","last_unstale":"2024-04-25T00:01:20.511726+0000","last_undegraded":"2024-04-25T00:01:20.511726+0000","last_fullsized":"2024-04-25T00:01:20.511726+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:26.243635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"26'3","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582955+0000","last_change":"2024-04-25T00:01:18.473113+0000","last_active":"2024-04-25T00:01:20.582955+0000","last_peered":"2024-04-25T00:01:20.582955+0000","last_clean":"2024-04-25T00:01:20.582955+0000","last_became_active":"2024-04-25T00:01:18.472931+0000","last_became_peered":"2024-04-25T00:01:18.472931+0000","last_unstale":"2024-04-25T00:01:20.582955+0000","last_undegraded":"2024-04-25T00:01:20.582955+0000","last_fullsized":"2024-04-25T00:01:20.582955+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:07.498985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T00:01:19.542557+0000","last_change":"2024-04-25T00:01:19.533664+0000","last_active":"2024-04-25T00:01:19.542557+0000","last_peered":"2024-04-25T00:01:19.542557+0000","last_clean":"2024-04-25T00:01:19.542557+0000","last_became_active":"2024-04-25T00:01:19.533048+0000","last_became_peered":"2024-04-25T00:01:19.533048+0000","last_unstale":"2024-04-25T00:01:19.542557+0000","last_undegraded":"2024-04-25T00:01:19.542557+0000","last_fullsized":"2024-04-25T00:01:19.542557+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:30:19.851252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582298+0000","last_change":"2024-04-25T00:01:05.385827+0000","last_active":"2024-04-25T00:01:20.582298+0000","last_peered":"2024-04-25T00:01:20.582298+0000","last_clean":"2024-04-25T00:01:20.582298+0000","last_became_active":"2024-04-25T00:01:05.385258+0000","last_became_peered":"2024-04-25T00:01:05.385258+0000","last_unstale":"2024-04-25T00:01:20.582298+0000","last_undegraded":"2024-04-25T00:01:20.582298+0000","last_fullsized":"2024-04-25T00:01:20.582298+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:23.870121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"22'6","reported_seq":28,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.512128+0000","last_change":"2024-04-25T00:01:04.365185+0000","last_active":"2024-04-25T00:01:20.512128+0000","last_peered":"2024-04-25T00:01:20.512128+0000","last_clean":"2024-04-25T00:01:20.512128+0000","last_became_active":"2024-04-25T00:01:04.364930+0000","last_became_peered":"2024-04-25T00:01:04.364930+0000","last_unstale":"2024-04-25T00:01:20.512128+0000","last_undegraded":"2024-04-25T00:01:20.512128+0000","last_fullsized":"2024-04-25T00:01:20.512128+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:45:03.618340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T00:01:19.496459+0000","last_change":"2024-04-25T00:01:18.496220+0000","last_active":"2024-04-25T00:01:19.496459+0000","last_peered":"2024-04-25T00:01:19.496459+0000","last_clean":"2024-04-25T00:01:19.496459+0000","last_became_active":"2024-04-25T00:01:18.492850+0000","last_became_peered":"2024-04-25T00:01:18.492850+0000","last_unstale":"2024-04-25T00:01:19.496459+0000","last_undegraded":"2024-04-25T00:01:19.496459+0000","last_fullsized":"2024-04-25T00:01:19.496459+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:50:16.922314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578332+0000","last_change":"2024-04-25T00:01:19.515725+0000","last_active":"2024-04-25T00:01:20.578332+0000","last_peered":"2024-04-25T00:01:20.578332+0000","last_clean":"2024-04-25T00:01:20.578332+0000","last_became_active":"2024-04-25T00:01:19.509285+0000","last_became_peered":"2024-04-25T00:01:19.509285+0000","last_unstale":"2024-04-25T00:01:20.578332+0000","last_undegraded":"2024-04-25T00:01:20.578332+0000","last_fullsized":"2024-04-25T00:01:20.578332+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:25:20.185355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.581530+0000","last_change":"2024-04-25T00:01:05.371061+0000","last_active":"2024-04-25T00:01:20.581530+0000","last_peered":"2024-04-25T00:01:20.581530+0000","last_clean":"2024-04-25T00:01:20.581530+0000","last_became_active":"2024-04-25T00:01:05.369950+0000","last_became_peered":"2024-04-25T00:01:05.369950+0000","last_unstale":"2024-04-25T00:01:20.581530+0000","last_undegraded":"2024-04-25T00:01:20.581530+0000","last_fullsized":"2024-04-25T00:01:20.581530+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:02:43.237902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"22'3","reported_seq":25,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.513687+0000","last_change":"2024-04-25T00:01:04.366503+0000","last_active":"2024-04-25T00:01:20.513687+0000","last_peered":"2024-04-25T00:01:20.513687+0000","last_clean":"2024-04-25T00:01:20.513687+0000","last_became_active":"2024-04-25T00:01:04.365945+0000","last_became_peered":"2024-04-25T00:01:04.365945+0000","last_unstale":"2024-04-25T00:01:20.513687+0000","last_undegraded":"2024-04-25T00:01:20.513687+0000","last_fullsized":"2024-04-25T00:01:20.513687+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"26'2","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511434+0000","last_change":"2024-04-25T00:01:18.476456+0000","last_active":"2024-04-25T00:01:20.511434+0000","last_peered":"2024-04-25T00:01:20.511434+0000","last_clean":"2024-04-25T00:01:20.511434+0000","last_became_active":"2024-04-25T00:01:18.475876+0000","last_became_peered":"2024-04-25T00:01:18.475876+0000","last_unstale":"2024-04-25T00:01:20.511434+0000","last_undegraded":"2024-04-25T00:01:20.511434+0000","last_fullsized":"2024-04-25T00:01:20.511434+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:56.741250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.580191+0000","last_change":"2024-04-25T00:01:19.515376+0000","last_active":"2024-04-25T00:01:20.580191+0000","last_peered":"2024-04-25T00:01:20.580191+0000","last_clean":"2024-04-25T00:01:20.580191+0000","last_became_active":"2024-04-25T00:01:19.512425+0000","last_became_peered":"2024-04-25T00:01:19.512425+0000","last_unstale":"2024-04-25T00:01:20.580191+0000","last_undegraded":"2024-04-25T00:01:20.580191+0000","last_fullsized":"2024-04-25T00:01:20.580191+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:43:16.957273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.520943+0000","last_change":"2024-04-25T00:01:05.361723+0000","last_active":"2024-04-25T00:01:20.520943+0000","last_peered":"2024-04-25T00:01:20.520943+0000","last_clean":"2024-04-25T00:01:20.520943+0000","last_became_active":"2024-04-25T00:01:05.360956+0000","last_became_peered":"2024-04-25T00:01:05.360956+0000","last_unstale":"2024-04-25T00:01:20.520943+0000","last_undegraded":"2024-04-25T00:01:20.520943+0000","last_fullsized":"2024-04-25T00:01:20.520943+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:23.870121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.580983+0000","last_change":"2024-04-25T00:01:04.362129+0000","last_active":"2024-04-25T00:01:20.580983+0000","last_peered":"2024-04-25T00:01:20.580983+0000","last_clean":"2024-04-25T00:01:20.580983+0000","last_became_active":"2024-04-25T00:01:04.361827+0000","last_became_peered":"2024-04-25T00:01:04.361827+0000","last_unstale":"2024-04-25T00:01:20.580983+0000","last_undegraded":"2024-04-25T00:01:20.580983+0000","last_fullsized":"2024-04-25T00:01:20.580983+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:26.243635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"26'8","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.510624+0000","last_change":"2024-04-25T00:01:18.496129+0000","last_active":"2024-04-25T00:01:20.510624+0000","last_peered":"2024-04-25T00:01:20.510624+0000","last_clean":"2024-04-25T00:01:20.510624+0000","last_became_active":"2024-04-25T00:01:18.495415+0000","last_became_peered":"2024-04-25T00:01:18.495415+0000","last_unstale":"2024-04-25T00:01:20.510624+0000","last_undegraded":"2024-04-25T00:01:20.510624+0000","last_fullsized":"2024-04-25T00:01:20.510624+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:38.089411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.510763+0000","last_change":"2024-04-25T00:01:19.509542+0000","last_active":"2024-04-25T00:01:20.510763+0000","last_peered":"2024-04-25T00:01:20.510763+0000","last_clean":"2024-04-25T00:01:20.510763+0000","last_became_active":"2024-04-25T00:01:19.509015+0000","last_became_peered":"2024-04-25T00:01:19.509015+0000","last_unstale":"2024-04-25T00:01:20.510763+0000","last_undegraded":"2024-04-25T00:01:20.510763+0000","last_fullsized":"2024-04-25T00:01:20.510763+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:25:20.185355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511808+0000","last_change":"2024-04-25T00:01:05.365413+0000","last_active":"2024-04-25T00:01:20.511808+0000","last_peered":"2024-04-25T00:01:20.511808+0000","last_clean":"2024-04-25T00:01:20.511808+0000","last_became_active":"2024-04-25T00:01:05.361865+0000","last_became_peered":"2024-04-25T00:01:05.361865+0000","last_unstale":"2024-04-25T00:01:20.511808+0000","last_undegraded":"2024-04-25T00:01:20.511808+0000","last_fullsized":"2024-04-25T00:01:20.511808+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:30.228060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582499+0000","last_change":"2024-04-25T00:01:04.360804+0000","last_active":"2024-04-25T00:01:20.582499+0000","last_peered":"2024-04-25T00:01:20.582499+0000","last_clean":"2024-04-25T00:01:20.582499+0000","last_became_active":"2024-04-25T00:01:04.360374+0000","last_became_peered":"2024-04-25T00:01:04.360374+0000","last_unstale":"2024-04-25T00:01:20.582499+0000","last_undegraded":"2024-04-25T00:01:20.582499+0000","last_fullsized":"2024-04-25T00:01:20.582499+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T00:01:19.494208+0000","last_change":"2024-04-25T00:01:18.480003+0000","last_active":"2024-04-25T00:01:19.494208+0000","last_peered":"2024-04-25T00:01:19.494208+0000","last_clean":"2024-04-25T00:01:19.494208+0000","last_became_active":"2024-04-25T00:01:18.479270+0000","last_became_peered":"2024-04-25T00:01:18.479270+0000","last_unstale":"2024-04-25T00:01:19.494208+0000","last_undegraded":"2024-04-25T00:01:19.494208+0000","last_fullsized":"2024-04-25T00:01:19.494208+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:52:40.942502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511885+0000","last_change":"2024-04-25T00:01:19.524463+0000","last_active":"2024-04-25T00:01:20.511885+0000","last_peered":"2024-04-25T00:01:20.511885+0000","last_clean":"2024-04-25T00:01:20.511885+0000","last_became_active":"2024-04-25T00:01:19.524166+0000","last_became_peered":"2024-04-25T00:01:19.524166+0000","last_unstale":"2024-04-25T00:01:20.511885+0000","last_undegraded":"2024-04-25T00:01:20.511885+0000","last_fullsized":"2024-04-25T00:01:20.511885+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:50.351599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"26'1","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T00:01:19.542913+0000","last_change":"2024-04-25T00:01:18.497096+0000","last_active":"2024-04-25T00:01:19.542913+0000","last_peered":"2024-04-25T00:01:19.542913+0000","last_clean":"2024-04-25T00:01:19.542913+0000","last_became_active":"2024-04-25T00:01:18.496289+0000","last_became_peered":"2024-04-25T00:01:18.496289+0000","last_unstale":"2024-04-25T00:01:19.542913+0000","last_undegraded":"2024-04-25T00:01:19.542913+0000","last_fullsized":"2024-04-25T00:01:19.542913+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:59:40.316831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T00:01:19.494675+0000","last_change":"2024-04-25T00:01:05.370043+0000","last_active":"2024-04-25T00:01:19.494675+0000","last_peered":"2024-04-25T00:01:19.494675+0000","last_clean":"2024-04-25T00:01:19.494675+0000","last_became_active":"2024-04-25T00:01:05.368080+0000","last_became_peered":"2024-04-25T00:01:05.368080+0000","last_unstale":"2024-04-25T00:01:19.494675+0000","last_undegraded":"2024-04-25T00:01:19.494675+0000","last_fullsized":"2024-04-25T00:01:19.494675+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:58:07.610923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.580256+0000","last_change":"2024-04-25T00:01:04.364523+0000","last_active":"2024-04-25T00:01:20.580256+0000","last_peered":"2024-04-25T00:01:20.580256+0000","last_clean":"2024-04-25T00:01:20.580256+0000","last_became_active":"2024-04-25T00:01:04.364050+0000","last_became_peered":"2024-04-25T00:01:04.364050+0000","last_unstale":"2024-04-25T00:01:20.580256+0000","last_undegraded":"2024-04-25T00:01:20.580256+0000","last_fullsized":"2024-04-25T00:01:20.580256+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:49.479804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.579360+0000","last_change":"2024-04-25T00:01:19.491523+0000","last_active":"2024-04-25T00:01:20.579360+0000","last_peered":"2024-04-25T00:01:20.579360+0000","last_clean":"2024-04-25T00:01:20.579360+0000","last_became_active":"2024-04-25T00:01:19.490755+0000","last_became_peered":"2024-04-25T00:01:19.490755+0000","last_unstale":"2024-04-25T00:01:20.579360+0000","last_undegraded":"2024-04-25T00:01:20.579360+0000","last_fullsized":"2024-04-25T00:01:20.579360+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:47:19.526845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"26'4","reported_seq":18,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.584296+0000","last_change":"2024-04-25T00:01:18.472794+0000","last_active":"2024-04-25T00:01:20.584296+0000","last_peered":"2024-04-25T00:01:20.584296+0000","last_clean":"2024-04-25T00:01:20.584296+0000","last_became_active":"2024-04-25T00:01:18.472534+0000","last_became_peered":"2024-04-25T00:01:18.472534+0000","last_unstale":"2024-04-25T00:01:20.584296+0000","last_undegraded":"2024-04-25T00:01:20.584296+0000","last_fullsized":"2024-04-25T00:01:20.584296+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:55:45.843808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.518835+0000","last_change":"2024-04-25T00:01:05.378867+0000","last_active":"2024-04-25T00:01:20.518835+0000","last_peered":"2024-04-25T00:01:20.518835+0000","last_clean":"2024-04-25T00:01:20.518835+0000","last_became_active":"2024-04-25T00:01:05.378056+0000","last_became_peered":"2024-04-25T00:01:05.378056+0000","last_unstale":"2024-04-25T00:01:20.518835+0000","last_undegraded":"2024-04-25T00:01:20.518835+0000","last_fullsized":"2024-04-25T00:01:20.518835+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:36:10.377757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.577806+0000","last_change":"2024-04-25T00:01:04.370896+0000","last_active":"2024-04-25T00:01:20.577806+0000","last_peered":"2024-04-25T00:01:20.577806+0000","last_clean":"2024-04-25T00:01:20.577806+0000","last_became_active":"2024-04-25T00:01:04.370273+0000","last_became_peered":"2024-04-25T00:01:04.370273+0000","last_unstale":"2024-04-25T00:01:20.577806+0000","last_undegraded":"2024-04-25T00:01:20.577806+0000","last_fullsized":"2024-04-25T00:01:20.577806+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:49:22.673421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T00:01:19.541208+0000","last_change":"2024-04-25T00:01:19.521143+0000","last_active":"2024-04-25T00:01:19.541208+0000","last_peered":"2024-04-25T00:01:19.541208+0000","last_clean":"2024-04-25T00:01:19.541208+0000","last_became_active":"2024-04-25T00:01:19.514664+0000","last_became_peered":"2024-04-25T00:01:19.514664+0000","last_unstale":"2024-04-25T00:01:19.541208+0000","last_undegraded":"2024-04-25T00:01:19.541208+0000","last_fullsized":"2024-04-25T00:01:19.541208+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:43:14.020286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"22'1","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.580606+0000","last_change":"2024-04-25T00:01:04.360254+0000","last_active":"2024-04-25T00:01:20.580606+0000","last_peered":"2024-04-25T00:01:20.580606+0000","last_clean":"2024-04-25T00:01:20.580606+0000","last_became_active":"2024-04-25T00:01:04.359586+0000","last_became_peered":"2024-04-25T00:01:04.359586+0000","last_unstale":"2024-04-25T00:01:20.580606+0000","last_undegraded":"2024-04-25T00:01:20.580606+0000","last_fullsized":"2024-04-25T00:01:20.580606+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:47.192930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T00:01:19.500989+0000","last_change":"2024-04-25T00:01:05.370723+0000","last_active":"2024-04-25T00:01:19.500989+0000","last_peered":"2024-04-25T00:01:19.500989+0000","last_clean":"2024-04-25T00:01:19.500989+0000","last_became_active":"2024-04-25T00:01:05.368970+0000","last_became_peered":"2024-04-25T00:01:05.368970+0000","last_unstale":"2024-04-25T00:01:19.500989+0000","last_undegraded":"2024-04-25T00:01:19.500989+0000","last_fullsized":"2024-04-25T00:01:19.500989+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:06:53.743074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"26'2","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T00:01:19.546428+0000","last_change":"2024-04-25T00:01:18.498570+0000","last_active":"2024-04-25T00:01:19.546428+0000","last_peered":"2024-04-25T00:01:19.546428+0000","last_clean":"2024-04-25T00:01:19.546428+0000","last_became_active":"2024-04-25T00:01:18.497745+0000","last_became_peered":"2024-04-25T00:01:18.497745+0000","last_unstale":"2024-04-25T00:01:19.546428+0000","last_undegraded":"2024-04-25T00:01:19.546428+0000","last_fullsized":"2024-04-25T00:01:19.546428+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:26:42.047149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511888+0000","last_change":"2024-04-25T00:01:19.514919+0000","last_active":"2024-04-25T00:01:20.511888+0000","last_peered":"2024-04-25T00:01:20.511888+0000","last_clean":"2024-04-25T00:01:20.511888+0000","last_became_active":"2024-04-25T00:01:19.514280+0000","last_became_peered":"2024-04-25T00:01:19.514280+0000","last_unstale":"2024-04-25T00:01:20.511888+0000","last_undegraded":"2024-04-25T00:01:20.511888+0000","last_fullsized":"2024-04-25T00:01:20.511888+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:37.975265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"22'1","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.519944+0000","last_change":"2024-04-25T00:01:04.365607+0000","last_active":"2024-04-25T00:01:20.519944+0000","last_peered":"2024-04-25T00:01:20.519944+0000","last_clean":"2024-04-25T00:01:20.519944+0000","last_became_active":"2024-04-25T00:01:04.364788+0000","last_became_peered":"2024-04-25T00:01:04.364788+0000","last_unstale":"2024-04-25T00:01:20.519944+0000","last_undegraded":"2024-04-25T00:01:20.519944+0000","last_fullsized":"2024-04-25T00:01:20.519944+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.579248+0000","last_change":"2024-04-25T00:01:05.375142+0000","last_active":"2024-04-25T00:01:20.579248+0000","last_peered":"2024-04-25T00:01:20.579248+0000","last_clean":"2024-04-25T00:01:20.579248+0000","last_became_active":"2024-04-25T00:01:05.373857+0000","last_became_peered":"2024-04-25T00:01:05.373857+0000","last_unstale":"2024-04-25T00:01:20.579248+0000","last_undegraded":"2024-04-25T00:01:20.579248+0000","last_fullsized":"2024-04-25T00:01:20.579248+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:15.879673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"26'2","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.513534+0000","last_change":"2024-04-25T00:01:18.502786+0000","last_active":"2024-04-25T00:01:20.513534+0000","last_peered":"2024-04-25T00:01:20.513534+0000","last_clean":"2024-04-25T00:01:20.513534+0000","last_became_active":"2024-04-25T00:01:18.501785+0000","last_became_peered":"2024-04-25T00:01:18.501785+0000","last_unstale":"2024-04-25T00:01:20.513534+0000","last_undegraded":"2024-04-25T00:01:20.513534+0000","last_fullsized":"2024-04-25T00:01:20.513534+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:09:31.441044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":9,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T00:01:19.537533+0000","last_change":"2024-04-25T00:01:19.511239+0000","last_active":"2024-04-25T00:01:19.537533+0000","last_peered":"2024-04-25T00:01:19.537533+0000","last_clean":"2024-04-25T00:01:19.537533+0000","last_became_active":"2024-04-25T00:01:19.510155+0000","last_became_peered":"2024-04-25T00:01:19.510155+0000","last_unstale":"2024-04-25T00:01:19.537533+0000","last_undegraded":"2024-04-25T00:01:19.537533+0000","last_fullsized":"2024-04-25T00:01:19.537533+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:33:56.057791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511273+0000","last_change":"2024-04-25T00:01:05.378517+0000","last_active":"2024-04-25T00:01:20.511273+0000","last_peered":"2024-04-25T00:01:20.511273+0000","last_clean":"2024-04-25T00:01:20.511273+0000","last_became_active":"2024-04-25T00:01:05.377742+0000","last_became_peered":"2024-04-25T00:01:05.377742+0000","last_unstale":"2024-04-25T00:01:20.511273+0000","last_undegraded":"2024-04-25T00:01:20.511273+0000","last_fullsized":"2024-04-25T00:01:20.511273+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:06.450369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":9,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T00:01:19.537322+0000","last_change":"2024-04-25T00:01:19.508999+0000","last_active":"2024-04-25T00:01:19.537322+0000","last_peered":"2024-04-25T00:01:19.537322+0000","last_clean":"2024-04-25T00:01:19.537322+0000","last_became_active":"2024-04-25T00:01:19.507960+0000","last_became_peered":"2024-04-25T00:01:19.507960+0000","last_unstale":"2024-04-25T00:01:19.537322+0000","last_undegraded":"2024-04-25T00:01:19.537322+0000","last_fullsized":"2024-04-25T00:01:19.537322+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:48:00.647773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.510941+0000","last_change":"2024-04-25T00:01:05.372927+0000","last_active":"2024-04-25T00:01:20.510941+0000","last_peered":"2024-04-25T00:01:20.510941+0000","last_clean":"2024-04-25T00:01:20.510941+0000","last_became_active":"2024-04-25T00:01:05.372050+0000","last_became_peered":"2024-04-25T00:01:05.372050+0000","last_unstale":"2024-04-25T00:01:20.510941+0000","last_undegraded":"2024-04-25T00:01:20.510941+0000","last_fullsized":"2024-04-25T00:01:20.510941+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:48.202251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511854+0000","last_change":"2024-04-25T00:01:19.531072+0000","last_active":"2024-04-25T00:01:20.511854+0000","last_peered":"2024-04-25T00:01:20.511854+0000","last_clean":"2024-04-25T00:01:20.511854+0000","last_became_active":"2024-04-25T00:01:19.530836+0000","last_became_peered":"2024-04-25T00:01:19.530836+0000","last_unstale":"2024-04-25T00:01:20.511854+0000","last_undegraded":"2024-04-25T00:01:20.511854+0000","last_fullsized":"2024-04-25T00:01:20.511854+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:08.508741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.580209+0000","last_change":"2024-04-25T00:01:05.362960+0000","last_active":"2024-04-25T00:01:20.580209+0000","last_peered":"2024-04-25T00:01:20.580209+0000","last_clean":"2024-04-25T00:01:20.580209+0000","last_became_active":"2024-04-25T00:01:05.361067+0000","last_became_peered":"2024-04-25T00:01:05.361067+0000","last_unstale":"2024-04-25T00:01:20.580209+0000","last_undegraded":"2024-04-25T00:01:20.580209+0000","last_fullsized":"2024-04-25T00:01:20.580209+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:24.017263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.520235+0000","last_change":"2024-04-25T00:01:19.506029+0000","last_active":"2024-04-25T00:01:20.520235+0000","last_peered":"2024-04-25T00:01:20.520235+0000","last_clean":"2024-04-25T00:01:20.520235+0000","last_became_active":"2024-04-25T00:01:19.505184+0000","last_became_peered":"2024-04-25T00:01:19.505184+0000","last_unstale":"2024-04-25T00:01:20.520235+0000","last_undegraded":"2024-04-25T00:01:20.520235+0000","last_fullsized":"2024-04-25T00:01:20.520235+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:08.508741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.519736+0000","last_change":"2024-04-25T00:01:05.358690+0000","last_active":"2024-04-25T00:01:20.519736+0000","last_peered":"2024-04-25T00:01:20.519736+0000","last_clean":"2024-04-25T00:01:20.519736+0000","last_became_active":"2024-04-25T00:01:05.357340+0000","last_became_peered":"2024-04-25T00:01:05.357340+0000","last_unstale":"2024-04-25T00:01:20.519736+0000","last_undegraded":"2024-04-25T00:01:20.519736+0000","last_fullsized":"2024-04-25T00:01:20.519736+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:36:55.436661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.577993+0000","last_change":"2024-04-25T00:01:19.508691+0000","last_active":"2024-04-25T00:01:20.577993+0000","last_peered":"2024-04-25T00:01:20.577993+0000","last_clean":"2024-04-25T00:01:20.577993+0000","last_became_active":"2024-04-25T00:01:19.508123+0000","last_became_peered":"2024-04-25T00:01:19.508123+0000","last_unstale":"2024-04-25T00:01:20.577993+0000","last_undegraded":"2024-04-25T00:01:20.577993+0000","last_fullsized":"2024-04-25T00:01:20.577993+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:47:19.526845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578984+0000","last_change":"2024-04-25T00:01:05.387057+0000","last_active":"2024-04-25T00:01:20.578984+0000","last_peered":"2024-04-25T00:01:20.578984+0000","last_clean":"2024-04-25T00:01:20.578984+0000","last_became_active":"2024-04-25T00:01:05.386724+0000","last_became_peered":"2024-04-25T00:01:05.386724+0000","last_unstale":"2024-04-25T00:01:20.578984+0000","last_undegraded":"2024-04-25T00:01:20.578984+0000","last_fullsized":"2024-04-25T00:01:20.578984+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:55:09.073286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.580476+0000","last_change":"2024-04-25T00:01:19.522760+0000","last_active":"2024-04-25T00:01:20.580476+0000","last_peered":"2024-04-25T00:01:20.580476+0000","last_clean":"2024-04-25T00:01:20.580476+0000","last_became_active":"2024-04-25T00:01:19.522380+0000","last_became_peered":"2024-04-25T00:01:19.522380+0000","last_unstale":"2024-04-25T00:01:20.580476+0000","last_undegraded":"2024-04-25T00:01:20.580476+0000","last_fullsized":"2024-04-25T00:01:20.580476+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:55:46.853565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578828+0000","last_change":"2024-04-25T00:01:05.367296+0000","last_active":"2024-04-25T00:01:20.578828+0000","last_peered":"2024-04-25T00:01:20.578828+0000","last_clean":"2024-04-25T00:01:20.578828+0000","last_became_active":"2024-04-25T00:01:05.363838+0000","last_became_peered":"2024-04-25T00:01:05.363838+0000","last_unstale":"2024-04-25T00:01:20.578828+0000","last_undegraded":"2024-04-25T00:01:20.578828+0000","last_fullsized":"2024-04-25T00:01:20.578828+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:43:07.053074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.579599+0000","last_change":"2024-04-25T00:01:19.510902+0000","last_active":"2024-04-25T00:01:20.579599+0000","last_peered":"2024-04-25T00:01:20.579599+0000","last_clean":"2024-04-25T00:01:20.579599+0000","last_became_active":"2024-04-25T00:01:19.510008+0000","last_became_peered":"2024-04-25T00:01:19.510008+0000","last_unstale":"2024-04-25T00:01:20.579599+0000","last_undegraded":"2024-04-25T00:01:20.579599+0000","last_fullsized":"2024-04-25T00:01:20.579599+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:40:32.292140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511704+0000","last_change":"2024-04-25T00:01:05.365761+0000","last_active":"2024-04-25T00:01:20.511704+0000","last_peered":"2024-04-25T00:01:20.511704+0000","last_clean":"2024-04-25T00:01:20.511704+0000","last_became_active":"2024-04-25T00:01:05.361119+0000","last_became_peered":"2024-04-25T00:01:05.361119+0000","last_unstale":"2024-04-25T00:01:20.511704+0000","last_undegraded":"2024-04-25T00:01:20.511704+0000","last_fullsized":"2024-04-25T00:01:20.511704+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.510983+0000","last_change":"2024-04-25T00:01:19.518534+0000","last_active":"2024-04-25T00:01:20.510983+0000","last_peered":"2024-04-25T00:01:20.510983+0000","last_clean":"2024-04-25T00:01:20.510983+0000","last_became_active":"2024-04-25T00:01:19.518110+0000","last_became_peered":"2024-04-25T00:01:19.518110+0000","last_unstale":"2024-04-25T00:01:20.510983+0000","last_undegraded":"2024-04-25T00:01:20.510983+0000","last_fullsized":"2024-04-25T00:01:20.510983+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:05:09.219823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.520770+0000","last_change":"2024-04-25T00:01:05.373464+0000","last_active":"2024-04-25T00:01:20.520770+0000","last_peered":"2024-04-25T00:01:20.520770+0000","last_clean":"2024-04-25T00:01:20.520770+0000","last_became_active":"2024-04-25T00:01:05.372611+0000","last_became_peered":"2024-04-25T00:01:05.372611+0000","last_unstale":"2024-04-25T00:01:20.520770+0000","last_undegraded":"2024-04-25T00:01:20.520770+0000","last_fullsized":"2024-04-25T00:01:20.520770+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:31.456792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.512974+0000","last_change":"2024-04-25T00:01:19.542475+0000","last_active":"2024-04-25T00:01:20.512974+0000","last_peered":"2024-04-25T00:01:20.512974+0000","last_clean":"2024-04-25T00:01:20.512974+0000","last_became_active":"2024-04-25T00:01:19.541877+0000","last_became_peered":"2024-04-25T00:01:19.541877+0000","last_unstale":"2024-04-25T00:01:20.512974+0000","last_undegraded":"2024-04-25T00:01:20.512974+0000","last_fullsized":"2024-04-25T00:01:20.512974+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:02:57.373014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.580023+0000","last_change":"2024-04-25T00:01:05.372906+0000","last_active":"2024-04-25T00:01:20.580023+0000","last_peered":"2024-04-25T00:01:20.580023+0000","last_clean":"2024-04-25T00:01:20.580023+0000","last_became_active":"2024-04-25T00:01:05.372148+0000","last_became_peered":"2024-04-25T00:01:05.372148+0000","last_unstale":"2024-04-25T00:01:20.580023+0000","last_undegraded":"2024-04-25T00:01:20.580023+0000","last_fullsized":"2024-04-25T00:01:20.580023+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:43.615895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.584868+0000","last_change":"2024-04-25T00:01:19.498515+0000","last_active":"2024-04-25T00:01:20.584868+0000","last_peered":"2024-04-25T00:01:20.584868+0000","last_clean":"2024-04-25T00:01:20.584868+0000","last_became_active":"2024-04-25T00:01:19.496310+0000","last_became_peered":"2024-04-25T00:01:19.496310+0000","last_unstale":"2024-04-25T00:01:20.584868+0000","last_undegraded":"2024-04-25T00:01:20.584868+0000","last_fullsized":"2024-04-25T00:01:20.584868+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:20:06.661558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.519325+0000","last_change":"2024-04-25T00:01:05.366340+0000","last_active":"2024-04-25T00:01:20.519325+0000","last_peered":"2024-04-25T00:01:20.519325+0000","last_clean":"2024-04-25T00:01:20.519325+0000","last_became_active":"2024-04-25T00:01:05.361952+0000","last_became_peered":"2024-04-25T00:01:05.361952+0000","last_unstale":"2024-04-25T00:01:20.519325+0000","last_undegraded":"2024-04-25T00:01:20.519325+0000","last_fullsized":"2024-04-25T00:01:20.519325+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:41.948662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578558+0000","last_change":"2024-04-25T00:01:19.508257+0000","last_active":"2024-04-25T00:01:20.578558+0000","last_peered":"2024-04-25T00:01:20.578558+0000","last_clean":"2024-04-25T00:01:20.578558+0000","last_became_active":"2024-04-25T00:01:19.507516+0000","last_became_peered":"2024-04-25T00:01:19.507516+0000","last_unstale":"2024-04-25T00:01:20.578558+0000","last_undegraded":"2024-04-25T00:01:20.578558+0000","last_fullsized":"2024-04-25T00:01:20.578558+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:45.121513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.579066+0000","last_change":"2024-04-25T00:01:05.387457+0000","last_active":"2024-04-25T00:01:20.579066+0000","last_peered":"2024-04-25T00:01:20.579066+0000","last_clean":"2024-04-25T00:01:20.579066+0000","last_became_active":"2024-04-25T00:01:05.386919+0000","last_became_peered":"2024-04-25T00:01:05.386919+0000","last_unstale":"2024-04-25T00:01:20.579066+0000","last_undegraded":"2024-04-25T00:01:20.579066+0000","last_fullsized":"2024-04-25T00:01:20.579066+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.512398+0000","last_change":"2024-04-25T00:01:19.520648+0000","last_active":"2024-04-25T00:01:20.512398+0000","last_peered":"2024-04-25T00:01:20.512398+0000","last_clean":"2024-04-25T00:01:20.512398+0000","last_became_active":"2024-04-25T00:01:19.519303+0000","last_became_peered":"2024-04-25T00:01:19.519303+0000","last_unstale":"2024-04-25T00:01:20.512398+0000","last_undegraded":"2024-04-25T00:01:20.512398+0000","last_fullsized":"2024-04-25T00:01:20.512398+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:23.993729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T00:01:19.500747+0000","last_change":"2024-04-25T00:01:05.366249+0000","last_active":"2024-04-25T00:01:19.500747+0000","last_peered":"2024-04-25T00:01:19.500747+0000","last_clean":"2024-04-25T00:01:19.500747+0000","last_became_active":"2024-04-25T00:01:05.365548+0000","last_became_peered":"2024-04-25T00:01:05.365548+0000","last_unstale":"2024-04-25T00:01:19.500747+0000","last_undegraded":"2024-04-25T00:01:19.500747+0000","last_fullsized":"2024-04-25T00:01:19.500747+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:22:21.143639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.510804+0000","last_change":"2024-04-25T00:01:19.514659+0000","last_active":"2024-04-25T00:01:20.510804+0000","last_peered":"2024-04-25T00:01:20.510804+0000","last_clean":"2024-04-25T00:01:20.510804+0000","last_became_active":"2024-04-25T00:01:19.514232+0000","last_became_peered":"2024-04-25T00:01:19.514232+0000","last_unstale":"2024-04-25T00:01:20.510804+0000","last_undegraded":"2024-04-25T00:01:20.510804+0000","last_fullsized":"2024-04-25T00:01:20.510804+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:45.121513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582289+0000","last_change":"2024-04-25T00:01:05.385253+0000","last_active":"2024-04-25T00:01:20.582289+0000","last_peered":"2024-04-25T00:01:20.582289+0000","last_clean":"2024-04-25T00:01:20.582289+0000","last_became_active":"2024-04-25T00:01:05.379429+0000","last_became_peered":"2024-04-25T00:01:05.379429+0000","last_unstale":"2024-04-25T00:01:20.582289+0000","last_undegraded":"2024-04-25T00:01:20.582289+0000","last_fullsized":"2024-04-25T00:01:20.582289+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:36:10.377757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.581087+0000","last_change":"2024-04-25T00:01:19.503419+0000","last_active":"2024-04-25T00:01:20.581087+0000","last_peered":"2024-04-25T00:01:20.581087+0000","last_clean":"2024-04-25T00:01:20.581087+0000","last_became_active":"2024-04-25T00:01:19.500568+0000","last_became_peered":"2024-04-25T00:01:19.500568+0000","last_unstale":"2024-04-25T00:01:20.581087+0000","last_undegraded":"2024-04-25T00:01:20.581087+0000","last_fullsized":"2024-04-25T00:01:20.581087+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:08.508741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.583587+0000","last_change":"2024-04-25T00:01:05.370653+0000","last_active":"2024-04-25T00:01:20.583587+0000","last_peered":"2024-04-25T00:01:20.583587+0000","last_clean":"2024-04-25T00:01:20.583587+0000","last_became_active":"2024-04-25T00:01:05.369919+0000","last_became_peered":"2024-04-25T00:01:05.369919+0000","last_unstale":"2024-04-25T00:01:20.583587+0000","last_undegraded":"2024-04-25T00:01:20.583587+0000","last_fullsized":"2024-04-25T00:01:20.583587+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:30:08.311326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.510560+0000","last_change":"2024-04-25T00:01:19.540104+0000","last_active":"2024-04-25T00:01:20.510560+0000","last_peered":"2024-04-25T00:01:20.510560+0000","last_clean":"2024-04-25T00:01:20.510560+0000","last_became_active":"2024-04-25T00:01:19.539775+0000","last_became_peered":"2024-04-25T00:01:19.539775+0000","last_unstale":"2024-04-25T00:01:20.510560+0000","last_undegraded":"2024-04-25T00:01:20.510560+0000","last_fullsized":"2024-04-25T00:01:20.510560+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:53:14.768520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.514367+0000","last_change":"2024-04-25T00:01:05.367001+0000","last_active":"2024-04-25T00:01:20.514367+0000","last_peered":"2024-04-25T00:01:20.514367+0000","last_clean":"2024-04-25T00:01:20.514367+0000","last_became_active":"2024-04-25T00:01:05.366264+0000","last_became_peered":"2024-04-25T00:01:05.366264+0000","last_unstale":"2024-04-25T00:01:20.514367+0000","last_undegraded":"2024-04-25T00:01:20.514367+0000","last_fullsized":"2024-04-25T00:01:20.514367+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578038+0000","last_change":"2024-04-25T00:01:19.497247+0000","last_active":"2024-04-25T00:01:20.578038+0000","last_peered":"2024-04-25T00:01:20.578038+0000","last_clean":"2024-04-25T00:01:20.578038+0000","last_became_active":"2024-04-25T00:01:19.496517+0000","last_became_peered":"2024-04-25T00:01:19.496517+0000","last_unstale":"2024-04-25T00:01:20.578038+0000","last_undegraded":"2024-04-25T00:01:20.578038+0000","last_fullsized":"2024-04-25T00:01:20.578038+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:30.129056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.584723+0000","last_change":"2024-04-25T00:01:05.379958+0000","last_active":"2024-04-25T00:01:20.584723+0000","last_peered":"2024-04-25T00:01:20.584723+0000","last_clean":"2024-04-25T00:01:20.584723+0000","last_became_active":"2024-04-25T00:01:05.379090+0000","last_became_peered":"2024-04-25T00:01:05.379090+0000","last_unstale":"2024-04-25T00:01:20.584723+0000","last_undegraded":"2024-04-25T00:01:20.584723+0000","last_fullsized":"2024-04-25T00:01:20.584723+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:59:50.832958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":9,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T00:01:19.543038+0000","last_change":"2024-04-25T00:01:19.530176+0000","last_active":"2024-04-25T00:01:19.543038+0000","last_peered":"2024-04-25T00:01:19.543038+0000","last_clean":"2024-04-25T00:01:19.543038+0000","last_became_active":"2024-04-25T00:01:19.529410+0000","last_became_peered":"2024-04-25T00:01:19.529410+0000","last_unstale":"2024-04-25T00:01:19.543038+0000","last_undegraded":"2024-04-25T00:01:19.543038+0000","last_fullsized":"2024-04-25T00:01:19.543038+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:24:44.480639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.579746+0000","last_change":"2024-04-25T00:01:05.370826+0000","last_active":"2024-04-25T00:01:20.579746+0000","last_peered":"2024-04-25T00:01:20.579746+0000","last_clean":"2024-04-25T00:01:20.579746+0000","last_became_active":"2024-04-25T00:01:05.369848+0000","last_became_peered":"2024-04-25T00:01:05.369848+0000","last_unstale":"2024-04-25T00:01:20.579746+0000","last_undegraded":"2024-04-25T00:01:20.579746+0000","last_fullsized":"2024-04-25T00:01:20.579746+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:23.870121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.581888+0000","last_change":"2024-04-25T00:01:19.511625+0000","last_active":"2024-04-25T00:01:20.581888+0000","last_peered":"2024-04-25T00:01:20.581888+0000","last_clean":"2024-04-25T00:01:20.581888+0000","last_became_active":"2024-04-25T00:01:19.511349+0000","last_became_peered":"2024-04-25T00:01:19.511349+0000","last_unstale":"2024-04-25T00:01:20.581888+0000","last_undegraded":"2024-04-25T00:01:20.581888+0000","last_fullsized":"2024-04-25T00:01:20.581888+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:02:57.373014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.518885+0000","last_change":"2024-04-25T00:01:05.390341+0000","last_active":"2024-04-25T00:01:20.518885+0000","last_peered":"2024-04-25T00:01:20.518885+0000","last_clean":"2024-04-25T00:01:20.518885+0000","last_became_active":"2024-04-25T00:01:05.389707+0000","last_became_peered":"2024-04-25T00:01:05.389707+0000","last_unstale":"2024-04-25T00:01:20.518885+0000","last_undegraded":"2024-04-25T00:01:20.518885+0000","last_fullsized":"2024-04-25T00:01:20.518885+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:54:21.138947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578289+0000","last_change":"2024-04-25T00:01:19.513630+0000","last_active":"2024-04-25T00:01:20.578289+0000","last_peered":"2024-04-25T00:01:20.578289+0000","last_clean":"2024-04-25T00:01:20.578289+0000","last_became_active":"2024-04-25T00:01:19.512978+0000","last_became_peered":"2024-04-25T00:01:19.512978+0000","last_unstale":"2024-04-25T00:01:20.578289+0000","last_undegraded":"2024-04-25T00:01:20.578289+0000","last_fullsized":"2024-04-25T00:01:20.578289+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:57.751007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578080+0000","last_change":"2024-04-25T00:01:05.378373+0000","last_active":"2024-04-25T00:01:20.578080+0000","last_peered":"2024-04-25T00:01:20.578080+0000","last_clean":"2024-04-25T00:01:20.578080+0000","last_became_active":"2024-04-25T00:01:05.376570+0000","last_became_peered":"2024-04-25T00:01:05.376570+0000","last_unstale":"2024-04-25T00:01:20.578080+0000","last_undegraded":"2024-04-25T00:01:20.578080+0000","last_fullsized":"2024-04-25T00:01:20.578080+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:30.228060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582484+0000","last_change":"2024-04-25T00:01:19.516716+0000","last_active":"2024-04-25T00:01:20.582484+0000","last_peered":"2024-04-25T00:01:20.582484+0000","last_clean":"2024-04-25T00:01:20.582484+0000","last_became_active":"2024-04-25T00:01:19.516142+0000","last_became_peered":"2024-04-25T00:01:19.516142+0000","last_unstale":"2024-04-25T00:01:20.582484+0000","last_undegraded":"2024-04-25T00:01:20.582484+0000","last_fullsized":"2024-04-25T00:01:20.582484+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:28:18.234546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578828+0000","last_change":"2024-04-25T00:01:05.381011+0000","last_active":"2024-04-25T00:01:20.578828+0000","last_peered":"2024-04-25T00:01:20.578828+0000","last_clean":"2024-04-25T00:01:20.578828+0000","last_became_active":"2024-04-25T00:01:05.380236+0000","last_became_peered":"2024-04-25T00:01:05.380236+0000","last_unstale":"2024-04-25T00:01:20.578828+0000","last_undegraded":"2024-04-25T00:01:20.578828+0000","last_fullsized":"2024-04-25T00:01:20.578828+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:06:19.307378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":9,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T00:01:19.542220+0000","last_change":"2024-04-25T00:01:19.529023+0000","last_active":"2024-04-25T00:01:19.542220+0000","last_peered":"2024-04-25T00:01:19.542220+0000","last_clean":"2024-04-25T00:01:19.542220+0000","last_became_active":"2024-04-25T00:01:19.524609+0000","last_became_peered":"2024-04-25T00:01:19.524609+0000","last_unstale":"2024-04-25T00:01:19.542220+0000","last_undegraded":"2024-04-25T00:01:19.542220+0000","last_fullsized":"2024-04-25T00:01:19.542220+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:38:43.961691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582056+0000","last_change":"2024-04-25T00:01:05.385853+0000","last_active":"2024-04-25T00:01:20.582056+0000","last_peered":"2024-04-25T00:01:20.582056+0000","last_clean":"2024-04-25T00:01:20.582056+0000","last_became_active":"2024-04-25T00:01:05.384997+0000","last_became_peered":"2024-04-25T00:01:05.384997+0000","last_unstale":"2024-04-25T00:01:20.582056+0000","last_undegraded":"2024-04-25T00:01:20.582056+0000","last_fullsized":"2024-04-25T00:01:20.582056+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:36:55.436661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578299+0000","last_change":"2024-04-25T00:01:19.516867+0000","last_active":"2024-04-25T00:01:20.578299+0000","last_peered":"2024-04-25T00:01:20.578299+0000","last_clean":"2024-04-25T00:01:20.578299+0000","last_became_active":"2024-04-25T00:01:19.515764+0000","last_became_peered":"2024-04-25T00:01:19.515764+0000","last_unstale":"2024-04-25T00:01:20.578299+0000","last_undegraded":"2024-04-25T00:01:20.578299+0000","last_fullsized":"2024-04-25T00:01:20.578299+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:53:14.768520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582072+0000","last_change":"2024-04-25T00:01:05.389424+0000","last_active":"2024-04-25T00:01:20.582072+0000","last_peered":"2024-04-25T00:01:20.582072+0000","last_clean":"2024-04-25T00:01:20.582072+0000","last_became_active":"2024-04-25T00:01:05.388999+0000","last_became_peered":"2024-04-25T00:01:05.388999+0000","last_unstale":"2024-04-25T00:01:20.582072+0000","last_undegraded":"2024-04-25T00:01:20.582072+0000","last_fullsized":"2024-04-25T00:01:20.582072+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:31.456792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578192+0000","last_change":"2024-04-25T00:01:19.519365+0000","last_active":"2024-04-25T00:01:20.578192+0000","last_peered":"2024-04-25T00:01:20.578192+0000","last_clean":"2024-04-25T00:01:20.578192+0000","last_became_active":"2024-04-25T00:01:19.519085+0000","last_became_peered":"2024-04-25T00:01:19.519085+0000","last_unstale":"2024-04-25T00:01:20.578192+0000","last_undegraded":"2024-04-25T00:01:20.578192+0000","last_fullsized":"2024-04-25T00:01:20.578192+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:05:09.219823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.580630+0000","last_change":"2024-04-25T00:01:05.368974+0000","last_active":"2024-04-25T00:01:20.580630+0000","last_peered":"2024-04-25T00:01:20.580630+0000","last_clean":"2024-04-25T00:01:20.580630+0000","last_became_active":"2024-04-25T00:01:05.367893+0000","last_became_peered":"2024-04-25T00:01:05.367893+0000","last_unstale":"2024-04-25T00:01:20.580630+0000","last_undegraded":"2024-04-25T00:01:20.580630+0000","last_fullsized":"2024-04-25T00:01:20.580630+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:30:23.557381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578274+0000","last_change":"2024-04-25T00:01:19.513514+0000","last_active":"2024-04-25T00:01:20.578274+0000","last_peered":"2024-04-25T00:01:20.578274+0000","last_clean":"2024-04-25T00:01:20.578274+0000","last_became_active":"2024-04-25T00:01:19.513184+0000","last_became_peered":"2024-04-25T00:01:19.513184+0000","last_unstale":"2024-04-25T00:01:20.578274+0000","last_undegraded":"2024-04-25T00:01:20.578274+0000","last_fullsized":"2024-04-25T00:01:20.578274+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:47.859626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582384+0000","last_change":"2024-04-25T00:01:05.362303+0000","last_active":"2024-04-25T00:01:20.582384+0000","last_peered":"2024-04-25T00:01:20.582384+0000","last_clean":"2024-04-25T00:01:20.582384+0000","last_became_active":"2024-04-25T00:01:05.361586+0000","last_became_peered":"2024-04-25T00:01:05.361586+0000","last_unstale":"2024-04-25T00:01:20.582384+0000","last_undegraded":"2024-04-25T00:01:20.582384+0000","last_fullsized":"2024-04-25T00:01:20.582384+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:30.228060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.520395+0000","last_change":"2024-04-25T00:01:19.515101+0000","last_active":"2024-04-25T00:01:20.520395+0000","last_peered":"2024-04-25T00:01:20.520395+0000","last_clean":"2024-04-25T00:01:20.520395+0000","last_became_active":"2024-04-25T00:01:19.513314+0000","last_became_peered":"2024-04-25T00:01:19.513314+0000","last_unstale":"2024-04-25T00:01:20.520395+0000","last_undegraded":"2024-04-25T00:01:20.520395+0000","last_fullsized":"2024-04-25T00:01:20.520395+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:50.351599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.518642+0000","last_change":"2024-04-25T00:01:05.390256+0000","last_active":"2024-04-25T00:01:20.518642+0000","last_peered":"2024-04-25T00:01:20.518642+0000","last_clean":"2024-04-25T00:01:20.518642+0000","last_became_active":"2024-04-25T00:01:05.389724+0000","last_became_peered":"2024-04-25T00:01:05.389724+0000","last_unstale":"2024-04-25T00:01:20.518642+0000","last_undegraded":"2024-04-25T00:01:20.518642+0000","last_fullsized":"2024-04-25T00:01:20.518642+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:09:18.315688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3f","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511570+0000","last_change":"2024-04-25T00:01:19.511493+0000","last_active":"2024-04-25T00:01:20.511570+0000","last_peered":"2024-04-25T00:01:20.511570+0000","last_clean":"2024-04-25T00:01:20.511570+0000","last_became_active":"2024-04-25T00:01:19.508049+0000","last_became_peered":"2024-04-25T00:01:19.508049+0000","last_unstale":"2024-04-25T00:01:20.511570+0000","last_undegraded":"2024-04-25T00:01:20.511570+0000","last_fullsized":"2024-04-25T00:01:20.511570+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:55:46.853565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578830+0000","last_change":"2024-04-25T00:01:05.374766+0000","last_active":"2024-04-25T00:01:20.578830+0000","last_peered":"2024-04-25T00:01:20.578830+0000","last_clean":"2024-04-25T00:01:20.578830+0000","last_became_active":"2024-04-25T00:01:05.369739+0000","last_became_peered":"2024-04-25T00:01:05.369739+0000","last_unstale":"2024-04-25T00:01:20.578830+0000","last_undegraded":"2024-04-25T00:01:20.578830+0000","last_fullsized":"2024-04-25T00:01:20.578830+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:09:18.315688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":9,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T00:01:19.541277+0000","last_change":"2024-04-25T00:01:19.521456+0000","last_active":"2024-04-25T00:01:19.541277+0000","last_peered":"2024-04-25T00:01:19.541277+0000","last_clean":"2024-04-25T00:01:19.541277+0000","last_became_active":"2024-04-25T00:01:19.520787+0000","last_became_peered":"2024-04-25T00:01:19.520787+0000","last_unstale":"2024-04-25T00:01:19.541277+0000","last_undegraded":"2024-04-25T00:01:19.541277+0000","last_fullsized":"2024-04-25T00:01:19.541277+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:30.865281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511048+0000","last_change":"2024-04-25T00:01:05.359253+0000","last_active":"2024-04-25T00:01:20.511048+0000","last_peered":"2024-04-25T00:01:20.511048+0000","last_clean":"2024-04-25T00:01:20.511048+0000","last_became_active":"2024-04-25T00:01:05.358477+0000","last_became_peered":"2024-04-25T00:01:05.358477+0000","last_unstale":"2024-04-25T00:01:20.511048+0000","last_undegraded":"2024-04-25T00:01:20.511048+0000","last_fullsized":"2024-04-25T00:01:20.511048+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:22.739214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.579384+0000","last_change":"2024-04-25T00:01:19.509505+0000","last_active":"2024-04-25T00:01:20.579384+0000","last_peered":"2024-04-25T00:01:20.579384+0000","last_clean":"2024-04-25T00:01:20.579384+0000","last_became_active":"2024-04-25T00:01:19.507812+0000","last_became_peered":"2024-04-25T00:01:19.507812+0000","last_unstale":"2024-04-25T00:01:20.579384+0000","last_undegraded":"2024-04-25T00:01:20.579384+0000","last_fullsized":"2024-04-25T00:01:20.579384+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:23.993729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.581200+0000","last_change":"2024-04-25T00:01:05.386068+0000","last_active":"2024-04-25T00:01:20.581200+0000","last_peered":"2024-04-25T00:01:20.581200+0000","last_clean":"2024-04-25T00:01:20.581200+0000","last_became_active":"2024-04-25T00:01:05.378650+0000","last_became_peered":"2024-04-25T00:01:05.378650+0000","last_unstale":"2024-04-25T00:01:20.581200+0000","last_undegraded":"2024-04-25T00:01:20.581200+0000","last_fullsized":"2024-04-25T00:01:20.581200+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:12:52.846250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511288+0000","last_change":"2024-04-25T00:01:19.514843+0000","last_active":"2024-04-25T00:01:20.511288+0000","last_peered":"2024-04-25T00:01:20.511288+0000","last_clean":"2024-04-25T00:01:20.511288+0000","last_became_active":"2024-04-25T00:01:19.513956+0000","last_became_peered":"2024-04-25T00:01:19.513956+0000","last_unstale":"2024-04-25T00:01:20.511288+0000","last_undegraded":"2024-04-25T00:01:20.511288+0000","last_fullsized":"2024-04-25T00:01:20.511288+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:38.005233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511996+0000","last_change":"2024-04-25T00:01:05.359716+0000","last_active":"2024-04-25T00:01:20.511996+0000","last_peered":"2024-04-25T00:01:20.511996+0000","last_clean":"2024-04-25T00:01:20.511996+0000","last_became_active":"2024-04-25T00:01:05.358866+0000","last_became_peered":"2024-04-25T00:01:05.358866+0000","last_unstale":"2024-04-25T00:01:20.511996+0000","last_undegraded":"2024-04-25T00:01:20.511996+0000","last_fullsized":"2024-04-25T00:01:20.511996+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:50.489125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.519466+0000","last_change":"2024-04-25T00:01:19.499574+0000","last_active":"2024-04-25T00:01:20.519466+0000","last_peered":"2024-04-25T00:01:20.519466+0000","last_clean":"2024-04-25T00:01:20.519466+0000","last_became_active":"2024-04-25T00:01:19.498699+0000","last_became_peered":"2024-04-25T00:01:19.498699+0000","last_unstale":"2024-04-25T00:01:20.519466+0000","last_undegraded":"2024-04-25T00:01:20.519466+0000","last_fullsized":"2024-04-25T00:01:20.519466+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:47.859626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.520083+0000","last_change":"2024-04-25T00:01:05.371291+0000","last_active":"2024-04-25T00:01:20.520083+0000","last_peered":"2024-04-25T00:01:20.520083+0000","last_clean":"2024-04-25T00:01:20.520083+0000","last_became_active":"2024-04-25T00:01:05.370522+0000","last_became_peered":"2024-04-25T00:01:05.370522+0000","last_unstale":"2024-04-25T00:01:20.520083+0000","last_undegraded":"2024-04-25T00:01:20.520083+0000","last_fullsized":"2024-04-25T00:01:20.520083+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:11:27.044893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511353+0000","last_change":"2024-04-25T00:01:19.523798+0000","last_active":"2024-04-25T00:01:20.511353+0000","last_peered":"2024-04-25T00:01:20.511353+0000","last_clean":"2024-04-25T00:01:20.511353+0000","last_became_active":"2024-04-25T00:01:19.523450+0000","last_became_peered":"2024-04-25T00:01:19.523450+0000","last_unstale":"2024-04-25T00:01:20.511353+0000","last_undegraded":"2024-04-25T00:01:20.511353+0000","last_fullsized":"2024-04-25T00:01:20.511353+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:40:36.350832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.520565+0000","last_change":"2024-04-25T00:01:05.368190+0000","last_active":"2024-04-25T00:01:20.520565+0000","last_peered":"2024-04-25T00:01:20.520565+0000","last_clean":"2024-04-25T00:01:20.520565+0000","last_became_active":"2024-04-25T00:01:05.362606+0000","last_became_peered":"2024-04-25T00:01:05.362606+0000","last_unstale":"2024-04-25T00:01:20.520565+0000","last_undegraded":"2024-04-25T00:01:20.520565+0000","last_fullsized":"2024-04-25T00:01:20.520565+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:12:52.846250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.510932+0000","last_change":"2024-04-25T00:01:19.505737+0000","last_active":"2024-04-25T00:01:20.510932+0000","last_peered":"2024-04-25T00:01:20.510932+0000","last_clean":"2024-04-25T00:01:20.510932+0000","last_became_active":"2024-04-25T00:01:19.504991+0000","last_became_peered":"2024-04-25T00:01:19.504991+0000","last_unstale":"2024-04-25T00:01:20.510932+0000","last_undegraded":"2024-04-25T00:01:20.510932+0000","last_fullsized":"2024-04-25T00:01:20.510932+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:47:19.526845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.583309+0000","last_change":"2024-04-25T00:01:05.389596+0000","last_active":"2024-04-25T00:01:20.583309+0000","last_peered":"2024-04-25T00:01:20.583309+0000","last_clean":"2024-04-25T00:01:20.583309+0000","last_became_active":"2024-04-25T00:01:05.389127+0000","last_became_peered":"2024-04-25T00:01:05.389127+0000","last_unstale":"2024-04-25T00:01:20.583309+0000","last_undegraded":"2024-04-25T00:01:20.583309+0000","last_fullsized":"2024-04-25T00:01:20.583309+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:09.858617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509449,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":109716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509449,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":111182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509448,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27896,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343944,"statfs":{"total":96636764160,"available":96608198656,"internally_reserved":0,"allocated":888832,"data_stored":700612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509449,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27324,"kb_used_data":296,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344516,"statfs":{"total":96636764160,"available":96608784384,"internally_reserved":0,"allocated":303104,"data_stored":110928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509450,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27320,"kb_used_data":292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344520,"statfs":{"total":96636764160,"available":96608788480,"internally_reserved":0,"allocated":299008,"data_stored":111216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509449,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27892,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343948,"statfs":{"total":96636764160,"available":96608202752,"internally_reserved":0,"allocated":884736,"data_stored":700078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509449,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":109716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":278528,"data_stored":97170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T00:01:24.558 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T00:01:24.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.718+0000 7fe12c89f640 1 Processor -- start 2024-04-25T00:01:24.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.718+0000 7fe12c89f640 1 -- start start 2024-04-25T00:01:24.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.718+0000 7fe12c89f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe128071d90 0x7fe1281563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:24.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.718+0000 7fe12c89f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe128071220 con 0x7fe1280a94a0 2024-04-25T00:01:24.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.718+0000 7fe12c89f640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe128152420 con 0x7fe12814c720 2024-04-25T00:01:24.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.718+0000 7fe12c89f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fe128153460 con 0x7fe128071d90 2024-04-25T00:01:24.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.718+0000 7fe11ffff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fe1280a94a0 0x7fe128071680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:60944/0 (socket says 172.21.15.116:60944) 2024-04-25T00:01:24.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.718+0000 7fe11ffff640 1 -- 172.21.15.116:0/173528295 learned_addr learned my addr 172.21.15.116:0/173528295 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:24.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.718+0000 7fe126d9d640 1 --2- 172.21.15.116:0/173528295 >> v2:172.21.15.116:3300/0 conn(0x7fe128071d90 0x7fe1281563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:24.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.718+0000 7fe125d9b640 1 -- 172.21.15.116:0/173528295 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2556357955 0 0) 0x7fe128071220 con 0x7fe1280a94a0 2024-04-25T00:01:24.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.718+0000 7fe125d9b640 1 -- 172.21.15.116:0/173528295 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe114003530 con 0x7fe1280a94a0 2024-04-25T00:01:24.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.718+0000 7fe126d9d640 1 -- 172.21.15.116:0/173528295 >> v1:172.21.15.174:6789/0 conn(0x7fe12814c720 legacy=0x7fe1280c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:24.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.718+0000 7fe126d9d640 1 -- 172.21.15.116:0/173528295 >> v1:172.21.15.116:6789/0 conn(0x7fe1280a94a0 legacy=0x7fe128071680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:24.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.718+0000 7fe126d9d640 1 -- 172.21.15.116:0/173528295 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1280721c0 con 0x7fe128071d90 2024-04-25T00:01:24.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.722+0000 7fe126d9d640 1 --2- 172.21.15.116:0/173528295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe128071d90 0x7fe1281563d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe120009d90 tx=0x7fe12001c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e62d31632959addb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:24.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.722+0000 7fe125d9b640 1 -- 172.21.15.116:0/173528295 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1200b2070 con 0x7fe128071d90 2024-04-25T00:01:24.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.722+0000 7fe125d9b640 1 -- 172.21.15.116:0/173528295 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1200a59a0 con 0x7fe128071d90 2024-04-25T00:01:24.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.722+0000 7fe125d9b640 1 -- 172.21.15.116:0/173528295 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1200a5d70 con 0x7fe128071d90 2024-04-25T00:01:24.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.722+0000 7fe12c89f640 1 -- 172.21.15.116:0/173528295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe128071d90 msgr2=0x7fe1281563d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:24.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.722+0000 7fe12c89f640 1 --2- 172.21.15.116:0/173528295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe128071d90 0x7fe1281563d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe120009d90 tx=0x7fe12001c8e0 comp rx=0 tx=0).stop 2024-04-25T00:01:24.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.722+0000 7fe12c89f640 1 -- 172.21.15.116:0/173528295 shutdown_connections 2024-04-25T00:01:24.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.722+0000 7fe12c89f640 1 --2- 172.21.15.116:0/173528295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe128071d90 0x7fe1281563d0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:24.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.722+0000 7fe12c89f640 1 -- 172.21.15.116:0/173528295 >> 172.21.15.116:0/173528295 conn(0x7fe1280c3b00 msgr2=0x7fe1280c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:24.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.722+0000 7fe12c89f640 1 -- 172.21.15.116:0/173528295 shutdown_connections 2024-04-25T00:01:24.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.722+0000 7fe12c89f640 1 -- 172.21.15.116:0/173528295 wait complete. 2024-04-25T00:01:24.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.722+0000 7fe12c89f640 1 Processor -- start 2024-04-25T00:01:24.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.722+0000 7fe12c89f640 1 -- start start 2024-04-25T00:01:24.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.722+0000 7fe12c89f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe128071d90 0x7fe128154850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:24.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.722+0000 7fe12c89f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1280a94a0 0x7fe1281ee0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:24.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.722+0000 7fe12c89f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe12814c720 0x7fe1280c0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:24.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.722+0000 7fe12c89f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe128152f40 con 0x7fe1280a94a0 2024-04-25T00:01:24.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.722+0000 7fe12c89f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe128152c60 con 0x7fe128071d90 2024-04-25T00:01:24.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.722+0000 7fe12c89f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe1280c3120 con 0x7fe12814c720 2024-04-25T00:01:24.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.722+0000 7fe11ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1280a94a0 0x7fe1281ee0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:24.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.722+0000 7fe11ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1280a94a0 0x7fe1281ee0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54560/0 (socket says 172.21.15.116:54560) 2024-04-25T00:01:24.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.722+0000 7fe11ffff640 1 -- 172.21.15.116:0/972771167 learned_addr learned my addr 172.21.15.116:0/972771167 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:24.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.722+0000 7fe12659c640 1 --2- 172.21.15.116:0/972771167 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe128071d90 0x7fe128154850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:24.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.722+0000 7fe126d9d640 1 --2- 172.21.15.116:0/972771167 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe12814c720 0x7fe1280c0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:24.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.722+0000 7fe11ffff640 1 -- 172.21.15.116:0/972771167 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe12814c720 msgr2=0x7fe1280c0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:24.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.722+0000 7fe11ffff640 1 --2- 172.21.15.116:0/972771167 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe12814c720 0x7fe1280c0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:24.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.722+0000 7fe11ffff640 1 -- 172.21.15.116:0/972771167 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe128071d90 msgr2=0x7fe128154850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:24.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.722+0000 7fe11ffff640 1 --2- 172.21.15.116:0/972771167 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe128071d90 0x7fe128154850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:24.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.722+0000 7fe11ffff640 1 -- 172.21.15.116:0/972771167 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1281ef080 con 0x7fe1280a94a0 2024-04-25T00:01:24.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.722+0000 7fe11ffff640 1 --2- 172.21.15.116:0/972771167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1280a94a0 0x7fe1281ee0f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe11000c500 tx=0x7fe11000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:24.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.722+0000 7fe11effd640 1 -- 172.21.15.116:0/972771167 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe110017070 con 0x7fe1280a94a0 2024-04-25T00:01:24.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.722+0000 7fe11effd640 1 -- 172.21.15.116:0/972771167 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe110005cf0 con 0x7fe1280a94a0 2024-04-25T00:01:24.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.722+0000 7fe12c89f640 1 -- 172.21.15.116:0/972771167 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1281ee800 con 0x7fe1280a94a0 2024-04-25T00:01:24.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.722+0000 7fe11effd640 1 -- 172.21.15.116:0/972771167 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1100072f0 con 0x7fe1280a94a0 2024-04-25T00:01:24.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.722+0000 7fe12c89f640 1 -- 172.21.15.116:0/972771167 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1281eeda0 con 0x7fe1280a94a0 2024-04-25T00:01:24.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.726+0000 7fe12c89f640 1 -- 172.21.15.116:0/972771167 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe128068350 con 0x7fe1280a94a0 2024-04-25T00:01:24.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.726+0000 7fe11effd640 1 -- 172.21.15.116:0/972771167 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe11001a070 con 0x7fe1280a94a0 2024-04-25T00:01:24.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.726+0000 7fe11effd640 1 --2- 172.21.15.116:0/972771167 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe1080678c0 0x7fe108069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:24.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.726+0000 7fe11effd640 1 -- 172.21.15.116:0/972771167 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7fe1100852e0 con 0x7fe1280a94a0 2024-04-25T00:01:24.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.726+0000 7fe12659c640 1 --2- 172.21.15.116:0/972771167 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe1080678c0 0x7fe108069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:24.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.726+0000 7fe12659c640 1 --2- 172.21.15.116:0/972771167 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe1080678c0 0x7fe108069d80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe1180015b0 tx=0x7fe118009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:24.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.730+0000 7fe11effd640 1 -- 172.21.15.116:0/972771167 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fe110055530 con 0x7fe1280a94a0 2024-04-25T00:01:24.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.874+0000 7fe12c89f640 1 -- 172.21.15.116:0/972771167 --> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe1280c2f00 con 0x7fe1080678c0 2024-04-25T00:01:24.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.906+0000 7fe11effd640 1 -- 172.21.15.116:0/972771167 <== mgr.4105 v2:172.21.15.174:6832/2004448211 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+490414 (secure 0 0 0) 0x7fe1280c2f00 con 0x7fe1080678c0 2024-04-25T00:01:24.912 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:01:24.918 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-04-25T00:01:24.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.914+0000 7fe12c89f640 1 -- 172.21.15.116:0/972771167 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe1080678c0 msgr2=0x7fe108069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:24.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.914+0000 7fe12c89f640 1 --2- 172.21.15.116:0/972771167 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe1080678c0 0x7fe108069d80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe1180015b0 tx=0x7fe118009040 comp rx=0 tx=0).stop 2024-04-25T00:01:24.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.914+0000 7fe12c89f640 1 -- 172.21.15.116:0/972771167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1280a94a0 msgr2=0x7fe1281ee0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:24.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.914+0000 7fe12c89f640 1 --2- 172.21.15.116:0/972771167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1280a94a0 0x7fe1281ee0f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe11000c500 tx=0x7fe11000c9c0 comp rx=0 tx=0).stop 2024-04-25T00:01:24.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.914+0000 7fe12c89f640 1 -- 172.21.15.116:0/972771167 shutdown_connections 2024-04-25T00:01:24.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.914+0000 7fe12c89f640 1 --2- 172.21.15.116:0/972771167 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe1080678c0 0x7fe108069d80 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:24.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.914+0000 7fe12c89f640 1 --2- 172.21.15.116:0/972771167 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe12814c720 0x7fe1280c0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:24.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.914+0000 7fe12c89f640 1 --2- 172.21.15.116:0/972771167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1280a94a0 0x7fe1281ee0f0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:24.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.914+0000 7fe12c89f640 1 --2- 172.21.15.116:0/972771167 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe128071d90 0x7fe128154850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:24.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.914+0000 7fe12c89f640 1 -- 172.21.15.116:0/972771167 >> 172.21.15.116:0/972771167 conn(0x7fe1280c3b00 msgr2=0x7fe1280c3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:24.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.914+0000 7fe12c89f640 1 -- 172.21.15.116:0/972771167 shutdown_connections 2024-04-25T00:01:24.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:24.914+0000 7fe12c89f640 1 -- 172.21.15.116:0/972771167 wait complete. 2024-04-25T00:01:24.931 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-04-25T00:01:23.256272+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595188,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":249,"num_write_kb":2444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":195,"ondisk_log_size":195,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":372,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219668,"kb_used_data":3456,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755052,"statfs":{"total":773094113280,"available":772869173248,"internally_reserved":0,"allocated":3538944,"data_stored":2050618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":53,"apply_latency_ms":53,"commit_latency_ns":53000000,"apply_latency_ns":53000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.804465"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.577366+0000","last_change":"2024-04-25T00:01:19.508728+0000","last_active":"2024-04-25T00:01:20.577366+0000","last_peered":"2024-04-25T00:01:20.577366+0000","last_clean":"2024-04-25T00:01:20.577366+0000","last_became_active":"2024-04-25T00:01:19.508390+0000","last_became_peered":"2024-04-25T00:01:19.508390+0000","last_unstale":"2024-04-25T00:01:20.577366+0000","last_undegraded":"2024-04-25T00:01:20.577366+0000","last_fullsized":"2024-04-25T00:01:20.577366+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:45.591905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1f","version":"22'4","reported_seq":24,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T00:01:19.494954+0000","last_change":"2024-04-25T00:01:04.356032+0000","last_active":"2024-04-25T00:01:19.494954+0000","last_peered":"2024-04-25T00:01:19.494954+0000","last_clean":"2024-04-25T00:01:19.494954+0000","last_became_active":"2024-04-25T00:01:04.355293+0000","last_became_peered":"2024-04-25T00:01:04.355293+0000","last_unstale":"2024-04-25T00:01:19.494954+0000","last_undegraded":"2024-04-25T00:01:19.494954+0000","last_fullsized":"2024-04-25T00:01:19.494954+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:56:16.678132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.520751+0000","last_change":"2024-04-25T00:01:05.367207+0000","last_active":"2024-04-25T00:01:20.520751+0000","last_peered":"2024-04-25T00:01:20.520751+0000","last_clean":"2024-04-25T00:01:20.520751+0000","last_became_active":"2024-04-25T00:01:05.366471+0000","last_became_peered":"2024-04-25T00:01:05.366471+0000","last_unstale":"2024-04-25T00:01:20.520751+0000","last_undegraded":"2024-04-25T00:01:20.520751+0000","last_fullsized":"2024-04-25T00:01:20.520751+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:30:23.557381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511325+0000","last_change":"2024-04-25T00:01:18.502364+0000","last_active":"2024-04-25T00:01:20.511325+0000","last_peered":"2024-04-25T00:01:20.511325+0000","last_clean":"2024-04-25T00:01:20.511325+0000","last_became_active":"2024-04-25T00:01:18.501633+0000","last_became_peered":"2024-04-25T00:01:18.501633+0000","last_unstale":"2024-04-25T00:01:20.511325+0000","last_undegraded":"2024-04-25T00:01:20.511325+0000","last_fullsized":"2024-04-25T00:01:20.511325+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:55.074018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T00:01:19.541052+0000","last_change":"2024-04-25T00:01:19.512822+0000","last_active":"2024-04-25T00:01:19.541052+0000","last_peered":"2024-04-25T00:01:19.541052+0000","last_clean":"2024-04-25T00:01:19.541052+0000","last_became_active":"2024-04-25T00:01:19.512105+0000","last_became_peered":"2024-04-25T00:01:19.512105+0000","last_unstale":"2024-04-25T00:01:19.541052+0000","last_undegraded":"2024-04-25T00:01:19.541052+0000","last_fullsized":"2024-04-25T00:01:19.541052+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:07:42.932302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"22'1","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.520903+0000","last_change":"2024-04-25T00:01:04.361809+0000","last_active":"2024-04-25T00:01:20.520903+0000","last_peered":"2024-04-25T00:01:20.520903+0000","last_clean":"2024-04-25T00:01:20.520903+0000","last_became_active":"2024-04-25T00:01:04.361059+0000","last_became_peered":"2024-04-25T00:01:04.361059+0000","last_unstale":"2024-04-25T00:01:20.520903+0000","last_undegraded":"2024-04-25T00:01:20.520903+0000","last_fullsized":"2024-04-25T00:01:20.520903+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.520801+0000","last_change":"2024-04-25T00:01:05.370818+0000","last_active":"2024-04-25T00:01:20.520801+0000","last_peered":"2024-04-25T00:01:20.520801+0000","last_clean":"2024-04-25T00:01:20.520801+0000","last_became_active":"2024-04-25T00:01:05.369991+0000","last_became_peered":"2024-04-25T00:01:05.369991+0000","last_unstale":"2024-04-25T00:01:20.520801+0000","last_undegraded":"2024-04-25T00:01:20.520801+0000","last_fullsized":"2024-04-25T00:01:20.520801+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:43.615895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.510695+0000","last_change":"2024-04-25T00:01:18.497682+0000","last_active":"2024-04-25T00:01:20.510695+0000","last_peered":"2024-04-25T00:01:20.510695+0000","last_clean":"2024-04-25T00:01:20.510695+0000","last_became_active":"2024-04-25T00:01:18.497397+0000","last_became_peered":"2024-04-25T00:01:18.497397+0000","last_unstale":"2024-04-25T00:01:20.510695+0000","last_undegraded":"2024-04-25T00:01:20.510695+0000","last_fullsized":"2024-04-25T00:01:20.510695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:22.983972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578826+0000","last_change":"2024-04-25T00:01:19.510591+0000","last_active":"2024-04-25T00:01:20.578826+0000","last_peered":"2024-04-25T00:01:20.578826+0000","last_clean":"2024-04-25T00:01:20.578826+0000","last_became_active":"2024-04-25T00:01:19.510252+0000","last_became_peered":"2024-04-25T00:01:19.510252+0000","last_unstale":"2024-04-25T00:01:20.578826+0000","last_undegraded":"2024-04-25T00:01:20.578826+0000","last_fullsized":"2024-04-25T00:01:20.578826+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:38.005233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"22'2","reported_seq":24,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582530+0000","last_change":"2024-04-25T00:01:04.362030+0000","last_active":"2024-04-25T00:01:20.582530+0000","last_peered":"2024-04-25T00:01:20.582530+0000","last_clean":"2024-04-25T00:01:20.582530+0000","last_became_active":"2024-04-25T00:01:04.361708+0000","last_became_peered":"2024-04-25T00:01:04.361708+0000","last_unstale":"2024-04-25T00:01:20.582530+0000","last_undegraded":"2024-04-25T00:01:20.582530+0000","last_fullsized":"2024-04-25T00:01:20.582530+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:38:33.392719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578882+0000","last_change":"2024-04-25T00:01:05.387195+0000","last_active":"2024-04-25T00:01:20.578882+0000","last_peered":"2024-04-25T00:01:20.578882+0000","last_clean":"2024-04-25T00:01:20.578882+0000","last_became_active":"2024-04-25T00:01:05.386853+0000","last_became_peered":"2024-04-25T00:01:05.386853+0000","last_unstale":"2024-04-25T00:01:20.578882+0000","last_undegraded":"2024-04-25T00:01:20.578882+0000","last_fullsized":"2024-04-25T00:01:20.578882+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:06:19.307378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"26'1","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511288+0000","last_change":"2024-04-25T00:01:18.487835+0000","last_active":"2024-04-25T00:01:20.511288+0000","last_peered":"2024-04-25T00:01:20.511288+0000","last_clean":"2024-04-25T00:01:20.511288+0000","last_became_active":"2024-04-25T00:01:18.487571+0000","last_became_peered":"2024-04-25T00:01:18.487571+0000","last_unstale":"2024-04-25T00:01:20.511288+0000","last_undegraded":"2024-04-25T00:01:20.511288+0000","last_fullsized":"2024-04-25T00:01:20.511288+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:44.582148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511094+0000","last_change":"2024-04-25T00:01:19.519834+0000","last_active":"2024-04-25T00:01:20.511094+0000","last_peered":"2024-04-25T00:01:20.511094+0000","last_clean":"2024-04-25T00:01:20.511094+0000","last_became_active":"2024-04-25T00:01:19.519470+0000","last_became_peered":"2024-04-25T00:01:19.519470+0000","last_unstale":"2024-04-25T00:01:20.511094+0000","last_undegraded":"2024-04-25T00:01:20.511094+0000","last_fullsized":"2024-04-25T00:01:20.511094+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:47.859626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"22'2","reported_seq":24,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582640+0000","last_change":"2024-04-25T00:01:04.362832+0000","last_active":"2024-04-25T00:01:20.582640+0000","last_peered":"2024-04-25T00:01:20.582640+0000","last_clean":"2024-04-25T00:01:20.582640+0000","last_became_active":"2024-04-25T00:01:04.362059+0000","last_became_peered":"2024-04-25T00:01:04.362059+0000","last_unstale":"2024-04-25T00:01:20.582640+0000","last_undegraded":"2024-04-25T00:01:20.582640+0000","last_fullsized":"2024-04-25T00:01:20.582640+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:05.441048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.583017+0000","last_change":"2024-04-25T00:01:05.381120+0000","last_active":"2024-04-25T00:01:20.583017+0000","last_peered":"2024-04-25T00:01:20.583017+0000","last_clean":"2024-04-25T00:01:20.583017+0000","last_became_active":"2024-04-25T00:01:05.378320+0000","last_became_peered":"2024-04-25T00:01:05.378320+0000","last_unstale":"2024-04-25T00:01:20.583017+0000","last_undegraded":"2024-04-25T00:01:20.583017+0000","last_fullsized":"2024-04-25T00:01:20.583017+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:34:06.175918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"26'1","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.510926+0000","last_change":"2024-04-25T00:01:18.490300+0000","last_active":"2024-04-25T00:01:20.510926+0000","last_peered":"2024-04-25T00:01:20.510926+0000","last_clean":"2024-04-25T00:01:20.510926+0000","last_became_active":"2024-04-25T00:01:18.490050+0000","last_became_peered":"2024-04-25T00:01:18.490050+0000","last_unstale":"2024-04-25T00:01:20.510926+0000","last_undegraded":"2024-04-25T00:01:20.510926+0000","last_fullsized":"2024-04-25T00:01:20.510926+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:13:05.971606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.519505+0000","last_change":"2024-04-25T00:01:19.500026+0000","last_active":"2024-04-25T00:01:20.519505+0000","last_peered":"2024-04-25T00:01:20.519505+0000","last_clean":"2024-04-25T00:01:20.519505+0000","last_became_active":"2024-04-25T00:01:19.499244+0000","last_became_peered":"2024-04-25T00:01:19.499244+0000","last_unstale":"2024-04-25T00:01:20.519505+0000","last_undegraded":"2024-04-25T00:01:20.519505+0000","last_fullsized":"2024-04-25T00:01:20.519505+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:40:36.350832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.580552+0000","last_change":"2024-04-25T00:01:05.376984+0000","last_active":"2024-04-25T00:01:20.580552+0000","last_peered":"2024-04-25T00:01:20.580552+0000","last_clean":"2024-04-25T00:01:20.580552+0000","last_became_active":"2024-04-25T00:01:05.376259+0000","last_became_peered":"2024-04-25T00:01:05.376259+0000","last_unstale":"2024-04-25T00:01:20.580552+0000","last_undegraded":"2024-04-25T00:01:20.580552+0000","last_fullsized":"2024-04-25T00:01:20.580552+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:13:28.543302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"22'2","reported_seq":24,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.581267+0000","last_change":"2024-04-25T00:01:04.359262+0000","last_active":"2024-04-25T00:01:20.581267+0000","last_peered":"2024-04-25T00:01:20.581267+0000","last_clean":"2024-04-25T00:01:20.581267+0000","last_became_active":"2024-04-25T00:01:04.357971+0000","last_became_peered":"2024-04-25T00:01:04.357971+0000","last_unstale":"2024-04-25T00:01:20.581267+0000","last_undegraded":"2024-04-25T00:01:20.581267+0000","last_fullsized":"2024-04-25T00:01:20.581267+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:05.441048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"26'1","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582746+0000","last_change":"2024-04-25T00:01:18.475624+0000","last_active":"2024-04-25T00:01:20.582746+0000","last_peered":"2024-04-25T00:01:20.582746+0000","last_clean":"2024-04-25T00:01:20.582746+0000","last_became_active":"2024-04-25T00:01:18.475149+0000","last_became_peered":"2024-04-25T00:01:18.475149+0000","last_unstale":"2024-04-25T00:01:20.582746+0000","last_undegraded":"2024-04-25T00:01:20.582746+0000","last_fullsized":"2024-04-25T00:01:20.582746+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:40:35.341075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.512956+0000","last_change":"2024-04-25T00:01:19.531113+0000","last_active":"2024-04-25T00:01:20.512956+0000","last_peered":"2024-04-25T00:01:20.512956+0000","last_clean":"2024-04-25T00:01:20.512956+0000","last_became_active":"2024-04-25T00:01:19.520150+0000","last_became_peered":"2024-04-25T00:01:19.520150+0000","last_unstale":"2024-04-25T00:01:20.512956+0000","last_undegraded":"2024-04-25T00:01:20.512956+0000","last_fullsized":"2024-04-25T00:01:20.512956+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:38.152376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511363+0000","last_change":"2024-04-25T00:01:05.362611+0000","last_active":"2024-04-25T00:01:20.511363+0000","last_peered":"2024-04-25T00:01:20.511363+0000","last_clean":"2024-04-25T00:01:20.511363+0000","last_became_active":"2024-04-25T00:01:05.361817+0000","last_became_peered":"2024-04-25T00:01:05.361817+0000","last_unstale":"2024-04-25T00:01:20.511363+0000","last_undegraded":"2024-04-25T00:01:20.511363+0000","last_fullsized":"2024-04-25T00:01:20.511363+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:50.489125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578606+0000","last_change":"2024-04-25T00:01:04.371935+0000","last_active":"2024-04-25T00:01:20.578606+0000","last_peered":"2024-04-25T00:01:20.578606+0000","last_clean":"2024-04-25T00:01:20.578606+0000","last_became_active":"2024-04-25T00:01:04.371420+0000","last_became_peered":"2024-04-25T00:01:04.371420+0000","last_unstale":"2024-04-25T00:01:20.578606+0000","last_undegraded":"2024-04-25T00:01:20.578606+0000","last_fullsized":"2024-04-25T00:01:20.578606+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582229+0000","last_change":"2024-04-25T00:01:18.488623+0000","last_active":"2024-04-25T00:01:20.582229+0000","last_peered":"2024-04-25T00:01:20.582229+0000","last_clean":"2024-04-25T00:01:20.582229+0000","last_became_active":"2024-04-25T00:01:18.488291+0000","last_became_peered":"2024-04-25T00:01:18.488291+0000","last_unstale":"2024-04-25T00:01:20.582229+0000","last_undegraded":"2024-04-25T00:01:20.582229+0000","last_fullsized":"2024-04-25T00:01:20.582229+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:30:21.436681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.584490+0000","last_change":"2024-04-25T00:01:19.516731+0000","last_active":"2024-04-25T00:01:20.584490+0000","last_peered":"2024-04-25T00:01:20.584490+0000","last_clean":"2024-04-25T00:01:20.584490+0000","last_became_active":"2024-04-25T00:01:19.516405+0000","last_became_peered":"2024-04-25T00:01:19.516405+0000","last_unstale":"2024-04-25T00:01:20.584490+0000","last_undegraded":"2024-04-25T00:01:20.584490+0000","last_fullsized":"2024-04-25T00:01:20.584490+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:44:37.936092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T00:01:19.493937+0000","last_change":"2024-04-25T00:01:05.355455+0000","last_active":"2024-04-25T00:01:19.493937+0000","last_peered":"2024-04-25T00:01:19.493937+0000","last_clean":"2024-04-25T00:01:19.493937+0000","last_became_active":"2024-04-25T00:01:05.354699+0000","last_became_peered":"2024-04-25T00:01:05.354699+0000","last_unstale":"2024-04-25T00:01:19.493937+0000","last_undegraded":"2024-04-25T00:01:19.493937+0000","last_fullsized":"2024-04-25T00:01:19.493937+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:43:58.069271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":20,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T00:01:19.494923+0000","last_change":"2024-04-25T00:01:04.362268+0000","last_active":"2024-04-25T00:01:19.494923+0000","last_peered":"2024-04-25T00:01:19.494923+0000","last_clean":"2024-04-25T00:01:19.494923+0000","last_became_active":"2024-04-25T00:01:04.361622+0000","last_became_peered":"2024-04-25T00:01:04.361622+0000","last_unstale":"2024-04-25T00:01:19.494923+0000","last_undegraded":"2024-04-25T00:01:19.494923+0000","last_fullsized":"2024-04-25T00:01:19.494923+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:58:46.021343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"26'1","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.514314+0000","last_change":"2024-04-25T00:01:18.494047+0000","last_active":"2024-04-25T00:01:20.514314+0000","last_peered":"2024-04-25T00:01:20.514314+0000","last_clean":"2024-04-25T00:01:20.514314+0000","last_became_active":"2024-04-25T00:01:18.493614+0000","last_became_peered":"2024-04-25T00:01:18.493614+0000","last_unstale":"2024-04-25T00:01:20.514314+0000","last_undegraded":"2024-04-25T00:01:20.514314+0000","last_fullsized":"2024-04-25T00:01:20.514314+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:37:08.562016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582182+0000","last_change":"2024-04-25T00:01:19.505397+0000","last_active":"2024-04-25T00:01:20.582182+0000","last_peered":"2024-04-25T00:01:20.582182+0000","last_clean":"2024-04-25T00:01:20.582182+0000","last_became_active":"2024-04-25T00:01:19.505126+0000","last_became_peered":"2024-04-25T00:01:19.505126+0000","last_unstale":"2024-04-25T00:01:20.582182+0000","last_undegraded":"2024-04-25T00:01:20.582182+0000","last_fullsized":"2024-04-25T00:01:20.582182+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:08:49.640267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.581145+0000","last_change":"2024-04-25T00:01:05.354953+0000","last_active":"2024-04-25T00:01:20.581145+0000","last_peered":"2024-04-25T00:01:20.581145+0000","last_clean":"2024-04-25T00:01:20.581145+0000","last_became_active":"2024-04-25T00:01:05.354118+0000","last_became_peered":"2024-04-25T00:01:05.354118+0000","last_unstale":"2024-04-25T00:01:20.581145+0000","last_undegraded":"2024-04-25T00:01:20.581145+0000","last_fullsized":"2024-04-25T00:01:20.581145+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:09.858617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578494+0000","last_change":"2024-04-25T00:01:04.365515+0000","last_active":"2024-04-25T00:01:20.578494+0000","last_peered":"2024-04-25T00:01:20.578494+0000","last_clean":"2024-04-25T00:01:20.578494+0000","last_became_active":"2024-04-25T00:01:04.364942+0000","last_became_peered":"2024-04-25T00:01:04.364942+0000","last_unstale":"2024-04-25T00:01:20.578494+0000","last_undegraded":"2024-04-25T00:01:20.578494+0000","last_fullsized":"2024-04-25T00:01:20.578494+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"26'2","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511306+0000","last_change":"2024-04-25T00:01:18.476207+0000","last_active":"2024-04-25T00:01:20.511306+0000","last_peered":"2024-04-25T00:01:20.511306+0000","last_clean":"2024-04-25T00:01:20.511306+0000","last_became_active":"2024-04-25T00:01:18.475862+0000","last_became_peered":"2024-04-25T00:01:18.475862+0000","last_unstale":"2024-04-25T00:01:20.511306+0000","last_undegraded":"2024-04-25T00:01:20.511306+0000","last_fullsized":"2024-04-25T00:01:20.511306+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:11:40.170249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T00:01:19.542432+0000","last_change":"2024-04-25T00:01:19.529607+0000","last_active":"2024-04-25T00:01:19.542432+0000","last_peered":"2024-04-25T00:01:19.542432+0000","last_clean":"2024-04-25T00:01:19.542432+0000","last_became_active":"2024-04-25T00:01:19.528509+0000","last_became_peered":"2024-04-25T00:01:19.528509+0000","last_unstale":"2024-04-25T00:01:19.542432+0000","last_undegraded":"2024-04-25T00:01:19.542432+0000","last_fullsized":"2024-04-25T00:01:19.542432+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:47:26.096921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T00:01:19.497197+0000","last_change":"2024-04-25T00:01:05.358106+0000","last_active":"2024-04-25T00:01:19.497197+0000","last_peered":"2024-04-25T00:01:19.497197+0000","last_clean":"2024-04-25T00:01:19.497197+0000","last_became_active":"2024-04-25T00:01:05.357135+0000","last_became_peered":"2024-04-25T00:01:05.357135+0000","last_unstale":"2024-04-25T00:01:19.497197+0000","last_undegraded":"2024-04-25T00:01:19.497197+0000","last_fullsized":"2024-04-25T00:01:19.497197+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:41:42.228234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"22'2","reported_seq":24,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.584846+0000","last_change":"2024-04-25T00:01:04.369917+0000","last_active":"2024-04-25T00:01:20.584846+0000","last_peered":"2024-04-25T00:01:20.584846+0000","last_clean":"2024-04-25T00:01:20.584846+0000","last_became_active":"2024-04-25T00:01:04.369309+0000","last_became_peered":"2024-04-25T00:01:04.369309+0000","last_unstale":"2024-04-25T00:01:20.584846+0000","last_undegraded":"2024-04-25T00:01:20.584846+0000","last_fullsized":"2024-04-25T00:01:20.584846+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:55:08.063965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511314+0000","last_change":"2024-04-25T00:01:18.477189+0000","last_active":"2024-04-25T00:01:20.511314+0000","last_peered":"2024-04-25T00:01:20.511314+0000","last_clean":"2024-04-25T00:01:20.511314+0000","last_became_active":"2024-04-25T00:01:18.476909+0000","last_became_peered":"2024-04-25T00:01:18.476909+0000","last_unstale":"2024-04-25T00:01:20.511314+0000","last_undegraded":"2024-04-25T00:01:20.511314+0000","last_fullsized":"2024-04-25T00:01:20.511314+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:36.995477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.580550+0000","last_change":"2024-04-25T00:01:19.517500+0000","last_active":"2024-04-25T00:01:20.580550+0000","last_peered":"2024-04-25T00:01:20.580550+0000","last_clean":"2024-04-25T00:01:20.580550+0000","last_became_active":"2024-04-25T00:01:19.517182+0000","last_became_peered":"2024-04-25T00:01:19.517182+0000","last_unstale":"2024-04-25T00:01:20.580550+0000","last_undegraded":"2024-04-25T00:01:20.580550+0000","last_fullsized":"2024-04-25T00:01:20.580550+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:58:49.738233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T00:01:19.494701+0000","last_change":"2024-04-25T00:01:05.364829+0000","last_active":"2024-04-25T00:01:19.494701+0000","last_peered":"2024-04-25T00:01:19.494701+0000","last_clean":"2024-04-25T00:01:19.494701+0000","last_became_active":"2024-04-25T00:01:05.364125+0000","last_became_peered":"2024-04-25T00:01:05.364125+0000","last_unstale":"2024-04-25T00:01:19.494701+0000","last_undegraded":"2024-04-25T00:01:19.494701+0000","last_fullsized":"2024-04-25T00:01:19.494701+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:15:45.531760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"23'1","reported_seq":21,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T00:01:19.498006+0000","last_change":"2024-04-25T00:01:04.361603+0000","last_active":"2024-04-25T00:01:19.498006+0000","last_peered":"2024-04-25T00:01:19.498006+0000","last_clean":"2024-04-25T00:01:19.498006+0000","last_became_active":"2024-04-25T00:01:04.361046+0000","last_became_peered":"2024-04-25T00:01:04.361046+0000","last_unstale":"2024-04-25T00:01:19.498006+0000","last_undegraded":"2024-04-25T00:01:19.498006+0000","last_fullsized":"2024-04-25T00:01:19.498006+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:35:18.036843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T00:01:19.494649+0000","last_change":"2024-04-25T00:01:18.488352+0000","last_active":"2024-04-25T00:01:19.494649+0000","last_peered":"2024-04-25T00:01:19.494649+0000","last_clean":"2024-04-25T00:01:19.494649+0000","last_became_active":"2024-04-25T00:01:18.487321+0000","last_became_peered":"2024-04-25T00:01:18.487321+0000","last_unstale":"2024-04-25T00:01:19.494649+0000","last_undegraded":"2024-04-25T00:01:19.494649+0000","last_fullsized":"2024-04-25T00:01:19.494649+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:37:22.329655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578359+0000","last_change":"2024-04-25T00:01:19.519653+0000","last_active":"2024-04-25T00:01:20.578359+0000","last_peered":"2024-04-25T00:01:20.578359+0000","last_clean":"2024-04-25T00:01:20.578359+0000","last_became_active":"2024-04-25T00:01:19.519052+0000","last_became_peered":"2024-04-25T00:01:19.519052+0000","last_unstale":"2024-04-25T00:01:20.578359+0000","last_undegraded":"2024-04-25T00:01:20.578359+0000","last_fullsized":"2024-04-25T00:01:20.578359+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:40:36.350832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.579276+0000","last_change":"2024-04-25T00:01:05.374011+0000","last_active":"2024-04-25T00:01:20.579276+0000","last_peered":"2024-04-25T00:01:20.579276+0000","last_clean":"2024-04-25T00:01:20.579276+0000","last_became_active":"2024-04-25T00:01:05.373276+0000","last_became_peered":"2024-04-25T00:01:05.373276+0000","last_unstale":"2024-04-25T00:01:20.579276+0000","last_undegraded":"2024-04-25T00:01:20.579276+0000","last_fullsized":"2024-04-25T00:01:20.579276+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:40:53.349428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.583723+0000","last_change":"2024-04-25T00:01:04.365391+0000","last_active":"2024-04-25T00:01:20.583723+0000","last_peered":"2024-04-25T00:01:20.583723+0000","last_clean":"2024-04-25T00:01:20.583723+0000","last_became_active":"2024-04-25T00:01:04.364989+0000","last_became_peered":"2024-04-25T00:01:04.364989+0000","last_unstale":"2024-04-25T00:01:20.583723+0000","last_undegraded":"2024-04-25T00:01:20.583723+0000","last_fullsized":"2024-04-25T00:01:20.583723+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:06:18.298057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578642+0000","last_change":"2024-04-25T00:01:18.491597+0000","last_active":"2024-04-25T00:01:20.578642+0000","last_peered":"2024-04-25T00:01:20.578642+0000","last_clean":"2024-04-25T00:01:20.578642+0000","last_became_active":"2024-04-25T00:01:18.490859+0000","last_became_peered":"2024-04-25T00:01:18.490859+0000","last_unstale":"2024-04-25T00:01:20.578642+0000","last_undegraded":"2024-04-25T00:01:20.578642+0000","last_fullsized":"2024-04-25T00:01:20.578642+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:36.995477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T00:01:19.542507+0000","last_change":"2024-04-25T00:01:19.532926+0000","last_active":"2024-04-25T00:01:19.542507+0000","last_peered":"2024-04-25T00:01:19.542507+0000","last_clean":"2024-04-25T00:01:19.542507+0000","last_became_active":"2024-04-25T00:01:19.529742+0000","last_became_peered":"2024-04-25T00:01:19.529742+0000","last_unstale":"2024-04-25T00:01:19.542507+0000","last_undegraded":"2024-04-25T00:01:19.542507+0000","last_fullsized":"2024-04-25T00:01:19.542507+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:14:12.893663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511333+0000","last_change":"2024-04-25T00:01:05.378340+0000","last_active":"2024-04-25T00:01:20.511333+0000","last_peered":"2024-04-25T00:01:20.511333+0000","last_clean":"2024-04-25T00:01:20.511333+0000","last_became_active":"2024-04-25T00:01:05.377593+0000","last_became_peered":"2024-04-25T00:01:05.377593+0000","last_unstale":"2024-04-25T00:01:20.511333+0000","last_undegraded":"2024-04-25T00:01:20.511333+0000","last_fullsized":"2024-04-25T00:01:20.511333+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:48.202251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.519859+0000","last_change":"2024-04-25T00:01:04.366577+0000","last_active":"2024-04-25T00:01:20.519859+0000","last_peered":"2024-04-25T00:01:20.519859+0000","last_clean":"2024-04-25T00:01:20.519859+0000","last_became_active":"2024-04-25T00:01:04.365881+0000","last_became_peered":"2024-04-25T00:01:04.365881+0000","last_unstale":"2024-04-25T00:01:20.519859+0000","last_undegraded":"2024-04-25T00:01:20.519859+0000","last_fullsized":"2024-04-25T00:01:20.519859+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:49:22.673421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.577736+0000","last_change":"2024-04-25T00:01:18.504341+0000","last_active":"2024-04-25T00:01:20.577736+0000","last_peered":"2024-04-25T00:01:20.577736+0000","last_clean":"2024-04-25T00:01:20.577736+0000","last_became_active":"2024-04-25T00:01:18.491498+0000","last_became_peered":"2024-04-25T00:01:18.491498+0000","last_unstale":"2024-04-25T00:01:20.577736+0000","last_undegraded":"2024-04-25T00:01:20.577736+0000","last_fullsized":"2024-04-25T00:01:20.577736+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:56.741250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.513341+0000","last_change":"2024-04-25T00:01:19.539074+0000","last_active":"2024-04-25T00:01:20.513341+0000","last_peered":"2024-04-25T00:01:20.513341+0000","last_clean":"2024-04-25T00:01:20.513341+0000","last_became_active":"2024-04-25T00:01:19.538447+0000","last_became_peered":"2024-04-25T00:01:19.538447+0000","last_unstale":"2024-04-25T00:01:20.513341+0000","last_undegraded":"2024-04-25T00:01:20.513341+0000","last_fullsized":"2024-04-25T00:01:20.513341+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:08:49.640267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.581346+0000","last_change":"2024-04-25T00:01:05.368769+0000","last_active":"2024-04-25T00:01:20.581346+0000","last_peered":"2024-04-25T00:01:20.581346+0000","last_clean":"2024-04-25T00:01:20.581346+0000","last_became_active":"2024-04-25T00:01:05.366817+0000","last_became_peered":"2024-04-25T00:01:05.366817+0000","last_unstale":"2024-04-25T00:01:20.581346+0000","last_undegraded":"2024-04-25T00:01:20.581346+0000","last_fullsized":"2024-04-25T00:01:20.581346+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:11:27.044893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.581006+0000","last_change":"2024-04-25T00:01:04.361900+0000","last_active":"2024-04-25T00:01:20.581006+0000","last_peered":"2024-04-25T00:01:20.581006+0000","last_clean":"2024-04-25T00:01:20.581006+0000","last_became_active":"2024-04-25T00:01:04.361572+0000","last_became_peered":"2024-04-25T00:01:04.361572+0000","last_unstale":"2024-04-25T00:01:20.581006+0000","last_undegraded":"2024-04-25T00:01:20.581006+0000","last_fullsized":"2024-04-25T00:01:20.581006+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:29.218739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.579029+0000","last_change":"2024-04-25T00:01:18.504139+0000","last_active":"2024-04-25T00:01:20.579029+0000","last_peered":"2024-04-25T00:01:20.579029+0000","last_clean":"2024-04-25T00:01:20.579029+0000","last_became_active":"2024-04-25T00:01:18.503379+0000","last_became_peered":"2024-04-25T00:01:18.503379+0000","last_unstale":"2024-04-25T00:01:20.579029+0000","last_undegraded":"2024-04-25T00:01:20.579029+0000","last_fullsized":"2024-04-25T00:01:20.579029+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:02:56.363258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.514229+0000","last_change":"2024-04-25T00:01:19.531756+0000","last_active":"2024-04-25T00:01:20.514229+0000","last_peered":"2024-04-25T00:01:20.514229+0000","last_clean":"2024-04-25T00:01:20.514229+0000","last_became_active":"2024-04-25T00:01:19.531145+0000","last_became_peered":"2024-04-25T00:01:19.531145+0000","last_unstale":"2024-04-25T00:01:20.514229+0000","last_undegraded":"2024-04-25T00:01:20.514229+0000","last_fullsized":"2024-04-25T00:01:20.514229+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:13:42.678415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.518780+0000","last_change":"2024-04-25T00:01:05.354557+0000","last_active":"2024-04-25T00:01:20.518780+0000","last_peered":"2024-04-25T00:01:20.518780+0000","last_clean":"2024-04-25T00:01:20.518780+0000","last_became_active":"2024-04-25T00:01:05.353556+0000","last_became_peered":"2024-04-25T00:01:05.353556+0000","last_unstale":"2024-04-25T00:01:20.518780+0000","last_undegraded":"2024-04-25T00:01:20.518780+0000","last_fullsized":"2024-04-25T00:01:20.518780+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:24.964055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511447+0000","last_change":"2024-04-25T00:01:04.372916+0000","last_active":"2024-04-25T00:01:20.511447+0000","last_peered":"2024-04-25T00:01:20.511447+0000","last_clean":"2024-04-25T00:01:20.511447+0000","last_became_active":"2024-04-25T00:01:04.372185+0000","last_became_peered":"2024-04-25T00:01:04.372185+0000","last_unstale":"2024-04-25T00:01:20.511447+0000","last_undegraded":"2024-04-25T00:01:20.511447+0000","last_fullsized":"2024-04-25T00:01:20.511447+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"26'2","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.579299+0000","last_change":"2024-04-25T00:01:18.504614+0000","last_active":"2024-04-25T00:01:20.579299+0000","last_peered":"2024-04-25T00:01:20.579299+0000","last_clean":"2024-04-25T00:01:20.579299+0000","last_became_active":"2024-04-25T00:01:18.501448+0000","last_became_peered":"2024-04-25T00:01:18.501448+0000","last_unstale":"2024-04-25T00:01:20.579299+0000","last_undegraded":"2024-04-25T00:01:20.579299+0000","last_fullsized":"2024-04-25T00:01:20.579299+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:22.983972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.584367+0000","last_change":"2024-04-25T00:01:19.517510+0000","last_active":"2024-04-25T00:01:20.584367+0000","last_peered":"2024-04-25T00:01:20.584367+0000","last_clean":"2024-04-25T00:01:20.584367+0000","last_became_active":"2024-04-25T00:01:19.517243+0000","last_became_peered":"2024-04-25T00:01:19.517243+0000","last_unstale":"2024-04-25T00:01:20.584367+0000","last_undegraded":"2024-04-25T00:01:20.584367+0000","last_fullsized":"2024-04-25T00:01:20.584367+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:42:38.149502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.577555+0000","last_change":"2024-04-25T00:01:05.370968+0000","last_active":"2024-04-25T00:01:20.577555+0000","last_peered":"2024-04-25T00:01:20.577555+0000","last_clean":"2024-04-25T00:01:20.577555+0000","last_became_active":"2024-04-25T00:01:05.369987+0000","last_became_peered":"2024-04-25T00:01:05.369987+0000","last_unstale":"2024-04-25T00:01:20.577555+0000","last_undegraded":"2024-04-25T00:01:20.577555+0000","last_fullsized":"2024-04-25T00:01:20.577555+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.580498+0000","last_change":"2024-04-25T00:01:04.370226+0000","last_active":"2024-04-25T00:01:20.580498+0000","last_peered":"2024-04-25T00:01:20.580498+0000","last_clean":"2024-04-25T00:01:20.580498+0000","last_became_active":"2024-04-25T00:01:04.369904+0000","last_became_peered":"2024-04-25T00:01:04.369904+0000","last_unstale":"2024-04-25T00:01:20.580498+0000","last_undegraded":"2024-04-25T00:01:20.580498+0000","last_fullsized":"2024-04-25T00:01:20.580498+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:47:25.595275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"27'1","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.523846+0000","last_change":"2024-04-25T00:01:18.497156+0000","last_active":"2024-04-25T00:01:20.523846+0000","last_peered":"2024-04-25T00:01:20.523846+0000","last_clean":"2024-04-25T00:01:20.523846+0000","last_became_active":"2024-04-25T00:01:18.495573+0000","last_became_peered":"2024-04-25T00:01:18.495573+0000","last_unstale":"2024-04-25T00:01:20.523846+0000","last_undegraded":"2024-04-25T00:01:20.523846+0000","last_fullsized":"2024-04-25T00:01:20.523846+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:29.119299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.581909+0000","last_change":"2024-04-25T00:01:19.519411+0000","last_active":"2024-04-25T00:01:20.581909+0000","last_peered":"2024-04-25T00:01:20.581909+0000","last_clean":"2024-04-25T00:01:20.581909+0000","last_became_active":"2024-04-25T00:01:19.519020+0000","last_became_peered":"2024-04-25T00:01:19.519020+0000","last_unstale":"2024-04-25T00:01:20.581909+0000","last_undegraded":"2024-04-25T00:01:20.581909+0000","last_fullsized":"2024-04-25T00:01:20.581909+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:50.351599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T00:01:19.496766+0000","last_change":"2024-04-25T00:01:05.364939+0000","last_active":"2024-04-25T00:01:19.496766+0000","last_peered":"2024-04-25T00:01:19.496766+0000","last_clean":"2024-04-25T00:01:19.496766+0000","last_became_active":"2024-04-25T00:01:05.364272+0000","last_became_peered":"2024-04-25T00:01:05.364272+0000","last_unstale":"2024-04-25T00:01:19.496766+0000","last_undegraded":"2024-04-25T00:01:19.496766+0000","last_fullsized":"2024-04-25T00:01:19.496766+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:24:59.679908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.581536+0000","last_change":"2024-04-25T00:01:04.357028+0000","last_active":"2024-04-25T00:01:20.581536+0000","last_peered":"2024-04-25T00:01:20.581536+0000","last_clean":"2024-04-25T00:01:20.581536+0000","last_became_active":"2024-04-25T00:01:04.356644+0000","last_became_peered":"2024-04-25T00:01:04.356644+0000","last_unstale":"2024-04-25T00:01:20.581536+0000","last_undegraded":"2024-04-25T00:01:20.581536+0000","last_fullsized":"2024-04-25T00:01:20.581536+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:21.729893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":178,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.577513+0000","last_change":"2024-04-25T00:00:55.084492+0000","last_active":"2024-04-25T00:01:20.577513+0000","last_peered":"2024-04-25T00:01:20.577513+0000","last_clean":"2024-04-25T00:01:20.577513+0000","last_became_active":"2024-04-25T00:00:55.083493+0000","last_became_peered":"2024-04-25T00:00:55.083493+0000","last_unstale":"2024-04-25T00:01:20.577513+0000","last_undegraded":"2024-04-25T00:01:20.577513+0000","last_fullsized":"2024-04-25T00:01:20.577513+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:00:54.060578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:00:54.060578+0000","last_clean_scrub_stamp":"2024-04-25T00:00:54.060578+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:14:58.472477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.583214+0000","last_change":"2024-04-25T00:01:18.476298+0000","last_active":"2024-04-25T00:01:20.583214+0000","last_peered":"2024-04-25T00:01:20.583214+0000","last_clean":"2024-04-25T00:01:20.583214+0000","last_became_active":"2024-04-25T00:01:18.475898+0000","last_became_peered":"2024-04-25T00:01:18.475898+0000","last_unstale":"2024-04-25T00:01:20.583214+0000","last_undegraded":"2024-04-25T00:01:20.583214+0000","last_fullsized":"2024-04-25T00:01:20.583214+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:49.341842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.519562+0000","last_change":"2024-04-25T00:01:19.518102+0000","last_active":"2024-04-25T00:01:20.519562+0000","last_peered":"2024-04-25T00:01:20.519562+0000","last_clean":"2024-04-25T00:01:20.519562+0000","last_became_active":"2024-04-25T00:01:19.517376+0000","last_became_peered":"2024-04-25T00:01:19.517376+0000","last_unstale":"2024-04-25T00:01:20.519562+0000","last_undegraded":"2024-04-25T00:01:20.519562+0000","last_fullsized":"2024-04-25T00:01:20.519562+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:55:46.853565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.583333+0000","last_change":"2024-04-25T00:01:05.376609+0000","last_active":"2024-04-25T00:01:20.583333+0000","last_peered":"2024-04-25T00:01:20.583333+0000","last_clean":"2024-04-25T00:01:20.583333+0000","last_became_active":"2024-04-25T00:01:05.375881+0000","last_became_peered":"2024-04-25T00:01:05.375881+0000","last_unstale":"2024-04-25T00:01:20.583333+0000","last_undegraded":"2024-04-25T00:01:20.583333+0000","last_fullsized":"2024-04-25T00:01:20.583333+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:46:05.592233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.579387+0000","last_change":"2024-04-25T00:01:04.368954+0000","last_active":"2024-04-25T00:01:20.579387+0000","last_peered":"2024-04-25T00:01:20.579387+0000","last_clean":"2024-04-25T00:01:20.579387+0000","last_became_active":"2024-04-25T00:01:04.367903+0000","last_became_peered":"2024-04-25T00:01:04.367903+0000","last_unstale":"2024-04-25T00:01:20.579387+0000","last_undegraded":"2024-04-25T00:01:20.579387+0000","last_fullsized":"2024-04-25T00:01:20.579387+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511154+0000","last_change":"2024-04-25T00:01:18.476525+0000","last_active":"2024-04-25T00:01:20.511154+0000","last_peered":"2024-04-25T00:01:20.511154+0000","last_clean":"2024-04-25T00:01:20.511154+0000","last_became_active":"2024-04-25T00:01:18.476247+0000","last_became_peered":"2024-04-25T00:01:18.476247+0000","last_unstale":"2024-04-25T00:01:20.511154+0000","last_undegraded":"2024-04-25T00:01:20.511154+0000","last_fullsized":"2024-04-25T00:01:20.511154+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:30:36.682736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.577886+0000","last_change":"2024-04-25T00:01:19.514615+0000","last_active":"2024-04-25T00:01:20.577886+0000","last_peered":"2024-04-25T00:01:20.577886+0000","last_clean":"2024-04-25T00:01:20.577886+0000","last_became_active":"2024-04-25T00:01:19.513640+0000","last_became_peered":"2024-04-25T00:01:19.513640+0000","last_unstale":"2024-04-25T00:01:20.577886+0000","last_undegraded":"2024-04-25T00:01:20.577886+0000","last_fullsized":"2024-04-25T00:01:20.577886+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:08.620602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582498+0000","last_change":"2024-04-25T00:01:05.385670+0000","last_active":"2024-04-25T00:01:20.582498+0000","last_peered":"2024-04-25T00:01:20.582498+0000","last_clean":"2024-04-25T00:01:20.582498+0000","last_became_active":"2024-04-25T00:01:05.384822+0000","last_became_peered":"2024-04-25T00:01:05.384822+0000","last_unstale":"2024-04-25T00:01:20.582498+0000","last_undegraded":"2024-04-25T00:01:20.582498+0000","last_fullsized":"2024-04-25T00:01:20.582498+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:24.017263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"22'1","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.520366+0000","last_change":"2024-04-25T00:01:04.356627+0000","last_active":"2024-04-25T00:01:20.520366+0000","last_peered":"2024-04-25T00:01:20.520366+0000","last_clean":"2024-04-25T00:01:20.520366+0000","last_became_active":"2024-04-25T00:01:04.354419+0000","last_became_peered":"2024-04-25T00:01:04.354419+0000","last_unstale":"2024-04-25T00:01:20.520366+0000","last_undegraded":"2024-04-25T00:01:20.520366+0000","last_fullsized":"2024-04-25T00:01:20.520366+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:26.243635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"26'2","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.519762+0000","last_change":"2024-04-25T00:01:18.490511+0000","last_active":"2024-04-25T00:01:20.519762+0000","last_peered":"2024-04-25T00:01:20.519762+0000","last_clean":"2024-04-25T00:01:20.519762+0000","last_became_active":"2024-04-25T00:01:18.489629+0000","last_became_peered":"2024-04-25T00:01:18.489629+0000","last_unstale":"2024-04-25T00:01:20.519762+0000","last_undegraded":"2024-04-25T00:01:20.519762+0000","last_fullsized":"2024-04-25T00:01:20.519762+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:07.610846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.579469+0000","last_change":"2024-04-25T00:01:19.508931+0000","last_active":"2024-04-25T00:01:20.579469+0000","last_peered":"2024-04-25T00:01:20.579469+0000","last_clean":"2024-04-25T00:01:20.579469+0000","last_became_active":"2024-04-25T00:01:19.507867+0000","last_became_peered":"2024-04-25T00:01:19.507867+0000","last_unstale":"2024-04-25T00:01:20.579469+0000","last_undegraded":"2024-04-25T00:01:20.579469+0000","last_fullsized":"2024-04-25T00:01:20.579469+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:38.152376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.583381+0000","last_change":"2024-04-25T00:01:05.378840+0000","last_active":"2024-04-25T00:01:20.583381+0000","last_peered":"2024-04-25T00:01:20.583381+0000","last_clean":"2024-04-25T00:01:20.583381+0000","last_became_active":"2024-04-25T00:01:05.378112+0000","last_became_peered":"2024-04-25T00:01:05.378112+0000","last_unstale":"2024-04-25T00:01:20.583381+0000","last_undegraded":"2024-04-25T00:01:20.583381+0000","last_fullsized":"2024-04-25T00:01:20.583381+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:43.615895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"22'2","reported_seq":22,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T00:01:19.497702+0000","last_change":"2024-04-25T00:01:04.370794+0000","last_active":"2024-04-25T00:01:19.497702+0000","last_peered":"2024-04-25T00:01:19.497702+0000","last_clean":"2024-04-25T00:01:19.497702+0000","last_became_active":"2024-04-25T00:01:04.370099+0000","last_became_peered":"2024-04-25T00:01:04.370099+0000","last_unstale":"2024-04-25T00:01:19.497702+0000","last_undegraded":"2024-04-25T00:01:19.497702+0000","last_fullsized":"2024-04-25T00:01:19.497702+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:42:03.181036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.579618+0000","last_change":"2024-04-25T00:01:18.475256+0000","last_active":"2024-04-25T00:01:20.579618+0000","last_peered":"2024-04-25T00:01:20.579618+0000","last_clean":"2024-04-25T00:01:20.579618+0000","last_became_active":"2024-04-25T00:01:18.474952+0000","last_became_peered":"2024-04-25T00:01:18.474952+0000","last_unstale":"2024-04-25T00:01:20.579618+0000","last_undegraded":"2024-04-25T00:01:20.579618+0000","last_fullsized":"2024-04-25T00:01:20.579618+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:46.849870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.577592+0000","last_change":"2024-04-25T00:01:19.516833+0000","last_active":"2024-04-25T00:01:20.577592+0000","last_peered":"2024-04-25T00:01:20.577592+0000","last_clean":"2024-04-25T00:01:20.577592+0000","last_became_active":"2024-04-25T00:01:19.515926+0000","last_became_peered":"2024-04-25T00:01:19.515926+0000","last_unstale":"2024-04-25T00:01:20.577592+0000","last_undegraded":"2024-04-25T00:01:20.577592+0000","last_fullsized":"2024-04-25T00:01:20.577592+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:23:48.855381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511807+0000","last_change":"2024-04-25T00:01:05.372591+0000","last_active":"2024-04-25T00:01:20.511807+0000","last_peered":"2024-04-25T00:01:20.511807+0000","last_clean":"2024-04-25T00:01:20.511807+0000","last_became_active":"2024-04-25T00:01:05.371204+0000","last_became_peered":"2024-04-25T00:01:05.371204+0000","last_unstale":"2024-04-25T00:01:20.511807+0000","last_undegraded":"2024-04-25T00:01:20.511807+0000","last_fullsized":"2024-04-25T00:01:20.511807+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:15.879673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"22'1","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.583451+0000","last_change":"2024-04-25T00:01:04.372376+0000","last_active":"2024-04-25T00:01:20.583451+0000","last_peered":"2024-04-25T00:01:20.583451+0000","last_clean":"2024-04-25T00:01:20.583451+0000","last_became_active":"2024-04-25T00:01:04.372040+0000","last_became_peered":"2024-04-25T00:01:04.372040+0000","last_unstale":"2024-04-25T00:01:20.583451+0000","last_undegraded":"2024-04-25T00:01:20.583451+0000","last_fullsized":"2024-04-25T00:01:20.583451+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:21.729893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578134+0000","last_change":"2024-04-25T00:01:18.487548+0000","last_active":"2024-04-25T00:01:20.578134+0000","last_peered":"2024-04-25T00:01:20.578134+0000","last_clean":"2024-04-25T00:01:20.578134+0000","last_became_active":"2024-04-25T00:01:18.487227+0000","last_became_peered":"2024-04-25T00:01:18.487227+0000","last_unstale":"2024-04-25T00:01:20.578134+0000","last_undegraded":"2024-04-25T00:01:20.578134+0000","last_fullsized":"2024-04-25T00:01:20.578134+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:55.074018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T00:01:19.542702+0000","last_change":"2024-04-25T00:01:19.528574+0000","last_active":"2024-04-25T00:01:19.542702+0000","last_peered":"2024-04-25T00:01:19.542702+0000","last_clean":"2024-04-25T00:01:19.542702+0000","last_became_active":"2024-04-25T00:01:19.525531+0000","last_became_peered":"2024-04-25T00:01:19.525531+0000","last_unstale":"2024-04-25T00:01:19.542702+0000","last_undegraded":"2024-04-25T00:01:19.542702+0000","last_fullsized":"2024-04-25T00:01:19.542702+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:20:40.636536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.580931+0000","last_change":"2024-04-25T00:01:05.364103+0000","last_active":"2024-04-25T00:01:20.580931+0000","last_peered":"2024-04-25T00:01:20.580931+0000","last_clean":"2024-04-25T00:01:20.580931+0000","last_became_active":"2024-04-25T00:01:05.363320+0000","last_became_peered":"2024-04-25T00:01:05.363320+0000","last_unstale":"2024-04-25T00:01:20.580931+0000","last_undegraded":"2024-04-25T00:01:20.580931+0000","last_fullsized":"2024-04-25T00:01:20.580931+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:31.456792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"22'8","reported_seq":28,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T00:01:19.494273+0000","last_change":"2024-04-25T00:01:04.358365+0000","last_active":"2024-04-25T00:01:19.494273+0000","last_peered":"2024-04-25T00:01:19.494273+0000","last_clean":"2024-04-25T00:01:19.494273+0000","last_became_active":"2024-04-25T00:01:04.357571+0000","last_became_peered":"2024-04-25T00:01:04.357571+0000","last_unstale":"2024-04-25T00:01:19.494273+0000","last_undegraded":"2024-04-25T00:01:19.494273+0000","last_fullsized":"2024-04-25T00:01:19.494273+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:50:23.023114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.520269+0000","last_change":"2024-04-25T00:01:18.497679+0000","last_active":"2024-04-25T00:01:20.520269+0000","last_peered":"2024-04-25T00:01:20.520269+0000","last_clean":"2024-04-25T00:01:20.520269+0000","last_became_active":"2024-04-25T00:01:18.496489+0000","last_became_peered":"2024-04-25T00:01:18.496489+0000","last_unstale":"2024-04-25T00:01:20.520269+0000","last_undegraded":"2024-04-25T00:01:20.520269+0000","last_fullsized":"2024-04-25T00:01:20.520269+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:23:47.845624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.584174+0000","last_change":"2024-04-25T00:01:19.498082+0000","last_active":"2024-04-25T00:01:20.584174+0000","last_peered":"2024-04-25T00:01:20.584174+0000","last_clean":"2024-04-25T00:01:20.584174+0000","last_became_active":"2024-04-25T00:01:19.496805+0000","last_became_peered":"2024-04-25T00:01:19.496805+0000","last_unstale":"2024-04-25T00:01:20.584174+0000","last_undegraded":"2024-04-25T00:01:20.584174+0000","last_fullsized":"2024-04-25T00:01:20.584174+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:57:36.256250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.577453+0000","last_change":"2024-04-25T00:01:05.373927+0000","last_active":"2024-04-25T00:01:20.577453+0000","last_peered":"2024-04-25T00:01:20.577453+0000","last_clean":"2024-04-25T00:01:20.577453+0000","last_became_active":"2024-04-25T00:01:05.373077+0000","last_became_peered":"2024-04-25T00:01:05.373077+0000","last_unstale":"2024-04-25T00:01:20.577453+0000","last_undegraded":"2024-04-25T00:01:20.577453+0000","last_fullsized":"2024-04-25T00:01:20.577453+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:50.489125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578902+0000","last_change":"2024-04-25T00:01:04.357265+0000","last_active":"2024-04-25T00:01:20.578902+0000","last_peered":"2024-04-25T00:01:20.578902+0000","last_clean":"2024-04-25T00:01:20.578902+0000","last_became_active":"2024-04-25T00:01:04.357004+0000","last_became_peered":"2024-04-25T00:01:04.357004+0000","last_unstale":"2024-04-25T00:01:20.578902+0000","last_undegraded":"2024-04-25T00:01:20.578902+0000","last_fullsized":"2024-04-25T00:01:20.578902+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:47:25.595275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.520612+0000","last_change":"2024-04-25T00:01:18.487276+0000","last_active":"2024-04-25T00:01:20.520612+0000","last_peered":"2024-04-25T00:01:20.520612+0000","last_clean":"2024-04-25T00:01:20.520612+0000","last_became_active":"2024-04-25T00:01:18.486279+0000","last_became_peered":"2024-04-25T00:01:18.486279+0000","last_unstale":"2024-04-25T00:01:20.520612+0000","last_undegraded":"2024-04-25T00:01:20.520612+0000","last_fullsized":"2024-04-25T00:01:20.520612+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:17:37.141843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582117+0000","last_change":"2024-04-25T00:01:19.516827+0000","last_active":"2024-04-25T00:01:20.582117+0000","last_peered":"2024-04-25T00:01:20.582117+0000","last_clean":"2024-04-25T00:01:20.582117+0000","last_became_active":"2024-04-25T00:01:19.515765+0000","last_became_peered":"2024-04-25T00:01:19.515765+0000","last_unstale":"2024-04-25T00:01:20.582117+0000","last_undegraded":"2024-04-25T00:01:20.582117+0000","last_fullsized":"2024-04-25T00:01:20.582117+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:13:34.153117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.579960+0000","last_change":"2024-04-25T00:01:05.379501+0000","last_active":"2024-04-25T00:01:20.579960+0000","last_peered":"2024-04-25T00:01:20.579960+0000","last_clean":"2024-04-25T00:01:20.579960+0000","last_became_active":"2024-04-25T00:01:05.378644+0000","last_became_peered":"2024-04-25T00:01:05.378644+0000","last_unstale":"2024-04-25T00:01:20.579960+0000","last_undegraded":"2024-04-25T00:01:20.579960+0000","last_fullsized":"2024-04-25T00:01:20.579960+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:08:35.505154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.580791+0000","last_change":"2024-04-25T00:01:04.360987+0000","last_active":"2024-04-25T00:01:20.580791+0000","last_peered":"2024-04-25T00:01:20.580791+0000","last_clean":"2024-04-25T00:01:20.580791+0000","last_became_active":"2024-04-25T00:01:04.360664+0000","last_became_peered":"2024-04-25T00:01:04.360664+0000","last_unstale":"2024-04-25T00:01:20.580791+0000","last_undegraded":"2024-04-25T00:01:20.580791+0000","last_fullsized":"2024-04-25T00:01:20.580791+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:55.283076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"26'6","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578827+0000","last_change":"2024-04-25T00:01:18.505577+0000","last_active":"2024-04-25T00:01:20.578827+0000","last_peered":"2024-04-25T00:01:20.578827+0000","last_clean":"2024-04-25T00:01:20.578827+0000","last_became_active":"2024-04-25T00:01:18.504963+0000","last_became_peered":"2024-04-25T00:01:18.504963+0000","last_unstale":"2024-04-25T00:01:20.578827+0000","last_undegraded":"2024-04-25T00:01:20.578827+0000","last_fullsized":"2024-04-25T00:01:20.578827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:37.142619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T00:01:19.541800+0000","last_change":"2024-04-25T00:01:19.524723+0000","last_active":"2024-04-25T00:01:19.541800+0000","last_peered":"2024-04-25T00:01:19.541800+0000","last_clean":"2024-04-25T00:01:19.541800+0000","last_became_active":"2024-04-25T00:01:19.523753+0000","last_became_peered":"2024-04-25T00:01:19.523753+0000","last_unstale":"2024-04-25T00:01:19.541800+0000","last_undegraded":"2024-04-25T00:01:19.541800+0000","last_fullsized":"2024-04-25T00:01:19.541800+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:37:16.164314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578487+0000","last_change":"2024-04-25T00:01:05.379102+0000","last_active":"2024-04-25T00:01:20.578487+0000","last_peered":"2024-04-25T00:01:20.578487+0000","last_clean":"2024-04-25T00:01:20.578487+0000","last_became_active":"2024-04-25T00:01:05.377625+0000","last_became_peered":"2024-04-25T00:01:05.377625+0000","last_unstale":"2024-04-25T00:01:20.578487+0000","last_undegraded":"2024-04-25T00:01:20.578487+0000","last_fullsized":"2024-04-25T00:01:20.578487+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:15.879673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"22'2","reported_seq":24,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511726+0000","last_change":"2024-04-25T00:01:04.370640+0000","last_active":"2024-04-25T00:01:20.511726+0000","last_peered":"2024-04-25T00:01:20.511726+0000","last_clean":"2024-04-25T00:01:20.511726+0000","last_became_active":"2024-04-25T00:01:04.370031+0000","last_became_peered":"2024-04-25T00:01:04.370031+0000","last_unstale":"2024-04-25T00:01:20.511726+0000","last_undegraded":"2024-04-25T00:01:20.511726+0000","last_fullsized":"2024-04-25T00:01:20.511726+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:26.243635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"26'3","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582955+0000","last_change":"2024-04-25T00:01:18.473113+0000","last_active":"2024-04-25T00:01:20.582955+0000","last_peered":"2024-04-25T00:01:20.582955+0000","last_clean":"2024-04-25T00:01:20.582955+0000","last_became_active":"2024-04-25T00:01:18.472931+0000","last_became_peered":"2024-04-25T00:01:18.472931+0000","last_unstale":"2024-04-25T00:01:20.582955+0000","last_undegraded":"2024-04-25T00:01:20.582955+0000","last_fullsized":"2024-04-25T00:01:20.582955+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:07.498985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T00:01:19.542557+0000","last_change":"2024-04-25T00:01:19.533664+0000","last_active":"2024-04-25T00:01:19.542557+0000","last_peered":"2024-04-25T00:01:19.542557+0000","last_clean":"2024-04-25T00:01:19.542557+0000","last_became_active":"2024-04-25T00:01:19.533048+0000","last_became_peered":"2024-04-25T00:01:19.533048+0000","last_unstale":"2024-04-25T00:01:19.542557+0000","last_undegraded":"2024-04-25T00:01:19.542557+0000","last_fullsized":"2024-04-25T00:01:19.542557+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:30:19.851252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582298+0000","last_change":"2024-04-25T00:01:05.385827+0000","last_active":"2024-04-25T00:01:20.582298+0000","last_peered":"2024-04-25T00:01:20.582298+0000","last_clean":"2024-04-25T00:01:20.582298+0000","last_became_active":"2024-04-25T00:01:05.385258+0000","last_became_peered":"2024-04-25T00:01:05.385258+0000","last_unstale":"2024-04-25T00:01:20.582298+0000","last_undegraded":"2024-04-25T00:01:20.582298+0000","last_fullsized":"2024-04-25T00:01:20.582298+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:23.870121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"22'6","reported_seq":28,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.512128+0000","last_change":"2024-04-25T00:01:04.365185+0000","last_active":"2024-04-25T00:01:20.512128+0000","last_peered":"2024-04-25T00:01:20.512128+0000","last_clean":"2024-04-25T00:01:20.512128+0000","last_became_active":"2024-04-25T00:01:04.364930+0000","last_became_peered":"2024-04-25T00:01:04.364930+0000","last_unstale":"2024-04-25T00:01:20.512128+0000","last_undegraded":"2024-04-25T00:01:20.512128+0000","last_fullsized":"2024-04-25T00:01:20.512128+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:45:03.618340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T00:01:19.496459+0000","last_change":"2024-04-25T00:01:18.496220+0000","last_active":"2024-04-25T00:01:19.496459+0000","last_peered":"2024-04-25T00:01:19.496459+0000","last_clean":"2024-04-25T00:01:19.496459+0000","last_became_active":"2024-04-25T00:01:18.492850+0000","last_became_peered":"2024-04-25T00:01:18.492850+0000","last_unstale":"2024-04-25T00:01:19.496459+0000","last_undegraded":"2024-04-25T00:01:19.496459+0000","last_fullsized":"2024-04-25T00:01:19.496459+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:50:16.922314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578332+0000","last_change":"2024-04-25T00:01:19.515725+0000","last_active":"2024-04-25T00:01:20.578332+0000","last_peered":"2024-04-25T00:01:20.578332+0000","last_clean":"2024-04-25T00:01:20.578332+0000","last_became_active":"2024-04-25T00:01:19.509285+0000","last_became_peered":"2024-04-25T00:01:19.509285+0000","last_unstale":"2024-04-25T00:01:20.578332+0000","last_undegraded":"2024-04-25T00:01:20.578332+0000","last_fullsized":"2024-04-25T00:01:20.578332+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:25:20.185355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.581530+0000","last_change":"2024-04-25T00:01:05.371061+0000","last_active":"2024-04-25T00:01:20.581530+0000","last_peered":"2024-04-25T00:01:20.581530+0000","last_clean":"2024-04-25T00:01:20.581530+0000","last_became_active":"2024-04-25T00:01:05.369950+0000","last_became_peered":"2024-04-25T00:01:05.369950+0000","last_unstale":"2024-04-25T00:01:20.581530+0000","last_undegraded":"2024-04-25T00:01:20.581530+0000","last_fullsized":"2024-04-25T00:01:20.581530+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:02:43.237902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"22'3","reported_seq":25,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.513687+0000","last_change":"2024-04-25T00:01:04.366503+0000","last_active":"2024-04-25T00:01:20.513687+0000","last_peered":"2024-04-25T00:01:20.513687+0000","last_clean":"2024-04-25T00:01:20.513687+0000","last_became_active":"2024-04-25T00:01:04.365945+0000","last_became_peered":"2024-04-25T00:01:04.365945+0000","last_unstale":"2024-04-25T00:01:20.513687+0000","last_undegraded":"2024-04-25T00:01:20.513687+0000","last_fullsized":"2024-04-25T00:01:20.513687+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"26'2","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511434+0000","last_change":"2024-04-25T00:01:18.476456+0000","last_active":"2024-04-25T00:01:20.511434+0000","last_peered":"2024-04-25T00:01:20.511434+0000","last_clean":"2024-04-25T00:01:20.511434+0000","last_became_active":"2024-04-25T00:01:18.475876+0000","last_became_peered":"2024-04-25T00:01:18.475876+0000","last_unstale":"2024-04-25T00:01:20.511434+0000","last_undegraded":"2024-04-25T00:01:20.511434+0000","last_fullsized":"2024-04-25T00:01:20.511434+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:56.741250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.580191+0000","last_change":"2024-04-25T00:01:19.515376+0000","last_active":"2024-04-25T00:01:20.580191+0000","last_peered":"2024-04-25T00:01:20.580191+0000","last_clean":"2024-04-25T00:01:20.580191+0000","last_became_active":"2024-04-25T00:01:19.512425+0000","last_became_peered":"2024-04-25T00:01:19.512425+0000","last_unstale":"2024-04-25T00:01:20.580191+0000","last_undegraded":"2024-04-25T00:01:20.580191+0000","last_fullsized":"2024-04-25T00:01:20.580191+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:43:16.957273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.520943+0000","last_change":"2024-04-25T00:01:05.361723+0000","last_active":"2024-04-25T00:01:20.520943+0000","last_peered":"2024-04-25T00:01:20.520943+0000","last_clean":"2024-04-25T00:01:20.520943+0000","last_became_active":"2024-04-25T00:01:05.360956+0000","last_became_peered":"2024-04-25T00:01:05.360956+0000","last_unstale":"2024-04-25T00:01:20.520943+0000","last_undegraded":"2024-04-25T00:01:20.520943+0000","last_fullsized":"2024-04-25T00:01:20.520943+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:23.870121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.580983+0000","last_change":"2024-04-25T00:01:04.362129+0000","last_active":"2024-04-25T00:01:20.580983+0000","last_peered":"2024-04-25T00:01:20.580983+0000","last_clean":"2024-04-25T00:01:20.580983+0000","last_became_active":"2024-04-25T00:01:04.361827+0000","last_became_peered":"2024-04-25T00:01:04.361827+0000","last_unstale":"2024-04-25T00:01:20.580983+0000","last_undegraded":"2024-04-25T00:01:20.580983+0000","last_fullsized":"2024-04-25T00:01:20.580983+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:26.243635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"26'8","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.510624+0000","last_change":"2024-04-25T00:01:18.496129+0000","last_active":"2024-04-25T00:01:20.510624+0000","last_peered":"2024-04-25T00:01:20.510624+0000","last_clean":"2024-04-25T00:01:20.510624+0000","last_became_active":"2024-04-25T00:01:18.495415+0000","last_became_peered":"2024-04-25T00:01:18.495415+0000","last_unstale":"2024-04-25T00:01:20.510624+0000","last_undegraded":"2024-04-25T00:01:20.510624+0000","last_fullsized":"2024-04-25T00:01:20.510624+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:38.089411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.510763+0000","last_change":"2024-04-25T00:01:19.509542+0000","last_active":"2024-04-25T00:01:20.510763+0000","last_peered":"2024-04-25T00:01:20.510763+0000","last_clean":"2024-04-25T00:01:20.510763+0000","last_became_active":"2024-04-25T00:01:19.509015+0000","last_became_peered":"2024-04-25T00:01:19.509015+0000","last_unstale":"2024-04-25T00:01:20.510763+0000","last_undegraded":"2024-04-25T00:01:20.510763+0000","last_fullsized":"2024-04-25T00:01:20.510763+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:25:20.185355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511808+0000","last_change":"2024-04-25T00:01:05.365413+0000","last_active":"2024-04-25T00:01:20.511808+0000","last_peered":"2024-04-25T00:01:20.511808+0000","last_clean":"2024-04-25T00:01:20.511808+0000","last_became_active":"2024-04-25T00:01:05.361865+0000","last_became_peered":"2024-04-25T00:01:05.361865+0000","last_unstale":"2024-04-25T00:01:20.511808+0000","last_undegraded":"2024-04-25T00:01:20.511808+0000","last_fullsized":"2024-04-25T00:01:20.511808+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:30.228060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582499+0000","last_change":"2024-04-25T00:01:04.360804+0000","last_active":"2024-04-25T00:01:20.582499+0000","last_peered":"2024-04-25T00:01:20.582499+0000","last_clean":"2024-04-25T00:01:20.582499+0000","last_became_active":"2024-04-25T00:01:04.360374+0000","last_became_peered":"2024-04-25T00:01:04.360374+0000","last_unstale":"2024-04-25T00:01:20.582499+0000","last_undegraded":"2024-04-25T00:01:20.582499+0000","last_fullsized":"2024-04-25T00:01:20.582499+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T00:01:19.494208+0000","last_change":"2024-04-25T00:01:18.480003+0000","last_active":"2024-04-25T00:01:19.494208+0000","last_peered":"2024-04-25T00:01:19.494208+0000","last_clean":"2024-04-25T00:01:19.494208+0000","last_became_active":"2024-04-25T00:01:18.479270+0000","last_became_peered":"2024-04-25T00:01:18.479270+0000","last_unstale":"2024-04-25T00:01:19.494208+0000","last_undegraded":"2024-04-25T00:01:19.494208+0000","last_fullsized":"2024-04-25T00:01:19.494208+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:52:40.942502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511885+0000","last_change":"2024-04-25T00:01:19.524463+0000","last_active":"2024-04-25T00:01:20.511885+0000","last_peered":"2024-04-25T00:01:20.511885+0000","last_clean":"2024-04-25T00:01:20.511885+0000","last_became_active":"2024-04-25T00:01:19.524166+0000","last_became_peered":"2024-04-25T00:01:19.524166+0000","last_unstale":"2024-04-25T00:01:20.511885+0000","last_undegraded":"2024-04-25T00:01:20.511885+0000","last_fullsized":"2024-04-25T00:01:20.511885+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:50.351599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"26'1","reported_seq":13,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T00:01:19.542913+0000","last_change":"2024-04-25T00:01:18.497096+0000","last_active":"2024-04-25T00:01:19.542913+0000","last_peered":"2024-04-25T00:01:19.542913+0000","last_clean":"2024-04-25T00:01:19.542913+0000","last_became_active":"2024-04-25T00:01:18.496289+0000","last_became_peered":"2024-04-25T00:01:18.496289+0000","last_unstale":"2024-04-25T00:01:19.542913+0000","last_undegraded":"2024-04-25T00:01:19.542913+0000","last_fullsized":"2024-04-25T00:01:19.542913+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:59:40.316831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T00:01:19.494675+0000","last_change":"2024-04-25T00:01:05.370043+0000","last_active":"2024-04-25T00:01:19.494675+0000","last_peered":"2024-04-25T00:01:19.494675+0000","last_clean":"2024-04-25T00:01:19.494675+0000","last_became_active":"2024-04-25T00:01:05.368080+0000","last_became_peered":"2024-04-25T00:01:05.368080+0000","last_unstale":"2024-04-25T00:01:19.494675+0000","last_undegraded":"2024-04-25T00:01:19.494675+0000","last_fullsized":"2024-04-25T00:01:19.494675+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:58:07.610923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.580256+0000","last_change":"2024-04-25T00:01:04.364523+0000","last_active":"2024-04-25T00:01:20.580256+0000","last_peered":"2024-04-25T00:01:20.580256+0000","last_clean":"2024-04-25T00:01:20.580256+0000","last_became_active":"2024-04-25T00:01:04.364050+0000","last_became_peered":"2024-04-25T00:01:04.364050+0000","last_unstale":"2024-04-25T00:01:20.580256+0000","last_undegraded":"2024-04-25T00:01:20.580256+0000","last_fullsized":"2024-04-25T00:01:20.580256+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:49.479804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.579360+0000","last_change":"2024-04-25T00:01:19.491523+0000","last_active":"2024-04-25T00:01:20.579360+0000","last_peered":"2024-04-25T00:01:20.579360+0000","last_clean":"2024-04-25T00:01:20.579360+0000","last_became_active":"2024-04-25T00:01:19.490755+0000","last_became_peered":"2024-04-25T00:01:19.490755+0000","last_unstale":"2024-04-25T00:01:20.579360+0000","last_undegraded":"2024-04-25T00:01:20.579360+0000","last_fullsized":"2024-04-25T00:01:20.579360+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:47:19.526845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"26'4","reported_seq":18,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.584296+0000","last_change":"2024-04-25T00:01:18.472794+0000","last_active":"2024-04-25T00:01:20.584296+0000","last_peered":"2024-04-25T00:01:20.584296+0000","last_clean":"2024-04-25T00:01:20.584296+0000","last_became_active":"2024-04-25T00:01:18.472534+0000","last_became_peered":"2024-04-25T00:01:18.472534+0000","last_unstale":"2024-04-25T00:01:20.584296+0000","last_undegraded":"2024-04-25T00:01:20.584296+0000","last_fullsized":"2024-04-25T00:01:20.584296+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:55:45.843808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.518835+0000","last_change":"2024-04-25T00:01:05.378867+0000","last_active":"2024-04-25T00:01:20.518835+0000","last_peered":"2024-04-25T00:01:20.518835+0000","last_clean":"2024-04-25T00:01:20.518835+0000","last_became_active":"2024-04-25T00:01:05.378056+0000","last_became_peered":"2024-04-25T00:01:05.378056+0000","last_unstale":"2024-04-25T00:01:20.518835+0000","last_undegraded":"2024-04-25T00:01:20.518835+0000","last_fullsized":"2024-04-25T00:01:20.518835+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:36:10.377757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.577806+0000","last_change":"2024-04-25T00:01:04.370896+0000","last_active":"2024-04-25T00:01:20.577806+0000","last_peered":"2024-04-25T00:01:20.577806+0000","last_clean":"2024-04-25T00:01:20.577806+0000","last_became_active":"2024-04-25T00:01:04.370273+0000","last_became_peered":"2024-04-25T00:01:04.370273+0000","last_unstale":"2024-04-25T00:01:20.577806+0000","last_undegraded":"2024-04-25T00:01:20.577806+0000","last_fullsized":"2024-04-25T00:01:20.577806+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:49:22.673421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T00:01:19.541208+0000","last_change":"2024-04-25T00:01:19.521143+0000","last_active":"2024-04-25T00:01:19.541208+0000","last_peered":"2024-04-25T00:01:19.541208+0000","last_clean":"2024-04-25T00:01:19.541208+0000","last_became_active":"2024-04-25T00:01:19.514664+0000","last_became_peered":"2024-04-25T00:01:19.514664+0000","last_unstale":"2024-04-25T00:01:19.541208+0000","last_undegraded":"2024-04-25T00:01:19.541208+0000","last_fullsized":"2024-04-25T00:01:19.541208+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:43:14.020286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"22'1","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.580606+0000","last_change":"2024-04-25T00:01:04.360254+0000","last_active":"2024-04-25T00:01:20.580606+0000","last_peered":"2024-04-25T00:01:20.580606+0000","last_clean":"2024-04-25T00:01:20.580606+0000","last_became_active":"2024-04-25T00:01:04.359586+0000","last_became_peered":"2024-04-25T00:01:04.359586+0000","last_unstale":"2024-04-25T00:01:20.580606+0000","last_undegraded":"2024-04-25T00:01:20.580606+0000","last_fullsized":"2024-04-25T00:01:20.580606+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:47.192930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T00:01:19.500989+0000","last_change":"2024-04-25T00:01:05.370723+0000","last_active":"2024-04-25T00:01:19.500989+0000","last_peered":"2024-04-25T00:01:19.500989+0000","last_clean":"2024-04-25T00:01:19.500989+0000","last_became_active":"2024-04-25T00:01:05.368970+0000","last_became_peered":"2024-04-25T00:01:05.368970+0000","last_unstale":"2024-04-25T00:01:19.500989+0000","last_undegraded":"2024-04-25T00:01:19.500989+0000","last_fullsized":"2024-04-25T00:01:19.500989+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:06:53.743074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"26'2","reported_seq":14,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T00:01:19.546428+0000","last_change":"2024-04-25T00:01:18.498570+0000","last_active":"2024-04-25T00:01:19.546428+0000","last_peered":"2024-04-25T00:01:19.546428+0000","last_clean":"2024-04-25T00:01:19.546428+0000","last_became_active":"2024-04-25T00:01:18.497745+0000","last_became_peered":"2024-04-25T00:01:18.497745+0000","last_unstale":"2024-04-25T00:01:19.546428+0000","last_undegraded":"2024-04-25T00:01:19.546428+0000","last_fullsized":"2024-04-25T00:01:19.546428+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:26:42.047149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511888+0000","last_change":"2024-04-25T00:01:19.514919+0000","last_active":"2024-04-25T00:01:20.511888+0000","last_peered":"2024-04-25T00:01:20.511888+0000","last_clean":"2024-04-25T00:01:20.511888+0000","last_became_active":"2024-04-25T00:01:19.514280+0000","last_became_peered":"2024-04-25T00:01:19.514280+0000","last_unstale":"2024-04-25T00:01:20.511888+0000","last_undegraded":"2024-04-25T00:01:20.511888+0000","last_fullsized":"2024-04-25T00:01:20.511888+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:37.975265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"22'1","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.519944+0000","last_change":"2024-04-25T00:01:04.365607+0000","last_active":"2024-04-25T00:01:20.519944+0000","last_peered":"2024-04-25T00:01:20.519944+0000","last_clean":"2024-04-25T00:01:20.519944+0000","last_became_active":"2024-04-25T00:01:04.364788+0000","last_became_peered":"2024-04-25T00:01:04.364788+0000","last_unstale":"2024-04-25T00:01:20.519944+0000","last_undegraded":"2024-04-25T00:01:20.519944+0000","last_fullsized":"2024-04-25T00:01:20.519944+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.579248+0000","last_change":"2024-04-25T00:01:05.375142+0000","last_active":"2024-04-25T00:01:20.579248+0000","last_peered":"2024-04-25T00:01:20.579248+0000","last_clean":"2024-04-25T00:01:20.579248+0000","last_became_active":"2024-04-25T00:01:05.373857+0000","last_became_peered":"2024-04-25T00:01:05.373857+0000","last_unstale":"2024-04-25T00:01:20.579248+0000","last_undegraded":"2024-04-25T00:01:20.579248+0000","last_fullsized":"2024-04-25T00:01:20.579248+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:15.879673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"26'2","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.513534+0000","last_change":"2024-04-25T00:01:18.502786+0000","last_active":"2024-04-25T00:01:20.513534+0000","last_peered":"2024-04-25T00:01:20.513534+0000","last_clean":"2024-04-25T00:01:20.513534+0000","last_became_active":"2024-04-25T00:01:18.501785+0000","last_became_peered":"2024-04-25T00:01:18.501785+0000","last_unstale":"2024-04-25T00:01:20.513534+0000","last_undegraded":"2024-04-25T00:01:20.513534+0000","last_fullsized":"2024-04-25T00:01:20.513534+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:09:31.441044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":9,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T00:01:19.537533+0000","last_change":"2024-04-25T00:01:19.511239+0000","last_active":"2024-04-25T00:01:19.537533+0000","last_peered":"2024-04-25T00:01:19.537533+0000","last_clean":"2024-04-25T00:01:19.537533+0000","last_became_active":"2024-04-25T00:01:19.510155+0000","last_became_peered":"2024-04-25T00:01:19.510155+0000","last_unstale":"2024-04-25T00:01:19.537533+0000","last_undegraded":"2024-04-25T00:01:19.537533+0000","last_fullsized":"2024-04-25T00:01:19.537533+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:33:56.057791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511273+0000","last_change":"2024-04-25T00:01:05.378517+0000","last_active":"2024-04-25T00:01:20.511273+0000","last_peered":"2024-04-25T00:01:20.511273+0000","last_clean":"2024-04-25T00:01:20.511273+0000","last_became_active":"2024-04-25T00:01:05.377742+0000","last_became_peered":"2024-04-25T00:01:05.377742+0000","last_unstale":"2024-04-25T00:01:20.511273+0000","last_undegraded":"2024-04-25T00:01:20.511273+0000","last_fullsized":"2024-04-25T00:01:20.511273+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:06.450369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":9,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T00:01:19.537322+0000","last_change":"2024-04-25T00:01:19.508999+0000","last_active":"2024-04-25T00:01:19.537322+0000","last_peered":"2024-04-25T00:01:19.537322+0000","last_clean":"2024-04-25T00:01:19.537322+0000","last_became_active":"2024-04-25T00:01:19.507960+0000","last_became_peered":"2024-04-25T00:01:19.507960+0000","last_unstale":"2024-04-25T00:01:19.537322+0000","last_undegraded":"2024-04-25T00:01:19.537322+0000","last_fullsized":"2024-04-25T00:01:19.537322+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:48:00.647773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.510941+0000","last_change":"2024-04-25T00:01:05.372927+0000","last_active":"2024-04-25T00:01:20.510941+0000","last_peered":"2024-04-25T00:01:20.510941+0000","last_clean":"2024-04-25T00:01:20.510941+0000","last_became_active":"2024-04-25T00:01:05.372050+0000","last_became_peered":"2024-04-25T00:01:05.372050+0000","last_unstale":"2024-04-25T00:01:20.510941+0000","last_undegraded":"2024-04-25T00:01:20.510941+0000","last_fullsized":"2024-04-25T00:01:20.510941+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:48.202251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511854+0000","last_change":"2024-04-25T00:01:19.531072+0000","last_active":"2024-04-25T00:01:20.511854+0000","last_peered":"2024-04-25T00:01:20.511854+0000","last_clean":"2024-04-25T00:01:20.511854+0000","last_became_active":"2024-04-25T00:01:19.530836+0000","last_became_peered":"2024-04-25T00:01:19.530836+0000","last_unstale":"2024-04-25T00:01:20.511854+0000","last_undegraded":"2024-04-25T00:01:20.511854+0000","last_fullsized":"2024-04-25T00:01:20.511854+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:08.508741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.580209+0000","last_change":"2024-04-25T00:01:05.362960+0000","last_active":"2024-04-25T00:01:20.580209+0000","last_peered":"2024-04-25T00:01:20.580209+0000","last_clean":"2024-04-25T00:01:20.580209+0000","last_became_active":"2024-04-25T00:01:05.361067+0000","last_became_peered":"2024-04-25T00:01:05.361067+0000","last_unstale":"2024-04-25T00:01:20.580209+0000","last_undegraded":"2024-04-25T00:01:20.580209+0000","last_fullsized":"2024-04-25T00:01:20.580209+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:24.017263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.520235+0000","last_change":"2024-04-25T00:01:19.506029+0000","last_active":"2024-04-25T00:01:20.520235+0000","last_peered":"2024-04-25T00:01:20.520235+0000","last_clean":"2024-04-25T00:01:20.520235+0000","last_became_active":"2024-04-25T00:01:19.505184+0000","last_became_peered":"2024-04-25T00:01:19.505184+0000","last_unstale":"2024-04-25T00:01:20.520235+0000","last_undegraded":"2024-04-25T00:01:20.520235+0000","last_fullsized":"2024-04-25T00:01:20.520235+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:08.508741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.519736+0000","last_change":"2024-04-25T00:01:05.358690+0000","last_active":"2024-04-25T00:01:20.519736+0000","last_peered":"2024-04-25T00:01:20.519736+0000","last_clean":"2024-04-25T00:01:20.519736+0000","last_became_active":"2024-04-25T00:01:05.357340+0000","last_became_peered":"2024-04-25T00:01:05.357340+0000","last_unstale":"2024-04-25T00:01:20.519736+0000","last_undegraded":"2024-04-25T00:01:20.519736+0000","last_fullsized":"2024-04-25T00:01:20.519736+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:36:55.436661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.577993+0000","last_change":"2024-04-25T00:01:19.508691+0000","last_active":"2024-04-25T00:01:20.577993+0000","last_peered":"2024-04-25T00:01:20.577993+0000","last_clean":"2024-04-25T00:01:20.577993+0000","last_became_active":"2024-04-25T00:01:19.508123+0000","last_became_peered":"2024-04-25T00:01:19.508123+0000","last_unstale":"2024-04-25T00:01:20.577993+0000","last_undegraded":"2024-04-25T00:01:20.577993+0000","last_fullsized":"2024-04-25T00:01:20.577993+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:47:19.526845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578984+0000","last_change":"2024-04-25T00:01:05.387057+0000","last_active":"2024-04-25T00:01:20.578984+0000","last_peered":"2024-04-25T00:01:20.578984+0000","last_clean":"2024-04-25T00:01:20.578984+0000","last_became_active":"2024-04-25T00:01:05.386724+0000","last_became_peered":"2024-04-25T00:01:05.386724+0000","last_unstale":"2024-04-25T00:01:20.578984+0000","last_undegraded":"2024-04-25T00:01:20.578984+0000","last_fullsized":"2024-04-25T00:01:20.578984+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:55:09.073286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.580476+0000","last_change":"2024-04-25T00:01:19.522760+0000","last_active":"2024-04-25T00:01:20.580476+0000","last_peered":"2024-04-25T00:01:20.580476+0000","last_clean":"2024-04-25T00:01:20.580476+0000","last_became_active":"2024-04-25T00:01:19.522380+0000","last_became_peered":"2024-04-25T00:01:19.522380+0000","last_unstale":"2024-04-25T00:01:20.580476+0000","last_undegraded":"2024-04-25T00:01:20.580476+0000","last_fullsized":"2024-04-25T00:01:20.580476+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:55:46.853565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578828+0000","last_change":"2024-04-25T00:01:05.367296+0000","last_active":"2024-04-25T00:01:20.578828+0000","last_peered":"2024-04-25T00:01:20.578828+0000","last_clean":"2024-04-25T00:01:20.578828+0000","last_became_active":"2024-04-25T00:01:05.363838+0000","last_became_peered":"2024-04-25T00:01:05.363838+0000","last_unstale":"2024-04-25T00:01:20.578828+0000","last_undegraded":"2024-04-25T00:01:20.578828+0000","last_fullsized":"2024-04-25T00:01:20.578828+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:43:07.053074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.579599+0000","last_change":"2024-04-25T00:01:19.510902+0000","last_active":"2024-04-25T00:01:20.579599+0000","last_peered":"2024-04-25T00:01:20.579599+0000","last_clean":"2024-04-25T00:01:20.579599+0000","last_became_active":"2024-04-25T00:01:19.510008+0000","last_became_peered":"2024-04-25T00:01:19.510008+0000","last_unstale":"2024-04-25T00:01:20.579599+0000","last_undegraded":"2024-04-25T00:01:20.579599+0000","last_fullsized":"2024-04-25T00:01:20.579599+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:40:32.292140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511704+0000","last_change":"2024-04-25T00:01:05.365761+0000","last_active":"2024-04-25T00:01:20.511704+0000","last_peered":"2024-04-25T00:01:20.511704+0000","last_clean":"2024-04-25T00:01:20.511704+0000","last_became_active":"2024-04-25T00:01:05.361119+0000","last_became_peered":"2024-04-25T00:01:05.361119+0000","last_unstale":"2024-04-25T00:01:20.511704+0000","last_undegraded":"2024-04-25T00:01:20.511704+0000","last_fullsized":"2024-04-25T00:01:20.511704+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.510983+0000","last_change":"2024-04-25T00:01:19.518534+0000","last_active":"2024-04-25T00:01:20.510983+0000","last_peered":"2024-04-25T00:01:20.510983+0000","last_clean":"2024-04-25T00:01:20.510983+0000","last_became_active":"2024-04-25T00:01:19.518110+0000","last_became_peered":"2024-04-25T00:01:19.518110+0000","last_unstale":"2024-04-25T00:01:20.510983+0000","last_undegraded":"2024-04-25T00:01:20.510983+0000","last_fullsized":"2024-04-25T00:01:20.510983+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:05:09.219823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.520770+0000","last_change":"2024-04-25T00:01:05.373464+0000","last_active":"2024-04-25T00:01:20.520770+0000","last_peered":"2024-04-25T00:01:20.520770+0000","last_clean":"2024-04-25T00:01:20.520770+0000","last_became_active":"2024-04-25T00:01:05.372611+0000","last_became_peered":"2024-04-25T00:01:05.372611+0000","last_unstale":"2024-04-25T00:01:20.520770+0000","last_undegraded":"2024-04-25T00:01:20.520770+0000","last_fullsized":"2024-04-25T00:01:20.520770+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:31.456792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.512974+0000","last_change":"2024-04-25T00:01:19.542475+0000","last_active":"2024-04-25T00:01:20.512974+0000","last_peered":"2024-04-25T00:01:20.512974+0000","last_clean":"2024-04-25T00:01:20.512974+0000","last_became_active":"2024-04-25T00:01:19.541877+0000","last_became_peered":"2024-04-25T00:01:19.541877+0000","last_unstale":"2024-04-25T00:01:20.512974+0000","last_undegraded":"2024-04-25T00:01:20.512974+0000","last_fullsized":"2024-04-25T00:01:20.512974+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:02:57.373014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.580023+0000","last_change":"2024-04-25T00:01:05.372906+0000","last_active":"2024-04-25T00:01:20.580023+0000","last_peered":"2024-04-25T00:01:20.580023+0000","last_clean":"2024-04-25T00:01:20.580023+0000","last_became_active":"2024-04-25T00:01:05.372148+0000","last_became_peered":"2024-04-25T00:01:05.372148+0000","last_unstale":"2024-04-25T00:01:20.580023+0000","last_undegraded":"2024-04-25T00:01:20.580023+0000","last_fullsized":"2024-04-25T00:01:20.580023+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:43.615895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.584868+0000","last_change":"2024-04-25T00:01:19.498515+0000","last_active":"2024-04-25T00:01:20.584868+0000","last_peered":"2024-04-25T00:01:20.584868+0000","last_clean":"2024-04-25T00:01:20.584868+0000","last_became_active":"2024-04-25T00:01:19.496310+0000","last_became_peered":"2024-04-25T00:01:19.496310+0000","last_unstale":"2024-04-25T00:01:20.584868+0000","last_undegraded":"2024-04-25T00:01:20.584868+0000","last_fullsized":"2024-04-25T00:01:20.584868+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:20:06.661558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.519325+0000","last_change":"2024-04-25T00:01:05.366340+0000","last_active":"2024-04-25T00:01:20.519325+0000","last_peered":"2024-04-25T00:01:20.519325+0000","last_clean":"2024-04-25T00:01:20.519325+0000","last_became_active":"2024-04-25T00:01:05.361952+0000","last_became_peered":"2024-04-25T00:01:05.361952+0000","last_unstale":"2024-04-25T00:01:20.519325+0000","last_undegraded":"2024-04-25T00:01:20.519325+0000","last_fullsized":"2024-04-25T00:01:20.519325+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:41.948662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578558+0000","last_change":"2024-04-25T00:01:19.508257+0000","last_active":"2024-04-25T00:01:20.578558+0000","last_peered":"2024-04-25T00:01:20.578558+0000","last_clean":"2024-04-25T00:01:20.578558+0000","last_became_active":"2024-04-25T00:01:19.507516+0000","last_became_peered":"2024-04-25T00:01:19.507516+0000","last_unstale":"2024-04-25T00:01:20.578558+0000","last_undegraded":"2024-04-25T00:01:20.578558+0000","last_fullsized":"2024-04-25T00:01:20.578558+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:45.121513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.579066+0000","last_change":"2024-04-25T00:01:05.387457+0000","last_active":"2024-04-25T00:01:20.579066+0000","last_peered":"2024-04-25T00:01:20.579066+0000","last_clean":"2024-04-25T00:01:20.579066+0000","last_became_active":"2024-04-25T00:01:05.386919+0000","last_became_peered":"2024-04-25T00:01:05.386919+0000","last_unstale":"2024-04-25T00:01:20.579066+0000","last_undegraded":"2024-04-25T00:01:20.579066+0000","last_fullsized":"2024-04-25T00:01:20.579066+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.512398+0000","last_change":"2024-04-25T00:01:19.520648+0000","last_active":"2024-04-25T00:01:20.512398+0000","last_peered":"2024-04-25T00:01:20.512398+0000","last_clean":"2024-04-25T00:01:20.512398+0000","last_became_active":"2024-04-25T00:01:19.519303+0000","last_became_peered":"2024-04-25T00:01:19.519303+0000","last_unstale":"2024-04-25T00:01:20.512398+0000","last_undegraded":"2024-04-25T00:01:20.512398+0000","last_fullsized":"2024-04-25T00:01:20.512398+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:23.993729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T00:01:19.500747+0000","last_change":"2024-04-25T00:01:05.366249+0000","last_active":"2024-04-25T00:01:19.500747+0000","last_peered":"2024-04-25T00:01:19.500747+0000","last_clean":"2024-04-25T00:01:19.500747+0000","last_became_active":"2024-04-25T00:01:05.365548+0000","last_became_peered":"2024-04-25T00:01:05.365548+0000","last_unstale":"2024-04-25T00:01:19.500747+0000","last_undegraded":"2024-04-25T00:01:19.500747+0000","last_fullsized":"2024-04-25T00:01:19.500747+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:22:21.143639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.510804+0000","last_change":"2024-04-25T00:01:19.514659+0000","last_active":"2024-04-25T00:01:20.510804+0000","last_peered":"2024-04-25T00:01:20.510804+0000","last_clean":"2024-04-25T00:01:20.510804+0000","last_became_active":"2024-04-25T00:01:19.514232+0000","last_became_peered":"2024-04-25T00:01:19.514232+0000","last_unstale":"2024-04-25T00:01:20.510804+0000","last_undegraded":"2024-04-25T00:01:20.510804+0000","last_fullsized":"2024-04-25T00:01:20.510804+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:45.121513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582289+0000","last_change":"2024-04-25T00:01:05.385253+0000","last_active":"2024-04-25T00:01:20.582289+0000","last_peered":"2024-04-25T00:01:20.582289+0000","last_clean":"2024-04-25T00:01:20.582289+0000","last_became_active":"2024-04-25T00:01:05.379429+0000","last_became_peered":"2024-04-25T00:01:05.379429+0000","last_unstale":"2024-04-25T00:01:20.582289+0000","last_undegraded":"2024-04-25T00:01:20.582289+0000","last_fullsized":"2024-04-25T00:01:20.582289+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:36:10.377757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.581087+0000","last_change":"2024-04-25T00:01:19.503419+0000","last_active":"2024-04-25T00:01:20.581087+0000","last_peered":"2024-04-25T00:01:20.581087+0000","last_clean":"2024-04-25T00:01:20.581087+0000","last_became_active":"2024-04-25T00:01:19.500568+0000","last_became_peered":"2024-04-25T00:01:19.500568+0000","last_unstale":"2024-04-25T00:01:20.581087+0000","last_undegraded":"2024-04-25T00:01:20.581087+0000","last_fullsized":"2024-04-25T00:01:20.581087+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:08.508741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.583587+0000","last_change":"2024-04-25T00:01:05.370653+0000","last_active":"2024-04-25T00:01:20.583587+0000","last_peered":"2024-04-25T00:01:20.583587+0000","last_clean":"2024-04-25T00:01:20.583587+0000","last_became_active":"2024-04-25T00:01:05.369919+0000","last_became_peered":"2024-04-25T00:01:05.369919+0000","last_unstale":"2024-04-25T00:01:20.583587+0000","last_undegraded":"2024-04-25T00:01:20.583587+0000","last_fullsized":"2024-04-25T00:01:20.583587+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:30:08.311326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.510560+0000","last_change":"2024-04-25T00:01:19.540104+0000","last_active":"2024-04-25T00:01:20.510560+0000","last_peered":"2024-04-25T00:01:20.510560+0000","last_clean":"2024-04-25T00:01:20.510560+0000","last_became_active":"2024-04-25T00:01:19.539775+0000","last_became_peered":"2024-04-25T00:01:19.539775+0000","last_unstale":"2024-04-25T00:01:20.510560+0000","last_undegraded":"2024-04-25T00:01:20.510560+0000","last_fullsized":"2024-04-25T00:01:20.510560+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:53:14.768520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.514367+0000","last_change":"2024-04-25T00:01:05.367001+0000","last_active":"2024-04-25T00:01:20.514367+0000","last_peered":"2024-04-25T00:01:20.514367+0000","last_clean":"2024-04-25T00:01:20.514367+0000","last_became_active":"2024-04-25T00:01:05.366264+0000","last_became_peered":"2024-04-25T00:01:05.366264+0000","last_unstale":"2024-04-25T00:01:20.514367+0000","last_undegraded":"2024-04-25T00:01:20.514367+0000","last_fullsized":"2024-04-25T00:01:20.514367+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578038+0000","last_change":"2024-04-25T00:01:19.497247+0000","last_active":"2024-04-25T00:01:20.578038+0000","last_peered":"2024-04-25T00:01:20.578038+0000","last_clean":"2024-04-25T00:01:20.578038+0000","last_became_active":"2024-04-25T00:01:19.496517+0000","last_became_peered":"2024-04-25T00:01:19.496517+0000","last_unstale":"2024-04-25T00:01:20.578038+0000","last_undegraded":"2024-04-25T00:01:20.578038+0000","last_fullsized":"2024-04-25T00:01:20.578038+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:30.129056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.584723+0000","last_change":"2024-04-25T00:01:05.379958+0000","last_active":"2024-04-25T00:01:20.584723+0000","last_peered":"2024-04-25T00:01:20.584723+0000","last_clean":"2024-04-25T00:01:20.584723+0000","last_became_active":"2024-04-25T00:01:05.379090+0000","last_became_peered":"2024-04-25T00:01:05.379090+0000","last_unstale":"2024-04-25T00:01:20.584723+0000","last_undegraded":"2024-04-25T00:01:20.584723+0000","last_fullsized":"2024-04-25T00:01:20.584723+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:59:50.832958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":9,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T00:01:19.543038+0000","last_change":"2024-04-25T00:01:19.530176+0000","last_active":"2024-04-25T00:01:19.543038+0000","last_peered":"2024-04-25T00:01:19.543038+0000","last_clean":"2024-04-25T00:01:19.543038+0000","last_became_active":"2024-04-25T00:01:19.529410+0000","last_became_peered":"2024-04-25T00:01:19.529410+0000","last_unstale":"2024-04-25T00:01:19.543038+0000","last_undegraded":"2024-04-25T00:01:19.543038+0000","last_fullsized":"2024-04-25T00:01:19.543038+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:24:44.480639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.579746+0000","last_change":"2024-04-25T00:01:05.370826+0000","last_active":"2024-04-25T00:01:20.579746+0000","last_peered":"2024-04-25T00:01:20.579746+0000","last_clean":"2024-04-25T00:01:20.579746+0000","last_became_active":"2024-04-25T00:01:05.369848+0000","last_became_peered":"2024-04-25T00:01:05.369848+0000","last_unstale":"2024-04-25T00:01:20.579746+0000","last_undegraded":"2024-04-25T00:01:20.579746+0000","last_fullsized":"2024-04-25T00:01:20.579746+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:23.870121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.581888+0000","last_change":"2024-04-25T00:01:19.511625+0000","last_active":"2024-04-25T00:01:20.581888+0000","last_peered":"2024-04-25T00:01:20.581888+0000","last_clean":"2024-04-25T00:01:20.581888+0000","last_became_active":"2024-04-25T00:01:19.511349+0000","last_became_peered":"2024-04-25T00:01:19.511349+0000","last_unstale":"2024-04-25T00:01:20.581888+0000","last_undegraded":"2024-04-25T00:01:20.581888+0000","last_fullsized":"2024-04-25T00:01:20.581888+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:02:57.373014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.518885+0000","last_change":"2024-04-25T00:01:05.390341+0000","last_active":"2024-04-25T00:01:20.518885+0000","last_peered":"2024-04-25T00:01:20.518885+0000","last_clean":"2024-04-25T00:01:20.518885+0000","last_became_active":"2024-04-25T00:01:05.389707+0000","last_became_peered":"2024-04-25T00:01:05.389707+0000","last_unstale":"2024-04-25T00:01:20.518885+0000","last_undegraded":"2024-04-25T00:01:20.518885+0000","last_fullsized":"2024-04-25T00:01:20.518885+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:54:21.138947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578289+0000","last_change":"2024-04-25T00:01:19.513630+0000","last_active":"2024-04-25T00:01:20.578289+0000","last_peered":"2024-04-25T00:01:20.578289+0000","last_clean":"2024-04-25T00:01:20.578289+0000","last_became_active":"2024-04-25T00:01:19.512978+0000","last_became_peered":"2024-04-25T00:01:19.512978+0000","last_unstale":"2024-04-25T00:01:20.578289+0000","last_undegraded":"2024-04-25T00:01:20.578289+0000","last_fullsized":"2024-04-25T00:01:20.578289+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:57.751007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578080+0000","last_change":"2024-04-25T00:01:05.378373+0000","last_active":"2024-04-25T00:01:20.578080+0000","last_peered":"2024-04-25T00:01:20.578080+0000","last_clean":"2024-04-25T00:01:20.578080+0000","last_became_active":"2024-04-25T00:01:05.376570+0000","last_became_peered":"2024-04-25T00:01:05.376570+0000","last_unstale":"2024-04-25T00:01:20.578080+0000","last_undegraded":"2024-04-25T00:01:20.578080+0000","last_fullsized":"2024-04-25T00:01:20.578080+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:30.228060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582484+0000","last_change":"2024-04-25T00:01:19.516716+0000","last_active":"2024-04-25T00:01:20.582484+0000","last_peered":"2024-04-25T00:01:20.582484+0000","last_clean":"2024-04-25T00:01:20.582484+0000","last_became_active":"2024-04-25T00:01:19.516142+0000","last_became_peered":"2024-04-25T00:01:19.516142+0000","last_unstale":"2024-04-25T00:01:20.582484+0000","last_undegraded":"2024-04-25T00:01:20.582484+0000","last_fullsized":"2024-04-25T00:01:20.582484+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:28:18.234546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578828+0000","last_change":"2024-04-25T00:01:05.381011+0000","last_active":"2024-04-25T00:01:20.578828+0000","last_peered":"2024-04-25T00:01:20.578828+0000","last_clean":"2024-04-25T00:01:20.578828+0000","last_became_active":"2024-04-25T00:01:05.380236+0000","last_became_peered":"2024-04-25T00:01:05.380236+0000","last_unstale":"2024-04-25T00:01:20.578828+0000","last_undegraded":"2024-04-25T00:01:20.578828+0000","last_fullsized":"2024-04-25T00:01:20.578828+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:06:19.307378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":9,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T00:01:19.542220+0000","last_change":"2024-04-25T00:01:19.529023+0000","last_active":"2024-04-25T00:01:19.542220+0000","last_peered":"2024-04-25T00:01:19.542220+0000","last_clean":"2024-04-25T00:01:19.542220+0000","last_became_active":"2024-04-25T00:01:19.524609+0000","last_became_peered":"2024-04-25T00:01:19.524609+0000","last_unstale":"2024-04-25T00:01:19.542220+0000","last_undegraded":"2024-04-25T00:01:19.542220+0000","last_fullsized":"2024-04-25T00:01:19.542220+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:38:43.961691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582056+0000","last_change":"2024-04-25T00:01:05.385853+0000","last_active":"2024-04-25T00:01:20.582056+0000","last_peered":"2024-04-25T00:01:20.582056+0000","last_clean":"2024-04-25T00:01:20.582056+0000","last_became_active":"2024-04-25T00:01:05.384997+0000","last_became_peered":"2024-04-25T00:01:05.384997+0000","last_unstale":"2024-04-25T00:01:20.582056+0000","last_undegraded":"2024-04-25T00:01:20.582056+0000","last_fullsized":"2024-04-25T00:01:20.582056+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:36:55.436661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578299+0000","last_change":"2024-04-25T00:01:19.516867+0000","last_active":"2024-04-25T00:01:20.578299+0000","last_peered":"2024-04-25T00:01:20.578299+0000","last_clean":"2024-04-25T00:01:20.578299+0000","last_became_active":"2024-04-25T00:01:19.515764+0000","last_became_peered":"2024-04-25T00:01:19.515764+0000","last_unstale":"2024-04-25T00:01:20.578299+0000","last_undegraded":"2024-04-25T00:01:20.578299+0000","last_fullsized":"2024-04-25T00:01:20.578299+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:53:14.768520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582072+0000","last_change":"2024-04-25T00:01:05.389424+0000","last_active":"2024-04-25T00:01:20.582072+0000","last_peered":"2024-04-25T00:01:20.582072+0000","last_clean":"2024-04-25T00:01:20.582072+0000","last_became_active":"2024-04-25T00:01:05.388999+0000","last_became_peered":"2024-04-25T00:01:05.388999+0000","last_unstale":"2024-04-25T00:01:20.582072+0000","last_undegraded":"2024-04-25T00:01:20.582072+0000","last_fullsized":"2024-04-25T00:01:20.582072+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:31.456792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578192+0000","last_change":"2024-04-25T00:01:19.519365+0000","last_active":"2024-04-25T00:01:20.578192+0000","last_peered":"2024-04-25T00:01:20.578192+0000","last_clean":"2024-04-25T00:01:20.578192+0000","last_became_active":"2024-04-25T00:01:19.519085+0000","last_became_peered":"2024-04-25T00:01:19.519085+0000","last_unstale":"2024-04-25T00:01:20.578192+0000","last_undegraded":"2024-04-25T00:01:20.578192+0000","last_fullsized":"2024-04-25T00:01:20.578192+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:05:09.219823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.580630+0000","last_change":"2024-04-25T00:01:05.368974+0000","last_active":"2024-04-25T00:01:20.580630+0000","last_peered":"2024-04-25T00:01:20.580630+0000","last_clean":"2024-04-25T00:01:20.580630+0000","last_became_active":"2024-04-25T00:01:05.367893+0000","last_became_peered":"2024-04-25T00:01:05.367893+0000","last_unstale":"2024-04-25T00:01:20.580630+0000","last_undegraded":"2024-04-25T00:01:20.580630+0000","last_fullsized":"2024-04-25T00:01:20.580630+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:30:23.557381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578274+0000","last_change":"2024-04-25T00:01:19.513514+0000","last_active":"2024-04-25T00:01:20.578274+0000","last_peered":"2024-04-25T00:01:20.578274+0000","last_clean":"2024-04-25T00:01:20.578274+0000","last_became_active":"2024-04-25T00:01:19.513184+0000","last_became_peered":"2024-04-25T00:01:19.513184+0000","last_unstale":"2024-04-25T00:01:20.578274+0000","last_undegraded":"2024-04-25T00:01:20.578274+0000","last_fullsized":"2024-04-25T00:01:20.578274+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:47.859626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582384+0000","last_change":"2024-04-25T00:01:05.362303+0000","last_active":"2024-04-25T00:01:20.582384+0000","last_peered":"2024-04-25T00:01:20.582384+0000","last_clean":"2024-04-25T00:01:20.582384+0000","last_became_active":"2024-04-25T00:01:05.361586+0000","last_became_peered":"2024-04-25T00:01:05.361586+0000","last_unstale":"2024-04-25T00:01:20.582384+0000","last_undegraded":"2024-04-25T00:01:20.582384+0000","last_fullsized":"2024-04-25T00:01:20.582384+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:30.228060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.520395+0000","last_change":"2024-04-25T00:01:19.515101+0000","last_active":"2024-04-25T00:01:20.520395+0000","last_peered":"2024-04-25T00:01:20.520395+0000","last_clean":"2024-04-25T00:01:20.520395+0000","last_became_active":"2024-04-25T00:01:19.513314+0000","last_became_peered":"2024-04-25T00:01:19.513314+0000","last_unstale":"2024-04-25T00:01:20.520395+0000","last_undegraded":"2024-04-25T00:01:20.520395+0000","last_fullsized":"2024-04-25T00:01:20.520395+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:50.351599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.518642+0000","last_change":"2024-04-25T00:01:05.390256+0000","last_active":"2024-04-25T00:01:20.518642+0000","last_peered":"2024-04-25T00:01:20.518642+0000","last_clean":"2024-04-25T00:01:20.518642+0000","last_became_active":"2024-04-25T00:01:05.389724+0000","last_became_peered":"2024-04-25T00:01:05.389724+0000","last_unstale":"2024-04-25T00:01:20.518642+0000","last_undegraded":"2024-04-25T00:01:20.518642+0000","last_fullsized":"2024-04-25T00:01:20.518642+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:09:18.315688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3f","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511570+0000","last_change":"2024-04-25T00:01:19.511493+0000","last_active":"2024-04-25T00:01:20.511570+0000","last_peered":"2024-04-25T00:01:20.511570+0000","last_clean":"2024-04-25T00:01:20.511570+0000","last_became_active":"2024-04-25T00:01:19.508049+0000","last_became_peered":"2024-04-25T00:01:19.508049+0000","last_unstale":"2024-04-25T00:01:20.511570+0000","last_undegraded":"2024-04-25T00:01:20.511570+0000","last_fullsized":"2024-04-25T00:01:20.511570+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:55:46.853565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578830+0000","last_change":"2024-04-25T00:01:05.374766+0000","last_active":"2024-04-25T00:01:20.578830+0000","last_peered":"2024-04-25T00:01:20.578830+0000","last_clean":"2024-04-25T00:01:20.578830+0000","last_became_active":"2024-04-25T00:01:05.369739+0000","last_became_peered":"2024-04-25T00:01:05.369739+0000","last_unstale":"2024-04-25T00:01:20.578830+0000","last_undegraded":"2024-04-25T00:01:20.578830+0000","last_fullsized":"2024-04-25T00:01:20.578830+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:09:18.315688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":9,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-04-25T00:01:19.541277+0000","last_change":"2024-04-25T00:01:19.521456+0000","last_active":"2024-04-25T00:01:19.541277+0000","last_peered":"2024-04-25T00:01:19.541277+0000","last_clean":"2024-04-25T00:01:19.541277+0000","last_became_active":"2024-04-25T00:01:19.520787+0000","last_became_peered":"2024-04-25T00:01:19.520787+0000","last_unstale":"2024-04-25T00:01:19.541277+0000","last_undegraded":"2024-04-25T00:01:19.541277+0000","last_fullsized":"2024-04-25T00:01:19.541277+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:59:30.865281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511048+0000","last_change":"2024-04-25T00:01:05.359253+0000","last_active":"2024-04-25T00:01:20.511048+0000","last_peered":"2024-04-25T00:01:20.511048+0000","last_clean":"2024-04-25T00:01:20.511048+0000","last_became_active":"2024-04-25T00:01:05.358477+0000","last_became_peered":"2024-04-25T00:01:05.358477+0000","last_unstale":"2024-04-25T00:01:20.511048+0000","last_undegraded":"2024-04-25T00:01:20.511048+0000","last_fullsized":"2024-04-25T00:01:20.511048+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:22.739214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.579384+0000","last_change":"2024-04-25T00:01:19.509505+0000","last_active":"2024-04-25T00:01:20.579384+0000","last_peered":"2024-04-25T00:01:20.579384+0000","last_clean":"2024-04-25T00:01:20.579384+0000","last_became_active":"2024-04-25T00:01:19.507812+0000","last_became_peered":"2024-04-25T00:01:19.507812+0000","last_unstale":"2024-04-25T00:01:20.579384+0000","last_undegraded":"2024-04-25T00:01:20.579384+0000","last_fullsized":"2024-04-25T00:01:20.579384+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:23.993729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.581200+0000","last_change":"2024-04-25T00:01:05.386068+0000","last_active":"2024-04-25T00:01:20.581200+0000","last_peered":"2024-04-25T00:01:20.581200+0000","last_clean":"2024-04-25T00:01:20.581200+0000","last_became_active":"2024-04-25T00:01:05.378650+0000","last_became_peered":"2024-04-25T00:01:05.378650+0000","last_unstale":"2024-04-25T00:01:20.581200+0000","last_undegraded":"2024-04-25T00:01:20.581200+0000","last_fullsized":"2024-04-25T00:01:20.581200+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:12:52.846250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511288+0000","last_change":"2024-04-25T00:01:19.514843+0000","last_active":"2024-04-25T00:01:20.511288+0000","last_peered":"2024-04-25T00:01:20.511288+0000","last_clean":"2024-04-25T00:01:20.511288+0000","last_became_active":"2024-04-25T00:01:19.513956+0000","last_became_peered":"2024-04-25T00:01:19.513956+0000","last_unstale":"2024-04-25T00:01:20.511288+0000","last_undegraded":"2024-04-25T00:01:20.511288+0000","last_fullsized":"2024-04-25T00:01:20.511288+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:38.005233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511996+0000","last_change":"2024-04-25T00:01:05.359716+0000","last_active":"2024-04-25T00:01:20.511996+0000","last_peered":"2024-04-25T00:01:20.511996+0000","last_clean":"2024-04-25T00:01:20.511996+0000","last_became_active":"2024-04-25T00:01:05.358866+0000","last_became_peered":"2024-04-25T00:01:05.358866+0000","last_unstale":"2024-04-25T00:01:20.511996+0000","last_undegraded":"2024-04-25T00:01:20.511996+0000","last_fullsized":"2024-04-25T00:01:20.511996+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:50.489125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.519466+0000","last_change":"2024-04-25T00:01:19.499574+0000","last_active":"2024-04-25T00:01:20.519466+0000","last_peered":"2024-04-25T00:01:20.519466+0000","last_clean":"2024-04-25T00:01:20.519466+0000","last_became_active":"2024-04-25T00:01:19.498699+0000","last_became_peered":"2024-04-25T00:01:19.498699+0000","last_unstale":"2024-04-25T00:01:20.519466+0000","last_undegraded":"2024-04-25T00:01:20.519466+0000","last_fullsized":"2024-04-25T00:01:20.519466+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:47.859626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.520083+0000","last_change":"2024-04-25T00:01:05.371291+0000","last_active":"2024-04-25T00:01:20.520083+0000","last_peered":"2024-04-25T00:01:20.520083+0000","last_clean":"2024-04-25T00:01:20.520083+0000","last_became_active":"2024-04-25T00:01:05.370522+0000","last_became_peered":"2024-04-25T00:01:05.370522+0000","last_unstale":"2024-04-25T00:01:20.520083+0000","last_undegraded":"2024-04-25T00:01:20.520083+0000","last_fullsized":"2024-04-25T00:01:20.520083+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:11:27.044893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511353+0000","last_change":"2024-04-25T00:01:19.523798+0000","last_active":"2024-04-25T00:01:20.511353+0000","last_peered":"2024-04-25T00:01:20.511353+0000","last_clean":"2024-04-25T00:01:20.511353+0000","last_became_active":"2024-04-25T00:01:19.523450+0000","last_became_peered":"2024-04-25T00:01:19.523450+0000","last_unstale":"2024-04-25T00:01:20.511353+0000","last_undegraded":"2024-04-25T00:01:20.511353+0000","last_fullsized":"2024-04-25T00:01:20.511353+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:40:36.350832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.520565+0000","last_change":"2024-04-25T00:01:05.368190+0000","last_active":"2024-04-25T00:01:20.520565+0000","last_peered":"2024-04-25T00:01:20.520565+0000","last_clean":"2024-04-25T00:01:20.520565+0000","last_became_active":"2024-04-25T00:01:05.362606+0000","last_became_peered":"2024-04-25T00:01:05.362606+0000","last_unstale":"2024-04-25T00:01:20.520565+0000","last_undegraded":"2024-04-25T00:01:20.520565+0000","last_fullsized":"2024-04-25T00:01:20.520565+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:12:52.846250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.510932+0000","last_change":"2024-04-25T00:01:19.505737+0000","last_active":"2024-04-25T00:01:20.510932+0000","last_peered":"2024-04-25T00:01:20.510932+0000","last_clean":"2024-04-25T00:01:20.510932+0000","last_became_active":"2024-04-25T00:01:19.504991+0000","last_became_peered":"2024-04-25T00:01:19.504991+0000","last_unstale":"2024-04-25T00:01:20.510932+0000","last_undegraded":"2024-04-25T00:01:20.510932+0000","last_fullsized":"2024-04-25T00:01:20.510932+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:47:19.526845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.583309+0000","last_change":"2024-04-25T00:01:05.389596+0000","last_active":"2024-04-25T00:01:20.583309+0000","last_peered":"2024-04-25T00:01:20.583309+0000","last_clean":"2024-04-25T00:01:20.583309+0000","last_became_active":"2024-04-25T00:01:05.389127+0000","last_became_peered":"2024-04-25T00:01:05.389127+0000","last_unstale":"2024-04-25T00:01:20.583309+0000","last_undegraded":"2024-04-25T00:01:20.583309+0000","last_fullsized":"2024-04-25T00:01:20.583309+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:09.858617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509449,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":109716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509449,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":111182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509448,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27896,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343944,"statfs":{"total":96636764160,"available":96608198656,"internally_reserved":0,"allocated":888832,"data_stored":700612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509449,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27324,"kb_used_data":296,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344516,"statfs":{"total":96636764160,"available":96608784384,"internally_reserved":0,"allocated":303104,"data_stored":110928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509450,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27320,"kb_used_data":292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344520,"statfs":{"total":96636764160,"available":96608788480,"internally_reserved":0,"allocated":299008,"data_stored":111216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509449,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27892,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343948,"statfs":{"total":96636764160,"available":96608202752,"internally_reserved":0,"allocated":884736,"data_stored":700078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509449,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":109716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":278528,"data_stored":97170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T00:01:24.939 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-25T00:01:24.939 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-25T00:01:24.940 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-25T00:01:24.940 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-25T00:01:24.940 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-25T00:01:25.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.090+0000 7f5287054640 1 Processor -- start 2024-04-25T00:01:25.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.090+0000 7f5287054640 1 -- start start 2024-04-25T00:01:25.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.094+0000 7f5287054640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5280150d20 0x7f52800c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:25.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.094+0000 7f5287054640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f52800c2f50 0x7f52801538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:25.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.094+0000 7f5287054640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52800c3910 con 0x7f52800a94a0 2024-04-25T00:01:25.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.094+0000 7f5287054640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f5280154530 con 0x7f5280150d20 2024-04-25T00:01:25.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.094+0000 7f5287054640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f52800c16e0 con 0x7f52800c2f50 2024-04-25T00:01:25.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.094+0000 7f5284df0640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5280150d20 0x7f52800c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:25.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.094+0000 7f5284df0640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5280150d20 0x7f52800c2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55986/0 (socket says 172.21.15.116:55986) 2024-04-25T00:01:25.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.094+0000 7f5284df0640 1 -- 172.21.15.116:0/3858051450 learned_addr learned my addr 172.21.15.116:0/3858051450 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:25.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.094+0000 7f527ffff640 1 --2- 172.21.15.116:0/3858051450 >> v2:172.21.15.174:3300/0 conn(0x7f52800c2f50 0x7f52801538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:25.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.094+0000 7f5284df0640 1 -- 172.21.15.116:0/3858051450 >> v2:172.21.15.174:3300/0 conn(0x7f52800c2f50 msgr2=0x7f52801538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:25.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.094+0000 7f5284df0640 1 --2- 172.21.15.116:0/3858051450 >> v2:172.21.15.174:3300/0 conn(0x7f52800c2f50 0x7f52801538e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:25.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.094+0000 7f5284df0640 1 -- 172.21.15.116:0/3858051450 >> v1:172.21.15.116:6789/0 conn(0x7f52800a94a0 legacy=0x7f5280153e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:25.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.094+0000 7f5284df0640 1 -- 172.21.15.116:0/3858051450 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52800c1400 con 0x7f5280150d20 2024-04-25T00:01:25.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.094+0000 7f527ffff640 1 --2- 172.21.15.116:0/3858051450 >> v2:172.21.15.174:3300/0 conn(0x7f52800c2f50 0x7f52801538e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:25.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.094+0000 7f5284df0640 1 --2- 172.21.15.116:0/3858051450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5280150d20 0x7f52800c2a10 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5280070710 tx=0x7f527001c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec31491b7eabf8dc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:25.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.094+0000 7f527f7fe640 1 -- 172.21.15.116:0/3858051450 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52700a57d0 con 0x7f5280150d20 2024-04-25T00:01:25.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.094+0000 7f527f7fe640 1 -- 172.21.15.116:0/3858051450 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52700a5970 con 0x7f5280150d20 2024-04-25T00:01:25.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.094+0000 7f527f7fe640 1 -- 172.21.15.116:0/3858051450 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52700a5d40 con 0x7f5280150d20 2024-04-25T00:01:25.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.098+0000 7f5287054640 1 -- 172.21.15.116:0/3858051450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5280150d20 msgr2=0x7f52800c2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:25.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.098+0000 7f5287054640 1 --2- 172.21.15.116:0/3858051450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5280150d20 0x7f52800c2a10 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5280070710 tx=0x7f527001c9f0 comp rx=0 tx=0).stop 2024-04-25T00:01:25.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.098+0000 7f5287054640 1 -- 172.21.15.116:0/3858051450 shutdown_connections 2024-04-25T00:01:25.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.098+0000 7f5287054640 1 --2- 172.21.15.116:0/3858051450 >> v2:172.21.15.174:3300/0 conn(0x7f52800c2f50 0x7f52801538e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:25.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.098+0000 7f5287054640 1 --2- 172.21.15.116:0/3858051450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5280150d20 0x7f52800c2a10 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:25.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.098+0000 7f5287054640 1 -- 172.21.15.116:0/3858051450 >> 172.21.15.116:0/3858051450 conn(0x7f52800c3b00 msgr2=0x7f52800c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:25.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.098+0000 7f5287054640 1 -- 172.21.15.116:0/3858051450 shutdown_connections 2024-04-25T00:01:25.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.098+0000 7f5287054640 1 -- 172.21.15.116:0/3858051450 wait complete. 2024-04-25T00:01:25.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.098+0000 7f5287054640 1 Processor -- start 2024-04-25T00:01:25.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.098+0000 7f5287054640 1 -- start start 2024-04-25T00:01:25.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.098+0000 7f5287054640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52800a94a0 0x7f5280145e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:25.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.098+0000 7f5287054640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52800c2f50 0x7f528014cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:25.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.098+0000 7f5287054640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5280150d20 0x7f528013fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:25.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.098+0000 7f5287054640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f52801608b0 con 0x7f52800c2f50 2024-04-25T00:01:25.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.098+0000 7f5287054640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5280160730 con 0x7f52800a94a0 2024-04-25T00:01:25.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.098+0000 7f5287054640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f5280160a30 con 0x7f5280150d20 2024-04-25T00:01:25.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.098+0000 7f527ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52800c2f50 0x7f528014cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:25.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.098+0000 7f527ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52800c2f50 0x7f528014cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55998/0 (socket says 172.21.15.116:55998) 2024-04-25T00:01:25.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.098+0000 7f527ffff640 1 -- 172.21.15.116:0/1932399925 learned_addr learned my addr 172.21.15.116:0/1932399925 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:25.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.098+0000 7f527ffff640 1 -- 172.21.15.116:0/1932399925 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5280150d20 msgr2=0x7f528013fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:25.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.098+0000 7f5284df0640 1 --2- 172.21.15.116:0/1932399925 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52800a94a0 0x7f5280145e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:25.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.098+0000 7f52855f1640 1 --2- 172.21.15.116:0/1932399925 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5280150d20 0x7f528013fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:25.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.098+0000 7f527ffff640 1 --2- 172.21.15.116:0/1932399925 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5280150d20 0x7f528013fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:25.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.098+0000 7f527ffff640 1 -- 172.21.15.116:0/1932399925 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52800a94a0 msgr2=0x7f5280145e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:25.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.098+0000 7f527ffff640 1 --2- 172.21.15.116:0/1932399925 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52800a94a0 0x7f5280145e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:25.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.098+0000 7f527ffff640 1 -- 172.21.15.116:0/1932399925 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5280142440 con 0x7f52800c2f50 2024-04-25T00:01:25.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.098+0000 7f527ffff640 1 --2- 172.21.15.116:0/1932399925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52800c2f50 0x7f528014cf40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f526800c7c0 tx=0x7f526800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:25.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.098+0000 7f527d7fa640 1 -- 172.21.15.116:0/1932399925 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5268007a30 con 0x7f52800c2f50 2024-04-25T00:01:25.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.098+0000 7f527d7fa640 1 -- 172.21.15.116:0/1932399925 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5268007bd0 con 0x7f52800c2f50 2024-04-25T00:01:25.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.098+0000 7f527d7fa640 1 -- 172.21.15.116:0/1932399925 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52680113f0 con 0x7f52800c2f50 2024-04-25T00:01:25.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.098+0000 7f5287054640 1 -- 172.21.15.116:0/1932399925 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f528014dc30 con 0x7f52800c2f50 2024-04-25T00:01:25.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.098+0000 7f5287054640 1 -- 172.21.15.116:0/1932399925 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52801469b0 con 0x7f52800c2f50 2024-04-25T00:01:25.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.102+0000 7f5287054640 1 -- 172.21.15.116:0/1932399925 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5280068350 con 0x7f52800c2f50 2024-04-25T00:01:25.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.102+0000 7f527d7fa640 1 -- 172.21.15.116:0/1932399925 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5268011590 con 0x7f52800c2f50 2024-04-25T00:01:25.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.102+0000 7f527d7fa640 1 --2- 172.21.15.116:0/1932399925 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f52600678c0 0x7f5260069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:25.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.102+0000 7f527d7fa640 1 -- 172.21.15.116:0/1932399925 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7f52680861d0 con 0x7f52800c2f50 2024-04-25T00:01:25.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.102+0000 7f5284df0640 1 --2- 172.21.15.116:0/1932399925 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f52600678c0 0x7f5260069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:25.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.102+0000 7f5284df0640 1 --2- 172.21.15.116:0/1932399925 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f52600678c0 0x7f5260069d80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f52700ae000 tx=0x7f52700a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:25.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.106+0000 7f527d7fa640 1 -- 172.21.15.116:0/1932399925 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f5268015020 con 0x7f52800c2f50 2024-04-25T00:01:25.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.302+0000 7f5287054640 1 -- 172.21.15.116:0/1932399925 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f52800c34f0 con 0x7f52800c2f50 2024-04-25T00:01:25.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.310+0000 7f527d7fa640 1 -- 172.21.15.116:0/1932399925 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+166890 (secure 0 0 0) 0x7f52680569e0 con 0x7f52800c2f50 2024-04-25T00:01:25.313 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:01:25.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.314+0000 7f5287054640 1 -- 172.21.15.116:0/1932399925 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f52600678c0 msgr2=0x7f5260069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:25.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.314+0000 7f5287054640 1 --2- 172.21.15.116:0/1932399925 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f52600678c0 0x7f5260069d80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f52700ae000 tx=0x7f52700a7070 comp rx=0 tx=0).stop 2024-04-25T00:01:25.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.314+0000 7f5287054640 1 -- 172.21.15.116:0/1932399925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52800c2f50 msgr2=0x7f528014cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:25.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.314+0000 7f5287054640 1 --2- 172.21.15.116:0/1932399925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52800c2f50 0x7f528014cf40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f526800c7c0 tx=0x7f526800cc80 comp rx=0 tx=0).stop 2024-04-25T00:01:25.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.314+0000 7f5287054640 1 -- 172.21.15.116:0/1932399925 shutdown_connections 2024-04-25T00:01:25.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.314+0000 7f5287054640 1 --2- 172.21.15.116:0/1932399925 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f52600678c0 0x7f5260069d80 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:25.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.314+0000 7f5287054640 1 --2- 172.21.15.116:0/1932399925 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5280150d20 0x7f528013fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:25.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.314+0000 7f5287054640 1 --2- 172.21.15.116:0/1932399925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52800c2f50 0x7f528014cf40 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:25.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.314+0000 7f5287054640 1 --2- 172.21.15.116:0/1932399925 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52800a94a0 0x7f5280145e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:25.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.314+0000 7f5287054640 1 -- 172.21.15.116:0/1932399925 >> 172.21.15.116:0/1932399925 conn(0x7f52800c3b00 msgr2=0x7f528015d340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:25.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.314+0000 7f5287054640 1 -- 172.21.15.116:0/1932399925 shutdown_connections 2024-04-25T00:01:25.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.314+0000 7f5287054640 1 -- 172.21.15.116:0/1932399925 wait complete. 2024-04-25T00:01:25.329 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":5,"flags":0,"active_gid":4105,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":2004448211},{"type":"v1","addr":"172.21.15.174:6833","nonce":2004448211}]},"active_addr":"172.21.15.174:6833/2004448211","active_change":"2024-04-25T00:00:45.236704+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4104,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.174:0","nonce":325135112}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.174:0","nonce":7733609}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.174:0","nonce":1372722381}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.174:0","nonce":1143943554}]}]} 2024-04-25T00:01:25.333 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-25T00:01:25.333 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-25T00:01:25.333 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T00:01:25.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.450+0000 7ff3477fd640 1 Processor -- start 2024-04-25T00:01:25.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.450+0000 7ff3477fd640 1 -- start start 2024-04-25T00:01:25.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.450+0000 7ff3477fd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff340070770 0x7ff340154170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:25.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.450+0000 7ff3477fd640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff340070180 con 0x7ff3400c4c20 2024-04-25T00:01:25.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.450+0000 7ff3477fd640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff340070460 con 0x7ff340150d40 2024-04-25T00:01:25.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.450+0000 7ff3477fd640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7ff3400c7530 con 0x7ff340070770 2024-04-25T00:01:25.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.450+0000 7ff345d9a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff340070770 0x7ff340154170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:25.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.450+0000 7ff344d98640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7ff3400c4c20 0x7ff3400ab200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:41596/0 (socket says 172.21.15.116:41596) 2024-04-25T00:01:25.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.450+0000 7ff345d9a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff340070770 0x7ff340154170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56014/0 (socket says 172.21.15.116:56014) 2024-04-25T00:01:25.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.450+0000 7ff344d98640 1 -- 172.21.15.116:0/3105826109 learned_addr learned my addr 172.21.15.116:0/3105826109 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:25.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.450+0000 7ff345d9a640 1 -- 172.21.15.116:0/3105826109 >> v1:172.21.15.174:6789/0 conn(0x7ff340150d40 legacy=0x7ff3400c6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:25.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.450+0000 7ff345d9a640 1 -- 172.21.15.116:0/3105826109 >> v1:172.21.15.116:6789/0 conn(0x7ff3400c4c20 legacy=0x7ff3400ab200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:25.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.450+0000 7ff345d9a640 1 -- 172.21.15.116:0/3105826109 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff340153970 con 0x7ff340070770 2024-04-25T00:01:25.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.450+0000 7ff337fff640 1 -- 172.21.15.116:0/3105826109 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3207341622 0 0) 0x7ff340070180 con 0x7ff3400c4c20 2024-04-25T00:01:25.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.454+0000 7ff345d9a640 1 --2- 172.21.15.116:0/3105826109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff340070770 0x7ff340154170 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff338009d90 tx=0x7ff33801c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6b112aa6bbf6d5f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:25.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.454+0000 7ff337fff640 1 -- 172.21.15.116:0/3105826109 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3380b2070 con 0x7ff340070770 2024-04-25T00:01:25.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.454+0000 7ff337fff640 1 -- 172.21.15.116:0/3105826109 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3380a5760 con 0x7ff340070770 2024-04-25T00:01:25.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.454+0000 7ff337fff640 1 -- 172.21.15.116:0/3105826109 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3380a5b30 con 0x7ff340070770 2024-04-25T00:01:25.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.454+0000 7ff3477fd640 1 -- 172.21.15.116:0/3105826109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff340070770 msgr2=0x7ff340154170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:25.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.454+0000 7ff3477fd640 1 --2- 172.21.15.116:0/3105826109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff340070770 0x7ff340154170 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff338009d90 tx=0x7ff33801c8a0 comp rx=0 tx=0).stop 2024-04-25T00:01:25.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.454+0000 7ff3477fd640 1 -- 172.21.15.116:0/3105826109 shutdown_connections 2024-04-25T00:01:25.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.454+0000 7ff3477fd640 1 --2- 172.21.15.116:0/3105826109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff340070770 0x7ff340154170 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:25.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.454+0000 7ff3477fd640 1 -- 172.21.15.116:0/3105826109 >> 172.21.15.116:0/3105826109 conn(0x7ff34006f0c0 msgr2=0x7ff34006f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:25.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.454+0000 7ff3477fd640 1 -- 172.21.15.116:0/3105826109 shutdown_connections 2024-04-25T00:01:25.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.454+0000 7ff3477fd640 1 -- 172.21.15.116:0/3105826109 wait complete. 2024-04-25T00:01:25.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.454+0000 7ff3477fd640 1 Processor -- start 2024-04-25T00:01:25.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.454+0000 7ff3477fd640 1 -- start start 2024-04-25T00:01:25.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.454+0000 7ff3477fd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff340070770 0x7ff34014c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:25.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.454+0000 7ff3477fd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3400c4c20 0x7ff3400c25e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:25.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.454+0000 7ff3477fd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff340150d40 0x7ff3401ee870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:25.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.454+0000 7ff3477fd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff340151120 con 0x7ff3400c4c20 2024-04-25T00:01:25.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.454+0000 7ff3477fd640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff34006ff30 con 0x7ff340070770 2024-04-25T00:01:25.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.454+0000 7ff3477fd640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff3401512a0 con 0x7ff340150d40 2024-04-25T00:01:25.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.454+0000 7ff344d98640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3400c4c20 0x7ff3400c25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:25.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.454+0000 7ff344d98640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3400c4c20 0x7ff3400c25e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56030/0 (socket says 172.21.15.116:56030) 2024-04-25T00:01:25.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.454+0000 7ff344d98640 1 -- 172.21.15.116:0/3137544148 learned_addr learned my addr 172.21.15.116:0/3137544148 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:25.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.454+0000 7ff345599640 1 --2- 172.21.15.116:0/3137544148 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff340070770 0x7ff34014c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:25.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.454+0000 7ff345d9a640 1 --2- 172.21.15.116:0/3137544148 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff340150d40 0x7ff3401ee870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:25.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.454+0000 7ff344d98640 1 -- 172.21.15.116:0/3137544148 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff340150d40 msgr2=0x7ff3401ee870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:25.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.454+0000 7ff344d98640 1 --2- 172.21.15.116:0/3137544148 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff340150d40 0x7ff3401ee870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:25.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.454+0000 7ff344d98640 1 -- 172.21.15.116:0/3137544148 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff340070770 msgr2=0x7ff34014c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:25.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.454+0000 7ff344d98640 1 --2- 172.21.15.116:0/3137544148 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff340070770 0x7ff34014c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:25.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.454+0000 7ff344d98640 1 -- 172.21.15.116:0/3137544148 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3401f29f0 con 0x7ff3400c4c20 2024-04-25T00:01:25.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.454+0000 7ff344d98640 1 --2- 172.21.15.116:0/3137544148 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3400c4c20 0x7ff3400c25e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff33000c480 tx=0x7ff33000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:25.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.454+0000 7ff335ffb640 1 -- 172.21.15.116:0/3137544148 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff330018070 con 0x7ff3400c4c20 2024-04-25T00:01:25.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.454+0000 7ff335ffb640 1 -- 172.21.15.116:0/3137544148 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff330007500 con 0x7ff3400c4c20 2024-04-25T00:01:25.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.454+0000 7ff3477fd640 1 -- 172.21.15.116:0/3137544148 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff34006dcb0 con 0x7ff3400c4c20 2024-04-25T00:01:25.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.454+0000 7ff335ffb640 1 -- 172.21.15.116:0/3137544148 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff330004a30 con 0x7ff3400c4c20 2024-04-25T00:01:25.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.454+0000 7ff3477fd640 1 -- 172.21.15.116:0/3137544148 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3401f2680 con 0x7ff3400c4c20 2024-04-25T00:01:25.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.458+0000 7ff3477fd640 1 -- 172.21.15.116:0/3137544148 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff340068a60 con 0x7ff3400c4c20 2024-04-25T00:01:25.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.458+0000 7ff335ffb640 1 -- 172.21.15.116:0/3137544148 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff33001b070 con 0x7ff3400c4c20 2024-04-25T00:01:25.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.458+0000 7ff335ffb640 1 --2- 172.21.15.116:0/3137544148 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff3240678f0 0x7ff324069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:25.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.458+0000 7ff335ffb640 1 -- 172.21.15.116:0/3137544148 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7ff330085210 con 0x7ff3400c4c20 2024-04-25T00:01:25.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.458+0000 7ff345599640 1 --2- 172.21.15.116:0/3137544148 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff3240678f0 0x7ff324069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:25.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.458+0000 7ff345599640 1 --2- 172.21.15.116:0/3137544148 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff3240678f0 0x7ff324069db0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff340152c10 tx=0x7ff33c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:25.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.462+0000 7ff335ffb640 1 -- 172.21.15.116:0/3137544148 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ff330055ab0 con 0x7ff3400c4c20 2024-04-25T00:01:25.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.626+0000 7ff3477fd640 1 -- 172.21.15.116:0/3137544148 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff3401ee5d0 con 0x7ff3400c4c20 2024-04-25T00:01:25.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.626+0000 7ff335ffb640 1 -- 172.21.15.116:0/3137544148 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+20311 (secure 0 0 0) 0x7ff3300553b0 con 0x7ff3400c4c20 2024-04-25T00:01:25.629 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:01:25.629 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":27,"fsid":"6d6a1aa7-0478-4c83-8057-05917e81f046","created":"2024-04-25T00:00:40.682493+0000","modified":"2024-04-25T00:01:20.501508+0000","last_up_change":"2024-04-25T00:00:51.822174+0000","last_in_change":"2024-04-25T00:00:42.980473+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T00:00:53.324376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-04-25T00:01:03.197326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-04-25T00:01:03.728618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-04-25T00:01:17.125210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-04-25T00:01:17.821873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8b6ddcb2-1ad9-4167-961c-bd6936bb9060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6809","nonce":3751765682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6811","nonce":3751765682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6815","nonce":3751765682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6813","nonce":3751765682}]},"public_addr":"172.21.15.116:6809/3751765682","cluster_addr":"172.21.15.116:6811/3751765682","heartbeat_back_addr":"172.21.15.116:6815/3751765682","heartbeat_front_addr":"172.21.15.116:6813/3751765682","state":["exists","up"]},{"osd":1,"uuid":"a6431768-994f-41da-8853-886f5687fb91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6801","nonce":377879124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6803","nonce":377879124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6807","nonce":377879124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6805","nonce":377879124}]},"public_addr":"172.21.15.116:6801/377879124","cluster_addr":"172.21.15.116:6803/377879124","heartbeat_back_addr":"172.21.15.116:6807/377879124","heartbeat_front_addr":"172.21.15.116:6805/377879124","state":["exists","up"]},{"osd":2,"uuid":"4f69bd40-5a23-45ff-8693-e1de75e1aa9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6817","nonce":1099282267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6819","nonce":1099282267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6823","nonce":1099282267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6821","nonce":1099282267}]},"public_addr":"172.21.15.116:6817/1099282267","cluster_addr":"172.21.15.116:6819/1099282267","heartbeat_back_addr":"172.21.15.116:6823/1099282267","heartbeat_front_addr":"172.21.15.116:6821/1099282267","state":["exists","up"]},{"osd":3,"uuid":"6a0d5344-8b6f-40d3-918d-0f0a4be688e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6825","nonce":1179942661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6827","nonce":1179942661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6831","nonce":1179942661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6829","nonce":1179942661}]},"public_addr":"172.21.15.116:6825/1179942661","cluster_addr":"172.21.15.116:6827/1179942661","heartbeat_back_addr":"172.21.15.116:6831/1179942661","heartbeat_front_addr":"172.21.15.116:6829/1179942661","state":["exists","up"]},{"osd":4,"uuid":"132e92e4-b825-441e-965a-ca00533fa86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6801","nonce":3132444330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6803","nonce":3132444330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6807","nonce":3132444330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6805","nonce":3132444330}]},"public_addr":"172.21.15.174:6801/3132444330","cluster_addr":"172.21.15.174:6803/3132444330","heartbeat_back_addr":"172.21.15.174:6807/3132444330","heartbeat_front_addr":"172.21.15.174:6805/3132444330","state":["exists","up"]},{"osd":5,"uuid":"a2eeb5b6-1b31-43cc-8306-a0adf3ee15bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6809","nonce":881817164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6811","nonce":881817164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6815","nonce":881817164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6813","nonce":881817164}]},"public_addr":"172.21.15.174:6809/881817164","cluster_addr":"172.21.15.174:6811/881817164","heartbeat_back_addr":"172.21.15.174:6815/881817164","heartbeat_front_addr":"172.21.15.174:6813/881817164","state":["exists","up"]},{"osd":6,"uuid":"588e873a-c841-4ba7-90f2-8956c87a9983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6825","nonce":117574251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6827","nonce":117574251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6831","nonce":117574251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6829","nonce":117574251}]},"public_addr":"172.21.15.174:6825/117574251","cluster_addr":"172.21.15.174:6827/117574251","heartbeat_back_addr":"172.21.15.174:6831/117574251","heartbeat_front_addr":"172.21.15.174:6829/117574251","state":["exists","up"]},{"osd":7,"uuid":"3b90f9b0-8a0e-4883-b088-ca1d4012870c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6817","nonce":2543899196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6819","nonce":2543899196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6823","nonce":2543899196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6821","nonce":2543899196}]},"public_addr":"172.21.15.174:6817/2543899196","cluster_addr":"172.21.15.174:6819/2543899196","heartbeat_back_addr":"172.21.15.174:6823/2543899196","heartbeat_front_addr":"172.21.15.174:6821/2543899196","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.293769+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.310034+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.404865+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.549658+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.192174+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.297266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.493649+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.452309+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:01:25.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.630+0000 7ff3477fd640 1 -- 172.21.15.116:0/3137544148 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff3240678f0 msgr2=0x7ff324069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:25.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.630+0000 7ff3477fd640 1 --2- 172.21.15.116:0/3137544148 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff3240678f0 0x7ff324069db0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff340152c10 tx=0x7ff33c009040 comp rx=0 tx=0).stop 2024-04-25T00:01:25.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.630+0000 7ff3477fd640 1 -- 172.21.15.116:0/3137544148 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3400c4c20 msgr2=0x7ff3400c25e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:25.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.630+0000 7ff3477fd640 1 --2- 172.21.15.116:0/3137544148 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3400c4c20 0x7ff3400c25e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff33000c480 tx=0x7ff33000c940 comp rx=0 tx=0).stop 2024-04-25T00:01:25.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.630+0000 7ff3477fd640 1 -- 172.21.15.116:0/3137544148 shutdown_connections 2024-04-25T00:01:25.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.630+0000 7ff3477fd640 1 --2- 172.21.15.116:0/3137544148 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff3240678f0 0x7ff324069db0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:25.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.630+0000 7ff3477fd640 1 --2- 172.21.15.116:0/3137544148 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff340150d40 0x7ff3401ee870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:25.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.630+0000 7ff3477fd640 1 --2- 172.21.15.116:0/3137544148 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff3400c4c20 0x7ff3400c25e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:25.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.630+0000 7ff3477fd640 1 --2- 172.21.15.116:0/3137544148 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff340070770 0x7ff34014c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:25.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.630+0000 7ff3477fd640 1 -- 172.21.15.116:0/3137544148 >> 172.21.15.116:0/3137544148 conn(0x7ff34006f0c0 msgr2=0x7ff340150370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:25.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.630+0000 7ff3477fd640 1 -- 172.21.15.116:0/3137544148 shutdown_connections 2024-04-25T00:01:25.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.630+0000 7ff3477fd640 1 -- 172.21.15.116:0/3137544148 wait complete. 2024-04-25T00:01:25.648 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-25T00:01:25.649 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T00:01:25.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.754+0000 7efff35e5640 1 Processor -- start 2024-04-25T00:01:25.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.754+0000 7efff35e5640 1 -- start start 2024-04-25T00:01:25.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.758+0000 7efff35e5640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7effec0cacc0 0x7effec14c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:25.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.758+0000 7efff35e5640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7effec0c5ac0 0x7effec0c9040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:25.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.758+0000 7efff35e5640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7effec156720 con 0x7effec14bf10 2024-04-25T00:01:25.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.758+0000 7efff35e5640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7effec0c8460 con 0x7effec0c5ac0 2024-04-25T00:01:25.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.758+0000 7efff35e5640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7effec156440 con 0x7effec0cacc0 2024-04-25T00:01:25.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.758+0000 7efff0b80640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7effec0cacc0 0x7effec14c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:25.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.758+0000 7efff1b82640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7effec0c5ac0 0x7effec0c9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:25.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.758+0000 7efff1b82640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7effec0c5ac0 0x7effec0c9040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:37136/0 (socket says 172.21.15.116:37136) 2024-04-25T00:01:25.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.758+0000 7efff1b82640 1 -- 172.21.15.116:0/1841629089 learned_addr learned my addr 172.21.15.116:0/1841629089 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:25.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.758+0000 7efff0b80640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7effec0cacc0 0x7effec14c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:55874/0 (socket says 172.21.15.116:55874) 2024-04-25T00:01:25.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.758+0000 7efff1b82640 1 -- 172.21.15.116:0/1841629089 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7effec0cacc0 msgr2=0x7effec14c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:25.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.758+0000 7efff1b82640 1 --2- 172.21.15.116:0/1841629089 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7effec0cacc0 0x7effec14c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:25.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.758+0000 7efff1b82640 1 -- 172.21.15.116:0/1841629089 >> v1:172.21.15.174:6789/0 conn(0x7effec14bf10 legacy=0x7effec06be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:25.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.758+0000 7efff1b82640 1 -- 172.21.15.116:0/1841629089 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effec0c7c60 con 0x7effec0c5ac0 2024-04-25T00:01:25.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.758+0000 7efff1b82640 1 --2- 172.21.15.116:0/1841629089 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7effec0c5ac0 0x7effec0c9040 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7effe4009e50 tx=0x7effe401ca30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6af04bf0e053fc9a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:25.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.758+0000 7effe3fff640 1 -- 172.21.15.116:0/1841629089 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe40a5720 con 0x7effec0c5ac0 2024-04-25T00:01:25.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.758+0000 7effe3fff640 1 -- 172.21.15.116:0/1841629089 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effe40a58c0 con 0x7effec0c5ac0 2024-04-25T00:01:25.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.758+0000 7effe3fff640 1 -- 172.21.15.116:0/1841629089 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe40a5c90 con 0x7effec0c5ac0 2024-04-25T00:01:25.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.762+0000 7efff35e5640 1 -- 172.21.15.116:0/1841629089 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7effec0c5ac0 msgr2=0x7effec0c9040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:25.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.762+0000 7efff35e5640 1 --2- 172.21.15.116:0/1841629089 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7effec0c5ac0 0x7effec0c9040 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7effe4009e50 tx=0x7effe401ca30 comp rx=0 tx=0).stop 2024-04-25T00:01:25.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.762+0000 7efff35e5640 1 -- 172.21.15.116:0/1841629089 shutdown_connections 2024-04-25T00:01:25.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.762+0000 7efff35e5640 1 --2- 172.21.15.116:0/1841629089 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7effec0c5ac0 0x7effec0c9040 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:25.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.762+0000 7efff35e5640 1 --2- 172.21.15.116:0/1841629089 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7effec0cacc0 0x7effec14c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:25.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.762+0000 7efff35e5640 1 -- 172.21.15.116:0/1841629089 >> 172.21.15.116:0/1841629089 conn(0x7effec0ace40 msgr2=0x7effec0ad250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:25.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.762+0000 7efff35e5640 1 -- 172.21.15.116:0/1841629089 shutdown_connections 2024-04-25T00:01:25.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.762+0000 7efff35e5640 1 -- 172.21.15.116:0/1841629089 wait complete. 2024-04-25T00:01:25.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.762+0000 7efff35e5640 1 Processor -- start 2024-04-25T00:01:25.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.762+0000 7efff35e5640 1 -- start start 2024-04-25T00:01:25.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.762+0000 7efff35e5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effec0c5ac0 0x7effec1603c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:25.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.762+0000 7efff35e5640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7effec0cacc0 0x7effec160900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:25.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.762+0000 7efff35e5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7effec14bf10 0x7effec0c4870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:25.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.762+0000 7efff35e5640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7effec0c9b10 con 0x7effec0c5ac0 2024-04-25T00:01:25.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.762+0000 7efff35e5640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7effec0c9990 con 0x7effec14bf10 2024-04-25T00:01:25.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.762+0000 7efff35e5640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7effec0b07c0 con 0x7effec0cacc0 2024-04-25T00:01:25.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.762+0000 7efff1381640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effec0c5ac0 0x7effec1603c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:25.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.762+0000 7efff1381640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effec0c5ac0 0x7effec1603c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56032/0 (socket says 172.21.15.116:56032) 2024-04-25T00:01:25.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.762+0000 7efff1381640 1 -- 172.21.15.116:0/1625102987 learned_addr learned my addr 172.21.15.116:0/1625102987 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:25.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.762+0000 7efff0b80640 1 --2- 172.21.15.116:0/1625102987 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7effec0cacc0 0x7effec160900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:25.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.762+0000 7efff1b82640 1 --2- 172.21.15.116:0/1625102987 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7effec14bf10 0x7effec0c4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:25.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.762+0000 7efff1381640 1 -- 172.21.15.116:0/1625102987 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7effec0cacc0 msgr2=0x7effec160900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:25.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.762+0000 7efff1381640 1 --2- 172.21.15.116:0/1625102987 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7effec0cacc0 0x7effec160900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:25.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.762+0000 7efff1381640 1 -- 172.21.15.116:0/1625102987 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7effec14bf10 msgr2=0x7effec0c4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:25.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.762+0000 7efff1381640 1 --2- 172.21.15.116:0/1625102987 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7effec14bf10 0x7effec0c4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:25.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.762+0000 7efff1381640 1 -- 172.21.15.116:0/1625102987 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effec0c2d30 con 0x7effec0c5ac0 2024-04-25T00:01:25.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.762+0000 7efff0b80640 1 --2- 172.21.15.116:0/1625102987 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7effec0cacc0 0x7effec160900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:25.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.762+0000 7efff1b82640 1 --2- 172.21.15.116:0/1625102987 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7effec14bf10 0x7effec0c4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:25.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.762+0000 7efff1381640 1 --2- 172.21.15.116:0/1625102987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effec0c5ac0 0x7effec1603c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7effe800c3a0 tx=0x7effe800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:25.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.762+0000 7effe1ffb640 1 -- 172.21.15.116:0/1625102987 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe80052d0 con 0x7effec0c5ac0 2024-04-25T00:01:25.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.762+0000 7effe1ffb640 1 -- 172.21.15.116:0/1625102987 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effe8005470 con 0x7effec0c5ac0 2024-04-25T00:01:25.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.762+0000 7effe1ffb640 1 -- 172.21.15.116:0/1625102987 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe8007260 con 0x7effec0c5ac0 2024-04-25T00:01:25.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.762+0000 7efff35e5640 1 -- 172.21.15.116:0/1625102987 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effec159890 con 0x7effec0c5ac0 2024-04-25T00:01:25.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.762+0000 7efff35e5640 1 -- 172.21.15.116:0/1625102987 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effec0c2520 con 0x7effec0c5ac0 2024-04-25T00:01:25.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.766+0000 7efff35e5640 1 -- 172.21.15.116:0/1625102987 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effec1596b0 con 0x7effec0c5ac0 2024-04-25T00:01:25.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.766+0000 7effe1ffb640 1 -- 172.21.15.116:0/1625102987 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7effe8007400 con 0x7effec0c5ac0 2024-04-25T00:01:25.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.766+0000 7effe1ffb640 1 --2- 172.21.15.116:0/1625102987 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7effd00678f0 0x7effd0069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:25.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.766+0000 7effe1ffb640 1 -- 172.21.15.116:0/1625102987 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7effe80853a0 con 0x7effec0c5ac0 2024-04-25T00:01:25.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.766+0000 7efff0b80640 1 --2- 172.21.15.116:0/1625102987 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7effd00678f0 0x7effd0069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:25.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.766+0000 7efff0b80640 1 --2- 172.21.15.116:0/1625102987 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7effd00678f0 0x7effd0069db0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7effec0c2050 tx=0x7effdc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:25.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.770+0000 7effe1ffb640 1 -- 172.21.15.116:0/1625102987 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7effec1596b0 con 0x7effec0c5ac0 2024-04-25T00:01:25.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.914+0000 7efff35e5640 1 -- 172.21.15.116:0/1625102987 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7effec1eeeb0 con 0x7effec0c5ac0 2024-04-25T00:01:25.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.918+0000 7effe1ffb640 1 -- 172.21.15.116:0/1625102987 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+20311 (secure 0 0 0) 0x7effe8055c40 con 0x7effec0c5ac0 2024-04-25T00:01:25.922 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:01:25.922 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":27,"fsid":"6d6a1aa7-0478-4c83-8057-05917e81f046","created":"2024-04-25T00:00:40.682493+0000","modified":"2024-04-25T00:01:20.501508+0000","last_up_change":"2024-04-25T00:00:51.822174+0000","last_in_change":"2024-04-25T00:00:42.980473+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T00:00:53.324376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-04-25T00:01:03.197326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-04-25T00:01:03.728618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-04-25T00:01:17.125210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-04-25T00:01:17.821873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8b6ddcb2-1ad9-4167-961c-bd6936bb9060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6809","nonce":3751765682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6811","nonce":3751765682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6815","nonce":3751765682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6813","nonce":3751765682}]},"public_addr":"172.21.15.116:6809/3751765682","cluster_addr":"172.21.15.116:6811/3751765682","heartbeat_back_addr":"172.21.15.116:6815/3751765682","heartbeat_front_addr":"172.21.15.116:6813/3751765682","state":["exists","up"]},{"osd":1,"uuid":"a6431768-994f-41da-8853-886f5687fb91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6801","nonce":377879124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6803","nonce":377879124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6807","nonce":377879124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6805","nonce":377879124}]},"public_addr":"172.21.15.116:6801/377879124","cluster_addr":"172.21.15.116:6803/377879124","heartbeat_back_addr":"172.21.15.116:6807/377879124","heartbeat_front_addr":"172.21.15.116:6805/377879124","state":["exists","up"]},{"osd":2,"uuid":"4f69bd40-5a23-45ff-8693-e1de75e1aa9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6817","nonce":1099282267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6819","nonce":1099282267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6823","nonce":1099282267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6821","nonce":1099282267}]},"public_addr":"172.21.15.116:6817/1099282267","cluster_addr":"172.21.15.116:6819/1099282267","heartbeat_back_addr":"172.21.15.116:6823/1099282267","heartbeat_front_addr":"172.21.15.116:6821/1099282267","state":["exists","up"]},{"osd":3,"uuid":"6a0d5344-8b6f-40d3-918d-0f0a4be688e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6825","nonce":1179942661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6827","nonce":1179942661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6831","nonce":1179942661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6829","nonce":1179942661}]},"public_addr":"172.21.15.116:6825/1179942661","cluster_addr":"172.21.15.116:6827/1179942661","heartbeat_back_addr":"172.21.15.116:6831/1179942661","heartbeat_front_addr":"172.21.15.116:6829/1179942661","state":["exists","up"]},{"osd":4,"uuid":"132e92e4-b825-441e-965a-ca00533fa86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6801","nonce":3132444330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6803","nonce":3132444330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6807","nonce":3132444330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6805","nonce":3132444330}]},"public_addr":"172.21.15.174:6801/3132444330","cluster_addr":"172.21.15.174:6803/3132444330","heartbeat_back_addr":"172.21.15.174:6807/3132444330","heartbeat_front_addr":"172.21.15.174:6805/3132444330","state":["exists","up"]},{"osd":5,"uuid":"a2eeb5b6-1b31-43cc-8306-a0adf3ee15bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6809","nonce":881817164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6811","nonce":881817164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6815","nonce":881817164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6813","nonce":881817164}]},"public_addr":"172.21.15.174:6809/881817164","cluster_addr":"172.21.15.174:6811/881817164","heartbeat_back_addr":"172.21.15.174:6815/881817164","heartbeat_front_addr":"172.21.15.174:6813/881817164","state":["exists","up"]},{"osd":6,"uuid":"588e873a-c841-4ba7-90f2-8956c87a9983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6825","nonce":117574251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6827","nonce":117574251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6831","nonce":117574251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6829","nonce":117574251}]},"public_addr":"172.21.15.174:6825/117574251","cluster_addr":"172.21.15.174:6827/117574251","heartbeat_back_addr":"172.21.15.174:6831/117574251","heartbeat_front_addr":"172.21.15.174:6829/117574251","state":["exists","up"]},{"osd":7,"uuid":"3b90f9b0-8a0e-4883-b088-ca1d4012870c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6817","nonce":2543899196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6819","nonce":2543899196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6823","nonce":2543899196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6821","nonce":2543899196}]},"public_addr":"172.21.15.174:6817/2543899196","cluster_addr":"172.21.15.174:6819/2543899196","heartbeat_back_addr":"172.21.15.174:6823/2543899196","heartbeat_front_addr":"172.21.15.174:6821/2543899196","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.293769+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.310034+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.404865+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.549658+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.192174+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.297266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.493649+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.452309+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:01:25.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.922+0000 7efff35e5640 1 -- 172.21.15.116:0/1625102987 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7effd00678f0 msgr2=0x7effd0069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:25.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.922+0000 7efff35e5640 1 --2- 172.21.15.116:0/1625102987 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7effd00678f0 0x7effd0069db0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7effec0c2050 tx=0x7effdc008040 comp rx=0 tx=0).stop 2024-04-25T00:01:25.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.922+0000 7efff35e5640 1 -- 172.21.15.116:0/1625102987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effec0c5ac0 msgr2=0x7effec1603c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:25.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.922+0000 7efff35e5640 1 --2- 172.21.15.116:0/1625102987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effec0c5ac0 0x7effec1603c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7effe800c3a0 tx=0x7effe800c860 comp rx=0 tx=0).stop 2024-04-25T00:01:25.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.922+0000 7efff35e5640 1 -- 172.21.15.116:0/1625102987 shutdown_connections 2024-04-25T00:01:25.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.922+0000 7efff35e5640 1 --2- 172.21.15.116:0/1625102987 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7effd00678f0 0x7effd0069db0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:25.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.922+0000 7efff35e5640 1 --2- 172.21.15.116:0/1625102987 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7effec14bf10 0x7effec0c4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:25.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.922+0000 7efff35e5640 1 --2- 172.21.15.116:0/1625102987 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7effec0cacc0 0x7effec160900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:25.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.922+0000 7efff35e5640 1 --2- 172.21.15.116:0/1625102987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effec0c5ac0 0x7effec1603c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:25.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.922+0000 7efff35e5640 1 -- 172.21.15.116:0/1625102987 >> 172.21.15.116:0/1625102987 conn(0x7effec0ace40 msgr2=0x7effec0af0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:25.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.922+0000 7efff35e5640 1 -- 172.21.15.116:0/1625102987 shutdown_connections 2024-04-25T00:01:25.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:25.922+0000 7efff35e5640 1 -- 172.21.15.116:0/1625102987 wait complete. 2024-04-25T00:01:25.939 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-25T00:01:25.940 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-25T00:01:25.940 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-25T00:01:25.941 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-25T00:01:25.941 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-25T00:01:25.941 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-25T00:01:25.942 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-04-25T00:01:25.942 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-04-25T00:01:26.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.114+0000 7fbf10b3c640 1 Processor -- start 2024-04-25T00:01:26.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.114+0000 7f69a9d94640 1 Processor -- start 2024-04-25T00:01:26.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.114+0000 7f0f0c266640 1 Processor -- start 2024-04-25T00:01:26.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.114+0000 7f4481952640 1 Processor -- start 2024-04-25T00:01:26.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.114+0000 7f4481952640 1 -- start start 2024-04-25T00:01:26.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f4481952640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f447c157240 0x7f447c159bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.114+0000 7f9b7976c640 1 Processor -- start 2024-04-25T00:01:26.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.114+0000 7f9b7976c640 1 -- start start 2024-04-25T00:01:26.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f9b7976c640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f9b7406b210 0x7f9b74072160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f9b7976c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9b740a9ff0 0x7f9b740726a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f9b7976c640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b740716a0 con 0x7f9b74157310 2024-04-25T00:01:26.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f9b7976c640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f9b7415ab30 con 0x7f9b740a9ff0 2024-04-25T00:01:26.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.114+0000 7fbf10b3c640 1 -- start start 2024-04-25T00:01:26.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7fbf10b3c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbf0400b0c0 0x7fbf040e2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f9b7976c640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f9b74071960 con 0x7f9b7406b210 2024-04-25T00:01:26.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f9b737fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9b740a9ff0 0x7f9b740726a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f9b737fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9b740a9ff0 0x7f9b740726a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56038/0 (socket says 172.21.15.116:56038) 2024-04-25T00:01:26.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f9b737fe640 1 -- 172.21.15.116:0/2669722268 learned_addr learned my addr 172.21.15.116:0/2669722268 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:26.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7fbf10b3c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbf040ded20 0x7fbf040df1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7fbf10b3c640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf040de870 con 0x7fbf040dbce0 2024-04-25T00:01:26.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7fbf10b3c640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fbf040e2c50 con 0x7fbf040ded20 2024-04-25T00:01:26.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7fbf10b3c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fbf040df710 con 0x7fbf0400b0c0 2024-04-25T00:01:26.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.114+0000 7f0f0c266640 1 -- start start 2024-04-25T00:01:26.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f0f0c266640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f0f04070dd0 0x7f0f04071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f0f0c266640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f04154df0 con 0x7f0f0414b830 2024-04-25T00:01:26.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f0f0c266640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f041550d0 con 0x7f0f0406e100 2024-04-25T00:01:26.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f0f0c266640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f0f0414bc10 con 0x7f0f04070dd0 2024-04-25T00:01:26.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f9b727fc640 1 --2- 172.21.15.116:0/2669722268 >> v2:172.21.15.174:3300/0 conn(0x7f9b7406b210 0x7f9b74072160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.114+0000 7f87d10e2640 1 Processor -- start 2024-04-25T00:01:26.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f0f09801640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f0f0406e100 0x7f0f040c2a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:35522/0 (socket says 172.21.15.116:35522) 2024-04-25T00:01:26.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f0f09801640 1 -- 172.21.15.116:0/1581818836 learned_addr learned my addr 172.21.15.116:0/1581818836 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:26.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.114+0000 7f87d10e2640 1 -- start start 2024-04-25T00:01:26.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f69a9d94640 1 -- start start 2024-04-25T00:01:26.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7fbf0bfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbf040ded20 0x7fbf040df1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7fbf0bfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbf040ded20 0x7fbf040df1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56046/0 (socket says 172.21.15.116:56046) 2024-04-25T00:01:26.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7fbf0bfff640 1 -- 172.21.15.116:0/3400997428 learned_addr learned my addr 172.21.15.116:0/3400997428 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:26.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f4480950640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f447c157240 0x7f447c159bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f4480950640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f447c157240 0x7f447c159bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56074/0 (socket says 172.21.15.116:56074) 2024-04-25T00:01:26.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f447bfff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f447c15d130 0x7f447c0c2850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:41624/0 (socket says 172.21.15.116:41624) 2024-04-25T00:01:26.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f87d10e2640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f87cc0c2ab0 0x7f87cc15f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f87d10e2640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f87cc165ac0 0x7f87cc15fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f87d10e2640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f87cc070ff0 0x7f87cc071440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f87d10e2640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f87cc1629a0 con 0x7f87cc0c2ab0 2024-04-25T00:01:26.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f87d10e2640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f87cc162c80 con 0x7f87cc165ac0 2024-04-25T00:01:26.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f87d10e2640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f87cc162f60 con 0x7f87cc070ff0 2024-04-25T00:01:26.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f87d08e1640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f87cc070ff0 0x7f87cc071440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f87cbfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f87cc0c2ab0 0x7f87cc15f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f87cbfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f87cc0c2ab0 0x7f87cc15f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56062/0 (socket says 172.21.15.116:56062) 2024-04-25T00:01:26.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f87cbfff640 1 -- 172.21.15.116:0/2739205918 learned_addr learned my addr 172.21.15.116:0/2739205918 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:26.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f9b71ffb640 1 -- 172.21.15.116:0/2669722268 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2801631968 0 0) 0x7f9b740716a0 con 0x7f9b74157310 2024-04-25T00:01:26.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f9b71ffb640 1 -- 172.21.15.116:0/2669722268 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b5c003530 con 0x7f9b74157310 2024-04-25T00:01:26.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f9b737fe640 1 -- 172.21.15.116:0/2669722268 >> v2:172.21.15.174:3300/0 conn(0x7f9b7406b210 msgr2=0x7f9b74072160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:26.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f9b737fe640 1 --2- 172.21.15.116:0/2669722268 >> v2:172.21.15.174:3300/0 conn(0x7f9b7406b210 0x7f9b74072160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f9b737fe640 1 -- 172.21.15.116:0/2669722268 >> v1:172.21.15.174:6789/0 conn(0x7f9b74157310 legacy=0x7f9b740c3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f9b737fe640 1 -- 172.21.15.116:0/2669722268 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b74071c40 con 0x7f9b740a9ff0 2024-04-25T00:01:26.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f87cb7fe640 1 --2- 172.21.15.116:0/2739205918 >> v2:172.21.15.174:3300/0 conn(0x7f87cc165ac0 0x7f87cc15fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7fbf0affd640 1 --2- 172.21.15.116:0/3400997428 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbf0400b0c0 0x7fbf040e2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f0f09000640 1 -- 172.21.15.116:0/1581818836 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3667908305 0 0) 0x7f0f041550d0 con 0x7f0f0406e100 2024-04-25T00:01:26.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f0f09000640 1 -- 172.21.15.116:0/1581818836 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ef0003530 con 0x7f0f0406e100 2024-04-25T00:01:26.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f87d08e1640 1 -- 172.21.15.116:0/2739205918 >> v2:172.21.15.174:3300/0 conn(0x7f87cc165ac0 msgr2=0x7f87cc15fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:26.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f0f09000640 1 -- 172.21.15.116:0/1581818836 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 251131629 0 0) 0x7f0f04154df0 con 0x7f0f0414b830 2024-04-25T00:01:26.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f0f09000640 1 -- 172.21.15.116:0/1581818836 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f041550d0 con 0x7f0f0414b830 2024-04-25T00:01:26.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f0f09000640 1 -- 172.21.15.116:0/1581818836 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2053329401 0 0) 0x7f0ef0003530 con 0x7f0f0406e100 2024-04-25T00:01:26.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f0f09000640 1 -- 172.21.15.116:0/1581818836 >> v2:172.21.15.174:3300/0 conn(0x7f0f04070dd0 msgr2=0x7f0f04071260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T00:01:26.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f87d08e1640 1 --2- 172.21.15.116:0/2739205918 >> v2:172.21.15.174:3300/0 conn(0x7f87cc165ac0 0x7f87cc15fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f87d08e1640 1 -- 172.21.15.116:0/2739205918 >> v2:172.21.15.116:3300/0 conn(0x7f87cc0c2ab0 msgr2=0x7f87cc15f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:26.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f69a9d94640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f69a40c3f70 0x7f69a41557c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f69a9d94640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69a4153220 con 0x7f69a40c3a20 2024-04-25T00:01:26.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f69a9d94640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69a40c7a90 con 0x7f69a4152e40 2024-04-25T00:01:26.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f69a9d94640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f69a40c4c90 con 0x7f69a40c3f70 2024-04-25T00:01:26.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7fbf0bfff640 1 -- 172.21.15.116:0/3400997428 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbf0400b0c0 msgr2=0x7fbf040e2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:26.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7fbf0bfff640 1 --2- 172.21.15.116:0/3400997428 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbf0400b0c0 0x7fbf040e2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7fbf0bfff640 1 -- 172.21.15.116:0/3400997428 >> v1:172.21.15.116:6789/0 conn(0x7fbf040dbce0 legacy=0x7fbf04006eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:26.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7fbf0bfff640 1 -- 172.21.15.116:0/3400997428 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf04047620 con 0x7fbf040ded20 2024-04-25T00:01:26.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f87d08e1640 1 --2- 172.21.15.116:0/2739205918 >> v2:172.21.15.116:3300/0 conn(0x7f87cc0c2ab0 0x7f87cc15f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f87d08e1640 1 -- 172.21.15.116:0/2739205918 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87cc163240 con 0x7f87cc070ff0 2024-04-25T00:01:26.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f0f09000640 1 --2- 172.21.15.116:0/1581818836 >> v2:172.21.15.174:3300/0 conn(0x7f0f04070dd0 0x7f0f04071260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f0f09000640 1 -- 172.21.15.116:0/1581818836 >> v1:172.21.15.116:6789/0 conn(0x7f0f0414b830 legacy=0x7f0f04152150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f0f09000640 1 -- 172.21.15.116:0/1581818836 --> v1:172.21.15.174:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f041553b0 con 0x7f0f0406e100 2024-04-25T00:01:26.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f0f09000640 1 -- 172.21.15.116:0/1581818836 <== mon.1 v1:172.21.15.174:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f0ef8005040 con 0x7f0f0406e100 2024-04-25T00:01:26.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f69a37fe640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f69a4152e40 0x7f69a40710b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:35528/0 (socket says 172.21.15.116:35528) 2024-04-25T00:01:26.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f69a37fe640 1 -- 172.21.15.116:0/2600413532 learned_addr learned my addr 172.21.15.116:0/2600413532 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:26.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f69a2ffd640 1 --2- 172.21.15.116:0/2600413532 >> v2:172.21.15.174:3300/0 conn(0x7f69a40c3f70 0x7f69a41557c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f4481952640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f447c071130 0x7f447c0715c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f4481952640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f447c157620 con 0x7f447c15d130 2024-04-25T00:01:26.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f4481952640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f447c070ca0 con 0x7f447c157240 2024-04-25T00:01:26.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f4481952640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f447c15a1a0 con 0x7f447c071130 2024-04-25T00:01:26.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f0f09000640 1 -- 172.21.15.116:0/1581818836 >> v1:172.21.15.174:6789/0 conn(0x7f0f0406e100 legacy=0x7f0f040c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f0f09000640 1 --2- 172.21.15.116:0/1581818836 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ef0003b50 0x7f0ef0003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f0f09000640 1 --2- 172.21.15.116:0/1581818836 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0ef0004810 0x7f0ef0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f0f09000640 1 --2- 172.21.15.116:0/1581818836 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0ef00053b0 0x7f0ef00079f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f0f09000640 1 -- 172.21.15.116:0/1581818836 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0f0414bc10 con 0x7f0ef0003b50 2024-04-25T00:01:26.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f0f09000640 1 -- 172.21.15.116:0/1581818836 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef00082d0 con 0x7f0ef0004810 2024-04-25T00:01:26.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f0f09000640 1 -- 172.21.15.116:0/1581818836 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f0ef00084d0 con 0x7f0ef00053b0 2024-04-25T00:01:26.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f4480950640 1 -- 172.21.15.116:0/3020099349 learned_addr learned my addr 172.21.15.116:0/3020099349 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:26.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f6afd325640 1 Processor -- start 2024-04-25T00:01:26.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f4481151640 1 --2- 172.21.15.116:0/3020099349 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f447c071130 0x7f447c0715c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f6afd325640 1 -- start start 2024-04-25T00:01:26.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f69a2ffd640 1 -- 172.21.15.116:0/2600413532 >> v1:172.21.15.174:6789/0 conn(0x7f69a4152e40 legacy=0x7f69a40710b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f69a2ffd640 1 -- 172.21.15.116:0/2600413532 >> v1:172.21.15.116:6789/0 conn(0x7f69a40c3a20 legacy=0x7f69a41600e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f69a2ffd640 1 -- 172.21.15.116:0/2600413532 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69a40c7d70 con 0x7f69a40c3f70 2024-04-25T00:01:26.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f94da64e640 1 Processor -- start 2024-04-25T00:01:26.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f94da64e640 1 -- start start 2024-04-25T00:01:26.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f4481151640 1 -- 172.21.15.116:0/3020099349 >> v2:172.21.15.116:3300/0 conn(0x7f447c157240 msgr2=0x7f447c159bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:26.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f4481151640 1 --2- 172.21.15.116:0/3020099349 >> v2:172.21.15.116:3300/0 conn(0x7f447c157240 0x7f447c159bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f4481151640 1 -- 172.21.15.116:0/3020099349 >> v1:172.21.15.116:6789/0 conn(0x7f447c15d130 legacy=0x7f447c0c2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:26.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f4481151640 1 -- 172.21.15.116:0/3020099349 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f447c15a480 con 0x7f447c071130 2024-04-25T00:01:26.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f0f09801640 1 --2- 172.21.15.116:0/1581818836 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0ef0004810 0x7f0ef0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f0f0a002640 1 --2- 172.21.15.116:0/1581818836 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ef0003b50 0x7f0ef0003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f6afd325640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f6af80c45b0 0x7f6af813a100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f6afd325640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6af813a640 0x7f6af80c29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f6afd325640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6af8073040 con 0x7f6af80c2ee0 2024-04-25T00:01:26.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f6afd325640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f6af8070cf0 con 0x7f6af80c45b0 2024-04-25T00:01:26.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f6afd325640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6af81557e0 con 0x7f6af813a640 2024-04-25T00:01:26.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f94da64e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f94d4152ee0 0x7f94d40c5950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f94da64e640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f94d40be660 0x7f94d4155f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f94da64e640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94d4159c20 con 0x7f94d40c8ed0 2024-04-25T00:01:26.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f94da64e640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f94d41596f0 con 0x7f94d40be660 2024-04-25T00:01:26.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f94da64e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f94d4159940 con 0x7f94d4152ee0 2024-04-25T00:01:26.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f94d8beb640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f94d40be660 0x7f94d4155f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f94d8beb640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f94d40be660 0x7f94d4155f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:37188/0 (socket says 172.21.15.116:37188) 2024-04-25T00:01:26.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f94d8beb640 1 -- 172.21.15.116:0/2553293899 learned_addr learned my addr 172.21.15.116:0/2553293899 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:26.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7fbf0bfff640 1 --2- 172.21.15.116:0/3400997428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf040ded20 0x7fbf040df1d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c04fa30 tx=0x7fbf0c097fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1ce7bd36f0ffa8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:26.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7fbf0a7fc640 1 -- 172.21.15.116:0/3400997428 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf0c1208b0 con 0x7fbf040ded20 2024-04-25T00:01:26.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7fbf0a7fc640 1 -- 172.21.15.116:0/3400997428 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf0c120a50 con 0x7fbf040ded20 2024-04-25T00:01:26.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7fbf0a7fc640 1 -- 172.21.15.116:0/3400997428 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf0c128440 con 0x7fbf040ded20 2024-04-25T00:01:26.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f0f09801640 1 -- 172.21.15.116:0/1581818836 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0ef00053b0 msgr2=0x7f0ef00079f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T00:01:26.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f0f09801640 1 --2- 172.21.15.116:0/1581818836 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0ef00053b0 0x7f0ef00079f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f0f09801640 1 -- 172.21.15.116:0/1581818836 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ef0003b50 msgr2=0x7f0ef0003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:26.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f0f09801640 1 --2- 172.21.15.116:0/1581818836 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ef0003b50 0x7f0ef0003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.118+0000 7f0f09801640 1 -- 172.21.15.116:0/1581818836 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ef0003530 con 0x7f0ef0004810 2024-04-25T00:01:26.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f0f0a803640 1 -- 172.21.15.116:0/1581818836 reap_dead start 2024-04-25T00:01:26.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f6af6ffd640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f6af80c45b0 0x7f6af813a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f6af6ffd640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f6af80c45b0 0x7f6af813a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:37176/0 (socket says 172.21.15.116:37176) 2024-04-25T00:01:26.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f6af6ffd640 1 -- 172.21.15.116:0/1366276459 learned_addr learned my addr 172.21.15.116:0/1366276459 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:26.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f94d3fff640 1 --2- 172.21.15.116:0/2553293899 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f94d4152ee0 0x7f94d40c5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f6af67fc640 1 --2- 172.21.15.116:0/1366276459 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6af813a640 0x7f6af80c29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7fbf10b3c640 1 -- 172.21.15.116:0/3400997428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf040ded20 msgr2=0x7fbf040df1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7fbf10b3c640 1 --2- 172.21.15.116:0/3400997428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf040ded20 0x7fbf040df1d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c04fa30 tx=0x7fbf0c097fa0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f6af6ffd640 1 -- 172.21.15.116:0/1366276459 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6af813a640 msgr2=0x7f6af80c29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:26.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f6af6ffd640 1 --2- 172.21.15.116:0/1366276459 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6af813a640 0x7f6af80c29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f6af6ffd640 1 -- 172.21.15.116:0/1366276459 >> v1:172.21.15.174:6789/0 conn(0x7f6af80c2ee0 legacy=0x7f6af8071180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:26.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f6af6ffd640 1 -- 172.21.15.116:0/1366276459 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6af8155ac0 con 0x7f6af80c45b0 2024-04-25T00:01:26.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f9b737fe640 1 --2- 172.21.15.116:0/2669722268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b740a9ff0 0x7f9b740726a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9b740c8700 tx=0x7f9b6c01c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2584f13f5fe1a2fe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:26.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f9b71ffb640 1 -- 172.21.15.116:0/2669722268 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b6c0a56a0 con 0x7f9b740a9ff0 2024-04-25T00:01:26.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f9b71ffb640 1 -- 172.21.15.116:0/2669722268 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b6c0a5840 con 0x7f9b740a9ff0 2024-04-25T00:01:26.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f9b71ffb640 1 -- 172.21.15.116:0/2669722268 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b6c0a5c10 con 0x7f9b740a9ff0 2024-04-25T00:01:26.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f69a2ffd640 1 --2- 172.21.15.116:0/2600413532 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69a40c3f70 0x7f69a41557c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f69980097c0 tx=0x7f699801c420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad6d2abefe70d74b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:26.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f94d3fff640 1 -- 172.21.15.116:0/2553293899 >> v2:172.21.15.174:3300/0 conn(0x7f94d40be660 msgr2=0x7f94d4155f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:26.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f94d3fff640 1 --2- 172.21.15.116:0/2553293899 >> v2:172.21.15.174:3300/0 conn(0x7f94d40be660 0x7f94d4155f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f94d3fff640 1 -- 172.21.15.116:0/2553293899 >> v1:172.21.15.116:6789/0 conn(0x7f94d40c8ed0 legacy=0x7f94d4155860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f94d3fff640 1 -- 172.21.15.116:0/2553293899 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94d40ad010 con 0x7f94d4152ee0 2024-04-25T00:01:26.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7fbf10b3c640 1 -- 172.21.15.116:0/3400997428 shutdown_connections 2024-04-25T00:01:26.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7fbf10b3c640 1 --2- 172.21.15.116:0/3400997428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf040ded20 0x7fbf040df1d0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7fbf10b3c640 1 --2- 172.21.15.116:0/3400997428 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbf0400b0c0 0x7fbf040e2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7fbf10b3c640 1 -- 172.21.15.116:0/3400997428 >> 172.21.15.116:0/3400997428 conn(0x7fbf04048260 msgr2=0x7fbf04048670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:26.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f87d08e1640 1 --2- 172.21.15.116:0/2739205918 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f87cc070ff0 0x7f87cc071440 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f87cc070a60 tx=0x7f87c4020ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ad44aa4e391e9d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:26.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7fbf10b3c640 1 -- 172.21.15.116:0/3400997428 shutdown_connections 2024-04-25T00:01:26.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f69a27fc640 1 -- 172.21.15.116:0/2600413532 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69980ab600 con 0x7f69a40c3f70 2024-04-25T00:01:26.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f69a27fc640 1 -- 172.21.15.116:0/2600413532 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69980b0070 con 0x7f69a40c3f70 2024-04-25T00:01:26.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f69a27fc640 1 -- 172.21.15.116:0/2600413532 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69980aba00 con 0x7f69a40c3f70 2024-04-25T00:01:26.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f4481151640 1 --2- 172.21.15.116:0/3020099349 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f447c071130 0x7f447c0715c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4474009d90 tx=0x7f447401f140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa3eb05a4a5fc3d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:26.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f87caffd640 1 -- 172.21.15.116:0/2739205918 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c400e2d0 con 0x7f87cc070ff0 2024-04-25T00:01:26.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f87caffd640 1 -- 172.21.15.116:0/2739205918 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87c400e470 con 0x7f87cc070ff0 2024-04-25T00:01:26.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f87caffd640 1 -- 172.21.15.116:0/2739205918 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c40a9b50 con 0x7f87cc070ff0 2024-04-25T00:01:26.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f9b7976c640 1 -- 172.21.15.116:0/2669722268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b740a9ff0 msgr2=0x7f9b740726a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f9b7976c640 1 --2- 172.21.15.116:0/2669722268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b740a9ff0 0x7f9b740726a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9b740c8700 tx=0x7f9b6c01c8d0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f9b7976c640 1 -- 172.21.15.116:0/2669722268 shutdown_connections 2024-04-25T00:01:26.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f9b7976c640 1 --2- 172.21.15.116:0/2669722268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b740a9ff0 0x7f9b740726a0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f9b7976c640 1 --2- 172.21.15.116:0/2669722268 >> v2:172.21.15.174:3300/0 conn(0x7f9b7406b210 0x7f9b74072160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f9b7976c640 1 -- 172.21.15.116:0/2669722268 >> 172.21.15.116:0/2669722268 conn(0x7f9b740c4200 msgr2=0x7f9b740c4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:26.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f9b7976c640 1 -- 172.21.15.116:0/2669722268 shutdown_connections 2024-04-25T00:01:26.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f447b7fe640 1 -- 172.21.15.116:0/3020099349 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44740b5070 con 0x7f447c071130 2024-04-25T00:01:26.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f447b7fe640 1 -- 172.21.15.116:0/3020099349 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44740b0440 con 0x7f447c071130 2024-04-25T00:01:26.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f447b7fe640 1 -- 172.21.15.116:0/3020099349 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44740b0810 con 0x7f447c071130 2024-04-25T00:01:26.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f69a9d94640 1 -- 172.21.15.116:0/2600413532 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69a40c3f70 msgr2=0x7f69a41557c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f69a9d94640 1 --2- 172.21.15.116:0/2600413532 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69a40c3f70 0x7f69a41557c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f69980097c0 tx=0x7f699801c420 comp rx=0 tx=0).stop 2024-04-25T00:01:26.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f87d10e2640 1 -- 172.21.15.116:0/2739205918 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f87cc070ff0 msgr2=0x7f87cc071440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f87d10e2640 1 --2- 172.21.15.116:0/2739205918 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f87cc070ff0 0x7f87cc071440 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f87cc070a60 tx=0x7f87c4020ca0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7fbf10b3c640 1 -- 172.21.15.116:0/3400997428 wait complete. 2024-04-25T00:01:26.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f87d10e2640 1 -- 172.21.15.116:0/2739205918 shutdown_connections 2024-04-25T00:01:26.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f87d10e2640 1 --2- 172.21.15.116:0/2739205918 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f87cc070ff0 0x7f87cc071440 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f87d10e2640 1 --2- 172.21.15.116:0/2739205918 >> v2:172.21.15.174:3300/0 conn(0x7f87cc165ac0 0x7f87cc15fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f87d10e2640 1 --2- 172.21.15.116:0/2739205918 >> v2:172.21.15.116:3300/0 conn(0x7f87cc0c2ab0 0x7f87cc15f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f87d10e2640 1 -- 172.21.15.116:0/2739205918 >> 172.21.15.116:0/2739205918 conn(0x7f87cc0c3940 msgr2=0x7f87cc0c3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:26.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7fbf10b3c640 1 Processor -- start 2024-04-25T00:01:26.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.122+0000 7f9b7976c640 1 -- 172.21.15.116:0/2669722268 wait complete. 2024-04-25T00:01:26.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f69a9d94640 1 -- 172.21.15.116:0/2600413532 shutdown_connections 2024-04-25T00:01:26.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f69a9d94640 1 --2- 172.21.15.116:0/2600413532 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69a40c3f70 0x7f69a41557c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f69a9d94640 1 -- 172.21.15.116:0/2600413532 >> 172.21.15.116:0/2600413532 conn(0x7f69a40c9160 msgr2=0x7f69a40c9570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:26.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7fbf10b3c640 1 -- start start 2024-04-25T00:01:26.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f4481952640 1 -- 172.21.15.116:0/3020099349 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f447c071130 msgr2=0x7f447c0715c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f4481952640 1 --2- 172.21.15.116:0/3020099349 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f447c071130 0x7f447c0715c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4474009d90 tx=0x7f447401f140 comp rx=0 tx=0).stop 2024-04-25T00:01:26.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f0f09801640 1 --2- 172.21.15.116:0/1581818836 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0ef0004810 0x7f0ef0004c90 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0ef8001640 tx=0x7f0ef801cf40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9cc2ac936e97c97a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:26.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f4481952640 1 -- 172.21.15.116:0/3020099349 shutdown_connections 2024-04-25T00:01:26.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f4481952640 1 --2- 172.21.15.116:0/3020099349 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f447c071130 0x7f447c0715c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f4481952640 1 --2- 172.21.15.116:0/3020099349 >> v2:172.21.15.116:3300/0 conn(0x7f447c157240 0x7f447c159bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f4481952640 1 -- 172.21.15.116:0/3020099349 >> 172.21.15.116:0/3020099349 conn(0x7f447c0c3940 msgr2=0x7f447c0c3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:26.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f69a9d94640 1 -- 172.21.15.116:0/2600413532 shutdown_connections 2024-04-25T00:01:26.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f0f09000640 1 -- 172.21.15.116:0/1581818836 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef8008c80 con 0x7f0ef0004810 2024-04-25T00:01:26.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f87d10e2640 1 -- 172.21.15.116:0/2739205918 shutdown_connections 2024-04-25T00:01:26.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f4481952640 1 -- 172.21.15.116:0/3020099349 shutdown_connections 2024-04-25T00:01:26.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f9b7976c640 1 Processor -- start 2024-04-25T00:01:26.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f6af6ffd640 1 --2- 172.21.15.116:0/1366276459 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6af80c45b0 0x7f6af813a100 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6aec009870 tx=0x7f6aec01c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ffb599e8d84490c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:26.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f6af5ffb640 1 -- 172.21.15.116:0/1366276459 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aec0b0070 con 0x7f6af80c45b0 2024-04-25T00:01:26.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f6af5ffb640 1 -- 172.21.15.116:0/1366276459 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6aec0ab600 con 0x7f6af80c45b0 2024-04-25T00:01:26.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f0f09000640 1 -- 172.21.15.116:0/1581818836 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ef80a9070 con 0x7f0ef0004810 2024-04-25T00:01:26.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f0f09000640 1 -- 172.21.15.116:0/1581818836 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef80a59e0 con 0x7f0ef0004810 2024-04-25T00:01:26.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f6af5ffb640 1 -- 172.21.15.116:0/1366276459 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aec0ab9d0 con 0x7f6af80c45b0 2024-04-25T00:01:26.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f87d10e2640 1 -- 172.21.15.116:0/2739205918 wait complete. 2024-04-25T00:01:26.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f69a9d94640 1 -- 172.21.15.116:0/2600413532 wait complete. 2024-04-25T00:01:26.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7fbf10b3c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf0400b0c0 0x7fbf0402e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7fbf10b3c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbf040dbce0 0x7fbf040d4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7fbf10b3c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf040ded20 0x7fbf040d5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7fbf10b3c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbf040dfc50 con 0x7fbf0400b0c0 2024-04-25T00:01:26.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7fbf10b3c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbf04047ec0 con 0x7fbf040ded20 2024-04-25T00:01:26.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7fbf10b3c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fbf040dc0c0 con 0x7fbf040dbce0 2024-04-25T00:01:26.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f9b7976c640 1 -- start start 2024-04-25T00:01:26.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f9b7976c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b7406b210 0x7f9b7415c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f9b7976c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9b740a9ff0 0x7f9b741fc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f9b7976c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9b74157310 0x7f9b7415d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f9b7976c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9b740c3cb0 con 0x7f9b7406b210 2024-04-25T00:01:26.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f9b7976c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9b740c3b30 con 0x7f9b740a9ff0 2024-04-25T00:01:26.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f9b7976c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f9b741576f0 con 0x7f9b74157310 2024-04-25T00:01:26.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f4481952640 1 -- 172.21.15.116:0/3020099349 wait complete. 2024-04-25T00:01:26.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f69a9d94640 1 Processor -- start 2024-04-25T00:01:26.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f87d10e2640 1 Processor -- start 2024-04-25T00:01:26.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7fbf0b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf0400b0c0 0x7fbf0402e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7fbf0b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf0400b0c0 0x7fbf0402e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56090/0 (socket says 172.21.15.116:56090) 2024-04-25T00:01:26.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7fbf0b7fe640 1 -- 172.21.15.116:0/2317525846 learned_addr learned my addr 172.21.15.116:0/2317525846 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:26.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f9b727fc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9b740a9ff0 0x7f9b741fc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7fbf0affd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbf040dbce0 0x7fbf040d4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f0f0c266640 1 -- 172.21.15.116:0/1581818836 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0ef0004810 msgr2=0x7f0ef0004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f0f0c266640 1 --2- 172.21.15.116:0/1581818836 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0ef0004810 0x7f0ef0004c90 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0ef8001640 tx=0x7f0ef801cf40 comp rx=0 tx=0).stop 2024-04-25T00:01:26.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f6afd325640 1 -- 172.21.15.116:0/1366276459 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6af80c45b0 msgr2=0x7f6af813a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f6afd325640 1 --2- 172.21.15.116:0/1366276459 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6af80c45b0 0x7f6af813a100 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6aec009870 tx=0x7f6aec01c490 comp rx=0 tx=0).stop 2024-04-25T00:01:26.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f9b727fc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9b740a9ff0 0x7f9b741fc4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:37198/0 (socket says 172.21.15.116:37198) 2024-04-25T00:01:26.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f9b727fc640 1 -- 172.21.15.116:0/2612583262 learned_addr learned my addr 172.21.15.116:0/2612583262 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:26.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f9b737fe640 1 --2- 172.21.15.116:0/2612583262 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9b74157310 0x7f9b7415d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f94d3fff640 1 --2- 172.21.15.116:0/2553293899 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f94d4152ee0 0x7f94d40c5950 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f94c80097c0 tx=0x7f94c801c380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a51bc4956fa20a44 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:26.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f4481952640 1 Processor -- start 2024-04-25T00:01:26.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f6afd325640 1 -- 172.21.15.116:0/1366276459 shutdown_connections 2024-04-25T00:01:26.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f6afd325640 1 --2- 172.21.15.116:0/1366276459 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6af813a640 0x7f6af80c29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f6afd325640 1 --2- 172.21.15.116:0/1366276459 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6af80c45b0 0x7f6af813a100 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f6afd325640 1 -- 172.21.15.116:0/1366276459 >> 172.21.15.116:0/1366276459 conn(0x7f6af80c3a90 msgr2=0x7f6af80c3ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:26.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7fbf0bfff640 1 --2- 172.21.15.116:0/2317525846 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf040ded20 0x7fbf040d5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f94d2ffd640 1 -- 172.21.15.116:0/2553293899 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94c80b0070 con 0x7f94d4152ee0 2024-04-25T00:01:26.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f9b72ffd640 1 --2- 172.21.15.116:0/2612583262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b7406b210 0x7f9b7415c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f6afd325640 1 -- 172.21.15.116:0/1366276459 shutdown_connections 2024-04-25T00:01:26.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f0f0c266640 1 -- 172.21.15.116:0/1581818836 shutdown_connections 2024-04-25T00:01:26.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f0f0c266640 1 --2- 172.21.15.116:0/1581818836 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0ef0004810 0x7f0ef0004c90 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f0f0c266640 1 -- 172.21.15.116:0/1581818836 >> 172.21.15.116:0/1581818836 conn(0x7f0f040c3b00 msgr2=0x7f0f040c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:26.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f0f0c266640 1 -- 172.21.15.116:0/1581818836 shutdown_connections 2024-04-25T00:01:26.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f9b737fe640 1 -- 172.21.15.116:0/2612583262 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9b740a9ff0 msgr2=0x7f9b741fc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f9b737fe640 1 --2- 172.21.15.116:0/2612583262 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9b740a9ff0 0x7f9b741fc4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f9b737fe640 1 -- 172.21.15.116:0/2612583262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b7406b210 msgr2=0x7f9b7415c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f9b737fe640 1 --2- 172.21.15.116:0/2612583262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b7406b210 0x7f9b7415c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f9b737fe640 1 -- 172.21.15.116:0/2612583262 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b7415b5d0 con 0x7f9b74157310 2024-04-25T00:01:26.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f9b72ffd640 1 --2- 172.21.15.116:0/2612583262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b7406b210 0x7f9b7415c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:01:26.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f69a9d94640 1 -- start start 2024-04-25T00:01:26.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f69a9d94640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a40c3a20 0x7f69a406ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f69a9d94640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69a40c3f70 0x7f69a4159000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f69a9d94640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f69a4152e40 0x7f69a4159540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f69a9d94640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f69a4071940 con 0x7f69a40c3a20 2024-04-25T00:01:26.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f69a9d94640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f69a40717c0 con 0x7f69a40c3f70 2024-04-25T00:01:26.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f69a9d94640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f69a40c5120 con 0x7f69a4152e40 2024-04-25T00:01:26.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f69a37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a40c3a20 0x7f69a406ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f69a37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a40c3a20 0x7f69a406ffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56124/0 (socket says 172.21.15.116:56124) 2024-04-25T00:01:26.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f69a37fe640 1 -- 172.21.15.116:0/3526902096 learned_addr learned my addr 172.21.15.116:0/3526902096 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:26.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f87d10e2640 1 -- start start 2024-04-25T00:01:26.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f87d10e2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f87cc070ff0 0x7f87cc1603c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f87d10e2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87cc0c2ab0 0x7f87cc153700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f87d10e2640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f87cc165ac0 0x7f87cc1faf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f87d10e2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f87cc0c2e90 con 0x7f87cc0c2ab0 2024-04-25T00:01:26.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f87d10e2640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f87cc162780 con 0x7f87cc070ff0 2024-04-25T00:01:26.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f87d10e2640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f87cc0c3010 con 0x7f87cc165ac0 2024-04-25T00:01:26.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f87cb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87cc0c2ab0 0x7f87cc153700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f87cb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87cc0c2ab0 0x7f87cc153700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56120/0 (socket says 172.21.15.116:56120) 2024-04-25T00:01:26.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f87cb7fe640 1 -- 172.21.15.116:0/4213611597 learned_addr learned my addr 172.21.15.116:0/4213611597 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:26.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f4481952640 1 -- start start 2024-04-25T00:01:26.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f4481952640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f447c071130 0x7f447c159660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f4481952640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f447c157240 0x7f447c1f5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f9b737fe640 1 --2- 172.21.15.116:0/2612583262 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9b74157310 0x7f9b7415d410 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9b6c0ab000 tx=0x7f9b6c019670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:26.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f69a3fff640 1 --2- 172.21.15.116:0/3526902096 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f69a4152e40 0x7f69a4159540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f69a2ffd640 1 --2- 172.21.15.116:0/3526902096 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69a40c3f70 0x7f69a4159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f4481952640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f447c15d130 0x7f447c1f5c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f4481952640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f447c0c1470 con 0x7f447c157240 2024-04-25T00:01:26.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f4481952640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f447c15aa40 con 0x7f447c071130 2024-04-25T00:01:26.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f4481952640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f447c0c2f60 con 0x7f447c15d130 2024-04-25T00:01:26.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f87cb7fe640 1 -- 172.21.15.116:0/4213611597 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f87cc165ac0 msgr2=0x7f87cc1faf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f87cb7fe640 1 --2- 172.21.15.116:0/4213611597 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f87cc165ac0 0x7f87cc1faf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f87cb7fe640 1 -- 172.21.15.116:0/4213611597 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f87cc070ff0 msgr2=0x7f87cc1603c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T00:01:26.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f87cb7fe640 1 --2- 172.21.15.116:0/4213611597 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f87cc070ff0 0x7f87cc1603c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f87cb7fe640 1 -- 172.21.15.116:0/4213611597 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87cc1654a0 con 0x7f87cc0c2ab0 2024-04-25T00:01:26.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f94d2ffd640 1 -- 172.21.15.116:0/2553293899 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94c80ab600 con 0x7f94d4152ee0 2024-04-25T00:01:26.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f94d2ffd640 1 -- 172.21.15.116:0/2553293899 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94c80ab9d0 con 0x7f94d4152ee0 2024-04-25T00:01:26.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f94da64e640 1 -- 172.21.15.116:0/2553293899 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f94d4152ee0 msgr2=0x7f94d40c5950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f94da64e640 1 --2- 172.21.15.116:0/2553293899 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f94d4152ee0 0x7f94d40c5950 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f94c80097c0 tx=0x7f94c801c380 comp rx=0 tx=0).stop 2024-04-25T00:01:26.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f447bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f447c157240 0x7f447c1f5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f447bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f447c157240 0x7f447c1f5720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56136/0 (socket says 172.21.15.116:56136) 2024-04-25T00:01:26.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f447bfff640 1 -- 172.21.15.116:0/227852055 learned_addr learned my addr 172.21.15.116:0/227852055 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:26.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f4480950640 1 --2- 172.21.15.116:0/227852055 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f447c071130 0x7f447c159660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7fbf0b7fe640 1 -- 172.21.15.116:0/2317525846 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbf040dbce0 msgr2=0x7fbf040d4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7fbf0b7fe640 1 --2- 172.21.15.116:0/2317525846 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbf040dbce0 0x7fbf040d4fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7fbf0b7fe640 1 -- 172.21.15.116:0/2317525846 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf040ded20 msgr2=0x7fbf040d5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7fbf0b7fe640 1 --2- 172.21.15.116:0/2317525846 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf040ded20 0x7fbf040d5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7fbf0b7fe640 1 -- 172.21.15.116:0/2317525846 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf04183b70 con 0x7fbf0400b0c0 2024-04-25T00:01:26.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f6afd325640 1 -- 172.21.15.116:0/1366276459 wait complete. 2024-04-25T00:01:26.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f6afd325640 1 Processor -- start 2024-04-25T00:01:26.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f69a37fe640 1 -- 172.21.15.116:0/3526902096 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f69a4152e40 msgr2=0x7f69a4159540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f69a37fe640 1 --2- 172.21.15.116:0/3526902096 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f69a4152e40 0x7f69a4159540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f69a37fe640 1 -- 172.21.15.116:0/3526902096 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69a40c3f70 msgr2=0x7f69a4159000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f69a37fe640 1 --2- 172.21.15.116:0/3526902096 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69a40c3f70 0x7f69a4159000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f69a37fe640 1 -- 172.21.15.116:0/3526902096 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69a414e330 con 0x7f69a40c3a20 2024-04-25T00:01:26.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f69a2ffd640 1 --2- 172.21.15.116:0/3526902096 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69a40c3f70 0x7f69a4159000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:01:26.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f9b5b7fe640 1 -- 172.21.15.116:0/2612583262 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b6c0bc030 con 0x7f9b74157310 2024-04-25T00:01:26.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f9b5b7fe640 1 -- 172.21.15.116:0/2612583262 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b6c0b8470 con 0x7f9b74157310 2024-04-25T00:01:26.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f9b5b7fe640 1 -- 172.21.15.116:0/2612583262 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b6c009510 con 0x7f9b74157310 2024-04-25T00:01:26.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f9b7976c640 1 -- 172.21.15.116:0/2612583262 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b7415b8b0 con 0x7f9b74157310 2024-04-25T00:01:26.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f9b7976c640 1 -- 172.21.15.116:0/2612583262 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b7415be70 con 0x7f9b74157310 2024-04-25T00:01:26.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f4481151640 1 --2- 172.21.15.116:0/227852055 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f447c15d130 0x7f447c1f5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.126+0000 7f0f0c266640 1 -- 172.21.15.116:0/1581818836 wait complete. 2024-04-25T00:01:26.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f0f0c266640 1 Processor -- start 2024-04-25T00:01:26.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f87cb7fe640 1 --2- 172.21.15.116:0/4213611597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87cc0c2ab0 0x7f87cc153700 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f87c000d800 tx=0x7f87c000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:26.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f447bfff640 1 -- 172.21.15.116:0/227852055 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f447c15d130 msgr2=0x7f447c1f5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f447bfff640 1 --2- 172.21.15.116:0/227852055 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f447c15d130 0x7f447c1f5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f447bfff640 1 -- 172.21.15.116:0/227852055 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f447c071130 msgr2=0x7f447c159660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f447bfff640 1 --2- 172.21.15.116:0/227852055 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f447c071130 0x7f447c159660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f447bfff640 1 -- 172.21.15.116:0/227852055 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f447c155e10 con 0x7f447c157240 2024-04-25T00:01:26.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f6afd325640 1 -- start start 2024-04-25T00:01:26.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f6afd325640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6af80c2ee0 0x7f6af8154590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f6afd325640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6af80c45b0 0x7f6af80ac650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f6afd325640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6af813a640 0x7f6af80acb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f6afd325640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6af8071c20 con 0x7f6af813a640 2024-04-25T00:01:26.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f6afd325640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6af8071aa0 con 0x7f6af80c2ee0 2024-04-25T00:01:26.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f6afd325640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6af81588b0 con 0x7f6af80c45b0 2024-04-25T00:01:26.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7fbf0b7fe640 1 --2- 172.21.15.116:0/2317525846 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf0400b0c0 0x7fbf0402e7f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fbf0000c480 tx=0x7fbf0000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:26.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f69a37fe640 1 --2- 172.21.15.116:0/3526902096 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a40c3a20 0x7f69a406ffb0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f699c00cba0 tx=0x7f699c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:26.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7fbef7fff640 1 -- 172.21.15.116:0/2317525846 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf00012830 con 0x7fbf0400b0c0 2024-04-25T00:01:26.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7fbef7fff640 1 -- 172.21.15.116:0/2317525846 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf00017070 con 0x7fbf0400b0c0 2024-04-25T00:01:26.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7fbef7fff640 1 -- 172.21.15.116:0/2317525846 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf00012c30 con 0x7fbf0400b0c0 2024-04-25T00:01:26.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f6af67fc640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6af80c45b0 0x7f6af80ac650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f447bfff640 1 --2- 172.21.15.116:0/227852055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f447c157240 0x7f447c1f5720 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f446800b430 tx=0x7f446800b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:26.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f6af67fc640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6af80c45b0 0x7f6af80ac650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:56004/0 (socket says 172.21.15.116:56004) 2024-04-25T00:01:26.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f6af67fc640 1 -- 172.21.15.116:0/3122722302 learned_addr learned my addr 172.21.15.116:0/3122722302 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:26.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f698bfff640 1 -- 172.21.15.116:0/3526902096 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f699c017070 con 0x7f69a40c3a20 2024-04-25T00:01:26.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f6af6ffd640 1 --2- 172.21.15.116:0/3122722302 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6af80c2ee0 0x7f6af8154590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f698bfff640 1 -- 172.21.15.116:0/3526902096 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f699c00ad40 con 0x7f69a40c3a20 2024-04-25T00:01:26.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f69a9d94640 1 -- 172.21.15.116:0/3526902096 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69a414e610 con 0x7f69a40c3a20 2024-04-25T00:01:26.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f698bfff640 1 -- 172.21.15.116:0/3526902096 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f699c012600 con 0x7f69a40c3a20 2024-04-25T00:01:26.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f69a9d94640 1 -- 172.21.15.116:0/3526902096 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69a414ebd0 con 0x7f69a40c3a20 2024-04-25T00:01:26.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f44797fa640 1 -- 172.21.15.116:0/227852055 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4468015020 con 0x7f447c157240 2024-04-25T00:01:26.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f0f0c266640 1 -- start start 2024-04-25T00:01:26.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f0f0c266640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0ef0004810 0x7f0f040ac160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f0f0c266640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0f041ffc80 0x7f0f040ac9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f0f0c266640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f04140a80 0x7f0f04142da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f0f0c266640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0f040715f0 con 0x7f0f04140a80 2024-04-25T00:01:26.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f0f0c266640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0f04071470 con 0x7f0ef0004810 2024-04-25T00:01:26.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f0f0c266640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f0f04071770 con 0x7f0f041ffc80 2024-04-25T00:01:26.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7fbf10b3c640 1 -- 172.21.15.116:0/2317525846 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf04183e50 con 0x7fbf0400b0c0 2024-04-25T00:01:26.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f44797fa640 1 -- 172.21.15.116:0/227852055 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4468011830 con 0x7f447c157240 2024-04-25T00:01:26.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f4481952640 1 -- 172.21.15.116:0/227852055 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f447c1f61a0 con 0x7f447c157240 2024-04-25T00:01:26.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f44797fa640 1 -- 172.21.15.116:0/227852055 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4468011c60 con 0x7f447c157240 2024-04-25T00:01:26.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f4481952640 1 -- 172.21.15.116:0/227852055 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f447c14deb0 con 0x7f447c157240 2024-04-25T00:01:26.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f9b7976c640 1 -- 172.21.15.116:0/2612583262 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9b74068a60 con 0x7f9b74157310 2024-04-25T00:01:26.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f6af6ffd640 1 -- 172.21.15.116:0/3122722302 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6af80c45b0 msgr2=0x7f6af80ac650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f6af6ffd640 1 --2- 172.21.15.116:0/3122722302 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6af80c45b0 0x7f6af80ac650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f6af6ffd640 1 -- 172.21.15.116:0/3122722302 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6af813a640 msgr2=0x7f6af80acb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f6af6ffd640 1 --2- 172.21.15.116:0/3122722302 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6af813a640 0x7f6af80acb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f6af6ffd640 1 -- 172.21.15.116:0/3122722302 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6af80c5d70 con 0x7f6af80c2ee0 2024-04-25T00:01:26.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f6af6ffd640 1 --2- 172.21.15.116:0/3122722302 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6af80c2ee0 0x7f6af8154590 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6aec0ab000 tx=0x7f6aec01acd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:26.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7fbf10b3c640 1 -- 172.21.15.116:0/2317525846 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf040433b0 con 0x7fbf0400b0c0 2024-04-25T00:01:26.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f4481952640 1 -- 172.21.15.116:0/227852055 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f447c04f510 con 0x7f447c157240 2024-04-25T00:01:26.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f0f0a803640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f04140a80 0x7f0f04142da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f0f0a803640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f04140a80 0x7f0f04142da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56146/0 (socket says 172.21.15.116:56146) 2024-04-25T00:01:26.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f0f0a803640 1 -- 172.21.15.116:0/1018652658 learned_addr learned my addr 172.21.15.116:0/1018652658 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:26.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f0f0a002640 1 --2- 172.21.15.116:0/1018652658 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0ef0004810 0x7f0f040ac160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f0f09801640 1 --2- 172.21.15.116:0/1018652658 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0f041ffc80 0x7f0f040ac9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f6adf7fe640 1 -- 172.21.15.116:0/3122722302 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aec0ba020 con 0x7f6af80c2ee0 2024-04-25T00:01:26.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f6adf7fe640 1 -- 172.21.15.116:0/3122722302 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6aec0aa750 con 0x7f6af80c2ee0 2024-04-25T00:01:26.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f6adf7fe640 1 -- 172.21.15.116:0/3122722302 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aec0b0040 con 0x7f6af80c2ee0 2024-04-25T00:01:26.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f6afd325640 1 -- 172.21.15.116:0/3122722302 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6af80c6050 con 0x7f6af80c2ee0 2024-04-25T00:01:26.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f6afd325640 1 -- 172.21.15.116:0/3122722302 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6af8154ad0 con 0x7f6af80c2ee0 2024-04-25T00:01:26.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f0f0a002640 1 -- 172.21.15.116:0/1018652658 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0f041ffc80 msgr2=0x7f0f040ac9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f0f0a002640 1 --2- 172.21.15.116:0/1018652658 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0f041ffc80 0x7f0f040ac9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f0f0a002640 1 -- 172.21.15.116:0/1018652658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f04140a80 msgr2=0x7f0f04142da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f0f0a002640 1 --2- 172.21.15.116:0/1018652658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f04140a80 0x7f0f04142da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7f0f0a002640 1 -- 172.21.15.116:0/1018652658 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f040ab3a0 con 0x7f0ef0004810 2024-04-25T00:01:26.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f87c8ff9640 1 -- 172.21.15.116:0/4213611597 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c000b6c0 con 0x7f87cc0c2ab0 2024-04-25T00:01:26.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.130+0000 7fbf10b3c640 1 -- 172.21.15.116:0/2317525846 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbf040053a0 con 0x7fbf0400b0c0 2024-04-25T00:01:26.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f87c8ff9640 1 -- 172.21.15.116:0/4213611597 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87c0019070 con 0x7f87cc0c2ab0 2024-04-25T00:01:26.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f0f0a002640 1 --2- 172.21.15.116:0/1018652658 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0ef0004810 0x7f0f040ac160 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0f040c7f90 tx=0x7f0f0000fc00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:26.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f6afd325640 1 -- 172.21.15.116:0/3122722302 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6af804f580 con 0x7f6af80c2ee0 2024-04-25T00:01:26.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f94da64e640 1 -- 172.21.15.116:0/2553293899 shutdown_connections 2024-04-25T00:01:26.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f94da64e640 1 --2- 172.21.15.116:0/2553293899 >> v2:172.21.15.174:3300/0 conn(0x7f94d40be660 0x7f94d4155f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f94da64e640 1 --2- 172.21.15.116:0/2553293899 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f94d4152ee0 0x7f94d40c5950 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f94da64e640 1 -- 172.21.15.116:0/2553293899 >> 172.21.15.116:0/2553293899 conn(0x7f94d4074650 msgr2=0x7f94d4074a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:26.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f44797fa640 1 -- 172.21.15.116:0/227852055 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4468018020 con 0x7f447c157240 2024-04-25T00:01:26.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f94da64e640 1 -- 172.21.15.116:0/2553293899 shutdown_connections 2024-04-25T00:01:26.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f94da64e640 1 -- 172.21.15.116:0/2553293899 wait complete. 2024-04-25T00:01:26.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f0ef67fc640 1 -- 172.21.15.116:0/1018652658 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f0000fe50 con 0x7f0ef0004810 2024-04-25T00:01:26.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f0f0c266640 1 -- 172.21.15.116:0/1018652658 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f040ab680 con 0x7f0ef0004810 2024-04-25T00:01:26.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f0f0c266640 1 -- 172.21.15.116:0/1018652658 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f040abc40 con 0x7f0ef0004810 2024-04-25T00:01:26.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f0ef67fc640 1 -- 172.21.15.116:0/1018652658 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f00018070 con 0x7f0ef0004810 2024-04-25T00:01:26.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f0ef67fc640 1 -- 172.21.15.116:0/1018652658 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f00003600 con 0x7f0ef0004810 2024-04-25T00:01:26.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f87c8ff9640 1 -- 172.21.15.116:0/4213611597 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c000ba60 con 0x7f87cc0c2ab0 2024-04-25T00:01:26.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f69a9d94640 1 -- 172.21.15.116:0/3526902096 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f69a4068a60 con 0x7f69a40c3a20 2024-04-25T00:01:26.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f0f0c266640 1 -- 172.21.15.116:0/1018652658 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0eec001040 con 0x7f0ef0004810 2024-04-25T00:01:26.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f9b5b7fe640 1 -- 172.21.15.116:0/2612583262 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f9b6c0bf070 con 0x7f9b74157310 2024-04-25T00:01:26.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f9b5b7fe640 1 --2- 172.21.15.116:0/2612583262 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9b600679c0 0x7f9b60069e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f9b5b7fe640 1 -- 172.21.15.116:0/2612583262 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7f9b6c120e30 con 0x7f9b74157310 2024-04-25T00:01:26.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f9b5b7fe640 1 --2- 172.21.15.116:0/2612583262 >> [v2:172.21.15.174:6816/2543899196,v1:172.21.15.174:6817/2543899196] conn(0x7f9b60072260 0x7f9b60074680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f9b5b7fe640 1 -- 172.21.15.116:0/2612583262 --> [v2:172.21.15.174:6816/2543899196,v1:172.21.15.174:6817/2543899196] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9b6c0b5e50 con 0x7f9b60072260 2024-04-25T00:01:26.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f9b5b7fe640 1 -- 172.21.15.116:0/2612583262 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9b6c121220 con 0x7f9b74157310 2024-04-25T00:01:26.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f87d10e2640 1 -- 172.21.15.116:0/4213611597 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87cc1ff090 con 0x7f87cc0c2ab0 2024-04-25T00:01:26.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f9b72ffd640 1 --2- 172.21.15.116:0/2612583262 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9b600679c0 0x7f9b60069e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f9b727fc640 1 --2- 172.21.15.116:0/2612583262 >> [v2:172.21.15.174:6816/2543899196,v1:172.21.15.174:6817/2543899196] conn(0x7f9b60072260 0x7f9b60074680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f94da64e640 1 Processor -- start 2024-04-25T00:01:26.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f44797fa640 1 --2- 172.21.15.116:0/227852055 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4460067910 0x7f4460069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f44797fa640 1 -- 172.21.15.116:0/227852055 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7f4468085db0 con 0x7f447c157240 2024-04-25T00:01:26.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f44797fa640 1 --2- 172.21.15.116:0/227852055 >> [v2:172.21.15.116:6824/1179942661,v1:172.21.15.116:6825/1179942661] conn(0x7f4460072120 0x7f4460074540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f44797fa640 1 -- 172.21.15.116:0/227852055 --> [v2:172.21.15.116:6824/1179942661,v1:172.21.15.116:6825/1179942661] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4468007e50 con 0x7f4460072120 2024-04-25T00:01:26.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f4480950640 1 --2- 172.21.15.116:0/227852055 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4460067910 0x7f4460069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f94da64e640 1 -- start start 2024-04-25T00:01:26.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f4481151640 1 --2- 172.21.15.116:0/227852055 >> [v2:172.21.15.116:6824/1179942661,v1:172.21.15.116:6825/1179942661] conn(0x7f4460072120 0x7f4460074540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f87d10e2640 1 -- 172.21.15.116:0/4213611597 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87cc153c40 con 0x7f87cc0c2ab0 2024-04-25T00:01:26.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f87d10e2640 1 -- 172.21.15.116:0/4213611597 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f87cc165250 con 0x7f87cc0c2ab0 2024-04-25T00:01:26.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f6adf7fe640 1 -- 172.21.15.116:0/3122722302 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6aec0bd020 con 0x7f6af80c2ee0 2024-04-25T00:01:26.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f6adf7fe640 1 --2- 172.21.15.116:0/3122722302 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6ae40678f0 0x7f6ae4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f6adf7fe640 1 -- 172.21.15.116:0/3122722302 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7f6aec1208e0 con 0x7f6af80c2ee0 2024-04-25T00:01:26.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f6adf7fe640 1 --2- 172.21.15.116:0/3122722302 >> [v2:172.21.15.174:6824/117574251,v1:172.21.15.174:6825/117574251] conn(0x7f6ae4072190 0x7f6ae40745b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f6adf7fe640 1 -- 172.21.15.116:0/3122722302 --> [v2:172.21.15.174:6824/117574251,v1:172.21.15.174:6825/117574251] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6aec0c5e50 con 0x7f6ae4072190 2024-04-25T00:01:26.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f6adf7fe640 1 -- 172.21.15.116:0/3122722302 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6aec120cd0 con 0x7f6af80c2ee0 2024-04-25T00:01:26.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f6af67fc640 1 --2- 172.21.15.116:0/3122722302 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6ae40678f0 0x7f6ae4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f6af77fe640 1 --2- 172.21.15.116:0/3122722302 >> [v2:172.21.15.174:6824/117574251,v1:172.21.15.174:6825/117574251] conn(0x7f6ae4072190 0x7f6ae40745b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f4481151640 1 --2- 172.21.15.116:0/227852055 >> [v2:172.21.15.116:6824/1179942661,v1:172.21.15.116:6825/1179942661] conn(0x7f4460072120 0x7f4460074540 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:26.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f94da64e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f94d40be660 0x7f94d4151380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f94da64e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94d40c8ed0 0x7f94d40c0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f94da64e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f94d4152ee0 0x7f94d40c13d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f94da64e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f94d4162890 con 0x7f94d40c8ed0 2024-04-25T00:01:26.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f94da64e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f94d406d8a0 con 0x7f94d40be660 2024-04-25T00:01:26.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f94da64e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f94d4162a10 con 0x7f94d4152ee0 2024-04-25T00:01:26.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f698bfff640 1 -- 172.21.15.116:0/3526902096 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f699c01a070 con 0x7f69a40c3a20 2024-04-25T00:01:26.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f94d8beb640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f94d4152ee0 0x7f94d40c13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f0ef67fc640 1 -- 172.21.15.116:0/1018652658 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f0f000037a0 con 0x7f0ef0004810 2024-04-25T00:01:26.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f0ef67fc640 1 --2- 172.21.15.116:0/1018652658 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0ee8067990 0x7f0ee8069e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f0ef67fc640 1 -- 172.21.15.116:0/1018652658 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7f0f000866e0 con 0x7f0ef0004810 2024-04-25T00:01:26.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f0ef67fc640 1 --2- 172.21.15.116:0/1018652658 >> [v2:172.21.15.174:6808/881817164,v1:172.21.15.174:6809/881817164] conn(0x7f0ee8072230 0x7f0ee8074650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f0ef67fc640 1 -- 172.21.15.116:0/1018652658 --> [v2:172.21.15.174:6808/881817164,v1:172.21.15.174:6809/881817164] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0f00028e50 con 0x7f0ee8072230 2024-04-25T00:01:26.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f0ef67fc640 1 -- 172.21.15.116:0/1018652658 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0f00086ad0 con 0x7f0ef0004810 2024-04-25T00:01:26.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f94d8beb640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f94d4152ee0 0x7f94d40c13d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:56022/0 (socket says 172.21.15.116:56022) 2024-04-25T00:01:26.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.134+0000 7f94d8beb640 1 -- 172.21.15.116:0/3596976693 learned_addr learned my addr 172.21.15.116:0/3596976693 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:26.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.138+0000 7f6af67fc640 1 --2- 172.21.15.116:0/3122722302 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6ae40678f0 0x7f6ae4069db0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6af81f95a0 tx=0x7f6af00098b0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:26.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.138+0000 7f698bfff640 1 --2- 172.21.15.116:0/3526902096 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6990067990 0x7f6990069e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.138+0000 7f94d37fe640 1 --2- 172.21.15.116:0/3596976693 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94d40c8ed0 0x7f94d40c0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.138+0000 7f0f09801640 1 --2- 172.21.15.116:0/1018652658 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0ee8067990 0x7f0ee8069e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.138+0000 7f6af77fe640 1 --2- 172.21.15.116:0/3122722302 >> [v2:172.21.15.174:6824/117574251,v1:172.21.15.174:6825/117574251] conn(0x7f6ae4072190 0x7f6ae40745b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:26.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.138+0000 7f698bfff640 1 -- 172.21.15.116:0/3526902096 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7f699c085cd0 con 0x7f69a40c3a20 2024-04-25T00:01:26.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.138+0000 7f9b727fc640 1 --2- 172.21.15.116:0/2612583262 >> [v2:172.21.15.174:6816/2543899196,v1:172.21.15.174:6817/2543899196] conn(0x7f9b60072260 0x7f9b60074680 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:26.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.138+0000 7f698bfff640 1 --2- 172.21.15.116:0/3526902096 >> [v2:172.21.15.116:6808/3751765682,v1:172.21.15.116:6809/3751765682] conn(0x7f69900721a0 0x7f69900745c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.138+0000 7f698bfff640 1 -- 172.21.15.116:0/3526902096 --> [v2:172.21.15.116:6808/3751765682,v1:172.21.15.116:6809/3751765682] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f699c013e50 con 0x7f69900721a0 2024-04-25T00:01:26.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.138+0000 7f0f0a803640 1 --2- 172.21.15.116:0/1018652658 >> [v2:172.21.15.174:6808/881817164,v1:172.21.15.174:6809/881817164] conn(0x7f0ee8072230 0x7f0ee8074650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.138+0000 7f9b72ffd640 1 --2- 172.21.15.116:0/2612583262 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9b600679c0 0x7f9b60069e80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9b68001bb0 tx=0x7f9b68009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:26.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.138+0000 7f94d8beb640 1 -- 172.21.15.116:0/3596976693 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f94d40be660 msgr2=0x7f94d4151380 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T00:01:26.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.138+0000 7f94d8beb640 1 --2- 172.21.15.116:0/3596976693 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f94d40be660 0x7f94d4151380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.138+0000 7f94d8beb640 1 -- 172.21.15.116:0/3596976693 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94d40c8ed0 msgr2=0x7f94d40c0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.138+0000 7f94d8beb640 1 --2- 172.21.15.116:0/3596976693 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94d40c8ed0 0x7f94d40c0e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.138+0000 7f94d8beb640 1 -- 172.21.15.116:0/3596976693 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94d41faf30 con 0x7f94d4152ee0 2024-04-25T00:01:26.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.138+0000 7f69a2ffd640 1 --2- 172.21.15.116:0/3526902096 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6990067990 0x7f6990069e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.138+0000 7f94d8beb640 1 --2- 172.21.15.116:0/3596976693 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f94d4152ee0 0x7f94d40c13d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f94cc00efc0 tx=0x7f94cc00c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:26.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.138+0000 7fbef7fff640 1 -- 172.21.15.116:0/2317525846 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbf0001c460 con 0x7fbf0400b0c0 2024-04-25T00:01:26.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.138+0000 7f0f0a803640 1 --2- 172.21.15.116:0/1018652658 >> [v2:172.21.15.174:6808/881817164,v1:172.21.15.174:6809/881817164] conn(0x7f0ee8072230 0x7f0ee8074650 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:26.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.138+0000 7f4480950640 1 --2- 172.21.15.116:0/227852055 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4460067910 0x7f4460069dd0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4470009b50 tx=0x7f4470009240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:26.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.138+0000 7f44797fa640 1 -- 172.21.15.116:0/227852055 <== osd.3 v2:172.21.15.116:6824/1179942661 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4468007e50 con 0x7f4460072120 2024-04-25T00:01:26.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.138+0000 7f44797fa640 1 -- 172.21.15.116:0/227852055 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4468052280 con 0x7f447c157240 2024-04-25T00:01:26.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.138+0000 7f69a3fff640 1 --2- 172.21.15.116:0/3526902096 >> [v2:172.21.15.116:6808/3751765682,v1:172.21.15.116:6809/3751765682] conn(0x7f69900721a0 0x7f69900745c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.138+0000 7f698bfff640 1 -- 172.21.15.116:0/3526902096 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f699c055860 con 0x7f69a40c3a20 2024-04-25T00:01:26.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.138+0000 7f69a3fff640 1 --2- 172.21.15.116:0/3526902096 >> [v2:172.21.15.116:6808/3751765682,v1:172.21.15.116:6809/3751765682] conn(0x7f69900721a0 0x7f69900745c0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:26.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.138+0000 7f698bfff640 1 -- 172.21.15.116:0/3526902096 <== osd.0 v2:172.21.15.116:6808/3751765682 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f699c013e50 con 0x7f69900721a0 2024-04-25T00:01:26.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.138+0000 7f69a2ffd640 1 --2- 172.21.15.116:0/3526902096 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6990067990 0x7f6990069e50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f69980af000 tx=0x7f69980a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:26.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.138+0000 7f94d0ff9640 1 -- 172.21.15.116:0/3596976693 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94cc009e60 con 0x7f94d4152ee0 2024-04-25T00:01:26.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.138+0000 7f94d0ff9640 1 -- 172.21.15.116:0/3596976693 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94cc00cd20 con 0x7f94d4152ee0 2024-04-25T00:01:26.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.138+0000 7f94da64e640 1 -- 172.21.15.116:0/3596976693 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94d41fb210 con 0x7f94d4152ee0 2024-04-25T00:01:26.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.138+0000 7f94da64e640 1 -- 172.21.15.116:0/3596976693 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f94d41fb7d0 con 0x7f94d4152ee0 2024-04-25T00:01:26.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.138+0000 7f94d0ff9640 1 -- 172.21.15.116:0/3596976693 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94cc003be0 con 0x7f94d4152ee0 2024-04-25T00:01:26.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.138+0000 7fbef7fff640 1 --2- 172.21.15.116:0/2317525846 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbef0067940 0x7fbef0069e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.138+0000 7fbef7fff640 1 -- 172.21.15.116:0/2317525846 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7fbf00086620 con 0x7fbf0400b0c0 2024-04-25T00:01:26.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.138+0000 7fbef7fff640 1 --2- 172.21.15.116:0/2317525846 >> [v2:172.21.15.174:6800/3132444330,v1:172.21.15.174:6801/3132444330] conn(0x7fbef00721d0 0x7fbef00745f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.138+0000 7fbef7fff640 1 -- 172.21.15.116:0/2317525846 --> [v2:172.21.15.174:6800/3132444330,v1:172.21.15.174:6801/3132444330] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbef0074ca0 con 0x7fbef00721d0 2024-04-25T00:01:26.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.138+0000 7fbef7fff640 1 -- 172.21.15.116:0/2317525846 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fbf00086b00 con 0x7fbf0400b0c0 2024-04-25T00:01:26.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.138+0000 7fbf0bfff640 1 --2- 172.21.15.116:0/2317525846 >> [v2:172.21.15.174:6800/3132444330,v1:172.21.15.174:6801/3132444330] conn(0x7fbef00721d0 0x7fbef00745f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.138+0000 7fbf0affd640 1 --2- 172.21.15.116:0/2317525846 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbef0067940 0x7fbef0069e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.138+0000 7f0f09801640 1 --2- 172.21.15.116:0/1018652658 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0ee8067990 0x7f0ee8069e50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0ef80ab000 tx=0x7f0ef80a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:26.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.138+0000 7fbf0bfff640 1 --2- 172.21.15.116:0/2317525846 >> [v2:172.21.15.174:6800/3132444330,v1:172.21.15.174:6801/3132444330] conn(0x7fbef00721d0 0x7fbef00745f0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:26.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.138+0000 7f0ef67fc640 1 -- 172.21.15.116:0/1018652658 <== osd.5 v2:172.21.15.174:6808/881817164 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0f00028e50 con 0x7f0ee8072230 2024-04-25T00:01:26.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.142+0000 7f94da64e640 1 -- 172.21.15.116:0/3596976693 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f94d404f580 con 0x7f94d4152ee0 2024-04-25T00:01:26.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.142+0000 7f9b5b7fe640 1 -- 172.21.15.116:0/2612583262 <== osd.7 v2:172.21.15.174:6816/2543899196 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9b6c0b5e50 con 0x7f9b60072260 2024-04-25T00:01:26.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.142+0000 7f94d0ff9640 1 -- 172.21.15.116:0/3596976693 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f94cc012050 con 0x7f94d4152ee0 2024-04-25T00:01:26.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.142+0000 7f94d0ff9640 1 --2- 172.21.15.116:0/3596976693 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f94c00678f0 0x7f94c0069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.142+0000 7f94d0ff9640 1 -- 172.21.15.116:0/3596976693 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7f94cc086410 con 0x7f94d4152ee0 2024-04-25T00:01:26.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.142+0000 7f94d0ff9640 1 --2- 172.21.15.116:0/3596976693 >> [v2:172.21.15.116:6816/1099282267,v1:172.21.15.116:6817/1099282267] conn(0x7f94c0072080 0x7f94c00744a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.142+0000 7f94d0ff9640 1 -- 172.21.15.116:0/3596976693 --> [v2:172.21.15.116:6816/1099282267,v1:172.21.15.116:6817/1099282267] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f94cc01ae70 con 0x7f94c0072080 2024-04-25T00:01:26.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.142+0000 7f94d0ff9640 1 -- 172.21.15.116:0/3596976693 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f94cc08d050 con 0x7f94d4152ee0 2024-04-25T00:01:26.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.142+0000 7f94d3fff640 1 --2- 172.21.15.116:0/3596976693 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f94c00678f0 0x7f94c0069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.142+0000 7f94d37fe640 1 --2- 172.21.15.116:0/3596976693 >> [v2:172.21.15.116:6816/1099282267,v1:172.21.15.116:6817/1099282267] conn(0x7f94c0072080 0x7f94c00744a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.142+0000 7f94d37fe640 1 --2- 172.21.15.116:0/3596976693 >> [v2:172.21.15.116:6816/1099282267,v1:172.21.15.116:6817/1099282267] conn(0x7f94c0072080 0x7f94c00744a0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:26.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.142+0000 7f94d3fff640 1 --2- 172.21.15.116:0/3596976693 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f94c00678f0 0x7f94c0069db0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f94c80ad000 tx=0x7f94c8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:26.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.142+0000 7f94d0ff9640 1 -- 172.21.15.116:0/3596976693 <== osd.2 v2:172.21.15.116:6816/1099282267 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f94cc01ae70 con 0x7f94c0072080 2024-04-25T00:01:26.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.142+0000 7f6adf7fe640 1 -- 172.21.15.116:0/3122722302 <== osd.6 v2:172.21.15.174:6824/117574251 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6aec0c5e50 con 0x7f6ae4072190 2024-04-25T00:01:26.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.142+0000 7f87c8ff9640 1 -- 172.21.15.116:0/4213611597 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f87c000bc00 con 0x7f87cc0c2ab0 2024-04-25T00:01:26.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.146+0000 7f87c8ff9640 1 --2- 172.21.15.116:0/4213611597 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f87b8067990 0x7f87b8069e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.146+0000 7f87c8ff9640 1 -- 172.21.15.116:0/4213611597 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7f87c0086e20 con 0x7f87cc0c2ab0 2024-04-25T00:01:26.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.146+0000 7f87c8ff9640 1 --2- 172.21.15.116:0/4213611597 >> [v2:172.21.15.116:6800/377879124,v1:172.21.15.116:6801/377879124] conn(0x7f87b8072230 0x7f87b8074650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.146+0000 7f87c8ff9640 1 -- 172.21.15.116:0/4213611597 --> [v2:172.21.15.116:6800/377879124,v1:172.21.15.116:6801/377879124] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f87c0019070 con 0x7f87b8072230 2024-04-25T00:01:26.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.146+0000 7f87c8ff9640 1 -- 172.21.15.116:0/4213611597 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f87c0087210 con 0x7f87cc0c2ab0 2024-04-25T00:01:26.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.146+0000 7f87cbfff640 1 --2- 172.21.15.116:0/4213611597 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f87b8067990 0x7f87b8069e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.146+0000 7f87d08e1640 1 --2- 172.21.15.116:0/4213611597 >> [v2:172.21.15.116:6800/377879124,v1:172.21.15.116:6801/377879124] conn(0x7f87b8072230 0x7f87b8074650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.146+0000 7f87d08e1640 1 --2- 172.21.15.116:0/4213611597 >> [v2:172.21.15.116:6800/377879124,v1:172.21.15.116:6801/377879124] conn(0x7f87b8072230 0x7f87b8074650 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:26.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.146+0000 7f87cbfff640 1 --2- 172.21.15.116:0/4213611597 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f87b8067990 0x7f87b8069e50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f87bc00b310 tx=0x7f87bc00e040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:26.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.146+0000 7f87c8ff9640 1 -- 172.21.15.116:0/4213611597 <== osd.1 v2:172.21.15.116:6800/377879124 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f87c0019070 con 0x7f87b8072230 2024-04-25T00:01:26.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.146+0000 7fbef7fff640 1 -- 172.21.15.116:0/2317525846 <== osd.4 v2:172.21.15.174:6800/3132444330 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbef0074ca0 con 0x7fbef00721d0 2024-04-25T00:01:26.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.158+0000 7fbf0affd640 1 --2- 172.21.15.116:0/2317525846 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbef0067940 0x7fbef0069e00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fbf04044af0 tx=0x7fbef8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:26.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.166+0000 7f0f0c266640 1 -- 172.21.15.116:0/1018652658 --> [v2:172.21.15.174:6808/881817164,v1:172.21.15.174:6809/881817164] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0eec002de0 con 0x7f0ee8072230 2024-04-25T00:01:26.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.166+0000 7f0ef67fc640 1 -- 172.21.15.116:0/1018652658 <== osd.5 v2:172.21.15.174:6808/881817164 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f0eec002de0 con 0x7f0ee8072230 2024-04-25T00:01:26.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.166+0000 7f0ee7fff640 1 -- 172.21.15.116:0/1018652658 >> [v2:172.21.15.174:6808/881817164,v1:172.21.15.174:6809/881817164] conn(0x7f0ee8072230 msgr2=0x7f0ee8074650 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.166+0000 7f0ee7fff640 1 --2- 172.21.15.116:0/1018652658 >> [v2:172.21.15.174:6808/881817164,v1:172.21.15.174:6809/881817164] conn(0x7f0ee8072230 0x7f0ee8074650 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.166+0000 7f0ee7fff640 1 -- 172.21.15.116:0/1018652658 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0ee8067990 msgr2=0x7f0ee8069e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.166+0000 7f0ee7fff640 1 --2- 172.21.15.116:0/1018652658 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0ee8067990 0x7f0ee8069e50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0ef80ab000 tx=0x7f0ef80a7070 comp rx=0 tx=0).stop 2024-04-25T00:01:26.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.166+0000 7f0ee7fff640 1 -- 172.21.15.116:0/1018652658 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0ef0004810 msgr2=0x7f0f040ac160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.166+0000 7f0ee7fff640 1 --2- 172.21.15.116:0/1018652658 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0ef0004810 0x7f0f040ac160 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0f040c7f90 tx=0x7f0f0000fc00 comp rx=0 tx=0).stop 2024-04-25T00:01:26.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.166+0000 7f0f0a803640 1 -- 172.21.15.116:0/1018652658 reap_dead start 2024-04-25T00:01:26.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.166+0000 7f0ee7fff640 1 -- 172.21.15.116:0/1018652658 shutdown_connections 2024-04-25T00:01:26.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.166+0000 7f69a9d94640 1 -- 172.21.15.116:0/3526902096 --> [v2:172.21.15.116:6808/3751765682,v1:172.21.15.116:6809/3751765682] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f69a406c940 con 0x7f69900721a0 2024-04-25T00:01:26.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.166+0000 7f0ee7fff640 1 -- 172.21.15.116:0/1018652658 >> 172.21.15.116:0/1018652658 conn(0x7f0f040c3b00 msgr2=0x7f0f0406e7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:26.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.166+0000 7f0ee7fff640 1 -- 172.21.15.116:0/1018652658 shutdown_connections 2024-04-25T00:01:26.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.166+0000 7f0ee7fff640 1 -- 172.21.15.116:0/1018652658 wait complete. 2024-04-25T00:01:26.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.166+0000 7f698bfff640 1 -- 172.21.15.116:0/3526902096 <== osd.0 v2:172.21.15.116:6808/3751765682 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f69a406c940 con 0x7f69900721a0 2024-04-25T00:01:26.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.170+0000 7f6989ffb640 1 -- 172.21.15.116:0/3526902096 >> [v2:172.21.15.116:6808/3751765682,v1:172.21.15.116:6809/3751765682] conn(0x7f69900721a0 msgr2=0x7f69900745c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.170+0000 7f6989ffb640 1 --2- 172.21.15.116:0/3526902096 >> [v2:172.21.15.116:6808/3751765682,v1:172.21.15.116:6809/3751765682] conn(0x7f69900721a0 0x7f69900745c0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.170+0000 7f6989ffb640 1 -- 172.21.15.116:0/3526902096 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6990067990 msgr2=0x7f6990069e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.170+0000 7f6989ffb640 1 --2- 172.21.15.116:0/3526902096 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6990067990 0x7f6990069e50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f69980af000 tx=0x7f69980a7000 comp rx=0 tx=0).stop 2024-04-25T00:01:26.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.170+0000 7f6989ffb640 1 -- 172.21.15.116:0/3526902096 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a40c3a20 msgr2=0x7f69a406ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.170+0000 7f6989ffb640 1 --2- 172.21.15.116:0/3526902096 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69a40c3a20 0x7f69a406ffb0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f699c00cba0 tx=0x7f699c00a430 comp rx=0 tx=0).stop 2024-04-25T00:01:26.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.170+0000 7f69a3fff640 1 -- 172.21.15.116:0/3526902096 reap_dead start 2024-04-25T00:01:26.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.170+0000 7f6989ffb640 1 -- 172.21.15.116:0/3526902096 shutdown_connections 2024-04-25T00:01:26.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.170+0000 7f6989ffb640 1 -- 172.21.15.116:0/3526902096 >> 172.21.15.116:0/3526902096 conn(0x7f69a40c9160 msgr2=0x7f69a406d180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:26.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.170+0000 7f6989ffb640 1 -- 172.21.15.116:0/3526902096 shutdown_connections 2024-04-25T00:01:26.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.170+0000 7f6989ffb640 1 -- 172.21.15.116:0/3526902096 wait complete. 2024-04-25T00:01:26.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.170+0000 7f9b7976c640 1 -- 172.21.15.116:0/2612583262 --> [v2:172.21.15.174:6816/2543899196,v1:172.21.15.174:6817/2543899196] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9b74157a20 con 0x7f9b60072260 2024-04-25T00:01:26.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.170+0000 7fbf10b3c640 1 -- 172.21.15.116:0/2317525846 --> [v2:172.21.15.174:6800/3132444330,v1:172.21.15.174:6801/3132444330] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fbf0403ef90 con 0x7fbef00721d0 2024-04-25T00:01:26.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.170+0000 7f9b5b7fe640 1 -- 172.21.15.116:0/2612583262 <== osd.7 v2:172.21.15.174:6816/2543899196 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f9b74157a20 con 0x7f9b60072260 2024-04-25T00:01:26.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.170+0000 7f94da64e640 1 -- 172.21.15.116:0/3596976693 --> [v2:172.21.15.116:6816/1099282267,v1:172.21.15.116:6817/1099282267] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f94d4074f50 con 0x7f94c0072080 2024-04-25T00:01:26.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.174+0000 7f9b7976c640 1 -- 172.21.15.116:0/2612583262 >> [v2:172.21.15.174:6816/2543899196,v1:172.21.15.174:6817/2543899196] conn(0x7f9b60072260 msgr2=0x7f9b60074680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.174+0000 7f9b7976c640 1 --2- 172.21.15.116:0/2612583262 >> [v2:172.21.15.174:6816/2543899196,v1:172.21.15.174:6817/2543899196] conn(0x7f9b60072260 0x7f9b60074680 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.174+0000 7f9b7976c640 1 -- 172.21.15.116:0/2612583262 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9b600679c0 msgr2=0x7f9b60069e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.174+0000 7f9b7976c640 1 --2- 172.21.15.116:0/2612583262 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9b600679c0 0x7f9b60069e80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9b68001bb0 tx=0x7f9b68009040 comp rx=0 tx=0).stop 2024-04-25T00:01:26.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.174+0000 7f9b7976c640 1 -- 172.21.15.116:0/2612583262 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9b74157310 msgr2=0x7f9b7415d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.174+0000 7f9b7976c640 1 --2- 172.21.15.116:0/2612583262 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9b74157310 0x7f9b7415d410 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9b6c0ab000 tx=0x7f9b6c019670 comp rx=0 tx=0).stop 2024-04-25T00:01:26.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.174+0000 7f9b737fe640 1 -- 172.21.15.116:0/2612583262 reap_dead start 2024-04-25T00:01:26.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.174+0000 7f94d0ff9640 1 -- 172.21.15.116:0/3596976693 <== osd.2 v2:172.21.15.116:6816/1099282267 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f94d4074f50 con 0x7f94c0072080 2024-04-25T00:01:26.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.174+0000 7f9b7976c640 1 -- 172.21.15.116:0/2612583262 shutdown_connections 2024-04-25T00:01:26.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.174+0000 7f9b7976c640 1 -- 172.21.15.116:0/2612583262 >> 172.21.15.116:0/2612583262 conn(0x7f9b740c4200 msgr2=0x7f9b741554f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:26.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.174+0000 7f9b7976c640 1 -- 172.21.15.116:0/2612583262 shutdown_connections 2024-04-25T00:01:26.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.174+0000 7f9b7976c640 1 -- 172.21.15.116:0/2612583262 wait complete. 2024-04-25T00:01:26.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.174+0000 7f6afd325640 1 -- 172.21.15.116:0/3122722302 --> [v2:172.21.15.174:6824/117574251,v1:172.21.15.174:6825/117574251] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6af8137b60 con 0x7f6ae4072190 2024-04-25T00:01:26.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.174+0000 7f94da64e640 1 -- 172.21.15.116:0/3596976693 >> [v2:172.21.15.116:6816/1099282267,v1:172.21.15.116:6817/1099282267] conn(0x7f94c0072080 msgr2=0x7f94c00744a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.174+0000 7f94da64e640 1 --2- 172.21.15.116:0/3596976693 >> [v2:172.21.15.116:6816/1099282267,v1:172.21.15.116:6817/1099282267] conn(0x7f94c0072080 0x7f94c00744a0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.174+0000 7f94da64e640 1 -- 172.21.15.116:0/3596976693 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f94c00678f0 msgr2=0x7f94c0069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.174+0000 7f94da64e640 1 --2- 172.21.15.116:0/3596976693 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f94c00678f0 0x7f94c0069db0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f94c80ad000 tx=0x7f94c8008040 comp rx=0 tx=0).stop 2024-04-25T00:01:26.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.174+0000 7f94da64e640 1 -- 172.21.15.116:0/3596976693 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f94d4152ee0 msgr2=0x7f94d40c13d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.174+0000 7f94da64e640 1 --2- 172.21.15.116:0/3596976693 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f94d4152ee0 0x7f94d40c13d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f94cc00efc0 tx=0x7f94cc00c490 comp rx=0 tx=0).stop 2024-04-25T00:01:26.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.174+0000 7f94d8beb640 1 -- 172.21.15.116:0/3596976693 reap_dead start 2024-04-25T00:01:26.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.174+0000 7f94da64e640 1 -- 172.21.15.116:0/3596976693 shutdown_connections 2024-04-25T00:01:26.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.174+0000 7f94da64e640 1 -- 172.21.15.116:0/3596976693 >> 172.21.15.116:0/3596976693 conn(0x7f94d4074650 msgr2=0x7f94d41533c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:26.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.174+0000 7fbef7fff640 1 -- 172.21.15.116:0/2317525846 <== osd.4 v2:172.21.15.174:6800/3132444330 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fbf0403ef90 con 0x7fbef00721d0 2024-04-25T00:01:26.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.174+0000 7f94da64e640 1 -- 172.21.15.116:0/3596976693 shutdown_connections 2024-04-25T00:01:26.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.174+0000 7f94da64e640 1 -- 172.21.15.116:0/3596976693 wait complete. 2024-04-25T00:01:26.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.174+0000 7f6adf7fe640 1 -- 172.21.15.116:0/3122722302 <== osd.6 v2:172.21.15.174:6824/117574251 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f6ae80050b0 con 0x7f6ae4072190 2024-04-25T00:01:26.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.174+0000 7f4481952640 1 -- 172.21.15.116:0/227852055 --> [v2:172.21.15.116:6824/1179942661,v1:172.21.15.116:6825/1179942661] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f447c150930 con 0x7f4460072120 2024-04-25T00:01:26.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.182+0000 7f87d10e2640 1 -- 172.21.15.116:0/4213611597 --> [v2:172.21.15.116:6800/377879124,v1:172.21.15.116:6801/377879124] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f87cc04f510 con 0x7f87b8072230 2024-04-25T00:01:26.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.182+0000 7fbf10b3c640 1 -- 172.21.15.116:0/2317525846 >> [v2:172.21.15.174:6800/3132444330,v1:172.21.15.174:6801/3132444330] conn(0x7fbef00721d0 msgr2=0x7fbef00745f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.182+0000 7fbf10b3c640 1 --2- 172.21.15.116:0/2317525846 >> [v2:172.21.15.174:6800/3132444330,v1:172.21.15.174:6801/3132444330] conn(0x7fbef00721d0 0x7fbef00745f0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.182+0000 7f87c8ff9640 1 -- 172.21.15.116:0/4213611597 <== osd.1 v2:172.21.15.116:6800/377879124 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f87cc04f510 con 0x7f87b8072230 2024-04-25T00:01:26.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.182+0000 7fbf10b3c640 1 -- 172.21.15.116:0/2317525846 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbef0067940 msgr2=0x7fbef0069e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.182+0000 7fbf10b3c640 1 --2- 172.21.15.116:0/2317525846 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbef0067940 0x7fbef0069e00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fbf04044af0 tx=0x7fbef8008040 comp rx=0 tx=0).stop 2024-04-25T00:01:26.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.182+0000 7f6afd325640 1 -- 172.21.15.116:0/3122722302 >> [v2:172.21.15.174:6824/117574251,v1:172.21.15.174:6825/117574251] conn(0x7f6ae4072190 msgr2=0x7f6ae40745b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.182+0000 7f6afd325640 1 --2- 172.21.15.116:0/3122722302 >> [v2:172.21.15.174:6824/117574251,v1:172.21.15.174:6825/117574251] conn(0x7f6ae4072190 0x7f6ae40745b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.182+0000 7fbf10b3c640 1 -- 172.21.15.116:0/2317525846 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf0400b0c0 msgr2=0x7fbf0402e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.182+0000 7fbf10b3c640 1 --2- 172.21.15.116:0/2317525846 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf0400b0c0 0x7fbf0402e7f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fbf0000c480 tx=0x7fbf0000c940 comp rx=0 tx=0).stop 2024-04-25T00:01:26.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.182+0000 7fbf0bfff640 1 -- 172.21.15.116:0/2317525846 reap_dead start 2024-04-25T00:01:26.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.182+0000 7f6afd325640 1 -- 172.21.15.116:0/3122722302 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6ae40678f0 msgr2=0x7f6ae4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.182+0000 7fbf10b3c640 1 -- 172.21.15.116:0/2317525846 shutdown_connections 2024-04-25T00:01:26.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.182+0000 7fbf10b3c640 1 -- 172.21.15.116:0/2317525846 >> 172.21.15.116:0/2317525846 conn(0x7fbf04048260 msgr2=0x7fbf040ddaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:26.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.182+0000 7f6afd325640 1 --2- 172.21.15.116:0/3122722302 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6ae40678f0 0x7f6ae4069db0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6af81f95a0 tx=0x7f6af00098b0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.182+0000 7f6afd325640 1 -- 172.21.15.116:0/3122722302 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6af80c2ee0 msgr2=0x7f6af8154590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.182+0000 7f6afd325640 1 --2- 172.21.15.116:0/3122722302 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6af80c2ee0 0x7f6af8154590 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6aec0ab000 tx=0x7f6aec01acd0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.182+0000 7fbf10b3c640 1 -- 172.21.15.116:0/2317525846 shutdown_connections 2024-04-25T00:01:26.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.182+0000 7fbf10b3c640 1 -- 172.21.15.116:0/2317525846 wait complete. 2024-04-25T00:01:26.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.182+0000 7f87d10e2640 1 -- 172.21.15.116:0/4213611597 >> [v2:172.21.15.116:6800/377879124,v1:172.21.15.116:6801/377879124] conn(0x7f87b8072230 msgr2=0x7f87b8074650 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.182+0000 7f87d10e2640 1 --2- 172.21.15.116:0/4213611597 >> [v2:172.21.15.116:6800/377879124,v1:172.21.15.116:6801/377879124] conn(0x7f87b8072230 0x7f87b8074650 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.182+0000 7f44797fa640 1 -- 172.21.15.116:0/227852055 <== osd.3 v2:172.21.15.116:6824/1179942661 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f447c150930 con 0x7f4460072120 2024-04-25T00:01:26.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.182+0000 7f87d10e2640 1 -- 172.21.15.116:0/4213611597 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f87b8067990 msgr2=0x7f87b8069e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.182+0000 7f87d10e2640 1 --2- 172.21.15.116:0/4213611597 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f87b8067990 0x7f87b8069e50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f87bc00b310 tx=0x7f87bc00e040 comp rx=0 tx=0).stop 2024-04-25T00:01:26.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.182+0000 7f87d10e2640 1 -- 172.21.15.116:0/4213611597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87cc0c2ab0 msgr2=0x7f87cc153700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.182+0000 7f87d10e2640 1 --2- 172.21.15.116:0/4213611597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87cc0c2ab0 0x7f87cc153700 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f87c000d800 tx=0x7f87c000dcc0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.182+0000 7f6af77fe640 1 -- 172.21.15.116:0/3122722302 reap_dead start 2024-04-25T00:01:26.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.182+0000 7f6afd325640 1 -- 172.21.15.116:0/3122722302 shutdown_connections 2024-04-25T00:01:26.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.182+0000 7f6afd325640 1 -- 172.21.15.116:0/3122722302 >> 172.21.15.116:0/3122722302 conn(0x7f6af80c3a90 msgr2=0x7f6af8071180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:26.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.182+0000 7f87d08e1640 1 -- 172.21.15.116:0/4213611597 reap_dead start 2024-04-25T00:01:26.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.182+0000 7f87d10e2640 1 -- 172.21.15.116:0/4213611597 shutdown_connections 2024-04-25T00:01:26.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.182+0000 7f87d10e2640 1 -- 172.21.15.116:0/4213611597 >> 172.21.15.116:0/4213611597 conn(0x7f87cc0c3940 msgr2=0x7f87cc072b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:26.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.182+0000 7f87d10e2640 1 -- 172.21.15.116:0/4213611597 shutdown_connections 2024-04-25T00:01:26.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.182+0000 7f87d10e2640 1 -- 172.21.15.116:0/4213611597 wait complete. 2024-04-25T00:01:26.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.182+0000 7f6afd325640 1 -- 172.21.15.116:0/3122722302 shutdown_connections 2024-04-25T00:01:26.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.182+0000 7f6afd325640 1 -- 172.21.15.116:0/3122722302 wait complete. 2024-04-25T00:01:26.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.182+0000 7f4481952640 1 -- 172.21.15.116:0/227852055 >> [v2:172.21.15.116:6824/1179942661,v1:172.21.15.116:6825/1179942661] conn(0x7f4460072120 msgr2=0x7f4460074540 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.182+0000 7f4481952640 1 --2- 172.21.15.116:0/227852055 >> [v2:172.21.15.116:6824/1179942661,v1:172.21.15.116:6825/1179942661] conn(0x7f4460072120 0x7f4460074540 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.182+0000 7f4481952640 1 -- 172.21.15.116:0/227852055 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4460067910 msgr2=0x7f4460069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.182+0000 7f4481952640 1 --2- 172.21.15.116:0/227852055 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4460067910 0x7f4460069dd0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4470009b50 tx=0x7f4470009240 comp rx=0 tx=0).stop 2024-04-25T00:01:26.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.182+0000 7f4481952640 1 -- 172.21.15.116:0/227852055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f447c157240 msgr2=0x7f447c1f5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.182+0000 7f4481952640 1 --2- 172.21.15.116:0/227852055 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f447c157240 0x7f447c1f5720 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f446800b430 tx=0x7f446800b8f0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.182+0000 7f4481151640 1 -- 172.21.15.116:0/227852055 reap_dead start 2024-04-25T00:01:26.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.182+0000 7f4481952640 1 -- 172.21.15.116:0/227852055 shutdown_connections 2024-04-25T00:01:26.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.182+0000 7f4481952640 1 -- 172.21.15.116:0/227852055 >> 172.21.15.116:0/227852055 conn(0x7f447c0c3940 msgr2=0x7f447c0c2580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:26.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.182+0000 7f4481952640 1 -- 172.21.15.116:0/227852055 shutdown_connections 2024-04-25T00:01:26.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.182+0000 7f4481952640 1 -- 172.21.15.116:0/227852055 wait complete. 2024-04-25T00:01:26.232 INFO:teuthology.orchestra.run.smithi116.stdout:64424509449 2024-04-25T00:01:26.232 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-25T00:01:26.233 INFO:teuthology.orchestra.run.smithi116.stdout:64424509451 2024-04-25T00:01:26.233 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-25T00:01:26.234 INFO:teuthology.orchestra.run.smithi116.stdout:64424509450 2024-04-25T00:01:26.234 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-25T00:01:26.235 INFO:teuthology.orchestra.run.smithi116.stdout:64424509450 2024-04-25T00:01:26.235 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-25T00:01:26.236 INFO:teuthology.orchestra.run.smithi116.stdout:64424509450 2024-04-25T00:01:26.236 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-25T00:01:26.237 INFO:teuthology.orchestra.run.smithi116.stdout:64424509450 2024-04-25T00:01:26.237 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-25T00:01:26.238 INFO:teuthology.orchestra.run.smithi116.stdout:55834574859 2024-04-25T00:01:26.239 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-25T00:01:26.239 INFO:teuthology.orchestra.run.smithi116.stdout:64424509450 2024-04-25T00:01:26.240 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-25T00:01:26.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.422+0000 7fc1eed36640 1 Processor -- start 2024-04-25T00:01:26.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.422+0000 7fc1eed36640 1 -- start start 2024-04-25T00:01:26.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.426+0000 7fc1eed36640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc1e813acb0 0x7fc1e8071180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.426+0000 7fc1eed36640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1e806cb70 0x7fc1e80716c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.426+0000 7fc1eed36640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1e8073320 con 0x7fc1e813a360 2024-04-25T00:01:26.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.426+0000 7fc1eed36640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fc1e8070cf0 con 0x7fc1e813acb0 2024-04-25T00:01:26.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.426+0000 7fc1eed36640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fc1e8073040 con 0x7fc1e806cb70 2024-04-25T00:01:26.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.426+0000 7fc1ecad2640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fc1e813a360 0x7fc1e80c29a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:41652/0 (socket says 172.21.15.116:41652) 2024-04-25T00:01:26.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.426+0000 7fc1ecad2640 1 -- 172.21.15.116:0/1794088815 learned_addr learned my addr 172.21.15.116:0/1794088815 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:26.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.426+0000 7fc1e7fff640 1 --2- 172.21.15.116:0/1794088815 >> v2:172.21.15.116:3300/0 conn(0x7fc1e813acb0 0x7fc1e8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.426+0000 7fc1e7fff640 1 -- 172.21.15.116:0/1794088815 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1e806cb70 msgr2=0x7fc1e80716c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T00:01:26.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.426+0000 7fc1e7fff640 1 --2- 172.21.15.116:0/1794088815 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1e806cb70 0x7fc1e80716c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.426+0000 7fc1e7fff640 1 -- 172.21.15.116:0/1794088815 >> v1:172.21.15.116:6789/0 conn(0x7fc1e813a360 legacy=0x7fc1e80c29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.426+0000 7fc1e7fff640 1 -- 172.21.15.116:0/1794088815 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1e81557e0 con 0x7fc1e813acb0 2024-04-25T00:01:26.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.430+0000 7fc1e7fff640 1 --2- 172.21.15.116:0/1794088815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1e813acb0 0x7fc1e8071180 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc1d80097c0 tx=0x7fc1d801e1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a31f5c413348a24d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:26.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.438+0000 7fc1e77fe640 1 -- 172.21.15.116:0/1794088815 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1d80ad070 con 0x7fc1e813acb0 2024-04-25T00:01:26.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.438+0000 7fc1e77fe640 1 -- 172.21.15.116:0/1794088815 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1d80a7640 con 0x7fc1e813acb0 2024-04-25T00:01:26.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.438+0000 7fc1e77fe640 1 -- 172.21.15.116:0/1794088815 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1d80a7a10 con 0x7fc1e813acb0 2024-04-25T00:01:26.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.438+0000 7fc1eed36640 1 -- 172.21.15.116:0/1794088815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1e813acb0 msgr2=0x7fc1e8071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.438+0000 7fc1eed36640 1 --2- 172.21.15.116:0/1794088815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1e813acb0 0x7fc1e8071180 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc1d80097c0 tx=0x7fc1d801e1c0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.438+0000 7fc1eed36640 1 -- 172.21.15.116:0/1794088815 shutdown_connections 2024-04-25T00:01:26.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.438+0000 7fc1eed36640 1 --2- 172.21.15.116:0/1794088815 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1e806cb70 0x7fc1e80716c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.438+0000 7fc1eed36640 1 --2- 172.21.15.116:0/1794088815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1e813acb0 0x7fc1e8071180 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.438+0000 7fc1eed36640 1 -- 172.21.15.116:0/1794088815 >> 172.21.15.116:0/1794088815 conn(0x7fc1e80c3a90 msgr2=0x7fc1e80c3ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:26.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.438+0000 7fc1eed36640 1 -- 172.21.15.116:0/1794088815 shutdown_connections 2024-04-25T00:01:26.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.438+0000 7fc1eed36640 1 -- 172.21.15.116:0/1794088815 wait complete. 2024-04-25T00:01:26.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.438+0000 7fc1eed36640 1 Processor -- start 2024-04-25T00:01:26.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.442+0000 7fc1eed36640 1 -- start start 2024-04-25T00:01:26.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.442+0000 7fc1eed36640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1e806cb70 0x7fc1e8151710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.442+0000 7fc1eed36640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1e813a360 0x7fc1e814fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.442+0000 7fc1eed36640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1e813acb0 0x7fc1e81f73d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.442+0000 7fc1eed36640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc1e8163690 con 0x7fc1e813acb0 2024-04-25T00:01:26.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.442+0000 7fc1eed36640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc1e8163510 con 0x7fc1e813a360 2024-04-25T00:01:26.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.442+0000 7fc1eed36640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fc1e8163810 con 0x7fc1e806cb70 2024-04-25T00:01:26.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.442+0000 7fc1ecad2640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1e806cb70 0x7fc1e8151710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.442+0000 7fc1ecad2640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1e806cb70 0x7fc1e8151710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:56030/0 (socket says 172.21.15.116:56030) 2024-04-25T00:01:26.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.442+0000 7fc1ecad2640 1 -- 172.21.15.116:0/3898419650 learned_addr learned my addr 172.21.15.116:0/3898419650 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:26.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.442+0000 7fc1ed2d3640 1 --2- 172.21.15.116:0/3898419650 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1e813acb0 0x7fc1e81f73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.442+0000 7fc1ecad2640 1 -- 172.21.15.116:0/3898419650 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1e813a360 msgr2=0x7fc1e814fb10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T00:01:26.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.442+0000 7fc1ecad2640 1 --2- 172.21.15.116:0/3898419650 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1e813a360 0x7fc1e814fb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.442+0000 7fc1ecad2640 1 -- 172.21.15.116:0/3898419650 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1e813acb0 msgr2=0x7fc1e81f73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.442+0000 7fc1ecad2640 1 --2- 172.21.15.116:0/3898419650 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1e813acb0 0x7fc1e81f73d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.442+0000 7fc1ecad2640 1 -- 172.21.15.116:0/3898419650 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1e80ab3f0 con 0x7fc1e806cb70 2024-04-25T00:01:26.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.442+0000 7fc1ecad2640 1 --2- 172.21.15.116:0/3898419650 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1e806cb70 0x7fc1e8151710 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc1e000cb20 tx=0x7fc1e00021a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:26.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.442+0000 7fc1e57fa640 1 -- 172.21.15.116:0/3898419650 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1e0018070 con 0x7fc1e806cb70 2024-04-25T00:01:26.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.442+0000 7fc1e57fa640 1 -- 172.21.15.116:0/3898419650 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1e000a7c0 con 0x7fc1e806cb70 2024-04-25T00:01:26.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.442+0000 7fc1e57fa640 1 -- 172.21.15.116:0/3898419650 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1e000ab70 con 0x7fc1e806cb70 2024-04-25T00:01:26.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.442+0000 7fc1eed36640 1 -- 172.21.15.116:0/3898419650 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1e8154710 con 0x7fc1e806cb70 2024-04-25T00:01:26.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.442+0000 7fc1eed36640 1 -- 172.21.15.116:0/3898419650 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1e8154cd0 con 0x7fc1e806cb70 2024-04-25T00:01:26.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.446+0000 7fc1eed36640 1 -- 172.21.15.116:0/3898419650 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1c80052a0 con 0x7fc1e806cb70 2024-04-25T00:01:26.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.446+0000 7f2e2ed10640 1 Processor -- start 2024-04-25T00:01:26.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.446+0000 7f2e2ed10640 1 -- start start 2024-04-25T00:01:26.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.446+0000 7f2e2ed10640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2e200db6e0 0x7f2e200dbaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.446+0000 7f2e2ed10640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e20049fe0 con 0x7f2e200d5780 2024-04-25T00:01:26.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.446+0000 7f2e2ed10640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e2004a5d0 con 0x7f2e2000aa20 2024-04-25T00:01:26.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.446+0000 7f2e2ed10640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f2e20030750 con 0x7f2e200db6e0 2024-04-25T00:01:26.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.446+0000 7f2e2dd0e640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f2e200d5780 0x7f2e200496e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:41668/0 (socket says 172.21.15.116:41668) 2024-04-25T00:01:26.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.446+0000 7f2e2dd0e640 1 -- 172.21.15.116:0/2870979904 learned_addr learned my addr 172.21.15.116:0/2870979904 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:26.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.446+0000 7f2e2e50f640 1 --2- 172.21.15.116:0/2870979904 >> v2:172.21.15.116:3300/0 conn(0x7f2e200db6e0 0x7f2e200dbaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.446+0000 7f3235981640 1 Processor -- start 2024-04-25T00:01:26.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.446+0000 7f3235981640 1 -- start start 2024-04-25T00:01:26.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.446+0000 7fc1e57fa640 1 -- 172.21.15.116:0/3898419650 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc1e000ad10 con 0x7fc1e806cb70 2024-04-25T00:01:26.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.446+0000 7fc1e57fa640 1 --2- 172.21.15.116:0/3898419650 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc1d40678f0 0x7fc1d4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.446+0000 7fc1e57fa640 1 -- 172.21.15.116:0/3898419650 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7fc1e0085a30 con 0x7fc1e806cb70 2024-04-25T00:01:26.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.446+0000 7f454173d640 1 Processor -- start 2024-04-25T00:01:26.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.446+0000 7f454173d640 1 -- start start 2024-04-25T00:01:26.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.446+0000 7fc1e7fff640 1 --2- 172.21.15.116:0/3898419650 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc1d40678f0 0x7fc1d4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.446+0000 7f3235981640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f3230152d80 0x7f3230159950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.446+0000 7f2e2e50f640 1 -- 172.21.15.116:0/2870979904 >> v1:172.21.15.174:6789/0 conn(0x7f2e2000aa20 legacy=0x7f2e20006eb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T00:01:26.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.446+0000 7f2e2e50f640 1 -- 172.21.15.116:0/2870979904 >> v1:172.21.15.116:6789/0 conn(0x7f2e200d5780 legacy=0x7f2e200496e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.446+0000 7f2e2e50f640 1 -- 172.21.15.116:0/2870979904 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e2004a8b0 con 0x7f2e200db6e0 2024-04-25T00:01:26.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.446+0000 7f30e1182640 1 Processor -- start 2024-04-25T00:01:26.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.446+0000 7f3235981640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f32301562a0 0x7f32300c5020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.446+0000 7f322f7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f32301562a0 0x7f32300c5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.446+0000 7f322f7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f32301562a0 0x7f32300c5020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56192/0 (socket says 172.21.15.116:56192) 2024-04-25T00:01:26.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.446+0000 7f322f7fe640 1 -- 172.21.15.116:0/1750336278 learned_addr learned my addr 172.21.15.116:0/1750336278 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:26.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.446+0000 7fc1e7fff640 1 --2- 172.21.15.116:0/3898419650 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc1d40678f0 0x7fc1d4069db0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc1d80af000 tx=0x7fc1d80a8000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:26.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.446+0000 7f30e1182640 1 -- start start 2024-04-25T00:01:26.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.446+0000 7f3235981640 1 -- 172.21.15.116:0/1750336278 --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32300730b0 con 0x7f3230072080 2024-04-25T00:01:26.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.446+0000 7f3235981640 1 -- 172.21.15.116:0/1750336278 --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f3230072e60 con 0x7f32301562a0 2024-04-25T00:01:26.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.446+0000 7f3235981640 1 -- 172.21.15.116:0/1750336278 --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f3230073390 con 0x7f3230152d80 2024-04-25T00:01:26.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.446+0000 7f322effd640 1 --2- 172.21.15.116:0/1750336278 >> v2:172.21.15.174:3300/0 conn(0x7f3230152d80 0x7f3230159950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7f322f7fe640 1 -- 172.21.15.116:0/1750336278 >> v2:172.21.15.174:3300/0 conn(0x7f3230152d80 msgr2=0x7f3230159950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:26.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7f322f7fe640 1 --2- 172.21.15.116:0/1750336278 >> v2:172.21.15.174:3300/0 conn(0x7f3230152d80 0x7f3230159950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7f322f7fe640 1 -- 172.21.15.116:0/1750336278 >> v1:172.21.15.174:6789/0 conn(0x7f3230072080 legacy=0x7f32300c36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7f322f7fe640 1 -- 172.21.15.116:0/1750336278 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3230158e70 con 0x7f32301562a0 2024-04-25T00:01:26.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7fc43559b640 1 Processor -- start 2024-04-25T00:01:26.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7f454173d640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f453400f8d0 0x7f453404dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7f454173d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f453400ea20 0x7f4534007220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7f454173d640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f453400e290 con 0x7f45340d7a50 2024-04-25T00:01:26.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7f454173d640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f45340dab00 con 0x7f453400f8d0 2024-04-25T00:01:26.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7f454173d640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f453400e590 con 0x7f453400ea20 2024-04-25T00:01:26.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7f453b7fe640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f453400f8d0 0x7f453404dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7f453b7fe640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f453400f8d0 0x7f453404dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:37280/0 (socket says 172.21.15.116:37280) 2024-04-25T00:01:26.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7f453b7fe640 1 -- 172.21.15.116:0/1321191695 learned_addr learned my addr 172.21.15.116:0/1321191695 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:26.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7f4540f3c640 1 --2- 172.21.15.116:0/1321191695 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f453400ea20 0x7f4534007220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7fc43559b640 1 -- start start 2024-04-25T00:01:26.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7f453b7fe640 1 -- 172.21.15.116:0/1321191695 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f453400ea20 msgr2=0x7f4534007220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:26.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7f453b7fe640 1 --2- 172.21.15.116:0/1321191695 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f453400ea20 0x7f4534007220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7f453b7fe640 1 -- 172.21.15.116:0/1321191695 >> v1:172.21.15.174:6789/0 conn(0x7f45340d7a50 legacy=0x7f45340da3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7f453b7fe640 1 -- 172.21.15.116:0/1321191695 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45340d7e30 con 0x7f453400f8d0 2024-04-25T00:01:26.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7ffb74daa640 1 Processor -- start 2024-04-25T00:01:26.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7ffb74daa640 1 -- start start 2024-04-25T00:01:26.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7f2e2e50f640 1 --2- 172.21.15.116:0/2870979904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e200db6e0 0x7f2e200dbaf0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2e2802d710 tx=0x7f2e28097f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89da922a31fc1716 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:26.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7fc43559b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc43006b210 0x7fc43006b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7fc43559b640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fc43006bb30 0x7fc43006e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7fc43559b640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc430159a90 con 0x7fc43015d080 2024-04-25T00:01:26.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7fc43559b640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fc4300c4720 con 0x7fc43006b210 2024-04-25T00:01:26.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7fc43559b640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fc430159d50 con 0x7fc43006bb30 2024-04-25T00:01:26.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7ffb74daa640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffb68048380 0x7ffb680d82e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7ffb74daa640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ffb6800be80 0x7ffb68048760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7ffb74daa640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb6800c380 con 0x7ffb680d25c0 2024-04-25T00:01:26.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7ffb74daa640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7ffb6800cf90 con 0x7ffb6800be80 2024-04-25T00:01:26.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7ffb74daa640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ffb6800c660 con 0x7ffb68048380 2024-04-25T00:01:26.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7f2e2cd0c640 1 -- 172.21.15.116:0/2870979904 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e28120650 con 0x7f2e200db6e0 2024-04-25T00:01:26.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7fc42effd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc43006b210 0x7fc43006b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7fc42effd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc43006b210 0x7fc43006b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56200/0 (socket says 172.21.15.116:56200) 2024-04-25T00:01:26.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7fc42e7fc640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fc43015d080 0x7fc43006dee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:35580/0 (socket says 172.21.15.116:35580) 2024-04-25T00:01:26.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7fc42e7fc640 1 -- 172.21.15.116:0/2395906088 learned_addr learned my addr 172.21.15.116:0/2395906088 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:26.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7fc42f7fe640 1 --2- 172.21.15.116:0/2395906088 >> v2:172.21.15.174:3300/0 conn(0x7fc43006bb30 0x7fc43006e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7f2e2cd0c640 1 -- 172.21.15.116:0/2870979904 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e281207f0 con 0x7f2e200db6e0 2024-04-25T00:01:26.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7f2e2cd0c640 1 -- 172.21.15.116:0/2870979904 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e28120c60 con 0x7f2e200db6e0 2024-04-25T00:01:26.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7ffb6e59c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffb68048380 0x7ffb680d82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7fc42f7fe640 1 -- 172.21.15.116:0/2395906088 >> v2:172.21.15.116:3300/0 conn(0x7fc43006b210 msgr2=0x7fc43006b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:26.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7fc42f7fe640 1 --2- 172.21.15.116:0/2395906088 >> v2:172.21.15.116:3300/0 conn(0x7fc43006b210 0x7fc43006b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7fc42f7fe640 1 -- 172.21.15.116:0/2395906088 >> v1:172.21.15.174:6789/0 conn(0x7fc43015d080 legacy=0x7fc43006dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7fc42f7fe640 1 -- 172.21.15.116:0/2395906088 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc43015a030 con 0x7fc43006bb30 2024-04-25T00:01:26.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7ffb6e59c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffb68048380 0x7ffb680d82e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:56040/0 (socket says 172.21.15.116:56040) 2024-04-25T00:01:26.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7ffb6e59c640 1 -- 172.21.15.116:0/1361792891 learned_addr learned my addr 172.21.15.116:0/1361792891 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:26.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7ffb6e59c640 1 -- 172.21.15.116:0/1361792891 >> v2:172.21.15.174:3300/0 conn(0x7ffb6800be80 msgr2=0x7ffb68048760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:26.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7ffb6e59c640 1 --2- 172.21.15.116:0/1361792891 >> v2:172.21.15.174:3300/0 conn(0x7ffb6800be80 0x7ffb68048760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7ffb6e59c640 1 -- 172.21.15.116:0/1361792891 >> v1:172.21.15.174:6789/0 conn(0x7ffb680d25c0 legacy=0x7ffb6804bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7ffb6e59c640 1 -- 172.21.15.116:0/1361792891 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb6800c940 con 0x7ffb68048380 2024-04-25T00:01:26.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7f2e2ed10640 1 -- 172.21.15.116:0/2870979904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e200db6e0 msgr2=0x7f2e200dbaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7f2e2ed10640 1 --2- 172.21.15.116:0/2870979904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e200db6e0 0x7f2e200dbaf0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2e2802d710 tx=0x7f2e28097f70 comp rx=0 tx=0).stop 2024-04-25T00:01:26.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7f322f7fe640 1 --2- 172.21.15.116:0/1750336278 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32301562a0 0x7f32300c5020 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f32300c7fb0 tx=0x7f322801c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=796a818899140fbe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:26.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7f2e2ed10640 1 -- 172.21.15.116:0/2870979904 shutdown_connections 2024-04-25T00:01:26.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7f2e2ed10640 1 --2- 172.21.15.116:0/2870979904 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e200db6e0 0x7f2e200dbaf0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7f2e2ed10640 1 -- 172.21.15.116:0/2870979904 >> 172.21.15.116:0/2870979904 conn(0x7f2e20048260 msgr2=0x7f2e20048670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:26.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7f322dffb640 1 -- 172.21.15.116:0/1750336278 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32280a57f0 con 0x7f32301562a0 2024-04-25T00:01:26.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7f2e2ed10640 1 -- 172.21.15.116:0/2870979904 shutdown_connections 2024-04-25T00:01:26.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7f2e2ed10640 1 -- 172.21.15.116:0/2870979904 wait complete. 2024-04-25T00:01:26.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7f322dffb640 1 -- 172.21.15.116:0/1750336278 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32280a5990 con 0x7f32301562a0 2024-04-25T00:01:26.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7f322dffb640 1 -- 172.21.15.116:0/1750336278 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32280a5d60 con 0x7f32301562a0 2024-04-25T00:01:26.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.450+0000 7f453b7fe640 1 --2- 172.21.15.116:0/1321191695 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f453400f8d0 0x7f453404dcf0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f45280098a0 tx=0x7f452801c4d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d86075a068169055 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:26.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f30e1182640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f30d400b0c0 0x7f30d40e2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f30e1182640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f30d40ded20 0x7f30d40df1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f30e1182640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30d40de870 con 0x7f30d40dbce0 2024-04-25T00:01:26.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f30e1182640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f30d40e2c50 con 0x7f30d40ded20 2024-04-25T00:01:26.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f30e1182640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f30d40df710 con 0x7f30d400b0c0 2024-04-25T00:01:26.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f30db7fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f30d400b0c0 0x7f30d40e2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f30db7fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f30d400b0c0 0x7f30d40e2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:56052/0 (socket says 172.21.15.116:56052) 2024-04-25T00:01:26.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f30db7fe640 1 -- 172.21.15.116:0/4063931671 learned_addr learned my addr 172.21.15.116:0/4063931671 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:26.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f3235981640 1 -- 172.21.15.116:0/1750336278 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32301562a0 msgr2=0x7f32300c5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f3235981640 1 --2- 172.21.15.116:0/1750336278 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32301562a0 0x7f32300c5020 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f32300c7fb0 tx=0x7f322801c930 comp rx=0 tx=0).stop 2024-04-25T00:01:26.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f30e0981640 1 --2- 172.21.15.116:0/4063931671 >> v2:172.21.15.174:3300/0 conn(0x7f30d40ded20 0x7f30d40df1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f453affd640 1 -- 172.21.15.116:0/1321191695 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45280b0070 con 0x7f453400f8d0 2024-04-25T00:01:26.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f453affd640 1 -- 172.21.15.116:0/1321191695 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45280ab600 con 0x7f453400f8d0 2024-04-25T00:01:26.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f453affd640 1 -- 172.21.15.116:0/1321191695 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45280aba10 con 0x7f453400f8d0 2024-04-25T00:01:26.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f3235981640 1 -- 172.21.15.116:0/1750336278 shutdown_connections 2024-04-25T00:01:26.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f3235981640 1 --2- 172.21.15.116:0/1750336278 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f32301562a0 0x7f32300c5020 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f3235981640 1 --2- 172.21.15.116:0/1750336278 >> v2:172.21.15.174:3300/0 conn(0x7f3230152d80 0x7f3230159950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f3235981640 1 -- 172.21.15.116:0/1750336278 >> 172.21.15.116:0/1750336278 conn(0x7f323006c720 msgr2=0x7f323006cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:26.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f3235981640 1 -- 172.21.15.116:0/1750336278 shutdown_connections 2024-04-25T00:01:26.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f2e2ed10640 1 Processor -- start 2024-04-25T00:01:26.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f2e2ed10640 1 -- start start 2024-04-25T00:01:26.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7ffb6e59c640 1 --2- 172.21.15.116:0/1361792891 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffb68048380 0x7ffb680d82e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ffb60009870 tx=0x7ffb6001c4f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e15ff65640cef02c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:26.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f2e2ed10640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e2000aa20 0x7f2e20041740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f2e2ed10640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e200d5780 0x7f2e200ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f2e2ed10640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e200db6e0 0x7f2e200ceef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f2e2ed10640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2e20007740 con 0x7f2e2000aa20 2024-04-25T00:01:26.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f2e2ed10640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2e200075c0 con 0x7f2e200d5780 2024-04-25T00:01:26.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f2e2ed10640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f2e200dc030 con 0x7f2e200db6e0 2024-04-25T00:01:26.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f2e2dd0e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e2000aa20 0x7f2e20041740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7fc42f7fe640 1 --2- 172.21.15.116:0/2395906088 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc43006bb30 0x7fc43006e5f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc4300c5620 tx=0x7fc42401c3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=777894e73d0f4436 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:26.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f2e2dd0e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e2000aa20 0x7f2e20041740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56214/0 (socket says 172.21.15.116:56214) 2024-04-25T00:01:26.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f2e2dd0e640 1 -- 172.21.15.116:0/189297053 learned_addr learned my addr 172.21.15.116:0/189297053 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:26.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f2e2e50f640 1 --2- 172.21.15.116:0/189297053 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e200db6e0 0x7f2e200ceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f2e2d50d640 1 --2- 172.21.15.116:0/189297053 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e200d5780 0x7f2e200ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7fc42dffb640 1 -- 172.21.15.116:0/2395906088 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4240b0070 con 0x7fc43006bb30 2024-04-25T00:01:26.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f2e2dd0e640 1 -- 172.21.15.116:0/189297053 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e200db6e0 msgr2=0x7f2e200ceef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f2e2dd0e640 1 --2- 172.21.15.116:0/189297053 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e200db6e0 0x7f2e200ceef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f2e2dd0e640 1 -- 172.21.15.116:0/189297053 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e200d5780 msgr2=0x7f2e200ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f2e2dd0e640 1 --2- 172.21.15.116:0/189297053 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e200d5780 0x7f2e200ce9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f2e2dd0e640 1 -- 172.21.15.116:0/189297053 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e20044ae0 con 0x7f2e2000aa20 2024-04-25T00:01:26.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f2e2dd0e640 1 --2- 172.21.15.116:0/189297053 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e2000aa20 0x7f2e20041740 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2e1800c420 tx=0x7f2e1800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:26.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f2e1e7fc640 1 -- 172.21.15.116:0/189297053 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e18017070 con 0x7f2e2000aa20 2024-04-25T00:01:26.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f2e1e7fc640 1 -- 172.21.15.116:0/189297053 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e18005cf0 con 0x7f2e2000aa20 2024-04-25T00:01:26.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f2e1e7fc640 1 -- 172.21.15.116:0/189297053 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e180072f0 con 0x7f2e2000aa20 2024-04-25T00:01:26.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7fc42dffb640 1 -- 172.21.15.116:0/2395906088 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4240ab600 con 0x7fc43006bb30 2024-04-25T00:01:26.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7fc42dffb640 1 -- 172.21.15.116:0/2395906088 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4240ab9d0 con 0x7fc43006bb30 2024-04-25T00:01:26.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7fc43559b640 1 -- 172.21.15.116:0/2395906088 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc43006bb30 msgr2=0x7fc43006e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7fc43559b640 1 --2- 172.21.15.116:0/2395906088 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc43006bb30 0x7fc43006e5f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc4300c5620 tx=0x7fc42401c3f0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f454173d640 1 -- 172.21.15.116:0/1321191695 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f453400f8d0 msgr2=0x7f453404dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f454173d640 1 --2- 172.21.15.116:0/1321191695 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f453400f8d0 0x7f453404dcf0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f45280098a0 tx=0x7f452801c4d0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f454173d640 1 -- 172.21.15.116:0/1321191695 shutdown_connections 2024-04-25T00:01:26.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f454173d640 1 --2- 172.21.15.116:0/1321191695 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f453400ea20 0x7f4534007220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f454173d640 1 --2- 172.21.15.116:0/1321191695 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f453400f8d0 0x7f453404dcf0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f454173d640 1 -- 172.21.15.116:0/1321191695 >> 172.21.15.116:0/1321191695 conn(0x7f4534031740 msgr2=0x7f4534031b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:26.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f454173d640 1 -- 172.21.15.116:0/1321191695 shutdown_connections 2024-04-25T00:01:26.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f454173d640 1 -- 172.21.15.116:0/1321191695 wait complete. 2024-04-25T00:01:26.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f2e2ed10640 1 -- 172.21.15.116:0/189297053 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e20044dc0 con 0x7f2e2000aa20 2024-04-25T00:01:26.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f2e2ed10640 1 -- 172.21.15.116:0/189297053 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e200e38a0 con 0x7f2e2000aa20 2024-04-25T00:01:26.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f2e2ed10640 1 -- 172.21.15.116:0/189297053 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e200053a0 con 0x7f2e2000aa20 2024-04-25T00:01:26.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f3235981640 1 -- 172.21.15.116:0/1750336278 wait complete. 2024-04-25T00:01:26.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f3235981640 1 Processor -- start 2024-04-25T00:01:26.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f3235981640 1 -- start start 2024-04-25T00:01:26.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f3235981640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3230072080 0x7f3230202900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f3235981640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3230152d80 0x7f32301fc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f3235981640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f32301562a0 0x7f32300a9a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f3235981640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3230153160 con 0x7f3230072080 2024-04-25T00:01:26.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f3235981640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f32301566f0 con 0x7f3230152d80 2024-04-25T00:01:26.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f3235981640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f32301532e0 con 0x7f32301562a0 2024-04-25T00:01:26.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f30db7fe640 1 -- 172.21.15.116:0/4063931671 >> v2:172.21.15.174:3300/0 conn(0x7f30d40ded20 msgr2=0x7f30d40df1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:26.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f30db7fe640 1 --2- 172.21.15.116:0/4063931671 >> v2:172.21.15.174:3300/0 conn(0x7f30d40ded20 0x7f30d40df1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f30db7fe640 1 -- 172.21.15.116:0/4063931671 >> v1:172.21.15.116:6789/0 conn(0x7f30d40dbce0 legacy=0x7f30d4006eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f30db7fe640 1 -- 172.21.15.116:0/4063931671 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30d4047620 con 0x7f30d400b0c0 2024-04-25T00:01:26.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f322f7fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f32301562a0 0x7f32300a9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f322f7fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f32301562a0 0x7f32300a9a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:56078/0 (socket says 172.21.15.116:56078) 2024-04-25T00:01:26.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f322f7fe640 1 -- 172.21.15.116:0/1619660765 learned_addr learned my addr 172.21.15.116:0/1619660765 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:26.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f322f7fe640 1 -- 172.21.15.116:0/1619660765 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3230152d80 msgr2=0x7f32301fc7e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T00:01:26.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f322f7fe640 1 --2- 172.21.15.116:0/1619660765 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3230152d80 0x7f32301fc7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f322f7fe640 1 -- 172.21.15.116:0/1619660765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3230072080 msgr2=0x7f3230202900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f322f7fe640 1 --2- 172.21.15.116:0/1619660765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3230072080 0x7f3230202900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.454+0000 7f322f7fe640 1 -- 172.21.15.116:0/1619660765 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3230202ee0 con 0x7f32301562a0 2024-04-25T00:01:26.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.458+0000 7f322f7fe640 1 --2- 172.21.15.116:0/1619660765 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f32301562a0 0x7f32300a9a40 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f32280af040 tx=0x7f32280196d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:26.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.458+0000 7f454173d640 1 Processor -- start 2024-04-25T00:01:26.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.458+0000 7f30db7fe640 1 --2- 172.21.15.116:0/4063931671 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f30d400b0c0 0x7f30d40e2710 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f30c8009870 tx=0x7f30c801c4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3da59ab390373c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:26.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.458+0000 7fc43559b640 1 -- 172.21.15.116:0/2395906088 shutdown_connections 2024-04-25T00:01:26.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.458+0000 7fc43559b640 1 --2- 172.21.15.116:0/2395906088 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc43006bb30 0x7fc43006e5f0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.458+0000 7fc43559b640 1 --2- 172.21.15.116:0/2395906088 >> v2:172.21.15.116:3300/0 conn(0x7fc43006b210 0x7fc43006b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.458+0000 7fc43559b640 1 -- 172.21.15.116:0/2395906088 >> 172.21.15.116:0/2395906088 conn(0x7fc4300c4950 msgr2=0x7fc4300c4d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:26.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.458+0000 7f454173d640 1 -- start start 2024-04-25T00:01:26.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.458+0000 7f454173d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f453400ea20 0x7f45341730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.458+0000 7f454173d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f453400f8d0 0x7f4534177000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.458+0000 7f454173d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45340d7a50 0x7f4534033e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.458+0000 7f454173d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f45340e7710 con 0x7f45340d7a50 2024-04-25T00:01:26.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.458+0000 7f454173d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f45340e7590 con 0x7f453400f8d0 2024-04-25T00:01:26.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.458+0000 7f454173d640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f45340e7890 con 0x7f453400ea20 2024-04-25T00:01:26.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.458+0000 7f4540f3c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45340d7a50 0x7f4534033e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.458+0000 7f4540f3c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45340d7a50 0x7f4534033e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56224/0 (socket says 172.21.15.116:56224) 2024-04-25T00:01:26.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.458+0000 7f4540f3c640 1 -- 172.21.15.116:0/3245013540 learned_addr learned my addr 172.21.15.116:0/3245013540 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:26.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.458+0000 7f453b7fe640 1 --2- 172.21.15.116:0/3245013540 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f453400f8d0 0x7f4534177000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.458+0000 7f453bfff640 1 --2- 172.21.15.116:0/3245013540 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f453400ea20 0x7f45341730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.458+0000 7f4540f3c640 1 -- 172.21.15.116:0/3245013540 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f453400ea20 msgr2=0x7f45341730b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.458+0000 7f4540f3c640 1 --2- 172.21.15.116:0/3245013540 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f453400ea20 0x7f45341730b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.458+0000 7f4540f3c640 1 -- 172.21.15.116:0/3245013540 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f453400f8d0 msgr2=0x7f4534177000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.458+0000 7f4540f3c640 1 --2- 172.21.15.116:0/3245013540 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f453400f8d0 0x7f4534177000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.458+0000 7f4540f3c640 1 -- 172.21.15.116:0/3245013540 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45341736f0 con 0x7f45340d7a50 2024-04-25T00:01:26.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.458+0000 7f2e1e7fc640 1 -- 172.21.15.116:0/189297053 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2e1801a070 con 0x7f2e2000aa20 2024-04-25T00:01:26.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.458+0000 7f2e1e7fc640 1 --2- 172.21.15.116:0/189297053 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2e0c0679c0 0x7f2e0c069e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.458+0000 7f2e1e7fc640 1 -- 172.21.15.116:0/189297053 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7f2e180859d0 con 0x7f2e2000aa20 2024-04-25T00:01:26.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.458+0000 7f4540f3c640 1 --2- 172.21.15.116:0/3245013540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45340d7a50 0x7f4534033e10 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f453c06a1e0 tx=0x7f453c04f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:26.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.462+0000 7f2e2d50d640 1 --2- 172.21.15.116:0/189297053 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2e0c0679c0 0x7f2e0c069e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.462+0000 7f2e2d50d640 1 --2- 172.21.15.116:0/189297053 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2e0c0679c0 0x7f2e0c069e80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2e200051f0 tx=0x7f2e24009060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:26.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.462+0000 7fc43559b640 1 -- 172.21.15.116:0/2395906088 shutdown_connections 2024-04-25T00:01:26.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.462+0000 7fc43559b640 1 -- 172.21.15.116:0/2395906088 wait complete. 2024-04-25T00:01:26.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.462+0000 7fc43559b640 1 Processor -- start 2024-04-25T00:01:26.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.462+0000 7fc43559b640 1 -- start start 2024-04-25T00:01:26.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.462+0000 7ffb6d59a640 1 -- 172.21.15.116:0/1361792891 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb600b0070 con 0x7ffb68048380 2024-04-25T00:01:26.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.462+0000 7f30daffd640 1 -- 172.21.15.116:0/4063931671 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30c80ab600 con 0x7f30d400b0c0 2024-04-25T00:01:26.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.462+0000 7f3217fff640 1 -- 172.21.15.116:0/1619660765 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32280b8670 con 0x7f32301562a0 2024-04-25T00:01:26.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.462+0000 7ffb6d59a640 1 -- 172.21.15.116:0/1361792891 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb600ab600 con 0x7ffb68048380 2024-04-25T00:01:26.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.462+0000 7ffb6d59a640 1 -- 172.21.15.116:0/1361792891 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb600ab9d0 con 0x7ffb68048380 2024-04-25T00:01:26.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.462+0000 7fc1e57fa640 1 -- 172.21.15.116:0/3898419650 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fc1e0021080 con 0x7fc1e806cb70 2024-04-25T00:01:26.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.462+0000 7f30daffd640 1 -- 172.21.15.116:0/4063931671 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30c80b0070 con 0x7f30d400b0c0 2024-04-25T00:01:26.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.462+0000 7f30daffd640 1 -- 172.21.15.116:0/4063931671 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30c80aba00 con 0x7f30d400b0c0 2024-04-25T00:01:26.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.462+0000 7f3235981640 1 -- 172.21.15.116:0/1619660765 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32302031c0 con 0x7f32301562a0 2024-04-25T00:01:26.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.462+0000 7f3235981640 1 -- 172.21.15.116:0/1619660765 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3230150ea0 con 0x7f32301562a0 2024-04-25T00:01:26.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.462+0000 7f3217fff640 1 -- 172.21.15.116:0/1619660765 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32280be070 con 0x7f32301562a0 2024-04-25T00:01:26.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.462+0000 7f3217fff640 1 -- 172.21.15.116:0/1619660765 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32280b8a50 con 0x7f32301562a0 2024-04-25T00:01:26.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.462+0000 7f30e1182640 1 -- 172.21.15.116:0/4063931671 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f30d400b0c0 msgr2=0x7f30d40e2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.462+0000 7f30e1182640 1 --2- 172.21.15.116:0/4063931671 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f30d400b0c0 0x7f30d40e2710 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f30c8009870 tx=0x7f30c801c4e0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.462+0000 7fc43559b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc43006b210 0x7fc43015c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.462+0000 7fc43559b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc43006bb30 0x7fc4301f30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.462+0000 7fc43559b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc430155060 0x7fc4301f3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.462+0000 7fc43559b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc4300c4430 con 0x7fc43006bb30 2024-04-25T00:01:26.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.462+0000 7fc43559b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc4300c42b0 con 0x7fc430155060 2024-04-25T00:01:26.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.462+0000 7fc43559b640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fc430166c30 con 0x7fc43006b210 2024-04-25T00:01:26.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.462+0000 7f30e1182640 1 -- 172.21.15.116:0/4063931671 shutdown_connections 2024-04-25T00:01:26.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.462+0000 7f30e1182640 1 --2- 172.21.15.116:0/4063931671 >> v2:172.21.15.174:3300/0 conn(0x7f30d40ded20 0x7f30d40df1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.462+0000 7f30e1182640 1 --2- 172.21.15.116:0/4063931671 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f30d400b0c0 0x7f30d40e2710 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.462+0000 7f30e1182640 1 -- 172.21.15.116:0/4063931671 >> 172.21.15.116:0/4063931671 conn(0x7f30d4048260 msgr2=0x7f30d4048670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:26.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.462+0000 7fc42f7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc430155060 0x7fc4301f3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.462+0000 7fc42effd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc43006b210 0x7fc43015c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.462+0000 7fc42e7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc43006bb30 0x7fc4301f30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.462+0000 7fc42e7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc43006bb30 0x7fc4301f30c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56232/0 (socket says 172.21.15.116:56232) 2024-04-25T00:01:26.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.462+0000 7fc42e7fc640 1 -- 172.21.15.116:0/511534478 learned_addr learned my addr 172.21.15.116:0/511534478 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:26.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.462+0000 7fc42e7fc640 1 -- 172.21.15.116:0/511534478 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc43006b210 msgr2=0x7fc43015c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.462+0000 7fc42e7fc640 1 --2- 172.21.15.116:0/511534478 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc43006b210 0x7fc43015c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.462+0000 7fc42e7fc640 1 -- 172.21.15.116:0/511534478 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc430155060 msgr2=0x7fc4301f3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.462+0000 7fc42e7fc640 1 --2- 172.21.15.116:0/511534478 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc430155060 0x7fc4301f3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.462+0000 7fc42e7fc640 1 -- 172.21.15.116:0/511534478 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4300c5ae0 con 0x7fc43006bb30 2024-04-25T00:01:26.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.462+0000 7f3235981640 1 -- 172.21.15.116:0/1619660765 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32100052a0 con 0x7f32301562a0 2024-04-25T00:01:26.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.462+0000 7f2e1e7fc640 1 -- 172.21.15.116:0/189297053 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f2e18021070 con 0x7f2e2000aa20 2024-04-25T00:01:26.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.462+0000 7fc42e7fc640 1 --2- 172.21.15.116:0/511534478 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc43006bb30 0x7fc4301f30c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc42000c500 tx=0x7fc42000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:26.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.462+0000 7ffb74daa640 1 -- 172.21.15.116:0/1361792891 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffb68048380 msgr2=0x7ffb680d82e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.462+0000 7ffb74daa640 1 --2- 172.21.15.116:0/1361792891 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffb68048380 0x7ffb680d82e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ffb60009870 tx=0x7ffb6001c4f0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.462+0000 7ffb74daa640 1 -- 172.21.15.116:0/1361792891 shutdown_connections 2024-04-25T00:01:26.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.462+0000 7ffb74daa640 1 --2- 172.21.15.116:0/1361792891 >> v2:172.21.15.174:3300/0 conn(0x7ffb6800be80 0x7ffb68048760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.462+0000 7ffb74daa640 1 --2- 172.21.15.116:0/1361792891 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffb68048380 0x7ffb680d82e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.462+0000 7ffb74daa640 1 -- 172.21.15.116:0/1361792891 >> 172.21.15.116:0/1361792891 conn(0x7ffb6802f4f0 msgr2=0x7ffb6802f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:26.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.462+0000 7ffb74daa640 1 -- 172.21.15.116:0/1361792891 shutdown_connections 2024-04-25T00:01:26.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.462+0000 7ffb74daa640 1 -- 172.21.15.116:0/1361792891 wait complete. 2024-04-25T00:01:26.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.466+0000 7f30e1182640 1 -- 172.21.15.116:0/4063931671 shutdown_connections 2024-04-25T00:01:26.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.466+0000 7f30e1182640 1 -- 172.21.15.116:0/4063931671 wait complete. 2024-04-25T00:01:26.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.466+0000 7fc4177fe640 1 -- 172.21.15.116:0/511534478 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc420017070 con 0x7fc43006bb30 2024-04-25T00:01:26.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.466+0000 7ffb74daa640 1 Processor -- start 2024-04-25T00:01:26.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.466+0000 7f3217fff640 1 -- 172.21.15.116:0/1619660765 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f32280b8bf0 con 0x7f32301562a0 2024-04-25T00:01:26.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.466+0000 7fc43559b640 1 -- 172.21.15.116:0/511534478 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc430149930 con 0x7fc43006bb30 2024-04-25T00:01:26.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.466+0000 7fc43559b640 1 -- 172.21.15.116:0/511534478 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc43015c670 con 0x7fc43006bb30 2024-04-25T00:01:26.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.466+0000 7f3217fff640 1 --2- 172.21.15.116:0/1619660765 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f321c0678f0 0x7f321c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.466+0000 7f3217fff640 1 -- 172.21.15.116:0/1619660765 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7f32281213b0 con 0x7f32301562a0 2024-04-25T00:01:26.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.466+0000 7f322effd640 1 --2- 172.21.15.116:0/1619660765 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f321c0678f0 0x7f321c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.466+0000 7ffb74daa640 1 -- start start 2024-04-25T00:01:26.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.466+0000 7ffb74daa640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb6800be80 0x7ffb6802e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.466+0000 7ffb74daa640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffb68048380 0x7ffb680cae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.466+0000 7ffb74daa640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffb680d25c0 0x7ffb681771b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.466+0000 7ffb74daa640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ffb68007d80 con 0x7ffb6800be80 2024-04-25T00:01:26.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.466+0000 7ffb74daa640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ffb68007c00 con 0x7ffb68048380 2024-04-25T00:01:26.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.466+0000 7ffb74daa640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ffb68007f00 con 0x7ffb680d25c0 2024-04-25T00:01:26.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.466+0000 7fc43559b640 1 -- 172.21.15.116:0/511534478 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc43004f580 con 0x7fc43006bb30 2024-04-25T00:01:26.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.466+0000 7f30e1182640 1 Processor -- start 2024-04-25T00:01:26.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.466+0000 7f30e1182640 1 -- start start 2024-04-25T00:01:26.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.466+0000 7ffb6e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb6800be80 0x7ffb6802e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.466+0000 7ffb6e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb6800be80 0x7ffb6802e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56238/0 (socket says 172.21.15.116:56238) 2024-04-25T00:01:26.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.466+0000 7ffb6e59c640 1 -- 172.21.15.116:0/1884896888 learned_addr learned my addr 172.21.15.116:0/1884896888 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:26.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.466+0000 7ffb6ed9d640 1 --2- 172.21.15.116:0/1884896888 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffb680d25c0 0x7ffb681771b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.466+0000 7ffb6dd9b640 1 --2- 172.21.15.116:0/1884896888 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffb68048380 0x7ffb680cae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.466+0000 7fc4177fe640 1 -- 172.21.15.116:0/511534478 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc420005cf0 con 0x7fc43006bb30 2024-04-25T00:01:26.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.466+0000 7f322effd640 1 --2- 172.21.15.116:0/1619660765 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f321c0678f0 0x7f321c069db0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3220006aa0 tx=0x7f3220006a30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:26.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.466+0000 7fc4177fe640 1 -- 172.21.15.116:0/511534478 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4200072f0 con 0x7fc43006bb30 2024-04-25T00:01:26.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.466+0000 7ffb6e59c640 1 -- 172.21.15.116:0/1884896888 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffb680d25c0 msgr2=0x7ffb681771b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.466+0000 7ffb6e59c640 1 --2- 172.21.15.116:0/1884896888 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffb680d25c0 0x7ffb681771b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.466+0000 7ffb6e59c640 1 -- 172.21.15.116:0/1884896888 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffb68048380 msgr2=0x7ffb680cae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.466+0000 7ffb6e59c640 1 --2- 172.21.15.116:0/1884896888 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffb68048380 0x7ffb680cae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.466+0000 7ffb6e59c640 1 -- 172.21.15.116:0/1884896888 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb6802d860 con 0x7ffb6800be80 2024-04-25T00:01:26.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.466+0000 7f30e1182640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30d400b0c0 0x7f30d4043710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.466+0000 7f30e1182640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30d40dbce0 0x7f30d40da670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.466+0000 7f30e1182640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f30d40ded20 0x7f30d40dabb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.466+0000 7f30e1182640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f30d40dfc50 con 0x7f30d400b0c0 2024-04-25T00:01:26.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.466+0000 7f30e1182640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f30d4047ec0 con 0x7f30d40dbce0 2024-04-25T00:01:26.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.466+0000 7f30e1182640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f30d40dc0c0 con 0x7f30d40ded20 2024-04-25T00:01:26.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.466+0000 7f30dbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30d400b0c0 0x7f30d4043710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.466+0000 7f30dbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30d400b0c0 0x7f30d4043710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56244/0 (socket says 172.21.15.116:56244) 2024-04-25T00:01:26.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.466+0000 7f30dbfff640 1 -- 172.21.15.116:0/2423118928 learned_addr learned my addr 172.21.15.116:0/2423118928 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:26.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.466+0000 7ffb6e59c640 1 --2- 172.21.15.116:0/1884896888 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb6800be80 0x7ffb6802e100 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ffb60009870 tx=0x7ffb6001ad30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:26.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.466+0000 7f30e0981640 1 --2- 172.21.15.116:0/2423118928 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f30d40ded20 0x7f30d40dabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.466+0000 7f30db7fe640 1 --2- 172.21.15.116:0/2423118928 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30d40dbce0 0x7f30d40da670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.466+0000 7f30dbfff640 1 -- 172.21.15.116:0/2423118928 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f30d40ded20 msgr2=0x7f30d40dabb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.466+0000 7f30dbfff640 1 --2- 172.21.15.116:0/2423118928 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f30d40ded20 0x7f30d40dabb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.466+0000 7f30dbfff640 1 -- 172.21.15.116:0/2423118928 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30d40dbce0 msgr2=0x7f30d40da670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.466+0000 7f30dbfff640 1 --2- 172.21.15.116:0/2423118928 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30d40dbce0 0x7f30d40da670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.466+0000 7f30dbfff640 1 -- 172.21.15.116:0/2423118928 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30d400ae90 con 0x7f30d400b0c0 2024-04-25T00:01:26.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.470+0000 7f4538ff9640 1 -- 172.21.15.116:0/3245013540 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f453c073390 con 0x7f45340d7a50 2024-04-25T00:01:26.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.470+0000 7f30dbfff640 1 --2- 172.21.15.116:0/2423118928 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30d400b0c0 0x7f30d4043710 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f30d000c450 tx=0x7f30d000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:26.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.470+0000 7fc4177fe640 1 -- 172.21.15.116:0/511534478 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc42001a070 con 0x7fc43006bb30 2024-04-25T00:01:26.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.470+0000 7ffb5effd640 1 -- 172.21.15.116:0/1884896888 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb600ba020 con 0x7ffb6800be80 2024-04-25T00:01:26.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.470+0000 7f454173d640 1 -- 172.21.15.116:0/3245013540 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45340d9c30 con 0x7f45340d7a50 2024-04-25T00:01:26.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.470+0000 7f454173d640 1 -- 172.21.15.116:0/3245013540 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4534176800 con 0x7f45340d7a50 2024-04-25T00:01:26.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.470+0000 7f4538ff9640 1 -- 172.21.15.116:0/3245013540 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f453c073530 con 0x7f45340d7a50 2024-04-25T00:01:26.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.470+0000 7f4538ff9640 1 -- 172.21.15.116:0/3245013540 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f453c06e410 con 0x7f45340d7a50 2024-04-25T00:01:26.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.470+0000 7ffb74daa640 1 -- 172.21.15.116:0/1884896888 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb6802db40 con 0x7ffb6800be80 2024-04-25T00:01:26.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.470+0000 7ffb74daa640 1 -- 172.21.15.116:0/1884896888 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb68176c90 con 0x7ffb6800be80 2024-04-25T00:01:26.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.470+0000 7ffb74daa640 1 -- 172.21.15.116:0/1884896888 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb68005820 con 0x7ffb6800be80 2024-04-25T00:01:26.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.470+0000 7ffb5effd640 1 -- 172.21.15.116:0/1884896888 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb600b6620 con 0x7ffb6800be80 2024-04-25T00:01:26.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.470+0000 7f454173d640 1 -- 172.21.15.116:0/3245013540 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f451c0052a0 con 0x7f45340d7a50 2024-04-25T00:01:26.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.470+0000 7fc4177fe640 1 --2- 172.21.15.116:0/511534478 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc41c067990 0x7fc41c069e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.470+0000 7fc4177fe640 1 -- 172.21.15.116:0/511534478 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7fc420085270 con 0x7fc43006bb30 2024-04-25T00:01:26.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.470+0000 7fc42effd640 1 --2- 172.21.15.116:0/511534478 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc41c067990 0x7fc41c069e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.470+0000 7ffb5effd640 1 -- 172.21.15.116:0/1884896888 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb600b0040 con 0x7ffb6800be80 2024-04-25T00:01:26.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.470+0000 7f30d8ff9640 1 -- 172.21.15.116:0/2423118928 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d00052d0 con 0x7f30d400b0c0 2024-04-25T00:01:26.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.470+0000 7f30e1182640 1 -- 172.21.15.116:0/2423118928 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30d4042120 con 0x7f30d400b0c0 2024-04-25T00:01:26.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.470+0000 7f30e1182640 1 -- 172.21.15.116:0/2423118928 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30d403f740 con 0x7f30d400b0c0 2024-04-25T00:01:26.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.470+0000 7fc42effd640 1 --2- 172.21.15.116:0/511534478 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc41c067990 0x7fc41c069e50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc428003ab0 tx=0x7fc428009830 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:26.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.474+0000 7f30e1182640 1 -- 172.21.15.116:0/2423118928 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30d40446e0 con 0x7f30d400b0c0 2024-04-25T00:01:26.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.474+0000 7f30d8ff9640 1 -- 172.21.15.116:0/2423118928 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30d0001130 con 0x7f30d400b0c0 2024-04-25T00:01:26.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.474+0000 7f30d8ff9640 1 -- 172.21.15.116:0/2423118928 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30d0020060 con 0x7f30d400b0c0 2024-04-25T00:01:26.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.478+0000 7fc4177fe640 1 -- 172.21.15.116:0/511534478 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fc4200554a0 con 0x7fc43006bb30 2024-04-25T00:01:26.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.478+0000 7f4538ff9640 1 -- 172.21.15.116:0/3245013540 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f453c06e5b0 con 0x7f45340d7a50 2024-04-25T00:01:26.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.478+0000 7f4538ff9640 1 --2- 172.21.15.116:0/3245013540 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4520067940 0x7f4520069e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.478+0000 7f453bfff640 1 --2- 172.21.15.116:0/3245013540 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4520067940 0x7f4520069e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.478+0000 7f3217fff640 1 -- 172.21.15.116:0/1619660765 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f32280165e0 con 0x7f32301562a0 2024-04-25T00:01:26.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.478+0000 7f30d8ff9640 1 -- 172.21.15.116:0/2423118928 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f30d0003600 con 0x7f30d400b0c0 2024-04-25T00:01:26.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.478+0000 7f30d8ff9640 1 --2- 172.21.15.116:0/2423118928 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f30c00678c0 0x7f30c0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.478+0000 7f30db7fe640 1 --2- 172.21.15.116:0/2423118928 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f30c00678c0 0x7f30c0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.478+0000 7f4538ff9640 1 -- 172.21.15.116:0/3245013540 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7f453c095df0 con 0x7f45340d7a50 2024-04-25T00:01:26.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.478+0000 7f30d8ff9640 1 -- 172.21.15.116:0/2423118928 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7f30d0053b00 con 0x7f30d400b0c0 2024-04-25T00:01:26.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.478+0000 7f453bfff640 1 --2- 172.21.15.116:0/3245013540 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4520067940 0x7f4520069e00 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4530001770 tx=0x7f4530009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:26.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.482+0000 7ffb5effd640 1 -- 172.21.15.116:0/1884896888 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ffb600bd020 con 0x7ffb6800be80 2024-04-25T00:01:26.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.482+0000 7f30db7fe640 1 --2- 172.21.15.116:0/2423118928 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f30c00678c0 0x7f30c0069d80 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f30c80af000 tx=0x7f30c80a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:26.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.482+0000 7ffb5effd640 1 --2- 172.21.15.116:0/1884896888 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ffb500678c0 0x7ffb50069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.482+0000 7ffb5effd640 1 -- 172.21.15.116:0/1884896888 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7ffb601205d0 con 0x7ffb6800be80 2024-04-25T00:01:26.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.482+0000 7ffb6dd9b640 1 --2- 172.21.15.116:0/1884896888 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ffb500678c0 0x7ffb50069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.482+0000 7ffb6dd9b640 1 --2- 172.21.15.116:0/1884896888 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ffb500678c0 0x7ffb50069d80 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ffb7004fa30 tx=0x7ffb70072040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:26.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.486+0000 7f4538ff9640 1 -- 172.21.15.116:0/3245013540 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f453c0a3080 con 0x7f45340d7a50 2024-04-25T00:01:26.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.490+0000 7f6c24d2f640 1 Processor -- start 2024-04-25T00:01:26.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.490+0000 7f6c24d2f640 1 -- start start 2024-04-25T00:01:26.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.490+0000 7ffb5effd640 1 -- 172.21.15.116:0/1884896888 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ffb600c4030 con 0x7ffb6800be80 2024-04-25T00:01:26.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.490+0000 7f6c24d2f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6c20155b20 0x7f6c20152c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.490+0000 7f6c24d2f640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f6c200aacd0 0x7f6c20153180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.490+0000 7f6c24d2f640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c200a9e60 con 0x7f6c2014c700 2024-04-25T00:01:26.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.490+0000 7f6c24d2f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f6c201527b0 con 0x7f6c20155b20 2024-04-25T00:01:26.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.490+0000 7f6c24d2f640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f6c200c3ef0 con 0x7f6c200aacd0 2024-04-25T00:01:26.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.490+0000 7f6c1dd9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6c20155b20 0x7f6c20152c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.490+0000 7f6c1dd9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6c20155b20 0x7f6c20152c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56254/0 (socket says 172.21.15.116:56254) 2024-04-25T00:01:26.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.490+0000 7f6c1dd9b640 1 -- 172.21.15.116:0/558353700 learned_addr learned my addr 172.21.15.116:0/558353700 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:26.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.490+0000 7f6c1ed9d640 1 --2- 172.21.15.116:0/558353700 >> v2:172.21.15.174:3300/0 conn(0x7f6c200aacd0 0x7f6c20153180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.494+0000 7f6c1dd9b640 1 -- 172.21.15.116:0/558353700 >> v2:172.21.15.174:3300/0 conn(0x7f6c200aacd0 msgr2=0x7f6c20153180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:26.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.494+0000 7f6c1dd9b640 1 --2- 172.21.15.116:0/558353700 >> v2:172.21.15.174:3300/0 conn(0x7f6c200aacd0 0x7f6c20153180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.494+0000 7f6c1dd9b640 1 -- 172.21.15.116:0/558353700 >> v1:172.21.15.174:6789/0 conn(0x7f6c2014c700 legacy=0x7f6c200c5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.494+0000 7f6c1dd9b640 1 -- 172.21.15.116:0/558353700 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c200c41d0 con 0x7f6c20155b20 2024-04-25T00:01:26.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.494+0000 7f6c1dd9b640 1 --2- 172.21.15.116:0/558353700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c20155b20 0x7f6c20152c40 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6c20072e90 tx=0x7f6c1001c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25d0388cfb2129d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:26.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.498+0000 7f30d8ff9640 1 -- 172.21.15.116:0/2423118928 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f30d0056160 con 0x7f30d400b0c0 2024-04-25T00:01:26.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.498+0000 7f6c1d59a640 1 -- 172.21.15.116:0/558353700 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c100b0070 con 0x7f6c20155b20 2024-04-25T00:01:26.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.498+0000 7f6c1d59a640 1 -- 172.21.15.116:0/558353700 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c100ab600 con 0x7f6c20155b20 2024-04-25T00:01:26.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.498+0000 7f6c24d2f640 1 -- 172.21.15.116:0/558353700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c20155b20 msgr2=0x7f6c20152c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.498+0000 7f6c24d2f640 1 --2- 172.21.15.116:0/558353700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c20155b20 0x7f6c20152c40 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6c20072e90 tx=0x7f6c1001c400 comp rx=0 tx=0).stop 2024-04-25T00:01:26.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.498+0000 7f6c24d2f640 1 -- 172.21.15.116:0/558353700 shutdown_connections 2024-04-25T00:01:26.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.498+0000 7f6c24d2f640 1 --2- 172.21.15.116:0/558353700 >> v2:172.21.15.174:3300/0 conn(0x7f6c200aacd0 0x7f6c20153180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.498+0000 7f6c24d2f640 1 --2- 172.21.15.116:0/558353700 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c20155b20 0x7f6c20152c40 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.498+0000 7f6c24d2f640 1 -- 172.21.15.116:0/558353700 >> 172.21.15.116:0/558353700 conn(0x7f6c2006c720 msgr2=0x7f6c2006cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:26.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.498+0000 7f6c24d2f640 1 -- 172.21.15.116:0/558353700 shutdown_connections 2024-04-25T00:01:26.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.498+0000 7f6c24d2f640 1 -- 172.21.15.116:0/558353700 wait complete. 2024-04-25T00:01:26.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.502+0000 7f6c24d2f640 1 Processor -- start 2024-04-25T00:01:26.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.502+0000 7f6c24d2f640 1 -- start start 2024-04-25T00:01:26.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.502+0000 7f6c24d2f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c200aacd0 0x7f6c20149bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.502+0000 7f6c24d2f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c2014c700 0x7f6c2014a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.502+0000 7f6c24d2f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c20155b20 0x7f6c2015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.502+0000 7f6c24d2f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6c2006bdf0 con 0x7f6c200aacd0 2024-04-25T00:01:26.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.502+0000 7f6c24d2f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6c2006bc70 con 0x7f6c2014c700 2024-04-25T00:01:26.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.502+0000 7f6c24d2f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6c2006bf70 con 0x7f6c20155b20 2024-04-25T00:01:26.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.502+0000 7f6c1e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c200aacd0 0x7f6c20149bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.502+0000 7f6c1e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c200aacd0 0x7f6c20149bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56260/0 (socket says 172.21.15.116:56260) 2024-04-25T00:01:26.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.502+0000 7f6c1e59c640 1 -- 172.21.15.116:0/550517207 learned_addr learned my addr 172.21.15.116:0/550517207 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:26.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.502+0000 7f6c1dd9b640 1 --2- 172.21.15.116:0/550517207 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c2014c700 0x7f6c2014a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.502+0000 7f6c1e59c640 1 -- 172.21.15.116:0/550517207 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c20155b20 msgr2=0x7f6c2015dca0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T00:01:26.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.502+0000 7f6c1e59c640 1 --2- 172.21.15.116:0/550517207 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c20155b20 0x7f6c2015dca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.502+0000 7f6c1e59c640 1 -- 172.21.15.116:0/550517207 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c2014c700 msgr2=0x7f6c2014a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.502+0000 7f6c1e59c640 1 --2- 172.21.15.116:0/550517207 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c2014c700 0x7f6c2014a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.502+0000 7f6c1e59c640 1 -- 172.21.15.116:0/550517207 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c200ac9f0 con 0x7f6c200aacd0 2024-04-25T00:01:26.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.502+0000 7f6c1e59c640 1 --2- 172.21.15.116:0/550517207 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c200aacd0 0x7f6c20149bc0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f6c1400c530 tx=0x7f6c1400c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:26.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.502+0000 7f6c0effd640 1 -- 172.21.15.116:0/550517207 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c14016020 con 0x7f6c200aacd0 2024-04-25T00:01:26.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.502+0000 7f6c0effd640 1 -- 172.21.15.116:0/550517207 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c14005220 con 0x7f6c200aacd0 2024-04-25T00:01:26.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.502+0000 7f6c0effd640 1 -- 172.21.15.116:0/550517207 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c14007030 con 0x7f6c200aacd0 2024-04-25T00:01:26.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.502+0000 7f6c24d2f640 1 -- 172.21.15.116:0/550517207 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c2014a820 con 0x7f6c200aacd0 2024-04-25T00:01:26.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.502+0000 7f6c24d2f640 1 -- 172.21.15.116:0/550517207 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c200c3400 con 0x7f6c200aacd0 2024-04-25T00:01:26.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.506+0000 7f6c24d2f640 1 -- 172.21.15.116:0/550517207 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c2014a640 con 0x7f6c200aacd0 2024-04-25T00:01:26.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.506+0000 7f6c0effd640 1 -- 172.21.15.116:0/550517207 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6c140071d0 con 0x7f6c200aacd0 2024-04-25T00:01:26.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.506+0000 7f6c0effd640 1 --2- 172.21.15.116:0/550517207 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6c000678f0 0x7f6c00069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:26.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.506+0000 7f6c0effd640 1 -- 172.21.15.116:0/550517207 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7f6c140867e0 con 0x7f6c200aacd0 2024-04-25T00:01:26.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.506+0000 7f6c1dd9b640 1 --2- 172.21.15.116:0/550517207 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6c000678f0 0x7f6c00069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:26.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.510+0000 7f6c1dd9b640 1 --2- 172.21.15.116:0/550517207 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6c000678f0 0x7f6c00069db0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6c100ad000 tx=0x7f6c100a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:26.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.514+0000 7f6c0effd640 1 -- 172.21.15.116:0/550517207 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f6c2014a640 con 0x7f6c200aacd0 2024-04-25T00:01:26.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.782+0000 7fc1eed36640 1 -- 172.21.15.116:0/3898419650 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fc1c8005860 con 0x7fc1e806cb70 2024-04-25T00:01:26.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.782+0000 7fc1e57fa640 1 -- 172.21.15.116:0/3898419650 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc1e0055c60 con 0x7fc1e806cb70 2024-04-25T00:01:26.787 INFO:teuthology.orchestra.run.smithi116.stdout:64424509448 2024-04-25T00:01:26.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.786+0000 7f2e2ed10640 1 -- 172.21.15.116:0/189297053 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f2e200cf430 con 0x7f2e2000aa20 2024-04-25T00:01:26.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.786+0000 7f2e1e7fc640 1 -- 172.21.15.116:0/189297053 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f2e18055c00 con 0x7f2e2000aa20 2024-04-25T00:01:26.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.786+0000 7fc1eed36640 1 -- 172.21.15.116:0/3898419650 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc1d40678f0 msgr2=0x7fc1d4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.786+0000 7fc1eed36640 1 --2- 172.21.15.116:0/3898419650 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc1d40678f0 0x7fc1d4069db0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc1d80af000 tx=0x7fc1d80a8000 comp rx=0 tx=0).stop 2024-04-25T00:01:26.788 INFO:teuthology.orchestra.run.smithi116.stdout:64424509449 2024-04-25T00:01:26.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.786+0000 7fc1eed36640 1 -- 172.21.15.116:0/3898419650 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1e806cb70 msgr2=0x7fc1e8151710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.786+0000 7fc1eed36640 1 --2- 172.21.15.116:0/3898419650 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1e806cb70 0x7fc1e8151710 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc1e000cb20 tx=0x7fc1e00021a0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.786+0000 7fc1eed36640 1 -- 172.21.15.116:0/3898419650 shutdown_connections 2024-04-25T00:01:26.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.786+0000 7fc1eed36640 1 --2- 172.21.15.116:0/3898419650 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc1d40678f0 0x7fc1d4069db0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.786+0000 7fc1eed36640 1 --2- 172.21.15.116:0/3898419650 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1e813acb0 0x7fc1e81f73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.786+0000 7fc1eed36640 1 --2- 172.21.15.116:0/3898419650 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc1e813a360 0x7fc1e814fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.786+0000 7fc1eed36640 1 --2- 172.21.15.116:0/3898419650 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1e806cb70 0x7fc1e8151710 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.786+0000 7fc1eed36640 1 -- 172.21.15.116:0/3898419650 >> 172.21.15.116:0/3898419650 conn(0x7fc1e80c3a90 msgr2=0x7fc1e80c3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:26.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.786+0000 7fc1eed36640 1 -- 172.21.15.116:0/3898419650 shutdown_connections 2024-04-25T00:01:26.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.786+0000 7fc1eed36640 1 -- 172.21.15.116:0/3898419650 wait complete. 2024-04-25T00:01:26.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.798+0000 7fc43559b640 1 -- 172.21.15.116:0/511534478 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fc430150fa0 con 0x7fc43006bb30 2024-04-25T00:01:26.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.798+0000 7f2e2ed10640 1 -- 172.21.15.116:0/189297053 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2e0c0679c0 msgr2=0x7f2e0c069e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.798+0000 7f2e2ed10640 1 --2- 172.21.15.116:0/189297053 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2e0c0679c0 0x7f2e0c069e80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2e200051f0 tx=0x7f2e24009060 comp rx=0 tx=0).stop 2024-04-25T00:01:26.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.798+0000 7f2e2ed10640 1 -- 172.21.15.116:0/189297053 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e2000aa20 msgr2=0x7f2e20041740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.798+0000 7f2e2ed10640 1 --2- 172.21.15.116:0/189297053 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e2000aa20 0x7f2e20041740 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2e1800c420 tx=0x7f2e1800c8e0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.798+0000 7f2e2ed10640 1 -- 172.21.15.116:0/189297053 shutdown_connections 2024-04-25T00:01:26.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.798+0000 7f2e2ed10640 1 --2- 172.21.15.116:0/189297053 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2e0c0679c0 0x7f2e0c069e80 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.798+0000 7f2e2ed10640 1 --2- 172.21.15.116:0/189297053 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e200db6e0 0x7f2e200ceef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.798+0000 7f2e2ed10640 1 --2- 172.21.15.116:0/189297053 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e200d5780 0x7f2e200ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.798+0000 7f2e2ed10640 1 --2- 172.21.15.116:0/189297053 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e2000aa20 0x7f2e20041740 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.798+0000 7f2e2ed10640 1 -- 172.21.15.116:0/189297053 >> 172.21.15.116:0/189297053 conn(0x7f2e20048260 msgr2=0x7f2e2002c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:26.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.798+0000 7f2e2ed10640 1 -- 172.21.15.116:0/189297053 shutdown_connections 2024-04-25T00:01:26.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.798+0000 7fc4177fe640 1 -- 172.21.15.116:0/511534478 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc4200597a0 con 0x7fc43006bb30 2024-04-25T00:01:26.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.798+0000 7f2e2ed10640 1 -- 172.21.15.116:0/189297053 wait complete. 2024-04-25T00:01:26.801 INFO:teuthology.orchestra.run.smithi116.stdout:64424509449 2024-04-25T00:01:26.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.802+0000 7fc43559b640 1 -- 172.21.15.116:0/511534478 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc41c067990 msgr2=0x7fc41c069e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.802+0000 7fc43559b640 1 --2- 172.21.15.116:0/511534478 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc41c067990 0x7fc41c069e50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc428003ab0 tx=0x7fc428009830 comp rx=0 tx=0).stop 2024-04-25T00:01:26.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.802+0000 7fc43559b640 1 -- 172.21.15.116:0/511534478 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc43006bb30 msgr2=0x7fc4301f30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.802+0000 7fc43559b640 1 --2- 172.21.15.116:0/511534478 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc43006bb30 0x7fc4301f30c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc42000c500 tx=0x7fc42000c9c0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.802+0000 7fc43559b640 1 -- 172.21.15.116:0/511534478 shutdown_connections 2024-04-25T00:01:26.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.802+0000 7fc43559b640 1 --2- 172.21.15.116:0/511534478 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc41c067990 0x7fc41c069e50 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.802+0000 7fc43559b640 1 --2- 172.21.15.116:0/511534478 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc430155060 0x7fc4301f3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.802+0000 7fc43559b640 1 --2- 172.21.15.116:0/511534478 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc43006bb30 0x7fc4301f30c0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.802+0000 7fc43559b640 1 --2- 172.21.15.116:0/511534478 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc43006b210 0x7fc43015c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.802+0000 7fc43559b640 1 -- 172.21.15.116:0/511534478 >> 172.21.15.116:0/511534478 conn(0x7fc4300c4950 msgr2=0x7fc4300c4d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:26.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.806+0000 7fc43559b640 1 -- 172.21.15.116:0/511534478 shutdown_connections 2024-04-25T00:01:26.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.806+0000 7fc43559b640 1 -- 172.21.15.116:0/511534478 wait complete. 2024-04-25T00:01:26.808 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509449 got 64424509448 for osd.5 2024-04-25T00:01:26.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.810+0000 7f454173d640 1 -- 172.21.15.116:0/3245013540 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f451c005860 con 0x7f45340d7a50 2024-04-25T00:01:26.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.810+0000 7f4538ff9640 1 -- 172.21.15.116:0/3245013540 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f453c0d1030 con 0x7f45340d7a50 2024-04-25T00:01:26.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.810+0000 7ffb74daa640 1 -- 172.21.15.116:0/1884896888 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7ffb68033980 con 0x7ffb6800be80 2024-04-25T00:01:26.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.810+0000 7f3235981640 1 -- 172.21.15.116:0/1619660765 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f3210005860 con 0x7f32301562a0 2024-04-25T00:01:26.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.814+0000 7ffb5effd640 1 -- 172.21.15.116:0/1884896888 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ffb60016170 con 0x7ffb6800be80 2024-04-25T00:01:26.814 INFO:teuthology.orchestra.run.smithi116.stdout:64424509450 2024-04-25T00:01:26.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.814+0000 7f3217fff640 1 -- 172.21.15.116:0/1619660765 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f32280f1da0 con 0x7f32301562a0 2024-04-25T00:01:26.817 INFO:teuthology.orchestra.run.smithi116.stdout:64424509449 2024-04-25T00:01:26.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.814+0000 7ffb74daa640 1 -- 172.21.15.116:0/1884896888 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ffb500678c0 msgr2=0x7ffb50069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.814+0000 7ffb74daa640 1 --2- 172.21.15.116:0/1884896888 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ffb500678c0 0x7ffb50069d80 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ffb7004fa30 tx=0x7ffb70072040 comp rx=0 tx=0).stop 2024-04-25T00:01:26.818 INFO:teuthology.orchestra.run.smithi116.stdout:55834574857 2024-04-25T00:01:26.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.814+0000 7ffb74daa640 1 -- 172.21.15.116:0/1884896888 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb6800be80 msgr2=0x7ffb6802e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.814+0000 7ffb74daa640 1 --2- 172.21.15.116:0/1884896888 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb6800be80 0x7ffb6802e100 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ffb60009870 tx=0x7ffb6001ad30 comp rx=0 tx=0).stop 2024-04-25T00:01:26.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.814+0000 7ffb74daa640 1 -- 172.21.15.116:0/1884896888 shutdown_connections 2024-04-25T00:01:26.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.818+0000 7ffb74daa640 1 --2- 172.21.15.116:0/1884896888 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ffb500678c0 0x7ffb50069d80 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.818+0000 7ffb74daa640 1 --2- 172.21.15.116:0/1884896888 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffb680d25c0 0x7ffb681771b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.818+0000 7ffb74daa640 1 --2- 172.21.15.116:0/1884896888 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffb68048380 0x7ffb680cae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.818+0000 7ffb74daa640 1 --2- 172.21.15.116:0/1884896888 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb6800be80 0x7ffb6802e100 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.818+0000 7ffb74daa640 1 -- 172.21.15.116:0/1884896888 >> 172.21.15.116:0/1884896888 conn(0x7ffb6802f4f0 msgr2=0x7ffb680d7890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:26.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.818+0000 7ffb74daa640 1 -- 172.21.15.116:0/1884896888 shutdown_connections 2024-04-25T00:01:26.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.818+0000 7ffb74daa640 1 -- 172.21.15.116:0/1884896888 wait complete. 2024-04-25T00:01:26.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.818+0000 7f3235981640 1 -- 172.21.15.116:0/1619660765 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f321c0678f0 msgr2=0x7f321c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.818+0000 7f3235981640 1 --2- 172.21.15.116:0/1619660765 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f321c0678f0 0x7f321c069db0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3220006aa0 tx=0x7f3220006a30 comp rx=0 tx=0).stop 2024-04-25T00:01:26.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.818+0000 7f454173d640 1 -- 172.21.15.116:0/3245013540 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4520067940 msgr2=0x7f4520069e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.818+0000 7f454173d640 1 --2- 172.21.15.116:0/3245013540 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4520067940 0x7f4520069e00 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4530001770 tx=0x7f4530009040 comp rx=0 tx=0).stop 2024-04-25T00:01:26.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.818+0000 7f454173d640 1 -- 172.21.15.116:0/3245013540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45340d7a50 msgr2=0x7f4534033e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.818+0000 7f454173d640 1 --2- 172.21.15.116:0/3245013540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45340d7a50 0x7f4534033e10 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f453c06a1e0 tx=0x7f453c04f740 comp rx=0 tx=0).stop 2024-04-25T00:01:26.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.818+0000 7f454173d640 1 -- 172.21.15.116:0/3245013540 shutdown_connections 2024-04-25T00:01:26.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.818+0000 7f454173d640 1 --2- 172.21.15.116:0/3245013540 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4520067940 0x7f4520069e00 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.818+0000 7f454173d640 1 --2- 172.21.15.116:0/3245013540 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45340d7a50 0x7f4534033e10 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.818+0000 7f454173d640 1 --2- 172.21.15.116:0/3245013540 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f453400f8d0 0x7f4534177000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.818+0000 7f454173d640 1 --2- 172.21.15.116:0/3245013540 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f453400ea20 0x7f45341730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.818+0000 7f454173d640 1 -- 172.21.15.116:0/3245013540 >> 172.21.15.116:0/3245013540 conn(0x7f4534031740 msgr2=0x7f4534030f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:26.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.818+0000 7f3235981640 1 -- 172.21.15.116:0/1619660765 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f32301562a0 msgr2=0x7f32300a9a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.818+0000 7f3235981640 1 --2- 172.21.15.116:0/1619660765 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f32301562a0 0x7f32300a9a40 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f32280af040 tx=0x7f32280196d0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.818+0000 7f454173d640 1 -- 172.21.15.116:0/3245013540 shutdown_connections 2024-04-25T00:01:26.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.818+0000 7f454173d640 1 -- 172.21.15.116:0/3245013540 wait complete. 2024-04-25T00:01:26.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.818+0000 7f3235981640 1 -- 172.21.15.116:0/1619660765 shutdown_connections 2024-04-25T00:01:26.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.818+0000 7f3235981640 1 --2- 172.21.15.116:0/1619660765 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f321c0678f0 0x7f321c069db0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.818+0000 7f3235981640 1 --2- 172.21.15.116:0/1619660765 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f32301562a0 0x7f32300a9a40 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.818+0000 7f3235981640 1 --2- 172.21.15.116:0/1619660765 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3230152d80 0x7f32301fc7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.818+0000 7f3235981640 1 --2- 172.21.15.116:0/1619660765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3230072080 0x7f3230202900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.818+0000 7f3235981640 1 -- 172.21.15.116:0/1619660765 >> 172.21.15.116:0/1619660765 conn(0x7f323006c720 msgr2=0x7f3230157ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:26.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.818+0000 7f3235981640 1 -- 172.21.15.116:0/1619660765 shutdown_connections 2024-04-25T00:01:26.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.818+0000 7f3235981640 1 -- 172.21.15.116:0/1619660765 wait complete. 2024-04-25T00:01:26.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.822+0000 7f30e1182640 1 -- 172.21.15.116:0/2423118928 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f30d40053a0 con 0x7f30d400b0c0 2024-04-25T00:01:26.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.822+0000 7f30d8ff9640 1 -- 172.21.15.116:0/2423118928 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f30d0055a60 con 0x7f30d400b0c0 2024-04-25T00:01:26.826 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509450 got 64424509449 for osd.0 2024-04-25T00:01:26.826 INFO:teuthology.orchestra.run.smithi116.stdout:64424509449 2024-04-25T00:01:26.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.822+0000 7f30e1182640 1 -- 172.21.15.116:0/2423118928 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f30c00678c0 msgr2=0x7f30c0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.822+0000 7f30e1182640 1 --2- 172.21.15.116:0/2423118928 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f30c00678c0 0x7f30c0069d80 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f30c80af000 tx=0x7f30c80a7000 comp rx=0 tx=0).stop 2024-04-25T00:01:26.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.822+0000 7f30e1182640 1 -- 172.21.15.116:0/2423118928 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30d400b0c0 msgr2=0x7f30d4043710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.822+0000 7f30e1182640 1 --2- 172.21.15.116:0/2423118928 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30d400b0c0 0x7f30d4043710 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f30d000c450 tx=0x7f30d000c850 comp rx=0 tx=0).stop 2024-04-25T00:01:26.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.826+0000 7f30e1182640 1 -- 172.21.15.116:0/2423118928 shutdown_connections 2024-04-25T00:01:26.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.826+0000 7f30e1182640 1 --2- 172.21.15.116:0/2423118928 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f30c00678c0 0x7f30c0069d80 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.826+0000 7f30e1182640 1 --2- 172.21.15.116:0/2423118928 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f30d40ded20 0x7f30d40dabb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.826+0000 7f30e1182640 1 --2- 172.21.15.116:0/2423118928 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f30d40dbce0 0x7f30d40da670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.826+0000 7f30e1182640 1 --2- 172.21.15.116:0/2423118928 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f30d400b0c0 0x7f30d4043710 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.826+0000 7f30e1182640 1 -- 172.21.15.116:0/2423118928 >> 172.21.15.116:0/2423118928 conn(0x7f30d4048260 msgr2=0x7f30d40de040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:26.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.826+0000 7f30e1182640 1 -- 172.21.15.116:0/2423118928 shutdown_connections 2024-04-25T00:01:26.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.826+0000 7f30e1182640 1 -- 172.21.15.116:0/2423118928 wait complete. 2024-04-25T00:01:26.830 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509450 got 64424509449 for osd.1 2024-04-25T00:01:26.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.834+0000 7f6c24d2f640 1 -- 172.21.15.116:0/550517207 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f6c20068a60 con 0x7f6c200aacd0 2024-04-25T00:01:26.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.834+0000 7f6c0effd640 1 -- 172.21.15.116:0/550517207 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f6c14020020 con 0x7f6c200aacd0 2024-04-25T00:01:26.835 INFO:teuthology.orchestra.run.smithi116.stdout:64424509449 2024-04-25T00:01:26.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.838+0000 7f6c24d2f640 1 -- 172.21.15.116:0/550517207 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6c000678f0 msgr2=0x7f6c00069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.838+0000 7f6c24d2f640 1 --2- 172.21.15.116:0/550517207 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6c000678f0 0x7f6c00069db0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6c100ad000 tx=0x7f6c100a6000 comp rx=0 tx=0).stop 2024-04-25T00:01:26.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.838+0000 7f6c24d2f640 1 -- 172.21.15.116:0/550517207 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c200aacd0 msgr2=0x7f6c20149bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:26.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.838+0000 7f6c24d2f640 1 --2- 172.21.15.116:0/550517207 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c200aacd0 0x7f6c20149bc0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f6c1400c530 tx=0x7f6c1400c9f0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.838+0000 7f6c24d2f640 1 -- 172.21.15.116:0/550517207 shutdown_connections 2024-04-25T00:01:26.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.838+0000 7f6c24d2f640 1 --2- 172.21.15.116:0/550517207 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6c000678f0 0x7f6c00069db0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.838+0000 7f6c24d2f640 1 --2- 172.21.15.116:0/550517207 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c20155b20 0x7f6c2015dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.838+0000 7f6c24d2f640 1 --2- 172.21.15.116:0/550517207 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c2014c700 0x7f6c2014a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.838+0000 7f6c24d2f640 1 --2- 172.21.15.116:0/550517207 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c200aacd0 0x7f6c20149bc0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:26.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.838+0000 7f6c24d2f640 1 -- 172.21.15.116:0/550517207 >> 172.21.15.116:0/550517207 conn(0x7f6c2006c720 msgr2=0x7f6c200c5090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:26.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.838+0000 7f6c24d2f640 1 -- 172.21.15.116:0/550517207 shutdown_connections 2024-04-25T00:01:26.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:26.838+0000 7f6c24d2f640 1 -- 172.21.15.116:0/550517207 wait complete. 2024-04-25T00:01:26.845 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574857 for osd.4 2024-04-25T00:01:26.846 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509451 got 64424509450 for osd.2 2024-04-25T00:01:26.850 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509450 got 64424509449 for osd.6 2024-04-25T00:01:26.854 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509450 got 64424509449 for osd.7 2024-04-25T00:01:26.861 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509450 got 64424509449 for osd.3 2024-04-25T00:01:27.808 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-25T00:01:27.826 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-25T00:01:27.831 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-25T00:01:27.846 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-25T00:01:27.846 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-25T00:01:27.851 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-25T00:01:27.855 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-25T00:01:27.862 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-25T00:01:27.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:27.982+0000 7f45910e1640 1 Processor -- start 2024-04-25T00:01:27.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:27.982+0000 7f45910e1640 1 -- start start 2024-04-25T00:01:27.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:27.982+0000 7f45910e1640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f458c13acb0 0x7f458c071180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:27.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:27.982+0000 7f45910e1640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f458c073320 con 0x7f458c06cb70 2024-04-25T00:01:27.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:27.982+0000 7f45910e1640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f458c073040 con 0x7f458c13a360 2024-04-25T00:01:27.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:27.982+0000 7f45910e1640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f458c070cf0 con 0x7f458c13acb0 2024-04-25T00:01:27.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:27.982+0000 7f458b59e640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f458c06cb70 0x7f458c0716c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:41690/0 (socket says 172.21.15.116:41690) 2024-04-25T00:01:27.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:27.982+0000 7f458b59e640 1 -- 172.21.15.116:0/3568345711 learned_addr learned my addr 172.21.15.116:0/3568345711 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:27.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:27.986+0000 7f458a59c640 1 --2- 172.21.15.116:0/3568345711 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f458c13acb0 0x7f458c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:27.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:27.986+0000 7f458a59c640 1 -- 172.21.15.116:0/3568345711 >> v1:172.21.15.174:6789/0 conn(0x7f458c13a360 legacy=0x7f458c0c29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:27.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:27.986+0000 7f458a59c640 1 -- 172.21.15.116:0/3568345711 >> v1:172.21.15.116:6789/0 conn(0x7f458c06cb70 legacy=0x7f458c0716c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:27.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:27.986+0000 7f458a59c640 1 -- 172.21.15.116:0/3568345711 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f458c1557e0 con 0x7f458c13acb0 2024-04-25T00:01:27.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:27.990+0000 7f458a59c640 1 --2- 172.21.15.116:0/3568345711 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f458c13acb0 0x7f458c071180 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f45800087f0 tx=0x7f458001e410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=740334172c99e86 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:27.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:27.990+0000 7f4589d9b640 1 -- 172.21.15.116:0/3568345711 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45800ac070 con 0x7f458c13acb0 2024-04-25T00:01:27.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:27.994+0000 7f4589d9b640 1 -- 172.21.15.116:0/3568345711 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45800a6a30 con 0x7f458c13acb0 2024-04-25T00:01:27.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:27.994+0000 7f4589d9b640 1 -- 172.21.15.116:0/3568345711 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45800a6e00 con 0x7f458c13acb0 2024-04-25T00:01:27.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:27.994+0000 7f45910e1640 1 -- 172.21.15.116:0/3568345711 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f458c13acb0 msgr2=0x7f458c071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:27.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:27.994+0000 7f45910e1640 1 --2- 172.21.15.116:0/3568345711 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f458c13acb0 0x7f458c071180 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f45800087f0 tx=0x7f458001e410 comp rx=0 tx=0).stop 2024-04-25T00:01:27.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:27.994+0000 7f45910e1640 1 -- 172.21.15.116:0/3568345711 shutdown_connections 2024-04-25T00:01:27.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:27.994+0000 7f45910e1640 1 --2- 172.21.15.116:0/3568345711 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f458c13acb0 0x7f458c071180 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:27.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:27.994+0000 7f45910e1640 1 -- 172.21.15.116:0/3568345711 >> 172.21.15.116:0/3568345711 conn(0x7f458c0c3a90 msgr2=0x7f458c0c3ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:27.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:27.994+0000 7f45910e1640 1 -- 172.21.15.116:0/3568345711 shutdown_connections 2024-04-25T00:01:27.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:27.994+0000 7f45910e1640 1 -- 172.21.15.116:0/3568345711 wait complete. 2024-04-25T00:01:27.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:27.994+0000 7f45910e1640 1 Processor -- start 2024-04-25T00:01:27.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:27.994+0000 7f45910e1640 1 -- start start 2024-04-25T00:01:27.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:27.998+0000 7f45910e1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f458c06cb70 0x7f458c1eddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:27.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:27.998+0000 7f45910e1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f458c13a360 0x7f458c133790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:27.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:27.998+0000 7f45910e1640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f458c13acb0 0x7f458c133cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:27.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:27.998+0000 7f45910e1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f458c13a8c0 con 0x7f458c06cb70 2024-04-25T00:01:27.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:27.998+0000 7f45910e1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f458c13a740 con 0x7f458c13a360 2024-04-25T00:01:27.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:27.998+0000 7f45910e1640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f458c15f8f0 con 0x7f458c13acb0 2024-04-25T00:01:27.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:27.998+0000 7f458ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f458c06cb70 0x7f458c1eddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:27.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:27.998+0000 7f458ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f458c06cb70 0x7f458c1eddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56270/0 (socket says 172.21.15.116:56270) 2024-04-25T00:01:27.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:27.998+0000 7f458ad9d640 1 -- 172.21.15.116:0/2951034814 learned_addr learned my addr 172.21.15.116:0/2951034814 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:28.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:27.998+0000 7f458a59c640 1 --2- 172.21.15.116:0/2951034814 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f458c13a360 0x7f458c133790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:28.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:27.998+0000 7f458b59e640 1 --2- 172.21.15.116:0/2951034814 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f458c13acb0 0x7f458c133cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:28.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:27.998+0000 7f458ad9d640 1 -- 172.21.15.116:0/2951034814 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f458c13acb0 msgr2=0x7f458c133cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:28.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:27.998+0000 7f458ad9d640 1 --2- 172.21.15.116:0/2951034814 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f458c13acb0 0x7f458c133cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:27.998+0000 7f458ad9d640 1 -- 172.21.15.116:0/2951034814 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f458c13a360 msgr2=0x7f458c133790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:28.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:27.998+0000 7f458ad9d640 1 --2- 172.21.15.116:0/2951034814 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f458c13a360 0x7f458c133790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:27.998+0000 7f458ad9d640 1 -- 172.21.15.116:0/2951034814 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f458c154b70 con 0x7f458c06cb70 2024-04-25T00:01:28.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:27.998+0000 7f458ad9d640 1 --2- 172.21.15.116:0/2951034814 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f458c06cb70 0x7f458c1eddf0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f457c007d40 tx=0x7f457c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:28.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:27.998+0000 7f45737fe640 1 -- 172.21.15.116:0/2951034814 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f457c017070 con 0x7f458c06cb70 2024-04-25T00:01:28.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:27.998+0000 7f45737fe640 1 -- 172.21.15.116:0/2951034814 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f457c005070 con 0x7f458c06cb70 2024-04-25T00:01:28.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:27.998+0000 7f45910e1640 1 -- 172.21.15.116:0/2951034814 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f458c153af0 con 0x7f458c06cb70 2024-04-25T00:01:28.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:27.998+0000 7f45737fe640 1 -- 172.21.15.116:0/2951034814 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f457c005460 con 0x7f458c06cb70 2024-04-25T00:01:28.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:27.998+0000 7f45910e1640 1 -- 172.21.15.116:0/2951034814 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f458c1ee7f0 con 0x7f458c06cb70 2024-04-25T00:01:28.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:27.998+0000 7f45910e1640 1 -- 172.21.15.116:0/2951034814 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f458c04f580 con 0x7f458c06cb70 2024-04-25T00:01:28.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.002+0000 7f45737fe640 1 -- 172.21.15.116:0/2951034814 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f457c01a070 con 0x7f458c06cb70 2024-04-25T00:01:28.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.002+0000 7f45737fe640 1 --2- 172.21.15.116:0/2951034814 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4578067910 0x7f4578069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.002+0000 7f45737fe640 1 -- 172.21.15.116:0/2951034814 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7f457c085fb0 con 0x7f458c06cb70 2024-04-25T00:01:28.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.002+0000 7f458a59c640 1 --2- 172.21.15.116:0/2951034814 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4578067910 0x7f4578069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:28.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.002+0000 7f458a59c640 1 --2- 172.21.15.116:0/2951034814 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4578067910 0x7f4578069dd0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f45800b1000 tx=0x7f458000a040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:28.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.010+0000 7f45737fe640 1 -- 172.21.15.116:0/2951034814 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f457c0b25c0 con 0x7f458c06cb70 2024-04-25T00:01:28.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.022+0000 7f17ac3ac640 1 Processor -- start 2024-04-25T00:01:28.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.022+0000 7f17ac3ac640 1 -- start start 2024-04-25T00:01:28.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.022+0000 7f17ac3ac640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f17a414b830 0x7f17a4152150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.022+0000 7f17ac3ac640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f17a4070dd0 0x7f17a4071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.022+0000 7f17ac3ac640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17a4152690 con 0x7f17a406e100 2024-04-25T00:01:28.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.022+0000 7f17ac3ac640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f17a414bc10 con 0x7f17a414b830 2024-04-25T00:01:28.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.022+0000 7f17ac3ac640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f17a4154df0 con 0x7f17a4070dd0 2024-04-25T00:01:28.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.022+0000 7f17a9947640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f17a406e100 0x7f17a40c2a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:41696/0 (socket says 172.21.15.116:41696) 2024-04-25T00:01:28.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.022+0000 7f17a9947640 1 -- 172.21.15.116:0/3001838501 learned_addr learned my addr 172.21.15.116:0/3001838501 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:28.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.022+0000 7f17aa148640 1 --2- 172.21.15.116:0/3001838501 >> v2:172.21.15.116:3300/0 conn(0x7f17a414b830 0x7f17a4152150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:28.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.022+0000 7f17aa148640 1 -- 172.21.15.116:0/3001838501 >> v2:172.21.15.174:3300/0 conn(0x7f17a4070dd0 msgr2=0x7f17a4071260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T00:01:28.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.022+0000 7f17aa148640 1 --2- 172.21.15.116:0/3001838501 >> v2:172.21.15.174:3300/0 conn(0x7f17a4070dd0 0x7f17a4071260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.022+0000 7f17aa148640 1 -- 172.21.15.116:0/3001838501 >> v1:172.21.15.116:6789/0 conn(0x7f17a406e100 legacy=0x7f17a40c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:28.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.022+0000 7f17aa148640 1 -- 172.21.15.116:0/3001838501 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17a41550d0 con 0x7f17a414b830 2024-04-25T00:01:28.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.026+0000 7f17aa148640 1 --2- 172.21.15.116:0/3001838501 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17a414b830 0x7f17a4152150 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f17a4073260 tx=0x7f17a0020510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf8041aa9cb0a01e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:28.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.026+0000 7f17a9146640 1 -- 172.21.15.116:0/3001838501 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17a00ae070 con 0x7f17a414b830 2024-04-25T00:01:28.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.026+0000 7f17a9146640 1 -- 172.21.15.116:0/3001838501 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17a00a8ae0 con 0x7f17a414b830 2024-04-25T00:01:28.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.026+0000 7f17a9146640 1 -- 172.21.15.116:0/3001838501 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17a00b2400 con 0x7f17a414b830 2024-04-25T00:01:28.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.026+0000 7f17ac3ac640 1 -- 172.21.15.116:0/3001838501 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17a414b830 msgr2=0x7f17a4152150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:28.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.026+0000 7f17ac3ac640 1 --2- 172.21.15.116:0/3001838501 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17a414b830 0x7f17a4152150 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f17a4073260 tx=0x7f17a0020510 comp rx=0 tx=0).stop 2024-04-25T00:01:28.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.030+0000 7f17ac3ac640 1 -- 172.21.15.116:0/3001838501 shutdown_connections 2024-04-25T00:01:28.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.030+0000 7f17ac3ac640 1 --2- 172.21.15.116:0/3001838501 >> v2:172.21.15.174:3300/0 conn(0x7f17a4070dd0 0x7f17a4071260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.030+0000 7f17ac3ac640 1 --2- 172.21.15.116:0/3001838501 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17a414b830 0x7f17a4152150 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.030+0000 7f17ac3ac640 1 -- 172.21.15.116:0/3001838501 >> 172.21.15.116:0/3001838501 conn(0x7f17a40c3b00 msgr2=0x7f17a40c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:28.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.030+0000 7f17ac3ac640 1 -- 172.21.15.116:0/3001838501 shutdown_connections 2024-04-25T00:01:28.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.030+0000 7fbec65b8640 1 Processor -- start 2024-04-25T00:01:28.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.030+0000 7f17ac3ac640 1 -- 172.21.15.116:0/3001838501 wait complete. 2024-04-25T00:01:28.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.030+0000 7fbec65b8640 1 -- start start 2024-04-25T00:01:28.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.030+0000 7fbec65b8640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbec00c4300 0x7fbec00aa170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.030+0000 7f17ac3ac640 1 Processor -- start 2024-04-25T00:01:28.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.030+0000 7fbec65b8640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbec01526a0 con 0x7fbec00c5520 2024-04-25T00:01:28.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.030+0000 7fbec65b8640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbec0152980 con 0x7fbec00c5100 2024-04-25T00:01:28.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.030+0000 7fbec4b55640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fbec00c5520 0x7fbec015ce80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:41706/0 (socket says 172.21.15.116:41706) 2024-04-25T00:01:28.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.030+0000 7fbec4b55640 1 -- 172.21.15.116:0/1846214273 learned_addr learned my addr 172.21.15.116:0/1846214273 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:28.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.030+0000 7fbec65b8640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fbec0153460 con 0x7fbec00c4300 2024-04-25T00:01:28.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.030+0000 7fbebffff640 1 --2- 172.21.15.116:0/1846214273 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbec00c4300 0x7fbec00aa170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:28.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.030+0000 7f17ac3ac640 1 -- start start 2024-04-25T00:01:28.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.030+0000 7fbebeffd640 1 -- 172.21.15.116:0/1846214273 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1979948673 0 0) 0x7fbec01526a0 con 0x7fbec00c5520 2024-04-25T00:01:28.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.030+0000 7fbebeffd640 1 -- 172.21.15.116:0/1846214273 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbea8005740 con 0x7fbec00c5520 2024-04-25T00:01:28.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.030+0000 7f17ac3ac640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f17a406e100 0x7f17a41f2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.030+0000 7f17ac3ac640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17a4070dd0 0x7f17a41f2640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.030+0000 7f17ac3ac640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f17a414b830 0x7f17a414c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.030+0000 7f17ac3ac640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f17a415f010 con 0x7f17a4070dd0 2024-04-25T00:01:28.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.030+0000 7f17ac3ac640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f17a415ee90 con 0x7f17a414b830 2024-04-25T00:01:28.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.030+0000 7f17ac3ac640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f17a415f190 con 0x7f17a406e100 2024-04-25T00:01:28.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.030+0000 7f17a9947640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17a4070dd0 0x7f17a41f2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:28.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.030+0000 7f17a9947640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17a4070dd0 0x7f17a41f2640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56296/0 (socket says 172.21.15.116:56296) 2024-04-25T00:01:28.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.030+0000 7f17a9947640 1 -- 172.21.15.116:0/2607796615 learned_addr learned my addr 172.21.15.116:0/2607796615 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:28.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.030+0000 7f17aa148640 1 --2- 172.21.15.116:0/2607796615 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f17a406e100 0x7f17a41f2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:28.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.030+0000 7f17aa949640 1 --2- 172.21.15.116:0/2607796615 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f17a414b830 0x7f17a414c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:28.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.034+0000 7f17a9947640 1 -- 172.21.15.116:0/2607796615 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f17a406e100 msgr2=0x7f17a41f2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:28.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.034+0000 7f17a9947640 1 --2- 172.21.15.116:0/2607796615 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f17a406e100 0x7f17a41f2100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.034+0000 7f17a9947640 1 -- 172.21.15.116:0/2607796615 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f17a414b830 msgr2=0x7f17a414c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:28.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.034+0000 7f17a9947640 1 --2- 172.21.15.116:0/2607796615 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f17a414b830 0x7f17a414c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.034+0000 7f17a9947640 1 -- 172.21.15.116:0/2607796615 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17a41f2b80 con 0x7f17a4070dd0 2024-04-25T00:01:28.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.034+0000 7f17a9947640 1 --2- 172.21.15.116:0/2607796615 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17a4070dd0 0x7f17a41f2640 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f179c00ce70 tx=0x7f179c00a630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:28.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.034+0000 7f179affd640 1 -- 172.21.15.116:0/2607796615 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f179c00ae00 con 0x7f17a4070dd0 2024-04-25T00:01:28.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.034+0000 7f179affd640 1 -- 172.21.15.116:0/2607796615 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f179c00f670 con 0x7f17a4070dd0 2024-04-25T00:01:28.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.034+0000 7f179affd640 1 -- 172.21.15.116:0/2607796615 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f179c00fa60 con 0x7f17a4070dd0 2024-04-25T00:01:28.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.034+0000 7f17ac3ac640 1 -- 172.21.15.116:0/2607796615 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17a414ec20 con 0x7f17a4070dd0 2024-04-25T00:01:28.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.034+0000 7f17ac3ac640 1 -- 172.21.15.116:0/2607796615 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17a414dc50 con 0x7f17a4070dd0 2024-04-25T00:01:28.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.030+0000 7fbebffff640 1 -- 172.21.15.116:0/1846214273 >> v1:172.21.15.174:6789/0 conn(0x7fbec00c5100 legacy=0x7fbec0155af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:28.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.030+0000 7fbebffff640 1 -- 172.21.15.116:0/1846214273 >> v1:172.21.15.116:6789/0 conn(0x7fbec00c5520 legacy=0x7fbec015ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:28.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.030+0000 7fbebffff640 1 -- 172.21.15.116:0/1846214273 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbec0152c60 con 0x7fbec00c4300 2024-04-25T00:01:28.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.034+0000 7fbebffff640 1 --2- 172.21.15.116:0/1846214273 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbec00c4300 0x7fbec00aa170 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fbeb00097f0 tx=0x7fbeb001c500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9acf88e4f3193a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:28.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.034+0000 7fbebeffd640 1 -- 172.21.15.116:0/1846214273 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeb00b0070 con 0x7fbec00c4300 2024-04-25T00:01:28.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.034+0000 7fbebeffd640 1 -- 172.21.15.116:0/1846214273 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbeb00ab600 con 0x7fbec00c4300 2024-04-25T00:01:28.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.034+0000 7fbebeffd640 1 -- 172.21.15.116:0/1846214273 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeb00ab9d0 con 0x7fbec00c4300 2024-04-25T00:01:28.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.034+0000 7f179affd640 1 -- 172.21.15.116:0/2607796615 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f179c00fc00 con 0x7f17a4070dd0 2024-04-25T00:01:28.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.034+0000 7f179affd640 1 --2- 172.21.15.116:0/2607796615 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1788067910 0x7f1788069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.034+0000 7f179affd640 1 -- 172.21.15.116:0/2607796615 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7f179c086c50 con 0x7f17a4070dd0 2024-04-25T00:01:28.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.034+0000 7f17aa148640 1 --2- 172.21.15.116:0/2607796615 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1788067910 0x7f1788069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:28.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.038+0000 7fbec65b8640 1 -- 172.21.15.116:0/1846214273 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbec00c4300 msgr2=0x7fbec00aa170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:28.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.038+0000 7fbec65b8640 1 --2- 172.21.15.116:0/1846214273 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbec00c4300 0x7fbec00aa170 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fbeb00097f0 tx=0x7fbeb001c500 comp rx=0 tx=0).stop 2024-04-25T00:01:28.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.038+0000 7fbec65b8640 1 -- 172.21.15.116:0/1846214273 shutdown_connections 2024-04-25T00:01:28.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.038+0000 7fbec65b8640 1 --2- 172.21.15.116:0/1846214273 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbec00c4300 0x7fbec00aa170 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.038+0000 7fbec65b8640 1 -- 172.21.15.116:0/1846214273 >> 172.21.15.116:0/1846214273 conn(0x7fbec006c720 msgr2=0x7fbec006cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:28.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.038+0000 7fbec65b8640 1 -- 172.21.15.116:0/1846214273 shutdown_connections 2024-04-25T00:01:28.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.038+0000 7f17aa148640 1 --2- 172.21.15.116:0/2607796615 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1788067910 0x7f1788069dd0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f17a00b3000 tx=0x7f17a00aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:28.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.038+0000 7fbec65b8640 1 -- 172.21.15.116:0/1846214273 wait complete. 2024-04-25T00:01:28.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.038+0000 7fbec65b8640 1 Processor -- start 2024-04-25T00:01:28.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.038+0000 7fbec65b8640 1 -- start start 2024-04-25T00:01:28.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.038+0000 7f17ac3ac640 1 -- 172.21.15.116:0/2607796615 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17a4068350 con 0x7f17a4070dd0 2024-04-25T00:01:28.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.038+0000 7fcedf59e640 1 Processor -- start 2024-04-25T00:01:28.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.038+0000 7fcedf59e640 1 -- start start 2024-04-25T00:01:28.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.038+0000 7fbec65b8640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbec00c4300 0x7fbec00ac400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.038+0000 7fbec65b8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbec00c5100 0x7fbec01fee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.038+0000 7fbec65b8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbec00c5520 0x7fbec01ff380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.038+0000 7fbec65b8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbec006ffd0 con 0x7fbec00c5100 2024-04-25T00:01:28.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.038+0000 7fbec65b8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbec00aaee0 con 0x7fbec00c5520 2024-04-25T00:01:28.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.038+0000 7fbec65b8640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fbec00c5930 con 0x7fbec00c4300 2024-04-25T00:01:28.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.038+0000 7fbec4b55640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbec00c5520 0x7fbec01ff380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:28.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.038+0000 7fbec4b55640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbec00c5520 0x7fbec01ff380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:37386/0 (socket says 172.21.15.116:37386) 2024-04-25T00:01:28.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.038+0000 7fbec4b55640 1 -- 172.21.15.116:0/171848925 learned_addr learned my addr 172.21.15.116:0/171848925 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:28.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.038+0000 7fbebffff640 1 --2- 172.21.15.116:0/171848925 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbec00c4300 0x7fbec00ac400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:28.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.038+0000 7fe8d169d640 1 Processor -- start 2024-04-25T00:01:28.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.038+0000 7fbebf7fe640 1 --2- 172.21.15.116:0/171848925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbec00c5100 0x7fbec01fee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:28.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.038+0000 7fbec4b55640 1 -- 172.21.15.116:0/171848925 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbec00c4300 msgr2=0x7fbec00ac400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:28.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.038+0000 7fbec4b55640 1 --2- 172.21.15.116:0/171848925 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbec00c4300 0x7fbec00ac400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.038+0000 7fbec4b55640 1 -- 172.21.15.116:0/171848925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbec00c5100 msgr2=0x7fbec01fee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:28.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.038+0000 7fbec4b55640 1 --2- 172.21.15.116:0/171848925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbec00c5100 0x7fbec01fee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.038+0000 7fbec4b55640 1 -- 172.21.15.116:0/171848925 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbec0163750 con 0x7fbec00c5520 2024-04-25T00:01:28.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.042+0000 7fbebf7fe640 1 --2- 172.21.15.116:0/171848925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbec00c5100 0x7fbec01fee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:01:28.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.042+0000 7fcedf59e640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fced80d4360 0x7fced80071c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.042+0000 7fcedf59e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fced8007700 0x7fced800ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.042+0000 7fceddd9b640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fced804daf0 0x7fced800e600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:41708/0 (socket says 172.21.15.116:41708) 2024-04-25T00:01:28.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.042+0000 7fceddd9b640 1 -- 172.21.15.116:0/3328312672 learned_addr learned my addr 172.21.15.116:0/3328312672 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:28.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.042+0000 7fbec4b55640 1 --2- 172.21.15.116:0/171848925 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbec00c5520 0x7fbec01ff380 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fbeb800caa0 tx=0x7fbeb800cf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:28.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.042+0000 7fceded9d640 1 --2- 172.21.15.116:0/3328312672 >> v2:172.21.15.116:3300/0 conn(0x7fced8007700 0x7fced800ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:28.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.042+0000 7fcedf59e640 1 -- 172.21.15.116:0/3328312672 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fced804a6f0 con 0x7fced804daf0 2024-04-25T00:01:28.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.042+0000 7fcedf59e640 1 -- 172.21.15.116:0/3328312672 --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fced800e170 con 0x7fced8007700 2024-04-25T00:01:28.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.042+0000 7fcedf59e640 1 -- 172.21.15.116:0/3328312672 --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fced804a9d0 con 0x7fced80d4360 2024-04-25T00:01:28.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.038+0000 7fe8d169d640 1 -- start start 2024-04-25T00:01:28.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.042+0000 7fe8d169d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe8c40462f0 0x7fe8c40070d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.042+0000 7fe8d169d640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fe8c40496e0 0x7fe8c4049b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.042+0000 7fe8d169d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8c4046820 con 0x7fe8c40d3ab0 2024-04-25T00:01:28.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.042+0000 7fe8d169d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fe8c4046fb0 con 0x7fe8c40462f0 2024-04-25T00:01:28.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.042+0000 7fe8d169d640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fe8c4007720 con 0x7fe8c40496e0 2024-04-25T00:01:28.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.042+0000 7fe8cbfff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fe8c40d3ab0 0x7fe8c40069c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:41722/0 (socket says 172.21.15.116:41722) 2024-04-25T00:01:28.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.042+0000 7fe8cbfff640 1 -- 172.21.15.116:0/1556496434 learned_addr learned my addr 172.21.15.116:0/1556496434 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:28.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.042+0000 7fcede59c640 1 --2- 172.21.15.116:0/3328312672 >> v2:172.21.15.174:3300/0 conn(0x7fced80d4360 0x7fced80071c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:28.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.042+0000 7fceded9d640 1 -- 172.21.15.116:0/3328312672 >> v2:172.21.15.174:3300/0 conn(0x7fced80d4360 msgr2=0x7fced80071c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:28.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.042+0000 7fceded9d640 1 --2- 172.21.15.116:0/3328312672 >> v2:172.21.15.174:3300/0 conn(0x7fced80d4360 0x7fced80071c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.042+0000 7fceded9d640 1 -- 172.21.15.116:0/3328312672 >> v1:172.21.15.116:6789/0 conn(0x7fced804daf0 legacy=0x7fced800e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:28.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.042+0000 7fceded9d640 1 -- 172.21.15.116:0/3328312672 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fced804acb0 con 0x7fced8007700 2024-04-25T00:01:28.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.042+0000 7fcede59c640 1 --2- 172.21.15.116:0/3328312672 >> v2:172.21.15.174:3300/0 conn(0x7fced80d4360 0x7fced80071c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:28.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.042+0000 7fe8d0e9c640 1 --2- 172.21.15.116:0/1556496434 >> v2:172.21.15.174:3300/0 conn(0x7fe8c40496e0 0x7fe8c4049b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:28.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.042+0000 7fe8cb7fe640 1 --2- 172.21.15.116:0/1556496434 >> v2:172.21.15.116:3300/0 conn(0x7fe8c40462f0 0x7fe8c40070d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:28.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.042+0000 7fe8cb7fe640 1 -- 172.21.15.116:0/1556496434 >> v2:172.21.15.174:3300/0 conn(0x7fe8c40496e0 msgr2=0x7fe8c4049b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:28.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.042+0000 7fbebcff9640 1 -- 172.21.15.116:0/171848925 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeb8017070 con 0x7fbec00c5520 2024-04-25T00:01:28.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.042+0000 7fbebcff9640 1 -- 172.21.15.116:0/171848925 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbeb800ac00 con 0x7fbec00c5520 2024-04-25T00:01:28.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.042+0000 7fbebcff9640 1 -- 172.21.15.116:0/171848925 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeb8012400 con 0x7fbec00c5520 2024-04-25T00:01:28.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.042+0000 7fbec65b8640 1 -- 172.21.15.116:0/171848925 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbec01639f0 con 0x7fbec00c5520 2024-04-25T00:01:28.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.042+0000 7fbec65b8640 1 -- 172.21.15.116:0/171848925 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbec0163fb0 con 0x7fbec00c5520 2024-04-25T00:01:28.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.042+0000 7fe8cb7fe640 1 --2- 172.21.15.116:0/1556496434 >> v2:172.21.15.174:3300/0 conn(0x7fe8c40496e0 0x7fe8c4049b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.042+0000 7fe8cb7fe640 1 -- 172.21.15.116:0/1556496434 >> v1:172.21.15.116:6789/0 conn(0x7fe8c40d3ab0 legacy=0x7fe8c40069c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:28.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.042+0000 7fe8cb7fe640 1 -- 172.21.15.116:0/1556496434 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8c40d7040 con 0x7fe8c40462f0 2024-04-25T00:01:28.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.042+0000 7fbec65b8640 1 -- 172.21.15.116:0/171848925 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbec0068a60 con 0x7fbec00c5520 2024-04-25T00:01:28.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.046+0000 7fceded9d640 1 --2- 172.21.15.116:0/3328312672 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fced8007700 0x7fced800ed10 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fcee002d710 tx=0x7fcee0097f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc490f7fd010cfeb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:28.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.046+0000 7fbebcff9640 1 -- 172.21.15.116:0/171848925 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbeb801a070 con 0x7fbec00c5520 2024-04-25T00:01:28.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.046+0000 7fbebcff9640 1 --2- 172.21.15.116:0/171848925 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbeac067990 0x7fbeac069e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.046+0000 7fbebcff9640 1 -- 172.21.15.116:0/171848925 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7fbeb8086170 con 0x7fbec00c5520 2024-04-25T00:01:28.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.046+0000 7fbebffff640 1 --2- 172.21.15.116:0/171848925 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbeac067990 0x7fbeac069e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:28.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.046+0000 7fe8cb7fe640 1 --2- 172.21.15.116:0/1556496434 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8c40462f0 0x7fe8c40070d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe8c4046780 tx=0x7fe8b801c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5025afd3f3dfa817 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:28.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.046+0000 7f179affd640 1 -- 172.21.15.116:0/2607796615 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f17a41f2b80 con 0x7f17a4070dd0 2024-04-25T00:01:28.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.046+0000 7fcedd59a640 1 -- 172.21.15.116:0/3328312672 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcee0120600 con 0x7fced8007700 2024-04-25T00:01:28.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.046+0000 7fbebffff640 1 --2- 172.21.15.116:0/171848925 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbeac067990 0x7fbeac069e50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbeb00ad000 tx=0x7fbeb00a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:28.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.046+0000 7f82d7fff640 1 Processor -- start 2024-04-25T00:01:28.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.046+0000 7f82d7fff640 1 -- start start 2024-04-25T00:01:28.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.046+0000 7f82d7fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f82d80c2ab0 0x7f82d815f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.046+0000 7f82d7fff640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82d81629a0 con 0x7f82d8165ac0 2024-04-25T00:01:28.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.046+0000 7f82d7fff640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82d8162c80 con 0x7f82d815fba0 2024-04-25T00:01:28.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.046+0000 7f82d7fff640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f82d815fff0 con 0x7f82d80c2ab0 2024-04-25T00:01:28.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.050+0000 7f82cffff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f82d8165ac0 0x7f82d8070ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:41724/0 (socket says 172.21.15.116:41724) 2024-04-25T00:01:28.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.050+0000 7f82cffff640 1 -- 172.21.15.116:0/91958015 learned_addr learned my addr 172.21.15.116:0/91958015 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:28.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.050+0000 7fcedd59a640 1 -- 172.21.15.116:0/3328312672 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcee01207a0 con 0x7fced8007700 2024-04-25T00:01:28.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.050+0000 7fe8caffd640 1 -- 172.21.15.116:0/1556496434 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b80b0070 con 0x7fe8c40462f0 2024-04-25T00:01:28.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.050+0000 7f82d67fc640 1 -- 172.21.15.116:0/91958015 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1065935521 0 0) 0x7f82d81629a0 con 0x7f82d8165ac0 2024-04-25T00:01:28.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.050+0000 7f82d67fc640 1 -- 172.21.15.116:0/91958015 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f82c4003530 con 0x7f82d8165ac0 2024-04-25T00:01:28.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.050+0000 7fe8caffd640 1 -- 172.21.15.116:0/1556496434 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8b80ab600 con 0x7fe8c40462f0 2024-04-25T00:01:28.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.050+0000 7fe8caffd640 1 -- 172.21.15.116:0/1556496434 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b80ab9d0 con 0x7fe8c40462f0 2024-04-25T00:01:28.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.050+0000 7fcedf59e640 1 -- 172.21.15.116:0/3328312672 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fced8007700 msgr2=0x7fced800ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:28.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.050+0000 7fcedf59e640 1 --2- 172.21.15.116:0/3328312672 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fced8007700 0x7fced800ed10 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fcee002d710 tx=0x7fcee0097f20 comp rx=0 tx=0).stop 2024-04-25T00:01:28.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.050+0000 7f82d67fc640 1 -- 172.21.15.116:0/91958015 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2534779711 0 0) 0x7f82c4003530 con 0x7f82d8165ac0 2024-04-25T00:01:28.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.050+0000 7fcedf59e640 1 -- 172.21.15.116:0/3328312672 shutdown_connections 2024-04-25T00:01:28.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.050+0000 7fcedf59e640 1 --2- 172.21.15.116:0/3328312672 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fced8007700 0x7fced800ed10 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.050+0000 7fcedf59e640 1 --2- 172.21.15.116:0/3328312672 >> v2:172.21.15.174:3300/0 conn(0x7fced80d4360 0x7fced80071c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.050+0000 7fcedf59e640 1 -- 172.21.15.116:0/3328312672 >> 172.21.15.116:0/3328312672 conn(0x7fced804c670 msgr2=0x7fced804ca80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:28.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.050+0000 7f82d67fc640 1 -- 172.21.15.116:0/91958015 >> v2:172.21.15.116:3300/0 conn(0x7f82d80c2ab0 msgr2=0x7f82d815f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:28.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.050+0000 7f82d67fc640 1 --2- 172.21.15.116:0/91958015 >> v2:172.21.15.116:3300/0 conn(0x7f82d80c2ab0 0x7f82d815f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.050+0000 7f82d67fc640 1 -- 172.21.15.116:0/91958015 >> v1:172.21.15.174:6789/0 conn(0x7f82d815fba0 legacy=0x7f82d8071700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:28.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.050+0000 7f82d67fc640 1 -- 172.21.15.116:0/91958015 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82d8162f60 con 0x7f82d8165ac0 2024-04-25T00:01:28.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.050+0000 7f82d67fc640 1 -- 172.21.15.116:0/91958015 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f82c0001320 con 0x7f82d8165ac0 2024-04-25T00:01:28.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.050+0000 7f82d67fc640 1 -- 172.21.15.116:0/91958015 >> v1:172.21.15.116:6789/0 conn(0x7f82d8165ac0 legacy=0x7f82d8070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:28.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.050+0000 7f82d67fc640 1 --2- 172.21.15.116:0/91958015 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82c40039c0 0x7f82c4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.050+0000 7f82d67fc640 1 --2- 172.21.15.116:0/91958015 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f82c4004810 0x7f82c4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.050+0000 7f82d67fc640 1 --2- 172.21.15.116:0/91958015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82c4005380 0x7f82c40079b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.050+0000 7f82d67fc640 1 -- 172.21.15.116:0/91958015 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f82d815fff0 con 0x7f82c4005380 2024-04-25T00:01:28.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.050+0000 7f82d67fc640 1 -- 172.21.15.116:0/91958015 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f82c4008270 con 0x7f82c40039c0 2024-04-25T00:01:28.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.050+0000 7f82d67fc640 1 -- 172.21.15.116:0/91958015 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f82c4008470 con 0x7f82c4004810 2024-04-25T00:01:28.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.050+0000 7fcedf59e640 1 -- 172.21.15.116:0/3328312672 shutdown_connections 2024-04-25T00:01:28.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.050+0000 7fe8d169d640 1 -- 172.21.15.116:0/1556496434 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8c40462f0 msgr2=0x7fe8c40070d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:28.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.050+0000 7fe8d169d640 1 --2- 172.21.15.116:0/1556496434 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8c40462f0 0x7fe8c40070d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe8c4046780 tx=0x7fe8b801c3f0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.050+0000 7f82d77fe640 1 --2- 172.21.15.116:0/91958015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82c4005380 0x7f82c40079b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:28.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.050+0000 7f82d6ffd640 1 --2- 172.21.15.116:0/91958015 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82c40039c0 0x7f82c4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:28.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.050+0000 7fcedf59e640 1 -- 172.21.15.116:0/3328312672 wait complete. 2024-04-25T00:01:28.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.050+0000 7f82cffff640 1 --2- 172.21.15.116:0/91958015 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f82c4004810 0x7f82c4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:28.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.050+0000 7fe8d169d640 1 -- 172.21.15.116:0/1556496434 shutdown_connections 2024-04-25T00:01:28.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.050+0000 7fe8d169d640 1 --2- 172.21.15.116:0/1556496434 >> v2:172.21.15.174:3300/0 conn(0x7fe8c40496e0 0x7fe8c4049b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.050+0000 7fe8d169d640 1 --2- 172.21.15.116:0/1556496434 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8c40462f0 0x7fe8c40070d0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.050+0000 7fe8d169d640 1 -- 172.21.15.116:0/1556496434 >> 172.21.15.116:0/1556496434 conn(0x7fe8c4005ea0 msgr2=0x7fe8c40062b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:28.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.050+0000 7f82d77fe640 1 -- 172.21.15.116:0/91958015 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f82c4004810 msgr2=0x7f82c4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:28.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.050+0000 7fe8d169d640 1 -- 172.21.15.116:0/1556496434 shutdown_connections 2024-04-25T00:01:28.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.050+0000 7f82d77fe640 1 --2- 172.21.15.116:0/91958015 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f82c4004810 0x7f82c4004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.050+0000 7f82d77fe640 1 -- 172.21.15.116:0/91958015 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82c40039c0 msgr2=0x7f82c4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:28.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.050+0000 7f82d77fe640 1 --2- 172.21.15.116:0/91958015 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82c40039c0 0x7f82c4003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.050+0000 7f82d77fe640 1 -- 172.21.15.116:0/91958015 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82d8162f60 con 0x7f82c4005380 2024-04-25T00:01:28.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.050+0000 7fcedf59e640 1 Processor -- start 2024-04-25T00:01:28.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.050+0000 7fe8d169d640 1 -- 172.21.15.116:0/1556496434 wait complete. 2024-04-25T00:01:28.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.054+0000 7fe8d169d640 1 Processor -- start 2024-04-25T00:01:28.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.054+0000 7fcedf59e640 1 -- start start 2024-04-25T00:01:28.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.054+0000 7fcedf59e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fced8007700 0x7fced81722d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.054+0000 7fcedf59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fced804daf0 0x7fced8045c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.054+0000 7fcedf59e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fced80d4360 0x7fced80461b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.054+0000 7fcedf59e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fced800c700 con 0x7fced804daf0 2024-04-25T00:01:28.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.054+0000 7fcedf59e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fced800c580 con 0x7fced80d4360 2024-04-25T00:01:28.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.054+0000 7fcedf59e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fced800c880 con 0x7fced8007700 2024-04-25T00:01:28.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.054+0000 7fceddd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fced804daf0 0x7fced8045c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:28.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.054+0000 7fceddd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fced804daf0 0x7fced8045c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56336/0 (socket says 172.21.15.116:56336) 2024-04-25T00:01:28.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.054+0000 7fceddd9b640 1 -- 172.21.15.116:0/2106852205 learned_addr learned my addr 172.21.15.116:0/2106852205 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:28.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.054+0000 7fcede59c640 1 --2- 172.21.15.116:0/2106852205 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fced8007700 0x7fced81722d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:28.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.054+0000 7fceded9d640 1 --2- 172.21.15.116:0/2106852205 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fced80d4360 0x7fced80461b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:28.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.054+0000 7fe8d169d640 1 -- start start 2024-04-25T00:01:28.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.054+0000 7fceddd9b640 1 -- 172.21.15.116:0/2106852205 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fced8007700 msgr2=0x7fced81722d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:28.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.054+0000 7fceddd9b640 1 --2- 172.21.15.116:0/2106852205 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fced8007700 0x7fced81722d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.054+0000 7fceddd9b640 1 -- 172.21.15.116:0/2106852205 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fced80d4360 msgr2=0x7fced80461b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:28.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.054+0000 7fceddd9b640 1 --2- 172.21.15.116:0/2106852205 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fced80d4360 0x7fced80461b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.054+0000 7fceddd9b640 1 -- 172.21.15.116:0/2106852205 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fced8031240 con 0x7fced804daf0 2024-04-25T00:01:28.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.054+0000 7fceded9d640 1 --2- 172.21.15.116:0/2106852205 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fced80d4360 0x7fced80461b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:28.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.054+0000 7f82d77fe640 1 -- 172.21.15.116:0/91958015 reap_dead start 2024-04-25T00:01:28.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.054+0000 7fceddd9b640 1 --2- 172.21.15.116:0/2106852205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fced804daf0 0x7fced8045c70 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fced000c500 tx=0x7fced000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:28.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.054+0000 7f82d77fe640 1 --2- 172.21.15.116:0/91958015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82c4005380 0x7f82c40079b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f82d0001e50 tx=0x7f82d001c780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f17ccd94ab63d92 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:28.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.054+0000 7fe8d169d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe8c40462f0 0x7fe8c40064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.054+0000 7fe8d169d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8c40496e0 0x7fe8c4178840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.054+0000 7fe8d169d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe8c40d3ab0 0x7fe8c4178d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.054+0000 7fe8d169d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe8c400c400 con 0x7fe8c40496e0 2024-04-25T00:01:28.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.054+0000 7fe8d169d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe8c400c280 con 0x7fe8c40d3ab0 2024-04-25T00:01:28.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.054+0000 7fe8d169d640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe8c40e35e0 con 0x7fe8c40462f0 2024-04-25T00:01:28.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.054+0000 7fe8cb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8c40496e0 0x7fe8c4178840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:28.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.054+0000 7fe8cb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8c40496e0 0x7fe8c4178840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56350/0 (socket says 172.21.15.116:56350) 2024-04-25T00:01:28.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.054+0000 7fe8cb7fe640 1 -- 172.21.15.116:0/2044269259 learned_addr learned my addr 172.21.15.116:0/2044269259 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:28.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.054+0000 7fe8cbfff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe8c40462f0 0x7fe8c40064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:28.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.054+0000 7fe8d0e9c640 1 --2- 172.21.15.116:0/2044269259 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe8c40d3ab0 0x7fe8c4178d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:28.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.054+0000 7fe8cb7fe640 1 -- 172.21.15.116:0/2044269259 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe8c40462f0 msgr2=0x7fe8c40064e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:28.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.054+0000 7fe8cb7fe640 1 --2- 172.21.15.116:0/2044269259 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe8c40462f0 0x7fe8c40064e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.054+0000 7fe8cb7fe640 1 -- 172.21.15.116:0/2044269259 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe8c40d3ab0 msgr2=0x7fe8c4178d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:28.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.054+0000 7fe8cb7fe640 1 --2- 172.21.15.116:0/2044269259 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe8c40d3ab0 0x7fe8c4178d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.054+0000 7fe8cb7fe640 1 -- 172.21.15.116:0/2044269259 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8c40413b0 con 0x7fe8c40496e0 2024-04-25T00:01:28.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.054+0000 7f82d67fc640 1 -- 172.21.15.116:0/91958015 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d0001a70 con 0x7f82c4005380 2024-04-25T00:01:28.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.054+0000 7f82d67fc640 1 -- 172.21.15.116:0/91958015 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82d0003ba0 con 0x7f82c4005380 2024-04-25T00:01:28.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.054+0000 7fe8d0e9c640 1 --2- 172.21.15.116:0/2044269259 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe8c40d3ab0 0x7fe8c4178d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:01:28.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.054+0000 7f82d67fc640 1 -- 172.21.15.116:0/91958015 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d0005020 con 0x7f82c4005380 2024-04-25T00:01:28.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.054+0000 7fceceffd640 1 -- 172.21.15.116:0/2106852205 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fced0017070 con 0x7fced804daf0 2024-04-25T00:01:28.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.054+0000 7fceceffd640 1 -- 172.21.15.116:0/2106852205 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fced0005cf0 con 0x7fced804daf0 2024-04-25T00:01:28.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.054+0000 7fceceffd640 1 -- 172.21.15.116:0/2106852205 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fced00072f0 con 0x7fced804daf0 2024-04-25T00:01:28.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.054+0000 7fcedf59e640 1 -- 172.21.15.116:0/2106852205 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fced8043680 con 0x7fced804daf0 2024-04-25T00:01:28.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.054+0000 7fcedf59e640 1 -- 172.21.15.116:0/2106852205 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fced8172f00 con 0x7fced804daf0 2024-04-25T00:01:28.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.054+0000 7f82d7fff640 1 -- 172.21.15.116:0/91958015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82c4005380 msgr2=0x7f82c40079b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:28.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.054+0000 7f82d7fff640 1 --2- 172.21.15.116:0/91958015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82c4005380 0x7f82c40079b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f82d0001e50 tx=0x7f82d001c780 comp rx=0 tx=0).stop 2024-04-25T00:01:28.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.054+0000 7f82d7fff640 1 -- 172.21.15.116:0/91958015 shutdown_connections 2024-04-25T00:01:28.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.054+0000 7f82d7fff640 1 --2- 172.21.15.116:0/91958015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82c4005380 0x7f82c40079b0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.054+0000 7f82d7fff640 1 -- 172.21.15.116:0/91958015 >> 172.21.15.116:0/91958015 conn(0x7f82d80c3940 msgr2=0x7f82d80c3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:28.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.054+0000 7f82d7fff640 1 -- 172.21.15.116:0/91958015 shutdown_connections 2024-04-25T00:01:28.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.054+0000 7f82d7fff640 1 -- 172.21.15.116:0/91958015 wait complete. 2024-04-25T00:01:28.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.058+0000 7fe8cb7fe640 1 --2- 172.21.15.116:0/2044269259 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8c40496e0 0x7fe8c4178840 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe8b80ae040 tx=0x7fe8b8019190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:28.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.058+0000 7fe8c8ff9640 1 -- 172.21.15.116:0/2044269259 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b80c45b0 con 0x7fe8c40496e0 2024-04-25T00:01:28.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.058+0000 7fe8c8ff9640 1 -- 172.21.15.116:0/2044269259 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8b80ca070 con 0x7fe8c40496e0 2024-04-25T00:01:28.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.058+0000 7fe8c8ff9640 1 -- 172.21.15.116:0/2044269259 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b80c4990 con 0x7fe8c40496e0 2024-04-25T00:01:28.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.058+0000 7fe8d169d640 1 -- 172.21.15.116:0/2044269259 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8c4041690 con 0x7fe8c40496e0 2024-04-25T00:01:28.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.058+0000 7fe8d169d640 1 -- 172.21.15.116:0/2044269259 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8c4041c50 con 0x7fe8c40496e0 2024-04-25T00:01:28.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.058+0000 7f82d7fff640 1 Processor -- start 2024-04-25T00:01:28.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.058+0000 7fceceffd640 1 -- 172.21.15.116:0/2106852205 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fced001a070 con 0x7fced804daf0 2024-04-25T00:01:28.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.058+0000 7fe8b67fc640 1 -- 172.21.15.116:0/2044269259 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8c40053a0 con 0x7fe8c40496e0 2024-04-25T00:01:28.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.058+0000 7fceceffd640 1 --2- 172.21.15.116:0/2106852205 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcec0067990 0x7fcec0069e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.058+0000 7fceceffd640 1 -- 172.21.15.116:0/2106852205 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7fced0085260 con 0x7fced804daf0 2024-04-25T00:01:28.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.058+0000 7fcede59c640 1 --2- 172.21.15.116:0/2106852205 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcec0067990 0x7fcec0069e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:28.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.058+0000 7fe8c8ff9640 1 -- 172.21.15.116:0/2044269259 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe8b80c4b30 con 0x7fe8c40496e0 2024-04-25T00:01:28.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.058+0000 7f82d7fff640 1 -- start start 2024-04-25T00:01:28.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.058+0000 7f82d7fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82c4005380 0x7f82d81603c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.058+0000 7f82d7fff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f82d81616f0 0x7f82d80c5f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.058+0000 7f82d7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82d80a9d70 0x7f82d80c6470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.058+0000 7f82d7fff640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f82d80c1470 con 0x7f82d80a9d70 2024-04-25T00:01:28.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.058+0000 7f82d7fff640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f82d806ee40 con 0x7f82c4005380 2024-04-25T00:01:28.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.058+0000 7f82d7fff640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f82d80c2e90 con 0x7f82d81616f0 2024-04-25T00:01:28.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.058+0000 7f82d77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82d80a9d70 0x7f82d80c6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:28.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.058+0000 7fbebcff9640 1 -- 172.21.15.116:0/171848925 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fbeb8021020 con 0x7fbec00c5520 2024-04-25T00:01:28.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.058+0000 7f82d77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82d80a9d70 0x7f82d80c6470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56362/0 (socket says 172.21.15.116:56362) 2024-04-25T00:01:28.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.058+0000 7f82d77fe640 1 -- 172.21.15.116:0/4178188501 learned_addr learned my addr 172.21.15.116:0/4178188501 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:28.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.058+0000 7fe8c8ff9640 1 --2- 172.21.15.116:0/2044269259 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe8b00678c0 0x7fe8b0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.058+0000 7fcede59c640 1 --2- 172.21.15.116:0/2106852205 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcec0067990 0x7fcec0069e50 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fced4006fd0 tx=0x7fced400fa00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:28.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.058+0000 7f82d77fe640 1 -- 172.21.15.116:0/4178188501 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f82d81616f0 msgr2=0x7f82d80c5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:28.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.058+0000 7f82d77fe640 1 --2- 172.21.15.116:0/4178188501 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f82d81616f0 0x7f82d80c5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.058+0000 7f82d77fe640 1 -- 172.21.15.116:0/4178188501 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82c4005380 msgr2=0x7f82d81603c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T00:01:28.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.058+0000 7f82d77fe640 1 --2- 172.21.15.116:0/4178188501 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82c4005380 0x7f82d81603c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.058+0000 7fcedf59e640 1 -- 172.21.15.116:0/2106852205 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcec40052a0 con 0x7fced804daf0 2024-04-25T00:01:28.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.058+0000 7f82d77fe640 1 -- 172.21.15.116:0/4178188501 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82d81f8820 con 0x7f82d80a9d70 2024-04-25T00:01:28.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.062+0000 7f82d77fe640 1 --2- 172.21.15.116:0/4178188501 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82d80a9d70 0x7f82d80c6470 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f82d001c5b0 tx=0x7f82c4005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:28.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.062+0000 7fe8c8ff9640 1 -- 172.21.15.116:0/2044269259 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7fe8b8135060 con 0x7fe8c40496e0 2024-04-25T00:01:28.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.062+0000 7fe8cbfff640 1 --2- 172.21.15.116:0/2044269259 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe8b00678c0 0x7fe8b0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:28.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.062+0000 7fe8cbfff640 1 --2- 172.21.15.116:0/2044269259 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe8b00678c0 0x7fe8b0069d80 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe8c0005890 tx=0x7fe8c0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:28.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.066+0000 7fe8c8ff9640 1 -- 172.21.15.116:0/2044269259 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fe8b8016070 con 0x7fe8c40496e0 2024-04-25T00:01:28.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.066+0000 7f82cf7fe640 1 -- 172.21.15.116:0/4178188501 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d0005790 con 0x7f82d80a9d70 2024-04-25T00:01:28.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.066+0000 7f82d7fff640 1 -- 172.21.15.116:0/4178188501 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82d8162120 con 0x7f82d80a9d70 2024-04-25T00:01:28.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.066+0000 7f82d7fff640 1 -- 172.21.15.116:0/4178188501 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82d8072fc0 con 0x7f82d80a9d70 2024-04-25T00:01:28.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.070+0000 7f82cd7fa640 1 -- 172.21.15.116:0/4178188501 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82d804f510 con 0x7f82d80a9d70 2024-04-25T00:01:28.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.070+0000 7fbecb539640 1 Processor -- start 2024-04-25T00:01:28.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.070+0000 7fbecb539640 1 -- start start 2024-04-25T00:01:28.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.070+0000 7fbecb539640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbebc0409a0 0x7fbebc049770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.070+0000 7fbecb539640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fbebc0df2e0 0x7fbebc040d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.070+0000 7fbecb539640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbebc00c560 con 0x7fbebc00d6e0 2024-04-25T00:01:28.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.070+0000 7fbecb539640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fbebc049dc0 con 0x7fbebc0df2e0 2024-04-25T00:01:28.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.070+0000 7fbecb539640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fbebc04a0a0 con 0x7fbebc0409a0 2024-04-25T00:01:28.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.070+0000 7fbec9ad6640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fbebc0df2e0 0x7fbebc040d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:28.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.070+0000 7fbec92d5640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbebc0409a0 0x7fbebc049770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:28.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.070+0000 7fbec9ad6640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fbebc0df2e0 0x7fbebc040d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:37434/0 (socket says 172.21.15.116:37434) 2024-04-25T00:01:28.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.070+0000 7fbec92d5640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbebc0409a0 0x7fbebc049770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:56222/0 (socket says 172.21.15.116:56222) 2024-04-25T00:01:28.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.070+0000 7fbec9ad6640 1 -- 172.21.15.116:0/751466584 learned_addr learned my addr 172.21.15.116:0/751466584 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:28.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.070+0000 7fbec9ad6640 1 -- 172.21.15.116:0/751466584 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbebc0409a0 msgr2=0x7fbebc049770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:28.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.070+0000 7fbec9ad6640 1 --2- 172.21.15.116:0/751466584 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbebc0409a0 0x7fbebc049770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.070+0000 7fbec9ad6640 1 -- 172.21.15.116:0/751466584 >> v1:172.21.15.174:6789/0 conn(0x7fbebc00d6e0 legacy=0x7fbebc00be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:28.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.070+0000 7fbec9ad6640 1 -- 172.21.15.116:0/751466584 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbebc0deae0 con 0x7fbebc0df2e0 2024-04-25T00:01:28.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.070+0000 7fceceffd640 1 -- 172.21.15.116:0/2106852205 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fced8031240 con 0x7fced804daf0 2024-04-25T00:01:28.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.074+0000 7f82cf7fe640 1 -- 172.21.15.116:0/4178188501 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82d0003ba0 con 0x7f82d80a9d70 2024-04-25T00:01:28.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.074+0000 7f82cf7fe640 1 -- 172.21.15.116:0/4178188501 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82d00a6870 con 0x7f82d80a9d70 2024-04-25T00:01:28.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.074+0000 7fbec9ad6640 1 --2- 172.21.15.116:0/751466584 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbebc0df2e0 0x7fbebc040d80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fbec406a4d0 tx=0x7fbec409a900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f5eec0c243ab0e20 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:28.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.074+0000 7fbeb7fff640 1 -- 172.21.15.116:0/751466584 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec406e410 con 0x7fbebc0df2e0 2024-04-25T00:01:28.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.074+0000 7fbeb7fff640 1 -- 172.21.15.116:0/751466584 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbec406e5b0 con 0x7fbebc0df2e0 2024-04-25T00:01:28.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.074+0000 7f82cf7fe640 1 -- 172.21.15.116:0/4178188501 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f82d00a6a10 con 0x7f82d80a9d70 2024-04-25T00:01:28.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.074+0000 7f82cf7fe640 1 --2- 172.21.15.116:0/4178188501 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f82c4071cb0 0x7f82c4074170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.074+0000 7fbeb7fff640 1 -- 172.21.15.116:0/751466584 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec406e9e0 con 0x7fbebc0df2e0 2024-04-25T00:01:28.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.074+0000 7fbecb539640 1 -- 172.21.15.116:0/751466584 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbebc0df2e0 msgr2=0x7fbebc040d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:28.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.074+0000 7fbecb539640 1 --2- 172.21.15.116:0/751466584 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbebc0df2e0 0x7fbebc040d80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fbec406a4d0 tx=0x7fbec409a900 comp rx=0 tx=0).stop 2024-04-25T00:01:28.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.074+0000 7f82cf7fe640 1 -- 172.21.15.116:0/4178188501 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7f82d00edb90 con 0x7f82d80a9d70 2024-04-25T00:01:28.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.074+0000 7f82d6ffd640 1 --2- 172.21.15.116:0/4178188501 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f82c4071cb0 0x7f82c4074170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:28.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.078+0000 7f82d6ffd640 1 --2- 172.21.15.116:0/4178188501 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f82c4071cb0 0x7f82c4074170 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f82c80047b0 tx=0x7f82c8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:28.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.082+0000 7fbecb539640 1 -- 172.21.15.116:0/751466584 shutdown_connections 2024-04-25T00:01:28.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.082+0000 7fbecb539640 1 --2- 172.21.15.116:0/751466584 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbebc0df2e0 0x7fbebc040d80 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.082+0000 7fbecb539640 1 --2- 172.21.15.116:0/751466584 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbebc0409a0 0x7fbebc049770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.082+0000 7fbecb539640 1 -- 172.21.15.116:0/751466584 >> 172.21.15.116:0/751466584 conn(0x7fbebc02f4c0 msgr2=0x7fbebc02f8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:28.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.082+0000 7fbecb539640 1 -- 172.21.15.116:0/751466584 shutdown_connections 2024-04-25T00:01:28.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.082+0000 7f82cf7fe640 1 -- 172.21.15.116:0/4178188501 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f82d0016490 con 0x7f82d80a9d70 2024-04-25T00:01:28.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.082+0000 7fbecb539640 1 -- 172.21.15.116:0/751466584 wait complete. 2024-04-25T00:01:28.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.082+0000 7fbecb539640 1 Processor -- start 2024-04-25T00:01:28.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.082+0000 7fbecb539640 1 -- start start 2024-04-25T00:01:28.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.082+0000 7fbecb539640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbebc00d6e0 0x7fbebc0e08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.082+0000 7fbecb539640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbebc0409a0 0x7fbebc0df8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.082+0000 7fbecb539640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbebc0df2e0 0x7fbebc0dfe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.082+0000 7fbecb539640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbebc0302d0 con 0x7fbebc0df2e0 2024-04-25T00:01:28.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.082+0000 7fbecb539640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbebc0de7b0 con 0x7fbebc0409a0 2024-04-25T00:01:28.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.082+0000 7fbecb539640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fbebc0dc250 con 0x7fbebc00d6e0 2024-04-25T00:01:28.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.082+0000 7fbec9ad6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbebc0df2e0 0x7fbebc0dfe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:28.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.082+0000 7fbec9ad6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbebc0df2e0 0x7fbebc0dfe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56376/0 (socket says 172.21.15.116:56376) 2024-04-25T00:01:28.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.082+0000 7fbec9ad6640 1 -- 172.21.15.116:0/3315917463 learned_addr learned my addr 172.21.15.116:0/3315917463 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:28.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.082+0000 7fbec92d5640 1 --2- 172.21.15.116:0/3315917463 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbebc00d6e0 0x7fbebc0e08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:28.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.082+0000 7fbec8ad4640 1 --2- 172.21.15.116:0/3315917463 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbebc0409a0 0x7fbebc0df8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:28.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.082+0000 7fbec9ad6640 1 -- 172.21.15.116:0/3315917463 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbebc00d6e0 msgr2=0x7fbebc0e08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:28.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.082+0000 7fbec9ad6640 1 --2- 172.21.15.116:0/3315917463 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbebc00d6e0 0x7fbebc0e08c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.082+0000 7fbec9ad6640 1 -- 172.21.15.116:0/3315917463 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbebc0409a0 msgr2=0x7fbebc0df8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:28.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.082+0000 7fbec9ad6640 1 --2- 172.21.15.116:0/3315917463 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbebc0409a0 0x7fbebc0df8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.082+0000 7fbec9ad6640 1 -- 172.21.15.116:0/3315917463 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbebc0e8e20 con 0x7fbebc0df2e0 2024-04-25T00:01:28.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.082+0000 7fbec9ad6640 1 --2- 172.21.15.116:0/3315917463 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbebc0df2e0 0x7fbebc0dfe00 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fbec402d710 tx=0x7fbec4097740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:28.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.086+0000 7fbeb5ffb640 1 -- 172.21.15.116:0/3315917463 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec413a070 con 0x7fbebc0df2e0 2024-04-25T00:01:28.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.086+0000 7fbeb5ffb640 1 -- 172.21.15.116:0/3315917463 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbec4070190 con 0x7fbebc0df2e0 2024-04-25T00:01:28.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.086+0000 7fbecb539640 1 -- 172.21.15.116:0/3315917463 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbebc0e9100 con 0x7fbebc0df2e0 2024-04-25T00:01:28.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.086+0000 7fbeb5ffb640 1 -- 172.21.15.116:0/3315917463 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec406e410 con 0x7fbebc0df2e0 2024-04-25T00:01:28.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.086+0000 7fbecb539640 1 -- 172.21.15.116:0/3315917463 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbebc043100 con 0x7fbebc0df2e0 2024-04-25T00:01:28.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.086+0000 7fbeb5ffb640 1 -- 172.21.15.116:0/3315917463 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbec4132070 con 0x7fbebc0df2e0 2024-04-25T00:01:28.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.086+0000 7fbeb5ffb640 1 --2- 172.21.15.116:0/3315917463 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbea80678c0 0x7fbea8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.086+0000 7fbeb5ffb640 1 -- 172.21.15.116:0/3315917463 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7fbec419e480 con 0x7fbebc0df2e0 2024-04-25T00:01:28.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.086+0000 7fbec92d5640 1 --2- 172.21.15.116:0/3315917463 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbea80678c0 0x7fbea8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:28.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.090+0000 7fbec92d5640 1 --2- 172.21.15.116:0/3315917463 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbea80678c0 0x7fbea8069d80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fbeb8009b60 tx=0x7fbeb8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:28.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.098+0000 7fbecb539640 1 -- 172.21.15.116:0/3315917463 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbebc0057f0 con 0x7fbebc0df2e0 2024-04-25T00:01:28.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.106+0000 7fbeb5ffb640 1 -- 172.21.15.116:0/3315917463 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fbec4094530 con 0x7fbebc0df2e0 2024-04-25T00:01:28.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.106+0000 7fb10bfff640 1 Processor -- start 2024-04-25T00:01:28.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.106+0000 7fb10bfff640 1 -- start start 2024-04-25T00:01:28.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.106+0000 7fb10bfff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb10c0be730 0x7fb10c06c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.106+0000 7fb10bfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb10c0c8f00 0x7fb10c06ca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.106+0000 7fb10bfff640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb10c0b0f50 con 0x7fb10c157310 2024-04-25T00:01:28.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.106+0000 7fb10bfff640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fb10c15a060 con 0x7fb10c0c8f00 2024-04-25T00:01:28.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.106+0000 7fb10bfff640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fb10c15a340 con 0x7fb10c0be730 2024-04-25T00:01:28.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.106+0000 7fb10b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb10c0c8f00 0x7fb10c06ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:28.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.106+0000 7fb103fff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb10c0be730 0x7fb10c06c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:28.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.106+0000 7fb103fff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb10c0be730 0x7fb10c06c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:56232/0 (socket says 172.21.15.116:56232) 2024-04-25T00:01:28.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.106+0000 7fb103fff640 1 -- 172.21.15.116:0/2879531144 learned_addr learned my addr 172.21.15.116:0/2879531144 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:28.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.110+0000 7fb103fff640 1 -- 172.21.15.116:0/2879531144 >> v2:172.21.15.116:3300/0 conn(0x7fb10c0c8f00 msgr2=0x7fb10c06ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:28.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.110+0000 7fb103fff640 1 --2- 172.21.15.116:0/2879531144 >> v2:172.21.15.116:3300/0 conn(0x7fb10c0c8f00 0x7fb10c06ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.110+0000 7fb103fff640 1 -- 172.21.15.116:0/2879531144 >> v1:172.21.15.116:6789/0 conn(0x7fb10c157310 legacy=0x7fb10c0c5980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T00:01:28.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.110+0000 7fb103fff640 1 -- 172.21.15.116:0/2879531144 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb10c15a620 con 0x7fb10c0be730 2024-04-25T00:01:28.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.110+0000 7fb10b7fe640 1 --2- 172.21.15.116:0/2879531144 >> v2:172.21.15.116:3300/0 conn(0x7fb10c0c8f00 0x7fb10c06ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:01:28.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.114+0000 7fb103fff640 1 --2- 172.21.15.116:0/2879531144 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb10c0be730 0x7fb10c06c550 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc009870 tx=0x7fb0fc01cf70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=219ca251de5745d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:28.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.114+0000 7fb10a7fc640 1 -- 172.21.15.116:0/2879531144 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0fc0aa650 con 0x7fb10c0be730 2024-04-25T00:01:28.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.114+0000 7fb10a7fc640 1 -- 172.21.15.116:0/2879531144 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0fc0af070 con 0x7fb10c0be730 2024-04-25T00:01:28.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.114+0000 7fb10a7fc640 1 -- 172.21.15.116:0/2879531144 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0fc0aa9f0 con 0x7fb10c0be730 2024-04-25T00:01:28.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.114+0000 7fb10bfff640 1 -- 172.21.15.116:0/2879531144 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb10c0be730 msgr2=0x7fb10c06c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:28.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.114+0000 7fb10bfff640 1 --2- 172.21.15.116:0/2879531144 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb10c0be730 0x7fb10c06c550 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc009870 tx=0x7fb0fc01cf70 comp rx=0 tx=0).stop 2024-04-25T00:01:28.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.114+0000 7fb10bfff640 1 -- 172.21.15.116:0/2879531144 shutdown_connections 2024-04-25T00:01:28.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.114+0000 7fb10bfff640 1 --2- 172.21.15.116:0/2879531144 >> v2:172.21.15.116:3300/0 conn(0x7fb10c0c8f00 0x7fb10c06ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.114+0000 7fb10bfff640 1 --2- 172.21.15.116:0/2879531144 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb10c0be730 0x7fb10c06c550 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.114+0000 7fb10bfff640 1 -- 172.21.15.116:0/2879531144 >> 172.21.15.116:0/2879531144 conn(0x7fb10c0acce0 msgr2=0x7fb10c0ad0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:28.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.114+0000 7fb10bfff640 1 -- 172.21.15.116:0/2879531144 shutdown_connections 2024-04-25T00:01:28.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.114+0000 7fb10bfff640 1 -- 172.21.15.116:0/2879531144 wait complete. 2024-04-25T00:01:28.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.118+0000 7fb10bfff640 1 Processor -- start 2024-04-25T00:01:28.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.118+0000 7fb10bfff640 1 -- start start 2024-04-25T00:01:28.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.118+0000 7fb10bfff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb10c0be730 0x7fb10c0bf490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.118+0000 7fb10bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb10c0c8f00 0x7fb10c15cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.118+0000 7fb103fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb10c0c8f00 0x7fb10c15cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:28.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.118+0000 7fb103fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb10c0c8f00 0x7fb10c15cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56394/0 (socket says 172.21.15.116:56394) 2024-04-25T00:01:28.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.118+0000 7fb103fff640 1 -- 172.21.15.116:0/2837989791 learned_addr learned my addr 172.21.15.116:0/2837989791 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:28.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.118+0000 7fb10bfff640 1 --2- 172.21.15.116:0/2837989791 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb10c157310 0x7fb10c156360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.118+0000 7fb10bfff640 1 -- 172.21.15.116:0/2837989791 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb10c0c6090 con 0x7fb10c0c8f00 2024-04-25T00:01:28.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.118+0000 7fb10bfff640 1 -- 172.21.15.116:0/2837989791 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb10c159d10 con 0x7fb10c157310 2024-04-25T00:01:28.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.118+0000 7fb10bfff640 1 -- 172.21.15.116:0/2837989791 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fb10c0c6210 con 0x7fb10c0be730 2024-04-25T00:01:28.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.118+0000 7fb103fff640 1 -- 172.21.15.116:0/2837989791 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb10c0be730 msgr2=0x7fb10c0bf490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T00:01:28.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.118+0000 7fb103fff640 1 --2- 172.21.15.116:0/2837989791 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb10c0be730 0x7fb10c0bf490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.118+0000 7fb103fff640 1 -- 172.21.15.116:0/2837989791 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb10c157310 msgr2=0x7fb10c156360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:28.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.118+0000 7fb103fff640 1 --2- 172.21.15.116:0/2837989791 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb10c157310 0x7fb10c156360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.118+0000 7fb103fff640 1 -- 172.21.15.116:0/2837989791 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb10c0bfb90 con 0x7fb10c0c8f00 2024-04-25T00:01:28.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.122+0000 7fb103fff640 1 --2- 172.21.15.116:0/2837989791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb10c0c8f00 0x7fb10c15cd90 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc01cdb0 tx=0x7fb0fc0192b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:28.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.122+0000 7fb108ff9640 1 -- 172.21.15.116:0/2837989791 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0fc003bf0 con 0x7fb10c0c8f00 2024-04-25T00:01:28.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.122+0000 7fb10bfff640 1 -- 172.21.15.116:0/2837989791 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb10c0c4670 con 0x7fb10c0c8f00 2024-04-25T00:01:28.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.122+0000 7fb10bfff640 1 -- 172.21.15.116:0/2837989791 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb10c0c1df0 con 0x7fb10c0c8f00 2024-04-25T00:01:28.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.122+0000 7fb108ff9640 1 -- 172.21.15.116:0/2837989791 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0fc0a6040 con 0x7fb10c0c8f00 2024-04-25T00:01:28.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.126+0000 7fb108ff9640 1 -- 172.21.15.116:0/2837989791 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0fc0aa450 con 0x7fb10c0c8f00 2024-04-25T00:01:28.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.126+0000 7fb108ff9640 1 -- 172.21.15.116:0/2837989791 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb0fc0aa5f0 con 0x7fb10c0c8f00 2024-04-25T00:01:28.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.126+0000 7fb108ff9640 1 --2- 172.21.15.116:0/2837989791 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb0f8067b00 0x7fb0f8069fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.126+0000 7fb108ff9640 1 -- 172.21.15.116:0/2837989791 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7fb0fc1200c0 con 0x7fb10c0c8f00 2024-04-25T00:01:28.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.126+0000 7fb10affd640 1 --2- 172.21.15.116:0/2837989791 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb0f8067b00 0x7fb0f8069fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:28.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.126+0000 7fb10affd640 1 --2- 172.21.15.116:0/2837989791 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb0f8067b00 0x7fb0f8069fc0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb0f4009690 tx=0x7fb0f4006be0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:28.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.138+0000 7fb10bfff640 1 -- 172.21.15.116:0/2837989791 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0f00052a0 con 0x7fb10c0c8f00 2024-04-25T00:01:28.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.146+0000 7fb108ff9640 1 -- 172.21.15.116:0/2837989791 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fb0fc016130 con 0x7fb10c0c8f00 2024-04-25T00:01:28.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.302+0000 7f45910e1640 1 -- 172.21.15.116:0/2951034814 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f458c0c13c0 con 0x7f458c06cb70 2024-04-25T00:01:28.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.302+0000 7f45737fe640 1 -- 172.21.15.116:0/2951034814 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f457c021020 con 0x7f458c06cb70 2024-04-25T00:01:28.305 INFO:teuthology.orchestra.run.smithi116.stdout:64424509449 2024-04-25T00:01:28.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.302+0000 7f45717fa640 1 -- 172.21.15.116:0/2951034814 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4578067910 msgr2=0x7f4578069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:28.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.302+0000 7f45717fa640 1 --2- 172.21.15.116:0/2951034814 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4578067910 0x7f4578069dd0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f45800b1000 tx=0x7f458000a040 comp rx=0 tx=0).stop 2024-04-25T00:01:28.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.302+0000 7f45717fa640 1 -- 172.21.15.116:0/2951034814 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f458c06cb70 msgr2=0x7f458c1eddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:28.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.306+0000 7f45717fa640 1 --2- 172.21.15.116:0/2951034814 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f458c06cb70 0x7f458c1eddf0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f457c007d40 tx=0x7f457c00a430 comp rx=0 tx=0).stop 2024-04-25T00:01:28.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.306+0000 7f45717fa640 1 -- 172.21.15.116:0/2951034814 shutdown_connections 2024-04-25T00:01:28.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.306+0000 7f45717fa640 1 --2- 172.21.15.116:0/2951034814 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4578067910 0x7f4578069dd0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.306+0000 7f45717fa640 1 --2- 172.21.15.116:0/2951034814 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f458c13acb0 0x7f458c133cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.306+0000 7f45717fa640 1 --2- 172.21.15.116:0/2951034814 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f458c13a360 0x7f458c133790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.306+0000 7f45717fa640 1 --2- 172.21.15.116:0/2951034814 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f458c06cb70 0x7f458c1eddf0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.306+0000 7f45717fa640 1 -- 172.21.15.116:0/2951034814 >> 172.21.15.116:0/2951034814 conn(0x7f458c0c3a90 msgr2=0x7f458c0c3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:28.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.306+0000 7f45717fa640 1 -- 172.21.15.116:0/2951034814 shutdown_connections 2024-04-25T00:01:28.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.306+0000 7f45717fa640 1 -- 172.21.15.116:0/2951034814 wait complete. 2024-04-25T00:01:28.329 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509449 got 64424509449 for osd.5 2024-04-25T00:01:28.329 DEBUG:teuthology.parallel:result is None 2024-04-25T00:01:28.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.366+0000 7f17ac3ac640 1 -- 172.21.15.116:0/2607796615 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f17a4152470 con 0x7f17a4070dd0 2024-04-25T00:01:28.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.366+0000 7f179affd640 1 -- 172.21.15.116:0/2607796615 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f179c013030 con 0x7f17a4070dd0 2024-04-25T00:01:28.369 INFO:teuthology.orchestra.run.smithi116.stdout:64424509451 2024-04-25T00:01:28.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.370+0000 7f17ac3ac640 1 -- 172.21.15.116:0/2607796615 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1788067910 msgr2=0x7f1788069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:28.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.370+0000 7f17ac3ac640 1 --2- 172.21.15.116:0/2607796615 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1788067910 0x7f1788069dd0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f17a00b3000 tx=0x7f17a00aa070 comp rx=0 tx=0).stop 2024-04-25T00:01:28.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.370+0000 7f17ac3ac640 1 -- 172.21.15.116:0/2607796615 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17a4070dd0 msgr2=0x7f17a41f2640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:28.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.370+0000 7f17ac3ac640 1 --2- 172.21.15.116:0/2607796615 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17a4070dd0 0x7f17a41f2640 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f179c00ce70 tx=0x7f179c00a630 comp rx=0 tx=0).stop 2024-04-25T00:01:28.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.370+0000 7f17ac3ac640 1 -- 172.21.15.116:0/2607796615 shutdown_connections 2024-04-25T00:01:28.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.370+0000 7f17ac3ac640 1 --2- 172.21.15.116:0/2607796615 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1788067910 0x7f1788069dd0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.370+0000 7f17ac3ac640 1 --2- 172.21.15.116:0/2607796615 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f17a414b830 0x7f17a414c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.370+0000 7f17ac3ac640 1 --2- 172.21.15.116:0/2607796615 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17a4070dd0 0x7f17a41f2640 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.370+0000 7f17ac3ac640 1 --2- 172.21.15.116:0/2607796615 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f17a406e100 0x7f17a41f2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.370+0000 7f17ac3ac640 1 -- 172.21.15.116:0/2607796615 >> 172.21.15.116:0/2607796615 conn(0x7f17a40c3b00 msgr2=0x7f17a414eee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:28.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.370+0000 7f17ac3ac640 1 -- 172.21.15.116:0/2607796615 shutdown_connections 2024-04-25T00:01:28.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.370+0000 7f17ac3ac640 1 -- 172.21.15.116:0/2607796615 wait complete. 2024-04-25T00:01:28.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.378+0000 7fcedf59e640 1 -- 172.21.15.116:0/2106852205 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fcec4005590 con 0x7fced804daf0 2024-04-25T00:01:28.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.378+0000 7fceceffd640 1 -- 172.21.15.116:0/2106852205 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fced0055490 con 0x7fced804daf0 2024-04-25T00:01:28.381 INFO:teuthology.orchestra.run.smithi116.stdout:64424509450 2024-04-25T00:01:28.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.382+0000 7fbec65b8640 1 -- 172.21.15.116:0/171848925 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fbec01feba0 con 0x7fbec00c5520 2024-04-25T00:01:28.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.382+0000 7fbecb539640 1 -- 172.21.15.116:0/3315917463 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fbebc008da0 con 0x7fbebc0df2e0 2024-04-25T00:01:28.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.382+0000 7fbeb5ffb640 1 -- 172.21.15.116:0/3315917463 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fbec416e5d0 con 0x7fbebc0df2e0 2024-04-25T00:01:28.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.382+0000 7fcedf59e640 1 -- 172.21.15.116:0/2106852205 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcec0067990 msgr2=0x7fcec0069e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:28.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.382+0000 7fcedf59e640 1 --2- 172.21.15.116:0/2106852205 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcec0067990 0x7fcec0069e50 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fced4006fd0 tx=0x7fced400fa00 comp rx=0 tx=0).stop 2024-04-25T00:01:28.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.382+0000 7fcedf59e640 1 -- 172.21.15.116:0/2106852205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fced804daf0 msgr2=0x7fced8045c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:28.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.382+0000 7fcedf59e640 1 --2- 172.21.15.116:0/2106852205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fced804daf0 0x7fced8045c70 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fced000c500 tx=0x7fced000c9c0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.382+0000 7fbebcff9640 1 -- 172.21.15.116:0/171848925 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fbeb80563a0 con 0x7fbec00c5520 2024-04-25T00:01:28.385 INFO:teuthology.orchestra.run.smithi116.stdout:64424509450 2024-04-25T00:01:28.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.382+0000 7fcedf59e640 1 -- 172.21.15.116:0/2106852205 shutdown_connections 2024-04-25T00:01:28.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.382+0000 7fcedf59e640 1 --2- 172.21.15.116:0/2106852205 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcec0067990 0x7fcec0069e50 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.382+0000 7fcedf59e640 1 --2- 172.21.15.116:0/2106852205 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fced80d4360 0x7fced80461b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.382+0000 7fcedf59e640 1 --2- 172.21.15.116:0/2106852205 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fced804daf0 0x7fced8045c70 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.382+0000 7fcedf59e640 1 --2- 172.21.15.116:0/2106852205 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fced8007700 0x7fced81722d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.382+0000 7fcedf59e640 1 -- 172.21.15.116:0/2106852205 >> 172.21.15.116:0/2106852205 conn(0x7fced804c670 msgr2=0x7fced804b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:28.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.382+0000 7fcedf59e640 1 -- 172.21.15.116:0/2106852205 shutdown_connections 2024-04-25T00:01:28.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.382+0000 7fcedf59e640 1 -- 172.21.15.116:0/2106852205 wait complete. 2024-04-25T00:01:28.387 INFO:teuthology.orchestra.run.smithi116.stdout:55834574859 2024-04-25T00:01:28.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.386+0000 7fbecb539640 1 -- 172.21.15.116:0/3315917463 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbea80678c0 msgr2=0x7fbea8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:28.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.386+0000 7fbecb539640 1 --2- 172.21.15.116:0/3315917463 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbea80678c0 0x7fbea8069d80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fbeb8009b60 tx=0x7fbeb8009190 comp rx=0 tx=0).stop 2024-04-25T00:01:28.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.386+0000 7fbecb539640 1 -- 172.21.15.116:0/3315917463 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbebc0df2e0 msgr2=0x7fbebc0dfe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:28.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.386+0000 7fbecb539640 1 --2- 172.21.15.116:0/3315917463 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbebc0df2e0 0x7fbebc0dfe00 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fbec402d710 tx=0x7fbec4097740 comp rx=0 tx=0).stop 2024-04-25T00:01:28.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.386+0000 7fbecb539640 1 -- 172.21.15.116:0/3315917463 shutdown_connections 2024-04-25T00:01:28.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.386+0000 7fbecb539640 1 --2- 172.21.15.116:0/3315917463 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbea80678c0 0x7fbea8069d80 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.386+0000 7fbecb539640 1 --2- 172.21.15.116:0/3315917463 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbebc0df2e0 0x7fbebc0dfe00 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.386+0000 7fbecb539640 1 --2- 172.21.15.116:0/3315917463 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbebc0409a0 0x7fbebc0df8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.386+0000 7fbecb539640 1 --2- 172.21.15.116:0/3315917463 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbebc00d6e0 0x7fbebc0e08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.386+0000 7fbecb539640 1 -- 172.21.15.116:0/3315917463 >> 172.21.15.116:0/3315917463 conn(0x7fbebc02f4c0 msgr2=0x7fbebc02f8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:28.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.386+0000 7fbecb539640 1 -- 172.21.15.116:0/3315917463 shutdown_connections 2024-04-25T00:01:28.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.386+0000 7fbecb539640 1 -- 172.21.15.116:0/3315917463 wait complete. 2024-04-25T00:01:28.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.386+0000 7fbec65b8640 1 -- 172.21.15.116:0/171848925 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbeac067990 msgr2=0x7fbeac069e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:28.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.386+0000 7fbec65b8640 1 --2- 172.21.15.116:0/171848925 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbeac067990 0x7fbeac069e50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbeb00ad000 tx=0x7fbeb00a6000 comp rx=0 tx=0).stop 2024-04-25T00:01:28.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.386+0000 7fbec65b8640 1 -- 172.21.15.116:0/171848925 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbec00c5520 msgr2=0x7fbec01ff380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:28.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.386+0000 7fbec65b8640 1 --2- 172.21.15.116:0/171848925 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbec00c5520 0x7fbec01ff380 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fbeb800caa0 tx=0x7fbeb800cf60 comp rx=0 tx=0).stop 2024-04-25T00:01:28.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.386+0000 7fbec65b8640 1 -- 172.21.15.116:0/171848925 shutdown_connections 2024-04-25T00:01:28.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.386+0000 7fbec65b8640 1 --2- 172.21.15.116:0/171848925 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbeac067990 0x7fbeac069e50 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.386+0000 7fbec65b8640 1 --2- 172.21.15.116:0/171848925 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbec00c5520 0x7fbec01ff380 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.386+0000 7fbec65b8640 1 --2- 172.21.15.116:0/171848925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbec00c5100 0x7fbec01fee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.386+0000 7fbec65b8640 1 --2- 172.21.15.116:0/171848925 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbec00c4300 0x7fbec00ac400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.386+0000 7fbec65b8640 1 -- 172.21.15.116:0/171848925 >> 172.21.15.116:0/171848925 conn(0x7fbec006c720 msgr2=0x7fbec006cb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:28.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.386+0000 7fbec65b8640 1 -- 172.21.15.116:0/171848925 shutdown_connections 2024-04-25T00:01:28.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.386+0000 7fbec65b8640 1 -- 172.21.15.116:0/171848925 wait complete. 2024-04-25T00:01:28.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.390+0000 7fe8b67fc640 1 -- 172.21.15.116:0/2044269259 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fe8c402e160 con 0x7fe8c40496e0 2024-04-25T00:01:28.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.390+0000 7fe8c8ff9640 1 -- 172.21.15.116:0/2044269259 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fe8b8101100 con 0x7fe8c40496e0 2024-04-25T00:01:28.392 INFO:teuthology.orchestra.run.smithi116.stdout:64424509450 2024-04-25T00:01:28.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.394+0000 7fe8b67fc640 1 -- 172.21.15.116:0/2044269259 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe8b00678c0 msgr2=0x7fe8b0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:28.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.394+0000 7fe8b67fc640 1 --2- 172.21.15.116:0/2044269259 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe8b00678c0 0x7fe8b0069d80 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe8c0005890 tx=0x7fe8c0009040 comp rx=0 tx=0).stop 2024-04-25T00:01:28.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.394+0000 7fe8b67fc640 1 -- 172.21.15.116:0/2044269259 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8c40496e0 msgr2=0x7fe8c4178840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:28.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.394+0000 7fe8b67fc640 1 --2- 172.21.15.116:0/2044269259 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8c40496e0 0x7fe8c4178840 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe8b80ae040 tx=0x7fe8b8019190 comp rx=0 tx=0).stop 2024-04-25T00:01:28.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.394+0000 7fe8b67fc640 1 -- 172.21.15.116:0/2044269259 shutdown_connections 2024-04-25T00:01:28.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.394+0000 7fe8b67fc640 1 --2- 172.21.15.116:0/2044269259 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe8b00678c0 0x7fe8b0069d80 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.394+0000 7fe8b67fc640 1 --2- 172.21.15.116:0/2044269259 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe8c40d3ab0 0x7fe8c4178d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.394+0000 7fe8b67fc640 1 --2- 172.21.15.116:0/2044269259 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8c40496e0 0x7fe8c4178840 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.394+0000 7fe8b67fc640 1 --2- 172.21.15.116:0/2044269259 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe8c40462f0 0x7fe8c40064e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.394+0000 7fe8b67fc640 1 -- 172.21.15.116:0/2044269259 >> 172.21.15.116:0/2044269259 conn(0x7fe8c4005ea0 msgr2=0x7fe8c402b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:28.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.394+0000 7fe8b67fc640 1 -- 172.21.15.116:0/2044269259 shutdown_connections 2024-04-25T00:01:28.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.394+0000 7fe8b67fc640 1 -- 172.21.15.116:0/2044269259 wait complete. 2024-04-25T00:01:28.396 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509451 got 64424509451 for osd.2 2024-04-25T00:01:28.396 DEBUG:teuthology.parallel:result is None 2024-04-25T00:01:28.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.394+0000 7f82cd7fa640 1 -- 172.21.15.116:0/4178188501 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f82d806f5d0 con 0x7f82d80a9d70 2024-04-25T00:01:28.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.394+0000 7f82cf7fe640 1 -- 172.21.15.116:0/4178188501 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f82d00f01f0 con 0x7f82d80a9d70 2024-04-25T00:01:28.397 INFO:teuthology.orchestra.run.smithi116.stdout:64424509450 2024-04-25T00:01:28.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.398+0000 7f82cd7fa640 1 -- 172.21.15.116:0/4178188501 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f82c4071cb0 msgr2=0x7f82c4074170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:28.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.398+0000 7f82cd7fa640 1 --2- 172.21.15.116:0/4178188501 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f82c4071cb0 0x7f82c4074170 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f82c80047b0 tx=0x7f82c8008040 comp rx=0 tx=0).stop 2024-04-25T00:01:28.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.398+0000 7f82cd7fa640 1 -- 172.21.15.116:0/4178188501 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82d80a9d70 msgr2=0x7f82d80c6470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:28.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.398+0000 7f82cd7fa640 1 --2- 172.21.15.116:0/4178188501 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82d80a9d70 0x7f82d80c6470 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f82d001c5b0 tx=0x7f82c4005840 comp rx=0 tx=0).stop 2024-04-25T00:01:28.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.398+0000 7f82cd7fa640 1 -- 172.21.15.116:0/4178188501 shutdown_connections 2024-04-25T00:01:28.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.398+0000 7f82cd7fa640 1 --2- 172.21.15.116:0/4178188501 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f82c4071cb0 0x7f82c4074170 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.398+0000 7f82cd7fa640 1 --2- 172.21.15.116:0/4178188501 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82d80a9d70 0x7f82d80c6470 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.398+0000 7f82cd7fa640 1 --2- 172.21.15.116:0/4178188501 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f82d81616f0 0x7f82d80c5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.398+0000 7f82cd7fa640 1 --2- 172.21.15.116:0/4178188501 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82c4005380 0x7f82d81603c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.398+0000 7f82cd7fa640 1 -- 172.21.15.116:0/4178188501 >> 172.21.15.116:0/4178188501 conn(0x7f82d80c3940 msgr2=0x7f82d806c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:28.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.398+0000 7f82cd7fa640 1 -- 172.21.15.116:0/4178188501 shutdown_connections 2024-04-25T00:01:28.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.398+0000 7f82cd7fa640 1 -- 172.21.15.116:0/4178188501 wait complete. 2024-04-25T00:01:28.411 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509450 got 64424509450 for osd.7 2024-04-25T00:01:28.412 DEBUG:teuthology.parallel:result is None 2024-04-25T00:01:28.417 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.4 2024-04-25T00:01:28.418 DEBUG:teuthology.parallel:result is None 2024-04-25T00:01:28.418 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509450 got 64424509450 for osd.0 2024-04-25T00:01:28.418 DEBUG:teuthology.parallel:result is None 2024-04-25T00:01:28.426 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509450 got 64424509450 for osd.6 2024-04-25T00:01:28.427 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509450 got 64424509450 for osd.1 2024-04-25T00:01:28.427 DEBUG:teuthology.parallel:result is None 2024-04-25T00:01:28.427 DEBUG:teuthology.parallel:result is None 2024-04-25T00:01:28.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.442+0000 7fb10bfff640 1 -- 172.21.15.116:0/2837989791 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fb0f0005590 con 0x7fb10c0c8f00 2024-04-25T00:01:28.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.442+0000 7fb108ff9640 1 -- 172.21.15.116:0/2837989791 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb0fc0f0940 con 0x7fb10c0c8f00 2024-04-25T00:01:28.446 INFO:teuthology.orchestra.run.smithi116.stdout:64424509450 2024-04-25T00:01:28.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.446+0000 7fb10bfff640 1 -- 172.21.15.116:0/2837989791 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb0f8067b00 msgr2=0x7fb0f8069fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:28.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.446+0000 7fb10bfff640 1 --2- 172.21.15.116:0/2837989791 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb0f8067b00 0x7fb0f8069fc0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb0f4009690 tx=0x7fb0f4006be0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.446+0000 7fb10bfff640 1 -- 172.21.15.116:0/2837989791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb10c0c8f00 msgr2=0x7fb10c15cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:28.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.446+0000 7fb10bfff640 1 --2- 172.21.15.116:0/2837989791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb10c0c8f00 0x7fb10c15cd90 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc01cdb0 tx=0x7fb0fc0192b0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.446+0000 7fb10bfff640 1 -- 172.21.15.116:0/2837989791 shutdown_connections 2024-04-25T00:01:28.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.446+0000 7fb10bfff640 1 --2- 172.21.15.116:0/2837989791 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb0f8067b00 0x7fb0f8069fc0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.446+0000 7fb10bfff640 1 --2- 172.21.15.116:0/2837989791 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb10c157310 0x7fb10c156360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.446+0000 7fb10bfff640 1 --2- 172.21.15.116:0/2837989791 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb10c0c8f00 0x7fb10c15cd90 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.446+0000 7fb10bfff640 1 --2- 172.21.15.116:0/2837989791 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb10c0be730 0x7fb10c0bf490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.446+0000 7fb10bfff640 1 -- 172.21.15.116:0/2837989791 >> 172.21.15.116:0/2837989791 conn(0x7fb10c0acce0 msgr2=0x7fb10c1592e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:28.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.446+0000 7fb10bfff640 1 -- 172.21.15.116:0/2837989791 shutdown_connections 2024-04-25T00:01:28.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.446+0000 7fb10bfff640 1 -- 172.21.15.116:0/2837989791 wait complete. 2024-04-25T00:01:28.462 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509450 got 64424509450 for osd.3 2024-04-25T00:01:28.462 DEBUG:teuthology.parallel:result is None 2024-04-25T00:01:28.462 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-25T00:01:28.462 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T00:01:28.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.606+0000 7fc6577d4640 1 Processor -- start 2024-04-25T00:01:28.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.606+0000 7fc6577d4640 1 -- start start 2024-04-25T00:01:28.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.610+0000 7fc6577d4640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fc6500a94a0 0x7fc6501563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.610+0000 7fc6577d4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc650071680 0x7fc650071ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.610+0000 7fc6577d4640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc650071220 con 0x7fc65014c720 2024-04-25T00:01:28.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.610+0000 7fc6577d4640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fc650153460 con 0x7fc6500a94a0 2024-04-25T00:01:28.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.610+0000 7fc6577d4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fc650152420 con 0x7fc650071680 2024-04-25T00:01:28.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.610+0000 7fc655570640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fc65014c720 0x7fc6500c2a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:41736/0 (socket says 172.21.15.116:41736) 2024-04-25T00:01:28.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.610+0000 7fc655570640 1 -- 172.21.15.116:0/3207312554 learned_addr learned my addr 172.21.15.116:0/3207312554 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:28.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.610+0000 7fc643fff640 1 -- 172.21.15.116:0/3207312554 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1142623766 0 0) 0x7fc650071220 con 0x7fc65014c720 2024-04-25T00:01:28.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.610+0000 7fc643fff640 1 -- 172.21.15.116:0/3207312554 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc63c003530 con 0x7fc65014c720 2024-04-25T00:01:28.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.610+0000 7fc654d6f640 1 --2- 172.21.15.116:0/3207312554 >> v2:172.21.15.174:3300/0 conn(0x7fc6500a94a0 0x7fc6501563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:28.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.610+0000 7fc655d71640 1 --2- 172.21.15.116:0/3207312554 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc650071680 0x7fc650071ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:28.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.610+0000 7fc643fff640 1 -- 172.21.15.116:0/3207312554 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3897311165 0 0) 0x7fc63c003530 con 0x7fc65014c720 2024-04-25T00:01:28.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.610+0000 7fc643fff640 1 -- 172.21.15.116:0/3207312554 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc650071680 msgr2=0x7fc650071ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:28.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.610+0000 7fc643fff640 1 --2- 172.21.15.116:0/3207312554 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc650071680 0x7fc650071ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.610+0000 7fc643fff640 1 -- 172.21.15.116:0/3207312554 >> v2:172.21.15.174:3300/0 conn(0x7fc6500a94a0 msgr2=0x7fc6501563d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:28.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.610+0000 7fc643fff640 1 --2- 172.21.15.116:0/3207312554 >> v2:172.21.15.174:3300/0 conn(0x7fc6500a94a0 0x7fc6501563d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.610+0000 7fc643fff640 1 -- 172.21.15.116:0/3207312554 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc65014cb00 con 0x7fc65014c720 2024-04-25T00:01:28.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.610+0000 7fc643fff640 1 -- 172.21.15.116:0/3207312554 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7fc64c001680 con 0x7fc65014c720 2024-04-25T00:01:28.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.610+0000 7fc643fff640 1 -- 172.21.15.116:0/3207312554 >> v1:172.21.15.116:6789/0 conn(0x7fc65014c720 legacy=0x7fc6500c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:28.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.610+0000 7fc643fff640 1 --2- 172.21.15.116:0/3207312554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc63c0039c0 0x7fc63c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.610+0000 7fc643fff640 1 --2- 172.21.15.116:0/3207312554 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc650071680 0x7fc63c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.610+0000 7fc643fff640 1 --2- 172.21.15.116:0/3207312554 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc63c004d50 0x7fc63c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.610+0000 7fc643fff640 1 -- 172.21.15.116:0/3207312554 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc650153460 con 0x7fc63c0039c0 2024-04-25T00:01:28.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.610+0000 7fc643fff640 1 -- 172.21.15.116:0/3207312554 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc650152420 con 0x7fc63c004d50 2024-04-25T00:01:28.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.610+0000 7fc643fff640 1 -- 172.21.15.116:0/3207312554 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fc63c00bb60 con 0x7fc650071680 2024-04-25T00:01:28.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.610+0000 7fc654d6f640 1 --2- 172.21.15.116:0/3207312554 >> v2:172.21.15.174:3300/0 conn(0x7fc6500a94a0 0x7fc6501563d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:01:28.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.610+0000 7fc655570640 1 --2- 172.21.15.116:0/3207312554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc63c0039c0 0x7fc63c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:28.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.610+0000 7fc655d71640 1 --2- 172.21.15.116:0/3207312554 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc63c004d50 0x7fc63c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:28.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.610+0000 7fc655570640 1 -- 172.21.15.116:0/3207312554 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc650071680 msgr2=0x7fc63c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:28.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.610+0000 7fc655570640 1 --2- 172.21.15.116:0/3207312554 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc650071680 0x7fc63c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.610+0000 7fc655570640 1 -- 172.21.15.116:0/3207312554 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc63c004d50 msgr2=0x7fc63c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:28.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.610+0000 7fc655570640 1 --2- 172.21.15.116:0/3207312554 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc63c004d50 0x7fc63c0051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.610+0000 7fc655570640 1 -- 172.21.15.116:0/3207312554 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc63c003530 con 0x7fc63c0039c0 2024-04-25T00:01:28.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.614+0000 7fc655570640 1 --2- 172.21.15.116:0/3207312554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc63c0039c0 0x7fc63c003de0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc64c00c240 tx=0x7fc64c01cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3d02d060139db5e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:28.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.614+0000 7fc643fff640 1 -- 172.21.15.116:0/3207312554 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc64c007d60 con 0x7fc63c0039c0 2024-04-25T00:01:28.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.614+0000 7fc643fff640 1 -- 172.21.15.116:0/3207312554 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc64c0a9020 con 0x7fc63c0039c0 2024-04-25T00:01:28.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.614+0000 7fc643fff640 1 -- 172.21.15.116:0/3207312554 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc64c005ae0 con 0x7fc63c0039c0 2024-04-25T00:01:28.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.614+0000 7fc6577d4640 1 -- 172.21.15.116:0/3207312554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc63c0039c0 msgr2=0x7fc63c003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:28.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.614+0000 7fc6577d4640 1 --2- 172.21.15.116:0/3207312554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc63c0039c0 0x7fc63c003de0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc64c00c240 tx=0x7fc64c01cc90 comp rx=0 tx=0).stop 2024-04-25T00:01:28.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.614+0000 7fc655d71640 1 -- 172.21.15.116:0/3207312554 reap_dead start 2024-04-25T00:01:28.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.614+0000 7fc6577d4640 1 -- 172.21.15.116:0/3207312554 shutdown_connections 2024-04-25T00:01:28.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.614+0000 7fc6577d4640 1 -- 172.21.15.116:0/3207312554 >> 172.21.15.116:0/3207312554 conn(0x7fc6500c3b00 msgr2=0x7fc6500c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:28.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.614+0000 7fc6577d4640 1 -- 172.21.15.116:0/3207312554 shutdown_connections 2024-04-25T00:01:28.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.614+0000 7fc6577d4640 1 -- 172.21.15.116:0/3207312554 wait complete. 2024-04-25T00:01:28.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.614+0000 7fc6577d4640 1 Processor -- start 2024-04-25T00:01:28.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.614+0000 7fc6577d4640 1 -- start start 2024-04-25T00:01:28.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.614+0000 7fc6577d4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc63c0039c0 0x7fc650140c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.614+0000 7fc6577d4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc65013c790 0x7fc65013cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.614+0000 7fc6577d4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc6500c4bf0 0x7fc6500c5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.614+0000 7fc6577d4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc6500c3120 con 0x7fc63c0039c0 2024-04-25T00:01:28.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.614+0000 7fc6577d4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc650152f40 con 0x7fc65013c790 2024-04-25T00:01:28.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.614+0000 7fc6577d4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fc6500c32a0 con 0x7fc6500c4bf0 2024-04-25T00:01:28.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.614+0000 7fc655570640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc63c0039c0 0x7fc650140c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:28.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.614+0000 7fc655570640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc63c0039c0 0x7fc650140c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56402/0 (socket says 172.21.15.116:56402) 2024-04-25T00:01:28.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.614+0000 7fc655570640 1 -- 172.21.15.116:0/2258098823 learned_addr learned my addr 172.21.15.116:0/2258098823 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:28.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.614+0000 7fc655570640 1 -- 172.21.15.116:0/2258098823 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc6500c4bf0 msgr2=0x7fc6500c5070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:28.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.614+0000 7fc655570640 1 --2- 172.21.15.116:0/2258098823 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc6500c4bf0 0x7fc6500c5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.614+0000 7fc655570640 1 -- 172.21.15.116:0/2258098823 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc65013c790 msgr2=0x7fc65013cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:28.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.614+0000 7fc655570640 1 --2- 172.21.15.116:0/2258098823 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc65013c790 0x7fc65013cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.614+0000 7fc655570640 1 -- 172.21.15.116:0/2258098823 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc650153680 con 0x7fc63c0039c0 2024-04-25T00:01:28.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.614+0000 7fc655570640 1 --2- 172.21.15.116:0/2258098823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc63c0039c0 0x7fc650140c10 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc64c00c240 tx=0x7fc63c005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:28.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.614+0000 7fc641ffb640 1 -- 172.21.15.116:0/2258098823 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc64c0a5a60 con 0x7fc63c0039c0 2024-04-25T00:01:28.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.614+0000 7fc641ffb640 1 -- 172.21.15.116:0/2258098823 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc64c0b3030 con 0x7fc63c0039c0 2024-04-25T00:01:28.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.614+0000 7fc6577d4640 1 -- 172.21.15.116:0/2258098823 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6500c6a70 con 0x7fc63c0039c0 2024-04-25T00:01:28.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.614+0000 7fc641ffb640 1 -- 172.21.15.116:0/2258098823 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc64c0a5e20 con 0x7fc63c0039c0 2024-04-25T00:01:28.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.614+0000 7fc6577d4640 1 -- 172.21.15.116:0/2258098823 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6501ef3e0 con 0x7fc63c0039c0 2024-04-25T00:01:28.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.618+0000 7fc6577d4640 1 -- 172.21.15.116:0/2258098823 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc650068350 con 0x7fc63c0039c0 2024-04-25T00:01:28.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.618+0000 7fc641ffb640 1 -- 172.21.15.116:0/2258098823 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc64c0047f0 con 0x7fc63c0039c0 2024-04-25T00:01:28.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.618+0000 7fc641ffb640 1 --2- 172.21.15.116:0/2258098823 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc6340678f0 0x7fc634069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.618+0000 7fc641ffb640 1 -- 172.21.15.116:0/2258098823 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7fc64c11f580 con 0x7fc63c0039c0 2024-04-25T00:01:28.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.618+0000 7fc654d6f640 1 --2- 172.21.15.116:0/2258098823 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc6340678f0 0x7fc634069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:28.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.618+0000 7fc654d6f640 1 --2- 172.21.15.116:0/2258098823 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc6340678f0 0x7fc634069db0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc650146660 tx=0x7fc644008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:28.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.622+0000 7fc641ffb640 1 -- 172.21.15.116:0/2258098823 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fc64c0169d0 con 0x7fc63c0039c0 2024-04-25T00:01:28.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.766+0000 7fc6577d4640 1 -- 172.21.15.116:0/2258098823 --> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc6500c3640 con 0x7fc6340678f0 2024-04-25T00:01:28.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.802+0000 7fc641ffb640 1 -- 172.21.15.116:0/2258098823 <== mgr.4105 v2:172.21.15.174:6832/2004448211 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+490307 (secure 0 0 0) 0x7fc6500c3640 con 0x7fc6340678f0 2024-04-25T00:01:28.807 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:01:28.810 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-04-25T00:01:28.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.806+0000 7fc6577d4640 1 -- 172.21.15.116:0/2258098823 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc6340678f0 msgr2=0x7fc634069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:28.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.806+0000 7fc6577d4640 1 --2- 172.21.15.116:0/2258098823 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc6340678f0 0x7fc634069db0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc650146660 tx=0x7fc644008040 comp rx=0 tx=0).stop 2024-04-25T00:01:28.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.806+0000 7fc6577d4640 1 -- 172.21.15.116:0/2258098823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc63c0039c0 msgr2=0x7fc650140c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:28.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.806+0000 7fc6577d4640 1 --2- 172.21.15.116:0/2258098823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc63c0039c0 0x7fc650140c10 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc64c00c240 tx=0x7fc63c005840 comp rx=0 tx=0).stop 2024-04-25T00:01:28.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.806+0000 7fc6577d4640 1 -- 172.21.15.116:0/2258098823 shutdown_connections 2024-04-25T00:01:28.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.806+0000 7fc6577d4640 1 --2- 172.21.15.116:0/2258098823 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc6340678f0 0x7fc634069db0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.806+0000 7fc6577d4640 1 --2- 172.21.15.116:0/2258098823 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc6500c4bf0 0x7fc6500c5070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.806+0000 7fc6577d4640 1 --2- 172.21.15.116:0/2258098823 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc65013c790 0x7fc65013cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.806+0000 7fc6577d4640 1 --2- 172.21.15.116:0/2258098823 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc63c0039c0 0x7fc650140c10 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.806+0000 7fc6577d4640 1 -- 172.21.15.116:0/2258098823 >> 172.21.15.116:0/2258098823 conn(0x7fc6500c3b00 msgr2=0x7fc6500723b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:28.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.806+0000 7fc6577d4640 1 -- 172.21.15.116:0/2258098823 shutdown_connections 2024-04-25T00:01:28.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.806+0000 7fc6577d4640 1 -- 172.21.15.116:0/2258098823 wait complete. 2024-04-25T00:01:28.822 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2024-04-25T00:01:27.258101+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595188,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":249,"num_write_kb":2444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":195,"ondisk_log_size":195,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219708,"kb_used_data":3496,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755012,"statfs":{"total":773094113280,"available":772869132288,"internally_reserved":0,"allocated":3579904,"data_stored":2066936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003150"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.577366+0000","last_change":"2024-04-25T00:01:19.508728+0000","last_active":"2024-04-25T00:01:20.577366+0000","last_peered":"2024-04-25T00:01:20.577366+0000","last_clean":"2024-04-25T00:01:20.577366+0000","last_became_active":"2024-04-25T00:01:19.508390+0000","last_became_peered":"2024-04-25T00:01:19.508390+0000","last_unstale":"2024-04-25T00:01:20.577366+0000","last_undegraded":"2024-04-25T00:01:20.577366+0000","last_fullsized":"2024-04-25T00:01:20.577366+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:45.591905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1f","version":"22'4","reported_seq":26,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.515265+0000","last_change":"2024-04-25T00:01:04.356032+0000","last_active":"2024-04-25T00:01:20.515265+0000","last_peered":"2024-04-25T00:01:20.515265+0000","last_clean":"2024-04-25T00:01:20.515265+0000","last_became_active":"2024-04-25T00:01:04.355293+0000","last_became_peered":"2024-04-25T00:01:04.355293+0000","last_unstale":"2024-04-25T00:01:20.515265+0000","last_undegraded":"2024-04-25T00:01:20.515265+0000","last_fullsized":"2024-04-25T00:01:20.515265+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:56:16.678132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.520751+0000","last_change":"2024-04-25T00:01:05.367207+0000","last_active":"2024-04-25T00:01:20.520751+0000","last_peered":"2024-04-25T00:01:20.520751+0000","last_clean":"2024-04-25T00:01:20.520751+0000","last_became_active":"2024-04-25T00:01:05.366471+0000","last_became_peered":"2024-04-25T00:01:05.366471+0000","last_unstale":"2024-04-25T00:01:20.520751+0000","last_undegraded":"2024-04-25T00:01:20.520751+0000","last_fullsized":"2024-04-25T00:01:20.520751+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:30:23.557381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511325+0000","last_change":"2024-04-25T00:01:18.502364+0000","last_active":"2024-04-25T00:01:20.511325+0000","last_peered":"2024-04-25T00:01:20.511325+0000","last_clean":"2024-04-25T00:01:20.511325+0000","last_became_active":"2024-04-25T00:01:18.501633+0000","last_became_peered":"2024-04-25T00:01:18.501633+0000","last_unstale":"2024-04-25T00:01:20.511325+0000","last_undegraded":"2024-04-25T00:01:20.511325+0000","last_fullsized":"2024-04-25T00:01:20.511325+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:55.074018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.510575+0000","last_change":"2024-04-25T00:01:19.512822+0000","last_active":"2024-04-25T00:01:20.510575+0000","last_peered":"2024-04-25T00:01:20.510575+0000","last_clean":"2024-04-25T00:01:20.510575+0000","last_became_active":"2024-04-25T00:01:19.512105+0000","last_became_peered":"2024-04-25T00:01:19.512105+0000","last_unstale":"2024-04-25T00:01:20.510575+0000","last_undegraded":"2024-04-25T00:01:20.510575+0000","last_fullsized":"2024-04-25T00:01:20.510575+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:35:29.722362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"22'1","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.520903+0000","last_change":"2024-04-25T00:01:04.361809+0000","last_active":"2024-04-25T00:01:20.520903+0000","last_peered":"2024-04-25T00:01:20.520903+0000","last_clean":"2024-04-25T00:01:20.520903+0000","last_became_active":"2024-04-25T00:01:04.361059+0000","last_became_peered":"2024-04-25T00:01:04.361059+0000","last_unstale":"2024-04-25T00:01:20.520903+0000","last_undegraded":"2024-04-25T00:01:20.520903+0000","last_fullsized":"2024-04-25T00:01:20.520903+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.520801+0000","last_change":"2024-04-25T00:01:05.370818+0000","last_active":"2024-04-25T00:01:20.520801+0000","last_peered":"2024-04-25T00:01:20.520801+0000","last_clean":"2024-04-25T00:01:20.520801+0000","last_became_active":"2024-04-25T00:01:05.369991+0000","last_became_peered":"2024-04-25T00:01:05.369991+0000","last_unstale":"2024-04-25T00:01:20.520801+0000","last_undegraded":"2024-04-25T00:01:20.520801+0000","last_fullsized":"2024-04-25T00:01:20.520801+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:43.615895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.510695+0000","last_change":"2024-04-25T00:01:18.497682+0000","last_active":"2024-04-25T00:01:20.510695+0000","last_peered":"2024-04-25T00:01:20.510695+0000","last_clean":"2024-04-25T00:01:20.510695+0000","last_became_active":"2024-04-25T00:01:18.497397+0000","last_became_peered":"2024-04-25T00:01:18.497397+0000","last_unstale":"2024-04-25T00:01:20.510695+0000","last_undegraded":"2024-04-25T00:01:20.510695+0000","last_fullsized":"2024-04-25T00:01:20.510695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:22.983972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578826+0000","last_change":"2024-04-25T00:01:19.510591+0000","last_active":"2024-04-25T00:01:20.578826+0000","last_peered":"2024-04-25T00:01:20.578826+0000","last_clean":"2024-04-25T00:01:20.578826+0000","last_became_active":"2024-04-25T00:01:19.510252+0000","last_became_peered":"2024-04-25T00:01:19.510252+0000","last_unstale":"2024-04-25T00:01:20.578826+0000","last_undegraded":"2024-04-25T00:01:20.578826+0000","last_fullsized":"2024-04-25T00:01:20.578826+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:38.005233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"22'2","reported_seq":24,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582530+0000","last_change":"2024-04-25T00:01:04.362030+0000","last_active":"2024-04-25T00:01:20.582530+0000","last_peered":"2024-04-25T00:01:20.582530+0000","last_clean":"2024-04-25T00:01:20.582530+0000","last_became_active":"2024-04-25T00:01:04.361708+0000","last_became_peered":"2024-04-25T00:01:04.361708+0000","last_unstale":"2024-04-25T00:01:20.582530+0000","last_undegraded":"2024-04-25T00:01:20.582530+0000","last_fullsized":"2024-04-25T00:01:20.582530+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:38:33.392719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578882+0000","last_change":"2024-04-25T00:01:05.387195+0000","last_active":"2024-04-25T00:01:20.578882+0000","last_peered":"2024-04-25T00:01:20.578882+0000","last_clean":"2024-04-25T00:01:20.578882+0000","last_became_active":"2024-04-25T00:01:05.386853+0000","last_became_peered":"2024-04-25T00:01:05.386853+0000","last_unstale":"2024-04-25T00:01:20.578882+0000","last_undegraded":"2024-04-25T00:01:20.578882+0000","last_fullsized":"2024-04-25T00:01:20.578882+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:06:19.307378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"26'1","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511288+0000","last_change":"2024-04-25T00:01:18.487835+0000","last_active":"2024-04-25T00:01:20.511288+0000","last_peered":"2024-04-25T00:01:20.511288+0000","last_clean":"2024-04-25T00:01:20.511288+0000","last_became_active":"2024-04-25T00:01:18.487571+0000","last_became_peered":"2024-04-25T00:01:18.487571+0000","last_unstale":"2024-04-25T00:01:20.511288+0000","last_undegraded":"2024-04-25T00:01:20.511288+0000","last_fullsized":"2024-04-25T00:01:20.511288+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:44.582148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511094+0000","last_change":"2024-04-25T00:01:19.519834+0000","last_active":"2024-04-25T00:01:20.511094+0000","last_peered":"2024-04-25T00:01:20.511094+0000","last_clean":"2024-04-25T00:01:20.511094+0000","last_became_active":"2024-04-25T00:01:19.519470+0000","last_became_peered":"2024-04-25T00:01:19.519470+0000","last_unstale":"2024-04-25T00:01:20.511094+0000","last_undegraded":"2024-04-25T00:01:20.511094+0000","last_fullsized":"2024-04-25T00:01:20.511094+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:47.859626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"22'2","reported_seq":24,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582640+0000","last_change":"2024-04-25T00:01:04.362832+0000","last_active":"2024-04-25T00:01:20.582640+0000","last_peered":"2024-04-25T00:01:20.582640+0000","last_clean":"2024-04-25T00:01:20.582640+0000","last_became_active":"2024-04-25T00:01:04.362059+0000","last_became_peered":"2024-04-25T00:01:04.362059+0000","last_unstale":"2024-04-25T00:01:20.582640+0000","last_undegraded":"2024-04-25T00:01:20.582640+0000","last_fullsized":"2024-04-25T00:01:20.582640+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:05.441048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.583017+0000","last_change":"2024-04-25T00:01:05.381120+0000","last_active":"2024-04-25T00:01:20.583017+0000","last_peered":"2024-04-25T00:01:20.583017+0000","last_clean":"2024-04-25T00:01:20.583017+0000","last_became_active":"2024-04-25T00:01:05.378320+0000","last_became_peered":"2024-04-25T00:01:05.378320+0000","last_unstale":"2024-04-25T00:01:20.583017+0000","last_undegraded":"2024-04-25T00:01:20.583017+0000","last_fullsized":"2024-04-25T00:01:20.583017+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:34:06.175918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"26'1","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.510926+0000","last_change":"2024-04-25T00:01:18.490300+0000","last_active":"2024-04-25T00:01:20.510926+0000","last_peered":"2024-04-25T00:01:20.510926+0000","last_clean":"2024-04-25T00:01:20.510926+0000","last_became_active":"2024-04-25T00:01:18.490050+0000","last_became_peered":"2024-04-25T00:01:18.490050+0000","last_unstale":"2024-04-25T00:01:20.510926+0000","last_undegraded":"2024-04-25T00:01:20.510926+0000","last_fullsized":"2024-04-25T00:01:20.510926+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:13:05.971606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.519505+0000","last_change":"2024-04-25T00:01:19.500026+0000","last_active":"2024-04-25T00:01:20.519505+0000","last_peered":"2024-04-25T00:01:20.519505+0000","last_clean":"2024-04-25T00:01:20.519505+0000","last_became_active":"2024-04-25T00:01:19.499244+0000","last_became_peered":"2024-04-25T00:01:19.499244+0000","last_unstale":"2024-04-25T00:01:20.519505+0000","last_undegraded":"2024-04-25T00:01:20.519505+0000","last_fullsized":"2024-04-25T00:01:20.519505+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:40:36.350832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.580552+0000","last_change":"2024-04-25T00:01:05.376984+0000","last_active":"2024-04-25T00:01:20.580552+0000","last_peered":"2024-04-25T00:01:20.580552+0000","last_clean":"2024-04-25T00:01:20.580552+0000","last_became_active":"2024-04-25T00:01:05.376259+0000","last_became_peered":"2024-04-25T00:01:05.376259+0000","last_unstale":"2024-04-25T00:01:20.580552+0000","last_undegraded":"2024-04-25T00:01:20.580552+0000","last_fullsized":"2024-04-25T00:01:20.580552+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:13:28.543302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"22'2","reported_seq":24,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.581267+0000","last_change":"2024-04-25T00:01:04.359262+0000","last_active":"2024-04-25T00:01:20.581267+0000","last_peered":"2024-04-25T00:01:20.581267+0000","last_clean":"2024-04-25T00:01:20.581267+0000","last_became_active":"2024-04-25T00:01:04.357971+0000","last_became_peered":"2024-04-25T00:01:04.357971+0000","last_unstale":"2024-04-25T00:01:20.581267+0000","last_undegraded":"2024-04-25T00:01:20.581267+0000","last_fullsized":"2024-04-25T00:01:20.581267+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:05.441048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"26'1","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582746+0000","last_change":"2024-04-25T00:01:18.475624+0000","last_active":"2024-04-25T00:01:20.582746+0000","last_peered":"2024-04-25T00:01:20.582746+0000","last_clean":"2024-04-25T00:01:20.582746+0000","last_became_active":"2024-04-25T00:01:18.475149+0000","last_became_peered":"2024-04-25T00:01:18.475149+0000","last_unstale":"2024-04-25T00:01:20.582746+0000","last_undegraded":"2024-04-25T00:01:20.582746+0000","last_fullsized":"2024-04-25T00:01:20.582746+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:40:35.341075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.512956+0000","last_change":"2024-04-25T00:01:19.531113+0000","last_active":"2024-04-25T00:01:20.512956+0000","last_peered":"2024-04-25T00:01:20.512956+0000","last_clean":"2024-04-25T00:01:20.512956+0000","last_became_active":"2024-04-25T00:01:19.520150+0000","last_became_peered":"2024-04-25T00:01:19.520150+0000","last_unstale":"2024-04-25T00:01:20.512956+0000","last_undegraded":"2024-04-25T00:01:20.512956+0000","last_fullsized":"2024-04-25T00:01:20.512956+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:38.152376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511363+0000","last_change":"2024-04-25T00:01:05.362611+0000","last_active":"2024-04-25T00:01:20.511363+0000","last_peered":"2024-04-25T00:01:20.511363+0000","last_clean":"2024-04-25T00:01:20.511363+0000","last_became_active":"2024-04-25T00:01:05.361817+0000","last_became_peered":"2024-04-25T00:01:05.361817+0000","last_unstale":"2024-04-25T00:01:20.511363+0000","last_undegraded":"2024-04-25T00:01:20.511363+0000","last_fullsized":"2024-04-25T00:01:20.511363+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:50.489125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578606+0000","last_change":"2024-04-25T00:01:04.371935+0000","last_active":"2024-04-25T00:01:20.578606+0000","last_peered":"2024-04-25T00:01:20.578606+0000","last_clean":"2024-04-25T00:01:20.578606+0000","last_became_active":"2024-04-25T00:01:04.371420+0000","last_became_peered":"2024-04-25T00:01:04.371420+0000","last_unstale":"2024-04-25T00:01:20.578606+0000","last_undegraded":"2024-04-25T00:01:20.578606+0000","last_fullsized":"2024-04-25T00:01:20.578606+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582229+0000","last_change":"2024-04-25T00:01:18.488623+0000","last_active":"2024-04-25T00:01:20.582229+0000","last_peered":"2024-04-25T00:01:20.582229+0000","last_clean":"2024-04-25T00:01:20.582229+0000","last_became_active":"2024-04-25T00:01:18.488291+0000","last_became_peered":"2024-04-25T00:01:18.488291+0000","last_unstale":"2024-04-25T00:01:20.582229+0000","last_undegraded":"2024-04-25T00:01:20.582229+0000","last_fullsized":"2024-04-25T00:01:20.582229+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:30:21.436681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.584490+0000","last_change":"2024-04-25T00:01:19.516731+0000","last_active":"2024-04-25T00:01:20.584490+0000","last_peered":"2024-04-25T00:01:20.584490+0000","last_clean":"2024-04-25T00:01:20.584490+0000","last_became_active":"2024-04-25T00:01:19.516405+0000","last_became_peered":"2024-04-25T00:01:19.516405+0000","last_unstale":"2024-04-25T00:01:20.584490+0000","last_undegraded":"2024-04-25T00:01:20.584490+0000","last_fullsized":"2024-04-25T00:01:20.584490+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:44:37.936092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.510871+0000","last_change":"2024-04-25T00:01:05.355455+0000","last_active":"2024-04-25T00:01:20.510871+0000","last_peered":"2024-04-25T00:01:20.510871+0000","last_clean":"2024-04-25T00:01:20.510871+0000","last_became_active":"2024-04-25T00:01:05.354699+0000","last_became_peered":"2024-04-25T00:01:05.354699+0000","last_unstale":"2024-04-25T00:01:20.510871+0000","last_undegraded":"2024-04-25T00:01:20.510871+0000","last_fullsized":"2024-04-25T00:01:20.510871+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:43:58.069271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.512349+0000","last_change":"2024-04-25T00:01:04.362268+0000","last_active":"2024-04-25T00:01:20.512349+0000","last_peered":"2024-04-25T00:01:20.512349+0000","last_clean":"2024-04-25T00:01:20.512349+0000","last_became_active":"2024-04-25T00:01:04.361622+0000","last_became_peered":"2024-04-25T00:01:04.361622+0000","last_unstale":"2024-04-25T00:01:20.512349+0000","last_undegraded":"2024-04-25T00:01:20.512349+0000","last_fullsized":"2024-04-25T00:01:20.512349+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:58:46.021343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"26'1","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.514314+0000","last_change":"2024-04-25T00:01:18.494047+0000","last_active":"2024-04-25T00:01:20.514314+0000","last_peered":"2024-04-25T00:01:20.514314+0000","last_clean":"2024-04-25T00:01:20.514314+0000","last_became_active":"2024-04-25T00:01:18.493614+0000","last_became_peered":"2024-04-25T00:01:18.493614+0000","last_unstale":"2024-04-25T00:01:20.514314+0000","last_undegraded":"2024-04-25T00:01:20.514314+0000","last_fullsized":"2024-04-25T00:01:20.514314+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:37:08.562016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582182+0000","last_change":"2024-04-25T00:01:19.505397+0000","last_active":"2024-04-25T00:01:20.582182+0000","last_peered":"2024-04-25T00:01:20.582182+0000","last_clean":"2024-04-25T00:01:20.582182+0000","last_became_active":"2024-04-25T00:01:19.505126+0000","last_became_peered":"2024-04-25T00:01:19.505126+0000","last_unstale":"2024-04-25T00:01:20.582182+0000","last_undegraded":"2024-04-25T00:01:20.582182+0000","last_fullsized":"2024-04-25T00:01:20.582182+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:08:49.640267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.581145+0000","last_change":"2024-04-25T00:01:05.354953+0000","last_active":"2024-04-25T00:01:20.581145+0000","last_peered":"2024-04-25T00:01:20.581145+0000","last_clean":"2024-04-25T00:01:20.581145+0000","last_became_active":"2024-04-25T00:01:05.354118+0000","last_became_peered":"2024-04-25T00:01:05.354118+0000","last_unstale":"2024-04-25T00:01:20.581145+0000","last_undegraded":"2024-04-25T00:01:20.581145+0000","last_fullsized":"2024-04-25T00:01:20.581145+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:09.858617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578494+0000","last_change":"2024-04-25T00:01:04.365515+0000","last_active":"2024-04-25T00:01:20.578494+0000","last_peered":"2024-04-25T00:01:20.578494+0000","last_clean":"2024-04-25T00:01:20.578494+0000","last_became_active":"2024-04-25T00:01:04.364942+0000","last_became_peered":"2024-04-25T00:01:04.364942+0000","last_unstale":"2024-04-25T00:01:20.578494+0000","last_undegraded":"2024-04-25T00:01:20.578494+0000","last_fullsized":"2024-04-25T00:01:20.578494+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"26'2","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511306+0000","last_change":"2024-04-25T00:01:18.476207+0000","last_active":"2024-04-25T00:01:20.511306+0000","last_peered":"2024-04-25T00:01:20.511306+0000","last_clean":"2024-04-25T00:01:20.511306+0000","last_became_active":"2024-04-25T00:01:18.475862+0000","last_became_peered":"2024-04-25T00:01:18.475862+0000","last_unstale":"2024-04-25T00:01:20.511306+0000","last_undegraded":"2024-04-25T00:01:20.511306+0000","last_fullsized":"2024-04-25T00:01:20.511306+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:11:40.170249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511258+0000","last_change":"2024-04-25T00:01:19.529607+0000","last_active":"2024-04-25T00:01:20.511258+0000","last_peered":"2024-04-25T00:01:20.511258+0000","last_clean":"2024-04-25T00:01:20.511258+0000","last_became_active":"2024-04-25T00:01:19.528509+0000","last_became_peered":"2024-04-25T00:01:19.528509+0000","last_unstale":"2024-04-25T00:01:20.511258+0000","last_undegraded":"2024-04-25T00:01:20.511258+0000","last_fullsized":"2024-04-25T00:01:20.511258+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:45:01.068091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.514958+0000","last_change":"2024-04-25T00:01:05.358106+0000","last_active":"2024-04-25T00:01:20.514958+0000","last_peered":"2024-04-25T00:01:20.514958+0000","last_clean":"2024-04-25T00:01:20.514958+0000","last_became_active":"2024-04-25T00:01:05.357135+0000","last_became_peered":"2024-04-25T00:01:05.357135+0000","last_unstale":"2024-04-25T00:01:20.514958+0000","last_undegraded":"2024-04-25T00:01:20.514958+0000","last_fullsized":"2024-04-25T00:01:20.514958+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:41:42.228234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"22'2","reported_seq":24,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.584846+0000","last_change":"2024-04-25T00:01:04.369917+0000","last_active":"2024-04-25T00:01:20.584846+0000","last_peered":"2024-04-25T00:01:20.584846+0000","last_clean":"2024-04-25T00:01:20.584846+0000","last_became_active":"2024-04-25T00:01:04.369309+0000","last_became_peered":"2024-04-25T00:01:04.369309+0000","last_unstale":"2024-04-25T00:01:20.584846+0000","last_undegraded":"2024-04-25T00:01:20.584846+0000","last_fullsized":"2024-04-25T00:01:20.584846+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:55:08.063965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511314+0000","last_change":"2024-04-25T00:01:18.477189+0000","last_active":"2024-04-25T00:01:20.511314+0000","last_peered":"2024-04-25T00:01:20.511314+0000","last_clean":"2024-04-25T00:01:20.511314+0000","last_became_active":"2024-04-25T00:01:18.476909+0000","last_became_peered":"2024-04-25T00:01:18.476909+0000","last_unstale":"2024-04-25T00:01:20.511314+0000","last_undegraded":"2024-04-25T00:01:20.511314+0000","last_fullsized":"2024-04-25T00:01:20.511314+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:36.995477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.580550+0000","last_change":"2024-04-25T00:01:19.517500+0000","last_active":"2024-04-25T00:01:20.580550+0000","last_peered":"2024-04-25T00:01:20.580550+0000","last_clean":"2024-04-25T00:01:20.580550+0000","last_became_active":"2024-04-25T00:01:19.517182+0000","last_became_peered":"2024-04-25T00:01:19.517182+0000","last_unstale":"2024-04-25T00:01:20.580550+0000","last_undegraded":"2024-04-25T00:01:20.580550+0000","last_fullsized":"2024-04-25T00:01:20.580550+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:58:49.738233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.515146+0000","last_change":"2024-04-25T00:01:05.364829+0000","last_active":"2024-04-25T00:01:20.515146+0000","last_peered":"2024-04-25T00:01:20.515146+0000","last_clean":"2024-04-25T00:01:20.515146+0000","last_became_active":"2024-04-25T00:01:05.364125+0000","last_became_peered":"2024-04-25T00:01:05.364125+0000","last_unstale":"2024-04-25T00:01:20.515146+0000","last_undegraded":"2024-04-25T00:01:20.515146+0000","last_fullsized":"2024-04-25T00:01:20.515146+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:15:45.531760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"23'1","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.515821+0000","last_change":"2024-04-25T00:01:04.361603+0000","last_active":"2024-04-25T00:01:20.515821+0000","last_peered":"2024-04-25T00:01:20.515821+0000","last_clean":"2024-04-25T00:01:20.515821+0000","last_became_active":"2024-04-25T00:01:04.361046+0000","last_became_peered":"2024-04-25T00:01:04.361046+0000","last_unstale":"2024-04-25T00:01:20.515821+0000","last_undegraded":"2024-04-25T00:01:20.515821+0000","last_fullsized":"2024-04-25T00:01:20.515821+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:35:18.036843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.510620+0000","last_change":"2024-04-25T00:01:18.488352+0000","last_active":"2024-04-25T00:01:20.510620+0000","last_peered":"2024-04-25T00:01:20.510620+0000","last_clean":"2024-04-25T00:01:20.510620+0000","last_became_active":"2024-04-25T00:01:18.487321+0000","last_became_peered":"2024-04-25T00:01:18.487321+0000","last_unstale":"2024-04-25T00:01:20.510620+0000","last_undegraded":"2024-04-25T00:01:20.510620+0000","last_fullsized":"2024-04-25T00:01:20.510620+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:37:22.329655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578359+0000","last_change":"2024-04-25T00:01:19.519653+0000","last_active":"2024-04-25T00:01:20.578359+0000","last_peered":"2024-04-25T00:01:20.578359+0000","last_clean":"2024-04-25T00:01:20.578359+0000","last_became_active":"2024-04-25T00:01:19.519052+0000","last_became_peered":"2024-04-25T00:01:19.519052+0000","last_unstale":"2024-04-25T00:01:20.578359+0000","last_undegraded":"2024-04-25T00:01:20.578359+0000","last_fullsized":"2024-04-25T00:01:20.578359+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:40:36.350832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.579276+0000","last_change":"2024-04-25T00:01:05.374011+0000","last_active":"2024-04-25T00:01:20.579276+0000","last_peered":"2024-04-25T00:01:20.579276+0000","last_clean":"2024-04-25T00:01:20.579276+0000","last_became_active":"2024-04-25T00:01:05.373276+0000","last_became_peered":"2024-04-25T00:01:05.373276+0000","last_unstale":"2024-04-25T00:01:20.579276+0000","last_undegraded":"2024-04-25T00:01:20.579276+0000","last_fullsized":"2024-04-25T00:01:20.579276+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:40:53.349428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.583723+0000","last_change":"2024-04-25T00:01:04.365391+0000","last_active":"2024-04-25T00:01:20.583723+0000","last_peered":"2024-04-25T00:01:20.583723+0000","last_clean":"2024-04-25T00:01:20.583723+0000","last_became_active":"2024-04-25T00:01:04.364989+0000","last_became_peered":"2024-04-25T00:01:04.364989+0000","last_unstale":"2024-04-25T00:01:20.583723+0000","last_undegraded":"2024-04-25T00:01:20.583723+0000","last_fullsized":"2024-04-25T00:01:20.583723+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:06:18.298057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578642+0000","last_change":"2024-04-25T00:01:18.491597+0000","last_active":"2024-04-25T00:01:20.578642+0000","last_peered":"2024-04-25T00:01:20.578642+0000","last_clean":"2024-04-25T00:01:20.578642+0000","last_became_active":"2024-04-25T00:01:18.490859+0000","last_became_peered":"2024-04-25T00:01:18.490859+0000","last_unstale":"2024-04-25T00:01:20.578642+0000","last_undegraded":"2024-04-25T00:01:20.578642+0000","last_fullsized":"2024-04-25T00:01:20.578642+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:36.995477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.512778+0000","last_change":"2024-04-25T00:01:19.532926+0000","last_active":"2024-04-25T00:01:20.512778+0000","last_peered":"2024-04-25T00:01:20.512778+0000","last_clean":"2024-04-25T00:01:20.512778+0000","last_became_active":"2024-04-25T00:01:19.529742+0000","last_became_peered":"2024-04-25T00:01:19.529742+0000","last_unstale":"2024-04-25T00:01:20.512778+0000","last_undegraded":"2024-04-25T00:01:20.512778+0000","last_fullsized":"2024-04-25T00:01:20.512778+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:34:35.706412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511333+0000","last_change":"2024-04-25T00:01:05.378340+0000","last_active":"2024-04-25T00:01:20.511333+0000","last_peered":"2024-04-25T00:01:20.511333+0000","last_clean":"2024-04-25T00:01:20.511333+0000","last_became_active":"2024-04-25T00:01:05.377593+0000","last_became_peered":"2024-04-25T00:01:05.377593+0000","last_unstale":"2024-04-25T00:01:20.511333+0000","last_undegraded":"2024-04-25T00:01:20.511333+0000","last_fullsized":"2024-04-25T00:01:20.511333+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:48.202251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.519859+0000","last_change":"2024-04-25T00:01:04.366577+0000","last_active":"2024-04-25T00:01:20.519859+0000","last_peered":"2024-04-25T00:01:20.519859+0000","last_clean":"2024-04-25T00:01:20.519859+0000","last_became_active":"2024-04-25T00:01:04.365881+0000","last_became_peered":"2024-04-25T00:01:04.365881+0000","last_unstale":"2024-04-25T00:01:20.519859+0000","last_undegraded":"2024-04-25T00:01:20.519859+0000","last_fullsized":"2024-04-25T00:01:20.519859+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:49:22.673421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.577736+0000","last_change":"2024-04-25T00:01:18.504341+0000","last_active":"2024-04-25T00:01:20.577736+0000","last_peered":"2024-04-25T00:01:20.577736+0000","last_clean":"2024-04-25T00:01:20.577736+0000","last_became_active":"2024-04-25T00:01:18.491498+0000","last_became_peered":"2024-04-25T00:01:18.491498+0000","last_unstale":"2024-04-25T00:01:20.577736+0000","last_undegraded":"2024-04-25T00:01:20.577736+0000","last_fullsized":"2024-04-25T00:01:20.577736+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:56.741250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.513341+0000","last_change":"2024-04-25T00:01:19.539074+0000","last_active":"2024-04-25T00:01:20.513341+0000","last_peered":"2024-04-25T00:01:20.513341+0000","last_clean":"2024-04-25T00:01:20.513341+0000","last_became_active":"2024-04-25T00:01:19.538447+0000","last_became_peered":"2024-04-25T00:01:19.538447+0000","last_unstale":"2024-04-25T00:01:20.513341+0000","last_undegraded":"2024-04-25T00:01:20.513341+0000","last_fullsized":"2024-04-25T00:01:20.513341+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:08:49.640267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.581346+0000","last_change":"2024-04-25T00:01:05.368769+0000","last_active":"2024-04-25T00:01:20.581346+0000","last_peered":"2024-04-25T00:01:20.581346+0000","last_clean":"2024-04-25T00:01:20.581346+0000","last_became_active":"2024-04-25T00:01:05.366817+0000","last_became_peered":"2024-04-25T00:01:05.366817+0000","last_unstale":"2024-04-25T00:01:20.581346+0000","last_undegraded":"2024-04-25T00:01:20.581346+0000","last_fullsized":"2024-04-25T00:01:20.581346+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:11:27.044893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.581006+0000","last_change":"2024-04-25T00:01:04.361900+0000","last_active":"2024-04-25T00:01:20.581006+0000","last_peered":"2024-04-25T00:01:20.581006+0000","last_clean":"2024-04-25T00:01:20.581006+0000","last_became_active":"2024-04-25T00:01:04.361572+0000","last_became_peered":"2024-04-25T00:01:04.361572+0000","last_unstale":"2024-04-25T00:01:20.581006+0000","last_undegraded":"2024-04-25T00:01:20.581006+0000","last_fullsized":"2024-04-25T00:01:20.581006+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:29.218739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.579029+0000","last_change":"2024-04-25T00:01:18.504139+0000","last_active":"2024-04-25T00:01:20.579029+0000","last_peered":"2024-04-25T00:01:20.579029+0000","last_clean":"2024-04-25T00:01:20.579029+0000","last_became_active":"2024-04-25T00:01:18.503379+0000","last_became_peered":"2024-04-25T00:01:18.503379+0000","last_unstale":"2024-04-25T00:01:20.579029+0000","last_undegraded":"2024-04-25T00:01:20.579029+0000","last_fullsized":"2024-04-25T00:01:20.579029+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:02:56.363258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.514229+0000","last_change":"2024-04-25T00:01:19.531756+0000","last_active":"2024-04-25T00:01:20.514229+0000","last_peered":"2024-04-25T00:01:20.514229+0000","last_clean":"2024-04-25T00:01:20.514229+0000","last_became_active":"2024-04-25T00:01:19.531145+0000","last_became_peered":"2024-04-25T00:01:19.531145+0000","last_unstale":"2024-04-25T00:01:20.514229+0000","last_undegraded":"2024-04-25T00:01:20.514229+0000","last_fullsized":"2024-04-25T00:01:20.514229+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:13:42.678415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.518780+0000","last_change":"2024-04-25T00:01:05.354557+0000","last_active":"2024-04-25T00:01:20.518780+0000","last_peered":"2024-04-25T00:01:20.518780+0000","last_clean":"2024-04-25T00:01:20.518780+0000","last_became_active":"2024-04-25T00:01:05.353556+0000","last_became_peered":"2024-04-25T00:01:05.353556+0000","last_unstale":"2024-04-25T00:01:20.518780+0000","last_undegraded":"2024-04-25T00:01:20.518780+0000","last_fullsized":"2024-04-25T00:01:20.518780+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:24.964055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511447+0000","last_change":"2024-04-25T00:01:04.372916+0000","last_active":"2024-04-25T00:01:20.511447+0000","last_peered":"2024-04-25T00:01:20.511447+0000","last_clean":"2024-04-25T00:01:20.511447+0000","last_became_active":"2024-04-25T00:01:04.372185+0000","last_became_peered":"2024-04-25T00:01:04.372185+0000","last_unstale":"2024-04-25T00:01:20.511447+0000","last_undegraded":"2024-04-25T00:01:20.511447+0000","last_fullsized":"2024-04-25T00:01:20.511447+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"26'2","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.579299+0000","last_change":"2024-04-25T00:01:18.504614+0000","last_active":"2024-04-25T00:01:20.579299+0000","last_peered":"2024-04-25T00:01:20.579299+0000","last_clean":"2024-04-25T00:01:20.579299+0000","last_became_active":"2024-04-25T00:01:18.501448+0000","last_became_peered":"2024-04-25T00:01:18.501448+0000","last_unstale":"2024-04-25T00:01:20.579299+0000","last_undegraded":"2024-04-25T00:01:20.579299+0000","last_fullsized":"2024-04-25T00:01:20.579299+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:22.983972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.584367+0000","last_change":"2024-04-25T00:01:19.517510+0000","last_active":"2024-04-25T00:01:20.584367+0000","last_peered":"2024-04-25T00:01:20.584367+0000","last_clean":"2024-04-25T00:01:20.584367+0000","last_became_active":"2024-04-25T00:01:19.517243+0000","last_became_peered":"2024-04-25T00:01:19.517243+0000","last_unstale":"2024-04-25T00:01:20.584367+0000","last_undegraded":"2024-04-25T00:01:20.584367+0000","last_fullsized":"2024-04-25T00:01:20.584367+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:42:38.149502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.577555+0000","last_change":"2024-04-25T00:01:05.370968+0000","last_active":"2024-04-25T00:01:20.577555+0000","last_peered":"2024-04-25T00:01:20.577555+0000","last_clean":"2024-04-25T00:01:20.577555+0000","last_became_active":"2024-04-25T00:01:05.369987+0000","last_became_peered":"2024-04-25T00:01:05.369987+0000","last_unstale":"2024-04-25T00:01:20.577555+0000","last_undegraded":"2024-04-25T00:01:20.577555+0000","last_fullsized":"2024-04-25T00:01:20.577555+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.580498+0000","last_change":"2024-04-25T00:01:04.370226+0000","last_active":"2024-04-25T00:01:20.580498+0000","last_peered":"2024-04-25T00:01:20.580498+0000","last_clean":"2024-04-25T00:01:20.580498+0000","last_became_active":"2024-04-25T00:01:04.369904+0000","last_became_peered":"2024-04-25T00:01:04.369904+0000","last_unstale":"2024-04-25T00:01:20.580498+0000","last_undegraded":"2024-04-25T00:01:20.580498+0000","last_fullsized":"2024-04-25T00:01:20.580498+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:47:25.595275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"27'1","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.523846+0000","last_change":"2024-04-25T00:01:18.497156+0000","last_active":"2024-04-25T00:01:20.523846+0000","last_peered":"2024-04-25T00:01:20.523846+0000","last_clean":"2024-04-25T00:01:20.523846+0000","last_became_active":"2024-04-25T00:01:18.495573+0000","last_became_peered":"2024-04-25T00:01:18.495573+0000","last_unstale":"2024-04-25T00:01:20.523846+0000","last_undegraded":"2024-04-25T00:01:20.523846+0000","last_fullsized":"2024-04-25T00:01:20.523846+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:29.119299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.581909+0000","last_change":"2024-04-25T00:01:19.519411+0000","last_active":"2024-04-25T00:01:20.581909+0000","last_peered":"2024-04-25T00:01:20.581909+0000","last_clean":"2024-04-25T00:01:20.581909+0000","last_became_active":"2024-04-25T00:01:19.519020+0000","last_became_peered":"2024-04-25T00:01:19.519020+0000","last_unstale":"2024-04-25T00:01:20.581909+0000","last_undegraded":"2024-04-25T00:01:20.581909+0000","last_fullsized":"2024-04-25T00:01:20.581909+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:50.351599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.513395+0000","last_change":"2024-04-25T00:01:05.364939+0000","last_active":"2024-04-25T00:01:20.513395+0000","last_peered":"2024-04-25T00:01:20.513395+0000","last_clean":"2024-04-25T00:01:20.513395+0000","last_became_active":"2024-04-25T00:01:05.364272+0000","last_became_peered":"2024-04-25T00:01:05.364272+0000","last_unstale":"2024-04-25T00:01:20.513395+0000","last_undegraded":"2024-04-25T00:01:20.513395+0000","last_fullsized":"2024-04-25T00:01:20.513395+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:24:59.679908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.581536+0000","last_change":"2024-04-25T00:01:04.357028+0000","last_active":"2024-04-25T00:01:20.581536+0000","last_peered":"2024-04-25T00:01:20.581536+0000","last_clean":"2024-04-25T00:01:20.581536+0000","last_became_active":"2024-04-25T00:01:04.356644+0000","last_became_peered":"2024-04-25T00:01:04.356644+0000","last_unstale":"2024-04-25T00:01:20.581536+0000","last_undegraded":"2024-04-25T00:01:20.581536+0000","last_fullsized":"2024-04-25T00:01:20.581536+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:21.729893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":178,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.577513+0000","last_change":"2024-04-25T00:00:55.084492+0000","last_active":"2024-04-25T00:01:20.577513+0000","last_peered":"2024-04-25T00:01:20.577513+0000","last_clean":"2024-04-25T00:01:20.577513+0000","last_became_active":"2024-04-25T00:00:55.083493+0000","last_became_peered":"2024-04-25T00:00:55.083493+0000","last_unstale":"2024-04-25T00:01:20.577513+0000","last_undegraded":"2024-04-25T00:01:20.577513+0000","last_fullsized":"2024-04-25T00:01:20.577513+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:00:54.060578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:00:54.060578+0000","last_clean_scrub_stamp":"2024-04-25T00:00:54.060578+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:14:58.472477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.583214+0000","last_change":"2024-04-25T00:01:18.476298+0000","last_active":"2024-04-25T00:01:20.583214+0000","last_peered":"2024-04-25T00:01:20.583214+0000","last_clean":"2024-04-25T00:01:20.583214+0000","last_became_active":"2024-04-25T00:01:18.475898+0000","last_became_peered":"2024-04-25T00:01:18.475898+0000","last_unstale":"2024-04-25T00:01:20.583214+0000","last_undegraded":"2024-04-25T00:01:20.583214+0000","last_fullsized":"2024-04-25T00:01:20.583214+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:49.341842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.519562+0000","last_change":"2024-04-25T00:01:19.518102+0000","last_active":"2024-04-25T00:01:20.519562+0000","last_peered":"2024-04-25T00:01:20.519562+0000","last_clean":"2024-04-25T00:01:20.519562+0000","last_became_active":"2024-04-25T00:01:19.517376+0000","last_became_peered":"2024-04-25T00:01:19.517376+0000","last_unstale":"2024-04-25T00:01:20.519562+0000","last_undegraded":"2024-04-25T00:01:20.519562+0000","last_fullsized":"2024-04-25T00:01:20.519562+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:55:46.853565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.583333+0000","last_change":"2024-04-25T00:01:05.376609+0000","last_active":"2024-04-25T00:01:20.583333+0000","last_peered":"2024-04-25T00:01:20.583333+0000","last_clean":"2024-04-25T00:01:20.583333+0000","last_became_active":"2024-04-25T00:01:05.375881+0000","last_became_peered":"2024-04-25T00:01:05.375881+0000","last_unstale":"2024-04-25T00:01:20.583333+0000","last_undegraded":"2024-04-25T00:01:20.583333+0000","last_fullsized":"2024-04-25T00:01:20.583333+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:46:05.592233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.579387+0000","last_change":"2024-04-25T00:01:04.368954+0000","last_active":"2024-04-25T00:01:20.579387+0000","last_peered":"2024-04-25T00:01:20.579387+0000","last_clean":"2024-04-25T00:01:20.579387+0000","last_became_active":"2024-04-25T00:01:04.367903+0000","last_became_peered":"2024-04-25T00:01:04.367903+0000","last_unstale":"2024-04-25T00:01:20.579387+0000","last_undegraded":"2024-04-25T00:01:20.579387+0000","last_fullsized":"2024-04-25T00:01:20.579387+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511154+0000","last_change":"2024-04-25T00:01:18.476525+0000","last_active":"2024-04-25T00:01:20.511154+0000","last_peered":"2024-04-25T00:01:20.511154+0000","last_clean":"2024-04-25T00:01:20.511154+0000","last_became_active":"2024-04-25T00:01:18.476247+0000","last_became_peered":"2024-04-25T00:01:18.476247+0000","last_unstale":"2024-04-25T00:01:20.511154+0000","last_undegraded":"2024-04-25T00:01:20.511154+0000","last_fullsized":"2024-04-25T00:01:20.511154+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:30:36.682736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.577886+0000","last_change":"2024-04-25T00:01:19.514615+0000","last_active":"2024-04-25T00:01:20.577886+0000","last_peered":"2024-04-25T00:01:20.577886+0000","last_clean":"2024-04-25T00:01:20.577886+0000","last_became_active":"2024-04-25T00:01:19.513640+0000","last_became_peered":"2024-04-25T00:01:19.513640+0000","last_unstale":"2024-04-25T00:01:20.577886+0000","last_undegraded":"2024-04-25T00:01:20.577886+0000","last_fullsized":"2024-04-25T00:01:20.577886+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:08.620602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582498+0000","last_change":"2024-04-25T00:01:05.385670+0000","last_active":"2024-04-25T00:01:20.582498+0000","last_peered":"2024-04-25T00:01:20.582498+0000","last_clean":"2024-04-25T00:01:20.582498+0000","last_became_active":"2024-04-25T00:01:05.384822+0000","last_became_peered":"2024-04-25T00:01:05.384822+0000","last_unstale":"2024-04-25T00:01:20.582498+0000","last_undegraded":"2024-04-25T00:01:20.582498+0000","last_fullsized":"2024-04-25T00:01:20.582498+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:24.017263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"22'1","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.520366+0000","last_change":"2024-04-25T00:01:04.356627+0000","last_active":"2024-04-25T00:01:20.520366+0000","last_peered":"2024-04-25T00:01:20.520366+0000","last_clean":"2024-04-25T00:01:20.520366+0000","last_became_active":"2024-04-25T00:01:04.354419+0000","last_became_peered":"2024-04-25T00:01:04.354419+0000","last_unstale":"2024-04-25T00:01:20.520366+0000","last_undegraded":"2024-04-25T00:01:20.520366+0000","last_fullsized":"2024-04-25T00:01:20.520366+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:26.243635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"26'2","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.519762+0000","last_change":"2024-04-25T00:01:18.490511+0000","last_active":"2024-04-25T00:01:20.519762+0000","last_peered":"2024-04-25T00:01:20.519762+0000","last_clean":"2024-04-25T00:01:20.519762+0000","last_became_active":"2024-04-25T00:01:18.489629+0000","last_became_peered":"2024-04-25T00:01:18.489629+0000","last_unstale":"2024-04-25T00:01:20.519762+0000","last_undegraded":"2024-04-25T00:01:20.519762+0000","last_fullsized":"2024-04-25T00:01:20.519762+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:07.610846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.579469+0000","last_change":"2024-04-25T00:01:19.508931+0000","last_active":"2024-04-25T00:01:20.579469+0000","last_peered":"2024-04-25T00:01:20.579469+0000","last_clean":"2024-04-25T00:01:20.579469+0000","last_became_active":"2024-04-25T00:01:19.507867+0000","last_became_peered":"2024-04-25T00:01:19.507867+0000","last_unstale":"2024-04-25T00:01:20.579469+0000","last_undegraded":"2024-04-25T00:01:20.579469+0000","last_fullsized":"2024-04-25T00:01:20.579469+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:38.152376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.583381+0000","last_change":"2024-04-25T00:01:05.378840+0000","last_active":"2024-04-25T00:01:20.583381+0000","last_peered":"2024-04-25T00:01:20.583381+0000","last_clean":"2024-04-25T00:01:20.583381+0000","last_became_active":"2024-04-25T00:01:05.378112+0000","last_became_peered":"2024-04-25T00:01:05.378112+0000","last_unstale":"2024-04-25T00:01:20.583381+0000","last_undegraded":"2024-04-25T00:01:20.583381+0000","last_fullsized":"2024-04-25T00:01:20.583381+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:43.615895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"22'2","reported_seq":24,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.515607+0000","last_change":"2024-04-25T00:01:04.370794+0000","last_active":"2024-04-25T00:01:20.515607+0000","last_peered":"2024-04-25T00:01:20.515607+0000","last_clean":"2024-04-25T00:01:20.515607+0000","last_became_active":"2024-04-25T00:01:04.370099+0000","last_became_peered":"2024-04-25T00:01:04.370099+0000","last_unstale":"2024-04-25T00:01:20.515607+0000","last_undegraded":"2024-04-25T00:01:20.515607+0000","last_fullsized":"2024-04-25T00:01:20.515607+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:42:03.181036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.579618+0000","last_change":"2024-04-25T00:01:18.475256+0000","last_active":"2024-04-25T00:01:20.579618+0000","last_peered":"2024-04-25T00:01:20.579618+0000","last_clean":"2024-04-25T00:01:20.579618+0000","last_became_active":"2024-04-25T00:01:18.474952+0000","last_became_peered":"2024-04-25T00:01:18.474952+0000","last_unstale":"2024-04-25T00:01:20.579618+0000","last_undegraded":"2024-04-25T00:01:20.579618+0000","last_fullsized":"2024-04-25T00:01:20.579618+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:46.849870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.577592+0000","last_change":"2024-04-25T00:01:19.516833+0000","last_active":"2024-04-25T00:01:20.577592+0000","last_peered":"2024-04-25T00:01:20.577592+0000","last_clean":"2024-04-25T00:01:20.577592+0000","last_became_active":"2024-04-25T00:01:19.515926+0000","last_became_peered":"2024-04-25T00:01:19.515926+0000","last_unstale":"2024-04-25T00:01:20.577592+0000","last_undegraded":"2024-04-25T00:01:20.577592+0000","last_fullsized":"2024-04-25T00:01:20.577592+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:23:48.855381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511807+0000","last_change":"2024-04-25T00:01:05.372591+0000","last_active":"2024-04-25T00:01:20.511807+0000","last_peered":"2024-04-25T00:01:20.511807+0000","last_clean":"2024-04-25T00:01:20.511807+0000","last_became_active":"2024-04-25T00:01:05.371204+0000","last_became_peered":"2024-04-25T00:01:05.371204+0000","last_unstale":"2024-04-25T00:01:20.511807+0000","last_undegraded":"2024-04-25T00:01:20.511807+0000","last_fullsized":"2024-04-25T00:01:20.511807+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:15.879673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"22'1","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.583451+0000","last_change":"2024-04-25T00:01:04.372376+0000","last_active":"2024-04-25T00:01:20.583451+0000","last_peered":"2024-04-25T00:01:20.583451+0000","last_clean":"2024-04-25T00:01:20.583451+0000","last_became_active":"2024-04-25T00:01:04.372040+0000","last_became_peered":"2024-04-25T00:01:04.372040+0000","last_unstale":"2024-04-25T00:01:20.583451+0000","last_undegraded":"2024-04-25T00:01:20.583451+0000","last_fullsized":"2024-04-25T00:01:20.583451+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:21.729893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578134+0000","last_change":"2024-04-25T00:01:18.487548+0000","last_active":"2024-04-25T00:01:20.578134+0000","last_peered":"2024-04-25T00:01:20.578134+0000","last_clean":"2024-04-25T00:01:20.578134+0000","last_became_active":"2024-04-25T00:01:18.487227+0000","last_became_peered":"2024-04-25T00:01:18.487227+0000","last_unstale":"2024-04-25T00:01:20.578134+0000","last_undegraded":"2024-04-25T00:01:20.578134+0000","last_fullsized":"2024-04-25T00:01:20.578134+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:55.074018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.514828+0000","last_change":"2024-04-25T00:01:19.528574+0000","last_active":"2024-04-25T00:01:20.514828+0000","last_peered":"2024-04-25T00:01:20.514828+0000","last_clean":"2024-04-25T00:01:20.514828+0000","last_became_active":"2024-04-25T00:01:19.525531+0000","last_became_peered":"2024-04-25T00:01:19.525531+0000","last_unstale":"2024-04-25T00:01:20.514828+0000","last_undegraded":"2024-04-25T00:01:20.514828+0000","last_fullsized":"2024-04-25T00:01:20.514828+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:47:08.132895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.580931+0000","last_change":"2024-04-25T00:01:05.364103+0000","last_active":"2024-04-25T00:01:20.580931+0000","last_peered":"2024-04-25T00:01:20.580931+0000","last_clean":"2024-04-25T00:01:20.580931+0000","last_became_active":"2024-04-25T00:01:05.363320+0000","last_became_peered":"2024-04-25T00:01:05.363320+0000","last_unstale":"2024-04-25T00:01:20.580931+0000","last_undegraded":"2024-04-25T00:01:20.580931+0000","last_fullsized":"2024-04-25T00:01:20.580931+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:31.456792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"22'8","reported_seq":30,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511508+0000","last_change":"2024-04-25T00:01:04.358365+0000","last_active":"2024-04-25T00:01:20.511508+0000","last_peered":"2024-04-25T00:01:20.511508+0000","last_clean":"2024-04-25T00:01:20.511508+0000","last_became_active":"2024-04-25T00:01:04.357571+0000","last_became_peered":"2024-04-25T00:01:04.357571+0000","last_unstale":"2024-04-25T00:01:20.511508+0000","last_undegraded":"2024-04-25T00:01:20.511508+0000","last_fullsized":"2024-04-25T00:01:20.511508+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:50:23.023114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.520269+0000","last_change":"2024-04-25T00:01:18.497679+0000","last_active":"2024-04-25T00:01:20.520269+0000","last_peered":"2024-04-25T00:01:20.520269+0000","last_clean":"2024-04-25T00:01:20.520269+0000","last_became_active":"2024-04-25T00:01:18.496489+0000","last_became_peered":"2024-04-25T00:01:18.496489+0000","last_unstale":"2024-04-25T00:01:20.520269+0000","last_undegraded":"2024-04-25T00:01:20.520269+0000","last_fullsized":"2024-04-25T00:01:20.520269+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:23:47.845624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.584174+0000","last_change":"2024-04-25T00:01:19.498082+0000","last_active":"2024-04-25T00:01:20.584174+0000","last_peered":"2024-04-25T00:01:20.584174+0000","last_clean":"2024-04-25T00:01:20.584174+0000","last_became_active":"2024-04-25T00:01:19.496805+0000","last_became_peered":"2024-04-25T00:01:19.496805+0000","last_unstale":"2024-04-25T00:01:20.584174+0000","last_undegraded":"2024-04-25T00:01:20.584174+0000","last_fullsized":"2024-04-25T00:01:20.584174+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:57:36.256250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.577453+0000","last_change":"2024-04-25T00:01:05.373927+0000","last_active":"2024-04-25T00:01:20.577453+0000","last_peered":"2024-04-25T00:01:20.577453+0000","last_clean":"2024-04-25T00:01:20.577453+0000","last_became_active":"2024-04-25T00:01:05.373077+0000","last_became_peered":"2024-04-25T00:01:05.373077+0000","last_unstale":"2024-04-25T00:01:20.577453+0000","last_undegraded":"2024-04-25T00:01:20.577453+0000","last_fullsized":"2024-04-25T00:01:20.577453+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:50.489125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578902+0000","last_change":"2024-04-25T00:01:04.357265+0000","last_active":"2024-04-25T00:01:20.578902+0000","last_peered":"2024-04-25T00:01:20.578902+0000","last_clean":"2024-04-25T00:01:20.578902+0000","last_became_active":"2024-04-25T00:01:04.357004+0000","last_became_peered":"2024-04-25T00:01:04.357004+0000","last_unstale":"2024-04-25T00:01:20.578902+0000","last_undegraded":"2024-04-25T00:01:20.578902+0000","last_fullsized":"2024-04-25T00:01:20.578902+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:47:25.595275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.520612+0000","last_change":"2024-04-25T00:01:18.487276+0000","last_active":"2024-04-25T00:01:20.520612+0000","last_peered":"2024-04-25T00:01:20.520612+0000","last_clean":"2024-04-25T00:01:20.520612+0000","last_became_active":"2024-04-25T00:01:18.486279+0000","last_became_peered":"2024-04-25T00:01:18.486279+0000","last_unstale":"2024-04-25T00:01:20.520612+0000","last_undegraded":"2024-04-25T00:01:20.520612+0000","last_fullsized":"2024-04-25T00:01:20.520612+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:17:37.141843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582117+0000","last_change":"2024-04-25T00:01:19.516827+0000","last_active":"2024-04-25T00:01:20.582117+0000","last_peered":"2024-04-25T00:01:20.582117+0000","last_clean":"2024-04-25T00:01:20.582117+0000","last_became_active":"2024-04-25T00:01:19.515765+0000","last_became_peered":"2024-04-25T00:01:19.515765+0000","last_unstale":"2024-04-25T00:01:20.582117+0000","last_undegraded":"2024-04-25T00:01:20.582117+0000","last_fullsized":"2024-04-25T00:01:20.582117+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:13:34.153117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.579960+0000","last_change":"2024-04-25T00:01:05.379501+0000","last_active":"2024-04-25T00:01:20.579960+0000","last_peered":"2024-04-25T00:01:20.579960+0000","last_clean":"2024-04-25T00:01:20.579960+0000","last_became_active":"2024-04-25T00:01:05.378644+0000","last_became_peered":"2024-04-25T00:01:05.378644+0000","last_unstale":"2024-04-25T00:01:20.579960+0000","last_undegraded":"2024-04-25T00:01:20.579960+0000","last_fullsized":"2024-04-25T00:01:20.579960+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:08:35.505154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.580791+0000","last_change":"2024-04-25T00:01:04.360987+0000","last_active":"2024-04-25T00:01:20.580791+0000","last_peered":"2024-04-25T00:01:20.580791+0000","last_clean":"2024-04-25T00:01:20.580791+0000","last_became_active":"2024-04-25T00:01:04.360664+0000","last_became_peered":"2024-04-25T00:01:04.360664+0000","last_unstale":"2024-04-25T00:01:20.580791+0000","last_undegraded":"2024-04-25T00:01:20.580791+0000","last_fullsized":"2024-04-25T00:01:20.580791+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:55.283076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"26'6","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578827+0000","last_change":"2024-04-25T00:01:18.505577+0000","last_active":"2024-04-25T00:01:20.578827+0000","last_peered":"2024-04-25T00:01:20.578827+0000","last_clean":"2024-04-25T00:01:20.578827+0000","last_became_active":"2024-04-25T00:01:18.504963+0000","last_became_peered":"2024-04-25T00:01:18.504963+0000","last_unstale":"2024-04-25T00:01:20.578827+0000","last_undegraded":"2024-04-25T00:01:20.578827+0000","last_fullsized":"2024-04-25T00:01:20.578827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:37.142619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.513446+0000","last_change":"2024-04-25T00:01:19.524723+0000","last_active":"2024-04-25T00:01:20.513446+0000","last_peered":"2024-04-25T00:01:20.513446+0000","last_clean":"2024-04-25T00:01:20.513446+0000","last_became_active":"2024-04-25T00:01:19.523753+0000","last_became_peered":"2024-04-25T00:01:19.523753+0000","last_unstale":"2024-04-25T00:01:20.513446+0000","last_undegraded":"2024-04-25T00:01:20.513446+0000","last_fullsized":"2024-04-25T00:01:20.513446+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:19:17.969671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578487+0000","last_change":"2024-04-25T00:01:05.379102+0000","last_active":"2024-04-25T00:01:20.578487+0000","last_peered":"2024-04-25T00:01:20.578487+0000","last_clean":"2024-04-25T00:01:20.578487+0000","last_became_active":"2024-04-25T00:01:05.377625+0000","last_became_peered":"2024-04-25T00:01:05.377625+0000","last_unstale":"2024-04-25T00:01:20.578487+0000","last_undegraded":"2024-04-25T00:01:20.578487+0000","last_fullsized":"2024-04-25T00:01:20.578487+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:15.879673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"22'2","reported_seq":24,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511726+0000","last_change":"2024-04-25T00:01:04.370640+0000","last_active":"2024-04-25T00:01:20.511726+0000","last_peered":"2024-04-25T00:01:20.511726+0000","last_clean":"2024-04-25T00:01:20.511726+0000","last_became_active":"2024-04-25T00:01:04.370031+0000","last_became_peered":"2024-04-25T00:01:04.370031+0000","last_unstale":"2024-04-25T00:01:20.511726+0000","last_undegraded":"2024-04-25T00:01:20.511726+0000","last_fullsized":"2024-04-25T00:01:20.511726+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:26.243635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"26'3","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582955+0000","last_change":"2024-04-25T00:01:18.473113+0000","last_active":"2024-04-25T00:01:20.582955+0000","last_peered":"2024-04-25T00:01:20.582955+0000","last_clean":"2024-04-25T00:01:20.582955+0000","last_became_active":"2024-04-25T00:01:18.472931+0000","last_became_peered":"2024-04-25T00:01:18.472931+0000","last_unstale":"2024-04-25T00:01:20.582955+0000","last_undegraded":"2024-04-25T00:01:20.582955+0000","last_fullsized":"2024-04-25T00:01:20.582955+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:07.498985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.512763+0000","last_change":"2024-04-25T00:01:19.533664+0000","last_active":"2024-04-25T00:01:20.512763+0000","last_peered":"2024-04-25T00:01:20.512763+0000","last_clean":"2024-04-25T00:01:20.512763+0000","last_became_active":"2024-04-25T00:01:19.533048+0000","last_became_peered":"2024-04-25T00:01:19.533048+0000","last_unstale":"2024-04-25T00:01:20.512763+0000","last_undegraded":"2024-04-25T00:01:20.512763+0000","last_fullsized":"2024-04-25T00:01:20.512763+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:45:43.141569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582298+0000","last_change":"2024-04-25T00:01:05.385827+0000","last_active":"2024-04-25T00:01:20.582298+0000","last_peered":"2024-04-25T00:01:20.582298+0000","last_clean":"2024-04-25T00:01:20.582298+0000","last_became_active":"2024-04-25T00:01:05.385258+0000","last_became_peered":"2024-04-25T00:01:05.385258+0000","last_unstale":"2024-04-25T00:01:20.582298+0000","last_undegraded":"2024-04-25T00:01:20.582298+0000","last_fullsized":"2024-04-25T00:01:20.582298+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:23.870121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"22'6","reported_seq":28,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.512128+0000","last_change":"2024-04-25T00:01:04.365185+0000","last_active":"2024-04-25T00:01:20.512128+0000","last_peered":"2024-04-25T00:01:20.512128+0000","last_clean":"2024-04-25T00:01:20.512128+0000","last_became_active":"2024-04-25T00:01:04.364930+0000","last_became_peered":"2024-04-25T00:01:04.364930+0000","last_unstale":"2024-04-25T00:01:20.512128+0000","last_undegraded":"2024-04-25T00:01:20.512128+0000","last_fullsized":"2024-04-25T00:01:20.512128+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:45:03.618340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.512982+0000","last_change":"2024-04-25T00:01:18.496220+0000","last_active":"2024-04-25T00:01:20.512982+0000","last_peered":"2024-04-25T00:01:20.512982+0000","last_clean":"2024-04-25T00:01:20.512982+0000","last_became_active":"2024-04-25T00:01:18.492850+0000","last_became_peered":"2024-04-25T00:01:18.492850+0000","last_unstale":"2024-04-25T00:01:20.512982+0000","last_undegraded":"2024-04-25T00:01:20.512982+0000","last_fullsized":"2024-04-25T00:01:20.512982+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:50:16.922314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578332+0000","last_change":"2024-04-25T00:01:19.515725+0000","last_active":"2024-04-25T00:01:20.578332+0000","last_peered":"2024-04-25T00:01:20.578332+0000","last_clean":"2024-04-25T00:01:20.578332+0000","last_became_active":"2024-04-25T00:01:19.509285+0000","last_became_peered":"2024-04-25T00:01:19.509285+0000","last_unstale":"2024-04-25T00:01:20.578332+0000","last_undegraded":"2024-04-25T00:01:20.578332+0000","last_fullsized":"2024-04-25T00:01:20.578332+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:25:20.185355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.581530+0000","last_change":"2024-04-25T00:01:05.371061+0000","last_active":"2024-04-25T00:01:20.581530+0000","last_peered":"2024-04-25T00:01:20.581530+0000","last_clean":"2024-04-25T00:01:20.581530+0000","last_became_active":"2024-04-25T00:01:05.369950+0000","last_became_peered":"2024-04-25T00:01:05.369950+0000","last_unstale":"2024-04-25T00:01:20.581530+0000","last_undegraded":"2024-04-25T00:01:20.581530+0000","last_fullsized":"2024-04-25T00:01:20.581530+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:02:43.237902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"22'3","reported_seq":25,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.513687+0000","last_change":"2024-04-25T00:01:04.366503+0000","last_active":"2024-04-25T00:01:20.513687+0000","last_peered":"2024-04-25T00:01:20.513687+0000","last_clean":"2024-04-25T00:01:20.513687+0000","last_became_active":"2024-04-25T00:01:04.365945+0000","last_became_peered":"2024-04-25T00:01:04.365945+0000","last_unstale":"2024-04-25T00:01:20.513687+0000","last_undegraded":"2024-04-25T00:01:20.513687+0000","last_fullsized":"2024-04-25T00:01:20.513687+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"26'2","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511434+0000","last_change":"2024-04-25T00:01:18.476456+0000","last_active":"2024-04-25T00:01:20.511434+0000","last_peered":"2024-04-25T00:01:20.511434+0000","last_clean":"2024-04-25T00:01:20.511434+0000","last_became_active":"2024-04-25T00:01:18.475876+0000","last_became_peered":"2024-04-25T00:01:18.475876+0000","last_unstale":"2024-04-25T00:01:20.511434+0000","last_undegraded":"2024-04-25T00:01:20.511434+0000","last_fullsized":"2024-04-25T00:01:20.511434+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:56.741250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.580191+0000","last_change":"2024-04-25T00:01:19.515376+0000","last_active":"2024-04-25T00:01:20.580191+0000","last_peered":"2024-04-25T00:01:20.580191+0000","last_clean":"2024-04-25T00:01:20.580191+0000","last_became_active":"2024-04-25T00:01:19.512425+0000","last_became_peered":"2024-04-25T00:01:19.512425+0000","last_unstale":"2024-04-25T00:01:20.580191+0000","last_undegraded":"2024-04-25T00:01:20.580191+0000","last_fullsized":"2024-04-25T00:01:20.580191+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:43:16.957273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.520943+0000","last_change":"2024-04-25T00:01:05.361723+0000","last_active":"2024-04-25T00:01:20.520943+0000","last_peered":"2024-04-25T00:01:20.520943+0000","last_clean":"2024-04-25T00:01:20.520943+0000","last_became_active":"2024-04-25T00:01:05.360956+0000","last_became_peered":"2024-04-25T00:01:05.360956+0000","last_unstale":"2024-04-25T00:01:20.520943+0000","last_undegraded":"2024-04-25T00:01:20.520943+0000","last_fullsized":"2024-04-25T00:01:20.520943+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:23.870121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.580983+0000","last_change":"2024-04-25T00:01:04.362129+0000","last_active":"2024-04-25T00:01:20.580983+0000","last_peered":"2024-04-25T00:01:20.580983+0000","last_clean":"2024-04-25T00:01:20.580983+0000","last_became_active":"2024-04-25T00:01:04.361827+0000","last_became_peered":"2024-04-25T00:01:04.361827+0000","last_unstale":"2024-04-25T00:01:20.580983+0000","last_undegraded":"2024-04-25T00:01:20.580983+0000","last_fullsized":"2024-04-25T00:01:20.580983+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:26.243635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"26'8","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.510624+0000","last_change":"2024-04-25T00:01:18.496129+0000","last_active":"2024-04-25T00:01:20.510624+0000","last_peered":"2024-04-25T00:01:20.510624+0000","last_clean":"2024-04-25T00:01:20.510624+0000","last_became_active":"2024-04-25T00:01:18.495415+0000","last_became_peered":"2024-04-25T00:01:18.495415+0000","last_unstale":"2024-04-25T00:01:20.510624+0000","last_undegraded":"2024-04-25T00:01:20.510624+0000","last_fullsized":"2024-04-25T00:01:20.510624+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:38.089411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.510763+0000","last_change":"2024-04-25T00:01:19.509542+0000","last_active":"2024-04-25T00:01:20.510763+0000","last_peered":"2024-04-25T00:01:20.510763+0000","last_clean":"2024-04-25T00:01:20.510763+0000","last_became_active":"2024-04-25T00:01:19.509015+0000","last_became_peered":"2024-04-25T00:01:19.509015+0000","last_unstale":"2024-04-25T00:01:20.510763+0000","last_undegraded":"2024-04-25T00:01:20.510763+0000","last_fullsized":"2024-04-25T00:01:20.510763+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:25:20.185355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511808+0000","last_change":"2024-04-25T00:01:05.365413+0000","last_active":"2024-04-25T00:01:20.511808+0000","last_peered":"2024-04-25T00:01:20.511808+0000","last_clean":"2024-04-25T00:01:20.511808+0000","last_became_active":"2024-04-25T00:01:05.361865+0000","last_became_peered":"2024-04-25T00:01:05.361865+0000","last_unstale":"2024-04-25T00:01:20.511808+0000","last_undegraded":"2024-04-25T00:01:20.511808+0000","last_fullsized":"2024-04-25T00:01:20.511808+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:30.228060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582499+0000","last_change":"2024-04-25T00:01:04.360804+0000","last_active":"2024-04-25T00:01:20.582499+0000","last_peered":"2024-04-25T00:01:20.582499+0000","last_clean":"2024-04-25T00:01:20.582499+0000","last_became_active":"2024-04-25T00:01:04.360374+0000","last_became_peered":"2024-04-25T00:01:04.360374+0000","last_unstale":"2024-04-25T00:01:20.582499+0000","last_undegraded":"2024-04-25T00:01:20.582499+0000","last_fullsized":"2024-04-25T00:01:20.582499+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511980+0000","last_change":"2024-04-25T00:01:18.480003+0000","last_active":"2024-04-25T00:01:20.511980+0000","last_peered":"2024-04-25T00:01:20.511980+0000","last_clean":"2024-04-25T00:01:20.511980+0000","last_became_active":"2024-04-25T00:01:18.479270+0000","last_became_peered":"2024-04-25T00:01:18.479270+0000","last_unstale":"2024-04-25T00:01:20.511980+0000","last_undegraded":"2024-04-25T00:01:20.511980+0000","last_fullsized":"2024-04-25T00:01:20.511980+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:52:40.942502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511885+0000","last_change":"2024-04-25T00:01:19.524463+0000","last_active":"2024-04-25T00:01:20.511885+0000","last_peered":"2024-04-25T00:01:20.511885+0000","last_clean":"2024-04-25T00:01:20.511885+0000","last_became_active":"2024-04-25T00:01:19.524166+0000","last_became_peered":"2024-04-25T00:01:19.524166+0000","last_unstale":"2024-04-25T00:01:20.511885+0000","last_undegraded":"2024-04-25T00:01:20.511885+0000","last_fullsized":"2024-04-25T00:01:20.511885+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:50.351599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"26'1","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.514720+0000","last_change":"2024-04-25T00:01:18.497096+0000","last_active":"2024-04-25T00:01:20.514720+0000","last_peered":"2024-04-25T00:01:20.514720+0000","last_clean":"2024-04-25T00:01:20.514720+0000","last_became_active":"2024-04-25T00:01:18.496289+0000","last_became_peered":"2024-04-25T00:01:18.496289+0000","last_unstale":"2024-04-25T00:01:20.514720+0000","last_undegraded":"2024-04-25T00:01:20.514720+0000","last_fullsized":"2024-04-25T00:01:20.514720+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:59:40.316831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.512256+0000","last_change":"2024-04-25T00:01:05.370043+0000","last_active":"2024-04-25T00:01:20.512256+0000","last_peered":"2024-04-25T00:01:20.512256+0000","last_clean":"2024-04-25T00:01:20.512256+0000","last_became_active":"2024-04-25T00:01:05.368080+0000","last_became_peered":"2024-04-25T00:01:05.368080+0000","last_unstale":"2024-04-25T00:01:20.512256+0000","last_undegraded":"2024-04-25T00:01:20.512256+0000","last_fullsized":"2024-04-25T00:01:20.512256+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:58:07.610923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.580256+0000","last_change":"2024-04-25T00:01:04.364523+0000","last_active":"2024-04-25T00:01:20.580256+0000","last_peered":"2024-04-25T00:01:20.580256+0000","last_clean":"2024-04-25T00:01:20.580256+0000","last_became_active":"2024-04-25T00:01:04.364050+0000","last_became_peered":"2024-04-25T00:01:04.364050+0000","last_unstale":"2024-04-25T00:01:20.580256+0000","last_undegraded":"2024-04-25T00:01:20.580256+0000","last_fullsized":"2024-04-25T00:01:20.580256+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:49.479804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.579360+0000","last_change":"2024-04-25T00:01:19.491523+0000","last_active":"2024-04-25T00:01:20.579360+0000","last_peered":"2024-04-25T00:01:20.579360+0000","last_clean":"2024-04-25T00:01:20.579360+0000","last_became_active":"2024-04-25T00:01:19.490755+0000","last_became_peered":"2024-04-25T00:01:19.490755+0000","last_unstale":"2024-04-25T00:01:20.579360+0000","last_undegraded":"2024-04-25T00:01:20.579360+0000","last_fullsized":"2024-04-25T00:01:20.579360+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:47:19.526845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"26'4","reported_seq":18,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.584296+0000","last_change":"2024-04-25T00:01:18.472794+0000","last_active":"2024-04-25T00:01:20.584296+0000","last_peered":"2024-04-25T00:01:20.584296+0000","last_clean":"2024-04-25T00:01:20.584296+0000","last_became_active":"2024-04-25T00:01:18.472534+0000","last_became_peered":"2024-04-25T00:01:18.472534+0000","last_unstale":"2024-04-25T00:01:20.584296+0000","last_undegraded":"2024-04-25T00:01:20.584296+0000","last_fullsized":"2024-04-25T00:01:20.584296+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:55:45.843808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.518835+0000","last_change":"2024-04-25T00:01:05.378867+0000","last_active":"2024-04-25T00:01:20.518835+0000","last_peered":"2024-04-25T00:01:20.518835+0000","last_clean":"2024-04-25T00:01:20.518835+0000","last_became_active":"2024-04-25T00:01:05.378056+0000","last_became_peered":"2024-04-25T00:01:05.378056+0000","last_unstale":"2024-04-25T00:01:20.518835+0000","last_undegraded":"2024-04-25T00:01:20.518835+0000","last_fullsized":"2024-04-25T00:01:20.518835+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:36:10.377757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.577806+0000","last_change":"2024-04-25T00:01:04.370896+0000","last_active":"2024-04-25T00:01:20.577806+0000","last_peered":"2024-04-25T00:01:20.577806+0000","last_clean":"2024-04-25T00:01:20.577806+0000","last_became_active":"2024-04-25T00:01:04.370273+0000","last_became_peered":"2024-04-25T00:01:04.370273+0000","last_unstale":"2024-04-25T00:01:20.577806+0000","last_undegraded":"2024-04-25T00:01:20.577806+0000","last_fullsized":"2024-04-25T00:01:20.577806+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:49:22.673421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.514211+0000","last_change":"2024-04-25T00:01:19.521143+0000","last_active":"2024-04-25T00:01:20.514211+0000","last_peered":"2024-04-25T00:01:20.514211+0000","last_clean":"2024-04-25T00:01:20.514211+0000","last_became_active":"2024-04-25T00:01:19.514664+0000","last_became_peered":"2024-04-25T00:01:19.514664+0000","last_unstale":"2024-04-25T00:01:20.514211+0000","last_undegraded":"2024-04-25T00:01:20.514211+0000","last_fullsized":"2024-04-25T00:01:20.514211+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:03:47.440442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"22'1","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.580606+0000","last_change":"2024-04-25T00:01:04.360254+0000","last_active":"2024-04-25T00:01:20.580606+0000","last_peered":"2024-04-25T00:01:20.580606+0000","last_clean":"2024-04-25T00:01:20.580606+0000","last_became_active":"2024-04-25T00:01:04.359586+0000","last_became_peered":"2024-04-25T00:01:04.359586+0000","last_unstale":"2024-04-25T00:01:20.580606+0000","last_undegraded":"2024-04-25T00:01:20.580606+0000","last_fullsized":"2024-04-25T00:01:20.580606+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:47.192930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.513293+0000","last_change":"2024-04-25T00:01:05.370723+0000","last_active":"2024-04-25T00:01:20.513293+0000","last_peered":"2024-04-25T00:01:20.513293+0000","last_clean":"2024-04-25T00:01:20.513293+0000","last_became_active":"2024-04-25T00:01:05.368970+0000","last_became_peered":"2024-04-25T00:01:05.368970+0000","last_unstale":"2024-04-25T00:01:20.513293+0000","last_undegraded":"2024-04-25T00:01:20.513293+0000","last_fullsized":"2024-04-25T00:01:20.513293+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:06:53.743074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"26'2","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.513582+0000","last_change":"2024-04-25T00:01:18.498570+0000","last_active":"2024-04-25T00:01:20.513582+0000","last_peered":"2024-04-25T00:01:20.513582+0000","last_clean":"2024-04-25T00:01:20.513582+0000","last_became_active":"2024-04-25T00:01:18.497745+0000","last_became_peered":"2024-04-25T00:01:18.497745+0000","last_unstale":"2024-04-25T00:01:20.513582+0000","last_undegraded":"2024-04-25T00:01:20.513582+0000","last_fullsized":"2024-04-25T00:01:20.513582+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:26:42.047149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511888+0000","last_change":"2024-04-25T00:01:19.514919+0000","last_active":"2024-04-25T00:01:20.511888+0000","last_peered":"2024-04-25T00:01:20.511888+0000","last_clean":"2024-04-25T00:01:20.511888+0000","last_became_active":"2024-04-25T00:01:19.514280+0000","last_became_peered":"2024-04-25T00:01:19.514280+0000","last_unstale":"2024-04-25T00:01:20.511888+0000","last_undegraded":"2024-04-25T00:01:20.511888+0000","last_fullsized":"2024-04-25T00:01:20.511888+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:37.975265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"22'1","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.519944+0000","last_change":"2024-04-25T00:01:04.365607+0000","last_active":"2024-04-25T00:01:20.519944+0000","last_peered":"2024-04-25T00:01:20.519944+0000","last_clean":"2024-04-25T00:01:20.519944+0000","last_became_active":"2024-04-25T00:01:04.364788+0000","last_became_peered":"2024-04-25T00:01:04.364788+0000","last_unstale":"2024-04-25T00:01:20.519944+0000","last_undegraded":"2024-04-25T00:01:20.519944+0000","last_fullsized":"2024-04-25T00:01:20.519944+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.579248+0000","last_change":"2024-04-25T00:01:05.375142+0000","last_active":"2024-04-25T00:01:20.579248+0000","last_peered":"2024-04-25T00:01:20.579248+0000","last_clean":"2024-04-25T00:01:20.579248+0000","last_became_active":"2024-04-25T00:01:05.373857+0000","last_became_peered":"2024-04-25T00:01:05.373857+0000","last_unstale":"2024-04-25T00:01:20.579248+0000","last_undegraded":"2024-04-25T00:01:20.579248+0000","last_fullsized":"2024-04-25T00:01:20.579248+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:15.879673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"26'2","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.513534+0000","last_change":"2024-04-25T00:01:18.502786+0000","last_active":"2024-04-25T00:01:20.513534+0000","last_peered":"2024-04-25T00:01:20.513534+0000","last_clean":"2024-04-25T00:01:20.513534+0000","last_became_active":"2024-04-25T00:01:18.501785+0000","last_became_peered":"2024-04-25T00:01:18.501785+0000","last_unstale":"2024-04-25T00:01:20.513534+0000","last_undegraded":"2024-04-25T00:01:20.513534+0000","last_fullsized":"2024-04-25T00:01:20.513534+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:09:31.441044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.514560+0000","last_change":"2024-04-25T00:01:19.511239+0000","last_active":"2024-04-25T00:01:20.514560+0000","last_peered":"2024-04-25T00:01:20.514560+0000","last_clean":"2024-04-25T00:01:20.514560+0000","last_became_active":"2024-04-25T00:01:19.510155+0000","last_became_peered":"2024-04-25T00:01:19.510155+0000","last_unstale":"2024-04-25T00:01:20.514560+0000","last_undegraded":"2024-04-25T00:01:20.514560+0000","last_fullsized":"2024-04-25T00:01:20.514560+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:43:14.452663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511273+0000","last_change":"2024-04-25T00:01:05.378517+0000","last_active":"2024-04-25T00:01:20.511273+0000","last_peered":"2024-04-25T00:01:20.511273+0000","last_clean":"2024-04-25T00:01:20.511273+0000","last_became_active":"2024-04-25T00:01:05.377742+0000","last_became_peered":"2024-04-25T00:01:05.377742+0000","last_unstale":"2024-04-25T00:01:20.511273+0000","last_undegraded":"2024-04-25T00:01:20.511273+0000","last_fullsized":"2024-04-25T00:01:20.511273+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:06.450369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.514828+0000","last_change":"2024-04-25T00:01:19.508999+0000","last_active":"2024-04-25T00:01:20.514828+0000","last_peered":"2024-04-25T00:01:20.514828+0000","last_clean":"2024-04-25T00:01:20.514828+0000","last_became_active":"2024-04-25T00:01:19.507960+0000","last_became_peered":"2024-04-25T00:01:19.507960+0000","last_unstale":"2024-04-25T00:01:20.514828+0000","last_undegraded":"2024-04-25T00:01:20.514828+0000","last_fullsized":"2024-04-25T00:01:20.514828+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:32:40.274908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.510941+0000","last_change":"2024-04-25T00:01:05.372927+0000","last_active":"2024-04-25T00:01:20.510941+0000","last_peered":"2024-04-25T00:01:20.510941+0000","last_clean":"2024-04-25T00:01:20.510941+0000","last_became_active":"2024-04-25T00:01:05.372050+0000","last_became_peered":"2024-04-25T00:01:05.372050+0000","last_unstale":"2024-04-25T00:01:20.510941+0000","last_undegraded":"2024-04-25T00:01:20.510941+0000","last_fullsized":"2024-04-25T00:01:20.510941+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:48.202251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511854+0000","last_change":"2024-04-25T00:01:19.531072+0000","last_active":"2024-04-25T00:01:20.511854+0000","last_peered":"2024-04-25T00:01:20.511854+0000","last_clean":"2024-04-25T00:01:20.511854+0000","last_became_active":"2024-04-25T00:01:19.530836+0000","last_became_peered":"2024-04-25T00:01:19.530836+0000","last_unstale":"2024-04-25T00:01:20.511854+0000","last_undegraded":"2024-04-25T00:01:20.511854+0000","last_fullsized":"2024-04-25T00:01:20.511854+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:08.508741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.580209+0000","last_change":"2024-04-25T00:01:05.362960+0000","last_active":"2024-04-25T00:01:20.580209+0000","last_peered":"2024-04-25T00:01:20.580209+0000","last_clean":"2024-04-25T00:01:20.580209+0000","last_became_active":"2024-04-25T00:01:05.361067+0000","last_became_peered":"2024-04-25T00:01:05.361067+0000","last_unstale":"2024-04-25T00:01:20.580209+0000","last_undegraded":"2024-04-25T00:01:20.580209+0000","last_fullsized":"2024-04-25T00:01:20.580209+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:24.017263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.520235+0000","last_change":"2024-04-25T00:01:19.506029+0000","last_active":"2024-04-25T00:01:20.520235+0000","last_peered":"2024-04-25T00:01:20.520235+0000","last_clean":"2024-04-25T00:01:20.520235+0000","last_became_active":"2024-04-25T00:01:19.505184+0000","last_became_peered":"2024-04-25T00:01:19.505184+0000","last_unstale":"2024-04-25T00:01:20.520235+0000","last_undegraded":"2024-04-25T00:01:20.520235+0000","last_fullsized":"2024-04-25T00:01:20.520235+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:08.508741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.519736+0000","last_change":"2024-04-25T00:01:05.358690+0000","last_active":"2024-04-25T00:01:20.519736+0000","last_peered":"2024-04-25T00:01:20.519736+0000","last_clean":"2024-04-25T00:01:20.519736+0000","last_became_active":"2024-04-25T00:01:05.357340+0000","last_became_peered":"2024-04-25T00:01:05.357340+0000","last_unstale":"2024-04-25T00:01:20.519736+0000","last_undegraded":"2024-04-25T00:01:20.519736+0000","last_fullsized":"2024-04-25T00:01:20.519736+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:36:55.436661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.577993+0000","last_change":"2024-04-25T00:01:19.508691+0000","last_active":"2024-04-25T00:01:20.577993+0000","last_peered":"2024-04-25T00:01:20.577993+0000","last_clean":"2024-04-25T00:01:20.577993+0000","last_became_active":"2024-04-25T00:01:19.508123+0000","last_became_peered":"2024-04-25T00:01:19.508123+0000","last_unstale":"2024-04-25T00:01:20.577993+0000","last_undegraded":"2024-04-25T00:01:20.577993+0000","last_fullsized":"2024-04-25T00:01:20.577993+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:47:19.526845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578984+0000","last_change":"2024-04-25T00:01:05.387057+0000","last_active":"2024-04-25T00:01:20.578984+0000","last_peered":"2024-04-25T00:01:20.578984+0000","last_clean":"2024-04-25T00:01:20.578984+0000","last_became_active":"2024-04-25T00:01:05.386724+0000","last_became_peered":"2024-04-25T00:01:05.386724+0000","last_unstale":"2024-04-25T00:01:20.578984+0000","last_undegraded":"2024-04-25T00:01:20.578984+0000","last_fullsized":"2024-04-25T00:01:20.578984+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:55:09.073286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.580476+0000","last_change":"2024-04-25T00:01:19.522760+0000","last_active":"2024-04-25T00:01:20.580476+0000","last_peered":"2024-04-25T00:01:20.580476+0000","last_clean":"2024-04-25T00:01:20.580476+0000","last_became_active":"2024-04-25T00:01:19.522380+0000","last_became_peered":"2024-04-25T00:01:19.522380+0000","last_unstale":"2024-04-25T00:01:20.580476+0000","last_undegraded":"2024-04-25T00:01:20.580476+0000","last_fullsized":"2024-04-25T00:01:20.580476+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:55:46.853565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578828+0000","last_change":"2024-04-25T00:01:05.367296+0000","last_active":"2024-04-25T00:01:20.578828+0000","last_peered":"2024-04-25T00:01:20.578828+0000","last_clean":"2024-04-25T00:01:20.578828+0000","last_became_active":"2024-04-25T00:01:05.363838+0000","last_became_peered":"2024-04-25T00:01:05.363838+0000","last_unstale":"2024-04-25T00:01:20.578828+0000","last_undegraded":"2024-04-25T00:01:20.578828+0000","last_fullsized":"2024-04-25T00:01:20.578828+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:43:07.053074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.579599+0000","last_change":"2024-04-25T00:01:19.510902+0000","last_active":"2024-04-25T00:01:20.579599+0000","last_peered":"2024-04-25T00:01:20.579599+0000","last_clean":"2024-04-25T00:01:20.579599+0000","last_became_active":"2024-04-25T00:01:19.510008+0000","last_became_peered":"2024-04-25T00:01:19.510008+0000","last_unstale":"2024-04-25T00:01:20.579599+0000","last_undegraded":"2024-04-25T00:01:20.579599+0000","last_fullsized":"2024-04-25T00:01:20.579599+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:40:32.292140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511704+0000","last_change":"2024-04-25T00:01:05.365761+0000","last_active":"2024-04-25T00:01:20.511704+0000","last_peered":"2024-04-25T00:01:20.511704+0000","last_clean":"2024-04-25T00:01:20.511704+0000","last_became_active":"2024-04-25T00:01:05.361119+0000","last_became_peered":"2024-04-25T00:01:05.361119+0000","last_unstale":"2024-04-25T00:01:20.511704+0000","last_undegraded":"2024-04-25T00:01:20.511704+0000","last_fullsized":"2024-04-25T00:01:20.511704+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.510983+0000","last_change":"2024-04-25T00:01:19.518534+0000","last_active":"2024-04-25T00:01:20.510983+0000","last_peered":"2024-04-25T00:01:20.510983+0000","last_clean":"2024-04-25T00:01:20.510983+0000","last_became_active":"2024-04-25T00:01:19.518110+0000","last_became_peered":"2024-04-25T00:01:19.518110+0000","last_unstale":"2024-04-25T00:01:20.510983+0000","last_undegraded":"2024-04-25T00:01:20.510983+0000","last_fullsized":"2024-04-25T00:01:20.510983+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:05:09.219823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.520770+0000","last_change":"2024-04-25T00:01:05.373464+0000","last_active":"2024-04-25T00:01:20.520770+0000","last_peered":"2024-04-25T00:01:20.520770+0000","last_clean":"2024-04-25T00:01:20.520770+0000","last_became_active":"2024-04-25T00:01:05.372611+0000","last_became_peered":"2024-04-25T00:01:05.372611+0000","last_unstale":"2024-04-25T00:01:20.520770+0000","last_undegraded":"2024-04-25T00:01:20.520770+0000","last_fullsized":"2024-04-25T00:01:20.520770+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:31.456792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.512974+0000","last_change":"2024-04-25T00:01:19.542475+0000","last_active":"2024-04-25T00:01:20.512974+0000","last_peered":"2024-04-25T00:01:20.512974+0000","last_clean":"2024-04-25T00:01:20.512974+0000","last_became_active":"2024-04-25T00:01:19.541877+0000","last_became_peered":"2024-04-25T00:01:19.541877+0000","last_unstale":"2024-04-25T00:01:20.512974+0000","last_undegraded":"2024-04-25T00:01:20.512974+0000","last_fullsized":"2024-04-25T00:01:20.512974+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:02:57.373014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.580023+0000","last_change":"2024-04-25T00:01:05.372906+0000","last_active":"2024-04-25T00:01:20.580023+0000","last_peered":"2024-04-25T00:01:20.580023+0000","last_clean":"2024-04-25T00:01:20.580023+0000","last_became_active":"2024-04-25T00:01:05.372148+0000","last_became_peered":"2024-04-25T00:01:05.372148+0000","last_unstale":"2024-04-25T00:01:20.580023+0000","last_undegraded":"2024-04-25T00:01:20.580023+0000","last_fullsized":"2024-04-25T00:01:20.580023+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:43.615895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.584868+0000","last_change":"2024-04-25T00:01:19.498515+0000","last_active":"2024-04-25T00:01:20.584868+0000","last_peered":"2024-04-25T00:01:20.584868+0000","last_clean":"2024-04-25T00:01:20.584868+0000","last_became_active":"2024-04-25T00:01:19.496310+0000","last_became_peered":"2024-04-25T00:01:19.496310+0000","last_unstale":"2024-04-25T00:01:20.584868+0000","last_undegraded":"2024-04-25T00:01:20.584868+0000","last_fullsized":"2024-04-25T00:01:20.584868+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:20:06.661558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.519325+0000","last_change":"2024-04-25T00:01:05.366340+0000","last_active":"2024-04-25T00:01:20.519325+0000","last_peered":"2024-04-25T00:01:20.519325+0000","last_clean":"2024-04-25T00:01:20.519325+0000","last_became_active":"2024-04-25T00:01:05.361952+0000","last_became_peered":"2024-04-25T00:01:05.361952+0000","last_unstale":"2024-04-25T00:01:20.519325+0000","last_undegraded":"2024-04-25T00:01:20.519325+0000","last_fullsized":"2024-04-25T00:01:20.519325+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:41.948662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578558+0000","last_change":"2024-04-25T00:01:19.508257+0000","last_active":"2024-04-25T00:01:20.578558+0000","last_peered":"2024-04-25T00:01:20.578558+0000","last_clean":"2024-04-25T00:01:20.578558+0000","last_became_active":"2024-04-25T00:01:19.507516+0000","last_became_peered":"2024-04-25T00:01:19.507516+0000","last_unstale":"2024-04-25T00:01:20.578558+0000","last_undegraded":"2024-04-25T00:01:20.578558+0000","last_fullsized":"2024-04-25T00:01:20.578558+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:45.121513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.579066+0000","last_change":"2024-04-25T00:01:05.387457+0000","last_active":"2024-04-25T00:01:20.579066+0000","last_peered":"2024-04-25T00:01:20.579066+0000","last_clean":"2024-04-25T00:01:20.579066+0000","last_became_active":"2024-04-25T00:01:05.386919+0000","last_became_peered":"2024-04-25T00:01:05.386919+0000","last_unstale":"2024-04-25T00:01:20.579066+0000","last_undegraded":"2024-04-25T00:01:20.579066+0000","last_fullsized":"2024-04-25T00:01:20.579066+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.512398+0000","last_change":"2024-04-25T00:01:19.520648+0000","last_active":"2024-04-25T00:01:20.512398+0000","last_peered":"2024-04-25T00:01:20.512398+0000","last_clean":"2024-04-25T00:01:20.512398+0000","last_became_active":"2024-04-25T00:01:19.519303+0000","last_became_peered":"2024-04-25T00:01:19.519303+0000","last_unstale":"2024-04-25T00:01:20.512398+0000","last_undegraded":"2024-04-25T00:01:20.512398+0000","last_fullsized":"2024-04-25T00:01:20.512398+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:23.993729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.513073+0000","last_change":"2024-04-25T00:01:05.366249+0000","last_active":"2024-04-25T00:01:20.513073+0000","last_peered":"2024-04-25T00:01:20.513073+0000","last_clean":"2024-04-25T00:01:20.513073+0000","last_became_active":"2024-04-25T00:01:05.365548+0000","last_became_peered":"2024-04-25T00:01:05.365548+0000","last_unstale":"2024-04-25T00:01:20.513073+0000","last_undegraded":"2024-04-25T00:01:20.513073+0000","last_fullsized":"2024-04-25T00:01:20.513073+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:22:21.143639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.510804+0000","last_change":"2024-04-25T00:01:19.514659+0000","last_active":"2024-04-25T00:01:20.510804+0000","last_peered":"2024-04-25T00:01:20.510804+0000","last_clean":"2024-04-25T00:01:20.510804+0000","last_became_active":"2024-04-25T00:01:19.514232+0000","last_became_peered":"2024-04-25T00:01:19.514232+0000","last_unstale":"2024-04-25T00:01:20.510804+0000","last_undegraded":"2024-04-25T00:01:20.510804+0000","last_fullsized":"2024-04-25T00:01:20.510804+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:45.121513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582289+0000","last_change":"2024-04-25T00:01:05.385253+0000","last_active":"2024-04-25T00:01:20.582289+0000","last_peered":"2024-04-25T00:01:20.582289+0000","last_clean":"2024-04-25T00:01:20.582289+0000","last_became_active":"2024-04-25T00:01:05.379429+0000","last_became_peered":"2024-04-25T00:01:05.379429+0000","last_unstale":"2024-04-25T00:01:20.582289+0000","last_undegraded":"2024-04-25T00:01:20.582289+0000","last_fullsized":"2024-04-25T00:01:20.582289+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:36:10.377757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.581087+0000","last_change":"2024-04-25T00:01:19.503419+0000","last_active":"2024-04-25T00:01:20.581087+0000","last_peered":"2024-04-25T00:01:20.581087+0000","last_clean":"2024-04-25T00:01:20.581087+0000","last_became_active":"2024-04-25T00:01:19.500568+0000","last_became_peered":"2024-04-25T00:01:19.500568+0000","last_unstale":"2024-04-25T00:01:20.581087+0000","last_undegraded":"2024-04-25T00:01:20.581087+0000","last_fullsized":"2024-04-25T00:01:20.581087+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:08.508741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.583587+0000","last_change":"2024-04-25T00:01:05.370653+0000","last_active":"2024-04-25T00:01:20.583587+0000","last_peered":"2024-04-25T00:01:20.583587+0000","last_clean":"2024-04-25T00:01:20.583587+0000","last_became_active":"2024-04-25T00:01:05.369919+0000","last_became_peered":"2024-04-25T00:01:05.369919+0000","last_unstale":"2024-04-25T00:01:20.583587+0000","last_undegraded":"2024-04-25T00:01:20.583587+0000","last_fullsized":"2024-04-25T00:01:20.583587+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:30:08.311326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.510560+0000","last_change":"2024-04-25T00:01:19.540104+0000","last_active":"2024-04-25T00:01:20.510560+0000","last_peered":"2024-04-25T00:01:20.510560+0000","last_clean":"2024-04-25T00:01:20.510560+0000","last_became_active":"2024-04-25T00:01:19.539775+0000","last_became_peered":"2024-04-25T00:01:19.539775+0000","last_unstale":"2024-04-25T00:01:20.510560+0000","last_undegraded":"2024-04-25T00:01:20.510560+0000","last_fullsized":"2024-04-25T00:01:20.510560+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:53:14.768520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.514367+0000","last_change":"2024-04-25T00:01:05.367001+0000","last_active":"2024-04-25T00:01:20.514367+0000","last_peered":"2024-04-25T00:01:20.514367+0000","last_clean":"2024-04-25T00:01:20.514367+0000","last_became_active":"2024-04-25T00:01:05.366264+0000","last_became_peered":"2024-04-25T00:01:05.366264+0000","last_unstale":"2024-04-25T00:01:20.514367+0000","last_undegraded":"2024-04-25T00:01:20.514367+0000","last_fullsized":"2024-04-25T00:01:20.514367+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578038+0000","last_change":"2024-04-25T00:01:19.497247+0000","last_active":"2024-04-25T00:01:20.578038+0000","last_peered":"2024-04-25T00:01:20.578038+0000","last_clean":"2024-04-25T00:01:20.578038+0000","last_became_active":"2024-04-25T00:01:19.496517+0000","last_became_peered":"2024-04-25T00:01:19.496517+0000","last_unstale":"2024-04-25T00:01:20.578038+0000","last_undegraded":"2024-04-25T00:01:20.578038+0000","last_fullsized":"2024-04-25T00:01:20.578038+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:30.129056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.584723+0000","last_change":"2024-04-25T00:01:05.379958+0000","last_active":"2024-04-25T00:01:20.584723+0000","last_peered":"2024-04-25T00:01:20.584723+0000","last_clean":"2024-04-25T00:01:20.584723+0000","last_became_active":"2024-04-25T00:01:05.379090+0000","last_became_peered":"2024-04-25T00:01:05.379090+0000","last_unstale":"2024-04-25T00:01:20.584723+0000","last_undegraded":"2024-04-25T00:01:20.584723+0000","last_fullsized":"2024-04-25T00:01:20.584723+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:59:50.832958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.515390+0000","last_change":"2024-04-25T00:01:19.530176+0000","last_active":"2024-04-25T00:01:20.515390+0000","last_peered":"2024-04-25T00:01:20.515390+0000","last_clean":"2024-04-25T00:01:20.515390+0000","last_became_active":"2024-04-25T00:01:19.529410+0000","last_became_peered":"2024-04-25T00:01:19.529410+0000","last_unstale":"2024-04-25T00:01:20.515390+0000","last_undegraded":"2024-04-25T00:01:20.515390+0000","last_fullsized":"2024-04-25T00:01:20.515390+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:40:13.204880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.579746+0000","last_change":"2024-04-25T00:01:05.370826+0000","last_active":"2024-04-25T00:01:20.579746+0000","last_peered":"2024-04-25T00:01:20.579746+0000","last_clean":"2024-04-25T00:01:20.579746+0000","last_became_active":"2024-04-25T00:01:05.369848+0000","last_became_peered":"2024-04-25T00:01:05.369848+0000","last_unstale":"2024-04-25T00:01:20.579746+0000","last_undegraded":"2024-04-25T00:01:20.579746+0000","last_fullsized":"2024-04-25T00:01:20.579746+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:23.870121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.581888+0000","last_change":"2024-04-25T00:01:19.511625+0000","last_active":"2024-04-25T00:01:20.581888+0000","last_peered":"2024-04-25T00:01:20.581888+0000","last_clean":"2024-04-25T00:01:20.581888+0000","last_became_active":"2024-04-25T00:01:19.511349+0000","last_became_peered":"2024-04-25T00:01:19.511349+0000","last_unstale":"2024-04-25T00:01:20.581888+0000","last_undegraded":"2024-04-25T00:01:20.581888+0000","last_fullsized":"2024-04-25T00:01:20.581888+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:02:57.373014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.518885+0000","last_change":"2024-04-25T00:01:05.390341+0000","last_active":"2024-04-25T00:01:20.518885+0000","last_peered":"2024-04-25T00:01:20.518885+0000","last_clean":"2024-04-25T00:01:20.518885+0000","last_became_active":"2024-04-25T00:01:05.389707+0000","last_became_peered":"2024-04-25T00:01:05.389707+0000","last_unstale":"2024-04-25T00:01:20.518885+0000","last_undegraded":"2024-04-25T00:01:20.518885+0000","last_fullsized":"2024-04-25T00:01:20.518885+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:54:21.138947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578289+0000","last_change":"2024-04-25T00:01:19.513630+0000","last_active":"2024-04-25T00:01:20.578289+0000","last_peered":"2024-04-25T00:01:20.578289+0000","last_clean":"2024-04-25T00:01:20.578289+0000","last_became_active":"2024-04-25T00:01:19.512978+0000","last_became_peered":"2024-04-25T00:01:19.512978+0000","last_unstale":"2024-04-25T00:01:20.578289+0000","last_undegraded":"2024-04-25T00:01:20.578289+0000","last_fullsized":"2024-04-25T00:01:20.578289+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:57.751007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578080+0000","last_change":"2024-04-25T00:01:05.378373+0000","last_active":"2024-04-25T00:01:20.578080+0000","last_peered":"2024-04-25T00:01:20.578080+0000","last_clean":"2024-04-25T00:01:20.578080+0000","last_became_active":"2024-04-25T00:01:05.376570+0000","last_became_peered":"2024-04-25T00:01:05.376570+0000","last_unstale":"2024-04-25T00:01:20.578080+0000","last_undegraded":"2024-04-25T00:01:20.578080+0000","last_fullsized":"2024-04-25T00:01:20.578080+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:30.228060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582484+0000","last_change":"2024-04-25T00:01:19.516716+0000","last_active":"2024-04-25T00:01:20.582484+0000","last_peered":"2024-04-25T00:01:20.582484+0000","last_clean":"2024-04-25T00:01:20.582484+0000","last_became_active":"2024-04-25T00:01:19.516142+0000","last_became_peered":"2024-04-25T00:01:19.516142+0000","last_unstale":"2024-04-25T00:01:20.582484+0000","last_undegraded":"2024-04-25T00:01:20.582484+0000","last_fullsized":"2024-04-25T00:01:20.582484+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:28:18.234546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578828+0000","last_change":"2024-04-25T00:01:05.381011+0000","last_active":"2024-04-25T00:01:20.578828+0000","last_peered":"2024-04-25T00:01:20.578828+0000","last_clean":"2024-04-25T00:01:20.578828+0000","last_became_active":"2024-04-25T00:01:05.380236+0000","last_became_peered":"2024-04-25T00:01:05.380236+0000","last_unstale":"2024-04-25T00:01:20.578828+0000","last_undegraded":"2024-04-25T00:01:20.578828+0000","last_fullsized":"2024-04-25T00:01:20.578828+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:06:19.307378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.513107+0000","last_change":"2024-04-25T00:01:19.529023+0000","last_active":"2024-04-25T00:01:20.513107+0000","last_peered":"2024-04-25T00:01:20.513107+0000","last_clean":"2024-04-25T00:01:20.513107+0000","last_became_active":"2024-04-25T00:01:19.524609+0000","last_became_peered":"2024-04-25T00:01:19.524609+0000","last_unstale":"2024-04-25T00:01:20.513107+0000","last_undegraded":"2024-04-25T00:01:20.513107+0000","last_fullsized":"2024-04-25T00:01:20.513107+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:41:19.150476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582056+0000","last_change":"2024-04-25T00:01:05.385853+0000","last_active":"2024-04-25T00:01:20.582056+0000","last_peered":"2024-04-25T00:01:20.582056+0000","last_clean":"2024-04-25T00:01:20.582056+0000","last_became_active":"2024-04-25T00:01:05.384997+0000","last_became_peered":"2024-04-25T00:01:05.384997+0000","last_unstale":"2024-04-25T00:01:20.582056+0000","last_undegraded":"2024-04-25T00:01:20.582056+0000","last_fullsized":"2024-04-25T00:01:20.582056+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:36:55.436661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578299+0000","last_change":"2024-04-25T00:01:19.516867+0000","last_active":"2024-04-25T00:01:20.578299+0000","last_peered":"2024-04-25T00:01:20.578299+0000","last_clean":"2024-04-25T00:01:20.578299+0000","last_became_active":"2024-04-25T00:01:19.515764+0000","last_became_peered":"2024-04-25T00:01:19.515764+0000","last_unstale":"2024-04-25T00:01:20.578299+0000","last_undegraded":"2024-04-25T00:01:20.578299+0000","last_fullsized":"2024-04-25T00:01:20.578299+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:53:14.768520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582072+0000","last_change":"2024-04-25T00:01:05.389424+0000","last_active":"2024-04-25T00:01:20.582072+0000","last_peered":"2024-04-25T00:01:20.582072+0000","last_clean":"2024-04-25T00:01:20.582072+0000","last_became_active":"2024-04-25T00:01:05.388999+0000","last_became_peered":"2024-04-25T00:01:05.388999+0000","last_unstale":"2024-04-25T00:01:20.582072+0000","last_undegraded":"2024-04-25T00:01:20.582072+0000","last_fullsized":"2024-04-25T00:01:20.582072+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:31.456792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578192+0000","last_change":"2024-04-25T00:01:19.519365+0000","last_active":"2024-04-25T00:01:20.578192+0000","last_peered":"2024-04-25T00:01:20.578192+0000","last_clean":"2024-04-25T00:01:20.578192+0000","last_became_active":"2024-04-25T00:01:19.519085+0000","last_became_peered":"2024-04-25T00:01:19.519085+0000","last_unstale":"2024-04-25T00:01:20.578192+0000","last_undegraded":"2024-04-25T00:01:20.578192+0000","last_fullsized":"2024-04-25T00:01:20.578192+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:05:09.219823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.580630+0000","last_change":"2024-04-25T00:01:05.368974+0000","last_active":"2024-04-25T00:01:20.580630+0000","last_peered":"2024-04-25T00:01:20.580630+0000","last_clean":"2024-04-25T00:01:20.580630+0000","last_became_active":"2024-04-25T00:01:05.367893+0000","last_became_peered":"2024-04-25T00:01:05.367893+0000","last_unstale":"2024-04-25T00:01:20.580630+0000","last_undegraded":"2024-04-25T00:01:20.580630+0000","last_fullsized":"2024-04-25T00:01:20.580630+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:30:23.557381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578274+0000","last_change":"2024-04-25T00:01:19.513514+0000","last_active":"2024-04-25T00:01:20.578274+0000","last_peered":"2024-04-25T00:01:20.578274+0000","last_clean":"2024-04-25T00:01:20.578274+0000","last_became_active":"2024-04-25T00:01:19.513184+0000","last_became_peered":"2024-04-25T00:01:19.513184+0000","last_unstale":"2024-04-25T00:01:20.578274+0000","last_undegraded":"2024-04-25T00:01:20.578274+0000","last_fullsized":"2024-04-25T00:01:20.578274+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:47.859626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582384+0000","last_change":"2024-04-25T00:01:05.362303+0000","last_active":"2024-04-25T00:01:20.582384+0000","last_peered":"2024-04-25T00:01:20.582384+0000","last_clean":"2024-04-25T00:01:20.582384+0000","last_became_active":"2024-04-25T00:01:05.361586+0000","last_became_peered":"2024-04-25T00:01:05.361586+0000","last_unstale":"2024-04-25T00:01:20.582384+0000","last_undegraded":"2024-04-25T00:01:20.582384+0000","last_fullsized":"2024-04-25T00:01:20.582384+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:30.228060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.520395+0000","last_change":"2024-04-25T00:01:19.515101+0000","last_active":"2024-04-25T00:01:20.520395+0000","last_peered":"2024-04-25T00:01:20.520395+0000","last_clean":"2024-04-25T00:01:20.520395+0000","last_became_active":"2024-04-25T00:01:19.513314+0000","last_became_peered":"2024-04-25T00:01:19.513314+0000","last_unstale":"2024-04-25T00:01:20.520395+0000","last_undegraded":"2024-04-25T00:01:20.520395+0000","last_fullsized":"2024-04-25T00:01:20.520395+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:50.351599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.518642+0000","last_change":"2024-04-25T00:01:05.390256+0000","last_active":"2024-04-25T00:01:20.518642+0000","last_peered":"2024-04-25T00:01:20.518642+0000","last_clean":"2024-04-25T00:01:20.518642+0000","last_became_active":"2024-04-25T00:01:05.389724+0000","last_became_peered":"2024-04-25T00:01:05.389724+0000","last_unstale":"2024-04-25T00:01:20.518642+0000","last_undegraded":"2024-04-25T00:01:20.518642+0000","last_fullsized":"2024-04-25T00:01:20.518642+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:09:18.315688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3f","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511570+0000","last_change":"2024-04-25T00:01:19.511493+0000","last_active":"2024-04-25T00:01:20.511570+0000","last_peered":"2024-04-25T00:01:20.511570+0000","last_clean":"2024-04-25T00:01:20.511570+0000","last_became_active":"2024-04-25T00:01:19.508049+0000","last_became_peered":"2024-04-25T00:01:19.508049+0000","last_unstale":"2024-04-25T00:01:20.511570+0000","last_undegraded":"2024-04-25T00:01:20.511570+0000","last_fullsized":"2024-04-25T00:01:20.511570+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:55:46.853565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578830+0000","last_change":"2024-04-25T00:01:05.374766+0000","last_active":"2024-04-25T00:01:20.578830+0000","last_peered":"2024-04-25T00:01:20.578830+0000","last_clean":"2024-04-25T00:01:20.578830+0000","last_became_active":"2024-04-25T00:01:05.369739+0000","last_became_peered":"2024-04-25T00:01:05.369739+0000","last_unstale":"2024-04-25T00:01:20.578830+0000","last_undegraded":"2024-04-25T00:01:20.578830+0000","last_fullsized":"2024-04-25T00:01:20.578830+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:09:18.315688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.513219+0000","last_change":"2024-04-25T00:01:19.521456+0000","last_active":"2024-04-25T00:01:20.513219+0000","last_peered":"2024-04-25T00:01:20.513219+0000","last_clean":"2024-04-25T00:01:20.513219+0000","last_became_active":"2024-04-25T00:01:19.520787+0000","last_became_peered":"2024-04-25T00:01:19.520787+0000","last_unstale":"2024-04-25T00:01:20.513219+0000","last_undegraded":"2024-04-25T00:01:20.513219+0000","last_fullsized":"2024-04-25T00:01:20.513219+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:25:38.981364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511048+0000","last_change":"2024-04-25T00:01:05.359253+0000","last_active":"2024-04-25T00:01:20.511048+0000","last_peered":"2024-04-25T00:01:20.511048+0000","last_clean":"2024-04-25T00:01:20.511048+0000","last_became_active":"2024-04-25T00:01:05.358477+0000","last_became_peered":"2024-04-25T00:01:05.358477+0000","last_unstale":"2024-04-25T00:01:20.511048+0000","last_undegraded":"2024-04-25T00:01:20.511048+0000","last_fullsized":"2024-04-25T00:01:20.511048+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:22.739214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.579384+0000","last_change":"2024-04-25T00:01:19.509505+0000","last_active":"2024-04-25T00:01:20.579384+0000","last_peered":"2024-04-25T00:01:20.579384+0000","last_clean":"2024-04-25T00:01:20.579384+0000","last_became_active":"2024-04-25T00:01:19.507812+0000","last_became_peered":"2024-04-25T00:01:19.507812+0000","last_unstale":"2024-04-25T00:01:20.579384+0000","last_undegraded":"2024-04-25T00:01:20.579384+0000","last_fullsized":"2024-04-25T00:01:20.579384+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:23.993729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.581200+0000","last_change":"2024-04-25T00:01:05.386068+0000","last_active":"2024-04-25T00:01:20.581200+0000","last_peered":"2024-04-25T00:01:20.581200+0000","last_clean":"2024-04-25T00:01:20.581200+0000","last_became_active":"2024-04-25T00:01:05.378650+0000","last_became_peered":"2024-04-25T00:01:05.378650+0000","last_unstale":"2024-04-25T00:01:20.581200+0000","last_undegraded":"2024-04-25T00:01:20.581200+0000","last_fullsized":"2024-04-25T00:01:20.581200+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:12:52.846250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511288+0000","last_change":"2024-04-25T00:01:19.514843+0000","last_active":"2024-04-25T00:01:20.511288+0000","last_peered":"2024-04-25T00:01:20.511288+0000","last_clean":"2024-04-25T00:01:20.511288+0000","last_became_active":"2024-04-25T00:01:19.513956+0000","last_became_peered":"2024-04-25T00:01:19.513956+0000","last_unstale":"2024-04-25T00:01:20.511288+0000","last_undegraded":"2024-04-25T00:01:20.511288+0000","last_fullsized":"2024-04-25T00:01:20.511288+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:38.005233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511996+0000","last_change":"2024-04-25T00:01:05.359716+0000","last_active":"2024-04-25T00:01:20.511996+0000","last_peered":"2024-04-25T00:01:20.511996+0000","last_clean":"2024-04-25T00:01:20.511996+0000","last_became_active":"2024-04-25T00:01:05.358866+0000","last_became_peered":"2024-04-25T00:01:05.358866+0000","last_unstale":"2024-04-25T00:01:20.511996+0000","last_undegraded":"2024-04-25T00:01:20.511996+0000","last_fullsized":"2024-04-25T00:01:20.511996+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:50.489125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.519466+0000","last_change":"2024-04-25T00:01:19.499574+0000","last_active":"2024-04-25T00:01:20.519466+0000","last_peered":"2024-04-25T00:01:20.519466+0000","last_clean":"2024-04-25T00:01:20.519466+0000","last_became_active":"2024-04-25T00:01:19.498699+0000","last_became_peered":"2024-04-25T00:01:19.498699+0000","last_unstale":"2024-04-25T00:01:20.519466+0000","last_undegraded":"2024-04-25T00:01:20.519466+0000","last_fullsized":"2024-04-25T00:01:20.519466+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:47.859626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.520083+0000","last_change":"2024-04-25T00:01:05.371291+0000","last_active":"2024-04-25T00:01:20.520083+0000","last_peered":"2024-04-25T00:01:20.520083+0000","last_clean":"2024-04-25T00:01:20.520083+0000","last_became_active":"2024-04-25T00:01:05.370522+0000","last_became_peered":"2024-04-25T00:01:05.370522+0000","last_unstale":"2024-04-25T00:01:20.520083+0000","last_undegraded":"2024-04-25T00:01:20.520083+0000","last_fullsized":"2024-04-25T00:01:20.520083+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:11:27.044893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511353+0000","last_change":"2024-04-25T00:01:19.523798+0000","last_active":"2024-04-25T00:01:20.511353+0000","last_peered":"2024-04-25T00:01:20.511353+0000","last_clean":"2024-04-25T00:01:20.511353+0000","last_became_active":"2024-04-25T00:01:19.523450+0000","last_became_peered":"2024-04-25T00:01:19.523450+0000","last_unstale":"2024-04-25T00:01:20.511353+0000","last_undegraded":"2024-04-25T00:01:20.511353+0000","last_fullsized":"2024-04-25T00:01:20.511353+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:40:36.350832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.520565+0000","last_change":"2024-04-25T00:01:05.368190+0000","last_active":"2024-04-25T00:01:20.520565+0000","last_peered":"2024-04-25T00:01:20.520565+0000","last_clean":"2024-04-25T00:01:20.520565+0000","last_became_active":"2024-04-25T00:01:05.362606+0000","last_became_peered":"2024-04-25T00:01:05.362606+0000","last_unstale":"2024-04-25T00:01:20.520565+0000","last_undegraded":"2024-04-25T00:01:20.520565+0000","last_fullsized":"2024-04-25T00:01:20.520565+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:12:52.846250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.510932+0000","last_change":"2024-04-25T00:01:19.505737+0000","last_active":"2024-04-25T00:01:20.510932+0000","last_peered":"2024-04-25T00:01:20.510932+0000","last_clean":"2024-04-25T00:01:20.510932+0000","last_became_active":"2024-04-25T00:01:19.504991+0000","last_became_peered":"2024-04-25T00:01:19.504991+0000","last_unstale":"2024-04-25T00:01:20.510932+0000","last_undegraded":"2024-04-25T00:01:20.510932+0000","last_fullsized":"2024-04-25T00:01:20.510932+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:47:19.526845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.583309+0000","last_change":"2024-04-25T00:01:05.389596+0000","last_active":"2024-04-25T00:01:20.583309+0000","last_peered":"2024-04-25T00:01:20.583309+0000","last_clean":"2024-04-25T00:01:20.583309+0000","last_became_active":"2024-04-25T00:01:05.389127+0000","last_became_peered":"2024-04-25T00:01:05.389127+0000","last_unstale":"2024-04-25T00:01:20.583309+0000","last_undegraded":"2024-04-25T00:01:20.583309+0000","last_fullsized":"2024-04-25T00:01:20.583309+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:09.858617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509450,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":109716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509450,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":111182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509449,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27896,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343944,"statfs":{"total":96636764160,"available":96608198656,"internally_reserved":0,"allocated":888832,"data_stored":700612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509450,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27324,"kb_used_data":296,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344516,"statfs":{"total":96636764160,"available":96608784384,"internally_reserved":0,"allocated":303104,"data_stored":110928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509451,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27320,"kb_used_data":292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344520,"statfs":{"total":96636764160,"available":96608788480,"internally_reserved":0,"allocated":299008,"data_stored":111216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509450,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27892,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343948,"statfs":{"total":96636764160,"available":96608202752,"internally_reserved":0,"allocated":884736,"data_stored":700078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509450,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":109716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574859,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27336,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344504,"statfs":{"total":96636764160,"available":96608772096,"internally_reserved":0,"allocated":319488,"data_stored":113488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T00:01:28.830 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T00:01:28.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.970+0000 7f2169f8f640 1 Processor -- start 2024-04-25T00:01:28.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.970+0000 7f2169f8f640 1 -- start start 2024-04-25T00:01:28.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.974+0000 7f2169f8f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f216406d5f0 0x7f2164156c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.974+0000 7f2169f8f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2164155810 0x7f21641571c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.974+0000 7f2169f8f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21640c9150 con 0x7f2164152d80 2024-04-25T00:01:28.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.974+0000 7f2169f8f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f21640c9d60 con 0x7f216406d5f0 2024-04-25T00:01:28.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.974+0000 7f2169f8f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f21640c9430 con 0x7f2164155810 2024-04-25T00:01:28.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.974+0000 7f2162ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f216406d5f0 0x7f2164156c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:28.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.974+0000 7f21637fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f2164152d80 0x7f216406bd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:41744/0 (socket says 172.21.15.116:41744) 2024-04-25T00:01:28.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.974+0000 7f2162ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f216406d5f0 0x7f2164156c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56414/0 (socket says 172.21.15.116:56414) 2024-04-25T00:01:28.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.974+0000 7f21637fe640 1 -- 172.21.15.116:0/3025542059 learned_addr learned my addr 172.21.15.116:0/3025542059 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:28.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.974+0000 7f2163fff640 1 --2- 172.21.15.116:0/3025542059 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2164155810 0x7f21641571c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:28.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.974+0000 7f2162ffd640 1 -- 172.21.15.116:0/3025542059 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2164155810 msgr2=0x7f21641571c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:28.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.974+0000 7f2162ffd640 1 --2- 172.21.15.116:0/3025542059 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2164155810 0x7f21641571c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.974+0000 7f2162ffd640 1 -- 172.21.15.116:0/3025542059 >> v1:172.21.15.116:6789/0 conn(0x7f2164152d80 legacy=0x7f216406bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:28.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.974+0000 7f21627fc640 1 -- 172.21.15.116:0/3025542059 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4215169917 0 0) 0x7f21640c9150 con 0x7f2164152d80 2024-04-25T00:01:28.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.974+0000 7f2162ffd640 1 -- 172.21.15.116:0/3025542059 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21640c9710 con 0x7f216406d5f0 2024-04-25T00:01:28.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.974+0000 7f2162ffd640 1 --2- 172.21.15.116:0/3025542059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f216406d5f0 0x7f2164156c80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f21580097c0 tx=0x7f215801c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57a72b8c619cdc9f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:28.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.974+0000 7f21627fc640 1 -- 172.21.15.116:0/3025542059 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21580b1070 con 0x7f216406d5f0 2024-04-25T00:01:28.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.974+0000 7f21627fc640 1 -- 172.21.15.116:0/3025542059 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21580ac640 con 0x7f216406d5f0 2024-04-25T00:01:28.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.974+0000 7f21627fc640 1 -- 172.21.15.116:0/3025542059 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21580aca10 con 0x7f216406d5f0 2024-04-25T00:01:28.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.974+0000 7f2169f8f640 1 -- 172.21.15.116:0/3025542059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f216406d5f0 msgr2=0x7f2164156c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:28.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.974+0000 7f2169f8f640 1 --2- 172.21.15.116:0/3025542059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f216406d5f0 0x7f2164156c80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f21580097c0 tx=0x7f215801c3e0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.974+0000 7f2169f8f640 1 -- 172.21.15.116:0/3025542059 shutdown_connections 2024-04-25T00:01:28.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.974+0000 7f2169f8f640 1 --2- 172.21.15.116:0/3025542059 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2164155810 0x7f21641571c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.974+0000 7f2169f8f640 1 --2- 172.21.15.116:0/3025542059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f216406d5f0 0x7f2164156c80 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.974+0000 7f2169f8f640 1 -- 172.21.15.116:0/3025542059 >> 172.21.15.116:0/3025542059 conn(0x7f21640acdd0 msgr2=0x7f21640ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:28.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.974+0000 7f2169f8f640 1 -- 172.21.15.116:0/3025542059 shutdown_connections 2024-04-25T00:01:28.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.978+0000 7f2169f8f640 1 -- 172.21.15.116:0/3025542059 wait complete. 2024-04-25T00:01:28.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.978+0000 7f2169f8f640 1 Processor -- start 2024-04-25T00:01:28.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.978+0000 7f2169f8f640 1 -- start start 2024-04-25T00:01:28.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.978+0000 7f2169f8f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f216406d5f0 0x7f2164149d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.978+0000 7f2169f8f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2164152d80 0x7f2164154470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.978+0000 7f2169f8f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2164155810 0x7f2164148420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.978+0000 7f2169f8f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f21640caf00 con 0x7f216406d5f0 2024-04-25T00:01:28.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.978+0000 7f2169f8f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f21640cad80 con 0x7f2164152d80 2024-04-25T00:01:28.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.978+0000 7f2169f8f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f21641628d0 con 0x7f2164155810 2024-04-25T00:01:28.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.978+0000 7f21637fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f216406d5f0 0x7f2164149d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:28.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.978+0000 7f21637fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f216406d5f0 0x7f2164149d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56428/0 (socket says 172.21.15.116:56428) 2024-04-25T00:01:28.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.978+0000 7f21637fe640 1 -- 172.21.15.116:0/1560887154 learned_addr learned my addr 172.21.15.116:0/1560887154 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:28.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.978+0000 7f21637fe640 1 -- 172.21.15.116:0/1560887154 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2164155810 msgr2=0x7f2164148420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:28.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.978+0000 7f21637fe640 1 --2- 172.21.15.116:0/1560887154 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2164155810 0x7f2164148420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.978+0000 7f21637fe640 1 -- 172.21.15.116:0/1560887154 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2164152d80 msgr2=0x7f2164154470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:28.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.978+0000 7f21637fe640 1 --2- 172.21.15.116:0/1560887154 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2164152d80 0x7f2164154470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:28.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.978+0000 7f21637fe640 1 -- 172.21.15.116:0/1560887154 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21641550c0 con 0x7f216406d5f0 2024-04-25T00:01:28.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.978+0000 7f21637fe640 1 --2- 172.21.15.116:0/1560887154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f216406d5f0 0x7f2164149d60 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f215400c480 tx=0x7f215400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:28.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.978+0000 7f214bfff640 1 -- 172.21.15.116:0/1560887154 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2154018070 con 0x7f216406d5f0 2024-04-25T00:01:28.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.978+0000 7f214bfff640 1 -- 172.21.15.116:0/1560887154 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2154007500 con 0x7f216406d5f0 2024-04-25T00:01:28.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.978+0000 7f214bfff640 1 -- 172.21.15.116:0/1560887154 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2154004a30 con 0x7f216406d5f0 2024-04-25T00:01:28.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.978+0000 7f2169f8f640 1 -- 172.21.15.116:0/1560887154 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21640c1cd0 con 0x7f216406d5f0 2024-04-25T00:01:28.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.978+0000 7f2169f8f640 1 -- 172.21.15.116:0/1560887154 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2164155380 con 0x7f216406d5f0 2024-04-25T00:01:28.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.978+0000 7f2169f8f640 1 -- 172.21.15.116:0/1560887154 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2164068350 con 0x7f216406d5f0 2024-04-25T00:01:28.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.978+0000 7f214bfff640 1 -- 172.21.15.116:0/1560887154 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f215401b070 con 0x7f216406d5f0 2024-04-25T00:01:28.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.982+0000 7f214bfff640 1 --2- 172.21.15.116:0/1560887154 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f21500678f0 0x7f2150069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:28.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.982+0000 7f214bfff640 1 -- 172.21.15.116:0/1560887154 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7f21540852e0 con 0x7f216406d5f0 2024-04-25T00:01:28.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.982+0000 7f2162ffd640 1 --2- 172.21.15.116:0/1560887154 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f21500678f0 0x7f2150069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:28.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.982+0000 7f2162ffd640 1 --2- 172.21.15.116:0/1560887154 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f21500678f0 0x7f2150069db0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f21580ae000 tx=0x7f2158004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:28.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:28.986+0000 7f214bfff640 1 -- 172.21.15.116:0/1560887154 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f2154055ae0 con 0x7f216406d5f0 2024-04-25T00:01:29.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.130+0000 7f2169f8f640 1 -- 172.21.15.116:0/1560887154 --> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f21640cab60 con 0x7f21500678f0 2024-04-25T00:01:29.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.162+0000 7f214bfff640 1 -- 172.21.15.116:0/1560887154 <== mgr.4105 v2:172.21.15.174:6832/2004448211 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+490307 (secure 0 0 0) 0x7f21640cab60 con 0x7f21500678f0 2024-04-25T00:01:29.170 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:01:29.174 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-04-25T00:01:29.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.170+0000 7f2169f8f640 1 -- 172.21.15.116:0/1560887154 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f21500678f0 msgr2=0x7f2150069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:29.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.170+0000 7f2169f8f640 1 --2- 172.21.15.116:0/1560887154 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f21500678f0 0x7f2150069db0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f21580ae000 tx=0x7f2158004740 comp rx=0 tx=0).stop 2024-04-25T00:01:29.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.170+0000 7f2169f8f640 1 -- 172.21.15.116:0/1560887154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f216406d5f0 msgr2=0x7f2164149d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:29.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.170+0000 7f2169f8f640 1 --2- 172.21.15.116:0/1560887154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f216406d5f0 0x7f2164149d60 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f215400c480 tx=0x7f215400c940 comp rx=0 tx=0).stop 2024-04-25T00:01:29.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.170+0000 7f2169f8f640 1 -- 172.21.15.116:0/1560887154 shutdown_connections 2024-04-25T00:01:29.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.170+0000 7f2169f8f640 1 --2- 172.21.15.116:0/1560887154 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f21500678f0 0x7f2150069db0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:29.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.170+0000 7f2169f8f640 1 --2- 172.21.15.116:0/1560887154 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2164155810 0x7f2164148420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:29.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.170+0000 7f2169f8f640 1 --2- 172.21.15.116:0/1560887154 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2164152d80 0x7f2164154470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:29.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.170+0000 7f2169f8f640 1 --2- 172.21.15.116:0/1560887154 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f216406d5f0 0x7f2164149d60 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:29.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.170+0000 7f2169f8f640 1 -- 172.21.15.116:0/1560887154 >> 172.21.15.116:0/1560887154 conn(0x7f21640acdd0 msgr2=0x7f216414e1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:29.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.170+0000 7f2169f8f640 1 -- 172.21.15.116:0/1560887154 shutdown_connections 2024-04-25T00:01:29.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.170+0000 7f2169f8f640 1 -- 172.21.15.116:0/1560887154 wait complete. 2024-04-25T00:01:29.188 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2024-04-25T00:01:27.258101+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595188,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":249,"num_write_kb":2444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":195,"ondisk_log_size":195,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219708,"kb_used_data":3496,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755012,"statfs":{"total":773094113280,"available":772869132288,"internally_reserved":0,"allocated":3579904,"data_stored":2066936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003150"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.577366+0000","last_change":"2024-04-25T00:01:19.508728+0000","last_active":"2024-04-25T00:01:20.577366+0000","last_peered":"2024-04-25T00:01:20.577366+0000","last_clean":"2024-04-25T00:01:20.577366+0000","last_became_active":"2024-04-25T00:01:19.508390+0000","last_became_peered":"2024-04-25T00:01:19.508390+0000","last_unstale":"2024-04-25T00:01:20.577366+0000","last_undegraded":"2024-04-25T00:01:20.577366+0000","last_fullsized":"2024-04-25T00:01:20.577366+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:45.591905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1f","version":"22'4","reported_seq":26,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.515265+0000","last_change":"2024-04-25T00:01:04.356032+0000","last_active":"2024-04-25T00:01:20.515265+0000","last_peered":"2024-04-25T00:01:20.515265+0000","last_clean":"2024-04-25T00:01:20.515265+0000","last_became_active":"2024-04-25T00:01:04.355293+0000","last_became_peered":"2024-04-25T00:01:04.355293+0000","last_unstale":"2024-04-25T00:01:20.515265+0000","last_undegraded":"2024-04-25T00:01:20.515265+0000","last_fullsized":"2024-04-25T00:01:20.515265+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:56:16.678132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.520751+0000","last_change":"2024-04-25T00:01:05.367207+0000","last_active":"2024-04-25T00:01:20.520751+0000","last_peered":"2024-04-25T00:01:20.520751+0000","last_clean":"2024-04-25T00:01:20.520751+0000","last_became_active":"2024-04-25T00:01:05.366471+0000","last_became_peered":"2024-04-25T00:01:05.366471+0000","last_unstale":"2024-04-25T00:01:20.520751+0000","last_undegraded":"2024-04-25T00:01:20.520751+0000","last_fullsized":"2024-04-25T00:01:20.520751+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:30:23.557381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511325+0000","last_change":"2024-04-25T00:01:18.502364+0000","last_active":"2024-04-25T00:01:20.511325+0000","last_peered":"2024-04-25T00:01:20.511325+0000","last_clean":"2024-04-25T00:01:20.511325+0000","last_became_active":"2024-04-25T00:01:18.501633+0000","last_became_peered":"2024-04-25T00:01:18.501633+0000","last_unstale":"2024-04-25T00:01:20.511325+0000","last_undegraded":"2024-04-25T00:01:20.511325+0000","last_fullsized":"2024-04-25T00:01:20.511325+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:55.074018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.510575+0000","last_change":"2024-04-25T00:01:19.512822+0000","last_active":"2024-04-25T00:01:20.510575+0000","last_peered":"2024-04-25T00:01:20.510575+0000","last_clean":"2024-04-25T00:01:20.510575+0000","last_became_active":"2024-04-25T00:01:19.512105+0000","last_became_peered":"2024-04-25T00:01:19.512105+0000","last_unstale":"2024-04-25T00:01:20.510575+0000","last_undegraded":"2024-04-25T00:01:20.510575+0000","last_fullsized":"2024-04-25T00:01:20.510575+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:35:29.722362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"22'1","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.520903+0000","last_change":"2024-04-25T00:01:04.361809+0000","last_active":"2024-04-25T00:01:20.520903+0000","last_peered":"2024-04-25T00:01:20.520903+0000","last_clean":"2024-04-25T00:01:20.520903+0000","last_became_active":"2024-04-25T00:01:04.361059+0000","last_became_peered":"2024-04-25T00:01:04.361059+0000","last_unstale":"2024-04-25T00:01:20.520903+0000","last_undegraded":"2024-04-25T00:01:20.520903+0000","last_fullsized":"2024-04-25T00:01:20.520903+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.520801+0000","last_change":"2024-04-25T00:01:05.370818+0000","last_active":"2024-04-25T00:01:20.520801+0000","last_peered":"2024-04-25T00:01:20.520801+0000","last_clean":"2024-04-25T00:01:20.520801+0000","last_became_active":"2024-04-25T00:01:05.369991+0000","last_became_peered":"2024-04-25T00:01:05.369991+0000","last_unstale":"2024-04-25T00:01:20.520801+0000","last_undegraded":"2024-04-25T00:01:20.520801+0000","last_fullsized":"2024-04-25T00:01:20.520801+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:43.615895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.510695+0000","last_change":"2024-04-25T00:01:18.497682+0000","last_active":"2024-04-25T00:01:20.510695+0000","last_peered":"2024-04-25T00:01:20.510695+0000","last_clean":"2024-04-25T00:01:20.510695+0000","last_became_active":"2024-04-25T00:01:18.497397+0000","last_became_peered":"2024-04-25T00:01:18.497397+0000","last_unstale":"2024-04-25T00:01:20.510695+0000","last_undegraded":"2024-04-25T00:01:20.510695+0000","last_fullsized":"2024-04-25T00:01:20.510695+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:22.983972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578826+0000","last_change":"2024-04-25T00:01:19.510591+0000","last_active":"2024-04-25T00:01:20.578826+0000","last_peered":"2024-04-25T00:01:20.578826+0000","last_clean":"2024-04-25T00:01:20.578826+0000","last_became_active":"2024-04-25T00:01:19.510252+0000","last_became_peered":"2024-04-25T00:01:19.510252+0000","last_unstale":"2024-04-25T00:01:20.578826+0000","last_undegraded":"2024-04-25T00:01:20.578826+0000","last_fullsized":"2024-04-25T00:01:20.578826+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:38.005233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"22'2","reported_seq":24,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582530+0000","last_change":"2024-04-25T00:01:04.362030+0000","last_active":"2024-04-25T00:01:20.582530+0000","last_peered":"2024-04-25T00:01:20.582530+0000","last_clean":"2024-04-25T00:01:20.582530+0000","last_became_active":"2024-04-25T00:01:04.361708+0000","last_became_peered":"2024-04-25T00:01:04.361708+0000","last_unstale":"2024-04-25T00:01:20.582530+0000","last_undegraded":"2024-04-25T00:01:20.582530+0000","last_fullsized":"2024-04-25T00:01:20.582530+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:38:33.392719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578882+0000","last_change":"2024-04-25T00:01:05.387195+0000","last_active":"2024-04-25T00:01:20.578882+0000","last_peered":"2024-04-25T00:01:20.578882+0000","last_clean":"2024-04-25T00:01:20.578882+0000","last_became_active":"2024-04-25T00:01:05.386853+0000","last_became_peered":"2024-04-25T00:01:05.386853+0000","last_unstale":"2024-04-25T00:01:20.578882+0000","last_undegraded":"2024-04-25T00:01:20.578882+0000","last_fullsized":"2024-04-25T00:01:20.578882+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:06:19.307378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"26'1","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511288+0000","last_change":"2024-04-25T00:01:18.487835+0000","last_active":"2024-04-25T00:01:20.511288+0000","last_peered":"2024-04-25T00:01:20.511288+0000","last_clean":"2024-04-25T00:01:20.511288+0000","last_became_active":"2024-04-25T00:01:18.487571+0000","last_became_peered":"2024-04-25T00:01:18.487571+0000","last_unstale":"2024-04-25T00:01:20.511288+0000","last_undegraded":"2024-04-25T00:01:20.511288+0000","last_fullsized":"2024-04-25T00:01:20.511288+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:44.582148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511094+0000","last_change":"2024-04-25T00:01:19.519834+0000","last_active":"2024-04-25T00:01:20.511094+0000","last_peered":"2024-04-25T00:01:20.511094+0000","last_clean":"2024-04-25T00:01:20.511094+0000","last_became_active":"2024-04-25T00:01:19.519470+0000","last_became_peered":"2024-04-25T00:01:19.519470+0000","last_unstale":"2024-04-25T00:01:20.511094+0000","last_undegraded":"2024-04-25T00:01:20.511094+0000","last_fullsized":"2024-04-25T00:01:20.511094+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:47.859626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"22'2","reported_seq":24,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582640+0000","last_change":"2024-04-25T00:01:04.362832+0000","last_active":"2024-04-25T00:01:20.582640+0000","last_peered":"2024-04-25T00:01:20.582640+0000","last_clean":"2024-04-25T00:01:20.582640+0000","last_became_active":"2024-04-25T00:01:04.362059+0000","last_became_peered":"2024-04-25T00:01:04.362059+0000","last_unstale":"2024-04-25T00:01:20.582640+0000","last_undegraded":"2024-04-25T00:01:20.582640+0000","last_fullsized":"2024-04-25T00:01:20.582640+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:05.441048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.583017+0000","last_change":"2024-04-25T00:01:05.381120+0000","last_active":"2024-04-25T00:01:20.583017+0000","last_peered":"2024-04-25T00:01:20.583017+0000","last_clean":"2024-04-25T00:01:20.583017+0000","last_became_active":"2024-04-25T00:01:05.378320+0000","last_became_peered":"2024-04-25T00:01:05.378320+0000","last_unstale":"2024-04-25T00:01:20.583017+0000","last_undegraded":"2024-04-25T00:01:20.583017+0000","last_fullsized":"2024-04-25T00:01:20.583017+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:34:06.175918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"26'1","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.510926+0000","last_change":"2024-04-25T00:01:18.490300+0000","last_active":"2024-04-25T00:01:20.510926+0000","last_peered":"2024-04-25T00:01:20.510926+0000","last_clean":"2024-04-25T00:01:20.510926+0000","last_became_active":"2024-04-25T00:01:18.490050+0000","last_became_peered":"2024-04-25T00:01:18.490050+0000","last_unstale":"2024-04-25T00:01:20.510926+0000","last_undegraded":"2024-04-25T00:01:20.510926+0000","last_fullsized":"2024-04-25T00:01:20.510926+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:13:05.971606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.519505+0000","last_change":"2024-04-25T00:01:19.500026+0000","last_active":"2024-04-25T00:01:20.519505+0000","last_peered":"2024-04-25T00:01:20.519505+0000","last_clean":"2024-04-25T00:01:20.519505+0000","last_became_active":"2024-04-25T00:01:19.499244+0000","last_became_peered":"2024-04-25T00:01:19.499244+0000","last_unstale":"2024-04-25T00:01:20.519505+0000","last_undegraded":"2024-04-25T00:01:20.519505+0000","last_fullsized":"2024-04-25T00:01:20.519505+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:40:36.350832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.580552+0000","last_change":"2024-04-25T00:01:05.376984+0000","last_active":"2024-04-25T00:01:20.580552+0000","last_peered":"2024-04-25T00:01:20.580552+0000","last_clean":"2024-04-25T00:01:20.580552+0000","last_became_active":"2024-04-25T00:01:05.376259+0000","last_became_peered":"2024-04-25T00:01:05.376259+0000","last_unstale":"2024-04-25T00:01:20.580552+0000","last_undegraded":"2024-04-25T00:01:20.580552+0000","last_fullsized":"2024-04-25T00:01:20.580552+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:13:28.543302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"22'2","reported_seq":24,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.581267+0000","last_change":"2024-04-25T00:01:04.359262+0000","last_active":"2024-04-25T00:01:20.581267+0000","last_peered":"2024-04-25T00:01:20.581267+0000","last_clean":"2024-04-25T00:01:20.581267+0000","last_became_active":"2024-04-25T00:01:04.357971+0000","last_became_peered":"2024-04-25T00:01:04.357971+0000","last_unstale":"2024-04-25T00:01:20.581267+0000","last_undegraded":"2024-04-25T00:01:20.581267+0000","last_fullsized":"2024-04-25T00:01:20.581267+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:05.441048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"26'1","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582746+0000","last_change":"2024-04-25T00:01:18.475624+0000","last_active":"2024-04-25T00:01:20.582746+0000","last_peered":"2024-04-25T00:01:20.582746+0000","last_clean":"2024-04-25T00:01:20.582746+0000","last_became_active":"2024-04-25T00:01:18.475149+0000","last_became_peered":"2024-04-25T00:01:18.475149+0000","last_unstale":"2024-04-25T00:01:20.582746+0000","last_undegraded":"2024-04-25T00:01:20.582746+0000","last_fullsized":"2024-04-25T00:01:20.582746+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:40:35.341075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.512956+0000","last_change":"2024-04-25T00:01:19.531113+0000","last_active":"2024-04-25T00:01:20.512956+0000","last_peered":"2024-04-25T00:01:20.512956+0000","last_clean":"2024-04-25T00:01:20.512956+0000","last_became_active":"2024-04-25T00:01:19.520150+0000","last_became_peered":"2024-04-25T00:01:19.520150+0000","last_unstale":"2024-04-25T00:01:20.512956+0000","last_undegraded":"2024-04-25T00:01:20.512956+0000","last_fullsized":"2024-04-25T00:01:20.512956+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:38.152376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511363+0000","last_change":"2024-04-25T00:01:05.362611+0000","last_active":"2024-04-25T00:01:20.511363+0000","last_peered":"2024-04-25T00:01:20.511363+0000","last_clean":"2024-04-25T00:01:20.511363+0000","last_became_active":"2024-04-25T00:01:05.361817+0000","last_became_peered":"2024-04-25T00:01:05.361817+0000","last_unstale":"2024-04-25T00:01:20.511363+0000","last_undegraded":"2024-04-25T00:01:20.511363+0000","last_fullsized":"2024-04-25T00:01:20.511363+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:50.489125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578606+0000","last_change":"2024-04-25T00:01:04.371935+0000","last_active":"2024-04-25T00:01:20.578606+0000","last_peered":"2024-04-25T00:01:20.578606+0000","last_clean":"2024-04-25T00:01:20.578606+0000","last_became_active":"2024-04-25T00:01:04.371420+0000","last_became_peered":"2024-04-25T00:01:04.371420+0000","last_unstale":"2024-04-25T00:01:20.578606+0000","last_undegraded":"2024-04-25T00:01:20.578606+0000","last_fullsized":"2024-04-25T00:01:20.578606+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582229+0000","last_change":"2024-04-25T00:01:18.488623+0000","last_active":"2024-04-25T00:01:20.582229+0000","last_peered":"2024-04-25T00:01:20.582229+0000","last_clean":"2024-04-25T00:01:20.582229+0000","last_became_active":"2024-04-25T00:01:18.488291+0000","last_became_peered":"2024-04-25T00:01:18.488291+0000","last_unstale":"2024-04-25T00:01:20.582229+0000","last_undegraded":"2024-04-25T00:01:20.582229+0000","last_fullsized":"2024-04-25T00:01:20.582229+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:30:21.436681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.584490+0000","last_change":"2024-04-25T00:01:19.516731+0000","last_active":"2024-04-25T00:01:20.584490+0000","last_peered":"2024-04-25T00:01:20.584490+0000","last_clean":"2024-04-25T00:01:20.584490+0000","last_became_active":"2024-04-25T00:01:19.516405+0000","last_became_peered":"2024-04-25T00:01:19.516405+0000","last_unstale":"2024-04-25T00:01:20.584490+0000","last_undegraded":"2024-04-25T00:01:20.584490+0000","last_fullsized":"2024-04-25T00:01:20.584490+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:44:37.936092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.510871+0000","last_change":"2024-04-25T00:01:05.355455+0000","last_active":"2024-04-25T00:01:20.510871+0000","last_peered":"2024-04-25T00:01:20.510871+0000","last_clean":"2024-04-25T00:01:20.510871+0000","last_became_active":"2024-04-25T00:01:05.354699+0000","last_became_peered":"2024-04-25T00:01:05.354699+0000","last_unstale":"2024-04-25T00:01:20.510871+0000","last_undegraded":"2024-04-25T00:01:20.510871+0000","last_fullsized":"2024-04-25T00:01:20.510871+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:43:58.069271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.512349+0000","last_change":"2024-04-25T00:01:04.362268+0000","last_active":"2024-04-25T00:01:20.512349+0000","last_peered":"2024-04-25T00:01:20.512349+0000","last_clean":"2024-04-25T00:01:20.512349+0000","last_became_active":"2024-04-25T00:01:04.361622+0000","last_became_peered":"2024-04-25T00:01:04.361622+0000","last_unstale":"2024-04-25T00:01:20.512349+0000","last_undegraded":"2024-04-25T00:01:20.512349+0000","last_fullsized":"2024-04-25T00:01:20.512349+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:58:46.021343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"26'1","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.514314+0000","last_change":"2024-04-25T00:01:18.494047+0000","last_active":"2024-04-25T00:01:20.514314+0000","last_peered":"2024-04-25T00:01:20.514314+0000","last_clean":"2024-04-25T00:01:20.514314+0000","last_became_active":"2024-04-25T00:01:18.493614+0000","last_became_peered":"2024-04-25T00:01:18.493614+0000","last_unstale":"2024-04-25T00:01:20.514314+0000","last_undegraded":"2024-04-25T00:01:20.514314+0000","last_fullsized":"2024-04-25T00:01:20.514314+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:37:08.562016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582182+0000","last_change":"2024-04-25T00:01:19.505397+0000","last_active":"2024-04-25T00:01:20.582182+0000","last_peered":"2024-04-25T00:01:20.582182+0000","last_clean":"2024-04-25T00:01:20.582182+0000","last_became_active":"2024-04-25T00:01:19.505126+0000","last_became_peered":"2024-04-25T00:01:19.505126+0000","last_unstale":"2024-04-25T00:01:20.582182+0000","last_undegraded":"2024-04-25T00:01:20.582182+0000","last_fullsized":"2024-04-25T00:01:20.582182+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:08:49.640267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.581145+0000","last_change":"2024-04-25T00:01:05.354953+0000","last_active":"2024-04-25T00:01:20.581145+0000","last_peered":"2024-04-25T00:01:20.581145+0000","last_clean":"2024-04-25T00:01:20.581145+0000","last_became_active":"2024-04-25T00:01:05.354118+0000","last_became_peered":"2024-04-25T00:01:05.354118+0000","last_unstale":"2024-04-25T00:01:20.581145+0000","last_undegraded":"2024-04-25T00:01:20.581145+0000","last_fullsized":"2024-04-25T00:01:20.581145+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:09.858617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578494+0000","last_change":"2024-04-25T00:01:04.365515+0000","last_active":"2024-04-25T00:01:20.578494+0000","last_peered":"2024-04-25T00:01:20.578494+0000","last_clean":"2024-04-25T00:01:20.578494+0000","last_became_active":"2024-04-25T00:01:04.364942+0000","last_became_peered":"2024-04-25T00:01:04.364942+0000","last_unstale":"2024-04-25T00:01:20.578494+0000","last_undegraded":"2024-04-25T00:01:20.578494+0000","last_fullsized":"2024-04-25T00:01:20.578494+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"26'2","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511306+0000","last_change":"2024-04-25T00:01:18.476207+0000","last_active":"2024-04-25T00:01:20.511306+0000","last_peered":"2024-04-25T00:01:20.511306+0000","last_clean":"2024-04-25T00:01:20.511306+0000","last_became_active":"2024-04-25T00:01:18.475862+0000","last_became_peered":"2024-04-25T00:01:18.475862+0000","last_unstale":"2024-04-25T00:01:20.511306+0000","last_undegraded":"2024-04-25T00:01:20.511306+0000","last_fullsized":"2024-04-25T00:01:20.511306+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:11:40.170249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511258+0000","last_change":"2024-04-25T00:01:19.529607+0000","last_active":"2024-04-25T00:01:20.511258+0000","last_peered":"2024-04-25T00:01:20.511258+0000","last_clean":"2024-04-25T00:01:20.511258+0000","last_became_active":"2024-04-25T00:01:19.528509+0000","last_became_peered":"2024-04-25T00:01:19.528509+0000","last_unstale":"2024-04-25T00:01:20.511258+0000","last_undegraded":"2024-04-25T00:01:20.511258+0000","last_fullsized":"2024-04-25T00:01:20.511258+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:45:01.068091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.514958+0000","last_change":"2024-04-25T00:01:05.358106+0000","last_active":"2024-04-25T00:01:20.514958+0000","last_peered":"2024-04-25T00:01:20.514958+0000","last_clean":"2024-04-25T00:01:20.514958+0000","last_became_active":"2024-04-25T00:01:05.357135+0000","last_became_peered":"2024-04-25T00:01:05.357135+0000","last_unstale":"2024-04-25T00:01:20.514958+0000","last_undegraded":"2024-04-25T00:01:20.514958+0000","last_fullsized":"2024-04-25T00:01:20.514958+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:41:42.228234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"22'2","reported_seq":24,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.584846+0000","last_change":"2024-04-25T00:01:04.369917+0000","last_active":"2024-04-25T00:01:20.584846+0000","last_peered":"2024-04-25T00:01:20.584846+0000","last_clean":"2024-04-25T00:01:20.584846+0000","last_became_active":"2024-04-25T00:01:04.369309+0000","last_became_peered":"2024-04-25T00:01:04.369309+0000","last_unstale":"2024-04-25T00:01:20.584846+0000","last_undegraded":"2024-04-25T00:01:20.584846+0000","last_fullsized":"2024-04-25T00:01:20.584846+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:55:08.063965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511314+0000","last_change":"2024-04-25T00:01:18.477189+0000","last_active":"2024-04-25T00:01:20.511314+0000","last_peered":"2024-04-25T00:01:20.511314+0000","last_clean":"2024-04-25T00:01:20.511314+0000","last_became_active":"2024-04-25T00:01:18.476909+0000","last_became_peered":"2024-04-25T00:01:18.476909+0000","last_unstale":"2024-04-25T00:01:20.511314+0000","last_undegraded":"2024-04-25T00:01:20.511314+0000","last_fullsized":"2024-04-25T00:01:20.511314+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:36.995477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.580550+0000","last_change":"2024-04-25T00:01:19.517500+0000","last_active":"2024-04-25T00:01:20.580550+0000","last_peered":"2024-04-25T00:01:20.580550+0000","last_clean":"2024-04-25T00:01:20.580550+0000","last_became_active":"2024-04-25T00:01:19.517182+0000","last_became_peered":"2024-04-25T00:01:19.517182+0000","last_unstale":"2024-04-25T00:01:20.580550+0000","last_undegraded":"2024-04-25T00:01:20.580550+0000","last_fullsized":"2024-04-25T00:01:20.580550+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:58:49.738233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.515146+0000","last_change":"2024-04-25T00:01:05.364829+0000","last_active":"2024-04-25T00:01:20.515146+0000","last_peered":"2024-04-25T00:01:20.515146+0000","last_clean":"2024-04-25T00:01:20.515146+0000","last_became_active":"2024-04-25T00:01:05.364125+0000","last_became_peered":"2024-04-25T00:01:05.364125+0000","last_unstale":"2024-04-25T00:01:20.515146+0000","last_undegraded":"2024-04-25T00:01:20.515146+0000","last_fullsized":"2024-04-25T00:01:20.515146+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:15:45.531760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"23'1","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.515821+0000","last_change":"2024-04-25T00:01:04.361603+0000","last_active":"2024-04-25T00:01:20.515821+0000","last_peered":"2024-04-25T00:01:20.515821+0000","last_clean":"2024-04-25T00:01:20.515821+0000","last_became_active":"2024-04-25T00:01:04.361046+0000","last_became_peered":"2024-04-25T00:01:04.361046+0000","last_unstale":"2024-04-25T00:01:20.515821+0000","last_undegraded":"2024-04-25T00:01:20.515821+0000","last_fullsized":"2024-04-25T00:01:20.515821+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:35:18.036843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.510620+0000","last_change":"2024-04-25T00:01:18.488352+0000","last_active":"2024-04-25T00:01:20.510620+0000","last_peered":"2024-04-25T00:01:20.510620+0000","last_clean":"2024-04-25T00:01:20.510620+0000","last_became_active":"2024-04-25T00:01:18.487321+0000","last_became_peered":"2024-04-25T00:01:18.487321+0000","last_unstale":"2024-04-25T00:01:20.510620+0000","last_undegraded":"2024-04-25T00:01:20.510620+0000","last_fullsized":"2024-04-25T00:01:20.510620+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:37:22.329655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578359+0000","last_change":"2024-04-25T00:01:19.519653+0000","last_active":"2024-04-25T00:01:20.578359+0000","last_peered":"2024-04-25T00:01:20.578359+0000","last_clean":"2024-04-25T00:01:20.578359+0000","last_became_active":"2024-04-25T00:01:19.519052+0000","last_became_peered":"2024-04-25T00:01:19.519052+0000","last_unstale":"2024-04-25T00:01:20.578359+0000","last_undegraded":"2024-04-25T00:01:20.578359+0000","last_fullsized":"2024-04-25T00:01:20.578359+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:40:36.350832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.579276+0000","last_change":"2024-04-25T00:01:05.374011+0000","last_active":"2024-04-25T00:01:20.579276+0000","last_peered":"2024-04-25T00:01:20.579276+0000","last_clean":"2024-04-25T00:01:20.579276+0000","last_became_active":"2024-04-25T00:01:05.373276+0000","last_became_peered":"2024-04-25T00:01:05.373276+0000","last_unstale":"2024-04-25T00:01:20.579276+0000","last_undegraded":"2024-04-25T00:01:20.579276+0000","last_fullsized":"2024-04-25T00:01:20.579276+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:40:53.349428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.583723+0000","last_change":"2024-04-25T00:01:04.365391+0000","last_active":"2024-04-25T00:01:20.583723+0000","last_peered":"2024-04-25T00:01:20.583723+0000","last_clean":"2024-04-25T00:01:20.583723+0000","last_became_active":"2024-04-25T00:01:04.364989+0000","last_became_peered":"2024-04-25T00:01:04.364989+0000","last_unstale":"2024-04-25T00:01:20.583723+0000","last_undegraded":"2024-04-25T00:01:20.583723+0000","last_fullsized":"2024-04-25T00:01:20.583723+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:06:18.298057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578642+0000","last_change":"2024-04-25T00:01:18.491597+0000","last_active":"2024-04-25T00:01:20.578642+0000","last_peered":"2024-04-25T00:01:20.578642+0000","last_clean":"2024-04-25T00:01:20.578642+0000","last_became_active":"2024-04-25T00:01:18.490859+0000","last_became_peered":"2024-04-25T00:01:18.490859+0000","last_unstale":"2024-04-25T00:01:20.578642+0000","last_undegraded":"2024-04-25T00:01:20.578642+0000","last_fullsized":"2024-04-25T00:01:20.578642+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:36.995477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.512778+0000","last_change":"2024-04-25T00:01:19.532926+0000","last_active":"2024-04-25T00:01:20.512778+0000","last_peered":"2024-04-25T00:01:20.512778+0000","last_clean":"2024-04-25T00:01:20.512778+0000","last_became_active":"2024-04-25T00:01:19.529742+0000","last_became_peered":"2024-04-25T00:01:19.529742+0000","last_unstale":"2024-04-25T00:01:20.512778+0000","last_undegraded":"2024-04-25T00:01:20.512778+0000","last_fullsized":"2024-04-25T00:01:20.512778+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:34:35.706412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511333+0000","last_change":"2024-04-25T00:01:05.378340+0000","last_active":"2024-04-25T00:01:20.511333+0000","last_peered":"2024-04-25T00:01:20.511333+0000","last_clean":"2024-04-25T00:01:20.511333+0000","last_became_active":"2024-04-25T00:01:05.377593+0000","last_became_peered":"2024-04-25T00:01:05.377593+0000","last_unstale":"2024-04-25T00:01:20.511333+0000","last_undegraded":"2024-04-25T00:01:20.511333+0000","last_fullsized":"2024-04-25T00:01:20.511333+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:48.202251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.519859+0000","last_change":"2024-04-25T00:01:04.366577+0000","last_active":"2024-04-25T00:01:20.519859+0000","last_peered":"2024-04-25T00:01:20.519859+0000","last_clean":"2024-04-25T00:01:20.519859+0000","last_became_active":"2024-04-25T00:01:04.365881+0000","last_became_peered":"2024-04-25T00:01:04.365881+0000","last_unstale":"2024-04-25T00:01:20.519859+0000","last_undegraded":"2024-04-25T00:01:20.519859+0000","last_fullsized":"2024-04-25T00:01:20.519859+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:49:22.673421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.577736+0000","last_change":"2024-04-25T00:01:18.504341+0000","last_active":"2024-04-25T00:01:20.577736+0000","last_peered":"2024-04-25T00:01:20.577736+0000","last_clean":"2024-04-25T00:01:20.577736+0000","last_became_active":"2024-04-25T00:01:18.491498+0000","last_became_peered":"2024-04-25T00:01:18.491498+0000","last_unstale":"2024-04-25T00:01:20.577736+0000","last_undegraded":"2024-04-25T00:01:20.577736+0000","last_fullsized":"2024-04-25T00:01:20.577736+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:56.741250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.513341+0000","last_change":"2024-04-25T00:01:19.539074+0000","last_active":"2024-04-25T00:01:20.513341+0000","last_peered":"2024-04-25T00:01:20.513341+0000","last_clean":"2024-04-25T00:01:20.513341+0000","last_became_active":"2024-04-25T00:01:19.538447+0000","last_became_peered":"2024-04-25T00:01:19.538447+0000","last_unstale":"2024-04-25T00:01:20.513341+0000","last_undegraded":"2024-04-25T00:01:20.513341+0000","last_fullsized":"2024-04-25T00:01:20.513341+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:08:49.640267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.581346+0000","last_change":"2024-04-25T00:01:05.368769+0000","last_active":"2024-04-25T00:01:20.581346+0000","last_peered":"2024-04-25T00:01:20.581346+0000","last_clean":"2024-04-25T00:01:20.581346+0000","last_became_active":"2024-04-25T00:01:05.366817+0000","last_became_peered":"2024-04-25T00:01:05.366817+0000","last_unstale":"2024-04-25T00:01:20.581346+0000","last_undegraded":"2024-04-25T00:01:20.581346+0000","last_fullsized":"2024-04-25T00:01:20.581346+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:11:27.044893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.581006+0000","last_change":"2024-04-25T00:01:04.361900+0000","last_active":"2024-04-25T00:01:20.581006+0000","last_peered":"2024-04-25T00:01:20.581006+0000","last_clean":"2024-04-25T00:01:20.581006+0000","last_became_active":"2024-04-25T00:01:04.361572+0000","last_became_peered":"2024-04-25T00:01:04.361572+0000","last_unstale":"2024-04-25T00:01:20.581006+0000","last_undegraded":"2024-04-25T00:01:20.581006+0000","last_fullsized":"2024-04-25T00:01:20.581006+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:29.218739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.579029+0000","last_change":"2024-04-25T00:01:18.504139+0000","last_active":"2024-04-25T00:01:20.579029+0000","last_peered":"2024-04-25T00:01:20.579029+0000","last_clean":"2024-04-25T00:01:20.579029+0000","last_became_active":"2024-04-25T00:01:18.503379+0000","last_became_peered":"2024-04-25T00:01:18.503379+0000","last_unstale":"2024-04-25T00:01:20.579029+0000","last_undegraded":"2024-04-25T00:01:20.579029+0000","last_fullsized":"2024-04-25T00:01:20.579029+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:02:56.363258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.514229+0000","last_change":"2024-04-25T00:01:19.531756+0000","last_active":"2024-04-25T00:01:20.514229+0000","last_peered":"2024-04-25T00:01:20.514229+0000","last_clean":"2024-04-25T00:01:20.514229+0000","last_became_active":"2024-04-25T00:01:19.531145+0000","last_became_peered":"2024-04-25T00:01:19.531145+0000","last_unstale":"2024-04-25T00:01:20.514229+0000","last_undegraded":"2024-04-25T00:01:20.514229+0000","last_fullsized":"2024-04-25T00:01:20.514229+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:13:42.678415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.518780+0000","last_change":"2024-04-25T00:01:05.354557+0000","last_active":"2024-04-25T00:01:20.518780+0000","last_peered":"2024-04-25T00:01:20.518780+0000","last_clean":"2024-04-25T00:01:20.518780+0000","last_became_active":"2024-04-25T00:01:05.353556+0000","last_became_peered":"2024-04-25T00:01:05.353556+0000","last_unstale":"2024-04-25T00:01:20.518780+0000","last_undegraded":"2024-04-25T00:01:20.518780+0000","last_fullsized":"2024-04-25T00:01:20.518780+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:24.964055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511447+0000","last_change":"2024-04-25T00:01:04.372916+0000","last_active":"2024-04-25T00:01:20.511447+0000","last_peered":"2024-04-25T00:01:20.511447+0000","last_clean":"2024-04-25T00:01:20.511447+0000","last_became_active":"2024-04-25T00:01:04.372185+0000","last_became_peered":"2024-04-25T00:01:04.372185+0000","last_unstale":"2024-04-25T00:01:20.511447+0000","last_undegraded":"2024-04-25T00:01:20.511447+0000","last_fullsized":"2024-04-25T00:01:20.511447+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"26'2","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.579299+0000","last_change":"2024-04-25T00:01:18.504614+0000","last_active":"2024-04-25T00:01:20.579299+0000","last_peered":"2024-04-25T00:01:20.579299+0000","last_clean":"2024-04-25T00:01:20.579299+0000","last_became_active":"2024-04-25T00:01:18.501448+0000","last_became_peered":"2024-04-25T00:01:18.501448+0000","last_unstale":"2024-04-25T00:01:20.579299+0000","last_undegraded":"2024-04-25T00:01:20.579299+0000","last_fullsized":"2024-04-25T00:01:20.579299+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:22.983972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.584367+0000","last_change":"2024-04-25T00:01:19.517510+0000","last_active":"2024-04-25T00:01:20.584367+0000","last_peered":"2024-04-25T00:01:20.584367+0000","last_clean":"2024-04-25T00:01:20.584367+0000","last_became_active":"2024-04-25T00:01:19.517243+0000","last_became_peered":"2024-04-25T00:01:19.517243+0000","last_unstale":"2024-04-25T00:01:20.584367+0000","last_undegraded":"2024-04-25T00:01:20.584367+0000","last_fullsized":"2024-04-25T00:01:20.584367+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:42:38.149502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.577555+0000","last_change":"2024-04-25T00:01:05.370968+0000","last_active":"2024-04-25T00:01:20.577555+0000","last_peered":"2024-04-25T00:01:20.577555+0000","last_clean":"2024-04-25T00:01:20.577555+0000","last_became_active":"2024-04-25T00:01:05.369987+0000","last_became_peered":"2024-04-25T00:01:05.369987+0000","last_unstale":"2024-04-25T00:01:20.577555+0000","last_undegraded":"2024-04-25T00:01:20.577555+0000","last_fullsized":"2024-04-25T00:01:20.577555+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.580498+0000","last_change":"2024-04-25T00:01:04.370226+0000","last_active":"2024-04-25T00:01:20.580498+0000","last_peered":"2024-04-25T00:01:20.580498+0000","last_clean":"2024-04-25T00:01:20.580498+0000","last_became_active":"2024-04-25T00:01:04.369904+0000","last_became_peered":"2024-04-25T00:01:04.369904+0000","last_unstale":"2024-04-25T00:01:20.580498+0000","last_undegraded":"2024-04-25T00:01:20.580498+0000","last_fullsized":"2024-04-25T00:01:20.580498+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:47:25.595275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"27'1","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.523846+0000","last_change":"2024-04-25T00:01:18.497156+0000","last_active":"2024-04-25T00:01:20.523846+0000","last_peered":"2024-04-25T00:01:20.523846+0000","last_clean":"2024-04-25T00:01:20.523846+0000","last_became_active":"2024-04-25T00:01:18.495573+0000","last_became_peered":"2024-04-25T00:01:18.495573+0000","last_unstale":"2024-04-25T00:01:20.523846+0000","last_undegraded":"2024-04-25T00:01:20.523846+0000","last_fullsized":"2024-04-25T00:01:20.523846+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:29.119299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.581909+0000","last_change":"2024-04-25T00:01:19.519411+0000","last_active":"2024-04-25T00:01:20.581909+0000","last_peered":"2024-04-25T00:01:20.581909+0000","last_clean":"2024-04-25T00:01:20.581909+0000","last_became_active":"2024-04-25T00:01:19.519020+0000","last_became_peered":"2024-04-25T00:01:19.519020+0000","last_unstale":"2024-04-25T00:01:20.581909+0000","last_undegraded":"2024-04-25T00:01:20.581909+0000","last_fullsized":"2024-04-25T00:01:20.581909+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:50.351599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.513395+0000","last_change":"2024-04-25T00:01:05.364939+0000","last_active":"2024-04-25T00:01:20.513395+0000","last_peered":"2024-04-25T00:01:20.513395+0000","last_clean":"2024-04-25T00:01:20.513395+0000","last_became_active":"2024-04-25T00:01:05.364272+0000","last_became_peered":"2024-04-25T00:01:05.364272+0000","last_unstale":"2024-04-25T00:01:20.513395+0000","last_undegraded":"2024-04-25T00:01:20.513395+0000","last_fullsized":"2024-04-25T00:01:20.513395+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:24:59.679908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.581536+0000","last_change":"2024-04-25T00:01:04.357028+0000","last_active":"2024-04-25T00:01:20.581536+0000","last_peered":"2024-04-25T00:01:20.581536+0000","last_clean":"2024-04-25T00:01:20.581536+0000","last_became_active":"2024-04-25T00:01:04.356644+0000","last_became_peered":"2024-04-25T00:01:04.356644+0000","last_unstale":"2024-04-25T00:01:20.581536+0000","last_undegraded":"2024-04-25T00:01:20.581536+0000","last_fullsized":"2024-04-25T00:01:20.581536+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:21.729893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":178,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.577513+0000","last_change":"2024-04-25T00:00:55.084492+0000","last_active":"2024-04-25T00:01:20.577513+0000","last_peered":"2024-04-25T00:01:20.577513+0000","last_clean":"2024-04-25T00:01:20.577513+0000","last_became_active":"2024-04-25T00:00:55.083493+0000","last_became_peered":"2024-04-25T00:00:55.083493+0000","last_unstale":"2024-04-25T00:01:20.577513+0000","last_undegraded":"2024-04-25T00:01:20.577513+0000","last_fullsized":"2024-04-25T00:01:20.577513+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:00:54.060578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:00:54.060578+0000","last_clean_scrub_stamp":"2024-04-25T00:00:54.060578+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:14:58.472477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.583214+0000","last_change":"2024-04-25T00:01:18.476298+0000","last_active":"2024-04-25T00:01:20.583214+0000","last_peered":"2024-04-25T00:01:20.583214+0000","last_clean":"2024-04-25T00:01:20.583214+0000","last_became_active":"2024-04-25T00:01:18.475898+0000","last_became_peered":"2024-04-25T00:01:18.475898+0000","last_unstale":"2024-04-25T00:01:20.583214+0000","last_undegraded":"2024-04-25T00:01:20.583214+0000","last_fullsized":"2024-04-25T00:01:20.583214+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:49.341842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.519562+0000","last_change":"2024-04-25T00:01:19.518102+0000","last_active":"2024-04-25T00:01:20.519562+0000","last_peered":"2024-04-25T00:01:20.519562+0000","last_clean":"2024-04-25T00:01:20.519562+0000","last_became_active":"2024-04-25T00:01:19.517376+0000","last_became_peered":"2024-04-25T00:01:19.517376+0000","last_unstale":"2024-04-25T00:01:20.519562+0000","last_undegraded":"2024-04-25T00:01:20.519562+0000","last_fullsized":"2024-04-25T00:01:20.519562+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:55:46.853565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.583333+0000","last_change":"2024-04-25T00:01:05.376609+0000","last_active":"2024-04-25T00:01:20.583333+0000","last_peered":"2024-04-25T00:01:20.583333+0000","last_clean":"2024-04-25T00:01:20.583333+0000","last_became_active":"2024-04-25T00:01:05.375881+0000","last_became_peered":"2024-04-25T00:01:05.375881+0000","last_unstale":"2024-04-25T00:01:20.583333+0000","last_undegraded":"2024-04-25T00:01:20.583333+0000","last_fullsized":"2024-04-25T00:01:20.583333+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:46:05.592233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.579387+0000","last_change":"2024-04-25T00:01:04.368954+0000","last_active":"2024-04-25T00:01:20.579387+0000","last_peered":"2024-04-25T00:01:20.579387+0000","last_clean":"2024-04-25T00:01:20.579387+0000","last_became_active":"2024-04-25T00:01:04.367903+0000","last_became_peered":"2024-04-25T00:01:04.367903+0000","last_unstale":"2024-04-25T00:01:20.579387+0000","last_undegraded":"2024-04-25T00:01:20.579387+0000","last_fullsized":"2024-04-25T00:01:20.579387+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511154+0000","last_change":"2024-04-25T00:01:18.476525+0000","last_active":"2024-04-25T00:01:20.511154+0000","last_peered":"2024-04-25T00:01:20.511154+0000","last_clean":"2024-04-25T00:01:20.511154+0000","last_became_active":"2024-04-25T00:01:18.476247+0000","last_became_peered":"2024-04-25T00:01:18.476247+0000","last_unstale":"2024-04-25T00:01:20.511154+0000","last_undegraded":"2024-04-25T00:01:20.511154+0000","last_fullsized":"2024-04-25T00:01:20.511154+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:30:36.682736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.577886+0000","last_change":"2024-04-25T00:01:19.514615+0000","last_active":"2024-04-25T00:01:20.577886+0000","last_peered":"2024-04-25T00:01:20.577886+0000","last_clean":"2024-04-25T00:01:20.577886+0000","last_became_active":"2024-04-25T00:01:19.513640+0000","last_became_peered":"2024-04-25T00:01:19.513640+0000","last_unstale":"2024-04-25T00:01:20.577886+0000","last_undegraded":"2024-04-25T00:01:20.577886+0000","last_fullsized":"2024-04-25T00:01:20.577886+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:08.620602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582498+0000","last_change":"2024-04-25T00:01:05.385670+0000","last_active":"2024-04-25T00:01:20.582498+0000","last_peered":"2024-04-25T00:01:20.582498+0000","last_clean":"2024-04-25T00:01:20.582498+0000","last_became_active":"2024-04-25T00:01:05.384822+0000","last_became_peered":"2024-04-25T00:01:05.384822+0000","last_unstale":"2024-04-25T00:01:20.582498+0000","last_undegraded":"2024-04-25T00:01:20.582498+0000","last_fullsized":"2024-04-25T00:01:20.582498+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:24.017263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"22'1","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.520366+0000","last_change":"2024-04-25T00:01:04.356627+0000","last_active":"2024-04-25T00:01:20.520366+0000","last_peered":"2024-04-25T00:01:20.520366+0000","last_clean":"2024-04-25T00:01:20.520366+0000","last_became_active":"2024-04-25T00:01:04.354419+0000","last_became_peered":"2024-04-25T00:01:04.354419+0000","last_unstale":"2024-04-25T00:01:20.520366+0000","last_undegraded":"2024-04-25T00:01:20.520366+0000","last_fullsized":"2024-04-25T00:01:20.520366+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:26.243635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"26'2","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.519762+0000","last_change":"2024-04-25T00:01:18.490511+0000","last_active":"2024-04-25T00:01:20.519762+0000","last_peered":"2024-04-25T00:01:20.519762+0000","last_clean":"2024-04-25T00:01:20.519762+0000","last_became_active":"2024-04-25T00:01:18.489629+0000","last_became_peered":"2024-04-25T00:01:18.489629+0000","last_unstale":"2024-04-25T00:01:20.519762+0000","last_undegraded":"2024-04-25T00:01:20.519762+0000","last_fullsized":"2024-04-25T00:01:20.519762+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:07.610846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.579469+0000","last_change":"2024-04-25T00:01:19.508931+0000","last_active":"2024-04-25T00:01:20.579469+0000","last_peered":"2024-04-25T00:01:20.579469+0000","last_clean":"2024-04-25T00:01:20.579469+0000","last_became_active":"2024-04-25T00:01:19.507867+0000","last_became_peered":"2024-04-25T00:01:19.507867+0000","last_unstale":"2024-04-25T00:01:20.579469+0000","last_undegraded":"2024-04-25T00:01:20.579469+0000","last_fullsized":"2024-04-25T00:01:20.579469+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:38.152376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.583381+0000","last_change":"2024-04-25T00:01:05.378840+0000","last_active":"2024-04-25T00:01:20.583381+0000","last_peered":"2024-04-25T00:01:20.583381+0000","last_clean":"2024-04-25T00:01:20.583381+0000","last_became_active":"2024-04-25T00:01:05.378112+0000","last_became_peered":"2024-04-25T00:01:05.378112+0000","last_unstale":"2024-04-25T00:01:20.583381+0000","last_undegraded":"2024-04-25T00:01:20.583381+0000","last_fullsized":"2024-04-25T00:01:20.583381+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:43.615895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"22'2","reported_seq":24,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.515607+0000","last_change":"2024-04-25T00:01:04.370794+0000","last_active":"2024-04-25T00:01:20.515607+0000","last_peered":"2024-04-25T00:01:20.515607+0000","last_clean":"2024-04-25T00:01:20.515607+0000","last_became_active":"2024-04-25T00:01:04.370099+0000","last_became_peered":"2024-04-25T00:01:04.370099+0000","last_unstale":"2024-04-25T00:01:20.515607+0000","last_undegraded":"2024-04-25T00:01:20.515607+0000","last_fullsized":"2024-04-25T00:01:20.515607+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:42:03.181036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.579618+0000","last_change":"2024-04-25T00:01:18.475256+0000","last_active":"2024-04-25T00:01:20.579618+0000","last_peered":"2024-04-25T00:01:20.579618+0000","last_clean":"2024-04-25T00:01:20.579618+0000","last_became_active":"2024-04-25T00:01:18.474952+0000","last_became_peered":"2024-04-25T00:01:18.474952+0000","last_unstale":"2024-04-25T00:01:20.579618+0000","last_undegraded":"2024-04-25T00:01:20.579618+0000","last_fullsized":"2024-04-25T00:01:20.579618+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:46.849870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.577592+0000","last_change":"2024-04-25T00:01:19.516833+0000","last_active":"2024-04-25T00:01:20.577592+0000","last_peered":"2024-04-25T00:01:20.577592+0000","last_clean":"2024-04-25T00:01:20.577592+0000","last_became_active":"2024-04-25T00:01:19.515926+0000","last_became_peered":"2024-04-25T00:01:19.515926+0000","last_unstale":"2024-04-25T00:01:20.577592+0000","last_undegraded":"2024-04-25T00:01:20.577592+0000","last_fullsized":"2024-04-25T00:01:20.577592+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:23:48.855381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511807+0000","last_change":"2024-04-25T00:01:05.372591+0000","last_active":"2024-04-25T00:01:20.511807+0000","last_peered":"2024-04-25T00:01:20.511807+0000","last_clean":"2024-04-25T00:01:20.511807+0000","last_became_active":"2024-04-25T00:01:05.371204+0000","last_became_peered":"2024-04-25T00:01:05.371204+0000","last_unstale":"2024-04-25T00:01:20.511807+0000","last_undegraded":"2024-04-25T00:01:20.511807+0000","last_fullsized":"2024-04-25T00:01:20.511807+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:15.879673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"22'1","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.583451+0000","last_change":"2024-04-25T00:01:04.372376+0000","last_active":"2024-04-25T00:01:20.583451+0000","last_peered":"2024-04-25T00:01:20.583451+0000","last_clean":"2024-04-25T00:01:20.583451+0000","last_became_active":"2024-04-25T00:01:04.372040+0000","last_became_peered":"2024-04-25T00:01:04.372040+0000","last_unstale":"2024-04-25T00:01:20.583451+0000","last_undegraded":"2024-04-25T00:01:20.583451+0000","last_fullsized":"2024-04-25T00:01:20.583451+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:21.729893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578134+0000","last_change":"2024-04-25T00:01:18.487548+0000","last_active":"2024-04-25T00:01:20.578134+0000","last_peered":"2024-04-25T00:01:20.578134+0000","last_clean":"2024-04-25T00:01:20.578134+0000","last_became_active":"2024-04-25T00:01:18.487227+0000","last_became_peered":"2024-04-25T00:01:18.487227+0000","last_unstale":"2024-04-25T00:01:20.578134+0000","last_undegraded":"2024-04-25T00:01:20.578134+0000","last_fullsized":"2024-04-25T00:01:20.578134+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:55.074018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.514828+0000","last_change":"2024-04-25T00:01:19.528574+0000","last_active":"2024-04-25T00:01:20.514828+0000","last_peered":"2024-04-25T00:01:20.514828+0000","last_clean":"2024-04-25T00:01:20.514828+0000","last_became_active":"2024-04-25T00:01:19.525531+0000","last_became_peered":"2024-04-25T00:01:19.525531+0000","last_unstale":"2024-04-25T00:01:20.514828+0000","last_undegraded":"2024-04-25T00:01:20.514828+0000","last_fullsized":"2024-04-25T00:01:20.514828+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:47:08.132895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.580931+0000","last_change":"2024-04-25T00:01:05.364103+0000","last_active":"2024-04-25T00:01:20.580931+0000","last_peered":"2024-04-25T00:01:20.580931+0000","last_clean":"2024-04-25T00:01:20.580931+0000","last_became_active":"2024-04-25T00:01:05.363320+0000","last_became_peered":"2024-04-25T00:01:05.363320+0000","last_unstale":"2024-04-25T00:01:20.580931+0000","last_undegraded":"2024-04-25T00:01:20.580931+0000","last_fullsized":"2024-04-25T00:01:20.580931+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:31.456792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"22'8","reported_seq":30,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511508+0000","last_change":"2024-04-25T00:01:04.358365+0000","last_active":"2024-04-25T00:01:20.511508+0000","last_peered":"2024-04-25T00:01:20.511508+0000","last_clean":"2024-04-25T00:01:20.511508+0000","last_became_active":"2024-04-25T00:01:04.357571+0000","last_became_peered":"2024-04-25T00:01:04.357571+0000","last_unstale":"2024-04-25T00:01:20.511508+0000","last_undegraded":"2024-04-25T00:01:20.511508+0000","last_fullsized":"2024-04-25T00:01:20.511508+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:50:23.023114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.520269+0000","last_change":"2024-04-25T00:01:18.497679+0000","last_active":"2024-04-25T00:01:20.520269+0000","last_peered":"2024-04-25T00:01:20.520269+0000","last_clean":"2024-04-25T00:01:20.520269+0000","last_became_active":"2024-04-25T00:01:18.496489+0000","last_became_peered":"2024-04-25T00:01:18.496489+0000","last_unstale":"2024-04-25T00:01:20.520269+0000","last_undegraded":"2024-04-25T00:01:20.520269+0000","last_fullsized":"2024-04-25T00:01:20.520269+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:23:47.845624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.584174+0000","last_change":"2024-04-25T00:01:19.498082+0000","last_active":"2024-04-25T00:01:20.584174+0000","last_peered":"2024-04-25T00:01:20.584174+0000","last_clean":"2024-04-25T00:01:20.584174+0000","last_became_active":"2024-04-25T00:01:19.496805+0000","last_became_peered":"2024-04-25T00:01:19.496805+0000","last_unstale":"2024-04-25T00:01:20.584174+0000","last_undegraded":"2024-04-25T00:01:20.584174+0000","last_fullsized":"2024-04-25T00:01:20.584174+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:57:36.256250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.577453+0000","last_change":"2024-04-25T00:01:05.373927+0000","last_active":"2024-04-25T00:01:20.577453+0000","last_peered":"2024-04-25T00:01:20.577453+0000","last_clean":"2024-04-25T00:01:20.577453+0000","last_became_active":"2024-04-25T00:01:05.373077+0000","last_became_peered":"2024-04-25T00:01:05.373077+0000","last_unstale":"2024-04-25T00:01:20.577453+0000","last_undegraded":"2024-04-25T00:01:20.577453+0000","last_fullsized":"2024-04-25T00:01:20.577453+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:50.489125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578902+0000","last_change":"2024-04-25T00:01:04.357265+0000","last_active":"2024-04-25T00:01:20.578902+0000","last_peered":"2024-04-25T00:01:20.578902+0000","last_clean":"2024-04-25T00:01:20.578902+0000","last_became_active":"2024-04-25T00:01:04.357004+0000","last_became_peered":"2024-04-25T00:01:04.357004+0000","last_unstale":"2024-04-25T00:01:20.578902+0000","last_undegraded":"2024-04-25T00:01:20.578902+0000","last_fullsized":"2024-04-25T00:01:20.578902+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:47:25.595275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.520612+0000","last_change":"2024-04-25T00:01:18.487276+0000","last_active":"2024-04-25T00:01:20.520612+0000","last_peered":"2024-04-25T00:01:20.520612+0000","last_clean":"2024-04-25T00:01:20.520612+0000","last_became_active":"2024-04-25T00:01:18.486279+0000","last_became_peered":"2024-04-25T00:01:18.486279+0000","last_unstale":"2024-04-25T00:01:20.520612+0000","last_undegraded":"2024-04-25T00:01:20.520612+0000","last_fullsized":"2024-04-25T00:01:20.520612+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:17:37.141843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582117+0000","last_change":"2024-04-25T00:01:19.516827+0000","last_active":"2024-04-25T00:01:20.582117+0000","last_peered":"2024-04-25T00:01:20.582117+0000","last_clean":"2024-04-25T00:01:20.582117+0000","last_became_active":"2024-04-25T00:01:19.515765+0000","last_became_peered":"2024-04-25T00:01:19.515765+0000","last_unstale":"2024-04-25T00:01:20.582117+0000","last_undegraded":"2024-04-25T00:01:20.582117+0000","last_fullsized":"2024-04-25T00:01:20.582117+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:13:34.153117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.579960+0000","last_change":"2024-04-25T00:01:05.379501+0000","last_active":"2024-04-25T00:01:20.579960+0000","last_peered":"2024-04-25T00:01:20.579960+0000","last_clean":"2024-04-25T00:01:20.579960+0000","last_became_active":"2024-04-25T00:01:05.378644+0000","last_became_peered":"2024-04-25T00:01:05.378644+0000","last_unstale":"2024-04-25T00:01:20.579960+0000","last_undegraded":"2024-04-25T00:01:20.579960+0000","last_fullsized":"2024-04-25T00:01:20.579960+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:08:35.505154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.580791+0000","last_change":"2024-04-25T00:01:04.360987+0000","last_active":"2024-04-25T00:01:20.580791+0000","last_peered":"2024-04-25T00:01:20.580791+0000","last_clean":"2024-04-25T00:01:20.580791+0000","last_became_active":"2024-04-25T00:01:04.360664+0000","last_became_peered":"2024-04-25T00:01:04.360664+0000","last_unstale":"2024-04-25T00:01:20.580791+0000","last_undegraded":"2024-04-25T00:01:20.580791+0000","last_fullsized":"2024-04-25T00:01:20.580791+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:55.283076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"26'6","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578827+0000","last_change":"2024-04-25T00:01:18.505577+0000","last_active":"2024-04-25T00:01:20.578827+0000","last_peered":"2024-04-25T00:01:20.578827+0000","last_clean":"2024-04-25T00:01:20.578827+0000","last_became_active":"2024-04-25T00:01:18.504963+0000","last_became_peered":"2024-04-25T00:01:18.504963+0000","last_unstale":"2024-04-25T00:01:20.578827+0000","last_undegraded":"2024-04-25T00:01:20.578827+0000","last_fullsized":"2024-04-25T00:01:20.578827+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:37.142619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.513446+0000","last_change":"2024-04-25T00:01:19.524723+0000","last_active":"2024-04-25T00:01:20.513446+0000","last_peered":"2024-04-25T00:01:20.513446+0000","last_clean":"2024-04-25T00:01:20.513446+0000","last_became_active":"2024-04-25T00:01:19.523753+0000","last_became_peered":"2024-04-25T00:01:19.523753+0000","last_unstale":"2024-04-25T00:01:20.513446+0000","last_undegraded":"2024-04-25T00:01:20.513446+0000","last_fullsized":"2024-04-25T00:01:20.513446+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:19:17.969671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578487+0000","last_change":"2024-04-25T00:01:05.379102+0000","last_active":"2024-04-25T00:01:20.578487+0000","last_peered":"2024-04-25T00:01:20.578487+0000","last_clean":"2024-04-25T00:01:20.578487+0000","last_became_active":"2024-04-25T00:01:05.377625+0000","last_became_peered":"2024-04-25T00:01:05.377625+0000","last_unstale":"2024-04-25T00:01:20.578487+0000","last_undegraded":"2024-04-25T00:01:20.578487+0000","last_fullsized":"2024-04-25T00:01:20.578487+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:15.879673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"22'2","reported_seq":24,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511726+0000","last_change":"2024-04-25T00:01:04.370640+0000","last_active":"2024-04-25T00:01:20.511726+0000","last_peered":"2024-04-25T00:01:20.511726+0000","last_clean":"2024-04-25T00:01:20.511726+0000","last_became_active":"2024-04-25T00:01:04.370031+0000","last_became_peered":"2024-04-25T00:01:04.370031+0000","last_unstale":"2024-04-25T00:01:20.511726+0000","last_undegraded":"2024-04-25T00:01:20.511726+0000","last_fullsized":"2024-04-25T00:01:20.511726+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:26.243635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"26'3","reported_seq":17,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582955+0000","last_change":"2024-04-25T00:01:18.473113+0000","last_active":"2024-04-25T00:01:20.582955+0000","last_peered":"2024-04-25T00:01:20.582955+0000","last_clean":"2024-04-25T00:01:20.582955+0000","last_became_active":"2024-04-25T00:01:18.472931+0000","last_became_peered":"2024-04-25T00:01:18.472931+0000","last_unstale":"2024-04-25T00:01:20.582955+0000","last_undegraded":"2024-04-25T00:01:20.582955+0000","last_fullsized":"2024-04-25T00:01:20.582955+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:07.498985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.512763+0000","last_change":"2024-04-25T00:01:19.533664+0000","last_active":"2024-04-25T00:01:20.512763+0000","last_peered":"2024-04-25T00:01:20.512763+0000","last_clean":"2024-04-25T00:01:20.512763+0000","last_became_active":"2024-04-25T00:01:19.533048+0000","last_became_peered":"2024-04-25T00:01:19.533048+0000","last_unstale":"2024-04-25T00:01:20.512763+0000","last_undegraded":"2024-04-25T00:01:20.512763+0000","last_fullsized":"2024-04-25T00:01:20.512763+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:45:43.141569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582298+0000","last_change":"2024-04-25T00:01:05.385827+0000","last_active":"2024-04-25T00:01:20.582298+0000","last_peered":"2024-04-25T00:01:20.582298+0000","last_clean":"2024-04-25T00:01:20.582298+0000","last_became_active":"2024-04-25T00:01:05.385258+0000","last_became_peered":"2024-04-25T00:01:05.385258+0000","last_unstale":"2024-04-25T00:01:20.582298+0000","last_undegraded":"2024-04-25T00:01:20.582298+0000","last_fullsized":"2024-04-25T00:01:20.582298+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:23.870121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"22'6","reported_seq":28,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.512128+0000","last_change":"2024-04-25T00:01:04.365185+0000","last_active":"2024-04-25T00:01:20.512128+0000","last_peered":"2024-04-25T00:01:20.512128+0000","last_clean":"2024-04-25T00:01:20.512128+0000","last_became_active":"2024-04-25T00:01:04.364930+0000","last_became_peered":"2024-04-25T00:01:04.364930+0000","last_unstale":"2024-04-25T00:01:20.512128+0000","last_undegraded":"2024-04-25T00:01:20.512128+0000","last_fullsized":"2024-04-25T00:01:20.512128+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:45:03.618340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.512982+0000","last_change":"2024-04-25T00:01:18.496220+0000","last_active":"2024-04-25T00:01:20.512982+0000","last_peered":"2024-04-25T00:01:20.512982+0000","last_clean":"2024-04-25T00:01:20.512982+0000","last_became_active":"2024-04-25T00:01:18.492850+0000","last_became_peered":"2024-04-25T00:01:18.492850+0000","last_unstale":"2024-04-25T00:01:20.512982+0000","last_undegraded":"2024-04-25T00:01:20.512982+0000","last_fullsized":"2024-04-25T00:01:20.512982+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:50:16.922314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578332+0000","last_change":"2024-04-25T00:01:19.515725+0000","last_active":"2024-04-25T00:01:20.578332+0000","last_peered":"2024-04-25T00:01:20.578332+0000","last_clean":"2024-04-25T00:01:20.578332+0000","last_became_active":"2024-04-25T00:01:19.509285+0000","last_became_peered":"2024-04-25T00:01:19.509285+0000","last_unstale":"2024-04-25T00:01:20.578332+0000","last_undegraded":"2024-04-25T00:01:20.578332+0000","last_fullsized":"2024-04-25T00:01:20.578332+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:25:20.185355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.581530+0000","last_change":"2024-04-25T00:01:05.371061+0000","last_active":"2024-04-25T00:01:20.581530+0000","last_peered":"2024-04-25T00:01:20.581530+0000","last_clean":"2024-04-25T00:01:20.581530+0000","last_became_active":"2024-04-25T00:01:05.369950+0000","last_became_peered":"2024-04-25T00:01:05.369950+0000","last_unstale":"2024-04-25T00:01:20.581530+0000","last_undegraded":"2024-04-25T00:01:20.581530+0000","last_fullsized":"2024-04-25T00:01:20.581530+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:02:43.237902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"22'3","reported_seq":25,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.513687+0000","last_change":"2024-04-25T00:01:04.366503+0000","last_active":"2024-04-25T00:01:20.513687+0000","last_peered":"2024-04-25T00:01:20.513687+0000","last_clean":"2024-04-25T00:01:20.513687+0000","last_became_active":"2024-04-25T00:01:04.365945+0000","last_became_peered":"2024-04-25T00:01:04.365945+0000","last_unstale":"2024-04-25T00:01:20.513687+0000","last_undegraded":"2024-04-25T00:01:20.513687+0000","last_fullsized":"2024-04-25T00:01:20.513687+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"26'2","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511434+0000","last_change":"2024-04-25T00:01:18.476456+0000","last_active":"2024-04-25T00:01:20.511434+0000","last_peered":"2024-04-25T00:01:20.511434+0000","last_clean":"2024-04-25T00:01:20.511434+0000","last_became_active":"2024-04-25T00:01:18.475876+0000","last_became_peered":"2024-04-25T00:01:18.475876+0000","last_unstale":"2024-04-25T00:01:20.511434+0000","last_undegraded":"2024-04-25T00:01:20.511434+0000","last_fullsized":"2024-04-25T00:01:20.511434+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:56.741250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.580191+0000","last_change":"2024-04-25T00:01:19.515376+0000","last_active":"2024-04-25T00:01:20.580191+0000","last_peered":"2024-04-25T00:01:20.580191+0000","last_clean":"2024-04-25T00:01:20.580191+0000","last_became_active":"2024-04-25T00:01:19.512425+0000","last_became_peered":"2024-04-25T00:01:19.512425+0000","last_unstale":"2024-04-25T00:01:20.580191+0000","last_undegraded":"2024-04-25T00:01:20.580191+0000","last_fullsized":"2024-04-25T00:01:20.580191+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:43:16.957273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.520943+0000","last_change":"2024-04-25T00:01:05.361723+0000","last_active":"2024-04-25T00:01:20.520943+0000","last_peered":"2024-04-25T00:01:20.520943+0000","last_clean":"2024-04-25T00:01:20.520943+0000","last_became_active":"2024-04-25T00:01:05.360956+0000","last_became_peered":"2024-04-25T00:01:05.360956+0000","last_unstale":"2024-04-25T00:01:20.520943+0000","last_undegraded":"2024-04-25T00:01:20.520943+0000","last_fullsized":"2024-04-25T00:01:20.520943+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:23.870121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.580983+0000","last_change":"2024-04-25T00:01:04.362129+0000","last_active":"2024-04-25T00:01:20.580983+0000","last_peered":"2024-04-25T00:01:20.580983+0000","last_clean":"2024-04-25T00:01:20.580983+0000","last_became_active":"2024-04-25T00:01:04.361827+0000","last_became_peered":"2024-04-25T00:01:04.361827+0000","last_unstale":"2024-04-25T00:01:20.580983+0000","last_undegraded":"2024-04-25T00:01:20.580983+0000","last_fullsized":"2024-04-25T00:01:20.580983+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:26.243635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"26'8","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.510624+0000","last_change":"2024-04-25T00:01:18.496129+0000","last_active":"2024-04-25T00:01:20.510624+0000","last_peered":"2024-04-25T00:01:20.510624+0000","last_clean":"2024-04-25T00:01:20.510624+0000","last_became_active":"2024-04-25T00:01:18.495415+0000","last_became_peered":"2024-04-25T00:01:18.495415+0000","last_unstale":"2024-04-25T00:01:20.510624+0000","last_undegraded":"2024-04-25T00:01:20.510624+0000","last_fullsized":"2024-04-25T00:01:20.510624+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:38.089411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.510763+0000","last_change":"2024-04-25T00:01:19.509542+0000","last_active":"2024-04-25T00:01:20.510763+0000","last_peered":"2024-04-25T00:01:20.510763+0000","last_clean":"2024-04-25T00:01:20.510763+0000","last_became_active":"2024-04-25T00:01:19.509015+0000","last_became_peered":"2024-04-25T00:01:19.509015+0000","last_unstale":"2024-04-25T00:01:20.510763+0000","last_undegraded":"2024-04-25T00:01:20.510763+0000","last_fullsized":"2024-04-25T00:01:20.510763+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:25:20.185355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511808+0000","last_change":"2024-04-25T00:01:05.365413+0000","last_active":"2024-04-25T00:01:20.511808+0000","last_peered":"2024-04-25T00:01:20.511808+0000","last_clean":"2024-04-25T00:01:20.511808+0000","last_became_active":"2024-04-25T00:01:05.361865+0000","last_became_peered":"2024-04-25T00:01:05.361865+0000","last_unstale":"2024-04-25T00:01:20.511808+0000","last_undegraded":"2024-04-25T00:01:20.511808+0000","last_fullsized":"2024-04-25T00:01:20.511808+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:30.228060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582499+0000","last_change":"2024-04-25T00:01:04.360804+0000","last_active":"2024-04-25T00:01:20.582499+0000","last_peered":"2024-04-25T00:01:20.582499+0000","last_clean":"2024-04-25T00:01:20.582499+0000","last_became_active":"2024-04-25T00:01:04.360374+0000","last_became_peered":"2024-04-25T00:01:04.360374+0000","last_unstale":"2024-04-25T00:01:20.582499+0000","last_undegraded":"2024-04-25T00:01:20.582499+0000","last_fullsized":"2024-04-25T00:01:20.582499+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511980+0000","last_change":"2024-04-25T00:01:18.480003+0000","last_active":"2024-04-25T00:01:20.511980+0000","last_peered":"2024-04-25T00:01:20.511980+0000","last_clean":"2024-04-25T00:01:20.511980+0000","last_became_active":"2024-04-25T00:01:18.479270+0000","last_became_peered":"2024-04-25T00:01:18.479270+0000","last_unstale":"2024-04-25T00:01:20.511980+0000","last_undegraded":"2024-04-25T00:01:20.511980+0000","last_fullsized":"2024-04-25T00:01:20.511980+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:52:40.942502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511885+0000","last_change":"2024-04-25T00:01:19.524463+0000","last_active":"2024-04-25T00:01:20.511885+0000","last_peered":"2024-04-25T00:01:20.511885+0000","last_clean":"2024-04-25T00:01:20.511885+0000","last_became_active":"2024-04-25T00:01:19.524166+0000","last_became_peered":"2024-04-25T00:01:19.524166+0000","last_unstale":"2024-04-25T00:01:20.511885+0000","last_undegraded":"2024-04-25T00:01:20.511885+0000","last_fullsized":"2024-04-25T00:01:20.511885+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:50.351599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"26'1","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.514720+0000","last_change":"2024-04-25T00:01:18.497096+0000","last_active":"2024-04-25T00:01:20.514720+0000","last_peered":"2024-04-25T00:01:20.514720+0000","last_clean":"2024-04-25T00:01:20.514720+0000","last_became_active":"2024-04-25T00:01:18.496289+0000","last_became_peered":"2024-04-25T00:01:18.496289+0000","last_unstale":"2024-04-25T00:01:20.514720+0000","last_undegraded":"2024-04-25T00:01:20.514720+0000","last_fullsized":"2024-04-25T00:01:20.514720+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:59:40.316831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.512256+0000","last_change":"2024-04-25T00:01:05.370043+0000","last_active":"2024-04-25T00:01:20.512256+0000","last_peered":"2024-04-25T00:01:20.512256+0000","last_clean":"2024-04-25T00:01:20.512256+0000","last_became_active":"2024-04-25T00:01:05.368080+0000","last_became_peered":"2024-04-25T00:01:05.368080+0000","last_unstale":"2024-04-25T00:01:20.512256+0000","last_undegraded":"2024-04-25T00:01:20.512256+0000","last_fullsized":"2024-04-25T00:01:20.512256+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:58:07.610923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.580256+0000","last_change":"2024-04-25T00:01:04.364523+0000","last_active":"2024-04-25T00:01:20.580256+0000","last_peered":"2024-04-25T00:01:20.580256+0000","last_clean":"2024-04-25T00:01:20.580256+0000","last_became_active":"2024-04-25T00:01:04.364050+0000","last_became_peered":"2024-04-25T00:01:04.364050+0000","last_unstale":"2024-04-25T00:01:20.580256+0000","last_undegraded":"2024-04-25T00:01:20.580256+0000","last_fullsized":"2024-04-25T00:01:20.580256+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:49.479804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.579360+0000","last_change":"2024-04-25T00:01:19.491523+0000","last_active":"2024-04-25T00:01:20.579360+0000","last_peered":"2024-04-25T00:01:20.579360+0000","last_clean":"2024-04-25T00:01:20.579360+0000","last_became_active":"2024-04-25T00:01:19.490755+0000","last_became_peered":"2024-04-25T00:01:19.490755+0000","last_unstale":"2024-04-25T00:01:20.579360+0000","last_undegraded":"2024-04-25T00:01:20.579360+0000","last_fullsized":"2024-04-25T00:01:20.579360+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:47:19.526845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"26'4","reported_seq":18,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.584296+0000","last_change":"2024-04-25T00:01:18.472794+0000","last_active":"2024-04-25T00:01:20.584296+0000","last_peered":"2024-04-25T00:01:20.584296+0000","last_clean":"2024-04-25T00:01:20.584296+0000","last_became_active":"2024-04-25T00:01:18.472534+0000","last_became_peered":"2024-04-25T00:01:18.472534+0000","last_unstale":"2024-04-25T00:01:20.584296+0000","last_undegraded":"2024-04-25T00:01:20.584296+0000","last_fullsized":"2024-04-25T00:01:20.584296+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:55:45.843808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.518835+0000","last_change":"2024-04-25T00:01:05.378867+0000","last_active":"2024-04-25T00:01:20.518835+0000","last_peered":"2024-04-25T00:01:20.518835+0000","last_clean":"2024-04-25T00:01:20.518835+0000","last_became_active":"2024-04-25T00:01:05.378056+0000","last_became_peered":"2024-04-25T00:01:05.378056+0000","last_unstale":"2024-04-25T00:01:20.518835+0000","last_undegraded":"2024-04-25T00:01:20.518835+0000","last_fullsized":"2024-04-25T00:01:20.518835+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:36:10.377757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.577806+0000","last_change":"2024-04-25T00:01:04.370896+0000","last_active":"2024-04-25T00:01:20.577806+0000","last_peered":"2024-04-25T00:01:20.577806+0000","last_clean":"2024-04-25T00:01:20.577806+0000","last_became_active":"2024-04-25T00:01:04.370273+0000","last_became_peered":"2024-04-25T00:01:04.370273+0000","last_unstale":"2024-04-25T00:01:20.577806+0000","last_undegraded":"2024-04-25T00:01:20.577806+0000","last_fullsized":"2024-04-25T00:01:20.577806+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:49:22.673421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.514211+0000","last_change":"2024-04-25T00:01:19.521143+0000","last_active":"2024-04-25T00:01:20.514211+0000","last_peered":"2024-04-25T00:01:20.514211+0000","last_clean":"2024-04-25T00:01:20.514211+0000","last_became_active":"2024-04-25T00:01:19.514664+0000","last_became_peered":"2024-04-25T00:01:19.514664+0000","last_unstale":"2024-04-25T00:01:20.514211+0000","last_undegraded":"2024-04-25T00:01:20.514211+0000","last_fullsized":"2024-04-25T00:01:20.514211+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:03:47.440442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"22'1","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.580606+0000","last_change":"2024-04-25T00:01:04.360254+0000","last_active":"2024-04-25T00:01:20.580606+0000","last_peered":"2024-04-25T00:01:20.580606+0000","last_clean":"2024-04-25T00:01:20.580606+0000","last_became_active":"2024-04-25T00:01:04.359586+0000","last_became_peered":"2024-04-25T00:01:04.359586+0000","last_unstale":"2024-04-25T00:01:20.580606+0000","last_undegraded":"2024-04-25T00:01:20.580606+0000","last_fullsized":"2024-04-25T00:01:20.580606+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:47.192930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.513293+0000","last_change":"2024-04-25T00:01:05.370723+0000","last_active":"2024-04-25T00:01:20.513293+0000","last_peered":"2024-04-25T00:01:20.513293+0000","last_clean":"2024-04-25T00:01:20.513293+0000","last_became_active":"2024-04-25T00:01:05.368970+0000","last_became_peered":"2024-04-25T00:01:05.368970+0000","last_unstale":"2024-04-25T00:01:20.513293+0000","last_undegraded":"2024-04-25T00:01:20.513293+0000","last_fullsized":"2024-04-25T00:01:20.513293+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:06:53.743074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"26'2","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.513582+0000","last_change":"2024-04-25T00:01:18.498570+0000","last_active":"2024-04-25T00:01:20.513582+0000","last_peered":"2024-04-25T00:01:20.513582+0000","last_clean":"2024-04-25T00:01:20.513582+0000","last_became_active":"2024-04-25T00:01:18.497745+0000","last_became_peered":"2024-04-25T00:01:18.497745+0000","last_unstale":"2024-04-25T00:01:20.513582+0000","last_undegraded":"2024-04-25T00:01:20.513582+0000","last_fullsized":"2024-04-25T00:01:20.513582+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:26:42.047149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511888+0000","last_change":"2024-04-25T00:01:19.514919+0000","last_active":"2024-04-25T00:01:20.511888+0000","last_peered":"2024-04-25T00:01:20.511888+0000","last_clean":"2024-04-25T00:01:20.511888+0000","last_became_active":"2024-04-25T00:01:19.514280+0000","last_became_peered":"2024-04-25T00:01:19.514280+0000","last_unstale":"2024-04-25T00:01:20.511888+0000","last_undegraded":"2024-04-25T00:01:20.511888+0000","last_fullsized":"2024-04-25T00:01:20.511888+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:37.975265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"22'1","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.519944+0000","last_change":"2024-04-25T00:01:04.365607+0000","last_active":"2024-04-25T00:01:20.519944+0000","last_peered":"2024-04-25T00:01:20.519944+0000","last_clean":"2024-04-25T00:01:20.519944+0000","last_became_active":"2024-04-25T00:01:04.364788+0000","last_became_peered":"2024-04-25T00:01:04.364788+0000","last_unstale":"2024-04-25T00:01:20.519944+0000","last_undegraded":"2024-04-25T00:01:20.519944+0000","last_fullsized":"2024-04-25T00:01:20.519944+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.579248+0000","last_change":"2024-04-25T00:01:05.375142+0000","last_active":"2024-04-25T00:01:20.579248+0000","last_peered":"2024-04-25T00:01:20.579248+0000","last_clean":"2024-04-25T00:01:20.579248+0000","last_became_active":"2024-04-25T00:01:05.373857+0000","last_became_peered":"2024-04-25T00:01:05.373857+0000","last_unstale":"2024-04-25T00:01:20.579248+0000","last_undegraded":"2024-04-25T00:01:20.579248+0000","last_fullsized":"2024-04-25T00:01:20.579248+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:15.879673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"26'2","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.513534+0000","last_change":"2024-04-25T00:01:18.502786+0000","last_active":"2024-04-25T00:01:20.513534+0000","last_peered":"2024-04-25T00:01:20.513534+0000","last_clean":"2024-04-25T00:01:20.513534+0000","last_became_active":"2024-04-25T00:01:18.501785+0000","last_became_peered":"2024-04-25T00:01:18.501785+0000","last_unstale":"2024-04-25T00:01:20.513534+0000","last_undegraded":"2024-04-25T00:01:20.513534+0000","last_fullsized":"2024-04-25T00:01:20.513534+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:09:31.441044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.514560+0000","last_change":"2024-04-25T00:01:19.511239+0000","last_active":"2024-04-25T00:01:20.514560+0000","last_peered":"2024-04-25T00:01:20.514560+0000","last_clean":"2024-04-25T00:01:20.514560+0000","last_became_active":"2024-04-25T00:01:19.510155+0000","last_became_peered":"2024-04-25T00:01:19.510155+0000","last_unstale":"2024-04-25T00:01:20.514560+0000","last_undegraded":"2024-04-25T00:01:20.514560+0000","last_fullsized":"2024-04-25T00:01:20.514560+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:43:14.452663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511273+0000","last_change":"2024-04-25T00:01:05.378517+0000","last_active":"2024-04-25T00:01:20.511273+0000","last_peered":"2024-04-25T00:01:20.511273+0000","last_clean":"2024-04-25T00:01:20.511273+0000","last_became_active":"2024-04-25T00:01:05.377742+0000","last_became_peered":"2024-04-25T00:01:05.377742+0000","last_unstale":"2024-04-25T00:01:20.511273+0000","last_undegraded":"2024-04-25T00:01:20.511273+0000","last_fullsized":"2024-04-25T00:01:20.511273+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:06.450369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.514828+0000","last_change":"2024-04-25T00:01:19.508999+0000","last_active":"2024-04-25T00:01:20.514828+0000","last_peered":"2024-04-25T00:01:20.514828+0000","last_clean":"2024-04-25T00:01:20.514828+0000","last_became_active":"2024-04-25T00:01:19.507960+0000","last_became_peered":"2024-04-25T00:01:19.507960+0000","last_unstale":"2024-04-25T00:01:20.514828+0000","last_undegraded":"2024-04-25T00:01:20.514828+0000","last_fullsized":"2024-04-25T00:01:20.514828+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:32:40.274908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.510941+0000","last_change":"2024-04-25T00:01:05.372927+0000","last_active":"2024-04-25T00:01:20.510941+0000","last_peered":"2024-04-25T00:01:20.510941+0000","last_clean":"2024-04-25T00:01:20.510941+0000","last_became_active":"2024-04-25T00:01:05.372050+0000","last_became_peered":"2024-04-25T00:01:05.372050+0000","last_unstale":"2024-04-25T00:01:20.510941+0000","last_undegraded":"2024-04-25T00:01:20.510941+0000","last_fullsized":"2024-04-25T00:01:20.510941+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:48.202251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511854+0000","last_change":"2024-04-25T00:01:19.531072+0000","last_active":"2024-04-25T00:01:20.511854+0000","last_peered":"2024-04-25T00:01:20.511854+0000","last_clean":"2024-04-25T00:01:20.511854+0000","last_became_active":"2024-04-25T00:01:19.530836+0000","last_became_peered":"2024-04-25T00:01:19.530836+0000","last_unstale":"2024-04-25T00:01:20.511854+0000","last_undegraded":"2024-04-25T00:01:20.511854+0000","last_fullsized":"2024-04-25T00:01:20.511854+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:08.508741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.580209+0000","last_change":"2024-04-25T00:01:05.362960+0000","last_active":"2024-04-25T00:01:20.580209+0000","last_peered":"2024-04-25T00:01:20.580209+0000","last_clean":"2024-04-25T00:01:20.580209+0000","last_became_active":"2024-04-25T00:01:05.361067+0000","last_became_peered":"2024-04-25T00:01:05.361067+0000","last_unstale":"2024-04-25T00:01:20.580209+0000","last_undegraded":"2024-04-25T00:01:20.580209+0000","last_fullsized":"2024-04-25T00:01:20.580209+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:24.017263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.520235+0000","last_change":"2024-04-25T00:01:19.506029+0000","last_active":"2024-04-25T00:01:20.520235+0000","last_peered":"2024-04-25T00:01:20.520235+0000","last_clean":"2024-04-25T00:01:20.520235+0000","last_became_active":"2024-04-25T00:01:19.505184+0000","last_became_peered":"2024-04-25T00:01:19.505184+0000","last_unstale":"2024-04-25T00:01:20.520235+0000","last_undegraded":"2024-04-25T00:01:20.520235+0000","last_fullsized":"2024-04-25T00:01:20.520235+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:08.508741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.519736+0000","last_change":"2024-04-25T00:01:05.358690+0000","last_active":"2024-04-25T00:01:20.519736+0000","last_peered":"2024-04-25T00:01:20.519736+0000","last_clean":"2024-04-25T00:01:20.519736+0000","last_became_active":"2024-04-25T00:01:05.357340+0000","last_became_peered":"2024-04-25T00:01:05.357340+0000","last_unstale":"2024-04-25T00:01:20.519736+0000","last_undegraded":"2024-04-25T00:01:20.519736+0000","last_fullsized":"2024-04-25T00:01:20.519736+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:36:55.436661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.577993+0000","last_change":"2024-04-25T00:01:19.508691+0000","last_active":"2024-04-25T00:01:20.577993+0000","last_peered":"2024-04-25T00:01:20.577993+0000","last_clean":"2024-04-25T00:01:20.577993+0000","last_became_active":"2024-04-25T00:01:19.508123+0000","last_became_peered":"2024-04-25T00:01:19.508123+0000","last_unstale":"2024-04-25T00:01:20.577993+0000","last_undegraded":"2024-04-25T00:01:20.577993+0000","last_fullsized":"2024-04-25T00:01:20.577993+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:47:19.526845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578984+0000","last_change":"2024-04-25T00:01:05.387057+0000","last_active":"2024-04-25T00:01:20.578984+0000","last_peered":"2024-04-25T00:01:20.578984+0000","last_clean":"2024-04-25T00:01:20.578984+0000","last_became_active":"2024-04-25T00:01:05.386724+0000","last_became_peered":"2024-04-25T00:01:05.386724+0000","last_unstale":"2024-04-25T00:01:20.578984+0000","last_undegraded":"2024-04-25T00:01:20.578984+0000","last_fullsized":"2024-04-25T00:01:20.578984+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:55:09.073286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.580476+0000","last_change":"2024-04-25T00:01:19.522760+0000","last_active":"2024-04-25T00:01:20.580476+0000","last_peered":"2024-04-25T00:01:20.580476+0000","last_clean":"2024-04-25T00:01:20.580476+0000","last_became_active":"2024-04-25T00:01:19.522380+0000","last_became_peered":"2024-04-25T00:01:19.522380+0000","last_unstale":"2024-04-25T00:01:20.580476+0000","last_undegraded":"2024-04-25T00:01:20.580476+0000","last_fullsized":"2024-04-25T00:01:20.580476+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:55:46.853565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578828+0000","last_change":"2024-04-25T00:01:05.367296+0000","last_active":"2024-04-25T00:01:20.578828+0000","last_peered":"2024-04-25T00:01:20.578828+0000","last_clean":"2024-04-25T00:01:20.578828+0000","last_became_active":"2024-04-25T00:01:05.363838+0000","last_became_peered":"2024-04-25T00:01:05.363838+0000","last_unstale":"2024-04-25T00:01:20.578828+0000","last_undegraded":"2024-04-25T00:01:20.578828+0000","last_fullsized":"2024-04-25T00:01:20.578828+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:43:07.053074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.579599+0000","last_change":"2024-04-25T00:01:19.510902+0000","last_active":"2024-04-25T00:01:20.579599+0000","last_peered":"2024-04-25T00:01:20.579599+0000","last_clean":"2024-04-25T00:01:20.579599+0000","last_became_active":"2024-04-25T00:01:19.510008+0000","last_became_peered":"2024-04-25T00:01:19.510008+0000","last_unstale":"2024-04-25T00:01:20.579599+0000","last_undegraded":"2024-04-25T00:01:20.579599+0000","last_fullsized":"2024-04-25T00:01:20.579599+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:40:32.292140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511704+0000","last_change":"2024-04-25T00:01:05.365761+0000","last_active":"2024-04-25T00:01:20.511704+0000","last_peered":"2024-04-25T00:01:20.511704+0000","last_clean":"2024-04-25T00:01:20.511704+0000","last_became_active":"2024-04-25T00:01:05.361119+0000","last_became_peered":"2024-04-25T00:01:05.361119+0000","last_unstale":"2024-04-25T00:01:20.511704+0000","last_undegraded":"2024-04-25T00:01:20.511704+0000","last_fullsized":"2024-04-25T00:01:20.511704+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.510983+0000","last_change":"2024-04-25T00:01:19.518534+0000","last_active":"2024-04-25T00:01:20.510983+0000","last_peered":"2024-04-25T00:01:20.510983+0000","last_clean":"2024-04-25T00:01:20.510983+0000","last_became_active":"2024-04-25T00:01:19.518110+0000","last_became_peered":"2024-04-25T00:01:19.518110+0000","last_unstale":"2024-04-25T00:01:20.510983+0000","last_undegraded":"2024-04-25T00:01:20.510983+0000","last_fullsized":"2024-04-25T00:01:20.510983+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:05:09.219823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.520770+0000","last_change":"2024-04-25T00:01:05.373464+0000","last_active":"2024-04-25T00:01:20.520770+0000","last_peered":"2024-04-25T00:01:20.520770+0000","last_clean":"2024-04-25T00:01:20.520770+0000","last_became_active":"2024-04-25T00:01:05.372611+0000","last_became_peered":"2024-04-25T00:01:05.372611+0000","last_unstale":"2024-04-25T00:01:20.520770+0000","last_undegraded":"2024-04-25T00:01:20.520770+0000","last_fullsized":"2024-04-25T00:01:20.520770+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:31.456792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.512974+0000","last_change":"2024-04-25T00:01:19.542475+0000","last_active":"2024-04-25T00:01:20.512974+0000","last_peered":"2024-04-25T00:01:20.512974+0000","last_clean":"2024-04-25T00:01:20.512974+0000","last_became_active":"2024-04-25T00:01:19.541877+0000","last_became_peered":"2024-04-25T00:01:19.541877+0000","last_unstale":"2024-04-25T00:01:20.512974+0000","last_undegraded":"2024-04-25T00:01:20.512974+0000","last_fullsized":"2024-04-25T00:01:20.512974+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:02:57.373014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.580023+0000","last_change":"2024-04-25T00:01:05.372906+0000","last_active":"2024-04-25T00:01:20.580023+0000","last_peered":"2024-04-25T00:01:20.580023+0000","last_clean":"2024-04-25T00:01:20.580023+0000","last_became_active":"2024-04-25T00:01:05.372148+0000","last_became_peered":"2024-04-25T00:01:05.372148+0000","last_unstale":"2024-04-25T00:01:20.580023+0000","last_undegraded":"2024-04-25T00:01:20.580023+0000","last_fullsized":"2024-04-25T00:01:20.580023+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:43.615895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.584868+0000","last_change":"2024-04-25T00:01:19.498515+0000","last_active":"2024-04-25T00:01:20.584868+0000","last_peered":"2024-04-25T00:01:20.584868+0000","last_clean":"2024-04-25T00:01:20.584868+0000","last_became_active":"2024-04-25T00:01:19.496310+0000","last_became_peered":"2024-04-25T00:01:19.496310+0000","last_unstale":"2024-04-25T00:01:20.584868+0000","last_undegraded":"2024-04-25T00:01:20.584868+0000","last_fullsized":"2024-04-25T00:01:20.584868+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:20:06.661558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.519325+0000","last_change":"2024-04-25T00:01:05.366340+0000","last_active":"2024-04-25T00:01:20.519325+0000","last_peered":"2024-04-25T00:01:20.519325+0000","last_clean":"2024-04-25T00:01:20.519325+0000","last_became_active":"2024-04-25T00:01:05.361952+0000","last_became_peered":"2024-04-25T00:01:05.361952+0000","last_unstale":"2024-04-25T00:01:20.519325+0000","last_undegraded":"2024-04-25T00:01:20.519325+0000","last_fullsized":"2024-04-25T00:01:20.519325+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:41.948662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578558+0000","last_change":"2024-04-25T00:01:19.508257+0000","last_active":"2024-04-25T00:01:20.578558+0000","last_peered":"2024-04-25T00:01:20.578558+0000","last_clean":"2024-04-25T00:01:20.578558+0000","last_became_active":"2024-04-25T00:01:19.507516+0000","last_became_peered":"2024-04-25T00:01:19.507516+0000","last_unstale":"2024-04-25T00:01:20.578558+0000","last_undegraded":"2024-04-25T00:01:20.578558+0000","last_fullsized":"2024-04-25T00:01:20.578558+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:45.121513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.579066+0000","last_change":"2024-04-25T00:01:05.387457+0000","last_active":"2024-04-25T00:01:20.579066+0000","last_peered":"2024-04-25T00:01:20.579066+0000","last_clean":"2024-04-25T00:01:20.579066+0000","last_became_active":"2024-04-25T00:01:05.386919+0000","last_became_peered":"2024-04-25T00:01:05.386919+0000","last_unstale":"2024-04-25T00:01:20.579066+0000","last_undegraded":"2024-04-25T00:01:20.579066+0000","last_fullsized":"2024-04-25T00:01:20.579066+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.512398+0000","last_change":"2024-04-25T00:01:19.520648+0000","last_active":"2024-04-25T00:01:20.512398+0000","last_peered":"2024-04-25T00:01:20.512398+0000","last_clean":"2024-04-25T00:01:20.512398+0000","last_became_active":"2024-04-25T00:01:19.519303+0000","last_became_peered":"2024-04-25T00:01:19.519303+0000","last_unstale":"2024-04-25T00:01:20.512398+0000","last_undegraded":"2024-04-25T00:01:20.512398+0000","last_fullsized":"2024-04-25T00:01:20.512398+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:23.993729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.513073+0000","last_change":"2024-04-25T00:01:05.366249+0000","last_active":"2024-04-25T00:01:20.513073+0000","last_peered":"2024-04-25T00:01:20.513073+0000","last_clean":"2024-04-25T00:01:20.513073+0000","last_became_active":"2024-04-25T00:01:05.365548+0000","last_became_peered":"2024-04-25T00:01:05.365548+0000","last_unstale":"2024-04-25T00:01:20.513073+0000","last_undegraded":"2024-04-25T00:01:20.513073+0000","last_fullsized":"2024-04-25T00:01:20.513073+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:22:21.143639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.510804+0000","last_change":"2024-04-25T00:01:19.514659+0000","last_active":"2024-04-25T00:01:20.510804+0000","last_peered":"2024-04-25T00:01:20.510804+0000","last_clean":"2024-04-25T00:01:20.510804+0000","last_became_active":"2024-04-25T00:01:19.514232+0000","last_became_peered":"2024-04-25T00:01:19.514232+0000","last_unstale":"2024-04-25T00:01:20.510804+0000","last_undegraded":"2024-04-25T00:01:20.510804+0000","last_fullsized":"2024-04-25T00:01:20.510804+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:45.121513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582289+0000","last_change":"2024-04-25T00:01:05.385253+0000","last_active":"2024-04-25T00:01:20.582289+0000","last_peered":"2024-04-25T00:01:20.582289+0000","last_clean":"2024-04-25T00:01:20.582289+0000","last_became_active":"2024-04-25T00:01:05.379429+0000","last_became_peered":"2024-04-25T00:01:05.379429+0000","last_unstale":"2024-04-25T00:01:20.582289+0000","last_undegraded":"2024-04-25T00:01:20.582289+0000","last_fullsized":"2024-04-25T00:01:20.582289+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:36:10.377757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.581087+0000","last_change":"2024-04-25T00:01:19.503419+0000","last_active":"2024-04-25T00:01:20.581087+0000","last_peered":"2024-04-25T00:01:20.581087+0000","last_clean":"2024-04-25T00:01:20.581087+0000","last_became_active":"2024-04-25T00:01:19.500568+0000","last_became_peered":"2024-04-25T00:01:19.500568+0000","last_unstale":"2024-04-25T00:01:20.581087+0000","last_undegraded":"2024-04-25T00:01:20.581087+0000","last_fullsized":"2024-04-25T00:01:20.581087+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:08.508741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.583587+0000","last_change":"2024-04-25T00:01:05.370653+0000","last_active":"2024-04-25T00:01:20.583587+0000","last_peered":"2024-04-25T00:01:20.583587+0000","last_clean":"2024-04-25T00:01:20.583587+0000","last_became_active":"2024-04-25T00:01:05.369919+0000","last_became_peered":"2024-04-25T00:01:05.369919+0000","last_unstale":"2024-04-25T00:01:20.583587+0000","last_undegraded":"2024-04-25T00:01:20.583587+0000","last_fullsized":"2024-04-25T00:01:20.583587+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:30:08.311326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.510560+0000","last_change":"2024-04-25T00:01:19.540104+0000","last_active":"2024-04-25T00:01:20.510560+0000","last_peered":"2024-04-25T00:01:20.510560+0000","last_clean":"2024-04-25T00:01:20.510560+0000","last_became_active":"2024-04-25T00:01:19.539775+0000","last_became_peered":"2024-04-25T00:01:19.539775+0000","last_unstale":"2024-04-25T00:01:20.510560+0000","last_undegraded":"2024-04-25T00:01:20.510560+0000","last_fullsized":"2024-04-25T00:01:20.510560+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:53:14.768520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.514367+0000","last_change":"2024-04-25T00:01:05.367001+0000","last_active":"2024-04-25T00:01:20.514367+0000","last_peered":"2024-04-25T00:01:20.514367+0000","last_clean":"2024-04-25T00:01:20.514367+0000","last_became_active":"2024-04-25T00:01:05.366264+0000","last_became_peered":"2024-04-25T00:01:05.366264+0000","last_unstale":"2024-04-25T00:01:20.514367+0000","last_undegraded":"2024-04-25T00:01:20.514367+0000","last_fullsized":"2024-04-25T00:01:20.514367+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578038+0000","last_change":"2024-04-25T00:01:19.497247+0000","last_active":"2024-04-25T00:01:20.578038+0000","last_peered":"2024-04-25T00:01:20.578038+0000","last_clean":"2024-04-25T00:01:20.578038+0000","last_became_active":"2024-04-25T00:01:19.496517+0000","last_became_peered":"2024-04-25T00:01:19.496517+0000","last_unstale":"2024-04-25T00:01:20.578038+0000","last_undegraded":"2024-04-25T00:01:20.578038+0000","last_fullsized":"2024-04-25T00:01:20.578038+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:30.129056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.584723+0000","last_change":"2024-04-25T00:01:05.379958+0000","last_active":"2024-04-25T00:01:20.584723+0000","last_peered":"2024-04-25T00:01:20.584723+0000","last_clean":"2024-04-25T00:01:20.584723+0000","last_became_active":"2024-04-25T00:01:05.379090+0000","last_became_peered":"2024-04-25T00:01:05.379090+0000","last_unstale":"2024-04-25T00:01:20.584723+0000","last_undegraded":"2024-04-25T00:01:20.584723+0000","last_fullsized":"2024-04-25T00:01:20.584723+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:59:50.832958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.515390+0000","last_change":"2024-04-25T00:01:19.530176+0000","last_active":"2024-04-25T00:01:20.515390+0000","last_peered":"2024-04-25T00:01:20.515390+0000","last_clean":"2024-04-25T00:01:20.515390+0000","last_became_active":"2024-04-25T00:01:19.529410+0000","last_became_peered":"2024-04-25T00:01:19.529410+0000","last_unstale":"2024-04-25T00:01:20.515390+0000","last_undegraded":"2024-04-25T00:01:20.515390+0000","last_fullsized":"2024-04-25T00:01:20.515390+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:40:13.204880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.579746+0000","last_change":"2024-04-25T00:01:05.370826+0000","last_active":"2024-04-25T00:01:20.579746+0000","last_peered":"2024-04-25T00:01:20.579746+0000","last_clean":"2024-04-25T00:01:20.579746+0000","last_became_active":"2024-04-25T00:01:05.369848+0000","last_became_peered":"2024-04-25T00:01:05.369848+0000","last_unstale":"2024-04-25T00:01:20.579746+0000","last_undegraded":"2024-04-25T00:01:20.579746+0000","last_fullsized":"2024-04-25T00:01:20.579746+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:23.870121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.581888+0000","last_change":"2024-04-25T00:01:19.511625+0000","last_active":"2024-04-25T00:01:20.581888+0000","last_peered":"2024-04-25T00:01:20.581888+0000","last_clean":"2024-04-25T00:01:20.581888+0000","last_became_active":"2024-04-25T00:01:19.511349+0000","last_became_peered":"2024-04-25T00:01:19.511349+0000","last_unstale":"2024-04-25T00:01:20.581888+0000","last_undegraded":"2024-04-25T00:01:20.581888+0000","last_fullsized":"2024-04-25T00:01:20.581888+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:02:57.373014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.518885+0000","last_change":"2024-04-25T00:01:05.390341+0000","last_active":"2024-04-25T00:01:20.518885+0000","last_peered":"2024-04-25T00:01:20.518885+0000","last_clean":"2024-04-25T00:01:20.518885+0000","last_became_active":"2024-04-25T00:01:05.389707+0000","last_became_peered":"2024-04-25T00:01:05.389707+0000","last_unstale":"2024-04-25T00:01:20.518885+0000","last_undegraded":"2024-04-25T00:01:20.518885+0000","last_fullsized":"2024-04-25T00:01:20.518885+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:54:21.138947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578289+0000","last_change":"2024-04-25T00:01:19.513630+0000","last_active":"2024-04-25T00:01:20.578289+0000","last_peered":"2024-04-25T00:01:20.578289+0000","last_clean":"2024-04-25T00:01:20.578289+0000","last_became_active":"2024-04-25T00:01:19.512978+0000","last_became_peered":"2024-04-25T00:01:19.512978+0000","last_unstale":"2024-04-25T00:01:20.578289+0000","last_undegraded":"2024-04-25T00:01:20.578289+0000","last_fullsized":"2024-04-25T00:01:20.578289+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:57.751007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578080+0000","last_change":"2024-04-25T00:01:05.378373+0000","last_active":"2024-04-25T00:01:20.578080+0000","last_peered":"2024-04-25T00:01:20.578080+0000","last_clean":"2024-04-25T00:01:20.578080+0000","last_became_active":"2024-04-25T00:01:05.376570+0000","last_became_peered":"2024-04-25T00:01:05.376570+0000","last_unstale":"2024-04-25T00:01:20.578080+0000","last_undegraded":"2024-04-25T00:01:20.578080+0000","last_fullsized":"2024-04-25T00:01:20.578080+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:30.228060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582484+0000","last_change":"2024-04-25T00:01:19.516716+0000","last_active":"2024-04-25T00:01:20.582484+0000","last_peered":"2024-04-25T00:01:20.582484+0000","last_clean":"2024-04-25T00:01:20.582484+0000","last_became_active":"2024-04-25T00:01:19.516142+0000","last_became_peered":"2024-04-25T00:01:19.516142+0000","last_unstale":"2024-04-25T00:01:20.582484+0000","last_undegraded":"2024-04-25T00:01:20.582484+0000","last_fullsized":"2024-04-25T00:01:20.582484+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:28:18.234546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578828+0000","last_change":"2024-04-25T00:01:05.381011+0000","last_active":"2024-04-25T00:01:20.578828+0000","last_peered":"2024-04-25T00:01:20.578828+0000","last_clean":"2024-04-25T00:01:20.578828+0000","last_became_active":"2024-04-25T00:01:05.380236+0000","last_became_peered":"2024-04-25T00:01:05.380236+0000","last_unstale":"2024-04-25T00:01:20.578828+0000","last_undegraded":"2024-04-25T00:01:20.578828+0000","last_fullsized":"2024-04-25T00:01:20.578828+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:06:19.307378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.513107+0000","last_change":"2024-04-25T00:01:19.529023+0000","last_active":"2024-04-25T00:01:20.513107+0000","last_peered":"2024-04-25T00:01:20.513107+0000","last_clean":"2024-04-25T00:01:20.513107+0000","last_became_active":"2024-04-25T00:01:19.524609+0000","last_became_peered":"2024-04-25T00:01:19.524609+0000","last_unstale":"2024-04-25T00:01:20.513107+0000","last_undegraded":"2024-04-25T00:01:20.513107+0000","last_fullsized":"2024-04-25T00:01:20.513107+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:41:19.150476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582056+0000","last_change":"2024-04-25T00:01:05.385853+0000","last_active":"2024-04-25T00:01:20.582056+0000","last_peered":"2024-04-25T00:01:20.582056+0000","last_clean":"2024-04-25T00:01:20.582056+0000","last_became_active":"2024-04-25T00:01:05.384997+0000","last_became_peered":"2024-04-25T00:01:05.384997+0000","last_unstale":"2024-04-25T00:01:20.582056+0000","last_undegraded":"2024-04-25T00:01:20.582056+0000","last_fullsized":"2024-04-25T00:01:20.582056+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:36:55.436661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578299+0000","last_change":"2024-04-25T00:01:19.516867+0000","last_active":"2024-04-25T00:01:20.578299+0000","last_peered":"2024-04-25T00:01:20.578299+0000","last_clean":"2024-04-25T00:01:20.578299+0000","last_became_active":"2024-04-25T00:01:19.515764+0000","last_became_peered":"2024-04-25T00:01:19.515764+0000","last_unstale":"2024-04-25T00:01:20.578299+0000","last_undegraded":"2024-04-25T00:01:20.578299+0000","last_fullsized":"2024-04-25T00:01:20.578299+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:53:14.768520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582072+0000","last_change":"2024-04-25T00:01:05.389424+0000","last_active":"2024-04-25T00:01:20.582072+0000","last_peered":"2024-04-25T00:01:20.582072+0000","last_clean":"2024-04-25T00:01:20.582072+0000","last_became_active":"2024-04-25T00:01:05.388999+0000","last_became_peered":"2024-04-25T00:01:05.388999+0000","last_unstale":"2024-04-25T00:01:20.582072+0000","last_undegraded":"2024-04-25T00:01:20.582072+0000","last_fullsized":"2024-04-25T00:01:20.582072+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:31.456792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578192+0000","last_change":"2024-04-25T00:01:19.519365+0000","last_active":"2024-04-25T00:01:20.578192+0000","last_peered":"2024-04-25T00:01:20.578192+0000","last_clean":"2024-04-25T00:01:20.578192+0000","last_became_active":"2024-04-25T00:01:19.519085+0000","last_became_peered":"2024-04-25T00:01:19.519085+0000","last_unstale":"2024-04-25T00:01:20.578192+0000","last_undegraded":"2024-04-25T00:01:20.578192+0000","last_fullsized":"2024-04-25T00:01:20.578192+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:05:09.219823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.580630+0000","last_change":"2024-04-25T00:01:05.368974+0000","last_active":"2024-04-25T00:01:20.580630+0000","last_peered":"2024-04-25T00:01:20.580630+0000","last_clean":"2024-04-25T00:01:20.580630+0000","last_became_active":"2024-04-25T00:01:05.367893+0000","last_became_peered":"2024-04-25T00:01:05.367893+0000","last_unstale":"2024-04-25T00:01:20.580630+0000","last_undegraded":"2024-04-25T00:01:20.580630+0000","last_fullsized":"2024-04-25T00:01:20.580630+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:30:23.557381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578274+0000","last_change":"2024-04-25T00:01:19.513514+0000","last_active":"2024-04-25T00:01:20.578274+0000","last_peered":"2024-04-25T00:01:20.578274+0000","last_clean":"2024-04-25T00:01:20.578274+0000","last_became_active":"2024-04-25T00:01:19.513184+0000","last_became_peered":"2024-04-25T00:01:19.513184+0000","last_unstale":"2024-04-25T00:01:20.578274+0000","last_undegraded":"2024-04-25T00:01:20.578274+0000","last_fullsized":"2024-04-25T00:01:20.578274+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:47.859626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.582384+0000","last_change":"2024-04-25T00:01:05.362303+0000","last_active":"2024-04-25T00:01:20.582384+0000","last_peered":"2024-04-25T00:01:20.582384+0000","last_clean":"2024-04-25T00:01:20.582384+0000","last_became_active":"2024-04-25T00:01:05.361586+0000","last_became_peered":"2024-04-25T00:01:05.361586+0000","last_unstale":"2024-04-25T00:01:20.582384+0000","last_undegraded":"2024-04-25T00:01:20.582384+0000","last_fullsized":"2024-04-25T00:01:20.582384+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:30.228060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.520395+0000","last_change":"2024-04-25T00:01:19.515101+0000","last_active":"2024-04-25T00:01:20.520395+0000","last_peered":"2024-04-25T00:01:20.520395+0000","last_clean":"2024-04-25T00:01:20.520395+0000","last_became_active":"2024-04-25T00:01:19.513314+0000","last_became_peered":"2024-04-25T00:01:19.513314+0000","last_unstale":"2024-04-25T00:01:20.520395+0000","last_undegraded":"2024-04-25T00:01:20.520395+0000","last_fullsized":"2024-04-25T00:01:20.520395+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:50.351599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.518642+0000","last_change":"2024-04-25T00:01:05.390256+0000","last_active":"2024-04-25T00:01:20.518642+0000","last_peered":"2024-04-25T00:01:20.518642+0000","last_clean":"2024-04-25T00:01:20.518642+0000","last_became_active":"2024-04-25T00:01:05.389724+0000","last_became_peered":"2024-04-25T00:01:05.389724+0000","last_unstale":"2024-04-25T00:01:20.518642+0000","last_undegraded":"2024-04-25T00:01:20.518642+0000","last_fullsized":"2024-04-25T00:01:20.518642+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:09:18.315688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3f","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511570+0000","last_change":"2024-04-25T00:01:19.511493+0000","last_active":"2024-04-25T00:01:20.511570+0000","last_peered":"2024-04-25T00:01:20.511570+0000","last_clean":"2024-04-25T00:01:20.511570+0000","last_became_active":"2024-04-25T00:01:19.508049+0000","last_became_peered":"2024-04-25T00:01:19.508049+0000","last_unstale":"2024-04-25T00:01:20.511570+0000","last_undegraded":"2024-04-25T00:01:20.511570+0000","last_fullsized":"2024-04-25T00:01:20.511570+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:55:46.853565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.578830+0000","last_change":"2024-04-25T00:01:05.374766+0000","last_active":"2024-04-25T00:01:20.578830+0000","last_peered":"2024-04-25T00:01:20.578830+0000","last_clean":"2024-04-25T00:01:20.578830+0000","last_became_active":"2024-04-25T00:01:05.369739+0000","last_became_peered":"2024-04-25T00:01:05.369739+0000","last_unstale":"2024-04-25T00:01:20.578830+0000","last_undegraded":"2024-04-25T00:01:20.578830+0000","last_fullsized":"2024-04-25T00:01:20.578830+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:09:18.315688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.513219+0000","last_change":"2024-04-25T00:01:19.521456+0000","last_active":"2024-04-25T00:01:20.513219+0000","last_peered":"2024-04-25T00:01:20.513219+0000","last_clean":"2024-04-25T00:01:20.513219+0000","last_became_active":"2024-04-25T00:01:19.520787+0000","last_became_peered":"2024-04-25T00:01:19.520787+0000","last_unstale":"2024-04-25T00:01:20.513219+0000","last_undegraded":"2024-04-25T00:01:20.513219+0000","last_fullsized":"2024-04-25T00:01:20.513219+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:25:38.981364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511048+0000","last_change":"2024-04-25T00:01:05.359253+0000","last_active":"2024-04-25T00:01:20.511048+0000","last_peered":"2024-04-25T00:01:20.511048+0000","last_clean":"2024-04-25T00:01:20.511048+0000","last_became_active":"2024-04-25T00:01:05.358477+0000","last_became_peered":"2024-04-25T00:01:05.358477+0000","last_unstale":"2024-04-25T00:01:20.511048+0000","last_undegraded":"2024-04-25T00:01:20.511048+0000","last_fullsized":"2024-04-25T00:01:20.511048+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:22.739214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.579384+0000","last_change":"2024-04-25T00:01:19.509505+0000","last_active":"2024-04-25T00:01:20.579384+0000","last_peered":"2024-04-25T00:01:20.579384+0000","last_clean":"2024-04-25T00:01:20.579384+0000","last_became_active":"2024-04-25T00:01:19.507812+0000","last_became_peered":"2024-04-25T00:01:19.507812+0000","last_unstale":"2024-04-25T00:01:20.579384+0000","last_undegraded":"2024-04-25T00:01:20.579384+0000","last_fullsized":"2024-04-25T00:01:20.579384+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:23.993729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.581200+0000","last_change":"2024-04-25T00:01:05.386068+0000","last_active":"2024-04-25T00:01:20.581200+0000","last_peered":"2024-04-25T00:01:20.581200+0000","last_clean":"2024-04-25T00:01:20.581200+0000","last_became_active":"2024-04-25T00:01:05.378650+0000","last_became_peered":"2024-04-25T00:01:05.378650+0000","last_unstale":"2024-04-25T00:01:20.581200+0000","last_undegraded":"2024-04-25T00:01:20.581200+0000","last_fullsized":"2024-04-25T00:01:20.581200+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:12:52.846250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511288+0000","last_change":"2024-04-25T00:01:19.514843+0000","last_active":"2024-04-25T00:01:20.511288+0000","last_peered":"2024-04-25T00:01:20.511288+0000","last_clean":"2024-04-25T00:01:20.511288+0000","last_became_active":"2024-04-25T00:01:19.513956+0000","last_became_peered":"2024-04-25T00:01:19.513956+0000","last_unstale":"2024-04-25T00:01:20.511288+0000","last_undegraded":"2024-04-25T00:01:20.511288+0000","last_fullsized":"2024-04-25T00:01:20.511288+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:38.005233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511996+0000","last_change":"2024-04-25T00:01:05.359716+0000","last_active":"2024-04-25T00:01:20.511996+0000","last_peered":"2024-04-25T00:01:20.511996+0000","last_clean":"2024-04-25T00:01:20.511996+0000","last_became_active":"2024-04-25T00:01:05.358866+0000","last_became_peered":"2024-04-25T00:01:05.358866+0000","last_unstale":"2024-04-25T00:01:20.511996+0000","last_undegraded":"2024-04-25T00:01:20.511996+0000","last_fullsized":"2024-04-25T00:01:20.511996+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:50.489125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.519466+0000","last_change":"2024-04-25T00:01:19.499574+0000","last_active":"2024-04-25T00:01:20.519466+0000","last_peered":"2024-04-25T00:01:20.519466+0000","last_clean":"2024-04-25T00:01:20.519466+0000","last_became_active":"2024-04-25T00:01:19.498699+0000","last_became_peered":"2024-04-25T00:01:19.498699+0000","last_unstale":"2024-04-25T00:01:20.519466+0000","last_undegraded":"2024-04-25T00:01:20.519466+0000","last_fullsized":"2024-04-25T00:01:20.519466+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:47.859626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.520083+0000","last_change":"2024-04-25T00:01:05.371291+0000","last_active":"2024-04-25T00:01:20.520083+0000","last_peered":"2024-04-25T00:01:20.520083+0000","last_clean":"2024-04-25T00:01:20.520083+0000","last_became_active":"2024-04-25T00:01:05.370522+0000","last_became_peered":"2024-04-25T00:01:05.370522+0000","last_unstale":"2024-04-25T00:01:20.520083+0000","last_undegraded":"2024-04-25T00:01:20.520083+0000","last_fullsized":"2024-04-25T00:01:20.520083+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:11:27.044893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511353+0000","last_change":"2024-04-25T00:01:19.523798+0000","last_active":"2024-04-25T00:01:20.511353+0000","last_peered":"2024-04-25T00:01:20.511353+0000","last_clean":"2024-04-25T00:01:20.511353+0000","last_became_active":"2024-04-25T00:01:19.523450+0000","last_became_peered":"2024-04-25T00:01:19.523450+0000","last_unstale":"2024-04-25T00:01:20.511353+0000","last_undegraded":"2024-04-25T00:01:20.511353+0000","last_fullsized":"2024-04-25T00:01:20.511353+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:40:36.350832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.520565+0000","last_change":"2024-04-25T00:01:05.368190+0000","last_active":"2024-04-25T00:01:20.520565+0000","last_peered":"2024-04-25T00:01:20.520565+0000","last_clean":"2024-04-25T00:01:20.520565+0000","last_became_active":"2024-04-25T00:01:05.362606+0000","last_became_peered":"2024-04-25T00:01:05.362606+0000","last_unstale":"2024-04-25T00:01:20.520565+0000","last_undegraded":"2024-04-25T00:01:20.520565+0000","last_fullsized":"2024-04-25T00:01:20.520565+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:12:52.846250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.510932+0000","last_change":"2024-04-25T00:01:19.505737+0000","last_active":"2024-04-25T00:01:20.510932+0000","last_peered":"2024-04-25T00:01:20.510932+0000","last_clean":"2024-04-25T00:01:20.510932+0000","last_became_active":"2024-04-25T00:01:19.504991+0000","last_became_peered":"2024-04-25T00:01:19.504991+0000","last_unstale":"2024-04-25T00:01:20.510932+0000","last_undegraded":"2024-04-25T00:01:20.510932+0000","last_fullsized":"2024-04-25T00:01:20.510932+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:47:19.526845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.583309+0000","last_change":"2024-04-25T00:01:05.389596+0000","last_active":"2024-04-25T00:01:20.583309+0000","last_peered":"2024-04-25T00:01:20.583309+0000","last_clean":"2024-04-25T00:01:20.583309+0000","last_became_active":"2024-04-25T00:01:05.389127+0000","last_became_peered":"2024-04-25T00:01:05.389127+0000","last_unstale":"2024-04-25T00:01:20.583309+0000","last_undegraded":"2024-04-25T00:01:20.583309+0000","last_fullsized":"2024-04-25T00:01:20.583309+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:09.858617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509450,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":109716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509450,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27316,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344524,"statfs":{"total":96636764160,"available":96608792576,"internally_reserved":0,"allocated":294912,"data_stored":111182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509449,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27896,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343944,"statfs":{"total":96636764160,"available":96608198656,"internally_reserved":0,"allocated":888832,"data_stored":700612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509450,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27324,"kb_used_data":296,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344516,"statfs":{"total":96636764160,"available":96608784384,"internally_reserved":0,"allocated":303104,"data_stored":110928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509451,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27320,"kb_used_data":292,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344520,"statfs":{"total":96636764160,"available":96608788480,"internally_reserved":0,"allocated":299008,"data_stored":111216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509450,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27892,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343948,"statfs":{"total":96636764160,"available":96608202752,"internally_reserved":0,"allocated":884736,"data_stored":700078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509450,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":288,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":294912,"data_stored":109716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574859,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27336,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344504,"statfs":{"total":96636764160,"available":96608772096,"internally_reserved":0,"allocated":319488,"data_stored":113488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T00:01:29.195 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-25T00:01:29.195 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-25T00:01:29.195 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-25T00:01:29.195 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-25T00:01:29.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.338+0000 7f4994824640 1 Processor -- start 2024-04-25T00:01:29.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.338+0000 7f4994824640 1 -- start start 2024-04-25T00:01:29.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.342+0000 7f4994824640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f499014fe30 0x7f4990073350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:29.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.342+0000 7f4994824640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4990152890 0x7f4990152d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:29.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.342+0000 7f4994824640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49900a7280 con 0x7f49900729c0 2024-04-25T00:01:29.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.342+0000 7f4994824640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f49900c7450 con 0x7f4990152890 2024-04-25T00:01:29.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.342+0000 7f4994824640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f499006ff20 con 0x7f499014fe30 2024-04-25T00:01:29.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.342+0000 7f498ed9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4990152890 0x7f4990152d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:29.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.342+0000 7f498ed9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4990152890 0x7f4990152d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56444/0 (socket says 172.21.15.116:56444) 2024-04-25T00:01:29.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.342+0000 7f498ed9d640 1 -- 172.21.15.116:0/3123535045 learned_addr learned my addr 172.21.15.116:0/3123535045 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:29.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.342+0000 7f498e59c640 1 --2- 172.21.15.116:0/3123535045 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f499014fe30 0x7f4990073350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:29.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.342+0000 7f498ed9d640 1 -- 172.21.15.116:0/3123535045 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f499014fe30 msgr2=0x7f4990073350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:29.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.342+0000 7f498ed9d640 1 --2- 172.21.15.116:0/3123535045 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f499014fe30 0x7f4990073350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:29.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.342+0000 7f498ed9d640 1 -- 172.21.15.116:0/3123535045 >> v1:172.21.15.174:6789/0 conn(0x7f49900729c0 legacy=0x7f499006c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:29.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.342+0000 7f498ed9d640 1 -- 172.21.15.116:0/3123535045 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4990073070 con 0x7f4990152890 2024-04-25T00:01:29.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.342+0000 7f498ed9d640 1 --2- 172.21.15.116:0/3123535045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4990152890 0x7f4990152d40 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f4988009e50 tx=0x7f498801c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce8b2ec16d2becc9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:29.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.342+0000 7f498dd9b640 1 -- 172.21.15.116:0/3123535045 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49880a5550 con 0x7f4990152890 2024-04-25T00:01:29.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.342+0000 7f498dd9b640 1 -- 172.21.15.116:0/3123535045 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49880a56f0 con 0x7f4990152890 2024-04-25T00:01:29.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.342+0000 7f498dd9b640 1 -- 172.21.15.116:0/3123535045 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49880a5ac0 con 0x7f4990152890 2024-04-25T00:01:29.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.346+0000 7f4994824640 1 -- 172.21.15.116:0/3123535045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4990152890 msgr2=0x7f4990152d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:29.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.346+0000 7f4994824640 1 --2- 172.21.15.116:0/3123535045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4990152890 0x7f4990152d40 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f4988009e50 tx=0x7f498801c8d0 comp rx=0 tx=0).stop 2024-04-25T00:01:29.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.346+0000 7f4994824640 1 -- 172.21.15.116:0/3123535045 shutdown_connections 2024-04-25T00:01:29.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.346+0000 7f4994824640 1 --2- 172.21.15.116:0/3123535045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4990152890 0x7f4990152d40 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:29.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.346+0000 7f4994824640 1 --2- 172.21.15.116:0/3123535045 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f499014fe30 0x7f4990073350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:29.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.346+0000 7f4994824640 1 -- 172.21.15.116:0/3123535045 >> 172.21.15.116:0/3123535045 conn(0x7f49900c4270 msgr2=0x7f49900c4680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:29.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.346+0000 7f4994824640 1 -- 172.21.15.116:0/3123535045 shutdown_connections 2024-04-25T00:01:29.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.346+0000 7f4994824640 1 -- 172.21.15.116:0/3123535045 wait complete. 2024-04-25T00:01:29.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.346+0000 7f4994824640 1 Processor -- start 2024-04-25T00:01:29.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.346+0000 7f4994824640 1 -- start start 2024-04-25T00:01:29.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.346+0000 7f4994824640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f49900729c0 0x7f49901f20d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:29.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.346+0000 7f4994824640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f499014fe30 0x7f49900c0e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:29.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.346+0000 7f4994824640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4990152890 0x7f49901ee200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:29.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.346+0000 7f4994824640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4990153400 con 0x7f4990152890 2024-04-25T00:01:29.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.346+0000 7f4994824640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4990153280 con 0x7f499014fe30 2024-04-25T00:01:29.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.346+0000 7f4994824640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f4990150210 con 0x7f49900729c0 2024-04-25T00:01:29.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.346+0000 7f498ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4990152890 0x7f49901ee200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:29.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.346+0000 7f498ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4990152890 0x7f49901ee200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56446/0 (socket says 172.21.15.116:56446) 2024-04-25T00:01:29.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.346+0000 7f498ed9d640 1 -- 172.21.15.116:0/3860932170 learned_addr learned my addr 172.21.15.116:0/3860932170 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:29.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.346+0000 7f4987fff640 1 --2- 172.21.15.116:0/3860932170 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f499014fe30 0x7f49900c0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:29.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.346+0000 7f498e59c640 1 --2- 172.21.15.116:0/3860932170 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f49900729c0 0x7f49901f20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:29.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.346+0000 7f498ed9d640 1 -- 172.21.15.116:0/3860932170 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f49900729c0 msgr2=0x7f49901f20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:29.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.346+0000 7f498ed9d640 1 --2- 172.21.15.116:0/3860932170 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f49900729c0 0x7f49901f20d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:29.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.346+0000 7f498ed9d640 1 -- 172.21.15.116:0/3860932170 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f499014fe30 msgr2=0x7f49900c0e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:29.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.346+0000 7f498ed9d640 1 --2- 172.21.15.116:0/3860932170 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f499014fe30 0x7f49900c0e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:29.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.346+0000 7f498ed9d640 1 -- 172.21.15.116:0/3860932170 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49900abd20 con 0x7f4990152890 2024-04-25T00:01:29.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.346+0000 7f498ed9d640 1 --2- 172.21.15.116:0/3860932170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4990152890 0x7f49901ee200 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f498801af80 tx=0x7f4988019670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:29.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.346+0000 7f4986ffd640 1 -- 172.21.15.116:0/3860932170 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49880bc020 con 0x7f4990152890 2024-04-25T00:01:29.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.346+0000 7f4986ffd640 1 -- 172.21.15.116:0/3860932170 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49880b8670 con 0x7f4990152890 2024-04-25T00:01:29.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.346+0000 7f4994824640 1 -- 172.21.15.116:0/3860932170 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49901f1910 con 0x7f4990152890 2024-04-25T00:01:29.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.346+0000 7f4986ffd640 1 -- 172.21.15.116:0/3860932170 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4988009510 con 0x7f4990152890 2024-04-25T00:01:29.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.346+0000 7f4994824640 1 -- 172.21.15.116:0/3860932170 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49900ad8c0 con 0x7f4990152890 2024-04-25T00:01:29.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.346+0000 7f4994824640 1 -- 172.21.15.116:0/3860932170 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4990068a60 con 0x7f4990152890 2024-04-25T00:01:29.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.346+0000 7f4986ffd640 1 -- 172.21.15.116:0/3860932170 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f49880bf020 con 0x7f4990152890 2024-04-25T00:01:29.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.350+0000 7f4986ffd640 1 --2- 172.21.15.116:0/3860932170 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f49700678c0 0x7f4970069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:29.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.350+0000 7f4986ffd640 1 -- 172.21.15.116:0/3860932170 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7f4988120ea0 con 0x7f4990152890 2024-04-25T00:01:29.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.350+0000 7f498e59c640 1 --2- 172.21.15.116:0/3860932170 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f49700678c0 0x7f4970069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:29.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.350+0000 7f498e59c640 1 --2- 172.21.15.116:0/3860932170 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f49700678c0 0x7f4970069d80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f498000b7c0 tx=0x7f4980008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:29.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.354+0000 7f4986ffd640 1 -- 172.21.15.116:0/3860932170 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f49880c6030 con 0x7f4990152890 2024-04-25T00:01:29.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.534+0000 7f4994824640 1 -- 172.21.15.116:0/3860932170 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f4990153790 con 0x7f4990152890 2024-04-25T00:01:29.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.534+0000 7f4986ffd640 1 -- 172.21.15.116:0/3860932170 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f49880165e0 con 0x7f4990152890 2024-04-25T00:01:29.538 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:01:29.538 INFO:teuthology.orchestra.run.smithi116.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-25T00:01:29.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.538+0000 7f4994824640 1 -- 172.21.15.116:0/3860932170 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f49700678c0 msgr2=0x7f4970069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:29.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.538+0000 7f4994824640 1 --2- 172.21.15.116:0/3860932170 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f49700678c0 0x7f4970069d80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f498000b7c0 tx=0x7f4980008040 comp rx=0 tx=0).stop 2024-04-25T00:01:29.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.538+0000 7f4994824640 1 -- 172.21.15.116:0/3860932170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4990152890 msgr2=0x7f49901ee200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:29.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.538+0000 7f4994824640 1 --2- 172.21.15.116:0/3860932170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4990152890 0x7f49901ee200 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f498801af80 tx=0x7f4988019670 comp rx=0 tx=0).stop 2024-04-25T00:01:29.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.538+0000 7f4994824640 1 -- 172.21.15.116:0/3860932170 shutdown_connections 2024-04-25T00:01:29.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.538+0000 7f4994824640 1 --2- 172.21.15.116:0/3860932170 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f49700678c0 0x7f4970069d80 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:29.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.538+0000 7f4994824640 1 --2- 172.21.15.116:0/3860932170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4990152890 0x7f49901ee200 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:29.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.538+0000 7f4994824640 1 --2- 172.21.15.116:0/3860932170 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f499014fe30 0x7f49900c0e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:29.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.538+0000 7f4994824640 1 --2- 172.21.15.116:0/3860932170 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f49900729c0 0x7f49901f20d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:29.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.538+0000 7f4994824640 1 -- 172.21.15.116:0/3860932170 >> 172.21.15.116:0/3860932170 conn(0x7f49900c4270 msgr2=0x7f499014eec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:29.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.538+0000 7f4994824640 1 -- 172.21.15.116:0/3860932170 shutdown_connections 2024-04-25T00:01:29.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.538+0000 7f4994824640 1 -- 172.21.15.116:0/3860932170 wait complete. 2024-04-25T00:01:29.556 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-25T00:01:29.556 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T00:01:29.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.670+0000 7f149fb59640 1 Processor -- start 2024-04-25T00:01:29.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.670+0000 7f149fb59640 1 -- start start 2024-04-25T00:01:29.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.670+0000 7f149fb59640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f14980a94a0 0x7f14981563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:29.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.670+0000 7f149fb59640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1498071220 con 0x7f1498071680 2024-04-25T00:01:29.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.670+0000 7f149fb59640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1498152420 con 0x7f149814c720 2024-04-25T00:01:29.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.670+0000 7f149fb59640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f1498153460 con 0x7f14980a94a0 2024-04-25T00:01:29.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.670+0000 7f149d0f4640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f14980a94a0 0x7f14981563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:29.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.670+0000 7f149e0f6640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f1498071680 0x7f1498071ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:41746/0 (socket says 172.21.15.116:41746) 2024-04-25T00:01:29.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.670+0000 7f149e0f6640 1 -- 172.21.15.116:0/2623908180 learned_addr learned my addr 172.21.15.116:0/2623908180 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:29.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.670+0000 7f149d0f4640 1 -- 172.21.15.116:0/2623908180 >> v1:172.21.15.174:6789/0 conn(0x7f149814c720 legacy=0x7f14980c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:29.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.670+0000 7f149d0f4640 1 -- 172.21.15.116:0/2623908180 >> v1:172.21.15.116:6789/0 conn(0x7f1498071680 legacy=0x7f1498071ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:29.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.670+0000 7f149d0f4640 1 -- 172.21.15.116:0/2623908180 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14980721c0 con 0x7f14980a94a0 2024-04-25T00:01:29.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.674+0000 7f149d0f4640 1 --2- 172.21.15.116:0/2623908180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14980a94a0 0x7f14981563d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f14880097c0 tx=0x7f148801c1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52c76f717baea6d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:29.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.674+0000 7f149c8f3640 1 -- 172.21.15.116:0/2623908180 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14880b0070 con 0x7f14980a94a0 2024-04-25T00:01:29.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.674+0000 7f149c8f3640 1 -- 172.21.15.116:0/2623908180 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14880ab600 con 0x7f14980a94a0 2024-04-25T00:01:29.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.674+0000 7f149c8f3640 1 -- 172.21.15.116:0/2623908180 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14880ab9d0 con 0x7f14980a94a0 2024-04-25T00:01:29.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.674+0000 7f149fb59640 1 -- 172.21.15.116:0/2623908180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14980a94a0 msgr2=0x7f14981563d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:29.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.674+0000 7f149fb59640 1 --2- 172.21.15.116:0/2623908180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14980a94a0 0x7f14981563d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f14880097c0 tx=0x7f148801c1f0 comp rx=0 tx=0).stop 2024-04-25T00:01:29.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.674+0000 7f149fb59640 1 -- 172.21.15.116:0/2623908180 shutdown_connections 2024-04-25T00:01:29.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.674+0000 7f149fb59640 1 --2- 172.21.15.116:0/2623908180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14980a94a0 0x7f14981563d0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:29.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.674+0000 7f149fb59640 1 -- 172.21.15.116:0/2623908180 >> 172.21.15.116:0/2623908180 conn(0x7f14980c3b00 msgr2=0x7f14980c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:29.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.674+0000 7f149fb59640 1 -- 172.21.15.116:0/2623908180 shutdown_connections 2024-04-25T00:01:29.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.674+0000 7f149fb59640 1 -- 172.21.15.116:0/2623908180 wait complete. 2024-04-25T00:01:29.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.674+0000 7f149fb59640 1 Processor -- start 2024-04-25T00:01:29.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.674+0000 7f149fb59640 1 -- start start 2024-04-25T00:01:29.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.674+0000 7f149fb59640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1498071680 0x7f149813c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:29.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.674+0000 7f149fb59640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14980a94a0 0x7f14980c0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:29.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.674+0000 7f149fb59640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f149814c720 0x7f14981376b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:29.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.674+0000 7f149fb59640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1498152f40 con 0x7f14980a94a0 2024-04-25T00:01:29.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.674+0000 7f149fb59640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1498152c60 con 0x7f149814c720 2024-04-25T00:01:29.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.674+0000 7f149fb59640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f14980c3120 con 0x7f1498071680 2024-04-25T00:01:29.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.674+0000 7f149d0f4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14980a94a0 0x7f14980c0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:29.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.674+0000 7f149d0f4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14980a94a0 0x7f14980c0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56474/0 (socket says 172.21.15.116:56474) 2024-04-25T00:01:29.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.674+0000 7f149d0f4640 1 -- 172.21.15.116:0/3729453156 learned_addr learned my addr 172.21.15.116:0/3729453156 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:29.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.674+0000 7f149d0f4640 1 -- 172.21.15.116:0/3729453156 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1498071680 msgr2=0x7f149813c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:29.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.674+0000 7f149d0f4640 1 --2- 172.21.15.116:0/3729453156 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1498071680 0x7f149813c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:29.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.674+0000 7f149d0f4640 1 -- 172.21.15.116:0/3729453156 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f149814c720 msgr2=0x7f14981376b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:29.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.674+0000 7f149d0f4640 1 --2- 172.21.15.116:0/3729453156 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f149814c720 0x7f14981376b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:29.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.674+0000 7f149d0f4640 1 -- 172.21.15.116:0/3729453156 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f149813ba40 con 0x7f14980a94a0 2024-04-25T00:01:29.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.674+0000 7f149d0f4640 1 --2- 172.21.15.116:0/3729453156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14980a94a0 0x7f14980c0690 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f14880097c0 tx=0x7f1488018f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:29.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.674+0000 7f148e7fc640 1 -- 172.21.15.116:0/3729453156 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14880b8030 con 0x7f14980a94a0 2024-04-25T00:01:29.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.674+0000 7f148e7fc640 1 -- 172.21.15.116:0/3729453156 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14880b4410 con 0x7f14980a94a0 2024-04-25T00:01:29.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.674+0000 7f149fb59640 1 -- 172.21.15.116:0/3729453156 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f149813bd20 con 0x7f14980a94a0 2024-04-25T00:01:29.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.674+0000 7f148e7fc640 1 -- 172.21.15.116:0/3729453156 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14880b0040 con 0x7f14980a94a0 2024-04-25T00:01:29.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.678+0000 7f149fb59640 1 -- 172.21.15.116:0/3729453156 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14980c0bd0 con 0x7f14980a94a0 2024-04-25T00:01:29.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.678+0000 7f149fb59640 1 -- 172.21.15.116:0/3729453156 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1498068350 con 0x7f14980a94a0 2024-04-25T00:01:29.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.678+0000 7f148e7fc640 1 -- 172.21.15.116:0/3729453156 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f14880bb070 con 0x7f14980a94a0 2024-04-25T00:01:29.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.678+0000 7f148e7fc640 1 --2- 172.21.15.116:0/3729453156 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f147c0678f0 0x7f147c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:29.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.678+0000 7f148e7fc640 1 -- 172.21.15.116:0/3729453156 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7f148811fb00 con 0x7f14980a94a0 2024-04-25T00:01:29.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.678+0000 7f149d8f5640 1 --2- 172.21.15.116:0/3729453156 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f147c0678f0 0x7f147c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:29.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.678+0000 7f149d8f5640 1 --2- 172.21.15.116:0/3729453156 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f147c0678f0 0x7f147c069db0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1494009650 tx=0x7f1494006bb0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:29.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.682+0000 7f148e7fc640 1 -- 172.21.15.116:0/3729453156 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f1498152420 con 0x7f14980a94a0 2024-04-25T00:01:29.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.830+0000 7f149fb59640 1 -- 172.21.15.116:0/3729453156 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f14980c3640 con 0x7f14980a94a0 2024-04-25T00:01:29.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.834+0000 7f148e7fc640 1 -- 172.21.15.116:0/3729453156 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+20311 (secure 0 0 0) 0x7f14880f03a0 con 0x7f14980a94a0 2024-04-25T00:01:29.837 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:01:29.837 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":27,"fsid":"6d6a1aa7-0478-4c83-8057-05917e81f046","created":"2024-04-25T00:00:40.682493+0000","modified":"2024-04-25T00:01:20.501508+0000","last_up_change":"2024-04-25T00:00:51.822174+0000","last_in_change":"2024-04-25T00:00:42.980473+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T00:00:53.324376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-04-25T00:01:03.197326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-04-25T00:01:03.728618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-04-25T00:01:17.125210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-04-25T00:01:17.821873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8b6ddcb2-1ad9-4167-961c-bd6936bb9060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6809","nonce":3751765682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6811","nonce":3751765682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6815","nonce":3751765682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6813","nonce":3751765682}]},"public_addr":"172.21.15.116:6809/3751765682","cluster_addr":"172.21.15.116:6811/3751765682","heartbeat_back_addr":"172.21.15.116:6815/3751765682","heartbeat_front_addr":"172.21.15.116:6813/3751765682","state":["exists","up"]},{"osd":1,"uuid":"a6431768-994f-41da-8853-886f5687fb91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6801","nonce":377879124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6803","nonce":377879124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6807","nonce":377879124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6805","nonce":377879124}]},"public_addr":"172.21.15.116:6801/377879124","cluster_addr":"172.21.15.116:6803/377879124","heartbeat_back_addr":"172.21.15.116:6807/377879124","heartbeat_front_addr":"172.21.15.116:6805/377879124","state":["exists","up"]},{"osd":2,"uuid":"4f69bd40-5a23-45ff-8693-e1de75e1aa9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6817","nonce":1099282267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6819","nonce":1099282267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6823","nonce":1099282267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6821","nonce":1099282267}]},"public_addr":"172.21.15.116:6817/1099282267","cluster_addr":"172.21.15.116:6819/1099282267","heartbeat_back_addr":"172.21.15.116:6823/1099282267","heartbeat_front_addr":"172.21.15.116:6821/1099282267","state":["exists","up"]},{"osd":3,"uuid":"6a0d5344-8b6f-40d3-918d-0f0a4be688e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6825","nonce":1179942661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6827","nonce":1179942661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6831","nonce":1179942661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6829","nonce":1179942661}]},"public_addr":"172.21.15.116:6825/1179942661","cluster_addr":"172.21.15.116:6827/1179942661","heartbeat_back_addr":"172.21.15.116:6831/1179942661","heartbeat_front_addr":"172.21.15.116:6829/1179942661","state":["exists","up"]},{"osd":4,"uuid":"132e92e4-b825-441e-965a-ca00533fa86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6801","nonce":3132444330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6803","nonce":3132444330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6807","nonce":3132444330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6805","nonce":3132444330}]},"public_addr":"172.21.15.174:6801/3132444330","cluster_addr":"172.21.15.174:6803/3132444330","heartbeat_back_addr":"172.21.15.174:6807/3132444330","heartbeat_front_addr":"172.21.15.174:6805/3132444330","state":["exists","up"]},{"osd":5,"uuid":"a2eeb5b6-1b31-43cc-8306-a0adf3ee15bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6809","nonce":881817164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6811","nonce":881817164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6815","nonce":881817164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6813","nonce":881817164}]},"public_addr":"172.21.15.174:6809/881817164","cluster_addr":"172.21.15.174:6811/881817164","heartbeat_back_addr":"172.21.15.174:6815/881817164","heartbeat_front_addr":"172.21.15.174:6813/881817164","state":["exists","up"]},{"osd":6,"uuid":"588e873a-c841-4ba7-90f2-8956c87a9983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6825","nonce":117574251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6827","nonce":117574251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6831","nonce":117574251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6829","nonce":117574251}]},"public_addr":"172.21.15.174:6825/117574251","cluster_addr":"172.21.15.174:6827/117574251","heartbeat_back_addr":"172.21.15.174:6831/117574251","heartbeat_front_addr":"172.21.15.174:6829/117574251","state":["exists","up"]},{"osd":7,"uuid":"3b90f9b0-8a0e-4883-b088-ca1d4012870c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6817","nonce":2543899196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6819","nonce":2543899196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6823","nonce":2543899196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6821","nonce":2543899196}]},"public_addr":"172.21.15.174:6817/2543899196","cluster_addr":"172.21.15.174:6819/2543899196","heartbeat_back_addr":"172.21.15.174:6823/2543899196","heartbeat_front_addr":"172.21.15.174:6821/2543899196","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.293769+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.310034+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.404865+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.549658+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.192174+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.297266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.493649+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.452309+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:01:29.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.838+0000 7f149fb59640 1 -- 172.21.15.116:0/3729453156 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f147c0678f0 msgr2=0x7f147c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:29.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.838+0000 7f149fb59640 1 --2- 172.21.15.116:0/3729453156 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f147c0678f0 0x7f147c069db0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1494009650 tx=0x7f1494006bb0 comp rx=0 tx=0).stop 2024-04-25T00:01:29.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.838+0000 7f149fb59640 1 -- 172.21.15.116:0/3729453156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14980a94a0 msgr2=0x7f14980c0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:29.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.838+0000 7f149fb59640 1 --2- 172.21.15.116:0/3729453156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14980a94a0 0x7f14980c0690 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f14880097c0 tx=0x7f1488018f90 comp rx=0 tx=0).stop 2024-04-25T00:01:29.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.838+0000 7f149fb59640 1 -- 172.21.15.116:0/3729453156 shutdown_connections 2024-04-25T00:01:29.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.838+0000 7f149fb59640 1 --2- 172.21.15.116:0/3729453156 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f147c0678f0 0x7f147c069db0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:29.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.838+0000 7f149fb59640 1 --2- 172.21.15.116:0/3729453156 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f149814c720 0x7f14981376b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:29.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.838+0000 7f149fb59640 1 --2- 172.21.15.116:0/3729453156 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14980a94a0 0x7f14980c0690 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:29.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.838+0000 7f149fb59640 1 --2- 172.21.15.116:0/3729453156 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1498071680 0x7f149813c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:29.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.838+0000 7f149fb59640 1 -- 172.21.15.116:0/3729453156 >> 172.21.15.116:0/3729453156 conn(0x7f14980c3b00 msgr2=0x7f149806cc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:29.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.838+0000 7f149fb59640 1 -- 172.21.15.116:0/3729453156 shutdown_connections 2024-04-25T00:01:29.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.838+0000 7f149fb59640 1 -- 172.21.15.116:0/3729453156 wait complete. 2024-04-25T00:01:29.855 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T00:00:53.324376+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'a_metadata', 'create_time': '2024-04-25T00:01:03.197326+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'a'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'a_data', 'create_time': '2024-04-25T00:01:03.728618+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'a'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'b_metadata', 'create_time': '2024-04-25T00:01:17.125210+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'b'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'b_data', 'create_time': '2024-04-25T00:01:17.821873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'b'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T00:01:29.855 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T00:01:29.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.974+0000 7f4ef8989640 1 Processor -- start 2024-04-25T00:01:29.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.974+0000 7f4ef8989640 1 -- start start 2024-04-25T00:01:29.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.974+0000 7f4ef8989640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4ef40c7250 0x7f4ef4156620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:29.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.974+0000 7f4ef8989640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ef406c3f0 con 0x7f4ef40c63b0 2024-04-25T00:01:29.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.974+0000 7f4ef8989640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ef4155b40 con 0x7f4ef40c5e30 2024-04-25T00:01:29.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.974+0000 7f4ef8989640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f4ef4156190 con 0x7f4ef40c7250 2024-04-25T00:01:29.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.974+0000 7f4ef259c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4ef40c7250 0x7f4ef4156620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:29.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.974+0000 7f4ef259c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4ef40c7250 0x7f4ef4156620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56486/0 (socket says 172.21.15.116:56486) 2024-04-25T00:01:29.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.974+0000 7f4ef259c640 1 -- 172.21.15.116:0/4036302080 learned_addr learned my addr 172.21.15.116:0/4036302080 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:29.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.974+0000 7f4ef1d9b640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f4ef40c63b0 0x7f4ef40c9030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:41748/0 (socket says 172.21.15.116:41748) 2024-04-25T00:01:29.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.974+0000 7f4ef259c640 1 -- 172.21.15.116:0/4036302080 >> v1:172.21.15.174:6789/0 conn(0x7f4ef40c5e30 legacy=0x7f4ef4152d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:29.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.974+0000 7f4ef259c640 1 -- 172.21.15.116:0/4036302080 >> v1:172.21.15.116:6789/0 conn(0x7f4ef40c63b0 legacy=0x7f4ef40c9030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:29.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.974+0000 7f4ef159a640 1 -- 172.21.15.116:0/4036302080 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 396265570 0 0) 0x7f4ef406c3f0 con 0x7f4ef40c63b0 2024-04-25T00:01:29.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.974+0000 7f4ef259c640 1 -- 172.21.15.116:0/4036302080 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ef4155e20 con 0x7f4ef40c7250 2024-04-25T00:01:29.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.978+0000 7f4ef259c640 1 --2- 172.21.15.116:0/4036302080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ef40c7250 0x7f4ef4156620 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4ee4009870 tx=0x7f4ee401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6330be7d1d92b51f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:29.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.978+0000 7f4ef159a640 1 -- 172.21.15.116:0/4036302080 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ee40b0070 con 0x7f4ef40c7250 2024-04-25T00:01:29.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.978+0000 7f4ef159a640 1 -- 172.21.15.116:0/4036302080 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ee40ab600 con 0x7f4ef40c7250 2024-04-25T00:01:29.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.978+0000 7f4ef159a640 1 -- 172.21.15.116:0/4036302080 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ee40ab9d0 con 0x7f4ef40c7250 2024-04-25T00:01:29.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.978+0000 7f4ef8989640 1 -- 172.21.15.116:0/4036302080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ef40c7250 msgr2=0x7f4ef4156620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:29.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.978+0000 7f4ef8989640 1 --2- 172.21.15.116:0/4036302080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ef40c7250 0x7f4ef4156620 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4ee4009870 tx=0x7f4ee401c490 comp rx=0 tx=0).stop 2024-04-25T00:01:29.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.978+0000 7f4ef8989640 1 -- 172.21.15.116:0/4036302080 shutdown_connections 2024-04-25T00:01:29.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.978+0000 7f4ef8989640 1 --2- 172.21.15.116:0/4036302080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ef40c7250 0x7f4ef4156620 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:29.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.978+0000 7f4ef8989640 1 -- 172.21.15.116:0/4036302080 >> 172.21.15.116:0/4036302080 conn(0x7f4ef40cb300 msgr2=0x7f4ef40cb710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:29.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.978+0000 7f4ef8989640 1 -- 172.21.15.116:0/4036302080 shutdown_connections 2024-04-25T00:01:29.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.978+0000 7f4ef8989640 1 -- 172.21.15.116:0/4036302080 wait complete. 2024-04-25T00:01:29.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.978+0000 7f4ef8989640 1 Processor -- start 2024-04-25T00:01:29.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.978+0000 7f4ef8989640 1 -- start start 2024-04-25T00:01:29.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.978+0000 7f4ef8989640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ef40c5e30 0x7f4ef41ee340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:29.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.978+0000 7f4ef8989640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ef40c63b0 0x7f4ef414e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:29.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.978+0000 7f4ef8989640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ef40c7250 0x7f4ef41ef7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:29.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.978+0000 7f4ef8989640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef4163b00 con 0x7f4ef40c63b0 2024-04-25T00:01:29.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.978+0000 7f4ef8989640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef40c7840 con 0x7f4ef40c7250 2024-04-25T00:01:29.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.978+0000 7f4ef8989640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f4ef4163c80 con 0x7f4ef40c5e30 2024-04-25T00:01:29.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.978+0000 7f4ef1d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ef40c63b0 0x7f4ef414e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:29.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.978+0000 7f4ef1d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ef40c63b0 0x7f4ef414e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56496/0 (socket says 172.21.15.116:56496) 2024-04-25T00:01:29.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.978+0000 7f4ef1d9b640 1 -- 172.21.15.116:0/1315251041 learned_addr learned my addr 172.21.15.116:0/1315251041 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:29.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.978+0000 7f4ef259c640 1 --2- 172.21.15.116:0/1315251041 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ef40c5e30 0x7f4ef41ee340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:29.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.978+0000 7f4ef1d9b640 1 -- 172.21.15.116:0/1315251041 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ef40c5e30 msgr2=0x7f4ef41ee340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:29.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.978+0000 7f4ef1d9b640 1 --2- 172.21.15.116:0/1315251041 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ef40c5e30 0x7f4ef41ee340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:29.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.978+0000 7f4ef2d9d640 1 --2- 172.21.15.116:0/1315251041 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ef40c7250 0x7f4ef41ef7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:29.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.978+0000 7f4ef1d9b640 1 -- 172.21.15.116:0/1315251041 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ef40c7250 msgr2=0x7f4ef41ef7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:29.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.978+0000 7f4ef1d9b640 1 --2- 172.21.15.116:0/1315251041 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ef40c7250 0x7f4ef41ef7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:29.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.978+0000 7f4ef1d9b640 1 -- 172.21.15.116:0/1315251041 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ef414e8e0 con 0x7f4ef40c63b0 2024-04-25T00:01:29.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.978+0000 7f4ef1d9b640 1 --2- 172.21.15.116:0/1315251041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ef40c63b0 0x7f4ef414e3a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4ee800c4b0 tx=0x7f4ee800c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:29.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.978+0000 7f4ee2ffd640 1 -- 172.21.15.116:0/1315251041 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ee8018070 con 0x7f4ef40c63b0 2024-04-25T00:01:29.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.978+0000 7f4ee2ffd640 1 -- 172.21.15.116:0/1315251041 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ee80074e0 con 0x7f4ef40c63b0 2024-04-25T00:01:29.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.978+0000 7f4ee2ffd640 1 -- 172.21.15.116:0/1315251041 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ee8004a30 con 0x7f4ef40c63b0 2024-04-25T00:01:29.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.978+0000 7f4ef8989640 1 -- 172.21.15.116:0/1315251041 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ef41610e0 con 0x7f4ef40c63b0 2024-04-25T00:01:29.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.978+0000 7f4ef8989640 1 -- 172.21.15.116:0/1315251041 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ef414d5e0 con 0x7f4ef40c63b0 2024-04-25T00:01:29.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.982+0000 7f4ef8989640 1 -- 172.21.15.116:0/1315251041 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ef404f510 con 0x7f4ef40c63b0 2024-04-25T00:01:29.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.982+0000 7f4ee2ffd640 1 -- 172.21.15.116:0/1315251041 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4ee801b070 con 0x7f4ef40c63b0 2024-04-25T00:01:29.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.982+0000 7f4ee2ffd640 1 --2- 172.21.15.116:0/1315251041 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4ed40678c0 0x7f4ed4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:29.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.982+0000 7f4ee2ffd640 1 -- 172.21.15.116:0/1315251041 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7f4ee80852a0 con 0x7f4ef40c63b0 2024-04-25T00:01:29.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.982+0000 7f4ef259c640 1 --2- 172.21.15.116:0/1315251041 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4ed40678c0 0x7f4ed4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:29.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.982+0000 7f4ef259c640 1 --2- 172.21.15.116:0/1315251041 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4ed40678c0 0x7f4ed4069d80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4ee40ad000 tx=0x7f4ee40a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:29.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:29.986+0000 7f4ee2ffd640 1 -- 172.21.15.116:0/1315251041 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f4ee8055aa0 con 0x7f4ef40c63b0 2024-04-25T00:01:30.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.146+0000 7f4ef8989640 1 -- 172.21.15.116:0/1315251041 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4ef41efcf0 con 0x7f4ef40c63b0 2024-04-25T00:01:30.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.146+0000 7f4ee2ffd640 1 -- 172.21.15.116:0/1315251041 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f4ee80553a0 con 0x7f4ef40c63b0 2024-04-25T00:01:30.148 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-04-25T00:01:30.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.150+0000 7f4ef8989640 1 -- 172.21.15.116:0/1315251041 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4ed40678c0 msgr2=0x7f4ed4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:30.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.150+0000 7f4ef8989640 1 --2- 172.21.15.116:0/1315251041 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4ed40678c0 0x7f4ed4069d80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4ee40ad000 tx=0x7f4ee40a6000 comp rx=0 tx=0).stop 2024-04-25T00:01:30.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.150+0000 7f4ef8989640 1 -- 172.21.15.116:0/1315251041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ef40c63b0 msgr2=0x7f4ef414e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:30.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.150+0000 7f4ef8989640 1 --2- 172.21.15.116:0/1315251041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ef40c63b0 0x7f4ef414e3a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4ee800c4b0 tx=0x7f4ee800c970 comp rx=0 tx=0).stop 2024-04-25T00:01:30.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.150+0000 7f4ef8989640 1 -- 172.21.15.116:0/1315251041 shutdown_connections 2024-04-25T00:01:30.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.150+0000 7f4ef8989640 1 --2- 172.21.15.116:0/1315251041 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4ed40678c0 0x7f4ed4069d80 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:30.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.150+0000 7f4ef8989640 1 --2- 172.21.15.116:0/1315251041 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ef40c7250 0x7f4ef41ef7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:30.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.150+0000 7f4ef8989640 1 --2- 172.21.15.116:0/1315251041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ef40c63b0 0x7f4ef414e3a0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:30.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.150+0000 7f4ef8989640 1 --2- 172.21.15.116:0/1315251041 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ef40c5e30 0x7f4ef41ee340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:30.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.150+0000 7f4ef8989640 1 -- 172.21.15.116:0/1315251041 >> 172.21.15.116:0/1315251041 conn(0x7f4ef40cb300 msgr2=0x7f4ef406b940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:30.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.150+0000 7f4ef8989640 1 -- 172.21.15.116:0/1315251041 shutdown_connections 2024-04-25T00:01:30.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.150+0000 7f4ef8989640 1 -- 172.21.15.116:0/1315251041 wait complete. 2024-04-25T00:01:30.168 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_metadata pg_num 2024-04-25T00:01:30.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.278+0000 7f9c1e714640 1 Processor -- start 2024-04-25T00:01:30.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.278+0000 7f9c1e714640 1 -- start start 2024-04-25T00:01:30.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.278+0000 7f9c1e714640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9c180c7040 0x7f9c180c63b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:30.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.278+0000 7f9c1e714640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f9c180c5dd0 0x7f9c1806c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:30.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.278+0000 7f9c1e714640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c180c77e0 con 0x7f9c18152f50 2024-04-25T00:01:30.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.278+0000 7f9c1e714640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f9c180c7590 con 0x7f9c180c7040 2024-04-25T00:01:30.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.278+0000 7f9c1e714640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f9c180c6a20 con 0x7f9c180c5dd0 2024-04-25T00:01:30.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.278+0000 7f9c17fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f9c18152f50 0x7f9c180c9060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:41752/0 (socket says 172.21.15.116:41752) 2024-04-25T00:01:30.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.278+0000 7f9c17fff640 1 -- 172.21.15.116:0/3393565794 learned_addr learned my addr 172.21.15.116:0/3393565794 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:30.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.278+0000 7f9c177fe640 1 --2- 172.21.15.116:0/3393565794 >> v2:172.21.15.116:3300/0 conn(0x7f9c180c7040 0x7f9c180c63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:30.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.278+0000 7f9c16ffd640 1 -- 172.21.15.116:0/3393565794 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 574747881 0 0) 0x7f9c180c77e0 con 0x7f9c18152f50 2024-04-25T00:01:30.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.282+0000 7f9c1ccb1640 1 --2- 172.21.15.116:0/3393565794 >> v2:172.21.15.174:3300/0 conn(0x7f9c180c5dd0 0x7f9c1806c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:30.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.282+0000 7f9c16ffd640 1 -- 172.21.15.116:0/3393565794 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c00003530 con 0x7f9c18152f50 2024-04-25T00:01:30.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.282+0000 7f9c177fe640 1 -- 172.21.15.116:0/3393565794 >> v2:172.21.15.174:3300/0 conn(0x7f9c180c5dd0 msgr2=0x7f9c1806c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:30.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.282+0000 7f9c177fe640 1 --2- 172.21.15.116:0/3393565794 >> v2:172.21.15.174:3300/0 conn(0x7f9c180c5dd0 0x7f9c1806c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:30.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.282+0000 7f9c177fe640 1 -- 172.21.15.116:0/3393565794 >> v1:172.21.15.116:6789/0 conn(0x7f9c18152f50 legacy=0x7f9c180c9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:30.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.282+0000 7f9c177fe640 1 -- 172.21.15.116:0/3393565794 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c180c6c70 con 0x7f9c180c7040 2024-04-25T00:01:30.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.282+0000 7f9c177fe640 1 --2- 172.21.15.116:0/3393565794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c180c7040 0x7f9c180c63b0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f9c1806e1b0 tx=0x7f9c0c01c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95169e65ab9a1e4f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:30.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.282+0000 7f9c16ffd640 1 -- 172.21.15.116:0/3393565794 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c0c0b0070 con 0x7f9c180c7040 2024-04-25T00:01:30.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.282+0000 7f9c16ffd640 1 -- 172.21.15.116:0/3393565794 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c0c0ab600 con 0x7f9c180c7040 2024-04-25T00:01:30.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.282+0000 7f9c16ffd640 1 -- 172.21.15.116:0/3393565794 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c0c0ab9d0 con 0x7f9c180c7040 2024-04-25T00:01:30.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.282+0000 7f9c1e714640 1 -- 172.21.15.116:0/3393565794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c180c7040 msgr2=0x7f9c180c63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:30.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.282+0000 7f9c1e714640 1 --2- 172.21.15.116:0/3393565794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c180c7040 0x7f9c180c63b0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f9c1806e1b0 tx=0x7f9c0c01c3f0 comp rx=0 tx=0).stop 2024-04-25T00:01:30.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.282+0000 7f9c1e714640 1 -- 172.21.15.116:0/3393565794 shutdown_connections 2024-04-25T00:01:30.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.282+0000 7f9c1e714640 1 --2- 172.21.15.116:0/3393565794 >> v2:172.21.15.174:3300/0 conn(0x7f9c180c5dd0 0x7f9c1806c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:30.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.282+0000 7f9c1e714640 1 --2- 172.21.15.116:0/3393565794 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c180c7040 0x7f9c180c63b0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:30.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.282+0000 7f9c1e714640 1 -- 172.21.15.116:0/3393565794 >> 172.21.15.116:0/3393565794 conn(0x7f9c180cb160 msgr2=0x7f9c180cb570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:30.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.282+0000 7f9c1e714640 1 -- 172.21.15.116:0/3393565794 shutdown_connections 2024-04-25T00:01:30.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.282+0000 7f9c1e714640 1 -- 172.21.15.116:0/3393565794 wait complete. 2024-04-25T00:01:30.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.286+0000 7f9c1e714640 1 Processor -- start 2024-04-25T00:01:30.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.286+0000 7f9c1e714640 1 -- start start 2024-04-25T00:01:30.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.286+0000 7f9c1e714640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9c180c5dd0 0x7f9c180c3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:30.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.286+0000 7f9c1e714640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c180c7040 0x7f9c180c4290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:30.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.286+0000 7f9c1e714640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9c18152f50 0x7f9c180c0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:30.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.286+0000 7f9c1e714640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9c18162ca0 con 0x7f9c180c7040 2024-04-25T00:01:30.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.286+0000 7f9c1e714640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9c180caf70 con 0x7f9c180c5dd0 2024-04-25T00:01:30.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.286+0000 7f9c1e714640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f9c18162e20 con 0x7f9c18152f50 2024-04-25T00:01:30.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.286+0000 7f9c177fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c180c7040 0x7f9c180c4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:30.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.286+0000 7f9c177fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c180c7040 0x7f9c180c4290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56514/0 (socket says 172.21.15.116:56514) 2024-04-25T00:01:30.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.286+0000 7f9c177fe640 1 -- 172.21.15.116:0/3407926083 learned_addr learned my addr 172.21.15.116:0/3407926083 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:30.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.286+0000 7f9c17fff640 1 --2- 172.21.15.116:0/3407926083 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9c180c5dd0 0x7f9c180c3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:30.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.286+0000 7f9c1ccb1640 1 --2- 172.21.15.116:0/3407926083 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9c18152f50 0x7f9c180c0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:30.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.286+0000 7f9c177fe640 1 -- 172.21.15.116:0/3407926083 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9c18152f50 msgr2=0x7f9c180c0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:30.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.286+0000 7f9c177fe640 1 --2- 172.21.15.116:0/3407926083 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9c18152f50 0x7f9c180c0d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:30.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.286+0000 7f9c177fe640 1 -- 172.21.15.116:0/3407926083 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9c180c5dd0 msgr2=0x7f9c180c3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:30.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.286+0000 7f9c177fe640 1 --2- 172.21.15.116:0/3407926083 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9c180c5dd0 0x7f9c180c3d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:30.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.286+0000 7f9c177fe640 1 -- 172.21.15.116:0/3407926083 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c180c5040 con 0x7f9c180c7040 2024-04-25T00:01:30.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.286+0000 7f9c177fe640 1 --2- 172.21.15.116:0/3407926083 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c180c7040 0x7f9c180c4290 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c0ae040 tx=0x7f9c0c019190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:30.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.286+0000 7f9c14ff9640 1 -- 172.21.15.116:0/3407926083 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c0c0b8020 con 0x7f9c180c7040 2024-04-25T00:01:30.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.286+0000 7f9c14ff9640 1 -- 172.21.15.116:0/3407926083 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c0c0b4630 con 0x7f9c180c7040 2024-04-25T00:01:30.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.286+0000 7f9c1e714640 1 -- 172.21.15.116:0/3407926083 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c181eebb0 con 0x7f9c180c7040 2024-04-25T00:01:30.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.286+0000 7f9c14ff9640 1 -- 172.21.15.116:0/3407926083 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c0c0b0040 con 0x7f9c180c7040 2024-04-25T00:01:30.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.286+0000 7f9c1e714640 1 -- 172.21.15.116:0/3407926083 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c180c4c20 con 0x7f9c180c7040 2024-04-25T00:01:30.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.286+0000 7f9c1e714640 1 -- 172.21.15.116:0/3407926083 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c181ee9a0 con 0x7f9c180c7040 2024-04-25T00:01:30.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.286+0000 7f9c14ff9640 1 -- 172.21.15.116:0/3407926083 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f9c0c0bb020 con 0x7f9c180c7040 2024-04-25T00:01:30.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.286+0000 7f9c14ff9640 1 --2- 172.21.15.116:0/3407926083 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9c040678c0 0x7f9c04069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:30.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.286+0000 7f9c14ff9640 1 -- 172.21.15.116:0/3407926083 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7f9c0c11fd00 con 0x7f9c180c7040 2024-04-25T00:01:30.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.290+0000 7f9c17fff640 1 --2- 172.21.15.116:0/3407926083 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9c040678c0 0x7f9c04069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:30.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.290+0000 7f9c17fff640 1 --2- 172.21.15.116:0/3407926083 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9c040678c0 0x7f9c04069d80 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9c08007fd0 tx=0x7f9c08009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:30.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.294+0000 7f9c14ff9640 1 -- 172.21.15.116:0/3407926083 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f9c0c0c2030 con 0x7f9c180c7040 2024-04-25T00:01:30.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.438+0000 7f9c1e714640 1 -- 172.21.15.116:0/3407926083 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"} v 0) -- 0x7f9c1804f510 con 0x7f9c180c7040 2024-04-25T00:01:30.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.438+0000 7f9c14ff9640 1 -- 172.21.15.116:0/3407926083 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"}]=0 v27) ==== 99+0+11 (secure 0 0 0) 0x7f9c0c016070 con 0x7f9c180c7040 2024-04-25T00:01:30.441 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-04-25T00:01:30.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.442+0000 7f9c1e714640 1 -- 172.21.15.116:0/3407926083 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9c040678c0 msgr2=0x7f9c04069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:30.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.442+0000 7f9c1e714640 1 --2- 172.21.15.116:0/3407926083 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9c040678c0 0x7f9c04069d80 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9c08007fd0 tx=0x7f9c08009040 comp rx=0 tx=0).stop 2024-04-25T00:01:30.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.442+0000 7f9c1e714640 1 -- 172.21.15.116:0/3407926083 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c180c7040 msgr2=0x7f9c180c4290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:30.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.442+0000 7f9c1e714640 1 --2- 172.21.15.116:0/3407926083 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c180c7040 0x7f9c180c4290 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c0ae040 tx=0x7f9c0c019190 comp rx=0 tx=0).stop 2024-04-25T00:01:30.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.442+0000 7f9c1e714640 1 -- 172.21.15.116:0/3407926083 shutdown_connections 2024-04-25T00:01:30.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.442+0000 7f9c1e714640 1 --2- 172.21.15.116:0/3407926083 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9c040678c0 0x7f9c04069d80 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:30.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.442+0000 7f9c1e714640 1 --2- 172.21.15.116:0/3407926083 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9c18152f50 0x7f9c180c0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:30.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.442+0000 7f9c1e714640 1 --2- 172.21.15.116:0/3407926083 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c180c7040 0x7f9c180c4290 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:30.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.442+0000 7f9c1e714640 1 --2- 172.21.15.116:0/3407926083 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9c180c5dd0 0x7f9c180c3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:30.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.442+0000 7f9c1e714640 1 -- 172.21.15.116:0/3407926083 >> 172.21.15.116:0/3407926083 conn(0x7f9c180cb160 msgr2=0x7f9c180abac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:30.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.442+0000 7f9c1e714640 1 -- 172.21.15.116:0/3407926083 shutdown_connections 2024-04-25T00:01:30.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.442+0000 7f9c1e714640 1 -- 172.21.15.116:0/3407926083 wait complete. 2024-04-25T00:01:30.459 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_data pg_num 2024-04-25T00:01:30.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.578+0000 7f0b50df3640 1 Processor -- start 2024-04-25T00:01:30.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.578+0000 7f0b50df3640 1 -- start start 2024-04-25T00:01:30.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.582+0000 7f0b50df3640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f0b4c14ca60 0x7f0b4c0ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:30.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.582+0000 7f0b50df3640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0b4c0c3c00 0x7f0b4c0c2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:30.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.582+0000 7f0b50df3640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b4c0abe30 con 0x7f0b4c06b210 2024-04-25T00:01:30.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.582+0000 7f0b50df3640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f0b4c153130 con 0x7f0b4c0c3c00 2024-04-25T00:01:30.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.582+0000 7f0b50df3640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f0b4c0ac0d0 con 0x7f0b4c14ca60 2024-04-25T00:01:30.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.582+0000 7f0b49d9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0b4c0c3c00 0x7f0b4c0c2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:30.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.582+0000 7f0b4ad9d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f0b4c06b210 0x7f0b4c152510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:41754/0 (socket says 172.21.15.116:41754) 2024-04-25T00:01:30.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.582+0000 7f0b4ad9d640 1 -- 172.21.15.116:0/1369893850 learned_addr learned my addr 172.21.15.116:0/1369893850 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:30.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.582+0000 7f0b4a59c640 1 --2- 172.21.15.116:0/1369893850 >> v2:172.21.15.174:3300/0 conn(0x7f0b4c14ca60 0x7f0b4c0ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:30.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.582+0000 7f0b49d9b640 1 -- 172.21.15.116:0/1369893850 >> v2:172.21.15.174:3300/0 conn(0x7f0b4c14ca60 msgr2=0x7f0b4c0ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:30.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.582+0000 7f0b49d9b640 1 --2- 172.21.15.116:0/1369893850 >> v2:172.21.15.174:3300/0 conn(0x7f0b4c14ca60 0x7f0b4c0ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:30.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.582+0000 7f0b49d9b640 1 -- 172.21.15.116:0/1369893850 >> v1:172.21.15.116:6789/0 conn(0x7f0b4c06b210 legacy=0x7f0b4c152510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:30.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.582+0000 7f0b4959a640 1 -- 172.21.15.116:0/1369893850 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2592090701 0 0) 0x7f0b4c0abe30 con 0x7f0b4c06b210 2024-04-25T00:01:30.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.582+0000 7f0b49d9b640 1 -- 172.21.15.116:0/1369893850 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b4c0ac3b0 con 0x7f0b4c0c3c00 2024-04-25T00:01:30.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.582+0000 7f0b4a59c640 1 --2- 172.21.15.116:0/1369893850 >> v2:172.21.15.174:3300/0 conn(0x7f0b4c14ca60 0x7f0b4c0ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:30.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.582+0000 7f0b49d9b640 1 --2- 172.21.15.116:0/1369893850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b4c0c3c00 0x7f0b4c0c2e50 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c0745f0 tx=0x7f0b4401c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f5e8d7bb3bc1e25 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:30.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.582+0000 7f0b4959a640 1 -- 172.21.15.116:0/1369893850 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b440a8630 con 0x7f0b4c0c3c00 2024-04-25T00:01:30.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.582+0000 7f0b4959a640 1 -- 172.21.15.116:0/1369893850 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b440a87d0 con 0x7f0b4c0c3c00 2024-04-25T00:01:30.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.582+0000 7f0b4959a640 1 -- 172.21.15.116:0/1369893850 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b440a8ba0 con 0x7f0b4c0c3c00 2024-04-25T00:01:30.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.582+0000 7f0b50df3640 1 -- 172.21.15.116:0/1369893850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b4c0c3c00 msgr2=0x7f0b4c0c2e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:30.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.582+0000 7f0b50df3640 1 --2- 172.21.15.116:0/1369893850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b4c0c3c00 0x7f0b4c0c2e50 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c0745f0 tx=0x7f0b4401c810 comp rx=0 tx=0).stop 2024-04-25T00:01:30.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.582+0000 7f0b50df3640 1 -- 172.21.15.116:0/1369893850 shutdown_connections 2024-04-25T00:01:30.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.582+0000 7f0b50df3640 1 --2- 172.21.15.116:0/1369893850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b4c0c3c00 0x7f0b4c0c2e50 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:30.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.582+0000 7f0b50df3640 1 --2- 172.21.15.116:0/1369893850 >> v2:172.21.15.174:3300/0 conn(0x7f0b4c14ca60 0x7f0b4c0ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:30.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.582+0000 7f0b50df3640 1 -- 172.21.15.116:0/1369893850 >> 172.21.15.116:0/1369893850 conn(0x7f0b4c0c7a90 msgr2=0x7f0b4c0c7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:30.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.582+0000 7f0b50df3640 1 -- 172.21.15.116:0/1369893850 shutdown_connections 2024-04-25T00:01:30.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.582+0000 7f0b50df3640 1 -- 172.21.15.116:0/1369893850 wait complete. 2024-04-25T00:01:30.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.586+0000 7f0b50df3640 1 Processor -- start 2024-04-25T00:01:30.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.586+0000 7f0b50df3640 1 -- start start 2024-04-25T00:01:30.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.586+0000 7f0b50df3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b4c06b210 0x7f0b4c1f28a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:30.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.586+0000 7f0b50df3640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b4c0c3c00 0x7f0b4c14a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:30.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.586+0000 7f0b50df3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b4c14ca60 0x7f0b4c0c0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:30.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.586+0000 7f0b50df3640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0b4c06ea60 con 0x7f0b4c14ca60 2024-04-25T00:01:30.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.586+0000 7f0b50df3640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0b4c06e8e0 con 0x7f0b4c06b210 2024-04-25T00:01:30.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.586+0000 7f0b50df3640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f0b4c06ebe0 con 0x7f0b4c0c3c00 2024-04-25T00:01:30.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.586+0000 7f0b4ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b4c14ca60 0x7f0b4c0c0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:30.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.586+0000 7f0b4ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b4c14ca60 0x7f0b4c0c0d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56528/0 (socket says 172.21.15.116:56528) 2024-04-25T00:01:30.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.586+0000 7f0b4ad9d640 1 -- 172.21.15.116:0/2890923801 learned_addr learned my addr 172.21.15.116:0/2890923801 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:30.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.586+0000 7f0b49d9b640 1 --2- 172.21.15.116:0/2890923801 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b4c0c3c00 0x7f0b4c14a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:30.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.586+0000 7f0b4ad9d640 1 -- 172.21.15.116:0/2890923801 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b4c0c3c00 msgr2=0x7f0b4c14a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:30.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.586+0000 7f0b4ad9d640 1 --2- 172.21.15.116:0/2890923801 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b4c0c3c00 0x7f0b4c14a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:30.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.586+0000 7f0b4ad9d640 1 -- 172.21.15.116:0/2890923801 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b4c06b210 msgr2=0x7f0b4c1f28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:30.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.586+0000 7f0b4a59c640 1 --2- 172.21.15.116:0/2890923801 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b4c06b210 0x7f0b4c1f28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:30.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.586+0000 7f0b4ad9d640 1 --2- 172.21.15.116:0/2890923801 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b4c06b210 0x7f0b4c1f28a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:30.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.586+0000 7f0b4ad9d640 1 -- 172.21.15.116:0/2890923801 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b4c14ac10 con 0x7f0b4c14ca60 2024-04-25T00:01:30.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.586+0000 7f0b4ad9d640 1 --2- 172.21.15.116:0/2890923801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b4c14ca60 0x7f0b4c0c0d20 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0b3c00c480 tx=0x7f0b3c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:30.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.586+0000 7f0b42ffd640 1 -- 172.21.15.116:0/2890923801 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b3c018070 con 0x7f0b4c14ca60 2024-04-25T00:01:30.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.586+0000 7f0b42ffd640 1 -- 172.21.15.116:0/2890923801 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b3c0074e0 con 0x7f0b4c14ca60 2024-04-25T00:01:30.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.586+0000 7f0b50df3640 1 -- 172.21.15.116:0/2890923801 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b4c14a3f0 con 0x7f0b4c14ca60 2024-04-25T00:01:30.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.586+0000 7f0b42ffd640 1 -- 172.21.15.116:0/2890923801 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b3c004a30 con 0x7f0b4c14ca60 2024-04-25T00:01:30.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.586+0000 7f0b50df3640 1 -- 172.21.15.116:0/2890923801 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b4c1f20a0 con 0x7f0b4c14ca60 2024-04-25T00:01:30.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.586+0000 7f0b50df3640 1 -- 172.21.15.116:0/2890923801 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b4c04f510 con 0x7f0b4c14ca60 2024-04-25T00:01:30.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.586+0000 7f0b42ffd640 1 -- 172.21.15.116:0/2890923801 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f0b3c01b070 con 0x7f0b4c14ca60 2024-04-25T00:01:30.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.590+0000 7f0b42ffd640 1 --2- 172.21.15.116:0/2890923801 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0b2c0678c0 0x7f0b2c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:30.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.590+0000 7f0b42ffd640 1 -- 172.21.15.116:0/2890923801 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7f0b3c085280 con 0x7f0b4c14ca60 2024-04-25T00:01:30.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.590+0000 7f0b4a59c640 1 --2- 172.21.15.116:0/2890923801 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0b2c0678c0 0x7f0b2c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:30.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.590+0000 7f0b4a59c640 1 --2- 172.21.15.116:0/2890923801 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0b2c0678c0 0x7f0b2c069d80 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0b34006fd0 tx=0x7f0b34008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:30.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.594+0000 7f0b42ffd640 1 -- 172.21.15.116:0/2890923801 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f0b3c055b20 con 0x7f0b4c14ca60 2024-04-25T00:01:30.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.754+0000 7f0b50df3640 1 -- 172.21.15.116:0/2890923801 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"} v 0) -- 0x7f0b4c067540 con 0x7f0b4c14ca60 2024-04-25T00:01:30.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.754+0000 7f0b42ffd640 1 -- 172.21.15.116:0/2890923801 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"}]=0 v27) ==== 95+0+11 (secure 0 0 0) 0x7f0b3c055420 con 0x7f0b4c14ca60 2024-04-25T00:01:30.755 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-04-25T00:01:30.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.754+0000 7f0b50df3640 1 -- 172.21.15.116:0/2890923801 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0b2c0678c0 msgr2=0x7f0b2c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:30.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.754+0000 7f0b50df3640 1 --2- 172.21.15.116:0/2890923801 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0b2c0678c0 0x7f0b2c069d80 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0b34006fd0 tx=0x7f0b34008040 comp rx=0 tx=0).stop 2024-04-25T00:01:30.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.754+0000 7f0b50df3640 1 -- 172.21.15.116:0/2890923801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b4c14ca60 msgr2=0x7f0b4c0c0d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:30.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.754+0000 7f0b50df3640 1 --2- 172.21.15.116:0/2890923801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b4c14ca60 0x7f0b4c0c0d20 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0b3c00c480 tx=0x7f0b3c00c940 comp rx=0 tx=0).stop 2024-04-25T00:01:30.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.754+0000 7f0b50df3640 1 -- 172.21.15.116:0/2890923801 shutdown_connections 2024-04-25T00:01:30.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.754+0000 7f0b50df3640 1 --2- 172.21.15.116:0/2890923801 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0b2c0678c0 0x7f0b2c069d80 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:30.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.754+0000 7f0b50df3640 1 --2- 172.21.15.116:0/2890923801 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b4c14ca60 0x7f0b4c0c0d20 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:30.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.754+0000 7f0b50df3640 1 --2- 172.21.15.116:0/2890923801 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b4c0c3c00 0x7f0b4c14a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:30.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.754+0000 7f0b50df3640 1 --2- 172.21.15.116:0/2890923801 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b4c06b210 0x7f0b4c1f28a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:30.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.754+0000 7f0b50df3640 1 -- 172.21.15.116:0/2890923801 >> 172.21.15.116:0/2890923801 conn(0x7f0b4c0c7a90 msgr2=0x7f0b4c0c7e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:30.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.754+0000 7f0b50df3640 1 -- 172.21.15.116:0/2890923801 shutdown_connections 2024-04-25T00:01:30.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.754+0000 7f0b50df3640 1 -- 172.21.15.116:0/2890923801 wait complete. 2024-04-25T00:01:30.777 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_metadata pg_num 2024-04-25T00:01:30.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.898+0000 7fc8a804f640 1 Processor -- start 2024-04-25T00:01:30.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.898+0000 7fc8a804f640 1 -- start start 2024-04-25T00:01:30.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.898+0000 7fc8a804f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc8a0150d60 0x7fc8a00c2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:30.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.898+0000 7fc8a804f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc8a00c2e70 0x7fc8a0157510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:30.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.898+0000 7fc8a804f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8a00a9cb0 con 0x7fc8a00a9630 2024-04-25T00:01:30.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.898+0000 7fc8a804f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fc8a00a9f90 con 0x7fc8a00c2e70 2024-04-25T00:01:30.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.898+0000 7fc8a804f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fc8a00aa270 con 0x7fc8a0150d60 2024-04-25T00:01:30.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.902+0000 7fc8a65ec640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc8a00c2e70 0x7fc8a0157510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:30.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.902+0000 7fc8a55ea640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fc8a00a9630 0x7fc8a0156e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:41770/0 (socket says 172.21.15.116:41770) 2024-04-25T00:01:30.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.902+0000 7fc8a55ea640 1 -- 172.21.15.116:0/3796530590 learned_addr learned my addr 172.21.15.116:0/3796530590 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:30.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.902+0000 7fc8a5deb640 1 --2- 172.21.15.116:0/3796530590 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc8a0150d60 0x7fc8a00c2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:30.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.902+0000 7fc8a65ec640 1 -- 172.21.15.116:0/3796530590 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc8a0150d60 msgr2=0x7fc8a00c2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:30.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.902+0000 7fc8a65ec640 1 --2- 172.21.15.116:0/3796530590 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc8a0150d60 0x7fc8a00c2930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:30.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.902+0000 7fc8a65ec640 1 -- 172.21.15.116:0/3796530590 >> v1:172.21.15.116:6789/0 conn(0x7fc8a00a9630 legacy=0x7fc8a0156e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:30.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.902+0000 7fc8a65ec640 1 -- 172.21.15.116:0/3796530590 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8a00aa550 con 0x7fc8a00c2e70 2024-04-25T00:01:30.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.902+0000 7fc8a65ec640 1 --2- 172.21.15.116:0/3796530590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8a00c2e70 0x7fc8a0157510 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc898009d90 tx=0x7fc89801c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d5a8feb53d37c56 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:30.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.902+0000 7fc8a4de9640 1 -- 172.21.15.116:0/3796530590 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8980b2070 con 0x7fc8a00c2e70 2024-04-25T00:01:30.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.902+0000 7fc8a4de9640 1 -- 172.21.15.116:0/3796530590 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8980a5900 con 0x7fc8a00c2e70 2024-04-25T00:01:30.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.902+0000 7fc8a4de9640 1 -- 172.21.15.116:0/3796530590 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8980a5cd0 con 0x7fc8a00c2e70 2024-04-25T00:01:30.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.902+0000 7fc8a804f640 1 -- 172.21.15.116:0/3796530590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8a00c2e70 msgr2=0x7fc8a0157510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:30.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.902+0000 7fc8a804f640 1 --2- 172.21.15.116:0/3796530590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8a00c2e70 0x7fc8a0157510 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc898009d90 tx=0x7fc89801c820 comp rx=0 tx=0).stop 2024-04-25T00:01:30.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.902+0000 7fc8a804f640 1 -- 172.21.15.116:0/3796530590 shutdown_connections 2024-04-25T00:01:30.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.902+0000 7fc8a804f640 1 --2- 172.21.15.116:0/3796530590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8a00c2e70 0x7fc8a0157510 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:30.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.902+0000 7fc8a804f640 1 --2- 172.21.15.116:0/3796530590 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc8a0150d60 0x7fc8a00c2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:30.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.902+0000 7fc8a804f640 1 -- 172.21.15.116:0/3796530590 >> 172.21.15.116:0/3796530590 conn(0x7fc8a00c3a20 msgr2=0x7fc8a00c3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:30.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.902+0000 7fc8a804f640 1 -- 172.21.15.116:0/3796530590 shutdown_connections 2024-04-25T00:01:30.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.902+0000 7fc8a804f640 1 -- 172.21.15.116:0/3796530590 wait complete. 2024-04-25T00:01:30.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.906+0000 7fc8a804f640 1 Processor -- start 2024-04-25T00:01:30.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.906+0000 7fc8a804f640 1 -- start start 2024-04-25T00:01:30.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.906+0000 7fc8a804f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc8a00a9630 0x7fc8a01436d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:30.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.906+0000 7fc8a804f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8a00c2e70 0x7fc8a0143c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:30.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.906+0000 7fc8a804f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc8a0150d60 0x7fc8a0145940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:30.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.906+0000 7fc8a804f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc8a0072150 con 0x7fc8a00c2e70 2024-04-25T00:01:30.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.906+0000 7fc8a804f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc8a0071fd0 con 0x7fc8a00a9630 2024-04-25T00:01:30.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.906+0000 7fc8a804f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fc8a0160770 con 0x7fc8a0150d60 2024-04-25T00:01:30.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.906+0000 7fc8a55ea640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8a00c2e70 0x7fc8a0143c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:30.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.906+0000 7fc8a55ea640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8a00c2e70 0x7fc8a0143c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56546/0 (socket says 172.21.15.116:56546) 2024-04-25T00:01:30.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.906+0000 7fc8a55ea640 1 -- 172.21.15.116:0/3512009825 learned_addr learned my addr 172.21.15.116:0/3512009825 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:30.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.906+0000 7fc8a5deb640 1 --2- 172.21.15.116:0/3512009825 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc8a00a9630 0x7fc8a01436d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:30.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.906+0000 7fc8a65ec640 1 --2- 172.21.15.116:0/3512009825 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc8a0150d60 0x7fc8a0145940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:30.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.906+0000 7fc8a55ea640 1 -- 172.21.15.116:0/3512009825 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc8a0150d60 msgr2=0x7fc8a0145940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:30.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.906+0000 7fc8a55ea640 1 --2- 172.21.15.116:0/3512009825 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc8a0150d60 0x7fc8a0145940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:30.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.906+0000 7fc8a55ea640 1 -- 172.21.15.116:0/3512009825 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc8a00a9630 msgr2=0x7fc8a01436d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:30.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.906+0000 7fc8a55ea640 1 --2- 172.21.15.116:0/3512009825 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc8a00a9630 0x7fc8a01436d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:30.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.906+0000 7fc8a55ea640 1 -- 172.21.15.116:0/3512009825 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8a014b780 con 0x7fc8a00c2e70 2024-04-25T00:01:30.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.906+0000 7fc8a5deb640 1 --2- 172.21.15.116:0/3512009825 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc8a00a9630 0x7fc8a01436d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:30.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.906+0000 7fc8a55ea640 1 --2- 172.21.15.116:0/3512009825 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8a00c2e70 0x7fc8a0143c10 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fc89000c400 tx=0x7fc89000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:30.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.906+0000 7fc8967fc640 1 -- 172.21.15.116:0/3512009825 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc890017070 con 0x7fc8a00c2e70 2024-04-25T00:01:30.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.906+0000 7fc8967fc640 1 -- 172.21.15.116:0/3512009825 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc890005cf0 con 0x7fc8a00c2e70 2024-04-25T00:01:30.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.906+0000 7fc8a804f640 1 -- 172.21.15.116:0/3512009825 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8a0144150 con 0x7fc8a00c2e70 2024-04-25T00:01:30.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.906+0000 7fc8967fc640 1 -- 172.21.15.116:0/3512009825 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8900072d0 con 0x7fc8a00c2e70 2024-04-25T00:01:30.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.906+0000 7fc8a804f640 1 -- 172.21.15.116:0/3512009825 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8a0141770 con 0x7fc8a00c2e70 2024-04-25T00:01:30.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.906+0000 7fc8a804f640 1 -- 172.21.15.116:0/3512009825 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8a004f510 con 0x7fc8a00c2e70 2024-04-25T00:01:30.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.906+0000 7fc8967fc640 1 -- 172.21.15.116:0/3512009825 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc89001a070 con 0x7fc8a00c2e70 2024-04-25T00:01:30.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.906+0000 7fc8967fc640 1 --2- 172.21.15.116:0/3512009825 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc8840678f0 0x7fc884069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:30.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.910+0000 7fc8967fc640 1 -- 172.21.15.116:0/3512009825 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7fc8900851b0 con 0x7fc8a00c2e70 2024-04-25T00:01:30.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.910+0000 7fc8a5deb640 1 --2- 172.21.15.116:0/3512009825 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc8840678f0 0x7fc884069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:30.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.910+0000 7fc8a5deb640 1 --2- 172.21.15.116:0/3512009825 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc8840678f0 0x7fc884069db0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fc8a014e160 tx=0x7fc89c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:30.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:30.914+0000 7fc8967fc640 1 -- 172.21.15.116:0/3512009825 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fc890055400 con 0x7fc8a00c2e70 2024-04-25T00:01:31.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.070+0000 7fc8a804f640 1 -- 172.21.15.116:0/3512009825 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"} v 0) -- 0x7fc8a014ee00 con 0x7fc8a00c2e70 2024-04-25T00:01:31.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.070+0000 7fc8967fc640 1 -- 172.21.15.116:0/3512009825 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"}]=0 v27) ==== 99+0+11 (secure 0 0 0) 0x7fc890059700 con 0x7fc8a00c2e70 2024-04-25T00:01:31.073 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-04-25T00:01:31.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.074+0000 7fc8a804f640 1 -- 172.21.15.116:0/3512009825 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc8840678f0 msgr2=0x7fc884069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:31.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.074+0000 7fc8a804f640 1 --2- 172.21.15.116:0/3512009825 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc8840678f0 0x7fc884069db0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fc8a014e160 tx=0x7fc89c008040 comp rx=0 tx=0).stop 2024-04-25T00:01:31.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.074+0000 7fc8a804f640 1 -- 172.21.15.116:0/3512009825 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8a00c2e70 msgr2=0x7fc8a0143c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:31.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.074+0000 7fc8a804f640 1 --2- 172.21.15.116:0/3512009825 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8a00c2e70 0x7fc8a0143c10 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fc89000c400 tx=0x7fc89000c8c0 comp rx=0 tx=0).stop 2024-04-25T00:01:31.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.074+0000 7fc8a804f640 1 -- 172.21.15.116:0/3512009825 shutdown_connections 2024-04-25T00:01:31.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.074+0000 7fc8a804f640 1 --2- 172.21.15.116:0/3512009825 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc8840678f0 0x7fc884069db0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:31.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.074+0000 7fc8a804f640 1 --2- 172.21.15.116:0/3512009825 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc8a0150d60 0x7fc8a0145940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:31.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.074+0000 7fc8a804f640 1 --2- 172.21.15.116:0/3512009825 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8a00c2e70 0x7fc8a0143c10 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:31.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.074+0000 7fc8a804f640 1 --2- 172.21.15.116:0/3512009825 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc8a00a9630 0x7fc8a01436d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:31.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.074+0000 7fc8a804f640 1 -- 172.21.15.116:0/3512009825 >> 172.21.15.116:0/3512009825 conn(0x7fc8a00c3a20 msgr2=0x7fc8a015d6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:31.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.074+0000 7fc8a804f640 1 -- 172.21.15.116:0/3512009825 shutdown_connections 2024-04-25T00:01:31.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.074+0000 7fc8a804f640 1 -- 172.21.15.116:0/3512009825 wait complete. 2024-04-25T00:01:31.092 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_data pg_num 2024-04-25T00:01:31.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.214+0000 7fd3ab052640 1 Processor -- start 2024-04-25T00:01:31.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.218+0000 7fd3ab052640 1 -- start start 2024-04-25T00:01:31.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.218+0000 7fd3ab052640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd3a4150d60 0x7fd3a40c2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:31.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.218+0000 7fd3ab052640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fd3a40a9630 0x7fd3a4156e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:31.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.218+0000 7fd3ab052640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3a40c2e70 0x7fd3a4157340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:31.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.218+0000 7fd3ab052640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fd3a40a9cb0 con 0x7fd3a4150d60 2024-04-25T00:01:31.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.218+0000 7fd3ab052640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fd3a40a9f90 con 0x7fd3a40a9630 2024-04-25T00:01:31.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.218+0000 7fd3ab052640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fd3a40aa270 con 0x7fd3a40c2e70 2024-04-25T00:01:31.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.218+0000 7fd3a8dee640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd3a4150d60 0x7fd3a40c2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:31.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.218+0000 7fd3a8dee640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd3a4150d60 0x7fd3a40c2930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56556/0 (socket says 172.21.15.116:56556) 2024-04-25T00:01:31.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.218+0000 7fd3a8dee640 1 -- 172.21.15.116:0/2056676400 learned_addr learned my addr 172.21.15.116:0/2056676400 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:31.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.218+0000 7fd3a95ef640 1 --2- 172.21.15.116:0/2056676400 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3a40c2e70 0x7fd3a4157340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:31.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.218+0000 7fd39bfff640 1 --2- 172.21.15.116:0/2056676400 >> v2:172.21.15.174:3300/0 conn(0x7fd3a40a9630 0x7fd3a4156e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:31.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.218+0000 7fd3a8dee640 1 -- 172.21.15.116:0/2056676400 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3a40c2e70 msgr2=0x7fd3a4157340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:31.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.218+0000 7fd3a8dee640 1 --2- 172.21.15.116:0/2056676400 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3a40c2e70 0x7fd3a4157340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:31.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.218+0000 7fd3a8dee640 1 -- 172.21.15.116:0/2056676400 >> v2:172.21.15.174:3300/0 conn(0x7fd3a40a9630 msgr2=0x7fd3a4156e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:31.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.218+0000 7fd3a8dee640 1 --2- 172.21.15.116:0/2056676400 >> v2:172.21.15.174:3300/0 conn(0x7fd3a40a9630 0x7fd3a4156e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:31.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.218+0000 7fd3a8dee640 1 -- 172.21.15.116:0/2056676400 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3a40aa550 con 0x7fd3a4150d60 2024-04-25T00:01:31.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.218+0000 7fd3a8dee640 1 --2- 172.21.15.116:0/2056676400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3a4150d60 0x7fd3a40c2930 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd3a40c1350 tx=0x7fd3a001c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41661cfaa859d997 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:31.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.222+0000 7fd39b7fe640 1 -- 172.21.15.116:0/2056676400 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a00b1070 con 0x7fd3a4150d60 2024-04-25T00:01:31.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.222+0000 7fd39b7fe640 1 -- 172.21.15.116:0/2056676400 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3a40ab7a0 con 0x7fd3a4150d60 2024-04-25T00:01:31.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.222+0000 7fd39b7fe640 1 -- 172.21.15.116:0/2056676400 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a00ac630 con 0x7fd3a4150d60 2024-04-25T00:01:31.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.222+0000 7fd3ab052640 1 -- 172.21.15.116:0/2056676400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3a4150d60 msgr2=0x7fd3a40c2930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:31.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.222+0000 7fd3ab052640 1 --2- 172.21.15.116:0/2056676400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3a4150d60 0x7fd3a40c2930 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd3a40c1350 tx=0x7fd3a001c3d0 comp rx=0 tx=0).stop 2024-04-25T00:01:31.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.222+0000 7fd3ab052640 1 -- 172.21.15.116:0/2056676400 shutdown_connections 2024-04-25T00:01:31.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.222+0000 7fd3ab052640 1 --2- 172.21.15.116:0/2056676400 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3a40c2e70 0x7fd3a4157340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:31.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.222+0000 7fd3ab052640 1 --2- 172.21.15.116:0/2056676400 >> v2:172.21.15.174:3300/0 conn(0x7fd3a40a9630 0x7fd3a4156e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:31.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.222+0000 7fd3ab052640 1 --2- 172.21.15.116:0/2056676400 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3a4150d60 0x7fd3a40c2930 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:31.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.222+0000 7fd3ab052640 1 -- 172.21.15.116:0/2056676400 >> 172.21.15.116:0/2056676400 conn(0x7fd3a40c3a20 msgr2=0x7fd3a40c3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:31.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.222+0000 7fd3ab052640 1 -- 172.21.15.116:0/2056676400 shutdown_connections 2024-04-25T00:01:31.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.222+0000 7fd3ab052640 1 -- 172.21.15.116:0/2056676400 wait complete. 2024-04-25T00:01:31.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.222+0000 7fd3ab052640 1 Processor -- start 2024-04-25T00:01:31.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.222+0000 7fd3ab052640 1 -- start start 2024-04-25T00:01:31.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.222+0000 7fd3ab052640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd3a40a9630 0x7fd3a41ec5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:31.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.222+0000 7fd3ab052640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3a40c2e70 0x7fd3a414af60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:31.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.222+0000 7fd3ab052640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3a4150d60 0x7fd3a41427c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:31.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.222+0000 7fd3ab052640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd3a40715b0 con 0x7fd3a40c2e70 2024-04-25T00:01:31.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.222+0000 7fd3ab052640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd3a40721e0 con 0x7fd3a40a9630 2024-04-25T00:01:31.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.222+0000 7fd3ab052640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fd3a4071730 con 0x7fd3a4150d60 2024-04-25T00:01:31.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.222+0000 7fd39bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3a40c2e70 0x7fd3a414af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:31.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.222+0000 7fd39bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3a40c2e70 0x7fd3a414af60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56568/0 (socket says 172.21.15.116:56568) 2024-04-25T00:01:31.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.222+0000 7fd39bfff640 1 -- 172.21.15.116:0/776767152 learned_addr learned my addr 172.21.15.116:0/776767152 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:31.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.222+0000 7fd3a95ef640 1 --2- 172.21.15.116:0/776767152 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3a4150d60 0x7fd3a41427c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:31.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.222+0000 7fd3a8dee640 1 --2- 172.21.15.116:0/776767152 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd3a40a9630 0x7fd3a41ec5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:31.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.222+0000 7fd39bfff640 1 -- 172.21.15.116:0/776767152 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3a4150d60 msgr2=0x7fd3a41427c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:31.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.222+0000 7fd39bfff640 1 --2- 172.21.15.116:0/776767152 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3a4150d60 0x7fd3a41427c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:31.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.222+0000 7fd39bfff640 1 -- 172.21.15.116:0/776767152 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd3a40a9630 msgr2=0x7fd3a41ec5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:31.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.222+0000 7fd39bfff640 1 --2- 172.21.15.116:0/776767152 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd3a40a9630 0x7fd3a41ec5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:31.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.222+0000 7fd39bfff640 1 -- 172.21.15.116:0/776767152 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3a41519a0 con 0x7fd3a40c2e70 2024-04-25T00:01:31.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.222+0000 7fd39bfff640 1 --2- 172.21.15.116:0/776767152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3a40c2e70 0x7fd3a414af60 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd39400c7f0 tx=0x7fd39400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:31.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.222+0000 7fd3997fa640 1 -- 172.21.15.116:0/776767152 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd394015070 con 0x7fd3a40c2e70 2024-04-25T00:01:31.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.222+0000 7fd3997fa640 1 -- 172.21.15.116:0/776767152 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd394004220 con 0x7fd3a40c2e70 2024-04-25T00:01:31.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.222+0000 7fd3ab052640 1 -- 172.21.15.116:0/776767152 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3a414b660 con 0x7fd3a40c2e70 2024-04-25T00:01:31.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.222+0000 7fd3997fa640 1 -- 172.21.15.116:0/776767152 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd394003620 con 0x7fd3a40c2e70 2024-04-25T00:01:31.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.222+0000 7fd3ab052640 1 -- 172.21.15.116:0/776767152 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3a414bc20 con 0x7fd3a40c2e70 2024-04-25T00:01:31.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.226+0000 7fd3ab052640 1 -- 172.21.15.116:0/776767152 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3a414bf60 con 0x7fd3a40c2e70 2024-04-25T00:01:31.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.226+0000 7fd3997fa640 1 -- 172.21.15.116:0/776767152 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd3940039d0 con 0x7fd3a40c2e70 2024-04-25T00:01:31.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.226+0000 7fd3997fa640 1 --2- 172.21.15.116:0/776767152 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd3900678c0 0x7fd390069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:31.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.226+0000 7fd3997fa640 1 -- 172.21.15.116:0/776767152 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7fd394085f90 con 0x7fd3a40c2e70 2024-04-25T00:01:31.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.226+0000 7fd3a8dee640 1 --2- 172.21.15.116:0/776767152 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd3900678c0 0x7fd390069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:31.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.226+0000 7fd3a8dee640 1 --2- 172.21.15.116:0/776767152 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd3900678c0 0x7fd390069d80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd3a00ae000 tx=0x7fd3a0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:31.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.230+0000 7fd3997fa640 1 -- 172.21.15.116:0/776767152 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fd3a414bf60 con 0x7fd3a40c2e70 2024-04-25T00:01:31.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.386+0000 7fd3ab052640 1 -- 172.21.15.116:0/776767152 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"} v 0) -- 0x7fd3a404f510 con 0x7fd3a40c2e70 2024-04-25T00:01:31.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.386+0000 7fd3997fa640 1 -- 172.21.15.116:0/776767152 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"}]=0 v27) ==== 95+0+11 (secure 0 0 0) 0x7fd394056830 con 0x7fd3a40c2e70 2024-04-25T00:01:31.387 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-04-25T00:01:31.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.386+0000 7fd3ab052640 1 -- 172.21.15.116:0/776767152 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd3900678c0 msgr2=0x7fd390069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:31.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.386+0000 7fd3ab052640 1 --2- 172.21.15.116:0/776767152 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd3900678c0 0x7fd390069d80 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd3a00ae000 tx=0x7fd3a0008040 comp rx=0 tx=0).stop 2024-04-25T00:01:31.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.386+0000 7fd3ab052640 1 -- 172.21.15.116:0/776767152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3a40c2e70 msgr2=0x7fd3a414af60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:31.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.386+0000 7fd3ab052640 1 --2- 172.21.15.116:0/776767152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3a40c2e70 0x7fd3a414af60 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd39400c7f0 tx=0x7fd39400ccb0 comp rx=0 tx=0).stop 2024-04-25T00:01:31.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.386+0000 7fd3ab052640 1 -- 172.21.15.116:0/776767152 shutdown_connections 2024-04-25T00:01:31.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.386+0000 7fd3ab052640 1 --2- 172.21.15.116:0/776767152 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd3900678c0 0x7fd390069d80 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:31.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.386+0000 7fd3ab052640 1 --2- 172.21.15.116:0/776767152 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3a4150d60 0x7fd3a41427c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:31.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.386+0000 7fd3ab052640 1 --2- 172.21.15.116:0/776767152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3a40c2e70 0x7fd3a414af60 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:31.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.386+0000 7fd3ab052640 1 --2- 172.21.15.116:0/776767152 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd3a40a9630 0x7fd3a41ec5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:31.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.386+0000 7fd3ab052640 1 -- 172.21.15.116:0/776767152 >> 172.21.15.116:0/776767152 conn(0x7fd3a40c3a20 msgr2=0x7fd3a414fd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:31.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.386+0000 7fd3ab052640 1 -- 172.21.15.116:0/776767152 shutdown_connections 2024-04-25T00:01:31.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.386+0000 7fd3ab052640 1 -- 172.21.15.116:0/776767152 wait complete. 2024-04-25T00:01:31.406 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:01:31.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.522+0000 7f6cd6bda640 1 Processor -- start 2024-04-25T00:01:31.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.522+0000 7f6cd6bda640 1 -- start start 2024-04-25T00:01:31.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.526+0000 7f6cd6bda640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6cd0150d20 0x7f6cd00c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:31.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.526+0000 7f6cd6bda640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6cd00c3910 con 0x7f6cd00c2f50 2024-04-25T00:01:31.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.526+0000 7f6cd6bda640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6cd00c16e0 con 0x7f6cd00a94a0 2024-04-25T00:01:31.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.526+0000 7f6cd6bda640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6cd00c1400 con 0x7f6cd0150d20 2024-04-25T00:01:31.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.526+0000 7f6ccffff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f6cd00c2f50 0x7f6cd01538e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:41784/0 (socket says 172.21.15.116:41784) 2024-04-25T00:01:31.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.526+0000 7f6ccffff640 1 -- 172.21.15.116:0/2978399789 learned_addr learned my addr 172.21.15.116:0/2978399789 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:31.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.526+0000 7f6ccf7fe640 1 -- 172.21.15.116:0/2978399789 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1269569746 0 0) 0x7f6cd00c3910 con 0x7f6cd00c2f50 2024-04-25T00:01:31.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.526+0000 7f6ccf7fe640 1 -- 172.21.15.116:0/2978399789 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6cc0003530 con 0x7f6cd00c2f50 2024-04-25T00:01:31.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.526+0000 7f6cd4976640 1 --2- 172.21.15.116:0/2978399789 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6cd0150d20 0x7f6cd00c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:31.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.526+0000 7f6ccf7fe640 1 -- 172.21.15.116:0/2978399789 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 166821128 0 0) 0x7f6cc0003530 con 0x7f6cd00c2f50 2024-04-25T00:01:31.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.526+0000 7f6ccf7fe640 1 -- 172.21.15.116:0/2978399789 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6cd0150d20 msgr2=0x7f6cd00c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:31.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.526+0000 7f6ccf7fe640 1 --2- 172.21.15.116:0/2978399789 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6cd0150d20 0x7f6cd00c2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:31.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.526+0000 7f6ccf7fe640 1 -- 172.21.15.116:0/2978399789 >> v1:172.21.15.174:6789/0 conn(0x7f6cd00a94a0 legacy=0x7f6cd0153ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:31.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.526+0000 7f6ccf7fe640 1 -- 172.21.15.116:0/2978399789 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cd006e660 con 0x7f6cd00c2f50 2024-04-25T00:01:31.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.526+0000 7f6ccf7fe640 1 -- 172.21.15.116:0/2978399789 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f6cbc001680 con 0x7f6cd00c2f50 2024-04-25T00:01:31.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.526+0000 7f6ccf7fe640 1 -- 172.21.15.116:0/2978399789 >> v1:172.21.15.116:6789/0 conn(0x7f6cd00c2f50 legacy=0x7f6cd01538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:31.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.526+0000 7f6ccf7fe640 1 --2- 172.21.15.116:0/2978399789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cc00039f0 0x7f6cc0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:31.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.526+0000 7f6ccf7fe640 1 --2- 172.21.15.116:0/2978399789 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6cd0150d20 0x7f6cc0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:31.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.526+0000 7f6ccf7fe640 1 --2- 172.21.15.116:0/2978399789 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6cc0004d50 0x7f6cc00051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:31.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.526+0000 7f6ccf7fe640 1 -- 172.21.15.116:0/2978399789 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6cd00c1400 con 0x7f6cc00039f0 2024-04-25T00:01:31.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.526+0000 7f6ccf7fe640 1 -- 172.21.15.116:0/2978399789 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6cc000ba80 con 0x7f6cc0004d50 2024-04-25T00:01:31.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.526+0000 7f6ccf7fe640 1 -- 172.21.15.116:0/2978399789 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6cc000bc80 con 0x7f6cd0150d20 2024-04-25T00:01:31.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.526+0000 7f6cd4976640 1 --2- 172.21.15.116:0/2978399789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cc00039f0 0x7f6cc0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:31.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.526+0000 7f6ccffff640 1 --2- 172.21.15.116:0/2978399789 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6cd0150d20 0x7f6cc0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:31.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.526+0000 7f6cd4976640 1 -- 172.21.15.116:0/2978399789 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6cd0150d20 msgr2=0x7f6cc0004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:31.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.526+0000 7f6cd4976640 1 --2- 172.21.15.116:0/2978399789 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6cd0150d20 0x7f6cc0004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:31.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.526+0000 7f6cd4976640 1 -- 172.21.15.116:0/2978399789 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6cc0004d50 msgr2=0x7f6cc00051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:31.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.526+0000 7f6cd4976640 1 --2- 172.21.15.116:0/2978399789 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6cc0004d50 0x7f6cc00051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:31.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.526+0000 7f6cd4976640 1 -- 172.21.15.116:0/2978399789 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cc0003530 con 0x7f6cc00039f0 2024-04-25T00:01:31.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.526+0000 7f6ccffff640 1 --2- 172.21.15.116:0/2978399789 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6cd0150d20 0x7f6cc0004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:31.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.526+0000 7f6cd4976640 1 --2- 172.21.15.116:0/2978399789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cc00039f0 0x7f6cc0003e10 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6cc4009ab0 tx=0x7f6cc401ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd50f995c92308ce server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:31.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.526+0000 7f6ccf7fe640 1 -- 172.21.15.116:0/2978399789 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc40b2070 con 0x7f6cc00039f0 2024-04-25T00:01:31.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.526+0000 7f6ccf7fe640 1 -- 172.21.15.116:0/2978399789 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cc40a5530 con 0x7f6cc00039f0 2024-04-25T00:01:31.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.526+0000 7f6ccf7fe640 1 -- 172.21.15.116:0/2978399789 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc40096b0 con 0x7f6cc00039f0 2024-04-25T00:01:31.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.530+0000 7f6cd6bda640 1 -- 172.21.15.116:0/2978399789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cc00039f0 msgr2=0x7f6cc0003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:31.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.530+0000 7f6cd6bda640 1 --2- 172.21.15.116:0/2978399789 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cc00039f0 0x7f6cc0003e10 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6cc4009ab0 tx=0x7f6cc401ca50 comp rx=0 tx=0).stop 2024-04-25T00:01:31.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.530+0000 7f6cd5177640 1 -- 172.21.15.116:0/2978399789 reap_dead start 2024-04-25T00:01:31.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.530+0000 7f6cd6bda640 1 -- 172.21.15.116:0/2978399789 shutdown_connections 2024-04-25T00:01:31.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.530+0000 7f6cd6bda640 1 -- 172.21.15.116:0/2978399789 >> 172.21.15.116:0/2978399789 conn(0x7f6cd00c3b00 msgr2=0x7f6cd00c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:31.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.530+0000 7f6cd6bda640 1 -- 172.21.15.116:0/2978399789 shutdown_connections 2024-04-25T00:01:31.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.530+0000 7f6cd6bda640 1 -- 172.21.15.116:0/2978399789 wait complete. 2024-04-25T00:01:31.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.530+0000 7f6cd6bda640 1 Processor -- start 2024-04-25T00:01:31.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.530+0000 7f6cd6bda640 1 -- start start 2024-04-25T00:01:31.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.530+0000 7f6cd6bda640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6cc00039f0 0x7f6cd0145d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:31.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.530+0000 7f6cd6bda640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cd0151e10 0x7f6cd0141e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:31.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.530+0000 7f6cd6bda640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6cd00c6590 0x7f6cd00c4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:31.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.530+0000 7f6cd6bda640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6cd015fac0 con 0x7f6cd0151e10 2024-04-25T00:01:31.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.530+0000 7f6cd6bda640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6cd015f940 con 0x7f6cc00039f0 2024-04-25T00:01:31.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.530+0000 7f6cd6bda640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6cd015fc40 con 0x7f6cd00c6590 2024-04-25T00:01:31.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.530+0000 7f6ccffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cd0151e10 0x7f6cd0141e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:31.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.530+0000 7f6ccffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cd0151e10 0x7f6cd0141e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56578/0 (socket says 172.21.15.116:56578) 2024-04-25T00:01:31.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.530+0000 7f6ccffff640 1 -- 172.21.15.116:0/1367755441 learned_addr learned my addr 172.21.15.116:0/1367755441 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:31.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.530+0000 7f6cd4976640 1 --2- 172.21.15.116:0/1367755441 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6cc00039f0 0x7f6cd0145d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:31.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.530+0000 7f6cd5177640 1 --2- 172.21.15.116:0/1367755441 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6cd00c6590 0x7f6cd00c4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:31.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.530+0000 7f6ccffff640 1 -- 172.21.15.116:0/1367755441 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6cd00c6590 msgr2=0x7f6cd00c4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:31.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.530+0000 7f6ccffff640 1 --2- 172.21.15.116:0/1367755441 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6cd00c6590 0x7f6cd00c4fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:31.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.530+0000 7f6ccffff640 1 -- 172.21.15.116:0/1367755441 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6cc00039f0 msgr2=0x7f6cd0145d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:31.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.530+0000 7f6ccffff640 1 --2- 172.21.15.116:0/1367755441 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6cc00039f0 0x7f6cd0145d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:31.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.530+0000 7f6ccffff640 1 -- 172.21.15.116:0/1367755441 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cd0151a20 con 0x7f6cd0151e10 2024-04-25T00:01:31.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.530+0000 7f6cd5177640 1 --2- 172.21.15.116:0/1367755441 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6cd00c6590 0x7f6cd00c4fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:31.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.530+0000 7f6ccffff640 1 --2- 172.21.15.116:0/1367755441 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cd0151e10 0x7f6cd0141e00 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc00d940 tx=0x7f6cbc00de00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:31.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.530+0000 7f6ccd7fa640 1 -- 172.21.15.116:0/1367755441 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cbc019070 con 0x7f6cd0151e10 2024-04-25T00:01:31.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.530+0000 7f6ccd7fa640 1 -- 172.21.15.116:0/1367755441 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cbc0074e0 con 0x7f6cd0151e10 2024-04-25T00:01:31.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.530+0000 7f6cd6bda640 1 -- 172.21.15.116:0/1367755441 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cd014d7f0 con 0x7f6cd0151e10 2024-04-25T00:01:31.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.530+0000 7f6ccd7fa640 1 -- 172.21.15.116:0/1367755441 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cbc0049f0 con 0x7f6cd0151e10 2024-04-25T00:01:31.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.530+0000 7f6cd6bda640 1 -- 172.21.15.116:0/1367755441 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cd01ed850 con 0x7f6cd0151e10 2024-04-25T00:01:31.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.534+0000 7f6cd6bda640 1 -- 172.21.15.116:0/1367755441 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cd0068350 con 0x7f6cd0151e10 2024-04-25T00:01:31.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.534+0000 7f6ccd7fa640 1 -- 172.21.15.116:0/1367755441 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6cbc004b90 con 0x7f6cd0151e10 2024-04-25T00:01:31.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.534+0000 7f6ccd7fa640 1 --2- 172.21.15.116:0/1367755441 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6cb40678f0 0x7f6cb4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:31.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.534+0000 7f6ccd7fa640 1 -- 172.21.15.116:0/1367755441 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7f6cbc085a20 con 0x7f6cd0151e10 2024-04-25T00:01:31.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.534+0000 7f6cd4976640 1 --2- 172.21.15.116:0/1367755441 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6cb40678f0 0x7f6cb4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:31.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.534+0000 7f6cd4976640 1 --2- 172.21.15.116:0/1367755441 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6cb40678f0 0x7f6cb4069db0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6cc40af000 tx=0x7f6cc40046d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:31.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.538+0000 7f6ccd7fa640 1 -- 172.21.15.116:0/1367755441 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f6cbc0562c0 con 0x7f6cd0151e10 2024-04-25T00:01:31.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.718+0000 7f6cd6bda640 1 -- 172.21.15.116:0/1367755441 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6cd0151bf0 con 0x7f6cd0151e10 2024-04-25T00:01:31.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.722+0000 7f6ccd7fa640 1 -- 172.21.15.116:0/1367755441 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6492 (secure 0 0 0) 0x7f6cbc055bc0 con 0x7f6cd0151e10 2024-04-25T00:01:31.723 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:01:31.723 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6834/1459177921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1459177921},{"type":"v1","addr":"172.21.15.116:6834","nonce":1459177921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:01:08.748957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6837/1762356055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1762356055},{"type":"v1","addr":"172.21.15.174:6837","nonce":1762356055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270]},"id":1},{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:01:22.888641+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6835/174781672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":174781672},{"type":"v1","addr":"172.21.15.116:6835","nonce":174781672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4245,"qdb_cluster":[4245]},"id":2}]} 2024-04-25T00:01:31.724 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 14 2024-04-25T00:01:31.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.722+0000 7f6cd6bda640 1 -- 172.21.15.116:0/1367755441 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6cb40678f0 msgr2=0x7f6cb4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:31.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.722+0000 7f6cd6bda640 1 --2- 172.21.15.116:0/1367755441 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6cb40678f0 0x7f6cb4069db0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6cc40af000 tx=0x7f6cc40046d0 comp rx=0 tx=0).stop 2024-04-25T00:01:31.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.722+0000 7f6cd6bda640 1 -- 172.21.15.116:0/1367755441 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cd0151e10 msgr2=0x7f6cd0141e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:31.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.722+0000 7f6cd6bda640 1 --2- 172.21.15.116:0/1367755441 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cd0151e10 0x7f6cd0141e00 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc00d940 tx=0x7f6cbc00de00 comp rx=0 tx=0).stop 2024-04-25T00:01:31.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.722+0000 7f6cd6bda640 1 -- 172.21.15.116:0/1367755441 shutdown_connections 2024-04-25T00:01:31.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.722+0000 7f6cd6bda640 1 --2- 172.21.15.116:0/1367755441 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6cb40678f0 0x7f6cb4069db0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:31.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.722+0000 7f6cd6bda640 1 --2- 172.21.15.116:0/1367755441 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6cd00c6590 0x7f6cd00c4fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:31.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.722+0000 7f6cd6bda640 1 --2- 172.21.15.116:0/1367755441 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cd0151e10 0x7f6cd0141e00 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:31.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.722+0000 7f6cd6bda640 1 --2- 172.21.15.116:0/1367755441 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6cc00039f0 0x7f6cd0145d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:31.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.722+0000 7f6cd6bda640 1 -- 172.21.15.116:0/1367755441 >> 172.21.15.116:0/1367755441 conn(0x7f6cd00c3b00 msgr2=0x7f6cd014a4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:31.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.722+0000 7f6cd6bda640 1 -- 172.21.15.116:0/1367755441 shutdown_connections 2024-04-25T00:01:31.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.722+0000 7f6cd6bda640 1 -- 172.21.15.116:0/1367755441 wait complete. 2024-04-25T00:01:31.742 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T00:01:31.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.854+0000 7fb12726c640 1 Processor -- start 2024-04-25T00:01:31.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.854+0000 7fb12726c640 1 -- start start 2024-04-25T00:01:31.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.854+0000 7fb12726c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb12014d810 0x7fb1200c8340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:31.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.854+0000 7fb12726c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb1200ae260 0x7fb120071090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:31.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.854+0000 7fb12726c640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb120153650 con 0x7fb1200715d0 2024-04-25T00:01:31.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.854+0000 7fb12726c640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fb120153910 con 0x7fb1200ae260 2024-04-25T00:01:31.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.854+0000 7fb12726c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fb120153bf0 con 0x7fb12014d810 2024-04-25T00:01:31.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.854+0000 7fb124807640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb1200ae260 0x7fb120071090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:31.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.854+0000 7fb125809640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fb1200715d0 0x7fb120156c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:41794/0 (socket says 172.21.15.116:41794) 2024-04-25T00:01:31.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.854+0000 7fb125809640 1 -- 172.21.15.116:0/2942990995 learned_addr learned my addr 172.21.15.116:0/2942990995 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:31.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.858+0000 7fb125008640 1 --2- 172.21.15.116:0/2942990995 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb12014d810 0x7fb1200c8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:31.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.858+0000 7fb124807640 1 -- 172.21.15.116:0/2942990995 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb12014d810 msgr2=0x7fb1200c8340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:31.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.858+0000 7fb124807640 1 --2- 172.21.15.116:0/2942990995 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb12014d810 0x7fb1200c8340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:31.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.858+0000 7fb124807640 1 -- 172.21.15.116:0/2942990995 >> v1:172.21.15.116:6789/0 conn(0x7fb1200715d0 legacy=0x7fb120156c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:31.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.858+0000 7fb124807640 1 -- 172.21.15.116:0/2942990995 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb120153ed0 con 0x7fb1200ae260 2024-04-25T00:01:31.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.858+0000 7fb117fff640 1 -- 172.21.15.116:0/2942990995 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3863541578 0 0) 0x7fb120153650 con 0x7fb1200715d0 2024-04-25T00:01:31.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.858+0000 7fb125008640 1 --2- 172.21.15.116:0/2942990995 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb12014d810 0x7fb1200c8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:31.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.858+0000 7fb124807640 1 --2- 172.21.15.116:0/2942990995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1200ae260 0x7fb120071090 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb1100097c0 tx=0x7fb11001c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a28b5fd1ce549afa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:31.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.858+0000 7fb117fff640 1 -- 172.21.15.116:0/2942990995 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1100b0070 con 0x7fb1200ae260 2024-04-25T00:01:31.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.858+0000 7fb117fff640 1 -- 172.21.15.116:0/2942990995 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1100ab600 con 0x7fb1200ae260 2024-04-25T00:01:31.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.858+0000 7fb117fff640 1 -- 172.21.15.116:0/2942990995 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1100ab9d0 con 0x7fb1200ae260 2024-04-25T00:01:31.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.858+0000 7fb12726c640 1 -- 172.21.15.116:0/2942990995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1200ae260 msgr2=0x7fb120071090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:31.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.858+0000 7fb12726c640 1 --2- 172.21.15.116:0/2942990995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1200ae260 0x7fb120071090 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb1100097c0 tx=0x7fb11001c380 comp rx=0 tx=0).stop 2024-04-25T00:01:31.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.858+0000 7fb12726c640 1 -- 172.21.15.116:0/2942990995 shutdown_connections 2024-04-25T00:01:31.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.858+0000 7fb12726c640 1 --2- 172.21.15.116:0/2942990995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1200ae260 0x7fb120071090 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:31.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.858+0000 7fb12726c640 1 --2- 172.21.15.116:0/2942990995 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb12014d810 0x7fb1200c8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:31.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.858+0000 7fb12726c640 1 -- 172.21.15.116:0/2942990995 >> 172.21.15.116:0/2942990995 conn(0x7fb120074a80 msgr2=0x7fb1200c8f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:31.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.858+0000 7fb12726c640 1 -- 172.21.15.116:0/2942990995 shutdown_connections 2024-04-25T00:01:31.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.858+0000 7fb12726c640 1 -- 172.21.15.116:0/2942990995 wait complete. 2024-04-25T00:01:31.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.862+0000 7fb12726c640 1 Processor -- start 2024-04-25T00:01:31.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.862+0000 7fb12726c640 1 -- start start 2024-04-25T00:01:31.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.862+0000 7fb12726c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb1200715d0 0x7fb120155350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:31.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.862+0000 7fb12726c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb1200ae260 0x7fb120148940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:31.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.862+0000 7fb12726c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb12014d810 0x7fb120070540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:31.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.862+0000 7fb12726c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb120160a00 con 0x7fb12014d810 2024-04-25T00:01:31.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.862+0000 7fb12726c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb120157970 con 0x7fb1200715d0 2024-04-25T00:01:31.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.862+0000 7fb12726c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fb120160b80 con 0x7fb1200ae260 2024-04-25T00:01:31.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.862+0000 7fb125809640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb12014d810 0x7fb120070540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:31.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.862+0000 7fb125809640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb12014d810 0x7fb120070540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56604/0 (socket says 172.21.15.116:56604) 2024-04-25T00:01:31.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.862+0000 7fb125809640 1 -- 172.21.15.116:0/3709220634 learned_addr learned my addr 172.21.15.116:0/3709220634 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:31.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.862+0000 7fb125008640 1 --2- 172.21.15.116:0/3709220634 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb1200715d0 0x7fb120155350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:31.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.862+0000 7fb125809640 1 -- 172.21.15.116:0/3709220634 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb1200ae260 msgr2=0x7fb120148940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:31.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.862+0000 7fb125809640 1 --2- 172.21.15.116:0/3709220634 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb1200ae260 0x7fb120148940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:31.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.862+0000 7fb125809640 1 -- 172.21.15.116:0/3709220634 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb1200715d0 msgr2=0x7fb120155350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:31.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.862+0000 7fb125809640 1 --2- 172.21.15.116:0/3709220634 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb1200715d0 0x7fb120155350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:31.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.862+0000 7fb125809640 1 -- 172.21.15.116:0/3709220634 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb12014a700 con 0x7fb12014d810 2024-04-25T00:01:31.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.862+0000 7fb125809640 1 --2- 172.21.15.116:0/3709220634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb12014d810 0x7fb120070540 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb11800c700 tx=0x7fb11800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:31.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.862+0000 7fb115ffb640 1 -- 172.21.15.116:0/3709220634 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb118018070 con 0x7fb12014d810 2024-04-25T00:01:31.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.862+0000 7fb115ffb640 1 -- 172.21.15.116:0/3709220634 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb11800ce10 con 0x7fb12014d810 2024-04-25T00:01:31.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.862+0000 7fb12726c640 1 -- 172.21.15.116:0/3709220634 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb12014a9e0 con 0x7fb12014d810 2024-04-25T00:01:31.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.862+0000 7fb115ffb640 1 -- 172.21.15.116:0/3709220634 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb11800a860 con 0x7fb12014d810 2024-04-25T00:01:31.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.862+0000 7fb12726c640 1 -- 172.21.15.116:0/3709220634 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb120070a80 con 0x7fb12014d810 2024-04-25T00:01:31.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.862+0000 7fb12726c640 1 -- 172.21.15.116:0/3709220634 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb120155890 con 0x7fb12014d810 2024-04-25T00:01:31.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.862+0000 7fb115ffb640 1 -- 172.21.15.116:0/3709220634 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb11800aa00 con 0x7fb12014d810 2024-04-25T00:01:31.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.862+0000 7fb115ffb640 1 --2- 172.21.15.116:0/3709220634 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb1040678c0 0x7fb104069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:31.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.866+0000 7fb115ffb640 1 -- 172.21.15.116:0/3709220634 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7fb118085d10 con 0x7fb12014d810 2024-04-25T00:01:31.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.866+0000 7fb125008640 1 --2- 172.21.15.116:0/3709220634 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb1040678c0 0x7fb104069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:31.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.866+0000 7fb125008640 1 --2- 172.21.15.116:0/3709220634 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb1040678c0 0x7fb104069d80 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb11c006fd0 tx=0x7fb11c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:31.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:31.866+0000 7fb115ffb640 1 -- 172.21.15.116:0/3709220634 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fb118021070 con 0x7fb12014d810 2024-04-25T00:01:32.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.022+0000 7fb12726c640 1 -- 172.21.15.116:0/3709220634 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb120068350 con 0x7fb12014d810 2024-04-25T00:01:32.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.026+0000 7fb115ffb640 1 -- 172.21.15.116:0/3709220634 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+20311 (secure 0 0 0) 0x7fb1180565b0 con 0x7fb12014d810 2024-04-25T00:01:32.028 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:01:32.029 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":27,"fsid":"6d6a1aa7-0478-4c83-8057-05917e81f046","created":"2024-04-25T00:00:40.682493+0000","modified":"2024-04-25T00:01:20.501508+0000","last_up_change":"2024-04-25T00:00:51.822174+0000","last_in_change":"2024-04-25T00:00:42.980473+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T00:00:53.324376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-04-25T00:01:03.197326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-04-25T00:01:03.728618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-04-25T00:01:17.125210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-04-25T00:01:17.821873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8b6ddcb2-1ad9-4167-961c-bd6936bb9060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6809","nonce":3751765682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6811","nonce":3751765682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6815","nonce":3751765682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6813","nonce":3751765682}]},"public_addr":"172.21.15.116:6809/3751765682","cluster_addr":"172.21.15.116:6811/3751765682","heartbeat_back_addr":"172.21.15.116:6815/3751765682","heartbeat_front_addr":"172.21.15.116:6813/3751765682","state":["exists","up"]},{"osd":1,"uuid":"a6431768-994f-41da-8853-886f5687fb91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6801","nonce":377879124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6803","nonce":377879124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6807","nonce":377879124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6805","nonce":377879124}]},"public_addr":"172.21.15.116:6801/377879124","cluster_addr":"172.21.15.116:6803/377879124","heartbeat_back_addr":"172.21.15.116:6807/377879124","heartbeat_front_addr":"172.21.15.116:6805/377879124","state":["exists","up"]},{"osd":2,"uuid":"4f69bd40-5a23-45ff-8693-e1de75e1aa9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6817","nonce":1099282267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6819","nonce":1099282267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6823","nonce":1099282267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6821","nonce":1099282267}]},"public_addr":"172.21.15.116:6817/1099282267","cluster_addr":"172.21.15.116:6819/1099282267","heartbeat_back_addr":"172.21.15.116:6823/1099282267","heartbeat_front_addr":"172.21.15.116:6821/1099282267","state":["exists","up"]},{"osd":3,"uuid":"6a0d5344-8b6f-40d3-918d-0f0a4be688e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6825","nonce":1179942661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6827","nonce":1179942661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6831","nonce":1179942661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6829","nonce":1179942661}]},"public_addr":"172.21.15.116:6825/1179942661","cluster_addr":"172.21.15.116:6827/1179942661","heartbeat_back_addr":"172.21.15.116:6831/1179942661","heartbeat_front_addr":"172.21.15.116:6829/1179942661","state":["exists","up"]},{"osd":4,"uuid":"132e92e4-b825-441e-965a-ca00533fa86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6801","nonce":3132444330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6803","nonce":3132444330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6807","nonce":3132444330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6805","nonce":3132444330}]},"public_addr":"172.21.15.174:6801/3132444330","cluster_addr":"172.21.15.174:6803/3132444330","heartbeat_back_addr":"172.21.15.174:6807/3132444330","heartbeat_front_addr":"172.21.15.174:6805/3132444330","state":["exists","up"]},{"osd":5,"uuid":"a2eeb5b6-1b31-43cc-8306-a0adf3ee15bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6809","nonce":881817164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6811","nonce":881817164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6815","nonce":881817164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6813","nonce":881817164}]},"public_addr":"172.21.15.174:6809/881817164","cluster_addr":"172.21.15.174:6811/881817164","heartbeat_back_addr":"172.21.15.174:6815/881817164","heartbeat_front_addr":"172.21.15.174:6813/881817164","state":["exists","up"]},{"osd":6,"uuid":"588e873a-c841-4ba7-90f2-8956c87a9983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6825","nonce":117574251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6827","nonce":117574251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6831","nonce":117574251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6829","nonce":117574251}]},"public_addr":"172.21.15.174:6825/117574251","cluster_addr":"172.21.15.174:6827/117574251","heartbeat_back_addr":"172.21.15.174:6831/117574251","heartbeat_front_addr":"172.21.15.174:6829/117574251","state":["exists","up"]},{"osd":7,"uuid":"3b90f9b0-8a0e-4883-b088-ca1d4012870c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6817","nonce":2543899196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6819","nonce":2543899196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6823","nonce":2543899196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6821","nonce":2543899196}]},"public_addr":"172.21.15.174:6817/2543899196","cluster_addr":"172.21.15.174:6819/2543899196","heartbeat_back_addr":"172.21.15.174:6823/2543899196","heartbeat_front_addr":"172.21.15.174:6821/2543899196","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.293769+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.310034+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.404865+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.549658+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.192174+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.297266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.493649+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.452309+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:01:32.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.026+0000 7fb12726c640 1 -- 172.21.15.116:0/3709220634 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb1040678c0 msgr2=0x7fb104069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:32.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.026+0000 7fb12726c640 1 --2- 172.21.15.116:0/3709220634 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb1040678c0 0x7fb104069d80 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb11c006fd0 tx=0x7fb11c008040 comp rx=0 tx=0).stop 2024-04-25T00:01:32.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.026+0000 7fb12726c640 1 -- 172.21.15.116:0/3709220634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb12014d810 msgr2=0x7fb120070540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:32.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.026+0000 7fb12726c640 1 --2- 172.21.15.116:0/3709220634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb12014d810 0x7fb120070540 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb11800c700 tx=0x7fb11800cbc0 comp rx=0 tx=0).stop 2024-04-25T00:01:32.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.030+0000 7fb12726c640 1 -- 172.21.15.116:0/3709220634 shutdown_connections 2024-04-25T00:01:32.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.030+0000 7fb12726c640 1 --2- 172.21.15.116:0/3709220634 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb1040678c0 0x7fb104069d80 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:32.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.030+0000 7fb12726c640 1 --2- 172.21.15.116:0/3709220634 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb12014d810 0x7fb120070540 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:32.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.030+0000 7fb12726c640 1 --2- 172.21.15.116:0/3709220634 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb1200ae260 0x7fb120148940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:32.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.030+0000 7fb12726c640 1 --2- 172.21.15.116:0/3709220634 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb1200715d0 0x7fb120155350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:32.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.030+0000 7fb12726c640 1 -- 172.21.15.116:0/3709220634 >> 172.21.15.116:0/3709220634 conn(0x7fb120074a80 msgr2=0x7fb120070eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:32.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.030+0000 7fb12726c640 1 -- 172.21.15.116:0/3709220634 shutdown_connections 2024-04-25T00:01:32.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.030+0000 7fb12726c640 1 -- 172.21.15.116:0/3709220634 wait complete. 2024-04-25T00:01:32.046 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T00:00:53.324376+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'a_metadata', 'create_time': '2024-04-25T00:01:03.197326+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'a'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'a_data', 'create_time': '2024-04-25T00:01:03.728618+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'a'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'b_metadata', 'create_time': '2024-04-25T00:01:17.125210+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'b'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'b_data', 'create_time': '2024-04-25T00:01:17.821873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'b'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T00:01:32.047 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T00:01:32.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.166+0000 7fe6a3363640 1 Processor -- start 2024-04-25T00:01:32.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.166+0000 7fe6a3363640 1 -- start start 2024-04-25T00:01:32.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.166+0000 7fe6a3363640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fe69c0c8740 0x7fe69c157a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:32.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.166+0000 7fe6a3363640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe69c153cb0 con 0x7fe69c0c7750 2024-04-25T00:01:32.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.166+0000 7fe6a3363640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe69c153f90 con 0x7fe69c151260 2024-04-25T00:01:32.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.166+0000 7fe6a3363640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fe69c1548c0 con 0x7fe69c0c8740 2024-04-25T00:01:32.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.166+0000 7fe6a1900640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fe69c0c7750 0x7fe69c0c4920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:41810/0 (socket says 172.21.15.116:41810) 2024-04-25T00:01:32.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.166+0000 7fe6a1900640 1 -- 172.21.15.116:0/3993481118 learned_addr learned my addr 172.21.15.116:0/3993481118 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:32.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.166+0000 7fe6a08fe640 1 --2- 172.21.15.116:0/3993481118 >> v2:172.21.15.174:3300/0 conn(0x7fe69c0c8740 0x7fe69c157a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:32.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.166+0000 7fe693fff640 1 -- 172.21.15.116:0/3993481118 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3777038303 0 0) 0x7fe69c153cb0 con 0x7fe69c0c7750 2024-04-25T00:01:32.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.166+0000 7fe693fff640 1 -- 172.21.15.116:0/3993481118 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe688003530 con 0x7fe69c0c7750 2024-04-25T00:01:32.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.166+0000 7fe693fff640 1 -- 172.21.15.116:0/3993481118 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3594198825 0 0) 0x7fe688003530 con 0x7fe69c0c7750 2024-04-25T00:01:32.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.166+0000 7fe693fff640 1 -- 172.21.15.116:0/3993481118 >> v2:172.21.15.174:3300/0 conn(0x7fe69c0c8740 msgr2=0x7fe69c157a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:32.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.166+0000 7fe693fff640 1 --2- 172.21.15.116:0/3993481118 >> v2:172.21.15.174:3300/0 conn(0x7fe69c0c8740 0x7fe69c157a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:32.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.166+0000 7fe693fff640 1 -- 172.21.15.116:0/3993481118 >> v1:172.21.15.174:6789/0 conn(0x7fe69c151260 legacy=0x7fe69c0c7040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:32.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.166+0000 7fe693fff640 1 -- 172.21.15.116:0/3993481118 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe69c154270 con 0x7fe69c0c7750 2024-04-25T00:01:32.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.166+0000 7fe693fff640 1 -- 172.21.15.116:0/3993481118 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7fe694001a00 con 0x7fe69c0c7750 2024-04-25T00:01:32.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.166+0000 7fe693fff640 1 -- 172.21.15.116:0/3993481118 >> v1:172.21.15.116:6789/0 conn(0x7fe69c0c7750 legacy=0x7fe69c0c4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:32.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.166+0000 7fe693fff640 1 --2- 172.21.15.116:0/3993481118 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6880039c0 0x7fe688003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:32.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.166+0000 7fe693fff640 1 --2- 172.21.15.116:0/3993481118 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe688004810 0x7fe688004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:32.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.166+0000 7fe693fff640 1 --2- 172.21.15.116:0/3993481118 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe6880051d0 0x7fe68800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:32.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.166+0000 7fe693fff640 1 -- 172.21.15.116:0/3993481118 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe69c1548c0 con 0x7fe6880039c0 2024-04-25T00:01:32.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.166+0000 7fe693fff640 1 -- 172.21.15.116:0/3993481118 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe68800c170 con 0x7fe6880051d0 2024-04-25T00:01:32.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.166+0000 7fe693fff640 1 -- 172.21.15.116:0/3993481118 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe68800c370 con 0x7fe688004810 2024-04-25T00:01:32.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.166+0000 7fe6a10ff640 1 --2- 172.21.15.116:0/3993481118 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6880039c0 0x7fe688003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:32.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.166+0000 7fe6a1900640 1 --2- 172.21.15.116:0/3993481118 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe6880051d0 0x7fe68800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:32.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.166+0000 7fe6a10ff640 1 -- 172.21.15.116:0/3993481118 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe688004810 msgr2=0x7fe688004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:32.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.166+0000 7fe6a08fe640 1 --2- 172.21.15.116:0/3993481118 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe688004810 0x7fe688004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:32.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.166+0000 7fe6a10ff640 1 --2- 172.21.15.116:0/3993481118 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe688004810 0x7fe688004c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:32.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.166+0000 7fe6a10ff640 1 -- 172.21.15.116:0/3993481118 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe6880051d0 msgr2=0x7fe68800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:32.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.166+0000 7fe6a10ff640 1 --2- 172.21.15.116:0/3993481118 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe6880051d0 0x7fe68800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:32.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.166+0000 7fe6a10ff640 1 -- 172.21.15.116:0/3993481118 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe688003530 con 0x7fe6880039c0 2024-04-25T00:01:32.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.166+0000 7fe6a1900640 1 --2- 172.21.15.116:0/3993481118 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe6880051d0 0x7fe68800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:32.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.166+0000 7fe6a1900640 1 -- 172.21.15.116:0/3993481118 reap_dead start 2024-04-25T00:01:32.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.170+0000 7fe6a10ff640 1 --2- 172.21.15.116:0/3993481118 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6880039c0 0x7fe688003de0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe698001780 tx=0x7fe69801c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=def1aa551c087264 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:32.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.170+0000 7fe693fff640 1 -- 172.21.15.116:0/3993481118 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6980013a0 con 0x7fe6880039c0 2024-04-25T00:01:32.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.170+0000 7fe693fff640 1 -- 172.21.15.116:0/3993481118 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6980052f0 con 0x7fe6880039c0 2024-04-25T00:01:32.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.170+0000 7fe693fff640 1 -- 172.21.15.116:0/3993481118 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe698003840 con 0x7fe6880039c0 2024-04-25T00:01:32.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.170+0000 7fe6a3363640 1 -- 172.21.15.116:0/3993481118 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6880039c0 msgr2=0x7fe688003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:32.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.170+0000 7fe6a3363640 1 --2- 172.21.15.116:0/3993481118 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6880039c0 0x7fe688003de0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe698001780 tx=0x7fe69801c400 comp rx=0 tx=0).stop 2024-04-25T00:01:32.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.170+0000 7fe6a3363640 1 -- 172.21.15.116:0/3993481118 shutdown_connections 2024-04-25T00:01:32.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.170+0000 7fe6a3363640 1 --2- 172.21.15.116:0/3993481118 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe6880039c0 0x7fe688003de0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:32.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.170+0000 7fe6a3363640 1 -- 172.21.15.116:0/3993481118 >> 172.21.15.116:0/3993481118 conn(0x7fe69c0c8f90 msgr2=0x7fe69c0c93a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:32.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.170+0000 7fe6a3363640 1 -- 172.21.15.116:0/3993481118 shutdown_connections 2024-04-25T00:01:32.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.170+0000 7fe6a3363640 1 -- 172.21.15.116:0/3993481118 wait complete. 2024-04-25T00:01:32.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.170+0000 7fe6a3363640 1 Processor -- start 2024-04-25T00:01:32.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.170+0000 7fe6a3363640 1 -- start start 2024-04-25T00:01:32.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.170+0000 7fe6a3363640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe6880039c0 0x7fe69c152960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:32.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.170+0000 7fe6a3363640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe69c0c5520 0x7fe69c14f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:32.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.170+0000 7fe6a3363640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe69c1f0590 0x7fe69c14fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:32.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.170+0000 7fe6a3363640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe69c0c75b0 con 0x7fe69c1f0590 2024-04-25T00:01:32.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.170+0000 7fe6a3363640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe69c0c7430 con 0x7fe6880039c0 2024-04-25T00:01:32.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.170+0000 7fe6a3363640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe69c0c7b30 con 0x7fe69c0c5520 2024-04-25T00:01:32.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.170+0000 7fe6a1900640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe69c1f0590 0x7fe69c14fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:32.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.170+0000 7fe6a1900640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe69c1f0590 0x7fe69c14fec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56620/0 (socket says 172.21.15.116:56620) 2024-04-25T00:01:32.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.170+0000 7fe6a1900640 1 -- 172.21.15.116:0/1768990135 learned_addr learned my addr 172.21.15.116:0/1768990135 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:32.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.170+0000 7fe6a1900640 1 -- 172.21.15.116:0/1768990135 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe69c0c5520 msgr2=0x7fe69c14f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:32.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.170+0000 7fe6a1900640 1 --2- 172.21.15.116:0/1768990135 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe69c0c5520 0x7fe69c14f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:32.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.170+0000 7fe6a10ff640 1 --2- 172.21.15.116:0/1768990135 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe6880039c0 0x7fe69c152960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:32.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.174+0000 7fe6a1900640 1 -- 172.21.15.116:0/1768990135 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe6880039c0 msgr2=0x7fe69c152960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:32.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.174+0000 7fe6a1900640 1 --2- 172.21.15.116:0/1768990135 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe6880039c0 0x7fe69c152960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:32.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.174+0000 7fe6a1900640 1 -- 172.21.15.116:0/1768990135 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe69c0a9270 con 0x7fe69c1f0590 2024-04-25T00:01:32.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.174+0000 7fe6a1900640 1 --2- 172.21.15.116:0/1768990135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe69c1f0590 0x7fe69c14fec0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fe69c1580d0 tx=0x7fe69400b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:32.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.174+0000 7fe691ffb640 1 -- 172.21.15.116:0/1768990135 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe694004d70 con 0x7fe69c1f0590 2024-04-25T00:01:32.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.174+0000 7fe691ffb640 1 -- 172.21.15.116:0/1768990135 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe69400bb80 con 0x7fe69c1f0590 2024-04-25T00:01:32.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.174+0000 7fe6a3363640 1 -- 172.21.15.116:0/1768990135 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe69c1f0a50 con 0x7fe69c1f0590 2024-04-25T00:01:32.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.174+0000 7fe691ffb640 1 -- 172.21.15.116:0/1768990135 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe694007070 con 0x7fe69c1f0590 2024-04-25T00:01:32.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.174+0000 7fe6a3363640 1 -- 172.21.15.116:0/1768990135 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe69c150890 con 0x7fe69c1f0590 2024-04-25T00:01:32.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.174+0000 7fe6a3363640 1 -- 172.21.15.116:0/1768990135 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe69c04f510 con 0x7fe69c1f0590 2024-04-25T00:01:32.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.174+0000 7fe691ffb640 1 -- 172.21.15.116:0/1768990135 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe69401b020 con 0x7fe69c1f0590 2024-04-25T00:01:32.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.174+0000 7fe691ffb640 1 --2- 172.21.15.116:0/1768990135 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe6800678c0 0x7fe680069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:32.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.174+0000 7fe691ffb640 1 -- 172.21.15.116:0/1768990135 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7fe6940860c0 con 0x7fe69c1f0590 2024-04-25T00:01:32.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.174+0000 7fe6a10ff640 1 --2- 172.21.15.116:0/1768990135 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe6800678c0 0x7fe680069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:32.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.174+0000 7fe6a10ff640 1 --2- 172.21.15.116:0/1768990135 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe6800678c0 0x7fe680069d80 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe698001750 tx=0x7fe6980011a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:32.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.178+0000 7fe691ffb640 1 -- 172.21.15.116:0/1768990135 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fe694056310 con 0x7fe69c1f0590 2024-04-25T00:01:32.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.338+0000 7fe6a3363640 1 -- 172.21.15.116:0/1768990135 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe69c150a60 con 0x7fe69c1f0590 2024-04-25T00:01:32.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.338+0000 7fe691ffb640 1 -- 172.21.15.116:0/1768990135 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7fe69405a610 con 0x7fe69c1f0590 2024-04-25T00:01:32.342 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-04-25T00:01:32.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.342+0000 7fe6a3363640 1 -- 172.21.15.116:0/1768990135 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe6800678c0 msgr2=0x7fe680069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:32.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.342+0000 7fe6a3363640 1 --2- 172.21.15.116:0/1768990135 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe6800678c0 0x7fe680069d80 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe698001750 tx=0x7fe6980011a0 comp rx=0 tx=0).stop 2024-04-25T00:01:32.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.342+0000 7fe6a3363640 1 -- 172.21.15.116:0/1768990135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe69c1f0590 msgr2=0x7fe69c14fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:32.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.342+0000 7fe6a3363640 1 --2- 172.21.15.116:0/1768990135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe69c1f0590 0x7fe69c14fec0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fe69c1580d0 tx=0x7fe69400b4a0 comp rx=0 tx=0).stop 2024-04-25T00:01:32.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.342+0000 7fe6a3363640 1 -- 172.21.15.116:0/1768990135 shutdown_connections 2024-04-25T00:01:32.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.342+0000 7fe6a3363640 1 --2- 172.21.15.116:0/1768990135 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe6800678c0 0x7fe680069d80 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:32.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.342+0000 7fe6a3363640 1 --2- 172.21.15.116:0/1768990135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe69c1f0590 0x7fe69c14fec0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:32.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.342+0000 7fe6a3363640 1 --2- 172.21.15.116:0/1768990135 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe69c0c5520 0x7fe69c14f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:32.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.342+0000 7fe6a3363640 1 --2- 172.21.15.116:0/1768990135 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe6880039c0 0x7fe69c152960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:32.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.342+0000 7fe6a3363640 1 -- 172.21.15.116:0/1768990135 >> 172.21.15.116:0/1768990135 conn(0x7fe69c0c8f90 msgr2=0x7fe69c0ac640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:32.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.342+0000 7fe6a3363640 1 -- 172.21.15.116:0/1768990135 shutdown_connections 2024-04-25T00:01:32.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.342+0000 7fe6a3363640 1 -- 172.21.15.116:0/1768990135 wait complete. 2024-04-25T00:01:32.361 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_metadata pg_num 2024-04-25T00:01:32.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.470+0000 7f76a4d9b640 1 Processor -- start 2024-04-25T00:01:32.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.470+0000 7f76a4d9b640 1 -- start start 2024-04-25T00:01:32.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.474+0000 7f76a4d9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f76a00a9630 0x7f76a0156e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:32.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.474+0000 7f76a4d9b640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76a00a9cb0 con 0x7f76a0157340 2024-04-25T00:01:32.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.474+0000 7f76a4d9b640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76a00a9f90 con 0x7f76a0150d60 2024-04-25T00:01:32.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.474+0000 7f76a4d9b640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f76a00aa270 con 0x7f76a00a9630 2024-04-25T00:01:32.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.474+0000 7f769dd9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f76a00a9630 0x7f76a0156e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:32.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.474+0000 7f769ed9d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f76a0157340 0x7f76a015df50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:41812/0 (socket says 172.21.15.116:41812) 2024-04-25T00:01:32.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.474+0000 7f769dd9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f76a00a9630 0x7f76a0156e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56624/0 (socket says 172.21.15.116:56624) 2024-04-25T00:01:32.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.474+0000 7f769ed9d640 1 -- 172.21.15.116:0/2909615827 learned_addr learned my addr 172.21.15.116:0/2909615827 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:32.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.474+0000 7f769dd9b640 1 -- 172.21.15.116:0/2909615827 >> v1:172.21.15.174:6789/0 conn(0x7f76a0150d60 legacy=0x7f76a00c2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:32.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.474+0000 7f769dd9b640 1 -- 172.21.15.116:0/2909615827 >> v1:172.21.15.116:6789/0 conn(0x7f76a0157340 legacy=0x7f76a015df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:32.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.474+0000 7f769d59a640 1 -- 172.21.15.116:0/2909615827 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4141122124 0 0) 0x7f76a00a9cb0 con 0x7f76a0157340 2024-04-25T00:01:32.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.474+0000 7f769dd9b640 1 -- 172.21.15.116:0/2909615827 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76a00aa550 con 0x7f76a00a9630 2024-04-25T00:01:32.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.474+0000 7f769dd9b640 1 --2- 172.21.15.116:0/2909615827 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76a00a9630 0x7f76a0156e00 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f76940097c0 tx=0x7f769401c310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57926b1dbd24b119 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:32.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.474+0000 7f769d59a640 1 -- 172.21.15.116:0/2909615827 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76940b0070 con 0x7f76a00a9630 2024-04-25T00:01:32.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.474+0000 7f769d59a640 1 -- 172.21.15.116:0/2909615827 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76940ab600 con 0x7f76a00a9630 2024-04-25T00:01:32.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.474+0000 7f769d59a640 1 -- 172.21.15.116:0/2909615827 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76940ab9d0 con 0x7f76a00a9630 2024-04-25T00:01:32.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.474+0000 7f76a4d9b640 1 -- 172.21.15.116:0/2909615827 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76a00a9630 msgr2=0x7f76a0156e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:32.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.474+0000 7f76a4d9b640 1 --2- 172.21.15.116:0/2909615827 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76a00a9630 0x7f76a0156e00 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f76940097c0 tx=0x7f769401c310 comp rx=0 tx=0).stop 2024-04-25T00:01:32.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.478+0000 7f76a4d9b640 1 -- 172.21.15.116:0/2909615827 shutdown_connections 2024-04-25T00:01:32.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.478+0000 7f76a4d9b640 1 --2- 172.21.15.116:0/2909615827 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76a00a9630 0x7f76a0156e00 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:32.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.478+0000 7f76a4d9b640 1 -- 172.21.15.116:0/2909615827 >> 172.21.15.116:0/2909615827 conn(0x7f76a00c3a20 msgr2=0x7f76a00c3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:32.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.478+0000 7f76a4d9b640 1 -- 172.21.15.116:0/2909615827 shutdown_connections 2024-04-25T00:01:32.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.478+0000 7f76a4d9b640 1 -- 172.21.15.116:0/2909615827 wait complete. 2024-04-25T00:01:32.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.478+0000 7f76a4d9b640 1 Processor -- start 2024-04-25T00:01:32.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.478+0000 7f76a4d9b640 1 -- start start 2024-04-25T00:01:32.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.478+0000 7f76a4d9b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f76a00a9630 0x7f76a014f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:32.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.478+0000 7f76a4d9b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f76a0150d60 0x7f76a0151a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:32.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.478+0000 7f76a4d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76a0157340 0x7f76a0151f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:32.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.478+0000 7f76a4d9b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f76a0072150 con 0x7f76a0157340 2024-04-25T00:01:32.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.478+0000 7f76a4d9b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f76a0071fd0 con 0x7f76a0150d60 2024-04-25T00:01:32.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.478+0000 7f76a4d9b640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f76a0161e90 con 0x7f76a00a9630 2024-04-25T00:01:32.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.478+0000 7f769ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76a0157340 0x7f76a0151f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:32.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.478+0000 7f769ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76a0157340 0x7f76a0151f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56630/0 (socket says 172.21.15.116:56630) 2024-04-25T00:01:32.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.478+0000 7f769ed9d640 1 -- 172.21.15.116:0/756222259 learned_addr learned my addr 172.21.15.116:0/756222259 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:32.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.478+0000 7f769dd9b640 1 --2- 172.21.15.116:0/756222259 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f76a0150d60 0x7f76a0151a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:32.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.478+0000 7f769ed9d640 1 -- 172.21.15.116:0/756222259 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f76a00a9630 msgr2=0x7f76a014f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:32.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.478+0000 7f769ed9d640 1 --2- 172.21.15.116:0/756222259 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f76a00a9630 0x7f76a014f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:32.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.478+0000 7f769ed9d640 1 -- 172.21.15.116:0/756222259 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f76a0150d60 msgr2=0x7f76a0151a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:32.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.478+0000 7f769ed9d640 1 --2- 172.21.15.116:0/756222259 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f76a0150d60 0x7f76a0151a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:32.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.478+0000 7f769ed9d640 1 -- 172.21.15.116:0/756222259 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76a013fda0 con 0x7f76a0157340 2024-04-25T00:01:32.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.478+0000 7f769ed9d640 1 --2- 172.21.15.116:0/756222259 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76a0157340 0x7f76a0151f90 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f769800c700 tx=0x7f769800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:32.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.478+0000 7f768effd640 1 -- 172.21.15.116:0/756222259 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7698018070 con 0x7f76a0157340 2024-04-25T00:01:32.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.478+0000 7f768effd640 1 -- 172.21.15.116:0/756222259 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f769800ce10 con 0x7f76a0157340 2024-04-25T00:01:32.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.478+0000 7f76a4d9b640 1 -- 172.21.15.116:0/756222259 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76a013f500 con 0x7f76a0157340 2024-04-25T00:01:32.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.478+0000 7f768effd640 1 -- 172.21.15.116:0/756222259 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f769800a880 con 0x7f76a0157340 2024-04-25T00:01:32.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.478+0000 7f76a4d9b640 1 -- 172.21.15.116:0/756222259 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76a013fac0 con 0x7f76a0157340 2024-04-25T00:01:32.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.478+0000 7f76a4d9b640 1 -- 172.21.15.116:0/756222259 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76a004f510 con 0x7f76a0157340 2024-04-25T00:01:32.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.482+0000 7f768effd640 1 -- 172.21.15.116:0/756222259 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f769801b070 con 0x7f76a0157340 2024-04-25T00:01:32.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.482+0000 7f768effd640 1 --2- 172.21.15.116:0/756222259 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f76800678c0 0x7f7680069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:32.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.482+0000 7f768effd640 1 -- 172.21.15.116:0/756222259 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7f7698085490 con 0x7f76a0157340 2024-04-25T00:01:32.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.482+0000 7f769e59c640 1 --2- 172.21.15.116:0/756222259 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f76800678c0 0x7f7680069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:32.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.482+0000 7f769e59c640 1 --2- 172.21.15.116:0/756222259 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f76800678c0 0x7f7680069d80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7690009670 tx=0x7f7690006bd0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:32.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.486+0000 7f768effd640 1 -- 172.21.15.116:0/756222259 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f7698055d30 con 0x7f76a0157340 2024-04-25T00:01:32.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.646+0000 7f76a4d9b640 1 -- 172.21.15.116:0/756222259 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"} v 0) -- 0x7f76a0071db0 con 0x7f76a0157340 2024-04-25T00:01:32.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.646+0000 7f768effd640 1 -- 172.21.15.116:0/756222259 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"}]=0 v27) ==== 99+0+11 (secure 0 0 0) 0x7f7698055630 con 0x7f76a0157340 2024-04-25T00:01:32.647 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-04-25T00:01:32.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.646+0000 7f76a4d9b640 1 -- 172.21.15.116:0/756222259 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f76800678c0 msgr2=0x7f7680069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:32.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.646+0000 7f76a4d9b640 1 --2- 172.21.15.116:0/756222259 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f76800678c0 0x7f7680069d80 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7690009670 tx=0x7f7690006bd0 comp rx=0 tx=0).stop 2024-04-25T00:01:32.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.646+0000 7f76a4d9b640 1 -- 172.21.15.116:0/756222259 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76a0157340 msgr2=0x7f76a0151f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:32.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.646+0000 7f76a4d9b640 1 --2- 172.21.15.116:0/756222259 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76a0157340 0x7f76a0151f90 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f769800c700 tx=0x7f769800cbc0 comp rx=0 tx=0).stop 2024-04-25T00:01:32.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.646+0000 7f76a4d9b640 1 -- 172.21.15.116:0/756222259 shutdown_connections 2024-04-25T00:01:32.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.646+0000 7f76a4d9b640 1 --2- 172.21.15.116:0/756222259 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f76800678c0 0x7f7680069d80 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:32.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.646+0000 7f76a4d9b640 1 --2- 172.21.15.116:0/756222259 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76a0157340 0x7f76a0151f90 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:32.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.646+0000 7f76a4d9b640 1 --2- 172.21.15.116:0/756222259 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f76a0150d60 0x7f76a0151a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:32.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.646+0000 7f76a4d9b640 1 --2- 172.21.15.116:0/756222259 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f76a00a9630 0x7f76a014f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:32.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.646+0000 7f76a4d9b640 1 -- 172.21.15.116:0/756222259 >> 172.21.15.116:0/756222259 conn(0x7f76a00c3a20 msgr2=0x7f76a0072360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:32.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.646+0000 7f76a4d9b640 1 -- 172.21.15.116:0/756222259 shutdown_connections 2024-04-25T00:01:32.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.646+0000 7f76a4d9b640 1 -- 172.21.15.116:0/756222259 wait complete. 2024-04-25T00:01:32.666 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_data pg_num 2024-04-25T00:01:32.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.786+0000 7f853aa7c640 1 Processor -- start 2024-04-25T00:01:32.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.790+0000 7f853aa7c640 1 -- start start 2024-04-25T00:01:32.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.790+0000 7f853aa7c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8534153130 0x7f85341562e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:32.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.790+0000 7f853aa7c640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f85340ca1b0 0x7f853406bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:32.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.790+0000 7f853aa7c640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8534156820 con 0x7f85340c9260 2024-04-25T00:01:32.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.790+0000 7f853aa7c640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f85340ca820 con 0x7f8534153130 2024-04-25T00:01:32.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.790+0000 7f853aa7c640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f8534155e50 con 0x7f85340ca1b0 2024-04-25T00:01:32.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.790+0000 7f8538818640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8534153130 0x7f85341562e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:32.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.790+0000 7f8538818640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8534153130 0x7f85341562e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56646/0 (socket says 172.21.15.116:56646) 2024-04-25T00:01:32.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.790+0000 7f8538818640 1 -- 172.21.15.116:0/804082452 learned_addr learned my addr 172.21.15.116:0/804082452 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:32.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.790+0000 7f8533fff640 1 --2- 172.21.15.116:0/804082452 >> v2:172.21.15.174:3300/0 conn(0x7f85340ca1b0 0x7f853406bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:32.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.790+0000 7f8538818640 1 -- 172.21.15.116:0/804082452 >> v2:172.21.15.174:3300/0 conn(0x7f85340ca1b0 msgr2=0x7f853406bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:32.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.790+0000 7f8538818640 1 --2- 172.21.15.116:0/804082452 >> v2:172.21.15.174:3300/0 conn(0x7f85340ca1b0 0x7f853406bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:32.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.790+0000 7f8538818640 1 -- 172.21.15.116:0/804082452 >> v1:172.21.15.174:6789/0 conn(0x7f85340c9260 legacy=0x7f85340c9710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:32.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.790+0000 7f8538818640 1 -- 172.21.15.116:0/804082452 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f853406c300 con 0x7f8534153130 2024-04-25T00:01:32.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.790+0000 7f8538818640 1 --2- 172.21.15.116:0/804082452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8534153130 0x7f85341562e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f8534068f10 tx=0x7f852801c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=992e7ca7913c2796 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:32.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.790+0000 7f85337fe640 1 -- 172.21.15.116:0/804082452 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85280b0070 con 0x7f8534153130 2024-04-25T00:01:32.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.790+0000 7f85337fe640 1 -- 172.21.15.116:0/804082452 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85280a4e50 con 0x7f8534153130 2024-04-25T00:01:32.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.790+0000 7f85337fe640 1 -- 172.21.15.116:0/804082452 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85280ab820 con 0x7f8534153130 2024-04-25T00:01:32.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.794+0000 7f853aa7c640 1 -- 172.21.15.116:0/804082452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8534153130 msgr2=0x7f85341562e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:32.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.794+0000 7f853aa7c640 1 --2- 172.21.15.116:0/804082452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8534153130 0x7f85341562e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f8534068f10 tx=0x7f852801c3c0 comp rx=0 tx=0).stop 2024-04-25T00:01:32.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.794+0000 7f853aa7c640 1 -- 172.21.15.116:0/804082452 shutdown_connections 2024-04-25T00:01:32.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.794+0000 7f853aa7c640 1 --2- 172.21.15.116:0/804082452 >> v2:172.21.15.174:3300/0 conn(0x7f85340ca1b0 0x7f853406bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:32.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.794+0000 7f853aa7c640 1 --2- 172.21.15.116:0/804082452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8534153130 0x7f85341562e0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:32.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.794+0000 7f853aa7c640 1 -- 172.21.15.116:0/804082452 >> 172.21.15.116:0/804082452 conn(0x7f85340cb310 msgr2=0x7f85340cb720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:32.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.794+0000 7f853aa7c640 1 -- 172.21.15.116:0/804082452 shutdown_connections 2024-04-25T00:01:32.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.794+0000 7f853aa7c640 1 -- 172.21.15.116:0/804082452 wait complete. 2024-04-25T00:01:32.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.794+0000 7f853aa7c640 1 Processor -- start 2024-04-25T00:01:32.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.794+0000 7f853aa7c640 1 -- start start 2024-04-25T00:01:32.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.794+0000 7f853aa7c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f85340c9260 0x7f853414c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:32.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.794+0000 7f853aa7c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85340ca1b0 0x7f853414eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:32.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.794+0000 7f853aa7c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8534153130 0x7f853414f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:32.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.794+0000 7f853aa7c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f85340bea70 con 0x7f85340ca1b0 2024-04-25T00:01:32.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.794+0000 7f853aa7c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f85340be8f0 con 0x7f85340c9260 2024-04-25T00:01:32.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.794+0000 7f853aa7c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f8534162990 con 0x7f8534153130 2024-04-25T00:01:32.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.794+0000 7f8533fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85340ca1b0 0x7f853414eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:32.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.794+0000 7f8533fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85340ca1b0 0x7f853414eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56660/0 (socket says 172.21.15.116:56660) 2024-04-25T00:01:32.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.794+0000 7f8533fff640 1 -- 172.21.15.116:0/3303063254 learned_addr learned my addr 172.21.15.116:0/3303063254 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:32.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.794+0000 7f8538818640 1 --2- 172.21.15.116:0/3303063254 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f85340c9260 0x7f853414c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:32.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.794+0000 7f8539019640 1 --2- 172.21.15.116:0/3303063254 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8534153130 0x7f853414f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:32.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.794+0000 7f8533fff640 1 -- 172.21.15.116:0/3303063254 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8534153130 msgr2=0x7f853414f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:32.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.794+0000 7f8533fff640 1 --2- 172.21.15.116:0/3303063254 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8534153130 0x7f853414f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:32.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.794+0000 7f8533fff640 1 -- 172.21.15.116:0/3303063254 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f85340c9260 msgr2=0x7f853414c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:32.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.794+0000 7f8533fff640 1 --2- 172.21.15.116:0/3303063254 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f85340c9260 0x7f853414c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:32.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.794+0000 7f8533fff640 1 -- 172.21.15.116:0/3303063254 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85341edf60 con 0x7f85340ca1b0 2024-04-25T00:01:32.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.794+0000 7f8533fff640 1 --2- 172.21.15.116:0/3303063254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85340ca1b0 0x7f853414eae0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f852400c820 tx=0x7f852400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:32.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.794+0000 7f85317fa640 1 -- 172.21.15.116:0/3303063254 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8524004230 con 0x7f85340ca1b0 2024-04-25T00:01:32.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.794+0000 7f85317fa640 1 -- 172.21.15.116:0/3303063254 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85240043d0 con 0x7f85340ca1b0 2024-04-25T00:01:32.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.794+0000 7f853aa7c640 1 -- 172.21.15.116:0/3303063254 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85341ee220 con 0x7f85340ca1b0 2024-04-25T00:01:32.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.794+0000 7f85317fa640 1 -- 172.21.15.116:0/3303063254 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8524007a70 con 0x7f85340ca1b0 2024-04-25T00:01:32.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.794+0000 7f853aa7c640 1 -- 172.21.15.116:0/3303063254 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85341ee7e0 con 0x7f85340ca1b0 2024-04-25T00:01:32.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.794+0000 7f853aa7c640 1 -- 172.21.15.116:0/3303063254 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f853404f510 con 0x7f85340ca1b0 2024-04-25T00:01:32.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.798+0000 7f85317fa640 1 -- 172.21.15.116:0/3303063254 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8524007c10 con 0x7f85340ca1b0 2024-04-25T00:01:32.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.798+0000 7f85317fa640 1 --2- 172.21.15.116:0/3303063254 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f85200678c0 0x7f8520069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:32.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.798+0000 7f85317fa640 1 -- 172.21.15.116:0/3303063254 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7f8524085f60 con 0x7f85340ca1b0 2024-04-25T00:01:32.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.798+0000 7f8538818640 1 --2- 172.21.15.116:0/3303063254 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f85200678c0 0x7f8520069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:32.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.798+0000 7f8538818640 1 --2- 172.21.15.116:0/3303063254 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f85200678c0 0x7f8520069d80 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f85280ad000 tx=0x7f8528008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:32.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.802+0000 7f85317fa640 1 -- 172.21.15.116:0/3303063254 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f8524015020 con 0x7f85340ca1b0 2024-04-25T00:01:32.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.962+0000 7f853aa7c640 1 -- 172.21.15.116:0/3303063254 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"} v 0) -- 0x7f853406da40 con 0x7f85340ca1b0 2024-04-25T00:01:32.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.962+0000 7f85317fa640 1 -- 172.21.15.116:0/3303063254 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"}]=0 v27) ==== 95+0+11 (secure 0 0 0) 0x7f8524056800 con 0x7f85340ca1b0 2024-04-25T00:01:32.966 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-04-25T00:01:32.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.966+0000 7f853aa7c640 1 -- 172.21.15.116:0/3303063254 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f85200678c0 msgr2=0x7f8520069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:32.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.966+0000 7f853aa7c640 1 --2- 172.21.15.116:0/3303063254 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f85200678c0 0x7f8520069d80 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f85280ad000 tx=0x7f8528008040 comp rx=0 tx=0).stop 2024-04-25T00:01:32.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.966+0000 7f853aa7c640 1 -- 172.21.15.116:0/3303063254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85340ca1b0 msgr2=0x7f853414eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:32.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.966+0000 7f853aa7c640 1 --2- 172.21.15.116:0/3303063254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85340ca1b0 0x7f853414eae0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f852400c820 tx=0x7f852400cce0 comp rx=0 tx=0).stop 2024-04-25T00:01:32.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.966+0000 7f853aa7c640 1 -- 172.21.15.116:0/3303063254 shutdown_connections 2024-04-25T00:01:32.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.966+0000 7f853aa7c640 1 --2- 172.21.15.116:0/3303063254 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f85200678c0 0x7f8520069d80 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:32.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.966+0000 7f853aa7c640 1 --2- 172.21.15.116:0/3303063254 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8534153130 0x7f853414f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:32.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.966+0000 7f853aa7c640 1 --2- 172.21.15.116:0/3303063254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85340ca1b0 0x7f853414eae0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:32.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.966+0000 7f853aa7c640 1 --2- 172.21.15.116:0/3303063254 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f85340c9260 0x7f853414c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:32.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.966+0000 7f853aa7c640 1 -- 172.21.15.116:0/3303063254 >> 172.21.15.116:0/3303063254 conn(0x7f85340cb310 msgr2=0x7f85341501a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:32.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.966+0000 7f853aa7c640 1 -- 172.21.15.116:0/3303063254 shutdown_connections 2024-04-25T00:01:32.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:32.966+0000 7f853aa7c640 1 -- 172.21.15.116:0/3303063254 wait complete. 2024-04-25T00:01:32.985 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_metadata pg_num 2024-04-25T00:01:33.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.106+0000 7faa93971640 1 Processor -- start 2024-04-25T00:01:33.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.106+0000 7faa93971640 1 -- start start 2024-04-25T00:01:33.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.106+0000 7faa93971640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa8c153130 0x7faa8c1562e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:33.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.106+0000 7faa93971640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7faa8c0ca1b0 0x7faa8c06bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:33.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.106+0000 7faa93971640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa8c156820 con 0x7faa8c0c9260 2024-04-25T00:01:33.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.106+0000 7faa93971640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7faa8c0c9dd0 con 0x7faa8c0ca1b0 2024-04-25T00:01:33.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.106+0000 7faa93971640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7faa8c0ca820 con 0x7faa8c153130 2024-04-25T00:01:33.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.106+0000 7faa90f0c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7faa8c0ca1b0 0x7faa8c06bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:33.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.106+0000 7faa91f0e640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7faa8c0c9260 0x7faa8c0c96c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:41824/0 (socket says 172.21.15.116:41824) 2024-04-25T00:01:33.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.106+0000 7faa91f0e640 1 -- 172.21.15.116:0/3985357097 learned_addr learned my addr 172.21.15.116:0/3985357097 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:33.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.106+0000 7faa9170d640 1 --2- 172.21.15.116:0/3985357097 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa8c153130 0x7faa8c1562e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:33.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.106+0000 7faa90f0c640 1 -- 172.21.15.116:0/3985357097 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa8c153130 msgr2=0x7faa8c1562e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:33.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.106+0000 7faa90f0c640 1 --2- 172.21.15.116:0/3985357097 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa8c153130 0x7faa8c1562e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:33.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.106+0000 7faa90f0c640 1 -- 172.21.15.116:0/3985357097 >> v1:172.21.15.116:6789/0 conn(0x7faa8c0c9260 legacy=0x7faa8c0c96c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:33.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.106+0000 7faa90f0c640 1 -- 172.21.15.116:0/3985357097 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa8c06c300 con 0x7faa8c0ca1b0 2024-04-25T00:01:33.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.106+0000 7faa90f0c640 1 --2- 172.21.15.116:0/3985357097 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa8c0ca1b0 0x7faa8c06bdc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7faa880097c0 tx=0x7faa8801bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20506a3cfd9c4f6e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:33.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.110+0000 7faa83fff640 1 -- 172.21.15.116:0/3985357097 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa880a9650 con 0x7faa8c0ca1b0 2024-04-25T00:01:33.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.110+0000 7faa83fff640 1 -- 172.21.15.116:0/3985357097 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa880ae070 con 0x7faa8c0ca1b0 2024-04-25T00:01:33.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.110+0000 7faa83fff640 1 -- 172.21.15.116:0/3985357097 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa880a9a10 con 0x7faa8c0ca1b0 2024-04-25T00:01:33.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.110+0000 7faa93971640 1 -- 172.21.15.116:0/3985357097 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa8c0ca1b0 msgr2=0x7faa8c06bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:33.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.110+0000 7faa93971640 1 --2- 172.21.15.116:0/3985357097 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa8c0ca1b0 0x7faa8c06bdc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7faa880097c0 tx=0x7faa8801bf80 comp rx=0 tx=0).stop 2024-04-25T00:01:33.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.110+0000 7faa93971640 1 -- 172.21.15.116:0/3985357097 shutdown_connections 2024-04-25T00:01:33.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.110+0000 7faa93971640 1 --2- 172.21.15.116:0/3985357097 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa8c0ca1b0 0x7faa8c06bdc0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:33.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.110+0000 7faa93971640 1 --2- 172.21.15.116:0/3985357097 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa8c153130 0x7faa8c1562e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:33.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.110+0000 7faa93971640 1 -- 172.21.15.116:0/3985357097 >> 172.21.15.116:0/3985357097 conn(0x7faa8c0cb310 msgr2=0x7faa8c0cb720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:33.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.110+0000 7faa93971640 1 -- 172.21.15.116:0/3985357097 shutdown_connections 2024-04-25T00:01:33.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.110+0000 7faa93971640 1 -- 172.21.15.116:0/3985357097 wait complete. 2024-04-25T00:01:33.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.110+0000 7faa93971640 1 Processor -- start 2024-04-25T00:01:33.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.110+0000 7faa93971640 1 -- start start 2024-04-25T00:01:33.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.110+0000 7faa93971640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa8c0c9260 0x7faa8c14c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:33.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.110+0000 7faa93971640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa8c0ca1b0 0x7faa8c1583c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:33.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.110+0000 7faa93971640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa8c153130 0x7faa8c0c1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:33.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.110+0000 7faa93971640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7faa8c163540 con 0x7faa8c0ca1b0 2024-04-25T00:01:33.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.110+0000 7faa93971640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7faa8c1633c0 con 0x7faa8c0c9260 2024-04-25T00:01:33.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.110+0000 7faa93971640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7faa8c1636c0 con 0x7faa8c153130 2024-04-25T00:01:33.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.110+0000 7faa90f0c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa8c0ca1b0 0x7faa8c1583c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:33.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.110+0000 7faa90f0c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa8c0ca1b0 0x7faa8c1583c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56676/0 (socket says 172.21.15.116:56676) 2024-04-25T00:01:33.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.110+0000 7faa90f0c640 1 -- 172.21.15.116:0/2906253409 learned_addr learned my addr 172.21.15.116:0/2906253409 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:33.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.110+0000 7faa91f0e640 1 --2- 172.21.15.116:0/2906253409 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa8c153130 0x7faa8c0c1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:33.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.110+0000 7faa9170d640 1 --2- 172.21.15.116:0/2906253409 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa8c0c9260 0x7faa8c14c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:33.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.110+0000 7faa90f0c640 1 -- 172.21.15.116:0/2906253409 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa8c153130 msgr2=0x7faa8c0c1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:33.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.110+0000 7faa90f0c640 1 --2- 172.21.15.116:0/2906253409 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa8c153130 0x7faa8c0c1120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:33.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.110+0000 7faa90f0c640 1 -- 172.21.15.116:0/2906253409 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa8c0c9260 msgr2=0x7faa8c14c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:33.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.110+0000 7faa90f0c640 1 --2- 172.21.15.116:0/2906253409 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa8c0c9260 0x7faa8c14c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:33.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.110+0000 7faa90f0c640 1 -- 172.21.15.116:0/2906253409 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa8c1571b0 con 0x7faa8c0ca1b0 2024-04-25T00:01:33.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.110+0000 7faa9170d640 1 --2- 172.21.15.116:0/2906253409 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa8c0c9260 0x7faa8c14c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:33.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.110+0000 7faa90f0c640 1 --2- 172.21.15.116:0/2906253409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa8c0ca1b0 0x7faa8c1583c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7faa880ab040 tx=0x7faa8801a7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:33.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.110+0000 7faa81ffb640 1 -- 172.21.15.116:0/2906253409 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa880bd410 con 0x7faa8c0ca1b0 2024-04-25T00:01:33.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.110+0000 7faa81ffb640 1 -- 172.21.15.116:0/2906253409 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa880a5040 con 0x7faa8c0ca1b0 2024-04-25T00:01:33.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.110+0000 7faa93971640 1 -- 172.21.15.116:0/2906253409 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa8c157490 con 0x7faa8c0ca1b0 2024-04-25T00:01:33.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.110+0000 7faa81ffb640 1 -- 172.21.15.116:0/2906253409 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa880bd7f0 con 0x7faa8c0ca1b0 2024-04-25T00:01:33.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.110+0000 7faa93971640 1 -- 172.21.15.116:0/2906253409 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa8c157a50 con 0x7faa8c0ca1b0 2024-04-25T00:01:33.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.114+0000 7faa93971640 1 -- 172.21.15.116:0/2906253409 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa8c04f510 con 0x7faa8c0ca1b0 2024-04-25T00:01:33.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.114+0000 7faa81ffb640 1 -- 172.21.15.116:0/2906253409 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7faa880bb020 con 0x7faa8c0ca1b0 2024-04-25T00:01:33.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.114+0000 7faa81ffb640 1 --2- 172.21.15.116:0/2906253409 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faa700678c0 0x7faa70069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:33.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.114+0000 7faa81ffb640 1 -- 172.21.15.116:0/2906253409 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7faa8811f5d0 con 0x7faa8c0ca1b0 2024-04-25T00:01:33.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.114+0000 7faa9170d640 1 --2- 172.21.15.116:0/2906253409 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faa700678c0 0x7faa70069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:33.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.114+0000 7faa9170d640 1 --2- 172.21.15.116:0/2906253409 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faa700678c0 0x7faa70069d80 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7faa7c009840 tx=0x7faa7c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:33.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.118+0000 7faa81ffb640 1 -- 172.21.15.116:0/2906253409 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7faa88015fb0 con 0x7faa8c0ca1b0 2024-04-25T00:01:33.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.266+0000 7faa93971640 1 -- 172.21.15.116:0/2906253409 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"} v 0) -- 0x7faa8c06da40 con 0x7faa8c0ca1b0 2024-04-25T00:01:33.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.266+0000 7faa81ffb640 1 -- 172.21.15.116:0/2906253409 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"}]=0 v27) ==== 99+0+11 (secure 0 0 0) 0x7faa880eff90 con 0x7faa8c0ca1b0 2024-04-25T00:01:33.268 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-04-25T00:01:33.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.266+0000 7faa93971640 1 -- 172.21.15.116:0/2906253409 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faa700678c0 msgr2=0x7faa70069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:33.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.266+0000 7faa93971640 1 --2- 172.21.15.116:0/2906253409 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faa700678c0 0x7faa70069d80 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7faa7c009840 tx=0x7faa7c008040 comp rx=0 tx=0).stop 2024-04-25T00:01:33.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.266+0000 7faa93971640 1 -- 172.21.15.116:0/2906253409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa8c0ca1b0 msgr2=0x7faa8c1583c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:33.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.266+0000 7faa93971640 1 --2- 172.21.15.116:0/2906253409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa8c0ca1b0 0x7faa8c1583c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7faa880ab040 tx=0x7faa8801a7c0 comp rx=0 tx=0).stop 2024-04-25T00:01:33.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.266+0000 7faa93971640 1 -- 172.21.15.116:0/2906253409 shutdown_connections 2024-04-25T00:01:33.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.266+0000 7faa93971640 1 --2- 172.21.15.116:0/2906253409 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faa700678c0 0x7faa70069d80 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:33.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.266+0000 7faa93971640 1 --2- 172.21.15.116:0/2906253409 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa8c153130 0x7faa8c0c1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:33.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.266+0000 7faa93971640 1 --2- 172.21.15.116:0/2906253409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa8c0ca1b0 0x7faa8c1583c0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:33.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.266+0000 7faa93971640 1 --2- 172.21.15.116:0/2906253409 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa8c0c9260 0x7faa8c14c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:33.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.266+0000 7faa93971640 1 -- 172.21.15.116:0/2906253409 >> 172.21.15.116:0/2906253409 conn(0x7faa8c0cb310 msgr2=0x7faa8c0c9680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:33.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.266+0000 7faa93971640 1 -- 172.21.15.116:0/2906253409 shutdown_connections 2024-04-25T00:01:33.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.266+0000 7faa93971640 1 -- 172.21.15.116:0/2906253409 wait complete. 2024-04-25T00:01:33.285 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_data pg_num 2024-04-25T00:01:33.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.410+0000 7f8b8e8e3640 1 Processor -- start 2024-04-25T00:01:33.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.410+0000 7f8b8e8e3640 1 -- start start 2024-04-25T00:01:33.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.410+0000 7f8b8e8e3640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8b8814db70 0x7f8b880aa740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:33.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.410+0000 7f8b8e8e3640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8b8806b2a0 0x7f8b8806b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:33.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.410+0000 7f8b8e8e3640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b88156d70 con 0x7f8b8806c6a0 2024-04-25T00:01:33.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.410+0000 7f8b8e8e3640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f8b88157940 con 0x7f8b8814db70 2024-04-25T00:01:33.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.410+0000 7f8b8e8e3640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f8b88157010 con 0x7f8b8806b2a0 2024-04-25T00:01:33.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.410+0000 7f8b87fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8b8814db70 0x7f8b880aa740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:33.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.410+0000 7f8b877fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f8b8806c6a0 0x7f8b880ac330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:41836/0 (socket says 172.21.15.116:41836) 2024-04-25T00:01:33.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.410+0000 7f8b877fe640 1 -- 172.21.15.116:0/3337687380 learned_addr learned my addr 172.21.15.116:0/3337687380 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:33.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.410+0000 7f8b8ce80640 1 --2- 172.21.15.116:0/3337687380 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8b8806b2a0 0x7f8b8806b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:33.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.410+0000 7f8b87fff640 1 -- 172.21.15.116:0/3337687380 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8b8806b2a0 msgr2=0x7f8b8806b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:33.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.410+0000 7f8b87fff640 1 --2- 172.21.15.116:0/3337687380 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8b8806b2a0 0x7f8b8806b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:33.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.410+0000 7f8b87fff640 1 -- 172.21.15.116:0/3337687380 >> v1:172.21.15.116:6789/0 conn(0x7f8b8806c6a0 legacy=0x7f8b880ac330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:33.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.410+0000 7f8b87fff640 1 -- 172.21.15.116:0/3337687380 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b881572f0 con 0x7f8b8814db70 2024-04-25T00:01:33.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.410+0000 7f8b86ffd640 1 -- 172.21.15.116:0/3337687380 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2973247928 0 0) 0x7f8b88156d70 con 0x7f8b8806c6a0 2024-04-25T00:01:33.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.414+0000 7f8b87fff640 1 --2- 172.21.15.116:0/3337687380 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b8814db70 0x7f8b880aa740 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f8b88157ac0 tx=0x7f8b8001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acd8e7e6f560fc43 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:33.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.414+0000 7f8b86ffd640 1 -- 172.21.15.116:0/3337687380 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b800b0070 con 0x7f8b8814db70 2024-04-25T00:01:33.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.414+0000 7f8b86ffd640 1 -- 172.21.15.116:0/3337687380 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b800ab600 con 0x7f8b8814db70 2024-04-25T00:01:33.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.414+0000 7f8b86ffd640 1 -- 172.21.15.116:0/3337687380 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b800ab9d0 con 0x7f8b8814db70 2024-04-25T00:01:33.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.414+0000 7f8b8e8e3640 1 -- 172.21.15.116:0/3337687380 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b8814db70 msgr2=0x7f8b880aa740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:33.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.414+0000 7f8b8e8e3640 1 --2- 172.21.15.116:0/3337687380 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b8814db70 0x7f8b880aa740 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f8b88157ac0 tx=0x7f8b8001c3e0 comp rx=0 tx=0).stop 2024-04-25T00:01:33.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.414+0000 7f8b8e8e3640 1 -- 172.21.15.116:0/3337687380 shutdown_connections 2024-04-25T00:01:33.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.414+0000 7f8b8e8e3640 1 --2- 172.21.15.116:0/3337687380 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8b8806b2a0 0x7f8b8806b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:33.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.414+0000 7f8b8e8e3640 1 --2- 172.21.15.116:0/3337687380 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b8814db70 0x7f8b880aa740 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:33.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.414+0000 7f8b8e8e3640 1 -- 172.21.15.116:0/3337687380 >> 172.21.15.116:0/3337687380 conn(0x7f8b880c7a90 msgr2=0x7f8b880c7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:33.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.414+0000 7f8b8e8e3640 1 -- 172.21.15.116:0/3337687380 shutdown_connections 2024-04-25T00:01:33.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.414+0000 7f8b8e8e3640 1 -- 172.21.15.116:0/3337687380 wait complete. 2024-04-25T00:01:33.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.418+0000 7f8b8e8e3640 1 Processor -- start 2024-04-25T00:01:33.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.418+0000 7f8b8e8e3640 1 -- start start 2024-04-25T00:01:33.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.418+0000 7f8b8e8e3640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8b8806b2a0 0x7f8b88150360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:33.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.418+0000 7f8b8e8e3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b8814db70 0x7f8b88149440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:33.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.418+0000 7f8b8e8e3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8b880c7060 0x7f8b88149980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:33.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.418+0000 7f8b8e8e3640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8b880aae90 con 0x7f8b8814db70 2024-04-25T00:01:33.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.418+0000 7f8b8e8e3640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8b880acc50 con 0x7f8b880c7060 2024-04-25T00:01:33.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.418+0000 7f8b8e8e3640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f8b881607f0 con 0x7f8b8806b2a0 2024-04-25T00:01:33.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.418+0000 7f8b877fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b8814db70 0x7f8b88149440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:33.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.418+0000 7f8b877fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b8814db70 0x7f8b88149440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56702/0 (socket says 172.21.15.116:56702) 2024-04-25T00:01:33.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.418+0000 7f8b877fe640 1 -- 172.21.15.116:0/3998526608 learned_addr learned my addr 172.21.15.116:0/3998526608 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:33.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.418+0000 7f8b87fff640 1 --2- 172.21.15.116:0/3998526608 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8b8806b2a0 0x7f8b88150360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:33.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.418+0000 7f8b877fe640 1 -- 172.21.15.116:0/3998526608 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8b8806b2a0 msgr2=0x7f8b88150360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:33.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.418+0000 7f8b877fe640 1 --2- 172.21.15.116:0/3998526608 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8b8806b2a0 0x7f8b88150360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:33.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.418+0000 7f8b877fe640 1 -- 172.21.15.116:0/3998526608 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8b880c7060 msgr2=0x7f8b88149980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:33.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.418+0000 7f8b8ce80640 1 --2- 172.21.15.116:0/3998526608 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8b880c7060 0x7f8b88149980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:33.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.418+0000 7f8b877fe640 1 --2- 172.21.15.116:0/3998526608 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8b880c7060 0x7f8b88149980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:33.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.418+0000 7f8b877fe640 1 -- 172.21.15.116:0/3998526608 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b88147cb0 con 0x7f8b8814db70 2024-04-25T00:01:33.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.418+0000 7f8b877fe640 1 --2- 172.21.15.116:0/3998526608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b8814db70 0x7f8b88149440 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f8b7000c550 tx=0x7f8b7000ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:33.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.418+0000 7f8b84ff9640 1 -- 172.21.15.116:0/3998526608 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b70017070 con 0x7f8b8814db70 2024-04-25T00:01:33.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.418+0000 7f8b84ff9640 1 -- 172.21.15.116:0/3998526608 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b70005d10 con 0x7f8b8814db70 2024-04-25T00:01:33.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.418+0000 7f8b84ff9640 1 -- 172.21.15.116:0/3998526608 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b70007330 con 0x7f8b8814db70 2024-04-25T00:01:33.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.418+0000 7f8b8e8e3640 1 -- 172.21.15.116:0/3998526608 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b881eefd0 con 0x7f8b8814db70 2024-04-25T00:01:33.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.418+0000 7f8b8e8e3640 1 -- 172.21.15.116:0/3998526608 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b88150e00 con 0x7f8b8814db70 2024-04-25T00:01:33.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.418+0000 7f8b8e8e3640 1 -- 172.21.15.116:0/3998526608 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b8804f510 con 0x7f8b8814db70 2024-04-25T00:01:33.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.418+0000 7f8b84ff9640 1 -- 172.21.15.116:0/3998526608 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8b7001a070 con 0x7f8b8814db70 2024-04-25T00:01:33.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.422+0000 7f8b84ff9640 1 --2- 172.21.15.116:0/3998526608 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8b680678c0 0x7f8b68069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:33.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.422+0000 7f8b84ff9640 1 -- 172.21.15.116:0/3998526608 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7f8b700853e0 con 0x7f8b8814db70 2024-04-25T00:01:33.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.422+0000 7f8b87fff640 1 --2- 172.21.15.116:0/3998526608 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8b680678c0 0x7f8b68069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:33.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.422+0000 7f8b87fff640 1 --2- 172.21.15.116:0/3998526608 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8b680678c0 0x7f8b68069d80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8b800ad000 tx=0x7f8b800a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:33.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.426+0000 7f8b84ff9640 1 -- 172.21.15.116:0/3998526608 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f8b70055c80 con 0x7f8b8814db70 2024-04-25T00:01:33.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.586+0000 7f8b8e8e3640 1 -- 172.21.15.116:0/3998526608 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"} v 0) -- 0x7f8b8806c3d0 con 0x7f8b8814db70 2024-04-25T00:01:33.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.586+0000 7f8b84ff9640 1 -- 172.21.15.116:0/3998526608 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"}]=0 v27) ==== 95+0+11 (secure 0 0 0) 0x7f8b70055580 con 0x7f8b8814db70 2024-04-25T00:01:33.587 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-04-25T00:01:33.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.586+0000 7f8b8e8e3640 1 -- 172.21.15.116:0/3998526608 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8b680678c0 msgr2=0x7f8b68069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:33.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.586+0000 7f8b8e8e3640 1 --2- 172.21.15.116:0/3998526608 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8b680678c0 0x7f8b68069d80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8b800ad000 tx=0x7f8b800a6000 comp rx=0 tx=0).stop 2024-04-25T00:01:33.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.586+0000 7f8b8e8e3640 1 -- 172.21.15.116:0/3998526608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b8814db70 msgr2=0x7f8b88149440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:33.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.586+0000 7f8b8e8e3640 1 --2- 172.21.15.116:0/3998526608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b8814db70 0x7f8b88149440 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f8b7000c550 tx=0x7f8b7000ca10 comp rx=0 tx=0).stop 2024-04-25T00:01:33.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.586+0000 7f8b8e8e3640 1 -- 172.21.15.116:0/3998526608 shutdown_connections 2024-04-25T00:01:33.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.586+0000 7f8b8e8e3640 1 --2- 172.21.15.116:0/3998526608 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8b680678c0 0x7f8b68069d80 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:33.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.586+0000 7f8b8e8e3640 1 --2- 172.21.15.116:0/3998526608 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8b880c7060 0x7f8b88149980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:33.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.586+0000 7f8b8e8e3640 1 --2- 172.21.15.116:0/3998526608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b8814db70 0x7f8b88149440 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:33.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.586+0000 7f8b8e8e3640 1 --2- 172.21.15.116:0/3998526608 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8b8806b2a0 0x7f8b88150360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:33.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.586+0000 7f8b8e8e3640 1 -- 172.21.15.116:0/3998526608 >> 172.21.15.116:0/3998526608 conn(0x7f8b880c7a90 msgr2=0x7f8b880c1f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:33.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.586+0000 7f8b8e8e3640 1 -- 172.21.15.116:0/3998526608 shutdown_connections 2024-04-25T00:01:33.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.586+0000 7f8b8e8e3640 1 -- 172.21.15.116:0/3998526608 wait complete. 2024-04-25T00:01:33.606 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:01:33.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.718+0000 7faf46c46640 1 Processor -- start 2024-04-25T00:01:33.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.718+0000 7faf46c46640 1 -- start start 2024-04-25T00:01:33.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.722+0000 7faf46c46640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faf40153f40 0x7faf4006e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:33.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.722+0000 7faf46c46640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7faf400c2e70 0x7faf4015a610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:33.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.722+0000 7faf46c46640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf40071a30 con 0x7faf400c2a10 2024-04-25T00:01:33.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.722+0000 7faf46c46640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7faf400725d0 con 0x7faf400c2e70 2024-04-25T00:01:33.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.722+0000 7faf46c46640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7faf40071ca0 con 0x7faf40153f40 2024-04-25T00:01:33.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.722+0000 7faf451e3640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7faf400c2e70 0x7faf4015a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:33.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.722+0000 7faf3ffff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7faf400c2a10 0x7faf40159f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:41850/0 (socket says 172.21.15.116:41850) 2024-04-25T00:01:33.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.722+0000 7faf3ffff640 1 -- 172.21.15.116:0/3913650944 learned_addr learned my addr 172.21.15.116:0/3913650944 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:33.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.722+0000 7faf451e3640 1 -- 172.21.15.116:0/3913650944 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faf40153f40 msgr2=0x7faf4006e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:33.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.722+0000 7faf451e3640 1 --2- 172.21.15.116:0/3913650944 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faf40153f40 0x7faf4006e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:33.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.722+0000 7faf451e3640 1 -- 172.21.15.116:0/3913650944 >> v1:172.21.15.116:6789/0 conn(0x7faf400c2a10 legacy=0x7faf40159f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:33.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.722+0000 7faf451e3640 1 -- 172.21.15.116:0/3913650944 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf40071f80 con 0x7faf400c2e70 2024-04-25T00:01:33.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.722+0000 7faf3f7fe640 1 -- 172.21.15.116:0/3913650944 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 599599852 0 0) 0x7faf40071a30 con 0x7faf400c2a10 2024-04-25T00:01:33.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.722+0000 7faf451e3640 1 --2- 172.21.15.116:0/3913650944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf400c2e70 0x7faf4015a610 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7faf38009d90 tx=0x7faf3801c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dba2c6c5c8cbd02 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:33.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.722+0000 7faf3f7fe640 1 -- 172.21.15.116:0/3913650944 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf380b2070 con 0x7faf400c2e70 2024-04-25T00:01:33.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.722+0000 7faf3f7fe640 1 -- 172.21.15.116:0/3913650944 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf380a58b0 con 0x7faf400c2e70 2024-04-25T00:01:33.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.722+0000 7faf3f7fe640 1 -- 172.21.15.116:0/3913650944 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf380a5c80 con 0x7faf400c2e70 2024-04-25T00:01:33.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.722+0000 7faf46c46640 1 -- 172.21.15.116:0/3913650944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf400c2e70 msgr2=0x7faf4015a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:33.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.722+0000 7faf46c46640 1 --2- 172.21.15.116:0/3913650944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf400c2e70 0x7faf4015a610 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7faf38009d90 tx=0x7faf3801c8e0 comp rx=0 tx=0).stop 2024-04-25T00:01:33.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.722+0000 7faf46c46640 1 -- 172.21.15.116:0/3913650944 shutdown_connections 2024-04-25T00:01:33.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.722+0000 7faf46c46640 1 --2- 172.21.15.116:0/3913650944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf400c2e70 0x7faf4015a610 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:33.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.722+0000 7faf46c46640 1 --2- 172.21.15.116:0/3913650944 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faf40153f40 0x7faf4006e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:33.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.722+0000 7faf46c46640 1 -- 172.21.15.116:0/3913650944 >> 172.21.15.116:0/3913650944 conn(0x7faf400c3b00 msgr2=0x7faf400c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:33.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.722+0000 7faf46c46640 1 -- 172.21.15.116:0/3913650944 shutdown_connections 2024-04-25T00:01:33.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.722+0000 7faf46c46640 1 -- 172.21.15.116:0/3913650944 wait complete. 2024-04-25T00:01:33.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.726+0000 7faf46c46640 1 Processor -- start 2024-04-25T00:01:33.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.726+0000 7faf46c46640 1 -- start start 2024-04-25T00:01:33.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.726+0000 7faf46c46640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf400c2a10 0x7faf4014f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:33.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.726+0000 7faf46c46640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf400c2e70 0x7faf40149980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:33.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.726+0000 7faf46c46640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faf40153f40 0x7faf40159470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:33.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.726+0000 7faf46c46640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7faf400c36c0 con 0x7faf400c2a10 2024-04-25T00:01:33.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.726+0000 7faf46c46640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7faf4006eee0 con 0x7faf400c2e70 2024-04-25T00:01:33.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.726+0000 7faf46c46640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7faf400c3840 con 0x7faf40153f40 2024-04-25T00:01:33.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.726+0000 7faf449e2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf400c2a10 0x7faf4014f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:33.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.726+0000 7faf449e2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf400c2a10 0x7faf4014f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56716/0 (socket says 172.21.15.116:56716) 2024-04-25T00:01:33.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.726+0000 7faf449e2640 1 -- 172.21.15.116:0/2551978093 learned_addr learned my addr 172.21.15.116:0/2551978093 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:33.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.726+0000 7faf451e3640 1 --2- 172.21.15.116:0/2551978093 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faf40153f40 0x7faf40159470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:33.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.726+0000 7faf449e2640 1 -- 172.21.15.116:0/2551978093 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faf40153f40 msgr2=0x7faf40159470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:33.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.726+0000 7faf449e2640 1 --2- 172.21.15.116:0/2551978093 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faf40153f40 0x7faf40159470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:33.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.726+0000 7faf449e2640 1 -- 172.21.15.116:0/2551978093 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf400c2e70 msgr2=0x7faf40149980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:33.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.726+0000 7faf449e2640 1 --2- 172.21.15.116:0/2551978093 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf400c2e70 0x7faf40149980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:33.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.726+0000 7faf449e2640 1 -- 172.21.15.116:0/2551978093 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf40158cb0 con 0x7faf400c2a10 2024-04-25T00:01:33.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.726+0000 7faf449e2640 1 --2- 172.21.15.116:0/2551978093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf400c2a10 0x7faf4014f430 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7faf3400b440 tx=0x7faf3400b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:33.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.726+0000 7faf3d7fa640 1 -- 172.21.15.116:0/2551978093 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf34015020 con 0x7faf400c2a10 2024-04-25T00:01:33.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.726+0000 7faf3d7fa640 1 -- 172.21.15.116:0/2551978093 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf340118b0 con 0x7faf400c2a10 2024-04-25T00:01:33.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.726+0000 7faf46c46640 1 -- 172.21.15.116:0/2551978093 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf401f6900 con 0x7faf400c2a10 2024-04-25T00:01:33.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.726+0000 7faf3d7fa640 1 -- 172.21.15.116:0/2551978093 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf34011ca0 con 0x7faf400c2a10 2024-04-25T00:01:33.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.726+0000 7faf46c46640 1 -- 172.21.15.116:0/2551978093 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf401f6be0 con 0x7faf400c2a10 2024-04-25T00:01:33.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.726+0000 7faf46c46640 1 -- 172.21.15.116:0/2551978093 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf40068350 con 0x7faf400c2a10 2024-04-25T00:01:33.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.726+0000 7faf3d7fa640 1 -- 172.21.15.116:0/2551978093 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7faf34018020 con 0x7faf400c2a10 2024-04-25T00:01:33.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.730+0000 7faf3d7fa640 1 --2- 172.21.15.116:0/2551978093 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faf2c0678c0 0x7faf2c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:33.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.730+0000 7faf3d7fa640 1 -- 172.21.15.116:0/2551978093 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7faf340860a0 con 0x7faf400c2a10 2024-04-25T00:01:33.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.730+0000 7faf3ffff640 1 --2- 172.21.15.116:0/2551978093 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faf2c0678c0 0x7faf2c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:33.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.730+0000 7faf3ffff640 1 --2- 172.21.15.116:0/2551978093 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faf2c0678c0 0x7faf2c069d80 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7faf300011f0 tx=0x7faf30007930 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:33.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.734+0000 7faf3d7fa640 1 -- 172.21.15.116:0/2551978093 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7faf3401f020 con 0x7faf400c2a10 2024-04-25T00:01:33.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.930+0000 7faf46c46640 1 -- 172.21.15.116:0/2551978093 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faf400c16e0 con 0x7faf400c2a10 2024-04-25T00:01:33.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.930+0000 7faf3d7fa640 1 -- 172.21.15.116:0/2551978093 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6492 (secure 0 0 0) 0x7faf3400d040 con 0x7faf400c2a10 2024-04-25T00:01:33.932 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:01:33.932 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6834/1459177921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1459177921},{"type":"v1","addr":"172.21.15.116:6834","nonce":1459177921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:01:08.748957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6837/1762356055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1762356055},{"type":"v1","addr":"172.21.15.174:6837","nonce":1762356055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270]},"id":1},{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:01:22.888641+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6835/174781672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":174781672},{"type":"v1","addr":"172.21.15.116:6835","nonce":174781672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4245,"qdb_cluster":[4245]},"id":2}]} 2024-04-25T00:01:33.932 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 14 2024-04-25T00:01:33.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.930+0000 7faf46c46640 1 -- 172.21.15.116:0/2551978093 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faf2c0678c0 msgr2=0x7faf2c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:33.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.930+0000 7faf46c46640 1 --2- 172.21.15.116:0/2551978093 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faf2c0678c0 0x7faf2c069d80 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7faf300011f0 tx=0x7faf30007930 comp rx=0 tx=0).stop 2024-04-25T00:01:33.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.930+0000 7faf46c46640 1 -- 172.21.15.116:0/2551978093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf400c2a10 msgr2=0x7faf4014f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:33.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.930+0000 7faf46c46640 1 --2- 172.21.15.116:0/2551978093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf400c2a10 0x7faf4014f430 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7faf3400b440 tx=0x7faf3400b900 comp rx=0 tx=0).stop 2024-04-25T00:01:33.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.934+0000 7faf46c46640 1 -- 172.21.15.116:0/2551978093 shutdown_connections 2024-04-25T00:01:33.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.934+0000 7faf46c46640 1 --2- 172.21.15.116:0/2551978093 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faf2c0678c0 0x7faf2c069d80 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:33.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.934+0000 7faf46c46640 1 --2- 172.21.15.116:0/2551978093 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faf40153f40 0x7faf40159470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:33.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.934+0000 7faf46c46640 1 --2- 172.21.15.116:0/2551978093 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf400c2e70 0x7faf40149980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:33.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.934+0000 7faf46c46640 1 --2- 172.21.15.116:0/2551978093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf400c2a10 0x7faf4014f430 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:33.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.934+0000 7faf46c46640 1 -- 172.21.15.116:0/2551978093 >> 172.21.15.116:0/2551978093 conn(0x7faf400c3b00 msgr2=0x7faf4014d910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:33.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.934+0000 7faf46c46640 1 -- 172.21.15.116:0/2551978093 shutdown_connections 2024-04-25T00:01:33.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:33.934+0000 7faf46c46640 1 -- 172.21.15.116:0/2551978093 wait complete. 2024-04-25T00:01:33.950 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T00:01:34.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.058+0000 7fdefbfea640 1 Processor -- start 2024-04-25T00:01:34.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.058+0000 7fdefbfea640 1 -- start start 2024-04-25T00:01:34.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.058+0000 7fdefbfea640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdef4152d80 0x7fdef406bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:34.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.058+0000 7fdefbfea640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fdef406d5f0 0x7fdef4156c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:34.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.058+0000 7fdefbfea640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdef40c9150 con 0x7fdef406c290 2024-04-25T00:01:34.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.058+0000 7fdefbfea640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fdef40c9d60 con 0x7fdef406d5f0 2024-04-25T00:01:34.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.058+0000 7fdefbfea640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fdef40c9430 con 0x7fdef4152d80 2024-04-25T00:01:34.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.058+0000 7fdef9585640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fdef406d5f0 0x7fdef4156c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:34.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.058+0000 7fdefa587640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fdef406c290 0x7fdef41571c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:41866/0 (socket says 172.21.15.116:41866) 2024-04-25T00:01:34.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.058+0000 7fdefa587640 1 -- 172.21.15.116:0/3296468863 learned_addr learned my addr 172.21.15.116:0/3296468863 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:34.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.058+0000 7fdef9d86640 1 --2- 172.21.15.116:0/3296468863 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdef4152d80 0x7fdef406bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:34.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.058+0000 7fdef9585640 1 -- 172.21.15.116:0/3296468863 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdef4152d80 msgr2=0x7fdef406bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:34.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.058+0000 7fdef9585640 1 --2- 172.21.15.116:0/3296468863 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdef4152d80 0x7fdef406bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:34.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.058+0000 7fdef9585640 1 -- 172.21.15.116:0/3296468863 >> v1:172.21.15.116:6789/0 conn(0x7fdef406c290 legacy=0x7fdef41571c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:34.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.058+0000 7fdef8d84640 1 -- 172.21.15.116:0/3296468863 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2352874206 0 0) 0x7fdef40c9150 con 0x7fdef406c290 2024-04-25T00:01:34.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.058+0000 7fdef9585640 1 -- 172.21.15.116:0/3296468863 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdef40c9710 con 0x7fdef406d5f0 2024-04-25T00:01:34.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.062+0000 7fdef9585640 1 --2- 172.21.15.116:0/3296468863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdef406d5f0 0x7fdef4156c80 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fdef00097c0 tx=0x7fdef001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29637fd3612cdb64 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:34.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.062+0000 7fdef8d84640 1 -- 172.21.15.116:0/3296468863 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef00ab600 con 0x7fdef406d5f0 2024-04-25T00:01:34.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.062+0000 7fdef8d84640 1 -- 172.21.15.116:0/3296468863 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdef00b0070 con 0x7fdef406d5f0 2024-04-25T00:01:34.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.062+0000 7fdef8d84640 1 -- 172.21.15.116:0/3296468863 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdef00ab9c0 con 0x7fdef406d5f0 2024-04-25T00:01:34.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.062+0000 7fdefbfea640 1 -- 172.21.15.116:0/3296468863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdef406d5f0 msgr2=0x7fdef4156c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:34.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.062+0000 7fdefbfea640 1 --2- 172.21.15.116:0/3296468863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdef406d5f0 0x7fdef4156c80 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fdef00097c0 tx=0x7fdef001c3e0 comp rx=0 tx=0).stop 2024-04-25T00:01:34.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.062+0000 7fdefbfea640 1 -- 172.21.15.116:0/3296468863 shutdown_connections 2024-04-25T00:01:34.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.062+0000 7fdefbfea640 1 --2- 172.21.15.116:0/3296468863 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdef406d5f0 0x7fdef4156c80 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:34.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.062+0000 7fdefbfea640 1 --2- 172.21.15.116:0/3296468863 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdef4152d80 0x7fdef406bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:34.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.062+0000 7fdefbfea640 1 -- 172.21.15.116:0/3296468863 >> 172.21.15.116:0/3296468863 conn(0x7fdef40acdd0 msgr2=0x7fdef40ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:34.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.062+0000 7fdefbfea640 1 -- 172.21.15.116:0/3296468863 shutdown_connections 2024-04-25T00:01:34.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.062+0000 7fdefbfea640 1 -- 172.21.15.116:0/3296468863 wait complete. 2024-04-25T00:01:34.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.066+0000 7fdefbfea640 1 Processor -- start 2024-04-25T00:01:34.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.066+0000 7fdefbfea640 1 -- start start 2024-04-25T00:01:34.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.066+0000 7fdefbfea640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdef406c290 0x7fdef4147220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:34.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.066+0000 7fdefbfea640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdef406d5f0 0x7fdef41f1a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:34.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.066+0000 7fdefbfea640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdef4152d80 0x7fdef41f1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:34.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.066+0000 7fdefbfea640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdef40caf00 con 0x7fdef406c290 2024-04-25T00:01:34.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.066+0000 7fdefbfea640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fdef40cad80 con 0x7fdef406d5f0 2024-04-25T00:01:34.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.066+0000 7fdefbfea640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fdef41628d0 con 0x7fdef4152d80 2024-04-25T00:01:34.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.066+0000 7fdef9d86640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdef406c290 0x7fdef4147220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:34.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.066+0000 7fdef9d86640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdef406c290 0x7fdef4147220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56738/0 (socket says 172.21.15.116:56738) 2024-04-25T00:01:34.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.066+0000 7fdef9d86640 1 -- 172.21.15.116:0/2241484573 learned_addr learned my addr 172.21.15.116:0/2241484573 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:34.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.066+0000 7fdef9d86640 1 -- 172.21.15.116:0/2241484573 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdef4152d80 msgr2=0x7fdef41f1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:34.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.066+0000 7fdef9d86640 1 --2- 172.21.15.116:0/2241484573 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdef4152d80 0x7fdef41f1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:34.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.066+0000 7fdef9d86640 1 -- 172.21.15.116:0/2241484573 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdef406d5f0 msgr2=0x7fdef41f1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:34.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.066+0000 7fdef9585640 1 --2- 172.21.15.116:0/2241484573 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdef406d5f0 0x7fdef41f1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:34.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.066+0000 7fdef9d86640 1 --2- 172.21.15.116:0/2241484573 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdef406d5f0 0x7fdef41f1a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:34.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.066+0000 7fdef9d86640 1 -- 172.21.15.116:0/2241484573 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdef40ab480 con 0x7fdef406c290 2024-04-25T00:01:34.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.066+0000 7fdef9d86640 1 --2- 172.21.15.116:0/2241484573 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdef406c290 0x7fdef4147220 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fdee400c7f0 tx=0x7fdee400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:34.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.066+0000 7fdeea7fc640 1 -- 172.21.15.116:0/2241484573 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdee4015070 con 0x7fdef406c290 2024-04-25T00:01:34.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.066+0000 7fdeea7fc640 1 -- 172.21.15.116:0/2241484573 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdee4004240 con 0x7fdef406c290 2024-04-25T00:01:34.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.066+0000 7fdefbfea640 1 -- 172.21.15.116:0/2241484573 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdef41f2500 con 0x7fdef406c290 2024-04-25T00:01:34.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.066+0000 7fdeea7fc640 1 -- 172.21.15.116:0/2241484573 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdee4003620 con 0x7fdef406c290 2024-04-25T00:01:34.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.066+0000 7fdefbfea640 1 -- 172.21.15.116:0/2241484573 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdef4146c90 con 0x7fdef406c290 2024-04-25T00:01:34.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.066+0000 7fdefbfea640 1 -- 172.21.15.116:0/2241484573 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdef4150600 con 0x7fdef406c290 2024-04-25T00:01:34.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.066+0000 7fdeea7fc640 1 -- 172.21.15.116:0/2241484573 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fdee40039d0 con 0x7fdef406c290 2024-04-25T00:01:34.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.070+0000 7fdeea7fc640 1 --2- 172.21.15.116:0/2241484573 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fded80678c0 0x7fded8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:34.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.070+0000 7fdeea7fc640 1 -- 172.21.15.116:0/2241484573 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7fdee4085fc0 con 0x7fdef406c290 2024-04-25T00:01:34.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.070+0000 7fdef9585640 1 --2- 172.21.15.116:0/2241484573 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fded80678c0 0x7fded8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:34.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.070+0000 7fdef9585640 1 --2- 172.21.15.116:0/2241484573 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fded80678c0 0x7fded8069d80 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fdef00af000 tx=0x7fdef00a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:34.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.074+0000 7fdeea7fc640 1 -- 172.21.15.116:0/2241484573 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fdee4056860 con 0x7fdef406c290 2024-04-25T00:01:34.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.218+0000 7fdefbfea640 1 -- 172.21.15.116:0/2241484573 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdef4068350 con 0x7fdef406c290 2024-04-25T00:01:34.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.222+0000 7fdeea7fc640 1 -- 172.21.15.116:0/2241484573 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+20311 (secure 0 0 0) 0x7fdee4056160 con 0x7fdef406c290 2024-04-25T00:01:34.224 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:01:34.224 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":27,"fsid":"6d6a1aa7-0478-4c83-8057-05917e81f046","created":"2024-04-25T00:00:40.682493+0000","modified":"2024-04-25T00:01:20.501508+0000","last_up_change":"2024-04-25T00:00:51.822174+0000","last_in_change":"2024-04-25T00:00:42.980473+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T00:00:53.324376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-04-25T00:01:03.197326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-04-25T00:01:03.728618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-04-25T00:01:17.125210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-04-25T00:01:17.821873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8b6ddcb2-1ad9-4167-961c-bd6936bb9060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6809","nonce":3751765682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6811","nonce":3751765682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6815","nonce":3751765682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6813","nonce":3751765682}]},"public_addr":"172.21.15.116:6809/3751765682","cluster_addr":"172.21.15.116:6811/3751765682","heartbeat_back_addr":"172.21.15.116:6815/3751765682","heartbeat_front_addr":"172.21.15.116:6813/3751765682","state":["exists","up"]},{"osd":1,"uuid":"a6431768-994f-41da-8853-886f5687fb91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6801","nonce":377879124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6803","nonce":377879124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6807","nonce":377879124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6805","nonce":377879124}]},"public_addr":"172.21.15.116:6801/377879124","cluster_addr":"172.21.15.116:6803/377879124","heartbeat_back_addr":"172.21.15.116:6807/377879124","heartbeat_front_addr":"172.21.15.116:6805/377879124","state":["exists","up"]},{"osd":2,"uuid":"4f69bd40-5a23-45ff-8693-e1de75e1aa9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6817","nonce":1099282267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6819","nonce":1099282267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6823","nonce":1099282267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6821","nonce":1099282267}]},"public_addr":"172.21.15.116:6817/1099282267","cluster_addr":"172.21.15.116:6819/1099282267","heartbeat_back_addr":"172.21.15.116:6823/1099282267","heartbeat_front_addr":"172.21.15.116:6821/1099282267","state":["exists","up"]},{"osd":3,"uuid":"6a0d5344-8b6f-40d3-918d-0f0a4be688e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6825","nonce":1179942661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6827","nonce":1179942661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6831","nonce":1179942661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6829","nonce":1179942661}]},"public_addr":"172.21.15.116:6825/1179942661","cluster_addr":"172.21.15.116:6827/1179942661","heartbeat_back_addr":"172.21.15.116:6831/1179942661","heartbeat_front_addr":"172.21.15.116:6829/1179942661","state":["exists","up"]},{"osd":4,"uuid":"132e92e4-b825-441e-965a-ca00533fa86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6801","nonce":3132444330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6803","nonce":3132444330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6807","nonce":3132444330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6805","nonce":3132444330}]},"public_addr":"172.21.15.174:6801/3132444330","cluster_addr":"172.21.15.174:6803/3132444330","heartbeat_back_addr":"172.21.15.174:6807/3132444330","heartbeat_front_addr":"172.21.15.174:6805/3132444330","state":["exists","up"]},{"osd":5,"uuid":"a2eeb5b6-1b31-43cc-8306-a0adf3ee15bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6809","nonce":881817164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6811","nonce":881817164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6815","nonce":881817164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6813","nonce":881817164}]},"public_addr":"172.21.15.174:6809/881817164","cluster_addr":"172.21.15.174:6811/881817164","heartbeat_back_addr":"172.21.15.174:6815/881817164","heartbeat_front_addr":"172.21.15.174:6813/881817164","state":["exists","up"]},{"osd":6,"uuid":"588e873a-c841-4ba7-90f2-8956c87a9983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6825","nonce":117574251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6827","nonce":117574251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6831","nonce":117574251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6829","nonce":117574251}]},"public_addr":"172.21.15.174:6825/117574251","cluster_addr":"172.21.15.174:6827/117574251","heartbeat_back_addr":"172.21.15.174:6831/117574251","heartbeat_front_addr":"172.21.15.174:6829/117574251","state":["exists","up"]},{"osd":7,"uuid":"3b90f9b0-8a0e-4883-b088-ca1d4012870c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6817","nonce":2543899196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6819","nonce":2543899196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6823","nonce":2543899196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6821","nonce":2543899196}]},"public_addr":"172.21.15.174:6817/2543899196","cluster_addr":"172.21.15.174:6819/2543899196","heartbeat_back_addr":"172.21.15.174:6823/2543899196","heartbeat_front_addr":"172.21.15.174:6821/2543899196","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.293769+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.310034+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.404865+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.549658+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.192174+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.297266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.493649+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.452309+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:01:34.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.222+0000 7fdefbfea640 1 -- 172.21.15.116:0/2241484573 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fded80678c0 msgr2=0x7fded8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:34.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.222+0000 7fdefbfea640 1 --2- 172.21.15.116:0/2241484573 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fded80678c0 0x7fded8069d80 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fdef00af000 tx=0x7fdef00a7000 comp rx=0 tx=0).stop 2024-04-25T00:01:34.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.222+0000 7fdefbfea640 1 -- 172.21.15.116:0/2241484573 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdef406c290 msgr2=0x7fdef4147220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:34.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.222+0000 7fdefbfea640 1 --2- 172.21.15.116:0/2241484573 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdef406c290 0x7fdef4147220 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fdee400c7f0 tx=0x7fdee400ccb0 comp rx=0 tx=0).stop 2024-04-25T00:01:34.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.222+0000 7fdefbfea640 1 -- 172.21.15.116:0/2241484573 shutdown_connections 2024-04-25T00:01:34.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.222+0000 7fdefbfea640 1 --2- 172.21.15.116:0/2241484573 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fded80678c0 0x7fded8069d80 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:34.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.222+0000 7fdefbfea640 1 --2- 172.21.15.116:0/2241484573 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdef4152d80 0x7fdef41f1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:34.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.222+0000 7fdefbfea640 1 --2- 172.21.15.116:0/2241484573 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdef406d5f0 0x7fdef41f1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:34.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.222+0000 7fdefbfea640 1 --2- 172.21.15.116:0/2241484573 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdef406c290 0x7fdef4147220 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:34.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.222+0000 7fdefbfea640 1 -- 172.21.15.116:0/2241484573 >> 172.21.15.116:0/2241484573 conn(0x7fdef40acdd0 msgr2=0x7fdef415f570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:34.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.222+0000 7fdefbfea640 1 -- 172.21.15.116:0/2241484573 shutdown_connections 2024-04-25T00:01:34.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.222+0000 7fdefbfea640 1 -- 172.21.15.116:0/2241484573 wait complete. 2024-04-25T00:01:34.241 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:01:34.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.358+0000 7f13b587c640 1 Processor -- start 2024-04-25T00:01:34.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.358+0000 7f13b587c640 1 -- start start 2024-04-25T00:01:34.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.358+0000 7f13b587c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f13b00acef0 0x7f13b01541d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:34.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.358+0000 7f13b587c640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f13b00ad2d0 0x7f13b00a7d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:34.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.358+0000 7f13b587c640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13b00a9ef0 con 0x7f13b00a82c0 2024-04-25T00:01:34.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.358+0000 7f13b587c640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f13b00c2b60 con 0x7f13b00acef0 2024-04-25T00:01:34.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.358+0000 7f13b587c640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f13b00c2e40 con 0x7f13b00ad2d0 2024-04-25T00:01:34.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.358+0000 7f13aeffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f13b00acef0 0x7f13b01541d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:34.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.358+0000 7f13af7fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f13b00a82c0 0x7f13b00c32d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:41882/0 (socket says 172.21.15.116:41882) 2024-04-25T00:01:34.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.358+0000 7f13af7fe640 1 -- 172.21.15.116:0/117150373 learned_addr learned my addr 172.21.15.116:0/117150373 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:34.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.358+0000 7f13aeffd640 1 -- 172.21.15.116:0/117150373 >> v2:172.21.15.174:3300/0 conn(0x7f13b00ad2d0 msgr2=0x7f13b00a7d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:34.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.358+0000 7f13aeffd640 1 --2- 172.21.15.116:0/117150373 >> v2:172.21.15.174:3300/0 conn(0x7f13b00ad2d0 0x7f13b00a7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:34.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.358+0000 7f13aeffd640 1 -- 172.21.15.116:0/117150373 >> v1:172.21.15.116:6789/0 conn(0x7f13b00a82c0 legacy=0x7f13b00c32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:34.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.358+0000 7f13aeffd640 1 -- 172.21.15.116:0/117150373 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13b00aa1d0 con 0x7f13b00acef0 2024-04-25T00:01:34.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.362+0000 7f13aeffd640 1 --2- 172.21.15.116:0/117150373 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b00acef0 0x7f13b01541d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f13b00748c0 tx=0x7f13a801c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93efc3bc747a6d47 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:34.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.362+0000 7f13adffb640 1 -- 172.21.15.116:0/117150373 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a80b2070 con 0x7f13b00acef0 2024-04-25T00:01:34.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.362+0000 7f13adffb640 1 -- 172.21.15.116:0/117150373 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13a80a5760 con 0x7f13b00acef0 2024-04-25T00:01:34.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.362+0000 7f13adffb640 1 -- 172.21.15.116:0/117150373 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a80a5b30 con 0x7f13b00acef0 2024-04-25T00:01:34.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.362+0000 7f13b587c640 1 -- 172.21.15.116:0/117150373 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b00acef0 msgr2=0x7f13b01541d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:34.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.362+0000 7f13b587c640 1 --2- 172.21.15.116:0/117150373 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b00acef0 0x7f13b01541d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f13b00748c0 tx=0x7f13a801c8a0 comp rx=0 tx=0).stop 2024-04-25T00:01:34.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.362+0000 7f13b587c640 1 -- 172.21.15.116:0/117150373 shutdown_connections 2024-04-25T00:01:34.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.362+0000 7f13b587c640 1 --2- 172.21.15.116:0/117150373 >> v2:172.21.15.174:3300/0 conn(0x7f13b00ad2d0 0x7f13b00a7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:34.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.362+0000 7f13b587c640 1 --2- 172.21.15.116:0/117150373 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b00acef0 0x7f13b01541d0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:34.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.362+0000 7f13b587c640 1 -- 172.21.15.116:0/117150373 >> 172.21.15.116:0/117150373 conn(0x7f13b00c4200 msgr2=0x7f13b00c4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:34.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.362+0000 7f13b587c640 1 -- 172.21.15.116:0/117150373 shutdown_connections 2024-04-25T00:01:34.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.362+0000 7f13b587c640 1 -- 172.21.15.116:0/117150373 wait complete. 2024-04-25T00:01:34.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.362+0000 7f13b587c640 1 Processor -- start 2024-04-25T00:01:34.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.362+0000 7f13b587c640 1 -- start start 2024-04-25T00:01:34.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.362+0000 7f13b587c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f13b00a82c0 0x7f13b0159350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:34.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.362+0000 7f13b587c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b00acef0 0x7f13b0156f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:34.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.362+0000 7f13b587c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f13b00ad2d0 0x7f13b014ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:34.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.362+0000 7f13b587c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f13b01609e0 con 0x7f13b00acef0 2024-04-25T00:01:34.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.362+0000 7f13b587c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f13b0160860 con 0x7f13b00a82c0 2024-04-25T00:01:34.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.362+0000 7f13b587c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f13b0160b60 con 0x7f13b00ad2d0 2024-04-25T00:01:34.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.362+0000 7f13ae7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b00acef0 0x7f13b0156f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:34.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.362+0000 7f13ae7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b00acef0 0x7f13b0156f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56768/0 (socket says 172.21.15.116:56768) 2024-04-25T00:01:34.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.362+0000 7f13ae7fc640 1 -- 172.21.15.116:0/1771078468 learned_addr learned my addr 172.21.15.116:0/1771078468 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:34.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.362+0000 7f13aeffd640 1 --2- 172.21.15.116:0/1771078468 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f13b00a82c0 0x7f13b0159350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:34.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.362+0000 7f13ae7fc640 1 -- 172.21.15.116:0/1771078468 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f13b00ad2d0 msgr2=0x7f13b014ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:34.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.362+0000 7f13af7fe640 1 --2- 172.21.15.116:0/1771078468 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f13b00ad2d0 0x7f13b014ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:34.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.362+0000 7f13ae7fc640 1 --2- 172.21.15.116:0/1771078468 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f13b00ad2d0 0x7f13b014ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:34.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.362+0000 7f13ae7fc640 1 -- 172.21.15.116:0/1771078468 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f13b00a82c0 msgr2=0x7f13b0159350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:34.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.362+0000 7f13ae7fc640 1 --2- 172.21.15.116:0/1771078468 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f13b00a82c0 0x7f13b0159350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:34.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.362+0000 7f13ae7fc640 1 -- 172.21.15.116:0/1771078468 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13b01ed1e0 con 0x7f13b00acef0 2024-04-25T00:01:34.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.362+0000 7f13aeffd640 1 --2- 172.21.15.116:0/1771078468 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f13b00a82c0 0x7f13b0159350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:34.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.362+0000 7f13ae7fc640 1 --2- 172.21.15.116:0/1771078468 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b00acef0 0x7f13b0156f40 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f139800b440 tx=0x7f139800b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:34.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.366+0000 7f13a7fff640 1 -- 172.21.15.116:0/1771078468 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1398015020 con 0x7f13b00acef0 2024-04-25T00:01:34.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.366+0000 7f13a7fff640 1 -- 172.21.15.116:0/1771078468 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13980118b0 con 0x7f13b00acef0 2024-04-25T00:01:34.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.366+0000 7f13b587c640 1 -- 172.21.15.116:0/1771078468 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13b01ec660 con 0x7f13b00acef0 2024-04-25T00:01:34.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.366+0000 7f13a7fff640 1 -- 172.21.15.116:0/1771078468 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1398011ca0 con 0x7f13b00acef0 2024-04-25T00:01:34.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.366+0000 7f13b587c640 1 -- 172.21.15.116:0/1771078468 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13b01ecc20 con 0x7f13b00acef0 2024-04-25T00:01:34.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.366+0000 7f13b587c640 1 -- 172.21.15.116:0/1771078468 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13b0068a60 con 0x7f13b00acef0 2024-04-25T00:01:34.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.366+0000 7f13a7fff640 1 -- 172.21.15.116:0/1771078468 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1398018020 con 0x7f13b00acef0 2024-04-25T00:01:34.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.366+0000 7f13a7fff640 1 --2- 172.21.15.116:0/1771078468 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f13900678c0 0x7f1390069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:34.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.366+0000 7f13a7fff640 1 -- 172.21.15.116:0/1771078468 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7f13980860a0 con 0x7f13b00acef0 2024-04-25T00:01:34.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.366+0000 7f13aeffd640 1 --2- 172.21.15.116:0/1771078468 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f13900678c0 0x7f1390069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:34.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.366+0000 7f13aeffd640 1 --2- 172.21.15.116:0/1771078468 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f13900678c0 0x7f1390069d80 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f13a80af000 tx=0x7f13a80a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:34.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.370+0000 7f13a7fff640 1 -- 172.21.15.116:0/1771078468 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f139801f020 con 0x7f13b00acef0 2024-04-25T00:01:34.455 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:01:34.459 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:01:34.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.554+0000 7f13b587c640 1 -- 172.21.15.116:0/1771078468 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f13b00c23f0 con 0x7f13b00acef0 2024-04-25T00:01:34.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.554+0000 7f13a7fff640 1 -- 172.21.15.116:0/1771078468 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6492 (secure 0 0 0) 0x7f139800d040 con 0x7f13b00acef0 2024-04-25T00:01:34.557 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:01:34.557 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6834/1459177921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1459177921},{"type":"v1","addr":"172.21.15.116:6834","nonce":1459177921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:01:08.748957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6837/1762356055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1762356055},{"type":"v1","addr":"172.21.15.174:6837","nonce":1762356055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270]},"id":1},{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:01:22.888641+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6835/174781672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":174781672},{"type":"v1","addr":"172.21.15.116:6835","nonce":174781672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4245,"qdb_cluster":[4245]},"id":2}]} 2024-04-25T00:01:34.557 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 14 2024-04-25T00:01:34.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.554+0000 7f13b587c640 1 -- 172.21.15.116:0/1771078468 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f13900678c0 msgr2=0x7f1390069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:34.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.554+0000 7f13b587c640 1 --2- 172.21.15.116:0/1771078468 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f13900678c0 0x7f1390069d80 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f13a80af000 tx=0x7f13a80a7070 comp rx=0 tx=0).stop 2024-04-25T00:01:34.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.554+0000 7f13b587c640 1 -- 172.21.15.116:0/1771078468 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b00acef0 msgr2=0x7f13b0156f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:34.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.554+0000 7f13b587c640 1 --2- 172.21.15.116:0/1771078468 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b00acef0 0x7f13b0156f40 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f139800b440 tx=0x7f139800b900 comp rx=0 tx=0).stop 2024-04-25T00:01:34.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.558+0000 7f13b587c640 1 -- 172.21.15.116:0/1771078468 shutdown_connections 2024-04-25T00:01:34.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.558+0000 7f13b587c640 1 --2- 172.21.15.116:0/1771078468 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f13900678c0 0x7f1390069d80 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:34.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.558+0000 7f13b587c640 1 --2- 172.21.15.116:0/1771078468 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f13b00ad2d0 0x7f13b014ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:34.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.558+0000 7f13b587c640 1 --2- 172.21.15.116:0/1771078468 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13b00acef0 0x7f13b0156f40 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:34.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.558+0000 7f13b587c640 1 --2- 172.21.15.116:0/1771078468 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f13b00a82c0 0x7f13b0159350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:34.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.558+0000 7f13b587c640 1 -- 172.21.15.116:0/1771078468 >> 172.21.15.116:0/1771078468 conn(0x7f13b00c4200 msgr2=0x7f13b00c5fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:34.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.558+0000 7f13b587c640 1 -- 172.21.15.116:0/1771078468 shutdown_connections 2024-04-25T00:01:34.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:34.558+0000 7f13b587c640 1 -- 172.21.15.116:0/1771078468 wait complete. 2024-04-25T00:01:34.976 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 9, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T00:01:04.678737+0000', 'modified': '2024-04-25T00:01:08.748957+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4270': {'gid': 4270, 'name': 'b', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.174:6837/1762356055', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.174:6836', 'nonce': 1762356055}, {'type': 'v1', 'addr': '172.21.15.174:6837', 'nonce': 1762356055}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'a', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4270, 'qdb_cluster': [4270]} 2024-04-25T00:01:34.976 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-25T00:01:34.976 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-25T00:01:35.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.110+0000 7f31913e1640 1 Processor -- start 2024-04-25T00:01:35.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.110+0000 7f31913e1640 1 -- start start 2024-04-25T00:01:35.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.110+0000 7f31913e1640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f318c152d80 0x7f318c06bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:35.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.110+0000 7f31913e1640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f318c06c290 0x7f318c157390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:35.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.110+0000 7f31913e1640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f318c0c9150 con 0x7f318c06d5f0 2024-04-25T00:01:35.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.110+0000 7f31913e1640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f318c0c9d60 con 0x7f318c152d80 2024-04-25T00:01:35.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.110+0000 7f31913e1640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f318c0c9430 con 0x7f318c06c290 2024-04-25T00:01:35.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.110+0000 7f318affd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f318c152d80 0x7f318c06bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:35.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.110+0000 7f31827fc640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f318c06d5f0 0x7f318c156c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:55326/0 (socket says 172.21.15.116:55326) 2024-04-25T00:01:35.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.110+0000 7f31827fc640 1 -- 172.21.15.116:0/3226045010 learned_addr learned my addr 172.21.15.116:0/3226045010 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:35.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.110+0000 7f318b7fe640 1 --2- 172.21.15.116:0/3226045010 >> v2:172.21.15.174:3300/0 conn(0x7f318c06c290 0x7f318c157390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:35.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.110+0000 7f318affd640 1 -- 172.21.15.116:0/3226045010 >> v2:172.21.15.174:3300/0 conn(0x7f318c06c290 msgr2=0x7f318c157390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:35.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.110+0000 7f318affd640 1 --2- 172.21.15.116:0/3226045010 >> v2:172.21.15.174:3300/0 conn(0x7f318c06c290 0x7f318c157390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.110+0000 7f318affd640 1 -- 172.21.15.116:0/3226045010 >> v1:172.21.15.116:6789/0 conn(0x7f318c06d5f0 legacy=0x7f318c156c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:35.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.110+0000 7f318affd640 1 -- 172.21.15.116:0/3226045010 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f318c0c9710 con 0x7f318c152d80 2024-04-25T00:01:35.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.110+0000 7f318b7fe640 1 --2- 172.21.15.116:0/3226045010 >> v2:172.21.15.174:3300/0 conn(0x7f318c06c290 0x7f318c157390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:35.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.110+0000 7f318a7fc640 1 -- 172.21.15.116:0/3226045010 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4113056107 0 0) 0x7f318c0c9150 con 0x7f318c06d5f0 2024-04-25T00:01:35.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.110+0000 7f318affd640 1 --2- 172.21.15.116:0/3226045010 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f318c152d80 0x7f318c06bd50 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f317c0097c0 tx=0x7f317c01c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4758ae3f00cdeafe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:35.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.114+0000 7f318a7fc640 1 -- 172.21.15.116:0/3226045010 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f317c0b1070 con 0x7f318c152d80 2024-04-25T00:01:35.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.114+0000 7f318a7fc640 1 -- 172.21.15.116:0/3226045010 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f317c0ac620 con 0x7f318c152d80 2024-04-25T00:01:35.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.114+0000 7f318a7fc640 1 -- 172.21.15.116:0/3226045010 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f317c0ac9f0 con 0x7f318c152d80 2024-04-25T00:01:35.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.114+0000 7f31913e1640 1 -- 172.21.15.116:0/3226045010 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f318c152d80 msgr2=0x7f318c06bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:35.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.114+0000 7f31913e1640 1 --2- 172.21.15.116:0/3226045010 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f318c152d80 0x7f318c06bd50 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f317c0097c0 tx=0x7f317c01c3a0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.114+0000 7f31913e1640 1 -- 172.21.15.116:0/3226045010 shutdown_connections 2024-04-25T00:01:35.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.114+0000 7f31913e1640 1 --2- 172.21.15.116:0/3226045010 >> v2:172.21.15.174:3300/0 conn(0x7f318c06c290 0x7f318c157390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.114+0000 7f31913e1640 1 --2- 172.21.15.116:0/3226045010 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f318c152d80 0x7f318c06bd50 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.114+0000 7f31913e1640 1 -- 172.21.15.116:0/3226045010 >> 172.21.15.116:0/3226045010 conn(0x7f318c0acdd0 msgr2=0x7f318c0ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:35.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.114+0000 7f31913e1640 1 -- 172.21.15.116:0/3226045010 shutdown_connections 2024-04-25T00:01:35.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.114+0000 7f31913e1640 1 -- 172.21.15.116:0/3226045010 wait complete. 2024-04-25T00:01:35.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.114+0000 7f31913e1640 1 Processor -- start 2024-04-25T00:01:35.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.114+0000 7f31913e1640 1 -- start start 2024-04-25T00:01:35.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.114+0000 7f31913e1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f318c06c290 0x7f318c14bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:35.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.114+0000 7f31913e1640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f318c06d5f0 0x7f318c154db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:35.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.114+0000 7f31913e1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f318c152d80 0x7f318c1f1370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:35.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.114+0000 7f31913e1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f318c0caf00 con 0x7f318c06c290 2024-04-25T00:01:35.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.114+0000 7f31913e1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f318c0cad80 con 0x7f318c152d80 2024-04-25T00:01:35.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.114+0000 7f31913e1640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f318c1628d0 con 0x7f318c06d5f0 2024-04-25T00:01:35.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.114+0000 7f318affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f318c06c290 0x7f318c14bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:35.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.114+0000 7f318affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f318c06c290 0x7f318c14bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42838/0 (socket says 172.21.15.116:42838) 2024-04-25T00:01:35.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.114+0000 7f318affd640 1 -- 172.21.15.116:0/3465920248 learned_addr learned my addr 172.21.15.116:0/3465920248 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:35.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.114+0000 7f318affd640 1 -- 172.21.15.116:0/3465920248 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f318c06d5f0 msgr2=0x7f318c154db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:35.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.114+0000 7f318affd640 1 --2- 172.21.15.116:0/3465920248 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f318c06d5f0 0x7f318c154db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.114+0000 7f318affd640 1 -- 172.21.15.116:0/3465920248 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f318c152d80 msgr2=0x7f318c1f1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:35.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.114+0000 7f318b7fe640 1 --2- 172.21.15.116:0/3465920248 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f318c152d80 0x7f318c1f1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:35.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.114+0000 7f318affd640 1 --2- 172.21.15.116:0/3465920248 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f318c152d80 0x7f318c1f1370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.114+0000 7f318affd640 1 -- 172.21.15.116:0/3465920248 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f318c0ac780 con 0x7f318c06c290 2024-04-25T00:01:35.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.114+0000 7f318affd640 1 --2- 172.21.15.116:0/3465920248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f318c06c290 0x7f318c14bac0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f317c018fb0 tx=0x7f317c019140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:35.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.114+0000 7f3183fff640 1 -- 172.21.15.116:0/3465920248 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f317c0b9070 con 0x7f318c06c290 2024-04-25T00:01:35.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.114+0000 7f3183fff640 1 -- 172.21.15.116:0/3465920248 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f317c0b5630 con 0x7f318c06c290 2024-04-25T00:01:35.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.114+0000 7f31913e1640 1 -- 172.21.15.116:0/3465920248 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f318c1eead0 con 0x7f318c06c290 2024-04-25T00:01:35.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.114+0000 7f3183fff640 1 -- 172.21.15.116:0/3465920248 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f317c0b1040 con 0x7f318c06c290 2024-04-25T00:01:35.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.114+0000 7f31913e1640 1 -- 172.21.15.116:0/3465920248 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f318c14c8a0 con 0x7f318c06c290 2024-04-25T00:01:35.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.118+0000 7f3183fff640 1 -- 172.21.15.116:0/3465920248 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f317c0bc070 con 0x7f318c06c290 2024-04-25T00:01:35.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.118+0000 7f3183fff640 1 --2- 172.21.15.116:0/3465920248 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f316c0678c0 0x7f316c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:35.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.118+0000 7f3183fff640 1 -- 172.21.15.116:0/3465920248 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7f317c11fef0 con 0x7f318c06c290 2024-04-25T00:01:35.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.118+0000 7f31827fc640 1 --2- 172.21.15.116:0/3465920248 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f316c0678c0 0x7f316c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:35.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.118+0000 7f319455a000 1 Processor -- start 2024-04-25T00:01:35.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.118+0000 7f319455a000 1 -- start start 2024-04-25T00:01:35.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.118+0000 7f31827fc640 1 --2- 172.21.15.116:0/3465920248 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f316c0678c0 0x7f316c069d80 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f31740011f0 tx=0x7f3174007930 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:35.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.118+0000 7f319455a000 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b150522b80 0x55b150522f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:35.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.118+0000 7f319455a000 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b1505234a0 0x55b150523880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:35.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.118+0000 7f319455a000 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55b150523dc0 0x55b15056cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:35.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.118+0000 7f319455a000 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b1503735e0 con 0x55b150522b80 2024-04-25T00:01:35.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.118+0000 7f319455a000 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55b1502fac50 con 0x55b1505234a0 2024-04-25T00:01:35.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.118+0000 7f319455a000 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x55b15036f5a0 con 0x55b150523dc0 2024-04-25T00:01:35.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.118+0000 7f318affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b150522b80 0x55b150522f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:35.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.118+0000 7f318affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b150522b80 0x55b150522f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42844/0 (socket says 172.21.15.116:42844) 2024-04-25T00:01:35.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.118+0000 7f318affd640 1 -- 172.21.15.116:0/1705478815 learned_addr learned my addr 172.21.15.116:0/1705478815 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:35.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.118+0000 7f318affd640 1 -- 172.21.15.116:0/1705478815 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55b150523dc0 msgr2=0x55b15056cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:35.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.118+0000 7f31827fc640 1 --2- 172.21.15.116:0/1705478815 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b1505234a0 0x55b150523880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:35.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.118+0000 7f318affd640 1 --2- 172.21.15.116:0/1705478815 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55b150523dc0 0x55b15056cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.118+0000 7f318affd640 1 -- 172.21.15.116:0/1705478815 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b1505234a0 msgr2=0x55b150523880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:35.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.118+0000 7f318affd640 1 --2- 172.21.15.116:0/1705478815 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b1505234a0 0x55b150523880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.118+0000 7f318affd640 1 -- 172.21.15.116:0/1705478815 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b1502f8d10 con 0x55b150522b80 2024-04-25T00:01:35.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.118+0000 7f318affd640 1 --2- 172.21.15.116:0/1705478815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b150522b80 0x55b150522f60 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f317c0f8740 tx=0x7f317c0eff60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=625dabc6ddfbe805 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:35.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.118+0000 7f3180ff9640 1 -- 172.21.15.116:0/1705478815 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f317c0c3070 con 0x55b150522b80 2024-04-25T00:01:35.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.118+0000 7f3180ff9640 1 -- 172.21.15.116:0/1705478815 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f317c0f4440 con 0x55b150522b80 2024-04-25T00:01:35.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.118+0000 7f3180ff9640 1 -- 172.21.15.116:0/1705478815 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f317c0f8c70 con 0x55b150522b80 2024-04-25T00:01:35.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.122+0000 7f319455a000 1 -- 172.21.15.116:0/1705478815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b150522b80 msgr2=0x55b150522f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:35.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.122+0000 7f319455a000 1 --2- 172.21.15.116:0/1705478815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b150522b80 0x55b150522f60 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f317c0f8740 tx=0x7f317c0eff60 comp rx=0 tx=0).stop 2024-04-25T00:01:35.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.122+0000 7f319455a000 1 -- 172.21.15.116:0/1705478815 shutdown_connections 2024-04-25T00:01:35.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.122+0000 7f319455a000 1 --2- 172.21.15.116:0/1705478815 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55b150523dc0 0x55b15056cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.122+0000 7f319455a000 1 --2- 172.21.15.116:0/1705478815 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b1505234a0 0x55b150523880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.122+0000 7f319455a000 1 --2- 172.21.15.116:0/1705478815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b150522b80 0x55b150522f60 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.122+0000 7f319455a000 1 -- 172.21.15.116:0/1705478815 >> 172.21.15.116:0/1705478815 conn(0x55b150560c00 msgr2=0x55b150562ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:35.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.122+0000 7f319455a000 1 -- 172.21.15.116:0/1705478815 shutdown_connections 2024-04-25T00:01:35.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.122+0000 7f319455a000 1 -- 172.21.15.116:0/1705478815 wait complete. 2024-04-25T00:01:35.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.122+0000 7f319455a000 1 -- start start 2024-04-25T00:01:35.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.122+0000 7f319455a000 1 Processor -- start 2024-04-25T00:01:35.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.122+0000 7f319455a000 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55b150522b80 0x55b15055ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:35.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.122+0000 7f319455a000 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b1505234a0 0x55b15055f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:35.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.122+0000 7f319455a000 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b150523dc0 0x55b15055f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:35.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.122+0000 7f319455a000 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b150377c10 con 0x55b1505234a0 2024-04-25T00:01:35.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.122+0000 7f319455a000 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x55b15032d520 con 0x55b150523dc0 2024-04-25T00:01:35.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.122+0000 7f319455a000 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x55b15052b610 con 0x55b150522b80 2024-04-25T00:01:35.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.122+0000 7f319455a000 10 client.0 _ll_register_callbacks cb 0x55b1505105c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:01:35.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.122+0000 7f31827fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b1505234a0 0x55b15055f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:35.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.122+0000 7f31827fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b1505234a0 0x55b15055f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42854/0 (socket says 172.21.15.116:42854) 2024-04-25T00:01:35.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.122+0000 7f31827fc640 1 -- 172.21.15.116:0/414936635 learned_addr learned my addr 172.21.15.116:0/414936635 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:35.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.122+0000 7f318affd640 1 --2- 172.21.15.116:0/414936635 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55b150522b80 0x55b15055ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:35.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.122+0000 7f318b7fe640 1 --2- 172.21.15.116:0/414936635 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b150523dc0 0x55b15055f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:35.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.122+0000 7f31827fc640 1 -- 172.21.15.116:0/414936635 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55b150522b80 msgr2=0x55b15055ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:35.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.122+0000 7f31827fc640 1 --2- 172.21.15.116:0/414936635 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55b150522b80 0x55b15055ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.122+0000 7f31827fc640 1 -- 172.21.15.116:0/414936635 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b150523dc0 msgr2=0x55b15055f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:35.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.122+0000 7f31827fc640 1 --2- 172.21.15.116:0/414936635 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b150523dc0 0x55b15055f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.122+0000 7f31827fc640 1 -- 172.21.15.116:0/414936635 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b15030b7b0 con 0x55b1505234a0 2024-04-25T00:01:35.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.122+0000 7f318b7fe640 1 --2- 172.21.15.116:0/414936635 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b150523dc0 0x55b15055f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:35.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.122+0000 7f31827fc640 1 --2- 172.21.15.116:0/414936635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b1505234a0 0x55b15055f2c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f3174005b50 tx=0x7f3174001090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7f07a989a423309 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:35.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.122+0000 7f3169ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-04-25T00:01:35.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.122+0000 7f3169ffb640 1 -- 172.21.15.116:0/414936635 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3174012440 con 0x55b1505234a0 2024-04-25T00:01:35.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.122+0000 7f3169ffb640 1 -- 172.21.15.116:0/414936635 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7f3174022e00 con 0x55b1505234a0 2024-04-25T00:01:35.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.122+0000 7f319455a000 1 -- 172.21.15.116:0/414936635 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f317c0f4440 con 0x55b1505234a0 2024-04-25T00:01:35.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.122+0000 7f3169ffb640 1 -- 172.21.15.116:0/414936635 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=14) ==== 24+0+0 (secure 0 0 0) 0x7f31740125e0 con 0x55b1505234a0 2024-04-25T00:01:35.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.122+0000 7f319455a000 10 client.4860 fetch_fsmap learned FSMap version 14 2024-04-25T00:01:35.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.122+0000 7f319455a000 1 -- 172.21.15.116:0/414936635 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=14,monmap=2+}) -- 0x55b1502faea0 con 0x55b1505234a0 2024-04-25T00:01:35.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.122+0000 7f3169ffb640 1 -- 172.21.15.116:0/414936635 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f317c0f4440 con 0x55b1505234a0 2024-04-25T00:01:35.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.122+0000 7f3169ffb640 1 -- 172.21.15.116:0/414936635 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 14) ==== 4128+0+0 (secure 0 0 0) 0x7f3174012780 con 0x55b1505234a0 2024-04-25T00:01:35.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.122+0000 7f319455a000 10 client.4860 fetch_fsmap finished waiting for FSMap version 14 2024-04-25T00:01:35.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.122+0000 7f319455a000 10 client.4860 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T00:01:35.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.122+0000 7f319455a000 20 client.4860 populate_metadata read hostname 'smithi116' 2024-04-25T00:01:35.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.122+0000 7f319455a000 1 --2- 172.21.15.116:0/414936635 >> [v2:172.21.15.174:6836/1762356055,v1:172.21.15.174:6837/1762356055] conn(0x55b1505758d0 0x55b150577cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:35.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.122+0000 7f319455a000 4 client.4860 mds_command: new command op to 4270 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:01:35.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.122+0000 7f319455a000 1 -- 172.21.15.116:0/414936635 --> [v2:172.21.15.174:6836/1762356055,v1:172.21.15.174:6837/1762356055] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b15030eab0 con 0x55b1505758d0 2024-04-25T00:01:35.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.122+0000 7f318affd640 1 --2- 172.21.15.116:0/414936635 >> [v2:172.21.15.174:6836/1762356055,v1:172.21.15.174:6837/1762356055] conn(0x55b1505758d0 0x55b150577cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:35.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.126+0000 7f318affd640 1 --2- 172.21.15.116:0/414936635 >> [v2:172.21.15.174:6836/1762356055,v1:172.21.15.174:6837/1762356055] conn(0x55b1505758d0 0x55b150577cc0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d410f847fc195e server_cookie=7117ee29efcc1264 in_seq=0 out_seq=0 2024-04-25T00:01:35.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.126+0000 7f3169ffb640 10 client.4860 ms_handle_connect on v2:172.21.15.174:6836/1762356055 2024-04-25T00:01:35.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.126+0000 7f3169ffb640 1 -- 172.21.15.116:0/414936635 <== mds.0 v2:172.21.15.174:6836/1762356055 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b15030eab0 con 0x55b1505758d0 2024-04-25T00:01:35.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.126+0000 7f3169ffb640 10 client.4860 handle_command_reply: tid=0 2024-04-25T00:01:35.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.126+0000 7f319455a000 1 client.4860 shutdown 2024-04-25T00:01:35.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.126+0000 7f319455a000 1 -- 172.21.15.116:0/414936635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b1505234a0 msgr2=0x55b15055f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:35.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.126+0000 7f319455a000 1 --2- 172.21.15.116:0/414936635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b1505234a0 0x55b15055f2c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f3174005b50 tx=0x7f3174001090 comp rx=0 tx=0).stop 2024-04-25T00:01:35.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.126+0000 7f319455a000 1 -- 172.21.15.116:0/414936635 shutdown_connections 2024-04-25T00:01:35.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.126+0000 7f319455a000 1 --2- 172.21.15.116:0/414936635 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b150523dc0 0x55b15055f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.126+0000 7f319455a000 1 --2- 172.21.15.116:0/414936635 >> [v2:172.21.15.174:6836/1762356055,v1:172.21.15.174:6837/1762356055] conn(0x55b1505758d0 0x55b150577cc0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.126+0000 7f319455a000 1 --2- 172.21.15.116:0/414936635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b1505234a0 0x55b15055f2c0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.126+0000 7f319455a000 1 --2- 172.21.15.116:0/414936635 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55b150522b80 0x55b15055ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.126+0000 7f3169ffb640 0 client.4860 ms_handle_reset on v2:172.21.15.174:6836/1762356055 2024-04-25T00:01:35.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.126+0000 7f319455a000 1 -- 172.21.15.116:0/414936635 >> 172.21.15.116:0/414936635 conn(0x55b150560c00 msgr2=0x55b150522280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:35.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.126+0000 7f319455a000 1 -- 172.21.15.116:0/414936635 shutdown_connections 2024-04-25T00:01:35.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.126+0000 7f319455a000 1 -- 172.21.15.116:0/414936635 wait complete. 2024-04-25T00:01:35.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.126+0000 7f319455a000 20 client.4860 trim_cache size 0 max 16384 2024-04-25T00:01:35.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.142+0000 7f319455a000 1 Processor -- start 2024-04-25T00:01:35.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.142+0000 7f319455a000 1 -- start start 2024-04-25T00:01:35.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.142+0000 7f319455a000 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55b150522b80 0x55b1505222f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:35.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.142+0000 7f319455a000 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b1505234a0 0x55b150577cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:35.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.142+0000 7f319455a000 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b150523dc0 0x55b150578200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:35.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.142+0000 7f319455a000 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f318c1fcd10 con 0x55b150523dc0 2024-04-25T00:01:35.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.142+0000 7f319455a000 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f318c1fdda0 con 0x55b1505234a0 2024-04-25T00:01:35.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.142+0000 7f319455a000 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f318c1fb800 con 0x55b150522b80 2024-04-25T00:01:35.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f318b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b150523dc0 0x55b150578200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:35.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f318b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b150523dc0 0x55b150578200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42858/0 (socket says 172.21.15.116:42858) 2024-04-25T00:01:35.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f318b7fe640 1 -- 172.21.15.116:0/2627756949 learned_addr learned my addr 172.21.15.116:0/2627756949 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:35.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f31827fc640 1 --2- 172.21.15.116:0/2627756949 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b1505234a0 0x55b150577cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:35.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f318affd640 1 --2- 172.21.15.116:0/2627756949 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55b150522b80 0x55b1505222f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:35.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f318b7fe640 1 -- 172.21.15.116:0/2627756949 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55b150522b80 msgr2=0x55b1505222f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:35.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f318b7fe640 1 --2- 172.21.15.116:0/2627756949 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55b150522b80 0x55b1505222f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f318b7fe640 1 -- 172.21.15.116:0/2627756949 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b1505234a0 msgr2=0x55b150577cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:35.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f318b7fe640 1 --2- 172.21.15.116:0/2627756949 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b1505234a0 0x55b150577cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f318b7fe640 1 -- 172.21.15.116:0/2627756949 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b1502fbe80 con 0x55b150523dc0 2024-04-25T00:01:35.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f31827fc640 1 --2- 172.21.15.116:0/2627756949 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b1505234a0 0x55b150577cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:35.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f318b7fe640 1 --2- 172.21.15.116:0/2627756949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b150523dc0 0x55b150578200 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f3184008e40 tx=0x7f318400e7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=889bb05ca842a7e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:35.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f316bfff640 1 -- 172.21.15.116:0/2627756949 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f318400f860 con 0x55b150523dc0 2024-04-25T00:01:35.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f316bfff640 1 -- 172.21.15.116:0/2627756949 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f318400fa00 con 0x55b150523dc0 2024-04-25T00:01:35.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f316bfff640 1 -- 172.21.15.116:0/2627756949 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f318400fdf0 con 0x55b150523dc0 2024-04-25T00:01:35.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f319455a000 1 -- 172.21.15.116:0/2627756949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b150523dc0 msgr2=0x55b150578200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:35.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f319455a000 1 --2- 172.21.15.116:0/2627756949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b150523dc0 0x55b150578200 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f3184008e40 tx=0x7f318400e7b0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f319455a000 1 -- 172.21.15.116:0/2627756949 shutdown_connections 2024-04-25T00:01:35.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f319455a000 1 --2- 172.21.15.116:0/2627756949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b150523dc0 0x55b150578200 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f319455a000 1 --2- 172.21.15.116:0/2627756949 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b1505234a0 0x55b150577cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f319455a000 1 --2- 172.21.15.116:0/2627756949 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55b150522b80 0x55b1505222f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f319455a000 1 -- 172.21.15.116:0/2627756949 >> 172.21.15.116:0/2627756949 conn(0x55b150560c00 msgr2=0x55b1505280a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:35.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f319455a000 1 -- 172.21.15.116:0/2627756949 shutdown_connections 2024-04-25T00:01:35.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f319455a000 1 -- 172.21.15.116:0/2627756949 wait complete. 2024-04-25T00:01:35.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f319455a000 1 -- start start 2024-04-25T00:01:35.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f319455a000 1 Processor -- start 2024-04-25T00:01:35.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f319455a000 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b150522b80 0x55b15057a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:35.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f319455a000 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55b1505234a0 0x55b1505d5f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:35.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f319455a000 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b150523dc0 0x55b1505d6450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:35.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f319455a000 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x55b1503728f0 con 0x55b150523dc0 2024-04-25T00:01:35.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f319455a000 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f318c1fbe20 con 0x55b150522b80 2024-04-25T00:01:35.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f319455a000 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x55b150319810 con 0x55b1505234a0 2024-04-25T00:01:35.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f319455a000 10 client.0 _ll_register_callbacks cb 0x7f3174001320 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:01:35.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f318b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b150523dc0 0x55b1505d6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:35.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f318b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b150523dc0 0x55b1505d6450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42860/0 (socket says 172.21.15.116:42860) 2024-04-25T00:01:35.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f318b7fe640 1 -- 172.21.15.116:0/3839107968 learned_addr learned my addr 172.21.15.116:0/3839107968 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:35.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f318affd640 1 --2- 172.21.15.116:0/3839107968 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b150522b80 0x55b15057a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:35.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f31827fc640 1 --2- 172.21.15.116:0/3839107968 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55b1505234a0 0x55b1505d5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:35.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f318b7fe640 1 -- 172.21.15.116:0/3839107968 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55b1505234a0 msgr2=0x55b1505d5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:35.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f318b7fe640 1 --2- 172.21.15.116:0/3839107968 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55b1505234a0 0x55b1505d5f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f318b7fe640 1 -- 172.21.15.116:0/3839107968 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b150522b80 msgr2=0x55b15057a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:35.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f318b7fe640 1 --2- 172.21.15.116:0/3839107968 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b150522b80 0x55b15057a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f318b7fe640 1 -- 172.21.15.116:0/3839107968 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b150340760 con 0x55b150523dc0 2024-04-25T00:01:35.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f318affd640 1 --2- 172.21.15.116:0/3839107968 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b150522b80 0x55b15057a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:35.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f318b7fe640 1 --2- 172.21.15.116:0/3839107968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b150523dc0 0x55b1505d6450 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x55b150418b00 tx=0x7f3184004ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66496af1a2f42f79 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:35.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f3169ffb640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-04-25T00:01:35.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f3169ffb640 1 -- 172.21.15.116:0/3839107968 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3184027020 con 0x55b150523dc0 2024-04-25T00:01:35.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f3169ffb640 1 -- 172.21.15.116:0/3839107968 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7f3184023680 con 0x55b150523dc0 2024-04-25T00:01:35.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f319455a000 1 -- 172.21.15.116:0/3839107968 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f318400fa00 con 0x55b150523dc0 2024-04-25T00:01:35.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f3169ffb640 1 -- 172.21.15.116:0/3839107968 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=14) ==== 24+0+0 (secure 0 0 0) 0x7f3184031050 con 0x55b150523dc0 2024-04-25T00:01:35.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f319455a000 10 client.4866 fetch_fsmap learned FSMap version 14 2024-04-25T00:01:35.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f319455a000 1 -- 172.21.15.116:0/3839107968 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=14,monmap=2+}) -- 0x55b1503cc9e0 con 0x55b150523dc0 2024-04-25T00:01:35.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f3169ffb640 1 -- 172.21.15.116:0/3839107968 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f318400fa00 con 0x55b150523dc0 2024-04-25T00:01:35.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f3169ffb640 1 -- 172.21.15.116:0/3839107968 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 14) ==== 4128+0+0 (secure 0 0 0) 0x7f3184034020 con 0x55b150523dc0 2024-04-25T00:01:35.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f319455a000 10 client.4866 fetch_fsmap finished waiting for FSMap version 14 2024-04-25T00:01:35.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f319455a000 10 client.4866 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T00:01:35.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f319455a000 20 client.4866 populate_metadata read hostname 'smithi116' 2024-04-25T00:01:35.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f319455a000 1 --2- 172.21.15.116:0/3839107968 >> [v2:172.21.15.174:6836/1762356055,v1:172.21.15.174:6837/1762356055] conn(0x55b1505d2c60 0x55b1505d3040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:35.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f319455a000 4 client.4866 mds_command: new command op to 4270 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T00:01:35.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.146+0000 7f319455a000 1 -- 172.21.15.116:0/3839107968 --> [v2:172.21.15.174:6836/1762356055,v1:172.21.15.174:6837/1762356055] -- command(tid 0: {"prefix": "status"}) -- 0x55b150380b20 con 0x55b1505d2c60 2024-04-25T00:01:35.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.150+0000 7f318affd640 1 --2- 172.21.15.116:0/3839107968 >> [v2:172.21.15.174:6836/1762356055,v1:172.21.15.174:6837/1762356055] conn(0x55b1505d2c60 0x55b1505d3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:35.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.150+0000 7f318affd640 1 --2- 172.21.15.116:0/3839107968 >> [v2:172.21.15.174:6836/1762356055,v1:172.21.15.174:6837/1762356055] conn(0x55b1505d2c60 0x55b1505d3040 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5b7211bbce3adae server_cookie=78928545548eda83 in_seq=0 out_seq=0 2024-04-25T00:01:35.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.150+0000 7f3169ffb640 10 client.4866 ms_handle_connect on v2:172.21.15.174:6836/1762356055 2024-04-25T00:01:35.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.150+0000 7f3169ffb640 1 -- 172.21.15.116:0/3839107968 <== mds.0 v2:172.21.15.174:6836/1762356055 1 ==== command_reply(tid 0: 0 ) ==== 8+0+313 (crc 0 0 0) 0x55b150380b20 con 0x55b1505d2c60 2024-04-25T00:01:35.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.150+0000 7f3169ffb640 10 client.4866 handle_command_reply: tid=0 2024-04-25T00:01:35.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.150+0000 7f319455a000 1 client.4866 shutdown 2024-04-25T00:01:35.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.150+0000 7f319455a000 1 -- 172.21.15.116:0/3839107968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b150523dc0 msgr2=0x55b1505d6450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:35.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.150+0000 7f319455a000 1 --2- 172.21.15.116:0/3839107968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b150523dc0 0x55b1505d6450 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x55b150418b00 tx=0x7f3184004ac0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.150+0000 7f319455a000 1 -- 172.21.15.116:0/3839107968 shutdown_connections 2024-04-25T00:01:35.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.150+0000 7f319455a000 1 --2- 172.21.15.116:0/3839107968 >> [v2:172.21.15.174:6836/1762356055,v1:172.21.15.174:6837/1762356055] conn(0x55b1505d2c60 0x55b1505d3040 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.150+0000 7f319455a000 1 --2- 172.21.15.116:0/3839107968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x55b150523dc0 0x55b1505d6450 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.150+0000 7f319455a000 1 --2- 172.21.15.116:0/3839107968 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x55b1505234a0 0x55b1505d5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.150+0000 7f3169ffb640 0 client.4866 ms_handle_reset on v2:172.21.15.174:6836/1762356055 2024-04-25T00:01:35.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.150+0000 7f319455a000 1 --2- 172.21.15.116:0/3839107968 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x55b150522b80 0x55b15057a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.150+0000 7f319455a000 1 -- 172.21.15.116:0/3839107968 >> 172.21.15.116:0/3839107968 conn(0x55b150560c00 msgr2=0x55b1505212e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:35.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.150+0000 7f319455a000 1 -- 172.21.15.116:0/3839107968 shutdown_connections 2024-04-25T00:01:35.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.150+0000 7f319455a000 1 -- 172.21.15.116:0/3839107968 wait complete. 2024-04-25T00:01:35.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.150+0000 7f319455a000 20 client.4866 trim_cache size 0 max 16384 2024-04-25T00:01:35.158 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-25T00:01:35.158 INFO:teuthology.orchestra.run.smithi116.stdout: "cluster_fsid": "6d6a1aa7-0478-4c83-8057-05917e81f046", 2024-04-25T00:01:35.158 INFO:teuthology.orchestra.run.smithi116.stdout: "whoami": 0, 2024-04-25T00:01:35.158 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 4270, 2024-04-25T00:01:35.158 INFO:teuthology.orchestra.run.smithi116.stdout: "want_state": "up:active", 2024-04-25T00:01:35.158 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "up:active", 2024-04-25T00:01:35.159 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "a", 2024-04-25T00:01:35.159 INFO:teuthology.orchestra.run.smithi116.stdout: "rank_uptime": 29.767733695, 2024-04-25T00:01:35.159 INFO:teuthology.orchestra.run.smithi116.stdout: "mdsmap_epoch": 9, 2024-04-25T00:01:35.159 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch": 27, 2024-04-25T00:01:35.159 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch_barrier": 23, 2024-04-25T00:01:35.159 INFO:teuthology.orchestra.run.smithi116.stdout: "uptime": 36.403928082 2024-04-25T00:01:35.159 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-25T00:01:35.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.150+0000 7f31913e1640 1 -- 172.21.15.116:0/3465920248 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f316c0678c0 msgr2=0x7f316c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:35.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.150+0000 7f31913e1640 1 --2- 172.21.15.116:0/3465920248 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f316c0678c0 0x7f316c069d80 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f31740011f0 tx=0x7f3174007930 comp rx=0 tx=0).stop 2024-04-25T00:01:35.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.150+0000 7f31913e1640 1 -- 172.21.15.116:0/3465920248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f318c06c290 msgr2=0x7f318c14bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:35.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.150+0000 7f31913e1640 1 --2- 172.21.15.116:0/3465920248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f318c06c290 0x7f318c14bac0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f317c018fb0 tx=0x7f317c019140 comp rx=0 tx=0).stop 2024-04-25T00:01:35.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.150+0000 7f31913e1640 1 -- 172.21.15.116:0/3465920248 shutdown_connections 2024-04-25T00:01:35.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.150+0000 7f31913e1640 1 --2- 172.21.15.116:0/3465920248 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f316c0678c0 0x7f316c069d80 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.150+0000 7f31913e1640 1 --2- 172.21.15.116:0/3465920248 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f318c152d80 0x7f318c1f1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.150+0000 7f31913e1640 1 --2- 172.21.15.116:0/3465920248 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f318c06d5f0 0x7f318c154db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.150+0000 7f31913e1640 1 --2- 172.21.15.116:0/3465920248 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f318c06c290 0x7f318c14bac0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.150+0000 7f31913e1640 1 -- 172.21.15.116:0/3465920248 >> 172.21.15.116:0/3465920248 conn(0x7f318c0acdd0 msgr2=0x7f318c14e150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:35.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.150+0000 7f31913e1640 1 -- 172.21.15.116:0/3465920248 shutdown_connections 2024-04-25T00:01:35.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.150+0000 7f31913e1640 1 -- 172.21.15.116:0/3465920248 wait complete. 2024-04-25T00:01:35.167 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T00:01:35.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.266+0000 7ff04df04640 1 Processor -- start 2024-04-25T00:01:35.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.270+0000 7ff04df04640 1 -- start start 2024-04-25T00:01:35.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.270+0000 7ff04df04640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff048070dd0 0x7ff048071270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:35.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.270+0000 7ff04df04640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff04806e890 0x7ff0480717b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:35.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.270+0000 7ff04df04640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff048152c10 con 0x7ff048148890 2024-04-25T00:01:35.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.270+0000 7ff04df04640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7ff048071cf0 con 0x7ff048070dd0 2024-04-25T00:01:35.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.270+0000 7ff04df04640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff048151e50 con 0x7ff04806e890 2024-04-25T00:01:35.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.270+0000 7ff047fff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff04806e890 0x7ff0480717b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:35.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.270+0000 7ff046ffd640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff048070dd0 0x7ff048071270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:35.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.270+0000 7ff046ffd640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff048070dd0 0x7ff048071270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:39840/0 (socket says 172.21.15.116:39840) 2024-04-25T00:01:35.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.270+0000 7ff046ffd640 1 -- 172.21.15.116:0/2546371423 learned_addr learned my addr 172.21.15.116:0/2546371423 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:35.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.270+0000 7ff047fff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff04806e890 0x7ff0480717b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:35666/0 (socket says 172.21.15.116:35666) 2024-04-25T00:01:35.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.270+0000 7ff047fff640 1 -- 172.21.15.116:0/2546371423 >> v2:172.21.15.174:3300/0 conn(0x7ff048070dd0 msgr2=0x7ff048071270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:35.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.270+0000 7ff047fff640 1 --2- 172.21.15.116:0/2546371423 >> v2:172.21.15.174:3300/0 conn(0x7ff048070dd0 0x7ff048071270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.270+0000 7ff047fff640 1 -- 172.21.15.116:0/2546371423 >> v1:172.21.15.174:6789/0 conn(0x7ff048148890 legacy=0x7ff048072590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:35.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.270+0000 7ff047fff640 1 -- 172.21.15.116:0/2546371423 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff048152130 con 0x7ff04806e890 2024-04-25T00:01:35.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.274+0000 7ff047fff640 1 --2- 172.21.15.116:0/2546371423 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff04806e890 0x7ff0480717b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff040009d90 tx=0x7ff04001c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7977bffbe3dd37af server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:35.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.274+0000 7ff0467fc640 1 -- 172.21.15.116:0/2546371423 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0400b2070 con 0x7ff04806e890 2024-04-25T00:01:35.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.274+0000 7ff0467fc640 1 -- 172.21.15.116:0/2546371423 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0400a57b0 con 0x7ff04806e890 2024-04-25T00:01:35.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.274+0000 7ff0467fc640 1 -- 172.21.15.116:0/2546371423 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0400a5b80 con 0x7ff04806e890 2024-04-25T00:01:35.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.274+0000 7ff04df04640 1 -- 172.21.15.116:0/2546371423 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff04806e890 msgr2=0x7ff0480717b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:35.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.274+0000 7ff04df04640 1 --2- 172.21.15.116:0/2546371423 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff04806e890 0x7ff0480717b0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff040009d90 tx=0x7ff04001c970 comp rx=0 tx=0).stop 2024-04-25T00:01:35.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.274+0000 7ff04df04640 1 -- 172.21.15.116:0/2546371423 shutdown_connections 2024-04-25T00:01:35.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.274+0000 7ff04df04640 1 --2- 172.21.15.116:0/2546371423 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff04806e890 0x7ff0480717b0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.274+0000 7ff04df04640 1 --2- 172.21.15.116:0/2546371423 >> v2:172.21.15.174:3300/0 conn(0x7ff048070dd0 0x7ff048071270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.274+0000 7ff04df04640 1 -- 172.21.15.116:0/2546371423 >> 172.21.15.116:0/2546371423 conn(0x7ff0480c3b00 msgr2=0x7ff0480c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:35.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.274+0000 7ff04df04640 1 -- 172.21.15.116:0/2546371423 shutdown_connections 2024-04-25T00:01:35.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.274+0000 7ff04df04640 1 -- 172.21.15.116:0/2546371423 wait complete. 2024-04-25T00:01:35.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.274+0000 7ff04df04640 1 Processor -- start 2024-04-25T00:01:35.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.274+0000 7ff04df04640 1 -- start start 2024-04-25T00:01:35.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.274+0000 7ff04df04640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff04806e890 0x7ff04814e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:35.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.274+0000 7ff04df04640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff048070dd0 0x7ff0480c6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:35.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.274+0000 7ff04df04640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff048148890 0x7ff04814c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:35.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.274+0000 7ff04df04640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff048072e20 con 0x7ff048070dd0 2024-04-25T00:01:35.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.274+0000 7ff04df04640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff048072ca0 con 0x7ff04806e890 2024-04-25T00:01:35.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.274+0000 7ff04df04640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff0480ab590 con 0x7ff048148890 2024-04-25T00:01:35.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.274+0000 7ff046ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff048070dd0 0x7ff0480c6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:35.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.274+0000 7ff046ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff048070dd0 0x7ff0480c6270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42864/0 (socket says 172.21.15.116:42864) 2024-04-25T00:01:35.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.274+0000 7ff046ffd640 1 -- 172.21.15.116:0/365201949 learned_addr learned my addr 172.21.15.116:0/365201949 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:35.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.274+0000 7ff0477fe640 1 --2- 172.21.15.116:0/365201949 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff04806e890 0x7ff04814e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:35.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.274+0000 7ff047fff640 1 --2- 172.21.15.116:0/365201949 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff048148890 0x7ff04814c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:35.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.274+0000 7ff046ffd640 1 -- 172.21.15.116:0/365201949 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff048148890 msgr2=0x7ff04814c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:35.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.274+0000 7ff046ffd640 1 --2- 172.21.15.116:0/365201949 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff048148890 0x7ff04814c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.274+0000 7ff046ffd640 1 -- 172.21.15.116:0/365201949 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff04806e890 msgr2=0x7ff04814e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:35.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.274+0000 7ff046ffd640 1 --2- 172.21.15.116:0/365201949 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff04806e890 0x7ff04814e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.274+0000 7ff046ffd640 1 -- 172.21.15.116:0/365201949 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff04814e700 con 0x7ff048070dd0 2024-04-25T00:01:35.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.274+0000 7ff047fff640 1 --2- 172.21.15.116:0/365201949 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff048148890 0x7ff04814c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:35.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.274+0000 7ff046ffd640 1 --2- 172.21.15.116:0/365201949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff048070dd0 0x7ff0480c6270 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ff03c010b40 tx=0x7ff03c00e360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:35.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.278+0000 7ff02ffff640 1 -- 172.21.15.116:0/365201949 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff03c01e070 con 0x7ff048070dd0 2024-04-25T00:01:35.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.278+0000 7ff02ffff640 1 -- 172.21.15.116:0/365201949 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff03c011450 con 0x7ff048070dd0 2024-04-25T00:01:35.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.278+0000 7ff04df04640 1 -- 172.21.15.116:0/365201949 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0481555d0 con 0x7ff048070dd0 2024-04-25T00:01:35.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.278+0000 7ff02ffff640 1 -- 172.21.15.116:0/365201949 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff03c011800 con 0x7ff048070dd0 2024-04-25T00:01:35.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.278+0000 7ff04df04640 1 -- 172.21.15.116:0/365201949 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0480c67b0 con 0x7ff048070dd0 2024-04-25T00:01:35.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.278+0000 7ff04df04640 1 -- 172.21.15.116:0/365201949 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff04814eb80 con 0x7ff048070dd0 2024-04-25T00:01:35.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.278+0000 7ff02ffff640 1 -- 172.21.15.116:0/365201949 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff03c021070 con 0x7ff048070dd0 2024-04-25T00:01:35.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.278+0000 7ff02ffff640 1 --2- 172.21.15.116:0/365201949 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff0340678f0 0x7ff034069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:35.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.278+0000 7ff02ffff640 1 -- 172.21.15.116:0/365201949 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7ff03c088290 con 0x7ff048070dd0 2024-04-25T00:01:35.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.278+0000 7ff0477fe640 1 --2- 172.21.15.116:0/365201949 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff0340678f0 0x7ff034069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:35.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.278+0000 7ff0477fe640 1 --2- 172.21.15.116:0/365201949 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff0340678f0 0x7ff034069db0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff0380015b0 tx=0x7ff038009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:35.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.282+0000 7ff02ffff640 1 -- 172.21.15.116:0/365201949 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ff03c058ab0 con 0x7ff048070dd0 2024-04-25T00:01:35.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.434+0000 7ff04df04640 1 -- 172.21.15.116:0/365201949 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff048068350 con 0x7ff048070dd0 2024-04-25T00:01:35.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.438+0000 7ff02ffff640 1 -- 172.21.15.116:0/365201949 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+20311 (secure 0 0 0) 0x7ff03c0583b0 con 0x7ff048070dd0 2024-04-25T00:01:35.439 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:01:35.439 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":27,"fsid":"6d6a1aa7-0478-4c83-8057-05917e81f046","created":"2024-04-25T00:00:40.682493+0000","modified":"2024-04-25T00:01:20.501508+0000","last_up_change":"2024-04-25T00:00:51.822174+0000","last_in_change":"2024-04-25T00:00:42.980473+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T00:00:53.324376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-04-25T00:01:03.197326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-04-25T00:01:03.728618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-04-25T00:01:17.125210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-04-25T00:01:17.821873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8b6ddcb2-1ad9-4167-961c-bd6936bb9060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6809","nonce":3751765682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6811","nonce":3751765682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6815","nonce":3751765682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6813","nonce":3751765682}]},"public_addr":"172.21.15.116:6809/3751765682","cluster_addr":"172.21.15.116:6811/3751765682","heartbeat_back_addr":"172.21.15.116:6815/3751765682","heartbeat_front_addr":"172.21.15.116:6813/3751765682","state":["exists","up"]},{"osd":1,"uuid":"a6431768-994f-41da-8853-886f5687fb91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6801","nonce":377879124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6803","nonce":377879124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6807","nonce":377879124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6805","nonce":377879124}]},"public_addr":"172.21.15.116:6801/377879124","cluster_addr":"172.21.15.116:6803/377879124","heartbeat_back_addr":"172.21.15.116:6807/377879124","heartbeat_front_addr":"172.21.15.116:6805/377879124","state":["exists","up"]},{"osd":2,"uuid":"4f69bd40-5a23-45ff-8693-e1de75e1aa9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6817","nonce":1099282267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6819","nonce":1099282267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6823","nonce":1099282267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6821","nonce":1099282267}]},"public_addr":"172.21.15.116:6817/1099282267","cluster_addr":"172.21.15.116:6819/1099282267","heartbeat_back_addr":"172.21.15.116:6823/1099282267","heartbeat_front_addr":"172.21.15.116:6821/1099282267","state":["exists","up"]},{"osd":3,"uuid":"6a0d5344-8b6f-40d3-918d-0f0a4be688e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6825","nonce":1179942661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6827","nonce":1179942661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6831","nonce":1179942661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6829","nonce":1179942661}]},"public_addr":"172.21.15.116:6825/1179942661","cluster_addr":"172.21.15.116:6827/1179942661","heartbeat_back_addr":"172.21.15.116:6831/1179942661","heartbeat_front_addr":"172.21.15.116:6829/1179942661","state":["exists","up"]},{"osd":4,"uuid":"132e92e4-b825-441e-965a-ca00533fa86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6801","nonce":3132444330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6803","nonce":3132444330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6807","nonce":3132444330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6805","nonce":3132444330}]},"public_addr":"172.21.15.174:6801/3132444330","cluster_addr":"172.21.15.174:6803/3132444330","heartbeat_back_addr":"172.21.15.174:6807/3132444330","heartbeat_front_addr":"172.21.15.174:6805/3132444330","state":["exists","up"]},{"osd":5,"uuid":"a2eeb5b6-1b31-43cc-8306-a0adf3ee15bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6809","nonce":881817164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6811","nonce":881817164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6815","nonce":881817164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6813","nonce":881817164}]},"public_addr":"172.21.15.174:6809/881817164","cluster_addr":"172.21.15.174:6811/881817164","heartbeat_back_addr":"172.21.15.174:6815/881817164","heartbeat_front_addr":"172.21.15.174:6813/881817164","state":["exists","up"]},{"osd":6,"uuid":"588e873a-c841-4ba7-90f2-8956c87a9983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6825","nonce":117574251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6827","nonce":117574251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6831","nonce":117574251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6829","nonce":117574251}]},"public_addr":"172.21.15.174:6825/117574251","cluster_addr":"172.21.15.174:6827/117574251","heartbeat_back_addr":"172.21.15.174:6831/117574251","heartbeat_front_addr":"172.21.15.174:6829/117574251","state":["exists","up"]},{"osd":7,"uuid":"3b90f9b0-8a0e-4883-b088-ca1d4012870c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6817","nonce":2543899196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6819","nonce":2543899196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6823","nonce":2543899196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6821","nonce":2543899196}]},"public_addr":"172.21.15.174:6817/2543899196","cluster_addr":"172.21.15.174:6819/2543899196","heartbeat_back_addr":"172.21.15.174:6823/2543899196","heartbeat_front_addr":"172.21.15.174:6821/2543899196","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.293769+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.310034+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.404865+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.549658+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.192174+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.297266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.493649+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.452309+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:01:35.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.438+0000 7ff04df04640 1 -- 172.21.15.116:0/365201949 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff0340678f0 msgr2=0x7ff034069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:35.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.438+0000 7ff04df04640 1 --2- 172.21.15.116:0/365201949 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff0340678f0 0x7ff034069db0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff0380015b0 tx=0x7ff038009040 comp rx=0 tx=0).stop 2024-04-25T00:01:35.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.438+0000 7ff04df04640 1 -- 172.21.15.116:0/365201949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff048070dd0 msgr2=0x7ff0480c6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:35.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.438+0000 7ff04df04640 1 --2- 172.21.15.116:0/365201949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff048070dd0 0x7ff0480c6270 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ff03c010b40 tx=0x7ff03c00e360 comp rx=0 tx=0).stop 2024-04-25T00:01:35.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.438+0000 7ff04df04640 1 -- 172.21.15.116:0/365201949 shutdown_connections 2024-04-25T00:01:35.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.438+0000 7ff04df04640 1 --2- 172.21.15.116:0/365201949 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff0340678f0 0x7ff034069db0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.438+0000 7ff04df04640 1 --2- 172.21.15.116:0/365201949 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff048148890 0x7ff04814c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.438+0000 7ff04df04640 1 --2- 172.21.15.116:0/365201949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff048070dd0 0x7ff0480c6270 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.438+0000 7ff04df04640 1 --2- 172.21.15.116:0/365201949 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff04806e890 0x7ff04814e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.438+0000 7ff04df04640 1 -- 172.21.15.116:0/365201949 >> 172.21.15.116:0/365201949 conn(0x7ff0480c3b00 msgr2=0x7ff0480c4bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:35.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.438+0000 7ff04df04640 1 -- 172.21.15.116:0/365201949 shutdown_connections 2024-04-25T00:01:35.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.438+0000 7ff04df04640 1 -- 172.21.15.116:0/365201949 wait complete. 2024-04-25T00:01:35.457 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T00:00:53.324376+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'a_metadata', 'create_time': '2024-04-25T00:01:03.197326+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'a'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'a_data', 'create_time': '2024-04-25T00:01:03.728618+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'a'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'b_metadata', 'create_time': '2024-04-25T00:01:17.125210+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'b'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'b_data', 'create_time': '2024-04-25T00:01:17.821873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'b'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T00:01:35.457 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T00:01:35.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.574+0000 7f7f33bdf640 1 Processor -- start 2024-04-25T00:01:35.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.574+0000 7f7f33bdf640 1 -- start start 2024-04-25T00:01:35.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.574+0000 7f7f33bdf640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7f2c06d000 0x7f7f2c0c0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:35.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.574+0000 7f7f33bdf640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f2c155aa0 con 0x7f7f2c13edf0 2024-04-25T00:01:35.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.574+0000 7f7f33bdf640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f2c155d80 con 0x7f7f2c070da0 2024-04-25T00:01:35.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.574+0000 7f7f33bdf640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f7f2c0c0fa0 con 0x7f7f2c06d000 2024-04-25T00:01:35.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.574+0000 7f7f3117a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7f2c06d000 0x7f7f2c0c0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:35.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.574+0000 7f7f3197b640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f7f2c13edf0 0x7f7f2c0c2930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:55340/0 (socket says 172.21.15.116:55340) 2024-04-25T00:01:35.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.574+0000 7f7f3197b640 1 -- 172.21.15.116:0/2714081911 learned_addr learned my addr 172.21.15.116:0/2714081911 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:35.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.574+0000 7f7f3117a640 1 -- 172.21.15.116:0/2714081911 >> v1:172.21.15.174:6789/0 conn(0x7f7f2c070da0 legacy=0x7f7f2c0c0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:35.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.574+0000 7f7f3117a640 1 -- 172.21.15.116:0/2714081911 >> v1:172.21.15.116:6789/0 conn(0x7f7f2c13edf0 legacy=0x7f7f2c0c2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:35.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.574+0000 7f7f3117a640 1 -- 172.21.15.116:0/2714081911 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f2c156060 con 0x7f7f2c06d000 2024-04-25T00:01:35.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.574+0000 7f7f30979640 1 -- 172.21.15.116:0/2714081911 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 945876427 0 0) 0x7f7f2c155aa0 con 0x7f7f2c13edf0 2024-04-25T00:01:35.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.578+0000 7f7f3117a640 1 --2- 172.21.15.116:0/2714081911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f2c06d000 0x7f7f2c0c0350 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f7f1c0097c0 tx=0x7f7f1c01c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e9766f8d176969a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:35.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.582+0000 7f7f30979640 1 -- 172.21.15.116:0/2714081911 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f1c0b0070 con 0x7f7f2c06d000 2024-04-25T00:01:35.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.582+0000 7f7f30979640 1 -- 172.21.15.116:0/2714081911 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f1c0ab600 con 0x7f7f2c06d000 2024-04-25T00:01:35.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.582+0000 7f7f30979640 1 -- 172.21.15.116:0/2714081911 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f1c0ab9d0 con 0x7f7f2c06d000 2024-04-25T00:01:35.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.582+0000 7f7f33bdf640 1 -- 172.21.15.116:0/2714081911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f2c06d000 msgr2=0x7f7f2c0c0350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:35.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.582+0000 7f7f33bdf640 1 --2- 172.21.15.116:0/2714081911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f2c06d000 0x7f7f2c0c0350 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f7f1c0097c0 tx=0x7f7f1c01c3d0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.582+0000 7f7f33bdf640 1 -- 172.21.15.116:0/2714081911 shutdown_connections 2024-04-25T00:01:35.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.582+0000 7f7f33bdf640 1 --2- 172.21.15.116:0/2714081911 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f2c06d000 0x7f7f2c0c0350 secure :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f7f1c0097c0 tx=0x7f7f1c01c3d0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.582+0000 7f7f33bdf640 1 -- 172.21.15.116:0/2714081911 >> 172.21.15.116:0/2714081911 conn(0x7f7f2c0c3a20 msgr2=0x7f7f2c0c3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:35.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.582+0000 7f7f33bdf640 1 -- 172.21.15.116:0/2714081911 shutdown_connections 2024-04-25T00:01:35.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.582+0000 7f7f33bdf640 1 -- 172.21.15.116:0/2714081911 wait complete. 2024-04-25T00:01:35.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.582+0000 7f7f33bdf640 1 Processor -- start 2024-04-25T00:01:35.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.582+0000 7f7f33bdf640 1 -- start start 2024-04-25T00:01:35.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.582+0000 7f7f33bdf640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7f2c070da0 0x7f7f2c0a9700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:35.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.582+0000 7f7f33bdf640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7f2c13edf0 0x7f7f2c0ab310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:35.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.582+0000 7f7f33bdf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f2c1f15a0 0x7f7f2c0ab850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:35.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.582+0000 7f7f33bdf640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7f2c0c3820 con 0x7f7f2c1f15a0 2024-04-25T00:01:35.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.582+0000 7f7f33bdf640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7f2c0c36a0 con 0x7f7f2c070da0 2024-04-25T00:01:35.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.582+0000 7f7f33bdf640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f7f2c163900 con 0x7f7f2c13edf0 2024-04-25T00:01:35.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.582+0000 7f7f3217c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f2c1f15a0 0x7f7f2c0ab850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:35.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.582+0000 7f7f3217c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f2c1f15a0 0x7f7f2c0ab850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42892/0 (socket says 172.21.15.116:42892) 2024-04-25T00:01:35.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.582+0000 7f7f3217c640 1 -- 172.21.15.116:0/529122761 learned_addr learned my addr 172.21.15.116:0/529122761 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:35.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.582+0000 7f7f3197b640 1 --2- 172.21.15.116:0/529122761 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7f2c070da0 0x7f7f2c0a9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:35.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.582+0000 7f7f3217c640 1 -- 172.21.15.116:0/529122761 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7f2c13edf0 msgr2=0x7f7f2c0ab310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:35.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.582+0000 7f7f3117a640 1 --2- 172.21.15.116:0/529122761 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7f2c13edf0 0x7f7f2c0ab310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:35.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.582+0000 7f7f3217c640 1 --2- 172.21.15.116:0/529122761 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7f2c13edf0 0x7f7f2c0ab310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.582+0000 7f7f3217c640 1 -- 172.21.15.116:0/529122761 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7f2c070da0 msgr2=0x7f7f2c0a9700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:35.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.582+0000 7f7f3217c640 1 --2- 172.21.15.116:0/529122761 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7f2c070da0 0x7f7f2c0a9700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.582+0000 7f7f3217c640 1 -- 172.21.15.116:0/529122761 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f2c0c6c80 con 0x7f7f2c1f15a0 2024-04-25T00:01:35.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.582+0000 7f7f3197b640 1 --2- 172.21.15.116:0/529122761 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7f2c070da0 0x7f7f2c0a9700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:35.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.586+0000 7f7f3217c640 1 --2- 172.21.15.116:0/529122761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f2c1f15a0 0x7f7f2c0ab850 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f7f2400c780 tx=0x7f7f2400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:35.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.586+0000 7f7f227fc640 1 -- 172.21.15.116:0/529122761 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f24017070 con 0x7f7f2c1f15a0 2024-04-25T00:01:35.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.586+0000 7f7f227fc640 1 -- 172.21.15.116:0/529122761 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f24003db0 con 0x7f7f2c1f15a0 2024-04-25T00:01:35.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.586+0000 7f7f33bdf640 1 -- 172.21.15.116:0/529122761 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f2c1ee380 con 0x7f7f2c1f15a0 2024-04-25T00:01:35.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.586+0000 7f7f227fc640 1 -- 172.21.15.116:0/529122761 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f240052b0 con 0x7f7f2c1f15a0 2024-04-25T00:01:35.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.586+0000 7f7f33bdf640 1 -- 172.21.15.116:0/529122761 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f2c1eec20 con 0x7f7f2c1f15a0 2024-04-25T00:01:35.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.586+0000 7f7f33bdf640 1 -- 172.21.15.116:0/529122761 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f2c1ee550 con 0x7f7f2c1f15a0 2024-04-25T00:01:35.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.586+0000 7f7f227fc640 1 -- 172.21.15.116:0/529122761 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7f2401a070 con 0x7f7f2c1f15a0 2024-04-25T00:01:35.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.586+0000 7f7f227fc640 1 --2- 172.21.15.116:0/529122761 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7f100678c0 0x7f7f10069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:35.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.586+0000 7f7f227fc640 1 -- 172.21.15.116:0/529122761 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7f7f24085970 con 0x7f7f2c1f15a0 2024-04-25T00:01:35.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.586+0000 7f7f3197b640 1 --2- 172.21.15.116:0/529122761 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7f100678c0 0x7f7f10069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:35.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.586+0000 7f7f3197b640 1 --2- 172.21.15.116:0/529122761 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7f100678c0 0x7f7f10069d80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7f28000f70 tx=0x7f7f28009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:35.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.590+0000 7f7f227fc640 1 -- 172.21.15.116:0/529122761 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f7f24021020 con 0x7f7f2c1f15a0 2024-04-25T00:01:35.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.742+0000 7f7f33bdf640 1 -- 172.21.15.116:0/529122761 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7f2c04f510 con 0x7f7f2c1f15a0 2024-04-25T00:01:35.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.746+0000 7f7f227fc640 1 -- 172.21.15.116:0/529122761 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f7f24056260 con 0x7f7f2c1f15a0 2024-04-25T00:01:35.746 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-04-25T00:01:35.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.746+0000 7f7f33bdf640 1 -- 172.21.15.116:0/529122761 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7f100678c0 msgr2=0x7f7f10069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:35.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.746+0000 7f7f33bdf640 1 --2- 172.21.15.116:0/529122761 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7f100678c0 0x7f7f10069d80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7f28000f70 tx=0x7f7f28009040 comp rx=0 tx=0).stop 2024-04-25T00:01:35.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.746+0000 7f7f33bdf640 1 -- 172.21.15.116:0/529122761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f2c1f15a0 msgr2=0x7f7f2c0ab850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:35.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.746+0000 7f7f33bdf640 1 --2- 172.21.15.116:0/529122761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f2c1f15a0 0x7f7f2c0ab850 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f7f2400c780 tx=0x7f7f2400cc40 comp rx=0 tx=0).stop 2024-04-25T00:01:35.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.746+0000 7f7f33bdf640 1 -- 172.21.15.116:0/529122761 shutdown_connections 2024-04-25T00:01:35.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.746+0000 7f7f33bdf640 1 --2- 172.21.15.116:0/529122761 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7f100678c0 0x7f7f10069d80 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.746+0000 7f7f33bdf640 1 --2- 172.21.15.116:0/529122761 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f2c1f15a0 0x7f7f2c0ab850 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.746+0000 7f7f33bdf640 1 --2- 172.21.15.116:0/529122761 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7f2c13edf0 0x7f7f2c0ab310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.746+0000 7f7f33bdf640 1 --2- 172.21.15.116:0/529122761 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7f2c070da0 0x7f7f2c0a9700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.746+0000 7f7f33bdf640 1 -- 172.21.15.116:0/529122761 >> 172.21.15.116:0/529122761 conn(0x7f7f2c0c3a20 msgr2=0x7f7f2c0c0350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:35.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.746+0000 7f7f33bdf640 1 -- 172.21.15.116:0/529122761 shutdown_connections 2024-04-25T00:01:35.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.746+0000 7f7f33bdf640 1 -- 172.21.15.116:0/529122761 wait complete. 2024-04-25T00:01:35.763 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_metadata pg_num 2024-04-25T00:01:35.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.870+0000 7f43dff20640 1 Processor -- start 2024-04-25T00:01:35.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.870+0000 7f43dff20640 1 -- start start 2024-04-25T00:01:35.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.870+0000 7f43dff20640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43d8148f20 0x7f43d80c2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:35.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.870+0000 7f43dff20640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f43d806d510 0x7f43d8072500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:35.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.870+0000 7f43dff20640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f43d806d000 0x7f43d8072a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:35.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.870+0000 7f43dff20640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f43d8153350 con 0x7f43d806d000 2024-04-25T00:01:35.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.870+0000 7f43dff20640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f43d8149300 con 0x7f43d806d510 2024-04-25T00:01:35.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.870+0000 7f43dff20640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f43d80a95d0 con 0x7f43d8148f20 2024-04-25T00:01:35.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.870+0000 7f43de4bd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f43d806d000 0x7f43d8072a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:35.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.870+0000 7f43de4bd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f43d806d000 0x7f43d8072a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42900/0 (socket says 172.21.15.116:42900) 2024-04-25T00:01:35.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.874+0000 7f43de4bd640 1 -- 172.21.15.116:0/565086681 learned_addr learned my addr 172.21.15.116:0/565086681 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:35.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.874+0000 7f43ddcbc640 1 --2- 172.21.15.116:0/565086681 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43d8148f20 0x7f43d80c2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:35.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.874+0000 7f43dd4bb640 1 --2- 172.21.15.116:0/565086681 >> v2:172.21.15.174:3300/0 conn(0x7f43d806d510 0x7f43d8072500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:35.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.874+0000 7f43de4bd640 1 -- 172.21.15.116:0/565086681 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43d8148f20 msgr2=0x7f43d80c2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:35.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.874+0000 7f43de4bd640 1 --2- 172.21.15.116:0/565086681 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43d8148f20 0x7f43d80c2930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.874+0000 7f43de4bd640 1 -- 172.21.15.116:0/565086681 >> v2:172.21.15.174:3300/0 conn(0x7f43d806d510 msgr2=0x7f43d8072500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:35.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.874+0000 7f43de4bd640 1 --2- 172.21.15.116:0/565086681 >> v2:172.21.15.174:3300/0 conn(0x7f43d806d510 0x7f43d8072500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.874+0000 7f43de4bd640 1 -- 172.21.15.116:0/565086681 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43d8153070 con 0x7f43d806d000 2024-04-25T00:01:35.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.874+0000 7f43dd4bb640 1 --2- 172.21.15.116:0/565086681 >> v2:172.21.15.174:3300/0 conn(0x7f43d806d510 0x7f43d8072500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:35.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.874+0000 7f43de4bd640 1 --2- 172.21.15.116:0/565086681 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43d806d000 0x7f43d8072a40 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f43d8068110 tx=0x7f43d001c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3136523b060323b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:35.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.874+0000 7f43dccba640 1 -- 172.21.15.116:0/565086681 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d00a5550 con 0x7f43d806d000 2024-04-25T00:01:35.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.874+0000 7f43dccba640 1 -- 172.21.15.116:0/565086681 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43d00a56f0 con 0x7f43d806d000 2024-04-25T00:01:35.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.874+0000 7f43dccba640 1 -- 172.21.15.116:0/565086681 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d00a5ac0 con 0x7f43d806d000 2024-04-25T00:01:35.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.874+0000 7f43dff20640 1 -- 172.21.15.116:0/565086681 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43d806d000 msgr2=0x7f43d8072a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:35.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.874+0000 7f43dff20640 1 --2- 172.21.15.116:0/565086681 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43d806d000 0x7f43d8072a40 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f43d8068110 tx=0x7f43d001c8d0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.874+0000 7f43dff20640 1 -- 172.21.15.116:0/565086681 shutdown_connections 2024-04-25T00:01:35.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.874+0000 7f43dff20640 1 --2- 172.21.15.116:0/565086681 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43d806d000 0x7f43d8072a40 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.874+0000 7f43dff20640 1 --2- 172.21.15.116:0/565086681 >> v2:172.21.15.174:3300/0 conn(0x7f43d806d510 0x7f43d8072500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.874+0000 7f43dff20640 1 --2- 172.21.15.116:0/565086681 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43d8148f20 0x7f43d80c2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.874+0000 7f43dff20640 1 -- 172.21.15.116:0/565086681 >> 172.21.15.116:0/565086681 conn(0x7f43d80c3a20 msgr2=0x7f43d80c3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:35.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.874+0000 7f43dff20640 1 -- 172.21.15.116:0/565086681 shutdown_connections 2024-04-25T00:01:35.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.874+0000 7f43dff20640 1 -- 172.21.15.116:0/565086681 wait complete. 2024-04-25T00:01:35.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.878+0000 7f43dff20640 1 Processor -- start 2024-04-25T00:01:35.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.878+0000 7f43dff20640 1 -- start start 2024-04-25T00:01:35.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.878+0000 7f43dff20640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43d806d000 0x7f43d815b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:35.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.878+0000 7f43dff20640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43d806d510 0x7f43d80ac070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:35.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.878+0000 7f43dff20640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43d8148f20 0x7f43d80ac5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:35.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.878+0000 7f43dff20640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f43d81563d0 con 0x7f43d8148f20 2024-04-25T00:01:35.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.878+0000 7f43dff20640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f43d8156250 con 0x7f43d806d000 2024-04-25T00:01:35.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.878+0000 7f43dff20640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f43d8156550 con 0x7f43d806d510 2024-04-25T00:01:35.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.878+0000 7f43de4bd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43d8148f20 0x7f43d80ac5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:35.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.878+0000 7f43de4bd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43d8148f20 0x7f43d80ac5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42910/0 (socket says 172.21.15.116:42910) 2024-04-25T00:01:35.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.878+0000 7f43de4bd640 1 -- 172.21.15.116:0/300382710 learned_addr learned my addr 172.21.15.116:0/300382710 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:35.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.878+0000 7f43dd4bb640 1 --2- 172.21.15.116:0/300382710 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43d806d510 0x7f43d80ac070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:35.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.878+0000 7f43ddcbc640 1 --2- 172.21.15.116:0/300382710 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43d806d000 0x7f43d815b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:35.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.878+0000 7f43de4bd640 1 -- 172.21.15.116:0/300382710 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43d806d510 msgr2=0x7f43d80ac070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:35.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.878+0000 7f43de4bd640 1 --2- 172.21.15.116:0/300382710 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43d806d510 0x7f43d80ac070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.878+0000 7f43de4bd640 1 -- 172.21.15.116:0/300382710 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43d806d000 msgr2=0x7f43d815b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:35.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.878+0000 7f43de4bd640 1 --2- 172.21.15.116:0/300382710 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43d806d000 0x7f43d815b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:35.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.878+0000 7f43de4bd640 1 -- 172.21.15.116:0/300382710 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43d81ee4f0 con 0x7f43d8148f20 2024-04-25T00:01:35.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.878+0000 7f43de4bd640 1 --2- 172.21.15.116:0/300382710 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43d8148f20 0x7f43d80ac5b0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f43d00af040 tx=0x7f43d0019670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:35.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.878+0000 7f43ce7fc640 1 -- 172.21.15.116:0/300382710 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d00bc020 con 0x7f43d8148f20 2024-04-25T00:01:35.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.878+0000 7f43ce7fc640 1 -- 172.21.15.116:0/300382710 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43d00b8670 con 0x7f43d8148f20 2024-04-25T00:01:35.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.878+0000 7f43dff20640 1 -- 172.21.15.116:0/300382710 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43d81ee7d0 con 0x7f43d8148f20 2024-04-25T00:01:35.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.878+0000 7f43ce7fc640 1 -- 172.21.15.116:0/300382710 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d0009510 con 0x7f43d8148f20 2024-04-25T00:01:35.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.878+0000 7f43dff20640 1 -- 172.21.15.116:0/300382710 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43d81eed90 con 0x7f43d8148f20 2024-04-25T00:01:35.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.878+0000 7f43dff20640 1 -- 172.21.15.116:0/300382710 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43d804f510 con 0x7f43d8148f20 2024-04-25T00:01:35.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.878+0000 7f43ce7fc640 1 -- 172.21.15.116:0/300382710 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f43d00bf020 con 0x7f43d8148f20 2024-04-25T00:01:35.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.878+0000 7f43ce7fc640 1 --2- 172.21.15.116:0/300382710 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f43bc0678c0 0x7f43bc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:35.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.878+0000 7f43ce7fc640 1 -- 172.21.15.116:0/300382710 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7f43d0120f90 con 0x7f43d8148f20 2024-04-25T00:01:35.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.878+0000 7f43ddcbc640 1 --2- 172.21.15.116:0/300382710 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f43bc0678c0 0x7f43bc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:35.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.882+0000 7f43ddcbc640 1 --2- 172.21.15.116:0/300382710 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f43bc0678c0 0x7f43bc069d80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f43d400b7c0 tx=0x7f43d4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:35.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:35.882+0000 7f43ce7fc640 1 -- 172.21.15.116:0/300382710 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f43d00c6030 con 0x7f43d8148f20 2024-04-25T00:01:36.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.030+0000 7f43dff20640 1 -- 172.21.15.116:0/300382710 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"} v 0) -- 0x7f43d80c34b0 con 0x7f43d8148f20 2024-04-25T00:01:36.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.030+0000 7f43ce7fc640 1 -- 172.21.15.116:0/300382710 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"}]=0 v27) ==== 99+0+11 (secure 0 0 0) 0x7f43d00165e0 con 0x7f43d8148f20 2024-04-25T00:01:36.034 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-04-25T00:01:36.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.034+0000 7f43dff20640 1 -- 172.21.15.116:0/300382710 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f43bc0678c0 msgr2=0x7f43bc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:36.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.034+0000 7f43dff20640 1 --2- 172.21.15.116:0/300382710 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f43bc0678c0 0x7f43bc069d80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f43d400b7c0 tx=0x7f43d4008040 comp rx=0 tx=0).stop 2024-04-25T00:01:36.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.034+0000 7f43dff20640 1 -- 172.21.15.116:0/300382710 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43d8148f20 msgr2=0x7f43d80ac5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:36.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.034+0000 7f43dff20640 1 --2- 172.21.15.116:0/300382710 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43d8148f20 0x7f43d80ac5b0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f43d00af040 tx=0x7f43d0019670 comp rx=0 tx=0).stop 2024-04-25T00:01:36.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.034+0000 7f43dff20640 1 -- 172.21.15.116:0/300382710 shutdown_connections 2024-04-25T00:01:36.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.034+0000 7f43dff20640 1 --2- 172.21.15.116:0/300382710 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f43bc0678c0 0x7f43bc069d80 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:36.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.034+0000 7f43dff20640 1 --2- 172.21.15.116:0/300382710 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43d8148f20 0x7f43d80ac5b0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:36.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.034+0000 7f43dff20640 1 --2- 172.21.15.116:0/300382710 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43d806d510 0x7f43d80ac070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:36.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.034+0000 7f43dff20640 1 --2- 172.21.15.116:0/300382710 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43d806d000 0x7f43d815b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:36.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.034+0000 7f43dff20640 1 -- 172.21.15.116:0/300382710 >> 172.21.15.116:0/300382710 conn(0x7f43d80c3a20 msgr2=0x7f43d80c3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:36.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.034+0000 7f43dff20640 1 -- 172.21.15.116:0/300382710 shutdown_connections 2024-04-25T00:01:36.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.034+0000 7f43dff20640 1 -- 172.21.15.116:0/300382710 wait complete. 2024-04-25T00:01:36.051 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_data pg_num 2024-04-25T00:01:36.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.165+0000 7f6dab827640 1 Processor -- start 2024-04-25T00:01:36.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.165+0000 7f6dab827640 1 -- start start 2024-04-25T00:01:36.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.165+0000 7f6dab827640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f6da4148f20 0x7f6da40c2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:36.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.165+0000 7f6dab827640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6da406d000 0x7f6da415cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:36.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.165+0000 7f6dab827640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6da4149300 con 0x7f6da406d510 2024-04-25T00:01:36.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.165+0000 7f6dab827640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f6da4153350 con 0x7f6da4148f20 2024-04-25T00:01:36.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.165+0000 7f6dab827640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6da40a95d0 con 0x7f6da406d000 2024-04-25T00:01:36.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.169+0000 7f6da95c3640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f6da4148f20 0x7f6da40c2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:36.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.169+0000 7f6da9dc4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6da406d000 0x7f6da415cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:36.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.169+0000 7f6da95c3640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f6da4148f20 0x7f6da40c2930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:39896/0 (socket says 172.21.15.116:39896) 2024-04-25T00:01:36.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.169+0000 7f6da95c3640 1 -- 172.21.15.116:0/3338915562 learned_addr learned my addr 172.21.15.116:0/3338915562 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:36.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.169+0000 7f6da95c3640 1 -- 172.21.15.116:0/3338915562 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6da406d000 msgr2=0x7f6da415cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:36.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.169+0000 7f6da95c3640 1 --2- 172.21.15.116:0/3338915562 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6da406d000 0x7f6da415cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:36.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.169+0000 7f6da95c3640 1 -- 172.21.15.116:0/3338915562 >> v1:172.21.15.174:6789/0 conn(0x7f6da406d510 legacy=0x7f6da4072500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:36.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.169+0000 7f6da95c3640 1 -- 172.21.15.116:0/3338915562 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6da4153070 con 0x7f6da4148f20 2024-04-25T00:01:36.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.169+0000 7f6da9dc4640 1 --2- 172.21.15.116:0/3338915562 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6da406d000 0x7f6da415cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:01:36.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.169+0000 7f6da95c3640 1 --2- 172.21.15.116:0/3338915562 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6da4148f20 0x7f6da40c2930 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6da0009870 tx=0x7f6da001c440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6571f6a2a42822c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:36.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.169+0000 7f6d9bfff640 1 -- 172.21.15.116:0/3338915562 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da00b0070 con 0x7f6da4148f20 2024-04-25T00:01:36.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.173+0000 7f6d9bfff640 1 -- 172.21.15.116:0/3338915562 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6da0004230 con 0x7f6da4148f20 2024-04-25T00:01:36.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.173+0000 7f6d9bfff640 1 -- 172.21.15.116:0/3338915562 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da0003620 con 0x7f6da4148f20 2024-04-25T00:01:36.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.173+0000 7f6dab827640 1 -- 172.21.15.116:0/3338915562 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6da4148f20 msgr2=0x7f6da40c2930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:36.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.173+0000 7f6dab827640 1 --2- 172.21.15.116:0/3338915562 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6da4148f20 0x7f6da40c2930 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6da0009870 tx=0x7f6da001c440 comp rx=0 tx=0).stop 2024-04-25T00:01:36.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.173+0000 7f6dab827640 1 -- 172.21.15.116:0/3338915562 shutdown_connections 2024-04-25T00:01:36.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.173+0000 7f6dab827640 1 --2- 172.21.15.116:0/3338915562 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6da406d000 0x7f6da415cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:36.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.173+0000 7f6dab827640 1 --2- 172.21.15.116:0/3338915562 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6da4148f20 0x7f6da40c2930 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:36.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.173+0000 7f6dab827640 1 -- 172.21.15.116:0/3338915562 >> 172.21.15.116:0/3338915562 conn(0x7f6da40c3a20 msgr2=0x7f6da40c3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:36.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.173+0000 7f6dab827640 1 -- 172.21.15.116:0/3338915562 shutdown_connections 2024-04-25T00:01:36.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.173+0000 7f6dab827640 1 -- 172.21.15.116:0/3338915562 wait complete. 2024-04-25T00:01:36.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.173+0000 7f6dab827640 1 Processor -- start 2024-04-25T00:01:36.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.173+0000 7f6dab827640 1 -- start start 2024-04-25T00:01:36.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.173+0000 7f6dab827640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6da406d000 0x7f6da41edca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:36.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.173+0000 7f6dab827640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6da406d510 0x7f6da41eb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:36.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.173+0000 7f6dab827640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6da4148f20 0x7f6da41ebef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:36.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.173+0000 7f6dab827640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6da4070f20 con 0x7f6da406d510 2024-04-25T00:01:36.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.173+0000 7f6dab827640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6da4070da0 con 0x7f6da4148f20 2024-04-25T00:01:36.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.173+0000 7f6dab827640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6da40710a0 con 0x7f6da406d000 2024-04-25T00:01:36.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.173+0000 7f6da8dc2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6da406d510 0x7f6da41eb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:36.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.173+0000 7f6da8dc2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6da406d510 0x7f6da41eb9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42922/0 (socket says 172.21.15.116:42922) 2024-04-25T00:01:36.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.173+0000 7f6da8dc2640 1 -- 172.21.15.116:0/3737060659 learned_addr learned my addr 172.21.15.116:0/3737060659 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:36.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.173+0000 7f6da9dc4640 1 --2- 172.21.15.116:0/3737060659 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6da4148f20 0x7f6da41ebef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:36.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.173+0000 7f6da95c3640 1 --2- 172.21.15.116:0/3737060659 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6da406d000 0x7f6da41edca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:36.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.173+0000 7f6da8dc2640 1 -- 172.21.15.116:0/3737060659 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6da406d000 msgr2=0x7f6da41edca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:36.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.173+0000 7f6da8dc2640 1 --2- 172.21.15.116:0/3737060659 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6da406d000 0x7f6da41edca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:36.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.173+0000 7f6da8dc2640 1 -- 172.21.15.116:0/3737060659 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6da4148f20 msgr2=0x7f6da41ebef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:36.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.173+0000 7f6da8dc2640 1 --2- 172.21.15.116:0/3737060659 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6da4148f20 0x7f6da41ebef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:36.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.173+0000 7f6da8dc2640 1 -- 172.21.15.116:0/3737060659 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6da41ee730 con 0x7f6da406d510 2024-04-25T00:01:36.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.173+0000 7f6da9dc4640 1 --2- 172.21.15.116:0/3737060659 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6da4148f20 0x7f6da41ebef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:36.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.173+0000 7f6da8dc2640 1 --2- 172.21.15.116:0/3737060659 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6da406d510 0x7f6da41eb9b0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6d9400c420 tx=0x7f6d9400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:36.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.173+0000 7f6d99ffb640 1 -- 172.21.15.116:0/3737060659 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d940052d0 con 0x7f6da406d510 2024-04-25T00:01:36.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.173+0000 7f6d99ffb640 1 -- 172.21.15.116:0/3737060659 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d94005470 con 0x7f6da406d510 2024-04-25T00:01:36.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.173+0000 7f6dab827640 1 -- 172.21.15.116:0/3737060659 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6da4149890 con 0x7f6da406d510 2024-04-25T00:01:36.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.173+0000 7f6d99ffb640 1 -- 172.21.15.116:0/3737060659 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d94007260 con 0x7f6da406d510 2024-04-25T00:01:36.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.173+0000 7f6dab827640 1 -- 172.21.15.116:0/3737060659 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6da406e900 con 0x7f6da406d510 2024-04-25T00:01:36.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.177+0000 7f6dab827640 1 -- 172.21.15.116:0/3737060659 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6da404f510 con 0x7f6da406d510 2024-04-25T00:01:36.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.177+0000 7f6d99ffb640 1 -- 172.21.15.116:0/3737060659 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6d94007400 con 0x7f6da406d510 2024-04-25T00:01:36.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.177+0000 7f6d99ffb640 1 --2- 172.21.15.116:0/3737060659 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6d880678f0 0x7f6d88069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:36.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.177+0000 7f6d99ffb640 1 -- 172.21.15.116:0/3737060659 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7f6d94085400 con 0x7f6da406d510 2024-04-25T00:01:36.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.177+0000 7f6da95c3640 1 --2- 172.21.15.116:0/3737060659 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6d880678f0 0x7f6d88069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:36.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.177+0000 7f6da95c3640 1 --2- 172.21.15.116:0/3737060659 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6d880678f0 0x7f6d88069db0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6da00ad000 tx=0x7f6da00a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:36.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.181+0000 7f6d99ffb640 1 -- 172.21.15.116:0/3737060659 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f6da4149300 con 0x7f6da406d510 2024-04-25T00:01:36.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.341+0000 7f6dab827640 1 -- 172.21.15.116:0/3737060659 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"} v 0) -- 0x7f6da40c34b0 con 0x7f6da406d510 2024-04-25T00:01:36.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.341+0000 7f6d99ffb640 1 -- 172.21.15.116:0/3737060659 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"}]=0 v27) ==== 95+0+11 (secure 0 0 0) 0x7f6d94055ca0 con 0x7f6da406d510 2024-04-25T00:01:36.344 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-04-25T00:01:36.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.341+0000 7f6dab827640 1 -- 172.21.15.116:0/3737060659 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6d880678f0 msgr2=0x7f6d88069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:36.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.341+0000 7f6dab827640 1 --2- 172.21.15.116:0/3737060659 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6d880678f0 0x7f6d88069db0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f6da00ad000 tx=0x7f6da00a6000 comp rx=0 tx=0).stop 2024-04-25T00:01:36.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.341+0000 7f6dab827640 1 -- 172.21.15.116:0/3737060659 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6da406d510 msgr2=0x7f6da41eb9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:36.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.341+0000 7f6dab827640 1 --2- 172.21.15.116:0/3737060659 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6da406d510 0x7f6da41eb9b0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6d9400c420 tx=0x7f6d9400c8e0 comp rx=0 tx=0).stop 2024-04-25T00:01:36.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.341+0000 7f6dab827640 1 -- 172.21.15.116:0/3737060659 shutdown_connections 2024-04-25T00:01:36.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.341+0000 7f6dab827640 1 --2- 172.21.15.116:0/3737060659 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6d880678f0 0x7f6d88069db0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:36.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.341+0000 7f6dab827640 1 --2- 172.21.15.116:0/3737060659 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6da4148f20 0x7f6da41ebef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:36.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.341+0000 7f6dab827640 1 --2- 172.21.15.116:0/3737060659 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6da406d510 0x7f6da41eb9b0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:36.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.341+0000 7f6dab827640 1 --2- 172.21.15.116:0/3737060659 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6da406d000 0x7f6da41edca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:36.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.341+0000 7f6dab827640 1 -- 172.21.15.116:0/3737060659 >> 172.21.15.116:0/3737060659 conn(0x7f6da40c3a20 msgr2=0x7f6da41475c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:36.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.341+0000 7f6dab827640 1 -- 172.21.15.116:0/3737060659 shutdown_connections 2024-04-25T00:01:36.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.341+0000 7f6dab827640 1 -- 172.21.15.116:0/3737060659 wait complete. 2024-04-25T00:01:36.363 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_metadata pg_num 2024-04-25T00:01:36.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.477+0000 7f4ff1071640 1 Processor -- start 2024-04-25T00:01:36.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.477+0000 7f4ff1071640 1 -- start start 2024-04-25T00:01:36.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.481+0000 7f4ff1071640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4fec13edf0 0x7f4fec0c2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:36.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.481+0000 7f4ff1071640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4fec06d000 0x7f4fec0c0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:36.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.481+0000 7f4ff1071640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f4fec070da0 0x7f4fec0c0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:36.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.481+0000 7f4ff1071640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f4fec155aa0 con 0x7f4fec06d000 2024-04-25T00:01:36.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.481+0000 7f4ff1071640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f4fec155d80 con 0x7f4fec070da0 2024-04-25T00:01:36.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.481+0000 7f4ff1071640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f4fec156060 con 0x7f4fec13edf0 2024-04-25T00:01:36.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.481+0000 7f4fea59c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4fec06d000 0x7f4fec0c0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:36.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.481+0000 7f4fea59c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4fec06d000 0x7f4fec0c0350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42936/0 (socket says 172.21.15.116:42936) 2024-04-25T00:01:36.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.481+0000 7f4fea59c640 1 -- 172.21.15.116:0/3247364161 learned_addr learned my addr 172.21.15.116:0/3247364161 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:36.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.481+0000 7f4fead9d640 1 --2- 172.21.15.116:0/3247364161 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4fec13edf0 0x7f4fec0c2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:36.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.481+0000 7f4fea59c640 1 -- 172.21.15.116:0/3247364161 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4fec13edf0 msgr2=0x7f4fec0c2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:36.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.481+0000 7f4fea59c640 1 --2- 172.21.15.116:0/3247364161 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4fec13edf0 0x7f4fec0c2930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:36.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.481+0000 7f4feb59e640 1 --2- 172.21.15.116:0/3247364161 >> v2:172.21.15.174:3300/0 conn(0x7f4fec070da0 0x7f4fec0c0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:36.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.481+0000 7f4fea59c640 1 -- 172.21.15.116:0/3247364161 >> v2:172.21.15.174:3300/0 conn(0x7f4fec070da0 msgr2=0x7f4fec0c0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:36.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.481+0000 7f4fea59c640 1 --2- 172.21.15.116:0/3247364161 >> v2:172.21.15.174:3300/0 conn(0x7f4fec070da0 0x7f4fec0c0890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:36.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.481+0000 7f4fea59c640 1 -- 172.21.15.116:0/3247364161 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fec156340 con 0x7f4fec06d000 2024-04-25T00:01:36.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.481+0000 7f4fead9d640 1 --2- 172.21.15.116:0/3247364161 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4fec13edf0 0x7f4fec0c2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:36.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.481+0000 7f4fea59c640 1 --2- 172.21.15.116:0/3247364161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fec06d000 0x7f4fec0c0350 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f4fec070b60 tx=0x7f4fe001c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eadae124647e8517 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:36.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.481+0000 7f4fe9d9b640 1 -- 172.21.15.116:0/3247364161 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fe00b2070 con 0x7f4fec06d000 2024-04-25T00:01:36.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.481+0000 7f4fe9d9b640 1 -- 172.21.15.116:0/3247364161 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fe00a9070 con 0x7f4fec06d000 2024-04-25T00:01:36.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.481+0000 7f4fe9d9b640 1 -- 172.21.15.116:0/3247364161 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fe00ad880 con 0x7f4fec06d000 2024-04-25T00:01:36.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.481+0000 7f4ff1071640 1 -- 172.21.15.116:0/3247364161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fec06d000 msgr2=0x7f4fec0c0350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:36.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.481+0000 7f4ff1071640 1 --2- 172.21.15.116:0/3247364161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fec06d000 0x7f4fec0c0350 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f4fec070b60 tx=0x7f4fe001c490 comp rx=0 tx=0).stop 2024-04-25T00:01:36.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.485+0000 7f4ff1071640 1 -- 172.21.15.116:0/3247364161 shutdown_connections 2024-04-25T00:01:36.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.485+0000 7f4ff1071640 1 --2- 172.21.15.116:0/3247364161 >> v2:172.21.15.174:3300/0 conn(0x7f4fec070da0 0x7f4fec0c0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:36.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.485+0000 7f4ff1071640 1 --2- 172.21.15.116:0/3247364161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fec06d000 0x7f4fec0c0350 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:36.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.485+0000 7f4ff1071640 1 --2- 172.21.15.116:0/3247364161 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4fec13edf0 0x7f4fec0c2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:36.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.485+0000 7f4ff1071640 1 -- 172.21.15.116:0/3247364161 >> 172.21.15.116:0/3247364161 conn(0x7f4fec0c3a20 msgr2=0x7f4fec0c3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:36.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.485+0000 7f4ff1071640 1 -- 172.21.15.116:0/3247364161 shutdown_connections 2024-04-25T00:01:36.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.485+0000 7f4ff1071640 1 -- 172.21.15.116:0/3247364161 wait complete. 2024-04-25T00:01:36.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.485+0000 7f4ff1071640 1 Processor -- start 2024-04-25T00:01:36.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.485+0000 7f4ff1071640 1 -- start start 2024-04-25T00:01:36.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.485+0000 7f4ff1071640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4fec06d000 0x7f4fec13cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:36.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.485+0000 7f4ff1071640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fec070da0 0x7f4fec0712b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:36.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.485+0000 7f4ff1071640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4fec13edf0 0x7f4fec1365e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:36.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.485+0000 7f4ff1071640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4fec159550 con 0x7f4fec070da0 2024-04-25T00:01:36.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.485+0000 7f4ff1071640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4fec0c0fe0 con 0x7f4fec06d000 2024-04-25T00:01:36.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.485+0000 7f4ff1071640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f4fec1596d0 con 0x7f4fec13edf0 2024-04-25T00:01:36.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.485+0000 7f4fea59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fec070da0 0x7f4fec0712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:36.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.485+0000 7f4fea59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fec070da0 0x7f4fec0712b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42940/0 (socket says 172.21.15.116:42940) 2024-04-25T00:01:36.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.485+0000 7f4fea59c640 1 -- 172.21.15.116:0/1850534415 learned_addr learned my addr 172.21.15.116:0/1850534415 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:36.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.485+0000 7f4feb59e640 1 --2- 172.21.15.116:0/1850534415 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4fec13edf0 0x7f4fec1365e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:36.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.485+0000 7f4fea59c640 1 -- 172.21.15.116:0/1850534415 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4fec13edf0 msgr2=0x7f4fec1365e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:36.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.485+0000 7f4fead9d640 1 --2- 172.21.15.116:0/1850534415 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4fec06d000 0x7f4fec13cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:36.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.485+0000 7f4fea59c640 1 --2- 172.21.15.116:0/1850534415 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4fec13edf0 0x7f4fec1365e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:36.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.485+0000 7f4fea59c640 1 -- 172.21.15.116:0/1850534415 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4fec06d000 msgr2=0x7f4fec13cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:36.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.485+0000 7f4fea59c640 1 --2- 172.21.15.116:0/1850534415 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4fec06d000 0x7f4fec13cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:36.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.485+0000 7f4fea59c640 1 -- 172.21.15.116:0/1850534415 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fec1eeed0 con 0x7f4fec070da0 2024-04-25T00:01:36.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.485+0000 7f4fea59c640 1 --2- 172.21.15.116:0/1850534415 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fec070da0 0x7f4fec0712b0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f4fe001c970 tx=0x7f4fe0019230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:36.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.485+0000 7f4fd37fe640 1 -- 172.21.15.116:0/1850534415 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fe00b2040 con 0x7f4fec070da0 2024-04-25T00:01:36.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.485+0000 7f4fd37fe640 1 -- 172.21.15.116:0/1850534415 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fe00b6640 con 0x7f4fec070da0 2024-04-25T00:01:36.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.485+0000 7f4ff1071640 1 -- 172.21.15.116:0/1850534415 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fec0ac1b0 con 0x7f4fec070da0 2024-04-25T00:01:36.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.485+0000 7f4fd37fe640 1 -- 172.21.15.116:0/1850534415 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fe00b6a30 con 0x7f4fec070da0 2024-04-25T00:01:36.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.485+0000 7f4ff1071640 1 -- 172.21.15.116:0/1850534415 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fec1ef170 con 0x7f4fec070da0 2024-04-25T00:01:36.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.485+0000 7f4ff1071640 1 -- 172.21.15.116:0/1850534415 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fec04f510 con 0x7f4fec070da0 2024-04-25T00:01:36.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.489+0000 7f4fd37fe640 1 -- 172.21.15.116:0/1850534415 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4fe00ab070 con 0x7f4fec070da0 2024-04-25T00:01:36.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.489+0000 7f4fd37fe640 1 --2- 172.21.15.116:0/1850534415 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4fd80678f0 0x7f4fd8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:36.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.489+0000 7f4fd37fe640 1 -- 172.21.15.116:0/1850534415 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7f4fe011fd40 con 0x7f4fec070da0 2024-04-25T00:01:36.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.489+0000 7f4fead9d640 1 --2- 172.21.15.116:0/1850534415 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4fd80678f0 0x7f4fd8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:36.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.489+0000 7f4fead9d640 1 --2- 172.21.15.116:0/1850534415 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4fd80678f0 0x7f4fd8069db0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc00b7e0 tx=0x7f4fdc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:36.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.493+0000 7f4fd37fe640 1 -- 172.21.15.116:0/1850534415 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f4fe0016110 con 0x7f4fec070da0 2024-04-25T00:01:36.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.641+0000 7f4ff1071640 1 -- 172.21.15.116:0/1850534415 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"} v 0) -- 0x7f4fec0c3480 con 0x7f4fec070da0 2024-04-25T00:01:36.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.641+0000 7f4fd37fe640 1 -- 172.21.15.116:0/1850534415 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"}]=0 v27) ==== 99+0+11 (secure 0 0 0) 0x7f4fe00bd300 con 0x7f4fec070da0 2024-04-25T00:01:36.643 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-04-25T00:01:36.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.641+0000 7f4ff1071640 1 -- 172.21.15.116:0/1850534415 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4fd80678f0 msgr2=0x7f4fd8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:36.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.641+0000 7f4ff1071640 1 --2- 172.21.15.116:0/1850534415 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4fd80678f0 0x7f4fd8069db0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc00b7e0 tx=0x7f4fdc008040 comp rx=0 tx=0).stop 2024-04-25T00:01:36.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.641+0000 7f4ff1071640 1 -- 172.21.15.116:0/1850534415 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fec070da0 msgr2=0x7f4fec0712b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:36.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.641+0000 7f4ff1071640 1 --2- 172.21.15.116:0/1850534415 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fec070da0 0x7f4fec0712b0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f4fe001c970 tx=0x7f4fe0019230 comp rx=0 tx=0).stop 2024-04-25T00:01:36.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.641+0000 7f4ff1071640 1 -- 172.21.15.116:0/1850534415 shutdown_connections 2024-04-25T00:01:36.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.641+0000 7f4ff1071640 1 --2- 172.21.15.116:0/1850534415 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4fd80678f0 0x7f4fd8069db0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:36.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.641+0000 7f4ff1071640 1 --2- 172.21.15.116:0/1850534415 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4fec13edf0 0x7f4fec1365e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:36.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.641+0000 7f4ff1071640 1 --2- 172.21.15.116:0/1850534415 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fec070da0 0x7f4fec0712b0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:36.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.641+0000 7f4ff1071640 1 --2- 172.21.15.116:0/1850534415 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4fec06d000 0x7f4fec13cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:36.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.641+0000 7f4ff1071640 1 -- 172.21.15.116:0/1850534415 >> 172.21.15.116:0/1850534415 conn(0x7f4fec0c3a20 msgr2=0x7f4fec13a6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:36.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.641+0000 7f4ff1071640 1 -- 172.21.15.116:0/1850534415 shutdown_connections 2024-04-25T00:01:36.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.641+0000 7f4ff1071640 1 -- 172.21.15.116:0/1850534415 wait complete. 2024-04-25T00:01:36.660 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_data pg_num 2024-04-25T00:01:36.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.769+0000 7fbe9e01d640 1 Processor -- start 2024-04-25T00:01:36.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.769+0000 7fbe9e01d640 1 -- start start 2024-04-25T00:01:36.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.769+0000 7fbe9e01d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbe98148f20 0x7fbe980c2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:36.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.769+0000 7fbe9e01d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe98149300 con 0x7fbe9806d000 2024-04-25T00:01:36.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.769+0000 7fbe9e01d640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe980a95d0 con 0x7fbe9806d510 2024-04-25T00:01:36.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.769+0000 7fbe9e01d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fbe98153350 con 0x7fbe98148f20 2024-04-25T00:01:36.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.769+0000 7fbe977fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbe98148f20 0x7fbe980c2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:36.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.769+0000 7fbe977fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbe98148f20 0x7fbe980c2930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42942/0 (socket says 172.21.15.116:42942) 2024-04-25T00:01:36.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.769+0000 7fbe977fe640 1 -- 172.21.15.116:0/1596508628 learned_addr learned my addr 172.21.15.116:0/1596508628 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:36.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.769+0000 7fbe977fe640 1 -- 172.21.15.116:0/1596508628 >> v1:172.21.15.174:6789/0 conn(0x7fbe9806d510 legacy=0x7fbe98072500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:36.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.769+0000 7fbe977fe640 1 -- 172.21.15.116:0/1596508628 >> v1:172.21.15.116:6789/0 conn(0x7fbe9806d000 legacy=0x7fbe9815cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:36.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.769+0000 7fbe977fe640 1 -- 172.21.15.116:0/1596508628 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe98153070 con 0x7fbe98148f20 2024-04-25T00:01:36.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.773+0000 7fbe977fe640 1 --2- 172.21.15.116:0/1596508628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe98148f20 0x7fbe980c2930 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fbe981534d0 tx=0x7fbe8801c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40d6f6f23e82810f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:36.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.773+0000 7fbe967fc640 1 -- 172.21.15.116:0/1596508628 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe880b0070 con 0x7fbe98148f20 2024-04-25T00:01:36.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.773+0000 7fbe967fc640 1 -- 172.21.15.116:0/1596508628 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe880ab600 con 0x7fbe98148f20 2024-04-25T00:01:36.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.773+0000 7fbe967fc640 1 -- 172.21.15.116:0/1596508628 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe880ab9d0 con 0x7fbe98148f20 2024-04-25T00:01:36.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.773+0000 7fbe9e01d640 1 -- 172.21.15.116:0/1596508628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe98148f20 msgr2=0x7fbe980c2930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:36.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.773+0000 7fbe9e01d640 1 --2- 172.21.15.116:0/1596508628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe98148f20 0x7fbe980c2930 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fbe981534d0 tx=0x7fbe8801c3e0 comp rx=0 tx=0).stop 2024-04-25T00:01:36.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.773+0000 7fbe9e01d640 1 -- 172.21.15.116:0/1596508628 shutdown_connections 2024-04-25T00:01:36.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.773+0000 7fbe9e01d640 1 --2- 172.21.15.116:0/1596508628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe98148f20 0x7fbe980c2930 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:36.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.773+0000 7fbe9e01d640 1 -- 172.21.15.116:0/1596508628 >> 172.21.15.116:0/1596508628 conn(0x7fbe980c3a20 msgr2=0x7fbe980c3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:36.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.773+0000 7fbe9e01d640 1 -- 172.21.15.116:0/1596508628 shutdown_connections 2024-04-25T00:01:36.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.773+0000 7fbe9e01d640 1 -- 172.21.15.116:0/1596508628 wait complete. 2024-04-25T00:01:36.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.773+0000 7fbe9e01d640 1 Processor -- start 2024-04-25T00:01:36.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.773+0000 7fbe9e01d640 1 -- start start 2024-04-25T00:01:36.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.773+0000 7fbe9e01d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe9806d000 0x7fbe98151410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:36.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.773+0000 7fbe9e01d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe9806d510 0x7fbe98145d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:36.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.773+0000 7fbe9e01d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe98148f20 0x7fbe98146290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:36.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.773+0000 7fbe9e01d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbe98160e90 con 0x7fbe9806d000 2024-04-25T00:01:36.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.773+0000 7fbe9e01d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbe98160d10 con 0x7fbe9806d510 2024-04-25T00:01:36.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.773+0000 7fbe9e01d640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fbe98161010 con 0x7fbe98148f20 2024-04-25T00:01:36.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.773+0000 7fbe977fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe9806d000 0x7fbe98151410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:36.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.773+0000 7fbe977fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe9806d000 0x7fbe98151410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42954/0 (socket says 172.21.15.116:42954) 2024-04-25T00:01:36.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.773+0000 7fbe977fe640 1 -- 172.21.15.116:0/1435841793 learned_addr learned my addr 172.21.15.116:0/1435841793 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:36.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.773+0000 7fbe96ffd640 1 --2- 172.21.15.116:0/1435841793 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe9806d510 0x7fbe98145d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:36.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.773+0000 7fbe977fe640 1 -- 172.21.15.116:0/1435841793 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe98148f20 msgr2=0x7fbe98146290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:36.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.773+0000 7fbe977fe640 1 --2- 172.21.15.116:0/1435841793 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe98148f20 0x7fbe98146290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:36.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.773+0000 7fbe977fe640 1 -- 172.21.15.116:0/1435841793 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe9806d510 msgr2=0x7fbe98145d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:36.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.773+0000 7fbe977fe640 1 --2- 172.21.15.116:0/1435841793 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe9806d510 0x7fbe98145d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:36.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.773+0000 7fbe977fe640 1 -- 172.21.15.116:0/1435841793 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe980abfb0 con 0x7fbe9806d000 2024-04-25T00:01:36.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.773+0000 7fbe977fe640 1 --2- 172.21.15.116:0/1435841793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe9806d000 0x7fbe98151410 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fbe880ae040 tx=0x7fbe8801ac20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:36.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.773+0000 7fbe7ffff640 1 -- 172.21.15.116:0/1435841793 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe880ba020 con 0x7fbe9806d000 2024-04-25T00:01:36.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.773+0000 7fbe7ffff640 1 -- 172.21.15.116:0/1435841793 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe880b6420 con 0x7fbe9806d000 2024-04-25T00:01:36.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.773+0000 7fbe7ffff640 1 -- 172.21.15.116:0/1435841793 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe880b0040 con 0x7fbe9806d000 2024-04-25T00:01:36.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.773+0000 7fbe9e01d640 1 -- 172.21.15.116:0/1435841793 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe981eec40 con 0x7fbe9806d000 2024-04-25T00:01:36.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.773+0000 7fbe9e01d640 1 -- 172.21.15.116:0/1435841793 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe981ef200 con 0x7fbe9806d000 2024-04-25T00:01:36.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.777+0000 7fbe9e01d640 1 -- 172.21.15.116:0/1435841793 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe9804f510 con 0x7fbe9806d000 2024-04-25T00:01:36.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.777+0000 7fbe7ffff640 1 -- 172.21.15.116:0/1435841793 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbe880bd020 con 0x7fbe9806d000 2024-04-25T00:01:36.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.777+0000 7fbe7ffff640 1 --2- 172.21.15.116:0/1435841793 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbe840678f0 0x7fbe84069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:36.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.777+0000 7fbe7ffff640 1 -- 172.21.15.116:0/1435841793 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7fbe88120150 con 0x7fbe9806d000 2024-04-25T00:01:36.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.777+0000 7fbe96ffd640 1 --2- 172.21.15.116:0/1435841793 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbe840678f0 0x7fbe84069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:36.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.777+0000 7fbe96ffd640 1 --2- 172.21.15.116:0/1435841793 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbe840678f0 0x7fbe84069db0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fbe98152110 tx=0x7fbe8c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:36.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.781+0000 7fbe7ffff640 1 -- 172.21.15.116:0/1435841793 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fbe880c4030 con 0x7fbe9806d000 2024-04-25T00:01:36.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.929+0000 7fbe9e01d640 1 -- 172.21.15.116:0/1435841793 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"} v 0) -- 0x7fbe980c34b0 con 0x7fbe9806d000 2024-04-25T00:01:36.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.929+0000 7fbe7ffff640 1 -- 172.21.15.116:0/1435841793 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"}]=0 v27) ==== 95+0+11 (secure 0 0 0) 0x7fbe880160f0 con 0x7fbe9806d000 2024-04-25T00:01:36.932 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-04-25T00:01:36.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.933+0000 7fbe9e01d640 1 -- 172.21.15.116:0/1435841793 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbe840678f0 msgr2=0x7fbe84069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:36.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.933+0000 7fbe9e01d640 1 --2- 172.21.15.116:0/1435841793 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbe840678f0 0x7fbe84069db0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fbe98152110 tx=0x7fbe8c009040 comp rx=0 tx=0).stop 2024-04-25T00:01:36.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.933+0000 7fbe9e01d640 1 -- 172.21.15.116:0/1435841793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe9806d000 msgr2=0x7fbe98151410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:36.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.933+0000 7fbe9e01d640 1 --2- 172.21.15.116:0/1435841793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe9806d000 0x7fbe98151410 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fbe880ae040 tx=0x7fbe8801ac20 comp rx=0 tx=0).stop 2024-04-25T00:01:36.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.933+0000 7fbe9e01d640 1 -- 172.21.15.116:0/1435841793 shutdown_connections 2024-04-25T00:01:36.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.933+0000 7fbe9e01d640 1 --2- 172.21.15.116:0/1435841793 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbe840678f0 0x7fbe84069db0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:36.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.933+0000 7fbe9e01d640 1 --2- 172.21.15.116:0/1435841793 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe98148f20 0x7fbe98146290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:36.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.933+0000 7fbe9e01d640 1 --2- 172.21.15.116:0/1435841793 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe9806d510 0x7fbe98145d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:36.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.933+0000 7fbe9e01d640 1 --2- 172.21.15.116:0/1435841793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe9806d000 0x7fbe98151410 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:36.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.933+0000 7fbe9e01d640 1 -- 172.21.15.116:0/1435841793 >> 172.21.15.116:0/1435841793 conn(0x7fbe980c3a20 msgr2=0x7fbe9806df40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:36.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.933+0000 7fbe9e01d640 1 -- 172.21.15.116:0/1435841793 shutdown_connections 2024-04-25T00:01:36.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:36.933+0000 7fbe9e01d640 1 -- 172.21.15.116:0/1435841793 wait complete. 2024-04-25T00:01:36.949 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:01:37.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.061+0000 7fd101a0b640 1 Processor -- start 2024-04-25T00:01:37.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.061+0000 7fd101a0b640 1 -- start start 2024-04-25T00:01:37.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.065+0000 7fd101a0b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd0fc148890 0x7fd0fc072590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:37.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.065+0000 7fd101a0b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd0fc070dd0 0x7fd0fc071270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:37.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.065+0000 7fd101a0b640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fd0fc06e890 0x7fd0fc0717b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:37.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.065+0000 7fd101a0b640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fd0fc152c10 con 0x7fd0fc148890 2024-04-25T00:01:37.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.065+0000 7fd101a0b640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fd0fc071cf0 con 0x7fd0fc06e890 2024-04-25T00:01:37.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.065+0000 7fd101a0b640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fd0fc151e50 con 0x7fd0fc070dd0 2024-04-25T00:01:37.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.065+0000 7fd0faffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd0fc148890 0x7fd0fc072590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:37.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.065+0000 7fd0faffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd0fc148890 0x7fd0fc072590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42968/0 (socket says 172.21.15.116:42968) 2024-04-25T00:01:37.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.065+0000 7fd0faffd640 1 -- 172.21.15.116:0/1012345839 learned_addr learned my addr 172.21.15.116:0/1012345839 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:37.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.065+0000 7fd0faffd640 1 -- 172.21.15.116:0/1012345839 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd0fc070dd0 msgr2=0x7fd0fc071270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:37.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.065+0000 7fd0faffd640 1 --2- 172.21.15.116:0/1012345839 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd0fc070dd0 0x7fd0fc071270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:37.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.065+0000 7fd0faffd640 1 -- 172.21.15.116:0/1012345839 >> v2:172.21.15.174:3300/0 conn(0x7fd0fc06e890 msgr2=0x7fd0fc0717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:37.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.065+0000 7fd0faffd640 1 --2- 172.21.15.116:0/1012345839 >> v2:172.21.15.174:3300/0 conn(0x7fd0fc06e890 0x7fd0fc0717b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:37.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.065+0000 7fd0faffd640 1 -- 172.21.15.116:0/1012345839 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0fc152130 con 0x7fd0fc148890 2024-04-25T00:01:37.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.065+0000 7fd0faffd640 1 --2- 172.21.15.116:0/1012345839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0fc148890 0x7fd0fc072590 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc073530 tx=0x7fd0ec01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10feae0abcae4344 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:37.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.065+0000 7fd0fa7fc640 1 -- 172.21.15.116:0/1012345839 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0ec0b0070 con 0x7fd0fc148890 2024-04-25T00:01:37.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.065+0000 7fd0fa7fc640 1 -- 172.21.15.116:0/1012345839 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0ec0ab600 con 0x7fd0fc148890 2024-04-25T00:01:37.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.065+0000 7fd0fa7fc640 1 -- 172.21.15.116:0/1012345839 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0ec0ab9d0 con 0x7fd0fc148890 2024-04-25T00:01:37.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.069+0000 7fd101a0b640 1 -- 172.21.15.116:0/1012345839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0fc148890 msgr2=0x7fd0fc072590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:37.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.069+0000 7fd101a0b640 1 --2- 172.21.15.116:0/1012345839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0fc148890 0x7fd0fc072590 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc073530 tx=0x7fd0ec01c520 comp rx=0 tx=0).stop 2024-04-25T00:01:37.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.069+0000 7fd101a0b640 1 -- 172.21.15.116:0/1012345839 shutdown_connections 2024-04-25T00:01:37.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.069+0000 7fd101a0b640 1 --2- 172.21.15.116:0/1012345839 >> v2:172.21.15.174:3300/0 conn(0x7fd0fc06e890 0x7fd0fc0717b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:37.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.069+0000 7fd101a0b640 1 --2- 172.21.15.116:0/1012345839 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd0fc070dd0 0x7fd0fc071270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:37.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.069+0000 7fd101a0b640 1 --2- 172.21.15.116:0/1012345839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0fc148890 0x7fd0fc072590 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:37.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.069+0000 7fd101a0b640 1 -- 172.21.15.116:0/1012345839 >> 172.21.15.116:0/1012345839 conn(0x7fd0fc0c3b00 msgr2=0x7fd0fc0c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:37.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.069+0000 7fd101a0b640 1 -- 172.21.15.116:0/1012345839 shutdown_connections 2024-04-25T00:01:37.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.069+0000 7fd101a0b640 1 -- 172.21.15.116:0/1012345839 wait complete. 2024-04-25T00:01:37.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.069+0000 7fd101a0b640 1 Processor -- start 2024-04-25T00:01:37.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.069+0000 7fd101a0b640 1 -- start start 2024-04-25T00:01:37.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.069+0000 7fd101a0b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0fc06e890 0x7fd0fc155630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:37.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.069+0000 7fd101a0b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0fc070dd0 0x7fd0fc1f02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:37.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.069+0000 7fd101a0b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd0fc148890 0x7fd0fc141ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:37.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.069+0000 7fd101a0b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd0fc0ab710 con 0x7fd0fc06e890 2024-04-25T00:01:37.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.069+0000 7fd101a0b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd0fc0ab590 con 0x7fd0fc070dd0 2024-04-25T00:01:37.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.069+0000 7fd101a0b640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fd0fc0ab890 con 0x7fd0fc148890 2024-04-25T00:01:37.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.069+0000 7fd0faffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0fc06e890 0x7fd0fc155630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:37.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.069+0000 7fd0faffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0fc06e890 0x7fd0fc155630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42978/0 (socket says 172.21.15.116:42978) 2024-04-25T00:01:37.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.069+0000 7fd0faffd640 1 -- 172.21.15.116:0/680246167 learned_addr learned my addr 172.21.15.116:0/680246167 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:37.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.069+0000 7fd0faffd640 1 -- 172.21.15.116:0/680246167 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd0fc148890 msgr2=0x7fd0fc141ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:37.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.069+0000 7fd0faffd640 1 --2- 172.21.15.116:0/680246167 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd0fc148890 0x7fd0fc141ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:37.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.069+0000 7fd0faffd640 1 -- 172.21.15.116:0/680246167 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0fc070dd0 msgr2=0x7fd0fc1f02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:37.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.069+0000 7fd0faffd640 1 --2- 172.21.15.116:0/680246167 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0fc070dd0 0x7fd0fc1f02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:37.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.069+0000 7fd0faffd640 1 -- 172.21.15.116:0/680246167 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0fc1f0cb0 con 0x7fd0fc06e890 2024-04-25T00:01:37.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.069+0000 7fd0faffd640 1 --2- 172.21.15.116:0/680246167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0fc06e890 0x7fd0fc155630 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec01abd0 tx=0x7fd0ec01ad60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:37.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.069+0000 7fd0f8ff9640 1 -- 172.21.15.116:0/680246167 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0ec0ba020 con 0x7fd0fc06e890 2024-04-25T00:01:37.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.069+0000 7fd0f8ff9640 1 -- 172.21.15.116:0/680246167 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0ec0b6620 con 0x7fd0fc06e890 2024-04-25T00:01:37.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.069+0000 7fd0f8ff9640 1 -- 172.21.15.116:0/680246167 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0ec0b0040 con 0x7fd0fc06e890 2024-04-25T00:01:37.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.069+0000 7fd101a0b640 1 -- 172.21.15.116:0/680246167 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0fc140fe0 con 0x7fd0fc06e890 2024-04-25T00:01:37.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.069+0000 7fd101a0b640 1 -- 172.21.15.116:0/680246167 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0fc1f07f0 con 0x7fd0fc06e890 2024-04-25T00:01:37.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.069+0000 7fd101a0b640 1 -- 172.21.15.116:0/680246167 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0fc068350 con 0x7fd0fc06e890 2024-04-25T00:01:37.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.073+0000 7fd0f8ff9640 1 -- 172.21.15.116:0/680246167 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd0ec0bd020 con 0x7fd0fc06e890 2024-04-25T00:01:37.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.073+0000 7fd0f8ff9640 1 --2- 172.21.15.116:0/680246167 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd0dc0678f0 0x7fd0dc069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:37.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.073+0000 7fd0f8ff9640 1 -- 172.21.15.116:0/680246167 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7fd0ec1201a0 con 0x7fd0fc06e890 2024-04-25T00:01:37.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.073+0000 7fd0f3fff640 1 --2- 172.21.15.116:0/680246167 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd0dc0678f0 0x7fd0dc069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:37.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.073+0000 7fd0f3fff640 1 --2- 172.21.15.116:0/680246167 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd0dc0678f0 0x7fd0dc069db0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc1edc10 tx=0x7fd0e4006ca0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:37.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.077+0000 7fd0f8ff9640 1 -- 172.21.15.116:0/680246167 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fd0ec0c4030 con 0x7fd0fc06e890 2024-04-25T00:01:37.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.253+0000 7fd101a0b640 1 -- 172.21.15.116:0/680246167 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd0fc06e670 con 0x7fd0fc06e890 2024-04-25T00:01:37.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.253+0000 7fd0f8ff9640 1 -- 172.21.15.116:0/680246167 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6492 (secure 0 0 0) 0x7fd0ec016110 con 0x7fd0fc06e890 2024-04-25T00:01:37.257 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:01:37.258 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6834/1459177921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1459177921},{"type":"v1","addr":"172.21.15.116:6834","nonce":1459177921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:01:08.748957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6837/1762356055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1762356055},{"type":"v1","addr":"172.21.15.174:6837","nonce":1762356055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270]},"id":1},{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:01:22.888641+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6835/174781672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":174781672},{"type":"v1","addr":"172.21.15.116:6835","nonce":174781672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4245,"qdb_cluster":[4245]},"id":2}]} 2024-04-25T00:01:37.258 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 14 2024-04-25T00:01:37.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.257+0000 7fd101a0b640 1 -- 172.21.15.116:0/680246167 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd0dc0678f0 msgr2=0x7fd0dc069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:37.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.257+0000 7fd101a0b640 1 --2- 172.21.15.116:0/680246167 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd0dc0678f0 0x7fd0dc069db0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc1edc10 tx=0x7fd0e4006ca0 comp rx=0 tx=0).stop 2024-04-25T00:01:37.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.257+0000 7fd101a0b640 1 -- 172.21.15.116:0/680246167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0fc06e890 msgr2=0x7fd0fc155630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:37.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.257+0000 7fd101a0b640 1 --2- 172.21.15.116:0/680246167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0fc06e890 0x7fd0fc155630 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec01abd0 tx=0x7fd0ec01ad60 comp rx=0 tx=0).stop 2024-04-25T00:01:37.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.257+0000 7fd101a0b640 1 -- 172.21.15.116:0/680246167 shutdown_connections 2024-04-25T00:01:37.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.257+0000 7fd101a0b640 1 --2- 172.21.15.116:0/680246167 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd0dc0678f0 0x7fd0dc069db0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:37.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.257+0000 7fd101a0b640 1 --2- 172.21.15.116:0/680246167 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd0fc148890 0x7fd0fc141ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:37.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.257+0000 7fd101a0b640 1 --2- 172.21.15.116:0/680246167 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0fc070dd0 0x7fd0fc1f02b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:37.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.257+0000 7fd101a0b640 1 --2- 172.21.15.116:0/680246167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0fc06e890 0x7fd0fc155630 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:37.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.257+0000 7fd101a0b640 1 -- 172.21.15.116:0/680246167 >> 172.21.15.116:0/680246167 conn(0x7fd0fc0c3b00 msgr2=0x7fd0fc071200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:37.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.257+0000 7fd101a0b640 1 -- 172.21.15.116:0/680246167 shutdown_connections 2024-04-25T00:01:37.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.257+0000 7fd101a0b640 1 -- 172.21.15.116:0/680246167 wait complete. 2024-04-25T00:01:37.275 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T00:01:37.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.381+0000 7fe026690640 1 Processor -- start 2024-04-25T00:01:37.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.381+0000 7fe026690640 1 -- start start 2024-04-25T00:01:37.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.385+0000 7fe026690640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe020150bf0 0x7fe02006c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:37.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.385+0000 7fe026690640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fe02006c760 0x7fe02006f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:37.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.385+0000 7fe026690640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe020153aa0 con 0x7fe020153570 2024-04-25T00:01:37.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.385+0000 7fe026690640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fe02006e630 con 0x7fe020150bf0 2024-04-25T00:01:37.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.385+0000 7fe026690640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fe020150fd0 con 0x7fe02006c760 2024-04-25T00:01:37.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.385+0000 7fe01ffff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe020150bf0 0x7fe02006c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:37.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.385+0000 7fe017fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fe020153570 0x7fe02006eac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:55356/0 (socket says 172.21.15.116:55356) 2024-04-25T00:01:37.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.385+0000 7fe017fff640 1 -- 172.21.15.116:0/286652293 learned_addr learned my addr 172.21.15.116:0/286652293 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:37.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.385+0000 7fe024c2d640 1 --2- 172.21.15.116:0/286652293 >> v2:172.21.15.174:3300/0 conn(0x7fe02006c760 0x7fe02006f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:37.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.385+0000 7fe01f7fe640 1 -- 172.21.15.116:0/286652293 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2880475556 0 0) 0x7fe020153aa0 con 0x7fe020153570 2024-04-25T00:01:37.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.385+0000 7fe01f7fe640 1 -- 172.21.15.116:0/286652293 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe00c003530 con 0x7fe020153570 2024-04-25T00:01:37.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.385+0000 7fe01ffff640 1 -- 172.21.15.116:0/286652293 >> v2:172.21.15.174:3300/0 conn(0x7fe02006c760 msgr2=0x7fe02006f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:37.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.385+0000 7fe01ffff640 1 --2- 172.21.15.116:0/286652293 >> v2:172.21.15.174:3300/0 conn(0x7fe02006c760 0x7fe02006f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:37.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.385+0000 7fe01ffff640 1 -- 172.21.15.116:0/286652293 >> v1:172.21.15.116:6789/0 conn(0x7fe020153570 legacy=0x7fe02006eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:37.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.385+0000 7fe01ffff640 1 -- 172.21.15.116:0/286652293 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe020153d80 con 0x7fe020150bf0 2024-04-25T00:01:37.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.385+0000 7fe01ffff640 1 --2- 172.21.15.116:0/286652293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe020150bf0 0x7fe02006c220 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fe0200c5200 tx=0x7fe01001c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd42973448a26200 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:37.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.385+0000 7fe01f7fe640 1 -- 172.21.15.116:0/286652293 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0100b1070 con 0x7fe020150bf0 2024-04-25T00:01:37.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.385+0000 7fe01f7fe640 1 -- 172.21.15.116:0/286652293 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0100ac620 con 0x7fe020150bf0 2024-04-25T00:01:37.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.385+0000 7fe01f7fe640 1 -- 172.21.15.116:0/286652293 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0100ac9f0 con 0x7fe020150bf0 2024-04-25T00:01:37.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.385+0000 7fe026690640 1 -- 172.21.15.116:0/286652293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe020150bf0 msgr2=0x7fe02006c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:37.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.385+0000 7fe026690640 1 --2- 172.21.15.116:0/286652293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe020150bf0 0x7fe02006c220 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fe0200c5200 tx=0x7fe01001c3c0 comp rx=0 tx=0).stop 2024-04-25T00:01:37.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.389+0000 7fe026690640 1 -- 172.21.15.116:0/286652293 shutdown_connections 2024-04-25T00:01:37.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.389+0000 7fe026690640 1 --2- 172.21.15.116:0/286652293 >> v2:172.21.15.174:3300/0 conn(0x7fe02006c760 0x7fe02006f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:37.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.389+0000 7fe026690640 1 --2- 172.21.15.116:0/286652293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe020150bf0 0x7fe02006c220 secure :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fe0200c5200 tx=0x7fe01001c3c0 comp rx=0 tx=0).stop 2024-04-25T00:01:37.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.389+0000 7fe026690640 1 -- 172.21.15.116:0/286652293 >> 172.21.15.116:0/286652293 conn(0x7fe0200c4270 msgr2=0x7fe0200c4680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:37.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.389+0000 7fe026690640 1 -- 172.21.15.116:0/286652293 shutdown_connections 2024-04-25T00:01:37.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.389+0000 7fe026690640 1 -- 172.21.15.116:0/286652293 wait complete. 2024-04-25T00:01:37.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.389+0000 7fe026690640 1 Processor -- start 2024-04-25T00:01:37.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.389+0000 7fe026690640 1 -- start start 2024-04-25T00:01:37.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.389+0000 7fe026690640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe02006c760 0x7fe0201460f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:37.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.389+0000 7fe026690640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe020153570 0x7fe0201eeaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:37.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.389+0000 7fe026690640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe0201409b0 0x7fe0201ef030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:37.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.389+0000 7fe026690640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe0200a72b0 con 0x7fe02006c760 2024-04-25T00:01:37.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.389+0000 7fe026690640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe0200a7130 con 0x7fe0201409b0 2024-04-25T00:01:37.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.389+0000 7fe026690640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe020160710 con 0x7fe020153570 2024-04-25T00:01:37.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.389+0000 7fe01ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe02006c760 0x7fe0201460f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:37.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.389+0000 7fe01ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe02006c760 0x7fe0201460f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42994/0 (socket says 172.21.15.116:42994) 2024-04-25T00:01:37.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.389+0000 7fe01ffff640 1 -- 172.21.15.116:0/831808218 learned_addr learned my addr 172.21.15.116:0/831808218 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:37.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.389+0000 7fe01ffff640 1 -- 172.21.15.116:0/831808218 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe020153570 msgr2=0x7fe0201eeaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:37.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.389+0000 7fe01ffff640 1 --2- 172.21.15.116:0/831808218 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe020153570 0x7fe0201eeaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:37.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.389+0000 7fe01ffff640 1 -- 172.21.15.116:0/831808218 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe0201409b0 msgr2=0x7fe0201ef030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:37.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.389+0000 7fe01ffff640 1 --2- 172.21.15.116:0/831808218 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe0201409b0 0x7fe0201ef030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:37.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.389+0000 7fe01ffff640 1 -- 172.21.15.116:0/831808218 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe02014fbd0 con 0x7fe02006c760 2024-04-25T00:01:37.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.389+0000 7fe01ffff640 1 --2- 172.21.15.116:0/831808218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe02006c760 0x7fe0201460f0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fe010019ff0 tx=0x7fe010019160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:37.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.389+0000 7fe01d7fa640 1 -- 172.21.15.116:0/831808218 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0100b1040 con 0x7fe02006c760 2024-04-25T00:01:37.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.389+0000 7fe01d7fa640 1 -- 172.21.15.116:0/831808218 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0100bc070 con 0x7fe02006c760 2024-04-25T00:01:37.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.389+0000 7fe026690640 1 -- 172.21.15.116:0/831808218 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe02014feb0 con 0x7fe02006c760 2024-04-25T00:01:37.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.389+0000 7fe01d7fa640 1 -- 172.21.15.116:0/831808218 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe010006e20 con 0x7fe02006c760 2024-04-25T00:01:37.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.389+0000 7fe026690640 1 -- 172.21.15.116:0/831808218 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe020146630 con 0x7fe02006c760 2024-04-25T00:01:37.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.389+0000 7fe026690640 1 -- 172.21.15.116:0/831808218 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe020068a60 con 0x7fe02006c760 2024-04-25T00:01:37.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.393+0000 7fe01d7fa640 1 -- 172.21.15.116:0/831808218 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe0100b5630 con 0x7fe02006c760 2024-04-25T00:01:37.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.393+0000 7fe01d7fa640 1 --2- 172.21.15.116:0/831808218 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe0000678c0 0x7fe000069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:37.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.393+0000 7fe01d7fa640 1 -- 172.21.15.116:0/831808218 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7fe01011fd50 con 0x7fe02006c760 2024-04-25T00:01:37.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.393+0000 7fe017fff640 1 --2- 172.21.15.116:0/831808218 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe0000678c0 0x7fe000069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:37.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.393+0000 7fe017fff640 1 --2- 172.21.15.116:0/831808218 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe0000678c0 0x7fe000069d80 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe008007fd0 tx=0x7fe008007900 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:37.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.397+0000 7fe01d7fa640 1 -- 172.21.15.116:0/831808218 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fe010123030 con 0x7fe02006c760 2024-04-25T00:01:37.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.557+0000 7fe026690640 1 -- 172.21.15.116:0/831808218 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe020074aa0 con 0x7fe02006c760 2024-04-25T00:01:37.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.557+0000 7fe01d7fa640 1 -- 172.21.15.116:0/831808218 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+20311 (secure 0 0 0) 0x7fe010016070 con 0x7fe02006c760 2024-04-25T00:01:37.561 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:01:37.561 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":27,"fsid":"6d6a1aa7-0478-4c83-8057-05917e81f046","created":"2024-04-25T00:00:40.682493+0000","modified":"2024-04-25T00:01:20.501508+0000","last_up_change":"2024-04-25T00:00:51.822174+0000","last_in_change":"2024-04-25T00:00:42.980473+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T00:00:53.324376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-04-25T00:01:03.197326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-04-25T00:01:03.728618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-04-25T00:01:17.125210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-04-25T00:01:17.821873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8b6ddcb2-1ad9-4167-961c-bd6936bb9060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6809","nonce":3751765682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6811","nonce":3751765682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6815","nonce":3751765682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6813","nonce":3751765682}]},"public_addr":"172.21.15.116:6809/3751765682","cluster_addr":"172.21.15.116:6811/3751765682","heartbeat_back_addr":"172.21.15.116:6815/3751765682","heartbeat_front_addr":"172.21.15.116:6813/3751765682","state":["exists","up"]},{"osd":1,"uuid":"a6431768-994f-41da-8853-886f5687fb91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6801","nonce":377879124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6803","nonce":377879124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6807","nonce":377879124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6805","nonce":377879124}]},"public_addr":"172.21.15.116:6801/377879124","cluster_addr":"172.21.15.116:6803/377879124","heartbeat_back_addr":"172.21.15.116:6807/377879124","heartbeat_front_addr":"172.21.15.116:6805/377879124","state":["exists","up"]},{"osd":2,"uuid":"4f69bd40-5a23-45ff-8693-e1de75e1aa9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6817","nonce":1099282267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6819","nonce":1099282267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6823","nonce":1099282267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6821","nonce":1099282267}]},"public_addr":"172.21.15.116:6817/1099282267","cluster_addr":"172.21.15.116:6819/1099282267","heartbeat_back_addr":"172.21.15.116:6823/1099282267","heartbeat_front_addr":"172.21.15.116:6821/1099282267","state":["exists","up"]},{"osd":3,"uuid":"6a0d5344-8b6f-40d3-918d-0f0a4be688e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6825","nonce":1179942661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6827","nonce":1179942661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6831","nonce":1179942661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6829","nonce":1179942661}]},"public_addr":"172.21.15.116:6825/1179942661","cluster_addr":"172.21.15.116:6827/1179942661","heartbeat_back_addr":"172.21.15.116:6831/1179942661","heartbeat_front_addr":"172.21.15.116:6829/1179942661","state":["exists","up"]},{"osd":4,"uuid":"132e92e4-b825-441e-965a-ca00533fa86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6801","nonce":3132444330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6803","nonce":3132444330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6807","nonce":3132444330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6805","nonce":3132444330}]},"public_addr":"172.21.15.174:6801/3132444330","cluster_addr":"172.21.15.174:6803/3132444330","heartbeat_back_addr":"172.21.15.174:6807/3132444330","heartbeat_front_addr":"172.21.15.174:6805/3132444330","state":["exists","up"]},{"osd":5,"uuid":"a2eeb5b6-1b31-43cc-8306-a0adf3ee15bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6809","nonce":881817164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6811","nonce":881817164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6815","nonce":881817164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6813","nonce":881817164}]},"public_addr":"172.21.15.174:6809/881817164","cluster_addr":"172.21.15.174:6811/881817164","heartbeat_back_addr":"172.21.15.174:6815/881817164","heartbeat_front_addr":"172.21.15.174:6813/881817164","state":["exists","up"]},{"osd":6,"uuid":"588e873a-c841-4ba7-90f2-8956c87a9983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6825","nonce":117574251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6827","nonce":117574251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6831","nonce":117574251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6829","nonce":117574251}]},"public_addr":"172.21.15.174:6825/117574251","cluster_addr":"172.21.15.174:6827/117574251","heartbeat_back_addr":"172.21.15.174:6831/117574251","heartbeat_front_addr":"172.21.15.174:6829/117574251","state":["exists","up"]},{"osd":7,"uuid":"3b90f9b0-8a0e-4883-b088-ca1d4012870c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6817","nonce":2543899196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6819","nonce":2543899196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6823","nonce":2543899196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6821","nonce":2543899196}]},"public_addr":"172.21.15.174:6817/2543899196","cluster_addr":"172.21.15.174:6819/2543899196","heartbeat_back_addr":"172.21.15.174:6823/2543899196","heartbeat_front_addr":"172.21.15.174:6821/2543899196","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.293769+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.310034+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.404865+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.549658+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.192174+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.297266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.493649+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.452309+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:01:37.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.561+0000 7fe026690640 1 -- 172.21.15.116:0/831808218 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe0000678c0 msgr2=0x7fe000069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:37.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.561+0000 7fe026690640 1 --2- 172.21.15.116:0/831808218 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe0000678c0 0x7fe000069d80 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe008007fd0 tx=0x7fe008007900 comp rx=0 tx=0).stop 2024-04-25T00:01:37.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.561+0000 7fe026690640 1 -- 172.21.15.116:0/831808218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe02006c760 msgr2=0x7fe0201460f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:37.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.561+0000 7fe026690640 1 --2- 172.21.15.116:0/831808218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe02006c760 0x7fe0201460f0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fe010019ff0 tx=0x7fe010019160 comp rx=0 tx=0).stop 2024-04-25T00:01:37.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.561+0000 7fe026690640 1 -- 172.21.15.116:0/831808218 shutdown_connections 2024-04-25T00:01:37.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.561+0000 7fe026690640 1 --2- 172.21.15.116:0/831808218 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe0000678c0 0x7fe000069d80 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:37.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.561+0000 7fe026690640 1 --2- 172.21.15.116:0/831808218 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe0201409b0 0x7fe0201ef030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:37.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.561+0000 7fe026690640 1 --2- 172.21.15.116:0/831808218 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe020153570 0x7fe0201eeaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:37.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.561+0000 7fe026690640 1 --2- 172.21.15.116:0/831808218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe02006c760 0x7fe0201460f0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:37.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.561+0000 7fe026690640 1 -- 172.21.15.116:0/831808218 >> 172.21.15.116:0/831808218 conn(0x7fe0200c4270 msgr2=0x7fe0200a7d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:37.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.561+0000 7fe026690640 1 -- 172.21.15.116:0/831808218 shutdown_connections 2024-04-25T00:01:37.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.561+0000 7fe026690640 1 -- 172.21.15.116:0/831808218 wait complete. 2024-04-25T00:01:37.582 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:01:37.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.701+0000 7f7863056640 1 Processor -- start 2024-04-25T00:01:37.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.701+0000 7f7863056640 1 -- start start 2024-04-25T00:01:37.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.705+0000 7f7863056640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f785c150d20 0x7f785c0c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:37.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.705+0000 7f7863056640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f785c0c2f50 0x7f785c1538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:37.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.705+0000 7f7863056640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f785c0c3910 con 0x7f785c0a94a0 2024-04-25T00:01:37.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.705+0000 7f7863056640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f785c154530 con 0x7f785c150d20 2024-04-25T00:01:37.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.705+0000 7f7863056640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f785c0c16e0 con 0x7f785c0c2f50 2024-04-25T00:01:37.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.705+0000 7f78615f3640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f785c0a94a0 0x7f785c153e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:55366/0 (socket says 172.21.15.116:55366) 2024-04-25T00:01:37.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.705+0000 7f78615f3640 1 -- 172.21.15.116:0/3886822881 learned_addr learned my addr 172.21.15.116:0/3886822881 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:37.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.705+0000 7f7860df2640 1 --2- 172.21.15.116:0/3886822881 >> v2:172.21.15.174:3300/0 conn(0x7f785c150d20 0x7f785c0c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:37.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.705+0000 7f785bfff640 1 --2- 172.21.15.116:0/3886822881 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f785c0c2f50 0x7f785c1538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:37.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.705+0000 7f785b7fe640 1 -- 172.21.15.116:0/3886822881 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2948624287 0 0) 0x7f785c0c3910 con 0x7f785c0a94a0 2024-04-25T00:01:37.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.705+0000 7f785b7fe640 1 -- 172.21.15.116:0/3886822881 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7844003530 con 0x7f785c0a94a0 2024-04-25T00:01:37.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.705+0000 7f785b7fe640 1 -- 172.21.15.116:0/3886822881 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2190529713 0 0) 0x7f7844003530 con 0x7f785c0a94a0 2024-04-25T00:01:37.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.705+0000 7f785b7fe640 1 -- 172.21.15.116:0/3886822881 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f785c0c2f50 msgr2=0x7f785c1538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:37.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.705+0000 7f785b7fe640 1 --2- 172.21.15.116:0/3886822881 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f785c0c2f50 0x7f785c1538e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:37.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.705+0000 7f785b7fe640 1 -- 172.21.15.116:0/3886822881 >> v2:172.21.15.174:3300/0 conn(0x7f785c150d20 msgr2=0x7f785c0c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:37.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.705+0000 7f785b7fe640 1 --2- 172.21.15.116:0/3886822881 >> v2:172.21.15.174:3300/0 conn(0x7f785c150d20 0x7f785c0c2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:37.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.705+0000 7f785b7fe640 1 -- 172.21.15.116:0/3886822881 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f785c0c1400 con 0x7f785c0a94a0 2024-04-25T00:01:37.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.705+0000 7f785b7fe640 1 -- 172.21.15.116:0/3886822881 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f7854001a00 con 0x7f785c0a94a0 2024-04-25T00:01:37.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.705+0000 7f785b7fe640 1 -- 172.21.15.116:0/3886822881 >> v1:172.21.15.116:6789/0 conn(0x7f785c0a94a0 legacy=0x7f785c153e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:37.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.705+0000 7f785b7fe640 1 --2- 172.21.15.116:0/3886822881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78440039c0 0x7f7844003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:37.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.705+0000 7f785b7fe640 1 --2- 172.21.15.116:0/3886822881 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f785c0c2f50 0x7f7844004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:37.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.705+0000 7f785b7fe640 1 --2- 172.21.15.116:0/3886822881 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7844004d50 0x7f78440051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:37.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.705+0000 7f785b7fe640 1 -- 172.21.15.116:0/3886822881 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f785c154530 con 0x7f78440039c0 2024-04-25T00:01:37.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.705+0000 7f785b7fe640 1 -- 172.21.15.116:0/3886822881 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f785c0c16e0 con 0x7f7844004d50 2024-04-25T00:01:37.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.705+0000 7f785b7fe640 1 -- 172.21.15.116:0/3886822881 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f784400b9d0 con 0x7f785c0c2f50 2024-04-25T00:01:37.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.705+0000 7f7860df2640 1 --2- 172.21.15.116:0/3886822881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78440039c0 0x7f7844003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:37.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.705+0000 7f785bfff640 1 --2- 172.21.15.116:0/3886822881 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f785c0c2f50 0x7f7844004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:37.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.705+0000 7f78615f3640 1 --2- 172.21.15.116:0/3886822881 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7844004d50 0x7f78440051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:37.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.705+0000 7f7860df2640 1 -- 172.21.15.116:0/3886822881 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f785c0c2f50 msgr2=0x7f7844004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:37.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.705+0000 7f7860df2640 1 --2- 172.21.15.116:0/3886822881 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f785c0c2f50 0x7f7844004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:37.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.705+0000 7f7860df2640 1 -- 172.21.15.116:0/3886822881 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7844004d50 msgr2=0x7f78440051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:37.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.705+0000 7f7860df2640 1 --2- 172.21.15.116:0/3886822881 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7844004d50 0x7f78440051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:37.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.705+0000 7f7860df2640 1 -- 172.21.15.116:0/3886822881 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7844003530 con 0x7f78440039c0 2024-04-25T00:01:37.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.705+0000 7f78615f3640 1 --2- 172.21.15.116:0/3886822881 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7844004d50 0x7f78440051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:37.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.709+0000 7f7860df2640 1 --2- 172.21.15.116:0/3886822881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78440039c0 0x7f7844003de0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f784c009ae0 tx=0x7f784c01cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a80fc39ea5b5613 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:37.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.709+0000 7f785b7fe640 1 -- 172.21.15.116:0/3886822881 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f784c0b2070 con 0x7f78440039c0 2024-04-25T00:01:37.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.709+0000 7f785b7fe640 1 -- 172.21.15.116:0/3886822881 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f784c0a5590 con 0x7f78440039c0 2024-04-25T00:01:37.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.709+0000 7f785b7fe640 1 -- 172.21.15.116:0/3886822881 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f784c0096b0 con 0x7f78440039c0 2024-04-25T00:01:37.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.709+0000 7f7863056640 1 -- 172.21.15.116:0/3886822881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78440039c0 msgr2=0x7f7844003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:37.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.709+0000 7f7863056640 1 --2- 172.21.15.116:0/3886822881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78440039c0 0x7f7844003de0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f784c009ae0 tx=0x7f784c01cab0 comp rx=0 tx=0).stop 2024-04-25T00:01:37.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.709+0000 7f78615f3640 1 -- 172.21.15.116:0/3886822881 reap_dead start 2024-04-25T00:01:37.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.709+0000 7f7863056640 1 -- 172.21.15.116:0/3886822881 shutdown_connections 2024-04-25T00:01:37.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.709+0000 7f7863056640 1 -- 172.21.15.116:0/3886822881 >> 172.21.15.116:0/3886822881 conn(0x7f785c0c3b00 msgr2=0x7f785c0c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:37.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.709+0000 7f7863056640 1 -- 172.21.15.116:0/3886822881 shutdown_connections 2024-04-25T00:01:37.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.709+0000 7f7863056640 1 -- 172.21.15.116:0/3886822881 wait complete. 2024-04-25T00:01:37.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.709+0000 7f7863056640 1 Processor -- start 2024-04-25T00:01:37.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.709+0000 7f7863056640 1 -- start start 2024-04-25T00:01:37.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.709+0000 7f7863056640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f785c145c00 0x7f785c0c6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:37.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.709+0000 7f7863056640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f785c151400 0x7f785c145fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:37.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.709+0000 7f7863056640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f785c14e550 0x7f785c0c4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:37.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.709+0000 7f7863056640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f785c15e8a0 con 0x7f785c145c00 2024-04-25T00:01:37.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.709+0000 7f7863056640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f785c15e720 con 0x7f785c151400 2024-04-25T00:01:37.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.709+0000 7f7863056640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f785c15ea20 con 0x7f785c14e550 2024-04-25T00:01:37.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.713+0000 7f7860df2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f785c145c00 0x7f785c0c6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:37.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.713+0000 7f7860df2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f785c145c00 0x7f785c0c6860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43012/0 (socket says 172.21.15.116:43012) 2024-04-25T00:01:37.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.713+0000 7f7860df2640 1 -- 172.21.15.116:0/2102284412 learned_addr learned my addr 172.21.15.116:0/2102284412 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:37.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.713+0000 7f78615f3640 1 --2- 172.21.15.116:0/2102284412 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f785c14e550 0x7f785c0c4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:37.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.713+0000 7f785bfff640 1 --2- 172.21.15.116:0/2102284412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f785c151400 0x7f785c145fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:37.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.713+0000 7f7860df2640 1 -- 172.21.15.116:0/2102284412 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f785c14e550 msgr2=0x7f785c0c4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:37.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.713+0000 7f7860df2640 1 --2- 172.21.15.116:0/2102284412 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f785c14e550 0x7f785c0c4cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:37.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.713+0000 7f7860df2640 1 -- 172.21.15.116:0/2102284412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f785c151400 msgr2=0x7f785c145fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:37.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.713+0000 7f7860df2640 1 --2- 172.21.15.116:0/2102284412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f785c151400 0x7f785c145fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:37.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.713+0000 7f7860df2640 1 -- 172.21.15.116:0/2102284412 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f785c14ffe0 con 0x7f785c145c00 2024-04-25T00:01:37.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.713+0000 7f78615f3640 1 --2- 172.21.15.116:0/2102284412 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f785c14e550 0x7f785c0c4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:37.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.713+0000 7f785bfff640 1 --2- 172.21.15.116:0/2102284412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f785c151400 0x7f785c145fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:37.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.713+0000 7f7860df2640 1 --2- 172.21.15.116:0/2102284412 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f785c145c00 0x7f785c0c6860 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f784c009ae0 tx=0x7f784c019850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:37.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.713+0000 7f78597fa640 1 -- 172.21.15.116:0/2102284412 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f784c0b2040 con 0x7f785c145c00 2024-04-25T00:01:37.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.713+0000 7f78597fa640 1 -- 172.21.15.116:0/2102284412 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f784c00f000 con 0x7f785c145c00 2024-04-25T00:01:37.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.713+0000 7f7863056640 1 -- 172.21.15.116:0/2102284412 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f785c1edaa0 con 0x7f785c145c00 2024-04-25T00:01:37.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.713+0000 7f78597fa640 1 -- 172.21.15.116:0/2102284412 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f784c0096b0 con 0x7f785c145c00 2024-04-25T00:01:37.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.713+0000 7f7863056640 1 -- 172.21.15.116:0/2102284412 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f785c14e9d0 con 0x7f785c145c00 2024-04-25T00:01:37.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.713+0000 7f7863056640 1 -- 172.21.15.116:0/2102284412 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f785c068350 con 0x7f785c145c00 2024-04-25T00:01:37.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.713+0000 7f78597fa640 1 -- 172.21.15.116:0/2102284412 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f784c0bf030 con 0x7f785c145c00 2024-04-25T00:01:37.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.713+0000 7f78597fa640 1 --2- 172.21.15.116:0/2102284412 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f78480678f0 0x7f7848069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:37.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.713+0000 7f78597fa640 1 -- 172.21.15.116:0/2102284412 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7f784c11f520 con 0x7f785c145c00 2024-04-25T00:01:37.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.713+0000 7f785bfff640 1 --2- 172.21.15.116:0/2102284412 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f78480678f0 0x7f7848069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:37.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.713+0000 7f785bfff640 1 --2- 172.21.15.116:0/2102284412 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f78480678f0 0x7f7848069db0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f785c14fed0 tx=0x7f7850008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:37.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.717+0000 7f78597fa640 1 -- 172.21.15.116:0/2102284412 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f784c0167c0 con 0x7f785c145c00 2024-04-25T00:01:37.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.913+0000 7f7863056640 1 -- 172.21.15.116:0/2102284412 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f785c075410 con 0x7f785c145c00 2024-04-25T00:01:37.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.913+0000 7f78597fa640 1 -- 172.21.15.116:0/2102284412 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6492 (secure 0 0 0) 0x7f784c0efdc0 con 0x7f785c145c00 2024-04-25T00:01:37.917 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:01:37.918 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6834/1459177921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1459177921},{"type":"v1","addr":"172.21.15.116:6834","nonce":1459177921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:01:08.748957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6837/1762356055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1762356055},{"type":"v1","addr":"172.21.15.174:6837","nonce":1762356055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270]},"id":1},{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:01:22.888641+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6835/174781672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":174781672},{"type":"v1","addr":"172.21.15.116:6835","nonce":174781672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4245,"qdb_cluster":[4245]},"id":2}]} 2024-04-25T00:01:37.918 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 14 2024-04-25T00:01:37.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.917+0000 7f7863056640 1 -- 172.21.15.116:0/2102284412 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f78480678f0 msgr2=0x7f7848069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:37.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.917+0000 7f7863056640 1 --2- 172.21.15.116:0/2102284412 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f78480678f0 0x7f7848069db0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f785c14fed0 tx=0x7f7850008040 comp rx=0 tx=0).stop 2024-04-25T00:01:37.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.917+0000 7f7863056640 1 -- 172.21.15.116:0/2102284412 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f785c145c00 msgr2=0x7f785c0c6860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:37.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.917+0000 7f7863056640 1 --2- 172.21.15.116:0/2102284412 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f785c145c00 0x7f785c0c6860 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f784c009ae0 tx=0x7f784c019850 comp rx=0 tx=0).stop 2024-04-25T00:01:37.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.917+0000 7f7863056640 1 -- 172.21.15.116:0/2102284412 shutdown_connections 2024-04-25T00:01:37.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.917+0000 7f7863056640 1 --2- 172.21.15.116:0/2102284412 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f78480678f0 0x7f7848069db0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:37.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.917+0000 7f7863056640 1 --2- 172.21.15.116:0/2102284412 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f785c14e550 0x7f785c0c4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:37.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.917+0000 7f7863056640 1 --2- 172.21.15.116:0/2102284412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f785c151400 0x7f785c145fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:37.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.917+0000 7f7863056640 1 --2- 172.21.15.116:0/2102284412 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f785c145c00 0x7f785c0c6860 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:37.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.917+0000 7f7863056640 1 -- 172.21.15.116:0/2102284412 >> 172.21.15.116:0/2102284412 conn(0x7f785c0c3b00 msgr2=0x7f785c0a9f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:37.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.917+0000 7f7863056640 1 -- 172.21.15.116:0/2102284412 shutdown_connections 2024-04-25T00:01:37.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:37.917+0000 7f7863056640 1 -- 172.21.15.116:0/2102284412 wait complete. 2024-04-25T00:01:38.338 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 14, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T00:01:18.825887+0000', 'modified': '2024-04-25T00:01:22.888641+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4245}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4245': {'gid': 4245, 'name': 'a', 'rank': 0, 'incarnation': 10, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.116:6835/174781672', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.116:6833', 'nonce': 174781672}, {'type': 'v1', 'addr': '172.21.15.116:6835', 'nonce': 174781672}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'b', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4245, 'qdb_cluster': [4245]} 2024-04-25T00:01:38.339 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-25T00:01:38.339 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-25T00:01:38.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.469+0000 7f57d281f640 1 Processor -- start 2024-04-25T00:01:38.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.469+0000 7f57d281f640 1 -- start start 2024-04-25T00:01:38.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.469+0000 7f57d281f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f57cc0a94a0 0x7f57cc1538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:38.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.469+0000 7f57d281f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f57cc0a99f0 0x7f57cc153e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:38.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.469+0000 7f57d281f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57cc0c3910 con 0x7f57cc150d20 2024-04-25T00:01:38.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.469+0000 7f57d281f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f57cc0c16e0 con 0x7f57cc0a99f0 2024-04-25T00:01:38.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.469+0000 7f57d281f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f57cc0c1400 con 0x7f57cc0a94a0 2024-04-25T00:01:38.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.469+0000 7f57d0dbc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f57cc0a99f0 0x7f57cc153e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:38.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.469+0000 7f57cbfff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f57cc150d20 0x7f57cc0c2a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:55376/0 (socket says 172.21.15.116:55376) 2024-04-25T00:01:38.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.469+0000 7f57cbfff640 1 -- 172.21.15.116:0/4162334784 learned_addr learned my addr 172.21.15.116:0/4162334784 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:38.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.469+0000 7f57caffd640 1 -- 172.21.15.116:0/4162334784 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 259284257 0 0) 0x7f57cc0c3910 con 0x7f57cc150d20 2024-04-25T00:01:38.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.469+0000 7f57caffd640 1 -- 172.21.15.116:0/4162334784 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f57b8003530 con 0x7f57cc150d20 2024-04-25T00:01:38.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.469+0000 7f57cb7fe640 1 --2- 172.21.15.116:0/4162334784 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f57cc0a94a0 0x7f57cc1538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:38.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.469+0000 7f57d0dbc640 1 -- 172.21.15.116:0/4162334784 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f57cc0a94a0 msgr2=0x7f57cc1538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:38.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.469+0000 7f57d0dbc640 1 --2- 172.21.15.116:0/4162334784 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f57cc0a94a0 0x7f57cc1538e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:38.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.469+0000 7f57d0dbc640 1 -- 172.21.15.116:0/4162334784 >> v1:172.21.15.116:6789/0 conn(0x7f57cc150d20 legacy=0x7f57cc0c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:38.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.469+0000 7f57d0dbc640 1 -- 172.21.15.116:0/4162334784 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57cc06e660 con 0x7f57cc0a99f0 2024-04-25T00:01:38.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.473+0000 7f57d0dbc640 1 --2- 172.21.15.116:0/4162334784 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57cc0a99f0 0x7f57cc153e20 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f57c4009d90 tx=0x7f57c401c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5cd155c3f39ee09 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:38.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.473+0000 7f57caffd640 1 -- 172.21.15.116:0/4162334784 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57c40b2070 con 0x7f57cc0a99f0 2024-04-25T00:01:38.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.473+0000 7f57caffd640 1 -- 172.21.15.116:0/4162334784 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57c40a5830 con 0x7f57cc0a99f0 2024-04-25T00:01:38.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.473+0000 7f57caffd640 1 -- 172.21.15.116:0/4162334784 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57c40a5c00 con 0x7f57cc0a99f0 2024-04-25T00:01:38.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.473+0000 7f57d281f640 1 -- 172.21.15.116:0/4162334784 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57cc0a99f0 msgr2=0x7f57cc153e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:38.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.473+0000 7f57d281f640 1 --2- 172.21.15.116:0/4162334784 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57cc0a99f0 0x7f57cc153e20 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f57c4009d90 tx=0x7f57c401c970 comp rx=0 tx=0).stop 2024-04-25T00:01:38.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.473+0000 7f57d281f640 1 -- 172.21.15.116:0/4162334784 shutdown_connections 2024-04-25T00:01:38.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.473+0000 7f57d281f640 1 --2- 172.21.15.116:0/4162334784 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57cc0a99f0 0x7f57cc153e20 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:38.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.473+0000 7f57d281f640 1 --2- 172.21.15.116:0/4162334784 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f57cc0a94a0 0x7f57cc1538e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:38.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.473+0000 7f57d281f640 1 -- 172.21.15.116:0/4162334784 >> 172.21.15.116:0/4162334784 conn(0x7f57cc0c3b00 msgr2=0x7f57cc0c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:38.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.473+0000 7f57d281f640 1 -- 172.21.15.116:0/4162334784 shutdown_connections 2024-04-25T00:01:38.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.473+0000 7f57d281f640 1 -- 172.21.15.116:0/4162334784 wait complete. 2024-04-25T00:01:38.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.473+0000 7f57d281f640 1 Processor -- start 2024-04-25T00:01:38.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.473+0000 7f57d281f640 1 -- start start 2024-04-25T00:01:38.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.473+0000 7f57d281f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f57cc0a94a0 0x7f57cc156760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:38.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.473+0000 7f57d281f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f57cc0a99f0 0x7f57cc156ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:38.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.473+0000 7f57d281f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57cc150d20 0x7f57cc1408d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:38.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.473+0000 7f57d281f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f57cc1544e0 con 0x7f57cc150d20 2024-04-25T00:01:38.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.473+0000 7f57d281f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f57cc154360 con 0x7f57cc0a94a0 2024-04-25T00:01:38.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.473+0000 7f57d281f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f57cc160730 con 0x7f57cc0a99f0 2024-04-25T00:01:38.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.473+0000 7f57d0dbc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57cc150d20 0x7f57cc1408d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:38.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.473+0000 7f57d0dbc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57cc150d20 0x7f57cc1408d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43040/0 (socket says 172.21.15.116:43040) 2024-04-25T00:01:38.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.473+0000 7f57d0dbc640 1 -- 172.21.15.116:0/603653508 learned_addr learned my addr 172.21.15.116:0/603653508 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:38.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.477+0000 7f57cb7fe640 1 --2- 172.21.15.116:0/603653508 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f57cc0a99f0 0x7f57cc156ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:38.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.477+0000 7f57d0dbc640 1 -- 172.21.15.116:0/603653508 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f57cc0a99f0 msgr2=0x7f57cc156ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:38.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.477+0000 7f57cbfff640 1 --2- 172.21.15.116:0/603653508 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f57cc0a94a0 0x7f57cc156760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:38.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.477+0000 7f57d0dbc640 1 --2- 172.21.15.116:0/603653508 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f57cc0a99f0 0x7f57cc156ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:38.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.477+0000 7f57d0dbc640 1 -- 172.21.15.116:0/603653508 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f57cc0a94a0 msgr2=0x7f57cc156760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:38.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.477+0000 7f57d0dbc640 1 --2- 172.21.15.116:0/603653508 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f57cc0a94a0 0x7f57cc156760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:38.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.477+0000 7f57d0dbc640 1 -- 172.21.15.116:0/603653508 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57cc1571e0 con 0x7f57cc150d20 2024-04-25T00:01:38.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.477+0000 7f57cbfff640 1 --2- 172.21.15.116:0/603653508 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f57cc0a94a0 0x7f57cc156760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:38.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.477+0000 7f57d0dbc640 1 --2- 172.21.15.116:0/603653508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57cc150d20 0x7f57cc1408d0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f57c4009d90 tx=0x7f57c4019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:38.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.477+0000 7f57c8ff9640 1 -- 172.21.15.116:0/603653508 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57c40b2050 con 0x7f57cc150d20 2024-04-25T00:01:38.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.477+0000 7f57c8ff9640 1 -- 172.21.15.116:0/603653508 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57c40be070 con 0x7f57cc150d20 2024-04-25T00:01:38.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.477+0000 7f57d281f640 1 -- 172.21.15.116:0/603653508 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57cc1ec6a0 con 0x7f57cc150d20 2024-04-25T00:01:38.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.477+0000 7f57c8ff9640 1 -- 172.21.15.116:0/603653508 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57c40ada90 con 0x7f57cc150d20 2024-04-25T00:01:38.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.477+0000 7f57d281f640 1 -- 172.21.15.116:0/603653508 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57cc154ab0 con 0x7f57cc150d20 2024-04-25T00:01:38.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.477+0000 7f57c8ff9640 1 -- 172.21.15.116:0/603653508 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f57c40a56b0 con 0x7f57cc150d20 2024-04-25T00:01:38.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.477+0000 7f57c8ff9640 1 --2- 172.21.15.116:0/603653508 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f57ac0678c0 0x7f57ac069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:38.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.477+0000 7f57c8ff9640 1 -- 172.21.15.116:0/603653508 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7f57c4121020 con 0x7f57cc150d20 2024-04-25T00:01:38.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.477+0000 7f57cbfff640 1 --2- 172.21.15.116:0/603653508 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f57ac0678c0 0x7f57ac069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:38.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.477+0000 7f57d5998000 1 Processor -- start 2024-04-25T00:01:38.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.477+0000 7f57d5998000 1 -- start start 2024-04-25T00:01:38.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.477+0000 7f57cbfff640 1 --2- 172.21.15.116:0/603653508 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f57ac0678c0 0x7f57ac069d80 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f57bc005890 tx=0x7f57bc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:38.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.481+0000 7f57d5998000 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x559b10279b80 0x559b10279f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:38.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.481+0000 7f57d5998000 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b1027a4a0 0x559b1027a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:38.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.481+0000 7f57d5998000 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559b1027adc0 0x559b102c3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:38.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.481+0000 7f57d5998000 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559b100ca5e0 con 0x559b1027a4a0 2024-04-25T00:01:38.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.481+0000 7f57d5998000 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x559b10051c50 con 0x559b1027adc0 2024-04-25T00:01:38.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.481+0000 7f57d5998000 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x559b100c65a0 con 0x559b10279b80 2024-04-25T00:01:38.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.481+0000 7f57cb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b1027a4a0 0x559b1027a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:38.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.481+0000 7f57cb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b1027a4a0 0x559b1027a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43052/0 (socket says 172.21.15.116:43052) 2024-04-25T00:01:38.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.481+0000 7f57cb7fe640 1 -- 172.21.15.116:0/3977096662 learned_addr learned my addr 172.21.15.116:0/3977096662 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:38.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.481+0000 7f57cbfff640 1 --2- 172.21.15.116:0/3977096662 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x559b10279b80 0x559b10279f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:38.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.481+0000 7f57d0dbc640 1 --2- 172.21.15.116:0/3977096662 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559b1027adc0 0x559b102c3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:38.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.481+0000 7f57cb7fe640 1 -- 172.21.15.116:0/3977096662 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x559b10279b80 msgr2=0x559b10279f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:38.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.481+0000 7f57cb7fe640 1 --2- 172.21.15.116:0/3977096662 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x559b10279b80 0x559b10279f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:38.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.481+0000 7f57cb7fe640 1 -- 172.21.15.116:0/3977096662 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559b1027adc0 msgr2=0x559b102c3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:38.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.481+0000 7f57cb7fe640 1 --2- 172.21.15.116:0/3977096662 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559b1027adc0 0x559b102c3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:38.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.481+0000 7f57cb7fe640 1 -- 172.21.15.116:0/3977096662 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559b1004fd10 con 0x559b1027a4a0 2024-04-25T00:01:38.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.481+0000 7f57d0dbc640 1 --2- 172.21.15.116:0/3977096662 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559b1027adc0 0x559b102c3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:38.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.481+0000 7f57cb7fe640 1 --2- 172.21.15.116:0/3977096662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b1027a4a0 0x559b1027a880 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f57b400b810 tx=0x7f57b400bcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a0a9728d26d8ff server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:38.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.481+0000 7f57c1ffb640 1 -- 172.21.15.116:0/3977096662 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57b400dbb0 con 0x559b1027a4a0 2024-04-25T00:01:38.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.481+0000 7f57c1ffb640 1 -- 172.21.15.116:0/3977096662 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57b400dd50 con 0x559b1027a4a0 2024-04-25T00:01:38.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.481+0000 7f57c1ffb640 1 -- 172.21.15.116:0/3977096662 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57b4004250 con 0x559b1027a4a0 2024-04-25T00:01:38.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.481+0000 7f57d5998000 1 -- 172.21.15.116:0/3977096662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b1027a4a0 msgr2=0x559b1027a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:38.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.481+0000 7f57d5998000 1 --2- 172.21.15.116:0/3977096662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b1027a4a0 0x559b1027a880 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f57b400b810 tx=0x7f57b400bcd0 comp rx=0 tx=0).stop 2024-04-25T00:01:38.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.481+0000 7f57d5998000 1 -- 172.21.15.116:0/3977096662 shutdown_connections 2024-04-25T00:01:38.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.481+0000 7f57d5998000 1 --2- 172.21.15.116:0/3977096662 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559b1027adc0 0x559b102c3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:38.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.481+0000 7f57d5998000 1 --2- 172.21.15.116:0/3977096662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b1027a4a0 0x559b1027a880 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:38.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.481+0000 7f57d5998000 1 --2- 172.21.15.116:0/3977096662 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x559b10279b80 0x559b10279f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:38.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.481+0000 7f57d5998000 1 -- 172.21.15.116:0/3977096662 >> 172.21.15.116:0/3977096662 conn(0x559b102b7c00 msgr2=0x559b102b9ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:38.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.481+0000 7f57d5998000 1 -- 172.21.15.116:0/3977096662 shutdown_connections 2024-04-25T00:01:38.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.481+0000 7f57d5998000 1 -- 172.21.15.116:0/3977096662 wait complete. 2024-04-25T00:01:38.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.481+0000 7f57d5998000 1 -- start start 2024-04-25T00:01:38.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.481+0000 7f57d5998000 1 Processor -- start 2024-04-25T00:01:38.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.481+0000 7f57d5998000 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b10279b80 0x559b102b5d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:38.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.481+0000 7f57d5998000 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x559b1027a4a0 0x559b102b62c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:38.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.481+0000 7f57d5998000 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559b1027adc0 0x559b102b6800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:38.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.481+0000 7f57d5998000 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559b100cec10 con 0x559b10279b80 2024-04-25T00:01:38.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.481+0000 7f57d5998000 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x559b10084520 con 0x559b1027adc0 2024-04-25T00:01:38.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.481+0000 7f57d5998000 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x559b10282610 con 0x559b1027a4a0 2024-04-25T00:01:38.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.481+0000 7f57d5998000 10 client.0 _ll_register_callbacks cb 0x559b102675c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:01:38.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.481+0000 7f57cbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b10279b80 0x559b102b5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:38.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.481+0000 7f57cbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b10279b80 0x559b102b5d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43064/0 (socket says 172.21.15.116:43064) 2024-04-25T00:01:38.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.481+0000 7f57cbfff640 1 -- 172.21.15.116:0/2534225088 learned_addr learned my addr 172.21.15.116:0/2534225088 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:38.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.481+0000 7f57cbfff640 1 -- 172.21.15.116:0/2534225088 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x559b1027a4a0 msgr2=0x559b102b62c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:38.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.481+0000 7f57cbfff640 1 --2- 172.21.15.116:0/2534225088 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x559b1027a4a0 0x559b102b62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:38.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.481+0000 7f57cbfff640 1 -- 172.21.15.116:0/2534225088 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559b1027adc0 msgr2=0x559b102b6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:38.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.481+0000 7f57cbfff640 1 --2- 172.21.15.116:0/2534225088 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559b1027adc0 0x559b102b6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:38.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.481+0000 7f57cbfff640 1 -- 172.21.15.116:0/2534225088 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559b100627b0 con 0x559b10279b80 2024-04-25T00:01:38.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.481+0000 7f57cbfff640 1 --2- 172.21.15.116:0/2534225088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b10279b80 0x559b102b5d80 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f57bc001570 tx=0x7f57bc000bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6405ea94f0c1bcea server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:38.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.481+0000 7f57aaffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-04-25T00:01:38.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.481+0000 7f57aaffd640 1 -- 172.21.15.116:0/2534225088 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57bc026060 con 0x559b10279b80 2024-04-25T00:01:38.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.481+0000 7f57aaffd640 1 -- 172.21.15.116:0/2534225088 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7f57bc021dc0 con 0x559b10279b80 2024-04-25T00:01:38.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.481+0000 7f57d5998000 1 -- 172.21.15.116:0/2534225088 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57b400dd50 con 0x559b10279b80 2024-04-25T00:01:38.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.485+0000 7f57aaffd640 1 -- 172.21.15.116:0/2534225088 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=14) ==== 24+0+0 (secure 0 0 0) 0x7f57bc012410 con 0x559b10279b80 2024-04-25T00:01:38.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.485+0000 7f57d5998000 10 client.4938 fetch_fsmap learned FSMap version 14 2024-04-25T00:01:38.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.485+0000 7f57d5998000 1 -- 172.21.15.116:0/2534225088 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=14,monmap=2+}) -- 0x559b10051ea0 con 0x559b10279b80 2024-04-25T00:01:38.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.485+0000 7f57aaffd640 1 -- 172.21.15.116:0/2534225088 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57b400dd50 con 0x559b10279b80 2024-04-25T00:01:38.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.485+0000 7f57aaffd640 1 -- 172.21.15.116:0/2534225088 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 14) ==== 4128+0+0 (secure 0 0 0) 0x7f57bc021460 con 0x559b10279b80 2024-04-25T00:01:38.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.485+0000 7f57d5998000 10 client.4938 fetch_fsmap finished waiting for FSMap version 14 2024-04-25T00:01:38.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.485+0000 7f57d5998000 10 client.4938 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-25T00:01:38.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.485+0000 7f57d5998000 20 client.4938 populate_metadata read hostname 'smithi116' 2024-04-25T00:01:38.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.485+0000 7f57d5998000 1 --2- 172.21.15.116:0/2534225088 >> [v2:172.21.15.116:6833/174781672,v1:172.21.15.116:6835/174781672] conn(0x559b102cc8d0 0x559b102cecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:38.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.485+0000 7f57d5998000 4 client.4938 mds_command: new command op to 4245 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T00:01:38.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.485+0000 7f57d5998000 1 -- 172.21.15.116:0/2534225088 --> [v2:172.21.15.116:6833/174781672,v1:172.21.15.116:6835/174781672] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559b10065ab0 con 0x559b102cc8d0 2024-04-25T00:01:38.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.485+0000 7f57cb7fe640 1 --2- 172.21.15.116:0/2534225088 >> [v2:172.21.15.116:6833/174781672,v1:172.21.15.116:6835/174781672] conn(0x559b102cc8d0 0x559b102cecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:38.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.485+0000 7f57aaffd640 10 client.4938 ms_handle_connect on v2:172.21.15.116:6833/174781672 2024-04-25T00:01:38.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.485+0000 7f57cb7fe640 1 --2- 172.21.15.116:0/2534225088 >> [v2:172.21.15.116:6833/174781672,v1:172.21.15.116:6835/174781672] conn(0x559b102cc8d0 0x559b102cecc0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=62bd09804682ec86 server_cookie=db84b9bc5f6bd2c in_seq=0 out_seq=0 2024-04-25T00:01:38.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.485+0000 7f57aaffd640 1 -- 172.21.15.116:0/2534225088 <== mds.0 v2:172.21.15.116:6833/174781672 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559b10065ab0 con 0x559b102cc8d0 2024-04-25T00:01:38.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.485+0000 7f57aaffd640 10 client.4938 handle_command_reply: tid=0 2024-04-25T00:01:38.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.485+0000 7f57d5998000 1 client.4938 shutdown 2024-04-25T00:01:38.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.485+0000 7f57d5998000 1 -- 172.21.15.116:0/2534225088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b10279b80 msgr2=0x559b102b5d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:38.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.485+0000 7f57d5998000 1 --2- 172.21.15.116:0/2534225088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b10279b80 0x559b102b5d80 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f57bc001570 tx=0x7f57bc000bd0 comp rx=0 tx=0).stop 2024-04-25T00:01:38.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.485+0000 7f57d5998000 1 -- 172.21.15.116:0/2534225088 shutdown_connections 2024-04-25T00:01:38.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.485+0000 7f57d5998000 1 --2- 172.21.15.116:0/2534225088 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559b1027adc0 0x559b102b6800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:38.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.485+0000 7f57d5998000 1 --2- 172.21.15.116:0/2534225088 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x559b1027a4a0 0x559b102b62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:38.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.485+0000 7f57d5998000 1 --2- 172.21.15.116:0/2534225088 >> [v2:172.21.15.116:6833/174781672,v1:172.21.15.116:6835/174781672] conn(0x559b102cc8d0 0x559b102cecc0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:38.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.485+0000 7f57d5998000 1 --2- 172.21.15.116:0/2534225088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b10279b80 0x559b102b5d80 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:38.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.485+0000 7f57aaffd640 0 client.4938 ms_handle_reset on v2:172.21.15.116:6833/174781672 2024-04-25T00:01:38.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.485+0000 7f57d5998000 1 -- 172.21.15.116:0/2534225088 >> 172.21.15.116:0/2534225088 conn(0x559b102b7c00 msgr2=0x559b10279280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:38.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.485+0000 7f57d5998000 1 -- 172.21.15.116:0/2534225088 shutdown_connections 2024-04-25T00:01:38.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.485+0000 7f57d5998000 1 -- 172.21.15.116:0/2534225088 wait complete. 2024-04-25T00:01:38.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.485+0000 7f57d5998000 20 client.4938 trim_cache size 0 max 16384 2024-04-25T00:01:38.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.501+0000 7f57d5998000 1 Processor -- start 2024-04-25T00:01:38.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.501+0000 7f57d5998000 1 -- start start 2024-04-25T00:01:38.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.501+0000 7f57d5998000 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x559b10279b80 0x559b10329830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:38.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.501+0000 7f57d5998000 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b1027a4a0 0x559b102d1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:38.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.501+0000 7f57d5998000 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559b1027adc0 0x559b102d2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:38.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.501+0000 7f57d5998000 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f57cc1fb900 con 0x559b1027a4a0 2024-04-25T00:01:38.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.501+0000 7f57d5998000 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f57cc1f9370 con 0x559b1027adc0 2024-04-25T00:01:38.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.501+0000 7f57d5998000 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f57cc1f9960 con 0x559b10279b80 2024-04-25T00:01:38.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.501+0000 7f57cb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b1027a4a0 0x559b102d1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:38.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.501+0000 7f57cb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b1027a4a0 0x559b102d1cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43066/0 (socket says 172.21.15.116:43066) 2024-04-25T00:01:38.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.501+0000 7f57cb7fe640 1 -- 172.21.15.116:0/1335231879 learned_addr learned my addr 172.21.15.116:0/1335231879 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:38.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.501+0000 7f57cb7fe640 1 -- 172.21.15.116:0/1335231879 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x559b10279b80 msgr2=0x559b10329830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:38.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.501+0000 7f57cb7fe640 1 --2- 172.21.15.116:0/1335231879 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x559b10279b80 0x559b10329830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:38.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.501+0000 7f57cb7fe640 1 -- 172.21.15.116:0/1335231879 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559b1027adc0 msgr2=0x559b102d2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:38.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.501+0000 7f57cb7fe640 1 --2- 172.21.15.116:0/1335231879 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559b1027adc0 0x559b102d2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:38.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.501+0000 7f57cb7fe640 1 -- 172.21.15.116:0/1335231879 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559b10052e80 con 0x559b1027a4a0 2024-04-25T00:01:38.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.501+0000 7f57cb7fe640 1 --2- 172.21.15.116:0/1335231879 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b1027a4a0 0x559b102d1cf0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f57b400b210 tx=0x7f57b4012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8a9363ff7b03fb3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:38.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.505+0000 7f57c17fa640 1 -- 172.21.15.116:0/1335231879 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57b4004030 con 0x559b1027a4a0 2024-04-25T00:01:38.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.505+0000 7f57c17fa640 1 -- 172.21.15.116:0/1335231879 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57b400b040 con 0x559b1027a4a0 2024-04-25T00:01:38.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.505+0000 7f57c17fa640 1 -- 172.21.15.116:0/1335231879 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57b4004410 con 0x559b1027a4a0 2024-04-25T00:01:38.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.505+0000 7f57d5998000 1 -- 172.21.15.116:0/1335231879 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b1027a4a0 msgr2=0x559b102d1cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:38.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.505+0000 7f57d5998000 1 --2- 172.21.15.116:0/1335231879 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b1027a4a0 0x559b102d1cf0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f57b400b210 tx=0x7f57b4012000 comp rx=0 tx=0).stop 2024-04-25T00:01:38.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.505+0000 7f57d5998000 1 -- 172.21.15.116:0/1335231879 shutdown_connections 2024-04-25T00:01:38.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.505+0000 7f57d5998000 1 --2- 172.21.15.116:0/1335231879 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559b1027adc0 0x559b102d2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:38.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.505+0000 7f57d5998000 1 --2- 172.21.15.116:0/1335231879 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b1027a4a0 0x559b102d1cf0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:38.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.505+0000 7f57d5998000 1 --2- 172.21.15.116:0/1335231879 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x559b10279b80 0x559b10329830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:38.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.505+0000 7f57d5998000 1 -- 172.21.15.116:0/1335231879 >> 172.21.15.116:0/1335231879 conn(0x559b102b7c00 msgr2=0x559b102782e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:38.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.505+0000 7f57d5998000 1 -- 172.21.15.116:0/1335231879 shutdown_connections 2024-04-25T00:01:38.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.505+0000 7f57d5998000 1 -- 172.21.15.116:0/1335231879 wait complete. 2024-04-25T00:01:38.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.505+0000 7f57d5998000 1 -- start start 2024-04-25T00:01:38.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.505+0000 7f57d5998000 1 Processor -- start 2024-04-25T00:01:38.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.505+0000 7f57d5998000 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559b10279b80 0x559b102f18e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:38.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.505+0000 7f57d5998000 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x559b1027a4a0 0x559b102f1e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:38.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.505+0000 7f57d5998000 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b1027adc0 0x559b102f2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:38.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.505+0000 7f57d5998000 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x559b10285250 con 0x559b1027adc0 2024-04-25T00:01:38.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.505+0000 7f57d5998000 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x559b10070810 con 0x559b10279b80 2024-04-25T00:01:38.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.505+0000 7f57d5998000 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x559b1008da70 con 0x559b1027a4a0 2024-04-25T00:01:38.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.505+0000 7f57d5998000 10 client.0 _ll_register_callbacks cb 0x7f57bc001cf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T00:01:38.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.505+0000 7f57d0dbc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b1027adc0 0x559b102f2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:38.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.505+0000 7f57d0dbc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b1027adc0 0x559b102f2360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43078/0 (socket says 172.21.15.116:43078) 2024-04-25T00:01:38.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.505+0000 7f57d0dbc640 1 -- 172.21.15.116:0/3871472333 learned_addr learned my addr 172.21.15.116:0/3871472333 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:38.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.505+0000 7f57d0dbc640 1 -- 172.21.15.116:0/3871472333 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x559b1027a4a0 msgr2=0x559b102f1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:38.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.505+0000 7f57cbfff640 1 --2- 172.21.15.116:0/3871472333 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559b10279b80 0x559b102f18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:38.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.505+0000 7f57d0dbc640 1 --2- 172.21.15.116:0/3871472333 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x559b1027a4a0 0x559b102f1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:38.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.505+0000 7f57d0dbc640 1 -- 172.21.15.116:0/3871472333 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559b10279b80 msgr2=0x559b102f18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:38.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.505+0000 7f57d0dbc640 1 --2- 172.21.15.116:0/3871472333 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559b10279b80 0x559b102f18e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:38.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.505+0000 7f57d0dbc640 1 -- 172.21.15.116:0/3871472333 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559b10097760 con 0x559b1027adc0 2024-04-25T00:01:38.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.505+0000 7f57d0dbc640 1 --2- 172.21.15.116:0/3871472333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b1027adc0 0x559b102f2360 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f57c4098350 tx=0x7f57c40ab340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3961fba93a1b39a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:38.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.505+0000 7f57aaffd640 10 client.0 ms_handle_connect on v2:172.21.15.116:3300/0 2024-04-25T00:01:38.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.505+0000 7f57aaffd640 1 -- 172.21.15.116:0/3871472333 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57c40f5570 con 0x559b1027adc0 2024-04-25T00:01:38.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.505+0000 7f57aaffd640 1 -- 172.21.15.116:0/3871472333 <== mon.0 v2:172.21.15.116:3300/0 2 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7f57c40f1420 con 0x559b1027adc0 2024-04-25T00:01:38.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.505+0000 7f57d5998000 1 -- 172.21.15.116:0/3871472333 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57b400b040 con 0x559b1027adc0 2024-04-25T00:01:38.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.505+0000 7f57aaffd640 1 -- 172.21.15.116:0/3871472333 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_get_version_reply(handle=1 version=14) ==== 24+0+0 (secure 0 0 0) 0x7f57c40f9da0 con 0x559b1027adc0 2024-04-25T00:01:38.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.505+0000 7f57d5998000 10 client.4944 fetch_fsmap learned FSMap version 14 2024-04-25T00:01:38.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.505+0000 7f57d5998000 1 -- 172.21.15.116:0/3871472333 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,fsmap=14,monmap=2+}) -- 0x559b101239e0 con 0x559b1027adc0 2024-04-25T00:01:38.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.505+0000 7f57aaffd640 1 -- 172.21.15.116:0/3871472333 <== mon.0 v2:172.21.15.116:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57b400b040 con 0x559b1027adc0 2024-04-25T00:01:38.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.505+0000 7f57aaffd640 1 -- 172.21.15.116:0/3871472333 <== mon.0 v2:172.21.15.116:3300/0 5 ==== fsmap(e 14) ==== 4128+0+0 (secure 0 0 0) 0x7f57c40f9910 con 0x559b1027adc0 2024-04-25T00:01:38.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.505+0000 7f57d5998000 10 client.4944 fetch_fsmap finished waiting for FSMap version 14 2024-04-25T00:01:38.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.505+0000 7f57d5998000 10 client.4944 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-25T00:01:38.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.505+0000 7f57d5998000 20 client.4944 populate_metadata read hostname 'smithi116' 2024-04-25T00:01:38.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.505+0000 7f57d5998000 1 --2- 172.21.15.116:0/3871472333 >> [v2:172.21.15.116:6833/174781672,v1:172.21.15.116:6835/174781672] conn(0x559b102f2cc0 0x559b103356e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:38.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.505+0000 7f57d5998000 4 client.4944 mds_command: new command op to 4245 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T00:01:38.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.505+0000 7f57d5998000 1 -- 172.21.15.116:0/3871472333 --> [v2:172.21.15.116:6833/174781672,v1:172.21.15.116:6835/174781672] -- command(tid 0: {"prefix": "status"}) -- 0x559b100d7b20 con 0x559b102f2cc0 2024-04-25T00:01:38.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.505+0000 7f57cbfff640 1 --2- 172.21.15.116:0/3871472333 >> [v2:172.21.15.116:6833/174781672,v1:172.21.15.116:6835/174781672] conn(0x559b102f2cc0 0x559b103356e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:38.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.505+0000 7f57cbfff640 1 --2- 172.21.15.116:0/3871472333 >> [v2:172.21.15.116:6833/174781672,v1:172.21.15.116:6835/174781672] conn(0x559b102f2cc0 0x559b103356e0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90821bdfd9e20e86 server_cookie=8ae6cfefc32203c0 in_seq=0 out_seq=0 2024-04-25T00:01:38.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.505+0000 7f57aaffd640 10 client.4944 ms_handle_connect on v2:172.21.15.116:6833/174781672 2024-04-25T00:01:38.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.509+0000 7f57aaffd640 1 -- 172.21.15.116:0/3871472333 <== mds.0 v2:172.21.15.116:6833/174781672 1 ==== command_reply(tid 0: 0 ) ==== 8+0+314 (crc 0 0 0) 0x559b100d7b20 con 0x559b102f2cc0 2024-04-25T00:01:38.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.509+0000 7f57aaffd640 10 client.4944 handle_command_reply: tid=0 2024-04-25T00:01:38.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.509+0000 7f57d5998000 1 client.4944 shutdown 2024-04-25T00:01:38.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.509+0000 7f57d5998000 1 -- 172.21.15.116:0/3871472333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b1027adc0 msgr2=0x559b102f2360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:38.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.509+0000 7f57d5998000 1 --2- 172.21.15.116:0/3871472333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b1027adc0 0x559b102f2360 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f57c4098350 tx=0x7f57c40ab340 comp rx=0 tx=0).stop 2024-04-25T00:01:38.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.509+0000 7f57d5998000 1 -- 172.21.15.116:0/3871472333 shutdown_connections 2024-04-25T00:01:38.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.509+0000 7f57d5998000 1 --2- 172.21.15.116:0/3871472333 >> [v2:172.21.15.116:6833/174781672,v1:172.21.15.116:6835/174781672] conn(0x559b102f2cc0 0x559b103356e0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:38.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.509+0000 7f57d5998000 1 --2- 172.21.15.116:0/3871472333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x559b1027adc0 0x559b102f2360 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:38.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.509+0000 7f57d5998000 1 --2- 172.21.15.116:0/3871472333 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x559b1027a4a0 0x559b102f1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:38.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.509+0000 7f57aaffd640 0 client.4944 ms_handle_reset on v2:172.21.15.116:6833/174781672 2024-04-25T00:01:38.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.509+0000 7f57d5998000 1 --2- 172.21.15.116:0/3871472333 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x559b10279b80 0x559b102f18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:38.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.509+0000 7f57d5998000 1 -- 172.21.15.116:0/3871472333 >> 172.21.15.116:0/3871472333 conn(0x559b102b7c00 msgr2=0x559b10278b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:38.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.509+0000 7f57d5998000 1 -- 172.21.15.116:0/3871472333 shutdown_connections 2024-04-25T00:01:38.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.509+0000 7f57d5998000 1 -- 172.21.15.116:0/3871472333 wait complete. 2024-04-25T00:01:38.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.509+0000 7f57d5998000 20 client.4944 trim_cache size 0 max 16384 2024-04-25T00:01:38.517 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-25T00:01:38.517 INFO:teuthology.orchestra.run.smithi116.stdout: "cluster_fsid": "6d6a1aa7-0478-4c83-8057-05917e81f046", 2024-04-25T00:01:38.517 INFO:teuthology.orchestra.run.smithi116.stdout: "whoami": 0, 2024-04-25T00:01:38.517 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 4245, 2024-04-25T00:01:38.517 INFO:teuthology.orchestra.run.smithi116.stdout: "want_state": "up:active", 2024-04-25T00:01:38.517 INFO:teuthology.orchestra.run.smithi116.stdout: "state": "up:active", 2024-04-25T00:01:38.517 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "b", 2024-04-25T00:01:38.518 INFO:teuthology.orchestra.run.smithi116.stdout: "rank_uptime": 19.007922503, 2024-04-25T00:01:38.518 INFO:teuthology.orchestra.run.smithi116.stdout: "mdsmap_epoch": 14, 2024-04-25T00:01:38.518 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch": 27, 2024-04-25T00:01:38.518 INFO:teuthology.orchestra.run.smithi116.stdout: "osdmap_epoch_barrier": 27, 2024-04-25T00:01:38.518 INFO:teuthology.orchestra.run.smithi116.stdout: "uptime": 39.787713474 2024-04-25T00:01:38.518 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-25T00:01:38.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.509+0000 7f57d281f640 1 -- 172.21.15.116:0/603653508 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f57ac0678c0 msgr2=0x7f57ac069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:38.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.509+0000 7f57d281f640 1 --2- 172.21.15.116:0/603653508 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f57ac0678c0 0x7f57ac069d80 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f57bc005890 tx=0x7f57bc009040 comp rx=0 tx=0).stop 2024-04-25T00:01:38.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.509+0000 7f57d281f640 1 -- 172.21.15.116:0/603653508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57cc150d20 msgr2=0x7f57cc1408d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:38.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.509+0000 7f57d281f640 1 --2- 172.21.15.116:0/603653508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57cc150d20 0x7f57cc1408d0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f57c4009d90 tx=0x7f57c4019710 comp rx=0 tx=0).stop 2024-04-25T00:01:38.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.509+0000 7f57d281f640 1 -- 172.21.15.116:0/603653508 shutdown_connections 2024-04-25T00:01:38.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.509+0000 7f57d281f640 1 --2- 172.21.15.116:0/603653508 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f57ac0678c0 0x7f57ac069d80 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:38.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.509+0000 7f57d281f640 1 --2- 172.21.15.116:0/603653508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57cc150d20 0x7f57cc1408d0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:38.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.509+0000 7f57d281f640 1 --2- 172.21.15.116:0/603653508 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f57cc0a99f0 0x7f57cc156ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:38.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.509+0000 7f57d281f640 1 --2- 172.21.15.116:0/603653508 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f57cc0a94a0 0x7f57cc156760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:38.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.509+0000 7f57d281f640 1 -- 172.21.15.116:0/603653508 >> 172.21.15.116:0/603653508 conn(0x7f57cc0c3b00 msgr2=0x7f57cc146fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:38.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.509+0000 7f57d281f640 1 -- 172.21.15.116:0/603653508 shutdown_connections 2024-04-25T00:01:38.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.509+0000 7f57d281f640 1 -- 172.21.15.116:0/603653508 wait complete. 2024-04-25T00:01:38.527 INFO:teuthology.run_tasks:Running task sequential... 2024-04-25T00:01:38.616 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-25T00:01:38.617 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-25T00:01:38.617 INFO:teuthology.task.print:Enabling mgr modules 2024-04-25T00:01:38.617 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-04-25T00:01:38.672 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-04-25T00:01:38.672 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.1': {}} 2024-04-25T00:01:38.672 INFO:tasks.ceph_fuse:client.0 config is {'cephfs_name': 'a'} 2024-04-25T00:01:38.672 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-25T00:01:38.672 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-25T00:01:38.672 INFO:tasks.ceph_fuse:client.1 config is {'cephfs_name': 'b'} 2024-04-25T00:01:38.673 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-25T00:01:38.673 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-25T00:01:38.673 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:01:38.673 DEBUG:teuthology.orchestra.run.smithi116:> ip netns list 2024-04-25T00:01:38.679 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:01:38.680 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link delete ceph-brx 2024-04-25T00:01:38.738 INFO:teuthology.orchestra.run.smithi116.stderr:Cannot find device "ceph-brx" 2024-04-25T00:01:38.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T00:01:38.740 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:01:38.740 DEBUG:teuthology.orchestra.run.smithi174:> ip netns list 2024-04-25T00:01:38.747 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:01:38.747 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip link delete ceph-brx 2024-04-25T00:01:38.806 INFO:teuthology.orchestra.run.smithi174.stderr:Cannot find device "ceph-brx" 2024-04-25T00:01:38.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T00:01:38.807 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-04-25T00:01:38.807 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T00:01:38.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.929+0000 7f237b917640 1 Processor -- start 2024-04-25T00:01:38.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.929+0000 7f237b917640 1 -- start start 2024-04-25T00:01:38.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.929+0000 7f237b917640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f2374070dd0 0x7f2374148890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:38.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.929+0000 7f237b917640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f237406e890 0x7f23740c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:38.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.929+0000 7f237b917640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f23740c3540 0x7f2374072590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:38.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.929+0000 7f237b917640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f2374152c10 con 0x7f237406e890 2024-04-25T00:01:38.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.929+0000 7f237b917640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f2374151e50 con 0x7f2374070dd0 2024-04-25T00:01:38.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.929+0000 7f237b917640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f2374152130 con 0x7f23740c3540 2024-04-25T00:01:38.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.929+0000 7f2378eb2640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f237406e890 0x7f23740c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:38.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.929+0000 7f2378eb2640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f237406e890 0x7f23740c2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43094/0 (socket says 172.21.15.116:43094) 2024-04-25T00:01:38.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.929+0000 7f2378eb2640 1 -- 172.21.15.116:0/787687401 learned_addr learned my addr 172.21.15.116:0/787687401 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:38.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.929+0000 7f2379eb4640 1 --2- 172.21.15.116:0/787687401 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f23740c3540 0x7f2374072590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:38.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.929+0000 7f23796b3640 1 --2- 172.21.15.116:0/787687401 >> v2:172.21.15.174:3300/0 conn(0x7f2374070dd0 0x7f2374148890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:38.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.929+0000 7f2378eb2640 1 -- 172.21.15.116:0/787687401 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f23740c3540 msgr2=0x7f2374072590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:38.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.929+0000 7f2378eb2640 1 --2- 172.21.15.116:0/787687401 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f23740c3540 0x7f2374072590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:38.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.929+0000 7f2378eb2640 1 -- 172.21.15.116:0/787687401 >> v2:172.21.15.174:3300/0 conn(0x7f2374070dd0 msgr2=0x7f2374148890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:38.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.929+0000 7f2378eb2640 1 --2- 172.21.15.116:0/787687401 >> v2:172.21.15.174:3300/0 conn(0x7f2374070dd0 0x7f2374148890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:38.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.929+0000 7f2378eb2640 1 -- 172.21.15.116:0/787687401 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2374152410 con 0x7f237406e890 2024-04-25T00:01:38.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.929+0000 7f23796b3640 1 --2- 172.21.15.116:0/787687401 >> v2:172.21.15.174:3300/0 conn(0x7f2374070dd0 0x7f2374148890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:38.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.933+0000 7f2378eb2640 1 --2- 172.21.15.116:0/787687401 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f237406e890 0x7f23740c2a10 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f237406f3d0 tx=0x7f237001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=588ac273199e6fa9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:38.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.933+0000 7f236bfff640 1 -- 172.21.15.116:0/787687401 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23700b0070 con 0x7f237406e890 2024-04-25T00:01:38.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.933+0000 7f236bfff640 1 -- 172.21.15.116:0/787687401 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23700ab600 con 0x7f237406e890 2024-04-25T00:01:38.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.933+0000 7f236bfff640 1 -- 172.21.15.116:0/787687401 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23700ab9d0 con 0x7f237406e890 2024-04-25T00:01:38.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.933+0000 7f237b917640 1 -- 172.21.15.116:0/787687401 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f237406e890 msgr2=0x7f23740c2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:38.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.933+0000 7f237b917640 1 --2- 172.21.15.116:0/787687401 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f237406e890 0x7f23740c2a10 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f237406f3d0 tx=0x7f237001c520 comp rx=0 tx=0).stop 2024-04-25T00:01:38.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.933+0000 7f237b917640 1 -- 172.21.15.116:0/787687401 shutdown_connections 2024-04-25T00:01:38.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.933+0000 7f237b917640 1 --2- 172.21.15.116:0/787687401 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f23740c3540 0x7f2374072590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:38.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.933+0000 7f237b917640 1 --2- 172.21.15.116:0/787687401 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f237406e890 0x7f23740c2a10 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:38.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.933+0000 7f237b917640 1 --2- 172.21.15.116:0/787687401 >> v2:172.21.15.174:3300/0 conn(0x7f2374070dd0 0x7f2374148890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:38.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.933+0000 7f237b917640 1 -- 172.21.15.116:0/787687401 >> 172.21.15.116:0/787687401 conn(0x7f23740c3b00 msgr2=0x7f23740c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:38.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.933+0000 7f237b917640 1 -- 172.21.15.116:0/787687401 shutdown_connections 2024-04-25T00:01:38.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.933+0000 7f237b917640 1 -- 172.21.15.116:0/787687401 wait complete. 2024-04-25T00:01:38.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.933+0000 7f237b917640 1 Processor -- start 2024-04-25T00:01:38.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.933+0000 7f237b917640 1 -- start start 2024-04-25T00:01:38.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.933+0000 7f237b917640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f237406e890 0x7f2374148af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:38.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.933+0000 7f237b917640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2374070dd0 0x7f23741548e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:38.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.933+0000 7f237b917640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f23740c3540 0x7f2374154e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:38.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.933+0000 7f237b917640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2374071c00 con 0x7f2374070dd0 2024-04-25T00:01:38.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.933+0000 7f237b917640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2374071a80 con 0x7f237406e890 2024-04-25T00:01:38.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.933+0000 7f237b917640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f23740ab590 con 0x7f23740c3540 2024-04-25T00:01:38.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.933+0000 7f2378eb2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2374070dd0 0x7f23741548e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:38.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.933+0000 7f2378eb2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2374070dd0 0x7f23741548e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43100/0 (socket says 172.21.15.116:43100) 2024-04-25T00:01:38.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.933+0000 7f2378eb2640 1 -- 172.21.15.116:0/1810540947 learned_addr learned my addr 172.21.15.116:0/1810540947 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:38.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.937+0000 7f23796b3640 1 --2- 172.21.15.116:0/1810540947 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f237406e890 0x7f2374148af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:38.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.937+0000 7f2378eb2640 1 -- 172.21.15.116:0/1810540947 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f23740c3540 msgr2=0x7f2374154e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:38.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.937+0000 7f2378eb2640 1 --2- 172.21.15.116:0/1810540947 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f23740c3540 0x7f2374154e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:38.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.937+0000 7f2378eb2640 1 -- 172.21.15.116:0/1810540947 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f237406e890 msgr2=0x7f2374148af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:38.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.937+0000 7f2378eb2640 1 --2- 172.21.15.116:0/1810540947 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f237406e890 0x7f2374148af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:38.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.937+0000 7f2378eb2640 1 -- 172.21.15.116:0/1810540947 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f237414b360 con 0x7f2374070dd0 2024-04-25T00:01:38.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.937+0000 7f2378eb2640 1 --2- 172.21.15.116:0/1810540947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2374070dd0 0x7f23741548e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f237001abd0 tx=0x7f23700192c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:38.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.937+0000 7f2369ffb640 1 -- 172.21.15.116:0/1810540947 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23700b0040 con 0x7f2374070dd0 2024-04-25T00:01:38.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.937+0000 7f2369ffb640 1 -- 172.21.15.116:0/1810540947 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23700b4620 con 0x7f2374070dd0 2024-04-25T00:01:38.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.937+0000 7f2369ffb640 1 -- 172.21.15.116:0/1810540947 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23700b4a10 con 0x7f2374070dd0 2024-04-25T00:01:38.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.937+0000 7f237b917640 1 -- 172.21.15.116:0/1810540947 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f237414b5e0 con 0x7f2374070dd0 2024-04-25T00:01:38.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.937+0000 7f237b917640 1 -- 172.21.15.116:0/1810540947 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f237414f370 con 0x7f2374070dd0 2024-04-25T00:01:38.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.937+0000 7f237b917640 1 -- 172.21.15.116:0/1810540947 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2374068350 con 0x7f2374070dd0 2024-04-25T00:01:38.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.937+0000 7f2369ffb640 1 -- 172.21.15.116:0/1810540947 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f23700ab4c0 con 0x7f2374070dd0 2024-04-25T00:01:38.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.937+0000 7f2369ffb640 1 --2- 172.21.15.116:0/1810540947 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f23580678f0 0x7f2358069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:38.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.937+0000 7f2369ffb640 1 -- 172.21.15.116:0/1810540947 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7f237011fcb0 con 0x7f2374070dd0 2024-04-25T00:01:38.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.937+0000 7f23796b3640 1 --2- 172.21.15.116:0/1810540947 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f23580678f0 0x7f2358069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:38.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.941+0000 7f23796b3640 1 --2- 172.21.15.116:0/1810540947 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f23580678f0 0x7f2358069db0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f2364006fd0 tx=0x7f2364008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:38.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:38.941+0000 7f2369ffb640 1 -- 172.21.15.116:0/1810540947 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f237414b360 con 0x7f2374070dd0 2024-04-25T00:01:39.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:39.121+0000 7f237b917640 1 -- 172.21.15.116:0/1810540947 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f23740c1e10 con 0x7f2374070dd0 2024-04-25T00:01:39.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:39.121+0000 7f2369ffb640 1 -- 172.21.15.116:0/1810540947 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v14) ==== 53+0+116 (secure 0 0 0) 0x7f2370016110 con 0x7f2374070dd0 2024-04-25T00:01:39.126 INFO:teuthology.orchestra.run.smithi116.stdout:name: a, metadata pool: a_metadata, data pools: [a_data ] 2024-04-25T00:01:39.126 INFO:teuthology.orchestra.run.smithi116.stdout:name: b, metadata pool: b_metadata, data pools: [b_data ] 2024-04-25T00:01:39.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:39.125+0000 7f237b917640 1 -- 172.21.15.116:0/1810540947 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f23580678f0 msgr2=0x7f2358069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:39.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:39.125+0000 7f237b917640 1 --2- 172.21.15.116:0/1810540947 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f23580678f0 0x7f2358069db0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f2364006fd0 tx=0x7f2364008040 comp rx=0 tx=0).stop 2024-04-25T00:01:39.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:39.125+0000 7f237b917640 1 -- 172.21.15.116:0/1810540947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2374070dd0 msgr2=0x7f23741548e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:39.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:39.125+0000 7f237b917640 1 --2- 172.21.15.116:0/1810540947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2374070dd0 0x7f23741548e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f237001abd0 tx=0x7f23700192c0 comp rx=0 tx=0).stop 2024-04-25T00:01:39.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:39.125+0000 7f237b917640 1 -- 172.21.15.116:0/1810540947 shutdown_connections 2024-04-25T00:01:39.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:39.125+0000 7f237b917640 1 --2- 172.21.15.116:0/1810540947 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f23580678f0 0x7f2358069db0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:39.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:39.125+0000 7f237b917640 1 --2- 172.21.15.116:0/1810540947 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f23740c3540 0x7f2374154e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:39.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:39.125+0000 7f237b917640 1 --2- 172.21.15.116:0/1810540947 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2374070dd0 0x7f23741548e0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:39.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:39.125+0000 7f237b917640 1 --2- 172.21.15.116:0/1810540947 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f237406e890 0x7f2374148af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:39.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:39.125+0000 7f237b917640 1 -- 172.21.15.116:0/1810540947 >> 172.21.15.116:0/1810540947 conn(0x7f23740c3b00 msgr2=0x7f23740ab9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:39.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:39.125+0000 7f237b917640 1 -- 172.21.15.116:0/1810540947 shutdown_connections 2024-04-25T00:01:39.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:39.125+0000 7f237b917640 1 -- 172.21.15.116:0/1810540947 wait complete. 2024-04-25T00:01:39.143 INFO:tasks.cephfs.mount:Mounting Ceph FS a; just confirmed its presence on cluster 2024-04-25T00:01:39.143 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T00:01:39.143 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi116.front.sepia.ceph.com 2024-04-25T00:01:39.143 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-25T00:01:39.143 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-25T00:01:39.143 INFO:tasks.cephfs.mount:self.cephfs_name = a 2024-04-25T00:01:39.144 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-25T00:01:39.144 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T00:01:39.144 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-25T00:01:39.144 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:01:39.144 DEBUG:teuthology.orchestra.run.smithi116:> ip addr 2024-04-25T00:01:39.150 INFO:teuthology.orchestra.run.smithi116.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T00:01:39.150 INFO:teuthology.orchestra.run.smithi116.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T00:01:39.150 INFO:teuthology.orchestra.run.smithi116.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T00:01:39.150 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2024-04-25T00:01:39.150 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 ::1/128 scope host 2024-04-25T00:01:39.150 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2024-04-25T00:01:39.150 INFO:teuthology.orchestra.run.smithi116.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-25T00:01:39.150 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:d8:cb:90 brd ff:ff:ff:ff:ff:ff 2024-04-25T00:01:39.150 INFO:teuthology.orchestra.run.smithi116.stdout: altname enp7s0f0 2024-04-25T00:01:39.151 INFO:teuthology.orchestra.run.smithi116.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-25T00:01:39.151 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:d8:cb:91 brd ff:ff:ff:ff:ff:ff 2024-04-25T00:01:39.151 INFO:teuthology.orchestra.run.smithi116.stdout: altname enp7s0f1 2024-04-25T00:01:39.151 INFO:teuthology.orchestra.run.smithi116.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-25T00:01:39.151 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:8f:3f:66 brd ff:ff:ff:ff:ff:ff 2024-04-25T00:01:39.151 INFO:teuthology.orchestra.run.smithi116.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T00:01:39.151 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 0c:c4:7a:8f:3f:67 brd ff:ff:ff:ff:ff:ff 2024-04-25T00:01:39.151 INFO:teuthology.orchestra.run.smithi116.stdout: inet 172.21.15.116/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-25T00:01:39.151 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft 42481sec preferred_lft 42481sec 2024-04-25T00:01:39.151 INFO:teuthology.orchestra.run.smithi116.stdout: inet6 fe80::ec4:7aff:fe8f:3f67/64 scope link 2024-04-25T00:01:39.151 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2024-04-25T00:01:39.151 INFO:teuthology.orchestra.run.smithi116.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-04-25T00:01:39.151 INFO:teuthology.orchestra.run.smithi116.stdout: link/ether 02:42:32:3a:03:50 brd ff:ff:ff:ff:ff:ff 2024-04-25T00:01:39.152 INFO:teuthology.orchestra.run.smithi116.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-04-25T00:01:39.152 INFO:teuthology.orchestra.run.smithi116.stdout: valid_lft forever preferred_lft forever 2024-04-25T00:01:39.152 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-25T00:01:39.152 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T00:01:39.152 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-04-25T00:01:39.152 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link add name ceph-brx type bridge 2024-04-25T00:01:39.152 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip addr flush dev ceph-brx 2024-04-25T00:01:39.152 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link set ceph-brx up 2024-04-25T00:01:39.153 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-25T00:01:39.153 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-04-25T00:01:39.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:39 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T00:01:39.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T00:01:39.272 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:01:39.272 DEBUG:teuthology.orchestra.run.smithi116:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T00:01:39.322 INFO:teuthology.orchestra.run.smithi116.stdout:1 2024-04-25T00:01:39.324 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:01:39.324 DEBUG:teuthology.orchestra.run.smithi116:> route 2024-04-25T00:01:39.455 INFO:teuthology.orchestra.run.smithi116.stdout:Kernel IP routing table 2024-04-25T00:01:39.456 INFO:teuthology.orchestra.run.smithi116.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T00:01:39.456 INFO:teuthology.orchestra.run.smithi116.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-04-25T00:01:39.456 INFO:teuthology.orchestra.run.smithi116.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T00:01:39.456 INFO:teuthology.orchestra.run.smithi116.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-04-25T00:01:39.456 INFO:teuthology.orchestra.run.smithi116.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-04-25T00:01:39.456 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T00:01:39.456 INFO:teuthology.orchestra.run.smithi116.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-25T00:01:39.456 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-25T00:01:39.456 INFO:teuthology.orchestra.run.smithi116.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-25T00:01:39.456 INFO:teuthology.orchestra.run.smithi116.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T00:01:39.457 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T00:01:39.457 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-04-25T00:01:39.457 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T00:01:39.457 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T00:01:39.457 DEBUG:teuthology.orchestra.run.smithi116:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T00:01:39.457 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-04-25T00:01:39.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:39 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T00:01:39.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T00:01:39.539 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:01:39.540 DEBUG:teuthology.orchestra.run.smithi116:> ip netns list 2024-04-25T00:01:39.546 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:01:39.546 DEBUG:teuthology.orchestra.run.smithi116:> ip netns list-id 2024-04-25T00:01:39.591 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T00:01:39.591 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-04-25T00:01:39.591 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T00:01:39.592 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-25T00:01:39.592 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-04-25T00:01:39.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:39 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T00:01:39.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T00:01:39.714 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-25T00:01:39.715 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T00:01:39.715 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-04-25T00:01:39.715 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-25T00:01:39.715 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-25T00:01:39.715 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-25T00:01:39.715 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-25T00:01:39.715 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-25T00:01:39.715 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-04-25T00:01:39.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:39 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T00:01:39.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T00:01:39.811 DEBUG:teuthology.orchestra.run.smithi116:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T00:01:39.811 DEBUG:teuthology.orchestra.run.smithi116:> set -e 2024-04-25T00:01:39.811 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link set brx.0 up 2024-04-25T00:01:39.811 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link set dev brx.0 master ceph-brx 2024-04-25T00:01:39.812 DEBUG:teuthology.orchestra.run.smithi116:> ') 2024-04-25T00:01:39.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:39 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T00:01:39.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T00:01:39.916 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {'cephfs_name': 'a'} 2024-04-25T00:01:39.917 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T00:01:39.917 DEBUG:teuthology.orchestra.run.smithi116:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-25T00:01:39.966 INFO:teuthology.orchestra.run.smithi116.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-25T00:01:39.967 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T00:01:39.967 DEBUG:teuthology.orchestra.run.smithi116:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-25T00:01:40.014 DEBUG:teuthology.orchestra.run.smithi116:> sudo modprobe fuse 2024-04-25T00:01:40.071 DEBUG:teuthology.orchestra.run.smithi116:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T00:01:40.121 INFO:teuthology.orchestra.run.smithi116.stdout:/sys 2024-04-25T00:01:40.173 INFO:teuthology.orchestra.run.smithi116.stdout:/proc 2024-04-25T00:01:40.173 INFO:teuthology.orchestra.run.smithi116.stdout:/dev 2024-04-25T00:01:40.173 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/pts 2024-04-25T00:01:40.173 INFO:teuthology.orchestra.run.smithi116.stdout:/run 2024-04-25T00:01:40.173 INFO:teuthology.orchestra.run.smithi116.stdout:/ 2024-04-25T00:01:40.173 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/security 2024-04-25T00:01:40.173 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/shm 2024-04-25T00:01:40.173 INFO:teuthology.orchestra.run.smithi116.stdout:/run/lock 2024-04-25T00:01:40.174 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/cgroup 2024-04-25T00:01:40.174 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/pstore 2024-04-25T00:01:40.174 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/bpf 2024-04-25T00:01:40.174 INFO:teuthology.orchestra.run.smithi116.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T00:01:40.174 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/hugepages 2024-04-25T00:01:40.174 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/mqueue 2024-04-25T00:01:40.174 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/debug 2024-04-25T00:01:40.174 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/tracing 2024-04-25T00:01:40.174 INFO:teuthology.orchestra.run.smithi116.stdout:/proc/fs/nfsd 2024-04-25T00:01:40.174 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/fuse/connections 2024-04-25T00:01:40.174 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/config 2024-04-25T00:01:40.174 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T00:01:40.175 INFO:teuthology.orchestra.run.smithi116.stdout:/run/rpc_pipefs 2024-04-25T00:01:40.175 INFO:teuthology.orchestra.run.smithi116.stdout:/run/user/1000 2024-04-25T00:01:40.175 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph 2024-04-25T00:01:40.175 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph/osd/ceph-0 2024-04-25T00:01:40.175 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph/osd/ceph-1 2024-04-25T00:01:40.175 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph/osd/ceph-2 2024-04-25T00:01:40.175 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph/osd/ceph-3 2024-04-25T00:01:40.175 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns 2024-04-25T00:01:40.175 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T00:01:40.175 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T00:01:40.176 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:01:40.177 DEBUG:teuthology.orchestra.run.smithi116:> ls /sys/fs/fuse/connections 2024-04-25T00:01:40.223 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-25T00:01:40.223 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0 --client_fs=a) 2024-04-25T00:01:40.268 DEBUG:teuthology.orchestra.run.smithi116:> sudo modprobe fuse 2024-04-25T00:01:40.283 DEBUG:teuthology.orchestra.run.smithi116:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T00:01:40.331 INFO:teuthology.orchestra.run.smithi116.stdout:/sys 2024-04-25T00:01:40.331 INFO:teuthology.orchestra.run.smithi116.stdout:/proc 2024-04-25T00:01:40.331 INFO:teuthology.orchestra.run.smithi116.stdout:/dev 2024-04-25T00:01:40.331 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/pts 2024-04-25T00:01:40.331 INFO:teuthology.orchestra.run.smithi116.stdout:/run 2024-04-25T00:01:40.331 INFO:teuthology.orchestra.run.smithi116.stdout:/ 2024-04-25T00:01:40.331 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/security 2024-04-25T00:01:40.332 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/shm 2024-04-25T00:01:40.332 INFO:teuthology.orchestra.run.smithi116.stdout:/run/lock 2024-04-25T00:01:40.332 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/cgroup 2024-04-25T00:01:40.332 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/pstore 2024-04-25T00:01:40.332 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/bpf 2024-04-25T00:01:40.332 INFO:teuthology.orchestra.run.smithi116.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T00:01:40.332 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/hugepages 2024-04-25T00:01:40.332 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/mqueue 2024-04-25T00:01:40.332 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/debug 2024-04-25T00:01:40.332 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/tracing 2024-04-25T00:01:40.332 INFO:teuthology.orchestra.run.smithi116.stdout:/proc/fs/nfsd 2024-04-25T00:01:40.332 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/fuse/connections 2024-04-25T00:01:40.332 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/config 2024-04-25T00:01:40.333 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T00:01:40.333 INFO:teuthology.orchestra.run.smithi116.stdout:/run/rpc_pipefs 2024-04-25T00:01:40.333 INFO:teuthology.orchestra.run.smithi116.stdout:/run/user/1000 2024-04-25T00:01:40.333 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph 2024-04-25T00:01:40.333 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph/osd/ceph-0 2024-04-25T00:01:40.333 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph/osd/ceph-1 2024-04-25T00:01:40.333 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph/osd/ceph-2 2024-04-25T00:01:40.333 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph/osd/ceph-3 2024-04-25T00:01:40.333 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns 2024-04-25T00:01:40.333 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T00:01:40.333 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T00:01:40.334 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:01:40.334 DEBUG:teuthology.orchestra.run.smithi116:> ls /sys/fs/fuse/connections 2024-04-25T00:01:40.349 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi116.stderr:2024-04-25T00:01:40.345+0000 7f80f0350580 -1 init, newargv = 0x55fdfef7dda0 newargc=15 2024-04-25T00:01:40.350 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi116.stderr:ceph-fuse[36045]: starting ceph client 2024-04-25T00:01:40.356 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi116.stderr:ceph-fuse[36045]: starting fuse 2024-04-25T00:01:41.454 DEBUG:teuthology.orchestra.run.smithi116:> sudo modprobe fuse 2024-04-25T00:01:41.470 DEBUG:teuthology.orchestra.run.smithi116:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T00:01:41.520 INFO:teuthology.orchestra.run.smithi116.stdout:/sys 2024-04-25T00:01:41.520 INFO:teuthology.orchestra.run.smithi116.stdout:/proc 2024-04-25T00:01:41.520 INFO:teuthology.orchestra.run.smithi116.stdout:/dev 2024-04-25T00:01:41.520 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/pts 2024-04-25T00:01:41.520 INFO:teuthology.orchestra.run.smithi116.stdout:/run 2024-04-25T00:01:41.521 INFO:teuthology.orchestra.run.smithi116.stdout:/ 2024-04-25T00:01:41.521 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/security 2024-04-25T00:01:41.521 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/shm 2024-04-25T00:01:41.521 INFO:teuthology.orchestra.run.smithi116.stdout:/run/lock 2024-04-25T00:01:41.521 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/cgroup 2024-04-25T00:01:41.521 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/pstore 2024-04-25T00:01:41.521 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/bpf 2024-04-25T00:01:41.522 INFO:teuthology.orchestra.run.smithi116.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T00:01:41.522 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/hugepages 2024-04-25T00:01:41.522 INFO:teuthology.orchestra.run.smithi116.stdout:/dev/mqueue 2024-04-25T00:01:41.522 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/debug 2024-04-25T00:01:41.522 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/tracing 2024-04-25T00:01:41.522 INFO:teuthology.orchestra.run.smithi116.stdout:/proc/fs/nfsd 2024-04-25T00:01:41.522 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/fs/fuse/connections 2024-04-25T00:01:41.522 INFO:teuthology.orchestra.run.smithi116.stdout:/sys/kernel/config 2024-04-25T00:01:41.522 INFO:teuthology.orchestra.run.smithi116.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T00:01:41.522 INFO:teuthology.orchestra.run.smithi116.stdout:/run/rpc_pipefs 2024-04-25T00:01:41.522 INFO:teuthology.orchestra.run.smithi116.stdout:/run/user/1000 2024-04-25T00:01:41.522 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph 2024-04-25T00:01:41.523 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph/osd/ceph-0 2024-04-25T00:01:41.523 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph/osd/ceph-1 2024-04-25T00:01:41.523 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph/osd/ceph-2 2024-04-25T00:01:41.523 INFO:teuthology.orchestra.run.smithi116.stdout:/var/lib/ceph/osd/ceph-3 2024-04-25T00:01:41.523 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns 2024-04-25T00:01:41.523 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T00:01:41.523 INFO:teuthology.orchestra.run.smithi116.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T00:01:41.523 INFO:teuthology.orchestra.run.smithi116.stdout:/home/ubuntu/cephtest/mnt.0 2024-04-25T00:01:41.524 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:01:41.524 DEBUG:teuthology.orchestra.run.smithi116:> ls /sys/fs/fuse/connections 2024-04-25T00:01:41.571 INFO:teuthology.orchestra.run.smithi116.stdout:45 2024-04-25T00:01:41.571 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2024-04-25T00:01:41.572 DEBUG:teuthology.orchestra.run.smithi116:> sudo stdin-killer -- python3 -c ' 2024-04-25T00:01:41.572 DEBUG:teuthology.orchestra.run.smithi116:> import glob 2024-04-25T00:01:41.572 DEBUG:teuthology.orchestra.run.smithi116:> import re 2024-04-25T00:01:41.572 DEBUG:teuthology.orchestra.run.smithi116:> import os 2024-04-25T00:01:41.572 DEBUG:teuthology.orchestra.run.smithi116:> import subprocess 2024-04-25T00:01:41.572 DEBUG:teuthology.orchestra.run.smithi116:> 2024-04-25T00:01:41.572 DEBUG:teuthology.orchestra.run.smithi116:> def _find_admin_socket(client_name): 2024-04-25T00:01:41.572 DEBUG:teuthology.orchestra.run.smithi116:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-25T00:01:41.572 DEBUG:teuthology.orchestra.run.smithi116:> files = glob.glob(asok_path) 2024-04-25T00:01:41.572 DEBUG:teuthology.orchestra.run.smithi116:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-25T00:01:41.572 DEBUG:teuthology.orchestra.run.smithi116:> 2024-04-25T00:01:41.572 DEBUG:teuthology.orchestra.run.smithi116:> # Given a non-glob path, it better be there 2024-04-25T00:01:41.572 DEBUG:teuthology.orchestra.run.smithi116:> if "*" not in asok_path: 2024-04-25T00:01:41.572 DEBUG:teuthology.orchestra.run.smithi116:> assert(len(files) == 1) 2024-04-25T00:01:41.573 DEBUG:teuthology.orchestra.run.smithi116:> return files[0] 2024-04-25T00:01:41.573 DEBUG:teuthology.orchestra.run.smithi116:> 2024-04-25T00:01:41.573 DEBUG:teuthology.orchestra.run.smithi116:> for f in files: 2024-04-25T00:01:41.573 DEBUG:teuthology.orchestra.run.smithi116:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-25T00:01:41.573 DEBUG:teuthology.orchestra.run.smithi116:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-25T00:01:41.573 DEBUG:teuthology.orchestra.run.smithi116:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-25T00:01:41.573 DEBUG:teuthology.orchestra.run.smithi116:> contents = proc_f.read() 2024-04-25T00:01:41.573 DEBUG:teuthology.orchestra.run.smithi116:> if mountpoint in contents: 2024-04-25T00:01:41.573 DEBUG:teuthology.orchestra.run.smithi116:> return f 2024-04-25T00:01:41.573 DEBUG:teuthology.orchestra.run.smithi116:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-25T00:01:41.573 DEBUG:teuthology.orchestra.run.smithi116:> 2024-04-25T00:01:41.573 DEBUG:teuthology.orchestra.run.smithi116:> print(_find_admin_socket("client.0")) 2024-04-25T00:01:41.573 DEBUG:teuthology.orchestra.run.smithi116:> ' 2024-04-25T00:01:41.698 INFO:teuthology.orchestra.run.smithi116.stdout:/var/run/ceph/ceph-client.0.36045.asok 2024-04-25T00:01:41.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T00:01:41.709 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.36045.asok 2024-04-25T00:01:41.709 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:01:41.709 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.36045.asok status 2024-04-25T00:01:41.859 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-25T00:01:41.859 INFO:teuthology.orchestra.run.smithi116.stdout: "metadata": { 2024-04-25T00:01:41.859 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_sha1": "d2c45d977142e024a59b68a8f1dce8f0b9840551", 2024-04-25T00:01:41.859 INFO:teuthology.orchestra.run.smithi116.stdout: "ceph_version": "ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev)", 2024-04-25T00:01:41.859 INFO:teuthology.orchestra.run.smithi116.stdout: "entity_id": "0", 2024-04-25T00:01:41.859 INFO:teuthology.orchestra.run.smithi116.stdout: "hostname": "smithi116", 2024-04-25T00:01:41.859 INFO:teuthology.orchestra.run.smithi116.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-25T00:01:41.859 INFO:teuthology.orchestra.run.smithi116.stdout: "pid": "36045", 2024-04-25T00:01:41.860 INFO:teuthology.orchestra.run.smithi116.stdout: "root": "/" 2024-04-25T00:01:41.860 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-25T00:01:41.860 INFO:teuthology.orchestra.run.smithi116.stdout: "dentry_count": 0, 2024-04-25T00:01:41.860 INFO:teuthology.orchestra.run.smithi116.stdout: "dentry_pinned_count": 0, 2024-04-25T00:01:41.860 INFO:teuthology.orchestra.run.smithi116.stdout: "id": 4959, 2024-04-25T00:01:41.860 INFO:teuthology.orchestra.run.smithi116.stdout: "inst": { 2024-04-25T00:01:41.860 INFO:teuthology.orchestra.run.smithi116.stdout: "name": { 2024-04-25T00:01:41.860 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "client", 2024-04-25T00:01:41.860 INFO:teuthology.orchestra.run.smithi116.stdout: "num": 4959 2024-04-25T00:01:41.860 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-25T00:01:41.860 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": { 2024-04-25T00:01:41.860 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-04-25T00:01:41.860 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "192.168.0.1:0", 2024-04-25T00:01:41.861 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 1255704380 2024-04-25T00:01:41.861 INFO:teuthology.orchestra.run.smithi116.stdout: } 2024-04-25T00:01:41.861 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-25T00:01:41.861 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": { 2024-04-25T00:01:41.861 INFO:teuthology.orchestra.run.smithi116.stdout: "type": "v1", 2024-04-25T00:01:41.861 INFO:teuthology.orchestra.run.smithi116.stdout: "addr": "192.168.0.1:0", 2024-04-25T00:01:41.861 INFO:teuthology.orchestra.run.smithi116.stdout: "nonce": 1255704380 2024-04-25T00:01:41.861 INFO:teuthology.orchestra.run.smithi116.stdout: }, 2024-04-25T00:01:41.861 INFO:teuthology.orchestra.run.smithi116.stdout: "inst_str": "client.4959 192.168.0.1:0/1255704380", 2024-04-25T00:01:41.861 INFO:teuthology.orchestra.run.smithi116.stdout: "addr_str": "192.168.0.1:0/1255704380", 2024-04-25T00:01:41.861 INFO:teuthology.orchestra.run.smithi116.stdout: "inode_count": 1, 2024-04-25T00:01:41.861 INFO:teuthology.orchestra.run.smithi116.stdout: "mds_epoch": 9, 2024-04-25T00:01:41.861 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_epoch": 27, 2024-04-25T00:01:41.861 INFO:teuthology.orchestra.run.smithi116.stdout: "osd_epoch_barrier": 0, 2024-04-25T00:01:41.862 INFO:teuthology.orchestra.run.smithi116.stdout: "blocklisted": false, 2024-04-25T00:01:41.862 INFO:teuthology.orchestra.run.smithi116.stdout: "fs_name": "a" 2024-04-25T00:01:41.862 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-25T00:01:41.872 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T00:01:41.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:41.989+0000 7f69c3b4a640 1 Processor -- start 2024-04-25T00:01:41.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:41.989+0000 7f69c3b4a640 1 -- start start 2024-04-25T00:01:41.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:41.993+0000 7f69c3b4a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f69bc0a94a0 0x7f69bc1538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:41.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:41.993+0000 7f69c3b4a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f69bc0a99f0 0x7f69bc153e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:41.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:41.993+0000 7f69c3b4a640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69bc0c3910 con 0x7f69bc150d20 2024-04-25T00:01:41.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:41.993+0000 7f69c3b4a640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f69bc0c16e0 con 0x7f69bc0a99f0 2024-04-25T00:01:41.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:41.993+0000 7f69c3b4a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f69bc0c1400 con 0x7f69bc0a94a0 2024-04-25T00:01:41.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:41.993+0000 7f69c20e7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f69bc0a99f0 0x7f69bc153e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:41.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:41.993+0000 7f69c18e6640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f69bc150d20 0x7f69bc0c2a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:55386/0 (socket says 172.21.15.116:55386) 2024-04-25T00:01:41.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:41.993+0000 7f69c20e7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f69bc0a99f0 0x7f69bc153e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43110/0 (socket says 172.21.15.116:43110) 2024-04-25T00:01:41.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:41.993+0000 7f69c18e6640 1 -- 172.21.15.116:0/3684229805 learned_addr learned my addr 172.21.15.116:0/3684229805 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:41.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:41.993+0000 7f69c10e5640 1 --2- 172.21.15.116:0/3684229805 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f69bc0a94a0 0x7f69bc1538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:41.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:41.993+0000 7f69c20e7640 1 -- 172.21.15.116:0/3684229805 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f69bc0a94a0 msgr2=0x7f69bc1538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:41.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:41.993+0000 7f69c20e7640 1 --2- 172.21.15.116:0/3684229805 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f69bc0a94a0 0x7f69bc1538e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:41.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:41.993+0000 7f69c20e7640 1 -- 172.21.15.116:0/3684229805 >> v1:172.21.15.116:6789/0 conn(0x7f69bc150d20 legacy=0x7f69bc0c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:41.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:41.993+0000 7f69c20e7640 1 -- 172.21.15.116:0/3684229805 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69bc06e660 con 0x7f69bc0a99f0 2024-04-25T00:01:41.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:41.993+0000 7f69c20e7640 1 --2- 172.21.15.116:0/3684229805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69bc0a99f0 0x7f69bc153e20 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f69ac0097c0 tx=0x7f69ac01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dd21c9f83633b07 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:41.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:41.993+0000 7f69c08e4640 1 -- 172.21.15.116:0/3684229805 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69ac0b0070 con 0x7f69bc0a99f0 2024-04-25T00:01:41.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:41.993+0000 7f69c08e4640 1 -- 172.21.15.116:0/3684229805 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69ac0ab600 con 0x7f69bc0a99f0 2024-04-25T00:01:41.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:41.993+0000 7f69c08e4640 1 -- 172.21.15.116:0/3684229805 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69ac0ab9d0 con 0x7f69bc0a99f0 2024-04-25T00:01:41.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:41.997+0000 7f69c3b4a640 1 -- 172.21.15.116:0/3684229805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69bc0a99f0 msgr2=0x7f69bc153e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:41.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:41.997+0000 7f69c3b4a640 1 --2- 172.21.15.116:0/3684229805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69bc0a99f0 0x7f69bc153e20 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f69ac0097c0 tx=0x7f69ac01c4a0 comp rx=0 tx=0).stop 2024-04-25T00:01:41.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:41.997+0000 7f69c3b4a640 1 -- 172.21.15.116:0/3684229805 shutdown_connections 2024-04-25T00:01:41.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:41.997+0000 7f69c3b4a640 1 --2- 172.21.15.116:0/3684229805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69bc0a99f0 0x7f69bc153e20 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:41.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:41.997+0000 7f69c3b4a640 1 --2- 172.21.15.116:0/3684229805 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f69bc0a94a0 0x7f69bc1538e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:41.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:41.997+0000 7f69c3b4a640 1 -- 172.21.15.116:0/3684229805 >> 172.21.15.116:0/3684229805 conn(0x7f69bc0c3b00 msgr2=0x7f69bc0c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:41.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:41.997+0000 7f69c3b4a640 1 -- 172.21.15.116:0/3684229805 shutdown_connections 2024-04-25T00:01:41.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:41.997+0000 7f69c3b4a640 1 -- 172.21.15.116:0/3684229805 wait complete. 2024-04-25T00:01:42.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:41.997+0000 7f69c3b4a640 1 Processor -- start 2024-04-25T00:01:42.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:41.997+0000 7f69c3b4a640 1 -- start start 2024-04-25T00:01:42.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:41.997+0000 7f69c3b4a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f69bc0a94a0 0x7f69bc0c60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:42.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:41.997+0000 7f69c3b4a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69bc0a99f0 0x7f69bc13b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:42.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:41.997+0000 7f69c3b4a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69bc150d20 0x7f69bc13bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:42.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:41.997+0000 7f69c3b4a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f69bc1544e0 con 0x7f69bc150d20 2024-04-25T00:01:42.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:41.997+0000 7f69c3b4a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f69bc154360 con 0x7f69bc0a99f0 2024-04-25T00:01:42.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:41.997+0000 7f69c3b4a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f69bc160730 con 0x7f69bc0a94a0 2024-04-25T00:01:42.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:41.997+0000 7f69c20e7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69bc150d20 0x7f69bc13bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:42.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:41.997+0000 7f69c20e7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69bc150d20 0x7f69bc13bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43122/0 (socket says 172.21.15.116:43122) 2024-04-25T00:01:42.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:41.997+0000 7f69c20e7640 1 -- 172.21.15.116:0/3250194194 learned_addr learned my addr 172.21.15.116:0/3250194194 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:42.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:41.997+0000 7f69c20e7640 1 -- 172.21.15.116:0/3250194194 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f69bc0a94a0 msgr2=0x7f69bc0c60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:42.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:41.997+0000 7f69c18e6640 1 --2- 172.21.15.116:0/3250194194 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f69bc0a94a0 0x7f69bc0c60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:42.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:41.997+0000 7f69c10e5640 1 --2- 172.21.15.116:0/3250194194 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69bc0a99f0 0x7f69bc13b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:42.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:41.997+0000 7f69c20e7640 1 --2- 172.21.15.116:0/3250194194 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f69bc0a94a0 0x7f69bc0c60b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:42.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:41.997+0000 7f69c20e7640 1 -- 172.21.15.116:0/3250194194 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69bc0a99f0 msgr2=0x7f69bc13b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:42.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:41.997+0000 7f69c20e7640 1 --2- 172.21.15.116:0/3250194194 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69bc0a99f0 0x7f69bc13b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:42.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:41.997+0000 7f69c20e7640 1 -- 172.21.15.116:0/3250194194 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69bc1ece50 con 0x7f69bc150d20 2024-04-25T00:01:42.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:41.997+0000 7f69c10e5640 1 --2- 172.21.15.116:0/3250194194 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69bc0a99f0 0x7f69bc13b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:42.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:41.997+0000 7f69c20e7640 1 --2- 172.21.15.116:0/3250194194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69bc150d20 0x7f69bc13bcc0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f69ac0097c0 tx=0x7f69ac019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:42.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:41.997+0000 7f69ba7fc640 1 -- 172.21.15.116:0/3250194194 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69ac0b0040 con 0x7f69bc150d20 2024-04-25T00:01:42.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:41.997+0000 7f69ba7fc640 1 -- 172.21.15.116:0/3250194194 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69ac0b9070 con 0x7f69bc150d20 2024-04-25T00:01:42.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:41.997+0000 7f69c3b4a640 1 -- 172.21.15.116:0/3250194194 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69bc0c69a0 con 0x7f69bc150d20 2024-04-25T00:01:42.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:41.997+0000 7f69ba7fc640 1 -- 172.21.15.116:0/3250194194 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69ac0b4630 con 0x7f69bc150d20 2024-04-25T00:01:42.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:41.997+0000 7f69c3b4a640 1 -- 172.21.15.116:0/3250194194 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69bc0c6c40 con 0x7f69bc150d20 2024-04-25T00:01:42.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:42.001+0000 7f69c3b4a640 1 -- 172.21.15.116:0/3250194194 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69bc068350 con 0x7f69bc150d20 2024-04-25T00:01:42.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:42.001+0000 7f69ba7fc640 1 -- 172.21.15.116:0/3250194194 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f69ac0b47d0 con 0x7f69bc150d20 2024-04-25T00:01:42.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:42.001+0000 7f69ba7fc640 1 --2- 172.21.15.116:0/3250194194 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f69a80678f0 0x7f69a8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:42.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:42.001+0000 7f69ba7fc640 1 -- 172.21.15.116:0/3250194194 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7f69ac11fd60 con 0x7f69bc150d20 2024-04-25T00:01:42.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:42.001+0000 7f69c18e6640 1 --2- 172.21.15.116:0/3250194194 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f69a80678f0 0x7f69a8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:42.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:42.001+0000 7f69c18e6640 1 --2- 172.21.15.116:0/3250194194 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f69a80678f0 0x7f69a8069db0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f69bc1447a0 tx=0x7f69b4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:42.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:42.005+0000 7f69ba7fc640 1 -- 172.21.15.116:0/3250194194 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f69ac016090 con 0x7f69bc150d20 2024-04-25T00:01:42.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:42.201+0000 7f69c3b4a640 1 -- 172.21.15.116:0/3250194194 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f69bc0c34f0 con 0x7f69bc150d20 2024-04-25T00:01:42.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:42.201+0000 7f69ba7fc640 1 -- 172.21.15.116:0/3250194194 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v14) ==== 53+0+116 (secure 0 0 0) 0x7f69ac0f0600 con 0x7f69bc150d20 2024-04-25T00:01:42.205 INFO:teuthology.orchestra.run.smithi116.stdout:name: a, metadata pool: a_metadata, data pools: [a_data ] 2024-04-25T00:01:42.205 INFO:teuthology.orchestra.run.smithi116.stdout:name: b, metadata pool: b_metadata, data pools: [b_data ] 2024-04-25T00:01:42.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:42.205+0000 7f69c3b4a640 1 -- 172.21.15.116:0/3250194194 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f69a80678f0 msgr2=0x7f69a8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:42.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:42.205+0000 7f69c3b4a640 1 --2- 172.21.15.116:0/3250194194 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f69a80678f0 0x7f69a8069db0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f69bc1447a0 tx=0x7f69b4009040 comp rx=0 tx=0).stop 2024-04-25T00:01:42.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:42.205+0000 7f69c3b4a640 1 -- 172.21.15.116:0/3250194194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69bc150d20 msgr2=0x7f69bc13bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:42.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:42.205+0000 7f69c3b4a640 1 --2- 172.21.15.116:0/3250194194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69bc150d20 0x7f69bc13bcc0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f69ac0097c0 tx=0x7f69ac019240 comp rx=0 tx=0).stop 2024-04-25T00:01:42.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:42.205+0000 7f69c3b4a640 1 -- 172.21.15.116:0/3250194194 shutdown_connections 2024-04-25T00:01:42.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:42.205+0000 7f69c3b4a640 1 --2- 172.21.15.116:0/3250194194 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f69a80678f0 0x7f69a8069db0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:42.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:42.205+0000 7f69c3b4a640 1 --2- 172.21.15.116:0/3250194194 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69bc150d20 0x7f69bc13bcc0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:42.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:42.205+0000 7f69c3b4a640 1 --2- 172.21.15.116:0/3250194194 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69bc0a99f0 0x7f69bc13b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:42.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:42.205+0000 7f69c3b4a640 1 --2- 172.21.15.116:0/3250194194 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f69bc0a94a0 0x7f69bc0c60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:42.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:42.205+0000 7f69c3b4a640 1 -- 172.21.15.116:0/3250194194 >> 172.21.15.116:0/3250194194 conn(0x7f69bc0c3b00 msgr2=0x7f69bc153730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:42.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:42.205+0000 7f69c3b4a640 1 -- 172.21.15.116:0/3250194194 shutdown_connections 2024-04-25T00:01:42.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:42.205+0000 7f69c3b4a640 1 -- 172.21.15.116:0/3250194194 wait complete. 2024-04-25T00:01:42.224 INFO:tasks.cephfs.mount:Mounting Ceph FS b; just confirmed its presence on cluster 2024-04-25T00:01:42.224 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T00:01:42.224 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi174.front.sepia.ceph.com 2024-04-25T00:01:42.224 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-25T00:01:42.224 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-25T00:01:42.224 INFO:tasks.cephfs.mount:self.cephfs_name = b 2024-04-25T00:01:42.224 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-25T00:01:42.224 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T00:01:42.225 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-25T00:01:42.225 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:01:42.225 DEBUG:teuthology.orchestra.run.smithi174:> ip addr 2024-04-25T00:01:42.232 INFO:teuthology.orchestra.run.smithi174.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T00:01:42.232 INFO:teuthology.orchestra.run.smithi174.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T00:01:42.232 INFO:teuthology.orchestra.run.smithi174.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T00:01:42.232 INFO:teuthology.orchestra.run.smithi174.stdout: valid_lft forever preferred_lft forever 2024-04-25T00:01:42.233 INFO:teuthology.orchestra.run.smithi174.stdout: inet6 ::1/128 scope host 2024-04-25T00:01:42.233 INFO:teuthology.orchestra.run.smithi174.stdout: valid_lft forever preferred_lft forever 2024-04-25T00:01:42.233 INFO:teuthology.orchestra.run.smithi174.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-25T00:01:42.233 INFO:teuthology.orchestra.run.smithi174.stdout: link/ether 0c:c4:7a:d9:36:e0 brd ff:ff:ff:ff:ff:ff 2024-04-25T00:01:42.233 INFO:teuthology.orchestra.run.smithi174.stdout: altname enp7s0f0 2024-04-25T00:01:42.233 INFO:teuthology.orchestra.run.smithi174.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-25T00:01:42.233 INFO:teuthology.orchestra.run.smithi174.stdout: link/ether 0c:c4:7a:d9:36:e1 brd ff:ff:ff:ff:ff:ff 2024-04-25T00:01:42.233 INFO:teuthology.orchestra.run.smithi174.stdout: altname enp7s0f1 2024-04-25T00:01:42.233 INFO:teuthology.orchestra.run.smithi174.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-04-25T00:01:42.233 INFO:teuthology.orchestra.run.smithi174.stdout: link/ether 0c:c4:7a:88:77:1a brd ff:ff:ff:ff:ff:ff 2024-04-25T00:01:42.233 INFO:teuthology.orchestra.run.smithi174.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T00:01:42.233 INFO:teuthology.orchestra.run.smithi174.stdout: link/ether 0c:c4:7a:88:77:1b brd ff:ff:ff:ff:ff:ff 2024-04-25T00:01:42.233 INFO:teuthology.orchestra.run.smithi174.stdout: inet 172.21.15.174/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-04-25T00:01:42.233 INFO:teuthology.orchestra.run.smithi174.stdout: valid_lft 42467sec preferred_lft 42467sec 2024-04-25T00:01:42.234 INFO:teuthology.orchestra.run.smithi174.stdout: inet6 fe80::ec4:7aff:fe88:771b/64 scope link 2024-04-25T00:01:42.234 INFO:teuthology.orchestra.run.smithi174.stdout: valid_lft forever preferred_lft forever 2024-04-25T00:01:42.234 INFO:teuthology.orchestra.run.smithi174.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-04-25T00:01:42.234 INFO:teuthology.orchestra.run.smithi174.stdout: link/ether 02:42:9a:5d:39:a8 brd ff:ff:ff:ff:ff:ff 2024-04-25T00:01:42.234 INFO:teuthology.orchestra.run.smithi174.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-04-25T00:01:42.234 INFO:teuthology.orchestra.run.smithi174.stdout: valid_lft forever preferred_lft forever 2024-04-25T00:01:42.234 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-25T00:01:42.234 DEBUG:teuthology.orchestra.run.smithi174:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T00:01:42.234 DEBUG:teuthology.orchestra.run.smithi174:> set -e 2024-04-25T00:01:42.234 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip link add name ceph-brx type bridge 2024-04-25T00:01:42.235 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip addr flush dev ceph-brx 2024-04-25T00:01:42.235 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip link set ceph-brx up 2024-04-25T00:01:42.235 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-25T00:01:42.235 DEBUG:teuthology.orchestra.run.smithi174:> ') 2024-04-25T00:01:42.325 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:01:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T00:01:42.360 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:01:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T00:01:42.367 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:01:42.367 DEBUG:teuthology.orchestra.run.smithi174:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T00:01:42.378 INFO:teuthology.orchestra.run.smithi174.stdout:1 2024-04-25T00:01:42.379 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:01:42.379 DEBUG:teuthology.orchestra.run.smithi174:> route 2024-04-25T00:01:42.523 INFO:teuthology.orchestra.run.smithi174.stdout:Kernel IP routing table 2024-04-25T00:01:42.523 INFO:teuthology.orchestra.run.smithi174.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T00:01:42.523 INFO:teuthology.orchestra.run.smithi174.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-04-25T00:01:42.523 INFO:teuthology.orchestra.run.smithi174.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T00:01:42.524 INFO:teuthology.orchestra.run.smithi174.stdout:158.69.67.47 _gateway 255.255.255.255 UGH 100 0 0 enp3s0f1 2024-04-25T00:01:42.524 INFO:teuthology.orchestra.run.smithi174.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-04-25T00:01:42.524 INFO:teuthology.orchestra.run.smithi174.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T00:01:42.524 INFO:teuthology.orchestra.run.smithi174.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-25T00:01:42.524 INFO:teuthology.orchestra.run.smithi174.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-25T00:01:42.524 INFO:teuthology.orchestra.run.smithi174.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-04-25T00:01:42.524 INFO:teuthology.orchestra.run.smithi174.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T00:01:42.524 DEBUG:teuthology.orchestra.run.smithi174:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T00:01:42.524 DEBUG:teuthology.orchestra.run.smithi174:> set -e 2024-04-25T00:01:42.524 DEBUG:teuthology.orchestra.run.smithi174:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T00:01:42.524 DEBUG:teuthology.orchestra.run.smithi174:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T00:01:42.525 DEBUG:teuthology.orchestra.run.smithi174:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T00:01:42.525 DEBUG:teuthology.orchestra.run.smithi174:> ') 2024-04-25T00:01:42.573 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:01:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T00:01:42.600 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:01:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T00:01:42.607 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:01:42.608 DEBUG:teuthology.orchestra.run.smithi174:> ip netns list 2024-04-25T00:01:42.614 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:01:42.614 DEBUG:teuthology.orchestra.run.smithi174:> ip netns list-id 2024-04-25T00:01:42.662 DEBUG:teuthology.orchestra.run.smithi174:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T00:01:42.663 DEBUG:teuthology.orchestra.run.smithi174:> set -e 2024-04-25T00:01:42.663 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T00:01:42.663 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-04-25T00:01:42.663 DEBUG:teuthology.orchestra.run.smithi174:> ') 2024-04-25T00:01:42.754 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:01:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T00:01:42.771 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:01:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T00:01:42.778 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-04-25T00:01:42.778 DEBUG:teuthology.orchestra.run.smithi174:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T00:01:42.778 DEBUG:teuthology.orchestra.run.smithi174:> set -e 2024-04-25T00:01:42.778 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-04-25T00:01:42.778 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-25T00:01:42.778 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-04-25T00:01:42.778 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-04-25T00:01:42.779 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-04-25T00:01:42.779 DEBUG:teuthology.orchestra.run.smithi174:> ') 2024-04-25T00:01:42.821 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:01:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T00:01:42.886 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:01:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T00:01:42.893 DEBUG:teuthology.orchestra.run.smithi174:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T00:01:42.893 DEBUG:teuthology.orchestra.run.smithi174:> set -e 2024-04-25T00:01:42.893 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip link set brx.0 up 2024-04-25T00:01:42.893 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip link set dev brx.0 master ceph-brx 2024-04-25T00:01:42.893 DEBUG:teuthology.orchestra.run.smithi174:> ') 2024-04-25T00:01:42.976 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:01:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T00:01:42.991 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:01:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T00:01:42.998 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {'cephfs_name': 'b'} 2024-04-25T00:01:42.998 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T00:01:42.998 DEBUG:teuthology.orchestra.run.smithi174:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-25T00:01:43.045 INFO:teuthology.orchestra.run.smithi174.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-25T00:01:43.045 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T00:01:43.045 DEBUG:teuthology.orchestra.run.smithi174:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-25T00:01:43.093 DEBUG:teuthology.orchestra.run.smithi174:> sudo modprobe fuse 2024-04-25T00:01:43.152 DEBUG:teuthology.orchestra.run.smithi174:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T00:01:43.199 INFO:teuthology.orchestra.run.smithi174.stdout:/sys 2024-04-25T00:01:43.200 INFO:teuthology.orchestra.run.smithi174.stdout:/proc 2024-04-25T00:01:43.200 INFO:teuthology.orchestra.run.smithi174.stdout:/dev 2024-04-25T00:01:43.200 INFO:teuthology.orchestra.run.smithi174.stdout:/dev/pts 2024-04-25T00:01:43.200 INFO:teuthology.orchestra.run.smithi174.stdout:/run 2024-04-25T00:01:43.200 INFO:teuthology.orchestra.run.smithi174.stdout:/ 2024-04-25T00:01:43.200 INFO:teuthology.orchestra.run.smithi174.stdout:/sys/kernel/security 2024-04-25T00:01:43.200 INFO:teuthology.orchestra.run.smithi174.stdout:/dev/shm 2024-04-25T00:01:43.200 INFO:teuthology.orchestra.run.smithi174.stdout:/run/lock 2024-04-25T00:01:43.200 INFO:teuthology.orchestra.run.smithi174.stdout:/sys/fs/cgroup 2024-04-25T00:01:43.201 INFO:teuthology.orchestra.run.smithi174.stdout:/sys/fs/pstore 2024-04-25T00:01:43.201 INFO:teuthology.orchestra.run.smithi174.stdout:/sys/fs/bpf 2024-04-25T00:01:43.201 INFO:teuthology.orchestra.run.smithi174.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T00:01:43.201 INFO:teuthology.orchestra.run.smithi174.stdout:/dev/hugepages 2024-04-25T00:01:43.201 INFO:teuthology.orchestra.run.smithi174.stdout:/dev/mqueue 2024-04-25T00:01:43.201 INFO:teuthology.orchestra.run.smithi174.stdout:/sys/kernel/debug 2024-04-25T00:01:43.201 INFO:teuthology.orchestra.run.smithi174.stdout:/sys/kernel/tracing 2024-04-25T00:01:43.201 INFO:teuthology.orchestra.run.smithi174.stdout:/proc/fs/nfsd 2024-04-25T00:01:43.201 INFO:teuthology.orchestra.run.smithi174.stdout:/sys/fs/fuse/connections 2024-04-25T00:01:43.201 INFO:teuthology.orchestra.run.smithi174.stdout:/sys/kernel/config 2024-04-25T00:01:43.201 INFO:teuthology.orchestra.run.smithi174.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T00:01:43.201 INFO:teuthology.orchestra.run.smithi174.stdout:/run/rpc_pipefs 2024-04-25T00:01:43.201 INFO:teuthology.orchestra.run.smithi174.stdout:/run/user/1000 2024-04-25T00:01:43.201 INFO:teuthology.orchestra.run.smithi174.stdout:/var/lib/ceph 2024-04-25T00:01:43.201 INFO:teuthology.orchestra.run.smithi174.stdout:/var/lib/ceph/osd/ceph-4 2024-04-25T00:01:43.202 INFO:teuthology.orchestra.run.smithi174.stdout:/var/lib/ceph/osd/ceph-5 2024-04-25T00:01:43.202 INFO:teuthology.orchestra.run.smithi174.stdout:/var/lib/ceph/osd/ceph-6 2024-04-25T00:01:43.202 INFO:teuthology.orchestra.run.smithi174.stdout:/var/lib/ceph/osd/ceph-7 2024-04-25T00:01:43.202 INFO:teuthology.orchestra.run.smithi174.stdout:/run/netns 2024-04-25T00:01:43.202 INFO:teuthology.orchestra.run.smithi174.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T00:01:43.202 INFO:teuthology.orchestra.run.smithi174.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T00:01:43.203 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:01:43.203 DEBUG:teuthology.orchestra.run.smithi174:> ls /sys/fs/fuse/connections 2024-04-25T00:01:43.249 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-25T00:01:43.249 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1 --client_fs=b) 2024-04-25T00:01:43.294 DEBUG:teuthology.orchestra.run.smithi174:> sudo modprobe fuse 2024-04-25T00:01:43.308 DEBUG:teuthology.orchestra.run.smithi174:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T00:01:43.354 INFO:teuthology.orchestra.run.smithi174.stdout:/sys 2024-04-25T00:01:43.354 INFO:teuthology.orchestra.run.smithi174.stdout:/proc 2024-04-25T00:01:43.354 INFO:teuthology.orchestra.run.smithi174.stdout:/dev 2024-04-25T00:01:43.354 INFO:teuthology.orchestra.run.smithi174.stdout:/dev/pts 2024-04-25T00:01:43.354 INFO:teuthology.orchestra.run.smithi174.stdout:/run 2024-04-25T00:01:43.354 INFO:teuthology.orchestra.run.smithi174.stdout:/ 2024-04-25T00:01:43.354 INFO:teuthology.orchestra.run.smithi174.stdout:/sys/kernel/security 2024-04-25T00:01:43.354 INFO:teuthology.orchestra.run.smithi174.stdout:/dev/shm 2024-04-25T00:01:43.354 INFO:teuthology.orchestra.run.smithi174.stdout:/run/lock 2024-04-25T00:01:43.355 INFO:teuthology.orchestra.run.smithi174.stdout:/sys/fs/cgroup 2024-04-25T00:01:43.355 INFO:teuthology.orchestra.run.smithi174.stdout:/sys/fs/pstore 2024-04-25T00:01:43.355 INFO:teuthology.orchestra.run.smithi174.stdout:/sys/fs/bpf 2024-04-25T00:01:43.355 INFO:teuthology.orchestra.run.smithi174.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T00:01:43.355 INFO:teuthology.orchestra.run.smithi174.stdout:/dev/hugepages 2024-04-25T00:01:43.355 INFO:teuthology.orchestra.run.smithi174.stdout:/dev/mqueue 2024-04-25T00:01:43.355 INFO:teuthology.orchestra.run.smithi174.stdout:/sys/kernel/debug 2024-04-25T00:01:43.355 INFO:teuthology.orchestra.run.smithi174.stdout:/sys/kernel/tracing 2024-04-25T00:01:43.355 INFO:teuthology.orchestra.run.smithi174.stdout:/proc/fs/nfsd 2024-04-25T00:01:43.355 INFO:teuthology.orchestra.run.smithi174.stdout:/sys/fs/fuse/connections 2024-04-25T00:01:43.355 INFO:teuthology.orchestra.run.smithi174.stdout:/sys/kernel/config 2024-04-25T00:01:43.355 INFO:teuthology.orchestra.run.smithi174.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T00:01:43.355 INFO:teuthology.orchestra.run.smithi174.stdout:/run/rpc_pipefs 2024-04-25T00:01:43.356 INFO:teuthology.orchestra.run.smithi174.stdout:/run/user/1000 2024-04-25T00:01:43.356 INFO:teuthology.orchestra.run.smithi174.stdout:/var/lib/ceph 2024-04-25T00:01:43.356 INFO:teuthology.orchestra.run.smithi174.stdout:/var/lib/ceph/osd/ceph-4 2024-04-25T00:01:43.356 INFO:teuthology.orchestra.run.smithi174.stdout:/var/lib/ceph/osd/ceph-5 2024-04-25T00:01:43.356 INFO:teuthology.orchestra.run.smithi174.stdout:/var/lib/ceph/osd/ceph-6 2024-04-25T00:01:43.356 INFO:teuthology.orchestra.run.smithi174.stdout:/var/lib/ceph/osd/ceph-7 2024-04-25T00:01:43.356 INFO:teuthology.orchestra.run.smithi174.stdout:/run/netns 2024-04-25T00:01:43.356 INFO:teuthology.orchestra.run.smithi174.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T00:01:43.356 INFO:teuthology.orchestra.run.smithi174.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T00:01:43.357 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:01:43.357 DEBUG:teuthology.orchestra.run.smithi174:> ls /sys/fs/fuse/connections 2024-04-25T00:01:43.370 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi174.stderr:2024-04-25T00:01:43.370+0000 7f7ba9a43580 -1 init, newargv = 0x5573389f2260 newargc=15 2024-04-25T00:01:43.371 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi174.stderr:ceph-fuse[33357]: starting ceph client 2024-04-25T00:01:43.378 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi174.stderr:ceph-fuse[33357]: starting fuse 2024-04-25T00:01:44.477 DEBUG:teuthology.orchestra.run.smithi174:> sudo modprobe fuse 2024-04-25T00:01:44.492 DEBUG:teuthology.orchestra.run.smithi174:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T00:01:44.539 INFO:teuthology.orchestra.run.smithi174.stdout:/sys 2024-04-25T00:01:44.539 INFO:teuthology.orchestra.run.smithi174.stdout:/proc 2024-04-25T00:01:44.540 INFO:teuthology.orchestra.run.smithi174.stdout:/dev 2024-04-25T00:01:44.540 INFO:teuthology.orchestra.run.smithi174.stdout:/dev/pts 2024-04-25T00:01:44.540 INFO:teuthology.orchestra.run.smithi174.stdout:/run 2024-04-25T00:01:44.540 INFO:teuthology.orchestra.run.smithi174.stdout:/ 2024-04-25T00:01:44.540 INFO:teuthology.orchestra.run.smithi174.stdout:/sys/kernel/security 2024-04-25T00:01:44.540 INFO:teuthology.orchestra.run.smithi174.stdout:/dev/shm 2024-04-25T00:01:44.540 INFO:teuthology.orchestra.run.smithi174.stdout:/run/lock 2024-04-25T00:01:44.540 INFO:teuthology.orchestra.run.smithi174.stdout:/sys/fs/cgroup 2024-04-25T00:01:44.540 INFO:teuthology.orchestra.run.smithi174.stdout:/sys/fs/pstore 2024-04-25T00:01:44.540 INFO:teuthology.orchestra.run.smithi174.stdout:/sys/fs/bpf 2024-04-25T00:01:44.540 INFO:teuthology.orchestra.run.smithi174.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T00:01:44.541 INFO:teuthology.orchestra.run.smithi174.stdout:/dev/hugepages 2024-04-25T00:01:44.541 INFO:teuthology.orchestra.run.smithi174.stdout:/dev/mqueue 2024-04-25T00:01:44.541 INFO:teuthology.orchestra.run.smithi174.stdout:/sys/kernel/debug 2024-04-25T00:01:44.541 INFO:teuthology.orchestra.run.smithi174.stdout:/sys/kernel/tracing 2024-04-25T00:01:44.541 INFO:teuthology.orchestra.run.smithi174.stdout:/proc/fs/nfsd 2024-04-25T00:01:44.541 INFO:teuthology.orchestra.run.smithi174.stdout:/sys/fs/fuse/connections 2024-04-25T00:01:44.541 INFO:teuthology.orchestra.run.smithi174.stdout:/sys/kernel/config 2024-04-25T00:01:44.541 INFO:teuthology.orchestra.run.smithi174.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T00:01:44.541 INFO:teuthology.orchestra.run.smithi174.stdout:/run/rpc_pipefs 2024-04-25T00:01:44.541 INFO:teuthology.orchestra.run.smithi174.stdout:/run/user/1000 2024-04-25T00:01:44.541 INFO:teuthology.orchestra.run.smithi174.stdout:/var/lib/ceph 2024-04-25T00:01:44.541 INFO:teuthology.orchestra.run.smithi174.stdout:/var/lib/ceph/osd/ceph-4 2024-04-25T00:01:44.542 INFO:teuthology.orchestra.run.smithi174.stdout:/var/lib/ceph/osd/ceph-5 2024-04-25T00:01:44.542 INFO:teuthology.orchestra.run.smithi174.stdout:/var/lib/ceph/osd/ceph-6 2024-04-25T00:01:44.542 INFO:teuthology.orchestra.run.smithi174.stdout:/var/lib/ceph/osd/ceph-7 2024-04-25T00:01:44.542 INFO:teuthology.orchestra.run.smithi174.stdout:/run/netns 2024-04-25T00:01:44.542 INFO:teuthology.orchestra.run.smithi174.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T00:01:44.542 INFO:teuthology.orchestra.run.smithi174.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T00:01:44.542 INFO:teuthology.orchestra.run.smithi174.stdout:/home/ubuntu/cephtest/mnt.1 2024-04-25T00:01:44.542 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:01:44.543 DEBUG:teuthology.orchestra.run.smithi174:> ls /sys/fs/fuse/connections 2024-04-25T00:01:44.585 INFO:teuthology.orchestra.run.smithi174.stdout:45 2024-04-25T00:01:44.586 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [45] 2024-04-25T00:01:44.586 DEBUG:teuthology.orchestra.run.smithi174:> sudo stdin-killer -- python3 -c ' 2024-04-25T00:01:44.586 DEBUG:teuthology.orchestra.run.smithi174:> import glob 2024-04-25T00:01:44.586 DEBUG:teuthology.orchestra.run.smithi174:> import re 2024-04-25T00:01:44.586 DEBUG:teuthology.orchestra.run.smithi174:> import os 2024-04-25T00:01:44.586 DEBUG:teuthology.orchestra.run.smithi174:> import subprocess 2024-04-25T00:01:44.586 DEBUG:teuthology.orchestra.run.smithi174:> 2024-04-25T00:01:44.586 DEBUG:teuthology.orchestra.run.smithi174:> def _find_admin_socket(client_name): 2024-04-25T00:01:44.587 DEBUG:teuthology.orchestra.run.smithi174:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-04-25T00:01:44.587 DEBUG:teuthology.orchestra.run.smithi174:> files = glob.glob(asok_path) 2024-04-25T00:01:44.587 DEBUG:teuthology.orchestra.run.smithi174:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-04-25T00:01:44.587 DEBUG:teuthology.orchestra.run.smithi174:> 2024-04-25T00:01:44.587 DEBUG:teuthology.orchestra.run.smithi174:> # Given a non-glob path, it better be there 2024-04-25T00:01:44.587 DEBUG:teuthology.orchestra.run.smithi174:> if "*" not in asok_path: 2024-04-25T00:01:44.587 DEBUG:teuthology.orchestra.run.smithi174:> assert(len(files) == 1) 2024-04-25T00:01:44.587 DEBUG:teuthology.orchestra.run.smithi174:> return files[0] 2024-04-25T00:01:44.587 DEBUG:teuthology.orchestra.run.smithi174:> 2024-04-25T00:01:44.587 DEBUG:teuthology.orchestra.run.smithi174:> for f in files: 2024-04-25T00:01:44.587 DEBUG:teuthology.orchestra.run.smithi174:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-25T00:01:44.587 DEBUG:teuthology.orchestra.run.smithi174:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-25T00:01:44.587 DEBUG:teuthology.orchestra.run.smithi174:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-25T00:01:44.588 DEBUG:teuthology.orchestra.run.smithi174:> contents = proc_f.read() 2024-04-25T00:01:44.588 DEBUG:teuthology.orchestra.run.smithi174:> if mountpoint in contents: 2024-04-25T00:01:44.588 DEBUG:teuthology.orchestra.run.smithi174:> return f 2024-04-25T00:01:44.588 DEBUG:teuthology.orchestra.run.smithi174:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-25T00:01:44.588 DEBUG:teuthology.orchestra.run.smithi174:> 2024-04-25T00:01:44.588 DEBUG:teuthology.orchestra.run.smithi174:> print(_find_admin_socket("client.1")) 2024-04-25T00:01:44.588 DEBUG:teuthology.orchestra.run.smithi174:> ' 2024-04-25T00:01:44.728 INFO:teuthology.orchestra.run.smithi174.stdout:/var/run/ceph/ceph-client.1.33357.asok 2024-04-25T00:01:44.731 INFO:teuthology.orchestra.run.smithi174.stderr:2024-04-25T00:01:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T00:01:44.741 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.33357.asok 2024-04-25T00:01:44.741 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:01:44.741 DEBUG:teuthology.orchestra.run.smithi174:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.33357.asok status 2024-04-25T00:01:44.898 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-04-25T00:01:44.898 INFO:teuthology.orchestra.run.smithi174.stdout: "metadata": { 2024-04-25T00:01:44.898 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_sha1": "d2c45d977142e024a59b68a8f1dce8f0b9840551", 2024-04-25T00:01:44.899 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_version": "ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev)", 2024-04-25T00:01:44.899 INFO:teuthology.orchestra.run.smithi174.stdout: "entity_id": "1", 2024-04-25T00:01:44.899 INFO:teuthology.orchestra.run.smithi174.stdout: "hostname": "smithi174", 2024-04-25T00:01:44.899 INFO:teuthology.orchestra.run.smithi174.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-04-25T00:01:44.899 INFO:teuthology.orchestra.run.smithi174.stdout: "pid": "33357", 2024-04-25T00:01:44.899 INFO:teuthology.orchestra.run.smithi174.stdout: "root": "/" 2024-04-25T00:01:44.899 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-25T00:01:44.899 INFO:teuthology.orchestra.run.smithi174.stdout: "dentry_count": 0, 2024-04-25T00:01:44.900 INFO:teuthology.orchestra.run.smithi174.stdout: "dentry_pinned_count": 0, 2024-04-25T00:01:44.900 INFO:teuthology.orchestra.run.smithi174.stdout: "id": 4439, 2024-04-25T00:01:44.900 INFO:teuthology.orchestra.run.smithi174.stdout: "inst": { 2024-04-25T00:01:44.900 INFO:teuthology.orchestra.run.smithi174.stdout: "name": { 2024-04-25T00:01:44.900 INFO:teuthology.orchestra.run.smithi174.stdout: "type": "client", 2024-04-25T00:01:44.900 INFO:teuthology.orchestra.run.smithi174.stdout: "num": 4439 2024-04-25T00:01:44.900 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-25T00:01:44.900 INFO:teuthology.orchestra.run.smithi174.stdout: "addr": { 2024-04-25T00:01:44.900 INFO:teuthology.orchestra.run.smithi174.stdout: "type": "v1", 2024-04-25T00:01:44.901 INFO:teuthology.orchestra.run.smithi174.stdout: "addr": "192.168.0.1:0", 2024-04-25T00:01:44.901 INFO:teuthology.orchestra.run.smithi174.stdout: "nonce": 3700357214 2024-04-25T00:01:44.901 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-04-25T00:01:44.901 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-25T00:01:44.901 INFO:teuthology.orchestra.run.smithi174.stdout: "addr": { 2024-04-25T00:01:44.901 INFO:teuthology.orchestra.run.smithi174.stdout: "type": "v1", 2024-04-25T00:01:44.901 INFO:teuthology.orchestra.run.smithi174.stdout: "addr": "192.168.0.1:0", 2024-04-25T00:01:44.901 INFO:teuthology.orchestra.run.smithi174.stdout: "nonce": 3700357214 2024-04-25T00:01:44.901 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-25T00:01:44.901 INFO:teuthology.orchestra.run.smithi174.stdout: "inst_str": "client.4439 192.168.0.1:0/3700357214", 2024-04-25T00:01:44.902 INFO:teuthology.orchestra.run.smithi174.stdout: "addr_str": "192.168.0.1:0/3700357214", 2024-04-25T00:01:44.902 INFO:teuthology.orchestra.run.smithi174.stdout: "inode_count": 1, 2024-04-25T00:01:44.902 INFO:teuthology.orchestra.run.smithi174.stdout: "mds_epoch": 14, 2024-04-25T00:01:44.902 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_epoch": 27, 2024-04-25T00:01:44.902 INFO:teuthology.orchestra.run.smithi174.stdout: "osd_epoch_barrier": 0, 2024-04-25T00:01:44.902 INFO:teuthology.orchestra.run.smithi174.stdout: "blocklisted": false, 2024-04-25T00:01:44.902 INFO:teuthology.orchestra.run.smithi174.stdout: "fs_name": "b" 2024-04-25T00:01:44.902 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-04-25T00:01:44.913 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:01:44.913 DEBUG:teuthology.orchestra.run.smithi116:> stat --file-system '--printf=%T 2024-04-25T00:01:44.913 DEBUG:teuthology.orchestra.run.smithi116:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-25T00:01:44.921 INFO:teuthology.orchestra.run.smithi116.stdout:fuseblk 2024-04-25T00:01:44.921 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-25T00:01:44.921 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:01:44.922 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-25T00:01:44.978 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:01:44.978 DEBUG:teuthology.orchestra.run.smithi174:> stat --file-system '--printf=%T 2024-04-25T00:01:44.978 DEBUG:teuthology.orchestra.run.smithi174:> ' -- /home/ubuntu/cephtest/mnt.1 2024-04-25T00:01:44.985 INFO:teuthology.orchestra.run.smithi174.stdout:fuseblk 2024-04-25T00:01:44.985 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-04-25T00:01:44.985 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:01:44.985 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-25T00:01:45.043 INFO:teuthology.run_tasks:Running task mds_thrash... 2024-04-25T00:01:45.064 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T00:01:45.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.193+0000 7fa6b1960640 1 Processor -- start 2024-04-25T00:01:45.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.193+0000 7fa6b1960640 1 -- start start 2024-04-25T00:01:45.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.193+0000 7fa6b1960640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa6ac06d5f0 0x7fa6ac156c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:45.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.193+0000 7fa6b1960640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fa6ac155810 0x7fa6ac1571c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:45.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.193+0000 7fa6b1960640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6ac0c9150 con 0x7fa6ac152d80 2024-04-25T00:01:45.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.193+0000 7fa6b1960640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fa6ac0c9d60 con 0x7fa6ac06d5f0 2024-04-25T00:01:45.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.193+0000 7fa6b1960640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fa6ac0c9430 con 0x7fa6ac155810 2024-04-25T00:01:45.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.197+0000 7fa6aa7fc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa6ac06d5f0 0x7fa6ac156c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:45.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.197+0000 7fa6aa7fc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa6ac06d5f0 0x7fa6ac156c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43112/0 (socket says 172.21.15.116:43112) 2024-04-25T00:01:45.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.197+0000 7fa6aa7fc640 1 -- 172.21.15.116:0/81001865 learned_addr learned my addr 172.21.15.116:0/81001865 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:45.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.197+0000 7fa6ab7fe640 1 --2- 172.21.15.116:0/81001865 >> v2:172.21.15.174:3300/0 conn(0x7fa6ac155810 0x7fa6ac1571c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:45.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.197+0000 7fa6aa7fc640 1 -- 172.21.15.116:0/81001865 >> v2:172.21.15.174:3300/0 conn(0x7fa6ac155810 msgr2=0x7fa6ac1571c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:45.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.197+0000 7fa6aa7fc640 1 --2- 172.21.15.116:0/81001865 >> v2:172.21.15.174:3300/0 conn(0x7fa6ac155810 0x7fa6ac1571c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:45.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.197+0000 7fa6aa7fc640 1 -- 172.21.15.116:0/81001865 >> v1:172.21.15.174:6789/0 conn(0x7fa6ac152d80 legacy=0x7fa6ac06bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:45.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.197+0000 7fa6aa7fc640 1 -- 172.21.15.116:0/81001865 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6ac0c9710 con 0x7fa6ac06d5f0 2024-04-25T00:01:45.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.197+0000 7fa6aa7fc640 1 --2- 172.21.15.116:0/81001865 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6ac06d5f0 0x7fa6ac156c80 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac155ca0 tx=0x7fa69401c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f88c0cdd831ea04 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:45.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.197+0000 7fa6a9ffb640 1 -- 172.21.15.116:0/81001865 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6940b0070 con 0x7fa6ac06d5f0 2024-04-25T00:01:45.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.197+0000 7fa6a9ffb640 1 -- 172.21.15.116:0/81001865 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa694004230 con 0x7fa6ac06d5f0 2024-04-25T00:01:45.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.197+0000 7fa6a9ffb640 1 -- 172.21.15.116:0/81001865 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa694003620 con 0x7fa6ac06d5f0 2024-04-25T00:01:45.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.197+0000 7fa6b1960640 1 -- 172.21.15.116:0/81001865 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6ac06d5f0 msgr2=0x7fa6ac156c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:45.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.197+0000 7fa6b1960640 1 --2- 172.21.15.116:0/81001865 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6ac06d5f0 0x7fa6ac156c80 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac155ca0 tx=0x7fa69401c420 comp rx=0 tx=0).stop 2024-04-25T00:01:45.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.197+0000 7fa6b1960640 1 -- 172.21.15.116:0/81001865 shutdown_connections 2024-04-25T00:01:45.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.197+0000 7fa6b1960640 1 --2- 172.21.15.116:0/81001865 >> v2:172.21.15.174:3300/0 conn(0x7fa6ac155810 0x7fa6ac1571c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:45.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.197+0000 7fa6b1960640 1 --2- 172.21.15.116:0/81001865 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6ac06d5f0 0x7fa6ac156c80 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:45.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.197+0000 7fa6b1960640 1 -- 172.21.15.116:0/81001865 >> 172.21.15.116:0/81001865 conn(0x7fa6ac0acdd0 msgr2=0x7fa6ac0ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:45.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.197+0000 7fa6b1960640 1 -- 172.21.15.116:0/81001865 shutdown_connections 2024-04-25T00:01:45.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.197+0000 7fa6b1960640 1 -- 172.21.15.116:0/81001865 wait complete. 2024-04-25T00:01:45.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.201+0000 7fa6b1960640 1 Processor -- start 2024-04-25T00:01:45.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.201+0000 7fa6b1960640 1 -- start start 2024-04-25T00:01:45.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.201+0000 7fa6b1960640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6ac06d5f0 0x7fa6ac0b0a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:45.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.201+0000 7fa6b1960640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa6ac152d80 0x7fa6ac14c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:45.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.201+0000 7fa6b1960640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa6ac155810 0x7fa6ac151e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:45.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.201+0000 7fa6b1960640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa6ac0caf00 con 0x7fa6ac06d5f0 2024-04-25T00:01:45.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.201+0000 7fa6b1960640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa6ac0cad80 con 0x7fa6ac152d80 2024-04-25T00:01:45.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.201+0000 7fa6b1960640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa6ac1628d0 con 0x7fa6ac155810 2024-04-25T00:01:45.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.201+0000 7fa6aaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6ac06d5f0 0x7fa6ac0b0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:45.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.201+0000 7fa6aaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6ac06d5f0 0x7fa6ac0b0a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43114/0 (socket says 172.21.15.116:43114) 2024-04-25T00:01:45.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.201+0000 7fa6aaffd640 1 -- 172.21.15.116:0/1665908311 learned_addr learned my addr 172.21.15.116:0/1665908311 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:45.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.201+0000 7fa6aa7fc640 1 --2- 172.21.15.116:0/1665908311 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa6ac152d80 0x7fa6ac14c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:45.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.201+0000 7fa6aaffd640 1 -- 172.21.15.116:0/1665908311 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa6ac155810 msgr2=0x7fa6ac151e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:45.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.201+0000 7fa6ab7fe640 1 --2- 172.21.15.116:0/1665908311 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa6ac155810 0x7fa6ac151e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:45.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.201+0000 7fa6aaffd640 1 --2- 172.21.15.116:0/1665908311 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa6ac155810 0x7fa6ac151e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:45.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.201+0000 7fa6aaffd640 1 -- 172.21.15.116:0/1665908311 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa6ac152d80 msgr2=0x7fa6ac14c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:45.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.201+0000 7fa6aaffd640 1 --2- 172.21.15.116:0/1665908311 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa6ac152d80 0x7fa6ac14c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:45.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.201+0000 7fa6aaffd640 1 -- 172.21.15.116:0/1665908311 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6ac1502d0 con 0x7fa6ac06d5f0 2024-04-25T00:01:45.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.201+0000 7fa6aaffd640 1 --2- 172.21.15.116:0/1665908311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6ac06d5f0 0x7fa6ac0b0a60 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa6a400c490 tx=0x7fa6a4001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:45.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.201+0000 7fa6a3fff640 1 -- 172.21.15.116:0/1665908311 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6a4017070 con 0x7fa6ac06d5f0 2024-04-25T00:01:45.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.201+0000 7fa6a3fff640 1 -- 172.21.15.116:0/1665908311 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6a4003600 con 0x7fa6ac06d5f0 2024-04-25T00:01:45.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.201+0000 7fa6b1960640 1 -- 172.21.15.116:0/1665908311 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6ac1505b0 con 0x7fa6ac06d5f0 2024-04-25T00:01:45.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.201+0000 7fa6a3fff640 1 -- 172.21.15.116:0/1665908311 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6a4003a30 con 0x7fa6ac06d5f0 2024-04-25T00:01:45.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.201+0000 7fa6b1960640 1 -- 172.21.15.116:0/1665908311 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6ac150b70 con 0x7fa6ac06d5f0 2024-04-25T00:01:45.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.201+0000 7fa6b1960640 1 -- 172.21.15.116:0/1665908311 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6ac151060 con 0x7fa6ac06d5f0 2024-04-25T00:01:45.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.201+0000 7fa6a3fff640 1 -- 172.21.15.116:0/1665908311 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa6a401a070 con 0x7fa6ac06d5f0 2024-04-25T00:01:45.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.205+0000 7fa6a3fff640 1 --2- 172.21.15.116:0/1665908311 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa68c0678c0 0x7fa68c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:45.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.205+0000 7fa6a3fff640 1 -- 172.21.15.116:0/1665908311 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7fa6a4085390 con 0x7fa6ac06d5f0 2024-04-25T00:01:45.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.205+0000 7fa6aa7fc640 1 --2- 172.21.15.116:0/1665908311 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa68c0678c0 0x7fa68c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:45.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.205+0000 7fa6aa7fc640 1 --2- 172.21.15.116:0/1665908311 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa68c0678c0 0x7fa68c069d80 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa6940ad000 tx=0x7fa694008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:45.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.209+0000 7fa6a3fff640 1 -- 172.21.15.116:0/1665908311 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fa6a4055c30 con 0x7fa6ac06d5f0 2024-04-25T00:01:45.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.381+0000 7fa6b1960640 1 -- 172.21.15.116:0/1665908311 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa6ac1523d0 con 0x7fa6ac06d5f0 2024-04-25T00:01:45.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.385+0000 7fa6a3fff640 1 -- 172.21.15.116:0/1665908311 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+20311 (secure 0 0 0) 0x7fa6a4055530 con 0x7fa6ac06d5f0 2024-04-25T00:01:45.388 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:01:45.389 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":27,"fsid":"6d6a1aa7-0478-4c83-8057-05917e81f046","created":"2024-04-25T00:00:40.682493+0000","modified":"2024-04-25T00:01:20.501508+0000","last_up_change":"2024-04-25T00:00:51.822174+0000","last_in_change":"2024-04-25T00:00:42.980473+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T00:00:53.324376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-04-25T00:01:03.197326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-04-25T00:01:03.728618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-04-25T00:01:17.125210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-04-25T00:01:17.821873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8b6ddcb2-1ad9-4167-961c-bd6936bb9060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6809","nonce":3751765682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6811","nonce":3751765682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6815","nonce":3751765682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6813","nonce":3751765682}]},"public_addr":"172.21.15.116:6809/3751765682","cluster_addr":"172.21.15.116:6811/3751765682","heartbeat_back_addr":"172.21.15.116:6815/3751765682","heartbeat_front_addr":"172.21.15.116:6813/3751765682","state":["exists","up"]},{"osd":1,"uuid":"a6431768-994f-41da-8853-886f5687fb91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6801","nonce":377879124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6803","nonce":377879124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6807","nonce":377879124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6805","nonce":377879124}]},"public_addr":"172.21.15.116:6801/377879124","cluster_addr":"172.21.15.116:6803/377879124","heartbeat_back_addr":"172.21.15.116:6807/377879124","heartbeat_front_addr":"172.21.15.116:6805/377879124","state":["exists","up"]},{"osd":2,"uuid":"4f69bd40-5a23-45ff-8693-e1de75e1aa9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6817","nonce":1099282267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6819","nonce":1099282267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6823","nonce":1099282267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6821","nonce":1099282267}]},"public_addr":"172.21.15.116:6817/1099282267","cluster_addr":"172.21.15.116:6819/1099282267","heartbeat_back_addr":"172.21.15.116:6823/1099282267","heartbeat_front_addr":"172.21.15.116:6821/1099282267","state":["exists","up"]},{"osd":3,"uuid":"6a0d5344-8b6f-40d3-918d-0f0a4be688e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6825","nonce":1179942661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6827","nonce":1179942661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6831","nonce":1179942661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6829","nonce":1179942661}]},"public_addr":"172.21.15.116:6825/1179942661","cluster_addr":"172.21.15.116:6827/1179942661","heartbeat_back_addr":"172.21.15.116:6831/1179942661","heartbeat_front_addr":"172.21.15.116:6829/1179942661","state":["exists","up"]},{"osd":4,"uuid":"132e92e4-b825-441e-965a-ca00533fa86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6801","nonce":3132444330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6803","nonce":3132444330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6807","nonce":3132444330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6805","nonce":3132444330}]},"public_addr":"172.21.15.174:6801/3132444330","cluster_addr":"172.21.15.174:6803/3132444330","heartbeat_back_addr":"172.21.15.174:6807/3132444330","heartbeat_front_addr":"172.21.15.174:6805/3132444330","state":["exists","up"]},{"osd":5,"uuid":"a2eeb5b6-1b31-43cc-8306-a0adf3ee15bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6809","nonce":881817164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6811","nonce":881817164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6815","nonce":881817164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6813","nonce":881817164}]},"public_addr":"172.21.15.174:6809/881817164","cluster_addr":"172.21.15.174:6811/881817164","heartbeat_back_addr":"172.21.15.174:6815/881817164","heartbeat_front_addr":"172.21.15.174:6813/881817164","state":["exists","up"]},{"osd":6,"uuid":"588e873a-c841-4ba7-90f2-8956c87a9983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6825","nonce":117574251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6827","nonce":117574251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6831","nonce":117574251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6829","nonce":117574251}]},"public_addr":"172.21.15.174:6825/117574251","cluster_addr":"172.21.15.174:6827/117574251","heartbeat_back_addr":"172.21.15.174:6831/117574251","heartbeat_front_addr":"172.21.15.174:6829/117574251","state":["exists","up"]},{"osd":7,"uuid":"3b90f9b0-8a0e-4883-b088-ca1d4012870c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6817","nonce":2543899196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6819","nonce":2543899196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6823","nonce":2543899196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6821","nonce":2543899196}]},"public_addr":"172.21.15.174:6817/2543899196","cluster_addr":"172.21.15.174:6819/2543899196","heartbeat_back_addr":"172.21.15.174:6823/2543899196","heartbeat_front_addr":"172.21.15.174:6821/2543899196","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.293769+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.310034+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.404865+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.549658+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.192174+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.297266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.493649+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.452309+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:01:45.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.385+0000 7fa6b1960640 1 -- 172.21.15.116:0/1665908311 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa68c0678c0 msgr2=0x7fa68c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:45.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.385+0000 7fa6b1960640 1 --2- 172.21.15.116:0/1665908311 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa68c0678c0 0x7fa68c069d80 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa6940ad000 tx=0x7fa694008040 comp rx=0 tx=0).stop 2024-04-25T00:01:45.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.385+0000 7fa6b1960640 1 -- 172.21.15.116:0/1665908311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6ac06d5f0 msgr2=0x7fa6ac0b0a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:45.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.385+0000 7fa6b1960640 1 --2- 172.21.15.116:0/1665908311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6ac06d5f0 0x7fa6ac0b0a60 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa6a400c490 tx=0x7fa6a4001e80 comp rx=0 tx=0).stop 2024-04-25T00:01:45.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.385+0000 7fa6b1960640 1 -- 172.21.15.116:0/1665908311 shutdown_connections 2024-04-25T00:01:45.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.385+0000 7fa6b1960640 1 --2- 172.21.15.116:0/1665908311 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa68c0678c0 0x7fa68c069d80 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:45.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.385+0000 7fa6b1960640 1 --2- 172.21.15.116:0/1665908311 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa6ac155810 0x7fa6ac151e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:45.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.385+0000 7fa6b1960640 1 --2- 172.21.15.116:0/1665908311 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa6ac152d80 0x7fa6ac14c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:45.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.385+0000 7fa6b1960640 1 --2- 172.21.15.116:0/1665908311 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6ac06d5f0 0x7fa6ac0b0a60 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:45.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.385+0000 7fa6b1960640 1 -- 172.21.15.116:0/1665908311 >> 172.21.15.116:0/1665908311 conn(0x7fa6ac0acdd0 msgr2=0x7fa6ac155d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:45.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.389+0000 7fa6b1960640 1 -- 172.21.15.116:0/1665908311 shutdown_connections 2024-04-25T00:01:45.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.389+0000 7fa6b1960640 1 -- 172.21.15.116:0/1665908311 wait complete. 2024-04-25T00:01:45.406 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T00:00:53.324376+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'a_metadata', 'create_time': '2024-04-25T00:01:03.197326+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'a'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'a_data', 'create_time': '2024-04-25T00:01:03.728618+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'a'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'b_metadata', 'create_time': '2024-04-25T00:01:17.125210+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'b'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'b_data', 'create_time': '2024-04-25T00:01:17.821873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'b'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T00:01:45.406 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T00:01:45.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.529+0000 7f4a15612640 1 Processor -- start 2024-04-25T00:01:45.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.529+0000 7f4a15612640 1 -- start start 2024-04-25T00:01:45.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.529+0000 7f4a15612640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4a100c3500 0x7f4a1014edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:45.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.529+0000 7f4a15612640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4a100aa550 0x7f4a1014f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:45.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.529+0000 7f4a15612640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a1006d1a0 con 0x7f4a1006ea20 2024-04-25T00:01:45.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.529+0000 7f4a15612640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f4a10152c30 con 0x7f4a100c3500 2024-04-25T00:01:45.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.529+0000 7f4a15612640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f4a1014f830 con 0x7f4a100aa550 2024-04-25T00:01:45.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.529+0000 7f4a0e7fc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4a100c3500 0x7f4a1014edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:45.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.529+0000 7f4a0e7fc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4a100c3500 0x7f4a1014edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43130/0 (socket says 172.21.15.116:43130) 2024-04-25T00:01:45.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.529+0000 7f4a0e7fc640 1 -- 172.21.15.116:0/255768181 learned_addr learned my addr 172.21.15.116:0/255768181 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:45.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.529+0000 7f4a0f7fe640 1 --2- 172.21.15.116:0/255768181 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4a100aa550 0x7f4a1014f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:45.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.529+0000 7f4a0e7fc640 1 -- 172.21.15.116:0/255768181 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4a100aa550 msgr2=0x7f4a1014f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:45.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.529+0000 7f4a0e7fc640 1 --2- 172.21.15.116:0/255768181 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4a100aa550 0x7f4a1014f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:45.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.529+0000 7f4a0e7fc640 1 -- 172.21.15.116:0/255768181 >> v1:172.21.15.174:6789/0 conn(0x7f4a1006ea20 legacy=0x7f4a1006dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:45.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.529+0000 7f4a0e7fc640 1 -- 172.21.15.116:0/255768181 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a10152670 con 0x7f4a100c3500 2024-04-25T00:01:45.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.529+0000 7f4a0e7fc640 1 --2- 172.21.15.116:0/255768181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a100c3500 0x7f4a1014edb0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f4a10073440 tx=0x7f4a0401d820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fd1aaa45cefe01e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:45.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.533+0000 7f4a0dffb640 1 -- 172.21.15.116:0/255768181 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a0400b040 con 0x7f4a100c3500 2024-04-25T00:01:45.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.533+0000 7f4a0dffb640 1 -- 172.21.15.116:0/255768181 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a040a7710 con 0x7f4a100c3500 2024-04-25T00:01:45.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.533+0000 7f4a0dffb640 1 -- 172.21.15.116:0/255768181 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a040a7ae0 con 0x7f4a100c3500 2024-04-25T00:01:45.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.533+0000 7f4a15612640 1 -- 172.21.15.116:0/255768181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a100c3500 msgr2=0x7f4a1014edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:45.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.533+0000 7f4a15612640 1 --2- 172.21.15.116:0/255768181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a100c3500 0x7f4a1014edb0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f4a10073440 tx=0x7f4a0401d820 comp rx=0 tx=0).stop 2024-04-25T00:01:45.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.533+0000 7f4a15612640 1 -- 172.21.15.116:0/255768181 shutdown_connections 2024-04-25T00:01:45.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.533+0000 7f4a15612640 1 --2- 172.21.15.116:0/255768181 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4a100aa550 0x7f4a1014f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:45.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.533+0000 7f4a15612640 1 --2- 172.21.15.116:0/255768181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a100c3500 0x7f4a1014edb0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:45.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.533+0000 7f4a15612640 1 -- 172.21.15.116:0/255768181 >> 172.21.15.116:0/255768181 conn(0x7f4a100c3a20 msgr2=0x7f4a100c3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:45.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.533+0000 7f4a15612640 1 -- 172.21.15.116:0/255768181 shutdown_connections 2024-04-25T00:01:45.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.533+0000 7f4a15612640 1 -- 172.21.15.116:0/255768181 wait complete. 2024-04-25T00:01:45.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.533+0000 7f4a15612640 1 Processor -- start 2024-04-25T00:01:45.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.533+0000 7f4a15612640 1 -- start start 2024-04-25T00:01:45.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.533+0000 7f4a15612640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4a1006ea20 0x7f4a1014fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:45.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.533+0000 7f4a15612640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a100aa550 0x7f4a101eb540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:45.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.533+0000 7f4a15612640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4a100c3500 0x7f4a100c0350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:45.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.533+0000 7f4a15612640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4a1006e4d0 con 0x7f4a100aa550 2024-04-25T00:01:45.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.533+0000 7f4a15612640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4a1006e350 con 0x7f4a1006ea20 2024-04-25T00:01:45.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.533+0000 7f4a15612640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f4a1006ee00 con 0x7f4a100c3500 2024-04-25T00:01:45.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.533+0000 7f4a0e7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a100aa550 0x7f4a101eb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:45.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.533+0000 7f4a0e7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a100aa550 0x7f4a101eb540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43132/0 (socket says 172.21.15.116:43132) 2024-04-25T00:01:45.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.533+0000 7f4a0e7fc640 1 -- 172.21.15.116:0/120553836 learned_addr learned my addr 172.21.15.116:0/120553836 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:45.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.533+0000 7f4a0effd640 1 --2- 172.21.15.116:0/120553836 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4a1006ea20 0x7f4a1014fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:45.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.533+0000 7f4a0f7fe640 1 --2- 172.21.15.116:0/120553836 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4a100c3500 0x7f4a100c0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:45.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.533+0000 7f4a0e7fc640 1 -- 172.21.15.116:0/120553836 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4a100c3500 msgr2=0x7f4a100c0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:45.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.533+0000 7f4a0e7fc640 1 --2- 172.21.15.116:0/120553836 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4a100c3500 0x7f4a100c0350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:45.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.533+0000 7f4a0e7fc640 1 -- 172.21.15.116:0/120553836 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4a1006ea20 msgr2=0x7f4a1014fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:45.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.533+0000 7f4a0e7fc640 1 --2- 172.21.15.116:0/120553836 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4a1006ea20 0x7f4a1014fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:45.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.533+0000 7f4a0e7fc640 1 -- 172.21.15.116:0/120553836 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a1013aa40 con 0x7f4a100aa550 2024-04-25T00:01:45.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.533+0000 7f4a0e7fc640 1 --2- 172.21.15.116:0/120553836 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a100aa550 0x7f4a101eb540 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f4a0401bed0 tx=0x7f4a0401c060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:45.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.533+0000 7f49f77fe640 1 -- 172.21.15.116:0/120553836 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a040bd020 con 0x7f4a100aa550 2024-04-25T00:01:45.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.537+0000 7f49f77fe640 1 -- 172.21.15.116:0/120553836 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a040af850 con 0x7f4a100aa550 2024-04-25T00:01:45.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.537+0000 7f4a15612640 1 -- 172.21.15.116:0/120553836 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a1014c380 con 0x7f4a100aa550 2024-04-25T00:01:45.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.537+0000 7f49f77fe640 1 -- 172.21.15.116:0/120553836 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a040afc40 con 0x7f4a100aa550 2024-04-25T00:01:45.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.537+0000 7f4a15612640 1 -- 172.21.15.116:0/120553836 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a101eec00 con 0x7f4a100aa550 2024-04-25T00:01:45.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.537+0000 7f4a15612640 1 -- 172.21.15.116:0/120553836 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a1004f510 con 0x7f4a100aa550 2024-04-25T00:01:45.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.537+0000 7f49f77fe640 1 -- 172.21.15.116:0/120553836 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4a040c0020 con 0x7f4a100aa550 2024-04-25T00:01:45.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.537+0000 7f49f77fe640 1 --2- 172.21.15.116:0/120553836 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f49fc0678f0 0x7f49fc069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:45.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.537+0000 7f49f77fe640 1 -- 172.21.15.116:0/120553836 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6521+0+0 (secure 0 0 0) 0x7f4a04120870 con 0x7f4a100aa550 2024-04-25T00:01:45.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.537+0000 7f4a0effd640 1 --2- 172.21.15.116:0/120553836 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f49fc0678f0 0x7f49fc069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:45.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.537+0000 7f4a0effd640 1 --2- 172.21.15.116:0/120553836 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f49fc0678f0 0x7f49fc069db0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f4a0000b810 tx=0x7f4a00008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:45.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.541+0000 7f49f77fe640 1 -- 172.21.15.116:0/120553836 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f4a040c8020 con 0x7f4a100aa550 2024-04-25T00:01:45.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.709+0000 7f4a15612640 1 -- 172.21.15.116:0/120553836 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4a1006c220 con 0x7f4a100aa550 2024-04-25T00:01:45.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.709+0000 7f49f77fe640 1 -- 172.21.15.116:0/120553836 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) ==== 93+0+10 (secure 0 0 0) 0x7f4a040174a0 con 0x7f4a100aa550 2024-04-25T00:01:45.711 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-04-25T00:01:45.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.709+0000 7f4a15612640 1 -- 172.21.15.116:0/120553836 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f49fc0678f0 msgr2=0x7f49fc069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:45.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.709+0000 7f4a15612640 1 --2- 172.21.15.116:0/120553836 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f49fc0678f0 0x7f49fc069db0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f4a0000b810 tx=0x7f4a00008040 comp rx=0 tx=0).stop 2024-04-25T00:01:45.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.709+0000 7f4a15612640 1 -- 172.21.15.116:0/120553836 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a100aa550 msgr2=0x7f4a101eb540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:45.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.709+0000 7f4a15612640 1 --2- 172.21.15.116:0/120553836 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a100aa550 0x7f4a101eb540 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f4a0401bed0 tx=0x7f4a0401c060 comp rx=0 tx=0).stop 2024-04-25T00:01:45.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.709+0000 7f4a15612640 1 -- 172.21.15.116:0/120553836 shutdown_connections 2024-04-25T00:01:45.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.709+0000 7f4a15612640 1 --2- 172.21.15.116:0/120553836 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f49fc0678f0 0x7f49fc069db0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:45.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.709+0000 7f4a15612640 1 --2- 172.21.15.116:0/120553836 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4a100c3500 0x7f4a100c0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:45.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.709+0000 7f4a15612640 1 --2- 172.21.15.116:0/120553836 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a100aa550 0x7f4a101eb540 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:45.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.709+0000 7f4a15612640 1 --2- 172.21.15.116:0/120553836 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4a1006ea20 0x7f4a1014fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:45.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.709+0000 7f4a15612640 1 -- 172.21.15.116:0/120553836 >> 172.21.15.116:0/120553836 conn(0x7f4a100c3a20 msgr2=0x7f4a101497e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:45.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.709+0000 7f4a15612640 1 -- 172.21.15.116:0/120553836 shutdown_connections 2024-04-25T00:01:45.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.709+0000 7f4a15612640 1 -- 172.21.15.116:0/120553836 wait complete. 2024-04-25T00:01:45.730 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_metadata pg_num 2024-04-25T00:01:45.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.849+0000 7f0096155640 1 Processor -- start 2024-04-25T00:01:45.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.853+0000 7f0096155640 1 -- start start 2024-04-25T00:01:45.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.853+0000 7f0096155640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f00900c5990 0x7f00900c5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:45.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.853+0000 7f0096155640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f00900c4490 0x7f00900c4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:45.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.853+0000 7f0096155640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00901567b0 con 0x7f0090153280 2024-04-25T00:01:45.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.853+0000 7f0096155640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f00900c4000 con 0x7f00900c4490 2024-04-25T00:01:45.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.853+0000 7f0096155640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f0090156a90 con 0x7f00900c5990 2024-04-25T00:01:45.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.853+0000 7f008f7fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f0090153280 0x7f0090155d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:47500/0 (socket says 172.21.15.116:47500) 2024-04-25T00:01:45.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.853+0000 7f008f7fe640 1 -- 172.21.15.116:0/3760548367 learned_addr learned my addr 172.21.15.116:0/3760548367 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:45.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.853+0000 7f008effd640 1 --2- 172.21.15.116:0/3760548367 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f00900c5990 0x7f00900c5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:45.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.853+0000 7f008e7fc640 1 -- 172.21.15.116:0/3760548367 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3988653152 0 0) 0x7f00901567b0 con 0x7f0090153280 2024-04-25T00:01:45.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.853+0000 7f008e7fc640 1 -- 172.21.15.116:0/3760548367 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0078003530 con 0x7f0090153280 2024-04-25T00:01:45.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.853+0000 7f008ffff640 1 --2- 172.21.15.116:0/3760548367 >> v2:172.21.15.174:3300/0 conn(0x7f00900c4490 0x7f00900c4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:45.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.853+0000 7f008e7fc640 1 -- 172.21.15.116:0/3760548367 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3511872900 0 0) 0x7f0078003530 con 0x7f0090153280 2024-04-25T00:01:45.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.853+0000 7f008e7fc640 1 -- 172.21.15.116:0/3760548367 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f00900c5990 msgr2=0x7f00900c5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:45.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.853+0000 7f008e7fc640 1 --2- 172.21.15.116:0/3760548367 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f00900c5990 0x7f00900c5dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:45.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.853+0000 7f008e7fc640 1 -- 172.21.15.116:0/3760548367 >> v2:172.21.15.174:3300/0 conn(0x7f00900c4490 msgr2=0x7f00900c4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:45.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.853+0000 7f008e7fc640 1 --2- 172.21.15.116:0/3760548367 >> v2:172.21.15.174:3300/0 conn(0x7f00900c4490 0x7f00900c4920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:45.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.853+0000 7f008e7fc640 1 -- 172.21.15.116:0/3760548367 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00900c39b0 con 0x7f0090153280 2024-04-25T00:01:45.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.853+0000 7f008e7fc640 1 -- 172.21.15.116:0/3760548367 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f0084001680 con 0x7f0090153280 2024-04-25T00:01:45.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.853+0000 7f008e7fc640 1 -- 172.21.15.116:0/3760548367 >> v1:172.21.15.116:6789/0 conn(0x7f0090153280 legacy=0x7f0090155d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:45.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.853+0000 7f008e7fc640 1 --2- 172.21.15.116:0/3760548367 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f00900c5990 0x7f00780039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:45.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.853+0000 7f008e7fc640 1 --2- 172.21.15.116:0/3760548367 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0078003f40 0x7f0078004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:45.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.853+0000 7f008e7fc640 1 --2- 172.21.15.116:0/3760548367 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0078004d50 0x7f00780051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:45.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.853+0000 7f008e7fc640 1 -- 172.21.15.116:0/3760548367 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f00900c4000 con 0x7f0078004d50 2024-04-25T00:01:45.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.853+0000 7f008e7fc640 1 -- 172.21.15.116:0/3760548367 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0090156a90 con 0x7f0078003f40 2024-04-25T00:01:45.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.853+0000 7f008e7fc640 1 -- 172.21.15.116:0/3760548367 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f007800bad0 con 0x7f00900c5990 2024-04-25T00:01:45.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.853+0000 7f008ffff640 1 --2- 172.21.15.116:0/3760548367 >> v2:172.21.15.174:3300/0 conn(0x7f00900c4490 0x7f00900c4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:01:45.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.853+0000 7f008ffff640 1 --2- 172.21.15.116:0/3760548367 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0078004d50 0x7f00780051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:45.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.853+0000 7f008effd640 1 --2- 172.21.15.116:0/3760548367 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0078003f40 0x7f0078004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:45.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.853+0000 7f008f7fe640 1 --2- 172.21.15.116:0/3760548367 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f00900c5990 0x7f00780039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:45.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.853+0000 7f008ffff640 1 -- 172.21.15.116:0/3760548367 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f00900c5990 msgr2=0x7f00780039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:45.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.853+0000 7f008ffff640 1 --2- 172.21.15.116:0/3760548367 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f00900c5990 0x7f00780039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:45.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.853+0000 7f008ffff640 1 -- 172.21.15.116:0/3760548367 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0078003f40 msgr2=0x7f0078004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:45.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.853+0000 7f008ffff640 1 --2- 172.21.15.116:0/3760548367 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0078003f40 0x7f0078004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:45.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.853+0000 7f008ffff640 1 -- 172.21.15.116:0/3760548367 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0078003530 con 0x7f0078004d50 2024-04-25T00:01:45.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.853+0000 7f008f7fe640 1 --2- 172.21.15.116:0/3760548367 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f00900c5990 0x7f00780039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:45.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.857+0000 7f008ffff640 1 --2- 172.21.15.116:0/3760548367 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0078004d50 0x7f00780051d0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f0088007410 tx=0x7f008801cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce0c28d3486b6b7b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:45.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.857+0000 7f008e7fc640 1 -- 172.21.15.116:0/3760548367 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00880b2070 con 0x7f0078004d50 2024-04-25T00:01:45.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.857+0000 7f008e7fc640 1 -- 172.21.15.116:0/3760548367 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00880a5e40 con 0x7f0078004d50 2024-04-25T00:01:45.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.857+0000 7f008e7fc640 1 -- 172.21.15.116:0/3760548367 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0088009c70 con 0x7f0078004d50 2024-04-25T00:01:45.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.857+0000 7f0096155640 1 -- 172.21.15.116:0/3760548367 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0078004d50 msgr2=0x7f00780051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:45.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.857+0000 7f0096155640 1 --2- 172.21.15.116:0/3760548367 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0078004d50 0x7f00780051d0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f0088007410 tx=0x7f008801cf40 comp rx=0 tx=0).stop 2024-04-25T00:01:45.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.857+0000 7f008ffff640 1 -- 172.21.15.116:0/3760548367 reap_dead start 2024-04-25T00:01:45.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.857+0000 7f0096155640 1 -- 172.21.15.116:0/3760548367 shutdown_connections 2024-04-25T00:01:45.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.857+0000 7f0096155640 1 -- 172.21.15.116:0/3760548367 >> 172.21.15.116:0/3760548367 conn(0x7f00900c8f90 msgr2=0x7f00900c93a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:45.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.857+0000 7f0096155640 1 -- 172.21.15.116:0/3760548367 shutdown_connections 2024-04-25T00:01:45.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.857+0000 7f0096155640 1 -- 172.21.15.116:0/3760548367 wait complete. 2024-04-25T00:01:45.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.857+0000 7f0096155640 1 Processor -- start 2024-04-25T00:01:45.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.857+0000 7f0096155640 1 -- start start 2024-04-25T00:01:45.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.857+0000 7f0096155640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0078004d50 0x7f00901ee550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:45.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.857+0000 7f0096155640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00901eea90 0x7f00900a9a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:45.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.857+0000 7f0096155640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f00900bf800 0x7f00900a9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:45.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.857+0000 7f0096155640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f009006d7b0 con 0x7f00901eea90 2024-04-25T00:01:45.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.857+0000 7f0096155640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0090156440 con 0x7f00900bf800 2024-04-25T00:01:45.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.857+0000 7f0096155640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f009006d930 con 0x7f0078004d50 2024-04-25T00:01:45.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.857+0000 7f008effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00901eea90 0x7f00900a9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:45.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.857+0000 7f008effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00901eea90 0x7f00900a9a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43152/0 (socket says 172.21.15.116:43152) 2024-04-25T00:01:45.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.857+0000 7f008effd640 1 -- 172.21.15.116:0/1977806319 learned_addr learned my addr 172.21.15.116:0/1977806319 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:45.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.857+0000 7f008f7fe640 1 --2- 172.21.15.116:0/1977806319 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0078004d50 0x7f00901ee550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:45.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.857+0000 7f008ffff640 1 --2- 172.21.15.116:0/1977806319 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f00900bf800 0x7f00900a9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:45.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.857+0000 7f008effd640 1 -- 172.21.15.116:0/1977806319 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0078004d50 msgr2=0x7f00901ee550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:45.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.857+0000 7f008effd640 1 --2- 172.21.15.116:0/1977806319 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0078004d50 0x7f00901ee550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:45.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.857+0000 7f008effd640 1 -- 172.21.15.116:0/1977806319 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f00900bf800 msgr2=0x7f00900a9fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:45.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.857+0000 7f008effd640 1 --2- 172.21.15.116:0/1977806319 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f00900bf800 0x7f00900a9fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:45.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.857+0000 7f008effd640 1 -- 172.21.15.116:0/1977806319 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00901eef50 con 0x7f00901eea90 2024-04-25T00:01:45.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.857+0000 7f008effd640 1 --2- 172.21.15.116:0/1977806319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00901eea90 0x7f00900a9a70 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f0080011560 tx=0x7f0080011a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:45.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.857+0000 7f0077fff640 1 -- 172.21.15.116:0/1977806319 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0080015930 con 0x7f00901eea90 2024-04-25T00:01:45.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.861+0000 7f0077fff640 1 -- 172.21.15.116:0/1977806319 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0080015ad0 con 0x7f00901eea90 2024-04-25T00:01:45.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.861+0000 7f0096155640 1 -- 172.21.15.116:0/1977806319 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0090159bf0 con 0x7f00901eea90 2024-04-25T00:01:45.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.861+0000 7f0077fff640 1 -- 172.21.15.116:0/1977806319 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f008001d400 con 0x7f00901eea90 2024-04-25T00:01:45.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.861+0000 7f0096155640 1 -- 172.21.15.116:0/1977806319 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0090157960 con 0x7f00901eea90 2024-04-25T00:01:45.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.861+0000 7f0096155640 1 -- 172.21.15.116:0/1977806319 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f009004f510 con 0x7f00901eea90 2024-04-25T00:01:45.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.861+0000 7f0077fff640 1 -- 172.21.15.116:0/1977806319 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f008001d5a0 con 0x7f00901eea90 2024-04-25T00:01:45.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.861+0000 7f0077fff640 1 --2- 172.21.15.116:0/1977806319 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f007c067940 0x7f007c069e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:45.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.861+0000 7f0077fff640 1 -- 172.21.15.116:0/1977806319 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6666+0+0 (secure 0 0 0) 0x7f0080086ba0 con 0x7f00901eea90 2024-04-25T00:01:45.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.861+0000 7f008f7fe640 1 --2- 172.21.15.116:0/1977806319 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f007c067940 0x7f007c069e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:45.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.861+0000 7f008f7fe640 1 --2- 172.21.15.116:0/1977806319 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f007c067940 0x7f007c069e00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f00840017f0 tx=0x7f0084001660 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:45.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:45.865+0000 7f0077fff640 1 -- 172.21.15.116:0/1977806319 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f0080019030 con 0x7f00901eea90 2024-04-25T00:01:46.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.013+0000 7f0096155640 1 -- 172.21.15.116:0/1977806319 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"} v 0) -- 0x7f00900715a0 con 0x7f00901eea90 2024-04-25T00:01:46.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.013+0000 7f0077fff640 1 -- 172.21.15.116:0/1977806319 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"}]=0 v28) ==== 99+0+11 (secure 0 0 0) 0x7f0080057490 con 0x7f00901eea90 2024-04-25T00:01:46.016 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-04-25T00:01:46.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.017+0000 7f0096155640 1 -- 172.21.15.116:0/1977806319 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f007c067940 msgr2=0x7f007c069e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:46.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.017+0000 7f0096155640 1 --2- 172.21.15.116:0/1977806319 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f007c067940 0x7f007c069e00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f00840017f0 tx=0x7f0084001660 comp rx=0 tx=0).stop 2024-04-25T00:01:46.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.017+0000 7f0096155640 1 -- 172.21.15.116:0/1977806319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00901eea90 msgr2=0x7f00900a9a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:46.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.017+0000 7f0096155640 1 --2- 172.21.15.116:0/1977806319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00901eea90 0x7f00900a9a70 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f0080011560 tx=0x7f0080011a20 comp rx=0 tx=0).stop 2024-04-25T00:01:46.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.017+0000 7f0096155640 1 -- 172.21.15.116:0/1977806319 shutdown_connections 2024-04-25T00:01:46.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.017+0000 7f0096155640 1 --2- 172.21.15.116:0/1977806319 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f007c067940 0x7f007c069e00 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:46.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.017+0000 7f0096155640 1 --2- 172.21.15.116:0/1977806319 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f00900bf800 0x7f00900a9fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:46.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.017+0000 7f0096155640 1 --2- 172.21.15.116:0/1977806319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00901eea90 0x7f00900a9a70 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:46.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.017+0000 7f0096155640 1 --2- 172.21.15.116:0/1977806319 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0078004d50 0x7f00901ee550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:46.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.017+0000 7f0096155640 1 -- 172.21.15.116:0/1977806319 >> 172.21.15.116:0/1977806319 conn(0x7f00900c8f90 msgr2=0x7f00900c87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:46.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.017+0000 7f0096155640 1 -- 172.21.15.116:0/1977806319 shutdown_connections 2024-04-25T00:01:46.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.017+0000 7f0096155640 1 -- 172.21.15.116:0/1977806319 wait complete. 2024-04-25T00:01:46.034 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_data pg_num 2024-04-25T00:01:46.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.153+0000 7f7b44116640 1 Processor -- start 2024-04-25T00:01:46.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.153+0000 7f7b44116640 1 -- start start 2024-04-25T00:01:46.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.153+0000 7f7b44116640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7b3c150d60 0x7f7b3c0c2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:46.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.153+0000 7f7b44116640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b3c0a9630 0x7f7b3c156e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:46.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.153+0000 7f7b44116640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f7b3c0c2e70 0x7f7b3c157340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:46.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.153+0000 7f7b44116640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f7b3c0a9cb0 con 0x7f7b3c150d60 2024-04-25T00:01:46.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.153+0000 7f7b44116640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f7b3c0a9f90 con 0x7f7b3c0c2e70 2024-04-25T00:01:46.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.153+0000 7f7b44116640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f7b3c0aa270 con 0x7f7b3c0a9630 2024-04-25T00:01:46.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.153+0000 7f7b41eb2640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7b3c150d60 0x7f7b3c0c2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:46.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.153+0000 7f7b41eb2640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7b3c150d60 0x7f7b3c0c2930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43166/0 (socket says 172.21.15.116:43166) 2024-04-25T00:01:46.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.153+0000 7f7b41eb2640 1 -- 172.21.15.116:0/500247368 learned_addr learned my addr 172.21.15.116:0/500247368 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:46.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.153+0000 7f7b416b1640 1 --2- 172.21.15.116:0/500247368 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b3c0a9630 0x7f7b3c156e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:46.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.157+0000 7f7b426b3640 1 --2- 172.21.15.116:0/500247368 >> v2:172.21.15.174:3300/0 conn(0x7f7b3c0c2e70 0x7f7b3c157340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:46.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.157+0000 7f7b41eb2640 1 -- 172.21.15.116:0/500247368 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b3c0a9630 msgr2=0x7f7b3c156e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:46.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.157+0000 7f7b41eb2640 1 --2- 172.21.15.116:0/500247368 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b3c0a9630 0x7f7b3c156e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:46.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.157+0000 7f7b41eb2640 1 -- 172.21.15.116:0/500247368 >> v2:172.21.15.174:3300/0 conn(0x7f7b3c0c2e70 msgr2=0x7f7b3c157340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:46.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.157+0000 7f7b41eb2640 1 --2- 172.21.15.116:0/500247368 >> v2:172.21.15.174:3300/0 conn(0x7f7b3c0c2e70 0x7f7b3c157340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:46.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.157+0000 7f7b41eb2640 1 -- 172.21.15.116:0/500247368 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b3c0aa550 con 0x7f7b3c150d60 2024-04-25T00:01:46.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.157+0000 7f7b426b3640 1 --2- 172.21.15.116:0/500247368 >> v2:172.21.15.174:3300/0 conn(0x7f7b3c0c2e70 0x7f7b3c157340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:46.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.157+0000 7f7b41eb2640 1 --2- 172.21.15.116:0/500247368 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b3c150d60 0x7f7b3c0c2930 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c0c1350 tx=0x7f7b3401c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e9c09d75224d1e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:46.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.157+0000 7f7b40eb0640 1 -- 172.21.15.116:0/500247368 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b340a5740 con 0x7f7b3c150d60 2024-04-25T00:01:46.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.157+0000 7f7b40eb0640 1 -- 172.21.15.116:0/500247368 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b3c0ab7a0 con 0x7f7b3c150d60 2024-04-25T00:01:46.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.157+0000 7f7b40eb0640 1 -- 172.21.15.116:0/500247368 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b340a5b00 con 0x7f7b3c150d60 2024-04-25T00:01:46.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.157+0000 7f7b44116640 1 -- 172.21.15.116:0/500247368 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b3c150d60 msgr2=0x7f7b3c0c2930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:46.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.157+0000 7f7b44116640 1 --2- 172.21.15.116:0/500247368 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b3c150d60 0x7f7b3c0c2930 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c0c1350 tx=0x7f7b3401c8a0 comp rx=0 tx=0).stop 2024-04-25T00:01:46.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.157+0000 7f7b44116640 1 -- 172.21.15.116:0/500247368 shutdown_connections 2024-04-25T00:01:46.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.157+0000 7f7b44116640 1 --2- 172.21.15.116:0/500247368 >> v2:172.21.15.174:3300/0 conn(0x7f7b3c0c2e70 0x7f7b3c157340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:46.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.157+0000 7f7b44116640 1 --2- 172.21.15.116:0/500247368 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b3c0a9630 0x7f7b3c156e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:46.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.157+0000 7f7b44116640 1 --2- 172.21.15.116:0/500247368 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b3c150d60 0x7f7b3c0c2930 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:46.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.157+0000 7f7b44116640 1 -- 172.21.15.116:0/500247368 >> 172.21.15.116:0/500247368 conn(0x7f7b3c0c3a20 msgr2=0x7f7b3c0c3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:46.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.157+0000 7f7b44116640 1 -- 172.21.15.116:0/500247368 shutdown_connections 2024-04-25T00:01:46.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.157+0000 7f7b44116640 1 -- 172.21.15.116:0/500247368 wait complete. 2024-04-25T00:01:46.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.161+0000 7f7b44116640 1 Processor -- start 2024-04-25T00:01:46.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.161+0000 7f7b44116640 1 -- start start 2024-04-25T00:01:46.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.161+0000 7f7b44116640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b3c0a9630 0x7f7b3c144d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:46.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.161+0000 7f7b44116640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b3c0c2e70 0x7f7b3c13efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:46.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.161+0000 7f7b44116640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b3c150d60 0x7f7b3c146b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:46.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.161+0000 7f7b44116640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7b3c0715b0 con 0x7f7b3c150d60 2024-04-25T00:01:46.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.161+0000 7f7b44116640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7b3c0721e0 con 0x7f7b3c0c2e70 2024-04-25T00:01:46.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.161+0000 7f7b44116640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f7b3c071730 con 0x7f7b3c0a9630 2024-04-25T00:01:46.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.161+0000 7f7b426b3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b3c150d60 0x7f7b3c146b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:46.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.161+0000 7f7b426b3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b3c150d60 0x7f7b3c146b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43170/0 (socket says 172.21.15.116:43170) 2024-04-25T00:01:46.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.161+0000 7f7b426b3640 1 -- 172.21.15.116:0/742551695 learned_addr learned my addr 172.21.15.116:0/742551695 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:46.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.161+0000 7f7b416b1640 1 --2- 172.21.15.116:0/742551695 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b3c0c2e70 0x7f7b3c13efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:46.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.161+0000 7f7b41eb2640 1 --2- 172.21.15.116:0/742551695 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b3c0a9630 0x7f7b3c144d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:46.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.161+0000 7f7b426b3640 1 -- 172.21.15.116:0/742551695 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b3c0a9630 msgr2=0x7f7b3c144d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:46.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.161+0000 7f7b426b3640 1 --2- 172.21.15.116:0/742551695 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b3c0a9630 0x7f7b3c144d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:46.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.161+0000 7f7b426b3640 1 -- 172.21.15.116:0/742551695 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b3c0c2e70 msgr2=0x7f7b3c13efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:46.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.161+0000 7f7b426b3640 1 --2- 172.21.15.116:0/742551695 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b3c0c2e70 0x7f7b3c13efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:46.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.161+0000 7f7b426b3640 1 -- 172.21.15.116:0/742551695 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b3c0c4cb0 con 0x7f7b3c150d60 2024-04-25T00:01:46.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.161+0000 7f7b426b3640 1 --2- 172.21.15.116:0/742551695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b3c150d60 0x7f7b3c146b90 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f7b3800c7f0 tx=0x7f7b3800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:46.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.161+0000 7f7b327fc640 1 -- 172.21.15.116:0/742551695 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b38015070 con 0x7f7b3c150d60 2024-04-25T00:01:46.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.161+0000 7f7b327fc640 1 -- 172.21.15.116:0/742551695 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b38007860 con 0x7f7b3c150d60 2024-04-25T00:01:46.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.161+0000 7f7b327fc640 1 -- 172.21.15.116:0/742551695 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b38007c50 con 0x7f7b3c150d60 2024-04-25T00:01:46.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.161+0000 7f7b44116640 1 -- 172.21.15.116:0/742551695 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b3c0c4f50 con 0x7f7b3c150d60 2024-04-25T00:01:46.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.161+0000 7f7b44116640 1 -- 172.21.15.116:0/742551695 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b3c1470d0 con 0x7f7b3c150d60 2024-04-25T00:01:46.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.161+0000 7f7b44116640 1 -- 172.21.15.116:0/742551695 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b3c04f510 con 0x7f7b3c150d60 2024-04-25T00:01:46.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.161+0000 7f7b327fc640 1 -- 172.21.15.116:0/742551695 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7b38018070 con 0x7f7b3c150d60 2024-04-25T00:01:46.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.161+0000 7f7b327fc640 1 --2- 172.21.15.116:0/742551695 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7b200678f0 0x7f7b20069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:46.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.165+0000 7f7b327fc640 1 -- 172.21.15.116:0/742551695 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6666+0+0 (secure 0 0 0) 0x7f7b38085f60 con 0x7f7b3c150d60 2024-04-25T00:01:46.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.165+0000 7f7b41eb2640 1 --2- 172.21.15.116:0/742551695 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7b200678f0 0x7f7b20069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:46.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.165+0000 7f7b41eb2640 1 --2- 172.21.15.116:0/742551695 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7b200678f0 0x7f7b20069db0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7b340ae000 tx=0x7f7b340a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:46.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.169+0000 7f7b327fc640 1 -- 172.21.15.116:0/742551695 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f7b38056770 con 0x7f7b3c150d60 2024-04-25T00:01:46.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.329+0000 7f7b44116640 1 -- 172.21.15.116:0/742551695 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"} v 0) -- 0x7f7b3c073560 con 0x7f7b3c150d60 2024-04-25T00:01:46.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.329+0000 7f7b327fc640 1 -- 172.21.15.116:0/742551695 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"}]=0 v28) ==== 95+0+11 (secure 0 0 0) 0x7f7b38056070 con 0x7f7b3c150d60 2024-04-25T00:01:46.334 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-04-25T00:01:46.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.333+0000 7f7b44116640 1 -- 172.21.15.116:0/742551695 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7b200678f0 msgr2=0x7f7b20069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:46.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.333+0000 7f7b44116640 1 --2- 172.21.15.116:0/742551695 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7b200678f0 0x7f7b20069db0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7b340ae000 tx=0x7f7b340a7070 comp rx=0 tx=0).stop 2024-04-25T00:01:46.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.333+0000 7f7b44116640 1 -- 172.21.15.116:0/742551695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b3c150d60 msgr2=0x7f7b3c146b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:46.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.333+0000 7f7b44116640 1 --2- 172.21.15.116:0/742551695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b3c150d60 0x7f7b3c146b90 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f7b3800c7f0 tx=0x7f7b3800ccb0 comp rx=0 tx=0).stop 2024-04-25T00:01:46.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.333+0000 7f7b44116640 1 -- 172.21.15.116:0/742551695 shutdown_connections 2024-04-25T00:01:46.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.333+0000 7f7b44116640 1 --2- 172.21.15.116:0/742551695 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7b200678f0 0x7f7b20069db0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:46.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.333+0000 7f7b44116640 1 --2- 172.21.15.116:0/742551695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b3c150d60 0x7f7b3c146b90 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:46.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.333+0000 7f7b44116640 1 --2- 172.21.15.116:0/742551695 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b3c0c2e70 0x7f7b3c13efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:46.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.333+0000 7f7b44116640 1 --2- 172.21.15.116:0/742551695 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b3c0a9630 0x7f7b3c144d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:46.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.333+0000 7f7b44116640 1 -- 172.21.15.116:0/742551695 >> 172.21.15.116:0/742551695 conn(0x7f7b3c0c3a20 msgr2=0x7f7b3c0ac020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:46.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.333+0000 7f7b44116640 1 -- 172.21.15.116:0/742551695 shutdown_connections 2024-04-25T00:01:46.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.333+0000 7f7b44116640 1 -- 172.21.15.116:0/742551695 wait complete. 2024-04-25T00:01:46.353 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_metadata pg_num 2024-04-25T00:01:46.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.469+0000 7f7e1c541640 1 Processor -- start 2024-04-25T00:01:46.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.469+0000 7f7e1c541640 1 -- start start 2024-04-25T00:01:46.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.473+0000 7f7e1c541640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7e140ca1b0 0x7f7e1406bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:46.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.473+0000 7f7e1c541640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f7e140c9260 0x7f7e140c9710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:46.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.473+0000 7f7e1c541640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e1406c300 con 0x7f7e14153130 2024-04-25T00:01:46.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.473+0000 7f7e1c541640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f7e140ca820 con 0x7f7e140ca1b0 2024-04-25T00:01:46.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.473+0000 7f7e1c541640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f7e14155e50 con 0x7f7e140c9260 2024-04-25T00:01:46.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.473+0000 7f7e19adc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7e140ca1b0 0x7f7e1406bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:46.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.473+0000 7f7e1a2dd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f7e14153130 0x7f7e141562e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:47504/0 (socket says 172.21.15.116:47504) 2024-04-25T00:01:46.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.473+0000 7f7e1a2dd640 1 -- 172.21.15.116:0/1949303484 learned_addr learned my addr 172.21.15.116:0/1949303484 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:46.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.473+0000 7f7e1aade640 1 --2- 172.21.15.116:0/1949303484 >> v2:172.21.15.174:3300/0 conn(0x7f7e140c9260 0x7f7e140c9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:46.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.473+0000 7f7e19adc640 1 -- 172.21.15.116:0/1949303484 >> v2:172.21.15.174:3300/0 conn(0x7f7e140c9260 msgr2=0x7f7e140c9710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:46.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.473+0000 7f7e19adc640 1 --2- 172.21.15.116:0/1949303484 >> v2:172.21.15.174:3300/0 conn(0x7f7e140c9260 0x7f7e140c9710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:46.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.473+0000 7f7e19adc640 1 -- 172.21.15.116:0/1949303484 >> v1:172.21.15.116:6789/0 conn(0x7f7e14153130 legacy=0x7f7e141562e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:46.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.473+0000 7f7e192db640 1 -- 172.21.15.116:0/1949303484 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3961862978 0 0) 0x7f7e1406c300 con 0x7f7e14153130 2024-04-25T00:01:46.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.473+0000 7f7e19adc640 1 -- 172.21.15.116:0/1949303484 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e140caa70 con 0x7f7e140ca1b0 2024-04-25T00:01:46.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.473+0000 7f7e19adc640 1 --2- 172.21.15.116:0/1949303484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e140ca1b0 0x7f7e1406bdc0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f7e1406b9f0 tx=0x7f7e0401c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=970bced3b442abc3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:46.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.473+0000 7f7e192db640 1 -- 172.21.15.116:0/1949303484 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e040b0070 con 0x7f7e140ca1b0 2024-04-25T00:01:46.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.473+0000 7f7e192db640 1 -- 172.21.15.116:0/1949303484 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e040ab600 con 0x7f7e140ca1b0 2024-04-25T00:01:46.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.473+0000 7f7e192db640 1 -- 172.21.15.116:0/1949303484 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e040ab9d0 con 0x7f7e140ca1b0 2024-04-25T00:01:46.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.473+0000 7f7e1c541640 1 -- 172.21.15.116:0/1949303484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e140ca1b0 msgr2=0x7f7e1406bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:46.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.473+0000 7f7e1c541640 1 --2- 172.21.15.116:0/1949303484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e140ca1b0 0x7f7e1406bdc0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f7e1406b9f0 tx=0x7f7e0401c360 comp rx=0 tx=0).stop 2024-04-25T00:01:46.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.473+0000 7f7e1c541640 1 -- 172.21.15.116:0/1949303484 shutdown_connections 2024-04-25T00:01:46.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.473+0000 7f7e1c541640 1 --2- 172.21.15.116:0/1949303484 >> v2:172.21.15.174:3300/0 conn(0x7f7e140c9260 0x7f7e140c9710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:46.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.473+0000 7f7e1c541640 1 --2- 172.21.15.116:0/1949303484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e140ca1b0 0x7f7e1406bdc0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:46.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.473+0000 7f7e1c541640 1 -- 172.21.15.116:0/1949303484 >> 172.21.15.116:0/1949303484 conn(0x7f7e140cb310 msgr2=0x7f7e140cb720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:46.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.477+0000 7f7e1c541640 1 -- 172.21.15.116:0/1949303484 shutdown_connections 2024-04-25T00:01:46.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.477+0000 7f7e1c541640 1 -- 172.21.15.116:0/1949303484 wait complete. 2024-04-25T00:01:46.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.477+0000 7f7e1c541640 1 Processor -- start 2024-04-25T00:01:46.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.477+0000 7f7e1c541640 1 -- start start 2024-04-25T00:01:46.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.477+0000 7f7e1c541640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e140c9260 0x7f7e140abe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:46.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.477+0000 7f7e1c541640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e140ca1b0 0x7f7e141ef440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:46.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.477+0000 7f7e1c541640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7e14153130 0x7f7e1406c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:46.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.477+0000 7f7e1c541640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7e140bea70 con 0x7f7e140ca1b0 2024-04-25T00:01:46.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.477+0000 7f7e1c541640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7e140be8f0 con 0x7f7e140c9260 2024-04-25T00:01:46.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.477+0000 7f7e1c541640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f7e14162990 con 0x7f7e14153130 2024-04-25T00:01:46.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.477+0000 7f7e19adc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e140ca1b0 0x7f7e141ef440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:46.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.477+0000 7f7e19adc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e140ca1b0 0x7f7e141ef440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43176/0 (socket says 172.21.15.116:43176) 2024-04-25T00:01:46.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.477+0000 7f7e19adc640 1 -- 172.21.15.116:0/1754289768 learned_addr learned my addr 172.21.15.116:0/1754289768 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:46.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.477+0000 7f7e1aade640 1 --2- 172.21.15.116:0/1754289768 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7e14153130 0x7f7e1406c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:46.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.477+0000 7f7e1a2dd640 1 --2- 172.21.15.116:0/1754289768 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e140c9260 0x7f7e140abe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:46.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.477+0000 7f7e19adc640 1 -- 172.21.15.116:0/1754289768 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7e14153130 msgr2=0x7f7e1406c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:46.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.477+0000 7f7e19adc640 1 --2- 172.21.15.116:0/1754289768 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7e14153130 0x7f7e1406c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:46.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.477+0000 7f7e19adc640 1 -- 172.21.15.116:0/1754289768 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e140c9260 msgr2=0x7f7e140abe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:46.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.477+0000 7f7e19adc640 1 --2- 172.21.15.116:0/1754289768 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e140c9260 0x7f7e140abe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:46.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.477+0000 7f7e19adc640 1 -- 172.21.15.116:0/1754289768 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e140aabc0 con 0x7f7e140ca1b0 2024-04-25T00:01:46.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.477+0000 7f7e1a2dd640 1 --2- 172.21.15.116:0/1754289768 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e140c9260 0x7f7e140abe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:46.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.477+0000 7f7e19adc640 1 --2- 172.21.15.116:0/1754289768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e140ca1b0 0x7f7e141ef440 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f7e040ae040 tx=0x7f7e0401aba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:46.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.477+0000 7f7e0affd640 1 -- 172.21.15.116:0/1754289768 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e040b0050 con 0x7f7e140ca1b0 2024-04-25T00:01:46.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.477+0000 7f7e0affd640 1 -- 172.21.15.116:0/1754289768 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e040bc070 con 0x7f7e140ca1b0 2024-04-25T00:01:46.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.477+0000 7f7e1c541640 1 -- 172.21.15.116:0/1754289768 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e140aaea0 con 0x7f7e140ca1b0 2024-04-25T00:01:46.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.477+0000 7f7e0affd640 1 -- 172.21.15.116:0/1754289768 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e040b5830 con 0x7f7e140ca1b0 2024-04-25T00:01:46.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.477+0000 7f7e1c541640 1 -- 172.21.15.116:0/1754289768 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e140ab460 con 0x7f7e140ca1b0 2024-04-25T00:01:46.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.477+0000 7f7e1c541640 1 -- 172.21.15.116:0/1754289768 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e1404f510 con 0x7f7e140ca1b0 2024-04-25T00:01:46.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.481+0000 7f7e0affd640 1 -- 172.21.15.116:0/1754289768 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7e040b59d0 con 0x7f7e140ca1b0 2024-04-25T00:01:46.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.481+0000 7f7e0affd640 1 --2- 172.21.15.116:0/1754289768 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7df80678c0 0x7f7df8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:46.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.481+0000 7f7e0affd640 1 -- 172.21.15.116:0/1754289768 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6666+0+0 (secure 0 0 0) 0x7f7e04120780 con 0x7f7e140ca1b0 2024-04-25T00:01:46.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.481+0000 7f7e1a2dd640 1 --2- 172.21.15.116:0/1754289768 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7df80678c0 0x7f7df8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:46.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.481+0000 7f7e1a2dd640 1 --2- 172.21.15.116:0/1754289768 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7df80678c0 0x7f7df8069d80 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f7e10000f70 tx=0x7f7e10009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:46.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.485+0000 7f7e0affd640 1 -- 172.21.15.116:0/1754289768 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f7e04016070 con 0x7f7e140ca1b0 2024-04-25T00:01:46.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.633+0000 7f7e1c541640 1 -- 172.21.15.116:0/1754289768 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"} v 0) -- 0x7f7e140c9040 con 0x7f7e140ca1b0 2024-04-25T00:01:46.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.637+0000 7f7e0affd640 1 -- 172.21.15.116:0/1754289768 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"}]=0 v28) ==== 99+0+11 (secure 0 0 0) 0x7f7e040f0f90 con 0x7f7e140ca1b0 2024-04-25T00:01:46.639 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-04-25T00:01:46.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.637+0000 7f7e1c541640 1 -- 172.21.15.116:0/1754289768 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7df80678c0 msgr2=0x7f7df8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:46.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.637+0000 7f7e1c541640 1 --2- 172.21.15.116:0/1754289768 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7df80678c0 0x7f7df8069d80 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f7e10000f70 tx=0x7f7e10009040 comp rx=0 tx=0).stop 2024-04-25T00:01:46.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.637+0000 7f7e1c541640 1 -- 172.21.15.116:0/1754289768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e140ca1b0 msgr2=0x7f7e141ef440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:46.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.637+0000 7f7e1c541640 1 --2- 172.21.15.116:0/1754289768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e140ca1b0 0x7f7e141ef440 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f7e040ae040 tx=0x7f7e0401aba0 comp rx=0 tx=0).stop 2024-04-25T00:01:46.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.637+0000 7f7e1c541640 1 -- 172.21.15.116:0/1754289768 shutdown_connections 2024-04-25T00:01:46.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.637+0000 7f7e1c541640 1 --2- 172.21.15.116:0/1754289768 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7df80678c0 0x7f7df8069d80 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:46.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.637+0000 7f7e1c541640 1 --2- 172.21.15.116:0/1754289768 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7e14153130 0x7f7e1406c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:46.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.637+0000 7f7e1c541640 1 --2- 172.21.15.116:0/1754289768 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e140ca1b0 0x7f7e141ef440 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:46.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.637+0000 7f7e1c541640 1 --2- 172.21.15.116:0/1754289768 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e140c9260 0x7f7e140abe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:46.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.637+0000 7f7e1c541640 1 -- 172.21.15.116:0/1754289768 >> 172.21.15.116:0/1754289768 conn(0x7f7e140cb310 msgr2=0x7f7e14150960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:46.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.637+0000 7f7e1c541640 1 -- 172.21.15.116:0/1754289768 shutdown_connections 2024-04-25T00:01:46.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.637+0000 7f7e1c541640 1 -- 172.21.15.116:0/1754289768 wait complete. 2024-04-25T00:01:46.661 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_data pg_num 2024-04-25T00:01:46.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.769+0000 7fb1aeaca640 1 Processor -- start 2024-04-25T00:01:46.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.769+0000 7fb1aeaca640 1 -- start start 2024-04-25T00:01:46.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.769+0000 7fb1aeaca640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb1a8150d60 0x7fb1a80c2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:46.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.769+0000 7fb1aeaca640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb1a80c2e70 0x7fb1a8157510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:46.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.769+0000 7fb1aeaca640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1a80a9cb0 con 0x7fb1a80a9630 2024-04-25T00:01:46.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.769+0000 7fb1aeaca640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fb1a80a9f90 con 0x7fb1a80c2e70 2024-04-25T00:01:46.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.769+0000 7fb1aeaca640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fb1a80aa270 con 0x7fb1a8150d60 2024-04-25T00:01:46.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.769+0000 7fb1ad067640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb1a80c2e70 0x7fb1a8157510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:46.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.769+0000 7fb1ad067640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb1a80c2e70 0x7fb1a8157510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43188/0 (socket says 172.21.15.116:43188) 2024-04-25T00:01:46.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.769+0000 7fb1ad067640 1 -- 172.21.15.116:0/2216792108 learned_addr learned my addr 172.21.15.116:0/2216792108 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:46.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.769+0000 7fb1ac866640 1 --2- 172.21.15.116:0/2216792108 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb1a8150d60 0x7fb1a80c2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:46.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.769+0000 7fb1ad067640 1 -- 172.21.15.116:0/2216792108 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb1a8150d60 msgr2=0x7fb1a80c2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:46.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.769+0000 7fb1ad067640 1 --2- 172.21.15.116:0/2216792108 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb1a8150d60 0x7fb1a80c2930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:46.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.769+0000 7fb1ad067640 1 -- 172.21.15.116:0/2216792108 >> v1:172.21.15.116:6789/0 conn(0x7fb1a80a9630 legacy=0x7fb1a8156e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:46.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.769+0000 7fb1ad067640 1 -- 172.21.15.116:0/2216792108 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1a80aa550 con 0x7fb1a80c2e70 2024-04-25T00:01:46.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.769+0000 7fb1a77fe640 1 -- 172.21.15.116:0/2216792108 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1051026411 0 0) 0x7fb1a80a9cb0 con 0x7fb1a80a9630 2024-04-25T00:01:46.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.769+0000 7fb1ac866640 1 --2- 172.21.15.116:0/2216792108 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb1a8150d60 0x7fb1a80c2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:46.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.773+0000 7fb1ad067640 1 --2- 172.21.15.116:0/2216792108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1a80c2e70 0x7fb1a8157510 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fb1a0009e50 tx=0x7fb1a001c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1538cf9534a5d923 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:46.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.773+0000 7fb1a77fe640 1 -- 172.21.15.116:0/2216792108 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a00a57c0 con 0x7fb1a80c2e70 2024-04-25T00:01:46.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.773+0000 7fb1a77fe640 1 -- 172.21.15.116:0/2216792108 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1a00a5960 con 0x7fb1a80c2e70 2024-04-25T00:01:46.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.773+0000 7fb1a77fe640 1 -- 172.21.15.116:0/2216792108 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a00a5d30 con 0x7fb1a80c2e70 2024-04-25T00:01:46.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.773+0000 7fb1aeaca640 1 -- 172.21.15.116:0/2216792108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1a80c2e70 msgr2=0x7fb1a8157510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:46.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.773+0000 7fb1aeaca640 1 --2- 172.21.15.116:0/2216792108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1a80c2e70 0x7fb1a8157510 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fb1a0009e50 tx=0x7fb1a001c8e0 comp rx=0 tx=0).stop 2024-04-25T00:01:46.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.773+0000 7fb1aeaca640 1 -- 172.21.15.116:0/2216792108 shutdown_connections 2024-04-25T00:01:46.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.773+0000 7fb1aeaca640 1 --2- 172.21.15.116:0/2216792108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1a80c2e70 0x7fb1a8157510 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:46.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.773+0000 7fb1aeaca640 1 --2- 172.21.15.116:0/2216792108 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb1a8150d60 0x7fb1a80c2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:46.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.773+0000 7fb1aeaca640 1 -- 172.21.15.116:0/2216792108 >> 172.21.15.116:0/2216792108 conn(0x7fb1a80c3a20 msgr2=0x7fb1a80c3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:46.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.773+0000 7fb1aeaca640 1 -- 172.21.15.116:0/2216792108 shutdown_connections 2024-04-25T00:01:46.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.773+0000 7fb1aeaca640 1 -- 172.21.15.116:0/2216792108 wait complete. 2024-04-25T00:01:46.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.773+0000 7fb1aeaca640 1 Processor -- start 2024-04-25T00:01:46.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.773+0000 7fb1aeaca640 1 -- start start 2024-04-25T00:01:46.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.773+0000 7fb1aeaca640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb1a80a9630 0x7fb1a8152d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:46.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.773+0000 7fb1aeaca640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb1a80c2e70 0x7fb1a81432c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:46.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.773+0000 7fb1aeaca640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1a8150d60 0x7fb1a8141a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:46.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.773+0000 7fb1aeaca640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a8071fd0 con 0x7fb1a8150d60 2024-04-25T00:01:46.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.773+0000 7fb1aeaca640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a80ab7a0 con 0x7fb1a80c2e70 2024-04-25T00:01:46.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.773+0000 7fb1aeaca640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fb1a8072150 con 0x7fb1a80a9630 2024-04-25T00:01:46.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.773+0000 7fb1ad067640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1a8150d60 0x7fb1a8141a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:46.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.773+0000 7fb1ad067640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1a8150d60 0x7fb1a8141a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43196/0 (socket says 172.21.15.116:43196) 2024-04-25T00:01:46.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.773+0000 7fb1ad067640 1 -- 172.21.15.116:0/1059861042 learned_addr learned my addr 172.21.15.116:0/1059861042 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:46.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.773+0000 7fb1ac866640 1 --2- 172.21.15.116:0/1059861042 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb1a80a9630 0x7fb1a8152d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:46.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.773+0000 7fb1ad067640 1 -- 172.21.15.116:0/1059861042 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb1a80a9630 msgr2=0x7fb1a8152d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:46.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.773+0000 7fb1ad067640 1 --2- 172.21.15.116:0/1059861042 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb1a80a9630 0x7fb1a8152d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:46.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.773+0000 7fb1ad067640 1 -- 172.21.15.116:0/1059861042 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb1a80c2e70 msgr2=0x7fb1a81432c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:46.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.773+0000 7fb1ad067640 1 --2- 172.21.15.116:0/1059861042 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb1a80c2e70 0x7fb1a81432c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:46.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.773+0000 7fb1ad067640 1 -- 172.21.15.116:0/1059861042 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1a8153680 con 0x7fb1a8150d60 2024-04-25T00:01:46.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.773+0000 7fb1ac866640 1 --2- 172.21.15.116:0/1059861042 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb1a80a9630 0x7fb1a8152d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:46.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.773+0000 7fb1ad067640 1 --2- 172.21.15.116:0/1059861042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1a8150d60 0x7fb1a8141a10 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fb1a00af040 tx=0x7fb1a0019680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:46.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.773+0000 7fb1a57fa640 1 -- 172.21.15.116:0/1059861042 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a00bc020 con 0x7fb1a8150d60 2024-04-25T00:01:46.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.773+0000 7fb1a57fa640 1 -- 172.21.15.116:0/1059861042 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1a00b8670 con 0x7fb1a8150d60 2024-04-25T00:01:46.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.773+0000 7fb1a57fa640 1 -- 172.21.15.116:0/1059861042 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1a00b8a60 con 0x7fb1a8150d60 2024-04-25T00:01:46.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.773+0000 7fb1aeaca640 1 -- 172.21.15.116:0/1059861042 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1a8153960 con 0x7fb1a8150d60 2024-04-25T00:01:46.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.773+0000 7fb1aeaca640 1 -- 172.21.15.116:0/1059861042 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1a8154440 con 0x7fb1a8150d60 2024-04-25T00:01:46.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.777+0000 7fb1aeaca640 1 -- 172.21.15.116:0/1059861042 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1a804f510 con 0x7fb1a8150d60 2024-04-25T00:01:46.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.777+0000 7fb1a57fa640 1 -- 172.21.15.116:0/1059861042 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb1a00bf020 con 0x7fb1a8150d60 2024-04-25T00:01:46.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.777+0000 7fb1a57fa640 1 --2- 172.21.15.116:0/1059861042 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb1940678c0 0x7fb194069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:46.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.777+0000 7fb1a57fa640 1 -- 172.21.15.116:0/1059861042 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6666+0+0 (secure 0 0 0) 0x7fb1a01210e0 con 0x7fb1a8150d60 2024-04-25T00:01:46.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.777+0000 7fb1ac866640 1 --2- 172.21.15.116:0/1059861042 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb1940678c0 0x7fb194069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:46.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.777+0000 7fb1ac866640 1 --2- 172.21.15.116:0/1059861042 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb1940678c0 0x7fb194069d80 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb1a8142e20 tx=0x7fb198008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:46.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.781+0000 7fb1a57fa640 1 -- 172.21.15.116:0/1059861042 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fb1a00c6030 con 0x7fb1a8150d60 2024-04-25T00:01:46.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.929+0000 7fb1aeaca640 1 -- 172.21.15.116:0/1059861042 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"} v 0) -- 0x7fb1a8157590 con 0x7fb1a8150d60 2024-04-25T00:01:46.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.929+0000 7fb1a57fa640 1 -- 172.21.15.116:0/1059861042 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"}]=0 v29) ==== 95+0+11 (secure 0 0 0) 0x7fb1a0016620 con 0x7fb1a8150d60 2024-04-25T00:01:46.931 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-04-25T00:01:46.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.929+0000 7fb1aeaca640 1 -- 172.21.15.116:0/1059861042 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb1940678c0 msgr2=0x7fb194069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:46.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.929+0000 7fb1aeaca640 1 --2- 172.21.15.116:0/1059861042 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb1940678c0 0x7fb194069d80 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb1a8142e20 tx=0x7fb198008040 comp rx=0 tx=0).stop 2024-04-25T00:01:46.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.929+0000 7fb1aeaca640 1 -- 172.21.15.116:0/1059861042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1a8150d60 msgr2=0x7fb1a8141a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:46.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.929+0000 7fb1aeaca640 1 --2- 172.21.15.116:0/1059861042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1a8150d60 0x7fb1a8141a10 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fb1a00af040 tx=0x7fb1a0019680 comp rx=0 tx=0).stop 2024-04-25T00:01:46.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.929+0000 7fb1aeaca640 1 -- 172.21.15.116:0/1059861042 shutdown_connections 2024-04-25T00:01:46.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.929+0000 7fb1aeaca640 1 --2- 172.21.15.116:0/1059861042 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb1940678c0 0x7fb194069d80 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:46.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.929+0000 7fb1aeaca640 1 --2- 172.21.15.116:0/1059861042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb1a8150d60 0x7fb1a8141a10 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:46.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.929+0000 7fb1aeaca640 1 --2- 172.21.15.116:0/1059861042 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb1a80c2e70 0x7fb1a81432c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:46.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.929+0000 7fb1aeaca640 1 --2- 172.21.15.116:0/1059861042 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb1a80a9630 0x7fb1a8152d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:46.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.929+0000 7fb1aeaca640 1 -- 172.21.15.116:0/1059861042 >> 172.21.15.116:0/1059861042 conn(0x7fb1a80c3a20 msgr2=0x7fb1a81470e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:46.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.929+0000 7fb1aeaca640 1 -- 172.21.15.116:0/1059861042 shutdown_connections 2024-04-25T00:01:46.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:46.929+0000 7fb1aeaca640 1 -- 172.21.15.116:0/1059861042 wait complete. 2024-04-25T00:01:46.948 INFO:tasks.mds_thrash:mds thrasher using random seed: 1714003306 2024-04-25T00:01:46.949 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T00:01:47.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.049+0000 7f477b8af640 1 Processor -- start 2024-04-25T00:01:47.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.049+0000 7f477b8af640 1 -- start start 2024-04-25T00:01:47.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.053+0000 7f477b8af640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4774152d80 0x7f477406bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:47.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.053+0000 7f477b8af640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f477406d5f0 0x7f4774156c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:47.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.053+0000 7f477b8af640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f477406c290 0x7f47741571c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:47.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.053+0000 7f477b8af640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f47740c9d60 con 0x7f477406d5f0 2024-04-25T00:01:47.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.053+0000 7f477b8af640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f47740c9150 con 0x7f477406c290 2024-04-25T00:01:47.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.053+0000 7f477b8af640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f47740c9430 con 0x7f4774152d80 2024-04-25T00:01:47.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.053+0000 7f4778e4a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f477406d5f0 0x7f4774156c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:47.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.053+0000 7f4778e4a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f477406d5f0 0x7f4774156c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43210/0 (socket says 172.21.15.116:43210) 2024-04-25T00:01:47.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.053+0000 7f4778e4a640 1 -- 172.21.15.116:0/2471647479 learned_addr learned my addr 172.21.15.116:0/2471647479 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:47.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.053+0000 7f477964b640 1 --2- 172.21.15.116:0/2471647479 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4774152d80 0x7f477406bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:47.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.053+0000 7f4779e4c640 1 --2- 172.21.15.116:0/2471647479 >> v2:172.21.15.174:3300/0 conn(0x7f477406c290 0x7f47741571c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:47.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.053+0000 7f4778e4a640 1 -- 172.21.15.116:0/2471647479 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4774152d80 msgr2=0x7f477406bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:47.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.053+0000 7f4778e4a640 1 --2- 172.21.15.116:0/2471647479 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4774152d80 0x7f477406bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:47.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.053+0000 7f4778e4a640 1 -- 172.21.15.116:0/2471647479 >> v2:172.21.15.174:3300/0 conn(0x7f477406c290 msgr2=0x7f47741571c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:47.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.053+0000 7f4778e4a640 1 --2- 172.21.15.116:0/2471647479 >> v2:172.21.15.174:3300/0 conn(0x7f477406c290 0x7f47741571c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:47.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.053+0000 7f4778e4a640 1 -- 172.21.15.116:0/2471647479 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47740c9710 con 0x7f477406d5f0 2024-04-25T00:01:47.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.053+0000 7f4779e4c640 1 --2- 172.21.15.116:0/2471647479 >> v2:172.21.15.174:3300/0 conn(0x7f477406c290 0x7f47741571c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:47.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.053+0000 7f4778e4a640 1 --2- 172.21.15.116:0/2471647479 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f477406d5f0 0x7f4774156c80 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f47740c86f0 tx=0x7f476c01c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67806894a0a46e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:47.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.053+0000 7f476bfff640 1 -- 172.21.15.116:0/2471647479 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f476c0b0070 con 0x7f477406d5f0 2024-04-25T00:01:47.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.057+0000 7f476bfff640 1 -- 172.21.15.116:0/2471647479 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f476c0ab600 con 0x7f477406d5f0 2024-04-25T00:01:47.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.057+0000 7f476bfff640 1 -- 172.21.15.116:0/2471647479 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f476c0ab9d0 con 0x7f477406d5f0 2024-04-25T00:01:47.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.057+0000 7f477b8af640 1 -- 172.21.15.116:0/2471647479 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f477406d5f0 msgr2=0x7f4774156c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:47.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.057+0000 7f477b8af640 1 --2- 172.21.15.116:0/2471647479 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f477406d5f0 0x7f4774156c80 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f47740c86f0 tx=0x7f476c01c460 comp rx=0 tx=0).stop 2024-04-25T00:01:47.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.057+0000 7f477b8af640 1 -- 172.21.15.116:0/2471647479 shutdown_connections 2024-04-25T00:01:47.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.057+0000 7f477b8af640 1 --2- 172.21.15.116:0/2471647479 >> v2:172.21.15.174:3300/0 conn(0x7f477406c290 0x7f47741571c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:47.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.057+0000 7f477b8af640 1 --2- 172.21.15.116:0/2471647479 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f477406d5f0 0x7f4774156c80 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:47.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.057+0000 7f477b8af640 1 --2- 172.21.15.116:0/2471647479 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4774152d80 0x7f477406bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:47.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.057+0000 7f477b8af640 1 -- 172.21.15.116:0/2471647479 >> 172.21.15.116:0/2471647479 conn(0x7f47740acdd0 msgr2=0x7f47740ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:47.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.057+0000 7f477b8af640 1 -- 172.21.15.116:0/2471647479 shutdown_connections 2024-04-25T00:01:47.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.057+0000 7f477b8af640 1 -- 172.21.15.116:0/2471647479 wait complete. 2024-04-25T00:01:47.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.057+0000 7f477b8af640 1 Processor -- start 2024-04-25T00:01:47.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.057+0000 7f477b8af640 1 -- start start 2024-04-25T00:01:47.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.057+0000 7f477b8af640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f477406c290 0x7f47741f2630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:47.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.057+0000 7f477b8af640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f477406d5f0 0x7f4774151900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:47.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.057+0000 7f477b8af640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4774152d80 0x7f47740c2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:47.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.057+0000 7f477b8af640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4774159710 con 0x7f477406c290 2024-04-25T00:01:47.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.057+0000 7f477b8af640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4774159590 con 0x7f4774152d80 2024-04-25T00:01:47.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.057+0000 7f477b8af640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f4774159890 con 0x7f477406d5f0 2024-04-25T00:01:47.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.057+0000 7f477964b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f477406c290 0x7f47741f2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:47.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.057+0000 7f477964b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f477406c290 0x7f47741f2630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43216/0 (socket says 172.21.15.116:43216) 2024-04-25T00:01:47.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.057+0000 7f477964b640 1 -- 172.21.15.116:0/930073818 learned_addr learned my addr 172.21.15.116:0/930073818 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:47.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.057+0000 7f4779e4c640 1 --2- 172.21.15.116:0/930073818 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4774152d80 0x7f47740c2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:47.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.057+0000 7f4778e4a640 1 --2- 172.21.15.116:0/930073818 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f477406d5f0 0x7f4774151900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:47.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.057+0000 7f477964b640 1 -- 172.21.15.116:0/930073818 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f477406d5f0 msgr2=0x7f4774151900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:47.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.057+0000 7f477964b640 1 --2- 172.21.15.116:0/930073818 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f477406d5f0 0x7f4774151900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:47.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.057+0000 7f477964b640 1 -- 172.21.15.116:0/930073818 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4774152d80 msgr2=0x7f47740c2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:47.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.057+0000 7f477964b640 1 --2- 172.21.15.116:0/930073818 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4774152d80 0x7f47740c2ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:47.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.057+0000 7f477964b640 1 -- 172.21.15.116:0/930073818 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47741ef0c0 con 0x7f477406c290 2024-04-25T00:01:47.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.057+0000 7f477964b640 1 --2- 172.21.15.116:0/930073818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f477406c290 0x7f47741f2630 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f4770007e90 tx=0x7f477000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:47.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.057+0000 7f4769ffb640 1 -- 172.21.15.116:0/930073818 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4770014420 con 0x7f477406c290 2024-04-25T00:01:47.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.057+0000 7f4769ffb640 1 -- 172.21.15.116:0/930073818 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4770019070 con 0x7f477406c290 2024-04-25T00:01:47.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.057+0000 7f4769ffb640 1 -- 172.21.15.116:0/930073818 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4770014820 con 0x7f477406c290 2024-04-25T00:01:47.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.057+0000 7f477b8af640 1 -- 172.21.15.116:0/930073818 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47741f1b50 con 0x7f477406c290 2024-04-25T00:01:47.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.057+0000 7f477b8af640 1 -- 172.21.15.116:0/930073818 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47741f2110 con 0x7f477406c290 2024-04-25T00:01:47.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.061+0000 7f477b8af640 1 -- 172.21.15.116:0/930073818 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4774068350 con 0x7f477406c290 2024-04-25T00:01:47.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.061+0000 7f4769ffb640 1 -- 172.21.15.116:0/930073818 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f47700149c0 con 0x7f477406c290 2024-04-25T00:01:47.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.061+0000 7f4769ffb640 1 --2- 172.21.15.116:0/930073818 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f47600678c0 0x7f4760069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:47.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.061+0000 7f4769ffb640 1 -- 172.21.15.116:0/930073818 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6666+0+0 (secure 0 0 0) 0x7f47700866d0 con 0x7f477406c290 2024-04-25T00:01:47.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.061+0000 7f4778e4a640 1 --2- 172.21.15.116:0/930073818 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f47600678c0 0x7f4760069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:47.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.061+0000 7f4778e4a640 1 --2- 172.21.15.116:0/930073818 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f47600678c0 0x7f4760069d80 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f47740c8c30 tx=0x7f476c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:47.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.065+0000 7f4769ffb640 1 -- 172.21.15.116:0/930073818 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f4770056890 con 0x7f477406c290 2024-04-25T00:01:47.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.225+0000 7f477b8af640 1 -- 172.21.15.116:0/930073818 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f47740ad1b0 con 0x7f477406c290 2024-04-25T00:01:47.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.229+0000 7f4769ffb640 1 -- 172.21.15.116:0/930073818 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) ==== 74+0+20606 (secure 0 0 0) 0x7f477005ab90 con 0x7f477406c290 2024-04-25T00:01:47.232 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:01:47.233 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":29,"fsid":"6d6a1aa7-0478-4c83-8057-05917e81f046","created":"2024-04-25T00:00:40.682493+0000","modified":"2024-04-25T00:01:46.633464+0000","last_up_change":"2024-04-25T00:00:51.822174+0000","last_in_change":"2024-04-25T00:00:42.980473+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T00:00:53.324376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-04-25T00:01:03.197326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-04-25T00:01:03.728618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-04-25T00:01:17.125210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-04-25T00:01:17.821873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8b6ddcb2-1ad9-4167-961c-bd6936bb9060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6809","nonce":3751765682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6811","nonce":3751765682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6815","nonce":3751765682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6813","nonce":3751765682}]},"public_addr":"172.21.15.116:6809/3751765682","cluster_addr":"172.21.15.116:6811/3751765682","heartbeat_back_addr":"172.21.15.116:6815/3751765682","heartbeat_front_addr":"172.21.15.116:6813/3751765682","state":["exists","up"]},{"osd":1,"uuid":"a6431768-994f-41da-8853-886f5687fb91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6801","nonce":377879124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6803","nonce":377879124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6807","nonce":377879124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6805","nonce":377879124}]},"public_addr":"172.21.15.116:6801/377879124","cluster_addr":"172.21.15.116:6803/377879124","heartbeat_back_addr":"172.21.15.116:6807/377879124","heartbeat_front_addr":"172.21.15.116:6805/377879124","state":["exists","up"]},{"osd":2,"uuid":"4f69bd40-5a23-45ff-8693-e1de75e1aa9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6817","nonce":1099282267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6819","nonce":1099282267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6823","nonce":1099282267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6821","nonce":1099282267}]},"public_addr":"172.21.15.116:6817/1099282267","cluster_addr":"172.21.15.116:6819/1099282267","heartbeat_back_addr":"172.21.15.116:6823/1099282267","heartbeat_front_addr":"172.21.15.116:6821/1099282267","state":["exists","up"]},{"osd":3,"uuid":"6a0d5344-8b6f-40d3-918d-0f0a4be688e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6825","nonce":1179942661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6827","nonce":1179942661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6831","nonce":1179942661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6829","nonce":1179942661}]},"public_addr":"172.21.15.116:6825/1179942661","cluster_addr":"172.21.15.116:6827/1179942661","heartbeat_back_addr":"172.21.15.116:6831/1179942661","heartbeat_front_addr":"172.21.15.116:6829/1179942661","state":["exists","up"]},{"osd":4,"uuid":"132e92e4-b825-441e-965a-ca00533fa86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6801","nonce":3132444330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6803","nonce":3132444330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6807","nonce":3132444330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6805","nonce":3132444330}]},"public_addr":"172.21.15.174:6801/3132444330","cluster_addr":"172.21.15.174:6803/3132444330","heartbeat_back_addr":"172.21.15.174:6807/3132444330","heartbeat_front_addr":"172.21.15.174:6805/3132444330","state":["exists","up"]},{"osd":5,"uuid":"a2eeb5b6-1b31-43cc-8306-a0adf3ee15bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6809","nonce":881817164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6811","nonce":881817164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6815","nonce":881817164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6813","nonce":881817164}]},"public_addr":"172.21.15.174:6809/881817164","cluster_addr":"172.21.15.174:6811/881817164","heartbeat_back_addr":"172.21.15.174:6815/881817164","heartbeat_front_addr":"172.21.15.174:6813/881817164","state":["exists","up"]},{"osd":6,"uuid":"588e873a-c841-4ba7-90f2-8956c87a9983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6825","nonce":117574251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6827","nonce":117574251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6831","nonce":117574251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6829","nonce":117574251}]},"public_addr":"172.21.15.174:6825/117574251","cluster_addr":"172.21.15.174:6827/117574251","heartbeat_back_addr":"172.21.15.174:6831/117574251","heartbeat_front_addr":"172.21.15.174:6829/117574251","state":["exists","up"]},{"osd":7,"uuid":"3b90f9b0-8a0e-4883-b088-ca1d4012870c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6817","nonce":2543899196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6819","nonce":2543899196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6823","nonce":2543899196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6821","nonce":2543899196}]},"public_addr":"172.21.15.174:6817/2543899196","cluster_addr":"172.21.15.174:6819/2543899196","heartbeat_back_addr":"172.21.15.174:6823/2543899196","heartbeat_front_addr":"172.21.15.174:6821/2543899196","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.293769+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.310034+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.404865+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.549658+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.192174+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.297266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.493649+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.452309+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2c","mappings":[{"from":0,"to":7}]},{"pgid":"5.3","mappings":[{"from":4,"to":5}]},{"pgid":"5.5","mappings":[{"from":4,"to":5}]},{"pgid":"5.14","mappings":[{"from":4,"to":5}]},{"pgid":"5.19","mappings":[{"from":4,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:01:47.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.233+0000 7f477b8af640 1 -- 172.21.15.116:0/930073818 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f47600678c0 msgr2=0x7f4760069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:47.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.233+0000 7f477b8af640 1 --2- 172.21.15.116:0/930073818 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f47600678c0 0x7f4760069d80 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f47740c8c30 tx=0x7f476c008040 comp rx=0 tx=0).stop 2024-04-25T00:01:47.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.233+0000 7f477b8af640 1 -- 172.21.15.116:0/930073818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f477406c290 msgr2=0x7f47741f2630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:47.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.233+0000 7f477b8af640 1 --2- 172.21.15.116:0/930073818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f477406c290 0x7f47741f2630 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f4770007e90 tx=0x7f477000e4b0 comp rx=0 tx=0).stop 2024-04-25T00:01:47.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.233+0000 7f477b8af640 1 -- 172.21.15.116:0/930073818 shutdown_connections 2024-04-25T00:01:47.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.233+0000 7f477b8af640 1 --2- 172.21.15.116:0/930073818 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f47600678c0 0x7f4760069d80 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:47.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.233+0000 7f477b8af640 1 --2- 172.21.15.116:0/930073818 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4774152d80 0x7f47740c2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:47.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.233+0000 7f477b8af640 1 --2- 172.21.15.116:0/930073818 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f477406d5f0 0x7f4774151900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:47.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.233+0000 7f477b8af640 1 --2- 172.21.15.116:0/930073818 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f477406c290 0x7f47741f2630 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:47.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.233+0000 7f477b8af640 1 -- 172.21.15.116:0/930073818 >> 172.21.15.116:0/930073818 conn(0x7f47740acdd0 msgr2=0x7f47740caf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:47.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.233+0000 7f477b8af640 1 -- 172.21.15.116:0/930073818 shutdown_connections 2024-04-25T00:01:47.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.233+0000 7f477b8af640 1 -- 172.21.15.116:0/930073818 wait complete. 2024-04-25T00:01:47.252 INFO:tasks.mds_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T00:00:53.324376+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'a_metadata', 'create_time': '2024-04-25T00:01:03.197326+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'a'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'a_data', 'create_time': '2024-04-25T00:01:03.728618+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'a'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'b_metadata', 'create_time': '2024-04-25T00:01:17.125210+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'b'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'b_data', 'create_time': '2024-04-25T00:01:17.821873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'b'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T00:01:47.252 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T00:01:47.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.361+0000 7f8440173640 1 Processor -- start 2024-04-25T00:01:47.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.361+0000 7f8440173640 1 -- start start 2024-04-25T00:01:47.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.361+0000 7f8440173640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f843806d510 0x7f8438072500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:47.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.361+0000 7f8440173640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f843806d000 0x7f8438072a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:47.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.361+0000 7f8440173640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8438149300 con 0x7f8438148f20 2024-04-25T00:01:47.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.361+0000 7f8440173640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f8438153350 con 0x7f843806d000 2024-04-25T00:01:47.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.361+0000 7f8440173640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f84380a95d0 con 0x7f843806d510 2024-04-25T00:01:47.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.361+0000 7f843df0f640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f8438148f20 0x7f84380c2930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:47514/0 (socket says 172.21.15.116:47514) 2024-04-25T00:01:47.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.361+0000 7f843df0f640 1 -- 172.21.15.116:0/3719210299 learned_addr learned my addr 172.21.15.116:0/3719210299 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:47.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.361+0000 7f843d70e640 1 --2- 172.21.15.116:0/3719210299 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f843806d510 0x7f8438072500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:47.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.361+0000 7f843e710640 1 --2- 172.21.15.116:0/3719210299 >> v2:172.21.15.174:3300/0 conn(0x7f843806d000 0x7f8438072a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:47.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.361+0000 7f843cf0d640 1 -- 172.21.15.116:0/3719210299 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3379951046 0 0) 0x7f8434001120 con 0x7f8438148f20 2024-04-25T00:01:47.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.361+0000 7f843cf0d640 1 -- 172.21.15.116:0/3719210299 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8424005740 con 0x7f8438148f20 2024-04-25T00:01:47.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.361+0000 7f843cf0d640 1 -- 172.21.15.116:0/3719210299 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2528968049 0 0) 0x7f8424005740 con 0x7f8438148f20 2024-04-25T00:01:47.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.361+0000 7f843cf0d640 1 -- 172.21.15.116:0/3719210299 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f843806d510 msgr2=0x7f8438072500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:47.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.361+0000 7f843cf0d640 1 --2- 172.21.15.116:0/3719210299 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f843806d510 0x7f8438072500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:47.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.361+0000 7f843cf0d640 1 -- 172.21.15.116:0/3719210299 >> v2:172.21.15.174:3300/0 conn(0x7f843806d000 msgr2=0x7f8438072a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:47.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.361+0000 7f843cf0d640 1 --2- 172.21.15.116:0/3719210299 >> v2:172.21.15.174:3300/0 conn(0x7f843806d000 0x7f8438072a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:47.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.361+0000 7f843cf0d640 1 -- 172.21.15.116:0/3719210299 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8438153070 con 0x7f8438148f20 2024-04-25T00:01:47.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.361+0000 7f843cf0d640 1 -- 172.21.15.116:0/3719210299 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f8434001850 con 0x7f8438148f20 2024-04-25T00:01:47.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.361+0000 7f843cf0d640 1 -- 172.21.15.116:0/3719210299 >> v1:172.21.15.116:6789/0 conn(0x7f8438148f20 legacy=0x7f84380c2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:47.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.361+0000 7f843cf0d640 1 --2- 172.21.15.116:0/3719210299 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8424005bd0 0x7f8424005ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:47.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.361+0000 7f843cf0d640 1 --2- 172.21.15.116:0/3719210299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8424006a20 0x7f8424006ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:47.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.361+0000 7f843cf0d640 1 --2- 172.21.15.116:0/3719210299 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f843806d510 0x7f84240073e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:47.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.361+0000 7f843cf0d640 1 -- 172.21.15.116:0/3719210299 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8438153350 con 0x7f8424006a20 2024-04-25T00:01:47.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.361+0000 7f843cf0d640 1 -- 172.21.15.116:0/3719210299 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f84380a95d0 con 0x7f8424005bd0 2024-04-25T00:01:47.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.361+0000 7f843cf0d640 1 -- 172.21.15.116:0/3719210299 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f842400dc60 con 0x7f843806d510 2024-04-25T00:01:47.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.361+0000 7f843d70e640 1 --2- 172.21.15.116:0/3719210299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8424006a20 0x7f8424006ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:47.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.361+0000 7f843df0f640 1 --2- 172.21.15.116:0/3719210299 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8424005bd0 0x7f8424005ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:47.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.361+0000 7f843e710640 1 --2- 172.21.15.116:0/3719210299 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f843806d510 0x7f84240073e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:47.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.361+0000 7f843d70e640 1 -- 172.21.15.116:0/3719210299 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f843806d510 msgr2=0x7f84240073e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:47.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.361+0000 7f843d70e640 1 --2- 172.21.15.116:0/3719210299 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f843806d510 0x7f84240073e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:47.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.361+0000 7f843d70e640 1 -- 172.21.15.116:0/3719210299 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8424005bd0 msgr2=0x7f8424005ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:47.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.361+0000 7f843d70e640 1 --2- 172.21.15.116:0/3719210299 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8424005bd0 0x7f8424005ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:47.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.361+0000 7f843d70e640 1 -- 172.21.15.116:0/3719210299 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8424005740 con 0x7f8424006a20 2024-04-25T00:01:47.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.365+0000 7f843d70e640 1 --2- 172.21.15.116:0/3719210299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8424006a20 0x7f8424006ea0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f8428009ac0 tx=0x7f842801ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=341b072264777038 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:47.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.365+0000 7f843cf0d640 1 -- 172.21.15.116:0/3719210299 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8428009690 con 0x7f8424006a20 2024-04-25T00:01:47.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.365+0000 7f843cf0d640 1 -- 172.21.15.116:0/3719210299 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84280a9430 con 0x7f8424006a20 2024-04-25T00:01:47.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.365+0000 7f843cf0d640 1 -- 172.21.15.116:0/3719210299 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84280a9800 con 0x7f8424006a20 2024-04-25T00:01:47.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.365+0000 7f8440173640 1 -- 172.21.15.116:0/3719210299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8424006a20 msgr2=0x7f8424006ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:47.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.365+0000 7f8440173640 1 --2- 172.21.15.116:0/3719210299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8424006a20 0x7f8424006ea0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f8428009ac0 tx=0x7f842801ca50 comp rx=0 tx=0).stop 2024-04-25T00:01:47.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.365+0000 7f843e710640 1 -- 172.21.15.116:0/3719210299 reap_dead start 2024-04-25T00:01:47.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.365+0000 7f8440173640 1 -- 172.21.15.116:0/3719210299 shutdown_connections 2024-04-25T00:01:47.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.365+0000 7f8440173640 1 -- 172.21.15.116:0/3719210299 >> 172.21.15.116:0/3719210299 conn(0x7f84380c3a20 msgr2=0x7f84380c3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:47.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.365+0000 7f8440173640 1 -- 172.21.15.116:0/3719210299 shutdown_connections 2024-04-25T00:01:47.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.365+0000 7f8440173640 1 -- 172.21.15.116:0/3719210299 wait complete. 2024-04-25T00:01:47.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.365+0000 7f8440173640 1 Processor -- start 2024-04-25T00:01:47.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.365+0000 7f8440173640 1 -- start start 2024-04-25T00:01:47.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.365+0000 7f8440173640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8424006a20 0x7f8438147b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:47.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.365+0000 7f8440173640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8438144790 0x7f84380c6590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:47.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.365+0000 7f8440173640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f84380c6ad0 0x7f843813dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:47.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.365+0000 7f8440173640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f84380c3850 con 0x7f8424006a20 2024-04-25T00:01:47.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.365+0000 7f8440173640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f84380c36d0 con 0x7f8438144790 2024-04-25T00:01:47.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.365+0000 7f8440173640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f84380715b0 con 0x7f84380c6ad0 2024-04-25T00:01:47.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.365+0000 7f843df0f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8424006a20 0x7f8438147b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:47.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.365+0000 7f843df0f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8424006a20 0x7f8438147b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43236/0 (socket says 172.21.15.116:43236) 2024-04-25T00:01:47.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.365+0000 7f843df0f640 1 -- 172.21.15.116:0/54643803 learned_addr learned my addr 172.21.15.116:0/54643803 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:47.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.369+0000 7f843e710640 1 --2- 172.21.15.116:0/54643803 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f84380c6ad0 0x7f843813dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:47.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.369+0000 7f843d70e640 1 --2- 172.21.15.116:0/54643803 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8438144790 0x7f84380c6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:47.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.369+0000 7f843df0f640 1 -- 172.21.15.116:0/54643803 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f84380c6ad0 msgr2=0x7f843813dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:47.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.369+0000 7f843df0f640 1 --2- 172.21.15.116:0/54643803 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f84380c6ad0 0x7f843813dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:47.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.369+0000 7f843df0f640 1 -- 172.21.15.116:0/54643803 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8438144790 msgr2=0x7f84380c6590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:47.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.369+0000 7f843df0f640 1 --2- 172.21.15.116:0/54643803 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8438144790 0x7f84380c6590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:47.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.369+0000 7f843df0f640 1 -- 172.21.15.116:0/54643803 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84381484a0 con 0x7f8424006a20 2024-04-25T00:01:47.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.369+0000 7f843df0f640 1 --2- 172.21.15.116:0/54643803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8424006a20 0x7f8438147b90 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f843400d700 tx=0x7f843400dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:47.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.369+0000 7f842e7fc640 1 -- 172.21.15.116:0/54643803 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8434019070 con 0x7f8424006a20 2024-04-25T00:01:47.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.369+0000 7f842e7fc640 1 -- 172.21.15.116:0/54643803 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84340074e0 con 0x7f8424006a20 2024-04-25T00:01:47.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.369+0000 7f8440173640 1 -- 172.21.15.116:0/54643803 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f843813ebb0 con 0x7f8424006a20 2024-04-25T00:01:47.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.369+0000 7f842e7fc640 1 -- 172.21.15.116:0/54643803 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84340049f0 con 0x7f8424006a20 2024-04-25T00:01:47.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.369+0000 7f8440173640 1 -- 172.21.15.116:0/54643803 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84380c4df0 con 0x7f8424006a20 2024-04-25T00:01:47.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.369+0000 7f8440173640 1 -- 172.21.15.116:0/54643803 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f843804f510 con 0x7f8424006a20 2024-04-25T00:01:47.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.369+0000 7f842e7fc640 1 -- 172.21.15.116:0/54643803 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f843401c070 con 0x7f8424006a20 2024-04-25T00:01:47.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.369+0000 7f842e7fc640 1 --2- 172.21.15.116:0/54643803 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f841c0678f0 0x7f841c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:47.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.369+0000 7f842e7fc640 1 -- 172.21.15.116:0/54643803 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6666+0+0 (secure 0 0 0) 0x7f8434085950 con 0x7f8424006a20 2024-04-25T00:01:47.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.369+0000 7f843d70e640 1 --2- 172.21.15.116:0/54643803 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f841c0678f0 0x7f841c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:47.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.369+0000 7f843d70e640 1 --2- 172.21.15.116:0/54643803 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f841c0678f0 0x7f841c069db0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f84280ae000 tx=0x7f84280046d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:47.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.373+0000 7f842e7fc640 1 -- 172.21.15.116:0/54643803 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f84381484a0 con 0x7f8424006a20 2024-04-25T00:01:47.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.525+0000 7f8440173640 1 -- 172.21.15.116:0/54643803 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f84380c34b0 con 0x7f8424006a20 2024-04-25T00:01:47.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.525+0000 7f842e7fc640 1 -- 172.21.15.116:0/54643803 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) ==== 93+0+10 (secure 0 0 0) 0x7f8438149300 con 0x7f8424006a20 2024-04-25T00:01:47.528 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-04-25T00:01:47.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.525+0000 7f8440173640 1 -- 172.21.15.116:0/54643803 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f841c0678f0 msgr2=0x7f841c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:47.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.525+0000 7f8440173640 1 --2- 172.21.15.116:0/54643803 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f841c0678f0 0x7f841c069db0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f84280ae000 tx=0x7f84280046d0 comp rx=0 tx=0).stop 2024-04-25T00:01:47.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.525+0000 7f8440173640 1 -- 172.21.15.116:0/54643803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8424006a20 msgr2=0x7f8438147b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:47.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.525+0000 7f8440173640 1 --2- 172.21.15.116:0/54643803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8424006a20 0x7f8438147b90 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f843400d700 tx=0x7f843400dbc0 comp rx=0 tx=0).stop 2024-04-25T00:01:47.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.525+0000 7f8440173640 1 -- 172.21.15.116:0/54643803 shutdown_connections 2024-04-25T00:01:47.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.525+0000 7f8440173640 1 --2- 172.21.15.116:0/54643803 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f841c0678f0 0x7f841c069db0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:47.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.525+0000 7f8440173640 1 --2- 172.21.15.116:0/54643803 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f84380c6ad0 0x7f843813dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:47.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.525+0000 7f8440173640 1 --2- 172.21.15.116:0/54643803 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8438144790 0x7f84380c6590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:47.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.525+0000 7f8440173640 1 --2- 172.21.15.116:0/54643803 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8424006a20 0x7f8438147b90 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:47.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.525+0000 7f8440173640 1 -- 172.21.15.116:0/54643803 >> 172.21.15.116:0/54643803 conn(0x7f84380c3a20 msgr2=0x7f84380c3e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:47.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.529+0000 7f8440173640 1 -- 172.21.15.116:0/54643803 shutdown_connections 2024-04-25T00:01:47.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.529+0000 7f8440173640 1 -- 172.21.15.116:0/54643803 wait complete. 2024-04-25T00:01:47.545 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_metadata pg_num 2024-04-25T00:01:47.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.665+0000 7f67d316c640 1 Processor -- start 2024-04-25T00:01:47.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.665+0000 7f67d316c640 1 -- start start 2024-04-25T00:01:47.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.669+0000 7f67d316c640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f67cc0a9630 0x7f67cc156e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:47.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.669+0000 7f67d316c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f67cc157340 0x7f67cc157790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:47.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.669+0000 7f67d316c640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67cc0a9cb0 con 0x7f67cc150d60 2024-04-25T00:01:47.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.669+0000 7f67d316c640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f67cc0a9f90 con 0x7f67cc157340 2024-04-25T00:01:47.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.669+0000 7f67d316c640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f67cc0aa270 con 0x7f67cc0a9630 2024-04-25T00:01:47.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.669+0000 7f67d1709640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f67cc157340 0x7f67cc157790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:47.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.669+0000 7f67d1709640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f67cc157340 0x7f67cc157790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43238/0 (socket says 172.21.15.116:43238) 2024-04-25T00:01:47.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.669+0000 7f67d1709640 1 -- 172.21.15.116:0/1724933125 learned_addr learned my addr 172.21.15.116:0/1724933125 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:47.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.669+0000 7f67cbfff640 1 --2- 172.21.15.116:0/1724933125 >> v2:172.21.15.174:3300/0 conn(0x7f67cc0a9630 0x7f67cc156e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:47.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.669+0000 7f67d1709640 1 -- 172.21.15.116:0/1724933125 >> v2:172.21.15.174:3300/0 conn(0x7f67cc0a9630 msgr2=0x7f67cc156e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:47.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.669+0000 7f67d1709640 1 --2- 172.21.15.116:0/1724933125 >> v2:172.21.15.174:3300/0 conn(0x7f67cc0a9630 0x7f67cc156e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:47.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.669+0000 7f67d1709640 1 -- 172.21.15.116:0/1724933125 >> v1:172.21.15.174:6789/0 conn(0x7f67cc150d60 legacy=0x7f67cc0c2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:47.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.669+0000 7f67d1709640 1 -- 172.21.15.116:0/1724933125 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67cc0aa550 con 0x7f67cc157340 2024-04-25T00:01:47.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.669+0000 7f67d1709640 1 --2- 172.21.15.116:0/1724933125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67cc157340 0x7f67cc157790 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f67cc0c1350 tx=0x7f67c401c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0d9300c42610bb5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:47.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.669+0000 7f67cb7fe640 1 -- 172.21.15.116:0/1724933125 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c40a5650 con 0x7f67cc157340 2024-04-25T00:01:47.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.669+0000 7f67cb7fe640 1 -- 172.21.15.116:0/1724933125 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67c40a57f0 con 0x7f67cc157340 2024-04-25T00:01:47.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.669+0000 7f67cb7fe640 1 -- 172.21.15.116:0/1724933125 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c40a5bc0 con 0x7f67cc157340 2024-04-25T00:01:47.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.673+0000 7f67d316c640 1 -- 172.21.15.116:0/1724933125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67cc157340 msgr2=0x7f67cc157790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:47.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.673+0000 7f67d316c640 1 --2- 172.21.15.116:0/1724933125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67cc157340 0x7f67cc157790 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f67cc0c1350 tx=0x7f67c401c860 comp rx=0 tx=0).stop 2024-04-25T00:01:47.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.673+0000 7f67d316c640 1 -- 172.21.15.116:0/1724933125 shutdown_connections 2024-04-25T00:01:47.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.673+0000 7f67d316c640 1 --2- 172.21.15.116:0/1724933125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67cc157340 0x7f67cc157790 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:47.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.673+0000 7f67d316c640 1 --2- 172.21.15.116:0/1724933125 >> v2:172.21.15.174:3300/0 conn(0x7f67cc0a9630 0x7f67cc156e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:47.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.673+0000 7f67d316c640 1 -- 172.21.15.116:0/1724933125 >> 172.21.15.116:0/1724933125 conn(0x7f67cc0c3a20 msgr2=0x7f67cc0c3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:47.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.673+0000 7f67d316c640 1 -- 172.21.15.116:0/1724933125 shutdown_connections 2024-04-25T00:01:47.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.673+0000 7f67d316c640 1 -- 172.21.15.116:0/1724933125 wait complete. 2024-04-25T00:01:47.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.673+0000 7f67d316c640 1 Processor -- start 2024-04-25T00:01:47.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.673+0000 7f67d316c640 1 -- start start 2024-04-25T00:01:47.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.673+0000 7f67d316c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f67cc0a9630 0x7f67cc146a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:47.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.673+0000 7f67d316c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67cc150d60 0x7f67cc13f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:47.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.673+0000 7f67d316c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f67cc157340 0x7f67cc13ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:47.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.673+0000 7f67d316c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f67cc072150 con 0x7f67cc150d60 2024-04-25T00:01:47.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.673+0000 7f67d316c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f67cc071fd0 con 0x7f67cc157340 2024-04-25T00:01:47.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.673+0000 7f67d316c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f67cc160770 con 0x7f67cc0a9630 2024-04-25T00:01:47.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.673+0000 7f67cbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67cc150d60 0x7f67cc13f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:47.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.673+0000 7f67cbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67cc150d60 0x7f67cc13f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43240/0 (socket says 172.21.15.116:43240) 2024-04-25T00:01:47.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.673+0000 7f67cbfff640 1 -- 172.21.15.116:0/888402310 learned_addr learned my addr 172.21.15.116:0/888402310 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:47.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.673+0000 7f67d1709640 1 --2- 172.21.15.116:0/888402310 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f67cc157340 0x7f67cc13ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:47.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.673+0000 7f67d0f08640 1 --2- 172.21.15.116:0/888402310 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f67cc0a9630 0x7f67cc146a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:47.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.673+0000 7f67cbfff640 1 -- 172.21.15.116:0/888402310 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f67cc0a9630 msgr2=0x7f67cc146a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:47.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.673+0000 7f67cbfff640 1 --2- 172.21.15.116:0/888402310 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f67cc0a9630 0x7f67cc146a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:47.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.673+0000 7f67cbfff640 1 -- 172.21.15.116:0/888402310 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f67cc157340 msgr2=0x7f67cc13ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:47.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.673+0000 7f67cbfff640 1 --2- 172.21.15.116:0/888402310 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f67cc157340 0x7f67cc13ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:47.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.673+0000 7f67cbfff640 1 -- 172.21.15.116:0/888402310 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67cc1414c0 con 0x7f67cc150d60 2024-04-25T00:01:47.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.673+0000 7f67cbfff640 1 --2- 172.21.15.116:0/888402310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67cc150d60 0x7f67cc13f830 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f67c000c7f0 tx=0x7f67c000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:47.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.673+0000 7f67c97fa640 1 -- 172.21.15.116:0/888402310 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c0015070 con 0x7f67cc150d60 2024-04-25T00:01:47.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.673+0000 7f67c97fa640 1 -- 172.21.15.116:0/888402310 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67c0007820 con 0x7f67cc150d60 2024-04-25T00:01:47.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.673+0000 7f67d316c640 1 -- 172.21.15.116:0/888402310 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67cc1ef5c0 con 0x7f67cc150d60 2024-04-25T00:01:47.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.673+0000 7f67c97fa640 1 -- 172.21.15.116:0/888402310 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c0007c10 con 0x7f67cc150d60 2024-04-25T00:01:47.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.677+0000 7f67d316c640 1 -- 172.21.15.116:0/888402310 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67cc146f80 con 0x7f67cc150d60 2024-04-25T00:01:47.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.677+0000 7f67d316c640 1 -- 172.21.15.116:0/888402310 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67cc04f510 con 0x7f67cc150d60 2024-04-25T00:01:47.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.677+0000 7f67c97fa640 1 -- 172.21.15.116:0/888402310 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f67c0018070 con 0x7f67cc150d60 2024-04-25T00:01:47.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.677+0000 7f67c97fa640 1 --2- 172.21.15.116:0/888402310 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f67b80678c0 0x7f67b8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:47.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.677+0000 7f67c97fa640 1 -- 172.21.15.116:0/888402310 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6666+0+0 (secure 0 0 0) 0x7f67c0086040 con 0x7f67cc150d60 2024-04-25T00:01:47.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.677+0000 7f67d0f08640 1 --2- 172.21.15.116:0/888402310 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f67b80678c0 0x7f67b8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:47.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.677+0000 7f67d0f08640 1 --2- 172.21.15.116:0/888402310 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f67b80678c0 0x7f67b8069d80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f67bc00b810 tx=0x7f67bc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:47.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.681+0000 7f67c97fa640 1 -- 172.21.15.116:0/888402310 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f67c0056850 con 0x7f67cc150d60 2024-04-25T00:01:47.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.841+0000 7f67d316c640 1 -- 172.21.15.116:0/888402310 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"} v 0) -- 0x7f67cc071db0 con 0x7f67cc150d60 2024-04-25T00:01:47.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.845+0000 7f67c97fa640 1 -- 172.21.15.116:0/888402310 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"}]=0 v29) ==== 99+0+11 (secure 0 0 0) 0x7f67c0056150 con 0x7f67cc150d60 2024-04-25T00:01:47.847 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-04-25T00:01:47.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.845+0000 7f67d316c640 1 -- 172.21.15.116:0/888402310 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f67b80678c0 msgr2=0x7f67b8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:47.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.845+0000 7f67d316c640 1 --2- 172.21.15.116:0/888402310 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f67b80678c0 0x7f67b8069d80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f67bc00b810 tx=0x7f67bc008040 comp rx=0 tx=0).stop 2024-04-25T00:01:47.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.845+0000 7f67d316c640 1 -- 172.21.15.116:0/888402310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67cc150d60 msgr2=0x7f67cc13f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:47.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.845+0000 7f67d316c640 1 --2- 172.21.15.116:0/888402310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67cc150d60 0x7f67cc13f830 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f67c000c7f0 tx=0x7f67c000ccb0 comp rx=0 tx=0).stop 2024-04-25T00:01:47.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.845+0000 7f67d316c640 1 -- 172.21.15.116:0/888402310 shutdown_connections 2024-04-25T00:01:47.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.845+0000 7f67d316c640 1 --2- 172.21.15.116:0/888402310 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f67b80678c0 0x7f67b8069d80 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:47.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.845+0000 7f67d316c640 1 --2- 172.21.15.116:0/888402310 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f67cc157340 0x7f67cc13ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:47.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.845+0000 7f67d316c640 1 --2- 172.21.15.116:0/888402310 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f67cc150d60 0x7f67cc13f830 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:47.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.845+0000 7f67d316c640 1 --2- 172.21.15.116:0/888402310 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f67cc0a9630 0x7f67cc146a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:47.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.845+0000 7f67d316c640 1 -- 172.21.15.116:0/888402310 >> 172.21.15.116:0/888402310 conn(0x7f67cc0c3a20 msgr2=0x7f67cc0c1820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:47.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.845+0000 7f67d316c640 1 -- 172.21.15.116:0/888402310 shutdown_connections 2024-04-25T00:01:47.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.845+0000 7f67d316c640 1 -- 172.21.15.116:0/888402310 wait complete. 2024-04-25T00:01:47.866 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_data pg_num 2024-04-25T00:01:47.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.989+0000 7f1a9970f640 1 Processor -- start 2024-04-25T00:01:47.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.989+0000 7f1a9970f640 1 -- start start 2024-04-25T00:01:47.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.989+0000 7f1a9970f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1a940a9630 0x7f1a94156e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:47.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.989+0000 7f1a9970f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a940a9cb0 con 0x7f1a94157340 2024-04-25T00:01:47.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.989+0000 7f1a9970f640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a940a9f90 con 0x7f1a94150d60 2024-04-25T00:01:47.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.989+0000 7f1a9970f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f1a940aa270 con 0x7f1a940a9630 2024-04-25T00:01:47.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.989+0000 7f1a937fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f1a94157340 0x7f1a9415df50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:47520/0 (socket says 172.21.15.116:47520) 2024-04-25T00:01:47.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.989+0000 7f1a937fe640 1 -- 172.21.15.116:0/774362370 learned_addr learned my addr 172.21.15.116:0/774362370 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:47.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.989+0000 7f1a8bfff640 1 --2- 172.21.15.116:0/774362370 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1a940a9630 0x7f1a94156e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:47.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.989+0000 7f1a927fc640 1 -- 172.21.15.116:0/774362370 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3362622203 0 0) 0x7f1a940a9cb0 con 0x7f1a94157340 2024-04-25T00:01:47.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.989+0000 7f1a927fc640 1 -- 172.21.15.116:0/774362370 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a80003530 con 0x7f1a94157340 2024-04-25T00:01:47.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.989+0000 7f1a927fc640 1 -- 172.21.15.116:0/774362370 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3584247243 0 0) 0x7f1a80003530 con 0x7f1a94157340 2024-04-25T00:01:47.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.989+0000 7f1a927fc640 1 -- 172.21.15.116:0/774362370 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1a940a9630 msgr2=0x7f1a94156e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:47.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.989+0000 7f1a927fc640 1 --2- 172.21.15.116:0/774362370 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1a940a9630 0x7f1a94156e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:47.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.989+0000 7f1a927fc640 1 -- 172.21.15.116:0/774362370 >> v1:172.21.15.174:6789/0 conn(0x7f1a94150d60 legacy=0x7f1a940c2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:47.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.989+0000 7f1a927fc640 1 -- 172.21.15.116:0/774362370 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a940aa550 con 0x7f1a94157340 2024-04-25T00:01:47.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.989+0000 7f1a927fc640 1 -- 172.21.15.116:0/774362370 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f1a8c001a00 con 0x7f1a94157340 2024-04-25T00:01:47.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.989+0000 7f1a927fc640 1 -- 172.21.15.116:0/774362370 >> v1:172.21.15.116:6789/0 conn(0x7f1a94157340 legacy=0x7f1a9415df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:47.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.989+0000 7f1a927fc640 1 --2- 172.21.15.116:0/774362370 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1a940a9630 0x7f1a800039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:47.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.989+0000 7f1a927fc640 1 --2- 172.21.15.116:0/774362370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a80003f70 0x7f1a80004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:47.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.989+0000 7f1a927fc640 1 --2- 172.21.15.116:0/774362370 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a80004d50 0x7f1a800051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:47.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.989+0000 7f1a927fc640 1 -- 172.21.15.116:0/774362370 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1a940aa270 con 0x7f1a80003f70 2024-04-25T00:01:47.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.989+0000 7f1a927fc640 1 -- 172.21.15.116:0/774362370 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1a8000bc10 con 0x7f1a80004d50 2024-04-25T00:01:47.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.989+0000 7f1a927fc640 1 -- 172.21.15.116:0/774362370 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f1a8000be10 con 0x7f1a940a9630 2024-04-25T00:01:47.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.989+0000 7f1a8bfff640 1 --2- 172.21.15.116:0/774362370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a80003f70 0x7f1a80004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:47.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.989+0000 7f1a92ffd640 1 --2- 172.21.15.116:0/774362370 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1a940a9630 0x7f1a800039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:47.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.989+0000 7f1a8bfff640 1 -- 172.21.15.116:0/774362370 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1a940a9630 msgr2=0x7f1a800039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:47.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.989+0000 7f1a8bfff640 1 --2- 172.21.15.116:0/774362370 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1a940a9630 0x7f1a800039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:47.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.989+0000 7f1a8bfff640 1 -- 172.21.15.116:0/774362370 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a80004d50 msgr2=0x7f1a800051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:47.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.989+0000 7f1a937fe640 1 --2- 172.21.15.116:0/774362370 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a80004d50 0x7f1a800051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:47.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.989+0000 7f1a8bfff640 1 --2- 172.21.15.116:0/774362370 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a80004d50 0x7f1a800051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:47.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.989+0000 7f1a8bfff640 1 -- 172.21.15.116:0/774362370 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a80003530 con 0x7f1a80003f70 2024-04-25T00:01:48.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.993+0000 7f1a8bfff640 1 --2- 172.21.15.116:0/774362370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a80003f70 0x7f1a80004810 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f1a7c009ae0 tx=0x7f1a7c01ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c26605e06682aaa4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:48.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.993+0000 7f1a927fc640 1 -- 172.21.15.116:0/774362370 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a7c0b2070 con 0x7f1a80003f70 2024-04-25T00:01:48.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.993+0000 7f1a927fc640 1 -- 172.21.15.116:0/774362370 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a7c0a5520 con 0x7f1a80003f70 2024-04-25T00:01:48.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.993+0000 7f1a927fc640 1 -- 172.21.15.116:0/774362370 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a7c0096b0 con 0x7f1a80003f70 2024-04-25T00:01:48.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.993+0000 7f1a9970f640 1 -- 172.21.15.116:0/774362370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a80003f70 msgr2=0x7f1a80004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:48.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.993+0000 7f1a9970f640 1 --2- 172.21.15.116:0/774362370 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a80003f70 0x7f1a80004810 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f1a7c009ae0 tx=0x7f1a7c01ca40 comp rx=0 tx=0).stop 2024-04-25T00:01:48.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.993+0000 7f1a937fe640 1 -- 172.21.15.116:0/774362370 reap_dead start 2024-04-25T00:01:48.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.993+0000 7f1a9970f640 1 -- 172.21.15.116:0/774362370 shutdown_connections 2024-04-25T00:01:48.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.993+0000 7f1a9970f640 1 -- 172.21.15.116:0/774362370 >> 172.21.15.116:0/774362370 conn(0x7f1a940c3a20 msgr2=0x7f1a940c3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:48.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.993+0000 7f1a9970f640 1 -- 172.21.15.116:0/774362370 shutdown_connections 2024-04-25T00:01:48.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.993+0000 7f1a9970f640 1 -- 172.21.15.116:0/774362370 wait complete. 2024-04-25T00:01:48.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.993+0000 7f1a9970f640 1 Processor -- start 2024-04-25T00:01:48.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.993+0000 7f1a9970f640 1 -- start start 2024-04-25T00:01:48.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.993+0000 7f1a9970f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1a94140c90 0x7f1a941549c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:48.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.993+0000 7f1a9970f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a94144de0 0x7f1a941ef810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:48.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.993+0000 7f1a9970f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a94151440 0x7f1a941f23b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:48.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.993+0000 7f1a9970f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1a94157b60 con 0x7f1a94144de0 2024-04-25T00:01:48.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.993+0000 7f1a9970f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1a940ab7a0 con 0x7f1a94151440 2024-04-25T00:01:48.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.993+0000 7f1a9970f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f1a94071fd0 con 0x7f1a94140c90 2024-04-25T00:01:48.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.993+0000 7f1a8bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a94144de0 0x7f1a941ef810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:48.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.993+0000 7f1a8bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a94144de0 0x7f1a941ef810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43260/0 (socket says 172.21.15.116:43260) 2024-04-25T00:01:48.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.993+0000 7f1a8bfff640 1 -- 172.21.15.116:0/2843170023 learned_addr learned my addr 172.21.15.116:0/2843170023 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:48.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.997+0000 7f1a937fe640 1 --2- 172.21.15.116:0/2843170023 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a94151440 0x7f1a941f23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:48.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.997+0000 7f1a92ffd640 1 --2- 172.21.15.116:0/2843170023 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1a94140c90 0x7f1a941549c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:48.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.997+0000 7f1a8bfff640 1 -- 172.21.15.116:0/2843170023 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1a94140c90 msgr2=0x7f1a941549c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:48.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.997+0000 7f1a8bfff640 1 --2- 172.21.15.116:0/2843170023 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1a94140c90 0x7f1a941549c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:48.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.997+0000 7f1a8bfff640 1 -- 172.21.15.116:0/2843170023 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a94151440 msgr2=0x7f1a941f23b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:48.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.997+0000 7f1a8bfff640 1 --2- 172.21.15.116:0/2843170023 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a94151440 0x7f1a941f23b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:48.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.997+0000 7f1a8bfff640 1 -- 172.21.15.116:0/2843170023 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a941505e0 con 0x7f1a94144de0 2024-04-25T00:01:48.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.997+0000 7f1a92ffd640 1 --2- 172.21.15.116:0/2843170023 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1a94140c90 0x7f1a941549c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:48.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.997+0000 7f1a8bfff640 1 --2- 172.21.15.116:0/2843170023 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a94144de0 0x7f1a941ef810 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f1a7c01c870 tx=0x7f1a7c0197e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:48.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.997+0000 7f1a8b7fe640 1 -- 172.21.15.116:0/2843170023 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a7c0b2040 con 0x7f1a94144de0 2024-04-25T00:01:48.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.997+0000 7f1a8b7fe640 1 -- 172.21.15.116:0/2843170023 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a7c00f000 con 0x7f1a94144de0 2024-04-25T00:01:48.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.997+0000 7f1a9970f640 1 -- 172.21.15.116:0/2843170023 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a94156750 con 0x7f1a94144de0 2024-04-25T00:01:48.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.997+0000 7f1a8b7fe640 1 -- 172.21.15.116:0/2843170023 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a7c004020 con 0x7f1a94144de0 2024-04-25T00:01:48.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.997+0000 7f1a9970f640 1 -- 172.21.15.116:0/2843170023 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a94152dc0 con 0x7f1a94144de0 2024-04-25T00:01:48.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.997+0000 7f1a9970f640 1 -- 172.21.15.116:0/2843170023 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a9404f510 con 0x7f1a94144de0 2024-04-25T00:01:48.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.997+0000 7f1a8b7fe640 1 -- 172.21.15.116:0/2843170023 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1a7c0bf030 con 0x7f1a94144de0 2024-04-25T00:01:48.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.997+0000 7f1a8b7fe640 1 --2- 172.21.15.116:0/2843170023 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1a740678c0 0x7f1a74069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:48.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.997+0000 7f1a8b7fe640 1 -- 172.21.15.116:0/2843170023 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6666+0+0 (secure 0 0 0) 0x7f1a7c11fbb0 con 0x7f1a94144de0 2024-04-25T00:01:48.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.997+0000 7f1a92ffd640 1 --2- 172.21.15.116:0/2843170023 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1a740678c0 0x7f1a74069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:48.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:47.997+0000 7f1a92ffd640 1 --2- 172.21.15.116:0/2843170023 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1a740678c0 0x7f1a74069d80 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1a840016e0 tx=0x7f1a84009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:48.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.001+0000 7f1a8b7fe640 1 -- 172.21.15.116:0/2843170023 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f1a7c0c6060 con 0x7f1a94144de0 2024-04-25T00:01:48.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.149+0000 7f1a9970f640 1 -- 172.21.15.116:0/2843170023 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"} v 0) -- 0x7f1a9415f450 con 0x7f1a94144de0 2024-04-25T00:01:48.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.149+0000 7f1a8b7fe640 1 -- 172.21.15.116:0/2843170023 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"}]=0 v29) ==== 95+0+11 (secure 0 0 0) 0x7f1a7c016780 con 0x7f1a94144de0 2024-04-25T00:01:48.154 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-04-25T00:01:48.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.153+0000 7f1a9970f640 1 -- 172.21.15.116:0/2843170023 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1a740678c0 msgr2=0x7f1a74069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:48.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.153+0000 7f1a9970f640 1 --2- 172.21.15.116:0/2843170023 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1a740678c0 0x7f1a74069d80 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1a840016e0 tx=0x7f1a84009040 comp rx=0 tx=0).stop 2024-04-25T00:01:48.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.153+0000 7f1a9970f640 1 -- 172.21.15.116:0/2843170023 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a94144de0 msgr2=0x7f1a941ef810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:48.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.153+0000 7f1a9970f640 1 --2- 172.21.15.116:0/2843170023 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a94144de0 0x7f1a941ef810 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f1a7c01c870 tx=0x7f1a7c0197e0 comp rx=0 tx=0).stop 2024-04-25T00:01:48.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.153+0000 7f1a9970f640 1 -- 172.21.15.116:0/2843170023 shutdown_connections 2024-04-25T00:01:48.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.153+0000 7f1a9970f640 1 --2- 172.21.15.116:0/2843170023 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1a740678c0 0x7f1a74069d80 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:48.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.153+0000 7f1a9970f640 1 --2- 172.21.15.116:0/2843170023 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1a94151440 0x7f1a941f23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:48.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.153+0000 7f1a9970f640 1 --2- 172.21.15.116:0/2843170023 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1a94144de0 0x7f1a941ef810 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:48.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.153+0000 7f1a9970f640 1 --2- 172.21.15.116:0/2843170023 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1a94140c90 0x7f1a941549c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:48.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.153+0000 7f1a9970f640 1 -- 172.21.15.116:0/2843170023 >> 172.21.15.116:0/2843170023 conn(0x7f1a940c3a20 msgr2=0x7f1a94153ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:48.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.153+0000 7f1a9970f640 1 -- 172.21.15.116:0/2843170023 shutdown_connections 2024-04-25T00:01:48.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.153+0000 7f1a9970f640 1 -- 172.21.15.116:0/2843170023 wait complete. 2024-04-25T00:01:48.170 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_metadata pg_num 2024-04-25T00:01:48.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.289+0000 7fc98e24f640 1 Processor -- start 2024-04-25T00:01:48.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.289+0000 7fc98e24f640 1 -- start start 2024-04-25T00:01:48.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.289+0000 7fc98e24f640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fc988148f20 0x7fc9880c2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:48.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.289+0000 7fc98e24f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc988149300 con 0x7fc98806d000 2024-04-25T00:01:48.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.289+0000 7fc98e24f640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9880a95d0 con 0x7fc98806d510 2024-04-25T00:01:48.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.289+0000 7fc98e24f640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fc988153350 con 0x7fc988148f20 2024-04-25T00:01:48.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.289+0000 7fc987fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fc98806d000 0x7fc98815cdd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:47536/0 (socket says 172.21.15.116:47536) 2024-04-25T00:01:48.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.289+0000 7fc987fff640 1 -- 172.21.15.116:0/470204153 learned_addr learned my addr 172.21.15.116:0/470204153 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:48.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.289+0000 7fc9877fe640 1 --2- 172.21.15.116:0/470204153 >> v2:172.21.15.174:3300/0 conn(0x7fc988148f20 0x7fc9880c2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:48.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.289+0000 7fc9867fc640 1 -- 172.21.15.116:0/470204153 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1670669268 0 0) 0x7fc9800014a0 con 0x7fc98806d000 2024-04-25T00:01:48.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.289+0000 7fc9867fc640 1 -- 172.21.15.116:0/470204153 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc970005740 con 0x7fc98806d000 2024-04-25T00:01:48.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.289+0000 7fc9867fc640 1 -- 172.21.15.116:0/470204153 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3028041819 0 0) 0x7fc970005740 con 0x7fc98806d000 2024-04-25T00:01:48.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.289+0000 7fc9867fc640 1 -- 172.21.15.116:0/470204153 >> v2:172.21.15.174:3300/0 conn(0x7fc988148f20 msgr2=0x7fc9880c2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:48.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.289+0000 7fc9867fc640 1 --2- 172.21.15.116:0/470204153 >> v2:172.21.15.174:3300/0 conn(0x7fc988148f20 0x7fc9880c2930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:48.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.289+0000 7fc9867fc640 1 -- 172.21.15.116:0/470204153 >> v1:172.21.15.174:6789/0 conn(0x7fc98806d510 legacy=0x7fc988072500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:48.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.289+0000 7fc9867fc640 1 -- 172.21.15.116:0/470204153 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc988153070 con 0x7fc98806d000 2024-04-25T00:01:48.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.289+0000 7fc9867fc640 1 -- 172.21.15.116:0/470204153 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7fc980001bd0 con 0x7fc98806d000 2024-04-25T00:01:48.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.289+0000 7fc9867fc640 1 -- 172.21.15.116:0/470204153 >> v1:172.21.15.116:6789/0 conn(0x7fc98806d000 legacy=0x7fc98815cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:48.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.289+0000 7fc9867fc640 1 --2- 172.21.15.116:0/470204153 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc970005bd0 0x7fc970006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:48.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.289+0000 7fc9867fc640 1 --2- 172.21.15.116:0/470204153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc970006a20 0x7fc970006ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:48.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.289+0000 7fc9867fc640 1 --2- 172.21.15.116:0/470204153 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc970007500 0x7fc97000dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:48.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.289+0000 7fc9867fc640 1 -- 172.21.15.116:0/470204153 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc988153350 con 0x7fc970006a20 2024-04-25T00:01:48.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.289+0000 7fc9867fc640 1 -- 172.21.15.116:0/470204153 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc97000e3d0 con 0x7fc970007500 2024-04-25T00:01:48.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.289+0000 7fc9867fc640 1 -- 172.21.15.116:0/470204153 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fc97000e5d0 con 0x7fc970005bd0 2024-04-25T00:01:48.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.289+0000 7fc986ffd640 1 --2- 172.21.15.116:0/470204153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc970006a20 0x7fc970006ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:48.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.289+0000 7fc9877fe640 1 --2- 172.21.15.116:0/470204153 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc970005bd0 0x7fc970006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:48.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.289+0000 7fc987fff640 1 --2- 172.21.15.116:0/470204153 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc970007500 0x7fc97000dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:48.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.289+0000 7fc986ffd640 1 -- 172.21.15.116:0/470204153 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc970005bd0 msgr2=0x7fc970006010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:48.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.289+0000 7fc986ffd640 1 --2- 172.21.15.116:0/470204153 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc970005bd0 0x7fc970006010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:48.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.289+0000 7fc986ffd640 1 -- 172.21.15.116:0/470204153 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc970007500 msgr2=0x7fc97000dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:48.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.289+0000 7fc986ffd640 1 --2- 172.21.15.116:0/470204153 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc970007500 0x7fc97000dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:48.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.289+0000 7fc986ffd640 1 -- 172.21.15.116:0/470204153 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc970005740 con 0x7fc970006a20 2024-04-25T00:01:48.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.289+0000 7fc987fff640 1 --2- 172.21.15.116:0/470204153 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc970007500 0x7fc97000dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:48.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.289+0000 7fc987fff640 1 -- 172.21.15.116:0/470204153 reap_dead start 2024-04-25T00:01:48.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.289+0000 7fc9877fe640 1 --2- 172.21.15.116:0/470204153 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc970005bd0 0x7fc970006010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:48.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.293+0000 7fc986ffd640 1 --2- 172.21.15.116:0/470204153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc970006a20 0x7fc970006ea0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fc97c001780 tx=0x7fc97c01c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40400b01d8263b6f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:48.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.293+0000 7fc9867fc640 1 -- 172.21.15.116:0/470204153 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc97c0013a0 con 0x7fc970006a20 2024-04-25T00:01:48.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.293+0000 7fc9867fc640 1 -- 172.21.15.116:0/470204153 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc97c0050d0 con 0x7fc970006a20 2024-04-25T00:01:48.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.293+0000 7fc9867fc640 1 -- 172.21.15.116:0/470204153 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc97c003600 con 0x7fc970006a20 2024-04-25T00:01:48.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.293+0000 7fc98e24f640 1 -- 172.21.15.116:0/470204153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc970006a20 msgr2=0x7fc970006ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:48.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.293+0000 7fc98e24f640 1 --2- 172.21.15.116:0/470204153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc970006a20 0x7fc970006ea0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fc97c001780 tx=0x7fc97c01c400 comp rx=0 tx=0).stop 2024-04-25T00:01:48.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.293+0000 7fc98e24f640 1 -- 172.21.15.116:0/470204153 shutdown_connections 2024-04-25T00:01:48.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.293+0000 7fc98e24f640 1 --2- 172.21.15.116:0/470204153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc970006a20 0x7fc970006ea0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:48.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.293+0000 7fc98e24f640 1 -- 172.21.15.116:0/470204153 >> 172.21.15.116:0/470204153 conn(0x7fc9880c3a20 msgr2=0x7fc9880c3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:48.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.293+0000 7fc98e24f640 1 -- 172.21.15.116:0/470204153 shutdown_connections 2024-04-25T00:01:48.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.293+0000 7fc98e24f640 1 -- 172.21.15.116:0/470204153 wait complete. 2024-04-25T00:01:48.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.293+0000 7fc98e24f640 1 Processor -- start 2024-04-25T00:01:48.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.293+0000 7fc98e24f640 1 -- start start 2024-04-25T00:01:48.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.293+0000 7fc98e24f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc970006a20 0x7fc9880c5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:48.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.293+0000 7fc98e24f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc9880aafc0 0x7fc988141700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:48.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.293+0000 7fc98e24f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc988141c40 0x7fc988146c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:48.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.293+0000 7fc98e24f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc988072680 con 0x7fc970006a20 2024-04-25T00:01:48.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.293+0000 7fc98e24f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc988072500 con 0x7fc9880aafc0 2024-04-25T00:01:48.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.293+0000 7fc98e24f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fc988072800 con 0x7fc988141c40 2024-04-25T00:01:48.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.293+0000 7fc9877fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc970006a20 0x7fc9880c5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:48.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.293+0000 7fc9877fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc970006a20 0x7fc9880c5ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43276/0 (socket says 172.21.15.116:43276) 2024-04-25T00:01:48.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.293+0000 7fc9877fe640 1 -- 172.21.15.116:0/2653796830 learned_addr learned my addr 172.21.15.116:0/2653796830 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:48.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.293+0000 7fc986ffd640 1 --2- 172.21.15.116:0/2653796830 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc9880aafc0 0x7fc988141700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:48.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.293+0000 7fc987fff640 1 --2- 172.21.15.116:0/2653796830 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc988141c40 0x7fc988146c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:48.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.293+0000 7fc9877fe640 1 -- 172.21.15.116:0/2653796830 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc988141c40 msgr2=0x7fc988146c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:48.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.293+0000 7fc9877fe640 1 --2- 172.21.15.116:0/2653796830 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc988141c40 0x7fc988146c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:48.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.293+0000 7fc9877fe640 1 -- 172.21.15.116:0/2653796830 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc9880aafc0 msgr2=0x7fc988141700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:48.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.293+0000 7fc9877fe640 1 --2- 172.21.15.116:0/2653796830 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc9880aafc0 0x7fc988141700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:48.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.293+0000 7fc9877fe640 1 -- 172.21.15.116:0/2653796830 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc988154a20 con 0x7fc970006a20 2024-04-25T00:01:48.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.297+0000 7fc9877fe640 1 --2- 172.21.15.116:0/2653796830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc970006a20 0x7fc9880c5ef0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fc978010820 tx=0x7fc978010ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:48.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.297+0000 7fc96ffff640 1 -- 172.21.15.116:0/2653796830 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc97800f040 con 0x7fc970006a20 2024-04-25T00:01:48.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.297+0000 7fc96ffff640 1 -- 172.21.15.116:0/2653796830 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc978004220 con 0x7fc970006a20 2024-04-25T00:01:48.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.297+0000 7fc98e24f640 1 -- 172.21.15.116:0/2653796830 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc988148250 con 0x7fc970006a20 2024-04-25T00:01:48.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.297+0000 7fc96ffff640 1 -- 172.21.15.116:0/2653796830 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc97800e900 con 0x7fc970006a20 2024-04-25T00:01:48.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.297+0000 7fc98e24f640 1 -- 172.21.15.116:0/2653796830 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9881ef650 con 0x7fc970006a20 2024-04-25T00:01:48.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.297+0000 7fc98e24f640 1 -- 172.21.15.116:0/2653796830 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc98804f510 con 0x7fc970006a20 2024-04-25T00:01:48.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.297+0000 7fc96ffff640 1 -- 172.21.15.116:0/2653796830 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc97800eaa0 con 0x7fc970006a20 2024-04-25T00:01:48.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.297+0000 7fc96ffff640 1 --2- 172.21.15.116:0/2653796830 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc9740678f0 0x7fc974069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:48.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.297+0000 7fc96ffff640 1 -- 172.21.15.116:0/2653796830 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6666+0+0 (secure 0 0 0) 0x7fc9780860d0 con 0x7fc970006a20 2024-04-25T00:01:48.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.297+0000 7fc986ffd640 1 --2- 172.21.15.116:0/2653796830 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc9740678f0 0x7fc974069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:48.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.297+0000 7fc986ffd640 1 --2- 172.21.15.116:0/2653796830 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc9740678f0 0x7fc974069db0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc97c001750 tx=0x7fc97c0011a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:48.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.301+0000 7fc96ffff640 1 -- 172.21.15.116:0/2653796830 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fc978018030 con 0x7fc970006a20 2024-04-25T00:01:48.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.457+0000 7fc98e24f640 1 -- 172.21.15.116:0/2653796830 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"} v 0) -- 0x7fc988142020 con 0x7fc970006a20 2024-04-25T00:01:48.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.457+0000 7fc96ffff640 1 -- 172.21.15.116:0/2653796830 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"}]=0 v29) ==== 99+0+11 (secure 0 0 0) 0x7fc9780568e0 con 0x7fc970006a20 2024-04-25T00:01:48.462 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-04-25T00:01:48.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.461+0000 7fc98e24f640 1 -- 172.21.15.116:0/2653796830 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc9740678f0 msgr2=0x7fc974069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:48.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.461+0000 7fc98e24f640 1 --2- 172.21.15.116:0/2653796830 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc9740678f0 0x7fc974069db0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc97c001750 tx=0x7fc97c0011a0 comp rx=0 tx=0).stop 2024-04-25T00:01:48.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.461+0000 7fc98e24f640 1 -- 172.21.15.116:0/2653796830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc970006a20 msgr2=0x7fc9880c5ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:48.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.461+0000 7fc98e24f640 1 --2- 172.21.15.116:0/2653796830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc970006a20 0x7fc9880c5ef0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fc978010820 tx=0x7fc978010ce0 comp rx=0 tx=0).stop 2024-04-25T00:01:48.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.461+0000 7fc98e24f640 1 -- 172.21.15.116:0/2653796830 shutdown_connections 2024-04-25T00:01:48.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.461+0000 7fc98e24f640 1 --2- 172.21.15.116:0/2653796830 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc9740678f0 0x7fc974069db0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:48.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.461+0000 7fc98e24f640 1 --2- 172.21.15.116:0/2653796830 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc988141c40 0x7fc988146c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:48.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.461+0000 7fc98e24f640 1 --2- 172.21.15.116:0/2653796830 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc9880aafc0 0x7fc988141700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:48.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.461+0000 7fc98e24f640 1 --2- 172.21.15.116:0/2653796830 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc970006a20 0x7fc9880c5ef0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:48.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.461+0000 7fc98e24f640 1 -- 172.21.15.116:0/2653796830 >> 172.21.15.116:0/2653796830 conn(0x7fc9880c3a20 msgr2=0x7fc988072b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:48.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.461+0000 7fc98e24f640 1 -- 172.21.15.116:0/2653796830 shutdown_connections 2024-04-25T00:01:48.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.461+0000 7fc98e24f640 1 -- 172.21.15.116:0/2653796830 wait complete. 2024-04-25T00:01:48.480 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_data pg_num 2024-04-25T00:01:48.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.593+0000 7f2f9c2e0640 1 Processor -- start 2024-04-25T00:01:48.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.593+0000 7f2f9c2e0640 1 -- start start 2024-04-25T00:01:48.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.597+0000 7f2f9c2e0640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2f9406d000 0x7f2f9415dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:48.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.597+0000 7f2f9c2e0640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f94149300 con 0x7f2f9406d510 2024-04-25T00:01:48.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.597+0000 7f2f9c2e0640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f940a95d0 con 0x7f2f94148f20 2024-04-25T00:01:48.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.597+0000 7f2f9c2e0640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f2f94153350 con 0x7f2f9406d000 2024-04-25T00:01:48.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.597+0000 7f2f9987b640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f2f9406d510 0x7f2f94072500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:47550/0 (socket says 172.21.15.116:47550) 2024-04-25T00:01:48.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.597+0000 7f2f9987b640 1 -- 172.21.15.116:0/2182453245 learned_addr learned my addr 172.21.15.116:0/2182453245 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:48.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.597+0000 7f2f9a87d640 1 --2- 172.21.15.116:0/2182453245 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2f9406d000 0x7f2f9415dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:48.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.597+0000 7f2f9907a640 1 -- 172.21.15.116:0/2182453245 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1308745999 0 0) 0x7f2f84001120 con 0x7f2f9406d510 2024-04-25T00:01:48.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.597+0000 7f2f9907a640 1 -- 172.21.15.116:0/2182453245 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f80003530 con 0x7f2f9406d510 2024-04-25T00:01:48.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.597+0000 7f2f9907a640 1 -- 172.21.15.116:0/2182453245 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 670341121 0 0) 0x7f2f80003530 con 0x7f2f9406d510 2024-04-25T00:01:48.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.597+0000 7f2f9907a640 1 -- 172.21.15.116:0/2182453245 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2f9406d000 msgr2=0x7f2f9415dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:48.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.597+0000 7f2f9907a640 1 --2- 172.21.15.116:0/2182453245 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2f9406d000 0x7f2f9415dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:48.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.597+0000 7f2f9907a640 1 -- 172.21.15.116:0/2182453245 >> v1:172.21.15.174:6789/0 conn(0x7f2f94148f20 legacy=0x7f2f940c2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:48.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.597+0000 7f2f9907a640 1 -- 172.21.15.116:0/2182453245 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f94153070 con 0x7f2f9406d510 2024-04-25T00:01:48.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.597+0000 7f2f9907a640 1 -- 172.21.15.116:0/2182453245 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f2f84001850 con 0x7f2f9406d510 2024-04-25T00:01:48.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.597+0000 7f2f9907a640 1 -- 172.21.15.116:0/2182453245 >> v1:172.21.15.116:6789/0 conn(0x7f2f9406d510 legacy=0x7f2f94072500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:48.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.597+0000 7f2f9907a640 1 --2- 172.21.15.116:0/2182453245 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2f9406d000 0x7f2f800039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:48.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.597+0000 7f2f9907a640 1 --2- 172.21.15.116:0/2182453245 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f80003f70 0x7f2f80004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:48.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.597+0000 7f2f9907a640 1 --2- 172.21.15.116:0/2182453245 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f80004d50 0x7f2f800051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:48.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.597+0000 7f2f9907a640 1 -- 172.21.15.116:0/2182453245 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2f94153350 con 0x7f2f80004d50 2024-04-25T00:01:48.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.597+0000 7f2f9907a640 1 -- 172.21.15.116:0/2182453245 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2f8000bc10 con 0x7f2f80003f70 2024-04-25T00:01:48.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.597+0000 7f2f9907a640 1 -- 172.21.15.116:0/2182453245 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f2f8000e020 con 0x7f2f9406d000 2024-04-25T00:01:48.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.597+0000 7f2f9a87d640 1 --2- 172.21.15.116:0/2182453245 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f80004d50 0x7f2f800051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:48.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.597+0000 7f2f9a07c640 1 --2- 172.21.15.116:0/2182453245 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2f9406d000 0x7f2f800039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:48.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.597+0000 7f2f9987b640 1 --2- 172.21.15.116:0/2182453245 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f80003f70 0x7f2f80004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:48.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.597+0000 7f2f9a87d640 1 -- 172.21.15.116:0/2182453245 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2f9406d000 msgr2=0x7f2f800039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:48.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.597+0000 7f2f9a87d640 1 --2- 172.21.15.116:0/2182453245 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2f9406d000 0x7f2f800039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:48.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.597+0000 7f2f9a87d640 1 -- 172.21.15.116:0/2182453245 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f80003f70 msgr2=0x7f2f80004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:48.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.597+0000 7f2f9a87d640 1 --2- 172.21.15.116:0/2182453245 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f80003f70 0x7f2f80004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:48.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.597+0000 7f2f9a87d640 1 -- 172.21.15.116:0/2182453245 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f80003530 con 0x7f2f80004d50 2024-04-25T00:01:48.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.597+0000 7f2f9a87d640 1 --2- 172.21.15.116:0/2182453245 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f80004d50 0x7f2f800051d0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c009fa0 tx=0x7f2f8c01cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93c92f05ab39cb39 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:48.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.597+0000 7f2f9907a640 1 -- 172.21.15.116:0/2182453245 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f8c0b2070 con 0x7f2f80004d50 2024-04-25T00:01:48.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.597+0000 7f2f9907a640 1 -- 172.21.15.116:0/2182453245 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f8c0a5b00 con 0x7f2f80004d50 2024-04-25T00:01:48.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.597+0000 7f2f9907a640 1 -- 172.21.15.116:0/2182453245 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f8c009bf0 con 0x7f2f80004d50 2024-04-25T00:01:48.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.601+0000 7f2f9c2e0640 1 -- 172.21.15.116:0/2182453245 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f80004d50 msgr2=0x7f2f800051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:48.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.601+0000 7f2f9c2e0640 1 --2- 172.21.15.116:0/2182453245 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f80004d50 0x7f2f800051d0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c009fa0 tx=0x7f2f8c01cc40 comp rx=0 tx=0).stop 2024-04-25T00:01:48.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.601+0000 7f2f9a87d640 1 -- 172.21.15.116:0/2182453245 reap_dead start 2024-04-25T00:01:48.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.601+0000 7f2f9c2e0640 1 -- 172.21.15.116:0/2182453245 shutdown_connections 2024-04-25T00:01:48.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.601+0000 7f2f9c2e0640 1 -- 172.21.15.116:0/2182453245 >> 172.21.15.116:0/2182453245 conn(0x7f2f940c3a20 msgr2=0x7f2f940c3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:48.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.601+0000 7f2f9c2e0640 1 -- 172.21.15.116:0/2182453245 shutdown_connections 2024-04-25T00:01:48.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.601+0000 7f2f9c2e0640 1 -- 172.21.15.116:0/2182453245 wait complete. 2024-04-25T00:01:48.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.601+0000 7f2f9c2e0640 1 Processor -- start 2024-04-25T00:01:48.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.601+0000 7f2f9c2e0640 1 -- start start 2024-04-25T00:01:48.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.601+0000 7f2f9c2e0640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2f80004d50 0x7f2f94151570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:48.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.601+0000 7f2f9c2e0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f94149df0 0x7f2f941e9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:48.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.601+0000 7f2f9c2e0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f941ea420 0x7f2f941ed350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:48.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.601+0000 7f2f9c2e0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2f940c3850 con 0x7f2f94149df0 2024-04-25T00:01:48.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.601+0000 7f2f9c2e0640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2f940c36d0 con 0x7f2f941ea420 2024-04-25T00:01:48.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.601+0000 7f2f9c2e0640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f2f940715b0 con 0x7f2f80004d50 2024-04-25T00:01:48.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.601+0000 7f2f9987b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f94149df0 0x7f2f941e9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:48.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.601+0000 7f2f9987b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f94149df0 0x7f2f941e9ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43298/0 (socket says 172.21.15.116:43298) 2024-04-25T00:01:48.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.601+0000 7f2f9987b640 1 -- 172.21.15.116:0/3978449997 learned_addr learned my addr 172.21.15.116:0/3978449997 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:48.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.601+0000 7f2f9a07c640 1 --2- 172.21.15.116:0/3978449997 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2f80004d50 0x7f2f94151570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:48.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.601+0000 7f2f9a87d640 1 --2- 172.21.15.116:0/3978449997 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f941ea420 0x7f2f941ed350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:48.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.601+0000 7f2f9987b640 1 -- 172.21.15.116:0/3978449997 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2f80004d50 msgr2=0x7f2f94151570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:48.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.601+0000 7f2f9987b640 1 --2- 172.21.15.116:0/3978449997 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2f80004d50 0x7f2f94151570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:48.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.601+0000 7f2f9987b640 1 -- 172.21.15.116:0/3978449997 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f941ea420 msgr2=0x7f2f941ed350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:48.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.601+0000 7f2f9987b640 1 --2- 172.21.15.116:0/3978449997 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f941ea420 0x7f2f941ed350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:48.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.601+0000 7f2f9987b640 1 -- 172.21.15.116:0/3978449997 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f9414a2f0 con 0x7f2f94149df0 2024-04-25T00:01:48.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.601+0000 7f2f9a87d640 1 --2- 172.21.15.116:0/3978449997 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f941ea420 0x7f2f941ed350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:48.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.601+0000 7f2f9987b640 1 --2- 172.21.15.116:0/3978449997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f94149df0 0x7f2f941e9ee0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f2f84007330 tx=0x7f2f840077f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:48.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.601+0000 7f2f8affd640 1 -- 172.21.15.116:0/3978449997 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f8400e920 con 0x7f2f94149df0 2024-04-25T00:01:48.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.601+0000 7f2f8affd640 1 -- 172.21.15.116:0/3978449997 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f84018070 con 0x7f2f94149df0 2024-04-25T00:01:48.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.601+0000 7f2f9c2e0640 1 -- 172.21.15.116:0/3978449997 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f9414ba10 con 0x7f2f94149df0 2024-04-25T00:01:48.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.601+0000 7f2f8affd640 1 -- 172.21.15.116:0/3978449997 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f8400ece0 con 0x7f2f94149df0 2024-04-25T00:01:48.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.601+0000 7f2f9c2e0640 1 -- 172.21.15.116:0/3978449997 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f9414bfd0 con 0x7f2f94149df0 2024-04-25T00:01:48.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.605+0000 7f2f9c2e0640 1 -- 172.21.15.116:0/3978449997 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f9404f510 con 0x7f2f94149df0 2024-04-25T00:01:48.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.605+0000 7f2f8affd640 1 -- 172.21.15.116:0/3978449997 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2f8401d460 con 0x7f2f94149df0 2024-04-25T00:01:48.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.605+0000 7f2f8affd640 1 --2- 172.21.15.116:0/3978449997 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2f780678f0 0x7f2f78069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:48.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.605+0000 7f2f8affd640 1 -- 172.21.15.116:0/3978449997 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6666+0+0 (secure 0 0 0) 0x7f2f84086060 con 0x7f2f94149df0 2024-04-25T00:01:48.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.605+0000 7f2f9a07c640 1 --2- 172.21.15.116:0/3978449997 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2f780678f0 0x7f2f78069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:48.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.605+0000 7f2f9a07c640 1 --2- 172.21.15.116:0/3978449997 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2f780678f0 0x7f2f78069db0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2f94149a50 tx=0x7f2f90009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:48.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.609+0000 7f2f8affd640 1 -- 172.21.15.116:0/3978449997 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f2f94149300 con 0x7f2f94149df0 2024-04-25T00:01:48.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.765+0000 7f2f9c2e0640 1 -- 172.21.15.116:0/3978449997 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"} v 0) -- 0x7f2f940c34b0 con 0x7f2f94149df0 2024-04-25T00:01:48.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.769+0000 7f2f8affd640 1 -- 172.21.15.116:0/3978449997 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"}]=0 v29) ==== 95+0+11 (secure 0 0 0) 0x7f2f840567e0 con 0x7f2f94149df0 2024-04-25T00:01:48.771 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-04-25T00:01:48.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.769+0000 7f2f9c2e0640 1 -- 172.21.15.116:0/3978449997 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2f780678f0 msgr2=0x7f2f78069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:48.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.769+0000 7f2f9c2e0640 1 --2- 172.21.15.116:0/3978449997 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2f780678f0 0x7f2f78069db0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2f94149a50 tx=0x7f2f90009040 comp rx=0 tx=0).stop 2024-04-25T00:01:48.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.769+0000 7f2f9c2e0640 1 -- 172.21.15.116:0/3978449997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f94149df0 msgr2=0x7f2f941e9ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:48.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.769+0000 7f2f9c2e0640 1 --2- 172.21.15.116:0/3978449997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f94149df0 0x7f2f941e9ee0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f2f84007330 tx=0x7f2f840077f0 comp rx=0 tx=0).stop 2024-04-25T00:01:48.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.769+0000 7f2f9c2e0640 1 -- 172.21.15.116:0/3978449997 shutdown_connections 2024-04-25T00:01:48.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.769+0000 7f2f9c2e0640 1 --2- 172.21.15.116:0/3978449997 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2f780678f0 0x7f2f78069db0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:48.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.769+0000 7f2f9c2e0640 1 --2- 172.21.15.116:0/3978449997 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2f941ea420 0x7f2f941ed350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:48.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.769+0000 7f2f9c2e0640 1 --2- 172.21.15.116:0/3978449997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2f94149df0 0x7f2f941e9ee0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:48.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.769+0000 7f2f9c2e0640 1 --2- 172.21.15.116:0/3978449997 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2f80004d50 0x7f2f94151570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:48.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.769+0000 7f2f9c2e0640 1 -- 172.21.15.116:0/3978449997 >> 172.21.15.116:0/3978449997 conn(0x7f2f940c3a20 msgr2=0x7f2f94147ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:48.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.769+0000 7f2f9c2e0640 1 -- 172.21.15.116:0/3978449997 shutdown_connections 2024-04-25T00:01:48.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.769+0000 7f2f9c2e0640 1 -- 172.21.15.116:0/3978449997 wait complete. 2024-04-25T00:01:48.790 INFO:tasks.mds_thrash:Wait for all MDSs to reach steady state... 2024-04-25T00:01:48.791 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:01:48.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.905+0000 7f602af3e640 1 Processor -- start 2024-04-25T00:01:48.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.905+0000 7f602af3e640 1 -- start start 2024-04-25T00:01:48.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.909+0000 7f602af3e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6024152d80 0x7f602406bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:48.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.909+0000 7f602af3e640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f602406d5f0 0x7f6024156c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:48.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.909+0000 7f602af3e640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60240c9150 con 0x7f602406c290 2024-04-25T00:01:48.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.909+0000 7f602af3e640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f60240c9d60 con 0x7f602406d5f0 2024-04-25T00:01:48.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.909+0000 7f602af3e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f60240c9430 con 0x7f6024152d80 2024-04-25T00:01:48.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.909+0000 7f6028cda640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6024152d80 0x7f602406bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:48.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.909+0000 7f6023fff640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f602406d5f0 0x7f6024156c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:48.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.909+0000 7f6028cda640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6024152d80 0x7f602406bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:38322/0 (socket says 172.21.15.116:38322) 2024-04-25T00:01:48.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.909+0000 7f6028cda640 1 -- 172.21.15.116:0/4210323308 learned_addr learned my addr 172.21.15.116:0/4210323308 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:48.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.909+0000 7f6028cda640 1 -- 172.21.15.116:0/4210323308 >> v2:172.21.15.174:3300/0 conn(0x7f602406d5f0 msgr2=0x7f6024156c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:48.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.909+0000 7f6028cda640 1 --2- 172.21.15.116:0/4210323308 >> v2:172.21.15.174:3300/0 conn(0x7f602406d5f0 0x7f6024156c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:48.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.909+0000 7f6028cda640 1 -- 172.21.15.116:0/4210323308 >> v1:172.21.15.174:6789/0 conn(0x7f602406c290 legacy=0x7f60241571c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:48.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.909+0000 7f6028cda640 1 -- 172.21.15.116:0/4210323308 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60240c9710 con 0x7f6024152d80 2024-04-25T00:01:48.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.909+0000 7f6023fff640 1 --2- 172.21.15.116:0/4210323308 >> v2:172.21.15.174:3300/0 conn(0x7f602406d5f0 0x7f6024156c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:01:48.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.909+0000 7f6028cda640 1 --2- 172.21.15.116:0/4210323308 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6024152d80 0x7f602406bd50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6014009870 tx=0x7f601401c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95a47ccd6d6ba0f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:48.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.913+0000 7f60237fe640 1 -- 172.21.15.116:0/4210323308 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60140aa620 con 0x7f6024152d80 2024-04-25T00:01:48.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.913+0000 7f60237fe640 1 -- 172.21.15.116:0/4210323308 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60140af070 con 0x7f6024152d80 2024-04-25T00:01:48.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.913+0000 7f60237fe640 1 -- 172.21.15.116:0/4210323308 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60140aa9e0 con 0x7f6024152d80 2024-04-25T00:01:48.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.913+0000 7f602af3e640 1 -- 172.21.15.116:0/4210323308 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6024152d80 msgr2=0x7f602406bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:48.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.913+0000 7f602af3e640 1 --2- 172.21.15.116:0/4210323308 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6024152d80 0x7f602406bd50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6014009870 tx=0x7f601401c4a0 comp rx=0 tx=0).stop 2024-04-25T00:01:48.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.913+0000 7f602af3e640 1 -- 172.21.15.116:0/4210323308 shutdown_connections 2024-04-25T00:01:48.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.913+0000 7f602af3e640 1 --2- 172.21.15.116:0/4210323308 >> v2:172.21.15.174:3300/0 conn(0x7f602406d5f0 0x7f6024156c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:48.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.913+0000 7f602af3e640 1 --2- 172.21.15.116:0/4210323308 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6024152d80 0x7f602406bd50 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:48.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.913+0000 7f602af3e640 1 -- 172.21.15.116:0/4210323308 >> 172.21.15.116:0/4210323308 conn(0x7f60240acdd0 msgr2=0x7f60240ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:48.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.913+0000 7f602af3e640 1 -- 172.21.15.116:0/4210323308 shutdown_connections 2024-04-25T00:01:48.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.913+0000 7f602af3e640 1 -- 172.21.15.116:0/4210323308 wait complete. 2024-04-25T00:01:48.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.913+0000 7f602af3e640 1 Processor -- start 2024-04-25T00:01:48.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.913+0000 7f602af3e640 1 -- start start 2024-04-25T00:01:48.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.913+0000 7f602af3e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f602406c290 0x7f6024146fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:48.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.913+0000 7f602af3e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f602406d5f0 0x7f60240aaa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:48.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.913+0000 7f602af3e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6024152d80 0x7f60240aafb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:48.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.913+0000 7f602af3e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f60240caf00 con 0x7f6024152d80 2024-04-25T00:01:48.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.913+0000 7f602af3e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f60240cad80 con 0x7f602406d5f0 2024-04-25T00:01:48.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.913+0000 7f602af3e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f60241628d0 con 0x7f602406c290 2024-04-25T00:01:48.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.913+0000 7f60294db640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6024152d80 0x7f60240aafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:48.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.913+0000 7f60294db640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6024152d80 0x7f60240aafb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43308/0 (socket says 172.21.15.116:43308) 2024-04-25T00:01:48.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.913+0000 7f60294db640 1 -- 172.21.15.116:0/2524125389 learned_addr learned my addr 172.21.15.116:0/2524125389 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:48.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.913+0000 7f6023fff640 1 --2- 172.21.15.116:0/2524125389 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f602406d5f0 0x7f60240aaa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:48.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.913+0000 7f6028cda640 1 --2- 172.21.15.116:0/2524125389 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f602406c290 0x7f6024146fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:48.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.913+0000 7f60294db640 1 -- 172.21.15.116:0/2524125389 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f602406c290 msgr2=0x7f6024146fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:48.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.913+0000 7f60294db640 1 --2- 172.21.15.116:0/2524125389 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f602406c290 0x7f6024146fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:48.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.913+0000 7f60294db640 1 -- 172.21.15.116:0/2524125389 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f602406d5f0 msgr2=0x7f60240aaa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:48.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.913+0000 7f60294db640 1 --2- 172.21.15.116:0/2524125389 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f602406d5f0 0x7f60240aaa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:48.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.913+0000 7f60294db640 1 -- 172.21.15.116:0/2524125389 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60240c7780 con 0x7f6024152d80 2024-04-25T00:01:48.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.913+0000 7f60294db640 1 --2- 172.21.15.116:0/2524125389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6024152d80 0x7f60240aafb0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f601c00c760 tx=0x7f601c00cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:48.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.913+0000 7f60217fa640 1 -- 172.21.15.116:0/2524125389 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f601c016020 con 0x7f6024152d80 2024-04-25T00:01:48.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.913+0000 7f60217fa640 1 -- 172.21.15.116:0/2524125389 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f601c006d60 con 0x7f6024152d80 2024-04-25T00:01:48.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.913+0000 7f602af3e640 1 -- 172.21.15.116:0/2524125389 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60240c74e0 con 0x7f6024152d80 2024-04-25T00:01:48.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.913+0000 7f60217fa640 1 -- 172.21.15.116:0/2524125389 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f601c003bc0 con 0x7f6024152d80 2024-04-25T00:01:48.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.913+0000 7f602af3e640 1 -- 172.21.15.116:0/2524125389 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60241f27e0 con 0x7f6024152d80 2024-04-25T00:01:48.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.917+0000 7f602af3e640 1 -- 172.21.15.116:0/2524125389 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6024068350 con 0x7f6024152d80 2024-04-25T00:01:48.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.917+0000 7f60217fa640 1 -- 172.21.15.116:0/2524125389 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f601c0050d0 con 0x7f6024152d80 2024-04-25T00:01:48.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.917+0000 7f60217fa640 1 --2- 172.21.15.116:0/2524125389 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f60100678f0 0x7f6010069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:48.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.917+0000 7f60217fa640 1 -- 172.21.15.116:0/2524125389 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6666+0+0 (secure 0 0 0) 0x7f601c085bf0 con 0x7f6024152d80 2024-04-25T00:01:48.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.917+0000 7f6028cda640 1 --2- 172.21.15.116:0/2524125389 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f60100678f0 0x7f6010069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:48.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.917+0000 7f6028cda640 1 --2- 172.21.15.116:0/2524125389 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f60100678f0 0x7f6010069db0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f60140ae000 tx=0x7f60140a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:48.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:48.921+0000 7f60217fa640 1 -- 172.21.15.116:0/2524125389 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f601c020020 con 0x7f6024152d80 2024-04-25T00:01:49.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.117+0000 7f602af3e640 1 -- 172.21.15.116:0/2524125389 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f60240cab60 con 0x7f6024152d80 2024-04-25T00:01:49.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.117+0000 7f60217fa640 1 -- 172.21.15.116:0/2524125389 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6492 (secure 0 0 0) 0x7f601c056400 con 0x7f6024152d80 2024-04-25T00:01:49.122 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:01:49.122 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6834/1459177921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1459177921},{"type":"v1","addr":"172.21.15.116:6834","nonce":1459177921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:01:08.748957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6837/1762356055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1762356055},{"type":"v1","addr":"172.21.15.174:6837","nonce":1762356055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270]},"id":1},{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:01:22.888641+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6835/174781672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":174781672},{"type":"v1","addr":"172.21.15.116:6835","nonce":174781672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4245,"qdb_cluster":[4245]},"id":2}]} 2024-04-25T00:01:49.122 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 14 2024-04-25T00:01:49.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.121+0000 7f602af3e640 1 -- 172.21.15.116:0/2524125389 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f60100678f0 msgr2=0x7f6010069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:49.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.121+0000 7f602af3e640 1 --2- 172.21.15.116:0/2524125389 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f60100678f0 0x7f6010069db0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f60140ae000 tx=0x7f60140a7000 comp rx=0 tx=0).stop 2024-04-25T00:01:49.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.121+0000 7f602af3e640 1 -- 172.21.15.116:0/2524125389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6024152d80 msgr2=0x7f60240aafb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:49.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.121+0000 7f602af3e640 1 --2- 172.21.15.116:0/2524125389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6024152d80 0x7f60240aafb0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f601c00c760 tx=0x7f601c00cb60 comp rx=0 tx=0).stop 2024-04-25T00:01:49.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.121+0000 7f602af3e640 1 -- 172.21.15.116:0/2524125389 shutdown_connections 2024-04-25T00:01:49.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.121+0000 7f602af3e640 1 --2- 172.21.15.116:0/2524125389 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f60100678f0 0x7f6010069db0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:49.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.121+0000 7f602af3e640 1 --2- 172.21.15.116:0/2524125389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6024152d80 0x7f60240aafb0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:49.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.121+0000 7f602af3e640 1 --2- 172.21.15.116:0/2524125389 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f602406d5f0 0x7f60240aaa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:49.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.121+0000 7f602af3e640 1 --2- 172.21.15.116:0/2524125389 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f602406c290 0x7f6024146fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:49.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.121+0000 7f602af3e640 1 -- 172.21.15.116:0/2524125389 >> 172.21.15.116:0/2524125389 conn(0x7f60240acdd0 msgr2=0x7f6024155d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:49.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.121+0000 7f602af3e640 1 -- 172.21.15.116:0/2524125389 shutdown_connections 2024-04-25T00:01:49.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.121+0000 7f602af3e640 1 -- 172.21.15.116:0/2524125389 wait complete. 2024-04-25T00:01:49.141 INFO:tasks.mds_thrash:Ready to start thrashing 2024-04-25T00:01:49.141 INFO:tasks.mds_thrash.ceph_manager:waiting for clean 2024-04-25T00:01:49.142 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T00:01:49.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.257+0000 7fb7f70b9640 1 Processor -- start 2024-04-25T00:01:49.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.257+0000 7fb7f70b9640 1 -- start start 2024-04-25T00:01:49.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.257+0000 7fb7f70b9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb7f0150d10 0x7fb7f01541f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:49.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.257+0000 7fb7f70b9640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fb7f00c2b10 0x7fb7f00a94a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:49.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.257+0000 7fb7f70b9640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7f00734b0 con 0x7fb7f00a99e0 2024-04-25T00:01:49.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.257+0000 7fb7f70b9640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fb7f0153750 con 0x7fb7f0150d10 2024-04-25T00:01:49.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.257+0000 7fb7f70b9640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fb7f00731d0 con 0x7fb7f00c2b10 2024-04-25T00:01:49.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.257+0000 7fb7f4e55640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb7f0150d10 0x7fb7f01541f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:49.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.257+0000 7fb7f4e55640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb7f0150d10 0x7fb7f01541f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43324/0 (socket says 172.21.15.116:43324) 2024-04-25T00:01:49.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.257+0000 7fb7f4e55640 1 -- 172.21.15.116:0/3084548594 learned_addr learned my addr 172.21.15.116:0/3084548594 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:49.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.257+0000 7fb7e7fff640 1 --2- 172.21.15.116:0/3084548594 >> v2:172.21.15.174:3300/0 conn(0x7fb7f00c2b10 0x7fb7f00a94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:49.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.257+0000 7fb7f4e55640 1 -- 172.21.15.116:0/3084548594 >> v2:172.21.15.174:3300/0 conn(0x7fb7f00c2b10 msgr2=0x7fb7f00a94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:49.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.257+0000 7fb7f4e55640 1 --2- 172.21.15.116:0/3084548594 >> v2:172.21.15.174:3300/0 conn(0x7fb7f00c2b10 0x7fb7f00a94a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:49.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.257+0000 7fb7f4e55640 1 -- 172.21.15.116:0/3084548594 >> v1:172.21.15.174:6789/0 conn(0x7fb7f00a99e0 legacy=0x7fb7f00c3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:49.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.257+0000 7fb7f4e55640 1 -- 172.21.15.116:0/3084548594 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7f01539f0 con 0x7fb7f0150d10 2024-04-25T00:01:49.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.261+0000 7fb7f4e55640 1 --2- 172.21.15.116:0/3084548594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7f0150d10 0x7fb7f01541f0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0073700 tx=0x7fb7ec01c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b34ef575bc711b3a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:49.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.261+0000 7fb7e77fe640 1 -- 172.21.15.116:0/3084548594 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7ec0b2070 con 0x7fb7f0150d10 2024-04-25T00:01:49.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.261+0000 7fb7e77fe640 1 -- 172.21.15.116:0/3084548594 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7ec0a9070 con 0x7fb7f0150d10 2024-04-25T00:01:49.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.261+0000 7fb7e77fe640 1 -- 172.21.15.116:0/3084548594 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7ec0ad880 con 0x7fb7f0150d10 2024-04-25T00:01:49.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.261+0000 7fb7f70b9640 1 -- 172.21.15.116:0/3084548594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7f0150d10 msgr2=0x7fb7f01541f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:49.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.261+0000 7fb7f70b9640 1 --2- 172.21.15.116:0/3084548594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7f0150d10 0x7fb7f01541f0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fb7f0073700 tx=0x7fb7ec01c450 comp rx=0 tx=0).stop 2024-04-25T00:01:49.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.261+0000 7fb7f70b9640 1 -- 172.21.15.116:0/3084548594 shutdown_connections 2024-04-25T00:01:49.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.261+0000 7fb7f70b9640 1 --2- 172.21.15.116:0/3084548594 >> v2:172.21.15.174:3300/0 conn(0x7fb7f00c2b10 0x7fb7f00a94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:49.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.261+0000 7fb7f70b9640 1 --2- 172.21.15.116:0/3084548594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7f0150d10 0x7fb7f01541f0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:49.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.261+0000 7fb7f70b9640 1 -- 172.21.15.116:0/3084548594 >> 172.21.15.116:0/3084548594 conn(0x7fb7f00c4c10 msgr2=0x7fb7f00c5020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:49.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.261+0000 7fb7f70b9640 1 -- 172.21.15.116:0/3084548594 shutdown_connections 2024-04-25T00:01:49.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.261+0000 7fb7f70b9640 1 -- 172.21.15.116:0/3084548594 wait complete. 2024-04-25T00:01:49.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.265+0000 7fb7f70b9640 1 Processor -- start 2024-04-25T00:01:49.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.265+0000 7fb7f70b9640 1 -- start start 2024-04-25T00:01:49.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.265+0000 7fb7f70b9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7f00c2b10 0x7fb7f006ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:49.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.265+0000 7fb7f70b9640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb7f0150d10 0x7fb7f006e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:49.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.265+0000 7fb7f70b9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb7f00c7c30 0x7fb7f015c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:49.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.265+0000 7fb7f70b9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb7f0151270 con 0x7fb7f00c2b10 2024-04-25T00:01:49.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.265+0000 7fb7f70b9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb7f01510f0 con 0x7fb7f00c7c30 2024-04-25T00:01:49.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.265+0000 7fb7f70b9640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fb7f0160750 con 0x7fb7f0150d10 2024-04-25T00:01:49.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.265+0000 7fb7f4e55640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7f00c2b10 0x7fb7f006ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:49.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.265+0000 7fb7f4e55640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7f00c2b10 0x7fb7f006ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43340/0 (socket says 172.21.15.116:43340) 2024-04-25T00:01:49.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.265+0000 7fb7f4e55640 1 -- 172.21.15.116:0/1373430719 learned_addr learned my addr 172.21.15.116:0/1373430719 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:49.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.265+0000 7fb7f4e55640 1 -- 172.21.15.116:0/1373430719 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb7f0150d10 msgr2=0x7fb7f006e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:49.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.265+0000 7fb7f4e55640 1 --2- 172.21.15.116:0/1373430719 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb7f0150d10 0x7fb7f006e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:49.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.265+0000 7fb7f4e55640 1 -- 172.21.15.116:0/1373430719 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb7f00c7c30 msgr2=0x7fb7f015c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:49.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.265+0000 7fb7f4e55640 1 --2- 172.21.15.116:0/1373430719 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb7f00c7c30 0x7fb7f015c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:49.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.265+0000 7fb7f4e55640 1 -- 172.21.15.116:0/1373430719 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7f006fd60 con 0x7fb7f00c2b10 2024-04-25T00:01:49.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.265+0000 7fb7f4e55640 1 --2- 172.21.15.116:0/1373430719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7f00c2b10 0x7fb7f006ef40 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec0b0040 tx=0x7fb7ec01ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:49.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.265+0000 7fb7e57fa640 1 -- 172.21.15.116:0/1373430719 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7ec0bc020 con 0x7fb7f00c2b10 2024-04-25T00:01:49.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.265+0000 7fb7e57fa640 1 -- 172.21.15.116:0/1373430719 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7ec0b8420 con 0x7fb7f00c2b10 2024-04-25T00:01:49.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.265+0000 7fb7f70b9640 1 -- 172.21.15.116:0/1373430719 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7f013f7a0 con 0x7fb7f00c2b10 2024-04-25T00:01:49.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.265+0000 7fb7e57fa640 1 -- 172.21.15.116:0/1373430719 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7ec0b2040 con 0x7fb7f00c2b10 2024-04-25T00:01:49.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.265+0000 7fb7f70b9640 1 -- 172.21.15.116:0/1373430719 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7f0152800 con 0x7fb7f00c2b10 2024-04-25T00:01:49.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.265+0000 7fb7f70b9640 1 -- 172.21.15.116:0/1373430719 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7f0068a60 con 0x7fb7f00c2b10 2024-04-25T00:01:49.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.265+0000 7fb7e57fa640 1 -- 172.21.15.116:0/1373430719 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb7ec0bf020 con 0x7fb7f00c2b10 2024-04-25T00:01:49.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.269+0000 7fb7e57fa640 1 --2- 172.21.15.116:0/1373430719 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb7dc0678c0 0x7fb7dc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:49.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.269+0000 7fb7e57fa640 1 -- 172.21.15.116:0/1373430719 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6666+0+0 (secure 0 0 0) 0x7fb7ec120670 con 0x7fb7f00c2b10 2024-04-25T00:01:49.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.269+0000 7fb7e7fff640 1 --2- 172.21.15.116:0/1373430719 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb7dc0678c0 0x7fb7dc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:49.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.269+0000 7fb7e7fff640 1 --2- 172.21.15.116:0/1373430719 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb7dc0678c0 0x7fb7dc069d80 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb7f014ac60 tx=0x7fb7e0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:49.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.273+0000 7fb7e57fa640 1 -- 172.21.15.116:0/1373430719 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fb7ec0c6030 con 0x7fb7f00c2b10 2024-04-25T00:01:49.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.417+0000 7fb7f70b9640 1 -- 172.21.15.116:0/1373430719 --> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb7f00c6780 con 0x7fb7dc0678c0 2024-04-25T00:01:49.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.449+0000 7fb7e57fa640 1 -- 172.21.15.116:0/1373430719 <== mgr.4105 v2:172.21.15.174:6832/2004448211 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+490400 (secure 0 0 0) 0x7fb7e000adc0 con 0x7fb7dc0678c0 2024-04-25T00:01:49.459 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:01:49.462 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-04-25T00:01:49.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.457+0000 7fb7f70b9640 1 -- 172.21.15.116:0/1373430719 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb7dc0678c0 msgr2=0x7fb7dc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:49.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.457+0000 7fb7f70b9640 1 --2- 172.21.15.116:0/1373430719 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb7dc0678c0 0x7fb7dc069d80 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb7f014ac60 tx=0x7fb7e0008040 comp rx=0 tx=0).stop 2024-04-25T00:01:49.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.457+0000 7fb7f70b9640 1 -- 172.21.15.116:0/1373430719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7f00c2b10 msgr2=0x7fb7f006ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:49.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.457+0000 7fb7f70b9640 1 --2- 172.21.15.116:0/1373430719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7f00c2b10 0x7fb7f006ef40 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec0b0040 tx=0x7fb7ec01ac90 comp rx=0 tx=0).stop 2024-04-25T00:01:49.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.457+0000 7fb7f70b9640 1 -- 172.21.15.116:0/1373430719 shutdown_connections 2024-04-25T00:01:49.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.457+0000 7fb7f70b9640 1 --2- 172.21.15.116:0/1373430719 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb7dc0678c0 0x7fb7dc069d80 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:49.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.457+0000 7fb7f70b9640 1 --2- 172.21.15.116:0/1373430719 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb7f00c7c30 0x7fb7f015c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:49.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.457+0000 7fb7f70b9640 1 --2- 172.21.15.116:0/1373430719 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb7f0150d10 0x7fb7f006e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:49.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.457+0000 7fb7f70b9640 1 --2- 172.21.15.116:0/1373430719 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7f00c2b10 0x7fb7f006ef40 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:49.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.457+0000 7fb7f70b9640 1 -- 172.21.15.116:0/1373430719 >> 172.21.15.116:0/1373430719 conn(0x7fb7f00c4c10 msgr2=0x7fb7f00c2ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:49.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.457+0000 7fb7f70b9640 1 -- 172.21.15.116:0/1373430719 shutdown_connections 2024-04-25T00:01:49.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.457+0000 7fb7f70b9640 1 -- 172.21.15.116:0/1373430719 wait complete. 2024-04-25T00:01:49.475 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":52,"stamp":"2024-04-25T00:01:49.268391+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596070,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":253,"num_write_kb":2448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":199,"ondisk_log_size":199,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":387,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219800,"kb_used_data":3628,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754920,"statfs":{"total":773094113280,"available":772869038080,"internally_reserved":0,"allocated":3715072,"data_stored":2145305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":882,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003190"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.656917+0000","last_change":"2024-04-25T00:01:19.508728+0000","last_active":"2024-04-25T00:01:46.656917+0000","last_peered":"2024-04-25T00:01:46.656917+0000","last_clean":"2024-04-25T00:01:46.656917+0000","last_became_active":"2024-04-25T00:01:19.508390+0000","last_became_peered":"2024-04-25T00:01:19.508390+0000","last_unstale":"2024-04-25T00:01:46.656917+0000","last_undegraded":"2024-04-25T00:01:46.656917+0000","last_fullsized":"2024-04-25T00:01:46.656917+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:45.591905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1f","version":"22'4","reported_seq":26,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.515265+0000","last_change":"2024-04-25T00:01:04.356032+0000","last_active":"2024-04-25T00:01:20.515265+0000","last_peered":"2024-04-25T00:01:20.515265+0000","last_clean":"2024-04-25T00:01:20.515265+0000","last_became_active":"2024-04-25T00:01:04.355293+0000","last_became_peered":"2024-04-25T00:01:04.355293+0000","last_unstale":"2024-04-25T00:01:20.515265+0000","last_undegraded":"2024-04-25T00:01:20.515265+0000","last_fullsized":"2024-04-25T00:01:20.515265+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:56:16.678132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.651891+0000","last_change":"2024-04-25T00:01:05.367207+0000","last_active":"2024-04-25T00:01:46.651891+0000","last_peered":"2024-04-25T00:01:46.651891+0000","last_clean":"2024-04-25T00:01:46.651891+0000","last_became_active":"2024-04-25T00:01:05.366471+0000","last_became_peered":"2024-04-25T00:01:05.366471+0000","last_unstale":"2024-04-25T00:01:46.651891+0000","last_undegraded":"2024-04-25T00:01:46.651891+0000","last_fullsized":"2024-04-25T00:01:46.651891+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:30:23.557381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659979+0000","last_change":"2024-04-25T00:01:18.502364+0000","last_active":"2024-04-25T00:01:46.659979+0000","last_peered":"2024-04-25T00:01:46.659979+0000","last_clean":"2024-04-25T00:01:46.659979+0000","last_became_active":"2024-04-25T00:01:18.501633+0000","last_became_peered":"2024-04-25T00:01:18.501633+0000","last_unstale":"2024-04-25T00:01:46.659979+0000","last_undegraded":"2024-04-25T00:01:46.659979+0000","last_fullsized":"2024-04-25T00:01:46.659979+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:55.074018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":10,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.665473+0000","last_change":"2024-04-25T00:01:46.662019+0000","last_active":"2024-04-25T00:01:46.665473+0000","last_peered":"2024-04-25T00:01:46.665473+0000","last_clean":"2024-04-25T00:01:46.665473+0000","last_became_active":"2024-04-25T00:01:46.661167+0000","last_became_peered":"2024-04-25T00:01:46.661167+0000","last_unstale":"2024-04-25T00:01:46.665473+0000","last_undegraded":"2024-04-25T00:01:46.665473+0000","last_fullsized":"2024-04-25T00:01:46.665473+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:13:34.153117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"22'1","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.652395+0000","last_change":"2024-04-25T00:01:04.361809+0000","last_active":"2024-04-25T00:01:46.652395+0000","last_peered":"2024-04-25T00:01:46.652395+0000","last_clean":"2024-04-25T00:01:46.652395+0000","last_became_active":"2024-04-25T00:01:04.361059+0000","last_became_peered":"2024-04-25T00:01:04.361059+0000","last_unstale":"2024-04-25T00:01:46.652395+0000","last_undegraded":"2024-04-25T00:01:46.652395+0000","last_fullsized":"2024-04-25T00:01:46.652395+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.653796+0000","last_change":"2024-04-25T00:01:05.370818+0000","last_active":"2024-04-25T00:01:46.653796+0000","last_peered":"2024-04-25T00:01:46.653796+0000","last_clean":"2024-04-25T00:01:46.653796+0000","last_became_active":"2024-04-25T00:01:05.369991+0000","last_became_peered":"2024-04-25T00:01:05.369991+0000","last_unstale":"2024-04-25T00:01:46.653796+0000","last_undegraded":"2024-04-25T00:01:46.653796+0000","last_fullsized":"2024-04-25T00:01:46.653796+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:43.615895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.647007+0000","last_change":"2024-04-25T00:01:18.497682+0000","last_active":"2024-04-25T00:01:46.647007+0000","last_peered":"2024-04-25T00:01:46.647007+0000","last_clean":"2024-04-25T00:01:46.647007+0000","last_became_active":"2024-04-25T00:01:18.497397+0000","last_became_peered":"2024-04-25T00:01:18.497397+0000","last_unstale":"2024-04-25T00:01:46.647007+0000","last_undegraded":"2024-04-25T00:01:46.647007+0000","last_fullsized":"2024-04-25T00:01:46.647007+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:22.983972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.658174+0000","last_change":"2024-04-25T00:01:19.510591+0000","last_active":"2024-04-25T00:01:46.658174+0000","last_peered":"2024-04-25T00:01:46.658174+0000","last_clean":"2024-04-25T00:01:46.658174+0000","last_became_active":"2024-04-25T00:01:19.510252+0000","last_became_peered":"2024-04-25T00:01:19.510252+0000","last_unstale":"2024-04-25T00:01:46.658174+0000","last_undegraded":"2024-04-25T00:01:46.658174+0000","last_fullsized":"2024-04-25T00:01:46.658174+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:38.005233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"22'2","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660361+0000","last_change":"2024-04-25T00:01:04.362030+0000","last_active":"2024-04-25T00:01:46.660361+0000","last_peered":"2024-04-25T00:01:46.660361+0000","last_clean":"2024-04-25T00:01:46.660361+0000","last_became_active":"2024-04-25T00:01:04.361708+0000","last_became_peered":"2024-04-25T00:01:04.361708+0000","last_unstale":"2024-04-25T00:01:46.660361+0000","last_undegraded":"2024-04-25T00:01:46.660361+0000","last_fullsized":"2024-04-25T00:01:46.660361+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:38:33.392719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.658174+0000","last_change":"2024-04-25T00:01:05.387195+0000","last_active":"2024-04-25T00:01:46.658174+0000","last_peered":"2024-04-25T00:01:46.658174+0000","last_clean":"2024-04-25T00:01:46.658174+0000","last_became_active":"2024-04-25T00:01:05.386853+0000","last_became_peered":"2024-04-25T00:01:05.386853+0000","last_unstale":"2024-04-25T00:01:46.658174+0000","last_undegraded":"2024-04-25T00:01:46.658174+0000","last_fullsized":"2024-04-25T00:01:46.658174+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:06:19.307378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"26'1","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.647549+0000","last_change":"2024-04-25T00:01:18.487835+0000","last_active":"2024-04-25T00:01:46.647549+0000","last_peered":"2024-04-25T00:01:46.647549+0000","last_clean":"2024-04-25T00:01:46.647549+0000","last_became_active":"2024-04-25T00:01:18.487571+0000","last_became_peered":"2024-04-25T00:01:18.487571+0000","last_unstale":"2024-04-25T00:01:46.647549+0000","last_undegraded":"2024-04-25T00:01:46.647549+0000","last_fullsized":"2024-04-25T00:01:46.647549+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:44.582148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.647421+0000","last_change":"2024-04-25T00:01:19.519834+0000","last_active":"2024-04-25T00:01:46.647421+0000","last_peered":"2024-04-25T00:01:46.647421+0000","last_clean":"2024-04-25T00:01:46.647421+0000","last_became_active":"2024-04-25T00:01:19.519470+0000","last_became_peered":"2024-04-25T00:01:19.519470+0000","last_unstale":"2024-04-25T00:01:46.647421+0000","last_undegraded":"2024-04-25T00:01:46.647421+0000","last_fullsized":"2024-04-25T00:01:46.647421+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:47.859626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"22'2","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660830+0000","last_change":"2024-04-25T00:01:04.362832+0000","last_active":"2024-04-25T00:01:46.660830+0000","last_peered":"2024-04-25T00:01:46.660830+0000","last_clean":"2024-04-25T00:01:46.660830+0000","last_became_active":"2024-04-25T00:01:04.362059+0000","last_became_peered":"2024-04-25T00:01:04.362059+0000","last_unstale":"2024-04-25T00:01:46.660830+0000","last_undegraded":"2024-04-25T00:01:46.660830+0000","last_fullsized":"2024-04-25T00:01:46.660830+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:05.441048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659460+0000","last_change":"2024-04-25T00:01:05.381120+0000","last_active":"2024-04-25T00:01:46.659460+0000","last_peered":"2024-04-25T00:01:46.659460+0000","last_clean":"2024-04-25T00:01:46.659460+0000","last_became_active":"2024-04-25T00:01:05.378320+0000","last_became_peered":"2024-04-25T00:01:05.378320+0000","last_unstale":"2024-04-25T00:01:46.659460+0000","last_undegraded":"2024-04-25T00:01:46.659460+0000","last_fullsized":"2024-04-25T00:01:46.659460+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:34:06.175918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"26'1","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.647038+0000","last_change":"2024-04-25T00:01:18.490300+0000","last_active":"2024-04-25T00:01:46.647038+0000","last_peered":"2024-04-25T00:01:46.647038+0000","last_clean":"2024-04-25T00:01:46.647038+0000","last_became_active":"2024-04-25T00:01:18.490050+0000","last_became_peered":"2024-04-25T00:01:18.490050+0000","last_unstale":"2024-04-25T00:01:46.647038+0000","last_undegraded":"2024-04-25T00:01:46.647038+0000","last_fullsized":"2024-04-25T00:01:46.647038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:13:05.971606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.649896+0000","last_change":"2024-04-25T00:01:19.500026+0000","last_active":"2024-04-25T00:01:46.649896+0000","last_peered":"2024-04-25T00:01:46.649896+0000","last_clean":"2024-04-25T00:01:46.649896+0000","last_became_active":"2024-04-25T00:01:19.499244+0000","last_became_peered":"2024-04-25T00:01:19.499244+0000","last_unstale":"2024-04-25T00:01:46.649896+0000","last_undegraded":"2024-04-25T00:01:46.649896+0000","last_fullsized":"2024-04-25T00:01:46.649896+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:40:36.350832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.658085+0000","last_change":"2024-04-25T00:01:05.376984+0000","last_active":"2024-04-25T00:01:46.658085+0000","last_peered":"2024-04-25T00:01:46.658085+0000","last_clean":"2024-04-25T00:01:46.658085+0000","last_became_active":"2024-04-25T00:01:05.376259+0000","last_became_peered":"2024-04-25T00:01:05.376259+0000","last_unstale":"2024-04-25T00:01:46.658085+0000","last_undegraded":"2024-04-25T00:01:46.658085+0000","last_fullsized":"2024-04-25T00:01:46.658085+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:13:28.543302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"22'2","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.658522+0000","last_change":"2024-04-25T00:01:04.359262+0000","last_active":"2024-04-25T00:01:46.658522+0000","last_peered":"2024-04-25T00:01:46.658522+0000","last_clean":"2024-04-25T00:01:46.658522+0000","last_became_active":"2024-04-25T00:01:04.357971+0000","last_became_peered":"2024-04-25T00:01:04.357971+0000","last_unstale":"2024-04-25T00:01:46.658522+0000","last_undegraded":"2024-04-25T00:01:46.658522+0000","last_fullsized":"2024-04-25T00:01:46.658522+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:05.441048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"26'1","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659290+0000","last_change":"2024-04-25T00:01:18.475624+0000","last_active":"2024-04-25T00:01:46.659290+0000","last_peered":"2024-04-25T00:01:46.659290+0000","last_clean":"2024-04-25T00:01:46.659290+0000","last_became_active":"2024-04-25T00:01:18.475149+0000","last_became_peered":"2024-04-25T00:01:18.475149+0000","last_unstale":"2024-04-25T00:01:46.659290+0000","last_undegraded":"2024-04-25T00:01:46.659290+0000","last_fullsized":"2024-04-25T00:01:46.659290+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:40:35.341075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.664518+0000","last_change":"2024-04-25T00:01:19.531113+0000","last_active":"2024-04-25T00:01:46.664518+0000","last_peered":"2024-04-25T00:01:46.664518+0000","last_clean":"2024-04-25T00:01:46.664518+0000","last_became_active":"2024-04-25T00:01:19.520150+0000","last_became_peered":"2024-04-25T00:01:19.520150+0000","last_unstale":"2024-04-25T00:01:46.664518+0000","last_undegraded":"2024-04-25T00:01:46.664518+0000","last_fullsized":"2024-04-25T00:01:46.664518+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:38.152376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659977+0000","last_change":"2024-04-25T00:01:05.362611+0000","last_active":"2024-04-25T00:01:46.659977+0000","last_peered":"2024-04-25T00:01:46.659977+0000","last_clean":"2024-04-25T00:01:46.659977+0000","last_became_active":"2024-04-25T00:01:05.361817+0000","last_became_peered":"2024-04-25T00:01:05.361817+0000","last_unstale":"2024-04-25T00:01:46.659977+0000","last_undegraded":"2024-04-25T00:01:46.659977+0000","last_fullsized":"2024-04-25T00:01:46.659977+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:50.489125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.663735+0000","last_change":"2024-04-25T00:01:04.371935+0000","last_active":"2024-04-25T00:01:46.663735+0000","last_peered":"2024-04-25T00:01:46.663735+0000","last_clean":"2024-04-25T00:01:46.663735+0000","last_became_active":"2024-04-25T00:01:04.371420+0000","last_became_peered":"2024-04-25T00:01:04.371420+0000","last_unstale":"2024-04-25T00:01:46.663735+0000","last_undegraded":"2024-04-25T00:01:46.663735+0000","last_fullsized":"2024-04-25T00:01:46.663735+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660605+0000","last_change":"2024-04-25T00:01:18.488623+0000","last_active":"2024-04-25T00:01:46.660605+0000","last_peered":"2024-04-25T00:01:46.660605+0000","last_clean":"2024-04-25T00:01:46.660605+0000","last_became_active":"2024-04-25T00:01:18.488291+0000","last_became_peered":"2024-04-25T00:01:18.488291+0000","last_unstale":"2024-04-25T00:01:46.660605+0000","last_undegraded":"2024-04-25T00:01:46.660605+0000","last_fullsized":"2024-04-25T00:01:46.660605+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:30:21.436681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.658546+0000","last_change":"2024-04-25T00:01:19.516731+0000","last_active":"2024-04-25T00:01:46.658546+0000","last_peered":"2024-04-25T00:01:46.658546+0000","last_clean":"2024-04-25T00:01:46.658546+0000","last_became_active":"2024-04-25T00:01:19.516405+0000","last_became_peered":"2024-04-25T00:01:19.516405+0000","last_unstale":"2024-04-25T00:01:46.658546+0000","last_undegraded":"2024-04-25T00:01:46.658546+0000","last_fullsized":"2024-04-25T00:01:46.658546+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:44:37.936092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.510871+0000","last_change":"2024-04-25T00:01:05.355455+0000","last_active":"2024-04-25T00:01:20.510871+0000","last_peered":"2024-04-25T00:01:20.510871+0000","last_clean":"2024-04-25T00:01:20.510871+0000","last_became_active":"2024-04-25T00:01:05.354699+0000","last_became_peered":"2024-04-25T00:01:05.354699+0000","last_unstale":"2024-04-25T00:01:20.510871+0000","last_undegraded":"2024-04-25T00:01:20.510871+0000","last_fullsized":"2024-04-25T00:01:20.510871+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:43:58.069271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.512349+0000","last_change":"2024-04-25T00:01:04.362268+0000","last_active":"2024-04-25T00:01:20.512349+0000","last_peered":"2024-04-25T00:01:20.512349+0000","last_clean":"2024-04-25T00:01:20.512349+0000","last_became_active":"2024-04-25T00:01:04.361622+0000","last_became_peered":"2024-04-25T00:01:04.361622+0000","last_unstale":"2024-04-25T00:01:20.512349+0000","last_undegraded":"2024-04-25T00:01:20.512349+0000","last_fullsized":"2024-04-25T00:01:20.512349+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:58:46.021343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"26'1","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.664549+0000","last_change":"2024-04-25T00:01:18.494047+0000","last_active":"2024-04-25T00:01:46.664549+0000","last_peered":"2024-04-25T00:01:46.664549+0000","last_clean":"2024-04-25T00:01:46.664549+0000","last_became_active":"2024-04-25T00:01:18.493614+0000","last_became_peered":"2024-04-25T00:01:18.493614+0000","last_unstale":"2024-04-25T00:01:46.664549+0000","last_undegraded":"2024-04-25T00:01:46.664549+0000","last_fullsized":"2024-04-25T00:01:46.664549+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:37:08.562016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657743+0000","last_change":"2024-04-25T00:01:19.505397+0000","last_active":"2024-04-25T00:01:46.657743+0000","last_peered":"2024-04-25T00:01:46.657743+0000","last_clean":"2024-04-25T00:01:46.657743+0000","last_became_active":"2024-04-25T00:01:19.505126+0000","last_became_peered":"2024-04-25T00:01:19.505126+0000","last_unstale":"2024-04-25T00:01:46.657743+0000","last_undegraded":"2024-04-25T00:01:46.657743+0000","last_fullsized":"2024-04-25T00:01:46.657743+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:08:49.640267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657635+0000","last_change":"2024-04-25T00:01:05.354953+0000","last_active":"2024-04-25T00:01:46.657635+0000","last_peered":"2024-04-25T00:01:46.657635+0000","last_clean":"2024-04-25T00:01:46.657635+0000","last_became_active":"2024-04-25T00:01:05.354118+0000","last_became_peered":"2024-04-25T00:01:05.354118+0000","last_unstale":"2024-04-25T00:01:46.657635+0000","last_undegraded":"2024-04-25T00:01:46.657635+0000","last_fullsized":"2024-04-25T00:01:46.657635+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:09.858617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657897+0000","last_change":"2024-04-25T00:01:04.365515+0000","last_active":"2024-04-25T00:01:46.657897+0000","last_peered":"2024-04-25T00:01:46.657897+0000","last_clean":"2024-04-25T00:01:46.657897+0000","last_became_active":"2024-04-25T00:01:04.364942+0000","last_became_peered":"2024-04-25T00:01:04.364942+0000","last_unstale":"2024-04-25T00:01:46.657897+0000","last_undegraded":"2024-04-25T00:01:46.657897+0000","last_fullsized":"2024-04-25T00:01:46.657897+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"26'2","reported_seq":20,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.648107+0000","last_change":"2024-04-25T00:01:18.476207+0000","last_active":"2024-04-25T00:01:46.648107+0000","last_peered":"2024-04-25T00:01:46.648107+0000","last_clean":"2024-04-25T00:01:46.648107+0000","last_became_active":"2024-04-25T00:01:18.475862+0000","last_became_peered":"2024-04-25T00:01:18.475862+0000","last_unstale":"2024-04-25T00:01:46.648107+0000","last_undegraded":"2024-04-25T00:01:46.648107+0000","last_fullsized":"2024-04-25T00:01:46.648107+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:11:40.170249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511258+0000","last_change":"2024-04-25T00:01:19.529607+0000","last_active":"2024-04-25T00:01:20.511258+0000","last_peered":"2024-04-25T00:01:20.511258+0000","last_clean":"2024-04-25T00:01:20.511258+0000","last_became_active":"2024-04-25T00:01:19.528509+0000","last_became_peered":"2024-04-25T00:01:19.528509+0000","last_unstale":"2024-04-25T00:01:20.511258+0000","last_undegraded":"2024-04-25T00:01:20.511258+0000","last_fullsized":"2024-04-25T00:01:20.511258+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:45:01.068091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.514958+0000","last_change":"2024-04-25T00:01:05.358106+0000","last_active":"2024-04-25T00:01:20.514958+0000","last_peered":"2024-04-25T00:01:20.514958+0000","last_clean":"2024-04-25T00:01:20.514958+0000","last_became_active":"2024-04-25T00:01:05.357135+0000","last_became_peered":"2024-04-25T00:01:05.357135+0000","last_unstale":"2024-04-25T00:01:20.514958+0000","last_undegraded":"2024-04-25T00:01:20.514958+0000","last_fullsized":"2024-04-25T00:01:20.514958+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:41:42.228234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"22'2","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660058+0000","last_change":"2024-04-25T00:01:04.369917+0000","last_active":"2024-04-25T00:01:46.660058+0000","last_peered":"2024-04-25T00:01:46.660058+0000","last_clean":"2024-04-25T00:01:46.660058+0000","last_became_active":"2024-04-25T00:01:04.369309+0000","last_became_peered":"2024-04-25T00:01:04.369309+0000","last_unstale":"2024-04-25T00:01:46.660058+0000","last_undegraded":"2024-04-25T00:01:46.660058+0000","last_fullsized":"2024-04-25T00:01:46.660058+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:55:08.063965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.647179+0000","last_change":"2024-04-25T00:01:18.477189+0000","last_active":"2024-04-25T00:01:46.647179+0000","last_peered":"2024-04-25T00:01:46.647179+0000","last_clean":"2024-04-25T00:01:46.647179+0000","last_became_active":"2024-04-25T00:01:18.476909+0000","last_became_peered":"2024-04-25T00:01:18.476909+0000","last_unstale":"2024-04-25T00:01:46.647179+0000","last_undegraded":"2024-04-25T00:01:46.647179+0000","last_fullsized":"2024-04-25T00:01:46.647179+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:36.995477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657252+0000","last_change":"2024-04-25T00:01:19.517500+0000","last_active":"2024-04-25T00:01:46.657252+0000","last_peered":"2024-04-25T00:01:46.657252+0000","last_clean":"2024-04-25T00:01:46.657252+0000","last_became_active":"2024-04-25T00:01:19.517182+0000","last_became_peered":"2024-04-25T00:01:19.517182+0000","last_unstale":"2024-04-25T00:01:46.657252+0000","last_undegraded":"2024-04-25T00:01:46.657252+0000","last_fullsized":"2024-04-25T00:01:46.657252+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:58:49.738233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.515146+0000","last_change":"2024-04-25T00:01:05.364829+0000","last_active":"2024-04-25T00:01:20.515146+0000","last_peered":"2024-04-25T00:01:20.515146+0000","last_clean":"2024-04-25T00:01:20.515146+0000","last_became_active":"2024-04-25T00:01:05.364125+0000","last_became_peered":"2024-04-25T00:01:05.364125+0000","last_unstale":"2024-04-25T00:01:20.515146+0000","last_undegraded":"2024-04-25T00:01:20.515146+0000","last_fullsized":"2024-04-25T00:01:20.515146+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:15:45.531760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"23'1","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.515821+0000","last_change":"2024-04-25T00:01:04.361603+0000","last_active":"2024-04-25T00:01:20.515821+0000","last_peered":"2024-04-25T00:01:20.515821+0000","last_clean":"2024-04-25T00:01:20.515821+0000","last_became_active":"2024-04-25T00:01:04.361046+0000","last_became_peered":"2024-04-25T00:01:04.361046+0000","last_unstale":"2024-04-25T00:01:20.515821+0000","last_undegraded":"2024-04-25T00:01:20.515821+0000","last_fullsized":"2024-04-25T00:01:20.515821+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:35:18.036843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.510620+0000","last_change":"2024-04-25T00:01:18.488352+0000","last_active":"2024-04-25T00:01:20.510620+0000","last_peered":"2024-04-25T00:01:20.510620+0000","last_clean":"2024-04-25T00:01:20.510620+0000","last_became_active":"2024-04-25T00:01:18.487321+0000","last_became_peered":"2024-04-25T00:01:18.487321+0000","last_unstale":"2024-04-25T00:01:20.510620+0000","last_undegraded":"2024-04-25T00:01:20.510620+0000","last_fullsized":"2024-04-25T00:01:20.510620+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:37:22.329655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.658811+0000","last_change":"2024-04-25T00:01:19.519653+0000","last_active":"2024-04-25T00:01:46.658811+0000","last_peered":"2024-04-25T00:01:46.658811+0000","last_clean":"2024-04-25T00:01:46.658811+0000","last_became_active":"2024-04-25T00:01:19.519052+0000","last_became_peered":"2024-04-25T00:01:19.519052+0000","last_unstale":"2024-04-25T00:01:46.658811+0000","last_undegraded":"2024-04-25T00:01:46.658811+0000","last_fullsized":"2024-04-25T00:01:46.658811+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:40:36.350832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.665974+0000","last_change":"2024-04-25T00:01:05.374011+0000","last_active":"2024-04-25T00:01:46.665974+0000","last_peered":"2024-04-25T00:01:46.665974+0000","last_clean":"2024-04-25T00:01:46.665974+0000","last_became_active":"2024-04-25T00:01:05.373276+0000","last_became_peered":"2024-04-25T00:01:05.373276+0000","last_unstale":"2024-04-25T00:01:46.665974+0000","last_undegraded":"2024-04-25T00:01:46.665974+0000","last_fullsized":"2024-04-25T00:01:46.665974+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:40:53.349428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659709+0000","last_change":"2024-04-25T00:01:04.365391+0000","last_active":"2024-04-25T00:01:46.659709+0000","last_peered":"2024-04-25T00:01:46.659709+0000","last_clean":"2024-04-25T00:01:46.659709+0000","last_became_active":"2024-04-25T00:01:04.364989+0000","last_became_peered":"2024-04-25T00:01:04.364989+0000","last_unstale":"2024-04-25T00:01:46.659709+0000","last_undegraded":"2024-04-25T00:01:46.659709+0000","last_fullsized":"2024-04-25T00:01:46.659709+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:06:18.298057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.663291+0000","last_change":"2024-04-25T00:01:18.491597+0000","last_active":"2024-04-25T00:01:46.663291+0000","last_peered":"2024-04-25T00:01:46.663291+0000","last_clean":"2024-04-25T00:01:46.663291+0000","last_became_active":"2024-04-25T00:01:18.490859+0000","last_became_peered":"2024-04-25T00:01:18.490859+0000","last_unstale":"2024-04-25T00:01:46.663291+0000","last_undegraded":"2024-04-25T00:01:46.663291+0000","last_fullsized":"2024-04-25T00:01:46.663291+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:36.995477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":10,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.670049+0000","last_change":"2024-04-25T00:01:46.667934+0000","last_active":"2024-04-25T00:01:46.670049+0000","last_peered":"2024-04-25T00:01:46.670049+0000","last_clean":"2024-04-25T00:01:46.670049+0000","last_became_active":"2024-04-25T00:01:46.667075+0000","last_became_peered":"2024-04-25T00:01:46.667075+0000","last_unstale":"2024-04-25T00:01:46.670049+0000","last_undegraded":"2024-04-25T00:01:46.670049+0000","last_fullsized":"2024-04-25T00:01:46.670049+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:28:18.234546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.648286+0000","last_change":"2024-04-25T00:01:05.378340+0000","last_active":"2024-04-25T00:01:46.648286+0000","last_peered":"2024-04-25T00:01:46.648286+0000","last_clean":"2024-04-25T00:01:46.648286+0000","last_became_active":"2024-04-25T00:01:05.377593+0000","last_became_peered":"2024-04-25T00:01:05.377593+0000","last_unstale":"2024-04-25T00:01:46.648286+0000","last_undegraded":"2024-04-25T00:01:46.648286+0000","last_fullsized":"2024-04-25T00:01:46.648286+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:48.202251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.650594+0000","last_change":"2024-04-25T00:01:04.366577+0000","last_active":"2024-04-25T00:01:46.650594+0000","last_peered":"2024-04-25T00:01:46.650594+0000","last_clean":"2024-04-25T00:01:46.650594+0000","last_became_active":"2024-04-25T00:01:04.365881+0000","last_became_peered":"2024-04-25T00:01:04.365881+0000","last_unstale":"2024-04-25T00:01:46.650594+0000","last_undegraded":"2024-04-25T00:01:46.650594+0000","last_fullsized":"2024-04-25T00:01:46.650594+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:49:22.673421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.662487+0000","last_change":"2024-04-25T00:01:18.504341+0000","last_active":"2024-04-25T00:01:46.662487+0000","last_peered":"2024-04-25T00:01:46.662487+0000","last_clean":"2024-04-25T00:01:46.662487+0000","last_became_active":"2024-04-25T00:01:18.491498+0000","last_became_peered":"2024-04-25T00:01:18.491498+0000","last_unstale":"2024-04-25T00:01:46.662487+0000","last_undegraded":"2024-04-25T00:01:46.662487+0000","last_fullsized":"2024-04-25T00:01:46.662487+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:56.741250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.667022+0000","last_change":"2024-04-25T00:01:46.667022+0000","last_active":"2024-04-25T00:01:46.667022+0000","last_peered":"2024-04-25T00:01:46.667022+0000","last_clean":"2024-04-25T00:01:46.667022+0000","last_became_active":"2024-04-25T00:01:46.666224+0000","last_became_peered":"2024-04-25T00:01:46.666224+0000","last_unstale":"2024-04-25T00:01:46.667022+0000","last_undegraded":"2024-04-25T00:01:46.667022+0000","last_fullsized":"2024-04-25T00:01:46.667022+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:25:20.185355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659841+0000","last_change":"2024-04-25T00:01:05.368769+0000","last_active":"2024-04-25T00:01:46.659841+0000","last_peered":"2024-04-25T00:01:46.659841+0000","last_clean":"2024-04-25T00:01:46.659841+0000","last_became_active":"2024-04-25T00:01:05.366817+0000","last_became_peered":"2024-04-25T00:01:05.366817+0000","last_unstale":"2024-04-25T00:01:46.659841+0000","last_undegraded":"2024-04-25T00:01:46.659841+0000","last_fullsized":"2024-04-25T00:01:46.659841+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:11:27.044893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.658528+0000","last_change":"2024-04-25T00:01:04.361900+0000","last_active":"2024-04-25T00:01:46.658528+0000","last_peered":"2024-04-25T00:01:46.658528+0000","last_clean":"2024-04-25T00:01:46.658528+0000","last_became_active":"2024-04-25T00:01:04.361572+0000","last_became_peered":"2024-04-25T00:01:04.361572+0000","last_unstale":"2024-04-25T00:01:46.658528+0000","last_undegraded":"2024-04-25T00:01:46.658528+0000","last_fullsized":"2024-04-25T00:01:46.658528+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:29.218739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.665621+0000","last_change":"2024-04-25T00:01:18.504139+0000","last_active":"2024-04-25T00:01:46.665621+0000","last_peered":"2024-04-25T00:01:46.665621+0000","last_clean":"2024-04-25T00:01:46.665621+0000","last_became_active":"2024-04-25T00:01:18.503379+0000","last_became_peered":"2024-04-25T00:01:18.503379+0000","last_unstale":"2024-04-25T00:01:46.665621+0000","last_undegraded":"2024-04-25T00:01:46.665621+0000","last_fullsized":"2024-04-25T00:01:46.665621+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:02:56.363258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.664313+0000","last_change":"2024-04-25T00:01:19.531756+0000","last_active":"2024-04-25T00:01:46.664313+0000","last_peered":"2024-04-25T00:01:46.664313+0000","last_clean":"2024-04-25T00:01:46.664313+0000","last_became_active":"2024-04-25T00:01:19.531145+0000","last_became_peered":"2024-04-25T00:01:19.531145+0000","last_unstale":"2024-04-25T00:01:46.664313+0000","last_undegraded":"2024-04-25T00:01:46.664313+0000","last_fullsized":"2024-04-25T00:01:46.664313+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:13:42.678415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.649319+0000","last_change":"2024-04-25T00:01:05.354557+0000","last_active":"2024-04-25T00:01:46.649319+0000","last_peered":"2024-04-25T00:01:46.649319+0000","last_clean":"2024-04-25T00:01:46.649319+0000","last_became_active":"2024-04-25T00:01:05.353556+0000","last_became_peered":"2024-04-25T00:01:05.353556+0000","last_unstale":"2024-04-25T00:01:46.649319+0000","last_undegraded":"2024-04-25T00:01:46.649319+0000","last_fullsized":"2024-04-25T00:01:46.649319+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:24.964055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660230+0000","last_change":"2024-04-25T00:01:04.372916+0000","last_active":"2024-04-25T00:01:46.660230+0000","last_peered":"2024-04-25T00:01:46.660230+0000","last_clean":"2024-04-25T00:01:46.660230+0000","last_became_active":"2024-04-25T00:01:04.372185+0000","last_became_peered":"2024-04-25T00:01:04.372185+0000","last_unstale":"2024-04-25T00:01:46.660230+0000","last_undegraded":"2024-04-25T00:01:46.660230+0000","last_fullsized":"2024-04-25T00:01:46.660230+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"26'2","reported_seq":20,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.665571+0000","last_change":"2024-04-25T00:01:18.504614+0000","last_active":"2024-04-25T00:01:46.665571+0000","last_peered":"2024-04-25T00:01:46.665571+0000","last_clean":"2024-04-25T00:01:46.665571+0000","last_became_active":"2024-04-25T00:01:18.501448+0000","last_became_peered":"2024-04-25T00:01:18.501448+0000","last_unstale":"2024-04-25T00:01:46.665571+0000","last_undegraded":"2024-04-25T00:01:46.665571+0000","last_fullsized":"2024-04-25T00:01:46.665571+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:22.983972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659568+0000","last_change":"2024-04-25T00:01:19.517510+0000","last_active":"2024-04-25T00:01:46.659568+0000","last_peered":"2024-04-25T00:01:46.659568+0000","last_clean":"2024-04-25T00:01:46.659568+0000","last_became_active":"2024-04-25T00:01:19.517243+0000","last_became_peered":"2024-04-25T00:01:19.517243+0000","last_unstale":"2024-04-25T00:01:46.659568+0000","last_undegraded":"2024-04-25T00:01:46.659568+0000","last_fullsized":"2024-04-25T00:01:46.659568+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:42:38.149502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.662201+0000","last_change":"2024-04-25T00:01:05.370968+0000","last_active":"2024-04-25T00:01:46.662201+0000","last_peered":"2024-04-25T00:01:46.662201+0000","last_clean":"2024-04-25T00:01:46.662201+0000","last_became_active":"2024-04-25T00:01:05.369987+0000","last_became_peered":"2024-04-25T00:01:05.369987+0000","last_unstale":"2024-04-25T00:01:46.662201+0000","last_undegraded":"2024-04-25T00:01:46.662201+0000","last_fullsized":"2024-04-25T00:01:46.662201+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657919+0000","last_change":"2024-04-25T00:01:04.370226+0000","last_active":"2024-04-25T00:01:46.657919+0000","last_peered":"2024-04-25T00:01:46.657919+0000","last_clean":"2024-04-25T00:01:46.657919+0000","last_became_active":"2024-04-25T00:01:04.369904+0000","last_became_peered":"2024-04-25T00:01:04.369904+0000","last_unstale":"2024-04-25T00:01:46.657919+0000","last_undegraded":"2024-04-25T00:01:46.657919+0000","last_fullsized":"2024-04-25T00:01:46.657919+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:47:25.595275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"27'1","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.652131+0000","last_change":"2024-04-25T00:01:18.497156+0000","last_active":"2024-04-25T00:01:46.652131+0000","last_peered":"2024-04-25T00:01:46.652131+0000","last_clean":"2024-04-25T00:01:46.652131+0000","last_became_active":"2024-04-25T00:01:18.495573+0000","last_became_peered":"2024-04-25T00:01:18.495573+0000","last_unstale":"2024-04-25T00:01:46.652131+0000","last_undegraded":"2024-04-25T00:01:46.652131+0000","last_fullsized":"2024-04-25T00:01:46.652131+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:29.119299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660477+0000","last_change":"2024-04-25T00:01:19.519411+0000","last_active":"2024-04-25T00:01:46.660477+0000","last_peered":"2024-04-25T00:01:46.660477+0000","last_clean":"2024-04-25T00:01:46.660477+0000","last_became_active":"2024-04-25T00:01:19.519020+0000","last_became_peered":"2024-04-25T00:01:19.519020+0000","last_unstale":"2024-04-25T00:01:46.660477+0000","last_undegraded":"2024-04-25T00:01:46.660477+0000","last_fullsized":"2024-04-25T00:01:46.660477+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:50.351599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.513395+0000","last_change":"2024-04-25T00:01:05.364939+0000","last_active":"2024-04-25T00:01:20.513395+0000","last_peered":"2024-04-25T00:01:20.513395+0000","last_clean":"2024-04-25T00:01:20.513395+0000","last_became_active":"2024-04-25T00:01:05.364272+0000","last_became_peered":"2024-04-25T00:01:05.364272+0000","last_unstale":"2024-04-25T00:01:20.513395+0000","last_undegraded":"2024-04-25T00:01:20.513395+0000","last_fullsized":"2024-04-25T00:01:20.513395+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:24:59.679908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.658666+0000","last_change":"2024-04-25T00:01:04.357028+0000","last_active":"2024-04-25T00:01:46.658666+0000","last_peered":"2024-04-25T00:01:46.658666+0000","last_clean":"2024-04-25T00:01:46.658666+0000","last_became_active":"2024-04-25T00:01:04.356644+0000","last_became_peered":"2024-04-25T00:01:04.356644+0000","last_unstale":"2024-04-25T00:01:46.658666+0000","last_undegraded":"2024-04-25T00:01:46.658666+0000","last_fullsized":"2024-04-25T00:01:46.658666+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:21.729893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":182,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657375+0000","last_change":"2024-04-25T00:00:55.084492+0000","last_active":"2024-04-25T00:01:46.657375+0000","last_peered":"2024-04-25T00:01:46.657375+0000","last_clean":"2024-04-25T00:01:46.657375+0000","last_became_active":"2024-04-25T00:00:55.083493+0000","last_became_peered":"2024-04-25T00:00:55.083493+0000","last_unstale":"2024-04-25T00:01:46.657375+0000","last_undegraded":"2024-04-25T00:01:46.657375+0000","last_fullsized":"2024-04-25T00:01:46.657375+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:00:54.060578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:00:54.060578+0000","last_clean_scrub_stamp":"2024-04-25T00:00:54.060578+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:14:58.472477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660166+0000","last_change":"2024-04-25T00:01:18.476298+0000","last_active":"2024-04-25T00:01:46.660166+0000","last_peered":"2024-04-25T00:01:46.660166+0000","last_clean":"2024-04-25T00:01:46.660166+0000","last_became_active":"2024-04-25T00:01:18.475898+0000","last_became_peered":"2024-04-25T00:01:18.475898+0000","last_unstale":"2024-04-25T00:01:46.660166+0000","last_undegraded":"2024-04-25T00:01:46.660166+0000","last_fullsized":"2024-04-25T00:01:46.660166+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:49.341842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.652839+0000","last_change":"2024-04-25T00:01:19.518102+0000","last_active":"2024-04-25T00:01:46.652839+0000","last_peered":"2024-04-25T00:01:46.652839+0000","last_clean":"2024-04-25T00:01:46.652839+0000","last_became_active":"2024-04-25T00:01:19.517376+0000","last_became_peered":"2024-04-25T00:01:19.517376+0000","last_unstale":"2024-04-25T00:01:46.652839+0000","last_undegraded":"2024-04-25T00:01:46.652839+0000","last_fullsized":"2024-04-25T00:01:46.652839+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:55:46.853565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659553+0000","last_change":"2024-04-25T00:01:05.376609+0000","last_active":"2024-04-25T00:01:46.659553+0000","last_peered":"2024-04-25T00:01:46.659553+0000","last_clean":"2024-04-25T00:01:46.659553+0000","last_became_active":"2024-04-25T00:01:05.375881+0000","last_became_peered":"2024-04-25T00:01:05.375881+0000","last_unstale":"2024-04-25T00:01:46.659553+0000","last_undegraded":"2024-04-25T00:01:46.659553+0000","last_fullsized":"2024-04-25T00:01:46.659553+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:46:05.592233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.666092+0000","last_change":"2024-04-25T00:01:04.368954+0000","last_active":"2024-04-25T00:01:46.666092+0000","last_peered":"2024-04-25T00:01:46.666092+0000","last_clean":"2024-04-25T00:01:46.666092+0000","last_became_active":"2024-04-25T00:01:04.367903+0000","last_became_peered":"2024-04-25T00:01:04.367903+0000","last_unstale":"2024-04-25T00:01:46.666092+0000","last_undegraded":"2024-04-25T00:01:46.666092+0000","last_fullsized":"2024-04-25T00:01:46.666092+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.647171+0000","last_change":"2024-04-25T00:01:18.476525+0000","last_active":"2024-04-25T00:01:46.647171+0000","last_peered":"2024-04-25T00:01:46.647171+0000","last_clean":"2024-04-25T00:01:46.647171+0000","last_became_active":"2024-04-25T00:01:18.476247+0000","last_became_peered":"2024-04-25T00:01:18.476247+0000","last_unstale":"2024-04-25T00:01:46.647171+0000","last_undegraded":"2024-04-25T00:01:46.647171+0000","last_fullsized":"2024-04-25T00:01:46.647171+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:30:36.682736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.662181+0000","last_change":"2024-04-25T00:01:19.514615+0000","last_active":"2024-04-25T00:01:46.662181+0000","last_peered":"2024-04-25T00:01:46.662181+0000","last_clean":"2024-04-25T00:01:46.662181+0000","last_became_active":"2024-04-25T00:01:19.513640+0000","last_became_peered":"2024-04-25T00:01:19.513640+0000","last_unstale":"2024-04-25T00:01:46.662181+0000","last_undegraded":"2024-04-25T00:01:46.662181+0000","last_fullsized":"2024-04-25T00:01:46.662181+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:08.620602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660785+0000","last_change":"2024-04-25T00:01:05.385670+0000","last_active":"2024-04-25T00:01:46.660785+0000","last_peered":"2024-04-25T00:01:46.660785+0000","last_clean":"2024-04-25T00:01:46.660785+0000","last_became_active":"2024-04-25T00:01:05.384822+0000","last_became_peered":"2024-04-25T00:01:05.384822+0000","last_unstale":"2024-04-25T00:01:46.660785+0000","last_undegraded":"2024-04-25T00:01:46.660785+0000","last_fullsized":"2024-04-25T00:01:46.660785+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:24.017263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"22'1","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.650515+0000","last_change":"2024-04-25T00:01:04.356627+0000","last_active":"2024-04-25T00:01:46.650515+0000","last_peered":"2024-04-25T00:01:46.650515+0000","last_clean":"2024-04-25T00:01:46.650515+0000","last_became_active":"2024-04-25T00:01:04.354419+0000","last_became_peered":"2024-04-25T00:01:04.354419+0000","last_unstale":"2024-04-25T00:01:46.650515+0000","last_undegraded":"2024-04-25T00:01:46.650515+0000","last_fullsized":"2024-04-25T00:01:46.650515+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:26.243635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"26'2","reported_seq":20,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.650452+0000","last_change":"2024-04-25T00:01:18.490511+0000","last_active":"2024-04-25T00:01:46.650452+0000","last_peered":"2024-04-25T00:01:46.650452+0000","last_clean":"2024-04-25T00:01:46.650452+0000","last_became_active":"2024-04-25T00:01:18.489629+0000","last_became_peered":"2024-04-25T00:01:18.489629+0000","last_unstale":"2024-04-25T00:01:46.650452+0000","last_undegraded":"2024-04-25T00:01:46.650452+0000","last_fullsized":"2024-04-25T00:01:46.650452+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:07.610846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657253+0000","last_change":"2024-04-25T00:01:19.508931+0000","last_active":"2024-04-25T00:01:46.657253+0000","last_peered":"2024-04-25T00:01:46.657253+0000","last_clean":"2024-04-25T00:01:46.657253+0000","last_became_active":"2024-04-25T00:01:19.507867+0000","last_became_peered":"2024-04-25T00:01:19.507867+0000","last_unstale":"2024-04-25T00:01:46.657253+0000","last_undegraded":"2024-04-25T00:01:46.657253+0000","last_fullsized":"2024-04-25T00:01:46.657253+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:38.152376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660540+0000","last_change":"2024-04-25T00:01:05.378840+0000","last_active":"2024-04-25T00:01:46.660540+0000","last_peered":"2024-04-25T00:01:46.660540+0000","last_clean":"2024-04-25T00:01:46.660540+0000","last_became_active":"2024-04-25T00:01:05.378112+0000","last_became_peered":"2024-04-25T00:01:05.378112+0000","last_unstale":"2024-04-25T00:01:46.660540+0000","last_undegraded":"2024-04-25T00:01:46.660540+0000","last_fullsized":"2024-04-25T00:01:46.660540+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:43.615895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"22'2","reported_seq":24,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.515607+0000","last_change":"2024-04-25T00:01:04.370794+0000","last_active":"2024-04-25T00:01:20.515607+0000","last_peered":"2024-04-25T00:01:20.515607+0000","last_clean":"2024-04-25T00:01:20.515607+0000","last_became_active":"2024-04-25T00:01:04.370099+0000","last_became_peered":"2024-04-25T00:01:04.370099+0000","last_unstale":"2024-04-25T00:01:20.515607+0000","last_undegraded":"2024-04-25T00:01:20.515607+0000","last_fullsized":"2024-04-25T00:01:20.515607+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:42:03.181036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657372+0000","last_change":"2024-04-25T00:01:18.475256+0000","last_active":"2024-04-25T00:01:46.657372+0000","last_peered":"2024-04-25T00:01:46.657372+0000","last_clean":"2024-04-25T00:01:46.657372+0000","last_became_active":"2024-04-25T00:01:18.474952+0000","last_became_peered":"2024-04-25T00:01:18.474952+0000","last_unstale":"2024-04-25T00:01:46.657372+0000","last_undegraded":"2024-04-25T00:01:46.657372+0000","last_fullsized":"2024-04-25T00:01:46.657372+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:46.849870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660859+0000","last_change":"2024-04-25T00:01:19.516833+0000","last_active":"2024-04-25T00:01:46.660859+0000","last_peered":"2024-04-25T00:01:46.660859+0000","last_clean":"2024-04-25T00:01:46.660859+0000","last_became_active":"2024-04-25T00:01:19.515926+0000","last_became_peered":"2024-04-25T00:01:19.515926+0000","last_unstale":"2024-04-25T00:01:46.660859+0000","last_undegraded":"2024-04-25T00:01:46.660859+0000","last_fullsized":"2024-04-25T00:01:46.660859+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:23:48.855381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.648328+0000","last_change":"2024-04-25T00:01:05.372591+0000","last_active":"2024-04-25T00:01:46.648328+0000","last_peered":"2024-04-25T00:01:46.648328+0000","last_clean":"2024-04-25T00:01:46.648328+0000","last_became_active":"2024-04-25T00:01:05.371204+0000","last_became_peered":"2024-04-25T00:01:05.371204+0000","last_unstale":"2024-04-25T00:01:46.648328+0000","last_undegraded":"2024-04-25T00:01:46.648328+0000","last_fullsized":"2024-04-25T00:01:46.648328+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:15.879673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"22'1","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660665+0000","last_change":"2024-04-25T00:01:04.372376+0000","last_active":"2024-04-25T00:01:46.660665+0000","last_peered":"2024-04-25T00:01:46.660665+0000","last_clean":"2024-04-25T00:01:46.660665+0000","last_became_active":"2024-04-25T00:01:04.372040+0000","last_became_peered":"2024-04-25T00:01:04.372040+0000","last_unstale":"2024-04-25T00:01:46.660665+0000","last_undegraded":"2024-04-25T00:01:46.660665+0000","last_fullsized":"2024-04-25T00:01:46.660665+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:21.729893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657413+0000","last_change":"2024-04-25T00:01:18.487548+0000","last_active":"2024-04-25T00:01:46.657413+0000","last_peered":"2024-04-25T00:01:46.657413+0000","last_clean":"2024-04-25T00:01:46.657413+0000","last_became_active":"2024-04-25T00:01:18.487227+0000","last_became_peered":"2024-04-25T00:01:18.487227+0000","last_unstale":"2024-04-25T00:01:46.657413+0000","last_undegraded":"2024-04-25T00:01:46.657413+0000","last_fullsized":"2024-04-25T00:01:46.657413+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:55.074018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.514828+0000","last_change":"2024-04-25T00:01:19.528574+0000","last_active":"2024-04-25T00:01:20.514828+0000","last_peered":"2024-04-25T00:01:20.514828+0000","last_clean":"2024-04-25T00:01:20.514828+0000","last_became_active":"2024-04-25T00:01:19.525531+0000","last_became_peered":"2024-04-25T00:01:19.525531+0000","last_unstale":"2024-04-25T00:01:20.514828+0000","last_undegraded":"2024-04-25T00:01:20.514828+0000","last_fullsized":"2024-04-25T00:01:20.514828+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:47:08.132895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657517+0000","last_change":"2024-04-25T00:01:05.364103+0000","last_active":"2024-04-25T00:01:46.657517+0000","last_peered":"2024-04-25T00:01:46.657517+0000","last_clean":"2024-04-25T00:01:46.657517+0000","last_became_active":"2024-04-25T00:01:05.363320+0000","last_became_peered":"2024-04-25T00:01:05.363320+0000","last_unstale":"2024-04-25T00:01:46.657517+0000","last_undegraded":"2024-04-25T00:01:46.657517+0000","last_fullsized":"2024-04-25T00:01:46.657517+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:31.456792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"22'8","reported_seq":30,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511508+0000","last_change":"2024-04-25T00:01:04.358365+0000","last_active":"2024-04-25T00:01:20.511508+0000","last_peered":"2024-04-25T00:01:20.511508+0000","last_clean":"2024-04-25T00:01:20.511508+0000","last_became_active":"2024-04-25T00:01:04.357571+0000","last_became_peered":"2024-04-25T00:01:04.357571+0000","last_unstale":"2024-04-25T00:01:20.511508+0000","last_undegraded":"2024-04-25T00:01:20.511508+0000","last_fullsized":"2024-04-25T00:01:20.511508+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:50:23.023114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.651115+0000","last_change":"2024-04-25T00:01:18.497679+0000","last_active":"2024-04-25T00:01:46.651115+0000","last_peered":"2024-04-25T00:01:46.651115+0000","last_clean":"2024-04-25T00:01:46.651115+0000","last_became_active":"2024-04-25T00:01:18.496489+0000","last_became_peered":"2024-04-25T00:01:18.496489+0000","last_unstale":"2024-04-25T00:01:46.651115+0000","last_undegraded":"2024-04-25T00:01:46.651115+0000","last_fullsized":"2024-04-25T00:01:46.651115+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:23:47.845624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.658432+0000","last_change":"2024-04-25T00:01:19.498082+0000","last_active":"2024-04-25T00:01:46.658432+0000","last_peered":"2024-04-25T00:01:46.658432+0000","last_clean":"2024-04-25T00:01:46.658432+0000","last_became_active":"2024-04-25T00:01:19.496805+0000","last_became_peered":"2024-04-25T00:01:19.496805+0000","last_unstale":"2024-04-25T00:01:46.658432+0000","last_undegraded":"2024-04-25T00:01:46.658432+0000","last_fullsized":"2024-04-25T00:01:46.658432+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:57:36.256250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657234+0000","last_change":"2024-04-25T00:01:05.373927+0000","last_active":"2024-04-25T00:01:46.657234+0000","last_peered":"2024-04-25T00:01:46.657234+0000","last_clean":"2024-04-25T00:01:46.657234+0000","last_became_active":"2024-04-25T00:01:05.373077+0000","last_became_peered":"2024-04-25T00:01:05.373077+0000","last_unstale":"2024-04-25T00:01:46.657234+0000","last_undegraded":"2024-04-25T00:01:46.657234+0000","last_fullsized":"2024-04-25T00:01:46.657234+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:50.489125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.658723+0000","last_change":"2024-04-25T00:01:04.357265+0000","last_active":"2024-04-25T00:01:46.658723+0000","last_peered":"2024-04-25T00:01:46.658723+0000","last_clean":"2024-04-25T00:01:46.658723+0000","last_became_active":"2024-04-25T00:01:04.357004+0000","last_became_peered":"2024-04-25T00:01:04.357004+0000","last_unstale":"2024-04-25T00:01:46.658723+0000","last_undegraded":"2024-04-25T00:01:46.658723+0000","last_fullsized":"2024-04-25T00:01:46.658723+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:47:25.595275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.653006+0000","last_change":"2024-04-25T00:01:18.487276+0000","last_active":"2024-04-25T00:01:46.653006+0000","last_peered":"2024-04-25T00:01:46.653006+0000","last_clean":"2024-04-25T00:01:46.653006+0000","last_became_active":"2024-04-25T00:01:18.486279+0000","last_became_peered":"2024-04-25T00:01:18.486279+0000","last_unstale":"2024-04-25T00:01:46.653006+0000","last_undegraded":"2024-04-25T00:01:46.653006+0000","last_fullsized":"2024-04-25T00:01:46.653006+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:17:37.141843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659022+0000","last_change":"2024-04-25T00:01:19.516827+0000","last_active":"2024-04-25T00:01:46.659022+0000","last_peered":"2024-04-25T00:01:46.659022+0000","last_clean":"2024-04-25T00:01:46.659022+0000","last_became_active":"2024-04-25T00:01:19.515765+0000","last_became_peered":"2024-04-25T00:01:19.515765+0000","last_unstale":"2024-04-25T00:01:46.659022+0000","last_undegraded":"2024-04-25T00:01:46.659022+0000","last_fullsized":"2024-04-25T00:01:46.659022+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:13:34.153117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.658378+0000","last_change":"2024-04-25T00:01:05.379501+0000","last_active":"2024-04-25T00:01:46.658378+0000","last_peered":"2024-04-25T00:01:46.658378+0000","last_clean":"2024-04-25T00:01:46.658378+0000","last_became_active":"2024-04-25T00:01:05.378644+0000","last_became_peered":"2024-04-25T00:01:05.378644+0000","last_unstale":"2024-04-25T00:01:46.658378+0000","last_undegraded":"2024-04-25T00:01:46.658378+0000","last_fullsized":"2024-04-25T00:01:46.658378+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:08:35.505154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.658345+0000","last_change":"2024-04-25T00:01:04.360987+0000","last_active":"2024-04-25T00:01:46.658345+0000","last_peered":"2024-04-25T00:01:46.658345+0000","last_clean":"2024-04-25T00:01:46.658345+0000","last_became_active":"2024-04-25T00:01:04.360664+0000","last_became_peered":"2024-04-25T00:01:04.360664+0000","last_unstale":"2024-04-25T00:01:46.658345+0000","last_undegraded":"2024-04-25T00:01:46.658345+0000","last_fullsized":"2024-04-25T00:01:46.658345+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:55.283076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"27'8","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.665406+0000","last_change":"2024-04-25T00:01:18.505577+0000","last_active":"2024-04-25T00:01:46.665406+0000","last_peered":"2024-04-25T00:01:46.665406+0000","last_clean":"2024-04-25T00:01:46.665406+0000","last_became_active":"2024-04-25T00:01:18.504963+0000","last_became_peered":"2024-04-25T00:01:18.504963+0000","last_unstale":"2024-04-25T00:01:46.665406+0000","last_undegraded":"2024-04-25T00:01:46.665406+0000","last_fullsized":"2024-04-25T00:01:46.665406+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:37.142619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2053,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":10,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.662316+0000","last_change":"2024-04-25T00:01:46.658442+0000","last_active":"2024-04-25T00:01:46.662316+0000","last_peered":"2024-04-25T00:01:46.662316+0000","last_clean":"2024-04-25T00:01:46.662316+0000","last_became_active":"2024-04-25T00:01:46.657588+0000","last_became_peered":"2024-04-25T00:01:46.657588+0000","last_unstale":"2024-04-25T00:01:46.662316+0000","last_undegraded":"2024-04-25T00:01:46.662316+0000","last_fullsized":"2024-04-25T00:01:46.662316+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:58:49.738233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.663479+0000","last_change":"2024-04-25T00:01:05.379102+0000","last_active":"2024-04-25T00:01:46.663479+0000","last_peered":"2024-04-25T00:01:46.663479+0000","last_clean":"2024-04-25T00:01:46.663479+0000","last_became_active":"2024-04-25T00:01:05.377625+0000","last_became_peered":"2024-04-25T00:01:05.377625+0000","last_unstale":"2024-04-25T00:01:46.663479+0000","last_undegraded":"2024-04-25T00:01:46.663479+0000","last_fullsized":"2024-04-25T00:01:46.663479+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:15.879673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"22'2","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660428+0000","last_change":"2024-04-25T00:01:04.370640+0000","last_active":"2024-04-25T00:01:46.660428+0000","last_peered":"2024-04-25T00:01:46.660428+0000","last_clean":"2024-04-25T00:01:46.660428+0000","last_became_active":"2024-04-25T00:01:04.370031+0000","last_became_peered":"2024-04-25T00:01:04.370031+0000","last_unstale":"2024-04-25T00:01:46.660428+0000","last_undegraded":"2024-04-25T00:01:46.660428+0000","last_fullsized":"2024-04-25T00:01:46.660428+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:26.243635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"26'3","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659370+0000","last_change":"2024-04-25T00:01:18.473113+0000","last_active":"2024-04-25T00:01:46.659370+0000","last_peered":"2024-04-25T00:01:46.659370+0000","last_clean":"2024-04-25T00:01:46.659370+0000","last_became_active":"2024-04-25T00:01:18.472931+0000","last_became_peered":"2024-04-25T00:01:18.472931+0000","last_unstale":"2024-04-25T00:01:46.659370+0000","last_undegraded":"2024-04-25T00:01:46.659370+0000","last_fullsized":"2024-04-25T00:01:46.659370+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:07.498985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.512763+0000","last_change":"2024-04-25T00:01:19.533664+0000","last_active":"2024-04-25T00:01:20.512763+0000","last_peered":"2024-04-25T00:01:20.512763+0000","last_clean":"2024-04-25T00:01:20.512763+0000","last_became_active":"2024-04-25T00:01:19.533048+0000","last_became_peered":"2024-04-25T00:01:19.533048+0000","last_unstale":"2024-04-25T00:01:20.512763+0000","last_undegraded":"2024-04-25T00:01:20.512763+0000","last_fullsized":"2024-04-25T00:01:20.512763+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:45:43.141569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660317+0000","last_change":"2024-04-25T00:01:05.385827+0000","last_active":"2024-04-25T00:01:46.660317+0000","last_peered":"2024-04-25T00:01:46.660317+0000","last_clean":"2024-04-25T00:01:46.660317+0000","last_became_active":"2024-04-25T00:01:05.385258+0000","last_became_peered":"2024-04-25T00:01:05.385258+0000","last_unstale":"2024-04-25T00:01:46.660317+0000","last_undegraded":"2024-04-25T00:01:46.660317+0000","last_fullsized":"2024-04-25T00:01:46.660317+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:23.870121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"27'8","reported_seq":34,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.648163+0000","last_change":"2024-04-25T00:01:04.365185+0000","last_active":"2024-04-25T00:01:46.648163+0000","last_peered":"2024-04-25T00:01:46.648163+0000","last_clean":"2024-04-25T00:01:46.648163+0000","last_became_active":"2024-04-25T00:01:04.364930+0000","last_became_peered":"2024-04-25T00:01:04.364930+0000","last_unstale":"2024-04-25T00:01:46.648163+0000","last_undegraded":"2024-04-25T00:01:46.648163+0000","last_fullsized":"2024-04-25T00:01:46.648163+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:45:03.618340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2053,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.512982+0000","last_change":"2024-04-25T00:01:18.496220+0000","last_active":"2024-04-25T00:01:20.512982+0000","last_peered":"2024-04-25T00:01:20.512982+0000","last_clean":"2024-04-25T00:01:20.512982+0000","last_became_active":"2024-04-25T00:01:18.492850+0000","last_became_peered":"2024-04-25T00:01:18.492850+0000","last_unstale":"2024-04-25T00:01:20.512982+0000","last_undegraded":"2024-04-25T00:01:20.512982+0000","last_fullsized":"2024-04-25T00:01:20.512982+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:50:16.922314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.662739+0000","last_change":"2024-04-25T00:01:19.515725+0000","last_active":"2024-04-25T00:01:46.662739+0000","last_peered":"2024-04-25T00:01:46.662739+0000","last_clean":"2024-04-25T00:01:46.662739+0000","last_became_active":"2024-04-25T00:01:19.509285+0000","last_became_peered":"2024-04-25T00:01:19.509285+0000","last_unstale":"2024-04-25T00:01:46.662739+0000","last_undegraded":"2024-04-25T00:01:46.662739+0000","last_fullsized":"2024-04-25T00:01:46.662739+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:25:20.185355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.658824+0000","last_change":"2024-04-25T00:01:05.371061+0000","last_active":"2024-04-25T00:01:46.658824+0000","last_peered":"2024-04-25T00:01:46.658824+0000","last_clean":"2024-04-25T00:01:46.658824+0000","last_became_active":"2024-04-25T00:01:05.369950+0000","last_became_peered":"2024-04-25T00:01:05.369950+0000","last_unstale":"2024-04-25T00:01:46.658824+0000","last_undegraded":"2024-04-25T00:01:46.658824+0000","last_fullsized":"2024-04-25T00:01:46.658824+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:02:43.237902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"22'3","reported_seq":29,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.665160+0000","last_change":"2024-04-25T00:01:04.366503+0000","last_active":"2024-04-25T00:01:46.665160+0000","last_peered":"2024-04-25T00:01:46.665160+0000","last_clean":"2024-04-25T00:01:46.665160+0000","last_became_active":"2024-04-25T00:01:04.365945+0000","last_became_peered":"2024-04-25T00:01:04.365945+0000","last_unstale":"2024-04-25T00:01:46.665160+0000","last_undegraded":"2024-04-25T00:01:46.665160+0000","last_fullsized":"2024-04-25T00:01:46.665160+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"26'2","reported_seq":20,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.647716+0000","last_change":"2024-04-25T00:01:18.476456+0000","last_active":"2024-04-25T00:01:46.647716+0000","last_peered":"2024-04-25T00:01:46.647716+0000","last_clean":"2024-04-25T00:01:46.647716+0000","last_became_active":"2024-04-25T00:01:18.475876+0000","last_became_peered":"2024-04-25T00:01:18.475876+0000","last_unstale":"2024-04-25T00:01:46.647716+0000","last_undegraded":"2024-04-25T00:01:46.647716+0000","last_fullsized":"2024-04-25T00:01:46.647716+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:56.741250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657126+0000","last_change":"2024-04-25T00:01:19.515376+0000","last_active":"2024-04-25T00:01:46.657126+0000","last_peered":"2024-04-25T00:01:46.657126+0000","last_clean":"2024-04-25T00:01:46.657126+0000","last_became_active":"2024-04-25T00:01:19.512425+0000","last_became_peered":"2024-04-25T00:01:19.512425+0000","last_unstale":"2024-04-25T00:01:46.657126+0000","last_undegraded":"2024-04-25T00:01:46.657126+0000","last_fullsized":"2024-04-25T00:01:46.657126+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:43:16.957273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.654022+0000","last_change":"2024-04-25T00:01:05.361723+0000","last_active":"2024-04-25T00:01:46.654022+0000","last_peered":"2024-04-25T00:01:46.654022+0000","last_clean":"2024-04-25T00:01:46.654022+0000","last_became_active":"2024-04-25T00:01:05.360956+0000","last_became_peered":"2024-04-25T00:01:05.360956+0000","last_unstale":"2024-04-25T00:01:46.654022+0000","last_undegraded":"2024-04-25T00:01:46.654022+0000","last_fullsized":"2024-04-25T00:01:46.654022+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:23.870121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660056+0000","last_change":"2024-04-25T00:01:04.362129+0000","last_active":"2024-04-25T00:01:46.660056+0000","last_peered":"2024-04-25T00:01:46.660056+0000","last_clean":"2024-04-25T00:01:46.660056+0000","last_became_active":"2024-04-25T00:01:04.361827+0000","last_became_peered":"2024-04-25T00:01:04.361827+0000","last_unstale":"2024-04-25T00:01:46.660056+0000","last_undegraded":"2024-04-25T00:01:46.660056+0000","last_fullsized":"2024-04-25T00:01:46.660056+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:26.243635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"26'8","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659186+0000","last_change":"2024-04-25T00:01:18.496129+0000","last_active":"2024-04-25T00:01:46.659186+0000","last_peered":"2024-04-25T00:01:46.659186+0000","last_clean":"2024-04-25T00:01:46.659186+0000","last_became_active":"2024-04-25T00:01:18.495415+0000","last_became_peered":"2024-04-25T00:01:18.495415+0000","last_unstale":"2024-04-25T00:01:46.659186+0000","last_undegraded":"2024-04-25T00:01:46.659186+0000","last_fullsized":"2024-04-25T00:01:46.659186+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:38.089411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.646840+0000","last_change":"2024-04-25T00:01:19.509542+0000","last_active":"2024-04-25T00:01:46.646840+0000","last_peered":"2024-04-25T00:01:46.646840+0000","last_clean":"2024-04-25T00:01:46.646840+0000","last_became_active":"2024-04-25T00:01:19.509015+0000","last_became_peered":"2024-04-25T00:01:19.509015+0000","last_unstale":"2024-04-25T00:01:46.646840+0000","last_undegraded":"2024-04-25T00:01:46.646840+0000","last_fullsized":"2024-04-25T00:01:46.646840+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:25:20.185355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.648263+0000","last_change":"2024-04-25T00:01:05.365413+0000","last_active":"2024-04-25T00:01:46.648263+0000","last_peered":"2024-04-25T00:01:46.648263+0000","last_clean":"2024-04-25T00:01:46.648263+0000","last_became_active":"2024-04-25T00:01:05.361865+0000","last_became_peered":"2024-04-25T00:01:05.361865+0000","last_unstale":"2024-04-25T00:01:46.648263+0000","last_undegraded":"2024-04-25T00:01:46.648263+0000","last_fullsized":"2024-04-25T00:01:46.648263+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:30.228060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.658791+0000","last_change":"2024-04-25T00:01:04.360804+0000","last_active":"2024-04-25T00:01:46.658791+0000","last_peered":"2024-04-25T00:01:46.658791+0000","last_clean":"2024-04-25T00:01:46.658791+0000","last_became_active":"2024-04-25T00:01:04.360374+0000","last_became_peered":"2024-04-25T00:01:04.360374+0000","last_unstale":"2024-04-25T00:01:46.658791+0000","last_undegraded":"2024-04-25T00:01:46.658791+0000","last_fullsized":"2024-04-25T00:01:46.658791+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511980+0000","last_change":"2024-04-25T00:01:18.480003+0000","last_active":"2024-04-25T00:01:20.511980+0000","last_peered":"2024-04-25T00:01:20.511980+0000","last_clean":"2024-04-25T00:01:20.511980+0000","last_became_active":"2024-04-25T00:01:18.479270+0000","last_became_peered":"2024-04-25T00:01:18.479270+0000","last_unstale":"2024-04-25T00:01:20.511980+0000","last_undegraded":"2024-04-25T00:01:20.511980+0000","last_fullsized":"2024-04-25T00:01:20.511980+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:52:40.942502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.648354+0000","last_change":"2024-04-25T00:01:19.524463+0000","last_active":"2024-04-25T00:01:46.648354+0000","last_peered":"2024-04-25T00:01:46.648354+0000","last_clean":"2024-04-25T00:01:46.648354+0000","last_became_active":"2024-04-25T00:01:19.524166+0000","last_became_peered":"2024-04-25T00:01:19.524166+0000","last_unstale":"2024-04-25T00:01:46.648354+0000","last_undegraded":"2024-04-25T00:01:46.648354+0000","last_fullsized":"2024-04-25T00:01:46.648354+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:50.351599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"26'1","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.514720+0000","last_change":"2024-04-25T00:01:18.497096+0000","last_active":"2024-04-25T00:01:20.514720+0000","last_peered":"2024-04-25T00:01:20.514720+0000","last_clean":"2024-04-25T00:01:20.514720+0000","last_became_active":"2024-04-25T00:01:18.496289+0000","last_became_peered":"2024-04-25T00:01:18.496289+0000","last_unstale":"2024-04-25T00:01:20.514720+0000","last_undegraded":"2024-04-25T00:01:20.514720+0000","last_fullsized":"2024-04-25T00:01:20.514720+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:59:40.316831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.512256+0000","last_change":"2024-04-25T00:01:05.370043+0000","last_active":"2024-04-25T00:01:20.512256+0000","last_peered":"2024-04-25T00:01:20.512256+0000","last_clean":"2024-04-25T00:01:20.512256+0000","last_became_active":"2024-04-25T00:01:05.368080+0000","last_became_peered":"2024-04-25T00:01:05.368080+0000","last_unstale":"2024-04-25T00:01:20.512256+0000","last_undegraded":"2024-04-25T00:01:20.512256+0000","last_fullsized":"2024-04-25T00:01:20.512256+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:58:07.610923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657790+0000","last_change":"2024-04-25T00:01:04.364523+0000","last_active":"2024-04-25T00:01:46.657790+0000","last_peered":"2024-04-25T00:01:46.657790+0000","last_clean":"2024-04-25T00:01:46.657790+0000","last_became_active":"2024-04-25T00:01:04.364050+0000","last_became_peered":"2024-04-25T00:01:04.364050+0000","last_unstale":"2024-04-25T00:01:46.657790+0000","last_undegraded":"2024-04-25T00:01:46.657790+0000","last_fullsized":"2024-04-25T00:01:46.657790+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:49.479804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.664982+0000","last_change":"2024-04-25T00:01:19.491523+0000","last_active":"2024-04-25T00:01:46.664982+0000","last_peered":"2024-04-25T00:01:46.664982+0000","last_clean":"2024-04-25T00:01:46.664982+0000","last_became_active":"2024-04-25T00:01:19.490755+0000","last_became_peered":"2024-04-25T00:01:19.490755+0000","last_unstale":"2024-04-25T00:01:46.664982+0000","last_undegraded":"2024-04-25T00:01:46.664982+0000","last_fullsized":"2024-04-25T00:01:46.664982+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:47:19.526845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"26'4","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659642+0000","last_change":"2024-04-25T00:01:18.472794+0000","last_active":"2024-04-25T00:01:46.659642+0000","last_peered":"2024-04-25T00:01:46.659642+0000","last_clean":"2024-04-25T00:01:46.659642+0000","last_became_active":"2024-04-25T00:01:18.472534+0000","last_became_peered":"2024-04-25T00:01:18.472534+0000","last_unstale":"2024-04-25T00:01:46.659642+0000","last_undegraded":"2024-04-25T00:01:46.659642+0000","last_fullsized":"2024-04-25T00:01:46.659642+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:55:45.843808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.649586+0000","last_change":"2024-04-25T00:01:05.378867+0000","last_active":"2024-04-25T00:01:46.649586+0000","last_peered":"2024-04-25T00:01:46.649586+0000","last_clean":"2024-04-25T00:01:46.649586+0000","last_became_active":"2024-04-25T00:01:05.378056+0000","last_became_peered":"2024-04-25T00:01:05.378056+0000","last_unstale":"2024-04-25T00:01:46.649586+0000","last_undegraded":"2024-04-25T00:01:46.649586+0000","last_fullsized":"2024-04-25T00:01:46.649586+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:36:10.377757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.662462+0000","last_change":"2024-04-25T00:01:04.370896+0000","last_active":"2024-04-25T00:01:46.662462+0000","last_peered":"2024-04-25T00:01:46.662462+0000","last_clean":"2024-04-25T00:01:46.662462+0000","last_became_active":"2024-04-25T00:01:04.370273+0000","last_became_peered":"2024-04-25T00:01:04.370273+0000","last_unstale":"2024-04-25T00:01:46.662462+0000","last_undegraded":"2024-04-25T00:01:46.662462+0000","last_fullsized":"2024-04-25T00:01:46.662462+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:49:22.673421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.514211+0000","last_change":"2024-04-25T00:01:19.521143+0000","last_active":"2024-04-25T00:01:20.514211+0000","last_peered":"2024-04-25T00:01:20.514211+0000","last_clean":"2024-04-25T00:01:20.514211+0000","last_became_active":"2024-04-25T00:01:19.514664+0000","last_became_peered":"2024-04-25T00:01:19.514664+0000","last_unstale":"2024-04-25T00:01:20.514211+0000","last_undegraded":"2024-04-25T00:01:20.514211+0000","last_fullsized":"2024-04-25T00:01:20.514211+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:03:47.440442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"22'1","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.658205+0000","last_change":"2024-04-25T00:01:04.360254+0000","last_active":"2024-04-25T00:01:46.658205+0000","last_peered":"2024-04-25T00:01:46.658205+0000","last_clean":"2024-04-25T00:01:46.658205+0000","last_became_active":"2024-04-25T00:01:04.359586+0000","last_became_peered":"2024-04-25T00:01:04.359586+0000","last_unstale":"2024-04-25T00:01:46.658205+0000","last_undegraded":"2024-04-25T00:01:46.658205+0000","last_fullsized":"2024-04-25T00:01:46.658205+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:47.192930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.513293+0000","last_change":"2024-04-25T00:01:05.370723+0000","last_active":"2024-04-25T00:01:20.513293+0000","last_peered":"2024-04-25T00:01:20.513293+0000","last_clean":"2024-04-25T00:01:20.513293+0000","last_became_active":"2024-04-25T00:01:05.368970+0000","last_became_peered":"2024-04-25T00:01:05.368970+0000","last_unstale":"2024-04-25T00:01:20.513293+0000","last_undegraded":"2024-04-25T00:01:20.513293+0000","last_fullsized":"2024-04-25T00:01:20.513293+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:06:53.743074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"26'2","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.513582+0000","last_change":"2024-04-25T00:01:18.498570+0000","last_active":"2024-04-25T00:01:20.513582+0000","last_peered":"2024-04-25T00:01:20.513582+0000","last_clean":"2024-04-25T00:01:20.513582+0000","last_became_active":"2024-04-25T00:01:18.497745+0000","last_became_peered":"2024-04-25T00:01:18.497745+0000","last_unstale":"2024-04-25T00:01:20.513582+0000","last_undegraded":"2024-04-25T00:01:20.513582+0000","last_fullsized":"2024-04-25T00:01:20.513582+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:26:42.047149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.647984+0000","last_change":"2024-04-25T00:01:19.514919+0000","last_active":"2024-04-25T00:01:46.647984+0000","last_peered":"2024-04-25T00:01:46.647984+0000","last_clean":"2024-04-25T00:01:46.647984+0000","last_became_active":"2024-04-25T00:01:19.514280+0000","last_became_peered":"2024-04-25T00:01:19.514280+0000","last_unstale":"2024-04-25T00:01:46.647984+0000","last_undegraded":"2024-04-25T00:01:46.647984+0000","last_fullsized":"2024-04-25T00:01:46.647984+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:37.975265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"22'1","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.653328+0000","last_change":"2024-04-25T00:01:04.365607+0000","last_active":"2024-04-25T00:01:46.653328+0000","last_peered":"2024-04-25T00:01:46.653328+0000","last_clean":"2024-04-25T00:01:46.653328+0000","last_became_active":"2024-04-25T00:01:04.364788+0000","last_became_peered":"2024-04-25T00:01:04.364788+0000","last_unstale":"2024-04-25T00:01:46.653328+0000","last_undegraded":"2024-04-25T00:01:46.653328+0000","last_fullsized":"2024-04-25T00:01:46.653328+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.661022+0000","last_change":"2024-04-25T00:01:05.375142+0000","last_active":"2024-04-25T00:01:46.661022+0000","last_peered":"2024-04-25T00:01:46.661022+0000","last_clean":"2024-04-25T00:01:46.661022+0000","last_became_active":"2024-04-25T00:01:05.373857+0000","last_became_peered":"2024-04-25T00:01:05.373857+0000","last_unstale":"2024-04-25T00:01:46.661022+0000","last_undegraded":"2024-04-25T00:01:46.661022+0000","last_fullsized":"2024-04-25T00:01:46.661022+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:15.879673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"26'2","reported_seq":20,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.664841+0000","last_change":"2024-04-25T00:01:18.502786+0000","last_active":"2024-04-25T00:01:46.664841+0000","last_peered":"2024-04-25T00:01:46.664841+0000","last_clean":"2024-04-25T00:01:46.664841+0000","last_became_active":"2024-04-25T00:01:18.501785+0000","last_became_peered":"2024-04-25T00:01:18.501785+0000","last_unstale":"2024-04-25T00:01:46.664841+0000","last_undegraded":"2024-04-25T00:01:46.664841+0000","last_fullsized":"2024-04-25T00:01:46.664841+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:09:31.441044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.514560+0000","last_change":"2024-04-25T00:01:19.511239+0000","last_active":"2024-04-25T00:01:20.514560+0000","last_peered":"2024-04-25T00:01:20.514560+0000","last_clean":"2024-04-25T00:01:20.514560+0000","last_became_active":"2024-04-25T00:01:19.510155+0000","last_became_peered":"2024-04-25T00:01:19.510155+0000","last_unstale":"2024-04-25T00:01:20.514560+0000","last_undegraded":"2024-04-25T00:01:20.514560+0000","last_fullsized":"2024-04-25T00:01:20.514560+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:43:14.452663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.647454+0000","last_change":"2024-04-25T00:01:05.378517+0000","last_active":"2024-04-25T00:01:46.647454+0000","last_peered":"2024-04-25T00:01:46.647454+0000","last_clean":"2024-04-25T00:01:46.647454+0000","last_became_active":"2024-04-25T00:01:05.377742+0000","last_became_peered":"2024-04-25T00:01:05.377742+0000","last_unstale":"2024-04-25T00:01:46.647454+0000","last_undegraded":"2024-04-25T00:01:46.647454+0000","last_fullsized":"2024-04-25T00:01:46.647454+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:06.450369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.514828+0000","last_change":"2024-04-25T00:01:19.508999+0000","last_active":"2024-04-25T00:01:20.514828+0000","last_peered":"2024-04-25T00:01:20.514828+0000","last_clean":"2024-04-25T00:01:20.514828+0000","last_became_active":"2024-04-25T00:01:19.507960+0000","last_became_peered":"2024-04-25T00:01:19.507960+0000","last_unstale":"2024-04-25T00:01:20.514828+0000","last_undegraded":"2024-04-25T00:01:20.514828+0000","last_fullsized":"2024-04-25T00:01:20.514828+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:32:40.274908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659704+0000","last_change":"2024-04-25T00:01:05.372927+0000","last_active":"2024-04-25T00:01:46.659704+0000","last_peered":"2024-04-25T00:01:46.659704+0000","last_clean":"2024-04-25T00:01:46.659704+0000","last_became_active":"2024-04-25T00:01:05.372050+0000","last_became_peered":"2024-04-25T00:01:05.372050+0000","last_unstale":"2024-04-25T00:01:46.659704+0000","last_undegraded":"2024-04-25T00:01:46.659704+0000","last_fullsized":"2024-04-25T00:01:46.659704+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:48.202251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.647316+0000","last_change":"2024-04-25T00:01:19.531072+0000","last_active":"2024-04-25T00:01:46.647316+0000","last_peered":"2024-04-25T00:01:46.647316+0000","last_clean":"2024-04-25T00:01:46.647316+0000","last_became_active":"2024-04-25T00:01:19.530836+0000","last_became_peered":"2024-04-25T00:01:19.530836+0000","last_unstale":"2024-04-25T00:01:46.647316+0000","last_undegraded":"2024-04-25T00:01:46.647316+0000","last_fullsized":"2024-04-25T00:01:46.647316+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:08.508741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657903+0000","last_change":"2024-04-25T00:01:05.362960+0000","last_active":"2024-04-25T00:01:46.657903+0000","last_peered":"2024-04-25T00:01:46.657903+0000","last_clean":"2024-04-25T00:01:46.657903+0000","last_became_active":"2024-04-25T00:01:05.361067+0000","last_became_peered":"2024-04-25T00:01:05.361067+0000","last_unstale":"2024-04-25T00:01:46.657903+0000","last_undegraded":"2024-04-25T00:01:46.657903+0000","last_fullsized":"2024-04-25T00:01:46.657903+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:24.017263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.650119+0000","last_change":"2024-04-25T00:01:19.506029+0000","last_active":"2024-04-25T00:01:46.650119+0000","last_peered":"2024-04-25T00:01:46.650119+0000","last_clean":"2024-04-25T00:01:46.650119+0000","last_became_active":"2024-04-25T00:01:19.505184+0000","last_became_peered":"2024-04-25T00:01:19.505184+0000","last_unstale":"2024-04-25T00:01:46.650119+0000","last_undegraded":"2024-04-25T00:01:46.650119+0000","last_fullsized":"2024-04-25T00:01:46.650119+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:08.508741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.653074+0000","last_change":"2024-04-25T00:01:05.358690+0000","last_active":"2024-04-25T00:01:46.653074+0000","last_peered":"2024-04-25T00:01:46.653074+0000","last_clean":"2024-04-25T00:01:46.653074+0000","last_became_active":"2024-04-25T00:01:05.357340+0000","last_became_peered":"2024-04-25T00:01:05.357340+0000","last_unstale":"2024-04-25T00:01:46.653074+0000","last_undegraded":"2024-04-25T00:01:46.653074+0000","last_fullsized":"2024-04-25T00:01:46.653074+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:36:55.436661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.658667+0000","last_change":"2024-04-25T00:01:19.508691+0000","last_active":"2024-04-25T00:01:46.658667+0000","last_peered":"2024-04-25T00:01:46.658667+0000","last_clean":"2024-04-25T00:01:46.658667+0000","last_became_active":"2024-04-25T00:01:19.508123+0000","last_became_peered":"2024-04-25T00:01:19.508123+0000","last_unstale":"2024-04-25T00:01:46.658667+0000","last_undegraded":"2024-04-25T00:01:46.658667+0000","last_fullsized":"2024-04-25T00:01:46.658667+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:47:19.526845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.665745+0000","last_change":"2024-04-25T00:01:05.387057+0000","last_active":"2024-04-25T00:01:46.665745+0000","last_peered":"2024-04-25T00:01:46.665745+0000","last_clean":"2024-04-25T00:01:46.665745+0000","last_became_active":"2024-04-25T00:01:05.386724+0000","last_became_peered":"2024-04-25T00:01:05.386724+0000","last_unstale":"2024-04-25T00:01:46.665745+0000","last_undegraded":"2024-04-25T00:01:46.665745+0000","last_fullsized":"2024-04-25T00:01:46.665745+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:55:09.073286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659072+0000","last_change":"2024-04-25T00:01:19.522760+0000","last_active":"2024-04-25T00:01:46.659072+0000","last_peered":"2024-04-25T00:01:46.659072+0000","last_clean":"2024-04-25T00:01:46.659072+0000","last_became_active":"2024-04-25T00:01:19.522380+0000","last_became_peered":"2024-04-25T00:01:19.522380+0000","last_unstale":"2024-04-25T00:01:46.659072+0000","last_undegraded":"2024-04-25T00:01:46.659072+0000","last_fullsized":"2024-04-25T00:01:46.659072+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:55:46.853565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.665267+0000","last_change":"2024-04-25T00:01:05.367296+0000","last_active":"2024-04-25T00:01:46.665267+0000","last_peered":"2024-04-25T00:01:46.665267+0000","last_clean":"2024-04-25T00:01:46.665267+0000","last_became_active":"2024-04-25T00:01:05.363838+0000","last_became_peered":"2024-04-25T00:01:05.363838+0000","last_unstale":"2024-04-25T00:01:46.665267+0000","last_undegraded":"2024-04-25T00:01:46.665267+0000","last_fullsized":"2024-04-25T00:01:46.665267+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:43:07.053074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657236+0000","last_change":"2024-04-25T00:01:19.510902+0000","last_active":"2024-04-25T00:01:46.657236+0000","last_peered":"2024-04-25T00:01:46.657236+0000","last_clean":"2024-04-25T00:01:46.657236+0000","last_became_active":"2024-04-25T00:01:19.510008+0000","last_became_peered":"2024-04-25T00:01:19.510008+0000","last_unstale":"2024-04-25T00:01:46.657236+0000","last_undegraded":"2024-04-25T00:01:46.657236+0000","last_fullsized":"2024-04-25T00:01:46.657236+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:40:32.292140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.648277+0000","last_change":"2024-04-25T00:01:05.365761+0000","last_active":"2024-04-25T00:01:46.648277+0000","last_peered":"2024-04-25T00:01:46.648277+0000","last_clean":"2024-04-25T00:01:46.648277+0000","last_became_active":"2024-04-25T00:01:05.361119+0000","last_became_peered":"2024-04-25T00:01:05.361119+0000","last_unstale":"2024-04-25T00:01:46.648277+0000","last_undegraded":"2024-04-25T00:01:46.648277+0000","last_fullsized":"2024-04-25T00:01:46.648277+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.647042+0000","last_change":"2024-04-25T00:01:19.518534+0000","last_active":"2024-04-25T00:01:46.647042+0000","last_peered":"2024-04-25T00:01:46.647042+0000","last_clean":"2024-04-25T00:01:46.647042+0000","last_became_active":"2024-04-25T00:01:19.518110+0000","last_became_peered":"2024-04-25T00:01:19.518110+0000","last_unstale":"2024-04-25T00:01:46.647042+0000","last_undegraded":"2024-04-25T00:01:46.647042+0000","last_fullsized":"2024-04-25T00:01:46.647042+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:05:09.219823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.653554+0000","last_change":"2024-04-25T00:01:05.373464+0000","last_active":"2024-04-25T00:01:46.653554+0000","last_peered":"2024-04-25T00:01:46.653554+0000","last_clean":"2024-04-25T00:01:46.653554+0000","last_became_active":"2024-04-25T00:01:05.372611+0000","last_became_peered":"2024-04-25T00:01:05.372611+0000","last_unstale":"2024-04-25T00:01:46.653554+0000","last_undegraded":"2024-04-25T00:01:46.653554+0000","last_fullsized":"2024-04-25T00:01:46.653554+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:31.456792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.664300+0000","last_change":"2024-04-25T00:01:19.542475+0000","last_active":"2024-04-25T00:01:46.664300+0000","last_peered":"2024-04-25T00:01:46.664300+0000","last_clean":"2024-04-25T00:01:46.664300+0000","last_became_active":"2024-04-25T00:01:19.541877+0000","last_became_peered":"2024-04-25T00:01:19.541877+0000","last_unstale":"2024-04-25T00:01:46.664300+0000","last_undegraded":"2024-04-25T00:01:46.664300+0000","last_fullsized":"2024-04-25T00:01:46.664300+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:02:57.373014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657634+0000","last_change":"2024-04-25T00:01:05.372906+0000","last_active":"2024-04-25T00:01:46.657634+0000","last_peered":"2024-04-25T00:01:46.657634+0000","last_clean":"2024-04-25T00:01:46.657634+0000","last_became_active":"2024-04-25T00:01:05.372148+0000","last_became_peered":"2024-04-25T00:01:05.372148+0000","last_unstale":"2024-04-25T00:01:46.657634+0000","last_undegraded":"2024-04-25T00:01:46.657634+0000","last_fullsized":"2024-04-25T00:01:46.657634+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:43.615895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659888+0000","last_change":"2024-04-25T00:01:19.498515+0000","last_active":"2024-04-25T00:01:46.659888+0000","last_peered":"2024-04-25T00:01:46.659888+0000","last_clean":"2024-04-25T00:01:46.659888+0000","last_became_active":"2024-04-25T00:01:19.496310+0000","last_became_peered":"2024-04-25T00:01:19.496310+0000","last_unstale":"2024-04-25T00:01:46.659888+0000","last_undegraded":"2024-04-25T00:01:46.659888+0000","last_fullsized":"2024-04-25T00:01:46.659888+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:20:06.661558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.651069+0000","last_change":"2024-04-25T00:01:05.366340+0000","last_active":"2024-04-25T00:01:46.651069+0000","last_peered":"2024-04-25T00:01:46.651069+0000","last_clean":"2024-04-25T00:01:46.651069+0000","last_became_active":"2024-04-25T00:01:05.361952+0000","last_became_peered":"2024-04-25T00:01:05.361952+0000","last_unstale":"2024-04-25T00:01:46.651069+0000","last_undegraded":"2024-04-25T00:01:46.651069+0000","last_fullsized":"2024-04-25T00:01:46.651069+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:41.948662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.661872+0000","last_change":"2024-04-25T00:01:19.508257+0000","last_active":"2024-04-25T00:01:46.661872+0000","last_peered":"2024-04-25T00:01:46.661872+0000","last_clean":"2024-04-25T00:01:46.661872+0000","last_became_active":"2024-04-25T00:01:19.507516+0000","last_became_peered":"2024-04-25T00:01:19.507516+0000","last_unstale":"2024-04-25T00:01:46.661872+0000","last_undegraded":"2024-04-25T00:01:46.661872+0000","last_fullsized":"2024-04-25T00:01:46.661872+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:45.121513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660969+0000","last_change":"2024-04-25T00:01:05.387457+0000","last_active":"2024-04-25T00:01:46.660969+0000","last_peered":"2024-04-25T00:01:46.660969+0000","last_clean":"2024-04-25T00:01:46.660969+0000","last_became_active":"2024-04-25T00:01:05.386919+0000","last_became_peered":"2024-04-25T00:01:05.386919+0000","last_unstale":"2024-04-25T00:01:46.660969+0000","last_undegraded":"2024-04-25T00:01:46.660969+0000","last_fullsized":"2024-04-25T00:01:46.660969+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.664224+0000","last_change":"2024-04-25T00:01:19.520648+0000","last_active":"2024-04-25T00:01:46.664224+0000","last_peered":"2024-04-25T00:01:46.664224+0000","last_clean":"2024-04-25T00:01:46.664224+0000","last_became_active":"2024-04-25T00:01:19.519303+0000","last_became_peered":"2024-04-25T00:01:19.519303+0000","last_unstale":"2024-04-25T00:01:46.664224+0000","last_undegraded":"2024-04-25T00:01:46.664224+0000","last_fullsized":"2024-04-25T00:01:46.664224+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:23.993729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.513073+0000","last_change":"2024-04-25T00:01:05.366249+0000","last_active":"2024-04-25T00:01:20.513073+0000","last_peered":"2024-04-25T00:01:20.513073+0000","last_clean":"2024-04-25T00:01:20.513073+0000","last_became_active":"2024-04-25T00:01:05.365548+0000","last_became_peered":"2024-04-25T00:01:05.365548+0000","last_unstale":"2024-04-25T00:01:20.513073+0000","last_undegraded":"2024-04-25T00:01:20.513073+0000","last_fullsized":"2024-04-25T00:01:20.513073+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:22:21.143639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.647068+0000","last_change":"2024-04-25T00:01:19.514659+0000","last_active":"2024-04-25T00:01:46.647068+0000","last_peered":"2024-04-25T00:01:46.647068+0000","last_clean":"2024-04-25T00:01:46.647068+0000","last_became_active":"2024-04-25T00:01:19.514232+0000","last_became_peered":"2024-04-25T00:01:19.514232+0000","last_unstale":"2024-04-25T00:01:46.647068+0000","last_undegraded":"2024-04-25T00:01:46.647068+0000","last_fullsized":"2024-04-25T00:01:46.647068+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:45.121513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":10,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.668794+0000","last_change":"2024-04-25T00:01:46.665943+0000","last_active":"2024-04-25T00:01:46.668794+0000","last_peered":"2024-04-25T00:01:46.668794+0000","last_clean":"2024-04-25T00:01:46.668794+0000","last_became_active":"2024-04-25T00:01:46.663874+0000","last_became_peered":"2024-04-25T00:01:46.663874+0000","last_unstale":"2024-04-25T00:01:46.668794+0000","last_undegraded":"2024-04-25T00:01:46.668794+0000","last_fullsized":"2024-04-25T00:01:46.668794+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:53:00.633407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659279+0000","last_change":"2024-04-25T00:01:19.503419+0000","last_active":"2024-04-25T00:01:46.659279+0000","last_peered":"2024-04-25T00:01:46.659279+0000","last_clean":"2024-04-25T00:01:46.659279+0000","last_became_active":"2024-04-25T00:01:19.500568+0000","last_became_peered":"2024-04-25T00:01:19.500568+0000","last_unstale":"2024-04-25T00:01:46.659279+0000","last_undegraded":"2024-04-25T00:01:46.659279+0000","last_fullsized":"2024-04-25T00:01:46.659279+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:08.508741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659631+0000","last_change":"2024-04-25T00:01:05.370653+0000","last_active":"2024-04-25T00:01:46.659631+0000","last_peered":"2024-04-25T00:01:46.659631+0000","last_clean":"2024-04-25T00:01:46.659631+0000","last_became_active":"2024-04-25T00:01:05.369919+0000","last_became_peered":"2024-04-25T00:01:05.369919+0000","last_unstale":"2024-04-25T00:01:46.659631+0000","last_undegraded":"2024-04-25T00:01:46.659631+0000","last_fullsized":"2024-04-25T00:01:46.659631+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:30:08.311326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.646840+0000","last_change":"2024-04-25T00:01:19.540104+0000","last_active":"2024-04-25T00:01:46.646840+0000","last_peered":"2024-04-25T00:01:46.646840+0000","last_clean":"2024-04-25T00:01:46.646840+0000","last_became_active":"2024-04-25T00:01:19.539775+0000","last_became_peered":"2024-04-25T00:01:19.539775+0000","last_unstale":"2024-04-25T00:01:46.646840+0000","last_undegraded":"2024-04-25T00:01:46.646840+0000","last_fullsized":"2024-04-25T00:01:46.646840+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:53:14.768520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.664786+0000","last_change":"2024-04-25T00:01:05.367001+0000","last_active":"2024-04-25T00:01:46.664786+0000","last_peered":"2024-04-25T00:01:46.664786+0000","last_clean":"2024-04-25T00:01:46.664786+0000","last_became_active":"2024-04-25T00:01:05.366264+0000","last_became_peered":"2024-04-25T00:01:05.366264+0000","last_unstale":"2024-04-25T00:01:46.664786+0000","last_undegraded":"2024-04-25T00:01:46.664786+0000","last_fullsized":"2024-04-25T00:01:46.664786+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.661915+0000","last_change":"2024-04-25T00:01:19.497247+0000","last_active":"2024-04-25T00:01:46.661915+0000","last_peered":"2024-04-25T00:01:46.661915+0000","last_clean":"2024-04-25T00:01:46.661915+0000","last_became_active":"2024-04-25T00:01:19.496517+0000","last_became_peered":"2024-04-25T00:01:19.496517+0000","last_unstale":"2024-04-25T00:01:46.661915+0000","last_undegraded":"2024-04-25T00:01:46.661915+0000","last_fullsized":"2024-04-25T00:01:46.661915+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:30.129056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659982+0000","last_change":"2024-04-25T00:01:05.379958+0000","last_active":"2024-04-25T00:01:46.659982+0000","last_peered":"2024-04-25T00:01:46.659982+0000","last_clean":"2024-04-25T00:01:46.659982+0000","last_became_active":"2024-04-25T00:01:05.379090+0000","last_became_peered":"2024-04-25T00:01:05.379090+0000","last_unstale":"2024-04-25T00:01:46.659982+0000","last_undegraded":"2024-04-25T00:01:46.659982+0000","last_fullsized":"2024-04-25T00:01:46.659982+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:59:50.832958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.515390+0000","last_change":"2024-04-25T00:01:19.530176+0000","last_active":"2024-04-25T00:01:20.515390+0000","last_peered":"2024-04-25T00:01:20.515390+0000","last_clean":"2024-04-25T00:01:20.515390+0000","last_became_active":"2024-04-25T00:01:19.529410+0000","last_became_peered":"2024-04-25T00:01:19.529410+0000","last_unstale":"2024-04-25T00:01:20.515390+0000","last_undegraded":"2024-04-25T00:01:20.515390+0000","last_fullsized":"2024-04-25T00:01:20.515390+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:40:13.204880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657503+0000","last_change":"2024-04-25T00:01:05.370826+0000","last_active":"2024-04-25T00:01:46.657503+0000","last_peered":"2024-04-25T00:01:46.657503+0000","last_clean":"2024-04-25T00:01:46.657503+0000","last_became_active":"2024-04-25T00:01:05.369848+0000","last_became_peered":"2024-04-25T00:01:05.369848+0000","last_unstale":"2024-04-25T00:01:46.657503+0000","last_undegraded":"2024-04-25T00:01:46.657503+0000","last_fullsized":"2024-04-25T00:01:46.657503+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:23.870121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657618+0000","last_change":"2024-04-25T00:01:19.511625+0000","last_active":"2024-04-25T00:01:46.657618+0000","last_peered":"2024-04-25T00:01:46.657618+0000","last_clean":"2024-04-25T00:01:46.657618+0000","last_became_active":"2024-04-25T00:01:19.511349+0000","last_became_peered":"2024-04-25T00:01:19.511349+0000","last_unstale":"2024-04-25T00:01:46.657618+0000","last_undegraded":"2024-04-25T00:01:46.657618+0000","last_fullsized":"2024-04-25T00:01:46.657618+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:02:57.373014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.650675+0000","last_change":"2024-04-25T00:01:05.390341+0000","last_active":"2024-04-25T00:01:46.650675+0000","last_peered":"2024-04-25T00:01:46.650675+0000","last_clean":"2024-04-25T00:01:46.650675+0000","last_became_active":"2024-04-25T00:01:05.389707+0000","last_became_peered":"2024-04-25T00:01:05.389707+0000","last_unstale":"2024-04-25T00:01:46.650675+0000","last_undegraded":"2024-04-25T00:01:46.650675+0000","last_fullsized":"2024-04-25T00:01:46.650675+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:54:21.138947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657624+0000","last_change":"2024-04-25T00:01:19.513630+0000","last_active":"2024-04-25T00:01:46.657624+0000","last_peered":"2024-04-25T00:01:46.657624+0000","last_clean":"2024-04-25T00:01:46.657624+0000","last_became_active":"2024-04-25T00:01:19.512978+0000","last_became_peered":"2024-04-25T00:01:19.512978+0000","last_unstale":"2024-04-25T00:01:46.657624+0000","last_undegraded":"2024-04-25T00:01:46.657624+0000","last_fullsized":"2024-04-25T00:01:46.657624+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:57.751007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.662982+0000","last_change":"2024-04-25T00:01:05.378373+0000","last_active":"2024-04-25T00:01:46.662982+0000","last_peered":"2024-04-25T00:01:46.662982+0000","last_clean":"2024-04-25T00:01:46.662982+0000","last_became_active":"2024-04-25T00:01:05.376570+0000","last_became_peered":"2024-04-25T00:01:05.376570+0000","last_unstale":"2024-04-25T00:01:46.662982+0000","last_undegraded":"2024-04-25T00:01:46.662982+0000","last_fullsized":"2024-04-25T00:01:46.662982+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:30.228060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659177+0000","last_change":"2024-04-25T00:01:19.516716+0000","last_active":"2024-04-25T00:01:46.659177+0000","last_peered":"2024-04-25T00:01:46.659177+0000","last_clean":"2024-04-25T00:01:46.659177+0000","last_became_active":"2024-04-25T00:01:19.516142+0000","last_became_peered":"2024-04-25T00:01:19.516142+0000","last_unstale":"2024-04-25T00:01:46.659177+0000","last_undegraded":"2024-04-25T00:01:46.659177+0000","last_fullsized":"2024-04-25T00:01:46.659177+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:28:18.234546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.663972+0000","last_change":"2024-04-25T00:01:05.381011+0000","last_active":"2024-04-25T00:01:46.663972+0000","last_peered":"2024-04-25T00:01:46.663972+0000","last_clean":"2024-04-25T00:01:46.663972+0000","last_became_active":"2024-04-25T00:01:05.380236+0000","last_became_peered":"2024-04-25T00:01:05.380236+0000","last_unstale":"2024-04-25T00:01:46.663972+0000","last_undegraded":"2024-04-25T00:01:46.663972+0000","last_fullsized":"2024-04-25T00:01:46.663972+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:06:19.307378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.513107+0000","last_change":"2024-04-25T00:01:19.529023+0000","last_active":"2024-04-25T00:01:20.513107+0000","last_peered":"2024-04-25T00:01:20.513107+0000","last_clean":"2024-04-25T00:01:20.513107+0000","last_became_active":"2024-04-25T00:01:19.524609+0000","last_became_peered":"2024-04-25T00:01:19.524609+0000","last_unstale":"2024-04-25T00:01:20.513107+0000","last_undegraded":"2024-04-25T00:01:20.513107+0000","last_fullsized":"2024-04-25T00:01:20.513107+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:41:19.150476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660549+0000","last_change":"2024-04-25T00:01:05.385853+0000","last_active":"2024-04-25T00:01:46.660549+0000","last_peered":"2024-04-25T00:01:46.660549+0000","last_clean":"2024-04-25T00:01:46.660549+0000","last_became_active":"2024-04-25T00:01:05.384997+0000","last_became_peered":"2024-04-25T00:01:05.384997+0000","last_unstale":"2024-04-25T00:01:46.660549+0000","last_undegraded":"2024-04-25T00:01:46.660549+0000","last_fullsized":"2024-04-25T00:01:46.660549+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:36:55.436661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.662855+0000","last_change":"2024-04-25T00:01:19.516867+0000","last_active":"2024-04-25T00:01:46.662855+0000","last_peered":"2024-04-25T00:01:46.662855+0000","last_clean":"2024-04-25T00:01:46.662855+0000","last_became_active":"2024-04-25T00:01:19.515764+0000","last_became_peered":"2024-04-25T00:01:19.515764+0000","last_unstale":"2024-04-25T00:01:46.662855+0000","last_undegraded":"2024-04-25T00:01:46.662855+0000","last_fullsized":"2024-04-25T00:01:46.662855+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:53:14.768520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660272+0000","last_change":"2024-04-25T00:01:05.389424+0000","last_active":"2024-04-25T00:01:46.660272+0000","last_peered":"2024-04-25T00:01:46.660272+0000","last_clean":"2024-04-25T00:01:46.660272+0000","last_became_active":"2024-04-25T00:01:05.388999+0000","last_became_peered":"2024-04-25T00:01:05.388999+0000","last_unstale":"2024-04-25T00:01:46.660272+0000","last_undegraded":"2024-04-25T00:01:46.660272+0000","last_fullsized":"2024-04-25T00:01:46.660272+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:31.456792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657679+0000","last_change":"2024-04-25T00:01:19.519365+0000","last_active":"2024-04-25T00:01:46.657679+0000","last_peered":"2024-04-25T00:01:46.657679+0000","last_clean":"2024-04-25T00:01:46.657679+0000","last_became_active":"2024-04-25T00:01:19.519085+0000","last_became_peered":"2024-04-25T00:01:19.519085+0000","last_unstale":"2024-04-25T00:01:46.657679+0000","last_undegraded":"2024-04-25T00:01:46.657679+0000","last_fullsized":"2024-04-25T00:01:46.657679+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:05:09.219823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660140+0000","last_change":"2024-04-25T00:01:05.368974+0000","last_active":"2024-04-25T00:01:46.660140+0000","last_peered":"2024-04-25T00:01:46.660140+0000","last_clean":"2024-04-25T00:01:46.660140+0000","last_became_active":"2024-04-25T00:01:05.367893+0000","last_became_peered":"2024-04-25T00:01:05.367893+0000","last_unstale":"2024-04-25T00:01:46.660140+0000","last_undegraded":"2024-04-25T00:01:46.660140+0000","last_fullsized":"2024-04-25T00:01:46.660140+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:30:23.557381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.658127+0000","last_change":"2024-04-25T00:01:19.513514+0000","last_active":"2024-04-25T00:01:46.658127+0000","last_peered":"2024-04-25T00:01:46.658127+0000","last_clean":"2024-04-25T00:01:46.658127+0000","last_became_active":"2024-04-25T00:01:19.513184+0000","last_became_peered":"2024-04-25T00:01:19.513184+0000","last_unstale":"2024-04-25T00:01:46.658127+0000","last_undegraded":"2024-04-25T00:01:46.658127+0000","last_fullsized":"2024-04-25T00:01:46.658127+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:47.859626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657909+0000","last_change":"2024-04-25T00:01:05.362303+0000","last_active":"2024-04-25T00:01:46.657909+0000","last_peered":"2024-04-25T00:01:46.657909+0000","last_clean":"2024-04-25T00:01:46.657909+0000","last_became_active":"2024-04-25T00:01:05.361586+0000","last_became_peered":"2024-04-25T00:01:05.361586+0000","last_unstale":"2024-04-25T00:01:46.657909+0000","last_undegraded":"2024-04-25T00:01:46.657909+0000","last_fullsized":"2024-04-25T00:01:46.657909+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:30.228060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.650059+0000","last_change":"2024-04-25T00:01:19.515101+0000","last_active":"2024-04-25T00:01:46.650059+0000","last_peered":"2024-04-25T00:01:46.650059+0000","last_clean":"2024-04-25T00:01:46.650059+0000","last_became_active":"2024-04-25T00:01:19.513314+0000","last_became_peered":"2024-04-25T00:01:19.513314+0000","last_unstale":"2024-04-25T00:01:46.650059+0000","last_undegraded":"2024-04-25T00:01:46.650059+0000","last_fullsized":"2024-04-25T00:01:46.650059+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:50.351599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.649234+0000","last_change":"2024-04-25T00:01:05.390256+0000","last_active":"2024-04-25T00:01:46.649234+0000","last_peered":"2024-04-25T00:01:46.649234+0000","last_clean":"2024-04-25T00:01:46.649234+0000","last_became_active":"2024-04-25T00:01:05.389724+0000","last_became_peered":"2024-04-25T00:01:05.389724+0000","last_unstale":"2024-04-25T00:01:46.649234+0000","last_undegraded":"2024-04-25T00:01:46.649234+0000","last_fullsized":"2024-04-25T00:01:46.649234+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:09:18.315688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3f","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.648222+0000","last_change":"2024-04-25T00:01:19.511493+0000","last_active":"2024-04-25T00:01:46.648222+0000","last_peered":"2024-04-25T00:01:46.648222+0000","last_clean":"2024-04-25T00:01:46.648222+0000","last_became_active":"2024-04-25T00:01:19.508049+0000","last_became_peered":"2024-04-25T00:01:19.508049+0000","last_unstale":"2024-04-25T00:01:46.648222+0000","last_undegraded":"2024-04-25T00:01:46.648222+0000","last_fullsized":"2024-04-25T00:01:46.648222+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:55:46.853565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.658614+0000","last_change":"2024-04-25T00:01:05.374766+0000","last_active":"2024-04-25T00:01:46.658614+0000","last_peered":"2024-04-25T00:01:46.658614+0000","last_clean":"2024-04-25T00:01:46.658614+0000","last_became_active":"2024-04-25T00:01:05.369739+0000","last_became_peered":"2024-04-25T00:01:05.369739+0000","last_unstale":"2024-04-25T00:01:46.658614+0000","last_undegraded":"2024-04-25T00:01:46.658614+0000","last_fullsized":"2024-04-25T00:01:46.658614+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:09:18.315688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.513219+0000","last_change":"2024-04-25T00:01:19.521456+0000","last_active":"2024-04-25T00:01:20.513219+0000","last_peered":"2024-04-25T00:01:20.513219+0000","last_clean":"2024-04-25T00:01:20.513219+0000","last_became_active":"2024-04-25T00:01:19.520787+0000","last_became_peered":"2024-04-25T00:01:19.520787+0000","last_unstale":"2024-04-25T00:01:20.513219+0000","last_undegraded":"2024-04-25T00:01:20.513219+0000","last_fullsized":"2024-04-25T00:01:20.513219+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:25:38.981364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.647277+0000","last_change":"2024-04-25T00:01:05.359253+0000","last_active":"2024-04-25T00:01:46.647277+0000","last_peered":"2024-04-25T00:01:46.647277+0000","last_clean":"2024-04-25T00:01:46.647277+0000","last_became_active":"2024-04-25T00:01:05.358477+0000","last_became_peered":"2024-04-25T00:01:05.358477+0000","last_unstale":"2024-04-25T00:01:46.647277+0000","last_undegraded":"2024-04-25T00:01:46.647277+0000","last_fullsized":"2024-04-25T00:01:46.647277+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:22.739214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.656961+0000","last_change":"2024-04-25T00:01:19.509505+0000","last_active":"2024-04-25T00:01:46.656961+0000","last_peered":"2024-04-25T00:01:46.656961+0000","last_clean":"2024-04-25T00:01:46.656961+0000","last_became_active":"2024-04-25T00:01:19.507812+0000","last_became_peered":"2024-04-25T00:01:19.507812+0000","last_unstale":"2024-04-25T00:01:46.656961+0000","last_undegraded":"2024-04-25T00:01:46.656961+0000","last_fullsized":"2024-04-25T00:01:46.656961+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:23.993729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659707+0000","last_change":"2024-04-25T00:01:05.386068+0000","last_active":"2024-04-25T00:01:46.659707+0000","last_peered":"2024-04-25T00:01:46.659707+0000","last_clean":"2024-04-25T00:01:46.659707+0000","last_became_active":"2024-04-25T00:01:05.378650+0000","last_became_peered":"2024-04-25T00:01:05.378650+0000","last_unstale":"2024-04-25T00:01:46.659707+0000","last_undegraded":"2024-04-25T00:01:46.659707+0000","last_fullsized":"2024-04-25T00:01:46.659707+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:12:52.846250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660257+0000","last_change":"2024-04-25T00:01:19.514843+0000","last_active":"2024-04-25T00:01:46.660257+0000","last_peered":"2024-04-25T00:01:46.660257+0000","last_clean":"2024-04-25T00:01:46.660257+0000","last_became_active":"2024-04-25T00:01:19.513956+0000","last_became_peered":"2024-04-25T00:01:19.513956+0000","last_unstale":"2024-04-25T00:01:46.660257+0000","last_undegraded":"2024-04-25T00:01:46.660257+0000","last_fullsized":"2024-04-25T00:01:46.660257+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:38.005233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.648398+0000","last_change":"2024-04-25T00:01:05.359716+0000","last_active":"2024-04-25T00:01:46.648398+0000","last_peered":"2024-04-25T00:01:46.648398+0000","last_clean":"2024-04-25T00:01:46.648398+0000","last_became_active":"2024-04-25T00:01:05.358866+0000","last_became_peered":"2024-04-25T00:01:05.358866+0000","last_unstale":"2024-04-25T00:01:46.648398+0000","last_undegraded":"2024-04-25T00:01:46.648398+0000","last_fullsized":"2024-04-25T00:01:46.648398+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:50.489125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.649702+0000","last_change":"2024-04-25T00:01:19.499574+0000","last_active":"2024-04-25T00:01:46.649702+0000","last_peered":"2024-04-25T00:01:46.649702+0000","last_clean":"2024-04-25T00:01:46.649702+0000","last_became_active":"2024-04-25T00:01:19.498699+0000","last_became_peered":"2024-04-25T00:01:19.498699+0000","last_unstale":"2024-04-25T00:01:46.649702+0000","last_undegraded":"2024-04-25T00:01:46.649702+0000","last_fullsized":"2024-04-25T00:01:46.649702+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:47.859626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.650324+0000","last_change":"2024-04-25T00:01:05.371291+0000","last_active":"2024-04-25T00:01:46.650324+0000","last_peered":"2024-04-25T00:01:46.650324+0000","last_clean":"2024-04-25T00:01:46.650324+0000","last_became_active":"2024-04-25T00:01:05.370522+0000","last_became_peered":"2024-04-25T00:01:05.370522+0000","last_unstale":"2024-04-25T00:01:46.650324+0000","last_undegraded":"2024-04-25T00:01:46.650324+0000","last_fullsized":"2024-04-25T00:01:46.650324+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:11:27.044893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.647355+0000","last_change":"2024-04-25T00:01:19.523798+0000","last_active":"2024-04-25T00:01:46.647355+0000","last_peered":"2024-04-25T00:01:46.647355+0000","last_clean":"2024-04-25T00:01:46.647355+0000","last_became_active":"2024-04-25T00:01:19.523450+0000","last_became_peered":"2024-04-25T00:01:19.523450+0000","last_unstale":"2024-04-25T00:01:46.647355+0000","last_undegraded":"2024-04-25T00:01:46.647355+0000","last_fullsized":"2024-04-25T00:01:46.647355+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:40:36.350832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.651674+0000","last_change":"2024-04-25T00:01:05.368190+0000","last_active":"2024-04-25T00:01:46.651674+0000","last_peered":"2024-04-25T00:01:46.651674+0000","last_clean":"2024-04-25T00:01:46.651674+0000","last_became_active":"2024-04-25T00:01:05.362606+0000","last_became_peered":"2024-04-25T00:01:05.362606+0000","last_unstale":"2024-04-25T00:01:46.651674+0000","last_undegraded":"2024-04-25T00:01:46.651674+0000","last_fullsized":"2024-04-25T00:01:46.651674+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:12:52.846250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.646901+0000","last_change":"2024-04-25T00:01:19.505737+0000","last_active":"2024-04-25T00:01:46.646901+0000","last_peered":"2024-04-25T00:01:46.646901+0000","last_clean":"2024-04-25T00:01:46.646901+0000","last_became_active":"2024-04-25T00:01:19.504991+0000","last_became_peered":"2024-04-25T00:01:19.504991+0000","last_unstale":"2024-04-25T00:01:46.646901+0000","last_undegraded":"2024-04-25T00:01:46.646901+0000","last_fullsized":"2024-04-25T00:01:46.646901+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:47:19.526845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660392+0000","last_change":"2024-04-25T00:01:05.389596+0000","last_active":"2024-04-25T00:01:46.660392+0000","last_peered":"2024-04-25T00:01:46.660392+0000","last_clean":"2024-04-25T00:01:46.660392+0000","last_became_active":"2024-04-25T00:01:05.389127+0000","last_became_peered":"2024-04-25T00:01:05.389127+0000","last_unstale":"2024-04-25T00:01:46.660392+0000","last_undegraded":"2024-04-25T00:01:46.660392+0000","last_fullsized":"2024-04-25T00:01:46.660392+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:09.858617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2851,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":5702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2851,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":5702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509455,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27336,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344504,"statfs":{"total":96636764160,"available":96608772096,"internally_reserved":0,"allocated":319488,"data_stored":123609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509455,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27320,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344520,"statfs":{"total":96636764160,"available":96608788480,"internally_reserved":0,"allocated":307200,"data_stored":118634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509454,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27900,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343940,"statfs":{"total":96636764160,"available":96608194560,"internally_reserved":0,"allocated":901120,"data_stored":707623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509455,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":327680,"data_stored":124821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509456,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":323584,"data_stored":125550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509455,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27912,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343928,"statfs":{"total":96636764160,"available":96608182272,"internally_reserved":0,"allocated":909312,"data_stored":713971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509455,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27320,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344520,"statfs":{"total":96636764160,"available":96608788480,"internally_reserved":0,"allocated":307200,"data_stored":116727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574863,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27328,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344512,"statfs":{"total":96636764160,"available":96608780288,"internally_reserved":0,"allocated":319488,"data_stored":114370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T00:01:49.484 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T00:01:49.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.629+0000 7fa33614f640 1 Processor -- start 2024-04-25T00:01:49.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.629+0000 7fa33614f640 1 -- start start 2024-04-25T00:01:49.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.633+0000 7fa33614f640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fa330150d20 0x7fa3300c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:49.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.633+0000 7fa33614f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3300a94a0 0x7fa330153ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:49.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.633+0000 7fa33614f640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3300c3910 con 0x7fa3300c2f50 2024-04-25T00:01:49.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.633+0000 7fa33614f640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fa330154530 con 0x7fa330150d20 2024-04-25T00:01:49.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.633+0000 7fa33614f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa3300c16e0 con 0x7fa3300a94a0 2024-04-25T00:01:49.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.633+0000 7fa32f7fe640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fa330150d20 0x7fa3300c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:49.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.633+0000 7fa32ffff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3300a94a0 0x7fa330153ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:49.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.633+0000 7fa32ffff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3300a94a0 0x7fa330153ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:38338/0 (socket says 172.21.15.116:38338) 2024-04-25T00:01:49.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.633+0000 7fa32ffff640 1 -- 172.21.15.116:0/1982150834 learned_addr learned my addr 172.21.15.116:0/1982150834 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:49.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.633+0000 7fa32f7fe640 1 -- 172.21.15.116:0/1982150834 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3300a94a0 msgr2=0x7fa330153ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:49.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.633+0000 7fa32f7fe640 1 --2- 172.21.15.116:0/1982150834 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3300a94a0 0x7fa330153ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:49.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.633+0000 7fa32f7fe640 1 -- 172.21.15.116:0/1982150834 >> v1:172.21.15.174:6789/0 conn(0x7fa3300c2f50 legacy=0x7fa3301538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:49.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.633+0000 7fa32f7fe640 1 -- 172.21.15.116:0/1982150834 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3300c1400 con 0x7fa330150d20 2024-04-25T00:01:49.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.633+0000 7fa32f7fe640 1 --2- 172.21.15.116:0/1982150834 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa330150d20 0x7fa3300c2a10 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa3280097c0 tx=0x7fa32801c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9dae96906b85e48e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:49.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.637+0000 7fa32e7fc640 1 -- 172.21.15.116:0/1982150834 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3280b1070 con 0x7fa330150d20 2024-04-25T00:01:49.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.637+0000 7fa32e7fc640 1 -- 172.21.15.116:0/1982150834 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3280ac410 con 0x7fa330150d20 2024-04-25T00:01:49.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.637+0000 7fa32e7fc640 1 -- 172.21.15.116:0/1982150834 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3280ac7e0 con 0x7fa330150d20 2024-04-25T00:01:49.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.637+0000 7fa33614f640 1 -- 172.21.15.116:0/1982150834 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa330150d20 msgr2=0x7fa3300c2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:49.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.637+0000 7fa33614f640 1 --2- 172.21.15.116:0/1982150834 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa330150d20 0x7fa3300c2a10 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa3280097c0 tx=0x7fa32801c4a0 comp rx=0 tx=0).stop 2024-04-25T00:01:49.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.637+0000 7fa33614f640 1 -- 172.21.15.116:0/1982150834 shutdown_connections 2024-04-25T00:01:49.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.637+0000 7fa33614f640 1 --2- 172.21.15.116:0/1982150834 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3300a94a0 0x7fa330153ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:49.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.637+0000 7fa33614f640 1 --2- 172.21.15.116:0/1982150834 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa330150d20 0x7fa3300c2a10 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:49.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.637+0000 7fa33614f640 1 -- 172.21.15.116:0/1982150834 >> 172.21.15.116:0/1982150834 conn(0x7fa3300c3b00 msgr2=0x7fa3300c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:49.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.637+0000 7fa33614f640 1 -- 172.21.15.116:0/1982150834 shutdown_connections 2024-04-25T00:01:49.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.637+0000 7fa33614f640 1 -- 172.21.15.116:0/1982150834 wait complete. 2024-04-25T00:01:49.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.637+0000 7fa33614f640 1 Processor -- start 2024-04-25T00:01:49.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.637+0000 7fa33614f640 1 -- start start 2024-04-25T00:01:49.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.637+0000 7fa33614f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3300a94a0 0x7fa3300c6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:49.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.637+0000 7fa33614f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3300c2f50 0x7fa3301efec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:49.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.637+0000 7fa33614f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa330150d20 0x7fa330138300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:49.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.637+0000 7fa33614f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa3301608b0 con 0x7fa330150d20 2024-04-25T00:01:49.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.637+0000 7fa33614f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa330160730 con 0x7fa3300a94a0 2024-04-25T00:01:49.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.637+0000 7fa33614f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa330160a30 con 0x7fa3300c2f50 2024-04-25T00:01:49.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.637+0000 7fa32ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa330150d20 0x7fa330138300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:49.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.637+0000 7fa32ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa330150d20 0x7fa330138300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43352/0 (socket says 172.21.15.116:43352) 2024-04-25T00:01:49.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.637+0000 7fa32ffff640 1 -- 172.21.15.116:0/541306071 learned_addr learned my addr 172.21.15.116:0/541306071 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:49.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.637+0000 7fa32effd640 1 --2- 172.21.15.116:0/541306071 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3300c2f50 0x7fa3301efec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:49.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.637+0000 7fa32f7fe640 1 --2- 172.21.15.116:0/541306071 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3300a94a0 0x7fa3300c6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:49.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.637+0000 7fa32ffff640 1 -- 172.21.15.116:0/541306071 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3300c2f50 msgr2=0x7fa3301efec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:49.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.637+0000 7fa32ffff640 1 --2- 172.21.15.116:0/541306071 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3300c2f50 0x7fa3301efec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:49.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.637+0000 7fa32ffff640 1 -- 172.21.15.116:0/541306071 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3300a94a0 msgr2=0x7fa3300c6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:49.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.637+0000 7fa32ffff640 1 --2- 172.21.15.116:0/541306071 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3300a94a0 0x7fa3300c6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:49.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.637+0000 7fa32ffff640 1 -- 172.21.15.116:0/541306071 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3300c69c0 con 0x7fa330150d20 2024-04-25T00:01:49.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.637+0000 7fa32ffff640 1 --2- 172.21.15.116:0/541306071 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa330150d20 0x7fa330138300 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fa32000efc0 tx=0x7fa32000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:49.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.637+0000 7fa327fff640 1 -- 172.21.15.116:0/541306071 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa32000cd20 con 0x7fa330150d20 2024-04-25T00:01:49.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.637+0000 7fa327fff640 1 -- 172.21.15.116:0/541306071 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3200039a0 con 0x7fa330150d20 2024-04-25T00:01:49.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.637+0000 7fa33614f640 1 -- 172.21.15.116:0/541306071 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3301ec110 con 0x7fa330150d20 2024-04-25T00:01:49.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.637+0000 7fa327fff640 1 -- 172.21.15.116:0/541306071 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa320003d90 con 0x7fa330150d20 2024-04-25T00:01:49.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.637+0000 7fa33614f640 1 -- 172.21.15.116:0/541306071 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3301ec6d0 con 0x7fa330150d20 2024-04-25T00:01:49.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.641+0000 7fa33614f640 1 -- 172.21.15.116:0/541306071 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa330068350 con 0x7fa330150d20 2024-04-25T00:01:49.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.641+0000 7fa327fff640 1 -- 172.21.15.116:0/541306071 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa320012050 con 0x7fa330150d20 2024-04-25T00:01:49.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.641+0000 7fa327fff640 1 --2- 172.21.15.116:0/541306071 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa3100678c0 0x7fa310069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:49.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.641+0000 7fa327fff640 1 -- 172.21.15.116:0/541306071 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6666+0+0 (secure 0 0 0) 0x7fa320086520 con 0x7fa330150d20 2024-04-25T00:01:49.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.641+0000 7fa32f7fe640 1 --2- 172.21.15.116:0/541306071 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa3100678c0 0x7fa310069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:49.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.641+0000 7fa32f7fe640 1 --2- 172.21.15.116:0/541306071 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa3100678c0 0x7fa310069d80 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa3280ae000 tx=0x7fa328008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:49.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.645+0000 7fa327fff640 1 -- 172.21.15.116:0/541306071 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fa320010040 con 0x7fa330150d20 2024-04-25T00:01:49.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.789+0000 7fa33614f640 1 -- 172.21.15.116:0/541306071 --> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa3300c34f0 con 0x7fa3100678c0 2024-04-25T00:01:49.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.821+0000 7fa327fff640 1 -- 172.21.15.116:0/541306071 <== mgr.4105 v2:172.21.15.174:6832/2004448211 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+490400 (secure 0 0 0) 0x7fa3300c34f0 con 0x7fa3100678c0 2024-04-25T00:01:49.833 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:01:49.838 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-04-25T00:01:49.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.829+0000 7fa33614f640 1 -- 172.21.15.116:0/541306071 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa3100678c0 msgr2=0x7fa310069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:49.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.829+0000 7fa33614f640 1 --2- 172.21.15.116:0/541306071 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa3100678c0 0x7fa310069d80 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa3280ae000 tx=0x7fa328008040 comp rx=0 tx=0).stop 2024-04-25T00:01:49.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.829+0000 7fa33614f640 1 -- 172.21.15.116:0/541306071 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa330150d20 msgr2=0x7fa330138300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:49.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.829+0000 7fa33614f640 1 --2- 172.21.15.116:0/541306071 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa330150d20 0x7fa330138300 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fa32000efc0 tx=0x7fa32000c490 comp rx=0 tx=0).stop 2024-04-25T00:01:49.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.829+0000 7fa33614f640 1 -- 172.21.15.116:0/541306071 shutdown_connections 2024-04-25T00:01:49.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.829+0000 7fa33614f640 1 --2- 172.21.15.116:0/541306071 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa3100678c0 0x7fa310069d80 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:49.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.829+0000 7fa33614f640 1 --2- 172.21.15.116:0/541306071 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa330150d20 0x7fa330138300 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:49.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.829+0000 7fa33614f640 1 --2- 172.21.15.116:0/541306071 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3300c2f50 0x7fa3301efec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:49.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.829+0000 7fa33614f640 1 --2- 172.21.15.116:0/541306071 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3300a94a0 0x7fa3300c6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:49.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.829+0000 7fa33614f640 1 -- 172.21.15.116:0/541306071 >> 172.21.15.116:0/541306071 conn(0x7fa3300c3b00 msgr2=0x7fa33006d840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:49.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.829+0000 7fa33614f640 1 -- 172.21.15.116:0/541306071 shutdown_connections 2024-04-25T00:01:49.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:49.829+0000 7fa33614f640 1 -- 172.21.15.116:0/541306071 wait complete. 2024-04-25T00:01:49.847 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":52,"stamp":"2024-04-25T00:01:49.268391+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596070,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":253,"num_write_kb":2448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":199,"ondisk_log_size":199,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":387,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219800,"kb_used_data":3628,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754920,"statfs":{"total":773094113280,"available":772869038080,"internally_reserved":0,"allocated":3715072,"data_stored":2145305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":882,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003190"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.656917+0000","last_change":"2024-04-25T00:01:19.508728+0000","last_active":"2024-04-25T00:01:46.656917+0000","last_peered":"2024-04-25T00:01:46.656917+0000","last_clean":"2024-04-25T00:01:46.656917+0000","last_became_active":"2024-04-25T00:01:19.508390+0000","last_became_peered":"2024-04-25T00:01:19.508390+0000","last_unstale":"2024-04-25T00:01:46.656917+0000","last_undegraded":"2024-04-25T00:01:46.656917+0000","last_fullsized":"2024-04-25T00:01:46.656917+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:45.591905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1f","version":"22'4","reported_seq":26,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.515265+0000","last_change":"2024-04-25T00:01:04.356032+0000","last_active":"2024-04-25T00:01:20.515265+0000","last_peered":"2024-04-25T00:01:20.515265+0000","last_clean":"2024-04-25T00:01:20.515265+0000","last_became_active":"2024-04-25T00:01:04.355293+0000","last_became_peered":"2024-04-25T00:01:04.355293+0000","last_unstale":"2024-04-25T00:01:20.515265+0000","last_undegraded":"2024-04-25T00:01:20.515265+0000","last_fullsized":"2024-04-25T00:01:20.515265+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:56:16.678132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.651891+0000","last_change":"2024-04-25T00:01:05.367207+0000","last_active":"2024-04-25T00:01:46.651891+0000","last_peered":"2024-04-25T00:01:46.651891+0000","last_clean":"2024-04-25T00:01:46.651891+0000","last_became_active":"2024-04-25T00:01:05.366471+0000","last_became_peered":"2024-04-25T00:01:05.366471+0000","last_unstale":"2024-04-25T00:01:46.651891+0000","last_undegraded":"2024-04-25T00:01:46.651891+0000","last_fullsized":"2024-04-25T00:01:46.651891+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:30:23.557381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659979+0000","last_change":"2024-04-25T00:01:18.502364+0000","last_active":"2024-04-25T00:01:46.659979+0000","last_peered":"2024-04-25T00:01:46.659979+0000","last_clean":"2024-04-25T00:01:46.659979+0000","last_became_active":"2024-04-25T00:01:18.501633+0000","last_became_peered":"2024-04-25T00:01:18.501633+0000","last_unstale":"2024-04-25T00:01:46.659979+0000","last_undegraded":"2024-04-25T00:01:46.659979+0000","last_fullsized":"2024-04-25T00:01:46.659979+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:55.074018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":10,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.665473+0000","last_change":"2024-04-25T00:01:46.662019+0000","last_active":"2024-04-25T00:01:46.665473+0000","last_peered":"2024-04-25T00:01:46.665473+0000","last_clean":"2024-04-25T00:01:46.665473+0000","last_became_active":"2024-04-25T00:01:46.661167+0000","last_became_peered":"2024-04-25T00:01:46.661167+0000","last_unstale":"2024-04-25T00:01:46.665473+0000","last_undegraded":"2024-04-25T00:01:46.665473+0000","last_fullsized":"2024-04-25T00:01:46.665473+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:13:34.153117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"22'1","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.652395+0000","last_change":"2024-04-25T00:01:04.361809+0000","last_active":"2024-04-25T00:01:46.652395+0000","last_peered":"2024-04-25T00:01:46.652395+0000","last_clean":"2024-04-25T00:01:46.652395+0000","last_became_active":"2024-04-25T00:01:04.361059+0000","last_became_peered":"2024-04-25T00:01:04.361059+0000","last_unstale":"2024-04-25T00:01:46.652395+0000","last_undegraded":"2024-04-25T00:01:46.652395+0000","last_fullsized":"2024-04-25T00:01:46.652395+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.653796+0000","last_change":"2024-04-25T00:01:05.370818+0000","last_active":"2024-04-25T00:01:46.653796+0000","last_peered":"2024-04-25T00:01:46.653796+0000","last_clean":"2024-04-25T00:01:46.653796+0000","last_became_active":"2024-04-25T00:01:05.369991+0000","last_became_peered":"2024-04-25T00:01:05.369991+0000","last_unstale":"2024-04-25T00:01:46.653796+0000","last_undegraded":"2024-04-25T00:01:46.653796+0000","last_fullsized":"2024-04-25T00:01:46.653796+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:43.615895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.647007+0000","last_change":"2024-04-25T00:01:18.497682+0000","last_active":"2024-04-25T00:01:46.647007+0000","last_peered":"2024-04-25T00:01:46.647007+0000","last_clean":"2024-04-25T00:01:46.647007+0000","last_became_active":"2024-04-25T00:01:18.497397+0000","last_became_peered":"2024-04-25T00:01:18.497397+0000","last_unstale":"2024-04-25T00:01:46.647007+0000","last_undegraded":"2024-04-25T00:01:46.647007+0000","last_fullsized":"2024-04-25T00:01:46.647007+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:22.983972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.658174+0000","last_change":"2024-04-25T00:01:19.510591+0000","last_active":"2024-04-25T00:01:46.658174+0000","last_peered":"2024-04-25T00:01:46.658174+0000","last_clean":"2024-04-25T00:01:46.658174+0000","last_became_active":"2024-04-25T00:01:19.510252+0000","last_became_peered":"2024-04-25T00:01:19.510252+0000","last_unstale":"2024-04-25T00:01:46.658174+0000","last_undegraded":"2024-04-25T00:01:46.658174+0000","last_fullsized":"2024-04-25T00:01:46.658174+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:38.005233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"22'2","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660361+0000","last_change":"2024-04-25T00:01:04.362030+0000","last_active":"2024-04-25T00:01:46.660361+0000","last_peered":"2024-04-25T00:01:46.660361+0000","last_clean":"2024-04-25T00:01:46.660361+0000","last_became_active":"2024-04-25T00:01:04.361708+0000","last_became_peered":"2024-04-25T00:01:04.361708+0000","last_unstale":"2024-04-25T00:01:46.660361+0000","last_undegraded":"2024-04-25T00:01:46.660361+0000","last_fullsized":"2024-04-25T00:01:46.660361+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:38:33.392719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.658174+0000","last_change":"2024-04-25T00:01:05.387195+0000","last_active":"2024-04-25T00:01:46.658174+0000","last_peered":"2024-04-25T00:01:46.658174+0000","last_clean":"2024-04-25T00:01:46.658174+0000","last_became_active":"2024-04-25T00:01:05.386853+0000","last_became_peered":"2024-04-25T00:01:05.386853+0000","last_unstale":"2024-04-25T00:01:46.658174+0000","last_undegraded":"2024-04-25T00:01:46.658174+0000","last_fullsized":"2024-04-25T00:01:46.658174+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:06:19.307378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"26'1","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.647549+0000","last_change":"2024-04-25T00:01:18.487835+0000","last_active":"2024-04-25T00:01:46.647549+0000","last_peered":"2024-04-25T00:01:46.647549+0000","last_clean":"2024-04-25T00:01:46.647549+0000","last_became_active":"2024-04-25T00:01:18.487571+0000","last_became_peered":"2024-04-25T00:01:18.487571+0000","last_unstale":"2024-04-25T00:01:46.647549+0000","last_undegraded":"2024-04-25T00:01:46.647549+0000","last_fullsized":"2024-04-25T00:01:46.647549+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:44.582148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.647421+0000","last_change":"2024-04-25T00:01:19.519834+0000","last_active":"2024-04-25T00:01:46.647421+0000","last_peered":"2024-04-25T00:01:46.647421+0000","last_clean":"2024-04-25T00:01:46.647421+0000","last_became_active":"2024-04-25T00:01:19.519470+0000","last_became_peered":"2024-04-25T00:01:19.519470+0000","last_unstale":"2024-04-25T00:01:46.647421+0000","last_undegraded":"2024-04-25T00:01:46.647421+0000","last_fullsized":"2024-04-25T00:01:46.647421+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:47.859626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"22'2","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660830+0000","last_change":"2024-04-25T00:01:04.362832+0000","last_active":"2024-04-25T00:01:46.660830+0000","last_peered":"2024-04-25T00:01:46.660830+0000","last_clean":"2024-04-25T00:01:46.660830+0000","last_became_active":"2024-04-25T00:01:04.362059+0000","last_became_peered":"2024-04-25T00:01:04.362059+0000","last_unstale":"2024-04-25T00:01:46.660830+0000","last_undegraded":"2024-04-25T00:01:46.660830+0000","last_fullsized":"2024-04-25T00:01:46.660830+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:05.441048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659460+0000","last_change":"2024-04-25T00:01:05.381120+0000","last_active":"2024-04-25T00:01:46.659460+0000","last_peered":"2024-04-25T00:01:46.659460+0000","last_clean":"2024-04-25T00:01:46.659460+0000","last_became_active":"2024-04-25T00:01:05.378320+0000","last_became_peered":"2024-04-25T00:01:05.378320+0000","last_unstale":"2024-04-25T00:01:46.659460+0000","last_undegraded":"2024-04-25T00:01:46.659460+0000","last_fullsized":"2024-04-25T00:01:46.659460+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:34:06.175918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"26'1","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.647038+0000","last_change":"2024-04-25T00:01:18.490300+0000","last_active":"2024-04-25T00:01:46.647038+0000","last_peered":"2024-04-25T00:01:46.647038+0000","last_clean":"2024-04-25T00:01:46.647038+0000","last_became_active":"2024-04-25T00:01:18.490050+0000","last_became_peered":"2024-04-25T00:01:18.490050+0000","last_unstale":"2024-04-25T00:01:46.647038+0000","last_undegraded":"2024-04-25T00:01:46.647038+0000","last_fullsized":"2024-04-25T00:01:46.647038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:13:05.971606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.649896+0000","last_change":"2024-04-25T00:01:19.500026+0000","last_active":"2024-04-25T00:01:46.649896+0000","last_peered":"2024-04-25T00:01:46.649896+0000","last_clean":"2024-04-25T00:01:46.649896+0000","last_became_active":"2024-04-25T00:01:19.499244+0000","last_became_peered":"2024-04-25T00:01:19.499244+0000","last_unstale":"2024-04-25T00:01:46.649896+0000","last_undegraded":"2024-04-25T00:01:46.649896+0000","last_fullsized":"2024-04-25T00:01:46.649896+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:40:36.350832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.658085+0000","last_change":"2024-04-25T00:01:05.376984+0000","last_active":"2024-04-25T00:01:46.658085+0000","last_peered":"2024-04-25T00:01:46.658085+0000","last_clean":"2024-04-25T00:01:46.658085+0000","last_became_active":"2024-04-25T00:01:05.376259+0000","last_became_peered":"2024-04-25T00:01:05.376259+0000","last_unstale":"2024-04-25T00:01:46.658085+0000","last_undegraded":"2024-04-25T00:01:46.658085+0000","last_fullsized":"2024-04-25T00:01:46.658085+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:13:28.543302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"22'2","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.658522+0000","last_change":"2024-04-25T00:01:04.359262+0000","last_active":"2024-04-25T00:01:46.658522+0000","last_peered":"2024-04-25T00:01:46.658522+0000","last_clean":"2024-04-25T00:01:46.658522+0000","last_became_active":"2024-04-25T00:01:04.357971+0000","last_became_peered":"2024-04-25T00:01:04.357971+0000","last_unstale":"2024-04-25T00:01:46.658522+0000","last_undegraded":"2024-04-25T00:01:46.658522+0000","last_fullsized":"2024-04-25T00:01:46.658522+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:05.441048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"26'1","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659290+0000","last_change":"2024-04-25T00:01:18.475624+0000","last_active":"2024-04-25T00:01:46.659290+0000","last_peered":"2024-04-25T00:01:46.659290+0000","last_clean":"2024-04-25T00:01:46.659290+0000","last_became_active":"2024-04-25T00:01:18.475149+0000","last_became_peered":"2024-04-25T00:01:18.475149+0000","last_unstale":"2024-04-25T00:01:46.659290+0000","last_undegraded":"2024-04-25T00:01:46.659290+0000","last_fullsized":"2024-04-25T00:01:46.659290+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:40:35.341075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.664518+0000","last_change":"2024-04-25T00:01:19.531113+0000","last_active":"2024-04-25T00:01:46.664518+0000","last_peered":"2024-04-25T00:01:46.664518+0000","last_clean":"2024-04-25T00:01:46.664518+0000","last_became_active":"2024-04-25T00:01:19.520150+0000","last_became_peered":"2024-04-25T00:01:19.520150+0000","last_unstale":"2024-04-25T00:01:46.664518+0000","last_undegraded":"2024-04-25T00:01:46.664518+0000","last_fullsized":"2024-04-25T00:01:46.664518+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:38.152376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659977+0000","last_change":"2024-04-25T00:01:05.362611+0000","last_active":"2024-04-25T00:01:46.659977+0000","last_peered":"2024-04-25T00:01:46.659977+0000","last_clean":"2024-04-25T00:01:46.659977+0000","last_became_active":"2024-04-25T00:01:05.361817+0000","last_became_peered":"2024-04-25T00:01:05.361817+0000","last_unstale":"2024-04-25T00:01:46.659977+0000","last_undegraded":"2024-04-25T00:01:46.659977+0000","last_fullsized":"2024-04-25T00:01:46.659977+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:50.489125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.663735+0000","last_change":"2024-04-25T00:01:04.371935+0000","last_active":"2024-04-25T00:01:46.663735+0000","last_peered":"2024-04-25T00:01:46.663735+0000","last_clean":"2024-04-25T00:01:46.663735+0000","last_became_active":"2024-04-25T00:01:04.371420+0000","last_became_peered":"2024-04-25T00:01:04.371420+0000","last_unstale":"2024-04-25T00:01:46.663735+0000","last_undegraded":"2024-04-25T00:01:46.663735+0000","last_fullsized":"2024-04-25T00:01:46.663735+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660605+0000","last_change":"2024-04-25T00:01:18.488623+0000","last_active":"2024-04-25T00:01:46.660605+0000","last_peered":"2024-04-25T00:01:46.660605+0000","last_clean":"2024-04-25T00:01:46.660605+0000","last_became_active":"2024-04-25T00:01:18.488291+0000","last_became_peered":"2024-04-25T00:01:18.488291+0000","last_unstale":"2024-04-25T00:01:46.660605+0000","last_undegraded":"2024-04-25T00:01:46.660605+0000","last_fullsized":"2024-04-25T00:01:46.660605+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:30:21.436681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.658546+0000","last_change":"2024-04-25T00:01:19.516731+0000","last_active":"2024-04-25T00:01:46.658546+0000","last_peered":"2024-04-25T00:01:46.658546+0000","last_clean":"2024-04-25T00:01:46.658546+0000","last_became_active":"2024-04-25T00:01:19.516405+0000","last_became_peered":"2024-04-25T00:01:19.516405+0000","last_unstale":"2024-04-25T00:01:46.658546+0000","last_undegraded":"2024-04-25T00:01:46.658546+0000","last_fullsized":"2024-04-25T00:01:46.658546+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:44:37.936092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.510871+0000","last_change":"2024-04-25T00:01:05.355455+0000","last_active":"2024-04-25T00:01:20.510871+0000","last_peered":"2024-04-25T00:01:20.510871+0000","last_clean":"2024-04-25T00:01:20.510871+0000","last_became_active":"2024-04-25T00:01:05.354699+0000","last_became_peered":"2024-04-25T00:01:05.354699+0000","last_unstale":"2024-04-25T00:01:20.510871+0000","last_undegraded":"2024-04-25T00:01:20.510871+0000","last_fullsized":"2024-04-25T00:01:20.510871+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:43:58.069271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.512349+0000","last_change":"2024-04-25T00:01:04.362268+0000","last_active":"2024-04-25T00:01:20.512349+0000","last_peered":"2024-04-25T00:01:20.512349+0000","last_clean":"2024-04-25T00:01:20.512349+0000","last_became_active":"2024-04-25T00:01:04.361622+0000","last_became_peered":"2024-04-25T00:01:04.361622+0000","last_unstale":"2024-04-25T00:01:20.512349+0000","last_undegraded":"2024-04-25T00:01:20.512349+0000","last_fullsized":"2024-04-25T00:01:20.512349+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:58:46.021343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"26'1","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.664549+0000","last_change":"2024-04-25T00:01:18.494047+0000","last_active":"2024-04-25T00:01:46.664549+0000","last_peered":"2024-04-25T00:01:46.664549+0000","last_clean":"2024-04-25T00:01:46.664549+0000","last_became_active":"2024-04-25T00:01:18.493614+0000","last_became_peered":"2024-04-25T00:01:18.493614+0000","last_unstale":"2024-04-25T00:01:46.664549+0000","last_undegraded":"2024-04-25T00:01:46.664549+0000","last_fullsized":"2024-04-25T00:01:46.664549+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:37:08.562016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657743+0000","last_change":"2024-04-25T00:01:19.505397+0000","last_active":"2024-04-25T00:01:46.657743+0000","last_peered":"2024-04-25T00:01:46.657743+0000","last_clean":"2024-04-25T00:01:46.657743+0000","last_became_active":"2024-04-25T00:01:19.505126+0000","last_became_peered":"2024-04-25T00:01:19.505126+0000","last_unstale":"2024-04-25T00:01:46.657743+0000","last_undegraded":"2024-04-25T00:01:46.657743+0000","last_fullsized":"2024-04-25T00:01:46.657743+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:08:49.640267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657635+0000","last_change":"2024-04-25T00:01:05.354953+0000","last_active":"2024-04-25T00:01:46.657635+0000","last_peered":"2024-04-25T00:01:46.657635+0000","last_clean":"2024-04-25T00:01:46.657635+0000","last_became_active":"2024-04-25T00:01:05.354118+0000","last_became_peered":"2024-04-25T00:01:05.354118+0000","last_unstale":"2024-04-25T00:01:46.657635+0000","last_undegraded":"2024-04-25T00:01:46.657635+0000","last_fullsized":"2024-04-25T00:01:46.657635+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:09.858617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657897+0000","last_change":"2024-04-25T00:01:04.365515+0000","last_active":"2024-04-25T00:01:46.657897+0000","last_peered":"2024-04-25T00:01:46.657897+0000","last_clean":"2024-04-25T00:01:46.657897+0000","last_became_active":"2024-04-25T00:01:04.364942+0000","last_became_peered":"2024-04-25T00:01:04.364942+0000","last_unstale":"2024-04-25T00:01:46.657897+0000","last_undegraded":"2024-04-25T00:01:46.657897+0000","last_fullsized":"2024-04-25T00:01:46.657897+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"26'2","reported_seq":20,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.648107+0000","last_change":"2024-04-25T00:01:18.476207+0000","last_active":"2024-04-25T00:01:46.648107+0000","last_peered":"2024-04-25T00:01:46.648107+0000","last_clean":"2024-04-25T00:01:46.648107+0000","last_became_active":"2024-04-25T00:01:18.475862+0000","last_became_peered":"2024-04-25T00:01:18.475862+0000","last_unstale":"2024-04-25T00:01:46.648107+0000","last_undegraded":"2024-04-25T00:01:46.648107+0000","last_fullsized":"2024-04-25T00:01:46.648107+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:11:40.170249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511258+0000","last_change":"2024-04-25T00:01:19.529607+0000","last_active":"2024-04-25T00:01:20.511258+0000","last_peered":"2024-04-25T00:01:20.511258+0000","last_clean":"2024-04-25T00:01:20.511258+0000","last_became_active":"2024-04-25T00:01:19.528509+0000","last_became_peered":"2024-04-25T00:01:19.528509+0000","last_unstale":"2024-04-25T00:01:20.511258+0000","last_undegraded":"2024-04-25T00:01:20.511258+0000","last_fullsized":"2024-04-25T00:01:20.511258+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:45:01.068091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.514958+0000","last_change":"2024-04-25T00:01:05.358106+0000","last_active":"2024-04-25T00:01:20.514958+0000","last_peered":"2024-04-25T00:01:20.514958+0000","last_clean":"2024-04-25T00:01:20.514958+0000","last_became_active":"2024-04-25T00:01:05.357135+0000","last_became_peered":"2024-04-25T00:01:05.357135+0000","last_unstale":"2024-04-25T00:01:20.514958+0000","last_undegraded":"2024-04-25T00:01:20.514958+0000","last_fullsized":"2024-04-25T00:01:20.514958+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:41:42.228234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"22'2","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660058+0000","last_change":"2024-04-25T00:01:04.369917+0000","last_active":"2024-04-25T00:01:46.660058+0000","last_peered":"2024-04-25T00:01:46.660058+0000","last_clean":"2024-04-25T00:01:46.660058+0000","last_became_active":"2024-04-25T00:01:04.369309+0000","last_became_peered":"2024-04-25T00:01:04.369309+0000","last_unstale":"2024-04-25T00:01:46.660058+0000","last_undegraded":"2024-04-25T00:01:46.660058+0000","last_fullsized":"2024-04-25T00:01:46.660058+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:55:08.063965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.647179+0000","last_change":"2024-04-25T00:01:18.477189+0000","last_active":"2024-04-25T00:01:46.647179+0000","last_peered":"2024-04-25T00:01:46.647179+0000","last_clean":"2024-04-25T00:01:46.647179+0000","last_became_active":"2024-04-25T00:01:18.476909+0000","last_became_peered":"2024-04-25T00:01:18.476909+0000","last_unstale":"2024-04-25T00:01:46.647179+0000","last_undegraded":"2024-04-25T00:01:46.647179+0000","last_fullsized":"2024-04-25T00:01:46.647179+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:36.995477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657252+0000","last_change":"2024-04-25T00:01:19.517500+0000","last_active":"2024-04-25T00:01:46.657252+0000","last_peered":"2024-04-25T00:01:46.657252+0000","last_clean":"2024-04-25T00:01:46.657252+0000","last_became_active":"2024-04-25T00:01:19.517182+0000","last_became_peered":"2024-04-25T00:01:19.517182+0000","last_unstale":"2024-04-25T00:01:46.657252+0000","last_undegraded":"2024-04-25T00:01:46.657252+0000","last_fullsized":"2024-04-25T00:01:46.657252+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:58:49.738233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.515146+0000","last_change":"2024-04-25T00:01:05.364829+0000","last_active":"2024-04-25T00:01:20.515146+0000","last_peered":"2024-04-25T00:01:20.515146+0000","last_clean":"2024-04-25T00:01:20.515146+0000","last_became_active":"2024-04-25T00:01:05.364125+0000","last_became_peered":"2024-04-25T00:01:05.364125+0000","last_unstale":"2024-04-25T00:01:20.515146+0000","last_undegraded":"2024-04-25T00:01:20.515146+0000","last_fullsized":"2024-04-25T00:01:20.515146+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:15:45.531760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"23'1","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.515821+0000","last_change":"2024-04-25T00:01:04.361603+0000","last_active":"2024-04-25T00:01:20.515821+0000","last_peered":"2024-04-25T00:01:20.515821+0000","last_clean":"2024-04-25T00:01:20.515821+0000","last_became_active":"2024-04-25T00:01:04.361046+0000","last_became_peered":"2024-04-25T00:01:04.361046+0000","last_unstale":"2024-04-25T00:01:20.515821+0000","last_undegraded":"2024-04-25T00:01:20.515821+0000","last_fullsized":"2024-04-25T00:01:20.515821+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:35:18.036843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.510620+0000","last_change":"2024-04-25T00:01:18.488352+0000","last_active":"2024-04-25T00:01:20.510620+0000","last_peered":"2024-04-25T00:01:20.510620+0000","last_clean":"2024-04-25T00:01:20.510620+0000","last_became_active":"2024-04-25T00:01:18.487321+0000","last_became_peered":"2024-04-25T00:01:18.487321+0000","last_unstale":"2024-04-25T00:01:20.510620+0000","last_undegraded":"2024-04-25T00:01:20.510620+0000","last_fullsized":"2024-04-25T00:01:20.510620+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:37:22.329655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.658811+0000","last_change":"2024-04-25T00:01:19.519653+0000","last_active":"2024-04-25T00:01:46.658811+0000","last_peered":"2024-04-25T00:01:46.658811+0000","last_clean":"2024-04-25T00:01:46.658811+0000","last_became_active":"2024-04-25T00:01:19.519052+0000","last_became_peered":"2024-04-25T00:01:19.519052+0000","last_unstale":"2024-04-25T00:01:46.658811+0000","last_undegraded":"2024-04-25T00:01:46.658811+0000","last_fullsized":"2024-04-25T00:01:46.658811+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:40:36.350832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.665974+0000","last_change":"2024-04-25T00:01:05.374011+0000","last_active":"2024-04-25T00:01:46.665974+0000","last_peered":"2024-04-25T00:01:46.665974+0000","last_clean":"2024-04-25T00:01:46.665974+0000","last_became_active":"2024-04-25T00:01:05.373276+0000","last_became_peered":"2024-04-25T00:01:05.373276+0000","last_unstale":"2024-04-25T00:01:46.665974+0000","last_undegraded":"2024-04-25T00:01:46.665974+0000","last_fullsized":"2024-04-25T00:01:46.665974+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:40:53.349428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659709+0000","last_change":"2024-04-25T00:01:04.365391+0000","last_active":"2024-04-25T00:01:46.659709+0000","last_peered":"2024-04-25T00:01:46.659709+0000","last_clean":"2024-04-25T00:01:46.659709+0000","last_became_active":"2024-04-25T00:01:04.364989+0000","last_became_peered":"2024-04-25T00:01:04.364989+0000","last_unstale":"2024-04-25T00:01:46.659709+0000","last_undegraded":"2024-04-25T00:01:46.659709+0000","last_fullsized":"2024-04-25T00:01:46.659709+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:06:18.298057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.663291+0000","last_change":"2024-04-25T00:01:18.491597+0000","last_active":"2024-04-25T00:01:46.663291+0000","last_peered":"2024-04-25T00:01:46.663291+0000","last_clean":"2024-04-25T00:01:46.663291+0000","last_became_active":"2024-04-25T00:01:18.490859+0000","last_became_peered":"2024-04-25T00:01:18.490859+0000","last_unstale":"2024-04-25T00:01:46.663291+0000","last_undegraded":"2024-04-25T00:01:46.663291+0000","last_fullsized":"2024-04-25T00:01:46.663291+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:36.995477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":10,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.670049+0000","last_change":"2024-04-25T00:01:46.667934+0000","last_active":"2024-04-25T00:01:46.670049+0000","last_peered":"2024-04-25T00:01:46.670049+0000","last_clean":"2024-04-25T00:01:46.670049+0000","last_became_active":"2024-04-25T00:01:46.667075+0000","last_became_peered":"2024-04-25T00:01:46.667075+0000","last_unstale":"2024-04-25T00:01:46.670049+0000","last_undegraded":"2024-04-25T00:01:46.670049+0000","last_fullsized":"2024-04-25T00:01:46.670049+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:28:18.234546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.648286+0000","last_change":"2024-04-25T00:01:05.378340+0000","last_active":"2024-04-25T00:01:46.648286+0000","last_peered":"2024-04-25T00:01:46.648286+0000","last_clean":"2024-04-25T00:01:46.648286+0000","last_became_active":"2024-04-25T00:01:05.377593+0000","last_became_peered":"2024-04-25T00:01:05.377593+0000","last_unstale":"2024-04-25T00:01:46.648286+0000","last_undegraded":"2024-04-25T00:01:46.648286+0000","last_fullsized":"2024-04-25T00:01:46.648286+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:48.202251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.650594+0000","last_change":"2024-04-25T00:01:04.366577+0000","last_active":"2024-04-25T00:01:46.650594+0000","last_peered":"2024-04-25T00:01:46.650594+0000","last_clean":"2024-04-25T00:01:46.650594+0000","last_became_active":"2024-04-25T00:01:04.365881+0000","last_became_peered":"2024-04-25T00:01:04.365881+0000","last_unstale":"2024-04-25T00:01:46.650594+0000","last_undegraded":"2024-04-25T00:01:46.650594+0000","last_fullsized":"2024-04-25T00:01:46.650594+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:49:22.673421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.662487+0000","last_change":"2024-04-25T00:01:18.504341+0000","last_active":"2024-04-25T00:01:46.662487+0000","last_peered":"2024-04-25T00:01:46.662487+0000","last_clean":"2024-04-25T00:01:46.662487+0000","last_became_active":"2024-04-25T00:01:18.491498+0000","last_became_peered":"2024-04-25T00:01:18.491498+0000","last_unstale":"2024-04-25T00:01:46.662487+0000","last_undegraded":"2024-04-25T00:01:46.662487+0000","last_fullsized":"2024-04-25T00:01:46.662487+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:56.741250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.667022+0000","last_change":"2024-04-25T00:01:46.667022+0000","last_active":"2024-04-25T00:01:46.667022+0000","last_peered":"2024-04-25T00:01:46.667022+0000","last_clean":"2024-04-25T00:01:46.667022+0000","last_became_active":"2024-04-25T00:01:46.666224+0000","last_became_peered":"2024-04-25T00:01:46.666224+0000","last_unstale":"2024-04-25T00:01:46.667022+0000","last_undegraded":"2024-04-25T00:01:46.667022+0000","last_fullsized":"2024-04-25T00:01:46.667022+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:25:20.185355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659841+0000","last_change":"2024-04-25T00:01:05.368769+0000","last_active":"2024-04-25T00:01:46.659841+0000","last_peered":"2024-04-25T00:01:46.659841+0000","last_clean":"2024-04-25T00:01:46.659841+0000","last_became_active":"2024-04-25T00:01:05.366817+0000","last_became_peered":"2024-04-25T00:01:05.366817+0000","last_unstale":"2024-04-25T00:01:46.659841+0000","last_undegraded":"2024-04-25T00:01:46.659841+0000","last_fullsized":"2024-04-25T00:01:46.659841+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:11:27.044893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.658528+0000","last_change":"2024-04-25T00:01:04.361900+0000","last_active":"2024-04-25T00:01:46.658528+0000","last_peered":"2024-04-25T00:01:46.658528+0000","last_clean":"2024-04-25T00:01:46.658528+0000","last_became_active":"2024-04-25T00:01:04.361572+0000","last_became_peered":"2024-04-25T00:01:04.361572+0000","last_unstale":"2024-04-25T00:01:46.658528+0000","last_undegraded":"2024-04-25T00:01:46.658528+0000","last_fullsized":"2024-04-25T00:01:46.658528+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:29.218739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.665621+0000","last_change":"2024-04-25T00:01:18.504139+0000","last_active":"2024-04-25T00:01:46.665621+0000","last_peered":"2024-04-25T00:01:46.665621+0000","last_clean":"2024-04-25T00:01:46.665621+0000","last_became_active":"2024-04-25T00:01:18.503379+0000","last_became_peered":"2024-04-25T00:01:18.503379+0000","last_unstale":"2024-04-25T00:01:46.665621+0000","last_undegraded":"2024-04-25T00:01:46.665621+0000","last_fullsized":"2024-04-25T00:01:46.665621+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:02:56.363258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.664313+0000","last_change":"2024-04-25T00:01:19.531756+0000","last_active":"2024-04-25T00:01:46.664313+0000","last_peered":"2024-04-25T00:01:46.664313+0000","last_clean":"2024-04-25T00:01:46.664313+0000","last_became_active":"2024-04-25T00:01:19.531145+0000","last_became_peered":"2024-04-25T00:01:19.531145+0000","last_unstale":"2024-04-25T00:01:46.664313+0000","last_undegraded":"2024-04-25T00:01:46.664313+0000","last_fullsized":"2024-04-25T00:01:46.664313+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:13:42.678415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.649319+0000","last_change":"2024-04-25T00:01:05.354557+0000","last_active":"2024-04-25T00:01:46.649319+0000","last_peered":"2024-04-25T00:01:46.649319+0000","last_clean":"2024-04-25T00:01:46.649319+0000","last_became_active":"2024-04-25T00:01:05.353556+0000","last_became_peered":"2024-04-25T00:01:05.353556+0000","last_unstale":"2024-04-25T00:01:46.649319+0000","last_undegraded":"2024-04-25T00:01:46.649319+0000","last_fullsized":"2024-04-25T00:01:46.649319+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:24.964055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660230+0000","last_change":"2024-04-25T00:01:04.372916+0000","last_active":"2024-04-25T00:01:46.660230+0000","last_peered":"2024-04-25T00:01:46.660230+0000","last_clean":"2024-04-25T00:01:46.660230+0000","last_became_active":"2024-04-25T00:01:04.372185+0000","last_became_peered":"2024-04-25T00:01:04.372185+0000","last_unstale":"2024-04-25T00:01:46.660230+0000","last_undegraded":"2024-04-25T00:01:46.660230+0000","last_fullsized":"2024-04-25T00:01:46.660230+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"26'2","reported_seq":20,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.665571+0000","last_change":"2024-04-25T00:01:18.504614+0000","last_active":"2024-04-25T00:01:46.665571+0000","last_peered":"2024-04-25T00:01:46.665571+0000","last_clean":"2024-04-25T00:01:46.665571+0000","last_became_active":"2024-04-25T00:01:18.501448+0000","last_became_peered":"2024-04-25T00:01:18.501448+0000","last_unstale":"2024-04-25T00:01:46.665571+0000","last_undegraded":"2024-04-25T00:01:46.665571+0000","last_fullsized":"2024-04-25T00:01:46.665571+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:22.983972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659568+0000","last_change":"2024-04-25T00:01:19.517510+0000","last_active":"2024-04-25T00:01:46.659568+0000","last_peered":"2024-04-25T00:01:46.659568+0000","last_clean":"2024-04-25T00:01:46.659568+0000","last_became_active":"2024-04-25T00:01:19.517243+0000","last_became_peered":"2024-04-25T00:01:19.517243+0000","last_unstale":"2024-04-25T00:01:46.659568+0000","last_undegraded":"2024-04-25T00:01:46.659568+0000","last_fullsized":"2024-04-25T00:01:46.659568+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:42:38.149502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.662201+0000","last_change":"2024-04-25T00:01:05.370968+0000","last_active":"2024-04-25T00:01:46.662201+0000","last_peered":"2024-04-25T00:01:46.662201+0000","last_clean":"2024-04-25T00:01:46.662201+0000","last_became_active":"2024-04-25T00:01:05.369987+0000","last_became_peered":"2024-04-25T00:01:05.369987+0000","last_unstale":"2024-04-25T00:01:46.662201+0000","last_undegraded":"2024-04-25T00:01:46.662201+0000","last_fullsized":"2024-04-25T00:01:46.662201+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657919+0000","last_change":"2024-04-25T00:01:04.370226+0000","last_active":"2024-04-25T00:01:46.657919+0000","last_peered":"2024-04-25T00:01:46.657919+0000","last_clean":"2024-04-25T00:01:46.657919+0000","last_became_active":"2024-04-25T00:01:04.369904+0000","last_became_peered":"2024-04-25T00:01:04.369904+0000","last_unstale":"2024-04-25T00:01:46.657919+0000","last_undegraded":"2024-04-25T00:01:46.657919+0000","last_fullsized":"2024-04-25T00:01:46.657919+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:47:25.595275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"27'1","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.652131+0000","last_change":"2024-04-25T00:01:18.497156+0000","last_active":"2024-04-25T00:01:46.652131+0000","last_peered":"2024-04-25T00:01:46.652131+0000","last_clean":"2024-04-25T00:01:46.652131+0000","last_became_active":"2024-04-25T00:01:18.495573+0000","last_became_peered":"2024-04-25T00:01:18.495573+0000","last_unstale":"2024-04-25T00:01:46.652131+0000","last_undegraded":"2024-04-25T00:01:46.652131+0000","last_fullsized":"2024-04-25T00:01:46.652131+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:29.119299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660477+0000","last_change":"2024-04-25T00:01:19.519411+0000","last_active":"2024-04-25T00:01:46.660477+0000","last_peered":"2024-04-25T00:01:46.660477+0000","last_clean":"2024-04-25T00:01:46.660477+0000","last_became_active":"2024-04-25T00:01:19.519020+0000","last_became_peered":"2024-04-25T00:01:19.519020+0000","last_unstale":"2024-04-25T00:01:46.660477+0000","last_undegraded":"2024-04-25T00:01:46.660477+0000","last_fullsized":"2024-04-25T00:01:46.660477+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:50.351599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.513395+0000","last_change":"2024-04-25T00:01:05.364939+0000","last_active":"2024-04-25T00:01:20.513395+0000","last_peered":"2024-04-25T00:01:20.513395+0000","last_clean":"2024-04-25T00:01:20.513395+0000","last_became_active":"2024-04-25T00:01:05.364272+0000","last_became_peered":"2024-04-25T00:01:05.364272+0000","last_unstale":"2024-04-25T00:01:20.513395+0000","last_undegraded":"2024-04-25T00:01:20.513395+0000","last_fullsized":"2024-04-25T00:01:20.513395+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:24:59.679908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.658666+0000","last_change":"2024-04-25T00:01:04.357028+0000","last_active":"2024-04-25T00:01:46.658666+0000","last_peered":"2024-04-25T00:01:46.658666+0000","last_clean":"2024-04-25T00:01:46.658666+0000","last_became_active":"2024-04-25T00:01:04.356644+0000","last_became_peered":"2024-04-25T00:01:04.356644+0000","last_unstale":"2024-04-25T00:01:46.658666+0000","last_undegraded":"2024-04-25T00:01:46.658666+0000","last_fullsized":"2024-04-25T00:01:46.658666+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:21.729893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":182,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657375+0000","last_change":"2024-04-25T00:00:55.084492+0000","last_active":"2024-04-25T00:01:46.657375+0000","last_peered":"2024-04-25T00:01:46.657375+0000","last_clean":"2024-04-25T00:01:46.657375+0000","last_became_active":"2024-04-25T00:00:55.083493+0000","last_became_peered":"2024-04-25T00:00:55.083493+0000","last_unstale":"2024-04-25T00:01:46.657375+0000","last_undegraded":"2024-04-25T00:01:46.657375+0000","last_fullsized":"2024-04-25T00:01:46.657375+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:00:54.060578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:00:54.060578+0000","last_clean_scrub_stamp":"2024-04-25T00:00:54.060578+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:14:58.472477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660166+0000","last_change":"2024-04-25T00:01:18.476298+0000","last_active":"2024-04-25T00:01:46.660166+0000","last_peered":"2024-04-25T00:01:46.660166+0000","last_clean":"2024-04-25T00:01:46.660166+0000","last_became_active":"2024-04-25T00:01:18.475898+0000","last_became_peered":"2024-04-25T00:01:18.475898+0000","last_unstale":"2024-04-25T00:01:46.660166+0000","last_undegraded":"2024-04-25T00:01:46.660166+0000","last_fullsized":"2024-04-25T00:01:46.660166+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:49.341842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.652839+0000","last_change":"2024-04-25T00:01:19.518102+0000","last_active":"2024-04-25T00:01:46.652839+0000","last_peered":"2024-04-25T00:01:46.652839+0000","last_clean":"2024-04-25T00:01:46.652839+0000","last_became_active":"2024-04-25T00:01:19.517376+0000","last_became_peered":"2024-04-25T00:01:19.517376+0000","last_unstale":"2024-04-25T00:01:46.652839+0000","last_undegraded":"2024-04-25T00:01:46.652839+0000","last_fullsized":"2024-04-25T00:01:46.652839+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:55:46.853565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659553+0000","last_change":"2024-04-25T00:01:05.376609+0000","last_active":"2024-04-25T00:01:46.659553+0000","last_peered":"2024-04-25T00:01:46.659553+0000","last_clean":"2024-04-25T00:01:46.659553+0000","last_became_active":"2024-04-25T00:01:05.375881+0000","last_became_peered":"2024-04-25T00:01:05.375881+0000","last_unstale":"2024-04-25T00:01:46.659553+0000","last_undegraded":"2024-04-25T00:01:46.659553+0000","last_fullsized":"2024-04-25T00:01:46.659553+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:46:05.592233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.666092+0000","last_change":"2024-04-25T00:01:04.368954+0000","last_active":"2024-04-25T00:01:46.666092+0000","last_peered":"2024-04-25T00:01:46.666092+0000","last_clean":"2024-04-25T00:01:46.666092+0000","last_became_active":"2024-04-25T00:01:04.367903+0000","last_became_peered":"2024-04-25T00:01:04.367903+0000","last_unstale":"2024-04-25T00:01:46.666092+0000","last_undegraded":"2024-04-25T00:01:46.666092+0000","last_fullsized":"2024-04-25T00:01:46.666092+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.647171+0000","last_change":"2024-04-25T00:01:18.476525+0000","last_active":"2024-04-25T00:01:46.647171+0000","last_peered":"2024-04-25T00:01:46.647171+0000","last_clean":"2024-04-25T00:01:46.647171+0000","last_became_active":"2024-04-25T00:01:18.476247+0000","last_became_peered":"2024-04-25T00:01:18.476247+0000","last_unstale":"2024-04-25T00:01:46.647171+0000","last_undegraded":"2024-04-25T00:01:46.647171+0000","last_fullsized":"2024-04-25T00:01:46.647171+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:30:36.682736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.662181+0000","last_change":"2024-04-25T00:01:19.514615+0000","last_active":"2024-04-25T00:01:46.662181+0000","last_peered":"2024-04-25T00:01:46.662181+0000","last_clean":"2024-04-25T00:01:46.662181+0000","last_became_active":"2024-04-25T00:01:19.513640+0000","last_became_peered":"2024-04-25T00:01:19.513640+0000","last_unstale":"2024-04-25T00:01:46.662181+0000","last_undegraded":"2024-04-25T00:01:46.662181+0000","last_fullsized":"2024-04-25T00:01:46.662181+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:08.620602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660785+0000","last_change":"2024-04-25T00:01:05.385670+0000","last_active":"2024-04-25T00:01:46.660785+0000","last_peered":"2024-04-25T00:01:46.660785+0000","last_clean":"2024-04-25T00:01:46.660785+0000","last_became_active":"2024-04-25T00:01:05.384822+0000","last_became_peered":"2024-04-25T00:01:05.384822+0000","last_unstale":"2024-04-25T00:01:46.660785+0000","last_undegraded":"2024-04-25T00:01:46.660785+0000","last_fullsized":"2024-04-25T00:01:46.660785+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:24.017263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"22'1","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.650515+0000","last_change":"2024-04-25T00:01:04.356627+0000","last_active":"2024-04-25T00:01:46.650515+0000","last_peered":"2024-04-25T00:01:46.650515+0000","last_clean":"2024-04-25T00:01:46.650515+0000","last_became_active":"2024-04-25T00:01:04.354419+0000","last_became_peered":"2024-04-25T00:01:04.354419+0000","last_unstale":"2024-04-25T00:01:46.650515+0000","last_undegraded":"2024-04-25T00:01:46.650515+0000","last_fullsized":"2024-04-25T00:01:46.650515+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:26.243635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"26'2","reported_seq":20,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.650452+0000","last_change":"2024-04-25T00:01:18.490511+0000","last_active":"2024-04-25T00:01:46.650452+0000","last_peered":"2024-04-25T00:01:46.650452+0000","last_clean":"2024-04-25T00:01:46.650452+0000","last_became_active":"2024-04-25T00:01:18.489629+0000","last_became_peered":"2024-04-25T00:01:18.489629+0000","last_unstale":"2024-04-25T00:01:46.650452+0000","last_undegraded":"2024-04-25T00:01:46.650452+0000","last_fullsized":"2024-04-25T00:01:46.650452+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:07.610846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657253+0000","last_change":"2024-04-25T00:01:19.508931+0000","last_active":"2024-04-25T00:01:46.657253+0000","last_peered":"2024-04-25T00:01:46.657253+0000","last_clean":"2024-04-25T00:01:46.657253+0000","last_became_active":"2024-04-25T00:01:19.507867+0000","last_became_peered":"2024-04-25T00:01:19.507867+0000","last_unstale":"2024-04-25T00:01:46.657253+0000","last_undegraded":"2024-04-25T00:01:46.657253+0000","last_fullsized":"2024-04-25T00:01:46.657253+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:38.152376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660540+0000","last_change":"2024-04-25T00:01:05.378840+0000","last_active":"2024-04-25T00:01:46.660540+0000","last_peered":"2024-04-25T00:01:46.660540+0000","last_clean":"2024-04-25T00:01:46.660540+0000","last_became_active":"2024-04-25T00:01:05.378112+0000","last_became_peered":"2024-04-25T00:01:05.378112+0000","last_unstale":"2024-04-25T00:01:46.660540+0000","last_undegraded":"2024-04-25T00:01:46.660540+0000","last_fullsized":"2024-04-25T00:01:46.660540+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:43.615895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"22'2","reported_seq":24,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.515607+0000","last_change":"2024-04-25T00:01:04.370794+0000","last_active":"2024-04-25T00:01:20.515607+0000","last_peered":"2024-04-25T00:01:20.515607+0000","last_clean":"2024-04-25T00:01:20.515607+0000","last_became_active":"2024-04-25T00:01:04.370099+0000","last_became_peered":"2024-04-25T00:01:04.370099+0000","last_unstale":"2024-04-25T00:01:20.515607+0000","last_undegraded":"2024-04-25T00:01:20.515607+0000","last_fullsized":"2024-04-25T00:01:20.515607+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:42:03.181036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657372+0000","last_change":"2024-04-25T00:01:18.475256+0000","last_active":"2024-04-25T00:01:46.657372+0000","last_peered":"2024-04-25T00:01:46.657372+0000","last_clean":"2024-04-25T00:01:46.657372+0000","last_became_active":"2024-04-25T00:01:18.474952+0000","last_became_peered":"2024-04-25T00:01:18.474952+0000","last_unstale":"2024-04-25T00:01:46.657372+0000","last_undegraded":"2024-04-25T00:01:46.657372+0000","last_fullsized":"2024-04-25T00:01:46.657372+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:46.849870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660859+0000","last_change":"2024-04-25T00:01:19.516833+0000","last_active":"2024-04-25T00:01:46.660859+0000","last_peered":"2024-04-25T00:01:46.660859+0000","last_clean":"2024-04-25T00:01:46.660859+0000","last_became_active":"2024-04-25T00:01:19.515926+0000","last_became_peered":"2024-04-25T00:01:19.515926+0000","last_unstale":"2024-04-25T00:01:46.660859+0000","last_undegraded":"2024-04-25T00:01:46.660859+0000","last_fullsized":"2024-04-25T00:01:46.660859+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:23:48.855381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.648328+0000","last_change":"2024-04-25T00:01:05.372591+0000","last_active":"2024-04-25T00:01:46.648328+0000","last_peered":"2024-04-25T00:01:46.648328+0000","last_clean":"2024-04-25T00:01:46.648328+0000","last_became_active":"2024-04-25T00:01:05.371204+0000","last_became_peered":"2024-04-25T00:01:05.371204+0000","last_unstale":"2024-04-25T00:01:46.648328+0000","last_undegraded":"2024-04-25T00:01:46.648328+0000","last_fullsized":"2024-04-25T00:01:46.648328+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:15.879673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"22'1","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660665+0000","last_change":"2024-04-25T00:01:04.372376+0000","last_active":"2024-04-25T00:01:46.660665+0000","last_peered":"2024-04-25T00:01:46.660665+0000","last_clean":"2024-04-25T00:01:46.660665+0000","last_became_active":"2024-04-25T00:01:04.372040+0000","last_became_peered":"2024-04-25T00:01:04.372040+0000","last_unstale":"2024-04-25T00:01:46.660665+0000","last_undegraded":"2024-04-25T00:01:46.660665+0000","last_fullsized":"2024-04-25T00:01:46.660665+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:21.729893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657413+0000","last_change":"2024-04-25T00:01:18.487548+0000","last_active":"2024-04-25T00:01:46.657413+0000","last_peered":"2024-04-25T00:01:46.657413+0000","last_clean":"2024-04-25T00:01:46.657413+0000","last_became_active":"2024-04-25T00:01:18.487227+0000","last_became_peered":"2024-04-25T00:01:18.487227+0000","last_unstale":"2024-04-25T00:01:46.657413+0000","last_undegraded":"2024-04-25T00:01:46.657413+0000","last_fullsized":"2024-04-25T00:01:46.657413+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:55.074018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.514828+0000","last_change":"2024-04-25T00:01:19.528574+0000","last_active":"2024-04-25T00:01:20.514828+0000","last_peered":"2024-04-25T00:01:20.514828+0000","last_clean":"2024-04-25T00:01:20.514828+0000","last_became_active":"2024-04-25T00:01:19.525531+0000","last_became_peered":"2024-04-25T00:01:19.525531+0000","last_unstale":"2024-04-25T00:01:20.514828+0000","last_undegraded":"2024-04-25T00:01:20.514828+0000","last_fullsized":"2024-04-25T00:01:20.514828+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:47:08.132895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657517+0000","last_change":"2024-04-25T00:01:05.364103+0000","last_active":"2024-04-25T00:01:46.657517+0000","last_peered":"2024-04-25T00:01:46.657517+0000","last_clean":"2024-04-25T00:01:46.657517+0000","last_became_active":"2024-04-25T00:01:05.363320+0000","last_became_peered":"2024-04-25T00:01:05.363320+0000","last_unstale":"2024-04-25T00:01:46.657517+0000","last_undegraded":"2024-04-25T00:01:46.657517+0000","last_fullsized":"2024-04-25T00:01:46.657517+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:31.456792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"22'8","reported_seq":30,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511508+0000","last_change":"2024-04-25T00:01:04.358365+0000","last_active":"2024-04-25T00:01:20.511508+0000","last_peered":"2024-04-25T00:01:20.511508+0000","last_clean":"2024-04-25T00:01:20.511508+0000","last_became_active":"2024-04-25T00:01:04.357571+0000","last_became_peered":"2024-04-25T00:01:04.357571+0000","last_unstale":"2024-04-25T00:01:20.511508+0000","last_undegraded":"2024-04-25T00:01:20.511508+0000","last_fullsized":"2024-04-25T00:01:20.511508+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:50:23.023114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.651115+0000","last_change":"2024-04-25T00:01:18.497679+0000","last_active":"2024-04-25T00:01:46.651115+0000","last_peered":"2024-04-25T00:01:46.651115+0000","last_clean":"2024-04-25T00:01:46.651115+0000","last_became_active":"2024-04-25T00:01:18.496489+0000","last_became_peered":"2024-04-25T00:01:18.496489+0000","last_unstale":"2024-04-25T00:01:46.651115+0000","last_undegraded":"2024-04-25T00:01:46.651115+0000","last_fullsized":"2024-04-25T00:01:46.651115+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:23:47.845624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.658432+0000","last_change":"2024-04-25T00:01:19.498082+0000","last_active":"2024-04-25T00:01:46.658432+0000","last_peered":"2024-04-25T00:01:46.658432+0000","last_clean":"2024-04-25T00:01:46.658432+0000","last_became_active":"2024-04-25T00:01:19.496805+0000","last_became_peered":"2024-04-25T00:01:19.496805+0000","last_unstale":"2024-04-25T00:01:46.658432+0000","last_undegraded":"2024-04-25T00:01:46.658432+0000","last_fullsized":"2024-04-25T00:01:46.658432+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:57:36.256250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657234+0000","last_change":"2024-04-25T00:01:05.373927+0000","last_active":"2024-04-25T00:01:46.657234+0000","last_peered":"2024-04-25T00:01:46.657234+0000","last_clean":"2024-04-25T00:01:46.657234+0000","last_became_active":"2024-04-25T00:01:05.373077+0000","last_became_peered":"2024-04-25T00:01:05.373077+0000","last_unstale":"2024-04-25T00:01:46.657234+0000","last_undegraded":"2024-04-25T00:01:46.657234+0000","last_fullsized":"2024-04-25T00:01:46.657234+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:50.489125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.658723+0000","last_change":"2024-04-25T00:01:04.357265+0000","last_active":"2024-04-25T00:01:46.658723+0000","last_peered":"2024-04-25T00:01:46.658723+0000","last_clean":"2024-04-25T00:01:46.658723+0000","last_became_active":"2024-04-25T00:01:04.357004+0000","last_became_peered":"2024-04-25T00:01:04.357004+0000","last_unstale":"2024-04-25T00:01:46.658723+0000","last_undegraded":"2024-04-25T00:01:46.658723+0000","last_fullsized":"2024-04-25T00:01:46.658723+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:47:25.595275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.653006+0000","last_change":"2024-04-25T00:01:18.487276+0000","last_active":"2024-04-25T00:01:46.653006+0000","last_peered":"2024-04-25T00:01:46.653006+0000","last_clean":"2024-04-25T00:01:46.653006+0000","last_became_active":"2024-04-25T00:01:18.486279+0000","last_became_peered":"2024-04-25T00:01:18.486279+0000","last_unstale":"2024-04-25T00:01:46.653006+0000","last_undegraded":"2024-04-25T00:01:46.653006+0000","last_fullsized":"2024-04-25T00:01:46.653006+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:17:37.141843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659022+0000","last_change":"2024-04-25T00:01:19.516827+0000","last_active":"2024-04-25T00:01:46.659022+0000","last_peered":"2024-04-25T00:01:46.659022+0000","last_clean":"2024-04-25T00:01:46.659022+0000","last_became_active":"2024-04-25T00:01:19.515765+0000","last_became_peered":"2024-04-25T00:01:19.515765+0000","last_unstale":"2024-04-25T00:01:46.659022+0000","last_undegraded":"2024-04-25T00:01:46.659022+0000","last_fullsized":"2024-04-25T00:01:46.659022+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:13:34.153117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.658378+0000","last_change":"2024-04-25T00:01:05.379501+0000","last_active":"2024-04-25T00:01:46.658378+0000","last_peered":"2024-04-25T00:01:46.658378+0000","last_clean":"2024-04-25T00:01:46.658378+0000","last_became_active":"2024-04-25T00:01:05.378644+0000","last_became_peered":"2024-04-25T00:01:05.378644+0000","last_unstale":"2024-04-25T00:01:46.658378+0000","last_undegraded":"2024-04-25T00:01:46.658378+0000","last_fullsized":"2024-04-25T00:01:46.658378+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:08:35.505154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.658345+0000","last_change":"2024-04-25T00:01:04.360987+0000","last_active":"2024-04-25T00:01:46.658345+0000","last_peered":"2024-04-25T00:01:46.658345+0000","last_clean":"2024-04-25T00:01:46.658345+0000","last_became_active":"2024-04-25T00:01:04.360664+0000","last_became_peered":"2024-04-25T00:01:04.360664+0000","last_unstale":"2024-04-25T00:01:46.658345+0000","last_undegraded":"2024-04-25T00:01:46.658345+0000","last_fullsized":"2024-04-25T00:01:46.658345+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:55.283076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"27'8","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.665406+0000","last_change":"2024-04-25T00:01:18.505577+0000","last_active":"2024-04-25T00:01:46.665406+0000","last_peered":"2024-04-25T00:01:46.665406+0000","last_clean":"2024-04-25T00:01:46.665406+0000","last_became_active":"2024-04-25T00:01:18.504963+0000","last_became_peered":"2024-04-25T00:01:18.504963+0000","last_unstale":"2024-04-25T00:01:46.665406+0000","last_undegraded":"2024-04-25T00:01:46.665406+0000","last_fullsized":"2024-04-25T00:01:46.665406+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:37.142619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2053,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":10,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.662316+0000","last_change":"2024-04-25T00:01:46.658442+0000","last_active":"2024-04-25T00:01:46.662316+0000","last_peered":"2024-04-25T00:01:46.662316+0000","last_clean":"2024-04-25T00:01:46.662316+0000","last_became_active":"2024-04-25T00:01:46.657588+0000","last_became_peered":"2024-04-25T00:01:46.657588+0000","last_unstale":"2024-04-25T00:01:46.662316+0000","last_undegraded":"2024-04-25T00:01:46.662316+0000","last_fullsized":"2024-04-25T00:01:46.662316+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:58:49.738233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.663479+0000","last_change":"2024-04-25T00:01:05.379102+0000","last_active":"2024-04-25T00:01:46.663479+0000","last_peered":"2024-04-25T00:01:46.663479+0000","last_clean":"2024-04-25T00:01:46.663479+0000","last_became_active":"2024-04-25T00:01:05.377625+0000","last_became_peered":"2024-04-25T00:01:05.377625+0000","last_unstale":"2024-04-25T00:01:46.663479+0000","last_undegraded":"2024-04-25T00:01:46.663479+0000","last_fullsized":"2024-04-25T00:01:46.663479+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:15.879673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"22'2","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660428+0000","last_change":"2024-04-25T00:01:04.370640+0000","last_active":"2024-04-25T00:01:46.660428+0000","last_peered":"2024-04-25T00:01:46.660428+0000","last_clean":"2024-04-25T00:01:46.660428+0000","last_became_active":"2024-04-25T00:01:04.370031+0000","last_became_peered":"2024-04-25T00:01:04.370031+0000","last_unstale":"2024-04-25T00:01:46.660428+0000","last_undegraded":"2024-04-25T00:01:46.660428+0000","last_fullsized":"2024-04-25T00:01:46.660428+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:26.243635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"26'3","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659370+0000","last_change":"2024-04-25T00:01:18.473113+0000","last_active":"2024-04-25T00:01:46.659370+0000","last_peered":"2024-04-25T00:01:46.659370+0000","last_clean":"2024-04-25T00:01:46.659370+0000","last_became_active":"2024-04-25T00:01:18.472931+0000","last_became_peered":"2024-04-25T00:01:18.472931+0000","last_unstale":"2024-04-25T00:01:46.659370+0000","last_undegraded":"2024-04-25T00:01:46.659370+0000","last_fullsized":"2024-04-25T00:01:46.659370+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:07.498985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.512763+0000","last_change":"2024-04-25T00:01:19.533664+0000","last_active":"2024-04-25T00:01:20.512763+0000","last_peered":"2024-04-25T00:01:20.512763+0000","last_clean":"2024-04-25T00:01:20.512763+0000","last_became_active":"2024-04-25T00:01:19.533048+0000","last_became_peered":"2024-04-25T00:01:19.533048+0000","last_unstale":"2024-04-25T00:01:20.512763+0000","last_undegraded":"2024-04-25T00:01:20.512763+0000","last_fullsized":"2024-04-25T00:01:20.512763+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:45:43.141569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660317+0000","last_change":"2024-04-25T00:01:05.385827+0000","last_active":"2024-04-25T00:01:46.660317+0000","last_peered":"2024-04-25T00:01:46.660317+0000","last_clean":"2024-04-25T00:01:46.660317+0000","last_became_active":"2024-04-25T00:01:05.385258+0000","last_became_peered":"2024-04-25T00:01:05.385258+0000","last_unstale":"2024-04-25T00:01:46.660317+0000","last_undegraded":"2024-04-25T00:01:46.660317+0000","last_fullsized":"2024-04-25T00:01:46.660317+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:23.870121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"27'8","reported_seq":34,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.648163+0000","last_change":"2024-04-25T00:01:04.365185+0000","last_active":"2024-04-25T00:01:46.648163+0000","last_peered":"2024-04-25T00:01:46.648163+0000","last_clean":"2024-04-25T00:01:46.648163+0000","last_became_active":"2024-04-25T00:01:04.364930+0000","last_became_peered":"2024-04-25T00:01:04.364930+0000","last_unstale":"2024-04-25T00:01:46.648163+0000","last_undegraded":"2024-04-25T00:01:46.648163+0000","last_fullsized":"2024-04-25T00:01:46.648163+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:45:03.618340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2053,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.512982+0000","last_change":"2024-04-25T00:01:18.496220+0000","last_active":"2024-04-25T00:01:20.512982+0000","last_peered":"2024-04-25T00:01:20.512982+0000","last_clean":"2024-04-25T00:01:20.512982+0000","last_became_active":"2024-04-25T00:01:18.492850+0000","last_became_peered":"2024-04-25T00:01:18.492850+0000","last_unstale":"2024-04-25T00:01:20.512982+0000","last_undegraded":"2024-04-25T00:01:20.512982+0000","last_fullsized":"2024-04-25T00:01:20.512982+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:50:16.922314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.662739+0000","last_change":"2024-04-25T00:01:19.515725+0000","last_active":"2024-04-25T00:01:46.662739+0000","last_peered":"2024-04-25T00:01:46.662739+0000","last_clean":"2024-04-25T00:01:46.662739+0000","last_became_active":"2024-04-25T00:01:19.509285+0000","last_became_peered":"2024-04-25T00:01:19.509285+0000","last_unstale":"2024-04-25T00:01:46.662739+0000","last_undegraded":"2024-04-25T00:01:46.662739+0000","last_fullsized":"2024-04-25T00:01:46.662739+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:25:20.185355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.658824+0000","last_change":"2024-04-25T00:01:05.371061+0000","last_active":"2024-04-25T00:01:46.658824+0000","last_peered":"2024-04-25T00:01:46.658824+0000","last_clean":"2024-04-25T00:01:46.658824+0000","last_became_active":"2024-04-25T00:01:05.369950+0000","last_became_peered":"2024-04-25T00:01:05.369950+0000","last_unstale":"2024-04-25T00:01:46.658824+0000","last_undegraded":"2024-04-25T00:01:46.658824+0000","last_fullsized":"2024-04-25T00:01:46.658824+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:02:43.237902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"22'3","reported_seq":29,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.665160+0000","last_change":"2024-04-25T00:01:04.366503+0000","last_active":"2024-04-25T00:01:46.665160+0000","last_peered":"2024-04-25T00:01:46.665160+0000","last_clean":"2024-04-25T00:01:46.665160+0000","last_became_active":"2024-04-25T00:01:04.365945+0000","last_became_peered":"2024-04-25T00:01:04.365945+0000","last_unstale":"2024-04-25T00:01:46.665160+0000","last_undegraded":"2024-04-25T00:01:46.665160+0000","last_fullsized":"2024-04-25T00:01:46.665160+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"26'2","reported_seq":20,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.647716+0000","last_change":"2024-04-25T00:01:18.476456+0000","last_active":"2024-04-25T00:01:46.647716+0000","last_peered":"2024-04-25T00:01:46.647716+0000","last_clean":"2024-04-25T00:01:46.647716+0000","last_became_active":"2024-04-25T00:01:18.475876+0000","last_became_peered":"2024-04-25T00:01:18.475876+0000","last_unstale":"2024-04-25T00:01:46.647716+0000","last_undegraded":"2024-04-25T00:01:46.647716+0000","last_fullsized":"2024-04-25T00:01:46.647716+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:56.741250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657126+0000","last_change":"2024-04-25T00:01:19.515376+0000","last_active":"2024-04-25T00:01:46.657126+0000","last_peered":"2024-04-25T00:01:46.657126+0000","last_clean":"2024-04-25T00:01:46.657126+0000","last_became_active":"2024-04-25T00:01:19.512425+0000","last_became_peered":"2024-04-25T00:01:19.512425+0000","last_unstale":"2024-04-25T00:01:46.657126+0000","last_undegraded":"2024-04-25T00:01:46.657126+0000","last_fullsized":"2024-04-25T00:01:46.657126+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:43:16.957273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.654022+0000","last_change":"2024-04-25T00:01:05.361723+0000","last_active":"2024-04-25T00:01:46.654022+0000","last_peered":"2024-04-25T00:01:46.654022+0000","last_clean":"2024-04-25T00:01:46.654022+0000","last_became_active":"2024-04-25T00:01:05.360956+0000","last_became_peered":"2024-04-25T00:01:05.360956+0000","last_unstale":"2024-04-25T00:01:46.654022+0000","last_undegraded":"2024-04-25T00:01:46.654022+0000","last_fullsized":"2024-04-25T00:01:46.654022+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:23.870121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660056+0000","last_change":"2024-04-25T00:01:04.362129+0000","last_active":"2024-04-25T00:01:46.660056+0000","last_peered":"2024-04-25T00:01:46.660056+0000","last_clean":"2024-04-25T00:01:46.660056+0000","last_became_active":"2024-04-25T00:01:04.361827+0000","last_became_peered":"2024-04-25T00:01:04.361827+0000","last_unstale":"2024-04-25T00:01:46.660056+0000","last_undegraded":"2024-04-25T00:01:46.660056+0000","last_fullsized":"2024-04-25T00:01:46.660056+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:26.243635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"26'8","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659186+0000","last_change":"2024-04-25T00:01:18.496129+0000","last_active":"2024-04-25T00:01:46.659186+0000","last_peered":"2024-04-25T00:01:46.659186+0000","last_clean":"2024-04-25T00:01:46.659186+0000","last_became_active":"2024-04-25T00:01:18.495415+0000","last_became_peered":"2024-04-25T00:01:18.495415+0000","last_unstale":"2024-04-25T00:01:46.659186+0000","last_undegraded":"2024-04-25T00:01:46.659186+0000","last_fullsized":"2024-04-25T00:01:46.659186+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:38.089411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.646840+0000","last_change":"2024-04-25T00:01:19.509542+0000","last_active":"2024-04-25T00:01:46.646840+0000","last_peered":"2024-04-25T00:01:46.646840+0000","last_clean":"2024-04-25T00:01:46.646840+0000","last_became_active":"2024-04-25T00:01:19.509015+0000","last_became_peered":"2024-04-25T00:01:19.509015+0000","last_unstale":"2024-04-25T00:01:46.646840+0000","last_undegraded":"2024-04-25T00:01:46.646840+0000","last_fullsized":"2024-04-25T00:01:46.646840+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:25:20.185355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.648263+0000","last_change":"2024-04-25T00:01:05.365413+0000","last_active":"2024-04-25T00:01:46.648263+0000","last_peered":"2024-04-25T00:01:46.648263+0000","last_clean":"2024-04-25T00:01:46.648263+0000","last_became_active":"2024-04-25T00:01:05.361865+0000","last_became_peered":"2024-04-25T00:01:05.361865+0000","last_unstale":"2024-04-25T00:01:46.648263+0000","last_undegraded":"2024-04-25T00:01:46.648263+0000","last_fullsized":"2024-04-25T00:01:46.648263+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:30.228060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.658791+0000","last_change":"2024-04-25T00:01:04.360804+0000","last_active":"2024-04-25T00:01:46.658791+0000","last_peered":"2024-04-25T00:01:46.658791+0000","last_clean":"2024-04-25T00:01:46.658791+0000","last_became_active":"2024-04-25T00:01:04.360374+0000","last_became_peered":"2024-04-25T00:01:04.360374+0000","last_unstale":"2024-04-25T00:01:46.658791+0000","last_undegraded":"2024-04-25T00:01:46.658791+0000","last_fullsized":"2024-04-25T00:01:46.658791+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511980+0000","last_change":"2024-04-25T00:01:18.480003+0000","last_active":"2024-04-25T00:01:20.511980+0000","last_peered":"2024-04-25T00:01:20.511980+0000","last_clean":"2024-04-25T00:01:20.511980+0000","last_became_active":"2024-04-25T00:01:18.479270+0000","last_became_peered":"2024-04-25T00:01:18.479270+0000","last_unstale":"2024-04-25T00:01:20.511980+0000","last_undegraded":"2024-04-25T00:01:20.511980+0000","last_fullsized":"2024-04-25T00:01:20.511980+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:52:40.942502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.648354+0000","last_change":"2024-04-25T00:01:19.524463+0000","last_active":"2024-04-25T00:01:46.648354+0000","last_peered":"2024-04-25T00:01:46.648354+0000","last_clean":"2024-04-25T00:01:46.648354+0000","last_became_active":"2024-04-25T00:01:19.524166+0000","last_became_peered":"2024-04-25T00:01:19.524166+0000","last_unstale":"2024-04-25T00:01:46.648354+0000","last_undegraded":"2024-04-25T00:01:46.648354+0000","last_fullsized":"2024-04-25T00:01:46.648354+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:50.351599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"26'1","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.514720+0000","last_change":"2024-04-25T00:01:18.497096+0000","last_active":"2024-04-25T00:01:20.514720+0000","last_peered":"2024-04-25T00:01:20.514720+0000","last_clean":"2024-04-25T00:01:20.514720+0000","last_became_active":"2024-04-25T00:01:18.496289+0000","last_became_peered":"2024-04-25T00:01:18.496289+0000","last_unstale":"2024-04-25T00:01:20.514720+0000","last_undegraded":"2024-04-25T00:01:20.514720+0000","last_fullsized":"2024-04-25T00:01:20.514720+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:59:40.316831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.512256+0000","last_change":"2024-04-25T00:01:05.370043+0000","last_active":"2024-04-25T00:01:20.512256+0000","last_peered":"2024-04-25T00:01:20.512256+0000","last_clean":"2024-04-25T00:01:20.512256+0000","last_became_active":"2024-04-25T00:01:05.368080+0000","last_became_peered":"2024-04-25T00:01:05.368080+0000","last_unstale":"2024-04-25T00:01:20.512256+0000","last_undegraded":"2024-04-25T00:01:20.512256+0000","last_fullsized":"2024-04-25T00:01:20.512256+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:58:07.610923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657790+0000","last_change":"2024-04-25T00:01:04.364523+0000","last_active":"2024-04-25T00:01:46.657790+0000","last_peered":"2024-04-25T00:01:46.657790+0000","last_clean":"2024-04-25T00:01:46.657790+0000","last_became_active":"2024-04-25T00:01:04.364050+0000","last_became_peered":"2024-04-25T00:01:04.364050+0000","last_unstale":"2024-04-25T00:01:46.657790+0000","last_undegraded":"2024-04-25T00:01:46.657790+0000","last_fullsized":"2024-04-25T00:01:46.657790+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:49.479804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.664982+0000","last_change":"2024-04-25T00:01:19.491523+0000","last_active":"2024-04-25T00:01:46.664982+0000","last_peered":"2024-04-25T00:01:46.664982+0000","last_clean":"2024-04-25T00:01:46.664982+0000","last_became_active":"2024-04-25T00:01:19.490755+0000","last_became_peered":"2024-04-25T00:01:19.490755+0000","last_unstale":"2024-04-25T00:01:46.664982+0000","last_undegraded":"2024-04-25T00:01:46.664982+0000","last_fullsized":"2024-04-25T00:01:46.664982+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:47:19.526845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"26'4","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659642+0000","last_change":"2024-04-25T00:01:18.472794+0000","last_active":"2024-04-25T00:01:46.659642+0000","last_peered":"2024-04-25T00:01:46.659642+0000","last_clean":"2024-04-25T00:01:46.659642+0000","last_became_active":"2024-04-25T00:01:18.472534+0000","last_became_peered":"2024-04-25T00:01:18.472534+0000","last_unstale":"2024-04-25T00:01:46.659642+0000","last_undegraded":"2024-04-25T00:01:46.659642+0000","last_fullsized":"2024-04-25T00:01:46.659642+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:55:45.843808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.649586+0000","last_change":"2024-04-25T00:01:05.378867+0000","last_active":"2024-04-25T00:01:46.649586+0000","last_peered":"2024-04-25T00:01:46.649586+0000","last_clean":"2024-04-25T00:01:46.649586+0000","last_became_active":"2024-04-25T00:01:05.378056+0000","last_became_peered":"2024-04-25T00:01:05.378056+0000","last_unstale":"2024-04-25T00:01:46.649586+0000","last_undegraded":"2024-04-25T00:01:46.649586+0000","last_fullsized":"2024-04-25T00:01:46.649586+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:36:10.377757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.662462+0000","last_change":"2024-04-25T00:01:04.370896+0000","last_active":"2024-04-25T00:01:46.662462+0000","last_peered":"2024-04-25T00:01:46.662462+0000","last_clean":"2024-04-25T00:01:46.662462+0000","last_became_active":"2024-04-25T00:01:04.370273+0000","last_became_peered":"2024-04-25T00:01:04.370273+0000","last_unstale":"2024-04-25T00:01:46.662462+0000","last_undegraded":"2024-04-25T00:01:46.662462+0000","last_fullsized":"2024-04-25T00:01:46.662462+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:49:22.673421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.514211+0000","last_change":"2024-04-25T00:01:19.521143+0000","last_active":"2024-04-25T00:01:20.514211+0000","last_peered":"2024-04-25T00:01:20.514211+0000","last_clean":"2024-04-25T00:01:20.514211+0000","last_became_active":"2024-04-25T00:01:19.514664+0000","last_became_peered":"2024-04-25T00:01:19.514664+0000","last_unstale":"2024-04-25T00:01:20.514211+0000","last_undegraded":"2024-04-25T00:01:20.514211+0000","last_fullsized":"2024-04-25T00:01:20.514211+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:03:47.440442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"22'1","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.658205+0000","last_change":"2024-04-25T00:01:04.360254+0000","last_active":"2024-04-25T00:01:46.658205+0000","last_peered":"2024-04-25T00:01:46.658205+0000","last_clean":"2024-04-25T00:01:46.658205+0000","last_became_active":"2024-04-25T00:01:04.359586+0000","last_became_peered":"2024-04-25T00:01:04.359586+0000","last_unstale":"2024-04-25T00:01:46.658205+0000","last_undegraded":"2024-04-25T00:01:46.658205+0000","last_fullsized":"2024-04-25T00:01:46.658205+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:47.192930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.513293+0000","last_change":"2024-04-25T00:01:05.370723+0000","last_active":"2024-04-25T00:01:20.513293+0000","last_peered":"2024-04-25T00:01:20.513293+0000","last_clean":"2024-04-25T00:01:20.513293+0000","last_became_active":"2024-04-25T00:01:05.368970+0000","last_became_peered":"2024-04-25T00:01:05.368970+0000","last_unstale":"2024-04-25T00:01:20.513293+0000","last_undegraded":"2024-04-25T00:01:20.513293+0000","last_fullsized":"2024-04-25T00:01:20.513293+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:06:53.743074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"26'2","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.513582+0000","last_change":"2024-04-25T00:01:18.498570+0000","last_active":"2024-04-25T00:01:20.513582+0000","last_peered":"2024-04-25T00:01:20.513582+0000","last_clean":"2024-04-25T00:01:20.513582+0000","last_became_active":"2024-04-25T00:01:18.497745+0000","last_became_peered":"2024-04-25T00:01:18.497745+0000","last_unstale":"2024-04-25T00:01:20.513582+0000","last_undegraded":"2024-04-25T00:01:20.513582+0000","last_fullsized":"2024-04-25T00:01:20.513582+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:26:42.047149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.647984+0000","last_change":"2024-04-25T00:01:19.514919+0000","last_active":"2024-04-25T00:01:46.647984+0000","last_peered":"2024-04-25T00:01:46.647984+0000","last_clean":"2024-04-25T00:01:46.647984+0000","last_became_active":"2024-04-25T00:01:19.514280+0000","last_became_peered":"2024-04-25T00:01:19.514280+0000","last_unstale":"2024-04-25T00:01:46.647984+0000","last_undegraded":"2024-04-25T00:01:46.647984+0000","last_fullsized":"2024-04-25T00:01:46.647984+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:37.975265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"22'1","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.653328+0000","last_change":"2024-04-25T00:01:04.365607+0000","last_active":"2024-04-25T00:01:46.653328+0000","last_peered":"2024-04-25T00:01:46.653328+0000","last_clean":"2024-04-25T00:01:46.653328+0000","last_became_active":"2024-04-25T00:01:04.364788+0000","last_became_peered":"2024-04-25T00:01:04.364788+0000","last_unstale":"2024-04-25T00:01:46.653328+0000","last_undegraded":"2024-04-25T00:01:46.653328+0000","last_fullsized":"2024-04-25T00:01:46.653328+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.661022+0000","last_change":"2024-04-25T00:01:05.375142+0000","last_active":"2024-04-25T00:01:46.661022+0000","last_peered":"2024-04-25T00:01:46.661022+0000","last_clean":"2024-04-25T00:01:46.661022+0000","last_became_active":"2024-04-25T00:01:05.373857+0000","last_became_peered":"2024-04-25T00:01:05.373857+0000","last_unstale":"2024-04-25T00:01:46.661022+0000","last_undegraded":"2024-04-25T00:01:46.661022+0000","last_fullsized":"2024-04-25T00:01:46.661022+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:15.879673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"26'2","reported_seq":20,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.664841+0000","last_change":"2024-04-25T00:01:18.502786+0000","last_active":"2024-04-25T00:01:46.664841+0000","last_peered":"2024-04-25T00:01:46.664841+0000","last_clean":"2024-04-25T00:01:46.664841+0000","last_became_active":"2024-04-25T00:01:18.501785+0000","last_became_peered":"2024-04-25T00:01:18.501785+0000","last_unstale":"2024-04-25T00:01:46.664841+0000","last_undegraded":"2024-04-25T00:01:46.664841+0000","last_fullsized":"2024-04-25T00:01:46.664841+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:09:31.441044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.514560+0000","last_change":"2024-04-25T00:01:19.511239+0000","last_active":"2024-04-25T00:01:20.514560+0000","last_peered":"2024-04-25T00:01:20.514560+0000","last_clean":"2024-04-25T00:01:20.514560+0000","last_became_active":"2024-04-25T00:01:19.510155+0000","last_became_peered":"2024-04-25T00:01:19.510155+0000","last_unstale":"2024-04-25T00:01:20.514560+0000","last_undegraded":"2024-04-25T00:01:20.514560+0000","last_fullsized":"2024-04-25T00:01:20.514560+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:43:14.452663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.647454+0000","last_change":"2024-04-25T00:01:05.378517+0000","last_active":"2024-04-25T00:01:46.647454+0000","last_peered":"2024-04-25T00:01:46.647454+0000","last_clean":"2024-04-25T00:01:46.647454+0000","last_became_active":"2024-04-25T00:01:05.377742+0000","last_became_peered":"2024-04-25T00:01:05.377742+0000","last_unstale":"2024-04-25T00:01:46.647454+0000","last_undegraded":"2024-04-25T00:01:46.647454+0000","last_fullsized":"2024-04-25T00:01:46.647454+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:06.450369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.514828+0000","last_change":"2024-04-25T00:01:19.508999+0000","last_active":"2024-04-25T00:01:20.514828+0000","last_peered":"2024-04-25T00:01:20.514828+0000","last_clean":"2024-04-25T00:01:20.514828+0000","last_became_active":"2024-04-25T00:01:19.507960+0000","last_became_peered":"2024-04-25T00:01:19.507960+0000","last_unstale":"2024-04-25T00:01:20.514828+0000","last_undegraded":"2024-04-25T00:01:20.514828+0000","last_fullsized":"2024-04-25T00:01:20.514828+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:32:40.274908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659704+0000","last_change":"2024-04-25T00:01:05.372927+0000","last_active":"2024-04-25T00:01:46.659704+0000","last_peered":"2024-04-25T00:01:46.659704+0000","last_clean":"2024-04-25T00:01:46.659704+0000","last_became_active":"2024-04-25T00:01:05.372050+0000","last_became_peered":"2024-04-25T00:01:05.372050+0000","last_unstale":"2024-04-25T00:01:46.659704+0000","last_undegraded":"2024-04-25T00:01:46.659704+0000","last_fullsized":"2024-04-25T00:01:46.659704+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:48.202251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.647316+0000","last_change":"2024-04-25T00:01:19.531072+0000","last_active":"2024-04-25T00:01:46.647316+0000","last_peered":"2024-04-25T00:01:46.647316+0000","last_clean":"2024-04-25T00:01:46.647316+0000","last_became_active":"2024-04-25T00:01:19.530836+0000","last_became_peered":"2024-04-25T00:01:19.530836+0000","last_unstale":"2024-04-25T00:01:46.647316+0000","last_undegraded":"2024-04-25T00:01:46.647316+0000","last_fullsized":"2024-04-25T00:01:46.647316+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:08.508741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657903+0000","last_change":"2024-04-25T00:01:05.362960+0000","last_active":"2024-04-25T00:01:46.657903+0000","last_peered":"2024-04-25T00:01:46.657903+0000","last_clean":"2024-04-25T00:01:46.657903+0000","last_became_active":"2024-04-25T00:01:05.361067+0000","last_became_peered":"2024-04-25T00:01:05.361067+0000","last_unstale":"2024-04-25T00:01:46.657903+0000","last_undegraded":"2024-04-25T00:01:46.657903+0000","last_fullsized":"2024-04-25T00:01:46.657903+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:24.017263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.650119+0000","last_change":"2024-04-25T00:01:19.506029+0000","last_active":"2024-04-25T00:01:46.650119+0000","last_peered":"2024-04-25T00:01:46.650119+0000","last_clean":"2024-04-25T00:01:46.650119+0000","last_became_active":"2024-04-25T00:01:19.505184+0000","last_became_peered":"2024-04-25T00:01:19.505184+0000","last_unstale":"2024-04-25T00:01:46.650119+0000","last_undegraded":"2024-04-25T00:01:46.650119+0000","last_fullsized":"2024-04-25T00:01:46.650119+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:08.508741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.653074+0000","last_change":"2024-04-25T00:01:05.358690+0000","last_active":"2024-04-25T00:01:46.653074+0000","last_peered":"2024-04-25T00:01:46.653074+0000","last_clean":"2024-04-25T00:01:46.653074+0000","last_became_active":"2024-04-25T00:01:05.357340+0000","last_became_peered":"2024-04-25T00:01:05.357340+0000","last_unstale":"2024-04-25T00:01:46.653074+0000","last_undegraded":"2024-04-25T00:01:46.653074+0000","last_fullsized":"2024-04-25T00:01:46.653074+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:36:55.436661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.658667+0000","last_change":"2024-04-25T00:01:19.508691+0000","last_active":"2024-04-25T00:01:46.658667+0000","last_peered":"2024-04-25T00:01:46.658667+0000","last_clean":"2024-04-25T00:01:46.658667+0000","last_became_active":"2024-04-25T00:01:19.508123+0000","last_became_peered":"2024-04-25T00:01:19.508123+0000","last_unstale":"2024-04-25T00:01:46.658667+0000","last_undegraded":"2024-04-25T00:01:46.658667+0000","last_fullsized":"2024-04-25T00:01:46.658667+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:47:19.526845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.665745+0000","last_change":"2024-04-25T00:01:05.387057+0000","last_active":"2024-04-25T00:01:46.665745+0000","last_peered":"2024-04-25T00:01:46.665745+0000","last_clean":"2024-04-25T00:01:46.665745+0000","last_became_active":"2024-04-25T00:01:05.386724+0000","last_became_peered":"2024-04-25T00:01:05.386724+0000","last_unstale":"2024-04-25T00:01:46.665745+0000","last_undegraded":"2024-04-25T00:01:46.665745+0000","last_fullsized":"2024-04-25T00:01:46.665745+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:55:09.073286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659072+0000","last_change":"2024-04-25T00:01:19.522760+0000","last_active":"2024-04-25T00:01:46.659072+0000","last_peered":"2024-04-25T00:01:46.659072+0000","last_clean":"2024-04-25T00:01:46.659072+0000","last_became_active":"2024-04-25T00:01:19.522380+0000","last_became_peered":"2024-04-25T00:01:19.522380+0000","last_unstale":"2024-04-25T00:01:46.659072+0000","last_undegraded":"2024-04-25T00:01:46.659072+0000","last_fullsized":"2024-04-25T00:01:46.659072+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:55:46.853565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.665267+0000","last_change":"2024-04-25T00:01:05.367296+0000","last_active":"2024-04-25T00:01:46.665267+0000","last_peered":"2024-04-25T00:01:46.665267+0000","last_clean":"2024-04-25T00:01:46.665267+0000","last_became_active":"2024-04-25T00:01:05.363838+0000","last_became_peered":"2024-04-25T00:01:05.363838+0000","last_unstale":"2024-04-25T00:01:46.665267+0000","last_undegraded":"2024-04-25T00:01:46.665267+0000","last_fullsized":"2024-04-25T00:01:46.665267+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:43:07.053074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657236+0000","last_change":"2024-04-25T00:01:19.510902+0000","last_active":"2024-04-25T00:01:46.657236+0000","last_peered":"2024-04-25T00:01:46.657236+0000","last_clean":"2024-04-25T00:01:46.657236+0000","last_became_active":"2024-04-25T00:01:19.510008+0000","last_became_peered":"2024-04-25T00:01:19.510008+0000","last_unstale":"2024-04-25T00:01:46.657236+0000","last_undegraded":"2024-04-25T00:01:46.657236+0000","last_fullsized":"2024-04-25T00:01:46.657236+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:40:32.292140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.648277+0000","last_change":"2024-04-25T00:01:05.365761+0000","last_active":"2024-04-25T00:01:46.648277+0000","last_peered":"2024-04-25T00:01:46.648277+0000","last_clean":"2024-04-25T00:01:46.648277+0000","last_became_active":"2024-04-25T00:01:05.361119+0000","last_became_peered":"2024-04-25T00:01:05.361119+0000","last_unstale":"2024-04-25T00:01:46.648277+0000","last_undegraded":"2024-04-25T00:01:46.648277+0000","last_fullsized":"2024-04-25T00:01:46.648277+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.647042+0000","last_change":"2024-04-25T00:01:19.518534+0000","last_active":"2024-04-25T00:01:46.647042+0000","last_peered":"2024-04-25T00:01:46.647042+0000","last_clean":"2024-04-25T00:01:46.647042+0000","last_became_active":"2024-04-25T00:01:19.518110+0000","last_became_peered":"2024-04-25T00:01:19.518110+0000","last_unstale":"2024-04-25T00:01:46.647042+0000","last_undegraded":"2024-04-25T00:01:46.647042+0000","last_fullsized":"2024-04-25T00:01:46.647042+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:05:09.219823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.653554+0000","last_change":"2024-04-25T00:01:05.373464+0000","last_active":"2024-04-25T00:01:46.653554+0000","last_peered":"2024-04-25T00:01:46.653554+0000","last_clean":"2024-04-25T00:01:46.653554+0000","last_became_active":"2024-04-25T00:01:05.372611+0000","last_became_peered":"2024-04-25T00:01:05.372611+0000","last_unstale":"2024-04-25T00:01:46.653554+0000","last_undegraded":"2024-04-25T00:01:46.653554+0000","last_fullsized":"2024-04-25T00:01:46.653554+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:31.456792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.664300+0000","last_change":"2024-04-25T00:01:19.542475+0000","last_active":"2024-04-25T00:01:46.664300+0000","last_peered":"2024-04-25T00:01:46.664300+0000","last_clean":"2024-04-25T00:01:46.664300+0000","last_became_active":"2024-04-25T00:01:19.541877+0000","last_became_peered":"2024-04-25T00:01:19.541877+0000","last_unstale":"2024-04-25T00:01:46.664300+0000","last_undegraded":"2024-04-25T00:01:46.664300+0000","last_fullsized":"2024-04-25T00:01:46.664300+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:02:57.373014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657634+0000","last_change":"2024-04-25T00:01:05.372906+0000","last_active":"2024-04-25T00:01:46.657634+0000","last_peered":"2024-04-25T00:01:46.657634+0000","last_clean":"2024-04-25T00:01:46.657634+0000","last_became_active":"2024-04-25T00:01:05.372148+0000","last_became_peered":"2024-04-25T00:01:05.372148+0000","last_unstale":"2024-04-25T00:01:46.657634+0000","last_undegraded":"2024-04-25T00:01:46.657634+0000","last_fullsized":"2024-04-25T00:01:46.657634+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:43.615895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659888+0000","last_change":"2024-04-25T00:01:19.498515+0000","last_active":"2024-04-25T00:01:46.659888+0000","last_peered":"2024-04-25T00:01:46.659888+0000","last_clean":"2024-04-25T00:01:46.659888+0000","last_became_active":"2024-04-25T00:01:19.496310+0000","last_became_peered":"2024-04-25T00:01:19.496310+0000","last_unstale":"2024-04-25T00:01:46.659888+0000","last_undegraded":"2024-04-25T00:01:46.659888+0000","last_fullsized":"2024-04-25T00:01:46.659888+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:20:06.661558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.651069+0000","last_change":"2024-04-25T00:01:05.366340+0000","last_active":"2024-04-25T00:01:46.651069+0000","last_peered":"2024-04-25T00:01:46.651069+0000","last_clean":"2024-04-25T00:01:46.651069+0000","last_became_active":"2024-04-25T00:01:05.361952+0000","last_became_peered":"2024-04-25T00:01:05.361952+0000","last_unstale":"2024-04-25T00:01:46.651069+0000","last_undegraded":"2024-04-25T00:01:46.651069+0000","last_fullsized":"2024-04-25T00:01:46.651069+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:41.948662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.661872+0000","last_change":"2024-04-25T00:01:19.508257+0000","last_active":"2024-04-25T00:01:46.661872+0000","last_peered":"2024-04-25T00:01:46.661872+0000","last_clean":"2024-04-25T00:01:46.661872+0000","last_became_active":"2024-04-25T00:01:19.507516+0000","last_became_peered":"2024-04-25T00:01:19.507516+0000","last_unstale":"2024-04-25T00:01:46.661872+0000","last_undegraded":"2024-04-25T00:01:46.661872+0000","last_fullsized":"2024-04-25T00:01:46.661872+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:45.121513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660969+0000","last_change":"2024-04-25T00:01:05.387457+0000","last_active":"2024-04-25T00:01:46.660969+0000","last_peered":"2024-04-25T00:01:46.660969+0000","last_clean":"2024-04-25T00:01:46.660969+0000","last_became_active":"2024-04-25T00:01:05.386919+0000","last_became_peered":"2024-04-25T00:01:05.386919+0000","last_unstale":"2024-04-25T00:01:46.660969+0000","last_undegraded":"2024-04-25T00:01:46.660969+0000","last_fullsized":"2024-04-25T00:01:46.660969+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.664224+0000","last_change":"2024-04-25T00:01:19.520648+0000","last_active":"2024-04-25T00:01:46.664224+0000","last_peered":"2024-04-25T00:01:46.664224+0000","last_clean":"2024-04-25T00:01:46.664224+0000","last_became_active":"2024-04-25T00:01:19.519303+0000","last_became_peered":"2024-04-25T00:01:19.519303+0000","last_unstale":"2024-04-25T00:01:46.664224+0000","last_undegraded":"2024-04-25T00:01:46.664224+0000","last_fullsized":"2024-04-25T00:01:46.664224+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:23.993729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.513073+0000","last_change":"2024-04-25T00:01:05.366249+0000","last_active":"2024-04-25T00:01:20.513073+0000","last_peered":"2024-04-25T00:01:20.513073+0000","last_clean":"2024-04-25T00:01:20.513073+0000","last_became_active":"2024-04-25T00:01:05.365548+0000","last_became_peered":"2024-04-25T00:01:05.365548+0000","last_unstale":"2024-04-25T00:01:20.513073+0000","last_undegraded":"2024-04-25T00:01:20.513073+0000","last_fullsized":"2024-04-25T00:01:20.513073+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:22:21.143639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.647068+0000","last_change":"2024-04-25T00:01:19.514659+0000","last_active":"2024-04-25T00:01:46.647068+0000","last_peered":"2024-04-25T00:01:46.647068+0000","last_clean":"2024-04-25T00:01:46.647068+0000","last_became_active":"2024-04-25T00:01:19.514232+0000","last_became_peered":"2024-04-25T00:01:19.514232+0000","last_unstale":"2024-04-25T00:01:46.647068+0000","last_undegraded":"2024-04-25T00:01:46.647068+0000","last_fullsized":"2024-04-25T00:01:46.647068+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:45.121513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":10,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.668794+0000","last_change":"2024-04-25T00:01:46.665943+0000","last_active":"2024-04-25T00:01:46.668794+0000","last_peered":"2024-04-25T00:01:46.668794+0000","last_clean":"2024-04-25T00:01:46.668794+0000","last_became_active":"2024-04-25T00:01:46.663874+0000","last_became_peered":"2024-04-25T00:01:46.663874+0000","last_unstale":"2024-04-25T00:01:46.668794+0000","last_undegraded":"2024-04-25T00:01:46.668794+0000","last_fullsized":"2024-04-25T00:01:46.668794+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:53:00.633407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659279+0000","last_change":"2024-04-25T00:01:19.503419+0000","last_active":"2024-04-25T00:01:46.659279+0000","last_peered":"2024-04-25T00:01:46.659279+0000","last_clean":"2024-04-25T00:01:46.659279+0000","last_became_active":"2024-04-25T00:01:19.500568+0000","last_became_peered":"2024-04-25T00:01:19.500568+0000","last_unstale":"2024-04-25T00:01:46.659279+0000","last_undegraded":"2024-04-25T00:01:46.659279+0000","last_fullsized":"2024-04-25T00:01:46.659279+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:08.508741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659631+0000","last_change":"2024-04-25T00:01:05.370653+0000","last_active":"2024-04-25T00:01:46.659631+0000","last_peered":"2024-04-25T00:01:46.659631+0000","last_clean":"2024-04-25T00:01:46.659631+0000","last_became_active":"2024-04-25T00:01:05.369919+0000","last_became_peered":"2024-04-25T00:01:05.369919+0000","last_unstale":"2024-04-25T00:01:46.659631+0000","last_undegraded":"2024-04-25T00:01:46.659631+0000","last_fullsized":"2024-04-25T00:01:46.659631+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:30:08.311326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.646840+0000","last_change":"2024-04-25T00:01:19.540104+0000","last_active":"2024-04-25T00:01:46.646840+0000","last_peered":"2024-04-25T00:01:46.646840+0000","last_clean":"2024-04-25T00:01:46.646840+0000","last_became_active":"2024-04-25T00:01:19.539775+0000","last_became_peered":"2024-04-25T00:01:19.539775+0000","last_unstale":"2024-04-25T00:01:46.646840+0000","last_undegraded":"2024-04-25T00:01:46.646840+0000","last_fullsized":"2024-04-25T00:01:46.646840+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:53:14.768520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.664786+0000","last_change":"2024-04-25T00:01:05.367001+0000","last_active":"2024-04-25T00:01:46.664786+0000","last_peered":"2024-04-25T00:01:46.664786+0000","last_clean":"2024-04-25T00:01:46.664786+0000","last_became_active":"2024-04-25T00:01:05.366264+0000","last_became_peered":"2024-04-25T00:01:05.366264+0000","last_unstale":"2024-04-25T00:01:46.664786+0000","last_undegraded":"2024-04-25T00:01:46.664786+0000","last_fullsized":"2024-04-25T00:01:46.664786+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.661915+0000","last_change":"2024-04-25T00:01:19.497247+0000","last_active":"2024-04-25T00:01:46.661915+0000","last_peered":"2024-04-25T00:01:46.661915+0000","last_clean":"2024-04-25T00:01:46.661915+0000","last_became_active":"2024-04-25T00:01:19.496517+0000","last_became_peered":"2024-04-25T00:01:19.496517+0000","last_unstale":"2024-04-25T00:01:46.661915+0000","last_undegraded":"2024-04-25T00:01:46.661915+0000","last_fullsized":"2024-04-25T00:01:46.661915+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:30.129056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659982+0000","last_change":"2024-04-25T00:01:05.379958+0000","last_active":"2024-04-25T00:01:46.659982+0000","last_peered":"2024-04-25T00:01:46.659982+0000","last_clean":"2024-04-25T00:01:46.659982+0000","last_became_active":"2024-04-25T00:01:05.379090+0000","last_became_peered":"2024-04-25T00:01:05.379090+0000","last_unstale":"2024-04-25T00:01:46.659982+0000","last_undegraded":"2024-04-25T00:01:46.659982+0000","last_fullsized":"2024-04-25T00:01:46.659982+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:59:50.832958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.515390+0000","last_change":"2024-04-25T00:01:19.530176+0000","last_active":"2024-04-25T00:01:20.515390+0000","last_peered":"2024-04-25T00:01:20.515390+0000","last_clean":"2024-04-25T00:01:20.515390+0000","last_became_active":"2024-04-25T00:01:19.529410+0000","last_became_peered":"2024-04-25T00:01:19.529410+0000","last_unstale":"2024-04-25T00:01:20.515390+0000","last_undegraded":"2024-04-25T00:01:20.515390+0000","last_fullsized":"2024-04-25T00:01:20.515390+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:40:13.204880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657503+0000","last_change":"2024-04-25T00:01:05.370826+0000","last_active":"2024-04-25T00:01:46.657503+0000","last_peered":"2024-04-25T00:01:46.657503+0000","last_clean":"2024-04-25T00:01:46.657503+0000","last_became_active":"2024-04-25T00:01:05.369848+0000","last_became_peered":"2024-04-25T00:01:05.369848+0000","last_unstale":"2024-04-25T00:01:46.657503+0000","last_undegraded":"2024-04-25T00:01:46.657503+0000","last_fullsized":"2024-04-25T00:01:46.657503+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:23.870121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657618+0000","last_change":"2024-04-25T00:01:19.511625+0000","last_active":"2024-04-25T00:01:46.657618+0000","last_peered":"2024-04-25T00:01:46.657618+0000","last_clean":"2024-04-25T00:01:46.657618+0000","last_became_active":"2024-04-25T00:01:19.511349+0000","last_became_peered":"2024-04-25T00:01:19.511349+0000","last_unstale":"2024-04-25T00:01:46.657618+0000","last_undegraded":"2024-04-25T00:01:46.657618+0000","last_fullsized":"2024-04-25T00:01:46.657618+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:02:57.373014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.650675+0000","last_change":"2024-04-25T00:01:05.390341+0000","last_active":"2024-04-25T00:01:46.650675+0000","last_peered":"2024-04-25T00:01:46.650675+0000","last_clean":"2024-04-25T00:01:46.650675+0000","last_became_active":"2024-04-25T00:01:05.389707+0000","last_became_peered":"2024-04-25T00:01:05.389707+0000","last_unstale":"2024-04-25T00:01:46.650675+0000","last_undegraded":"2024-04-25T00:01:46.650675+0000","last_fullsized":"2024-04-25T00:01:46.650675+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:54:21.138947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657624+0000","last_change":"2024-04-25T00:01:19.513630+0000","last_active":"2024-04-25T00:01:46.657624+0000","last_peered":"2024-04-25T00:01:46.657624+0000","last_clean":"2024-04-25T00:01:46.657624+0000","last_became_active":"2024-04-25T00:01:19.512978+0000","last_became_peered":"2024-04-25T00:01:19.512978+0000","last_unstale":"2024-04-25T00:01:46.657624+0000","last_undegraded":"2024-04-25T00:01:46.657624+0000","last_fullsized":"2024-04-25T00:01:46.657624+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:57.751007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.662982+0000","last_change":"2024-04-25T00:01:05.378373+0000","last_active":"2024-04-25T00:01:46.662982+0000","last_peered":"2024-04-25T00:01:46.662982+0000","last_clean":"2024-04-25T00:01:46.662982+0000","last_became_active":"2024-04-25T00:01:05.376570+0000","last_became_peered":"2024-04-25T00:01:05.376570+0000","last_unstale":"2024-04-25T00:01:46.662982+0000","last_undegraded":"2024-04-25T00:01:46.662982+0000","last_fullsized":"2024-04-25T00:01:46.662982+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:30.228060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659177+0000","last_change":"2024-04-25T00:01:19.516716+0000","last_active":"2024-04-25T00:01:46.659177+0000","last_peered":"2024-04-25T00:01:46.659177+0000","last_clean":"2024-04-25T00:01:46.659177+0000","last_became_active":"2024-04-25T00:01:19.516142+0000","last_became_peered":"2024-04-25T00:01:19.516142+0000","last_unstale":"2024-04-25T00:01:46.659177+0000","last_undegraded":"2024-04-25T00:01:46.659177+0000","last_fullsized":"2024-04-25T00:01:46.659177+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:28:18.234546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.663972+0000","last_change":"2024-04-25T00:01:05.381011+0000","last_active":"2024-04-25T00:01:46.663972+0000","last_peered":"2024-04-25T00:01:46.663972+0000","last_clean":"2024-04-25T00:01:46.663972+0000","last_became_active":"2024-04-25T00:01:05.380236+0000","last_became_peered":"2024-04-25T00:01:05.380236+0000","last_unstale":"2024-04-25T00:01:46.663972+0000","last_undegraded":"2024-04-25T00:01:46.663972+0000","last_fullsized":"2024-04-25T00:01:46.663972+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:06:19.307378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.513107+0000","last_change":"2024-04-25T00:01:19.529023+0000","last_active":"2024-04-25T00:01:20.513107+0000","last_peered":"2024-04-25T00:01:20.513107+0000","last_clean":"2024-04-25T00:01:20.513107+0000","last_became_active":"2024-04-25T00:01:19.524609+0000","last_became_peered":"2024-04-25T00:01:19.524609+0000","last_unstale":"2024-04-25T00:01:20.513107+0000","last_undegraded":"2024-04-25T00:01:20.513107+0000","last_fullsized":"2024-04-25T00:01:20.513107+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:41:19.150476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660549+0000","last_change":"2024-04-25T00:01:05.385853+0000","last_active":"2024-04-25T00:01:46.660549+0000","last_peered":"2024-04-25T00:01:46.660549+0000","last_clean":"2024-04-25T00:01:46.660549+0000","last_became_active":"2024-04-25T00:01:05.384997+0000","last_became_peered":"2024-04-25T00:01:05.384997+0000","last_unstale":"2024-04-25T00:01:46.660549+0000","last_undegraded":"2024-04-25T00:01:46.660549+0000","last_fullsized":"2024-04-25T00:01:46.660549+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:36:55.436661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.662855+0000","last_change":"2024-04-25T00:01:19.516867+0000","last_active":"2024-04-25T00:01:46.662855+0000","last_peered":"2024-04-25T00:01:46.662855+0000","last_clean":"2024-04-25T00:01:46.662855+0000","last_became_active":"2024-04-25T00:01:19.515764+0000","last_became_peered":"2024-04-25T00:01:19.515764+0000","last_unstale":"2024-04-25T00:01:46.662855+0000","last_undegraded":"2024-04-25T00:01:46.662855+0000","last_fullsized":"2024-04-25T00:01:46.662855+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:53:14.768520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660272+0000","last_change":"2024-04-25T00:01:05.389424+0000","last_active":"2024-04-25T00:01:46.660272+0000","last_peered":"2024-04-25T00:01:46.660272+0000","last_clean":"2024-04-25T00:01:46.660272+0000","last_became_active":"2024-04-25T00:01:05.388999+0000","last_became_peered":"2024-04-25T00:01:05.388999+0000","last_unstale":"2024-04-25T00:01:46.660272+0000","last_undegraded":"2024-04-25T00:01:46.660272+0000","last_fullsized":"2024-04-25T00:01:46.660272+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:31.456792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657679+0000","last_change":"2024-04-25T00:01:19.519365+0000","last_active":"2024-04-25T00:01:46.657679+0000","last_peered":"2024-04-25T00:01:46.657679+0000","last_clean":"2024-04-25T00:01:46.657679+0000","last_became_active":"2024-04-25T00:01:19.519085+0000","last_became_peered":"2024-04-25T00:01:19.519085+0000","last_unstale":"2024-04-25T00:01:46.657679+0000","last_undegraded":"2024-04-25T00:01:46.657679+0000","last_fullsized":"2024-04-25T00:01:46.657679+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:05:09.219823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660140+0000","last_change":"2024-04-25T00:01:05.368974+0000","last_active":"2024-04-25T00:01:46.660140+0000","last_peered":"2024-04-25T00:01:46.660140+0000","last_clean":"2024-04-25T00:01:46.660140+0000","last_became_active":"2024-04-25T00:01:05.367893+0000","last_became_peered":"2024-04-25T00:01:05.367893+0000","last_unstale":"2024-04-25T00:01:46.660140+0000","last_undegraded":"2024-04-25T00:01:46.660140+0000","last_fullsized":"2024-04-25T00:01:46.660140+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:30:23.557381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.658127+0000","last_change":"2024-04-25T00:01:19.513514+0000","last_active":"2024-04-25T00:01:46.658127+0000","last_peered":"2024-04-25T00:01:46.658127+0000","last_clean":"2024-04-25T00:01:46.658127+0000","last_became_active":"2024-04-25T00:01:19.513184+0000","last_became_peered":"2024-04-25T00:01:19.513184+0000","last_unstale":"2024-04-25T00:01:46.658127+0000","last_undegraded":"2024-04-25T00:01:46.658127+0000","last_fullsized":"2024-04-25T00:01:46.658127+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:47.859626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657909+0000","last_change":"2024-04-25T00:01:05.362303+0000","last_active":"2024-04-25T00:01:46.657909+0000","last_peered":"2024-04-25T00:01:46.657909+0000","last_clean":"2024-04-25T00:01:46.657909+0000","last_became_active":"2024-04-25T00:01:05.361586+0000","last_became_peered":"2024-04-25T00:01:05.361586+0000","last_unstale":"2024-04-25T00:01:46.657909+0000","last_undegraded":"2024-04-25T00:01:46.657909+0000","last_fullsized":"2024-04-25T00:01:46.657909+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:30.228060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.650059+0000","last_change":"2024-04-25T00:01:19.515101+0000","last_active":"2024-04-25T00:01:46.650059+0000","last_peered":"2024-04-25T00:01:46.650059+0000","last_clean":"2024-04-25T00:01:46.650059+0000","last_became_active":"2024-04-25T00:01:19.513314+0000","last_became_peered":"2024-04-25T00:01:19.513314+0000","last_unstale":"2024-04-25T00:01:46.650059+0000","last_undegraded":"2024-04-25T00:01:46.650059+0000","last_fullsized":"2024-04-25T00:01:46.650059+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:50.351599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.649234+0000","last_change":"2024-04-25T00:01:05.390256+0000","last_active":"2024-04-25T00:01:46.649234+0000","last_peered":"2024-04-25T00:01:46.649234+0000","last_clean":"2024-04-25T00:01:46.649234+0000","last_became_active":"2024-04-25T00:01:05.389724+0000","last_became_peered":"2024-04-25T00:01:05.389724+0000","last_unstale":"2024-04-25T00:01:46.649234+0000","last_undegraded":"2024-04-25T00:01:46.649234+0000","last_fullsized":"2024-04-25T00:01:46.649234+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:09:18.315688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3f","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.648222+0000","last_change":"2024-04-25T00:01:19.511493+0000","last_active":"2024-04-25T00:01:46.648222+0000","last_peered":"2024-04-25T00:01:46.648222+0000","last_clean":"2024-04-25T00:01:46.648222+0000","last_became_active":"2024-04-25T00:01:19.508049+0000","last_became_peered":"2024-04-25T00:01:19.508049+0000","last_unstale":"2024-04-25T00:01:46.648222+0000","last_undegraded":"2024-04-25T00:01:46.648222+0000","last_fullsized":"2024-04-25T00:01:46.648222+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:55:46.853565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.658614+0000","last_change":"2024-04-25T00:01:05.374766+0000","last_active":"2024-04-25T00:01:46.658614+0000","last_peered":"2024-04-25T00:01:46.658614+0000","last_clean":"2024-04-25T00:01:46.658614+0000","last_became_active":"2024-04-25T00:01:05.369739+0000","last_became_peered":"2024-04-25T00:01:05.369739+0000","last_unstale":"2024-04-25T00:01:46.658614+0000","last_undegraded":"2024-04-25T00:01:46.658614+0000","last_fullsized":"2024-04-25T00:01:46.658614+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:09:18.315688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.513219+0000","last_change":"2024-04-25T00:01:19.521456+0000","last_active":"2024-04-25T00:01:20.513219+0000","last_peered":"2024-04-25T00:01:20.513219+0000","last_clean":"2024-04-25T00:01:20.513219+0000","last_became_active":"2024-04-25T00:01:19.520787+0000","last_became_peered":"2024-04-25T00:01:19.520787+0000","last_unstale":"2024-04-25T00:01:20.513219+0000","last_undegraded":"2024-04-25T00:01:20.513219+0000","last_fullsized":"2024-04-25T00:01:20.513219+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:25:38.981364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.647277+0000","last_change":"2024-04-25T00:01:05.359253+0000","last_active":"2024-04-25T00:01:46.647277+0000","last_peered":"2024-04-25T00:01:46.647277+0000","last_clean":"2024-04-25T00:01:46.647277+0000","last_became_active":"2024-04-25T00:01:05.358477+0000","last_became_peered":"2024-04-25T00:01:05.358477+0000","last_unstale":"2024-04-25T00:01:46.647277+0000","last_undegraded":"2024-04-25T00:01:46.647277+0000","last_fullsized":"2024-04-25T00:01:46.647277+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:22.739214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.656961+0000","last_change":"2024-04-25T00:01:19.509505+0000","last_active":"2024-04-25T00:01:46.656961+0000","last_peered":"2024-04-25T00:01:46.656961+0000","last_clean":"2024-04-25T00:01:46.656961+0000","last_became_active":"2024-04-25T00:01:19.507812+0000","last_became_peered":"2024-04-25T00:01:19.507812+0000","last_unstale":"2024-04-25T00:01:46.656961+0000","last_undegraded":"2024-04-25T00:01:46.656961+0000","last_fullsized":"2024-04-25T00:01:46.656961+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:23.993729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659707+0000","last_change":"2024-04-25T00:01:05.386068+0000","last_active":"2024-04-25T00:01:46.659707+0000","last_peered":"2024-04-25T00:01:46.659707+0000","last_clean":"2024-04-25T00:01:46.659707+0000","last_became_active":"2024-04-25T00:01:05.378650+0000","last_became_peered":"2024-04-25T00:01:05.378650+0000","last_unstale":"2024-04-25T00:01:46.659707+0000","last_undegraded":"2024-04-25T00:01:46.659707+0000","last_fullsized":"2024-04-25T00:01:46.659707+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:12:52.846250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660257+0000","last_change":"2024-04-25T00:01:19.514843+0000","last_active":"2024-04-25T00:01:46.660257+0000","last_peered":"2024-04-25T00:01:46.660257+0000","last_clean":"2024-04-25T00:01:46.660257+0000","last_became_active":"2024-04-25T00:01:19.513956+0000","last_became_peered":"2024-04-25T00:01:19.513956+0000","last_unstale":"2024-04-25T00:01:46.660257+0000","last_undegraded":"2024-04-25T00:01:46.660257+0000","last_fullsized":"2024-04-25T00:01:46.660257+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:38.005233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.648398+0000","last_change":"2024-04-25T00:01:05.359716+0000","last_active":"2024-04-25T00:01:46.648398+0000","last_peered":"2024-04-25T00:01:46.648398+0000","last_clean":"2024-04-25T00:01:46.648398+0000","last_became_active":"2024-04-25T00:01:05.358866+0000","last_became_peered":"2024-04-25T00:01:05.358866+0000","last_unstale":"2024-04-25T00:01:46.648398+0000","last_undegraded":"2024-04-25T00:01:46.648398+0000","last_fullsized":"2024-04-25T00:01:46.648398+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:50.489125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.649702+0000","last_change":"2024-04-25T00:01:19.499574+0000","last_active":"2024-04-25T00:01:46.649702+0000","last_peered":"2024-04-25T00:01:46.649702+0000","last_clean":"2024-04-25T00:01:46.649702+0000","last_became_active":"2024-04-25T00:01:19.498699+0000","last_became_peered":"2024-04-25T00:01:19.498699+0000","last_unstale":"2024-04-25T00:01:46.649702+0000","last_undegraded":"2024-04-25T00:01:46.649702+0000","last_fullsized":"2024-04-25T00:01:46.649702+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:47.859626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.650324+0000","last_change":"2024-04-25T00:01:05.371291+0000","last_active":"2024-04-25T00:01:46.650324+0000","last_peered":"2024-04-25T00:01:46.650324+0000","last_clean":"2024-04-25T00:01:46.650324+0000","last_became_active":"2024-04-25T00:01:05.370522+0000","last_became_peered":"2024-04-25T00:01:05.370522+0000","last_unstale":"2024-04-25T00:01:46.650324+0000","last_undegraded":"2024-04-25T00:01:46.650324+0000","last_fullsized":"2024-04-25T00:01:46.650324+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:11:27.044893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.647355+0000","last_change":"2024-04-25T00:01:19.523798+0000","last_active":"2024-04-25T00:01:46.647355+0000","last_peered":"2024-04-25T00:01:46.647355+0000","last_clean":"2024-04-25T00:01:46.647355+0000","last_became_active":"2024-04-25T00:01:19.523450+0000","last_became_peered":"2024-04-25T00:01:19.523450+0000","last_unstale":"2024-04-25T00:01:46.647355+0000","last_undegraded":"2024-04-25T00:01:46.647355+0000","last_fullsized":"2024-04-25T00:01:46.647355+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:40:36.350832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.651674+0000","last_change":"2024-04-25T00:01:05.368190+0000","last_active":"2024-04-25T00:01:46.651674+0000","last_peered":"2024-04-25T00:01:46.651674+0000","last_clean":"2024-04-25T00:01:46.651674+0000","last_became_active":"2024-04-25T00:01:05.362606+0000","last_became_peered":"2024-04-25T00:01:05.362606+0000","last_unstale":"2024-04-25T00:01:46.651674+0000","last_undegraded":"2024-04-25T00:01:46.651674+0000","last_fullsized":"2024-04-25T00:01:46.651674+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:12:52.846250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.646901+0000","last_change":"2024-04-25T00:01:19.505737+0000","last_active":"2024-04-25T00:01:46.646901+0000","last_peered":"2024-04-25T00:01:46.646901+0000","last_clean":"2024-04-25T00:01:46.646901+0000","last_became_active":"2024-04-25T00:01:19.504991+0000","last_became_peered":"2024-04-25T00:01:19.504991+0000","last_unstale":"2024-04-25T00:01:46.646901+0000","last_undegraded":"2024-04-25T00:01:46.646901+0000","last_fullsized":"2024-04-25T00:01:46.646901+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:47:19.526845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660392+0000","last_change":"2024-04-25T00:01:05.389596+0000","last_active":"2024-04-25T00:01:46.660392+0000","last_peered":"2024-04-25T00:01:46.660392+0000","last_clean":"2024-04-25T00:01:46.660392+0000","last_became_active":"2024-04-25T00:01:05.389127+0000","last_became_peered":"2024-04-25T00:01:05.389127+0000","last_unstale":"2024-04-25T00:01:46.660392+0000","last_undegraded":"2024-04-25T00:01:46.660392+0000","last_fullsized":"2024-04-25T00:01:46.660392+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:09.858617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2851,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":5702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2851,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":5702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509455,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27336,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344504,"statfs":{"total":96636764160,"available":96608772096,"internally_reserved":0,"allocated":319488,"data_stored":123609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509455,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27320,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344520,"statfs":{"total":96636764160,"available":96608788480,"internally_reserved":0,"allocated":307200,"data_stored":118634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509454,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27900,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343940,"statfs":{"total":96636764160,"available":96608194560,"internally_reserved":0,"allocated":901120,"data_stored":707623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509455,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":327680,"data_stored":124821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509456,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":323584,"data_stored":125550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509455,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27912,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343928,"statfs":{"total":96636764160,"available":96608182272,"internally_reserved":0,"allocated":909312,"data_stored":713971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509455,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27320,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344520,"statfs":{"total":96636764160,"available":96608788480,"internally_reserved":0,"allocated":307200,"data_stored":116727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574863,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27328,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344512,"statfs":{"total":96636764160,"available":96608780288,"internally_reserved":0,"allocated":319488,"data_stored":114370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T00:01:49.858 INFO:tasks.mds_thrash.ceph_manager:clean! 2024-04-25T00:01:49.858 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T00:01:50.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.001+0000 7f4466b05640 1 Processor -- start 2024-04-25T00:01:50.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.005+0000 7f4466b05640 1 -- start start 2024-04-25T00:01:50.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.005+0000 7f4466b05640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f44601562d0 0x7f4460161130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:50.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.005+0000 7f4466b05640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4460073200 con 0x7f4460152e60 2024-04-25T00:01:50.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.005+0000 7f4466b05640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4460158ec0 con 0x7f44601599a0 2024-04-25T00:01:50.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.005+0000 7f4466b05640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f4460156760 con 0x7f44601562d0 2024-04-25T00:01:50.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.005+0000 7f44648a1640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f4460152e60 0x7f44600c36e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:47564/0 (socket says 172.21.15.116:47564) 2024-04-25T00:01:50.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.005+0000 7f44648a1640 1 -- 172.21.15.116:0/4009066115 learned_addr learned my addr 172.21.15.116:0/4009066115 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:50.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.005+0000 7f44650a2640 1 --2- 172.21.15.116:0/4009066115 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f44601562d0 0x7f4460161130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:50.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.005+0000 7f445f7fe640 1 -- 172.21.15.116:0/4009066115 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 961334363 0 0) 0x7f4460073200 con 0x7f4460152e60 2024-04-25T00:01:50.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.005+0000 7f445f7fe640 1 -- 172.21.15.116:0/4009066115 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4450003530 con 0x7f4460152e60 2024-04-25T00:01:50.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.005+0000 7f445f7fe640 1 -- 172.21.15.116:0/4009066115 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 167771801 0 0) 0x7f4460158ec0 con 0x7f44601599a0 2024-04-25T00:01:50.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.005+0000 7f445f7fe640 1 -- 172.21.15.116:0/4009066115 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4460073200 con 0x7f44601599a0 2024-04-25T00:01:50.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.005+0000 7f445f7fe640 1 -- 172.21.15.116:0/4009066115 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 728282720 0 0) 0x7f4450003530 con 0x7f4460152e60 2024-04-25T00:01:50.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.005+0000 7f445f7fe640 1 -- 172.21.15.116:0/4009066115 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f44601562d0 msgr2=0x7f4460161130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:50.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.005+0000 7f445f7fe640 1 --2- 172.21.15.116:0/4009066115 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f44601562d0 0x7f4460161130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:50.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.005+0000 7f445f7fe640 1 -- 172.21.15.116:0/4009066115 >> v1:172.21.15.174:6789/0 conn(0x7f44601599a0 legacy=0x7f44600c5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:50.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.005+0000 7f445f7fe640 1 -- 172.21.15.116:0/4009066115 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44601591a0 con 0x7f4460152e60 2024-04-25T00:01:50.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.005+0000 7f445f7fe640 1 -- 172.21.15.116:0/4009066115 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f4454001680 con 0x7f4460152e60 2024-04-25T00:01:50.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.005+0000 7f445f7fe640 1 -- 172.21.15.116:0/4009066115 >> v1:172.21.15.116:6789/0 conn(0x7f4460152e60 legacy=0x7f44600c36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:50.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.005+0000 7f445f7fe640 1 --2- 172.21.15.116:0/4009066115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4450003b80 0x7f4450003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:50.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.005+0000 7f445f7fe640 1 --2- 172.21.15.116:0/4009066115 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f44601562d0 0x7f4450004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:50.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.005+0000 7f445f7fe640 1 --2- 172.21.15.116:0/4009066115 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4450004d50 0x7f44500051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:50.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.005+0000 7f445f7fe640 1 -- 172.21.15.116:0/4009066115 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4460156760 con 0x7f4450003b80 2024-04-25T00:01:50.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.005+0000 7f445f7fe640 1 -- 172.21.15.116:0/4009066115 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f445000bad0 con 0x7f4450004d50 2024-04-25T00:01:50.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.005+0000 7f445f7fe640 1 -- 172.21.15.116:0/4009066115 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f445000bcd0 con 0x7f44601562d0 2024-04-25T00:01:50.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.005+0000 7f44648a1640 1 --2- 172.21.15.116:0/4009066115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4450003b80 0x7f4450003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:50.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.005+0000 7f445ffff640 1 --2- 172.21.15.116:0/4009066115 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f44601562d0 0x7f4450004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:50.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.005+0000 7f44650a2640 1 --2- 172.21.15.116:0/4009066115 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4450004d50 0x7f44500051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:50.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.005+0000 7f44648a1640 1 -- 172.21.15.116:0/4009066115 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f44601562d0 msgr2=0x7f4450004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:50.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.005+0000 7f44648a1640 1 --2- 172.21.15.116:0/4009066115 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f44601562d0 0x7f4450004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:50.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.005+0000 7f44648a1640 1 -- 172.21.15.116:0/4009066115 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4450004d50 msgr2=0x7f44500051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:50.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.005+0000 7f44648a1640 1 --2- 172.21.15.116:0/4009066115 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4450004d50 0x7f44500051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:50.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.005+0000 7f44648a1640 1 -- 172.21.15.116:0/4009066115 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4450003530 con 0x7f4450003b80 2024-04-25T00:01:50.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.005+0000 7f44650a2640 1 --2- 172.21.15.116:0/4009066115 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4450004d50 0x7f44500051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:50.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.009+0000 7f44648a1640 1 --2- 172.21.15.116:0/4009066115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4450003b80 0x7f4450003f80 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f445400c1c0 tx=0x7f445401cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49a770d2f923e824 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:50.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.009+0000 7f445f7fe640 1 -- 172.21.15.116:0/4009066115 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4454007cf0 con 0x7f4450003b80 2024-04-25T00:01:50.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.009+0000 7f445f7fe640 1 -- 172.21.15.116:0/4009066115 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44540a9020 con 0x7f4450003b80 2024-04-25T00:01:50.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.009+0000 7f445f7fe640 1 -- 172.21.15.116:0/4009066115 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4454005b00 con 0x7f4450003b80 2024-04-25T00:01:50.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.009+0000 7f4466b05640 1 -- 172.21.15.116:0/4009066115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4450003b80 msgr2=0x7f4450003f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:50.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.009+0000 7f4466b05640 1 --2- 172.21.15.116:0/4009066115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4450003b80 0x7f4450003f80 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f445400c1c0 tx=0x7f445401cc20 comp rx=0 tx=0).stop 2024-04-25T00:01:50.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.009+0000 7f44650a2640 1 -- 172.21.15.116:0/4009066115 reap_dead start 2024-04-25T00:01:50.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.009+0000 7f4466b05640 1 -- 172.21.15.116:0/4009066115 shutdown_connections 2024-04-25T00:01:50.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.009+0000 7f4466b05640 1 -- 172.21.15.116:0/4009066115 >> 172.21.15.116:0/4009066115 conn(0x7f446006c720 msgr2=0x7f446006cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:50.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.009+0000 7f4466b05640 1 -- 172.21.15.116:0/4009066115 shutdown_connections 2024-04-25T00:01:50.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.009+0000 7f4466b05640 1 -- 172.21.15.116:0/4009066115 wait complete. 2024-04-25T00:01:50.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.009+0000 7f4466b05640 1 Processor -- start 2024-04-25T00:01:50.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.009+0000 7f4466b05640 1 -- start start 2024-04-25T00:01:50.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.009+0000 7f4466b05640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4450003b80 0x7f446014a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:50.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.009+0000 7f4466b05640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44601f1100 0x7f44601f1590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:50.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.009+0000 7f4466b05640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f446014a6d0 0x7f44601f1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:50.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.009+0000 7f4466b05640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f44600734e0 con 0x7f44601f1100 2024-04-25T00:01:50.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.009+0000 7f4466b05640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4460153240 con 0x7f446014a6d0 2024-04-25T00:01:50.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.009+0000 7f4466b05640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f44600c5720 con 0x7f4450003b80 2024-04-25T00:01:50.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.009+0000 7f445ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44601f1100 0x7f44601f1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:50.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.009+0000 7f445ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44601f1100 0x7f44601f1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43378/0 (socket says 172.21.15.116:43378) 2024-04-25T00:01:50.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.009+0000 7f445ffff640 1 -- 172.21.15.116:0/3931716799 learned_addr learned my addr 172.21.15.116:0/3931716799 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:50.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.009+0000 7f44648a1640 1 --2- 172.21.15.116:0/3931716799 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4450003b80 0x7f446014a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:50.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.009+0000 7f44650a2640 1 --2- 172.21.15.116:0/3931716799 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f446014a6d0 0x7f44601f1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:50.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.009+0000 7f445ffff640 1 -- 172.21.15.116:0/3931716799 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4450003b80 msgr2=0x7f446014a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:50.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.009+0000 7f445ffff640 1 --2- 172.21.15.116:0/3931716799 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4450003b80 0x7f446014a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:50.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.009+0000 7f445ffff640 1 -- 172.21.15.116:0/3931716799 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f446014a6d0 msgr2=0x7f44601f1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:50.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.009+0000 7f445ffff640 1 --2- 172.21.15.116:0/3931716799 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f446014a6d0 0x7f44601f1ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:50.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.009+0000 7f445ffff640 1 -- 172.21.15.116:0/3931716799 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44601f2010 con 0x7f44601f1100 2024-04-25T00:01:50.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.009+0000 7f44650a2640 1 --2- 172.21.15.116:0/3931716799 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f446014a6d0 0x7f44601f1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:50.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.013+0000 7f445ffff640 1 --2- 172.21.15.116:0/3931716799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44601f1100 0x7f44601f1590 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f444c00e7b0 tx=0x7f444c00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:50.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.013+0000 7f445d7fa640 1 -- 172.21.15.116:0/3931716799 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f444c0050d0 con 0x7f44601f1100 2024-04-25T00:01:50.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.013+0000 7f445d7fa640 1 -- 172.21.15.116:0/3931716799 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f444c018070 con 0x7f44601f1100 2024-04-25T00:01:50.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.013+0000 7f4466b05640 1 -- 172.21.15.116:0/3931716799 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f446014ae40 con 0x7f44601f1100 2024-04-25T00:01:50.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.013+0000 7f445d7fa640 1 -- 172.21.15.116:0/3931716799 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f444c005490 con 0x7f44601f1100 2024-04-25T00:01:50.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.013+0000 7f4466b05640 1 -- 172.21.15.116:0/3931716799 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44601ede50 con 0x7f44601f1100 2024-04-25T00:01:50.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.013+0000 7f4466b05640 1 -- 172.21.15.116:0/3931716799 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4460068a60 con 0x7f44601f1100 2024-04-25T00:01:50.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.013+0000 7f445d7fa640 1 -- 172.21.15.116:0/3931716799 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f444c00c6c0 con 0x7f44601f1100 2024-04-25T00:01:50.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.013+0000 7f445d7fa640 1 --2- 172.21.15.116:0/3931716799 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4444067a90 0x7f4444069f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:50.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.013+0000 7f445d7fa640 1 -- 172.21.15.116:0/3931716799 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6666+0+0 (secure 0 0 0) 0x7f444c085d40 con 0x7f44601f1100 2024-04-25T00:01:50.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.013+0000 7f44648a1640 1 --2- 172.21.15.116:0/3931716799 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4444067a90 0x7f4444069f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:50.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.013+0000 7f44648a1640 1 --2- 172.21.15.116:0/3931716799 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4444067a90 0x7f4444069f50 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f44540aa000 tx=0x7f4454001a40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:50.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.017+0000 7f445d7fa640 1 -- 172.21.15.116:0/3931716799 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f44601f2010 con 0x7f44601f1100 2024-04-25T00:01:50.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.161+0000 7f4466b05640 1 -- 172.21.15.116:0/3931716799 --> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f44600aa650 con 0x7f4444067a90 2024-04-25T00:01:50.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.193+0000 7f445d7fa640 1 -- 172.21.15.116:0/3931716799 <== mgr.4105 v2:172.21.15.174:6832/2004448211 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+490400 (secure 0 0 0) 0x7f44600aa650 con 0x7f4444067a90 2024-04-25T00:01:50.200 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:01:50.205 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-04-25T00:01:50.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.201+0000 7f4466b05640 1 -- 172.21.15.116:0/3931716799 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4444067a90 msgr2=0x7f4444069f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:50.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.201+0000 7f4466b05640 1 --2- 172.21.15.116:0/3931716799 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4444067a90 0x7f4444069f50 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f44540aa000 tx=0x7f4454001a40 comp rx=0 tx=0).stop 2024-04-25T00:01:50.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.201+0000 7f4466b05640 1 -- 172.21.15.116:0/3931716799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44601f1100 msgr2=0x7f44601f1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:50.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.201+0000 7f4466b05640 1 --2- 172.21.15.116:0/3931716799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44601f1100 0x7f44601f1590 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f444c00e7b0 tx=0x7f444c00ec70 comp rx=0 tx=0).stop 2024-04-25T00:01:50.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.201+0000 7f4466b05640 1 -- 172.21.15.116:0/3931716799 shutdown_connections 2024-04-25T00:01:50.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.201+0000 7f4466b05640 1 --2- 172.21.15.116:0/3931716799 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4444067a90 0x7f4444069f50 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:50.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.201+0000 7f4466b05640 1 --2- 172.21.15.116:0/3931716799 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f446014a6d0 0x7f44601f1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:50.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.201+0000 7f4466b05640 1 --2- 172.21.15.116:0/3931716799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44601f1100 0x7f44601f1590 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:50.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.201+0000 7f4466b05640 1 --2- 172.21.15.116:0/3931716799 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4450003b80 0x7f446014a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:50.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.201+0000 7f4466b05640 1 -- 172.21.15.116:0/3931716799 >> 172.21.15.116:0/3931716799 conn(0x7f446006c720 msgr2=0x7f446006f7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:50.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.201+0000 7f4466b05640 1 -- 172.21.15.116:0/3931716799 shutdown_connections 2024-04-25T00:01:50.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.201+0000 7f4466b05640 1 -- 172.21.15.116:0/3931716799 wait complete. 2024-04-25T00:01:50.218 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":52,"stamp":"2024-04-25T00:01:49.268391+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596070,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":253,"num_write_kb":2448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":199,"ondisk_log_size":199,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":387,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219800,"kb_used_data":3628,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754920,"statfs":{"total":773094113280,"available":772869038080,"internally_reserved":0,"allocated":3715072,"data_stored":2145305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":882,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003190"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.656917+0000","last_change":"2024-04-25T00:01:19.508728+0000","last_active":"2024-04-25T00:01:46.656917+0000","last_peered":"2024-04-25T00:01:46.656917+0000","last_clean":"2024-04-25T00:01:46.656917+0000","last_became_active":"2024-04-25T00:01:19.508390+0000","last_became_peered":"2024-04-25T00:01:19.508390+0000","last_unstale":"2024-04-25T00:01:46.656917+0000","last_undegraded":"2024-04-25T00:01:46.656917+0000","last_fullsized":"2024-04-25T00:01:46.656917+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:45.591905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1f","version":"22'4","reported_seq":26,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.515265+0000","last_change":"2024-04-25T00:01:04.356032+0000","last_active":"2024-04-25T00:01:20.515265+0000","last_peered":"2024-04-25T00:01:20.515265+0000","last_clean":"2024-04-25T00:01:20.515265+0000","last_became_active":"2024-04-25T00:01:04.355293+0000","last_became_peered":"2024-04-25T00:01:04.355293+0000","last_unstale":"2024-04-25T00:01:20.515265+0000","last_undegraded":"2024-04-25T00:01:20.515265+0000","last_fullsized":"2024-04-25T00:01:20.515265+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:56:16.678132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.651891+0000","last_change":"2024-04-25T00:01:05.367207+0000","last_active":"2024-04-25T00:01:46.651891+0000","last_peered":"2024-04-25T00:01:46.651891+0000","last_clean":"2024-04-25T00:01:46.651891+0000","last_became_active":"2024-04-25T00:01:05.366471+0000","last_became_peered":"2024-04-25T00:01:05.366471+0000","last_unstale":"2024-04-25T00:01:46.651891+0000","last_undegraded":"2024-04-25T00:01:46.651891+0000","last_fullsized":"2024-04-25T00:01:46.651891+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:30:23.557381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659979+0000","last_change":"2024-04-25T00:01:18.502364+0000","last_active":"2024-04-25T00:01:46.659979+0000","last_peered":"2024-04-25T00:01:46.659979+0000","last_clean":"2024-04-25T00:01:46.659979+0000","last_became_active":"2024-04-25T00:01:18.501633+0000","last_became_peered":"2024-04-25T00:01:18.501633+0000","last_unstale":"2024-04-25T00:01:46.659979+0000","last_undegraded":"2024-04-25T00:01:46.659979+0000","last_fullsized":"2024-04-25T00:01:46.659979+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:55.074018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":10,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.665473+0000","last_change":"2024-04-25T00:01:46.662019+0000","last_active":"2024-04-25T00:01:46.665473+0000","last_peered":"2024-04-25T00:01:46.665473+0000","last_clean":"2024-04-25T00:01:46.665473+0000","last_became_active":"2024-04-25T00:01:46.661167+0000","last_became_peered":"2024-04-25T00:01:46.661167+0000","last_unstale":"2024-04-25T00:01:46.665473+0000","last_undegraded":"2024-04-25T00:01:46.665473+0000","last_fullsized":"2024-04-25T00:01:46.665473+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:13:34.153117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"22'1","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.652395+0000","last_change":"2024-04-25T00:01:04.361809+0000","last_active":"2024-04-25T00:01:46.652395+0000","last_peered":"2024-04-25T00:01:46.652395+0000","last_clean":"2024-04-25T00:01:46.652395+0000","last_became_active":"2024-04-25T00:01:04.361059+0000","last_became_peered":"2024-04-25T00:01:04.361059+0000","last_unstale":"2024-04-25T00:01:46.652395+0000","last_undegraded":"2024-04-25T00:01:46.652395+0000","last_fullsized":"2024-04-25T00:01:46.652395+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.653796+0000","last_change":"2024-04-25T00:01:05.370818+0000","last_active":"2024-04-25T00:01:46.653796+0000","last_peered":"2024-04-25T00:01:46.653796+0000","last_clean":"2024-04-25T00:01:46.653796+0000","last_became_active":"2024-04-25T00:01:05.369991+0000","last_became_peered":"2024-04-25T00:01:05.369991+0000","last_unstale":"2024-04-25T00:01:46.653796+0000","last_undegraded":"2024-04-25T00:01:46.653796+0000","last_fullsized":"2024-04-25T00:01:46.653796+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:43.615895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.647007+0000","last_change":"2024-04-25T00:01:18.497682+0000","last_active":"2024-04-25T00:01:46.647007+0000","last_peered":"2024-04-25T00:01:46.647007+0000","last_clean":"2024-04-25T00:01:46.647007+0000","last_became_active":"2024-04-25T00:01:18.497397+0000","last_became_peered":"2024-04-25T00:01:18.497397+0000","last_unstale":"2024-04-25T00:01:46.647007+0000","last_undegraded":"2024-04-25T00:01:46.647007+0000","last_fullsized":"2024-04-25T00:01:46.647007+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:22.983972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.658174+0000","last_change":"2024-04-25T00:01:19.510591+0000","last_active":"2024-04-25T00:01:46.658174+0000","last_peered":"2024-04-25T00:01:46.658174+0000","last_clean":"2024-04-25T00:01:46.658174+0000","last_became_active":"2024-04-25T00:01:19.510252+0000","last_became_peered":"2024-04-25T00:01:19.510252+0000","last_unstale":"2024-04-25T00:01:46.658174+0000","last_undegraded":"2024-04-25T00:01:46.658174+0000","last_fullsized":"2024-04-25T00:01:46.658174+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:38.005233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"22'2","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660361+0000","last_change":"2024-04-25T00:01:04.362030+0000","last_active":"2024-04-25T00:01:46.660361+0000","last_peered":"2024-04-25T00:01:46.660361+0000","last_clean":"2024-04-25T00:01:46.660361+0000","last_became_active":"2024-04-25T00:01:04.361708+0000","last_became_peered":"2024-04-25T00:01:04.361708+0000","last_unstale":"2024-04-25T00:01:46.660361+0000","last_undegraded":"2024-04-25T00:01:46.660361+0000","last_fullsized":"2024-04-25T00:01:46.660361+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:38:33.392719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.658174+0000","last_change":"2024-04-25T00:01:05.387195+0000","last_active":"2024-04-25T00:01:46.658174+0000","last_peered":"2024-04-25T00:01:46.658174+0000","last_clean":"2024-04-25T00:01:46.658174+0000","last_became_active":"2024-04-25T00:01:05.386853+0000","last_became_peered":"2024-04-25T00:01:05.386853+0000","last_unstale":"2024-04-25T00:01:46.658174+0000","last_undegraded":"2024-04-25T00:01:46.658174+0000","last_fullsized":"2024-04-25T00:01:46.658174+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:06:19.307378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"26'1","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.647549+0000","last_change":"2024-04-25T00:01:18.487835+0000","last_active":"2024-04-25T00:01:46.647549+0000","last_peered":"2024-04-25T00:01:46.647549+0000","last_clean":"2024-04-25T00:01:46.647549+0000","last_became_active":"2024-04-25T00:01:18.487571+0000","last_became_peered":"2024-04-25T00:01:18.487571+0000","last_unstale":"2024-04-25T00:01:46.647549+0000","last_undegraded":"2024-04-25T00:01:46.647549+0000","last_fullsized":"2024-04-25T00:01:46.647549+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:44.582148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.647421+0000","last_change":"2024-04-25T00:01:19.519834+0000","last_active":"2024-04-25T00:01:46.647421+0000","last_peered":"2024-04-25T00:01:46.647421+0000","last_clean":"2024-04-25T00:01:46.647421+0000","last_became_active":"2024-04-25T00:01:19.519470+0000","last_became_peered":"2024-04-25T00:01:19.519470+0000","last_unstale":"2024-04-25T00:01:46.647421+0000","last_undegraded":"2024-04-25T00:01:46.647421+0000","last_fullsized":"2024-04-25T00:01:46.647421+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:47.859626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"22'2","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660830+0000","last_change":"2024-04-25T00:01:04.362832+0000","last_active":"2024-04-25T00:01:46.660830+0000","last_peered":"2024-04-25T00:01:46.660830+0000","last_clean":"2024-04-25T00:01:46.660830+0000","last_became_active":"2024-04-25T00:01:04.362059+0000","last_became_peered":"2024-04-25T00:01:04.362059+0000","last_unstale":"2024-04-25T00:01:46.660830+0000","last_undegraded":"2024-04-25T00:01:46.660830+0000","last_fullsized":"2024-04-25T00:01:46.660830+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:05.441048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659460+0000","last_change":"2024-04-25T00:01:05.381120+0000","last_active":"2024-04-25T00:01:46.659460+0000","last_peered":"2024-04-25T00:01:46.659460+0000","last_clean":"2024-04-25T00:01:46.659460+0000","last_became_active":"2024-04-25T00:01:05.378320+0000","last_became_peered":"2024-04-25T00:01:05.378320+0000","last_unstale":"2024-04-25T00:01:46.659460+0000","last_undegraded":"2024-04-25T00:01:46.659460+0000","last_fullsized":"2024-04-25T00:01:46.659460+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:34:06.175918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"26'1","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.647038+0000","last_change":"2024-04-25T00:01:18.490300+0000","last_active":"2024-04-25T00:01:46.647038+0000","last_peered":"2024-04-25T00:01:46.647038+0000","last_clean":"2024-04-25T00:01:46.647038+0000","last_became_active":"2024-04-25T00:01:18.490050+0000","last_became_peered":"2024-04-25T00:01:18.490050+0000","last_unstale":"2024-04-25T00:01:46.647038+0000","last_undegraded":"2024-04-25T00:01:46.647038+0000","last_fullsized":"2024-04-25T00:01:46.647038+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:13:05.971606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.649896+0000","last_change":"2024-04-25T00:01:19.500026+0000","last_active":"2024-04-25T00:01:46.649896+0000","last_peered":"2024-04-25T00:01:46.649896+0000","last_clean":"2024-04-25T00:01:46.649896+0000","last_became_active":"2024-04-25T00:01:19.499244+0000","last_became_peered":"2024-04-25T00:01:19.499244+0000","last_unstale":"2024-04-25T00:01:46.649896+0000","last_undegraded":"2024-04-25T00:01:46.649896+0000","last_fullsized":"2024-04-25T00:01:46.649896+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:40:36.350832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.658085+0000","last_change":"2024-04-25T00:01:05.376984+0000","last_active":"2024-04-25T00:01:46.658085+0000","last_peered":"2024-04-25T00:01:46.658085+0000","last_clean":"2024-04-25T00:01:46.658085+0000","last_became_active":"2024-04-25T00:01:05.376259+0000","last_became_peered":"2024-04-25T00:01:05.376259+0000","last_unstale":"2024-04-25T00:01:46.658085+0000","last_undegraded":"2024-04-25T00:01:46.658085+0000","last_fullsized":"2024-04-25T00:01:46.658085+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:13:28.543302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"22'2","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.658522+0000","last_change":"2024-04-25T00:01:04.359262+0000","last_active":"2024-04-25T00:01:46.658522+0000","last_peered":"2024-04-25T00:01:46.658522+0000","last_clean":"2024-04-25T00:01:46.658522+0000","last_became_active":"2024-04-25T00:01:04.357971+0000","last_became_peered":"2024-04-25T00:01:04.357971+0000","last_unstale":"2024-04-25T00:01:46.658522+0000","last_undegraded":"2024-04-25T00:01:46.658522+0000","last_fullsized":"2024-04-25T00:01:46.658522+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:05.441048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"26'1","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659290+0000","last_change":"2024-04-25T00:01:18.475624+0000","last_active":"2024-04-25T00:01:46.659290+0000","last_peered":"2024-04-25T00:01:46.659290+0000","last_clean":"2024-04-25T00:01:46.659290+0000","last_became_active":"2024-04-25T00:01:18.475149+0000","last_became_peered":"2024-04-25T00:01:18.475149+0000","last_unstale":"2024-04-25T00:01:46.659290+0000","last_undegraded":"2024-04-25T00:01:46.659290+0000","last_fullsized":"2024-04-25T00:01:46.659290+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:40:35.341075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.664518+0000","last_change":"2024-04-25T00:01:19.531113+0000","last_active":"2024-04-25T00:01:46.664518+0000","last_peered":"2024-04-25T00:01:46.664518+0000","last_clean":"2024-04-25T00:01:46.664518+0000","last_became_active":"2024-04-25T00:01:19.520150+0000","last_became_peered":"2024-04-25T00:01:19.520150+0000","last_unstale":"2024-04-25T00:01:46.664518+0000","last_undegraded":"2024-04-25T00:01:46.664518+0000","last_fullsized":"2024-04-25T00:01:46.664518+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:38.152376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659977+0000","last_change":"2024-04-25T00:01:05.362611+0000","last_active":"2024-04-25T00:01:46.659977+0000","last_peered":"2024-04-25T00:01:46.659977+0000","last_clean":"2024-04-25T00:01:46.659977+0000","last_became_active":"2024-04-25T00:01:05.361817+0000","last_became_peered":"2024-04-25T00:01:05.361817+0000","last_unstale":"2024-04-25T00:01:46.659977+0000","last_undegraded":"2024-04-25T00:01:46.659977+0000","last_fullsized":"2024-04-25T00:01:46.659977+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:50.489125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.663735+0000","last_change":"2024-04-25T00:01:04.371935+0000","last_active":"2024-04-25T00:01:46.663735+0000","last_peered":"2024-04-25T00:01:46.663735+0000","last_clean":"2024-04-25T00:01:46.663735+0000","last_became_active":"2024-04-25T00:01:04.371420+0000","last_became_peered":"2024-04-25T00:01:04.371420+0000","last_unstale":"2024-04-25T00:01:46.663735+0000","last_undegraded":"2024-04-25T00:01:46.663735+0000","last_fullsized":"2024-04-25T00:01:46.663735+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660605+0000","last_change":"2024-04-25T00:01:18.488623+0000","last_active":"2024-04-25T00:01:46.660605+0000","last_peered":"2024-04-25T00:01:46.660605+0000","last_clean":"2024-04-25T00:01:46.660605+0000","last_became_active":"2024-04-25T00:01:18.488291+0000","last_became_peered":"2024-04-25T00:01:18.488291+0000","last_unstale":"2024-04-25T00:01:46.660605+0000","last_undegraded":"2024-04-25T00:01:46.660605+0000","last_fullsized":"2024-04-25T00:01:46.660605+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:30:21.436681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.658546+0000","last_change":"2024-04-25T00:01:19.516731+0000","last_active":"2024-04-25T00:01:46.658546+0000","last_peered":"2024-04-25T00:01:46.658546+0000","last_clean":"2024-04-25T00:01:46.658546+0000","last_became_active":"2024-04-25T00:01:19.516405+0000","last_became_peered":"2024-04-25T00:01:19.516405+0000","last_unstale":"2024-04-25T00:01:46.658546+0000","last_undegraded":"2024-04-25T00:01:46.658546+0000","last_fullsized":"2024-04-25T00:01:46.658546+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:44:37.936092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.510871+0000","last_change":"2024-04-25T00:01:05.355455+0000","last_active":"2024-04-25T00:01:20.510871+0000","last_peered":"2024-04-25T00:01:20.510871+0000","last_clean":"2024-04-25T00:01:20.510871+0000","last_became_active":"2024-04-25T00:01:05.354699+0000","last_became_peered":"2024-04-25T00:01:05.354699+0000","last_unstale":"2024-04-25T00:01:20.510871+0000","last_undegraded":"2024-04-25T00:01:20.510871+0000","last_fullsized":"2024-04-25T00:01:20.510871+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:43:58.069271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.512349+0000","last_change":"2024-04-25T00:01:04.362268+0000","last_active":"2024-04-25T00:01:20.512349+0000","last_peered":"2024-04-25T00:01:20.512349+0000","last_clean":"2024-04-25T00:01:20.512349+0000","last_became_active":"2024-04-25T00:01:04.361622+0000","last_became_peered":"2024-04-25T00:01:04.361622+0000","last_unstale":"2024-04-25T00:01:20.512349+0000","last_undegraded":"2024-04-25T00:01:20.512349+0000","last_fullsized":"2024-04-25T00:01:20.512349+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:58:46.021343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"26'1","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.664549+0000","last_change":"2024-04-25T00:01:18.494047+0000","last_active":"2024-04-25T00:01:46.664549+0000","last_peered":"2024-04-25T00:01:46.664549+0000","last_clean":"2024-04-25T00:01:46.664549+0000","last_became_active":"2024-04-25T00:01:18.493614+0000","last_became_peered":"2024-04-25T00:01:18.493614+0000","last_unstale":"2024-04-25T00:01:46.664549+0000","last_undegraded":"2024-04-25T00:01:46.664549+0000","last_fullsized":"2024-04-25T00:01:46.664549+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:37:08.562016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657743+0000","last_change":"2024-04-25T00:01:19.505397+0000","last_active":"2024-04-25T00:01:46.657743+0000","last_peered":"2024-04-25T00:01:46.657743+0000","last_clean":"2024-04-25T00:01:46.657743+0000","last_became_active":"2024-04-25T00:01:19.505126+0000","last_became_peered":"2024-04-25T00:01:19.505126+0000","last_unstale":"2024-04-25T00:01:46.657743+0000","last_undegraded":"2024-04-25T00:01:46.657743+0000","last_fullsized":"2024-04-25T00:01:46.657743+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:08:49.640267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657635+0000","last_change":"2024-04-25T00:01:05.354953+0000","last_active":"2024-04-25T00:01:46.657635+0000","last_peered":"2024-04-25T00:01:46.657635+0000","last_clean":"2024-04-25T00:01:46.657635+0000","last_became_active":"2024-04-25T00:01:05.354118+0000","last_became_peered":"2024-04-25T00:01:05.354118+0000","last_unstale":"2024-04-25T00:01:46.657635+0000","last_undegraded":"2024-04-25T00:01:46.657635+0000","last_fullsized":"2024-04-25T00:01:46.657635+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:09.858617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657897+0000","last_change":"2024-04-25T00:01:04.365515+0000","last_active":"2024-04-25T00:01:46.657897+0000","last_peered":"2024-04-25T00:01:46.657897+0000","last_clean":"2024-04-25T00:01:46.657897+0000","last_became_active":"2024-04-25T00:01:04.364942+0000","last_became_peered":"2024-04-25T00:01:04.364942+0000","last_unstale":"2024-04-25T00:01:46.657897+0000","last_undegraded":"2024-04-25T00:01:46.657897+0000","last_fullsized":"2024-04-25T00:01:46.657897+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"26'2","reported_seq":20,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.648107+0000","last_change":"2024-04-25T00:01:18.476207+0000","last_active":"2024-04-25T00:01:46.648107+0000","last_peered":"2024-04-25T00:01:46.648107+0000","last_clean":"2024-04-25T00:01:46.648107+0000","last_became_active":"2024-04-25T00:01:18.475862+0000","last_became_peered":"2024-04-25T00:01:18.475862+0000","last_unstale":"2024-04-25T00:01:46.648107+0000","last_undegraded":"2024-04-25T00:01:46.648107+0000","last_fullsized":"2024-04-25T00:01:46.648107+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:11:40.170249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511258+0000","last_change":"2024-04-25T00:01:19.529607+0000","last_active":"2024-04-25T00:01:20.511258+0000","last_peered":"2024-04-25T00:01:20.511258+0000","last_clean":"2024-04-25T00:01:20.511258+0000","last_became_active":"2024-04-25T00:01:19.528509+0000","last_became_peered":"2024-04-25T00:01:19.528509+0000","last_unstale":"2024-04-25T00:01:20.511258+0000","last_undegraded":"2024-04-25T00:01:20.511258+0000","last_fullsized":"2024-04-25T00:01:20.511258+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:45:01.068091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.514958+0000","last_change":"2024-04-25T00:01:05.358106+0000","last_active":"2024-04-25T00:01:20.514958+0000","last_peered":"2024-04-25T00:01:20.514958+0000","last_clean":"2024-04-25T00:01:20.514958+0000","last_became_active":"2024-04-25T00:01:05.357135+0000","last_became_peered":"2024-04-25T00:01:05.357135+0000","last_unstale":"2024-04-25T00:01:20.514958+0000","last_undegraded":"2024-04-25T00:01:20.514958+0000","last_fullsized":"2024-04-25T00:01:20.514958+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:41:42.228234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"22'2","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660058+0000","last_change":"2024-04-25T00:01:04.369917+0000","last_active":"2024-04-25T00:01:46.660058+0000","last_peered":"2024-04-25T00:01:46.660058+0000","last_clean":"2024-04-25T00:01:46.660058+0000","last_became_active":"2024-04-25T00:01:04.369309+0000","last_became_peered":"2024-04-25T00:01:04.369309+0000","last_unstale":"2024-04-25T00:01:46.660058+0000","last_undegraded":"2024-04-25T00:01:46.660058+0000","last_fullsized":"2024-04-25T00:01:46.660058+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:55:08.063965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.647179+0000","last_change":"2024-04-25T00:01:18.477189+0000","last_active":"2024-04-25T00:01:46.647179+0000","last_peered":"2024-04-25T00:01:46.647179+0000","last_clean":"2024-04-25T00:01:46.647179+0000","last_became_active":"2024-04-25T00:01:18.476909+0000","last_became_peered":"2024-04-25T00:01:18.476909+0000","last_unstale":"2024-04-25T00:01:46.647179+0000","last_undegraded":"2024-04-25T00:01:46.647179+0000","last_fullsized":"2024-04-25T00:01:46.647179+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:36.995477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657252+0000","last_change":"2024-04-25T00:01:19.517500+0000","last_active":"2024-04-25T00:01:46.657252+0000","last_peered":"2024-04-25T00:01:46.657252+0000","last_clean":"2024-04-25T00:01:46.657252+0000","last_became_active":"2024-04-25T00:01:19.517182+0000","last_became_peered":"2024-04-25T00:01:19.517182+0000","last_unstale":"2024-04-25T00:01:46.657252+0000","last_undegraded":"2024-04-25T00:01:46.657252+0000","last_fullsized":"2024-04-25T00:01:46.657252+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:58:49.738233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.515146+0000","last_change":"2024-04-25T00:01:05.364829+0000","last_active":"2024-04-25T00:01:20.515146+0000","last_peered":"2024-04-25T00:01:20.515146+0000","last_clean":"2024-04-25T00:01:20.515146+0000","last_became_active":"2024-04-25T00:01:05.364125+0000","last_became_peered":"2024-04-25T00:01:05.364125+0000","last_unstale":"2024-04-25T00:01:20.515146+0000","last_undegraded":"2024-04-25T00:01:20.515146+0000","last_fullsized":"2024-04-25T00:01:20.515146+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:15:45.531760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"23'1","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.515821+0000","last_change":"2024-04-25T00:01:04.361603+0000","last_active":"2024-04-25T00:01:20.515821+0000","last_peered":"2024-04-25T00:01:20.515821+0000","last_clean":"2024-04-25T00:01:20.515821+0000","last_became_active":"2024-04-25T00:01:04.361046+0000","last_became_peered":"2024-04-25T00:01:04.361046+0000","last_unstale":"2024-04-25T00:01:20.515821+0000","last_undegraded":"2024-04-25T00:01:20.515821+0000","last_fullsized":"2024-04-25T00:01:20.515821+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:35:18.036843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.510620+0000","last_change":"2024-04-25T00:01:18.488352+0000","last_active":"2024-04-25T00:01:20.510620+0000","last_peered":"2024-04-25T00:01:20.510620+0000","last_clean":"2024-04-25T00:01:20.510620+0000","last_became_active":"2024-04-25T00:01:18.487321+0000","last_became_peered":"2024-04-25T00:01:18.487321+0000","last_unstale":"2024-04-25T00:01:20.510620+0000","last_undegraded":"2024-04-25T00:01:20.510620+0000","last_fullsized":"2024-04-25T00:01:20.510620+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:37:22.329655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.658811+0000","last_change":"2024-04-25T00:01:19.519653+0000","last_active":"2024-04-25T00:01:46.658811+0000","last_peered":"2024-04-25T00:01:46.658811+0000","last_clean":"2024-04-25T00:01:46.658811+0000","last_became_active":"2024-04-25T00:01:19.519052+0000","last_became_peered":"2024-04-25T00:01:19.519052+0000","last_unstale":"2024-04-25T00:01:46.658811+0000","last_undegraded":"2024-04-25T00:01:46.658811+0000","last_fullsized":"2024-04-25T00:01:46.658811+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:40:36.350832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.665974+0000","last_change":"2024-04-25T00:01:05.374011+0000","last_active":"2024-04-25T00:01:46.665974+0000","last_peered":"2024-04-25T00:01:46.665974+0000","last_clean":"2024-04-25T00:01:46.665974+0000","last_became_active":"2024-04-25T00:01:05.373276+0000","last_became_peered":"2024-04-25T00:01:05.373276+0000","last_unstale":"2024-04-25T00:01:46.665974+0000","last_undegraded":"2024-04-25T00:01:46.665974+0000","last_fullsized":"2024-04-25T00:01:46.665974+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:40:53.349428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659709+0000","last_change":"2024-04-25T00:01:04.365391+0000","last_active":"2024-04-25T00:01:46.659709+0000","last_peered":"2024-04-25T00:01:46.659709+0000","last_clean":"2024-04-25T00:01:46.659709+0000","last_became_active":"2024-04-25T00:01:04.364989+0000","last_became_peered":"2024-04-25T00:01:04.364989+0000","last_unstale":"2024-04-25T00:01:46.659709+0000","last_undegraded":"2024-04-25T00:01:46.659709+0000","last_fullsized":"2024-04-25T00:01:46.659709+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:06:18.298057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.663291+0000","last_change":"2024-04-25T00:01:18.491597+0000","last_active":"2024-04-25T00:01:46.663291+0000","last_peered":"2024-04-25T00:01:46.663291+0000","last_clean":"2024-04-25T00:01:46.663291+0000","last_became_active":"2024-04-25T00:01:18.490859+0000","last_became_peered":"2024-04-25T00:01:18.490859+0000","last_unstale":"2024-04-25T00:01:46.663291+0000","last_undegraded":"2024-04-25T00:01:46.663291+0000","last_fullsized":"2024-04-25T00:01:46.663291+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:36.995477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":10,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.670049+0000","last_change":"2024-04-25T00:01:46.667934+0000","last_active":"2024-04-25T00:01:46.670049+0000","last_peered":"2024-04-25T00:01:46.670049+0000","last_clean":"2024-04-25T00:01:46.670049+0000","last_became_active":"2024-04-25T00:01:46.667075+0000","last_became_peered":"2024-04-25T00:01:46.667075+0000","last_unstale":"2024-04-25T00:01:46.670049+0000","last_undegraded":"2024-04-25T00:01:46.670049+0000","last_fullsized":"2024-04-25T00:01:46.670049+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:28:18.234546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.648286+0000","last_change":"2024-04-25T00:01:05.378340+0000","last_active":"2024-04-25T00:01:46.648286+0000","last_peered":"2024-04-25T00:01:46.648286+0000","last_clean":"2024-04-25T00:01:46.648286+0000","last_became_active":"2024-04-25T00:01:05.377593+0000","last_became_peered":"2024-04-25T00:01:05.377593+0000","last_unstale":"2024-04-25T00:01:46.648286+0000","last_undegraded":"2024-04-25T00:01:46.648286+0000","last_fullsized":"2024-04-25T00:01:46.648286+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:48.202251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.650594+0000","last_change":"2024-04-25T00:01:04.366577+0000","last_active":"2024-04-25T00:01:46.650594+0000","last_peered":"2024-04-25T00:01:46.650594+0000","last_clean":"2024-04-25T00:01:46.650594+0000","last_became_active":"2024-04-25T00:01:04.365881+0000","last_became_peered":"2024-04-25T00:01:04.365881+0000","last_unstale":"2024-04-25T00:01:46.650594+0000","last_undegraded":"2024-04-25T00:01:46.650594+0000","last_fullsized":"2024-04-25T00:01:46.650594+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:49:22.673421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.662487+0000","last_change":"2024-04-25T00:01:18.504341+0000","last_active":"2024-04-25T00:01:46.662487+0000","last_peered":"2024-04-25T00:01:46.662487+0000","last_clean":"2024-04-25T00:01:46.662487+0000","last_became_active":"2024-04-25T00:01:18.491498+0000","last_became_peered":"2024-04-25T00:01:18.491498+0000","last_unstale":"2024-04-25T00:01:46.662487+0000","last_undegraded":"2024-04-25T00:01:46.662487+0000","last_fullsized":"2024-04-25T00:01:46.662487+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:56.741250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.667022+0000","last_change":"2024-04-25T00:01:46.667022+0000","last_active":"2024-04-25T00:01:46.667022+0000","last_peered":"2024-04-25T00:01:46.667022+0000","last_clean":"2024-04-25T00:01:46.667022+0000","last_became_active":"2024-04-25T00:01:46.666224+0000","last_became_peered":"2024-04-25T00:01:46.666224+0000","last_unstale":"2024-04-25T00:01:46.667022+0000","last_undegraded":"2024-04-25T00:01:46.667022+0000","last_fullsized":"2024-04-25T00:01:46.667022+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:25:20.185355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659841+0000","last_change":"2024-04-25T00:01:05.368769+0000","last_active":"2024-04-25T00:01:46.659841+0000","last_peered":"2024-04-25T00:01:46.659841+0000","last_clean":"2024-04-25T00:01:46.659841+0000","last_became_active":"2024-04-25T00:01:05.366817+0000","last_became_peered":"2024-04-25T00:01:05.366817+0000","last_unstale":"2024-04-25T00:01:46.659841+0000","last_undegraded":"2024-04-25T00:01:46.659841+0000","last_fullsized":"2024-04-25T00:01:46.659841+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:11:27.044893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.658528+0000","last_change":"2024-04-25T00:01:04.361900+0000","last_active":"2024-04-25T00:01:46.658528+0000","last_peered":"2024-04-25T00:01:46.658528+0000","last_clean":"2024-04-25T00:01:46.658528+0000","last_became_active":"2024-04-25T00:01:04.361572+0000","last_became_peered":"2024-04-25T00:01:04.361572+0000","last_unstale":"2024-04-25T00:01:46.658528+0000","last_undegraded":"2024-04-25T00:01:46.658528+0000","last_fullsized":"2024-04-25T00:01:46.658528+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:29.218739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.665621+0000","last_change":"2024-04-25T00:01:18.504139+0000","last_active":"2024-04-25T00:01:46.665621+0000","last_peered":"2024-04-25T00:01:46.665621+0000","last_clean":"2024-04-25T00:01:46.665621+0000","last_became_active":"2024-04-25T00:01:18.503379+0000","last_became_peered":"2024-04-25T00:01:18.503379+0000","last_unstale":"2024-04-25T00:01:46.665621+0000","last_undegraded":"2024-04-25T00:01:46.665621+0000","last_fullsized":"2024-04-25T00:01:46.665621+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:02:56.363258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.664313+0000","last_change":"2024-04-25T00:01:19.531756+0000","last_active":"2024-04-25T00:01:46.664313+0000","last_peered":"2024-04-25T00:01:46.664313+0000","last_clean":"2024-04-25T00:01:46.664313+0000","last_became_active":"2024-04-25T00:01:19.531145+0000","last_became_peered":"2024-04-25T00:01:19.531145+0000","last_unstale":"2024-04-25T00:01:46.664313+0000","last_undegraded":"2024-04-25T00:01:46.664313+0000","last_fullsized":"2024-04-25T00:01:46.664313+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:13:42.678415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.649319+0000","last_change":"2024-04-25T00:01:05.354557+0000","last_active":"2024-04-25T00:01:46.649319+0000","last_peered":"2024-04-25T00:01:46.649319+0000","last_clean":"2024-04-25T00:01:46.649319+0000","last_became_active":"2024-04-25T00:01:05.353556+0000","last_became_peered":"2024-04-25T00:01:05.353556+0000","last_unstale":"2024-04-25T00:01:46.649319+0000","last_undegraded":"2024-04-25T00:01:46.649319+0000","last_fullsized":"2024-04-25T00:01:46.649319+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:24.964055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660230+0000","last_change":"2024-04-25T00:01:04.372916+0000","last_active":"2024-04-25T00:01:46.660230+0000","last_peered":"2024-04-25T00:01:46.660230+0000","last_clean":"2024-04-25T00:01:46.660230+0000","last_became_active":"2024-04-25T00:01:04.372185+0000","last_became_peered":"2024-04-25T00:01:04.372185+0000","last_unstale":"2024-04-25T00:01:46.660230+0000","last_undegraded":"2024-04-25T00:01:46.660230+0000","last_fullsized":"2024-04-25T00:01:46.660230+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"26'2","reported_seq":20,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.665571+0000","last_change":"2024-04-25T00:01:18.504614+0000","last_active":"2024-04-25T00:01:46.665571+0000","last_peered":"2024-04-25T00:01:46.665571+0000","last_clean":"2024-04-25T00:01:46.665571+0000","last_became_active":"2024-04-25T00:01:18.501448+0000","last_became_peered":"2024-04-25T00:01:18.501448+0000","last_unstale":"2024-04-25T00:01:46.665571+0000","last_undegraded":"2024-04-25T00:01:46.665571+0000","last_fullsized":"2024-04-25T00:01:46.665571+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:22.983972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659568+0000","last_change":"2024-04-25T00:01:19.517510+0000","last_active":"2024-04-25T00:01:46.659568+0000","last_peered":"2024-04-25T00:01:46.659568+0000","last_clean":"2024-04-25T00:01:46.659568+0000","last_became_active":"2024-04-25T00:01:19.517243+0000","last_became_peered":"2024-04-25T00:01:19.517243+0000","last_unstale":"2024-04-25T00:01:46.659568+0000","last_undegraded":"2024-04-25T00:01:46.659568+0000","last_fullsized":"2024-04-25T00:01:46.659568+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:42:38.149502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.662201+0000","last_change":"2024-04-25T00:01:05.370968+0000","last_active":"2024-04-25T00:01:46.662201+0000","last_peered":"2024-04-25T00:01:46.662201+0000","last_clean":"2024-04-25T00:01:46.662201+0000","last_became_active":"2024-04-25T00:01:05.369987+0000","last_became_peered":"2024-04-25T00:01:05.369987+0000","last_unstale":"2024-04-25T00:01:46.662201+0000","last_undegraded":"2024-04-25T00:01:46.662201+0000","last_fullsized":"2024-04-25T00:01:46.662201+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657919+0000","last_change":"2024-04-25T00:01:04.370226+0000","last_active":"2024-04-25T00:01:46.657919+0000","last_peered":"2024-04-25T00:01:46.657919+0000","last_clean":"2024-04-25T00:01:46.657919+0000","last_became_active":"2024-04-25T00:01:04.369904+0000","last_became_peered":"2024-04-25T00:01:04.369904+0000","last_unstale":"2024-04-25T00:01:46.657919+0000","last_undegraded":"2024-04-25T00:01:46.657919+0000","last_fullsized":"2024-04-25T00:01:46.657919+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:47:25.595275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"27'1","reported_seq":19,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.652131+0000","last_change":"2024-04-25T00:01:18.497156+0000","last_active":"2024-04-25T00:01:46.652131+0000","last_peered":"2024-04-25T00:01:46.652131+0000","last_clean":"2024-04-25T00:01:46.652131+0000","last_became_active":"2024-04-25T00:01:18.495573+0000","last_became_peered":"2024-04-25T00:01:18.495573+0000","last_unstale":"2024-04-25T00:01:46.652131+0000","last_undegraded":"2024-04-25T00:01:46.652131+0000","last_fullsized":"2024-04-25T00:01:46.652131+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:29.119299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660477+0000","last_change":"2024-04-25T00:01:19.519411+0000","last_active":"2024-04-25T00:01:46.660477+0000","last_peered":"2024-04-25T00:01:46.660477+0000","last_clean":"2024-04-25T00:01:46.660477+0000","last_became_active":"2024-04-25T00:01:19.519020+0000","last_became_peered":"2024-04-25T00:01:19.519020+0000","last_unstale":"2024-04-25T00:01:46.660477+0000","last_undegraded":"2024-04-25T00:01:46.660477+0000","last_fullsized":"2024-04-25T00:01:46.660477+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:50.351599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.513395+0000","last_change":"2024-04-25T00:01:05.364939+0000","last_active":"2024-04-25T00:01:20.513395+0000","last_peered":"2024-04-25T00:01:20.513395+0000","last_clean":"2024-04-25T00:01:20.513395+0000","last_became_active":"2024-04-25T00:01:05.364272+0000","last_became_peered":"2024-04-25T00:01:05.364272+0000","last_unstale":"2024-04-25T00:01:20.513395+0000","last_undegraded":"2024-04-25T00:01:20.513395+0000","last_fullsized":"2024-04-25T00:01:20.513395+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:24:59.679908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.658666+0000","last_change":"2024-04-25T00:01:04.357028+0000","last_active":"2024-04-25T00:01:46.658666+0000","last_peered":"2024-04-25T00:01:46.658666+0000","last_clean":"2024-04-25T00:01:46.658666+0000","last_became_active":"2024-04-25T00:01:04.356644+0000","last_became_peered":"2024-04-25T00:01:04.356644+0000","last_unstale":"2024-04-25T00:01:46.658666+0000","last_undegraded":"2024-04-25T00:01:46.658666+0000","last_fullsized":"2024-04-25T00:01:46.658666+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:21.729893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":182,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657375+0000","last_change":"2024-04-25T00:00:55.084492+0000","last_active":"2024-04-25T00:01:46.657375+0000","last_peered":"2024-04-25T00:01:46.657375+0000","last_clean":"2024-04-25T00:01:46.657375+0000","last_became_active":"2024-04-25T00:00:55.083493+0000","last_became_peered":"2024-04-25T00:00:55.083493+0000","last_unstale":"2024-04-25T00:01:46.657375+0000","last_undegraded":"2024-04-25T00:01:46.657375+0000","last_fullsized":"2024-04-25T00:01:46.657375+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:00:54.060578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:00:54.060578+0000","last_clean_scrub_stamp":"2024-04-25T00:00:54.060578+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:14:58.472477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660166+0000","last_change":"2024-04-25T00:01:18.476298+0000","last_active":"2024-04-25T00:01:46.660166+0000","last_peered":"2024-04-25T00:01:46.660166+0000","last_clean":"2024-04-25T00:01:46.660166+0000","last_became_active":"2024-04-25T00:01:18.475898+0000","last_became_peered":"2024-04-25T00:01:18.475898+0000","last_unstale":"2024-04-25T00:01:46.660166+0000","last_undegraded":"2024-04-25T00:01:46.660166+0000","last_fullsized":"2024-04-25T00:01:46.660166+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:49.341842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.652839+0000","last_change":"2024-04-25T00:01:19.518102+0000","last_active":"2024-04-25T00:01:46.652839+0000","last_peered":"2024-04-25T00:01:46.652839+0000","last_clean":"2024-04-25T00:01:46.652839+0000","last_became_active":"2024-04-25T00:01:19.517376+0000","last_became_peered":"2024-04-25T00:01:19.517376+0000","last_unstale":"2024-04-25T00:01:46.652839+0000","last_undegraded":"2024-04-25T00:01:46.652839+0000","last_fullsized":"2024-04-25T00:01:46.652839+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:55:46.853565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659553+0000","last_change":"2024-04-25T00:01:05.376609+0000","last_active":"2024-04-25T00:01:46.659553+0000","last_peered":"2024-04-25T00:01:46.659553+0000","last_clean":"2024-04-25T00:01:46.659553+0000","last_became_active":"2024-04-25T00:01:05.375881+0000","last_became_peered":"2024-04-25T00:01:05.375881+0000","last_unstale":"2024-04-25T00:01:46.659553+0000","last_undegraded":"2024-04-25T00:01:46.659553+0000","last_fullsized":"2024-04-25T00:01:46.659553+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:46:05.592233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.666092+0000","last_change":"2024-04-25T00:01:04.368954+0000","last_active":"2024-04-25T00:01:46.666092+0000","last_peered":"2024-04-25T00:01:46.666092+0000","last_clean":"2024-04-25T00:01:46.666092+0000","last_became_active":"2024-04-25T00:01:04.367903+0000","last_became_peered":"2024-04-25T00:01:04.367903+0000","last_unstale":"2024-04-25T00:01:46.666092+0000","last_undegraded":"2024-04-25T00:01:46.666092+0000","last_fullsized":"2024-04-25T00:01:46.666092+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.647171+0000","last_change":"2024-04-25T00:01:18.476525+0000","last_active":"2024-04-25T00:01:46.647171+0000","last_peered":"2024-04-25T00:01:46.647171+0000","last_clean":"2024-04-25T00:01:46.647171+0000","last_became_active":"2024-04-25T00:01:18.476247+0000","last_became_peered":"2024-04-25T00:01:18.476247+0000","last_unstale":"2024-04-25T00:01:46.647171+0000","last_undegraded":"2024-04-25T00:01:46.647171+0000","last_fullsized":"2024-04-25T00:01:46.647171+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:30:36.682736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.662181+0000","last_change":"2024-04-25T00:01:19.514615+0000","last_active":"2024-04-25T00:01:46.662181+0000","last_peered":"2024-04-25T00:01:46.662181+0000","last_clean":"2024-04-25T00:01:46.662181+0000","last_became_active":"2024-04-25T00:01:19.513640+0000","last_became_peered":"2024-04-25T00:01:19.513640+0000","last_unstale":"2024-04-25T00:01:46.662181+0000","last_undegraded":"2024-04-25T00:01:46.662181+0000","last_fullsized":"2024-04-25T00:01:46.662181+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:08.620602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660785+0000","last_change":"2024-04-25T00:01:05.385670+0000","last_active":"2024-04-25T00:01:46.660785+0000","last_peered":"2024-04-25T00:01:46.660785+0000","last_clean":"2024-04-25T00:01:46.660785+0000","last_became_active":"2024-04-25T00:01:05.384822+0000","last_became_peered":"2024-04-25T00:01:05.384822+0000","last_unstale":"2024-04-25T00:01:46.660785+0000","last_undegraded":"2024-04-25T00:01:46.660785+0000","last_fullsized":"2024-04-25T00:01:46.660785+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:24.017263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"22'1","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.650515+0000","last_change":"2024-04-25T00:01:04.356627+0000","last_active":"2024-04-25T00:01:46.650515+0000","last_peered":"2024-04-25T00:01:46.650515+0000","last_clean":"2024-04-25T00:01:46.650515+0000","last_became_active":"2024-04-25T00:01:04.354419+0000","last_became_peered":"2024-04-25T00:01:04.354419+0000","last_unstale":"2024-04-25T00:01:46.650515+0000","last_undegraded":"2024-04-25T00:01:46.650515+0000","last_fullsized":"2024-04-25T00:01:46.650515+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:26.243635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"26'2","reported_seq":20,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.650452+0000","last_change":"2024-04-25T00:01:18.490511+0000","last_active":"2024-04-25T00:01:46.650452+0000","last_peered":"2024-04-25T00:01:46.650452+0000","last_clean":"2024-04-25T00:01:46.650452+0000","last_became_active":"2024-04-25T00:01:18.489629+0000","last_became_peered":"2024-04-25T00:01:18.489629+0000","last_unstale":"2024-04-25T00:01:46.650452+0000","last_undegraded":"2024-04-25T00:01:46.650452+0000","last_fullsized":"2024-04-25T00:01:46.650452+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:07.610846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657253+0000","last_change":"2024-04-25T00:01:19.508931+0000","last_active":"2024-04-25T00:01:46.657253+0000","last_peered":"2024-04-25T00:01:46.657253+0000","last_clean":"2024-04-25T00:01:46.657253+0000","last_became_active":"2024-04-25T00:01:19.507867+0000","last_became_peered":"2024-04-25T00:01:19.507867+0000","last_unstale":"2024-04-25T00:01:46.657253+0000","last_undegraded":"2024-04-25T00:01:46.657253+0000","last_fullsized":"2024-04-25T00:01:46.657253+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:38.152376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660540+0000","last_change":"2024-04-25T00:01:05.378840+0000","last_active":"2024-04-25T00:01:46.660540+0000","last_peered":"2024-04-25T00:01:46.660540+0000","last_clean":"2024-04-25T00:01:46.660540+0000","last_became_active":"2024-04-25T00:01:05.378112+0000","last_became_peered":"2024-04-25T00:01:05.378112+0000","last_unstale":"2024-04-25T00:01:46.660540+0000","last_undegraded":"2024-04-25T00:01:46.660540+0000","last_fullsized":"2024-04-25T00:01:46.660540+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:43.615895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"22'2","reported_seq":24,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.515607+0000","last_change":"2024-04-25T00:01:04.370794+0000","last_active":"2024-04-25T00:01:20.515607+0000","last_peered":"2024-04-25T00:01:20.515607+0000","last_clean":"2024-04-25T00:01:20.515607+0000","last_became_active":"2024-04-25T00:01:04.370099+0000","last_became_peered":"2024-04-25T00:01:04.370099+0000","last_unstale":"2024-04-25T00:01:20.515607+0000","last_undegraded":"2024-04-25T00:01:20.515607+0000","last_fullsized":"2024-04-25T00:01:20.515607+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:42:03.181036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657372+0000","last_change":"2024-04-25T00:01:18.475256+0000","last_active":"2024-04-25T00:01:46.657372+0000","last_peered":"2024-04-25T00:01:46.657372+0000","last_clean":"2024-04-25T00:01:46.657372+0000","last_became_active":"2024-04-25T00:01:18.474952+0000","last_became_peered":"2024-04-25T00:01:18.474952+0000","last_unstale":"2024-04-25T00:01:46.657372+0000","last_undegraded":"2024-04-25T00:01:46.657372+0000","last_fullsized":"2024-04-25T00:01:46.657372+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:46.849870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660859+0000","last_change":"2024-04-25T00:01:19.516833+0000","last_active":"2024-04-25T00:01:46.660859+0000","last_peered":"2024-04-25T00:01:46.660859+0000","last_clean":"2024-04-25T00:01:46.660859+0000","last_became_active":"2024-04-25T00:01:19.515926+0000","last_became_peered":"2024-04-25T00:01:19.515926+0000","last_unstale":"2024-04-25T00:01:46.660859+0000","last_undegraded":"2024-04-25T00:01:46.660859+0000","last_fullsized":"2024-04-25T00:01:46.660859+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:23:48.855381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.648328+0000","last_change":"2024-04-25T00:01:05.372591+0000","last_active":"2024-04-25T00:01:46.648328+0000","last_peered":"2024-04-25T00:01:46.648328+0000","last_clean":"2024-04-25T00:01:46.648328+0000","last_became_active":"2024-04-25T00:01:05.371204+0000","last_became_peered":"2024-04-25T00:01:05.371204+0000","last_unstale":"2024-04-25T00:01:46.648328+0000","last_undegraded":"2024-04-25T00:01:46.648328+0000","last_fullsized":"2024-04-25T00:01:46.648328+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:15.879673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"22'1","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660665+0000","last_change":"2024-04-25T00:01:04.372376+0000","last_active":"2024-04-25T00:01:46.660665+0000","last_peered":"2024-04-25T00:01:46.660665+0000","last_clean":"2024-04-25T00:01:46.660665+0000","last_became_active":"2024-04-25T00:01:04.372040+0000","last_became_peered":"2024-04-25T00:01:04.372040+0000","last_unstale":"2024-04-25T00:01:46.660665+0000","last_undegraded":"2024-04-25T00:01:46.660665+0000","last_fullsized":"2024-04-25T00:01:46.660665+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:21.729893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657413+0000","last_change":"2024-04-25T00:01:18.487548+0000","last_active":"2024-04-25T00:01:46.657413+0000","last_peered":"2024-04-25T00:01:46.657413+0000","last_clean":"2024-04-25T00:01:46.657413+0000","last_became_active":"2024-04-25T00:01:18.487227+0000","last_became_peered":"2024-04-25T00:01:18.487227+0000","last_unstale":"2024-04-25T00:01:46.657413+0000","last_undegraded":"2024-04-25T00:01:46.657413+0000","last_fullsized":"2024-04-25T00:01:46.657413+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:55.074018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.514828+0000","last_change":"2024-04-25T00:01:19.528574+0000","last_active":"2024-04-25T00:01:20.514828+0000","last_peered":"2024-04-25T00:01:20.514828+0000","last_clean":"2024-04-25T00:01:20.514828+0000","last_became_active":"2024-04-25T00:01:19.525531+0000","last_became_peered":"2024-04-25T00:01:19.525531+0000","last_unstale":"2024-04-25T00:01:20.514828+0000","last_undegraded":"2024-04-25T00:01:20.514828+0000","last_fullsized":"2024-04-25T00:01:20.514828+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:47:08.132895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657517+0000","last_change":"2024-04-25T00:01:05.364103+0000","last_active":"2024-04-25T00:01:46.657517+0000","last_peered":"2024-04-25T00:01:46.657517+0000","last_clean":"2024-04-25T00:01:46.657517+0000","last_became_active":"2024-04-25T00:01:05.363320+0000","last_became_peered":"2024-04-25T00:01:05.363320+0000","last_unstale":"2024-04-25T00:01:46.657517+0000","last_undegraded":"2024-04-25T00:01:46.657517+0000","last_fullsized":"2024-04-25T00:01:46.657517+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:31.456792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"22'8","reported_seq":30,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511508+0000","last_change":"2024-04-25T00:01:04.358365+0000","last_active":"2024-04-25T00:01:20.511508+0000","last_peered":"2024-04-25T00:01:20.511508+0000","last_clean":"2024-04-25T00:01:20.511508+0000","last_became_active":"2024-04-25T00:01:04.357571+0000","last_became_peered":"2024-04-25T00:01:04.357571+0000","last_unstale":"2024-04-25T00:01:20.511508+0000","last_undegraded":"2024-04-25T00:01:20.511508+0000","last_fullsized":"2024-04-25T00:01:20.511508+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:50:23.023114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.651115+0000","last_change":"2024-04-25T00:01:18.497679+0000","last_active":"2024-04-25T00:01:46.651115+0000","last_peered":"2024-04-25T00:01:46.651115+0000","last_clean":"2024-04-25T00:01:46.651115+0000","last_became_active":"2024-04-25T00:01:18.496489+0000","last_became_peered":"2024-04-25T00:01:18.496489+0000","last_unstale":"2024-04-25T00:01:46.651115+0000","last_undegraded":"2024-04-25T00:01:46.651115+0000","last_fullsized":"2024-04-25T00:01:46.651115+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:23:47.845624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.658432+0000","last_change":"2024-04-25T00:01:19.498082+0000","last_active":"2024-04-25T00:01:46.658432+0000","last_peered":"2024-04-25T00:01:46.658432+0000","last_clean":"2024-04-25T00:01:46.658432+0000","last_became_active":"2024-04-25T00:01:19.496805+0000","last_became_peered":"2024-04-25T00:01:19.496805+0000","last_unstale":"2024-04-25T00:01:46.658432+0000","last_undegraded":"2024-04-25T00:01:46.658432+0000","last_fullsized":"2024-04-25T00:01:46.658432+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:57:36.256250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657234+0000","last_change":"2024-04-25T00:01:05.373927+0000","last_active":"2024-04-25T00:01:46.657234+0000","last_peered":"2024-04-25T00:01:46.657234+0000","last_clean":"2024-04-25T00:01:46.657234+0000","last_became_active":"2024-04-25T00:01:05.373077+0000","last_became_peered":"2024-04-25T00:01:05.373077+0000","last_unstale":"2024-04-25T00:01:46.657234+0000","last_undegraded":"2024-04-25T00:01:46.657234+0000","last_fullsized":"2024-04-25T00:01:46.657234+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:50.489125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.658723+0000","last_change":"2024-04-25T00:01:04.357265+0000","last_active":"2024-04-25T00:01:46.658723+0000","last_peered":"2024-04-25T00:01:46.658723+0000","last_clean":"2024-04-25T00:01:46.658723+0000","last_became_active":"2024-04-25T00:01:04.357004+0000","last_became_peered":"2024-04-25T00:01:04.357004+0000","last_unstale":"2024-04-25T00:01:46.658723+0000","last_undegraded":"2024-04-25T00:01:46.658723+0000","last_fullsized":"2024-04-25T00:01:46.658723+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:47:25.595275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.653006+0000","last_change":"2024-04-25T00:01:18.487276+0000","last_active":"2024-04-25T00:01:46.653006+0000","last_peered":"2024-04-25T00:01:46.653006+0000","last_clean":"2024-04-25T00:01:46.653006+0000","last_became_active":"2024-04-25T00:01:18.486279+0000","last_became_peered":"2024-04-25T00:01:18.486279+0000","last_unstale":"2024-04-25T00:01:46.653006+0000","last_undegraded":"2024-04-25T00:01:46.653006+0000","last_fullsized":"2024-04-25T00:01:46.653006+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:17:37.141843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659022+0000","last_change":"2024-04-25T00:01:19.516827+0000","last_active":"2024-04-25T00:01:46.659022+0000","last_peered":"2024-04-25T00:01:46.659022+0000","last_clean":"2024-04-25T00:01:46.659022+0000","last_became_active":"2024-04-25T00:01:19.515765+0000","last_became_peered":"2024-04-25T00:01:19.515765+0000","last_unstale":"2024-04-25T00:01:46.659022+0000","last_undegraded":"2024-04-25T00:01:46.659022+0000","last_fullsized":"2024-04-25T00:01:46.659022+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:13:34.153117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.658378+0000","last_change":"2024-04-25T00:01:05.379501+0000","last_active":"2024-04-25T00:01:46.658378+0000","last_peered":"2024-04-25T00:01:46.658378+0000","last_clean":"2024-04-25T00:01:46.658378+0000","last_became_active":"2024-04-25T00:01:05.378644+0000","last_became_peered":"2024-04-25T00:01:05.378644+0000","last_unstale":"2024-04-25T00:01:46.658378+0000","last_undegraded":"2024-04-25T00:01:46.658378+0000","last_fullsized":"2024-04-25T00:01:46.658378+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:08:35.505154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.658345+0000","last_change":"2024-04-25T00:01:04.360987+0000","last_active":"2024-04-25T00:01:46.658345+0000","last_peered":"2024-04-25T00:01:46.658345+0000","last_clean":"2024-04-25T00:01:46.658345+0000","last_became_active":"2024-04-25T00:01:04.360664+0000","last_became_peered":"2024-04-25T00:01:04.360664+0000","last_unstale":"2024-04-25T00:01:46.658345+0000","last_undegraded":"2024-04-25T00:01:46.658345+0000","last_fullsized":"2024-04-25T00:01:46.658345+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:55.283076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"27'8","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.665406+0000","last_change":"2024-04-25T00:01:18.505577+0000","last_active":"2024-04-25T00:01:46.665406+0000","last_peered":"2024-04-25T00:01:46.665406+0000","last_clean":"2024-04-25T00:01:46.665406+0000","last_became_active":"2024-04-25T00:01:18.504963+0000","last_became_peered":"2024-04-25T00:01:18.504963+0000","last_unstale":"2024-04-25T00:01:46.665406+0000","last_undegraded":"2024-04-25T00:01:46.665406+0000","last_fullsized":"2024-04-25T00:01:46.665406+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:37.142619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2053,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":10,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.662316+0000","last_change":"2024-04-25T00:01:46.658442+0000","last_active":"2024-04-25T00:01:46.662316+0000","last_peered":"2024-04-25T00:01:46.662316+0000","last_clean":"2024-04-25T00:01:46.662316+0000","last_became_active":"2024-04-25T00:01:46.657588+0000","last_became_peered":"2024-04-25T00:01:46.657588+0000","last_unstale":"2024-04-25T00:01:46.662316+0000","last_undegraded":"2024-04-25T00:01:46.662316+0000","last_fullsized":"2024-04-25T00:01:46.662316+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:58:49.738233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.663479+0000","last_change":"2024-04-25T00:01:05.379102+0000","last_active":"2024-04-25T00:01:46.663479+0000","last_peered":"2024-04-25T00:01:46.663479+0000","last_clean":"2024-04-25T00:01:46.663479+0000","last_became_active":"2024-04-25T00:01:05.377625+0000","last_became_peered":"2024-04-25T00:01:05.377625+0000","last_unstale":"2024-04-25T00:01:46.663479+0000","last_undegraded":"2024-04-25T00:01:46.663479+0000","last_fullsized":"2024-04-25T00:01:46.663479+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:15.879673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"22'2","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660428+0000","last_change":"2024-04-25T00:01:04.370640+0000","last_active":"2024-04-25T00:01:46.660428+0000","last_peered":"2024-04-25T00:01:46.660428+0000","last_clean":"2024-04-25T00:01:46.660428+0000","last_became_active":"2024-04-25T00:01:04.370031+0000","last_became_peered":"2024-04-25T00:01:04.370031+0000","last_unstale":"2024-04-25T00:01:46.660428+0000","last_undegraded":"2024-04-25T00:01:46.660428+0000","last_fullsized":"2024-04-25T00:01:46.660428+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:26.243635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"26'3","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659370+0000","last_change":"2024-04-25T00:01:18.473113+0000","last_active":"2024-04-25T00:01:46.659370+0000","last_peered":"2024-04-25T00:01:46.659370+0000","last_clean":"2024-04-25T00:01:46.659370+0000","last_became_active":"2024-04-25T00:01:18.472931+0000","last_became_peered":"2024-04-25T00:01:18.472931+0000","last_unstale":"2024-04-25T00:01:46.659370+0000","last_undegraded":"2024-04-25T00:01:46.659370+0000","last_fullsized":"2024-04-25T00:01:46.659370+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:07.498985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.512763+0000","last_change":"2024-04-25T00:01:19.533664+0000","last_active":"2024-04-25T00:01:20.512763+0000","last_peered":"2024-04-25T00:01:20.512763+0000","last_clean":"2024-04-25T00:01:20.512763+0000","last_became_active":"2024-04-25T00:01:19.533048+0000","last_became_peered":"2024-04-25T00:01:19.533048+0000","last_unstale":"2024-04-25T00:01:20.512763+0000","last_undegraded":"2024-04-25T00:01:20.512763+0000","last_fullsized":"2024-04-25T00:01:20.512763+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:45:43.141569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660317+0000","last_change":"2024-04-25T00:01:05.385827+0000","last_active":"2024-04-25T00:01:46.660317+0000","last_peered":"2024-04-25T00:01:46.660317+0000","last_clean":"2024-04-25T00:01:46.660317+0000","last_became_active":"2024-04-25T00:01:05.385258+0000","last_became_peered":"2024-04-25T00:01:05.385258+0000","last_unstale":"2024-04-25T00:01:46.660317+0000","last_undegraded":"2024-04-25T00:01:46.660317+0000","last_fullsized":"2024-04-25T00:01:46.660317+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:23.870121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"27'8","reported_seq":34,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.648163+0000","last_change":"2024-04-25T00:01:04.365185+0000","last_active":"2024-04-25T00:01:46.648163+0000","last_peered":"2024-04-25T00:01:46.648163+0000","last_clean":"2024-04-25T00:01:46.648163+0000","last_became_active":"2024-04-25T00:01:04.364930+0000","last_became_peered":"2024-04-25T00:01:04.364930+0000","last_unstale":"2024-04-25T00:01:46.648163+0000","last_undegraded":"2024-04-25T00:01:46.648163+0000","last_fullsized":"2024-04-25T00:01:46.648163+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:45:03.618340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2053,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.512982+0000","last_change":"2024-04-25T00:01:18.496220+0000","last_active":"2024-04-25T00:01:20.512982+0000","last_peered":"2024-04-25T00:01:20.512982+0000","last_clean":"2024-04-25T00:01:20.512982+0000","last_became_active":"2024-04-25T00:01:18.492850+0000","last_became_peered":"2024-04-25T00:01:18.492850+0000","last_unstale":"2024-04-25T00:01:20.512982+0000","last_undegraded":"2024-04-25T00:01:20.512982+0000","last_fullsized":"2024-04-25T00:01:20.512982+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:50:16.922314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.662739+0000","last_change":"2024-04-25T00:01:19.515725+0000","last_active":"2024-04-25T00:01:46.662739+0000","last_peered":"2024-04-25T00:01:46.662739+0000","last_clean":"2024-04-25T00:01:46.662739+0000","last_became_active":"2024-04-25T00:01:19.509285+0000","last_became_peered":"2024-04-25T00:01:19.509285+0000","last_unstale":"2024-04-25T00:01:46.662739+0000","last_undegraded":"2024-04-25T00:01:46.662739+0000","last_fullsized":"2024-04-25T00:01:46.662739+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:25:20.185355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.658824+0000","last_change":"2024-04-25T00:01:05.371061+0000","last_active":"2024-04-25T00:01:46.658824+0000","last_peered":"2024-04-25T00:01:46.658824+0000","last_clean":"2024-04-25T00:01:46.658824+0000","last_became_active":"2024-04-25T00:01:05.369950+0000","last_became_peered":"2024-04-25T00:01:05.369950+0000","last_unstale":"2024-04-25T00:01:46.658824+0000","last_undegraded":"2024-04-25T00:01:46.658824+0000","last_fullsized":"2024-04-25T00:01:46.658824+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:02:43.237902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"22'3","reported_seq":29,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.665160+0000","last_change":"2024-04-25T00:01:04.366503+0000","last_active":"2024-04-25T00:01:46.665160+0000","last_peered":"2024-04-25T00:01:46.665160+0000","last_clean":"2024-04-25T00:01:46.665160+0000","last_became_active":"2024-04-25T00:01:04.365945+0000","last_became_peered":"2024-04-25T00:01:04.365945+0000","last_unstale":"2024-04-25T00:01:46.665160+0000","last_undegraded":"2024-04-25T00:01:46.665160+0000","last_fullsized":"2024-04-25T00:01:46.665160+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"26'2","reported_seq":20,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.647716+0000","last_change":"2024-04-25T00:01:18.476456+0000","last_active":"2024-04-25T00:01:46.647716+0000","last_peered":"2024-04-25T00:01:46.647716+0000","last_clean":"2024-04-25T00:01:46.647716+0000","last_became_active":"2024-04-25T00:01:18.475876+0000","last_became_peered":"2024-04-25T00:01:18.475876+0000","last_unstale":"2024-04-25T00:01:46.647716+0000","last_undegraded":"2024-04-25T00:01:46.647716+0000","last_fullsized":"2024-04-25T00:01:46.647716+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:56.741250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657126+0000","last_change":"2024-04-25T00:01:19.515376+0000","last_active":"2024-04-25T00:01:46.657126+0000","last_peered":"2024-04-25T00:01:46.657126+0000","last_clean":"2024-04-25T00:01:46.657126+0000","last_became_active":"2024-04-25T00:01:19.512425+0000","last_became_peered":"2024-04-25T00:01:19.512425+0000","last_unstale":"2024-04-25T00:01:46.657126+0000","last_undegraded":"2024-04-25T00:01:46.657126+0000","last_fullsized":"2024-04-25T00:01:46.657126+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:43:16.957273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.654022+0000","last_change":"2024-04-25T00:01:05.361723+0000","last_active":"2024-04-25T00:01:46.654022+0000","last_peered":"2024-04-25T00:01:46.654022+0000","last_clean":"2024-04-25T00:01:46.654022+0000","last_became_active":"2024-04-25T00:01:05.360956+0000","last_became_peered":"2024-04-25T00:01:05.360956+0000","last_unstale":"2024-04-25T00:01:46.654022+0000","last_undegraded":"2024-04-25T00:01:46.654022+0000","last_fullsized":"2024-04-25T00:01:46.654022+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:23.870121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660056+0000","last_change":"2024-04-25T00:01:04.362129+0000","last_active":"2024-04-25T00:01:46.660056+0000","last_peered":"2024-04-25T00:01:46.660056+0000","last_clean":"2024-04-25T00:01:46.660056+0000","last_became_active":"2024-04-25T00:01:04.361827+0000","last_became_peered":"2024-04-25T00:01:04.361827+0000","last_unstale":"2024-04-25T00:01:46.660056+0000","last_undegraded":"2024-04-25T00:01:46.660056+0000","last_fullsized":"2024-04-25T00:01:46.660056+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:26.243635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"26'8","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659186+0000","last_change":"2024-04-25T00:01:18.496129+0000","last_active":"2024-04-25T00:01:46.659186+0000","last_peered":"2024-04-25T00:01:46.659186+0000","last_clean":"2024-04-25T00:01:46.659186+0000","last_became_active":"2024-04-25T00:01:18.495415+0000","last_became_peered":"2024-04-25T00:01:18.495415+0000","last_unstale":"2024-04-25T00:01:46.659186+0000","last_undegraded":"2024-04-25T00:01:46.659186+0000","last_fullsized":"2024-04-25T00:01:46.659186+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:38.089411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.646840+0000","last_change":"2024-04-25T00:01:19.509542+0000","last_active":"2024-04-25T00:01:46.646840+0000","last_peered":"2024-04-25T00:01:46.646840+0000","last_clean":"2024-04-25T00:01:46.646840+0000","last_became_active":"2024-04-25T00:01:19.509015+0000","last_became_peered":"2024-04-25T00:01:19.509015+0000","last_unstale":"2024-04-25T00:01:46.646840+0000","last_undegraded":"2024-04-25T00:01:46.646840+0000","last_fullsized":"2024-04-25T00:01:46.646840+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:25:20.185355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.648263+0000","last_change":"2024-04-25T00:01:05.365413+0000","last_active":"2024-04-25T00:01:46.648263+0000","last_peered":"2024-04-25T00:01:46.648263+0000","last_clean":"2024-04-25T00:01:46.648263+0000","last_became_active":"2024-04-25T00:01:05.361865+0000","last_became_peered":"2024-04-25T00:01:05.361865+0000","last_unstale":"2024-04-25T00:01:46.648263+0000","last_undegraded":"2024-04-25T00:01:46.648263+0000","last_fullsized":"2024-04-25T00:01:46.648263+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:30.228060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.658791+0000","last_change":"2024-04-25T00:01:04.360804+0000","last_active":"2024-04-25T00:01:46.658791+0000","last_peered":"2024-04-25T00:01:46.658791+0000","last_clean":"2024-04-25T00:01:46.658791+0000","last_became_active":"2024-04-25T00:01:04.360374+0000","last_became_peered":"2024-04-25T00:01:04.360374+0000","last_unstale":"2024-04-25T00:01:46.658791+0000","last_undegraded":"2024-04-25T00:01:46.658791+0000","last_fullsized":"2024-04-25T00:01:46.658791+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.511980+0000","last_change":"2024-04-25T00:01:18.480003+0000","last_active":"2024-04-25T00:01:20.511980+0000","last_peered":"2024-04-25T00:01:20.511980+0000","last_clean":"2024-04-25T00:01:20.511980+0000","last_became_active":"2024-04-25T00:01:18.479270+0000","last_became_peered":"2024-04-25T00:01:18.479270+0000","last_unstale":"2024-04-25T00:01:20.511980+0000","last_undegraded":"2024-04-25T00:01:20.511980+0000","last_fullsized":"2024-04-25T00:01:20.511980+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:52:40.942502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.648354+0000","last_change":"2024-04-25T00:01:19.524463+0000","last_active":"2024-04-25T00:01:46.648354+0000","last_peered":"2024-04-25T00:01:46.648354+0000","last_clean":"2024-04-25T00:01:46.648354+0000","last_became_active":"2024-04-25T00:01:19.524166+0000","last_became_peered":"2024-04-25T00:01:19.524166+0000","last_unstale":"2024-04-25T00:01:46.648354+0000","last_undegraded":"2024-04-25T00:01:46.648354+0000","last_fullsized":"2024-04-25T00:01:46.648354+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:50.351599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"26'1","reported_seq":15,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.514720+0000","last_change":"2024-04-25T00:01:18.497096+0000","last_active":"2024-04-25T00:01:20.514720+0000","last_peered":"2024-04-25T00:01:20.514720+0000","last_clean":"2024-04-25T00:01:20.514720+0000","last_became_active":"2024-04-25T00:01:18.496289+0000","last_became_peered":"2024-04-25T00:01:18.496289+0000","last_unstale":"2024-04-25T00:01:20.514720+0000","last_undegraded":"2024-04-25T00:01:20.514720+0000","last_fullsized":"2024-04-25T00:01:20.514720+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:59:40.316831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.512256+0000","last_change":"2024-04-25T00:01:05.370043+0000","last_active":"2024-04-25T00:01:20.512256+0000","last_peered":"2024-04-25T00:01:20.512256+0000","last_clean":"2024-04-25T00:01:20.512256+0000","last_became_active":"2024-04-25T00:01:05.368080+0000","last_became_peered":"2024-04-25T00:01:05.368080+0000","last_unstale":"2024-04-25T00:01:20.512256+0000","last_undegraded":"2024-04-25T00:01:20.512256+0000","last_fullsized":"2024-04-25T00:01:20.512256+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:58:07.610923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657790+0000","last_change":"2024-04-25T00:01:04.364523+0000","last_active":"2024-04-25T00:01:46.657790+0000","last_peered":"2024-04-25T00:01:46.657790+0000","last_clean":"2024-04-25T00:01:46.657790+0000","last_became_active":"2024-04-25T00:01:04.364050+0000","last_became_peered":"2024-04-25T00:01:04.364050+0000","last_unstale":"2024-04-25T00:01:46.657790+0000","last_undegraded":"2024-04-25T00:01:46.657790+0000","last_fullsized":"2024-04-25T00:01:46.657790+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:49.479804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.664982+0000","last_change":"2024-04-25T00:01:19.491523+0000","last_active":"2024-04-25T00:01:46.664982+0000","last_peered":"2024-04-25T00:01:46.664982+0000","last_clean":"2024-04-25T00:01:46.664982+0000","last_became_active":"2024-04-25T00:01:19.490755+0000","last_became_peered":"2024-04-25T00:01:19.490755+0000","last_unstale":"2024-04-25T00:01:46.664982+0000","last_undegraded":"2024-04-25T00:01:46.664982+0000","last_fullsized":"2024-04-25T00:01:46.664982+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:47:19.526845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"26'4","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659642+0000","last_change":"2024-04-25T00:01:18.472794+0000","last_active":"2024-04-25T00:01:46.659642+0000","last_peered":"2024-04-25T00:01:46.659642+0000","last_clean":"2024-04-25T00:01:46.659642+0000","last_became_active":"2024-04-25T00:01:18.472534+0000","last_became_peered":"2024-04-25T00:01:18.472534+0000","last_unstale":"2024-04-25T00:01:46.659642+0000","last_undegraded":"2024-04-25T00:01:46.659642+0000","last_fullsized":"2024-04-25T00:01:46.659642+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:55:45.843808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.649586+0000","last_change":"2024-04-25T00:01:05.378867+0000","last_active":"2024-04-25T00:01:46.649586+0000","last_peered":"2024-04-25T00:01:46.649586+0000","last_clean":"2024-04-25T00:01:46.649586+0000","last_became_active":"2024-04-25T00:01:05.378056+0000","last_became_peered":"2024-04-25T00:01:05.378056+0000","last_unstale":"2024-04-25T00:01:46.649586+0000","last_undegraded":"2024-04-25T00:01:46.649586+0000","last_fullsized":"2024-04-25T00:01:46.649586+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:36:10.377757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.662462+0000","last_change":"2024-04-25T00:01:04.370896+0000","last_active":"2024-04-25T00:01:46.662462+0000","last_peered":"2024-04-25T00:01:46.662462+0000","last_clean":"2024-04-25T00:01:46.662462+0000","last_became_active":"2024-04-25T00:01:04.370273+0000","last_became_peered":"2024-04-25T00:01:04.370273+0000","last_unstale":"2024-04-25T00:01:46.662462+0000","last_undegraded":"2024-04-25T00:01:46.662462+0000","last_fullsized":"2024-04-25T00:01:46.662462+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:49:22.673421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.514211+0000","last_change":"2024-04-25T00:01:19.521143+0000","last_active":"2024-04-25T00:01:20.514211+0000","last_peered":"2024-04-25T00:01:20.514211+0000","last_clean":"2024-04-25T00:01:20.514211+0000","last_became_active":"2024-04-25T00:01:19.514664+0000","last_became_peered":"2024-04-25T00:01:19.514664+0000","last_unstale":"2024-04-25T00:01:20.514211+0000","last_undegraded":"2024-04-25T00:01:20.514211+0000","last_fullsized":"2024-04-25T00:01:20.514211+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:03:47.440442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"22'1","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.658205+0000","last_change":"2024-04-25T00:01:04.360254+0000","last_active":"2024-04-25T00:01:46.658205+0000","last_peered":"2024-04-25T00:01:46.658205+0000","last_clean":"2024-04-25T00:01:46.658205+0000","last_became_active":"2024-04-25T00:01:04.359586+0000","last_became_peered":"2024-04-25T00:01:04.359586+0000","last_unstale":"2024-04-25T00:01:46.658205+0000","last_undegraded":"2024-04-25T00:01:46.658205+0000","last_fullsized":"2024-04-25T00:01:46.658205+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:47.192930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.513293+0000","last_change":"2024-04-25T00:01:05.370723+0000","last_active":"2024-04-25T00:01:20.513293+0000","last_peered":"2024-04-25T00:01:20.513293+0000","last_clean":"2024-04-25T00:01:20.513293+0000","last_became_active":"2024-04-25T00:01:05.368970+0000","last_became_peered":"2024-04-25T00:01:05.368970+0000","last_unstale":"2024-04-25T00:01:20.513293+0000","last_undegraded":"2024-04-25T00:01:20.513293+0000","last_fullsized":"2024-04-25T00:01:20.513293+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:06:53.743074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"26'2","reported_seq":16,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.513582+0000","last_change":"2024-04-25T00:01:18.498570+0000","last_active":"2024-04-25T00:01:20.513582+0000","last_peered":"2024-04-25T00:01:20.513582+0000","last_clean":"2024-04-25T00:01:20.513582+0000","last_became_active":"2024-04-25T00:01:18.497745+0000","last_became_peered":"2024-04-25T00:01:18.497745+0000","last_unstale":"2024-04-25T00:01:20.513582+0000","last_undegraded":"2024-04-25T00:01:20.513582+0000","last_fullsized":"2024-04-25T00:01:20.513582+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:26:42.047149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.647984+0000","last_change":"2024-04-25T00:01:19.514919+0000","last_active":"2024-04-25T00:01:46.647984+0000","last_peered":"2024-04-25T00:01:46.647984+0000","last_clean":"2024-04-25T00:01:46.647984+0000","last_became_active":"2024-04-25T00:01:19.514280+0000","last_became_peered":"2024-04-25T00:01:19.514280+0000","last_unstale":"2024-04-25T00:01:46.647984+0000","last_undegraded":"2024-04-25T00:01:46.647984+0000","last_fullsized":"2024-04-25T00:01:46.647984+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:37.975265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"22'1","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.653328+0000","last_change":"2024-04-25T00:01:04.365607+0000","last_active":"2024-04-25T00:01:46.653328+0000","last_peered":"2024-04-25T00:01:46.653328+0000","last_clean":"2024-04-25T00:01:46.653328+0000","last_became_active":"2024-04-25T00:01:04.364788+0000","last_became_peered":"2024-04-25T00:01:04.364788+0000","last_unstale":"2024-04-25T00:01:46.653328+0000","last_undegraded":"2024-04-25T00:01:46.653328+0000","last_fullsized":"2024-04-25T00:01:46.653328+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.661022+0000","last_change":"2024-04-25T00:01:05.375142+0000","last_active":"2024-04-25T00:01:46.661022+0000","last_peered":"2024-04-25T00:01:46.661022+0000","last_clean":"2024-04-25T00:01:46.661022+0000","last_became_active":"2024-04-25T00:01:05.373857+0000","last_became_peered":"2024-04-25T00:01:05.373857+0000","last_unstale":"2024-04-25T00:01:46.661022+0000","last_undegraded":"2024-04-25T00:01:46.661022+0000","last_fullsized":"2024-04-25T00:01:46.661022+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:15.879673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"26'2","reported_seq":20,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.664841+0000","last_change":"2024-04-25T00:01:18.502786+0000","last_active":"2024-04-25T00:01:46.664841+0000","last_peered":"2024-04-25T00:01:46.664841+0000","last_clean":"2024-04-25T00:01:46.664841+0000","last_became_active":"2024-04-25T00:01:18.501785+0000","last_became_peered":"2024-04-25T00:01:18.501785+0000","last_unstale":"2024-04-25T00:01:46.664841+0000","last_undegraded":"2024-04-25T00:01:46.664841+0000","last_fullsized":"2024-04-25T00:01:46.664841+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:09:31.441044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.514560+0000","last_change":"2024-04-25T00:01:19.511239+0000","last_active":"2024-04-25T00:01:20.514560+0000","last_peered":"2024-04-25T00:01:20.514560+0000","last_clean":"2024-04-25T00:01:20.514560+0000","last_became_active":"2024-04-25T00:01:19.510155+0000","last_became_peered":"2024-04-25T00:01:19.510155+0000","last_unstale":"2024-04-25T00:01:20.514560+0000","last_undegraded":"2024-04-25T00:01:20.514560+0000","last_fullsized":"2024-04-25T00:01:20.514560+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:43:14.452663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.647454+0000","last_change":"2024-04-25T00:01:05.378517+0000","last_active":"2024-04-25T00:01:46.647454+0000","last_peered":"2024-04-25T00:01:46.647454+0000","last_clean":"2024-04-25T00:01:46.647454+0000","last_became_active":"2024-04-25T00:01:05.377742+0000","last_became_peered":"2024-04-25T00:01:05.377742+0000","last_unstale":"2024-04-25T00:01:46.647454+0000","last_undegraded":"2024-04-25T00:01:46.647454+0000","last_fullsized":"2024-04-25T00:01:46.647454+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:06.450369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.514828+0000","last_change":"2024-04-25T00:01:19.508999+0000","last_active":"2024-04-25T00:01:20.514828+0000","last_peered":"2024-04-25T00:01:20.514828+0000","last_clean":"2024-04-25T00:01:20.514828+0000","last_became_active":"2024-04-25T00:01:19.507960+0000","last_became_peered":"2024-04-25T00:01:19.507960+0000","last_unstale":"2024-04-25T00:01:20.514828+0000","last_undegraded":"2024-04-25T00:01:20.514828+0000","last_fullsized":"2024-04-25T00:01:20.514828+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:32:40.274908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659704+0000","last_change":"2024-04-25T00:01:05.372927+0000","last_active":"2024-04-25T00:01:46.659704+0000","last_peered":"2024-04-25T00:01:46.659704+0000","last_clean":"2024-04-25T00:01:46.659704+0000","last_became_active":"2024-04-25T00:01:05.372050+0000","last_became_peered":"2024-04-25T00:01:05.372050+0000","last_unstale":"2024-04-25T00:01:46.659704+0000","last_undegraded":"2024-04-25T00:01:46.659704+0000","last_fullsized":"2024-04-25T00:01:46.659704+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:48.202251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.647316+0000","last_change":"2024-04-25T00:01:19.531072+0000","last_active":"2024-04-25T00:01:46.647316+0000","last_peered":"2024-04-25T00:01:46.647316+0000","last_clean":"2024-04-25T00:01:46.647316+0000","last_became_active":"2024-04-25T00:01:19.530836+0000","last_became_peered":"2024-04-25T00:01:19.530836+0000","last_unstale":"2024-04-25T00:01:46.647316+0000","last_undegraded":"2024-04-25T00:01:46.647316+0000","last_fullsized":"2024-04-25T00:01:46.647316+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:08.508741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657903+0000","last_change":"2024-04-25T00:01:05.362960+0000","last_active":"2024-04-25T00:01:46.657903+0000","last_peered":"2024-04-25T00:01:46.657903+0000","last_clean":"2024-04-25T00:01:46.657903+0000","last_became_active":"2024-04-25T00:01:05.361067+0000","last_became_peered":"2024-04-25T00:01:05.361067+0000","last_unstale":"2024-04-25T00:01:46.657903+0000","last_undegraded":"2024-04-25T00:01:46.657903+0000","last_fullsized":"2024-04-25T00:01:46.657903+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:24.017263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.650119+0000","last_change":"2024-04-25T00:01:19.506029+0000","last_active":"2024-04-25T00:01:46.650119+0000","last_peered":"2024-04-25T00:01:46.650119+0000","last_clean":"2024-04-25T00:01:46.650119+0000","last_became_active":"2024-04-25T00:01:19.505184+0000","last_became_peered":"2024-04-25T00:01:19.505184+0000","last_unstale":"2024-04-25T00:01:46.650119+0000","last_undegraded":"2024-04-25T00:01:46.650119+0000","last_fullsized":"2024-04-25T00:01:46.650119+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:08.508741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.653074+0000","last_change":"2024-04-25T00:01:05.358690+0000","last_active":"2024-04-25T00:01:46.653074+0000","last_peered":"2024-04-25T00:01:46.653074+0000","last_clean":"2024-04-25T00:01:46.653074+0000","last_became_active":"2024-04-25T00:01:05.357340+0000","last_became_peered":"2024-04-25T00:01:05.357340+0000","last_unstale":"2024-04-25T00:01:46.653074+0000","last_undegraded":"2024-04-25T00:01:46.653074+0000","last_fullsized":"2024-04-25T00:01:46.653074+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:36:55.436661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.658667+0000","last_change":"2024-04-25T00:01:19.508691+0000","last_active":"2024-04-25T00:01:46.658667+0000","last_peered":"2024-04-25T00:01:46.658667+0000","last_clean":"2024-04-25T00:01:46.658667+0000","last_became_active":"2024-04-25T00:01:19.508123+0000","last_became_peered":"2024-04-25T00:01:19.508123+0000","last_unstale":"2024-04-25T00:01:46.658667+0000","last_undegraded":"2024-04-25T00:01:46.658667+0000","last_fullsized":"2024-04-25T00:01:46.658667+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:47:19.526845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.665745+0000","last_change":"2024-04-25T00:01:05.387057+0000","last_active":"2024-04-25T00:01:46.665745+0000","last_peered":"2024-04-25T00:01:46.665745+0000","last_clean":"2024-04-25T00:01:46.665745+0000","last_became_active":"2024-04-25T00:01:05.386724+0000","last_became_peered":"2024-04-25T00:01:05.386724+0000","last_unstale":"2024-04-25T00:01:46.665745+0000","last_undegraded":"2024-04-25T00:01:46.665745+0000","last_fullsized":"2024-04-25T00:01:46.665745+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:55:09.073286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659072+0000","last_change":"2024-04-25T00:01:19.522760+0000","last_active":"2024-04-25T00:01:46.659072+0000","last_peered":"2024-04-25T00:01:46.659072+0000","last_clean":"2024-04-25T00:01:46.659072+0000","last_became_active":"2024-04-25T00:01:19.522380+0000","last_became_peered":"2024-04-25T00:01:19.522380+0000","last_unstale":"2024-04-25T00:01:46.659072+0000","last_undegraded":"2024-04-25T00:01:46.659072+0000","last_fullsized":"2024-04-25T00:01:46.659072+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:55:46.853565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.665267+0000","last_change":"2024-04-25T00:01:05.367296+0000","last_active":"2024-04-25T00:01:46.665267+0000","last_peered":"2024-04-25T00:01:46.665267+0000","last_clean":"2024-04-25T00:01:46.665267+0000","last_became_active":"2024-04-25T00:01:05.363838+0000","last_became_peered":"2024-04-25T00:01:05.363838+0000","last_unstale":"2024-04-25T00:01:46.665267+0000","last_undegraded":"2024-04-25T00:01:46.665267+0000","last_fullsized":"2024-04-25T00:01:46.665267+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:43:07.053074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657236+0000","last_change":"2024-04-25T00:01:19.510902+0000","last_active":"2024-04-25T00:01:46.657236+0000","last_peered":"2024-04-25T00:01:46.657236+0000","last_clean":"2024-04-25T00:01:46.657236+0000","last_became_active":"2024-04-25T00:01:19.510008+0000","last_became_peered":"2024-04-25T00:01:19.510008+0000","last_unstale":"2024-04-25T00:01:46.657236+0000","last_undegraded":"2024-04-25T00:01:46.657236+0000","last_fullsized":"2024-04-25T00:01:46.657236+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:40:32.292140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.648277+0000","last_change":"2024-04-25T00:01:05.365761+0000","last_active":"2024-04-25T00:01:46.648277+0000","last_peered":"2024-04-25T00:01:46.648277+0000","last_clean":"2024-04-25T00:01:46.648277+0000","last_became_active":"2024-04-25T00:01:05.361119+0000","last_became_peered":"2024-04-25T00:01:05.361119+0000","last_unstale":"2024-04-25T00:01:46.648277+0000","last_undegraded":"2024-04-25T00:01:46.648277+0000","last_fullsized":"2024-04-25T00:01:46.648277+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.647042+0000","last_change":"2024-04-25T00:01:19.518534+0000","last_active":"2024-04-25T00:01:46.647042+0000","last_peered":"2024-04-25T00:01:46.647042+0000","last_clean":"2024-04-25T00:01:46.647042+0000","last_became_active":"2024-04-25T00:01:19.518110+0000","last_became_peered":"2024-04-25T00:01:19.518110+0000","last_unstale":"2024-04-25T00:01:46.647042+0000","last_undegraded":"2024-04-25T00:01:46.647042+0000","last_fullsized":"2024-04-25T00:01:46.647042+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:05:09.219823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.653554+0000","last_change":"2024-04-25T00:01:05.373464+0000","last_active":"2024-04-25T00:01:46.653554+0000","last_peered":"2024-04-25T00:01:46.653554+0000","last_clean":"2024-04-25T00:01:46.653554+0000","last_became_active":"2024-04-25T00:01:05.372611+0000","last_became_peered":"2024-04-25T00:01:05.372611+0000","last_unstale":"2024-04-25T00:01:46.653554+0000","last_undegraded":"2024-04-25T00:01:46.653554+0000","last_fullsized":"2024-04-25T00:01:46.653554+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:31.456792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.664300+0000","last_change":"2024-04-25T00:01:19.542475+0000","last_active":"2024-04-25T00:01:46.664300+0000","last_peered":"2024-04-25T00:01:46.664300+0000","last_clean":"2024-04-25T00:01:46.664300+0000","last_became_active":"2024-04-25T00:01:19.541877+0000","last_became_peered":"2024-04-25T00:01:19.541877+0000","last_unstale":"2024-04-25T00:01:46.664300+0000","last_undegraded":"2024-04-25T00:01:46.664300+0000","last_fullsized":"2024-04-25T00:01:46.664300+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:02:57.373014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657634+0000","last_change":"2024-04-25T00:01:05.372906+0000","last_active":"2024-04-25T00:01:46.657634+0000","last_peered":"2024-04-25T00:01:46.657634+0000","last_clean":"2024-04-25T00:01:46.657634+0000","last_became_active":"2024-04-25T00:01:05.372148+0000","last_became_peered":"2024-04-25T00:01:05.372148+0000","last_unstale":"2024-04-25T00:01:46.657634+0000","last_undegraded":"2024-04-25T00:01:46.657634+0000","last_fullsized":"2024-04-25T00:01:46.657634+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:43.615895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659888+0000","last_change":"2024-04-25T00:01:19.498515+0000","last_active":"2024-04-25T00:01:46.659888+0000","last_peered":"2024-04-25T00:01:46.659888+0000","last_clean":"2024-04-25T00:01:46.659888+0000","last_became_active":"2024-04-25T00:01:19.496310+0000","last_became_peered":"2024-04-25T00:01:19.496310+0000","last_unstale":"2024-04-25T00:01:46.659888+0000","last_undegraded":"2024-04-25T00:01:46.659888+0000","last_fullsized":"2024-04-25T00:01:46.659888+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:20:06.661558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.651069+0000","last_change":"2024-04-25T00:01:05.366340+0000","last_active":"2024-04-25T00:01:46.651069+0000","last_peered":"2024-04-25T00:01:46.651069+0000","last_clean":"2024-04-25T00:01:46.651069+0000","last_became_active":"2024-04-25T00:01:05.361952+0000","last_became_peered":"2024-04-25T00:01:05.361952+0000","last_unstale":"2024-04-25T00:01:46.651069+0000","last_undegraded":"2024-04-25T00:01:46.651069+0000","last_fullsized":"2024-04-25T00:01:46.651069+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:41.948662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.661872+0000","last_change":"2024-04-25T00:01:19.508257+0000","last_active":"2024-04-25T00:01:46.661872+0000","last_peered":"2024-04-25T00:01:46.661872+0000","last_clean":"2024-04-25T00:01:46.661872+0000","last_became_active":"2024-04-25T00:01:19.507516+0000","last_became_peered":"2024-04-25T00:01:19.507516+0000","last_unstale":"2024-04-25T00:01:46.661872+0000","last_undegraded":"2024-04-25T00:01:46.661872+0000","last_fullsized":"2024-04-25T00:01:46.661872+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:45.121513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660969+0000","last_change":"2024-04-25T00:01:05.387457+0000","last_active":"2024-04-25T00:01:46.660969+0000","last_peered":"2024-04-25T00:01:46.660969+0000","last_clean":"2024-04-25T00:01:46.660969+0000","last_became_active":"2024-04-25T00:01:05.386919+0000","last_became_peered":"2024-04-25T00:01:05.386919+0000","last_unstale":"2024-04-25T00:01:46.660969+0000","last_undegraded":"2024-04-25T00:01:46.660969+0000","last_fullsized":"2024-04-25T00:01:46.660969+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.664224+0000","last_change":"2024-04-25T00:01:19.520648+0000","last_active":"2024-04-25T00:01:46.664224+0000","last_peered":"2024-04-25T00:01:46.664224+0000","last_clean":"2024-04-25T00:01:46.664224+0000","last_became_active":"2024-04-25T00:01:19.519303+0000","last_became_peered":"2024-04-25T00:01:19.519303+0000","last_unstale":"2024-04-25T00:01:46.664224+0000","last_undegraded":"2024-04-25T00:01:46.664224+0000","last_fullsized":"2024-04-25T00:01:46.664224+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:23.993729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.513073+0000","last_change":"2024-04-25T00:01:05.366249+0000","last_active":"2024-04-25T00:01:20.513073+0000","last_peered":"2024-04-25T00:01:20.513073+0000","last_clean":"2024-04-25T00:01:20.513073+0000","last_became_active":"2024-04-25T00:01:05.365548+0000","last_became_peered":"2024-04-25T00:01:05.365548+0000","last_unstale":"2024-04-25T00:01:20.513073+0000","last_undegraded":"2024-04-25T00:01:20.513073+0000","last_fullsized":"2024-04-25T00:01:20.513073+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:22:21.143639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.647068+0000","last_change":"2024-04-25T00:01:19.514659+0000","last_active":"2024-04-25T00:01:46.647068+0000","last_peered":"2024-04-25T00:01:46.647068+0000","last_clean":"2024-04-25T00:01:46.647068+0000","last_became_active":"2024-04-25T00:01:19.514232+0000","last_became_peered":"2024-04-25T00:01:19.514232+0000","last_unstale":"2024-04-25T00:01:46.647068+0000","last_undegraded":"2024-04-25T00:01:46.647068+0000","last_fullsized":"2024-04-25T00:01:46.647068+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:45.121513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":10,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.668794+0000","last_change":"2024-04-25T00:01:46.665943+0000","last_active":"2024-04-25T00:01:46.668794+0000","last_peered":"2024-04-25T00:01:46.668794+0000","last_clean":"2024-04-25T00:01:46.668794+0000","last_became_active":"2024-04-25T00:01:46.663874+0000","last_became_peered":"2024-04-25T00:01:46.663874+0000","last_unstale":"2024-04-25T00:01:46.668794+0000","last_undegraded":"2024-04-25T00:01:46.668794+0000","last_fullsized":"2024-04-25T00:01:46.668794+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:53:00.633407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659279+0000","last_change":"2024-04-25T00:01:19.503419+0000","last_active":"2024-04-25T00:01:46.659279+0000","last_peered":"2024-04-25T00:01:46.659279+0000","last_clean":"2024-04-25T00:01:46.659279+0000","last_became_active":"2024-04-25T00:01:19.500568+0000","last_became_peered":"2024-04-25T00:01:19.500568+0000","last_unstale":"2024-04-25T00:01:46.659279+0000","last_undegraded":"2024-04-25T00:01:46.659279+0000","last_fullsized":"2024-04-25T00:01:46.659279+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:08.508741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659631+0000","last_change":"2024-04-25T00:01:05.370653+0000","last_active":"2024-04-25T00:01:46.659631+0000","last_peered":"2024-04-25T00:01:46.659631+0000","last_clean":"2024-04-25T00:01:46.659631+0000","last_became_active":"2024-04-25T00:01:05.369919+0000","last_became_peered":"2024-04-25T00:01:05.369919+0000","last_unstale":"2024-04-25T00:01:46.659631+0000","last_undegraded":"2024-04-25T00:01:46.659631+0000","last_fullsized":"2024-04-25T00:01:46.659631+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:30:08.311326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.646840+0000","last_change":"2024-04-25T00:01:19.540104+0000","last_active":"2024-04-25T00:01:46.646840+0000","last_peered":"2024-04-25T00:01:46.646840+0000","last_clean":"2024-04-25T00:01:46.646840+0000","last_became_active":"2024-04-25T00:01:19.539775+0000","last_became_peered":"2024-04-25T00:01:19.539775+0000","last_unstale":"2024-04-25T00:01:46.646840+0000","last_undegraded":"2024-04-25T00:01:46.646840+0000","last_fullsized":"2024-04-25T00:01:46.646840+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:53:14.768520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.664786+0000","last_change":"2024-04-25T00:01:05.367001+0000","last_active":"2024-04-25T00:01:46.664786+0000","last_peered":"2024-04-25T00:01:46.664786+0000","last_clean":"2024-04-25T00:01:46.664786+0000","last_became_active":"2024-04-25T00:01:05.366264+0000","last_became_peered":"2024-04-25T00:01:05.366264+0000","last_unstale":"2024-04-25T00:01:46.664786+0000","last_undegraded":"2024-04-25T00:01:46.664786+0000","last_fullsized":"2024-04-25T00:01:46.664786+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.661915+0000","last_change":"2024-04-25T00:01:19.497247+0000","last_active":"2024-04-25T00:01:46.661915+0000","last_peered":"2024-04-25T00:01:46.661915+0000","last_clean":"2024-04-25T00:01:46.661915+0000","last_became_active":"2024-04-25T00:01:19.496517+0000","last_became_peered":"2024-04-25T00:01:19.496517+0000","last_unstale":"2024-04-25T00:01:46.661915+0000","last_undegraded":"2024-04-25T00:01:46.661915+0000","last_fullsized":"2024-04-25T00:01:46.661915+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:30.129056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659982+0000","last_change":"2024-04-25T00:01:05.379958+0000","last_active":"2024-04-25T00:01:46.659982+0000","last_peered":"2024-04-25T00:01:46.659982+0000","last_clean":"2024-04-25T00:01:46.659982+0000","last_became_active":"2024-04-25T00:01:05.379090+0000","last_became_peered":"2024-04-25T00:01:05.379090+0000","last_unstale":"2024-04-25T00:01:46.659982+0000","last_undegraded":"2024-04-25T00:01:46.659982+0000","last_fullsized":"2024-04-25T00:01:46.659982+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:59:50.832958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.515390+0000","last_change":"2024-04-25T00:01:19.530176+0000","last_active":"2024-04-25T00:01:20.515390+0000","last_peered":"2024-04-25T00:01:20.515390+0000","last_clean":"2024-04-25T00:01:20.515390+0000","last_became_active":"2024-04-25T00:01:19.529410+0000","last_became_peered":"2024-04-25T00:01:19.529410+0000","last_unstale":"2024-04-25T00:01:20.515390+0000","last_undegraded":"2024-04-25T00:01:20.515390+0000","last_fullsized":"2024-04-25T00:01:20.515390+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:40:13.204880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657503+0000","last_change":"2024-04-25T00:01:05.370826+0000","last_active":"2024-04-25T00:01:46.657503+0000","last_peered":"2024-04-25T00:01:46.657503+0000","last_clean":"2024-04-25T00:01:46.657503+0000","last_became_active":"2024-04-25T00:01:05.369848+0000","last_became_peered":"2024-04-25T00:01:05.369848+0000","last_unstale":"2024-04-25T00:01:46.657503+0000","last_undegraded":"2024-04-25T00:01:46.657503+0000","last_fullsized":"2024-04-25T00:01:46.657503+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:23.870121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657618+0000","last_change":"2024-04-25T00:01:19.511625+0000","last_active":"2024-04-25T00:01:46.657618+0000","last_peered":"2024-04-25T00:01:46.657618+0000","last_clean":"2024-04-25T00:01:46.657618+0000","last_became_active":"2024-04-25T00:01:19.511349+0000","last_became_peered":"2024-04-25T00:01:19.511349+0000","last_unstale":"2024-04-25T00:01:46.657618+0000","last_undegraded":"2024-04-25T00:01:46.657618+0000","last_fullsized":"2024-04-25T00:01:46.657618+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:02:57.373014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.650675+0000","last_change":"2024-04-25T00:01:05.390341+0000","last_active":"2024-04-25T00:01:46.650675+0000","last_peered":"2024-04-25T00:01:46.650675+0000","last_clean":"2024-04-25T00:01:46.650675+0000","last_became_active":"2024-04-25T00:01:05.389707+0000","last_became_peered":"2024-04-25T00:01:05.389707+0000","last_unstale":"2024-04-25T00:01:46.650675+0000","last_undegraded":"2024-04-25T00:01:46.650675+0000","last_fullsized":"2024-04-25T00:01:46.650675+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:54:21.138947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657624+0000","last_change":"2024-04-25T00:01:19.513630+0000","last_active":"2024-04-25T00:01:46.657624+0000","last_peered":"2024-04-25T00:01:46.657624+0000","last_clean":"2024-04-25T00:01:46.657624+0000","last_became_active":"2024-04-25T00:01:19.512978+0000","last_became_peered":"2024-04-25T00:01:19.512978+0000","last_unstale":"2024-04-25T00:01:46.657624+0000","last_undegraded":"2024-04-25T00:01:46.657624+0000","last_fullsized":"2024-04-25T00:01:46.657624+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:57.751007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.662982+0000","last_change":"2024-04-25T00:01:05.378373+0000","last_active":"2024-04-25T00:01:46.662982+0000","last_peered":"2024-04-25T00:01:46.662982+0000","last_clean":"2024-04-25T00:01:46.662982+0000","last_became_active":"2024-04-25T00:01:05.376570+0000","last_became_peered":"2024-04-25T00:01:05.376570+0000","last_unstale":"2024-04-25T00:01:46.662982+0000","last_undegraded":"2024-04-25T00:01:46.662982+0000","last_fullsized":"2024-04-25T00:01:46.662982+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:30.228060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659177+0000","last_change":"2024-04-25T00:01:19.516716+0000","last_active":"2024-04-25T00:01:46.659177+0000","last_peered":"2024-04-25T00:01:46.659177+0000","last_clean":"2024-04-25T00:01:46.659177+0000","last_became_active":"2024-04-25T00:01:19.516142+0000","last_became_peered":"2024-04-25T00:01:19.516142+0000","last_unstale":"2024-04-25T00:01:46.659177+0000","last_undegraded":"2024-04-25T00:01:46.659177+0000","last_fullsized":"2024-04-25T00:01:46.659177+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:28:18.234546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.663972+0000","last_change":"2024-04-25T00:01:05.381011+0000","last_active":"2024-04-25T00:01:46.663972+0000","last_peered":"2024-04-25T00:01:46.663972+0000","last_clean":"2024-04-25T00:01:46.663972+0000","last_became_active":"2024-04-25T00:01:05.380236+0000","last_became_peered":"2024-04-25T00:01:05.380236+0000","last_unstale":"2024-04-25T00:01:46.663972+0000","last_undegraded":"2024-04-25T00:01:46.663972+0000","last_fullsized":"2024-04-25T00:01:46.663972+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:06:19.307378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.513107+0000","last_change":"2024-04-25T00:01:19.529023+0000","last_active":"2024-04-25T00:01:20.513107+0000","last_peered":"2024-04-25T00:01:20.513107+0000","last_clean":"2024-04-25T00:01:20.513107+0000","last_became_active":"2024-04-25T00:01:19.524609+0000","last_became_peered":"2024-04-25T00:01:19.524609+0000","last_unstale":"2024-04-25T00:01:20.513107+0000","last_undegraded":"2024-04-25T00:01:20.513107+0000","last_fullsized":"2024-04-25T00:01:20.513107+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:41:19.150476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660549+0000","last_change":"2024-04-25T00:01:05.385853+0000","last_active":"2024-04-25T00:01:46.660549+0000","last_peered":"2024-04-25T00:01:46.660549+0000","last_clean":"2024-04-25T00:01:46.660549+0000","last_became_active":"2024-04-25T00:01:05.384997+0000","last_became_peered":"2024-04-25T00:01:05.384997+0000","last_unstale":"2024-04-25T00:01:46.660549+0000","last_undegraded":"2024-04-25T00:01:46.660549+0000","last_fullsized":"2024-04-25T00:01:46.660549+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:36:55.436661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.662855+0000","last_change":"2024-04-25T00:01:19.516867+0000","last_active":"2024-04-25T00:01:46.662855+0000","last_peered":"2024-04-25T00:01:46.662855+0000","last_clean":"2024-04-25T00:01:46.662855+0000","last_became_active":"2024-04-25T00:01:19.515764+0000","last_became_peered":"2024-04-25T00:01:19.515764+0000","last_unstale":"2024-04-25T00:01:46.662855+0000","last_undegraded":"2024-04-25T00:01:46.662855+0000","last_fullsized":"2024-04-25T00:01:46.662855+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:53:14.768520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660272+0000","last_change":"2024-04-25T00:01:05.389424+0000","last_active":"2024-04-25T00:01:46.660272+0000","last_peered":"2024-04-25T00:01:46.660272+0000","last_clean":"2024-04-25T00:01:46.660272+0000","last_became_active":"2024-04-25T00:01:05.388999+0000","last_became_peered":"2024-04-25T00:01:05.388999+0000","last_unstale":"2024-04-25T00:01:46.660272+0000","last_undegraded":"2024-04-25T00:01:46.660272+0000","last_fullsized":"2024-04-25T00:01:46.660272+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:31.456792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657679+0000","last_change":"2024-04-25T00:01:19.519365+0000","last_active":"2024-04-25T00:01:46.657679+0000","last_peered":"2024-04-25T00:01:46.657679+0000","last_clean":"2024-04-25T00:01:46.657679+0000","last_became_active":"2024-04-25T00:01:19.519085+0000","last_became_peered":"2024-04-25T00:01:19.519085+0000","last_unstale":"2024-04-25T00:01:46.657679+0000","last_undegraded":"2024-04-25T00:01:46.657679+0000","last_fullsized":"2024-04-25T00:01:46.657679+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:05:09.219823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660140+0000","last_change":"2024-04-25T00:01:05.368974+0000","last_active":"2024-04-25T00:01:46.660140+0000","last_peered":"2024-04-25T00:01:46.660140+0000","last_clean":"2024-04-25T00:01:46.660140+0000","last_became_active":"2024-04-25T00:01:05.367893+0000","last_became_peered":"2024-04-25T00:01:05.367893+0000","last_unstale":"2024-04-25T00:01:46.660140+0000","last_undegraded":"2024-04-25T00:01:46.660140+0000","last_fullsized":"2024-04-25T00:01:46.660140+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:30:23.557381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.658127+0000","last_change":"2024-04-25T00:01:19.513514+0000","last_active":"2024-04-25T00:01:46.658127+0000","last_peered":"2024-04-25T00:01:46.658127+0000","last_clean":"2024-04-25T00:01:46.658127+0000","last_became_active":"2024-04-25T00:01:19.513184+0000","last_became_peered":"2024-04-25T00:01:19.513184+0000","last_unstale":"2024-04-25T00:01:46.658127+0000","last_undegraded":"2024-04-25T00:01:46.658127+0000","last_fullsized":"2024-04-25T00:01:46.658127+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:47.859626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.657909+0000","last_change":"2024-04-25T00:01:05.362303+0000","last_active":"2024-04-25T00:01:46.657909+0000","last_peered":"2024-04-25T00:01:46.657909+0000","last_clean":"2024-04-25T00:01:46.657909+0000","last_became_active":"2024-04-25T00:01:05.361586+0000","last_became_peered":"2024-04-25T00:01:05.361586+0000","last_unstale":"2024-04-25T00:01:46.657909+0000","last_undegraded":"2024-04-25T00:01:46.657909+0000","last_fullsized":"2024-04-25T00:01:46.657909+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:30.228060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.650059+0000","last_change":"2024-04-25T00:01:19.515101+0000","last_active":"2024-04-25T00:01:46.650059+0000","last_peered":"2024-04-25T00:01:46.650059+0000","last_clean":"2024-04-25T00:01:46.650059+0000","last_became_active":"2024-04-25T00:01:19.513314+0000","last_became_peered":"2024-04-25T00:01:19.513314+0000","last_unstale":"2024-04-25T00:01:46.650059+0000","last_undegraded":"2024-04-25T00:01:46.650059+0000","last_fullsized":"2024-04-25T00:01:46.650059+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:50.351599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.649234+0000","last_change":"2024-04-25T00:01:05.390256+0000","last_active":"2024-04-25T00:01:46.649234+0000","last_peered":"2024-04-25T00:01:46.649234+0000","last_clean":"2024-04-25T00:01:46.649234+0000","last_became_active":"2024-04-25T00:01:05.389724+0000","last_became_peered":"2024-04-25T00:01:05.389724+0000","last_unstale":"2024-04-25T00:01:46.649234+0000","last_undegraded":"2024-04-25T00:01:46.649234+0000","last_fullsized":"2024-04-25T00:01:46.649234+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:09:18.315688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3f","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.648222+0000","last_change":"2024-04-25T00:01:19.511493+0000","last_active":"2024-04-25T00:01:46.648222+0000","last_peered":"2024-04-25T00:01:46.648222+0000","last_clean":"2024-04-25T00:01:46.648222+0000","last_became_active":"2024-04-25T00:01:19.508049+0000","last_became_peered":"2024-04-25T00:01:19.508049+0000","last_unstale":"2024-04-25T00:01:46.648222+0000","last_undegraded":"2024-04-25T00:01:46.648222+0000","last_fullsized":"2024-04-25T00:01:46.648222+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:55:46.853565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.658614+0000","last_change":"2024-04-25T00:01:05.374766+0000","last_active":"2024-04-25T00:01:46.658614+0000","last_peered":"2024-04-25T00:01:46.658614+0000","last_clean":"2024-04-25T00:01:46.658614+0000","last_became_active":"2024-04-25T00:01:05.369739+0000","last_became_peered":"2024-04-25T00:01:05.369739+0000","last_unstale":"2024-04-25T00:01:46.658614+0000","last_undegraded":"2024-04-25T00:01:46.658614+0000","last_fullsized":"2024-04-25T00:01:46.658614+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:09:18.315688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":12,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-04-25T00:01:20.513219+0000","last_change":"2024-04-25T00:01:19.521456+0000","last_active":"2024-04-25T00:01:20.513219+0000","last_peered":"2024-04-25T00:01:20.513219+0000","last_clean":"2024-04-25T00:01:20.513219+0000","last_became_active":"2024-04-25T00:01:19.520787+0000","last_became_peered":"2024-04-25T00:01:19.520787+0000","last_unstale":"2024-04-25T00:01:20.513219+0000","last_undegraded":"2024-04-25T00:01:20.513219+0000","last_fullsized":"2024-04-25T00:01:20.513219+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:25:38.981364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.647277+0000","last_change":"2024-04-25T00:01:05.359253+0000","last_active":"2024-04-25T00:01:46.647277+0000","last_peered":"2024-04-25T00:01:46.647277+0000","last_clean":"2024-04-25T00:01:46.647277+0000","last_became_active":"2024-04-25T00:01:05.358477+0000","last_became_peered":"2024-04-25T00:01:05.358477+0000","last_unstale":"2024-04-25T00:01:46.647277+0000","last_undegraded":"2024-04-25T00:01:46.647277+0000","last_fullsized":"2024-04-25T00:01:46.647277+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:22.739214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.656961+0000","last_change":"2024-04-25T00:01:19.509505+0000","last_active":"2024-04-25T00:01:46.656961+0000","last_peered":"2024-04-25T00:01:46.656961+0000","last_clean":"2024-04-25T00:01:46.656961+0000","last_became_active":"2024-04-25T00:01:19.507812+0000","last_became_peered":"2024-04-25T00:01:19.507812+0000","last_unstale":"2024-04-25T00:01:46.656961+0000","last_undegraded":"2024-04-25T00:01:46.656961+0000","last_fullsized":"2024-04-25T00:01:46.656961+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:23.993729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.659707+0000","last_change":"2024-04-25T00:01:05.386068+0000","last_active":"2024-04-25T00:01:46.659707+0000","last_peered":"2024-04-25T00:01:46.659707+0000","last_clean":"2024-04-25T00:01:46.659707+0000","last_became_active":"2024-04-25T00:01:05.378650+0000","last_became_peered":"2024-04-25T00:01:05.378650+0000","last_unstale":"2024-04-25T00:01:46.659707+0000","last_undegraded":"2024-04-25T00:01:46.659707+0000","last_fullsized":"2024-04-25T00:01:46.659707+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:12:52.846250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660257+0000","last_change":"2024-04-25T00:01:19.514843+0000","last_active":"2024-04-25T00:01:46.660257+0000","last_peered":"2024-04-25T00:01:46.660257+0000","last_clean":"2024-04-25T00:01:46.660257+0000","last_became_active":"2024-04-25T00:01:19.513956+0000","last_became_peered":"2024-04-25T00:01:19.513956+0000","last_unstale":"2024-04-25T00:01:46.660257+0000","last_undegraded":"2024-04-25T00:01:46.660257+0000","last_fullsized":"2024-04-25T00:01:46.660257+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:38.005233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.648398+0000","last_change":"2024-04-25T00:01:05.359716+0000","last_active":"2024-04-25T00:01:46.648398+0000","last_peered":"2024-04-25T00:01:46.648398+0000","last_clean":"2024-04-25T00:01:46.648398+0000","last_became_active":"2024-04-25T00:01:05.358866+0000","last_became_peered":"2024-04-25T00:01:05.358866+0000","last_unstale":"2024-04-25T00:01:46.648398+0000","last_undegraded":"2024-04-25T00:01:46.648398+0000","last_fullsized":"2024-04-25T00:01:46.648398+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:50.489125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.649702+0000","last_change":"2024-04-25T00:01:19.499574+0000","last_active":"2024-04-25T00:01:46.649702+0000","last_peered":"2024-04-25T00:01:46.649702+0000","last_clean":"2024-04-25T00:01:46.649702+0000","last_became_active":"2024-04-25T00:01:19.498699+0000","last_became_peered":"2024-04-25T00:01:19.498699+0000","last_unstale":"2024-04-25T00:01:46.649702+0000","last_undegraded":"2024-04-25T00:01:46.649702+0000","last_fullsized":"2024-04-25T00:01:46.649702+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:47.859626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.650324+0000","last_change":"2024-04-25T00:01:05.371291+0000","last_active":"2024-04-25T00:01:46.650324+0000","last_peered":"2024-04-25T00:01:46.650324+0000","last_clean":"2024-04-25T00:01:46.650324+0000","last_became_active":"2024-04-25T00:01:05.370522+0000","last_became_peered":"2024-04-25T00:01:05.370522+0000","last_unstale":"2024-04-25T00:01:46.650324+0000","last_undegraded":"2024-04-25T00:01:46.650324+0000","last_fullsized":"2024-04-25T00:01:46.650324+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:11:27.044893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.647355+0000","last_change":"2024-04-25T00:01:19.523798+0000","last_active":"2024-04-25T00:01:46.647355+0000","last_peered":"2024-04-25T00:01:46.647355+0000","last_clean":"2024-04-25T00:01:46.647355+0000","last_became_active":"2024-04-25T00:01:19.523450+0000","last_became_peered":"2024-04-25T00:01:19.523450+0000","last_unstale":"2024-04-25T00:01:46.647355+0000","last_undegraded":"2024-04-25T00:01:46.647355+0000","last_fullsized":"2024-04-25T00:01:46.647355+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:40:36.350832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.651674+0000","last_change":"2024-04-25T00:01:05.368190+0000","last_active":"2024-04-25T00:01:46.651674+0000","last_peered":"2024-04-25T00:01:46.651674+0000","last_clean":"2024-04-25T00:01:46.651674+0000","last_became_active":"2024-04-25T00:01:05.362606+0000","last_became_peered":"2024-04-25T00:01:05.362606+0000","last_unstale":"2024-04-25T00:01:46.651674+0000","last_undegraded":"2024-04-25T00:01:46.651674+0000","last_fullsized":"2024-04-25T00:01:46.651674+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:12:52.846250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":16,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.646901+0000","last_change":"2024-04-25T00:01:19.505737+0000","last_active":"2024-04-25T00:01:46.646901+0000","last_peered":"2024-04-25T00:01:46.646901+0000","last_clean":"2024-04-25T00:01:46.646901+0000","last_became_active":"2024-04-25T00:01:19.504991+0000","last_became_peered":"2024-04-25T00:01:19.504991+0000","last_unstale":"2024-04-25T00:01:46.646901+0000","last_undegraded":"2024-04-25T00:01:46.646901+0000","last_fullsized":"2024-04-25T00:01:46.646901+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:47:19.526845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-04-25T00:01:46.660392+0000","last_change":"2024-04-25T00:01:05.389596+0000","last_active":"2024-04-25T00:01:46.660392+0000","last_peered":"2024-04-25T00:01:46.660392+0000","last_clean":"2024-04-25T00:01:46.660392+0000","last_became_active":"2024-04-25T00:01:05.389127+0000","last_became_peered":"2024-04-25T00:01:05.389127+0000","last_unstale":"2024-04-25T00:01:46.660392+0000","last_undegraded":"2024-04-25T00:01:46.660392+0000","last_fullsized":"2024-04-25T00:01:46.660392+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:09.858617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2851,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":5702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2851,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":5702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509455,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27336,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344504,"statfs":{"total":96636764160,"available":96608772096,"internally_reserved":0,"allocated":319488,"data_stored":123609,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509455,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27320,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344520,"statfs":{"total":96636764160,"available":96608788480,"internally_reserved":0,"allocated":307200,"data_stored":118634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509454,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27900,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343940,"statfs":{"total":96636764160,"available":96608194560,"internally_reserved":0,"allocated":901120,"data_stored":707623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509455,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":327680,"data_stored":124821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509456,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27340,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344500,"statfs":{"total":96636764160,"available":96608768000,"internally_reserved":0,"allocated":323584,"data_stored":125550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509455,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27912,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343928,"statfs":{"total":96636764160,"available":96608182272,"internally_reserved":0,"allocated":909312,"data_stored":713971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509455,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27320,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344520,"statfs":{"total":96636764160,"available":96608788480,"internally_reserved":0,"allocated":307200,"data_stored":116727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574863,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27328,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344512,"statfs":{"total":96636764160,"available":96608780288,"internally_reserved":0,"allocated":319488,"data_stored":114370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T00:01:50.225 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T00:01:50.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.377+0000 7f996024d640 1 Processor -- start 2024-04-25T00:01:50.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.377+0000 7f996024d640 1 -- start start 2024-04-25T00:01:50.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.381+0000 7f996024d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f995813e310 0x7f99580c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:50.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.381+0000 7f996024d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f995813ed60 0x7f99581560d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:50.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.381+0000 7f996024d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f995806d200 con 0x7f99580c2f50 2024-04-25T00:01:50.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.381+0000 7f996024d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f9958070d40 con 0x7f995813e310 2024-04-25T00:01:50.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.381+0000 7f996024d640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f9958071020 con 0x7f995813ed60 2024-04-25T00:01:50.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.381+0000 7f995dfe9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f995813e310 0x7f99580c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:50.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.381+0000 7f995dfe9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f995813e310 0x7f99580c2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43392/0 (socket says 172.21.15.116:43392) 2024-04-25T00:01:50.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.381+0000 7f995dfe9640 1 -- 172.21.15.116:0/3538309688 learned_addr learned my addr 172.21.15.116:0/3538309688 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:50.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.381+0000 7f995d7e8640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f99580c2f50 0x7f99581559c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:47576/0 (socket says 172.21.15.116:47576) 2024-04-25T00:01:50.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.381+0000 7f995e7ea640 1 --2- 172.21.15.116:0/3538309688 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f995813ed60 0x7f99581560d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:50.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.381+0000 7f995dfe9640 1 -- 172.21.15.116:0/3538309688 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f995813ed60 msgr2=0x7f99581560d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:50.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.381+0000 7f995dfe9640 1 --2- 172.21.15.116:0/3538309688 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f995813ed60 0x7f99581560d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:50.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.381+0000 7f995dfe9640 1 -- 172.21.15.116:0/3538309688 >> v1:172.21.15.116:6789/0 conn(0x7f99580c2f50 legacy=0x7f99581559c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:50.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.381+0000 7f995cfe7640 1 -- 172.21.15.116:0/3538309688 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2634558322 0 0) 0x7f995806d200 con 0x7f99580c2f50 2024-04-25T00:01:50.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.381+0000 7f995dfe9640 1 -- 172.21.15.116:0/3538309688 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9958156610 con 0x7f995813e310 2024-04-25T00:01:50.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.381+0000 7f995e7ea640 1 --2- 172.21.15.116:0/3538309688 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f995813ed60 0x7f99581560d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:50.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.381+0000 7f995dfe9640 1 --2- 172.21.15.116:0/3538309688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f995813e310 0x7f99580c2a10 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f9954009870 tx=0x7f995401c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a3e7c54afb1668 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:50.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.381+0000 7f995cfe7640 1 -- 172.21.15.116:0/3538309688 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99540b0070 con 0x7f995813e310 2024-04-25T00:01:50.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.381+0000 7f995cfe7640 1 -- 172.21.15.116:0/3538309688 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9954004230 con 0x7f995813e310 2024-04-25T00:01:50.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.381+0000 7f995cfe7640 1 -- 172.21.15.116:0/3538309688 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9954003620 con 0x7f995813e310 2024-04-25T00:01:50.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.385+0000 7f996024d640 1 -- 172.21.15.116:0/3538309688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f995813e310 msgr2=0x7f99580c2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:50.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.385+0000 7f996024d640 1 --2- 172.21.15.116:0/3538309688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f995813e310 0x7f99580c2a10 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f9954009870 tx=0x7f995401c440 comp rx=0 tx=0).stop 2024-04-25T00:01:50.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.385+0000 7f996024d640 1 -- 172.21.15.116:0/3538309688 shutdown_connections 2024-04-25T00:01:50.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.385+0000 7f996024d640 1 --2- 172.21.15.116:0/3538309688 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f995813ed60 0x7f99581560d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:50.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.385+0000 7f996024d640 1 --2- 172.21.15.116:0/3538309688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f995813e310 0x7f99580c2a10 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:50.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.385+0000 7f996024d640 1 -- 172.21.15.116:0/3538309688 >> 172.21.15.116:0/3538309688 conn(0x7f99580c3b00 msgr2=0x7f99580c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:50.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.385+0000 7f996024d640 1 -- 172.21.15.116:0/3538309688 shutdown_connections 2024-04-25T00:01:50.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.385+0000 7f996024d640 1 -- 172.21.15.116:0/3538309688 wait complete. 2024-04-25T00:01:50.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.385+0000 7f996024d640 1 Processor -- start 2024-04-25T00:01:50.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.385+0000 7f996024d640 1 -- start start 2024-04-25T00:01:50.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.385+0000 7f996024d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f99580c2f50 0x7f995815f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:50.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.385+0000 7f996024d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f995813e310 0x7f9958130150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:50.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.385+0000 7f996024d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f995813ed60 0x7f9958152860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:50.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.385+0000 7f996024d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9958162b70 con 0x7f995813ed60 2024-04-25T00:01:50.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.385+0000 7f996024d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f99581629f0 con 0x7f99580c2f50 2024-04-25T00:01:50.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.385+0000 7f996024d640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f9958162cf0 con 0x7f995813e310 2024-04-25T00:01:50.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.385+0000 7f995e7ea640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f995813ed60 0x7f9958152860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:50.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.385+0000 7f995e7ea640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f995813ed60 0x7f9958152860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43406/0 (socket says 172.21.15.116:43406) 2024-04-25T00:01:50.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.385+0000 7f995e7ea640 1 -- 172.21.15.116:0/3206123639 learned_addr learned my addr 172.21.15.116:0/3206123639 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:50.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.385+0000 7f995d7e8640 1 --2- 172.21.15.116:0/3206123639 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f995813e310 0x7f9958130150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:50.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.385+0000 7f995dfe9640 1 --2- 172.21.15.116:0/3206123639 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f99580c2f50 0x7f995815f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:50.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.385+0000 7f995e7ea640 1 -- 172.21.15.116:0/3206123639 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f995813e310 msgr2=0x7f9958130150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:50.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.385+0000 7f995e7ea640 1 --2- 172.21.15.116:0/3206123639 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f995813e310 0x7f9958130150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:50.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.385+0000 7f995e7ea640 1 -- 172.21.15.116:0/3206123639 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f99580c2f50 msgr2=0x7f995815f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:50.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.385+0000 7f995e7ea640 1 --2- 172.21.15.116:0/3206123639 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f99580c2f50 0x7f995815f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:50.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.385+0000 7f995e7ea640 1 -- 172.21.15.116:0/3206123639 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9958154c80 con 0x7f995813ed60 2024-04-25T00:01:50.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.385+0000 7f995e7ea640 1 --2- 172.21.15.116:0/3206123639 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f995813ed60 0x7f9958152860 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f995000cef0 tx=0x7f995000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:50.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.385+0000 7f994e7fc640 1 -- 172.21.15.116:0/3206123639 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f995001a070 con 0x7f995813ed60 2024-04-25T00:01:50.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.385+0000 7f994e7fc640 1 -- 172.21.15.116:0/3206123639 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f995000d860 con 0x7f995813ed60 2024-04-25T00:01:50.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.385+0000 7f996024d640 1 -- 172.21.15.116:0/3206123639 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9958154ef0 con 0x7f995813ed60 2024-04-25T00:01:50.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.385+0000 7f994e7fc640 1 -- 172.21.15.116:0/3206123639 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f995000dc50 con 0x7f995813ed60 2024-04-25T00:01:50.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.385+0000 7f996024d640 1 -- 172.21.15.116:0/3206123639 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99581554b0 con 0x7f995813ed60 2024-04-25T00:01:50.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.389+0000 7f996024d640 1 -- 172.21.15.116:0/3206123639 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9958068350 con 0x7f995813ed60 2024-04-25T00:01:50.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.389+0000 7f994e7fc640 1 -- 172.21.15.116:0/3206123639 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f995001d070 con 0x7f995813ed60 2024-04-25T00:01:50.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.389+0000 7f994e7fc640 1 --2- 172.21.15.116:0/3206123639 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f993c0678c0 0x7f993c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:50.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.389+0000 7f994e7fc640 1 -- 172.21.15.116:0/3206123639 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6666+0+0 (secure 0 0 0) 0x7f9950086880 con 0x7f995813ed60 2024-04-25T00:01:50.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.389+0000 7f995dfe9640 1 --2- 172.21.15.116:0/3206123639 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f993c0678c0 0x7f993c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:50.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.389+0000 7f995dfe9640 1 --2- 172.21.15.116:0/3206123639 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f993c0678c0 0x7f993c069d80 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f99540ad000 tx=0x7f9954008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:50.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.393+0000 7f994e7fc640 1 -- 172.21.15.116:0/3206123639 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f9950057150 con 0x7f995813ed60 2024-04-25T00:01:50.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.549+0000 7f996024d640 1 -- 172.21.15.116:0/3206123639 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f99580c34f0 con 0x7f995813ed60 2024-04-25T00:01:50.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.553+0000 7f994e7fc640 1 -- 172.21.15.116:0/3206123639 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) ==== 74+0+20606 (secure 0 0 0) 0x7f9950056a50 con 0x7f995813ed60 2024-04-25T00:01:50.557 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:01:50.558 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":29,"fsid":"6d6a1aa7-0478-4c83-8057-05917e81f046","created":"2024-04-25T00:00:40.682493+0000","modified":"2024-04-25T00:01:46.633464+0000","last_up_change":"2024-04-25T00:00:51.822174+0000","last_in_change":"2024-04-25T00:00:42.980473+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T00:00:53.324376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-04-25T00:01:03.197326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-04-25T00:01:03.728618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-04-25T00:01:17.125210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-04-25T00:01:17.821873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8b6ddcb2-1ad9-4167-961c-bd6936bb9060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6809","nonce":3751765682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6811","nonce":3751765682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6815","nonce":3751765682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6813","nonce":3751765682}]},"public_addr":"172.21.15.116:6809/3751765682","cluster_addr":"172.21.15.116:6811/3751765682","heartbeat_back_addr":"172.21.15.116:6815/3751765682","heartbeat_front_addr":"172.21.15.116:6813/3751765682","state":["exists","up"]},{"osd":1,"uuid":"a6431768-994f-41da-8853-886f5687fb91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6801","nonce":377879124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6803","nonce":377879124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6807","nonce":377879124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6805","nonce":377879124}]},"public_addr":"172.21.15.116:6801/377879124","cluster_addr":"172.21.15.116:6803/377879124","heartbeat_back_addr":"172.21.15.116:6807/377879124","heartbeat_front_addr":"172.21.15.116:6805/377879124","state":["exists","up"]},{"osd":2,"uuid":"4f69bd40-5a23-45ff-8693-e1de75e1aa9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6817","nonce":1099282267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6819","nonce":1099282267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6823","nonce":1099282267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6821","nonce":1099282267}]},"public_addr":"172.21.15.116:6817/1099282267","cluster_addr":"172.21.15.116:6819/1099282267","heartbeat_back_addr":"172.21.15.116:6823/1099282267","heartbeat_front_addr":"172.21.15.116:6821/1099282267","state":["exists","up"]},{"osd":3,"uuid":"6a0d5344-8b6f-40d3-918d-0f0a4be688e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6825","nonce":1179942661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6827","nonce":1179942661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6831","nonce":1179942661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6829","nonce":1179942661}]},"public_addr":"172.21.15.116:6825/1179942661","cluster_addr":"172.21.15.116:6827/1179942661","heartbeat_back_addr":"172.21.15.116:6831/1179942661","heartbeat_front_addr":"172.21.15.116:6829/1179942661","state":["exists","up"]},{"osd":4,"uuid":"132e92e4-b825-441e-965a-ca00533fa86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6801","nonce":3132444330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6803","nonce":3132444330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6807","nonce":3132444330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6805","nonce":3132444330}]},"public_addr":"172.21.15.174:6801/3132444330","cluster_addr":"172.21.15.174:6803/3132444330","heartbeat_back_addr":"172.21.15.174:6807/3132444330","heartbeat_front_addr":"172.21.15.174:6805/3132444330","state":["exists","up"]},{"osd":5,"uuid":"a2eeb5b6-1b31-43cc-8306-a0adf3ee15bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6809","nonce":881817164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6811","nonce":881817164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6815","nonce":881817164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6813","nonce":881817164}]},"public_addr":"172.21.15.174:6809/881817164","cluster_addr":"172.21.15.174:6811/881817164","heartbeat_back_addr":"172.21.15.174:6815/881817164","heartbeat_front_addr":"172.21.15.174:6813/881817164","state":["exists","up"]},{"osd":6,"uuid":"588e873a-c841-4ba7-90f2-8956c87a9983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6825","nonce":117574251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6827","nonce":117574251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6831","nonce":117574251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6829","nonce":117574251}]},"public_addr":"172.21.15.174:6825/117574251","cluster_addr":"172.21.15.174:6827/117574251","heartbeat_back_addr":"172.21.15.174:6831/117574251","heartbeat_front_addr":"172.21.15.174:6829/117574251","state":["exists","up"]},{"osd":7,"uuid":"3b90f9b0-8a0e-4883-b088-ca1d4012870c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6817","nonce":2543899196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6819","nonce":2543899196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6823","nonce":2543899196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6821","nonce":2543899196}]},"public_addr":"172.21.15.174:6817/2543899196","cluster_addr":"172.21.15.174:6819/2543899196","heartbeat_back_addr":"172.21.15.174:6823/2543899196","heartbeat_front_addr":"172.21.15.174:6821/2543899196","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.293769+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.310034+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.404865+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.549658+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.192174+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.297266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.493649+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.452309+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2c","mappings":[{"from":0,"to":7}]},{"pgid":"5.3","mappings":[{"from":4,"to":5}]},{"pgid":"5.5","mappings":[{"from":4,"to":5}]},{"pgid":"5.14","mappings":[{"from":4,"to":5}]},{"pgid":"5.19","mappings":[{"from":4,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:01:50.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.557+0000 7f996024d640 1 -- 172.21.15.116:0/3206123639 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f993c0678c0 msgr2=0x7f993c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:50.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.557+0000 7f996024d640 1 --2- 172.21.15.116:0/3206123639 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f993c0678c0 0x7f993c069d80 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f99540ad000 tx=0x7f9954008040 comp rx=0 tx=0).stop 2024-04-25T00:01:50.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.557+0000 7f996024d640 1 -- 172.21.15.116:0/3206123639 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f995813ed60 msgr2=0x7f9958152860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:50.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.557+0000 7f996024d640 1 --2- 172.21.15.116:0/3206123639 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f995813ed60 0x7f9958152860 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f995000cef0 tx=0x7f995000a710 comp rx=0 tx=0).stop 2024-04-25T00:01:50.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.557+0000 7f996024d640 1 -- 172.21.15.116:0/3206123639 shutdown_connections 2024-04-25T00:01:50.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.557+0000 7f996024d640 1 --2- 172.21.15.116:0/3206123639 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f993c0678c0 0x7f993c069d80 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:50.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.557+0000 7f996024d640 1 --2- 172.21.15.116:0/3206123639 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f995813ed60 0x7f9958152860 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:50.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.557+0000 7f996024d640 1 --2- 172.21.15.116:0/3206123639 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f995813e310 0x7f9958130150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:50.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.557+0000 7f996024d640 1 --2- 172.21.15.116:0/3206123639 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f99580c2f50 0x7f995815f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:50.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.557+0000 7f996024d640 1 -- 172.21.15.116:0/3206123639 >> 172.21.15.116:0/3206123639 conn(0x7f99580c3b00 msgr2=0x7f9958072810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:50.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.557+0000 7f996024d640 1 -- 172.21.15.116:0/3206123639 shutdown_connections 2024-04-25T00:01:50.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.557+0000 7f996024d640 1 -- 172.21.15.116:0/3206123639 wait complete. 2024-04-25T00:01:50.575 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T00:00:53.324376+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'a_metadata', 'create_time': '2024-04-25T00:01:03.197326+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'a'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'a_data', 'create_time': '2024-04-25T00:01:03.728618+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'a'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'b_metadata', 'create_time': '2024-04-25T00:01:17.125210+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'b'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'b_data', 'create_time': '2024-04-25T00:01:17.821873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'b'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T00:01:50.576 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T00:01:50.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.685+0000 7fb64b719640 1 Processor -- start 2024-04-25T00:01:50.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.685+0000 7fb64b719640 1 -- start start 2024-04-25T00:01:50.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.685+0000 7fb64b719640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb644157510 0x7fb64415ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:50.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.685+0000 7fb64b719640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6440a9cb0 con 0x7fb644150d60 2024-04-25T00:01:50.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.685+0000 7fb64b719640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6440a9f90 con 0x7fb6440a9630 2024-04-25T00:01:50.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.685+0000 7fb64b719640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fb6440aa270 con 0x7fb644157510 2024-04-25T00:01:50.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.685+0000 7fb649cb6640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb644157510 0x7fb64415ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:50.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.685+0000 7fb6494b5640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fb644150d60 0x7fb6440c2930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:47588/0 (socket says 172.21.15.116:47588) 2024-04-25T00:01:50.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.685+0000 7fb6494b5640 1 -- 172.21.15.116:0/1892341237 learned_addr learned my addr 172.21.15.116:0/1892341237 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:50.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.685+0000 7fb649cb6640 1 -- 172.21.15.116:0/1892341237 >> v1:172.21.15.174:6789/0 conn(0x7fb6440a9630 legacy=0x7fb644156e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:50.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.685+0000 7fb649cb6640 1 -- 172.21.15.116:0/1892341237 >> v1:172.21.15.116:6789/0 conn(0x7fb644150d60 legacy=0x7fb6440c2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:50.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.685+0000 7fb649cb6640 1 -- 172.21.15.116:0/1892341237 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6440aa550 con 0x7fb644157510 2024-04-25T00:01:50.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.685+0000 7fb63bfff640 1 -- 172.21.15.116:0/1892341237 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2516352268 0 0) 0x7fb6440a9cb0 con 0x7fb644150d60 2024-04-25T00:01:50.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.689+0000 7fb649cb6640 1 --2- 172.21.15.116:0/1892341237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb644157510 0x7fb64415ef60 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fb63c009d90 tx=0x7fb63c01c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddf0029ef8bb931a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:50.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.689+0000 7fb63bfff640 1 -- 172.21.15.116:0/1892341237 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb63c0b2070 con 0x7fb644157510 2024-04-25T00:01:50.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.689+0000 7fb63bfff640 1 -- 172.21.15.116:0/1892341237 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb63c0a57d0 con 0x7fb644157510 2024-04-25T00:01:50.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.689+0000 7fb63bfff640 1 -- 172.21.15.116:0/1892341237 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb63c0a5ba0 con 0x7fb644157510 2024-04-25T00:01:50.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.689+0000 7fb64b719640 1 -- 172.21.15.116:0/1892341237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb644157510 msgr2=0x7fb64415ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:50.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.689+0000 7fb64b719640 1 --2- 172.21.15.116:0/1892341237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb644157510 0x7fb64415ef60 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fb63c009d90 tx=0x7fb63c01c7e0 comp rx=0 tx=0).stop 2024-04-25T00:01:50.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.689+0000 7fb64b719640 1 -- 172.21.15.116:0/1892341237 shutdown_connections 2024-04-25T00:01:50.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.689+0000 7fb64b719640 1 --2- 172.21.15.116:0/1892341237 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb644157510 0x7fb64415ef60 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:50.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.689+0000 7fb64b719640 1 -- 172.21.15.116:0/1892341237 >> 172.21.15.116:0/1892341237 conn(0x7fb6440c3a20 msgr2=0x7fb6440c3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:50.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.689+0000 7fb64b719640 1 -- 172.21.15.116:0/1892341237 shutdown_connections 2024-04-25T00:01:50.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.689+0000 7fb64b719640 1 -- 172.21.15.116:0/1892341237 wait complete. 2024-04-25T00:01:50.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.689+0000 7fb64b719640 1 Processor -- start 2024-04-25T00:01:50.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.689+0000 7fb64b719640 1 -- start start 2024-04-25T00:01:50.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.693+0000 7fb64b719640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb6440a9630 0x7fb644152490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:50.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.693+0000 7fb64b719640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb644150d60 0x7fb64414b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:50.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.693+0000 7fb64b719640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb644157510 0x7fb6441ebef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:50.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.693+0000 7fb64b719640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb6440ab710 con 0x7fb644150d60 2024-04-25T00:01:50.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.693+0000 7fb64b719640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb6440ab590 con 0x7fb644157510 2024-04-25T00:01:50.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.693+0000 7fb64b719640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fb644071fd0 con 0x7fb6440a9630 2024-04-25T00:01:50.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.693+0000 7fb648cb4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb644150d60 0x7fb64414b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:50.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.693+0000 7fb648cb4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb644150d60 0x7fb64414b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43432/0 (socket says 172.21.15.116:43432) 2024-04-25T00:01:50.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.693+0000 7fb648cb4640 1 -- 172.21.15.116:0/2363041808 learned_addr learned my addr 172.21.15.116:0/2363041808 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:50.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.693+0000 7fb6494b5640 1 --2- 172.21.15.116:0/2363041808 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb6440a9630 0x7fb644152490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:50.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.693+0000 7fb649cb6640 1 --2- 172.21.15.116:0/2363041808 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb644157510 0x7fb6441ebef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:50.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.693+0000 7fb648cb4640 1 -- 172.21.15.116:0/2363041808 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb6440a9630 msgr2=0x7fb644152490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:50.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.693+0000 7fb648cb4640 1 --2- 172.21.15.116:0/2363041808 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb6440a9630 0x7fb644152490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:50.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.693+0000 7fb648cb4640 1 -- 172.21.15.116:0/2363041808 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb644157510 msgr2=0x7fb6441ebef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:50.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.693+0000 7fb648cb4640 1 --2- 172.21.15.116:0/2363041808 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb644157510 0x7fb6441ebef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:50.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.693+0000 7fb648cb4640 1 -- 172.21.15.116:0/2363041808 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6440c6840 con 0x7fb644150d60 2024-04-25T00:01:50.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.693+0000 7fb6494b5640 1 --2- 172.21.15.116:0/2363041808 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb6440a9630 0x7fb644152490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:50.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.693+0000 7fb648cb4640 1 --2- 172.21.15.116:0/2363041808 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb644150d60 0x7fb64414b240 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fb63400b410 tx=0x7fb63400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:50.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.693+0000 7fb639ffb640 1 -- 172.21.15.116:0/2363041808 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb634015030 con 0x7fb644150d60 2024-04-25T00:01:50.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.693+0000 7fb639ffb640 1 -- 172.21.15.116:0/2363041808 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb634004020 con 0x7fb644150d60 2024-04-25T00:01:50.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.693+0000 7fb64b719640 1 -- 172.21.15.116:0/2363041808 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6441ef5e0 con 0x7fb644150d60 2024-04-25T00:01:50.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.693+0000 7fb639ffb640 1 -- 172.21.15.116:0/2363041808 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb634004410 con 0x7fb644150d60 2024-04-25T00:01:50.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.693+0000 7fb64b719640 1 -- 172.21.15.116:0/2363041808 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6441efba0 con 0x7fb644150d60 2024-04-25T00:01:50.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.693+0000 7fb64b719640 1 -- 172.21.15.116:0/2363041808 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb64404f510 con 0x7fb644150d60 2024-04-25T00:01:50.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.693+0000 7fb639ffb640 1 -- 172.21.15.116:0/2363041808 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb634018070 con 0x7fb644150d60 2024-04-25T00:01:50.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.693+0000 7fb639ffb640 1 --2- 172.21.15.116:0/2363041808 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb6280678c0 0x7fb628069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:50.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.693+0000 7fb639ffb640 1 -- 172.21.15.116:0/2363041808 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6666+0+0 (secure 0 0 0) 0x7fb634085ab0 con 0x7fb644150d60 2024-04-25T00:01:50.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.693+0000 7fb6494b5640 1 --2- 172.21.15.116:0/2363041808 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb6280678c0 0x7fb628069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:50.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.693+0000 7fb6494b5640 1 --2- 172.21.15.116:0/2363041808 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb6280678c0 0x7fb628069d80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb640000f70 tx=0x7fb640009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:50.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.697+0000 7fb639ffb640 1 -- 172.21.15.116:0/2363041808 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fb6340562c0 con 0x7fb644150d60 2024-04-25T00:01:50.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.845+0000 7fb64b719640 1 -- 172.21.15.116:0/2363041808 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb644071db0 con 0x7fb644150d60 2024-04-25T00:01:50.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.845+0000 7fb639ffb640 1 -- 172.21.15.116:0/2363041808 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) ==== 93+0+10 (secure 0 0 0) 0x7fb634055bc0 con 0x7fb644150d60 2024-04-25T00:01:50.850 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-04-25T00:01:50.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.849+0000 7fb64b719640 1 -- 172.21.15.116:0/2363041808 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb6280678c0 msgr2=0x7fb628069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:50.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.849+0000 7fb64b719640 1 --2- 172.21.15.116:0/2363041808 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb6280678c0 0x7fb628069d80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb640000f70 tx=0x7fb640009040 comp rx=0 tx=0).stop 2024-04-25T00:01:50.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.849+0000 7fb64b719640 1 -- 172.21.15.116:0/2363041808 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb644150d60 msgr2=0x7fb64414b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:50.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.849+0000 7fb64b719640 1 --2- 172.21.15.116:0/2363041808 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb644150d60 0x7fb64414b240 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fb63400b410 tx=0x7fb63400b8d0 comp rx=0 tx=0).stop 2024-04-25T00:01:50.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.849+0000 7fb64b719640 1 -- 172.21.15.116:0/2363041808 shutdown_connections 2024-04-25T00:01:50.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.849+0000 7fb64b719640 1 --2- 172.21.15.116:0/2363041808 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb6280678c0 0x7fb628069d80 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:50.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.849+0000 7fb64b719640 1 --2- 172.21.15.116:0/2363041808 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb644157510 0x7fb6441ebef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:50.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.849+0000 7fb64b719640 1 --2- 172.21.15.116:0/2363041808 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb644150d60 0x7fb64414b240 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:50.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.849+0000 7fb64b719640 1 --2- 172.21.15.116:0/2363041808 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb6440a9630 0x7fb644152490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:50.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.849+0000 7fb64b719640 1 -- 172.21.15.116:0/2363041808 >> 172.21.15.116:0/2363041808 conn(0x7fb6440c3a20 msgr2=0x7fb644070ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:50.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.849+0000 7fb64b719640 1 -- 172.21.15.116:0/2363041808 shutdown_connections 2024-04-25T00:01:50.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.849+0000 7fb64b719640 1 -- 172.21.15.116:0/2363041808 wait complete. 2024-04-25T00:01:50.867 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_metadata pg_num 2024-04-25T00:01:50.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.973+0000 7f42a6f07640 1 Processor -- start 2024-04-25T00:01:50.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.973+0000 7f42a6f07640 1 -- start start 2024-04-25T00:01:50.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.973+0000 7f42a6f07640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f42a00a9630 0x7f42a0156e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:50.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.973+0000 7f42a6f07640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f42a0157340 0x7f42a01577d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:50.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.973+0000 7f42a6f07640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42a00a9cb0 con 0x7f42a0150d60 2024-04-25T00:01:50.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.973+0000 7f42a6f07640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f42a00a9f90 con 0x7f42a00a9630 2024-04-25T00:01:50.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.973+0000 7f42a6f07640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f42a00aa270 con 0x7f42a0157340 2024-04-25T00:01:50.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.973+0000 7f429ffff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f42a00a9630 0x7f42a0156e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:50.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.973+0000 7f429ffff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f42a00a9630 0x7f42a0156e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43434/0 (socket says 172.21.15.116:43434) 2024-04-25T00:01:50.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.973+0000 7f429ffff640 1 -- 172.21.15.116:0/377588930 learned_addr learned my addr 172.21.15.116:0/377588930 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:50.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.973+0000 7f42a54a4640 1 --2- 172.21.15.116:0/377588930 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f42a0157340 0x7f42a01577d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:50.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.973+0000 7f429ffff640 1 -- 172.21.15.116:0/377588930 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f42a0157340 msgr2=0x7f42a01577d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:50.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.973+0000 7f429ffff640 1 --2- 172.21.15.116:0/377588930 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f42a0157340 0x7f42a01577d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:50.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.973+0000 7f429ffff640 1 -- 172.21.15.116:0/377588930 >> v1:172.21.15.174:6789/0 conn(0x7f42a0150d60 legacy=0x7f42a00c2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:50.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.973+0000 7f429ffff640 1 -- 172.21.15.116:0/377588930 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42a00aa550 con 0x7f42a00a9630 2024-04-25T00:01:50.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.973+0000 7f429ffff640 1 --2- 172.21.15.116:0/377588930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42a00a9630 0x7f42a0156e00 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f4294009870 tx=0x7f429401c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8a659601fd5540e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:50.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.977+0000 7f429f7fe640 1 -- 172.21.15.116:0/377588930 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42940b0070 con 0x7f42a00a9630 2024-04-25T00:01:50.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.977+0000 7f429f7fe640 1 -- 172.21.15.116:0/377588930 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42940ab600 con 0x7f42a00a9630 2024-04-25T00:01:50.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.977+0000 7f429f7fe640 1 -- 172.21.15.116:0/377588930 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42940ab9d0 con 0x7f42a00a9630 2024-04-25T00:01:50.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.977+0000 7f42a6f07640 1 -- 172.21.15.116:0/377588930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42a00a9630 msgr2=0x7f42a0156e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:50.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.977+0000 7f42a6f07640 1 --2- 172.21.15.116:0/377588930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42a00a9630 0x7f42a0156e00 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f4294009870 tx=0x7f429401c410 comp rx=0 tx=0).stop 2024-04-25T00:01:50.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.977+0000 7f42a6f07640 1 -- 172.21.15.116:0/377588930 shutdown_connections 2024-04-25T00:01:50.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.977+0000 7f42a6f07640 1 --2- 172.21.15.116:0/377588930 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f42a0157340 0x7f42a01577d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:50.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.977+0000 7f42a6f07640 1 --2- 172.21.15.116:0/377588930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42a00a9630 0x7f42a0156e00 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:50.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.977+0000 7f42a6f07640 1 -- 172.21.15.116:0/377588930 >> 172.21.15.116:0/377588930 conn(0x7f42a00c3a20 msgr2=0x7f42a00c3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:50.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.977+0000 7f42a6f07640 1 -- 172.21.15.116:0/377588930 shutdown_connections 2024-04-25T00:01:50.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.977+0000 7f42a6f07640 1 -- 172.21.15.116:0/377588930 wait complete. 2024-04-25T00:01:50.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.977+0000 7f42a6f07640 1 Processor -- start 2024-04-25T00:01:50.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.977+0000 7f42a6f07640 1 -- start start 2024-04-25T00:01:50.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.977+0000 7f42a6f07640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f42a00a9630 0x7f42a00c5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:50.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.977+0000 7f42a6f07640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42a0150d60 0x7f42a0155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:50.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.977+0000 7f42a6f07640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f42a0157340 0x7f42a01564f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:50.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.977+0000 7f42a6f07640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f42a0071fd0 con 0x7f42a0150d60 2024-04-25T00:01:50.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.977+0000 7f42a6f07640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f42a00ab7a0 con 0x7f42a00a9630 2024-04-25T00:01:50.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.977+0000 7f42a6f07640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f42a0072150 con 0x7f42a0157340 2024-04-25T00:01:50.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.977+0000 7f429ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42a0150d60 0x7f42a0155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:50.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.977+0000 7f429ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42a0150d60 0x7f42a0155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43446/0 (socket says 172.21.15.116:43446) 2024-04-25T00:01:50.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.977+0000 7f429ffff640 1 -- 172.21.15.116:0/3336920354 learned_addr learned my addr 172.21.15.116:0/3336920354 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:50.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.977+0000 7f429ffff640 1 -- 172.21.15.116:0/3336920354 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f42a0157340 msgr2=0x7f42a01564f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:50.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.977+0000 7f429ffff640 1 --2- 172.21.15.116:0/3336920354 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f42a0157340 0x7f42a01564f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:50.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.977+0000 7f429ffff640 1 -- 172.21.15.116:0/3336920354 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f42a00a9630 msgr2=0x7f42a00c5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:50.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.977+0000 7f42a4ca3640 1 --2- 172.21.15.116:0/3336920354 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f42a00a9630 0x7f42a00c5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:50.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.977+0000 7f429ffff640 1 --2- 172.21.15.116:0/3336920354 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f42a00a9630 0x7f42a00c5520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:50.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.977+0000 7f429ffff640 1 -- 172.21.15.116:0/3336920354 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42a0155510 con 0x7f42a0150d60 2024-04-25T00:01:50.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.977+0000 7f429ffff640 1 --2- 172.21.15.116:0/3336920354 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42a0150d60 0x7f42a0155fb0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f429401aac0 tx=0x7f42940191b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:50.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.977+0000 7f429d7fa640 1 -- 172.21.15.116:0/3336920354 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42940b9070 con 0x7f42a0150d60 2024-04-25T00:01:50.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.977+0000 7f429d7fa640 1 -- 172.21.15.116:0/3336920354 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42940b4420 con 0x7f42a0150d60 2024-04-25T00:01:50.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.977+0000 7f42a6f07640 1 -- 172.21.15.116:0/3336920354 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42a0140d10 con 0x7f42a0150d60 2024-04-25T00:01:50.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.977+0000 7f429d7fa640 1 -- 172.21.15.116:0/3336920354 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42940b0040 con 0x7f42a0150d60 2024-04-25T00:01:50.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.977+0000 7f42a6f07640 1 -- 172.21.15.116:0/3336920354 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42a01557b0 con 0x7f42a0150d60 2024-04-25T00:01:50.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.981+0000 7f42a6f07640 1 -- 172.21.15.116:0/3336920354 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42a004f510 con 0x7f42a0150d60 2024-04-25T00:01:50.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.981+0000 7f429d7fa640 1 -- 172.21.15.116:0/3336920354 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f42940bc070 con 0x7f42a0150d60 2024-04-25T00:01:50.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.981+0000 7f429d7fa640 1 --2- 172.21.15.116:0/3336920354 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f428c0678c0 0x7f428c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:50.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.981+0000 7f429d7fa640 1 -- 172.21.15.116:0/3336920354 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6666+0+0 (secure 0 0 0) 0x7f429411f0f0 con 0x7f42a0150d60 2024-04-25T00:01:50.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.981+0000 7f42a4ca3640 1 --2- 172.21.15.116:0/3336920354 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f428c0678c0 0x7f428c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:50.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.981+0000 7f42a4ca3640 1 --2- 172.21.15.116:0/3336920354 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f428c0678c0 0x7f428c069d80 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4290001cc0 tx=0x7f4290009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:50.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:50.985+0000 7f429d7fa640 1 -- 172.21.15.116:0/3336920354 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f4294016150 con 0x7f42a0150d60 2024-04-25T00:01:51.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.133+0000 7f42a6f07640 1 -- 172.21.15.116:0/3336920354 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"} v 0) -- 0x7f42a0071db0 con 0x7f42a0150d60 2024-04-25T00:01:51.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.133+0000 7f429d7fa640 1 -- 172.21.15.116:0/3336920354 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"}]=0 v29) ==== 99+0+11 (secure 0 0 0) 0x7f42940ef900 con 0x7f42a0150d60 2024-04-25T00:01:51.136 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-04-25T00:01:51.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.133+0000 7f42a6f07640 1 -- 172.21.15.116:0/3336920354 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f428c0678c0 msgr2=0x7f428c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:51.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.133+0000 7f42a6f07640 1 --2- 172.21.15.116:0/3336920354 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f428c0678c0 0x7f428c069d80 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4290001cc0 tx=0x7f4290009040 comp rx=0 tx=0).stop 2024-04-25T00:01:51.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.133+0000 7f42a6f07640 1 -- 172.21.15.116:0/3336920354 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42a0150d60 msgr2=0x7f42a0155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:51.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.133+0000 7f42a6f07640 1 --2- 172.21.15.116:0/3336920354 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42a0150d60 0x7f42a0155fb0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f429401aac0 tx=0x7f42940191b0 comp rx=0 tx=0).stop 2024-04-25T00:01:51.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.133+0000 7f42a6f07640 1 -- 172.21.15.116:0/3336920354 shutdown_connections 2024-04-25T00:01:51.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.133+0000 7f42a6f07640 1 --2- 172.21.15.116:0/3336920354 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f428c0678c0 0x7f428c069d80 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:51.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.133+0000 7f42a6f07640 1 --2- 172.21.15.116:0/3336920354 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f42a0157340 0x7f42a01564f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:51.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.133+0000 7f42a6f07640 1 --2- 172.21.15.116:0/3336920354 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42a0150d60 0x7f42a0155fb0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:51.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.133+0000 7f42a6f07640 1 --2- 172.21.15.116:0/3336920354 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f42a00a9630 0x7f42a00c5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:51.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.133+0000 7f42a6f07640 1 -- 172.21.15.116:0/3336920354 >> 172.21.15.116:0/3336920354 conn(0x7f42a00c3a20 msgr2=0x7f42a0072360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:51.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.133+0000 7f42a6f07640 1 -- 172.21.15.116:0/3336920354 shutdown_connections 2024-04-25T00:01:51.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.133+0000 7f42a6f07640 1 -- 172.21.15.116:0/3336920354 wait complete. 2024-04-25T00:01:51.153 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_data pg_num 2024-04-25T00:01:51.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.273+0000 7ffbbf344640 1 Processor -- start 2024-04-25T00:01:51.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.273+0000 7ffbbf344640 1 -- start start 2024-04-25T00:01:51.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.273+0000 7ffbbf344640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ffbb8153130 0x7ffbb81562e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:51.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.273+0000 7ffbbf344640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffbb80c9260 0x7ffbb80c9710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:51.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.273+0000 7ffbbf344640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffbb8156820 con 0x7ffbb80ca1b0 2024-04-25T00:01:51.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.273+0000 7ffbbf344640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7ffbb80ca820 con 0x7ffbb8153130 2024-04-25T00:01:51.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.273+0000 7ffbbf344640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ffbb8155e50 con 0x7ffbb80c9260 2024-04-25T00:01:51.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.273+0000 7ffbbc8df640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7ffbb80ca1b0 0x7ffbb806bdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:47590/0 (socket says 172.21.15.116:47590) 2024-04-25T00:01:51.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.273+0000 7ffbbc8df640 1 -- 172.21.15.116:0/1482365788 learned_addr learned my addr 172.21.15.116:0/1482365788 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:51.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.273+0000 7ffbbd8e1640 1 --2- 172.21.15.116:0/1482365788 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffbb80c9260 0x7ffbb80c9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:51.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.273+0000 7ffbbd0e0640 1 --2- 172.21.15.116:0/1482365788 >> v2:172.21.15.174:3300/0 conn(0x7ffbb8153130 0x7ffbb81562e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:51.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.273+0000 7ffbaffff640 1 -- 172.21.15.116:0/1482365788 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2555035432 0 0) 0x7ffbb8156820 con 0x7ffbb80ca1b0 2024-04-25T00:01:51.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.273+0000 7ffbaffff640 1 -- 172.21.15.116:0/1482365788 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffba4003530 con 0x7ffbb80ca1b0 2024-04-25T00:01:51.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.273+0000 7ffbaffff640 1 -- 172.21.15.116:0/1482365788 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1510039892 0 0) 0x7ffba4003530 con 0x7ffbb80ca1b0 2024-04-25T00:01:51.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.273+0000 7ffbaffff640 1 -- 172.21.15.116:0/1482365788 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffbb80c9260 msgr2=0x7ffbb80c9710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:51.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.273+0000 7ffbaffff640 1 --2- 172.21.15.116:0/1482365788 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffbb80c9260 0x7ffbb80c9710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:51.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.273+0000 7ffbaffff640 1 -- 172.21.15.116:0/1482365788 >> v2:172.21.15.174:3300/0 conn(0x7ffbb8153130 msgr2=0x7ffbb81562e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:51.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.273+0000 7ffbaffff640 1 --2- 172.21.15.116:0/1482365788 >> v2:172.21.15.174:3300/0 conn(0x7ffbb8153130 0x7ffbb81562e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:51.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.273+0000 7ffbaffff640 1 -- 172.21.15.116:0/1482365788 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbb80caa70 con 0x7ffbb80ca1b0 2024-04-25T00:01:51.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.273+0000 7ffbaffff640 1 -- 172.21.15.116:0/1482365788 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7ffbb4001680 con 0x7ffbb80ca1b0 2024-04-25T00:01:51.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.273+0000 7ffbaffff640 1 -- 172.21.15.116:0/1482365788 >> v1:172.21.15.116:6789/0 conn(0x7ffbb80ca1b0 legacy=0x7ffbb806bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:51.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.273+0000 7ffbaffff640 1 --2- 172.21.15.116:0/1482365788 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffbb80c9260 0x7ffba40039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:51.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.273+0000 7ffbaffff640 1 --2- 172.21.15.116:0/1482365788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffba4003f40 0x7ffba4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:51.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.273+0000 7ffbaffff640 1 --2- 172.21.15.116:0/1482365788 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffba4004d50 0x7ffba40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:51.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.273+0000 7ffbaffff640 1 -- 172.21.15.116:0/1482365788 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ffbb80ca820 con 0x7ffba4003f40 2024-04-25T00:01:51.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.273+0000 7ffbaffff640 1 -- 172.21.15.116:0/1482365788 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ffbb8155e50 con 0x7ffba4004d50 2024-04-25T00:01:51.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.273+0000 7ffbaffff640 1 -- 172.21.15.116:0/1482365788 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ffba400bab0 con 0x7ffbb80c9260 2024-04-25T00:01:51.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.273+0000 7ffbbc8df640 1 --2- 172.21.15.116:0/1482365788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffba4003f40 0x7ffba4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:51.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.273+0000 7ffbbd0e0640 1 --2- 172.21.15.116:0/1482365788 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffbb80c9260 0x7ffba40039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:51.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.273+0000 7ffbbd8e1640 1 --2- 172.21.15.116:0/1482365788 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffba4004d50 0x7ffba40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:51.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.273+0000 7ffbbc8df640 1 -- 172.21.15.116:0/1482365788 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffbb80c9260 msgr2=0x7ffba40039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:51.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.273+0000 7ffbbc8df640 1 --2- 172.21.15.116:0/1482365788 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffbb80c9260 0x7ffba40039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:51.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.273+0000 7ffbbc8df640 1 -- 172.21.15.116:0/1482365788 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffba4004d50 msgr2=0x7ffba40051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:51.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.273+0000 7ffbbc8df640 1 --2- 172.21.15.116:0/1482365788 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffba4004d50 0x7ffba40051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:51.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.273+0000 7ffbbc8df640 1 -- 172.21.15.116:0/1482365788 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffba4003530 con 0x7ffba4003f40 2024-04-25T00:01:51.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.273+0000 7ffbbd8e1640 1 --2- 172.21.15.116:0/1482365788 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffba4004d50 0x7ffba40051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:51.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.277+0000 7ffbbc8df640 1 --2- 172.21.15.116:0/1482365788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffba4003f40 0x7ffba4004810 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7ffbb4001060 tx=0x7ffbb401cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36443a5c9996ef33 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:51.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.277+0000 7ffbaffff640 1 -- 172.21.15.116:0/1482365788 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb4007d80 con 0x7ffba4003f40 2024-04-25T00:01:51.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.277+0000 7ffbaffff640 1 -- 172.21.15.116:0/1482365788 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffbb40a9020 con 0x7ffba4003f40 2024-04-25T00:01:51.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.277+0000 7ffbaffff640 1 -- 172.21.15.116:0/1482365788 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbb4005ae0 con 0x7ffba4003f40 2024-04-25T00:01:51.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.277+0000 7ffbbf344640 1 -- 172.21.15.116:0/1482365788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffba4003f40 msgr2=0x7ffba4004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:51.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.277+0000 7ffbbf344640 1 --2- 172.21.15.116:0/1482365788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffba4003f40 0x7ffba4004810 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7ffbb4001060 tx=0x7ffbb401cc80 comp rx=0 tx=0).stop 2024-04-25T00:01:51.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.277+0000 7ffbbd8e1640 1 -- 172.21.15.116:0/1482365788 reap_dead start 2024-04-25T00:01:51.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.277+0000 7ffbbf344640 1 -- 172.21.15.116:0/1482365788 shutdown_connections 2024-04-25T00:01:51.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.277+0000 7ffbbf344640 1 -- 172.21.15.116:0/1482365788 >> 172.21.15.116:0/1482365788 conn(0x7ffbb80cb310 msgr2=0x7ffbb80cb720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:51.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.277+0000 7ffbbf344640 1 -- 172.21.15.116:0/1482365788 shutdown_connections 2024-04-25T00:01:51.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.277+0000 7ffbbf344640 1 -- 172.21.15.116:0/1482365788 wait complete. 2024-04-25T00:01:51.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.277+0000 7ffbbf344640 1 Processor -- start 2024-04-25T00:01:51.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.277+0000 7ffbbf344640 1 -- start start 2024-04-25T00:01:51.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.277+0000 7ffbbf344640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffba4003f40 0x7ffbb815b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:51.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.281+0000 7ffbbf344640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbb814d930 0x7ffbb815c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:51.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.281+0000 7ffbbf344640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffbb815caf0 0x7ffbb81f1a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:51.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.281+0000 7ffbbf344640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ffbb81602f0 con 0x7ffba4003f40 2024-04-25T00:01:51.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.281+0000 7ffbbf344640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ffbb8160170 con 0x7ffbb814d930 2024-04-25T00:01:51.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.281+0000 7ffbbf344640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ffbb8160470 con 0x7ffbb815caf0 2024-04-25T00:01:51.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.281+0000 7ffbbd0e0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffba4003f40 0x7ffbb815b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:51.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.281+0000 7ffbbd0e0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffba4003f40 0x7ffbb815b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43472/0 (socket says 172.21.15.116:43472) 2024-04-25T00:01:51.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.281+0000 7ffbbd0e0640 1 -- 172.21.15.116:0/3684147254 learned_addr learned my addr 172.21.15.116:0/3684147254 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:51.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.281+0000 7ffbbc8df640 1 --2- 172.21.15.116:0/3684147254 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbb814d930 0x7ffbb815c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:51.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.281+0000 7ffbbd8e1640 1 --2- 172.21.15.116:0/3684147254 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffbb815caf0 0x7ffbb81f1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:51.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.281+0000 7ffbbd0e0640 1 -- 172.21.15.116:0/3684147254 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffbb815caf0 msgr2=0x7ffbb81f1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:51.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.281+0000 7ffbbd0e0640 1 --2- 172.21.15.116:0/3684147254 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffbb815caf0 0x7ffbb81f1a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:51.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.281+0000 7ffbbd0e0640 1 -- 172.21.15.116:0/3684147254 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbb814d930 msgr2=0x7ffbb815c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:51.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.281+0000 7ffbbd0e0640 1 --2- 172.21.15.116:0/3684147254 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbb814d930 0x7ffbb815c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:51.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.281+0000 7ffbbd0e0640 1 -- 172.21.15.116:0/3684147254 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbb814ce70 con 0x7ffba4003f40 2024-04-25T00:01:51.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.281+0000 7ffbbd8e1640 1 --2- 172.21.15.116:0/3684147254 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffbb815caf0 0x7ffbb81f1a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:51.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.281+0000 7ffbbc8df640 1 --2- 172.21.15.116:0/3684147254 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbb814d930 0x7ffbb815c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:51.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.281+0000 7ffbbd0e0640 1 --2- 172.21.15.116:0/3684147254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffba4003f40 0x7ffbb815b560 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7ffba8011610 tx=0x7ffba8011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:51.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.281+0000 7ffbadffb640 1 -- 172.21.15.116:0/3684147254 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffba800f8f0 con 0x7ffba4003f40 2024-04-25T00:01:51.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.281+0000 7ffbadffb640 1 -- 172.21.15.116:0/3684147254 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffba800fa90 con 0x7ffba4003f40 2024-04-25T00:01:51.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.281+0000 7ffbbf344640 1 -- 172.21.15.116:0/3684147254 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbb814c500 con 0x7ffba4003f40 2024-04-25T00:01:51.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.281+0000 7ffbadffb640 1 -- 172.21.15.116:0/3684147254 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffba800fe40 con 0x7ffba4003f40 2024-04-25T00:01:51.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.281+0000 7ffbbf344640 1 -- 172.21.15.116:0/3684147254 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbb815b040 con 0x7ffba4003f40 2024-04-25T00:01:51.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.281+0000 7ffbbf344640 1 -- 172.21.15.116:0/3684147254 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffbb815ab50 con 0x7ffba4003f40 2024-04-25T00:01:51.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.281+0000 7ffbadffb640 1 -- 172.21.15.116:0/3684147254 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ffba8015460 con 0x7ffba4003f40 2024-04-25T00:01:51.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.281+0000 7ffbadffb640 1 --2- 172.21.15.116:0/3684147254 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ffb9c0678f0 0x7ffb9c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:51.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.281+0000 7ffbadffb640 1 -- 172.21.15.116:0/3684147254 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6666+0+0 (secure 0 0 0) 0x7ffba8086a80 con 0x7ffba4003f40 2024-04-25T00:01:51.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.281+0000 7ffbbc8df640 1 --2- 172.21.15.116:0/3684147254 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ffb9c0678f0 0x7ffb9c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:51.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.285+0000 7ffbbc8df640 1 --2- 172.21.15.116:0/3684147254 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ffb9c0678f0 0x7ffb9c069db0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ffbb40aa000 tx=0x7ffbb4001a50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:51.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.285+0000 7ffbadffb640 1 -- 172.21.15.116:0/3684147254 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ffba8019030 con 0x7ffba4003f40 2024-04-25T00:01:51.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.437+0000 7ffbbf344640 1 -- 172.21.15.116:0/3684147254 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"} v 0) -- 0x7ffbb804f510 con 0x7ffba4003f40 2024-04-25T00:01:51.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.437+0000 7ffbadffb640 1 -- 172.21.15.116:0/3684147254 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"}]=0 v29) ==== 95+0+11 (secure 0 0 0) 0x7ffba8057170 con 0x7ffba4003f40 2024-04-25T00:01:51.440 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-04-25T00:01:51.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.437+0000 7ffbbf344640 1 -- 172.21.15.116:0/3684147254 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ffb9c0678f0 msgr2=0x7ffb9c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:51.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.437+0000 7ffbbf344640 1 --2- 172.21.15.116:0/3684147254 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ffb9c0678f0 0x7ffb9c069db0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ffbb40aa000 tx=0x7ffbb4001a50 comp rx=0 tx=0).stop 2024-04-25T00:01:51.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.437+0000 7ffbbf344640 1 -- 172.21.15.116:0/3684147254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffba4003f40 msgr2=0x7ffbb815b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:51.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.437+0000 7ffbbf344640 1 --2- 172.21.15.116:0/3684147254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffba4003f40 0x7ffbb815b560 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7ffba8011610 tx=0x7ffba8011ad0 comp rx=0 tx=0).stop 2024-04-25T00:01:51.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.437+0000 7ffbbf344640 1 -- 172.21.15.116:0/3684147254 shutdown_connections 2024-04-25T00:01:51.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.437+0000 7ffbbf344640 1 --2- 172.21.15.116:0/3684147254 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ffb9c0678f0 0x7ffb9c069db0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:51.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.437+0000 7ffbbf344640 1 --2- 172.21.15.116:0/3684147254 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffbb815caf0 0x7ffbb81f1a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:51.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.437+0000 7ffbbf344640 1 --2- 172.21.15.116:0/3684147254 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbb814d930 0x7ffbb815c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:51.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.437+0000 7ffbbf344640 1 --2- 172.21.15.116:0/3684147254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffba4003f40 0x7ffbb815b560 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:51.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.437+0000 7ffbbf344640 1 -- 172.21.15.116:0/3684147254 >> 172.21.15.116:0/3684147254 conn(0x7ffbb80cb310 msgr2=0x7ffbb81511d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:51.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.437+0000 7ffbbf344640 1 -- 172.21.15.116:0/3684147254 shutdown_connections 2024-04-25T00:01:51.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.437+0000 7ffbbf344640 1 -- 172.21.15.116:0/3684147254 wait complete. 2024-04-25T00:01:51.458 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_metadata pg_num 2024-04-25T00:01:51.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.569+0000 7f284583f640 1 Processor -- start 2024-04-25T00:01:51.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.569+0000 7f284583f640 1 -- start start 2024-04-25T00:01:51.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.573+0000 7f284583f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2840150d60 0x7f28400c2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:51.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.573+0000 7f284583f640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f28400c2e70 0x7f2840157510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:51.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.573+0000 7f284583f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28400a9cb0 con 0x7f28400a9630 2024-04-25T00:01:51.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.573+0000 7f284583f640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f28400a9f90 con 0x7f28400c2e70 2024-04-25T00:01:51.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.573+0000 7f284583f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f28400aa270 con 0x7f2840150d60 2024-04-25T00:01:51.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.573+0000 7f28367fc640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f28400a9630 0x7f2840156e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:47604/0 (socket says 172.21.15.116:47604) 2024-04-25T00:01:51.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.573+0000 7f28367fc640 1 -- 172.21.15.116:0/3297896520 learned_addr learned my addr 172.21.15.116:0/3297896520 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:51.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.573+0000 7f283e7fc640 1 -- 172.21.15.116:0/3297896520 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4139941862 0 0) 0x7f28400a9cb0 con 0x7f28400a9630 2024-04-25T00:01:51.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.573+0000 7f283effd640 1 --2- 172.21.15.116:0/3297896520 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2840150d60 0x7f28400c2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:51.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.573+0000 7f283f7fe640 1 --2- 172.21.15.116:0/3297896520 >> v2:172.21.15.174:3300/0 conn(0x7f28400c2e70 0x7f2840157510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:51.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.573+0000 7f283e7fc640 1 -- 172.21.15.116:0/3297896520 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f282c003530 con 0x7f28400a9630 2024-04-25T00:01:51.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.573+0000 7f283e7fc640 1 -- 172.21.15.116:0/3297896520 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2674103317 0 0) 0x7f282c003530 con 0x7f28400a9630 2024-04-25T00:01:51.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.573+0000 7f283e7fc640 1 -- 172.21.15.116:0/3297896520 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2840150d60 msgr2=0x7f28400c2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:51.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.573+0000 7f283e7fc640 1 --2- 172.21.15.116:0/3297896520 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2840150d60 0x7f28400c2930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:51.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.573+0000 7f283e7fc640 1 -- 172.21.15.116:0/3297896520 >> v2:172.21.15.174:3300/0 conn(0x7f28400c2e70 msgr2=0x7f2840157510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:51.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.573+0000 7f283e7fc640 1 --2- 172.21.15.116:0/3297896520 >> v2:172.21.15.174:3300/0 conn(0x7f28400c2e70 0x7f2840157510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:51.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.573+0000 7f283e7fc640 1 -- 172.21.15.116:0/3297896520 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28400aa550 con 0x7f28400a9630 2024-04-25T00:01:51.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.573+0000 7f283e7fc640 1 -- 172.21.15.116:0/3297896520 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f2828001680 con 0x7f28400a9630 2024-04-25T00:01:51.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.573+0000 7f283e7fc640 1 -- 172.21.15.116:0/3297896520 >> v1:172.21.15.116:6789/0 conn(0x7f28400a9630 legacy=0x7f2840156e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:51.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.573+0000 7f283e7fc640 1 --2- 172.21.15.116:0/3297896520 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f282c0039c0 0x7f282c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:51.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.573+0000 7f283e7fc640 1 --2- 172.21.15.116:0/3297896520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f282c004810 0x7f282c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:51.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.573+0000 7f283e7fc640 1 --2- 172.21.15.116:0/3297896520 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2840150d60 0x7f282c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:51.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.573+0000 7f283e7fc640 1 -- 172.21.15.116:0/3297896520 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f28400a9f90 con 0x7f282c004810 2024-04-25T00:01:51.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.573+0000 7f283e7fc640 1 -- 172.21.15.116:0/3297896520 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f28400aa270 con 0x7f282c0039c0 2024-04-25T00:01:51.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.573+0000 7f283e7fc640 1 -- 172.21.15.116:0/3297896520 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f282c00bb00 con 0x7f2840150d60 2024-04-25T00:01:51.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.573+0000 7f28367fc640 1 --2- 172.21.15.116:0/3297896520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f282c004810 0x7f282c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:51.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.573+0000 7f283effd640 1 --2- 172.21.15.116:0/3297896520 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f282c0039c0 0x7f282c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:51.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.573+0000 7f283f7fe640 1 --2- 172.21.15.116:0/3297896520 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2840150d60 0x7f282c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:51.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.573+0000 7f28367fc640 1 -- 172.21.15.116:0/3297896520 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2840150d60 msgr2=0x7f282c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:51.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.573+0000 7f28367fc640 1 --2- 172.21.15.116:0/3297896520 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2840150d60 0x7f282c0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:51.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.573+0000 7f28367fc640 1 -- 172.21.15.116:0/3297896520 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f282c0039c0 msgr2=0x7f282c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:51.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.573+0000 7f28367fc640 1 --2- 172.21.15.116:0/3297896520 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f282c0039c0 0x7f282c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:51.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.573+0000 7f28367fc640 1 -- 172.21.15.116:0/3297896520 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f282c003530 con 0x7f282c004810 2024-04-25T00:01:51.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.577+0000 7f28367fc640 1 --2- 172.21.15.116:0/3297896520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f282c004810 0x7f282c004c90 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f282800c210 tx=0x7f282801cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9de0d287aebae0cd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:51.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.577+0000 7f283e7fc640 1 -- 172.21.15.116:0/3297896520 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2828007d60 con 0x7f282c004810 2024-04-25T00:01:51.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.577+0000 7f283e7fc640 1 -- 172.21.15.116:0/3297896520 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28280a9020 con 0x7f282c004810 2024-04-25T00:01:51.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.577+0000 7f283e7fc640 1 -- 172.21.15.116:0/3297896520 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2828005b20 con 0x7f282c004810 2024-04-25T00:01:51.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.577+0000 7f284583f640 1 -- 172.21.15.116:0/3297896520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f282c004810 msgr2=0x7f282c004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:51.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.577+0000 7f284583f640 1 --2- 172.21.15.116:0/3297896520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f282c004810 0x7f282c004c90 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f282800c210 tx=0x7f282801cc70 comp rx=0 tx=0).stop 2024-04-25T00:01:51.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.577+0000 7f283f7fe640 1 -- 172.21.15.116:0/3297896520 reap_dead start 2024-04-25T00:01:51.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.577+0000 7f284583f640 1 -- 172.21.15.116:0/3297896520 shutdown_connections 2024-04-25T00:01:51.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.577+0000 7f284583f640 1 -- 172.21.15.116:0/3297896520 >> 172.21.15.116:0/3297896520 conn(0x7f28400c3a20 msgr2=0x7f28400c3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:51.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.577+0000 7f284583f640 1 -- 172.21.15.116:0/3297896520 shutdown_connections 2024-04-25T00:01:51.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.577+0000 7f284583f640 1 -- 172.21.15.116:0/3297896520 wait complete. 2024-04-25T00:01:51.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.577+0000 7f284583f640 1 Processor -- start 2024-04-25T00:01:51.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.577+0000 7f284583f640 1 -- start start 2024-04-25T00:01:51.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.577+0000 7f284583f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f282c004810 0x7f28401ecca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:51.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.577+0000 7f284583f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28401522a0 0x7f28401efe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:51.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.577+0000 7f284583f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28400c6820 0x7f28400c4b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:51.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.577+0000 7f284583f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2840072150 con 0x7f28401522a0 2024-04-25T00:01:51.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.577+0000 7f284583f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2840071fd0 con 0x7f282c004810 2024-04-25T00:01:51.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.577+0000 7f284583f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f284015e760 con 0x7f28400c6820 2024-04-25T00:01:51.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.577+0000 7f28367fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28401522a0 0x7f28401efe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:51.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.577+0000 7f28367fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28401522a0 0x7f28401efe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43490/0 (socket says 172.21.15.116:43490) 2024-04-25T00:01:51.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.577+0000 7f28367fc640 1 -- 172.21.15.116:0/57653301 learned_addr learned my addr 172.21.15.116:0/57653301 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:51.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.577+0000 7f283f7fe640 1 --2- 172.21.15.116:0/57653301 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28400c6820 0x7f28400c4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:51.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.577+0000 7f283effd640 1 --2- 172.21.15.116:0/57653301 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f282c004810 0x7f28401ecca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:51.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.577+0000 7f28367fc640 1 -- 172.21.15.116:0/57653301 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28400c6820 msgr2=0x7f28400c4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:51.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.577+0000 7f28367fc640 1 --2- 172.21.15.116:0/57653301 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28400c6820 0x7f28400c4b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:51.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.577+0000 7f28367fc640 1 -- 172.21.15.116:0/57653301 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f282c004810 msgr2=0x7f28401ecca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:51.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.577+0000 7f28367fc640 1 --2- 172.21.15.116:0/57653301 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f282c004810 0x7f28401ecca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:51.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.577+0000 7f28367fc640 1 -- 172.21.15.116:0/57653301 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2840151e80 con 0x7f28401522a0 2024-04-25T00:01:51.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.577+0000 7f283effd640 1 --2- 172.21.15.116:0/57653301 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f282c004810 0x7f28401ecca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:51.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.581+0000 7f28367fc640 1 --2- 172.21.15.116:0/57653301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28401522a0 0x7f28401efe70 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f28280ab040 tx=0x7f282c005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:51.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.581+0000 7f283cff9640 1 -- 172.21.15.116:0/57653301 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28280a5a60 con 0x7f28401522a0 2024-04-25T00:01:51.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.581+0000 7f283cff9640 1 -- 172.21.15.116:0/57653301 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28280b3070 con 0x7f28401522a0 2024-04-25T00:01:51.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.581+0000 7f283cff9640 1 -- 172.21.15.116:0/57653301 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28280a5e20 con 0x7f28401522a0 2024-04-25T00:01:51.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.581+0000 7f284583f640 1 -- 172.21.15.116:0/57653301 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28400c5c40 con 0x7f28401522a0 2024-04-25T00:01:51.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.581+0000 7f284583f640 1 -- 172.21.15.116:0/57653301 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28400c6200 con 0x7f28401522a0 2024-04-25T00:01:51.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.581+0000 7f284583f640 1 -- 172.21.15.116:0/57653301 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28400c6400 con 0x7f28401522a0 2024-04-25T00:01:51.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.581+0000 7f283cff9640 1 -- 172.21.15.116:0/57653301 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f28280047f0 con 0x7f28401522a0 2024-04-25T00:01:51.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.581+0000 7f283cff9640 1 --2- 172.21.15.116:0/57653301 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f28200678f0 0x7f2820069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:51.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.581+0000 7f283cff9640 1 -- 172.21.15.116:0/57653301 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6666+0+0 (secure 0 0 0) 0x7f282811f990 con 0x7f28401522a0 2024-04-25T00:01:51.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.581+0000 7f283effd640 1 --2- 172.21.15.116:0/57653301 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f28200678f0 0x7f2820069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:51.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.581+0000 7f283effd640 1 --2- 172.21.15.116:0/57653301 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f28200678f0 0x7f2820069db0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2840072c40 tx=0x7f2830008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:51.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.585+0000 7f283cff9640 1 -- 172.21.15.116:0/57653301 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f28400c6400 con 0x7f28401522a0 2024-04-25T00:01:51.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.733+0000 7f284583f640 1 -- 172.21.15.116:0/57653301 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"} v 0) -- 0x7f2840071db0 con 0x7f28401522a0 2024-04-25T00:01:51.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.733+0000 7f283cff9640 1 -- 172.21.15.116:0/57653301 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"}]=0 v29) ==== 99+0+11 (secure 0 0 0) 0x7f28280169b0 con 0x7f28401522a0 2024-04-25T00:01:51.738 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-04-25T00:01:51.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.737+0000 7f284583f640 1 -- 172.21.15.116:0/57653301 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f28200678f0 msgr2=0x7f2820069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:51.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.737+0000 7f284583f640 1 --2- 172.21.15.116:0/57653301 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f28200678f0 0x7f2820069db0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2840072c40 tx=0x7f2830008040 comp rx=0 tx=0).stop 2024-04-25T00:01:51.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.737+0000 7f284583f640 1 -- 172.21.15.116:0/57653301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28401522a0 msgr2=0x7f28401efe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:51.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.737+0000 7f284583f640 1 --2- 172.21.15.116:0/57653301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28401522a0 0x7f28401efe70 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f28280ab040 tx=0x7f282c005840 comp rx=0 tx=0).stop 2024-04-25T00:01:51.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.737+0000 7f284583f640 1 -- 172.21.15.116:0/57653301 shutdown_connections 2024-04-25T00:01:51.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.737+0000 7f284583f640 1 --2- 172.21.15.116:0/57653301 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f28200678f0 0x7f2820069db0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:51.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.737+0000 7f284583f640 1 --2- 172.21.15.116:0/57653301 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28400c6820 0x7f28400c4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:51.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.737+0000 7f284583f640 1 --2- 172.21.15.116:0/57653301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28401522a0 0x7f28401efe70 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:51.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.737+0000 7f284583f640 1 --2- 172.21.15.116:0/57653301 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f282c004810 0x7f28401ecca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:51.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.737+0000 7f284583f640 1 -- 172.21.15.116:0/57653301 >> 172.21.15.116:0/57653301 conn(0x7f28400c3a20 msgr2=0x7f28400abf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:51.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.737+0000 7f284583f640 1 -- 172.21.15.116:0/57653301 shutdown_connections 2024-04-25T00:01:51.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.737+0000 7f284583f640 1 -- 172.21.15.116:0/57653301 wait complete. 2024-04-25T00:01:51.755 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_data pg_num 2024-04-25T00:01:51.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.865+0000 7fc5380b4640 1 Processor -- start 2024-04-25T00:01:51.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.865+0000 7fc5380b4640 1 -- start start 2024-04-25T00:01:51.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.865+0000 7fc5380b4640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc530148f20 0x7fc5300c2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:51.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.865+0000 7fc5380b4640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fc53006d000 0x7fc53015cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:51.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.865+0000 7fc5380b4640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc530149300 con 0x7fc53006d510 2024-04-25T00:01:51.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.865+0000 7fc5380b4640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fc530153350 con 0x7fc530148f20 2024-04-25T00:01:51.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.865+0000 7fc5380b4640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fc5300a95d0 con 0x7fc53006d000 2024-04-25T00:01:51.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.865+0000 7fc535e50640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc530148f20 0x7fc5300c2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:51.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.865+0000 7fc535e50640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc530148f20 0x7fc5300c2930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43494/0 (socket says 172.21.15.116:43494) 2024-04-25T00:01:51.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.865+0000 7fc535e50640 1 -- 172.21.15.116:0/3688652153 learned_addr learned my addr 172.21.15.116:0/3688652153 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:51.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.865+0000 7fc535e50640 1 -- 172.21.15.116:0/3688652153 >> v2:172.21.15.174:3300/0 conn(0x7fc53006d000 msgr2=0x7fc53015cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:51.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.865+0000 7fc536651640 1 --2- 172.21.15.116:0/3688652153 >> v2:172.21.15.174:3300/0 conn(0x7fc53006d000 0x7fc53015cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:51.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.865+0000 7fc535e50640 1 --2- 172.21.15.116:0/3688652153 >> v2:172.21.15.174:3300/0 conn(0x7fc53006d000 0x7fc53015cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:51.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.865+0000 7fc535e50640 1 -- 172.21.15.116:0/3688652153 >> v1:172.21.15.174:6789/0 conn(0x7fc53006d510 legacy=0x7fc530072500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:51.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.865+0000 7fc535e50640 1 -- 172.21.15.116:0/3688652153 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc530153070 con 0x7fc530148f20 2024-04-25T00:01:51.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.869+0000 7fc535e50640 1 --2- 172.21.15.116:0/3688652153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc530148f20 0x7fc5300c2930 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fc53006c190 tx=0x7fc52001c1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=300845651de772c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:51.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.869+0000 7fc534e4e640 1 -- 172.21.15.116:0/3688652153 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5200b0070 con 0x7fc530148f20 2024-04-25T00:01:51.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.869+0000 7fc534e4e640 1 -- 172.21.15.116:0/3688652153 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5200ab600 con 0x7fc530148f20 2024-04-25T00:01:51.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.869+0000 7fc534e4e640 1 -- 172.21.15.116:0/3688652153 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5200ab9d0 con 0x7fc530148f20 2024-04-25T00:01:51.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.869+0000 7fc5380b4640 1 -- 172.21.15.116:0/3688652153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc530148f20 msgr2=0x7fc5300c2930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:51.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.869+0000 7fc5380b4640 1 --2- 172.21.15.116:0/3688652153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc530148f20 0x7fc5300c2930 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fc53006c190 tx=0x7fc52001c1e0 comp rx=0 tx=0).stop 2024-04-25T00:01:51.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.869+0000 7fc5380b4640 1 -- 172.21.15.116:0/3688652153 shutdown_connections 2024-04-25T00:01:51.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.869+0000 7fc5380b4640 1 --2- 172.21.15.116:0/3688652153 >> v2:172.21.15.174:3300/0 conn(0x7fc53006d000 0x7fc53015cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:51.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.869+0000 7fc5380b4640 1 --2- 172.21.15.116:0/3688652153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc530148f20 0x7fc5300c2930 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:51.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.869+0000 7fc5380b4640 1 -- 172.21.15.116:0/3688652153 >> 172.21.15.116:0/3688652153 conn(0x7fc5300c3a20 msgr2=0x7fc5300c3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:51.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.869+0000 7fc5380b4640 1 -- 172.21.15.116:0/3688652153 shutdown_connections 2024-04-25T00:01:51.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.869+0000 7fc5380b4640 1 -- 172.21.15.116:0/3688652153 wait complete. 2024-04-25T00:01:51.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.869+0000 7fc5380b4640 1 Processor -- start 2024-04-25T00:01:51.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.869+0000 7fc5380b4640 1 -- start start 2024-04-25T00:01:51.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.869+0000 7fc5380b4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc53006d000 0x7fc5300c05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:51.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.869+0000 7fc5380b4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc53006d510 0x7fc5301eed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:51.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.869+0000 7fc5380b4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc530148f20 0x7fc5301ef260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:51.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.869+0000 7fc5380b4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc53015dca0 con 0x7fc53006d000 2024-04-25T00:01:51.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.869+0000 7fc5380b4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc53015db20 con 0x7fc53006d510 2024-04-25T00:01:51.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.869+0000 7fc5380b4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fc53015de20 con 0x7fc530148f20 2024-04-25T00:01:51.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.869+0000 7fc535e50640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc53006d000 0x7fc5300c05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:51.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.869+0000 7fc535e50640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc53006d000 0x7fc5300c05b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43500/0 (socket says 172.21.15.116:43500) 2024-04-25T00:01:51.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.869+0000 7fc535e50640 1 -- 172.21.15.116:0/951942661 learned_addr learned my addr 172.21.15.116:0/951942661 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:51.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.869+0000 7fc535e50640 1 -- 172.21.15.116:0/951942661 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc530148f20 msgr2=0x7fc5301ef260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:51.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.869+0000 7fc535e50640 1 --2- 172.21.15.116:0/951942661 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc530148f20 0x7fc5301ef260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:51.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.869+0000 7fc535e50640 1 -- 172.21.15.116:0/951942661 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc53006d510 msgr2=0x7fc5301eed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:51.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.869+0000 7fc535e50640 1 --2- 172.21.15.116:0/951942661 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc53006d510 0x7fc5301eed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:51.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.869+0000 7fc535e50640 1 -- 172.21.15.116:0/951942661 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5301f2270 con 0x7fc53006d000 2024-04-25T00:01:51.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.869+0000 7fc535e50640 1 --2- 172.21.15.116:0/951942661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc53006d000 0x7fc5300c05b0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fc5200ae040 tx=0x7fc52001aa20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:51.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.869+0000 7fc5267fc640 1 -- 172.21.15.116:0/951942661 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5200ba020 con 0x7fc53006d000 2024-04-25T00:01:51.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.873+0000 7fc5267fc640 1 -- 172.21.15.116:0/951942661 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5200b6420 con 0x7fc53006d000 2024-04-25T00:01:51.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.873+0000 7fc5380b4640 1 -- 172.21.15.116:0/951942661 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5301f2550 con 0x7fc53006d000 2024-04-25T00:01:51.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.873+0000 7fc5267fc640 1 -- 172.21.15.116:0/951942661 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5200b0040 con 0x7fc53006d000 2024-04-25T00:01:51.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.873+0000 7fc5380b4640 1 -- 172.21.15.116:0/951942661 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5301f2b10 con 0x7fc53006d000 2024-04-25T00:01:51.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.873+0000 7fc5380b4640 1 -- 172.21.15.116:0/951942661 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc53004f510 con 0x7fc53006d000 2024-04-25T00:01:51.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.873+0000 7fc5267fc640 1 -- 172.21.15.116:0/951942661 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc5200bd020 con 0x7fc53006d000 2024-04-25T00:01:51.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.873+0000 7fc5267fc640 1 --2- 172.21.15.116:0/951942661 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc5140678c0 0x7fc514069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:51.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.873+0000 7fc5267fc640 1 -- 172.21.15.116:0/951942661 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6666+0+0 (secure 0 0 0) 0x7fc520120210 con 0x7fc53006d000 2024-04-25T00:01:51.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.873+0000 7fc53564f640 1 --2- 172.21.15.116:0/951942661 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc5140678c0 0x7fc514069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:51.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.873+0000 7fc53564f640 1 --2- 172.21.15.116:0/951942661 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc5140678c0 0x7fc514069d80 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc52c007fd0 tx=0x7fc52c0079a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:51.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:51.877+0000 7fc5267fc640 1 -- 172.21.15.116:0/951942661 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fc5200c4030 con 0x7fc53006d000 2024-04-25T00:01:52.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.025+0000 7fc5380b4640 1 -- 172.21.15.116:0/951942661 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"} v 0) -- 0x7fc5300c34b0 con 0x7fc53006d000 2024-04-25T00:01:52.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.025+0000 7fc5267fc640 1 -- 172.21.15.116:0/951942661 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"}]=0 v29) ==== 95+0+11 (secure 0 0 0) 0x7fc530149300 con 0x7fc53006d000 2024-04-25T00:01:52.028 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-04-25T00:01:52.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.025+0000 7fc5380b4640 1 -- 172.21.15.116:0/951942661 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc5140678c0 msgr2=0x7fc514069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:52.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.025+0000 7fc5380b4640 1 --2- 172.21.15.116:0/951942661 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc5140678c0 0x7fc514069d80 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc52c007fd0 tx=0x7fc52c0079a0 comp rx=0 tx=0).stop 2024-04-25T00:01:52.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.025+0000 7fc5380b4640 1 -- 172.21.15.116:0/951942661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc53006d000 msgr2=0x7fc5300c05b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:52.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.025+0000 7fc5380b4640 1 --2- 172.21.15.116:0/951942661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc53006d000 0x7fc5300c05b0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fc5200ae040 tx=0x7fc52001aa20 comp rx=0 tx=0).stop 2024-04-25T00:01:52.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.025+0000 7fc5380b4640 1 -- 172.21.15.116:0/951942661 shutdown_connections 2024-04-25T00:01:52.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.025+0000 7fc5380b4640 1 --2- 172.21.15.116:0/951942661 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc5140678c0 0x7fc514069d80 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:52.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.025+0000 7fc5380b4640 1 --2- 172.21.15.116:0/951942661 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc530148f20 0x7fc5301ef260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:52.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.025+0000 7fc5380b4640 1 --2- 172.21.15.116:0/951942661 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc53006d510 0x7fc5301eed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:52.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.025+0000 7fc5380b4640 1 --2- 172.21.15.116:0/951942661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc53006d000 0x7fc5300c05b0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:52.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.025+0000 7fc5380b4640 1 -- 172.21.15.116:0/951942661 >> 172.21.15.116:0/951942661 conn(0x7fc5300c3a20 msgr2=0x7fc530070ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:52.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.029+0000 7fc5380b4640 1 -- 172.21.15.116:0/951942661 shutdown_connections 2024-04-25T00:01:52.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.029+0000 7fc5380b4640 1 -- 172.21.15.116:0/951942661 wait complete. 2024-04-25T00:01:52.045 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:01:52.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.149+0000 7fe3cc65f640 1 Processor -- start 2024-04-25T00:01:52.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.149+0000 7fe3cc65f640 1 -- start start 2024-04-25T00:01:52.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.149+0000 7fe3cc65f640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fe3c40cacc0 0x7fe3c414c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:52.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.149+0000 7fe3cc65f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3c4156720 con 0x7fe3c40c5ac0 2024-04-25T00:01:52.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.149+0000 7fe3cc65f640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3c4156440 con 0x7fe3c414bf10 2024-04-25T00:01:52.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.149+0000 7fe3cc65f640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fe3c40c8460 con 0x7fe3c40cacc0 2024-04-25T00:01:52.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.149+0000 7fe3cabfc640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fe3c40c5ac0 0x7fe3c40c9040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:47614/0 (socket says 172.21.15.116:47614) 2024-04-25T00:01:52.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.149+0000 7fe3cabfc640 1 -- 172.21.15.116:0/3839716320 learned_addr learned my addr 172.21.15.116:0/3839716320 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:52.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.149+0000 7fe3c9bfa640 1 --2- 172.21.15.116:0/3839716320 >> v2:172.21.15.174:3300/0 conn(0x7fe3c40cacc0 0x7fe3c414c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:52.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.149+0000 7fe3c93f9640 1 -- 172.21.15.116:0/3839716320 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1619437115 0 0) 0x7fe3c4156720 con 0x7fe3c40c5ac0 2024-04-25T00:01:52.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.149+0000 7fe3c93f9640 1 -- 172.21.15.116:0/3839716320 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe3b0003530 con 0x7fe3c40c5ac0 2024-04-25T00:01:52.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.149+0000 7fe3c93f9640 1 -- 172.21.15.116:0/3839716320 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 181574796 0 0) 0x7fe3b0003530 con 0x7fe3c40c5ac0 2024-04-25T00:01:52.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.149+0000 7fe3c93f9640 1 -- 172.21.15.116:0/3839716320 >> v2:172.21.15.174:3300/0 conn(0x7fe3c40cacc0 msgr2=0x7fe3c414c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:52.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.149+0000 7fe3c93f9640 1 --2- 172.21.15.116:0/3839716320 >> v2:172.21.15.174:3300/0 conn(0x7fe3c40cacc0 0x7fe3c414c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:52.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.149+0000 7fe3c93f9640 1 -- 172.21.15.116:0/3839716320 >> v1:172.21.15.174:6789/0 conn(0x7fe3c414bf10 legacy=0x7fe3c406be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:52.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.149+0000 7fe3c93f9640 1 -- 172.21.15.116:0/3839716320 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3c40c7c60 con 0x7fe3c40c5ac0 2024-04-25T00:01:52.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.149+0000 7fe3c93f9640 1 -- 172.21.15.116:0/3839716320 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7fe3bc001a00 con 0x7fe3c40c5ac0 2024-04-25T00:01:52.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.149+0000 7fe3c93f9640 1 -- 172.21.15.116:0/3839716320 >> v1:172.21.15.116:6789/0 conn(0x7fe3c40c5ac0 legacy=0x7fe3c40c9040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:52.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.149+0000 7fe3c93f9640 1 --2- 172.21.15.116:0/3839716320 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe3b00039f0 0x7fe3b0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:52.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.149+0000 7fe3c93f9640 1 --2- 172.21.15.116:0/3839716320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3b0004810 0x7fe3b0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:52.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.149+0000 7fe3c93f9640 1 --2- 172.21.15.116:0/3839716320 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe3b00051d0 0x7fe3b000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:52.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.149+0000 7fe3c93f9640 1 -- 172.21.15.116:0/3839716320 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe3b000c0a0 con 0x7fe3b0004810 2024-04-25T00:01:52.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.149+0000 7fe3c93f9640 1 -- 172.21.15.116:0/3839716320 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe3b000c220 con 0x7fe3b00039f0 2024-04-25T00:01:52.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.149+0000 7fe3c93f9640 1 -- 172.21.15.116:0/3839716320 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe3b000c420 con 0x7fe3b00051d0 2024-04-25T00:01:52.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.149+0000 7fe3c9bfa640 1 --2- 172.21.15.116:0/3839716320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3b0004810 0x7fe3b0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:52.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.149+0000 7fe3ca3fb640 1 --2- 172.21.15.116:0/3839716320 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe3b00039f0 0x7fe3b0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:52.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.149+0000 7fe3cabfc640 1 --2- 172.21.15.116:0/3839716320 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe3b00051d0 0x7fe3b000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:52.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.149+0000 7fe3c9bfa640 1 -- 172.21.15.116:0/3839716320 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe3b00051d0 msgr2=0x7fe3b000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:52.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.149+0000 7fe3c9bfa640 1 --2- 172.21.15.116:0/3839716320 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe3b00051d0 0x7fe3b000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:52.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.149+0000 7fe3c9bfa640 1 -- 172.21.15.116:0/3839716320 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe3b00039f0 msgr2=0x7fe3b0003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:52.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.149+0000 7fe3c9bfa640 1 --2- 172.21.15.116:0/3839716320 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe3b00039f0 0x7fe3b0003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:52.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.149+0000 7fe3c9bfa640 1 -- 172.21.15.116:0/3839716320 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3b0003530 con 0x7fe3b0004810 2024-04-25T00:01:52.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.149+0000 7fe3cabfc640 1 --2- 172.21.15.116:0/3839716320 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe3b00051d0 0x7fe3b000b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:52.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.149+0000 7fe3cabfc640 1 -- 172.21.15.116:0/3839716320 reap_dead start 2024-04-25T00:01:52.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.153+0000 7fe3c9bfa640 1 --2- 172.21.15.116:0/3839716320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3b0004810 0x7fe3b0004c90 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fe3c406bb30 tx=0x7fe3b401ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c217fbbc60c7eda server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:52.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.153+0000 7fe3c93f9640 1 -- 172.21.15.116:0/3839716320 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b40b2070 con 0x7fe3b0004810 2024-04-25T00:01:52.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.153+0000 7fe3c93f9640 1 -- 172.21.15.116:0/3839716320 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3b40a5600 con 0x7fe3b0004810 2024-04-25T00:01:52.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.153+0000 7fe3c93f9640 1 -- 172.21.15.116:0/3839716320 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b40096b0 con 0x7fe3b0004810 2024-04-25T00:01:52.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.153+0000 7fe3cc65f640 1 -- 172.21.15.116:0/3839716320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3b0004810 msgr2=0x7fe3b0004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:52.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.153+0000 7fe3cc65f640 1 --2- 172.21.15.116:0/3839716320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3b0004810 0x7fe3b0004c90 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fe3c406bb30 tx=0x7fe3b401ca10 comp rx=0 tx=0).stop 2024-04-25T00:01:52.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.153+0000 7fe3cc65f640 1 -- 172.21.15.116:0/3839716320 shutdown_connections 2024-04-25T00:01:52.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.153+0000 7fe3cc65f640 1 --2- 172.21.15.116:0/3839716320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3b0004810 0x7fe3b0004c90 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:52.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.153+0000 7fe3cc65f640 1 -- 172.21.15.116:0/3839716320 >> 172.21.15.116:0/3839716320 conn(0x7fe3c40ace40 msgr2=0x7fe3c40ad250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:52.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.153+0000 7fe3cc65f640 1 -- 172.21.15.116:0/3839716320 shutdown_connections 2024-04-25T00:01:52.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.153+0000 7fe3cc65f640 1 -- 172.21.15.116:0/3839716320 wait complete. 2024-04-25T00:01:52.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.153+0000 7fe3cc65f640 1 Processor -- start 2024-04-25T00:01:52.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.153+0000 7fe3cc65f640 1 -- start start 2024-04-25T00:01:52.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.153+0000 7fe3cc65f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3b0004810 0x7fe3c40c1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:52.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.153+0000 7fe3cc65f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe3c40ae260 0x7fe3c40bfd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:52.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.153+0000 7fe3cc65f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe3c40b0d20 0x7fe3c40c0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:52.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.153+0000 7fe3cc65f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c40be950 con 0x7fe3b0004810 2024-04-25T00:01:52.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.153+0000 7fe3cc65f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c40be7d0 con 0x7fe3c40b0d20 2024-04-25T00:01:52.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.153+0000 7fe3cc65f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe3c414c2f0 con 0x7fe3c40ae260 2024-04-25T00:01:52.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.153+0000 7fe3ca3fb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3b0004810 0x7fe3c40c1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:52.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.153+0000 7fe3ca3fb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3b0004810 0x7fe3c40c1280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43530/0 (socket says 172.21.15.116:43530) 2024-04-25T00:01:52.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.153+0000 7fe3ca3fb640 1 -- 172.21.15.116:0/1764995451 learned_addr learned my addr 172.21.15.116:0/1764995451 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:52.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.153+0000 7fe3c9bfa640 1 --2- 172.21.15.116:0/1764995451 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe3c40ae260 0x7fe3c40bfd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:52.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.153+0000 7fe3ca3fb640 1 -- 172.21.15.116:0/1764995451 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe3c40ae260 msgr2=0x7fe3c40bfd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:52.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.153+0000 7fe3ca3fb640 1 --2- 172.21.15.116:0/1764995451 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe3c40ae260 0x7fe3c40bfd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:52.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.153+0000 7fe3ca3fb640 1 -- 172.21.15.116:0/1764995451 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe3c40b0d20 msgr2=0x7fe3c40c0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:52.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.153+0000 7fe3cabfc640 1 --2- 172.21.15.116:0/1764995451 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe3c40b0d20 0x7fe3c40c0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:52.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.153+0000 7fe3ca3fb640 1 --2- 172.21.15.116:0/1764995451 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe3c40b0d20 0x7fe3c40c0280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:52.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.153+0000 7fe3ca3fb640 1 -- 172.21.15.116:0/1764995451 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3c40beca0 con 0x7fe3b0004810 2024-04-25T00:01:52.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.153+0000 7fe3c9bfa640 1 --2- 172.21.15.116:0/1764995451 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe3c40ae260 0x7fe3c40bfd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:52.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.153+0000 7fe3ca3fb640 1 --2- 172.21.15.116:0/1764995451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3b0004810 0x7fe3c40c1280 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fe3c000e550 tx=0x7fe3c000ea10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:52.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.153+0000 7fe3baffd640 1 -- 172.21.15.116:0/1764995451 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3c00052e0 con 0x7fe3b0004810 2024-04-25T00:01:52.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.153+0000 7fe3baffd640 1 -- 172.21.15.116:0/1764995451 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3c0018070 con 0x7fe3b0004810 2024-04-25T00:01:52.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.153+0000 7fe3cc65f640 1 -- 172.21.15.116:0/1764995451 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3c40bef80 con 0x7fe3b0004810 2024-04-25T00:01:52.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.153+0000 7fe3baffd640 1 -- 172.21.15.116:0/1764995451 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3c0003820 con 0x7fe3b0004810 2024-04-25T00:01:52.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.153+0000 7fe3cc65f640 1 -- 172.21.15.116:0/1764995451 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3c40bf540 con 0x7fe3b0004810 2024-04-25T00:01:52.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.157+0000 7fe3cc65f640 1 -- 172.21.15.116:0/1764995451 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3c4068350 con 0x7fe3b0004810 2024-04-25T00:01:52.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.157+0000 7fe3baffd640 1 -- 172.21.15.116:0/1764995451 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe3c00039c0 con 0x7fe3b0004810 2024-04-25T00:01:52.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.157+0000 7fe3baffd640 1 --2- 172.21.15.116:0/1764995451 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe3a80678c0 0x7fe3a8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:52.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.157+0000 7fe3baffd640 1 -- 172.21.15.116:0/1764995451 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6666+0+0 (secure 0 0 0) 0x7fe3c0085d30 con 0x7fe3b0004810 2024-04-25T00:01:52.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.157+0000 7fe3c9bfa640 1 --2- 172.21.15.116:0/1764995451 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe3a80678c0 0x7fe3a8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:52.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.157+0000 7fe3c9bfa640 1 --2- 172.21.15.116:0/1764995451 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe3a80678c0 0x7fe3a8069d80 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe3b40af000 tx=0x7fe3b4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:52.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.161+0000 7fe3baffd640 1 -- 172.21.15.116:0/1764995451 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fe3c0056540 con 0x7fe3b0004810 2024-04-25T00:01:52.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.357+0000 7fe3cc65f640 1 -- 172.21.15.116:0/1764995451 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe3c40caaa0 con 0x7fe3b0004810 2024-04-25T00:01:52.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.357+0000 7fe3baffd640 1 -- 172.21.15.116:0/1764995451 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6492 (secure 0 0 0) 0x7fe3c0005480 con 0x7fe3b0004810 2024-04-25T00:01:52.361 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:01:52.361 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6834/1459177921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1459177921},{"type":"v1","addr":"172.21.15.116:6834","nonce":1459177921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:01:08.748957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6837/1762356055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1762356055},{"type":"v1","addr":"172.21.15.174:6837","nonce":1762356055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270]},"id":1},{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:01:22.888641+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6835/174781672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":174781672},{"type":"v1","addr":"172.21.15.116:6835","nonce":174781672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4245,"qdb_cluster":[4245]},"id":2}]} 2024-04-25T00:01:52.361 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 14 2024-04-25T00:01:52.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.361+0000 7fe3cc65f640 1 -- 172.21.15.116:0/1764995451 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe3a80678c0 msgr2=0x7fe3a8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:52.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.361+0000 7fe3cc65f640 1 --2- 172.21.15.116:0/1764995451 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe3a80678c0 0x7fe3a8069d80 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe3b40af000 tx=0x7fe3b4008040 comp rx=0 tx=0).stop 2024-04-25T00:01:52.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.361+0000 7fe3cc65f640 1 -- 172.21.15.116:0/1764995451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3b0004810 msgr2=0x7fe3c40c1280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:52.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.361+0000 7fe3cc65f640 1 --2- 172.21.15.116:0/1764995451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3b0004810 0x7fe3c40c1280 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fe3c000e550 tx=0x7fe3c000ea10 comp rx=0 tx=0).stop 2024-04-25T00:01:52.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.361+0000 7fe3cc65f640 1 -- 172.21.15.116:0/1764995451 shutdown_connections 2024-04-25T00:01:52.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.361+0000 7fe3cc65f640 1 --2- 172.21.15.116:0/1764995451 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe3a80678c0 0x7fe3a8069d80 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:52.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.361+0000 7fe3cc65f640 1 --2- 172.21.15.116:0/1764995451 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe3c40b0d20 0x7fe3c40c0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:52.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.361+0000 7fe3cc65f640 1 --2- 172.21.15.116:0/1764995451 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe3c40ae260 0x7fe3c40bfd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:52.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.361+0000 7fe3cc65f640 1 --2- 172.21.15.116:0/1764995451 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3b0004810 0x7fe3c40c1280 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:52.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.361+0000 7fe3cc65f640 1 -- 172.21.15.116:0/1764995451 >> 172.21.15.116:0/1764995451 conn(0x7fe3c40ace40 msgr2=0x7fe3c40c6b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:52.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.361+0000 7fe3cc65f640 1 -- 172.21.15.116:0/1764995451 shutdown_connections 2024-04-25T00:01:52.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.361+0000 7fe3cc65f640 1 -- 172.21.15.116:0/1764995451 wait complete. 2024-04-25T00:01:52.379 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T00:01:52.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.481+0000 7fcac89ed640 1 Processor -- start 2024-04-25T00:01:52.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.481+0000 7fcac89ed640 1 -- start start 2024-04-25T00:01:52.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.485+0000 7fcac89ed640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcac4070dd0 0x7fcac4071270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:52.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.485+0000 7fcac89ed640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcac4152c10 con 0x7fcac4148890 2024-04-25T00:01:52.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.485+0000 7fcac89ed640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcac4151e50 con 0x7fcac406e890 2024-04-25T00:01:52.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.485+0000 7fcac89ed640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fcac4152130 con 0x7fcac4070dd0 2024-04-25T00:01:52.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.485+0000 7fcac259c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fcac4148890 0x7fcac4072590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:47628/0 (socket says 172.21.15.116:47628) 2024-04-25T00:01:52.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.485+0000 7fcac259c640 1 -- 172.21.15.116:0/1602021788 learned_addr learned my addr 172.21.15.116:0/1602021788 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:52.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.485+0000 7fcabbfff640 1 --2- 172.21.15.116:0/1602021788 >> v2:172.21.15.116:3300/0 conn(0x7fcac4070dd0 0x7fcac4071270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:52.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.485+0000 7fcabbfff640 1 -- 172.21.15.116:0/1602021788 >> v1:172.21.15.174:6789/0 conn(0x7fcac406e890 legacy=0x7fcac40717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:52.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.485+0000 7fcabbfff640 1 -- 172.21.15.116:0/1602021788 >> v1:172.21.15.116:6789/0 conn(0x7fcac4148890 legacy=0x7fcac4072590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:52.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.485+0000 7fcabbfff640 1 -- 172.21.15.116:0/1602021788 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcac4152410 con 0x7fcac4070dd0 2024-04-25T00:01:52.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.485+0000 7fcac1d9b640 1 -- 172.21.15.116:0/1602021788 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1593372281 0 0) 0x7fcac4152c10 con 0x7fcac4148890 2024-04-25T00:01:52.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.485+0000 7fcabbfff640 1 --2- 172.21.15.116:0/1602021788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcac4070dd0 0x7fcac4071270 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fcab40097f0 tx=0x7fcab401c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad7dea8f156fb97c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:52.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.485+0000 7fcac1d9b640 1 -- 172.21.15.116:0/1602021788 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcab40b0070 con 0x7fcac4070dd0 2024-04-25T00:01:52.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.485+0000 7fcac1d9b640 1 -- 172.21.15.116:0/1602021788 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcab40ab600 con 0x7fcac4070dd0 2024-04-25T00:01:52.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.485+0000 7fcac1d9b640 1 -- 172.21.15.116:0/1602021788 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcab40ab9d0 con 0x7fcac4070dd0 2024-04-25T00:01:52.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.489+0000 7fcac89ed640 1 -- 172.21.15.116:0/1602021788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcac4070dd0 msgr2=0x7fcac4071270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:52.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.489+0000 7fcac89ed640 1 --2- 172.21.15.116:0/1602021788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcac4070dd0 0x7fcac4071270 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fcab40097f0 tx=0x7fcab401c460 comp rx=0 tx=0).stop 2024-04-25T00:01:52.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.489+0000 7fcac89ed640 1 -- 172.21.15.116:0/1602021788 shutdown_connections 2024-04-25T00:01:52.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.489+0000 7fcac89ed640 1 --2- 172.21.15.116:0/1602021788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcac4070dd0 0x7fcac4071270 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:52.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.489+0000 7fcac89ed640 1 -- 172.21.15.116:0/1602021788 >> 172.21.15.116:0/1602021788 conn(0x7fcac40c3b00 msgr2=0x7fcac40c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:52.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.489+0000 7fcac89ed640 1 -- 172.21.15.116:0/1602021788 shutdown_connections 2024-04-25T00:01:52.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.489+0000 7fcac89ed640 1 -- 172.21.15.116:0/1602021788 wait complete. 2024-04-25T00:01:52.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.489+0000 7fcac89ed640 1 Processor -- start 2024-04-25T00:01:52.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.489+0000 7fcac89ed640 1 -- start start 2024-04-25T00:01:52.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.489+0000 7fcac89ed640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcac406e890 0x7fcac40c5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:52.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.489+0000 7fcac89ed640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcac4070dd0 0x7fcac413f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:52.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.489+0000 7fcac89ed640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcac4148890 0x7fcac40c66c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:52.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.489+0000 7fcac89ed640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcac40abf20 con 0x7fcac406e890 2024-04-25T00:01:52.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.489+0000 7fcac89ed640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcac40abda0 con 0x7fcac4070dd0 2024-04-25T00:01:52.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.489+0000 7fcac89ed640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fcac40ac0a0 con 0x7fcac4148890 2024-04-25T00:01:52.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.489+0000 7fcac259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcac406e890 0x7fcac40c5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:52.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.489+0000 7fcac259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcac406e890 0x7fcac40c5650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43544/0 (socket says 172.21.15.116:43544) 2024-04-25T00:01:52.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.489+0000 7fcac259c640 1 -- 172.21.15.116:0/1517688351 learned_addr learned my addr 172.21.15.116:0/1517688351 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:52.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.489+0000 7fcac259c640 1 -- 172.21.15.116:0/1517688351 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcac4148890 msgr2=0x7fcac40c66c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:52.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.489+0000 7fcac259c640 1 --2- 172.21.15.116:0/1517688351 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcac4148890 0x7fcac40c66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:52.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.489+0000 7fcac259c640 1 -- 172.21.15.116:0/1517688351 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcac4070dd0 msgr2=0x7fcac413f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:52.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.489+0000 7fcac259c640 1 --2- 172.21.15.116:0/1517688351 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcac4070dd0 0x7fcac413f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:52.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.489+0000 7fcac259c640 1 -- 172.21.15.116:0/1517688351 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcac40c6c00 con 0x7fcac406e890 2024-04-25T00:01:52.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.489+0000 7fcac259c640 1 --2- 172.21.15.116:0/1517688351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcac406e890 0x7fcac40c5650 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fcaac00c480 tx=0x7fcaac00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:52.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.489+0000 7fcabaffd640 1 -- 172.21.15.116:0/1517688351 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaac018070 con 0x7fcac406e890 2024-04-25T00:01:52.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.489+0000 7fcabaffd640 1 -- 172.21.15.116:0/1517688351 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcaac007500 con 0x7fcac406e890 2024-04-25T00:01:52.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.489+0000 7fcac89ed640 1 -- 172.21.15.116:0/1517688351 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcac4140930 con 0x7fcac406e890 2024-04-25T00:01:52.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.489+0000 7fcabaffd640 1 -- 172.21.15.116:0/1517688351 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaac004a30 con 0x7fcac406e890 2024-04-25T00:01:52.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.489+0000 7fcac89ed640 1 -- 172.21.15.116:0/1517688351 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcac41ef1f0 con 0x7fcac406e890 2024-04-25T00:01:52.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.489+0000 7fcac89ed640 1 -- 172.21.15.116:0/1517688351 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcac4068350 con 0x7fcac406e890 2024-04-25T00:01:52.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.493+0000 7fcabaffd640 1 -- 172.21.15.116:0/1517688351 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fcaac01b070 con 0x7fcac406e890 2024-04-25T00:01:52.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.493+0000 7fcabaffd640 1 --2- 172.21.15.116:0/1517688351 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcaa40678f0 0x7fcaa4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:52.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.493+0000 7fcabaffd640 1 -- 172.21.15.116:0/1517688351 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6666+0+0 (secure 0 0 0) 0x7fcaac085340 con 0x7fcac406e890 2024-04-25T00:01:52.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.493+0000 7fcabbfff640 1 --2- 172.21.15.116:0/1517688351 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcaa40678f0 0x7fcaa4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:52.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.493+0000 7fcabbfff640 1 --2- 172.21.15.116:0/1517688351 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcaa40678f0 0x7fcaa4069db0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fcab40ad000 tx=0x7fcab4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:52.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.497+0000 7fcabaffd640 1 -- 172.21.15.116:0/1517688351 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fcaac055ab0 con 0x7fcac406e890 2024-04-25T00:01:52.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.657+0000 7fcac89ed640 1 -- 172.21.15.116:0/1517688351 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcac4071a90 con 0x7fcac406e890 2024-04-25T00:01:52.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.661+0000 7fcabaffd640 1 -- 172.21.15.116:0/1517688351 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) ==== 74+0+20606 (secure 0 0 0) 0x7fcaac0553b0 con 0x7fcac406e890 2024-04-25T00:01:52.663 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:01:52.663 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":29,"fsid":"6d6a1aa7-0478-4c83-8057-05917e81f046","created":"2024-04-25T00:00:40.682493+0000","modified":"2024-04-25T00:01:46.633464+0000","last_up_change":"2024-04-25T00:00:51.822174+0000","last_in_change":"2024-04-25T00:00:42.980473+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T00:00:53.324376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-04-25T00:01:03.197326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-04-25T00:01:03.728618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-04-25T00:01:17.125210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-04-25T00:01:17.821873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8b6ddcb2-1ad9-4167-961c-bd6936bb9060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6809","nonce":3751765682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6811","nonce":3751765682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6815","nonce":3751765682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6813","nonce":3751765682}]},"public_addr":"172.21.15.116:6809/3751765682","cluster_addr":"172.21.15.116:6811/3751765682","heartbeat_back_addr":"172.21.15.116:6815/3751765682","heartbeat_front_addr":"172.21.15.116:6813/3751765682","state":["exists","up"]},{"osd":1,"uuid":"a6431768-994f-41da-8853-886f5687fb91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6801","nonce":377879124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6803","nonce":377879124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6807","nonce":377879124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6805","nonce":377879124}]},"public_addr":"172.21.15.116:6801/377879124","cluster_addr":"172.21.15.116:6803/377879124","heartbeat_back_addr":"172.21.15.116:6807/377879124","heartbeat_front_addr":"172.21.15.116:6805/377879124","state":["exists","up"]},{"osd":2,"uuid":"4f69bd40-5a23-45ff-8693-e1de75e1aa9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6817","nonce":1099282267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6819","nonce":1099282267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6823","nonce":1099282267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6821","nonce":1099282267}]},"public_addr":"172.21.15.116:6817/1099282267","cluster_addr":"172.21.15.116:6819/1099282267","heartbeat_back_addr":"172.21.15.116:6823/1099282267","heartbeat_front_addr":"172.21.15.116:6821/1099282267","state":["exists","up"]},{"osd":3,"uuid":"6a0d5344-8b6f-40d3-918d-0f0a4be688e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6825","nonce":1179942661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6827","nonce":1179942661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6831","nonce":1179942661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6829","nonce":1179942661}]},"public_addr":"172.21.15.116:6825/1179942661","cluster_addr":"172.21.15.116:6827/1179942661","heartbeat_back_addr":"172.21.15.116:6831/1179942661","heartbeat_front_addr":"172.21.15.116:6829/1179942661","state":["exists","up"]},{"osd":4,"uuid":"132e92e4-b825-441e-965a-ca00533fa86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6801","nonce":3132444330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6803","nonce":3132444330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6807","nonce":3132444330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6805","nonce":3132444330}]},"public_addr":"172.21.15.174:6801/3132444330","cluster_addr":"172.21.15.174:6803/3132444330","heartbeat_back_addr":"172.21.15.174:6807/3132444330","heartbeat_front_addr":"172.21.15.174:6805/3132444330","state":["exists","up"]},{"osd":5,"uuid":"a2eeb5b6-1b31-43cc-8306-a0adf3ee15bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6809","nonce":881817164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6811","nonce":881817164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6815","nonce":881817164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6813","nonce":881817164}]},"public_addr":"172.21.15.174:6809/881817164","cluster_addr":"172.21.15.174:6811/881817164","heartbeat_back_addr":"172.21.15.174:6815/881817164","heartbeat_front_addr":"172.21.15.174:6813/881817164","state":["exists","up"]},{"osd":6,"uuid":"588e873a-c841-4ba7-90f2-8956c87a9983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6825","nonce":117574251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6827","nonce":117574251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6831","nonce":117574251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6829","nonce":117574251}]},"public_addr":"172.21.15.174:6825/117574251","cluster_addr":"172.21.15.174:6827/117574251","heartbeat_back_addr":"172.21.15.174:6831/117574251","heartbeat_front_addr":"172.21.15.174:6829/117574251","state":["exists","up"]},{"osd":7,"uuid":"3b90f9b0-8a0e-4883-b088-ca1d4012870c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6817","nonce":2543899196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6819","nonce":2543899196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6823","nonce":2543899196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6821","nonce":2543899196}]},"public_addr":"172.21.15.174:6817/2543899196","cluster_addr":"172.21.15.174:6819/2543899196","heartbeat_back_addr":"172.21.15.174:6823/2543899196","heartbeat_front_addr":"172.21.15.174:6821/2543899196","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.293769+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.310034+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.404865+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.549658+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.192174+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.297266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.493649+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.452309+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2c","mappings":[{"from":0,"to":7}]},{"pgid":"5.3","mappings":[{"from":4,"to":5}]},{"pgid":"5.5","mappings":[{"from":4,"to":5}]},{"pgid":"5.14","mappings":[{"from":4,"to":5}]},{"pgid":"5.19","mappings":[{"from":4,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:01:52.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.665+0000 7fcac89ed640 1 -- 172.21.15.116:0/1517688351 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcaa40678f0 msgr2=0x7fcaa4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:52.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.665+0000 7fcac89ed640 1 --2- 172.21.15.116:0/1517688351 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcaa40678f0 0x7fcaa4069db0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fcab40ad000 tx=0x7fcab4008040 comp rx=0 tx=0).stop 2024-04-25T00:01:52.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.665+0000 7fcac89ed640 1 -- 172.21.15.116:0/1517688351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcac406e890 msgr2=0x7fcac40c5650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:52.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.665+0000 7fcac89ed640 1 --2- 172.21.15.116:0/1517688351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcac406e890 0x7fcac40c5650 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fcaac00c480 tx=0x7fcaac00c940 comp rx=0 tx=0).stop 2024-04-25T00:01:52.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.665+0000 7fcac89ed640 1 -- 172.21.15.116:0/1517688351 shutdown_connections 2024-04-25T00:01:52.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.665+0000 7fcac89ed640 1 --2- 172.21.15.116:0/1517688351 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcaa40678f0 0x7fcaa4069db0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:52.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.665+0000 7fcac89ed640 1 --2- 172.21.15.116:0/1517688351 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcac4148890 0x7fcac40c66c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:52.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.665+0000 7fcac89ed640 1 --2- 172.21.15.116:0/1517688351 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcac4070dd0 0x7fcac413f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:52.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.665+0000 7fcac89ed640 1 --2- 172.21.15.116:0/1517688351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcac406e890 0x7fcac40c5650 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:52.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.665+0000 7fcac89ed640 1 -- 172.21.15.116:0/1517688351 >> 172.21.15.116:0/1517688351 conn(0x7fcac40c3b00 msgr2=0x7fcac40c2a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:52.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.665+0000 7fcac89ed640 1 -- 172.21.15.116:0/1517688351 shutdown_connections 2024-04-25T00:01:52.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.665+0000 7fcac89ed640 1 -- 172.21.15.116:0/1517688351 wait complete. 2024-04-25T00:01:52.684 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T00:01:52.684 INFO:tasks.mds_thrash.fs.[a]:starting mds_do_thrash for fs a 2024-04-25T00:01:52.684 INFO:tasks.mds_thrash.fs.[a]:waiting for 89 secs before thrashing 2024-04-25T00:01:52.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.797+0000 7f95bcfc4640 1 Processor -- start 2024-04-25T00:01:52.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.801+0000 7f95bcfc4640 1 -- start start 2024-04-25T00:01:52.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.801+0000 7f95bcfc4640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f95b814c720 0x7f95b81563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:52.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.801+0000 7f95bcfc4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f95b80a94a0 0x7f95b80c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:52.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.801+0000 7f95bcfc4640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95b8071220 con 0x7f95b80c2f50 2024-04-25T00:01:52.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.801+0000 7f95bcfc4640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f95b8153460 con 0x7f95b814c720 2024-04-25T00:01:52.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.801+0000 7f95bcfc4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f95b8152420 con 0x7f95b80a94a0 2024-04-25T00:01:52.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.801+0000 7f95b659c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f95b814c720 0x7f95b81563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:52.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.801+0000 7f95b659c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f95b814c720 0x7f95b81563d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43548/0 (socket says 172.21.15.116:43548) 2024-04-25T00:01:52.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.801+0000 7f95b659c640 1 -- 172.21.15.116:0/1318256689 learned_addr learned my addr 172.21.15.116:0/1318256689 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:52.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.801+0000 7f95add9b640 1 --2- 172.21.15.116:0/1318256689 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f95b80a94a0 0x7f95b80c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:52.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.801+0000 7f95b659c640 1 -- 172.21.15.116:0/1318256689 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f95b80a94a0 msgr2=0x7f95b80c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:52.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.801+0000 7f95b659c640 1 --2- 172.21.15.116:0/1318256689 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f95b80a94a0 0x7f95b80c2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:52.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.801+0000 7f95b659c640 1 -- 172.21.15.116:0/1318256689 >> v1:172.21.15.174:6789/0 conn(0x7f95b80c2f50 legacy=0x7f95b8071680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:52.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.801+0000 7f95b659c640 1 -- 172.21.15.116:0/1318256689 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95b814cb00 con 0x7f95b814c720 2024-04-25T00:01:52.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.801+0000 7f95b659c640 1 --2- 172.21.15.116:0/1318256689 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95b814c720 0x7f95b81563d0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f95a0009870 tx=0x7f95a001c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9983cc61d1b1a357 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:52.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.801+0000 7f95b5d9b640 1 -- 172.21.15.116:0/1318256689 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95a00ab640 con 0x7f95b814c720 2024-04-25T00:01:52.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.801+0000 7f95b5d9b640 1 -- 172.21.15.116:0/1318256689 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95a00b0070 con 0x7f95b814c720 2024-04-25T00:01:52.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.801+0000 7f95b5d9b640 1 -- 172.21.15.116:0/1318256689 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95a00aba00 con 0x7f95b814c720 2024-04-25T00:01:52.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.805+0000 7f95bcfc4640 1 -- 172.21.15.116:0/1318256689 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95b814c720 msgr2=0x7f95b81563d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:52.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.805+0000 7f95bcfc4640 1 --2- 172.21.15.116:0/1318256689 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95b814c720 0x7f95b81563d0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f95a0009870 tx=0x7f95a001c460 comp rx=0 tx=0).stop 2024-04-25T00:01:52.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.805+0000 7f95bcfc4640 1 -- 172.21.15.116:0/1318256689 shutdown_connections 2024-04-25T00:01:52.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.805+0000 7f95bcfc4640 1 --2- 172.21.15.116:0/1318256689 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f95b80a94a0 0x7f95b80c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:52.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.805+0000 7f95bcfc4640 1 --2- 172.21.15.116:0/1318256689 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95b814c720 0x7f95b81563d0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:52.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.805+0000 7f95bcfc4640 1 -- 172.21.15.116:0/1318256689 >> 172.21.15.116:0/1318256689 conn(0x7f95b80c3b00 msgr2=0x7f95b80c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:52.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.805+0000 7f95bcfc4640 1 -- 172.21.15.116:0/1318256689 shutdown_connections 2024-04-25T00:01:52.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.805+0000 7f95bcfc4640 1 -- 172.21.15.116:0/1318256689 wait complete. 2024-04-25T00:01:52.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.805+0000 7f95bcfc4640 1 Processor -- start 2024-04-25T00:01:52.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.805+0000 7f95bcfc4640 1 -- start start 2024-04-25T00:01:52.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.805+0000 7f95bcfc4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f95b80a94a0 0x7f95b8144050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:52.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.805+0000 7f95bcfc4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95b80c2f50 0x7f95b8151880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:52.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.805+0000 7f95bcfc4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f95b814c720 0x7f95b8151dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:52.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.805+0000 7f95bcfc4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f95b8152f40 con 0x7f95b80c2f50 2024-04-25T00:01:52.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.805+0000 7f95bcfc4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f95b8152c60 con 0x7f95b814c720 2024-04-25T00:01:52.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.805+0000 7f95bcfc4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f95b815f5c0 con 0x7f95b80a94a0 2024-04-25T00:01:52.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.805+0000 7f95add9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95b80c2f50 0x7f95b8151880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:52.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.805+0000 7f95add9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95b80c2f50 0x7f95b8151880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43556/0 (socket says 172.21.15.116:43556) 2024-04-25T00:01:52.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.805+0000 7f95add9b640 1 -- 172.21.15.116:0/3161090332 learned_addr learned my addr 172.21.15.116:0/3161090332 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:52.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.805+0000 7f95b6d9d640 1 --2- 172.21.15.116:0/3161090332 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f95b814c720 0x7f95b8151dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:52.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.805+0000 7f95b659c640 1 --2- 172.21.15.116:0/3161090332 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f95b80a94a0 0x7f95b8144050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:52.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.805+0000 7f95add9b640 1 -- 172.21.15.116:0/3161090332 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f95b80a94a0 msgr2=0x7f95b8144050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:52.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.805+0000 7f95add9b640 1 --2- 172.21.15.116:0/3161090332 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f95b80a94a0 0x7f95b8144050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:52.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.805+0000 7f95add9b640 1 -- 172.21.15.116:0/3161090332 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f95b814c720 msgr2=0x7f95b8151dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:52.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.805+0000 7f95add9b640 1 --2- 172.21.15.116:0/3161090332 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f95b814c720 0x7f95b8151dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:52.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.805+0000 7f95add9b640 1 -- 172.21.15.116:0/3161090332 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95b80ac150 con 0x7f95b80c2f50 2024-04-25T00:01:52.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.805+0000 7f95add9b640 1 --2- 172.21.15.116:0/3161090332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95b80c2f50 0x7f95b8151880 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f95a800c7f0 tx=0x7f95a800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:52.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.805+0000 7f95af7fe640 1 -- 172.21.15.116:0/3161090332 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95a8015070 con 0x7f95b80c2f50 2024-04-25T00:01:52.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.805+0000 7f95af7fe640 1 -- 172.21.15.116:0/3161090332 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95a8004240 con 0x7f95b80c2f50 2024-04-25T00:01:52.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.805+0000 7f95af7fe640 1 -- 172.21.15.116:0/3161090332 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95a8003620 con 0x7f95b80c2f50 2024-04-25T00:01:52.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.805+0000 7f95bcfc4640 1 -- 172.21.15.116:0/3161090332 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95b81478a0 con 0x7f95b80c2f50 2024-04-25T00:01:52.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.805+0000 7f95bcfc4640 1 -- 172.21.15.116:0/3161090332 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95b8146ae0 con 0x7f95b80c2f50 2024-04-25T00:01:52.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.805+0000 7f95bcfc4640 1 -- 172.21.15.116:0/3161090332 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95b8068350 con 0x7f95b80c2f50 2024-04-25T00:01:52.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.809+0000 7f95af7fe640 1 -- 172.21.15.116:0/3161090332 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f95a80039d0 con 0x7f95b80c2f50 2024-04-25T00:01:52.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.809+0000 7f95af7fe640 1 --2- 172.21.15.116:0/3161090332 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f95980678c0 0x7f9598069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:52.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.809+0000 7f95af7fe640 1 -- 172.21.15.116:0/3161090332 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6666+0+0 (secure 0 0 0) 0x7f95a80861c0 con 0x7f95b80c2f50 2024-04-25T00:01:52.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.809+0000 7f95b659c640 1 --2- 172.21.15.116:0/3161090332 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f95980678c0 0x7f9598069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:52.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.809+0000 7f95b659c640 1 --2- 172.21.15.116:0/3161090332 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f95980678c0 0x7f9598069d80 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f95a00af000 tx=0x7f95a0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:52.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.813+0000 7f95af7fe640 1 -- 172.21.15.116:0/3161090332 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f95b80ac150 con 0x7f95b80c2f50 2024-04-25T00:01:52.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.961+0000 7f95bcfc4640 1 -- 172.21.15.116:0/3161090332 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f95b80c3640 con 0x7f95b80c2f50 2024-04-25T00:01:52.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.961+0000 7f95af7fe640 1 -- 172.21.15.116:0/3161090332 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) ==== 74+0+20606 (secure 0 0 0) 0x7f95a80568a0 con 0x7f95b80c2f50 2024-04-25T00:01:52.965 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:01:52.965 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":29,"fsid":"6d6a1aa7-0478-4c83-8057-05917e81f046","created":"2024-04-25T00:00:40.682493+0000","modified":"2024-04-25T00:01:46.633464+0000","last_up_change":"2024-04-25T00:00:51.822174+0000","last_in_change":"2024-04-25T00:00:42.980473+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T00:00:53.324376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-04-25T00:01:03.197326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-04-25T00:01:03.728618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-04-25T00:01:17.125210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-04-25T00:01:17.821873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8b6ddcb2-1ad9-4167-961c-bd6936bb9060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6809","nonce":3751765682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6811","nonce":3751765682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6815","nonce":3751765682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6813","nonce":3751765682}]},"public_addr":"172.21.15.116:6809/3751765682","cluster_addr":"172.21.15.116:6811/3751765682","heartbeat_back_addr":"172.21.15.116:6815/3751765682","heartbeat_front_addr":"172.21.15.116:6813/3751765682","state":["exists","up"]},{"osd":1,"uuid":"a6431768-994f-41da-8853-886f5687fb91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6801","nonce":377879124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6803","nonce":377879124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6807","nonce":377879124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6805","nonce":377879124}]},"public_addr":"172.21.15.116:6801/377879124","cluster_addr":"172.21.15.116:6803/377879124","heartbeat_back_addr":"172.21.15.116:6807/377879124","heartbeat_front_addr":"172.21.15.116:6805/377879124","state":["exists","up"]},{"osd":2,"uuid":"4f69bd40-5a23-45ff-8693-e1de75e1aa9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6817","nonce":1099282267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6819","nonce":1099282267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6823","nonce":1099282267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6821","nonce":1099282267}]},"public_addr":"172.21.15.116:6817/1099282267","cluster_addr":"172.21.15.116:6819/1099282267","heartbeat_back_addr":"172.21.15.116:6823/1099282267","heartbeat_front_addr":"172.21.15.116:6821/1099282267","state":["exists","up"]},{"osd":3,"uuid":"6a0d5344-8b6f-40d3-918d-0f0a4be688e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6825","nonce":1179942661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6827","nonce":1179942661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6831","nonce":1179942661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6829","nonce":1179942661}]},"public_addr":"172.21.15.116:6825/1179942661","cluster_addr":"172.21.15.116:6827/1179942661","heartbeat_back_addr":"172.21.15.116:6831/1179942661","heartbeat_front_addr":"172.21.15.116:6829/1179942661","state":["exists","up"]},{"osd":4,"uuid":"132e92e4-b825-441e-965a-ca00533fa86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6801","nonce":3132444330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6803","nonce":3132444330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6807","nonce":3132444330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6805","nonce":3132444330}]},"public_addr":"172.21.15.174:6801/3132444330","cluster_addr":"172.21.15.174:6803/3132444330","heartbeat_back_addr":"172.21.15.174:6807/3132444330","heartbeat_front_addr":"172.21.15.174:6805/3132444330","state":["exists","up"]},{"osd":5,"uuid":"a2eeb5b6-1b31-43cc-8306-a0adf3ee15bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6809","nonce":881817164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6811","nonce":881817164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6815","nonce":881817164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6813","nonce":881817164}]},"public_addr":"172.21.15.174:6809/881817164","cluster_addr":"172.21.15.174:6811/881817164","heartbeat_back_addr":"172.21.15.174:6815/881817164","heartbeat_front_addr":"172.21.15.174:6813/881817164","state":["exists","up"]},{"osd":6,"uuid":"588e873a-c841-4ba7-90f2-8956c87a9983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6825","nonce":117574251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6827","nonce":117574251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6831","nonce":117574251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6829","nonce":117574251}]},"public_addr":"172.21.15.174:6825/117574251","cluster_addr":"172.21.15.174:6827/117574251","heartbeat_back_addr":"172.21.15.174:6831/117574251","heartbeat_front_addr":"172.21.15.174:6829/117574251","state":["exists","up"]},{"osd":7,"uuid":"3b90f9b0-8a0e-4883-b088-ca1d4012870c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6817","nonce":2543899196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6819","nonce":2543899196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6823","nonce":2543899196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6821","nonce":2543899196}]},"public_addr":"172.21.15.174:6817/2543899196","cluster_addr":"172.21.15.174:6819/2543899196","heartbeat_back_addr":"172.21.15.174:6823/2543899196","heartbeat_front_addr":"172.21.15.174:6821/2543899196","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.293769+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.310034+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.404865+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.549658+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.192174+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.297266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.493649+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.452309+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2c","mappings":[{"from":0,"to":7}]},{"pgid":"5.3","mappings":[{"from":4,"to":5}]},{"pgid":"5.5","mappings":[{"from":4,"to":5}]},{"pgid":"5.14","mappings":[{"from":4,"to":5}]},{"pgid":"5.19","mappings":[{"from":4,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:01:52.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.965+0000 7f95bcfc4640 1 -- 172.21.15.116:0/3161090332 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f95980678c0 msgr2=0x7f9598069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:52.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.965+0000 7f95bcfc4640 1 --2- 172.21.15.116:0/3161090332 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f95980678c0 0x7f9598069d80 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f95a00af000 tx=0x7f95a0008040 comp rx=0 tx=0).stop 2024-04-25T00:01:52.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.965+0000 7f95bcfc4640 1 -- 172.21.15.116:0/3161090332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95b80c2f50 msgr2=0x7f95b8151880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:52.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.965+0000 7f95bcfc4640 1 --2- 172.21.15.116:0/3161090332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95b80c2f50 0x7f95b8151880 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f95a800c7f0 tx=0x7f95a800ccb0 comp rx=0 tx=0).stop 2024-04-25T00:01:52.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.965+0000 7f95bcfc4640 1 -- 172.21.15.116:0/3161090332 shutdown_connections 2024-04-25T00:01:52.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.965+0000 7f95bcfc4640 1 --2- 172.21.15.116:0/3161090332 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f95980678c0 0x7f9598069d80 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:52.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.965+0000 7f95bcfc4640 1 --2- 172.21.15.116:0/3161090332 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f95b814c720 0x7f95b8151dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:52.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.965+0000 7f95bcfc4640 1 --2- 172.21.15.116:0/3161090332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95b80c2f50 0x7f95b8151880 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:52.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.965+0000 7f95bcfc4640 1 --2- 172.21.15.116:0/3161090332 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f95b80a94a0 0x7f95b8144050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:52.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.965+0000 7f95bcfc4640 1 -- 172.21.15.116:0/3161090332 >> 172.21.15.116:0/3161090332 conn(0x7f95b80c3b00 msgr2=0x7f95b80c2810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:52.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.965+0000 7f95bcfc4640 1 -- 172.21.15.116:0/3161090332 shutdown_connections 2024-04-25T00:01:52.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:52.965+0000 7f95bcfc4640 1 -- 172.21.15.116:0/3161090332 wait complete. 2024-04-25T00:01:52.983 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T00:00:53.324376+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'a_metadata', 'create_time': '2024-04-25T00:01:03.197326+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'a'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'a_data', 'create_time': '2024-04-25T00:01:03.728618+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'a'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'b_metadata', 'create_time': '2024-04-25T00:01:17.125210+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'b'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'b_data', 'create_time': '2024-04-25T00:01:17.821873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'b'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T00:01:52.983 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T00:01:53.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.089+0000 7f6dff7cd640 1 Processor -- start 2024-04-25T00:01:53.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.089+0000 7f6dff7cd640 1 -- start start 2024-04-25T00:01:53.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.093+0000 7f6dff7cd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6df80c5990 0x7f6df80c5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:53.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.093+0000 7f6dff7cd640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f6df80c4490 0x7f6df80c48c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:53.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.093+0000 7f6dff7cd640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6df81567b0 con 0x7f6df8153280 2024-04-25T00:01:53.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.093+0000 7f6dff7cd640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f6df80c4000 con 0x7f6df80c5990 2024-04-25T00:01:53.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.093+0000 7f6dff7cd640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f6df8156a90 con 0x7f6df80c4490 2024-04-25T00:01:53.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.093+0000 7f6dfcd68640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6df80c5990 0x7f6df80c5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:53.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.093+0000 7f6dfcd68640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6df80c5990 0x7f6df80c5dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43558/0 (socket says 172.21.15.116:43558) 2024-04-25T00:01:53.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.093+0000 7f6dfcd68640 1 -- 172.21.15.116:0/3927945244 learned_addr learned my addr 172.21.15.116:0/3927945244 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:53.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.093+0000 7f6dfcd68640 1 -- 172.21.15.116:0/3927945244 >> v2:172.21.15.174:3300/0 conn(0x7f6df80c4490 msgr2=0x7f6df80c48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:53.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.093+0000 7f6dfcd68640 1 --2- 172.21.15.116:0/3927945244 >> v2:172.21.15.174:3300/0 conn(0x7f6df80c4490 0x7f6df80c48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:53.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.093+0000 7f6dfcd68640 1 -- 172.21.15.116:0/3927945244 >> v1:172.21.15.116:6789/0 conn(0x7f6df8153280 legacy=0x7f6df8155d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:53.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.093+0000 7f6dfcd68640 1 -- 172.21.15.116:0/3927945244 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6df80c39b0 con 0x7f6df80c5990 2024-04-25T00:01:53.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.093+0000 7f6dfcd68640 1 --2- 172.21.15.116:0/3927945244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6df80c5990 0x7f6df80c5dd0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f6df80c4180 tx=0x7f6de801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=278b6b1cc61d3799 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:53.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.093+0000 7f6deffff640 1 -- 172.21.15.116:0/3927945244 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de80b0070 con 0x7f6df80c5990 2024-04-25T00:01:53.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.093+0000 7f6deffff640 1 -- 172.21.15.116:0/3927945244 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6de80ab600 con 0x7f6df80c5990 2024-04-25T00:01:53.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.093+0000 7f6deffff640 1 -- 172.21.15.116:0/3927945244 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de80ab9d0 con 0x7f6df80c5990 2024-04-25T00:01:53.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.097+0000 7f6dff7cd640 1 -- 172.21.15.116:0/3927945244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6df80c5990 msgr2=0x7f6df80c5dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:53.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.097+0000 7f6dff7cd640 1 --2- 172.21.15.116:0/3927945244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6df80c5990 0x7f6df80c5dd0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f6df80c4180 tx=0x7f6de801c490 comp rx=0 tx=0).stop 2024-04-25T00:01:53.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.097+0000 7f6dff7cd640 1 -- 172.21.15.116:0/3927945244 shutdown_connections 2024-04-25T00:01:53.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.097+0000 7f6dff7cd640 1 --2- 172.21.15.116:0/3927945244 >> v2:172.21.15.174:3300/0 conn(0x7f6df80c4490 0x7f6df80c48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:53.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.097+0000 7f6dff7cd640 1 --2- 172.21.15.116:0/3927945244 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6df80c5990 0x7f6df80c5dd0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:53.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.097+0000 7f6dff7cd640 1 -- 172.21.15.116:0/3927945244 >> 172.21.15.116:0/3927945244 conn(0x7f6df80c8f90 msgr2=0x7f6df80c93a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:53.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.097+0000 7f6dff7cd640 1 -- 172.21.15.116:0/3927945244 shutdown_connections 2024-04-25T00:01:53.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.097+0000 7f6dff7cd640 1 -- 172.21.15.116:0/3927945244 wait complete. 2024-04-25T00:01:53.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.097+0000 7f6dff7cd640 1 Processor -- start 2024-04-25T00:01:53.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.097+0000 7f6dff7cd640 1 -- start start 2024-04-25T00:01:53.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.097+0000 7f6dff7cd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6df80c4490 0x7f6df81551a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:53.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.097+0000 7f6dff7cd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6df80c5990 0x7f6df81f24c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:53.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.097+0000 7f6dff7cd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6df8153280 0x7f6df81eed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:53.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.097+0000 7f6dff7cd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6df81537e0 con 0x7f6df80c4490 2024-04-25T00:01:53.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.097+0000 7f6dff7cd640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6df8153660 con 0x7f6df80c5990 2024-04-25T00:01:53.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.097+0000 7f6dff7cd640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6df8162ca0 con 0x7f6df8153280 2024-04-25T00:01:53.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.097+0000 7f6dfd569640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6df80c4490 0x7f6df81551a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:53.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.097+0000 7f6dfd569640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6df80c4490 0x7f6df81551a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43570/0 (socket says 172.21.15.116:43570) 2024-04-25T00:01:53.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.097+0000 7f6dfd569640 1 -- 172.21.15.116:0/313227167 learned_addr learned my addr 172.21.15.116:0/313227167 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:53.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.097+0000 7f6dfdd6a640 1 --2- 172.21.15.116:0/313227167 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6df8153280 0x7f6df81eed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:53.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.097+0000 7f6dfcd68640 1 --2- 172.21.15.116:0/313227167 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6df80c5990 0x7f6df81f24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:53.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.097+0000 7f6dfd569640 1 -- 172.21.15.116:0/313227167 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6df8153280 msgr2=0x7f6df81eed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:53.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.097+0000 7f6dfd569640 1 --2- 172.21.15.116:0/313227167 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6df8153280 0x7f6df81eed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:53.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.097+0000 7f6dfd569640 1 -- 172.21.15.116:0/313227167 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6df80c5990 msgr2=0x7f6df81f24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:53.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.097+0000 7f6dfd569640 1 --2- 172.21.15.116:0/313227167 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6df80c5990 0x7f6df81f24c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:53.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.097+0000 7f6dfd569640 1 -- 172.21.15.116:0/313227167 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6df81ef2c0 con 0x7f6df80c4490 2024-04-25T00:01:53.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.097+0000 7f6dfdd6a640 1 --2- 172.21.15.116:0/313227167 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6df8153280 0x7f6df81eed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:53.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.097+0000 7f6dfd569640 1 --2- 172.21.15.116:0/313227167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6df80c4490 0x7f6df81551a0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f6df400c370 tx=0x7f6df400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:53.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.097+0000 7f6dedffb640 1 -- 172.21.15.116:0/313227167 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6df40052d0 con 0x7f6df80c4490 2024-04-25T00:01:53.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.097+0000 7f6dedffb640 1 -- 172.21.15.116:0/313227167 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6df4005470 con 0x7f6df80c4490 2024-04-25T00:01:53.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.097+0000 7f6dff7cd640 1 -- 172.21.15.116:0/313227167 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6df8155a60 con 0x7f6df80c4490 2024-04-25T00:01:53.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.097+0000 7f6dedffb640 1 -- 172.21.15.116:0/313227167 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6df4007260 con 0x7f6df80c4490 2024-04-25T00:01:53.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.097+0000 7f6dff7cd640 1 -- 172.21.15.116:0/313227167 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6df81f19e0 con 0x7f6df80c4490 2024-04-25T00:01:53.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.101+0000 7f6dff7cd640 1 -- 172.21.15.116:0/313227167 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6df804f510 con 0x7f6df80c4490 2024-04-25T00:01:53.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.101+0000 7f6dedffb640 1 -- 172.21.15.116:0/313227167 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6df4007400 con 0x7f6df80c4490 2024-04-25T00:01:53.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.101+0000 7f6dedffb640 1 --2- 172.21.15.116:0/313227167 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6ddc0678c0 0x7f6ddc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:53.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.101+0000 7f6dedffb640 1 -- 172.21.15.116:0/313227167 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6666+0+0 (secure 0 0 0) 0x7f6df4085440 con 0x7f6df80c4490 2024-04-25T00:01:53.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.101+0000 7f6dfcd68640 1 --2- 172.21.15.116:0/313227167 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6ddc0678c0 0x7f6ddc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:53.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.101+0000 7f6dfcd68640 1 --2- 172.21.15.116:0/313227167 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6ddc0678c0 0x7f6ddc069d80 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6de80ad000 tx=0x7f6de8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:53.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.105+0000 7f6dedffb640 1 -- 172.21.15.116:0/313227167 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f6df4055c50 con 0x7f6df80c4490 2024-04-25T00:01:53.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.249+0000 7f6dff7cd640 1 -- 172.21.15.116:0/313227167 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6df80c5590 con 0x7f6df80c4490 2024-04-25T00:01:53.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.253+0000 7f6dedffb640 1 -- 172.21.15.116:0/313227167 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) ==== 93+0+10 (secure 0 0 0) 0x7f6df4055550 con 0x7f6df80c4490 2024-04-25T00:01:53.255 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-04-25T00:01:53.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.253+0000 7f6dff7cd640 1 -- 172.21.15.116:0/313227167 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6ddc0678c0 msgr2=0x7f6ddc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:53.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.253+0000 7f6dff7cd640 1 --2- 172.21.15.116:0/313227167 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6ddc0678c0 0x7f6ddc069d80 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6de80ad000 tx=0x7f6de8008040 comp rx=0 tx=0).stop 2024-04-25T00:01:53.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.253+0000 7f6dff7cd640 1 -- 172.21.15.116:0/313227167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6df80c4490 msgr2=0x7f6df81551a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:53.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.253+0000 7f6dff7cd640 1 --2- 172.21.15.116:0/313227167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6df80c4490 0x7f6df81551a0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f6df400c370 tx=0x7f6df400c830 comp rx=0 tx=0).stop 2024-04-25T00:01:53.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.253+0000 7f6dff7cd640 1 -- 172.21.15.116:0/313227167 shutdown_connections 2024-04-25T00:01:53.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.253+0000 7f6dff7cd640 1 --2- 172.21.15.116:0/313227167 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6ddc0678c0 0x7f6ddc069d80 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:53.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.253+0000 7f6dff7cd640 1 --2- 172.21.15.116:0/313227167 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6df8153280 0x7f6df81eed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:53.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.253+0000 7f6dff7cd640 1 --2- 172.21.15.116:0/313227167 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6df80c5990 0x7f6df81f24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:53.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.253+0000 7f6dff7cd640 1 --2- 172.21.15.116:0/313227167 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6df80c4490 0x7f6df81551a0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:53.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.253+0000 7f6dff7cd640 1 -- 172.21.15.116:0/313227167 >> 172.21.15.116:0/313227167 conn(0x7f6df80c8f90 msgr2=0x7f6df80c8750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:53.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.253+0000 7f6dff7cd640 1 -- 172.21.15.116:0/313227167 shutdown_connections 2024-04-25T00:01:53.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.253+0000 7f6dff7cd640 1 -- 172.21.15.116:0/313227167 wait complete. 2024-04-25T00:01:53.272 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_metadata pg_num 2024-04-25T00:01:53.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.377+0000 7f8e211ff640 1 Processor -- start 2024-04-25T00:01:53.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.377+0000 7f8e211ff640 1 -- start start 2024-04-25T00:01:53.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.377+0000 7f8e211ff640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f8e1c06c450 0x7f8e1c0c6040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:53.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.377+0000 7f8e211ff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8e1c1564d0 0x7f8e1c0c6e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:53.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.377+0000 7f8e211ff640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e1c06b9e0 con 0x7f8e1c0c5c60 2024-04-25T00:01:53.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.377+0000 7f8e211ff640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f8e1c0c96d0 con 0x7f8e1c1564d0 2024-04-25T00:01:53.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.377+0000 7f8e211ff640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f8e1c0c7380 con 0x7f8e1c06c450 2024-04-25T00:01:53.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.377+0000 7f8e1b59e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8e1c1564d0 0x7f8e1c0c6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:53.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.377+0000 7f8e1ad9d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f8e1c0c5c60 0x7f8e1c0c8fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:47644/0 (socket says 172.21.15.116:47644) 2024-04-25T00:01:53.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.377+0000 7f8e1ad9d640 1 -- 172.21.15.116:0/3402723490 learned_addr learned my addr 172.21.15.116:0/3402723490 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:53.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.377+0000 7f8e1a59c640 1 --2- 172.21.15.116:0/3402723490 >> v2:172.21.15.174:3300/0 conn(0x7f8e1c06c450 0x7f8e1c0c6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:53.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.377+0000 7f8e1b59e640 1 -- 172.21.15.116:0/3402723490 >> v2:172.21.15.174:3300/0 conn(0x7f8e1c06c450 msgr2=0x7f8e1c0c6040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:53.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.377+0000 7f8e1b59e640 1 --2- 172.21.15.116:0/3402723490 >> v2:172.21.15.174:3300/0 conn(0x7f8e1c06c450 0x7f8e1c0c6040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:53.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.377+0000 7f8e1b59e640 1 -- 172.21.15.116:0/3402723490 >> v1:172.21.15.116:6789/0 conn(0x7f8e1c0c5c60 legacy=0x7f8e1c0c8fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:53.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.377+0000 7f8e19d9b640 1 -- 172.21.15.116:0/3402723490 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3347283885 0 0) 0x7f8e1c06b9e0 con 0x7f8e1c0c5c60 2024-04-25T00:01:53.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.377+0000 7f8e1b59e640 1 -- 172.21.15.116:0/3402723490 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e1c0c75b0 con 0x7f8e1c1564d0 2024-04-25T00:01:53.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.377+0000 7f8e1a59c640 1 --2- 172.21.15.116:0/3402723490 >> v2:172.21.15.174:3300/0 conn(0x7f8e1c06c450 0x7f8e1c0c6040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:53.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.381+0000 7f8e1b59e640 1 --2- 172.21.15.116:0/3402723490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e1c1564d0 0x7f8e1c0c6e40 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c067d90 tx=0x7f8e1401c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a536cf3d4de6d54a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:53.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.381+0000 7f8e19d9b640 1 -- 172.21.15.116:0/3402723490 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e140b2070 con 0x7f8e1c1564d0 2024-04-25T00:01:53.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.381+0000 7f8e19d9b640 1 -- 172.21.15.116:0/3402723490 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e140a5760 con 0x7f8e1c1564d0 2024-04-25T00:01:53.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.381+0000 7f8e19d9b640 1 -- 172.21.15.116:0/3402723490 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e140a5b30 con 0x7f8e1c1564d0 2024-04-25T00:01:53.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.381+0000 7f8e211ff640 1 -- 172.21.15.116:0/3402723490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e1c1564d0 msgr2=0x7f8e1c0c6e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:53.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.381+0000 7f8e211ff640 1 --2- 172.21.15.116:0/3402723490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e1c1564d0 0x7f8e1c0c6e40 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c067d90 tx=0x7f8e1401c8a0 comp rx=0 tx=0).stop 2024-04-25T00:01:53.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.381+0000 7f8e211ff640 1 -- 172.21.15.116:0/3402723490 shutdown_connections 2024-04-25T00:01:53.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.381+0000 7f8e211ff640 1 --2- 172.21.15.116:0/3402723490 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e1c1564d0 0x7f8e1c0c6e40 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:53.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.381+0000 7f8e211ff640 1 --2- 172.21.15.116:0/3402723490 >> v2:172.21.15.174:3300/0 conn(0x7f8e1c06c450 0x7f8e1c0c6040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:53.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.381+0000 7f8e211ff640 1 -- 172.21.15.116:0/3402723490 >> 172.21.15.116:0/3402723490 conn(0x7f8e1c0c7d10 msgr2=0x7f8e1c0c8120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:53.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.381+0000 7f8e211ff640 1 -- 172.21.15.116:0/3402723490 shutdown_connections 2024-04-25T00:01:53.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.381+0000 7f8e211ff640 1 -- 172.21.15.116:0/3402723490 wait complete. 2024-04-25T00:01:53.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.381+0000 7f8e211ff640 1 Processor -- start 2024-04-25T00:01:53.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.381+0000 7f8e211ff640 1 -- start start 2024-04-25T00:01:53.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.385+0000 7f8e211ff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8e1c06c450 0x7f8e1c15f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:53.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.385+0000 7f8e211ff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8e1c0c5c60 0x7f8e1c0aac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:53.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.385+0000 7f8e211ff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e1c1564d0 0x7f8e1c1ee1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:53.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.385+0000 7f8e211ff640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8e1c1533f0 con 0x7f8e1c1564d0 2024-04-25T00:01:53.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.385+0000 7f8e211ff640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8e1c153270 con 0x7f8e1c06c450 2024-04-25T00:01:53.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.385+0000 7f8e211ff640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f8e1c162a80 con 0x7f8e1c0c5c60 2024-04-25T00:01:53.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.385+0000 7f8e1b59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e1c1564d0 0x7f8e1c1ee1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:53.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.385+0000 7f8e1b59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e1c1564d0 0x7f8e1c1ee1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43596/0 (socket says 172.21.15.116:43596) 2024-04-25T00:01:53.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.385+0000 7f8e1b59e640 1 -- 172.21.15.116:0/1434472041 learned_addr learned my addr 172.21.15.116:0/1434472041 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:53.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.385+0000 7f8e1ad9d640 1 --2- 172.21.15.116:0/1434472041 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8e1c06c450 0x7f8e1c15f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:53.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.385+0000 7f8e1a59c640 1 --2- 172.21.15.116:0/1434472041 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8e1c0c5c60 0x7f8e1c0aac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:53.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.385+0000 7f8e1b59e640 1 -- 172.21.15.116:0/1434472041 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8e1c0c5c60 msgr2=0x7f8e1c0aac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:53.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.385+0000 7f8e1b59e640 1 --2- 172.21.15.116:0/1434472041 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8e1c0c5c60 0x7f8e1c0aac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:53.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.385+0000 7f8e1b59e640 1 -- 172.21.15.116:0/1434472041 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8e1c06c450 msgr2=0x7f8e1c15f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:53.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.385+0000 7f8e1b59e640 1 --2- 172.21.15.116:0/1434472041 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8e1c06c450 0x7f8e1c15f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:53.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.385+0000 7f8e1b59e640 1 -- 172.21.15.116:0/1434472041 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e1c1550d0 con 0x7f8e1c1564d0 2024-04-25T00:01:53.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.385+0000 7f8e1ad9d640 1 --2- 172.21.15.116:0/1434472041 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8e1c06c450 0x7f8e1c15f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:53.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.385+0000 7f8e1b59e640 1 --2- 172.21.15.116:0/1434472041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e1c1564d0 0x7f8e1c1ee1b0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f8e140194b0 tx=0x7f8e14019640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:53.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.385+0000 7f8e037fe640 1 -- 172.21.15.116:0/1434472041 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e140bd020 con 0x7f8e1c1564d0 2024-04-25T00:01:53.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.385+0000 7f8e037fe640 1 -- 172.21.15.116:0/1434472041 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e140b3040 con 0x7f8e1c1564d0 2024-04-25T00:01:53.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.385+0000 7f8e211ff640 1 -- 172.21.15.116:0/1434472041 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e1c1553b0 con 0x7f8e1c1564d0 2024-04-25T00:01:53.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.385+0000 7f8e037fe640 1 -- 172.21.15.116:0/1434472041 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e140b2050 con 0x7f8e1c1564d0 2024-04-25T00:01:53.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.385+0000 7f8e211ff640 1 -- 172.21.15.116:0/1434472041 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e1c155690 con 0x7f8e1c1564d0 2024-04-25T00:01:53.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.385+0000 7f8e211ff640 1 -- 172.21.15.116:0/1434472041 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e1c04f510 con 0x7f8e1c1564d0 2024-04-25T00:01:53.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.385+0000 7f8e037fe640 1 -- 172.21.15.116:0/1434472041 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8e14003ba0 con 0x7f8e1c1564d0 2024-04-25T00:01:53.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.385+0000 7f8e037fe640 1 --2- 172.21.15.116:0/1434472041 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8e080678c0 0x7f8e08069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:53.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.385+0000 7f8e037fe640 1 -- 172.21.15.116:0/1434472041 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6666+0+0 (secure 0 0 0) 0x7f8e14120c10 con 0x7f8e1c1564d0 2024-04-25T00:01:53.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.385+0000 7f8e1ad9d640 1 --2- 172.21.15.116:0/1434472041 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8e080678c0 0x7f8e08069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:53.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.389+0000 7f8e1ad9d640 1 --2- 172.21.15.116:0/1434472041 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8e080678c0 0x7f8e08069d80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c007fd0 tx=0x7f8e0c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:53.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.389+0000 7f8e037fe640 1 -- 172.21.15.116:0/1434472041 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f8e140c7020 con 0x7f8e1c1564d0 2024-04-25T00:01:53.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.537+0000 7f8e211ff640 1 -- 172.21.15.116:0/1434472041 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"} v 0) -- 0x7f8e1c0cadd0 con 0x7f8e1c1564d0 2024-04-25T00:01:53.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.537+0000 7f8e037fe640 1 -- 172.21.15.116:0/1434472041 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"}]=0 v29) ==== 99+0+11 (secure 0 0 0) 0x7f8e14016520 con 0x7f8e1c1564d0 2024-04-25T00:01:53.540 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-04-25T00:01:53.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.537+0000 7f8e211ff640 1 -- 172.21.15.116:0/1434472041 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8e080678c0 msgr2=0x7f8e08069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:53.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.537+0000 7f8e211ff640 1 --2- 172.21.15.116:0/1434472041 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8e080678c0 0x7f8e08069d80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c007fd0 tx=0x7f8e0c009040 comp rx=0 tx=0).stop 2024-04-25T00:01:53.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.537+0000 7f8e211ff640 1 -- 172.21.15.116:0/1434472041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e1c1564d0 msgr2=0x7f8e1c1ee1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:53.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.537+0000 7f8e211ff640 1 --2- 172.21.15.116:0/1434472041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e1c1564d0 0x7f8e1c1ee1b0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f8e140194b0 tx=0x7f8e14019640 comp rx=0 tx=0).stop 2024-04-25T00:01:53.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.537+0000 7f8e211ff640 1 -- 172.21.15.116:0/1434472041 shutdown_connections 2024-04-25T00:01:53.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.537+0000 7f8e211ff640 1 --2- 172.21.15.116:0/1434472041 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8e080678c0 0x7f8e08069d80 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:53.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.537+0000 7f8e211ff640 1 --2- 172.21.15.116:0/1434472041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e1c1564d0 0x7f8e1c1ee1b0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:53.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.537+0000 7f8e211ff640 1 --2- 172.21.15.116:0/1434472041 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8e1c0c5c60 0x7f8e1c0aac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:53.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.537+0000 7f8e211ff640 1 --2- 172.21.15.116:0/1434472041 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8e1c06c450 0x7f8e1c15f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:53.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.537+0000 7f8e211ff640 1 -- 172.21.15.116:0/1434472041 >> 172.21.15.116:0/1434472041 conn(0x7f8e1c0c7d10 msgr2=0x7f8e1c152110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:53.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.537+0000 7f8e211ff640 1 -- 172.21.15.116:0/1434472041 shutdown_connections 2024-04-25T00:01:53.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.537+0000 7f8e211ff640 1 -- 172.21.15.116:0/1434472041 wait complete. 2024-04-25T00:01:53.557 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_data pg_num 2024-04-25T00:01:53.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.681+0000 7f3f31d81640 1 Processor -- start 2024-04-25T00:01:53.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.681+0000 7f3f31d81640 1 -- start start 2024-04-25T00:01:53.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.681+0000 7f3f31d81640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f3f2c150b20 0x7f3f2c0c2030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:53.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.681+0000 7f3f31d81640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f2c156b00 0x7f3f2c156fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:53.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.681+0000 7f3f31d81640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f2c0aa710 con 0x7f3f2c06d550 2024-04-25T00:01:53.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.681+0000 7f3f31d81640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f3f2c0ad1b0 con 0x7f3f2c150b20 2024-04-25T00:01:53.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.681+0000 7f3f31d81640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f3f2c0aa9f0 con 0x7f3f2c156b00 2024-04-25T00:01:53.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.681+0000 7f3f22ffd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f3f2c06d550 0x7f3f2c0c2570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:47652/0 (socket says 172.21.15.116:47652) 2024-04-25T00:01:53.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.681+0000 7f3f22ffd640 1 -- 172.21.15.116:0/3916272552 learned_addr learned my addr 172.21.15.116:0/3916272552 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:53.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.681+0000 7f3f2bfff640 1 --2- 172.21.15.116:0/3916272552 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f2c156b00 0x7f3f2c156fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:53.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.681+0000 7f3f2b7fe640 1 --2- 172.21.15.116:0/3916272552 >> v2:172.21.15.174:3300/0 conn(0x7f3f2c150b20 0x7f3f2c0c2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:53.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.681+0000 7f3f2affd640 1 -- 172.21.15.116:0/3916272552 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1395140026 0 0) 0x7f3f2c0aa710 con 0x7f3f2c06d550 2024-04-25T00:01:53.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.681+0000 7f3f2affd640 1 -- 172.21.15.116:0/3916272552 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f18003530 con 0x7f3f2c06d550 2024-04-25T00:01:53.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.681+0000 7f3f2affd640 1 -- 172.21.15.116:0/3916272552 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2313677994 0 0) 0x7f3f18003530 con 0x7f3f2c06d550 2024-04-25T00:01:53.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.681+0000 7f3f2affd640 1 -- 172.21.15.116:0/3916272552 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f2c156b00 msgr2=0x7f3f2c156fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:53.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.681+0000 7f3f2affd640 1 --2- 172.21.15.116:0/3916272552 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f2c156b00 0x7f3f2c156fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:53.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.681+0000 7f3f2affd640 1 -- 172.21.15.116:0/3916272552 >> v2:172.21.15.174:3300/0 conn(0x7f3f2c150b20 msgr2=0x7f3f2c0c2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:53.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.681+0000 7f3f2affd640 1 --2- 172.21.15.116:0/3916272552 >> v2:172.21.15.174:3300/0 conn(0x7f3f2c150b20 0x7f3f2c0c2030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:53.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.681+0000 7f3f2affd640 1 -- 172.21.15.116:0/3916272552 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f2c0acb60 con 0x7f3f2c06d550 2024-04-25T00:01:53.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.681+0000 7f3f2affd640 1 -- 172.21.15.116:0/3916272552 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f3f14001680 con 0x7f3f2c06d550 2024-04-25T00:01:53.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.681+0000 7f3f2affd640 1 -- 172.21.15.116:0/3916272552 >> v1:172.21.15.116:6789/0 conn(0x7f3f2c06d550 legacy=0x7f3f2c0c2570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:53.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.681+0000 7f3f2affd640 1 --2- 172.21.15.116:0/3916272552 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f2c156b00 0x7f3f180039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:53.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.681+0000 7f3f2affd640 1 --2- 172.21.15.116:0/3916272552 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f18003f90 0x7f3f18004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:53.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.681+0000 7f3f2affd640 1 --2- 172.21.15.116:0/3916272552 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f18004d50 0x7f3f180051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:53.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.681+0000 7f3f2affd640 1 -- 172.21.15.116:0/3916272552 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3f2c0aa9f0 con 0x7f3f18004d50 2024-04-25T00:01:53.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.681+0000 7f3f2affd640 1 -- 172.21.15.116:0/3916272552 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3f1800ba80 con 0x7f3f18003f90 2024-04-25T00:01:53.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.681+0000 7f3f2affd640 1 -- 172.21.15.116:0/3916272552 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f3f1800bc80 con 0x7f3f2c156b00 2024-04-25T00:01:53.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.681+0000 7f3f2bfff640 1 --2- 172.21.15.116:0/3916272552 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f18004d50 0x7f3f180051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:53.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.681+0000 7f3f22ffd640 1 --2- 172.21.15.116:0/3916272552 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f18003f90 0x7f3f18004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:53.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.681+0000 7f3f2b7fe640 1 --2- 172.21.15.116:0/3916272552 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f2c156b00 0x7f3f180039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:53.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.681+0000 7f3f2bfff640 1 -- 172.21.15.116:0/3916272552 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f2c156b00 msgr2=0x7f3f180039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:53.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.681+0000 7f3f2bfff640 1 --2- 172.21.15.116:0/3916272552 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f2c156b00 0x7f3f180039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:53.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.681+0000 7f3f2bfff640 1 -- 172.21.15.116:0/3916272552 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f18003f90 msgr2=0x7f3f18004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:53.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.681+0000 7f3f2bfff640 1 --2- 172.21.15.116:0/3916272552 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f18003f90 0x7f3f18004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:53.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.681+0000 7f3f2bfff640 1 -- 172.21.15.116:0/3916272552 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f18003530 con 0x7f3f18004d50 2024-04-25T00:01:53.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.681+0000 7f3f2b7fe640 1 --2- 172.21.15.116:0/3916272552 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f2c156b00 0x7f3f180039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:53.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.685+0000 7f3f2bfff640 1 --2- 172.21.15.116:0/3916272552 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f18004d50 0x7f3f180051d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f3f24009ac0 tx=0x7f3f2401ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29ef17f1786100d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:53.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.685+0000 7f3f2affd640 1 -- 172.21.15.116:0/3916272552 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f24009690 con 0x7f3f18004d50 2024-04-25T00:01:53.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.685+0000 7f3f2affd640 1 -- 172.21.15.116:0/3916272552 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f240a6500 con 0x7f3f18004d50 2024-04-25T00:01:53.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.685+0000 7f3f2affd640 1 -- 172.21.15.116:0/3916272552 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f240a68d0 con 0x7f3f18004d50 2024-04-25T00:01:53.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.685+0000 7f3f31d81640 1 -- 172.21.15.116:0/3916272552 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f18004d50 msgr2=0x7f3f180051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:53.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.685+0000 7f3f31d81640 1 --2- 172.21.15.116:0/3916272552 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f18004d50 0x7f3f180051d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f3f24009ac0 tx=0x7f3f2401ca50 comp rx=0 tx=0).stop 2024-04-25T00:01:53.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.685+0000 7f3f2bfff640 1 -- 172.21.15.116:0/3916272552 reap_dead start 2024-04-25T00:01:53.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.685+0000 7f3f31d81640 1 -- 172.21.15.116:0/3916272552 shutdown_connections 2024-04-25T00:01:53.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.685+0000 7f3f31d81640 1 -- 172.21.15.116:0/3916272552 >> 172.21.15.116:0/3916272552 conn(0x7f3f2c0c73e0 msgr2=0x7f3f2c0c77f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:53.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.685+0000 7f3f31d81640 1 -- 172.21.15.116:0/3916272552 shutdown_connections 2024-04-25T00:01:53.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.685+0000 7f3f31d81640 1 -- 172.21.15.116:0/3916272552 wait complete. 2024-04-25T00:01:53.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.685+0000 7f3f31d81640 1 Processor -- start 2024-04-25T00:01:53.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.685+0000 7f3f31d81640 1 -- start start 2024-04-25T00:01:53.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.685+0000 7f3f31d81640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f18004d50 0x7f3f2c1ef7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:53.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.685+0000 7f3f31d81640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f2c153ba0 0x7f3f2c14af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:53.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.685+0000 7f3f31d81640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f2c151b60 0x7f3f2c06d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:53.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.685+0000 7f3f31d81640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3f2c15de90 con 0x7f3f2c153ba0 2024-04-25T00:01:53.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.685+0000 7f3f31d81640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3f2c0aa3c0 con 0x7f3f18004d50 2024-04-25T00:01:53.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.685+0000 7f3f31d81640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f3f2c15e010 con 0x7f3f2c151b60 2024-04-25T00:01:53.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.685+0000 7f3f22ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f2c153ba0 0x7f3f2c14af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:53.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.685+0000 7f3f22ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f2c153ba0 0x7f3f2c14af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43620/0 (socket says 172.21.15.116:43620) 2024-04-25T00:01:53.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.685+0000 7f3f22ffd640 1 -- 172.21.15.116:0/3081861805 learned_addr learned my addr 172.21.15.116:0/3081861805 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:53.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.685+0000 7f3f2b7fe640 1 --2- 172.21.15.116:0/3081861805 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f18004d50 0x7f3f2c1ef7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:53.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.685+0000 7f3f22ffd640 1 -- 172.21.15.116:0/3081861805 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f2c151b60 msgr2=0x7f3f2c06d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:53.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.685+0000 7f3f22ffd640 1 --2- 172.21.15.116:0/3081861805 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f2c151b60 0x7f3f2c06d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:53.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.685+0000 7f3f22ffd640 1 -- 172.21.15.116:0/3081861805 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f18004d50 msgr2=0x7f3f2c1ef7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:53.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.685+0000 7f3f22ffd640 1 --2- 172.21.15.116:0/3081861805 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f18004d50 0x7f3f2c1ef7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:53.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.685+0000 7f3f22ffd640 1 -- 172.21.15.116:0/3081861805 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f2c151380 con 0x7f3f2c153ba0 2024-04-25T00:01:53.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.685+0000 7f3f22ffd640 1 --2- 172.21.15.116:0/3081861805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f2c153ba0 0x7f3f2c14af30 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f3f1400e700 tx=0x7f3f1400ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:53.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.689+0000 7f3f28ff9640 1 -- 172.21.15.116:0/3081861805 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f1400c6c0 con 0x7f3f2c153ba0 2024-04-25T00:01:53.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.689+0000 7f3f28ff9640 1 -- 172.21.15.116:0/3081861805 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f14018070 con 0x7f3f2c153ba0 2024-04-25T00:01:53.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.689+0000 7f3f31d81640 1 -- 172.21.15.116:0/3081861805 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f2c0c6120 con 0x7f3f2c153ba0 2024-04-25T00:01:53.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.689+0000 7f3f28ff9640 1 -- 172.21.15.116:0/3081861805 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f1400cae0 con 0x7f3f2c153ba0 2024-04-25T00:01:53.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.689+0000 7f3f31d81640 1 -- 172.21.15.116:0/3081861805 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f2c154b40 con 0x7f3f2c153ba0 2024-04-25T00:01:53.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.689+0000 7f3f31d81640 1 -- 172.21.15.116:0/3081861805 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f2c04f510 con 0x7f3f2c153ba0 2024-04-25T00:01:53.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.689+0000 7f3f28ff9640 1 -- 172.21.15.116:0/3081861805 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3f1400cc80 con 0x7f3f2c153ba0 2024-04-25T00:01:53.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.689+0000 7f3f28ff9640 1 --2- 172.21.15.116:0/3081861805 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3f0c0678f0 0x7f3f0c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:53.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.689+0000 7f3f28ff9640 1 -- 172.21.15.116:0/3081861805 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6666+0+0 (secure 0 0 0) 0x7f3f140859f0 con 0x7f3f2c153ba0 2024-04-25T00:01:53.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.689+0000 7f3f2b7fe640 1 --2- 172.21.15.116:0/3081861805 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3f0c0678f0 0x7f3f0c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:53.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.689+0000 7f3f2b7fe640 1 --2- 172.21.15.116:0/3081861805 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3f0c0678f0 0x7f3f0c069db0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c003bf0 tx=0x7f3f1c00b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:53.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.693+0000 7f3f28ff9640 1 -- 172.21.15.116:0/3081861805 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f3f14056200 con 0x7f3f2c153ba0 2024-04-25T00:01:53.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.841+0000 7f3f31d81640 1 -- 172.21.15.116:0/3081861805 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"} v 0) -- 0x7f3f2c0c1280 con 0x7f3f2c153ba0 2024-04-25T00:01:53.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.841+0000 7f3f28ff9640 1 -- 172.21.15.116:0/3081861805 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"}]=0 v29) ==== 95+0+11 (secure 0 0 0) 0x7f3f14055b00 con 0x7f3f2c153ba0 2024-04-25T00:01:53.845 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-04-25T00:01:53.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.845+0000 7f3f31d81640 1 -- 172.21.15.116:0/3081861805 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3f0c0678f0 msgr2=0x7f3f0c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:53.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.845+0000 7f3f31d81640 1 --2- 172.21.15.116:0/3081861805 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3f0c0678f0 0x7f3f0c069db0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c003bf0 tx=0x7f3f1c00b040 comp rx=0 tx=0).stop 2024-04-25T00:01:53.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.845+0000 7f3f31d81640 1 -- 172.21.15.116:0/3081861805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f2c153ba0 msgr2=0x7f3f2c14af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:53.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.845+0000 7f3f31d81640 1 --2- 172.21.15.116:0/3081861805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f2c153ba0 0x7f3f2c14af30 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f3f1400e700 tx=0x7f3f1400ebc0 comp rx=0 tx=0).stop 2024-04-25T00:01:53.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.845+0000 7f3f31d81640 1 -- 172.21.15.116:0/3081861805 shutdown_connections 2024-04-25T00:01:53.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.845+0000 7f3f31d81640 1 --2- 172.21.15.116:0/3081861805 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3f0c0678f0 0x7f3f0c069db0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:53.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.845+0000 7f3f31d81640 1 --2- 172.21.15.116:0/3081861805 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f2c151b60 0x7f3f2c06d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:53.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.845+0000 7f3f31d81640 1 --2- 172.21.15.116:0/3081861805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f2c153ba0 0x7f3f2c14af30 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:53.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.845+0000 7f3f31d81640 1 --2- 172.21.15.116:0/3081861805 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f18004d50 0x7f3f2c1ef7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:53.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.845+0000 7f3f31d81640 1 -- 172.21.15.116:0/3081861805 >> 172.21.15.116:0/3081861805 conn(0x7f3f2c0c73e0 msgr2=0x7f3f2c06ee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:53.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.845+0000 7f3f31d81640 1 -- 172.21.15.116:0/3081861805 shutdown_connections 2024-04-25T00:01:53.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.845+0000 7f3f31d81640 1 -- 172.21.15.116:0/3081861805 wait complete. 2024-04-25T00:01:53.863 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_metadata pg_num 2024-04-25T00:01:53.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.969+0000 7efdfd5ec640 1 Processor -- start 2024-04-25T00:01:53.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.969+0000 7efdfd5ec640 1 -- start start 2024-04-25T00:01:53.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.973+0000 7efdfd5ec640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7efdf80a9630 0x7efdf8156e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:53.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.973+0000 7efdfd5ec640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efdf8157340 0x7efdf81577d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:53.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.973+0000 7efdfd5ec640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efdf80a9cb0 con 0x7efdf8150d60 2024-04-25T00:01:53.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.973+0000 7efdfd5ec640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7efdf80a9f90 con 0x7efdf80a9630 2024-04-25T00:01:53.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.973+0000 7efdfd5ec640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7efdf80aa270 con 0x7efdf8157340 2024-04-25T00:01:53.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.973+0000 7efdeffff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7efdf80a9630 0x7efdf8156e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:53.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.973+0000 7efdeffff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7efdf80a9630 0x7efdf8156e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43626/0 (socket says 172.21.15.116:43626) 2024-04-25T00:01:53.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.973+0000 7efdeffff640 1 -- 172.21.15.116:0/1499257050 learned_addr learned my addr 172.21.15.116:0/1499257050 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:53.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.973+0000 7efdf77fe640 1 --2- 172.21.15.116:0/1499257050 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efdf8157340 0x7efdf81577d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:53.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.973+0000 7efdeffff640 1 -- 172.21.15.116:0/1499257050 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efdf8157340 msgr2=0x7efdf81577d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:53.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.973+0000 7efdeffff640 1 --2- 172.21.15.116:0/1499257050 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efdf8157340 0x7efdf81577d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:53.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.973+0000 7efdeffff640 1 -- 172.21.15.116:0/1499257050 >> v1:172.21.15.174:6789/0 conn(0x7efdf8150d60 legacy=0x7efdf80c2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:53.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.973+0000 7efdeffff640 1 -- 172.21.15.116:0/1499257050 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdf80aa550 con 0x7efdf80a9630 2024-04-25T00:01:53.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.973+0000 7efdeffff640 1 --2- 172.21.15.116:0/1499257050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdf80a9630 0x7efdf8156e00 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7efde0009870 tx=0x7efde001c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bafe69dc3dd4aca server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:53.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.973+0000 7efdf67fc640 1 -- 172.21.15.116:0/1499257050 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efde00b0070 con 0x7efdf80a9630 2024-04-25T00:01:53.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.973+0000 7efdf67fc640 1 -- 172.21.15.116:0/1499257050 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efde00ab600 con 0x7efdf80a9630 2024-04-25T00:01:53.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.973+0000 7efdf67fc640 1 -- 172.21.15.116:0/1499257050 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efde00ab9d0 con 0x7efdf80a9630 2024-04-25T00:01:53.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.973+0000 7efdfd5ec640 1 -- 172.21.15.116:0/1499257050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdf80a9630 msgr2=0x7efdf8156e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:53.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.973+0000 7efdfd5ec640 1 --2- 172.21.15.116:0/1499257050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdf80a9630 0x7efdf8156e00 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7efde0009870 tx=0x7efde001c3d0 comp rx=0 tx=0).stop 2024-04-25T00:01:53.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.973+0000 7efdfd5ec640 1 -- 172.21.15.116:0/1499257050 shutdown_connections 2024-04-25T00:01:53.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.973+0000 7efdfd5ec640 1 --2- 172.21.15.116:0/1499257050 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efdf8157340 0x7efdf81577d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:53.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.973+0000 7efdfd5ec640 1 --2- 172.21.15.116:0/1499257050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdf80a9630 0x7efdf8156e00 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:53.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.973+0000 7efdfd5ec640 1 -- 172.21.15.116:0/1499257050 >> 172.21.15.116:0/1499257050 conn(0x7efdf80c3a20 msgr2=0x7efdf80c3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:53.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.973+0000 7efdfd5ec640 1 -- 172.21.15.116:0/1499257050 shutdown_connections 2024-04-25T00:01:53.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.973+0000 7efdfd5ec640 1 -- 172.21.15.116:0/1499257050 wait complete. 2024-04-25T00:01:53.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.977+0000 7efdfd5ec640 1 Processor -- start 2024-04-25T00:01:53.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.977+0000 7efdfd5ec640 1 -- start start 2024-04-25T00:01:53.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.977+0000 7efdfd5ec640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efdf80a9630 0x7efdf8144a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:53.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.977+0000 7efdfd5ec640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efdf8150d60 0x7efdf815c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:53.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.977+0000 7efdfd5ec640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdf8157340 0x7efdf81ee5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:53.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.977+0000 7efdfd5ec640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7efdf8071fd0 con 0x7efdf8157340 2024-04-25T00:01:53.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.977+0000 7efdfd5ec640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7efdf80ab7a0 con 0x7efdf80a9630 2024-04-25T00:01:53.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.977+0000 7efdfd5ec640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7efdf8072150 con 0x7efdf8150d60 2024-04-25T00:01:53.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.977+0000 7efdf77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdf8157340 0x7efdf81ee5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:53.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.977+0000 7efdf77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdf8157340 0x7efdf81ee5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43640/0 (socket says 172.21.15.116:43640) 2024-04-25T00:01:53.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.977+0000 7efdf77fe640 1 -- 172.21.15.116:0/3832814028 learned_addr learned my addr 172.21.15.116:0/3832814028 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:53.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.977+0000 7efdf6ffd640 1 --2- 172.21.15.116:0/3832814028 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efdf80a9630 0x7efdf8144a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:53.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.977+0000 7efdeffff640 1 --2- 172.21.15.116:0/3832814028 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efdf8150d60 0x7efdf815c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:53.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.977+0000 7efdf77fe640 1 -- 172.21.15.116:0/3832814028 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efdf8150d60 msgr2=0x7efdf815c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:53.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.977+0000 7efdf77fe640 1 --2- 172.21.15.116:0/3832814028 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efdf8150d60 0x7efdf815c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:53.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.977+0000 7efdf77fe640 1 -- 172.21.15.116:0/3832814028 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efdf80a9630 msgr2=0x7efdf8144a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:53.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.977+0000 7efdf77fe640 1 --2- 172.21.15.116:0/3832814028 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efdf80a9630 0x7efdf8144a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:53.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.977+0000 7efdf77fe640 1 -- 172.21.15.116:0/3832814028 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdf806e970 con 0x7efdf8157340 2024-04-25T00:01:53.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.977+0000 7efdf6ffd640 1 --2- 172.21.15.116:0/3832814028 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efdf80a9630 0x7efdf8144a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:01:53.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.977+0000 7efdf77fe640 1 --2- 172.21.15.116:0/3832814028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdf8157340 0x7efdf81ee5d0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7efdf000cda0 tx=0x7efdf000a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:53.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.977+0000 7efdef7fe640 1 -- 172.21.15.116:0/3832814028 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdf0003ba0 con 0x7efdf8157340 2024-04-25T00:01:53.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.977+0000 7efdef7fe640 1 -- 172.21.15.116:0/3832814028 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efdf0003d40 con 0x7efdf8157340 2024-04-25T00:01:53.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.977+0000 7efdfd5ec640 1 -- 172.21.15.116:0/3832814028 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdf806ec50 con 0x7efdf8157340 2024-04-25T00:01:53.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.977+0000 7efdef7fe640 1 -- 172.21.15.116:0/3832814028 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdf0004240 con 0x7efdf8157340 2024-04-25T00:01:53.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.977+0000 7efdfd5ec640 1 -- 172.21.15.116:0/3832814028 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdf813be00 con 0x7efdf8157340 2024-04-25T00:01:53.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.977+0000 7efdfd5ec640 1 -- 172.21.15.116:0/3832814028 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efdf81edea0 con 0x7efdf8157340 2024-04-25T00:01:53.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.981+0000 7efdef7fe640 1 -- 172.21.15.116:0/3832814028 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7efdf00043e0 con 0x7efdf8157340 2024-04-25T00:01:53.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.981+0000 7efdef7fe640 1 --2- 172.21.15.116:0/3832814028 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efdd80678c0 0x7efdd8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:53.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.981+0000 7efdef7fe640 1 -- 172.21.15.116:0/3832814028 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6666+0+0 (secure 0 0 0) 0x7efdf0086fb0 con 0x7efdf8157340 2024-04-25T00:01:53.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.981+0000 7efdf6ffd640 1 --2- 172.21.15.116:0/3832814028 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efdd80678c0 0x7efdd8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:53.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.981+0000 7efdf6ffd640 1 --2- 172.21.15.116:0/3832814028 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efdd80678c0 0x7efdd8069d80 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7efde80098b0 tx=0x7efde8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:53.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:53.985+0000 7efdef7fe640 1 -- 172.21.15.116:0/3832814028 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7efdf00577a0 con 0x7efdf8157340 2024-04-25T00:01:54.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.133+0000 7efdfd5ec640 1 -- 172.21.15.116:0/3832814028 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"} v 0) -- 0x7efdf804f510 con 0x7efdf8157340 2024-04-25T00:01:54.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.133+0000 7efdef7fe640 1 -- 172.21.15.116:0/3832814028 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"}]=0 v29) ==== 99+0+11 (secure 0 0 0) 0x7efdf00570a0 con 0x7efdf8157340 2024-04-25T00:01:54.137 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-04-25T00:01:54.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.137+0000 7efdfd5ec640 1 -- 172.21.15.116:0/3832814028 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efdd80678c0 msgr2=0x7efdd8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:54.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.137+0000 7efdfd5ec640 1 --2- 172.21.15.116:0/3832814028 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efdd80678c0 0x7efdd8069d80 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7efde80098b0 tx=0x7efde8008040 comp rx=0 tx=0).stop 2024-04-25T00:01:54.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.137+0000 7efdfd5ec640 1 -- 172.21.15.116:0/3832814028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdf8157340 msgr2=0x7efdf81ee5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:54.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.137+0000 7efdfd5ec640 1 --2- 172.21.15.116:0/3832814028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdf8157340 0x7efdf81ee5d0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7efdf000cda0 tx=0x7efdf000a660 comp rx=0 tx=0).stop 2024-04-25T00:01:54.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.137+0000 7efdfd5ec640 1 -- 172.21.15.116:0/3832814028 shutdown_connections 2024-04-25T00:01:54.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.137+0000 7efdfd5ec640 1 --2- 172.21.15.116:0/3832814028 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efdd80678c0 0x7efdd8069d80 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:54.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.137+0000 7efdfd5ec640 1 --2- 172.21.15.116:0/3832814028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efdf8157340 0x7efdf81ee5d0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:54.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.137+0000 7efdfd5ec640 1 --2- 172.21.15.116:0/3832814028 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efdf8150d60 0x7efdf815c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:54.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.137+0000 7efdfd5ec640 1 --2- 172.21.15.116:0/3832814028 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efdf80a9630 0x7efdf8144a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:54.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.137+0000 7efdfd5ec640 1 -- 172.21.15.116:0/3832814028 >> 172.21.15.116:0/3832814028 conn(0x7efdf80c3a20 msgr2=0x7efdf80accc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:54.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.137+0000 7efdfd5ec640 1 -- 172.21.15.116:0/3832814028 shutdown_connections 2024-04-25T00:01:54.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.137+0000 7efdfd5ec640 1 -- 172.21.15.116:0/3832814028 wait complete. 2024-04-25T00:01:54.154 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_data pg_num 2024-04-25T00:01:54.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.273+0000 7f6d9892f640 1 Processor -- start 2024-04-25T00:01:54.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.273+0000 7f6d9892f640 1 -- start start 2024-04-25T00:01:54.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.273+0000 7f6d9892f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6d94153130 0x7f6d941562e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:54.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.273+0000 7f6d9892f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6d940c9260 0x7f6d940c96c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:54.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.273+0000 7f6d9892f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d94156820 con 0x7f6d940ca1b0 2024-04-25T00:01:54.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.273+0000 7f6d9892f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f6d940ca820 con 0x7f6d940c9260 2024-04-25T00:01:54.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.273+0000 7f6d9892f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6d94155e50 con 0x7f6d94153130 2024-04-25T00:01:54.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.273+0000 7f6d92d9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6d940c9260 0x7f6d940c96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:54.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.273+0000 7f6d91d9b640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f6d940ca1b0 0x7f6d9406bdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:47656/0 (socket says 172.21.15.116:47656) 2024-04-25T00:01:54.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.273+0000 7f6d91d9b640 1 -- 172.21.15.116:0/212725652 learned_addr learned my addr 172.21.15.116:0/212725652 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:54.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.273+0000 7f6d9259c640 1 --2- 172.21.15.116:0/212725652 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6d94153130 0x7f6d941562e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:54.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.273+0000 7f6d92d9d640 1 -- 172.21.15.116:0/212725652 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6d94153130 msgr2=0x7f6d941562e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:54.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.273+0000 7f6d92d9d640 1 --2- 172.21.15.116:0/212725652 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6d94153130 0x7f6d941562e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:54.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.273+0000 7f6d92d9d640 1 -- 172.21.15.116:0/212725652 >> v1:172.21.15.116:6789/0 conn(0x7f6d940ca1b0 legacy=0x7f6d9406bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:54.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.273+0000 7f6d9159a640 1 -- 172.21.15.116:0/212725652 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4066264113 0 0) 0x7f6d94156820 con 0x7f6d940ca1b0 2024-04-25T00:01:54.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.273+0000 7f6d92d9d640 1 -- 172.21.15.116:0/212725652 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d940caa70 con 0x7f6d940c9260 2024-04-25T00:01:54.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.277+0000 7f6d92d9d640 1 --2- 172.21.15.116:0/212725652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d940c9260 0x7f6d940c96c0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c009d90 tx=0x7f6d8c01c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b11530a688080676 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:54.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.277+0000 7f6d9159a640 1 -- 172.21.15.116:0/212725652 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d8c0b2070 con 0x7f6d940c9260 2024-04-25T00:01:54.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.277+0000 7f6d9159a640 1 -- 172.21.15.116:0/212725652 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d8c0a5800 con 0x7f6d940c9260 2024-04-25T00:01:54.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.277+0000 7f6d9159a640 1 -- 172.21.15.116:0/212725652 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d8c0a5bd0 con 0x7f6d940c9260 2024-04-25T00:01:54.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.277+0000 7f6d9892f640 1 -- 172.21.15.116:0/212725652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d940c9260 msgr2=0x7f6d940c96c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:54.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.277+0000 7f6d9892f640 1 --2- 172.21.15.116:0/212725652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d940c9260 0x7f6d940c96c0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c009d90 tx=0x7f6d8c01c850 comp rx=0 tx=0).stop 2024-04-25T00:01:54.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.277+0000 7f6d9892f640 1 -- 172.21.15.116:0/212725652 shutdown_connections 2024-04-25T00:01:54.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.277+0000 7f6d9892f640 1 --2- 172.21.15.116:0/212725652 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d940c9260 0x7f6d940c96c0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:54.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.277+0000 7f6d9892f640 1 --2- 172.21.15.116:0/212725652 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6d94153130 0x7f6d941562e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:54.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.277+0000 7f6d9892f640 1 -- 172.21.15.116:0/212725652 >> 172.21.15.116:0/212725652 conn(0x7f6d940cb310 msgr2=0x7f6d940cb720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:54.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.277+0000 7f6d9892f640 1 -- 172.21.15.116:0/212725652 shutdown_connections 2024-04-25T00:01:54.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.277+0000 7f6d9892f640 1 -- 172.21.15.116:0/212725652 wait complete. 2024-04-25T00:01:54.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.277+0000 7f6d9892f640 1 Processor -- start 2024-04-25T00:01:54.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.277+0000 7f6d9892f640 1 -- start start 2024-04-25T00:01:54.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.277+0000 7f6d9892f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6d940c9260 0x7f6d941f18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:54.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.277+0000 7f6d9892f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6d940ca1b0 0x7f6d941f1de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:54.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.277+0000 7f6d9892f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d94153130 0x7f6d940c4120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:54.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.277+0000 7f6d9892f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6d940c9d80 con 0x7f6d94153130 2024-04-25T00:01:54.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.277+0000 7f6d9892f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6d940c9c00 con 0x7f6d940c9260 2024-04-25T00:01:54.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.277+0000 7f6d9892f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6d94162990 con 0x7f6d940ca1b0 2024-04-25T00:01:54.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.277+0000 7f6d92d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d94153130 0x7f6d940c4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:54.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.277+0000 7f6d92d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d94153130 0x7f6d940c4120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43668/0 (socket says 172.21.15.116:43668) 2024-04-25T00:01:54.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.277+0000 7f6d92d9d640 1 -- 172.21.15.116:0/563833203 learned_addr learned my addr 172.21.15.116:0/563833203 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:54.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.277+0000 7f6d9259c640 1 --2- 172.21.15.116:0/563833203 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6d940c9260 0x7f6d941f18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:54.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.281+0000 7f6d91d9b640 1 --2- 172.21.15.116:0/563833203 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6d940ca1b0 0x7f6d941f1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:54.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.281+0000 7f6d92d9d640 1 -- 172.21.15.116:0/563833203 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6d940ca1b0 msgr2=0x7f6d941f1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:54.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.281+0000 7f6d92d9d640 1 --2- 172.21.15.116:0/563833203 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6d940ca1b0 0x7f6d941f1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:54.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.281+0000 7f6d92d9d640 1 -- 172.21.15.116:0/563833203 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6d940c9260 msgr2=0x7f6d941f18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:54.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.281+0000 7f6d92d9d640 1 --2- 172.21.15.116:0/563833203 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6d940c9260 0x7f6d941f18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:54.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.281+0000 7f6d92d9d640 1 -- 172.21.15.116:0/563833203 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d941f2320 con 0x7f6d94153130 2024-04-25T00:01:54.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.281+0000 7f6d92d9d640 1 --2- 172.21.15.116:0/563833203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d94153130 0x7f6d940c4120 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c0b0040 tx=0x7f6d8c0195f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:54.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.281+0000 7f6d82ffd640 1 -- 172.21.15.116:0/563833203 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d8c0b2050 con 0x7f6d94153130 2024-04-25T00:01:54.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.281+0000 7f6d82ffd640 1 -- 172.21.15.116:0/563833203 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d8c0b3040 con 0x7f6d94153130 2024-04-25T00:01:54.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.281+0000 7f6d9892f640 1 -- 172.21.15.116:0/563833203 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d94159890 con 0x7f6d94153130 2024-04-25T00:01:54.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.281+0000 7f6d82ffd640 1 -- 172.21.15.116:0/563833203 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d8c0ad850 con 0x7f6d94153130 2024-04-25T00:01:54.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.281+0000 7f6d9892f640 1 -- 172.21.15.116:0/563833203 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d940c22b0 con 0x7f6d94153130 2024-04-25T00:01:54.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.281+0000 7f6d9892f640 1 -- 172.21.15.116:0/563833203 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d9404f510 con 0x7f6d94153130 2024-04-25T00:01:54.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.281+0000 7f6d82ffd640 1 -- 172.21.15.116:0/563833203 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6d8c0c0020 con 0x7f6d94153130 2024-04-25T00:01:54.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.281+0000 7f6d82ffd640 1 --2- 172.21.15.116:0/563833203 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6d740678c0 0x7f6d74069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:54.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.281+0000 7f6d82ffd640 1 -- 172.21.15.116:0/563833203 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6666+0+0 (secure 0 0 0) 0x7f6d8c120910 con 0x7f6d94153130 2024-04-25T00:01:54.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.281+0000 7f6d9259c640 1 --2- 172.21.15.116:0/563833203 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6d740678c0 0x7f6d74069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:54.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.281+0000 7f6d9259c640 1 --2- 172.21.15.116:0/563833203 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6d740678c0 0x7f6d74069d80 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6d8400b930 tx=0x7f6d84008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:54.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.285+0000 7f6d82ffd640 1 -- 172.21.15.116:0/563833203 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f6d941f2320 con 0x7f6d94153130 2024-04-25T00:01:54.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.433+0000 7f6d9892f640 1 -- 172.21.15.116:0/563833203 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"} v 0) -- 0x7f6d9406da40 con 0x7f6d94153130 2024-04-25T00:01:54.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.433+0000 7f6d82ffd640 1 -- 172.21.15.116:0/563833203 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"}]=0 v29) ==== 95+0+11 (secure 0 0 0) 0x7f6d8c016560 con 0x7f6d94153130 2024-04-25T00:01:54.437 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-04-25T00:01:54.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.437+0000 7f6d9892f640 1 -- 172.21.15.116:0/563833203 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6d740678c0 msgr2=0x7f6d74069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:54.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.437+0000 7f6d9892f640 1 --2- 172.21.15.116:0/563833203 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6d740678c0 0x7f6d74069d80 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6d8400b930 tx=0x7f6d84008040 comp rx=0 tx=0).stop 2024-04-25T00:01:54.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.437+0000 7f6d9892f640 1 -- 172.21.15.116:0/563833203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d94153130 msgr2=0x7f6d940c4120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:54.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.437+0000 7f6d9892f640 1 --2- 172.21.15.116:0/563833203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d94153130 0x7f6d940c4120 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c0b0040 tx=0x7f6d8c0195f0 comp rx=0 tx=0).stop 2024-04-25T00:01:54.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.437+0000 7f6d9892f640 1 -- 172.21.15.116:0/563833203 shutdown_connections 2024-04-25T00:01:54.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.437+0000 7f6d9892f640 1 --2- 172.21.15.116:0/563833203 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6d740678c0 0x7f6d74069d80 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:54.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.437+0000 7f6d9892f640 1 --2- 172.21.15.116:0/563833203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6d94153130 0x7f6d940c4120 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:54.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.437+0000 7f6d9892f640 1 --2- 172.21.15.116:0/563833203 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6d940ca1b0 0x7f6d941f1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:54.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.437+0000 7f6d9892f640 1 --2- 172.21.15.116:0/563833203 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6d940c9260 0x7f6d941f18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:54.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.437+0000 7f6d9892f640 1 -- 172.21.15.116:0/563833203 >> 172.21.15.116:0/563833203 conn(0x7f6d940cb310 msgr2=0x7f6d9406cdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:54.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.437+0000 7f6d9892f640 1 -- 172.21.15.116:0/563833203 shutdown_connections 2024-04-25T00:01:54.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.437+0000 7f6d9892f640 1 -- 172.21.15.116:0/563833203 wait complete. 2024-04-25T00:01:54.455 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:01:54.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.565+0000 7ff01a0f1640 1 Processor -- start 2024-04-25T00:01:54.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.565+0000 7ff01a0f1640 1 -- start start 2024-04-25T00:01:54.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.565+0000 7ff01a0f1640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff0140a99f0 0x7ff014153ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:54.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.565+0000 7ff01a0f1640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0140c3910 con 0x7ff0140a94a0 2024-04-25T00:01:54.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.565+0000 7ff01a0f1640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff014154530 con 0x7ff014150d20 2024-04-25T00:01:54.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.565+0000 7ff01a0f1640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff0140c16e0 con 0x7ff0140a99f0 2024-04-25T00:01:54.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.565+0000 7ff012ffd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7ff0140a94a0 0x7ff0141538e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:47668/0 (socket says 172.21.15.116:47668) 2024-04-25T00:01:54.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.565+0000 7ff012ffd640 1 -- 172.21.15.116:0/2284337399 learned_addr learned my addr 172.21.15.116:0/2284337399 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:54.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.565+0000 7ff0127fc640 1 -- 172.21.15.116:0/2284337399 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3941346999 0 0) 0x7ff0140c3910 con 0x7ff0140a94a0 2024-04-25T00:01:54.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.565+0000 7ff0127fc640 1 -- 172.21.15.116:0/2284337399 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feffc003530 con 0x7ff0140a94a0 2024-04-25T00:01:54.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.565+0000 7ff013fff640 1 --2- 172.21.15.116:0/2284337399 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff0140a99f0 0x7ff014153ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:54.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.565+0000 7ff0127fc640 1 -- 172.21.15.116:0/2284337399 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2954549026 0 0) 0x7feffc003530 con 0x7ff0140a94a0 2024-04-25T00:01:54.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.565+0000 7ff0127fc640 1 -- 172.21.15.116:0/2284337399 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff0140a99f0 msgr2=0x7ff014153ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:54.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.565+0000 7ff0127fc640 1 --2- 172.21.15.116:0/2284337399 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff0140a99f0 0x7ff014153ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:54.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.565+0000 7ff0127fc640 1 -- 172.21.15.116:0/2284337399 >> v1:172.21.15.174:6789/0 conn(0x7ff014150d20 legacy=0x7ff0140c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:54.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.565+0000 7ff0127fc640 1 -- 172.21.15.116:0/2284337399 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0140c1400 con 0x7ff0140a94a0 2024-04-25T00:01:54.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.565+0000 7ff0127fc640 1 -- 172.21.15.116:0/2284337399 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7ff008001680 con 0x7ff0140a94a0 2024-04-25T00:01:54.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.565+0000 7ff0127fc640 1 -- 172.21.15.116:0/2284337399 >> v1:172.21.15.116:6789/0 conn(0x7ff0140a94a0 legacy=0x7ff0141538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:54.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.565+0000 7ff0127fc640 1 --2- 172.21.15.116:0/2284337399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feffc0039f0 0x7feffc003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:54.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.565+0000 7ff0127fc640 1 --2- 172.21.15.116:0/2284337399 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feffc004810 0x7feffc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:54.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.565+0000 7ff0127fc640 1 --2- 172.21.15.116:0/2284337399 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff0140a99f0 0x7feffc005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:54.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.565+0000 7ff0127fc640 1 -- 172.21.15.116:0/2284337399 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff0140c16e0 con 0x7feffc0039f0 2024-04-25T00:01:54.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.565+0000 7ff0127fc640 1 -- 172.21.15.116:0/2284337399 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7feffc00bb00 con 0x7feffc004810 2024-04-25T00:01:54.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.565+0000 7ff0127fc640 1 -- 172.21.15.116:0/2284337399 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7feffc00bd00 con 0x7ff0140a99f0 2024-04-25T00:01:54.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.565+0000 7ff0137fe640 1 --2- 172.21.15.116:0/2284337399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feffc0039f0 0x7feffc003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:54.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.565+0000 7ff012ffd640 1 --2- 172.21.15.116:0/2284337399 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feffc004810 0x7feffc004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:54.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.565+0000 7ff013fff640 1 --2- 172.21.15.116:0/2284337399 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff0140a99f0 0x7feffc005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:54.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.565+0000 7ff0137fe640 1 -- 172.21.15.116:0/2284337399 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff0140a99f0 msgr2=0x7feffc005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:54.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.565+0000 7ff0137fe640 1 --2- 172.21.15.116:0/2284337399 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff0140a99f0 0x7feffc005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:54.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.565+0000 7ff0137fe640 1 -- 172.21.15.116:0/2284337399 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feffc004810 msgr2=0x7feffc004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:54.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.565+0000 7ff0137fe640 1 --2- 172.21.15.116:0/2284337399 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feffc004810 0x7feffc004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:54.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.565+0000 7ff0137fe640 1 -- 172.21.15.116:0/2284337399 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feffc003530 con 0x7feffc0039f0 2024-04-25T00:01:54.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.569+0000 7ff0137fe640 1 --2- 172.21.15.116:0/2284337399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feffc0039f0 0x7feffc003e10 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7ff004001780 tx=0x7ff00401c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3aff6a1dc122c67d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:54.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.569+0000 7ff0127fc640 1 -- 172.21.15.116:0/2284337399 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0040013a0 con 0x7feffc0039f0 2024-04-25T00:01:54.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.569+0000 7ff0127fc640 1 -- 172.21.15.116:0/2284337399 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0040052f0 con 0x7feffc0039f0 2024-04-25T00:01:54.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.569+0000 7ff0127fc640 1 -- 172.21.15.116:0/2284337399 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff004003840 con 0x7feffc0039f0 2024-04-25T00:01:54.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.569+0000 7ff01a0f1640 1 -- 172.21.15.116:0/2284337399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feffc0039f0 msgr2=0x7feffc003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:54.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.569+0000 7ff01a0f1640 1 --2- 172.21.15.116:0/2284337399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feffc0039f0 0x7feffc003e10 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7ff004001780 tx=0x7ff00401c420 comp rx=0 tx=0).stop 2024-04-25T00:01:54.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.569+0000 7ff013fff640 1 -- 172.21.15.116:0/2284337399 reap_dead start 2024-04-25T00:01:54.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.569+0000 7ff01a0f1640 1 -- 172.21.15.116:0/2284337399 shutdown_connections 2024-04-25T00:01:54.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.569+0000 7ff01a0f1640 1 -- 172.21.15.116:0/2284337399 >> 172.21.15.116:0/2284337399 conn(0x7ff0140c3b00 msgr2=0x7ff0140c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:54.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.569+0000 7ff01a0f1640 1 -- 172.21.15.116:0/2284337399 shutdown_connections 2024-04-25T00:01:54.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.569+0000 7ff01a0f1640 1 -- 172.21.15.116:0/2284337399 wait complete. 2024-04-25T00:01:54.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.569+0000 7ff01a0f1640 1 Processor -- start 2024-04-25T00:01:54.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.569+0000 7ff01a0f1640 1 -- start start 2024-04-25T00:01:54.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.569+0000 7ff01a0f1640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7feffc0039f0 0x7ff014144b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:54.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.569+0000 7ff01a0f1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff01413cae0 0x7ff014152660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:54.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.573+0000 7ff01a0f1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0141ed0a0 0x7ff014145300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:54.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.573+0000 7ff01a0f1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff0140c32a0 con 0x7ff0141ed0a0 2024-04-25T00:01:54.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.573+0000 7ff01a0f1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff0140c3120 con 0x7ff01413cae0 2024-04-25T00:01:54.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.573+0000 7ff01a0f1640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff01415f730 con 0x7feffc0039f0 2024-04-25T00:01:54.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.573+0000 7ff013fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0141ed0a0 0x7ff014145300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:54.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.573+0000 7ff013fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0141ed0a0 0x7ff014145300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43690/0 (socket says 172.21.15.116:43690) 2024-04-25T00:01:54.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.573+0000 7ff013fff640 1 -- 172.21.15.116:0/3854914972 learned_addr learned my addr 172.21.15.116:0/3854914972 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:54.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.573+0000 7ff0137fe640 1 --2- 172.21.15.116:0/3854914972 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7feffc0039f0 0x7ff014144b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:54.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.573+0000 7ff013fff640 1 -- 172.21.15.116:0/3854914972 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7feffc0039f0 msgr2=0x7ff014144b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:54.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.573+0000 7ff012ffd640 1 --2- 172.21.15.116:0/3854914972 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff01413cae0 0x7ff014152660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:54.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.573+0000 7ff013fff640 1 --2- 172.21.15.116:0/3854914972 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7feffc0039f0 0x7ff014144b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:54.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.573+0000 7ff013fff640 1 -- 172.21.15.116:0/3854914972 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff01413cae0 msgr2=0x7ff014152660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:54.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.573+0000 7ff013fff640 1 --2- 172.21.15.116:0/3854914972 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff01413cae0 0x7ff014152660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:54.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.573+0000 7ff013fff640 1 -- 172.21.15.116:0/3854914972 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0141495a0 con 0x7ff0141ed0a0 2024-04-25T00:01:54.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.573+0000 7ff0137fe640 1 --2- 172.21.15.116:0/3854914972 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7feffc0039f0 0x7ff014144b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:54.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.573+0000 7ff013fff640 1 --2- 172.21.15.116:0/3854914972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0141ed0a0 0x7ff014145300 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7ff014070890 tx=0x7ff00c009dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:54.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.573+0000 7feffbfff640 1 -- 172.21.15.116:0/3854914972 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff00c007d50 con 0x7ff0141ed0a0 2024-04-25T00:01:54.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.573+0000 7feffbfff640 1 -- 172.21.15.116:0/3854914972 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff00c019070 con 0x7ff0141ed0a0 2024-04-25T00:01:54.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.573+0000 7ff01a0f1640 1 -- 172.21.15.116:0/3854914972 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff01413c800 con 0x7ff0141ed0a0 2024-04-25T00:01:54.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.573+0000 7feffbfff640 1 -- 172.21.15.116:0/3854914972 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff00c014650 con 0x7ff0141ed0a0 2024-04-25T00:01:54.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.573+0000 7ff01a0f1640 1 -- 172.21.15.116:0/3854914972 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0141f0bc0 con 0x7ff0141ed0a0 2024-04-25T00:01:54.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.573+0000 7ff01a0f1640 1 -- 172.21.15.116:0/3854914972 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff014068350 con 0x7ff0141ed0a0 2024-04-25T00:01:54.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.573+0000 7feffbfff640 1 -- 172.21.15.116:0/3854914972 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff00c0147f0 con 0x7ff0141ed0a0 2024-04-25T00:01:54.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.573+0000 7feffbfff640 1 --2- 172.21.15.116:0/3854914972 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff0000678f0 0x7ff000069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:54.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.573+0000 7feffbfff640 1 -- 172.21.15.116:0/3854914972 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6666+0+0 (secure 0 0 0) 0x7ff00c0864e0 con 0x7ff0141ed0a0 2024-04-25T00:01:54.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.573+0000 7ff0137fe640 1 --2- 172.21.15.116:0/3854914972 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff0000678f0 0x7ff000069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:54.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.573+0000 7ff0137fe640 1 --2- 172.21.15.116:0/3854914972 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff0000678f0 0x7ff000069db0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff004001750 tx=0x7ff0040011a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:54.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.577+0000 7feffbfff640 1 -- 172.21.15.116:0/3854914972 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ff00c056c50 con 0x7ff0141ed0a0 2024-04-25T00:01:54.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.765+0000 7ff01a0f1640 1 -- 172.21.15.116:0/3854914972 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff01406e1e0 con 0x7ff0141ed0a0 2024-04-25T00:01:54.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.765+0000 7feffbfff640 1 -- 172.21.15.116:0/3854914972 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6492 (secure 0 0 0) 0x7ff00c056550 con 0x7ff0141ed0a0 2024-04-25T00:01:54.769 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:01:54.769 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6834/1459177921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1459177921},{"type":"v1","addr":"172.21.15.116:6834","nonce":1459177921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:01:08.748957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6837/1762356055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1762356055},{"type":"v1","addr":"172.21.15.174:6837","nonce":1762356055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270]},"id":1},{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:01:22.888641+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6835/174781672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":174781672},{"type":"v1","addr":"172.21.15.116:6835","nonce":174781672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4245,"qdb_cluster":[4245]},"id":2}]} 2024-04-25T00:01:54.769 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 14 2024-04-25T00:01:54.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.769+0000 7ff01a0f1640 1 -- 172.21.15.116:0/3854914972 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff0000678f0 msgr2=0x7ff000069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:54.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.769+0000 7ff01a0f1640 1 --2- 172.21.15.116:0/3854914972 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff0000678f0 0x7ff000069db0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff004001750 tx=0x7ff0040011a0 comp rx=0 tx=0).stop 2024-04-25T00:01:54.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.769+0000 7ff01a0f1640 1 -- 172.21.15.116:0/3854914972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0141ed0a0 msgr2=0x7ff014145300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:54.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.769+0000 7ff01a0f1640 1 --2- 172.21.15.116:0/3854914972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0141ed0a0 0x7ff014145300 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7ff014070890 tx=0x7ff00c009dc0 comp rx=0 tx=0).stop 2024-04-25T00:01:54.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.769+0000 7ff01a0f1640 1 -- 172.21.15.116:0/3854914972 shutdown_connections 2024-04-25T00:01:54.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.769+0000 7ff01a0f1640 1 --2- 172.21.15.116:0/3854914972 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff0000678f0 0x7ff000069db0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:54.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.769+0000 7ff01a0f1640 1 --2- 172.21.15.116:0/3854914972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0141ed0a0 0x7ff014145300 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:54.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.769+0000 7ff01a0f1640 1 --2- 172.21.15.116:0/3854914972 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff01413cae0 0x7ff014152660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:54.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.769+0000 7ff01a0f1640 1 --2- 172.21.15.116:0/3854914972 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7feffc0039f0 0x7ff014144b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:54.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.769+0000 7ff01a0f1640 1 -- 172.21.15.116:0/3854914972 >> 172.21.15.116:0/3854914972 conn(0x7ff0140c3b00 msgr2=0x7ff0140ac0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:54.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.769+0000 7ff01a0f1640 1 -- 172.21.15.116:0/3854914972 shutdown_connections 2024-04-25T00:01:54.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.769+0000 7ff01a0f1640 1 -- 172.21.15.116:0/3854914972 wait complete. 2024-04-25T00:01:54.786 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T00:01:54.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.889+0000 7ff7b120e640 1 Processor -- start 2024-04-25T00:01:54.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.893+0000 7ff7b120e640 1 -- start start 2024-04-25T00:01:54.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.893+0000 7ff7b120e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff7ac0a94a0 0x7ff7ac1538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:54.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.893+0000 7ff7b120e640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff7ac0a99f0 0x7ff7ac153e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:54.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.893+0000 7ff7b120e640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7ac0c3910 con 0x7ff7ac150d20 2024-04-25T00:01:54.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.893+0000 7ff7b120e640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7ff7ac0c16e0 con 0x7ff7ac0a99f0 2024-04-25T00:01:54.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.893+0000 7ff7b120e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff7ac0c1400 con 0x7ff7ac0a94a0 2024-04-25T00:01:54.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.893+0000 7ff7aad9d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7ff7ac150d20 0x7ff7ac0c2a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:46956/0 (socket says 172.21.15.116:46956) 2024-04-25T00:01:54.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.893+0000 7ff7aad9d640 1 -- 172.21.15.116:0/446525279 learned_addr learned my addr 172.21.15.116:0/446525279 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:01:54.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.893+0000 7ff7aa59c640 1 --2- 172.21.15.116:0/446525279 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff7ac0a94a0 0x7ff7ac1538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:54.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.893+0000 7ff7ab59e640 1 --2- 172.21.15.116:0/446525279 >> v2:172.21.15.174:3300/0 conn(0x7ff7ac0a99f0 0x7ff7ac153e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:54.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.893+0000 7ff7a9d9b640 1 -- 172.21.15.116:0/446525279 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3417526125 0 0) 0x7ff7ac0c3910 con 0x7ff7ac150d20 2024-04-25T00:01:54.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.893+0000 7ff7a9d9b640 1 -- 172.21.15.116:0/446525279 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff794003530 con 0x7ff7ac150d20 2024-04-25T00:01:54.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.893+0000 7ff7a9d9b640 1 -- 172.21.15.116:0/446525279 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3617515078 0 0) 0x7ff794003530 con 0x7ff7ac150d20 2024-04-25T00:01:54.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.893+0000 7ff7a9d9b640 1 -- 172.21.15.116:0/446525279 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff7ac0a94a0 msgr2=0x7ff7ac1538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:54.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.893+0000 7ff7a9d9b640 1 --2- 172.21.15.116:0/446525279 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff7ac0a94a0 0x7ff7ac1538e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:54.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.893+0000 7ff7a9d9b640 1 -- 172.21.15.116:0/446525279 >> v2:172.21.15.174:3300/0 conn(0x7ff7ac0a99f0 msgr2=0x7ff7ac153e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:54.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.893+0000 7ff7a9d9b640 1 --2- 172.21.15.116:0/446525279 >> v2:172.21.15.174:3300/0 conn(0x7ff7ac0a99f0 0x7ff7ac153e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:54.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.893+0000 7ff7a9d9b640 1 -- 172.21.15.116:0/446525279 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7ac06e660 con 0x7ff7ac150d20 2024-04-25T00:01:54.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.893+0000 7ff7a9d9b640 1 -- 172.21.15.116:0/446525279 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7ff79c001680 con 0x7ff7ac150d20 2024-04-25T00:01:54.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.893+0000 7ff7a9d9b640 1 -- 172.21.15.116:0/446525279 >> v1:172.21.15.116:6789/0 conn(0x7ff7ac150d20 legacy=0x7ff7ac0c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:54.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.893+0000 7ff7a9d9b640 1 --2- 172.21.15.116:0/446525279 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7940039c0 0x7ff794003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:54.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.893+0000 7ff7a9d9b640 1 --2- 172.21.15.116:0/446525279 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff794004810 0x7ff794004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:54.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.893+0000 7ff7a9d9b640 1 --2- 172.21.15.116:0/446525279 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff7940051d0 0x7ff79400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:54.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.893+0000 7ff7a9d9b640 1 -- 172.21.15.116:0/446525279 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff7ac0c16e0 con 0x7ff7940039c0 2024-04-25T00:01:54.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.893+0000 7ff7a9d9b640 1 -- 172.21.15.116:0/446525279 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff7ac0c1400 con 0x7ff794004810 2024-04-25T00:01:54.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.893+0000 7ff7a9d9b640 1 -- 172.21.15.116:0/446525279 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff79400bf40 con 0x7ff7940051d0 2024-04-25T00:01:54.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.893+0000 7ff7aa59c640 1 --2- 172.21.15.116:0/446525279 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff7ac0a94a0 0x7ff7ac1538e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:01:54.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.893+0000 7ff7aad9d640 1 --2- 172.21.15.116:0/446525279 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7940039c0 0x7ff794003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:54.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.893+0000 7ff7aa59c640 1 --2- 172.21.15.116:0/446525279 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff794004810 0x7ff794004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:54.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.893+0000 7ff7aad9d640 1 -- 172.21.15.116:0/446525279 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff7940051d0 msgr2=0x7ff79400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:54.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.893+0000 7ff7aad9d640 1 --2- 172.21.15.116:0/446525279 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff7940051d0 0x7ff79400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:54.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.893+0000 7ff7aad9d640 1 -- 172.21.15.116:0/446525279 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff794004810 msgr2=0x7ff794004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:01:54.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.893+0000 7ff7aad9d640 1 --2- 172.21.15.116:0/446525279 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff794004810 0x7ff794004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:54.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.893+0000 7ff7aad9d640 1 -- 172.21.15.116:0/446525279 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff794003530 con 0x7ff7940039c0 2024-04-25T00:01:54.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.893+0000 7ff7aa59c640 1 --2- 172.21.15.116:0/446525279 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff794004810 0x7ff794004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:54.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.897+0000 7ff7aad9d640 1 --2- 172.21.15.116:0/446525279 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7940039c0 0x7ff794003de0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7ff79c00c240 tx=0x7ff79c01cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7296a82f379eaa04 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:54.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.897+0000 7ff7a9d9b640 1 -- 172.21.15.116:0/446525279 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff79c007d60 con 0x7ff7940039c0 2024-04-25T00:01:54.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.897+0000 7ff7a9d9b640 1 -- 172.21.15.116:0/446525279 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff79c0a9020 con 0x7ff7940039c0 2024-04-25T00:01:54.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.897+0000 7ff7a9d9b640 1 -- 172.21.15.116:0/446525279 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff79c005ae0 con 0x7ff7940039c0 2024-04-25T00:01:54.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.897+0000 7ff7b120e640 1 -- 172.21.15.116:0/446525279 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7940039c0 msgr2=0x7ff794003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:54.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.897+0000 7ff7b120e640 1 --2- 172.21.15.116:0/446525279 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7940039c0 0x7ff794003de0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7ff79c00c240 tx=0x7ff79c01cc60 comp rx=0 tx=0).stop 2024-04-25T00:01:54.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.897+0000 7ff7ab59e640 1 -- 172.21.15.116:0/446525279 reap_dead start 2024-04-25T00:01:54.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.897+0000 7ff7b120e640 1 -- 172.21.15.116:0/446525279 shutdown_connections 2024-04-25T00:01:54.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.897+0000 7ff7b120e640 1 -- 172.21.15.116:0/446525279 >> 172.21.15.116:0/446525279 conn(0x7ff7ac0c3b00 msgr2=0x7ff7ac0c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:54.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.897+0000 7ff7b120e640 1 -- 172.21.15.116:0/446525279 shutdown_connections 2024-04-25T00:01:54.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.897+0000 7ff7b120e640 1 -- 172.21.15.116:0/446525279 wait complete. 2024-04-25T00:01:54.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.897+0000 7ff7b120e640 1 Processor -- start 2024-04-25T00:01:54.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.897+0000 7ff7b120e640 1 -- start start 2024-04-25T00:01:54.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.897+0000 7ff7b120e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff7940039c0 0x7ff7ac14f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:54.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.897+0000 7ff7b120e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff7ac146a30 0x7ff7ac145e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:54.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.897+0000 7ff7b120e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7ac146340 0x7ff7ac0c4f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:54.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.897+0000 7ff7b120e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff7ac1544e0 con 0x7ff7ac146340 2024-04-25T00:01:54.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.897+0000 7ff7b120e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff7ac154360 con 0x7ff7ac146a30 2024-04-25T00:01:54.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.897+0000 7ff7b120e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff7ac071550 con 0x7ff7940039c0 2024-04-25T00:01:54.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.897+0000 7ff7ab59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7ac146340 0x7ff7ac0c4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:54.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.897+0000 7ff7ab59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7ac146340 0x7ff7ac0c4f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60296/0 (socket says 172.21.15.116:60296) 2024-04-25T00:01:54.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.897+0000 7ff7ab59e640 1 -- 172.21.15.116:0/2147173822 learned_addr learned my addr 172.21.15.116:0/2147173822 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:01:54.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.897+0000 7ff7aad9d640 1 --2- 172.21.15.116:0/2147173822 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff7940039c0 0x7ff7ac14f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:54.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.901+0000 7ff7aa59c640 1 --2- 172.21.15.116:0/2147173822 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff7ac146a30 0x7ff7ac145e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:54.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.901+0000 7ff7ab59e640 1 -- 172.21.15.116:0/2147173822 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff7940039c0 msgr2=0x7ff7ac14f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:54.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.901+0000 7ff7ab59e640 1 --2- 172.21.15.116:0/2147173822 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff7940039c0 0x7ff7ac14f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:54.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.901+0000 7ff7ab59e640 1 -- 172.21.15.116:0/2147173822 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff7ac146a30 msgr2=0x7ff7ac145e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:54.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.901+0000 7ff7ab59e640 1 --2- 172.21.15.116:0/2147173822 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff7ac146a30 0x7ff7ac145e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:54.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.901+0000 7ff7ab59e640 1 -- 172.21.15.116:0/2147173822 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7ac14ad90 con 0x7ff7ac146340 2024-04-25T00:01:54.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.901+0000 7ff7aa59c640 1 --2- 172.21.15.116:0/2147173822 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff7ac146a30 0x7ff7ac145e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:01:54.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.901+0000 7ff7ab59e640 1 --2- 172.21.15.116:0/2147173822 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7ac146340 0x7ff7ac0c4f20 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac157b40 tx=0x7ff7a400b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:54.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.901+0000 7ff7937fe640 1 -- 172.21.15.116:0/2147173822 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7a4018020 con 0x7ff7ac146340 2024-04-25T00:01:54.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.901+0000 7ff7937fe640 1 -- 172.21.15.116:0/2147173822 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7a400bab0 con 0x7ff7ac146340 2024-04-25T00:01:54.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.901+0000 7ff7b120e640 1 -- 172.21.15.116:0/2147173822 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7ac152ee0 con 0x7ff7ac146340 2024-04-25T00:01:54.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.901+0000 7ff7937fe640 1 -- 172.21.15.116:0/2147173822 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7a4014400 con 0x7ff7ac146340 2024-04-25T00:01:54.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.901+0000 7ff7b120e640 1 -- 172.21.15.116:0/2147173822 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7ac152190 con 0x7ff7ac146340 2024-04-25T00:01:54.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.901+0000 7ff7b120e640 1 -- 172.21.15.116:0/2147173822 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7ac068350 con 0x7ff7ac146340 2024-04-25T00:01:54.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.901+0000 7ff7937fe640 1 -- 172.21.15.116:0/2147173822 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff7a401b020 con 0x7ff7ac146340 2024-04-25T00:01:54.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.901+0000 7ff7937fe640 1 --2- 172.21.15.116:0/2147173822 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff7980678c0 0x7ff798069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:01:54.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.901+0000 7ff7937fe640 1 -- 172.21.15.116:0/2147173822 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6666+0+0 (secure 0 0 0) 0x7ff7a40864f0 con 0x7ff7ac146340 2024-04-25T00:01:54.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.901+0000 7ff7aad9d640 1 --2- 172.21.15.116:0/2147173822 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff7980678c0 0x7ff798069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:01:54.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.901+0000 7ff7aad9d640 1 --2- 172.21.15.116:0/2147173822 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff7980678c0 0x7ff798069d80 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff79c0aa000 tx=0x7ff79c0b0040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:01:54.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:54.905+0000 7ff7937fe640 1 -- 172.21.15.116:0/2147173822 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ff7a4022030 con 0x7ff7ac146340 2024-04-25T00:01:55.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:55.053+0000 7ff7b120e640 1 -- 172.21.15.116:0/2147173822 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff7ac0c34f0 con 0x7ff7ac146340 2024-04-25T00:01:55.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:55.057+0000 7ff7937fe640 1 -- 172.21.15.116:0/2147173822 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) ==== 74+0+20606 (secure 0 0 0) 0x7ff7a4056d00 con 0x7ff7ac146340 2024-04-25T00:01:55.060 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:01:55.061 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":29,"fsid":"6d6a1aa7-0478-4c83-8057-05917e81f046","created":"2024-04-25T00:00:40.682493+0000","modified":"2024-04-25T00:01:46.633464+0000","last_up_change":"2024-04-25T00:00:51.822174+0000","last_in_change":"2024-04-25T00:00:42.980473+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T00:00:53.324376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-04-25T00:01:03.197326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-04-25T00:01:03.728618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-04-25T00:01:17.125210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-04-25T00:01:17.821873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8b6ddcb2-1ad9-4167-961c-bd6936bb9060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6809","nonce":3751765682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6811","nonce":3751765682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6815","nonce":3751765682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6813","nonce":3751765682}]},"public_addr":"172.21.15.116:6809/3751765682","cluster_addr":"172.21.15.116:6811/3751765682","heartbeat_back_addr":"172.21.15.116:6815/3751765682","heartbeat_front_addr":"172.21.15.116:6813/3751765682","state":["exists","up"]},{"osd":1,"uuid":"a6431768-994f-41da-8853-886f5687fb91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6801","nonce":377879124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6803","nonce":377879124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6807","nonce":377879124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6805","nonce":377879124}]},"public_addr":"172.21.15.116:6801/377879124","cluster_addr":"172.21.15.116:6803/377879124","heartbeat_back_addr":"172.21.15.116:6807/377879124","heartbeat_front_addr":"172.21.15.116:6805/377879124","state":["exists","up"]},{"osd":2,"uuid":"4f69bd40-5a23-45ff-8693-e1de75e1aa9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6817","nonce":1099282267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6819","nonce":1099282267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6823","nonce":1099282267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6821","nonce":1099282267}]},"public_addr":"172.21.15.116:6817/1099282267","cluster_addr":"172.21.15.116:6819/1099282267","heartbeat_back_addr":"172.21.15.116:6823/1099282267","heartbeat_front_addr":"172.21.15.116:6821/1099282267","state":["exists","up"]},{"osd":3,"uuid":"6a0d5344-8b6f-40d3-918d-0f0a4be688e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6825","nonce":1179942661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6827","nonce":1179942661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6831","nonce":1179942661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6829","nonce":1179942661}]},"public_addr":"172.21.15.116:6825/1179942661","cluster_addr":"172.21.15.116:6827/1179942661","heartbeat_back_addr":"172.21.15.116:6831/1179942661","heartbeat_front_addr":"172.21.15.116:6829/1179942661","state":["exists","up"]},{"osd":4,"uuid":"132e92e4-b825-441e-965a-ca00533fa86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6801","nonce":3132444330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6803","nonce":3132444330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6807","nonce":3132444330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6805","nonce":3132444330}]},"public_addr":"172.21.15.174:6801/3132444330","cluster_addr":"172.21.15.174:6803/3132444330","heartbeat_back_addr":"172.21.15.174:6807/3132444330","heartbeat_front_addr":"172.21.15.174:6805/3132444330","state":["exists","up"]},{"osd":5,"uuid":"a2eeb5b6-1b31-43cc-8306-a0adf3ee15bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6809","nonce":881817164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6811","nonce":881817164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6815","nonce":881817164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6813","nonce":881817164}]},"public_addr":"172.21.15.174:6809/881817164","cluster_addr":"172.21.15.174:6811/881817164","heartbeat_back_addr":"172.21.15.174:6815/881817164","heartbeat_front_addr":"172.21.15.174:6813/881817164","state":["exists","up"]},{"osd":6,"uuid":"588e873a-c841-4ba7-90f2-8956c87a9983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6825","nonce":117574251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6827","nonce":117574251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6831","nonce":117574251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6829","nonce":117574251}]},"public_addr":"172.21.15.174:6825/117574251","cluster_addr":"172.21.15.174:6827/117574251","heartbeat_back_addr":"172.21.15.174:6831/117574251","heartbeat_front_addr":"172.21.15.174:6829/117574251","state":["exists","up"]},{"osd":7,"uuid":"3b90f9b0-8a0e-4883-b088-ca1d4012870c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6817","nonce":2543899196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6819","nonce":2543899196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6823","nonce":2543899196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6821","nonce":2543899196}]},"public_addr":"172.21.15.174:6817/2543899196","cluster_addr":"172.21.15.174:6819/2543899196","heartbeat_back_addr":"172.21.15.174:6823/2543899196","heartbeat_front_addr":"172.21.15.174:6821/2543899196","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.293769+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.310034+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.404865+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.549658+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.192174+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.297266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.493649+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.452309+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2c","mappings":[{"from":0,"to":7}]},{"pgid":"5.3","mappings":[{"from":4,"to":5}]},{"pgid":"5.5","mappings":[{"from":4,"to":5}]},{"pgid":"5.14","mappings":[{"from":4,"to":5}]},{"pgid":"5.19","mappings":[{"from":4,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:01:55.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:55.061+0000 7ff7b120e640 1 -- 172.21.15.116:0/2147173822 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff7980678c0 msgr2=0x7ff798069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:55.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:55.061+0000 7ff7b120e640 1 --2- 172.21.15.116:0/2147173822 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff7980678c0 0x7ff798069d80 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff79c0aa000 tx=0x7ff79c0b0040 comp rx=0 tx=0).stop 2024-04-25T00:01:55.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:55.061+0000 7ff7b120e640 1 -- 172.21.15.116:0/2147173822 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7ac146340 msgr2=0x7ff7ac0c4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:01:55.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:55.061+0000 7ff7b120e640 1 --2- 172.21.15.116:0/2147173822 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7ac146340 0x7ff7ac0c4f20 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac157b40 tx=0x7ff7a400b480 comp rx=0 tx=0).stop 2024-04-25T00:01:55.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:55.061+0000 7ff7b120e640 1 -- 172.21.15.116:0/2147173822 shutdown_connections 2024-04-25T00:01:55.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:55.061+0000 7ff7b120e640 1 --2- 172.21.15.116:0/2147173822 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff7980678c0 0x7ff798069d80 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:55.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:55.061+0000 7ff7b120e640 1 --2- 172.21.15.116:0/2147173822 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff7ac146340 0x7ff7ac0c4f20 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:55.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:55.061+0000 7ff7b120e640 1 --2- 172.21.15.116:0/2147173822 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff7ac146a30 0x7ff7ac145e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:55.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:55.061+0000 7ff7b120e640 1 --2- 172.21.15.116:0/2147173822 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff7940039c0 0x7ff7ac14f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:01:55.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:55.061+0000 7ff7b120e640 1 -- 172.21.15.116:0/2147173822 >> 172.21.15.116:0/2147173822 conn(0x7ff7ac0c3b00 msgr2=0x7ff7ac06cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:01:55.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:55.061+0000 7ff7b120e640 1 -- 172.21.15.116:0/2147173822 shutdown_connections 2024-04-25T00:01:55.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:01:55.061+0000 7ff7b120e640 1 -- 172.21.15.116:0/2147173822 wait complete. 2024-04-25T00:01:55.079 DEBUG:tasks.mds_thrash:Yielding 2024-04-25T00:01:55.079 INFO:teuthology.run_tasks:Running task workunit... 2024-04-25T00:01:55.118 INFO:tasks.workunit:Pulling workunits from ref d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T00:01:55.118 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-25T00:01:55.118 INFO:tasks.workunit:timeout=3h 2024-04-25T00:01:55.118 INFO:tasks.workunit:cleanup=True 2024-04-25T00:01:55.119 DEBUG:teuthology.orchestra.run.smithi116:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-25T00:01:55.119 INFO:tasks.mds_thrash.fs.[b]:starting mds_do_thrash for fs b 2024-04-25T00:01:55.119 INFO:tasks.mds_thrash.fs.[b]:waiting for 30 secs before thrashing 2024-04-25T00:01:55.126 INFO:teuthology.orchestra.run.smithi116.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-25T00:01:55.126 INFO:teuthology.orchestra.run.smithi116.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-04-25T00:01:55.126 INFO:teuthology.orchestra.run.smithi116.stdout:Device: 2dh/45d Inode: 1 Links: 2 2024-04-25T00:01:55.126 INFO:teuthology.orchestra.run.smithi116.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T00:01:55.126 INFO:teuthology.orchestra.run.smithi116.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-04-25T00:01:55.126 INFO:teuthology.orchestra.run.smithi116.stdout:Modify: 2024-04-25 00:01:05.385978098 +0000 2024-04-25T00:01:55.126 INFO:teuthology.orchestra.run.smithi116.stdout:Change: 2024-04-25 00:01:44.976088974 +0000 2024-04-25T00:01:55.126 INFO:teuthology.orchestra.run.smithi116.stdout: Birth: - 2024-04-25T00:01:55.126 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-25T00:01:55.127 DEBUG:teuthology.orchestra.run.smithi116:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-25T00:01:55.190 DEBUG:teuthology.orchestra.run.smithi174:> stat -- /home/ubuntu/cephtest/mnt.1 2024-04-25T00:01:55.199 INFO:teuthology.orchestra.run.smithi174.stdout: File: /home/ubuntu/cephtest/mnt.1 2024-04-25T00:01:55.199 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-04-25T00:01:55.199 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 2dh/45d Inode: 1 Links: 2 2024-04-25T00:01:55.199 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T00:01:55.199 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-04-25T00:01:55.199 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-04-25 00:01:19.505037564 +0000 2024-04-25T00:01:55.199 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-04-25 00:01:45.041652092 +0000 2024-04-25T00:01:55.199 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: - 2024-04-25T00:01:55.200 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2024-04-25T00:01:55.200 DEBUG:teuthology.orchestra.run.smithi174:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2024-04-25T00:01:55.263 DEBUG:teuthology.orchestra.run.smithi116:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T00:01:55.264 DEBUG:teuthology.orchestra.run.smithi174:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-25T00:01:55.271 INFO:tasks.workunit.client.0.smithi116.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-25T00:01:55.311 INFO:tasks.workunit.client.1.smithi174.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2024-04-25T00:02:04.569 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:02:04.573 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:02:25.120 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:02:25.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.245+0000 7f84c6e29640 1 Processor -- start 2024-04-25T00:02:25.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.245+0000 7f84c6e29640 1 -- start start 2024-04-25T00:02:25.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.249+0000 7f84c6e29640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f84c0150d10 0x7f84c00c5010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:25.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.249+0000 7f84c6e29640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f84c00c5550 0x7f84c0153a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:25.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.249+0000 7f84c6e29640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84c00c4950 con 0x7f84c0072ca0 2024-04-25T00:02:25.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.249+0000 7f84c6e29640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f84c01546d0 con 0x7f84c00c5550 2024-04-25T00:02:25.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.249+0000 7f84c6e29640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f84c01510f0 con 0x7f84c0150d10 2024-04-25T00:02:25.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.249+0000 7f84c53c6640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f84c0072ca0 0x7f84c0153fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:39502/0 (socket says 172.21.15.116:39502) 2024-04-25T00:02:25.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.249+0000 7f84c53c6640 1 -- 172.21.15.116:0/3625316889 learned_addr learned my addr 172.21.15.116:0/3625316889 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:02:25.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.249+0000 7f84c4bc5640 1 --2- 172.21.15.116:0/3625316889 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f84c0150d10 0x7f84c00c5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:25.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.249+0000 7f84bffff640 1 --2- 172.21.15.116:0/3625316889 >> v2:172.21.15.174:3300/0 conn(0x7f84c00c5550 0x7f84c0153a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:25.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.249+0000 7f84bf7fe640 1 -- 172.21.15.116:0/3625316889 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 506080002 0 0) 0x7f84c00c4950 con 0x7f84c0072ca0 2024-04-25T00:02:25.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.249+0000 7f84bf7fe640 1 -- 172.21.15.116:0/3625316889 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f84ac003530 con 0x7f84c0072ca0 2024-04-25T00:02:25.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.249+0000 7f84bf7fe640 1 -- 172.21.15.116:0/3625316889 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3514860771 0 0) 0x7f84ac003530 con 0x7f84c0072ca0 2024-04-25T00:02:25.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.249+0000 7f84bf7fe640 1 -- 172.21.15.116:0/3625316889 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f84c0150d10 msgr2=0x7f84c00c5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:02:25.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.249+0000 7f84bf7fe640 1 --2- 172.21.15.116:0/3625316889 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f84c0150d10 0x7f84c00c5010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:25.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.249+0000 7f84bf7fe640 1 -- 172.21.15.116:0/3625316889 >> v2:172.21.15.174:3300/0 conn(0x7f84c00c5550 msgr2=0x7f84c0153a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:02:25.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.249+0000 7f84bf7fe640 1 --2- 172.21.15.116:0/3625316889 >> v2:172.21.15.174:3300/0 conn(0x7f84c00c5550 0x7f84c0153a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:25.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.249+0000 7f84bf7fe640 1 -- 172.21.15.116:0/3625316889 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84c0157a20 con 0x7f84c0072ca0 2024-04-25T00:02:25.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.249+0000 7f84bf7fe640 1 -- 172.21.15.116:0/3625316889 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f84b8001680 con 0x7f84c0072ca0 2024-04-25T00:02:25.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.249+0000 7f84bf7fe640 1 -- 172.21.15.116:0/3625316889 >> v1:172.21.15.116:6789/0 conn(0x7f84c0072ca0 legacy=0x7f84c0153fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:25.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.249+0000 7f84bf7fe640 1 --2- 172.21.15.116:0/3625316889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84ac0039c0 0x7f84ac003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:25.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.249+0000 7f84bf7fe640 1 --2- 172.21.15.116:0/3625316889 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f84ac004810 0x7f84ac004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:25.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.249+0000 7f84bf7fe640 1 --2- 172.21.15.116:0/3625316889 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f84c0150d10 0x7f84ac0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:25.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.249+0000 7f84bf7fe640 1 -- 172.21.15.116:0/3625316889 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f84c01510f0 con 0x7f84ac0039c0 2024-04-25T00:02:25.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.249+0000 7f84bf7fe640 1 -- 172.21.15.116:0/3625316889 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f84ac00b9d0 con 0x7f84ac004810 2024-04-25T00:02:25.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.249+0000 7f84bf7fe640 1 -- 172.21.15.116:0/3625316889 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f84ac00bbd0 con 0x7f84c0150d10 2024-04-25T00:02:25.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.249+0000 7f84c4bc5640 1 --2- 172.21.15.116:0/3625316889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84ac0039c0 0x7f84ac003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:25.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.249+0000 7f84bffff640 1 --2- 172.21.15.116:0/3625316889 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f84ac004810 0x7f84ac004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:25.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.249+0000 7f84c53c6640 1 --2- 172.21.15.116:0/3625316889 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f84c0150d10 0x7f84ac0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:25.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.249+0000 7f84c4bc5640 1 -- 172.21.15.116:0/3625316889 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f84c0150d10 msgr2=0x7f84ac0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:02:25.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.249+0000 7f84c4bc5640 1 --2- 172.21.15.116:0/3625316889 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f84c0150d10 0x7f84ac0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:25.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.249+0000 7f84c4bc5640 1 -- 172.21.15.116:0/3625316889 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f84ac004810 msgr2=0x7f84ac004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:02:25.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.249+0000 7f84c4bc5640 1 --2- 172.21.15.116:0/3625316889 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f84ac004810 0x7f84ac004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:25.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.249+0000 7f84c4bc5640 1 -- 172.21.15.116:0/3625316889 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84ac003530 con 0x7f84ac0039c0 2024-04-25T00:02:25.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.249+0000 7f84c53c6640 1 --2- 172.21.15.116:0/3625316889 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f84c0150d10 0x7f84ac0051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:02:25.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.249+0000 7f84c4bc5640 1 --2- 172.21.15.116:0/3625316889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84ac0039c0 0x7f84ac003de0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f84b0009fc0 tx=0x7f84b001cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=715fdfe2d5499d28 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:02:25.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.249+0000 7f84bf7fe640 1 -- 172.21.15.116:0/3625316889 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84b00b2070 con 0x7f84ac0039c0 2024-04-25T00:02:25.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.249+0000 7f84bf7fe640 1 -- 172.21.15.116:0/3625316889 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84b00a5b80 con 0x7f84ac0039c0 2024-04-25T00:02:25.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.249+0000 7f84bf7fe640 1 -- 172.21.15.116:0/3625316889 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84b0009c10 con 0x7f84ac0039c0 2024-04-25T00:02:25.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.253+0000 7f84c6e29640 1 -- 172.21.15.116:0/3625316889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84ac0039c0 msgr2=0x7f84ac003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:25.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.253+0000 7f84c6e29640 1 --2- 172.21.15.116:0/3625316889 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84ac0039c0 0x7f84ac003de0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f84b0009fc0 tx=0x7f84b001cc80 comp rx=0 tx=0).stop 2024-04-25T00:02:25.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.253+0000 7f84c53c6640 1 -- 172.21.15.116:0/3625316889 reap_dead start 2024-04-25T00:02:25.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.253+0000 7f84c6e29640 1 -- 172.21.15.116:0/3625316889 shutdown_connections 2024-04-25T00:02:25.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.253+0000 7f84c6e29640 1 -- 172.21.15.116:0/3625316889 >> 172.21.15.116:0/3625316889 conn(0x7f84c006c720 msgr2=0x7f84c006cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:02:25.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.253+0000 7f84c6e29640 1 -- 172.21.15.116:0/3625316889 shutdown_connections 2024-04-25T00:02:25.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.253+0000 7f84c6e29640 1 -- 172.21.15.116:0/3625316889 wait complete. 2024-04-25T00:02:25.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.253+0000 7f84c6e29640 1 Processor -- start 2024-04-25T00:02:25.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.253+0000 7f84c6e29640 1 -- start start 2024-04-25T00:02:25.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.253+0000 7f84c6e29640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f84ac0039c0 0x7f84c01f1750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:25.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.253+0000 7f84c6e29640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f84c014afc0 0x7f84c0149b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:25.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.253+0000 7f84c6e29640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84c014edf0 0x7f84c014bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:25.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.253+0000 7f84c6e29640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f84c00c2640 con 0x7f84c014edf0 2024-04-25T00:02:25.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.253+0000 7f84c6e29640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f84c01576a0 con 0x7f84ac0039c0 2024-04-25T00:02:25.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.253+0000 7f84c6e29640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f84c00c27c0 con 0x7f84c014afc0 2024-04-25T00:02:25.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.253+0000 7f84c53c6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84c014edf0 0x7f84c014bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:25.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.253+0000 7f84c53c6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84c014edf0 0x7f84c014bdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41132/0 (socket says 172.21.15.116:41132) 2024-04-25T00:02:25.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.253+0000 7f84c53c6640 1 -- 172.21.15.116:0/2039077011 learned_addr learned my addr 172.21.15.116:0/2039077011 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:02:25.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.253+0000 7f84c4bc5640 1 --2- 172.21.15.116:0/2039077011 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f84ac0039c0 0x7f84c01f1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:25.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.253+0000 7f84bffff640 1 --2- 172.21.15.116:0/2039077011 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f84c014afc0 0x7f84c0149b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:25.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.253+0000 7f84c53c6640 1 -- 172.21.15.116:0/2039077011 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f84c014afc0 msgr2=0x7f84c0149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:25.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.253+0000 7f84c53c6640 1 --2- 172.21.15.116:0/2039077011 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f84c014afc0 0x7f84c0149b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:25.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.253+0000 7f84c53c6640 1 -- 172.21.15.116:0/2039077011 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f84ac0039c0 msgr2=0x7f84c01f1750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:25.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.253+0000 7f84c53c6640 1 --2- 172.21.15.116:0/2039077011 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f84ac0039c0 0x7f84c01f1750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:25.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.253+0000 7f84c53c6640 1 -- 172.21.15.116:0/2039077011 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84c0151fc0 con 0x7f84c014edf0 2024-04-25T00:02:25.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.253+0000 7f84c53c6640 1 --2- 172.21.15.116:0/2039077011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84c014edf0 0x7f84c014bdf0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f84c00c5c90 tx=0x7f84b800d900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:02:25.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.253+0000 7f84bd7fa640 1 -- 172.21.15.116:0/2039077011 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84b80074e0 con 0x7f84c014edf0 2024-04-25T00:02:25.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.253+0000 7f84bd7fa640 1 -- 172.21.15.116:0/2039077011 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84b80047f0 con 0x7f84c014edf0 2024-04-25T00:02:25.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.253+0000 7f84bd7fa640 1 -- 172.21.15.116:0/2039077011 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84b8007680 con 0x7f84c014edf0 2024-04-25T00:02:25.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.253+0000 7f84c6e29640 1 -- 172.21.15.116:0/2039077011 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84c01f20a0 con 0x7f84c014edf0 2024-04-25T00:02:25.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.253+0000 7f84c6e29640 1 -- 172.21.15.116:0/2039077011 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84c01f2380 con 0x7f84c014edf0 2024-04-25T00:02:25.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.257+0000 7f84c6e29640 1 -- 172.21.15.116:0/2039077011 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84c0068a60 con 0x7f84c014edf0 2024-04-25T00:02:25.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.257+0000 7f84bd7fa640 1 -- 172.21.15.116:0/2039077011 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f84b8004ba0 con 0x7f84c014edf0 2024-04-25T00:02:25.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.257+0000 7f84bd7fa640 1 --2- 172.21.15.116:0/2039077011 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f84a00678f0 0x7f84a0069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:25.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.257+0000 7f84bd7fa640 1 -- 172.21.15.116:0/2039077011 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6666+0+0 (secure 0 0 0) 0x7f84b8085bd0 con 0x7f84c014edf0 2024-04-25T00:02:25.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.257+0000 7f84c4bc5640 1 --2- 172.21.15.116:0/2039077011 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f84a00678f0 0x7f84a0069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:25.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.257+0000 7f84c4bc5640 1 --2- 172.21.15.116:0/2039077011 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f84a00678f0 0x7f84a0069db0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f84b00af000 tx=0x7f84b0004ac0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:02:25.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.261+0000 7f84bd7fa640 1 -- 172.21.15.116:0/2039077011 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f84b80563e0 con 0x7f84c014edf0 2024-04-25T00:02:25.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.465+0000 7f84c6e29640 1 -- 172.21.15.116:0/2039077011 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f84c0072a80 con 0x7f84c014edf0 2024-04-25T00:02:25.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.465+0000 7f84bd7fa640 1 -- 172.21.15.116:0/2039077011 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6492 (secure 0 0 0) 0x7f84b8055ce0 con 0x7f84c014edf0 2024-04-25T00:02:25.468 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:02:25.468 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6834/1459177921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1459177921},{"type":"v1","addr":"172.21.15.116:6834","nonce":1459177921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:01:08.748957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6837/1762356055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1762356055},{"type":"v1","addr":"172.21.15.174:6837","nonce":1762356055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270]},"id":1},{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:01:22.888641+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6835/174781672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":174781672},{"type":"v1","addr":"172.21.15.116:6835","nonce":174781672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4245,"qdb_cluster":[4245]},"id":2}]} 2024-04-25T00:02:25.468 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 14 2024-04-25T00:02:25.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.465+0000 7f84c6e29640 1 -- 172.21.15.116:0/2039077011 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f84a00678f0 msgr2=0x7f84a0069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:25.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.465+0000 7f84c6e29640 1 --2- 172.21.15.116:0/2039077011 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f84a00678f0 0x7f84a0069db0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f84b00af000 tx=0x7f84b0004ac0 comp rx=0 tx=0).stop 2024-04-25T00:02:25.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.465+0000 7f84c6e29640 1 -- 172.21.15.116:0/2039077011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84c014edf0 msgr2=0x7f84c014bdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:25.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.465+0000 7f84c6e29640 1 --2- 172.21.15.116:0/2039077011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84c014edf0 0x7f84c014bdf0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f84c00c5c90 tx=0x7f84b800d900 comp rx=0 tx=0).stop 2024-04-25T00:02:25.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.465+0000 7f84c6e29640 1 -- 172.21.15.116:0/2039077011 shutdown_connections 2024-04-25T00:02:25.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.465+0000 7f84c6e29640 1 --2- 172.21.15.116:0/2039077011 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f84a00678f0 0x7f84a0069db0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:25.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.465+0000 7f84c6e29640 1 --2- 172.21.15.116:0/2039077011 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84c014edf0 0x7f84c014bdf0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:25.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.465+0000 7f84c6e29640 1 --2- 172.21.15.116:0/2039077011 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f84c014afc0 0x7f84c0149b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:25.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.465+0000 7f84c6e29640 1 --2- 172.21.15.116:0/2039077011 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f84ac0039c0 0x7f84c01f1750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:25.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.465+0000 7f84c6e29640 1 -- 172.21.15.116:0/2039077011 >> 172.21.15.116:0/2039077011 conn(0x7f84c006c720 msgr2=0x7f84c006cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:02:25.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.465+0000 7f84c6e29640 1 -- 172.21.15.116:0/2039077011 shutdown_connections 2024-04-25T00:02:25.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.465+0000 7f84c6e29640 1 -- 172.21.15.116:0/2039077011 wait complete. 2024-04-25T00:02:25.487 INFO:tasks.mds_thrash.fs.[b]:kill mds.a (rank=0) 2024-04-25T00:02:25.487 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-25T00:02:25.487 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:02:25.505 INFO:tasks.ceph.mds.a:Stopped 2024-04-25T00:02:25.505 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:02:25.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.669+0000 7fcd3c9aa640 1 Processor -- start 2024-04-25T00:02:25.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.669+0000 7fcd3c9aa640 1 -- start start 2024-04-25T00:02:25.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.669+0000 7fcd3c9aa640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcd38150d20 0x7fcd380c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:25.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.669+0000 7fcd3c9aa640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcd380a94a0 0x7fcd38153ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:25.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.669+0000 7fcd3c9aa640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd380c3910 con 0x7fcd380c2f50 2024-04-25T00:02:25.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.669+0000 7fcd3c9aa640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fcd38154530 con 0x7fcd380a94a0 2024-04-25T00:02:25.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.669+0000 7fcd3c9aa640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fcd380c16e0 con 0x7fcd38150d20 2024-04-25T00:02:25.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.669+0000 7fcd36d9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcd380a94a0 0x7fcd38153ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:25.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.669+0000 7fcd36d9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcd380a94a0 0x7fcd38153ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41138/0 (socket says 172.21.15.116:41138) 2024-04-25T00:02:25.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.669+0000 7fcd36d9d640 1 -- 172.21.15.116:0/3377444602 learned_addr learned my addr 172.21.15.116:0/3377444602 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:02:25.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.669+0000 7fcd3659c640 1 --2- 172.21.15.116:0/3377444602 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcd38150d20 0x7fcd380c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:25.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.669+0000 7fcd36d9d640 1 -- 172.21.15.116:0/3377444602 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcd38150d20 msgr2=0x7fcd380c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:02:25.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.669+0000 7fcd36d9d640 1 --2- 172.21.15.116:0/3377444602 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcd38150d20 0x7fcd380c2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:25.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.669+0000 7fcd36d9d640 1 -- 172.21.15.116:0/3377444602 >> v1:172.21.15.174:6789/0 conn(0x7fcd380c2f50 legacy=0x7fcd381538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:02:25.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.669+0000 7fcd36d9d640 1 -- 172.21.15.116:0/3377444602 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd380c1400 con 0x7fcd380a94a0 2024-04-25T00:02:25.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.669+0000 7fcd36d9d640 1 --2- 172.21.15.116:0/3377444602 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd380a94a0 0x7fcd38153ff0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fcd30009e50 tx=0x7fcd3001c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=172442cd3eaad14b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:02:25.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.669+0000 7fcd3559a640 1 -- 172.21.15.116:0/3377444602 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd300a5780 con 0x7fcd380a94a0 2024-04-25T00:02:25.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.669+0000 7fcd3559a640 1 -- 172.21.15.116:0/3377444602 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd300a5920 con 0x7fcd380a94a0 2024-04-25T00:02:25.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.669+0000 7fcd3559a640 1 -- 172.21.15.116:0/3377444602 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd300a5cf0 con 0x7fcd380a94a0 2024-04-25T00:02:25.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.673+0000 7fcd3c9aa640 1 -- 172.21.15.116:0/3377444602 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd380a94a0 msgr2=0x7fcd38153ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:25.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.673+0000 7fcd3c9aa640 1 --2- 172.21.15.116:0/3377444602 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd380a94a0 0x7fcd38153ff0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fcd30009e50 tx=0x7fcd3001c9f0 comp rx=0 tx=0).stop 2024-04-25T00:02:25.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.673+0000 7fcd3c9aa640 1 -- 172.21.15.116:0/3377444602 shutdown_connections 2024-04-25T00:02:25.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.673+0000 7fcd3c9aa640 1 --2- 172.21.15.116:0/3377444602 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd380a94a0 0x7fcd38153ff0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:25.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.673+0000 7fcd3c9aa640 1 --2- 172.21.15.116:0/3377444602 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcd38150d20 0x7fcd380c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:25.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.673+0000 7fcd3c9aa640 1 -- 172.21.15.116:0/3377444602 >> 172.21.15.116:0/3377444602 conn(0x7fcd380c3b00 msgr2=0x7fcd380c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:02:25.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.673+0000 7fcd3c9aa640 1 -- 172.21.15.116:0/3377444602 shutdown_connections 2024-04-25T00:02:25.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.673+0000 7fcd3c9aa640 1 -- 172.21.15.116:0/3377444602 wait complete. 2024-04-25T00:02:25.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.673+0000 7fcd3c9aa640 1 Processor -- start 2024-04-25T00:02:25.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.673+0000 7fcd3c9aa640 1 -- start start 2024-04-25T00:02:25.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.673+0000 7fcd3c9aa640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd380a94a0 0x7fcd380728b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:25.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.673+0000 7fcd3c9aa640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcd380c2f50 0x7fcd380c0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:25.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.673+0000 7fcd3c9aa640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcd38150d20 0x7fcd38156c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:25.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.673+0000 7fcd3c9aa640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcd381608b0 con 0x7fcd380a94a0 2024-04-25T00:02:25.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.673+0000 7fcd3c9aa640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcd38160730 con 0x7fcd38150d20 2024-04-25T00:02:25.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.673+0000 7fcd3c9aa640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fcd38160a30 con 0x7fcd380c2f50 2024-04-25T00:02:25.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.673+0000 7fcd3659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd380a94a0 0x7fcd380728b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:25.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.673+0000 7fcd3659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd380a94a0 0x7fcd380728b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41154/0 (socket says 172.21.15.116:41154) 2024-04-25T00:02:25.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.673+0000 7fcd3659c640 1 -- 172.21.15.116:0/2960149922 learned_addr learned my addr 172.21.15.116:0/2960149922 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:02:25.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.673+0000 7fcd35d9b640 1 --2- 172.21.15.116:0/2960149922 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcd380c2f50 0x7fcd380c0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:25.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.673+0000 7fcd36d9d640 1 --2- 172.21.15.116:0/2960149922 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcd38150d20 0x7fcd38156c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:25.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.673+0000 7fcd35d9b640 1 -- 172.21.15.116:0/2960149922 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcd38150d20 msgr2=0x7fcd38156c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:25.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.673+0000 7fcd35d9b640 1 --2- 172.21.15.116:0/2960149922 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcd38150d20 0x7fcd38156c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:25.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.673+0000 7fcd35d9b640 1 -- 172.21.15.116:0/2960149922 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd380a94a0 msgr2=0x7fcd380728b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:25.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.673+0000 7fcd35d9b640 1 --2- 172.21.15.116:0/2960149922 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd380a94a0 0x7fcd380728b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:25.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.673+0000 7fcd35d9b640 1 -- 172.21.15.116:0/2960149922 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd38150390 con 0x7fcd380c2f50 2024-04-25T00:02:25.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.673+0000 7fcd35d9b640 1 --2- 172.21.15.116:0/2960149922 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcd380c2f50 0x7fcd380c0430 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcd2800c3e0 tx=0x7fcd2800c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:02:25.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.673+0000 7fcd2effd640 1 -- 172.21.15.116:0/2960149922 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd28011790 con 0x7fcd380c2f50 2024-04-25T00:02:25.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.677+0000 7fcd2effd640 1 -- 172.21.15.116:0/2960149922 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd28017070 con 0x7fcd380c2f50 2024-04-25T00:02:25.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.677+0000 7fcd2effd640 1 -- 172.21.15.116:0/2960149922 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd28011b70 con 0x7fcd380c2f50 2024-04-25T00:02:25.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.677+0000 7fcd3c9aa640 1 -- 172.21.15.116:0/2960149922 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd381ef360 con 0x7fcd380c2f50 2024-04-25T00:02:25.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.677+0000 7fcd3c9aa640 1 -- 172.21.15.116:0/2960149922 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd38154a20 con 0x7fcd380c2f50 2024-04-25T00:02:25.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.677+0000 7fcd3659c640 1 --2- 172.21.15.116:0/2960149922 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd380a94a0 0x7fcd380728b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:02:25.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.677+0000 7fcd3c9aa640 1 -- 172.21.15.116:0/2960149922 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd38068350 con 0x7fcd380c2f50 2024-04-25T00:02:25.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.677+0000 7fcd2effd640 1 -- 172.21.15.116:0/2960149922 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fcd28011d10 con 0x7fcd380c2f50 2024-04-25T00:02:25.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.677+0000 7fcd2effd640 1 --2- 172.21.15.116:0/2960149922 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcd180678f0 0x7fcd18069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:25.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.677+0000 7fcd2effd640 1 -- 172.21.15.116:0/2960149922 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(29..29 src has 1..29) ==== 6666+0+0 (secure 0 0 0) 0x7fcd280867c0 con 0x7fcd380c2f50 2024-04-25T00:02:25.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.677+0000 7fcd3659c640 1 --2- 172.21.15.116:0/2960149922 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcd180678f0 0x7fcd18069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:25.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.677+0000 7fcd3659c640 1 --2- 172.21.15.116:0/2960149922 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcd180678f0 0x7fcd18069db0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fcd20009f70 tx=0x7fcd20009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:02:25.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.693+0000 7fcd2effd640 1 -- 172.21.15.116:0/2960149922 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fcd280568e0 con 0x7fcd380c2f50 2024-04-25T00:02:25.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.889+0000 7fcd3c9aa640 1 -- 172.21.15.116:0/2960149922 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcd3806d4b0 con 0x7fcd380c2f50 2024-04-25T00:02:25.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.893+0000 7fcd2effd640 1 -- 172.21.15.116:0/2960149922 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6492 (secure 0 0 0) 0x7fcd2805abe0 con 0x7fcd380c2f50 2024-04-25T00:02:25.895 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:02:25.896 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6834/1459177921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1459177921},{"type":"v1","addr":"172.21.15.116:6834","nonce":1459177921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:01:08.748957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6837/1762356055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1762356055},{"type":"v1","addr":"172.21.15.174:6837","nonce":1762356055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270]},"id":1},{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:01:22.888641+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6835/174781672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":174781672},{"type":"v1","addr":"172.21.15.116:6835","nonce":174781672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4245,"qdb_cluster":[4245]},"id":2}]} 2024-04-25T00:02:25.896 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 14 2024-04-25T00:02:25.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.893+0000 7fcd3c9aa640 1 -- 172.21.15.116:0/2960149922 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcd180678f0 msgr2=0x7fcd18069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:25.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.893+0000 7fcd3c9aa640 1 --2- 172.21.15.116:0/2960149922 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcd180678f0 0x7fcd18069db0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fcd20009f70 tx=0x7fcd20009190 comp rx=0 tx=0).stop 2024-04-25T00:02:25.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.893+0000 7fcd3c9aa640 1 -- 172.21.15.116:0/2960149922 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcd380c2f50 msgr2=0x7fcd380c0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:25.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.893+0000 7fcd3c9aa640 1 --2- 172.21.15.116:0/2960149922 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcd380c2f50 0x7fcd380c0430 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcd2800c3e0 tx=0x7fcd2800c8a0 comp rx=0 tx=0).stop 2024-04-25T00:02:25.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.893+0000 7fcd3c9aa640 1 -- 172.21.15.116:0/2960149922 shutdown_connections 2024-04-25T00:02:25.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.893+0000 7fcd3c9aa640 1 --2- 172.21.15.116:0/2960149922 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcd180678f0 0x7fcd18069db0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:25.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.893+0000 7fcd3c9aa640 1 --2- 172.21.15.116:0/2960149922 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcd38150d20 0x7fcd38156c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:25.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.893+0000 7fcd3c9aa640 1 --2- 172.21.15.116:0/2960149922 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcd380c2f50 0x7fcd380c0430 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:25.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.893+0000 7fcd3c9aa640 1 --2- 172.21.15.116:0/2960149922 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd380a94a0 0x7fcd380728b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:25.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.893+0000 7fcd3c9aa640 1 -- 172.21.15.116:0/2960149922 >> 172.21.15.116:0/2960149922 conn(0x7fcd380c3b00 msgr2=0x7fcd380ac2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:02:25.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.893+0000 7fcd3c9aa640 1 -- 172.21.15.116:0/2960149922 shutdown_connections 2024-04-25T00:02:25.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:25.893+0000 7fcd3c9aa640 1 -- 172.21.15.116:0/2960149922 wait complete. 2024-04-25T00:02:25.915 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:02:27.916 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:02:28.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.033+0000 7f2da985c640 1 Processor -- start 2024-04-25T00:02:28.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.033+0000 7f2da985c640 1 -- start start 2024-04-25T00:02:28.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.033+0000 7f2da985c640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f2da4155b20 0x7f2da4152c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:28.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.033+0000 7f2da985c640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2da40a9e60 con 0x7f2da40aacd0 2024-04-25T00:02:28.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.033+0000 7f2da985c640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2da40c3ef0 con 0x7f2da414c700 2024-04-25T00:02:28.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.033+0000 7f2da985c640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f2da41527b0 con 0x7f2da4155b20 2024-04-25T00:02:28.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.033+0000 7f2da37fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f2da40aacd0 0x7f2da415ceb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:39518/0 (socket says 172.21.15.116:39518) 2024-04-25T00:02:28.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.033+0000 7f2da37fe640 1 -- 172.21.15.116:0/4007555190 learned_addr learned my addr 172.21.15.116:0/4007555190 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:02:28.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.033+0000 7f2da27fc640 1 --2- 172.21.15.116:0/4007555190 >> v2:172.21.15.174:3300/0 conn(0x7f2da4155b20 0x7f2da4152c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:28.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.033+0000 7f2da1ffb640 1 -- 172.21.15.116:0/4007555190 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4203158658 0 0) 0x7f2da40a9e60 con 0x7f2da40aacd0 2024-04-25T00:02:28.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.033+0000 7f2da1ffb640 1 -- 172.21.15.116:0/4007555190 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d8c003530 con 0x7f2da40aacd0 2024-04-25T00:02:28.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.033+0000 7f2da1ffb640 1 -- 172.21.15.116:0/4007555190 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1421700977 0 0) 0x7f2d8c003530 con 0x7f2da40aacd0 2024-04-25T00:02:28.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.033+0000 7f2da1ffb640 1 -- 172.21.15.116:0/4007555190 >> v2:172.21.15.174:3300/0 conn(0x7f2da4155b20 msgr2=0x7f2da4152c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:02:28.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.033+0000 7f2da1ffb640 1 --2- 172.21.15.116:0/4007555190 >> v2:172.21.15.174:3300/0 conn(0x7f2da4155b20 0x7f2da4152c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:28.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.033+0000 7f2da1ffb640 1 -- 172.21.15.116:0/4007555190 >> v1:172.21.15.174:6789/0 conn(0x7f2da414c700 legacy=0x7f2da40c5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:28.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.033+0000 7f2da1ffb640 1 -- 172.21.15.116:0/4007555190 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2da40c41d0 con 0x7f2da40aacd0 2024-04-25T00:02:28.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.033+0000 7f2da1ffb640 1 -- 172.21.15.116:0/4007555190 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f2d9c001a00 con 0x7f2da40aacd0 2024-04-25T00:02:28.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.033+0000 7f2da1ffb640 1 -- 172.21.15.116:0/4007555190 >> v1:172.21.15.116:6789/0 conn(0x7f2da40aacd0 legacy=0x7f2da415ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:28.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.033+0000 7f2da1ffb640 1 --2- 172.21.15.116:0/4007555190 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d8c0039c0 0x7f2d8c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:28.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.033+0000 7f2da1ffb640 1 --2- 172.21.15.116:0/4007555190 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d8c004810 0x7f2d8c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:28.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.033+0000 7f2da1ffb640 1 --2- 172.21.15.116:0/4007555190 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2d8c0052f0 0x7f2d8c00dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:28.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.033+0000 7f2da1ffb640 1 -- 172.21.15.116:0/4007555190 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2da41527b0 con 0x7f2d8c0039c0 2024-04-25T00:02:28.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.033+0000 7f2da1ffb640 1 -- 172.21.15.116:0/4007555190 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2d8c00e3d0 con 0x7f2d8c004810 2024-04-25T00:02:28.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.033+0000 7f2da1ffb640 1 -- 172.21.15.116:0/4007555190 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f2d8c00e5d0 con 0x7f2d8c0052f0 2024-04-25T00:02:28.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.033+0000 7f2da2ffd640 1 --2- 172.21.15.116:0/4007555190 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d8c0039c0 0x7f2d8c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:28.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.033+0000 7f2da27fc640 1 --2- 172.21.15.116:0/4007555190 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d8c004810 0x7f2d8c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:28.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.033+0000 7f2da37fe640 1 --2- 172.21.15.116:0/4007555190 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2d8c0052f0 0x7f2d8c00dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:28.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.033+0000 7f2da2ffd640 1 -- 172.21.15.116:0/4007555190 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2d8c0052f0 msgr2=0x7f2d8c00dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:02:28.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.033+0000 7f2da2ffd640 1 --2- 172.21.15.116:0/4007555190 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2d8c0052f0 0x7f2d8c00dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:28.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.033+0000 7f2da2ffd640 1 -- 172.21.15.116:0/4007555190 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d8c004810 msgr2=0x7f2d8c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:02:28.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.033+0000 7f2da2ffd640 1 --2- 172.21.15.116:0/4007555190 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d8c004810 0x7f2d8c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:28.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.033+0000 7f2da2ffd640 1 -- 172.21.15.116:0/4007555190 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d8c003530 con 0x7f2d8c0039c0 2024-04-25T00:02:28.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.033+0000 7f2da37fe640 1 --2- 172.21.15.116:0/4007555190 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2d8c0052f0 0x7f2d8c00dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:02:28.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.033+0000 7f2da27fc640 1 --2- 172.21.15.116:0/4007555190 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d8c004810 0x7f2d8c004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:02:28.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.033+0000 7f2da37fe640 1 -- 172.21.15.116:0/4007555190 reap_dead start 2024-04-25T00:02:28.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.037+0000 7f2da2ffd640 1 --2- 172.21.15.116:0/4007555190 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d8c0039c0 0x7f2d8c003de0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f2d98001360 tx=0x7f2d980207d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1637a25ee462841 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:02:28.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.037+0000 7f2da1ffb640 1 -- 172.21.15.116:0/4007555190 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d980b4070 con 0x7f2d8c0039c0 2024-04-25T00:02:28.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.037+0000 7f2da1ffb640 1 -- 172.21.15.116:0/4007555190 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d980050d0 con 0x7f2d8c0039c0 2024-04-25T00:02:28.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.037+0000 7f2da1ffb640 1 -- 172.21.15.116:0/4007555190 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d980013a0 con 0x7f2d8c0039c0 2024-04-25T00:02:28.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.037+0000 7f2da985c640 1 -- 172.21.15.116:0/4007555190 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d8c0039c0 msgr2=0x7f2d8c003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:28.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.037+0000 7f2da985c640 1 --2- 172.21.15.116:0/4007555190 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d8c0039c0 0x7f2d8c003de0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f2d98001360 tx=0x7f2d980207d0 comp rx=0 tx=0).stop 2024-04-25T00:02:28.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.037+0000 7f2da985c640 1 -- 172.21.15.116:0/4007555190 shutdown_connections 2024-04-25T00:02:28.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.037+0000 7f2da985c640 1 --2- 172.21.15.116:0/4007555190 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d8c0039c0 0x7f2d8c003de0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:28.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.037+0000 7f2da985c640 1 -- 172.21.15.116:0/4007555190 >> 172.21.15.116:0/4007555190 conn(0x7f2da406c720 msgr2=0x7f2da406cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:02:28.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.037+0000 7f2da985c640 1 -- 172.21.15.116:0/4007555190 shutdown_connections 2024-04-25T00:02:28.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.037+0000 7f2da985c640 1 -- 172.21.15.116:0/4007555190 wait complete. 2024-04-25T00:02:28.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.037+0000 7f2da985c640 1 Processor -- start 2024-04-25T00:02:28.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.037+0000 7f2da985c640 1 -- start start 2024-04-25T00:02:28.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.037+0000 7f2da985c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2d8c0039c0 0x7f2da41ed750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:28.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.037+0000 7f2da985c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2da41edc90 0x7f2da414b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:28.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.037+0000 7f2da985c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2da414ba30 0x7f2da41487e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:28.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.037+0000 7f2da985c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2da4152fb0 con 0x7f2da41edc90 2024-04-25T00:02:28.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.037+0000 7f2da985c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2da4152e30 con 0x7f2da414ba30 2024-04-25T00:02:28.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.037+0000 7f2da985c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f2da4153130 con 0x7f2d8c0039c0 2024-04-25T00:02:28.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.037+0000 7f2da27fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2da41edc90 0x7f2da414b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:28.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.037+0000 7f2da27fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2da41edc90 0x7f2da414b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41174/0 (socket says 172.21.15.116:41174) 2024-04-25T00:02:28.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.037+0000 7f2da27fc640 1 -- 172.21.15.116:0/1142260123 learned_addr learned my addr 172.21.15.116:0/1142260123 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:02:28.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.041+0000 7f2da2ffd640 1 --2- 172.21.15.116:0/1142260123 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2d8c0039c0 0x7f2da41ed750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:28.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.041+0000 7f2da27fc640 1 -- 172.21.15.116:0/1142260123 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2d8c0039c0 msgr2=0x7f2da41ed750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:28.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.041+0000 7f2da27fc640 1 --2- 172.21.15.116:0/1142260123 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2d8c0039c0 0x7f2da41ed750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:28.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.041+0000 7f2da27fc640 1 -- 172.21.15.116:0/1142260123 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2da414ba30 msgr2=0x7f2da41487e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:28.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.041+0000 7f2da27fc640 1 --2- 172.21.15.116:0/1142260123 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2da414ba30 0x7f2da41487e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:28.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.041+0000 7f2da27fc640 1 -- 172.21.15.116:0/1142260123 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2da414b010 con 0x7f2da41edc90 2024-04-25T00:02:28.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.041+0000 7f2da27fc640 1 --2- 172.21.15.116:0/1142260123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2da41edc90 0x7f2da414b4f0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f2d94010950 tx=0x7f2d94010e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:02:28.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.041+0000 7f2d8bfff640 1 -- 172.21.15.116:0/1142260123 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d9400f040 con 0x7f2da41edc90 2024-04-25T00:02:28.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.041+0000 7f2d8bfff640 1 -- 172.21.15.116:0/1142260123 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d94004220 con 0x7f2da41edc90 2024-04-25T00:02:28.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.041+0000 7f2d8bfff640 1 -- 172.21.15.116:0/1142260123 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d9400e9c0 con 0x7f2da41edc90 2024-04-25T00:02:28.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.041+0000 7f2da985c640 1 -- 172.21.15.116:0/1142260123 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2da41505a0 con 0x7f2da41edc90 2024-04-25T00:02:28.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.041+0000 7f2da985c640 1 -- 172.21.15.116:0/1142260123 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2da41ee1e0 con 0x7f2da41edc90 2024-04-25T00:02:28.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.041+0000 7f2da985c640 1 -- 172.21.15.116:0/1142260123 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2da4068a60 con 0x7f2da41edc90 2024-04-25T00:02:28.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.041+0000 7f2d8bfff640 1 -- 172.21.15.116:0/1142260123 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2d9400eb60 con 0x7f2da41edc90 2024-04-25T00:02:28.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.041+0000 7f2d8bfff640 1 --2- 172.21.15.116:0/1142260123 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2d900678f0 0x7f2d90069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:28.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.041+0000 7f2d8bfff640 1 -- 172.21.15.116:0/1142260123 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6666+0+0 (secure 0 0 0) 0x7f2d94086010 con 0x7f2da41edc90 2024-04-25T00:02:28.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.041+0000 7f2da2ffd640 1 --2- 172.21.15.116:0/1142260123 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2d900678f0 0x7f2d90069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:28.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.041+0000 7f2da2ffd640 1 --2- 172.21.15.116:0/1142260123 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2d900678f0 0x7f2d90069db0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c0057f0 tx=0x7f2d980011a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:02:28.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.045+0000 7f2d8bfff640 1 -- 172.21.15.116:0/1142260123 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f2d94018030 con 0x7f2da41edc90 2024-04-25T00:02:28.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.257+0000 7f2da985c640 1 -- 172.21.15.116:0/1142260123 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2da40ad630 con 0x7f2da41edc90 2024-04-25T00:02:28.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.257+0000 7f2d8bfff640 1 -- 172.21.15.116:0/1142260123 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6492 (secure 0 0 0) 0x7f2d94056790 con 0x7f2da41edc90 2024-04-25T00:02:28.262 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:02:28.263 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6834/1459177921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1459177921},{"type":"v1","addr":"172.21.15.116:6834","nonce":1459177921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:01:08.748957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6837/1762356055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1762356055},{"type":"v1","addr":"172.21.15.174:6837","nonce":1762356055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270]},"id":1},{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:01:22.888641+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6835/174781672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":174781672},{"type":"v1","addr":"172.21.15.116:6835","nonce":174781672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4245,"qdb_cluster":[4245]},"id":2}]} 2024-04-25T00:02:28.265 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 14 2024-04-25T00:02:28.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.261+0000 7f2da985c640 1 -- 172.21.15.116:0/1142260123 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2d900678f0 msgr2=0x7f2d90069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:28.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.261+0000 7f2da985c640 1 --2- 172.21.15.116:0/1142260123 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2d900678f0 0x7f2d90069db0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c0057f0 tx=0x7f2d980011a0 comp rx=0 tx=0).stop 2024-04-25T00:02:28.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.261+0000 7f2da985c640 1 -- 172.21.15.116:0/1142260123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2da41edc90 msgr2=0x7f2da414b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:28.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.261+0000 7f2da985c640 1 --2- 172.21.15.116:0/1142260123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2da41edc90 0x7f2da414b4f0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f2d94010950 tx=0x7f2d94010e10 comp rx=0 tx=0).stop 2024-04-25T00:02:28.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.261+0000 7f2da985c640 1 -- 172.21.15.116:0/1142260123 shutdown_connections 2024-04-25T00:02:28.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.261+0000 7f2da985c640 1 --2- 172.21.15.116:0/1142260123 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2d900678f0 0x7f2d90069db0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:28.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.261+0000 7f2da985c640 1 --2- 172.21.15.116:0/1142260123 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2da414ba30 0x7f2da41487e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:28.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.261+0000 7f2da985c640 1 --2- 172.21.15.116:0/1142260123 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2da41edc90 0x7f2da414b4f0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:28.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.261+0000 7f2da985c640 1 --2- 172.21.15.116:0/1142260123 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2d8c0039c0 0x7f2da41ed750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:28.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.261+0000 7f2da985c640 1 -- 172.21.15.116:0/1142260123 >> 172.21.15.116:0/1142260123 conn(0x7f2da406c720 msgr2=0x7f2da4154400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:02:28.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.261+0000 7f2da985c640 1 -- 172.21.15.116:0/1142260123 shutdown_connections 2024-04-25T00:02:28.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:28.261+0000 7f2da985c640 1 -- 172.21.15.116:0/1142260123 wait complete. 2024-04-25T00:02:28.279 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:02:30.280 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:02:30.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.401+0000 7fdee1b47640 1 Processor -- start 2024-04-25T00:02:30.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.401+0000 7fdee1b47640 1 -- start start 2024-04-25T00:02:30.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.401+0000 7fdee1b47640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fdedc0c5a60 0x7fdedc0c5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:30.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.401+0000 7fdee1b47640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fdedc1558c0 0x7fdedc155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:30.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.401+0000 7fdee1b47640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdedc158eb0 con 0x7fdedc152db0 2024-04-25T00:02:30.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.401+0000 7fdee1b47640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fdedc159190 con 0x7fdedc0c5a60 2024-04-25T00:02:30.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.401+0000 7fdee1b47640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fdedc159470 con 0x7fdedc1558c0 2024-04-25T00:02:30.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.401+0000 7fded3fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fdedc0c5a60 0x7fdedc0c5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:30.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.401+0000 7fdedb7fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fdedc152db0 0x7fdedc0c7af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:39534/0 (socket says 172.21.15.116:39534) 2024-04-25T00:02:30.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.401+0000 7fdedb7fe640 1 -- 172.21.15.116:0/334719138 learned_addr learned my addr 172.21.15.116:0/334719138 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:02:30.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.401+0000 7fdedbfff640 1 --2- 172.21.15.116:0/334719138 >> v2:172.21.15.174:3300/0 conn(0x7fdedc1558c0 0x7fdedc155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:30.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.401+0000 7fded3fff640 1 -- 172.21.15.116:0/334719138 >> v2:172.21.15.174:3300/0 conn(0x7fdedc1558c0 msgr2=0x7fdedc155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:02:30.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.401+0000 7fded3fff640 1 --2- 172.21.15.116:0/334719138 >> v2:172.21.15.174:3300/0 conn(0x7fdedc1558c0 0x7fdedc155cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:30.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.401+0000 7fded3fff640 1 -- 172.21.15.116:0/334719138 >> v1:172.21.15.116:6789/0 conn(0x7fdedc152db0 legacy=0x7fdedc0c7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:30.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.401+0000 7fded3fff640 1 -- 172.21.15.116:0/334719138 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdedc159750 con 0x7fdedc0c5a60 2024-04-25T00:02:30.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.401+0000 7fdedaffd640 1 -- 172.21.15.116:0/334719138 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2750939600 0 0) 0x7fdedc158eb0 con 0x7fdedc152db0 2024-04-25T00:02:30.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.401+0000 7fded3fff640 1 --2- 172.21.15.116:0/334719138 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdedc0c5a60 0x7fdedc0c5ef0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fdedc0b1230 tx=0x7fdec401ce20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=babbcf941585cb29 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:02:30.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.405+0000 7fdedaffd640 1 -- 172.21.15.116:0/334719138 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec40b1070 con 0x7fdedc0c5a60 2024-04-25T00:02:30.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.405+0000 7fdedaffd640 1 -- 172.21.15.116:0/334719138 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdec40a8050 con 0x7fdedc0c5a60 2024-04-25T00:02:30.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.405+0000 7fdedaffd640 1 -- 172.21.15.116:0/334719138 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec40ac860 con 0x7fdedc0c5a60 2024-04-25T00:02:30.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.405+0000 7fdee1b47640 1 -- 172.21.15.116:0/334719138 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdedc0c5a60 msgr2=0x7fdedc0c5ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:30.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.405+0000 7fdee1b47640 1 --2- 172.21.15.116:0/334719138 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdedc0c5a60 0x7fdedc0c5ef0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fdedc0b1230 tx=0x7fdec401ce20 comp rx=0 tx=0).stop 2024-04-25T00:02:30.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.405+0000 7fdee1b47640 1 -- 172.21.15.116:0/334719138 shutdown_connections 2024-04-25T00:02:30.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.405+0000 7fdee1b47640 1 --2- 172.21.15.116:0/334719138 >> v2:172.21.15.174:3300/0 conn(0x7fdedc1558c0 0x7fdedc155cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:30.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.405+0000 7fdee1b47640 1 --2- 172.21.15.116:0/334719138 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdedc0c5a60 0x7fdedc0c5ef0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:30.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.405+0000 7fdee1b47640 1 -- 172.21.15.116:0/334719138 >> 172.21.15.116:0/334719138 conn(0x7fdedc0acc10 msgr2=0x7fdedc0ad020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:02:30.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.405+0000 7fdee1b47640 1 -- 172.21.15.116:0/334719138 shutdown_connections 2024-04-25T00:02:30.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.405+0000 7fdee1b47640 1 -- 172.21.15.116:0/334719138 wait complete. 2024-04-25T00:02:30.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.405+0000 7fdee1b47640 1 Processor -- start 2024-04-25T00:02:30.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.405+0000 7fdee1b47640 1 -- start start 2024-04-25T00:02:30.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.405+0000 7fdee1b47640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdedc0c5a60 0x7fdedc146530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:30.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.405+0000 7fdee1b47640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdedc152db0 0x7fdedc14bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:30.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.405+0000 7fdee1b47640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdedc1558c0 0x7fdedc1f22d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:30.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.405+0000 7fdee1b47640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdedc153310 con 0x7fdedc152db0 2024-04-25T00:02:30.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.405+0000 7fdee1b47640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fdedc153190 con 0x7fdedc1558c0 2024-04-25T00:02:30.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.405+0000 7fdee1b47640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fdedc162a60 con 0x7fdedc0c5a60 2024-04-25T00:02:30.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.405+0000 7fded3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdedc152db0 0x7fdedc14bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:30.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.405+0000 7fded3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdedc152db0 0x7fdedc14bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41184/0 (socket says 172.21.15.116:41184) 2024-04-25T00:02:30.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.405+0000 7fded3fff640 1 -- 172.21.15.116:0/720066690 learned_addr learned my addr 172.21.15.116:0/720066690 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:02:30.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.405+0000 7fdedbfff640 1 --2- 172.21.15.116:0/720066690 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdedc1558c0 0x7fdedc1f22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:30.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.405+0000 7fdedb7fe640 1 --2- 172.21.15.116:0/720066690 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdedc0c5a60 0x7fdedc146530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:30.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.405+0000 7fded3fff640 1 -- 172.21.15.116:0/720066690 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdedc0c5a60 msgr2=0x7fdedc146530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:30.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.405+0000 7fded3fff640 1 --2- 172.21.15.116:0/720066690 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdedc0c5a60 0x7fdedc146530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:30.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.405+0000 7fded3fff640 1 -- 172.21.15.116:0/720066690 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdedc1558c0 msgr2=0x7fdedc1f22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:30.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.405+0000 7fded3fff640 1 --2- 172.21.15.116:0/720066690 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdedc1558c0 0x7fdedc1f22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:30.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.405+0000 7fded3fff640 1 -- 172.21.15.116:0/720066690 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdedc14cca0 con 0x7fdedc152db0 2024-04-25T00:02:30.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.405+0000 7fded3fff640 1 --2- 172.21.15.116:0/720066690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdedc152db0 0x7fdedc14bcb0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fdec4019ff0 tx=0x7fdec401ac00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:02:30.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.405+0000 7fded8ff9640 1 -- 172.21.15.116:0/720066690 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec40bd070 con 0x7fdedc152db0 2024-04-25T00:02:30.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.405+0000 7fded8ff9640 1 -- 172.21.15.116:0/720066690 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdec40b2040 con 0x7fdedc152db0 2024-04-25T00:02:30.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.405+0000 7fdee1b47640 1 -- 172.21.15.116:0/720066690 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdedc14cf80 con 0x7fdedc152db0 2024-04-25T00:02:30.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.405+0000 7fded8ff9640 1 -- 172.21.15.116:0/720066690 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec40b1050 con 0x7fdedc152db0 2024-04-25T00:02:30.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.405+0000 7fdee1b47640 1 -- 172.21.15.116:0/720066690 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdedc0ac710 con 0x7fdedc152db0 2024-04-25T00:02:30.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.409+0000 7fdee1b47640 1 -- 172.21.15.116:0/720066690 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdedc068a60 con 0x7fdedc152db0 2024-04-25T00:02:30.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.409+0000 7fded8ff9640 1 -- 172.21.15.116:0/720066690 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fdec40c0070 con 0x7fdedc152db0 2024-04-25T00:02:30.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.409+0000 7fded8ff9640 1 --2- 172.21.15.116:0/720066690 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdebc0678f0 0x7fdebc069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:30.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.409+0000 7fded8ff9640 1 -- 172.21.15.116:0/720066690 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6666+0+0 (secure 0 0 0) 0x7fdec411fc60 con 0x7fdedc152db0 2024-04-25T00:02:30.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.409+0000 7fdedb7fe640 1 --2- 172.21.15.116:0/720066690 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdebc0678f0 0x7fdebc069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:30.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.409+0000 7fdedb7fe640 1 --2- 172.21.15.116:0/720066690 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdebc0678f0 0x7fdebc069db0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fdecc000f70 tx=0x7fdecc007b80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:02:30.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.413+0000 7fded8ff9640 1 -- 172.21.15.116:0/720066690 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fdec4016070 con 0x7fdedc152db0 2024-04-25T00:02:30.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.613+0000 7fdee1b47640 1 -- 172.21.15.116:0/720066690 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdedc0c51e0 con 0x7fdedc152db0 2024-04-25T00:02:30.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.613+0000 7fded8ff9640 1 -- 172.21.15.116:0/720066690 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6492 (secure 0 0 0) 0x7fdec40f0470 con 0x7fdedc152db0 2024-04-25T00:02:30.617 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:02:30.617 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6834/1459177921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1459177921},{"type":"v1","addr":"172.21.15.116:6834","nonce":1459177921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:01:08.748957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6837/1762356055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1762356055},{"type":"v1","addr":"172.21.15.174:6837","nonce":1762356055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270]},"id":1},{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:01:22.888641+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6835/174781672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":174781672},{"type":"v1","addr":"172.21.15.116:6835","nonce":174781672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4245,"qdb_cluster":[4245]},"id":2}]} 2024-04-25T00:02:30.617 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 14 2024-04-25T00:02:30.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.617+0000 7fdee1b47640 1 -- 172.21.15.116:0/720066690 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdebc0678f0 msgr2=0x7fdebc069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:30.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.617+0000 7fdee1b47640 1 --2- 172.21.15.116:0/720066690 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdebc0678f0 0x7fdebc069db0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fdecc000f70 tx=0x7fdecc007b80 comp rx=0 tx=0).stop 2024-04-25T00:02:30.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.617+0000 7fdee1b47640 1 -- 172.21.15.116:0/720066690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdedc152db0 msgr2=0x7fdedc14bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:30.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.617+0000 7fdee1b47640 1 --2- 172.21.15.116:0/720066690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdedc152db0 0x7fdedc14bcb0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fdec4019ff0 tx=0x7fdec401ac00 comp rx=0 tx=0).stop 2024-04-25T00:02:30.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.617+0000 7fdee1b47640 1 -- 172.21.15.116:0/720066690 shutdown_connections 2024-04-25T00:02:30.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.617+0000 7fdee1b47640 1 --2- 172.21.15.116:0/720066690 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdebc0678f0 0x7fdebc069db0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:30.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.617+0000 7fdee1b47640 1 --2- 172.21.15.116:0/720066690 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdedc1558c0 0x7fdedc1f22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:30.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.617+0000 7fdee1b47640 1 --2- 172.21.15.116:0/720066690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdedc152db0 0x7fdedc14bcb0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:30.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.617+0000 7fdee1b47640 1 --2- 172.21.15.116:0/720066690 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdedc0c5a60 0x7fdedc146530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:30.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.617+0000 7fdee1b47640 1 -- 172.21.15.116:0/720066690 >> 172.21.15.116:0/720066690 conn(0x7fdedc0acc10 msgr2=0x7fdedc0c7af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:02:30.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.617+0000 7fdee1b47640 1 -- 172.21.15.116:0/720066690 shutdown_connections 2024-04-25T00:02:30.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:30.617+0000 7fdee1b47640 1 -- 172.21.15.116:0/720066690 wait complete. 2024-04-25T00:02:30.635 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:02:32.636 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:02:32.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.752+0000 7fa0fe15b640 1 Processor -- start 2024-04-25T00:02:32.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.752+0000 7fa0fe15b640 1 -- start start 2024-04-25T00:02:32.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.752+0000 7fa0fe15b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa0f80aeb20 0x7fa0f80c9310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:32.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.752+0000 7fa0fe15b640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fa0f8150f80 0x7fa0f80c5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:32.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.752+0000 7fa0fe15b640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0f81535d0 con 0x7fa0f80c9850 2024-04-25T00:02:32.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.752+0000 7fa0fe15b640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fa0f80c9cb0 con 0x7fa0f8150f80 2024-04-25T00:02:32.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.752+0000 7fa0fe15b640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa0f8153850 con 0x7fa0f80aeb20 2024-04-25T00:02:32.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.752+0000 7fa0eeffd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fa0f80c9850 0x7fa0f8150870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:39544/0 (socket says 172.21.15.116:39544) 2024-04-25T00:02:32.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.752+0000 7fa0eeffd640 1 -- 172.21.15.116:0/2433058604 learned_addr learned my addr 172.21.15.116:0/2433058604 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:02:32.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.752+0000 7fa0f77fe640 1 --2- 172.21.15.116:0/2433058604 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa0f80aeb20 0x7fa0f80c9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:32.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.752+0000 7fa0f7fff640 1 --2- 172.21.15.116:0/2433058604 >> v2:172.21.15.174:3300/0 conn(0x7fa0f8150f80 0x7fa0f80c5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:32.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.752+0000 7fa0f6ffd640 1 -- 172.21.15.116:0/2433058604 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2717061975 0 0) 0x7fa0e0001120 con 0x7fa0f80c9850 2024-04-25T00:02:32.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.752+0000 7fa0f6ffd640 1 -- 172.21.15.116:0/2433058604 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa0e4003530 con 0x7fa0f80c9850 2024-04-25T00:02:32.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.752+0000 7fa0f77fe640 1 -- 172.21.15.116:0/2433058604 >> v2:172.21.15.174:3300/0 conn(0x7fa0f8150f80 msgr2=0x7fa0f80c5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:02:32.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.752+0000 7fa0f77fe640 1 --2- 172.21.15.116:0/2433058604 >> v2:172.21.15.174:3300/0 conn(0x7fa0f8150f80 0x7fa0f80c5270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:32.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.752+0000 7fa0f6ffd640 1 -- 172.21.15.116:0/2433058604 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4077672403 0 0) 0x7fa0e4003530 con 0x7fa0f80c9850 2024-04-25T00:02:32.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.752+0000 7fa0f77fe640 1 -- 172.21.15.116:0/2433058604 >> v1:172.21.15.116:6789/0 conn(0x7fa0f80c9850 legacy=0x7fa0f8150870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:32.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.752+0000 7fa0f77fe640 1 -- 172.21.15.116:0/2433058604 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0f8153b30 con 0x7fa0f80aeb20 2024-04-25T00:02:32.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.756+0000 7fa0f77fe640 1 --2- 172.21.15.116:0/2433058604 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa0f80aeb20 0x7fa0f80c9310 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa0e80097c0 tx=0x7fa0e801c300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cac844d0191db49e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:02:32.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.756+0000 7fa0f6ffd640 1 -- 172.21.15.116:0/2433058604 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0e80b0070 con 0x7fa0f80aeb20 2024-04-25T00:02:32.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.756+0000 7fa0f6ffd640 1 -- 172.21.15.116:0/2433058604 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0e80ab400 con 0x7fa0f80aeb20 2024-04-25T00:02:32.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.756+0000 7fa0f6ffd640 1 -- 172.21.15.116:0/2433058604 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0e0001850 con 0x7fa0f80aeb20 2024-04-25T00:02:32.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.756+0000 7fa0fe15b640 1 -- 172.21.15.116:0/2433058604 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa0f80aeb20 msgr2=0x7fa0f80c9310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:32.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.756+0000 7fa0fe15b640 1 --2- 172.21.15.116:0/2433058604 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa0f80aeb20 0x7fa0f80c9310 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa0e80097c0 tx=0x7fa0e801c300 comp rx=0 tx=0).stop 2024-04-25T00:02:32.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.756+0000 7fa0fe15b640 1 -- 172.21.15.116:0/2433058604 shutdown_connections 2024-04-25T00:02:32.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.756+0000 7fa0fe15b640 1 --2- 172.21.15.116:0/2433058604 >> v2:172.21.15.174:3300/0 conn(0x7fa0f8150f80 0x7fa0f80c5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:32.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.756+0000 7fa0fe15b640 1 --2- 172.21.15.116:0/2433058604 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa0f80aeb20 0x7fa0f80c9310 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:32.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.756+0000 7fa0fe15b640 1 -- 172.21.15.116:0/2433058604 >> 172.21.15.116:0/2433058604 conn(0x7fa0f8074650 msgr2=0x7fa0f8074a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:02:32.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.756+0000 7fa0fe15b640 1 -- 172.21.15.116:0/2433058604 shutdown_connections 2024-04-25T00:02:32.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.756+0000 7fa0fe15b640 1 -- 172.21.15.116:0/2433058604 wait complete. 2024-04-25T00:02:32.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.756+0000 7fa0fe15b640 1 Processor -- start 2024-04-25T00:02:32.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.756+0000 7fa0fe15b640 1 -- start start 2024-04-25T00:02:32.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.756+0000 7fa0fe15b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa0f80aeb20 0x7fa0f80c2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:32.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.756+0000 7fa0fe15b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa0f80c9850 0x7fa0f80ac0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:32.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.756+0000 7fa0fe15b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0f8150f80 0x7fa0f80ac5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:32.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.756+0000 7fa0fe15b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa0f80af080 con 0x7fa0f8150f80 2024-04-25T00:02:32.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.756+0000 7fa0fe15b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa0f80aef00 con 0x7fa0f80c9850 2024-04-25T00:02:32.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.756+0000 7fa0fe15b640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa0f815ff60 con 0x7fa0f80aeb20 2024-04-25T00:02:32.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.756+0000 7fa0f7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0f8150f80 0x7fa0f80ac5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:32.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.756+0000 7fa0f7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0f8150f80 0x7fa0f80ac5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41194/0 (socket says 172.21.15.116:41194) 2024-04-25T00:02:32.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.756+0000 7fa0f7fff640 1 -- 172.21.15.116:0/398612386 learned_addr learned my addr 172.21.15.116:0/398612386 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:02:32.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.756+0000 7fa0f77fe640 1 --2- 172.21.15.116:0/398612386 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa0f80aeb20 0x7fa0f80c2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:32.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.756+0000 7fa0eeffd640 1 --2- 172.21.15.116:0/398612386 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa0f80c9850 0x7fa0f80ac0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:32.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.756+0000 7fa0f7fff640 1 -- 172.21.15.116:0/398612386 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa0f80aeb20 msgr2=0x7fa0f80c2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:32.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.756+0000 7fa0f7fff640 1 --2- 172.21.15.116:0/398612386 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa0f80aeb20 0x7fa0f80c2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:32.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.756+0000 7fa0f7fff640 1 -- 172.21.15.116:0/398612386 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa0f80c9850 msgr2=0x7fa0f80ac0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:32.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.756+0000 7fa0f7fff640 1 --2- 172.21.15.116:0/398612386 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa0f80c9850 0x7fa0f80ac0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:32.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.756+0000 7fa0f7fff640 1 -- 172.21.15.116:0/398612386 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0f80c7ce0 con 0x7fa0f8150f80 2024-04-25T00:02:32.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.756+0000 7fa0f7fff640 1 --2- 172.21.15.116:0/398612386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0f8150f80 0x7fa0f80ac5f0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fa0f0009d30 tx=0x7fa0f000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:02:32.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.760+0000 7fa0f4ff9640 1 -- 172.21.15.116:0/398612386 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f0018020 con 0x7fa0f8150f80 2024-04-25T00:02:32.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.760+0000 7fa0f4ff9640 1 -- 172.21.15.116:0/398612386 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0f0014480 con 0x7fa0f8150f80 2024-04-25T00:02:32.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.760+0000 7fa0f4ff9640 1 -- 172.21.15.116:0/398612386 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f0014870 con 0x7fa0f8150f80 2024-04-25T00:02:32.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.760+0000 7fa0fe15b640 1 -- 172.21.15.116:0/398612386 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0f80c7fc0 con 0x7fa0f8150f80 2024-04-25T00:02:32.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.760+0000 7fa0fe15b640 1 -- 172.21.15.116:0/398612386 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0f80c8580 con 0x7fa0f8150f80 2024-04-25T00:02:32.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.760+0000 7fa0fe15b640 1 -- 172.21.15.116:0/398612386 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0f8068a60 con 0x7fa0f8150f80 2024-04-25T00:02:32.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.760+0000 7fa0f4ff9640 1 -- 172.21.15.116:0/398612386 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa0f001b020 con 0x7fa0f8150f80 2024-04-25T00:02:32.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.760+0000 7fa0f4ff9640 1 --2- 172.21.15.116:0/398612386 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa0d80678c0 0x7fa0d8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:32.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.760+0000 7fa0f4ff9640 1 -- 172.21.15.116:0/398612386 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6666+0+0 (secure 0 0 0) 0x7fa0f0086d50 con 0x7fa0f8150f80 2024-04-25T00:02:32.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.760+0000 7fa0f77fe640 1 --2- 172.21.15.116:0/398612386 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa0d80678c0 0x7fa0d8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:32.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.764+0000 7fa0f77fe640 1 --2- 172.21.15.116:0/398612386 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa0d80678c0 0x7fa0d8069d80 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa0e80ad000 tx=0x7fa0e80a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:02:32.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.764+0000 7fa0f4ff9640 1 -- 172.21.15.116:0/398612386 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fa0f008a030 con 0x7fa0f8150f80 2024-04-25T00:02:32.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.976+0000 7fa0fe15b640 1 -- 172.21.15.116:0/398612386 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa0f81526f0 con 0x7fa0f8150f80 2024-04-25T00:02:32.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.976+0000 7fa0f4ff9640 1 -- 172.21.15.116:0/398612386 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6492 (secure 0 0 0) 0x7fa0f0022030 con 0x7fa0f8150f80 2024-04-25T00:02:32.982 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:02:32.982 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6834/1459177921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1459177921},{"type":"v1","addr":"172.21.15.116:6834","nonce":1459177921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:01:08.748957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6837/1762356055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1762356055},{"type":"v1","addr":"172.21.15.174:6837","nonce":1762356055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270]},"id":1},{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:01:22.888641+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6835/174781672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":174781672},{"type":"v1","addr":"172.21.15.116:6835","nonce":174781672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4245,"qdb_cluster":[4245]},"id":2}]} 2024-04-25T00:02:32.982 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 14 2024-04-25T00:02:32.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.980+0000 7fa0fe15b640 1 -- 172.21.15.116:0/398612386 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa0d80678c0 msgr2=0x7fa0d8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:32.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.980+0000 7fa0fe15b640 1 --2- 172.21.15.116:0/398612386 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa0d80678c0 0x7fa0d8069d80 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa0e80ad000 tx=0x7fa0e80a6000 comp rx=0 tx=0).stop 2024-04-25T00:02:32.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.980+0000 7fa0fe15b640 1 -- 172.21.15.116:0/398612386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0f8150f80 msgr2=0x7fa0f80ac5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:32.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.980+0000 7fa0fe15b640 1 --2- 172.21.15.116:0/398612386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0f8150f80 0x7fa0f80ac5f0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fa0f0009d30 tx=0x7fa0f000c490 comp rx=0 tx=0).stop 2024-04-25T00:02:32.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.980+0000 7fa0fe15b640 1 -- 172.21.15.116:0/398612386 shutdown_connections 2024-04-25T00:02:32.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.980+0000 7fa0fe15b640 1 --2- 172.21.15.116:0/398612386 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa0d80678c0 0x7fa0d8069d80 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:32.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.980+0000 7fa0fe15b640 1 --2- 172.21.15.116:0/398612386 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0f8150f80 0x7fa0f80ac5f0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:32.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.980+0000 7fa0fe15b640 1 --2- 172.21.15.116:0/398612386 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa0f80c9850 0x7fa0f80ac0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:32.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.980+0000 7fa0fe15b640 1 --2- 172.21.15.116:0/398612386 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa0f80aeb20 0x7fa0f80c2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:32.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.980+0000 7fa0fe15b640 1 -- 172.21.15.116:0/398612386 >> 172.21.15.116:0/398612386 conn(0x7fa0f8074650 msgr2=0x7fa0f8070f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:02:32.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.980+0000 7fa0fe15b640 1 -- 172.21.15.116:0/398612386 shutdown_connections 2024-04-25T00:02:32.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:32.980+0000 7fa0fe15b640 1 -- 172.21.15.116:0/398612386 wait complete. 2024-04-25T00:02:33.000 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:02:34.684 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:02:34.687 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:02:35.001 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:02:35.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.116+0000 7fad58bef640 1 Processor -- start 2024-04-25T00:02:35.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.116+0000 7fad58bef640 1 -- start start 2024-04-25T00:02:35.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.120+0000 7fad58bef640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fad5414f400 0x7fad540c2170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:35.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.120+0000 7fad58bef640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fad540c6080 0x7fad5406c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:35.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.120+0000 7fad58bef640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad5414f7e0 con 0x7fad5406c760 2024-04-25T00:02:35.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.120+0000 7fad58bef640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fad54152990 con 0x7fad540c6080 2024-04-25T00:02:35.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.120+0000 7fad58bef640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fad54152060 con 0x7fad5414f400 2024-04-25T00:02:35.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.120+0000 7fad51d9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fad540c6080 0x7fad5406c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:35.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.120+0000 7fad51d9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fad540c6080 0x7fad5406c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45396/0 (socket says 172.21.15.116:45396) 2024-04-25T00:02:35.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.120+0000 7fad52d9d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fad5406c760 0x7fad5406e630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:56778/0 (socket says 172.21.15.116:56778) 2024-04-25T00:02:35.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.120+0000 7fad51d9b640 1 -- 172.21.15.116:0/1954504299 learned_addr learned my addr 172.21.15.116:0/1954504299 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:02:35.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.120+0000 7fad5259c640 1 --2- 172.21.15.116:0/1954504299 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fad5414f400 0x7fad540c2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:35.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.120+0000 7fad51d9b640 1 -- 172.21.15.116:0/1954504299 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fad5414f400 msgr2=0x7fad540c2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:02:35.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.120+0000 7fad51d9b640 1 --2- 172.21.15.116:0/1954504299 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fad5414f400 0x7fad540c2170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:35.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.120+0000 7fad51d9b640 1 -- 172.21.15.116:0/1954504299 >> v1:172.21.15.116:6789/0 conn(0x7fad5406c760 legacy=0x7fad5406e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:35.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.120+0000 7fad51d9b640 1 -- 172.21.15.116:0/1954504299 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad54152340 con 0x7fad540c6080 2024-04-25T00:02:35.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.120+0000 7fad5159a640 1 -- 172.21.15.116:0/1954504299 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1196053004 0 0) 0x7fad5414f7e0 con 0x7fad5406c760 2024-04-25T00:02:35.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.120+0000 7fad51d9b640 1 --2- 172.21.15.116:0/1954504299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad540c6080 0x7fad5406c220 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fad48009870 tx=0x7fad4801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab6678557b354638 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:02:35.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.120+0000 7fad5159a640 1 -- 172.21.15.116:0/1954504299 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad480b0070 con 0x7fad540c6080 2024-04-25T00:02:35.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.120+0000 7fad5159a640 1 -- 172.21.15.116:0/1954504299 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad480ab600 con 0x7fad540c6080 2024-04-25T00:02:35.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.120+0000 7fad5159a640 1 -- 172.21.15.116:0/1954504299 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad480ab9d0 con 0x7fad540c6080 2024-04-25T00:02:35.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.124+0000 7fad58bef640 1 -- 172.21.15.116:0/1954504299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad540c6080 msgr2=0x7fad5406c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:35.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.124+0000 7fad58bef640 1 --2- 172.21.15.116:0/1954504299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad540c6080 0x7fad5406c220 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fad48009870 tx=0x7fad4801c4d0 comp rx=0 tx=0).stop 2024-04-25T00:02:35.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.124+0000 7fad58bef640 1 -- 172.21.15.116:0/1954504299 shutdown_connections 2024-04-25T00:02:35.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.124+0000 7fad58bef640 1 --2- 172.21.15.116:0/1954504299 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad540c6080 0x7fad5406c220 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:35.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.124+0000 7fad58bef640 1 --2- 172.21.15.116:0/1954504299 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fad5414f400 0x7fad540c2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:35.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.124+0000 7fad58bef640 1 -- 172.21.15.116:0/1954504299 >> 172.21.15.116:0/1954504299 conn(0x7fad540c4270 msgr2=0x7fad540c4680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:02:35.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.124+0000 7fad58bef640 1 -- 172.21.15.116:0/1954504299 shutdown_connections 2024-04-25T00:02:35.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.124+0000 7fad58bef640 1 -- 172.21.15.116:0/1954504299 wait complete. 2024-04-25T00:02:35.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.124+0000 7fad58bef640 1 Processor -- start 2024-04-25T00:02:35.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.124+0000 7fad58bef640 1 -- start start 2024-04-25T00:02:35.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.124+0000 7fad58bef640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fad5406c760 0x7fad540a8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:35.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.124+0000 7fad58bef640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad540c6080 0x7fad5413f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:35.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.124+0000 7fad58bef640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fad5414f400 0x7fad5415a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:35.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.124+0000 7fad58bef640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fad5406f2d0 con 0x7fad540c6080 2024-04-25T00:02:35.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.124+0000 7fad58bef640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fad5406f150 con 0x7fad5414f400 2024-04-25T00:02:35.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.124+0000 7fad58bef640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fad54160940 con 0x7fad5406c760 2024-04-25T00:02:35.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.124+0000 7fad51d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad540c6080 0x7fad5413f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:35.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.124+0000 7fad51d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad540c6080 0x7fad5413f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45402/0 (socket says 172.21.15.116:45402) 2024-04-25T00:02:35.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.124+0000 7fad51d9b640 1 -- 172.21.15.116:0/2814613614 learned_addr learned my addr 172.21.15.116:0/2814613614 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:02:35.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.124+0000 7fad52d9d640 1 --2- 172.21.15.116:0/2814613614 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fad5414f400 0x7fad5415a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:35.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.124+0000 7fad5259c640 1 --2- 172.21.15.116:0/2814613614 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fad5406c760 0x7fad540a8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:35.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.124+0000 7fad51d9b640 1 -- 172.21.15.116:0/2814613614 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fad5406c760 msgr2=0x7fad540a8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:35.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.124+0000 7fad51d9b640 1 --2- 172.21.15.116:0/2814613614 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fad5406c760 0x7fad540a8ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:35.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.124+0000 7fad51d9b640 1 -- 172.21.15.116:0/2814613614 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fad5414f400 msgr2=0x7fad5415a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:35.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.124+0000 7fad51d9b640 1 --2- 172.21.15.116:0/2814613614 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fad5414f400 0x7fad5415a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:35.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.124+0000 7fad51d9b640 1 -- 172.21.15.116:0/2814613614 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad54068c70 con 0x7fad540c6080 2024-04-25T00:02:35.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.124+0000 7fad5259c640 1 --2- 172.21.15.116:0/2814613614 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fad5406c760 0x7fad540a8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:02:35.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.124+0000 7fad51d9b640 1 --2- 172.21.15.116:0/2814613614 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad540c6080 0x7fad5413f450 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fad48009870 tx=0x7fad4801ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:02:35.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.124+0000 7fad42ffd640 1 -- 172.21.15.116:0/2814613614 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad480b0040 con 0x7fad540c6080 2024-04-25T00:02:35.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.124+0000 7fad42ffd640 1 -- 172.21.15.116:0/2814613614 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad48004030 con 0x7fad540c6080 2024-04-25T00:02:35.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.124+0000 7fad58bef640 1 -- 172.21.15.116:0/2814613614 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad5414ff00 con 0x7fad540c6080 2024-04-25T00:02:35.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.124+0000 7fad42ffd640 1 -- 172.21.15.116:0/2814613614 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad480043e0 con 0x7fad540c6080 2024-04-25T00:02:35.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.124+0000 7fad58bef640 1 -- 172.21.15.116:0/2814613614 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad54150fa0 con 0x7fad540c6080 2024-04-25T00:02:35.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.124+0000 7fad58bef640 1 -- 172.21.15.116:0/2814613614 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad54068a60 con 0x7fad540c6080 2024-04-25T00:02:35.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.128+0000 7fad42ffd640 1 -- 172.21.15.116:0/2814613614 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fad48003620 con 0x7fad540c6080 2024-04-25T00:02:35.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.128+0000 7fad42ffd640 1 --2- 172.21.15.116:0/2814613614 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fad340678c0 0x7fad34069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:35.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.128+0000 7fad42ffd640 1 -- 172.21.15.116:0/2814613614 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6666+0+0 (secure 0 0 0) 0x7fad48120de0 con 0x7fad540c6080 2024-04-25T00:02:35.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.128+0000 7fad5259c640 1 --2- 172.21.15.116:0/2814613614 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fad340678c0 0x7fad34069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:35.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.128+0000 7fad5259c640 1 --2- 172.21.15.116:0/2814613614 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fad340678c0 0x7fad34069d80 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fad4400b9f0 tx=0x7fad44008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:02:35.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.132+0000 7fad42ffd640 1 -- 172.21.15.116:0/2814613614 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fad5414ff00 con 0x7fad540c6080 2024-04-25T00:02:35.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.328+0000 7fad58bef640 1 -- 172.21.15.116:0/2814613614 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fad540a90e0 con 0x7fad540c6080 2024-04-25T00:02:35.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.328+0000 7fad42ffd640 1 -- 172.21.15.116:0/2814613614 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6492 (secure 0 0 0) 0x7fad48016150 con 0x7fad540c6080 2024-04-25T00:02:35.333 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:02:35.333 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6834/1459177921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1459177921},{"type":"v1","addr":"172.21.15.116:6834","nonce":1459177921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:01:08.748957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6837/1762356055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1762356055},{"type":"v1","addr":"172.21.15.174:6837","nonce":1762356055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270]},"id":1},{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:01:22.888641+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6835/174781672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":174781672},{"type":"v1","addr":"172.21.15.116:6835","nonce":174781672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4245,"qdb_cluster":[4245]},"id":2}]} 2024-04-25T00:02:35.334 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 14 2024-04-25T00:02:35.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.332+0000 7fad58bef640 1 -- 172.21.15.116:0/2814613614 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fad340678c0 msgr2=0x7fad34069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:35.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.332+0000 7fad58bef640 1 --2- 172.21.15.116:0/2814613614 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fad340678c0 0x7fad34069d80 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fad4400b9f0 tx=0x7fad44008040 comp rx=0 tx=0).stop 2024-04-25T00:02:35.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.332+0000 7fad58bef640 1 -- 172.21.15.116:0/2814613614 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad540c6080 msgr2=0x7fad5413f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:35.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.332+0000 7fad58bef640 1 --2- 172.21.15.116:0/2814613614 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad540c6080 0x7fad5413f450 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fad48009870 tx=0x7fad4801ad10 comp rx=0 tx=0).stop 2024-04-25T00:02:35.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.332+0000 7fad58bef640 1 -- 172.21.15.116:0/2814613614 shutdown_connections 2024-04-25T00:02:35.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.332+0000 7fad58bef640 1 --2- 172.21.15.116:0/2814613614 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fad340678c0 0x7fad34069d80 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:35.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.332+0000 7fad58bef640 1 --2- 172.21.15.116:0/2814613614 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fad5414f400 0x7fad5415a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:35.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.332+0000 7fad58bef640 1 --2- 172.21.15.116:0/2814613614 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad540c6080 0x7fad5413f450 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:35.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.332+0000 7fad58bef640 1 --2- 172.21.15.116:0/2814613614 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fad5406c760 0x7fad540a8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:35.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.332+0000 7fad58bef640 1 -- 172.21.15.116:0/2814613614 >> 172.21.15.116:0/2814613614 conn(0x7fad540c4270 msgr2=0x7fad5414a230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:02:35.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.332+0000 7fad58bef640 1 -- 172.21.15.116:0/2814613614 shutdown_connections 2024-04-25T00:02:35.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:35.332+0000 7fad58bef640 1 -- 172.21.15.116:0/2814613614 wait complete. 2024-04-25T00:02:35.352 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:02:37.353 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:02:37.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.480+0000 7f99e4319640 1 Processor -- start 2024-04-25T00:02:37.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.480+0000 7f99e4319640 1 -- start start 2024-04-25T00:02:37.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.484+0000 7f99e4319640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f99dc0c5280 0x7f99dc150840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:37.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.484+0000 7f99e4319640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99dc0c2c40 con 0x7f99dc072810 2024-04-25T00:02:37.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.484+0000 7f99e4319640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99dc157660 con 0x7f99dc072c90 2024-04-25T00:02:37.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.484+0000 7f99e4319640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f99dc157c20 con 0x7f99dc0c5280 2024-04-25T00:02:37.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.484+0000 7f99e28b6640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f99dc072810 0x7f99dc153760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:56788/0 (socket says 172.21.15.116:56788) 2024-04-25T00:02:37.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.484+0000 7f99e28b6640 1 -- 172.21.15.116:0/2682091220 learned_addr learned my addr 172.21.15.116:0/2682091220 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:02:37.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.484+0000 7f99e20b5640 1 --2- 172.21.15.116:0/2682091220 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f99dc0c5280 0x7f99dc150840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:37.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.484+0000 7f99e10b3640 1 -- 172.21.15.116:0/2682091220 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2064680218 0 0) 0x7f99dc0c2c40 con 0x7f99dc072810 2024-04-25T00:02:37.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.484+0000 7f99e10b3640 1 -- 172.21.15.116:0/2682091220 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f99c8003530 con 0x7f99dc072810 2024-04-25T00:02:37.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.484+0000 7f99e10b3640 1 -- 172.21.15.116:0/2682091220 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1183796138 0 0) 0x7f99c8003530 con 0x7f99dc072810 2024-04-25T00:02:37.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.484+0000 7f99e10b3640 1 -- 172.21.15.116:0/2682091220 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f99dc0c5280 msgr2=0x7f99dc150840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:02:37.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.484+0000 7f99e10b3640 1 --2- 172.21.15.116:0/2682091220 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f99dc0c5280 0x7f99dc150840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:37.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.484+0000 7f99e10b3640 1 -- 172.21.15.116:0/2682091220 >> v1:172.21.15.174:6789/0 conn(0x7f99dc072c90 legacy=0x7f99dc150d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:37.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.484+0000 7f99e10b3640 1 -- 172.21.15.116:0/2682091220 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99dc157940 con 0x7f99dc072810 2024-04-25T00:02:37.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.484+0000 7f99e10b3640 1 -- 172.21.15.116:0/2682091220 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f99d4001a20 con 0x7f99dc072810 2024-04-25T00:02:37.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.484+0000 7f99e10b3640 1 -- 172.21.15.116:0/2682091220 >> v1:172.21.15.116:6789/0 conn(0x7f99dc072810 legacy=0x7f99dc153760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:37.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.484+0000 7f99e10b3640 1 --2- 172.21.15.116:0/2682091220 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99c80039f0 0x7f99c8003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:37.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.484+0000 7f99e10b3640 1 --2- 172.21.15.116:0/2682091220 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f99dc0c5280 0x7f99c8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:37.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.484+0000 7f99e10b3640 1 --2- 172.21.15.116:0/2682091220 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f99c8004d50 0x7f99c80051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:37.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.484+0000 7f99e10b3640 1 -- 172.21.15.116:0/2682091220 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f99dc157c20 con 0x7f99c80039f0 2024-04-25T00:02:37.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.484+0000 7f99e10b3640 1 -- 172.21.15.116:0/2682091220 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f99c800bb00 con 0x7f99c8004d50 2024-04-25T00:02:37.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.484+0000 7f99e10b3640 1 -- 172.21.15.116:0/2682091220 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f99c800bd00 con 0x7f99dc0c5280 2024-04-25T00:02:37.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.484+0000 7f99e20b5640 1 --2- 172.21.15.116:0/2682091220 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99c80039f0 0x7f99c8003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:37.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.484+0000 7f99e18b4640 1 --2- 172.21.15.116:0/2682091220 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f99dc0c5280 0x7f99c8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:37.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.484+0000 7f99e28b6640 1 --2- 172.21.15.116:0/2682091220 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f99c8004d50 0x7f99c80051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:37.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.484+0000 7f99e20b5640 1 -- 172.21.15.116:0/2682091220 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f99dc0c5280 msgr2=0x7f99c8004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:02:37.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.484+0000 7f99e20b5640 1 --2- 172.21.15.116:0/2682091220 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f99dc0c5280 0x7f99c8004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:37.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.484+0000 7f99e20b5640 1 -- 172.21.15.116:0/2682091220 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f99c8004d50 msgr2=0x7f99c80051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:02:37.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.484+0000 7f99e20b5640 1 --2- 172.21.15.116:0/2682091220 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f99c8004d50 0x7f99c80051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:37.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.484+0000 7f99e20b5640 1 -- 172.21.15.116:0/2682091220 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99c8003530 con 0x7f99c80039f0 2024-04-25T00:02:37.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.484+0000 7f99e28b6640 1 --2- 172.21.15.116:0/2682091220 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f99c8004d50 0x7f99c80051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:02:37.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.488+0000 7f99e20b5640 1 --2- 172.21.15.116:0/2682091220 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99c80039f0 0x7f99c8003e10 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f99d8009ae0 tx=0x7f99d801c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=153eb0a338831135 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:02:37.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.488+0000 7f99e10b3640 1 -- 172.21.15.116:0/2682091220 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99d80096b0 con 0x7f99c80039f0 2024-04-25T00:02:37.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.488+0000 7f99e10b3640 1 -- 172.21.15.116:0/2682091220 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99d80b0400 con 0x7f99c80039f0 2024-04-25T00:02:37.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.488+0000 7f99e10b3640 1 -- 172.21.15.116:0/2682091220 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99d80b07f0 con 0x7f99c80039f0 2024-04-25T00:02:37.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.488+0000 7f99e4319640 1 -- 172.21.15.116:0/2682091220 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99c80039f0 msgr2=0x7f99c8003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:37.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.488+0000 7f99e4319640 1 --2- 172.21.15.116:0/2682091220 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99c80039f0 0x7f99c8003e10 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f99d8009ae0 tx=0x7f99d801c9b0 comp rx=0 tx=0).stop 2024-04-25T00:02:37.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.488+0000 7f99e28b6640 1 -- 172.21.15.116:0/2682091220 reap_dead start 2024-04-25T00:02:37.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.488+0000 7f99e4319640 1 -- 172.21.15.116:0/2682091220 shutdown_connections 2024-04-25T00:02:37.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.488+0000 7f99e4319640 1 -- 172.21.15.116:0/2682091220 >> 172.21.15.116:0/2682091220 conn(0x7f99dc06c720 msgr2=0x7f99dc06cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:02:37.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.488+0000 7f99e4319640 1 -- 172.21.15.116:0/2682091220 shutdown_connections 2024-04-25T00:02:37.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.488+0000 7f99e4319640 1 -- 172.21.15.116:0/2682091220 wait complete. 2024-04-25T00:02:37.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.488+0000 7f99e4319640 1 Processor -- start 2024-04-25T00:02:37.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.488+0000 7f99e4319640 1 -- start start 2024-04-25T00:02:37.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.488+0000 7f99e4319640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99c80039f0 0x7f99dc1ed440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:37.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.488+0000 7f99e4319640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f99dc0c6af0 0x7f99dc0c70c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:37.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.488+0000 7f99e4319640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f99dc06f120 0x7f99dc154e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:37.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.488+0000 7f99e4319640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f99dc154200 con 0x7f99c80039f0 2024-04-25T00:02:37.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.488+0000 7f99e4319640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f99dc154080 con 0x7f99dc06f120 2024-04-25T00:02:37.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.488+0000 7f99e4319640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f99dc154380 con 0x7f99dc0c6af0 2024-04-25T00:02:37.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.488+0000 7f99e20b5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99c80039f0 0x7f99dc1ed440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:37.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.488+0000 7f99e20b5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99c80039f0 0x7f99dc1ed440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45410/0 (socket says 172.21.15.116:45410) 2024-04-25T00:02:37.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.488+0000 7f99e20b5640 1 -- 172.21.15.116:0/2809108896 learned_addr learned my addr 172.21.15.116:0/2809108896 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:02:37.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.488+0000 7f99e28b6640 1 --2- 172.21.15.116:0/2809108896 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f99dc06f120 0x7f99dc154e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:37.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.488+0000 7f99e18b4640 1 --2- 172.21.15.116:0/2809108896 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f99dc0c6af0 0x7f99dc0c70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:37.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.488+0000 7f99e20b5640 1 -- 172.21.15.116:0/2809108896 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f99dc0c6af0 msgr2=0x7f99dc0c70c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:37.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.488+0000 7f99e20b5640 1 --2- 172.21.15.116:0/2809108896 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f99dc0c6af0 0x7f99dc0c70c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:37.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.488+0000 7f99e20b5640 1 -- 172.21.15.116:0/2809108896 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f99dc06f120 msgr2=0x7f99dc154e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:37.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.488+0000 7f99e20b5640 1 --2- 172.21.15.116:0/2809108896 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f99dc06f120 0x7f99dc154e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:37.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.488+0000 7f99e20b5640 1 -- 172.21.15.116:0/2809108896 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99dc0c77f0 con 0x7f99c80039f0 2024-04-25T00:02:37.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.488+0000 7f99e20b5640 1 --2- 172.21.15.116:0/2809108896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99c80039f0 0x7f99dc1ed440 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f99d8009ae0 tx=0x7f99d8019750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:02:37.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.488+0000 7f99d2ffd640 1 -- 172.21.15.116:0/2809108896 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99d80bb5f0 con 0x7f99c80039f0 2024-04-25T00:02:37.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.492+0000 7f99d2ffd640 1 -- 172.21.15.116:0/2809108896 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99d80bb790 con 0x7f99c80039f0 2024-04-25T00:02:37.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.492+0000 7f99e4319640 1 -- 172.21.15.116:0/2809108896 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99dc0c5d30 con 0x7f99c80039f0 2024-04-25T00:02:37.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.492+0000 7f99d2ffd640 1 -- 172.21.15.116:0/2809108896 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99d80bbb60 con 0x7f99c80039f0 2024-04-25T00:02:37.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.492+0000 7f99e4319640 1 -- 172.21.15.116:0/2809108896 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99dc0c62f0 con 0x7f99c80039f0 2024-04-25T00:02:37.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.492+0000 7f99e4319640 1 -- 172.21.15.116:0/2809108896 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99dc068a60 con 0x7f99c80039f0 2024-04-25T00:02:37.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.492+0000 7f99d2ffd640 1 -- 172.21.15.116:0/2809108896 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f99d80c2060 con 0x7f99c80039f0 2024-04-25T00:02:37.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.492+0000 7f99d2ffd640 1 --2- 172.21.15.116:0/2809108896 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f99c00678f0 0x7f99c0069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:37.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.492+0000 7f99d2ffd640 1 -- 172.21.15.116:0/2809108896 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6666+0+0 (secure 0 0 0) 0x7f99d8120710 con 0x7f99c80039f0 2024-04-25T00:02:37.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.492+0000 7f99e18b4640 1 --2- 172.21.15.116:0/2809108896 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f99c00678f0 0x7f99c0069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:37.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.492+0000 7f99e18b4640 1 --2- 172.21.15.116:0/2809108896 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f99c00678f0 0x7f99c0069db0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f99dc14f410 tx=0x7f99cc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:02:37.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.496+0000 7f99d2ffd640 1 -- 172.21.15.116:0/2809108896 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f99d80b6070 con 0x7f99c80039f0 2024-04-25T00:02:37.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.724+0000 7f99e4319640 1 -- 172.21.15.116:0/2809108896 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f99dc0c8f70 con 0x7f99c80039f0 2024-04-25T00:02:37.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.724+0000 7f99d2ffd640 1 -- 172.21.15.116:0/2809108896 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6492 (secure 0 0 0) 0x7f99d80ac030 con 0x7f99c80039f0 2024-04-25T00:02:37.729 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:02:37.729 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6834/1459177921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1459177921},{"type":"v1","addr":"172.21.15.116:6834","nonce":1459177921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:01:08.748957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6837/1762356055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1762356055},{"type":"v1","addr":"172.21.15.174:6837","nonce":1762356055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270]},"id":1},{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:01:22.888641+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6835/174781672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":174781672},{"type":"v1","addr":"172.21.15.116:6835","nonce":174781672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4245,"qdb_cluster":[4245]},"id":2}]} 2024-04-25T00:02:37.729 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 14 2024-04-25T00:02:37.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.728+0000 7f99e4319640 1 -- 172.21.15.116:0/2809108896 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f99c00678f0 msgr2=0x7f99c0069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:37.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.728+0000 7f99e4319640 1 --2- 172.21.15.116:0/2809108896 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f99c00678f0 0x7f99c0069db0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f99dc14f410 tx=0x7f99cc009040 comp rx=0 tx=0).stop 2024-04-25T00:02:37.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.728+0000 7f99e4319640 1 -- 172.21.15.116:0/2809108896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99c80039f0 msgr2=0x7f99dc1ed440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:37.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.728+0000 7f99e4319640 1 --2- 172.21.15.116:0/2809108896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99c80039f0 0x7f99dc1ed440 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f99d8009ae0 tx=0x7f99d8019750 comp rx=0 tx=0).stop 2024-04-25T00:02:37.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.728+0000 7f99e4319640 1 -- 172.21.15.116:0/2809108896 shutdown_connections 2024-04-25T00:02:37.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.728+0000 7f99e4319640 1 --2- 172.21.15.116:0/2809108896 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f99c00678f0 0x7f99c0069db0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:37.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.728+0000 7f99e4319640 1 --2- 172.21.15.116:0/2809108896 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f99dc06f120 0x7f99dc154e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:37.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.728+0000 7f99e4319640 1 --2- 172.21.15.116:0/2809108896 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f99dc0c6af0 0x7f99dc0c70c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:37.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.728+0000 7f99e4319640 1 --2- 172.21.15.116:0/2809108896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f99c80039f0 0x7f99dc1ed440 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:37.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.728+0000 7f99e4319640 1 -- 172.21.15.116:0/2809108896 >> 172.21.15.116:0/2809108896 conn(0x7f99dc06c720 msgr2=0x7f99dc14fd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:02:37.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.728+0000 7f99e4319640 1 -- 172.21.15.116:0/2809108896 shutdown_connections 2024-04-25T00:02:37.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:37.728+0000 7f99e4319640 1 -- 172.21.15.116:0/2809108896 wait complete. 2024-04-25T00:02:37.748 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:02:39.749 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:02:39.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.884+0000 7f196263d640 1 Processor -- start 2024-04-25T00:02:39.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.884+0000 7f196263d640 1 -- start start 2024-04-25T00:02:39.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.884+0000 7f196263d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f195c14fe10 0x7f195c06c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:39.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.884+0000 7f196263d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f195c071590 0x7f195c156180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:39.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.884+0000 7f196263d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f195c0a6cc0 con 0x7f195c06c760 2024-04-25T00:02:39.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.884+0000 7f196263d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f195c074790 con 0x7f195c14fe10 2024-04-25T00:02:39.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.884+0000 7f196263d640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f195c155cf0 con 0x7f195c071590 2024-04-25T00:02:39.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.884+0000 7f1960bda640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f195c06c760 0x7f195c1525c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:56798/0 (socket says 172.21.15.116:56798) 2024-04-25T00:02:39.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.884+0000 7f195bfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f195c14fe10 0x7f195c06c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:39.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.884+0000 7f1960bda640 1 -- 172.21.15.116:0/1977874909 learned_addr learned my addr 172.21.15.116:0/1977874909 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:02:39.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.884+0000 7f195b7fe640 1 --2- 172.21.15.116:0/1977874909 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f195c071590 0x7f195c156180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:39.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.884+0000 7f195affd640 1 -- 172.21.15.116:0/1977874909 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3356778132 0 0) 0x7f195c0a6cc0 con 0x7f195c06c760 2024-04-25T00:02:39.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.884+0000 7f195affd640 1 -- 172.21.15.116:0/1977874909 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1944003530 con 0x7f195c06c760 2024-04-25T00:02:39.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.884+0000 7f195bfff640 1 -- 172.21.15.116:0/1977874909 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f195c071590 msgr2=0x7f195c156180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:02:39.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.884+0000 7f195bfff640 1 --2- 172.21.15.116:0/1977874909 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f195c071590 0x7f195c156180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:39.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.884+0000 7f195bfff640 1 -- 172.21.15.116:0/1977874909 >> v1:172.21.15.116:6789/0 conn(0x7f195c06c760 legacy=0x7f195c1525c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:39.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.884+0000 7f195bfff640 1 -- 172.21.15.116:0/1977874909 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f195c0c6bb0 con 0x7f195c14fe10 2024-04-25T00:02:39.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.888+0000 7f195bfff640 1 --2- 172.21.15.116:0/1977874909 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f195c14fe10 0x7f195c06c220 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f194c0097c0 tx=0x7f194c01c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4bfa430b06e044b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:02:39.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.888+0000 7f195affd640 1 -- 172.21.15.116:0/1977874909 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f194c0b0070 con 0x7f195c14fe10 2024-04-25T00:02:39.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.888+0000 7f195affd640 1 -- 172.21.15.116:0/1977874909 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f194c0ab600 con 0x7f195c14fe10 2024-04-25T00:02:39.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.888+0000 7f195affd640 1 -- 172.21.15.116:0/1977874909 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f194c0b0070 con 0x7f195c14fe10 2024-04-25T00:02:39.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.888+0000 7f196263d640 1 -- 172.21.15.116:0/1977874909 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f195c14fe10 msgr2=0x7f195c06c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:39.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.888+0000 7f196263d640 1 --2- 172.21.15.116:0/1977874909 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f195c14fe10 0x7f195c06c220 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f194c0097c0 tx=0x7f194c01c380 comp rx=0 tx=0).stop 2024-04-25T00:02:39.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.888+0000 7f196263d640 1 -- 172.21.15.116:0/1977874909 shutdown_connections 2024-04-25T00:02:39.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.888+0000 7f196263d640 1 --2- 172.21.15.116:0/1977874909 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f195c071590 0x7f195c156180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:39.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.888+0000 7f196263d640 1 --2- 172.21.15.116:0/1977874909 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f195c14fe10 0x7f195c06c220 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:39.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.888+0000 7f196263d640 1 -- 172.21.15.116:0/1977874909 >> 172.21.15.116:0/1977874909 conn(0x7f195c0c42e0 msgr2=0x7f195c0c46f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:02:39.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.888+0000 7f196263d640 1 -- 172.21.15.116:0/1977874909 shutdown_connections 2024-04-25T00:02:39.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.888+0000 7f196263d640 1 -- 172.21.15.116:0/1977874909 wait complete. 2024-04-25T00:02:39.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.888+0000 7f196263d640 1 Processor -- start 2024-04-25T00:02:39.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.888+0000 7f196263d640 1 -- start start 2024-04-25T00:02:39.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.892+0000 7f196263d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f195c06c760 0x7f195c153b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:39.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.892+0000 7f196263d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f195c071590 0x7f195c0a7a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:39.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.892+0000 7f196263d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f195c14fe10 0x7f195c0a7f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:39.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.892+0000 7f196263d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f195c074d20 con 0x7f195c071590 2024-04-25T00:02:39.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.892+0000 7f196263d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f195c074ba0 con 0x7f195c14fe10 2024-04-25T00:02:39.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.892+0000 7f196263d640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f195c15deb0 con 0x7f195c06c760 2024-04-25T00:02:39.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.892+0000 7f195b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f195c071590 0x7f195c0a7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:39.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.892+0000 7f195b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f195c071590 0x7f195c0a7a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45424/0 (socket says 172.21.15.116:45424) 2024-04-25T00:02:39.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.892+0000 7f195b7fe640 1 -- 172.21.15.116:0/687976032 learned_addr learned my addr 172.21.15.116:0/687976032 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:02:39.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.892+0000 7f1960bda640 1 --2- 172.21.15.116:0/687976032 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f195c14fe10 0x7f195c0a7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:39.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.892+0000 7f195bfff640 1 --2- 172.21.15.116:0/687976032 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f195c06c760 0x7f195c153b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:39.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.892+0000 7f195b7fe640 1 -- 172.21.15.116:0/687976032 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f195c06c760 msgr2=0x7f195c153b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:39.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.892+0000 7f195b7fe640 1 --2- 172.21.15.116:0/687976032 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f195c06c760 0x7f195c153b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:39.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.892+0000 7f195b7fe640 1 -- 172.21.15.116:0/687976032 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f195c14fe10 msgr2=0x7f195c0a7f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:39.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.892+0000 7f195b7fe640 1 --2- 172.21.15.116:0/687976032 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f195c14fe10 0x7f195c0a7f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:39.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.892+0000 7f195b7fe640 1 -- 172.21.15.116:0/687976032 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f195c147850 con 0x7f195c071590 2024-04-25T00:02:39.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.892+0000 7f195b7fe640 1 --2- 172.21.15.116:0/687976032 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f195c071590 0x7f195c0a7a30 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f195000d800 tx=0x7f195000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:02:39.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.892+0000 7f1958ff9640 1 -- 172.21.15.116:0/687976032 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f195000b6c0 con 0x7f195c071590 2024-04-25T00:02:39.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.892+0000 7f1958ff9640 1 -- 172.21.15.116:0/687976032 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1950019070 con 0x7f195c071590 2024-04-25T00:02:39.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.892+0000 7f1958ff9640 1 -- 172.21.15.116:0/687976032 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f195000baa0 con 0x7f195c071590 2024-04-25T00:02:39.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.892+0000 7f196263d640 1 -- 172.21.15.116:0/687976032 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f195c0abc90 con 0x7f195c071590 2024-04-25T00:02:39.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.892+0000 7f196263d640 1 -- 172.21.15.116:0/687976032 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f195c154620 con 0x7f195c071590 2024-04-25T00:02:39.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.892+0000 7f196263d640 1 -- 172.21.15.116:0/687976032 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f195c068a60 con 0x7f195c071590 2024-04-25T00:02:39.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.896+0000 7f1958ff9640 1 -- 172.21.15.116:0/687976032 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f195000bc40 con 0x7f195c071590 2024-04-25T00:02:39.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.896+0000 7f1958ff9640 1 --2- 172.21.15.116:0/687976032 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f19480678f0 0x7f1948069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:39.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.896+0000 7f1958ff9640 1 -- 172.21.15.116:0/687976032 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6666+0+0 (secure 0 0 0) 0x7f1950086ed0 con 0x7f195c071590 2024-04-25T00:02:39.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.896+0000 7f195bfff640 1 --2- 172.21.15.116:0/687976032 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f19480678f0 0x7f1948069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:39.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.896+0000 7f195bfff640 1 --2- 172.21.15.116:0/687976032 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f19480678f0 0x7f1948069db0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f194c0ad000 tx=0x7f194c0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:02:39.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:39.896+0000 7f1958ff9640 1 -- 172.21.15.116:0/687976032 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f19500577c0 con 0x7f195c071590 2024-04-25T00:02:40.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:40.180+0000 7f196263d640 1 -- 172.21.15.116:0/687976032 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f195c073300 con 0x7f195c071590 2024-04-25T00:02:40.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:40.180+0000 7f1958ff9640 1 -- 172.21.15.116:0/687976032 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) ==== 94+0+6492 (secure 0 0 0) 0x7f19500570c0 con 0x7f195c071590 2024-04-25T00:02:40.185 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:02:40.185 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4242,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6834/1459177921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1459177921},{"type":"v1","addr":"172.21.15.116:6834","nonce":1459177921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:01:08.748957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6837/1762356055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1762356055},{"type":"v1","addr":"172.21.15.174:6837","nonce":1762356055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270]},"id":1},{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:01:22.888641+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"a","rank":0,"incarnation":10,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6835/174781672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":174781672},{"type":"v1","addr":"172.21.15.116:6835","nonce":174781672}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4245,"qdb_cluster":[4245]},"id":2}]} 2024-04-25T00:02:40.185 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 14 2024-04-25T00:02:40.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:40.184+0000 7f196263d640 1 -- 172.21.15.116:0/687976032 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f19480678f0 msgr2=0x7f1948069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:40.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:40.184+0000 7f196263d640 1 --2- 172.21.15.116:0/687976032 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f19480678f0 0x7f1948069db0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f194c0ad000 tx=0x7f194c0a6000 comp rx=0 tx=0).stop 2024-04-25T00:02:40.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:40.184+0000 7f196263d640 1 -- 172.21.15.116:0/687976032 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f195c071590 msgr2=0x7f195c0a7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:40.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:40.184+0000 7f196263d640 1 --2- 172.21.15.116:0/687976032 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f195c071590 0x7f195c0a7a30 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f195000d800 tx=0x7f195000dcc0 comp rx=0 tx=0).stop 2024-04-25T00:02:40.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:40.184+0000 7f196263d640 1 -- 172.21.15.116:0/687976032 shutdown_connections 2024-04-25T00:02:40.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:40.184+0000 7f196263d640 1 --2- 172.21.15.116:0/687976032 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f19480678f0 0x7f1948069db0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:40.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:40.184+0000 7f196263d640 1 --2- 172.21.15.116:0/687976032 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f195c14fe10 0x7f195c0a7f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:40.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:40.184+0000 7f196263d640 1 --2- 172.21.15.116:0/687976032 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f195c071590 0x7f195c0a7a30 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:40.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:40.184+0000 7f196263d640 1 --2- 172.21.15.116:0/687976032 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f195c06c760 0x7f195c153b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:40.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:40.184+0000 7f196263d640 1 -- 172.21.15.116:0/687976032 >> 172.21.15.116:0/687976032 conn(0x7f195c0c42e0 msgr2=0x7f195c0c2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:02:40.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:40.184+0000 7f196263d640 1 -- 172.21.15.116:0/687976032 shutdown_connections 2024-04-25T00:02:40.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:40.184+0000 7f196263d640 1 -- 172.21.15.116:0/687976032 wait complete. 2024-04-25T00:02:40.209 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:02:42.210 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:02:42.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.320+0000 7f528a9ae640 1 Processor -- start 2024-04-25T00:02:42.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.320+0000 7f528a9ae640 1 -- start start 2024-04-25T00:02:42.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.320+0000 7f528a9ae640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f528406c190 0x7f52841538a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:42.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.320+0000 7f528a9ae640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f528406c6e0 0x7f5284153de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:42.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.320+0000 7f528a9ae640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52840a7e80 con 0x7f5284150dd0 2024-04-25T00:02:42.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.320+0000 7f528a9ae640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f52840a8700 con 0x7f528406c190 2024-04-25T00:02:42.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.320+0000 7f528a9ae640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f52840a8140 con 0x7f528406c6e0 2024-04-25T00:02:42.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.320+0000 7f527bfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f528406c190 0x7f52841538a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:42.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.320+0000 7f5288f4b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f528406c6e0 0x7f5284153de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:42.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.320+0000 7f5283fff640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f5284150dd0 0x7f528406f5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:41854/0 (socket says 172.21.15.116:41854) 2024-04-25T00:02:42.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.320+0000 7f5283fff640 1 -- 172.21.15.116:0/1905049104 learned_addr learned my addr 172.21.15.116:0/1905049104 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:02:42.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.320+0000 7f527bfff640 1 -- 172.21.15.116:0/1905049104 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f528406c6e0 msgr2=0x7f5284153de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:02:42.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.320+0000 7f527bfff640 1 --2- 172.21.15.116:0/1905049104 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f528406c6e0 0x7f5284153de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:42.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.320+0000 7f527bfff640 1 -- 172.21.15.116:0/1905049104 >> v1:172.21.15.174:6789/0 conn(0x7f5284150dd0 legacy=0x7f528406f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:42.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.320+0000 7f52837fe640 1 -- 172.21.15.116:0/1905049104 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4064777641 0 0) 0x7f52840a7e80 con 0x7f5284150dd0 2024-04-25T00:02:42.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.320+0000 7f527bfff640 1 -- 172.21.15.116:0/1905049104 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52840a8420 con 0x7f528406c190 2024-04-25T00:02:42.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.320+0000 7f5288f4b640 1 --2- 172.21.15.116:0/1905049104 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f528406c6e0 0x7f5284153de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:02:42.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.324+0000 7f527bfff640 1 --2- 172.21.15.116:0/1905049104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f528406c190 0x7f52841538a0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f526c0097c0 tx=0x7f526c01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e74a075dbd31497 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:02:42.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.324+0000 7f52837fe640 1 -- 172.21.15.116:0/1905049104 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f526c0b1070 con 0x7f528406c190 2024-04-25T00:02:42.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.324+0000 7f52837fe640 1 -- 172.21.15.116:0/1905049104 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f526c0ac410 con 0x7f528406c190 2024-04-25T00:02:42.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.324+0000 7f52837fe640 1 -- 172.21.15.116:0/1905049104 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f526c0ac7e0 con 0x7f528406c190 2024-04-25T00:02:42.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.324+0000 7f528a9ae640 1 -- 172.21.15.116:0/1905049104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f528406c190 msgr2=0x7f52841538a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:42.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.324+0000 7f528a9ae640 1 --2- 172.21.15.116:0/1905049104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f528406c190 0x7f52841538a0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f526c0097c0 tx=0x7f526c01c4a0 comp rx=0 tx=0).stop 2024-04-25T00:02:42.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.324+0000 7f528a9ae640 1 -- 172.21.15.116:0/1905049104 shutdown_connections 2024-04-25T00:02:42.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.324+0000 7f528a9ae640 1 --2- 172.21.15.116:0/1905049104 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f528406c6e0 0x7f5284153de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:42.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.324+0000 7f528a9ae640 1 --2- 172.21.15.116:0/1905049104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f528406c190 0x7f52841538a0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:42.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.324+0000 7f528a9ae640 1 -- 172.21.15.116:0/1905049104 >> 172.21.15.116:0/1905049104 conn(0x7f528406b210 msgr2=0x7f528406b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:02:42.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.324+0000 7f528a9ae640 1 -- 172.21.15.116:0/1905049104 shutdown_connections 2024-04-25T00:02:42.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.324+0000 7f528a9ae640 1 -- 172.21.15.116:0/1905049104 wait complete. 2024-04-25T00:02:42.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.324+0000 7f528a9ae640 1 Processor -- start 2024-04-25T00:02:42.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.324+0000 7f528a9ae640 1 -- start start 2024-04-25T00:02:42.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.328+0000 7f528a9ae640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f528406c190 0x7f52840ab150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:42.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.328+0000 7f528a9ae640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f528406c6e0 0x7f528414e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:42.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.328+0000 7f528a9ae640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5284150dd0 0x7f52840acb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:42.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.328+0000 7f528a9ae640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f528406fcf0 con 0x7f528406c190 2024-04-25T00:02:42.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.328+0000 7f528a9ae640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f52840735a0 con 0x7f5284150dd0 2024-04-25T00:02:42.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.328+0000 7f528a9ae640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f528406fe70 con 0x7f528406c6e0 2024-04-25T00:02:42.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.328+0000 7f5283fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f528406c190 0x7f52840ab150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:42.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.328+0000 7f5283fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f528406c190 0x7f52840ab150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45448/0 (socket says 172.21.15.116:45448) 2024-04-25T00:02:42.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.328+0000 7f5283fff640 1 -- 172.21.15.116:0/3251437816 learned_addr learned my addr 172.21.15.116:0/3251437816 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:02:42.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.328+0000 7f527bfff640 1 --2- 172.21.15.116:0/3251437816 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f528406c6e0 0x7f528414e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:42.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.328+0000 7f5288f4b640 1 --2- 172.21.15.116:0/3251437816 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5284150dd0 0x7f52840acb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:42.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.328+0000 7f5283fff640 1 -- 172.21.15.116:0/3251437816 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f528406c6e0 msgr2=0x7f528414e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:42.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.328+0000 7f5283fff640 1 --2- 172.21.15.116:0/3251437816 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f528406c6e0 0x7f528414e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:42.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.328+0000 7f5283fff640 1 -- 172.21.15.116:0/3251437816 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5284150dd0 msgr2=0x7f52840acb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:42.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.328+0000 7f5283fff640 1 --2- 172.21.15.116:0/3251437816 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5284150dd0 0x7f52840acb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:42.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.328+0000 7f5283fff640 1 -- 172.21.15.116:0/3251437816 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52841eefc0 con 0x7f528406c190 2024-04-25T00:02:42.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.328+0000 7f5283fff640 1 --2- 172.21.15.116:0/3251437816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f528406c190 0x7f52840ab150 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f527400c410 tx=0x7f527400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:02:42.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.328+0000 7f52817fa640 1 -- 172.21.15.116:0/3251437816 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52740074e0 con 0x7f528406c190 2024-04-25T00:02:42.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.328+0000 7f52817fa640 1 -- 172.21.15.116:0/3251437816 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52740047f0 con 0x7f528406c190 2024-04-25T00:02:42.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.328+0000 7f528a9ae640 1 -- 172.21.15.116:0/3251437816 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5284150090 con 0x7f528406c190 2024-04-25T00:02:42.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.328+0000 7f52817fa640 1 -- 172.21.15.116:0/3251437816 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5274007680 con 0x7f528406c190 2024-04-25T00:02:42.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.328+0000 7f528a9ae640 1 -- 172.21.15.116:0/3251437816 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52841eecb0 con 0x7f528406c190 2024-04-25T00:02:42.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.328+0000 7f528a9ae640 1 -- 172.21.15.116:0/3251437816 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5284068350 con 0x7f528406c190 2024-04-25T00:02:42.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.328+0000 7f52817fa640 1 -- 172.21.15.116:0/3251437816 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5274011070 con 0x7f528406c190 2024-04-25T00:02:42.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.328+0000 7f52817fa640 1 --2- 172.21.15.116:0/3251437816 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f52640678c0 0x7f5264069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:42.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.328+0000 7f52817fa640 1 -- 172.21.15.116:0/3251437816 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6752+0+0 (secure 0 0 0) 0x7f5274085ad0 con 0x7f528406c190 2024-04-25T00:02:42.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.332+0000 7f527bfff640 1 --2- 172.21.15.116:0/3251437816 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f52640678c0 0x7f5264069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:42.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.332+0000 7f527bfff640 1 --2- 172.21.15.116:0/3251437816 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f52640678c0 0x7f5264069d80 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f526c0ae000 tx=0x7f526c0a6040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:02:42.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.340+0000 7f52817fa640 1 -- 172.21.15.116:0/3251437816 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f5274056290 con 0x7f528406c190 2024-04-25T00:02:42.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.612+0000 7f528a9ae640 1 -- 172.21.15.116:0/3251437816 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f528406bf70 con 0x7f528406c190 2024-04-25T00:02:42.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.612+0000 7f52817fa640 1 -- 172.21.15.116:0/3251437816 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5663 (secure 0 0 0) 0x7f5274055b90 con 0x7f528406c190 2024-04-25T00:02:42.615 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:02:42.615 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:01:08.748957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6837/1762356055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1762356055},{"type":"v1","addr":"172.21.15.174:6837","nonce":1762356055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270]},"id":1},{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:02:41.407879+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4242":{"gid":4242,"name":"c","rank":0,"incarnation":15,"state":"up:rejoin","state_seq":28,"addr":"172.21.15.116:6834/1459177921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1459177921},{"type":"v1","addr":"172.21.15.116:6834","nonce":1459177921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:02:42.616 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 17 2024-04-25T00:02:42.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.612+0000 7f528a9ae640 1 -- 172.21.15.116:0/3251437816 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f52640678c0 msgr2=0x7f5264069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:42.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.612+0000 7f528a9ae640 1 --2- 172.21.15.116:0/3251437816 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f52640678c0 0x7f5264069d80 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f526c0ae000 tx=0x7f526c0a6040 comp rx=0 tx=0).stop 2024-04-25T00:02:42.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.612+0000 7f528a9ae640 1 -- 172.21.15.116:0/3251437816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f528406c190 msgr2=0x7f52840ab150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:42.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.612+0000 7f528a9ae640 1 --2- 172.21.15.116:0/3251437816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f528406c190 0x7f52840ab150 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f527400c410 tx=0x7f527400c810 comp rx=0 tx=0).stop 2024-04-25T00:02:42.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.612+0000 7f528a9ae640 1 -- 172.21.15.116:0/3251437816 shutdown_connections 2024-04-25T00:02:42.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.612+0000 7f528a9ae640 1 --2- 172.21.15.116:0/3251437816 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f52640678c0 0x7f5264069d80 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:42.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.612+0000 7f528a9ae640 1 --2- 172.21.15.116:0/3251437816 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5284150dd0 0x7f52840acb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:42.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.612+0000 7f528a9ae640 1 --2- 172.21.15.116:0/3251437816 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f528406c6e0 0x7f528414e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:42.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.612+0000 7f528a9ae640 1 --2- 172.21.15.116:0/3251437816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f528406c190 0x7f52840ab150 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:42.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.612+0000 7f528a9ae640 1 -- 172.21.15.116:0/3251437816 >> 172.21.15.116:0/3251437816 conn(0x7f528406b210 msgr2=0x7f52840c6c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:02:42.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.612+0000 7f528a9ae640 1 -- 172.21.15.116:0/3251437816 shutdown_connections 2024-04-25T00:02:42.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.612+0000 7f528a9ae640 1 -- 172.21.15.116:0/3251437816 wait complete. 2024-04-25T00:02:42.637 INFO:tasks.mds_thrash.fs.[b]:mds.a down, removed from mdsmap 2024-04-25T00:02:42.637 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-04-25T00:02:42.637 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:02:42.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.736+0000 7f6b8a23f640 1 Processor -- start 2024-04-25T00:02:42.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.736+0000 7f6b8a23f640 1 -- start start 2024-04-25T00:02:42.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.736+0000 7f6b8a23f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6b840c1540 0x7f6b84158e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:42.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.736+0000 7f6b8a23f640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f6b84159370 0x7f6b84160160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:42.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.736+0000 7f6b8a23f640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b8413ee50 con 0x7f6b84152e30 2024-04-25T00:02:42.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.736+0000 7f6b8a23f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f6b84072ae0 con 0x7f6b840c1540 2024-04-25T00:02:42.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.736+0000 7f6b8a23f640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f6b84072dc0 con 0x7f6b84159370 2024-04-25T00:02:42.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.740+0000 7f6b82ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6b840c1540 0x7f6b84158e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:42.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.740+0000 7f6b82ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6b840c1540 0x7f6b84158e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45454/0 (socket says 172.21.15.116:45454) 2024-04-25T00:02:42.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.740+0000 7f6b82ffd640 1 -- 172.21.15.116:0/3432572556 learned_addr learned my addr 172.21.15.116:0/3432572556 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:02:42.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.740+0000 7f6b83fff640 1 --2- 172.21.15.116:0/3432572556 >> v2:172.21.15.174:3300/0 conn(0x7f6b84159370 0x7f6b84160160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:42.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.740+0000 7f6b82ffd640 1 -- 172.21.15.116:0/3432572556 >> v2:172.21.15.174:3300/0 conn(0x7f6b84159370 msgr2=0x7f6b84160160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:02:42.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.740+0000 7f6b82ffd640 1 --2- 172.21.15.116:0/3432572556 >> v2:172.21.15.174:3300/0 conn(0x7f6b84159370 0x7f6b84160160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:42.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.740+0000 7f6b82ffd640 1 -- 172.21.15.116:0/3432572556 >> v1:172.21.15.174:6789/0 conn(0x7f6b84152e30 legacy=0x7f6b840c3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:42.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.740+0000 7f6b82ffd640 1 -- 172.21.15.116:0/3432572556 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b840730a0 con 0x7f6b840c1540 2024-04-25T00:02:42.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.740+0000 7f6b82ffd640 1 --2- 172.21.15.116:0/3432572556 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b840c1540 0x7f6b84158e30 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f6b840720a0 tx=0x7f6b7801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85ff7988770c4ff3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:02:42.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.740+0000 7f6b827fc640 1 -- 172.21.15.116:0/3432572556 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b780b0070 con 0x7f6b840c1540 2024-04-25T00:02:42.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.740+0000 7f6b827fc640 1 -- 172.21.15.116:0/3432572556 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b780ab600 con 0x7f6b840c1540 2024-04-25T00:02:42.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.740+0000 7f6b827fc640 1 -- 172.21.15.116:0/3432572556 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b780ab9d0 con 0x7f6b840c1540 2024-04-25T00:02:42.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.740+0000 7f6b8a23f640 1 -- 172.21.15.116:0/3432572556 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b840c1540 msgr2=0x7f6b84158e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:42.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.740+0000 7f6b8a23f640 1 --2- 172.21.15.116:0/3432572556 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b840c1540 0x7f6b84158e30 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f6b840720a0 tx=0x7f6b7801c490 comp rx=0 tx=0).stop 2024-04-25T00:02:42.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.740+0000 7f6b8a23f640 1 -- 172.21.15.116:0/3432572556 shutdown_connections 2024-04-25T00:02:42.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.740+0000 7f6b8a23f640 1 --2- 172.21.15.116:0/3432572556 >> v2:172.21.15.174:3300/0 conn(0x7f6b84159370 0x7f6b84160160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:42.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.740+0000 7f6b8a23f640 1 --2- 172.21.15.116:0/3432572556 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b840c1540 0x7f6b84158e30 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:42.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.740+0000 7f6b8a23f640 1 -- 172.21.15.116:0/3432572556 >> 172.21.15.116:0/3432572556 conn(0x7f6b840c4200 msgr2=0x7f6b840c4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:02:42.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.740+0000 7f6b8a23f640 1 -- 172.21.15.116:0/3432572556 shutdown_connections 2024-04-25T00:02:42.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.740+0000 7f6b8a23f640 1 -- 172.21.15.116:0/3432572556 wait complete. 2024-04-25T00:02:42.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.744+0000 7f6b8a23f640 1 Processor -- start 2024-04-25T00:02:42.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.744+0000 7f6b8a23f640 1 -- start start 2024-04-25T00:02:42.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.744+0000 7f6b8a23f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b840c1540 0x7f6b8414cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:42.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.744+0000 7f6b8a23f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b84152e30 0x7f6b8414c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:42.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.744+0000 7f6b8a23f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b84159370 0x7f6b8413c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:42.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.744+0000 7f6b8a23f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6b84153210 con 0x7f6b84152e30 2024-04-25T00:02:42.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.744+0000 7f6b8a23f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6b840c1a50 con 0x7f6b84159370 2024-04-25T00:02:42.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.744+0000 7f6b8a23f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6b8414d220 con 0x7f6b840c1540 2024-04-25T00:02:42.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.744+0000 7f6b82ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b84152e30 0x7f6b8414c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:42.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.744+0000 7f6b837fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b840c1540 0x7f6b8414cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:42.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.744+0000 7f6b82ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b84152e30 0x7f6b8414c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45466/0 (socket says 172.21.15.116:45466) 2024-04-25T00:02:42.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.744+0000 7f6b82ffd640 1 -- 172.21.15.116:0/598389480 learned_addr learned my addr 172.21.15.116:0/598389480 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:02:42.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.744+0000 7f6b83fff640 1 --2- 172.21.15.116:0/598389480 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b84159370 0x7f6b8413c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:42.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.744+0000 7f6b82ffd640 1 -- 172.21.15.116:0/598389480 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b840c1540 msgr2=0x7f6b8414cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:42.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.744+0000 7f6b82ffd640 1 --2- 172.21.15.116:0/598389480 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b840c1540 0x7f6b8414cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:42.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.744+0000 7f6b82ffd640 1 -- 172.21.15.116:0/598389480 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b84159370 msgr2=0x7f6b8413c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:42.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.744+0000 7f6b82ffd640 1 --2- 172.21.15.116:0/598389480 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b84159370 0x7f6b8413c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:42.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.744+0000 7f6b82ffd640 1 -- 172.21.15.116:0/598389480 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b8414c810 con 0x7f6b84152e30 2024-04-25T00:02:42.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.744+0000 7f6b82ffd640 1 --2- 172.21.15.116:0/598389480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b84152e30 0x7f6b8414c2d0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f6b7801a0c0 tx=0x7f6b78019230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:02:42.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.744+0000 7f6b6bfff640 1 -- 172.21.15.116:0/598389480 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b780b9070 con 0x7f6b84152e30 2024-04-25T00:02:42.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.744+0000 7f6b6bfff640 1 -- 172.21.15.116:0/598389480 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b780ace50 con 0x7f6b84152e30 2024-04-25T00:02:42.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.744+0000 7f6b8a23f640 1 -- 172.21.15.116:0/598389480 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b8414d760 con 0x7f6b84152e30 2024-04-25T00:02:42.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.744+0000 7f6b6bfff640 1 -- 172.21.15.116:0/598389480 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b780b0040 con 0x7f6b84152e30 2024-04-25T00:02:42.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.744+0000 7f6b8a23f640 1 -- 172.21.15.116:0/598389480 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b8414ebe0 con 0x7f6b84152e30 2024-04-25T00:02:42.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.748+0000 7f6b8a23f640 1 -- 172.21.15.116:0/598389480 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b84068a60 con 0x7f6b84152e30 2024-04-25T00:02:42.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.748+0000 7f6b6bfff640 1 -- 172.21.15.116:0/598389480 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6b780bc070 con 0x7f6b84152e30 2024-04-25T00:02:42.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.748+0000 7f6b6bfff640 1 --2- 172.21.15.116:0/598389480 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6b700678c0 0x7f6b70069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:42.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.748+0000 7f6b6bfff640 1 -- 172.21.15.116:0/598389480 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6752+0+0 (secure 0 0 0) 0x7f6b7811efc0 con 0x7f6b84152e30 2024-04-25T00:02:42.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.748+0000 7f6b837fe640 1 --2- 172.21.15.116:0/598389480 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6b700678c0 0x7f6b70069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:42.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.748+0000 7f6b837fe640 1 --2- 172.21.15.116:0/598389480 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6b700678c0 0x7f6b70069d80 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6b840aa490 tx=0x7f6b74009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:02:42.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.756+0000 7f6b6bfff640 1 -- 172.21.15.116:0/598389480 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f6b78016110 con 0x7f6b84152e30 2024-04-25T00:02:43.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.996+0000 7f6b8a23f640 1 -- 172.21.15.116:0/598389480 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6b8406c440 con 0x7f6b84152e30 2024-04-25T00:02:43.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:42.996+0000 7f6b6bfff640 1 -- 172.21.15.116:0/598389480 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5663 (secure 0 0 0) 0x7f6b780ef130 con 0x7f6b84152e30 2024-04-25T00:02:43.000 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:02:43.000 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:01:08.748957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6837/1762356055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1762356055},{"type":"v1","addr":"172.21.15.174:6837","nonce":1762356055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270]},"id":1},{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:02:41.407879+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4242":{"gid":4242,"name":"c","rank":0,"incarnation":15,"state":"up:rejoin","state_seq":28,"addr":"172.21.15.116:6834/1459177921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1459177921},{"type":"v1","addr":"172.21.15.116:6834","nonce":1459177921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:02:43.001 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 17 2024-04-25T00:02:43.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:43.000+0000 7f6b8a23f640 1 -- 172.21.15.116:0/598389480 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6b700678c0 msgr2=0x7f6b70069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:43.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:43.000+0000 7f6b8a23f640 1 --2- 172.21.15.116:0/598389480 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6b700678c0 0x7f6b70069d80 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6b840aa490 tx=0x7f6b74009040 comp rx=0 tx=0).stop 2024-04-25T00:02:43.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:43.000+0000 7f6b8a23f640 1 -- 172.21.15.116:0/598389480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b84152e30 msgr2=0x7f6b8414c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:43.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:43.000+0000 7f6b8a23f640 1 --2- 172.21.15.116:0/598389480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b84152e30 0x7f6b8414c2d0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f6b7801a0c0 tx=0x7f6b78019230 comp rx=0 tx=0).stop 2024-04-25T00:02:43.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:43.000+0000 7f6b8a23f640 1 -- 172.21.15.116:0/598389480 shutdown_connections 2024-04-25T00:02:43.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:43.000+0000 7f6b8a23f640 1 --2- 172.21.15.116:0/598389480 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6b700678c0 0x7f6b70069d80 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:43.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:43.000+0000 7f6b8a23f640 1 --2- 172.21.15.116:0/598389480 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b84159370 0x7f6b8413c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:43.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:43.000+0000 7f6b8a23f640 1 --2- 172.21.15.116:0/598389480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b84152e30 0x7f6b8414c2d0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:43.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:43.000+0000 7f6b8a23f640 1 --2- 172.21.15.116:0/598389480 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b840c1540 0x7f6b8414cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:43.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:43.000+0000 7f6b8a23f640 1 -- 172.21.15.116:0/598389480 >> 172.21.15.116:0/598389480 conn(0x7f6b840c4200 msgr2=0x7f6b8406b210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:02:43.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:43.000+0000 7f6b8a23f640 1 -- 172.21.15.116:0/598389480 shutdown_connections 2024-04-25T00:02:43.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:43.000+0000 7f6b8a23f640 1 -- 172.21.15.116:0/598389480 wait complete. 2024-04-25T00:02:43.020 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 17, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:04.678737+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 9, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_4270": { "addr": "172.21.15.174:6837/1762356055", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6836", "nonce": 1762356055, "type": "v2" }, { "addr": "172.21.15.174:6837", "nonce": 1762356055, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4270, "incarnation": 4, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T00:01:08.748957+0000", "qdb_cluster": [ 4270 ], "qdb_leader": 4270, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4270 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:18.825887+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 17, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_4242": { "addr": "172.21.15.116:6834/1459177921", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6832", "nonce": 1459177921, "type": "v2" }, { "addr": "172.21.15.116:6834", "nonce": 1459177921, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4242, "incarnation": 15, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:rejoin", "state_seq": 28 } }, "last_failure": 0, "last_failure_osd_epoch": 30, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-04-25T00:02:41.407879+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4242 } } } ], "standbys": [ { "addr": "172.21.15.174:6835/2883220846", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6834", "nonce": 2883220846, "type": "v2" }, { "addr": "172.21.15.174:6835", "nonce": 2883220846, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4235, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-25T00:02:44.743 INFO:tasks.ceph.mds.c.smithi116.stderr:2024-04-25T00:02:44.740+0000 7f9568f5f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T00:02:45.021 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:02:45.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.172+0000 7fecf03b6640 1 Processor -- start 2024-04-25T00:02:45.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.172+0000 7fecf03b6640 1 -- start start 2024-04-25T00:02:45.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.176+0000 7fecf03b6640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fece8150d40 0x7fece8154170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:45.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.176+0000 7fecf03b6640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fece80c4c20 0x7fece80c6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:45.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.176+0000 7fecf03b6640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fece8070180 con 0x7fece8070770 2024-04-25T00:02:45.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.176+0000 7fecf03b6640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fece8071280 con 0x7fece8150d40 2024-04-25T00:02:45.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.176+0000 7fecf03b6640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fece806ff30 con 0x7fece80c4c20 2024-04-25T00:02:45.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.176+0000 7fecee152640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fece8150d40 0x7fece8154170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:45.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.176+0000 7feced951640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fece80c4c20 0x7fece80c6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:45.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.176+0000 7fecee152640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fece8150d40 0x7fece8154170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:52798/0 (socket says 172.21.15.116:52798) 2024-04-25T00:02:45.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.176+0000 7fecee152640 1 -- 172.21.15.116:0/145913151 learned_addr learned my addr 172.21.15.116:0/145913151 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:02:45.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.176+0000 7fecee152640 1 -- 172.21.15.116:0/145913151 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fece80c4c20 msgr2=0x7fece80c6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:02:45.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.176+0000 7fecee152640 1 --2- 172.21.15.116:0/145913151 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fece80c4c20 0x7fece80c6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:45.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.176+0000 7fecee152640 1 -- 172.21.15.116:0/145913151 >> v1:172.21.15.174:6789/0 conn(0x7fece8070770 legacy=0x7fece80ab200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:45.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.176+0000 7fecee152640 1 -- 172.21.15.116:0/145913151 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fece8070460 con 0x7fece8150d40 2024-04-25T00:02:45.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.180+0000 7fecee152640 1 --2- 172.21.15.116:0/145913151 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fece8150d40 0x7fece8154170 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fece40098a0 tx=0x7fece401c4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d53e8150a0486dd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:02:45.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.180+0000 7feced150640 1 -- 172.21.15.116:0/145913151 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece40b1070 con 0x7fece8150d40 2024-04-25T00:02:45.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.180+0000 7feced150640 1 -- 172.21.15.116:0/145913151 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fece40ac420 con 0x7fece8150d40 2024-04-25T00:02:45.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.180+0000 7feced150640 1 -- 172.21.15.116:0/145913151 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece40ac7f0 con 0x7fece8150d40 2024-04-25T00:02:45.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.180+0000 7fecf03b6640 1 -- 172.21.15.116:0/145913151 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fece8150d40 msgr2=0x7fece8154170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:45.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.180+0000 7fecf03b6640 1 --2- 172.21.15.116:0/145913151 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fece8150d40 0x7fece8154170 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fece40098a0 tx=0x7fece401c4b0 comp rx=0 tx=0).stop 2024-04-25T00:02:45.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.180+0000 7fecf03b6640 1 -- 172.21.15.116:0/145913151 shutdown_connections 2024-04-25T00:02:45.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.180+0000 7fecf03b6640 1 --2- 172.21.15.116:0/145913151 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fece80c4c20 0x7fece80c6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:45.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.180+0000 7fecf03b6640 1 --2- 172.21.15.116:0/145913151 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fece8150d40 0x7fece8154170 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:45.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.180+0000 7fecf03b6640 1 -- 172.21.15.116:0/145913151 >> 172.21.15.116:0/145913151 conn(0x7fece806f0c0 msgr2=0x7fece806f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:02:45.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.180+0000 7fecf03b6640 1 -- 172.21.15.116:0/145913151 shutdown_connections 2024-04-25T00:02:45.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.180+0000 7fecf03b6640 1 -- 172.21.15.116:0/145913151 wait complete. 2024-04-25T00:02:45.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.180+0000 7fecf03b6640 1 Processor -- start 2024-04-25T00:02:45.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.180+0000 7fecf03b6640 1 -- start start 2024-04-25T00:02:45.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.184+0000 7fecf03b6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fece8070770 0x7fece81ee150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:45.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.184+0000 7fecf03b6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fece80c4c20 0x7fece8155210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:45.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.184+0000 7fecf03b6640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fece8150d40 0x7fece814afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:45.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.184+0000 7fecf03b6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fece80aba90 con 0x7fece8070770 2024-04-25T00:02:45.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.184+0000 7fecee152640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fece8070770 0x7fece81ee150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:45.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.184+0000 7fecee152640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fece8070770 0x7fece81ee150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35886/0 (socket says 172.21.15.116:35886) 2024-04-25T00:02:45.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.184+0000 7fecee152640 1 -- 172.21.15.116:0/1397965363 learned_addr learned my addr 172.21.15.116:0/1397965363 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:02:45.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.184+0000 7fecf03b6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fece80ab910 con 0x7fece80c4c20 2024-04-25T00:02:45.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.184+0000 7feced951640 1 --2- 172.21.15.116:0/1397965363 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fece80c4c20 0x7fece8155210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:45.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.184+0000 7fecf03b6640 1 -- 172.21.15.116:0/1397965363 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fece80c7360 con 0x7fece8150d40 2024-04-25T00:02:45.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.184+0000 7fecee953640 1 --2- 172.21.15.116:0/1397965363 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fece8150d40 0x7fece814afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:45.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.184+0000 7fecee152640 1 -- 172.21.15.116:0/1397965363 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fece8150d40 msgr2=0x7fece814afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:45.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.184+0000 7fecee152640 1 --2- 172.21.15.116:0/1397965363 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fece8150d40 0x7fece814afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:45.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.184+0000 7fecee152640 1 -- 172.21.15.116:0/1397965363 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fece80c4c20 msgr2=0x7fece8155210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:45.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.184+0000 7fecee152640 1 --2- 172.21.15.116:0/1397965363 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fece80c4c20 0x7fece8155210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:45.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.184+0000 7fecee152640 1 -- 172.21.15.116:0/1397965363 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fece814a0a0 con 0x7fece8070770 2024-04-25T00:02:45.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.184+0000 7fecee152640 1 --2- 172.21.15.116:0/1397965363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fece8070770 0x7fece81ee150 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fece40af040 tx=0x7fece401acf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:02:45.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.184+0000 7fecdeffd640 1 -- 172.21.15.116:0/1397965363 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece40bb020 con 0x7fece8070770 2024-04-25T00:02:45.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.184+0000 7fecdeffd640 1 -- 172.21.15.116:0/1397965363 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fece40b7420 con 0x7fece8070770 2024-04-25T00:02:45.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.184+0000 7fecf03b6640 1 -- 172.21.15.116:0/1397965363 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fece814a380 con 0x7fece8070770 2024-04-25T00:02:45.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.184+0000 7fecf03b6640 1 -- 172.21.15.116:0/1397965363 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fece8150850 con 0x7fece8070770 2024-04-25T00:02:45.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.184+0000 7fecdeffd640 1 -- 172.21.15.116:0/1397965363 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece40b1040 con 0x7fece8070770 2024-04-25T00:02:45.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.184+0000 7fecf03b6640 1 -- 172.21.15.116:0/1397965363 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fece8068a60 con 0x7fece8070770 2024-04-25T00:02:45.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.188+0000 7fecdeffd640 1 -- 172.21.15.116:0/1397965363 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fece40be020 con 0x7fece8070770 2024-04-25T00:02:45.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.188+0000 7fecdeffd640 1 --2- 172.21.15.116:0/1397965363 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7feccc0678f0 0x7feccc069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:45.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.188+0000 7fecdeffd640 1 -- 172.21.15.116:0/1397965363 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6752+0+0 (secure 0 0 0) 0x7fece4120520 con 0x7fece8070770 2024-04-25T00:02:45.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.188+0000 7feced951640 1 --2- 172.21.15.116:0/1397965363 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7feccc0678f0 0x7feccc069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:45.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.188+0000 7feced951640 1 --2- 172.21.15.116:0/1397965363 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7feccc0678f0 0x7feccc069db0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fece81522c0 tx=0x7fecd800f040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:02:45.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.196+0000 7fecdeffd640 1 -- 172.21.15.116:0/1397965363 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fece40c5030 con 0x7fece8070770 2024-04-25T00:02:45.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.572+0000 7fecf03b6640 1 -- 172.21.15.116:0/1397965363 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fece80c6d20 con 0x7fece8070770 2024-04-25T00:02:45.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.572+0000 7fecdeffd640 1 -- 172.21.15.116:0/1397965363 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+5670 (secure 0 0 0) 0x7fece4016130 con 0x7fece8070770 2024-04-25T00:02:45.576 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:02:45.576 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:01:08.748957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6837/1762356055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1762356055},{"type":"v1","addr":"172.21.15.174:6837","nonce":1762356055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270]},"id":1},{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:02:43.411054+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4242":{"gid":4242,"name":"c","rank":0,"incarnation":15,"state":"up:active","state_seq":29,"addr":"172.21.15.116:6834/1459177921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1459177921},{"type":"v1","addr":"172.21.15.116:6834","nonce":1459177921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4242,"qdb_cluster":[4242]},"id":2}]} 2024-04-25T00:02:45.576 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 18 2024-04-25T00:02:45.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.576+0000 7fecf03b6640 1 -- 172.21.15.116:0/1397965363 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7feccc0678f0 msgr2=0x7feccc069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:45.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.576+0000 7fecf03b6640 1 --2- 172.21.15.116:0/1397965363 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7feccc0678f0 0x7feccc069db0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fece81522c0 tx=0x7fecd800f040 comp rx=0 tx=0).stop 2024-04-25T00:02:45.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.576+0000 7fecf03b6640 1 -- 172.21.15.116:0/1397965363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fece8070770 msgr2=0x7fece81ee150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:45.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.576+0000 7fecf03b6640 1 --2- 172.21.15.116:0/1397965363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fece8070770 0x7fece81ee150 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fece40af040 tx=0x7fece401acf0 comp rx=0 tx=0).stop 2024-04-25T00:02:45.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.576+0000 7fecf03b6640 1 -- 172.21.15.116:0/1397965363 shutdown_connections 2024-04-25T00:02:45.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.576+0000 7fecf03b6640 1 --2- 172.21.15.116:0/1397965363 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7feccc0678f0 0x7feccc069db0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:45.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.576+0000 7fecf03b6640 1 --2- 172.21.15.116:0/1397965363 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fece8150d40 0x7fece814afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:45.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.576+0000 7fecf03b6640 1 --2- 172.21.15.116:0/1397965363 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fece80c4c20 0x7fece8155210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:45.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.576+0000 7fecf03b6640 1 --2- 172.21.15.116:0/1397965363 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fece8070770 0x7fece81ee150 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:45.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.576+0000 7fecf03b6640 1 -- 172.21.15.116:0/1397965363 >> 172.21.15.116:0/1397965363 conn(0x7fece806f0c0 msgr2=0x7fece8070b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:02:45.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.576+0000 7fecf03b6640 1 -- 172.21.15.116:0/1397965363 shutdown_connections 2024-04-25T00:02:45.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:45.576+0000 7fecf03b6640 1 -- 172.21.15.116:0/1397965363 wait complete. 2024-04-25T00:02:45.599 INFO:tasks.mds_thrash.fs.[b]:mds.c has gained rank=0, replacing gid=4245 2024-04-25T00:02:45.599 INFO:tasks.mds_thrash.fs.[b]:waiting for 6 secs before reviving mds.a 2024-04-25T00:02:51.600 INFO:tasks.mds_thrash.fs.[b]:reviving mds.a 2024-04-25T00:02:51.600 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-25T00:02:51.600 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-25T00:02:51.603 INFO:tasks.ceph.mds.a:Started 2024-04-25T00:02:51.603 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:02:51.683 INFO:tasks.ceph.mds.a.smithi116.stdout:starting mds.a at 2024-04-25T00:02:51.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:51.756+0000 7f8f9496d640 1 Processor -- start 2024-04-25T00:02:51.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:51.756+0000 7f8f9496d640 1 -- start start 2024-04-25T00:02:51.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:51.756+0000 7f8f9496d640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f8f90150d20 0x7f8f90153620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:51.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:51.756+0000 7f8f9496d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f900c36a0 con 0x7f8f900c3240 2024-04-25T00:02:51.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:51.756+0000 7f8f9496d640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f9006f8c0 con 0x7f8f90071050 2024-04-25T00:02:51.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:51.756+0000 7f8f9496d640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f8f9006fba0 con 0x7f8f90150d20 2024-04-25T00:02:51.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:51.756+0000 7f8f8dd9b640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f8f900c3240 0x7f8f900c3e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:42584/0 (socket says 172.21.15.116:42584) 2024-04-25T00:02:51.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:51.756+0000 7f8f8dd9b640 1 -- 172.21.15.116:0/2121780757 learned_addr learned my addr 172.21.15.116:0/2121780757 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:02:51.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:51.756+0000 7f8f8e59c640 1 --2- 172.21.15.116:0/2121780757 >> v2:172.21.15.174:3300/0 conn(0x7f8f90150d20 0x7f8f90153620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:51.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:51.756+0000 7f8f8e59c640 1 -- 172.21.15.116:0/2121780757 >> v1:172.21.15.174:6789/0 conn(0x7f8f90071050 legacy=0x7f8f9015df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:51.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:51.756+0000 7f8f8d59a640 1 -- 172.21.15.116:0/2121780757 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3671783287 0 0) 0x7f8f900c36a0 con 0x7f8f900c3240 2024-04-25T00:02:51.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:51.756+0000 7f8f8e59c640 1 -- 172.21.15.116:0/2121780757 >> v1:172.21.15.116:6789/0 conn(0x7f8f900c3240 legacy=0x7f8f900c3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:51.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:51.756+0000 7f8f8e59c640 1 -- 172.21.15.116:0/2121780757 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f900c5420 con 0x7f8f90150d20 2024-04-25T00:02:51.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:51.760+0000 7f8f8e59c640 1 --2- 172.21.15.116:0/2121780757 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8f90150d20 0x7f8f90153620 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8f80009740 tx=0x7f8f8001c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e9040bbc6d4a400 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:02:51.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:51.760+0000 7f8f8d59a640 1 -- 172.21.15.116:0/2121780757 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f800b0070 con 0x7f8f90150d20 2024-04-25T00:02:51.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:51.760+0000 7f8f8d59a640 1 -- 172.21.15.116:0/2121780757 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f800ab600 con 0x7f8f90150d20 2024-04-25T00:02:51.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:51.760+0000 7f8f8d59a640 1 -- 172.21.15.116:0/2121780757 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f800ab9d0 con 0x7f8f90150d20 2024-04-25T00:02:51.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:51.760+0000 7f8f9496d640 1 -- 172.21.15.116:0/2121780757 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8f90150d20 msgr2=0x7f8f90153620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:51.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:51.760+0000 7f8f9496d640 1 --2- 172.21.15.116:0/2121780757 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8f90150d20 0x7f8f90153620 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8f80009740 tx=0x7f8f8001c2d0 comp rx=0 tx=0).stop 2024-04-25T00:02:51.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:51.760+0000 7f8f9496d640 1 -- 172.21.15.116:0/2121780757 shutdown_connections 2024-04-25T00:02:51.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:51.760+0000 7f8f9496d640 1 --2- 172.21.15.116:0/2121780757 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8f90150d20 0x7f8f90153620 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:51.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:51.760+0000 7f8f9496d640 1 -- 172.21.15.116:0/2121780757 >> 172.21.15.116:0/2121780757 conn(0x7f8f900c9160 msgr2=0x7f8f900c9570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:02:51.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:51.760+0000 7f8f9496d640 1 -- 172.21.15.116:0/2121780757 shutdown_connections 2024-04-25T00:02:51.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:51.760+0000 7f8f9496d640 1 -- 172.21.15.116:0/2121780757 wait complete. 2024-04-25T00:02:51.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:51.764+0000 7f8f9496d640 1 Processor -- start 2024-04-25T00:02:51.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:51.764+0000 7f8f9496d640 1 -- start start 2024-04-25T00:02:51.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:51.764+0000 7f8f9496d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8f90071050 0x7f8f900ae6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:51.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:51.764+0000 7f8f9496d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8f900c3240 0x7f8f901518e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:51.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:51.764+0000 7f8f9496d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f90150d20 0x7f8f9014c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:51.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:51.764+0000 7f8f9496d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8f900c4bd0 con 0x7f8f90150d20 2024-04-25T00:02:51.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:51.764+0000 7f8f9496d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8f900c4550 con 0x7f8f90071050 2024-04-25T00:02:51.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:51.764+0000 7f8f9496d640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f8f900c4d50 con 0x7f8f900c3240 2024-04-25T00:02:51.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:51.764+0000 7f8f8dd9b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8f900c3240 0x7f8f901518e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:51.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:51.764+0000 7f8f8dd9b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8f900c3240 0x7f8f901518e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:56712/0 (socket says 172.21.15.116:56712) 2024-04-25T00:02:51.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:51.764+0000 7f8f8e59c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8f90071050 0x7f8f900ae6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:51.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:51.764+0000 7f8f8dd9b640 1 -- 172.21.15.116:0/447853336 learned_addr learned my addr 172.21.15.116:0/447853336 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:02:51.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:51.764+0000 7f8f8ed9d640 1 --2- 172.21.15.116:0/447853336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f90150d20 0x7f8f9014c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:51.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:51.764+0000 7f8f8dd9b640 1 -- 172.21.15.116:0/447853336 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8f90071050 msgr2=0x7f8f900ae6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:51.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:51.764+0000 7f8f8dd9b640 1 --2- 172.21.15.116:0/447853336 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8f90071050 0x7f8f900ae6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:51.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:51.764+0000 7f8f8dd9b640 1 -- 172.21.15.116:0/447853336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f90150d20 msgr2=0x7f8f9014c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:51.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:51.764+0000 7f8f8dd9b640 1 --2- 172.21.15.116:0/447853336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f90150d20 0x7f8f9014c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:51.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:51.764+0000 7f8f8dd9b640 1 -- 172.21.15.116:0/447853336 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f90153380 con 0x7f8f900c3240 2024-04-25T00:02:51.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:51.764+0000 7f8f8e59c640 1 --2- 172.21.15.116:0/447853336 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8f90071050 0x7f8f900ae6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:02:51.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:51.764+0000 7f8f8ed9d640 1 --2- 172.21.15.116:0/447853336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f90150d20 0x7f8f9014c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:02:51.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:51.764+0000 7f8f8dd9b640 1 --2- 172.21.15.116:0/447853336 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8f900c3240 0x7f8f901518e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8f8400c4d0 tx=0x7f8f8400c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:02:51.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:51.764+0000 7f8f7effd640 1 -- 172.21.15.116:0/447853336 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f84018070 con 0x7f8f900c3240 2024-04-25T00:02:51.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:51.764+0000 7f8f7effd640 1 -- 172.21.15.116:0/447853336 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f840074e0 con 0x7f8f900c3240 2024-04-25T00:02:51.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:51.764+0000 7f8f9496d640 1 -- 172.21.15.116:0/447853336 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f90153660 con 0x7f8f900c3240 2024-04-25T00:02:51.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:51.764+0000 7f8f7effd640 1 -- 172.21.15.116:0/447853336 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f84004a30 con 0x7f8f900c3240 2024-04-25T00:02:51.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:51.768+0000 7f8f9496d640 1 -- 172.21.15.116:0/447853336 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f900aebe0 con 0x7f8f900c3240 2024-04-25T00:02:51.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:51.768+0000 7f8f9496d640 1 -- 172.21.15.116:0/447853336 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f90068a60 con 0x7f8f900c3240 2024-04-25T00:02:51.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:51.768+0000 7f8f7effd640 1 -- 172.21.15.116:0/447853336 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8f8401b070 con 0x7f8f900c3240 2024-04-25T00:02:51.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:51.768+0000 7f8f7effd640 1 --2- 172.21.15.116:0/447853336 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8f700678c0 0x7f8f70069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:51.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:51.768+0000 7f8f7effd640 1 -- 172.21.15.116:0/447853336 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(30..30 src has 1..30) ==== 6752+0+0 (secure 0 0 0) 0x7f8f840857a0 con 0x7f8f900c3240 2024-04-25T00:02:51.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:51.768+0000 7f8f8e59c640 1 --2- 172.21.15.116:0/447853336 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8f700678c0 0x7f8f70069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:51.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:51.768+0000 7f8f8e59c640 1 --2- 172.21.15.116:0/447853336 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8f700678c0 0x7f8f70069d80 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8f800ad000 tx=0x7f8f800a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:02:51.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:51.776+0000 7f8f7effd640 1 -- 172.21.15.116:0/447853336 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f8f84055a10 con 0x7f8f900c3240 2024-04-25T00:02:52.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:52.160+0000 7f8f9496d640 1 -- 172.21.15.116:0/447853336 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8f9015ec00 con 0x7f8f900c3240 2024-04-25T00:02:52.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:52.160+0000 7f8f7effd640 1 -- 172.21.15.116:0/447853336 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+5670 (secure 0 0 0) 0x7f8f84059d10 con 0x7f8f900c3240 2024-04-25T00:02:52.167 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:02:52.167 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:01:08.748957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6837/1762356055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1762356055},{"type":"v1","addr":"172.21.15.174:6837","nonce":1762356055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270]},"id":1},{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:02:43.411054+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4242":{"gid":4242,"name":"c","rank":0,"incarnation":15,"state":"up:active","state_seq":29,"addr":"172.21.15.116:6834/1459177921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1459177921},{"type":"v1","addr":"172.21.15.116:6834","nonce":1459177921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4242,"qdb_cluster":[4242]},"id":2}]} 2024-04-25T00:02:52.167 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 18 2024-04-25T00:02:52.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:52.164+0000 7f8f9496d640 1 -- 172.21.15.116:0/447853336 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8f700678c0 msgr2=0x7f8f70069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:52.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:52.164+0000 7f8f9496d640 1 --2- 172.21.15.116:0/447853336 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8f700678c0 0x7f8f70069d80 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8f800ad000 tx=0x7f8f800a6000 comp rx=0 tx=0).stop 2024-04-25T00:02:52.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:52.164+0000 7f8f9496d640 1 -- 172.21.15.116:0/447853336 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8f900c3240 msgr2=0x7f8f901518e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:52.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:52.164+0000 7f8f9496d640 1 --2- 172.21.15.116:0/447853336 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8f900c3240 0x7f8f901518e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8f8400c4d0 tx=0x7f8f8400c990 comp rx=0 tx=0).stop 2024-04-25T00:02:52.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:52.164+0000 7f8f9496d640 1 -- 172.21.15.116:0/447853336 shutdown_connections 2024-04-25T00:02:52.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:52.164+0000 7f8f9496d640 1 --2- 172.21.15.116:0/447853336 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8f700678c0 0x7f8f70069d80 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:52.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:52.164+0000 7f8f9496d640 1 --2- 172.21.15.116:0/447853336 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f90150d20 0x7f8f9014c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:52.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:52.164+0000 7f8f9496d640 1 --2- 172.21.15.116:0/447853336 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8f900c3240 0x7f8f901518e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:52.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:52.164+0000 7f8f9496d640 1 --2- 172.21.15.116:0/447853336 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8f90071050 0x7f8f900ae6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:52.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:52.164+0000 7f8f9496d640 1 -- 172.21.15.116:0/447853336 >> 172.21.15.116:0/447853336 conn(0x7f8f900c9160 msgr2=0x7f8f9013d0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:02:52.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:52.164+0000 7f8f9496d640 1 -- 172.21.15.116:0/447853336 shutdown_connections 2024-04-25T00:02:52.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:52.164+0000 7f8f9496d640 1 -- 172.21.15.116:0/447853336 wait complete. 2024-04-25T00:02:52.190 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2024-04-25T00:02:54.191 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:02:54.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.344+0000 7f276a479640 1 Processor -- start 2024-04-25T00:02:54.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.344+0000 7f276a479640 1 -- start start 2024-04-25T00:02:54.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.348+0000 7f276a479640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f27640c19f0 0x7f276406f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:54.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.348+0000 7f276a479640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2764153aa0 con 0x7f2764150bf0 2024-04-25T00:02:54.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.348+0000 7f276a479640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2764153d80 con 0x7f2764153570 2024-04-25T00:02:54.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.348+0000 7f276a479640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f276406e630 con 0x7f27640c19f0 2024-04-25T00:02:54.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.348+0000 7f2763fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f2764150bf0 0x7f276406c220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:42598/0 (socket says 172.21.15.116:42598) 2024-04-25T00:02:54.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.348+0000 7f2763fff640 1 -- 172.21.15.116:0/2009378806 learned_addr learned my addr 172.21.15.116:0/2009378806 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:02:54.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.348+0000 7f2768a16640 1 --2- 172.21.15.116:0/2009378806 >> v2:172.21.15.174:3300/0 conn(0x7f27640c19f0 0x7f276406f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:54.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.348+0000 7f27637fe640 1 -- 172.21.15.116:0/2009378806 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 90912946 0 0) 0x7f2764153aa0 con 0x7f2764150bf0 2024-04-25T00:02:54.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.348+0000 7f27637fe640 1 -- 172.21.15.116:0/2009378806 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2750003530 con 0x7f2764150bf0 2024-04-25T00:02:54.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.348+0000 7f27637fe640 1 -- 172.21.15.116:0/2009378806 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 240043822 0 0) 0x7f2764153d80 con 0x7f2764153570 2024-04-25T00:02:54.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.348+0000 7f27637fe640 1 -- 172.21.15.116:0/2009378806 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2764153aa0 con 0x7f2764153570 2024-04-25T00:02:54.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.348+0000 7f2768a16640 1 -- 172.21.15.116:0/2009378806 >> v1:172.21.15.174:6789/0 conn(0x7f2764153570 legacy=0x7f276406eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:54.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.348+0000 7f2768a16640 1 -- 172.21.15.116:0/2009378806 >> v1:172.21.15.116:6789/0 conn(0x7f2764150bf0 legacy=0x7f276406c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:54.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.348+0000 7f2768a16640 1 -- 172.21.15.116:0/2009378806 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2764154060 con 0x7f27640c19f0 2024-04-25T00:02:54.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.352+0000 7f2768a16640 1 --2- 172.21.15.116:0/2009378806 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f27640c19f0 0x7f276406f1d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f275c009d90 tx=0x7f275c01c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b12062f6c603c347 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:02:54.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.352+0000 7f27637fe640 1 -- 172.21.15.116:0/2009378806 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f275c0b2070 con 0x7f27640c19f0 2024-04-25T00:02:54.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.352+0000 7f27637fe640 1 -- 172.21.15.116:0/2009378806 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f275c0a57a0 con 0x7f27640c19f0 2024-04-25T00:02:54.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.352+0000 7f27637fe640 1 -- 172.21.15.116:0/2009378806 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f275c0a5b70 con 0x7f27640c19f0 2024-04-25T00:02:54.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.352+0000 7f276a479640 1 -- 172.21.15.116:0/2009378806 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f27640c19f0 msgr2=0x7f276406f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:54.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.352+0000 7f276a479640 1 --2- 172.21.15.116:0/2009378806 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f27640c19f0 0x7f276406f1d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f275c009d90 tx=0x7f275c01c8e0 comp rx=0 tx=0).stop 2024-04-25T00:02:54.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.352+0000 7f276a479640 1 -- 172.21.15.116:0/2009378806 shutdown_connections 2024-04-25T00:02:54.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.352+0000 7f276a479640 1 --2- 172.21.15.116:0/2009378806 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f27640c19f0 0x7f276406f1d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:54.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.352+0000 7f276a479640 1 -- 172.21.15.116:0/2009378806 >> 172.21.15.116:0/2009378806 conn(0x7f27640c4270 msgr2=0x7f27640c4680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:02:54.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.352+0000 7f276a479640 1 -- 172.21.15.116:0/2009378806 shutdown_connections 2024-04-25T00:02:54.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.352+0000 7f276a479640 1 -- 172.21.15.116:0/2009378806 wait complete. 2024-04-25T00:02:54.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.352+0000 7f276a479640 1 Processor -- start 2024-04-25T00:02:54.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.352+0000 7f276a479640 1 -- start start 2024-04-25T00:02:54.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.356+0000 7f276a479640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f27640c19f0 0x7f27640c0e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:54.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.356+0000 7f276a479640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2764150bf0 0x7f27641f1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:54.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.356+0000 7f276a479640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2764153570 0x7f27641f1e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:54.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.356+0000 7f276a479640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f27640a72b0 con 0x7f2764150bf0 2024-04-25T00:02:54.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.356+0000 7f275b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2764150bf0 0x7f27641f1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:54.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.356+0000 7f276a479640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f27640a7130 con 0x7f2764153570 2024-04-25T00:02:54.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.356+0000 7f275b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2764150bf0 0x7f27641f1940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35916/0 (socket says 172.21.15.116:35916) 2024-04-25T00:02:54.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.356+0000 7f275b7fe640 1 -- 172.21.15.116:0/3309329542 learned_addr learned my addr 172.21.15.116:0/3309329542 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:02:54.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.356+0000 7f276a479640 1 -- 172.21.15.116:0/3309329542 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f2764161720 con 0x7f27640c19f0 2024-04-25T00:02:54.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.356+0000 7f2763fff640 1 --2- 172.21.15.116:0/3309329542 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f27640c19f0 0x7f27640c0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:54.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.356+0000 7f2768a16640 1 --2- 172.21.15.116:0/3309329542 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2764153570 0x7f27641f1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:54.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.356+0000 7f275b7fe640 1 -- 172.21.15.116:0/3309329542 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f27640c19f0 msgr2=0x7f27640c0e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:54.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.356+0000 7f275b7fe640 1 --2- 172.21.15.116:0/3309329542 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f27640c19f0 0x7f27640c0e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:54.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.356+0000 7f275b7fe640 1 -- 172.21.15.116:0/3309329542 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2764153570 msgr2=0x7f27641f1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:54.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.356+0000 7f275b7fe640 1 --2- 172.21.15.116:0/3309329542 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2764153570 0x7f27641f1e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:54.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.356+0000 7f275b7fe640 1 -- 172.21.15.116:0/3309329542 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2764072740 con 0x7f2764150bf0 2024-04-25T00:02:54.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.356+0000 7f275b7fe640 1 --2- 172.21.15.116:0/3309329542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2764150bf0 0x7f27641f1940 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f274c00c430 tx=0x7f274c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:02:54.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.356+0000 7f27617fa640 1 -- 172.21.15.116:0/3309329542 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f274c017070 con 0x7f2764150bf0 2024-04-25T00:02:54.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.356+0000 7f2763fff640 1 --2- 172.21.15.116:0/3309329542 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f27640c19f0 0x7f27640c0e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:02:54.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.356+0000 7f27617fa640 1 -- 172.21.15.116:0/3309329542 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f274c0052f0 con 0x7f2764150bf0 2024-04-25T00:02:54.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.356+0000 7f27617fa640 1 -- 172.21.15.116:0/3309329542 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f274c011890 con 0x7f2764150bf0 2024-04-25T00:02:54.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.356+0000 7f2768a16640 1 --2- 172.21.15.116:0/3309329542 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2764153570 0x7f27641f1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:02:54.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.356+0000 7f276a479640 1 -- 172.21.15.116:0/3309329542 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2764144c10 con 0x7f2764150bf0 2024-04-25T00:02:54.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.356+0000 7f276a479640 1 -- 172.21.15.116:0/3309329542 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27640c1340 con 0x7f2764150bf0 2024-04-25T00:02:54.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.356+0000 7f275a7fc640 1 -- 172.21.15.116:0/3309329542 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f273c0052a0 con 0x7f2764150bf0 2024-04-25T00:02:54.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.360+0000 7f27617fa640 1 -- 172.21.15.116:0/3309329542 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f274c01a070 con 0x7f2764150bf0 2024-04-25T00:02:54.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.360+0000 7f27617fa640 1 --2- 172.21.15.116:0/3309329542 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f27440678f0 0x7f2744069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:02:54.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.360+0000 7f27617fa640 1 -- 172.21.15.116:0/3309329542 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6752+0+0 (secure 0 0 0) 0x7f274c0853b0 con 0x7f2764150bf0 2024-04-25T00:02:54.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.364+0000 7f2763fff640 1 --2- 172.21.15.116:0/3309329542 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f27440678f0 0x7f2744069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:02:54.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.364+0000 7f2763fff640 1 --2- 172.21.15.116:0/3309329542 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f27440678f0 0x7f2744069db0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2754005890 tx=0x7f2754009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:02:54.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.368+0000 7f27617fa640 1 -- 172.21.15.116:0/3309329542 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f274c055af0 con 0x7f2764150bf0 2024-04-25T00:02:54.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.744+0000 7f275a7fc640 1 -- 172.21.15.116:0/3309329542 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f273c005590 con 0x7f2764150bf0 2024-04-25T00:02:54.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.744+0000 7f27617fa640 1 -- 172.21.15.116:0/3309329542 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6498 (secure 0 0 0) 0x7f274c0553f0 con 0x7f2764150bf0 2024-04-25T00:02:54.751 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:02:54.751 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4735,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/2004876235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":2004876235},{"type":"v1","addr":"172.21.15.116:6835","nonce":2004876235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:01:08.748957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6837/1762356055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1762356055},{"type":"v1","addr":"172.21.15.174:6837","nonce":1762356055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270]},"id":1},{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:02:43.411054+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4242":{"gid":4242,"name":"c","rank":0,"incarnation":15,"state":"up:active","state_seq":29,"addr":"172.21.15.116:6834/1459177921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1459177921},{"type":"v1","addr":"172.21.15.116:6834","nonce":1459177921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4242,"qdb_cluster":[4242]},"id":2}]} 2024-04-25T00:02:54.751 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 19 2024-04-25T00:02:54.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.748+0000 7f275a7fc640 1 -- 172.21.15.116:0/3309329542 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f27440678f0 msgr2=0x7f2744069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:54.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.748+0000 7f275a7fc640 1 --2- 172.21.15.116:0/3309329542 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f27440678f0 0x7f2744069db0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2754005890 tx=0x7f2754009040 comp rx=0 tx=0).stop 2024-04-25T00:02:54.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.748+0000 7f275a7fc640 1 -- 172.21.15.116:0/3309329542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2764150bf0 msgr2=0x7f27641f1940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:02:54.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.748+0000 7f275a7fc640 1 --2- 172.21.15.116:0/3309329542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2764150bf0 0x7f27641f1940 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f274c00c430 tx=0x7f274c00c8f0 comp rx=0 tx=0).stop 2024-04-25T00:02:54.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.748+0000 7f275a7fc640 1 -- 172.21.15.116:0/3309329542 shutdown_connections 2024-04-25T00:02:54.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.748+0000 7f275a7fc640 1 --2- 172.21.15.116:0/3309329542 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f27440678f0 0x7f2744069db0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:54.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.748+0000 7f275a7fc640 1 --2- 172.21.15.116:0/3309329542 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2764153570 0x7f27641f1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:54.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.748+0000 7f275a7fc640 1 --2- 172.21.15.116:0/3309329542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2764150bf0 0x7f27641f1940 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:54.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.748+0000 7f275a7fc640 1 --2- 172.21.15.116:0/3309329542 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f27640c19f0 0x7f27640c0e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:02:54.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.748+0000 7f275a7fc640 1 -- 172.21.15.116:0/3309329542 >> 172.21.15.116:0/3309329542 conn(0x7f27640c4270 msgr2=0x7f27640a8ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:02:54.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.748+0000 7f275a7fc640 1 -- 172.21.15.116:0/3309329542 shutdown_connections 2024-04-25T00:02:54.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:02:54.748+0000 7f275a7fc640 1 -- 172.21.15.116:0/3309329542 wait complete. 2024-04-25T00:02:54.774 INFO:tasks.mds_thrash.fs.[b]:mds.a reported in up:standby state 2024-04-25T00:02:54.774 INFO:tasks.mds_thrash.fs.[b]:waiting for 21 secs before thrashing 2024-04-25T00:03:03.188 INFO:tasks.workunit.client.0.smithi116.stderr:Note: switching to 'd2c45d977142e024a59b68a8f1dce8f0b9840551'. 2024-04-25T00:03:03.188 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-04-25T00:03:03.188 INFO:tasks.workunit.client.0.smithi116.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-25T00:03:03.188 INFO:tasks.workunit.client.0.smithi116.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-25T00:03:03.188 INFO:tasks.workunit.client.0.smithi116.stderr:state without impacting any branches by switching back to a branch. 2024-04-25T00:03:03.188 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-04-25T00:03:03.188 INFO:tasks.workunit.client.0.smithi116.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-25T00:03:03.188 INFO:tasks.workunit.client.0.smithi116.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-25T00:03:03.189 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-04-25T00:03:03.189 INFO:tasks.workunit.client.0.smithi116.stderr: git switch -c 2024-04-25T00:03:03.189 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-04-25T00:03:03.189 INFO:tasks.workunit.client.0.smithi116.stderr:Or undo this operation with: 2024-04-25T00:03:03.189 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-04-25T00:03:03.189 INFO:tasks.workunit.client.0.smithi116.stderr: git switch - 2024-04-25T00:03:03.189 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-04-25T00:03:03.189 INFO:tasks.workunit.client.0.smithi116.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-25T00:03:03.189 INFO:tasks.workunit.client.0.smithi116.stderr: 2024-04-25T00:03:03.189 INFO:tasks.workunit.client.0.smithi116.stderr:HEAD is now at d2c45d97714 Merge pull request #56528 from adk3798/squid-mypy-bump 2024-04-25T00:03:03.192 DEBUG:teuthology.orchestra.run.smithi116:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-25T00:03:03.199 INFO:tasks.workunit.client.0.smithi116.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-25T00:03:03.202 INFO:tasks.workunit.client.0.smithi116.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-25T00:03:03.202 INFO:tasks.workunit.client.0.smithi116.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-25T00:03:03.939 INFO:tasks.workunit.client.0.smithi116.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-25T00:03:04.021 INFO:tasks.workunit.client.0.smithi116.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-25T00:03:04.072 INFO:tasks.workunit.client.0.smithi116.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-25T00:03:04.074 INFO:tasks.workunit.client.0.smithi116.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-25T00:03:04.074 INFO:tasks.workunit.client.0.smithi116.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-25T00:03:04.122 INFO:tasks.workunit.client.0.smithi116.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-25T00:03:04.126 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-04-25T00:03:04.126 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-25T00:03:04.175 INFO:tasks.workunit:Running workunits matching suites/ffsb.sh on client.0... 2024-04-25T00:03:04.176 INFO:tasks.workunit:Running workunit suites/ffsb.sh... 2024-04-25T00:03:04.176 DEBUG:teuthology.orchestra.run.smithi116:workunit test suites/ffsb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d2c45d977142e024a59b68a8f1dce8f0b9840551 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2024-04-25T00:03:04.238 INFO:tasks.workunit.client.0.smithi116.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2024-04-25T00:03:04.238 INFO:tasks.workunit.client.0.smithi116.stderr:+ mydir=/home/ubuntu/cephtest/clone.client.0/qa/workunits/suites 2024-04-25T00:03:04.239 INFO:tasks.workunit.client.0.smithi116.stderr:+ trap -- retry EXIT 2024-04-25T00:03:04.239 INFO:tasks.workunit.client.0.smithi116.stderr:+ rm -rf ffsb 2024-04-25T00:03:04.240 INFO:tasks.workunit.client.0.smithi116.stderr:+ timeout 1800 git clone https://git.ceph.com/ffsb.git --depth 1 2024-04-25T00:03:04.247 INFO:tasks.workunit.client.0.smithi116.stderr:Cloning into 'ffsb'... 2024-04-25T00:03:04.723 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:03:04.725 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:03:05.103 INFO:tasks.workunit.client.1.smithi174.stderr:Updating files: 90% (11324/12492) Updating files: 91% (11368/12492) Updating files: 92% (11493/12492) Updating files: 93% (11618/12492) Updating files: 94% (11743/12492) Updating files: 95% (11868/12492) Updating files: 96% (11993/12492) Updating files: 97% (12118/12492) Updating files: 98% (12243/12492) Updating files: 99% (12368/12492) Updating files: 100% (12492/12492) Updating files: 100% (12492/12492), done. 2024-04-25T00:03:05.430 INFO:tasks.workunit.client.1.smithi174.stderr:Note: switching to 'd2c45d977142e024a59b68a8f1dce8f0b9840551'. 2024-04-25T00:03:05.430 INFO:tasks.workunit.client.1.smithi174.stderr: 2024-04-25T00:03:05.431 INFO:tasks.workunit.client.1.smithi174.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-25T00:03:05.431 INFO:tasks.workunit.client.1.smithi174.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-25T00:03:05.431 INFO:tasks.workunit.client.1.smithi174.stderr:state without impacting any branches by switching back to a branch. 2024-04-25T00:03:05.431 INFO:tasks.workunit.client.1.smithi174.stderr: 2024-04-25T00:03:05.431 INFO:tasks.workunit.client.1.smithi174.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-25T00:03:05.431 INFO:tasks.workunit.client.1.smithi174.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-25T00:03:05.431 INFO:tasks.workunit.client.1.smithi174.stderr: 2024-04-25T00:03:05.431 INFO:tasks.workunit.client.1.smithi174.stderr: git switch -c 2024-04-25T00:03:05.431 INFO:tasks.workunit.client.1.smithi174.stderr: 2024-04-25T00:03:05.431 INFO:tasks.workunit.client.1.smithi174.stderr:Or undo this operation with: 2024-04-25T00:03:05.431 INFO:tasks.workunit.client.1.smithi174.stderr: 2024-04-25T00:03:05.431 INFO:tasks.workunit.client.1.smithi174.stderr: git switch - 2024-04-25T00:03:05.431 INFO:tasks.workunit.client.1.smithi174.stderr: 2024-04-25T00:03:05.431 INFO:tasks.workunit.client.1.smithi174.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-25T00:03:05.432 INFO:tasks.workunit.client.1.smithi174.stderr: 2024-04-25T00:03:05.432 INFO:tasks.workunit.client.1.smithi174.stderr:HEAD is now at d2c45d97714 Merge pull request #56528 from adk3798/squid-mypy-bump 2024-04-25T00:03:05.435 DEBUG:teuthology.orchestra.run.smithi174:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2024-04-25T00:03:05.441 INFO:tasks.workunit.client.1.smithi174.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-25T00:03:05.444 INFO:tasks.workunit.client.1.smithi174.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-04-25T00:03:05.444 INFO:tasks.workunit.client.1.smithi174.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-25T00:03:06.565 INFO:tasks.workunit.client.1.smithi174.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-25T00:03:06.640 INFO:tasks.workunit.client.1.smithi174.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-25T00:03:06.687 INFO:tasks.workunit.client.1.smithi174.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-04-25T00:03:06.688 INFO:tasks.workunit.client.1.smithi174.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-04-25T00:03:06.688 INFO:tasks.workunit.client.1.smithi174.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-25T00:03:06.734 INFO:tasks.workunit.client.1.smithi174.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-04-25T00:03:06.738 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-04-25T00:03:06.738 DEBUG:teuthology.orchestra.run.smithi174:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2024-04-25T00:03:06.785 INFO:tasks.workunit:Running workunits matching suites/ffsb.sh on client.1... 2024-04-25T00:03:06.787 INFO:tasks.workunit:Running workunit suites/ffsb.sh... 2024-04-25T00:03:06.787 DEBUG:teuthology.orchestra.run.smithi174:workunit test suites/ffsb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d2c45d977142e024a59b68a8f1dce8f0b9840551 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/ffsb.sh 2024-04-25T00:03:06.849 INFO:tasks.workunit.client.1.smithi174.stderr:++ dirname /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/ffsb.sh 2024-04-25T00:03:06.850 INFO:tasks.workunit.client.1.smithi174.stderr:+ mydir=/home/ubuntu/cephtest/clone.client.1/qa/workunits/suites 2024-04-25T00:03:06.850 INFO:tasks.workunit.client.1.smithi174.stderr:+ trap -- retry EXIT 2024-04-25T00:03:06.850 INFO:tasks.workunit.client.1.smithi174.stderr:+ rm -rf ffsb 2024-04-25T00:03:06.851 INFO:tasks.workunit.client.1.smithi174.stderr:+ timeout 1800 git clone https://git.ceph.com/ffsb.git --depth 1 2024-04-25T00:03:06.857 INFO:tasks.workunit.client.1.smithi174.stderr:Cloning into 'ffsb'... 2024-04-25T00:03:07.477 INFO:tasks.workunit.client.0.smithi116.stderr:+ trap - EXIT 2024-04-25T00:03:07.477 INFO:tasks.workunit.client.0.smithi116.stderr:+ cd ffsb 2024-04-25T00:03:07.478 INFO:tasks.workunit.client.0.smithi116.stderr:+ ./configure 2024-04-25T00:03:07.652 INFO:tasks.workunit.client.0.smithi116.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-04-25T00:03:08.672 INFO:tasks.workunit.client.0.smithi116.stdout:checking whether build environment is sane... yes 2024-04-25T00:03:08.697 INFO:tasks.workunit.client.0.smithi116.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2024-04-25T00:03:08.698 INFO:tasks.workunit.client.0.smithi116.stdout:checking for gawk... gawk 2024-04-25T00:03:08.743 INFO:tasks.workunit.client.0.smithi116.stdout:checking whether make sets $(MAKE)... yes 2024-04-25T00:03:08.802 INFO:tasks.workunit.client.0.smithi116.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-04-25T00:03:08.802 INFO:tasks.workunit.client.0.smithi116.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-04-25T00:03:08.834 INFO:tasks.workunit.client.0.smithi116.stdout:checking for style of include used by make... GNU 2024-04-25T00:03:08.843 INFO:tasks.workunit.client.0.smithi116.stdout:checking for gcc... gcc 2024-04-25T00:03:08.952 INFO:tasks.workunit.client.0.smithi116.stdout:checking for C compiler default output file name... a.out 2024-04-25T00:03:08.956 INFO:tasks.workunit.client.0.smithi116.stdout:checking whether the C compiler works... yes 2024-04-25T00:03:08.960 INFO:tasks.workunit.client.0.smithi116.stdout:checking whether we are cross compiling... no 2024-04-25T00:03:09.023 INFO:tasks.workunit.client.0.smithi116.stdout:checking for suffix of executables... 2024-04-25T00:03:09.073 INFO:tasks.workunit.client.0.smithi116.stdout:checking for suffix of object files... o 2024-04-25T00:03:09.123 INFO:tasks.workunit.client.0.smithi116.stdout:checking whether we are using the GNU C compiler... yes 2024-04-25T00:03:09.177 INFO:tasks.workunit.client.0.smithi116.stdout:checking whether gcc accepts -g... yes 2024-04-25T00:03:09.332 INFO:tasks.workunit.client.0.smithi116.stdout:checking for gcc option to accept ISO C89... none needed 2024-04-25T00:03:09.562 INFO:tasks.workunit.client.0.smithi116.stdout:checking dependency style of gcc... gcc3 2024-04-25T00:03:09.651 INFO:tasks.workunit.client.0.smithi116.stdout:checking how to run the C preprocessor... gcc -E 2024-04-25T00:03:09.718 INFO:tasks.workunit.client.0.smithi116.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-04-25T00:03:09.720 INFO:tasks.workunit.client.0.smithi116.stdout:checking for egrep... /usr/bin/grep -E 2024-04-25T00:03:09.732 INFO:tasks.workunit.client.0.smithi116.stdout:checking for AIX... no 2024-04-25T00:03:09.762 INFO:tasks.workunit.client.1.smithi174.stderr:+ trap - EXIT 2024-04-25T00:03:09.762 INFO:tasks.workunit.client.1.smithi174.stderr:+ cd ffsb 2024-04-25T00:03:09.762 INFO:tasks.workunit.client.0.smithi116.stdout:checking for gcc... (cached) gcc 2024-04-25T00:03:09.763 INFO:tasks.workunit.client.1.smithi174.stderr:+ ./configure 2024-04-25T00:03:09.775 INFO:tasks.workunit.client.0.smithi116.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-04-25T00:03:09.776 INFO:tasks.workunit.client.0.smithi116.stdout:checking whether gcc accepts -g... (cached) yes 2024-04-25T00:03:09.777 INFO:tasks.workunit.client.0.smithi116.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-04-25T00:03:09.778 INFO:tasks.workunit.client.0.smithi116.stdout:checking dependency style of gcc... (cached) gcc3 2024-04-25T00:03:09.865 INFO:tasks.workunit.client.0.smithi116.stdout:checking for main in -lm... yes 2024-04-25T00:03:09.923 INFO:tasks.workunit.client.1.smithi174.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-04-25T00:03:09.952 INFO:tasks.workunit.client.0.smithi116.stdout:checking for main in -lpthread... yes 2024-04-25T00:03:10.241 INFO:tasks.workunit.client.0.smithi116.stdout:checking for ANSI C header files... yes 2024-04-25T00:03:10.346 INFO:tasks.workunit.client.0.smithi116.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2024-04-25T00:03:10.430 INFO:tasks.workunit.client.0.smithi116.stdout:checking for sys/types.h... yes 2024-04-25T00:03:10.527 INFO:tasks.workunit.client.0.smithi116.stdout:checking for sys/stat.h... yes 2024-04-25T00:03:10.610 INFO:tasks.workunit.client.0.smithi116.stdout:checking for stdlib.h... yes 2024-04-25T00:03:10.701 INFO:tasks.workunit.client.0.smithi116.stdout:checking for string.h... yes 2024-04-25T00:03:10.790 INFO:tasks.workunit.client.0.smithi116.stdout:checking for memory.h... yes 2024-04-25T00:03:10.878 INFO:tasks.workunit.client.0.smithi116.stdout:checking for strings.h... yes 2024-04-25T00:03:10.938 INFO:tasks.workunit.client.1.smithi174.stdout:checking whether build environment is sane... yes 2024-04-25T00:03:10.959 INFO:tasks.workunit.client.1.smithi174.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2024-04-25T00:03:10.960 INFO:tasks.workunit.client.1.smithi174.stdout:checking for gawk... gawk 2024-04-25T00:03:10.975 INFO:tasks.workunit.client.0.smithi116.stdout:checking for inttypes.h... yes 2024-04-25T00:03:10.987 INFO:tasks.workunit.client.1.smithi174.stdout:checking whether make sets $(MAKE)... yes 2024-04-25T00:03:11.037 INFO:tasks.workunit.client.1.smithi174.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-04-25T00:03:11.038 INFO:tasks.workunit.client.1.smithi174.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-04-25T00:03:11.043 INFO:tasks.workunit.client.0.smithi116.stdout:checking for stdint.h... yes 2024-04-25T00:03:11.059 INFO:tasks.workunit.client.1.smithi174.stdout:checking for style of include used by make... GNU 2024-04-25T00:03:11.066 INFO:tasks.workunit.client.1.smithi174.stdout:checking for gcc... gcc 2024-04-25T00:03:11.119 INFO:tasks.workunit.client.0.smithi116.stdout:checking for unistd.h... yes 2024-04-25T00:03:11.163 INFO:tasks.workunit.client.1.smithi174.stdout:checking for C compiler default output file name... a.out 2024-04-25T00:03:11.166 INFO:tasks.workunit.client.1.smithi174.stdout:checking whether the C compiler works... yes 2024-04-25T00:03:11.171 INFO:tasks.workunit.client.1.smithi174.stdout:checking whether we are cross compiling... no 2024-04-25T00:03:11.206 INFO:tasks.workunit.client.0.smithi116.stdout:checking pthread.h usability... yes 2024-04-25T00:03:11.229 INFO:tasks.workunit.client.1.smithi174.stdout:checking for suffix of executables... 2024-04-25T00:03:11.235 INFO:tasks.workunit.client.0.smithi116.stdout:checking pthread.h presence... yes 2024-04-25T00:03:11.237 INFO:tasks.workunit.client.0.smithi116.stdout:checking for pthread.h... yes 2024-04-25T00:03:11.275 INFO:tasks.workunit.client.1.smithi174.stdout:checking for suffix of object files... o 2024-04-25T00:03:11.311 INFO:tasks.workunit.client.0.smithi116.stdout:checking fcntl.h usability... yes 2024-04-25T00:03:11.317 INFO:tasks.workunit.client.1.smithi174.stdout:checking whether we are using the GNU C compiler... yes 2024-04-25T00:03:11.337 INFO:tasks.workunit.client.0.smithi116.stdout:checking fcntl.h presence... yes 2024-04-25T00:03:11.338 INFO:tasks.workunit.client.0.smithi116.stdout:checking for fcntl.h... yes 2024-04-25T00:03:11.361 INFO:tasks.workunit.client.1.smithi174.stdout:checking whether gcc accepts -g... yes 2024-04-25T00:03:11.413 INFO:tasks.workunit.client.0.smithi116.stdout:checking limits.h usability... yes 2024-04-25T00:03:11.439 INFO:tasks.workunit.client.0.smithi116.stdout:checking limits.h presence... yes 2024-04-25T00:03:11.439 INFO:tasks.workunit.client.0.smithi116.stdout:checking for limits.h... yes 2024-04-25T00:03:11.462 INFO:tasks.workunit.client.0.smithi116.stdout:checking for stdint.h... (cached) yes 2024-04-25T00:03:11.470 INFO:tasks.workunit.client.1.smithi174.stdout:checking for gcc option to accept ISO C89... none needed 2024-04-25T00:03:11.533 INFO:tasks.workunit.client.0.smithi116.stdout:checking sys/time.h usability... yes 2024-04-25T00:03:11.561 INFO:tasks.workunit.client.0.smithi116.stdout:checking sys/time.h presence... yes 2024-04-25T00:03:11.562 INFO:tasks.workunit.client.0.smithi116.stdout:checking for sys/time.h... yes 2024-04-25T00:03:11.590 INFO:tasks.workunit.client.0.smithi116.stdout:checking for unistd.h... (cached) yes 2024-04-25T00:03:11.636 INFO:tasks.workunit.client.1.smithi174.stdout:checking dependency style of gcc... gcc3 2024-04-25T00:03:11.652 INFO:tasks.workunit.client.0.smithi116.stdout:checking sys/vfs.h usability... yes 2024-04-25T00:03:11.677 INFO:tasks.workunit.client.0.smithi116.stdout:checking sys/vfs.h presence... yes 2024-04-25T00:03:11.678 INFO:tasks.workunit.client.0.smithi116.stdout:checking for sys/vfs.h... yes 2024-04-25T00:03:11.722 INFO:tasks.workunit.client.1.smithi174.stdout:checking how to run the C preprocessor... gcc -E 2024-04-25T00:03:11.757 INFO:tasks.workunit.client.0.smithi116.stdout:checking sys/limits.h usability... no 2024-04-25T00:03:11.786 INFO:tasks.workunit.client.1.smithi174.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-04-25T00:03:11.788 INFO:tasks.workunit.client.0.smithi116.stdout:checking sys/limits.h presence... no 2024-04-25T00:03:11.788 INFO:tasks.workunit.client.1.smithi174.stdout:checking for egrep... /usr/bin/grep -E 2024-04-25T00:03:11.788 INFO:tasks.workunit.client.0.smithi116.stdout:checking for sys/limits.h... no 2024-04-25T00:03:11.801 INFO:tasks.workunit.client.1.smithi174.stdout:checking for AIX... no 2024-04-25T00:03:11.824 INFO:tasks.workunit.client.1.smithi174.stdout:checking for gcc... (cached) gcc 2024-04-25T00:03:11.831 INFO:tasks.workunit.client.0.smithi116.stdout:checking for an ANSI C-conforming const... yes 2024-04-25T00:03:11.837 INFO:tasks.workunit.client.1.smithi174.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-04-25T00:03:11.841 INFO:tasks.workunit.client.1.smithi174.stdout:checking whether gcc accepts -g... (cached) yes 2024-04-25T00:03:11.842 INFO:tasks.workunit.client.1.smithi174.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-04-25T00:03:11.842 INFO:tasks.workunit.client.1.smithi174.stdout:checking dependency style of gcc... (cached) gcc3 2024-04-25T00:03:11.873 INFO:tasks.workunit.client.0.smithi116.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-04-25T00:03:11.876 INFO:tasks.workunit.client.0.smithi116.stdout:checking for function prototypes... yes 2024-04-25T00:03:11.929 INFO:tasks.workunit.client.1.smithi174.stdout:checking for main in -lm... yes 2024-04-25T00:03:11.936 INFO:tasks.workunit.client.0.smithi116.stdout:checking whether setvbuf arguments are reversed... no 2024-04-25T00:03:12.007 INFO:tasks.workunit.client.1.smithi174.stdout:checking for main in -lpthread... yes 2024-04-25T00:03:12.021 INFO:tasks.workunit.client.0.smithi116.stdout:checking for system... yes 2024-04-25T00:03:12.104 INFO:tasks.workunit.client.0.smithi116.stdout:checking for gettimeofday... yes 2024-04-25T00:03:12.192 INFO:tasks.workunit.client.0.smithi116.stdout:checking for mkdir... yes 2024-04-25T00:03:12.219 INFO:tasks.workunit.client.1.smithi174.stdout:checking for ANSI C header files... yes 2024-04-25T00:03:12.276 INFO:tasks.workunit.client.1.smithi174.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2024-04-25T00:03:12.277 INFO:tasks.workunit.client.0.smithi116.stdout:checking for strerror... yes 2024-04-25T00:03:12.334 INFO:tasks.workunit.client.1.smithi174.stdout:checking for sys/types.h... yes 2024-04-25T00:03:12.365 INFO:tasks.workunit.client.0.smithi116.stdout:checking for open64... yes 2024-04-25T00:03:12.405 INFO:tasks.workunit.client.1.smithi174.stdout:checking for sys/stat.h... yes 2024-04-25T00:03:12.473 INFO:tasks.workunit.client.0.smithi116.stdout:checking for stat64... yes 2024-04-25T00:03:12.478 INFO:tasks.workunit.client.1.smithi174.stdout:checking for stdlib.h... yes 2024-04-25T00:03:12.551 INFO:tasks.workunit.client.1.smithi174.stdout:checking for string.h... yes 2024-04-25T00:03:12.577 INFO:tasks.workunit.client.0.smithi116.stdout:checking for fseeko64... yes 2024-04-25T00:03:12.617 INFO:tasks.workunit.client.1.smithi174.stdout:checking for memory.h... yes 2024-04-25T00:03:12.672 INFO:tasks.workunit.client.0.smithi116.stdout:checking for lrand48_r... yes 2024-04-25T00:03:12.681 INFO:tasks.workunit.client.1.smithi174.stdout:checking for strings.h... yes 2024-04-25T00:03:12.745 INFO:tasks.workunit.client.1.smithi174.stdout:checking for inttypes.h... yes 2024-04-25T00:03:12.759 INFO:tasks.workunit.client.0.smithi116.stdout:checking for srand48_r... yes 2024-04-25T00:03:12.793 INFO:tasks.workunit.client.0.smithi116.stdout:configure: creating ./config.status 2024-04-25T00:03:12.812 INFO:tasks.workunit.client.1.smithi174.stdout:checking for stdint.h... yes 2024-04-25T00:03:12.896 INFO:tasks.workunit.client.1.smithi174.stdout:checking for unistd.h... yes 2024-04-25T00:03:12.924 INFO:tasks.workunit.client.0.smithi116.stdout:config.status: creating Makefile 2024-04-25T00:03:12.943 INFO:tasks.workunit.client.0.smithi116.stdout:config.status: creating config.h 2024-04-25T00:03:12.968 INFO:tasks.workunit.client.0.smithi116.stdout:config.status: executing depfiles commands 2024-04-25T00:03:12.974 INFO:tasks.workunit.client.1.smithi174.stdout:checking pthread.h usability... yes 2024-04-25T00:03:13.003 INFO:tasks.workunit.client.1.smithi174.stdout:checking pthread.h presence... yes 2024-04-25T00:03:13.004 INFO:tasks.workunit.client.1.smithi174.stdout:checking for pthread.h... yes 2024-04-25T00:03:13.042 INFO:tasks.workunit.client.0.smithi116.stdout:config.status: executing default commands 2024-04-25T00:03:13.087 INFO:tasks.workunit.client.1.smithi174.stdout:checking fcntl.h usability... yes 2024-04-25T00:03:13.109 INFO:tasks.workunit.client.0.smithi116.stderr:+ make 2024-04-25T00:03:13.117 INFO:tasks.workunit.client.1.smithi174.stdout:checking fcntl.h presence... yes 2024-04-25T00:03:13.117 INFO:tasks.workunit.client.1.smithi174.stdout:checking for fcntl.h... yes 2024-04-25T00:03:13.140 INFO:tasks.workunit.client.0.smithi116.stdout:cd . && /bin/bash /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run aclocal-1.10 2024-04-25T00:03:13.146 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: aclocal-1.10: command not found 2024-04-25T00:03:13.148 INFO:tasks.workunit.client.0.smithi116.stderr:WARNING: `aclocal-1.10' is missing on your system. You should only need it if 2024-04-25T00:03:13.148 INFO:tasks.workunit.client.0.smithi116.stderr: you modified `acinclude.m4' or `configure.in'. You might want 2024-04-25T00:03:13.148 INFO:tasks.workunit.client.0.smithi116.stderr: to install the `Automake' and `Perl' packages. Grab them from 2024-04-25T00:03:13.148 INFO:tasks.workunit.client.0.smithi116.stderr: any GNU archive site. 2024-04-25T00:03:13.152 INFO:tasks.workunit.client.0.smithi116.stdout: cd . && /bin/bash /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run automake-1.10 --gnu 2024-04-25T00:03:13.157 INFO:tasks.workunit.client.0.smithi116.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: automake-1.10: command not found 2024-04-25T00:03:13.158 INFO:tasks.workunit.client.0.smithi116.stderr:WARNING: `automake-1.10' is missing on your system. You should only need it if 2024-04-25T00:03:13.158 INFO:tasks.workunit.client.0.smithi116.stderr: you modified `Makefile.am', `acinclude.m4' or `configure.in'. 2024-04-25T00:03:13.158 INFO:tasks.workunit.client.0.smithi116.stderr: You might want to install the `Automake' and `Perl' packages. 2024-04-25T00:03:13.158 INFO:tasks.workunit.client.0.smithi116.stderr: Grab them from any GNU archive site. 2024-04-25T00:03:13.196 INFO:tasks.workunit.client.1.smithi174.stdout:checking limits.h usability... yes 2024-04-25T00:03:13.222 INFO:tasks.workunit.client.1.smithi174.stdout:checking limits.h presence... yes 2024-04-25T00:03:13.223 INFO:tasks.workunit.client.1.smithi174.stdout:checking for limits.h... yes 2024-04-25T00:03:13.245 INFO:tasks.workunit.client.1.smithi174.stdout:checking for stdint.h... (cached) yes 2024-04-25T00:03:13.308 INFO:tasks.workunit.client.1.smithi174.stdout:checking sys/time.h usability... yes 2024-04-25T00:03:13.329 INFO:tasks.workunit.client.0.smithi116.stdout:cd . && /bin/bash /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoconf 2024-04-25T00:03:13.335 INFO:tasks.workunit.client.1.smithi174.stdout:checking sys/time.h presence... yes 2024-04-25T00:03:13.336 INFO:tasks.workunit.client.1.smithi174.stdout:checking for sys/time.h... yes 2024-04-25T00:03:13.357 INFO:tasks.workunit.client.1.smithi174.stdout:checking for unistd.h... (cached) yes 2024-04-25T00:03:13.431 INFO:tasks.workunit.client.1.smithi174.stdout:checking sys/vfs.h usability... yes 2024-04-25T00:03:13.465 INFO:tasks.workunit.client.1.smithi174.stdout:checking sys/vfs.h presence... yes 2024-04-25T00:03:13.465 INFO:tasks.workunit.client.1.smithi174.stdout:checking for sys/vfs.h... yes 2024-04-25T00:03:13.493 INFO:tasks.workunit.client.0.smithi116.stderr:aclocal.m4:16: warning: this file was generated for autoconf 2.61. 2024-04-25T00:03:13.494 INFO:tasks.workunit.client.0.smithi116.stderr:You have another version of autoconf. It may work, but is not guaranteed to. 2024-04-25T00:03:13.494 INFO:tasks.workunit.client.0.smithi116.stderr:If you have problems, you may need to regenerate the build system entirely. 2024-04-25T00:03:13.494 INFO:tasks.workunit.client.0.smithi116.stderr:To do so, use the procedure documented by the package, typically `autoreconf'. 2024-04-25T00:03:13.551 INFO:tasks.workunit.client.1.smithi174.stdout:checking sys/limits.h usability... no 2024-04-25T00:03:13.581 INFO:tasks.workunit.client.1.smithi174.stdout:checking sys/limits.h presence... no 2024-04-25T00:03:13.581 INFO:tasks.workunit.client.1.smithi174.stdout:checking for sys/limits.h... no 2024-04-25T00:03:13.602 INFO:tasks.workunit.client.0.smithi116.stderr:configure.in:4: warning: The macro `AC_FOREACH' is obsolete. 2024-04-25T00:03:13.603 INFO:tasks.workunit.client.0.smithi116.stderr:configure.in:4: You should run autoupdate. 2024-04-25T00:03:13.603 INFO:tasks.workunit.client.0.smithi116.stderr:./lib/autoconf/general.m4:191: AC_FOREACH is expanded from... 2024-04-25T00:03:13.603 INFO:tasks.workunit.client.0.smithi116.stderr:aclocal.m4:684: _AM_SET_OPTIONS is expanded from... 2024-04-25T00:03:13.603 INFO:tasks.workunit.client.0.smithi116.stderr:aclocal.m4:410: AM_INIT_AUTOMAKE is expanded from... 2024-04-25T00:03:13.603 INFO:tasks.workunit.client.0.smithi116.stderr:configure.in:4: the top level 2024-04-25T00:03:13.603 INFO:tasks.workunit.client.0.smithi116.stderr:configure.in:6: warning: The macro `AM_CONFIG_HEADER' is obsolete. 2024-04-25T00:03:13.603 INFO:tasks.workunit.client.0.smithi116.stderr:configure.in:6: You should run autoupdate. 2024-04-25T00:03:13.603 INFO:tasks.workunit.client.0.smithi116.stderr:aclocal.m4:383: AM_CONFIG_HEADER is expanded from... 2024-04-25T00:03:13.603 INFO:tasks.workunit.client.0.smithi116.stderr:configure.in:6: the top level 2024-04-25T00:03:13.603 INFO:tasks.workunit.client.0.smithi116.stderr:configure.in:10: warning: The macro `AC_AIX' is obsolete. 2024-04-25T00:03:13.603 INFO:tasks.workunit.client.0.smithi116.stderr:configure.in:10: You should run autoupdate. 2024-04-25T00:03:13.603 INFO:tasks.workunit.client.0.smithi116.stderr:./lib/autoconf/specific.m4:540: AC_AIX is expanded from... 2024-04-25T00:03:13.603 INFO:tasks.workunit.client.0.smithi116.stderr:configure.in:10: the top level 2024-04-25T00:03:13.603 INFO:tasks.workunit.client.0.smithi116.stderr:configure.in:36: warning: The macro `AC_HEADER_STDC' is obsolete. 2024-04-25T00:03:13.604 INFO:tasks.workunit.client.0.smithi116.stderr:configure.in:36: You should run autoupdate. 2024-04-25T00:03:13.604 INFO:tasks.workunit.client.0.smithi116.stderr:./lib/autoconf/headers.m4:704: AC_HEADER_STDC is expanded from... 2024-04-25T00:03:13.604 INFO:tasks.workunit.client.0.smithi116.stderr:configure.in:36: the top level 2024-04-25T00:03:13.604 INFO:tasks.workunit.client.0.smithi116.stderr:configure.in:42: warning: The macro `AC_HEADER_TIME' is obsolete. 2024-04-25T00:03:13.604 INFO:tasks.workunit.client.0.smithi116.stderr:configure.in:42: You should run autoupdate. 2024-04-25T00:03:13.604 INFO:tasks.workunit.client.0.smithi116.stderr:./lib/autoconf/headers.m4:743: AC_HEADER_TIME is expanded from... 2024-04-25T00:03:13.604 INFO:tasks.workunit.client.0.smithi116.stderr:configure.in:42: the top level 2024-04-25T00:03:13.604 INFO:tasks.workunit.client.0.smithi116.stderr:configure.in:45: warning: The macro `AC_FUNC_SETVBUF_REVERSED' is obsolete. Remove it and all references to SETVBUF_REVERSED. 2024-04-25T00:03:13.604 INFO:tasks.workunit.client.0.smithi116.stderr:./lib/autoconf/functions.m4:1766: AC_FUNC_SETVBUF_REVERSED is expanded from... 2024-04-25T00:03:13.604 INFO:tasks.workunit.client.0.smithi116.stderr:configure.in:45: the top level 2024-04-25T00:03:13.604 INFO:tasks.workunit.client.0.smithi116.stderr:configure.in:50: warning: AC_OUTPUT should be used without arguments. 2024-04-25T00:03:13.604 INFO:tasks.workunit.client.0.smithi116.stderr:configure.in:50: You should run autoupdate. 2024-04-25T00:03:13.622 INFO:tasks.workunit.client.1.smithi174.stdout:checking for an ANSI C-conforming const... yes 2024-04-25T00:03:13.664 INFO:tasks.workunit.client.1.smithi174.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-04-25T00:03:13.666 INFO:tasks.workunit.client.1.smithi174.stdout:checking for function prototypes... yes 2024-04-25T00:03:13.671 INFO:tasks.workunit.client.0.smithi116.stdout:/bin/bash ./config.status --recheck 2024-04-25T00:03:13.723 INFO:tasks.workunit.client.1.smithi174.stdout:checking whether setvbuf arguments are reversed... no 2024-04-25T00:03:13.740 INFO:tasks.workunit.client.0.smithi116.stdout:running CONFIG_SHELL=/bin/bash /bin/bash ./configure --no-create --no-recursion 2024-04-25T00:03:13.820 INFO:tasks.workunit.client.1.smithi174.stdout:checking for system... yes 2024-04-25T00:03:13.879 INFO:tasks.workunit.client.0.smithi116.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-04-25T00:03:13.907 INFO:tasks.workunit.client.1.smithi174.stdout:checking for gettimeofday... yes 2024-04-25T00:03:14.003 INFO:tasks.workunit.client.1.smithi174.stdout:checking for mkdir... yes 2024-04-25T00:03:14.109 INFO:tasks.workunit.client.1.smithi174.stdout:checking for strerror... yes 2024-04-25T00:03:14.215 INFO:tasks.workunit.client.1.smithi174.stdout:checking for open64... yes 2024-04-25T00:03:14.342 INFO:tasks.workunit.client.1.smithi174.stdout:checking for stat64... yes 2024-04-25T00:03:14.477 INFO:tasks.workunit.client.1.smithi174.stdout:checking for fseeko64... yes 2024-04-25T00:03:14.601 INFO:tasks.workunit.client.1.smithi174.stdout:checking for lrand48_r... yes 2024-04-25T00:03:14.721 INFO:tasks.workunit.client.1.smithi174.stdout:checking for srand48_r... yes 2024-04-25T00:03:14.777 INFO:tasks.workunit.client.1.smithi174.stdout:configure: creating ./config.status 2024-04-25T00:03:14.891 INFO:tasks.workunit.client.0.smithi116.stdout:checking whether build environment is sane... yes 2024-04-25T00:03:14.904 INFO:tasks.workunit.client.0.smithi116.stdout:checking for a race-free mkdir -p... /usr/bin/mkdir -p 2024-04-25T00:03:14.905 INFO:tasks.workunit.client.0.smithi116.stdout:checking for gawk... gawk 2024-04-25T00:03:14.930 INFO:tasks.workunit.client.0.smithi116.stdout:checking whether make sets $(MAKE)... yes 2024-04-25T00:03:14.931 INFO:tasks.workunit.client.1.smithi174.stdout:config.status: creating Makefile 2024-04-25T00:03:14.950 INFO:tasks.workunit.client.1.smithi174.stdout:config.status: creating config.h 2024-04-25T00:03:14.975 INFO:tasks.workunit.client.1.smithi174.stdout:config.status: executing depfiles commands 2024-04-25T00:03:14.978 INFO:tasks.workunit.client.0.smithi116.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-04-25T00:03:14.978 INFO:tasks.workunit.client.0.smithi116.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-04-25T00:03:14.997 INFO:tasks.workunit.client.0.smithi116.stdout:checking for style of include used by make... GNU 2024-04-25T00:03:15.003 INFO:tasks.workunit.client.0.smithi116.stdout:checking for gcc... gcc 2024-04-25T00:03:15.049 INFO:tasks.workunit.client.1.smithi174.stdout:config.status: executing default commands 2024-04-25T00:03:15.120 INFO:tasks.workunit.client.1.smithi174.stderr:+ make 2024-04-25T00:03:15.147 INFO:tasks.workunit.client.1.smithi174.stdout:cd . && /bin/bash /home/ubuntu/cephtest/mnt.1/client.1/tmp/ffsb/missing --run aclocal-1.10 2024-04-25T00:03:15.154 INFO:tasks.workunit.client.1.smithi174.stderr:/home/ubuntu/cephtest/mnt.1/client.1/tmp/ffsb/missing: line 46: aclocal-1.10: command not found 2024-04-25T00:03:15.155 INFO:tasks.workunit.client.1.smithi174.stderr:WARNING: `aclocal-1.10' is missing on your system. You should only need it if 2024-04-25T00:03:15.155 INFO:tasks.workunit.client.1.smithi174.stderr: you modified `acinclude.m4' or `configure.in'. You might want 2024-04-25T00:03:15.155 INFO:tasks.workunit.client.1.smithi174.stderr: to install the `Automake' and `Perl' packages. Grab them from 2024-04-25T00:03:15.155 INFO:tasks.workunit.client.1.smithi174.stderr: any GNU archive site. 2024-04-25T00:03:15.158 INFO:tasks.workunit.client.0.smithi116.stdout:checking whether the C compiler works... yes 2024-04-25T00:03:15.158 INFO:tasks.workunit.client.0.smithi116.stdout:checking for C compiler default output file name... a.out 2024-04-25T00:03:15.159 INFO:tasks.workunit.client.1.smithi174.stdout: cd . && /bin/bash /home/ubuntu/cephtest/mnt.1/client.1/tmp/ffsb/missing --run automake-1.10 --gnu 2024-04-25T00:03:15.164 INFO:tasks.workunit.client.1.smithi174.stderr:/home/ubuntu/cephtest/mnt.1/client.1/tmp/ffsb/missing: line 46: automake-1.10: command not found 2024-04-25T00:03:15.165 INFO:tasks.workunit.client.1.smithi174.stderr:WARNING: `automake-1.10' is missing on your system. You should only need it if 2024-04-25T00:03:15.165 INFO:tasks.workunit.client.1.smithi174.stderr: you modified `Makefile.am', `acinclude.m4' or `configure.in'. 2024-04-25T00:03:15.165 INFO:tasks.workunit.client.1.smithi174.stderr: You might want to install the `Automake' and `Perl' packages. 2024-04-25T00:03:15.165 INFO:tasks.workunit.client.1.smithi174.stderr: Grab them from any GNU archive site. 2024-04-25T00:03:15.216 INFO:tasks.workunit.client.0.smithi116.stdout:checking for suffix of executables... 2024-04-25T00:03:15.279 INFO:tasks.workunit.client.0.smithi116.stdout:checking whether we are cross compiling... no 2024-04-25T00:03:15.347 INFO:tasks.workunit.client.0.smithi116.stdout:checking for suffix of object files... o 2024-04-25T00:03:15.347 INFO:tasks.workunit.client.1.smithi174.stdout:cd . && /bin/bash /home/ubuntu/cephtest/mnt.1/client.1/tmp/ffsb/missing --run autoconf 2024-04-25T00:03:15.377 INFO:tasks.workunit.client.0.smithi116.stdout:checking whether the compiler supports GNU C... yes 2024-04-25T00:03:15.409 INFO:tasks.workunit.client.0.smithi116.stdout:checking whether gcc accepts -g... yes 2024-04-25T00:03:15.491 INFO:tasks.workunit.client.0.smithi116.stdout:checking for gcc option to enable C11 features... none needed 2024-04-25T00:03:15.503 INFO:tasks.workunit.client.1.smithi174.stderr:aclocal.m4:16: warning: this file was generated for autoconf 2.61. 2024-04-25T00:03:15.503 INFO:tasks.workunit.client.1.smithi174.stderr:You have another version of autoconf. It may work, but is not guaranteed to. 2024-04-25T00:03:15.503 INFO:tasks.workunit.client.1.smithi174.stderr:If you have problems, you may need to regenerate the build system entirely. 2024-04-25T00:03:15.503 INFO:tasks.workunit.client.1.smithi174.stderr:To do so, use the procedure documented by the package, typically `autoreconf'. 2024-04-25T00:03:15.625 INFO:tasks.workunit.client.1.smithi174.stderr:configure.in:4: warning: The macro `AC_FOREACH' is obsolete. 2024-04-25T00:03:15.625 INFO:tasks.workunit.client.1.smithi174.stderr:configure.in:4: You should run autoupdate. 2024-04-25T00:03:15.625 INFO:tasks.workunit.client.1.smithi174.stderr:./lib/autoconf/general.m4:191: AC_FOREACH is expanded from... 2024-04-25T00:03:15.625 INFO:tasks.workunit.client.1.smithi174.stderr:aclocal.m4:684: _AM_SET_OPTIONS is expanded from... 2024-04-25T00:03:15.625 INFO:tasks.workunit.client.1.smithi174.stderr:aclocal.m4:410: AM_INIT_AUTOMAKE is expanded from... 2024-04-25T00:03:15.625 INFO:tasks.workunit.client.1.smithi174.stderr:configure.in:4: the top level 2024-04-25T00:03:15.625 INFO:tasks.workunit.client.1.smithi174.stderr:configure.in:6: warning: The macro `AM_CONFIG_HEADER' is obsolete. 2024-04-25T00:03:15.626 INFO:tasks.workunit.client.1.smithi174.stderr:configure.in:6: You should run autoupdate. 2024-04-25T00:03:15.626 INFO:tasks.workunit.client.1.smithi174.stderr:aclocal.m4:383: AM_CONFIG_HEADER is expanded from... 2024-04-25T00:03:15.626 INFO:tasks.workunit.client.1.smithi174.stderr:configure.in:6: the top level 2024-04-25T00:03:15.626 INFO:tasks.workunit.client.1.smithi174.stderr:configure.in:10: warning: The macro `AC_AIX' is obsolete. 2024-04-25T00:03:15.626 INFO:tasks.workunit.client.1.smithi174.stderr:configure.in:10: You should run autoupdate. 2024-04-25T00:03:15.626 INFO:tasks.workunit.client.1.smithi174.stderr:./lib/autoconf/specific.m4:540: AC_AIX is expanded from... 2024-04-25T00:03:15.626 INFO:tasks.workunit.client.1.smithi174.stderr:configure.in:10: the top level 2024-04-25T00:03:15.626 INFO:tasks.workunit.client.1.smithi174.stderr:configure.in:36: warning: The macro `AC_HEADER_STDC' is obsolete. 2024-04-25T00:03:15.626 INFO:tasks.workunit.client.1.smithi174.stderr:configure.in:36: You should run autoupdate. 2024-04-25T00:03:15.626 INFO:tasks.workunit.client.1.smithi174.stderr:./lib/autoconf/headers.m4:704: AC_HEADER_STDC is expanded from... 2024-04-25T00:03:15.626 INFO:tasks.workunit.client.1.smithi174.stderr:configure.in:36: the top level 2024-04-25T00:03:15.626 INFO:tasks.workunit.client.1.smithi174.stderr:configure.in:42: warning: The macro `AC_HEADER_TIME' is obsolete. 2024-04-25T00:03:15.627 INFO:tasks.workunit.client.1.smithi174.stderr:configure.in:42: You should run autoupdate. 2024-04-25T00:03:15.627 INFO:tasks.workunit.client.1.smithi174.stderr:./lib/autoconf/headers.m4:743: AC_HEADER_TIME is expanded from... 2024-04-25T00:03:15.627 INFO:tasks.workunit.client.1.smithi174.stderr:configure.in:42: the top level 2024-04-25T00:03:15.627 INFO:tasks.workunit.client.1.smithi174.stderr:configure.in:45: warning: The macro `AC_FUNC_SETVBUF_REVERSED' is obsolete. Remove it and all references to SETVBUF_REVERSED. 2024-04-25T00:03:15.627 INFO:tasks.workunit.client.1.smithi174.stderr:./lib/autoconf/functions.m4:1766: AC_FUNC_SETVBUF_REVERSED is expanded from... 2024-04-25T00:03:15.627 INFO:tasks.workunit.client.1.smithi174.stderr:configure.in:45: the top level 2024-04-25T00:03:15.627 INFO:tasks.workunit.client.1.smithi174.stderr:configure.in:50: warning: AC_OUTPUT should be used without arguments. 2024-04-25T00:03:15.627 INFO:tasks.workunit.client.1.smithi174.stderr:configure.in:50: You should run autoupdate. 2024-04-25T00:03:15.649 INFO:tasks.workunit.client.0.smithi116.stdout:checking dependency style of gcc... gcc3 2024-04-25T00:03:15.684 INFO:tasks.workunit.client.0.smithi116.stdout:checking for stdio.h... yes 2024-04-25T00:03:15.714 INFO:tasks.workunit.client.1.smithi174.stdout:/bin/bash ./config.status --recheck 2024-04-25T00:03:15.734 INFO:tasks.workunit.client.0.smithi116.stdout:checking for stdlib.h... yes 2024-04-25T00:03:15.775 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:03:15.778 INFO:tasks.workunit.client.1.smithi174.stdout:running CONFIG_SHELL=/bin/bash /bin/bash ./configure --no-create --no-recursion 2024-04-25T00:03:15.779 INFO:tasks.workunit.client.0.smithi116.stdout:checking for string.h... yes 2024-04-25T00:03:15.825 INFO:tasks.workunit.client.0.smithi116.stdout:checking for inttypes.h... yes 2024-04-25T00:03:15.869 INFO:tasks.workunit.client.0.smithi116.stdout:checking for stdint.h... yes 2024-04-25T00:03:15.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.892+0000 7fa3cdae4640 1 Processor -- start 2024-04-25T00:03:15.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.892+0000 7fa3cdae4640 1 -- start start 2024-04-25T00:03:15.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.896+0000 7fa3cdae4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3c80c2bc0 0x7fa3c80c3020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:15.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.896+0000 7fa3cdae4640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa3c815d1c0 0x7fa3c815d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:15.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.896+0000 7fa3cdae4640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3c815aba0 con 0x7fa3c8157190 2024-04-25T00:03:15.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.896+0000 7fa3cdae4640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fa3c815a710 con 0x7fa3c815d1c0 2024-04-25T00:03:15.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.896+0000 7fa3cdae4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa3c8159de0 con 0x7fa3c80c2bc0 2024-04-25T00:03:15.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.896+0000 7fa3c7fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa3c815d1c0 0x7fa3c815d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:15.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.896+0000 7fa3c7fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa3c815d1c0 0x7fa3c815d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59210/0 (socket says 172.21.15.116:59210) 2024-04-25T00:03:15.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.896+0000 7fa3c7fff640 1 -- 172.21.15.116:0/3577062413 learned_addr learned my addr 172.21.15.116:0/3577062413 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:03:15.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.896+0000 7fa3c6ffd640 1 --2- 172.21.15.116:0/3577062413 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3c80c2bc0 0x7fa3c80c3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:15.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.896+0000 7fa3c7fff640 1 -- 172.21.15.116:0/3577062413 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3c80c2bc0 msgr2=0x7fa3c80c3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:03:15.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.896+0000 7fa3c7fff640 1 --2- 172.21.15.116:0/3577062413 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3c80c2bc0 0x7fa3c80c3020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:15.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.896+0000 7fa3c7fff640 1 -- 172.21.15.116:0/3577062413 >> v1:172.21.15.174:6789/0 conn(0x7fa3c8157190 legacy=0x7fa3c8072b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:15.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.896+0000 7fa3c7fff640 1 -- 172.21.15.116:0/3577062413 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3c815a0c0 con 0x7fa3c815d1c0 2024-04-25T00:03:15.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.896+0000 7fa3c7fff640 1 --2- 172.21.15.116:0/3577062413 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3c815d1c0 0x7fa3c815d630 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fa3c0009e50 tx=0x7fa3c001c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=760b54ee6bbe59c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:15.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.896+0000 7fa3c67fc640 1 -- 172.21.15.116:0/3577062413 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c00a55d0 con 0x7fa3c815d1c0 2024-04-25T00:03:15.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.896+0000 7fa3c67fc640 1 -- 172.21.15.116:0/3577062413 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3c00a5770 con 0x7fa3c815d1c0 2024-04-25T00:03:15.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.896+0000 7fa3c67fc640 1 -- 172.21.15.116:0/3577062413 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c00a5b40 con 0x7fa3c815d1c0 2024-04-25T00:03:15.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.900+0000 7fa3cdae4640 1 -- 172.21.15.116:0/3577062413 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3c815d1c0 msgr2=0x7fa3c815d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:15.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.900+0000 7fa3cdae4640 1 --2- 172.21.15.116:0/3577062413 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3c815d1c0 0x7fa3c815d630 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fa3c0009e50 tx=0x7fa3c001c910 comp rx=0 tx=0).stop 2024-04-25T00:03:15.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.900+0000 7fa3cdae4640 1 -- 172.21.15.116:0/3577062413 shutdown_connections 2024-04-25T00:03:15.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.900+0000 7fa3cdae4640 1 --2- 172.21.15.116:0/3577062413 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3c815d1c0 0x7fa3c815d630 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:15.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.900+0000 7fa3cdae4640 1 --2- 172.21.15.116:0/3577062413 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3c80c2bc0 0x7fa3c80c3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:15.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.900+0000 7fa3cdae4640 1 -- 172.21.15.116:0/3577062413 >> 172.21.15.116:0/3577062413 conn(0x7fa3c806c720 msgr2=0x7fa3c806cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:03:15.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.900+0000 7fa3cdae4640 1 -- 172.21.15.116:0/3577062413 shutdown_connections 2024-04-25T00:03:15.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.900+0000 7fa3cdae4640 1 -- 172.21.15.116:0/3577062413 wait complete. 2024-04-25T00:03:15.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.900+0000 7fa3cdae4640 1 Processor -- start 2024-04-25T00:03:15.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.900+0000 7fa3cdae4640 1 -- start start 2024-04-25T00:03:15.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.900+0000 7fa3cdae4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3c80c2bc0 0x7fa3c814c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:15.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.900+0000 7fa3cdae4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3c8157190 0x7fa3c8155810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:15.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.900+0000 7fa3cdae4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3c815d1c0 0x7fa3c80c5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:15.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.900+0000 7fa3cdae4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa3c8073420 con 0x7fa3c8157190 2024-04-25T00:03:15.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.900+0000 7fa3cdae4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa3c80732a0 con 0x7fa3c80c2bc0 2024-04-25T00:03:15.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.900+0000 7fa3cdae4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa3c815db70 con 0x7fa3c815d1c0 2024-04-25T00:03:15.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.900+0000 7fa3c6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3c8157190 0x7fa3c8155810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:15.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.900+0000 7fa3c6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3c8157190 0x7fa3c8155810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59226/0 (socket says 172.21.15.116:59226) 2024-04-25T00:03:15.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.900+0000 7fa3c77fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3c80c2bc0 0x7fa3c814c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:15.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.900+0000 7fa3c6ffd640 1 -- 172.21.15.116:0/137839948 learned_addr learned my addr 172.21.15.116:0/137839948 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:03:15.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.900+0000 7fa3c7fff640 1 --2- 172.21.15.116:0/137839948 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3c815d1c0 0x7fa3c80c5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:15.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.900+0000 7fa3c6ffd640 1 -- 172.21.15.116:0/137839948 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3c815d1c0 msgr2=0x7fa3c80c5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:15.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.900+0000 7fa3c6ffd640 1 --2- 172.21.15.116:0/137839948 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3c815d1c0 0x7fa3c80c5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:15.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.900+0000 7fa3c6ffd640 1 -- 172.21.15.116:0/137839948 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3c80c2bc0 msgr2=0x7fa3c814c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:15.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.900+0000 7fa3c6ffd640 1 --2- 172.21.15.116:0/137839948 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3c80c2bc0 0x7fa3c814c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:15.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.900+0000 7fa3c6ffd640 1 -- 172.21.15.116:0/137839948 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3c81562f0 con 0x7fa3c8157190 2024-04-25T00:03:15.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.900+0000 7fa3c7fff640 1 --2- 172.21.15.116:0/137839948 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3c815d1c0 0x7fa3c80c5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:03:15.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.900+0000 7fa3c77fe640 1 --2- 172.21.15.116:0/137839948 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3c80c2bc0 0x7fa3c814c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:03:15.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.900+0000 7fa3c6ffd640 1 --2- 172.21.15.116:0/137839948 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3c8157190 0x7fa3c8155810 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fa3bc00d830 tx=0x7fa3bc00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:15.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.904+0000 7fa3affff640 1 -- 172.21.15.116:0/137839948 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3bc00b6c0 con 0x7fa3c8157190 2024-04-25T00:03:15.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.904+0000 7fa3affff640 1 -- 172.21.15.116:0/137839948 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3bc019070 con 0x7fa3c8157190 2024-04-25T00:03:15.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.904+0000 7fa3affff640 1 -- 172.21.15.116:0/137839948 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3bc00bae0 con 0x7fa3c8157190 2024-04-25T00:03:15.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.904+0000 7fa3cdae4640 1 -- 172.21.15.116:0/137839948 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3c81567c0 con 0x7fa3c8157190 2024-04-25T00:03:15.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.904+0000 7fa3cdae4640 1 -- 172.21.15.116:0/137839948 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3c814c710 con 0x7fa3c8157190 2024-04-25T00:03:15.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.904+0000 7fa3cdae4640 1 -- 172.21.15.116:0/137839948 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3c8068a60 con 0x7fa3c8157190 2024-04-25T00:03:15.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.904+0000 7fa3affff640 1 -- 172.21.15.116:0/137839948 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa3bc01e460 con 0x7fa3c8157190 2024-04-25T00:03:15.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.904+0000 7fa3affff640 1 --2- 172.21.15.116:0/137839948 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa3b40678c0 0x7fa3b4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:15.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.904+0000 7fa3affff640 1 -- 172.21.15.116:0/137839948 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6752+0+0 (secure 0 0 0) 0x7fa3bc086650 con 0x7fa3c8157190 2024-04-25T00:03:15.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.904+0000 7fa3c77fe640 1 --2- 172.21.15.116:0/137839948 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa3b40678c0 0x7fa3b4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:15.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.904+0000 7fa3c77fe640 1 --2- 172.21.15.116:0/137839948 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa3b40678c0 0x7fa3b4069d80 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa3b8007fd0 tx=0x7fa3b8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:15.913 INFO:tasks.workunit.client.0.smithi116.stdout:checking for strings.h... yes 2024-04-25T00:03:15.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:15.908+0000 7fa3affff640 1 -- 172.21.15.116:0/137839948 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fa3bc056740 con 0x7fa3c8157190 2024-04-25T00:03:15.928 INFO:tasks.workunit.client.1.smithi174.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-04-25T00:03:15.957 INFO:tasks.workunit.client.0.smithi116.stdout:checking for sys/stat.h... yes 2024-04-25T00:03:16.002 INFO:tasks.workunit.client.0.smithi116.stdout:checking for sys/types.h... yes 2024-04-25T00:03:16.055 INFO:tasks.workunit.client.0.smithi116.stdout:checking for unistd.h... yes 2024-04-25T00:03:16.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.116+0000 7fa3cdae4640 1 -- 172.21.15.116:0/137839948 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa3c814bb10 con 0x7fa3c8157190 2024-04-25T00:03:16.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.116+0000 7fa3affff640 1 -- 172.21.15.116:0/137839948 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6498 (secure 0 0 0) 0x7fa3bc05aa40 con 0x7fa3c8157190 2024-04-25T00:03:16.120 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:03:16.120 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4735,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/2004876235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":2004876235},{"type":"v1","addr":"172.21.15.116:6835","nonce":2004876235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:01:08.748957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6837/1762356055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1762356055},{"type":"v1","addr":"172.21.15.174:6837","nonce":1762356055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270]},"id":1},{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:02:43.411054+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4242":{"gid":4242,"name":"c","rank":0,"incarnation":15,"state":"up:active","state_seq":29,"addr":"172.21.15.116:6834/1459177921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1459177921},{"type":"v1","addr":"172.21.15.116:6834","nonce":1459177921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4242,"qdb_cluster":[4242]},"id":2}]} 2024-04-25T00:03:16.120 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 19 2024-04-25T00:03:16.121 INFO:tasks.workunit.client.0.smithi116.stdout:checking for wchar.h... yes 2024-04-25T00:03:16.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.116+0000 7fa3cdae4640 1 -- 172.21.15.116:0/137839948 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa3b40678c0 msgr2=0x7fa3b4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:16.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.116+0000 7fa3cdae4640 1 --2- 172.21.15.116:0/137839948 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa3b40678c0 0x7fa3b4069d80 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa3b8007fd0 tx=0x7fa3b8009040 comp rx=0 tx=0).stop 2024-04-25T00:03:16.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.116+0000 7fa3cdae4640 1 -- 172.21.15.116:0/137839948 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3c8157190 msgr2=0x7fa3c8155810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:16.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.116+0000 7fa3cdae4640 1 --2- 172.21.15.116:0/137839948 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3c8157190 0x7fa3c8155810 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fa3bc00d830 tx=0x7fa3bc00dcf0 comp rx=0 tx=0).stop 2024-04-25T00:03:16.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.116+0000 7fa3cdae4640 1 -- 172.21.15.116:0/137839948 shutdown_connections 2024-04-25T00:03:16.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.116+0000 7fa3cdae4640 1 --2- 172.21.15.116:0/137839948 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa3b40678c0 0x7fa3b4069d80 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:16.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.116+0000 7fa3cdae4640 1 --2- 172.21.15.116:0/137839948 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3c815d1c0 0x7fa3c80c5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:16.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.116+0000 7fa3cdae4640 1 --2- 172.21.15.116:0/137839948 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3c8157190 0x7fa3c8155810 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:16.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.116+0000 7fa3cdae4640 1 --2- 172.21.15.116:0/137839948 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3c80c2bc0 0x7fa3c814c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:16.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.116+0000 7fa3cdae4640 1 -- 172.21.15.116:0/137839948 >> 172.21.15.116:0/137839948 conn(0x7fa3c806c720 msgr2=0x7fa3c80a9e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:03:16.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.116+0000 7fa3cdae4640 1 -- 172.21.15.116:0/137839948 shutdown_connections 2024-04-25T00:03:16.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.116+0000 7fa3cdae4640 1 -- 172.21.15.116:0/137839948 wait complete. 2024-04-25T00:03:16.138 INFO:tasks.mds_thrash.fs.[b]:kill mds.c (rank=0) 2024-04-25T00:03:16.138 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-25T00:03:16.138 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:03:16.192 INFO:tasks.ceph.mds.c:Stopped 2024-04-25T00:03:16.192 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:03:16.193 INFO:tasks.workunit.client.0.smithi116.stdout:checking for minix/config.h... no 2024-04-25T00:03:16.244 INFO:tasks.workunit.client.0.smithi116.stdout:checking for sys/time.h... yes 2024-04-25T00:03:16.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.292+0000 7ff67668e640 1 Processor -- start 2024-04-25T00:03:16.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.292+0000 7ff67668e640 1 -- start start 2024-04-25T00:03:16.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.292+0000 7ff67668e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff67014c700 0x7ff6700c5090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:16.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.292+0000 7ff67668e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff670155b20 0x7ff670152c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:16.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.292+0000 7ff67668e640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6700a9e60 con 0x7ff6700c55d0 2024-04-25T00:03:16.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.292+0000 7ff67668e640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7ff6701527b0 con 0x7ff67014c700 2024-04-25T00:03:16.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.292+0000 7ff67668e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff6700c3ef0 con 0x7ff670155b20 2024-04-25T00:03:16.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.292+0000 7ff66ffff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff67014c700 0x7ff6700c5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:16.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.292+0000 7ff66ffff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff67014c700 0x7ff6700c5090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59232/0 (socket says 172.21.15.116:59232) 2024-04-25T00:03:16.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.292+0000 7ff66ffff640 1 -- 172.21.15.116:0/2548905019 learned_addr learned my addr 172.21.15.116:0/2548905019 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:03:16.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.292+0000 7ff66f7fe640 1 --2- 172.21.15.116:0/2548905019 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff670155b20 0x7ff670152c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:16.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.296+0000 7ff66ffff640 1 -- 172.21.15.116:0/2548905019 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff670155b20 msgr2=0x7ff670152c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:03:16.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.296+0000 7ff66ffff640 1 --2- 172.21.15.116:0/2548905019 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff670155b20 0x7ff670152c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:16.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.296+0000 7ff66ffff640 1 -- 172.21.15.116:0/2548905019 >> v1:172.21.15.174:6789/0 conn(0x7ff6700c55d0 legacy=0x7ff670071df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:03:16.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.296+0000 7ff66ffff640 1 -- 172.21.15.116:0/2548905019 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6700c41d0 con 0x7ff67014c700 2024-04-25T00:03:16.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.296+0000 7ff66ffff640 1 --2- 172.21.15.116:0/2548905019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff67014c700 0x7ff6700c5090 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7ff670152930 tx=0x7ff66001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d914e3e370aab96c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:16.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.296+0000 7ff66effd640 1 -- 172.21.15.116:0/2548905019 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6600b0070 con 0x7ff67014c700 2024-04-25T00:03:16.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.296+0000 7ff66effd640 1 -- 172.21.15.116:0/2548905019 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6600ab600 con 0x7ff67014c700 2024-04-25T00:03:16.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.296+0000 7ff66effd640 1 -- 172.21.15.116:0/2548905019 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6600ab9d0 con 0x7ff67014c700 2024-04-25T00:03:16.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.296+0000 7ff67668e640 1 -- 172.21.15.116:0/2548905019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff67014c700 msgr2=0x7ff6700c5090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:16.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.296+0000 7ff67668e640 1 --2- 172.21.15.116:0/2548905019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff67014c700 0x7ff6700c5090 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7ff670152930 tx=0x7ff66001c3e0 comp rx=0 tx=0).stop 2024-04-25T00:03:16.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.296+0000 7ff67668e640 1 -- 172.21.15.116:0/2548905019 shutdown_connections 2024-04-25T00:03:16.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.296+0000 7ff67668e640 1 --2- 172.21.15.116:0/2548905019 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff670155b20 0x7ff670152c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:16.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.296+0000 7ff67668e640 1 --2- 172.21.15.116:0/2548905019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff67014c700 0x7ff6700c5090 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:16.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.296+0000 7ff67668e640 1 -- 172.21.15.116:0/2548905019 >> 172.21.15.116:0/2548905019 conn(0x7ff67006c720 msgr2=0x7ff67006cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:03:16.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.296+0000 7ff67668e640 1 -- 172.21.15.116:0/2548905019 shutdown_connections 2024-04-25T00:03:16.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.296+0000 7ff67668e640 1 -- 172.21.15.116:0/2548905019 wait complete. 2024-04-25T00:03:16.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.300+0000 7ff67668e640 1 Processor -- start 2024-04-25T00:03:16.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.300+0000 7ff67668e640 1 -- start start 2024-04-25T00:03:16.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.300+0000 7ff67668e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff6700c55d0 0x7ff670150600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:16.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.300+0000 7ff67668e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff67014c700 0x7ff670150b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:16.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.300+0000 7ff67668e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff670155b20 0x7ff6701448f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:16.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.300+0000 7ff67668e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff670153510 con 0x7ff67014c700 2024-04-25T00:03:16.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.300+0000 7ff67668e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff670153390 con 0x7ff670155b20 2024-04-25T00:03:16.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.300+0000 7ff67668e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff67015fed0 con 0x7ff6700c55d0 2024-04-25T00:03:16.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.300+0000 7ff66f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff67014c700 0x7ff670150b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:16.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.300+0000 7ff66f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff67014c700 0x7ff670150b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59242/0 (socket says 172.21.15.116:59242) 2024-04-25T00:03:16.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.300+0000 7ff66f7fe640 1 -- 172.21.15.116:0/3709580090 learned_addr learned my addr 172.21.15.116:0/3709580090 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:03:16.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.300+0000 7ff66f7fe640 1 -- 172.21.15.116:0/3709580090 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff6700c55d0 msgr2=0x7ff670150600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:16.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.300+0000 7ff66f7fe640 1 --2- 172.21.15.116:0/3709580090 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff6700c55d0 0x7ff670150600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:16.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.300+0000 7ff66f7fe640 1 -- 172.21.15.116:0/3709580090 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff670155b20 msgr2=0x7ff6701448f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:16.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.300+0000 7ff66f7fe640 1 --2- 172.21.15.116:0/3709580090 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff670155b20 0x7ff6701448f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:16.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.300+0000 7ff66f7fe640 1 -- 172.21.15.116:0/3709580090 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff670146250 con 0x7ff67014c700 2024-04-25T00:03:16.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.300+0000 7ff66f7fe640 1 --2- 172.21.15.116:0/3709580090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff67014c700 0x7ff670150b40 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7ff66400c820 tx=0x7ff66400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:16.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.300+0000 7ff66cff9640 1 -- 172.21.15.116:0/3709580090 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff664015070 con 0x7ff67014c700 2024-04-25T00:03:16.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.300+0000 7ff66cff9640 1 -- 172.21.15.116:0/3709580090 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff664004220 con 0x7ff67014c700 2024-04-25T00:03:16.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.300+0000 7ff67668e640 1 -- 172.21.15.116:0/3709580090 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6701efc30 con 0x7ff67014c700 2024-04-25T00:03:16.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.300+0000 7ff66cff9640 1 -- 172.21.15.116:0/3709580090 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff66400ce10 con 0x7ff67014c700 2024-04-25T00:03:16.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.300+0000 7ff67668e640 1 -- 172.21.15.116:0/3709580090 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff670143a90 con 0x7ff67014c700 2024-04-25T00:03:16.306 INFO:tasks.workunit.client.0.smithi116.stdout:checking whether it is safe to define __EXTENSIONS__... yes 2024-04-25T00:03:16.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.300+0000 7ff67668e640 1 -- 172.21.15.116:0/3709580090 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff670068a60 con 0x7ff67014c700 2024-04-25T00:03:16.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.300+0000 7ff66cff9640 1 -- 172.21.15.116:0/3709580090 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff664007830 con 0x7ff67014c700 2024-04-25T00:03:16.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.300+0000 7ff66cff9640 1 --2- 172.21.15.116:0/3709580090 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff65c0678c0 0x7ff65c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:16.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.300+0000 7ff66cff9640 1 -- 172.21.15.116:0/3709580090 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6752+0+0 (secure 0 0 0) 0x7ff664085e50 con 0x7ff67014c700 2024-04-25T00:03:16.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.300+0000 7ff66ffff640 1 --2- 172.21.15.116:0/3709580090 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff65c0678c0 0x7ff65c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:16.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.304+0000 7ff66ffff640 1 --2- 172.21.15.116:0/3709580090 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff65c0678c0 0x7ff65c069d80 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff6600ad000 tx=0x7ff660008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:16.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.304+0000 7ff66cff9640 1 -- 172.21.15.116:0/3709580090 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ff664056610 con 0x7ff67014c700 2024-04-25T00:03:16.350 INFO:tasks.workunit.client.0.smithi116.stdout:checking whether _XOPEN_SOURCE should be defined... no 2024-04-25T00:03:16.361 INFO:tasks.workunit.client.0.smithi116.stdout:checking for gcc... (cached) gcc 2024-04-25T00:03:16.462 INFO:tasks.workunit.client.0.smithi116.stdout:checking whether the compiler supports GNU C... (cached) yes 2024-04-25T00:03:16.462 INFO:tasks.workunit.client.0.smithi116.stdout:checking whether gcc accepts -g... (cached) yes 2024-04-25T00:03:16.462 INFO:tasks.workunit.client.0.smithi116.stdout:checking for gcc option to enable C11 features... (cached) none needed 2024-04-25T00:03:16.463 INFO:tasks.workunit.client.0.smithi116.stdout:checking dependency style of gcc... (cached) gcc3 2024-04-25T00:03:16.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.516+0000 7ff67668e640 1 -- 172.21.15.116:0/3709580090 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff6700759d0 con 0x7ff67014c700 2024-04-25T00:03:16.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.516+0000 7ff66cff9640 1 -- 172.21.15.116:0/3709580090 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6498 (secure 0 0 0) 0x7ff664055f10 con 0x7ff67014c700 2024-04-25T00:03:16.522 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:03:16.522 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4735,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/2004876235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":2004876235},{"type":"v1","addr":"172.21.15.116:6835","nonce":2004876235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:01:08.748957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6837/1762356055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1762356055},{"type":"v1","addr":"172.21.15.174:6837","nonce":1762356055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270]},"id":1},{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:02:43.411054+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4242":{"gid":4242,"name":"c","rank":0,"incarnation":15,"state":"up:active","state_seq":29,"addr":"172.21.15.116:6834/1459177921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1459177921},{"type":"v1","addr":"172.21.15.116:6834","nonce":1459177921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4242,"qdb_cluster":[4242]},"id":2}]} 2024-04-25T00:03:16.523 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 19 2024-04-25T00:03:16.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.520+0000 7ff67668e640 1 -- 172.21.15.116:0/3709580090 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff65c0678c0 msgr2=0x7ff65c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:16.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.520+0000 7ff67668e640 1 --2- 172.21.15.116:0/3709580090 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff65c0678c0 0x7ff65c069d80 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff6600ad000 tx=0x7ff660008040 comp rx=0 tx=0).stop 2024-04-25T00:03:16.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.520+0000 7ff67668e640 1 -- 172.21.15.116:0/3709580090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff67014c700 msgr2=0x7ff670150b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:16.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.520+0000 7ff67668e640 1 --2- 172.21.15.116:0/3709580090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff67014c700 0x7ff670150b40 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7ff66400c820 tx=0x7ff66400cce0 comp rx=0 tx=0).stop 2024-04-25T00:03:16.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.520+0000 7ff67668e640 1 -- 172.21.15.116:0/3709580090 shutdown_connections 2024-04-25T00:03:16.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.520+0000 7ff67668e640 1 --2- 172.21.15.116:0/3709580090 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff65c0678c0 0x7ff65c069d80 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:16.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.520+0000 7ff67668e640 1 --2- 172.21.15.116:0/3709580090 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff670155b20 0x7ff6701448f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:16.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.520+0000 7ff67668e640 1 --2- 172.21.15.116:0/3709580090 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff67014c700 0x7ff670150b40 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:16.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.520+0000 7ff67668e640 1 --2- 172.21.15.116:0/3709580090 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff6700c55d0 0x7ff670150600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:16.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.520+0000 7ff67668e640 1 -- 172.21.15.116:0/3709580090 >> 172.21.15.116:0/3709580090 conn(0x7ff67006c720 msgr2=0x7ff670071df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:03:16.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.520+0000 7ff67668e640 1 -- 172.21.15.116:0/3709580090 shutdown_connections 2024-04-25T00:03:16.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:16.520+0000 7ff67668e640 1 -- 172.21.15.116:0/3709580090 wait complete. 2024-04-25T00:03:16.534 INFO:tasks.workunit.client.0.smithi116.stdout:checking for main in -lm... yes 2024-04-25T00:03:16.539 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:03:16.611 INFO:tasks.workunit.client.0.smithi116.stdout:checking for main in -lpthread... yes 2024-04-25T00:03:16.615 INFO:tasks.workunit.client.0.smithi116.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-04-25T00:03:16.617 INFO:tasks.workunit.client.0.smithi116.stdout:checking for egrep... /usr/bin/grep -E 2024-04-25T00:03:16.678 INFO:tasks.workunit.client.0.smithi116.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2024-04-25T00:03:16.737 INFO:tasks.workunit.client.0.smithi116.stdout:checking for pthread.h... yes 2024-04-25T00:03:16.795 INFO:tasks.workunit.client.0.smithi116.stdout:checking for fcntl.h... yes 2024-04-25T00:03:16.874 INFO:tasks.workunit.client.0.smithi116.stdout:checking for limits.h... yes 2024-04-25T00:03:16.877 INFO:tasks.workunit.client.0.smithi116.stdout:checking for stdint.h... (cached) yes 2024-04-25T00:03:16.878 INFO:tasks.workunit.client.0.smithi116.stdout:checking for sys/time.h... (cached) yes 2024-04-25T00:03:16.879 INFO:tasks.workunit.client.0.smithi116.stdout:checking for unistd.h... (cached) yes 2024-04-25T00:03:16.937 INFO:tasks.workunit.client.0.smithi116.stdout:checking for sys/vfs.h... yes 2024-04-25T00:03:17.022 INFO:tasks.workunit.client.0.smithi116.stdout:checking for sys/limits.h... no 2024-04-25T00:03:17.066 INFO:tasks.workunit.client.0.smithi116.stdout:checking for an ANSI C-conforming const... yes 2024-04-25T00:03:17.156 INFO:tasks.workunit.client.0.smithi116.stdout:checking for system... yes 2024-04-25T00:03:17.238 INFO:tasks.workunit.client.0.smithi116.stdout:checking for gettimeofday... yes 2024-04-25T00:03:17.321 INFO:tasks.workunit.client.0.smithi116.stdout:checking for mkdir... yes 2024-04-25T00:03:17.401 INFO:tasks.workunit.client.0.smithi116.stdout:checking for strerror... yes 2024-04-25T00:03:17.479 INFO:tasks.workunit.client.0.smithi116.stdout:checking for open64... yes 2024-04-25T00:03:17.559 INFO:tasks.workunit.client.0.smithi116.stdout:checking for stat64... yes 2024-04-25T00:03:17.632 INFO:tasks.workunit.client.0.smithi116.stdout:checking for fseeko64... yes 2024-04-25T00:03:17.716 INFO:tasks.workunit.client.0.smithi116.stdout:checking for lrand48_r... yes 2024-04-25T00:03:17.794 INFO:tasks.workunit.client.0.smithi116.stdout:checking for srand48_r... yes 2024-04-25T00:03:17.819 INFO:tasks.workunit.client.0.smithi116.stdout:configure: creating ./config.status 2024-04-25T00:03:17.978 INFO:tasks.workunit.client.0.smithi116.stdout: /bin/bash ./config.status 2024-04-25T00:03:18.068 INFO:tasks.workunit.client.0.smithi116.stdout:config.status: creating Makefile 2024-04-25T00:03:18.088 INFO:tasks.workunit.client.0.smithi116.stdout:config.status: creating config.h 2024-04-25T00:03:18.135 INFO:tasks.workunit.client.0.smithi116.stdout:config.status: executing depfiles commands 2024-04-25T00:03:18.153 INFO:tasks.workunit.client.0.smithi116.stdout:config.status: executing default commands 2024-04-25T00:03:18.258 INFO:tasks.workunit.client.0.smithi116.stdout:cd . && /bin/bash /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoheader 2024-04-25T00:03:18.326 INFO:tasks.workunit.client.0.smithi116.stderr:autoheader: warning: autoconf input should be named 'configure.ac', not 'configure.in' 2024-04-25T00:03:18.416 INFO:tasks.workunit.client.0.smithi116.stderr:aclocal.m4:16: warning: this file was generated for autoconf 2.61. 2024-04-25T00:03:18.416 INFO:tasks.workunit.client.0.smithi116.stderr:You have another version of autoconf. It may work, but is not guaranteed to. 2024-04-25T00:03:18.416 INFO:tasks.workunit.client.0.smithi116.stderr:If you have problems, you may need to regenerate the build system entirely. 2024-04-25T00:03:18.416 INFO:tasks.workunit.client.0.smithi116.stderr:To do so, use the procedure documented by the package, typically `autoreconf'. 2024-04-25T00:03:18.540 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:03:18.598 INFO:tasks.workunit.client.0.smithi116.stdout:rm -f stamp-h1 2024-04-25T00:03:18.605 INFO:tasks.workunit.client.0.smithi116.stdout:touch config.h.in 2024-04-25T00:03:18.612 INFO:tasks.workunit.client.0.smithi116.stdout:cd . && /bin/bash ./config.status config.h 2024-04-25T00:03:18.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.648+0000 7f9036d8d640 1 Processor -- start 2024-04-25T00:03:18.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.648+0000 7f9036d8d640 1 -- start start 2024-04-25T00:03:18.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.648+0000 7f9036d8d640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f9030150150 0x7f9030072bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:18.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.648+0000 7f9036d8d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f903006c220 0x7f903006c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:18.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.648+0000 7f9036d8d640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9030152900 con 0x7f90300c1c10 2024-04-25T00:03:18.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.648+0000 7f9036d8d640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f90300c2880 con 0x7f9030150150 2024-04-25T00:03:18.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.648+0000 7f9036d8d640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f9030072740 con 0x7f903006c220 2024-04-25T00:03:18.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.648+0000 7f903532a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f903006c220 0x7f903006c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:18.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.648+0000 7f9034b29640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f9030150150 0x7f9030072bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:18.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.648+0000 7f903532a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f903006c220 0x7f903006c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:41558/0 (socket says 172.21.15.116:41558) 2024-04-25T00:03:18.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.648+0000 7f903532a640 1 -- 172.21.15.116:0/367749828 learned_addr learned my addr 172.21.15.116:0/367749828 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:03:18.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.648+0000 7f9034b29640 1 -- 172.21.15.116:0/367749828 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f903006c220 msgr2=0x7f903006c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:03:18.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.648+0000 7f9034b29640 1 --2- 172.21.15.116:0/367749828 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f903006c220 0x7f903006c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:18.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.648+0000 7f9034b29640 1 -- 172.21.15.116:0/367749828 >> v1:172.21.15.174:6789/0 conn(0x7f90300c1c10 legacy=0x7f90300c2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:18.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.648+0000 7f9034b29640 1 -- 172.21.15.116:0/367749828 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9030071e10 con 0x7f9030150150 2024-04-25T00:03:18.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.648+0000 7f903532a640 1 --2- 172.21.15.116:0/367749828 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f903006c220 0x7f903006c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:03:18.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.652+0000 7f9034b29640 1 --2- 172.21.15.116:0/367749828 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9030150150 0x7f9030072bd0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f90200097c0 tx=0x7f902001c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2dc0d251e55d73ef server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:18.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.652+0000 7f902f7fe640 1 -- 172.21.15.116:0/367749828 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90200b2070 con 0x7f9030150150 2024-04-25T00:03:18.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.652+0000 7f902f7fe640 1 -- 172.21.15.116:0/367749828 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90200ad640 con 0x7f9030150150 2024-04-25T00:03:18.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.652+0000 7f902f7fe640 1 -- 172.21.15.116:0/367749828 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90200ada10 con 0x7f9030150150 2024-04-25T00:03:18.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.652+0000 7f9036d8d640 1 -- 172.21.15.116:0/367749828 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9030150150 msgr2=0x7f9030072bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:18.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.652+0000 7f9036d8d640 1 --2- 172.21.15.116:0/367749828 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9030150150 0x7f9030072bd0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f90200097c0 tx=0x7f902001c4a0 comp rx=0 tx=0).stop 2024-04-25T00:03:18.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.652+0000 7f9036d8d640 1 -- 172.21.15.116:0/367749828 shutdown_connections 2024-04-25T00:03:18.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.652+0000 7f9036d8d640 1 --2- 172.21.15.116:0/367749828 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f903006c220 0x7f903006c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:18.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.652+0000 7f9036d8d640 1 --2- 172.21.15.116:0/367749828 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9030150150 0x7f9030072bd0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:18.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.652+0000 7f9036d8d640 1 -- 172.21.15.116:0/367749828 >> 172.21.15.116:0/367749828 conn(0x7f90300a9b20 msgr2=0x7f90300c4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:03:18.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.652+0000 7f9036d8d640 1 -- 172.21.15.116:0/367749828 shutdown_connections 2024-04-25T00:03:18.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.652+0000 7f9036d8d640 1 -- 172.21.15.116:0/367749828 wait complete. 2024-04-25T00:03:18.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.656+0000 7f9036d8d640 1 Processor -- start 2024-04-25T00:03:18.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.656+0000 7f9036d8d640 1 -- start start 2024-04-25T00:03:18.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.656+0000 7f9036d8d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f903006c220 0x7f9030148e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:18.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.656+0000 7f9036d8d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90300c1c10 0x7f9030146a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:18.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.656+0000 7f9036d8d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9030150150 0x7f9030146f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:18.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.656+0000 7f9036d8d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f90300c3f70 con 0x7f90300c1c10 2024-04-25T00:03:18.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.656+0000 7f9036d8d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f90300c50e0 con 0x7f903006c220 2024-04-25T00:03:18.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.656+0000 7f9036d8d640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f90300c40f0 con 0x7f9030150150 2024-04-25T00:03:18.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.656+0000 7f902ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90300c1c10 0x7f9030146a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:18.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.656+0000 7f902ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90300c1c10 0x7f9030146a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59256/0 (socket says 172.21.15.116:59256) 2024-04-25T00:03:18.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.656+0000 7f902ffff640 1 -- 172.21.15.116:0/1015029958 learned_addr learned my addr 172.21.15.116:0/1015029958 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:03:18.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.656+0000 7f903532a640 1 --2- 172.21.15.116:0/1015029958 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9030150150 0x7f9030146f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:18.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.656+0000 7f9034b29640 1 --2- 172.21.15.116:0/1015029958 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f903006c220 0x7f9030148e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:18.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.656+0000 7f902ffff640 1 -- 172.21.15.116:0/1015029958 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9030150150 msgr2=0x7f9030146f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:18.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.656+0000 7f902ffff640 1 --2- 172.21.15.116:0/1015029958 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9030150150 0x7f9030146f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:18.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.656+0000 7f902ffff640 1 -- 172.21.15.116:0/1015029958 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f903006c220 msgr2=0x7f9030148e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:18.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.656+0000 7f902ffff640 1 --2- 172.21.15.116:0/1015029958 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f903006c220 0x7f9030148e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:18.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.656+0000 7f902ffff640 1 -- 172.21.15.116:0/1015029958 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90301eecb0 con 0x7f90300c1c10 2024-04-25T00:03:18.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.656+0000 7f903532a640 1 --2- 172.21.15.116:0/1015029958 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9030150150 0x7f9030146f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:03:18.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.656+0000 7f9034b29640 1 --2- 172.21.15.116:0/1015029958 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f903006c220 0x7f9030148e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:03:18.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.656+0000 7f902ffff640 1 --2- 172.21.15.116:0/1015029958 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90300c1c10 0x7f9030146a10 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f902400c420 tx=0x7f902400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:18.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.656+0000 7f902d7fa640 1 -- 172.21.15.116:0/1015029958 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90240052d0 con 0x7f90300c1c10 2024-04-25T00:03:18.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.656+0000 7f902d7fa640 1 -- 172.21.15.116:0/1015029958 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9024005470 con 0x7f90300c1c10 2024-04-25T00:03:18.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.656+0000 7f9036d8d640 1 -- 172.21.15.116:0/1015029958 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90301eef90 con 0x7f90300c1c10 2024-04-25T00:03:18.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.656+0000 7f902d7fa640 1 -- 172.21.15.116:0/1015029958 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9024007260 con 0x7f90300c1c10 2024-04-25T00:03:18.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.656+0000 7f9036d8d640 1 -- 172.21.15.116:0/1015029958 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90301ef550 con 0x7f90300c1c10 2024-04-25T00:03:18.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.656+0000 7f9036d8d640 1 -- 172.21.15.116:0/1015029958 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9030068a60 con 0x7f90300c1c10 2024-04-25T00:03:18.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.656+0000 7f902d7fa640 1 -- 172.21.15.116:0/1015029958 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f9024007400 con 0x7f90300c1c10 2024-04-25T00:03:18.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.656+0000 7f902d7fa640 1 --2- 172.21.15.116:0/1015029958 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f901c0678c0 0x7f901c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:18.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.656+0000 7f902d7fa640 1 -- 172.21.15.116:0/1015029958 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6752+0+0 (secure 0 0 0) 0x7f9024085500 con 0x7f90300c1c10 2024-04-25T00:03:18.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.656+0000 7f9034b29640 1 --2- 172.21.15.116:0/1015029958 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f901c0678c0 0x7f901c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:18.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.660+0000 7f9034b29640 1 --2- 172.21.15.116:0/1015029958 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f901c0678c0 0x7f901c069d80 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f90200af000 tx=0x7f9020008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:18.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.664+0000 7f902d7fa640 1 -- 172.21.15.116:0/1015029958 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f9024055cc0 con 0x7f90300c1c10 2024-04-25T00:03:18.698 INFO:tasks.workunit.client.0.smithi116.stdout:config.status: creating config.h 2024-04-25T00:03:18.757 INFO:tasks.workunit.client.0.smithi116.stdout:make all-am 2024-04-25T00:03:18.786 INFO:tasks.workunit.client.0.smithi116.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2024-04-25T00:03:18.787 INFO:tasks.workunit.client.0.smithi116.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fileops.o -MD -MP -MF .deps/fileops.Tpo -c -o fileops.o fileops.c 2024-04-25T00:03:18.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.868+0000 7f9036d8d640 1 -- 172.21.15.116:0/1015029958 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f90300c17e0 con 0x7f90300c1c10 2024-04-25T00:03:18.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.868+0000 7f902d7fa640 1 -- 172.21.15.116:0/1015029958 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6498 (secure 0 0 0) 0x7f90240555c0 con 0x7f90300c1c10 2024-04-25T00:03:18.871 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:03:18.871 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4735,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/2004876235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":2004876235},{"type":"v1","addr":"172.21.15.116:6835","nonce":2004876235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:01:08.748957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6837/1762356055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1762356055},{"type":"v1","addr":"172.21.15.174:6837","nonce":1762356055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270]},"id":1},{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:02:43.411054+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4242":{"gid":4242,"name":"c","rank":0,"incarnation":15,"state":"up:active","state_seq":29,"addr":"172.21.15.116:6834/1459177921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1459177921},{"type":"v1","addr":"172.21.15.116:6834","nonce":1459177921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4242,"qdb_cluster":[4242]},"id":2}]} 2024-04-25T00:03:18.872 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 19 2024-04-25T00:03:18.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.868+0000 7f9036d8d640 1 -- 172.21.15.116:0/1015029958 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f901c0678c0 msgr2=0x7f901c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:18.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.868+0000 7f9036d8d640 1 --2- 172.21.15.116:0/1015029958 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f901c0678c0 0x7f901c069d80 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f90200af000 tx=0x7f9020008040 comp rx=0 tx=0).stop 2024-04-25T00:03:18.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.868+0000 7f9036d8d640 1 -- 172.21.15.116:0/1015029958 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90300c1c10 msgr2=0x7f9030146a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:18.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.868+0000 7f9036d8d640 1 --2- 172.21.15.116:0/1015029958 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90300c1c10 0x7f9030146a10 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f902400c420 tx=0x7f902400c8e0 comp rx=0 tx=0).stop 2024-04-25T00:03:18.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.868+0000 7f9036d8d640 1 -- 172.21.15.116:0/1015029958 shutdown_connections 2024-04-25T00:03:18.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.868+0000 7f9036d8d640 1 --2- 172.21.15.116:0/1015029958 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f901c0678c0 0x7f901c069d80 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:18.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.868+0000 7f9036d8d640 1 --2- 172.21.15.116:0/1015029958 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9030150150 0x7f9030146f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:18.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.868+0000 7f9036d8d640 1 --2- 172.21.15.116:0/1015029958 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90300c1c10 0x7f9030146a10 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:18.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.868+0000 7f9036d8d640 1 --2- 172.21.15.116:0/1015029958 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f903006c220 0x7f9030148e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:18.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.868+0000 7f9036d8d640 1 -- 172.21.15.116:0/1015029958 >> 172.21.15.116:0/1015029958 conn(0x7f90300a9b20 msgr2=0x7f903014c620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:03:18.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.868+0000 7f9036d8d640 1 -- 172.21.15.116:0/1015029958 shutdown_connections 2024-04-25T00:03:18.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:18.868+0000 7f9036d8d640 1 -- 172.21.15.116:0/1015029958 wait complete. 2024-04-25T00:03:18.889 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:03:18.987 INFO:tasks.workunit.client.0.smithi116.stdout:mv -f .deps/fileops.Tpo .deps/fileops.Po 2024-04-25T00:03:18.999 INFO:tasks.workunit.client.0.smithi116.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rand.o -MD -MP -MF .deps/rand.Tpo -c -o rand.o rand.c 2024-04-25T00:03:19.144 INFO:tasks.workunit.client.0.smithi116.stdout:mv -f .deps/rand.Tpo .deps/rand.Po 2024-04-25T00:03:19.157 INFO:tasks.workunit.client.0.smithi116.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT main.o -MD -MP -MF .deps/main.Tpo -c -o main.o main.c 2024-04-25T00:03:19.319 INFO:tasks.workunit.client.0.smithi116.stdout:mv -f .deps/main.Tpo .deps/main.Po 2024-04-25T00:03:19.331 INFO:tasks.workunit.client.0.smithi116.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fh.o -MD -MP -MF .deps/fh.Tpo -c -o fh.o fh.c 2024-04-25T00:03:19.499 INFO:tasks.workunit.client.0.smithi116.stdout:mv -f .deps/fh.Tpo .deps/fh.Po 2024-04-25T00:03:19.512 INFO:tasks.workunit.client.0.smithi116.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT filelist.o -MD -MP -MF .deps/filelist.Tpo -c -o filelist.o filelist.c 2024-04-25T00:03:19.691 INFO:tasks.workunit.client.0.smithi116.stdout:mv -f .deps/filelist.Tpo .deps/filelist.Po 2024-04-25T00:03:19.705 INFO:tasks.workunit.client.0.smithi116.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT metaops.o -MD -MP -MF .deps/metaops.Tpo -c -o metaops.o metaops.c 2024-04-25T00:03:19.815 INFO:tasks.workunit.client.0.smithi116.stdout:mv -f .deps/metaops.Tpo .deps/metaops.Po 2024-04-25T00:03:19.830 INFO:tasks.workunit.client.0.smithi116.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rwlock.o -MD -MP -MF .deps/rwlock.Tpo -c -o rwlock.o rwlock.c 2024-04-25T00:03:19.913 INFO:tasks.workunit.client.0.smithi116.stdout:mv -f .deps/rwlock.Tpo .deps/rwlock.Po 2024-04-25T00:03:19.929 INFO:tasks.workunit.client.0.smithi116.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT cirlist.o -MD -MP -MF .deps/cirlist.Tpo -c -o cirlist.o cirlist.c 2024-04-25T00:03:20.024 INFO:tasks.workunit.client.0.smithi116.stdout:mv -f .deps/cirlist.Tpo .deps/cirlist.Po 2024-04-25T00:03:20.040 INFO:tasks.workunit.client.0.smithi116.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rbt.o -MD -MP -MF .deps/rbt.Tpo -c -o rbt.o rbt.c 2024-04-25T00:03:20.312 INFO:tasks.workunit.client.0.smithi116.stdout:mv -f .deps/rbt.Tpo .deps/rbt.Po 2024-04-25T00:03:20.329 INFO:tasks.workunit.client.0.smithi116.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_tg.o -MD -MP -MF .deps/ffsb_tg.Tpo -c -o ffsb_tg.o ffsb_tg.c 2024-04-25T00:03:20.541 INFO:tasks.workunit.client.0.smithi116.stdout:mv -f .deps/ffsb_tg.Tpo .deps/ffsb_tg.Po 2024-04-25T00:03:20.553 INFO:tasks.workunit.client.0.smithi116.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fs.o -MD -MP -MF .deps/ffsb_fs.Tpo -c -o ffsb_fs.o ffsb_fs.c 2024-04-25T00:03:20.847 INFO:tasks.workunit.client.0.smithi116.stdout:mv -f .deps/ffsb_fs.Tpo .deps/ffsb_fs.Po 2024-04-25T00:03:20.861 INFO:tasks.workunit.client.0.smithi116.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_thread.o -MD -MP -MF .deps/ffsb_thread.Tpo -c -o ffsb_thread.o ffsb_thread.c 2024-04-25T00:03:20.890 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:03:21.002 INFO:tasks.workunit.client.0.smithi116.stdout:mv -f .deps/ffsb_thread.Tpo .deps/ffsb_thread.Po 2024-04-25T00:03:21.014 INFO:tasks.workunit.client.0.smithi116.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_op.o -MD -MP -MF .deps/ffsb_op.Tpo -c -o ffsb_op.o ffsb_op.c 2024-04-25T00:03:21.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.016+0000 7f086d42c640 1 Processor -- start 2024-04-25T00:03:21.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.016+0000 7f086d42c640 1 -- start start 2024-04-25T00:03:21.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.016+0000 7f086d42c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0868152d60 0x7f08680ab270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:21.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.016+0000 7f086d42c640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0868155c50 con 0x7f08680ab7b0 2024-04-25T00:03:21.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.016+0000 7f086d42c640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0868155f30 con 0x7f0868153140 2024-04-25T00:03:21.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.016+0000 7f086d42c640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f0868156210 con 0x7f0868152d60 2024-04-25T00:03:21.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.016+0000 7f0866ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0868152d60 0x7f08680ab270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:21.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.016+0000 7f0866ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0868152d60 0x7f08680ab270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59268/0 (socket says 172.21.15.116:59268) 2024-04-25T00:03:21.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.016+0000 7f0866ffd640 1 -- 172.21.15.116:0/3943003209 learned_addr learned my addr 172.21.15.116:0/3943003209 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:03:21.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.016+0000 7f085e7fc640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f08680ab7b0 0x7f08680c8450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:58312/0 (socket says 172.21.15.116:58312) 2024-04-25T00:03:21.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.016+0000 7f0866ffd640 1 -- 172.21.15.116:0/3943003209 >> v1:172.21.15.174:6789/0 conn(0x7f0868153140 legacy=0x7f08680c3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:03:21.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.016+0000 7f0866ffd640 1 -- 172.21.15.116:0/3943003209 >> v1:172.21.15.116:6789/0 conn(0x7f08680ab7b0 legacy=0x7f08680c8450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:21.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.016+0000 7f0866ffd640 1 -- 172.21.15.116:0/3943003209 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08681564f0 con 0x7f0868152d60 2024-04-25T00:03:21.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.016+0000 7f0866ffd640 1 --2- 172.21.15.116:0/3943003209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0868152d60 0x7f08680ab270 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f0858009870 tx=0x7f085801c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f08c368548746c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:21.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.020+0000 7f08667fc640 1 -- 172.21.15.116:0/3943003209 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08580b0070 con 0x7f0868152d60 2024-04-25T00:03:21.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.020+0000 7f08667fc640 1 -- 172.21.15.116:0/3943003209 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08580ab600 con 0x7f0868152d60 2024-04-25T00:03:21.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.020+0000 7f08667fc640 1 -- 172.21.15.116:0/3943003209 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08580aba10 con 0x7f0868152d60 2024-04-25T00:03:21.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.020+0000 7f086d42c640 1 -- 172.21.15.116:0/3943003209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0868152d60 msgr2=0x7f08680ab270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:21.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.020+0000 7f086d42c640 1 --2- 172.21.15.116:0/3943003209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0868152d60 0x7f08680ab270 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f0858009870 tx=0x7f085801c540 comp rx=0 tx=0).stop 2024-04-25T00:03:21.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.020+0000 7f086d42c640 1 -- 172.21.15.116:0/3943003209 shutdown_connections 2024-04-25T00:03:21.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.020+0000 7f086d42c640 1 --2- 172.21.15.116:0/3943003209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0868152d60 0x7f08680ab270 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:21.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.020+0000 7f086d42c640 1 -- 172.21.15.116:0/3943003209 >> 172.21.15.116:0/3943003209 conn(0x7f086806f0c0 msgr2=0x7f086806f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:03:21.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.020+0000 7f086d42c640 1 -- 172.21.15.116:0/3943003209 shutdown_connections 2024-04-25T00:03:21.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.020+0000 7f086d42c640 1 -- 172.21.15.116:0/3943003209 wait complete. 2024-04-25T00:03:21.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.020+0000 7f086d42c640 1 Processor -- start 2024-04-25T00:03:21.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.020+0000 7f086d42c640 1 -- start start 2024-04-25T00:03:21.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.020+0000 7f086d42c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f08680ab7b0 0x7f0868149ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:21.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.020+0000 7f086d42c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0868152d60 0x7f086814cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:21.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.020+0000 7f086d42c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0868153140 0x7f086814d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:21.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.020+0000 7f086d42c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f08681639b0 con 0x7f0868153140 2024-04-25T00:03:21.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.020+0000 7f086d42c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f08680c4a30 con 0x7f0868152d60 2024-04-25T00:03:21.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.020+0000 7f086d42c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f0868163b30 con 0x7f08680ab7b0 2024-04-25T00:03:21.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.020+0000 7f08677fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0868153140 0x7f086814d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:21.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.020+0000 7f08677fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0868153140 0x7f086814d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59282/0 (socket says 172.21.15.116:59282) 2024-04-25T00:03:21.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.020+0000 7f08677fe640 1 -- 172.21.15.116:0/1328998191 learned_addr learned my addr 172.21.15.116:0/1328998191 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:03:21.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.020+0000 7f085e7fc640 1 --2- 172.21.15.116:0/1328998191 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0868152d60 0x7f086814cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:21.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.020+0000 7f0866ffd640 1 --2- 172.21.15.116:0/1328998191 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f08680ab7b0 0x7f0868149ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:21.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.020+0000 7f08677fe640 1 -- 172.21.15.116:0/1328998191 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f08680ab7b0 msgr2=0x7f0868149ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:21.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.020+0000 7f08677fe640 1 --2- 172.21.15.116:0/1328998191 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f08680ab7b0 0x7f0868149ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:21.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.020+0000 7f08677fe640 1 -- 172.21.15.116:0/1328998191 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0868152d60 msgr2=0x7f086814cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:21.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.020+0000 7f08677fe640 1 --2- 172.21.15.116:0/1328998191 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0868152d60 0x7f086814cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:21.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.020+0000 7f08677fe640 1 -- 172.21.15.116:0/1328998191 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f086806d630 con 0x7f0868153140 2024-04-25T00:03:21.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.020+0000 7f085e7fc640 1 --2- 172.21.15.116:0/1328998191 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0868152d60 0x7f086814cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:03:21.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.020+0000 7f08677fe640 1 --2- 172.21.15.116:0/1328998191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0868153140 0x7f086814d440 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f086000c7b0 tx=0x7f086000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:21.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.020+0000 7f085ffff640 1 -- 172.21.15.116:0/1328998191 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f086000ce60 con 0x7f0868153140 2024-04-25T00:03:21.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.020+0000 7f085ffff640 1 -- 172.21.15.116:0/1328998191 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0860004b20 con 0x7f0868153140 2024-04-25T00:03:21.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.020+0000 7f086d42c640 1 -- 172.21.15.116:0/1328998191 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f086814b840 con 0x7f0868153140 2024-04-25T00:03:21.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.020+0000 7f085ffff640 1 -- 172.21.15.116:0/1328998191 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f086000a850 con 0x7f0868153140 2024-04-25T00:03:21.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.020+0000 7f086d42c640 1 -- 172.21.15.116:0/1328998191 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f086814bb00 con 0x7f0868153140 2024-04-25T00:03:21.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.024+0000 7f086d42c640 1 -- 172.21.15.116:0/1328998191 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0868068a60 con 0x7f0868153140 2024-04-25T00:03:21.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.024+0000 7f085ffff640 1 -- 172.21.15.116:0/1328998191 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f086000a9f0 con 0x7f0868153140 2024-04-25T00:03:21.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.024+0000 7f085ffff640 1 --2- 172.21.15.116:0/1328998191 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f084c0678c0 0x7f084c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:21.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.024+0000 7f085ffff640 1 -- 172.21.15.116:0/1328998191 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6752+0+0 (secure 0 0 0) 0x7f0860086170 con 0x7f0868153140 2024-04-25T00:03:21.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.024+0000 7f0866ffd640 1 --2- 172.21.15.116:0/1328998191 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f084c0678c0 0x7f084c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:21.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.024+0000 7f0866ffd640 1 --2- 172.21.15.116:0/1328998191 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f084c0678c0 0x7f084c069d80 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f08680c6ac0 tx=0x7f08580a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:21.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.032+0000 7f085ffff640 1 -- 172.21.15.116:0/1328998191 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f0860020070 con 0x7f0868153140 2024-04-25T00:03:21.070 INFO:tasks.workunit.client.0.smithi116.stderr:ffsb_op.c: In function 'print_op_results': 2024-04-25T00:03:21.070 INFO:tasks.workunit.client.0.smithi116.stderr:ffsb_op.c:83:37: warning: embedded '\0' in format [-Wformat-contains-nul] 2024-04-25T00:03:21.070 INFO:tasks.workunit.client.0.smithi116.stderr: 83 | sprintf(buf, "%s/sec\0", buf); 2024-04-25T00:03:21.070 INFO:tasks.workunit.client.0.smithi116.stderr: | ^~ 2024-04-25T00:03:21.070 INFO:tasks.workunit.client.0.smithi116.stderr:ffsb_op.c:86:33: warning: embedded '\0' in format [-Wformat-contains-nul] 2024-04-25T00:03:21.070 INFO:tasks.workunit.client.0.smithi116.stderr: 86 | sprintf(buf, "NA\0"); 2024-04-25T00:03:21.071 INFO:tasks.workunit.client.0.smithi116.stderr: | ^~ 2024-04-25T00:03:21.085 INFO:tasks.workunit.client.0.smithi116.stderr:ffsb_op.c: In function 'print_results': 2024-04-25T00:03:21.086 INFO:tasks.workunit.client.0.smithi116.stderr:ffsb_op.c:83:33: warning: '/sec' directive writing 4 bytes into a region of size between 1 and 256 [-Wformat-overflow=] 2024-04-25T00:03:21.086 INFO:tasks.workunit.client.0.smithi116.stderr: 83 | sprintf(buf, "%s/sec\0", buf); 2024-04-25T00:03:21.086 INFO:tasks.workunit.client.0.smithi116.stderr: | ^~~~ 2024-04-25T00:03:21.086 INFO:tasks.workunit.client.0.smithi116.stderr:In file included from /usr/include/stdio.h:894, 2024-04-25T00:03:21.086 INFO:tasks.workunit.client.0.smithi116.stderr: from ffsb_op.c:20: 2024-04-25T00:03:21.086 INFO:tasks.workunit.client.0.smithi116.stderr:/usr/include/x86_64-linux-gnu/bits/stdio2.h:38:10: note: '__builtin___sprintf_chk' output between 5 and 260 bytes into a destination of size 256 2024-04-25T00:03:21.086 INFO:tasks.workunit.client.0.smithi116.stderr: 38 | return __builtin___sprintf_chk (__s, __USE_FORTIFY_LEVEL - 1, 2024-04-25T00:03:21.086 INFO:tasks.workunit.client.0.smithi116.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-25T00:03:21.086 INFO:tasks.workunit.client.0.smithi116.stderr: 39 | __glibc_objsize (__s), __fmt, 2024-04-25T00:03:21.086 INFO:tasks.workunit.client.0.smithi116.stderr: | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-25T00:03:21.086 INFO:tasks.workunit.client.0.smithi116.stderr: 40 | __va_arg_pack ()); 2024-04-25T00:03:21.086 INFO:tasks.workunit.client.0.smithi116.stderr: | ~~~~~~~~~~~~~~~~~ 2024-04-25T00:03:21.135 INFO:tasks.workunit.client.0.smithi116.stdout:mv -f .deps/ffsb_op.Tpo .deps/ffsb_op.Po 2024-04-25T00:03:21.148 INFO:tasks.workunit.client.0.smithi116.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT util.o -MD -MP -MF .deps/util.Tpo -c -o util.o util.c 2024-04-25T00:03:21.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.240+0000 7f086d42c640 1 -- 172.21.15.116:0/1328998191 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f08680c6300 con 0x7f0868153140 2024-04-25T00:03:21.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.240+0000 7f085ffff640 1 -- 172.21.15.116:0/1328998191 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6498 (secure 0 0 0) 0x7f086806d630 con 0x7f0868153140 2024-04-25T00:03:21.246 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:03:21.246 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4735,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/2004876235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":2004876235},{"type":"v1","addr":"172.21.15.116:6835","nonce":2004876235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:01:08.748957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6837/1762356055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1762356055},{"type":"v1","addr":"172.21.15.174:6837","nonce":1762356055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270]},"id":1},{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:02:43.411054+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4242":{"gid":4242,"name":"c","rank":0,"incarnation":15,"state":"up:active","state_seq":29,"addr":"172.21.15.116:6834/1459177921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1459177921},{"type":"v1","addr":"172.21.15.116:6834","nonce":1459177921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4242,"qdb_cluster":[4242]},"id":2}]} 2024-04-25T00:03:21.247 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 19 2024-04-25T00:03:21.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.244+0000 7f086d42c640 1 -- 172.21.15.116:0/1328998191 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f084c0678c0 msgr2=0x7f084c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:21.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.244+0000 7f086d42c640 1 --2- 172.21.15.116:0/1328998191 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f084c0678c0 0x7f084c069d80 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f08680c6ac0 tx=0x7f08580a6000 comp rx=0 tx=0).stop 2024-04-25T00:03:21.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.244+0000 7f086d42c640 1 -- 172.21.15.116:0/1328998191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0868153140 msgr2=0x7f086814d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:21.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.244+0000 7f086d42c640 1 --2- 172.21.15.116:0/1328998191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0868153140 0x7f086814d440 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f086000c7b0 tx=0x7f086000cc70 comp rx=0 tx=0).stop 2024-04-25T00:03:21.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.244+0000 7f086d42c640 1 -- 172.21.15.116:0/1328998191 shutdown_connections 2024-04-25T00:03:21.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.244+0000 7f086d42c640 1 --2- 172.21.15.116:0/1328998191 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f084c0678c0 0x7f084c069d80 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:21.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.244+0000 7f086d42c640 1 --2- 172.21.15.116:0/1328998191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0868153140 0x7f086814d440 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:21.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.244+0000 7f086d42c640 1 --2- 172.21.15.116:0/1328998191 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0868152d60 0x7f086814cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:21.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.244+0000 7f086d42c640 1 --2- 172.21.15.116:0/1328998191 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f08680ab7b0 0x7f0868149ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:21.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.244+0000 7f086d42c640 1 -- 172.21.15.116:0/1328998191 >> 172.21.15.116:0/1328998191 conn(0x7f086806f0c0 msgr2=0x7f0868154a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:03:21.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.244+0000 7f086d42c640 1 -- 172.21.15.116:0/1328998191 shutdown_connections 2024-04-25T00:03:21.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.244+0000 7f086d42c640 1 -- 172.21.15.116:0/1328998191 wait complete. 2024-04-25T00:03:21.266 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:03:21.321 INFO:tasks.workunit.client.0.smithi116.stdout:mv -f .deps/util.Tpo .deps/util.Po 2024-04-25T00:03:21.332 INFO:tasks.workunit.client.0.smithi116.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT parser.o -MD -MP -MF .deps/parser.Tpo -c -o parser.o parser.c 2024-04-25T00:03:21.399 INFO:tasks.workunit.client.0.smithi116.stderr:In file included from parser.h:22, 2024-04-25T00:03:21.399 INFO:tasks.workunit.client.0.smithi116.stderr: from parser.c:25: 2024-04-25T00:03:21.399 INFO:tasks.workunit.client.0.smithi116.stderr:parser.c: In function 'init_filesys': 2024-04-25T00:03:21.401 INFO:tasks.workunit.client.0.smithi116.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-25T00:03:21.401 INFO:tasks.workunit.client.0.smithi116.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-25T00:03:21.402 INFO:tasks.workunit.client.0.smithi116.stderr: | ^ 2024-04-25T00:03:21.402 INFO:tasks.workunit.client.0.smithi116.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2024-04-25T00:03:21.402 INFO:tasks.workunit.client.0.smithi116.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-25T00:03:21.402 INFO:tasks.workunit.client.0.smithi116.stderr: | ^~~~~~~~ 2024-04-25T00:03:21.402 INFO:tasks.workunit.client.0.smithi116.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2024-04-25T00:03:21.402 INFO:tasks.workunit.client.0.smithi116.stderr: 36 | container_of(ptr, type, member) 2024-04-25T00:03:21.402 INFO:tasks.workunit.client.0.smithi116.stderr: | ^~~~~~~~~~~~ 2024-04-25T00:03:21.402 INFO:tasks.workunit.client.0.smithi116.stderr:list.h:39:20: note: in expansion of macro 'list_entry' 2024-04-25T00:03:21.402 INFO:tasks.workunit.client.0.smithi116.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-04-25T00:03:21.402 INFO:tasks.workunit.client.0.smithi116.stderr: | ^~~~~~~~~~ 2024-04-25T00:03:21.403 INFO:tasks.workunit.client.0.smithi116.stderr:parser.c:934:17: note: in expansion of macro 'list_for_each_entry' 2024-04-25T00:03:21.403 INFO:tasks.workunit.client.0.smithi116.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2024-04-25T00:03:21.403 INFO:tasks.workunit.client.0.smithi116.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-25T00:03:21.403 INFO:tasks.workunit.client.0.smithi116.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-25T00:03:21.403 INFO:tasks.workunit.client.0.smithi116.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-25T00:03:21.403 INFO:tasks.workunit.client.0.smithi116.stderr: | ^ 2024-04-25T00:03:21.403 INFO:tasks.workunit.client.0.smithi116.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2024-04-25T00:03:21.403 INFO:tasks.workunit.client.0.smithi116.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-25T00:03:21.403 INFO:tasks.workunit.client.0.smithi116.stderr: | ^~~~~~~~ 2024-04-25T00:03:21.403 INFO:tasks.workunit.client.0.smithi116.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2024-04-25T00:03:21.404 INFO:tasks.workunit.client.0.smithi116.stderr: 36 | container_of(ptr, type, member) 2024-04-25T00:03:21.404 INFO:tasks.workunit.client.0.smithi116.stderr: | ^~~~~~~~~~~~ 2024-04-25T00:03:21.404 INFO:tasks.workunit.client.0.smithi116.stderr:list.h:41:20: note: in expansion of macro 'list_entry' 2024-04-25T00:03:21.404 INFO:tasks.workunit.client.0.smithi116.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-04-25T00:03:21.404 INFO:tasks.workunit.client.0.smithi116.stderr: | ^~~~~~~~~~ 2024-04-25T00:03:21.404 INFO:tasks.workunit.client.0.smithi116.stderr:parser.c:934:17: note: in expansion of macro 'list_for_each_entry' 2024-04-25T00:03:21.404 INFO:tasks.workunit.client.0.smithi116.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2024-04-25T00:03:21.404 INFO:tasks.workunit.client.0.smithi116.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-25T00:03:21.404 INFO:tasks.workunit.client.0.smithi116.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-25T00:03:21.405 INFO:tasks.workunit.client.0.smithi116.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-25T00:03:21.405 INFO:tasks.workunit.client.0.smithi116.stderr: | ^ 2024-04-25T00:03:21.405 INFO:tasks.workunit.client.0.smithi116.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2024-04-25T00:03:21.405 INFO:tasks.workunit.client.0.smithi116.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-25T00:03:21.405 INFO:tasks.workunit.client.0.smithi116.stderr: | ^~~~~~~~ 2024-04-25T00:03:21.405 INFO:tasks.workunit.client.0.smithi116.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2024-04-25T00:03:21.405 INFO:tasks.workunit.client.0.smithi116.stderr: 36 | container_of(ptr, type, member) 2024-04-25T00:03:21.405 INFO:tasks.workunit.client.0.smithi116.stderr: | ^~~~~~~~~~~~ 2024-04-25T00:03:21.405 INFO:tasks.workunit.client.0.smithi116.stderr:list.h:39:20: note: in expansion of macro 'list_entry' 2024-04-25T00:03:21.406 INFO:tasks.workunit.client.0.smithi116.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-04-25T00:03:21.406 INFO:tasks.workunit.client.0.smithi116.stderr: | ^~~~~~~~~~ 2024-04-25T00:03:21.406 INFO:tasks.workunit.client.0.smithi116.stderr:parser.c:941:17: note: in expansion of macro 'list_for_each_entry' 2024-04-25T00:03:21.406 INFO:tasks.workunit.client.0.smithi116.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2024-04-25T00:03:21.406 INFO:tasks.workunit.client.0.smithi116.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-25T00:03:21.406 INFO:tasks.workunit.client.0.smithi116.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-25T00:03:21.406 INFO:tasks.workunit.client.0.smithi116.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-25T00:03:21.406 INFO:tasks.workunit.client.0.smithi116.stderr: | ^ 2024-04-25T00:03:21.407 INFO:tasks.workunit.client.0.smithi116.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2024-04-25T00:03:21.407 INFO:tasks.workunit.client.0.smithi116.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-25T00:03:21.407 INFO:tasks.workunit.client.0.smithi116.stderr: | ^~~~~~~~ 2024-04-25T00:03:21.407 INFO:tasks.workunit.client.0.smithi116.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2024-04-25T00:03:21.407 INFO:tasks.workunit.client.0.smithi116.stderr: 36 | container_of(ptr, type, member) 2024-04-25T00:03:21.407 INFO:tasks.workunit.client.0.smithi116.stderr: | ^~~~~~~~~~~~ 2024-04-25T00:03:21.407 INFO:tasks.workunit.client.0.smithi116.stderr:list.h:41:20: note: in expansion of macro 'list_entry' 2024-04-25T00:03:21.407 INFO:tasks.workunit.client.0.smithi116.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-04-25T00:03:21.407 INFO:tasks.workunit.client.0.smithi116.stderr: | ^~~~~~~~~~ 2024-04-25T00:03:21.408 INFO:tasks.workunit.client.0.smithi116.stderr:parser.c:941:17: note: in expansion of macro 'list_for_each_entry' 2024-04-25T00:03:21.408 INFO:tasks.workunit.client.0.smithi116.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2024-04-25T00:03:21.408 INFO:tasks.workunit.client.0.smithi116.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-25T00:03:21.408 INFO:tasks.workunit.client.0.smithi116.stderr:parser.c: In function 'init_tg_stats': 2024-04-25T00:03:21.408 INFO:tasks.workunit.client.0.smithi116.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-25T00:03:21.409 INFO:tasks.workunit.client.0.smithi116.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-25T00:03:21.409 INFO:tasks.workunit.client.0.smithi116.stderr: | ^ 2024-04-25T00:03:21.409 INFO:tasks.workunit.client.0.smithi116.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2024-04-25T00:03:21.409 INFO:tasks.workunit.client.0.smithi116.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-25T00:03:21.410 INFO:tasks.workunit.client.0.smithi116.stderr: | ^~~~~~~~ 2024-04-25T00:03:21.410 INFO:tasks.workunit.client.0.smithi116.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2024-04-25T00:03:21.411 INFO:tasks.workunit.client.0.smithi116.stderr: 36 | container_of(ptr, type, member) 2024-04-25T00:03:21.411 INFO:tasks.workunit.client.0.smithi116.stderr: | ^~~~~~~~~~~~ 2024-04-25T00:03:21.411 INFO:tasks.workunit.client.0.smithi116.stderr:list.h:39:20: note: in expansion of macro 'list_entry' 2024-04-25T00:03:21.411 INFO:tasks.workunit.client.0.smithi116.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-04-25T00:03:21.412 INFO:tasks.workunit.client.0.smithi116.stderr: | ^~~~~~~~~~ 2024-04-25T00:03:21.412 INFO:tasks.workunit.client.0.smithi116.stderr:parser.c:971:41: note: in expansion of macro 'list_for_each_entry' 2024-04-25T00:03:21.412 INFO:tasks.workunit.client.0.smithi116.stderr: 971 | list_for_each_entry(tmp_list, 2024-04-25T00:03:21.413 INFO:tasks.workunit.client.0.smithi116.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-25T00:03:21.413 INFO:tasks.workunit.client.0.smithi116.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-25T00:03:21.413 INFO:tasks.workunit.client.0.smithi116.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-25T00:03:21.414 INFO:tasks.workunit.client.0.smithi116.stderr: | ^ 2024-04-25T00:03:21.414 INFO:tasks.workunit.client.0.smithi116.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2024-04-25T00:03:21.414 INFO:tasks.workunit.client.0.smithi116.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-25T00:03:21.414 INFO:tasks.workunit.client.0.smithi116.stderr: | ^~~~~~~~ 2024-04-25T00:03:21.414 INFO:tasks.workunit.client.0.smithi116.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2024-04-25T00:03:21.415 INFO:tasks.workunit.client.0.smithi116.stderr: 36 | container_of(ptr, type, member) 2024-04-25T00:03:21.415 INFO:tasks.workunit.client.0.smithi116.stderr: | ^~~~~~~~~~~~ 2024-04-25T00:03:21.415 INFO:tasks.workunit.client.0.smithi116.stderr:list.h:41:20: note: in expansion of macro 'list_entry' 2024-04-25T00:03:21.415 INFO:tasks.workunit.client.0.smithi116.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-04-25T00:03:21.415 INFO:tasks.workunit.client.0.smithi116.stderr: | ^~~~~~~~~~ 2024-04-25T00:03:21.416 INFO:tasks.workunit.client.0.smithi116.stderr:parser.c:971:41: note: in expansion of macro 'list_for_each_entry' 2024-04-25T00:03:21.416 INFO:tasks.workunit.client.0.smithi116.stderr: 971 | list_for_each_entry(tmp_list, 2024-04-25T00:03:21.416 INFO:tasks.workunit.client.0.smithi116.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-25T00:03:21.417 INFO:tasks.workunit.client.0.smithi116.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-25T00:03:21.417 INFO:tasks.workunit.client.0.smithi116.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-25T00:03:21.417 INFO:tasks.workunit.client.0.smithi116.stderr: | ^ 2024-04-25T00:03:21.417 INFO:tasks.workunit.client.0.smithi116.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2024-04-25T00:03:21.417 INFO:tasks.workunit.client.0.smithi116.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-25T00:03:21.417 INFO:tasks.workunit.client.0.smithi116.stderr: | ^~~~~~~~ 2024-04-25T00:03:21.417 INFO:tasks.workunit.client.0.smithi116.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2024-04-25T00:03:21.417 INFO:tasks.workunit.client.0.smithi116.stderr: 36 | container_of(ptr, type, member) 2024-04-25T00:03:21.417 INFO:tasks.workunit.client.0.smithi116.stderr: | ^~~~~~~~~~~~ 2024-04-25T00:03:21.418 INFO:tasks.workunit.client.0.smithi116.stderr:list.h:39:20: note: in expansion of macro 'list_entry' 2024-04-25T00:03:21.418 INFO:tasks.workunit.client.0.smithi116.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-04-25T00:03:21.418 INFO:tasks.workunit.client.0.smithi116.stderr: | ^~~~~~~~~~ 2024-04-25T00:03:21.418 INFO:tasks.workunit.client.0.smithi116.stderr:parser.c:980:41: note: in expansion of macro 'list_for_each_entry' 2024-04-25T00:03:21.418 INFO:tasks.workunit.client.0.smithi116.stderr: 980 | list_for_each_entry(tmp_list, 2024-04-25T00:03:21.418 INFO:tasks.workunit.client.0.smithi116.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-25T00:03:21.418 INFO:tasks.workunit.client.0.smithi116.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-25T00:03:21.418 INFO:tasks.workunit.client.0.smithi116.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-25T00:03:21.419 INFO:tasks.workunit.client.0.smithi116.stderr: | ^ 2024-04-25T00:03:21.419 INFO:tasks.workunit.client.0.smithi116.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2024-04-25T00:03:21.419 INFO:tasks.workunit.client.0.smithi116.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-25T00:03:21.419 INFO:tasks.workunit.client.0.smithi116.stderr: | ^~~~~~~~ 2024-04-25T00:03:21.419 INFO:tasks.workunit.client.0.smithi116.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2024-04-25T00:03:21.419 INFO:tasks.workunit.client.0.smithi116.stderr: 36 | container_of(ptr, type, member) 2024-04-25T00:03:21.419 INFO:tasks.workunit.client.0.smithi116.stderr: | ^~~~~~~~~~~~ 2024-04-25T00:03:21.419 INFO:tasks.workunit.client.0.smithi116.stderr:list.h:41:20: note: in expansion of macro 'list_entry' 2024-04-25T00:03:21.419 INFO:tasks.workunit.client.0.smithi116.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-04-25T00:03:21.420 INFO:tasks.workunit.client.0.smithi116.stderr: | ^~~~~~~~~~ 2024-04-25T00:03:21.420 INFO:tasks.workunit.client.0.smithi116.stderr:parser.c:980:41: note: in expansion of macro 'list_for_each_entry' 2024-04-25T00:03:21.420 INFO:tasks.workunit.client.0.smithi116.stderr: 980 | list_for_each_entry(tmp_list, 2024-04-25T00:03:21.420 INFO:tasks.workunit.client.0.smithi116.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-25T00:03:21.478 INFO:tasks.workunit.client.0.smithi116.stderr:parser.c: In function 'get_optstr': 2024-04-25T00:03:21.478 INFO:tasks.workunit.client.0.smithi116.stderr:parser.c:206:23: warning: 'strnlen' specified bound 4095 exceeds source size 1024 [-Wstringop-overread] 2024-04-25T00:03:21.478 INFO:tasks.workunit.client.0.smithi116.stderr: 206 | len = strnlen(temp, 4095) + 1; 2024-04-25T00:03:21.478 INFO:tasks.workunit.client.0.smithi116.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-25T00:03:21.482 INFO:tasks.workunit.client.0.smithi116.stderr:parser.c:200:14: note: source object declared here 2024-04-25T00:03:21.482 INFO:tasks.workunit.client.0.smithi116.stderr: 200 | char temp[BUFSIZE]; 2024-04-25T00:03:21.482 INFO:tasks.workunit.client.0.smithi116.stderr: | ^~~~ 2024-04-25T00:03:21.496 INFO:tasks.workunit.client.0.smithi116.stderr:In file included from /usr/include/string.h:535, 2024-04-25T00:03:21.496 INFO:tasks.workunit.client.0.smithi116.stderr: from parser.c:19: 2024-04-25T00:03:21.496 INFO:tasks.workunit.client.0.smithi116.stderr:In function 'memcpy', 2024-04-25T00:03:21.496 INFO:tasks.workunit.client.0.smithi116.stderr: inlined from 'size64_convert' at parser.c:119:2: 2024-04-25T00:03:21.496 INFO:tasks.workunit.client.0.smithi116.stderr:/usr/include/x86_64-linux-gnu/bits/string_fortified.h:29:10: warning: '__builtin_memcpy' reading 3 bytes from a region of size 2 [-Wstringop-overread] 2024-04-25T00:03:21.496 INFO:tasks.workunit.client.0.smithi116.stderr: 29 | return __builtin___memcpy_chk (__dest, __src, __len, 2024-04-25T00:03:21.496 INFO:tasks.workunit.client.0.smithi116.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-25T00:03:21.496 INFO:tasks.workunit.client.0.smithi116.stderr: 30 | __glibc_objsize0 (__dest)); 2024-04-25T00:03:21.496 INFO:tasks.workunit.client.0.smithi116.stderr: | ~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-25T00:03:21.686 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:03:21.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.800+0000 7fbb15036640 1 Processor -- start 2024-04-25T00:03:21.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.800+0000 7fbb15036640 1 -- start start 2024-04-25T00:03:21.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.800+0000 7fbb15036640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb10152db0 0x7fbb10071070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:21.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.800+0000 7fbb15036640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbb100c8970 0x7fbb101557c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:21.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.800+0000 7fbb15036640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb100c3900 con 0x7fbb100715b0 2024-04-25T00:03:21.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.800+0000 7fbb15036640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fbb100c4b00 con 0x7fbb100c8970 2024-04-25T00:03:21.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.800+0000 7fbb15036640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fbb100c3be0 con 0x7fbb10152db0 2024-04-25T00:03:21.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.800+0000 7fbb0e59c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbb100c8970 0x7fbb101557c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:21.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.804+0000 7fbb0f59e640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fbb100715b0 0x7fbb100c2330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:58314/0 (socket says 172.21.15.116:58314) 2024-04-25T00:03:21.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.804+0000 7fbb0f59e640 1 -- 172.21.15.116:0/3803392582 learned_addr learned my addr 172.21.15.116:0/3803392582 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:03:21.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.804+0000 7fbb0ed9d640 1 --2- 172.21.15.116:0/3803392582 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb10152db0 0x7fbb10071070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:21.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.804+0000 7fbb0e59c640 1 -- 172.21.15.116:0/3803392582 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb10152db0 msgr2=0x7fbb10071070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:03:21.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.804+0000 7fbb0e59c640 1 --2- 172.21.15.116:0/3803392582 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb10152db0 0x7fbb10071070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:21.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.804+0000 7fbb0e59c640 1 -- 172.21.15.116:0/3803392582 >> v1:172.21.15.116:6789/0 conn(0x7fbb100715b0 legacy=0x7fbb100c2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:21.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.804+0000 7fbb0e59c640 1 -- 172.21.15.116:0/3803392582 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb100c3ec0 con 0x7fbb100c8970 2024-04-25T00:03:21.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.804+0000 7fbb0dd9b640 1 -- 172.21.15.116:0/3803392582 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1834945540 0 0) 0x7fbb100c3900 con 0x7fbb100715b0 2024-04-25T00:03:21.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.804+0000 7fbb0e59c640 1 --2- 172.21.15.116:0/3803392582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb100c8970 0x7fbb101557c0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fbaf80097f0 tx=0x7fbaf801c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1812366fc71f66b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:21.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.804+0000 7fbb0dd9b640 1 -- 172.21.15.116:0/3803392582 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaf80b0070 con 0x7fbb100c8970 2024-04-25T00:03:21.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.804+0000 7fbb0dd9b640 1 -- 172.21.15.116:0/3803392582 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbaf80ab600 con 0x7fbb100c8970 2024-04-25T00:03:21.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.804+0000 7fbb0dd9b640 1 -- 172.21.15.116:0/3803392582 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaf80ab9d0 con 0x7fbb100c8970 2024-04-25T00:03:21.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.808+0000 7fbb15036640 1 -- 172.21.15.116:0/3803392582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb100c8970 msgr2=0x7fbb101557c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:21.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.808+0000 7fbb15036640 1 --2- 172.21.15.116:0/3803392582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb100c8970 0x7fbb101557c0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fbaf80097f0 tx=0x7fbaf801c380 comp rx=0 tx=0).stop 2024-04-25T00:03:21.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.808+0000 7fbb15036640 1 -- 172.21.15.116:0/3803392582 shutdown_connections 2024-04-25T00:03:21.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.808+0000 7fbb15036640 1 --2- 172.21.15.116:0/3803392582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb100c8970 0x7fbb101557c0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:21.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.808+0000 7fbb15036640 1 --2- 172.21.15.116:0/3803392582 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb10152db0 0x7fbb10071070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:21.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.808+0000 7fbb15036640 1 -- 172.21.15.116:0/3803392582 >> 172.21.15.116:0/3803392582 conn(0x7fbb100c90d0 msgr2=0x7fbb100c94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:03:21.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.808+0000 7fbb15036640 1 -- 172.21.15.116:0/3803392582 shutdown_connections 2024-04-25T00:03:21.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.808+0000 7fbb15036640 1 -- 172.21.15.116:0/3803392582 wait complete. 2024-04-25T00:03:21.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.808+0000 7fbb15036640 1 Processor -- start 2024-04-25T00:03:21.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.808+0000 7fbb15036640 1 -- start start 2024-04-25T00:03:21.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.808+0000 7fbb15036640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb100715b0 0x7fbb1014c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:21.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.808+0000 7fbb15036640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb100c8970 0x7fbb1006f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:21.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.808+0000 7fbb15036640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb10152db0 0x7fbb1006f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:21.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.808+0000 7fbb15036640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbb100c4f70 con 0x7fbb100c8970 2024-04-25T00:03:21.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.808+0000 7fbb15036640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbb10156110 con 0x7fbb10152db0 2024-04-25T00:03:21.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.808+0000 7fbb15036640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fbb100c50f0 con 0x7fbb100715b0 2024-04-25T00:03:21.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.808+0000 7fbb0e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb100c8970 0x7fbb1006f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:21.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.808+0000 7fbb0e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb100c8970 0x7fbb1006f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59308/0 (socket says 172.21.15.116:59308) 2024-04-25T00:03:21.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.808+0000 7fbb0e59c640 1 -- 172.21.15.116:0/572571076 learned_addr learned my addr 172.21.15.116:0/572571076 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:03:21.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.808+0000 7fbb0f59e640 1 --2- 172.21.15.116:0/572571076 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb10152db0 0x7fbb1006f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:21.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.808+0000 7fbb0ed9d640 1 --2- 172.21.15.116:0/572571076 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb100715b0 0x7fbb1014c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:21.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.808+0000 7fbb0e59c640 1 -- 172.21.15.116:0/572571076 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb100715b0 msgr2=0x7fbb1014c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:21.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.808+0000 7fbb0e59c640 1 --2- 172.21.15.116:0/572571076 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb100715b0 0x7fbb1014c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:21.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.808+0000 7fbb0e59c640 1 -- 172.21.15.116:0/572571076 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb10152db0 msgr2=0x7fbb1006f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:21.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.808+0000 7fbb0e59c640 1 --2- 172.21.15.116:0/572571076 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb10152db0 0x7fbb1006f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:21.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.808+0000 7fbb0e59c640 1 -- 172.21.15.116:0/572571076 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb1014b660 con 0x7fbb100c8970 2024-04-25T00:03:21.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.808+0000 7fbb0f59e640 1 --2- 172.21.15.116:0/572571076 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb10152db0 0x7fbb1006f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:03:21.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.808+0000 7fbb0e59c640 1 --2- 172.21.15.116:0/572571076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb100c8970 0x7fbb1006f020 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fbaf80097f0 tx=0x7fbaf801abc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:21.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.808+0000 7fbb077fe640 1 -- 172.21.15.116:0/572571076 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaf80b0050 con 0x7fbb100c8970 2024-04-25T00:03:21.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.808+0000 7fbb077fe640 1 -- 172.21.15.116:0/572571076 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbaf80bc070 con 0x7fbb100c8970 2024-04-25T00:03:21.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.808+0000 7fbb077fe640 1 -- 172.21.15.116:0/572571076 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaf80b5850 con 0x7fbb100c8970 2024-04-25T00:03:21.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.808+0000 7fbb15036640 1 -- 172.21.15.116:0/572571076 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb1014b940 con 0x7fbb100c8970 2024-04-25T00:03:21.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.808+0000 7fbb15036640 1 -- 172.21.15.116:0/572571076 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb10152440 con 0x7fbb100c8970 2024-04-25T00:03:21.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.812+0000 7fbb15036640 1 -- 172.21.15.116:0/572571076 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb10068350 con 0x7fbb100c8970 2024-04-25T00:03:21.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.812+0000 7fbb077fe640 1 -- 172.21.15.116:0/572571076 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbaf80b59f0 con 0x7fbb100c8970 2024-04-25T00:03:21.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.812+0000 7fbb077fe640 1 --2- 172.21.15.116:0/572571076 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbaf00678f0 0x7fbaf0069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:21.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.812+0000 7fbb077fe640 1 -- 172.21.15.116:0/572571076 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6752+0+0 (secure 0 0 0) 0x7fbaf81208d0 con 0x7fbb100c8970 2024-04-25T00:03:21.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.812+0000 7fbb0ed9d640 1 --2- 172.21.15.116:0/572571076 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbaf00678f0 0x7fbaf0069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:21.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.812+0000 7fbb0ed9d640 1 --2- 172.21.15.116:0/572571076 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbaf00678f0 0x7fbaf0069db0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbb00009790 tx=0x7fbb00008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:21.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:21.816+0000 7fbb077fe640 1 -- 172.21.15.116:0/572571076 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fbaf8016090 con 0x7fbb100c8970 2024-04-25T00:03:21.824 INFO:tasks.workunit.client.0.smithi116.stdout:mv -f .deps/parser.Tpo .deps/parser.Po 2024-04-25T00:03:21.835 INFO:tasks.workunit.client.0.smithi116.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fc.o -MD -MP -MF .deps/ffsb_fc.Tpo -c -o ffsb_fc.o ffsb_fc.c 2024-04-25T00:03:21.921 INFO:tasks.workunit.client.0.smithi116.stdout:mv -f .deps/ffsb_fc.Tpo .deps/ffsb_fc.Po 2024-04-25T00:03:21.931 INFO:tasks.workunit.client.0.smithi116.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_stats.o -MD -MP -MF .deps/ffsb_stats.Tpo -c -o ffsb_stats.o ffsb_stats.c 2024-04-25T00:03:22.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.024+0000 7fbb15036640 1 -- 172.21.15.116:0/572571076 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbb10154c40 con 0x7fbb100c8970 2024-04-25T00:03:22.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.024+0000 7fbb077fe640 1 -- 172.21.15.116:0/572571076 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6498 (secure 0 0 0) 0x7fbaf80f0f70 con 0x7fbb100c8970 2024-04-25T00:03:22.027 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:03:22.028 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4735,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/2004876235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":2004876235},{"type":"v1","addr":"172.21.15.116:6835","nonce":2004876235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:01:08.748957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6837/1762356055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1762356055},{"type":"v1","addr":"172.21.15.174:6837","nonce":1762356055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270]},"id":1},{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:02:43.411054+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4242":{"gid":4242,"name":"c","rank":0,"incarnation":15,"state":"up:active","state_seq":29,"addr":"172.21.15.116:6834/1459177921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1459177921},{"type":"v1","addr":"172.21.15.116:6834","nonce":1459177921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4242,"qdb_cluster":[4242]},"id":2}]} 2024-04-25T00:03:22.028 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 19 2024-04-25T00:03:22.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.024+0000 7fbb15036640 1 -- 172.21.15.116:0/572571076 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbaf00678f0 msgr2=0x7fbaf0069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:22.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.024+0000 7fbb15036640 1 --2- 172.21.15.116:0/572571076 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbaf00678f0 0x7fbaf0069db0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbb00009790 tx=0x7fbb00008040 comp rx=0 tx=0).stop 2024-04-25T00:03:22.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.024+0000 7fbb15036640 1 -- 172.21.15.116:0/572571076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb100c8970 msgr2=0x7fbb1006f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:22.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.024+0000 7fbb15036640 1 --2- 172.21.15.116:0/572571076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb100c8970 0x7fbb1006f020 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fbaf80097f0 tx=0x7fbaf801abc0 comp rx=0 tx=0).stop 2024-04-25T00:03:22.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.024+0000 7fbb15036640 1 -- 172.21.15.116:0/572571076 shutdown_connections 2024-04-25T00:03:22.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.024+0000 7fbb15036640 1 --2- 172.21.15.116:0/572571076 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbaf00678f0 0x7fbaf0069db0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:22.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.024+0000 7fbb15036640 1 --2- 172.21.15.116:0/572571076 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb10152db0 0x7fbb1006f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:22.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.024+0000 7fbb15036640 1 --2- 172.21.15.116:0/572571076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb100c8970 0x7fbb1006f020 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:22.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.024+0000 7fbb15036640 1 --2- 172.21.15.116:0/572571076 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb100715b0 0x7fbb1014c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:22.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.024+0000 7fbb15036640 1 -- 172.21.15.116:0/572571076 >> 172.21.15.116:0/572571076 conn(0x7fbb100c90d0 msgr2=0x7fbb10153ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:03:22.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.024+0000 7fbb15036640 1 -- 172.21.15.116:0/572571076 shutdown_connections 2024-04-25T00:03:22.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.024+0000 7fbb15036640 1 -- 172.21.15.116:0/572571076 wait complete. 2024-04-25T00:03:22.046 INFO:tasks.mds_thrash.fs.[a]:kill mds.b (rank=0) 2024-04-25T00:03:22.047 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-25T00:03:22.047 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:03:22.048 INFO:tasks.workunit.client.0.smithi116.stdout:mv -f .deps/ffsb_stats.Tpo .deps/ffsb_stats.Po 2024-04-25T00:03:22.070 INFO:tasks.ceph.mds.b:Stopped 2024-04-25T00:03:22.071 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:03:22.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.176+0000 7f4be800d640 1 Processor -- start 2024-04-25T00:03:22.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.176+0000 7f4be800d640 1 -- start start 2024-04-25T00:03:22.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.180+0000 7f4be800d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4be0150d20 0x7f4be00c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:22.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.180+0000 7f4be800d640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f4be00c2f50 0x7f4be01538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:22.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.180+0000 7f4be800d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4be00a94a0 0x7f4be0153e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:22.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.180+0000 7f4be800d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f4be00c3910 con 0x7f4be00a94a0 2024-04-25T00:03:22.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.180+0000 7f4be800d640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f4be00c16e0 con 0x7f4be00c2f50 2024-04-25T00:03:22.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.180+0000 7f4be800d640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f4be00c1400 con 0x7f4be0150d20 2024-04-25T00:03:22.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.180+0000 7f4be65aa640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4be00a94a0 0x7f4be0153e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:22.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.180+0000 7f4be65aa640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4be00a94a0 0x7f4be0153e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59320/0 (socket says 172.21.15.116:59320) 2024-04-25T00:03:22.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.180+0000 7f4be65aa640 1 -- 172.21.15.116:0/3903311147 learned_addr learned my addr 172.21.15.116:0/3903311147 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:03:22.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.180+0000 7f4be55a8640 1 --2- 172.21.15.116:0/3903311147 >> v2:172.21.15.174:3300/0 conn(0x7f4be00c2f50 0x7f4be01538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:22.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.180+0000 7f4be5da9640 1 --2- 172.21.15.116:0/3903311147 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4be0150d20 0x7f4be00c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:22.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.180+0000 7f4be65aa640 1 -- 172.21.15.116:0/3903311147 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4be0150d20 msgr2=0x7f4be00c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:03:22.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.180+0000 7f4be65aa640 1 --2- 172.21.15.116:0/3903311147 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4be0150d20 0x7f4be00c2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:22.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.180+0000 7f4be65aa640 1 -- 172.21.15.116:0/3903311147 >> v2:172.21.15.174:3300/0 conn(0x7f4be00c2f50 msgr2=0x7f4be01538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:03:22.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.180+0000 7f4be65aa640 1 --2- 172.21.15.116:0/3903311147 >> v2:172.21.15.174:3300/0 conn(0x7f4be00c2f50 0x7f4be01538e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:22.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.180+0000 7f4be65aa640 1 -- 172.21.15.116:0/3903311147 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4be006e660 con 0x7f4be00a94a0 2024-04-25T00:03:22.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.180+0000 7f4be65aa640 1 --2- 172.21.15.116:0/3903311147 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4be00a94a0 0x7f4be0153e20 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f4be0070620 tx=0x7f4bd801c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48d7e72209b08725 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:22.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.180+0000 7f4be4da7640 1 -- 172.21.15.116:0/3903311147 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd80a5740 con 0x7f4be00a94a0 2024-04-25T00:03:22.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.180+0000 7f4be4da7640 1 -- 172.21.15.116:0/3903311147 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bd80a58e0 con 0x7f4be00a94a0 2024-04-25T00:03:22.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.180+0000 7f4be4da7640 1 -- 172.21.15.116:0/3903311147 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd80a5cb0 con 0x7f4be00a94a0 2024-04-25T00:03:22.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.180+0000 7f4be800d640 1 -- 172.21.15.116:0/3903311147 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4be00a94a0 msgr2=0x7f4be0153e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:22.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.180+0000 7f4be800d640 1 --2- 172.21.15.116:0/3903311147 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4be00a94a0 0x7f4be0153e20 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f4be0070620 tx=0x7f4bd801c9f0 comp rx=0 tx=0).stop 2024-04-25T00:03:22.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.180+0000 7f4be800d640 1 -- 172.21.15.116:0/3903311147 shutdown_connections 2024-04-25T00:03:22.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.180+0000 7f4be800d640 1 --2- 172.21.15.116:0/3903311147 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4be00a94a0 0x7f4be0153e20 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:22.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.180+0000 7f4be800d640 1 --2- 172.21.15.116:0/3903311147 >> v2:172.21.15.174:3300/0 conn(0x7f4be00c2f50 0x7f4be01538e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:22.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.180+0000 7f4be800d640 1 --2- 172.21.15.116:0/3903311147 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4be0150d20 0x7f4be00c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:22.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.180+0000 7f4be800d640 1 -- 172.21.15.116:0/3903311147 >> 172.21.15.116:0/3903311147 conn(0x7f4be00c3b00 msgr2=0x7f4be00c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:03:22.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.180+0000 7f4be800d640 1 -- 172.21.15.116:0/3903311147 shutdown_connections 2024-04-25T00:03:22.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.180+0000 7f4be800d640 1 -- 172.21.15.116:0/3903311147 wait complete. 2024-04-25T00:03:22.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.184+0000 7f4be800d640 1 Processor -- start 2024-04-25T00:03:22.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.184+0000 7f4be800d640 1 -- start start 2024-04-25T00:03:22.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.184+0000 7f4be800d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4be00a94a0 0x7f4be0146cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:22.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.184+0000 7f4be800d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4be00c2f50 0x7f4be0147200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:22.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.184+0000 7f4be800d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4be0150d20 0x7f4be013d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:22.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.184+0000 7f4be800d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4be0157340 con 0x7f4be00c2f50 2024-04-25T00:03:22.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.184+0000 7f4be800d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4be0154570 con 0x7f4be00a94a0 2024-04-25T00:03:22.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.184+0000 7f4be800d640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f4be01574c0 con 0x7f4be0150d20 2024-04-25T00:03:22.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.184+0000 7f4be55a8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4be00c2f50 0x7f4be0147200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:22.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.184+0000 7f4be55a8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4be00c2f50 0x7f4be0147200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59334/0 (socket says 172.21.15.116:59334) 2024-04-25T00:03:22.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.184+0000 7f4be55a8640 1 -- 172.21.15.116:0/669596671 learned_addr learned my addr 172.21.15.116:0/669596671 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:03:22.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.184+0000 7f4be5da9640 1 --2- 172.21.15.116:0/669596671 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4be00a94a0 0x7f4be0146cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:22.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.184+0000 7f4be65aa640 1 --2- 172.21.15.116:0/669596671 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4be0150d20 0x7f4be013d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:22.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.184+0000 7f4be55a8640 1 -- 172.21.15.116:0/669596671 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4be0150d20 msgr2=0x7f4be013d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:22.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.184+0000 7f4be55a8640 1 --2- 172.21.15.116:0/669596671 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4be0150d20 0x7f4be013d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:22.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.184+0000 7f4be55a8640 1 -- 172.21.15.116:0/669596671 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4be00a94a0 msgr2=0x7f4be0146cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:22.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.184+0000 7f4be55a8640 1 --2- 172.21.15.116:0/669596671 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4be00a94a0 0x7f4be0146cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:22.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.184+0000 7f4be55a8640 1 -- 172.21.15.116:0/669596671 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4be014c960 con 0x7f4be00c2f50 2024-04-25T00:03:22.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.184+0000 7f4be5da9640 1 --2- 172.21.15.116:0/669596671 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4be00a94a0 0x7f4be0146cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:03:22.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.184+0000 7f4be55a8640 1 --2- 172.21.15.116:0/669596671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4be00c2f50 0x7f4be0147200 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f4bd000c7f0 tx=0x7f4bd000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:22.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.184+0000 7f4bd67fc640 1 -- 172.21.15.116:0/669596671 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd0015070 con 0x7f4be00c2f50 2024-04-25T00:03:22.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.184+0000 7f4bd67fc640 1 -- 172.21.15.116:0/669596671 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bd0004220 con 0x7f4be00c2f50 2024-04-25T00:03:22.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.184+0000 7f4be800d640 1 -- 172.21.15.116:0/669596671 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4be014cc40 con 0x7f4be00c2f50 2024-04-25T00:03:22.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.184+0000 7f4bd67fc640 1 -- 172.21.15.116:0/669596671 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd0003620 con 0x7f4be00c2f50 2024-04-25T00:03:22.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.184+0000 7f4be800d640 1 -- 172.21.15.116:0/669596671 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4be01467a0 con 0x7f4be00c2f50 2024-04-25T00:03:22.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.184+0000 7f4be800d640 1 -- 172.21.15.116:0/669596671 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4be01469a0 con 0x7f4be00c2f50 2024-04-25T00:03:22.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.184+0000 7f4bd67fc640 1 -- 172.21.15.116:0/669596671 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4bd00039d0 con 0x7f4be00c2f50 2024-04-25T00:03:22.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.188+0000 7f4bd67fc640 1 --2- 172.21.15.116:0/669596671 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4bc40678c0 0x7f4bc4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:22.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.188+0000 7f4bd67fc640 1 -- 172.21.15.116:0/669596671 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6752+0+0 (secure 0 0 0) 0x7f4bd0086050 con 0x7f4be00c2f50 2024-04-25T00:03:22.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.188+0000 7f4be5da9640 1 --2- 172.21.15.116:0/669596671 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4bc40678c0 0x7f4bc4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:22.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.188+0000 7f4be5da9640 1 --2- 172.21.15.116:0/669596671 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4bc40678c0 0x7f4bc4069d80 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc006fd0 tx=0x7f4bdc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:22.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.192+0000 7f4bd67fc640 1 -- 172.21.15.116:0/669596671 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f4be01469a0 con 0x7f4be00c2f50 2024-04-25T00:03:22.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.388+0000 7f4be800d640 1 -- 172.21.15.116:0/669596671 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4be0068350 con 0x7f4be00c2f50 2024-04-25T00:03:22.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.388+0000 7f4bd67fc640 1 -- 172.21.15.116:0/669596671 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6498 (secure 0 0 0) 0x7f4bd0056810 con 0x7f4be00c2f50 2024-04-25T00:03:22.392 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:03:22.392 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4735,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/2004876235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":2004876235},{"type":"v1","addr":"172.21.15.116:6835","nonce":2004876235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:01:08.748957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6837/1762356055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1762356055},{"type":"v1","addr":"172.21.15.174:6837","nonce":1762356055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270]},"id":1},{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:02:43.411054+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4242":{"gid":4242,"name":"c","rank":0,"incarnation":15,"state":"up:active","state_seq":29,"addr":"172.21.15.116:6834/1459177921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1459177921},{"type":"v1","addr":"172.21.15.116:6834","nonce":1459177921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4242,"qdb_cluster":[4242]},"id":2}]} 2024-04-25T00:03:22.392 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 19 2024-04-25T00:03:22.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.388+0000 7f4be800d640 1 -- 172.21.15.116:0/669596671 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4bc40678c0 msgr2=0x7f4bc4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:22.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.388+0000 7f4be800d640 1 --2- 172.21.15.116:0/669596671 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4bc40678c0 0x7f4bc4069d80 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc006fd0 tx=0x7f4bdc008040 comp rx=0 tx=0).stop 2024-04-25T00:03:22.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.388+0000 7f4be800d640 1 -- 172.21.15.116:0/669596671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4be00c2f50 msgr2=0x7f4be0147200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:22.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.388+0000 7f4be800d640 1 --2- 172.21.15.116:0/669596671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4be00c2f50 0x7f4be0147200 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f4bd000c7f0 tx=0x7f4bd000ccb0 comp rx=0 tx=0).stop 2024-04-25T00:03:22.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.392+0000 7f4be800d640 1 -- 172.21.15.116:0/669596671 shutdown_connections 2024-04-25T00:03:22.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.392+0000 7f4be800d640 1 --2- 172.21.15.116:0/669596671 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4bc40678c0 0x7f4bc4069d80 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:22.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.392+0000 7f4be800d640 1 --2- 172.21.15.116:0/669596671 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4be0150d20 0x7f4be013d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:22.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.392+0000 7f4be800d640 1 --2- 172.21.15.116:0/669596671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4be00c2f50 0x7f4be0147200 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:22.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.392+0000 7f4be800d640 1 --2- 172.21.15.116:0/669596671 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4be00a94a0 0x7f4be0146cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:22.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.392+0000 7f4be800d640 1 -- 172.21.15.116:0/669596671 >> 172.21.15.116:0/669596671 conn(0x7f4be00c3b00 msgr2=0x7f4be00ac7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:03:22.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.392+0000 7f4be800d640 1 -- 172.21.15.116:0/669596671 shutdown_connections 2024-04-25T00:03:22.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:22.392+0000 7f4be800d640 1 -- 172.21.15.116:0/669596671 wait complete. 2024-04-25T00:03:22.412 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T00:03:23.268 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:03:23.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.392+0000 7fa9ff576640 1 Processor -- start 2024-04-25T00:03:23.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.392+0000 7fa9ff576640 1 -- start start 2024-04-25T00:03:23.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.396+0000 7fa9ff576640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa9f80a99f0 0x7fa9f8153ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:23.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.396+0000 7fa9ff576640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9f80c3910 con 0x7fa9f8150d20 2024-04-25T00:03:23.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.396+0000 7fa9ff576640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9f8154530 con 0x7fa9f80a94a0 2024-04-25T00:03:23.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.396+0000 7fa9ff576640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fa9f80c16e0 con 0x7fa9f80a99f0 2024-04-25T00:03:23.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.396+0000 7fa9fd312640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fa9f8150d20 0x7fa9f80c2a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:58320/0 (socket says 172.21.15.116:58320) 2024-04-25T00:03:23.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.396+0000 7fa9fd312640 1 -- 172.21.15.116:0/1225535437 learned_addr learned my addr 172.21.15.116:0/1225535437 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:03:23.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.396+0000 7fa9fdb13640 1 --2- 172.21.15.116:0/1225535437 >> v2:172.21.15.116:3300/0 conn(0x7fa9f80a99f0 0x7fa9f8153ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:23.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.396+0000 7fa9effff640 1 -- 172.21.15.116:0/1225535437 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3085958072 0 0) 0x7fa9f80c3910 con 0x7fa9f8150d20 2024-04-25T00:03:23.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.396+0000 7fa9effff640 1 -- 172.21.15.116:0/1225535437 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa9e4003530 con 0x7fa9f8150d20 2024-04-25T00:03:23.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.396+0000 7fa9fdb13640 1 -- 172.21.15.116:0/1225535437 >> v1:172.21.15.174:6789/0 conn(0x7fa9f80a94a0 legacy=0x7fa9f81538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:03:23.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.396+0000 7fa9fdb13640 1 -- 172.21.15.116:0/1225535437 >> v1:172.21.15.116:6789/0 conn(0x7fa9f8150d20 legacy=0x7fa9f80c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:23.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.396+0000 7fa9fdb13640 1 -- 172.21.15.116:0/1225535437 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9f80c1400 con 0x7fa9f80a99f0 2024-04-25T00:03:23.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.396+0000 7fa9fdb13640 1 --2- 172.21.15.116:0/1225535437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9f80a99f0 0x7fa9f8153ff0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fa9f0009d90 tx=0x7fa9f001c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87204ba8160f2cb6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:23.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.396+0000 7fa9effff640 1 -- 172.21.15.116:0/1225535437 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f00b2070 con 0x7fa9f80a99f0 2024-04-25T00:03:23.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.396+0000 7fa9effff640 1 -- 172.21.15.116:0/1225535437 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9f00a5830 con 0x7fa9f80a99f0 2024-04-25T00:03:23.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.396+0000 7fa9effff640 1 -- 172.21.15.116:0/1225535437 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f00a5c00 con 0x7fa9f80a99f0 2024-04-25T00:03:23.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.400+0000 7fa9ff576640 1 -- 172.21.15.116:0/1225535437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9f80a99f0 msgr2=0x7fa9f8153ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:23.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.400+0000 7fa9ff576640 1 --2- 172.21.15.116:0/1225535437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9f80a99f0 0x7fa9f8153ff0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fa9f0009d90 tx=0x7fa9f001c970 comp rx=0 tx=0).stop 2024-04-25T00:03:23.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.400+0000 7fa9ff576640 1 -- 172.21.15.116:0/1225535437 shutdown_connections 2024-04-25T00:03:23.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.400+0000 7fa9ff576640 1 --2- 172.21.15.116:0/1225535437 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9f80a99f0 0x7fa9f8153ff0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:23.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.400+0000 7fa9ff576640 1 -- 172.21.15.116:0/1225535437 >> 172.21.15.116:0/1225535437 conn(0x7fa9f80c3b00 msgr2=0x7fa9f80c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:03:23.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.400+0000 7fa9ff576640 1 -- 172.21.15.116:0/1225535437 shutdown_connections 2024-04-25T00:03:23.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.400+0000 7fa9ff576640 1 -- 172.21.15.116:0/1225535437 wait complete. 2024-04-25T00:03:23.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.400+0000 7fa9ff576640 1 Processor -- start 2024-04-25T00:03:23.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.400+0000 7fa9ff576640 1 -- start start 2024-04-25T00:03:23.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.400+0000 7fa9ff576640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9f80a94a0 0x7fa9f8142590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:23.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.400+0000 7fa9ff576640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9f80a99f0 0x7fa9f813c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:23.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.400+0000 7fa9ff576640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa9f8150d20 0x7fa9f813c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:23.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.400+0000 7fa9ff576640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa9f80c32a0 con 0x7fa9f80a99f0 2024-04-25T00:03:23.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.400+0000 7fa9ff576640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa9f80c3120 con 0x7fa9f80a94a0 2024-04-25T00:03:23.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.400+0000 7fa9ff576640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa9f8161740 con 0x7fa9f8150d20 2024-04-25T00:03:23.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.400+0000 7fa9fcb11640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9f80a99f0 0x7fa9f813c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:23.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.400+0000 7fa9fcb11640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9f80a99f0 0x7fa9f813c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59362/0 (socket says 172.21.15.116:59362) 2024-04-25T00:03:23.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.400+0000 7fa9fcb11640 1 -- 172.21.15.116:0/2267036388 learned_addr learned my addr 172.21.15.116:0/2267036388 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:03:23.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.400+0000 7fa9fd312640 1 --2- 172.21.15.116:0/2267036388 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9f80a94a0 0x7fa9f8142590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:23.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.400+0000 7fa9fdb13640 1 --2- 172.21.15.116:0/2267036388 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa9f8150d20 0x7fa9f813c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:23.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.400+0000 7fa9fcb11640 1 -- 172.21.15.116:0/2267036388 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa9f8150d20 msgr2=0x7fa9f813c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:23.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.400+0000 7fa9fcb11640 1 --2- 172.21.15.116:0/2267036388 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa9f8150d20 0x7fa9f813c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:23.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.400+0000 7fa9fcb11640 1 -- 172.21.15.116:0/2267036388 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9f80a94a0 msgr2=0x7fa9f8142590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:23.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.400+0000 7fa9fcb11640 1 --2- 172.21.15.116:0/2267036388 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9f80a94a0 0x7fa9f8142590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:23.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.400+0000 7fa9fcb11640 1 -- 172.21.15.116:0/2267036388 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9f81433f0 con 0x7fa9f80a99f0 2024-04-25T00:03:23.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.400+0000 7fa9fcb11640 1 --2- 172.21.15.116:0/2267036388 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9f80a99f0 0x7fa9f813c340 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fa9e800c420 tx=0x7fa9e800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:23.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.400+0000 7fa9edffb640 1 -- 172.21.15.116:0/2267036388 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9e80052d0 con 0x7fa9f80a99f0 2024-04-25T00:03:23.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.400+0000 7fa9edffb640 1 -- 172.21.15.116:0/2267036388 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9e8005470 con 0x7fa9f80a99f0 2024-04-25T00:03:23.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.400+0000 7fa9ff576640 1 -- 172.21.15.116:0/2267036388 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9f8144b80 con 0x7fa9f80a99f0 2024-04-25T00:03:23.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.400+0000 7fa9edffb640 1 -- 172.21.15.116:0/2267036388 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9e8007260 con 0x7fa9f80a99f0 2024-04-25T00:03:23.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.400+0000 7fa9ff576640 1 -- 172.21.15.116:0/2267036388 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9f8145140 con 0x7fa9f80a99f0 2024-04-25T00:03:23.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.400+0000 7fa9ff576640 1 -- 172.21.15.116:0/2267036388 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9f8068350 con 0x7fa9f80a99f0 2024-04-25T00:03:23.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.404+0000 7fa9edffb640 1 -- 172.21.15.116:0/2267036388 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa9e8007400 con 0x7fa9f80a99f0 2024-04-25T00:03:23.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.404+0000 7fa9edffb640 1 --2- 172.21.15.116:0/2267036388 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa9dc0678c0 0x7fa9dc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:23.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.404+0000 7fa9edffb640 1 -- 172.21.15.116:0/2267036388 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6752+0+0 (secure 0 0 0) 0x7fa9e80855a0 con 0x7fa9f80a99f0 2024-04-25T00:03:23.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.404+0000 7fa9fd312640 1 --2- 172.21.15.116:0/2267036388 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa9dc0678c0 0x7fa9dc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:23.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.404+0000 7fa9fd312640 1 --2- 172.21.15.116:0/2267036388 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa9dc0678c0 0x7fa9dc069d80 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa9f40011f0 tx=0x7fa9f4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:23.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.408+0000 7fa9edffb640 1 -- 172.21.15.116:0/2267036388 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fa9e8055cd0 con 0x7fa9f80a99f0 2024-04-25T00:03:23.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.604+0000 7fa9ff576640 1 -- 172.21.15.116:0/2267036388 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa9f80c34f0 con 0x7fa9f80a99f0 2024-04-25T00:03:23.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.604+0000 7fa9edffb640 1 -- 172.21.15.116:0/2267036388 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6498 (secure 0 0 0) 0x7fa9e80555d0 con 0x7fa9f80a99f0 2024-04-25T00:03:23.608 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:03:23.609 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4735,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/2004876235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":2004876235},{"type":"v1","addr":"172.21.15.116:6835","nonce":2004876235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:01:08.748957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6837/1762356055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1762356055},{"type":"v1","addr":"172.21.15.174:6837","nonce":1762356055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270]},"id":1},{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:02:43.411054+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4242":{"gid":4242,"name":"c","rank":0,"incarnation":15,"state":"up:active","state_seq":29,"addr":"172.21.15.116:6834/1459177921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1459177921},{"type":"v1","addr":"172.21.15.116:6834","nonce":1459177921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4242,"qdb_cluster":[4242]},"id":2}]} 2024-04-25T00:03:23.609 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 19 2024-04-25T00:03:23.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.608+0000 7fa9ff576640 1 -- 172.21.15.116:0/2267036388 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa9dc0678c0 msgr2=0x7fa9dc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:23.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.608+0000 7fa9ff576640 1 --2- 172.21.15.116:0/2267036388 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa9dc0678c0 0x7fa9dc069d80 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa9f40011f0 tx=0x7fa9f4009040 comp rx=0 tx=0).stop 2024-04-25T00:03:23.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.608+0000 7fa9ff576640 1 -- 172.21.15.116:0/2267036388 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9f80a99f0 msgr2=0x7fa9f813c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:23.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.608+0000 7fa9ff576640 1 --2- 172.21.15.116:0/2267036388 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9f80a99f0 0x7fa9f813c340 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fa9e800c420 tx=0x7fa9e800c8e0 comp rx=0 tx=0).stop 2024-04-25T00:03:23.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.608+0000 7fa9ff576640 1 -- 172.21.15.116:0/2267036388 shutdown_connections 2024-04-25T00:03:23.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.608+0000 7fa9ff576640 1 --2- 172.21.15.116:0/2267036388 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa9dc0678c0 0x7fa9dc069d80 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:23.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.608+0000 7fa9ff576640 1 --2- 172.21.15.116:0/2267036388 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa9f8150d20 0x7fa9f813c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:23.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.608+0000 7fa9ff576640 1 --2- 172.21.15.116:0/2267036388 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9f80a99f0 0x7fa9f813c340 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:23.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.608+0000 7fa9ff576640 1 --2- 172.21.15.116:0/2267036388 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9f80a94a0 0x7fa9f8142590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:23.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.608+0000 7fa9ff576640 1 -- 172.21.15.116:0/2267036388 >> 172.21.15.116:0/2267036388 conn(0x7fa9f80c3b00 msgr2=0x7fa9f8146e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:03:23.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.608+0000 7fa9ff576640 1 -- 172.21.15.116:0/2267036388 shutdown_connections 2024-04-25T00:03:23.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:23.608+0000 7fa9ff576640 1 -- 172.21.15.116:0/2267036388 wait complete. 2024-04-25T00:03:23.627 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:03:24.413 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:03:24.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.540+0000 7fdae8197640 1 Processor -- start 2024-04-25T00:03:24.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.540+0000 7fdae8197640 1 -- start start 2024-04-25T00:03:24.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.540+0000 7fdae8197640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fdae0153f40 0x7fdae006e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:24.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.540+0000 7fdae8197640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdae00c2a10 0x7fdae00c2e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:24.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.540+0000 7fdae8197640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdae0071a30 con 0x7fdae0159f00 2024-04-25T00:03:24.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.540+0000 7fdae8197640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fdae00725d0 con 0x7fdae0153f40 2024-04-25T00:03:24.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.540+0000 7fdae8197640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fdae0071ca0 con 0x7fdae00c2a10 2024-04-25T00:03:24.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.540+0000 7fdae5732640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdae00c2a10 0x7fdae00c2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:24.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.540+0000 7fdae5f33640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fdae0153f40 0x7fdae006e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:24.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.544+0000 7fdae5732640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdae00c2a10 0x7fdae00c2e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:41642/0 (socket says 172.21.15.116:41642) 2024-04-25T00:03:24.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.544+0000 7fdae5732640 1 -- 172.21.15.116:0/4019147578 learned_addr learned my addr 172.21.15.116:0/4019147578 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:03:24.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.544+0000 7fdae6734640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fdae0159f00 0x7fdae015a390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:46994/0 (socket says 172.21.15.116:46994) 2024-04-25T00:03:24.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.544+0000 7fdae5f33640 1 -- 172.21.15.116:0/4019147578 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdae00c2a10 msgr2=0x7fdae00c2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:03:24.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.544+0000 7fdae5f33640 1 --2- 172.21.15.116:0/4019147578 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdae00c2a10 0x7fdae00c2e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:24.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.544+0000 7fdae5f33640 1 -- 172.21.15.116:0/4019147578 >> v1:172.21.15.174:6789/0 conn(0x7fdae0159f00 legacy=0x7fdae015a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:24.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.544+0000 7fdae5f33640 1 -- 172.21.15.116:0/4019147578 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdae0071f80 con 0x7fdae0153f40 2024-04-25T00:03:24.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.544+0000 7fdae5f33640 1 --2- 172.21.15.116:0/4019147578 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdae0153f40 0x7fdae006e790 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fdadc009d90 tx=0x7fdadc01c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=933e1f70da0e4c3e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:24.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.544+0000 7fdae4f31640 1 -- 172.21.15.116:0/4019147578 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdadc0b2070 con 0x7fdae0153f40 2024-04-25T00:03:24.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.544+0000 7fdae4f31640 1 -- 172.21.15.116:0/4019147578 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdadc0a57b0 con 0x7fdae0153f40 2024-04-25T00:03:24.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.544+0000 7fdae4f31640 1 -- 172.21.15.116:0/4019147578 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdadc0a5b80 con 0x7fdae0153f40 2024-04-25T00:03:24.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.544+0000 7fdae8197640 1 -- 172.21.15.116:0/4019147578 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdae0153f40 msgr2=0x7fdae006e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:24.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.544+0000 7fdae8197640 1 --2- 172.21.15.116:0/4019147578 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdae0153f40 0x7fdae006e790 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fdadc009d90 tx=0x7fdadc01c8e0 comp rx=0 tx=0).stop 2024-04-25T00:03:24.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.544+0000 7fdae8197640 1 -- 172.21.15.116:0/4019147578 shutdown_connections 2024-04-25T00:03:24.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.544+0000 7fdae8197640 1 --2- 172.21.15.116:0/4019147578 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdae00c2a10 0x7fdae00c2e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:24.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.544+0000 7fdae8197640 1 --2- 172.21.15.116:0/4019147578 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdae0153f40 0x7fdae006e790 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:24.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.544+0000 7fdae8197640 1 -- 172.21.15.116:0/4019147578 >> 172.21.15.116:0/4019147578 conn(0x7fdae00c3b00 msgr2=0x7fdae00c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:03:24.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.544+0000 7fdae8197640 1 -- 172.21.15.116:0/4019147578 shutdown_connections 2024-04-25T00:03:24.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.544+0000 7fdae8197640 1 -- 172.21.15.116:0/4019147578 wait complete. 2024-04-25T00:03:24.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.548+0000 7fdae8197640 1 Processor -- start 2024-04-25T00:03:24.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.548+0000 7fdae8197640 1 -- start start 2024-04-25T00:03:24.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.548+0000 7fdae8197640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdae00c2a10 0x7fdae0148750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:24.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.548+0000 7fdae8197640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdae0153f40 0x7fdae01450d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:24.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.548+0000 7fdae8197640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdae0159f00 0x7fdae0145610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:24.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.548+0000 7fdae8197640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdae00c36c0 con 0x7fdae00c2a10 2024-04-25T00:03:24.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.548+0000 7fdae8197640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fdae006eee0 con 0x7fdae0159f00 2024-04-25T00:03:24.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.548+0000 7fdae8197640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fdae00c3840 con 0x7fdae0153f40 2024-04-25T00:03:24.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.548+0000 7fdae5f33640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdae00c2a10 0x7fdae0148750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:24.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.548+0000 7fdae5f33640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdae00c2a10 0x7fdae0148750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59370/0 (socket says 172.21.15.116:59370) 2024-04-25T00:03:24.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.548+0000 7fdae5f33640 1 -- 172.21.15.116:0/1243897405 learned_addr learned my addr 172.21.15.116:0/1243897405 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:03:24.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.548+0000 7fdae6734640 1 --2- 172.21.15.116:0/1243897405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdae0159f00 0x7fdae0145610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:24.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.548+0000 7fdae5732640 1 --2- 172.21.15.116:0/1243897405 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdae0153f40 0x7fdae01450d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:24.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.548+0000 7fdae5f33640 1 -- 172.21.15.116:0/1243897405 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdae0153f40 msgr2=0x7fdae01450d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:24.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.548+0000 7fdae5f33640 1 --2- 172.21.15.116:0/1243897405 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdae0153f40 0x7fdae01450d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:24.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.548+0000 7fdae5f33640 1 -- 172.21.15.116:0/1243897405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdae0159f00 msgr2=0x7fdae0145610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:24.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.548+0000 7fdae5f33640 1 --2- 172.21.15.116:0/1243897405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdae0159f00 0x7fdae0145610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:24.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.548+0000 7fdae5f33640 1 -- 172.21.15.116:0/1243897405 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdae00c4de0 con 0x7fdae00c2a10 2024-04-25T00:03:24.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.548+0000 7fdae5f33640 1 --2- 172.21.15.116:0/1243897405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdae00c2a10 0x7fdae0148750 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fdadc009d90 tx=0x7fdadc019680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:24.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.548+0000 7fdad67fc640 1 -- 172.21.15.116:0/1243897405 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdadc0bd020 con 0x7fdae00c2a10 2024-04-25T00:03:24.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.548+0000 7fdad67fc640 1 -- 172.21.15.116:0/1243897405 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdadc0b8470 con 0x7fdae00c2a10 2024-04-25T00:03:24.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.548+0000 7fdae8197640 1 -- 172.21.15.116:0/1243897405 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdae0145b50 con 0x7fdae00c2a10 2024-04-25T00:03:24.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.548+0000 7fdad67fc640 1 -- 172.21.15.116:0/1243897405 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdadc0b2040 con 0x7fdae00c2a10 2024-04-25T00:03:24.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.548+0000 7fdae8197640 1 -- 172.21.15.116:0/1243897405 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdae00c5080 con 0x7fdae00c2a10 2024-04-25T00:03:24.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.548+0000 7fdae8197640 1 -- 172.21.15.116:0/1243897405 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdae0068350 con 0x7fdae00c2a10 2024-04-25T00:03:24.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.552+0000 7fdad67fc640 1 -- 172.21.15.116:0/1243897405 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fdadc0c0020 con 0x7fdae00c2a10 2024-04-25T00:03:24.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.552+0000 7fdad67fc640 1 --2- 172.21.15.116:0/1243897405 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdac40678c0 0x7fdac4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:24.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.552+0000 7fdad67fc640 1 -- 172.21.15.116:0/1243897405 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6752+0+0 (secure 0 0 0) 0x7fdadc120f70 con 0x7fdae00c2a10 2024-04-25T00:03:24.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.552+0000 7fdae5732640 1 --2- 172.21.15.116:0/1243897405 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdac40678c0 0x7fdac4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:24.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.552+0000 7fdae5732640 1 --2- 172.21.15.116:0/1243897405 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdac40678c0 0x7fdac4069d80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fdae0149170 tx=0x7fdad0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:24.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.556+0000 7fdad67fc640 1 -- 172.21.15.116:0/1243897405 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fdadc0c7030 con 0x7fdae00c2a10 2024-04-25T00:03:24.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.748+0000 7fdae8197640 1 -- 172.21.15.116:0/1243897405 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdae014f1f0 con 0x7fdae00c2a10 2024-04-25T00:03:24.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.748+0000 7fdad67fc640 1 -- 172.21.15.116:0/1243897405 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6498 (secure 0 0 0) 0x7fdadc016560 con 0x7fdae00c2a10 2024-04-25T00:03:24.755 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:03:24.755 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4735,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/2004876235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":2004876235},{"type":"v1","addr":"172.21.15.116:6835","nonce":2004876235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:01:08.748957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6837/1762356055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1762356055},{"type":"v1","addr":"172.21.15.174:6837","nonce":1762356055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270]},"id":1},{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:02:43.411054+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4242":{"gid":4242,"name":"c","rank":0,"incarnation":15,"state":"up:active","state_seq":29,"addr":"172.21.15.116:6834/1459177921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1459177921},{"type":"v1","addr":"172.21.15.116:6834","nonce":1459177921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4242,"qdb_cluster":[4242]},"id":2}]} 2024-04-25T00:03:24.755 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 19 2024-04-25T00:03:24.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.752+0000 7fdae8197640 1 -- 172.21.15.116:0/1243897405 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdac40678c0 msgr2=0x7fdac4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:24.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.752+0000 7fdae8197640 1 --2- 172.21.15.116:0/1243897405 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdac40678c0 0x7fdac4069d80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fdae0149170 tx=0x7fdad0008040 comp rx=0 tx=0).stop 2024-04-25T00:03:24.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.752+0000 7fdae8197640 1 -- 172.21.15.116:0/1243897405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdae00c2a10 msgr2=0x7fdae0148750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:24.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.752+0000 7fdae8197640 1 --2- 172.21.15.116:0/1243897405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdae00c2a10 0x7fdae0148750 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fdadc009d90 tx=0x7fdadc019680 comp rx=0 tx=0).stop 2024-04-25T00:03:24.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.752+0000 7fdae8197640 1 -- 172.21.15.116:0/1243897405 shutdown_connections 2024-04-25T00:03:24.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.752+0000 7fdae8197640 1 --2- 172.21.15.116:0/1243897405 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdac40678c0 0x7fdac4069d80 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:24.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.752+0000 7fdae8197640 1 --2- 172.21.15.116:0/1243897405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdae0159f00 0x7fdae0145610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:24.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.752+0000 7fdae8197640 1 --2- 172.21.15.116:0/1243897405 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdae0153f40 0x7fdae01450d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:24.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.752+0000 7fdae8197640 1 --2- 172.21.15.116:0/1243897405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdae00c2a10 0x7fdae0148750 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:24.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.752+0000 7fdae8197640 1 -- 172.21.15.116:0/1243897405 >> 172.21.15.116:0/1243897405 conn(0x7fdae00c3b00 msgr2=0x7fdae014fea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:03:24.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.752+0000 7fdae8197640 1 -- 172.21.15.116:0/1243897405 shutdown_connections 2024-04-25T00:03:24.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:24.752+0000 7fdae8197640 1 -- 172.21.15.116:0/1243897405 wait complete. 2024-04-25T00:03:24.774 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T00:03:25.628 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:03:25.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.756+0000 7ff22fe83640 1 Processor -- start 2024-04-25T00:03:25.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.756+0000 7ff22fe83640 1 -- start start 2024-04-25T00:03:25.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.756+0000 7ff22fe83640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff228155790 0x7ff228156ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:25.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.756+0000 7ff22fe83640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff228157400 0x7ff228157810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:25.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.756+0000 7ff22fe83640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2280c9390 con 0x7ff2280b07d0 2024-04-25T00:03:25.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.756+0000 7ff22fe83640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7ff2280cadb0 con 0x7ff228155790 2024-04-25T00:03:25.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.756+0000 7ff22fe83640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff2280c9670 con 0x7ff228157400 2024-04-25T00:03:25.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.756+0000 7ff22dc1f640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7ff2280b07d0 0x7ff22806bd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:33116/0 (socket says 172.21.15.116:33116) 2024-04-25T00:03:25.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.756+0000 7ff22dc1f640 1 -- 172.21.15.116:0/2889666408 learned_addr learned my addr 172.21.15.116:0/2889666408 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:03:25.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.756+0000 7ff22d41e640 1 --2- 172.21.15.116:0/2889666408 >> v2:172.21.15.174:3300/0 conn(0x7ff228155790 0x7ff228156ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:25.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.756+0000 7ff22e420640 1 --2- 172.21.15.116:0/2889666408 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff228157400 0x7ff228157810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:25.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.756+0000 7ff22cc1d640 1 -- 172.21.15.116:0/2889666408 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 736363697 0 0) 0x7ff2280c9390 con 0x7ff2280b07d0 2024-04-25T00:03:25.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.756+0000 7ff22cc1d640 1 -- 172.21.15.116:0/2889666408 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff214003530 con 0x7ff2280b07d0 2024-04-25T00:03:25.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.756+0000 7ff22cc1d640 1 -- 172.21.15.116:0/2889666408 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2733191741 0 0) 0x7ff214003530 con 0x7ff2280b07d0 2024-04-25T00:03:25.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.756+0000 7ff22cc1d640 1 -- 172.21.15.116:0/2889666408 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff228157400 msgr2=0x7ff228157810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:03:25.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.756+0000 7ff22cc1d640 1 --2- 172.21.15.116:0/2889666408 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff228157400 0x7ff228157810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:25.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.756+0000 7ff22cc1d640 1 -- 172.21.15.116:0/2889666408 >> v2:172.21.15.174:3300/0 conn(0x7ff228155790 msgr2=0x7ff228156ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:03:25.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.756+0000 7ff22cc1d640 1 --2- 172.21.15.116:0/2889666408 >> v2:172.21.15.174:3300/0 conn(0x7ff228155790 0x7ff228156ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:25.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.756+0000 7ff22cc1d640 1 -- 172.21.15.116:0/2889666408 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2280c9950 con 0x7ff2280b07d0 2024-04-25T00:03:25.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.756+0000 7ff22cc1d640 1 -- 172.21.15.116:0/2889666408 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7ff224001680 con 0x7ff2280b07d0 2024-04-25T00:03:25.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.756+0000 7ff22cc1d640 1 -- 172.21.15.116:0/2889666408 >> v1:172.21.15.116:6789/0 conn(0x7ff2280b07d0 legacy=0x7ff22806bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:25.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.756+0000 7ff22cc1d640 1 --2- 172.21.15.116:0/2889666408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2140039c0 0x7ff214003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:25.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.756+0000 7ff22cc1d640 1 --2- 172.21.15.116:0/2889666408 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff214004810 0x7ff214004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:25.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.756+0000 7ff22cc1d640 1 --2- 172.21.15.116:0/2889666408 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff228157400 0x7ff2140051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:25.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.756+0000 7ff22cc1d640 1 -- 172.21.15.116:0/2889666408 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff2280c9670 con 0x7ff2140039c0 2024-04-25T00:03:25.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.756+0000 7ff22cc1d640 1 -- 172.21.15.116:0/2889666408 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff21400b9d0 con 0x7ff214004810 2024-04-25T00:03:25.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.756+0000 7ff22cc1d640 1 -- 172.21.15.116:0/2889666408 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff21400bb50 con 0x7ff228157400 2024-04-25T00:03:25.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.756+0000 7ff22dc1f640 1 --2- 172.21.15.116:0/2889666408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2140039c0 0x7ff214003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:25.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.756+0000 7ff22d41e640 1 --2- 172.21.15.116:0/2889666408 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff214004810 0x7ff214004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:25.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.756+0000 7ff22dc1f640 1 -- 172.21.15.116:0/2889666408 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff228157400 msgr2=0x7ff2140051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:03:25.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.756+0000 7ff22dc1f640 1 --2- 172.21.15.116:0/2889666408 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff228157400 0x7ff2140051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:25.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.756+0000 7ff22dc1f640 1 -- 172.21.15.116:0/2889666408 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff214004810 msgr2=0x7ff214004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:03:25.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.756+0000 7ff22dc1f640 1 --2- 172.21.15.116:0/2889666408 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff214004810 0x7ff214004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:25.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.756+0000 7ff22dc1f640 1 -- 172.21.15.116:0/2889666408 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff214003530 con 0x7ff2140039c0 2024-04-25T00:03:25.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.756+0000 7ff22d41e640 1 --2- 172.21.15.116:0/2889666408 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff214004810 0x7ff214004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:03:25.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.760+0000 7ff22dc1f640 1 --2- 172.21.15.116:0/2889666408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2140039c0 0x7ff214003de0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7ff22400c240 tx=0x7ff22401cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8afb8df8e693ddc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:25.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.760+0000 7ff22cc1d640 1 -- 172.21.15.116:0/2889666408 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff224007d60 con 0x7ff2140039c0 2024-04-25T00:03:25.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.760+0000 7ff22cc1d640 1 -- 172.21.15.116:0/2889666408 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2240a9020 con 0x7ff2140039c0 2024-04-25T00:03:25.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.760+0000 7ff22cc1d640 1 -- 172.21.15.116:0/2889666408 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff224005b20 con 0x7ff2140039c0 2024-04-25T00:03:25.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.760+0000 7ff22fe83640 1 -- 172.21.15.116:0/2889666408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2140039c0 msgr2=0x7ff214003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:25.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.760+0000 7ff22fe83640 1 --2- 172.21.15.116:0/2889666408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2140039c0 0x7ff214003de0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7ff22400c240 tx=0x7ff22401cc80 comp rx=0 tx=0).stop 2024-04-25T00:03:25.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.760+0000 7ff22e420640 1 -- 172.21.15.116:0/2889666408 reap_dead start 2024-04-25T00:03:25.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.760+0000 7ff22fe83640 1 -- 172.21.15.116:0/2889666408 shutdown_connections 2024-04-25T00:03:25.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.760+0000 7ff22fe83640 1 -- 172.21.15.116:0/2889666408 >> 172.21.15.116:0/2889666408 conn(0x7ff2280acdd0 msgr2=0x7ff2280ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:03:25.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.760+0000 7ff22fe83640 1 -- 172.21.15.116:0/2889666408 shutdown_connections 2024-04-25T00:03:25.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.760+0000 7ff22fe83640 1 -- 172.21.15.116:0/2889666408 wait complete. 2024-04-25T00:03:25.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.760+0000 7ff22fe83640 1 Processor -- start 2024-04-25T00:03:25.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.760+0000 7ff22fe83640 1 -- start start 2024-04-25T00:03:25.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.760+0000 7ff22fe83640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2140039c0 0x7ff2280c4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:25.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.760+0000 7ff22fe83640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff2280befc0 0x7ff2280bf420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:25.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.760+0000 7ff22fe83640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff2281f1340 0x7ff2281f17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:25.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.760+0000 7ff22fe83640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff22806c460 con 0x7ff2140039c0 2024-04-25T00:03:25.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.760+0000 7ff22fe83640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff2280be8d0 con 0x7ff2280befc0 2024-04-25T00:03:25.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.760+0000 7ff22fe83640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff22806c5e0 con 0x7ff2281f1340 2024-04-25T00:03:25.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.760+0000 7ff22dc1f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2140039c0 0x7ff2280c4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:25.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.760+0000 7ff22dc1f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2140039c0 0x7ff2280c4440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59674/0 (socket says 172.21.15.116:59674) 2024-04-25T00:03:25.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.760+0000 7ff22dc1f640 1 -- 172.21.15.116:0/447023984 learned_addr learned my addr 172.21.15.116:0/447023984 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:03:25.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.760+0000 7ff22d41e640 1 --2- 172.21.15.116:0/447023984 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff2280befc0 0x7ff2280bf420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:25.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.760+0000 7ff22e420640 1 --2- 172.21.15.116:0/447023984 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff2281f1340 0x7ff2281f17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:25.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.760+0000 7ff22dc1f640 1 -- 172.21.15.116:0/447023984 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff2281f1340 msgr2=0x7ff2281f17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:25.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.760+0000 7ff22dc1f640 1 --2- 172.21.15.116:0/447023984 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff2281f1340 0x7ff2281f17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:25.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.760+0000 7ff22dc1f640 1 -- 172.21.15.116:0/447023984 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff2280befc0 msgr2=0x7ff2280bf420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:25.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.760+0000 7ff22dc1f640 1 --2- 172.21.15.116:0/447023984 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff2280befc0 0x7ff2280bf420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:25.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.760+0000 7ff22dc1f640 1 -- 172.21.15.116:0/447023984 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2280be450 con 0x7ff2140039c0 2024-04-25T00:03:25.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.760+0000 7ff22e420640 1 --2- 172.21.15.116:0/447023984 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff2281f1340 0x7ff2281f17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:03:25.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.764+0000 7ff22dc1f640 1 --2- 172.21.15.116:0/447023984 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2140039c0 0x7ff2280c4440 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7ff22400c240 tx=0x7ff214005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:25.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.764+0000 7ff21e7fc640 1 -- 172.21.15.116:0/447023984 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff224005dc0 con 0x7ff2140039c0 2024-04-25T00:03:25.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.764+0000 7ff21e7fc640 1 -- 172.21.15.116:0/447023984 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2240047f0 con 0x7ff2140039c0 2024-04-25T00:03:25.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.764+0000 7ff21e7fc640 1 -- 172.21.15.116:0/447023984 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff224004be0 con 0x7ff2140039c0 2024-04-25T00:03:25.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.764+0000 7ff22fe83640 1 -- 172.21.15.116:0/447023984 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2280c4fb0 con 0x7ff2140039c0 2024-04-25T00:03:25.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.764+0000 7ff22fe83640 1 -- 172.21.15.116:0/447023984 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2280c2f60 con 0x7ff2140039c0 2024-04-25T00:03:25.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.764+0000 7ff22fe83640 1 -- 172.21.15.116:0/447023984 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff228068350 con 0x7ff2140039c0 2024-04-25T00:03:25.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.764+0000 7ff21e7fc640 1 -- 172.21.15.116:0/447023984 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff2240a5680 con 0x7ff2140039c0 2024-04-25T00:03:25.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.764+0000 7ff21e7fc640 1 --2- 172.21.15.116:0/447023984 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff20c0678c0 0x7ff20c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:25.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.764+0000 7ff21e7fc640 1 -- 172.21.15.116:0/447023984 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6752+0+0 (secure 0 0 0) 0x7ff22411f540 con 0x7ff2140039c0 2024-04-25T00:03:25.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.764+0000 7ff22d41e640 1 --2- 172.21.15.116:0/447023984 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff20c0678c0 0x7ff20c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:25.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.764+0000 7ff22d41e640 1 --2- 172.21.15.116:0/447023984 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff20c0678c0 0x7ff20c069d80 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ff228153ad0 tx=0x7ff218008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:25.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.768+0000 7ff21e7fc640 1 -- 172.21.15.116:0/447023984 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ff224016990 con 0x7ff2140039c0 2024-04-25T00:03:25.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.956+0000 7ff22fe83640 1 -- 172.21.15.116:0/447023984 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff2280cab20 con 0x7ff2140039c0 2024-04-25T00:03:25.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.956+0000 7ff21e7fc640 1 -- 172.21.15.116:0/447023984 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6498 (secure 0 0 0) 0x7ff2240efd00 con 0x7ff2140039c0 2024-04-25T00:03:25.960 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:03:25.960 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4735,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/2004876235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":2004876235},{"type":"v1","addr":"172.21.15.116:6835","nonce":2004876235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:01:08.748957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6837/1762356055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1762356055},{"type":"v1","addr":"172.21.15.174:6837","nonce":1762356055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270]},"id":1},{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:02:43.411054+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4242":{"gid":4242,"name":"c","rank":0,"incarnation":15,"state":"up:active","state_seq":29,"addr":"172.21.15.116:6834/1459177921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1459177921},{"type":"v1","addr":"172.21.15.116:6834","nonce":1459177921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4242,"qdb_cluster":[4242]},"id":2}]} 2024-04-25T00:03:25.961 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 19 2024-04-25T00:03:25.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.956+0000 7ff22fe83640 1 -- 172.21.15.116:0/447023984 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff20c0678c0 msgr2=0x7ff20c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:25.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.956+0000 7ff22fe83640 1 --2- 172.21.15.116:0/447023984 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff20c0678c0 0x7ff20c069d80 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7ff228153ad0 tx=0x7ff218008040 comp rx=0 tx=0).stop 2024-04-25T00:03:25.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.956+0000 7ff22fe83640 1 -- 172.21.15.116:0/447023984 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2140039c0 msgr2=0x7ff2280c4440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:25.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.956+0000 7ff22fe83640 1 --2- 172.21.15.116:0/447023984 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2140039c0 0x7ff2280c4440 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7ff22400c240 tx=0x7ff214005840 comp rx=0 tx=0).stop 2024-04-25T00:03:25.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.956+0000 7ff22fe83640 1 -- 172.21.15.116:0/447023984 shutdown_connections 2024-04-25T00:03:25.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.956+0000 7ff22fe83640 1 --2- 172.21.15.116:0/447023984 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff20c0678c0 0x7ff20c069d80 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:25.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.956+0000 7ff22fe83640 1 --2- 172.21.15.116:0/447023984 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff2281f1340 0x7ff2281f17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:25.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.956+0000 7ff22fe83640 1 --2- 172.21.15.116:0/447023984 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff2280befc0 0x7ff2280bf420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:25.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.956+0000 7ff22fe83640 1 --2- 172.21.15.116:0/447023984 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2140039c0 0x7ff2280c4440 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:25.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.960+0000 7ff22fe83640 1 -- 172.21.15.116:0/447023984 >> 172.21.15.116:0/447023984 conn(0x7ff2280acdd0 msgr2=0x7ff2280ac0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:03:25.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.960+0000 7ff22fe83640 1 -- 172.21.15.116:0/447023984 shutdown_connections 2024-04-25T00:03:25.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:25.960+0000 7ff22fe83640 1 -- 172.21.15.116:0/447023984 wait complete. 2024-04-25T00:03:25.978 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:03:26.775 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:03:26.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.900+0000 7f11de62c640 1 Processor -- start 2024-04-25T00:03:26.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.900+0000 7f11de62c640 1 -- start start 2024-04-25T00:03:26.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.900+0000 7f11de62c640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f11d80a94a0 0x7f11d81538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:26.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.900+0000 7f11de62c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11d80a99f0 0x7f11d8153e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:26.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.900+0000 7f11de62c640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11d80c3910 con 0x7f11d8150d20 2024-04-25T00:03:26.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.900+0000 7f11de62c640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f11d80c16e0 con 0x7f11d80a94a0 2024-04-25T00:03:26.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.900+0000 7f11de62c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f11d80c1400 con 0x7f11d80a99f0 2024-04-25T00:03:26.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.900+0000 7f11d7fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f11d8150d20 0x7f11d80c2a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:33124/0 (socket says 172.21.15.116:33124) 2024-04-25T00:03:26.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.900+0000 7f11d7fff640 1 -- 172.21.15.116:0/1968375147 learned_addr learned my addr 172.21.15.116:0/1968375147 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:03:26.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.900+0000 7f11dcbc9640 1 --2- 172.21.15.116:0/1968375147 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11d80a99f0 0x7f11d8153e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:26.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.900+0000 7f11cffff640 1 --2- 172.21.15.116:0/1968375147 >> v2:172.21.15.174:3300/0 conn(0x7f11d80a94a0 0x7f11d81538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:26.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.900+0000 7f11d77fe640 1 -- 172.21.15.116:0/1968375147 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2187300937 0 0) 0x7f11d80c3910 con 0x7f11d8150d20 2024-04-25T00:03:26.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.900+0000 7f11d77fe640 1 -- 172.21.15.116:0/1968375147 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f11c4003530 con 0x7f11d8150d20 2024-04-25T00:03:26.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.900+0000 7f11d77fe640 1 -- 172.21.15.116:0/1968375147 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3829816992 0 0) 0x7f11c4003530 con 0x7f11d8150d20 2024-04-25T00:03:26.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.900+0000 7f11d77fe640 1 -- 172.21.15.116:0/1968375147 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11d80a99f0 msgr2=0x7f11d8153e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:03:26.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.900+0000 7f11d77fe640 1 --2- 172.21.15.116:0/1968375147 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11d80a99f0 0x7f11d8153e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:26.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.900+0000 7f11d77fe640 1 -- 172.21.15.116:0/1968375147 >> v2:172.21.15.174:3300/0 conn(0x7f11d80a94a0 msgr2=0x7f11d81538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:03:26.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.900+0000 7f11d77fe640 1 --2- 172.21.15.116:0/1968375147 >> v2:172.21.15.174:3300/0 conn(0x7f11d80a94a0 0x7f11d81538e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:26.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.900+0000 7f11d77fe640 1 -- 172.21.15.116:0/1968375147 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11d806e660 con 0x7f11d8150d20 2024-04-25T00:03:26.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.900+0000 7f11d77fe640 1 -- 172.21.15.116:0/1968375147 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f11c8001680 con 0x7f11d8150d20 2024-04-25T00:03:26.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.900+0000 7f11d77fe640 1 -- 172.21.15.116:0/1968375147 >> v1:172.21.15.116:6789/0 conn(0x7f11d8150d20 legacy=0x7f11d80c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:26.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.900+0000 7f11d77fe640 1 --2- 172.21.15.116:0/1968375147 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11d80a99f0 0x7f11c40039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:26.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.900+0000 7f11d77fe640 1 --2- 172.21.15.116:0/1968375147 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11c4003f40 0x7f11c4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:26.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.900+0000 7f11d77fe640 1 --2- 172.21.15.116:0/1968375147 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11c4004d50 0x7f11c40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:26.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.900+0000 7f11d77fe640 1 -- 172.21.15.116:0/1968375147 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f11d80c16e0 con 0x7f11c4003f40 2024-04-25T00:03:26.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.900+0000 7f11d77fe640 1 -- 172.21.15.116:0/1968375147 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f11d80c1400 con 0x7f11c4004d50 2024-04-25T00:03:26.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.900+0000 7f11d77fe640 1 -- 172.21.15.116:0/1968375147 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f11c400b9d0 con 0x7f11d80a99f0 2024-04-25T00:03:26.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.900+0000 7f11cffff640 1 --2- 172.21.15.116:0/1968375147 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11c4003f40 0x7f11c4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:26.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.900+0000 7f11d7fff640 1 --2- 172.21.15.116:0/1968375147 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11d80a99f0 0x7f11c40039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:26.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.900+0000 7f11dcbc9640 1 --2- 172.21.15.116:0/1968375147 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11c4004d50 0x7f11c40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:26.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.900+0000 7f11cffff640 1 -- 172.21.15.116:0/1968375147 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11d80a99f0 msgr2=0x7f11c40039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:03:26.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.900+0000 7f11cffff640 1 --2- 172.21.15.116:0/1968375147 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11d80a99f0 0x7f11c40039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:26.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.900+0000 7f11cffff640 1 -- 172.21.15.116:0/1968375147 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11c4004d50 msgr2=0x7f11c40051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:03:26.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.900+0000 7f11cffff640 1 --2- 172.21.15.116:0/1968375147 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11c4004d50 0x7f11c40051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:26.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.900+0000 7f11cffff640 1 -- 172.21.15.116:0/1968375147 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11c4003530 con 0x7f11c4003f40 2024-04-25T00:03:26.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.900+0000 7f11dcbc9640 1 --2- 172.21.15.116:0/1968375147 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11c4004d50 0x7f11c40051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:03:26.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.904+0000 7f11cffff640 1 --2- 172.21.15.116:0/1968375147 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11c4003f40 0x7f11c4004810 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f11c0009ae0 tx=0x7f11c001ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f3fffdc286ee17b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:26.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.904+0000 7f11d77fe640 1 -- 172.21.15.116:0/1968375147 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c00b2070 con 0x7f11c4003f40 2024-04-25T00:03:26.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.904+0000 7f11d77fe640 1 -- 172.21.15.116:0/1968375147 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11c00a56a0 con 0x7f11c4003f40 2024-04-25T00:03:26.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.904+0000 7f11d77fe640 1 -- 172.21.15.116:0/1968375147 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11c00096b0 con 0x7f11c4003f40 2024-04-25T00:03:26.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.904+0000 7f11de62c640 1 -- 172.21.15.116:0/1968375147 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11c4003f40 msgr2=0x7f11c4004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:26.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.904+0000 7f11de62c640 1 --2- 172.21.15.116:0/1968375147 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11c4003f40 0x7f11c4004810 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f11c0009ae0 tx=0x7f11c001ca50 comp rx=0 tx=0).stop 2024-04-25T00:03:26.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.904+0000 7f11dcbc9640 1 -- 172.21.15.116:0/1968375147 reap_dead start 2024-04-25T00:03:26.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.904+0000 7f11de62c640 1 -- 172.21.15.116:0/1968375147 shutdown_connections 2024-04-25T00:03:26.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.904+0000 7f11de62c640 1 -- 172.21.15.116:0/1968375147 >> 172.21.15.116:0/1968375147 conn(0x7f11d80c3b00 msgr2=0x7f11d80c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:03:26.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.904+0000 7f11de62c640 1 -- 172.21.15.116:0/1968375147 shutdown_connections 2024-04-25T00:03:26.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.904+0000 7f11de62c640 1 -- 172.21.15.116:0/1968375147 wait complete. 2024-04-25T00:03:26.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.904+0000 7f11de62c640 1 Processor -- start 2024-04-25T00:03:26.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.904+0000 7f11de62c640 1 -- start start 2024-04-25T00:03:26.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.904+0000 7f11de62c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11c4003f40 0x7f11d814b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:26.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.904+0000 7f11de62c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11d813baa0 0x7f11d80c5b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:26.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.904+0000 7f11de62c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11d81e9940 0x7f11d80c6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:26.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.904+0000 7f11de62c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f11d80c3120 con 0x7f11d81e9940 2024-04-25T00:03:26.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.904+0000 7f11de62c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f11d806ec20 con 0x7f11c4003f40 2024-04-25T00:03:26.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.904+0000 7f11de62c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f11d80c32a0 con 0x7f11d813baa0 2024-04-25T00:03:26.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.904+0000 7f11dcbc9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11d81e9940 0x7f11d80c6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:26.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.904+0000 7f11dcbc9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11d81e9940 0x7f11d80c6050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59680/0 (socket says 172.21.15.116:59680) 2024-04-25T00:03:26.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.904+0000 7f11dcbc9640 1 -- 172.21.15.116:0/3470331339 learned_addr learned my addr 172.21.15.116:0/3470331339 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:03:26.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.908+0000 7f11d7fff640 1 --2- 172.21.15.116:0/3470331339 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11c4003f40 0x7f11d814b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:26.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.908+0000 7f11cffff640 1 --2- 172.21.15.116:0/3470331339 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11d813baa0 0x7f11d80c5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:26.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.908+0000 7f11dcbc9640 1 -- 172.21.15.116:0/3470331339 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11d813baa0 msgr2=0x7f11d80c5b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:26.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.908+0000 7f11dcbc9640 1 --2- 172.21.15.116:0/3470331339 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11d813baa0 0x7f11d80c5b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:26.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.908+0000 7f11dcbc9640 1 -- 172.21.15.116:0/3470331339 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11c4003f40 msgr2=0x7f11d814b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:26.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.908+0000 7f11dcbc9640 1 --2- 172.21.15.116:0/3470331339 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11c4003f40 0x7f11d814b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:26.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.908+0000 7f11dcbc9640 1 -- 172.21.15.116:0/3470331339 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11d81414c0 con 0x7f11d81e9940 2024-04-25T00:03:26.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.908+0000 7f11dcbc9640 1 --2- 172.21.15.116:0/3470331339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11d81e9940 0x7f11d80c6050 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f11d8070890 tx=0x7f11d000f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:26.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.908+0000 7f11d57fa640 1 -- 172.21.15.116:0/3470331339 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d001c070 con 0x7f11d81e9940 2024-04-25T00:03:26.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.908+0000 7f11d57fa640 1 -- 172.21.15.116:0/3470331339 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11d0017400 con 0x7f11d81e9940 2024-04-25T00:03:26.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.908+0000 7f11d57fa640 1 -- 172.21.15.116:0/3470331339 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d00177f0 con 0x7f11d81e9940 2024-04-25T00:03:26.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.908+0000 7f11de62c640 1 -- 172.21.15.116:0/3470331339 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11d80c4d70 con 0x7f11d81e9940 2024-04-25T00:03:26.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.908+0000 7f11de62c640 1 -- 172.21.15.116:0/3470331339 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11d80c5310 con 0x7f11d81e9940 2024-04-25T00:03:26.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.908+0000 7f11de62c640 1 -- 172.21.15.116:0/3470331339 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11d80c4af0 con 0x7f11d81e9940 2024-04-25T00:03:26.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.908+0000 7f11d57fa640 1 -- 172.21.15.116:0/3470331339 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f11d001f070 con 0x7f11d81e9940 2024-04-25T00:03:26.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.908+0000 7f11d57fa640 1 --2- 172.21.15.116:0/3470331339 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f11b80678c0 0x7f11b8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:26.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.908+0000 7f11d57fa640 1 -- 172.21.15.116:0/3470331339 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6752+0+0 (secure 0 0 0) 0x7f11d0085fd0 con 0x7f11d81e9940 2024-04-25T00:03:26.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.908+0000 7f11d7fff640 1 --2- 172.21.15.116:0/3470331339 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f11b80678c0 0x7f11b8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:26.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.912+0000 7f11d7fff640 1 --2- 172.21.15.116:0/3470331339 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f11b80678c0 0x7f11b8069d80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f11c8001060 tx=0x7f11c80013c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:26.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:26.912+0000 7f11d57fa640 1 -- 172.21.15.116:0/3470331339 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f11d80c4af0 con 0x7f11d81e9940 2024-04-25T00:03:27.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:27.100+0000 7f11de62c640 1 -- 172.21.15.116:0/3470331339 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f11d8068350 con 0x7f11d81e9940 2024-04-25T00:03:27.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:27.104+0000 7f11d57fa640 1 -- 172.21.15.116:0/3470331339 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6498 (secure 0 0 0) 0x7f11d0012070 con 0x7f11d81e9940 2024-04-25T00:03:27.108 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:03:27.109 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4735,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/2004876235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":2004876235},{"type":"v1","addr":"172.21.15.116:6835","nonce":2004876235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:01:08.748957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6837/1762356055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1762356055},{"type":"v1","addr":"172.21.15.174:6837","nonce":1762356055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270]},"id":1},{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:02:43.411054+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4242":{"gid":4242,"name":"c","rank":0,"incarnation":15,"state":"up:active","state_seq":29,"addr":"172.21.15.116:6834/1459177921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1459177921},{"type":"v1","addr":"172.21.15.116:6834","nonce":1459177921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4242,"qdb_cluster":[4242]},"id":2}]} 2024-04-25T00:03:27.110 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 19 2024-04-25T00:03:27.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:27.104+0000 7f11de62c640 1 -- 172.21.15.116:0/3470331339 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f11b80678c0 msgr2=0x7f11b8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:27.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:27.104+0000 7f11de62c640 1 --2- 172.21.15.116:0/3470331339 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f11b80678c0 0x7f11b8069d80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f11c8001060 tx=0x7f11c80013c0 comp rx=0 tx=0).stop 2024-04-25T00:03:27.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:27.104+0000 7f11de62c640 1 -- 172.21.15.116:0/3470331339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11d81e9940 msgr2=0x7f11d80c6050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:27.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:27.104+0000 7f11de62c640 1 --2- 172.21.15.116:0/3470331339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11d81e9940 0x7f11d80c6050 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f11d8070890 tx=0x7f11d000f4a0 comp rx=0 tx=0).stop 2024-04-25T00:03:27.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:27.104+0000 7f11de62c640 1 -- 172.21.15.116:0/3470331339 shutdown_connections 2024-04-25T00:03:27.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:27.104+0000 7f11de62c640 1 --2- 172.21.15.116:0/3470331339 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f11b80678c0 0x7f11b8069d80 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:27.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:27.104+0000 7f11de62c640 1 --2- 172.21.15.116:0/3470331339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11d81e9940 0x7f11d80c6050 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:27.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:27.104+0000 7f11de62c640 1 --2- 172.21.15.116:0/3470331339 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11d813baa0 0x7f11d80c5b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:27.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:27.104+0000 7f11de62c640 1 --2- 172.21.15.116:0/3470331339 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11c4003f40 0x7f11d814b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:27.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:27.104+0000 7f11de62c640 1 -- 172.21.15.116:0/3470331339 >> 172.21.15.116:0/3470331339 conn(0x7f11d80c3b00 msgr2=0x7f11d806cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:03:27.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:27.104+0000 7f11de62c640 1 -- 172.21.15.116:0/3470331339 shutdown_connections 2024-04-25T00:03:27.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:27.104+0000 7f11de62c640 1 -- 172.21.15.116:0/3470331339 wait complete. 2024-04-25T00:03:27.125 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T00:03:27.979 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:03:28.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.108+0000 7f9c85606640 1 Processor -- start 2024-04-25T00:03:28.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.108+0000 7f9c85606640 1 -- start start 2024-04-25T00:03:28.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.108+0000 7f9c85606640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f9c80150d20 0x7f9c800c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:28.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.108+0000 7f9c85606640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9c800a94a0 0x7f9c80153ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:28.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.108+0000 7f9c85606640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c800c3910 con 0x7f9c800c2f50 2024-04-25T00:03:28.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.108+0000 7f9c85606640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f9c80154530 con 0x7f9c800a94a0 2024-04-25T00:03:28.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.108+0000 7f9c85606640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f9c800c16e0 con 0x7f9c80150d20 2024-04-25T00:03:28.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.108+0000 7f9c7f7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9c800a94a0 0x7f9c80153ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:28.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.108+0000 7f9c7f7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9c800a94a0 0x7f9c80153ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59696/0 (socket says 172.21.15.116:59696) 2024-04-25T00:03:28.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.108+0000 7f9c7f7fe640 1 -- 172.21.15.116:0/326867287 learned_addr learned my addr 172.21.15.116:0/326867287 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:03:28.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.108+0000 7f9c7effd640 1 --2- 172.21.15.116:0/326867287 >> v2:172.21.15.174:3300/0 conn(0x7f9c80150d20 0x7f9c800c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:28.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.108+0000 7f9c7f7fe640 1 -- 172.21.15.116:0/326867287 >> v2:172.21.15.174:3300/0 conn(0x7f9c80150d20 msgr2=0x7f9c800c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:03:28.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.108+0000 7f9c7f7fe640 1 --2- 172.21.15.116:0/326867287 >> v2:172.21.15.174:3300/0 conn(0x7f9c80150d20 0x7f9c800c2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:28.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.108+0000 7f9c7f7fe640 1 -- 172.21.15.116:0/326867287 >> v1:172.21.15.174:6789/0 conn(0x7f9c800c2f50 legacy=0x7f9c801538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:03:28.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.108+0000 7f9c7f7fe640 1 -- 172.21.15.116:0/326867287 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c800c1400 con 0x7f9c800a94a0 2024-04-25T00:03:28.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.112+0000 7f9c7f7fe640 1 --2- 172.21.15.116:0/326867287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c800a94a0 0x7f9c80153ff0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f9c80073550 tx=0x7f9c7801c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=265c1ef5ba793e1c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:28.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.112+0000 7f9c7e7fc640 1 -- 172.21.15.116:0/326867287 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c780a5780 con 0x7f9c800a94a0 2024-04-25T00:03:28.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.112+0000 7f9c7e7fc640 1 -- 172.21.15.116:0/326867287 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c780a5920 con 0x7f9c800a94a0 2024-04-25T00:03:28.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.112+0000 7f9c7e7fc640 1 -- 172.21.15.116:0/326867287 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c780a5cf0 con 0x7f9c800a94a0 2024-04-25T00:03:28.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.112+0000 7f9c85606640 1 -- 172.21.15.116:0/326867287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c800a94a0 msgr2=0x7f9c80153ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:28.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.112+0000 7f9c85606640 1 --2- 172.21.15.116:0/326867287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c800a94a0 0x7f9c80153ff0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f9c80073550 tx=0x7f9c7801c9b0 comp rx=0 tx=0).stop 2024-04-25T00:03:28.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.112+0000 7f9c85606640 1 -- 172.21.15.116:0/326867287 shutdown_connections 2024-04-25T00:03:28.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.112+0000 7f9c85606640 1 --2- 172.21.15.116:0/326867287 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c800a94a0 0x7f9c80153ff0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:28.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.112+0000 7f9c85606640 1 --2- 172.21.15.116:0/326867287 >> v2:172.21.15.174:3300/0 conn(0x7f9c80150d20 0x7f9c800c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:28.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.112+0000 7f9c85606640 1 -- 172.21.15.116:0/326867287 >> 172.21.15.116:0/326867287 conn(0x7f9c800c3b00 msgr2=0x7f9c800c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:03:28.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.112+0000 7f9c85606640 1 -- 172.21.15.116:0/326867287 shutdown_connections 2024-04-25T00:03:28.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.112+0000 7f9c85606640 1 -- 172.21.15.116:0/326867287 wait complete. 2024-04-25T00:03:28.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.112+0000 7f9c85606640 1 Processor -- start 2024-04-25T00:03:28.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.112+0000 7f9c85606640 1 -- start start 2024-04-25T00:03:28.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.112+0000 7f9c85606640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9c800a94a0 0x7f9c801f24e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:28.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.112+0000 7f9c85606640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9c800c2f50 0x7f9c801466c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:28.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.112+0000 7f9c85606640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c80150d20 0x7f9c8013f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:28.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.112+0000 7f9c85606640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9c801608b0 con 0x7f9c80150d20 2024-04-25T00:03:28.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.112+0000 7f9c85606640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9c80160730 con 0x7f9c800c2f50 2024-04-25T00:03:28.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.112+0000 7f9c85606640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f9c80160a30 con 0x7f9c800a94a0 2024-04-25T00:03:28.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.112+0000 7f9c7f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c80150d20 0x7f9c8013f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:28.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.112+0000 7f9c7f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c80150d20 0x7f9c8013f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59712/0 (socket says 172.21.15.116:59712) 2024-04-25T00:03:28.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.112+0000 7f9c7f7fe640 1 -- 172.21.15.116:0/281313351 learned_addr learned my addr 172.21.15.116:0/281313351 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:03:28.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.112+0000 7f9c767fc640 1 --2- 172.21.15.116:0/281313351 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9c800c2f50 0x7f9c801466c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:28.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.112+0000 7f9c7effd640 1 --2- 172.21.15.116:0/281313351 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9c800a94a0 0x7f9c801f24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:28.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.112+0000 7f9c7f7fe640 1 -- 172.21.15.116:0/281313351 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9c800a94a0 msgr2=0x7f9c801f24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:28.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.112+0000 7f9c7f7fe640 1 --2- 172.21.15.116:0/281313351 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9c800a94a0 0x7f9c801f24e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:28.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.112+0000 7f9c7f7fe640 1 -- 172.21.15.116:0/281313351 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9c800c2f50 msgr2=0x7f9c801466c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:28.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.112+0000 7f9c7f7fe640 1 --2- 172.21.15.116:0/281313351 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9c800c2f50 0x7f9c801466c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:28.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.112+0000 7f9c7f7fe640 1 -- 172.21.15.116:0/281313351 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c8015cb90 con 0x7f9c80150d20 2024-04-25T00:03:28.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.112+0000 7f9c7effd640 1 --2- 172.21.15.116:0/281313351 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9c800a94a0 0x7f9c801f24e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:03:28.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.112+0000 7f9c7f7fe640 1 --2- 172.21.15.116:0/281313351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c80150d20 0x7f9c8013f730 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f9c78009e20 tx=0x7f9c78019750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:28.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.112+0000 7f9c77fff640 1 -- 172.21.15.116:0/281313351 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c780bc020 con 0x7f9c80150d20 2024-04-25T00:03:28.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.112+0000 7f9c77fff640 1 -- 172.21.15.116:0/281313351 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c780b8690 con 0x7f9c80150d20 2024-04-25T00:03:28.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.112+0000 7f9c85606640 1 -- 172.21.15.116:0/281313351 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c801ee8b0 con 0x7f9c80150d20 2024-04-25T00:03:28.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.112+0000 7f9c77fff640 1 -- 172.21.15.116:0/281313351 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c780b8a80 con 0x7f9c80150d20 2024-04-25T00:03:28.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.112+0000 7f9c85606640 1 -- 172.21.15.116:0/281313351 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c801eee70 con 0x7f9c80150d20 2024-04-25T00:03:28.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.116+0000 7f9c85606640 1 -- 172.21.15.116:0/281313351 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c80068350 con 0x7f9c80150d20 2024-04-25T00:03:28.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.116+0000 7f9c77fff640 1 -- 172.21.15.116:0/281313351 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f9c780bf020 con 0x7f9c80150d20 2024-04-25T00:03:28.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.116+0000 7f9c77fff640 1 --2- 172.21.15.116:0/281313351 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9c600678c0 0x7f9c60069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:28.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.116+0000 7f9c77fff640 1 -- 172.21.15.116:0/281313351 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6752+0+0 (secure 0 0 0) 0x7f9c78121050 con 0x7f9c80150d20 2024-04-25T00:03:28.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.116+0000 7f9c7effd640 1 --2- 172.21.15.116:0/281313351 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9c600678c0 0x7f9c60069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:28.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.116+0000 7f9c7effd640 1 --2- 172.21.15.116:0/281313351 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9c600678c0 0x7f9c60069d80 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f9c700097a0 tx=0x7f9c70008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:28.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.120+0000 7f9c77fff640 1 -- 172.21.15.116:0/281313351 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f9c780c6030 con 0x7f9c80150d20 2024-04-25T00:03:28.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.300+0000 7f9c85606640 1 -- 172.21.15.116:0/281313351 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9c800ac010 con 0x7f9c80150d20 2024-04-25T00:03:28.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.300+0000 7f9c77fff640 1 -- 172.21.15.116:0/281313351 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6498 (secure 0 0 0) 0x7f9c780165a0 con 0x7f9c80150d20 2024-04-25T00:03:28.306 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:03:28.307 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4735,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/2004876235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":2004876235},{"type":"v1","addr":"172.21.15.116:6835","nonce":2004876235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:01:08.748957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6837/1762356055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1762356055},{"type":"v1","addr":"172.21.15.174:6837","nonce":1762356055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270]},"id":1},{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:02:43.411054+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4242":{"gid":4242,"name":"c","rank":0,"incarnation":15,"state":"up:active","state_seq":29,"addr":"172.21.15.116:6834/1459177921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1459177921},{"type":"v1","addr":"172.21.15.116:6834","nonce":1459177921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4242,"qdb_cluster":[4242]},"id":2}]} 2024-04-25T00:03:28.307 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 19 2024-04-25T00:03:28.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.304+0000 7f9c85606640 1 -- 172.21.15.116:0/281313351 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9c600678c0 msgr2=0x7f9c60069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:28.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.304+0000 7f9c85606640 1 --2- 172.21.15.116:0/281313351 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9c600678c0 0x7f9c60069d80 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f9c700097a0 tx=0x7f9c70008040 comp rx=0 tx=0).stop 2024-04-25T00:03:28.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.304+0000 7f9c85606640 1 -- 172.21.15.116:0/281313351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c80150d20 msgr2=0x7f9c8013f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:28.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.304+0000 7f9c85606640 1 --2- 172.21.15.116:0/281313351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c80150d20 0x7f9c8013f730 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f9c78009e20 tx=0x7f9c78019750 comp rx=0 tx=0).stop 2024-04-25T00:03:28.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.304+0000 7f9c85606640 1 -- 172.21.15.116:0/281313351 shutdown_connections 2024-04-25T00:03:28.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.304+0000 7f9c85606640 1 --2- 172.21.15.116:0/281313351 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9c600678c0 0x7f9c60069d80 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:28.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.304+0000 7f9c85606640 1 --2- 172.21.15.116:0/281313351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9c80150d20 0x7f9c8013f730 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:28.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.304+0000 7f9c85606640 1 --2- 172.21.15.116:0/281313351 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9c800c2f50 0x7f9c801466c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:28.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.304+0000 7f9c85606640 1 --2- 172.21.15.116:0/281313351 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9c800a94a0 0x7f9c801f24e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:28.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.304+0000 7f9c85606640 1 -- 172.21.15.116:0/281313351 >> 172.21.15.116:0/281313351 conn(0x7f9c800c3b00 msgr2=0x7f9c80140480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:03:28.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.304+0000 7f9c85606640 1 -- 172.21.15.116:0/281313351 shutdown_connections 2024-04-25T00:03:28.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:28.304+0000 7f9c85606640 1 -- 172.21.15.116:0/281313351 wait complete. 2024-04-25T00:03:28.323 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:03:29.127 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:03:29.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.252+0000 7fe48d8c7640 1 Processor -- start 2024-04-25T00:03:29.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.252+0000 7fe48d8c7640 1 -- start start 2024-04-25T00:03:29.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.252+0000 7fe48d8c7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe488150d20 0x7fe4880c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:29.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.252+0000 7fe48d8c7640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fe4880c2f50 0x7fe4881538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:29.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.252+0000 7fe48d8c7640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe4880a94a0 0x7fe488153e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:29.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.252+0000 7fe48d8c7640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fe4880c3910 con 0x7fe488150d20 2024-04-25T00:03:29.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.252+0000 7fe48d8c7640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fe4880c16e0 con 0x7fe4880c2f50 2024-04-25T00:03:29.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.252+0000 7fe48d8c7640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe4880c1400 con 0x7fe4880a94a0 2024-04-25T00:03:29.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.252+0000 7fe486ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe488150d20 0x7fe4880c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:29.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.252+0000 7fe4877fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe4880a94a0 0x7fe488153e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:29.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.252+0000 7fe4867fc640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fe4880c2f50 0x7fe4881538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:29.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.252+0000 7fe486ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe488150d20 0x7fe4880c2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59728/0 (socket says 172.21.15.116:59728) 2024-04-25T00:03:29.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.252+0000 7fe486ffd640 1 -- 172.21.15.116:0/192087864 learned_addr learned my addr 172.21.15.116:0/192087864 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:03:29.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.252+0000 7fe486ffd640 1 -- 172.21.15.116:0/192087864 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe4880a94a0 msgr2=0x7fe488153e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:03:29.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.252+0000 7fe486ffd640 1 --2- 172.21.15.116:0/192087864 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe4880a94a0 0x7fe488153e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:29.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.252+0000 7fe486ffd640 1 -- 172.21.15.116:0/192087864 >> v2:172.21.15.174:3300/0 conn(0x7fe4880c2f50 msgr2=0x7fe4881538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:03:29.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.252+0000 7fe486ffd640 1 --2- 172.21.15.116:0/192087864 >> v2:172.21.15.174:3300/0 conn(0x7fe4880c2f50 0x7fe4881538e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:29.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.252+0000 7fe486ffd640 1 -- 172.21.15.116:0/192087864 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe48806e660 con 0x7fe488150d20 2024-04-25T00:03:29.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.256+0000 7fe486ffd640 1 --2- 172.21.15.116:0/192087864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe488150d20 0x7fe4880c2a10 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fe488070710 tx=0x7fe47c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76c92a8ac6ed2f45 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:29.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.256+0000 7fe485ffb640 1 -- 172.21.15.116:0/192087864 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe47c0b1070 con 0x7fe488150d20 2024-04-25T00:03:29.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.256+0000 7fe485ffb640 1 -- 172.21.15.116:0/192087864 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe47c0ac410 con 0x7fe488150d20 2024-04-25T00:03:29.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.256+0000 7fe485ffb640 1 -- 172.21.15.116:0/192087864 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe47c0ac7e0 con 0x7fe488150d20 2024-04-25T00:03:29.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.256+0000 7fe48d8c7640 1 -- 172.21.15.116:0/192087864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe488150d20 msgr2=0x7fe4880c2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:29.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.256+0000 7fe48d8c7640 1 --2- 172.21.15.116:0/192087864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe488150d20 0x7fe4880c2a10 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fe488070710 tx=0x7fe47c01c520 comp rx=0 tx=0).stop 2024-04-25T00:03:29.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.256+0000 7fe48d8c7640 1 -- 172.21.15.116:0/192087864 shutdown_connections 2024-04-25T00:03:29.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.256+0000 7fe48d8c7640 1 --2- 172.21.15.116:0/192087864 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe4880a94a0 0x7fe488153e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:29.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.256+0000 7fe48d8c7640 1 --2- 172.21.15.116:0/192087864 >> v2:172.21.15.174:3300/0 conn(0x7fe4880c2f50 0x7fe4881538e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:29.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.256+0000 7fe48d8c7640 1 --2- 172.21.15.116:0/192087864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe488150d20 0x7fe4880c2a10 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:29.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.256+0000 7fe48d8c7640 1 -- 172.21.15.116:0/192087864 >> 172.21.15.116:0/192087864 conn(0x7fe4880c3b00 msgr2=0x7fe4880c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:03:29.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.256+0000 7fe48d8c7640 1 -- 172.21.15.116:0/192087864 shutdown_connections 2024-04-25T00:03:29.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.256+0000 7fe48d8c7640 1 -- 172.21.15.116:0/192087864 wait complete. 2024-04-25T00:03:29.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.256+0000 7fe48d8c7640 1 Processor -- start 2024-04-25T00:03:29.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.256+0000 7fe48d8c7640 1 -- start start 2024-04-25T00:03:29.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.256+0000 7fe48d8c7640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe4880a94a0 0x7fe4880ac0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:29.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.256+0000 7fe48d8c7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe4880c2f50 0x7fe488152040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:29.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.256+0000 7fe48d8c7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe488150d20 0x7fe488152580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:29.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.260+0000 7fe48d8c7640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe488157340 con 0x7fe488150d20 2024-04-25T00:03:29.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.260+0000 7fe48d8c7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe488154570 con 0x7fe4880c2f50 2024-04-25T00:03:29.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.260+0000 7fe48d8c7640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe4881574c0 con 0x7fe4880a94a0 2024-04-25T00:03:29.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.260+0000 7fe4877fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe488150d20 0x7fe488152580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:29.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.260+0000 7fe4877fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe488150d20 0x7fe488152580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59738/0 (socket says 172.21.15.116:59738) 2024-04-25T00:03:29.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.260+0000 7fe4877fe640 1 -- 172.21.15.116:0/3317017291 learned_addr learned my addr 172.21.15.116:0/3317017291 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:03:29.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.260+0000 7fe4867fc640 1 --2- 172.21.15.116:0/3317017291 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe4880c2f50 0x7fe488152040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:29.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.260+0000 7fe486ffd640 1 --2- 172.21.15.116:0/3317017291 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe4880a94a0 0x7fe4880ac0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:29.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.260+0000 7fe4877fe640 1 -- 172.21.15.116:0/3317017291 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe4880a94a0 msgr2=0x7fe4880ac0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:29.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.260+0000 7fe4877fe640 1 --2- 172.21.15.116:0/3317017291 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe4880a94a0 0x7fe4880ac0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:29.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.260+0000 7fe4877fe640 1 -- 172.21.15.116:0/3317017291 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe4880c2f50 msgr2=0x7fe488152040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:29.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.260+0000 7fe4877fe640 1 --2- 172.21.15.116:0/3317017291 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe4880c2f50 0x7fe488152040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:29.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.260+0000 7fe4877fe640 1 -- 172.21.15.116:0/3317017291 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4881ec090 con 0x7fe488150d20 2024-04-25T00:03:29.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.260+0000 7fe4867fc640 1 --2- 172.21.15.116:0/3317017291 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe4880c2f50 0x7fe488152040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:03:29.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.260+0000 7fe4877fe640 1 --2- 172.21.15.116:0/3317017291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe488150d20 0x7fe488152580 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fe48000eed0 tx=0x7fe48000b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:29.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.260+0000 7fe46ffff640 1 -- 172.21.15.116:0/3317017291 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe480018020 con 0x7fe488150d20 2024-04-25T00:03:29.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.260+0000 7fe46ffff640 1 -- 172.21.15.116:0/3317017291 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe480014400 con 0x7fe488150d20 2024-04-25T00:03:29.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.260+0000 7fe48d8c7640 1 -- 172.21.15.116:0/3317017291 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4881f0340 con 0x7fe488150d20 2024-04-25T00:03:29.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.260+0000 7fe46ffff640 1 -- 172.21.15.116:0/3317017291 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4800147f0 con 0x7fe488150d20 2024-04-25T00:03:29.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.260+0000 7fe48d8c7640 1 -- 172.21.15.116:0/3317017291 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4881ec5f0 con 0x7fe488150d20 2024-04-25T00:03:29.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.260+0000 7fe48d8c7640 1 -- 172.21.15.116:0/3317017291 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe488068350 con 0x7fe488150d20 2024-04-25T00:03:29.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.260+0000 7fe46ffff640 1 -- 172.21.15.116:0/3317017291 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe48001b020 con 0x7fe488150d20 2024-04-25T00:03:29.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.260+0000 7fe46ffff640 1 --2- 172.21.15.116:0/3317017291 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe4740678c0 0x7fe474069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:29.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.260+0000 7fe46ffff640 1 -- 172.21.15.116:0/3317017291 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 6752+0+0 (secure 0 0 0) 0x7fe480086650 con 0x7fe488150d20 2024-04-25T00:03:29.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.260+0000 7fe486ffd640 1 --2- 172.21.15.116:0/3317017291 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe4740678c0 0x7fe474069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:29.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.260+0000 7fe486ffd640 1 --2- 172.21.15.116:0/3317017291 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe4740678c0 0x7fe474069d80 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe47c0ae000 tx=0x7fe47c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:29.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.264+0000 7fe46ffff640 1 -- 172.21.15.116:0/3317017291 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fe480022070 con 0x7fe488150d20 2024-04-25T00:03:29.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.452+0000 7fe48d8c7640 1 -- 172.21.15.116:0/3317017291 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe4880c34f0 con 0x7fe488150d20 2024-04-25T00:03:29.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.452+0000 7fe46ffff640 1 -- 172.21.15.116:0/3317017291 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+6498 (secure 0 0 0) 0x7fe4881ec090 con 0x7fe488150d20 2024-04-25T00:03:29.456 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:03:29.456 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4735,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/2004876235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":2004876235},{"type":"v1","addr":"172.21.15.116:6835","nonce":2004876235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:01:08.748957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6837/1762356055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1762356055},{"type":"v1","addr":"172.21.15.174:6837","nonce":1762356055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270]},"id":1},{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:02:43.411054+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4242},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4242":{"gid":4242,"name":"c","rank":0,"incarnation":15,"state":"up:active","state_seq":29,"addr":"172.21.15.116:6834/1459177921","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1459177921},{"type":"v1","addr":"172.21.15.116:6834","nonce":1459177921}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4242,"qdb_cluster":[4242]},"id":2}]} 2024-04-25T00:03:29.456 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 19 2024-04-25T00:03:29.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.452+0000 7fe48d8c7640 1 -- 172.21.15.116:0/3317017291 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe4740678c0 msgr2=0x7fe474069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:29.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.452+0000 7fe48d8c7640 1 --2- 172.21.15.116:0/3317017291 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe4740678c0 0x7fe474069d80 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe47c0ae000 tx=0x7fe47c008040 comp rx=0 tx=0).stop 2024-04-25T00:03:29.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.452+0000 7fe48d8c7640 1 -- 172.21.15.116:0/3317017291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe488150d20 msgr2=0x7fe488152580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:29.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.452+0000 7fe48d8c7640 1 --2- 172.21.15.116:0/3317017291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe488150d20 0x7fe488152580 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fe48000eed0 tx=0x7fe48000b480 comp rx=0 tx=0).stop 2024-04-25T00:03:29.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.452+0000 7fe48d8c7640 1 -- 172.21.15.116:0/3317017291 shutdown_connections 2024-04-25T00:03:29.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.452+0000 7fe48d8c7640 1 --2- 172.21.15.116:0/3317017291 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe4740678c0 0x7fe474069d80 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:29.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.452+0000 7fe48d8c7640 1 --2- 172.21.15.116:0/3317017291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe488150d20 0x7fe488152580 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:29.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.452+0000 7fe48d8c7640 1 --2- 172.21.15.116:0/3317017291 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe4880c2f50 0x7fe488152040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:29.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.452+0000 7fe48d8c7640 1 --2- 172.21.15.116:0/3317017291 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe4880a94a0 0x7fe4880ac0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:29.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.452+0000 7fe48d8c7640 1 -- 172.21.15.116:0/3317017291 >> 172.21.15.116:0/3317017291 conn(0x7fe4880c3b00 msgr2=0x7fe4880a9f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:03:29.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.452+0000 7fe48d8c7640 1 -- 172.21.15.116:0/3317017291 shutdown_connections 2024-04-25T00:03:29.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:29.452+0000 7fe48d8c7640 1 -- 172.21.15.116:0/3317017291 wait complete. 2024-04-25T00:03:29.473 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T00:03:30.324 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:03:30.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.451+0000 7f21796c3640 1 Processor -- start 2024-04-25T00:03:30.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.451+0000 7f21796c3640 1 -- start start 2024-04-25T00:03:30.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.451+0000 7f21796c3640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f2174152d60 0x7f21740ab250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:30.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.451+0000 7f21796c3640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f21740ab790 0x7f21740c8450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:30.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.451+0000 7f21796c3640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2174156760 con 0x7f2174153140 2024-04-25T00:03:30.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.451+0000 7f21796c3640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f2174155be0 con 0x7f21740ab790 2024-04-25T00:03:30.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.451+0000 7f21796c3640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f2174155ec0 con 0x7f2174152d60 2024-04-25T00:03:30.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.451+0000 7f21727fc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f21740ab790 0x7f21740c8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:30.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.451+0000 7f21727fc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f21740ab790 0x7f21740c8450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59744/0 (socket says 172.21.15.116:59744) 2024-04-25T00:03:30.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.451+0000 7f21727fc640 1 -- 172.21.15.116:0/1609120453 learned_addr learned my addr 172.21.15.116:0/1609120453 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:03:30.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.455+0000 7f2172ffd640 1 --2- 172.21.15.116:0/1609120453 >> v2:172.21.15.174:3300/0 conn(0x7f2174152d60 0x7f21740ab250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:30.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.455+0000 7f21727fc640 1 -- 172.21.15.116:0/1609120453 >> v2:172.21.15.174:3300/0 conn(0x7f2174152d60 msgr2=0x7f21740ab250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:03:30.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.455+0000 7f21727fc640 1 --2- 172.21.15.116:0/1609120453 >> v2:172.21.15.174:3300/0 conn(0x7f2174152d60 0x7f21740ab250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:30.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.455+0000 7f21727fc640 1 -- 172.21.15.116:0/1609120453 >> v1:172.21.15.174:6789/0 conn(0x7f2174153140 legacy=0x7f21740c3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:03:30.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.455+0000 7f21727fc640 1 -- 172.21.15.116:0/1609120453 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21741561a0 con 0x7f21740ab790 2024-04-25T00:03:30.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.455+0000 7f21727fc640 1 --2- 172.21.15.116:0/1609120453 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21740ab790 0x7f21740c8450 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f21740c35e0 tx=0x7f216801c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be9b4e4b91f856ad server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:30.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.455+0000 7f2171ffb640 1 -- 172.21.15.116:0/1609120453 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21680b2070 con 0x7f21740ab790 2024-04-25T00:03:30.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.455+0000 7f2171ffb640 1 -- 172.21.15.116:0/1609120453 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21680a9070 con 0x7f21740ab790 2024-04-25T00:03:30.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.455+0000 7f2171ffb640 1 -- 172.21.15.116:0/1609120453 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21680ad880 con 0x7f21740ab790 2024-04-25T00:03:30.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.455+0000 7f21796c3640 1 -- 172.21.15.116:0/1609120453 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21740ab790 msgr2=0x7f21740c8450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:30.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.455+0000 7f21796c3640 1 --2- 172.21.15.116:0/1609120453 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21740ab790 0x7f21740c8450 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f21740c35e0 tx=0x7f216801c450 comp rx=0 tx=0).stop 2024-04-25T00:03:30.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.455+0000 7f21796c3640 1 -- 172.21.15.116:0/1609120453 shutdown_connections 2024-04-25T00:03:30.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.455+0000 7f21796c3640 1 --2- 172.21.15.116:0/1609120453 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21740ab790 0x7f21740c8450 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:30.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.455+0000 7f21796c3640 1 --2- 172.21.15.116:0/1609120453 >> v2:172.21.15.174:3300/0 conn(0x7f2174152d60 0x7f21740ab250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:30.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.455+0000 7f21796c3640 1 -- 172.21.15.116:0/1609120453 >> 172.21.15.116:0/1609120453 conn(0x7f217406f0c0 msgr2=0x7f217406f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:03:30.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.455+0000 7f21796c3640 1 -- 172.21.15.116:0/1609120453 shutdown_connections 2024-04-25T00:03:30.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.455+0000 7f21796c3640 1 -- 172.21.15.116:0/1609120453 wait complete. 2024-04-25T00:03:30.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.459+0000 7f21796c3640 1 Processor -- start 2024-04-25T00:03:30.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.459+0000 7f21796c3640 1 -- start start 2024-04-25T00:03:30.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.459+0000 7f21796c3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f21740ab790 0x7f21741518a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:30.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.459+0000 7f21796c3640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2174152d60 0x7f217414e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:30.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.459+0000 7f21796c3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2174153140 0x7f217414e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:30.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.459+0000 7f21796c3640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2174162b20 con 0x7f2174153140 2024-04-25T00:03:30.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.459+0000 7f21796c3640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f21741629a0 con 0x7f21740ab790 2024-04-25T00:03:30.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.459+0000 7f21796c3640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f2174162ca0 con 0x7f2174152d60 2024-04-25T00:03:30.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.459+0000 7f21737fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2174153140 0x7f217414e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:30.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.459+0000 7f21737fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2174153140 0x7f217414e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59760/0 (socket says 172.21.15.116:59760) 2024-04-25T00:03:30.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.459+0000 7f21737fe640 1 -- 172.21.15.116:0/3605319296 learned_addr learned my addr 172.21.15.116:0/3605319296 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:03:30.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.459+0000 7f21727fc640 1 --2- 172.21.15.116:0/3605319296 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2174152d60 0x7f217414e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:30.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.459+0000 7f2172ffd640 1 --2- 172.21.15.116:0/3605319296 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f21740ab790 0x7f21741518a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:30.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.459+0000 7f21737fe640 1 -- 172.21.15.116:0/3605319296 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2174152d60 msgr2=0x7f217414e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:30.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.459+0000 7f21737fe640 1 --2- 172.21.15.116:0/3605319296 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2174152d60 0x7f217414e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:30.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.459+0000 7f21737fe640 1 -- 172.21.15.116:0/3605319296 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f21740ab790 msgr2=0x7f21741518a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:30.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.459+0000 7f21737fe640 1 --2- 172.21.15.116:0/3605319296 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f21740ab790 0x7f21741518a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:30.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.459+0000 7f21737fe640 1 -- 172.21.15.116:0/3605319296 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2174158f10 con 0x7f2174153140 2024-04-25T00:03:30.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.459+0000 7f21737fe640 1 --2- 172.21.15.116:0/3605319296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2174153140 0x7f217414e9f0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f216c00c7b0 tx=0x7f216c00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:30.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.459+0000 7f215b7fe640 1 -- 172.21.15.116:0/3605319296 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f216c00ce60 con 0x7f2174153140 2024-04-25T00:03:30.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.459+0000 7f215b7fe640 1 -- 172.21.15.116:0/3605319296 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f216c004b20 con 0x7f2174153140 2024-04-25T00:03:30.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.459+0000 7f21796c3640 1 -- 172.21.15.116:0/3605319296 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2174156bf0 con 0x7f2174153140 2024-04-25T00:03:30.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.459+0000 7f215b7fe640 1 -- 172.21.15.116:0/3605319296 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f216c00a850 con 0x7f2174153140 2024-04-25T00:03:30.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.459+0000 7f21796c3640 1 -- 172.21.15.116:0/3605319296 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21741571b0 con 0x7f2174153140 2024-04-25T00:03:30.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.459+0000 7f21796c3640 1 -- 172.21.15.116:0/3605319296 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2174068a60 con 0x7f2174153140 2024-04-25T00:03:30.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.463+0000 7f215b7fe640 1 -- 172.21.15.116:0/3605319296 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f216c00a9f0 con 0x7f2174153140 2024-04-25T00:03:30.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.463+0000 7f215b7fe640 1 --2- 172.21.15.116:0/3605319296 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f21600678f0 0x7f2160069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:30.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.463+0000 7f215b7fe640 1 -- 172.21.15.116:0/3605319296 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6838+0+0 (secure 0 0 0) 0x7f216c086510 con 0x7f2174153140 2024-04-25T00:03:30.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.463+0000 7f2172ffd640 1 --2- 172.21.15.116:0/3605319296 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f21600678f0 0x7f2160069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:30.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.463+0000 7f2172ffd640 1 --2- 172.21.15.116:0/3605319296 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f21600678f0 0x7f2160069db0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f21740aac80 tx=0x7f2164008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:30.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.467+0000 7f215b7fe640 1 -- 172.21.15.116:0/3605319296 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f216c056c70 con 0x7f2174153140 2024-04-25T00:03:30.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.679+0000 7f21796c3640 1 -- 172.21.15.116:0/3605319296 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2174067770 con 0x7f2174153140 2024-04-25T00:03:30.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.679+0000 7f215b7fe640 1 -- 172.21.15.116:0/3605319296 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+5662 (secure 0 0 0) 0x7f216c056570 con 0x7f2174153140 2024-04-25T00:03:30.685 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:03:30.685 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:01:08.748957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6837/1762356055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1762356055},{"type":"v1","addr":"172.21.15.174:6837","nonce":1762356055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270]},"id":1},{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:03:30.407122+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4735},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4735":{"gid":4735,"name":"a","rank":0,"incarnation":20,"state":"up:replay","state_seq":1,"addr":"172.21.15.116:6835/2004876235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":2004876235},{"type":"v1","addr":"172.21.15.116:6835","nonce":2004876235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:03:30.685 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 20 2024-04-25T00:03:30.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.683+0000 7f21796c3640 1 -- 172.21.15.116:0/3605319296 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f21600678f0 msgr2=0x7f2160069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:30.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.683+0000 7f21796c3640 1 --2- 172.21.15.116:0/3605319296 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f21600678f0 0x7f2160069db0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f21740aac80 tx=0x7f2164008040 comp rx=0 tx=0).stop 2024-04-25T00:03:30.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.683+0000 7f21796c3640 1 -- 172.21.15.116:0/3605319296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2174153140 msgr2=0x7f217414e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:30.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.683+0000 7f21796c3640 1 --2- 172.21.15.116:0/3605319296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2174153140 0x7f217414e9f0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f216c00c7b0 tx=0x7f216c00cc70 comp rx=0 tx=0).stop 2024-04-25T00:03:30.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.683+0000 7f21796c3640 1 -- 172.21.15.116:0/3605319296 shutdown_connections 2024-04-25T00:03:30.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.683+0000 7f21796c3640 1 --2- 172.21.15.116:0/3605319296 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f21600678f0 0x7f2160069db0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:30.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.683+0000 7f21796c3640 1 --2- 172.21.15.116:0/3605319296 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2174153140 0x7f217414e9f0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:30.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.683+0000 7f21796c3640 1 --2- 172.21.15.116:0/3605319296 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2174152d60 0x7f217414e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:30.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.683+0000 7f21796c3640 1 --2- 172.21.15.116:0/3605319296 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f21740ab790 0x7f21741518a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:30.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.683+0000 7f21796c3640 1 -- 172.21.15.116:0/3605319296 >> 172.21.15.116:0/3605319296 conn(0x7f217406f0c0 msgr2=0x7f21740c4fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:03:30.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.683+0000 7f21796c3640 1 -- 172.21.15.116:0/3605319296 shutdown_connections 2024-04-25T00:03:30.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.683+0000 7f21796c3640 1 -- 172.21.15.116:0/3605319296 wait complete. 2024-04-25T00:03:30.704 INFO:tasks.mds_thrash.fs.[b]:mds.c down, removed from mdsmap 2024-04-25T00:03:30.705 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-04-25T00:03:30.705 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:03:30.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.815+0000 7f589efea640 1 Processor -- start 2024-04-25T00:03:30.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.815+0000 7f589efea640 1 -- start start 2024-04-25T00:03:30.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.819+0000 7f589efea640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f589813e310 0x7f58980c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:30.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.819+0000 7f589efea640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f58980c2f50 0x7f58981559c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:30.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.819+0000 7f589efea640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f589806d200 con 0x7f589813ed60 2024-04-25T00:03:30.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.819+0000 7f589efea640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f5898070d40 con 0x7f589813e310 2024-04-25T00:03:30.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.819+0000 7f589efea640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f5898071020 con 0x7f58980c2f50 2024-04-25T00:03:30.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.819+0000 7f589cd86640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f589813e310 0x7f58980c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:30.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.819+0000 7f589cd86640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f589813e310 0x7f58980c2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59772/0 (socket says 172.21.15.116:59772) 2024-04-25T00:03:30.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.819+0000 7f589cd86640 1 -- 172.21.15.116:0/2575664326 learned_addr learned my addr 172.21.15.116:0/2575664326 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:03:30.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.819+0000 7f5897fff640 1 --2- 172.21.15.116:0/2575664326 >> v2:172.21.15.174:3300/0 conn(0x7f58980c2f50 0x7f58981559c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:30.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.819+0000 7f589cd86640 1 -- 172.21.15.116:0/2575664326 >> v2:172.21.15.174:3300/0 conn(0x7f58980c2f50 msgr2=0x7f58981559c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:03:30.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.819+0000 7f589cd86640 1 --2- 172.21.15.116:0/2575664326 >> v2:172.21.15.174:3300/0 conn(0x7f58980c2f50 0x7f58981559c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:30.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.819+0000 7f589cd86640 1 -- 172.21.15.116:0/2575664326 >> v1:172.21.15.174:6789/0 conn(0x7f589813ed60 legacy=0x7f5898155f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:03:30.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.819+0000 7f589cd86640 1 -- 172.21.15.116:0/2575664326 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5898156610 con 0x7f589813e310 2024-04-25T00:03:30.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.819+0000 7f5897fff640 1 --2- 172.21.15.116:0/2575664326 >> v2:172.21.15.174:3300/0 conn(0x7f58980c2f50 0x7f58981559c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:03:30.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.819+0000 7f589cd86640 1 --2- 172.21.15.116:0/2575664326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f589813e310 0x7f58980c2a10 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f5898070cd0 tx=0x7f588c01c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78754ed6d94292f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:30.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.819+0000 7f58977fe640 1 -- 172.21.15.116:0/2575664326 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f588c0b0070 con 0x7f589813e310 2024-04-25T00:03:30.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.819+0000 7f58977fe640 1 -- 172.21.15.116:0/2575664326 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f588c0ab600 con 0x7f589813e310 2024-04-25T00:03:30.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.819+0000 7f58977fe640 1 -- 172.21.15.116:0/2575664326 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f588c0ab9d0 con 0x7f589813e310 2024-04-25T00:03:30.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.823+0000 7f589efea640 1 -- 172.21.15.116:0/2575664326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f589813e310 msgr2=0x7f58980c2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:30.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.823+0000 7f589efea640 1 --2- 172.21.15.116:0/2575664326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f589813e310 0x7f58980c2a10 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f5898070cd0 tx=0x7f588c01c3c0 comp rx=0 tx=0).stop 2024-04-25T00:03:30.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.823+0000 7f589efea640 1 -- 172.21.15.116:0/2575664326 shutdown_connections 2024-04-25T00:03:30.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.823+0000 7f589efea640 1 --2- 172.21.15.116:0/2575664326 >> v2:172.21.15.174:3300/0 conn(0x7f58980c2f50 0x7f58981559c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:30.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.823+0000 7f589efea640 1 --2- 172.21.15.116:0/2575664326 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f589813e310 0x7f58980c2a10 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:30.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.823+0000 7f589efea640 1 -- 172.21.15.116:0/2575664326 >> 172.21.15.116:0/2575664326 conn(0x7f58980c3b00 msgr2=0x7f58980c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:03:30.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.823+0000 7f589efea640 1 -- 172.21.15.116:0/2575664326 shutdown_connections 2024-04-25T00:03:30.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.823+0000 7f589efea640 1 -- 172.21.15.116:0/2575664326 wait complete. 2024-04-25T00:03:30.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.823+0000 7f589efea640 1 Processor -- start 2024-04-25T00:03:30.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.823+0000 7f589efea640 1 -- start start 2024-04-25T00:03:30.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.823+0000 7f589efea640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58980c2f50 0x7f58981f1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:30.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.823+0000 7f589efea640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f589813e310 0x7f58980a9fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:30.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.823+0000 7f589efea640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f589813ed60 0x7f58980aa510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:30.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.823+0000 7f589efea640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5898162b70 con 0x7f589813e310 2024-04-25T00:03:30.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.823+0000 7f589efea640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f58981629f0 con 0x7f58980c2f50 2024-04-25T00:03:30.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.823+0000 7f589efea640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f5898162cf0 con 0x7f589813ed60 2024-04-25T00:03:30.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.823+0000 7f5897fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f589813e310 0x7f58980a9fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:30.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.823+0000 7f5897fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f589813e310 0x7f58980a9fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59786/0 (socket says 172.21.15.116:59786) 2024-04-25T00:03:30.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.823+0000 7f5897fff640 1 -- 172.21.15.116:0/3916576265 learned_addr learned my addr 172.21.15.116:0/3916576265 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:03:30.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.823+0000 7f589cd86640 1 --2- 172.21.15.116:0/3916576265 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58980c2f50 0x7f58981f1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:30.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.823+0000 7f589d587640 1 --2- 172.21.15.116:0/3916576265 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f589813ed60 0x7f58980aa510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:30.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.823+0000 7f5897fff640 1 -- 172.21.15.116:0/3916576265 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f589813ed60 msgr2=0x7f58980aa510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:30.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.823+0000 7f5897fff640 1 --2- 172.21.15.116:0/3916576265 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f589813ed60 0x7f58980aa510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:30.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.823+0000 7f5897fff640 1 -- 172.21.15.116:0/3916576265 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58980c2f50 msgr2=0x7f58981f1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:30.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.823+0000 7f5897fff640 1 --2- 172.21.15.116:0/3916576265 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58980c2f50 0x7f58981f1ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:30.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.823+0000 7f5897fff640 1 -- 172.21.15.116:0/3916576265 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58980ac000 con 0x7f589813e310 2024-04-25T00:03:30.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.823+0000 7f5897fff640 1 --2- 172.21.15.116:0/3916576265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f589813e310 0x7f58980a9fd0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f588400c820 tx=0x7f588400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:30.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.823+0000 7f58957fa640 1 -- 172.21.15.116:0/3916576265 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5884015070 con 0x7f589813e310 2024-04-25T00:03:30.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.823+0000 7f58957fa640 1 -- 172.21.15.116:0/3916576265 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5884007a30 con 0x7f589813e310 2024-04-25T00:03:30.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.823+0000 7f589efea640 1 -- 172.21.15.116:0/3916576265 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58981f2940 con 0x7f589813e310 2024-04-25T00:03:30.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.823+0000 7f58957fa640 1 -- 172.21.15.116:0/3916576265 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5884007e00 con 0x7f589813e310 2024-04-25T00:03:30.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.823+0000 7f589efea640 1 -- 172.21.15.116:0/3916576265 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58980aaa50 con 0x7f589813e310 2024-04-25T00:03:30.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.827+0000 7f589efea640 1 -- 172.21.15.116:0/3916576265 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5898068350 con 0x7f589813e310 2024-04-25T00:03:30.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.827+0000 7f58957fa640 1 -- 172.21.15.116:0/3916576265 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5884018070 con 0x7f589813e310 2024-04-25T00:03:30.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.827+0000 7f58957fa640 1 --2- 172.21.15.116:0/3916576265 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f587c0678c0 0x7f587c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:30.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.827+0000 7f58957fa640 1 -- 172.21.15.116:0/3916576265 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6838+0+0 (secure 0 0 0) 0x7f5884085f30 con 0x7f589813e310 2024-04-25T00:03:30.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.827+0000 7f589cd86640 1 --2- 172.21.15.116:0/3916576265 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f587c0678c0 0x7f587c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:30.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.827+0000 7f589cd86640 1 --2- 172.21.15.116:0/3916576265 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f587c0678c0 0x7f587c069d80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f588c0ad000 tx=0x7f588c0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:30.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:30.831+0000 7f58957fa640 1 -- 172.21.15.116:0/3916576265 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f5884056690 con 0x7f589813e310 2024-04-25T00:03:31.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.015+0000 7f589efea640 1 -- 172.21.15.116:0/3916576265 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f58981352f0 con 0x7f589813e310 2024-04-25T00:03:31.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.015+0000 7f58957fa640 1 -- 172.21.15.116:0/3916576265 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+5662 (secure 0 0 0) 0x7f5884055f90 con 0x7f589813e310 2024-04-25T00:03:31.022 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:03:31.022 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:01:08.748957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6837/1762356055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1762356055},{"type":"v1","addr":"172.21.15.174:6837","nonce":1762356055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270]},"id":1},{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:03:30.407122+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4735},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4735":{"gid":4735,"name":"a","rank":0,"incarnation":20,"state":"up:replay","state_seq":1,"addr":"172.21.15.116:6835/2004876235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":2004876235},{"type":"v1","addr":"172.21.15.116:6835","nonce":2004876235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:03:31.023 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 20 2024-04-25T00:03:31.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.019+0000 7f589efea640 1 -- 172.21.15.116:0/3916576265 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f587c0678c0 msgr2=0x7f587c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:31.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.019+0000 7f589efea640 1 --2- 172.21.15.116:0/3916576265 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f587c0678c0 0x7f587c069d80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f588c0ad000 tx=0x7f588c0a6000 comp rx=0 tx=0).stop 2024-04-25T00:03:31.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.019+0000 7f589efea640 1 -- 172.21.15.116:0/3916576265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f589813e310 msgr2=0x7f58980a9fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:31.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.019+0000 7f589efea640 1 --2- 172.21.15.116:0/3916576265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f589813e310 0x7f58980a9fd0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f588400c820 tx=0x7f588400cce0 comp rx=0 tx=0).stop 2024-04-25T00:03:31.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.019+0000 7f589efea640 1 -- 172.21.15.116:0/3916576265 shutdown_connections 2024-04-25T00:03:31.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.019+0000 7f589efea640 1 --2- 172.21.15.116:0/3916576265 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f587c0678c0 0x7f587c069d80 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:31.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.019+0000 7f589efea640 1 --2- 172.21.15.116:0/3916576265 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f589813ed60 0x7f58980aa510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:31.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.019+0000 7f589efea640 1 --2- 172.21.15.116:0/3916576265 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f589813e310 0x7f58980a9fd0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:31.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.019+0000 7f589efea640 1 --2- 172.21.15.116:0/3916576265 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58980c2f50 0x7f58981f1ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:31.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.019+0000 7f589efea640 1 -- 172.21.15.116:0/3916576265 >> 172.21.15.116:0/3916576265 conn(0x7f58980c3b00 msgr2=0x7f589806e610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:03:31.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.019+0000 7f589efea640 1 -- 172.21.15.116:0/3916576265 shutdown_connections 2024-04-25T00:03:31.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.019+0000 7f589efea640 1 -- 172.21.15.116:0/3916576265 wait complete. 2024-04-25T00:03:31.040 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 20, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:04.678737+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 9, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_4270": { "addr": "172.21.15.174:6837/1762356055", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6836", "nonce": 1762356055, "type": "v2" }, { "addr": "172.21.15.174:6837", "nonce": 1762356055, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4270, "incarnation": 4, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 0, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T00:01:08.748957+0000", "qdb_cluster": [ 4270 ], "qdb_leader": 4270, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4270 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:18.825887+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 20, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_4735": { "addr": "172.21.15.116:6835/2004876235", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6833", "nonce": 2004876235, "type": "v2" }, { "addr": "172.21.15.116:6835", "nonce": 2004876235, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4735, "incarnation": 20, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 31, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-04-25T00:03:30.407122+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4735 } } } ], "standbys": [ { "addr": "172.21.15.174:6835/2883220846", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6834", "nonce": 2883220846, "type": "v2" }, { "addr": "172.21.15.174:6835", "nonce": 2883220846, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4235, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-25T00:03:31.474 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:03:31.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.603+0000 7f476cf43640 1 Processor -- start 2024-04-25T00:03:31.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.603+0000 7f476cf43640 1 -- start start 2024-04-25T00:03:31.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.603+0000 7f476cf43640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f476806c480 0x7f47680c3060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:31.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.603+0000 7f476cf43640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f47680c1c30 0x7f4768071e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:31.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.603+0000 7f476cf43640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4768153170 0x7f476814fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:31.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.603+0000 7f476cf43640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f47680c35a0 con 0x7f4768153170 2024-04-25T00:03:31.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.603+0000 7f476cf43640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f4768153670 con 0x7f476806c480 2024-04-25T00:03:31.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.603+0000 7f476cf43640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f47680c22a0 con 0x7f47680c1c30 2024-04-25T00:03:31.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.603+0000 7f4766d9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4768153170 0x7f476814fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:31.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.603+0000 7f4766d9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4768153170 0x7f476814fa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59800/0 (socket says 172.21.15.116:59800) 2024-04-25T00:03:31.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.603+0000 7f4766d9d640 1 -- 172.21.15.116:0/402102125 learned_addr learned my addr 172.21.15.116:0/402102125 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:03:31.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.603+0000 7f4765d9b640 1 --2- 172.21.15.116:0/402102125 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f47680c1c30 0x7f4768071e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:31.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.603+0000 7f476659c640 1 --2- 172.21.15.116:0/402102125 >> v2:172.21.15.174:3300/0 conn(0x7f476806c480 0x7f47680c3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:31.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.603+0000 7f4766d9d640 1 -- 172.21.15.116:0/402102125 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f47680c1c30 msgr2=0x7f4768071e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:03:31.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.603+0000 7f4766d9d640 1 --2- 172.21.15.116:0/402102125 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f47680c1c30 0x7f4768071e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:31.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.603+0000 7f4766d9d640 1 -- 172.21.15.116:0/402102125 >> v2:172.21.15.174:3300/0 conn(0x7f476806c480 msgr2=0x7f47680c3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:03:31.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.603+0000 7f4766d9d640 1 --2- 172.21.15.116:0/402102125 >> v2:172.21.15.174:3300/0 conn(0x7f476806c480 0x7f47680c3060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:31.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.603+0000 7f4766d9d640 1 -- 172.21.15.116:0/402102125 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47680c2580 con 0x7f4768153170 2024-04-25T00:03:31.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.603+0000 7f4766d9d640 1 --2- 172.21.15.116:0/402102125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4768153170 0x7f476814fa80 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f47680aa4c0 tx=0x7f476001c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6c0d0599e5d62bb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:31.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.607+0000 7f476559a640 1 -- 172.21.15.116:0/402102125 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47600a5670 con 0x7f4768153170 2024-04-25T00:03:31.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.607+0000 7f476559a640 1 -- 172.21.15.116:0/402102125 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47600a5810 con 0x7f4768153170 2024-04-25T00:03:31.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.607+0000 7f476559a640 1 -- 172.21.15.116:0/402102125 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47600a5be0 con 0x7f4768153170 2024-04-25T00:03:31.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.607+0000 7f476cf43640 1 -- 172.21.15.116:0/402102125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4768153170 msgr2=0x7f476814fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:31.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.607+0000 7f476cf43640 1 --2- 172.21.15.116:0/402102125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4768153170 0x7f476814fa80 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f47680aa4c0 tx=0x7f476001c9f0 comp rx=0 tx=0).stop 2024-04-25T00:03:31.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.607+0000 7f476cf43640 1 -- 172.21.15.116:0/402102125 shutdown_connections 2024-04-25T00:03:31.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.607+0000 7f476cf43640 1 --2- 172.21.15.116:0/402102125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4768153170 0x7f476814fa80 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:31.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.607+0000 7f476cf43640 1 --2- 172.21.15.116:0/402102125 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f47680c1c30 0x7f4768071e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:31.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.607+0000 7f476cf43640 1 --2- 172.21.15.116:0/402102125 >> v2:172.21.15.174:3300/0 conn(0x7f476806c480 0x7f47680c3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:31.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.607+0000 7f476cf43640 1 -- 172.21.15.116:0/402102125 >> 172.21.15.116:0/402102125 conn(0x7f47680c4270 msgr2=0x7f47680c4680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:03:31.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.607+0000 7f476cf43640 1 -- 172.21.15.116:0/402102125 shutdown_connections 2024-04-25T00:03:31.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.607+0000 7f476cf43640 1 -- 172.21.15.116:0/402102125 wait complete. 2024-04-25T00:03:31.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.607+0000 7f476cf43640 1 Processor -- start 2024-04-25T00:03:31.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.607+0000 7f476cf43640 1 -- start start 2024-04-25T00:03:31.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.607+0000 7f476cf43640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f476806c480 0x7f47681ece90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:31.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.607+0000 7f476cf43640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47680c1c30 0x7f4768150f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:31.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.607+0000 7f476cf43640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4768153170 0x7f47681438e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:31.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.607+0000 7f476cf43640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f476806ca70 con 0x7f47680c1c30 2024-04-25T00:03:31.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.607+0000 7f476cf43640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4768156470 con 0x7f476806c480 2024-04-25T00:03:31.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.607+0000 7f476cf43640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f47681522d0 con 0x7f4768153170 2024-04-25T00:03:31.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.607+0000 7f4765d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47680c1c30 0x7f4768150f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:31.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.607+0000 7f4765d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47680c1c30 0x7f4768150f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59802/0 (socket says 172.21.15.116:59802) 2024-04-25T00:03:31.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.607+0000 7f4765d9b640 1 -- 172.21.15.116:0/439636905 learned_addr learned my addr 172.21.15.116:0/439636905 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:03:31.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.607+0000 7f4766d9d640 1 --2- 172.21.15.116:0/439636905 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4768153170 0x7f47681438e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:31.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.607+0000 7f476659c640 1 --2- 172.21.15.116:0/439636905 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f476806c480 0x7f47681ece90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:31.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.607+0000 7f4765d9b640 1 -- 172.21.15.116:0/439636905 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4768153170 msgr2=0x7f47681438e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:31.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.607+0000 7f4765d9b640 1 --2- 172.21.15.116:0/439636905 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4768153170 0x7f47681438e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:31.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.607+0000 7f4765d9b640 1 -- 172.21.15.116:0/439636905 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f476806c480 msgr2=0x7f47681ece90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:31.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.607+0000 7f4765d9b640 1 --2- 172.21.15.116:0/439636905 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f476806c480 0x7f47681ece90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:31.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.607+0000 7f4765d9b640 1 -- 172.21.15.116:0/439636905 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47680c5ca0 con 0x7f47680c1c30 2024-04-25T00:03:31.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.607+0000 7f4766d9d640 1 --2- 172.21.15.116:0/439636905 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4768153170 0x7f47681438e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:03:31.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.607+0000 7f4765d9b640 1 --2- 172.21.15.116:0/439636905 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47680c1c30 0x7f4768150f80 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f475c00d800 tx=0x7f475c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:31.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.607+0000 7f4756ffd640 1 -- 172.21.15.116:0/439636905 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f475c00b6c0 con 0x7f47680c1c30 2024-04-25T00:03:31.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.607+0000 7f4756ffd640 1 -- 172.21.15.116:0/439636905 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f475c019070 con 0x7f47680c1c30 2024-04-25T00:03:31.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.607+0000 7f4756ffd640 1 -- 172.21.15.116:0/439636905 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f475c00ba60 con 0x7f47680c1c30 2024-04-25T00:03:31.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.607+0000 7f476cf43640 1 -- 172.21.15.116:0/439636905 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47681514c0 con 0x7f47680c1c30 2024-04-25T00:03:31.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.607+0000 7f476cf43640 1 -- 172.21.15.116:0/439636905 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4768151ff0 con 0x7f47680c1c30 2024-04-25T00:03:31.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.611+0000 7f476cf43640 1 -- 172.21.15.116:0/439636905 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4768068a60 con 0x7f47680c1c30 2024-04-25T00:03:31.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.611+0000 7f4756ffd640 1 -- 172.21.15.116:0/439636905 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f475c00bc00 con 0x7f47680c1c30 2024-04-25T00:03:31.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.611+0000 7f4756ffd640 1 --2- 172.21.15.116:0/439636905 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f47480678c0 0x7f4748069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:31.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.611+0000 7f4756ffd640 1 -- 172.21.15.116:0/439636905 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6838+0+0 (secure 0 0 0) 0x7f475c086500 con 0x7f47680c1c30 2024-04-25T00:03:31.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.611+0000 7f476659c640 1 --2- 172.21.15.116:0/439636905 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f47480678c0 0x7f4748069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:31.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.611+0000 7f476659c640 1 --2- 172.21.15.116:0/439636905 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f47480678c0 0x7f4748069d80 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f475800ba30 tx=0x7f4758008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:31.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.615+0000 7f4756ffd640 1 -- 172.21.15.116:0/439636905 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f475c056c40 con 0x7f47680c1c30 2024-04-25T00:03:31.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.803+0000 7f476cf43640 1 -- 172.21.15.116:0/439636905 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4768150ce0 con 0x7f47680c1c30 2024-04-25T00:03:31.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.803+0000 7f4756ffd640 1 -- 172.21.15.116:0/439636905 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+5666 (secure 0 0 0) 0x7f475c056540 con 0x7f47680c1c30 2024-04-25T00:03:31.809 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:03:31.809 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:01:08.748957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6837/1762356055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1762356055},{"type":"v1","addr":"172.21.15.174:6837","nonce":1762356055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270]},"id":1},{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:03:30.591874+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4735},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4735":{"gid":4735,"name":"a","rank":0,"incarnation":20,"state":"up:reconnect","state_seq":11,"addr":"172.21.15.116:6835/2004876235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":2004876235},{"type":"v1","addr":"172.21.15.116:6835","nonce":2004876235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:03:31.810 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 21 2024-04-25T00:03:31.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.807+0000 7f476cf43640 1 -- 172.21.15.116:0/439636905 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f47480678c0 msgr2=0x7f4748069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:31.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.807+0000 7f476cf43640 1 --2- 172.21.15.116:0/439636905 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f47480678c0 0x7f4748069d80 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f475800ba30 tx=0x7f4758008040 comp rx=0 tx=0).stop 2024-04-25T00:03:31.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.807+0000 7f476cf43640 1 -- 172.21.15.116:0/439636905 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47680c1c30 msgr2=0x7f4768150f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:31.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.807+0000 7f476cf43640 1 --2- 172.21.15.116:0/439636905 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47680c1c30 0x7f4768150f80 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f475c00d800 tx=0x7f475c00dcc0 comp rx=0 tx=0).stop 2024-04-25T00:03:31.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.807+0000 7f476cf43640 1 -- 172.21.15.116:0/439636905 shutdown_connections 2024-04-25T00:03:31.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.807+0000 7f476cf43640 1 --2- 172.21.15.116:0/439636905 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f47480678c0 0x7f4748069d80 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:31.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.807+0000 7f476cf43640 1 --2- 172.21.15.116:0/439636905 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4768153170 0x7f47681438e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:31.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.807+0000 7f476cf43640 1 --2- 172.21.15.116:0/439636905 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47680c1c30 0x7f4768150f80 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:31.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.807+0000 7f476cf43640 1 --2- 172.21.15.116:0/439636905 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f476806c480 0x7f47681ece90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:31.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.807+0000 7f476cf43640 1 -- 172.21.15.116:0/439636905 >> 172.21.15.116:0/439636905 conn(0x7f47680c4270 msgr2=0x7f4768155470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:03:31.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.807+0000 7f476cf43640 1 -- 172.21.15.116:0/439636905 shutdown_connections 2024-04-25T00:03:31.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:31.807+0000 7f476cf43640 1 -- 172.21.15.116:0/439636905 wait complete. 2024-04-25T00:03:31.826 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T00:03:33.041 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:03:33.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.163+0000 7fe9839fe640 1 Processor -- start 2024-04-25T00:03:33.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.167+0000 7fe9839fe640 1 -- start start 2024-04-25T00:03:33.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.167+0000 7fe9839fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe97c071d90 0x7fe97c1563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:33.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.167+0000 7fe9839fe640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe97c071220 con 0x7fe97c0a94a0 2024-04-25T00:03:33.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.167+0000 7fe9839fe640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe97c152420 con 0x7fe97c14c720 2024-04-25T00:03:33.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.167+0000 7fe9839fe640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe97c153460 con 0x7fe97c071d90 2024-04-25T00:03:33.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.167+0000 7fe980f99640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fe97c0a94a0 0x7fe97c071680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:33126/0 (socket says 172.21.15.116:33126) 2024-04-25T00:03:33.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.167+0000 7fe980f99640 1 -- 172.21.15.116:0/74580975 learned_addr learned my addr 172.21.15.116:0/74580975 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:03:33.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.167+0000 7fe981f9b640 1 --2- 172.21.15.116:0/74580975 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe97c071d90 0x7fe97c1563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:33.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.167+0000 7fe973fff640 1 -- 172.21.15.116:0/74580975 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1269335200 0 0) 0x7fe97c071220 con 0x7fe97c0a94a0 2024-04-25T00:03:33.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.167+0000 7fe973fff640 1 -- 172.21.15.116:0/74580975 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe968003530 con 0x7fe97c0a94a0 2024-04-25T00:03:33.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.167+0000 7fe973fff640 1 -- 172.21.15.116:0/74580975 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1041653912 0 0) 0x7fe968003530 con 0x7fe97c0a94a0 2024-04-25T00:03:33.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.167+0000 7fe973fff640 1 -- 172.21.15.116:0/74580975 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe97c071d90 msgr2=0x7fe97c1563d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:03:33.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.167+0000 7fe973fff640 1 --2- 172.21.15.116:0/74580975 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe97c071d90 0x7fe97c1563d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:33.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.167+0000 7fe973fff640 1 -- 172.21.15.116:0/74580975 >> v1:172.21.15.174:6789/0 conn(0x7fe97c14c720 legacy=0x7fe97c0c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:33.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.167+0000 7fe973fff640 1 -- 172.21.15.116:0/74580975 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe97c0721c0 con 0x7fe97c0a94a0 2024-04-25T00:03:33.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.167+0000 7fe973fff640 1 -- 172.21.15.116:0/74580975 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7fe96c001680 con 0x7fe97c0a94a0 2024-04-25T00:03:33.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.167+0000 7fe973fff640 1 -- 172.21.15.116:0/74580975 >> v1:172.21.15.116:6789/0 conn(0x7fe97c0a94a0 legacy=0x7fe97c071680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:33.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.167+0000 7fe973fff640 1 --2- 172.21.15.116:0/74580975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9680039f0 0x7fe968003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:33.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.167+0000 7fe973fff640 1 --2- 172.21.15.116:0/74580975 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe97c071d90 0x7fe968004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:33.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.167+0000 7fe973fff640 1 --2- 172.21.15.116:0/74580975 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe968004d50 0x7fe9680051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:33.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.167+0000 7fe973fff640 1 -- 172.21.15.116:0/74580975 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe97c153460 con 0x7fe9680039f0 2024-04-25T00:03:33.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.167+0000 7fe973fff640 1 -- 172.21.15.116:0/74580975 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe96800bab0 con 0x7fe968004d50 2024-04-25T00:03:33.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.167+0000 7fe973fff640 1 -- 172.21.15.116:0/74580975 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe96800bcb0 con 0x7fe97c071d90 2024-04-25T00:03:33.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.167+0000 7fe98179a640 1 --2- 172.21.15.116:0/74580975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9680039f0 0x7fe968003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:33.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.167+0000 7fe981f9b640 1 --2- 172.21.15.116:0/74580975 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe968004d50 0x7fe9680051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:33.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.167+0000 7fe98179a640 1 -- 172.21.15.116:0/74580975 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe97c071d90 msgr2=0x7fe968004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:03:33.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.167+0000 7fe98179a640 1 --2- 172.21.15.116:0/74580975 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe97c071d90 0x7fe968004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:33.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.167+0000 7fe98179a640 1 -- 172.21.15.116:0/74580975 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe968004d50 msgr2=0x7fe9680051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:03:33.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.167+0000 7fe98179a640 1 --2- 172.21.15.116:0/74580975 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe968004d50 0x7fe9680051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:33.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.167+0000 7fe98179a640 1 -- 172.21.15.116:0/74580975 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe968003530 con 0x7fe9680039f0 2024-04-25T00:03:33.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.171+0000 7fe98179a640 1 --2- 172.21.15.116:0/74580975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9680039f0 0x7fe968003dd0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fe978001a30 tx=0x7fe97801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=238154fd71e1b68 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:33.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.171+0000 7fe973fff640 1 -- 172.21.15.116:0/74580975 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe978001630 con 0x7fe9680039f0 2024-04-25T00:03:33.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.171+0000 7fe973fff640 1 -- 172.21.15.116:0/74580975 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9780b0070 con 0x7fe9680039f0 2024-04-25T00:03:33.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.171+0000 7fe973fff640 1 -- 172.21.15.116:0/74580975 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9780ab880 con 0x7fe9680039f0 2024-04-25T00:03:33.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.171+0000 7fe9839fe640 1 -- 172.21.15.116:0/74580975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9680039f0 msgr2=0x7fe968003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:33.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.171+0000 7fe9839fe640 1 --2- 172.21.15.116:0/74580975 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9680039f0 0x7fe968003dd0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fe978001a30 tx=0x7fe97801c520 comp rx=0 tx=0).stop 2024-04-25T00:03:33.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.171+0000 7fe981f9b640 1 -- 172.21.15.116:0/74580975 reap_dead start 2024-04-25T00:03:33.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.171+0000 7fe9839fe640 1 -- 172.21.15.116:0/74580975 shutdown_connections 2024-04-25T00:03:33.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.171+0000 7fe9839fe640 1 -- 172.21.15.116:0/74580975 >> 172.21.15.116:0/74580975 conn(0x7fe97c0c3b00 msgr2=0x7fe97c0c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:03:33.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.171+0000 7fe9839fe640 1 -- 172.21.15.116:0/74580975 shutdown_connections 2024-04-25T00:03:33.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.171+0000 7fe9839fe640 1 -- 172.21.15.116:0/74580975 wait complete. 2024-04-25T00:03:33.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.171+0000 7fe9839fe640 1 Processor -- start 2024-04-25T00:03:33.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.171+0000 7fe9839fe640 1 -- start start 2024-04-25T00:03:33.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.171+0000 7fe9839fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9680039f0 0x7fe97c145ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:33.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.171+0000 7fe9839fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe97c0c2850 0x7fe97c0aae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:33.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.171+0000 7fe9839fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe97c0ab3b0 0x7fe97c1ef8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:33.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.171+0000 7fe9839fe640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe97c152c60 con 0x7fe97c0c2850 2024-04-25T00:03:33.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.171+0000 7fe9839fe640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe97c152980 con 0x7fe9680039f0 2024-04-25T00:03:33.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.171+0000 7fe9839fe640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe97c152f40 con 0x7fe97c0ab3b0 2024-04-25T00:03:33.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.171+0000 7fe980f99640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe97c0c2850 0x7fe97c0aae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:33.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.171+0000 7fe980f99640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe97c0c2850 0x7fe97c0aae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59822/0 (socket says 172.21.15.116:59822) 2024-04-25T00:03:33.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.171+0000 7fe980f99640 1 -- 172.21.15.116:0/2433709019 learned_addr learned my addr 172.21.15.116:0/2433709019 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:03:33.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.175+0000 7fe981f9b640 1 --2- 172.21.15.116:0/2433709019 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe97c0ab3b0 0x7fe97c1ef8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:33.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.175+0000 7fe98179a640 1 --2- 172.21.15.116:0/2433709019 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9680039f0 0x7fe97c145ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:33.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.175+0000 7fe980f99640 1 -- 172.21.15.116:0/2433709019 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe97c0ab3b0 msgr2=0x7fe97c1ef8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:33.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.175+0000 7fe980f99640 1 --2- 172.21.15.116:0/2433709019 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe97c0ab3b0 0x7fe97c1ef8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:33.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.175+0000 7fe980f99640 1 -- 172.21.15.116:0/2433709019 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9680039f0 msgr2=0x7fe97c145ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:33.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.175+0000 7fe980f99640 1 --2- 172.21.15.116:0/2433709019 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9680039f0 0x7fe97c145ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:33.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.175+0000 7fe980f99640 1 -- 172.21.15.116:0/2433709019 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe97c1ec6b0 con 0x7fe97c0c2850 2024-04-25T00:03:33.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.175+0000 7fe980f99640 1 --2- 172.21.15.116:0/2433709019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe97c0c2850 0x7fe97c0aae70 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fe96c00c6b0 tx=0x7fe96c00cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:33.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.175+0000 7fe971ffb640 1 -- 172.21.15.116:0/2433709019 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe96c018070 con 0x7fe97c0c2850 2024-04-25T00:03:33.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.175+0000 7fe971ffb640 1 -- 172.21.15.116:0/2433709019 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe96c0072f0 con 0x7fe97c0c2850 2024-04-25T00:03:33.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.175+0000 7fe9839fe640 1 -- 172.21.15.116:0/2433709019 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe97c1ec990 con 0x7fe97c0c2850 2024-04-25T00:03:33.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.175+0000 7fe971ffb640 1 -- 172.21.15.116:0/2433709019 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe96c0047f0 con 0x7fe97c0c2850 2024-04-25T00:03:33.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.175+0000 7fe9839fe640 1 -- 172.21.15.116:0/2433709019 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe97c1ecf50 con 0x7fe97c0c2850 2024-04-25T00:03:33.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.175+0000 7fe9839fe640 1 -- 172.21.15.116:0/2433709019 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe97c068350 con 0x7fe97c0c2850 2024-04-25T00:03:33.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.175+0000 7fe971ffb640 1 -- 172.21.15.116:0/2433709019 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe96c01b070 con 0x7fe97c0c2850 2024-04-25T00:03:33.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.175+0000 7fe971ffb640 1 --2- 172.21.15.116:0/2433709019 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe9600678c0 0x7fe960069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:33.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.175+0000 7fe971ffb640 1 -- 172.21.15.116:0/2433709019 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6838+0+0 (secure 0 0 0) 0x7fe96c0854d0 con 0x7fe97c0c2850 2024-04-25T00:03:33.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.175+0000 7fe98179a640 1 --2- 172.21.15.116:0/2433709019 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe9600678c0 0x7fe960069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:33.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.175+0000 7fe98179a640 1 --2- 172.21.15.116:0/2433709019 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe9600678c0 0x7fe960069d80 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fe9780af000 tx=0x7fe9780a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:33.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.179+0000 7fe971ffb640 1 -- 172.21.15.116:0/2433709019 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fe96c055c30 con 0x7fe97c0c2850 2024-04-25T00:03:33.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.383+0000 7fe9839fe640 1 -- 172.21.15.116:0/2433709019 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe97c0c3640 con 0x7fe97c0c2850 2024-04-25T00:03:33.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.383+0000 7fe971ffb640 1 -- 172.21.15.116:0/2433709019 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+5663 (secure 0 0 0) 0x7fe96c055530 con 0x7fe97c0c2850 2024-04-25T00:03:33.390 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:03:33.390 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:01:08.748957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6837/1762356055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1762356055},{"type":"v1","addr":"172.21.15.174:6837","nonce":1762356055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270]},"id":1},{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:03:31.570627+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4735},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4735":{"gid":4735,"name":"a","rank":0,"incarnation":20,"state":"up:rejoin","state_seq":12,"addr":"172.21.15.116:6835/2004876235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":2004876235},{"type":"v1","addr":"172.21.15.116:6835","nonce":2004876235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:03:33.390 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 22 2024-04-25T00:03:33.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.387+0000 7fe9839fe640 1 -- 172.21.15.116:0/2433709019 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe9600678c0 msgr2=0x7fe960069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:33.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.387+0000 7fe9839fe640 1 --2- 172.21.15.116:0/2433709019 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe9600678c0 0x7fe960069d80 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fe9780af000 tx=0x7fe9780a7000 comp rx=0 tx=0).stop 2024-04-25T00:03:33.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.387+0000 7fe9839fe640 1 -- 172.21.15.116:0/2433709019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe97c0c2850 msgr2=0x7fe97c0aae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:33.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.387+0000 7fe9839fe640 1 --2- 172.21.15.116:0/2433709019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe97c0c2850 0x7fe97c0aae70 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fe96c00c6b0 tx=0x7fe96c00cb70 comp rx=0 tx=0).stop 2024-04-25T00:03:33.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.387+0000 7fe9839fe640 1 -- 172.21.15.116:0/2433709019 shutdown_connections 2024-04-25T00:03:33.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.387+0000 7fe9839fe640 1 --2- 172.21.15.116:0/2433709019 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe9600678c0 0x7fe960069d80 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:33.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.387+0000 7fe9839fe640 1 --2- 172.21.15.116:0/2433709019 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe97c0ab3b0 0x7fe97c1ef8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:33.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.387+0000 7fe9839fe640 1 --2- 172.21.15.116:0/2433709019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe97c0c2850 0x7fe97c0aae70 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:33.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.387+0000 7fe9839fe640 1 --2- 172.21.15.116:0/2433709019 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9680039f0 0x7fe97c145ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:33.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.387+0000 7fe9839fe640 1 -- 172.21.15.116:0/2433709019 >> 172.21.15.116:0/2433709019 conn(0x7fe97c0c3b00 msgr2=0x7fe97c0c2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:03:33.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.387+0000 7fe9839fe640 1 -- 172.21.15.116:0/2433709019 shutdown_connections 2024-04-25T00:03:33.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.387+0000 7fe9839fe640 1 -- 172.21.15.116:0/2433709019 wait complete. 2024-04-25T00:03:33.409 INFO:tasks.mds_thrash.fs.[b]:no change 2024-04-25T00:03:33.827 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:03:33.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.955+0000 7f259f5c1640 1 Processor -- start 2024-04-25T00:03:33.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.955+0000 7f259f5c1640 1 -- start start 2024-04-25T00:03:33.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.955+0000 7f259f5c1640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2598152fb0 0x7f25980c25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:33.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.955+0000 7f259f5c1640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25980c3650 con 0x7f25980aa740 2024-04-25T00:03:33.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.955+0000 7f259f5c1640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25980c3930 con 0x7f25981592e0 2024-04-25T00:03:33.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.955+0000 7f259f5c1640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f25980c2bb0 con 0x7f2598152fb0 2024-04-25T00:03:33.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.955+0000 7f259d35d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2598152fb0 0x7f25980c25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:33.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.955+0000 7f259d35d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2598152fb0 0x7f25980c25d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59828/0 (socket says 172.21.15.116:59828) 2024-04-25T00:03:33.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.955+0000 7f259d35d640 1 -- 172.21.15.116:0/1166741561 learned_addr learned my addr 172.21.15.116:0/1166741561 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:03:33.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.955+0000 7f259cb5c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f25980aa740 0x7f25980c4150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:33128/0 (socket says 172.21.15.116:33128) 2024-04-25T00:03:33.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.955+0000 7f259d35d640 1 -- 172.21.15.116:0/1166741561 >> v1:172.21.15.174:6789/0 conn(0x7f25981592e0 legacy=0x7f2598159750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:03:33.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.955+0000 7f259d35d640 1 -- 172.21.15.116:0/1166741561 >> v1:172.21.15.116:6789/0 conn(0x7f25980aa740 legacy=0x7f25980c4150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:33.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.955+0000 7f259d35d640 1 -- 172.21.15.116:0/1166741561 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25980c3c10 con 0x7f2598152fb0 2024-04-25T00:03:33.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.955+0000 7f258ffff640 1 -- 172.21.15.116:0/1166741561 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 899772472 0 0) 0x7f25980c3650 con 0x7f25980aa740 2024-04-25T00:03:33.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.959+0000 7f259d35d640 1 --2- 172.21.15.116:0/1166741561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2598152fb0 0x7f25980c25d0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f2588009870 tx=0x7f258801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ea097fba8325453 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:33.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.959+0000 7f258ffff640 1 -- 172.21.15.116:0/1166741561 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25880b1070 con 0x7f2598152fb0 2024-04-25T00:03:33.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.959+0000 7f258ffff640 1 -- 172.21.15.116:0/1166741561 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25880ac600 con 0x7f2598152fb0 2024-04-25T00:03:33.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.959+0000 7f258ffff640 1 -- 172.21.15.116:0/1166741561 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25880ac9d0 con 0x7f2598152fb0 2024-04-25T00:03:33.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.959+0000 7f259f5c1640 1 -- 172.21.15.116:0/1166741561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2598152fb0 msgr2=0x7f25980c25d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:33.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.959+0000 7f259f5c1640 1 --2- 172.21.15.116:0/1166741561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2598152fb0 0x7f25980c25d0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f2588009870 tx=0x7f258801c490 comp rx=0 tx=0).stop 2024-04-25T00:03:33.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.959+0000 7f259f5c1640 1 -- 172.21.15.116:0/1166741561 shutdown_connections 2024-04-25T00:03:33.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.959+0000 7f259f5c1640 1 --2- 172.21.15.116:0/1166741561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2598152fb0 0x7f25980c25d0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:33.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.959+0000 7f259f5c1640 1 -- 172.21.15.116:0/1166741561 >> 172.21.15.116:0/1166741561 conn(0x7f259806b3e0 msgr2=0x7f259806b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:03:33.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.959+0000 7f259f5c1640 1 -- 172.21.15.116:0/1166741561 shutdown_connections 2024-04-25T00:03:33.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.959+0000 7f259f5c1640 1 -- 172.21.15.116:0/1166741561 wait complete. 2024-04-25T00:03:33.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.959+0000 7f259f5c1640 1 Processor -- start 2024-04-25T00:03:33.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.959+0000 7f259f5c1640 1 -- start start 2024-04-25T00:03:33.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.959+0000 7f259f5c1640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f25980aa740 0x7f25981eeb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:33.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.959+0000 7f259f5c1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2598152fb0 0x7f25981f1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:33.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.959+0000 7f259f5c1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25981592e0 0x7f25981584a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:33.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.959+0000 7f259f5c1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2598163a60 con 0x7f2598152fb0 2024-04-25T00:03:33.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.959+0000 7f259f5c1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f25981638e0 con 0x7f25981592e0 2024-04-25T00:03:33.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.959+0000 7f259f5c1640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f2598163be0 con 0x7f25980aa740 2024-04-25T00:03:33.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.959+0000 7f259cb5c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2598152fb0 0x7f25981f1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:33.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.959+0000 7f259cb5c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2598152fb0 0x7f25981f1f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59834/0 (socket says 172.21.15.116:59834) 2024-04-25T00:03:33.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.959+0000 7f259cb5c640 1 -- 172.21.15.116:0/2925164138 learned_addr learned my addr 172.21.15.116:0/2925164138 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:03:33.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.959+0000 7f259db5e640 1 --2- 172.21.15.116:0/2925164138 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25981592e0 0x7f25981584a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:33.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.959+0000 7f259d35d640 1 --2- 172.21.15.116:0/2925164138 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f25980aa740 0x7f25981eeb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:33.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.959+0000 7f259cb5c640 1 -- 172.21.15.116:0/2925164138 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f25980aa740 msgr2=0x7f25981eeb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:33.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.959+0000 7f259cb5c640 1 --2- 172.21.15.116:0/2925164138 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f25980aa740 0x7f25981eeb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:33.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.959+0000 7f259cb5c640 1 -- 172.21.15.116:0/2925164138 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25981592e0 msgr2=0x7f25981584a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:33.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.959+0000 7f259cb5c640 1 --2- 172.21.15.116:0/2925164138 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25981592e0 0x7f25981584a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:33.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.959+0000 7f259cb5c640 1 -- 172.21.15.116:0/2925164138 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2598145020 con 0x7f2598152fb0 2024-04-25T00:03:33.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.959+0000 7f259db5e640 1 --2- 172.21.15.116:0/2925164138 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25981592e0 0x7f25981584a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:03:33.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.959+0000 7f259cb5c640 1 --2- 172.21.15.116:0/2925164138 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2598152fb0 0x7f25981f1f80 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f259400c480 tx=0x7f259400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:33.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.959+0000 7f258dffb640 1 -- 172.21.15.116:0/2925164138 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2594018070 con 0x7f2598152fb0 2024-04-25T00:03:33.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.959+0000 7f258dffb640 1 -- 172.21.15.116:0/2925164138 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2594007500 con 0x7f2598152fb0 2024-04-25T00:03:33.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.959+0000 7f259f5c1640 1 -- 172.21.15.116:0/2925164138 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2598145300 con 0x7f2598152fb0 2024-04-25T00:03:33.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.959+0000 7f258dffb640 1 -- 172.21.15.116:0/2925164138 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2594004a30 con 0x7f2598152fb0 2024-04-25T00:03:33.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.959+0000 7f259f5c1640 1 -- 172.21.15.116:0/2925164138 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25981ee350 con 0x7f2598152fb0 2024-04-25T00:03:33.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.963+0000 7f259f5c1640 1 -- 172.21.15.116:0/2925164138 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2598068350 con 0x7f2598152fb0 2024-04-25T00:03:33.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.963+0000 7f258dffb640 1 -- 172.21.15.116:0/2925164138 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f259401b070 con 0x7f2598152fb0 2024-04-25T00:03:33.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.963+0000 7f258dffb640 1 --2- 172.21.15.116:0/2925164138 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f257c0678c0 0x7f257c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:33.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.963+0000 7f258dffb640 1 -- 172.21.15.116:0/2925164138 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6838+0+0 (secure 0 0 0) 0x7f25940853e0 con 0x7f2598152fb0 2024-04-25T00:03:33.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.963+0000 7f259d35d640 1 --2- 172.21.15.116:0/2925164138 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f257c0678c0 0x7f257c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:33.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.963+0000 7f259d35d640 1 --2- 172.21.15.116:0/2925164138 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f257c0678c0 0x7f257c069d80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f25880ae000 tx=0x7f2588008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:33.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:33.967+0000 7f258dffb640 1 -- 172.21.15.116:0/2925164138 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f2594055b40 con 0x7f2598152fb0 2024-04-25T00:03:34.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:34.163+0000 7f259f5c1640 1 -- 172.21.15.116:0/2925164138 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f25980c23b0 con 0x7f2598152fb0 2024-04-25T00:03:34.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:34.163+0000 7f258dffb640 1 -- 172.21.15.116:0/2925164138 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+5669 (secure 0 0 0) 0x7f2594055440 con 0x7f2598152fb0 2024-04-25T00:03:34.170 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:03:34.170 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:01:08.748957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6837/1762356055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1762356055},{"type":"v1","addr":"172.21.15.174:6837","nonce":1762356055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270]},"id":1},{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:03:32.579296+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4735},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4735":{"gid":4735,"name":"a","rank":0,"incarnation":20,"state":"up:clientreplay","state_seq":13,"addr":"172.21.15.116:6835/2004876235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":2004876235},{"type":"v1","addr":"172.21.15.116:6835","nonce":2004876235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:03:34.170 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 23 2024-04-25T00:03:34.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:34.167+0000 7f259f5c1640 1 -- 172.21.15.116:0/2925164138 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f257c0678c0 msgr2=0x7f257c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:34.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:34.167+0000 7f259f5c1640 1 --2- 172.21.15.116:0/2925164138 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f257c0678c0 0x7f257c069d80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f25880ae000 tx=0x7f2588008040 comp rx=0 tx=0).stop 2024-04-25T00:03:34.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:34.167+0000 7f259f5c1640 1 -- 172.21.15.116:0/2925164138 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2598152fb0 msgr2=0x7f25981f1f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:34.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:34.167+0000 7f259f5c1640 1 --2- 172.21.15.116:0/2925164138 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2598152fb0 0x7f25981f1f80 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f259400c480 tx=0x7f259400c940 comp rx=0 tx=0).stop 2024-04-25T00:03:34.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:34.167+0000 7f259f5c1640 1 -- 172.21.15.116:0/2925164138 shutdown_connections 2024-04-25T00:03:34.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:34.167+0000 7f259f5c1640 1 --2- 172.21.15.116:0/2925164138 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f257c0678c0 0x7f257c069d80 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:34.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:34.167+0000 7f259f5c1640 1 --2- 172.21.15.116:0/2925164138 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25981592e0 0x7f25981584a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:34.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:34.167+0000 7f259f5c1640 1 --2- 172.21.15.116:0/2925164138 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2598152fb0 0x7f25981f1f80 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:34.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:34.167+0000 7f259f5c1640 1 --2- 172.21.15.116:0/2925164138 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f25980aa740 0x7f25981eeb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:34.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:34.167+0000 7f259f5c1640 1 -- 172.21.15.116:0/2925164138 >> 172.21.15.116:0/2925164138 conn(0x7f259806b3e0 msgr2=0x7f25980aae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:03:34.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:34.167+0000 7f259f5c1640 1 -- 172.21.15.116:0/2925164138 shutdown_connections 2024-04-25T00:03:34.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:34.167+0000 7f259f5c1640 1 -- 172.21.15.116:0/2925164138 wait complete. 2024-04-25T00:03:34.189 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T00:03:34.609 INFO:tasks.workunit.client.1.smithi174.stdout:checking whether build environment is sane... yes 2024-04-25T00:03:34.630 INFO:tasks.workunit.client.1.smithi174.stdout:checking for a race-free mkdir -p... /usr/bin/mkdir -p 2024-04-25T00:03:34.632 INFO:tasks.workunit.client.1.smithi174.stdout:checking for gawk... gawk 2024-04-25T00:03:34.675 INFO:tasks.workunit.client.1.smithi174.stdout:checking whether make sets $(MAKE)... yes 2024-04-25T00:03:34.758 INFO:tasks.workunit.client.1.smithi174.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-04-25T00:03:34.758 INFO:tasks.workunit.client.1.smithi174.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-04-25T00:03:34.794 INFO:tasks.workunit.client.1.smithi174.stdout:checking for style of include used by make... GNU 2024-04-25T00:03:34.806 INFO:tasks.workunit.client.1.smithi174.stdout:checking for gcc... gcc 2024-04-25T00:03:34.838 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:03:34.841 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:03:35.040 INFO:tasks.workunit.client.1.smithi174.stdout:checking whether the C compiler works... yes 2024-04-25T00:03:35.041 INFO:tasks.workunit.client.1.smithi174.stdout:checking for C compiler default output file name... a.out 2024-04-25T00:03:35.145 INFO:tasks.workunit.client.1.smithi174.stdout:checking for suffix of executables... 2024-04-25T00:03:35.236 INFO:tasks.workunit.client.1.smithi174.stdout:checking whether we are cross compiling... no 2024-04-25T00:03:35.319 INFO:tasks.workunit.client.1.smithi174.stdout:checking for suffix of object files... o 2024-04-25T00:03:35.376 INFO:tasks.workunit.client.1.smithi174.stdout:checking whether the compiler supports GNU C... yes 2024-04-25T00:03:35.410 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:03:35.427 INFO:tasks.workunit.client.1.smithi174.stdout:checking whether gcc accepts -g... yes 2024-04-25T00:03:35.514 INFO:tasks.workunit.client.1.smithi174.stdout:checking for gcc option to enable C11 features... none needed 2024-04-25T00:03:35.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.535+0000 7f8e8665c640 1 Processor -- start 2024-04-25T00:03:35.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.535+0000 7f8e8665c640 1 -- start start 2024-04-25T00:03:35.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.539+0000 7f8e8665c640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f8e800aa170 0x7f8e80155af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:35.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.539+0000 7f8e8665c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8e80156030 0x7f8e8015ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:35.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.539+0000 7f8e8665c640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e801526a0 con 0x7f8e800c4300 2024-04-25T00:03:35.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.539+0000 7f8e8665c640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f8e80153460 con 0x7f8e800aa170 2024-04-25T00:03:35.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.539+0000 7f8e8665c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f8e8006ffd0 con 0x7f8e80156030 2024-04-25T00:03:35.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.539+0000 7f8e7ffff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f8e800c4300 0x7f8e800c5100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:53038/0 (socket says 172.21.15.116:53038) 2024-04-25T00:03:35.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.539+0000 7f8e7ffff640 1 -- 172.21.15.116:0/3845777967 learned_addr learned my addr 172.21.15.116:0/3845777967 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:03:35.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.539+0000 7f8e7f7fe640 1 --2- 172.21.15.116:0/3845777967 >> v2:172.21.15.174:3300/0 conn(0x7f8e800aa170 0x7f8e80155af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:35.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.539+0000 7f8e84bf9640 1 --2- 172.21.15.116:0/3845777967 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8e80156030 0x7f8e8015ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:35.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.539+0000 7f8e7effd640 1 -- 172.21.15.116:0/3845777967 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2340980452 0 0) 0x7f8e801526a0 con 0x7f8e800c4300 2024-04-25T00:03:35.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.539+0000 7f8e7effd640 1 -- 172.21.15.116:0/3845777967 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e68003530 con 0x7f8e800c4300 2024-04-25T00:03:35.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.539+0000 7f8e84bf9640 1 -- 172.21.15.116:0/3845777967 >> v2:172.21.15.174:3300/0 conn(0x7f8e800aa170 msgr2=0x7f8e80155af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:03:35.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.539+0000 7f8e84bf9640 1 --2- 172.21.15.116:0/3845777967 >> v2:172.21.15.174:3300/0 conn(0x7f8e800aa170 0x7f8e80155af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:35.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.539+0000 7f8e84bf9640 1 -- 172.21.15.116:0/3845777967 >> v1:172.21.15.116:6789/0 conn(0x7f8e800c4300 legacy=0x7f8e800c5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:35.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.539+0000 7f8e84bf9640 1 -- 172.21.15.116:0/3845777967 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e80152980 con 0x7f8e80156030 2024-04-25T00:03:35.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.539+0000 7f8e84bf9640 1 --2- 172.21.15.116:0/3845777967 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8e80156030 0x7f8e8015ce80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8e78009d90 tx=0x7f8e7801c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f69857953cf8c8f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:35.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.539+0000 7f8e7effd640 1 -- 172.21.15.116:0/3845777967 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e780b2070 con 0x7f8e80156030 2024-04-25T00:03:35.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.539+0000 7f8e7effd640 1 -- 172.21.15.116:0/3845777967 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e780a5830 con 0x7f8e80156030 2024-04-25T00:03:35.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.539+0000 7f8e7effd640 1 -- 172.21.15.116:0/3845777967 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e780a5c00 con 0x7f8e80156030 2024-04-25T00:03:35.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.543+0000 7f8e8665c640 1 -- 172.21.15.116:0/3845777967 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8e80156030 msgr2=0x7f8e8015ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:35.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.543+0000 7f8e8665c640 1 --2- 172.21.15.116:0/3845777967 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8e80156030 0x7f8e8015ce80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8e78009d90 tx=0x7f8e7801c970 comp rx=0 tx=0).stop 2024-04-25T00:03:35.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.543+0000 7f8e8665c640 1 -- 172.21.15.116:0/3845777967 shutdown_connections 2024-04-25T00:03:35.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.543+0000 7f8e8665c640 1 --2- 172.21.15.116:0/3845777967 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8e80156030 0x7f8e8015ce80 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:35.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.543+0000 7f8e8665c640 1 --2- 172.21.15.116:0/3845777967 >> v2:172.21.15.174:3300/0 conn(0x7f8e800aa170 0x7f8e80155af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:35.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.543+0000 7f8e8665c640 1 -- 172.21.15.116:0/3845777967 >> 172.21.15.116:0/3845777967 conn(0x7f8e8006c720 msgr2=0x7f8e8006cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:03:35.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.543+0000 7f8e8665c640 1 -- 172.21.15.116:0/3845777967 shutdown_connections 2024-04-25T00:03:35.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.543+0000 7f8e8665c640 1 -- 172.21.15.116:0/3845777967 wait complete. 2024-04-25T00:03:35.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.543+0000 7f8e8665c640 1 Processor -- start 2024-04-25T00:03:35.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.543+0000 7f8e8665c640 1 -- start start 2024-04-25T00:03:35.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.543+0000 7f8e8665c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e800aa170 0x7f8e801487c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:35.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.543+0000 7f8e8665c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8e800c4300 0x7f8e8013f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:35.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.543+0000 7f8e8665c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8e80156030 0x7f8e8013f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:35.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.543+0000 7f8e8665c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8e800c4860 con 0x7f8e800aa170 2024-04-25T00:03:35.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.543+0000 7f8e8665c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8e800c46e0 con 0x7f8e80156030 2024-04-25T00:03:35.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.543+0000 7f8e8665c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f8e800aacd0 con 0x7f8e800c4300 2024-04-25T00:03:35.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.543+0000 7f8e7ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e800aa170 0x7f8e801487c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:35.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.543+0000 7f8e7ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e800aa170 0x7f8e801487c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57796/0 (socket says 172.21.15.116:57796) 2024-04-25T00:03:35.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.543+0000 7f8e7ffff640 1 -- 172.21.15.116:0/3394907031 learned_addr learned my addr 172.21.15.116:0/3394907031 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:03:35.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.543+0000 7f8e7f7fe640 1 --2- 172.21.15.116:0/3394907031 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8e800c4300 0x7f8e8013f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:35.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.543+0000 7f8e84bf9640 1 --2- 172.21.15.116:0/3394907031 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8e80156030 0x7f8e8013f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:35.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.543+0000 7f8e7ffff640 1 -- 172.21.15.116:0/3394907031 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8e800c4300 msgr2=0x7f8e8013f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:35.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.543+0000 7f8e7ffff640 1 --2- 172.21.15.116:0/3394907031 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8e800c4300 0x7f8e8013f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:35.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.543+0000 7f8e7ffff640 1 -- 172.21.15.116:0/3394907031 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8e80156030 msgr2=0x7f8e8013f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:35.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.543+0000 7f8e7ffff640 1 --2- 172.21.15.116:0/3394907031 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8e80156030 0x7f8e8013f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:35.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.543+0000 7f8e7ffff640 1 -- 172.21.15.116:0/3394907031 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e80144950 con 0x7f8e800aa170 2024-04-25T00:03:35.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.543+0000 7f8e7ffff640 1 --2- 172.21.15.116:0/3394907031 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e800aa170 0x7f8e801487c0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f8e7000c480 tx=0x7f8e7000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:35.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.543+0000 7f8e7cff9640 1 -- 172.21.15.116:0/3394907031 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e70017070 con 0x7f8e800aa170 2024-04-25T00:03:35.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.543+0000 7f8e7cff9640 1 -- 172.21.15.116:0/3394907031 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e70005cf0 con 0x7f8e800aa170 2024-04-25T00:03:35.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.543+0000 7f8e8665c640 1 -- 172.21.15.116:0/3394907031 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e801538f0 con 0x7f8e800aa170 2024-04-25T00:03:35.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.543+0000 7f8e7cff9640 1 -- 172.21.15.116:0/3394907031 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e70007290 con 0x7f8e800aa170 2024-04-25T00:03:35.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.543+0000 7f8e8665c640 1 -- 172.21.15.116:0/3394907031 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e80148d00 con 0x7f8e800aa170 2024-04-25T00:03:35.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.547+0000 7f8e8665c640 1 -- 172.21.15.116:0/3394907031 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e80068a60 con 0x7f8e800aa170 2024-04-25T00:03:35.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.547+0000 7f8e7cff9640 1 -- 172.21.15.116:0/3394907031 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8e7001a070 con 0x7f8e800aa170 2024-04-25T00:03:35.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.547+0000 7f8e7cff9640 1 --2- 172.21.15.116:0/3394907031 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8e6c0678c0 0x7f8e6c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:35.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.547+0000 7f8e7cff9640 1 -- 172.21.15.116:0/3394907031 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6838+0+0 (secure 0 0 0) 0x7f8e700857f0 con 0x7f8e800aa170 2024-04-25T00:03:35.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.547+0000 7f8e7f7fe640 1 --2- 172.21.15.116:0/3394907031 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8e6c0678c0 0x7f8e6c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:35.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.547+0000 7f8e7f7fe640 1 --2- 172.21.15.116:0/3394907031 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8e6c0678c0 0x7f8e6c069d80 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f8e80148fe0 tx=0x7f8e74009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:35.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.551+0000 7f8e7cff9640 1 -- 172.21.15.116:0/3394907031 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f8e70021020 con 0x7f8e800aa170 2024-04-25T00:03:35.718 INFO:tasks.workunit.client.1.smithi174.stdout:checking dependency style of gcc... gcc3 2024-04-25T00:03:35.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.759+0000 7f8e8665c640 1 -- 172.21.15.116:0/3394907031 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8e800a9f50 con 0x7f8e800aa170 2024-04-25T00:03:35.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.759+0000 7f8e7cff9640 1 -- 172.21.15.116:0/3394907031 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+5670 (secure 0 0 0) 0x7f8e70055960 con 0x7f8e800aa170 2024-04-25T00:03:35.765 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:03:35.765 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:01:08.748957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6837/1762356055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1762356055},{"type":"v1","addr":"172.21.15.174:6837","nonce":1762356055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270]},"id":1},{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:03:34.579825+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4735},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4735":{"gid":4735,"name":"a","rank":0,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.116:6835/2004876235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":2004876235},{"type":"v1","addr":"172.21.15.116:6835","nonce":2004876235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4735,"qdb_cluster":[4735]},"id":2}]} 2024-04-25T00:03:35.765 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 24 2024-04-25T00:03:35.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.763+0000 7f8e8665c640 1 -- 172.21.15.116:0/3394907031 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8e6c0678c0 msgr2=0x7f8e6c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:35.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.763+0000 7f8e8665c640 1 --2- 172.21.15.116:0/3394907031 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8e6c0678c0 0x7f8e6c069d80 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f8e80148fe0 tx=0x7f8e74009190 comp rx=0 tx=0).stop 2024-04-25T00:03:35.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.763+0000 7f8e8665c640 1 -- 172.21.15.116:0/3394907031 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e800aa170 msgr2=0x7f8e801487c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:35.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.763+0000 7f8e8665c640 1 --2- 172.21.15.116:0/3394907031 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e800aa170 0x7f8e801487c0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f8e7000c480 tx=0x7f8e7000c940 comp rx=0 tx=0).stop 2024-04-25T00:03:35.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.763+0000 7f8e8665c640 1 -- 172.21.15.116:0/3394907031 shutdown_connections 2024-04-25T00:03:35.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.763+0000 7f8e8665c640 1 --2- 172.21.15.116:0/3394907031 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8e6c0678c0 0x7f8e6c069d80 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:35.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.763+0000 7f8e8665c640 1 --2- 172.21.15.116:0/3394907031 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8e80156030 0x7f8e8013f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:35.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.763+0000 7f8e8665c640 1 --2- 172.21.15.116:0/3394907031 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8e800c4300 0x7f8e8013f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:35.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.763+0000 7f8e8665c640 1 --2- 172.21.15.116:0/3394907031 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8e800aa170 0x7f8e801487c0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:35.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.763+0000 7f8e8665c640 1 -- 172.21.15.116:0/3394907031 >> 172.21.15.116:0/3394907031 conn(0x7f8e8006c720 msgr2=0x7f8e800c5100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:03:35.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.763+0000 7f8e8665c640 1 -- 172.21.15.116:0/3394907031 shutdown_connections 2024-04-25T00:03:35.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.763+0000 7f8e8665c640 1 -- 172.21.15.116:0/3394907031 wait complete. 2024-04-25T00:03:35.768 INFO:tasks.workunit.client.1.smithi174.stdout:checking for stdio.h... yes 2024-04-25T00:03:35.785 INFO:tasks.mds_thrash.fs.[b]:mds.a has gained rank=0, replacing gid=4242 2024-04-25T00:03:35.786 INFO:tasks.mds_thrash.fs.[b]:waiting for 0 secs before reviving mds.c 2024-04-25T00:03:35.786 INFO:tasks.mds_thrash.fs.[b]:reviving mds.c 2024-04-25T00:03:35.786 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-25T00:03:35.786 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-25T00:03:35.789 INFO:tasks.ceph.mds.c:Started 2024-04-25T00:03:35.789 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:03:35.809 INFO:tasks.workunit.client.1.smithi174.stdout:checking for stdlib.h... yes 2024-04-25T00:03:35.860 INFO:tasks.workunit.client.1.smithi174.stdout:checking for string.h... yes 2024-04-25T00:03:35.867 INFO:tasks.ceph.mds.c.smithi116.stdout:starting mds.c at 2024-04-25T00:03:35.909 INFO:tasks.workunit.client.1.smithi174.stdout:checking for inttypes.h... yes 2024-04-25T00:03:35.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.903+0000 7f14d25c8640 1 Processor -- start 2024-04-25T00:03:35.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.903+0000 7f14d25c8640 1 -- start start 2024-04-25T00:03:35.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.903+0000 7f14d25c8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f14cc150d40 0x7f14cc154170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:35.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.903+0000 7f14d25c8640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14cc070180 con 0x7f14cc0c4c20 2024-04-25T00:03:35.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.903+0000 7f14d25c8640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14cc070460 con 0x7f14cc070770 2024-04-25T00:03:35.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.903+0000 7f14d25c8640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f14cc0c7530 con 0x7f14cc150d40 2024-04-25T00:03:35.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.903+0000 7f14cbfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f14cc150d40 0x7f14cc154170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:35.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.903+0000 7f14cb7fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f14cc0c4c20 0x7f14cc0c6e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:53044/0 (socket says 172.21.15.116:53044) 2024-04-25T00:03:35.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.903+0000 7f14cb7fe640 1 -- 172.21.15.116:0/2954134439 learned_addr learned my addr 172.21.15.116:0/2954134439 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:03:35.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.907+0000 7f14cbfff640 1 -- 172.21.15.116:0/2954134439 >> v1:172.21.15.174:6789/0 conn(0x7f14cc070770 legacy=0x7f14cc0ab200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:35.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.907+0000 7f14cbfff640 1 -- 172.21.15.116:0/2954134439 >> v1:172.21.15.116:6789/0 conn(0x7f14cc0c4c20 legacy=0x7f14cc0c6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:35.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.907+0000 7f14cbfff640 1 -- 172.21.15.116:0/2954134439 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14cc153970 con 0x7f14cc150d40 2024-04-25T00:03:35.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.907+0000 7f14caffd640 1 -- 172.21.15.116:0/2954134439 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2887253975 0 0) 0x7f14cc070460 con 0x7f14cc070770 2024-04-25T00:03:35.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.907+0000 7f14cbfff640 1 --2- 172.21.15.116:0/2954134439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14cc150d40 0x7f14cc154170 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f14bc0097c0 tx=0x7f14bc01c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98923697ec35f7fb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:35.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.907+0000 7f14caffd640 1 -- 172.21.15.116:0/2954134439 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14bc0b0070 con 0x7f14cc150d40 2024-04-25T00:03:35.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.907+0000 7f14caffd640 1 -- 172.21.15.116:0/2954134439 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14bc0ab600 con 0x7f14cc150d40 2024-04-25T00:03:35.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.907+0000 7f14caffd640 1 -- 172.21.15.116:0/2954134439 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14bc0ab9d0 con 0x7f14cc150d40 2024-04-25T00:03:35.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.907+0000 7f14d25c8640 1 -- 172.21.15.116:0/2954134439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14cc150d40 msgr2=0x7f14cc154170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:35.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.907+0000 7f14d25c8640 1 --2- 172.21.15.116:0/2954134439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14cc150d40 0x7f14cc154170 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f14bc0097c0 tx=0x7f14bc01c410 comp rx=0 tx=0).stop 2024-04-25T00:03:35.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.911+0000 7f14d25c8640 1 -- 172.21.15.116:0/2954134439 shutdown_connections 2024-04-25T00:03:35.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.911+0000 7f14d25c8640 1 --2- 172.21.15.116:0/2954134439 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14cc150d40 0x7f14cc154170 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:35.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.911+0000 7f14d25c8640 1 -- 172.21.15.116:0/2954134439 >> 172.21.15.116:0/2954134439 conn(0x7f14cc06f0c0 msgr2=0x7f14cc06f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:03:35.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.911+0000 7f14d25c8640 1 -- 172.21.15.116:0/2954134439 shutdown_connections 2024-04-25T00:03:35.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.911+0000 7f14d25c8640 1 -- 172.21.15.116:0/2954134439 wait complete. 2024-04-25T00:03:35.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.911+0000 7f14d25c8640 1 Processor -- start 2024-04-25T00:03:35.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.911+0000 7f14d25c8640 1 -- start start 2024-04-25T00:03:35.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.911+0000 7f14d25c8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14cc070770 0x7f14cc148b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:35.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.911+0000 7f14d25c8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f14cc0c4c20 0x7f14cc0acc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:35.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.911+0000 7f14d25c8640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f14cc150d40 0x7f14cc0ad1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:35.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.911+0000 7f14d25c8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f14cc06ff30 con 0x7f14cc070770 2024-04-25T00:03:35.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.911+0000 7f14d25c8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f14cc071280 con 0x7f14cc0c4c20 2024-04-25T00:03:35.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.911+0000 7f14d25c8640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f14cc151120 con 0x7f14cc150d40 2024-04-25T00:03:35.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.911+0000 7f14cbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14cc070770 0x7f14cc148b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:35.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.911+0000 7f14cbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14cc070770 0x7f14cc148b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57834/0 (socket says 172.21.15.116:57834) 2024-04-25T00:03:35.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.911+0000 7f14cbfff640 1 -- 172.21.15.116:0/1070206169 learned_addr learned my addr 172.21.15.116:0/1070206169 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:03:35.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.911+0000 7f14cb7fe640 1 --2- 172.21.15.116:0/1070206169 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f14cc0c4c20 0x7f14cc0acc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:35.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.911+0000 7f14d0b65640 1 --2- 172.21.15.116:0/1070206169 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f14cc150d40 0x7f14cc0ad1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:35.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.911+0000 7f14cbfff640 1 -- 172.21.15.116:0/1070206169 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f14cc150d40 msgr2=0x7f14cc0ad1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:35.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.911+0000 7f14cbfff640 1 --2- 172.21.15.116:0/1070206169 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f14cc150d40 0x7f14cc0ad1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:35.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.911+0000 7f14cbfff640 1 -- 172.21.15.116:0/1070206169 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f14cc0c4c20 msgr2=0x7f14cc0acc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:35.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.911+0000 7f14cbfff640 1 --2- 172.21.15.116:0/1070206169 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f14cc0c4c20 0x7f14cc0acc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:35.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.911+0000 7f14cbfff640 1 -- 172.21.15.116:0/1070206169 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14cc0ac0f0 con 0x7f14cc070770 2024-04-25T00:03:35.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.911+0000 7f14cbfff640 1 --2- 172.21.15.116:0/1070206169 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14cc070770 0x7f14cc148b20 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f14bc01a040 tx=0x7f14bc0191b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:35.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.911+0000 7f14c8ff9640 1 -- 172.21.15.116:0/1070206169 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14bc0b8020 con 0x7f14cc070770 2024-04-25T00:03:35.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.911+0000 7f14c8ff9640 1 -- 172.21.15.116:0/1070206169 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14bc0b4610 con 0x7f14cc070770 2024-04-25T00:03:35.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.911+0000 7f14d25c8640 1 -- 172.21.15.116:0/1070206169 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14cc1eba00 con 0x7f14cc070770 2024-04-25T00:03:35.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.911+0000 7f14c8ff9640 1 -- 172.21.15.116:0/1070206169 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14bc0b0040 con 0x7f14cc070770 2024-04-25T00:03:35.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.911+0000 7f14d25c8640 1 -- 172.21.15.116:0/1070206169 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14cc1ebcc0 con 0x7f14cc070770 2024-04-25T00:03:35.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.915+0000 7f14d25c8640 1 -- 172.21.15.116:0/1070206169 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14cc068a60 con 0x7f14cc070770 2024-04-25T00:03:35.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.915+0000 7f14c8ff9640 1 -- 172.21.15.116:0/1070206169 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f14bc0bb020 con 0x7f14cc070770 2024-04-25T00:03:35.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.915+0000 7f14c8ff9640 1 --2- 172.21.15.116:0/1070206169 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f14ac0678c0 0x7f14ac069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:35.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.915+0000 7f14c8ff9640 1 -- 172.21.15.116:0/1070206169 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6838+0+0 (secure 0 0 0) 0x7f14bc11faf0 con 0x7f14cc070770 2024-04-25T00:03:35.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.915+0000 7f14cb7fe640 1 --2- 172.21.15.116:0/1070206169 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f14ac0678c0 0x7f14ac069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:35.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.915+0000 7f14cb7fe640 1 --2- 172.21.15.116:0/1070206169 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f14ac0678c0 0x7f14ac069d80 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f14cc150990 tx=0x7f14b4007e90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:35.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:35.919+0000 7f14c8ff9640 1 -- 172.21.15.116:0/1070206169 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f14bc0c2030 con 0x7f14cc070770 2024-04-25T00:03:35.953 INFO:tasks.workunit.client.1.smithi174.stdout:checking for stdint.h... yes 2024-04-25T00:03:35.998 INFO:tasks.workunit.client.1.smithi174.stdout:checking for strings.h... yes 2024-04-25T00:03:36.063 INFO:tasks.workunit.client.1.smithi174.stdout:checking for sys/stat.h... yes 2024-04-25T00:03:36.121 INFO:tasks.workunit.client.1.smithi174.stdout:checking for sys/types.h... yes 2024-04-25T00:03:36.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.131+0000 7f14d25c8640 1 -- 172.21.15.116:0/1070206169 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f14cc070d30 con 0x7f14cc070770 2024-04-25T00:03:36.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.135+0000 7f14c8ff9640 1 -- 172.21.15.116:0/1070206169 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+5670 (secure 0 0 0) 0x7f14cc0ac0f0 con 0x7f14cc070770 2024-04-25T00:03:36.139 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:03:36.139 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:01:08.748957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6837/1762356055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1762356055},{"type":"v1","addr":"172.21.15.174:6837","nonce":1762356055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270]},"id":1},{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:03:34.579825+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4735},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4735":{"gid":4735,"name":"a","rank":0,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.116:6835/2004876235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":2004876235},{"type":"v1","addr":"172.21.15.116:6835","nonce":2004876235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4735,"qdb_cluster":[4735]},"id":2}]} 2024-04-25T00:03:36.139 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 24 2024-04-25T00:03:36.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.135+0000 7f14d25c8640 1 -- 172.21.15.116:0/1070206169 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f14ac0678c0 msgr2=0x7f14ac069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:36.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.135+0000 7f14d25c8640 1 --2- 172.21.15.116:0/1070206169 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f14ac0678c0 0x7f14ac069d80 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f14cc150990 tx=0x7f14b4007e90 comp rx=0 tx=0).stop 2024-04-25T00:03:36.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.135+0000 7f14d25c8640 1 -- 172.21.15.116:0/1070206169 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14cc070770 msgr2=0x7f14cc148b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:36.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.135+0000 7f14d25c8640 1 --2- 172.21.15.116:0/1070206169 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14cc070770 0x7f14cc148b20 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f14bc01a040 tx=0x7f14bc0191b0 comp rx=0 tx=0).stop 2024-04-25T00:03:36.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.135+0000 7f14d25c8640 1 -- 172.21.15.116:0/1070206169 shutdown_connections 2024-04-25T00:03:36.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.135+0000 7f14d25c8640 1 --2- 172.21.15.116:0/1070206169 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f14ac0678c0 0x7f14ac069d80 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:36.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.135+0000 7f14d25c8640 1 --2- 172.21.15.116:0/1070206169 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f14cc150d40 0x7f14cc0ad1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:36.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.135+0000 7f14d25c8640 1 --2- 172.21.15.116:0/1070206169 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f14cc0c4c20 0x7f14cc0acc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:36.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.135+0000 7f14d25c8640 1 --2- 172.21.15.116:0/1070206169 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14cc070770 0x7f14cc148b20 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:36.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.135+0000 7f14d25c8640 1 -- 172.21.15.116:0/1070206169 >> 172.21.15.116:0/1070206169 conn(0x7f14cc06f0c0 msgr2=0x7f14cc0ab150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:03:36.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.135+0000 7f14d25c8640 1 -- 172.21.15.116:0/1070206169 shutdown_connections 2024-04-25T00:03:36.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.135+0000 7f14d25c8640 1 -- 172.21.15.116:0/1070206169 wait complete. 2024-04-25T00:03:36.158 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2024-04-25T00:03:36.183 INFO:tasks.workunit.client.1.smithi174.stdout:checking for unistd.h... yes 2024-04-25T00:03:36.191 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:03:36.239 INFO:tasks.workunit.client.1.smithi174.stdout:checking for wchar.h... yes 2024-04-25T00:03:36.293 INFO:tasks.workunit.client.1.smithi174.stdout:checking for minix/config.h... no 2024-04-25T00:03:36.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.315+0000 7f24b4dc2640 1 Processor -- start 2024-04-25T00:03:36.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.315+0000 7f24b4dc2640 1 -- start start 2024-04-25T00:03:36.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.315+0000 7f24b4dc2640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f24b014fc90 0x7f24b0073120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:36.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.315+0000 7f24b4dc2640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f24b006c7b0 0x7f24b0155ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:36.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.315+0000 7f24b4dc2640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f24b0071520 0x7f24b0155ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:36.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.315+0000 7f24b4dc2640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f24b006c320 con 0x7f24b014fc90 2024-04-25T00:03:36.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.315+0000 7f24b4dc2640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f24b0072810 con 0x7f24b0071520 2024-04-25T00:03:36.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.315+0000 7f24b4dc2640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f24b00a8a70 con 0x7f24b006c7b0 2024-04-25T00:03:36.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.315+0000 7f24ae59c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f24b014fc90 0x7f24b0073120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:36.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.315+0000 7f24ae59c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f24b014fc90 0x7f24b0073120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57844/0 (socket says 172.21.15.116:57844) 2024-04-25T00:03:36.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.315+0000 7f24ae59c640 1 -- 172.21.15.116:0/1200175376 learned_addr learned my addr 172.21.15.116:0/1200175376 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:03:36.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.315+0000 7f24add9b640 1 --2- 172.21.15.116:0/1200175376 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f24b006c7b0 0x7f24b0155ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:36.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.315+0000 7f24aed9d640 1 --2- 172.21.15.116:0/1200175376 >> v2:172.21.15.174:3300/0 conn(0x7f24b0071520 0x7f24b0155ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:36.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.315+0000 7f24ae59c640 1 -- 172.21.15.116:0/1200175376 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f24b006c7b0 msgr2=0x7f24b0155ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:03:36.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.315+0000 7f24ae59c640 1 --2- 172.21.15.116:0/1200175376 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f24b006c7b0 0x7f24b0155ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:36.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.315+0000 7f24ae59c640 1 -- 172.21.15.116:0/1200175376 >> v2:172.21.15.174:3300/0 conn(0x7f24b0071520 msgr2=0x7f24b0155ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:03:36.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.315+0000 7f24ae59c640 1 --2- 172.21.15.116:0/1200175376 >> v2:172.21.15.174:3300/0 conn(0x7f24b0071520 0x7f24b0155ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:36.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.315+0000 7f24ae59c640 1 -- 172.21.15.116:0/1200175376 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24b01568a0 con 0x7f24b014fc90 2024-04-25T00:03:36.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.319+0000 7f24ae59c640 1 --2- 172.21.15.116:0/1200175376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24b014fc90 0x7f24b0073120 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f24b0071b80 tx=0x7f24a801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3683d5ceb885760e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:36.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.319+0000 7f24ad59a640 1 -- 172.21.15.116:0/1200175376 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a80ab5f0 con 0x7f24b014fc90 2024-04-25T00:03:36.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.319+0000 7f24ad59a640 1 -- 172.21.15.116:0/1200175376 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24a80b0070 con 0x7f24b014fc90 2024-04-25T00:03:36.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.319+0000 7f24ad59a640 1 -- 172.21.15.116:0/1200175376 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24a80ab9b0 con 0x7f24b014fc90 2024-04-25T00:03:36.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.319+0000 7f24b4dc2640 1 -- 172.21.15.116:0/1200175376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24b014fc90 msgr2=0x7f24b0073120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:36.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.319+0000 7f24b4dc2640 1 --2- 172.21.15.116:0/1200175376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24b014fc90 0x7f24b0073120 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f24b0071b80 tx=0x7f24a801c520 comp rx=0 tx=0).stop 2024-04-25T00:03:36.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.319+0000 7f24b4dc2640 1 -- 172.21.15.116:0/1200175376 shutdown_connections 2024-04-25T00:03:36.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.319+0000 7f24b4dc2640 1 --2- 172.21.15.116:0/1200175376 >> v2:172.21.15.174:3300/0 conn(0x7f24b0071520 0x7f24b0155ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:36.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.319+0000 7f24b4dc2640 1 --2- 172.21.15.116:0/1200175376 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f24b006c7b0 0x7f24b0155ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:36.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.319+0000 7f24b4dc2640 1 --2- 172.21.15.116:0/1200175376 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24b014fc90 0x7f24b0073120 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:36.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.319+0000 7f24b4dc2640 1 -- 172.21.15.116:0/1200175376 >> 172.21.15.116:0/1200175376 conn(0x7f24b00c4200 msgr2=0x7f24b00c4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:03:36.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.319+0000 7f24b4dc2640 1 -- 172.21.15.116:0/1200175376 shutdown_connections 2024-04-25T00:03:36.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.319+0000 7f24b4dc2640 1 -- 172.21.15.116:0/1200175376 wait complete. 2024-04-25T00:03:36.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.319+0000 7f24b4dc2640 1 Processor -- start 2024-04-25T00:03:36.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.319+0000 7f24b4dc2640 1 -- start start 2024-04-25T00:03:36.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.319+0000 7f24b4dc2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f24b006c7b0 0x7f24b014e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:36.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.319+0000 7f24b4dc2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24b0071520 0x7f24b014d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:36.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.319+0000 7f24b4dc2640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f24b014fc90 0x7f24b01515c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:36.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.319+0000 7f24b4dc2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f24b00c3d30 con 0x7f24b0071520 2024-04-25T00:03:36.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.319+0000 7f24b4dc2640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f24b00c3bb0 con 0x7f24b006c7b0 2024-04-25T00:03:36.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.319+0000 7f24b4dc2640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f24b0073660 con 0x7f24b014fc90 2024-04-25T00:03:36.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.319+0000 7f24add9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24b0071520 0x7f24b014d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:36.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.319+0000 7f24add9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24b0071520 0x7f24b014d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57850/0 (socket says 172.21.15.116:57850) 2024-04-25T00:03:36.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.319+0000 7f24add9b640 1 -- 172.21.15.116:0/727879530 learned_addr learned my addr 172.21.15.116:0/727879530 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:03:36.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.319+0000 7f24aed9d640 1 --2- 172.21.15.116:0/727879530 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f24b014fc90 0x7f24b01515c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:36.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.319+0000 7f24ae59c640 1 --2- 172.21.15.116:0/727879530 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f24b006c7b0 0x7f24b014e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:36.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.319+0000 7f24add9b640 1 -- 172.21.15.116:0/727879530 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f24b014fc90 msgr2=0x7f24b01515c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:36.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.319+0000 7f24add9b640 1 --2- 172.21.15.116:0/727879530 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f24b014fc90 0x7f24b01515c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:36.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.319+0000 7f24add9b640 1 -- 172.21.15.116:0/727879530 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f24b006c7b0 msgr2=0x7f24b014e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:36.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.319+0000 7f24add9b640 1 --2- 172.21.15.116:0/727879530 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f24b006c7b0 0x7f24b014e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:36.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.319+0000 7f24add9b640 1 -- 172.21.15.116:0/727879530 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24b0150560 con 0x7f24b0071520 2024-04-25T00:03:36.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.319+0000 7f24add9b640 1 --2- 172.21.15.116:0/727879530 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24b0071520 0x7f24b014d280 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f249800e8a0 tx=0x7f249800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:36.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.319+0000 7f24a6ffd640 1 -- 172.21.15.116:0/727879530 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2498017070 con 0x7f24b0071520 2024-04-25T00:03:36.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.319+0000 7f24a6ffd640 1 -- 172.21.15.116:0/727879530 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f249800ee90 con 0x7f24b0071520 2024-04-25T00:03:36.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.319+0000 7f24b4dc2640 1 -- 172.21.15.116:0/727879530 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24b014e830 con 0x7f24b0071520 2024-04-25T00:03:36.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.319+0000 7f24a6ffd640 1 -- 172.21.15.116:0/727879530 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f249800cb60 con 0x7f24b0071520 2024-04-25T00:03:36.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.319+0000 7f24b4dc2640 1 -- 172.21.15.116:0/727879530 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24b0150dc0 con 0x7f24b0071520 2024-04-25T00:03:36.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.323+0000 7f24b4dc2640 1 -- 172.21.15.116:0/727879530 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24b0068a60 con 0x7f24b0071520 2024-04-25T00:03:36.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.323+0000 7f24a6ffd640 1 -- 172.21.15.116:0/727879530 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f249801a070 con 0x7f24b0071520 2024-04-25T00:03:36.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.323+0000 7f24a6ffd640 1 --2- 172.21.15.116:0/727879530 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f24900678c0 0x7f2490069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:36.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.323+0000 7f24a6ffd640 1 -- 172.21.15.116:0/727879530 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 6838+0+0 (secure 0 0 0) 0x7f2498085ea0 con 0x7f24b0071520 2024-04-25T00:03:36.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.323+0000 7f24ae59c640 1 --2- 172.21.15.116:0/727879530 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f24900678c0 0x7f2490069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:36.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.323+0000 7f24ae59c640 1 --2- 172.21.15.116:0/727879530 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f24900678c0 0x7f2490069d80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f24a80af000 tx=0x7f24a8004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:36.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.327+0000 7f24a6ffd640 1 -- 172.21.15.116:0/727879530 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f2498055fb0 con 0x7f24b0071520 2024-04-25T00:03:36.344 INFO:tasks.workunit.client.1.smithi174.stdout:checking for sys/time.h... yes 2024-04-25T00:03:36.402 INFO:tasks.workunit.client.1.smithi174.stdout:checking whether it is safe to define __EXTENSIONS__... yes 2024-04-25T00:03:36.446 INFO:tasks.workunit.client.1.smithi174.stdout:checking whether _XOPEN_SOURCE should be defined... no 2024-04-25T00:03:36.472 INFO:tasks.workunit.client.1.smithi174.stdout:checking for gcc... (cached) gcc 2024-04-25T00:03:36.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.543+0000 7f24b4dc2640 1 -- 172.21.15.116:0/727879530 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f24b0071be0 con 0x7f24b0071520 2024-04-25T00:03:36.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.543+0000 7f24a6ffd640 1 -- 172.21.15.116:0/727879530 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+5670 (secure 0 0 0) 0x7f249805a2b0 con 0x7f24b0071520 2024-04-25T00:03:36.548 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:03:36.548 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:01:08.748957+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.174:6837/1762356055","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1762356055},{"type":"v1","addr":"172.21.15.174:6837","nonce":1762356055}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4270,"qdb_cluster":[4270]},"id":1},{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:03:34.579825+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4735},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4735":{"gid":4735,"name":"a","rank":0,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.116:6835/2004876235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":2004876235},{"type":"v1","addr":"172.21.15.116:6835","nonce":2004876235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4735,"qdb_cluster":[4735]},"id":2}]} 2024-04-25T00:03:36.548 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 24 2024-04-25T00:03:36.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.543+0000 7f24b4dc2640 1 -- 172.21.15.116:0/727879530 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f24900678c0 msgr2=0x7f2490069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:36.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.543+0000 7f24b4dc2640 1 --2- 172.21.15.116:0/727879530 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f24900678c0 0x7f2490069d80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f24a80af000 tx=0x7f24a8004740 comp rx=0 tx=0).stop 2024-04-25T00:03:36.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.543+0000 7f24b4dc2640 1 -- 172.21.15.116:0/727879530 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24b0071520 msgr2=0x7f24b014d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:36.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.543+0000 7f24b4dc2640 1 --2- 172.21.15.116:0/727879530 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24b0071520 0x7f24b014d280 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f249800e8a0 tx=0x7f249800ed60 comp rx=0 tx=0).stop 2024-04-25T00:03:36.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.543+0000 7f24b4dc2640 1 -- 172.21.15.116:0/727879530 shutdown_connections 2024-04-25T00:03:36.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.543+0000 7f24b4dc2640 1 --2- 172.21.15.116:0/727879530 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f24900678c0 0x7f2490069d80 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:36.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.543+0000 7f24b4dc2640 1 --2- 172.21.15.116:0/727879530 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f24b014fc90 0x7f24b01515c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:36.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.543+0000 7f24b4dc2640 1 --2- 172.21.15.116:0/727879530 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24b0071520 0x7f24b014d280 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:36.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.543+0000 7f24b4dc2640 1 --2- 172.21.15.116:0/727879530 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f24b006c7b0 0x7f24b014e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:36.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.543+0000 7f24b4dc2640 1 -- 172.21.15.116:0/727879530 >> 172.21.15.116:0/727879530 conn(0x7f24b00c4200 msgr2=0x7f24b00c2bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:03:36.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.543+0000 7f24b4dc2640 1 -- 172.21.15.116:0/727879530 shutdown_connections 2024-04-25T00:03:36.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:36.543+0000 7f24b4dc2640 1 -- 172.21.15.116:0/727879530 wait complete. 2024-04-25T00:03:36.556 INFO:tasks.workunit.client.1.smithi174.stdout:checking whether the compiler supports GNU C... (cached) yes 2024-04-25T00:03:36.556 INFO:tasks.workunit.client.1.smithi174.stdout:checking whether gcc accepts -g... (cached) yes 2024-04-25T00:03:36.557 INFO:tasks.workunit.client.1.smithi174.stdout:checking for gcc option to enable C11 features... (cached) none needed 2024-04-25T00:03:36.558 INFO:tasks.workunit.client.1.smithi174.stdout:checking dependency style of gcc... (cached) gcc3 2024-04-25T00:03:36.566 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T00:03:36.634 INFO:tasks.workunit.client.1.smithi174.stdout:checking for main in -lm... yes 2024-04-25T00:03:36.722 INFO:tasks.workunit.client.1.smithi174.stdout:checking for main in -lpthread... yes 2024-04-25T00:03:36.726 INFO:tasks.workunit.client.1.smithi174.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-04-25T00:03:36.727 INFO:tasks.workunit.client.1.smithi174.stdout:checking for egrep... /usr/bin/grep -E 2024-04-25T00:03:36.811 INFO:tasks.workunit.client.1.smithi174.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2024-04-25T00:03:36.906 INFO:tasks.workunit.client.1.smithi174.stdout:checking for pthread.h... yes 2024-04-25T00:03:36.979 INFO:tasks.workunit.client.1.smithi174.stdout:checking for fcntl.h... yes 2024-04-25T00:03:37.159 INFO:tasks.workunit.client.1.smithi174.stdout:checking for limits.h... yes 2024-04-25T00:03:37.162 INFO:tasks.workunit.client.1.smithi174.stdout:checking for stdint.h... (cached) yes 2024-04-25T00:03:37.164 INFO:tasks.workunit.client.1.smithi174.stdout:checking for sys/time.h... (cached) yes 2024-04-25T00:03:37.166 INFO:tasks.workunit.client.1.smithi174.stdout:checking for unistd.h... (cached) yes 2024-04-25T00:03:37.251 INFO:tasks.workunit.client.1.smithi174.stdout:checking for sys/vfs.h... yes 2024-04-25T00:03:37.379 INFO:tasks.workunit.client.1.smithi174.stdout:checking for sys/limits.h... no 2024-04-25T00:03:37.452 INFO:tasks.workunit.client.1.smithi174.stdout:checking for an ANSI C-conforming const... yes 2024-04-25T00:03:37.692 INFO:tasks.workunit.client.1.smithi174.stdout:checking for system... yes 2024-04-25T00:03:37.911 INFO:tasks.workunit.client.1.smithi174.stdout:checking for gettimeofday... yes 2024-04-25T00:03:38.159 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:03:38.172 INFO:tasks.workunit.client.1.smithi174.stdout:checking for mkdir... yes 2024-04-25T00:03:38.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.283+0000 7fa83dcd2640 1 Processor -- start 2024-04-25T00:03:38.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.287+0000 7fa83dcd2640 1 -- start start 2024-04-25T00:03:38.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.287+0000 7fa83dcd2640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fa838150d20 0x7fa8380c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:38.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.287+0000 7fa83dcd2640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa8380c2f50 0x7fa8381538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:38.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.287+0000 7fa83dcd2640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa8380a94a0 0x7fa838153e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:38.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.287+0000 7fa83dcd2640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fa8380c3910 con 0x7fa8380a94a0 2024-04-25T00:03:38.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.287+0000 7fa83dcd2640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fa8380c16e0 con 0x7fa838150d20 2024-04-25T00:03:38.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.287+0000 7fa83dcd2640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa8380c1400 con 0x7fa8380c2f50 2024-04-25T00:03:38.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.287+0000 7fa837fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa8380a94a0 0x7fa838153e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:38.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.287+0000 7fa837fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa8380a94a0 0x7fa838153e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57858/0 (socket says 172.21.15.116:57858) 2024-04-25T00:03:38.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.287+0000 7fa837fff640 1 -- 172.21.15.116:0/1497932291 learned_addr learned my addr 172.21.15.116:0/1497932291 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:03:38.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.287+0000 7fa837fff640 1 -- 172.21.15.116:0/1497932291 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa8380c2f50 msgr2=0x7fa8381538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:03:38.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.287+0000 7fa836ffd640 1 --2- 172.21.15.116:0/1497932291 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa8380c2f50 0x7fa8381538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:38.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.287+0000 7fa837fff640 1 --2- 172.21.15.116:0/1497932291 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa8380c2f50 0x7fa8381538e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:38.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.287+0000 7fa837fff640 1 -- 172.21.15.116:0/1497932291 >> v2:172.21.15.174:3300/0 conn(0x7fa838150d20 msgr2=0x7fa8380c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:03:38.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.287+0000 7fa8377fe640 1 --2- 172.21.15.116:0/1497932291 >> v2:172.21.15.174:3300/0 conn(0x7fa838150d20 0x7fa8380c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:38.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.287+0000 7fa837fff640 1 --2- 172.21.15.116:0/1497932291 >> v2:172.21.15.174:3300/0 conn(0x7fa838150d20 0x7fa8380c2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:38.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.287+0000 7fa837fff640 1 -- 172.21.15.116:0/1497932291 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa83806e660 con 0x7fa8380a94a0 2024-04-25T00:03:38.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.287+0000 7fa8377fe640 1 --2- 172.21.15.116:0/1497932291 >> v2:172.21.15.174:3300/0 conn(0x7fa838150d20 0x7fa8380c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:03:38.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.291+0000 7fa837fff640 1 --2- 172.21.15.116:0/1497932291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8380a94a0 0x7fa838153e20 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fa838073550 tx=0x7fa83001c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=721e3d94f642c95f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:38.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.291+0000 7fa8367fc640 1 -- 172.21.15.116:0/1497932291 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8300a5740 con 0x7fa8380a94a0 2024-04-25T00:03:38.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.291+0000 7fa8367fc640 1 -- 172.21.15.116:0/1497932291 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8300a58e0 con 0x7fa8380a94a0 2024-04-25T00:03:38.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.291+0000 7fa8367fc640 1 -- 172.21.15.116:0/1497932291 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8300a5cb0 con 0x7fa8380a94a0 2024-04-25T00:03:38.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.291+0000 7fa83dcd2640 1 -- 172.21.15.116:0/1497932291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8380a94a0 msgr2=0x7fa838153e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:38.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.291+0000 7fa83dcd2640 1 --2- 172.21.15.116:0/1497932291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8380a94a0 0x7fa838153e20 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fa838073550 tx=0x7fa83001c9f0 comp rx=0 tx=0).stop 2024-04-25T00:03:38.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.291+0000 7fa83dcd2640 1 -- 172.21.15.116:0/1497932291 shutdown_connections 2024-04-25T00:03:38.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.291+0000 7fa83dcd2640 1 --2- 172.21.15.116:0/1497932291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8380a94a0 0x7fa838153e20 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:38.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.291+0000 7fa83dcd2640 1 --2- 172.21.15.116:0/1497932291 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa8380c2f50 0x7fa8381538e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:38.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.291+0000 7fa83dcd2640 1 --2- 172.21.15.116:0/1497932291 >> v2:172.21.15.174:3300/0 conn(0x7fa838150d20 0x7fa8380c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:38.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.291+0000 7fa83dcd2640 1 -- 172.21.15.116:0/1497932291 >> 172.21.15.116:0/1497932291 conn(0x7fa8380c3b00 msgr2=0x7fa8380c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:03:38.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.291+0000 7fa83dcd2640 1 -- 172.21.15.116:0/1497932291 shutdown_connections 2024-04-25T00:03:38.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.291+0000 7fa83dcd2640 1 -- 172.21.15.116:0/1497932291 wait complete. 2024-04-25T00:03:38.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.291+0000 7fa83dcd2640 1 Processor -- start 2024-04-25T00:03:38.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.291+0000 7fa83dcd2640 1 -- start start 2024-04-25T00:03:38.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.291+0000 7fa83dcd2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8380a94a0 0x7fa8380c5c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:38.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.291+0000 7fa83dcd2640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa8380c2f50 0x7fa83814ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:38.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.291+0000 7fa83dcd2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa838150d20 0x7fa83814f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:38.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.291+0000 7fa83dcd2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa838157340 con 0x7fa8380a94a0 2024-04-25T00:03:38.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.291+0000 7fa83dcd2640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa838154570 con 0x7fa838150d20 2024-04-25T00:03:38.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.291+0000 7fa83dcd2640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa8381574c0 con 0x7fa8380c2f50 2024-04-25T00:03:38.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.295+0000 7fa8377fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8380a94a0 0x7fa8380c5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:38.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.295+0000 7fa8377fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8380a94a0 0x7fa8380c5c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57870/0 (socket says 172.21.15.116:57870) 2024-04-25T00:03:38.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.295+0000 7fa8377fe640 1 -- 172.21.15.116:0/2531623771 learned_addr learned my addr 172.21.15.116:0/2531623771 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:03:38.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.295+0000 7fa836ffd640 1 --2- 172.21.15.116:0/2531623771 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa8380c2f50 0x7fa83814ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:38.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.295+0000 7fa837fff640 1 --2- 172.21.15.116:0/2531623771 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa838150d20 0x7fa83814f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:38.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.295+0000 7fa8377fe640 1 -- 172.21.15.116:0/2531623771 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa8380c2f50 msgr2=0x7fa83814ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:38.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.295+0000 7fa8377fe640 1 --2- 172.21.15.116:0/2531623771 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa8380c2f50 0x7fa83814ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:38.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.295+0000 7fa8377fe640 1 -- 172.21.15.116:0/2531623771 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa838150d20 msgr2=0x7fa83814f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:38.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.295+0000 7fa8377fe640 1 --2- 172.21.15.116:0/2531623771 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa838150d20 0x7fa83814f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:38.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.295+0000 7fa8377fe640 1 -- 172.21.15.116:0/2531623771 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa83813f700 con 0x7fa8380a94a0 2024-04-25T00:03:38.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.295+0000 7fa8377fe640 1 --2- 172.21.15.116:0/2531623771 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8380a94a0 0x7fa8380c5c10 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fa82800c7f0 tx=0x7fa82800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:38.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.295+0000 7fa81ffff640 1 -- 172.21.15.116:0/2531623771 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa828015070 con 0x7fa8380a94a0 2024-04-25T00:03:38.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.295+0000 7fa81ffff640 1 -- 172.21.15.116:0/2531623771 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa828007820 con 0x7fa8380a94a0 2024-04-25T00:03:38.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.295+0000 7fa83dcd2640 1 -- 172.21.15.116:0/2531623771 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa83813f9e0 con 0x7fa8380a94a0 2024-04-25T00:03:38.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.295+0000 7fa81ffff640 1 -- 172.21.15.116:0/2531623771 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa828007c10 con 0x7fa8380a94a0 2024-04-25T00:03:38.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.295+0000 7fa83dcd2640 1 -- 172.21.15.116:0/2531623771 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8380c6c40 con 0x7fa8380a94a0 2024-04-25T00:03:38.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.295+0000 7fa83dcd2640 1 -- 172.21.15.116:0/2531623771 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa838068350 con 0x7fa8380a94a0 2024-04-25T00:03:38.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.295+0000 7fa81ffff640 1 -- 172.21.15.116:0/2531623771 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa828018070 con 0x7fa8380a94a0 2024-04-25T00:03:38.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.295+0000 7fa81ffff640 1 --2- 172.21.15.116:0/2531623771 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa8240678c0 0x7fa824069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:38.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.295+0000 7fa81ffff640 1 -- 172.21.15.116:0/2531623771 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 6924+0+0 (secure 0 0 0) 0x7fa8280862f0 con 0x7fa8380a94a0 2024-04-25T00:03:38.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.295+0000 7fa836ffd640 1 --2- 172.21.15.116:0/2531623771 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa8240678c0 0x7fa824069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:38.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.295+0000 7fa836ffd640 1 --2- 172.21.15.116:0/2531623771 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa8240678c0 0x7fa824069d80 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa82c006fd0 tx=0x7fa82c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:38.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.299+0000 7fa81ffff640 1 -- 172.21.15.116:0/2531623771 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fa828056970 con 0x7fa8380a94a0 2024-04-25T00:03:38.435 INFO:tasks.ceph.mds.a.smithi116.stderr:2024-04-25T00:03:38.431+0000 7f4951dbb640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T00:03:38.464 INFO:tasks.workunit.client.1.smithi174.stdout:checking for strerror... yes 2024-04-25T00:03:38.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.515+0000 7fa83dcd2640 1 -- 172.21.15.116:0/2531623771 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa83813fbb0 con 0x7fa8380a94a0 2024-04-25T00:03:38.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.515+0000 7fa81ffff640 1 -- 172.21.15.116:0/2531623771 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+5669 (secure 0 0 0) 0x7fa828056270 con 0x7fa8380a94a0 2024-04-25T00:03:38.522 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:03:38.522 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:03:36.853039+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4828":{"gid":4828,"name":"c","rank":0,"incarnation":26,"state":"up:reconnect","state_seq":2,"addr":"172.21.15.116:6834/1649769369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1649769369},{"type":"v1","addr":"172.21.15.116:6834","nonce":1649769369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:03:34.579825+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4735},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4735":{"gid":4735,"name":"a","rank":0,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.116:6835/2004876235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":2004876235},{"type":"v1","addr":"172.21.15.116:6835","nonce":2004876235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4735,"qdb_cluster":[4735]},"id":2}]} 2024-04-25T00:03:38.522 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 27 2024-04-25T00:03:38.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.519+0000 7fa83dcd2640 1 -- 172.21.15.116:0/2531623771 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa8240678c0 msgr2=0x7fa824069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:38.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.519+0000 7fa83dcd2640 1 --2- 172.21.15.116:0/2531623771 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa8240678c0 0x7fa824069d80 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa82c006fd0 tx=0x7fa82c008040 comp rx=0 tx=0).stop 2024-04-25T00:03:38.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.519+0000 7fa83dcd2640 1 -- 172.21.15.116:0/2531623771 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8380a94a0 msgr2=0x7fa8380c5c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:38.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.519+0000 7fa83dcd2640 1 --2- 172.21.15.116:0/2531623771 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8380a94a0 0x7fa8380c5c10 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fa82800c7f0 tx=0x7fa82800ccb0 comp rx=0 tx=0).stop 2024-04-25T00:03:38.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.519+0000 7fa83dcd2640 1 -- 172.21.15.116:0/2531623771 shutdown_connections 2024-04-25T00:03:38.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.519+0000 7fa83dcd2640 1 --2- 172.21.15.116:0/2531623771 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa8240678c0 0x7fa824069d80 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:38.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.519+0000 7fa83dcd2640 1 --2- 172.21.15.116:0/2531623771 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa838150d20 0x7fa83814f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:38.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.519+0000 7fa83dcd2640 1 --2- 172.21.15.116:0/2531623771 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa8380c2f50 0x7fa83814ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:38.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.519+0000 7fa83dcd2640 1 --2- 172.21.15.116:0/2531623771 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8380a94a0 0x7fa8380c5c10 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:38.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.519+0000 7fa83dcd2640 1 -- 172.21.15.116:0/2531623771 >> 172.21.15.116:0/2531623771 conn(0x7fa8380c3b00 msgr2=0x7fa8380a9f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:03:38.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.519+0000 7fa83dcd2640 1 -- 172.21.15.116:0/2531623771 shutdown_connections 2024-04-25T00:03:38.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.519+0000 7fa83dcd2640 1 -- 172.21.15.116:0/2531623771 wait complete. 2024-04-25T00:03:38.541 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2024-04-25T00:03:38.567 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:03:38.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.683+0000 7fb782033640 1 Processor -- start 2024-04-25T00:03:38.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.683+0000 7fb782033640 1 -- start start 2024-04-25T00:03:38.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.687+0000 7fb782033640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fb77c14c700 0x7fb77c0c5090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:38.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.687+0000 7fb782033640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb77c155b20 0x7fb77c152c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:38.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.687+0000 7fb782033640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb77c0a9e60 con 0x7fb77c0c55d0 2024-04-25T00:03:38.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.687+0000 7fb782033640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fb77c1527b0 con 0x7fb77c14c700 2024-04-25T00:03:38.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.687+0000 7fb782033640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fb77c0c3ef0 con 0x7fb77c155b20 2024-04-25T00:03:38.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.687+0000 7fb77bfff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fb77c0c55d0 0x7fb77c071df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:53060/0 (socket says 172.21.15.116:53060) 2024-04-25T00:03:38.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.687+0000 7fb77bfff640 1 -- 172.21.15.116:0/290094559 learned_addr learned my addr 172.21.15.116:0/290094559 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:03:38.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.687+0000 7fb77b7fe640 1 --2- 172.21.15.116:0/290094559 >> v2:172.21.15.174:3300/0 conn(0x7fb77c14c700 0x7fb77c0c5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:38.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.687+0000 7fb77affd640 1 --2- 172.21.15.116:0/290094559 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb77c155b20 0x7fb77c152c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:38.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.687+0000 7fb77a7fc640 1 -- 172.21.15.116:0/290094559 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3155736428 0 0) 0x7fb77c0a9e60 con 0x7fb77c0c55d0 2024-04-25T00:03:38.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.687+0000 7fb77a7fc640 1 -- 172.21.15.116:0/290094559 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb764003530 con 0x7fb77c0c55d0 2024-04-25T00:03:38.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.687+0000 7fb77b7fe640 1 -- 172.21.15.116:0/290094559 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb77c155b20 msgr2=0x7fb77c152c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:03:38.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.687+0000 7fb77b7fe640 1 --2- 172.21.15.116:0/290094559 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb77c155b20 0x7fb77c152c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:38.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.687+0000 7fb77b7fe640 1 -- 172.21.15.116:0/290094559 >> v1:172.21.15.116:6789/0 conn(0x7fb77c0c55d0 legacy=0x7fb77c071df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:38.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.687+0000 7fb77b7fe640 1 -- 172.21.15.116:0/290094559 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb77c0c41d0 con 0x7fb77c14c700 2024-04-25T00:03:38.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.687+0000 7fb77affd640 1 --2- 172.21.15.116:0/290094559 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb77c155b20 0x7fb77c152c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:03:38.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.687+0000 7fb77b7fe640 1 --2- 172.21.15.116:0/290094559 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb77c14c700 0x7fb77c0c5090 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb77c152930 tx=0x7fb76c01c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b7b599670800d667 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:38.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.687+0000 7fb77a7fc640 1 -- 172.21.15.116:0/290094559 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb76c0b0070 con 0x7fb77c14c700 2024-04-25T00:03:38.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.691+0000 7fb77a7fc640 1 -- 172.21.15.116:0/290094559 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb76c0ab600 con 0x7fb77c14c700 2024-04-25T00:03:38.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.691+0000 7fb77a7fc640 1 -- 172.21.15.116:0/290094559 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb76c0aba10 con 0x7fb77c14c700 2024-04-25T00:03:38.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.691+0000 7fb782033640 1 -- 172.21.15.116:0/290094559 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb77c14c700 msgr2=0x7fb77c0c5090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:38.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.691+0000 7fb782033640 1 --2- 172.21.15.116:0/290094559 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb77c14c700 0x7fb77c0c5090 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb77c152930 tx=0x7fb76c01c3a0 comp rx=0 tx=0).stop 2024-04-25T00:03:38.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.691+0000 7fb782033640 1 -- 172.21.15.116:0/290094559 shutdown_connections 2024-04-25T00:03:38.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.691+0000 7fb782033640 1 --2- 172.21.15.116:0/290094559 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb77c155b20 0x7fb77c152c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:38.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.691+0000 7fb782033640 1 --2- 172.21.15.116:0/290094559 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb77c14c700 0x7fb77c0c5090 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:38.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.691+0000 7fb782033640 1 -- 172.21.15.116:0/290094559 >> 172.21.15.116:0/290094559 conn(0x7fb77c06c720 msgr2=0x7fb77c06cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:03:38.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.691+0000 7fb782033640 1 -- 172.21.15.116:0/290094559 shutdown_connections 2024-04-25T00:03:38.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.691+0000 7fb782033640 1 -- 172.21.15.116:0/290094559 wait complete. 2024-04-25T00:03:38.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.691+0000 7fb782033640 1 Processor -- start 2024-04-25T00:03:38.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.691+0000 7fb782033640 1 -- start start 2024-04-25T00:03:38.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.691+0000 7fb782033640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb77c0c55d0 0x7fb77c0acd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:38.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.691+0000 7fb782033640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb77c14c700 0x7fb77c0c14d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:38.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.691+0000 7fb782033640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb77c155b20 0x7fb77c1f1f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:38.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.691+0000 7fb782033640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb77c153510 con 0x7fb77c0c55d0 2024-04-25T00:03:38.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.691+0000 7fb782033640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb77c153390 con 0x7fb77c155b20 2024-04-25T00:03:38.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.691+0000 7fb782033640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fb77c15fed0 con 0x7fb77c14c700 2024-04-25T00:03:38.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.691+0000 7fb77b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb77c0c55d0 0x7fb77c0acd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:38.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.691+0000 7fb77b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb77c0c55d0 0x7fb77c0acd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57878/0 (socket says 172.21.15.116:57878) 2024-04-25T00:03:38.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.691+0000 7fb77b7fe640 1 -- 172.21.15.116:0/448846494 learned_addr learned my addr 172.21.15.116:0/448846494 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:03:38.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.691+0000 7fb77affd640 1 --2- 172.21.15.116:0/448846494 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb77c14c700 0x7fb77c0c14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:38.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.691+0000 7fb77bfff640 1 --2- 172.21.15.116:0/448846494 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb77c155b20 0x7fb77c1f1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:38.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.691+0000 7fb77b7fe640 1 -- 172.21.15.116:0/448846494 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb77c14c700 msgr2=0x7fb77c0c14d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:38.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.691+0000 7fb77b7fe640 1 --2- 172.21.15.116:0/448846494 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb77c14c700 0x7fb77c0c14d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:38.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.691+0000 7fb77b7fe640 1 -- 172.21.15.116:0/448846494 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb77c155b20 msgr2=0x7fb77c1f1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:38.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.691+0000 7fb77b7fe640 1 --2- 172.21.15.116:0/448846494 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb77c155b20 0x7fb77c1f1f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:38.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.691+0000 7fb77b7fe640 1 -- 172.21.15.116:0/448846494 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb77c0ad430 con 0x7fb77c0c55d0 2024-04-25T00:03:38.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.691+0000 7fb77bfff640 1 --2- 172.21.15.116:0/448846494 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb77c155b20 0x7fb77c1f1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:03:38.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.691+0000 7fb77b7fe640 1 --2- 172.21.15.116:0/448846494 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb77c0c55d0 0x7fb77c0acd50 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fb76c0ae040 tx=0x7fb76c01abe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:38.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.691+0000 7fb763fff640 1 -- 172.21.15.116:0/448846494 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb76c0b0050 con 0x7fb77c0c55d0 2024-04-25T00:03:38.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.691+0000 7fb763fff640 1 -- 172.21.15.116:0/448846494 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb76c0bc070 con 0x7fb77c0c55d0 2024-04-25T00:03:38.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.691+0000 7fb782033640 1 -- 172.21.15.116:0/448846494 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb77c0ab180 con 0x7fb77c0c55d0 2024-04-25T00:03:38.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.691+0000 7fb763fff640 1 -- 172.21.15.116:0/448846494 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb76c0b5a30 con 0x7fb77c0c55d0 2024-04-25T00:03:38.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.691+0000 7fb782033640 1 -- 172.21.15.116:0/448846494 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb77c1ef5c0 con 0x7fb77c0c55d0 2024-04-25T00:03:38.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.695+0000 7fb782033640 1 -- 172.21.15.116:0/448846494 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb77c068a60 con 0x7fb77c0c55d0 2024-04-25T00:03:38.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.695+0000 7fb763fff640 1 -- 172.21.15.116:0/448846494 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb76c0ab4c0 con 0x7fb77c0c55d0 2024-04-25T00:03:38.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.695+0000 7fb763fff640 1 --2- 172.21.15.116:0/448846494 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb7680678f0 0x7fb768069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:38.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.695+0000 7fb763fff640 1 -- 172.21.15.116:0/448846494 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 6924+0+0 (secure 0 0 0) 0x7fb76c121440 con 0x7fb77c0c55d0 2024-04-25T00:03:38.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.695+0000 7fb77affd640 1 --2- 172.21.15.116:0/448846494 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb7680678f0 0x7fb768069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:38.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.695+0000 7fb77affd640 1 --2- 172.21.15.116:0/448846494 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb7680678f0 0x7fb768069db0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb770009580 tx=0x7fb770009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:38.701 INFO:tasks.workunit.client.1.smithi174.stdout:checking for open64... yes 2024-04-25T00:03:38.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.699+0000 7fb763fff640 1 -- 172.21.15.116:0/448846494 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fb76c0160b0 con 0x7fb77c0c55d0 2024-04-25T00:03:38.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.895+0000 7fb782033640 1 -- 172.21.15.116:0/448846494 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb77c0759d0 con 0x7fb77c0c55d0 2024-04-25T00:03:38.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.895+0000 7fb763fff640 1 -- 172.21.15.116:0/448846494 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+5666 (secure 0 0 0) 0x7fb76c0f1b20 con 0x7fb77c0c55d0 2024-04-25T00:03:38.902 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:03:38.902 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:03:37.619154+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4828":{"gid":4828,"name":"c","rank":0,"incarnation":26,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.116:6834/1649769369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1649769369},{"type":"v1","addr":"172.21.15.116:6834","nonce":1649769369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:03:34.579825+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4735},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4735":{"gid":4735,"name":"a","rank":0,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.116:6835/2004876235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":2004876235},{"type":"v1","addr":"172.21.15.116:6835","nonce":2004876235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4735,"qdb_cluster":[4735]},"id":2}]} 2024-04-25T00:03:38.902 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 28 2024-04-25T00:03:38.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.899+0000 7fb782033640 1 -- 172.21.15.116:0/448846494 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb7680678f0 msgr2=0x7fb768069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:38.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.899+0000 7fb782033640 1 --2- 172.21.15.116:0/448846494 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb7680678f0 0x7fb768069db0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb770009580 tx=0x7fb770009190 comp rx=0 tx=0).stop 2024-04-25T00:03:38.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.899+0000 7fb782033640 1 -- 172.21.15.116:0/448846494 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb77c0c55d0 msgr2=0x7fb77c0acd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:38.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.899+0000 7fb782033640 1 --2- 172.21.15.116:0/448846494 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb77c0c55d0 0x7fb77c0acd50 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fb76c0ae040 tx=0x7fb76c01abe0 comp rx=0 tx=0).stop 2024-04-25T00:03:38.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.899+0000 7fb782033640 1 -- 172.21.15.116:0/448846494 shutdown_connections 2024-04-25T00:03:38.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.899+0000 7fb782033640 1 --2- 172.21.15.116:0/448846494 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb7680678f0 0x7fb768069db0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:38.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.899+0000 7fb782033640 1 --2- 172.21.15.116:0/448846494 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb77c155b20 0x7fb77c1f1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:38.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.899+0000 7fb782033640 1 --2- 172.21.15.116:0/448846494 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb77c14c700 0x7fb77c0c14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:38.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.899+0000 7fb782033640 1 --2- 172.21.15.116:0/448846494 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb77c0c55d0 0x7fb77c0acd50 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:38.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.899+0000 7fb782033640 1 -- 172.21.15.116:0/448846494 >> 172.21.15.116:0/448846494 conn(0x7fb77c06c720 msgr2=0x7fb77c15d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:03:38.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.899+0000 7fb782033640 1 -- 172.21.15.116:0/448846494 shutdown_connections 2024-04-25T00:03:38.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:38.899+0000 7fb782033640 1 -- 172.21.15.116:0/448846494 wait complete. 2024-04-25T00:03:38.914 INFO:tasks.workunit.client.1.smithi174.stdout:checking for stat64... yes 2024-04-25T00:03:38.923 INFO:tasks.mds_thrash.fs.[a]:mds.b down, removed from mdsmap 2024-04-25T00:03:38.923 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-04-25T00:03:38.923 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:03:39.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.043+0000 7faea9ce9640 1 Processor -- start 2024-04-25T00:03:39.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.043+0000 7faea9ce9640 1 -- start start 2024-04-25T00:03:39.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.043+0000 7faea9ce9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7faea414c720 0x7faea41563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:39.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.043+0000 7faea9ce9640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faea4071680 0x7faea4071ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:39.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.043+0000 7faea9ce9640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faea4071220 con 0x7faea40a94a0 2024-04-25T00:03:39.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.043+0000 7faea9ce9640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7faea4153460 con 0x7faea414c720 2024-04-25T00:03:39.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.043+0000 7faea9ce9640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7faea4152420 con 0x7faea4071680 2024-04-25T00:03:39.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.043+0000 7faea37fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7faea414c720 0x7faea41563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:39.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.043+0000 7fae9affd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7faea40a94a0 0x7faea40c2a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:53070/0 (socket says 172.21.15.116:53070) 2024-04-25T00:03:39.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.043+0000 7fae9affd640 1 -- 172.21.15.116:0/166712742 learned_addr learned my addr 172.21.15.116:0/166712742 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:03:39.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.043+0000 7faea3fff640 1 --2- 172.21.15.116:0/166712742 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faea4071680 0x7faea4071ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:39.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.043+0000 7faea37fe640 1 -- 172.21.15.116:0/166712742 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faea4071680 msgr2=0x7faea4071ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:03:39.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.043+0000 7faea37fe640 1 --2- 172.21.15.116:0/166712742 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faea4071680 0x7faea4071ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:39.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.043+0000 7faea37fe640 1 -- 172.21.15.116:0/166712742 >> v1:172.21.15.116:6789/0 conn(0x7faea40a94a0 legacy=0x7faea40c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:39.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.043+0000 7faea37fe640 1 -- 172.21.15.116:0/166712742 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faea414cb00 con 0x7faea414c720 2024-04-25T00:03:39.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.043+0000 7faea2ffd640 1 -- 172.21.15.116:0/166712742 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1500656177 0 0) 0x7faea4071220 con 0x7faea40a94a0 2024-04-25T00:03:39.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.043+0000 7faea37fe640 1 --2- 172.21.15.116:0/166712742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faea414c720 0x7faea41563d0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fae940097c0 tx=0x7fae9401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca5bcd487898d084 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:39.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.043+0000 7faea2ffd640 1 -- 172.21.15.116:0/166712742 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae940b1070 con 0x7faea414c720 2024-04-25T00:03:39.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.047+0000 7faea2ffd640 1 -- 172.21.15.116:0/166712742 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae940a8050 con 0x7faea414c720 2024-04-25T00:03:39.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.047+0000 7faea2ffd640 1 -- 172.21.15.116:0/166712742 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae940ac860 con 0x7faea414c720 2024-04-25T00:03:39.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.047+0000 7faea9ce9640 1 -- 172.21.15.116:0/166712742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faea414c720 msgr2=0x7faea41563d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:39.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.047+0000 7faea9ce9640 1 --2- 172.21.15.116:0/166712742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faea414c720 0x7faea41563d0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fae940097c0 tx=0x7fae9401c3e0 comp rx=0 tx=0).stop 2024-04-25T00:03:39.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.047+0000 7faea9ce9640 1 -- 172.21.15.116:0/166712742 shutdown_connections 2024-04-25T00:03:39.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.047+0000 7faea9ce9640 1 --2- 172.21.15.116:0/166712742 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faea4071680 0x7faea4071ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:39.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.047+0000 7faea9ce9640 1 --2- 172.21.15.116:0/166712742 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faea414c720 0x7faea41563d0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:39.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.047+0000 7faea9ce9640 1 -- 172.21.15.116:0/166712742 >> 172.21.15.116:0/166712742 conn(0x7faea40c3b00 msgr2=0x7faea40c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:03:39.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.047+0000 7faea9ce9640 1 -- 172.21.15.116:0/166712742 shutdown_connections 2024-04-25T00:03:39.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.047+0000 7faea9ce9640 1 -- 172.21.15.116:0/166712742 wait complete. 2024-04-25T00:03:39.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.047+0000 7faea9ce9640 1 Processor -- start 2024-04-25T00:03:39.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.047+0000 7faea9ce9640 1 -- start start 2024-04-25T00:03:39.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.047+0000 7faea9ce9640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faea4071680 0x7faea41557c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:39.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.047+0000 7faea9ce9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faea40a94a0 0x7faea406e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:39.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.047+0000 7faea9ce9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faea414c720 0x7faea4147b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:39.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.047+0000 7faea9ce9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7faea40c3120 con 0x7faea414c720 2024-04-25T00:03:39.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.047+0000 7faea9ce9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7faea4152f40 con 0x7faea40a94a0 2024-04-25T00:03:39.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.047+0000 7faea9ce9640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7faea40c32a0 con 0x7faea4071680 2024-04-25T00:03:39.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.047+0000 7faea3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faea414c720 0x7faea4147b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:39.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.047+0000 7faea3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faea414c720 0x7faea4147b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57900/0 (socket says 172.21.15.116:57900) 2024-04-25T00:03:39.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.047+0000 7faea3fff640 1 -- 172.21.15.116:0/2861478047 learned_addr learned my addr 172.21.15.116:0/2861478047 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:03:39.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.047+0000 7faea37fe640 1 --2- 172.21.15.116:0/2861478047 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faea4071680 0x7faea41557c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:39.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.047+0000 7fae9affd640 1 --2- 172.21.15.116:0/2861478047 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faea40a94a0 0x7faea406e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:39.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.047+0000 7faea3fff640 1 -- 172.21.15.116:0/2861478047 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faea4071680 msgr2=0x7faea41557c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:39.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.047+0000 7faea3fff640 1 --2- 172.21.15.116:0/2861478047 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faea4071680 0x7faea41557c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:39.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.047+0000 7faea3fff640 1 -- 172.21.15.116:0/2861478047 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faea40a94a0 msgr2=0x7faea406e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:39.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.047+0000 7faea3fff640 1 --2- 172.21.15.116:0/2861478047 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faea40a94a0 0x7faea406e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:39.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.047+0000 7faea3fff640 1 -- 172.21.15.116:0/2861478047 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faea40733b0 con 0x7faea414c720 2024-04-25T00:03:39.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.047+0000 7faea3fff640 1 --2- 172.21.15.116:0/2861478047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faea414c720 0x7faea4147b20 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fae9c007e40 tx=0x7fae9c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:39.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.047+0000 7faea0ff9640 1 -- 172.21.15.116:0/2861478047 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae9c014620 con 0x7faea414c720 2024-04-25T00:03:39.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.047+0000 7faea0ff9640 1 -- 172.21.15.116:0/2861478047 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae9c019070 con 0x7faea414c720 2024-04-25T00:03:39.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.047+0000 7faea9ce9640 1 -- 172.21.15.116:0/2861478047 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faea41ea610 con 0x7faea414c720 2024-04-25T00:03:39.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.047+0000 7faea0ff9640 1 -- 172.21.15.116:0/2861478047 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae9c014a20 con 0x7faea414c720 2024-04-25T00:03:39.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.047+0000 7faea9ce9640 1 -- 172.21.15.116:0/2861478047 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faea4153bd0 con 0x7faea414c720 2024-04-25T00:03:39.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.051+0000 7faea9ce9640 1 -- 172.21.15.116:0/2861478047 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faea4068350 con 0x7faea414c720 2024-04-25T00:03:39.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.051+0000 7faea0ff9640 1 -- 172.21.15.116:0/2861478047 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fae9c014bc0 con 0x7faea414c720 2024-04-25T00:03:39.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.051+0000 7faea0ff9640 1 --2- 172.21.15.116:0/2861478047 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fae840678c0 0x7fae84069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:39.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.051+0000 7faea0ff9640 1 -- 172.21.15.116:0/2861478047 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 6924+0+0 (secure 0 0 0) 0x7fae9c086880 con 0x7faea414c720 2024-04-25T00:03:39.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.051+0000 7faea37fe640 1 --2- 172.21.15.116:0/2861478047 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fae840678c0 0x7fae84069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:39.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.051+0000 7faea37fe640 1 --2- 172.21.15.116:0/2861478047 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fae840678c0 0x7fae84069d80 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fae940ae000 tx=0x7fae940a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:39.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.055+0000 7faea0ff9640 1 -- 172.21.15.116:0/2861478047 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fae9c0568b0 con 0x7faea414c720 2024-04-25T00:03:39.085 INFO:tasks.workunit.client.1.smithi174.stdout:checking for fseeko64... yes 2024-04-25T00:03:39.251 INFO:tasks.workunit.client.1.smithi174.stdout:checking for lrand48_r... yes 2024-04-25T00:03:39.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.263+0000 7faea9ce9640 1 -- 172.21.15.116:0/2861478047 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faea40c3640 con 0x7faea414c720 2024-04-25T00:03:39.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.263+0000 7faea0ff9640 1 -- 172.21.15.116:0/2861478047 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+5666 (secure 0 0 0) 0x7fae9c05abb0 con 0x7faea414c720 2024-04-25T00:03:39.271 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:03:39.271 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:03:37.619154+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4828":{"gid":4828,"name":"c","rank":0,"incarnation":26,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.116:6834/1649769369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1649769369},{"type":"v1","addr":"172.21.15.116:6834","nonce":1649769369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:03:34.579825+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4735},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4735":{"gid":4735,"name":"a","rank":0,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.116:6835/2004876235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":2004876235},{"type":"v1","addr":"172.21.15.116:6835","nonce":2004876235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4735,"qdb_cluster":[4735]},"id":2}]} 2024-04-25T00:03:39.271 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 28 2024-04-25T00:03:39.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.267+0000 7faea9ce9640 1 -- 172.21.15.116:0/2861478047 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fae840678c0 msgr2=0x7fae84069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:39.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.267+0000 7faea9ce9640 1 --2- 172.21.15.116:0/2861478047 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fae840678c0 0x7fae84069d80 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fae940ae000 tx=0x7fae940a7000 comp rx=0 tx=0).stop 2024-04-25T00:03:39.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.267+0000 7faea9ce9640 1 -- 172.21.15.116:0/2861478047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faea414c720 msgr2=0x7faea4147b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:39.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.267+0000 7faea9ce9640 1 --2- 172.21.15.116:0/2861478047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faea414c720 0x7faea4147b20 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fae9c007e40 tx=0x7fae9c00e4b0 comp rx=0 tx=0).stop 2024-04-25T00:03:39.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.267+0000 7faea9ce9640 1 -- 172.21.15.116:0/2861478047 shutdown_connections 2024-04-25T00:03:39.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.267+0000 7faea9ce9640 1 --2- 172.21.15.116:0/2861478047 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fae840678c0 0x7fae84069d80 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:39.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.267+0000 7faea9ce9640 1 --2- 172.21.15.116:0/2861478047 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faea414c720 0x7faea4147b20 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:39.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.267+0000 7faea9ce9640 1 --2- 172.21.15.116:0/2861478047 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faea40a94a0 0x7faea406e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:39.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.267+0000 7faea9ce9640 1 --2- 172.21.15.116:0/2861478047 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faea4071680 0x7faea41557c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:39.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.267+0000 7faea9ce9640 1 -- 172.21.15.116:0/2861478047 >> 172.21.15.116:0/2861478047 conn(0x7faea40c3b00 msgr2=0x7faea4149df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:03:39.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.267+0000 7faea9ce9640 1 -- 172.21.15.116:0/2861478047 shutdown_connections 2024-04-25T00:03:39.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:39.267+0000 7faea9ce9640 1 -- 172.21.15.116:0/2861478047 wait complete. 2024-04-25T00:03:39.291 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 28, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:04.678737+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 28, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_4828": { "addr": "172.21.15.116:6834/1649769369", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6832", "nonce": 1649769369, "type": "v2" }, { "addr": "172.21.15.116:6834", "nonce": 1649769369, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4828, "incarnation": 26, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:rejoin", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 32, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T00:03:37.619154+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4828 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:18.825887+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 24, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_4735": { "addr": "172.21.15.116:6835/2004876235", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6833", "nonce": 2004876235, "type": "v2" }, { "addr": "172.21.15.116:6835", "nonce": 2004876235, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4735, "incarnation": 20, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 14 } }, "last_failure": 0, "last_failure_osd_epoch": 31, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-04-25T00:03:34.579825+0000", "qdb_cluster": [ 4735 ], "qdb_leader": 4735, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4735 } } } ], "standbys": [ { "addr": "172.21.15.174:6835/2883220846", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6834", "nonce": 2883220846, "type": "v2" }, { "addr": "172.21.15.174:6835", "nonce": 2883220846, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4235, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-25T00:03:39.488 INFO:tasks.workunit.client.1.smithi174.stdout:checking for srand48_r... yes 2024-04-25T00:03:39.518 INFO:tasks.workunit.client.1.smithi174.stdout:configure: creating ./config.status 2024-04-25T00:03:39.705 INFO:tasks.workunit.client.1.smithi174.stdout: /bin/bash ./config.status 2024-04-25T00:03:39.812 INFO:tasks.workunit.client.1.smithi174.stdout:config.status: creating Makefile 2024-04-25T00:03:39.835 INFO:tasks.workunit.client.1.smithi174.stdout:config.status: creating config.h 2024-04-25T00:03:39.883 INFO:tasks.workunit.client.1.smithi174.stdout:config.status: executing depfiles commands 2024-04-25T00:03:39.917 INFO:tasks.workunit.client.1.smithi174.stdout:config.status: executing default commands 2024-04-25T00:03:40.008 INFO:tasks.workunit.client.1.smithi174.stdout:cd . && /bin/bash /home/ubuntu/cephtest/mnt.1/client.1/tmp/ffsb/missing --run autoheader 2024-04-25T00:03:40.542 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:03:40.653 INFO:tasks.workunit.client.0.smithi116.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT list.o -MD -MP -MF .deps/list.Tpo -c -o list.o list.c 2024-04-25T00:03:40.671 INFO:tasks.workunit.client.1.smithi174.stderr:autoheader: warning: autoconf input should be named 'configure.ac', not 'configure.in' 2024-04-25T00:03:40.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.667+0000 7f26255d8640 1 Processor -- start 2024-04-25T00:03:40.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.667+0000 7f26255d8640 1 -- start start 2024-04-25T00:03:40.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.667+0000 7f26255d8640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f2620150d40 0x7f2620154170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:40.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.667+0000 7f26255d8640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2620070770 0x7f26200ab200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:40.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.667+0000 7f26255d8640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2620070180 con 0x7f26200c4c20 2024-04-25T00:03:40.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.667+0000 7f26255d8640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f26200c7530 con 0x7f2620150d40 2024-04-25T00:03:40.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.667+0000 7f26255d8640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f2620071280 con 0x7f2620070770 2024-04-25T00:03:40.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.667+0000 7f261f7fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2620070770 0x7f26200ab200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:40.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.667+0000 7f261f7fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2620070770 0x7f26200ab200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:41992/0 (socket says 172.21.15.116:41992) 2024-04-25T00:03:40.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.667+0000 7f261f7fe640 1 -- 172.21.15.116:0/2181343242 learned_addr learned my addr 172.21.15.116:0/2181343242 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:03:40.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.667+0000 7f261effd640 1 --2- 172.21.15.116:0/2181343242 >> v2:172.21.15.174:3300/0 conn(0x7f2620150d40 0x7f2620154170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:40.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.671+0000 7f261f7fe640 1 -- 172.21.15.116:0/2181343242 >> v2:172.21.15.174:3300/0 conn(0x7f2620150d40 msgr2=0x7f2620154170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:03:40.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.671+0000 7f261f7fe640 1 --2- 172.21.15.116:0/2181343242 >> v2:172.21.15.174:3300/0 conn(0x7f2620150d40 0x7f2620154170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:40.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.671+0000 7f261f7fe640 1 -- 172.21.15.116:0/2181343242 >> v1:172.21.15.174:6789/0 conn(0x7f26200c4c20 legacy=0x7f26200c6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:40.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.671+0000 7f261f7fe640 1 -- 172.21.15.116:0/2181343242 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2620070460 con 0x7f2620070770 2024-04-25T00:03:40.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.671+0000 7f261f7fe640 1 --2- 172.21.15.116:0/2181343242 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2620070770 0x7f26200ab200 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2618009870 tx=0x7f261801c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8bdee7a4fefabfb6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:40.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.671+0000 7f261dffb640 1 -- 172.21.15.116:0/2181343242 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26180b1070 con 0x7f2620070770 2024-04-25T00:03:40.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.671+0000 7f261dffb640 1 -- 172.21.15.116:0/2181343242 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26180a8050 con 0x7f2620070770 2024-04-25T00:03:40.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.671+0000 7f261dffb640 1 -- 172.21.15.116:0/2181343242 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26180ac860 con 0x7f2620070770 2024-04-25T00:03:40.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.671+0000 7f26255d8640 1 -- 172.21.15.116:0/2181343242 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2620070770 msgr2=0x7f26200ab200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:40.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.671+0000 7f26255d8640 1 --2- 172.21.15.116:0/2181343242 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2620070770 0x7f26200ab200 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2618009870 tx=0x7f261801c4d0 comp rx=0 tx=0).stop 2024-04-25T00:03:40.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.671+0000 7f26255d8640 1 -- 172.21.15.116:0/2181343242 shutdown_connections 2024-04-25T00:03:40.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.671+0000 7f26255d8640 1 --2- 172.21.15.116:0/2181343242 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2620070770 0x7f26200ab200 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:40.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.671+0000 7f26255d8640 1 --2- 172.21.15.116:0/2181343242 >> v2:172.21.15.174:3300/0 conn(0x7f2620150d40 0x7f2620154170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:40.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.671+0000 7f26255d8640 1 -- 172.21.15.116:0/2181343242 >> 172.21.15.116:0/2181343242 conn(0x7f262006f0c0 msgr2=0x7f262006f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:03:40.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.671+0000 7f26255d8640 1 -- 172.21.15.116:0/2181343242 shutdown_connections 2024-04-25T00:03:40.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.675+0000 7f26255d8640 1 -- 172.21.15.116:0/2181343242 wait complete. 2024-04-25T00:03:40.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.675+0000 7f26255d8640 1 Processor -- start 2024-04-25T00:03:40.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.675+0000 7f26255d8640 1 -- start start 2024-04-25T00:03:40.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.675+0000 7f26255d8640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2620070770 0x7f2620143c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:40.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.675+0000 7f26255d8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f26200c4c20 0x7f262014ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:40.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.675+0000 7f26255d8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2620150d40 0x7f262014f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:40.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.675+0000 7f26255d8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2620151120 con 0x7f26200c4c20 2024-04-25T00:03:40.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.675+0000 7f26255d8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f262006ff30 con 0x7f2620150d40 2024-04-25T00:03:40.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.675+0000 7f26255d8640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f26201512a0 con 0x7f2620070770 2024-04-25T00:03:40.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.675+0000 7f261e7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f26200c4c20 0x7f262014ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:40.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.675+0000 7f261e7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f26200c4c20 0x7f262014ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57908/0 (socket says 172.21.15.116:57908) 2024-04-25T00:03:40.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.675+0000 7f261e7fc640 1 -- 172.21.15.116:0/1203390845 learned_addr learned my addr 172.21.15.116:0/1203390845 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:03:40.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.675+0000 7f261effd640 1 --2- 172.21.15.116:0/1203390845 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2620070770 0x7f2620143c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:40.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.675+0000 7f261f7fe640 1 --2- 172.21.15.116:0/1203390845 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2620150d40 0x7f262014f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:40.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.675+0000 7f261e7fc640 1 -- 172.21.15.116:0/1203390845 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2620070770 msgr2=0x7f2620143c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:40.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.675+0000 7f261e7fc640 1 --2- 172.21.15.116:0/1203390845 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2620070770 0x7f2620143c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:40.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.675+0000 7f261e7fc640 1 -- 172.21.15.116:0/1203390845 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2620150d40 msgr2=0x7f262014f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:40.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.675+0000 7f261e7fc640 1 --2- 172.21.15.116:0/1203390845 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2620150d40 0x7f262014f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:40.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.675+0000 7f261e7fc640 1 -- 172.21.15.116:0/1203390845 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2620146780 con 0x7f26200c4c20 2024-04-25T00:03:40.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.675+0000 7f261e7fc640 1 --2- 172.21.15.116:0/1203390845 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f26200c4c20 0x7f262014ed00 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f261400c420 tx=0x7f261400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:40.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.675+0000 7f26077fe640 1 -- 172.21.15.116:0/1203390845 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26140052d0 con 0x7f26200c4c20 2024-04-25T00:03:40.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.675+0000 7f26077fe640 1 -- 172.21.15.116:0/1203390845 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2614005470 con 0x7f26200c4c20 2024-04-25T00:03:40.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.675+0000 7f26255d8640 1 -- 172.21.15.116:0/1203390845 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2620146a60 con 0x7f26200c4c20 2024-04-25T00:03:40.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.675+0000 7f26077fe640 1 -- 172.21.15.116:0/1203390845 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2614011ad0 con 0x7f26200c4c20 2024-04-25T00:03:40.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.675+0000 7f26255d8640 1 -- 172.21.15.116:0/1203390845 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2620145820 con 0x7f26200c4c20 2024-04-25T00:03:40.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.679+0000 7f26255d8640 1 -- 172.21.15.116:0/1203390845 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2620068a60 con 0x7f26200c4c20 2024-04-25T00:03:40.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.679+0000 7f26077fe640 1 -- 172.21.15.116:0/1203390845 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2614011c70 con 0x7f26200c4c20 2024-04-25T00:03:40.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.679+0000 7f26077fe640 1 --2- 172.21.15.116:0/1203390845 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f260c0678c0 0x7f260c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:40.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.679+0000 7f26077fe640 1 -- 172.21.15.116:0/1203390845 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 6924+0+0 (secure 0 0 0) 0x7f2614089340 con 0x7f26200c4c20 2024-04-25T00:03:40.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.679+0000 7f261effd640 1 --2- 172.21.15.116:0/1203390845 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f260c0678c0 0x7f260c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:40.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.679+0000 7f261effd640 1 --2- 172.21.15.116:0/1203390845 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f260c0678c0 0x7f260c069d80 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2610011340 tx=0x7f261000b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:40.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.683+0000 7f26077fe640 1 -- 172.21.15.116:0/1203390845 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f2614059400 con 0x7f26200c4c20 2024-04-25T00:03:40.770 INFO:tasks.workunit.client.0.smithi116.stdout:mv -f .deps/list.Tpo .deps/list.Po 2024-04-25T00:03:40.771 INFO:tasks.workunit.client.1.smithi174.stderr:aclocal.m4:16: warning: this file was generated for autoconf 2.61. 2024-04-25T00:03:40.771 INFO:tasks.workunit.client.1.smithi174.stderr:You have another version of autoconf. It may work, but is not guaranteed to. 2024-04-25T00:03:40.771 INFO:tasks.workunit.client.1.smithi174.stderr:If you have problems, you may need to regenerate the build system entirely. 2024-04-25T00:03:40.771 INFO:tasks.workunit.client.1.smithi174.stderr:To do so, use the procedure documented by the package, typically `autoreconf'. 2024-04-25T00:03:40.784 INFO:tasks.workunit.client.0.smithi116.stdout:gcc -D_GNU_SOURCE -g -O2 -D_REENTRANT -o ffsb fileops.o rand.o main.o fh.o filelist.o metaops.o rwlock.o cirlist.o rbt.o ffsb_tg.o ffsb_fs.o ffsb_thread.o ffsb_op.o util.o parser.o ffsb_fc.o ffsb_stats.o list.o -lpthread -lm 2024-04-25T00:03:40.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.911+0000 7f26255d8640 1 -- 172.21.15.116:0/1203390845 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f26200aa190 con 0x7f26200c4c20 2024-04-25T00:03:40.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.911+0000 7f26077fe640 1 -- 172.21.15.116:0/1203390845 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+5673 (secure 0 0 0) 0x7f261405d700 con 0x7f26200c4c20 2024-04-25T00:03:40.917 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:03:40.917 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:03:40.629360+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4828":{"gid":4828,"name":"c","rank":0,"incarnation":26,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6834/1649769369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1649769369},{"type":"v1","addr":"172.21.15.116:6834","nonce":1649769369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4828,"qdb_cluster":[4828]},"id":1},{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:03:34.579825+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4735},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4735":{"gid":4735,"name":"a","rank":0,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.116:6835/2004876235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":2004876235},{"type":"v1","addr":"172.21.15.116:6835","nonce":2004876235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4735,"qdb_cluster":[4735]},"id":2}]} 2024-04-25T00:03:40.917 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 30 2024-04-25T00:03:40.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.915+0000 7f26255d8640 1 -- 172.21.15.116:0/1203390845 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f260c0678c0 msgr2=0x7f260c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:40.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.915+0000 7f26255d8640 1 --2- 172.21.15.116:0/1203390845 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f260c0678c0 0x7f260c069d80 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2610011340 tx=0x7f261000b040 comp rx=0 tx=0).stop 2024-04-25T00:03:40.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.915+0000 7f26255d8640 1 -- 172.21.15.116:0/1203390845 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f26200c4c20 msgr2=0x7f262014ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:40.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.915+0000 7f26255d8640 1 --2- 172.21.15.116:0/1203390845 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f26200c4c20 0x7f262014ed00 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f261400c420 tx=0x7f261400c8e0 comp rx=0 tx=0).stop 2024-04-25T00:03:40.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.915+0000 7f26255d8640 1 -- 172.21.15.116:0/1203390845 shutdown_connections 2024-04-25T00:03:40.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.915+0000 7f26255d8640 1 --2- 172.21.15.116:0/1203390845 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f260c0678c0 0x7f260c069d80 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:40.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.915+0000 7f26255d8640 1 --2- 172.21.15.116:0/1203390845 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2620150d40 0x7f262014f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:40.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.915+0000 7f26255d8640 1 --2- 172.21.15.116:0/1203390845 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f26200c4c20 0x7f262014ed00 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:40.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.915+0000 7f26255d8640 1 --2- 172.21.15.116:0/1203390845 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2620070770 0x7f2620143c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:40.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.915+0000 7f26255d8640 1 -- 172.21.15.116:0/1203390845 >> 172.21.15.116:0/1203390845 conn(0x7f262006f0c0 msgr2=0x7f262014d710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:03:40.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.915+0000 7f26255d8640 1 -- 172.21.15.116:0/1203390845 shutdown_connections 2024-04-25T00:03:40.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:40.915+0000 7f26255d8640 1 -- 172.21.15.116:0/1203390845 wait complete. 2024-04-25T00:03:40.923 INFO:tasks.workunit.client.1.smithi174.stdout:rm -f stamp-h1 2024-04-25T00:03:40.930 INFO:tasks.workunit.client.1.smithi174.stdout:touch config.h.in 2024-04-25T00:03:40.938 INFO:tasks.mds_thrash.fs.[b]:mds.c reported in up:active state 2024-04-25T00:03:40.938 INFO:tasks.mds_thrash.fs.[b]:waiting for 50 secs before thrashing 2024-04-25T00:03:40.938 INFO:tasks.workunit.client.1.smithi174.stdout:cd . && /bin/bash ./config.status config.h 2024-04-25T00:03:40.949 INFO:tasks.workunit.client.0.smithi116.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2024-04-25T00:03:40.950 INFO:tasks.workunit.client.0.smithi116.stderr:+ cd .. 2024-04-25T00:03:40.951 INFO:tasks.workunit.client.0.smithi116.stderr:+ mkdir tmp 2024-04-25T00:03:40.953 INFO:tasks.workunit.client.0.smithi116.stderr:+ cd tmp 2024-04-25T00:03:40.955 INFO:tasks.workunit.client.0.smithi116.stderr:+ for f in $mydir/*.ffsb 2024-04-25T00:03:40.955 INFO:tasks.workunit.client.0.smithi116.stderr:+ ../ffsb/ffsb /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/random_write.32.ffsb 2024-04-25T00:03:40.956 INFO:tasks.workunit.client.0.smithi116.stdout:FFSB version 6.0-RC2 started 2024-04-25T00:03:40.956 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-04-25T00:03:40.959 INFO:tasks.workunit.client.0.smithi116.stdout:benchmark time = 300 2024-04-25T00:03:40.959 INFO:tasks.workunit.client.0.smithi116.stdout:ThreadGroup 0 2024-04-25T00:03:40.959 INFO:tasks.workunit.client.0.smithi116.stdout:================ 2024-04-25T00:03:40.959 INFO:tasks.workunit.client.0.smithi116.stdout: num_threads = 32 2024-04-25T00:03:40.959 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-04-25T00:03:40.959 INFO:tasks.workunit.client.0.smithi116.stdout: read_random = off 2024-04-25T00:03:40.959 INFO:tasks.workunit.client.0.smithi116.stdout: read_size = 0 (0B) 2024-04-25T00:03:40.959 INFO:tasks.workunit.client.0.smithi116.stdout: read_blocksize = 0 (0B) 2024-04-25T00:03:40.959 INFO:tasks.workunit.client.0.smithi116.stdout: read_skip = off 2024-04-25T00:03:40.959 INFO:tasks.workunit.client.0.smithi116.stdout: read_skipsize = 0 (0B) 2024-04-25T00:03:40.959 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-04-25T00:03:40.960 INFO:tasks.workunit.client.0.smithi116.stdout: write_random = on 2024-04-25T00:03:40.960 INFO:tasks.workunit.client.0.smithi116.stdout: write_size = 5242880 (5MB) 2024-04-25T00:03:40.960 INFO:tasks.workunit.client.0.smithi116.stdout: fsync_file = 0 2024-04-25T00:03:40.960 INFO:tasks.workunit.client.0.smithi116.stdout: write_blocksize = 4096 (4KB) 2024-04-25T00:03:40.960 INFO:tasks.workunit.client.0.smithi116.stdout: wait time = 0 2024-04-25T00:03:40.960 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-04-25T00:03:40.960 INFO:tasks.workunit.client.0.smithi116.stdout: op weights 2024-04-25T00:03:40.960 INFO:tasks.workunit.client.0.smithi116.stdout: read = 0 (0.00%) 2024-04-25T00:03:40.960 INFO:tasks.workunit.client.0.smithi116.stdout: readall = 0 (0.00%) 2024-04-25T00:03:40.960 INFO:tasks.workunit.client.0.smithi116.stdout: write = 1 (100.00%) 2024-04-25T00:03:40.960 INFO:tasks.workunit.client.0.smithi116.stdout: create = 0 (0.00%) 2024-04-25T00:03:40.960 INFO:tasks.workunit.client.0.smithi116.stdout: append = 0 (0.00%) 2024-04-25T00:03:40.960 INFO:tasks.workunit.client.0.smithi116.stdout: delete = 0 (0.00%) 2024-04-25T00:03:40.960 INFO:tasks.workunit.client.0.smithi116.stdout: metaop = 0 (0.00%) 2024-04-25T00:03:40.960 INFO:tasks.workunit.client.0.smithi116.stdout: createdir = 0 (0.00%) 2024-04-25T00:03:40.961 INFO:tasks.workunit.client.0.smithi116.stdout: stat = 0 (0.00%) 2024-04-25T00:03:40.961 INFO:tasks.workunit.client.0.smithi116.stdout: writeall = 0 (0.00%) 2024-04-25T00:03:40.961 INFO:tasks.workunit.client.0.smithi116.stdout: writeall_fsync = 0 (0.00%) 2024-04-25T00:03:40.961 INFO:tasks.workunit.client.0.smithi116.stdout: open_close = 0 (0.00%) 2024-04-25T00:03:40.961 INFO:tasks.workunit.client.0.smithi116.stdout: write_fsync = 0 (0.00%) 2024-04-25T00:03:40.961 INFO:tasks.workunit.client.0.smithi116.stdout: create_fsync = 0 (0.00%) 2024-04-25T00:03:40.961 INFO:tasks.workunit.client.0.smithi116.stdout: append_fsync = 0 (0.00%) 2024-04-25T00:03:40.961 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-04-25T00:03:40.961 INFO:tasks.workunit.client.0.smithi116.stdout:FileSystem . 2024-04-25T00:03:40.961 INFO:tasks.workunit.client.0.smithi116.stdout:========== 2024-04-25T00:03:40.961 INFO:tasks.workunit.client.0.smithi116.stdout: num_dirs = 0 2024-04-25T00:03:40.961 INFO:tasks.workunit.client.0.smithi116.stdout: starting files = 128 2024-04-25T00:03:40.961 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-04-25T00:03:40.961 INFO:tasks.workunit.client.0.smithi116.stdout: min file size = 104857600 (100MB) 2024-04-25T00:03:40.961 INFO:tasks.workunit.client.0.smithi116.stdout: max file size = 104857600 (100MB) 2024-04-25T00:03:40.962 INFO:tasks.workunit.client.0.smithi116.stdout: directio = off 2024-04-25T00:03:40.962 INFO:tasks.workunit.client.0.smithi116.stdout: alignedio = on 2024-04-25T00:03:40.962 INFO:tasks.workunit.client.0.smithi116.stdout: bufferedio = off 2024-04-25T00:03:40.962 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-04-25T00:03:40.962 INFO:tasks.workunit.client.0.smithi116.stdout: aging is off 2024-04-25T00:03:40.962 INFO:tasks.workunit.client.0.smithi116.stdout: current utilization = 0.00% 2024-04-25T00:03:40.963 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-04-25T00:03:40.963 INFO:tasks.workunit.client.0.smithi116.stdout:checking existing fs: . 2024-04-25T00:03:40.963 INFO:tasks.workunit.client.0.smithi116.stdout:recreating new fileset 2024-04-25T00:03:40.963 INFO:tasks.workunit.client.0.smithi116.stderr:opendir: No such file or directory 2024-04-25T00:03:41.011 INFO:tasks.workunit.client.1.smithi174.stdout:config.status: creating config.h 2024-04-25T00:03:41.059 INFO:tasks.workunit.client.1.smithi174.stdout:make all-am 2024-04-25T00:03:41.089 INFO:tasks.workunit.client.1.smithi174.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/mnt.1/client.1/tmp/ffsb' 2024-04-25T00:03:41.089 INFO:tasks.workunit.client.1.smithi174.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fileops.o -MD -MP -MF .deps/fileops.Tpo -c -o fileops.o fileops.c 2024-04-25T00:03:41.291 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:03:41.375 INFO:tasks.workunit.client.1.smithi174.stdout:mv -f .deps/fileops.Tpo .deps/fileops.Po 2024-04-25T00:03:41.389 INFO:tasks.workunit.client.1.smithi174.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rand.o -MD -MP -MF .deps/rand.Tpo -c -o rand.o rand.c 2024-04-25T00:03:41.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.419+0000 7f4e2053f640 1 Processor -- start 2024-04-25T00:03:41.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.419+0000 7f4e2053f640 1 -- start start 2024-04-25T00:03:41.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.419+0000 7f4e2053f640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f4e180c5200 0x7f4e18154040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:41.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.419+0000 7f4e2053f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e180c2b40 con 0x7f4e18156bb0 2024-04-25T00:03:41.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.419+0000 7f4e2053f640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e18154580 con 0x7f4e180c4760 2024-04-25T00:03:41.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.419+0000 7f4e2053f640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f4e18072b80 con 0x7f4e180c5200 2024-04-25T00:03:41.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.419+0000 7f4e1eadc640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f4e18156bb0 0x7f4e18156fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:53080/0 (socket says 172.21.15.116:53080) 2024-04-25T00:03:41.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.419+0000 7f4e1eadc640 1 -- 172.21.15.116:0/1742903987 learned_addr learned my addr 172.21.15.116:0/1742903987 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:03:41.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.419+0000 7f4e1e2db640 1 --2- 172.21.15.116:0/1742903987 >> v2:172.21.15.174:3300/0 conn(0x7f4e180c5200 0x7f4e18154040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:41.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.419+0000 7f4e1d2d9640 1 -- 172.21.15.116:0/1742903987 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1467032119 0 0) 0x7f4e180c2b40 con 0x7f4e18156bb0 2024-04-25T00:03:41.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.419+0000 7f4e1d2d9640 1 -- 172.21.15.116:0/1742903987 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4e04003530 con 0x7f4e18156bb0 2024-04-25T00:03:41.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.419+0000 7f4e1e2db640 1 -- 172.21.15.116:0/1742903987 >> v1:172.21.15.174:6789/0 conn(0x7f4e180c4760 legacy=0x7f4e1814d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:41.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.419+0000 7f4e1d2d9640 1 -- 172.21.15.116:0/1742903987 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4196963886 0 0) 0x7f4e08001120 con 0x7f4e180c4760 2024-04-25T00:03:41.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.419+0000 7f4e1e2db640 1 -- 172.21.15.116:0/1742903987 >> v1:172.21.15.116:6789/0 conn(0x7f4e18156bb0 legacy=0x7f4e18156fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:41.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.419+0000 7f4e1e2db640 1 -- 172.21.15.116:0/1742903987 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e181576f0 con 0x7f4e180c5200 2024-04-25T00:03:41.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.423+0000 7f4e1e2db640 1 --2- 172.21.15.116:0/1742903987 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4e180c5200 0x7f4e18154040 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4e140097c0 tx=0x7f4e1401c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a0b5ca677b3b0a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:41.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.423+0000 7f4e1d2d9640 1 -- 172.21.15.116:0/1742903987 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e140b0070 con 0x7f4e180c5200 2024-04-25T00:03:41.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.423+0000 7f4e1d2d9640 1 -- 172.21.15.116:0/1742903987 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e140ab600 con 0x7f4e180c5200 2024-04-25T00:03:41.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.423+0000 7f4e1d2d9640 1 -- 172.21.15.116:0/1742903987 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e140ab9d0 con 0x7f4e180c5200 2024-04-25T00:03:41.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.423+0000 7f4e2053f640 1 -- 172.21.15.116:0/1742903987 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4e180c5200 msgr2=0x7f4e18154040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:41.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.423+0000 7f4e2053f640 1 --2- 172.21.15.116:0/1742903987 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4e180c5200 0x7f4e18154040 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4e140097c0 tx=0x7f4e1401c410 comp rx=0 tx=0).stop 2024-04-25T00:03:41.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.423+0000 7f4e2053f640 1 -- 172.21.15.116:0/1742903987 shutdown_connections 2024-04-25T00:03:41.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.423+0000 7f4e2053f640 1 --2- 172.21.15.116:0/1742903987 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4e180c5200 0x7f4e18154040 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:41.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.423+0000 7f4e2053f640 1 -- 172.21.15.116:0/1742903987 >> 172.21.15.116:0/1742903987 conn(0x7f4e1806c720 msgr2=0x7f4e1806cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:03:41.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.423+0000 7f4e2053f640 1 -- 172.21.15.116:0/1742903987 shutdown_connections 2024-04-25T00:03:41.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.423+0000 7f4e2053f640 1 -- 172.21.15.116:0/1742903987 wait complete. 2024-04-25T00:03:41.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.423+0000 7f4e2053f640 1 Processor -- start 2024-04-25T00:03:41.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.423+0000 7f4e2053f640 1 -- start start 2024-04-25T00:03:41.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.423+0000 7f4e2053f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e180c4760 0x7f4e1813f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:41.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.423+0000 7f4e2053f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4e180c5200 0x7f4e18147630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:41.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.423+0000 7f4e2053f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4e18156bb0 0x7f4e18147b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:41.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.427+0000 7f4e2053f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4e1814d940 con 0x7f4e180c4760 2024-04-25T00:03:41.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.427+0000 7f4e2053f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4e180c2720 con 0x7f4e18156bb0 2024-04-25T00:03:41.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.427+0000 7f4e2053f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f4e1814dac0 con 0x7f4e180c5200 2024-04-25T00:03:41.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.427+0000 7f4e1e2db640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e180c4760 0x7f4e1813f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:41.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.427+0000 7f4e1e2db640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e180c4760 0x7f4e1813f410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57918/0 (socket says 172.21.15.116:57918) 2024-04-25T00:03:41.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.427+0000 7f4e1e2db640 1 -- 172.21.15.116:0/1321005168 learned_addr learned my addr 172.21.15.116:0/1321005168 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:03:41.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.427+0000 7f4e1eadc640 1 --2- 172.21.15.116:0/1321005168 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4e18156bb0 0x7f4e18147b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:41.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.427+0000 7f4e1dada640 1 --2- 172.21.15.116:0/1321005168 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4e180c5200 0x7f4e18147630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:41.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.427+0000 7f4e1e2db640 1 -- 172.21.15.116:0/1321005168 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4e180c5200 msgr2=0x7f4e18147630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:41.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.427+0000 7f4e1e2db640 1 --2- 172.21.15.116:0/1321005168 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4e180c5200 0x7f4e18147630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:41.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.427+0000 7f4e1e2db640 1 -- 172.21.15.116:0/1321005168 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4e18156bb0 msgr2=0x7f4e18147b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:41.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.427+0000 7f4e1e2db640 1 --2- 172.21.15.116:0/1321005168 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4e18156bb0 0x7f4e18147b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:41.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.427+0000 7f4e1e2db640 1 -- 172.21.15.116:0/1321005168 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e18149750 con 0x7f4e180c4760 2024-04-25T00:03:41.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.427+0000 7f4e1e2db640 1 --2- 172.21.15.116:0/1321005168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e180c4760 0x7f4e1813f410 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f4e140ae040 tx=0x7f4e1401ac50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:41.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.427+0000 7f4e0effd640 1 -- 172.21.15.116:0/1321005168 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e140bb020 con 0x7f4e180c4760 2024-04-25T00:03:41.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.427+0000 7f4e0effd640 1 -- 172.21.15.116:0/1321005168 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e140b1040 con 0x7f4e180c4760 2024-04-25T00:03:41.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.427+0000 7f4e2053f640 1 -- 172.21.15.116:0/1321005168 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e18149470 con 0x7f4e180c4760 2024-04-25T00:03:41.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.427+0000 7f4e0effd640 1 -- 172.21.15.116:0/1321005168 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e140b0050 con 0x7f4e180c4760 2024-04-25T00:03:41.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.427+0000 7f4e2053f640 1 -- 172.21.15.116:0/1321005168 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e1813f950 con 0x7f4e180c4760 2024-04-25T00:03:41.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.427+0000 7f4e2053f640 1 -- 172.21.15.116:0/1321005168 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e18068a60 con 0x7f4e180c4760 2024-04-25T00:03:41.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.427+0000 7f4e0effd640 1 -- 172.21.15.116:0/1321005168 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4e140be070 con 0x7f4e180c4760 2024-04-25T00:03:41.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.427+0000 7f4e0effd640 1 --2- 172.21.15.116:0/1321005168 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4dfc0678c0 0x7f4dfc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:41.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.427+0000 7f4e0effd640 1 -- 172.21.15.116:0/1321005168 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 6924+0+0 (secure 0 0 0) 0x7f4e1411f5d0 con 0x7f4e180c4760 2024-04-25T00:03:41.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.427+0000 7f4e1dada640 1 --2- 172.21.15.116:0/1321005168 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4dfc0678c0 0x7f4dfc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:41.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.431+0000 7f4e1dada640 1 --2- 172.21.15.116:0/1321005168 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4dfc0678c0 0x7f4dfc069d80 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f4e180c8d00 tx=0x7f4e08009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:41.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.435+0000 7f4e0effd640 1 -- 172.21.15.116:0/1321005168 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f4e18149750 con 0x7f4e180c4760 2024-04-25T00:03:41.601 INFO:tasks.workunit.client.1.smithi174.stdout:mv -f .deps/rand.Tpo .deps/rand.Po 2024-04-25T00:03:41.613 INFO:tasks.workunit.client.1.smithi174.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT main.o -MD -MP -MF .deps/main.Tpo -c -o main.o main.c 2024-04-25T00:03:41.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.659+0000 7f4e2053f640 1 -- 172.21.15.116:0/1321005168 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4e18149ad0 con 0x7f4e180c4760 2024-04-25T00:03:41.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.659+0000 7f4e0effd640 1 -- 172.21.15.116:0/1321005168 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+5673 (secure 0 0 0) 0x7f4e14016090 con 0x7f4e180c4760 2024-04-25T00:03:41.665 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:03:41.665 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:03:40.629360+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4828":{"gid":4828,"name":"c","rank":0,"incarnation":26,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6834/1649769369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1649769369},{"type":"v1","addr":"172.21.15.116:6834","nonce":1649769369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4828,"qdb_cluster":[4828]},"id":1},{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:03:34.579825+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4735},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4735":{"gid":4735,"name":"a","rank":0,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.116:6835/2004876235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":2004876235},{"type":"v1","addr":"172.21.15.116:6835","nonce":2004876235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4735,"qdb_cluster":[4735]},"id":2}]} 2024-04-25T00:03:41.665 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 30 2024-04-25T00:03:41.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.663+0000 7f4e2053f640 1 -- 172.21.15.116:0/1321005168 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4dfc0678c0 msgr2=0x7f4dfc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:41.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.663+0000 7f4e2053f640 1 --2- 172.21.15.116:0/1321005168 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4dfc0678c0 0x7f4dfc069d80 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f4e180c8d00 tx=0x7f4e08009040 comp rx=0 tx=0).stop 2024-04-25T00:03:41.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.663+0000 7f4e2053f640 1 -- 172.21.15.116:0/1321005168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e180c4760 msgr2=0x7f4e1813f410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:41.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.663+0000 7f4e2053f640 1 --2- 172.21.15.116:0/1321005168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e180c4760 0x7f4e1813f410 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f4e140ae040 tx=0x7f4e1401ac50 comp rx=0 tx=0).stop 2024-04-25T00:03:41.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.663+0000 7f4e2053f640 1 -- 172.21.15.116:0/1321005168 shutdown_connections 2024-04-25T00:03:41.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.663+0000 7f4e2053f640 1 --2- 172.21.15.116:0/1321005168 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4dfc0678c0 0x7f4dfc069d80 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:41.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.663+0000 7f4e2053f640 1 --2- 172.21.15.116:0/1321005168 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4e18156bb0 0x7f4e18147b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:41.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.663+0000 7f4e2053f640 1 --2- 172.21.15.116:0/1321005168 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4e180c5200 0x7f4e18147630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:41.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.663+0000 7f4e2053f640 1 --2- 172.21.15.116:0/1321005168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e180c4760 0x7f4e1813f410 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:41.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.663+0000 7f4e2053f640 1 -- 172.21.15.116:0/1321005168 >> 172.21.15.116:0/1321005168 conn(0x7f4e1806c720 msgr2=0x7f4e18156fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:03:41.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.663+0000 7f4e2053f640 1 -- 172.21.15.116:0/1321005168 shutdown_connections 2024-04-25T00:03:41.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:41.663+0000 7f4e2053f640 1 -- 172.21.15.116:0/1321005168 wait complete. 2024-04-25T00:03:41.686 INFO:tasks.mds_thrash.fs.[a]:mds.c has gained rank=0, replacing gid=4270 2024-04-25T00:03:41.686 INFO:tasks.mds_thrash.fs.[a]:waiting for 6 secs before reviving mds.b 2024-04-25T00:03:41.838 INFO:tasks.workunit.client.1.smithi174.stdout:mv -f .deps/main.Tpo .deps/main.Po 2024-04-25T00:03:41.848 INFO:tasks.workunit.client.1.smithi174.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fh.o -MD -MP -MF .deps/fh.Tpo -c -o fh.o fh.c 2024-04-25T00:03:42.078 INFO:tasks.workunit.client.1.smithi174.stdout:mv -f .deps/fh.Tpo .deps/fh.Po 2024-04-25T00:03:42.090 INFO:tasks.workunit.client.1.smithi174.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT filelist.o -MD -MP -MF .deps/filelist.Tpo -c -o filelist.o filelist.c 2024-04-25T00:03:42.357 INFO:tasks.workunit.client.1.smithi174.stdout:mv -f .deps/filelist.Tpo .deps/filelist.Po 2024-04-25T00:03:42.374 INFO:tasks.workunit.client.1.smithi174.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT metaops.o -MD -MP -MF .deps/metaops.Tpo -c -o metaops.o metaops.c 2024-04-25T00:03:42.583 INFO:tasks.workunit.client.1.smithi174.stdout:mv -f .deps/metaops.Tpo .deps/metaops.Po 2024-04-25T00:03:42.595 INFO:tasks.workunit.client.1.smithi174.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rwlock.o -MD -MP -MF .deps/rwlock.Tpo -c -o rwlock.o rwlock.c 2024-04-25T00:03:42.602 INFO:tasks.ceph.mds.c.smithi116.stderr:2024-04-25T00:03:42.599+0000 7fec2030e640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T00:03:42.708 INFO:tasks.workunit.client.1.smithi174.stdout:mv -f .deps/rwlock.Tpo .deps/rwlock.Po 2024-04-25T00:03:42.721 INFO:tasks.workunit.client.1.smithi174.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT cirlist.o -MD -MP -MF .deps/cirlist.Tpo -c -o cirlist.o cirlist.c 2024-04-25T00:03:42.843 INFO:tasks.workunit.client.1.smithi174.stdout:mv -f .deps/cirlist.Tpo .deps/cirlist.Po 2024-04-25T00:03:42.857 INFO:tasks.workunit.client.1.smithi174.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rbt.o -MD -MP -MF .deps/rbt.Tpo -c -o rbt.o rbt.c 2024-04-25T00:03:43.171 INFO:tasks.workunit.client.1.smithi174.stdout:mv -f .deps/rbt.Tpo .deps/rbt.Po 2024-04-25T00:03:43.180 INFO:tasks.workunit.client.1.smithi174.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_tg.o -MD -MP -MF .deps/ffsb_tg.Tpo -c -o ffsb_tg.o ffsb_tg.c 2024-04-25T00:03:43.430 INFO:tasks.workunit.client.1.smithi174.stdout:mv -f .deps/ffsb_tg.Tpo .deps/ffsb_tg.Po 2024-04-25T00:03:43.445 INFO:tasks.workunit.client.1.smithi174.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fs.o -MD -MP -MF .deps/ffsb_fs.Tpo -c -o ffsb_fs.o ffsb_fs.c 2024-04-25T00:03:43.765 INFO:tasks.workunit.client.1.smithi174.stdout:mv -f .deps/ffsb_fs.Tpo .deps/ffsb_fs.Po 2024-04-25T00:03:43.779 INFO:tasks.workunit.client.1.smithi174.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_thread.o -MD -MP -MF .deps/ffsb_thread.Tpo -c -o ffsb_thread.o ffsb_thread.c 2024-04-25T00:03:43.983 INFO:tasks.workunit.client.1.smithi174.stdout:mv -f .deps/ffsb_thread.Tpo .deps/ffsb_thread.Po 2024-04-25T00:03:43.993 INFO:tasks.workunit.client.1.smithi174.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_op.o -MD -MP -MF .deps/ffsb_op.Tpo -c -o ffsb_op.o ffsb_op.c 2024-04-25T00:03:44.114 INFO:tasks.workunit.client.1.smithi174.stderr:ffsb_op.c: In function 'print_op_results': 2024-04-25T00:03:44.114 INFO:tasks.workunit.client.1.smithi174.stderr:ffsb_op.c:83:37: warning: embedded '\0' in format [-Wformat-contains-nul] 2024-04-25T00:03:44.114 INFO:tasks.workunit.client.1.smithi174.stderr: 83 | sprintf(buf, "%s/sec\0", buf); 2024-04-25T00:03:44.114 INFO:tasks.workunit.client.1.smithi174.stderr: | ^~ 2024-04-25T00:03:44.114 INFO:tasks.workunit.client.1.smithi174.stderr:ffsb_op.c:86:33: warning: embedded '\0' in format [-Wformat-contains-nul] 2024-04-25T00:03:44.114 INFO:tasks.workunit.client.1.smithi174.stderr: 86 | sprintf(buf, "NA\0"); 2024-04-25T00:03:44.114 INFO:tasks.workunit.client.1.smithi174.stderr: | ^~ 2024-04-25T00:03:44.134 INFO:tasks.workunit.client.1.smithi174.stderr:ffsb_op.c: In function 'print_results': 2024-04-25T00:03:44.134 INFO:tasks.workunit.client.1.smithi174.stderr:ffsb_op.c:83:33: warning: '/sec' directive writing 4 bytes into a region of size between 1 and 256 [-Wformat-overflow=] 2024-04-25T00:03:44.134 INFO:tasks.workunit.client.1.smithi174.stderr: 83 | sprintf(buf, "%s/sec\0", buf); 2024-04-25T00:03:44.134 INFO:tasks.workunit.client.1.smithi174.stderr: | ^~~~ 2024-04-25T00:03:44.135 INFO:tasks.workunit.client.1.smithi174.stderr:In file included from /usr/include/stdio.h:894, 2024-04-25T00:03:44.135 INFO:tasks.workunit.client.1.smithi174.stderr: from ffsb_op.c:20: 2024-04-25T00:03:44.135 INFO:tasks.workunit.client.1.smithi174.stderr:/usr/include/x86_64-linux-gnu/bits/stdio2.h:38:10: note: '__builtin___sprintf_chk' output between 5 and 260 bytes into a destination of size 256 2024-04-25T00:03:44.135 INFO:tasks.workunit.client.1.smithi174.stderr: 38 | return __builtin___sprintf_chk (__s, __USE_FORTIFY_LEVEL - 1, 2024-04-25T00:03:44.135 INFO:tasks.workunit.client.1.smithi174.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-25T00:03:44.135 INFO:tasks.workunit.client.1.smithi174.stderr: 39 | __glibc_objsize (__s), __fmt, 2024-04-25T00:03:44.135 INFO:tasks.workunit.client.1.smithi174.stderr: | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-25T00:03:44.136 INFO:tasks.workunit.client.1.smithi174.stderr: 40 | __va_arg_pack ()); 2024-04-25T00:03:44.136 INFO:tasks.workunit.client.1.smithi174.stderr: | ~~~~~~~~~~~~~~~~~ 2024-04-25T00:03:44.193 INFO:tasks.workunit.client.1.smithi174.stdout:mv -f .deps/ffsb_op.Tpo .deps/ffsb_op.Po 2024-04-25T00:03:44.205 INFO:tasks.workunit.client.1.smithi174.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT util.o -MD -MP -MF .deps/util.Tpo -c -o util.o util.c 2024-04-25T00:03:44.477 INFO:tasks.workunit.client.1.smithi174.stdout:mv -f .deps/util.Tpo .deps/util.Po 2024-04-25T00:03:44.489 INFO:tasks.workunit.client.1.smithi174.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT parser.o -MD -MP -MF .deps/parser.Tpo -c -o parser.o parser.c 2024-04-25T00:03:44.608 INFO:tasks.workunit.client.1.smithi174.stderr:In file included from parser.h:22, 2024-04-25T00:03:44.609 INFO:tasks.workunit.client.1.smithi174.stderr: from parser.c:25: 2024-04-25T00:03:44.609 INFO:tasks.workunit.client.1.smithi174.stderr:parser.c: In function 'init_filesys': 2024-04-25T00:03:44.611 INFO:tasks.workunit.client.1.smithi174.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-25T00:03:44.611 INFO:tasks.workunit.client.1.smithi174.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-25T00:03:44.611 INFO:tasks.workunit.client.1.smithi174.stderr: | ^ 2024-04-25T00:03:44.611 INFO:tasks.workunit.client.1.smithi174.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2024-04-25T00:03:44.611 INFO:tasks.workunit.client.1.smithi174.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-25T00:03:44.611 INFO:tasks.workunit.client.1.smithi174.stderr: | ^~~~~~~~ 2024-04-25T00:03:44.611 INFO:tasks.workunit.client.1.smithi174.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2024-04-25T00:03:44.611 INFO:tasks.workunit.client.1.smithi174.stderr: 36 | container_of(ptr, type, member) 2024-04-25T00:03:44.611 INFO:tasks.workunit.client.1.smithi174.stderr: | ^~~~~~~~~~~~ 2024-04-25T00:03:44.611 INFO:tasks.workunit.client.1.smithi174.stderr:list.h:39:20: note: in expansion of macro 'list_entry' 2024-04-25T00:03:44.611 INFO:tasks.workunit.client.1.smithi174.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-04-25T00:03:44.612 INFO:tasks.workunit.client.1.smithi174.stderr: | ^~~~~~~~~~ 2024-04-25T00:03:44.612 INFO:tasks.workunit.client.1.smithi174.stderr:parser.c:934:17: note: in expansion of macro 'list_for_each_entry' 2024-04-25T00:03:44.612 INFO:tasks.workunit.client.1.smithi174.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2024-04-25T00:03:44.612 INFO:tasks.workunit.client.1.smithi174.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-25T00:03:44.612 INFO:tasks.workunit.client.1.smithi174.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-25T00:03:44.612 INFO:tasks.workunit.client.1.smithi174.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-25T00:03:44.612 INFO:tasks.workunit.client.1.smithi174.stderr: | ^ 2024-04-25T00:03:44.612 INFO:tasks.workunit.client.1.smithi174.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2024-04-25T00:03:44.612 INFO:tasks.workunit.client.1.smithi174.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-25T00:03:44.612 INFO:tasks.workunit.client.1.smithi174.stderr: | ^~~~~~~~ 2024-04-25T00:03:44.612 INFO:tasks.workunit.client.1.smithi174.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2024-04-25T00:03:44.612 INFO:tasks.workunit.client.1.smithi174.stderr: 36 | container_of(ptr, type, member) 2024-04-25T00:03:44.612 INFO:tasks.workunit.client.1.smithi174.stderr: | ^~~~~~~~~~~~ 2024-04-25T00:03:44.612 INFO:tasks.workunit.client.1.smithi174.stderr:list.h:41:20: note: in expansion of macro 'list_entry' 2024-04-25T00:03:44.613 INFO:tasks.workunit.client.1.smithi174.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-04-25T00:03:44.613 INFO:tasks.workunit.client.1.smithi174.stderr: | ^~~~~~~~~~ 2024-04-25T00:03:44.613 INFO:tasks.workunit.client.1.smithi174.stderr:parser.c:934:17: note: in expansion of macro 'list_for_each_entry' 2024-04-25T00:03:44.613 INFO:tasks.workunit.client.1.smithi174.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2024-04-25T00:03:44.613 INFO:tasks.workunit.client.1.smithi174.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-25T00:03:44.613 INFO:tasks.workunit.client.1.smithi174.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-25T00:03:44.613 INFO:tasks.workunit.client.1.smithi174.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-25T00:03:44.613 INFO:tasks.workunit.client.1.smithi174.stderr: | ^ 2024-04-25T00:03:44.613 INFO:tasks.workunit.client.1.smithi174.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2024-04-25T00:03:44.613 INFO:tasks.workunit.client.1.smithi174.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-25T00:03:44.613 INFO:tasks.workunit.client.1.smithi174.stderr: | ^~~~~~~~ 2024-04-25T00:03:44.613 INFO:tasks.workunit.client.1.smithi174.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2024-04-25T00:03:44.613 INFO:tasks.workunit.client.1.smithi174.stderr: 36 | container_of(ptr, type, member) 2024-04-25T00:03:44.613 INFO:tasks.workunit.client.1.smithi174.stderr: | ^~~~~~~~~~~~ 2024-04-25T00:03:44.614 INFO:tasks.workunit.client.1.smithi174.stderr:list.h:39:20: note: in expansion of macro 'list_entry' 2024-04-25T00:03:44.614 INFO:tasks.workunit.client.1.smithi174.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-04-25T00:03:44.614 INFO:tasks.workunit.client.1.smithi174.stderr: | ^~~~~~~~~~ 2024-04-25T00:03:44.614 INFO:tasks.workunit.client.1.smithi174.stderr:parser.c:941:17: note: in expansion of macro 'list_for_each_entry' 2024-04-25T00:03:44.614 INFO:tasks.workunit.client.1.smithi174.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2024-04-25T00:03:44.614 INFO:tasks.workunit.client.1.smithi174.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-25T00:03:44.614 INFO:tasks.workunit.client.1.smithi174.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-25T00:03:44.614 INFO:tasks.workunit.client.1.smithi174.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-25T00:03:44.614 INFO:tasks.workunit.client.1.smithi174.stderr: | ^ 2024-04-25T00:03:44.614 INFO:tasks.workunit.client.1.smithi174.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2024-04-25T00:03:44.614 INFO:tasks.workunit.client.1.smithi174.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-25T00:03:44.614 INFO:tasks.workunit.client.1.smithi174.stderr: | ^~~~~~~~ 2024-04-25T00:03:44.615 INFO:tasks.workunit.client.1.smithi174.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2024-04-25T00:03:44.615 INFO:tasks.workunit.client.1.smithi174.stderr: 36 | container_of(ptr, type, member) 2024-04-25T00:03:44.615 INFO:tasks.workunit.client.1.smithi174.stderr: | ^~~~~~~~~~~~ 2024-04-25T00:03:44.615 INFO:tasks.workunit.client.1.smithi174.stderr:list.h:41:20: note: in expansion of macro 'list_entry' 2024-04-25T00:03:44.615 INFO:tasks.workunit.client.1.smithi174.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-04-25T00:03:44.615 INFO:tasks.workunit.client.1.smithi174.stderr: | ^~~~~~~~~~ 2024-04-25T00:03:44.615 INFO:tasks.workunit.client.1.smithi174.stderr:parser.c:941:17: note: in expansion of macro 'list_for_each_entry' 2024-04-25T00:03:44.615 INFO:tasks.workunit.client.1.smithi174.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2024-04-25T00:03:44.615 INFO:tasks.workunit.client.1.smithi174.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-25T00:03:44.615 INFO:tasks.workunit.client.1.smithi174.stderr:parser.c: In function 'init_tg_stats': 2024-04-25T00:03:44.615 INFO:tasks.workunit.client.1.smithi174.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-25T00:03:44.615 INFO:tasks.workunit.client.1.smithi174.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-25T00:03:44.615 INFO:tasks.workunit.client.1.smithi174.stderr: | ^ 2024-04-25T00:03:44.615 INFO:tasks.workunit.client.1.smithi174.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2024-04-25T00:03:44.616 INFO:tasks.workunit.client.1.smithi174.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-25T00:03:44.616 INFO:tasks.workunit.client.1.smithi174.stderr: | ^~~~~~~~ 2024-04-25T00:03:44.616 INFO:tasks.workunit.client.1.smithi174.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2024-04-25T00:03:44.616 INFO:tasks.workunit.client.1.smithi174.stderr: 36 | container_of(ptr, type, member) 2024-04-25T00:03:44.616 INFO:tasks.workunit.client.1.smithi174.stderr: | ^~~~~~~~~~~~ 2024-04-25T00:03:44.616 INFO:tasks.workunit.client.1.smithi174.stderr:list.h:39:20: note: in expansion of macro 'list_entry' 2024-04-25T00:03:44.616 INFO:tasks.workunit.client.1.smithi174.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-04-25T00:03:44.616 INFO:tasks.workunit.client.1.smithi174.stderr: | ^~~~~~~~~~ 2024-04-25T00:03:44.616 INFO:tasks.workunit.client.1.smithi174.stderr:parser.c:971:41: note: in expansion of macro 'list_for_each_entry' 2024-04-25T00:03:44.616 INFO:tasks.workunit.client.1.smithi174.stderr: 971 | list_for_each_entry(tmp_list, 2024-04-25T00:03:44.616 INFO:tasks.workunit.client.1.smithi174.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-25T00:03:44.616 INFO:tasks.workunit.client.1.smithi174.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-25T00:03:44.616 INFO:tasks.workunit.client.1.smithi174.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-25T00:03:44.616 INFO:tasks.workunit.client.1.smithi174.stderr: | ^ 2024-04-25T00:03:44.616 INFO:tasks.workunit.client.1.smithi174.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2024-04-25T00:03:44.617 INFO:tasks.workunit.client.1.smithi174.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-25T00:03:44.617 INFO:tasks.workunit.client.1.smithi174.stderr: | ^~~~~~~~ 2024-04-25T00:03:44.617 INFO:tasks.workunit.client.1.smithi174.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2024-04-25T00:03:44.617 INFO:tasks.workunit.client.1.smithi174.stderr: 36 | container_of(ptr, type, member) 2024-04-25T00:03:44.617 INFO:tasks.workunit.client.1.smithi174.stderr: | ^~~~~~~~~~~~ 2024-04-25T00:03:44.617 INFO:tasks.workunit.client.1.smithi174.stderr:list.h:41:20: note: in expansion of macro 'list_entry' 2024-04-25T00:03:44.617 INFO:tasks.workunit.client.1.smithi174.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-04-25T00:03:44.617 INFO:tasks.workunit.client.1.smithi174.stderr: | ^~~~~~~~~~ 2024-04-25T00:03:44.617 INFO:tasks.workunit.client.1.smithi174.stderr:parser.c:971:41: note: in expansion of macro 'list_for_each_entry' 2024-04-25T00:03:44.617 INFO:tasks.workunit.client.1.smithi174.stderr: 971 | list_for_each_entry(tmp_list, 2024-04-25T00:03:44.617 INFO:tasks.workunit.client.1.smithi174.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-25T00:03:44.617 INFO:tasks.workunit.client.1.smithi174.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-25T00:03:44.617 INFO:tasks.workunit.client.1.smithi174.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-25T00:03:44.617 INFO:tasks.workunit.client.1.smithi174.stderr: | ^ 2024-04-25T00:03:44.618 INFO:tasks.workunit.client.1.smithi174.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2024-04-25T00:03:44.618 INFO:tasks.workunit.client.1.smithi174.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-25T00:03:44.618 INFO:tasks.workunit.client.1.smithi174.stderr: | ^~~~~~~~ 2024-04-25T00:03:44.618 INFO:tasks.workunit.client.1.smithi174.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2024-04-25T00:03:44.618 INFO:tasks.workunit.client.1.smithi174.stderr: 36 | container_of(ptr, type, member) 2024-04-25T00:03:44.618 INFO:tasks.workunit.client.1.smithi174.stderr: | ^~~~~~~~~~~~ 2024-04-25T00:03:44.618 INFO:tasks.workunit.client.1.smithi174.stderr:list.h:39:20: note: in expansion of macro 'list_entry' 2024-04-25T00:03:44.618 INFO:tasks.workunit.client.1.smithi174.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-04-25T00:03:44.618 INFO:tasks.workunit.client.1.smithi174.stderr: | ^~~~~~~~~~ 2024-04-25T00:03:44.618 INFO:tasks.workunit.client.1.smithi174.stderr:parser.c:980:41: note: in expansion of macro 'list_for_each_entry' 2024-04-25T00:03:44.618 INFO:tasks.workunit.client.1.smithi174.stderr: 980 | list_for_each_entry(tmp_list, 2024-04-25T00:03:44.618 INFO:tasks.workunit.client.1.smithi174.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-25T00:03:44.618 INFO:tasks.workunit.client.1.smithi174.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-25T00:03:44.619 INFO:tasks.workunit.client.1.smithi174.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-25T00:03:44.619 INFO:tasks.workunit.client.1.smithi174.stderr: | ^ 2024-04-25T00:03:44.619 INFO:tasks.workunit.client.1.smithi174.stderr:list.h:33:35: note: in expansion of macro 'offsetof' 2024-04-25T00:03:44.619 INFO:tasks.workunit.client.1.smithi174.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-25T00:03:44.619 INFO:tasks.workunit.client.1.smithi174.stderr: | ^~~~~~~~ 2024-04-25T00:03:44.619 INFO:tasks.workunit.client.1.smithi174.stderr:list.h:36:9: note: in expansion of macro 'container_of' 2024-04-25T00:03:44.619 INFO:tasks.workunit.client.1.smithi174.stderr: 36 | container_of(ptr, type, member) 2024-04-25T00:03:44.619 INFO:tasks.workunit.client.1.smithi174.stderr: | ^~~~~~~~~~~~ 2024-04-25T00:03:44.619 INFO:tasks.workunit.client.1.smithi174.stderr:list.h:41:20: note: in expansion of macro 'list_entry' 2024-04-25T00:03:44.619 INFO:tasks.workunit.client.1.smithi174.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-04-25T00:03:44.619 INFO:tasks.workunit.client.1.smithi174.stderr: | ^~~~~~~~~~ 2024-04-25T00:03:44.619 INFO:tasks.workunit.client.1.smithi174.stderr:parser.c:980:41: note: in expansion of macro 'list_for_each_entry' 2024-04-25T00:03:44.619 INFO:tasks.workunit.client.1.smithi174.stderr: 980 | list_for_each_entry(tmp_list, 2024-04-25T00:03:44.619 INFO:tasks.workunit.client.1.smithi174.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-25T00:03:44.668 INFO:tasks.workunit.client.1.smithi174.stderr:parser.c: In function 'get_optstr': 2024-04-25T00:03:44.668 INFO:tasks.workunit.client.1.smithi174.stderr:parser.c:206:23: warning: 'strnlen' specified bound 4095 exceeds source size 1024 [-Wstringop-overread] 2024-04-25T00:03:44.668 INFO:tasks.workunit.client.1.smithi174.stderr: 206 | len = strnlen(temp, 4095) + 1; 2024-04-25T00:03:44.668 INFO:tasks.workunit.client.1.smithi174.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-25T00:03:44.668 INFO:tasks.workunit.client.1.smithi174.stderr:parser.c:200:14: note: source object declared here 2024-04-25T00:03:44.668 INFO:tasks.workunit.client.1.smithi174.stderr: 200 | char temp[BUFSIZE]; 2024-04-25T00:03:44.668 INFO:tasks.workunit.client.1.smithi174.stderr: | ^~~~ 2024-04-25T00:03:44.686 INFO:tasks.workunit.client.1.smithi174.stderr:In file included from /usr/include/string.h:535, 2024-04-25T00:03:44.686 INFO:tasks.workunit.client.1.smithi174.stderr: from parser.c:19: 2024-04-25T00:03:44.686 INFO:tasks.workunit.client.1.smithi174.stderr:In function 'memcpy', 2024-04-25T00:03:44.686 INFO:tasks.workunit.client.1.smithi174.stderr: inlined from 'size64_convert' at parser.c:119:2: 2024-04-25T00:03:44.686 INFO:tasks.workunit.client.1.smithi174.stderr:/usr/include/x86_64-linux-gnu/bits/string_fortified.h:29:10: warning: '__builtin_memcpy' reading 3 bytes from a region of size 2 [-Wstringop-overread] 2024-04-25T00:03:44.686 INFO:tasks.workunit.client.1.smithi174.stderr: 29 | return __builtin___memcpy_chk (__dest, __src, __len, 2024-04-25T00:03:44.686 INFO:tasks.workunit.client.1.smithi174.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-25T00:03:44.686 INFO:tasks.workunit.client.1.smithi174.stderr: 30 | __glibc_objsize0 (__dest)); 2024-04-25T00:03:44.687 INFO:tasks.workunit.client.1.smithi174.stderr: | ~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-04-25T00:03:45.033 INFO:tasks.workunit.client.1.smithi174.stdout:mv -f .deps/parser.Tpo .deps/parser.Po 2024-04-25T00:03:45.045 INFO:tasks.workunit.client.1.smithi174.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fc.o -MD -MP -MF .deps/ffsb_fc.Tpo -c -o ffsb_fc.o ffsb_fc.c 2024-04-25T00:03:45.244 INFO:tasks.workunit.client.1.smithi174.stdout:mv -f .deps/ffsb_fc.Tpo .deps/ffsb_fc.Po 2024-04-25T00:03:45.259 INFO:tasks.workunit.client.1.smithi174.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_stats.o -MD -MP -MF .deps/ffsb_stats.Tpo -c -o ffsb_stats.o ffsb_stats.c 2024-04-25T00:03:45.466 INFO:tasks.workunit.client.1.smithi174.stdout:mv -f .deps/ffsb_stats.Tpo .deps/ffsb_stats.Po 2024-04-25T00:03:45.477 INFO:tasks.workunit.client.1.smithi174.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT list.o -MD -MP -MF .deps/list.Tpo -c -o list.o list.c 2024-04-25T00:03:45.559 INFO:tasks.workunit.client.1.smithi174.stdout:mv -f .deps/list.Tpo .deps/list.Po 2024-04-25T00:03:45.573 INFO:tasks.workunit.client.1.smithi174.stdout:gcc -D_GNU_SOURCE -g -O2 -D_REENTRANT -o ffsb fileops.o rand.o main.o fh.o filelist.o metaops.o rwlock.o cirlist.o rbt.o ffsb_tg.o ffsb_fs.o ffsb_thread.o ffsb_op.o util.o parser.o ffsb_fc.o ffsb_stats.o list.o -lpthread -lm 2024-04-25T00:03:45.741 INFO:tasks.workunit.client.1.smithi174.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/mnt.1/client.1/tmp/ffsb' 2024-04-25T00:03:45.742 INFO:tasks.workunit.client.1.smithi174.stderr:+ cd .. 2024-04-25T00:03:45.744 INFO:tasks.workunit.client.1.smithi174.stderr:+ mkdir tmp 2024-04-25T00:03:45.747 INFO:tasks.workunit.client.1.smithi174.stderr:+ cd tmp 2024-04-25T00:03:45.748 INFO:tasks.workunit.client.1.smithi174.stderr:+ for f in $mydir/*.ffsb 2024-04-25T00:03:45.748 INFO:tasks.workunit.client.1.smithi174.stderr:+ ../ffsb/ffsb /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/random_write.32.ffsb 2024-04-25T00:03:45.749 INFO:tasks.workunit.client.1.smithi174.stdout:FFSB version 6.0-RC2 started 2024-04-25T00:03:45.749 INFO:tasks.workunit.client.1.smithi174.stdout: 2024-04-25T00:03:45.753 INFO:tasks.workunit.client.1.smithi174.stdout:benchmark time = 300 2024-04-25T00:03:45.754 INFO:tasks.workunit.client.1.smithi174.stdout:ThreadGroup 0 2024-04-25T00:03:45.754 INFO:tasks.workunit.client.1.smithi174.stdout:================ 2024-04-25T00:03:45.754 INFO:tasks.workunit.client.1.smithi174.stdout: num_threads = 32 2024-04-25T00:03:45.754 INFO:tasks.workunit.client.1.smithi174.stdout: 2024-04-25T00:03:45.754 INFO:tasks.workunit.client.1.smithi174.stdout: read_random = off 2024-04-25T00:03:45.754 INFO:tasks.workunit.client.1.smithi174.stdout: read_size = 0 (0B) 2024-04-25T00:03:45.754 INFO:tasks.workunit.client.1.smithi174.stdout: read_blocksize = 0 (0B) 2024-04-25T00:03:45.754 INFO:tasks.workunit.client.1.smithi174.stdout: read_skip = off 2024-04-25T00:03:45.754 INFO:tasks.workunit.client.1.smithi174.stdout: read_skipsize = 0 (0B) 2024-04-25T00:03:45.754 INFO:tasks.workunit.client.1.smithi174.stdout: 2024-04-25T00:03:45.754 INFO:tasks.workunit.client.1.smithi174.stdout: write_random = on 2024-04-25T00:03:45.754 INFO:tasks.workunit.client.1.smithi174.stdout: write_size = 5242880 (5MB) 2024-04-25T00:03:45.755 INFO:tasks.workunit.client.1.smithi174.stdout: fsync_file = 0 2024-04-25T00:03:45.755 INFO:tasks.workunit.client.1.smithi174.stdout: write_blocksize = 4096 (4KB) 2024-04-25T00:03:45.755 INFO:tasks.workunit.client.1.smithi174.stdout: wait time = 0 2024-04-25T00:03:45.755 INFO:tasks.workunit.client.1.smithi174.stdout: 2024-04-25T00:03:45.755 INFO:tasks.workunit.client.1.smithi174.stdout: op weights 2024-04-25T00:03:45.755 INFO:tasks.workunit.client.1.smithi174.stdout: read = 0 (0.00%) 2024-04-25T00:03:45.755 INFO:tasks.workunit.client.1.smithi174.stdout: readall = 0 (0.00%) 2024-04-25T00:03:45.755 INFO:tasks.workunit.client.1.smithi174.stdout: write = 1 (100.00%) 2024-04-25T00:03:45.755 INFO:tasks.workunit.client.1.smithi174.stdout: create = 0 (0.00%) 2024-04-25T00:03:45.755 INFO:tasks.workunit.client.1.smithi174.stdout: append = 0 (0.00%) 2024-04-25T00:03:45.755 INFO:tasks.workunit.client.1.smithi174.stdout: delete = 0 (0.00%) 2024-04-25T00:03:45.755 INFO:tasks.workunit.client.1.smithi174.stdout: metaop = 0 (0.00%) 2024-04-25T00:03:45.755 INFO:tasks.workunit.client.1.smithi174.stdout: createdir = 0 (0.00%) 2024-04-25T00:03:45.755 INFO:tasks.workunit.client.1.smithi174.stdout: stat = 0 (0.00%) 2024-04-25T00:03:45.756 INFO:tasks.workunit.client.1.smithi174.stdout: writeall = 0 (0.00%) 2024-04-25T00:03:45.756 INFO:tasks.workunit.client.1.smithi174.stdout: writeall_fsync = 0 (0.00%) 2024-04-25T00:03:45.756 INFO:tasks.workunit.client.1.smithi174.stdout: open_close = 0 (0.00%) 2024-04-25T00:03:45.756 INFO:tasks.workunit.client.1.smithi174.stdout: write_fsync = 0 (0.00%) 2024-04-25T00:03:45.756 INFO:tasks.workunit.client.1.smithi174.stdout: create_fsync = 0 (0.00%) 2024-04-25T00:03:45.756 INFO:tasks.workunit.client.1.smithi174.stdout: append_fsync = 0 (0.00%) 2024-04-25T00:03:45.756 INFO:tasks.workunit.client.1.smithi174.stdout: 2024-04-25T00:03:45.756 INFO:tasks.workunit.client.1.smithi174.stdout:FileSystem . 2024-04-25T00:03:45.756 INFO:tasks.workunit.client.1.smithi174.stdout:========== 2024-04-25T00:03:45.756 INFO:tasks.workunit.client.1.smithi174.stdout: num_dirs = 0 2024-04-25T00:03:45.756 INFO:tasks.workunit.client.1.smithi174.stdout: starting files = 128 2024-04-25T00:03:45.756 INFO:tasks.workunit.client.1.smithi174.stdout: 2024-04-25T00:03:45.756 INFO:tasks.workunit.client.1.smithi174.stdout: min file size = 104857600 (100MB) 2024-04-25T00:03:45.757 INFO:tasks.workunit.client.1.smithi174.stdout: max file size = 104857600 (100MB) 2024-04-25T00:03:45.757 INFO:tasks.workunit.client.1.smithi174.stdout: directio = off 2024-04-25T00:03:45.757 INFO:tasks.workunit.client.1.smithi174.stdout: alignedio = on 2024-04-25T00:03:45.757 INFO:tasks.workunit.client.1.smithi174.stdout: bufferedio = off 2024-04-25T00:03:45.757 INFO:tasks.workunit.client.1.smithi174.stdout: 2024-04-25T00:03:45.757 INFO:tasks.workunit.client.1.smithi174.stdout: aging is off 2024-04-25T00:03:45.757 INFO:tasks.workunit.client.1.smithi174.stdout: current utilization = 0.00% 2024-04-25T00:03:45.758 INFO:tasks.workunit.client.1.smithi174.stdout: 2024-04-25T00:03:45.758 INFO:tasks.workunit.client.1.smithi174.stdout:checking existing fs: . 2024-04-25T00:03:45.758 INFO:tasks.workunit.client.1.smithi174.stdout:recreating new fileset 2024-04-25T00:03:45.758 INFO:tasks.workunit.client.1.smithi174.stderr:opendir: No such file or directory 2024-04-25T00:03:47.686 INFO:tasks.mds_thrash.fs.[a]:reviving mds.b 2024-04-25T00:03:47.687 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-25T00:03:47.687 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-25T00:03:47.690 INFO:tasks.ceph.mds.b:Started 2024-04-25T00:03:47.690 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:03:47.769 INFO:tasks.ceph.mds.b.smithi174.stdout:starting mds.b at 2024-04-25T00:03:47.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:47.819+0000 7f3b824ad640 1 Processor -- start 2024-04-25T00:03:47.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:47.819+0000 7f3b824ad640 1 -- start start 2024-04-25T00:03:47.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:47.819+0000 7f3b824ad640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3b7c14c700 0x7f3b7c0c5090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:47.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:47.819+0000 7f3b824ad640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b7c0a9e60 con 0x7f3b7c0c55d0 2024-04-25T00:03:47.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:47.819+0000 7f3b824ad640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b7c0c3ef0 con 0x7f3b7c155b20 2024-04-25T00:03:47.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:47.819+0000 7f3b824ad640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f3b7c1527b0 con 0x7f3b7c14c700 2024-04-25T00:03:47.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:47.819+0000 7f3b80a4a640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f3b7c0c55d0 0x7f3b7c15ceb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:58250/0 (socket says 172.21.15.116:58250) 2024-04-25T00:03:47.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:47.819+0000 7f3b7bfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3b7c14c700 0x7f3b7c0c5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:47.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:47.819+0000 7f3b80a4a640 1 -- 172.21.15.116:0/4173828817 learned_addr learned my addr 172.21.15.116:0/4173828817 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:03:47.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:47.819+0000 7f3b7b7fe640 1 --1- 172.21.15.116:0/4173828817 >> v1:172.21.15.174:6789/0 conn(0x7f3b7c155b20 0x7f3b7c152c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:42082/0 (socket says 172.21.15.116:42082) 2024-04-25T00:03:47.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:47.819+0000 7f3b7bfff640 1 -- 172.21.15.116:0/4173828817 >> v1:172.21.15.174:6789/0 conn(0x7f3b7c155b20 legacy=0x7f3b7c152c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:47.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:47.819+0000 7f3b7affd640 1 -- 172.21.15.116:0/4173828817 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1380809068 0 0) 0x7f3b7c0c3ef0 con 0x7f3b7c155b20 2024-04-25T00:03:47.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:47.819+0000 7f3b7bfff640 1 -- 172.21.15.116:0/4173828817 >> v1:172.21.15.116:6789/0 conn(0x7f3b7c0c55d0 legacy=0x7f3b7c15ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:47.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:47.819+0000 7f3b7bfff640 1 -- 172.21.15.116:0/4173828817 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b7c0c41d0 con 0x7f3b7c14c700 2024-04-25T00:03:47.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:47.819+0000 7f3b7bfff640 1 --2- 172.21.15.116:0/4173828817 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b7c14c700 0x7f3b7c0c5090 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c0097c0 tx=0x7f3b6c01bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7abd2f0e49f05d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:47.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:47.819+0000 7f3b7affd640 1 -- 172.21.15.116:0/4173828817 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b6c0af070 con 0x7f3b7c14c700 2024-04-25T00:03:47.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:47.819+0000 7f3b7affd640 1 -- 172.21.15.116:0/4173828817 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b6c004230 con 0x7f3b7c14c700 2024-04-25T00:03:47.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:47.819+0000 7f3b7affd640 1 -- 172.21.15.116:0/4173828817 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b6c003620 con 0x7f3b7c14c700 2024-04-25T00:03:47.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:47.819+0000 7f3b824ad640 1 -- 172.21.15.116:0/4173828817 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b7c14c700 msgr2=0x7f3b7c0c5090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:47.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:47.819+0000 7f3b824ad640 1 --2- 172.21.15.116:0/4173828817 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b7c14c700 0x7f3b7c0c5090 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c0097c0 tx=0x7f3b6c01bf80 comp rx=0 tx=0).stop 2024-04-25T00:03:47.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:47.819+0000 7f3b824ad640 1 -- 172.21.15.116:0/4173828817 shutdown_connections 2024-04-25T00:03:47.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:47.819+0000 7f3b824ad640 1 --2- 172.21.15.116:0/4173828817 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b7c14c700 0x7f3b7c0c5090 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:47.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:47.819+0000 7f3b824ad640 1 -- 172.21.15.116:0/4173828817 >> 172.21.15.116:0/4173828817 conn(0x7f3b7c06c720 msgr2=0x7f3b7c06cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:03:47.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:47.819+0000 7f3b824ad640 1 -- 172.21.15.116:0/4173828817 shutdown_connections 2024-04-25T00:03:47.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:47.819+0000 7f3b824ad640 1 -- 172.21.15.116:0/4173828817 wait complete. 2024-04-25T00:03:47.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:47.823+0000 7f3b824ad640 1 Processor -- start 2024-04-25T00:03:47.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:47.823+0000 7f3b824ad640 1 -- start start 2024-04-25T00:03:47.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:47.823+0000 7f3b824ad640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3b7c0c55d0 0x7f3b7c0c1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:47.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:47.823+0000 7f3b824ad640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3b7c14c700 0x7f3b7c0c1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:47.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:47.823+0000 7f3b824ad640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b7c155b20 0x7f3b7c1487e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:47.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:47.823+0000 7f3b824ad640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3b7c1619a0 con 0x7f3b7c155b20 2024-04-25T00:03:47.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:47.823+0000 7f3b824ad640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3b7c161820 con 0x7f3b7c0c55d0 2024-04-25T00:03:47.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:47.823+0000 7f3b824ad640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f3b7c161b20 con 0x7f3b7c14c700 2024-04-25T00:03:47.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:47.823+0000 7f3b80a4a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b7c155b20 0x7f3b7c1487e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:47.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:47.823+0000 7f3b80a4a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b7c155b20 0x7f3b7c1487e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60346/0 (socket says 172.21.15.116:60346) 2024-04-25T00:03:47.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:47.823+0000 7f3b7bfff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3b7c0c55d0 0x7f3b7c0c1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:47.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:47.823+0000 7f3b80a4a640 1 -- 172.21.15.116:0/3698223568 learned_addr learned my addr 172.21.15.116:0/3698223568 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:03:47.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:47.823+0000 7f3b7b7fe640 1 --2- 172.21.15.116:0/3698223568 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3b7c14c700 0x7f3b7c0c1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:47.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:47.823+0000 7f3b80a4a640 1 -- 172.21.15.116:0/3698223568 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3b7c14c700 msgr2=0x7f3b7c0c1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:47.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:47.823+0000 7f3b80a4a640 1 --2- 172.21.15.116:0/3698223568 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3b7c14c700 0x7f3b7c0c1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:47.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:47.823+0000 7f3b80a4a640 1 -- 172.21.15.116:0/3698223568 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3b7c0c55d0 msgr2=0x7f3b7c0c1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:47.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:47.823+0000 7f3b80a4a640 1 --2- 172.21.15.116:0/3698223568 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3b7c0c55d0 0x7f3b7c0c1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:47.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:47.823+0000 7f3b80a4a640 1 -- 172.21.15.116:0/3698223568 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b7c14d120 con 0x7f3b7c155b20 2024-04-25T00:03:47.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:47.823+0000 7f3b80a4a640 1 --2- 172.21.15.116:0/3698223568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b7c155b20 0x7f3b7c1487e0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f3b7400caf0 tx=0x7f3b74002080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:47.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:47.823+0000 7f3b78ff9640 1 -- 172.21.15.116:0/3698223568 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b74018070 con 0x7f3b7c155b20 2024-04-25T00:03:47.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:47.823+0000 7f3b78ff9640 1 -- 172.21.15.116:0/3698223568 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b7400a700 con 0x7f3b7c155b20 2024-04-25T00:03:47.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:47.823+0000 7f3b824ad640 1 -- 172.21.15.116:0/3698223568 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b7c0c21b0 con 0x7f3b7c155b20 2024-04-25T00:03:47.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:47.823+0000 7f3b78ff9640 1 -- 172.21.15.116:0/3698223568 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b7400ab30 con 0x7f3b7c155b20 2024-04-25T00:03:47.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:47.823+0000 7f3b824ad640 1 -- 172.21.15.116:0/3698223568 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b7c14ce40 con 0x7f3b7c155b20 2024-04-25T00:03:47.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:47.827+0000 7f3b824ad640 1 -- 172.21.15.116:0/3698223568 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b7c0ab350 con 0x7f3b7c155b20 2024-04-25T00:03:47.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:47.827+0000 7f3b78ff9640 1 -- 172.21.15.116:0/3698223568 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3b7401b070 con 0x7f3b7c155b20 2024-04-25T00:03:47.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:47.827+0000 7f3b78ff9640 1 --2- 172.21.15.116:0/3698223568 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3b680678c0 0x7f3b68069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:47.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:47.827+0000 7f3b78ff9640 1 -- 172.21.15.116:0/3698223568 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 6924+0+0 (secure 0 0 0) 0x7f3b740857d0 con 0x7f3b7c155b20 2024-04-25T00:03:47.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:47.827+0000 7f3b7bfff640 1 --2- 172.21.15.116:0/3698223568 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3b680678c0 0x7f3b68069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:47.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:47.827+0000 7f3b7bfff640 1 --2- 172.21.15.116:0/3698223568 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3b680678c0 0x7f3b68069d80 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c0ac000 tx=0x7f3b6c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:47.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:47.831+0000 7f3b78ff9640 1 -- 172.21.15.116:0/3698223568 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f3b7c0c21b0 con 0x7f3b7c155b20 2024-04-25T00:03:48.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:48.071+0000 7f3b824ad640 1 -- 172.21.15.116:0/3698223568 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3b7c068a60 con 0x7f3b7c155b20 2024-04-25T00:03:48.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:48.071+0000 7f3b78ff9640 1 -- 172.21.15.116:0/3698223568 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+5673 (secure 0 0 0) 0x7f3b7c14d120 con 0x7f3b7c155b20 2024-04-25T00:03:48.077 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:03:48.078 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:03:40.629360+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4828":{"gid":4828,"name":"c","rank":0,"incarnation":26,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6834/1649769369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1649769369},{"type":"v1","addr":"172.21.15.116:6834","nonce":1649769369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4828,"qdb_cluster":[4828]},"id":1},{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:03:34.579825+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4735},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4735":{"gid":4735,"name":"a","rank":0,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.116:6835/2004876235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":2004876235},{"type":"v1","addr":"172.21.15.116:6835","nonce":2004876235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4735,"qdb_cluster":[4735]},"id":2}]} 2024-04-25T00:03:48.078 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 30 2024-04-25T00:03:48.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:48.075+0000 7f3b824ad640 1 -- 172.21.15.116:0/3698223568 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3b680678c0 msgr2=0x7f3b68069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:48.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:48.075+0000 7f3b824ad640 1 --2- 172.21.15.116:0/3698223568 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3b680678c0 0x7f3b68069d80 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c0ac000 tx=0x7f3b6c008040 comp rx=0 tx=0).stop 2024-04-25T00:03:48.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:48.075+0000 7f3b824ad640 1 -- 172.21.15.116:0/3698223568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b7c155b20 msgr2=0x7f3b7c1487e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:48.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:48.075+0000 7f3b824ad640 1 --2- 172.21.15.116:0/3698223568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b7c155b20 0x7f3b7c1487e0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f3b7400caf0 tx=0x7f3b74002080 comp rx=0 tx=0).stop 2024-04-25T00:03:48.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:48.075+0000 7f3b824ad640 1 -- 172.21.15.116:0/3698223568 shutdown_connections 2024-04-25T00:03:48.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:48.075+0000 7f3b824ad640 1 --2- 172.21.15.116:0/3698223568 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3b680678c0 0x7f3b68069d80 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:48.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:48.075+0000 7f3b824ad640 1 --2- 172.21.15.116:0/3698223568 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b7c155b20 0x7f3b7c1487e0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:48.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:48.075+0000 7f3b824ad640 1 --2- 172.21.15.116:0/3698223568 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3b7c14c700 0x7f3b7c0c1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:48.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:48.075+0000 7f3b824ad640 1 --2- 172.21.15.116:0/3698223568 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3b7c0c55d0 0x7f3b7c0c1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:48.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:48.075+0000 7f3b824ad640 1 -- 172.21.15.116:0/3698223568 >> 172.21.15.116:0/3698223568 conn(0x7f3b7c06c720 msgr2=0x7f3b7c151a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:03:48.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:48.075+0000 7f3b824ad640 1 -- 172.21.15.116:0/3698223568 shutdown_connections 2024-04-25T00:03:48.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:48.075+0000 7f3b824ad640 1 -- 172.21.15.116:0/3698223568 wait complete. 2024-04-25T00:03:48.100 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2024-04-25T00:03:50.102 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:03:50.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.251+0000 7f9df6069640 1 Processor -- start 2024-04-25T00:03:50.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.251+0000 7f9df6069640 1 -- start start 2024-04-25T00:03:50.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.255+0000 7f9df6069640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f9df0155a70 0x7f9df0155ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:50.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.255+0000 7f9df6069640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9df014fe70 con 0x7f9df014fa90 2024-04-25T00:03:50.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.255+0000 7f9df6069640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9df00c4770 con 0x7f9df0072d70 2024-04-25T00:03:50.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.255+0000 7f9df6069640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f9df00c3570 con 0x7f9df0155a70 2024-04-25T00:03:50.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.255+0000 7f9def7fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f9df014fa90 0x7f9df0153970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:58254/0 (socket says 172.21.15.116:58254) 2024-04-25T00:03:50.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.255+0000 7f9def7fe640 1 -- 172.21.15.116:0/376329929 learned_addr learned my addr 172.21.15.116:0/376329929 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:03:50.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.255+0000 7f9deffff640 1 --2- 172.21.15.116:0/376329929 >> v2:172.21.15.174:3300/0 conn(0x7f9df0155a70 0x7f9df0155ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:50.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.255+0000 7f9deeffd640 1 -- 172.21.15.116:0/376329929 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 561777759 0 0) 0x7f9df014fe70 con 0x7f9df014fa90 2024-04-25T00:03:50.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.255+0000 7f9deeffd640 1 -- 172.21.15.116:0/376329929 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ddc003530 con 0x7f9df014fa90 2024-04-25T00:03:50.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.255+0000 7f9deeffd640 1 -- 172.21.15.116:0/376329929 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2506293119 0 0) 0x7f9df00c4770 con 0x7f9df0072d70 2024-04-25T00:03:50.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.255+0000 7f9deeffd640 1 -- 172.21.15.116:0/376329929 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9df014fe70 con 0x7f9df0072d70 2024-04-25T00:03:50.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.255+0000 7f9deeffd640 1 -- 172.21.15.116:0/376329929 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2682957994 0 0) 0x7f9ddc003530 con 0x7f9df014fa90 2024-04-25T00:03:50.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.255+0000 7f9deeffd640 1 -- 172.21.15.116:0/376329929 >> v2:172.21.15.174:3300/0 conn(0x7f9df0155a70 msgr2=0x7f9df0155ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:03:50.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.255+0000 7f9deeffd640 1 --2- 172.21.15.116:0/376329929 >> v2:172.21.15.174:3300/0 conn(0x7f9df0155a70 0x7f9df0155ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:50.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.255+0000 7f9deeffd640 1 -- 172.21.15.116:0/376329929 >> v1:172.21.15.174:6789/0 conn(0x7f9df0072d70 legacy=0x7f9df0154080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:50.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.255+0000 7f9deeffd640 1 -- 172.21.15.116:0/376329929 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9df00c3850 con 0x7f9df014fa90 2024-04-25T00:03:50.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.255+0000 7f9deeffd640 1 -- 172.21.15.116:0/376329929 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f9de0001680 con 0x7f9df014fa90 2024-04-25T00:03:50.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.255+0000 7f9deeffd640 1 -- 172.21.15.116:0/376329929 >> v1:172.21.15.116:6789/0 conn(0x7f9df014fa90 legacy=0x7f9df0153970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:50.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.255+0000 7f9deeffd640 1 --2- 172.21.15.116:0/376329929 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ddc003ba0 0x7f9ddc003fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:50.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.255+0000 7f9deeffd640 1 --2- 172.21.15.116:0/376329929 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9ddc004810 0x7f9ddc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:50.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.255+0000 7f9deeffd640 1 --2- 172.21.15.116:0/376329929 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ddc005290 0x7f9ddc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:50.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.255+0000 7f9deeffd640 1 -- 172.21.15.116:0/376329929 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9df00c3570 con 0x7f9ddc005290 2024-04-25T00:03:50.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.255+0000 7f9deeffd640 1 -- 172.21.15.116:0/376329929 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9ddc00c1d0 con 0x7f9ddc003ba0 2024-04-25T00:03:50.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.255+0000 7f9deeffd640 1 -- 172.21.15.116:0/376329929 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f9ddc00c3d0 con 0x7f9ddc004810 2024-04-25T00:03:50.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.255+0000 7f9deffff640 1 --2- 172.21.15.116:0/376329929 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ddc005290 0x7f9ddc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:50.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.255+0000 7f9def7fe640 1 --2- 172.21.15.116:0/376329929 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ddc003ba0 0x7f9ddc003fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:50.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.255+0000 7f9deffff640 1 -- 172.21.15.116:0/376329929 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9ddc004810 msgr2=0x7f9ddc004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:03:50.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.255+0000 7f9de7fff640 1 --2- 172.21.15.116:0/376329929 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9ddc004810 0x7f9ddc004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:50.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.255+0000 7f9deffff640 1 --2- 172.21.15.116:0/376329929 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9ddc004810 0x7f9ddc004c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:50.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.255+0000 7f9deffff640 1 -- 172.21.15.116:0/376329929 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ddc003ba0 msgr2=0x7f9ddc003fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:03:50.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.255+0000 7f9deffff640 1 --2- 172.21.15.116:0/376329929 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ddc003ba0 0x7f9ddc003fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:50.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.255+0000 7f9deffff640 1 -- 172.21.15.116:0/376329929 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ddc003530 con 0x7f9ddc005290 2024-04-25T00:03:50.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.259+0000 7f9deffff640 1 -- 172.21.15.116:0/376329929 reap_dead start 2024-04-25T00:03:50.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.259+0000 7f9deffff640 1 --2- 172.21.15.116:0/376329929 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ddc005290 0x7f9ddc00b9d0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f9de8009fc0 tx=0x7f9de801ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=463782f434100527 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:50.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.259+0000 7f9deeffd640 1 -- 172.21.15.116:0/376329929 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de8009c10 con 0x7f9ddc005290 2024-04-25T00:03:50.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.259+0000 7f9deeffd640 1 -- 172.21.15.116:0/376329929 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9de8003ba0 con 0x7f9ddc005290 2024-04-25T00:03:50.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.259+0000 7f9deeffd640 1 -- 172.21.15.116:0/376329929 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de80a58c0 con 0x7f9ddc005290 2024-04-25T00:03:50.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.259+0000 7f9df6069640 1 -- 172.21.15.116:0/376329929 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ddc005290 msgr2=0x7f9ddc00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:50.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.259+0000 7f9df6069640 1 --2- 172.21.15.116:0/376329929 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ddc005290 0x7f9ddc00b9d0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f9de8009fc0 tx=0x7f9de801ccc0 comp rx=0 tx=0).stop 2024-04-25T00:03:50.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.259+0000 7f9df6069640 1 -- 172.21.15.116:0/376329929 shutdown_connections 2024-04-25T00:03:50.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.259+0000 7f9df6069640 1 --2- 172.21.15.116:0/376329929 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ddc005290 0x7f9ddc00b9d0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:50.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.259+0000 7f9df6069640 1 -- 172.21.15.116:0/376329929 >> 172.21.15.116:0/376329929 conn(0x7f9df00c4aa0 msgr2=0x7f9df00c4eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:03:50.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.259+0000 7f9df6069640 1 -- 172.21.15.116:0/376329929 shutdown_connections 2024-04-25T00:03:50.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.259+0000 7f9df6069640 1 -- 172.21.15.116:0/376329929 wait complete. 2024-04-25T00:03:50.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.259+0000 7f9df6069640 1 Processor -- start 2024-04-25T00:03:50.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.259+0000 7f9df6069640 1 -- start start 2024-04-25T00:03:50.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.259+0000 7f9df6069640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ddc005290 0x7f9df0145e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:50.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.259+0000 7f9df6069640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9df01494c0 0x7f9df00c7480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:50.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.259+0000 7f9df6069640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9df00ab5d0 0x7f9df00aba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:50.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.259+0000 7f9df6069640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9df0155e80 con 0x7f9ddc005290 2024-04-25T00:03:50.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.259+0000 7f9df6069640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9df0156a20 con 0x7f9df00ab5d0 2024-04-25T00:03:50.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.259+0000 7f9df6069640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f9df0156000 con 0x7f9df01494c0 2024-04-25T00:03:50.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.259+0000 7f9def7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ddc005290 0x7f9df0145e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:50.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.259+0000 7f9def7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ddc005290 0x7f9df0145e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60372/0 (socket says 172.21.15.116:60372) 2024-04-25T00:03:50.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.259+0000 7f9def7fe640 1 -- 172.21.15.116:0/2071122835 learned_addr learned my addr 172.21.15.116:0/2071122835 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:03:50.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.259+0000 7f9deffff640 1 --2- 172.21.15.116:0/2071122835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9df00ab5d0 0x7f9df00aba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:50.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.259+0000 7f9de7fff640 1 --2- 172.21.15.116:0/2071122835 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9df01494c0 0x7f9df00c7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:50.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.259+0000 7f9def7fe640 1 -- 172.21.15.116:0/2071122835 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9df01494c0 msgr2=0x7f9df00c7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:50.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.259+0000 7f9def7fe640 1 --2- 172.21.15.116:0/2071122835 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9df01494c0 0x7f9df00c7480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:50.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.259+0000 7f9def7fe640 1 -- 172.21.15.116:0/2071122835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9df00ab5d0 msgr2=0x7f9df00aba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:50.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.259+0000 7f9def7fe640 1 --2- 172.21.15.116:0/2071122835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9df00ab5d0 0x7f9df00aba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:50.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.259+0000 7f9def7fe640 1 -- 172.21.15.116:0/2071122835 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9df01ea160 con 0x7f9ddc005290 2024-04-25T00:03:50.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.259+0000 7f9def7fe640 1 --2- 172.21.15.116:0/2071122835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ddc005290 0x7f9df0145e80 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f9de0006070 tx=0x7f9de0006530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:50.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.259+0000 7f9decff9640 1 -- 172.21.15.116:0/2071122835 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de0019020 con 0x7f9ddc005290 2024-04-25T00:03:50.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.259+0000 7f9decff9640 1 -- 172.21.15.116:0/2071122835 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9de0011960 con 0x7f9ddc005290 2024-04-25T00:03:50.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.259+0000 7f9df6069640 1 -- 172.21.15.116:0/2071122835 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9df01ea440 con 0x7f9ddc005290 2024-04-25T00:03:50.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.259+0000 7f9decff9640 1 -- 172.21.15.116:0/2071122835 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de0011d30 con 0x7f9ddc005290 2024-04-25T00:03:50.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.263+0000 7f9df6069640 1 -- 172.21.15.116:0/2071122835 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9df01eaa00 con 0x7f9ddc005290 2024-04-25T00:03:50.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.263+0000 7f9de5ffb640 1 -- 172.21.15.116:0/2071122835 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9dc80052a0 con 0x7f9ddc005290 2024-04-25T00:03:50.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.263+0000 7f9decff9640 1 -- 172.21.15.116:0/2071122835 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f9de001c020 con 0x7f9ddc005290 2024-04-25T00:03:50.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.263+0000 7f9decff9640 1 --2- 172.21.15.116:0/2071122835 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9dd00678c0 0x7f9dd0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:03:50.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.263+0000 7f9decff9640 1 -- 172.21.15.116:0/2071122835 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 6924+0+0 (secure 0 0 0) 0x7f9de0088380 con 0x7f9ddc005290 2024-04-25T00:03:50.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.263+0000 7f9de7fff640 1 --2- 172.21.15.116:0/2071122835 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9dd00678c0 0x7f9dd0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:03:50.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.263+0000 7f9de7fff640 1 --2- 172.21.15.116:0/2071122835 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9dd00678c0 0x7f9dd0069d80 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9dd8000c10 tx=0x7f9dd8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:03:50.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.271+0000 7f9decff9640 1 -- 172.21.15.116:0/2071122835 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f9de0023020 con 0x7f9ddc005290 2024-04-25T00:03:50.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.547+0000 7f9de5ffb640 1 -- 172.21.15.116:0/2071122835 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9dc8005590 con 0x7f9ddc005290 2024-04-25T00:03:50.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.547+0000 7f9decff9640 1 -- 172.21.15.116:0/2071122835 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6501 (secure 0 0 0) 0x7f9de0058310 con 0x7f9ddc005290 2024-04-25T00:03:50.547 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:03:50.547 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4879,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/1889314958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1889314958},{"type":"v1","addr":"172.21.15.174:6837","nonce":1889314958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:03:40.629360+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4828":{"gid":4828,"name":"c","rank":0,"incarnation":26,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6834/1649769369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1649769369},{"type":"v1","addr":"172.21.15.116:6834","nonce":1649769369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4828,"qdb_cluster":[4828]},"id":1},{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:03:34.579825+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4735},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4735":{"gid":4735,"name":"a","rank":0,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.116:6835/2004876235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":2004876235},{"type":"v1","addr":"172.21.15.116:6835","nonce":2004876235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4735,"qdb_cluster":[4735]},"id":2}]} 2024-04-25T00:03:50.548 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 31 2024-04-25T00:03:50.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.547+0000 7f9de5ffb640 1 -- 172.21.15.116:0/2071122835 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9dd00678c0 msgr2=0x7f9dd0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:50.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.547+0000 7f9de5ffb640 1 --2- 172.21.15.116:0/2071122835 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9dd00678c0 0x7f9dd0069d80 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9dd8000c10 tx=0x7f9dd8009040 comp rx=0 tx=0).stop 2024-04-25T00:03:50.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.547+0000 7f9de5ffb640 1 -- 172.21.15.116:0/2071122835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ddc005290 msgr2=0x7f9df0145e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:03:50.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.547+0000 7f9de5ffb640 1 --2- 172.21.15.116:0/2071122835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ddc005290 0x7f9df0145e80 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f9de0006070 tx=0x7f9de0006530 comp rx=0 tx=0).stop 2024-04-25T00:03:50.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.547+0000 7f9de5ffb640 1 -- 172.21.15.116:0/2071122835 shutdown_connections 2024-04-25T00:03:50.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.547+0000 7f9de5ffb640 1 --2- 172.21.15.116:0/2071122835 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9dd00678c0 0x7f9dd0069d80 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:50.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.547+0000 7f9de5ffb640 1 --2- 172.21.15.116:0/2071122835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9df00ab5d0 0x7f9df00aba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:50.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.547+0000 7f9de5ffb640 1 --2- 172.21.15.116:0/2071122835 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9df01494c0 0x7f9df00c7480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:50.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.547+0000 7f9de5ffb640 1 --2- 172.21.15.116:0/2071122835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ddc005290 0x7f9df0145e80 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:03:50.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.547+0000 7f9de5ffb640 1 -- 172.21.15.116:0/2071122835 >> 172.21.15.116:0/2071122835 conn(0x7f9df00c4aa0 msgr2=0x7f9df00c2880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:03:50.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.547+0000 7f9de5ffb640 1 -- 172.21.15.116:0/2071122835 shutdown_connections 2024-04-25T00:03:50.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:03:50.547+0000 7f9de5ffb640 1 -- 172.21.15.116:0/2071122835 wait complete. 2024-04-25T00:03:50.568 INFO:tasks.mds_thrash.fs.[a]:mds.b reported in up:standby state 2024-04-25T00:03:50.568 INFO:tasks.mds_thrash.fs.[a]:waiting for 26 secs before thrashing 2024-04-25T00:04:04.892 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:04:04.896 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:04:16.569 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:04:16.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.715+0000 7fb76759e640 1 Processor -- start 2024-04-25T00:04:16.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.715+0000 7fb76759e640 1 -- start start 2024-04-25T00:04:16.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.715+0000 7fb76759e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb7680c7200 0x7fb7680a7e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:16.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.715+0000 7fb76759e640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb76806e430 con 0x7fb7680c8440 2024-04-25T00:04:16.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.715+0000 7fb76759e640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7680c3a20 con 0x7fb7681590d0 2024-04-25T00:04:16.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.715+0000 7fb76759e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fb7680c4750 con 0x7fb7680c7200 2024-04-25T00:04:16.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.715+0000 7fb76659c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fb7680c8440 0x7fb768156200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:47824/0 (socket says 172.21.15.116:47824) 2024-04-25T00:04:16.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.715+0000 7fb76659c640 1 -- 172.21.15.116:0/3209210295 learned_addr learned my addr 172.21.15.116:0/3209210295 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:04:16.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.715+0000 7fb765d9b640 1 --2- 172.21.15.116:0/3209210295 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb7680c7200 0x7fb7680a7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:16.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.715+0000 7fb76559a640 1 -- 172.21.15.116:0/3209210295 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2329563113 0 0) 0x7fb76806e430 con 0x7fb7680c8440 2024-04-25T00:04:16.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.715+0000 7fb76559a640 1 -- 172.21.15.116:0/3209210295 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb750003530 con 0x7fb7680c8440 2024-04-25T00:04:16.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.715+0000 7fb76559a640 1 -- 172.21.15.116:0/3209210295 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4045628366 0 0) 0x7fb750003530 con 0x7fb7680c8440 2024-04-25T00:04:16.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.715+0000 7fb76559a640 1 -- 172.21.15.116:0/3209210295 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb7680c7200 msgr2=0x7fb7680a7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:04:16.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.715+0000 7fb76559a640 1 --2- 172.21.15.116:0/3209210295 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb7680c7200 0x7fb7680a7e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:16.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.715+0000 7fb76559a640 1 -- 172.21.15.116:0/3209210295 >> v1:172.21.15.174:6789/0 conn(0x7fb7681590d0 legacy=0x7fb768159580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:16.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.715+0000 7fb76559a640 1 -- 172.21.15.116:0/3209210295 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7680c3d00 con 0x7fb7680c8440 2024-04-25T00:04:16.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.715+0000 7fb76559a640 1 -- 172.21.15.116:0/3209210295 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7fb758001680 con 0x7fb7680c8440 2024-04-25T00:04:16.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.715+0000 7fb76559a640 1 -- 172.21.15.116:0/3209210295 >> v1:172.21.15.116:6789/0 conn(0x7fb7680c8440 legacy=0x7fb768156200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:16.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.715+0000 7fb76559a640 1 --2- 172.21.15.116:0/3209210295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7500039f0 0x7fb750003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:16.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.715+0000 7fb76559a640 1 --2- 172.21.15.116:0/3209210295 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb7680c7200 0x7fb750004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:16.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.715+0000 7fb76559a640 1 --2- 172.21.15.116:0/3209210295 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb750004d50 0x7fb7500051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:16.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.715+0000 7fb76559a640 1 -- 172.21.15.116:0/3209210295 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb7680c4750 con 0x7fb7500039f0 2024-04-25T00:04:16.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.715+0000 7fb76559a640 1 -- 172.21.15.116:0/3209210295 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb75000bb00 con 0x7fb750004d50 2024-04-25T00:04:16.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.715+0000 7fb76559a640 1 -- 172.21.15.116:0/3209210295 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fb75000bd00 con 0x7fb7680c7200 2024-04-25T00:04:16.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.715+0000 7fb76659c640 1 --2- 172.21.15.116:0/3209210295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7500039f0 0x7fb750003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:16.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.715+0000 7fb766d9d640 1 --2- 172.21.15.116:0/3209210295 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb750004d50 0x7fb7500051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:16.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.715+0000 7fb76659c640 1 -- 172.21.15.116:0/3209210295 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb7680c7200 msgr2=0x7fb750004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:04:16.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.715+0000 7fb765d9b640 1 --2- 172.21.15.116:0/3209210295 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb7680c7200 0x7fb750004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:16.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.715+0000 7fb76659c640 1 --2- 172.21.15.116:0/3209210295 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb7680c7200 0x7fb750004810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:16.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.715+0000 7fb76659c640 1 -- 172.21.15.116:0/3209210295 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb750004d50 msgr2=0x7fb7500051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:04:16.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.715+0000 7fb76659c640 1 --2- 172.21.15.116:0/3209210295 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb750004d50 0x7fb7500051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:16.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.715+0000 7fb76659c640 1 -- 172.21.15.116:0/3209210295 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb750003530 con 0x7fb7500039f0 2024-04-25T00:04:16.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.715+0000 7fb766d9d640 1 --2- 172.21.15.116:0/3209210295 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb750004d50 0x7fb7500051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:04:16.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.719+0000 7fb76659c640 1 --2- 172.21.15.116:0/3209210295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7500039f0 0x7fb750003e10 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fb75800c240 tx=0x7fb75801cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3a2122c7c295f86 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:16.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.719+0000 7fb76559a640 1 -- 172.21.15.116:0/3209210295 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb758007d20 con 0x7fb7500039f0 2024-04-25T00:04:16.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.719+0000 7fb76559a640 1 -- 172.21.15.116:0/3209210295 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7580a9070 con 0x7fb7500039f0 2024-04-25T00:04:16.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.719+0000 7fb76559a640 1 -- 172.21.15.116:0/3209210295 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb758005d60 con 0x7fb7500039f0 2024-04-25T00:04:16.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.719+0000 7fb76759e640 1 -- 172.21.15.116:0/3209210295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7500039f0 msgr2=0x7fb750003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:16.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.719+0000 7fb76759e640 1 --2- 172.21.15.116:0/3209210295 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7500039f0 0x7fb750003e10 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fb75800c240 tx=0x7fb75801cbc0 comp rx=0 tx=0).stop 2024-04-25T00:04:16.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.719+0000 7fb766d9d640 1 -- 172.21.15.116:0/3209210295 reap_dead start 2024-04-25T00:04:16.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.719+0000 7fb76759e640 1 -- 172.21.15.116:0/3209210295 shutdown_connections 2024-04-25T00:04:16.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.719+0000 7fb76759e640 1 -- 172.21.15.116:0/3209210295 >> 172.21.15.116:0/3209210295 conn(0x7fb7680c8e80 msgr2=0x7fb7680c9290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:04:16.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.719+0000 7fb76759e640 1 -- 172.21.15.116:0/3209210295 shutdown_connections 2024-04-25T00:04:16.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.719+0000 7fb76759e640 1 -- 172.21.15.116:0/3209210295 wait complete. 2024-04-25T00:04:16.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.723+0000 7fb76759e640 1 Processor -- start 2024-04-25T00:04:16.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.723+0000 7fb76759e640 1 -- start start 2024-04-25T00:04:16.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.723+0000 7fb76759e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb7500039f0 0x7fb76814efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:16.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.723+0000 7fb76759e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb768156b40 0x7fb7681515e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:16.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.723+0000 7fb76759e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7681ee9e0 0x7fb76814fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:16.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.723+0000 7fb76759e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb7680c89a0 con 0x7fb7681ee9e0 2024-04-25T00:04:16.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.723+0000 7fb76759e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb7680c8820 con 0x7fb7500039f0 2024-04-25T00:04:16.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.723+0000 7fb76759e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fb768161b20 con 0x7fb768156b40 2024-04-25T00:04:16.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.723+0000 7fb766d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7681ee9e0 0x7fb76814fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:16.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.723+0000 7fb766d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7681ee9e0 0x7fb76814fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34940/0 (socket says 172.21.15.116:34940) 2024-04-25T00:04:16.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.723+0000 7fb766d9d640 1 -- 172.21.15.116:0/1312139997 learned_addr learned my addr 172.21.15.116:0/1312139997 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:04:16.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.723+0000 7fb76659c640 1 --2- 172.21.15.116:0/1312139997 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb7500039f0 0x7fb76814efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:16.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.723+0000 7fb765d9b640 1 --2- 172.21.15.116:0/1312139997 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb768156b40 0x7fb7681515e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:16.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.723+0000 7fb766d9d640 1 -- 172.21.15.116:0/1312139997 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb768156b40 msgr2=0x7fb7681515e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:16.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.723+0000 7fb766d9d640 1 --2- 172.21.15.116:0/1312139997 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb768156b40 0x7fb7681515e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:16.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.723+0000 7fb766d9d640 1 -- 172.21.15.116:0/1312139997 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb7500039f0 msgr2=0x7fb76814efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:16.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.723+0000 7fb766d9d640 1 --2- 172.21.15.116:0/1312139997 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb7500039f0 0x7fb76814efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:16.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.723+0000 7fb766d9d640 1 -- 172.21.15.116:0/1312139997 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb768158010 con 0x7fb7681ee9e0 2024-04-25T00:04:16.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.723+0000 7fb765d9b640 1 --2- 172.21.15.116:0/1312139997 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb768156b40 0x7fb7681515e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:04:16.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.723+0000 7fb766d9d640 1 --2- 172.21.15.116:0/1312139997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7681ee9e0 0x7fb76814fe60 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fb7681594b0 tx=0x7fb76000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:16.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.727+0000 7fb756ffd640 1 -- 172.21.15.116:0/1312139997 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb760006d60 con 0x7fb7681ee9e0 2024-04-25T00:04:16.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.727+0000 7fb756ffd640 1 -- 172.21.15.116:0/1312139997 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb760003980 con 0x7fb7681ee9e0 2024-04-25T00:04:16.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.727+0000 7fb756ffd640 1 -- 172.21.15.116:0/1312139997 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb760003d70 con 0x7fb7681ee9e0 2024-04-25T00:04:16.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.727+0000 7fb76759e640 1 -- 172.21.15.116:0/1312139997 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7681585f0 con 0x7fb7681ee9e0 2024-04-25T00:04:16.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.727+0000 7fb76759e640 1 -- 172.21.15.116:0/1312139997 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb768158b70 con 0x7fb7681ee9e0 2024-04-25T00:04:16.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.727+0000 7fb76759e640 1 -- 172.21.15.116:0/1312139997 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb76804f510 con 0x7fb7681ee9e0 2024-04-25T00:04:16.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.727+0000 7fb756ffd640 1 -- 172.21.15.116:0/1312139997 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb76001b020 con 0x7fb7681ee9e0 2024-04-25T00:04:16.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.727+0000 7fb756ffd640 1 --2- 172.21.15.116:0/1312139997 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb7480678f0 0x7fb748069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:16.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.727+0000 7fb756ffd640 1 -- 172.21.15.116:0/1312139997 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 6924+0+0 (secure 0 0 0) 0x7fb760085dd0 con 0x7fb7681ee9e0 2024-04-25T00:04:16.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.727+0000 7fb76659c640 1 --2- 172.21.15.116:0/1312139997 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb7480678f0 0x7fb748069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:16.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.731+0000 7fb76659c640 1 --2- 172.21.15.116:0/1312139997 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb7480678f0 0x7fb748069db0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb7580ab000 tx=0x7fb7580b1040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:16.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.731+0000 7fb756ffd640 1 -- 172.21.15.116:0/1312139997 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fb7600564e0 con 0x7fb7681ee9e0 2024-04-25T00:04:16.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.971+0000 7fb76759e640 1 -- 172.21.15.116:0/1312139997 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb7680c6fe0 con 0x7fb7681ee9e0 2024-04-25T00:04:16.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.971+0000 7fb756ffd640 1 -- 172.21.15.116:0/1312139997 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6501 (secure 0 0 0) 0x7fb760087090 con 0x7fb7681ee9e0 2024-04-25T00:04:16.972 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:04:16.972 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4879,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/1889314958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1889314958},{"type":"v1","addr":"172.21.15.174:6837","nonce":1889314958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:03:40.629360+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4828":{"gid":4828,"name":"c","rank":0,"incarnation":26,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6834/1649769369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1649769369},{"type":"v1","addr":"172.21.15.116:6834","nonce":1649769369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4828,"qdb_cluster":[4828]},"id":1},{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:03:34.579825+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4735},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4735":{"gid":4735,"name":"a","rank":0,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.116:6835/2004876235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":2004876235},{"type":"v1","addr":"172.21.15.116:6835","nonce":2004876235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4735,"qdb_cluster":[4735]},"id":2}]} 2024-04-25T00:04:16.973 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 31 2024-04-25T00:04:16.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.975+0000 7fb76759e640 1 -- 172.21.15.116:0/1312139997 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb7480678f0 msgr2=0x7fb748069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:16.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.975+0000 7fb76759e640 1 --2- 172.21.15.116:0/1312139997 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb7480678f0 0x7fb748069db0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb7580ab000 tx=0x7fb7580b1040 comp rx=0 tx=0).stop 2024-04-25T00:04:16.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.975+0000 7fb76759e640 1 -- 172.21.15.116:0/1312139997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7681ee9e0 msgr2=0x7fb76814fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:16.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.975+0000 7fb76759e640 1 --2- 172.21.15.116:0/1312139997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7681ee9e0 0x7fb76814fe60 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fb7681594b0 tx=0x7fb76000dc00 comp rx=0 tx=0).stop 2024-04-25T00:04:16.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.975+0000 7fb76759e640 1 -- 172.21.15.116:0/1312139997 shutdown_connections 2024-04-25T00:04:16.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.975+0000 7fb76759e640 1 --2- 172.21.15.116:0/1312139997 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb7480678f0 0x7fb748069db0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:16.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.975+0000 7fb76759e640 1 --2- 172.21.15.116:0/1312139997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7681ee9e0 0x7fb76814fe60 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:16.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.975+0000 7fb76759e640 1 --2- 172.21.15.116:0/1312139997 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb768156b40 0x7fb7681515e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:16.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.975+0000 7fb76759e640 1 --2- 172.21.15.116:0/1312139997 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb7500039f0 0x7fb76814efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:16.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.975+0000 7fb76759e640 1 -- 172.21.15.116:0/1312139997 >> 172.21.15.116:0/1312139997 conn(0x7fb7680c8e80 msgr2=0x7fb7680c9260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:04:16.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.975+0000 7fb76759e640 1 -- 172.21.15.116:0/1312139997 shutdown_connections 2024-04-25T00:04:16.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:16.975+0000 7fb76759e640 1 -- 172.21.15.116:0/1312139997 wait complete. 2024-04-25T00:04:16.992 INFO:tasks.mds_thrash.fs.[a]:kill mds.c (rank=0) 2024-04-25T00:04:16.992 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-25T00:04:16.992 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:04:17.004 INFO:tasks.ceph.mds.c:Stopped 2024-04-25T00:04:17.004 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:04:17.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.163+0000 7f5a74761640 1 Processor -- start 2024-04-25T00:04:17.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.163+0000 7f5a74761640 1 -- start start 2024-04-25T00:04:17.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.163+0000 7f5a74761640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f5a6c0c4300 0x7f5a6c0aa170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:17.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.163+0000 7f5a74761640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5a6c0c5550 0x7f5a6c156200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:17.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.163+0000 7f5a74761640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a6c1526a0 con 0x7f5a6c0c5100 2024-04-25T00:04:17.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.163+0000 7f5a74761640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f5a6c153460 con 0x7f5a6c0c4300 2024-04-25T00:04:17.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.163+0000 7f5a74761640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f5a6c06ffd0 con 0x7f5a6c0c5550 2024-04-25T00:04:17.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.163+0000 7f5a72cfe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5a6c0c5550 0x7f5a6c156200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:17.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.163+0000 7f5a724fd640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f5a6c0c4300 0x7f5a6c0aa170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:17.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.163+0000 7f5a72cfe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5a6c0c5550 0x7f5a6c156200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:40430/0 (socket says 172.21.15.116:40430) 2024-04-25T00:04:17.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.163+0000 7f5a72cfe640 1 -- 172.21.15.116:0/3399455529 learned_addr learned my addr 172.21.15.116:0/3399455529 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:04:17.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.163+0000 7f5a724fd640 1 -- 172.21.15.116:0/3399455529 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5a6c0c5550 msgr2=0x7f5a6c156200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:04:17.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.163+0000 7f5a724fd640 1 --2- 172.21.15.116:0/3399455529 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5a6c0c5550 0x7f5a6c156200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:17.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.163+0000 7f5a724fd640 1 -- 172.21.15.116:0/3399455529 >> v1:172.21.15.174:6789/0 conn(0x7f5a6c0c5100 legacy=0x7f5a6c155af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:17.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.163+0000 7f5a724fd640 1 -- 172.21.15.116:0/3399455529 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a6c152980 con 0x7f5a6c0c4300 2024-04-25T00:04:17.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.167+0000 7f5a724fd640 1 --2- 172.21.15.116:0/3399455529 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5a6c0c4300 0x7f5a6c0aa170 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5a680097c0 tx=0x7f5a6801c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d78c1b8f009d91ec server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:17.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.167+0000 7f5a714fb640 1 -- 172.21.15.116:0/3399455529 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a680b0070 con 0x7f5a6c0c4300 2024-04-25T00:04:17.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.167+0000 7f5a714fb640 1 -- 172.21.15.116:0/3399455529 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a680ab600 con 0x7f5a6c0c4300 2024-04-25T00:04:17.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.167+0000 7f5a714fb640 1 -- 172.21.15.116:0/3399455529 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a680ab9d0 con 0x7f5a6c0c4300 2024-04-25T00:04:17.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.167+0000 7f5a74761640 1 -- 172.21.15.116:0/3399455529 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5a6c0c4300 msgr2=0x7f5a6c0aa170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:17.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.167+0000 7f5a74761640 1 --2- 172.21.15.116:0/3399455529 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5a6c0c4300 0x7f5a6c0aa170 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5a680097c0 tx=0x7f5a6801c4a0 comp rx=0 tx=0).stop 2024-04-25T00:04:17.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.167+0000 7f5a74761640 1 -- 172.21.15.116:0/3399455529 shutdown_connections 2024-04-25T00:04:17.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.167+0000 7f5a74761640 1 --2- 172.21.15.116:0/3399455529 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5a6c0c5550 0x7f5a6c156200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:17.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.167+0000 7f5a74761640 1 --2- 172.21.15.116:0/3399455529 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5a6c0c4300 0x7f5a6c0aa170 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:17.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.167+0000 7f5a74761640 1 -- 172.21.15.116:0/3399455529 >> 172.21.15.116:0/3399455529 conn(0x7f5a6c06c720 msgr2=0x7f5a6c06cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:04:17.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.167+0000 7f5a74761640 1 -- 172.21.15.116:0/3399455529 shutdown_connections 2024-04-25T00:04:17.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.167+0000 7f5a74761640 1 -- 172.21.15.116:0/3399455529 wait complete. 2024-04-25T00:04:17.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.167+0000 7f5a74761640 1 Processor -- start 2024-04-25T00:04:17.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.167+0000 7f5a74761640 1 -- start start 2024-04-25T00:04:17.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.171+0000 7f5a74761640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5a6c0c4300 0x7f5a6c14bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:17.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.171+0000 7f5a74761640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5a6c0c5100 0x7f5a6c151f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:17.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.171+0000 7f5a74761640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a6c0c5550 0x7f5a6c0c6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:17.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.171+0000 7f5a74761640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5a6c06fd00 con 0x7f5a6c0c5550 2024-04-25T00:04:17.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.171+0000 7f5a74761640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5a6c06fb80 con 0x7f5a6c0c4300 2024-04-25T00:04:17.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.171+0000 7f5a74761640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f5a6c15f6a0 con 0x7f5a6c0c5100 2024-04-25T00:04:17.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.171+0000 7f5a72cfe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a6c0c5550 0x7f5a6c0c6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:17.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.171+0000 7f5a72cfe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a6c0c5550 0x7f5a6c0c6fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34952/0 (socket says 172.21.15.116:34952) 2024-04-25T00:04:17.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.171+0000 7f5a72cfe640 1 -- 172.21.15.116:0/3793575990 learned_addr learned my addr 172.21.15.116:0/3793575990 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:04:17.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.171+0000 7f5a724fd640 1 --2- 172.21.15.116:0/3793575990 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5a6c0c4300 0x7f5a6c14bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:17.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.171+0000 7f5a71cfc640 1 --2- 172.21.15.116:0/3793575990 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5a6c0c5100 0x7f5a6c151f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:17.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.171+0000 7f5a72cfe640 1 -- 172.21.15.116:0/3793575990 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5a6c0c5100 msgr2=0x7f5a6c151f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:17.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.171+0000 7f5a72cfe640 1 --2- 172.21.15.116:0/3793575990 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5a6c0c5100 0x7f5a6c151f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:17.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.171+0000 7f5a72cfe640 1 -- 172.21.15.116:0/3793575990 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5a6c0c4300 msgr2=0x7f5a6c14bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:17.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.171+0000 7f5a72cfe640 1 --2- 172.21.15.116:0/3793575990 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5a6c0c4300 0x7f5a6c14bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:17.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.171+0000 7f5a72cfe640 1 -- 172.21.15.116:0/3793575990 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a6c152460 con 0x7f5a6c0c5550 2024-04-25T00:04:17.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.171+0000 7f5a72cfe640 1 --2- 172.21.15.116:0/3793575990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a6c0c5550 0x7f5a6c0c6fe0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f5a6400efc0 tx=0x7f5a6400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:17.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.171+0000 7f5a62ffd640 1 -- 172.21.15.116:0/3793575990 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a6400ccc0 con 0x7f5a6c0c5550 2024-04-25T00:04:17.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.171+0000 7f5a62ffd640 1 -- 172.21.15.116:0/3793575990 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a640039a0 con 0x7f5a6c0c5550 2024-04-25T00:04:17.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.171+0000 7f5a74761640 1 -- 172.21.15.116:0/3793575990 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a6c1ee060 con 0x7f5a6c0c5550 2024-04-25T00:04:17.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.171+0000 7f5a62ffd640 1 -- 172.21.15.116:0/3793575990 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a6400ce60 con 0x7f5a6c0c5550 2024-04-25T00:04:17.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.171+0000 7f5a74761640 1 -- 172.21.15.116:0/3793575990 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a6c1ee600 con 0x7f5a6c0c5550 2024-04-25T00:04:17.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.171+0000 7f5a74761640 1 -- 172.21.15.116:0/3793575990 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a6c1eeff0 con 0x7f5a6c0c5550 2024-04-25T00:04:17.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.171+0000 7f5a62ffd640 1 -- 172.21.15.116:0/3793575990 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5a64012050 con 0x7f5a6c0c5550 2024-04-25T00:04:17.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.171+0000 7f5a62ffd640 1 --2- 172.21.15.116:0/3793575990 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5a500678f0 0x7f5a50069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:17.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.171+0000 7f5a62ffd640 1 -- 172.21.15.116:0/3793575990 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 6924+0+0 (secure 0 0 0) 0x7f5a640864c0 con 0x7f5a6c0c5550 2024-04-25T00:04:17.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.171+0000 7f5a724fd640 1 --2- 172.21.15.116:0/3793575990 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5a500678f0 0x7f5a50069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:17.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.175+0000 7f5a724fd640 1 --2- 172.21.15.116:0/3793575990 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5a500678f0 0x7f5a50069db0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5a680ad000 tx=0x7f5a68008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:17.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.179+0000 7f5a62ffd640 1 -- 172.21.15.116:0/3793575990 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f5a6c152460 con 0x7f5a6c0c5550 2024-04-25T00:04:17.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.419+0000 7f5a74761640 1 -- 172.21.15.116:0/3793575990 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5a6c0a9f50 con 0x7f5a6c0c5550 2024-04-25T00:04:17.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.419+0000 7f5a62ffd640 1 -- 172.21.15.116:0/3793575990 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6501 (secure 0 0 0) 0x7f5a64010040 con 0x7f5a6c0c5550 2024-04-25T00:04:17.419 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:04:17.419 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4879,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/1889314958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1889314958},{"type":"v1","addr":"172.21.15.174:6837","nonce":1889314958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:03:40.629360+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4828":{"gid":4828,"name":"c","rank":0,"incarnation":26,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6834/1649769369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1649769369},{"type":"v1","addr":"172.21.15.116:6834","nonce":1649769369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4828,"qdb_cluster":[4828]},"id":1},{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:03:34.579825+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4735},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4735":{"gid":4735,"name":"a","rank":0,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.116:6835/2004876235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":2004876235},{"type":"v1","addr":"172.21.15.116:6835","nonce":2004876235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4735,"qdb_cluster":[4735]},"id":2}]} 2024-04-25T00:04:17.419 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 31 2024-04-25T00:04:17.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.419+0000 7f5a74761640 1 -- 172.21.15.116:0/3793575990 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5a500678f0 msgr2=0x7f5a50069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:17.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.419+0000 7f5a74761640 1 --2- 172.21.15.116:0/3793575990 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5a500678f0 0x7f5a50069db0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5a680ad000 tx=0x7f5a68008040 comp rx=0 tx=0).stop 2024-04-25T00:04:17.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.419+0000 7f5a74761640 1 -- 172.21.15.116:0/3793575990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a6c0c5550 msgr2=0x7f5a6c0c6fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:17.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.419+0000 7f5a74761640 1 --2- 172.21.15.116:0/3793575990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a6c0c5550 0x7f5a6c0c6fe0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f5a6400efc0 tx=0x7f5a6400c490 comp rx=0 tx=0).stop 2024-04-25T00:04:17.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.419+0000 7f5a74761640 1 -- 172.21.15.116:0/3793575990 shutdown_connections 2024-04-25T00:04:17.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.419+0000 7f5a74761640 1 --2- 172.21.15.116:0/3793575990 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5a500678f0 0x7f5a50069db0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:17.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.419+0000 7f5a74761640 1 --2- 172.21.15.116:0/3793575990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a6c0c5550 0x7f5a6c0c6fe0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:17.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.419+0000 7f5a74761640 1 --2- 172.21.15.116:0/3793575990 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5a6c0c5100 0x7f5a6c151f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:17.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.419+0000 7f5a74761640 1 --2- 172.21.15.116:0/3793575990 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5a6c0c4300 0x7f5a6c14bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:17.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.419+0000 7f5a74761640 1 -- 172.21.15.116:0/3793575990 >> 172.21.15.116:0/3793575990 conn(0x7f5a6c06c720 msgr2=0x7f5a6c0ad160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:04:17.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.419+0000 7f5a74761640 1 -- 172.21.15.116:0/3793575990 shutdown_connections 2024-04-25T00:04:17.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:17.419+0000 7f5a74761640 1 -- 172.21.15.116:0/3793575990 wait complete. 2024-04-25T00:04:17.440 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:04:19.441 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:04:19.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.575+0000 7f6c5d26a640 1 Processor -- start 2024-04-25T00:04:19.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.575+0000 7f6c5d26a640 1 -- start start 2024-04-25T00:04:19.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.575+0000 7f6c5d26a640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f6c58150d20 0x7f6c580c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:19.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.575+0000 7f6c5d26a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c580c2f50 0x7f6c581538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:19.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.575+0000 7f6c5d26a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6c580a94a0 0x7f6c58153e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:19.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.575+0000 7f6c5d26a640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f6c580c3910 con 0x7f6c580a94a0 2024-04-25T00:04:19.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.575+0000 7f6c5d26a640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f6c580c16e0 con 0x7f6c58150d20 2024-04-25T00:04:19.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.575+0000 7f6c5d26a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6c580c1400 con 0x7f6c580c2f50 2024-04-25T00:04:19.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.575+0000 7f6c577fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6c580a94a0 0x7f6c58153e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:19.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.575+0000 7f6c577fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6c580a94a0 0x7f6c58153e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34954/0 (socket says 172.21.15.116:34954) 2024-04-25T00:04:19.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.575+0000 7f6c577fe640 1 -- 172.21.15.116:0/3742038113 learned_addr learned my addr 172.21.15.116:0/3742038113 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:04:19.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.575+0000 7f6c4ffff640 1 --2- 172.21.15.116:0/3742038113 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c580c2f50 0x7f6c581538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:19.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.575+0000 7f6c56ffd640 1 --2- 172.21.15.116:0/3742038113 >> v2:172.21.15.174:3300/0 conn(0x7f6c58150d20 0x7f6c580c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:19.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.575+0000 7f6c577fe640 1 -- 172.21.15.116:0/3742038113 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c580c2f50 msgr2=0x7f6c581538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:04:19.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.575+0000 7f6c577fe640 1 --2- 172.21.15.116:0/3742038113 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c580c2f50 0x7f6c581538e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:19.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.575+0000 7f6c577fe640 1 -- 172.21.15.116:0/3742038113 >> v2:172.21.15.174:3300/0 conn(0x7f6c58150d20 msgr2=0x7f6c580c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:04:19.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.575+0000 7f6c577fe640 1 --2- 172.21.15.116:0/3742038113 >> v2:172.21.15.174:3300/0 conn(0x7f6c58150d20 0x7f6c580c2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:19.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.575+0000 7f6c577fe640 1 -- 172.21.15.116:0/3742038113 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c5806e660 con 0x7f6c580a94a0 2024-04-25T00:04:19.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.579+0000 7f6c577fe640 1 --2- 172.21.15.116:0/3742038113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c580a94a0 0x7f6c58153e20 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f6c58073550 tx=0x7f6c5001c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb4679c3b2b2528a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:19.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.579+0000 7f6c567fc640 1 -- 172.21.15.116:0/3742038113 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c500a5740 con 0x7f6c580a94a0 2024-04-25T00:04:19.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.579+0000 7f6c567fc640 1 -- 172.21.15.116:0/3742038113 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c500a58e0 con 0x7f6c580a94a0 2024-04-25T00:04:19.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.579+0000 7f6c567fc640 1 -- 172.21.15.116:0/3742038113 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c500a5cb0 con 0x7f6c580a94a0 2024-04-25T00:04:19.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.579+0000 7f6c5d26a640 1 -- 172.21.15.116:0/3742038113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c580a94a0 msgr2=0x7f6c58153e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:19.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.579+0000 7f6c5d26a640 1 --2- 172.21.15.116:0/3742038113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c580a94a0 0x7f6c58153e20 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f6c58073550 tx=0x7f6c5001c9f0 comp rx=0 tx=0).stop 2024-04-25T00:04:19.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.579+0000 7f6c5d26a640 1 -- 172.21.15.116:0/3742038113 shutdown_connections 2024-04-25T00:04:19.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.579+0000 7f6c5d26a640 1 --2- 172.21.15.116:0/3742038113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c580a94a0 0x7f6c58153e20 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:19.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.579+0000 7f6c5d26a640 1 --2- 172.21.15.116:0/3742038113 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c580c2f50 0x7f6c581538e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:19.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.579+0000 7f6c5d26a640 1 --2- 172.21.15.116:0/3742038113 >> v2:172.21.15.174:3300/0 conn(0x7f6c58150d20 0x7f6c580c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:19.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.579+0000 7f6c5d26a640 1 -- 172.21.15.116:0/3742038113 >> 172.21.15.116:0/3742038113 conn(0x7f6c580c3b00 msgr2=0x7f6c580c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:04:19.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.579+0000 7f6c5d26a640 1 -- 172.21.15.116:0/3742038113 shutdown_connections 2024-04-25T00:04:19.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.579+0000 7f6c5d26a640 1 -- 172.21.15.116:0/3742038113 wait complete. 2024-04-25T00:04:19.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.579+0000 7f6c5d26a640 1 Processor -- start 2024-04-25T00:04:19.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.579+0000 7f6c5d26a640 1 -- start start 2024-04-25T00:04:19.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.579+0000 7f6c5d26a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c580a94a0 0x7f6c5815c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:19.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.579+0000 7f6c5d26a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c580c2f50 0x7f6c58155430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:19.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.579+0000 7f6c5d26a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c58150d20 0x7f6c58155970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:19.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.579+0000 7f6c5d26a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6c58157340 con 0x7f6c580a94a0 2024-04-25T00:04:19.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.579+0000 7f6c5d26a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6c58154570 con 0x7f6c58150d20 2024-04-25T00:04:19.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.579+0000 7f6c5d26a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6c581574c0 con 0x7f6c580c2f50 2024-04-25T00:04:19.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.579+0000 7f6c56ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c580a94a0 0x7f6c5815c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:19.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.579+0000 7f6c56ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c580a94a0 0x7f6c5815c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34968/0 (socket says 172.21.15.116:34968) 2024-04-25T00:04:19.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.579+0000 7f6c56ffd640 1 -- 172.21.15.116:0/4235257144 learned_addr learned my addr 172.21.15.116:0/4235257144 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:04:19.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.579+0000 7f6c4ffff640 1 --2- 172.21.15.116:0/4235257144 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c580c2f50 0x7f6c58155430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:19.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.579+0000 7f6c56ffd640 1 -- 172.21.15.116:0/4235257144 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c580c2f50 msgr2=0x7f6c58155430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:19.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.579+0000 7f6c56ffd640 1 --2- 172.21.15.116:0/4235257144 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c580c2f50 0x7f6c58155430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:19.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.579+0000 7f6c56ffd640 1 -- 172.21.15.116:0/4235257144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c58150d20 msgr2=0x7f6c58155970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:19.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.579+0000 7f6c577fe640 1 --2- 172.21.15.116:0/4235257144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c58150d20 0x7f6c58155970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:19.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.583+0000 7f6c56ffd640 1 --2- 172.21.15.116:0/4235257144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c58150d20 0x7f6c58155970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:19.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.583+0000 7f6c56ffd640 1 -- 172.21.15.116:0/4235257144 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c581551e0 con 0x7f6c580a94a0 2024-04-25T00:04:19.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.583+0000 7f6c56ffd640 1 --2- 172.21.15.116:0/4235257144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c580a94a0 0x7f6c5815c550 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f6c4800d5f0 tx=0x7f6c4800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:19.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.583+0000 7f6c4f7fe640 1 -- 172.21.15.116:0/4235257144 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c480125f0 con 0x7f6c580a94a0 2024-04-25T00:04:19.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.583+0000 7f6c4f7fe640 1 -- 172.21.15.116:0/4235257144 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c48019070 con 0x7f6c580a94a0 2024-04-25T00:04:19.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.583+0000 7f6c4f7fe640 1 -- 172.21.15.116:0/4235257144 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c480129b0 con 0x7f6c580a94a0 2024-04-25T00:04:19.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.583+0000 7f6c5d26a640 1 -- 172.21.15.116:0/4235257144 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c580725b0 con 0x7f6c580a94a0 2024-04-25T00:04:19.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.583+0000 7f6c5d26a640 1 -- 172.21.15.116:0/4235257144 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c58154cc0 con 0x7f6c580a94a0 2024-04-25T00:04:19.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.583+0000 7f6c5d26a640 1 -- 172.21.15.116:0/4235257144 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c58068350 con 0x7f6c580a94a0 2024-04-25T00:04:19.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.583+0000 7f6c4f7fe640 1 -- 172.21.15.116:0/4235257144 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6c48012b50 con 0x7f6c580a94a0 2024-04-25T00:04:19.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.583+0000 7f6c4f7fe640 1 --2- 172.21.15.116:0/4235257144 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6c380678c0 0x7f6c38069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:19.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.583+0000 7f6c4f7fe640 1 -- 172.21.15.116:0/4235257144 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 6924+0+0 (secure 0 0 0) 0x7f6c480864e0 con 0x7f6c580a94a0 2024-04-25T00:04:19.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.583+0000 7f6c4ffff640 1 --2- 172.21.15.116:0/4235257144 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6c380678c0 0x7f6c38069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:19.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.583+0000 7f6c4ffff640 1 --2- 172.21.15.116:0/4235257144 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6c380678c0 0x7f6c38069d80 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6c5814aa50 tx=0x7f6c40008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:19.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.587+0000 7f6c4f7fe640 1 -- 172.21.15.116:0/4235257144 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f6c48011340 con 0x7f6c580a94a0 2024-04-25T00:04:19.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.783+0000 7f6c5d26a640 1 -- 172.21.15.116:0/4235257144 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6c580c34f0 con 0x7f6c580a94a0 2024-04-25T00:04:19.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.783+0000 7f6c4f7fe640 1 -- 172.21.15.116:0/4235257144 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6501 (secure 0 0 0) 0x7f6c48056bf0 con 0x7f6c580a94a0 2024-04-25T00:04:19.785 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:04:19.785 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4879,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/1889314958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1889314958},{"type":"v1","addr":"172.21.15.174:6837","nonce":1889314958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:03:40.629360+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4828":{"gid":4828,"name":"c","rank":0,"incarnation":26,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6834/1649769369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1649769369},{"type":"v1","addr":"172.21.15.116:6834","nonce":1649769369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4828,"qdb_cluster":[4828]},"id":1},{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:03:34.579825+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4735},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4735":{"gid":4735,"name":"a","rank":0,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.116:6835/2004876235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":2004876235},{"type":"v1","addr":"172.21.15.116:6835","nonce":2004876235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4735,"qdb_cluster":[4735]},"id":2}]} 2024-04-25T00:04:19.786 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 31 2024-04-25T00:04:19.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.787+0000 7f6c5d26a640 1 -- 172.21.15.116:0/4235257144 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6c380678c0 msgr2=0x7f6c38069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:19.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.787+0000 7f6c5d26a640 1 --2- 172.21.15.116:0/4235257144 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6c380678c0 0x7f6c38069d80 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6c5814aa50 tx=0x7f6c40008040 comp rx=0 tx=0).stop 2024-04-25T00:04:19.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.787+0000 7f6c5d26a640 1 -- 172.21.15.116:0/4235257144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c580a94a0 msgr2=0x7f6c5815c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:19.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.787+0000 7f6c5d26a640 1 --2- 172.21.15.116:0/4235257144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c580a94a0 0x7f6c5815c550 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f6c4800d5f0 tx=0x7f6c4800dab0 comp rx=0 tx=0).stop 2024-04-25T00:04:19.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.787+0000 7f6c5d26a640 1 -- 172.21.15.116:0/4235257144 shutdown_connections 2024-04-25T00:04:19.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.787+0000 7f6c5d26a640 1 --2- 172.21.15.116:0/4235257144 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6c380678c0 0x7f6c38069d80 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:19.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.787+0000 7f6c5d26a640 1 --2- 172.21.15.116:0/4235257144 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c58150d20 0x7f6c58155970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:19.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.787+0000 7f6c5d26a640 1 --2- 172.21.15.116:0/4235257144 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c580c2f50 0x7f6c58155430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:19.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.787+0000 7f6c5d26a640 1 --2- 172.21.15.116:0/4235257144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c580a94a0 0x7f6c5815c550 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:19.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.787+0000 7f6c5d26a640 1 -- 172.21.15.116:0/4235257144 >> 172.21.15.116:0/4235257144 conn(0x7f6c580c3b00 msgr2=0x7f6c580ac1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:04:19.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.787+0000 7f6c5d26a640 1 -- 172.21.15.116:0/4235257144 shutdown_connections 2024-04-25T00:04:19.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:19.787+0000 7f6c5d26a640 1 -- 172.21.15.116:0/4235257144 wait complete. 2024-04-25T00:04:19.804 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:04:21.804 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:04:21.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.931+0000 7febec30f640 1 Processor -- start 2024-04-25T00:04:21.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.931+0000 7febec30f640 1 -- start start 2024-04-25T00:04:21.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.935+0000 7febec30f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7febe4150d20 0x7febe40c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:21.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.935+0000 7febec30f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7febe40c2f50 0x7febe41538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:21.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.935+0000 7febec30f640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7febe40a94a0 0x7febe4153e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:21.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.935+0000 7febec30f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7febe40c3910 con 0x7febe4150d20 2024-04-25T00:04:21.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.935+0000 7febec30f640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7febe40c16e0 con 0x7febe40a94a0 2024-04-25T00:04:21.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.935+0000 7febec30f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7febe40c1400 con 0x7febe40c2f50 2024-04-25T00:04:21.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.935+0000 7febea0ab640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7febe4150d20 0x7febe40c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:21.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.935+0000 7febea0ab640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7febe4150d20 0x7febe40c2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34982/0 (socket says 172.21.15.116:34982) 2024-04-25T00:04:21.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.935+0000 7febea0ab640 1 -- 172.21.15.116:0/362902443 learned_addr learned my addr 172.21.15.116:0/362902443 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:04:21.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.935+0000 7febea8ac640 1 --2- 172.21.15.116:0/362902443 >> v2:172.21.15.174:3300/0 conn(0x7febe40a94a0 0x7febe4153e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:21.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.935+0000 7febe98aa640 1 --2- 172.21.15.116:0/362902443 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7febe40c2f50 0x7febe41538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:21.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.935+0000 7febea0ab640 1 -- 172.21.15.116:0/362902443 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7febe40c2f50 msgr2=0x7febe41538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:04:21.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.935+0000 7febea0ab640 1 --2- 172.21.15.116:0/362902443 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7febe40c2f50 0x7febe41538e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:21.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.935+0000 7febea0ab640 1 -- 172.21.15.116:0/362902443 >> v2:172.21.15.174:3300/0 conn(0x7febe40a94a0 msgr2=0x7febe4153e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:04:21.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.935+0000 7febea0ab640 1 --2- 172.21.15.116:0/362902443 >> v2:172.21.15.174:3300/0 conn(0x7febe40a94a0 0x7febe4153e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:21.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.935+0000 7febea0ab640 1 -- 172.21.15.116:0/362902443 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febe406e660 con 0x7febe4150d20 2024-04-25T00:04:21.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.935+0000 7febea0ab640 1 --2- 172.21.15.116:0/362902443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7febe4150d20 0x7febe40c2a10 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7febe40732b0 tx=0x7febd401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49b5d530daa9055f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:21.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.935+0000 7febe90a9640 1 -- 172.21.15.116:0/362902443 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febd40b1070 con 0x7febe4150d20 2024-04-25T00:04:21.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.935+0000 7febe90a9640 1 -- 172.21.15.116:0/362902443 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febd40ac410 con 0x7febe4150d20 2024-04-25T00:04:21.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.935+0000 7febe90a9640 1 -- 172.21.15.116:0/362902443 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febd40ac7e0 con 0x7febe4150d20 2024-04-25T00:04:21.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.939+0000 7febec30f640 1 -- 172.21.15.116:0/362902443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7febe4150d20 msgr2=0x7febe40c2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:21.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.939+0000 7febec30f640 1 --2- 172.21.15.116:0/362902443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7febe4150d20 0x7febe40c2a10 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7febe40732b0 tx=0x7febd401c520 comp rx=0 tx=0).stop 2024-04-25T00:04:21.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.939+0000 7febec30f640 1 -- 172.21.15.116:0/362902443 shutdown_connections 2024-04-25T00:04:21.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.939+0000 7febec30f640 1 --2- 172.21.15.116:0/362902443 >> v2:172.21.15.174:3300/0 conn(0x7febe40a94a0 0x7febe4153e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:21.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.939+0000 7febec30f640 1 --2- 172.21.15.116:0/362902443 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7febe40c2f50 0x7febe41538e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:21.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.939+0000 7febec30f640 1 --2- 172.21.15.116:0/362902443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7febe4150d20 0x7febe40c2a10 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:21.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.939+0000 7febec30f640 1 -- 172.21.15.116:0/362902443 >> 172.21.15.116:0/362902443 conn(0x7febe40c3b00 msgr2=0x7febe40c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:04:21.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.939+0000 7febec30f640 1 -- 172.21.15.116:0/362902443 shutdown_connections 2024-04-25T00:04:21.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.939+0000 7febec30f640 1 -- 172.21.15.116:0/362902443 wait complete. 2024-04-25T00:04:21.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.939+0000 7febec30f640 1 Processor -- start 2024-04-25T00:04:21.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.939+0000 7febec30f640 1 -- start start 2024-04-25T00:04:21.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.939+0000 7febec30f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7febe40a94a0 0x7febe4155190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:21.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.939+0000 7febec30f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7febe40c2f50 0x7febe4152840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:21.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.939+0000 7febec30f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7febe4150d20 0x7febe41ef600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:21.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.939+0000 7febec30f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7febe4157340 con 0x7febe4150d20 2024-04-25T00:04:21.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.939+0000 7febec30f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7febe4154570 con 0x7febe40a94a0 2024-04-25T00:04:21.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.939+0000 7febec30f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7febe41574c0 con 0x7febe40c2f50 2024-04-25T00:04:21.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.939+0000 7febea8ac640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7febe4150d20 0x7febe41ef600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:21.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.939+0000 7febea8ac640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7febe4150d20 0x7febe41ef600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34984/0 (socket says 172.21.15.116:34984) 2024-04-25T00:04:21.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.939+0000 7febea8ac640 1 -- 172.21.15.116:0/218106421 learned_addr learned my addr 172.21.15.116:0/218106421 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:04:21.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.939+0000 7febe98aa640 1 --2- 172.21.15.116:0/218106421 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7febe40c2f50 0x7febe4152840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:21.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.939+0000 7febea0ab640 1 --2- 172.21.15.116:0/218106421 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7febe40a94a0 0x7febe4155190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:21.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.939+0000 7febea8ac640 1 -- 172.21.15.116:0/218106421 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7febe40c2f50 msgr2=0x7febe4152840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:21.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.939+0000 7febea8ac640 1 --2- 172.21.15.116:0/218106421 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7febe40c2f50 0x7febe4152840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:21.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.939+0000 7febea8ac640 1 -- 172.21.15.116:0/218106421 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7febe40a94a0 msgr2=0x7febe4155190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:21.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.939+0000 7febea8ac640 1 --2- 172.21.15.116:0/218106421 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7febe40a94a0 0x7febe4155190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:21.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.939+0000 7febea8ac640 1 -- 172.21.15.116:0/218106421 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febe4152d80 con 0x7febe4150d20 2024-04-25T00:04:21.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.939+0000 7febea8ac640 1 --2- 172.21.15.116:0/218106421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7febe4150d20 0x7febe41ef600 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7febdc007e90 tx=0x7febdc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:21.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.939+0000 7febe2ffd640 1 -- 172.21.15.116:0/218106421 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febdc014420 con 0x7febe4150d20 2024-04-25T00:04:21.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.939+0000 7febe2ffd640 1 -- 172.21.15.116:0/218106421 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febdc019070 con 0x7febe4150d20 2024-04-25T00:04:21.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.939+0000 7febec30f640 1 -- 172.21.15.116:0/218106421 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febe4154eb0 con 0x7febe4150d20 2024-04-25T00:04:21.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.939+0000 7febe2ffd640 1 -- 172.21.15.116:0/218106421 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febdc014820 con 0x7febe4150d20 2024-04-25T00:04:21.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.939+0000 7febec30f640 1 -- 172.21.15.116:0/218106421 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febe414ade0 con 0x7febe4150d20 2024-04-25T00:04:21.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.943+0000 7febec30f640 1 -- 172.21.15.116:0/218106421 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febe4068350 con 0x7febe4150d20 2024-04-25T00:04:21.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.943+0000 7febe2ffd640 1 -- 172.21.15.116:0/218106421 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7febdc0149c0 con 0x7febe4150d20 2024-04-25T00:04:21.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.943+0000 7febe2ffd640 1 --2- 172.21.15.116:0/218106421 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7febd00678c0 0x7febd0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:21.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.943+0000 7febe2ffd640 1 -- 172.21.15.116:0/218106421 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 6924+0+0 (secure 0 0 0) 0x7febdc086a80 con 0x7febe4150d20 2024-04-25T00:04:21.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.943+0000 7febea0ab640 1 --2- 172.21.15.116:0/218106421 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7febd00678c0 0x7febd0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:21.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.943+0000 7febea0ab640 1 --2- 172.21.15.116:0/218106421 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7febd00678c0 0x7febd0069d80 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7febd40ae000 tx=0x7febd4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:21.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:21.947+0000 7febe2ffd640 1 -- 172.21.15.116:0/218106421 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7febdc057060 con 0x7febe4150d20 2024-04-25T00:04:22.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:22.139+0000 7febec30f640 1 -- 172.21.15.116:0/218106421 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7febe40c34f0 con 0x7febe4150d20 2024-04-25T00:04:22.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:22.139+0000 7febe2ffd640 1 -- 172.21.15.116:0/218106421 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6501 (secure 0 0 0) 0x7febdc056960 con 0x7febe4150d20 2024-04-25T00:04:22.141 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:04:22.141 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4879,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/1889314958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1889314958},{"type":"v1","addr":"172.21.15.174:6837","nonce":1889314958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:03:40.629360+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4828":{"gid":4828,"name":"c","rank":0,"incarnation":26,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6834/1649769369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1649769369},{"type":"v1","addr":"172.21.15.116:6834","nonce":1649769369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4828,"qdb_cluster":[4828]},"id":1},{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:03:34.579825+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4735},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4735":{"gid":4735,"name":"a","rank":0,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.116:6835/2004876235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":2004876235},{"type":"v1","addr":"172.21.15.116:6835","nonce":2004876235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4735,"qdb_cluster":[4735]},"id":2}]} 2024-04-25T00:04:22.142 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 31 2024-04-25T00:04:22.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:22.143+0000 7febec30f640 1 -- 172.21.15.116:0/218106421 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7febd00678c0 msgr2=0x7febd0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:22.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:22.143+0000 7febec30f640 1 --2- 172.21.15.116:0/218106421 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7febd00678c0 0x7febd0069d80 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7febd40ae000 tx=0x7febd4008040 comp rx=0 tx=0).stop 2024-04-25T00:04:22.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:22.143+0000 7febec30f640 1 -- 172.21.15.116:0/218106421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7febe4150d20 msgr2=0x7febe41ef600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:22.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:22.143+0000 7febec30f640 1 --2- 172.21.15.116:0/218106421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7febe4150d20 0x7febe41ef600 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7febdc007e90 tx=0x7febdc00e4b0 comp rx=0 tx=0).stop 2024-04-25T00:04:22.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:22.143+0000 7febec30f640 1 -- 172.21.15.116:0/218106421 shutdown_connections 2024-04-25T00:04:22.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:22.143+0000 7febec30f640 1 --2- 172.21.15.116:0/218106421 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7febd00678c0 0x7febd0069d80 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:22.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:22.143+0000 7febec30f640 1 --2- 172.21.15.116:0/218106421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7febe4150d20 0x7febe41ef600 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:22.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:22.143+0000 7febec30f640 1 --2- 172.21.15.116:0/218106421 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7febe40c2f50 0x7febe4152840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:22.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:22.143+0000 7febec30f640 1 --2- 172.21.15.116:0/218106421 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7febe40a94a0 0x7febe4155190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:22.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:22.143+0000 7febec30f640 1 -- 172.21.15.116:0/218106421 >> 172.21.15.116:0/218106421 conn(0x7febe40c3b00 msgr2=0x7febe40c1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:04:22.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:22.143+0000 7febec30f640 1 -- 172.21.15.116:0/218106421 shutdown_connections 2024-04-25T00:04:22.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:22.143+0000 7febec30f640 1 -- 172.21.15.116:0/218106421 wait complete. 2024-04-25T00:04:22.159 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:04:24.160 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:04:24.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.291+0000 7fcdf1ba6640 1 Processor -- start 2024-04-25T00:04:24.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.291+0000 7fcdf1ba6640 1 -- start start 2024-04-25T00:04:24.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.291+0000 7fcdf1ba6640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcdec0a94a0 0x7fcdec1538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:24.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.291+0000 7fcdf1ba6640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcdec0a99f0 0x7fcdec153e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:24.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.291+0000 7fcdf1ba6640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcdec0c3910 con 0x7fcdec150d20 2024-04-25T00:04:24.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.291+0000 7fcdf1ba6640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fcdec0c16e0 con 0x7fcdec0a99f0 2024-04-25T00:04:24.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.291+0000 7fcdf1ba6640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fcdec0c1400 con 0x7fcdec0a94a0 2024-04-25T00:04:24.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.291+0000 7fcdebfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcdec0a99f0 0x7fcdec153e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:24.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.291+0000 7fcdebfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcdec0a99f0 0x7fcdec153e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34990/0 (socket says 172.21.15.116:34990) 2024-04-25T00:04:24.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.291+0000 7fcdebfff640 1 -- 172.21.15.116:0/192896671 learned_addr learned my addr 172.21.15.116:0/192896671 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:04:24.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.291+0000 7fcde3fff640 1 --2- 172.21.15.116:0/192896671 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcdec0a94a0 0x7fcdec1538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:24.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.291+0000 7fcdebfff640 1 -- 172.21.15.116:0/192896671 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcdec0a94a0 msgr2=0x7fcdec1538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:04:24.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.291+0000 7fcdebfff640 1 --2- 172.21.15.116:0/192896671 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcdec0a94a0 0x7fcdec1538e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:24.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.291+0000 7fcdebfff640 1 -- 172.21.15.116:0/192896671 >> v1:172.21.15.174:6789/0 conn(0x7fcdec150d20 legacy=0x7fcdec0c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:24.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.291+0000 7fcdebfff640 1 -- 172.21.15.116:0/192896671 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdec06e660 con 0x7fcdec0a99f0 2024-04-25T00:04:24.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.295+0000 7fcdebfff640 1 --2- 172.21.15.116:0/192896671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdec0a99f0 0x7fcdec153e20 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fcde4009e50 tx=0x7fcde401ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e94114b33a6b5623 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:24.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.295+0000 7fcdeaffd640 1 -- 172.21.15.116:0/192896671 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcde40a5700 con 0x7fcdec0a99f0 2024-04-25T00:04:24.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.295+0000 7fcdeaffd640 1 -- 172.21.15.116:0/192896671 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcde40a58a0 con 0x7fcdec0a99f0 2024-04-25T00:04:24.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.295+0000 7fcdeaffd640 1 -- 172.21.15.116:0/192896671 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcde40a5c70 con 0x7fcdec0a99f0 2024-04-25T00:04:24.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.295+0000 7fcdf1ba6640 1 -- 172.21.15.116:0/192896671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdec0a99f0 msgr2=0x7fcdec153e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:24.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.295+0000 7fcdf1ba6640 1 --2- 172.21.15.116:0/192896671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdec0a99f0 0x7fcdec153e20 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fcde4009e50 tx=0x7fcde401ca30 comp rx=0 tx=0).stop 2024-04-25T00:04:24.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.295+0000 7fcdf1ba6640 1 -- 172.21.15.116:0/192896671 shutdown_connections 2024-04-25T00:04:24.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.295+0000 7fcdf1ba6640 1 --2- 172.21.15.116:0/192896671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdec0a99f0 0x7fcdec153e20 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:24.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.295+0000 7fcdf1ba6640 1 --2- 172.21.15.116:0/192896671 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcdec0a94a0 0x7fcdec1538e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:24.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.295+0000 7fcdf1ba6640 1 -- 172.21.15.116:0/192896671 >> 172.21.15.116:0/192896671 conn(0x7fcdec0c3b00 msgr2=0x7fcdec0c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:04:24.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.295+0000 7fcdf1ba6640 1 -- 172.21.15.116:0/192896671 shutdown_connections 2024-04-25T00:04:24.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.295+0000 7fcdf1ba6640 1 -- 172.21.15.116:0/192896671 wait complete. 2024-04-25T00:04:24.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.295+0000 7fcdf1ba6640 1 Processor -- start 2024-04-25T00:04:24.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.295+0000 7fcdf1ba6640 1 -- start start 2024-04-25T00:04:24.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.295+0000 7fcdf1ba6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcdec0a94a0 0x7fcdec0ac0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:24.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.295+0000 7fcdf1ba6640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcdec0a99f0 0x7fcdec14afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:24.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.295+0000 7fcdf1ba6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdec150d20 0x7fcdec1407d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:24.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.295+0000 7fcdf1ba6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcdec1544e0 con 0x7fcdec150d20 2024-04-25T00:04:24.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.295+0000 7fcdf1ba6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcdec154360 con 0x7fcdec0a94a0 2024-04-25T00:04:24.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.295+0000 7fcdf1ba6640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fcdec160730 con 0x7fcdec0a99f0 2024-04-25T00:04:24.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.295+0000 7fcdebfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdec150d20 0x7fcdec1407d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:24.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.295+0000 7fcdebfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdec150d20 0x7fcdec1407d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34992/0 (socket says 172.21.15.116:34992) 2024-04-25T00:04:24.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.295+0000 7fcdebfff640 1 -- 172.21.15.116:0/1686426291 learned_addr learned my addr 172.21.15.116:0/1686426291 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:04:24.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.295+0000 7fcde3fff640 1 --2- 172.21.15.116:0/1686426291 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcdec0a99f0 0x7fcdec14afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:24.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.295+0000 7fcdeb7fe640 1 --2- 172.21.15.116:0/1686426291 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcdec0a94a0 0x7fcdec0ac0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:24.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.295+0000 7fcdebfff640 1 -- 172.21.15.116:0/1686426291 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcdec0a99f0 msgr2=0x7fcdec14afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:24.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.295+0000 7fcdebfff640 1 --2- 172.21.15.116:0/1686426291 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcdec0a99f0 0x7fcdec14afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:24.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.295+0000 7fcdebfff640 1 -- 172.21.15.116:0/1686426291 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcdec0a94a0 msgr2=0x7fcdec0ac0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:24.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.295+0000 7fcdebfff640 1 --2- 172.21.15.116:0/1686426291 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcdec0a94a0 0x7fcdec0ac0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:24.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.295+0000 7fcdebfff640 1 -- 172.21.15.116:0/1686426291 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdec14cfc0 con 0x7fcdec150d20 2024-04-25T00:04:24.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.295+0000 7fcdebfff640 1 --2- 172.21.15.116:0/1686426291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdec150d20 0x7fcdec1407d0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fcde401a660 tx=0x7fcde40197d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:24.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.295+0000 7fcde8ff9640 1 -- 172.21.15.116:0/1686426291 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcde40bc020 con 0x7fcdec150d20 2024-04-25T00:04:24.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.295+0000 7fcde8ff9640 1 -- 172.21.15.116:0/1686426291 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcde40b8670 con 0x7fcdec150d20 2024-04-25T00:04:24.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.295+0000 7fcdf1ba6640 1 -- 172.21.15.116:0/1686426291 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdec14d2a0 con 0x7fcdec150d20 2024-04-25T00:04:24.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.295+0000 7fcde8ff9640 1 -- 172.21.15.116:0/1686426291 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcde4009510 con 0x7fcdec150d20 2024-04-25T00:04:24.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.299+0000 7fcdf1ba6640 1 -- 172.21.15.116:0/1686426291 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdec14d860 con 0x7fcdec150d20 2024-04-25T00:04:24.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.299+0000 7fcdf1ba6640 1 -- 172.21.15.116:0/1686426291 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdec068350 con 0x7fcdec150d20 2024-04-25T00:04:24.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.299+0000 7fcde8ff9640 1 -- 172.21.15.116:0/1686426291 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fcde40bf020 con 0x7fcdec150d20 2024-04-25T00:04:24.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.299+0000 7fcde8ff9640 1 --2- 172.21.15.116:0/1686426291 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcdcc0678c0 0x7fcdcc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:24.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.299+0000 7fcde8ff9640 1 -- 172.21.15.116:0/1686426291 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 6924+0+0 (secure 0 0 0) 0x7fcde4120f30 con 0x7fcdec150d20 2024-04-25T00:04:24.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.299+0000 7fcdeb7fe640 1 --2- 172.21.15.116:0/1686426291 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcdcc0678c0 0x7fcdcc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:24.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.299+0000 7fcdeb7fe640 1 --2- 172.21.15.116:0/1686426291 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcdcc0678c0 0x7fcdcc069d80 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fcddc001650 tx=0x7fcddc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:24.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.303+0000 7fcde8ff9640 1 -- 172.21.15.116:0/1686426291 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fcde40c6030 con 0x7fcdec150d20 2024-04-25T00:04:24.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.499+0000 7fcdf1ba6640 1 -- 172.21.15.116:0/1686426291 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcdec0c34f0 con 0x7fcdec150d20 2024-04-25T00:04:24.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.503+0000 7fcde8ff9640 1 -- 172.21.15.116:0/1686426291 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6501 (secure 0 0 0) 0x7fcde4016620 con 0x7fcdec150d20 2024-04-25T00:04:24.503 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:04:24.503 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4879,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/1889314958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1889314958},{"type":"v1","addr":"172.21.15.174:6837","nonce":1889314958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:03:40.629360+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4828":{"gid":4828,"name":"c","rank":0,"incarnation":26,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6834/1649769369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1649769369},{"type":"v1","addr":"172.21.15.116:6834","nonce":1649769369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4828,"qdb_cluster":[4828]},"id":1},{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:03:34.579825+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4735},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4735":{"gid":4735,"name":"a","rank":0,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.116:6835/2004876235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":2004876235},{"type":"v1","addr":"172.21.15.116:6835","nonce":2004876235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4735,"qdb_cluster":[4735]},"id":2}]} 2024-04-25T00:04:24.503 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 31 2024-04-25T00:04:24.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.503+0000 7fcdf1ba6640 1 -- 172.21.15.116:0/1686426291 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcdcc0678c0 msgr2=0x7fcdcc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:24.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.503+0000 7fcdf1ba6640 1 --2- 172.21.15.116:0/1686426291 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcdcc0678c0 0x7fcdcc069d80 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fcddc001650 tx=0x7fcddc009040 comp rx=0 tx=0).stop 2024-04-25T00:04:24.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.503+0000 7fcdf1ba6640 1 -- 172.21.15.116:0/1686426291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdec150d20 msgr2=0x7fcdec1407d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:24.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.503+0000 7fcdf1ba6640 1 --2- 172.21.15.116:0/1686426291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdec150d20 0x7fcdec1407d0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fcde401a660 tx=0x7fcde40197d0 comp rx=0 tx=0).stop 2024-04-25T00:04:24.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.503+0000 7fcdf1ba6640 1 -- 172.21.15.116:0/1686426291 shutdown_connections 2024-04-25T00:04:24.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.503+0000 7fcdf1ba6640 1 --2- 172.21.15.116:0/1686426291 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcdcc0678c0 0x7fcdcc069d80 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:24.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.503+0000 7fcdf1ba6640 1 --2- 172.21.15.116:0/1686426291 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdec150d20 0x7fcdec1407d0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:24.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.503+0000 7fcdf1ba6640 1 --2- 172.21.15.116:0/1686426291 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcdec0a99f0 0x7fcdec14afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:24.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.503+0000 7fcdf1ba6640 1 --2- 172.21.15.116:0/1686426291 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcdec0a94a0 0x7fcdec0ac0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:24.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.503+0000 7fcdf1ba6640 1 -- 172.21.15.116:0/1686426291 >> 172.21.15.116:0/1686426291 conn(0x7fcdec0c3b00 msgr2=0x7fcdec146ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:04:24.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.503+0000 7fcdf1ba6640 1 -- 172.21.15.116:0/1686426291 shutdown_connections 2024-04-25T00:04:24.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:24.503+0000 7fcdf1ba6640 1 -- 172.21.15.116:0/1686426291 wait complete. 2024-04-25T00:04:24.520 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:04:26.521 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:04:26.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.651+0000 7f141aa93640 1 Processor -- start 2024-04-25T00:04:26.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.651+0000 7f141aa93640 1 -- start start 2024-04-25T00:04:26.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.655+0000 7f141aa93640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f1414150d20 0x7f14140c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:26.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.655+0000 7f141aa93640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f14140c2f50 0x7f14141538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:26.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.655+0000 7f141aa93640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14140c3910 con 0x7f14140a94a0 2024-04-25T00:04:26.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.655+0000 7f141aa93640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f1414154530 con 0x7f1414150d20 2024-04-25T00:04:26.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.655+0000 7f141aa93640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f14140c16e0 con 0x7f14140c2f50 2024-04-25T00:04:26.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.655+0000 7f1413fff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f14140c2f50 0x7f14141538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:26.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.655+0000 7f141882f640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f1414150d20 0x7f14140c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:26.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.655+0000 7f1413fff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f14140c2f50 0x7f14141538e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:47502/0 (socket says 172.21.15.116:47502) 2024-04-25T00:04:26.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.655+0000 7f1413fff640 1 -- 172.21.15.116:0/4125841098 learned_addr learned my addr 172.21.15.116:0/4125841098 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:04:26.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.655+0000 7f1413fff640 1 -- 172.21.15.116:0/4125841098 >> v2:172.21.15.174:3300/0 conn(0x7f1414150d20 msgr2=0x7f14140c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:04:26.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.655+0000 7f1413fff640 1 --2- 172.21.15.116:0/4125841098 >> v2:172.21.15.174:3300/0 conn(0x7f1414150d20 0x7f14140c2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:26.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.655+0000 7f1413fff640 1 -- 172.21.15.116:0/4125841098 >> v1:172.21.15.174:6789/0 conn(0x7f14140a94a0 legacy=0x7f1414153e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:26.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.655+0000 7f1413fff640 1 -- 172.21.15.116:0/4125841098 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14140c1400 con 0x7f14140c2f50 2024-04-25T00:04:26.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.659+0000 7f1413fff640 1 --2- 172.21.15.116:0/4125841098 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f14140c2f50 0x7f14141538e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1408009870 tx=0x7f140801c560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e4bd356ef11f30f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:26.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.659+0000 7f14137fe640 1 -- 172.21.15.116:0/4125841098 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14080a7070 con 0x7f14140c2f50 2024-04-25T00:04:26.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.659+0000 7f14137fe640 1 -- 172.21.15.116:0/4125841098 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14080b0070 con 0x7f14140c2f50 2024-04-25T00:04:26.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.659+0000 7f14137fe640 1 -- 172.21.15.116:0/4125841098 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14080ab890 con 0x7f14140c2f50 2024-04-25T00:04:26.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.659+0000 7f141aa93640 1 -- 172.21.15.116:0/4125841098 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f14140c2f50 msgr2=0x7f14141538e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:26.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.659+0000 7f141aa93640 1 --2- 172.21.15.116:0/4125841098 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f14140c2f50 0x7f14141538e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1408009870 tx=0x7f140801c560 comp rx=0 tx=0).stop 2024-04-25T00:04:26.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.659+0000 7f141aa93640 1 -- 172.21.15.116:0/4125841098 shutdown_connections 2024-04-25T00:04:26.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.659+0000 7f141aa93640 1 --2- 172.21.15.116:0/4125841098 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f14140c2f50 0x7f14141538e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:26.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.659+0000 7f141aa93640 1 --2- 172.21.15.116:0/4125841098 >> v2:172.21.15.174:3300/0 conn(0x7f1414150d20 0x7f14140c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:26.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.659+0000 7f141aa93640 1 -- 172.21.15.116:0/4125841098 >> 172.21.15.116:0/4125841098 conn(0x7f14140c3b00 msgr2=0x7f14140c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:04:26.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.659+0000 7f141aa93640 1 -- 172.21.15.116:0/4125841098 shutdown_connections 2024-04-25T00:04:26.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.659+0000 7f141aa93640 1 -- 172.21.15.116:0/4125841098 wait complete. 2024-04-25T00:04:26.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.659+0000 7f141aa93640 1 Processor -- start 2024-04-25T00:04:26.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.659+0000 7f141aa93640 1 -- start start 2024-04-25T00:04:26.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.659+0000 7f141aa93640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14140a94a0 0x7f14140c5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:26.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.659+0000 7f141aa93640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1414150d20 0x7f14141efd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:26.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.659+0000 7f141aa93640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f14140c69f0 0x7f14141f02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:26.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.659+0000 7f141aa93640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f14141608b0 con 0x7f14140a94a0 2024-04-25T00:04:26.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.659+0000 7f141aa93640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1414160730 con 0x7f1414150d20 2024-04-25T00:04:26.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.659+0000 7f141aa93640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f1414160a30 con 0x7f14140c69f0 2024-04-25T00:04:26.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.659+0000 7f141882f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14140a94a0 0x7f14140c5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:26.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.659+0000 7f141882f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14140a94a0 0x7f14140c5ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55084/0 (socket says 172.21.15.116:55084) 2024-04-25T00:04:26.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.659+0000 7f141882f640 1 -- 172.21.15.116:0/337967485 learned_addr learned my addr 172.21.15.116:0/337967485 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:04:26.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.659+0000 7f1413fff640 1 --2- 172.21.15.116:0/337967485 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1414150d20 0x7f14141efd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:26.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.659+0000 7f1419030640 1 --2- 172.21.15.116:0/337967485 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f14140c69f0 0x7f14141f02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:26.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.659+0000 7f141882f640 1 -- 172.21.15.116:0/337967485 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f14140c69f0 msgr2=0x7f14141f02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:26.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.659+0000 7f141882f640 1 --2- 172.21.15.116:0/337967485 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f14140c69f0 0x7f14141f02c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:26.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.659+0000 7f141882f640 1 -- 172.21.15.116:0/337967485 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1414150d20 msgr2=0x7f14141efd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:26.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.659+0000 7f141882f640 1 --2- 172.21.15.116:0/337967485 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1414150d20 0x7f14141efd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:26.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.659+0000 7f141882f640 1 -- 172.21.15.116:0/337967485 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f141414b220 con 0x7f14140a94a0 2024-04-25T00:04:26.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.659+0000 7f141882f640 1 --2- 172.21.15.116:0/337967485 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14140a94a0 0x7f14140c5ea0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f140400e870 tx=0x7f140400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:26.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.663+0000 7f14117fa640 1 -- 172.21.15.116:0/337967485 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1404017070 con 0x7f14140a94a0 2024-04-25T00:04:26.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.663+0000 7f14117fa640 1 -- 172.21.15.116:0/337967485 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f140400c940 con 0x7f14140a94a0 2024-04-25T00:04:26.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.663+0000 7f141aa93640 1 -- 172.21.15.116:0/337967485 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f141414b500 con 0x7f14140a94a0 2024-04-25T00:04:26.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.663+0000 7f14117fa640 1 -- 172.21.15.116:0/337967485 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f140400ccf0 con 0x7f14140a94a0 2024-04-25T00:04:26.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.663+0000 7f141aa93640 1 -- 172.21.15.116:0/337967485 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14141ef860 con 0x7f14140a94a0 2024-04-25T00:04:26.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.663+0000 7f141aa93640 1 -- 172.21.15.116:0/337967485 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1414068350 con 0x7f14140a94a0 2024-04-25T00:04:26.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.663+0000 7f14117fa640 1 -- 172.21.15.116:0/337967485 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f140401a070 con 0x7f14140a94a0 2024-04-25T00:04:26.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.663+0000 7f14117fa640 1 --2- 172.21.15.116:0/337967485 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f14000678c0 0x7f1400069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:26.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.663+0000 7f14117fa640 1 -- 172.21.15.116:0/337967485 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 6924+0+0 (secure 0 0 0) 0x7f14040863d0 con 0x7f14140a94a0 2024-04-25T00:04:26.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.663+0000 7f1413fff640 1 --2- 172.21.15.116:0/337967485 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f14000678c0 0x7f1400069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:26.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.663+0000 7f1413fff640 1 --2- 172.21.15.116:0/337967485 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f14000678c0 0x7f1400069d80 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f14080af000 tx=0x7f14080a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:26.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.667+0000 7f14117fa640 1 -- 172.21.15.116:0/337967485 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f1404056a50 con 0x7f14140a94a0 2024-04-25T00:04:26.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.863+0000 7f141aa93640 1 -- 172.21.15.116:0/337967485 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f14140c34f0 con 0x7f14140a94a0 2024-04-25T00:04:26.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.867+0000 7f14117fa640 1 -- 172.21.15.116:0/337967485 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6501 (secure 0 0 0) 0x7f1404056350 con 0x7f14140a94a0 2024-04-25T00:04:26.867 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:04:26.868 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4879,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/1889314958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1889314958},{"type":"v1","addr":"172.21.15.174:6837","nonce":1889314958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:03:40.629360+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4828":{"gid":4828,"name":"c","rank":0,"incarnation":26,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6834/1649769369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1649769369},{"type":"v1","addr":"172.21.15.116:6834","nonce":1649769369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4828,"qdb_cluster":[4828]},"id":1},{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:03:34.579825+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4735},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4735":{"gid":4735,"name":"a","rank":0,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.116:6835/2004876235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":2004876235},{"type":"v1","addr":"172.21.15.116:6835","nonce":2004876235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4735,"qdb_cluster":[4735]},"id":2}]} 2024-04-25T00:04:26.868 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 31 2024-04-25T00:04:26.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.867+0000 7f141aa93640 1 -- 172.21.15.116:0/337967485 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f14000678c0 msgr2=0x7f1400069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:26.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.867+0000 7f141aa93640 1 --2- 172.21.15.116:0/337967485 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f14000678c0 0x7f1400069d80 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f14080af000 tx=0x7f14080a7000 comp rx=0 tx=0).stop 2024-04-25T00:04:26.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.867+0000 7f141aa93640 1 -- 172.21.15.116:0/337967485 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14140a94a0 msgr2=0x7f14140c5ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:26.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.867+0000 7f141aa93640 1 --2- 172.21.15.116:0/337967485 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14140a94a0 0x7f14140c5ea0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f140400e870 tx=0x7f140400ec70 comp rx=0 tx=0).stop 2024-04-25T00:04:26.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.867+0000 7f141aa93640 1 -- 172.21.15.116:0/337967485 shutdown_connections 2024-04-25T00:04:26.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.867+0000 7f141aa93640 1 --2- 172.21.15.116:0/337967485 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f14000678c0 0x7f1400069d80 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:26.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.867+0000 7f141aa93640 1 --2- 172.21.15.116:0/337967485 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f14140c69f0 0x7f14141f02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:26.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.867+0000 7f141aa93640 1 --2- 172.21.15.116:0/337967485 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1414150d20 0x7f14141efd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:26.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.867+0000 7f141aa93640 1 --2- 172.21.15.116:0/337967485 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14140a94a0 0x7f14140c5ea0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:26.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.867+0000 7f141aa93640 1 -- 172.21.15.116:0/337967485 >> 172.21.15.116:0/337967485 conn(0x7f14140c3b00 msgr2=0x7f141414ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:04:26.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.867+0000 7f141aa93640 1 -- 172.21.15.116:0/337967485 shutdown_connections 2024-04-25T00:04:26.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:26.867+0000 7f141aa93640 1 -- 172.21.15.116:0/337967485 wait complete. 2024-04-25T00:04:26.886 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:04:28.887 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:04:29.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.015+0000 7f56685d1640 1 Processor -- start 2024-04-25T00:04:29.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.015+0000 7f56685d1640 1 -- start start 2024-04-25T00:04:29.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.019+0000 7f56685d1640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f566014c720 0x7f56601563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:29.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.019+0000 7f56685d1640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56600a94a0 0x7f56600c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:29.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.019+0000 7f56685d1640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5660071220 con 0x7f56600c2f50 2024-04-25T00:04:29.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.019+0000 7f56685d1640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f5660153460 con 0x7f566014c720 2024-04-25T00:04:29.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.019+0000 7f56685d1640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f5660152420 con 0x7f56600a94a0 2024-04-25T00:04:29.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.019+0000 7f566636d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f566014c720 0x7f56601563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:29.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.019+0000 7f5666b6e640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f56600c2f50 0x7f5660071680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:41990/0 (socket says 172.21.15.116:41990) 2024-04-25T00:04:29.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.019+0000 7f5666b6e640 1 -- 172.21.15.116:0/2687504148 learned_addr learned my addr 172.21.15.116:0/2687504148 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:04:29.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.019+0000 7f566636d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f566014c720 0x7f56601563d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55094/0 (socket says 172.21.15.116:55094) 2024-04-25T00:04:29.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.019+0000 7f5665b6c640 1 --2- 172.21.15.116:0/2687504148 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56600a94a0 0x7f56600c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:29.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.019+0000 7f566636d640 1 -- 172.21.15.116:0/2687504148 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56600a94a0 msgr2=0x7f56600c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:04:29.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.019+0000 7f566636d640 1 --2- 172.21.15.116:0/2687504148 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56600a94a0 0x7f56600c2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:29.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.019+0000 7f566636d640 1 -- 172.21.15.116:0/2687504148 >> v1:172.21.15.116:6789/0 conn(0x7f56600c2f50 legacy=0x7f5660071680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:29.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.019+0000 7f566636d640 1 -- 172.21.15.116:0/2687504148 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f566014cb00 con 0x7f566014c720 2024-04-25T00:04:29.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.019+0000 7f566536b640 1 -- 172.21.15.116:0/2687504148 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 721348425 0 0) 0x7f5660071220 con 0x7f56600c2f50 2024-04-25T00:04:29.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.019+0000 7f566636d640 1 --2- 172.21.15.116:0/2687504148 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f566014c720 0x7f56601563d0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f565c00ba60 tx=0x7f565c01eab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba0aed415e952dc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:29.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.019+0000 7f566536b640 1 -- 172.21.15.116:0/2687504148 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f565c0b4070 con 0x7f566014c720 2024-04-25T00:04:29.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.019+0000 7f566536b640 1 -- 172.21.15.116:0/2687504148 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f565c0a7aa0 con 0x7f566014c720 2024-04-25T00:04:29.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.019+0000 7f566536b640 1 -- 172.21.15.116:0/2687504148 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f565c0af440 con 0x7f566014c720 2024-04-25T00:04:29.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.023+0000 7f56685d1640 1 -- 172.21.15.116:0/2687504148 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f566014c720 msgr2=0x7f56601563d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:29.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.023+0000 7f56685d1640 1 --2- 172.21.15.116:0/2687504148 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f566014c720 0x7f56601563d0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f565c00ba60 tx=0x7f565c01eab0 comp rx=0 tx=0).stop 2024-04-25T00:04:29.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.023+0000 7f56685d1640 1 -- 172.21.15.116:0/2687504148 shutdown_connections 2024-04-25T00:04:29.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.023+0000 7f56685d1640 1 --2- 172.21.15.116:0/2687504148 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56600a94a0 0x7f56600c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:29.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.023+0000 7f56685d1640 1 --2- 172.21.15.116:0/2687504148 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f566014c720 0x7f56601563d0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:29.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.023+0000 7f56685d1640 1 -- 172.21.15.116:0/2687504148 >> 172.21.15.116:0/2687504148 conn(0x7f56600c3b00 msgr2=0x7f56600c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:04:29.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.023+0000 7f56685d1640 1 -- 172.21.15.116:0/2687504148 shutdown_connections 2024-04-25T00:04:29.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.023+0000 7f56685d1640 1 -- 172.21.15.116:0/2687504148 wait complete. 2024-04-25T00:04:29.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.023+0000 7f56685d1640 1 Processor -- start 2024-04-25T00:04:29.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.023+0000 7f56685d1640 1 -- start start 2024-04-25T00:04:29.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.023+0000 7f56685d1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56600a94a0 0x7f56600acce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:29.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.023+0000 7f56685d1640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56600c2f50 0x7f56600c0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:29.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.023+0000 7f56685d1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f566014c720 0x7f56601537f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:29.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.023+0000 7f56685d1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5660152f40 con 0x7f56600a94a0 2024-04-25T00:04:29.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.023+0000 7f56685d1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5660152c60 con 0x7f566014c720 2024-04-25T00:04:29.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.023+0000 7f56685d1640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f566006d460 con 0x7f56600c2f50 2024-04-25T00:04:29.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.023+0000 7f566636d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56600a94a0 0x7f56600acce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:29.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.023+0000 7f566636d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56600a94a0 0x7f56600acce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55096/0 (socket says 172.21.15.116:55096) 2024-04-25T00:04:29.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.023+0000 7f566636d640 1 -- 172.21.15.116:0/1295503786 learned_addr learned my addr 172.21.15.116:0/1295503786 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:04:29.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.023+0000 7f5665b6c640 1 --2- 172.21.15.116:0/1295503786 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56600c2f50 0x7f56600c0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:29.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.023+0000 7f5666b6e640 1 --2- 172.21.15.116:0/1295503786 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f566014c720 0x7f56601537f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:29.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.023+0000 7f566636d640 1 -- 172.21.15.116:0/1295503786 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56600c2f50 msgr2=0x7f56600c0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:29.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.023+0000 7f566636d640 1 --2- 172.21.15.116:0/1295503786 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56600c2f50 0x7f56600c0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:29.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.023+0000 7f566636d640 1 -- 172.21.15.116:0/1295503786 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f566014c720 msgr2=0x7f56601537f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:29.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.023+0000 7f566636d640 1 --2- 172.21.15.116:0/1295503786 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f566014c720 0x7f56601537f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:29.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.023+0000 7f566636d640 1 -- 172.21.15.116:0/1295503786 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5660155260 con 0x7f56600a94a0 2024-04-25T00:04:29.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.023+0000 7f5666b6e640 1 --2- 172.21.15.116:0/1295503786 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f566014c720 0x7f56601537f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:04:29.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.023+0000 7f5665b6c640 1 --2- 172.21.15.116:0/1295503786 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56600c2f50 0x7f56600c0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:04:29.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.023+0000 7f566636d640 1 --2- 172.21.15.116:0/1295503786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56600a94a0 0x7f56600acce0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f565c01c6e0 tx=0x7f565c01b850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:29.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.023+0000 7f5656ffd640 1 -- 172.21.15.116:0/1295503786 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f565c0b4040 con 0x7f56600a94a0 2024-04-25T00:04:29.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.023+0000 7f5656ffd640 1 -- 172.21.15.116:0/1295503786 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f565c0bd070 con 0x7f56600a94a0 2024-04-25T00:04:29.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.023+0000 7f5656ffd640 1 -- 172.21.15.116:0/1295503786 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f565c0afac0 con 0x7f56600a94a0 2024-04-25T00:04:29.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.023+0000 7f56685d1640 1 -- 172.21.15.116:0/1295503786 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5660154f80 con 0x7f56600a94a0 2024-04-25T00:04:29.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.023+0000 7f56685d1640 1 -- 172.21.15.116:0/1295503786 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5660146550 con 0x7f56600a94a0 2024-04-25T00:04:29.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.027+0000 7f56685d1640 1 -- 172.21.15.116:0/1295503786 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5660068350 con 0x7f56600a94a0 2024-04-25T00:04:29.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.027+0000 7f5656ffd640 1 -- 172.21.15.116:0/1295503786 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f565c0afc60 con 0x7f56600a94a0 2024-04-25T00:04:29.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.027+0000 7f5656ffd640 1 --2- 172.21.15.116:0/1295503786 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f56440678f0 0x7f5644069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:29.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.027+0000 7f5656ffd640 1 -- 172.21.15.116:0/1295503786 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 6924+0+0 (secure 0 0 0) 0x7f565c120890 con 0x7f56600a94a0 2024-04-25T00:04:29.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.027+0000 7f5665b6c640 1 --2- 172.21.15.116:0/1295503786 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f56440678f0 0x7f5644069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:29.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.027+0000 7f5665b6c640 1 --2- 172.21.15.116:0/1295503786 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f56440678f0 0x7f5644069db0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f56600c0ed0 tx=0x7f5650008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:29.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.031+0000 7f5656ffd640 1 -- 172.21.15.116:0/1295503786 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f565c018760 con 0x7f56600a94a0 2024-04-25T00:04:29.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.223+0000 7f56685d1640 1 -- 172.21.15.116:0/1295503786 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5660150510 con 0x7f56600a94a0 2024-04-25T00:04:29.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.227+0000 7f5656ffd640 1 -- 172.21.15.116:0/1295503786 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6501 (secure 0 0 0) 0x7f565c0f0e70 con 0x7f56600a94a0 2024-04-25T00:04:29.227 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:04:29.227 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4879,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/1889314958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1889314958},{"type":"v1","addr":"172.21.15.174:6837","nonce":1889314958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:03:40.629360+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4828":{"gid":4828,"name":"c","rank":0,"incarnation":26,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6834/1649769369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1649769369},{"type":"v1","addr":"172.21.15.116:6834","nonce":1649769369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4828,"qdb_cluster":[4828]},"id":1},{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:03:34.579825+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4735},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4735":{"gid":4735,"name":"a","rank":0,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.116:6835/2004876235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":2004876235},{"type":"v1","addr":"172.21.15.116:6835","nonce":2004876235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4735,"qdb_cluster":[4735]},"id":2}]} 2024-04-25T00:04:29.227 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 31 2024-04-25T00:04:29.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.227+0000 7f56685d1640 1 -- 172.21.15.116:0/1295503786 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f56440678f0 msgr2=0x7f5644069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:29.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.227+0000 7f56685d1640 1 --2- 172.21.15.116:0/1295503786 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f56440678f0 0x7f5644069db0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f56600c0ed0 tx=0x7f5650008040 comp rx=0 tx=0).stop 2024-04-25T00:04:29.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.227+0000 7f56685d1640 1 -- 172.21.15.116:0/1295503786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56600a94a0 msgr2=0x7f56600acce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:29.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.227+0000 7f56685d1640 1 --2- 172.21.15.116:0/1295503786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56600a94a0 0x7f56600acce0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f565c01c6e0 tx=0x7f565c01b850 comp rx=0 tx=0).stop 2024-04-25T00:04:29.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.227+0000 7f56685d1640 1 -- 172.21.15.116:0/1295503786 shutdown_connections 2024-04-25T00:04:29.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.227+0000 7f56685d1640 1 --2- 172.21.15.116:0/1295503786 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f56440678f0 0x7f5644069db0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:29.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.227+0000 7f56685d1640 1 --2- 172.21.15.116:0/1295503786 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f566014c720 0x7f56601537f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:29.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.227+0000 7f56685d1640 1 --2- 172.21.15.116:0/1295503786 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56600c2f50 0x7f56600c0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:29.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.227+0000 7f56685d1640 1 --2- 172.21.15.116:0/1295503786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56600a94a0 0x7f56600acce0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:29.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.227+0000 7f56685d1640 1 -- 172.21.15.116:0/1295503786 >> 172.21.15.116:0/1295503786 conn(0x7f56600c3b00 msgr2=0x7f56600c3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:04:29.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.227+0000 7f56685d1640 1 -- 172.21.15.116:0/1295503786 shutdown_connections 2024-04-25T00:04:29.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:29.227+0000 7f56685d1640 1 -- 172.21.15.116:0/1295503786 wait complete. 2024-04-25T00:04:29.245 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:04:30.939 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:04:31.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.067+0000 7f60bf6d4640 1 Processor -- start 2024-04-25T00:04:31.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.067+0000 7f60bf6d4640 1 -- start start 2024-04-25T00:04:31.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.071+0000 7f60bf6d4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f60b813e310 0x7f60b80c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:31.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.071+0000 7f60bf6d4640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f60b813ed60 0x7f60b81560d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:31.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.071+0000 7f60bf6d4640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60b806d200 con 0x7f60b80c2f50 2024-04-25T00:04:31.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.071+0000 7f60bf6d4640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f60b8070d40 con 0x7f60b813ed60 2024-04-25T00:04:31.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.071+0000 7f60bf6d4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f60b8071020 con 0x7f60b813e310 2024-04-25T00:04:31.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.071+0000 7f60bdc71640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f60b813ed60 0x7f60b81560d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:31.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.071+0000 7f60bcc6f640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f60b80c2f50 0x7f60b81559c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:41998/0 (socket says 172.21.15.116:41998) 2024-04-25T00:04:31.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.071+0000 7f60bcc6f640 1 -- 172.21.15.116:0/3211213962 learned_addr learned my addr 172.21.15.116:0/3211213962 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:04:31.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.071+0000 7f60bd470640 1 --2- 172.21.15.116:0/3211213962 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f60b813e310 0x7f60b80c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:31.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.071+0000 7f60bdc71640 1 -- 172.21.15.116:0/3211213962 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f60b813e310 msgr2=0x7f60b80c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:04:31.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.071+0000 7f60bdc71640 1 --2- 172.21.15.116:0/3211213962 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f60b813e310 0x7f60b80c2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:31.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.071+0000 7f60bdc71640 1 -- 172.21.15.116:0/3211213962 >> v1:172.21.15.116:6789/0 conn(0x7f60b80c2f50 legacy=0x7f60b81559c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:31.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.071+0000 7f60bdc71640 1 -- 172.21.15.116:0/3211213962 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60b8156610 con 0x7f60b813ed60 2024-04-25T00:04:31.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.071+0000 7f60bdc71640 1 --2- 172.21.15.116:0/3211213962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60b813ed60 0x7f60b81560d0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f60b0009d90 tx=0x7f60b001c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eea58fadb4651794 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:31.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.071+0000 7f60affff640 1 -- 172.21.15.116:0/3211213962 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60b00b2070 con 0x7f60b813ed60 2024-04-25T00:04:31.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.071+0000 7f60affff640 1 -- 172.21.15.116:0/3211213962 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60b00a5820 con 0x7f60b813ed60 2024-04-25T00:04:31.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.071+0000 7f60affff640 1 -- 172.21.15.116:0/3211213962 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60b00a5bf0 con 0x7f60b813ed60 2024-04-25T00:04:31.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.075+0000 7f60bf6d4640 1 -- 172.21.15.116:0/3211213962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60b813ed60 msgr2=0x7f60b81560d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:31.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.075+0000 7f60bf6d4640 1 --2- 172.21.15.116:0/3211213962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60b813ed60 0x7f60b81560d0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f60b0009d90 tx=0x7f60b001c850 comp rx=0 tx=0).stop 2024-04-25T00:04:31.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.075+0000 7f60bf6d4640 1 -- 172.21.15.116:0/3211213962 shutdown_connections 2024-04-25T00:04:31.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.075+0000 7f60bf6d4640 1 --2- 172.21.15.116:0/3211213962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60b813ed60 0x7f60b81560d0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:31.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.075+0000 7f60bf6d4640 1 --2- 172.21.15.116:0/3211213962 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f60b813e310 0x7f60b80c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:31.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.075+0000 7f60bf6d4640 1 -- 172.21.15.116:0/3211213962 >> 172.21.15.116:0/3211213962 conn(0x7f60b80c3b00 msgr2=0x7f60b80c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:04:31.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.075+0000 7f60bf6d4640 1 -- 172.21.15.116:0/3211213962 shutdown_connections 2024-04-25T00:04:31.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.075+0000 7f60bf6d4640 1 -- 172.21.15.116:0/3211213962 wait complete. 2024-04-25T00:04:31.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.075+0000 7f60bf6d4640 1 Processor -- start 2024-04-25T00:04:31.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.075+0000 7f60bf6d4640 1 -- start start 2024-04-25T00:04:31.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.075+0000 7f60bf6d4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60b80c2f50 0x7f60b8130250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:31.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.075+0000 7f60bf6d4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f60b813e310 0x7f60b8132420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:31.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.075+0000 7f60bf6d4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f60b813ed60 0x7f60b81526a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:31.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.075+0000 7f60bf6d4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f60b8162b70 con 0x7f60b80c2f50 2024-04-25T00:04:31.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.075+0000 7f60bf6d4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f60b81629f0 con 0x7f60b813e310 2024-04-25T00:04:31.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.075+0000 7f60bf6d4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f60b8162cf0 con 0x7f60b813ed60 2024-04-25T00:04:31.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.075+0000 7f60bd470640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60b80c2f50 0x7f60b8130250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:31.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.075+0000 7f60bd470640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60b80c2f50 0x7f60b8130250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55122/0 (socket says 172.21.15.116:55122) 2024-04-25T00:04:31.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.075+0000 7f60bd470640 1 -- 172.21.15.116:0/1896203460 learned_addr learned my addr 172.21.15.116:0/1896203460 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:04:31.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.075+0000 7f60bdc71640 1 --2- 172.21.15.116:0/1896203460 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f60b813ed60 0x7f60b81526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:31.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.075+0000 7f60bcc6f640 1 --2- 172.21.15.116:0/1896203460 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f60b813e310 0x7f60b8132420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:31.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.075+0000 7f60bd470640 1 -- 172.21.15.116:0/1896203460 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f60b813ed60 msgr2=0x7f60b81526a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:31.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.075+0000 7f60bd470640 1 --2- 172.21.15.116:0/1896203460 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f60b813ed60 0x7f60b81526a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:31.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.075+0000 7f60bd470640 1 -- 172.21.15.116:0/1896203460 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f60b813e310 msgr2=0x7f60b8132420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:31.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.075+0000 7f60bd470640 1 --2- 172.21.15.116:0/1896203460 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f60b813e310 0x7f60b8132420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:31.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.075+0000 7f60bd470640 1 -- 172.21.15.116:0/1896203460 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60b8155690 con 0x7f60b80c2f50 2024-04-25T00:04:31.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.075+0000 7f60bcc6f640 1 --2- 172.21.15.116:0/1896203460 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f60b813e310 0x7f60b8132420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:04:31.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.075+0000 7f60bdc71640 1 --2- 172.21.15.116:0/1896203460 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f60b813ed60 0x7f60b81526a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:04:31.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.075+0000 7f60bd470640 1 --2- 172.21.15.116:0/1896203460 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60b80c2f50 0x7f60b8130250 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f60a800d830 tx=0x7f60a800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:31.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.079+0000 7f60adffb640 1 -- 172.21.15.116:0/1896203460 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a800b6c0 con 0x7f60b80c2f50 2024-04-25T00:04:31.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.079+0000 7f60adffb640 1 -- 172.21.15.116:0/1896203460 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60a8019070 con 0x7f60b80c2f50 2024-04-25T00:04:31.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.079+0000 7f60bf6d4640 1 -- 172.21.15.116:0/1896203460 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60b81ef810 con 0x7f60b80c2f50 2024-04-25T00:04:31.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.079+0000 7f60adffb640 1 -- 172.21.15.116:0/1896203460 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60a800bae0 con 0x7f60b80c2f50 2024-04-25T00:04:31.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.079+0000 7f60bf6d4640 1 -- 172.21.15.116:0/1896203460 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60b81340f0 con 0x7f60b80c2f50 2024-04-25T00:04:31.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.079+0000 7f60bf6d4640 1 -- 172.21.15.116:0/1896203460 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60b8068350 con 0x7f60b80c2f50 2024-04-25T00:04:31.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.079+0000 7f60adffb640 1 -- 172.21.15.116:0/1896203460 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f60a801e460 con 0x7f60b80c2f50 2024-04-25T00:04:31.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.079+0000 7f60adffb640 1 --2- 172.21.15.116:0/1896203460 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f609c0678c0 0x7f609c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:31.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.079+0000 7f60adffb640 1 -- 172.21.15.116:0/1896203460 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 6924+0+0 (secure 0 0 0) 0x7f60a80868b0 con 0x7f60b80c2f50 2024-04-25T00:04:31.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.079+0000 7f60bcc6f640 1 --2- 172.21.15.116:0/1896203460 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f609c0678c0 0x7f609c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:31.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.079+0000 7f60bcc6f640 1 --2- 172.21.15.116:0/1896203460 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f609c0678c0 0x7f609c069d80 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f60b4007fd0 tx=0x7f60b4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:31.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.083+0000 7f60adffb640 1 -- 172.21.15.116:0/1896203460 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f60a8056f30 con 0x7f60b80c2f50 2024-04-25T00:04:31.246 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:04:31.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.283+0000 7f60bf6d4640 1 -- 172.21.15.116:0/1896203460 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f60b8131c40 con 0x7f60b80c2f50 2024-04-25T00:04:31.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.283+0000 7f60adffb640 1 -- 172.21.15.116:0/1896203460 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6501 (secure 0 0 0) 0x7f60a8056830 con 0x7f60b80c2f50 2024-04-25T00:04:31.285 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:04:31.285 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4879,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/1889314958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1889314958},{"type":"v1","addr":"172.21.15.174:6837","nonce":1889314958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:03:40.629360+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4828":{"gid":4828,"name":"c","rank":0,"incarnation":26,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6834/1649769369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1649769369},{"type":"v1","addr":"172.21.15.116:6834","nonce":1649769369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4828,"qdb_cluster":[4828]},"id":1},{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:03:34.579825+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4735},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4735":{"gid":4735,"name":"a","rank":0,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.116:6835/2004876235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":2004876235},{"type":"v1","addr":"172.21.15.116:6835","nonce":2004876235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4735,"qdb_cluster":[4735]},"id":2}]} 2024-04-25T00:04:31.285 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 31 2024-04-25T00:04:31.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.287+0000 7f60bf6d4640 1 -- 172.21.15.116:0/1896203460 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f609c0678c0 msgr2=0x7f609c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:31.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.287+0000 7f60bf6d4640 1 --2- 172.21.15.116:0/1896203460 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f609c0678c0 0x7f609c069d80 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f60b4007fd0 tx=0x7f60b4009040 comp rx=0 tx=0).stop 2024-04-25T00:04:31.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.287+0000 7f60bf6d4640 1 -- 172.21.15.116:0/1896203460 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60b80c2f50 msgr2=0x7f60b8130250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:31.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.287+0000 7f60bf6d4640 1 --2- 172.21.15.116:0/1896203460 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60b80c2f50 0x7f60b8130250 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f60a800d830 tx=0x7f60a800dcf0 comp rx=0 tx=0).stop 2024-04-25T00:04:31.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.287+0000 7f60bf6d4640 1 -- 172.21.15.116:0/1896203460 shutdown_connections 2024-04-25T00:04:31.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.287+0000 7f60bf6d4640 1 --2- 172.21.15.116:0/1896203460 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f609c0678c0 0x7f609c069d80 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:31.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.287+0000 7f60bf6d4640 1 --2- 172.21.15.116:0/1896203460 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f60b813ed60 0x7f60b81526a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:31.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.287+0000 7f60bf6d4640 1 --2- 172.21.15.116:0/1896203460 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f60b813e310 0x7f60b8132420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:31.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.287+0000 7f60bf6d4640 1 --2- 172.21.15.116:0/1896203460 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f60b80c2f50 0x7f60b8130250 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:31.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.287+0000 7f60bf6d4640 1 -- 172.21.15.116:0/1896203460 >> 172.21.15.116:0/1896203460 conn(0x7f60b80c3b00 msgr2=0x7f60b806d7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:04:31.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.287+0000 7f60bf6d4640 1 -- 172.21.15.116:0/1896203460 shutdown_connections 2024-04-25T00:04:31.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.287+0000 7f60bf6d4640 1 -- 172.21.15.116:0/1896203460 wait complete. 2024-04-25T00:04:31.304 INFO:tasks.mds_thrash.fs.[b]:kill mds.a (rank=0) 2024-04-25T00:04:31.305 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-25T00:04:31.305 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:04:31.359 INFO:tasks.ceph.mds.a:Stopped 2024-04-25T00:04:31.359 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:04:31.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.363+0000 7f699b06c640 1 Processor -- start 2024-04-25T00:04:31.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.363+0000 7f699b06c640 1 -- start start 2024-04-25T00:04:31.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.363+0000 7f699b06c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f69940c3f70 0x7f69941557c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:31.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.363+0000 7f699b06c640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f69940c3a20 0x7f6994155d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:31.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.363+0000 7f699b06c640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6994153220 con 0x7f6994152e40 2024-04-25T00:04:31.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.363+0000 7f699b06c640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f69940c4c90 con 0x7f69940c3a20 2024-04-25T00:04:31.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.363+0000 7f699b06c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6994156240 con 0x7f69940c3f70 2024-04-25T00:04:31.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.363+0000 7f6993fff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f69940c3f70 0x7f69941557c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:31.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.363+0000 7f6999609640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f69940c3a20 0x7f6994155d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:31.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.363+0000 7f6998e08640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f6994152e40 0x7f69940710b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:33806/0 (socket says 172.21.15.116:33806) 2024-04-25T00:04:31.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.363+0000 7f6998e08640 1 -- 172.21.15.116:0/1916267757 learned_addr learned my addr 172.21.15.116:0/1916267757 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:04:31.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.363+0000 7f6999609640 1 -- 172.21.15.116:0/1916267757 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f69940c3f70 msgr2=0x7f69941557c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:04:31.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.363+0000 7f6999609640 1 --2- 172.21.15.116:0/1916267757 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f69940c3f70 0x7f69941557c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:31.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.363+0000 7f6999609640 1 -- 172.21.15.116:0/1916267757 >> v1:172.21.15.174:6789/0 conn(0x7f6994152e40 legacy=0x7f69940710b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:31.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.363+0000 7f6999609640 1 -- 172.21.15.116:0/1916267757 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69940c7a90 con 0x7f69940c3a20 2024-04-25T00:04:31.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.363+0000 7f6993fff640 1 --2- 172.21.15.116:0/1916267757 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f69940c3f70 0x7f69941557c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:04:31.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.367+0000 7f6999609640 1 --2- 172.21.15.116:0/1916267757 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69940c3a20 0x7f6994155d00 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f698c009d90 tx=0x7f698c01c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f943e4b5cbad106e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:31.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.367+0000 7f69937fe640 1 -- 172.21.15.116:0/1916267757 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f698c0b2070 con 0x7f69940c3a20 2024-04-25T00:04:31.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.367+0000 7f69937fe640 1 -- 172.21.15.116:0/1916267757 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f698c0a5970 con 0x7f69940c3a20 2024-04-25T00:04:31.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.367+0000 7f69937fe640 1 -- 172.21.15.116:0/1916267757 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f698c0a5d40 con 0x7f69940c3a20 2024-04-25T00:04:31.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.367+0000 7f699b06c640 1 -- 172.21.15.116:0/1916267757 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69940c3a20 msgr2=0x7f6994155d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:31.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.367+0000 7f699b06c640 1 --2- 172.21.15.116:0/1916267757 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69940c3a20 0x7f6994155d00 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f698c009d90 tx=0x7f698c01c8b0 comp rx=0 tx=0).stop 2024-04-25T00:04:31.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.367+0000 7f699b06c640 1 -- 172.21.15.116:0/1916267757 shutdown_connections 2024-04-25T00:04:31.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.367+0000 7f699b06c640 1 --2- 172.21.15.116:0/1916267757 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69940c3a20 0x7f6994155d00 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:31.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.367+0000 7f699b06c640 1 --2- 172.21.15.116:0/1916267757 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f69940c3f70 0x7f69941557c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:31.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.367+0000 7f699b06c640 1 -- 172.21.15.116:0/1916267757 >> 172.21.15.116:0/1916267757 conn(0x7f69940c9160 msgr2=0x7f69940c9570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:04:31.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.367+0000 7f699b06c640 1 -- 172.21.15.116:0/1916267757 shutdown_connections 2024-04-25T00:04:31.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.367+0000 7f699b06c640 1 -- 172.21.15.116:0/1916267757 wait complete. 2024-04-25T00:04:31.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.367+0000 7f699b06c640 1 Processor -- start 2024-04-25T00:04:31.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.367+0000 7f699b06c640 1 -- start start 2024-04-25T00:04:31.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.371+0000 7f699b06c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f69940c3a20 0x7f699406f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:31.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.371+0000 7f699b06c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69940c3f70 0x7f6994154f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:31.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.371+0000 7f699b06c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6994152e40 0x7f6994144db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:31.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.371+0000 7f699b06c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6994071940 con 0x7f69940c3f70 2024-04-25T00:04:31.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.371+0000 7f699b06c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f69940717c0 con 0x7f6994152e40 2024-04-25T00:04:31.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.371+0000 7f699b06c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f69940c5100 con 0x7f69940c3a20 2024-04-25T00:04:31.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.371+0000 7f6993fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69940c3f70 0x7f6994154f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:31.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.371+0000 7f6993fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69940c3f70 0x7f6994154f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55128/0 (socket says 172.21.15.116:55128) 2024-04-25T00:04:31.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.371+0000 7f6993fff640 1 -- 172.21.15.116:0/2971871317 learned_addr learned my addr 172.21.15.116:0/2971871317 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:04:31.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.371+0000 7f6998e08640 1 --2- 172.21.15.116:0/2971871317 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f69940c3a20 0x7f699406f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:31.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.371+0000 7f6999609640 1 --2- 172.21.15.116:0/2971871317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6994152e40 0x7f6994144db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:31.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.371+0000 7f6993fff640 1 -- 172.21.15.116:0/2971871317 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f69940c3a20 msgr2=0x7f699406f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:31.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.371+0000 7f6993fff640 1 --2- 172.21.15.116:0/2971871317 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f69940c3a20 0x7f699406f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:31.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.371+0000 7f6993fff640 1 -- 172.21.15.116:0/2971871317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6994152e40 msgr2=0x7f6994144db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:31.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.371+0000 7f6993fff640 1 --2- 172.21.15.116:0/2971871317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6994152e40 0x7f6994144db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:31.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.371+0000 7f6993fff640 1 -- 172.21.15.116:0/2971871317 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6994146010 con 0x7f69940c3f70 2024-04-25T00:04:31.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.371+0000 7f6998e08640 1 --2- 172.21.15.116:0/2971871317 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f69940c3a20 0x7f699406f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:04:31.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.371+0000 7f6993fff640 1 --2- 172.21.15.116:0/2971871317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69940c3f70 0x7f6994154f10 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f698800ebe0 tx=0x7f698800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:31.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.371+0000 7f69917fa640 1 -- 172.21.15.116:0/2971871317 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6988017070 con 0x7f69940c3f70 2024-04-25T00:04:31.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.371+0000 7f69917fa640 1 -- 172.21.15.116:0/2971871317 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f698800cbd0 con 0x7f69940c3f70 2024-04-25T00:04:31.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.371+0000 7f699b06c640 1 -- 172.21.15.116:0/2971871317 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69941462f0 con 0x7f69940c3f70 2024-04-25T00:04:31.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.371+0000 7f69917fa640 1 -- 172.21.15.116:0/2971871317 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6988013470 con 0x7f69940c3f70 2024-04-25T00:04:31.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.371+0000 7f699b06c640 1 -- 172.21.15.116:0/2971871317 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69941468b0 con 0x7f69940c3f70 2024-04-25T00:04:31.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.371+0000 7f699b06c640 1 -- 172.21.15.116:0/2971871317 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6994068a60 con 0x7f69940c3f70 2024-04-25T00:04:31.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.375+0000 7f69917fa640 1 -- 172.21.15.116:0/2971871317 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f698801a070 con 0x7f69940c3f70 2024-04-25T00:04:31.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.375+0000 7f69917fa640 1 --2- 172.21.15.116:0/2971871317 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6980067910 0x7f6980069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:31.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.375+0000 7f69917fa640 1 -- 172.21.15.116:0/2971871317 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 6924+0+0 (secure 0 0 0) 0x7f69880867a0 con 0x7f69940c3f70 2024-04-25T00:04:31.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.375+0000 7f6998e08640 1 --2- 172.21.15.116:0/2971871317 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6980067910 0x7f6980069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:31.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.375+0000 7f6998e08640 1 --2- 172.21.15.116:0/2971871317 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6980067910 0x7f6980069dd0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f6984000f70 tx=0x7f6984009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:31.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.375+0000 7f69917fa640 1 -- 172.21.15.116:0/2971871317 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f6988056840 con 0x7f69940c3f70 2024-04-25T00:04:31.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.475+0000 7fce1918a640 1 Processor -- start 2024-04-25T00:04:31.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.475+0000 7fce1918a640 1 -- start start 2024-04-25T00:04:31.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.475+0000 7fce1918a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fce140c1c30 0x7fce140c3060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:31.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.475+0000 7fce1918a640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce140c22a0 con 0x7fce1406c480 2024-04-25T00:04:31.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.475+0000 7fce1918a640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce140c2580 con 0x7fce14153170 2024-04-25T00:04:31.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.475+0000 7fce1918a640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fce140c35a0 con 0x7fce140c1c30 2024-04-25T00:04:31.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.475+0000 7fce0bfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fce140c1c30 0x7fce140c3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:31.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.475+0000 7fce12d9d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fce1406c480 0x7fce14071e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:42010/0 (socket says 172.21.15.116:42010) 2024-04-25T00:04:31.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.475+0000 7fce0bfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fce140c1c30 0x7fce140c3060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55144/0 (socket says 172.21.15.116:55144) 2024-04-25T00:04:31.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.475+0000 7fce12d9d640 1 -- 172.21.15.116:0/126155594 learned_addr learned my addr 172.21.15.116:0/126155594 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:04:31.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.475+0000 7fce0bfff640 1 -- 172.21.15.116:0/126155594 >> v1:172.21.15.174:6789/0 conn(0x7fce14153170 legacy=0x7fce1414fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:04:31.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.475+0000 7fce1259c640 1 -- 172.21.15.116:0/126155594 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3000826470 0 0) 0x7fce140c22a0 con 0x7fce1406c480 2024-04-25T00:04:31.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.475+0000 7fce0bfff640 1 -- 172.21.15.116:0/126155594 >> v1:172.21.15.116:6789/0 conn(0x7fce1406c480 legacy=0x7fce14071e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:31.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.475+0000 7fce0bfff640 1 -- 172.21.15.116:0/126155594 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce140c2860 con 0x7fce140c1c30 2024-04-25T00:04:31.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.479+0000 7fce0bfff640 1 --2- 172.21.15.116:0/126155594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce140c1c30 0x7fce140c3060 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc0097c0 tx=0x7fcdfc01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e7197b18d7660bf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:31.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.479+0000 7fce1259c640 1 -- 172.21.15.116:0/126155594 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdfc0b0070 con 0x7fce140c1c30 2024-04-25T00:04:31.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.479+0000 7fce1259c640 1 -- 172.21.15.116:0/126155594 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdfc0ab600 con 0x7fce140c1c30 2024-04-25T00:04:31.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.479+0000 7fce1259c640 1 -- 172.21.15.116:0/126155594 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdfc0ab9d0 con 0x7fce140c1c30 2024-04-25T00:04:31.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.479+0000 7fce1918a640 1 -- 172.21.15.116:0/126155594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce140c1c30 msgr2=0x7fce140c3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:31.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.479+0000 7fce1918a640 1 --2- 172.21.15.116:0/126155594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce140c1c30 0x7fce140c3060 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc0097c0 tx=0x7fcdfc01c4a0 comp rx=0 tx=0).stop 2024-04-25T00:04:31.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.479+0000 7fce1918a640 1 -- 172.21.15.116:0/126155594 shutdown_connections 2024-04-25T00:04:31.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.479+0000 7fce1918a640 1 --2- 172.21.15.116:0/126155594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce140c1c30 0x7fce140c3060 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:31.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.479+0000 7fce1918a640 1 -- 172.21.15.116:0/126155594 >> 172.21.15.116:0/126155594 conn(0x7fce140c4270 msgr2=0x7fce140c4680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:04:31.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.479+0000 7fce1918a640 1 -- 172.21.15.116:0/126155594 shutdown_connections 2024-04-25T00:04:31.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.479+0000 7fce1918a640 1 -- 172.21.15.116:0/126155594 wait complete. 2024-04-25T00:04:31.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.483+0000 7fce1918a640 1 Processor -- start 2024-04-25T00:04:31.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.483+0000 7fce1918a640 1 -- start start 2024-04-25T00:04:31.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.483+0000 7fce1918a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce1406c480 0x7fce1406e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:31.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.483+0000 7fce1918a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fce140c1c30 0x7fce141ef700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:31.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.483+0000 7fce1918a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fce14153170 0x7fce141efc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:31.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.483+0000 7fce1918a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fce1406ca70 con 0x7fce1406c480 2024-04-25T00:04:31.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.483+0000 7fce1918a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fce14153670 con 0x7fce140c1c30 2024-04-25T00:04:31.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.483+0000 7fce1918a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fce141503a0 con 0x7fce14153170 2024-04-25T00:04:31.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.483+0000 7fce12d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce1406c480 0x7fce1406e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:31.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.483+0000 7fce12d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce1406c480 0x7fce1406e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55152/0 (socket says 172.21.15.116:55152) 2024-04-25T00:04:31.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.483+0000 7fce12d9d640 1 -- 172.21.15.116:0/3067050351 learned_addr learned my addr 172.21.15.116:0/3067050351 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:04:31.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.483+0000 7fce0bfff640 1 --2- 172.21.15.116:0/3067050351 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fce140c1c30 0x7fce141ef700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:31.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.483+0000 7fce1359e640 1 --2- 172.21.15.116:0/3067050351 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fce14153170 0x7fce141efc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:31.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.483+0000 7fce12d9d640 1 -- 172.21.15.116:0/3067050351 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fce14153170 msgr2=0x7fce141efc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:31.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.483+0000 7fce12d9d640 1 --2- 172.21.15.116:0/3067050351 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fce14153170 0x7fce141efc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:31.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.483+0000 7fce12d9d640 1 -- 172.21.15.116:0/3067050351 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fce140c1c30 msgr2=0x7fce141ef700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:31.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.483+0000 7fce12d9d640 1 --2- 172.21.15.116:0/3067050351 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fce140c1c30 0x7fce141ef700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:31.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.483+0000 7fce12d9d640 1 -- 172.21.15.116:0/3067050351 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce1414aa60 con 0x7fce1406c480 2024-04-25T00:04:31.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.483+0000 7fce0bfff640 1 --2- 172.21.15.116:0/3067050351 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fce140c1c30 0x7fce141ef700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:04:31.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.483+0000 7fce12d9d640 1 --2- 172.21.15.116:0/3067050351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce1406c480 0x7fce1406e890 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fce0400c4a0 tx=0x7fce0400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:31.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.483+0000 7fce0b7fe640 1 -- 172.21.15.116:0/3067050351 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce04018070 con 0x7fce1406c480 2024-04-25T00:04:31.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.483+0000 7fce0b7fe640 1 -- 172.21.15.116:0/3067050351 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce040074e0 con 0x7fce1406c480 2024-04-25T00:04:31.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.483+0000 7fce1918a640 1 -- 172.21.15.116:0/3067050351 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce1414ad40 con 0x7fce1406c480 2024-04-25T00:04:31.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.483+0000 7fce0b7fe640 1 -- 172.21.15.116:0/3067050351 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce04004a30 con 0x7fce1406c480 2024-04-25T00:04:31.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.483+0000 7fce1918a640 1 -- 172.21.15.116:0/3067050351 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce1414d320 con 0x7fce1406c480 2024-04-25T00:04:31.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.483+0000 7fce1918a640 1 -- 172.21.15.116:0/3067050351 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce14068a60 con 0x7fce1406c480 2024-04-25T00:04:31.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.487+0000 7fce0b7fe640 1 -- 172.21.15.116:0/3067050351 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fce04004bd0 con 0x7fce1406c480 2024-04-25T00:04:31.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.487+0000 7fce0b7fe640 1 --2- 172.21.15.116:0/3067050351 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcdf40678f0 0x7fcdf4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:31.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.487+0000 7fce0b7fe640 1 -- 172.21.15.116:0/3067050351 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 6924+0+0 (secure 0 0 0) 0x7fce040853c0 con 0x7fce1406c480 2024-04-25T00:04:31.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.487+0000 7fce0bfff640 1 --2- 172.21.15.116:0/3067050351 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcdf40678f0 0x7fcdf4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:31.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.487+0000 7fce0bfff640 1 --2- 172.21.15.116:0/3067050351 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcdf40678f0 0x7fcdf4069db0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc0ad000 tx=0x7fcdfc0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:31.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.491+0000 7fce0b7fe640 1 -- 172.21.15.116:0/3067050351 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fce04055a50 con 0x7fce1406c480 2024-04-25T00:04:31.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.583+0000 7f699b06c640 1 -- 172.21.15.116:0/2971871317 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f69940c7870 con 0x7f69940c3f70 2024-04-25T00:04:31.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.587+0000 7f69917fa640 1 -- 172.21.15.116:0/2971871317 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6501 (secure 0 0 0) 0x7f698805ab40 con 0x7f69940c3f70 2024-04-25T00:04:31.587 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:04:31.587 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4879,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/1889314958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1889314958},{"type":"v1","addr":"172.21.15.174:6837","nonce":1889314958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:03:40.629360+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4828":{"gid":4828,"name":"c","rank":0,"incarnation":26,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6834/1649769369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1649769369},{"type":"v1","addr":"172.21.15.116:6834","nonce":1649769369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4828,"qdb_cluster":[4828]},"id":1},{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:03:34.579825+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4735},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4735":{"gid":4735,"name":"a","rank":0,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.116:6835/2004876235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":2004876235},{"type":"v1","addr":"172.21.15.116:6835","nonce":2004876235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4735,"qdb_cluster":[4735]},"id":2}]} 2024-04-25T00:04:31.587 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 31 2024-04-25T00:04:31.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.587+0000 7f699b06c640 1 -- 172.21.15.116:0/2971871317 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6980067910 msgr2=0x7f6980069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:31.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.587+0000 7f699b06c640 1 --2- 172.21.15.116:0/2971871317 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6980067910 0x7f6980069dd0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f6984000f70 tx=0x7f6984009040 comp rx=0 tx=0).stop 2024-04-25T00:04:31.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.587+0000 7f699b06c640 1 -- 172.21.15.116:0/2971871317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69940c3f70 msgr2=0x7f6994154f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:31.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.587+0000 7f699b06c640 1 --2- 172.21.15.116:0/2971871317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69940c3f70 0x7f6994154f10 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f698800ebe0 tx=0x7f698800c520 comp rx=0 tx=0).stop 2024-04-25T00:04:31.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.587+0000 7f699b06c640 1 -- 172.21.15.116:0/2971871317 shutdown_connections 2024-04-25T00:04:31.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.587+0000 7f699b06c640 1 --2- 172.21.15.116:0/2971871317 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6980067910 0x7f6980069dd0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:31.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.587+0000 7f699b06c640 1 --2- 172.21.15.116:0/2971871317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6994152e40 0x7f6994144db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:31.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.587+0000 7f699b06c640 1 --2- 172.21.15.116:0/2971871317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69940c3f70 0x7f6994154f10 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:31.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.587+0000 7f699b06c640 1 --2- 172.21.15.116:0/2971871317 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f69940c3a20 0x7f699406f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:31.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.587+0000 7f699b06c640 1 -- 172.21.15.116:0/2971871317 >> 172.21.15.116:0/2971871317 conn(0x7f69940c9160 msgr2=0x7f6994150670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:04:31.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.587+0000 7f699b06c640 1 -- 172.21.15.116:0/2971871317 shutdown_connections 2024-04-25T00:04:31.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.587+0000 7f699b06c640 1 -- 172.21.15.116:0/2971871317 wait complete. 2024-04-25T00:04:31.606 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:04:31.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.691+0000 7fce1918a640 1 -- 172.21.15.116:0/3067050351 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fce141ef2b0 con 0x7fce1406c480 2024-04-25T00:04:31.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.691+0000 7fce0b7fe640 1 -- 172.21.15.116:0/3067050351 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6501 (secure 0 0 0) 0x7fce04055350 con 0x7fce1406c480 2024-04-25T00:04:31.692 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:04:31.692 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4879,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/1889314958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1889314958},{"type":"v1","addr":"172.21.15.174:6837","nonce":1889314958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:03:40.629360+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4828":{"gid":4828,"name":"c","rank":0,"incarnation":26,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6834/1649769369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1649769369},{"type":"v1","addr":"172.21.15.116:6834","nonce":1649769369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4828,"qdb_cluster":[4828]},"id":1},{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:03:34.579825+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4735},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4735":{"gid":4735,"name":"a","rank":0,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.116:6835/2004876235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":2004876235},{"type":"v1","addr":"172.21.15.116:6835","nonce":2004876235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4735,"qdb_cluster":[4735]},"id":2}]} 2024-04-25T00:04:31.693 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 31 2024-04-25T00:04:31.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.695+0000 7fce1918a640 1 -- 172.21.15.116:0/3067050351 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcdf40678f0 msgr2=0x7fcdf4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:31.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.695+0000 7fce1918a640 1 --2- 172.21.15.116:0/3067050351 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcdf40678f0 0x7fcdf4069db0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc0ad000 tx=0x7fcdfc0a6000 comp rx=0 tx=0).stop 2024-04-25T00:04:31.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.695+0000 7fce1918a640 1 -- 172.21.15.116:0/3067050351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce1406c480 msgr2=0x7fce1406e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:31.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.695+0000 7fce1918a640 1 --2- 172.21.15.116:0/3067050351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce1406c480 0x7fce1406e890 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fce0400c4a0 tx=0x7fce0400c960 comp rx=0 tx=0).stop 2024-04-25T00:04:31.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.695+0000 7fce1918a640 1 -- 172.21.15.116:0/3067050351 shutdown_connections 2024-04-25T00:04:31.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.695+0000 7fce1918a640 1 --2- 172.21.15.116:0/3067050351 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcdf40678f0 0x7fcdf4069db0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:31.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.695+0000 7fce1918a640 1 --2- 172.21.15.116:0/3067050351 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fce14153170 0x7fce141efc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:31.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.695+0000 7fce1918a640 1 --2- 172.21.15.116:0/3067050351 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fce140c1c30 0x7fce141ef700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:31.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.695+0000 7fce1918a640 1 --2- 172.21.15.116:0/3067050351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce1406c480 0x7fce1406e890 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:31.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.695+0000 7fce1918a640 1 -- 172.21.15.116:0/3067050351 >> 172.21.15.116:0/3067050351 conn(0x7fce140c4270 msgr2=0x7fce140c66a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:04:31.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.695+0000 7fce1918a640 1 -- 172.21.15.116:0/3067050351 shutdown_connections 2024-04-25T00:04:31.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:31.695+0000 7fce1918a640 1 -- 172.21.15.116:0/3067050351 wait complete. 2024-04-25T00:04:31.712 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:04:33.607 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:04:33.713 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:04:33.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.735+0000 7fa486654640 1 Processor -- start 2024-04-25T00:04:33.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.735+0000 7fa486654640 1 -- start start 2024-04-25T00:04:33.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.735+0000 7fa486654640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa480072080 0x7fa480159950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:33.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.735+0000 7fa486654640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa4801562a0 0x7fa4800c5020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:33.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.735+0000 7fa486654640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4800730b0 con 0x7fa480152d80 2024-04-25T00:04:33.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.735+0000 7fa486654640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fa480072e60 con 0x7fa480072080 2024-04-25T00:04:33.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.735+0000 7fa486654640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa480073390 con 0x7fa4801562a0 2024-04-25T00:04:33.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.735+0000 7fa477fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa480072080 0x7fa480159950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:33.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.735+0000 7fa477fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa480072080 0x7fa480159950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55160/0 (socket says 172.21.15.116:55160) 2024-04-25T00:04:33.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.735+0000 7fa477fff640 1 -- 172.21.15.116:0/3971600954 learned_addr learned my addr 172.21.15.116:0/3971600954 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:04:33.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.735+0000 7fa484bf1640 1 --2- 172.21.15.116:0/3971600954 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa4801562a0 0x7fa4800c5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:33.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.735+0000 7fa477fff640 1 -- 172.21.15.116:0/3971600954 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa4801562a0 msgr2=0x7fa4800c5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:04:33.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.735+0000 7fa477fff640 1 --2- 172.21.15.116:0/3971600954 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa4801562a0 0x7fa4800c5020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:33.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.735+0000 7fa477fff640 1 -- 172.21.15.116:0/3971600954 >> v1:172.21.15.174:6789/0 conn(0x7fa480152d80 legacy=0x7fa4800c36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:33.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.735+0000 7fa477fff640 1 -- 172.21.15.116:0/3971600954 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa480158e70 con 0x7fa480072080 2024-04-25T00:04:33.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.739+0000 7fa477fff640 1 --2- 172.21.15.116:0/3971600954 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa480072080 0x7fa480159950 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fa468009870 tx=0x7fa46801c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=708f48aadb9c3e22 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:33.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.739+0000 7fa47f7fe640 1 -- 172.21.15.116:0/3971600954 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4680aa600 con 0x7fa480072080 2024-04-25T00:04:33.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.739+0000 7fa47f7fe640 1 -- 172.21.15.116:0/3971600954 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4680af070 con 0x7fa480072080 2024-04-25T00:04:33.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.739+0000 7fa47f7fe640 1 -- 172.21.15.116:0/3971600954 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4680aa9c0 con 0x7fa480072080 2024-04-25T00:04:33.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.739+0000 7fa486654640 1 -- 172.21.15.116:0/3971600954 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa480072080 msgr2=0x7fa480159950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:33.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.739+0000 7fa486654640 1 --2- 172.21.15.116:0/3971600954 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa480072080 0x7fa480159950 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fa468009870 tx=0x7fa46801c4a0 comp rx=0 tx=0).stop 2024-04-25T00:04:33.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.739+0000 7fa486654640 1 -- 172.21.15.116:0/3971600954 shutdown_connections 2024-04-25T00:04:33.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.739+0000 7fa486654640 1 --2- 172.21.15.116:0/3971600954 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa4801562a0 0x7fa4800c5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:33.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.739+0000 7fa486654640 1 --2- 172.21.15.116:0/3971600954 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa480072080 0x7fa480159950 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:33.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.739+0000 7fa486654640 1 -- 172.21.15.116:0/3971600954 >> 172.21.15.116:0/3971600954 conn(0x7fa48006c720 msgr2=0x7fa48006cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:04:33.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.739+0000 7fa486654640 1 -- 172.21.15.116:0/3971600954 shutdown_connections 2024-04-25T00:04:33.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.739+0000 7fa486654640 1 -- 172.21.15.116:0/3971600954 wait complete. 2024-04-25T00:04:33.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.739+0000 7fa486654640 1 Processor -- start 2024-04-25T00:04:33.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.739+0000 7fa486654640 1 -- start start 2024-04-25T00:04:33.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.739+0000 7fa486654640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa480072080 0x7fa48014b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:33.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.739+0000 7fa486654640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa480152d80 0x7fa4801f2530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:33.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.739+0000 7fa486654640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa4801562a0 0x7fa48014d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:33.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.739+0000 7fa486654640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa480153160 con 0x7fa480152d80 2024-04-25T00:04:33.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.739+0000 7fa486654640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa4801566b0 con 0x7fa4801562a0 2024-04-25T00:04:33.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.739+0000 7fa486654640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa4801532e0 con 0x7fa480072080 2024-04-25T00:04:33.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.739+0000 7fa477fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa480152d80 0x7fa4801f2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:33.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.739+0000 7fa477fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa480152d80 0x7fa4801f2530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55170/0 (socket says 172.21.15.116:55170) 2024-04-25T00:04:33.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.739+0000 7fa477fff640 1 -- 172.21.15.116:0/451949481 learned_addr learned my addr 172.21.15.116:0/451949481 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:04:33.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.739+0000 7fa47ffff640 1 --2- 172.21.15.116:0/451949481 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa480072080 0x7fa48014b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:33.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.739+0000 7fa484bf1640 1 --2- 172.21.15.116:0/451949481 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa4801562a0 0x7fa48014d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:33.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.739+0000 7fa477fff640 1 -- 172.21.15.116:0/451949481 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa480072080 msgr2=0x7fa48014b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:33.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.739+0000 7fa477fff640 1 --2- 172.21.15.116:0/451949481 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa480072080 0x7fa48014b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:33.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.739+0000 7fa477fff640 1 -- 172.21.15.116:0/451949481 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa4801562a0 msgr2=0x7fa48014d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:33.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.739+0000 7fa477fff640 1 --2- 172.21.15.116:0/451949481 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa4801562a0 0x7fa48014d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:33.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.739+0000 7fa477fff640 1 -- 172.21.15.116:0/451949481 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4800ac860 con 0x7fa480152d80 2024-04-25T00:04:33.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.739+0000 7fa477fff640 1 --2- 172.21.15.116:0/451949481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa480152d80 0x7fa4801f2530 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fa46801a0d0 tx=0x7fa46801ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:33.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.743+0000 7fa47d7fa640 1 -- 172.21.15.116:0/451949481 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4680bd5f0 con 0x7fa480152d80 2024-04-25T00:04:33.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.743+0000 7fa47d7fa640 1 -- 172.21.15.116:0/451949481 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4680bd790 con 0x7fa480152d80 2024-04-25T00:04:33.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.743+0000 7fa47d7fa640 1 -- 172.21.15.116:0/451949481 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4680bdb80 con 0x7fa480152d80 2024-04-25T00:04:33.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.743+0000 7fa486654640 1 -- 172.21.15.116:0/451949481 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4800acb40 con 0x7fa480152d80 2024-04-25T00:04:33.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.743+0000 7fa486654640 1 -- 172.21.15.116:0/451949481 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa48014abf0 con 0x7fa480152d80 2024-04-25T00:04:33.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.743+0000 7fa486654640 1 -- 172.21.15.116:0/451949481 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa480068a60 con 0x7fa480152d80 2024-04-25T00:04:33.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.743+0000 7fa47d7fa640 1 -- 172.21.15.116:0/451949481 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa4680a9070 con 0x7fa480152d80 2024-04-25T00:04:33.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.743+0000 7fa47d7fa640 1 --2- 172.21.15.116:0/451949481 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa4600678f0 0x7fa460069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:33.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.743+0000 7fa47d7fa640 1 -- 172.21.15.116:0/451949481 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 6924+0+0 (secure 0 0 0) 0x7fa468120c60 con 0x7fa480152d80 2024-04-25T00:04:33.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.743+0000 7fa47ffff640 1 --2- 172.21.15.116:0/451949481 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa4600678f0 0x7fa460069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:33.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.743+0000 7fa47ffff640 1 --2- 172.21.15.116:0/451949481 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa4600678f0 0x7fa460069db0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa4800acda0 tx=0x7fa470009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:33.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.747+0000 7fa47d7fa640 1 -- 172.21.15.116:0/451949481 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fa468016150 con 0x7fa480152d80 2024-04-25T00:04:33.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.835+0000 7f70e5d19640 1 Processor -- start 2024-04-25T00:04:33.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.835+0000 7f70e5d19640 1 -- start start 2024-04-25T00:04:33.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.835+0000 7f70e5d19640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f70e00c2b10 0x7f70e01541f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:33.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.835+0000 7f70e5d19640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f70e006bc20 0x7f70e00c3260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:33.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.835+0000 7f70e5d19640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70e00734b0 con 0x7f70e0150d10 2024-04-25T00:04:33.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.835+0000 7f70e5d19640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f70e0153750 con 0x7f70e00c2b10 2024-04-25T00:04:33.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.835+0000 7f70e5d19640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f70e00a9bb0 con 0x7f70e006bc20 2024-04-25T00:04:33.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.835+0000 7f70deffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f70e00c2b10 0x7f70e01541f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:33.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.835+0000 7f70deffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f70e00c2b10 0x7f70e01541f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55184/0 (socket says 172.21.15.116:55184) 2024-04-25T00:04:33.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.835+0000 7f70deffd640 1 -- 172.21.15.116:0/1319864572 learned_addr learned my addr 172.21.15.116:0/1319864572 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:04:33.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.835+0000 7f70dffff640 1 --2- 172.21.15.116:0/1319864572 >> v2:172.21.15.174:3300/0 conn(0x7f70e006bc20 0x7f70e00c3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:33.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.835+0000 7f70deffd640 1 -- 172.21.15.116:0/1319864572 >> v2:172.21.15.174:3300/0 conn(0x7f70e006bc20 msgr2=0x7f70e00c3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:04:33.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.835+0000 7f70deffd640 1 --2- 172.21.15.116:0/1319864572 >> v2:172.21.15.174:3300/0 conn(0x7f70e006bc20 0x7f70e00c3260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:33.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.835+0000 7f70deffd640 1 -- 172.21.15.116:0/1319864572 >> v1:172.21.15.174:6789/0 conn(0x7f70e0150d10 legacy=0x7f70e00a94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:33.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.835+0000 7f70deffd640 1 -- 172.21.15.116:0/1319864572 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70e00731d0 con 0x7f70e00c2b10 2024-04-25T00:04:33.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.839+0000 7f70deffd640 1 --2- 172.21.15.116:0/1319864572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70e00c2b10 0x7f70e01541f0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f70e00ae400 tx=0x7f70d401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d86311133e6837f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:33.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.839+0000 7f70de7fc640 1 -- 172.21.15.116:0/1319864572 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70d40b2070 con 0x7f70e00c2b10 2024-04-25T00:04:33.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.839+0000 7f70de7fc640 1 -- 172.21.15.116:0/1319864572 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70d40ad680 con 0x7f70e00c2b10 2024-04-25T00:04:33.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.839+0000 7f70de7fc640 1 -- 172.21.15.116:0/1319864572 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70d40ada50 con 0x7f70e00c2b10 2024-04-25T00:04:33.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.839+0000 7f70e5d19640 1 -- 172.21.15.116:0/1319864572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70e00c2b10 msgr2=0x7f70e01541f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:33.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.839+0000 7f70e5d19640 1 --2- 172.21.15.116:0/1319864572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70e00c2b10 0x7f70e01541f0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f70e00ae400 tx=0x7f70d401c490 comp rx=0 tx=0).stop 2024-04-25T00:04:33.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.839+0000 7f70e5d19640 1 -- 172.21.15.116:0/1319864572 shutdown_connections 2024-04-25T00:04:33.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.839+0000 7f70e5d19640 1 --2- 172.21.15.116:0/1319864572 >> v2:172.21.15.174:3300/0 conn(0x7f70e006bc20 0x7f70e00c3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:33.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.839+0000 7f70e5d19640 1 --2- 172.21.15.116:0/1319864572 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70e00c2b10 0x7f70e01541f0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:33.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.839+0000 7f70e5d19640 1 -- 172.21.15.116:0/1319864572 >> 172.21.15.116:0/1319864572 conn(0x7f70e00c4c10 msgr2=0x7f70e00c5020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:04:33.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.839+0000 7f70e5d19640 1 -- 172.21.15.116:0/1319864572 shutdown_connections 2024-04-25T00:04:33.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.839+0000 7f70e5d19640 1 -- 172.21.15.116:0/1319864572 wait complete. 2024-04-25T00:04:33.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.839+0000 7f70e5d19640 1 Processor -- start 2024-04-25T00:04:33.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.839+0000 7f70e5d19640 1 -- start start 2024-04-25T00:04:33.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.839+0000 7f70e5d19640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70e006bc20 0x7f70e00ad4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:33.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.839+0000 7f70e5d19640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f70e00c2b10 0x7f70e013fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:33.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.839+0000 7f70e5d19640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f70e0150d10 0x7f70e01400c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:33.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.839+0000 7f70e5d19640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f70e0151270 con 0x7f70e006bc20 2024-04-25T00:04:33.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.839+0000 7f70e5d19640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f70e01510f0 con 0x7f70e0150d10 2024-04-25T00:04:33.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.839+0000 7f70e5d19640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f70e0160750 con 0x7f70e00c2b10 2024-04-25T00:04:33.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.839+0000 7f70df7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70e006bc20 0x7f70e00ad4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:33.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.839+0000 7f70df7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70e006bc20 0x7f70e00ad4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55188/0 (socket says 172.21.15.116:55188) 2024-04-25T00:04:33.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.839+0000 7f70df7fe640 1 -- 172.21.15.116:0/2962285621 learned_addr learned my addr 172.21.15.116:0/2962285621 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:04:33.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.839+0000 7f70deffd640 1 --2- 172.21.15.116:0/2962285621 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f70e00c2b10 0x7f70e013fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:33.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.839+0000 7f70dffff640 1 --2- 172.21.15.116:0/2962285621 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f70e0150d10 0x7f70e01400c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:33.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.839+0000 7f70df7fe640 1 -- 172.21.15.116:0/2962285621 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f70e00c2b10 msgr2=0x7f70e013fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:33.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.839+0000 7f70df7fe640 1 --2- 172.21.15.116:0/2962285621 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f70e00c2b10 0x7f70e013fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:33.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.839+0000 7f70df7fe640 1 -- 172.21.15.116:0/2962285621 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f70e0150d10 msgr2=0x7f70e01400c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:33.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.839+0000 7f70df7fe640 1 --2- 172.21.15.116:0/2962285621 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f70e0150d10 0x7f70e01400c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:33.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.839+0000 7f70df7fe640 1 -- 172.21.15.116:0/2962285621 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70e014b060 con 0x7f70e006bc20 2024-04-25T00:04:33.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.839+0000 7f70df7fe640 1 --2- 172.21.15.116:0/2962285621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70e006bc20 0x7f70e00ad4c0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f70d000c370 tx=0x7f70d000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:33.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.843+0000 7f70c7fff640 1 -- 172.21.15.116:0/2962285621 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70d00052d0 con 0x7f70e006bc20 2024-04-25T00:04:33.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.843+0000 7f70c7fff640 1 -- 172.21.15.116:0/2962285621 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70d0005470 con 0x7f70e006bc20 2024-04-25T00:04:33.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.843+0000 7f70e5d19640 1 -- 172.21.15.116:0/2962285621 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70e01f04b0 con 0x7f70e006bc20 2024-04-25T00:04:33.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.843+0000 7f70c7fff640 1 -- 172.21.15.116:0/2962285621 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70d0007260 con 0x7f70e006bc20 2024-04-25T00:04:33.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.843+0000 7f70e5d19640 1 -- 172.21.15.116:0/2962285621 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70e014b5e0 con 0x7f70e006bc20 2024-04-25T00:04:33.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.843+0000 7f70e5d19640 1 -- 172.21.15.116:0/2962285621 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70e0068a60 con 0x7f70e006bc20 2024-04-25T00:04:33.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.843+0000 7f70c7fff640 1 -- 172.21.15.116:0/2962285621 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f70d0007400 con 0x7f70e006bc20 2024-04-25T00:04:33.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.843+0000 7f70c7fff640 1 --2- 172.21.15.116:0/2962285621 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f70cc0678f0 0x7f70cc069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:33.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.843+0000 7f70c7fff640 1 -- 172.21.15.116:0/2962285621 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 6924+0+0 (secure 0 0 0) 0x7f70d0085510 con 0x7f70e006bc20 2024-04-25T00:04:33.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.843+0000 7f70deffd640 1 --2- 172.21.15.116:0/2962285621 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f70cc0678f0 0x7f70cc069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:33.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.843+0000 7f70deffd640 1 --2- 172.21.15.116:0/2962285621 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f70cc0678f0 0x7f70cc069db0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f70d40af000 tx=0x7f70d4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:33.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.847+0000 7f70c7fff640 1 -- 172.21.15.116:0/2962285621 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f70d0055c20 con 0x7f70e006bc20 2024-04-25T00:04:33.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.947+0000 7fa486654640 1 -- 172.21.15.116:0/451949481 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa4800c7d90 con 0x7fa480152d80 2024-04-25T00:04:33.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.947+0000 7fa47d7fa640 1 -- 172.21.15.116:0/451949481 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6501 (secure 0 0 0) 0x7fa4680f1350 con 0x7fa480152d80 2024-04-25T00:04:33.949 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:04:33.949 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4879,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/1889314958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1889314958},{"type":"v1","addr":"172.21.15.174:6837","nonce":1889314958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:03:40.629360+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4828":{"gid":4828,"name":"c","rank":0,"incarnation":26,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6834/1649769369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1649769369},{"type":"v1","addr":"172.21.15.116:6834","nonce":1649769369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4828,"qdb_cluster":[4828]},"id":1},{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:03:34.579825+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4735},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4735":{"gid":4735,"name":"a","rank":0,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.116:6835/2004876235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":2004876235},{"type":"v1","addr":"172.21.15.116:6835","nonce":2004876235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4735,"qdb_cluster":[4735]},"id":2}]} 2024-04-25T00:04:33.949 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 31 2024-04-25T00:04:33.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.951+0000 7fa486654640 1 -- 172.21.15.116:0/451949481 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa4600678f0 msgr2=0x7fa460069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:33.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.951+0000 7fa486654640 1 --2- 172.21.15.116:0/451949481 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa4600678f0 0x7fa460069db0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa4800acda0 tx=0x7fa470009040 comp rx=0 tx=0).stop 2024-04-25T00:04:33.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.951+0000 7fa486654640 1 -- 172.21.15.116:0/451949481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa480152d80 msgr2=0x7fa4801f2530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:33.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.951+0000 7fa486654640 1 --2- 172.21.15.116:0/451949481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa480152d80 0x7fa4801f2530 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fa46801a0d0 tx=0x7fa46801ace0 comp rx=0 tx=0).stop 2024-04-25T00:04:33.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.951+0000 7fa486654640 1 -- 172.21.15.116:0/451949481 shutdown_connections 2024-04-25T00:04:33.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.951+0000 7fa486654640 1 --2- 172.21.15.116:0/451949481 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa4600678f0 0x7fa460069db0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:33.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.951+0000 7fa486654640 1 --2- 172.21.15.116:0/451949481 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa4801562a0 0x7fa48014d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:33.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.951+0000 7fa486654640 1 --2- 172.21.15.116:0/451949481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa480152d80 0x7fa4801f2530 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:33.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.951+0000 7fa486654640 1 --2- 172.21.15.116:0/451949481 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa480072080 0x7fa48014b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:33.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.951+0000 7fa486654640 1 -- 172.21.15.116:0/451949481 >> 172.21.15.116:0/451949481 conn(0x7fa48006c720 msgr2=0x7fa48006cb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:04:33.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.951+0000 7fa486654640 1 -- 172.21.15.116:0/451949481 shutdown_connections 2024-04-25T00:04:33.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:33.951+0000 7fa486654640 1 -- 172.21.15.116:0/451949481 wait complete. 2024-04-25T00:04:33.967 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:04:34.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:34.043+0000 7f70e5d19640 1 -- 172.21.15.116:0/2962285621 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f70e00c2540 con 0x7f70e006bc20 2024-04-25T00:04:34.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:34.043+0000 7f70c7fff640 1 -- 172.21.15.116:0/2962285621 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) ==== 94+0+6501 (secure 0 0 0) 0x7f70d0055520 con 0x7f70e006bc20 2024-04-25T00:04:34.045 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:04:34.045 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4879,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/1889314958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1889314958},{"type":"v1","addr":"172.21.15.174:6837","nonce":1889314958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":31}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:03:40.629360+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":32,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4828":{"gid":4828,"name":"c","rank":0,"incarnation":26,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6834/1649769369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1649769369},{"type":"v1","addr":"172.21.15.116:6834","nonce":1649769369}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4828,"qdb_cluster":[4828]},"id":1},{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:03:34.579825+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4735},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4735":{"gid":4735,"name":"a","rank":0,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.116:6835/2004876235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":2004876235},{"type":"v1","addr":"172.21.15.116:6835","nonce":2004876235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4735,"qdb_cluster":[4735]},"id":2}]} 2024-04-25T00:04:34.046 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 31 2024-04-25T00:04:34.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:34.047+0000 7f70e5d19640 1 -- 172.21.15.116:0/2962285621 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f70cc0678f0 msgr2=0x7f70cc069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:34.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:34.047+0000 7f70e5d19640 1 --2- 172.21.15.116:0/2962285621 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f70cc0678f0 0x7f70cc069db0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f70d40af000 tx=0x7f70d4008040 comp rx=0 tx=0).stop 2024-04-25T00:04:34.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:34.047+0000 7f70e5d19640 1 -- 172.21.15.116:0/2962285621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70e006bc20 msgr2=0x7f70e00ad4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:34.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:34.047+0000 7f70e5d19640 1 --2- 172.21.15.116:0/2962285621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70e006bc20 0x7f70e00ad4c0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f70d000c370 tx=0x7f70d000c830 comp rx=0 tx=0).stop 2024-04-25T00:04:34.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:34.047+0000 7f70e5d19640 1 -- 172.21.15.116:0/2962285621 shutdown_connections 2024-04-25T00:04:34.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:34.047+0000 7f70e5d19640 1 --2- 172.21.15.116:0/2962285621 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f70cc0678f0 0x7f70cc069db0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:34.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:34.047+0000 7f70e5d19640 1 --2- 172.21.15.116:0/2962285621 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f70e0150d10 0x7f70e01400c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:34.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:34.047+0000 7f70e5d19640 1 --2- 172.21.15.116:0/2962285621 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f70e00c2b10 0x7f70e013fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:34.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:34.047+0000 7f70e5d19640 1 --2- 172.21.15.116:0/2962285621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70e006bc20 0x7f70e00ad4c0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:34.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:34.047+0000 7f70e5d19640 1 -- 172.21.15.116:0/2962285621 >> 172.21.15.116:0/2962285621 conn(0x7f70e00c4c10 msgr2=0x7f70e0151420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:04:34.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:34.047+0000 7f70e5d19640 1 -- 172.21.15.116:0/2962285621 shutdown_connections 2024-04-25T00:04:34.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:34.047+0000 7f70e5d19640 1 -- 172.21.15.116:0/2962285621 wait complete. 2024-04-25T00:04:34.064 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:04:34.944 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:04:34.948 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:04:35.968 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:04:36.066 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:04:36.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.099+0000 7f6681149640 1 Processor -- start 2024-04-25T00:04:36.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.099+0000 7f6681149640 1 -- start start 2024-04-25T00:04:36.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.099+0000 7f6681149640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f667c0c5280 0x7f667c150840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:36.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.099+0000 7f6681149640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f667c0c2c40 con 0x7f667c072810 2024-04-25T00:04:36.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.099+0000 7f6681149640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f667c157660 con 0x7f667c072c90 2024-04-25T00:04:36.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.099+0000 7f6681149640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f667c157c20 con 0x7f667c0c5280 2024-04-25T00:04:36.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.099+0000 7f667b59e640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f667c072810 0x7f667c153760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:49430/0 (socket says 172.21.15.116:49430) 2024-04-25T00:04:36.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.099+0000 7f667b59e640 1 -- 172.21.15.116:0/2187849185 learned_addr learned my addr 172.21.15.116:0/2187849185 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:04:36.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.099+0000 7f667ad9d640 1 --2- 172.21.15.116:0/2187849185 >> v2:172.21.15.174:3300/0 conn(0x7f667c0c5280 0x7f667c150840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:36.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.099+0000 7f6679d9b640 1 -- 172.21.15.116:0/2187849185 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3197970052 0 0) 0x7f667c0c2c40 con 0x7f667c072810 2024-04-25T00:04:36.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.099+0000 7f6679d9b640 1 -- 172.21.15.116:0/2187849185 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6668003530 con 0x7f667c072810 2024-04-25T00:04:36.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.099+0000 7f6679d9b640 1 -- 172.21.15.116:0/2187849185 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3808752726 0 0) 0x7f6668003530 con 0x7f667c072810 2024-04-25T00:04:36.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.099+0000 7f6679d9b640 1 -- 172.21.15.116:0/2187849185 >> v2:172.21.15.174:3300/0 conn(0x7f667c0c5280 msgr2=0x7f667c150840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:04:36.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.099+0000 7f6679d9b640 1 --2- 172.21.15.116:0/2187849185 >> v2:172.21.15.174:3300/0 conn(0x7f667c0c5280 0x7f667c150840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:36.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.099+0000 7f6679d9b640 1 -- 172.21.15.116:0/2187849185 >> v1:172.21.15.174:6789/0 conn(0x7f667c072c90 legacy=0x7f667c150d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:36.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.099+0000 7f6679d9b640 1 -- 172.21.15.116:0/2187849185 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f667c157940 con 0x7f667c072810 2024-04-25T00:04:36.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.099+0000 7f6679d9b640 1 -- 172.21.15.116:0/2187849185 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f666c001a20 con 0x7f667c072810 2024-04-25T00:04:36.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.099+0000 7f6679d9b640 1 -- 172.21.15.116:0/2187849185 >> v1:172.21.15.116:6789/0 conn(0x7f667c072810 legacy=0x7f667c153760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:36.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.099+0000 7f6679d9b640 1 --2- 172.21.15.116:0/2187849185 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f66680039c0 0x7f6668003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:36.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.099+0000 7f6679d9b640 1 --2- 172.21.15.116:0/2187849185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6668004810 0x7f6668004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:36.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.099+0000 7f6679d9b640 1 --2- 172.21.15.116:0/2187849185 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66680052f0 0x7f666800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:36.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.099+0000 7f6679d9b640 1 -- 172.21.15.116:0/2187849185 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f667c157c20 con 0x7f6668004810 2024-04-25T00:04:36.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.099+0000 7f6679d9b640 1 -- 172.21.15.116:0/2187849185 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f666800c1c0 con 0x7f66680052f0 2024-04-25T00:04:36.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.099+0000 7f6679d9b640 1 -- 172.21.15.116:0/2187849185 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f666800c3c0 con 0x7f66680039c0 2024-04-25T00:04:36.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.099+0000 7f667a59c640 1 --2- 172.21.15.116:0/2187849185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6668004810 0x7f6668004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:36.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.103+0000 7f667b59e640 1 --2- 172.21.15.116:0/2187849185 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66680052f0 0x7f666800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:36.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.103+0000 7f667ad9d640 1 --2- 172.21.15.116:0/2187849185 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f66680039c0 0x7f6668003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:36.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.103+0000 7f667a59c640 1 -- 172.21.15.116:0/2187849185 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f66680039c0 msgr2=0x7f6668003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:04:36.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.103+0000 7f667a59c640 1 --2- 172.21.15.116:0/2187849185 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f66680039c0 0x7f6668003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:36.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.103+0000 7f667a59c640 1 -- 172.21.15.116:0/2187849185 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66680052f0 msgr2=0x7f666800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:04:36.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.103+0000 7f667a59c640 1 --2- 172.21.15.116:0/2187849185 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66680052f0 0x7f666800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:36.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.103+0000 7f667b59e640 1 -- 172.21.15.116:0/2187849185 reap_dead start 2024-04-25T00:04:36.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.103+0000 7f667a59c640 1 -- 172.21.15.116:0/2187849185 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6668003530 con 0x7f6668004810 2024-04-25T00:04:36.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.107+0000 7f667a59c640 1 --2- 172.21.15.116:0/2187849185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6668004810 0x7f6668004c90 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f6664001750 tx=0x7f666401c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ad29ec3e8228a99 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:36.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.107+0000 7f6679d9b640 1 -- 172.21.15.116:0/2187849185 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66640013a0 con 0x7f6668004810 2024-04-25T00:04:36.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.107+0000 7f6679d9b640 1 -- 172.21.15.116:0/2187849185 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66640052f0 con 0x7f6668004810 2024-04-25T00:04:36.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.107+0000 7f6679d9b640 1 -- 172.21.15.116:0/2187849185 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6664003840 con 0x7f6668004810 2024-04-25T00:04:36.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.107+0000 7f6681149640 1 -- 172.21.15.116:0/2187849185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6668004810 msgr2=0x7f6668004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:36.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.107+0000 7f6681149640 1 --2- 172.21.15.116:0/2187849185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6668004810 0x7f6668004c90 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f6664001750 tx=0x7f666401c3c0 comp rx=0 tx=0).stop 2024-04-25T00:04:36.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.107+0000 7f6681149640 1 -- 172.21.15.116:0/2187849185 shutdown_connections 2024-04-25T00:04:36.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.107+0000 7f6681149640 1 --2- 172.21.15.116:0/2187849185 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6668004810 0x7f6668004c90 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:36.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.107+0000 7f6681149640 1 -- 172.21.15.116:0/2187849185 >> 172.21.15.116:0/2187849185 conn(0x7f667c06c720 msgr2=0x7f667c06cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:04:36.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.107+0000 7f6681149640 1 -- 172.21.15.116:0/2187849185 shutdown_connections 2024-04-25T00:04:36.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.107+0000 7f6681149640 1 -- 172.21.15.116:0/2187849185 wait complete. 2024-04-25T00:04:36.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.107+0000 7f6681149640 1 Processor -- start 2024-04-25T00:04:36.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.107+0000 7f6681149640 1 -- start start 2024-04-25T00:04:36.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.107+0000 7f6681149640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6668004810 0x7f667c0c1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:36.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.107+0000 7f6681149640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f667c147020 0x7f667c146470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:36.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.107+0000 7f6681149640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f667c154a50 0x7f667c1469b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:36.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.107+0000 7f6681149640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f667c153740 con 0x7f667c154a50 2024-04-25T00:04:36.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.107+0000 7f6681149640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f667c151260 con 0x7f6668004810 2024-04-25T00:04:36.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.107+0000 7f6681149640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f667c1538c0 con 0x7f667c147020 2024-04-25T00:04:36.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.107+0000 7f667b59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f667c154a50 0x7f667c1469b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:36.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.107+0000 7f667b59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f667c154a50 0x7f667c1469b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53710/0 (socket says 172.21.15.116:53710) 2024-04-25T00:04:36.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.107+0000 7f667b59e640 1 -- 172.21.15.116:0/2317671506 learned_addr learned my addr 172.21.15.116:0/2317671506 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:04:36.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.111+0000 7f667ad9d640 1 --2- 172.21.15.116:0/2317671506 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6668004810 0x7f667c0c1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:36.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.111+0000 7f667a59c640 1 --2- 172.21.15.116:0/2317671506 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f667c147020 0x7f667c146470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:36.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.111+0000 7f667b59e640 1 -- 172.21.15.116:0/2317671506 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f667c147020 msgr2=0x7f667c146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:36.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.111+0000 7f667b59e640 1 --2- 172.21.15.116:0/2317671506 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f667c147020 0x7f667c146470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:36.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.111+0000 7f667b59e640 1 -- 172.21.15.116:0/2317671506 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6668004810 msgr2=0x7f667c0c1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:36.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.111+0000 7f667b59e640 1 --2- 172.21.15.116:0/2317671506 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6668004810 0x7f667c0c1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:36.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.111+0000 7f667b59e640 1 -- 172.21.15.116:0/2317671506 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f667c1ee220 con 0x7f667c154a50 2024-04-25T00:04:36.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.111+0000 7f667a59c640 1 --2- 172.21.15.116:0/2317671506 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f667c147020 0x7f667c146470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:04:36.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.111+0000 7f667b59e640 1 --2- 172.21.15.116:0/2317671506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f667c154a50 0x7f667c1469b0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f666c001380 tx=0x7f666c007bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:36.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.111+0000 7f66737fe640 1 -- 172.21.15.116:0/2317671506 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f666c00ea80 con 0x7f667c154a50 2024-04-25T00:04:36.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.111+0000 7f66737fe640 1 -- 172.21.15.116:0/2317671506 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f666c018070 con 0x7f667c154a50 2024-04-25T00:04:36.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.111+0000 7f6681149640 1 -- 172.21.15.116:0/2317671506 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f667c156fa0 con 0x7f667c154a50 2024-04-25T00:04:36.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.111+0000 7f66737fe640 1 -- 172.21.15.116:0/2317671506 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f666c00ee80 con 0x7f667c154a50 2024-04-25T00:04:36.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.111+0000 7f6681149640 1 -- 172.21.15.116:0/2317671506 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f667c0acd60 con 0x7f667c154a50 2024-04-25T00:04:36.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.111+0000 7f6681149640 1 -- 172.21.15.116:0/2317671506 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f667c068a60 con 0x7f667c154a50 2024-04-25T00:04:36.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.111+0000 7f66737fe640 1 -- 172.21.15.116:0/2317671506 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f666c01d460 con 0x7f667c154a50 2024-04-25T00:04:36.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.111+0000 7f66737fe640 1 --2- 172.21.15.116:0/2317671506 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f665c0678c0 0x7f665c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:36.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.111+0000 7f66737fe640 1 -- 172.21.15.116:0/2317671506 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7010+0+0 (secure 0 0 0) 0x7f666c086410 con 0x7f667c154a50 2024-04-25T00:04:36.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.115+0000 7f667ad9d640 1 --2- 172.21.15.116:0/2317671506 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f665c0678c0 0x7f665c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:36.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.115+0000 7f667ad9d640 1 --2- 172.21.15.116:0/2317671506 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f665c0678c0 0x7f665c069d80 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6674009ab0 tx=0x7f6674008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:36.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.119+0000 7f66737fe640 1 -- 172.21.15.116:0/2317671506 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f666c056a40 con 0x7f667c154a50 2024-04-25T00:04:36.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.199+0000 7f739b7a4640 1 Processor -- start 2024-04-25T00:04:36.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.199+0000 7f739b7a4640 1 -- start start 2024-04-25T00:04:36.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.199+0000 7f739b7a4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f739414fa60 0x7f7394073490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:36.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.199+0000 7f739b7a4640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f73940c0b30 0x7f73941560a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:36.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.199+0000 7f739b7a4640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f73941565e0 0x7f73940c3780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:36.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.199+0000 7f739b7a4640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f7394155c10 con 0x7f73941565e0 2024-04-25T00:04:36.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.199+0000 7f739b7a4640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f739414fe40 con 0x7f73940c0b30 2024-04-25T00:04:36.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.199+0000 7f739b7a4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f73941558c0 con 0x7f739414fa60 2024-04-25T00:04:36.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.199+0000 7f7399d41640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f73941565e0 0x7f73940c3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:36.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.199+0000 7f7399d41640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f73941565e0 0x7f73940c3780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53712/0 (socket says 172.21.15.116:53712) 2024-04-25T00:04:36.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.199+0000 7f7399d41640 1 -- 172.21.15.116:0/2415892931 learned_addr learned my addr 172.21.15.116:0/2415892931 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:04:36.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.199+0000 7f7398d3f640 1 --2- 172.21.15.116:0/2415892931 >> v2:172.21.15.174:3300/0 conn(0x7f73940c0b30 0x7f73941560a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:36.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.199+0000 7f7399540640 1 --2- 172.21.15.116:0/2415892931 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f739414fa60 0x7f7394073490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:36.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.199+0000 7f7399d41640 1 -- 172.21.15.116:0/2415892931 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f739414fa60 msgr2=0x7f7394073490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:04:36.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.199+0000 7f7399d41640 1 --2- 172.21.15.116:0/2415892931 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f739414fa60 0x7f7394073490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:36.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.199+0000 7f7399d41640 1 -- 172.21.15.116:0/2415892931 >> v2:172.21.15.174:3300/0 conn(0x7f73940c0b30 msgr2=0x7f73941560a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:04:36.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.199+0000 7f7399d41640 1 --2- 172.21.15.116:0/2415892931 >> v2:172.21.15.174:3300/0 conn(0x7f73940c0b30 0x7f73941560a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:36.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.199+0000 7f7399d41640 1 -- 172.21.15.116:0/2415892931 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73940c1040 con 0x7f73941565e0 2024-04-25T00:04:36.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.199+0000 7f7399d41640 1 --2- 172.21.15.116:0/2415892931 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73941565e0 0x7f73940c3780 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f73940c86c0 tx=0x7f738c01c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a939ff9798ab099 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:36.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.199+0000 7f738bfff640 1 -- 172.21.15.116:0/2415892931 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f738c0a5720 con 0x7f73941565e0 2024-04-25T00:04:36.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.203+0000 7f738bfff640 1 -- 172.21.15.116:0/2415892931 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f738c0a58c0 con 0x7f73941565e0 2024-04-25T00:04:36.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.203+0000 7f738bfff640 1 -- 172.21.15.116:0/2415892931 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f738c0a5c90 con 0x7f73941565e0 2024-04-25T00:04:36.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.203+0000 7f739b7a4640 1 -- 172.21.15.116:0/2415892931 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73941565e0 msgr2=0x7f73940c3780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:36.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.203+0000 7f739b7a4640 1 --2- 172.21.15.116:0/2415892931 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73941565e0 0x7f73940c3780 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f73940c86c0 tx=0x7f738c01c8a0 comp rx=0 tx=0).stop 2024-04-25T00:04:36.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.203+0000 7f739b7a4640 1 -- 172.21.15.116:0/2415892931 shutdown_connections 2024-04-25T00:04:36.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.203+0000 7f739b7a4640 1 --2- 172.21.15.116:0/2415892931 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73941565e0 0x7f73940c3780 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:36.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.203+0000 7f739b7a4640 1 --2- 172.21.15.116:0/2415892931 >> v2:172.21.15.174:3300/0 conn(0x7f73940c0b30 0x7f73941560a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:36.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.203+0000 7f739b7a4640 1 --2- 172.21.15.116:0/2415892931 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f739414fa60 0x7f7394073490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:36.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.203+0000 7f739b7a4640 1 -- 172.21.15.116:0/2415892931 >> 172.21.15.116:0/2415892931 conn(0x7f73940c4200 msgr2=0x7f73940c4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:04:36.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.203+0000 7f739b7a4640 1 -- 172.21.15.116:0/2415892931 shutdown_connections 2024-04-25T00:04:36.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.203+0000 7f739b7a4640 1 -- 172.21.15.116:0/2415892931 wait complete. 2024-04-25T00:04:36.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.203+0000 7f739b7a4640 1 Processor -- start 2024-04-25T00:04:36.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.203+0000 7f739b7a4640 1 -- start start 2024-04-25T00:04:36.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.203+0000 7f739b7a4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73940c0b30 0x7f73940c6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:36.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.203+0000 7f739b7a4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f739414fa60 0x7f739414b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:36.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.203+0000 7f739b7a4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f73941565e0 0x7f739414c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:36.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.203+0000 7f739b7a4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f739413ef90 con 0x7f73940c0b30 2024-04-25T00:04:36.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.203+0000 7f739b7a4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f739413ee10 con 0x7f739414fa60 2024-04-25T00:04:36.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.203+0000 7f739b7a4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f739413f110 con 0x7f73941565e0 2024-04-25T00:04:36.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.203+0000 7f7399540640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73940c0b30 0x7f73940c6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:36.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.203+0000 7f7399540640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73940c0b30 0x7f73940c6fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53716/0 (socket says 172.21.15.116:53716) 2024-04-25T00:04:36.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.203+0000 7f7399540640 1 -- 172.21.15.116:0/2159826389 learned_addr learned my addr 172.21.15.116:0/2159826389 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:04:36.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.203+0000 7f7399d41640 1 --2- 172.21.15.116:0/2159826389 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f73941565e0 0x7f739414c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:36.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.203+0000 7f7398d3f640 1 --2- 172.21.15.116:0/2159826389 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f739414fa60 0x7f739414b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:36.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.203+0000 7f7399540640 1 -- 172.21.15.116:0/2159826389 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f73941565e0 msgr2=0x7f739414c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:36.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.203+0000 7f7399540640 1 --2- 172.21.15.116:0/2159826389 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f73941565e0 0x7f739414c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:36.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.203+0000 7f7399540640 1 -- 172.21.15.116:0/2159826389 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f739414fa60 msgr2=0x7f739414b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:36.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.203+0000 7f7399540640 1 --2- 172.21.15.116:0/2159826389 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f739414fa60 0x7f739414b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:36.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.203+0000 7f7399540640 1 -- 172.21.15.116:0/2159826389 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f739414ce60 con 0x7f73940c0b30 2024-04-25T00:04:36.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.203+0000 7f7398d3f640 1 --2- 172.21.15.116:0/2159826389 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f739414fa60 0x7f739414b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:04:36.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.203+0000 7f7399d41640 1 --2- 172.21.15.116:0/2159826389 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f73941565e0 0x7f739414c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:04:36.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.203+0000 7f7399540640 1 --2- 172.21.15.116:0/2159826389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73940c0b30 0x7f73940c6fc0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f739000d5f0 tx=0x7f739000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:36.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.203+0000 7f7389ffb640 1 -- 172.21.15.116:0/2159826389 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73900125f0 con 0x7f73940c0b30 2024-04-25T00:04:36.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.203+0000 7f739b7a4640 1 -- 172.21.15.116:0/2159826389 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f739414d140 con 0x7f73940c0b30 2024-04-25T00:04:36.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.203+0000 7f7389ffb640 1 -- 172.21.15.116:0/2159826389 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7390019070 con 0x7f73940c0b30 2024-04-25T00:04:36.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.203+0000 7f7389ffb640 1 -- 172.21.15.116:0/2159826389 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73900129d0 con 0x7f73940c0b30 2024-04-25T00:04:36.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.203+0000 7f739b7a4640 1 -- 172.21.15.116:0/2159826389 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73940c67c0 con 0x7f73940c0b30 2024-04-25T00:04:36.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.207+0000 7f739b7a4640 1 -- 172.21.15.116:0/2159826389 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7394068a60 con 0x7f73940c0b30 2024-04-25T00:04:36.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.207+0000 7f7389ffb640 1 -- 172.21.15.116:0/2159826389 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7390012b70 con 0x7f73940c0b30 2024-04-25T00:04:36.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.207+0000 7f7389ffb640 1 --2- 172.21.15.116:0/2159826389 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f73780678c0 0x7f7378069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:36.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.207+0000 7f7389ffb640 1 -- 172.21.15.116:0/2159826389 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7010+0+0 (secure 0 0 0) 0x7f7390086720 con 0x7f73940c0b30 2024-04-25T00:04:36.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.207+0000 7f7398d3f640 1 --2- 172.21.15.116:0/2159826389 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f73780678c0 0x7f7378069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:36.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.207+0000 7f7398d3f640 1 --2- 172.21.15.116:0/2159826389 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f73780678c0 0x7f7378069d80 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f73940c3990 tx=0x7f7384008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:36.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.211+0000 7f7389ffb640 1 -- 172.21.15.116:0/2159826389 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f7390056ca0 con 0x7f73940c0b30 2024-04-25T00:04:36.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.319+0000 7f6681149640 1 -- 172.21.15.116:0/2317671506 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f667c149db0 con 0x7f667c154a50 2024-04-25T00:04:36.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.319+0000 7f66737fe640 1 -- 172.21.15.116:0/2317671506 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+5666 (secure 0 0 0) 0x7f666c056340 con 0x7f667c154a50 2024-04-25T00:04:36.320 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:04:36.320 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:04:35.419303+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4879},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4879":{"gid":4879,"name":"b","rank":0,"incarnation":32,"state":"up:replay","state_seq":1,"addr":"172.21.15.174:6837/1889314958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1889314958},{"type":"v1","addr":"172.21.15.174:6837","nonce":1889314958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:03:34.579825+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4735},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4735":{"gid":4735,"name":"a","rank":0,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.116:6835/2004876235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":2004876235},{"type":"v1","addr":"172.21.15.116:6835","nonce":2004876235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4735,"qdb_cluster":[4735]},"id":2}]} 2024-04-25T00:04:36.320 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 32 2024-04-25T00:04:36.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.319+0000 7f6681149640 1 -- 172.21.15.116:0/2317671506 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f665c0678c0 msgr2=0x7f665c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:36.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.319+0000 7f6681149640 1 --2- 172.21.15.116:0/2317671506 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f665c0678c0 0x7f665c069d80 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6674009ab0 tx=0x7f6674008040 comp rx=0 tx=0).stop 2024-04-25T00:04:36.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.319+0000 7f6681149640 1 -- 172.21.15.116:0/2317671506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f667c154a50 msgr2=0x7f667c1469b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:36.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.319+0000 7f6681149640 1 --2- 172.21.15.116:0/2317671506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f667c154a50 0x7f667c1469b0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f666c001380 tx=0x7f666c007bc0 comp rx=0 tx=0).stop 2024-04-25T00:04:36.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.319+0000 7f6681149640 1 -- 172.21.15.116:0/2317671506 shutdown_connections 2024-04-25T00:04:36.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.319+0000 7f6681149640 1 --2- 172.21.15.116:0/2317671506 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f665c0678c0 0x7f665c069d80 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:36.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.319+0000 7f6681149640 1 --2- 172.21.15.116:0/2317671506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f667c154a50 0x7f667c1469b0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:36.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.319+0000 7f6681149640 1 --2- 172.21.15.116:0/2317671506 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f667c147020 0x7f667c146470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:36.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.319+0000 7f6681149640 1 --2- 172.21.15.116:0/2317671506 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6668004810 0x7f667c0c1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:36.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.319+0000 7f6681149640 1 -- 172.21.15.116:0/2317671506 >> 172.21.15.116:0/2317671506 conn(0x7f667c06c720 msgr2=0x7f667c152260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:04:36.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.323+0000 7f6681149640 1 -- 172.21.15.116:0/2317671506 shutdown_connections 2024-04-25T00:04:36.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.323+0000 7f6681149640 1 -- 172.21.15.116:0/2317671506 wait complete. 2024-04-25T00:04:36.339 INFO:tasks.mds_thrash.fs.[a]:mds.c down, removed from mdsmap 2024-04-25T00:04:36.339 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-04-25T00:04:36.340 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:04:36.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.407+0000 7f739b7a4640 1 -- 172.21.15.116:0/2159826389 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f73940a6e30 con 0x7f73940c0b30 2024-04-25T00:04:36.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.407+0000 7f7389ffb640 1 -- 172.21.15.116:0/2159826389 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+5666 (secure 0 0 0) 0x7f73900565a0 con 0x7f73940c0b30 2024-04-25T00:04:36.409 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:04:36.409 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:04:35.419303+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4879},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4879":{"gid":4879,"name":"b","rank":0,"incarnation":32,"state":"up:replay","state_seq":1,"addr":"172.21.15.174:6837/1889314958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1889314958},{"type":"v1","addr":"172.21.15.174:6837","nonce":1889314958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:03:34.579825+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4735},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4735":{"gid":4735,"name":"a","rank":0,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.116:6835/2004876235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":2004876235},{"type":"v1","addr":"172.21.15.116:6835","nonce":2004876235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4735,"qdb_cluster":[4735]},"id":2}]} 2024-04-25T00:04:36.409 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 32 2024-04-25T00:04:36.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.407+0000 7f739b7a4640 1 -- 172.21.15.116:0/2159826389 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f73780678c0 msgr2=0x7f7378069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:36.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.407+0000 7f739b7a4640 1 --2- 172.21.15.116:0/2159826389 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f73780678c0 0x7f7378069d80 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f73940c3990 tx=0x7f7384008040 comp rx=0 tx=0).stop 2024-04-25T00:04:36.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.407+0000 7f739b7a4640 1 -- 172.21.15.116:0/2159826389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73940c0b30 msgr2=0x7f73940c6fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:36.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.411+0000 7f739b7a4640 1 --2- 172.21.15.116:0/2159826389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73940c0b30 0x7f73940c6fc0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f739000d5f0 tx=0x7f739000dab0 comp rx=0 tx=0).stop 2024-04-25T00:04:36.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.411+0000 7f739b7a4640 1 -- 172.21.15.116:0/2159826389 shutdown_connections 2024-04-25T00:04:36.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.411+0000 7f739b7a4640 1 --2- 172.21.15.116:0/2159826389 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f73780678c0 0x7f7378069d80 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:36.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.411+0000 7f739b7a4640 1 --2- 172.21.15.116:0/2159826389 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f73941565e0 0x7f739414c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:36.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.411+0000 7f739b7a4640 1 --2- 172.21.15.116:0/2159826389 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f739414fa60 0x7f739414b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:36.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.411+0000 7f739b7a4640 1 --2- 172.21.15.116:0/2159826389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73940c0b30 0x7f73940c6fc0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:36.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.411+0000 7f739b7a4640 1 -- 172.21.15.116:0/2159826389 >> 172.21.15.116:0/2159826389 conn(0x7f73940c4200 msgr2=0x7f7394153500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:04:36.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.411+0000 7f739b7a4640 1 -- 172.21.15.116:0/2159826389 shutdown_connections 2024-04-25T00:04:36.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.411+0000 7f739b7a4640 1 -- 172.21.15.116:0/2159826389 wait complete. 2024-04-25T00:04:36.429 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:04:36.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.507+0000 7f50d2409640 1 Processor -- start 2024-04-25T00:04:36.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.507+0000 7f50d2409640 1 -- start start 2024-04-25T00:04:36.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.507+0000 7f50d2409640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50cc14c720 0x7f50cc1563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:36.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.507+0000 7f50d2409640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f50cc0a94a0 0x7f50cc0c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:36.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.507+0000 7f50d2409640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f50cc0c2f50 0x7f50cc071680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:36.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.507+0000 7f50d2409640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f50cc153460 con 0x7f50cc0c2f50 2024-04-25T00:04:36.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.507+0000 7f50d2409640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f50cc071220 con 0x7f50cc0a94a0 2024-04-25T00:04:36.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.507+0000 7f50d2409640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f50cc152420 con 0x7f50cc14c720 2024-04-25T00:04:36.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.507+0000 7f50d09a6640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f50cc0c2f50 0x7f50cc071680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:36.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.511+0000 7f50d09a6640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f50cc0c2f50 0x7f50cc071680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53728/0 (socket says 172.21.15.116:53728) 2024-04-25T00:04:36.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.511+0000 7f50d09a6640 1 -- 172.21.15.116:0/4102595289 learned_addr learned my addr 172.21.15.116:0/4102595289 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:04:36.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.511+0000 7f50cbfff640 1 --2- 172.21.15.116:0/4102595289 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50cc14c720 0x7f50cc1563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:36.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.511+0000 7f50cb7fe640 1 --2- 172.21.15.116:0/4102595289 >> v2:172.21.15.174:3300/0 conn(0x7f50cc0a94a0 0x7f50cc0c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:36.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.511+0000 7f50d09a6640 1 -- 172.21.15.116:0/4102595289 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50cc14c720 msgr2=0x7f50cc1563d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:04:36.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.511+0000 7f50d09a6640 1 --2- 172.21.15.116:0/4102595289 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50cc14c720 0x7f50cc1563d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:36.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.511+0000 7f50d09a6640 1 -- 172.21.15.116:0/4102595289 >> v2:172.21.15.174:3300/0 conn(0x7f50cc0a94a0 msgr2=0x7f50cc0c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:04:36.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.511+0000 7f50d09a6640 1 --2- 172.21.15.116:0/4102595289 >> v2:172.21.15.174:3300/0 conn(0x7f50cc0a94a0 0x7f50cc0c2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:36.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.511+0000 7f50d09a6640 1 -- 172.21.15.116:0/4102595289 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50cc14cb00 con 0x7f50cc0c2f50 2024-04-25T00:04:36.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.511+0000 7f50d09a6640 1 --2- 172.21.15.116:0/4102595289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50cc0c2f50 0x7f50cc071680 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f50cc0c4bc0 tx=0x7f50c401c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a8441bdd7df97c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:36.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.511+0000 7f50caffd640 1 -- 172.21.15.116:0/4102595289 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50c40a56a0 con 0x7f50cc0c2f50 2024-04-25T00:04:36.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.511+0000 7f50caffd640 1 -- 172.21.15.116:0/4102595289 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50c40a5840 con 0x7f50cc0c2f50 2024-04-25T00:04:36.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.511+0000 7f50caffd640 1 -- 172.21.15.116:0/4102595289 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50c40a5c10 con 0x7f50cc0c2f50 2024-04-25T00:04:36.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.511+0000 7f50d2409640 1 -- 172.21.15.116:0/4102595289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50cc0c2f50 msgr2=0x7f50cc071680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:36.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.511+0000 7f50d2409640 1 --2- 172.21.15.116:0/4102595289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50cc0c2f50 0x7f50cc071680 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f50cc0c4bc0 tx=0x7f50c401c930 comp rx=0 tx=0).stop 2024-04-25T00:04:36.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.511+0000 7f50d2409640 1 -- 172.21.15.116:0/4102595289 shutdown_connections 2024-04-25T00:04:36.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.511+0000 7f50d2409640 1 --2- 172.21.15.116:0/4102595289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50cc0c2f50 0x7f50cc071680 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:36.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.511+0000 7f50d2409640 1 --2- 172.21.15.116:0/4102595289 >> v2:172.21.15.174:3300/0 conn(0x7f50cc0a94a0 0x7f50cc0c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:36.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.511+0000 7f50d2409640 1 --2- 172.21.15.116:0/4102595289 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50cc14c720 0x7f50cc1563d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:36.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.511+0000 7f50d2409640 1 -- 172.21.15.116:0/4102595289 >> 172.21.15.116:0/4102595289 conn(0x7f50cc0c3b00 msgr2=0x7f50cc0c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:04:36.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.511+0000 7f50d2409640 1 -- 172.21.15.116:0/4102595289 shutdown_connections 2024-04-25T00:04:36.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.511+0000 7f50d2409640 1 -- 172.21.15.116:0/4102595289 wait complete. 2024-04-25T00:04:36.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.515+0000 7f50d2409640 1 Processor -- start 2024-04-25T00:04:36.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.515+0000 7f50d2409640 1 -- start start 2024-04-25T00:04:36.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.515+0000 7f50d2409640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50cc0a94a0 0x7f50cc1ee210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:36.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.515+0000 7f50d2409640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50cc0c2f50 0x7f50cc1f2020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:36.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.515+0000 7f50d2409640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50cc14c720 0x7f50cc1408d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:36.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.515+0000 7f50d2409640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f50cc071fd0 con 0x7f50cc0c2f50 2024-04-25T00:04:36.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.515+0000 7f50d2409640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f50cc152f40 con 0x7f50cc14c720 2024-04-25T00:04:36.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.515+0000 7f50d2409640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f50cc072150 con 0x7f50cc0a94a0 2024-04-25T00:04:36.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.515+0000 7f50cb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50cc0c2f50 0x7f50cc1f2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:36.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.515+0000 7f50cb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50cc0c2f50 0x7f50cc1f2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53736/0 (socket says 172.21.15.116:53736) 2024-04-25T00:04:36.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.515+0000 7f50cb7fe640 1 -- 172.21.15.116:0/4109971236 learned_addr learned my addr 172.21.15.116:0/4109971236 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:04:36.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.515+0000 7f50cbfff640 1 --2- 172.21.15.116:0/4109971236 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50cc0a94a0 0x7f50cc1ee210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:36.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.515+0000 7f50d09a6640 1 --2- 172.21.15.116:0/4109971236 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50cc14c720 0x7f50cc1408d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:36.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.515+0000 7f50cb7fe640 1 -- 172.21.15.116:0/4109971236 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50cc0a94a0 msgr2=0x7f50cc1ee210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:36.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.515+0000 7f50cb7fe640 1 --2- 172.21.15.116:0/4109971236 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50cc0a94a0 0x7f50cc1ee210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:36.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.515+0000 7f50cb7fe640 1 -- 172.21.15.116:0/4109971236 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50cc14c720 msgr2=0x7f50cc1408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:36.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.515+0000 7f50cb7fe640 1 --2- 172.21.15.116:0/4109971236 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50cc14c720 0x7f50cc1408d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:36.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.515+0000 7f50cb7fe640 1 -- 172.21.15.116:0/4109971236 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50cc14b8f0 con 0x7f50cc0c2f50 2024-04-25T00:04:36.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.515+0000 7f50d09a6640 1 --2- 172.21.15.116:0/4109971236 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50cc14c720 0x7f50cc1408d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:04:36.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.515+0000 7f50cb7fe640 1 --2- 172.21.15.116:0/4109971236 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50cc0c2f50 0x7f50cc1f2020 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f50bc00d800 tx=0x7f50bc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:36.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.515+0000 7f50c8ff9640 1 -- 172.21.15.116:0/4109971236 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50bc00b6c0 con 0x7f50cc0c2f50 2024-04-25T00:04:36.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.515+0000 7f50c8ff9640 1 -- 172.21.15.116:0/4109971236 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50bc019070 con 0x7f50cc0c2f50 2024-04-25T00:04:36.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.515+0000 7f50c8ff9640 1 -- 172.21.15.116:0/4109971236 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50bc00ba60 con 0x7f50cc0c2f50 2024-04-25T00:04:36.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.515+0000 7f50d2409640 1 -- 172.21.15.116:0/4109971236 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50cc14bbd0 con 0x7f50cc0c2f50 2024-04-25T00:04:36.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.515+0000 7f50d2409640 1 -- 172.21.15.116:0/4109971236 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50cc1f1820 con 0x7f50cc0c2f50 2024-04-25T00:04:36.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.515+0000 7f50d2409640 1 -- 172.21.15.116:0/4109971236 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50cc068350 con 0x7f50cc0c2f50 2024-04-25T00:04:36.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.519+0000 7f50c8ff9640 1 -- 172.21.15.116:0/4109971236 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f50bc00bc00 con 0x7f50cc0c2f50 2024-04-25T00:04:36.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.519+0000 7f50c8ff9640 1 --2- 172.21.15.116:0/4109971236 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f50b80678f0 0x7f50b8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:36.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.519+0000 7f50c8ff9640 1 -- 172.21.15.116:0/4109971236 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7010+0+0 (secure 0 0 0) 0x7f50bc086590 con 0x7f50cc0c2f50 2024-04-25T00:04:36.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.519+0000 7f50cbfff640 1 --2- 172.21.15.116:0/4109971236 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f50b80678f0 0x7f50b8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:36.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.519+0000 7f50cbfff640 1 --2- 172.21.15.116:0/4109971236 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f50b80678f0 0x7f50b8069db0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f50c0009670 tx=0x7f50c0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:36.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.523+0000 7f50c8ff9640 1 -- 172.21.15.116:0/4109971236 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f50bc056c40 con 0x7f50cc0c2f50 2024-04-25T00:04:36.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.715+0000 7f50d2409640 1 -- 172.21.15.116:0/4109971236 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f50cc0c3640 con 0x7f50cc0c2f50 2024-04-25T00:04:36.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.715+0000 7f50c8ff9640 1 -- 172.21.15.116:0/4109971236 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) ==== 94+0+5670 (secure 0 0 0) 0x7f50bc056540 con 0x7f50cc0c2f50 2024-04-25T00:04:36.717 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:04:36.717 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":33,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:04:35.680312+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4879},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4879":{"gid":4879,"name":"b","rank":0,"incarnation":32,"state":"up:reconnect","state_seq":13,"addr":"172.21.15.174:6837/1889314958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1889314958},{"type":"v1","addr":"172.21.15.174:6837","nonce":1889314958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:03:34.579825+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4735},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4735":{"gid":4735,"name":"a","rank":0,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.116:6835/2004876235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":2004876235},{"type":"v1","addr":"172.21.15.116:6835","nonce":2004876235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4735,"qdb_cluster":[4735]},"id":2}]} 2024-04-25T00:04:36.717 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 33 2024-04-25T00:04:36.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.719+0000 7f50d2409640 1 -- 172.21.15.116:0/4109971236 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f50b80678f0 msgr2=0x7f50b8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:36.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.719+0000 7f50d2409640 1 --2- 172.21.15.116:0/4109971236 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f50b80678f0 0x7f50b8069db0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f50c0009670 tx=0x7f50c0008040 comp rx=0 tx=0).stop 2024-04-25T00:04:36.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.719+0000 7f50d2409640 1 -- 172.21.15.116:0/4109971236 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50cc0c2f50 msgr2=0x7f50cc1f2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:36.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.719+0000 7f50d2409640 1 --2- 172.21.15.116:0/4109971236 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50cc0c2f50 0x7f50cc1f2020 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f50bc00d800 tx=0x7f50bc00dcc0 comp rx=0 tx=0).stop 2024-04-25T00:04:36.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.719+0000 7f50d2409640 1 -- 172.21.15.116:0/4109971236 shutdown_connections 2024-04-25T00:04:36.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.719+0000 7f50d2409640 1 --2- 172.21.15.116:0/4109971236 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f50b80678f0 0x7f50b8069db0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:36.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.719+0000 7f50d2409640 1 --2- 172.21.15.116:0/4109971236 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50cc14c720 0x7f50cc1408d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:36.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.719+0000 7f50d2409640 1 --2- 172.21.15.116:0/4109971236 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50cc0c2f50 0x7f50cc1f2020 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:36.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.719+0000 7f50d2409640 1 --2- 172.21.15.116:0/4109971236 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50cc0a94a0 0x7f50cc1ee210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:36.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.719+0000 7f50d2409640 1 -- 172.21.15.116:0/4109971236 >> 172.21.15.116:0/4109971236 conn(0x7f50cc0c3b00 msgr2=0x7f50cc150290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:04:36.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.719+0000 7f50d2409640 1 -- 172.21.15.116:0/4109971236 shutdown_connections 2024-04-25T00:04:36.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:36.719+0000 7f50d2409640 1 -- 172.21.15.116:0/4109971236 wait complete. 2024-04-25T00:04:36.737 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 33, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:04.678737+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 33, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_4879": { "addr": "172.21.15.174:6837/1889314958", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6836", "nonce": 1889314958, "type": "v2" }, { "addr": "172.21.15.174:6837", "nonce": 1889314958, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4879, "incarnation": 32, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:reconnect", "state_seq": 13 } }, "last_failure": 0, "last_failure_osd_epoch": 33, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T00:04:35.680312+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4879 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:18.825887+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 24, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_4735": { "addr": "172.21.15.116:6835/2004876235", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6833", "nonce": 2004876235, "type": "v2" }, { "addr": "172.21.15.116:6835", "nonce": 2004876235, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4735, "incarnation": 20, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 14 } }, "last_failure": 0, "last_failure_osd_epoch": 31, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-04-25T00:03:34.579825+0000", "qdb_cluster": [ 4735 ], "qdb_leader": 4735, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4735 } } } ], "standbys": [ { "addr": "172.21.15.174:6835/2883220846", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6834", "nonce": 2883220846, "type": "v2" }, { "addr": "172.21.15.174:6835", "nonce": 2883220846, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4235, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-25T00:04:38.430 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:04:38.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.567+0000 7f247c2d1640 1 Processor -- start 2024-04-25T00:04:38.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.567+0000 7f247c2d1640 1 -- start start 2024-04-25T00:04:38.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.567+0000 7f247c2d1640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2474152d80 0x7f247406bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:38.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.567+0000 7f247c2d1640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f247406c290 0x7f2474157390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:38.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.567+0000 7f247c2d1640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24740c9150 con 0x7f247406d5f0 2024-04-25T00:04:38.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.567+0000 7f247c2d1640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f24740c9d60 con 0x7f247406c290 2024-04-25T00:04:38.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.567+0000 7f247c2d1640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f24740c9430 con 0x7f2474152d80 2024-04-25T00:04:38.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.567+0000 7f247986c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f247406d5f0 0x7f2474156c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:49438/0 (socket says 172.21.15.116:49438) 2024-04-25T00:04:38.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.567+0000 7f247986c640 1 -- 172.21.15.116:0/1849621733 learned_addr learned my addr 172.21.15.116:0/1849621733 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:04:38.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.567+0000 7f247a86e640 1 --2- 172.21.15.116:0/1849621733 >> v2:172.21.15.174:3300/0 conn(0x7f247406c290 0x7f2474157390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:38.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.567+0000 7f247a06d640 1 --2- 172.21.15.116:0/1849621733 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2474152d80 0x7f247406bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:38.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.567+0000 7f247906b640 1 -- 172.21.15.116:0/1849621733 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 228916333 0 0) 0x7f24740c9150 con 0x7f247406d5f0 2024-04-25T00:04:38.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.567+0000 7f247906b640 1 -- 172.21.15.116:0/1849621733 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2460003530 con 0x7f247406d5f0 2024-04-25T00:04:38.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.567+0000 7f247906b640 1 -- 172.21.15.116:0/1849621733 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 951775262 0 0) 0x7f2460003530 con 0x7f247406d5f0 2024-04-25T00:04:38.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.567+0000 7f247906b640 1 -- 172.21.15.116:0/1849621733 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2474152d80 msgr2=0x7f247406bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:04:38.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.567+0000 7f247906b640 1 --2- 172.21.15.116:0/1849621733 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2474152d80 0x7f247406bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:38.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.567+0000 7f247906b640 1 -- 172.21.15.116:0/1849621733 >> v2:172.21.15.174:3300/0 conn(0x7f247406c290 msgr2=0x7f2474157390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:04:38.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.567+0000 7f247906b640 1 --2- 172.21.15.116:0/1849621733 >> v2:172.21.15.174:3300/0 conn(0x7f247406c290 0x7f2474157390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:38.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.567+0000 7f247906b640 1 -- 172.21.15.116:0/1849621733 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24740c9710 con 0x7f247406d5f0 2024-04-25T00:04:38.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.567+0000 7f247906b640 1 -- 172.21.15.116:0/1849621733 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f2464001680 con 0x7f247406d5f0 2024-04-25T00:04:38.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.567+0000 7f247906b640 1 -- 172.21.15.116:0/1849621733 >> v1:172.21.15.116:6789/0 conn(0x7f247406d5f0 legacy=0x7f2474156c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:38.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.567+0000 7f247906b640 1 --2- 172.21.15.116:0/1849621733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24600039c0 0x7f2460003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:38.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.567+0000 7f247906b640 1 --2- 172.21.15.116:0/1849621733 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2460004810 0x7f2460004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:38.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.567+0000 7f247906b640 1 --2- 172.21.15.116:0/1849621733 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2474152d80 0x7f24600051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:38.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.567+0000 7f247906b640 1 -- 172.21.15.116:0/1849621733 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f24740c9d60 con 0x7f24600039c0 2024-04-25T00:04:38.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.567+0000 7f247906b640 1 -- 172.21.15.116:0/1849621733 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f24740c9430 con 0x7f2460004810 2024-04-25T00:04:38.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.567+0000 7f247906b640 1 -- 172.21.15.116:0/1849621733 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f246000b9d0 con 0x7f2474152d80 2024-04-25T00:04:38.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.567+0000 7f247a06d640 1 --2- 172.21.15.116:0/1849621733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24600039c0 0x7f2460003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:38.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.567+0000 7f247a86e640 1 --2- 172.21.15.116:0/1849621733 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2474152d80 0x7f24600051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:38.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.567+0000 7f247986c640 1 --2- 172.21.15.116:0/1849621733 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2460004810 0x7f2460004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:38.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.567+0000 7f247a06d640 1 -- 172.21.15.116:0/1849621733 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2474152d80 msgr2=0x7f24600051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:04:38.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.567+0000 7f247a06d640 1 --2- 172.21.15.116:0/1849621733 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2474152d80 0x7f24600051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:38.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.567+0000 7f247a06d640 1 -- 172.21.15.116:0/1849621733 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2460004810 msgr2=0x7f2460004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:04:38.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.567+0000 7f247a06d640 1 --2- 172.21.15.116:0/1849621733 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2460004810 0x7f2460004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:38.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.567+0000 7f247a06d640 1 -- 172.21.15.116:0/1849621733 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2460003530 con 0x7f24600039c0 2024-04-25T00:04:38.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.571+0000 7f247a06d640 1 --2- 172.21.15.116:0/1849621733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24600039c0 0x7f2460003de0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f2470009b10 tx=0x7f247001ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ae05211ceda3f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:38.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.571+0000 7f247906b640 1 -- 172.21.15.116:0/1849621733 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24700b2070 con 0x7f24600039c0 2024-04-25T00:04:38.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.571+0000 7f247906b640 1 -- 172.21.15.116:0/1849621733 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24700a5550 con 0x7f24600039c0 2024-04-25T00:04:38.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.571+0000 7f247906b640 1 -- 172.21.15.116:0/1849621733 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24700096b0 con 0x7f24600039c0 2024-04-25T00:04:38.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.571+0000 7f247c2d1640 1 -- 172.21.15.116:0/1849621733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24600039c0 msgr2=0x7f2460003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:38.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.571+0000 7f247c2d1640 1 --2- 172.21.15.116:0/1849621733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f24600039c0 0x7f2460003de0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f2470009b10 tx=0x7f247001ca40 comp rx=0 tx=0).stop 2024-04-25T00:04:38.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.571+0000 7f247a86e640 1 -- 172.21.15.116:0/1849621733 reap_dead start 2024-04-25T00:04:38.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.571+0000 7f247c2d1640 1 -- 172.21.15.116:0/1849621733 shutdown_connections 2024-04-25T00:04:38.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.571+0000 7f247c2d1640 1 -- 172.21.15.116:0/1849621733 >> 172.21.15.116:0/1849621733 conn(0x7f24740acdd0 msgr2=0x7f24740ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:04:38.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.571+0000 7f247c2d1640 1 -- 172.21.15.116:0/1849621733 shutdown_connections 2024-04-25T00:04:38.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.571+0000 7f247c2d1640 1 -- 172.21.15.116:0/1849621733 wait complete. 2024-04-25T00:04:38.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.571+0000 7f247c2d1640 1 Processor -- start 2024-04-25T00:04:38.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.571+0000 7f247c2d1640 1 -- start start 2024-04-25T00:04:38.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.575+0000 7f247c2d1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f24600039c0 0x7f247414c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:38.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.575+0000 7f247c2d1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f247415eda0 0x7f247415e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:38.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.575+0000 7f247c2d1640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f24741f1d50 0x7f24741f2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:38.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.575+0000 7f247c2d1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f24740caf00 con 0x7f247415eda0 2024-04-25T00:04:38.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.575+0000 7f247c2d1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f24740cad80 con 0x7f24600039c0 2024-04-25T00:04:38.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.575+0000 7f247c2d1640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f24741608c0 con 0x7f24741f1d50 2024-04-25T00:04:38.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.575+0000 7f247986c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f247415eda0 0x7f247415e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:38.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.575+0000 7f247986c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f247415eda0 0x7f247415e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53754/0 (socket says 172.21.15.116:53754) 2024-04-25T00:04:38.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.575+0000 7f247986c640 1 -- 172.21.15.116:0/505914416 learned_addr learned my addr 172.21.15.116:0/505914416 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:04:38.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.575+0000 7f247a06d640 1 --2- 172.21.15.116:0/505914416 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f24600039c0 0x7f247414c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:38.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.575+0000 7f247a86e640 1 --2- 172.21.15.116:0/505914416 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f24741f1d50 0x7f24741f2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:38.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.575+0000 7f247986c640 1 -- 172.21.15.116:0/505914416 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f24741f1d50 msgr2=0x7f24741f2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:38.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.575+0000 7f247986c640 1 --2- 172.21.15.116:0/505914416 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f24741f1d50 0x7f24741f2200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:38.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.575+0000 7f247986c640 1 -- 172.21.15.116:0/505914416 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f24600039c0 msgr2=0x7f247414c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:38.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.575+0000 7f247986c640 1 --2- 172.21.15.116:0/505914416 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f24600039c0 0x7f247414c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:38.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.575+0000 7f247986c640 1 -- 172.21.15.116:0/505914416 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24741f2a40 con 0x7f247415eda0 2024-04-25T00:04:38.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.575+0000 7f247a06d640 1 --2- 172.21.15.116:0/505914416 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f24600039c0 0x7f247414c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:04:38.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.575+0000 7f247986c640 1 --2- 172.21.15.116:0/505914416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f247415eda0 0x7f247415e2c0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f2464001870 tx=0x7f246400ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:38.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.575+0000 7f246affd640 1 -- 172.21.15.116:0/505914416 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f246400ee60 con 0x7f247415eda0 2024-04-25T00:04:38.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.575+0000 7f246affd640 1 -- 172.21.15.116:0/505914416 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2464018070 con 0x7f247415eda0 2024-04-25T00:04:38.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.575+0000 7f246affd640 1 -- 172.21.15.116:0/505914416 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f246400cab0 con 0x7f247415eda0 2024-04-25T00:04:38.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.575+0000 7f247c2d1640 1 -- 172.21.15.116:0/505914416 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2474152150 con 0x7f247415eda0 2024-04-25T00:04:38.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.575+0000 7f247c2d1640 1 -- 172.21.15.116:0/505914416 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f247415c7b0 con 0x7f247415eda0 2024-04-25T00:04:38.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.575+0000 7f247c2d1640 1 -- 172.21.15.116:0/505914416 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2474068350 con 0x7f247415eda0 2024-04-25T00:04:38.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.575+0000 7f246affd640 1 -- 172.21.15.116:0/505914416 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f246400cc50 con 0x7f247415eda0 2024-04-25T00:04:38.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.575+0000 7f246affd640 1 --2- 172.21.15.116:0/505914416 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f24580678f0 0x7f2458069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:38.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.575+0000 7f246affd640 1 -- 172.21.15.116:0/505914416 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7010+0+0 (secure 0 0 0) 0x7f2464085c70 con 0x7f247415eda0 2024-04-25T00:04:38.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.575+0000 7f247a06d640 1 --2- 172.21.15.116:0/505914416 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f24580678f0 0x7f2458069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:38.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.579+0000 7f247a06d640 1 --2- 172.21.15.116:0/505914416 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f24580678f0 0x7f2458069db0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f24700af000 tx=0x7f24700046d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:38.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.583+0000 7f246affd640 1 -- 172.21.15.116:0/505914416 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f2464056320 con 0x7f247415eda0 2024-04-25T00:04:38.738 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:04:38.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.827+0000 7f247c2d1640 1 -- 172.21.15.116:0/505914416 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f24740cab60 con 0x7f247415eda0 2024-04-25T00:04:38.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.827+0000 7f246affd640 1 -- 172.21.15.116:0/505914416 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+5674 (secure 0 0 0) 0x7f2464055c20 con 0x7f247415eda0 2024-04-25T00:04:38.829 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:04:38.829 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:04:38.435723+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4879},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4879":{"gid":4879,"name":"b","rank":0,"incarnation":32,"state":"up:active","state_seq":15,"addr":"172.21.15.174:6837/1889314958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1889314958},{"type":"v1","addr":"172.21.15.174:6837","nonce":1889314958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4879,"qdb_cluster":[4879]},"id":1},{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:03:34.579825+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4735},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4735":{"gid":4735,"name":"a","rank":0,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.116:6835/2004876235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":2004876235},{"type":"v1","addr":"172.21.15.116:6835","nonce":2004876235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4735,"qdb_cluster":[4735]},"id":2}]} 2024-04-25T00:04:38.829 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 35 2024-04-25T00:04:38.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.831+0000 7f247c2d1640 1 -- 172.21.15.116:0/505914416 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f24580678f0 msgr2=0x7f2458069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:38.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.831+0000 7f247c2d1640 1 --2- 172.21.15.116:0/505914416 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f24580678f0 0x7f2458069db0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f24700af000 tx=0x7f24700046d0 comp rx=0 tx=0).stop 2024-04-25T00:04:38.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.831+0000 7f247c2d1640 1 -- 172.21.15.116:0/505914416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f247415eda0 msgr2=0x7f247415e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:38.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.831+0000 7f247c2d1640 1 --2- 172.21.15.116:0/505914416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f247415eda0 0x7f247415e2c0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f2464001870 tx=0x7f246400ec10 comp rx=0 tx=0).stop 2024-04-25T00:04:38.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.831+0000 7f247c2d1640 1 -- 172.21.15.116:0/505914416 shutdown_connections 2024-04-25T00:04:38.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.831+0000 7f247c2d1640 1 --2- 172.21.15.116:0/505914416 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f24580678f0 0x7f2458069db0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:38.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.831+0000 7f247c2d1640 1 --2- 172.21.15.116:0/505914416 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f24741f1d50 0x7f24741f2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:38.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.831+0000 7f247c2d1640 1 --2- 172.21.15.116:0/505914416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f247415eda0 0x7f247415e2c0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:38.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.831+0000 7f247c2d1640 1 --2- 172.21.15.116:0/505914416 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f24600039c0 0x7f247414c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:38.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.831+0000 7f247c2d1640 1 -- 172.21.15.116:0/505914416 >> 172.21.15.116:0/505914416 conn(0x7f24740acdd0 msgr2=0x7f247414e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:04:38.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.831+0000 7f247c2d1640 1 -- 172.21.15.116:0/505914416 shutdown_connections 2024-04-25T00:04:38.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.831+0000 7f247c2d1640 1 -- 172.21.15.116:0/505914416 wait complete. 2024-04-25T00:04:38.849 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:04:38.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.875+0000 7fc7df26a640 1 Processor -- start 2024-04-25T00:04:38.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.875+0000 7fc7df26a640 1 -- start start 2024-04-25T00:04:38.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.875+0000 7fc7df26a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc7d8150d20 0x7fc7d8153620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:38.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.875+0000 7fc7df26a640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7d80c36a0 con 0x7fc7d8071050 2024-04-25T00:04:38.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.875+0000 7fc7df26a640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7d806f8c0 con 0x7fc7d80c3240 2024-04-25T00:04:38.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.875+0000 7fc7df26a640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fc7d806fba0 con 0x7fc7d8150d20 2024-04-25T00:04:38.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.875+0000 7fc7dd006640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc7d8150d20 0x7fc7d8153620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:38.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.875+0000 7fc7dd807640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fc7d8071050 0x7fc7d815df10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:49446/0 (socket says 172.21.15.116:49446) 2024-04-25T00:04:38.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.875+0000 7fc7dd807640 1 -- 172.21.15.116:0/1531172998 learned_addr learned my addr 172.21.15.116:0/1531172998 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:04:38.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.875+0000 7fc7dd006640 1 -- 172.21.15.116:0/1531172998 >> v1:172.21.15.174:6789/0 conn(0x7fc7d80c3240 legacy=0x7fc7d80c3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:04:38.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.875+0000 7fc7dd006640 1 -- 172.21.15.116:0/1531172998 >> v1:172.21.15.116:6789/0 conn(0x7fc7d8071050 legacy=0x7fc7d815df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:38.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.875+0000 7fc7cffff640 1 -- 172.21.15.116:0/1531172998 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2006677302 0 0) 0x7fc7d80c36a0 con 0x7fc7d8071050 2024-04-25T00:04:38.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.875+0000 7fc7dd006640 1 -- 172.21.15.116:0/1531172998 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7d80c5420 con 0x7fc7d8150d20 2024-04-25T00:04:38.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.879+0000 7fc7dd006640 1 --2- 172.21.15.116:0/1531172998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7d8150d20 0x7fc7d8153620 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fc7d40097c0 tx=0x7fc7d401c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fc74d618c37a1b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:38.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.879+0000 7fc7cffff640 1 -- 172.21.15.116:0/1531172998 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7d40b0070 con 0x7fc7d8150d20 2024-04-25T00:04:38.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.879+0000 7fc7cffff640 1 -- 172.21.15.116:0/1531172998 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7d40ab600 con 0x7fc7d8150d20 2024-04-25T00:04:38.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.879+0000 7fc7cffff640 1 -- 172.21.15.116:0/1531172998 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7d40ab9d0 con 0x7fc7d8150d20 2024-04-25T00:04:38.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.879+0000 7fc7df26a640 1 -- 172.21.15.116:0/1531172998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7d8150d20 msgr2=0x7fc7d8153620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:38.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.879+0000 7fc7df26a640 1 --2- 172.21.15.116:0/1531172998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7d8150d20 0x7fc7d8153620 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fc7d40097c0 tx=0x7fc7d401c460 comp rx=0 tx=0).stop 2024-04-25T00:04:38.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.879+0000 7fc7df26a640 1 -- 172.21.15.116:0/1531172998 shutdown_connections 2024-04-25T00:04:38.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.879+0000 7fc7df26a640 1 --2- 172.21.15.116:0/1531172998 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7d8150d20 0x7fc7d8153620 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:38.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.879+0000 7fc7df26a640 1 -- 172.21.15.116:0/1531172998 >> 172.21.15.116:0/1531172998 conn(0x7fc7d80c9160 msgr2=0x7fc7d80c9570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:04:38.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.879+0000 7fc7df26a640 1 -- 172.21.15.116:0/1531172998 shutdown_connections 2024-04-25T00:04:38.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.879+0000 7fc7df26a640 1 -- 172.21.15.116:0/1531172998 wait complete. 2024-04-25T00:04:38.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.879+0000 7fc7df26a640 1 Processor -- start 2024-04-25T00:04:38.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.883+0000 7fc7df26a640 1 -- start start 2024-04-25T00:04:38.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.883+0000 7fc7df26a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc7d8071050 0x7fc7d8143570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:38.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.883+0000 7fc7df26a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7d80c3240 0x7fc7d8152a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:38.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.883+0000 7fc7df26a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7d8150d20 0x7fc7d8152fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:38.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.883+0000 7fc7df26a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d80c4d50 con 0x7fc7d80c3240 2024-04-25T00:04:38.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.883+0000 7fc7df26a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d80c4bd0 con 0x7fc7d8150d20 2024-04-25T00:04:38.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.883+0000 7fc7df26a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fc7d806f4c0 con 0x7fc7d8071050 2024-04-25T00:04:38.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.883+0000 7fc7dc805640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7d80c3240 0x7fc7d8152a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:38.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.883+0000 7fc7dc805640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7d80c3240 0x7fc7d8152a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53776/0 (socket says 172.21.15.116:53776) 2024-04-25T00:04:38.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.883+0000 7fc7dc805640 1 -- 172.21.15.116:0/1754271957 learned_addr learned my addr 172.21.15.116:0/1754271957 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:04:38.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.883+0000 7fc7dd006640 1 --2- 172.21.15.116:0/1754271957 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc7d8071050 0x7fc7d8143570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:38.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.883+0000 7fc7dd807640 1 --2- 172.21.15.116:0/1754271957 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7d8150d20 0x7fc7d8152fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:38.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.883+0000 7fc7dc805640 1 -- 172.21.15.116:0/1754271957 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc7d8071050 msgr2=0x7fc7d8143570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:38.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.883+0000 7fc7dc805640 1 --2- 172.21.15.116:0/1754271957 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc7d8071050 0x7fc7d8143570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:38.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.883+0000 7fc7dc805640 1 -- 172.21.15.116:0/1754271957 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7d8150d20 msgr2=0x7fc7d8152fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:38.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.883+0000 7fc7dc805640 1 --2- 172.21.15.116:0/1754271957 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7d8150d20 0x7fc7d8152fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:38.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.883+0000 7fc7dc805640 1 -- 172.21.15.116:0/1754271957 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7d81ecc10 con 0x7fc7d80c3240 2024-04-25T00:04:38.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.883+0000 7fc7dd006640 1 --2- 172.21.15.116:0/1754271957 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc7d8071050 0x7fc7d8143570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:04:38.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.883+0000 7fc7dc805640 1 --2- 172.21.15.116:0/1754271957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7d80c3240 0x7fc7d8152a70 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fc7c800c450 tx=0x7fc7c800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:38.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.883+0000 7fc7cdffb640 1 -- 172.21.15.116:0/1754271957 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c8012830 con 0x7fc7d80c3240 2024-04-25T00:04:38.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.883+0000 7fc7cdffb640 1 -- 172.21.15.116:0/1754271957 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7c8017070 con 0x7fc7d80c3240 2024-04-25T00:04:38.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.883+0000 7fc7cdffb640 1 -- 172.21.15.116:0/1754271957 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c8012c30 con 0x7fc7d80c3240 2024-04-25T00:04:38.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.883+0000 7fc7df26a640 1 -- 172.21.15.116:0/1754271957 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7d8151fd0 con 0x7fc7d80c3240 2024-04-25T00:04:38.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.883+0000 7fc7df26a640 1 -- 172.21.15.116:0/1754271957 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7d8151cf0 con 0x7fc7d80c3240 2024-04-25T00:04:38.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.883+0000 7fc7df26a640 1 -- 172.21.15.116:0/1754271957 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7d8068a60 con 0x7fc7d80c3240 2024-04-25T00:04:38.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.883+0000 7fc7cdffb640 1 -- 172.21.15.116:0/1754271957 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc7c801c460 con 0x7fc7d80c3240 2024-04-25T00:04:38.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.883+0000 7fc7cdffb640 1 --2- 172.21.15.116:0/1754271957 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc7bc0678f0 0x7fc7bc069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:38.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.883+0000 7fc7cdffb640 1 -- 172.21.15.116:0/1754271957 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7010+0+0 (secure 0 0 0) 0x7fc7c8085e40 con 0x7fc7d80c3240 2024-04-25T00:04:38.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.887+0000 7fc7dd006640 1 --2- 172.21.15.116:0/1754271957 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc7bc0678f0 0x7fc7bc069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:38.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.887+0000 7fc7dd006640 1 --2- 172.21.15.116:0/1754271957 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc7bc0678f0 0x7fc7bc069db0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fc7d40ad000 tx=0x7fc7d4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:38.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:38.891+0000 7fc7cdffb640 1 -- 172.21.15.116:0/1754271957 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fc7c80564f0 con 0x7fc7d80c3240 2024-04-25T00:04:39.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:39.103+0000 7fc7df26a640 1 -- 172.21.15.116:0/1754271957 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc7d8150130 con 0x7fc7d80c3240 2024-04-25T00:04:39.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:39.103+0000 7fc7cdffb640 1 -- 172.21.15.116:0/1754271957 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+5674 (secure 0 0 0) 0x7fc7c8055df0 con 0x7fc7d80c3240 2024-04-25T00:04:39.105 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:04:39.105 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:04:38.435723+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4879},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4879":{"gid":4879,"name":"b","rank":0,"incarnation":32,"state":"up:active","state_seq":15,"addr":"172.21.15.174:6837/1889314958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1889314958},{"type":"v1","addr":"172.21.15.174:6837","nonce":1889314958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4879,"qdb_cluster":[4879]},"id":1},{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:03:34.579825+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4735},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4735":{"gid":4735,"name":"a","rank":0,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.116:6835/2004876235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":2004876235},{"type":"v1","addr":"172.21.15.116:6835","nonce":2004876235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4735,"qdb_cluster":[4735]},"id":2}]} 2024-04-25T00:04:39.105 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 35 2024-04-25T00:04:39.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:39.107+0000 7fc7df26a640 1 -- 172.21.15.116:0/1754271957 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc7bc0678f0 msgr2=0x7fc7bc069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:39.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:39.107+0000 7fc7df26a640 1 --2- 172.21.15.116:0/1754271957 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc7bc0678f0 0x7fc7bc069db0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fc7d40ad000 tx=0x7fc7d4008040 comp rx=0 tx=0).stop 2024-04-25T00:04:39.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:39.107+0000 7fc7df26a640 1 -- 172.21.15.116:0/1754271957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7d80c3240 msgr2=0x7fc7d8152a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:39.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:39.107+0000 7fc7df26a640 1 --2- 172.21.15.116:0/1754271957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7d80c3240 0x7fc7d8152a70 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fc7c800c450 tx=0x7fc7c800c910 comp rx=0 tx=0).stop 2024-04-25T00:04:39.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:39.107+0000 7fc7df26a640 1 -- 172.21.15.116:0/1754271957 shutdown_connections 2024-04-25T00:04:39.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:39.107+0000 7fc7df26a640 1 --2- 172.21.15.116:0/1754271957 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc7bc0678f0 0x7fc7bc069db0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:39.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:39.107+0000 7fc7df26a640 1 --2- 172.21.15.116:0/1754271957 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7d8150d20 0x7fc7d8152fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:39.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:39.107+0000 7fc7df26a640 1 --2- 172.21.15.116:0/1754271957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7d80c3240 0x7fc7d8152a70 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:39.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:39.107+0000 7fc7df26a640 1 --2- 172.21.15.116:0/1754271957 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc7d8071050 0x7fc7d8143570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:39.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:39.107+0000 7fc7df26a640 1 -- 172.21.15.116:0/1754271957 >> 172.21.15.116:0/1754271957 conn(0x7fc7d80c9160 msgr2=0x7fc7d80c59e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:04:39.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:39.107+0000 7fc7df26a640 1 -- 172.21.15.116:0/1754271957 shutdown_connections 2024-04-25T00:04:39.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:39.107+0000 7fc7df26a640 1 -- 172.21.15.116:0/1754271957 wait complete. 2024-04-25T00:04:39.128 INFO:tasks.mds_thrash.fs.[a]:mds.b has gained rank=0, replacing gid=4828 2024-04-25T00:04:39.129 INFO:tasks.mds_thrash.fs.[a]:waiting for 3 secs before reviving mds.c 2024-04-25T00:04:39.664 INFO:tasks.ceph.mds.b.smithi174.stderr:2024-04-25T00:04:39.659+0000 7ffac2749640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T00:04:40.850 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:04:40.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.987+0000 7fe37ed09640 1 Processor -- start 2024-04-25T00:04:40.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.987+0000 7fe37ed09640 1 -- start start 2024-04-25T00:04:40.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.991+0000 7fe37ed09640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fe378150d20 0x7fe378153620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:40.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.991+0000 7fe37ed09640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3780c36a0 con 0x7fe3780c3240 2024-04-25T00:04:40.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.991+0000 7fe37ed09640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe37806f8c0 con 0x7fe378071050 2024-04-25T00:04:40.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.991+0000 7fe37ed09640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fe37806fba0 con 0x7fe378150d20 2024-04-25T00:04:40.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.991+0000 7fe377fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fe3780c3240 0x7fe3780c3e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:49454/0 (socket says 172.21.15.116:49454) 2024-04-25T00:04:40.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.991+0000 7fe377fff640 1 -- 172.21.15.116:0/2943835196 learned_addr learned my addr 172.21.15.116:0/2943835196 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:04:40.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.991+0000 7fe37caa5640 1 --2- 172.21.15.116:0/2943835196 >> v2:172.21.15.174:3300/0 conn(0x7fe378150d20 0x7fe378153620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:40.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.991+0000 7fe3777fe640 1 -- 172.21.15.116:0/2943835196 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3480516580 0 0) 0x7fe3780c36a0 con 0x7fe3780c3240 2024-04-25T00:04:40.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.991+0000 7fe3777fe640 1 -- 172.21.15.116:0/2943835196 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe368003530 con 0x7fe3780c3240 2024-04-25T00:04:40.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.991+0000 7fe3777fe640 1 -- 172.21.15.116:0/2943835196 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 855827849 0 0) 0x7fe368003530 con 0x7fe3780c3240 2024-04-25T00:04:40.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.991+0000 7fe3777fe640 1 -- 172.21.15.116:0/2943835196 >> v2:172.21.15.174:3300/0 conn(0x7fe378150d20 msgr2=0x7fe378153620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:04:40.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.991+0000 7fe3777fe640 1 --2- 172.21.15.116:0/2943835196 >> v2:172.21.15.174:3300/0 conn(0x7fe378150d20 0x7fe378153620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:40.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.991+0000 7fe3777fe640 1 -- 172.21.15.116:0/2943835196 >> v1:172.21.15.174:6789/0 conn(0x7fe378071050 legacy=0x7fe37815df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:40.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.991+0000 7fe3777fe640 1 -- 172.21.15.116:0/2943835196 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3780c5420 con 0x7fe3780c3240 2024-04-25T00:04:40.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.991+0000 7fe3777fe640 1 -- 172.21.15.116:0/2943835196 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7fe364001680 con 0x7fe3780c3240 2024-04-25T00:04:40.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.991+0000 7fe3777fe640 1 -- 172.21.15.116:0/2943835196 >> v1:172.21.15.116:6789/0 conn(0x7fe3780c3240 legacy=0x7fe3780c3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:40.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.991+0000 7fe3777fe640 1 --2- 172.21.15.116:0/2943835196 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe3680039c0 0x7fe368003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:40.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.991+0000 7fe3777fe640 1 --2- 172.21.15.116:0/2943835196 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe368004810 0x7fe368004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:40.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.991+0000 7fe3777fe640 1 --2- 172.21.15.116:0/2943835196 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3680051d0 0x7fe36800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:40.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.991+0000 7fe3777fe640 1 -- 172.21.15.116:0/2943835196 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe37806fba0 con 0x7fe3680051d0 2024-04-25T00:04:40.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.991+0000 7fe3777fe640 1 -- 172.21.15.116:0/2943835196 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe36800c070 con 0x7fe3680039c0 2024-04-25T00:04:40.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.991+0000 7fe3777fe640 1 -- 172.21.15.116:0/2943835196 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe36800c270 con 0x7fe368004810 2024-04-25T00:04:40.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.991+0000 7fe37d2a6640 1 --2- 172.21.15.116:0/2943835196 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3680051d0 0x7fe36800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:40.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.991+0000 7fe37caa5640 1 --2- 172.21.15.116:0/2943835196 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe3680039c0 0x7fe368003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:40.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.991+0000 7fe377fff640 1 --2- 172.21.15.116:0/2943835196 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe368004810 0x7fe368004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:40.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.991+0000 7fe37d2a6640 1 -- 172.21.15.116:0/2943835196 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe368004810 msgr2=0x7fe368004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:04:40.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.991+0000 7fe37d2a6640 1 --2- 172.21.15.116:0/2943835196 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe368004810 0x7fe368004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:40.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.991+0000 7fe37d2a6640 1 -- 172.21.15.116:0/2943835196 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe3680039c0 msgr2=0x7fe368003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:04:40.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.991+0000 7fe37d2a6640 1 --2- 172.21.15.116:0/2943835196 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe3680039c0 0x7fe368003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:40.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.991+0000 7fe37d2a6640 1 -- 172.21.15.116:0/2943835196 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe368003530 con 0x7fe3680051d0 2024-04-25T00:04:40.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.991+0000 7fe37d2a6640 1 -- 172.21.15.116:0/2943835196 reap_dead start 2024-04-25T00:04:40.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.991+0000 7fe37d2a6640 1 --2- 172.21.15.116:0/2943835196 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3680051d0 0x7fe36800b9d0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fe370001e70 tx=0x7fe37001c780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f8dd88a34d1e6be server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:40.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.991+0000 7fe3777fe640 1 -- 172.21.15.116:0/2943835196 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe370001a90 con 0x7fe3680051d0 2024-04-25T00:04:40.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.991+0000 7fe3777fe640 1 -- 172.21.15.116:0/2943835196 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe370003bc0 con 0x7fe3680051d0 2024-04-25T00:04:40.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.995+0000 7fe3777fe640 1 -- 172.21.15.116:0/2943835196 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe370005020 con 0x7fe3680051d0 2024-04-25T00:04:40.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.995+0000 7fe37ed09640 1 -- 172.21.15.116:0/2943835196 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3680051d0 msgr2=0x7fe36800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:40.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.995+0000 7fe37ed09640 1 --2- 172.21.15.116:0/2943835196 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3680051d0 0x7fe36800b9d0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fe370001e70 tx=0x7fe37001c780 comp rx=0 tx=0).stop 2024-04-25T00:04:40.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.995+0000 7fe37ed09640 1 -- 172.21.15.116:0/2943835196 shutdown_connections 2024-04-25T00:04:40.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.995+0000 7fe37ed09640 1 --2- 172.21.15.116:0/2943835196 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3680051d0 0x7fe36800b9d0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:40.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.995+0000 7fe37ed09640 1 -- 172.21.15.116:0/2943835196 >> 172.21.15.116:0/2943835196 conn(0x7fe3780c9160 msgr2=0x7fe3780c9570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:04:40.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.995+0000 7fe37ed09640 1 -- 172.21.15.116:0/2943835196 shutdown_connections 2024-04-25T00:04:40.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.995+0000 7fe37ed09640 1 -- 172.21.15.116:0/2943835196 wait complete. 2024-04-25T00:04:40.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.995+0000 7fe37ed09640 1 Processor -- start 2024-04-25T00:04:40.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.995+0000 7fe37ed09640 1 -- start start 2024-04-25T00:04:40.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.995+0000 7fe37ed09640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3680051d0 0x7fe3780c0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:40.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.995+0000 7fe37ed09640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe378148170 0x7fe3780c2660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:40.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.995+0000 7fe37ed09640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe3780c2ba0 0x7fe3781e8d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:40.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.995+0000 7fe37ed09640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe3780c4d50 con 0x7fe3680051d0 2024-04-25T00:04:40.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.995+0000 7fe37ed09640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe3780c4bd0 con 0x7fe3780c2ba0 2024-04-25T00:04:40.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.995+0000 7fe37ed09640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe378151100 con 0x7fe378148170 2024-04-25T00:04:40.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.995+0000 7fe37caa5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3680051d0 0x7fe3780c0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:40.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.995+0000 7fe37caa5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3680051d0 0x7fe3780c0c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53790/0 (socket says 172.21.15.116:53790) 2024-04-25T00:04:40.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.995+0000 7fe37caa5640 1 -- 172.21.15.116:0/539980788 learned_addr learned my addr 172.21.15.116:0/539980788 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:04:40.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.995+0000 7fe37d2a6640 1 --2- 172.21.15.116:0/539980788 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe3780c2ba0 0x7fe3781e8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:40.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.995+0000 7fe377fff640 1 --2- 172.21.15.116:0/539980788 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe378148170 0x7fe3780c2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:40.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.995+0000 7fe37caa5640 1 -- 172.21.15.116:0/539980788 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe378148170 msgr2=0x7fe3780c2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:40.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.995+0000 7fe37caa5640 1 --2- 172.21.15.116:0/539980788 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe378148170 0x7fe3780c2660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:40.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.995+0000 7fe37caa5640 1 -- 172.21.15.116:0/539980788 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe3780c2ba0 msgr2=0x7fe3781e8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:41.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.995+0000 7fe37caa5640 1 --2- 172.21.15.116:0/539980788 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe3780c2ba0 0x7fe3781e8d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:41.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.995+0000 7fe37caa5640 1 -- 172.21.15.116:0/539980788 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3781489d0 con 0x7fe3680051d0 2024-04-25T00:04:41.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.995+0000 7fe377fff640 1 --2- 172.21.15.116:0/539980788 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe378148170 0x7fe3780c2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:04:41.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.995+0000 7fe37d2a6640 1 --2- 172.21.15.116:0/539980788 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe3780c2ba0 0x7fe3781e8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:04:41.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.999+0000 7fe37caa5640 1 --2- 172.21.15.116:0/539980788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3680051d0 0x7fe3780c0c50 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fe36c011560 tx=0x7fe36c011a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:41.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.999+0000 7fe3757fa640 1 -- 172.21.15.116:0/539980788 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe36c015930 con 0x7fe3680051d0 2024-04-25T00:04:41.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.999+0000 7fe3757fa640 1 -- 172.21.15.116:0/539980788 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe36c015ad0 con 0x7fe3680051d0 2024-04-25T00:04:41.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.999+0000 7fe3757fa640 1 -- 172.21.15.116:0/539980788 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe36c01d400 con 0x7fe3680051d0 2024-04-25T00:04:41.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.999+0000 7fe37ed09640 1 -- 172.21.15.116:0/539980788 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe378153820 con 0x7fe3680051d0 2024-04-25T00:04:41.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.999+0000 7fe37ed09640 1 -- 172.21.15.116:0/539980788 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3780c1f60 con 0x7fe3680051d0 2024-04-25T00:04:41.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.999+0000 7fe37ed09640 1 -- 172.21.15.116:0/539980788 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe378068a60 con 0x7fe3680051d0 2024-04-25T00:04:41.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.999+0000 7fe3757fa640 1 -- 172.21.15.116:0/539980788 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe36c01d5a0 con 0x7fe3680051d0 2024-04-25T00:04:41.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.999+0000 7fe3757fa640 1 --2- 172.21.15.116:0/539980788 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe35c0678c0 0x7fe35c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:41.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.999+0000 7fe3757fa640 1 -- 172.21.15.116:0/539980788 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7010+0+0 (secure 0 0 0) 0x7fe36c0869d0 con 0x7fe3680051d0 2024-04-25T00:04:41.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.999+0000 7fe377fff640 1 --2- 172.21.15.116:0/539980788 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe35c0678c0 0x7fe35c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:41.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:40.999+0000 7fe377fff640 1 --2- 172.21.15.116:0/539980788 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe35c0678c0 0x7fe35c069d80 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe3781e9e10 tx=0x7fe364000ee0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:41.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:41.007+0000 7fe3757fa640 1 -- 172.21.15.116:0/539980788 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fe36c019030 con 0x7fe3680051d0 2024-04-25T00:04:41.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:41.223+0000 7fe37ed09640 1 -- 172.21.15.116:0/539980788 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe378071430 con 0x7fe3680051d0 2024-04-25T00:04:41.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:41.223+0000 7fe3757fa640 1 -- 172.21.15.116:0/539980788 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+5674 (secure 0 0 0) 0x7fe36c057080 con 0x7fe3680051d0 2024-04-25T00:04:41.223 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:04:41.223 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:04:38.435723+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4879},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4879":{"gid":4879,"name":"b","rank":0,"incarnation":32,"state":"up:active","state_seq":15,"addr":"172.21.15.174:6837/1889314958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1889314958},{"type":"v1","addr":"172.21.15.174:6837","nonce":1889314958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4879,"qdb_cluster":[4879]},"id":1},{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:03:34.579825+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4735},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4735":{"gid":4735,"name":"a","rank":0,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.116:6835/2004876235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":2004876235},{"type":"v1","addr":"172.21.15.116:6835","nonce":2004876235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4735,"qdb_cluster":[4735]},"id":2}]} 2024-04-25T00:04:41.224 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 35 2024-04-25T00:04:41.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:41.223+0000 7fe37ed09640 1 -- 172.21.15.116:0/539980788 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe35c0678c0 msgr2=0x7fe35c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:41.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:41.223+0000 7fe37ed09640 1 --2- 172.21.15.116:0/539980788 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe35c0678c0 0x7fe35c069d80 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe3781e9e10 tx=0x7fe364000ee0 comp rx=0 tx=0).stop 2024-04-25T00:04:41.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:41.223+0000 7fe37ed09640 1 -- 172.21.15.116:0/539980788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3680051d0 msgr2=0x7fe3780c0c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:41.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:41.223+0000 7fe37ed09640 1 --2- 172.21.15.116:0/539980788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3680051d0 0x7fe3780c0c50 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fe36c011560 tx=0x7fe36c011a20 comp rx=0 tx=0).stop 2024-04-25T00:04:41.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:41.223+0000 7fe37ed09640 1 -- 172.21.15.116:0/539980788 shutdown_connections 2024-04-25T00:04:41.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:41.223+0000 7fe37ed09640 1 --2- 172.21.15.116:0/539980788 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe35c0678c0 0x7fe35c069d80 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:41.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:41.223+0000 7fe37ed09640 1 --2- 172.21.15.116:0/539980788 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe3780c2ba0 0x7fe3781e8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:41.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:41.223+0000 7fe37ed09640 1 --2- 172.21.15.116:0/539980788 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe378148170 0x7fe3780c2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:41.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:41.223+0000 7fe37ed09640 1 --2- 172.21.15.116:0/539980788 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3680051d0 0x7fe3780c0c50 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:41.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:41.223+0000 7fe37ed09640 1 -- 172.21.15.116:0/539980788 >> 172.21.15.116:0/539980788 conn(0x7fe3780c9160 msgr2=0x7fe37814f070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:04:41.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:41.223+0000 7fe37ed09640 1 -- 172.21.15.116:0/539980788 shutdown_connections 2024-04-25T00:04:41.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:41.223+0000 7fe37ed09640 1 -- 172.21.15.116:0/539980788 wait complete. 2024-04-25T00:04:41.247 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:04:42.129 INFO:tasks.mds_thrash.fs.[a]:reviving mds.c 2024-04-25T00:04:42.129 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-25T00:04:42.129 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-25T00:04:42.132 INFO:tasks.ceph.mds.c:Started 2024-04-25T00:04:42.132 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:04:42.206 INFO:tasks.ceph.mds.c.smithi116.stdout:starting mds.c at 2024-04-25T00:04:42.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.275+0000 7f13ee058640 1 Processor -- start 2024-04-25T00:04:42.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.275+0000 7f13ee058640 1 -- start start 2024-04-25T00:04:42.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.275+0000 7f13ee058640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f13e8150d20 0x7f13e8153620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:42.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.275+0000 7f13ee058640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13e80c36a0 con 0x7f13e8071050 2024-04-25T00:04:42.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.275+0000 7f13ee058640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13e806f8c0 con 0x7f13e80c3240 2024-04-25T00:04:42.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.275+0000 7f13ee058640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f13e806fba0 con 0x7f13e8150d20 2024-04-25T00:04:42.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.275+0000 7f13e77fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f13e8150d20 0x7f13e8153620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:42.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.275+0000 7f13e77fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f13e8150d20 0x7f13e8153620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53810/0 (socket says 172.21.15.116:53810) 2024-04-25T00:04:42.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.275+0000 7f13e7fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f13e8071050 0x7f13e815df10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:49458/0 (socket says 172.21.15.116:49458) 2024-04-25T00:04:42.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.275+0000 7f13e77fe640 1 -- 172.21.15.116:0/174362638 learned_addr learned my addr 172.21.15.116:0/174362638 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:04:42.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.275+0000 7f13e67fc640 1 -- 172.21.15.116:0/174362638 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2404605649 0 0) 0x7f13e80c36a0 con 0x7f13e8071050 2024-04-25T00:04:42.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.275+0000 7f13e67fc640 1 -- 172.21.15.116:0/174362638 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13d4003530 con 0x7f13e8071050 2024-04-25T00:04:42.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.275+0000 7f13e77fe640 1 -- 172.21.15.116:0/174362638 >> v1:172.21.15.174:6789/0 conn(0x7f13e80c3240 legacy=0x7f13e80c3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:42.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.275+0000 7f13e77fe640 1 -- 172.21.15.116:0/174362638 >> v1:172.21.15.116:6789/0 conn(0x7f13e8071050 legacy=0x7f13e815df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:42.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.275+0000 7f13e77fe640 1 -- 172.21.15.116:0/174362638 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13e80c5420 con 0x7f13e8150d20 2024-04-25T00:04:42.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.279+0000 7f13e77fe640 1 --2- 172.21.15.116:0/174362638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13e8150d20 0x7f13e8153620 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f13d8009870 tx=0x7f13d801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c649a80c824c54ff server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:42.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.279+0000 7f13e67fc640 1 -- 172.21.15.116:0/174362638 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d80ab430 con 0x7f13e8150d20 2024-04-25T00:04:42.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.279+0000 7f13e67fc640 1 -- 172.21.15.116:0/174362638 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13d80b0070 con 0x7f13e8150d20 2024-04-25T00:04:42.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.279+0000 7f13e67fc640 1 -- 172.21.15.116:0/174362638 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d80ab7f0 con 0x7f13e8150d20 2024-04-25T00:04:42.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.279+0000 7f13ee058640 1 -- 172.21.15.116:0/174362638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13e8150d20 msgr2=0x7f13e8153620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:42.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.279+0000 7f13ee058640 1 --2- 172.21.15.116:0/174362638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13e8150d20 0x7f13e8153620 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f13d8009870 tx=0x7f13d801c520 comp rx=0 tx=0).stop 2024-04-25T00:04:42.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.279+0000 7f13ee058640 1 -- 172.21.15.116:0/174362638 shutdown_connections 2024-04-25T00:04:42.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.279+0000 7f13ee058640 1 --2- 172.21.15.116:0/174362638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13e8150d20 0x7f13e8153620 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:42.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.279+0000 7f13ee058640 1 -- 172.21.15.116:0/174362638 >> 172.21.15.116:0/174362638 conn(0x7f13e80c9160 msgr2=0x7f13e80c9570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:04:42.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.279+0000 7f13ee058640 1 -- 172.21.15.116:0/174362638 shutdown_connections 2024-04-25T00:04:42.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.279+0000 7f13ee058640 1 -- 172.21.15.116:0/174362638 wait complete. 2024-04-25T00:04:42.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.283+0000 7f13ee058640 1 Processor -- start 2024-04-25T00:04:42.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.283+0000 7f13ee058640 1 -- start start 2024-04-25T00:04:42.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.283+0000 7f13ee058640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13e8071050 0x7f13e806d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:42.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.283+0000 7f13ee058640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f13e80c3240 0x7f13e81445f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:42.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.283+0000 7f13ee058640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f13e8150d20 0x7f13e80ae590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:42.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.283+0000 7f13ee058640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f13e80c4d50 con 0x7f13e8071050 2024-04-25T00:04:42.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.283+0000 7f13ee058640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f13e80c4bd0 con 0x7f13e80c3240 2024-04-25T00:04:42.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.283+0000 7f13ee058640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f13e806f4c0 con 0x7f13e8150d20 2024-04-25T00:04:42.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.283+0000 7f13e77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13e8071050 0x7f13e806d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:42.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.283+0000 7f13e77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13e8071050 0x7f13e806d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53822/0 (socket says 172.21.15.116:53822) 2024-04-25T00:04:42.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.283+0000 7f13e77fe640 1 -- 172.21.15.116:0/3213530607 learned_addr learned my addr 172.21.15.116:0/3213530607 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:04:42.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.283+0000 7f13e7fff640 1 --2- 172.21.15.116:0/3213530607 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f13e8150d20 0x7f13e80ae590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:42.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.283+0000 7f13e77fe640 1 -- 172.21.15.116:0/3213530607 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f13e8150d20 msgr2=0x7f13e80ae590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:42.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.283+0000 7f13e6ffd640 1 --2- 172.21.15.116:0/3213530607 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f13e80c3240 0x7f13e81445f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:42.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.283+0000 7f13e77fe640 1 --2- 172.21.15.116:0/3213530607 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f13e8150d20 0x7f13e80ae590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:42.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.283+0000 7f13e77fe640 1 -- 172.21.15.116:0/3213530607 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f13e80c3240 msgr2=0x7f13e81445f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:42.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.283+0000 7f13e77fe640 1 --2- 172.21.15.116:0/3213530607 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f13e80c3240 0x7f13e81445f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:42.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.283+0000 7f13e77fe640 1 -- 172.21.15.116:0/3213530607 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13e80ac720 con 0x7f13e8071050 2024-04-25T00:04:42.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.283+0000 7f13e6ffd640 1 --2- 172.21.15.116:0/3213530607 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f13e80c3240 0x7f13e81445f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:04:42.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.283+0000 7f13e77fe640 1 --2- 172.21.15.116:0/3213530607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13e8071050 0x7f13e806d5d0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f13d8019130 tx=0x7f13d801ad60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:42.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.283+0000 7f13dffff640 1 -- 172.21.15.116:0/3213530607 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d80be5f0 con 0x7f13e8071050 2024-04-25T00:04:42.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.283+0000 7f13dffff640 1 -- 172.21.15.116:0/3213530607 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13d80be790 con 0x7f13e8071050 2024-04-25T00:04:42.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.283+0000 7f13ee058640 1 -- 172.21.15.116:0/3213530607 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13e80ac9c0 con 0x7f13e8071050 2024-04-25T00:04:42.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.283+0000 7f13dffff640 1 -- 172.21.15.116:0/3213530607 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d80beb80 con 0x7f13e8071050 2024-04-25T00:04:42.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.283+0000 7f13ee058640 1 -- 172.21.15.116:0/3213530607 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13e806cdd0 con 0x7f13e8071050 2024-04-25T00:04:42.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.283+0000 7f13ee058640 1 -- 172.21.15.116:0/3213530607 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13e806d0e0 con 0x7f13e8071050 2024-04-25T00:04:42.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.283+0000 7f13dffff640 1 -- 172.21.15.116:0/3213530607 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f13d80b43f0 con 0x7f13e8071050 2024-04-25T00:04:42.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.287+0000 7f13dffff640 1 --2- 172.21.15.116:0/3213530607 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f13c80678c0 0x7f13c8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:42.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.287+0000 7f13dffff640 1 -- 172.21.15.116:0/3213530607 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7010+0+0 (secure 0 0 0) 0x7f13d80a8070 con 0x7f13e8071050 2024-04-25T00:04:42.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.287+0000 7f13e6ffd640 1 --2- 172.21.15.116:0/3213530607 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f13c80678c0 0x7f13c8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:42.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.287+0000 7f13e6ffd640 1 --2- 172.21.15.116:0/3213530607 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f13c80678c0 0x7f13c8069d80 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f13d0000f70 tx=0x7f13d0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:42.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.291+0000 7f13dffff640 1 -- 172.21.15.116:0/3213530607 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f13d8016110 con 0x7f13e8071050 2024-04-25T00:04:42.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.531+0000 7f13ee058640 1 -- 172.21.15.116:0/3213530607 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f13e806d0e0 con 0x7f13e8071050 2024-04-25T00:04:42.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.531+0000 7f13dffff640 1 -- 172.21.15.116:0/3213530607 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+6502 (secure 0 0 0) 0x7f13d80f0ef0 con 0x7f13e8071050 2024-04-25T00:04:42.534 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:04:42.534 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5571,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/3841609112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3841609112},{"type":"v1","addr":"172.21.15.116:6833","nonce":3841609112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:04:38.435723+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4879},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4879":{"gid":4879,"name":"b","rank":0,"incarnation":32,"state":"up:active","state_seq":15,"addr":"172.21.15.174:6837/1889314958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1889314958},{"type":"v1","addr":"172.21.15.174:6837","nonce":1889314958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4879,"qdb_cluster":[4879]},"id":1},{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:03:34.579825+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4735},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4735":{"gid":4735,"name":"a","rank":0,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.116:6835/2004876235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":2004876235},{"type":"v1","addr":"172.21.15.116:6835","nonce":2004876235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4735,"qdb_cluster":[4735]},"id":2}]} 2024-04-25T00:04:42.534 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 36 2024-04-25T00:04:42.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.535+0000 7f13ee058640 1 -- 172.21.15.116:0/3213530607 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f13c80678c0 msgr2=0x7f13c8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:42.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.535+0000 7f13ee058640 1 --2- 172.21.15.116:0/3213530607 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f13c80678c0 0x7f13c8069d80 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f13d0000f70 tx=0x7f13d0009040 comp rx=0 tx=0).stop 2024-04-25T00:04:42.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.535+0000 7f13ee058640 1 -- 172.21.15.116:0/3213530607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13e8071050 msgr2=0x7f13e806d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:42.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.535+0000 7f13ee058640 1 --2- 172.21.15.116:0/3213530607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13e8071050 0x7f13e806d5d0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f13d8019130 tx=0x7f13d801ad60 comp rx=0 tx=0).stop 2024-04-25T00:04:42.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.535+0000 7f13ee058640 1 -- 172.21.15.116:0/3213530607 shutdown_connections 2024-04-25T00:04:42.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.535+0000 7f13ee058640 1 --2- 172.21.15.116:0/3213530607 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f13c80678c0 0x7f13c8069d80 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:42.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.535+0000 7f13ee058640 1 --2- 172.21.15.116:0/3213530607 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f13e8150d20 0x7f13e80ae590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:42.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.535+0000 7f13ee058640 1 --2- 172.21.15.116:0/3213530607 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f13e80c3240 0x7f13e81445f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:42.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.535+0000 7f13ee058640 1 --2- 172.21.15.116:0/3213530607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13e8071050 0x7f13e806d5d0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:42.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.535+0000 7f13ee058640 1 -- 172.21.15.116:0/3213530607 >> 172.21.15.116:0/3213530607 conn(0x7f13e80c9160 msgr2=0x7f13e815ee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:04:42.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.535+0000 7f13ee058640 1 -- 172.21.15.116:0/3213530607 shutdown_connections 2024-04-25T00:04:42.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:42.535+0000 7f13ee058640 1 -- 172.21.15.116:0/3213530607 wait complete. 2024-04-25T00:04:42.551 INFO:tasks.mds_thrash.fs.[a]:mds.c reported in up:standby state 2024-04-25T00:04:42.551 INFO:tasks.mds_thrash.fs.[a]:waiting for 91 secs before thrashing 2024-04-25T00:04:43.247 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:04:43.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.395+0000 7fd98163e640 1 Processor -- start 2024-04-25T00:04:43.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.395+0000 7fd98163e640 1 -- start start 2024-04-25T00:04:43.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.395+0000 7fd98163e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd97c152d60 0x7fd97c0ab270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:43.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.395+0000 7fd98163e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd97c0ab7b0 0x7fd97c0c8450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:43.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.395+0000 7fd98163e640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd97c155c50 con 0x7fd97c153140 2024-04-25T00:04:43.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.395+0000 7fd98163e640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fd97c155f30 con 0x7fd97c0ab7b0 2024-04-25T00:04:43.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.395+0000 7fd98163e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fd97c156210 con 0x7fd97c152d60 2024-04-25T00:04:43.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.395+0000 7fd97a7fc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd97c0ab7b0 0x7fd97c0c8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:43.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.395+0000 7fd97a7fc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd97c0ab7b0 0x7fd97c0c8450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53828/0 (socket says 172.21.15.116:53828) 2024-04-25T00:04:43.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.395+0000 7fd97a7fc640 1 -- 172.21.15.116:0/3418947667 learned_addr learned my addr 172.21.15.116:0/3418947667 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:04:43.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.395+0000 7fd97b7fe640 1 --1- 172.21.15.116:0/3418947667 >> v1:172.21.15.116:6789/0 conn(0x7fd97c153140 0x7fd97c0c3d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:49472/0 (socket says 172.21.15.116:49472) 2024-04-25T00:04:43.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.395+0000 7fd97affd640 1 --2- 172.21.15.116:0/3418947667 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd97c152d60 0x7fd97c0ab270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:43.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.395+0000 7fd97a7fc640 1 -- 172.21.15.116:0/3418947667 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd97c152d60 msgr2=0x7fd97c0ab270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:04:43.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.395+0000 7fd97a7fc640 1 --2- 172.21.15.116:0/3418947667 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd97c152d60 0x7fd97c0ab270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:43.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.395+0000 7fd97a7fc640 1 -- 172.21.15.116:0/3418947667 >> v1:172.21.15.116:6789/0 conn(0x7fd97c153140 legacy=0x7fd97c0c3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:43.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.395+0000 7fd979ffb640 1 -- 172.21.15.116:0/3418947667 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 580001584 0 0) 0x7fd97c155c50 con 0x7fd97c153140 2024-04-25T00:04:43.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.395+0000 7fd97a7fc640 1 -- 172.21.15.116:0/3418947667 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd97c1564f0 con 0x7fd97c0ab7b0 2024-04-25T00:04:43.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.395+0000 7fd97affd640 1 --2- 172.21.15.116:0/3418947667 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd97c152d60 0x7fd97c0ab270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:04:43.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.399+0000 7fd97a7fc640 1 --2- 172.21.15.116:0/3418947667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd97c0ab7b0 0x7fd97c0c8450 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fd970009870 tx=0x7fd97001c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9c18354b82847e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:43.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.399+0000 7fd979ffb640 1 -- 172.21.15.116:0/3418947667 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9700b2070 con 0x7fd97c0ab7b0 2024-04-25T00:04:43.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.399+0000 7fd979ffb640 1 -- 172.21.15.116:0/3418947667 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9700a9070 con 0x7fd97c0ab7b0 2024-04-25T00:04:43.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.399+0000 7fd979ffb640 1 -- 172.21.15.116:0/3418947667 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9700ad880 con 0x7fd97c0ab7b0 2024-04-25T00:04:43.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.399+0000 7fd98163e640 1 -- 172.21.15.116:0/3418947667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd97c0ab7b0 msgr2=0x7fd97c0c8450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:43.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.399+0000 7fd98163e640 1 --2- 172.21.15.116:0/3418947667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd97c0ab7b0 0x7fd97c0c8450 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fd970009870 tx=0x7fd97001c560 comp rx=0 tx=0).stop 2024-04-25T00:04:43.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.399+0000 7fd98163e640 1 -- 172.21.15.116:0/3418947667 shutdown_connections 2024-04-25T00:04:43.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.399+0000 7fd98163e640 1 --2- 172.21.15.116:0/3418947667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd97c0ab7b0 0x7fd97c0c8450 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:43.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.399+0000 7fd98163e640 1 --2- 172.21.15.116:0/3418947667 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd97c152d60 0x7fd97c0ab270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:43.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.399+0000 7fd98163e640 1 -- 172.21.15.116:0/3418947667 >> 172.21.15.116:0/3418947667 conn(0x7fd97c06f0c0 msgr2=0x7fd97c06f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:04:43.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.399+0000 7fd98163e640 1 -- 172.21.15.116:0/3418947667 shutdown_connections 2024-04-25T00:04:43.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.399+0000 7fd98163e640 1 -- 172.21.15.116:0/3418947667 wait complete. 2024-04-25T00:04:43.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.403+0000 7fd98163e640 1 Processor -- start 2024-04-25T00:04:43.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.403+0000 7fd98163e640 1 -- start start 2024-04-25T00:04:43.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.403+0000 7fd98163e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd97c0ab7b0 0x7fd97c159f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:43.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.403+0000 7fd98163e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd97c152d60 0x7fd97c0ad280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:43.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.403+0000 7fd98163e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd97c153140 0x7fd97c1f21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:43.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.403+0000 7fd98163e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd97c162bb0 con 0x7fd97c153140 2024-04-25T00:04:43.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.403+0000 7fd98163e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd97c155a10 con 0x7fd97c0ab7b0 2024-04-25T00:04:43.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.403+0000 7fd98163e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fd97c162d30 con 0x7fd97c152d60 2024-04-25T00:04:43.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.403+0000 7fd97b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd97c153140 0x7fd97c1f21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:43.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.403+0000 7fd97b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd97c153140 0x7fd97c1f21e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53838/0 (socket says 172.21.15.116:53838) 2024-04-25T00:04:43.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.403+0000 7fd97b7fe640 1 -- 172.21.15.116:0/1007322051 learned_addr learned my addr 172.21.15.116:0/1007322051 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:04:43.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.403+0000 7fd97a7fc640 1 --2- 172.21.15.116:0/1007322051 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd97c152d60 0x7fd97c0ad280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:43.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.403+0000 7fd97affd640 1 --2- 172.21.15.116:0/1007322051 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd97c0ab7b0 0x7fd97c159f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:43.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.403+0000 7fd97b7fe640 1 -- 172.21.15.116:0/1007322051 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd97c152d60 msgr2=0x7fd97c0ad280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:43.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.403+0000 7fd97b7fe640 1 --2- 172.21.15.116:0/1007322051 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd97c152d60 0x7fd97c0ad280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:43.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.403+0000 7fd97b7fe640 1 -- 172.21.15.116:0/1007322051 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd97c0ab7b0 msgr2=0x7fd97c159f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:43.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.403+0000 7fd97b7fe640 1 --2- 172.21.15.116:0/1007322051 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd97c0ab7b0 0x7fd97c159f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:43.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.403+0000 7fd97b7fe640 1 -- 172.21.15.116:0/1007322051 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd97c15aa40 con 0x7fd97c153140 2024-04-25T00:04:43.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.403+0000 7fd97affd640 1 --2- 172.21.15.116:0/1007322051 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd97c0ab7b0 0x7fd97c159f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:04:43.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.403+0000 7fd97b7fe640 1 --2- 172.21.15.116:0/1007322051 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd97c153140 0x7fd97c1f21e0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fd97400c780 tx=0x7fd97400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:43.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.403+0000 7fd9637fe640 1 -- 172.21.15.116:0/1007322051 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd974018070 con 0x7fd97c153140 2024-04-25T00:04:43.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.403+0000 7fd9637fe640 1 -- 172.21.15.116:0/1007322051 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd97400ce30 con 0x7fd97c153140 2024-04-25T00:04:43.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.403+0000 7fd98163e640 1 -- 172.21.15.116:0/1007322051 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd97c15ad20 con 0x7fd97c153140 2024-04-25T00:04:43.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.403+0000 7fd9637fe640 1 -- 172.21.15.116:0/1007322051 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd97400a800 con 0x7fd97c153140 2024-04-25T00:04:43.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.403+0000 7fd98163e640 1 -- 172.21.15.116:0/1007322051 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd97c15fe50 con 0x7fd97c153140 2024-04-25T00:04:43.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.403+0000 7fd98163e640 1 -- 172.21.15.116:0/1007322051 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd97c068a60 con 0x7fd97c153140 2024-04-25T00:04:43.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.407+0000 7fd9637fe640 1 -- 172.21.15.116:0/1007322051 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd97401b070 con 0x7fd97c153140 2024-04-25T00:04:43.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.407+0000 7fd9637fe640 1 --2- 172.21.15.116:0/1007322051 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd9680678c0 0x7fd968069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:43.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.407+0000 7fd9637fe640 1 -- 172.21.15.116:0/1007322051 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 7010+0+0 (secure 0 0 0) 0x7fd9740857c0 con 0x7fd97c153140 2024-04-25T00:04:43.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.407+0000 7fd97affd640 1 --2- 172.21.15.116:0/1007322051 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd9680678c0 0x7fd968069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:43.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.407+0000 7fd97affd640 1 --2- 172.21.15.116:0/1007322051 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd9680678c0 0x7fd968069d80 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd96c009980 tx=0x7fd96c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:43.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.411+0000 7fd9637fe640 1 -- 172.21.15.116:0/1007322051 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fd974055e70 con 0x7fd97c153140 2024-04-25T00:04:43.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.631+0000 7fd98163e640 1 -- 172.21.15.116:0/1007322051 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd97c15aef0 con 0x7fd97c153140 2024-04-25T00:04:43.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.635+0000 7fd9637fe640 1 -- 172.21.15.116:0/1007322051 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+6502 (secure 0 0 0) 0x7fd974055770 con 0x7fd97c153140 2024-04-25T00:04:43.635 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:04:43.635 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5571,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/3841609112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3841609112},{"type":"v1","addr":"172.21.15.116:6833","nonce":3841609112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:04:38.435723+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4879},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4879":{"gid":4879,"name":"b","rank":0,"incarnation":32,"state":"up:active","state_seq":15,"addr":"172.21.15.174:6837/1889314958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1889314958},{"type":"v1","addr":"172.21.15.174:6837","nonce":1889314958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4879,"qdb_cluster":[4879]},"id":1},{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:03:34.579825+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4735},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4735":{"gid":4735,"name":"a","rank":0,"incarnation":20,"state":"up:active","state_seq":14,"addr":"172.21.15.116:6835/2004876235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6833","nonce":2004876235},{"type":"v1","addr":"172.21.15.116:6835","nonce":2004876235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4735,"qdb_cluster":[4735]},"id":2}]} 2024-04-25T00:04:43.635 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 36 2024-04-25T00:04:43.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.635+0000 7fd98163e640 1 -- 172.21.15.116:0/1007322051 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd9680678c0 msgr2=0x7fd968069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:43.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.635+0000 7fd98163e640 1 --2- 172.21.15.116:0/1007322051 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd9680678c0 0x7fd968069d80 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd96c009980 tx=0x7fd96c008040 comp rx=0 tx=0).stop 2024-04-25T00:04:43.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.635+0000 7fd98163e640 1 -- 172.21.15.116:0/1007322051 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd97c153140 msgr2=0x7fd97c1f21e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:43.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.635+0000 7fd98163e640 1 --2- 172.21.15.116:0/1007322051 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd97c153140 0x7fd97c1f21e0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fd97400c780 tx=0x7fd97400cc40 comp rx=0 tx=0).stop 2024-04-25T00:04:43.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.635+0000 7fd98163e640 1 -- 172.21.15.116:0/1007322051 shutdown_connections 2024-04-25T00:04:43.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.635+0000 7fd98163e640 1 --2- 172.21.15.116:0/1007322051 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd9680678c0 0x7fd968069d80 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:43.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.635+0000 7fd98163e640 1 --2- 172.21.15.116:0/1007322051 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd97c153140 0x7fd97c1f21e0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:43.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.635+0000 7fd98163e640 1 --2- 172.21.15.116:0/1007322051 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd97c152d60 0x7fd97c0ad280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:43.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.635+0000 7fd98163e640 1 --2- 172.21.15.116:0/1007322051 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd97c0ab7b0 0x7fd97c159f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:43.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.635+0000 7fd98163e640 1 -- 172.21.15.116:0/1007322051 >> 172.21.15.116:0/1007322051 conn(0x7fd97c06f0c0 msgr2=0x7fd97c153670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:04:43.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.635+0000 7fd98163e640 1 -- 172.21.15.116:0/1007322051 shutdown_connections 2024-04-25T00:04:43.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:43.635+0000 7fd98163e640 1 -- 172.21.15.116:0/1007322051 wait complete. 2024-04-25T00:04:43.654 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:04:43.977 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi174.stderr:2024-04-25T00:04:43.975+0000 7f7ba7562640 -1 --2- 192.168.0.1:0/3700357214 >> [v2:172.21.15.116:6833/2004876235,v1:172.21.15.116:6835/2004876235] conn(0x7f7b8807eff0 0x7f7b8801ac90 unknown :-1 s=BANNER_CONNECTING pgs=12 cs=7 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner peer [v2:172.21.15.116:6833/2004876235,v1:172.21.15.116:6835/2004876235] is using msgr V1 protocol 2024-04-25T00:04:45.655 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:04:45.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:45.791+0000 7f09b5026640 1 Processor -- start 2024-04-25T00:04:45.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:45.791+0000 7f09b5026640 1 -- start start 2024-04-25T00:04:45.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:45.795+0000 7f09b5026640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f09b0152d80 0x7f09b006bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:45.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:45.795+0000 7f09b5026640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f09b006c290 0x7f09b0157390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:45.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:45.795+0000 7f09b5026640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09b00c9150 con 0x7f09b006d5f0 2024-04-25T00:04:45.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:45.795+0000 7f09b5026640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f09b00c9d60 con 0x7f09b006c290 2024-04-25T00:04:45.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:45.795+0000 7f09b5026640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f09b00c9430 con 0x7f09b0152d80 2024-04-25T00:04:45.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:45.795+0000 7f09af59e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f09b006c290 0x7f09b0157390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:45.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:45.795+0000 7f09af59e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f09b006c290 0x7f09b0157390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57908/0 (socket says 172.21.15.116:57908) 2024-04-25T00:04:45.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:45.795+0000 7f09af59e640 1 -- 172.21.15.116:0/1304606009 learned_addr learned my addr 172.21.15.116:0/1304606009 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:04:45.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:45.795+0000 7f09aed9d640 1 --2- 172.21.15.116:0/1304606009 >> v2:172.21.15.174:3300/0 conn(0x7f09b0152d80 0x7f09b006bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:45.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:45.795+0000 7f09af59e640 1 -- 172.21.15.116:0/1304606009 >> v2:172.21.15.174:3300/0 conn(0x7f09b0152d80 msgr2=0x7f09b006bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:04:45.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:45.795+0000 7f09af59e640 1 --2- 172.21.15.116:0/1304606009 >> v2:172.21.15.174:3300/0 conn(0x7f09b0152d80 0x7f09b006bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:45.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:45.795+0000 7f09af59e640 1 -- 172.21.15.116:0/1304606009 >> v1:172.21.15.174:6789/0 conn(0x7f09b006d5f0 legacy=0x7f09b0156c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:04:45.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:45.795+0000 7f09af59e640 1 -- 172.21.15.116:0/1304606009 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09b00c9710 con 0x7f09b006c290 2024-04-25T00:04:45.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:45.795+0000 7f09af59e640 1 --2- 172.21.15.116:0/1304606009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09b006c290 0x7f09b0157390 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f09a8009e50 tx=0x7f09a801c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6cbbc8d7863a1af server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:45.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:45.795+0000 7f09ae59c640 1 -- 172.21.15.116:0/1304606009 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a80a57d0 con 0x7f09b006c290 2024-04-25T00:04:45.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:45.795+0000 7f09ae59c640 1 -- 172.21.15.116:0/1304606009 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09a80a5970 con 0x7f09b006c290 2024-04-25T00:04:45.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:45.795+0000 7f09ae59c640 1 -- 172.21.15.116:0/1304606009 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a80a5d40 con 0x7f09b006c290 2024-04-25T00:04:45.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:45.799+0000 7f09b5026640 1 -- 172.21.15.116:0/1304606009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09b006c290 msgr2=0x7f09b0157390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:45.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:45.799+0000 7f09b5026640 1 --2- 172.21.15.116:0/1304606009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09b006c290 0x7f09b0157390 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f09a8009e50 tx=0x7f09a801c8f0 comp rx=0 tx=0).stop 2024-04-25T00:04:45.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:45.799+0000 7f09b5026640 1 -- 172.21.15.116:0/1304606009 shutdown_connections 2024-04-25T00:04:45.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:45.799+0000 7f09b5026640 1 --2- 172.21.15.116:0/1304606009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09b006c290 0x7f09b0157390 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:45.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:45.799+0000 7f09b5026640 1 --2- 172.21.15.116:0/1304606009 >> v2:172.21.15.174:3300/0 conn(0x7f09b0152d80 0x7f09b006bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:45.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:45.799+0000 7f09b5026640 1 -- 172.21.15.116:0/1304606009 >> 172.21.15.116:0/1304606009 conn(0x7f09b00acdd0 msgr2=0x7f09b00ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:04:45.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:45.799+0000 7f09b5026640 1 -- 172.21.15.116:0/1304606009 shutdown_connections 2024-04-25T00:04:45.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:45.799+0000 7f09b5026640 1 -- 172.21.15.116:0/1304606009 wait complete. 2024-04-25T00:04:45.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:45.799+0000 7f09b5026640 1 Processor -- start 2024-04-25T00:04:45.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:45.799+0000 7f09b5026640 1 -- start start 2024-04-25T00:04:45.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:45.799+0000 7f09b5026640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f09b006c290 0x7f09b0151c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:45.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:45.799+0000 7f09b5026640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f09b006d5f0 0x7f09b00af250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:45.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:45.799+0000 7f09b5026640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09b0152d80 0x7f09b00af790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:45.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:45.799+0000 7f09b5026640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f09b00caf00 con 0x7f09b0152d80 2024-04-25T00:04:45.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:45.799+0000 7f09b5026640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f09b00cad80 con 0x7f09b006d5f0 2024-04-25T00:04:45.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:45.799+0000 7f09b5026640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f09b01628d0 con 0x7f09b006c290 2024-04-25T00:04:45.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:45.799+0000 7f09af59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09b0152d80 0x7f09b00af790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:45.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:45.799+0000 7f09af59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09b0152d80 0x7f09b00af790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57912/0 (socket says 172.21.15.116:57912) 2024-04-25T00:04:45.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:45.799+0000 7f09af59e640 1 -- 172.21.15.116:0/1028181169 learned_addr learned my addr 172.21.15.116:0/1028181169 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:04:45.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:45.799+0000 7f09aed9d640 1 --2- 172.21.15.116:0/1028181169 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f09b006c290 0x7f09b0151c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:45.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:45.799+0000 7f09a7fff640 1 --2- 172.21.15.116:0/1028181169 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f09b006d5f0 0x7f09b00af250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:45.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:45.799+0000 7f09af59e640 1 -- 172.21.15.116:0/1028181169 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f09b006c290 msgr2=0x7f09b0151c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:45.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:45.799+0000 7f09af59e640 1 --2- 172.21.15.116:0/1028181169 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f09b006c290 0x7f09b0151c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:45.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:45.799+0000 7f09af59e640 1 -- 172.21.15.116:0/1028181169 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f09b006d5f0 msgr2=0x7f09b00af250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:45.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:45.799+0000 7f09af59e640 1 --2- 172.21.15.116:0/1028181169 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f09b006d5f0 0x7f09b00af250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:45.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:45.799+0000 7f09af59e640 1 -- 172.21.15.116:0/1028181169 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09b00b0d50 con 0x7f09b0152d80 2024-04-25T00:04:45.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:45.799+0000 7f09af59e640 1 --2- 172.21.15.116:0/1028181169 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09b0152d80 0x7f09b00af790 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f09a801a520 tx=0x7f09a8019690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:45.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:45.803+0000 7f09a77fe640 1 -- 172.21.15.116:0/1028181169 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a80bc020 con 0x7f09b0152d80 2024-04-25T00:04:45.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:45.803+0000 7f09a77fe640 1 -- 172.21.15.116:0/1028181169 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09a80b8670 con 0x7f09b0152d80 2024-04-25T00:04:45.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:45.803+0000 7f09b5026640 1 -- 172.21.15.116:0/1028181169 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09b00c54c0 con 0x7f09b0152d80 2024-04-25T00:04:45.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:45.803+0000 7f09a77fe640 1 -- 172.21.15.116:0/1028181169 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a80b8a20 con 0x7f09b0152d80 2024-04-25T00:04:45.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:45.803+0000 7f09b5026640 1 -- 172.21.15.116:0/1028181169 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09b00ae490 con 0x7f09b0152d80 2024-04-25T00:04:45.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:45.803+0000 7f09b5026640 1 -- 172.21.15.116:0/1028181169 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09b0068350 con 0x7f09b0152d80 2024-04-25T00:04:45.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:45.803+0000 7f09a77fe640 1 -- 172.21.15.116:0/1028181169 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f09a80bf020 con 0x7f09b0152d80 2024-04-25T00:04:45.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:45.803+0000 7f09a77fe640 1 --2- 172.21.15.116:0/1028181169 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f09900678c0 0x7f0990069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:45.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:45.803+0000 7f09a77fe640 1 -- 172.21.15.116:0/1028181169 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7096+0+0 (secure 0 0 0) 0x7f09a81212e0 con 0x7f09b0152d80 2024-04-25T00:04:45.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:45.803+0000 7f09aed9d640 1 --2- 172.21.15.116:0/1028181169 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f09900678c0 0x7f0990069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:45.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:45.803+0000 7f09aed9d640 1 --2- 172.21.15.116:0/1028181169 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f09900678c0 0x7f0990069d80 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f09b0068c40 tx=0x7f09a0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:45.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:45.807+0000 7f09a77fe640 1 -- 172.21.15.116:0/1028181169 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f09a80c6030 con 0x7f09b0152d80 2024-04-25T00:04:46.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.043+0000 7f09b5026640 1 -- 172.21.15.116:0/1028181169 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f09b00cab60 con 0x7f09b0152d80 2024-04-25T00:04:46.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.047+0000 7f09a77fe640 1 -- 172.21.15.116:0/1028181169 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+5666 (secure 0 0 0) 0x7f09a80165a0 con 0x7f09b0152d80 2024-04-25T00:04:46.046 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:04:46.046 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:04:38.435723+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4879},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4879":{"gid":4879,"name":"b","rank":0,"incarnation":32,"state":"up:active","state_seq":15,"addr":"172.21.15.174:6837/1889314958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1889314958},{"type":"v1","addr":"172.21.15.174:6837","nonce":1889314958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4879,"qdb_cluster":[4879]},"id":1},{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:04:45.422030+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5571},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5571":{"gid":5571,"name":"c","rank":0,"incarnation":37,"state":"up:replay","state_seq":1,"addr":"172.21.15.116:6833/3841609112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3841609112},{"type":"v1","addr":"172.21.15.116:6833","nonce":3841609112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:04:46.046 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 37 2024-04-25T00:04:46.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.047+0000 7f09b5026640 1 -- 172.21.15.116:0/1028181169 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f09900678c0 msgr2=0x7f0990069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:46.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.047+0000 7f09b5026640 1 --2- 172.21.15.116:0/1028181169 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f09900678c0 0x7f0990069d80 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f09b0068c40 tx=0x7f09a0008040 comp rx=0 tx=0).stop 2024-04-25T00:04:46.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.047+0000 7f09b5026640 1 -- 172.21.15.116:0/1028181169 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09b0152d80 msgr2=0x7f09b00af790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:46.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.047+0000 7f09b5026640 1 --2- 172.21.15.116:0/1028181169 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09b0152d80 0x7f09b00af790 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f09a801a520 tx=0x7f09a8019690 comp rx=0 tx=0).stop 2024-04-25T00:04:46.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.047+0000 7f09b5026640 1 -- 172.21.15.116:0/1028181169 shutdown_connections 2024-04-25T00:04:46.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.047+0000 7f09b5026640 1 --2- 172.21.15.116:0/1028181169 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f09900678c0 0x7f0990069d80 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:46.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.047+0000 7f09b5026640 1 --2- 172.21.15.116:0/1028181169 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f09b0152d80 0x7f09b00af790 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:46.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.047+0000 7f09b5026640 1 --2- 172.21.15.116:0/1028181169 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f09b006d5f0 0x7f09b00af250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:46.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.047+0000 7f09b5026640 1 --2- 172.21.15.116:0/1028181169 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f09b006c290 0x7f09b0151c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:46.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.047+0000 7f09b5026640 1 -- 172.21.15.116:0/1028181169 >> 172.21.15.116:0/1028181169 conn(0x7f09b00acdd0 msgr2=0x7f09b0154e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:04:46.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.047+0000 7f09b5026640 1 -- 172.21.15.116:0/1028181169 shutdown_connections 2024-04-25T00:04:46.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.047+0000 7f09b5026640 1 -- 172.21.15.116:0/1028181169 wait complete. 2024-04-25T00:04:46.065 INFO:tasks.mds_thrash.fs.[b]:mds.a down, removed from mdsmap 2024-04-25T00:04:46.065 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-04-25T00:04:46.065 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:04:46.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.195+0000 7faeb8c47640 1 Processor -- start 2024-04-25T00:04:46.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.195+0000 7faeb8c47640 1 -- start start 2024-04-25T00:04:46.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.195+0000 7faeb8c47640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faeb40aeb20 0x7faeb40c9310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:46.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.195+0000 7faeb8c47640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7faeb40c9850 0x7faeb4150870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:46.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.195+0000 7faeb8c47640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faeb41535d0 con 0x7faeb4150db0 2024-04-25T00:04:46.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.195+0000 7faeb8c47640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7faeb40c9cb0 con 0x7faeb40c9850 2024-04-25T00:04:46.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.195+0000 7faeb8c47640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7faeb4151220 con 0x7faeb40aeb20 2024-04-25T00:04:46.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.195+0000 7faeb1d9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7faeb40c9850 0x7faeb4150870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:46.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.195+0000 7faeb2d9d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7faeb4150db0 0x7faeb40c5270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:40166/0 (socket says 172.21.15.116:40166) 2024-04-25T00:04:46.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.195+0000 7faeb2d9d640 1 -- 172.21.15.116:0/1771732607 learned_addr learned my addr 172.21.15.116:0/1771732607 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:04:46.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.195+0000 7faeb259c640 1 --2- 172.21.15.116:0/1771732607 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faeb40aeb20 0x7faeb40c9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:46.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.195+0000 7faeb1d9b640 1 -- 172.21.15.116:0/1771732607 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faeb40aeb20 msgr2=0x7faeb40c9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:04:46.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.195+0000 7faeb1d9b640 1 --2- 172.21.15.116:0/1771732607 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faeb40aeb20 0x7faeb40c9310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:46.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.195+0000 7faeb1d9b640 1 -- 172.21.15.116:0/1771732607 >> v1:172.21.15.116:6789/0 conn(0x7faeb4150db0 legacy=0x7faeb40c5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:04:46.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.195+0000 7faeb1d9b640 1 -- 172.21.15.116:0/1771732607 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faeb4153850 con 0x7faeb40c9850 2024-04-25T00:04:46.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.199+0000 7faeb1d9b640 1 --2- 172.21.15.116:0/1771732607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faeb40c9850 0x7faeb4150870 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7faea80097c0 tx=0x7faea801c1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76d234e269d4f319 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:46.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.199+0000 7faeb159a640 1 -- 172.21.15.116:0/1771732607 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faea80b0070 con 0x7faeb40c9850 2024-04-25T00:04:46.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.199+0000 7faeb159a640 1 -- 172.21.15.116:0/1771732607 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faea80ab600 con 0x7faeb40c9850 2024-04-25T00:04:46.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.199+0000 7faeb159a640 1 -- 172.21.15.116:0/1771732607 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faea80ab9d0 con 0x7faeb40c9850 2024-04-25T00:04:46.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.199+0000 7faeb8c47640 1 -- 172.21.15.116:0/1771732607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faeb40c9850 msgr2=0x7faeb4150870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:46.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.199+0000 7faeb8c47640 1 --2- 172.21.15.116:0/1771732607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faeb40c9850 0x7faeb4150870 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7faea80097c0 tx=0x7faea801c1c0 comp rx=0 tx=0).stop 2024-04-25T00:04:46.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.199+0000 7faeb8c47640 1 -- 172.21.15.116:0/1771732607 shutdown_connections 2024-04-25T00:04:46.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.199+0000 7faeb8c47640 1 --2- 172.21.15.116:0/1771732607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faeb40c9850 0x7faeb4150870 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:46.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.199+0000 7faeb8c47640 1 --2- 172.21.15.116:0/1771732607 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faeb40aeb20 0x7faeb40c9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:46.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.199+0000 7faeb8c47640 1 -- 172.21.15.116:0/1771732607 >> 172.21.15.116:0/1771732607 conn(0x7faeb4074650 msgr2=0x7faeb4074a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:04:46.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.199+0000 7faeb8c47640 1 -- 172.21.15.116:0/1771732607 shutdown_connections 2024-04-25T00:04:46.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.199+0000 7faeb8c47640 1 -- 172.21.15.116:0/1771732607 wait complete. 2024-04-25T00:04:46.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.199+0000 7faeb8c47640 1 Processor -- start 2024-04-25T00:04:46.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.199+0000 7faeb8c47640 1 -- start start 2024-04-25T00:04:46.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.199+0000 7faeb8c47640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faeb40aeb20 0x7faeb406de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:46.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.199+0000 7faeb8c47640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faeb40c9850 0x7faeb40a8d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:46.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.199+0000 7faeb8c47640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faeb4150db0 0x7faeb40ad0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:46.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.199+0000 7faeb8c47640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7faeb40c5d90 con 0x7faeb40c9850 2024-04-25T00:04:46.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.199+0000 7faeb8c47640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7faeb41543d0 con 0x7faeb40aeb20 2024-04-25T00:04:46.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.199+0000 7faeb8c47640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7faeb40c5f10 con 0x7faeb4150db0 2024-04-25T00:04:46.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.199+0000 7faeb1d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faeb40c9850 0x7faeb40a8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:46.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.199+0000 7faeb1d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faeb40c9850 0x7faeb40a8d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57922/0 (socket says 172.21.15.116:57922) 2024-04-25T00:04:46.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.199+0000 7faeb1d9b640 1 -- 172.21.15.116:0/2242021549 learned_addr learned my addr 172.21.15.116:0/2242021549 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:04:46.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.199+0000 7faeb259c640 1 --2- 172.21.15.116:0/2242021549 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faeb40aeb20 0x7faeb406de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:46.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.199+0000 7faeb2d9d640 1 --2- 172.21.15.116:0/2242021549 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faeb4150db0 0x7faeb40ad0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:46.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.199+0000 7faeb1d9b640 1 -- 172.21.15.116:0/2242021549 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faeb4150db0 msgr2=0x7faeb40ad0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:46.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.199+0000 7faeb1d9b640 1 --2- 172.21.15.116:0/2242021549 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faeb4150db0 0x7faeb40ad0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:46.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.199+0000 7faeb1d9b640 1 -- 172.21.15.116:0/2242021549 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faeb40aeb20 msgr2=0x7faeb406de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:46.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.199+0000 7faeb1d9b640 1 --2- 172.21.15.116:0/2242021549 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faeb40aeb20 0x7faeb406de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:46.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.199+0000 7faeb1d9b640 1 -- 172.21.15.116:0/2242021549 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faeb41eeb00 con 0x7faeb40c9850 2024-04-25T00:04:46.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.199+0000 7faeb2d9d640 1 --2- 172.21.15.116:0/2242021549 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faeb4150db0 0x7faeb40ad0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:04:46.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.199+0000 7faeb259c640 1 --2- 172.21.15.116:0/2242021549 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faeb40aeb20 0x7faeb406de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:04:46.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.199+0000 7faeb1d9b640 1 --2- 172.21.15.116:0/2242021549 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faeb40c9850 0x7faeb40a8d00 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7faea801a870 tx=0x7faea8018f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:46.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.203+0000 7faea2ffd640 1 -- 172.21.15.116:0/2242021549 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faea80b8020 con 0x7faeb40c9850 2024-04-25T00:04:46.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.203+0000 7faea2ffd640 1 -- 172.21.15.116:0/2242021549 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faea80b4410 con 0x7faeb40c9850 2024-04-25T00:04:46.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.203+0000 7faeb8c47640 1 -- 172.21.15.116:0/2242021549 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faeb414d920 con 0x7faeb40c9850 2024-04-25T00:04:46.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.203+0000 7faea2ffd640 1 -- 172.21.15.116:0/2242021549 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faea80b0040 con 0x7faeb40c9850 2024-04-25T00:04:46.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.203+0000 7faeb8c47640 1 -- 172.21.15.116:0/2242021549 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faeb40c2f30 con 0x7faeb40c9850 2024-04-25T00:04:46.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.203+0000 7faeb8c47640 1 -- 172.21.15.116:0/2242021549 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faeb4068a60 con 0x7faeb40c9850 2024-04-25T00:04:46.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.203+0000 7faea2ffd640 1 -- 172.21.15.116:0/2242021549 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7faea80bb020 con 0x7faeb40c9850 2024-04-25T00:04:46.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.203+0000 7faea2ffd640 1 --2- 172.21.15.116:0/2242021549 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fae940678c0 0x7fae94069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:46.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.203+0000 7faea2ffd640 1 -- 172.21.15.116:0/2242021549 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7096+0+0 (secure 0 0 0) 0x7faea811fe60 con 0x7faeb40c9850 2024-04-25T00:04:46.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.203+0000 7faeb259c640 1 --2- 172.21.15.116:0/2242021549 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fae940678c0 0x7fae94069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:46.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.203+0000 7faeb259c640 1 --2- 172.21.15.116:0/2242021549 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fae940678c0 0x7fae94069d80 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7faea4009840 tx=0x7faea4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:46.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.211+0000 7faea2ffd640 1 -- 172.21.15.116:0/2242021549 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7faea80c2030 con 0x7faeb40c9850 2024-04-25T00:04:46.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.411+0000 7faeb8c47640 1 -- 172.21.15.116:0/2242021549 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faeb40a9960 con 0x7faeb40c9850 2024-04-25T00:04:46.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.415+0000 7faea2ffd640 1 -- 172.21.15.116:0/2242021549 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+5666 (secure 0 0 0) 0x7faeb41535d0 con 0x7faeb40c9850 2024-04-25T00:04:46.414 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:04:46.414 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:04:38.435723+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4879},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4879":{"gid":4879,"name":"b","rank":0,"incarnation":32,"state":"up:active","state_seq":15,"addr":"172.21.15.174:6837/1889314958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1889314958},{"type":"v1","addr":"172.21.15.174:6837","nonce":1889314958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4879,"qdb_cluster":[4879]},"id":1},{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:04:45.422030+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5571},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5571":{"gid":5571,"name":"c","rank":0,"incarnation":37,"state":"up:replay","state_seq":1,"addr":"172.21.15.116:6833/3841609112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3841609112},{"type":"v1","addr":"172.21.15.116:6833","nonce":3841609112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:04:46.415 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 37 2024-04-25T00:04:46.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.415+0000 7faeb8c47640 1 -- 172.21.15.116:0/2242021549 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fae940678c0 msgr2=0x7fae94069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:46.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.415+0000 7faeb8c47640 1 --2- 172.21.15.116:0/2242021549 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fae940678c0 0x7fae94069d80 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7faea4009840 tx=0x7faea4008040 comp rx=0 tx=0).stop 2024-04-25T00:04:46.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.415+0000 7faeb8c47640 1 -- 172.21.15.116:0/2242021549 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faeb40c9850 msgr2=0x7faeb40a8d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:46.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.415+0000 7faeb8c47640 1 --2- 172.21.15.116:0/2242021549 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faeb40c9850 0x7faeb40a8d00 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7faea801a870 tx=0x7faea8018f60 comp rx=0 tx=0).stop 2024-04-25T00:04:46.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.415+0000 7faeb8c47640 1 -- 172.21.15.116:0/2242021549 shutdown_connections 2024-04-25T00:04:46.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.415+0000 7faeb8c47640 1 --2- 172.21.15.116:0/2242021549 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fae940678c0 0x7fae94069d80 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:46.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.415+0000 7faeb8c47640 1 --2- 172.21.15.116:0/2242021549 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faeb4150db0 0x7faeb40ad0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:46.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.415+0000 7faeb8c47640 1 --2- 172.21.15.116:0/2242021549 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faeb40c9850 0x7faeb40a8d00 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:46.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.415+0000 7faeb8c47640 1 --2- 172.21.15.116:0/2242021549 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faeb40aeb20 0x7faeb406de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:46.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.415+0000 7faeb8c47640 1 -- 172.21.15.116:0/2242021549 >> 172.21.15.116:0/2242021549 conn(0x7faeb4074650 msgr2=0x7faeb4074a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:04:46.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.415+0000 7faeb8c47640 1 -- 172.21.15.116:0/2242021549 shutdown_connections 2024-04-25T00:04:46.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:46.415+0000 7faeb8c47640 1 -- 172.21.15.116:0/2242021549 wait complete. 2024-04-25T00:04:46.433 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 37, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:04.678737+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 35, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_4879": { "addr": "172.21.15.174:6837/1889314958", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6836", "nonce": 1889314958, "type": "v2" }, { "addr": "172.21.15.174:6837", "nonce": 1889314958, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4879, "incarnation": 32, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 15 } }, "last_failure": 0, "last_failure_osd_epoch": 33, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T00:04:38.435723+0000", "qdb_cluster": [ 4879 ], "qdb_leader": 4879, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4879 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:18.825887+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 37, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5571": { "addr": "172.21.15.116:6833/3841609112", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6832", "nonce": 3841609112, "type": "v2" }, { "addr": "172.21.15.116:6833", "nonce": 3841609112, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5571, "incarnation": 37, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 34, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-04-25T00:04:45.422030+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5571 } } } ], "standbys": [ { "addr": "172.21.15.174:6835/2883220846", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6834", "nonce": 2883220846, "type": "v2" }, { "addr": "172.21.15.174:6835", "nonce": 2883220846, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4235, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-25T00:04:48.434 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:04:48.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.587+0000 7f52cae1e640 1 Processor -- start 2024-04-25T00:04:48.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.587+0000 7f52cae1e640 1 -- start start 2024-04-25T00:04:48.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.587+0000 7f52cae1e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f52c40730a0 0x7f52c4154290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:48.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.587+0000 7f52cae1e640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52c4153810 con 0x7f52c40c2bf0 2024-04-25T00:04:48.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.587+0000 7f52cae1e640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52c40c3690 con 0x7f52c40c3140 2024-04-25T00:04:48.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.587+0000 7f52cae1e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f52c41547d0 con 0x7f52c40730a0 2024-04-25T00:04:48.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.587+0000 7f52c3fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f52c40c2bf0 0x7f52c40723b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:40176/0 (socket says 172.21.15.116:40176) 2024-04-25T00:04:48.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.587+0000 7f52c3fff640 1 -- 172.21.15.116:0/2725988105 learned_addr learned my addr 172.21.15.116:0/2725988105 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:04:48.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.587+0000 7f52c8bba640 1 --2- 172.21.15.116:0/2725988105 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f52c40730a0 0x7f52c4154290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:48.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.587+0000 7f52c37fe640 1 -- 172.21.15.116:0/2725988105 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1426484073 0 0) 0x7f52b8001120 con 0x7f52c40c2bf0 2024-04-25T00:04:48.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.587+0000 7f52c37fe640 1 -- 172.21.15.116:0/2725988105 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f52ac003530 con 0x7f52c40c2bf0 2024-04-25T00:04:48.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.587+0000 7f52c37fe640 1 -- 172.21.15.116:0/2725988105 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4029406845 0 0) 0x7f52ac003530 con 0x7f52c40c2bf0 2024-04-25T00:04:48.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.587+0000 7f52c37fe640 1 -- 172.21.15.116:0/2725988105 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f52c40730a0 msgr2=0x7f52c4154290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:04:48.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.587+0000 7f52c37fe640 1 --2- 172.21.15.116:0/2725988105 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f52c40730a0 0x7f52c4154290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:48.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.587+0000 7f52c37fe640 1 -- 172.21.15.116:0/2725988105 >> v1:172.21.15.174:6789/0 conn(0x7f52c40c3140 legacy=0x7f52c406c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:48.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.587+0000 7f52c37fe640 1 -- 172.21.15.116:0/2725988105 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52c40c3970 con 0x7f52c40c2bf0 2024-04-25T00:04:48.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.587+0000 7f52c37fe640 1 -- 172.21.15.116:0/2725988105 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f52b8001850 con 0x7f52c40c2bf0 2024-04-25T00:04:48.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.587+0000 7f52c37fe640 1 -- 172.21.15.116:0/2725988105 >> v1:172.21.15.116:6789/0 conn(0x7f52c40c2bf0 legacy=0x7f52c40723b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:48.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.587+0000 7f52c37fe640 1 --2- 172.21.15.116:0/2725988105 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52ac0039f0 0x7f52ac003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:48.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.587+0000 7f52c37fe640 1 --2- 172.21.15.116:0/2725988105 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f52c40730a0 0x7f52ac004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:48.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.587+0000 7f52c37fe640 1 --2- 172.21.15.116:0/2725988105 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52ac004d50 0x7f52ac0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:48.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.587+0000 7f52c37fe640 1 -- 172.21.15.116:0/2725988105 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f52c41547d0 con 0x7f52ac004d50 2024-04-25T00:04:48.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.587+0000 7f52c37fe640 1 -- 172.21.15.116:0/2725988105 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f52ac00ba80 con 0x7f52ac0039f0 2024-04-25T00:04:48.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.587+0000 7f52c37fe640 1 -- 172.21.15.116:0/2725988105 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f52ac010090 con 0x7f52c40730a0 2024-04-25T00:04:48.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.587+0000 7f52c93bb640 1 --2- 172.21.15.116:0/2725988105 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52ac004d50 0x7f52ac0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:48.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.587+0000 7f52c8bba640 1 --2- 172.21.15.116:0/2725988105 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52ac0039f0 0x7f52ac003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:48.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.587+0000 7f52c3fff640 1 --2- 172.21.15.116:0/2725988105 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f52c40730a0 0x7f52ac004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:48.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.587+0000 7f52c93bb640 1 -- 172.21.15.116:0/2725988105 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f52c40730a0 msgr2=0x7f52ac004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:04:48.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.587+0000 7f52c93bb640 1 --2- 172.21.15.116:0/2725988105 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f52c40730a0 0x7f52ac004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:48.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.587+0000 7f52c93bb640 1 -- 172.21.15.116:0/2725988105 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52ac0039f0 msgr2=0x7f52ac003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:04:48.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.587+0000 7f52c93bb640 1 --2- 172.21.15.116:0/2725988105 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52ac0039f0 0x7f52ac003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:48.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.587+0000 7f52c93bb640 1 -- 172.21.15.116:0/2725988105 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52ac003530 con 0x7f52ac004d50 2024-04-25T00:04:48.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.591+0000 7f52c93bb640 1 --2- 172.21.15.116:0/2725988105 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52ac004d50 0x7f52ac0051d0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f52bc001e40 tx=0x7f52bc01c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb200abe7cc563b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:48.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.591+0000 7f52c37fe640 1 -- 172.21.15.116:0/2725988105 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52bc001a90 con 0x7f52ac004d50 2024-04-25T00:04:48.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.591+0000 7f52c37fe640 1 -- 172.21.15.116:0/2725988105 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52bc003b80 con 0x7f52ac004d50 2024-04-25T00:04:48.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.591+0000 7f52c37fe640 1 -- 172.21.15.116:0/2725988105 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52bc005080 con 0x7f52ac004d50 2024-04-25T00:04:48.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.591+0000 7f52cae1e640 1 -- 172.21.15.116:0/2725988105 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52ac004d50 msgr2=0x7f52ac0051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:48.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.591+0000 7f52cae1e640 1 --2- 172.21.15.116:0/2725988105 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52ac004d50 0x7f52ac0051d0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f52bc001e40 tx=0x7f52bc01c760 comp rx=0 tx=0).stop 2024-04-25T00:04:48.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.591+0000 7f52c93bb640 1 -- 172.21.15.116:0/2725988105 reap_dead start 2024-04-25T00:04:48.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.591+0000 7f52cae1e640 1 -- 172.21.15.116:0/2725988105 shutdown_connections 2024-04-25T00:04:48.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.591+0000 7f52cae1e640 1 -- 172.21.15.116:0/2725988105 >> 172.21.15.116:0/2725988105 conn(0x7f52c40c42e0 msgr2=0x7f52c40c46f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:04:48.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.591+0000 7f52cae1e640 1 -- 172.21.15.116:0/2725988105 shutdown_connections 2024-04-25T00:04:48.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.591+0000 7f52cae1e640 1 -- 172.21.15.116:0/2725988105 wait complete. 2024-04-25T00:04:48.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.595+0000 7f52cae1e640 1 Processor -- start 2024-04-25T00:04:48.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.595+0000 7f52cae1e640 1 -- start start 2024-04-25T00:04:48.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.595+0000 7f52cae1e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52ac004d50 0x7f52c41e74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:48.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.595+0000 7f52cae1e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f52c41579b0 0x7f52c4156750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:48.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.595+0000 7f52cae1e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52c40bf9f0 0x7f52c40bfe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:48.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.595+0000 7f52cae1e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f52c4073480 con 0x7f52ac004d50 2024-04-25T00:04:48.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.595+0000 7f52cae1e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f52c4153d70 con 0x7f52c40bf9f0 2024-04-25T00:04:48.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.595+0000 7f52cae1e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f52c406c930 con 0x7f52c41579b0 2024-04-25T00:04:48.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.595+0000 7f52c8bba640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52ac004d50 0x7f52c41e74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:48.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.595+0000 7f52c8bba640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52ac004d50 0x7f52c41e74c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57936/0 (socket says 172.21.15.116:57936) 2024-04-25T00:04:48.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.595+0000 7f52c8bba640 1 -- 172.21.15.116:0/4139866829 learned_addr learned my addr 172.21.15.116:0/4139866829 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:04:48.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.595+0000 7f52c3fff640 1 --2- 172.21.15.116:0/4139866829 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f52c41579b0 0x7f52c4156750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:48.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.595+0000 7f52c93bb640 1 --2- 172.21.15.116:0/4139866829 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52c40bf9f0 0x7f52c40bfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:48.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.595+0000 7f52c8bba640 1 -- 172.21.15.116:0/4139866829 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f52c41579b0 msgr2=0x7f52c4156750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:48.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.595+0000 7f52c8bba640 1 --2- 172.21.15.116:0/4139866829 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f52c41579b0 0x7f52c4156750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:48.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.595+0000 7f52c8bba640 1 -- 172.21.15.116:0/4139866829 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52c40bf9f0 msgr2=0x7f52c40bfe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:48.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.595+0000 7f52c8bba640 1 --2- 172.21.15.116:0/4139866829 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52c40bf9f0 0x7f52c40bfe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:48.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.595+0000 7f52c8bba640 1 -- 172.21.15.116:0/4139866829 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52c40c1820 con 0x7f52ac004d50 2024-04-25T00:04:48.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.595+0000 7f52c8bba640 1 --2- 172.21.15.116:0/4139866829 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52ac004d50 0x7f52c41e74c0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f52b4011640 tx=0x7f52b4011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:48.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.595+0000 7f52c17fa640 1 -- 172.21.15.116:0/4139866829 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b400f920 con 0x7f52ac004d50 2024-04-25T00:04:48.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.595+0000 7f52c17fa640 1 -- 172.21.15.116:0/4139866829 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52b400fac0 con 0x7f52ac004d50 2024-04-25T00:04:48.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.595+0000 7f52cae1e640 1 -- 172.21.15.116:0/4139866829 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52c40a8780 con 0x7f52ac004d50 2024-04-25T00:04:48.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.595+0000 7f52c17fa640 1 -- 172.21.15.116:0/4139866829 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b400fe70 con 0x7f52ac004d50 2024-04-25T00:04:48.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.595+0000 7f52cae1e640 1 -- 172.21.15.116:0/4139866829 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52c40c1ae0 con 0x7f52ac004d50 2024-04-25T00:04:48.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.595+0000 7f52cae1e640 1 -- 172.21.15.116:0/4139866829 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52c4068a60 con 0x7f52ac004d50 2024-04-25T00:04:48.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.599+0000 7f52c17fa640 1 -- 172.21.15.116:0/4139866829 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f52b4015460 con 0x7f52ac004d50 2024-04-25T00:04:48.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.599+0000 7f52c17fa640 1 --2- 172.21.15.116:0/4139866829 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f52b00678f0 0x7f52b0069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:48.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.599+0000 7f52c17fa640 1 -- 172.21.15.116:0/4139866829 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7096+0+0 (secure 0 0 0) 0x7f52b4086b70 con 0x7f52ac004d50 2024-04-25T00:04:48.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.599+0000 7f52c3fff640 1 --2- 172.21.15.116:0/4139866829 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f52b00678f0 0x7f52b0069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:48.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.599+0000 7f52c3fff640 1 --2- 172.21.15.116:0/4139866829 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f52b00678f0 0x7f52b0069db0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f52c40a8410 tx=0x7f52b8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:48.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.603+0000 7f52c17fa640 1 -- 172.21.15.116:0/4139866829 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f52b4019030 con 0x7f52ac004d50 2024-04-25T00:04:48.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.827+0000 7f52cae1e640 1 -- 172.21.15.116:0/4139866829 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f52c4074980 con 0x7f52ac004d50 2024-04-25T00:04:48.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.827+0000 7f52c17fa640 1 -- 172.21.15.116:0/4139866829 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+5673 (secure 0 0 0) 0x7f52b40571d0 con 0x7f52ac004d50 2024-04-25T00:04:48.829 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:04:48.829 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:04:38.435723+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4879},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4879":{"gid":4879,"name":"b","rank":0,"incarnation":32,"state":"up:active","state_seq":15,"addr":"172.21.15.174:6837/1889314958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1889314958},{"type":"v1","addr":"172.21.15.174:6837","nonce":1889314958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4879,"qdb_cluster":[4879]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:04:48.496838+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5571},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5571":{"gid":5571,"name":"c","rank":0,"incarnation":37,"state":"up:active","state_seq":4,"addr":"172.21.15.116:6833/3841609112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3841609112},{"type":"v1","addr":"172.21.15.116:6833","nonce":3841609112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5571,"qdb_cluster":[5571]},"id":2}]} 2024-04-25T00:04:48.829 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 40 2024-04-25T00:04:48.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.831+0000 7f52cae1e640 1 -- 172.21.15.116:0/4139866829 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f52b00678f0 msgr2=0x7f52b0069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:48.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.831+0000 7f52cae1e640 1 --2- 172.21.15.116:0/4139866829 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f52b00678f0 0x7f52b0069db0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f52c40a8410 tx=0x7f52b8009040 comp rx=0 tx=0).stop 2024-04-25T00:04:48.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.831+0000 7f52cae1e640 1 -- 172.21.15.116:0/4139866829 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52ac004d50 msgr2=0x7f52c41e74c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:48.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.831+0000 7f52cae1e640 1 --2- 172.21.15.116:0/4139866829 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52ac004d50 0x7f52c41e74c0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f52b4011640 tx=0x7f52b4011b00 comp rx=0 tx=0).stop 2024-04-25T00:04:48.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.831+0000 7f52cae1e640 1 -- 172.21.15.116:0/4139866829 shutdown_connections 2024-04-25T00:04:48.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.831+0000 7f52cae1e640 1 --2- 172.21.15.116:0/4139866829 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f52b00678f0 0x7f52b0069db0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:48.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.831+0000 7f52cae1e640 1 --2- 172.21.15.116:0/4139866829 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52c40bf9f0 0x7f52c40bfe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:48.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.831+0000 7f52cae1e640 1 --2- 172.21.15.116:0/4139866829 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f52c41579b0 0x7f52c4156750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:48.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.831+0000 7f52cae1e640 1 --2- 172.21.15.116:0/4139866829 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52ac004d50 0x7f52c41e74c0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:48.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.831+0000 7f52cae1e640 1 -- 172.21.15.116:0/4139866829 >> 172.21.15.116:0/4139866829 conn(0x7f52c40c42e0 msgr2=0x7f52c4152de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:04:48.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.831+0000 7f52cae1e640 1 -- 172.21.15.116:0/4139866829 shutdown_connections 2024-04-25T00:04:48.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:48.831+0000 7f52cae1e640 1 -- 172.21.15.116:0/4139866829 wait complete. 2024-04-25T00:04:48.849 INFO:tasks.mds_thrash.fs.[b]:mds.c has gained rank=0, replacing gid=4735 2024-04-25T00:04:48.849 INFO:tasks.mds_thrash.fs.[b]:waiting for 4 secs before reviving mds.a 2024-04-25T00:04:52.850 INFO:tasks.mds_thrash.fs.[b]:reviving mds.a 2024-04-25T00:04:52.851 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-25T00:04:52.851 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-25T00:04:52.853 INFO:tasks.ceph.mds.a:Started 2024-04-25T00:04:52.854 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:04:52.930 INFO:tasks.ceph.mds.a.smithi116.stdout:starting mds.a at 2024-04-25T00:04:52.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.979+0000 7f88f611a640 1 Processor -- start 2024-04-25T00:04:52.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.979+0000 7f88f611a640 1 -- start start 2024-04-25T00:04:52.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.979+0000 7f88f611a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88f0152d60 0x7f88f00ab250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:52.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.979+0000 7f88f611a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f88f00ab790 0x7f88f00c8450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:52.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.979+0000 7f88f611a640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88f0156760 con 0x7f88f0153140 2024-04-25T00:04:52.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.979+0000 7f88f611a640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f88f0155be0 con 0x7f88f00ab790 2024-04-25T00:04:52.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.979+0000 7f88f611a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f88f0155ec0 con 0x7f88f0152d60 2024-04-25T00:04:52.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.979+0000 7f88effff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f88f0153140 0x7f88f00c3c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:40198/0 (socket says 172.21.15.116:40198) 2024-04-25T00:04:52.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.983+0000 7f88effff640 1 -- 172.21.15.116:0/3437561414 learned_addr learned my addr 172.21.15.116:0/3437561414 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:04:52.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.983+0000 7f88e7fff640 1 --2- 172.21.15.116:0/3437561414 >> v2:172.21.15.116:3300/0 conn(0x7f88f00ab790 0x7f88f00c8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:52.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.983+0000 7f88ef7fe640 1 --2- 172.21.15.116:0/3437561414 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88f0152d60 0x7f88f00ab250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:52.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.983+0000 7f88e7fff640 1 -- 172.21.15.116:0/3437561414 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88f0152d60 msgr2=0x7f88f00ab250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:04:52.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.983+0000 7f88e7fff640 1 --2- 172.21.15.116:0/3437561414 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88f0152d60 0x7f88f00ab250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:52.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.983+0000 7f88e7fff640 1 -- 172.21.15.116:0/3437561414 >> v1:172.21.15.116:6789/0 conn(0x7f88f0153140 legacy=0x7f88f00c3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:52.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.983+0000 7f88eeffd640 1 -- 172.21.15.116:0/3437561414 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 730725201 0 0) 0x7f88f0156760 con 0x7f88f0153140 2024-04-25T00:04:52.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.983+0000 7f88e7fff640 1 -- 172.21.15.116:0/3437561414 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88f01561a0 con 0x7f88f00ab790 2024-04-25T00:04:52.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.983+0000 7f88e7fff640 1 --2- 172.21.15.116:0/3437561414 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88f00ab790 0x7f88f00c8450 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f88d80097c0 tx=0x7f88d801c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38df80bdcf7d2e8f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:52.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.983+0000 7f88eeffd640 1 -- 172.21.15.116:0/3437561414 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d80b0070 con 0x7f88f00ab790 2024-04-25T00:04:52.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.983+0000 7f88eeffd640 1 -- 172.21.15.116:0/3437561414 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88d80ab600 con 0x7f88f00ab790 2024-04-25T00:04:52.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.983+0000 7f88eeffd640 1 -- 172.21.15.116:0/3437561414 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d80ab9d0 con 0x7f88f00ab790 2024-04-25T00:04:52.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.983+0000 7f88f611a640 1 -- 172.21.15.116:0/3437561414 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88f00ab790 msgr2=0x7f88f00c8450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:52.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.983+0000 7f88f611a640 1 --2- 172.21.15.116:0/3437561414 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88f00ab790 0x7f88f00c8450 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f88d80097c0 tx=0x7f88d801c410 comp rx=0 tx=0).stop 2024-04-25T00:04:52.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.983+0000 7f88f611a640 1 -- 172.21.15.116:0/3437561414 shutdown_connections 2024-04-25T00:04:52.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.983+0000 7f88f611a640 1 --2- 172.21.15.116:0/3437561414 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88f00ab790 0x7f88f00c8450 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:52.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.983+0000 7f88f611a640 1 --2- 172.21.15.116:0/3437561414 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88f0152d60 0x7f88f00ab250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:52.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.983+0000 7f88f611a640 1 -- 172.21.15.116:0/3437561414 >> 172.21.15.116:0/3437561414 conn(0x7f88f006f0c0 msgr2=0x7f88f006f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:04:52.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.987+0000 7f88f611a640 1 -- 172.21.15.116:0/3437561414 shutdown_connections 2024-04-25T00:04:52.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.987+0000 7f88f611a640 1 -- 172.21.15.116:0/3437561414 wait complete. 2024-04-25T00:04:52.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.987+0000 7f88f611a640 1 Processor -- start 2024-04-25T00:04:52.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.987+0000 7f88f611a640 1 -- start start 2024-04-25T00:04:52.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.987+0000 7f88f611a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88f00ab790 0x7f88f00abfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:52.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.987+0000 7f88f611a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88f0152d60 0x7f88f014cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:52.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.987+0000 7f88f611a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88f0153140 0x7f88f01f1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:52.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.987+0000 7f88f611a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f88f01629a0 con 0x7f88f00ab790 2024-04-25T00:04:52.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.987+0000 7f88f611a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f88f00c49c0 con 0x7f88f0153140 2024-04-25T00:04:52.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.987+0000 7f88f611a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f88f0162b20 con 0x7f88f0152d60 2024-04-25T00:04:52.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.987+0000 7f88ef7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88f00ab790 0x7f88f00abfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:52.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.987+0000 7f88ef7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88f00ab790 0x7f88f00abfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57974/0 (socket says 172.21.15.116:57974) 2024-04-25T00:04:52.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.987+0000 7f88ef7fe640 1 -- 172.21.15.116:0/1649831633 learned_addr learned my addr 172.21.15.116:0/1649831633 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:04:52.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.987+0000 7f88effff640 1 --2- 172.21.15.116:0/1649831633 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88f0153140 0x7f88f01f1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:52.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.987+0000 7f88e7fff640 1 --2- 172.21.15.116:0/1649831633 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88f0152d60 0x7f88f014cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:52.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.987+0000 7f88ef7fe640 1 -- 172.21.15.116:0/1649831633 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88f0152d60 msgr2=0x7f88f014cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:52.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.987+0000 7f88ef7fe640 1 --2- 172.21.15.116:0/1649831633 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88f0152d60 0x7f88f014cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:52.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.987+0000 7f88ef7fe640 1 -- 172.21.15.116:0/1649831633 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88f0153140 msgr2=0x7f88f01f1cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:52.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.987+0000 7f88ef7fe640 1 --2- 172.21.15.116:0/1649831633 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88f0153140 0x7f88f01f1cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:52.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.987+0000 7f88ef7fe640 1 -- 172.21.15.116:0/1649831633 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88f00ac510 con 0x7f88f00ab790 2024-04-25T00:04:52.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.987+0000 7f88effff640 1 --2- 172.21.15.116:0/1649831633 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88f0153140 0x7f88f01f1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:04:52.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.987+0000 7f88ef7fe640 1 --2- 172.21.15.116:0/1649831633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88f00ab790 0x7f88f00abfd0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f88e000d800 tx=0x7f88e000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:52.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.987+0000 7f88ecff9640 1 -- 172.21.15.116:0/1649831633 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e000b6c0 con 0x7f88f00ab790 2024-04-25T00:04:52.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.987+0000 7f88ecff9640 1 -- 172.21.15.116:0/1649831633 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88e0019070 con 0x7f88f00ab790 2024-04-25T00:04:52.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.987+0000 7f88f611a640 1 -- 172.21.15.116:0/1649831633 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88f0145370 con 0x7f88f00ab790 2024-04-25T00:04:52.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.987+0000 7f88ecff9640 1 -- 172.21.15.116:0/1649831633 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e000ba60 con 0x7f88f00ab790 2024-04-25T00:04:52.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.987+0000 7f88f611a640 1 -- 172.21.15.116:0/1649831633 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88f014ea20 con 0x7f88f00ab790 2024-04-25T00:04:52.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.991+0000 7f88e5ffb640 1 -- 172.21.15.116:0/1649831633 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88c80052a0 con 0x7f88f00ab790 2024-04-25T00:04:52.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.991+0000 7f88ecff9640 1 -- 172.21.15.116:0/1649831633 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f88e000bc00 con 0x7f88f00ab790 2024-04-25T00:04:52.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.991+0000 7f88ecff9640 1 --2- 172.21.15.116:0/1649831633 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f88d00678c0 0x7f88d0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:52.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.991+0000 7f88ecff9640 1 -- 172.21.15.116:0/1649831633 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7096+0+0 (secure 0 0 0) 0x7f88e0086640 con 0x7f88f00ab790 2024-04-25T00:04:52.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.991+0000 7f88e7fff640 1 --2- 172.21.15.116:0/1649831633 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f88d00678c0 0x7f88d0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:52.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.991+0000 7f88e7fff640 1 --2- 172.21.15.116:0/1649831633 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f88d00678c0 0x7f88d0069d80 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f88d80ad000 tx=0x7f88d8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:52.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:52.995+0000 7f88ecff9640 1 -- 172.21.15.116:0/1649831633 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f88e0056ca0 con 0x7f88f00ab790 2024-04-25T00:04:53.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:53.239+0000 7f88e5ffb640 1 -- 172.21.15.116:0/1649831633 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f88c8005860 con 0x7f88f00ab790 2024-04-25T00:04:53.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:53.239+0000 7f88ecff9640 1 -- 172.21.15.116:0/1649831633 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+5673 (secure 0 0 0) 0x7f88e00565a0 con 0x7f88f00ab790 2024-04-25T00:04:53.241 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:04:53.241 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:04:38.435723+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4879},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4879":{"gid":4879,"name":"b","rank":0,"incarnation":32,"state":"up:active","state_seq":15,"addr":"172.21.15.174:6837/1889314958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1889314958},{"type":"v1","addr":"172.21.15.174:6837","nonce":1889314958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4879,"qdb_cluster":[4879]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:04:48.496838+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5571},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5571":{"gid":5571,"name":"c","rank":0,"incarnation":37,"state":"up:active","state_seq":4,"addr":"172.21.15.116:6833/3841609112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3841609112},{"type":"v1","addr":"172.21.15.116:6833","nonce":3841609112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5571,"qdb_cluster":[5571]},"id":2}]} 2024-04-25T00:04:53.241 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 40 2024-04-25T00:04:53.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:53.243+0000 7f88e5ffb640 1 -- 172.21.15.116:0/1649831633 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f88d00678c0 msgr2=0x7f88d0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:53.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:53.243+0000 7f88e5ffb640 1 --2- 172.21.15.116:0/1649831633 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f88d00678c0 0x7f88d0069d80 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f88d80ad000 tx=0x7f88d8008040 comp rx=0 tx=0).stop 2024-04-25T00:04:53.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:53.243+0000 7f88e5ffb640 1 -- 172.21.15.116:0/1649831633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88f00ab790 msgr2=0x7f88f00abfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:53.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:53.243+0000 7f88e5ffb640 1 --2- 172.21.15.116:0/1649831633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88f00ab790 0x7f88f00abfd0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f88e000d800 tx=0x7f88e000dcc0 comp rx=0 tx=0).stop 2024-04-25T00:04:53.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:53.243+0000 7f88e5ffb640 1 -- 172.21.15.116:0/1649831633 shutdown_connections 2024-04-25T00:04:53.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:53.243+0000 7f88e5ffb640 1 --2- 172.21.15.116:0/1649831633 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f88d00678c0 0x7f88d0069d80 secure :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f88d80ad000 tx=0x7f88d8008040 comp rx=0 tx=0).stop 2024-04-25T00:04:53.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:53.243+0000 7f88e5ffb640 1 --2- 172.21.15.116:0/1649831633 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88f0153140 0x7f88f01f1cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:53.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:53.243+0000 7f88e5ffb640 1 --2- 172.21.15.116:0/1649831633 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88f0152d60 0x7f88f014cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:53.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:53.243+0000 7f88e5ffb640 1 --2- 172.21.15.116:0/1649831633 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88f00ab790 0x7f88f00abfd0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:53.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:53.243+0000 7f88e5ffb640 1 -- 172.21.15.116:0/1649831633 >> 172.21.15.116:0/1649831633 conn(0x7f88f006f0c0 msgr2=0x7f88f00c9260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:04:53.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:53.243+0000 7f88e5ffb640 1 -- 172.21.15.116:0/1649831633 shutdown_connections 2024-04-25T00:04:53.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:53.243+0000 7f88e5ffb640 1 -- 172.21.15.116:0/1649831633 wait complete. 2024-04-25T00:04:53.260 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2024-04-25T00:04:53.459 INFO:tasks.ceph.mds.c.smithi116.stderr:2024-04-25T00:04:53.459+0000 7ff52ea13640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T00:04:55.261 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:04:55.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.403+0000 7fd38c6b1640 1 Processor -- start 2024-04-25T00:04:55.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.403+0000 7fd38c6b1640 1 -- start start 2024-04-25T00:04:55.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.403+0000 7fd38c6b1640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fd3840c3c90 0x7fd3840c4100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:55.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.403+0000 7fd38c6b1640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd384156760 con 0x7fd384153140 2024-04-25T00:04:55.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.403+0000 7fd38c6b1640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd384155be0 con 0x7fd384152d60 2024-04-25T00:04:55.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.403+0000 7fd38c6b1640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fd384155ec0 con 0x7fd3840c3c90 2024-04-25T00:04:55.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.403+0000 7fd389c4c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fd384153140 0x7fd3840c8450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:51240/0 (socket says 172.21.15.116:51240) 2024-04-25T00:04:55.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.403+0000 7fd389c4c640 1 -- 172.21.15.116:0/1707398914 learned_addr learned my addr 172.21.15.116:0/1707398914 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:04:55.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.403+0000 7fd38ac4e640 1 --2- 172.21.15.116:0/1707398914 >> v2:172.21.15.174:3300/0 conn(0x7fd3840c3c90 0x7fd3840c4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:55.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.403+0000 7fd38944b640 1 -- 172.21.15.116:0/1707398914 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 831134468 0 0) 0x7fd384156760 con 0x7fd384153140 2024-04-25T00:04:55.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.403+0000 7fd38944b640 1 -- 172.21.15.116:0/1707398914 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd370003530 con 0x7fd384153140 2024-04-25T00:04:55.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.403+0000 7fd38944b640 1 -- 172.21.15.116:0/1707398914 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1829554568 0 0) 0x7fd370003530 con 0x7fd384153140 2024-04-25T00:04:55.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.403+0000 7fd38944b640 1 -- 172.21.15.116:0/1707398914 >> v2:172.21.15.174:3300/0 conn(0x7fd3840c3c90 msgr2=0x7fd3840c4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:04:55.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.403+0000 7fd38944b640 1 --2- 172.21.15.116:0/1707398914 >> v2:172.21.15.174:3300/0 conn(0x7fd3840c3c90 0x7fd3840c4100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:55.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.403+0000 7fd38944b640 1 -- 172.21.15.116:0/1707398914 >> v1:172.21.15.174:6789/0 conn(0x7fd384152d60 legacy=0x7fd3840ab250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:55.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.403+0000 7fd38944b640 1 -- 172.21.15.116:0/1707398914 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3841561a0 con 0x7fd384153140 2024-04-25T00:04:55.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.403+0000 7fd38944b640 1 -- 172.21.15.116:0/1707398914 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7fd380001680 con 0x7fd384153140 2024-04-25T00:04:55.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.403+0000 7fd38944b640 1 -- 172.21.15.116:0/1707398914 >> v1:172.21.15.116:6789/0 conn(0x7fd384153140 legacy=0x7fd3840c8450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:55.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.403+0000 7fd38944b640 1 --2- 172.21.15.116:0/1707398914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3700039c0 0x7fd370003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:55.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.403+0000 7fd38944b640 1 --2- 172.21.15.116:0/1707398914 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd370004810 0x7fd370004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:55.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.403+0000 7fd38944b640 1 --2- 172.21.15.116:0/1707398914 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3700051d0 0x7fd37000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:55.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.403+0000 7fd38944b640 1 -- 172.21.15.116:0/1707398914 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd384155ec0 con 0x7fd3700039c0 2024-04-25T00:04:55.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.403+0000 7fd38944b640 1 -- 172.21.15.116:0/1707398914 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd37000c170 con 0x7fd370004810 2024-04-25T00:04:55.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.403+0000 7fd38944b640 1 -- 172.21.15.116:0/1707398914 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fd37000c370 con 0x7fd3700051d0 2024-04-25T00:04:55.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.403+0000 7fd38a44d640 1 --2- 172.21.15.116:0/1707398914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3700039c0 0x7fd370003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:55.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.403+0000 7fd389c4c640 1 --2- 172.21.15.116:0/1707398914 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd370004810 0x7fd370004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:55.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.403+0000 7fd38ac4e640 1 --2- 172.21.15.116:0/1707398914 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3700051d0 0x7fd37000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:55.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.403+0000 7fd38a44d640 1 -- 172.21.15.116:0/1707398914 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3700051d0 msgr2=0x7fd37000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:04:55.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.403+0000 7fd38a44d640 1 --2- 172.21.15.116:0/1707398914 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3700051d0 0x7fd37000b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:55.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.403+0000 7fd38a44d640 1 -- 172.21.15.116:0/1707398914 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd370004810 msgr2=0x7fd370004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:04:55.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.403+0000 7fd38a44d640 1 --2- 172.21.15.116:0/1707398914 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd370004810 0x7fd370004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:55.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.403+0000 7fd38a44d640 1 -- 172.21.15.116:0/1707398914 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd370003530 con 0x7fd3700039c0 2024-04-25T00:04:55.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.403+0000 7fd389c4c640 1 --2- 172.21.15.116:0/1707398914 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd370004810 0x7fd370004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:04:55.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.403+0000 7fd38ac4e640 1 -- 172.21.15.116:0/1707398914 reap_dead start 2024-04-25T00:04:55.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.407+0000 7fd38a44d640 1 --2- 172.21.15.116:0/1707398914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3700039c0 0x7fd370003de0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fd374001780 tx=0x7fd37401c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6c106563a0fc3c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:55.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.407+0000 7fd38944b640 1 -- 172.21.15.116:0/1707398914 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3740013a0 con 0x7fd3700039c0 2024-04-25T00:04:55.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.407+0000 7fd38944b640 1 -- 172.21.15.116:0/1707398914 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3840aba60 con 0x7fd3700039c0 2024-04-25T00:04:55.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.407+0000 7fd38944b640 1 -- 172.21.15.116:0/1707398914 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd374003600 con 0x7fd3700039c0 2024-04-25T00:04:55.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.407+0000 7fd38c6b1640 1 -- 172.21.15.116:0/1707398914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3700039c0 msgr2=0x7fd370003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:55.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.407+0000 7fd38c6b1640 1 --2- 172.21.15.116:0/1707398914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3700039c0 0x7fd370003de0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fd374001780 tx=0x7fd37401c400 comp rx=0 tx=0).stop 2024-04-25T00:04:55.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.407+0000 7fd38c6b1640 1 -- 172.21.15.116:0/1707398914 shutdown_connections 2024-04-25T00:04:55.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.407+0000 7fd38c6b1640 1 --2- 172.21.15.116:0/1707398914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3700039c0 0x7fd370003de0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:55.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.407+0000 7fd38c6b1640 1 -- 172.21.15.116:0/1707398914 >> 172.21.15.116:0/1707398914 conn(0x7fd38406f0c0 msgr2=0x7fd38406f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:04:55.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.407+0000 7fd38c6b1640 1 -- 172.21.15.116:0/1707398914 shutdown_connections 2024-04-25T00:04:55.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.407+0000 7fd38c6b1640 1 -- 172.21.15.116:0/1707398914 wait complete. 2024-04-25T00:04:55.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.407+0000 7fd38c6b1640 1 Processor -- start 2024-04-25T00:04:55.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.407+0000 7fd38c6b1640 1 -- start start 2024-04-25T00:04:55.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.411+0000 7fd38c6b1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd3700039c0 0x7fd38406dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:55.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.411+0000 7fd38c6b1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd384146020 0x7fd3841464a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:55.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.411+0000 7fd38c6b1640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3841ee690 0x7fd3841eeb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:55.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.411+0000 7fd38c6b1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd3840c87e0 con 0x7fd384146020 2024-04-25T00:04:55.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.411+0000 7fd38c6b1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd3840c8660 con 0x7fd3700039c0 2024-04-25T00:04:55.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.411+0000 7fd38c6b1640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fd3840c8960 con 0x7fd3841ee690 2024-04-25T00:04:55.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.411+0000 7fd389c4c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd384146020 0x7fd3841464a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:55.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.411+0000 7fd389c4c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd384146020 0x7fd3841464a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44510/0 (socket says 172.21.15.116:44510) 2024-04-25T00:04:55.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.411+0000 7fd389c4c640 1 -- 172.21.15.116:0/2564825612 learned_addr learned my addr 172.21.15.116:0/2564825612 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:04:55.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.411+0000 7fd38a44d640 1 --2- 172.21.15.116:0/2564825612 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd3700039c0 0x7fd38406dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:55.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.411+0000 7fd38ac4e640 1 --2- 172.21.15.116:0/2564825612 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3841ee690 0x7fd3841eeb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:55.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.411+0000 7fd389c4c640 1 -- 172.21.15.116:0/2564825612 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3841ee690 msgr2=0x7fd3841eeb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:55.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.411+0000 7fd389c4c640 1 --2- 172.21.15.116:0/2564825612 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3841ee690 0x7fd3841eeb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:55.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.411+0000 7fd389c4c640 1 -- 172.21.15.116:0/2564825612 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd3700039c0 msgr2=0x7fd38406dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:55.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.411+0000 7fd389c4c640 1 --2- 172.21.15.116:0/2564825612 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd3700039c0 0x7fd38406dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:55.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.411+0000 7fd389c4c640 1 -- 172.21.15.116:0/2564825612 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd384149d50 con 0x7fd384146020 2024-04-25T00:04:55.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.411+0000 7fd38ac4e640 1 --2- 172.21.15.116:0/2564825612 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3841ee690 0x7fd3841eeb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:04:55.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.411+0000 7fd38a44d640 1 --2- 172.21.15.116:0/2564825612 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd3700039c0 0x7fd38406dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:04:55.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.411+0000 7fd389c4c640 1 --2- 172.21.15.116:0/2564825612 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd384146020 0x7fd3841464a0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fd380001ec0 tx=0x7fd38000ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:55.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.411+0000 7fd37affd640 1 -- 172.21.15.116:0/2564825612 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd38000c690 con 0x7fd384146020 2024-04-25T00:04:55.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.411+0000 7fd37affd640 1 -- 172.21.15.116:0/2564825612 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd380018070 con 0x7fd384146020 2024-04-25T00:04:55.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.411+0000 7fd37affd640 1 -- 172.21.15.116:0/2564825612 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd38000ca30 con 0x7fd384146020 2024-04-25T00:04:55.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.411+0000 7fd38c6b1640 1 -- 172.21.15.116:0/2564825612 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3840ac170 con 0x7fd384146020 2024-04-25T00:04:55.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.411+0000 7fd38c6b1640 1 -- 172.21.15.116:0/2564825612 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3840ade30 con 0x7fd384146020 2024-04-25T00:04:55.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.411+0000 7fd38c6b1640 1 -- 172.21.15.116:0/2564825612 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd384068a60 con 0x7fd384146020 2024-04-25T00:04:55.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.411+0000 7fd37affd640 1 -- 172.21.15.116:0/2564825612 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd38000cbd0 con 0x7fd384146020 2024-04-25T00:04:55.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.415+0000 7fd37affd640 1 --2- 172.21.15.116:0/2564825612 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd3680678c0 0x7fd368069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:04:55.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.415+0000 7fd37affd640 1 -- 172.21.15.116:0/2564825612 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7096+0+0 (secure 0 0 0) 0x7fd380086960 con 0x7fd384146020 2024-04-25T00:04:55.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.415+0000 7fd38a44d640 1 --2- 172.21.15.116:0/2564825612 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd3680678c0 0x7fd368069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:04:55.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.415+0000 7fd38a44d640 1 --2- 172.21.15.116:0/2564825612 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd3680678c0 0x7fd368069d80 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fd374004000 tx=0x7fd3740011a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:04:55.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.419+0000 7fd37affd640 1 -- 172.21.15.116:0/2564825612 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fd384149d50 con 0x7fd384146020 2024-04-25T00:04:55.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.655+0000 7fd38c6b1640 1 -- 172.21.15.116:0/2564825612 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd384155460 con 0x7fd384146020 2024-04-25T00:04:55.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.655+0000 7fd37affd640 1 -- 172.21.15.116:0/2564825612 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+6501 (secure 0 0 0) 0x7fd380057010 con 0x7fd384146020 2024-04-25T00:04:55.656 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:04:55.656 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1233829678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1233829678},{"type":"v1","addr":"172.21.15.116:6835","nonce":1233829678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:04:38.435723+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4879},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4879":{"gid":4879,"name":"b","rank":0,"incarnation":32,"state":"up:active","state_seq":15,"addr":"172.21.15.174:6837/1889314958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1889314958},{"type":"v1","addr":"172.21.15.174:6837","nonce":1889314958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4879,"qdb_cluster":[4879]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:04:48.496838+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5571},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5571":{"gid":5571,"name":"c","rank":0,"incarnation":37,"state":"up:active","state_seq":4,"addr":"172.21.15.116:6833/3841609112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3841609112},{"type":"v1","addr":"172.21.15.116:6833","nonce":3841609112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5571,"qdb_cluster":[5571]},"id":2}]} 2024-04-25T00:04:55.657 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 41 2024-04-25T00:04:55.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.659+0000 7fd38c6b1640 1 -- 172.21.15.116:0/2564825612 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd3680678c0 msgr2=0x7fd368069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:55.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.659+0000 7fd38c6b1640 1 --2- 172.21.15.116:0/2564825612 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd3680678c0 0x7fd368069d80 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fd374004000 tx=0x7fd3740011a0 comp rx=0 tx=0).stop 2024-04-25T00:04:55.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.659+0000 7fd38c6b1640 1 -- 172.21.15.116:0/2564825612 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd384146020 msgr2=0x7fd3841464a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:04:55.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.659+0000 7fd38c6b1640 1 --2- 172.21.15.116:0/2564825612 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd384146020 0x7fd3841464a0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fd380001ec0 tx=0x7fd38000ec10 comp rx=0 tx=0).stop 2024-04-25T00:04:55.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.659+0000 7fd38c6b1640 1 -- 172.21.15.116:0/2564825612 shutdown_connections 2024-04-25T00:04:55.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.659+0000 7fd38c6b1640 1 --2- 172.21.15.116:0/2564825612 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd3680678c0 0x7fd368069d80 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:55.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.659+0000 7fd38c6b1640 1 --2- 172.21.15.116:0/2564825612 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3841ee690 0x7fd3841eeb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:55.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.659+0000 7fd38c6b1640 1 --2- 172.21.15.116:0/2564825612 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd384146020 0x7fd3841464a0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:55.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.659+0000 7fd38c6b1640 1 --2- 172.21.15.116:0/2564825612 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd3700039c0 0x7fd38406dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:04:55.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.659+0000 7fd38c6b1640 1 -- 172.21.15.116:0/2564825612 >> 172.21.15.116:0/2564825612 conn(0x7fd38406f0c0 msgr2=0x7fd3840c9440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:04:55.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.659+0000 7fd38c6b1640 1 -- 172.21.15.116:0/2564825612 shutdown_connections 2024-04-25T00:04:55.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:04:55.659+0000 7fd38c6b1640 1 -- 172.21.15.116:0/2564825612 wait complete. 2024-04-25T00:04:55.677 INFO:tasks.mds_thrash.fs.[b]:mds.a reported in up:standby state 2024-04-25T00:04:55.677 INFO:tasks.mds_thrash.fs.[b]:waiting for 110 secs before thrashing 2024-04-25T00:05:05.134 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:05:05.137 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:05:35.332 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:05:35.336 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:06:05.497 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:06:05.501 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:06:13.553 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:06:13.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.682+0000 7fd3bd2df640 1 Processor -- start 2024-04-25T00:06:13.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.682+0000 7fd3bd2df640 1 -- start start 2024-04-25T00:06:13.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.682+0000 7fd3bd2df640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3b8152d80 0x7fd3b806bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:13.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.682+0000 7fd3bd2df640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fd3b806d5f0 0x7fd3b8156c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:13.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.682+0000 7fd3bd2df640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3b80c9150 con 0x7fd3b806c290 2024-04-25T00:06:13.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.682+0000 7fd3bd2df640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fd3b80c9d60 con 0x7fd3b806d5f0 2024-04-25T00:06:13.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.682+0000 7fd3bd2df640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fd3b80c9430 con 0x7fd3b8152d80 2024-04-25T00:06:13.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.682+0000 7fd3b6ffd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3b8152d80 0x7fd3b806bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:13.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.682+0000 7fd3b67fc640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fd3b806d5f0 0x7fd3b8156c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:13.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.682+0000 7fd3b77fe640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fd3b806c290 0x7fd3b81571c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:60310/0 (socket says 172.21.15.116:60310) 2024-04-25T00:06:13.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.682+0000 7fd3b67fc640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fd3b806d5f0 0x7fd3b8156c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:40940/0 (socket says 172.21.15.116:40940) 2024-04-25T00:06:13.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.682+0000 7fd3b6ffd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3b8152d80 0x7fd3b806bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:40966/0 (socket says 172.21.15.116:40966) 2024-04-25T00:06:13.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.682+0000 7fd3b77fe640 1 -- 172.21.15.116:0/1998583293 learned_addr learned my addr 172.21.15.116:0/1998583293 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:06:13.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.682+0000 7fd3b67fc640 1 -- 172.21.15.116:0/1998583293 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3b8152d80 msgr2=0x7fd3b806bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:06:13.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.682+0000 7fd3b67fc640 1 --2- 172.21.15.116:0/1998583293 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3b8152d80 0x7fd3b806bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:13.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.682+0000 7fd3b67fc640 1 -- 172.21.15.116:0/1998583293 >> v1:172.21.15.174:6789/0 conn(0x7fd3b806c290 legacy=0x7fd3b81571c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:13.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.682+0000 7fd3b67fc640 1 -- 172.21.15.116:0/1998583293 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3b80c9710 con 0x7fd3b806d5f0 2024-04-25T00:06:13.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.686+0000 7fd3b67fc640 1 --2- 172.21.15.116:0/1998583293 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd3b806d5f0 0x7fd3b8156c80 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac0097c0 tx=0x7fd3ac01c3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63c78b847414d065 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:13.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.686+0000 7fd3b5ffb640 1 -- 172.21.15.116:0/1998583293 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3ac0ab600 con 0x7fd3b806d5f0 2024-04-25T00:06:13.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.686+0000 7fd3b5ffb640 1 -- 172.21.15.116:0/1998583293 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3ac0b0070 con 0x7fd3b806d5f0 2024-04-25T00:06:13.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.686+0000 7fd3b5ffb640 1 -- 172.21.15.116:0/1998583293 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3ac0ab9c0 con 0x7fd3b806d5f0 2024-04-25T00:06:13.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.686+0000 7fd3bd2df640 1 -- 172.21.15.116:0/1998583293 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd3b806d5f0 msgr2=0x7fd3b8156c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:13.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.686+0000 7fd3bd2df640 1 --2- 172.21.15.116:0/1998583293 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd3b806d5f0 0x7fd3b8156c80 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac0097c0 tx=0x7fd3ac01c3e0 comp rx=0 tx=0).stop 2024-04-25T00:06:13.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.686+0000 7fd3bd2df640 1 -- 172.21.15.116:0/1998583293 shutdown_connections 2024-04-25T00:06:13.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.686+0000 7fd3bd2df640 1 --2- 172.21.15.116:0/1998583293 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd3b806d5f0 0x7fd3b8156c80 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:13.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.686+0000 7fd3bd2df640 1 --2- 172.21.15.116:0/1998583293 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3b8152d80 0x7fd3b806bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:13.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.686+0000 7fd3bd2df640 1 -- 172.21.15.116:0/1998583293 >> 172.21.15.116:0/1998583293 conn(0x7fd3b80acdd0 msgr2=0x7fd3b80ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:13.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.686+0000 7fd3bd2df640 1 -- 172.21.15.116:0/1998583293 shutdown_connections 2024-04-25T00:06:13.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.690+0000 7fd3bd2df640 1 -- 172.21.15.116:0/1998583293 wait complete. 2024-04-25T00:06:13.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.690+0000 7fd3bd2df640 1 Processor -- start 2024-04-25T00:06:13.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.690+0000 7fd3bd2df640 1 -- start start 2024-04-25T00:06:13.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.690+0000 7fd3bd2df640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3b806c290 0x7fd3b8151860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:13.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.690+0000 7fd3bd2df640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd3b806d5f0 0x7fd3b80aea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:13.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.690+0000 7fd3bd2df640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3b8152d80 0x7fd3b80aef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:13.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.690+0000 7fd3bd2df640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd3b80caf00 con 0x7fd3b806c290 2024-04-25T00:06:13.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.690+0000 7fd3bd2df640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd3b80cad80 con 0x7fd3b806d5f0 2024-04-25T00:06:13.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.690+0000 7fd3bd2df640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fd3b81628d0 con 0x7fd3b8152d80 2024-04-25T00:06:13.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.690+0000 7fd3b6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3b806c290 0x7fd3b8151860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:13.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.690+0000 7fd3b6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3b806c290 0x7fd3b8151860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53352/0 (socket says 172.21.15.116:53352) 2024-04-25T00:06:13.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.690+0000 7fd3b6ffd640 1 -- 172.21.15.116:0/3188631554 learned_addr learned my addr 172.21.15.116:0/3188631554 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:06:13.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.690+0000 7fd3b67fc640 1 --2- 172.21.15.116:0/3188631554 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd3b806d5f0 0x7fd3b80aea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:13.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.690+0000 7fd3b77fe640 1 --2- 172.21.15.116:0/3188631554 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3b8152d80 0x7fd3b80aef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:13.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.690+0000 7fd3b6ffd640 1 -- 172.21.15.116:0/3188631554 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3b8152d80 msgr2=0x7fd3b80aef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:13.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.690+0000 7fd3b6ffd640 1 --2- 172.21.15.116:0/3188631554 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3b8152d80 0x7fd3b80aef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:13.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.690+0000 7fd3b6ffd640 1 -- 172.21.15.116:0/3188631554 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd3b806d5f0 msgr2=0x7fd3b80aea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:13.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.690+0000 7fd3b6ffd640 1 --2- 172.21.15.116:0/3188631554 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd3b806d5f0 0x7fd3b80aea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:13.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.690+0000 7fd3b6ffd640 1 -- 172.21.15.116:0/3188631554 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3b80c63e0 con 0x7fd3b806c290 2024-04-25T00:06:13.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.690+0000 7fd3b6ffd640 1 --2- 172.21.15.116:0/3188631554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3b806c290 0x7fd3b8151860 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fd3a800e930 tx=0x7fd3a800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:13.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.690+0000 7fd39f7fe640 1 -- 172.21.15.116:0/3188631554 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a8017070 con 0x7fd3b806c290 2024-04-25T00:06:13.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.690+0000 7fd39f7fe640 1 -- 172.21.15.116:0/3188631554 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3a800c9a0 con 0x7fd3b806c290 2024-04-25T00:06:13.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.690+0000 7fd39f7fe640 1 -- 172.21.15.116:0/3188631554 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a800cd50 con 0x7fd3b806c290 2024-04-25T00:06:13.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.694+0000 7fd3bd2df640 1 -- 172.21.15.116:0/3188631554 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3b80c66c0 con 0x7fd3b806c290 2024-04-25T00:06:13.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.694+0000 7fd3bd2df640 1 -- 172.21.15.116:0/3188631554 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3b8151da0 con 0x7fd3b806c290 2024-04-25T00:06:13.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.694+0000 7fd3bd2df640 1 -- 172.21.15.116:0/3188631554 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3b80adff0 con 0x7fd3b806c290 2024-04-25T00:06:13.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.694+0000 7fd39f7fe640 1 -- 172.21.15.116:0/3188631554 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd3a801a070 con 0x7fd3b806c290 2024-04-25T00:06:13.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.694+0000 7fd39f7fe640 1 --2- 172.21.15.116:0/3188631554 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd3a40678f0 0x7fd3a4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:13.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.694+0000 7fd39f7fe640 1 -- 172.21.15.116:0/3188631554 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7096+0+0 (secure 0 0 0) 0x7fd3a80864b0 con 0x7fd3b806c290 2024-04-25T00:06:13.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.694+0000 7fd3b67fc640 1 --2- 172.21.15.116:0/3188631554 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd3a40678f0 0x7fd3a4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:13.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.694+0000 7fd3b67fc640 1 --2- 172.21.15.116:0/3188631554 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd3a40678f0 0x7fd3a4069db0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac0af000 tx=0x7fd3ac0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:13.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.698+0000 7fd39f7fe640 1 -- 172.21.15.116:0/3188631554 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fd3a8056500 con 0x7fd3b806c290 2024-04-25T00:06:13.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.946+0000 7fd3bd2df640 1 -- 172.21.15.116:0/3188631554 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd3b8068350 con 0x7fd3b806c290 2024-04-25T00:06:13.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.946+0000 7fd39f7fe640 1 -- 172.21.15.116:0/3188631554 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+6501 (secure 0 0 0) 0x7fd3a805a800 con 0x7fd3b806c290 2024-04-25T00:06:13.950 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:06:13.950 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1233829678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1233829678},{"type":"v1","addr":"172.21.15.116:6835","nonce":1233829678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:04:38.435723+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4879},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4879":{"gid":4879,"name":"b","rank":0,"incarnation":32,"state":"up:active","state_seq":15,"addr":"172.21.15.174:6837/1889314958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1889314958},{"type":"v1","addr":"172.21.15.174:6837","nonce":1889314958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4879,"qdb_cluster":[4879]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:04:48.496838+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5571},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5571":{"gid":5571,"name":"c","rank":0,"incarnation":37,"state":"up:active","state_seq":4,"addr":"172.21.15.116:6833/3841609112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3841609112},{"type":"v1","addr":"172.21.15.116:6833","nonce":3841609112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5571,"qdb_cluster":[5571]},"id":2}]} 2024-04-25T00:06:13.950 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 41 2024-04-25T00:06:13.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.950+0000 7fd3bd2df640 1 -- 172.21.15.116:0/3188631554 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd3a40678f0 msgr2=0x7fd3a4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:13.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.950+0000 7fd3bd2df640 1 --2- 172.21.15.116:0/3188631554 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd3a40678f0 0x7fd3a4069db0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac0af000 tx=0x7fd3ac0a7000 comp rx=0 tx=0).stop 2024-04-25T00:06:13.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.950+0000 7fd3bd2df640 1 -- 172.21.15.116:0/3188631554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3b806c290 msgr2=0x7fd3b8151860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:13.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.950+0000 7fd3bd2df640 1 --2- 172.21.15.116:0/3188631554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3b806c290 0x7fd3b8151860 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fd3a800e930 tx=0x7fd3a800edf0 comp rx=0 tx=0).stop 2024-04-25T00:06:13.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.950+0000 7fd3bd2df640 1 -- 172.21.15.116:0/3188631554 shutdown_connections 2024-04-25T00:06:13.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.950+0000 7fd3bd2df640 1 --2- 172.21.15.116:0/3188631554 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd3a40678f0 0x7fd3a4069db0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:13.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.950+0000 7fd3bd2df640 1 --2- 172.21.15.116:0/3188631554 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3b8152d80 0x7fd3b80aef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:13.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.950+0000 7fd3bd2df640 1 --2- 172.21.15.116:0/3188631554 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd3b806d5f0 0x7fd3b80aea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:13.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.950+0000 7fd3bd2df640 1 --2- 172.21.15.116:0/3188631554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3b806c290 0x7fd3b8151860 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:13.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.950+0000 7fd3bd2df640 1 -- 172.21.15.116:0/3188631554 >> 172.21.15.116:0/3188631554 conn(0x7fd3b80acdd0 msgr2=0x7fd3b814de60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:13.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.950+0000 7fd3bd2df640 1 -- 172.21.15.116:0/3188631554 shutdown_connections 2024-04-25T00:06:13.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:13.950+0000 7fd3bd2df640 1 -- 172.21.15.116:0/3188631554 wait complete. 2024-04-25T00:06:13.970 INFO:tasks.mds_thrash.fs.[a]:kill mds.b (rank=0) 2024-04-25T00:06:13.971 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-25T00:06:13.971 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:06:13.992 INFO:tasks.ceph.mds.b:Stopped 2024-04-25T00:06:13.992 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:06:14.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.118+0000 7f4a7eda8640 1 Processor -- start 2024-04-25T00:06:14.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.118+0000 7f4a7eda8640 1 -- start start 2024-04-25T00:06:14.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.118+0000 7f4a7eda8640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f4a780c5280 0x7f4a78150840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:14.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.118+0000 7f4a7eda8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4a78072810 0x7f4a78153760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:14.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.118+0000 7f4a7eda8640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a780c2c40 con 0x7f4a78072c90 2024-04-25T00:06:14.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.118+0000 7f4a7eda8640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f4a78157c20 con 0x7f4a78072810 2024-04-25T00:06:14.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.118+0000 7f4a7eda8640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f4a780c2f70 con 0x7f4a780c5280 2024-04-25T00:06:14.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.122+0000 7f4a7d345640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4a78072810 0x7f4a78153760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:14.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.122+0000 7f4a7d345640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4a78072810 0x7f4a78153760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53354/0 (socket says 172.21.15.116:53354) 2024-04-25T00:06:14.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.122+0000 7f4a7d345640 1 -- 172.21.15.116:0/1426397019 learned_addr learned my addr 172.21.15.116:0/1426397019 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:06:14.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.122+0000 7f4a7d345640 1 -- 172.21.15.116:0/1426397019 >> v2:172.21.15.174:3300/0 conn(0x7f4a780c5280 msgr2=0x7f4a78150840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:06:14.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.122+0000 7f4a7d345640 1 --2- 172.21.15.116:0/1426397019 >> v2:172.21.15.174:3300/0 conn(0x7f4a780c5280 0x7f4a78150840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:14.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.122+0000 7f4a7d345640 1 -- 172.21.15.116:0/1426397019 >> v1:172.21.15.174:6789/0 conn(0x7f4a78072c90 legacy=0x7f4a78150d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:06:14.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.122+0000 7f4a7d345640 1 -- 172.21.15.116:0/1426397019 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a78157660 con 0x7f4a78072810 2024-04-25T00:06:14.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.122+0000 7f4a7d345640 1 --2- 172.21.15.116:0/1426397019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a78072810 0x7f4a78153760 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f4a780c36e0 tx=0x7f4a7001c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6de0704e1c5a197f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:14.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.122+0000 7f4a777fe640 1 -- 172.21.15.116:0/1426397019 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a700a5740 con 0x7f4a78072810 2024-04-25T00:06:14.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.122+0000 7f4a777fe640 1 -- 172.21.15.116:0/1426397019 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a700a58e0 con 0x7f4a78072810 2024-04-25T00:06:14.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.122+0000 7f4a777fe640 1 -- 172.21.15.116:0/1426397019 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a700a5cb0 con 0x7f4a78072810 2024-04-25T00:06:14.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.126+0000 7f4a7eda8640 1 -- 172.21.15.116:0/1426397019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a78072810 msgr2=0x7f4a78153760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:14.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.126+0000 7f4a7eda8640 1 --2- 172.21.15.116:0/1426397019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a78072810 0x7f4a78153760 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f4a780c36e0 tx=0x7f4a7001c860 comp rx=0 tx=0).stop 2024-04-25T00:06:14.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.126+0000 7f4a7eda8640 1 -- 172.21.15.116:0/1426397019 shutdown_connections 2024-04-25T00:06:14.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.126+0000 7f4a7eda8640 1 --2- 172.21.15.116:0/1426397019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a78072810 0x7f4a78153760 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:14.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.126+0000 7f4a7eda8640 1 --2- 172.21.15.116:0/1426397019 >> v2:172.21.15.174:3300/0 conn(0x7f4a780c5280 0x7f4a78150840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:14.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.126+0000 7f4a7eda8640 1 -- 172.21.15.116:0/1426397019 >> 172.21.15.116:0/1426397019 conn(0x7f4a7806c720 msgr2=0x7f4a7806cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:14.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.126+0000 7f4a7eda8640 1 -- 172.21.15.116:0/1426397019 shutdown_connections 2024-04-25T00:06:14.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.126+0000 7f4a7eda8640 1 -- 172.21.15.116:0/1426397019 wait complete. 2024-04-25T00:06:14.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.126+0000 7f4a7eda8640 1 Processor -- start 2024-04-25T00:06:14.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.126+0000 7f4a7eda8640 1 -- start start 2024-04-25T00:06:14.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.126+0000 7f4a7eda8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a78072810 0x7f4a781f2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:14.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.126+0000 7f4a7eda8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4a78072c90 0x7f4a780accd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:14.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.126+0000 7f4a7eda8640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4a780c5280 0x7f4a781471e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:14.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.126+0000 7f4a7eda8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4a78160380 con 0x7f4a78072810 2024-04-25T00:06:14.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.126+0000 7f4a7eda8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4a78160200 con 0x7f4a78072c90 2024-04-25T00:06:14.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.126+0000 7f4a7eda8640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f4a78160500 con 0x7f4a780c5280 2024-04-25T00:06:14.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.126+0000 7f4a7cb44640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a78072810 0x7f4a781f2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:14.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.126+0000 7f4a7cb44640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a78072810 0x7f4a781f2110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53366/0 (socket says 172.21.15.116:53366) 2024-04-25T00:06:14.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.126+0000 7f4a7cb44640 1 -- 172.21.15.116:0/3978672957 learned_addr learned my addr 172.21.15.116:0/3978672957 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:06:14.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.126+0000 7f4a77fff640 1 --2- 172.21.15.116:0/3978672957 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4a78072c90 0x7f4a780accd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:14.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.126+0000 7f4a7d345640 1 --2- 172.21.15.116:0/3978672957 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4a780c5280 0x7f4a781471e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:14.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.126+0000 7f4a7cb44640 1 -- 172.21.15.116:0/3978672957 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4a780c5280 msgr2=0x7f4a781471e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:14.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.126+0000 7f4a7cb44640 1 --2- 172.21.15.116:0/3978672957 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4a780c5280 0x7f4a781471e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:14.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.126+0000 7f4a7cb44640 1 -- 172.21.15.116:0/3978672957 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4a78072c90 msgr2=0x7f4a780accd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:14.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.126+0000 7f4a7cb44640 1 --2- 172.21.15.116:0/3978672957 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4a78072c90 0x7f4a780accd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:14.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.126+0000 7f4a7cb44640 1 -- 172.21.15.116:0/3978672957 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a78148e70 con 0x7f4a78072810 2024-04-25T00:06:14.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.130+0000 7f4a7cb44640 1 --2- 172.21.15.116:0/3978672957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a78072810 0x7f4a781f2110 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c00b550 tx=0x7f4a6c00ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:14.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.130+0000 7f4a757fa640 1 -- 172.21.15.116:0/3978672957 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a6c014070 con 0x7f4a78072810 2024-04-25T00:06:14.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.130+0000 7f4a757fa640 1 -- 172.21.15.116:0/3978672957 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a6c010890 con 0x7f4a78072810 2024-04-25T00:06:14.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.130+0000 7f4a757fa640 1 -- 172.21.15.116:0/3978672957 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a6c010c80 con 0x7f4a78072810 2024-04-25T00:06:14.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.130+0000 7f4a7eda8640 1 -- 172.21.15.116:0/3978672957 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a780c73d0 con 0x7f4a78072810 2024-04-25T00:06:14.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.130+0000 7f4a7eda8640 1 -- 172.21.15.116:0/3978672957 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a780c7990 con 0x7f4a78072810 2024-04-25T00:06:14.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.130+0000 7f4a7eda8640 1 -- 172.21.15.116:0/3978672957 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a78068a60 con 0x7f4a78072810 2024-04-25T00:06:14.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.130+0000 7f4a757fa640 1 -- 172.21.15.116:0/3978672957 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4a6c017070 con 0x7f4a78072810 2024-04-25T00:06:14.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.130+0000 7f4a757fa640 1 --2- 172.21.15.116:0/3978672957 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4a5c0678c0 0x7f4a5c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:14.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.130+0000 7f4a757fa640 1 -- 172.21.15.116:0/3978672957 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7096+0+0 (secure 0 0 0) 0x7f4a6c085f20 con 0x7f4a78072810 2024-04-25T00:06:14.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.130+0000 7f4a77fff640 1 --2- 172.21.15.116:0/3978672957 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4a5c0678c0 0x7f4a5c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:14.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.134+0000 7f4a77fff640 1 --2- 172.21.15.116:0/3978672957 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4a5c0678c0 0x7f4a5c069d80 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f4a781f2650 tx=0x7f4a64009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:14.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.138+0000 7f4a757fa640 1 -- 172.21.15.116:0/3978672957 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f4a6c055ea0 con 0x7f4a78072810 2024-04-25T00:06:14.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.362+0000 7f4a7eda8640 1 -- 172.21.15.116:0/3978672957 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4a78150180 con 0x7f4a78072810 2024-04-25T00:06:14.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.366+0000 7f4a757fa640 1 -- 172.21.15.116:0/3978672957 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+6501 (secure 0 0 0) 0x7f4a6c010e20 con 0x7f4a78072810 2024-04-25T00:06:14.367 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:06:14.367 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1233829678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1233829678},{"type":"v1","addr":"172.21.15.116:6835","nonce":1233829678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:04:38.435723+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4879},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4879":{"gid":4879,"name":"b","rank":0,"incarnation":32,"state":"up:active","state_seq":15,"addr":"172.21.15.174:6837/1889314958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1889314958},{"type":"v1","addr":"172.21.15.174:6837","nonce":1889314958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4879,"qdb_cluster":[4879]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:04:48.496838+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5571},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5571":{"gid":5571,"name":"c","rank":0,"incarnation":37,"state":"up:active","state_seq":4,"addr":"172.21.15.116:6833/3841609112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3841609112},{"type":"v1","addr":"172.21.15.116:6833","nonce":3841609112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5571,"qdb_cluster":[5571]},"id":2}]} 2024-04-25T00:06:14.367 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 41 2024-04-25T00:06:14.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.366+0000 7f4a7eda8640 1 -- 172.21.15.116:0/3978672957 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4a5c0678c0 msgr2=0x7f4a5c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:14.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.366+0000 7f4a7eda8640 1 --2- 172.21.15.116:0/3978672957 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4a5c0678c0 0x7f4a5c069d80 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f4a781f2650 tx=0x7f4a64009040 comp rx=0 tx=0).stop 2024-04-25T00:06:14.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.366+0000 7f4a7eda8640 1 -- 172.21.15.116:0/3978672957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a78072810 msgr2=0x7f4a781f2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:14.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.366+0000 7f4a7eda8640 1 --2- 172.21.15.116:0/3978672957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a78072810 0x7f4a781f2110 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c00b550 tx=0x7f4a6c00ba10 comp rx=0 tx=0).stop 2024-04-25T00:06:14.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.366+0000 7f4a7eda8640 1 -- 172.21.15.116:0/3978672957 shutdown_connections 2024-04-25T00:06:14.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.366+0000 7f4a7eda8640 1 --2- 172.21.15.116:0/3978672957 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4a5c0678c0 0x7f4a5c069d80 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:14.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.366+0000 7f4a7eda8640 1 --2- 172.21.15.116:0/3978672957 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4a780c5280 0x7f4a781471e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:14.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.366+0000 7f4a7eda8640 1 --2- 172.21.15.116:0/3978672957 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4a78072c90 0x7f4a780accd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:14.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.366+0000 7f4a7eda8640 1 --2- 172.21.15.116:0/3978672957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a78072810 0x7f4a781f2110 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:14.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.366+0000 7f4a7eda8640 1 -- 172.21.15.116:0/3978672957 >> 172.21.15.116:0/3978672957 conn(0x7f4a7806c720 msgr2=0x7f4a780aa4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:14.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.366+0000 7f4a7eda8640 1 -- 172.21.15.116:0/3978672957 shutdown_connections 2024-04-25T00:06:14.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:14.366+0000 7f4a7eda8640 1 -- 172.21.15.116:0/3978672957 wait complete. 2024-04-25T00:06:14.385 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T00:06:16.386 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:06:16.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.514+0000 7efcbd1a1640 1 Processor -- start 2024-04-25T00:06:16.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.514+0000 7efcbd1a1640 1 -- start start 2024-04-25T00:06:16.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.518+0000 7efcbd1a1640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efcb8071d90 0x7efcb81563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:16.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.518+0000 7efcbd1a1640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efcb8071220 con 0x7efcb814c720 2024-04-25T00:06:16.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.518+0000 7efcbd1a1640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efcb8152420 con 0x7efcb80a94a0 2024-04-25T00:06:16.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.518+0000 7efcbd1a1640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7efcb8153460 con 0x7efcb8071d90 2024-04-25T00:06:16.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.518+0000 7efcb6d9d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7efcb814c720 0x7efcb80c2a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:35090/0 (socket says 172.21.15.116:35090) 2024-04-25T00:06:16.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.518+0000 7efcb6d9d640 1 -- 172.21.15.116:0/1009314824 learned_addr learned my addr 172.21.15.116:0/1009314824 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:06:16.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.518+0000 7efcb759e640 1 --2- 172.21.15.116:0/1009314824 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efcb8071d90 0x7efcb81563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:16.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.518+0000 7efcb659c640 1 -- 172.21.15.116:0/1009314824 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3452621852 0 0) 0x7efcb8071220 con 0x7efcb814c720 2024-04-25T00:06:16.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.518+0000 7efcb659c640 1 -- 172.21.15.116:0/1009314824 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efca4003530 con 0x7efcb814c720 2024-04-25T00:06:16.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.518+0000 7efcb659c640 1 -- 172.21.15.116:0/1009314824 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2316011897 0 0) 0x7efca4003530 con 0x7efcb814c720 2024-04-25T00:06:16.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.518+0000 7efcb659c640 1 -- 172.21.15.116:0/1009314824 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efcb8071d90 msgr2=0x7efcb81563d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:06:16.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.518+0000 7efcb659c640 1 --2- 172.21.15.116:0/1009314824 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efcb8071d90 0x7efcb81563d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:16.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.518+0000 7efcb659c640 1 -- 172.21.15.116:0/1009314824 >> v1:172.21.15.174:6789/0 conn(0x7efcb80a94a0 legacy=0x7efcb8071680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:16.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.518+0000 7efcb659c640 1 -- 172.21.15.116:0/1009314824 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcb80721c0 con 0x7efcb814c720 2024-04-25T00:06:16.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.518+0000 7efcb659c640 1 -- 172.21.15.116:0/1009314824 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7efca8001680 con 0x7efcb814c720 2024-04-25T00:06:16.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.518+0000 7efcb659c640 1 -- 172.21.15.116:0/1009314824 >> v1:172.21.15.116:6789/0 conn(0x7efcb814c720 legacy=0x7efcb80c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:16.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.518+0000 7efcb659c640 1 --2- 172.21.15.116:0/1009314824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efca40039f0 0x7efca4003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:16.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.518+0000 7efcb659c640 1 --2- 172.21.15.116:0/1009314824 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efca4004810 0x7efca4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:16.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.518+0000 7efcb659c640 1 --2- 172.21.15.116:0/1009314824 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efcb8071d90 0x7efca4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:16.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.518+0000 7efcb659c640 1 -- 172.21.15.116:0/1009314824 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7efcb8153460 con 0x7efca40039f0 2024-04-25T00:06:16.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.518+0000 7efcb659c640 1 -- 172.21.15.116:0/1009314824 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7efca400bb30 con 0x7efca4004810 2024-04-25T00:06:16.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.518+0000 7efcb659c640 1 -- 172.21.15.116:0/1009314824 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7efca400bd30 con 0x7efcb8071d90 2024-04-25T00:06:16.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.518+0000 7efcb6d9d640 1 --2- 172.21.15.116:0/1009314824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efca40039f0 0x7efca4003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:16.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.518+0000 7efcae59c640 1 --2- 172.21.15.116:0/1009314824 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efca4004810 0x7efca4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:16.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.518+0000 7efcb6d9d640 1 -- 172.21.15.116:0/1009314824 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efcb8071d90 msgr2=0x7efca4005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:06:16.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.518+0000 7efcb6d9d640 1 --2- 172.21.15.116:0/1009314824 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efcb8071d90 0x7efca4005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:16.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.518+0000 7efcb6d9d640 1 -- 172.21.15.116:0/1009314824 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efca4004810 msgr2=0x7efca4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:06:16.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.518+0000 7efcb6d9d640 1 --2- 172.21.15.116:0/1009314824 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efca4004810 0x7efca4004c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:16.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.518+0000 7efcb6d9d640 1 -- 172.21.15.116:0/1009314824 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efca4003530 con 0x7efca40039f0 2024-04-25T00:06:16.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.518+0000 7efcb6d9d640 1 --2- 172.21.15.116:0/1009314824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efca40039f0 0x7efca4003e10 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7efca800c290 tx=0x7efca801cd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97c8aebe11702428 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:16.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.518+0000 7efcb659c640 1 -- 172.21.15.116:0/1009314824 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efca8007d40 con 0x7efca40039f0 2024-04-25T00:06:16.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.518+0000 7efcb659c640 1 -- 172.21.15.116:0/1009314824 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efca80a9020 con 0x7efca40039f0 2024-04-25T00:06:16.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.518+0000 7efcb659c640 1 -- 172.21.15.116:0/1009314824 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efca80a5b70 con 0x7efca40039f0 2024-04-25T00:06:16.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.522+0000 7efcbd1a1640 1 -- 172.21.15.116:0/1009314824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efca40039f0 msgr2=0x7efca4003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:16.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.522+0000 7efcbd1a1640 1 --2- 172.21.15.116:0/1009314824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efca40039f0 0x7efca4003e10 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7efca800c290 tx=0x7efca801cd00 comp rx=0 tx=0).stop 2024-04-25T00:06:16.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.522+0000 7efcb759e640 1 -- 172.21.15.116:0/1009314824 reap_dead start 2024-04-25T00:06:16.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.522+0000 7efcbd1a1640 1 -- 172.21.15.116:0/1009314824 shutdown_connections 2024-04-25T00:06:16.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.522+0000 7efcbd1a1640 1 -- 172.21.15.116:0/1009314824 >> 172.21.15.116:0/1009314824 conn(0x7efcb80c3b00 msgr2=0x7efcb80c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:16.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.522+0000 7efcbd1a1640 1 -- 172.21.15.116:0/1009314824 shutdown_connections 2024-04-25T00:06:16.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.522+0000 7efcbd1a1640 1 -- 172.21.15.116:0/1009314824 wait complete. 2024-04-25T00:06:16.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.522+0000 7efcbd1a1640 1 Processor -- start 2024-04-25T00:06:16.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.522+0000 7efcbd1a1640 1 -- start start 2024-04-25T00:06:16.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.522+0000 7efcbd1a1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efca40039f0 0x7efcb81f1c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:16.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.522+0000 7efcbd1a1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efcb80acd80 0x7efcb80abe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:16.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.522+0000 7efcbd1a1640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efcb814cd10 0x7efcb80ac340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:16.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.522+0000 7efcbd1a1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7efcb8152c60 con 0x7efca40039f0 2024-04-25T00:06:16.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.522+0000 7efcbd1a1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7efcb8152980 con 0x7efcb80acd80 2024-04-25T00:06:16.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.522+0000 7efcbd1a1640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7efcb8152f40 con 0x7efcb814cd10 2024-04-25T00:06:16.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.522+0000 7efcb6d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efca40039f0 0x7efcb81f1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:16.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.522+0000 7efcb6d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efca40039f0 0x7efcb81f1c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54656/0 (socket says 172.21.15.116:54656) 2024-04-25T00:06:16.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.522+0000 7efcb6d9d640 1 -- 172.21.15.116:0/1064091042 learned_addr learned my addr 172.21.15.116:0/1064091042 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:06:16.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.522+0000 7efcae59c640 1 --2- 172.21.15.116:0/1064091042 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efcb80acd80 0x7efcb80abe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:16.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.522+0000 7efcb759e640 1 --2- 172.21.15.116:0/1064091042 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efcb814cd10 0x7efcb80ac340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:16.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.522+0000 7efcb6d9d640 1 -- 172.21.15.116:0/1064091042 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efcb814cd10 msgr2=0x7efcb80ac340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:16.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.522+0000 7efcb6d9d640 1 --2- 172.21.15.116:0/1064091042 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efcb814cd10 0x7efcb80ac340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:16.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.522+0000 7efcb6d9d640 1 -- 172.21.15.116:0/1064091042 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efcb80acd80 msgr2=0x7efcb80abe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:16.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.522+0000 7efcb6d9d640 1 --2- 172.21.15.116:0/1064091042 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efcb80acd80 0x7efcb80abe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:16.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.522+0000 7efcb6d9d640 1 -- 172.21.15.116:0/1064091042 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcb8145160 con 0x7efca40039f0 2024-04-25T00:06:16.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.522+0000 7efcb6d9d640 1 --2- 172.21.15.116:0/1064091042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efca40039f0 0x7efcb81f1c50 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7efca800c290 tx=0x7efca4005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:16.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.522+0000 7efcaffff640 1 -- 172.21.15.116:0/1064091042 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efca80a5b50 con 0x7efca40039f0 2024-04-25T00:06:16.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.522+0000 7efcaffff640 1 -- 172.21.15.116:0/1064091042 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efca80a5cf0 con 0x7efca40039f0 2024-04-25T00:06:16.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.522+0000 7efcaffff640 1 -- 172.21.15.116:0/1064091042 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efca80049f0 con 0x7efca40039f0 2024-04-25T00:06:16.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.522+0000 7efcbd1a1640 1 -- 172.21.15.116:0/1064091042 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcb8144c40 con 0x7efca40039f0 2024-04-25T00:06:16.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.522+0000 7efcbd1a1640 1 -- 172.21.15.116:0/1064091042 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcb81f2880 con 0x7efca40039f0 2024-04-25T00:06:16.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.526+0000 7efcbd1a1640 1 -- 172.21.15.116:0/1064091042 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efcb8068350 con 0x7efca40039f0 2024-04-25T00:06:16.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.526+0000 7efcaffff640 1 -- 172.21.15.116:0/1064091042 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7efca8004b90 con 0x7efca40039f0 2024-04-25T00:06:16.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.526+0000 7efcaffff640 1 --2- 172.21.15.116:0/1064091042 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efc980678c0 0x7efc98069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:16.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.526+0000 7efcaffff640 1 -- 172.21.15.116:0/1064091042 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7096+0+0 (secure 0 0 0) 0x7efca8120010 con 0x7efca40039f0 2024-04-25T00:06:16.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.526+0000 7efcae59c640 1 --2- 172.21.15.116:0/1064091042 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efc980678c0 0x7efc98069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:16.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.526+0000 7efcae59c640 1 --2- 172.21.15.116:0/1064091042 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efc980678c0 0x7efc98069d80 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7efcb80ad390 tx=0x7efca0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:16.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.530+0000 7efcaffff640 1 -- 172.21.15.116:0/1064091042 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7efca8016a10 con 0x7efca40039f0 2024-04-25T00:06:16.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.722+0000 7efcbd1a1640 1 -- 172.21.15.116:0/1064091042 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efcb80c3640 con 0x7efca40039f0 2024-04-25T00:06:16.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.726+0000 7efcaffff640 1 -- 172.21.15.116:0/1064091042 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+6501 (secure 0 0 0) 0x7efca80f0670 con 0x7efca40039f0 2024-04-25T00:06:16.732 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:06:16.733 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1233829678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1233829678},{"type":"v1","addr":"172.21.15.116:6835","nonce":1233829678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:04:38.435723+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4879},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4879":{"gid":4879,"name":"b","rank":0,"incarnation":32,"state":"up:active","state_seq":15,"addr":"172.21.15.174:6837/1889314958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1889314958},{"type":"v1","addr":"172.21.15.174:6837","nonce":1889314958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4879,"qdb_cluster":[4879]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:04:48.496838+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5571},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5571":{"gid":5571,"name":"c","rank":0,"incarnation":37,"state":"up:active","state_seq":4,"addr":"172.21.15.116:6833/3841609112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3841609112},{"type":"v1","addr":"172.21.15.116:6833","nonce":3841609112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5571,"qdb_cluster":[5571]},"id":2}]} 2024-04-25T00:06:16.734 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 41 2024-04-25T00:06:16.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.726+0000 7efcbd1a1640 1 -- 172.21.15.116:0/1064091042 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efc980678c0 msgr2=0x7efc98069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:16.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.726+0000 7efcbd1a1640 1 --2- 172.21.15.116:0/1064091042 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efc980678c0 0x7efc98069d80 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7efcb80ad390 tx=0x7efca0009040 comp rx=0 tx=0).stop 2024-04-25T00:06:16.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.726+0000 7efcbd1a1640 1 -- 172.21.15.116:0/1064091042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efca40039f0 msgr2=0x7efcb81f1c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:16.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.726+0000 7efcbd1a1640 1 --2- 172.21.15.116:0/1064091042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efca40039f0 0x7efcb81f1c50 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7efca800c290 tx=0x7efca4005840 comp rx=0 tx=0).stop 2024-04-25T00:06:16.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.726+0000 7efcbd1a1640 1 -- 172.21.15.116:0/1064091042 shutdown_connections 2024-04-25T00:06:16.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.726+0000 7efcbd1a1640 1 --2- 172.21.15.116:0/1064091042 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efc980678c0 0x7efc98069d80 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:16.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.726+0000 7efcbd1a1640 1 --2- 172.21.15.116:0/1064091042 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efcb814cd10 0x7efcb80ac340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:16.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.726+0000 7efcbd1a1640 1 --2- 172.21.15.116:0/1064091042 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efcb80acd80 0x7efcb80abe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:16.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.726+0000 7efcbd1a1640 1 --2- 172.21.15.116:0/1064091042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efca40039f0 0x7efcb81f1c50 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:16.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.726+0000 7efcbd1a1640 1 -- 172.21.15.116:0/1064091042 >> 172.21.15.116:0/1064091042 conn(0x7efcb80c3b00 msgr2=0x7efcb8150f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:16.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.726+0000 7efcbd1a1640 1 -- 172.21.15.116:0/1064091042 shutdown_connections 2024-04-25T00:06:16.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:16.726+0000 7efcbd1a1640 1 -- 172.21.15.116:0/1064091042 wait complete. 2024-04-25T00:06:16.744 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T00:06:18.745 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:06:18.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.874+0000 7f43b513c640 1 Processor -- start 2024-04-25T00:06:18.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.874+0000 7f43b513c640 1 -- start start 2024-04-25T00:06:18.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.874+0000 7f43b513c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43b0152d80 0x7f43b006bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:18.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.874+0000 7f43b513c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f43b006d5f0 0x7f43b0156c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:18.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.874+0000 7f43b513c640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f43b006c290 0x7f43b01571c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:18.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.874+0000 7f43b513c640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f43b00c9d60 con 0x7f43b006d5f0 2024-04-25T00:06:18.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.874+0000 7f43b513c640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f43b00c9150 con 0x7f43b006c290 2024-04-25T00:06:18.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.874+0000 7f43b513c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f43b00c9430 con 0x7f43b0152d80 2024-04-25T00:06:18.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.874+0000 7f43ae59c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f43b006d5f0 0x7f43b0156c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:18.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.874+0000 7f43ae59c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f43b006d5f0 0x7f43b0156c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54660/0 (socket says 172.21.15.116:54660) 2024-04-25T00:06:18.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.874+0000 7f43ae59c640 1 -- 172.21.15.116:0/2245590216 learned_addr learned my addr 172.21.15.116:0/2245590216 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:06:18.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.874+0000 7f43af59e640 1 --2- 172.21.15.116:0/2245590216 >> v2:172.21.15.174:3300/0 conn(0x7f43b006c290 0x7f43b01571c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:18.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.874+0000 7f43aed9d640 1 --2- 172.21.15.116:0/2245590216 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43b0152d80 0x7f43b006bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:18.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.874+0000 7f43ae59c640 1 -- 172.21.15.116:0/2245590216 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43b0152d80 msgr2=0x7f43b006bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:06:18.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.874+0000 7f43ae59c640 1 --2- 172.21.15.116:0/2245590216 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43b0152d80 0x7f43b006bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:18.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.874+0000 7f43ae59c640 1 -- 172.21.15.116:0/2245590216 >> v2:172.21.15.174:3300/0 conn(0x7f43b006c290 msgr2=0x7f43b01571c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:06:18.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.874+0000 7f43ae59c640 1 --2- 172.21.15.116:0/2245590216 >> v2:172.21.15.174:3300/0 conn(0x7f43b006c290 0x7f43b01571c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:18.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.874+0000 7f43ae59c640 1 -- 172.21.15.116:0/2245590216 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43b00c9710 con 0x7f43b006d5f0 2024-04-25T00:06:18.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.874+0000 7f43af59e640 1 --2- 172.21.15.116:0/2245590216 >> v2:172.21.15.174:3300/0 conn(0x7f43b006c290 0x7f43b01571c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:06:18.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.878+0000 7f43ae59c640 1 --2- 172.21.15.116:0/2245590216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43b006d5f0 0x7f43b0156c80 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f43b00c86f0 tx=0x7f43a401c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5898d08dcf22d99 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:18.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.878+0000 7f43add9b640 1 -- 172.21.15.116:0/2245590216 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a40b0070 con 0x7f43b006d5f0 2024-04-25T00:06:18.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.878+0000 7f43add9b640 1 -- 172.21.15.116:0/2245590216 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43a40ab600 con 0x7f43b006d5f0 2024-04-25T00:06:18.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.878+0000 7f43add9b640 1 -- 172.21.15.116:0/2245590216 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a40ab9d0 con 0x7f43b006d5f0 2024-04-25T00:06:18.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.878+0000 7f43b513c640 1 -- 172.21.15.116:0/2245590216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43b006d5f0 msgr2=0x7f43b0156c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:18.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.878+0000 7f43b513c640 1 --2- 172.21.15.116:0/2245590216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43b006d5f0 0x7f43b0156c80 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f43b00c86f0 tx=0x7f43a401c460 comp rx=0 tx=0).stop 2024-04-25T00:06:18.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.878+0000 7f43b513c640 1 -- 172.21.15.116:0/2245590216 shutdown_connections 2024-04-25T00:06:18.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.878+0000 7f43b513c640 1 --2- 172.21.15.116:0/2245590216 >> v2:172.21.15.174:3300/0 conn(0x7f43b006c290 0x7f43b01571c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:18.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.878+0000 7f43b513c640 1 --2- 172.21.15.116:0/2245590216 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43b006d5f0 0x7f43b0156c80 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:18.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.878+0000 7f43b513c640 1 --2- 172.21.15.116:0/2245590216 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43b0152d80 0x7f43b006bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:18.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.878+0000 7f43b513c640 1 -- 172.21.15.116:0/2245590216 >> 172.21.15.116:0/2245590216 conn(0x7f43b00acdd0 msgr2=0x7f43b00ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:18.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.878+0000 7f43b513c640 1 -- 172.21.15.116:0/2245590216 shutdown_connections 2024-04-25T00:06:18.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.878+0000 7f43b513c640 1 -- 172.21.15.116:0/2245590216 wait complete. 2024-04-25T00:06:18.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.878+0000 7f43b513c640 1 Processor -- start 2024-04-25T00:06:18.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.878+0000 7f43b513c640 1 -- start start 2024-04-25T00:06:18.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.878+0000 7f43b513c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43b006c290 0x7f43b01eeb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:18.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.878+0000 7f43b513c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43b006d5f0 0x7f43b01ef060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:18.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.878+0000 7f43b513c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43b0152d80 0x7f43b0144350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:18.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.878+0000 7f43b513c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f43b0159710 con 0x7f43b006c290 2024-04-25T00:06:18.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.878+0000 7f43b513c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f43b0159590 con 0x7f43b0152d80 2024-04-25T00:06:18.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.878+0000 7f43b513c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f43b0159890 con 0x7f43b006d5f0 2024-04-25T00:06:18.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.878+0000 7f43aed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43b006c290 0x7f43b01eeb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:18.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.878+0000 7f43aed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43b006c290 0x7f43b01eeb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54662/0 (socket says 172.21.15.116:54662) 2024-04-25T00:06:18.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.878+0000 7f43aed9d640 1 -- 172.21.15.116:0/359668333 learned_addr learned my addr 172.21.15.116:0/359668333 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:06:18.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.878+0000 7f43ae59c640 1 --2- 172.21.15.116:0/359668333 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43b006d5f0 0x7f43b01ef060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:18.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.878+0000 7f43af59e640 1 --2- 172.21.15.116:0/359668333 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43b0152d80 0x7f43b0144350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:18.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.878+0000 7f43aed9d640 1 -- 172.21.15.116:0/359668333 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43b006d5f0 msgr2=0x7f43b01ef060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:18.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.878+0000 7f43aed9d640 1 --2- 172.21.15.116:0/359668333 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43b006d5f0 0x7f43b01ef060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:18.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.878+0000 7f43aed9d640 1 -- 172.21.15.116:0/359668333 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43b0152d80 msgr2=0x7f43b0144350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:18.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.878+0000 7f43aed9d640 1 --2- 172.21.15.116:0/359668333 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43b0152d80 0x7f43b0144350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:18.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.878+0000 7f43aed9d640 1 -- 172.21.15.116:0/359668333 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43b00ae2f0 con 0x7f43b006c290 2024-04-25T00:06:18.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.878+0000 7f43ae59c640 1 --2- 172.21.15.116:0/359668333 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43b006d5f0 0x7f43b01ef060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:06:18.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.878+0000 7f43aed9d640 1 --2- 172.21.15.116:0/359668333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43b006c290 0x7f43b01eeb20 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f43a000d5f0 tx=0x7f43a000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:18.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.882+0000 7f43977fe640 1 -- 172.21.15.116:0/359668333 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a00125f0 con 0x7f43b006c290 2024-04-25T00:06:18.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.882+0000 7f43977fe640 1 -- 172.21.15.116:0/359668333 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43a0019070 con 0x7f43b006c290 2024-04-25T00:06:18.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.882+0000 7f43b513c640 1 -- 172.21.15.116:0/359668333 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43b014c670 con 0x7f43b006c290 2024-04-25T00:06:18.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.882+0000 7f43977fe640 1 -- 172.21.15.116:0/359668333 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a00129b0 con 0x7f43b006c290 2024-04-25T00:06:18.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.882+0000 7f43b513c640 1 -- 172.21.15.116:0/359668333 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43b00be6c0 con 0x7f43b006c290 2024-04-25T00:06:18.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.882+0000 7f43b513c640 1 -- 172.21.15.116:0/359668333 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43b0068350 con 0x7f43b006c290 2024-04-25T00:06:18.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.882+0000 7f43977fe640 1 -- 172.21.15.116:0/359668333 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f43a0012b50 con 0x7f43b006c290 2024-04-25T00:06:18.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.882+0000 7f43977fe640 1 --2- 172.21.15.116:0/359668333 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f439c0678c0 0x7f439c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:18.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.882+0000 7f43977fe640 1 -- 172.21.15.116:0/359668333 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7096+0+0 (secure 0 0 0) 0x7f43a0086540 con 0x7f43b006c290 2024-04-25T00:06:18.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.882+0000 7f43ae59c640 1 --2- 172.21.15.116:0/359668333 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f439c0678c0 0x7f439c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:18.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.882+0000 7f43ae59c640 1 --2- 172.21.15.116:0/359668333 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f439c0678c0 0x7f439c069d80 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f43b00c8c30 tx=0x7f43a4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:18.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:18.886+0000 7f43977fe640 1 -- 172.21.15.116:0/359668333 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f43a0056ba0 con 0x7f43b006c290 2024-04-25T00:06:19.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:19.082+0000 7f43b513c640 1 -- 172.21.15.116:0/359668333 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f43b0156620 con 0x7f43b006c290 2024-04-25T00:06:19.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:19.082+0000 7f43977fe640 1 -- 172.21.15.116:0/359668333 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+6501 (secure 0 0 0) 0x7f43a00564a0 con 0x7f43b006c290 2024-04-25T00:06:19.085 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:06:19.085 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1233829678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1233829678},{"type":"v1","addr":"172.21.15.116:6835","nonce":1233829678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:04:38.435723+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4879},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4879":{"gid":4879,"name":"b","rank":0,"incarnation":32,"state":"up:active","state_seq":15,"addr":"172.21.15.174:6837/1889314958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1889314958},{"type":"v1","addr":"172.21.15.174:6837","nonce":1889314958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4879,"qdb_cluster":[4879]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:04:48.496838+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5571},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5571":{"gid":5571,"name":"c","rank":0,"incarnation":37,"state":"up:active","state_seq":4,"addr":"172.21.15.116:6833/3841609112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3841609112},{"type":"v1","addr":"172.21.15.116:6833","nonce":3841609112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5571,"qdb_cluster":[5571]},"id":2}]} 2024-04-25T00:06:19.085 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 41 2024-04-25T00:06:19.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:19.086+0000 7f43b513c640 1 -- 172.21.15.116:0/359668333 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f439c0678c0 msgr2=0x7f439c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:19.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:19.086+0000 7f43b513c640 1 --2- 172.21.15.116:0/359668333 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f439c0678c0 0x7f439c069d80 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f43b00c8c30 tx=0x7f43a4008040 comp rx=0 tx=0).stop 2024-04-25T00:06:19.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:19.086+0000 7f43b513c640 1 -- 172.21.15.116:0/359668333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43b006c290 msgr2=0x7f43b01eeb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:19.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:19.086+0000 7f43b513c640 1 --2- 172.21.15.116:0/359668333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43b006c290 0x7f43b01eeb20 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f43a000d5f0 tx=0x7f43a000dab0 comp rx=0 tx=0).stop 2024-04-25T00:06:19.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:19.086+0000 7f43b513c640 1 -- 172.21.15.116:0/359668333 shutdown_connections 2024-04-25T00:06:19.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:19.086+0000 7f43b513c640 1 --2- 172.21.15.116:0/359668333 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f439c0678c0 0x7f439c069d80 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:19.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:19.086+0000 7f43b513c640 1 --2- 172.21.15.116:0/359668333 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43b0152d80 0x7f43b0144350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:19.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:19.086+0000 7f43b513c640 1 --2- 172.21.15.116:0/359668333 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43b006d5f0 0x7f43b01ef060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:19.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:19.086+0000 7f43b513c640 1 --2- 172.21.15.116:0/359668333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43b006c290 0x7f43b01eeb20 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:19.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:19.086+0000 7f43b513c640 1 -- 172.21.15.116:0/359668333 >> 172.21.15.116:0/359668333 conn(0x7f43b00acdd0 msgr2=0x7f43b014db40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:19.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:19.086+0000 7f43b513c640 1 -- 172.21.15.116:0/359668333 shutdown_connections 2024-04-25T00:06:19.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:19.086+0000 7f43b513c640 1 -- 172.21.15.116:0/359668333 wait complete. 2024-04-25T00:06:19.103 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T00:06:21.104 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:06:21.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.234+0000 7f7b69452640 1 Processor -- start 2024-04-25T00:06:21.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.234+0000 7f7b69452640 1 -- start start 2024-04-25T00:06:21.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.238+0000 7f7b69452640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f7b64150d20 0x7f7b640c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:21.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.238+0000 7f7b69452640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7b640a94a0 0x7f7b64153ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:21.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.238+0000 7f7b69452640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b640c3910 con 0x7f7b640c2f50 2024-04-25T00:06:21.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.238+0000 7f7b69452640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f7b64154530 con 0x7f7b640a94a0 2024-04-25T00:06:21.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.238+0000 7f7b69452640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f7b640c16e0 con 0x7f7b64150d20 2024-04-25T00:06:21.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.238+0000 7f7b637fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7b640a94a0 0x7f7b64153ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:21.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.238+0000 7f7b637fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7b640a94a0 0x7f7b64153ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54674/0 (socket says 172.21.15.116:54674) 2024-04-25T00:06:21.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.238+0000 7f7b637fe640 1 -- 172.21.15.116:0/3265561582 learned_addr learned my addr 172.21.15.116:0/3265561582 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:06:21.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.238+0000 7f7b62ffd640 1 --2- 172.21.15.116:0/3265561582 >> v2:172.21.15.174:3300/0 conn(0x7f7b64150d20 0x7f7b640c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:21.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.238+0000 7f7b637fe640 1 -- 172.21.15.116:0/3265561582 >> v2:172.21.15.174:3300/0 conn(0x7f7b64150d20 msgr2=0x7f7b640c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:06:21.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.238+0000 7f7b637fe640 1 --2- 172.21.15.116:0/3265561582 >> v2:172.21.15.174:3300/0 conn(0x7f7b64150d20 0x7f7b640c2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:21.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.238+0000 7f7b637fe640 1 -- 172.21.15.116:0/3265561582 >> v1:172.21.15.116:6789/0 conn(0x7f7b640c2f50 legacy=0x7f7b641538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:21.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.238+0000 7f7b637fe640 1 -- 172.21.15.116:0/3265561582 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b640c1400 con 0x7f7b640a94a0 2024-04-25T00:06:21.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.238+0000 7f7b61ffb640 1 -- 172.21.15.116:0/3265561582 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4165235711 0 0) 0x7f7b640c3910 con 0x7f7b640c2f50 2024-04-25T00:06:21.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.238+0000 7f7b637fe640 1 --2- 172.21.15.116:0/3265561582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b640a94a0 0x7f7b64153ff0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f7b64073550 tx=0x7f7b5c01c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9aafff8d46a7b19e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:21.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.238+0000 7f7b61ffb640 1 -- 172.21.15.116:0/3265561582 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b5c0a57d0 con 0x7f7b640a94a0 2024-04-25T00:06:21.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.238+0000 7f7b61ffb640 1 -- 172.21.15.116:0/3265561582 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b5c0a5970 con 0x7f7b640a94a0 2024-04-25T00:06:21.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.238+0000 7f7b61ffb640 1 -- 172.21.15.116:0/3265561582 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b5c0a5d40 con 0x7f7b640a94a0 2024-04-25T00:06:21.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.242+0000 7f7b69452640 1 -- 172.21.15.116:0/3265561582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b640a94a0 msgr2=0x7f7b64153ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:21.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.242+0000 7f7b69452640 1 --2- 172.21.15.116:0/3265561582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b640a94a0 0x7f7b64153ff0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f7b64073550 tx=0x7f7b5c01c9f0 comp rx=0 tx=0).stop 2024-04-25T00:06:21.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.242+0000 7f7b69452640 1 -- 172.21.15.116:0/3265561582 shutdown_connections 2024-04-25T00:06:21.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.242+0000 7f7b69452640 1 --2- 172.21.15.116:0/3265561582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b640a94a0 0x7f7b64153ff0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:21.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.242+0000 7f7b69452640 1 --2- 172.21.15.116:0/3265561582 >> v2:172.21.15.174:3300/0 conn(0x7f7b64150d20 0x7f7b640c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:21.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.242+0000 7f7b69452640 1 -- 172.21.15.116:0/3265561582 >> 172.21.15.116:0/3265561582 conn(0x7f7b640c3b00 msgr2=0x7f7b640c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:21.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.242+0000 7f7b69452640 1 -- 172.21.15.116:0/3265561582 shutdown_connections 2024-04-25T00:06:21.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.242+0000 7f7b69452640 1 -- 172.21.15.116:0/3265561582 wait complete. 2024-04-25T00:06:21.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.242+0000 7f7b69452640 1 Processor -- start 2024-04-25T00:06:21.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.242+0000 7f7b69452640 1 -- start start 2024-04-25T00:06:21.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.242+0000 7f7b69452640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b640a94a0 0x7f7b641ece40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:21.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.242+0000 7f7b69452640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b640c2f50 0x7f7b640c6910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:21.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.242+0000 7f7b69452640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b64150d20 0x7f7b64140490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:21.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.242+0000 7f7b69452640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7b641608b0 con 0x7f7b640a94a0 2024-04-25T00:06:21.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.242+0000 7f7b69452640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7b64160730 con 0x7f7b640c2f50 2024-04-25T00:06:21.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.242+0000 7f7b69452640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f7b64160a30 con 0x7f7b64150d20 2024-04-25T00:06:21.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.242+0000 7f7b62ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b640a94a0 0x7f7b641ece40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:21.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.242+0000 7f7b62ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b640a94a0 0x7f7b641ece40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54684/0 (socket says 172.21.15.116:54684) 2024-04-25T00:06:21.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.242+0000 7f7b62ffd640 1 -- 172.21.15.116:0/3825098927 learned_addr learned my addr 172.21.15.116:0/3825098927 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:06:21.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.242+0000 7f7b62ffd640 1 -- 172.21.15.116:0/3825098927 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b64150d20 msgr2=0x7f7b64140490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:21.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.242+0000 7f7b627fc640 1 --2- 172.21.15.116:0/3825098927 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b640c2f50 0x7f7b640c6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:21.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.242+0000 7f7b637fe640 1 --2- 172.21.15.116:0/3825098927 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b64150d20 0x7f7b64140490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:21.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.242+0000 7f7b62ffd640 1 --2- 172.21.15.116:0/3825098927 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b64150d20 0x7f7b64140490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:21.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.242+0000 7f7b62ffd640 1 -- 172.21.15.116:0/3825098927 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b640c2f50 msgr2=0x7f7b640c6910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:21.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.242+0000 7f7b62ffd640 1 --2- 172.21.15.116:0/3825098927 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b640c2f50 0x7f7b640c6910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:21.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.242+0000 7f7b62ffd640 1 -- 172.21.15.116:0/3825098927 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b64146980 con 0x7f7b640a94a0 2024-04-25T00:06:21.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.242+0000 7f7b62ffd640 1 --2- 172.21.15.116:0/3825098927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b640a94a0 0x7f7b641ece40 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f7b5800d5f0 tx=0x7f7b5800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:21.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.242+0000 7f7b4b7fe640 1 -- 172.21.15.116:0/3825098927 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b580125f0 con 0x7f7b640a94a0 2024-04-25T00:06:21.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.242+0000 7f7b4b7fe640 1 -- 172.21.15.116:0/3825098927 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b58019070 con 0x7f7b640a94a0 2024-04-25T00:06:21.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.242+0000 7f7b4b7fe640 1 -- 172.21.15.116:0/3825098927 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b580129d0 con 0x7f7b640a94a0 2024-04-25T00:06:21.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.242+0000 7f7b69452640 1 -- 172.21.15.116:0/3825098927 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b641466a0 con 0x7f7b640a94a0 2024-04-25T00:06:21.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.242+0000 7f7b69452640 1 -- 172.21.15.116:0/3825098927 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b641ec640 con 0x7f7b640a94a0 2024-04-25T00:06:21.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.246+0000 7f7b69452640 1 -- 172.21.15.116:0/3825098927 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b64068350 con 0x7f7b640a94a0 2024-04-25T00:06:21.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.246+0000 7f7b4b7fe640 1 -- 172.21.15.116:0/3825098927 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7b58012b70 con 0x7f7b640a94a0 2024-04-25T00:06:21.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.246+0000 7f7b4b7fe640 1 --2- 172.21.15.116:0/3825098927 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7b500678f0 0x7f7b50069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:21.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.246+0000 7f7b4b7fe640 1 -- 172.21.15.116:0/3825098927 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7096+0+0 (secure 0 0 0) 0x7f7b58086590 con 0x7f7b640a94a0 2024-04-25T00:06:21.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.246+0000 7f7b627fc640 1 --2- 172.21.15.116:0/3825098927 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7b500678f0 0x7f7b50069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:21.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.246+0000 7f7b627fc640 1 --2- 172.21.15.116:0/3825098927 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7b500678f0 0x7f7b50069db0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f7b64141190 tx=0x7f7b54009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:21.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.250+0000 7f7b4b7fe640 1 -- 172.21.15.116:0/3825098927 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f7b58056bf0 con 0x7f7b640a94a0 2024-04-25T00:06:21.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.442+0000 7f7b69452640 1 -- 172.21.15.116:0/3825098927 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7b640c34f0 con 0x7f7b640a94a0 2024-04-25T00:06:21.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.446+0000 7f7b4b7fe640 1 -- 172.21.15.116:0/3825098927 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+6501 (secure 0 0 0) 0x7f7b580564f0 con 0x7f7b640a94a0 2024-04-25T00:06:21.447 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:06:21.447 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1233829678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1233829678},{"type":"v1","addr":"172.21.15.116:6835","nonce":1233829678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:04:38.435723+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4879},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4879":{"gid":4879,"name":"b","rank":0,"incarnation":32,"state":"up:active","state_seq":15,"addr":"172.21.15.174:6837/1889314958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1889314958},{"type":"v1","addr":"172.21.15.174:6837","nonce":1889314958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4879,"qdb_cluster":[4879]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:04:48.496838+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5571},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5571":{"gid":5571,"name":"c","rank":0,"incarnation":37,"state":"up:active","state_seq":4,"addr":"172.21.15.116:6833/3841609112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3841609112},{"type":"v1","addr":"172.21.15.116:6833","nonce":3841609112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5571,"qdb_cluster":[5571]},"id":2}]} 2024-04-25T00:06:21.447 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 41 2024-04-25T00:06:21.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.446+0000 7f7b69452640 1 -- 172.21.15.116:0/3825098927 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7b500678f0 msgr2=0x7f7b50069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:21.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.446+0000 7f7b69452640 1 --2- 172.21.15.116:0/3825098927 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7b500678f0 0x7f7b50069db0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f7b64141190 tx=0x7f7b54009040 comp rx=0 tx=0).stop 2024-04-25T00:06:21.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.446+0000 7f7b69452640 1 -- 172.21.15.116:0/3825098927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b640a94a0 msgr2=0x7f7b641ece40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:21.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.446+0000 7f7b69452640 1 --2- 172.21.15.116:0/3825098927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b640a94a0 0x7f7b641ece40 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f7b5800d5f0 tx=0x7f7b5800dab0 comp rx=0 tx=0).stop 2024-04-25T00:06:21.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.446+0000 7f7b69452640 1 -- 172.21.15.116:0/3825098927 shutdown_connections 2024-04-25T00:06:21.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.446+0000 7f7b69452640 1 --2- 172.21.15.116:0/3825098927 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7b500678f0 0x7f7b50069db0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:21.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.446+0000 7f7b69452640 1 --2- 172.21.15.116:0/3825098927 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b64150d20 0x7f7b64140490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:21.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.446+0000 7f7b69452640 1 --2- 172.21.15.116:0/3825098927 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b640c2f50 0x7f7b640c6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:21.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.446+0000 7f7b69452640 1 --2- 172.21.15.116:0/3825098927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b640a94a0 0x7f7b641ece40 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:21.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.446+0000 7f7b69452640 1 -- 172.21.15.116:0/3825098927 >> 172.21.15.116:0/3825098927 conn(0x7f7b640c3b00 msgr2=0x7f7b640ac9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:21.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.446+0000 7f7b69452640 1 -- 172.21.15.116:0/3825098927 shutdown_connections 2024-04-25T00:06:21.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:21.446+0000 7f7b69452640 1 -- 172.21.15.116:0/3825098927 wait complete. 2024-04-25T00:06:21.465 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T00:06:23.466 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:06:23.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.602+0000 7f651357c640 1 Processor -- start 2024-04-25T00:06:23.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.602+0000 7f651357c640 1 -- start start 2024-04-25T00:06:23.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.606+0000 7f651357c640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f650c152e60 0x7f650c1599a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:23.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.606+0000 7f651357c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f650c0c5010 0x7f650c0c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:23.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.606+0000 7f651357c640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f650c073200 con 0x7f650c1562d0 2024-04-25T00:06:23.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.606+0000 7f651357c640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f650c156760 con 0x7f650c0c5010 2024-04-25T00:06:23.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.606+0000 7f651357c640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f650c153240 con 0x7f650c152e60 2024-04-25T00:06:23.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.606+0000 7f6511b19640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f650c0c5010 0x7f650c0c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:23.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.606+0000 7f6511b19640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f650c0c5010 0x7f650c0c5460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54686/0 (socket says 172.21.15.116:54686) 2024-04-25T00:06:23.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.606+0000 7f6511b19640 1 -- 172.21.15.116:0/2508307783 learned_addr learned my addr 172.21.15.116:0/2508307783 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:06:23.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.606+0000 7f6511318640 1 --2- 172.21.15.116:0/2508307783 >> v2:172.21.15.174:3300/0 conn(0x7f650c152e60 0x7f650c1599a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:23.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.606+0000 7f6511b19640 1 -- 172.21.15.116:0/2508307783 >> v2:172.21.15.174:3300/0 conn(0x7f650c152e60 msgr2=0x7f650c1599a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:06:23.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.606+0000 7f6511b19640 1 --2- 172.21.15.116:0/2508307783 >> v2:172.21.15.174:3300/0 conn(0x7f650c152e60 0x7f650c1599a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:23.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.606+0000 7f6511b19640 1 -- 172.21.15.116:0/2508307783 >> v1:172.21.15.174:6789/0 conn(0x7f650c1562d0 legacy=0x7f650c0c36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:06:23.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.606+0000 7f6511b19640 1 -- 172.21.15.116:0/2508307783 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f650c158ec0 con 0x7f650c0c5010 2024-04-25T00:06:23.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.606+0000 7f6511b19640 1 --2- 172.21.15.116:0/2508307783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f650c0c5010 0x7f650c0c5460 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f650c0c2080 tx=0x7f650401c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e81384e712aa5ff6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:23.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.606+0000 7f6503fff640 1 -- 172.21.15.116:0/2508307783 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65040a55e0 con 0x7f650c0c5010 2024-04-25T00:06:23.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.606+0000 7f6503fff640 1 -- 172.21.15.116:0/2508307783 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65040a5780 con 0x7f650c0c5010 2024-04-25T00:06:23.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.606+0000 7f6503fff640 1 -- 172.21.15.116:0/2508307783 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65040a5b50 con 0x7f650c0c5010 2024-04-25T00:06:23.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.610+0000 7f651357c640 1 -- 172.21.15.116:0/2508307783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f650c0c5010 msgr2=0x7f650c0c5460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:23.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.610+0000 7f651357c640 1 --2- 172.21.15.116:0/2508307783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f650c0c5010 0x7f650c0c5460 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f650c0c2080 tx=0x7f650401c920 comp rx=0 tx=0).stop 2024-04-25T00:06:23.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.610+0000 7f651357c640 1 -- 172.21.15.116:0/2508307783 shutdown_connections 2024-04-25T00:06:23.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.610+0000 7f651357c640 1 --2- 172.21.15.116:0/2508307783 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f650c0c5010 0x7f650c0c5460 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:23.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.610+0000 7f651357c640 1 --2- 172.21.15.116:0/2508307783 >> v2:172.21.15.174:3300/0 conn(0x7f650c152e60 0x7f650c1599a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:23.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.610+0000 7f651357c640 1 -- 172.21.15.116:0/2508307783 >> 172.21.15.116:0/2508307783 conn(0x7f650c06c720 msgr2=0x7f650c06cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:23.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.610+0000 7f651357c640 1 -- 172.21.15.116:0/2508307783 shutdown_connections 2024-04-25T00:06:23.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.610+0000 7f651357c640 1 -- 172.21.15.116:0/2508307783 wait complete. 2024-04-25T00:06:23.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.610+0000 7f651357c640 1 Processor -- start 2024-04-25T00:06:23.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.610+0000 7f651357c640 1 -- start start 2024-04-25T00:06:23.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.610+0000 7f651357c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f650c0c5010 0x7f650c06e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:23.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.610+0000 7f651357c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f650c152e60 0x7f650c14b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:23.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.610+0000 7f651357c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f650c1562d0 0x7f650c1f1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:23.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.610+0000 7f651357c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f650c072e20 con 0x7f650c1562d0 2024-04-25T00:06:23.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.610+0000 7f651357c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f650c0c3df0 con 0x7f650c0c5010 2024-04-25T00:06:23.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.610+0000 7f651357c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f650c072fa0 con 0x7f650c152e60 2024-04-25T00:06:23.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.610+0000 7f6511b19640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f650c1562d0 0x7f650c1f1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:23.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.610+0000 7f6511b19640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f650c1562d0 0x7f650c1f1ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54700/0 (socket says 172.21.15.116:54700) 2024-04-25T00:06:23.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.610+0000 7f6511b19640 1 -- 172.21.15.116:0/2060617898 learned_addr learned my addr 172.21.15.116:0/2060617898 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:06:23.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.610+0000 7f6510b17640 1 --2- 172.21.15.116:0/2060617898 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f650c152e60 0x7f650c14b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:23.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.610+0000 7f6511318640 1 --2- 172.21.15.116:0/2060617898 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f650c0c5010 0x7f650c06e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:23.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.610+0000 7f6511b19640 1 -- 172.21.15.116:0/2060617898 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f650c152e60 msgr2=0x7f650c14b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:23.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.610+0000 7f6511b19640 1 --2- 172.21.15.116:0/2060617898 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f650c152e60 0x7f650c14b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:23.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.610+0000 7f6511b19640 1 -- 172.21.15.116:0/2060617898 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f650c0c5010 msgr2=0x7f650c06e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:23.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.610+0000 7f6511b19640 1 --2- 172.21.15.116:0/2060617898 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f650c0c5010 0x7f650c06e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:23.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.610+0000 7f6511b19640 1 -- 172.21.15.116:0/2060617898 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f650c155cf0 con 0x7f650c1562d0 2024-04-25T00:06:23.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.610+0000 7f6511b19640 1 --2- 172.21.15.116:0/2060617898 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f650c1562d0 0x7f650c1f1ae0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f650401afd0 tx=0x7f65040196c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:23.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.614+0000 7f6501ffb640 1 -- 172.21.15.116:0/2060617898 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6504009510 con 0x7f650c1562d0 2024-04-25T00:06:23.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.614+0000 7f6501ffb640 1 -- 172.21.15.116:0/2060617898 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65040be070 con 0x7f650c1562d0 2024-04-25T00:06:23.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.614+0000 7f651357c640 1 -- 172.21.15.116:0/2060617898 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f650c0c1910 con 0x7f650c1562d0 2024-04-25T00:06:23.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.614+0000 7f6501ffb640 1 -- 172.21.15.116:0/2060617898 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65040b88b0 con 0x7f650c1562d0 2024-04-25T00:06:23.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.614+0000 7f651357c640 1 -- 172.21.15.116:0/2060617898 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f650c0ad470 con 0x7f650c1562d0 2024-04-25T00:06:23.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.614+0000 7f651357c640 1 -- 172.21.15.116:0/2060617898 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f650c068a60 con 0x7f650c1562d0 2024-04-25T00:06:23.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.614+0000 7f6501ffb640 1 -- 172.21.15.116:0/2060617898 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f65040b8a50 con 0x7f650c1562d0 2024-04-25T00:06:23.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.614+0000 7f6501ffb640 1 --2- 172.21.15.116:0/2060617898 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f64f00678f0 0x7f64f0069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:23.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.614+0000 7f6501ffb640 1 -- 172.21.15.116:0/2060617898 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7096+0+0 (secure 0 0 0) 0x7f6504121180 con 0x7f650c1562d0 2024-04-25T00:06:23.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.614+0000 7f6511318640 1 --2- 172.21.15.116:0/2060617898 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f64f00678f0 0x7f64f0069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:23.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.614+0000 7f6511318640 1 --2- 172.21.15.116:0/2060617898 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f64f00678f0 0x7f64f0069db0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f65080079c0 tx=0x7f6508008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:23.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.618+0000 7f6501ffb640 1 -- 172.21.15.116:0/2060617898 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f65040165a0 con 0x7f650c1562d0 2024-04-25T00:06:23.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.814+0000 7f651357c640 1 -- 172.21.15.116:0/2060617898 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f650c0c5600 con 0x7f650c1562d0 2024-04-25T00:06:23.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.814+0000 7f6501ffb640 1 -- 172.21.15.116:0/2060617898 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+6501 (secure 0 0 0) 0x7f65040f1190 con 0x7f650c1562d0 2024-04-25T00:06:23.816 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:06:23.816 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1233829678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1233829678},{"type":"v1","addr":"172.21.15.116:6835","nonce":1233829678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:04:38.435723+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4879},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4879":{"gid":4879,"name":"b","rank":0,"incarnation":32,"state":"up:active","state_seq":15,"addr":"172.21.15.174:6837/1889314958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1889314958},{"type":"v1","addr":"172.21.15.174:6837","nonce":1889314958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4879,"qdb_cluster":[4879]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:04:48.496838+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5571},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5571":{"gid":5571,"name":"c","rank":0,"incarnation":37,"state":"up:active","state_seq":4,"addr":"172.21.15.116:6833/3841609112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3841609112},{"type":"v1","addr":"172.21.15.116:6833","nonce":3841609112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5571,"qdb_cluster":[5571]},"id":2}]} 2024-04-25T00:06:23.817 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 41 2024-04-25T00:06:23.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.818+0000 7f651357c640 1 -- 172.21.15.116:0/2060617898 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f64f00678f0 msgr2=0x7f64f0069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:23.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.818+0000 7f651357c640 1 --2- 172.21.15.116:0/2060617898 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f64f00678f0 0x7f64f0069db0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f65080079c0 tx=0x7f6508008040 comp rx=0 tx=0).stop 2024-04-25T00:06:23.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.818+0000 7f651357c640 1 -- 172.21.15.116:0/2060617898 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f650c1562d0 msgr2=0x7f650c1f1ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:23.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.818+0000 7f651357c640 1 --2- 172.21.15.116:0/2060617898 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f650c1562d0 0x7f650c1f1ae0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f650401afd0 tx=0x7f65040196c0 comp rx=0 tx=0).stop 2024-04-25T00:06:23.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.818+0000 7f651357c640 1 -- 172.21.15.116:0/2060617898 shutdown_connections 2024-04-25T00:06:23.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.818+0000 7f651357c640 1 --2- 172.21.15.116:0/2060617898 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f64f00678f0 0x7f64f0069db0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:23.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.818+0000 7f651357c640 1 --2- 172.21.15.116:0/2060617898 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f650c1562d0 0x7f650c1f1ae0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:23.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.818+0000 7f651357c640 1 --2- 172.21.15.116:0/2060617898 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f650c152e60 0x7f650c14b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:23.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.818+0000 7f651357c640 1 --2- 172.21.15.116:0/2060617898 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f650c0c5010 0x7f650c06e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:23.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.818+0000 7f651357c640 1 -- 172.21.15.116:0/2060617898 >> 172.21.15.116:0/2060617898 conn(0x7f650c06c720 msgr2=0x7f650c0a94a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:23.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.818+0000 7f651357c640 1 -- 172.21.15.116:0/2060617898 shutdown_connections 2024-04-25T00:06:23.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:23.818+0000 7f651357c640 1 -- 172.21.15.116:0/2060617898 wait complete. 2024-04-25T00:06:23.835 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T00:06:25.836 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:06:25.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.966+0000 7ff19cfbd640 1 Processor -- start 2024-04-25T00:06:25.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.966+0000 7ff19cfbd640 1 -- start start 2024-04-25T00:06:25.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.970+0000 7ff19cfbd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff198148890 0x7ff198072590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:25.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.970+0000 7ff19cfbd640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff19806e890 0x7ff198071980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:25.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.970+0000 7ff19cfbd640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff198152c10 con 0x7ff198070dd0 2024-04-25T00:06:25.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.970+0000 7ff19cfbd640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7ff198151e50 con 0x7ff198148890 2024-04-25T00:06:25.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.970+0000 7ff19cfbd640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7ff198152130 con 0x7ff19806e890 2024-04-25T00:06:25.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.970+0000 7ff19659c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff198148890 0x7ff198072590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:25.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.970+0000 7ff19659c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff198148890 0x7ff198072590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35686/0 (socket says 172.21.15.116:35686) 2024-04-25T00:06:25.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.970+0000 7ff19659c640 1 -- 172.21.15.116:0/407759590 learned_addr learned my addr 172.21.15.116:0/407759590 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:06:25.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.970+0000 7ff195d9b640 1 --1- 172.21.15.116:0/407759590 >> v1:172.21.15.116:6789/0 conn(0x7ff198070dd0 0x7ff198071270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:50414/0 (socket says 172.21.15.116:50414) 2024-04-25T00:06:25.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.970+0000 7ff196d9d640 1 --2- 172.21.15.116:0/407759590 >> v2:172.21.15.174:3300/0 conn(0x7ff19806e890 0x7ff198071980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:25.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.970+0000 7ff19659c640 1 -- 172.21.15.116:0/407759590 >> v2:172.21.15.174:3300/0 conn(0x7ff19806e890 msgr2=0x7ff198071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:06:25.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.970+0000 7ff19659c640 1 --2- 172.21.15.116:0/407759590 >> v2:172.21.15.174:3300/0 conn(0x7ff19806e890 0x7ff198071980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:25.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.970+0000 7ff19659c640 1 -- 172.21.15.116:0/407759590 >> v1:172.21.15.116:6789/0 conn(0x7ff198070dd0 legacy=0x7ff198071270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:25.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.970+0000 7ff19659c640 1 -- 172.21.15.116:0/407759590 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff198152410 con 0x7ff198148890 2024-04-25T00:06:25.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.970+0000 7ff19559a640 1 -- 172.21.15.116:0/407759590 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1142884985 0 0) 0x7ff198152c10 con 0x7ff198070dd0 2024-04-25T00:06:25.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.970+0000 7ff196d9d640 1 --2- 172.21.15.116:0/407759590 >> v2:172.21.15.174:3300/0 conn(0x7ff19806e890 0x7ff198071980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:06:25.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.970+0000 7ff19659c640 1 --2- 172.21.15.116:0/407759590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff198148890 0x7ff198072590 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7ff18800bb10 tx=0x7ff18801ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6ab6e4106fdb5f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:25.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.970+0000 7ff19559a640 1 -- 172.21.15.116:0/407759590 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1880b4070 con 0x7ff198148890 2024-04-25T00:06:25.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.970+0000 7ff19559a640 1 -- 172.21.15.116:0/407759590 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1880a7ac0 con 0x7ff198148890 2024-04-25T00:06:25.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.970+0000 7ff19559a640 1 -- 172.21.15.116:0/407759590 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1880af440 con 0x7ff198148890 2024-04-25T00:06:25.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.974+0000 7ff19cfbd640 1 -- 172.21.15.116:0/407759590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff198148890 msgr2=0x7ff198072590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:25.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.974+0000 7ff19cfbd640 1 --2- 172.21.15.116:0/407759590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff198148890 0x7ff198072590 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7ff18800bb10 tx=0x7ff18801ebf0 comp rx=0 tx=0).stop 2024-04-25T00:06:25.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.974+0000 7ff19cfbd640 1 -- 172.21.15.116:0/407759590 shutdown_connections 2024-04-25T00:06:25.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.974+0000 7ff19cfbd640 1 --2- 172.21.15.116:0/407759590 >> v2:172.21.15.174:3300/0 conn(0x7ff19806e890 0x7ff198071980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:25.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.974+0000 7ff19cfbd640 1 --2- 172.21.15.116:0/407759590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff198148890 0x7ff198072590 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:25.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.974+0000 7ff19cfbd640 1 -- 172.21.15.116:0/407759590 >> 172.21.15.116:0/407759590 conn(0x7ff1980c3b00 msgr2=0x7ff1980c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:25.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.974+0000 7ff19cfbd640 1 -- 172.21.15.116:0/407759590 shutdown_connections 2024-04-25T00:06:25.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.974+0000 7ff19cfbd640 1 -- 172.21.15.116:0/407759590 wait complete. 2024-04-25T00:06:25.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.974+0000 7ff19cfbd640 1 Processor -- start 2024-04-25T00:06:25.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.974+0000 7ff19cfbd640 1 -- start start 2024-04-25T00:06:25.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.974+0000 7ff19cfbd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff19806e890 0x7ff19813ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:25.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.974+0000 7ff19cfbd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff198070dd0 0x7ff198141c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:25.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.974+0000 7ff19cfbd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff198148890 0x7ff198146c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:25.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.974+0000 7ff19cfbd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff1980ab920 con 0x7ff198148890 2024-04-25T00:06:25.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.974+0000 7ff19cfbd640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff1980ab7a0 con 0x7ff19806e890 2024-04-25T00:06:25.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.974+0000 7ff19cfbd640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff1980abaa0 con 0x7ff198070dd0 2024-04-25T00:06:25.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.974+0000 7ff196d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff198148890 0x7ff198146c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:25.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.974+0000 7ff196d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff198148890 0x7ff198146c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35690/0 (socket says 172.21.15.116:35690) 2024-04-25T00:06:25.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.974+0000 7ff196d9d640 1 -- 172.21.15.116:0/146901004 learned_addr learned my addr 172.21.15.116:0/146901004 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:06:25.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.974+0000 7ff19659c640 1 --2- 172.21.15.116:0/146901004 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff19806e890 0x7ff19813ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:25.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.974+0000 7ff195d9b640 1 --2- 172.21.15.116:0/146901004 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff198070dd0 0x7ff198141c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:25.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.974+0000 7ff196d9d640 1 -- 172.21.15.116:0/146901004 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff198070dd0 msgr2=0x7ff198141c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:25.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.974+0000 7ff196d9d640 1 --2- 172.21.15.116:0/146901004 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff198070dd0 0x7ff198141c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:25.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.974+0000 7ff196d9d640 1 -- 172.21.15.116:0/146901004 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff19806e890 msgr2=0x7ff19813ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:25.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.974+0000 7ff196d9d640 1 --2- 172.21.15.116:0/146901004 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff19806e890 0x7ff19813ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:25.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.974+0000 7ff196d9d640 1 -- 172.21.15.116:0/146901004 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1981ee010 con 0x7ff198148890 2024-04-25T00:06:25.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.974+0000 7ff196d9d640 1 --2- 172.21.15.116:0/146901004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff198148890 0x7ff198146c60 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7ff19000cea0 tx=0x7ff19000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:25.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.974+0000 7ff186ffd640 1 -- 172.21.15.116:0/146901004 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff190003ba0 con 0x7ff198148890 2024-04-25T00:06:25.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.974+0000 7ff186ffd640 1 -- 172.21.15.116:0/146901004 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff190003d60 con 0x7ff198148890 2024-04-25T00:06:25.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.974+0000 7ff19cfbd640 1 -- 172.21.15.116:0/146901004 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1981ee2b0 con 0x7ff198148890 2024-04-25T00:06:25.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.974+0000 7ff186ffd640 1 -- 172.21.15.116:0/146901004 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff19000d880 con 0x7ff198148890 2024-04-25T00:06:25.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.974+0000 7ff19cfbd640 1 -- 172.21.15.116:0/146901004 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff198142180 con 0x7ff198148890 2024-04-25T00:06:25.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.978+0000 7ff19cfbd640 1 -- 172.21.15.116:0/146901004 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff198068350 con 0x7ff198148890 2024-04-25T00:06:25.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.978+0000 7ff186ffd640 1 -- 172.21.15.116:0/146901004 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff19000da20 con 0x7ff198148890 2024-04-25T00:06:25.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.978+0000 7ff186ffd640 1 --2- 172.21.15.116:0/146901004 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff1780678c0 0x7ff178069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:25.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.978+0000 7ff186ffd640 1 -- 172.21.15.116:0/146901004 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7096+0+0 (secure 0 0 0) 0x7ff190086c10 con 0x7ff198148890 2024-04-25T00:06:25.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.978+0000 7ff19659c640 1 --2- 172.21.15.116:0/146901004 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff1780678c0 0x7ff178069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:25.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.978+0000 7ff19659c640 1 --2- 172.21.15.116:0/146901004 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff1780678c0 0x7ff178069d80 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff1880b1000 tx=0x7ff1880a9070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:25.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:25.982+0000 7ff186ffd640 1 -- 172.21.15.116:0/146901004 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ff1900571d0 con 0x7ff198148890 2024-04-25T00:06:26.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:26.178+0000 7ff19cfbd640 1 -- 172.21.15.116:0/146901004 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff19806e670 con 0x7ff198148890 2024-04-25T00:06:26.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:26.178+0000 7ff186ffd640 1 -- 172.21.15.116:0/146901004 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+6501 (secure 0 0 0) 0x7ff190056ad0 con 0x7ff198148890 2024-04-25T00:06:26.180 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:06:26.180 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1233829678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1233829678},{"type":"v1","addr":"172.21.15.116:6835","nonce":1233829678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:04:38.435723+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4879},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4879":{"gid":4879,"name":"b","rank":0,"incarnation":32,"state":"up:active","state_seq":15,"addr":"172.21.15.174:6837/1889314958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1889314958},{"type":"v1","addr":"172.21.15.174:6837","nonce":1889314958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4879,"qdb_cluster":[4879]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:04:48.496838+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5571},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5571":{"gid":5571,"name":"c","rank":0,"incarnation":37,"state":"up:active","state_seq":4,"addr":"172.21.15.116:6833/3841609112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3841609112},{"type":"v1","addr":"172.21.15.116:6833","nonce":3841609112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5571,"qdb_cluster":[5571]},"id":2}]} 2024-04-25T00:06:26.181 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 41 2024-04-25T00:06:26.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:26.182+0000 7ff19cfbd640 1 -- 172.21.15.116:0/146901004 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff1780678c0 msgr2=0x7ff178069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:26.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:26.182+0000 7ff19cfbd640 1 --2- 172.21.15.116:0/146901004 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff1780678c0 0x7ff178069d80 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff1880b1000 tx=0x7ff1880a9070 comp rx=0 tx=0).stop 2024-04-25T00:06:26.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:26.182+0000 7ff19cfbd640 1 -- 172.21.15.116:0/146901004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff198148890 msgr2=0x7ff198146c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:26.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:26.182+0000 7ff19cfbd640 1 --2- 172.21.15.116:0/146901004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff198148890 0x7ff198146c60 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7ff19000cea0 tx=0x7ff19000cf90 comp rx=0 tx=0).stop 2024-04-25T00:06:26.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:26.182+0000 7ff19cfbd640 1 -- 172.21.15.116:0/146901004 shutdown_connections 2024-04-25T00:06:26.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:26.182+0000 7ff19cfbd640 1 --2- 172.21.15.116:0/146901004 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff1780678c0 0x7ff178069d80 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:26.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:26.182+0000 7ff19cfbd640 1 --2- 172.21.15.116:0/146901004 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff198148890 0x7ff198146c60 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:26.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:26.182+0000 7ff19cfbd640 1 --2- 172.21.15.116:0/146901004 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff198070dd0 0x7ff198141c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:26.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:26.182+0000 7ff19cfbd640 1 --2- 172.21.15.116:0/146901004 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff19806e890 0x7ff19813ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:26.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:26.182+0000 7ff19cfbd640 1 -- 172.21.15.116:0/146901004 >> 172.21.15.116:0/146901004 conn(0x7ff1980c3b00 msgr2=0x7ff19806cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:26.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:26.182+0000 7ff19cfbd640 1 -- 172.21.15.116:0/146901004 shutdown_connections 2024-04-25T00:06:26.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:26.182+0000 7ff19cfbd640 1 -- 172.21.15.116:0/146901004 wait complete. 2024-04-25T00:06:26.199 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T00:06:28.200 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:06:28.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.326+0000 7fa610106640 1 Processor -- start 2024-04-25T00:06:28.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.326+0000 7fa610106640 1 -- start start 2024-04-25T00:06:28.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.330+0000 7fa610106640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa608150d20 0x7fa6080c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:28.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.330+0000 7fa610106640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fa6080c2f50 0x7fa6081538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:28.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.330+0000 7fa610106640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6080c3910 con 0x7fa6080a94a0 2024-04-25T00:06:28.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.330+0000 7fa610106640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fa608154530 con 0x7fa608150d20 2024-04-25T00:06:28.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.330+0000 7fa610106640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fa6080c16e0 con 0x7fa6080c2f50 2024-04-25T00:06:28.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.330+0000 7fa60dea2640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa608150d20 0x7fa6080c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:28.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.330+0000 7fa60dea2640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa608150d20 0x7fa6080c2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35700/0 (socket says 172.21.15.116:35700) 2024-04-25T00:06:28.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.330+0000 7fa60dea2640 1 -- 172.21.15.116:0/3578955962 learned_addr learned my addr 172.21.15.116:0/3578955962 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:06:28.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.330+0000 7fa60d6a1640 1 --2- 172.21.15.116:0/3578955962 >> v2:172.21.15.174:3300/0 conn(0x7fa6080c2f50 0x7fa6081538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:28.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.330+0000 7fa60dea2640 1 -- 172.21.15.116:0/3578955962 >> v2:172.21.15.174:3300/0 conn(0x7fa6080c2f50 msgr2=0x7fa6081538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:06:28.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.330+0000 7fa60dea2640 1 --2- 172.21.15.116:0/3578955962 >> v2:172.21.15.174:3300/0 conn(0x7fa6080c2f50 0x7fa6081538e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:28.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.330+0000 7fa60dea2640 1 -- 172.21.15.116:0/3578955962 >> v1:172.21.15.174:6789/0 conn(0x7fa6080a94a0 legacy=0x7fa608153e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:06:28.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.330+0000 7fa60dea2640 1 -- 172.21.15.116:0/3578955962 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6080c1400 con 0x7fa608150d20 2024-04-25T00:06:28.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.330+0000 7fa60dea2640 1 --2- 172.21.15.116:0/3578955962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa608150d20 0x7fa6080c2a10 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fa608070710 tx=0x7fa5fc01c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcf0fbd92e8c8c7e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:28.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.330+0000 7fa60cea0640 1 -- 172.21.15.116:0/3578955962 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5fc0ab5f0 con 0x7fa608150d20 2024-04-25T00:06:28.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.330+0000 7fa60cea0640 1 -- 172.21.15.116:0/3578955962 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5fc0b0070 con 0x7fa608150d20 2024-04-25T00:06:28.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.330+0000 7fa60cea0640 1 -- 172.21.15.116:0/3578955962 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5fc0ab9b0 con 0x7fa608150d20 2024-04-25T00:06:28.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.334+0000 7fa610106640 1 -- 172.21.15.116:0/3578955962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa608150d20 msgr2=0x7fa6080c2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:28.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.334+0000 7fa610106640 1 --2- 172.21.15.116:0/3578955962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa608150d20 0x7fa6080c2a10 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fa608070710 tx=0x7fa5fc01c4e0 comp rx=0 tx=0).stop 2024-04-25T00:06:28.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.334+0000 7fa610106640 1 -- 172.21.15.116:0/3578955962 shutdown_connections 2024-04-25T00:06:28.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.334+0000 7fa610106640 1 --2- 172.21.15.116:0/3578955962 >> v2:172.21.15.174:3300/0 conn(0x7fa6080c2f50 0x7fa6081538e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:28.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.334+0000 7fa610106640 1 --2- 172.21.15.116:0/3578955962 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa608150d20 0x7fa6080c2a10 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:28.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.334+0000 7fa610106640 1 -- 172.21.15.116:0/3578955962 >> 172.21.15.116:0/3578955962 conn(0x7fa6080c3b00 msgr2=0x7fa6080c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:28.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.334+0000 7fa610106640 1 -- 172.21.15.116:0/3578955962 shutdown_connections 2024-04-25T00:06:28.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.334+0000 7fa610106640 1 -- 172.21.15.116:0/3578955962 wait complete. 2024-04-25T00:06:28.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.334+0000 7fa610106640 1 Processor -- start 2024-04-25T00:06:28.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.334+0000 7fa610106640 1 -- start start 2024-04-25T00:06:28.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.334+0000 7fa610106640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa6080a94a0 0x7fa608144720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:28.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.334+0000 7fa610106640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6080c2f50 0x7fa608070d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:28.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.334+0000 7fa610106640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa608150d20 0x7fa60814ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:28.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.334+0000 7fa610106640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa6081608b0 con 0x7fa6080c2f50 2024-04-25T00:06:28.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.334+0000 7fa610106640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa608160730 con 0x7fa608150d20 2024-04-25T00:06:28.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.334+0000 7fa610106640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa608160a30 con 0x7fa6080a94a0 2024-04-25T00:06:28.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.334+0000 7fa60d6a1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6080c2f50 0x7fa608070d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:28.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.334+0000 7fa60d6a1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6080c2f50 0x7fa608070d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35714/0 (socket says 172.21.15.116:35714) 2024-04-25T00:06:28.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.334+0000 7fa60d6a1640 1 -- 172.21.15.116:0/2599251040 learned_addr learned my addr 172.21.15.116:0/2599251040 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:06:28.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.334+0000 7fa60e6a3640 1 --2- 172.21.15.116:0/2599251040 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa608150d20 0x7fa60814ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:28.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.334+0000 7fa60dea2640 1 --2- 172.21.15.116:0/2599251040 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa6080a94a0 0x7fa608144720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:28.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.334+0000 7fa60d6a1640 1 -- 172.21.15.116:0/2599251040 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa6080a94a0 msgr2=0x7fa608144720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:28.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.334+0000 7fa60d6a1640 1 --2- 172.21.15.116:0/2599251040 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa6080a94a0 0x7fa608144720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:28.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.334+0000 7fa60d6a1640 1 -- 172.21.15.116:0/2599251040 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa608150d20 msgr2=0x7fa60814ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:28.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.334+0000 7fa60d6a1640 1 --2- 172.21.15.116:0/2599251040 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa608150d20 0x7fa60814ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:28.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.334+0000 7fa60d6a1640 1 -- 172.21.15.116:0/2599251040 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6080c6e30 con 0x7fa6080c2f50 2024-04-25T00:06:28.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.334+0000 7fa60d6a1640 1 --2- 172.21.15.116:0/2599251040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6080c2f50 0x7fa608070d40 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fa60400d800 tx=0x7fa60400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:28.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.334+0000 7fa5fa7fc640 1 -- 172.21.15.116:0/2599251040 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa60400b660 con 0x7fa6080c2f50 2024-04-25T00:06:28.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.334+0000 7fa5fa7fc640 1 -- 172.21.15.116:0/2599251040 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa604019070 con 0x7fa6080c2f50 2024-04-25T00:06:28.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.334+0000 7fa610106640 1 -- 172.21.15.116:0/2599251040 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6080c6b50 con 0x7fa6080c2f50 2024-04-25T00:06:28.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.334+0000 7fa5fa7fc640 1 -- 172.21.15.116:0/2599251040 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa60400ba00 con 0x7fa6080c2f50 2024-04-25T00:06:28.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.334+0000 7fa610106640 1 -- 172.21.15.116:0/2599251040 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa608072ab0 con 0x7fa6080c2f50 2024-04-25T00:06:28.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.338+0000 7fa610106640 1 -- 172.21.15.116:0/2599251040 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa608068350 con 0x7fa6080c2f50 2024-04-25T00:06:28.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.338+0000 7fa5fa7fc640 1 -- 172.21.15.116:0/2599251040 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa60400bba0 con 0x7fa6080c2f50 2024-04-25T00:06:28.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.338+0000 7fa5fa7fc640 1 --2- 172.21.15.116:0/2599251040 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa5ec0678c0 0x7fa5ec069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:28.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.338+0000 7fa5fa7fc640 1 -- 172.21.15.116:0/2599251040 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 7096+0+0 (secure 0 0 0) 0x7fa604086600 con 0x7fa6080c2f50 2024-04-25T00:06:28.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.338+0000 7fa60dea2640 1 --2- 172.21.15.116:0/2599251040 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa5ec0678c0 0x7fa5ec069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:28.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.338+0000 7fa60dea2640 1 --2- 172.21.15.116:0/2599251040 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa5ec0678c0 0x7fa5ec069d80 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc0af000 tx=0x7fa5fc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:28.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.342+0000 7fa5fa7fc640 1 -- 172.21.15.116:0/2599251040 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fa604056c60 con 0x7fa6080c2f50 2024-04-25T00:06:28.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.538+0000 7fa610106640 1 -- 172.21.15.116:0/2599251040 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa6081504b0 con 0x7fa6080c2f50 2024-04-25T00:06:28.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.538+0000 7fa5fa7fc640 1 -- 172.21.15.116:0/2599251040 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) ==== 94+0+6501 (secure 0 0 0) 0x7fa604056560 con 0x7fa6080c2f50 2024-04-25T00:06:28.540 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:06:28.541 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1233829678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1233829678},{"type":"v1","addr":"172.21.15.116:6835","nonce":1233829678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":41}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:04:38.435723+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":33,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4879},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4879":{"gid":4879,"name":"b","rank":0,"incarnation":32,"state":"up:active","state_seq":15,"addr":"172.21.15.174:6837/1889314958","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1889314958},{"type":"v1","addr":"172.21.15.174:6837","nonce":1889314958}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4879,"qdb_cluster":[4879]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:04:48.496838+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5571},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5571":{"gid":5571,"name":"c","rank":0,"incarnation":37,"state":"up:active","state_seq":4,"addr":"172.21.15.116:6833/3841609112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3841609112},{"type":"v1","addr":"172.21.15.116:6833","nonce":3841609112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5571,"qdb_cluster":[5571]},"id":2}]} 2024-04-25T00:06:28.541 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 41 2024-04-25T00:06:28.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.542+0000 7fa610106640 1 -- 172.21.15.116:0/2599251040 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa5ec0678c0 msgr2=0x7fa5ec069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:28.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.542+0000 7fa610106640 1 --2- 172.21.15.116:0/2599251040 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa5ec0678c0 0x7fa5ec069d80 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc0af000 tx=0x7fa5fc008040 comp rx=0 tx=0).stop 2024-04-25T00:06:28.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.542+0000 7fa610106640 1 -- 172.21.15.116:0/2599251040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6080c2f50 msgr2=0x7fa608070d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:28.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.542+0000 7fa610106640 1 --2- 172.21.15.116:0/2599251040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6080c2f50 0x7fa608070d40 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fa60400d800 tx=0x7fa60400dcc0 comp rx=0 tx=0).stop 2024-04-25T00:06:28.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.542+0000 7fa610106640 1 -- 172.21.15.116:0/2599251040 shutdown_connections 2024-04-25T00:06:28.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.542+0000 7fa610106640 1 --2- 172.21.15.116:0/2599251040 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa5ec0678c0 0x7fa5ec069d80 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:28.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.542+0000 7fa610106640 1 --2- 172.21.15.116:0/2599251040 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa608150d20 0x7fa60814ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:28.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.542+0000 7fa610106640 1 --2- 172.21.15.116:0/2599251040 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa6080c2f50 0x7fa608070d40 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:28.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.542+0000 7fa610106640 1 --2- 172.21.15.116:0/2599251040 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa6080a94a0 0x7fa608144720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:28.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.542+0000 7fa610106640 1 -- 172.21.15.116:0/2599251040 >> 172.21.15.116:0/2599251040 conn(0x7fa6080c3b00 msgr2=0x7fa6080aa330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:28.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.542+0000 7fa610106640 1 -- 172.21.15.116:0/2599251040 shutdown_connections 2024-04-25T00:06:28.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:28.542+0000 7fa610106640 1 -- 172.21.15.116:0/2599251040 wait complete. 2024-04-25T00:06:28.558 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T00:06:30.559 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:06:30.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.670+0000 7f85f31c6640 1 Processor -- start 2024-04-25T00:06:30.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.670+0000 7f85f31c6640 1 -- start start 2024-04-25T00:06:30.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.670+0000 7f85f31c6640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f85ec150bf0 0x7f85ec06c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:30.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.670+0000 7f85f31c6640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f85ec153570 0x7f85ec06eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:30.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.670+0000 7f85f31c6640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85ec153aa0 con 0x7f85ec06c760 2024-04-25T00:06:30.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.670+0000 7f85f31c6640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f85ec06e630 con 0x7f85ec153570 2024-04-25T00:06:30.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.670+0000 7f85f31c6640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f85ec150fd0 con 0x7f85ec150bf0 2024-04-25T00:06:30.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.670+0000 7f85ebfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f85ec153570 0x7f85ec06eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:30.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.670+0000 7f85f1763640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f85ec06c760 0x7f85ec06f000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:50426/0 (socket says 172.21.15.116:50426) 2024-04-25T00:06:30.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.670+0000 7f85f1763640 1 -- 172.21.15.116:0/44249129 learned_addr learned my addr 172.21.15.116:0/44249129 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:06:30.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.670+0000 7f85f0f62640 1 --2- 172.21.15.116:0/44249129 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f85ec150bf0 0x7f85ec06c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:30.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.670+0000 7f85ebfff640 1 -- 172.21.15.116:0/44249129 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f85ec150bf0 msgr2=0x7f85ec06c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:06:30.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.670+0000 7f85ebfff640 1 --2- 172.21.15.116:0/44249129 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f85ec150bf0 0x7f85ec06c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:30.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.670+0000 7f85ebfff640 1 -- 172.21.15.116:0/44249129 >> v1:172.21.15.116:6789/0 conn(0x7f85ec06c760 legacy=0x7f85ec06f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:30.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.670+0000 7f85ebfff640 1 -- 172.21.15.116:0/44249129 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85ec153d80 con 0x7f85ec153570 2024-04-25T00:06:30.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.670+0000 7f85eb7fe640 1 -- 172.21.15.116:0/44249129 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 504492328 0 0) 0x7f85ec153aa0 con 0x7f85ec06c760 2024-04-25T00:06:30.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.674+0000 7f85ebfff640 1 --2- 172.21.15.116:0/44249129 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85ec153570 0x7f85ec06eac0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f85d80097c0 tx=0x7f85d801c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a599efb7a4303e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:30.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.674+0000 7f85eb7fe640 1 -- 172.21.15.116:0/44249129 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85d80b1070 con 0x7f85ec153570 2024-04-25T00:06:30.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.674+0000 7f85eb7fe640 1 -- 172.21.15.116:0/44249129 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85d80a8050 con 0x7f85ec153570 2024-04-25T00:06:30.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.674+0000 7f85eb7fe640 1 -- 172.21.15.116:0/44249129 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85d80ac860 con 0x7f85ec153570 2024-04-25T00:06:30.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.674+0000 7f85f31c6640 1 -- 172.21.15.116:0/44249129 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85ec153570 msgr2=0x7f85ec06eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:30.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.674+0000 7f85f31c6640 1 --2- 172.21.15.116:0/44249129 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85ec153570 0x7f85ec06eac0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f85d80097c0 tx=0x7f85d801c3e0 comp rx=0 tx=0).stop 2024-04-25T00:06:30.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.674+0000 7f85f31c6640 1 -- 172.21.15.116:0/44249129 shutdown_connections 2024-04-25T00:06:30.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.674+0000 7f85f31c6640 1 --2- 172.21.15.116:0/44249129 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85ec153570 0x7f85ec06eac0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:30.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.674+0000 7f85f31c6640 1 --2- 172.21.15.116:0/44249129 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f85ec150bf0 0x7f85ec06c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:30.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.674+0000 7f85f31c6640 1 -- 172.21.15.116:0/44249129 >> 172.21.15.116:0/44249129 conn(0x7f85ec0c4270 msgr2=0x7f85ec0c4680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:30.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.674+0000 7f85f31c6640 1 -- 172.21.15.116:0/44249129 shutdown_connections 2024-04-25T00:06:30.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.674+0000 7f85f31c6640 1 -- 172.21.15.116:0/44249129 wait complete. 2024-04-25T00:06:30.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.678+0000 7f85f31c6640 1 Processor -- start 2024-04-25T00:06:30.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.678+0000 7f85f31c6640 1 -- start start 2024-04-25T00:06:30.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.678+0000 7f85f31c6640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f85ec06c760 0x7f85ec14e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:30.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.678+0000 7f85f31c6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f85ec150bf0 0x7f85ec15c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:30.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.678+0000 7f85f31c6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85ec153570 0x7f85ec1eed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:30.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.678+0000 7f85f31c6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f85ec0a72b0 con 0x7f85ec153570 2024-04-25T00:06:30.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.678+0000 7f85f31c6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f85ec0a7130 con 0x7f85ec150bf0 2024-04-25T00:06:30.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.678+0000 7f85f31c6640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f85ec160710 con 0x7f85ec06c760 2024-04-25T00:06:30.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.678+0000 7f85f1763640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85ec153570 0x7f85ec1eed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:30.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.678+0000 7f85f1763640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85ec153570 0x7f85ec1eed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35746/0 (socket says 172.21.15.116:35746) 2024-04-25T00:06:30.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.678+0000 7f85f1763640 1 -- 172.21.15.116:0/3954081449 learned_addr learned my addr 172.21.15.116:0/3954081449 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:06:30.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.678+0000 7f85f0f62640 1 --2- 172.21.15.116:0/3954081449 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f85ec06c760 0x7f85ec14e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:30.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.678+0000 7f85ebfff640 1 --2- 172.21.15.116:0/3954081449 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f85ec150bf0 0x7f85ec15c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:30.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.678+0000 7f85f1763640 1 -- 172.21.15.116:0/3954081449 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f85ec06c760 msgr2=0x7f85ec14e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:30.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.678+0000 7f85f1763640 1 --2- 172.21.15.116:0/3954081449 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f85ec06c760 0x7f85ec14e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:30.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.678+0000 7f85f1763640 1 -- 172.21.15.116:0/3954081449 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f85ec150bf0 msgr2=0x7f85ec15c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:30.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.678+0000 7f85f1763640 1 --2- 172.21.15.116:0/3954081449 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f85ec150bf0 0x7f85ec15c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:30.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.678+0000 7f85f1763640 1 -- 172.21.15.116:0/3954081449 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85ec0c74e0 con 0x7f85ec153570 2024-04-25T00:06:30.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.678+0000 7f85ebfff640 1 --2- 172.21.15.116:0/3954081449 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f85ec150bf0 0x7f85ec15c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:06:30.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.678+0000 7f85f0f62640 1 --2- 172.21.15.116:0/3954081449 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f85ec06c760 0x7f85ec14e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:06:30.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.678+0000 7f85f1763640 1 --2- 172.21.15.116:0/3954081449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85ec153570 0x7f85ec1eed50 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f85e400c7a0 tx=0x7f85e400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:30.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.678+0000 7f85e97fa640 1 -- 172.21.15.116:0/3954081449 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e4018070 con 0x7f85ec153570 2024-04-25T00:06:30.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.678+0000 7f85e97fa640 1 -- 172.21.15.116:0/3954081449 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85e400a640 con 0x7f85ec153570 2024-04-25T00:06:30.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.678+0000 7f85e97fa640 1 -- 172.21.15.116:0/3954081449 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e400aa70 con 0x7f85ec153570 2024-04-25T00:06:30.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.678+0000 7f85f31c6640 1 -- 172.21.15.116:0/3954081449 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85ec073340 con 0x7f85ec153570 2024-04-25T00:06:30.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.678+0000 7f85f31c6640 1 -- 172.21.15.116:0/3954081449 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85ec1ee830 con 0x7f85ec153570 2024-04-25T00:06:30.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.678+0000 7f85f31c6640 1 -- 172.21.15.116:0/3954081449 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85ec068a60 con 0x7f85ec153570 2024-04-25T00:06:30.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.678+0000 7f85e97fa640 1 -- 172.21.15.116:0/3954081449 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f85e401b070 con 0x7f85ec153570 2024-04-25T00:06:30.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.682+0000 7f85e97fa640 1 --2- 172.21.15.116:0/3954081449 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f85d00678c0 0x7f85d0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:30.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.682+0000 7f85e97fa640 1 -- 172.21.15.116:0/3954081449 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7182+0+0 (secure 0 0 0) 0x7f85e4085790 con 0x7f85ec153570 2024-04-25T00:06:30.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.682+0000 7f85f0f62640 1 --2- 172.21.15.116:0/3954081449 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f85d00678c0 0x7f85d0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:30.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.682+0000 7f85f0f62640 1 --2- 172.21.15.116:0/3954081449 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f85d00678c0 0x7f85d0069d80 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f85ec153c00 tx=0x7f85e0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:30.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.690+0000 7f85e97fa640 1 -- 172.21.15.116:0/3954081449 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f85e4055da0 con 0x7f85ec153570 2024-04-25T00:06:30.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.882+0000 7f85f31c6640 1 -- 172.21.15.116:0/3954081449 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f85ec074aa0 con 0x7f85ec153570 2024-04-25T00:06:30.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.886+0000 7f85e97fa640 1 -- 172.21.15.116:0/3954081449 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+5665 (secure 0 0 0) 0x7f85e40556a0 con 0x7f85ec153570 2024-04-25T00:06:30.885 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:06:30.886 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:06:30.441877+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5613":{"gid":5613,"name":"a","rank":0,"incarnation":42,"state":"up:replay","state_seq":1,"addr":"172.21.15.116:6835/1233829678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1233829678},{"type":"v1","addr":"172.21.15.116:6835","nonce":1233829678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:04:48.496838+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5571},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5571":{"gid":5571,"name":"c","rank":0,"incarnation":37,"state":"up:active","state_seq":4,"addr":"172.21.15.116:6833/3841609112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3841609112},{"type":"v1","addr":"172.21.15.116:6833","nonce":3841609112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5571,"qdb_cluster":[5571]},"id":2}]} 2024-04-25T00:06:30.886 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 42 2024-04-25T00:06:30.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.886+0000 7f85f31c6640 1 -- 172.21.15.116:0/3954081449 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f85d00678c0 msgr2=0x7f85d0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:30.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.886+0000 7f85f31c6640 1 --2- 172.21.15.116:0/3954081449 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f85d00678c0 0x7f85d0069d80 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f85ec153c00 tx=0x7f85e0008040 comp rx=0 tx=0).stop 2024-04-25T00:06:30.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.886+0000 7f85f31c6640 1 -- 172.21.15.116:0/3954081449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85ec153570 msgr2=0x7f85ec1eed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:30.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.886+0000 7f85f31c6640 1 --2- 172.21.15.116:0/3954081449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85ec153570 0x7f85ec1eed50 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f85e400c7a0 tx=0x7f85e400cc60 comp rx=0 tx=0).stop 2024-04-25T00:06:30.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.886+0000 7f85f31c6640 1 -- 172.21.15.116:0/3954081449 shutdown_connections 2024-04-25T00:06:30.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.886+0000 7f85f31c6640 1 --2- 172.21.15.116:0/3954081449 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f85d00678c0 0x7f85d0069d80 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:30.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.886+0000 7f85f31c6640 1 --2- 172.21.15.116:0/3954081449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85ec153570 0x7f85ec1eed50 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:30.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.886+0000 7f85f31c6640 1 --2- 172.21.15.116:0/3954081449 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f85ec150bf0 0x7f85ec15c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:30.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.886+0000 7f85f31c6640 1 --2- 172.21.15.116:0/3954081449 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f85ec06c760 0x7f85ec14e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:30.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.886+0000 7f85f31c6640 1 -- 172.21.15.116:0/3954081449 >> 172.21.15.116:0/3954081449 conn(0x7f85ec0c4270 msgr2=0x7f85ec0671c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:30.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.886+0000 7f85f31c6640 1 -- 172.21.15.116:0/3954081449 shutdown_connections 2024-04-25T00:06:30.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:30.886+0000 7f85f31c6640 1 -- 172.21.15.116:0/3954081449 wait complete. 2024-04-25T00:06:30.903 INFO:tasks.mds_thrash.fs.[a]:mds.b down, removed from mdsmap 2024-04-25T00:06:30.903 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-04-25T00:06:30.904 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:06:31.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.022+0000 7f9a9a552640 1 Processor -- start 2024-04-25T00:06:31.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.022+0000 7f9a9a552640 1 -- start start 2024-04-25T00:06:31.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.022+0000 7f9a9a552640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9a9406d5f0 0x7f9a94156c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:31.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.022+0000 7f9a9a552640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9a94155810 0x7f9a941571c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:31.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.022+0000 7f9a9a552640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a940c9150 con 0x7f9a94152d80 2024-04-25T00:06:31.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.022+0000 7f9a9a552640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f9a940c9d60 con 0x7f9a94155810 2024-04-25T00:06:31.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.022+0000 7f9a9a552640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f9a940c9430 con 0x7f9a9406d5f0 2024-04-25T00:06:31.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.022+0000 7f9a98aef640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9a94155810 0x7f9a941571c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:31.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.022+0000 7f9a98aef640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9a94155810 0x7f9a941571c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35756/0 (socket says 172.21.15.116:35756) 2024-04-25T00:06:31.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.022+0000 7f9a98aef640 1 -- 172.21.15.116:0/174747991 learned_addr learned my addr 172.21.15.116:0/174747991 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:06:31.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.022+0000 7f9a8b7fe640 1 --2- 172.21.15.116:0/174747991 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9a9406d5f0 0x7f9a94156c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:31.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.022+0000 7f9a98aef640 1 -- 172.21.15.116:0/174747991 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9a9406d5f0 msgr2=0x7f9a94156c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:06:31.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.022+0000 7f9a98aef640 1 --2- 172.21.15.116:0/174747991 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9a9406d5f0 0x7f9a94156c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:31.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.022+0000 7f9a98aef640 1 -- 172.21.15.116:0/174747991 >> v1:172.21.15.174:6789/0 conn(0x7f9a94152d80 legacy=0x7f9a9406bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:06:31.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.022+0000 7f9a98aef640 1 -- 172.21.15.116:0/174747991 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a940c9710 con 0x7f9a94155810 2024-04-25T00:06:31.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.022+0000 7f9a98aef640 1 --2- 172.21.15.116:0/174747991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a94155810 0x7f9a941571c0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c009e50 tx=0x7f9a8c01c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81467661ecf46cd4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:31.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.022+0000 7f9a937fe640 1 -- 172.21.15.116:0/174747991 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a8c0a5650 con 0x7f9a94155810 2024-04-25T00:06:31.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.022+0000 7f9a937fe640 1 -- 172.21.15.116:0/174747991 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a8c0a57f0 con 0x7f9a94155810 2024-04-25T00:06:31.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.022+0000 7f9a937fe640 1 -- 172.21.15.116:0/174747991 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a8c0a5bc0 con 0x7f9a94155810 2024-04-25T00:06:31.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.026+0000 7f9a9a552640 1 -- 172.21.15.116:0/174747991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a94155810 msgr2=0x7f9a941571c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:31.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.026+0000 7f9a9a552640 1 --2- 172.21.15.116:0/174747991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a94155810 0x7f9a941571c0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c009e50 tx=0x7f9a8c01c8a0 comp rx=0 tx=0).stop 2024-04-25T00:06:31.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.026+0000 7f9a9a552640 1 -- 172.21.15.116:0/174747991 shutdown_connections 2024-04-25T00:06:31.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.026+0000 7f9a9a552640 1 --2- 172.21.15.116:0/174747991 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a94155810 0x7f9a941571c0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:31.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.026+0000 7f9a9a552640 1 --2- 172.21.15.116:0/174747991 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9a9406d5f0 0x7f9a94156c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:31.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.026+0000 7f9a9a552640 1 -- 172.21.15.116:0/174747991 >> 172.21.15.116:0/174747991 conn(0x7f9a940acdd0 msgr2=0x7f9a940ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:31.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.026+0000 7f9a9a552640 1 -- 172.21.15.116:0/174747991 shutdown_connections 2024-04-25T00:06:31.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.026+0000 7f9a9a552640 1 -- 172.21.15.116:0/174747991 wait complete. 2024-04-25T00:06:31.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.026+0000 7f9a9a552640 1 Processor -- start 2024-04-25T00:06:31.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.026+0000 7f9a9a552640 1 -- start start 2024-04-25T00:06:31.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.026+0000 7f9a9a552640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a9406d5f0 0x7f9a940c3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:31.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.026+0000 7f9a9a552640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a94152d80 0x7f9a940c4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:31.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.026+0000 7f9a9a552640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9a94155810 0x7f9a941f1de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:31.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.026+0000 7f9a9a552640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9a940caf00 con 0x7f9a9406d5f0 2024-04-25T00:06:31.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.026+0000 7f9a9a552640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9a940cad80 con 0x7f9a94152d80 2024-04-25T00:06:31.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.026+0000 7f9a9a552640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f9a941628d0 con 0x7f9a94155810 2024-04-25T00:06:31.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.026+0000 7f9a93fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a9406d5f0 0x7f9a940c3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:31.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.026+0000 7f9a93fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a9406d5f0 0x7f9a940c3b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35762/0 (socket says 172.21.15.116:35762) 2024-04-25T00:06:31.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.026+0000 7f9a93fff640 1 -- 172.21.15.116:0/1880399773 learned_addr learned my addr 172.21.15.116:0/1880399773 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:06:31.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.026+0000 7f9a98aef640 1 --2- 172.21.15.116:0/1880399773 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9a94155810 0x7f9a941f1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:31.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.026+0000 7f9a8b7fe640 1 --2- 172.21.15.116:0/1880399773 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a94152d80 0x7f9a940c4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:31.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.026+0000 7f9a93fff640 1 -- 172.21.15.116:0/1880399773 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9a94155810 msgr2=0x7f9a941f1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:31.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.026+0000 7f9a93fff640 1 --2- 172.21.15.116:0/1880399773 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9a94155810 0x7f9a941f1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:31.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.026+0000 7f9a93fff640 1 -- 172.21.15.116:0/1880399773 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a94152d80 msgr2=0x7f9a940c4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:31.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.026+0000 7f9a93fff640 1 --2- 172.21.15.116:0/1880399773 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a94152d80 0x7f9a940c4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:31.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.026+0000 7f9a93fff640 1 -- 172.21.15.116:0/1880399773 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a941517c0 con 0x7f9a9406d5f0 2024-04-25T00:06:31.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.026+0000 7f9a8b7fe640 1 --2- 172.21.15.116:0/1880399773 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a94152d80 0x7f9a940c4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:06:31.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.026+0000 7f9a93fff640 1 --2- 172.21.15.116:0/1880399773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a9406d5f0 0x7f9a940c3b40 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f9a8400b440 tx=0x7f9a8400b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:31.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.026+0000 7f9a917fa640 1 -- 172.21.15.116:0/1880399773 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a84015020 con 0x7f9a9406d5f0 2024-04-25T00:06:31.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.026+0000 7f9a917fa640 1 -- 172.21.15.116:0/1880399773 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a84011890 con 0x7f9a9406d5f0 2024-04-25T00:06:31.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.026+0000 7f9a917fa640 1 -- 172.21.15.116:0/1880399773 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a84011c80 con 0x7f9a9406d5f0 2024-04-25T00:06:31.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.026+0000 7f9a9a552640 1 -- 172.21.15.116:0/1880399773 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a94151aa0 con 0x7f9a9406d5f0 2024-04-25T00:06:31.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.026+0000 7f9a9a552640 1 -- 172.21.15.116:0/1880399773 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a940c4080 con 0x7f9a9406d5f0 2024-04-25T00:06:31.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.030+0000 7f9a9a552640 1 -- 172.21.15.116:0/1880399773 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a94068350 con 0x7f9a9406d5f0 2024-04-25T00:06:31.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.030+0000 7f9a917fa640 1 -- 172.21.15.116:0/1880399773 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f9a84018020 con 0x7f9a9406d5f0 2024-04-25T00:06:31.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.030+0000 7f9a917fa640 1 --2- 172.21.15.116:0/1880399773 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9a740678c0 0x7f9a74069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:31.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.030+0000 7f9a917fa640 1 -- 172.21.15.116:0/1880399773 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7182+0+0 (secure 0 0 0) 0x7f9a84086010 con 0x7f9a9406d5f0 2024-04-25T00:06:31.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.030+0000 7f9a8b7fe640 1 --2- 172.21.15.116:0/1880399773 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9a740678c0 0x7f9a74069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:31.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.030+0000 7f9a8b7fe640 1 --2- 172.21.15.116:0/1880399773 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9a740678c0 0x7f9a74069d80 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9a940be5b0 tx=0x7f9a7c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:31.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.034+0000 7f9a917fa640 1 -- 172.21.15.116:0/1880399773 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f9a8401f030 con 0x7f9a9406d5f0 2024-04-25T00:06:31.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.214+0000 7f9a9a552640 1 -- 172.21.15.116:0/1880399773 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9a941f1a00 con 0x7f9a9406d5f0 2024-04-25T00:06:31.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.218+0000 7f9a917fa640 1 -- 172.21.15.116:0/1880399773 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+5665 (secure 0 0 0) 0x7f9a8400d040 con 0x7f9a9406d5f0 2024-04-25T00:06:31.218 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:06:31.218 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:06:30.441877+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5613":{"gid":5613,"name":"a","rank":0,"incarnation":42,"state":"up:replay","state_seq":1,"addr":"172.21.15.116:6835/1233829678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1233829678},{"type":"v1","addr":"172.21.15.116:6835","nonce":1233829678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:04:48.496838+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5571},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5571":{"gid":5571,"name":"c","rank":0,"incarnation":37,"state":"up:active","state_seq":4,"addr":"172.21.15.116:6833/3841609112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3841609112},{"type":"v1","addr":"172.21.15.116:6833","nonce":3841609112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5571,"qdb_cluster":[5571]},"id":2}]} 2024-04-25T00:06:31.218 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 42 2024-04-25T00:06:31.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.218+0000 7f9a9a552640 1 -- 172.21.15.116:0/1880399773 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9a740678c0 msgr2=0x7f9a74069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:31.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.218+0000 7f9a9a552640 1 --2- 172.21.15.116:0/1880399773 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9a740678c0 0x7f9a74069d80 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9a940be5b0 tx=0x7f9a7c008040 comp rx=0 tx=0).stop 2024-04-25T00:06:31.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.218+0000 7f9a9a552640 1 -- 172.21.15.116:0/1880399773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a9406d5f0 msgr2=0x7f9a940c3b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:31.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.218+0000 7f9a9a552640 1 --2- 172.21.15.116:0/1880399773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a9406d5f0 0x7f9a940c3b40 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f9a8400b440 tx=0x7f9a8400b900 comp rx=0 tx=0).stop 2024-04-25T00:06:31.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.218+0000 7f9a9a552640 1 -- 172.21.15.116:0/1880399773 shutdown_connections 2024-04-25T00:06:31.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.218+0000 7f9a9a552640 1 --2- 172.21.15.116:0/1880399773 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9a740678c0 0x7f9a74069d80 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:31.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.218+0000 7f9a9a552640 1 --2- 172.21.15.116:0/1880399773 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9a94155810 0x7f9a941f1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:31.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.218+0000 7f9a9a552640 1 --2- 172.21.15.116:0/1880399773 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9a94152d80 0x7f9a940c4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:31.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.218+0000 7f9a9a552640 1 --2- 172.21.15.116:0/1880399773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9a9406d5f0 0x7f9a940c3b40 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:31.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.218+0000 7f9a9a552640 1 -- 172.21.15.116:0/1880399773 >> 172.21.15.116:0/1880399773 conn(0x7f9a940acdd0 msgr2=0x7f9a94154ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:31.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.218+0000 7f9a9a552640 1 -- 172.21.15.116:0/1880399773 shutdown_connections 2024-04-25T00:06:31.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:31.218+0000 7f9a9a552640 1 -- 172.21.15.116:0/1880399773 wait complete. 2024-04-25T00:06:31.237 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 42, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:04.678737+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 42, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5613": { "addr": "172.21.15.116:6835/1233829678", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6834", "nonce": 1233829678, "type": "v2" }, { "addr": "172.21.15.116:6835", "nonce": 1233829678, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5613, "incarnation": 42, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 35, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T00:06:30.441877+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5613 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:18.825887+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 40, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5571": { "addr": "172.21.15.116:6833/3841609112", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6832", "nonce": 3841609112, "type": "v2" }, { "addr": "172.21.15.116:6833", "nonce": 3841609112, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5571, "incarnation": 37, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 4 } }, "last_failure": 0, "last_failure_osd_epoch": 34, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-04-25T00:04:48.496838+0000", "qdb_cluster": [ 5571 ], "qdb_leader": 5571, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5571 } } } ], "standbys": [ { "addr": "172.21.15.174:6835/2883220846", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6834", "nonce": 2883220846, "type": "v2" }, { "addr": "172.21.15.174:6835", "nonce": 2883220846, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4235, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-25T00:06:33.238 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:06:33.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.370+0000 7ff93b0e9640 1 Processor -- start 2024-04-25T00:06:33.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.370+0000 7ff93b0e9640 1 -- start start 2024-04-25T00:06:33.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.370+0000 7ff93b0e9640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff93414c720 0x7ff9341563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:33.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.370+0000 7ff93b0e9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff934071680 0x7ff934071ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:33.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.370+0000 7ff93b0e9640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff934071220 con 0x7ff9340a94a0 2024-04-25T00:06:33.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.370+0000 7ff93b0e9640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7ff934153460 con 0x7ff934071680 2024-04-25T00:06:33.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.370+0000 7ff93b0e9640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7ff934152420 con 0x7ff93414c720 2024-04-25T00:06:33.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.370+0000 7ff939686640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff934071680 0x7ff934071ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:33.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.370+0000 7ff939686640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff934071680 0x7ff934071ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35774/0 (socket says 172.21.15.116:35774) 2024-04-25T00:06:33.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.370+0000 7ff939686640 1 -- 172.21.15.116:0/3033412125 learned_addr learned my addr 172.21.15.116:0/3033412125 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:06:33.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.370+0000 7ff938e85640 1 --2- 172.21.15.116:0/3033412125 >> v2:172.21.15.174:3300/0 conn(0x7ff93414c720 0x7ff9341563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:33.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.370+0000 7ff939686640 1 -- 172.21.15.116:0/3033412125 >> v2:172.21.15.174:3300/0 conn(0x7ff93414c720 msgr2=0x7ff9341563d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:06:33.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.370+0000 7ff939686640 1 --2- 172.21.15.116:0/3033412125 >> v2:172.21.15.174:3300/0 conn(0x7ff93414c720 0x7ff9341563d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:33.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.370+0000 7ff939686640 1 -- 172.21.15.116:0/3033412125 >> v1:172.21.15.174:6789/0 conn(0x7ff9340a94a0 legacy=0x7ff9340c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:33.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.370+0000 7ff939686640 1 -- 172.21.15.116:0/3033412125 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff93414cb00 con 0x7ff934071680 2024-04-25T00:06:33.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.370+0000 7ff938e85640 1 --2- 172.21.15.116:0/3033412125 >> v2:172.21.15.174:3300/0 conn(0x7ff93414c720 0x7ff9341563d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:06:33.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.374+0000 7ff939686640 1 --2- 172.21.15.116:0/3033412125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff934071680 0x7ff934071ab0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7ff934070190 tx=0x7ff92c01eb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cbb681c9474030b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:33.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.374+0000 7ff9337fe640 1 -- 172.21.15.116:0/3033412125 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff92c00b2d0 con 0x7ff934071680 2024-04-25T00:06:33.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.374+0000 7ff9337fe640 1 -- 172.21.15.116:0/3033412125 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff92c00b470 con 0x7ff934071680 2024-04-25T00:06:33.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.374+0000 7ff9337fe640 1 -- 172.21.15.116:0/3033412125 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff92c0a7b20 con 0x7ff934071680 2024-04-25T00:06:33.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.374+0000 7ff93b0e9640 1 -- 172.21.15.116:0/3033412125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff934071680 msgr2=0x7ff934071ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:33.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.374+0000 7ff93b0e9640 1 --2- 172.21.15.116:0/3033412125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff934071680 0x7ff934071ab0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7ff934070190 tx=0x7ff92c01eb30 comp rx=0 tx=0).stop 2024-04-25T00:06:33.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.374+0000 7ff93b0e9640 1 -- 172.21.15.116:0/3033412125 shutdown_connections 2024-04-25T00:06:33.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.374+0000 7ff93b0e9640 1 --2- 172.21.15.116:0/3033412125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff934071680 0x7ff934071ab0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:33.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.374+0000 7ff93b0e9640 1 --2- 172.21.15.116:0/3033412125 >> v2:172.21.15.174:3300/0 conn(0x7ff93414c720 0x7ff9341563d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:33.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.374+0000 7ff93b0e9640 1 -- 172.21.15.116:0/3033412125 >> 172.21.15.116:0/3033412125 conn(0x7ff9340c3b00 msgr2=0x7ff9340c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:33.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.374+0000 7ff93b0e9640 1 -- 172.21.15.116:0/3033412125 shutdown_connections 2024-04-25T00:06:33.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.374+0000 7ff93b0e9640 1 -- 172.21.15.116:0/3033412125 wait complete. 2024-04-25T00:06:33.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.374+0000 7ff93b0e9640 1 Processor -- start 2024-04-25T00:06:33.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.374+0000 7ff93b0e9640 1 -- start start 2024-04-25T00:06:33.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.374+0000 7ff93b0e9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff934071680 0x7ff934141cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:33.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.374+0000 7ff93b0e9640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff9340a94a0 0x7ff9341f2330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:33.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.374+0000 7ff93b0e9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff93414c720 0x7ff9340c0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:33.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.374+0000 7ff93b0e9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff9340c3120 con 0x7ff934071680 2024-04-25T00:06:33.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.374+0000 7ff93b0e9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff934152f40 con 0x7ff93414c720 2024-04-25T00:06:33.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.374+0000 7ff93b0e9640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff9340c32a0 con 0x7ff9340a94a0 2024-04-25T00:06:33.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.374+0000 7ff938e85640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff934071680 0x7ff934141cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:33.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.374+0000 7ff938e85640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff934071680 0x7ff934141cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35778/0 (socket says 172.21.15.116:35778) 2024-04-25T00:06:33.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.374+0000 7ff938e85640 1 -- 172.21.15.116:0/1084247723 learned_addr learned my addr 172.21.15.116:0/1084247723 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:06:33.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.374+0000 7ff933fff640 1 --2- 172.21.15.116:0/1084247723 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff9340a94a0 0x7ff9341f2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:33.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.374+0000 7ff939686640 1 --2- 172.21.15.116:0/1084247723 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff93414c720 0x7ff9340c0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:33.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.378+0000 7ff938e85640 1 -- 172.21.15.116:0/1084247723 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff9340a94a0 msgr2=0x7ff9341f2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:33.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.378+0000 7ff938e85640 1 --2- 172.21.15.116:0/1084247723 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff9340a94a0 0x7ff9341f2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:33.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.378+0000 7ff938e85640 1 -- 172.21.15.116:0/1084247723 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff93414c720 msgr2=0x7ff9340c0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:33.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.378+0000 7ff938e85640 1 --2- 172.21.15.116:0/1084247723 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff93414c720 0x7ff9340c0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:33.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.378+0000 7ff938e85640 1 -- 172.21.15.116:0/1084247723 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9340c6d50 con 0x7ff934071680 2024-04-25T00:06:33.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.378+0000 7ff938e85640 1 --2- 172.21.15.116:0/1084247723 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff934071680 0x7ff934141cc0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7ff92800d830 tx=0x7ff92800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:33.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.378+0000 7ff9317fa640 1 -- 172.21.15.116:0/1084247723 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff92800b6c0 con 0x7ff934071680 2024-04-25T00:06:33.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.378+0000 7ff9317fa640 1 -- 172.21.15.116:0/1084247723 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff928019070 con 0x7ff934071680 2024-04-25T00:06:33.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.378+0000 7ff93b0e9640 1 -- 172.21.15.116:0/1084247723 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9341422a0 con 0x7ff934071680 2024-04-25T00:06:33.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.378+0000 7ff9317fa640 1 -- 172.21.15.116:0/1084247723 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff92800baa0 con 0x7ff934071680 2024-04-25T00:06:33.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.378+0000 7ff93b0e9640 1 -- 172.21.15.116:0/1084247723 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9340c6270 con 0x7ff934071680 2024-04-25T00:06:33.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.378+0000 7ff93b0e9640 1 -- 172.21.15.116:0/1084247723 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff934068350 con 0x7ff934071680 2024-04-25T00:06:33.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.378+0000 7ff9317fa640 1 -- 172.21.15.116:0/1084247723 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff92801e460 con 0x7ff934071680 2024-04-25T00:06:33.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.378+0000 7ff9317fa640 1 --2- 172.21.15.116:0/1084247723 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff9180678c0 0x7ff918069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:33.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.378+0000 7ff9317fa640 1 -- 172.21.15.116:0/1084247723 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7182+0+0 (secure 0 0 0) 0x7ff928086810 con 0x7ff934071680 2024-04-25T00:06:33.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.378+0000 7ff933fff640 1 --2- 172.21.15.116:0/1084247723 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff9180678c0 0x7ff918069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:33.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.378+0000 7ff933fff640 1 --2- 172.21.15.116:0/1084247723 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff9180678c0 0x7ff918069d80 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ff934149bd0 tx=0x7ff920009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:33.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.382+0000 7ff9317fa640 1 -- 172.21.15.116:0/1084247723 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ff928056e20 con 0x7ff934071680 2024-04-25T00:06:33.536 INFO:tasks.ceph.mds.a.smithi116.stderr:2024-04-25T00:06:33.534+0000 7f4f51b0d640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T00:06:33.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.594+0000 7ff93b0e9640 1 -- 172.21.15.116:0/1084247723 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff9340c3640 con 0x7ff934071680 2024-04-25T00:06:33.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.594+0000 7ff9317fa640 1 -- 172.21.15.116:0/1084247723 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) ==== 94+0+5673 (secure 0 0 0) 0x7ff928056720 con 0x7ff934071680 2024-04-25T00:06:33.596 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:06:33.596 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:06:33.449109+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5613":{"gid":5613,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":28,"addr":"172.21.15.116:6835/1233829678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1233829678},{"type":"v1","addr":"172.21.15.116:6835","nonce":1233829678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5613,"qdb_cluster":[5613]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:04:48.496838+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5571},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5571":{"gid":5571,"name":"c","rank":0,"incarnation":37,"state":"up:active","state_seq":4,"addr":"172.21.15.116:6833/3841609112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3841609112},{"type":"v1","addr":"172.21.15.116:6833","nonce":3841609112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5571,"qdb_cluster":[5571]},"id":2}]} 2024-04-25T00:06:33.597 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 45 2024-04-25T00:06:33.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.598+0000 7ff93b0e9640 1 -- 172.21.15.116:0/1084247723 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff9180678c0 msgr2=0x7ff918069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:33.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.598+0000 7ff93b0e9640 1 --2- 172.21.15.116:0/1084247723 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff9180678c0 0x7ff918069d80 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ff934149bd0 tx=0x7ff920009040 comp rx=0 tx=0).stop 2024-04-25T00:06:33.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.598+0000 7ff93b0e9640 1 -- 172.21.15.116:0/1084247723 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff934071680 msgr2=0x7ff934141cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:33.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.598+0000 7ff93b0e9640 1 --2- 172.21.15.116:0/1084247723 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff934071680 0x7ff934141cc0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7ff92800d830 tx=0x7ff92800dcf0 comp rx=0 tx=0).stop 2024-04-25T00:06:33.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.598+0000 7ff93b0e9640 1 -- 172.21.15.116:0/1084247723 shutdown_connections 2024-04-25T00:06:33.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.598+0000 7ff93b0e9640 1 --2- 172.21.15.116:0/1084247723 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff9180678c0 0x7ff918069d80 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:33.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.598+0000 7ff93b0e9640 1 --2- 172.21.15.116:0/1084247723 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff93414c720 0x7ff9340c0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:33.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.598+0000 7ff93b0e9640 1 --2- 172.21.15.116:0/1084247723 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff9340a94a0 0x7ff9341f2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:33.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.598+0000 7ff93b0e9640 1 --2- 172.21.15.116:0/1084247723 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff934071680 0x7ff934141cc0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:33.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.598+0000 7ff93b0e9640 1 -- 172.21.15.116:0/1084247723 >> 172.21.15.116:0/1084247723 conn(0x7ff9340c3b00 msgr2=0x7ff9340c27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:33.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.598+0000 7ff93b0e9640 1 -- 172.21.15.116:0/1084247723 shutdown_connections 2024-04-25T00:06:33.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:33.598+0000 7ff93b0e9640 1 -- 172.21.15.116:0/1084247723 wait complete. 2024-04-25T00:06:33.615 INFO:tasks.mds_thrash.fs.[a]:mds.a has gained rank=0, replacing gid=4879 2024-04-25T00:06:33.616 INFO:tasks.mds_thrash.fs.[a]:waiting for 3 secs before reviving mds.b 2024-04-25T00:06:35.602 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:06:35.605 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:06:36.616 INFO:tasks.mds_thrash.fs.[a]:reviving mds.b 2024-04-25T00:06:36.616 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-25T00:06:36.616 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-25T00:06:36.619 INFO:tasks.ceph.mds.b:Started 2024-04-25T00:06:36.619 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:06:36.695 INFO:tasks.ceph.mds.b.smithi174.stdout:starting mds.b at 2024-04-25T00:06:36.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.758+0000 7f3f393d5640 1 Processor -- start 2024-04-25T00:06:36.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.758+0000 7f3f393d5640 1 -- start start 2024-04-25T00:06:36.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.762+0000 7f3f393d5640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f34153170 0x7f3f340c3060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:36.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.762+0000 7f3f393d5640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f340c22a0 con 0x7f3f3406c480 2024-04-25T00:06:36.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.762+0000 7f3f393d5640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f340c2580 con 0x7f3f340c1c30 2024-04-25T00:06:36.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.762+0000 7f3f32ffd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f3f3406c480 0x7f3f34071e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:59534/0 (socket says 172.21.15.116:59534) 2024-04-25T00:06:36.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.762+0000 7f3f32ffd640 1 -- 172.21.15.116:0/4180937572 learned_addr learned my addr 172.21.15.116:0/4180937572 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:06:36.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.762+0000 7f3f393d5640 1 -- 172.21.15.116:0/4180937572 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f3f340c35a0 con 0x7f3f34153170 2024-04-25T00:06:36.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.762+0000 7f3f337fe640 1 --2- 172.21.15.116:0/4180937572 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f34153170 0x7f3f340c3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:36.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.762+0000 7f3f337fe640 1 -- 172.21.15.116:0/4180937572 >> v1:172.21.15.174:6789/0 conn(0x7f3f340c1c30 legacy=0x7f3f3414fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:36.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.762+0000 7f3f337fe640 1 -- 172.21.15.116:0/4180937572 >> v1:172.21.15.116:6789/0 conn(0x7f3f3406c480 legacy=0x7f3f34071e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:36.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.762+0000 7f3f337fe640 1 -- 172.21.15.116:0/4180937572 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f340c2860 con 0x7f3f34153170 2024-04-25T00:06:36.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.762+0000 7f3f337fe640 1 --2- 172.21.15.116:0/4180937572 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f34153170 0x7f3f340c3060 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c009dc0 tx=0x7f3f2c01eba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2b82b37c75d9f3c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:36.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.766+0000 7f3f31ffb640 1 -- 172.21.15.116:0/4180937572 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f2c0b4070 con 0x7f3f34153170 2024-04-25T00:06:36.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.766+0000 7f3f31ffb640 1 -- 172.21.15.116:0/4180937572 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f2c0a7970 con 0x7f3f34153170 2024-04-25T00:06:36.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.766+0000 7f3f31ffb640 1 -- 172.21.15.116:0/4180937572 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f2c0a7d40 con 0x7f3f34153170 2024-04-25T00:06:36.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.766+0000 7f3f393d5640 1 -- 172.21.15.116:0/4180937572 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f34153170 msgr2=0x7f3f340c3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:36.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.766+0000 7f3f393d5640 1 --2- 172.21.15.116:0/4180937572 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f34153170 0x7f3f340c3060 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c009dc0 tx=0x7f3f2c01eba0 comp rx=0 tx=0).stop 2024-04-25T00:06:36.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.766+0000 7f3f393d5640 1 -- 172.21.15.116:0/4180937572 shutdown_connections 2024-04-25T00:06:36.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.766+0000 7f3f393d5640 1 --2- 172.21.15.116:0/4180937572 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f34153170 0x7f3f340c3060 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:36.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.766+0000 7f3f393d5640 1 -- 172.21.15.116:0/4180937572 >> 172.21.15.116:0/4180937572 conn(0x7f3f340c4270 msgr2=0x7f3f340c4680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:36.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.766+0000 7f3f393d5640 1 -- 172.21.15.116:0/4180937572 shutdown_connections 2024-04-25T00:06:36.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.766+0000 7f3f393d5640 1 -- 172.21.15.116:0/4180937572 wait complete. 2024-04-25T00:06:36.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.766+0000 7f3f393d5640 1 Processor -- start 2024-04-25T00:06:36.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.766+0000 7f3f393d5640 1 -- start start 2024-04-25T00:06:36.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.766+0000 7f3f393d5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f3406c480 0x7f3f340c6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:36.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.766+0000 7f3f393d5640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f340c1c30 0x7f3f341ec040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:36.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.766+0000 7f3f393d5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f34153170 0x7f3f341ec580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:36.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.766+0000 7f3f393d5640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3f3406f210 con 0x7f3f34153170 2024-04-25T00:06:36.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.766+0000 7f3f393d5640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3f3406f090 con 0x7f3f3406c480 2024-04-25T00:06:36.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.766+0000 7f3f393d5640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f3f3406f390 con 0x7f3f340c1c30 2024-04-25T00:06:36.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.766+0000 7f3f337fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f34153170 0x7f3f341ec580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:36.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.766+0000 7f3f337fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f34153170 0x7f3f341ec580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51096/0 (socket says 172.21.15.116:51096) 2024-04-25T00:06:36.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.766+0000 7f3f337fe640 1 -- 172.21.15.116:0/2412778097 learned_addr learned my addr 172.21.15.116:0/2412778097 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:06:36.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.766+0000 7f3f32ffd640 1 --2- 172.21.15.116:0/2412778097 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f3406c480 0x7f3f340c6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:36.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.770+0000 7f3f337fe640 1 -- 172.21.15.116:0/2412778097 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f340c1c30 msgr2=0x7f3f341ec040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:36.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.770+0000 7f3f337fe640 1 --2- 172.21.15.116:0/2412778097 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f340c1c30 0x7f3f341ec040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:36.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.770+0000 7f3f337fe640 1 -- 172.21.15.116:0/2412778097 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f3406c480 msgr2=0x7f3f340c6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:36.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.770+0000 7f3f337fe640 1 --2- 172.21.15.116:0/2412778097 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f3406c480 0x7f3f340c6610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:36.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.770+0000 7f3f337fe640 1 -- 172.21.15.116:0/2412778097 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f341ecac0 con 0x7f3f34153170 2024-04-25T00:06:36.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.770+0000 7f3f337fe640 1 --2- 172.21.15.116:0/2412778097 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f34153170 0x7f3f341ec580 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c0b2040 tx=0x7f3f2c01b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:36.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.770+0000 7f3f1b7fe640 1 -- 172.21.15.116:0/2412778097 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f2c0b4050 con 0x7f3f34153170 2024-04-25T00:06:36.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.770+0000 7f3f1b7fe640 1 -- 172.21.15.116:0/2412778097 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f2c0c0070 con 0x7f3f34153170 2024-04-25T00:06:36.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.770+0000 7f3f393d5640 1 -- 172.21.15.116:0/2412778097 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f34141610 con 0x7f3f34153170 2024-04-25T00:06:36.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.770+0000 7f3f1b7fe640 1 -- 172.21.15.116:0/2412778097 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f2c0afb30 con 0x7f3f34153170 2024-04-25T00:06:36.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.770+0000 7f3f393d5640 1 -- 172.21.15.116:0/2412778097 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f34141e60 con 0x7f3f34153170 2024-04-25T00:06:36.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.770+0000 7f3f393d5640 1 -- 172.21.15.116:0/2412778097 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f34068a60 con 0x7f3f34153170 2024-04-25T00:06:36.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.770+0000 7f3f1b7fe640 1 -- 172.21.15.116:0/2412778097 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3f2c0afcd0 con 0x7f3f34153170 2024-04-25T00:06:36.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.774+0000 7f3f1b7fe640 1 --2- 172.21.15.116:0/2412778097 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3f200678c0 0x7f3f20069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:36.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.774+0000 7f3f1b7fe640 1 -- 172.21.15.116:0/2412778097 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7182+0+0 (secure 0 0 0) 0x7f3f2c1233c0 con 0x7f3f34153170 2024-04-25T00:06:36.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.774+0000 7f3f32ffd640 1 --2- 172.21.15.116:0/2412778097 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3f200678c0 0x7f3f20069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:36.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.774+0000 7f3f32ffd640 1 --2- 172.21.15.116:0/2412778097 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3f200678c0 0x7f3f20069d80 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f3f24005890 tx=0x7f3f24009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:36.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.778+0000 7f3f1b7fe640 1 -- 172.21.15.116:0/2412778097 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f3f341ecac0 con 0x7f3f34153170 2024-04-25T00:06:36.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.990+0000 7f3f393d5640 1 -- 172.21.15.116:0/2412778097 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3f3414d2f0 con 0x7f3f34153170 2024-04-25T00:06:36.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.990+0000 7f3f1b7fe640 1 -- 172.21.15.116:0/2412778097 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+6501 (secure 0 0 0) 0x7f3f2c018790 con 0x7f3f34153170 2024-04-25T00:06:36.992 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:06:36.992 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4829,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3583791822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3583791822},{"type":"v1","addr":"172.21.15.174:6837","nonce":3583791822}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:06:33.449109+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5613":{"gid":5613,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":28,"addr":"172.21.15.116:6835/1233829678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1233829678},{"type":"v1","addr":"172.21.15.116:6835","nonce":1233829678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5613,"qdb_cluster":[5613]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:04:48.496838+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5571},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5571":{"gid":5571,"name":"c","rank":0,"incarnation":37,"state":"up:active","state_seq":4,"addr":"172.21.15.116:6833/3841609112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3841609112},{"type":"v1","addr":"172.21.15.116:6833","nonce":3841609112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5571,"qdb_cluster":[5571]},"id":2}]} 2024-04-25T00:06:36.992 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 46 2024-04-25T00:06:36.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.994+0000 7f3f393d5640 1 -- 172.21.15.116:0/2412778097 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3f200678c0 msgr2=0x7f3f20069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:36.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.994+0000 7f3f393d5640 1 --2- 172.21.15.116:0/2412778097 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3f200678c0 0x7f3f20069d80 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f3f24005890 tx=0x7f3f24009040 comp rx=0 tx=0).stop 2024-04-25T00:06:36.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.994+0000 7f3f393d5640 1 -- 172.21.15.116:0/2412778097 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f34153170 msgr2=0x7f3f341ec580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:36.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.994+0000 7f3f393d5640 1 --2- 172.21.15.116:0/2412778097 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f34153170 0x7f3f341ec580 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c0b2040 tx=0x7f3f2c01b940 comp rx=0 tx=0).stop 2024-04-25T00:06:36.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.994+0000 7f3f393d5640 1 -- 172.21.15.116:0/2412778097 shutdown_connections 2024-04-25T00:06:36.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.994+0000 7f3f393d5640 1 --2- 172.21.15.116:0/2412778097 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3f200678c0 0x7f3f20069d80 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:36.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.994+0000 7f3f393d5640 1 --2- 172.21.15.116:0/2412778097 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f34153170 0x7f3f341ec580 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:36.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.994+0000 7f3f393d5640 1 --2- 172.21.15.116:0/2412778097 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f340c1c30 0x7f3f341ec040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:36.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.994+0000 7f3f393d5640 1 --2- 172.21.15.116:0/2412778097 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f3406c480 0x7f3f340c6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:36.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.994+0000 7f3f393d5640 1 -- 172.21.15.116:0/2412778097 >> 172.21.15.116:0/2412778097 conn(0x7f3f340c4270 msgr2=0x7f3f340a8280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:36.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.994+0000 7f3f393d5640 1 -- 172.21.15.116:0/2412778097 shutdown_connections 2024-04-25T00:06:36.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:36.994+0000 7f3f393d5640 1 -- 172.21.15.116:0/2412778097 wait complete. 2024-04-25T00:06:37.016 INFO:tasks.mds_thrash.fs.[a]:mds.b reported in up:standby state 2024-04-25T00:06:37.016 INFO:tasks.mds_thrash.fs.[a]:waiting for 38 secs before thrashing 2024-04-25T00:06:45.678 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:06:45.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.809+0000 7f9fc2ea5640 1 Processor -- start 2024-04-25T00:06:45.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.809+0000 7f9fc2ea5640 1 -- start start 2024-04-25T00:06:45.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.813+0000 7f9fc2ea5640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f9fbc14d7e0 0x7f9fbc1540b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:45.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.813+0000 7f9fc2ea5640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9fbc0c3460 0x7f9fbc0a94a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:45.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.813+0000 7f9fc2ea5640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9fbc1545f0 con 0x7f9fbc0a99e0 2024-04-25T00:06:45.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.813+0000 7f9fc2ea5640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f9fbc0c2c30 con 0x7f9fbc0c3460 2024-04-25T00:06:45.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.813+0000 7f9fc2ea5640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f9fbc0702a0 con 0x7f9fbc14d7e0 2024-04-25T00:06:45.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.813+0000 7f9fc0c41640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f9fbc14d7e0 0x7f9fbc1540b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:45.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.813+0000 7f9fbbfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9fbc0c3460 0x7f9fbc0a94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:45.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.813+0000 7f9fc1442640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f9fbc0a99e0 0x7f9fbc1572e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:39884/0 (socket says 172.21.15.116:39884) 2024-04-25T00:06:45.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.813+0000 7f9fc1442640 1 -- 172.21.15.116:0/3527852161 learned_addr learned my addr 172.21.15.116:0/3527852161 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:06:45.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.813+0000 7f9fbbfff640 1 -- 172.21.15.116:0/3527852161 >> v2:172.21.15.174:3300/0 conn(0x7f9fbc14d7e0 msgr2=0x7f9fbc1540b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:06:45.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.813+0000 7f9fbbfff640 1 --2- 172.21.15.116:0/3527852161 >> v2:172.21.15.174:3300/0 conn(0x7f9fbc14d7e0 0x7f9fbc1540b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:45.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.813+0000 7f9fbbfff640 1 -- 172.21.15.116:0/3527852161 >> v1:172.21.15.174:6789/0 conn(0x7f9fbc0a99e0 legacy=0x7f9fbc1572e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:45.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.813+0000 7f9fbbfff640 1 -- 172.21.15.116:0/3527852161 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fbc06b210 con 0x7f9fbc0c3460 2024-04-25T00:06:45.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.813+0000 7f9fc0c41640 1 --2- 172.21.15.116:0/3527852161 >> v2:172.21.15.174:3300/0 conn(0x7f9fbc14d7e0 0x7f9fbc1540b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:06:45.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.813+0000 7f9fbbfff640 1 --2- 172.21.15.116:0/3527852161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fbc0c3460 0x7f9fbc0a94a0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc0c7f00 tx=0x7f9fa801c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c61c0b5f9e09e70 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:45.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.817+0000 7f9fbb7fe640 1 -- 172.21.15.116:0/3527852161 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fa80b0070 con 0x7f9fbc0c3460 2024-04-25T00:06:45.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.817+0000 7f9fbb7fe640 1 -- 172.21.15.116:0/3527852161 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fa80ab600 con 0x7f9fbc0c3460 2024-04-25T00:06:45.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.817+0000 7f9fbb7fe640 1 -- 172.21.15.116:0/3527852161 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fa80ab9d0 con 0x7f9fbc0c3460 2024-04-25T00:06:45.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.817+0000 7f9fc2ea5640 1 -- 172.21.15.116:0/3527852161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fbc0c3460 msgr2=0x7f9fbc0a94a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:45.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.817+0000 7f9fc2ea5640 1 --2- 172.21.15.116:0/3527852161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fbc0c3460 0x7f9fbc0a94a0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc0c7f00 tx=0x7f9fa801c3f0 comp rx=0 tx=0).stop 2024-04-25T00:06:45.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.817+0000 7f9fc2ea5640 1 -- 172.21.15.116:0/3527852161 shutdown_connections 2024-04-25T00:06:45.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.817+0000 7f9fc2ea5640 1 --2- 172.21.15.116:0/3527852161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fbc0c3460 0x7f9fbc0a94a0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:45.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.817+0000 7f9fc2ea5640 1 --2- 172.21.15.116:0/3527852161 >> v2:172.21.15.174:3300/0 conn(0x7f9fbc14d7e0 0x7f9fbc1540b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:45.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.817+0000 7f9fc2ea5640 1 -- 172.21.15.116:0/3527852161 >> 172.21.15.116:0/3527852161 conn(0x7f9fbc0c7fa0 msgr2=0x7f9fbc0c83b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:45.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.817+0000 7f9fc2ea5640 1 -- 172.21.15.116:0/3527852161 shutdown_connections 2024-04-25T00:06:45.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.817+0000 7f9fc2ea5640 1 -- 172.21.15.116:0/3527852161 wait complete. 2024-04-25T00:06:45.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.817+0000 7f9fc2ea5640 1 Processor -- start 2024-04-25T00:06:45.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.817+0000 7f9fc2ea5640 1 -- start start 2024-04-25T00:06:45.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.817+0000 7f9fc2ea5640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9fbc0a99e0 0x7f9fbc1ed000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:45.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.817+0000 7f9fc2ea5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fbc0c3460 0x7f9fbc14ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:45.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.817+0000 7f9fc2ea5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9fbc14d7e0 0x7f9fbc1ef900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:45.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.817+0000 7f9fc2ea5640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9fbc1579f0 con 0x7f9fbc0c3460 2024-04-25T00:06:45.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.817+0000 7f9fc2ea5640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9fbc156dc0 con 0x7f9fbc14d7e0 2024-04-25T00:06:45.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.817+0000 7f9fc2ea5640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f9fbc157b70 con 0x7f9fbc0a99e0 2024-04-25T00:06:45.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.817+0000 7f9fbbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fbc0c3460 0x7f9fbc14ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:45.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.821+0000 7f9fbbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fbc0c3460 0x7f9fbc14ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54074/0 (socket says 172.21.15.116:54074) 2024-04-25T00:06:45.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.821+0000 7f9fbbfff640 1 -- 172.21.15.116:0/3702276110 learned_addr learned my addr 172.21.15.116:0/3702276110 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:06:45.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.821+0000 7f9fc1442640 1 --2- 172.21.15.116:0/3702276110 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9fbc14d7e0 0x7f9fbc1ef900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:45.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.821+0000 7f9fc0c41640 1 --2- 172.21.15.116:0/3702276110 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9fbc0a99e0 0x7f9fbc1ed000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:45.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.821+0000 7f9fbbfff640 1 -- 172.21.15.116:0/3702276110 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9fbc0a99e0 msgr2=0x7f9fbc1ed000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:45.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.821+0000 7f9fbbfff640 1 --2- 172.21.15.116:0/3702276110 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9fbc0a99e0 0x7f9fbc1ed000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:45.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.821+0000 7f9fbbfff640 1 -- 172.21.15.116:0/3702276110 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9fbc14d7e0 msgr2=0x7f9fbc1ef900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:45.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.821+0000 7f9fbbfff640 1 --2- 172.21.15.116:0/3702276110 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9fbc14d7e0 0x7f9fbc1ef900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:45.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.821+0000 7f9fbbfff640 1 -- 172.21.15.116:0/3702276110 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fbc147c10 con 0x7f9fbc0c3460 2024-04-25T00:06:45.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.821+0000 7f9fc0c41640 1 --2- 172.21.15.116:0/3702276110 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9fbc0a99e0 0x7f9fbc1ed000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:06:45.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.821+0000 7f9fbbfff640 1 --2- 172.21.15.116:0/3702276110 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fbc0c3460 0x7f9fbc14ca70 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f9fa801a020 tx=0x7f9fa8019190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:45.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.821+0000 7f9fb97fa640 1 -- 172.21.15.116:0/3702276110 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fa80b0040 con 0x7f9fbc0c3460 2024-04-25T00:06:45.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.821+0000 7f9fb97fa640 1 -- 172.21.15.116:0/3702276110 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fa80b8030 con 0x7f9fbc0c3460 2024-04-25T00:06:45.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.821+0000 7f9fb97fa640 1 -- 172.21.15.116:0/3702276110 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fa80b4c60 con 0x7f9fbc0c3460 2024-04-25T00:06:45.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.821+0000 7f9fc2ea5640 1 -- 172.21.15.116:0/3702276110 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fbc147ef0 con 0x7f9fbc0c3460 2024-04-25T00:06:45.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.821+0000 7f9fc2ea5640 1 -- 172.21.15.116:0/3702276110 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fbc147560 con 0x7f9fbc0c3460 2024-04-25T00:06:45.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.821+0000 7f9fc2ea5640 1 -- 172.21.15.116:0/3702276110 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fbc068a60 con 0x7f9fbc0c3460 2024-04-25T00:06:45.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.821+0000 7f9fb97fa640 1 -- 172.21.15.116:0/3702276110 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f9fa80bc3d0 con 0x7f9fbc0c3460 2024-04-25T00:06:45.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.821+0000 7f9fb97fa640 1 --2- 172.21.15.116:0/3702276110 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9fa00678c0 0x7f9fa0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:45.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.821+0000 7f9fb97fa640 1 -- 172.21.15.116:0/3702276110 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7182+0+0 (secure 0 0 0) 0x7f9fa8123f40 con 0x7f9fbc0c3460 2024-04-25T00:06:45.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.825+0000 7f9fc0c41640 1 --2- 172.21.15.116:0/3702276110 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9fa00678c0 0x7f9fa0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:45.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.825+0000 7f9fc0c41640 1 --2- 172.21.15.116:0/3702276110 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9fa00678c0 0x7f9fa0069d80 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc14e310 tx=0x7f9fb0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:45.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:45.829+0000 7f9fb97fa640 1 -- 172.21.15.116:0/3702276110 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f9fa8016070 con 0x7f9fbc0c3460 2024-04-25T00:06:46.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.041+0000 7f9fc2ea5640 1 -- 172.21.15.116:0/3702276110 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9fbc070500 con 0x7f9fbc0c3460 2024-04-25T00:06:46.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.041+0000 7f9fb97fa640 1 -- 172.21.15.116:0/3702276110 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+6501 (secure 0 0 0) 0x7f9fa80f44c0 con 0x7f9fbc0c3460 2024-04-25T00:06:46.045 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:06:46.045 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4829,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3583791822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3583791822},{"type":"v1","addr":"172.21.15.174:6837","nonce":3583791822}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:06:33.449109+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5613":{"gid":5613,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":28,"addr":"172.21.15.116:6835/1233829678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1233829678},{"type":"v1","addr":"172.21.15.116:6835","nonce":1233829678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5613,"qdb_cluster":[5613]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:04:48.496838+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5571},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5571":{"gid":5571,"name":"c","rank":0,"incarnation":37,"state":"up:active","state_seq":4,"addr":"172.21.15.116:6833/3841609112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3841609112},{"type":"v1","addr":"172.21.15.116:6833","nonce":3841609112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5571,"qdb_cluster":[5571]},"id":2}]} 2024-04-25T00:06:46.045 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 46 2024-04-25T00:06:46.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.045+0000 7f9fc2ea5640 1 -- 172.21.15.116:0/3702276110 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9fa00678c0 msgr2=0x7f9fa0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:46.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.045+0000 7f9fc2ea5640 1 --2- 172.21.15.116:0/3702276110 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9fa00678c0 0x7f9fa0069d80 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc14e310 tx=0x7f9fb0008040 comp rx=0 tx=0).stop 2024-04-25T00:06:46.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.045+0000 7f9fc2ea5640 1 -- 172.21.15.116:0/3702276110 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fbc0c3460 msgr2=0x7f9fbc14ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:46.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.045+0000 7f9fc2ea5640 1 --2- 172.21.15.116:0/3702276110 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fbc0c3460 0x7f9fbc14ca70 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f9fa801a020 tx=0x7f9fa8019190 comp rx=0 tx=0).stop 2024-04-25T00:06:46.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.045+0000 7f9fc2ea5640 1 -- 172.21.15.116:0/3702276110 shutdown_connections 2024-04-25T00:06:46.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.045+0000 7f9fc2ea5640 1 --2- 172.21.15.116:0/3702276110 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9fa00678c0 0x7f9fa0069d80 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:46.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.045+0000 7f9fc2ea5640 1 --2- 172.21.15.116:0/3702276110 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9fbc14d7e0 0x7f9fbc1ef900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:46.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.045+0000 7f9fc2ea5640 1 --2- 172.21.15.116:0/3702276110 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9fbc0c3460 0x7f9fbc14ca70 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:46.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.045+0000 7f9fc2ea5640 1 --2- 172.21.15.116:0/3702276110 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9fbc0a99e0 0x7f9fbc1ed000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:46.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.045+0000 7f9fc2ea5640 1 -- 172.21.15.116:0/3702276110 >> 172.21.15.116:0/3702276110 conn(0x7f9fbc0c7fa0 msgr2=0x7f9fbc0c8380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:46.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.045+0000 7f9fc2ea5640 1 -- 172.21.15.116:0/3702276110 shutdown_connections 2024-04-25T00:06:46.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.045+0000 7f9fc2ea5640 1 -- 172.21.15.116:0/3702276110 wait complete. 2024-04-25T00:06:46.066 INFO:tasks.mds_thrash.fs.[b]:kill mds.c (rank=0) 2024-04-25T00:06:46.066 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-25T00:06:46.066 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:06:46.079 INFO:tasks.ceph.mds.c:Stopped 2024-04-25T00:06:46.079 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:06:46.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.257+0000 7fbc50e47640 1 Processor -- start 2024-04-25T00:06:46.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.257+0000 7fbc50e47640 1 -- start start 2024-04-25T00:06:46.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.261+0000 7fbc50e47640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbc4c0c7980 0x7fbc4c0c9610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:46.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.261+0000 7fbc50e47640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fbc4c0c3440 0x7fbc4c156280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:46.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.261+0000 7fbc50e47640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc4c1557c0 con 0x7fbc4c0c9230 2024-04-25T00:06:46.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.261+0000 7fbc50e47640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fbc4c1567c0 con 0x7fbc4c0c3440 2024-04-25T00:06:46.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.261+0000 7fbc50e47640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fbc4c0702d0 con 0x7fbc4c0c7980 2024-04-25T00:06:46.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.261+0000 7fbc49d9b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbc4c0c7980 0x7fbc4c0c9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:46.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.261+0000 7fbc49d9b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbc4c0c7980 0x7fbc4c0c9610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:44254/0 (socket says 172.21.15.116:44254) 2024-04-25T00:06:46.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.261+0000 7fbc49d9b640 1 -- 172.21.15.116:0/2434649501 learned_addr learned my addr 172.21.15.116:0/2434649501 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:06:46.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.261+0000 7fbc4ad9d640 1 --2- 172.21.15.116:0/2434649501 >> v2:172.21.15.174:3300/0 conn(0x7fbc4c0c3440 0x7fbc4c156280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:46.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.261+0000 7fbc49d9b640 1 -- 172.21.15.116:0/2434649501 >> v2:172.21.15.174:3300/0 conn(0x7fbc4c0c3440 msgr2=0x7fbc4c156280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:06:46.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.261+0000 7fbc49d9b640 1 --2- 172.21.15.116:0/2434649501 >> v2:172.21.15.174:3300/0 conn(0x7fbc4c0c3440 0x7fbc4c156280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:46.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.261+0000 7fbc49d9b640 1 -- 172.21.15.116:0/2434649501 >> v1:172.21.15.174:6789/0 conn(0x7fbc4c0c9230 legacy=0x7fbc4c156cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:46.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.261+0000 7fbc49d9b640 1 -- 172.21.15.116:0/2434649501 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc4c0705b0 con 0x7fbc4c0c7980 2024-04-25T00:06:46.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.261+0000 7fbc49d9b640 1 --2- 172.21.15.116:0/2434649501 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbc4c0c7980 0x7fbc4c0c9610 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fbc40009870 tx=0x7fbc4001c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7a5b06fbf31ce7a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:46.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.265+0000 7fbc4959a640 1 -- 172.21.15.116:0/2434649501 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc400b1070 con 0x7fbc4c0c7980 2024-04-25T00:06:46.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.265+0000 7fbc4959a640 1 -- 172.21.15.116:0/2434649501 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc400ac620 con 0x7fbc4c0c7980 2024-04-25T00:06:46.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.265+0000 7fbc4959a640 1 -- 172.21.15.116:0/2434649501 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc400ac9f0 con 0x7fbc4c0c7980 2024-04-25T00:06:46.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.265+0000 7fbc50e47640 1 -- 172.21.15.116:0/2434649501 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbc4c0c7980 msgr2=0x7fbc4c0c9610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:46.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.265+0000 7fbc50e47640 1 --2- 172.21.15.116:0/2434649501 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbc4c0c7980 0x7fbc4c0c9610 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fbc40009870 tx=0x7fbc4001c4d0 comp rx=0 tx=0).stop 2024-04-25T00:06:46.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.265+0000 7fbc50e47640 1 -- 172.21.15.116:0/2434649501 shutdown_connections 2024-04-25T00:06:46.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.265+0000 7fbc50e47640 1 --2- 172.21.15.116:0/2434649501 >> v2:172.21.15.174:3300/0 conn(0x7fbc4c0c3440 0x7fbc4c156280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:46.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.265+0000 7fbc50e47640 1 --2- 172.21.15.116:0/2434649501 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbc4c0c7980 0x7fbc4c0c9610 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:46.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.265+0000 7fbc50e47640 1 -- 172.21.15.116:0/2434649501 >> 172.21.15.116:0/2434649501 conn(0x7fbc4c074650 msgr2=0x7fbc4c074a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:46.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.265+0000 7fbc50e47640 1 -- 172.21.15.116:0/2434649501 shutdown_connections 2024-04-25T00:06:46.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.265+0000 7fbc50e47640 1 -- 172.21.15.116:0/2434649501 wait complete. 2024-04-25T00:06:46.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.265+0000 7fbc50e47640 1 Processor -- start 2024-04-25T00:06:46.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.265+0000 7fbc50e47640 1 -- start start 2024-04-25T00:06:46.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.265+0000 7fbc50e47640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbc4c0c3440 0x7fbc4c152850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:46.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.265+0000 7fbc50e47640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc4c0c7980 0x7fbc4c0c4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:46.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.265+0000 7fbc50e47640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc4c0c9230 0x7fbc4c0abe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:46.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.265+0000 7fbc50e47640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbc4c162cf0 con 0x7fbc4c0c9230 2024-04-25T00:06:46.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.265+0000 7fbc50e47640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbc4c162b70 con 0x7fbc4c0c7980 2024-04-25T00:06:46.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.265+0000 7fbc50e47640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fbc4c162e70 con 0x7fbc4c0c3440 2024-04-25T00:06:46.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.265+0000 7fbc4ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc4c0c9230 0x7fbc4c0abe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:46.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.265+0000 7fbc4ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc4c0c9230 0x7fbc4c0abe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54084/0 (socket says 172.21.15.116:54084) 2024-04-25T00:06:46.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.265+0000 7fbc4ad9d640 1 -- 172.21.15.116:0/694546307 learned_addr learned my addr 172.21.15.116:0/694546307 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:06:46.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.265+0000 7fbc49d9b640 1 --2- 172.21.15.116:0/694546307 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc4c0c7980 0x7fbc4c0c4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:46.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.265+0000 7fbc4a59c640 1 --2- 172.21.15.116:0/694546307 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbc4c0c3440 0x7fbc4c152850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:46.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.265+0000 7fbc4ad9d640 1 -- 172.21.15.116:0/694546307 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbc4c0c3440 msgr2=0x7fbc4c152850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:46.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.265+0000 7fbc4ad9d640 1 --2- 172.21.15.116:0/694546307 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbc4c0c3440 0x7fbc4c152850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:46.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.265+0000 7fbc4ad9d640 1 -- 172.21.15.116:0/694546307 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc4c0c7980 msgr2=0x7fbc4c0c4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:46.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.265+0000 7fbc4ad9d640 1 --2- 172.21.15.116:0/694546307 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc4c0c7980 0x7fbc4c0c4bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:46.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.265+0000 7fbc4ad9d640 1 -- 172.21.15.116:0/694546307 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc4c0acf50 con 0x7fbc4c0c9230 2024-04-25T00:06:46.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.265+0000 7fbc49d9b640 1 --2- 172.21.15.116:0/694546307 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc4c0c7980 0x7fbc4c0c4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:06:46.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.269+0000 7fbc4ad9d640 1 --2- 172.21.15.116:0/694546307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc4c0c9230 0x7fbc4c0abe10 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fbc4400efc0 tx=0x7fbc4400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:46.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.269+0000 7fbc3affd640 1 -- 172.21.15.116:0/694546307 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc44018020 con 0x7fbc4c0c9230 2024-04-25T00:06:46.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.269+0000 7fbc3affd640 1 -- 172.21.15.116:0/694546307 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc44014480 con 0x7fbc4c0c9230 2024-04-25T00:06:46.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.269+0000 7fbc50e47640 1 -- 172.21.15.116:0/694546307 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc4c0ad230 con 0x7fbc4c0c9230 2024-04-25T00:06:46.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.269+0000 7fbc3affd640 1 -- 172.21.15.116:0/694546307 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc44014870 con 0x7fbc4c0c9230 2024-04-25T00:06:46.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.269+0000 7fbc50e47640 1 -- 172.21.15.116:0/694546307 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc4c0c5640 con 0x7fbc4c0c9230 2024-04-25T00:06:46.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.269+0000 7fbc50e47640 1 -- 172.21.15.116:0/694546307 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc4c068a60 con 0x7fbc4c0c9230 2024-04-25T00:06:46.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.269+0000 7fbc3affd640 1 -- 172.21.15.116:0/694546307 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbc4401b020 con 0x7fbc4c0c9230 2024-04-25T00:06:46.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.269+0000 7fbc3affd640 1 --2- 172.21.15.116:0/694546307 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbc2c0678c0 0x7fbc2c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:46.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.269+0000 7fbc3affd640 1 -- 172.21.15.116:0/694546307 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7182+0+0 (secure 0 0 0) 0x7fbc44086ed0 con 0x7fbc4c0c9230 2024-04-25T00:06:46.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.269+0000 7fbc4a59c640 1 --2- 172.21.15.116:0/694546307 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbc2c0678c0 0x7fbc2c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:46.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.269+0000 7fbc4a59c640 1 --2- 172.21.15.116:0/694546307 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbc2c0678c0 0x7fbc2c069d80 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c0af020 tx=0x7fbc3c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:46.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.277+0000 7fbc3affd640 1 -- 172.21.15.116:0/694546307 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fbc44022030 con 0x7fbc4c0c9230 2024-04-25T00:06:46.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.529+0000 7fbc50e47640 1 -- 172.21.15.116:0/694546307 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbc4c0759d0 con 0x7fbc4c0c9230 2024-04-25T00:06:46.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.529+0000 7fbc3affd640 1 -- 172.21.15.116:0/694546307 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+6501 (secure 0 0 0) 0x7fbc44010040 con 0x7fbc4c0c9230 2024-04-25T00:06:46.531 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:06:46.532 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4829,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3583791822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3583791822},{"type":"v1","addr":"172.21.15.174:6837","nonce":3583791822}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:06:33.449109+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5613":{"gid":5613,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":28,"addr":"172.21.15.116:6835/1233829678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1233829678},{"type":"v1","addr":"172.21.15.116:6835","nonce":1233829678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5613,"qdb_cluster":[5613]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:04:48.496838+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5571},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5571":{"gid":5571,"name":"c","rank":0,"incarnation":37,"state":"up:active","state_seq":4,"addr":"172.21.15.116:6833/3841609112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3841609112},{"type":"v1","addr":"172.21.15.116:6833","nonce":3841609112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5571,"qdb_cluster":[5571]},"id":2}]} 2024-04-25T00:06:46.532 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 46 2024-04-25T00:06:46.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.533+0000 7fbc50e47640 1 -- 172.21.15.116:0/694546307 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbc2c0678c0 msgr2=0x7fbc2c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:46.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.533+0000 7fbc50e47640 1 --2- 172.21.15.116:0/694546307 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbc2c0678c0 0x7fbc2c069d80 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c0af020 tx=0x7fbc3c009040 comp rx=0 tx=0).stop 2024-04-25T00:06:46.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.533+0000 7fbc50e47640 1 -- 172.21.15.116:0/694546307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc4c0c9230 msgr2=0x7fbc4c0abe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:46.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.533+0000 7fbc50e47640 1 --2- 172.21.15.116:0/694546307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc4c0c9230 0x7fbc4c0abe10 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fbc4400efc0 tx=0x7fbc4400c490 comp rx=0 tx=0).stop 2024-04-25T00:06:46.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.533+0000 7fbc50e47640 1 -- 172.21.15.116:0/694546307 shutdown_connections 2024-04-25T00:06:46.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.533+0000 7fbc50e47640 1 --2- 172.21.15.116:0/694546307 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbc2c0678c0 0x7fbc2c069d80 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:46.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.533+0000 7fbc50e47640 1 --2- 172.21.15.116:0/694546307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc4c0c9230 0x7fbc4c0abe10 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:46.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.533+0000 7fbc50e47640 1 --2- 172.21.15.116:0/694546307 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc4c0c7980 0x7fbc4c0c4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:46.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.533+0000 7fbc50e47640 1 --2- 172.21.15.116:0/694546307 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbc4c0c3440 0x7fbc4c152850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:46.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.533+0000 7fbc50e47640 1 -- 172.21.15.116:0/694546307 >> 172.21.15.116:0/694546307 conn(0x7fbc4c074650 msgr2=0x7fbc4c156980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:46.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.533+0000 7fbc50e47640 1 -- 172.21.15.116:0/694546307 shutdown_connections 2024-04-25T00:06:46.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:46.533+0000 7fbc50e47640 1 -- 172.21.15.116:0/694546307 wait complete. 2024-04-25T00:06:46.550 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:06:48.551 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:06:48.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.673+0000 7faad3673640 1 Processor -- start 2024-04-25T00:06:48.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.673+0000 7faad3673640 1 -- start start 2024-04-25T00:06:48.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.677+0000 7faad3673640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7faacc0aeb20 0x7faacc0c9310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:48.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.677+0000 7faad3673640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faacc150f80 0x7faacc0c5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:48.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.677+0000 7faad3673640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faacc1535d0 con 0x7faacc0c9850 2024-04-25T00:06:48.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.677+0000 7faad3673640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7faacc153850 con 0x7faacc0aeb20 2024-04-25T00:06:48.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.677+0000 7faad3673640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7faacc153b30 con 0x7faacc150f80 2024-04-25T00:06:48.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.677+0000 7faad0c0e640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7faacc0c9850 0x7faacc150870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:43520/0 (socket says 172.21.15.116:43520) 2024-04-25T00:06:48.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.677+0000 7faad0c0e640 1 -- 172.21.15.116:0/2064343498 learned_addr learned my addr 172.21.15.116:0/2064343498 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:06:48.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.677+0000 7faad140f640 1 --2- 172.21.15.116:0/2064343498 >> v2:172.21.15.174:3300/0 conn(0x7faacc0aeb20 0x7faacc0c9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:48.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.677+0000 7faad1c10640 1 --2- 172.21.15.116:0/2064343498 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faacc150f80 0x7faacc0c5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:48.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.677+0000 7faac3fff640 1 -- 172.21.15.116:0/2064343498 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2701994940 0 0) 0x7faabc001120 con 0x7faacc0c9850 2024-04-25T00:06:48.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.677+0000 7faac3fff640 1 -- 172.21.15.116:0/2064343498 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faab8003530 con 0x7faacc0c9850 2024-04-25T00:06:48.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.677+0000 7faac3fff640 1 -- 172.21.15.116:0/2064343498 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3295997044 0 0) 0x7faab8003530 con 0x7faacc0c9850 2024-04-25T00:06:48.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.677+0000 7faac3fff640 1 -- 172.21.15.116:0/2064343498 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faacc150f80 msgr2=0x7faacc0c5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:06:48.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.677+0000 7faac3fff640 1 --2- 172.21.15.116:0/2064343498 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faacc150f80 0x7faacc0c5270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:48.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.677+0000 7faac3fff640 1 -- 172.21.15.116:0/2064343498 >> v2:172.21.15.174:3300/0 conn(0x7faacc0aeb20 msgr2=0x7faacc0c9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:06:48.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.677+0000 7faac3fff640 1 --2- 172.21.15.116:0/2064343498 >> v2:172.21.15.174:3300/0 conn(0x7faacc0aeb20 0x7faacc0c9310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:48.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.677+0000 7faac3fff640 1 -- 172.21.15.116:0/2064343498 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faacc153e10 con 0x7faacc0c9850 2024-04-25T00:06:48.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.677+0000 7faac3fff640 1 -- 172.21.15.116:0/2064343498 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7faabc001850 con 0x7faacc0c9850 2024-04-25T00:06:48.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.677+0000 7faac3fff640 1 -- 172.21.15.116:0/2064343498 >> v1:172.21.15.116:6789/0 conn(0x7faacc0c9850 legacy=0x7faacc150870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:48.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.677+0000 7faac3fff640 1 --2- 172.21.15.116:0/2064343498 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faab80039c0 0x7faab8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:48.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.677+0000 7faac3fff640 1 --2- 172.21.15.116:0/2064343498 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faacc150f80 0x7faab8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:48.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.677+0000 7faac3fff640 1 --2- 172.21.15.116:0/2064343498 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faab8004d50 0x7faab80051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:48.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.677+0000 7faac3fff640 1 -- 172.21.15.116:0/2064343498 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7faacc153850 con 0x7faab8004d50 2024-04-25T00:06:48.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.677+0000 7faac3fff640 1 -- 172.21.15.116:0/2064343498 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7faacc153b30 con 0x7faab80039c0 2024-04-25T00:06:48.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.677+0000 7faac3fff640 1 -- 172.21.15.116:0/2064343498 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7faab800ba00 con 0x7faacc150f80 2024-04-25T00:06:48.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.677+0000 7faad1c10640 1 --2- 172.21.15.116:0/2064343498 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faab8004d50 0x7faab80051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:48.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.677+0000 7faad0c0e640 1 --2- 172.21.15.116:0/2064343498 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faacc150f80 0x7faab8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:48.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.677+0000 7faad140f640 1 --2- 172.21.15.116:0/2064343498 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faab80039c0 0x7faab8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:48.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.677+0000 7faad1c10640 1 -- 172.21.15.116:0/2064343498 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faacc150f80 msgr2=0x7faab8004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:06:48.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.677+0000 7faad1c10640 1 --2- 172.21.15.116:0/2064343498 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faacc150f80 0x7faab8004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:48.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.677+0000 7faad1c10640 1 -- 172.21.15.116:0/2064343498 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faab80039c0 msgr2=0x7faab8003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:06:48.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.677+0000 7faad1c10640 1 --2- 172.21.15.116:0/2064343498 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faab80039c0 0x7faab8003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:48.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.677+0000 7faad1c10640 1 -- 172.21.15.116:0/2064343498 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faab8003530 con 0x7faab8004d50 2024-04-25T00:06:48.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.677+0000 7faad0c0e640 1 --2- 172.21.15.116:0/2064343498 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faacc150f80 0x7faab8004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:06:48.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.677+0000 7faad140f640 1 --2- 172.21.15.116:0/2064343498 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faab80039c0 0x7faab8003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:06:48.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.681+0000 7faad1c10640 1 --2- 172.21.15.116:0/2064343498 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faab8004d50 0x7faab80051d0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7faac4009fa0 tx=0x7faac401cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa3f71fcadade44f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:48.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.681+0000 7faac3fff640 1 -- 172.21.15.116:0/2064343498 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac40b2070 con 0x7faab8004d50 2024-04-25T00:06:48.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.681+0000 7faac3fff640 1 -- 172.21.15.116:0/2064343498 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faac40a5c00 con 0x7faab8004d50 2024-04-25T00:06:48.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.681+0000 7faac3fff640 1 -- 172.21.15.116:0/2064343498 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac4009bf0 con 0x7faab8004d50 2024-04-25T00:06:48.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.681+0000 7faad3673640 1 -- 172.21.15.116:0/2064343498 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faab8004d50 msgr2=0x7faab80051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:48.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.681+0000 7faad3673640 1 --2- 172.21.15.116:0/2064343498 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faab8004d50 0x7faab80051d0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7faac4009fa0 tx=0x7faac401cc40 comp rx=0 tx=0).stop 2024-04-25T00:06:48.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.681+0000 7faad1c10640 1 -- 172.21.15.116:0/2064343498 reap_dead start 2024-04-25T00:06:48.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.681+0000 7faad3673640 1 -- 172.21.15.116:0/2064343498 shutdown_connections 2024-04-25T00:06:48.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.681+0000 7faad3673640 1 -- 172.21.15.116:0/2064343498 >> 172.21.15.116:0/2064343498 conn(0x7faacc074650 msgr2=0x7faacc074a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:48.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.681+0000 7faad3673640 1 -- 172.21.15.116:0/2064343498 shutdown_connections 2024-04-25T00:06:48.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.681+0000 7faad3673640 1 -- 172.21.15.116:0/2064343498 wait complete. 2024-04-25T00:06:48.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.681+0000 7faad3673640 1 Processor -- start 2024-04-25T00:06:48.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.681+0000 7faad3673640 1 -- start start 2024-04-25T00:06:48.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.681+0000 7faad3673640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faacc0a9de0 0x7faacc0ac0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:48.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.681+0000 7faad3673640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faacc0adef0 0x7faacc06cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:48.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.681+0000 7faad3673640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faacc147bc0 0x7faacc147070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:48.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.681+0000 7faad3673640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7faacc0aef00 con 0x7faacc0a9de0 2024-04-25T00:06:48.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.681+0000 7faad3673640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7faacc1543d0 con 0x7faacc147bc0 2024-04-25T00:06:48.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.681+0000 7faad3673640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7faacc0af080 con 0x7faacc0adef0 2024-04-25T00:06:48.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.681+0000 7faad140f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faacc0a9de0 0x7faacc0ac0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:48.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.681+0000 7faad140f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faacc0a9de0 0x7faacc0ac0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54088/0 (socket says 172.21.15.116:54088) 2024-04-25T00:06:48.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.681+0000 7faad140f640 1 -- 172.21.15.116:0/3518168089 learned_addr learned my addr 172.21.15.116:0/3518168089 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:06:48.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.685+0000 7faad1c10640 1 --2- 172.21.15.116:0/3518168089 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faacc147bc0 0x7faacc147070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:48.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.685+0000 7faad0c0e640 1 --2- 172.21.15.116:0/3518168089 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faacc0adef0 0x7faacc06cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:48.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.685+0000 7faad140f640 1 -- 172.21.15.116:0/3518168089 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faacc0adef0 msgr2=0x7faacc06cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:48.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.685+0000 7faad140f640 1 --2- 172.21.15.116:0/3518168089 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faacc0adef0 0x7faacc06cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:48.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.685+0000 7faad140f640 1 -- 172.21.15.116:0/3518168089 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faacc147bc0 msgr2=0x7faacc147070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:48.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.685+0000 7faad140f640 1 --2- 172.21.15.116:0/3518168089 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faacc147bc0 0x7faacc147070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:48.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.685+0000 7faad140f640 1 -- 172.21.15.116:0/3518168089 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faacc0ac5f0 con 0x7faacc0a9de0 2024-04-25T00:06:48.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.685+0000 7faad1c10640 1 --2- 172.21.15.116:0/3518168089 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faacc147bc0 0x7faacc147070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:06:48.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.685+0000 7faad140f640 1 --2- 172.21.15.116:0/3518168089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faacc0a9de0 0x7faacc0ac0b0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7faac8010950 tx=0x7faac8010e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:48.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.685+0000 7faac1ffb640 1 -- 172.21.15.116:0/3518168089 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac800f040 con 0x7faacc0a9de0 2024-04-25T00:06:48.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.685+0000 7faac1ffb640 1 -- 172.21.15.116:0/3518168089 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faac8004220 con 0x7faacc0a9de0 2024-04-25T00:06:48.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.685+0000 7faad3673640 1 -- 172.21.15.116:0/3518168089 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faacc14e190 con 0x7faacc0a9de0 2024-04-25T00:06:48.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.685+0000 7faac1ffb640 1 -- 172.21.15.116:0/3518168089 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac800e9c0 con 0x7faacc0a9de0 2024-04-25T00:06:48.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.685+0000 7faad3673640 1 -- 172.21.15.116:0/3518168089 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faacc06dc70 con 0x7faacc0a9de0 2024-04-25T00:06:48.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.685+0000 7faad3673640 1 -- 172.21.15.116:0/3518168089 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faacc068a60 con 0x7faacc0a9de0 2024-04-25T00:06:48.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.685+0000 7faac1ffb640 1 -- 172.21.15.116:0/3518168089 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7faac800eb60 con 0x7faacc0a9de0 2024-04-25T00:06:48.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.685+0000 7faac1ffb640 1 --2- 172.21.15.116:0/3518168089 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faab00678f0 0x7faab0069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:48.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.685+0000 7faac1ffb640 1 -- 172.21.15.116:0/3518168089 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7182+0+0 (secure 0 0 0) 0x7faac80862a0 con 0x7faacc0a9de0 2024-04-25T00:06:48.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.685+0000 7faad0c0e640 1 --2- 172.21.15.116:0/3518168089 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faab00678f0 0x7faab0069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:48.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.689+0000 7faad0c0e640 1 --2- 172.21.15.116:0/3518168089 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faab00678f0 0x7faab0069db0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7faabc0019c0 tx=0x7faabc000ee0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:48.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.689+0000 7faac1ffb640 1 -- 172.21.15.116:0/3518168089 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7faac8018030 con 0x7faacc0a9de0 2024-04-25T00:06:48.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.937+0000 7faad3673640 1 -- 172.21.15.116:0/3518168089 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faacc0adaf0 con 0x7faacc0a9de0 2024-04-25T00:06:48.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.941+0000 7faac1ffb640 1 -- 172.21.15.116:0/3518168089 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+6501 (secure 0 0 0) 0x7faac8056890 con 0x7faacc0a9de0 2024-04-25T00:06:48.941 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:06:48.942 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4829,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3583791822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3583791822},{"type":"v1","addr":"172.21.15.174:6837","nonce":3583791822}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:06:33.449109+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5613":{"gid":5613,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":28,"addr":"172.21.15.116:6835/1233829678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1233829678},{"type":"v1","addr":"172.21.15.116:6835","nonce":1233829678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5613,"qdb_cluster":[5613]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:04:48.496838+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5571},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5571":{"gid":5571,"name":"c","rank":0,"incarnation":37,"state":"up:active","state_seq":4,"addr":"172.21.15.116:6833/3841609112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3841609112},{"type":"v1","addr":"172.21.15.116:6833","nonce":3841609112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5571,"qdb_cluster":[5571]},"id":2}]} 2024-04-25T00:06:48.942 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 46 2024-04-25T00:06:48.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.941+0000 7faad3673640 1 -- 172.21.15.116:0/3518168089 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faab00678f0 msgr2=0x7faab0069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:48.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.941+0000 7faad3673640 1 --2- 172.21.15.116:0/3518168089 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faab00678f0 0x7faab0069db0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7faabc0019c0 tx=0x7faabc000ee0 comp rx=0 tx=0).stop 2024-04-25T00:06:48.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.941+0000 7faad3673640 1 -- 172.21.15.116:0/3518168089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faacc0a9de0 msgr2=0x7faacc0ac0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:48.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.941+0000 7faad3673640 1 --2- 172.21.15.116:0/3518168089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faacc0a9de0 0x7faacc0ac0b0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7faac8010950 tx=0x7faac8010e10 comp rx=0 tx=0).stop 2024-04-25T00:06:48.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.941+0000 7faad3673640 1 -- 172.21.15.116:0/3518168089 shutdown_connections 2024-04-25T00:06:48.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.941+0000 7faad3673640 1 --2- 172.21.15.116:0/3518168089 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faab00678f0 0x7faab0069db0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:48.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.941+0000 7faad3673640 1 --2- 172.21.15.116:0/3518168089 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faacc147bc0 0x7faacc147070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:48.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.941+0000 7faad3673640 1 --2- 172.21.15.116:0/3518168089 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faacc0adef0 0x7faacc06cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:48.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.941+0000 7faad3673640 1 --2- 172.21.15.116:0/3518168089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faacc0a9de0 0x7faacc0ac0b0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:48.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.941+0000 7faad3673640 1 -- 172.21.15.116:0/3518168089 >> 172.21.15.116:0/3518168089 conn(0x7faacc074650 msgr2=0x7faacc0c5950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:48.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.941+0000 7faad3673640 1 -- 172.21.15.116:0/3518168089 shutdown_connections 2024-04-25T00:06:48.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:48.941+0000 7faad3673640 1 -- 172.21.15.116:0/3518168089 wait complete. 2024-04-25T00:06:48.961 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:06:50.962 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:06:51.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.093+0000 7fee3bf68640 1 Processor -- start 2024-04-25T00:06:51.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.093+0000 7fee3bf68640 1 -- start start 2024-04-25T00:06:51.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.097+0000 7fee3bf68640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fee34072080 0x7fee34159950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:51.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.097+0000 7fee3bf68640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fee341562a0 0x7fee340c5020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:51.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.097+0000 7fee3bf68640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee340730b0 con 0x7fee34152d80 2024-04-25T00:06:51.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.097+0000 7fee3bf68640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fee34072e60 con 0x7fee34072080 2024-04-25T00:06:51.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.097+0000 7fee3bf68640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fee34073390 con 0x7fee341562a0 2024-04-25T00:06:51.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.097+0000 7fee39503640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fee34072080 0x7fee34159950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:51.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.097+0000 7fee39503640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fee34072080 0x7fee34159950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54092/0 (socket says 172.21.15.116:54092) 2024-04-25T00:06:51.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.097+0000 7fee39503640 1 -- 172.21.15.116:0/2118234832 learned_addr learned my addr 172.21.15.116:0/2118234832 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:06:51.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.097+0000 7fee3a505640 1 --2- 172.21.15.116:0/2118234832 >> v2:172.21.15.174:3300/0 conn(0x7fee341562a0 0x7fee340c5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:51.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.097+0000 7fee39503640 1 -- 172.21.15.116:0/2118234832 >> v2:172.21.15.174:3300/0 conn(0x7fee341562a0 msgr2=0x7fee340c5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:06:51.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.097+0000 7fee39503640 1 --2- 172.21.15.116:0/2118234832 >> v2:172.21.15.174:3300/0 conn(0x7fee341562a0 0x7fee340c5020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:51.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.097+0000 7fee39503640 1 -- 172.21.15.116:0/2118234832 >> v1:172.21.15.116:6789/0 conn(0x7fee34152d80 legacy=0x7fee340c36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:51.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.097+0000 7fee39503640 1 -- 172.21.15.116:0/2118234832 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee34158e70 con 0x7fee34072080 2024-04-25T00:06:51.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.097+0000 7fee39503640 1 --2- 172.21.15.116:0/2118234832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee34072080 0x7fee34159950 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fee340c21e0 tx=0x7fee2401c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f460cf982e42069 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:51.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.097+0000 7fee38d02640 1 -- 172.21.15.116:0/2118234832 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee240ab660 con 0x7fee34072080 2024-04-25T00:06:51.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.097+0000 7fee38d02640 1 -- 172.21.15.116:0/2118234832 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee240b0070 con 0x7fee34072080 2024-04-25T00:06:51.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.097+0000 7fee38d02640 1 -- 172.21.15.116:0/2118234832 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee240aba20 con 0x7fee34072080 2024-04-25T00:06:51.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.101+0000 7fee3bf68640 1 -- 172.21.15.116:0/2118234832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee34072080 msgr2=0x7fee34159950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:51.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.101+0000 7fee3bf68640 1 --2- 172.21.15.116:0/2118234832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee34072080 0x7fee34159950 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fee340c21e0 tx=0x7fee2401c460 comp rx=0 tx=0).stop 2024-04-25T00:06:51.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.101+0000 7fee3bf68640 1 -- 172.21.15.116:0/2118234832 shutdown_connections 2024-04-25T00:06:51.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.101+0000 7fee3bf68640 1 --2- 172.21.15.116:0/2118234832 >> v2:172.21.15.174:3300/0 conn(0x7fee341562a0 0x7fee340c5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:51.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.101+0000 7fee3bf68640 1 --2- 172.21.15.116:0/2118234832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee34072080 0x7fee34159950 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:51.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.101+0000 7fee3bf68640 1 -- 172.21.15.116:0/2118234832 >> 172.21.15.116:0/2118234832 conn(0x7fee3406c720 msgr2=0x7fee3406cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:51.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.101+0000 7fee3bf68640 1 -- 172.21.15.116:0/2118234832 shutdown_connections 2024-04-25T00:06:51.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.101+0000 7fee3bf68640 1 -- 172.21.15.116:0/2118234832 wait complete. 2024-04-25T00:06:51.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.101+0000 7fee3bf68640 1 Processor -- start 2024-04-25T00:06:51.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.101+0000 7fee3bf68640 1 -- start start 2024-04-25T00:06:51.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.101+0000 7fee3bf68640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee34072080 0x7fee34143540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:51.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.101+0000 7fee3bf68640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fee34152d80 0x7fee340c6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:51.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.101+0000 7fee3bf68640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fee341562a0 0x7fee3414b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:51.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.101+0000 7fee3bf68640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fee34153160 con 0x7fee34072080 2024-04-25T00:06:51.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.101+0000 7fee3bf68640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fee341566b0 con 0x7fee341562a0 2024-04-25T00:06:51.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.101+0000 7fee3bf68640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fee341532e0 con 0x7fee34152d80 2024-04-25T00:06:51.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.101+0000 7fee39d04640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee34072080 0x7fee34143540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:51.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.101+0000 7fee39d04640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee34072080 0x7fee34143540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54098/0 (socket says 172.21.15.116:54098) 2024-04-25T00:06:51.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.101+0000 7fee39d04640 1 -- 172.21.15.116:0/872830854 learned_addr learned my addr 172.21.15.116:0/872830854 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:06:51.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.101+0000 7fee39503640 1 --2- 172.21.15.116:0/872830854 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fee34152d80 0x7fee340c6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:51.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.101+0000 7fee3a505640 1 --2- 172.21.15.116:0/872830854 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fee341562a0 0x7fee3414b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:51.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.101+0000 7fee39d04640 1 -- 172.21.15.116:0/872830854 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fee34152d80 msgr2=0x7fee340c6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:51.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.101+0000 7fee39d04640 1 --2- 172.21.15.116:0/872830854 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fee34152d80 0x7fee340c6eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:51.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.101+0000 7fee39d04640 1 -- 172.21.15.116:0/872830854 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fee341562a0 msgr2=0x7fee3414b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:51.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.101+0000 7fee39d04640 1 --2- 172.21.15.116:0/872830854 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fee341562a0 0x7fee3414b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:51.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.101+0000 7fee39d04640 1 -- 172.21.15.116:0/872830854 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee340c5e10 con 0x7fee34072080 2024-04-25T00:06:51.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.101+0000 7fee39503640 1 --2- 172.21.15.116:0/872830854 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fee34152d80 0x7fee340c6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:06:51.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.101+0000 7fee3a505640 1 --2- 172.21.15.116:0/872830854 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fee341562a0 0x7fee3414b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:06:51.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.101+0000 7fee39d04640 1 --2- 172.21.15.116:0/872830854 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee34072080 0x7fee34143540 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fee3000c450 tx=0x7fee3000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:51.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.101+0000 7fee2a7fc640 1 -- 172.21.15.116:0/872830854 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee30016020 con 0x7fee34072080 2024-04-25T00:06:51.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.101+0000 7fee2a7fc640 1 -- 172.21.15.116:0/872830854 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee300052d0 con 0x7fee34072080 2024-04-25T00:06:51.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.101+0000 7fee2a7fc640 1 -- 172.21.15.116:0/872830854 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee30007260 con 0x7fee34072080 2024-04-25T00:06:51.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.101+0000 7fee3bf68640 1 -- 172.21.15.116:0/872830854 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee340c60f0 con 0x7fee34072080 2024-04-25T00:06:51.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.101+0000 7fee3bf68640 1 -- 172.21.15.116:0/872830854 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee340c66b0 con 0x7fee34072080 2024-04-25T00:06:51.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.105+0000 7fee3bf68640 1 -- 172.21.15.116:0/872830854 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee34068a60 con 0x7fee34072080 2024-04-25T00:06:51.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.105+0000 7fee2a7fc640 1 -- 172.21.15.116:0/872830854 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fee30007400 con 0x7fee34072080 2024-04-25T00:06:51.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.105+0000 7fee2a7fc640 1 --2- 172.21.15.116:0/872830854 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fee18067910 0x7fee18069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:51.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.105+0000 7fee2a7fc640 1 -- 172.21.15.116:0/872830854 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7182+0+0 (secure 0 0 0) 0x7fee30085aa0 con 0x7fee34072080 2024-04-25T00:06:51.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.105+0000 7fee39503640 1 --2- 172.21.15.116:0/872830854 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fee18067910 0x7fee18069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:51.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.105+0000 7fee39503640 1 --2- 172.21.15.116:0/872830854 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fee18067910 0x7fee18069dd0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fee240af000 tx=0x7fee240a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:51.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.113+0000 7fee2a7fc640 1 -- 172.21.15.116:0/872830854 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fee30020020 con 0x7fee34072080 2024-04-25T00:06:51.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.349+0000 7fee3bf68640 1 -- 172.21.15.116:0/872830854 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fee3414f210 con 0x7fee34072080 2024-04-25T00:06:51.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.349+0000 7fee2a7fc640 1 -- 172.21.15.116:0/872830854 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+6501 (secure 0 0 0) 0x7fee30005480 con 0x7fee34072080 2024-04-25T00:06:51.350 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:06:51.350 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4829,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3583791822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3583791822},{"type":"v1","addr":"172.21.15.174:6837","nonce":3583791822}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:06:33.449109+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5613":{"gid":5613,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":28,"addr":"172.21.15.116:6835/1233829678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1233829678},{"type":"v1","addr":"172.21.15.116:6835","nonce":1233829678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5613,"qdb_cluster":[5613]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:04:48.496838+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5571},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5571":{"gid":5571,"name":"c","rank":0,"incarnation":37,"state":"up:active","state_seq":4,"addr":"172.21.15.116:6833/3841609112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3841609112},{"type":"v1","addr":"172.21.15.116:6833","nonce":3841609112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5571,"qdb_cluster":[5571]},"id":2}]} 2024-04-25T00:06:51.351 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 46 2024-04-25T00:06:51.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.349+0000 7fee3bf68640 1 -- 172.21.15.116:0/872830854 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fee18067910 msgr2=0x7fee18069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:51.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.349+0000 7fee3bf68640 1 --2- 172.21.15.116:0/872830854 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fee18067910 0x7fee18069dd0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fee240af000 tx=0x7fee240a7000 comp rx=0 tx=0).stop 2024-04-25T00:06:51.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.349+0000 7fee3bf68640 1 -- 172.21.15.116:0/872830854 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee34072080 msgr2=0x7fee34143540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:51.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.349+0000 7fee3bf68640 1 --2- 172.21.15.116:0/872830854 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee34072080 0x7fee34143540 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fee3000c450 tx=0x7fee3000c910 comp rx=0 tx=0).stop 2024-04-25T00:06:51.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.349+0000 7fee3bf68640 1 -- 172.21.15.116:0/872830854 shutdown_connections 2024-04-25T00:06:51.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.349+0000 7fee3bf68640 1 --2- 172.21.15.116:0/872830854 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fee18067910 0x7fee18069dd0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:51.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.349+0000 7fee3bf68640 1 --2- 172.21.15.116:0/872830854 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fee341562a0 0x7fee3414b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:51.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.349+0000 7fee3bf68640 1 --2- 172.21.15.116:0/872830854 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fee34152d80 0x7fee340c6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:51.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.349+0000 7fee3bf68640 1 --2- 172.21.15.116:0/872830854 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fee34072080 0x7fee34143540 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:51.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.349+0000 7fee3bf68640 1 -- 172.21.15.116:0/872830854 >> 172.21.15.116:0/872830854 conn(0x7fee3406c720 msgr2=0x7fee34149b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:51.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.353+0000 7fee3bf68640 1 -- 172.21.15.116:0/872830854 shutdown_connections 2024-04-25T00:06:51.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:51.353+0000 7fee3bf68640 1 -- 172.21.15.116:0/872830854 wait complete. 2024-04-25T00:06:51.374 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:06:53.375 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:06:53.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.517+0000 7f86acd0c640 1 Processor -- start 2024-04-25T00:06:53.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.517+0000 7f86acd0c640 1 -- start start 2024-04-25T00:06:53.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.517+0000 7f86acd0c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f86a8157870 0x7f86a806c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:53.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.517+0000 7f86acd0c640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f86a80c73e0 0x7f86a80728b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:53.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.517+0000 7f86acd0c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f86a806c760 0x7f86a8072df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:53.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.517+0000 7f86acd0c640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f86a8157c50 con 0x7f86a8157870 2024-04-25T00:06:53.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.517+0000 7f86acd0c640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f86a80723d0 con 0x7f86a80c73e0 2024-04-25T00:06:53.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.517+0000 7f86acd0c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f86a8073330 con 0x7f86a806c760 2024-04-25T00:06:53.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.517+0000 7f86a659c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f86a8157870 0x7f86a806c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:53.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.517+0000 7f86a659c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f86a8157870 0x7f86a806c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54104/0 (socket says 172.21.15.116:54104) 2024-04-25T00:06:53.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.517+0000 7f86a659c640 1 -- 172.21.15.116:0/987019058 learned_addr learned my addr 172.21.15.116:0/987019058 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:06:53.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.517+0000 7f86a5d9b640 1 --2- 172.21.15.116:0/987019058 >> v2:172.21.15.174:3300/0 conn(0x7f86a80c73e0 0x7f86a80728b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:53.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.517+0000 7f86a659c640 1 -- 172.21.15.116:0/987019058 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f86a806c760 msgr2=0x7f86a8072df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:06:53.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.517+0000 7f86a6d9d640 1 --2- 172.21.15.116:0/987019058 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f86a806c760 0x7f86a8072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:53.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.517+0000 7f86a659c640 1 --2- 172.21.15.116:0/987019058 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f86a806c760 0x7f86a8072df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:53.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.517+0000 7f86a659c640 1 -- 172.21.15.116:0/987019058 >> v2:172.21.15.174:3300/0 conn(0x7f86a80c73e0 msgr2=0x7f86a80728b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:06:53.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.517+0000 7f86a659c640 1 --2- 172.21.15.116:0/987019058 >> v2:172.21.15.174:3300/0 conn(0x7f86a80c73e0 0x7f86a80728b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:53.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.517+0000 7f86a659c640 1 -- 172.21.15.116:0/987019058 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86a8156ce0 con 0x7f86a8157870 2024-04-25T00:06:53.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.517+0000 7f86a5d9b640 1 --2- 172.21.15.116:0/987019058 >> v2:172.21.15.174:3300/0 conn(0x7f86a80c73e0 0x7f86a80728b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:06:53.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.521+0000 7f86a659c640 1 --2- 172.21.15.116:0/987019058 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86a8157870 0x7f86a806c220 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f86a8070b70 tx=0x7f869801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2601e8347660a6f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:53.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.521+0000 7f86a559a640 1 -- 172.21.15.116:0/987019058 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86980b0070 con 0x7f86a8157870 2024-04-25T00:06:53.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.521+0000 7f86a559a640 1 -- 172.21.15.116:0/987019058 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86980ab600 con 0x7f86a8157870 2024-04-25T00:06:53.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.521+0000 7f86a559a640 1 -- 172.21.15.116:0/987019058 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86980ab9d0 con 0x7f86a8157870 2024-04-25T00:06:53.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.521+0000 7f86acd0c640 1 -- 172.21.15.116:0/987019058 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86a8157870 msgr2=0x7f86a806c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:53.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.521+0000 7f86acd0c640 1 --2- 172.21.15.116:0/987019058 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86a8157870 0x7f86a806c220 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f86a8070b70 tx=0x7f869801c520 comp rx=0 tx=0).stop 2024-04-25T00:06:53.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.521+0000 7f86acd0c640 1 -- 172.21.15.116:0/987019058 shutdown_connections 2024-04-25T00:06:53.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.521+0000 7f86acd0c640 1 --2- 172.21.15.116:0/987019058 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f86a806c760 0x7f86a8072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:53.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.521+0000 7f86acd0c640 1 --2- 172.21.15.116:0/987019058 >> v2:172.21.15.174:3300/0 conn(0x7f86a80c73e0 0x7f86a80728b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:53.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.521+0000 7f86acd0c640 1 --2- 172.21.15.116:0/987019058 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86a8157870 0x7f86a806c220 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:53.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.521+0000 7f86acd0c640 1 -- 172.21.15.116:0/987019058 >> 172.21.15.116:0/987019058 conn(0x7f86a80c42e0 msgr2=0x7f86a80c46f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:53.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.521+0000 7f86acd0c640 1 -- 172.21.15.116:0/987019058 shutdown_connections 2024-04-25T00:06:53.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.521+0000 7f86acd0c640 1 -- 172.21.15.116:0/987019058 wait complete. 2024-04-25T00:06:53.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.521+0000 7f86acd0c640 1 Processor -- start 2024-04-25T00:06:53.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.521+0000 7f86acd0c640 1 -- start start 2024-04-25T00:06:53.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.521+0000 7f86acd0c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86a806c760 0x7f86a809aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:53.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.521+0000 7f86acd0c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f86a80c73e0 0x7f86a80c0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:53.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.521+0000 7f86acd0c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f86a8157870 0x7f86a81ebda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:53.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.521+0000 7f86acd0c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f86a80c5ef0 con 0x7f86a806c760 2024-04-25T00:06:53.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.521+0000 7f86acd0c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f86a80c6910 con 0x7f86a8157870 2024-04-25T00:06:53.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.521+0000 7f86acd0c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f86a80c6070 con 0x7f86a80c73e0 2024-04-25T00:06:53.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.521+0000 7f86a659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86a806c760 0x7f86a809aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:53.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.521+0000 7f86a659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86a806c760 0x7f86a809aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54114/0 (socket says 172.21.15.116:54114) 2024-04-25T00:06:53.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.521+0000 7f86a659c640 1 -- 172.21.15.116:0/2289029080 learned_addr learned my addr 172.21.15.116:0/2289029080 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:06:53.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.525+0000 7f86a6d9d640 1 --2- 172.21.15.116:0/2289029080 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f86a8157870 0x7f86a81ebda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:53.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.525+0000 7f86a5d9b640 1 --2- 172.21.15.116:0/2289029080 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f86a80c73e0 0x7f86a80c0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:53.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.525+0000 7f86a659c640 1 -- 172.21.15.116:0/2289029080 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f86a80c73e0 msgr2=0x7f86a80c0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:53.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.525+0000 7f86a659c640 1 --2- 172.21.15.116:0/2289029080 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f86a80c73e0 0x7f86a80c0ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:53.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.525+0000 7f86a659c640 1 -- 172.21.15.116:0/2289029080 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f86a8157870 msgr2=0x7f86a81ebda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:53.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.525+0000 7f86a659c640 1 --2- 172.21.15.116:0/2289029080 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f86a8157870 0x7f86a81ebda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:53.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.525+0000 7f86a659c640 1 -- 172.21.15.116:0/2289029080 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86a815c7f0 con 0x7f86a806c760 2024-04-25T00:06:53.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.525+0000 7f86a6d9d640 1 --2- 172.21.15.116:0/2289029080 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f86a8157870 0x7f86a81ebda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:06:53.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.525+0000 7f86a659c640 1 --2- 172.21.15.116:0/2289029080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86a806c760 0x7f86a809aa40 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f869801cdc0 tx=0x7f86980192c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:53.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.525+0000 7f8696ffd640 1 -- 172.21.15.116:0/2289029080 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86980b0040 con 0x7f86a806c760 2024-04-25T00:06:53.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.525+0000 7f8696ffd640 1 -- 172.21.15.116:0/2289029080 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86980b4640 con 0x7f86a806c760 2024-04-25T00:06:53.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.525+0000 7f86acd0c640 1 -- 172.21.15.116:0/2289029080 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86a81ef840 con 0x7f86a806c760 2024-04-25T00:06:53.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.525+0000 7f8696ffd640 1 -- 172.21.15.116:0/2289029080 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86980b4a30 con 0x7f86a806c760 2024-04-25T00:06:53.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.525+0000 7f86acd0c640 1 -- 172.21.15.116:0/2289029080 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86a80c13e0 con 0x7f86a806c760 2024-04-25T00:06:53.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.525+0000 7f86acd0c640 1 -- 172.21.15.116:0/2289029080 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86a8068a60 con 0x7f86a806c760 2024-04-25T00:06:53.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.525+0000 7f8696ffd640 1 -- 172.21.15.116:0/2289029080 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f86980ab4c0 con 0x7f86a806c760 2024-04-25T00:06:53.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.525+0000 7f8696ffd640 1 --2- 172.21.15.116:0/2289029080 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f86880678f0 0x7f8688069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:53.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.525+0000 7f8696ffd640 1 -- 172.21.15.116:0/2289029080 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7182+0+0 (secure 0 0 0) 0x7f869811fda0 con 0x7f86a806c760 2024-04-25T00:06:53.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.525+0000 7f86a5d9b640 1 --2- 172.21.15.116:0/2289029080 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f86880678f0 0x7f8688069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:53.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.525+0000 7f86a5d9b640 1 --2- 172.21.15.116:0/2289029080 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f86880678f0 0x7f8688069db0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f869c006fd0 tx=0x7f869c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:53.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.533+0000 7f8696ffd640 1 -- 172.21.15.116:0/2289029080 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f8698016110 con 0x7f86a806c760 2024-04-25T00:06:53.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.825+0000 7f86acd0c640 1 -- 172.21.15.116:0/2289029080 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f86a80c8630 con 0x7f86a806c760 2024-04-25T00:06:53.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.825+0000 7f8696ffd640 1 -- 172.21.15.116:0/2289029080 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+6501 (secure 0 0 0) 0x7f86980f04d0 con 0x7f86a806c760 2024-04-25T00:06:53.829 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:06:53.829 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4829,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3583791822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3583791822},{"type":"v1","addr":"172.21.15.174:6837","nonce":3583791822}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:06:33.449109+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5613":{"gid":5613,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":28,"addr":"172.21.15.116:6835/1233829678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1233829678},{"type":"v1","addr":"172.21.15.116:6835","nonce":1233829678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5613,"qdb_cluster":[5613]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:04:48.496838+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5571},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5571":{"gid":5571,"name":"c","rank":0,"incarnation":37,"state":"up:active","state_seq":4,"addr":"172.21.15.116:6833/3841609112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3841609112},{"type":"v1","addr":"172.21.15.116:6833","nonce":3841609112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5571,"qdb_cluster":[5571]},"id":2}]} 2024-04-25T00:06:53.829 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 46 2024-04-25T00:06:53.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.829+0000 7f86acd0c640 1 -- 172.21.15.116:0/2289029080 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f86880678f0 msgr2=0x7f8688069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:53.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.829+0000 7f86acd0c640 1 --2- 172.21.15.116:0/2289029080 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f86880678f0 0x7f8688069db0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f869c006fd0 tx=0x7f869c008040 comp rx=0 tx=0).stop 2024-04-25T00:06:53.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.829+0000 7f86acd0c640 1 -- 172.21.15.116:0/2289029080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86a806c760 msgr2=0x7f86a809aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:53.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.829+0000 7f86acd0c640 1 --2- 172.21.15.116:0/2289029080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86a806c760 0x7f86a809aa40 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f869801cdc0 tx=0x7f86980192c0 comp rx=0 tx=0).stop 2024-04-25T00:06:53.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.829+0000 7f86acd0c640 1 -- 172.21.15.116:0/2289029080 shutdown_connections 2024-04-25T00:06:53.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.829+0000 7f86acd0c640 1 --2- 172.21.15.116:0/2289029080 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f86880678f0 0x7f8688069db0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:53.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.829+0000 7f86acd0c640 1 --2- 172.21.15.116:0/2289029080 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f86a8157870 0x7f86a81ebda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:53.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.829+0000 7f86acd0c640 1 --2- 172.21.15.116:0/2289029080 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f86a80c73e0 0x7f86a80c0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:53.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.829+0000 7f86acd0c640 1 --2- 172.21.15.116:0/2289029080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f86a806c760 0x7f86a809aa40 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:53.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.829+0000 7f86acd0c640 1 -- 172.21.15.116:0/2289029080 >> 172.21.15.116:0/2289029080 conn(0x7f86a80c42e0 msgr2=0x7f86a815d790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:53.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.829+0000 7f86acd0c640 1 -- 172.21.15.116:0/2289029080 shutdown_connections 2024-04-25T00:06:53.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:53.829+0000 7f86acd0c640 1 -- 172.21.15.116:0/2289029080 wait complete. 2024-04-25T00:06:53.851 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:06:55.852 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:06:55.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.985+0000 7ff063ac6640 1 Processor -- start 2024-04-25T00:06:55.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.985+0000 7ff063ac6640 1 -- start start 2024-04-25T00:06:55.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.985+0000 7ff063ac6640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff05c152d80 0x7ff05c159950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:55.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.985+0000 7ff063ac6640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff05c1562a0 0x7ff05c0c5020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:55.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.985+0000 7ff063ac6640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff05c0730b0 con 0x7ff05c072080 2024-04-25T00:06:55.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.985+0000 7ff063ac6640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7ff05c072e60 con 0x7ff05c152d80 2024-04-25T00:06:55.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.985+0000 7ff063ac6640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff05c073390 con 0x7ff05c1562a0 2024-04-25T00:06:55.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.985+0000 7ff061061640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7ff05c072080 0x7ff05c0c36b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:56870/0 (socket says 172.21.15.116:56870) 2024-04-25T00:06:55.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.985+0000 7ff061061640 1 -- 172.21.15.116:0/1102252882 learned_addr learned my addr 172.21.15.116:0/1102252882 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:06:55.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.985+0000 7ff062063640 1 --2- 172.21.15.116:0/1102252882 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff05c1562a0 0x7ff05c0c5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:55.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.985+0000 7ff061862640 1 --2- 172.21.15.116:0/1102252882 >> v2:172.21.15.174:3300/0 conn(0x7ff05c152d80 0x7ff05c159950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:55.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.985+0000 7ff060860640 1 -- 172.21.15.116:0/1102252882 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3795278434 0 0) 0x7ff05c0730b0 con 0x7ff05c072080 2024-04-25T00:06:55.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.985+0000 7ff060860640 1 -- 172.21.15.116:0/1102252882 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff048003530 con 0x7ff05c072080 2024-04-25T00:06:55.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.989+0000 7ff060860640 1 -- 172.21.15.116:0/1102252882 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2844469599 0 0) 0x7ff048003530 con 0x7ff05c072080 2024-04-25T00:06:55.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.989+0000 7ff060860640 1 -- 172.21.15.116:0/1102252882 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff05c1562a0 msgr2=0x7ff05c0c5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:06:55.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.989+0000 7ff060860640 1 --2- 172.21.15.116:0/1102252882 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff05c1562a0 0x7ff05c0c5020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:55.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.989+0000 7ff060860640 1 -- 172.21.15.116:0/1102252882 >> v2:172.21.15.174:3300/0 conn(0x7ff05c152d80 msgr2=0x7ff05c159950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:06:55.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.989+0000 7ff060860640 1 --2- 172.21.15.116:0/1102252882 >> v2:172.21.15.174:3300/0 conn(0x7ff05c152d80 0x7ff05c159950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:55.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.989+0000 7ff060860640 1 -- 172.21.15.116:0/1102252882 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff05c158e70 con 0x7ff05c072080 2024-04-25T00:06:55.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.989+0000 7ff060860640 1 -- 172.21.15.116:0/1102252882 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7ff04c001680 con 0x7ff05c072080 2024-04-25T00:06:55.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.989+0000 7ff060860640 1 -- 172.21.15.116:0/1102252882 >> v1:172.21.15.116:6789/0 conn(0x7ff05c072080 legacy=0x7ff05c0c36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:55.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.989+0000 7ff060860640 1 --2- 172.21.15.116:0/1102252882 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff05c1562a0 0x7ff0480039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:55.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.989+0000 7ff060860640 1 --2- 172.21.15.116:0/1102252882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff048003f40 0x7ff048004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:55.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.989+0000 7ff060860640 1 --2- 172.21.15.116:0/1102252882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff048004d50 0x7ff0480051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:55.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.989+0000 7ff060860640 1 -- 172.21.15.116:0/1102252882 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff05c072e60 con 0x7ff048003f40 2024-04-25T00:06:55.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.989+0000 7ff060860640 1 -- 172.21.15.116:0/1102252882 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff05c073390 con 0x7ff048004d50 2024-04-25T00:06:55.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.989+0000 7ff060860640 1 -- 172.21.15.116:0/1102252882 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff04800ba20 con 0x7ff05c1562a0 2024-04-25T00:06:55.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.989+0000 7ff061061640 1 --2- 172.21.15.116:0/1102252882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff048003f40 0x7ff048004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:55.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.989+0000 7ff062063640 1 --2- 172.21.15.116:0/1102252882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff048004d50 0x7ff0480051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:55.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.989+0000 7ff061862640 1 --2- 172.21.15.116:0/1102252882 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff05c1562a0 0x7ff0480039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:55.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.989+0000 7ff061061640 1 -- 172.21.15.116:0/1102252882 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff05c1562a0 msgr2=0x7ff0480039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:06:55.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.989+0000 7ff061061640 1 --2- 172.21.15.116:0/1102252882 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff05c1562a0 0x7ff0480039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:55.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.989+0000 7ff061061640 1 -- 172.21.15.116:0/1102252882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff048004d50 msgr2=0x7ff0480051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:06:55.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.989+0000 7ff061061640 1 --2- 172.21.15.116:0/1102252882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff048004d50 0x7ff0480051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:55.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.989+0000 7ff061061640 1 -- 172.21.15.116:0/1102252882 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff048003530 con 0x7ff048003f40 2024-04-25T00:06:55.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.989+0000 7ff061061640 1 --2- 172.21.15.116:0/1102252882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff048003f40 0x7ff048004810 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7ff04c00c240 tx=0x7ff04c01cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=810546bafebb37f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:55.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.989+0000 7ff060860640 1 -- 172.21.15.116:0/1102252882 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff04c007d40 con 0x7ff048003f40 2024-04-25T00:06:55.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.989+0000 7ff060860640 1 -- 172.21.15.116:0/1102252882 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff04c0a9020 con 0x7ff048003f40 2024-04-25T00:06:55.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.989+0000 7ff060860640 1 -- 172.21.15.116:0/1102252882 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff04c0a5910 con 0x7ff048003f40 2024-04-25T00:06:55.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.993+0000 7ff063ac6640 1 -- 172.21.15.116:0/1102252882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff048003f40 msgr2=0x7ff048004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:55.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.993+0000 7ff063ac6640 1 --2- 172.21.15.116:0/1102252882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff048003f40 0x7ff048004810 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7ff04c00c240 tx=0x7ff04c01cc30 comp rx=0 tx=0).stop 2024-04-25T00:06:55.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.993+0000 7ff062063640 1 -- 172.21.15.116:0/1102252882 reap_dead start 2024-04-25T00:06:55.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.993+0000 7ff063ac6640 1 -- 172.21.15.116:0/1102252882 shutdown_connections 2024-04-25T00:06:55.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.993+0000 7ff063ac6640 1 -- 172.21.15.116:0/1102252882 >> 172.21.15.116:0/1102252882 conn(0x7ff05c06c720 msgr2=0x7ff05c06cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:55.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.993+0000 7ff063ac6640 1 -- 172.21.15.116:0/1102252882 shutdown_connections 2024-04-25T00:06:55.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.993+0000 7ff063ac6640 1 -- 172.21.15.116:0/1102252882 wait complete. 2024-04-25T00:06:55.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.993+0000 7ff063ac6640 1 Processor -- start 2024-04-25T00:06:55.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.993+0000 7ff063ac6640 1 -- start start 2024-04-25T00:06:55.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.993+0000 7ff063ac6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff048003f40 0x7ff05c1554a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:55.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.993+0000 7ff063ac6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff05c154d50 0x7ff05c1559e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:55.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.993+0000 7ff063ac6640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff05c14dc80 0x7ff05c14e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:55.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.993+0000 7ff063ac6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff05c156800 con 0x7ff048003f40 2024-04-25T00:06:55.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.993+0000 7ff063ac6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff05c156680 con 0x7ff05c154d50 2024-04-25T00:06:55.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.993+0000 7ff063ac6640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff05c153160 con 0x7ff05c14dc80 2024-04-25T00:06:55.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.993+0000 7ff061862640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff048003f40 0x7ff05c1554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:55.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.993+0000 7ff061862640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff048003f40 0x7ff05c1554a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56100/0 (socket says 172.21.15.116:56100) 2024-04-25T00:06:55.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.993+0000 7ff061862640 1 -- 172.21.15.116:0/2622388389 learned_addr learned my addr 172.21.15.116:0/2622388389 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:06:55.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.993+0000 7ff062063640 1 --2- 172.21.15.116:0/2622388389 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff05c14dc80 0x7ff05c14e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:55.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.993+0000 7ff061061640 1 --2- 172.21.15.116:0/2622388389 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff05c154d50 0x7ff05c1559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:55.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.993+0000 7ff061862640 1 -- 172.21.15.116:0/2622388389 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff05c14dc80 msgr2=0x7ff05c14e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:55.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.993+0000 7ff061862640 1 --2- 172.21.15.116:0/2622388389 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff05c14dc80 0x7ff05c14e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:55.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.993+0000 7ff061862640 1 -- 172.21.15.116:0/2622388389 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff05c154d50 msgr2=0x7ff05c1559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:55.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.993+0000 7ff061862640 1 --2- 172.21.15.116:0/2622388389 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff05c154d50 0x7ff05c1559e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:55.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.993+0000 7ff061862640 1 -- 172.21.15.116:0/2622388389 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff05c153720 con 0x7ff048003f40 2024-04-25T00:06:55.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.993+0000 7ff061862640 1 --2- 172.21.15.116:0/2622388389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff048003f40 0x7ff05c1554a0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7ff058011610 tx=0x7ff058011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:55.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.993+0000 7ff0527fc640 1 -- 172.21.15.116:0/2622388389 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff05800f8f0 con 0x7ff048003f40 2024-04-25T00:06:55.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.993+0000 7ff0527fc640 1 -- 172.21.15.116:0/2622388389 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff05800fa90 con 0x7ff048003f40 2024-04-25T00:06:55.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.993+0000 7ff063ac6640 1 -- 172.21.15.116:0/2622388389 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff05c15b330 con 0x7ff048003f40 2024-04-25T00:06:55.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.993+0000 7ff0527fc640 1 -- 172.21.15.116:0/2622388389 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff05800fe40 con 0x7ff048003f40 2024-04-25T00:06:55.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.993+0000 7ff063ac6640 1 -- 172.21.15.116:0/2622388389 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff05c15ace0 con 0x7ff048003f40 2024-04-25T00:06:55.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.997+0000 7ff063ac6640 1 -- 172.21.15.116:0/2622388389 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff05c068a60 con 0x7ff048003f40 2024-04-25T00:06:55.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.997+0000 7ff0527fc640 1 -- 172.21.15.116:0/2622388389 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff058015460 con 0x7ff048003f40 2024-04-25T00:06:55.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.997+0000 7ff0527fc640 1 --2- 172.21.15.116:0/2622388389 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff0400678f0 0x7ff040069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:55.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.997+0000 7ff0527fc640 1 -- 172.21.15.116:0/2622388389 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7182+0+0 (secure 0 0 0) 0x7ff058086b20 con 0x7ff048003f40 2024-04-25T00:06:55.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.997+0000 7ff061061640 1 --2- 172.21.15.116:0/2622388389 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff0400678f0 0x7ff040069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:56.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:55.997+0000 7ff061061640 1 --2- 172.21.15.116:0/2622388389 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff0400678f0 0x7ff040069db0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7ff04c0aa000 tx=0x7ff05c0c2010 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:56.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:56.001+0000 7ff0527fc640 1 -- 172.21.15.116:0/2622388389 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ff05c153720 con 0x7ff048003f40 2024-04-25T00:06:56.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:56.237+0000 7ff063ac6640 1 -- 172.21.15.116:0/2622388389 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff05c071dd0 con 0x7ff048003f40 2024-04-25T00:06:56.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:56.237+0000 7ff0527fc640 1 -- 172.21.15.116:0/2622388389 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+6501 (secure 0 0 0) 0x7ff058019030 con 0x7ff048003f40 2024-04-25T00:06:56.239 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:06:56.239 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4829,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3583791822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3583791822},{"type":"v1","addr":"172.21.15.174:6837","nonce":3583791822}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:06:33.449109+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5613":{"gid":5613,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":28,"addr":"172.21.15.116:6835/1233829678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1233829678},{"type":"v1","addr":"172.21.15.116:6835","nonce":1233829678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5613,"qdb_cluster":[5613]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:04:48.496838+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5571},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5571":{"gid":5571,"name":"c","rank":0,"incarnation":37,"state":"up:active","state_seq":4,"addr":"172.21.15.116:6833/3841609112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3841609112},{"type":"v1","addr":"172.21.15.116:6833","nonce":3841609112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5571,"qdb_cluster":[5571]},"id":2}]} 2024-04-25T00:06:56.239 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 46 2024-04-25T00:06:56.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:56.237+0000 7ff063ac6640 1 -- 172.21.15.116:0/2622388389 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff0400678f0 msgr2=0x7ff040069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:56.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:56.237+0000 7ff063ac6640 1 --2- 172.21.15.116:0/2622388389 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff0400678f0 0x7ff040069db0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7ff04c0aa000 tx=0x7ff05c0c2010 comp rx=0 tx=0).stop 2024-04-25T00:06:56.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:56.237+0000 7ff063ac6640 1 -- 172.21.15.116:0/2622388389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff048003f40 msgr2=0x7ff05c1554a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:56.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:56.237+0000 7ff063ac6640 1 --2- 172.21.15.116:0/2622388389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff048003f40 0x7ff05c1554a0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7ff058011610 tx=0x7ff058011ad0 comp rx=0 tx=0).stop 2024-04-25T00:06:56.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:56.237+0000 7ff063ac6640 1 -- 172.21.15.116:0/2622388389 shutdown_connections 2024-04-25T00:06:56.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:56.237+0000 7ff063ac6640 1 --2- 172.21.15.116:0/2622388389 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff0400678f0 0x7ff040069db0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:56.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:56.237+0000 7ff063ac6640 1 --2- 172.21.15.116:0/2622388389 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff05c14dc80 0x7ff05c14e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:56.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:56.237+0000 7ff063ac6640 1 --2- 172.21.15.116:0/2622388389 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff05c154d50 0x7ff05c1559e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:56.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:56.237+0000 7ff063ac6640 1 --2- 172.21.15.116:0/2622388389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff048003f40 0x7ff05c1554a0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:56.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:56.237+0000 7ff063ac6640 1 -- 172.21.15.116:0/2622388389 >> 172.21.15.116:0/2622388389 conn(0x7ff05c06c720 msgr2=0x7ff05c158500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:56.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:56.241+0000 7ff063ac6640 1 -- 172.21.15.116:0/2622388389 shutdown_connections 2024-04-25T00:06:56.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:56.241+0000 7ff063ac6640 1 -- 172.21.15.116:0/2622388389 wait complete. 2024-04-25T00:06:56.262 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:06:58.262 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:06:58.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.393+0000 7f23f6325640 1 Processor -- start 2024-04-25T00:06:58.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.393+0000 7f23f6325640 1 -- start start 2024-04-25T00:06:58.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.393+0000 7f23f6325640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f23f00c3c30 0x7f23f00c6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:58.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.393+0000 7f23f6325640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f23f006fb30 0x7f23f006c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:58.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.393+0000 7f23f6325640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23f006ff40 con 0x7f23f0152f60 2024-04-25T00:06:58.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.393+0000 7f23f6325640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f23f01595b0 con 0x7f23f00c3c30 2024-04-25T00:06:58.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.393+0000 7f23f6325640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f23f006e8f0 con 0x7f23f006fb30 2024-04-25T00:06:58.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.393+0000 7f23ef7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f23f00c3c30 0x7f23f00c6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:58.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.393+0000 7f23ef7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f23f00c3c30 0x7f23f00c6080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56102/0 (socket says 172.21.15.116:56102) 2024-04-25T00:06:58.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.393+0000 7f23ef7fe640 1 -- 172.21.15.116:0/2276309910 learned_addr learned my addr 172.21.15.116:0/2276309910 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:06:58.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.393+0000 7f23f48c2640 1 --2- 172.21.15.116:0/2276309910 >> v2:172.21.15.174:3300/0 conn(0x7f23f006fb30 0x7f23f006c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:58.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.393+0000 7f23ef7fe640 1 -- 172.21.15.116:0/2276309910 >> v2:172.21.15.174:3300/0 conn(0x7f23f006fb30 msgr2=0x7f23f006c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:06:58.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.393+0000 7f23ef7fe640 1 --2- 172.21.15.116:0/2276309910 >> v2:172.21.15.174:3300/0 conn(0x7f23f006fb30 0x7f23f006c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:58.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.393+0000 7f23ef7fe640 1 -- 172.21.15.116:0/2276309910 >> v1:172.21.15.116:6789/0 conn(0x7f23f0152f60 legacy=0x7f23f0158ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:58.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.393+0000 7f23ef7fe640 1 -- 172.21.15.116:0/2276309910 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23f006ebd0 con 0x7f23f00c3c30 2024-04-25T00:06:58.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.397+0000 7f23ef7fe640 1 --2- 172.21.15.116:0/2276309910 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23f00c3c30 0x7f23f00c6080 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f23f00739d0 tx=0x7f23e401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10d6eba658b6a48e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:58.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.397+0000 7f23eeffd640 1 -- 172.21.15.116:0/2276309910 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23e40b2070 con 0x7f23f00c3c30 2024-04-25T00:06:58.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.397+0000 7f23eeffd640 1 -- 172.21.15.116:0/2276309910 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23e40ad680 con 0x7f23f00c3c30 2024-04-25T00:06:58.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.397+0000 7f23eeffd640 1 -- 172.21.15.116:0/2276309910 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23e40ada50 con 0x7f23f00c3c30 2024-04-25T00:06:58.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.397+0000 7f23f6325640 1 -- 172.21.15.116:0/2276309910 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23f00c3c30 msgr2=0x7f23f00c6080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:58.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.397+0000 7f23f6325640 1 --2- 172.21.15.116:0/2276309910 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23f00c3c30 0x7f23f00c6080 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f23f00739d0 tx=0x7f23e401c490 comp rx=0 tx=0).stop 2024-04-25T00:06:58.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.397+0000 7f23f6325640 1 -- 172.21.15.116:0/2276309910 shutdown_connections 2024-04-25T00:06:58.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.397+0000 7f23f6325640 1 --2- 172.21.15.116:0/2276309910 >> v2:172.21.15.174:3300/0 conn(0x7f23f006fb30 0x7f23f006c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:58.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.397+0000 7f23f6325640 1 --2- 172.21.15.116:0/2276309910 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23f00c3c30 0x7f23f00c6080 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:58.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.397+0000 7f23f6325640 1 -- 172.21.15.116:0/2276309910 >> 172.21.15.116:0/2276309910 conn(0x7f23f00c4270 msgr2=0x7f23f00c4680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:58.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.397+0000 7f23f6325640 1 -- 172.21.15.116:0/2276309910 shutdown_connections 2024-04-25T00:06:58.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.397+0000 7f23f6325640 1 -- 172.21.15.116:0/2276309910 wait complete. 2024-04-25T00:06:58.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.397+0000 7f23f6325640 1 Processor -- start 2024-04-25T00:06:58.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.397+0000 7f23f6325640 1 -- start start 2024-04-25T00:06:58.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.397+0000 7f23f6325640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f23f006fb30 0x7f23f00aca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:58.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.397+0000 7f23f6325640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f23f00c3c30 0x7f23f015ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:58.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.397+0000 7f23f6325640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23f0152f60 0x7f23f014fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:58.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.397+0000 7f23f6325640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f23f01629f0 con 0x7f23f0152f60 2024-04-25T00:06:58.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.397+0000 7f23f6325640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f23f0074a70 con 0x7f23f00c3c30 2024-04-25T00:06:58.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.397+0000 7f23f6325640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f23f0162b70 con 0x7f23f006fb30 2024-04-25T00:06:58.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.397+0000 7f23f48c2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23f0152f60 0x7f23f014fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:58.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.397+0000 7f23f48c2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23f0152f60 0x7f23f014fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56118/0 (socket says 172.21.15.116:56118) 2024-04-25T00:06:58.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.397+0000 7f23f48c2640 1 -- 172.21.15.116:0/870254585 learned_addr learned my addr 172.21.15.116:0/870254585 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:06:58.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.397+0000 7f23ef7fe640 1 --2- 172.21.15.116:0/870254585 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f23f00c3c30 0x7f23f015ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:58.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.397+0000 7f23effff640 1 --2- 172.21.15.116:0/870254585 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f23f006fb30 0x7f23f00aca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:58.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.397+0000 7f23f48c2640 1 -- 172.21.15.116:0/870254585 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f23f006fb30 msgr2=0x7f23f00aca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:58.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.397+0000 7f23f48c2640 1 --2- 172.21.15.116:0/870254585 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f23f006fb30 0x7f23f00aca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:58.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.397+0000 7f23f48c2640 1 -- 172.21.15.116:0/870254585 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f23f00c3c30 msgr2=0x7f23f015ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:58.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.397+0000 7f23f48c2640 1 --2- 172.21.15.116:0/870254585 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f23f00c3c30 0x7f23f015ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:58.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.397+0000 7f23f48c2640 1 -- 172.21.15.116:0/870254585 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23f015ef30 con 0x7f23f0152f60 2024-04-25T00:06:58.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.397+0000 7f23effff640 1 --2- 172.21.15.116:0/870254585 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f23f006fb30 0x7f23f00aca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:06:58.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.401+0000 7f23f48c2640 1 --2- 172.21.15.116:0/870254585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23f0152f60 0x7f23f014fc20 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f23e800afd0 tx=0x7f23e800ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:58.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.401+0000 7f23ecff9640 1 -- 172.21.15.116:0/870254585 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23e8015070 con 0x7f23f0152f60 2024-04-25T00:06:58.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.401+0000 7f23ecff9640 1 -- 172.21.15.116:0/870254585 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23e8004220 con 0x7f23f0152f60 2024-04-25T00:06:58.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.401+0000 7f23f6325640 1 -- 172.21.15.116:0/870254585 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23f00c35f0 con 0x7f23f0152f60 2024-04-25T00:06:58.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.401+0000 7f23ecff9640 1 -- 172.21.15.116:0/870254585 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23e8003620 con 0x7f23f0152f60 2024-04-25T00:06:58.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.401+0000 7f23f6325640 1 -- 172.21.15.116:0/870254585 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23f015b900 con 0x7f23f0152f60 2024-04-25T00:06:58.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.401+0000 7f23f6325640 1 -- 172.21.15.116:0/870254585 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23f0068a60 con 0x7f23f0152f60 2024-04-25T00:06:58.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.401+0000 7f23ecff9640 1 -- 172.21.15.116:0/870254585 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f23e80037c0 con 0x7f23f0152f60 2024-04-25T00:06:58.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.401+0000 7f23ecff9640 1 --2- 172.21.15.116:0/870254585 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f23dc0678f0 0x7f23dc069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:06:58.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.401+0000 7f23ecff9640 1 -- 172.21.15.116:0/870254585 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 7182+0+0 (secure 0 0 0) 0x7f23e8086010 con 0x7f23f0152f60 2024-04-25T00:06:58.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.401+0000 7f23effff640 1 --2- 172.21.15.116:0/870254585 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f23dc0678f0 0x7f23dc069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:06:58.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.405+0000 7f23effff640 1 --2- 172.21.15.116:0/870254585 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f23dc0678f0 0x7f23dc069db0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f23e0007fd0 tx=0x7f23e0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:06:58.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.409+0000 7f23ecff9640 1 -- 172.21.15.116:0/870254585 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f23f015ef30 con 0x7f23f0152f60 2024-04-25T00:06:58.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.665+0000 7f23f6325640 1 -- 172.21.15.116:0/870254585 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f23f00a8330 con 0x7f23f0152f60 2024-04-25T00:06:58.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.669+0000 7f23ecff9640 1 -- 172.21.15.116:0/870254585 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) ==== 94+0+6501 (secure 0 0 0) 0x7f23e80565a0 con 0x7f23f0152f60 2024-04-25T00:06:58.670 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:06:58.671 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4829,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3583791822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3583791822},{"type":"v1","addr":"172.21.15.174:6837","nonce":3583791822}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":46}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:06:33.449109+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5613":{"gid":5613,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":28,"addr":"172.21.15.116:6835/1233829678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1233829678},{"type":"v1","addr":"172.21.15.116:6835","nonce":1233829678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5613,"qdb_cluster":[5613]},"id":1},{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:04:48.496838+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5571},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5571":{"gid":5571,"name":"c","rank":0,"incarnation":37,"state":"up:active","state_seq":4,"addr":"172.21.15.116:6833/3841609112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3841609112},{"type":"v1","addr":"172.21.15.116:6833","nonce":3841609112}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5571,"qdb_cluster":[5571]},"id":2}]} 2024-04-25T00:06:58.672 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 46 2024-04-25T00:06:58.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.669+0000 7f23f6325640 1 -- 172.21.15.116:0/870254585 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f23dc0678f0 msgr2=0x7f23dc069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:58.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.669+0000 7f23f6325640 1 --2- 172.21.15.116:0/870254585 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f23dc0678f0 0x7f23dc069db0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f23e0007fd0 tx=0x7f23e0009040 comp rx=0 tx=0).stop 2024-04-25T00:06:58.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.669+0000 7f23f6325640 1 -- 172.21.15.116:0/870254585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23f0152f60 msgr2=0x7f23f014fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:06:58.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.669+0000 7f23f6325640 1 --2- 172.21.15.116:0/870254585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23f0152f60 0x7f23f014fc20 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f23e800afd0 tx=0x7f23e800ca20 comp rx=0 tx=0).stop 2024-04-25T00:06:58.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.669+0000 7f23f6325640 1 -- 172.21.15.116:0/870254585 shutdown_connections 2024-04-25T00:06:58.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.669+0000 7f23f6325640 1 --2- 172.21.15.116:0/870254585 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f23dc0678f0 0x7f23dc069db0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:58.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.669+0000 7f23f6325640 1 --2- 172.21.15.116:0/870254585 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f23f0152f60 0x7f23f014fc20 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:58.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.669+0000 7f23f6325640 1 --2- 172.21.15.116:0/870254585 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f23f00c3c30 0x7f23f015ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:58.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.669+0000 7f23f6325640 1 --2- 172.21.15.116:0/870254585 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f23f006fb30 0x7f23f00aca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:06:58.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.669+0000 7f23f6325640 1 -- 172.21.15.116:0/870254585 >> 172.21.15.116:0/870254585 conn(0x7f23f00c4270 msgr2=0x7f23f00c0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:06:58.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.669+0000 7f23f6325640 1 -- 172.21.15.116:0/870254585 shutdown_connections 2024-04-25T00:06:58.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:06:58.669+0000 7f23f6325640 1 -- 172.21.15.116:0/870254585 wait complete. 2024-04-25T00:06:58.691 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:07:00.692 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:07:00.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:00.817+0000 7f43a5730640 1 Processor -- start 2024-04-25T00:07:00.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:00.817+0000 7f43a5730640 1 -- start start 2024-04-25T00:07:00.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:00.821+0000 7f43a5730640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f43a0157870 0x7f43a006c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:00.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:00.821+0000 7f43a5730640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43a00723d0 con 0x7f43a00c73e0 2024-04-25T00:07:00.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:00.821+0000 7f43a5730640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43a0156ce0 con 0x7f43a006c760 2024-04-25T00:07:00.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:00.821+0000 7f43a5730640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f43a0157c50 con 0x7f43a0157870 2024-04-25T00:07:00.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:00.821+0000 7f439effd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f43a0157870 0x7f43a006c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:00.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:00.821+0000 7f439effd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f43a0157870 0x7f43a006c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56126/0 (socket says 172.21.15.116:56126) 2024-04-25T00:07:00.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:00.821+0000 7f439effd640 1 -- 172.21.15.116:0/2919891182 learned_addr learned my addr 172.21.15.116:0/2919891182 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:07:00.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:00.821+0000 7f439effd640 1 -- 172.21.15.116:0/2919891182 >> v1:172.21.15.174:6789/0 conn(0x7f43a006c760 legacy=0x7f43a015e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:07:00.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:00.821+0000 7f439effd640 1 -- 172.21.15.116:0/2919891182 >> v1:172.21.15.116:6789/0 conn(0x7f43a00c73e0 legacy=0x7f43a00728b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:00.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:00.821+0000 7f439effd640 1 -- 172.21.15.116:0/2919891182 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43a00c2610 con 0x7f43a0157870 2024-04-25T00:07:00.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:00.821+0000 7f439effd640 1 --2- 172.21.15.116:0/2919891182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43a0157870 0x7f43a006c220 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f4390009870 tx=0x7f439001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25c52e1f5199dc00 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:00.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:00.821+0000 7f439dffb640 1 -- 172.21.15.116:0/2919891182 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43900b0070 con 0x7f43a0157870 2024-04-25T00:07:00.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:00.821+0000 7f439dffb640 1 -- 172.21.15.116:0/2919891182 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43900ab600 con 0x7f43a0157870 2024-04-25T00:07:00.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:00.821+0000 7f439dffb640 1 -- 172.21.15.116:0/2919891182 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43900ab9d0 con 0x7f43a0157870 2024-04-25T00:07:00.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:00.821+0000 7f43a5730640 1 -- 172.21.15.116:0/2919891182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43a0157870 msgr2=0x7f43a006c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:00.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:00.821+0000 7f43a5730640 1 --2- 172.21.15.116:0/2919891182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43a0157870 0x7f43a006c220 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f4390009870 tx=0x7f439001c520 comp rx=0 tx=0).stop 2024-04-25T00:07:00.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:00.821+0000 7f43a5730640 1 -- 172.21.15.116:0/2919891182 shutdown_connections 2024-04-25T00:07:00.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:00.821+0000 7f43a5730640 1 --2- 172.21.15.116:0/2919891182 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43a0157870 0x7f43a006c220 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:00.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:00.821+0000 7f43a5730640 1 -- 172.21.15.116:0/2919891182 >> 172.21.15.116:0/2919891182 conn(0x7f43a00c42e0 msgr2=0x7f43a00c46f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:00.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:00.821+0000 7f43a5730640 1 -- 172.21.15.116:0/2919891182 shutdown_connections 2024-04-25T00:07:00.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:00.821+0000 7f43a5730640 1 -- 172.21.15.116:0/2919891182 wait complete. 2024-04-25T00:07:00.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:00.825+0000 7f43a5730640 1 Processor -- start 2024-04-25T00:07:00.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:00.825+0000 7f43a5730640 1 -- start start 2024-04-25T00:07:00.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:00.825+0000 7f43a5730640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43a006c760 0x7f43a0155d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:00.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:00.825+0000 7f43a5730640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43a00c73e0 0x7f43a0150880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:00.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:00.825+0000 7f43a5730640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43a0157870 0x7f43a00bd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:00.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:00.825+0000 7f43a5730640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f43a01620d0 con 0x7f43a0157870 2024-04-25T00:07:00.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:00.825+0000 7f43a5730640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f43a0161f50 con 0x7f43a00c73e0 2024-04-25T00:07:00.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:00.825+0000 7f43a5730640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f43a0162250 con 0x7f43a006c760 2024-04-25T00:07:00.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:00.825+0000 7f439f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43a0157870 0x7f43a00bd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:00.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:00.825+0000 7f439f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43a0157870 0x7f43a00bd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56130/0 (socket says 172.21.15.116:56130) 2024-04-25T00:07:00.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:00.825+0000 7f439f7fe640 1 -- 172.21.15.116:0/2276996564 learned_addr learned my addr 172.21.15.116:0/2276996564 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:07:00.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:00.825+0000 7f439effd640 1 --2- 172.21.15.116:0/2276996564 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43a006c760 0x7f43a0155d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:00.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:00.825+0000 7f439f7fe640 1 -- 172.21.15.116:0/2276996564 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43a006c760 msgr2=0x7f43a0155d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:00.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:00.825+0000 7f439f7fe640 1 --2- 172.21.15.116:0/2276996564 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43a006c760 0x7f43a0155d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:00.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:00.825+0000 7f439e7fc640 1 --2- 172.21.15.116:0/2276996564 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43a00c73e0 0x7f43a0150880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:00.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:00.825+0000 7f439f7fe640 1 -- 172.21.15.116:0/2276996564 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43a00c73e0 msgr2=0x7f43a0150880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:00.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:00.825+0000 7f439f7fe640 1 --2- 172.21.15.116:0/2276996564 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43a00c73e0 0x7f43a0150880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:00.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:00.825+0000 7f439f7fe640 1 -- 172.21.15.116:0/2276996564 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43a00ac310 con 0x7f43a0157870 2024-04-25T00:07:00.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:00.825+0000 7f439effd640 1 --2- 172.21.15.116:0/2276996564 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43a006c760 0x7f43a0155d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:07:00.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:00.825+0000 7f439e7fc640 1 --2- 172.21.15.116:0/2276996564 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43a00c73e0 0x7f43a0150880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:07:00.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:00.825+0000 7f439f7fe640 1 --2- 172.21.15.116:0/2276996564 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43a0157870 0x7f43a00bd460 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f439800c780 tx=0x7f439800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:00.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:00.825+0000 7f43877fe640 1 -- 172.21.15.116:0/2276996564 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4398004b70 con 0x7f43a0157870 2024-04-25T00:07:00.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:00.825+0000 7f43877fe640 1 -- 172.21.15.116:0/2276996564 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4398004d10 con 0x7f43a0157870 2024-04-25T00:07:00.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:00.825+0000 7f43a5730640 1 -- 172.21.15.116:0/2276996564 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43a014fda0 con 0x7f43a0157870 2024-04-25T00:07:00.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:00.825+0000 7f43877fe640 1 -- 172.21.15.116:0/2276996564 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f439800aa90 con 0x7f43a0157870 2024-04-25T00:07:00.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:00.825+0000 7f43a5730640 1 -- 172.21.15.116:0/2276996564 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43a0150360 con 0x7f43a0157870 2024-04-25T00:07:00.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:00.829+0000 7f43a5730640 1 -- 172.21.15.116:0/2276996564 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43a0150560 con 0x7f43a0157870 2024-04-25T00:07:00.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:00.829+0000 7f43877fe640 1 -- 172.21.15.116:0/2276996564 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f439800ac30 con 0x7f43a0157870 2024-04-25T00:07:00.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:00.829+0000 7f43877fe640 1 --2- 172.21.15.116:0/2276996564 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f438c0678c0 0x7f438c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:00.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:00.829+0000 7f43877fe640 1 -- 172.21.15.116:0/2276996564 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 7268+0+0 (secure 0 0 0) 0x7f4398086610 con 0x7f43a0157870 2024-04-25T00:07:00.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:00.829+0000 7f439effd640 1 --2- 172.21.15.116:0/2276996564 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f438c0678c0 0x7f438c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:00.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:00.829+0000 7f439effd640 1 --2- 172.21.15.116:0/2276996564 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f438c0678c0 0x7f438c069d80 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f43900ad000 tx=0x7f4390008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:00.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:00.837+0000 7f43877fe640 1 -- 172.21.15.116:0/2276996564 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f43a0150560 con 0x7f43a0157870 2024-04-25T00:07:01.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.081+0000 7f43a5730640 1 -- 172.21.15.116:0/2276996564 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f43a0068a60 con 0x7f43a0157870 2024-04-25T00:07:01.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.085+0000 7f43877fe640 1 -- 172.21.15.116:0/2276996564 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) ==== 94+0+5666 (secure 0 0 0) 0x7f4398020070 con 0x7f43a0157870 2024-04-25T00:07:01.086 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:07:01.086 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:06:33.449109+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5613":{"gid":5613,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":28,"addr":"172.21.15.116:6835/1233829678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1233829678},{"type":"v1","addr":"172.21.15.116:6835","nonce":1233829678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5613,"qdb_cluster":[5613]},"id":1},{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:07:00.448973+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4829":{"gid":4829,"name":"b","rank":0,"incarnation":47,"state":"up:replay","state_seq":1,"addr":"172.21.15.174:6837/3583791822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3583791822},{"type":"v1","addr":"172.21.15.174:6837","nonce":3583791822}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:07:01.086 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 47 2024-04-25T00:07:01.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.085+0000 7f43a5730640 1 -- 172.21.15.116:0/2276996564 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f438c0678c0 msgr2=0x7f438c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:01.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.085+0000 7f43a5730640 1 --2- 172.21.15.116:0/2276996564 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f438c0678c0 0x7f438c069d80 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f43900ad000 tx=0x7f4390008040 comp rx=0 tx=0).stop 2024-04-25T00:07:01.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.085+0000 7f43a5730640 1 -- 172.21.15.116:0/2276996564 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43a0157870 msgr2=0x7f43a00bd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:01.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.085+0000 7f43a5730640 1 --2- 172.21.15.116:0/2276996564 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43a0157870 0x7f43a00bd460 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f439800c780 tx=0x7f439800cc40 comp rx=0 tx=0).stop 2024-04-25T00:07:01.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.085+0000 7f43a5730640 1 -- 172.21.15.116:0/2276996564 shutdown_connections 2024-04-25T00:07:01.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.085+0000 7f43a5730640 1 --2- 172.21.15.116:0/2276996564 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f438c0678c0 0x7f438c069d80 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:01.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.085+0000 7f43a5730640 1 --2- 172.21.15.116:0/2276996564 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43a0157870 0x7f43a00bd460 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:01.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.085+0000 7f43a5730640 1 --2- 172.21.15.116:0/2276996564 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43a00c73e0 0x7f43a0150880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:01.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.085+0000 7f43a5730640 1 --2- 172.21.15.116:0/2276996564 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43a006c760 0x7f43a0155d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:01.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.085+0000 7f43a5730640 1 -- 172.21.15.116:0/2276996564 >> 172.21.15.116:0/2276996564 conn(0x7f43a00c42e0 msgr2=0x7f43a0154070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:01.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.085+0000 7f43a5730640 1 -- 172.21.15.116:0/2276996564 shutdown_connections 2024-04-25T00:07:01.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.085+0000 7f43a5730640 1 -- 172.21.15.116:0/2276996564 wait complete. 2024-04-25T00:07:01.105 INFO:tasks.mds_thrash.fs.[b]:mds.c down, removed from mdsmap 2024-04-25T00:07:01.105 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-04-25T00:07:01.105 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:07:01.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.225+0000 7f8896a0d640 1 Processor -- start 2024-04-25T00:07:01.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.225+0000 7f8896a0d640 1 -- start start 2024-04-25T00:07:01.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.229+0000 7f8896a0d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88900730a0 0x7f8890154290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:01.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.229+0000 7f8896a0d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8890153810 con 0x7f88900c3140 2024-04-25T00:07:01.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.229+0000 7f8896a0d640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88900c3690 con 0x7f88900c2bf0 2024-04-25T00:07:01.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.229+0000 7f8896a0d640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f88901547d0 con 0x7f88900730a0 2024-04-25T00:07:01.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.229+0000 7f8894faa640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f88900c3140 0x7f889006c220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:56896/0 (socket says 172.21.15.116:56896) 2024-04-25T00:07:01.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.229+0000 7f8894faa640 1 -- 172.21.15.116:0/2528482642 learned_addr learned my addr 172.21.15.116:0/2528482642 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:07:01.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.229+0000 7f888ffff640 1 --2- 172.21.15.116:0/2528482642 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88900730a0 0x7f8890154290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:01.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.229+0000 7f888f7fe640 1 -- 172.21.15.116:0/2528482642 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 302169609 0 0) 0x7f88880014a0 con 0x7f88900c3140 2024-04-25T00:07:01.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.229+0000 7f888f7fe640 1 -- 172.21.15.116:0/2528482642 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f887c003530 con 0x7f88900c3140 2024-04-25T00:07:01.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.229+0000 7f888f7fe640 1 -- 172.21.15.116:0/2528482642 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2532361701 0 0) 0x7f887c003530 con 0x7f88900c3140 2024-04-25T00:07:01.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.229+0000 7f888f7fe640 1 -- 172.21.15.116:0/2528482642 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88900730a0 msgr2=0x7f8890154290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:07:01.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.229+0000 7f888f7fe640 1 --2- 172.21.15.116:0/2528482642 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88900730a0 0x7f8890154290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:01.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.229+0000 7f888f7fe640 1 -- 172.21.15.116:0/2528482642 >> v1:172.21.15.174:6789/0 conn(0x7f88900c2bf0 legacy=0x7f88900723b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:01.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.229+0000 7f888f7fe640 1 -- 172.21.15.116:0/2528482642 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88900c3970 con 0x7f88900c3140 2024-04-25T00:07:01.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.229+0000 7f888f7fe640 1 -- 172.21.15.116:0/2528482642 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f8888001bd0 con 0x7f88900c3140 2024-04-25T00:07:01.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.229+0000 7f888f7fe640 1 -- 172.21.15.116:0/2528482642 >> v1:172.21.15.116:6789/0 conn(0x7f88900c3140 legacy=0x7f889006c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:01.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.229+0000 7f888f7fe640 1 --2- 172.21.15.116:0/2528482642 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88900730a0 0x7f887c0039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:01.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.229+0000 7f888f7fe640 1 --2- 172.21.15.116:0/2528482642 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f887c003f70 0x7f887c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:01.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.229+0000 7f888f7fe640 1 --2- 172.21.15.116:0/2528482642 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f887c004d50 0x7f887c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:01.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.229+0000 7f888f7fe640 1 -- 172.21.15.116:0/2528482642 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f88901547d0 con 0x7f887c004d50 2024-04-25T00:07:01.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.229+0000 7f888f7fe640 1 -- 172.21.15.116:0/2528482642 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f887c00bab0 con 0x7f887c003f70 2024-04-25T00:07:01.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.229+0000 7f888f7fe640 1 -- 172.21.15.116:0/2528482642 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f887c00bcb0 con 0x7f88900730a0 2024-04-25T00:07:01.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.229+0000 7f8894faa640 1 --2- 172.21.15.116:0/2528482642 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f887c004d50 0x7f887c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:01.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.229+0000 7f8887fff640 1 --2- 172.21.15.116:0/2528482642 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f887c003f70 0x7f887c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:01.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.229+0000 7f888ffff640 1 --2- 172.21.15.116:0/2528482642 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88900730a0 0x7f887c0039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:01.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.229+0000 7f8894faa640 1 -- 172.21.15.116:0/2528482642 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88900730a0 msgr2=0x7f887c0039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:07:01.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.229+0000 7f8894faa640 1 --2- 172.21.15.116:0/2528482642 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88900730a0 0x7f887c0039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:01.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.229+0000 7f8894faa640 1 -- 172.21.15.116:0/2528482642 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f887c003f70 msgr2=0x7f887c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:07:01.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.229+0000 7f8894faa640 1 --2- 172.21.15.116:0/2528482642 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f887c003f70 0x7f887c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:01.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.229+0000 7f8894faa640 1 -- 172.21.15.116:0/2528482642 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f887c003530 con 0x7f887c004d50 2024-04-25T00:07:01.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.233+0000 7f8894faa640 1 --2- 172.21.15.116:0/2528482642 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f887c004d50 0x7f887c0051d0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f888800aa50 tx=0x7f888801d3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b14f561559fea5c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:01.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.233+0000 7f888f7fe640 1 -- 172.21.15.116:0/2528482642 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f888800a670 con 0x7f887c004d50 2024-04-25T00:07:01.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.233+0000 7f888f7fe640 1 -- 172.21.15.116:0/2528482642 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88880ad650 con 0x7f887c004d50 2024-04-25T00:07:01.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.233+0000 7f888f7fe640 1 -- 172.21.15.116:0/2528482642 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88880ada80 con 0x7f887c004d50 2024-04-25T00:07:01.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.233+0000 7f8896a0d640 1 -- 172.21.15.116:0/2528482642 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f887c004d50 msgr2=0x7f887c0051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:01.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.233+0000 7f8896a0d640 1 --2- 172.21.15.116:0/2528482642 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f887c004d50 0x7f887c0051d0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f888800aa50 tx=0x7f888801d3a0 comp rx=0 tx=0).stop 2024-04-25T00:07:01.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.233+0000 7f8894faa640 1 -- 172.21.15.116:0/2528482642 reap_dead start 2024-04-25T00:07:01.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.233+0000 7f8896a0d640 1 -- 172.21.15.116:0/2528482642 shutdown_connections 2024-04-25T00:07:01.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.233+0000 7f8896a0d640 1 -- 172.21.15.116:0/2528482642 >> 172.21.15.116:0/2528482642 conn(0x7f88900c42e0 msgr2=0x7f88900c46f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:01.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.233+0000 7f8896a0d640 1 -- 172.21.15.116:0/2528482642 shutdown_connections 2024-04-25T00:07:01.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.233+0000 7f8896a0d640 1 -- 172.21.15.116:0/2528482642 wait complete. 2024-04-25T00:07:01.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.233+0000 7f8896a0d640 1 Processor -- start 2024-04-25T00:07:01.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.233+0000 7f8896a0d640 1 -- start start 2024-04-25T00:07:01.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.233+0000 7f8896a0d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8890150bf0 0x7f88900bcb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:01.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.233+0000 7f8896a0d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88901e6c70 0x7f8890156cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:01.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.233+0000 7f8896a0d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88901577d0 0x7f889014dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:01.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.233+0000 7f8896a0d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8890073480 con 0x7f88901e6c70 2024-04-25T00:07:01.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.233+0000 7f8896a0d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8890153d70 con 0x7f8890150bf0 2024-04-25T00:07:01.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.233+0000 7f8896a0d640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f889006c930 con 0x7f88901577d0 2024-04-25T00:07:01.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.233+0000 7f8887fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88901e6c70 0x7f8890156cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:01.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.233+0000 7f8887fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88901e6c70 0x7f8890156cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56156/0 (socket says 172.21.15.116:56156) 2024-04-25T00:07:01.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.233+0000 7f8887fff640 1 -- 172.21.15.116:0/131641426 learned_addr learned my addr 172.21.15.116:0/131641426 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:07:01.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.237+0000 7f888ffff640 1 --2- 172.21.15.116:0/131641426 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8890150bf0 0x7f88900bcb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:01.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.237+0000 7f8894faa640 1 --2- 172.21.15.116:0/131641426 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88901577d0 0x7f889014dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:01.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.237+0000 7f8887fff640 1 -- 172.21.15.116:0/131641426 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88901577d0 msgr2=0x7f889014dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:01.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.237+0000 7f8887fff640 1 --2- 172.21.15.116:0/131641426 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88901577d0 0x7f889014dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:01.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.237+0000 7f8887fff640 1 -- 172.21.15.116:0/131641426 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8890150bf0 msgr2=0x7f88900bcb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:01.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.237+0000 7f8887fff640 1 --2- 172.21.15.116:0/131641426 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8890150bf0 0x7f88900bcb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:01.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.237+0000 7f8887fff640 1 -- 172.21.15.116:0/131641426 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88900c6190 con 0x7f88901e6c70 2024-04-25T00:07:01.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.237+0000 7f8894faa640 1 --2- 172.21.15.116:0/131641426 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88901577d0 0x7f889014dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:07:01.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.237+0000 7f8887fff640 1 --2- 172.21.15.116:0/131641426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88901e6c70 0x7f8890156cf0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f887800e490 tx=0x7f887800e950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:01.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.237+0000 7f888d7fa640 1 -- 172.21.15.116:0/131641426 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8878019070 con 0x7f88901e6c70 2024-04-25T00:07:01.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.237+0000 7f888d7fa640 1 -- 172.21.15.116:0/131641426 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88780052e0 con 0x7f88901e6c70 2024-04-25T00:07:01.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.237+0000 7f8896a0d640 1 -- 172.21.15.116:0/131641426 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8890155f30 con 0x7f88901e6c70 2024-04-25T00:07:01.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.237+0000 7f888d7fa640 1 -- 172.21.15.116:0/131641426 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8878003840 con 0x7f88901e6c70 2024-04-25T00:07:01.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.237+0000 7f8896a0d640 1 -- 172.21.15.116:0/131641426 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88901564f0 con 0x7f88901e6c70 2024-04-25T00:07:01.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.237+0000 7f8896a0d640 1 -- 172.21.15.116:0/131641426 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8890068a60 con 0x7f88901e6c70 2024-04-25T00:07:01.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.237+0000 7f888d7fa640 1 -- 172.21.15.116:0/131641426 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f887801c070 con 0x7f88901e6c70 2024-04-25T00:07:01.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.237+0000 7f888d7fa640 1 --2- 172.21.15.116:0/131641426 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f88700678f0 0x7f8870069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:01.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.237+0000 7f888d7fa640 1 -- 172.21.15.116:0/131641426 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 7268+0+0 (secure 0 0 0) 0x7f8878085710 con 0x7f88901e6c70 2024-04-25T00:07:01.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.237+0000 7f888ffff640 1 --2- 172.21.15.116:0/131641426 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f88700678f0 0x7f8870069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:01.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.241+0000 7f888ffff640 1 --2- 172.21.15.116:0/131641426 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f88700678f0 0x7f8870069db0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8880009a30 tx=0x7f8880008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:01.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.245+0000 7f888d7fa640 1 -- 172.21.15.116:0/131641426 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f8878055730 con 0x7f88901e6c70 2024-04-25T00:07:01.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.489+0000 7f8896a0d640 1 -- 172.21.15.116:0/131641426 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8890074980 con 0x7f88901e6c70 2024-04-25T00:07:01.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.489+0000 7f888d7fa640 1 -- 172.21.15.116:0/131641426 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+5669 (secure 0 0 0) 0x7f8878059a30 con 0x7f88901e6c70 2024-04-25T00:07:01.492 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:07:01.493 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:06:33.449109+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5613":{"gid":5613,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":28,"addr":"172.21.15.116:6835/1233829678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1233829678},{"type":"v1","addr":"172.21.15.116:6835","nonce":1233829678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5613,"qdb_cluster":[5613]},"id":1},{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:07:00.733975+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4829":{"gid":4829,"name":"b","rank":0,"incarnation":47,"state":"up:reconnect","state_seq":8,"addr":"172.21.15.174:6837/3583791822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3583791822},{"type":"v1","addr":"172.21.15.174:6837","nonce":3583791822}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:07:01.493 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 48 2024-04-25T00:07:01.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.493+0000 7f8896a0d640 1 -- 172.21.15.116:0/131641426 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f88700678f0 msgr2=0x7f8870069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:01.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.493+0000 7f8896a0d640 1 --2- 172.21.15.116:0/131641426 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f88700678f0 0x7f8870069db0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8880009a30 tx=0x7f8880008040 comp rx=0 tx=0).stop 2024-04-25T00:07:01.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.493+0000 7f8896a0d640 1 -- 172.21.15.116:0/131641426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88901e6c70 msgr2=0x7f8890156cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:01.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.493+0000 7f8896a0d640 1 --2- 172.21.15.116:0/131641426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88901e6c70 0x7f8890156cf0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f887800e490 tx=0x7f887800e950 comp rx=0 tx=0).stop 2024-04-25T00:07:01.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.493+0000 7f8896a0d640 1 -- 172.21.15.116:0/131641426 shutdown_connections 2024-04-25T00:07:01.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.493+0000 7f8896a0d640 1 --2- 172.21.15.116:0/131641426 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f88700678f0 0x7f8870069db0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:01.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.493+0000 7f8896a0d640 1 --2- 172.21.15.116:0/131641426 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88901577d0 0x7f889014dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:01.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.493+0000 7f8896a0d640 1 --2- 172.21.15.116:0/131641426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88901e6c70 0x7f8890156cf0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:01.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.493+0000 7f8896a0d640 1 --2- 172.21.15.116:0/131641426 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8890150bf0 0x7f88900bcb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:01.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.493+0000 7f8896a0d640 1 -- 172.21.15.116:0/131641426 >> 172.21.15.116:0/131641426 conn(0x7f88900c42e0 msgr2=0x7f88900c53d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:01.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.493+0000 7f8896a0d640 1 -- 172.21.15.116:0/131641426 shutdown_connections 2024-04-25T00:07:01.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:01.493+0000 7f8896a0d640 1 -- 172.21.15.116:0/131641426 wait complete. 2024-04-25T00:07:01.512 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 48, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:04.678737+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 45, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5613": { "addr": "172.21.15.116:6835/1233829678", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6834", "nonce": 1233829678, "type": "v2" }, { "addr": "172.21.15.116:6835", "nonce": 1233829678, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5613, "incarnation": 42, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 28 } }, "last_failure": 0, "last_failure_osd_epoch": 35, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T00:06:33.449109+0000", "qdb_cluster": [ 5613 ], "qdb_leader": 5613, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5613 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:18.825887+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 48, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_4829": { "addr": "172.21.15.174:6837/3583791822", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6836", "nonce": 3583791822, "type": "v2" }, { "addr": "172.21.15.174:6837", "nonce": 3583791822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4829, "incarnation": 47, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:reconnect", "state_seq": 8 } }, "last_failure": 0, "last_failure_osd_epoch": 36, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-04-25T00:07:00.733975+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4829 } } } ], "standbys": [ { "addr": "172.21.15.174:6835/2883220846", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6834", "nonce": 2883220846, "type": "v2" }, { "addr": "172.21.15.174:6835", "nonce": 2883220846, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4235, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-25T00:07:03.513 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:07:03.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.637+0000 7f4990ba8640 1 Processor -- start 2024-04-25T00:07:03.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.637+0000 7f4990ba8640 1 -- start start 2024-04-25T00:07:03.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.637+0000 7f4990ba8640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f498c155b20 0x7f498c152c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:03.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.637+0000 7f4990ba8640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f498c0a9e60 con 0x7f498c14c700 2024-04-25T00:07:03.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.637+0000 7f4990ba8640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f498c0c3ef0 con 0x7f498c0aacd0 2024-04-25T00:07:03.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.637+0000 7f4990ba8640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f498c1527b0 con 0x7f498c155b20 2024-04-25T00:07:03.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.637+0000 7f498a59c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f498c14c700 0x7f498c0c5090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:56900/0 (socket says 172.21.15.116:56900) 2024-04-25T00:07:03.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.637+0000 7f498a59c640 1 -- 172.21.15.116:0/2269334204 learned_addr learned my addr 172.21.15.116:0/2269334204 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:07:03.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.637+0000 7f4989d9b640 1 --2- 172.21.15.116:0/2269334204 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f498c155b20 0x7f498c152c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:03.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.637+0000 7f498959a640 1 -- 172.21.15.116:0/2269334204 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3238142447 0 0) 0x7f498c0a9e60 con 0x7f498c14c700 2024-04-25T00:07:03.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.637+0000 7f498959a640 1 -- 172.21.15.116:0/2269334204 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4974003530 con 0x7f498c14c700 2024-04-25T00:07:03.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.641+0000 7f498959a640 1 -- 172.21.15.116:0/2269334204 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1884957803 0 0) 0x7f498c0c3ef0 con 0x7f498c0aacd0 2024-04-25T00:07:03.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.641+0000 7f498959a640 1 -- 172.21.15.116:0/2269334204 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f498c0a9e60 con 0x7f498c0aacd0 2024-04-25T00:07:03.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.641+0000 7f4989d9b640 1 -- 172.21.15.116:0/2269334204 >> v1:172.21.15.174:6789/0 conn(0x7f498c0aacd0 legacy=0x7f498c15ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:03.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.641+0000 7f4989d9b640 1 -- 172.21.15.116:0/2269334204 >> v1:172.21.15.116:6789/0 conn(0x7f498c14c700 legacy=0x7f498c0c5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:03.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.641+0000 7f4989d9b640 1 -- 172.21.15.116:0/2269334204 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f498c0c41d0 con 0x7f498c155b20 2024-04-25T00:07:03.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.641+0000 7f4989d9b640 1 --2- 172.21.15.116:0/2269334204 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f498c155b20 0x7f498c152c40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f49800097f0 tx=0x7f498001c3c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73094c6a82cc5487 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:03.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.641+0000 7f498959a640 1 -- 172.21.15.116:0/2269334204 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49800b0070 con 0x7f498c155b20 2024-04-25T00:07:03.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.641+0000 7f498959a640 1 -- 172.21.15.116:0/2269334204 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49800a4e50 con 0x7f498c155b20 2024-04-25T00:07:03.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.641+0000 7f498959a640 1 -- 172.21.15.116:0/2269334204 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49800ab800 con 0x7f498c155b20 2024-04-25T00:07:03.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.641+0000 7f4990ba8640 1 -- 172.21.15.116:0/2269334204 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f498c155b20 msgr2=0x7f498c152c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:03.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.641+0000 7f4990ba8640 1 --2- 172.21.15.116:0/2269334204 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f498c155b20 0x7f498c152c40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f49800097f0 tx=0x7f498001c3c0 comp rx=0 tx=0).stop 2024-04-25T00:07:03.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.641+0000 7f4990ba8640 1 -- 172.21.15.116:0/2269334204 shutdown_connections 2024-04-25T00:07:03.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.641+0000 7f4990ba8640 1 --2- 172.21.15.116:0/2269334204 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f498c155b20 0x7f498c152c40 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:03.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.641+0000 7f4990ba8640 1 -- 172.21.15.116:0/2269334204 >> 172.21.15.116:0/2269334204 conn(0x7f498c06c720 msgr2=0x7f498c06cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:03.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.641+0000 7f4990ba8640 1 -- 172.21.15.116:0/2269334204 shutdown_connections 2024-04-25T00:07:03.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.641+0000 7f4990ba8640 1 -- 172.21.15.116:0/2269334204 wait complete. 2024-04-25T00:07:03.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.645+0000 7f4990ba8640 1 Processor -- start 2024-04-25T00:07:03.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.645+0000 7f4990ba8640 1 -- start start 2024-04-25T00:07:03.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.645+0000 7f4990ba8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f498c0aacd0 0x7f498c0c70f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:03.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.645+0000 7f4990ba8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f498c14c700 0x7f498c0c1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:03.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.645+0000 7f4990ba8640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f498c155b20 0x7f498c150050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:03.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.645+0000 7f4990ba8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f498c161180 con 0x7f498c14c700 2024-04-25T00:07:03.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.645+0000 7f4990ba8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f498c161000 con 0x7f498c0aacd0 2024-04-25T00:07:03.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.645+0000 7f4990ba8640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f498c161300 con 0x7f498c155b20 2024-04-25T00:07:03.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.645+0000 7f4989d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f498c14c700 0x7f498c0c1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:03.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.645+0000 7f4989d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f498c14c700 0x7f498c0c1730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56172/0 (socket says 172.21.15.116:56172) 2024-04-25T00:07:03.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.645+0000 7f4989d9b640 1 -- 172.21.15.116:0/3920394695 learned_addr learned my addr 172.21.15.116:0/3920394695 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:07:03.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.645+0000 7f498a59c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f498c0aacd0 0x7f498c0c70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:03.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.645+0000 7f498ad9d640 1 --2- 172.21.15.116:0/3920394695 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f498c155b20 0x7f498c150050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:03.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.645+0000 7f4989d9b640 1 -- 172.21.15.116:0/3920394695 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f498c155b20 msgr2=0x7f498c150050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:03.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.645+0000 7f4989d9b640 1 --2- 172.21.15.116:0/3920394695 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f498c155b20 0x7f498c150050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:03.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.645+0000 7f4989d9b640 1 -- 172.21.15.116:0/3920394695 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f498c0aacd0 msgr2=0x7f498c0c70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:03.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.645+0000 7f4989d9b640 1 --2- 172.21.15.116:0/3920394695 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f498c0aacd0 0x7f498c0c70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:03.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.645+0000 7f4989d9b640 1 -- 172.21.15.116:0/3920394695 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f498c0ab630 con 0x7f498c14c700 2024-04-25T00:07:03.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.645+0000 7f4989d9b640 1 --2- 172.21.15.116:0/3920394695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f498c14c700 0x7f498c0c1730 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f49800097f0 tx=0x7f4980019160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:03.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.645+0000 7f497affd640 1 -- 172.21.15.116:0/3920394695 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49800b0040 con 0x7f498c14c700 2024-04-25T00:07:03.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.645+0000 7f497affd640 1 -- 172.21.15.116:0/3920394695 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49800abd00 con 0x7f498c14c700 2024-04-25T00:07:03.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.645+0000 7f4990ba8640 1 -- 172.21.15.116:0/3920394695 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f498c0c6610 con 0x7f498c14c700 2024-04-25T00:07:03.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.645+0000 7f497affd640 1 -- 172.21.15.116:0/3920394695 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49800b45f0 con 0x7f498c14c700 2024-04-25T00:07:03.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.645+0000 7f4990ba8640 1 -- 172.21.15.116:0/3920394695 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f498c142080 con 0x7f498c14c700 2024-04-25T00:07:03.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.645+0000 7f4990ba8640 1 -- 172.21.15.116:0/3920394695 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f498c068a60 con 0x7f498c14c700 2024-04-25T00:07:03.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.649+0000 7f497affd640 1 -- 172.21.15.116:0/3920394695 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f49800b4790 con 0x7f498c14c700 2024-04-25T00:07:03.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.649+0000 7f497affd640 1 --2- 172.21.15.116:0/3920394695 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f496c0678c0 0x7f496c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:03.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.649+0000 7f497affd640 1 -- 172.21.15.116:0/3920394695 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 7268+0+0 (secure 0 0 0) 0x7f498011f9e0 con 0x7f498c14c700 2024-04-25T00:07:03.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.649+0000 7f498a59c640 1 --2- 172.21.15.116:0/3920394695 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f496c0678c0 0x7f496c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:03.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.649+0000 7f498a59c640 1 --2- 172.21.15.116:0/3920394695 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f496c0678c0 0x7f496c069d80 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f497c007fd0 tx=0x7f497c007d80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:03.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.653+0000 7f497affd640 1 -- 172.21.15.116:0/3920394695 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f49800160d0 con 0x7f498c14c700 2024-04-25T00:07:03.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.869+0000 7f4990ba8640 1 -- 172.21.15.116:0/3920394695 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f498c0ad630 con 0x7f498c14c700 2024-04-25T00:07:03.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.873+0000 7f497affd640 1 -- 172.21.15.116:0/3920394695 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) ==== 94+0+5674 (secure 0 0 0) 0x7f49800ef940 con 0x7f498c14c700 2024-04-25T00:07:03.873 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:07:03.874 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:06:33.449109+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5613":{"gid":5613,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":28,"addr":"172.21.15.116:6835/1233829678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1233829678},{"type":"v1","addr":"172.21.15.116:6835","nonce":1233829678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5613,"qdb_cluster":[5613]},"id":1},{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:07:03.468507+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4829":{"gid":4829,"name":"b","rank":0,"incarnation":47,"state":"up:active","state_seq":10,"addr":"172.21.15.174:6837/3583791822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3583791822},{"type":"v1","addr":"172.21.15.174:6837","nonce":3583791822}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4829,"qdb_cluster":[4829]},"id":2}]} 2024-04-25T00:07:03.874 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 50 2024-04-25T00:07:03.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.873+0000 7f4990ba8640 1 -- 172.21.15.116:0/3920394695 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f496c0678c0 msgr2=0x7f496c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:03.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.873+0000 7f4990ba8640 1 --2- 172.21.15.116:0/3920394695 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f496c0678c0 0x7f496c069d80 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f497c007fd0 tx=0x7f497c007d80 comp rx=0 tx=0).stop 2024-04-25T00:07:03.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.873+0000 7f4990ba8640 1 -- 172.21.15.116:0/3920394695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f498c14c700 msgr2=0x7f498c0c1730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:03.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.873+0000 7f4990ba8640 1 --2- 172.21.15.116:0/3920394695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f498c14c700 0x7f498c0c1730 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f49800097f0 tx=0x7f4980019160 comp rx=0 tx=0).stop 2024-04-25T00:07:03.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.873+0000 7f4990ba8640 1 -- 172.21.15.116:0/3920394695 shutdown_connections 2024-04-25T00:07:03.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.873+0000 7f4990ba8640 1 --2- 172.21.15.116:0/3920394695 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f496c0678c0 0x7f496c069d80 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:03.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.873+0000 7f4990ba8640 1 --2- 172.21.15.116:0/3920394695 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f498c155b20 0x7f498c150050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:03.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.873+0000 7f4990ba8640 1 --2- 172.21.15.116:0/3920394695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f498c14c700 0x7f498c0c1730 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:03.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.873+0000 7f4990ba8640 1 --2- 172.21.15.116:0/3920394695 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f498c0aacd0 0x7f498c0c70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:03.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.873+0000 7f4990ba8640 1 -- 172.21.15.116:0/3920394695 >> 172.21.15.116:0/3920394695 conn(0x7f498c06c720 msgr2=0x7f498c06f170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:03.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.873+0000 7f4990ba8640 1 -- 172.21.15.116:0/3920394695 shutdown_connections 2024-04-25T00:07:03.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:03.873+0000 7f4990ba8640 1 -- 172.21.15.116:0/3920394695 wait complete. 2024-04-25T00:07:03.898 INFO:tasks.mds_thrash.fs.[b]:mds.b has gained rank=0, replacing gid=5571 2024-04-25T00:07:03.898 INFO:tasks.mds_thrash.fs.[b]:waiting for 1 secs before reviving mds.c 2024-04-25T00:07:04.899 INFO:tasks.mds_thrash.fs.[b]:reviving mds.c 2024-04-25T00:07:04.899 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-25T00:07:04.899 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-25T00:07:04.901 INFO:tasks.ceph.mds.c:Started 2024-04-25T00:07:04.902 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:07:04.983 INFO:tasks.ceph.mds.c.smithi116.stdout:starting mds.c at 2024-04-25T00:07:05.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.041+0000 7f04b04c2640 1 Processor -- start 2024-04-25T00:07:05.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.041+0000 7f04b04c2640 1 -- start start 2024-04-25T00:07:05.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.041+0000 7f04b04c2640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f04a8152d80 0x7f04a806bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:05.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.041+0000 7f04b04c2640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f04a806c290 0x7f04a8157390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:05.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.041+0000 7f04b04c2640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04a80c9150 con 0x7f04a806d5f0 2024-04-25T00:07:05.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.041+0000 7f04b04c2640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f04a80c9d60 con 0x7f04a806c290 2024-04-25T00:07:05.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.041+0000 7f04b04c2640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f04a80c9430 con 0x7f04a8152d80 2024-04-25T00:07:05.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.041+0000 7f04aea5f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f04a806c290 0x7f04a8157390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:05.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.041+0000 7f04aea5f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f04a806c290 0x7f04a8157390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36136/0 (socket says 172.21.15.116:36136) 2024-04-25T00:07:05.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.041+0000 7f04aea5f640 1 -- 172.21.15.116:0/3966620158 learned_addr learned my addr 172.21.15.116:0/3966620158 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:07:05.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.041+0000 7f04ae25e640 1 --2- 172.21.15.116:0/3966620158 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f04a8152d80 0x7f04a806bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:05.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.041+0000 7f04aea5f640 1 -- 172.21.15.116:0/3966620158 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f04a8152d80 msgr2=0x7f04a806bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:07:05.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.041+0000 7f04aea5f640 1 --2- 172.21.15.116:0/3966620158 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f04a8152d80 0x7f04a806bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:05.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.041+0000 7f04aea5f640 1 -- 172.21.15.116:0/3966620158 >> v1:172.21.15.116:6789/0 conn(0x7f04a806d5f0 legacy=0x7f04a8156c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:05.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.041+0000 7f04aea5f640 1 -- 172.21.15.116:0/3966620158 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04a80c9710 con 0x7f04a806c290 2024-04-25T00:07:05.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.045+0000 7f04aea5f640 1 --2- 172.21.15.116:0/3966620158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04a806c290 0x7f04a8157390 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f04a0009870 tx=0x7f04a001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b82190a766bc50fb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:05.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.045+0000 7f04ad25c640 1 -- 172.21.15.116:0/3966620158 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a00aa620 con 0x7f04a806c290 2024-04-25T00:07:05.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.045+0000 7f04ad25c640 1 -- 172.21.15.116:0/3966620158 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04a00af070 con 0x7f04a806c290 2024-04-25T00:07:05.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.045+0000 7f04ad25c640 1 -- 172.21.15.116:0/3966620158 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a00aa9e0 con 0x7f04a806c290 2024-04-25T00:07:05.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.045+0000 7f04b04c2640 1 -- 172.21.15.116:0/3966620158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04a806c290 msgr2=0x7f04a8157390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:05.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.045+0000 7f04b04c2640 1 --2- 172.21.15.116:0/3966620158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04a806c290 0x7f04a8157390 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f04a0009870 tx=0x7f04a001c4a0 comp rx=0 tx=0).stop 2024-04-25T00:07:05.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.045+0000 7f04b04c2640 1 -- 172.21.15.116:0/3966620158 shutdown_connections 2024-04-25T00:07:05.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.045+0000 7f04b04c2640 1 --2- 172.21.15.116:0/3966620158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04a806c290 0x7f04a8157390 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:05.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.045+0000 7f04b04c2640 1 --2- 172.21.15.116:0/3966620158 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f04a8152d80 0x7f04a806bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:05.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.045+0000 7f04b04c2640 1 -- 172.21.15.116:0/3966620158 >> 172.21.15.116:0/3966620158 conn(0x7f04a80acdd0 msgr2=0x7f04a80ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:05.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.045+0000 7f04b04c2640 1 -- 172.21.15.116:0/3966620158 shutdown_connections 2024-04-25T00:07:05.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.045+0000 7f04b04c2640 1 -- 172.21.15.116:0/3966620158 wait complete. 2024-04-25T00:07:05.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.045+0000 7f04b04c2640 1 Processor -- start 2024-04-25T00:07:05.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.045+0000 7f04b04c2640 1 -- start start 2024-04-25T00:07:05.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.049+0000 7f04b04c2640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f04a806c290 0x7f04a80bf5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:05.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.049+0000 7f04b04c2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f04a806d5f0 0x7f04a81f1c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:05.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.049+0000 7f04b04c2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04a8152d80 0x7f04a80c0a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:05.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.049+0000 7f04b04c2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f04a80caf00 con 0x7f04a8152d80 2024-04-25T00:07:05.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.049+0000 7f04b04c2640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f04a80cad80 con 0x7f04a806d5f0 2024-04-25T00:07:05.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.049+0000 7f04b04c2640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f04a81628d0 con 0x7f04a806c290 2024-04-25T00:07:05.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.049+0000 7f04aea5f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04a8152d80 0x7f04a80c0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:05.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.049+0000 7f04aea5f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04a8152d80 0x7f04a80c0a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36140/0 (socket says 172.21.15.116:36140) 2024-04-25T00:07:05.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.049+0000 7f04aea5f640 1 -- 172.21.15.116:0/427875527 learned_addr learned my addr 172.21.15.116:0/427875527 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:07:05.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.049+0000 7f04ae25e640 1 --2- 172.21.15.116:0/427875527 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f04a806c290 0x7f04a80bf5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:05.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.049+0000 7f04ada5d640 1 --2- 172.21.15.116:0/427875527 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f04a806d5f0 0x7f04a81f1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:05.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.049+0000 7f04aea5f640 1 -- 172.21.15.116:0/427875527 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f04a806c290 msgr2=0x7f04a80bf5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:05.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.049+0000 7f04aea5f640 1 --2- 172.21.15.116:0/427875527 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f04a806c290 0x7f04a80bf5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:05.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.049+0000 7f04aea5f640 1 -- 172.21.15.116:0/427875527 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f04a806d5f0 msgr2=0x7f04a81f1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:05.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.049+0000 7f04aea5f640 1 --2- 172.21.15.116:0/427875527 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f04a806d5f0 0x7f04a81f1c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:05.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.049+0000 7f04aea5f640 1 -- 172.21.15.116:0/427875527 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04a814c700 con 0x7f04a8152d80 2024-04-25T00:07:05.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.049+0000 7f04aea5f640 1 --2- 172.21.15.116:0/427875527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04a8152d80 0x7f04a80c0a00 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f04a001a0d0 tx=0x7f04a001ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:05.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.049+0000 7f049effd640 1 -- 172.21.15.116:0/427875527 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a00aaca0 con 0x7f04a8152d80 2024-04-25T00:07:05.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.049+0000 7f049effd640 1 -- 172.21.15.116:0/427875527 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04a00aae40 con 0x7f04a8152d80 2024-04-25T00:07:05.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.049+0000 7f04b04c2640 1 -- 172.21.15.116:0/427875527 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04a81f2520 con 0x7f04a8152d80 2024-04-25T00:07:05.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.049+0000 7f049effd640 1 -- 172.21.15.116:0/427875527 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a00bd630 con 0x7f04a8152d80 2024-04-25T00:07:05.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.049+0000 7f04b04c2640 1 -- 172.21.15.116:0/427875527 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04a80c15c0 con 0x7f04a8152d80 2024-04-25T00:07:05.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.049+0000 7f04b04c2640 1 -- 172.21.15.116:0/427875527 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04a8068350 con 0x7f04a8152d80 2024-04-25T00:07:05.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.049+0000 7f049effd640 1 -- 172.21.15.116:0/427875527 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f04a00c6430 con 0x7f04a8152d80 2024-04-25T00:07:05.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.049+0000 7f049effd640 1 --2- 172.21.15.116:0/427875527 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f048c0678f0 0x7f048c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:05.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.053+0000 7f049effd640 1 -- 172.21.15.116:0/427875527 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 7268+0+0 (secure 0 0 0) 0x7f04a01205f0 con 0x7f04a8152d80 2024-04-25T00:07:05.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.053+0000 7f04ae25e640 1 --2- 172.21.15.116:0/427875527 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f048c0678f0 0x7f048c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:05.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.053+0000 7f04ae25e640 1 --2- 172.21.15.116:0/427875527 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f048c0678f0 0x7f048c069db0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f04a81593b0 tx=0x7f04a400b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:05.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.057+0000 7f049effd640 1 -- 172.21.15.116:0/427875527 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f04a814c700 con 0x7f04a8152d80 2024-04-25T00:07:05.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.261+0000 7f04b04c2640 1 -- 172.21.15.116:0/427875527 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f04a815fda0 con 0x7f04a8152d80 2024-04-25T00:07:05.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.261+0000 7f049effd640 1 -- 172.21.15.116:0/427875527 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) ==== 94+0+5674 (secure 0 0 0) 0x7f04a0016150 con 0x7f04a8152d80 2024-04-25T00:07:05.262 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:07:05.262 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:06:33.449109+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5613":{"gid":5613,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":28,"addr":"172.21.15.116:6835/1233829678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1233829678},{"type":"v1","addr":"172.21.15.116:6835","nonce":1233829678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5613,"qdb_cluster":[5613]},"id":1},{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:07:03.468507+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4829":{"gid":4829,"name":"b","rank":0,"incarnation":47,"state":"up:active","state_seq":10,"addr":"172.21.15.174:6837/3583791822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3583791822},{"type":"v1","addr":"172.21.15.174:6837","nonce":3583791822}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4829,"qdb_cluster":[4829]},"id":2}]} 2024-04-25T00:07:05.263 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 50 2024-04-25T00:07:05.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.261+0000 7f04b04c2640 1 -- 172.21.15.116:0/427875527 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f048c0678f0 msgr2=0x7f048c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:05.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.261+0000 7f04b04c2640 1 --2- 172.21.15.116:0/427875527 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f048c0678f0 0x7f048c069db0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f04a81593b0 tx=0x7f04a400b040 comp rx=0 tx=0).stop 2024-04-25T00:07:05.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.261+0000 7f04b04c2640 1 -- 172.21.15.116:0/427875527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04a8152d80 msgr2=0x7f04a80c0a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:05.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.261+0000 7f04b04c2640 1 --2- 172.21.15.116:0/427875527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04a8152d80 0x7f04a80c0a00 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f04a001a0d0 tx=0x7f04a001ace0 comp rx=0 tx=0).stop 2024-04-25T00:07:05.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.261+0000 7f04b04c2640 1 -- 172.21.15.116:0/427875527 shutdown_connections 2024-04-25T00:07:05.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.261+0000 7f04b04c2640 1 --2- 172.21.15.116:0/427875527 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f048c0678f0 0x7f048c069db0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:05.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.261+0000 7f04b04c2640 1 --2- 172.21.15.116:0/427875527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04a8152d80 0x7f04a80c0a00 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:05.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.261+0000 7f04b04c2640 1 --2- 172.21.15.116:0/427875527 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f04a806d5f0 0x7f04a81f1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:05.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.261+0000 7f04b04c2640 1 --2- 172.21.15.116:0/427875527 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f04a806c290 0x7f04a80bf5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:05.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.261+0000 7f04b04c2640 1 -- 172.21.15.116:0/427875527 >> 172.21.15.116:0/427875527 conn(0x7f04a80acdd0 msgr2=0x7f04a8155d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:05.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.261+0000 7f04b04c2640 1 -- 172.21.15.116:0/427875527 shutdown_connections 2024-04-25T00:07:05.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:05.261+0000 7f04b04c2640 1 -- 172.21.15.116:0/427875527 wait complete. 2024-04-25T00:07:05.280 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2024-04-25T00:07:05.700 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:07:05.703 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:07:06.776 INFO:tasks.ceph.mds.b.smithi174.stderr:2024-04-25T00:07:06.770+0000 7fb0cbb17640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T00:07:07.281 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:07:07.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.409+0000 7fa329039640 1 Processor -- start 2024-04-25T00:07:07.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.413+0000 7fa329039640 1 -- start start 2024-04-25T00:07:07.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.413+0000 7fa329039640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3240c5ef0 0x7fa324155fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:07.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.413+0000 7fa329039640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa324156510 con 0x7fa324152db0 2024-04-25T00:07:07.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.413+0000 7fa329039640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa324158eb0 con 0x7fa3240c5a60 2024-04-25T00:07:07.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.413+0000 7fa329039640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa324159190 con 0x7fa3240c5ef0 2024-04-25T00:07:07.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.413+0000 7fa322d9d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fa324152db0 0x7fa3240c7af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:40930/0 (socket says 172.21.15.116:40930) 2024-04-25T00:07:07.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.413+0000 7fa322d9d640 1 -- 172.21.15.116:0/161617873 learned_addr learned my addr 172.21.15.116:0/161617873 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:07:07.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.413+0000 7fa32359e640 1 --2- 172.21.15.116:0/161617873 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3240c5ef0 0x7fa324155fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:07.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.413+0000 7fa32259c640 1 -- 172.21.15.116:0/161617873 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 243567319 0 0) 0x7fa324156510 con 0x7fa324152db0 2024-04-25T00:07:07.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.413+0000 7fa32259c640 1 -- 172.21.15.116:0/161617873 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa310003530 con 0x7fa324152db0 2024-04-25T00:07:07.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.413+0000 7fa32259c640 1 -- 172.21.15.116:0/161617873 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3747369826 0 0) 0x7fa324158eb0 con 0x7fa3240c5a60 2024-04-25T00:07:07.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.413+0000 7fa32259c640 1 -- 172.21.15.116:0/161617873 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa324156510 con 0x7fa3240c5a60 2024-04-25T00:07:07.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.413+0000 7fa32359e640 1 -- 172.21.15.116:0/161617873 >> v1:172.21.15.174:6789/0 conn(0x7fa3240c5a60 legacy=0x7fa3241558c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:07.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.413+0000 7fa32359e640 1 -- 172.21.15.116:0/161617873 >> v1:172.21.15.116:6789/0 conn(0x7fa324152db0 legacy=0x7fa3240c7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:07.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.413+0000 7fa32359e640 1 -- 172.21.15.116:0/161617873 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa324159470 con 0x7fa3240c5ef0 2024-04-25T00:07:07.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.413+0000 7fa32359e640 1 --2- 172.21.15.116:0/161617873 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3240c5ef0 0x7fa324155fd0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa31c009dc0 tx=0x7fa31c01c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7273d3b05afad1d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:07.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.417+0000 7fa32259c640 1 -- 172.21.15.116:0/161617873 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa31c0b2070 con 0x7fa3240c5ef0 2024-04-25T00:07:07.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.417+0000 7fa32259c640 1 -- 172.21.15.116:0/161617873 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa31c0a58b0 con 0x7fa3240c5ef0 2024-04-25T00:07:07.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.417+0000 7fa32259c640 1 -- 172.21.15.116:0/161617873 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa31c0a5c80 con 0x7fa3240c5ef0 2024-04-25T00:07:07.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.417+0000 7fa329039640 1 -- 172.21.15.116:0/161617873 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3240c5ef0 msgr2=0x7fa324155fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:07.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.417+0000 7fa329039640 1 --2- 172.21.15.116:0/161617873 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3240c5ef0 0x7fa324155fd0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa31c009dc0 tx=0x7fa31c01c900 comp rx=0 tx=0).stop 2024-04-25T00:07:07.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.417+0000 7fa329039640 1 -- 172.21.15.116:0/161617873 shutdown_connections 2024-04-25T00:07:07.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.417+0000 7fa329039640 1 --2- 172.21.15.116:0/161617873 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3240c5ef0 0x7fa324155fd0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:07.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.417+0000 7fa329039640 1 -- 172.21.15.116:0/161617873 >> 172.21.15.116:0/161617873 conn(0x7fa3240acc10 msgr2=0x7fa3240ad020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:07.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.417+0000 7fa329039640 1 -- 172.21.15.116:0/161617873 shutdown_connections 2024-04-25T00:07:07.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.417+0000 7fa329039640 1 -- 172.21.15.116:0/161617873 wait complete. 2024-04-25T00:07:07.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.417+0000 7fa329039640 1 Processor -- start 2024-04-25T00:07:07.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.417+0000 7fa329039640 1 -- start start 2024-04-25T00:07:07.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.417+0000 7fa329039640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3240c5a60 0x7fa3241455c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:07.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.417+0000 7fa329039640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3240c5ef0 0x7fa3240c0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:07.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.417+0000 7fa329039640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa324152db0 0x7fa3240c0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:07.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.417+0000 7fa329039640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa324153190 con 0x7fa324152db0 2024-04-25T00:07:07.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.417+0000 7fa329039640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa3240cadf0 con 0x7fa3240c5ef0 2024-04-25T00:07:07.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.417+0000 7fa329039640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa324153310 con 0x7fa3240c5a60 2024-04-25T00:07:07.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.417+0000 7fa32359e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa324152db0 0x7fa3240c0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:07.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.417+0000 7fa32359e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa324152db0 0x7fa3240c0d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36156/0 (socket says 172.21.15.116:36156) 2024-04-25T00:07:07.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.417+0000 7fa32359e640 1 -- 172.21.15.116:0/1112476828 learned_addr learned my addr 172.21.15.116:0/1112476828 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:07:07.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.417+0000 7fa322d9d640 1 --2- 172.21.15.116:0/1112476828 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3240c5a60 0x7fa3241455c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:07.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.417+0000 7fa31bfff640 1 --2- 172.21.15.116:0/1112476828 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3240c5ef0 0x7fa3240c0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:07.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.417+0000 7fa32359e640 1 -- 172.21.15.116:0/1112476828 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3240c5a60 msgr2=0x7fa3241455c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:07.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.417+0000 7fa32359e640 1 --2- 172.21.15.116:0/1112476828 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3240c5a60 0x7fa3241455c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:07.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.417+0000 7fa32359e640 1 -- 172.21.15.116:0/1112476828 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3240c5ef0 msgr2=0x7fa3240c0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:07.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.417+0000 7fa32359e640 1 --2- 172.21.15.116:0/1112476828 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3240c5ef0 0x7fa3240c0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:07.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.417+0000 7fa32359e640 1 -- 172.21.15.116:0/1112476828 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3241469c0 con 0x7fa324152db0 2024-04-25T00:07:07.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.417+0000 7fa32359e640 1 --2- 172.21.15.116:0/1112476828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa324152db0 0x7fa3240c0d50 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fa31c01cde0 tx=0x7fa31c0196a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:07.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.417+0000 7fa31b7fe640 1 -- 172.21.15.116:0/1112476828 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa31c0b2050 con 0x7fa324152db0 2024-04-25T00:07:07.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.417+0000 7fa31b7fe640 1 -- 172.21.15.116:0/1112476828 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa31c0be070 con 0x7fa324152db0 2024-04-25T00:07:07.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.417+0000 7fa329039640 1 -- 172.21.15.116:0/1112476828 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3240c1690 con 0x7fa324152db0 2024-04-25T00:07:07.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.421+0000 7fa31b7fe640 1 -- 172.21.15.116:0/1112476828 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa31c0ad870 con 0x7fa324152db0 2024-04-25T00:07:07.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.421+0000 7fa329039640 1 -- 172.21.15.116:0/1112476828 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3241589a0 con 0x7fa324152db0 2024-04-25T00:07:07.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.421+0000 7fa329039640 1 -- 172.21.15.116:0/1112476828 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa324068a60 con 0x7fa324152db0 2024-04-25T00:07:07.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.421+0000 7fa31b7fe640 1 -- 172.21.15.116:0/1112476828 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa31c0ada10 con 0x7fa324152db0 2024-04-25T00:07:07.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.421+0000 7fa31b7fe640 1 --2- 172.21.15.116:0/1112476828 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa3040678c0 0x7fa304069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:07.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.421+0000 7fa31b7fe640 1 -- 172.21.15.116:0/1112476828 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 7268+0+0 (secure 0 0 0) 0x7fa31c1211f0 con 0x7fa324152db0 2024-04-25T00:07:07.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.421+0000 7fa322d9d640 1 --2- 172.21.15.116:0/1112476828 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa3040678c0 0x7fa304069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:07.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.421+0000 7fa322d9d640 1 --2- 172.21.15.116:0/1112476828 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa3040678c0 0x7fa304069d80 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa314007fd0 tx=0x7fa314007d60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:07.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.429+0000 7fa31b7fe640 1 -- 172.21.15.116:0/1112476828 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fa31c016580 con 0x7fa324152db0 2024-04-25T00:07:07.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.661+0000 7fa329039640 1 -- 172.21.15.116:0/1112476828 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa3240c51e0 con 0x7fa324152db0 2024-04-25T00:07:07.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.661+0000 7fa31b7fe640 1 -- 172.21.15.116:0/1112476828 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+6502 (secure 0 0 0) 0x7fa31c0add00 con 0x7fa324152db0 2024-04-25T00:07:07.665 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:07:07.665 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5775,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/3006671671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3006671671},{"type":"v1","addr":"172.21.15.116:6833","nonce":3006671671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:06:33.449109+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5613":{"gid":5613,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":28,"addr":"172.21.15.116:6835/1233829678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1233829678},{"type":"v1","addr":"172.21.15.116:6835","nonce":1233829678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5613,"qdb_cluster":[5613]},"id":1},{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:07:03.468507+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4829":{"gid":4829,"name":"b","rank":0,"incarnation":47,"state":"up:active","state_seq":10,"addr":"172.21.15.174:6837/3583791822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3583791822},{"type":"v1","addr":"172.21.15.174:6837","nonce":3583791822}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4829,"qdb_cluster":[4829]},"id":2}]} 2024-04-25T00:07:07.665 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 51 2024-04-25T00:07:07.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.665+0000 7fa329039640 1 -- 172.21.15.116:0/1112476828 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa3040678c0 msgr2=0x7fa304069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:07.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.665+0000 7fa329039640 1 --2- 172.21.15.116:0/1112476828 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa3040678c0 0x7fa304069d80 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa314007fd0 tx=0x7fa314007d60 comp rx=0 tx=0).stop 2024-04-25T00:07:07.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.665+0000 7fa329039640 1 -- 172.21.15.116:0/1112476828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa324152db0 msgr2=0x7fa3240c0d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:07.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.665+0000 7fa329039640 1 --2- 172.21.15.116:0/1112476828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa324152db0 0x7fa3240c0d50 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fa31c01cde0 tx=0x7fa31c0196a0 comp rx=0 tx=0).stop 2024-04-25T00:07:07.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.665+0000 7fa329039640 1 -- 172.21.15.116:0/1112476828 shutdown_connections 2024-04-25T00:07:07.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.665+0000 7fa329039640 1 --2- 172.21.15.116:0/1112476828 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa3040678c0 0x7fa304069d80 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:07.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.665+0000 7fa329039640 1 --2- 172.21.15.116:0/1112476828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa324152db0 0x7fa3240c0d50 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:07.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.665+0000 7fa329039640 1 --2- 172.21.15.116:0/1112476828 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3240c5ef0 0x7fa3240c0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:07.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.665+0000 7fa329039640 1 --2- 172.21.15.116:0/1112476828 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3240c5a60 0x7fa3241455c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:07.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.665+0000 7fa329039640 1 -- 172.21.15.116:0/1112476828 >> 172.21.15.116:0/1112476828 conn(0x7fa3240acc10 msgr2=0x7fa3240acff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:07.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.665+0000 7fa329039640 1 -- 172.21.15.116:0/1112476828 shutdown_connections 2024-04-25T00:07:07.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:07.665+0000 7fa329039640 1 -- 172.21.15.116:0/1112476828 wait complete. 2024-04-25T00:07:07.685 INFO:tasks.mds_thrash.fs.[b]:mds.c reported in up:standby state 2024-04-25T00:07:07.686 INFO:tasks.mds_thrash.fs.[b]:waiting for 36 secs before thrashing 2024-04-25T00:07:15.017 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:07:15.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.165+0000 7f5c58f80640 1 Processor -- start 2024-04-25T00:07:15.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.165+0000 7f5c58f80640 1 -- start start 2024-04-25T00:07:15.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.165+0000 7f5c58f80640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5c54150f30 0x7f5c540c39a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:15.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.165+0000 7f5c58f80640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5c54156bb0 0x7f5c54157060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:15.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.165+0000 7f5c58f80640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c540c81d0 con 0x7f5c54070750 2024-04-25T00:07:15.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.165+0000 7f5c58f80640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f5c5406edb0 con 0x7f5c54150f30 2024-04-25T00:07:15.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.165+0000 7f5c58f80640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f5c540c8470 con 0x7f5c54156bb0 2024-04-25T00:07:15.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.165+0000 7f5c52ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5c54150f30 0x7f5c540c39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:15.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.165+0000 7f5c52ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5c54150f30 0x7f5c540c39a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57320/0 (socket says 172.21.15.116:57320) 2024-04-25T00:07:15.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.165+0000 7f5c52ffd640 1 -- 172.21.15.116:0/1612296532 learned_addr learned my addr 172.21.15.116:0/1612296532 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:07:15.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.165+0000 7f5c537fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f5c54070750 0x7f5c54150820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:56854/0 (socket says 172.21.15.116:56854) 2024-04-25T00:07:15.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.165+0000 7f5c53fff640 1 --2- 172.21.15.116:0/1612296532 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5c54156bb0 0x7f5c54157060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:15.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.165+0000 7f5c52ffd640 1 -- 172.21.15.116:0/1612296532 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5c54156bb0 msgr2=0x7f5c54157060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:07:15.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.165+0000 7f5c52ffd640 1 --2- 172.21.15.116:0/1612296532 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5c54156bb0 0x7f5c54157060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:15.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.165+0000 7f5c52ffd640 1 -- 172.21.15.116:0/1612296532 >> v1:172.21.15.116:6789/0 conn(0x7f5c54070750 legacy=0x7f5c54150820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:15.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.165+0000 7f5c527fc640 1 -- 172.21.15.116:0/1612296532 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1527361618 0 0) 0x7f5c540c81d0 con 0x7f5c54070750 2024-04-25T00:07:15.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.165+0000 7f5c52ffd640 1 -- 172.21.15.116:0/1612296532 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c540c8750 con 0x7f5c54150f30 2024-04-25T00:07:15.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.169+0000 7f5c52ffd640 1 --2- 172.21.15.116:0/1612296532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c54150f30 0x7f5c540c39a0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f5c44009870 tx=0x7f5c4401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b999d3885db8ca87 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:15.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.169+0000 7f5c527fc640 1 -- 172.21.15.116:0/1612296532 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c440aa620 con 0x7f5c54150f30 2024-04-25T00:07:15.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.169+0000 7f5c527fc640 1 -- 172.21.15.116:0/1612296532 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c440af070 con 0x7f5c54150f30 2024-04-25T00:07:15.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.169+0000 7f5c527fc640 1 -- 172.21.15.116:0/1612296532 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c440aa9e0 con 0x7f5c54150f30 2024-04-25T00:07:15.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.169+0000 7f5c58f80640 1 -- 172.21.15.116:0/1612296532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c54150f30 msgr2=0x7f5c540c39a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:15.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.169+0000 7f5c58f80640 1 --2- 172.21.15.116:0/1612296532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c54150f30 0x7f5c540c39a0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f5c44009870 tx=0x7f5c4401c4a0 comp rx=0 tx=0).stop 2024-04-25T00:07:15.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.169+0000 7f5c58f80640 1 -- 172.21.15.116:0/1612296532 shutdown_connections 2024-04-25T00:07:15.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.169+0000 7f5c58f80640 1 --2- 172.21.15.116:0/1612296532 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5c54156bb0 0x7f5c54157060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:15.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.169+0000 7f5c58f80640 1 --2- 172.21.15.116:0/1612296532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c54150f30 0x7f5c540c39a0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:15.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.169+0000 7f5c58f80640 1 -- 172.21.15.116:0/1612296532 >> 172.21.15.116:0/1612296532 conn(0x7f5c540c8e80 msgr2=0x7f5c540c9290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:15.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.169+0000 7f5c58f80640 1 -- 172.21.15.116:0/1612296532 shutdown_connections 2024-04-25T00:07:15.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.169+0000 7f5c58f80640 1 -- 172.21.15.116:0/1612296532 wait complete. 2024-04-25T00:07:15.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.173+0000 7f5c58f80640 1 Processor -- start 2024-04-25T00:07:15.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.173+0000 7f5c58f80640 1 -- start start 2024-04-25T00:07:15.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.173+0000 7f5c58f80640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c54070750 0x7f5c541ee0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:15.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.173+0000 7f5c58f80640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5c54150f30 0x7f5c540c2510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:15.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.173+0000 7f5c58f80640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5c54156bb0 0x7f5c541f1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:15.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.173+0000 7f5c58f80640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5c5406e940 con 0x7f5c54070750 2024-04-25T00:07:15.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.173+0000 7f5c58f80640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5c54070d80 con 0x7f5c54150f30 2024-04-25T00:07:15.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.173+0000 7f5c58f80640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f5c5406eac0 con 0x7f5c54156bb0 2024-04-25T00:07:15.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.173+0000 7f5c537fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c54070750 0x7f5c541ee0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:15.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.173+0000 7f5c537fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c54070750 0x7f5c541ee0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57332/0 (socket says 172.21.15.116:57332) 2024-04-25T00:07:15.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.173+0000 7f5c537fe640 1 -- 172.21.15.116:0/1627294581 learned_addr learned my addr 172.21.15.116:0/1627294581 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:07:15.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.173+0000 7f5c53fff640 1 --2- 172.21.15.116:0/1627294581 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5c54156bb0 0x7f5c541f1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:15.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.173+0000 7f5c52ffd640 1 --2- 172.21.15.116:0/1627294581 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5c54150f30 0x7f5c540c2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:15.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.173+0000 7f5c537fe640 1 -- 172.21.15.116:0/1627294581 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5c54156bb0 msgr2=0x7f5c541f1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:15.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.173+0000 7f5c537fe640 1 --2- 172.21.15.116:0/1627294581 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5c54156bb0 0x7f5c541f1ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:15.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.173+0000 7f5c537fe640 1 -- 172.21.15.116:0/1627294581 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5c54150f30 msgr2=0x7f5c540c2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:15.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.173+0000 7f5c537fe640 1 --2- 172.21.15.116:0/1627294581 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5c54150f30 0x7f5c540c2510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:15.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.173+0000 7f5c537fe640 1 -- 172.21.15.116:0/1627294581 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c540ace80 con 0x7f5c54070750 2024-04-25T00:07:15.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.173+0000 7f5c53fff640 1 --2- 172.21.15.116:0/1627294581 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5c54156bb0 0x7f5c541f1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:07:15.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.173+0000 7f5c537fe640 1 --2- 172.21.15.116:0/1627294581 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c54070750 0x7f5c541ee0b0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f5c480011f0 tx=0x7f5c4800c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:15.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.173+0000 7f5c3bfff640 1 -- 172.21.15.116:0/1627294581 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c48018070 con 0x7f5c54070750 2024-04-25T00:07:15.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.173+0000 7f5c3bfff640 1 -- 172.21.15.116:0/1627294581 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c480074e0 con 0x7f5c54070750 2024-04-25T00:07:15.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.173+0000 7f5c3bfff640 1 -- 172.21.15.116:0/1627294581 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c48004a10 con 0x7f5c54070750 2024-04-25T00:07:15.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.173+0000 7f5c58f80640 1 -- 172.21.15.116:0/1627294581 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c540ac8c0 con 0x7f5c54070750 2024-04-25T00:07:15.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.173+0000 7f5c58f80640 1 -- 172.21.15.116:0/1627294581 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c541f16c0 con 0x7f5c54070750 2024-04-25T00:07:15.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.173+0000 7f5c58f80640 1 -- 172.21.15.116:0/1627294581 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c5404f510 con 0x7f5c54070750 2024-04-25T00:07:15.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.173+0000 7f5c3bfff640 1 -- 172.21.15.116:0/1627294581 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5c4801b070 con 0x7f5c54070750 2024-04-25T00:07:15.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.177+0000 7f5c3bfff640 1 --2- 172.21.15.116:0/1627294581 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5c400678c0 0x7f5c40069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:15.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.177+0000 7f5c3bfff640 1 -- 172.21.15.116:0/1627294581 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 7268+0+0 (secure 0 0 0) 0x7f5c480855c0 con 0x7f5c54070750 2024-04-25T00:07:15.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.177+0000 7f5c52ffd640 1 --2- 172.21.15.116:0/1627294581 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5c400678c0 0x7f5c40069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:15.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.177+0000 7f5c52ffd640 1 --2- 172.21.15.116:0/1627294581 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5c400678c0 0x7f5c40069d80 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f5c440ae000 tx=0x7f5c440a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:15.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.181+0000 7f5c3bfff640 1 -- 172.21.15.116:0/1627294581 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f5c48055b70 con 0x7f5c54070750 2024-04-25T00:07:15.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.401+0000 7f5c58f80640 1 -- 172.21.15.116:0/1627294581 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5c540addc0 con 0x7f5c54070750 2024-04-25T00:07:15.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.405+0000 7f5c3bfff640 1 -- 172.21.15.116:0/1627294581 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+6502 (secure 0 0 0) 0x7f5c48055470 con 0x7f5c54070750 2024-04-25T00:07:15.406 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:07:15.406 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5775,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/3006671671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3006671671},{"type":"v1","addr":"172.21.15.116:6833","nonce":3006671671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:06:33.449109+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5613":{"gid":5613,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":28,"addr":"172.21.15.116:6835/1233829678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1233829678},{"type":"v1","addr":"172.21.15.116:6835","nonce":1233829678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5613,"qdb_cluster":[5613]},"id":1},{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:07:03.468507+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4829":{"gid":4829,"name":"b","rank":0,"incarnation":47,"state":"up:active","state_seq":10,"addr":"172.21.15.174:6837/3583791822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3583791822},{"type":"v1","addr":"172.21.15.174:6837","nonce":3583791822}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4829,"qdb_cluster":[4829]},"id":2}]} 2024-04-25T00:07:15.406 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 51 2024-04-25T00:07:15.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.405+0000 7f5c58f80640 1 -- 172.21.15.116:0/1627294581 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5c400678c0 msgr2=0x7f5c40069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:15.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.405+0000 7f5c58f80640 1 --2- 172.21.15.116:0/1627294581 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5c400678c0 0x7f5c40069d80 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f5c440ae000 tx=0x7f5c440a7000 comp rx=0 tx=0).stop 2024-04-25T00:07:15.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.405+0000 7f5c58f80640 1 -- 172.21.15.116:0/1627294581 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c54070750 msgr2=0x7f5c541ee0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:15.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.405+0000 7f5c58f80640 1 --2- 172.21.15.116:0/1627294581 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c54070750 0x7f5c541ee0b0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f5c480011f0 tx=0x7f5c4800c9b0 comp rx=0 tx=0).stop 2024-04-25T00:07:15.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.405+0000 7f5c58f80640 1 -- 172.21.15.116:0/1627294581 shutdown_connections 2024-04-25T00:07:15.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.405+0000 7f5c58f80640 1 --2- 172.21.15.116:0/1627294581 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5c400678c0 0x7f5c40069d80 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:15.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.405+0000 7f5c58f80640 1 --2- 172.21.15.116:0/1627294581 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5c54156bb0 0x7f5c541f1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:15.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.405+0000 7f5c58f80640 1 --2- 172.21.15.116:0/1627294581 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5c54150f30 0x7f5c540c2510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:15.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.405+0000 7f5c58f80640 1 --2- 172.21.15.116:0/1627294581 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c54070750 0x7f5c541ee0b0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:15.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.405+0000 7f5c58f80640 1 -- 172.21.15.116:0/1627294581 >> 172.21.15.116:0/1627294581 conn(0x7f5c540c8e80 msgr2=0x7f5c5414fb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:15.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.405+0000 7f5c58f80640 1 -- 172.21.15.116:0/1627294581 shutdown_connections 2024-04-25T00:07:15.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.405+0000 7f5c58f80640 1 -- 172.21.15.116:0/1627294581 wait complete. 2024-04-25T00:07:15.425 INFO:tasks.mds_thrash.fs.[a]:kill mds.a (rank=0) 2024-04-25T00:07:15.425 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-25T00:07:15.425 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:07:15.437 INFO:tasks.ceph.mds.a:Stopped 2024-04-25T00:07:15.438 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:07:15.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.613+0000 7f66f9e6d640 1 Processor -- start 2024-04-25T00:07:15.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.613+0000 7f66f9e6d640 1 -- start start 2024-04-25T00:07:15.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.613+0000 7f66f9e6d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f66f40aa170 0x7f66f4155af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:15.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.613+0000 7f66f9e6d640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f66f4156030 0x7f66f415ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:15.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.613+0000 7f66f9e6d640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66f41526a0 con 0x7f66f40c4300 2024-04-25T00:07:15.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.613+0000 7f66f9e6d640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f66f4153460 con 0x7f66f4156030 2024-04-25T00:07:15.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.613+0000 7f66f9e6d640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f66f406ffd0 con 0x7f66f40aa170 2024-04-25T00:07:15.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.613+0000 7f66ebfff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f66f40aa170 0x7f66f4155af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:15.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.613+0000 7f66f3fff640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f66f4156030 0x7f66f415ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:15.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.613+0000 7f66f37fe640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f66f40c4300 0x7f66f40c5100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:34604/0 (socket says 172.21.15.116:34604) 2024-04-25T00:07:15.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.613+0000 7f66f37fe640 1 -- 172.21.15.116:0/857050992 learned_addr learned my addr 172.21.15.116:0/857050992 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:07:15.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.617+0000 7f66f3fff640 1 -- 172.21.15.116:0/857050992 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f66f40aa170 msgr2=0x7f66f4155af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:07:15.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.617+0000 7f66f3fff640 1 --2- 172.21.15.116:0/857050992 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f66f40aa170 0x7f66f4155af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:15.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.617+0000 7f66f3fff640 1 -- 172.21.15.116:0/857050992 >> v1:172.21.15.174:6789/0 conn(0x7f66f40c4300 legacy=0x7f66f40c5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:15.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.617+0000 7f66f3fff640 1 -- 172.21.15.116:0/857050992 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66f4152980 con 0x7f66f4156030 2024-04-25T00:07:15.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.617+0000 7f66f3fff640 1 --2- 172.21.15.116:0/857050992 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66f4156030 0x7f66f415ce80 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f66ec009d90 tx=0x7f66ec01c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6bb0f27fad3f0e10 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:15.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.617+0000 7f66f2ffd640 1 -- 172.21.15.116:0/857050992 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66ec0b2070 con 0x7f66f4156030 2024-04-25T00:07:15.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.617+0000 7f66f2ffd640 1 -- 172.21.15.116:0/857050992 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66ec0a5830 con 0x7f66f4156030 2024-04-25T00:07:15.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.617+0000 7f66f2ffd640 1 -- 172.21.15.116:0/857050992 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66ec0a5c00 con 0x7f66f4156030 2024-04-25T00:07:15.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.621+0000 7f66f9e6d640 1 -- 172.21.15.116:0/857050992 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66f4156030 msgr2=0x7f66f415ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:15.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.621+0000 7f66f9e6d640 1 --2- 172.21.15.116:0/857050992 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66f4156030 0x7f66f415ce80 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f66ec009d90 tx=0x7f66ec01c970 comp rx=0 tx=0).stop 2024-04-25T00:07:15.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.621+0000 7f66f9e6d640 1 -- 172.21.15.116:0/857050992 shutdown_connections 2024-04-25T00:07:15.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.621+0000 7f66f9e6d640 1 --2- 172.21.15.116:0/857050992 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66f4156030 0x7f66f415ce80 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:15.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.621+0000 7f66f9e6d640 1 --2- 172.21.15.116:0/857050992 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f66f40aa170 0x7f66f4155af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:15.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.621+0000 7f66f9e6d640 1 -- 172.21.15.116:0/857050992 >> 172.21.15.116:0/857050992 conn(0x7f66f406c720 msgr2=0x7f66f406cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:15.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.621+0000 7f66f9e6d640 1 -- 172.21.15.116:0/857050992 shutdown_connections 2024-04-25T00:07:15.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.621+0000 7f66f9e6d640 1 -- 172.21.15.116:0/857050992 wait complete. 2024-04-25T00:07:15.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.621+0000 7f66f9e6d640 1 Processor -- start 2024-04-25T00:07:15.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.621+0000 7f66f9e6d640 1 -- start start 2024-04-25T00:07:15.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.621+0000 7f66f9e6d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66f40aa170 0x7f66f4148660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:15.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.621+0000 7f66f9e6d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f66f40c4300 0x7f66f413f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:15.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.621+0000 7f66f9e6d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66f4156030 0x7f66f413f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:15.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.621+0000 7f66f9e6d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f66f40c4860 con 0x7f66f40aa170 2024-04-25T00:07:15.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.621+0000 7f66f9e6d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f66f40c46e0 con 0x7f66f4156030 2024-04-25T00:07:15.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.621+0000 7f66f9e6d640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f66f40aacd0 con 0x7f66f40c4300 2024-04-25T00:07:15.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.621+0000 7f66f37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66f40aa170 0x7f66f4148660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:15.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.621+0000 7f66f37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66f40aa170 0x7f66f4148660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57336/0 (socket says 172.21.15.116:57336) 2024-04-25T00:07:15.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.621+0000 7f66f37fe640 1 -- 172.21.15.116:0/937424297 learned_addr learned my addr 172.21.15.116:0/937424297 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:07:15.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.621+0000 7f66ebfff640 1 --2- 172.21.15.116:0/937424297 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f66f40c4300 0x7f66f413f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:15.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.621+0000 7f66f3fff640 1 --2- 172.21.15.116:0/937424297 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66f4156030 0x7f66f413f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:15.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.621+0000 7f66f37fe640 1 -- 172.21.15.116:0/937424297 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f66f40c4300 msgr2=0x7f66f413f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:15.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.621+0000 7f66f37fe640 1 --2- 172.21.15.116:0/937424297 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f66f40c4300 0x7f66f413f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:15.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.621+0000 7f66f37fe640 1 -- 172.21.15.116:0/937424297 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66f4156030 msgr2=0x7f66f413f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:15.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.621+0000 7f66f37fe640 1 --2- 172.21.15.116:0/937424297 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66f4156030 0x7f66f413f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:15.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.621+0000 7f66f37fe640 1 -- 172.21.15.116:0/937424297 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66f40aced0 con 0x7f66f40aa170 2024-04-25T00:07:15.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.621+0000 7f66ebfff640 1 --2- 172.21.15.116:0/937424297 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f66f40c4300 0x7f66f413f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:07:15.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.621+0000 7f66f37fe640 1 --2- 172.21.15.116:0/937424297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66f40aa170 0x7f66f4148660 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f66e400c420 tx=0x7f66e400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:15.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.621+0000 7f66f0ff9640 1 -- 172.21.15.116:0/937424297 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e4017070 con 0x7f66f40aa170 2024-04-25T00:07:15.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.621+0000 7f66f0ff9640 1 -- 172.21.15.116:0/937424297 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66e4005cf0 con 0x7f66f40aa170 2024-04-25T00:07:15.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.621+0000 7f66f9e6d640 1 -- 172.21.15.116:0/937424297 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66f41ecb20 con 0x7f66f40aa170 2024-04-25T00:07:15.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.621+0000 7f66f0ff9640 1 -- 172.21.15.116:0/937424297 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e40072f0 con 0x7f66f40aa170 2024-04-25T00:07:15.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.621+0000 7f66f9e6d640 1 -- 172.21.15.116:0/937424297 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66f41ec810 con 0x7f66f40aa170 2024-04-25T00:07:15.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.625+0000 7f66f9e6d640 1 -- 172.21.15.116:0/937424297 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66f40acce0 con 0x7f66f40aa170 2024-04-25T00:07:15.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.625+0000 7f66f0ff9640 1 -- 172.21.15.116:0/937424297 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f66e401a070 con 0x7f66f40aa170 2024-04-25T00:07:15.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.625+0000 7f66f0ff9640 1 --2- 172.21.15.116:0/937424297 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f66d40678f0 0x7f66d4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:15.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.625+0000 7f66f0ff9640 1 -- 172.21.15.116:0/937424297 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 7268+0+0 (secure 0 0 0) 0x7f66e4085660 con 0x7f66f40aa170 2024-04-25T00:07:15.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.625+0000 7f66ebfff640 1 --2- 172.21.15.116:0/937424297 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f66d40678f0 0x7f66d4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:15.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.625+0000 7f66ebfff640 1 --2- 172.21.15.116:0/937424297 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f66d40678f0 0x7f66d4069db0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f66f4143010 tx=0x7f66dc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:15.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.633+0000 7f66f0ff9640 1 -- 172.21.15.116:0/937424297 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f66e4055b80 con 0x7f66f40aa170 2024-04-25T00:07:15.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.897+0000 7f66f9e6d640 1 -- 172.21.15.116:0/937424297 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f66f4068a60 con 0x7f66f40aa170 2024-04-25T00:07:15.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.897+0000 7f66f0ff9640 1 -- 172.21.15.116:0/937424297 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+6502 (secure 0 0 0) 0x7f66e4055480 con 0x7f66f40aa170 2024-04-25T00:07:15.899 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:07:15.899 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5775,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/3006671671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3006671671},{"type":"v1","addr":"172.21.15.116:6833","nonce":3006671671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:06:33.449109+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5613":{"gid":5613,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":28,"addr":"172.21.15.116:6835/1233829678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1233829678},{"type":"v1","addr":"172.21.15.116:6835","nonce":1233829678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5613,"qdb_cluster":[5613]},"id":1},{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:07:03.468507+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4829":{"gid":4829,"name":"b","rank":0,"incarnation":47,"state":"up:active","state_seq":10,"addr":"172.21.15.174:6837/3583791822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3583791822},{"type":"v1","addr":"172.21.15.174:6837","nonce":3583791822}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4829,"qdb_cluster":[4829]},"id":2}]} 2024-04-25T00:07:15.899 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 51 2024-04-25T00:07:15.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.897+0000 7f66f9e6d640 1 -- 172.21.15.116:0/937424297 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f66d40678f0 msgr2=0x7f66d4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:15.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.897+0000 7f66f9e6d640 1 --2- 172.21.15.116:0/937424297 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f66d40678f0 0x7f66d4069db0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f66f4143010 tx=0x7f66dc008040 comp rx=0 tx=0).stop 2024-04-25T00:07:15.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.897+0000 7f66f9e6d640 1 -- 172.21.15.116:0/937424297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66f40aa170 msgr2=0x7f66f4148660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:15.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.897+0000 7f66f9e6d640 1 --2- 172.21.15.116:0/937424297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66f40aa170 0x7f66f4148660 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f66e400c420 tx=0x7f66e400c8e0 comp rx=0 tx=0).stop 2024-04-25T00:07:15.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.901+0000 7f66f9e6d640 1 -- 172.21.15.116:0/937424297 shutdown_connections 2024-04-25T00:07:15.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.901+0000 7f66f9e6d640 1 --2- 172.21.15.116:0/937424297 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f66d40678f0 0x7f66d4069db0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:15.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.901+0000 7f66f9e6d640 1 --2- 172.21.15.116:0/937424297 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66f4156030 0x7f66f413f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:15.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.901+0000 7f66f9e6d640 1 --2- 172.21.15.116:0/937424297 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f66f40c4300 0x7f66f413f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:15.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.901+0000 7f66f9e6d640 1 --2- 172.21.15.116:0/937424297 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66f40aa170 0x7f66f4148660 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:15.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.901+0000 7f66f9e6d640 1 -- 172.21.15.116:0/937424297 >> 172.21.15.116:0/937424297 conn(0x7f66f406c720 msgr2=0x7f66f406cb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:15.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.901+0000 7f66f9e6d640 1 -- 172.21.15.116:0/937424297 shutdown_connections 2024-04-25T00:07:15.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:15.901+0000 7f66f9e6d640 1 -- 172.21.15.116:0/937424297 wait complete. 2024-04-25T00:07:15.921 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:07:17.922 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:07:18.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.049+0000 7faa5353a640 1 Processor -- start 2024-04-25T00:07:18.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.049+0000 7faa5353a640 1 -- start start 2024-04-25T00:07:18.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.053+0000 7faa5353a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7faa4c148890 0x7faa4c072590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:18.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.053+0000 7faa5353a640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7faa4c070dd0 0x7faa4c071270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:18.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.053+0000 7faa5353a640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa4c152c10 con 0x7faa4c06e890 2024-04-25T00:07:18.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.053+0000 7faa5353a640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7faa4c151e50 con 0x7faa4c148890 2024-04-25T00:07:18.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.053+0000 7faa5353a640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7faa4c152130 con 0x7faa4c070dd0 2024-04-25T00:07:18.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.053+0000 7faa512d6640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7faa4c148890 0x7faa4c072590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:18.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.053+0000 7faa51ad7640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7faa4c06e890 0x7faa4c0717b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:56864/0 (socket says 172.21.15.116:56864) 2024-04-25T00:07:18.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.053+0000 7faa51ad7640 1 -- 172.21.15.116:0/1765187125 learned_addr learned my addr 172.21.15.116:0/1765187125 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:07:18.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.053+0000 7faa50ad5640 1 --2- 172.21.15.116:0/1765187125 >> v2:172.21.15.174:3300/0 conn(0x7faa4c070dd0 0x7faa4c071270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:18.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.053+0000 7faa4bfff640 1 -- 172.21.15.116:0/1765187125 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3632349944 0 0) 0x7faa4c152c10 con 0x7faa4c06e890 2024-04-25T00:07:18.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.053+0000 7faa4bfff640 1 -- 172.21.15.116:0/1765187125 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa34005740 con 0x7faa4c06e890 2024-04-25T00:07:18.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.053+0000 7faa512d6640 1 -- 172.21.15.116:0/1765187125 >> v2:172.21.15.174:3300/0 conn(0x7faa4c070dd0 msgr2=0x7faa4c071270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:07:18.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.053+0000 7faa512d6640 1 --2- 172.21.15.116:0/1765187125 >> v2:172.21.15.174:3300/0 conn(0x7faa4c070dd0 0x7faa4c071270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:18.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.053+0000 7faa512d6640 1 -- 172.21.15.116:0/1765187125 >> v1:172.21.15.116:6789/0 conn(0x7faa4c06e890 legacy=0x7faa4c0717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:18.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.053+0000 7faa512d6640 1 -- 172.21.15.116:0/1765187125 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa4c152410 con 0x7faa4c148890 2024-04-25T00:07:18.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.053+0000 7faa512d6640 1 --2- 172.21.15.116:0/1765187125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa4c148890 0x7faa4c072590 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7faa4c0711b0 tx=0x7faa4401c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57e7f2899a29d903 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:18.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.053+0000 7faa4bfff640 1 -- 172.21.15.116:0/1765187125 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa440b2070 con 0x7faa4c148890 2024-04-25T00:07:18.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.053+0000 7faa4bfff640 1 -- 172.21.15.116:0/1765187125 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa440a57f0 con 0x7faa4c148890 2024-04-25T00:07:18.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.053+0000 7faa4bfff640 1 -- 172.21.15.116:0/1765187125 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa440a5bc0 con 0x7faa4c148890 2024-04-25T00:07:18.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.053+0000 7faa5353a640 1 -- 172.21.15.116:0/1765187125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa4c148890 msgr2=0x7faa4c072590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:18.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.053+0000 7faa5353a640 1 --2- 172.21.15.116:0/1765187125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa4c148890 0x7faa4c072590 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7faa4c0711b0 tx=0x7faa4401c930 comp rx=0 tx=0).stop 2024-04-25T00:07:18.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.057+0000 7faa5353a640 1 -- 172.21.15.116:0/1765187125 shutdown_connections 2024-04-25T00:07:18.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.057+0000 7faa5353a640 1 --2- 172.21.15.116:0/1765187125 >> v2:172.21.15.174:3300/0 conn(0x7faa4c070dd0 0x7faa4c071270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:18.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.057+0000 7faa5353a640 1 --2- 172.21.15.116:0/1765187125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa4c148890 0x7faa4c072590 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:18.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.057+0000 7faa5353a640 1 -- 172.21.15.116:0/1765187125 >> 172.21.15.116:0/1765187125 conn(0x7faa4c0c3b00 msgr2=0x7faa4c0c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:18.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.057+0000 7faa5353a640 1 -- 172.21.15.116:0/1765187125 shutdown_connections 2024-04-25T00:07:18.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.057+0000 7faa5353a640 1 -- 172.21.15.116:0/1765187125 wait complete. 2024-04-25T00:07:18.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.057+0000 7faa5353a640 1 Processor -- start 2024-04-25T00:07:18.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.057+0000 7faa5353a640 1 -- start start 2024-04-25T00:07:18.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.057+0000 7faa5353a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa4c06e890 0x7faa4c146310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:18.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.057+0000 7faa5353a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa4c070dd0 0x7faa4c0c60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:18.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.057+0000 7faa5353a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa4c148890 0x7faa4c0c65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:18.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.057+0000 7faa5353a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7faa4c0ab920 con 0x7faa4c070dd0 2024-04-25T00:07:18.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.057+0000 7faa5353a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7faa4c0ab7a0 con 0x7faa4c06e890 2024-04-25T00:07:18.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.057+0000 7faa5353a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7faa4c0abaa0 con 0x7faa4c148890 2024-04-25T00:07:18.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.057+0000 7faa50ad5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa4c070dd0 0x7faa4c0c60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:18.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.057+0000 7faa50ad5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa4c070dd0 0x7faa4c0c60b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57346/0 (socket says 172.21.15.116:57346) 2024-04-25T00:07:18.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.057+0000 7faa50ad5640 1 -- 172.21.15.116:0/2117568254 learned_addr learned my addr 172.21.15.116:0/2117568254 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:07:18.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.057+0000 7faa512d6640 1 --2- 172.21.15.116:0/2117568254 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa4c06e890 0x7faa4c146310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:18.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.057+0000 7faa51ad7640 1 --2- 172.21.15.116:0/2117568254 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa4c148890 0x7faa4c0c65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:18.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.057+0000 7faa50ad5640 1 -- 172.21.15.116:0/2117568254 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa4c148890 msgr2=0x7faa4c0c65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:18.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.057+0000 7faa50ad5640 1 --2- 172.21.15.116:0/2117568254 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa4c148890 0x7faa4c0c65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:18.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.057+0000 7faa50ad5640 1 -- 172.21.15.116:0/2117568254 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa4c06e890 msgr2=0x7faa4c146310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:18.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.057+0000 7faa50ad5640 1 --2- 172.21.15.116:0/2117568254 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa4c06e890 0x7faa4c146310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:18.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.057+0000 7faa50ad5640 1 -- 172.21.15.116:0/2117568254 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa4c0c4d90 con 0x7faa4c070dd0 2024-04-25T00:07:18.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.057+0000 7faa50ad5640 1 --2- 172.21.15.116:0/2117568254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa4c070dd0 0x7faa4c0c60b0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7faa4000e900 tx=0x7faa4000edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:18.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.057+0000 7faa49ffb640 1 -- 172.21.15.116:0/2117568254 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa40017070 con 0x7faa4c070dd0 2024-04-25T00:07:18.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.057+0000 7faa49ffb640 1 -- 172.21.15.116:0/2117568254 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa4000ca00 con 0x7faa4c070dd0 2024-04-25T00:07:18.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.057+0000 7faa5353a640 1 -- 172.21.15.116:0/2117568254 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa4c0c5070 con 0x7faa4c070dd0 2024-04-25T00:07:18.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.057+0000 7faa49ffb640 1 -- 172.21.15.116:0/2117568254 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa4000cdf0 con 0x7faa4c070dd0 2024-04-25T00:07:18.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.057+0000 7faa5353a640 1 -- 172.21.15.116:0/2117568254 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa4c142850 con 0x7faa4c070dd0 2024-04-25T00:07:18.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.057+0000 7faa5353a640 1 -- 172.21.15.116:0/2117568254 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa4c068350 con 0x7faa4c070dd0 2024-04-25T00:07:18.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.061+0000 7faa49ffb640 1 -- 172.21.15.116:0/2117568254 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7faa4001a070 con 0x7faa4c070dd0 2024-04-25T00:07:18.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.061+0000 7faa49ffb640 1 --2- 172.21.15.116:0/2117568254 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faa380678c0 0x7faa38069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:18.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.061+0000 7faa49ffb640 1 -- 172.21.15.116:0/2117568254 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 7268+0+0 (secure 0 0 0) 0x7faa400865e0 con 0x7faa4c070dd0 2024-04-25T00:07:18.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.061+0000 7faa512d6640 1 --2- 172.21.15.116:0/2117568254 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faa380678c0 0x7faa38069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:18.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.061+0000 7faa512d6640 1 --2- 172.21.15.116:0/2117568254 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faa380678c0 0x7faa38069d80 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7faa440af000 tx=0x7faa440a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:18.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.065+0000 7faa49ffb640 1 -- 172.21.15.116:0/2117568254 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7faa40056b90 con 0x7faa4c070dd0 2024-04-25T00:07:18.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.261+0000 7faa5353a640 1 -- 172.21.15.116:0/2117568254 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faa4c06e670 con 0x7faa4c070dd0 2024-04-25T00:07:18.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.261+0000 7faa49ffb640 1 -- 172.21.15.116:0/2117568254 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+6502 (secure 0 0 0) 0x7faa40056490 con 0x7faa4c070dd0 2024-04-25T00:07:18.264 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:07:18.264 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5775,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/3006671671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3006671671},{"type":"v1","addr":"172.21.15.116:6833","nonce":3006671671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:06:33.449109+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5613":{"gid":5613,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":28,"addr":"172.21.15.116:6835/1233829678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1233829678},{"type":"v1","addr":"172.21.15.116:6835","nonce":1233829678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5613,"qdb_cluster":[5613]},"id":1},{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:07:03.468507+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4829":{"gid":4829,"name":"b","rank":0,"incarnation":47,"state":"up:active","state_seq":10,"addr":"172.21.15.174:6837/3583791822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3583791822},{"type":"v1","addr":"172.21.15.174:6837","nonce":3583791822}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4829,"qdb_cluster":[4829]},"id":2}]} 2024-04-25T00:07:18.264 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 51 2024-04-25T00:07:18.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.265+0000 7faa5353a640 1 -- 172.21.15.116:0/2117568254 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faa380678c0 msgr2=0x7faa38069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:18.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.265+0000 7faa5353a640 1 --2- 172.21.15.116:0/2117568254 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faa380678c0 0x7faa38069d80 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7faa440af000 tx=0x7faa440a7070 comp rx=0 tx=0).stop 2024-04-25T00:07:18.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.265+0000 7faa5353a640 1 -- 172.21.15.116:0/2117568254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa4c070dd0 msgr2=0x7faa4c0c60b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:18.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.265+0000 7faa5353a640 1 --2- 172.21.15.116:0/2117568254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa4c070dd0 0x7faa4c0c60b0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7faa4000e900 tx=0x7faa4000edc0 comp rx=0 tx=0).stop 2024-04-25T00:07:18.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.265+0000 7faa5353a640 1 -- 172.21.15.116:0/2117568254 shutdown_connections 2024-04-25T00:07:18.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.265+0000 7faa5353a640 1 --2- 172.21.15.116:0/2117568254 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faa380678c0 0x7faa38069d80 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:18.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.265+0000 7faa5353a640 1 --2- 172.21.15.116:0/2117568254 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa4c148890 0x7faa4c0c65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:18.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.265+0000 7faa5353a640 1 --2- 172.21.15.116:0/2117568254 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa4c070dd0 0x7faa4c0c60b0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:18.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.265+0000 7faa5353a640 1 --2- 172.21.15.116:0/2117568254 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa4c06e890 0x7faa4c146310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:18.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.265+0000 7faa5353a640 1 -- 172.21.15.116:0/2117568254 >> 172.21.15.116:0/2117568254 conn(0x7faa4c0c3b00 msgr2=0x7faa4c06d4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:18.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.265+0000 7faa5353a640 1 -- 172.21.15.116:0/2117568254 shutdown_connections 2024-04-25T00:07:18.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:18.265+0000 7faa5353a640 1 -- 172.21.15.116:0/2117568254 wait complete. 2024-04-25T00:07:18.282 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:07:20.283 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:07:20.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.413+0000 7fa807d9a640 1 Processor -- start 2024-04-25T00:07:20.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.413+0000 7fa807d9a640 1 -- start start 2024-04-25T00:07:20.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.417+0000 7fa807d9a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa8000aeb20 0x7fa8000c9310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:20.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.417+0000 7fa807d9a640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8001535d0 con 0x7fa8000c9850 2024-04-25T00:07:20.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.417+0000 7fa807d9a640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa800153850 con 0x7fa800150f80 2024-04-25T00:07:20.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.417+0000 7fa807d9a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa8000c9cb0 con 0x7fa8000aeb20 2024-04-25T00:07:20.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.417+0000 7fa805335640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fa8000c9850 0x7fa800150870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:56868/0 (socket says 172.21.15.116:56868) 2024-04-25T00:07:20.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.417+0000 7fa805335640 1 -- 172.21.15.116:0/2832328643 learned_addr learned my addr 172.21.15.116:0/2832328643 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:07:20.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.417+0000 7fa805b36640 1 --2- 172.21.15.116:0/2832328643 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa8000aeb20 0x7fa8000c9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:20.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.417+0000 7fa804b34640 1 -- 172.21.15.116:0/2832328643 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1251508192 0 0) 0x7fa7f0001120 con 0x7fa8000c9850 2024-04-25T00:07:20.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.417+0000 7fa804b34640 1 -- 172.21.15.116:0/2832328643 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7ec003530 con 0x7fa8000c9850 2024-04-25T00:07:20.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.417+0000 7fa804b34640 1 -- 172.21.15.116:0/2832328643 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2574289453 0 0) 0x7fa7ec003530 con 0x7fa8000c9850 2024-04-25T00:07:20.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.417+0000 7fa804b34640 1 -- 172.21.15.116:0/2832328643 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa8000aeb20 msgr2=0x7fa8000c9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:07:20.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.417+0000 7fa804b34640 1 --2- 172.21.15.116:0/2832328643 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa8000aeb20 0x7fa8000c9310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:20.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.417+0000 7fa804b34640 1 -- 172.21.15.116:0/2832328643 >> v1:172.21.15.174:6789/0 conn(0x7fa800150f80 legacy=0x7fa8000c5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:20.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.417+0000 7fa804b34640 1 -- 172.21.15.116:0/2832328643 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa800153b30 con 0x7fa8000c9850 2024-04-25T00:07:20.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.417+0000 7fa804b34640 1 -- 172.21.15.116:0/2832328643 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7fa7f0001850 con 0x7fa8000c9850 2024-04-25T00:07:20.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.417+0000 7fa804b34640 1 -- 172.21.15.116:0/2832328643 >> v1:172.21.15.116:6789/0 conn(0x7fa8000c9850 legacy=0x7fa800150870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:20.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.417+0000 7fa804b34640 1 --2- 172.21.15.116:0/2832328643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7ec0039f0 0x7fa7ec003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:20.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.417+0000 7fa804b34640 1 --2- 172.21.15.116:0/2832328643 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa8000aeb20 0x7fa7ec004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:20.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.417+0000 7fa804b34640 1 --2- 172.21.15.116:0/2832328643 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7ec004d50 0x7fa7ec0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:20.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.417+0000 7fa804b34640 1 -- 172.21.15.116:0/2832328643 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa7ec00bad0 con 0x7fa7ec0039f0 2024-04-25T00:07:20.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.417+0000 7fa804b34640 1 -- 172.21.15.116:0/2832328643 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa7ec00bc80 con 0x7fa7ec004d50 2024-04-25T00:07:20.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.417+0000 7fa804b34640 1 -- 172.21.15.116:0/2832328643 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa7ec00be80 con 0x7fa8000aeb20 2024-04-25T00:07:20.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.417+0000 7fa805b36640 1 --2- 172.21.15.116:0/2832328643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7ec0039f0 0x7fa7ec003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:20.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.417+0000 7fa805335640 1 --2- 172.21.15.116:0/2832328643 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa8000aeb20 0x7fa7ec004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:20.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.417+0000 7fa806337640 1 --2- 172.21.15.116:0/2832328643 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7ec004d50 0x7fa7ec0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:20.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.417+0000 7fa805b36640 1 -- 172.21.15.116:0/2832328643 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa8000aeb20 msgr2=0x7fa7ec004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:07:20.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.417+0000 7fa805b36640 1 --2- 172.21.15.116:0/2832328643 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa8000aeb20 0x7fa7ec004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:20.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.417+0000 7fa805b36640 1 -- 172.21.15.116:0/2832328643 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7ec004d50 msgr2=0x7fa7ec0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:07:20.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.417+0000 7fa805b36640 1 --2- 172.21.15.116:0/2832328643 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7ec004d50 0x7fa7ec0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:20.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.417+0000 7fa805b36640 1 -- 172.21.15.116:0/2832328643 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7ec003530 con 0x7fa7ec0039f0 2024-04-25T00:07:20.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.421+0000 7fa805b36640 1 --2- 172.21.15.116:0/2832328643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7ec0039f0 0x7fa7ec003e10 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc009ae0 tx=0x7fa7fc01c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91ad8b82ea90bdd7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:20.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.421+0000 7fa804b34640 1 -- 172.21.15.116:0/2832328643 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7fc0096b0 con 0x7fa7ec0039f0 2024-04-25T00:07:20.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.421+0000 7fa804b34640 1 -- 172.21.15.116:0/2832328643 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7fc0a6470 con 0x7fa7ec0039f0 2024-04-25T00:07:20.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.421+0000 7fa804b34640 1 -- 172.21.15.116:0/2832328643 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7fc0a6860 con 0x7fa7ec0039f0 2024-04-25T00:07:20.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.421+0000 7fa807d9a640 1 -- 172.21.15.116:0/2832328643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7ec0039f0 msgr2=0x7fa7ec003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:20.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.421+0000 7fa807d9a640 1 --2- 172.21.15.116:0/2832328643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7ec0039f0 0x7fa7ec003e10 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc009ae0 tx=0x7fa7fc01c980 comp rx=0 tx=0).stop 2024-04-25T00:07:20.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.421+0000 7fa806337640 1 -- 172.21.15.116:0/2832328643 reap_dead start 2024-04-25T00:07:20.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.421+0000 7fa807d9a640 1 -- 172.21.15.116:0/2832328643 shutdown_connections 2024-04-25T00:07:20.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.421+0000 7fa807d9a640 1 -- 172.21.15.116:0/2832328643 >> 172.21.15.116:0/2832328643 conn(0x7fa800074650 msgr2=0x7fa800074a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:20.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.421+0000 7fa807d9a640 1 -- 172.21.15.116:0/2832328643 shutdown_connections 2024-04-25T00:07:20.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.421+0000 7fa807d9a640 1 -- 172.21.15.116:0/2832328643 wait complete. 2024-04-25T00:07:20.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.421+0000 7fa807d9a640 1 Processor -- start 2024-04-25T00:07:20.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.421+0000 7fa807d9a640 1 -- start start 2024-04-25T00:07:20.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.421+0000 7fa807d9a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7ec0039f0 0x7fa80014a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:20.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.421+0000 7fa807d9a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa80014a310 0x7fa8001f00c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:20.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.421+0000 7fa807d9a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8001f0600 0x7fa80015d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:20.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.421+0000 7fa807d9a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa8000c5f10 con 0x7fa7ec0039f0 2024-04-25T00:07:20.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.421+0000 7fa807d9a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa8000c5d90 con 0x7fa8001f0600 2024-04-25T00:07:20.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.421+0000 7fa807d9a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa8000aef00 con 0x7fa80014a310 2024-04-25T00:07:20.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.425+0000 7fa805b36640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7ec0039f0 0x7fa80014a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:20.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.425+0000 7fa805b36640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7ec0039f0 0x7fa80014a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57366/0 (socket says 172.21.15.116:57366) 2024-04-25T00:07:20.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.425+0000 7fa805b36640 1 -- 172.21.15.116:0/2840271127 learned_addr learned my addr 172.21.15.116:0/2840271127 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:07:20.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.425+0000 7fa806337640 1 --2- 172.21.15.116:0/2840271127 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8001f0600 0x7fa80015d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:20.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.425+0000 7fa805335640 1 --2- 172.21.15.116:0/2840271127 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa80014a310 0x7fa8001f00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:20.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.425+0000 7fa805b36640 1 -- 172.21.15.116:0/2840271127 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa80014a310 msgr2=0x7fa8001f00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:20.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.425+0000 7fa805b36640 1 --2- 172.21.15.116:0/2840271127 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa80014a310 0x7fa8001f00c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:20.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.425+0000 7fa805b36640 1 -- 172.21.15.116:0/2840271127 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8001f0600 msgr2=0x7fa80015d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:20.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.425+0000 7fa805b36640 1 --2- 172.21.15.116:0/2840271127 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8001f0600 0x7fa80015d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:20.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.425+0000 7fa805b36640 1 -- 172.21.15.116:0/2840271127 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8001f0d60 con 0x7fa7ec0039f0 2024-04-25T00:07:20.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.425+0000 7fa805b36640 1 --2- 172.21.15.116:0/2840271127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7ec0039f0 0x7fa80014a860 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc009ae0 tx=0x7fa7fc019720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:20.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.425+0000 7fa7f67fc640 1 -- 172.21.15.116:0/2840271127 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7fc0a6d50 con 0x7fa7ec0039f0 2024-04-25T00:07:20.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.425+0000 7fa7f67fc640 1 -- 172.21.15.116:0/2840271127 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7fc0a7450 con 0x7fa7ec0039f0 2024-04-25T00:07:20.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.425+0000 7fa807d9a640 1 -- 172.21.15.116:0/2840271127 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8000ab0b0 con 0x7fa7ec0039f0 2024-04-25T00:07:20.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.425+0000 7fa7f67fc640 1 -- 172.21.15.116:0/2840271127 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7fc0a7820 con 0x7fa7ec0039f0 2024-04-25T00:07:20.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.425+0000 7fa807d9a640 1 -- 172.21.15.116:0/2840271127 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8000ab350 con 0x7fa7ec0039f0 2024-04-25T00:07:20.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.425+0000 7fa807d9a640 1 -- 172.21.15.116:0/2840271127 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa800068a60 con 0x7fa7ec0039f0 2024-04-25T00:07:20.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.425+0000 7fa7f67fc640 1 -- 172.21.15.116:0/2840271127 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa7fc0a79c0 con 0x7fa7ec0039f0 2024-04-25T00:07:20.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.425+0000 7fa7f67fc640 1 --2- 172.21.15.116:0/2840271127 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa7e40678c0 0x7fa7e4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:20.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.425+0000 7fa7f67fc640 1 -- 172.21.15.116:0/2840271127 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 7268+0+0 (secure 0 0 0) 0x7fa7fc1204a0 con 0x7fa7ec0039f0 2024-04-25T00:07:20.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.425+0000 7fa805335640 1 --2- 172.21.15.116:0/2840271127 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa7e40678c0 0x7fa7e4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:20.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.425+0000 7fa805335640 1 --2- 172.21.15.116:0/2840271127 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa7e40678c0 0x7fa7e4069d80 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fa7f0001dd0 tx=0x7fa7f0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:20.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.429+0000 7fa7f67fc640 1 -- 172.21.15.116:0/2840271127 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fa7fc0ac020 con 0x7fa7ec0039f0 2024-04-25T00:07:20.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.629+0000 7fa807d9a640 1 -- 172.21.15.116:0/2840271127 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa80014ff30 con 0x7fa7ec0039f0 2024-04-25T00:07:20.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.629+0000 7fa7f67fc640 1 -- 172.21.15.116:0/2840271127 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+6502 (secure 0 0 0) 0x7fa7fc0166c0 con 0x7fa7ec0039f0 2024-04-25T00:07:20.631 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:07:20.631 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5775,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/3006671671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3006671671},{"type":"v1","addr":"172.21.15.116:6833","nonce":3006671671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:06:33.449109+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5613":{"gid":5613,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":28,"addr":"172.21.15.116:6835/1233829678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1233829678},{"type":"v1","addr":"172.21.15.116:6835","nonce":1233829678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5613,"qdb_cluster":[5613]},"id":1},{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:07:03.468507+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4829":{"gid":4829,"name":"b","rank":0,"incarnation":47,"state":"up:active","state_seq":10,"addr":"172.21.15.174:6837/3583791822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3583791822},{"type":"v1","addr":"172.21.15.174:6837","nonce":3583791822}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4829,"qdb_cluster":[4829]},"id":2}]} 2024-04-25T00:07:20.631 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 51 2024-04-25T00:07:20.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.629+0000 7fa807d9a640 1 -- 172.21.15.116:0/2840271127 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa7e40678c0 msgr2=0x7fa7e4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:20.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.629+0000 7fa807d9a640 1 --2- 172.21.15.116:0/2840271127 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa7e40678c0 0x7fa7e4069d80 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fa7f0001dd0 tx=0x7fa7f0009040 comp rx=0 tx=0).stop 2024-04-25T00:07:20.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.633+0000 7fa807d9a640 1 -- 172.21.15.116:0/2840271127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7ec0039f0 msgr2=0x7fa80014a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:20.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.633+0000 7fa807d9a640 1 --2- 172.21.15.116:0/2840271127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7ec0039f0 0x7fa80014a860 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc009ae0 tx=0x7fa7fc019720 comp rx=0 tx=0).stop 2024-04-25T00:07:20.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.633+0000 7fa807d9a640 1 -- 172.21.15.116:0/2840271127 shutdown_connections 2024-04-25T00:07:20.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.633+0000 7fa807d9a640 1 --2- 172.21.15.116:0/2840271127 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa7e40678c0 0x7fa7e4069d80 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:20.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.633+0000 7fa807d9a640 1 --2- 172.21.15.116:0/2840271127 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8001f0600 0x7fa80015d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:20.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.633+0000 7fa807d9a640 1 --2- 172.21.15.116:0/2840271127 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa80014a310 0x7fa8001f00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:20.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.633+0000 7fa807d9a640 1 --2- 172.21.15.116:0/2840271127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa7ec0039f0 0x7fa80014a860 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:20.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.633+0000 7fa807d9a640 1 -- 172.21.15.116:0/2840271127 >> 172.21.15.116:0/2840271127 conn(0x7fa800074650 msgr2=0x7fa80014b4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:20.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.633+0000 7fa807d9a640 1 -- 172.21.15.116:0/2840271127 shutdown_connections 2024-04-25T00:07:20.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:20.633+0000 7fa807d9a640 1 -- 172.21.15.116:0/2840271127 wait complete. 2024-04-25T00:07:20.649 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:07:22.650 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:07:22.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.777+0000 7fa3c032c640 1 Processor -- start 2024-04-25T00:07:22.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.777+0000 7fa3c032c640 1 -- start start 2024-04-25T00:07:22.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.781+0000 7fa3c032c640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fa3b8150d20 0x7fa3b80c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:22.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.781+0000 7fa3c032c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa3b80c2f50 0x7fa3b81538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:22.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.781+0000 7fa3c032c640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3b80c3910 con 0x7fa3b80a94a0 2024-04-25T00:07:22.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.781+0000 7fa3c032c640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fa3b8154530 con 0x7fa3b80c2f50 2024-04-25T00:07:22.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.781+0000 7fa3c032c640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fa3b80c16e0 con 0x7fa3b8150d20 2024-04-25T00:07:22.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.781+0000 7fa3bd8c7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa3b80c2f50 0x7fa3b81538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:22.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.781+0000 7fa3bd8c7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa3b80c2f50 0x7fa3b81538e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57376/0 (socket says 172.21.15.116:57376) 2024-04-25T00:07:22.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.781+0000 7fa3bd8c7640 1 -- 172.21.15.116:0/2669151462 learned_addr learned my addr 172.21.15.116:0/2669151462 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:07:22.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.781+0000 7fa3be0c8640 1 --2- 172.21.15.116:0/2669151462 >> v2:172.21.15.174:3300/0 conn(0x7fa3b8150d20 0x7fa3b80c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:22.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.781+0000 7fa3bd8c7640 1 -- 172.21.15.116:0/2669151462 >> v2:172.21.15.174:3300/0 conn(0x7fa3b8150d20 msgr2=0x7fa3b80c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:07:22.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.781+0000 7fa3bd8c7640 1 --2- 172.21.15.116:0/2669151462 >> v2:172.21.15.174:3300/0 conn(0x7fa3b8150d20 0x7fa3b80c2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:22.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.781+0000 7fa3bd8c7640 1 -- 172.21.15.116:0/2669151462 >> v1:172.21.15.116:6789/0 conn(0x7fa3b80a94a0 legacy=0x7fa3b8153e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:22.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.781+0000 7fa3bd8c7640 1 -- 172.21.15.116:0/2669151462 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3b80c1400 con 0x7fa3b80c2f50 2024-04-25T00:07:22.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.781+0000 7fa3bd0c6640 1 -- 172.21.15.116:0/2669151462 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2243313924 0 0) 0x7fa3b80c3910 con 0x7fa3b80a94a0 2024-04-25T00:07:22.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.781+0000 7fa3bd8c7640 1 --2- 172.21.15.116:0/2669151462 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3b80c2f50 0x7fa3b81538e0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fa3b8073550 tx=0x7fa3b401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9aabfceeb827d449 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:22.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.781+0000 7fa3bd0c6640 1 -- 172.21.15.116:0/2669151462 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3b40ab430 con 0x7fa3b80c2f50 2024-04-25T00:07:22.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.781+0000 7fa3bd0c6640 1 -- 172.21.15.116:0/2669151462 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3b40b0070 con 0x7fa3b80c2f50 2024-04-25T00:07:22.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.781+0000 7fa3bd0c6640 1 -- 172.21.15.116:0/2669151462 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3b40ab7f0 con 0x7fa3b80c2f50 2024-04-25T00:07:22.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.785+0000 7fa3c032c640 1 -- 172.21.15.116:0/2669151462 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3b80c2f50 msgr2=0x7fa3b81538e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:22.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.785+0000 7fa3c032c640 1 --2- 172.21.15.116:0/2669151462 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3b80c2f50 0x7fa3b81538e0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fa3b8073550 tx=0x7fa3b401c520 comp rx=0 tx=0).stop 2024-04-25T00:07:22.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.785+0000 7fa3c032c640 1 -- 172.21.15.116:0/2669151462 shutdown_connections 2024-04-25T00:07:22.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.785+0000 7fa3c032c640 1 --2- 172.21.15.116:0/2669151462 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3b80c2f50 0x7fa3b81538e0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:22.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.785+0000 7fa3c032c640 1 --2- 172.21.15.116:0/2669151462 >> v2:172.21.15.174:3300/0 conn(0x7fa3b8150d20 0x7fa3b80c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:22.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.785+0000 7fa3c032c640 1 -- 172.21.15.116:0/2669151462 >> 172.21.15.116:0/2669151462 conn(0x7fa3b80c3b00 msgr2=0x7fa3b80c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:22.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.785+0000 7fa3c032c640 1 -- 172.21.15.116:0/2669151462 shutdown_connections 2024-04-25T00:07:22.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.785+0000 7fa3c032c640 1 -- 172.21.15.116:0/2669151462 wait complete. 2024-04-25T00:07:22.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.785+0000 7fa3c032c640 1 Processor -- start 2024-04-25T00:07:22.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.785+0000 7fa3c032c640 1 -- start start 2024-04-25T00:07:22.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.785+0000 7fa3c032c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3b80a94a0 0x7fa3b813e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:22.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.785+0000 7fa3c032c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3b80c2f50 0x7fa3b814e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:22.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.785+0000 7fa3c032c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3b8150d20 0x7fa3b814e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:22.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.785+0000 7fa3c032c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa3b81608b0 con 0x7fa3b8150d20 2024-04-25T00:07:22.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.785+0000 7fa3c032c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa3b8160730 con 0x7fa3b80c2f50 2024-04-25T00:07:22.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.785+0000 7fa3c032c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa3b8160a30 con 0x7fa3b80a94a0 2024-04-25T00:07:22.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.785+0000 7fa3be8c9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3b8150d20 0x7fa3b814e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:22.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.785+0000 7fa3be8c9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3b8150d20 0x7fa3b814e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57384/0 (socket says 172.21.15.116:57384) 2024-04-25T00:07:22.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.785+0000 7fa3be8c9640 1 -- 172.21.15.116:0/306267113 learned_addr learned my addr 172.21.15.116:0/306267113 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:07:22.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.785+0000 7fa3be0c8640 1 --2- 172.21.15.116:0/306267113 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3b80a94a0 0x7fa3b813e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:22.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.785+0000 7fa3bd8c7640 1 --2- 172.21.15.116:0/306267113 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3b80c2f50 0x7fa3b814e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:22.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.785+0000 7fa3be8c9640 1 -- 172.21.15.116:0/306267113 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3b80a94a0 msgr2=0x7fa3b813e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:22.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.785+0000 7fa3be8c9640 1 --2- 172.21.15.116:0/306267113 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3b80a94a0 0x7fa3b813e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:22.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.785+0000 7fa3be8c9640 1 -- 172.21.15.116:0/306267113 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3b80c2f50 msgr2=0x7fa3b814e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:22.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.785+0000 7fa3be8c9640 1 --2- 172.21.15.116:0/306267113 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3b80c2f50 0x7fa3b814e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:22.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.785+0000 7fa3be8c9640 1 -- 172.21.15.116:0/306267113 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3b8141290 con 0x7fa3b8150d20 2024-04-25T00:07:22.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.785+0000 7fa3bd8c7640 1 --2- 172.21.15.116:0/306267113 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3b80c2f50 0x7fa3b814e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:07:22.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.785+0000 7fa3be0c8640 1 --2- 172.21.15.116:0/306267113 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3b80a94a0 0x7fa3b813e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:07:22.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.785+0000 7fa3be8c9640 1 --2- 172.21.15.116:0/306267113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3b8150d20 0x7fa3b814e960 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fa3b0001570 tx=0x7fa3b000cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:22.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.785+0000 7fa3aeffd640 1 -- 172.21.15.116:0/306267113 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3b0017070 con 0x7fa3b8150d20 2024-04-25T00:07:22.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.785+0000 7fa3aeffd640 1 -- 172.21.15.116:0/306267113 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3b0003db0 con 0x7fa3b8150d20 2024-04-25T00:07:22.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.785+0000 7fa3c032c640 1 -- 172.21.15.116:0/306267113 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3b8142b60 con 0x7fa3b8150d20 2024-04-25T00:07:22.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.785+0000 7fa3aeffd640 1 -- 172.21.15.116:0/306267113 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3b00052b0 con 0x7fa3b8150d20 2024-04-25T00:07:22.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.785+0000 7fa3c032c640 1 -- 172.21.15.116:0/306267113 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3b81ecf00 con 0x7fa3b8150d20 2024-04-25T00:07:22.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.789+0000 7fa3c032c640 1 -- 172.21.15.116:0/306267113 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3b8068350 con 0x7fa3b8150d20 2024-04-25T00:07:22.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.789+0000 7fa3aeffd640 1 -- 172.21.15.116:0/306267113 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa3b001a070 con 0x7fa3b8150d20 2024-04-25T00:07:22.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.789+0000 7fa3aeffd640 1 --2- 172.21.15.116:0/306267113 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa39c0678f0 0x7fa39c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:22.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.789+0000 7fa3aeffd640 1 -- 172.21.15.116:0/306267113 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 7268+0+0 (secure 0 0 0) 0x7fa3b0085940 con 0x7fa3b8150d20 2024-04-25T00:07:22.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.789+0000 7fa3be0c8640 1 --2- 172.21.15.116:0/306267113 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa39c0678f0 0x7fa39c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:22.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.789+0000 7fa3be0c8640 1 --2- 172.21.15.116:0/306267113 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa39c0678f0 0x7fa39c069db0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fa3b806e030 tx=0x7fa3a8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:22.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.793+0000 7fa3aeffd640 1 -- 172.21.15.116:0/306267113 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fa3b0055e60 con 0x7fa3b8150d20 2024-04-25T00:07:22.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.989+0000 7fa3c032c640 1 -- 172.21.15.116:0/306267113 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa3b806d7b0 con 0x7fa3b8150d20 2024-04-25T00:07:22.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.989+0000 7fa3aeffd640 1 -- 172.21.15.116:0/306267113 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+6502 (secure 0 0 0) 0x7fa3b0055760 con 0x7fa3b8150d20 2024-04-25T00:07:22.991 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:07:22.991 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5775,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/3006671671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3006671671},{"type":"v1","addr":"172.21.15.116:6833","nonce":3006671671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:06:33.449109+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5613":{"gid":5613,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":28,"addr":"172.21.15.116:6835/1233829678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1233829678},{"type":"v1","addr":"172.21.15.116:6835","nonce":1233829678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5613,"qdb_cluster":[5613]},"id":1},{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:07:03.468507+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4829":{"gid":4829,"name":"b","rank":0,"incarnation":47,"state":"up:active","state_seq":10,"addr":"172.21.15.174:6837/3583791822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3583791822},{"type":"v1","addr":"172.21.15.174:6837","nonce":3583791822}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4829,"qdb_cluster":[4829]},"id":2}]} 2024-04-25T00:07:22.991 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 51 2024-04-25T00:07:22.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.989+0000 7fa3c032c640 1 -- 172.21.15.116:0/306267113 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa39c0678f0 msgr2=0x7fa39c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:22.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.989+0000 7fa3c032c640 1 --2- 172.21.15.116:0/306267113 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa39c0678f0 0x7fa39c069db0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fa3b806e030 tx=0x7fa3a8008040 comp rx=0 tx=0).stop 2024-04-25T00:07:22.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.989+0000 7fa3c032c640 1 -- 172.21.15.116:0/306267113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3b8150d20 msgr2=0x7fa3b814e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:22.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.989+0000 7fa3c032c640 1 --2- 172.21.15.116:0/306267113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3b8150d20 0x7fa3b814e960 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fa3b0001570 tx=0x7fa3b000cd20 comp rx=0 tx=0).stop 2024-04-25T00:07:22.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.989+0000 7fa3c032c640 1 -- 172.21.15.116:0/306267113 shutdown_connections 2024-04-25T00:07:22.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.989+0000 7fa3c032c640 1 --2- 172.21.15.116:0/306267113 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa39c0678f0 0x7fa39c069db0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:22.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.989+0000 7fa3c032c640 1 --2- 172.21.15.116:0/306267113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3b8150d20 0x7fa3b814e960 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:22.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.989+0000 7fa3c032c640 1 --2- 172.21.15.116:0/306267113 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3b80c2f50 0x7fa3b814e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:22.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.989+0000 7fa3c032c640 1 --2- 172.21.15.116:0/306267113 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3b80a94a0 0x7fa3b813e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:22.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.989+0000 7fa3c032c640 1 -- 172.21.15.116:0/306267113 >> 172.21.15.116:0/306267113 conn(0x7fa3b80c3b00 msgr2=0x7fa3b8146d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:22.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.993+0000 7fa3c032c640 1 -- 172.21.15.116:0/306267113 shutdown_connections 2024-04-25T00:07:22.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:22.993+0000 7fa3c032c640 1 -- 172.21.15.116:0/306267113 wait complete. 2024-04-25T00:07:23.011 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:07:25.012 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:07:25.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.137+0000 7f813a707640 1 Processor -- start 2024-04-25T00:07:25.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.137+0000 7f813a707640 1 -- start start 2024-04-25T00:07:25.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.141+0000 7f813a707640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f813414b7a0 0x7f8134152210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:25.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.141+0000 7f813a707640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f813406e270 0x7f81340c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:25.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.141+0000 7f813a707640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8134152750 con 0x7f81340c2f50 2024-04-25T00:07:25.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.141+0000 7f813a707640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f813406cd60 con 0x7f813406e270 2024-04-25T00:07:25.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.141+0000 7f813a707640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f813406d040 con 0x7f813414b7a0 2024-04-25T00:07:25.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.141+0000 7f81337fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f813406e270 0x7f81340c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:25.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.141+0000 7f8138ca4640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f81340c2f50 0x7f8134070e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57148/0 (socket says 172.21.15.116:57148) 2024-04-25T00:07:25.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.141+0000 7f8138ca4640 1 -- 172.21.15.116:0/2927608333 learned_addr learned my addr 172.21.15.116:0/2927608333 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:07:25.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.141+0000 7f8133fff640 1 --2- 172.21.15.116:0/2927608333 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f813414b7a0 0x7f8134152210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:25.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.141+0000 7f81337fe640 1 -- 172.21.15.116:0/2927608333 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f813414b7a0 msgr2=0x7f8134152210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:07:25.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.141+0000 7f81337fe640 1 --2- 172.21.15.116:0/2927608333 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f813414b7a0 0x7f8134152210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:25.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.141+0000 7f81337fe640 1 -- 172.21.15.116:0/2927608333 >> v1:172.21.15.116:6789/0 conn(0x7f81340c2f50 legacy=0x7f8134070e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:25.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.141+0000 7f81337fe640 1 -- 172.21.15.116:0/2927608333 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f813406d320 con 0x7f813406e270 2024-04-25T00:07:25.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.141+0000 7f81337fe640 1 --2- 172.21.15.116:0/2927608333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f813406e270 0x7f81340c2a10 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f81240097c0 tx=0x7f812401d490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de226d23d3d06cc0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:25.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.141+0000 7f8132ffd640 1 -- 172.21.15.116:0/2927608333 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81240b1070 con 0x7f813406e270 2024-04-25T00:07:25.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.141+0000 7f8132ffd640 1 -- 172.21.15.116:0/2927608333 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81240ac600 con 0x7f813406e270 2024-04-25T00:07:25.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.141+0000 7f8132ffd640 1 -- 172.21.15.116:0/2927608333 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81240ac9d0 con 0x7f813406e270 2024-04-25T00:07:25.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.145+0000 7f813a707640 1 -- 172.21.15.116:0/2927608333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f813406e270 msgr2=0x7f81340c2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:25.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.145+0000 7f813a707640 1 --2- 172.21.15.116:0/2927608333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f813406e270 0x7f81340c2a10 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f81240097c0 tx=0x7f812401d490 comp rx=0 tx=0).stop 2024-04-25T00:07:25.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.145+0000 7f813a707640 1 -- 172.21.15.116:0/2927608333 shutdown_connections 2024-04-25T00:07:25.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.145+0000 7f813a707640 1 --2- 172.21.15.116:0/2927608333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f813406e270 0x7f81340c2a10 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:25.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.145+0000 7f813a707640 1 --2- 172.21.15.116:0/2927608333 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f813414b7a0 0x7f8134152210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:25.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.145+0000 7f813a707640 1 -- 172.21.15.116:0/2927608333 >> 172.21.15.116:0/2927608333 conn(0x7f81340c3b00 msgr2=0x7f81340c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:25.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.145+0000 7f813a707640 1 -- 172.21.15.116:0/2927608333 shutdown_connections 2024-04-25T00:07:25.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.145+0000 7f813a707640 1 -- 172.21.15.116:0/2927608333 wait complete. 2024-04-25T00:07:25.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.145+0000 7f813a707640 1 Processor -- start 2024-04-25T00:07:25.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.145+0000 7f813a707640 1 -- start start 2024-04-25T00:07:25.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.145+0000 7f813a707640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f813406e270 0x7f813415a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:25.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.145+0000 7f813a707640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f81340c2f50 0x7f81340aa9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:25.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.145+0000 7f813a707640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f813414b7a0 0x7f8134154280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:25.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.145+0000 7f813a707640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f813415eab0 con 0x7f813414b7a0 2024-04-25T00:07:25.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.145+0000 7f813a707640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f813415e930 con 0x7f813406e270 2024-04-25T00:07:25.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.145+0000 7f813a707640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f813415ec30 con 0x7f81340c2f50 2024-04-25T00:07:25.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.145+0000 7f8138ca4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f813414b7a0 0x7f8134154280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:25.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.145+0000 7f8138ca4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f813414b7a0 0x7f8134154280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51662/0 (socket says 172.21.15.116:51662) 2024-04-25T00:07:25.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.145+0000 7f8138ca4640 1 -- 172.21.15.116:0/3114953300 learned_addr learned my addr 172.21.15.116:0/3114953300 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:07:25.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.145+0000 7f81337fe640 1 --2- 172.21.15.116:0/3114953300 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f81340c2f50 0x7f81340aa9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:25.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.145+0000 7f8133fff640 1 --2- 172.21.15.116:0/3114953300 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f813406e270 0x7f813415a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:25.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.145+0000 7f8138ca4640 1 -- 172.21.15.116:0/3114953300 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f81340c2f50 msgr2=0x7f81340aa9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:25.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.145+0000 7f8138ca4640 1 --2- 172.21.15.116:0/3114953300 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f81340c2f50 0x7f81340aa9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:25.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.145+0000 7f8138ca4640 1 -- 172.21.15.116:0/3114953300 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f813406e270 msgr2=0x7f813415a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:25.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.145+0000 7f8138ca4640 1 --2- 172.21.15.116:0/3114953300 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f813406e270 0x7f813415a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:25.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.145+0000 7f8138ca4640 1 -- 172.21.15.116:0/3114953300 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81341eeb60 con 0x7f813414b7a0 2024-04-25T00:07:25.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.145+0000 7f8133fff640 1 --2- 172.21.15.116:0/3114953300 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f813406e270 0x7f813415a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:07:25.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.145+0000 7f8138ca4640 1 --2- 172.21.15.116:0/3114953300 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f813414b7a0 0x7f8134154280 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f812c001a90 tx=0x7f812c001f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:25.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.145+0000 7f8130ff9640 1 -- 172.21.15.116:0/3114953300 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f812c018070 con 0x7f813414b7a0 2024-04-25T00:07:25.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.145+0000 7f8130ff9640 1 -- 172.21.15.116:0/3114953300 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f812c002080 con 0x7f813414b7a0 2024-04-25T00:07:25.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.145+0000 7f8130ff9640 1 -- 172.21.15.116:0/3114953300 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f812c00d8c0 con 0x7f813414b7a0 2024-04-25T00:07:25.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.145+0000 7f813a707640 1 -- 172.21.15.116:0/3114953300 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81341eee40 con 0x7f813414b7a0 2024-04-25T00:07:25.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.145+0000 7f813a707640 1 -- 172.21.15.116:0/3114953300 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81341ef400 con 0x7f813414b7a0 2024-04-25T00:07:25.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.149+0000 7f813a707640 1 -- 172.21.15.116:0/3114953300 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8134068350 con 0x7f813414b7a0 2024-04-25T00:07:25.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.149+0000 7f8130ff9640 1 -- 172.21.15.116:0/3114953300 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f812c01b070 con 0x7f813414b7a0 2024-04-25T00:07:25.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.149+0000 7f8130ff9640 1 --2- 172.21.15.116:0/3114953300 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f81200678c0 0x7f8120069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:25.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.149+0000 7f8130ff9640 1 -- 172.21.15.116:0/3114953300 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 7268+0+0 (secure 0 0 0) 0x7f812c0857e0 con 0x7f813414b7a0 2024-04-25T00:07:25.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.149+0000 7f8133fff640 1 --2- 172.21.15.116:0/3114953300 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f81200678c0 0x7f8120069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:25.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.149+0000 7f8133fff640 1 --2- 172.21.15.116:0/3114953300 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f81200678c0 0x7f8120069d80 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f8128009840 tx=0x7f8128008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:25.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.153+0000 7f8130ff9640 1 -- 172.21.15.116:0/3114953300 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f8134152750 con 0x7f813414b7a0 2024-04-25T00:07:25.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.349+0000 7f813a707640 1 -- 172.21.15.116:0/3114953300 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f813406df70 con 0x7f813414b7a0 2024-04-25T00:07:25.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.349+0000 7f8130ff9640 1 -- 172.21.15.116:0/3114953300 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+6502 (secure 0 0 0) 0x7f812c055770 con 0x7f813414b7a0 2024-04-25T00:07:25.352 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:07:25.352 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5775,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/3006671671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3006671671},{"type":"v1","addr":"172.21.15.116:6833","nonce":3006671671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:06:33.449109+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5613":{"gid":5613,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":28,"addr":"172.21.15.116:6835/1233829678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1233829678},{"type":"v1","addr":"172.21.15.116:6835","nonce":1233829678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5613,"qdb_cluster":[5613]},"id":1},{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:07:03.468507+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4829":{"gid":4829,"name":"b","rank":0,"incarnation":47,"state":"up:active","state_seq":10,"addr":"172.21.15.174:6837/3583791822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3583791822},{"type":"v1","addr":"172.21.15.174:6837","nonce":3583791822}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4829,"qdb_cluster":[4829]},"id":2}]} 2024-04-25T00:07:25.352 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 51 2024-04-25T00:07:25.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.353+0000 7f813a707640 1 -- 172.21.15.116:0/3114953300 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f81200678c0 msgr2=0x7f8120069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:25.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.353+0000 7f813a707640 1 --2- 172.21.15.116:0/3114953300 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f81200678c0 0x7f8120069d80 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f8128009840 tx=0x7f8128008040 comp rx=0 tx=0).stop 2024-04-25T00:07:25.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.353+0000 7f813a707640 1 -- 172.21.15.116:0/3114953300 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f813414b7a0 msgr2=0x7f8134154280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:25.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.353+0000 7f813a707640 1 --2- 172.21.15.116:0/3114953300 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f813414b7a0 0x7f8134154280 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f812c001a90 tx=0x7f812c001f50 comp rx=0 tx=0).stop 2024-04-25T00:07:25.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.353+0000 7f813a707640 1 -- 172.21.15.116:0/3114953300 shutdown_connections 2024-04-25T00:07:25.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.353+0000 7f813a707640 1 --2- 172.21.15.116:0/3114953300 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f81200678c0 0x7f8120069d80 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:25.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.353+0000 7f813a707640 1 --2- 172.21.15.116:0/3114953300 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f813414b7a0 0x7f8134154280 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:25.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.353+0000 7f813a707640 1 --2- 172.21.15.116:0/3114953300 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f81340c2f50 0x7f81340aa9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:25.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.353+0000 7f813a707640 1 --2- 172.21.15.116:0/3114953300 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f813406e270 0x7f813415a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:25.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.353+0000 7f813a707640 1 -- 172.21.15.116:0/3114953300 >> 172.21.15.116:0/3114953300 conn(0x7f81340c3b00 msgr2=0x7f81341487a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:25.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.353+0000 7f813a707640 1 -- 172.21.15.116:0/3114953300 shutdown_connections 2024-04-25T00:07:25.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:25.353+0000 7f813a707640 1 -- 172.21.15.116:0/3114953300 wait complete. 2024-04-25T00:07:25.370 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:07:27.372 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:07:27.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.501+0000 7fc86a66f640 1 Processor -- start 2024-04-25T00:07:27.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.501+0000 7fc86a66f640 1 -- start start 2024-04-25T00:07:27.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.505+0000 7fc86a66f640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fc864155870 0x7fc86415d500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:27.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.505+0000 7fc86a66f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc864155e50 con 0x7fc8640c2d10 2024-04-25T00:07:27.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.505+0000 7fc86a66f640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8640c3780 con 0x7fc86414f300 2024-04-25T00:07:27.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.505+0000 7fc86a66f640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fc8640c3a60 con 0x7fc864155870 2024-04-25T00:07:27.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.505+0000 7fc8637fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fc8640c2d10 0x7fc864155160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57162/0 (socket says 172.21.15.116:57162) 2024-04-25T00:07:27.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.505+0000 7fc8637fe640 1 -- 172.21.15.116:0/2197345159 learned_addr learned my addr 172.21.15.116:0/2197345159 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:07:27.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.505+0000 7fc868c0c640 1 --2- 172.21.15.116:0/2197345159 >> v2:172.21.15.174:3300/0 conn(0x7fc864155870 0x7fc86415d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:27.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.505+0000 7fc862ffd640 1 -- 172.21.15.116:0/2197345159 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4104326400 0 0) 0x7fc864155e50 con 0x7fc8640c2d10 2024-04-25T00:07:27.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.505+0000 7fc862ffd640 1 -- 172.21.15.116:0/2197345159 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc84c003450 con 0x7fc8640c2d10 2024-04-25T00:07:27.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.505+0000 7fc862ffd640 1 -- 172.21.15.116:0/2197345159 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1645928955 0 0) 0x7fc84c003450 con 0x7fc8640c2d10 2024-04-25T00:07:27.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.505+0000 7fc862ffd640 1 -- 172.21.15.116:0/2197345159 >> v2:172.21.15.174:3300/0 conn(0x7fc864155870 msgr2=0x7fc86415d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:07:27.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.505+0000 7fc862ffd640 1 --2- 172.21.15.116:0/2197345159 >> v2:172.21.15.174:3300/0 conn(0x7fc864155870 0x7fc86415d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:27.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.505+0000 7fc862ffd640 1 -- 172.21.15.116:0/2197345159 >> v1:172.21.15.174:6789/0 conn(0x7fc86414f300 legacy=0x7fc86406c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:27.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.505+0000 7fc862ffd640 1 -- 172.21.15.116:0/2197345159 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc86406bcc0 con 0x7fc8640c2d10 2024-04-25T00:07:27.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.505+0000 7fc862ffd640 1 -- 172.21.15.116:0/2197345159 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7fc858001680 con 0x7fc8640c2d10 2024-04-25T00:07:27.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.505+0000 7fc862ffd640 1 -- 172.21.15.116:0/2197345159 >> v1:172.21.15.116:6789/0 conn(0x7fc8640c2d10 legacy=0x7fc864155160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:27.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.505+0000 7fc862ffd640 1 --2- 172.21.15.116:0/2197345159 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc84c003ab0 0x7fc84c003ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:27.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.505+0000 7fc862ffd640 1 --2- 172.21.15.116:0/2197345159 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc84c008860 0x7fc84c00acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:27.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.505+0000 7fc862ffd640 1 --2- 172.21.15.116:0/2197345159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc84c00b570 0x7fc84c00da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:27.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.505+0000 7fc862ffd640 1 -- 172.21.15.116:0/2197345159 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc8640c3a60 con 0x7fc84c00b570 2024-04-25T00:07:27.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.505+0000 7fc862ffd640 1 -- 172.21.15.116:0/2197345159 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc84c00e310 con 0x7fc84c008860 2024-04-25T00:07:27.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.505+0000 7fc862ffd640 1 -- 172.21.15.116:0/2197345159 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fc84c00e510 con 0x7fc84c003ab0 2024-04-25T00:07:27.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.505+0000 7fc868c0c640 1 --2- 172.21.15.116:0/2197345159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc84c00b570 0x7fc84c00da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:27.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.505+0000 7fc863fff640 1 --2- 172.21.15.116:0/2197345159 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc84c003ab0 0x7fc84c003ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:27.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.505+0000 7fc868c0c640 1 -- 172.21.15.116:0/2197345159 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc84c003ab0 msgr2=0x7fc84c003ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:07:27.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.505+0000 7fc868c0c640 1 --2- 172.21.15.116:0/2197345159 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc84c003ab0 0x7fc84c003ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:27.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.505+0000 7fc868c0c640 1 -- 172.21.15.116:0/2197345159 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc84c008860 msgr2=0x7fc84c00acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:07:27.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.505+0000 7fc868c0c640 1 --2- 172.21.15.116:0/2197345159 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc84c008860 0x7fc84c00acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:27.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.505+0000 7fc868c0c640 1 -- 172.21.15.116:0/2197345159 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc84c003450 con 0x7fc84c00b570 2024-04-25T00:07:27.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.509+0000 7fc868c0c640 1 -- 172.21.15.116:0/2197345159 reap_dead start 2024-04-25T00:07:27.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.509+0000 7fc868c0c640 1 --2- 172.21.15.116:0/2197345159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc84c00b570 0x7fc84c00da30 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fc85c009fc0 tx=0x7fc85c01ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b4ad5560ecf0748 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:27.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.509+0000 7fc862ffd640 1 -- 172.21.15.116:0/2197345159 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc85c009c10 con 0x7fc84c00b570 2024-04-25T00:07:27.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.509+0000 7fc862ffd640 1 -- 172.21.15.116:0/2197345159 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc85c0a5cd0 con 0x7fc84c00b570 2024-04-25T00:07:27.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.509+0000 7fc862ffd640 1 -- 172.21.15.116:0/2197345159 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc85c003be0 con 0x7fc84c00b570 2024-04-25T00:07:27.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.509+0000 7fc86a66f640 1 -- 172.21.15.116:0/2197345159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc84c00b570 msgr2=0x7fc84c00da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:27.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.509+0000 7fc86a66f640 1 --2- 172.21.15.116:0/2197345159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc84c00b570 0x7fc84c00da30 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fc85c009fc0 tx=0x7fc85c01ccc0 comp rx=0 tx=0).stop 2024-04-25T00:07:27.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.509+0000 7fc86a66f640 1 -- 172.21.15.116:0/2197345159 shutdown_connections 2024-04-25T00:07:27.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.509+0000 7fc86a66f640 1 --2- 172.21.15.116:0/2197345159 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc84c00b570 0x7fc84c00da30 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:27.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.509+0000 7fc86a66f640 1 -- 172.21.15.116:0/2197345159 >> 172.21.15.116:0/2197345159 conn(0x7fc8640c4200 msgr2=0x7fc8640c4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:27.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.509+0000 7fc86a66f640 1 -- 172.21.15.116:0/2197345159 shutdown_connections 2024-04-25T00:07:27.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.509+0000 7fc86a66f640 1 -- 172.21.15.116:0/2197345159 wait complete. 2024-04-25T00:07:27.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.509+0000 7fc86a66f640 1 Processor -- start 2024-04-25T00:07:27.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.509+0000 7fc86a66f640 1 -- start start 2024-04-25T00:07:27.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.513+0000 7fc86a66f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc84c00b570 0x7fc8641e1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:27.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.513+0000 7fc86a66f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8641ee600 0x7fc8640a8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:27.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.513+0000 7fc86a66f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc8640c62c0 0x7fc86414a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:27.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.513+0000 7fc86a66f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc8640a7390 con 0x7fc8641ee600 2024-04-25T00:07:27.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.513+0000 7fc86a66f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc8640c3d40 con 0x7fc8640c62c0 2024-04-25T00:07:27.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.513+0000 7fc86a66f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fc8640a7510 con 0x7fc84c00b570 2024-04-25T00:07:27.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.513+0000 7fc8637fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8641ee600 0x7fc8640a8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:27.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.513+0000 7fc8637fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8641ee600 0x7fc8640a8c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51680/0 (socket says 172.21.15.116:51680) 2024-04-25T00:07:27.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.513+0000 7fc8637fe640 1 -- 172.21.15.116:0/2853320454 learned_addr learned my addr 172.21.15.116:0/2853320454 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:07:27.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.513+0000 7fc863fff640 1 --2- 172.21.15.116:0/2853320454 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc84c00b570 0x7fc8641e1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:27.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.513+0000 7fc868c0c640 1 --2- 172.21.15.116:0/2853320454 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc8640c62c0 0x7fc86414a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:27.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.513+0000 7fc8637fe640 1 -- 172.21.15.116:0/2853320454 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc84c00b570 msgr2=0x7fc8641e1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:27.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.513+0000 7fc8637fe640 1 --2- 172.21.15.116:0/2853320454 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc84c00b570 0x7fc8641e1ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:27.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.513+0000 7fc8637fe640 1 -- 172.21.15.116:0/2853320454 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc8640c62c0 msgr2=0x7fc86414a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:27.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.513+0000 7fc8637fe640 1 --2- 172.21.15.116:0/2853320454 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc8640c62c0 0x7fc86414a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:27.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.513+0000 7fc8637fe640 1 -- 172.21.15.116:0/2853320454 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8640a98c0 con 0x7fc8641ee600 2024-04-25T00:07:27.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.513+0000 7fc8637fe640 1 --2- 172.21.15.116:0/2853320454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8641ee600 0x7fc8640a8c00 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fc85800f3f0 tx=0x7fc85800f8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:27.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.513+0000 7fc860ff9640 1 -- 172.21.15.116:0/2853320454 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc858007d00 con 0x7fc8641ee600 2024-04-25T00:07:27.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.513+0000 7fc860ff9640 1 -- 172.21.15.116:0/2853320454 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc85800a0d0 con 0x7fc8641ee600 2024-04-25T00:07:27.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.513+0000 7fc86a66f640 1 -- 172.21.15.116:0/2853320454 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8640a9ba0 con 0x7fc8641ee600 2024-04-25T00:07:27.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.513+0000 7fc860ff9640 1 -- 172.21.15.116:0/2853320454 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc85800a4a0 con 0x7fc8641ee600 2024-04-25T00:07:27.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.513+0000 7fc86a66f640 1 -- 172.21.15.116:0/2853320454 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8640aa160 con 0x7fc8641ee600 2024-04-25T00:07:27.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.513+0000 7fc86a66f640 1 -- 172.21.15.116:0/2853320454 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc864068a60 con 0x7fc8641ee600 2024-04-25T00:07:27.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.517+0000 7fc860ff9640 1 -- 172.21.15.116:0/2853320454 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc858014070 con 0x7fc8641ee600 2024-04-25T00:07:27.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.517+0000 7fc860ff9640 1 --2- 172.21.15.116:0/2853320454 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc8500678c0 0x7fc850069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:27.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.517+0000 7fc860ff9640 1 -- 172.21.15.116:0/2853320454 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 7268+0+0 (secure 0 0 0) 0x7fc858089150 con 0x7fc8641ee600 2024-04-25T00:07:27.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.517+0000 7fc863fff640 1 --2- 172.21.15.116:0/2853320454 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc8500678c0 0x7fc850069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:27.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.517+0000 7fc863fff640 1 --2- 172.21.15.116:0/2853320454 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc8500678c0 0x7fc850069d80 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fc854001660 tx=0x7fc854009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:27.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.521+0000 7fc860ff9640 1 -- 172.21.15.116:0/2853320454 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fc858059070 con 0x7fc8641ee600 2024-04-25T00:07:27.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.713+0000 7fc86a66f640 1 -- 172.21.15.116:0/2853320454 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc86414e200 con 0x7fc8641ee600 2024-04-25T00:07:27.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.717+0000 7fc860ff9640 1 -- 172.21.15.116:0/2853320454 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+6502 (secure 0 0 0) 0x7fc85805d370 con 0x7fc8641ee600 2024-04-25T00:07:27.718 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:07:27.718 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5775,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/3006671671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3006671671},{"type":"v1","addr":"172.21.15.116:6833","nonce":3006671671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:06:33.449109+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5613":{"gid":5613,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":28,"addr":"172.21.15.116:6835/1233829678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1233829678},{"type":"v1","addr":"172.21.15.116:6835","nonce":1233829678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5613,"qdb_cluster":[5613]},"id":1},{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:07:03.468507+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4829":{"gid":4829,"name":"b","rank":0,"incarnation":47,"state":"up:active","state_seq":10,"addr":"172.21.15.174:6837/3583791822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3583791822},{"type":"v1","addr":"172.21.15.174:6837","nonce":3583791822}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4829,"qdb_cluster":[4829]},"id":2}]} 2024-04-25T00:07:27.719 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 51 2024-04-25T00:07:27.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.717+0000 7fc86a66f640 1 -- 172.21.15.116:0/2853320454 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc8500678c0 msgr2=0x7fc850069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:27.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.717+0000 7fc86a66f640 1 --2- 172.21.15.116:0/2853320454 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc8500678c0 0x7fc850069d80 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fc854001660 tx=0x7fc854009040 comp rx=0 tx=0).stop 2024-04-25T00:07:27.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.717+0000 7fc86a66f640 1 -- 172.21.15.116:0/2853320454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8641ee600 msgr2=0x7fc8640a8c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:27.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.717+0000 7fc86a66f640 1 --2- 172.21.15.116:0/2853320454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8641ee600 0x7fc8640a8c00 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fc85800f3f0 tx=0x7fc85800f8b0 comp rx=0 tx=0).stop 2024-04-25T00:07:27.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.717+0000 7fc86a66f640 1 -- 172.21.15.116:0/2853320454 shutdown_connections 2024-04-25T00:07:27.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.717+0000 7fc86a66f640 1 --2- 172.21.15.116:0/2853320454 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc8500678c0 0x7fc850069d80 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:27.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.717+0000 7fc86a66f640 1 --2- 172.21.15.116:0/2853320454 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc8640c62c0 0x7fc86414a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:27.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.717+0000 7fc86a66f640 1 --2- 172.21.15.116:0/2853320454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8641ee600 0x7fc8640a8c00 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:27.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.717+0000 7fc86a66f640 1 --2- 172.21.15.116:0/2853320454 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc84c00b570 0x7fc8641e1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:27.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.717+0000 7fc86a66f640 1 -- 172.21.15.116:0/2853320454 >> 172.21.15.116:0/2853320454 conn(0x7fc8640c4200 msgr2=0x7fc8640c45e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:27.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.717+0000 7fc86a66f640 1 -- 172.21.15.116:0/2853320454 shutdown_connections 2024-04-25T00:07:27.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:27.717+0000 7fc86a66f640 1 -- 172.21.15.116:0/2853320454 wait complete. 2024-04-25T00:07:27.737 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:07:29.738 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:07:29.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.869+0000 7fb32ebd2640 1 Processor -- start 2024-04-25T00:07:29.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.869+0000 7fb32ebd2640 1 -- start start 2024-04-25T00:07:29.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.873+0000 7fb32ebd2640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fb328070dd0 0x7fb328071270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:29.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.873+0000 7fb32ebd2640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb32806e890 0x7fb3280717b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:29.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.873+0000 7fb32ebd2640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb328152c10 con 0x7fb328148890 2024-04-25T00:07:29.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.873+0000 7fb32ebd2640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fb328071cf0 con 0x7fb32806e890 2024-04-25T00:07:29.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.873+0000 7fb32ebd2640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fb328151e50 con 0x7fb328070dd0 2024-04-25T00:07:29.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.873+0000 7fb32d16f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb32806e890 0x7fb3280717b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:29.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.873+0000 7fb32d16f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb32806e890 0x7fb3280717b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51682/0 (socket says 172.21.15.116:51682) 2024-04-25T00:07:29.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.873+0000 7fb32d16f640 1 -- 172.21.15.116:0/639553645 learned_addr learned my addr 172.21.15.116:0/639553645 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:07:29.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.873+0000 7fb327fff640 1 --2- 172.21.15.116:0/639553645 >> v2:172.21.15.174:3300/0 conn(0x7fb328070dd0 0x7fb328071270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:29.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.873+0000 7fb32d16f640 1 -- 172.21.15.116:0/639553645 >> v2:172.21.15.174:3300/0 conn(0x7fb328070dd0 msgr2=0x7fb328071270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:07:29.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.873+0000 7fb32d16f640 1 --2- 172.21.15.116:0/639553645 >> v2:172.21.15.174:3300/0 conn(0x7fb328070dd0 0x7fb328071270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:29.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.873+0000 7fb32d16f640 1 -- 172.21.15.116:0/639553645 >> v1:172.21.15.174:6789/0 conn(0x7fb328148890 legacy=0x7fb328072590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:29.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.873+0000 7fb32d16f640 1 -- 172.21.15.116:0/639553645 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb328152130 con 0x7fb32806e890 2024-04-25T00:07:29.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.877+0000 7fb32d16f640 1 --2- 172.21.15.116:0/639553645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb32806e890 0x7fb3280717b0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fb3281562f0 tx=0x7fb32001c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3235ef941d30b395 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:29.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.877+0000 7fb3277fe640 1 -- 172.21.15.116:0/639553645 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3200a5620 con 0x7fb32806e890 2024-04-25T00:07:29.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.877+0000 7fb3277fe640 1 -- 172.21.15.116:0/639553645 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3200a57c0 con 0x7fb32806e890 2024-04-25T00:07:29.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.877+0000 7fb3277fe640 1 -- 172.21.15.116:0/639553645 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3200a5b90 con 0x7fb32806e890 2024-04-25T00:07:29.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.877+0000 7fb32ebd2640 1 -- 172.21.15.116:0/639553645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb32806e890 msgr2=0x7fb3280717b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:29.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.877+0000 7fb32ebd2640 1 --2- 172.21.15.116:0/639553645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb32806e890 0x7fb3280717b0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fb3281562f0 tx=0x7fb32001c960 comp rx=0 tx=0).stop 2024-04-25T00:07:29.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.877+0000 7fb32ebd2640 1 -- 172.21.15.116:0/639553645 shutdown_connections 2024-04-25T00:07:29.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.877+0000 7fb32ebd2640 1 --2- 172.21.15.116:0/639553645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb32806e890 0x7fb3280717b0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:29.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.877+0000 7fb32ebd2640 1 --2- 172.21.15.116:0/639553645 >> v2:172.21.15.174:3300/0 conn(0x7fb328070dd0 0x7fb328071270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:29.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.877+0000 7fb32ebd2640 1 -- 172.21.15.116:0/639553645 >> 172.21.15.116:0/639553645 conn(0x7fb3280c3b00 msgr2=0x7fb3280c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:29.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.877+0000 7fb32ebd2640 1 -- 172.21.15.116:0/639553645 shutdown_connections 2024-04-25T00:07:29.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.877+0000 7fb32ebd2640 1 -- 172.21.15.116:0/639553645 wait complete. 2024-04-25T00:07:29.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.877+0000 7fb32ebd2640 1 Processor -- start 2024-04-25T00:07:29.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.877+0000 7fb32ebd2640 1 -- start start 2024-04-25T00:07:29.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.881+0000 7fb32ebd2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb32806e890 0x7fb32806cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:29.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.881+0000 7fb32ebd2640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb328070dd0 0x7fb3281ec4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:29.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.881+0000 7fb32ebd2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb328148890 0x7fb3281eca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:29.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.881+0000 7fb32ebd2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb328072e20 con 0x7fb32806e890 2024-04-25T00:07:29.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.881+0000 7fb32ebd2640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb328072ca0 con 0x7fb328148890 2024-04-25T00:07:29.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.881+0000 7fb32ebd2640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fb3280abda0 con 0x7fb328070dd0 2024-04-25T00:07:29.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.881+0000 7fb32c96e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb32806e890 0x7fb32806cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:29.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.881+0000 7fb32c96e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb32806e890 0x7fb32806cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51686/0 (socket says 172.21.15.116:51686) 2024-04-25T00:07:29.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.881+0000 7fb32c96e640 1 -- 172.21.15.116:0/457857029 learned_addr learned my addr 172.21.15.116:0/457857029 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:07:29.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.881+0000 7fb32d16f640 1 --2- 172.21.15.116:0/457857029 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb328148890 0x7fb3281eca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:29.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.881+0000 7fb327fff640 1 --2- 172.21.15.116:0/457857029 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb328070dd0 0x7fb3281ec4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:29.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.881+0000 7fb32c96e640 1 -- 172.21.15.116:0/457857029 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb328070dd0 msgr2=0x7fb3281ec4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:29.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.881+0000 7fb32c96e640 1 --2- 172.21.15.116:0/457857029 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb328070dd0 0x7fb3281ec4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:29.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.881+0000 7fb32c96e640 1 -- 172.21.15.116:0/457857029 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb328148890 msgr2=0x7fb3281eca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:29.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.881+0000 7fb32c96e640 1 --2- 172.21.15.116:0/457857029 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb328148890 0x7fb3281eca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:29.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.881+0000 7fb32c96e640 1 -- 172.21.15.116:0/457857029 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3281f06b0 con 0x7fb32806e890 2024-04-25T00:07:29.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.881+0000 7fb327fff640 1 --2- 172.21.15.116:0/457857029 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb328070dd0 0x7fb3281ec4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:07:29.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.881+0000 7fb32c96e640 1 --2- 172.21.15.116:0/457857029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb32806e890 0x7fb32806cdb0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fb31800c3c0 tx=0x7fb31800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:29.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.881+0000 7fb3257fa640 1 -- 172.21.15.116:0/457857029 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb318017070 con 0x7fb32806e890 2024-04-25T00:07:29.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.881+0000 7fb3257fa640 1 -- 172.21.15.116:0/457857029 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3180052d0 con 0x7fb32806e890 2024-04-25T00:07:29.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.881+0000 7fb32ebd2640 1 -- 172.21.15.116:0/457857029 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb328149610 con 0x7fb32806e890 2024-04-25T00:07:29.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.881+0000 7fb3257fa640 1 -- 172.21.15.116:0/457857029 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb318003840 con 0x7fb32806e890 2024-04-25T00:07:29.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.881+0000 7fb32ebd2640 1 -- 172.21.15.116:0/457857029 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb32806d2f0 con 0x7fb32806e890 2024-04-25T00:07:29.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.881+0000 7fb32ebd2640 1 -- 172.21.15.116:0/457857029 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb328068350 con 0x7fb32806e890 2024-04-25T00:07:29.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.881+0000 7fb3257fa640 1 -- 172.21.15.116:0/457857029 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb31801a070 con 0x7fb32806e890 2024-04-25T00:07:29.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.881+0000 7fb3257fa640 1 --2- 172.21.15.116:0/457857029 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb3140678c0 0x7fb314069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:29.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.881+0000 7fb3257fa640 1 -- 172.21.15.116:0/457857029 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 7268+0+0 (secure 0 0 0) 0x7fb3180858b0 con 0x7fb32806e890 2024-04-25T00:07:29.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.881+0000 7fb327fff640 1 --2- 172.21.15.116:0/457857029 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb3140678c0 0x7fb314069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:29.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.885+0000 7fb327fff640 1 --2- 172.21.15.116:0/457857029 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb3140678c0 0x7fb314069d80 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fb3280bd6c0 tx=0x7fb31c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:29.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:29.885+0000 7fb3257fa640 1 -- 172.21.15.116:0/457857029 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fb318021020 con 0x7fb32806e890 2024-04-25T00:07:30.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:30.085+0000 7fb32ebd2640 1 -- 172.21.15.116:0/457857029 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb328151850 con 0x7fb32806e890 2024-04-25T00:07:30.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:30.085+0000 7fb3257fa640 1 -- 172.21.15.116:0/457857029 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) ==== 94+0+6502 (secure 0 0 0) 0x7fb318005480 con 0x7fb32806e890 2024-04-25T00:07:30.089 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:07:30.089 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5775,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/3006671671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3006671671},{"type":"v1","addr":"172.21.15.116:6833","nonce":3006671671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:06:33.449109+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":35,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5613},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5613":{"gid":5613,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":28,"addr":"172.21.15.116:6835/1233829678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1233829678},{"type":"v1","addr":"172.21.15.116:6835","nonce":1233829678}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5613,"qdb_cluster":[5613]},"id":1},{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:07:03.468507+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4829":{"gid":4829,"name":"b","rank":0,"incarnation":47,"state":"up:active","state_seq":10,"addr":"172.21.15.174:6837/3583791822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3583791822},{"type":"v1","addr":"172.21.15.174:6837","nonce":3583791822}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4829,"qdb_cluster":[4829]},"id":2}]} 2024-04-25T00:07:30.089 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 51 2024-04-25T00:07:30.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:30.089+0000 7fb32ebd2640 1 -- 172.21.15.116:0/457857029 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb3140678c0 msgr2=0x7fb314069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:30.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:30.089+0000 7fb32ebd2640 1 --2- 172.21.15.116:0/457857029 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb3140678c0 0x7fb314069d80 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fb3280bd6c0 tx=0x7fb31c008040 comp rx=0 tx=0).stop 2024-04-25T00:07:30.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:30.089+0000 7fb32ebd2640 1 -- 172.21.15.116:0/457857029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb32806e890 msgr2=0x7fb32806cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:30.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:30.089+0000 7fb32ebd2640 1 --2- 172.21.15.116:0/457857029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb32806e890 0x7fb32806cdb0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fb31800c3c0 tx=0x7fb31800c880 comp rx=0 tx=0).stop 2024-04-25T00:07:30.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:30.089+0000 7fb32ebd2640 1 -- 172.21.15.116:0/457857029 shutdown_connections 2024-04-25T00:07:30.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:30.089+0000 7fb32ebd2640 1 --2- 172.21.15.116:0/457857029 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb3140678c0 0x7fb314069d80 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:30.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:30.089+0000 7fb32ebd2640 1 --2- 172.21.15.116:0/457857029 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb328148890 0x7fb3281eca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:30.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:30.089+0000 7fb32ebd2640 1 --2- 172.21.15.116:0/457857029 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb328070dd0 0x7fb3281ec4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:30.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:30.089+0000 7fb32ebd2640 1 --2- 172.21.15.116:0/457857029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb32806e890 0x7fb32806cdb0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:30.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:30.089+0000 7fb32ebd2640 1 -- 172.21.15.116:0/457857029 >> 172.21.15.116:0/457857029 conn(0x7fb3280c3b00 msgr2=0x7fb328072590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:30.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:30.089+0000 7fb32ebd2640 1 -- 172.21.15.116:0/457857029 shutdown_connections 2024-04-25T00:07:30.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:30.089+0000 7fb32ebd2640 1 -- 172.21.15.116:0/457857029 wait complete. 2024-04-25T00:07:30.108 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:07:32.109 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:07:32.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.237+0000 7f3556f16640 1 Processor -- start 2024-04-25T00:07:32.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.237+0000 7f3556f16640 1 -- start start 2024-04-25T00:07:32.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.237+0000 7f3556f16640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f3550150d20 0x7f35500c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:32.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.237+0000 7f3556f16640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f35500a94a0 0x7f3550153ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:32.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.237+0000 7f3556f16640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35500c3910 con 0x7f35500c2f50 2024-04-25T00:07:32.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.237+0000 7f3556f16640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f3550154530 con 0x7f3550150d20 2024-04-25T00:07:32.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.237+0000 7f3556f16640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f35500c16e0 con 0x7f35500a94a0 2024-04-25T00:07:32.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.237+0000 7f35554b3640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f35500a94a0 0x7f3550153ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:32.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.237+0000 7f3554cb2640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f3550150d20 0x7f35500c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:32.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.237+0000 7f35554b3640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f35500a94a0 0x7f3550153ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:33632/0 (socket says 172.21.15.116:33632) 2024-04-25T00:07:32.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.237+0000 7f354ffff640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f35500c2f50 0x7f35501538e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:50944/0 (socket says 172.21.15.116:50944) 2024-04-25T00:07:32.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.237+0000 7f35554b3640 1 -- 172.21.15.116:0/3564539043 learned_addr learned my addr 172.21.15.116:0/3564539043 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:07:32.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.237+0000 7f3554cb2640 1 -- 172.21.15.116:0/3564539043 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f35500a94a0 msgr2=0x7f3550153ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:07:32.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.237+0000 7f3554cb2640 1 --2- 172.21.15.116:0/3564539043 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f35500a94a0 0x7f3550153ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:32.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.237+0000 7f3554cb2640 1 -- 172.21.15.116:0/3564539043 >> v1:172.21.15.174:6789/0 conn(0x7f35500c2f50 legacy=0x7f35501538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:32.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.237+0000 7f3554cb2640 1 -- 172.21.15.116:0/3564539043 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35500c1400 con 0x7f3550150d20 2024-04-25T00:07:32.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.237+0000 7f35554b3640 1 --2- 172.21.15.116:0/3564539043 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f35500a94a0 0x7f3550153ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:07:32.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.241+0000 7f3554cb2640 1 --2- 172.21.15.116:0/3564539043 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3550150d20 0x7f35500c2a10 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f35440097c0 tx=0x7f354401c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f2fafde8ac86584 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:32.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.241+0000 7f354f7fe640 1 -- 172.21.15.116:0/3564539043 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35440b1070 con 0x7f3550150d20 2024-04-25T00:07:32.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.241+0000 7f354f7fe640 1 -- 172.21.15.116:0/3564539043 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35440ac410 con 0x7f3550150d20 2024-04-25T00:07:32.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.241+0000 7f354f7fe640 1 -- 172.21.15.116:0/3564539043 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35440ac7e0 con 0x7f3550150d20 2024-04-25T00:07:32.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.241+0000 7f3556f16640 1 -- 172.21.15.116:0/3564539043 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3550150d20 msgr2=0x7f35500c2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:32.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.241+0000 7f3556f16640 1 --2- 172.21.15.116:0/3564539043 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3550150d20 0x7f35500c2a10 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f35440097c0 tx=0x7f354401c4a0 comp rx=0 tx=0).stop 2024-04-25T00:07:32.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.241+0000 7f3556f16640 1 -- 172.21.15.116:0/3564539043 shutdown_connections 2024-04-25T00:07:32.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.241+0000 7f3556f16640 1 --2- 172.21.15.116:0/3564539043 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f35500a94a0 0x7f3550153ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:32.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.241+0000 7f3556f16640 1 --2- 172.21.15.116:0/3564539043 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3550150d20 0x7f35500c2a10 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:32.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.241+0000 7f3556f16640 1 -- 172.21.15.116:0/3564539043 >> 172.21.15.116:0/3564539043 conn(0x7f35500c3b00 msgr2=0x7f35500c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:32.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.241+0000 7f3556f16640 1 -- 172.21.15.116:0/3564539043 shutdown_connections 2024-04-25T00:07:32.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.241+0000 7f3556f16640 1 -- 172.21.15.116:0/3564539043 wait complete. 2024-04-25T00:07:32.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.241+0000 7f3556f16640 1 Processor -- start 2024-04-25T00:07:32.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.241+0000 7f3556f16640 1 -- start start 2024-04-25T00:07:32.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.245+0000 7f3556f16640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f35500a94a0 0x7f35500c5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:32.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.245+0000 7f3556f16640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35500c2f50 0x7f35501415c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:32.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.245+0000 7f3556f16640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3550150d20 0x7f3550141b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:32.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.245+0000 7f3556f16640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f35501608b0 con 0x7f35500c2f50 2024-04-25T00:07:32.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.245+0000 7f3556f16640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3550160730 con 0x7f3550150d20 2024-04-25T00:07:32.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.245+0000 7f3556f16640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f3550160a30 con 0x7f35500a94a0 2024-04-25T00:07:32.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.245+0000 7f354ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35500c2f50 0x7f35501415c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:32.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.245+0000 7f354ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35500c2f50 0x7f35501415c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51700/0 (socket says 172.21.15.116:51700) 2024-04-25T00:07:32.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.245+0000 7f354ffff640 1 -- 172.21.15.116:0/38542667 learned_addr learned my addr 172.21.15.116:0/38542667 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:07:32.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.245+0000 7f35554b3640 1 --2- 172.21.15.116:0/38542667 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3550150d20 0x7f3550141b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:32.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.245+0000 7f3554cb2640 1 --2- 172.21.15.116:0/38542667 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f35500a94a0 0x7f35500c5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:32.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.245+0000 7f354ffff640 1 -- 172.21.15.116:0/38542667 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f35500a94a0 msgr2=0x7f35500c5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:32.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.245+0000 7f354ffff640 1 --2- 172.21.15.116:0/38542667 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f35500a94a0 0x7f35500c5ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:32.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.245+0000 7f354ffff640 1 -- 172.21.15.116:0/38542667 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3550150d20 msgr2=0x7f3550141b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:32.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.245+0000 7f354ffff640 1 --2- 172.21.15.116:0/38542667 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3550150d20 0x7f3550141b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:32.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.245+0000 7f354ffff640 1 -- 172.21.15.116:0/38542667 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f355014b8f0 con 0x7f35500c2f50 2024-04-25T00:07:32.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.245+0000 7f354ffff640 1 --2- 172.21.15.116:0/38542667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35500c2f50 0x7f35501415c0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f354000c420 tx=0x7f354000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:32.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.245+0000 7f354d7fa640 1 -- 172.21.15.116:0/38542667 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35400074e0 con 0x7f35500c2f50 2024-04-25T00:07:32.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.245+0000 7f354d7fa640 1 -- 172.21.15.116:0/38542667 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3540007680 con 0x7f35500c2f50 2024-04-25T00:07:32.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.245+0000 7f354d7fa640 1 -- 172.21.15.116:0/38542667 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3540004a30 con 0x7f35500c2f50 2024-04-25T00:07:32.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.245+0000 7f3556f16640 1 -- 172.21.15.116:0/38542667 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3550142040 con 0x7f35500c2f50 2024-04-25T00:07:32.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.245+0000 7f3556f16640 1 -- 172.21.15.116:0/38542667 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f355014b610 con 0x7f35500c2f50 2024-04-25T00:07:32.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.245+0000 7f3556f16640 1 -- 172.21.15.116:0/38542667 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3550068350 con 0x7f35500c2f50 2024-04-25T00:07:32.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.245+0000 7f354d7fa640 1 -- 172.21.15.116:0/38542667 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f354001a020 con 0x7f35500c2f50 2024-04-25T00:07:32.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.245+0000 7f354d7fa640 1 --2- 172.21.15.116:0/38542667 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f353c0678c0 0x7f353c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:32.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.245+0000 7f354d7fa640 1 -- 172.21.15.116:0/38542667 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7354+0+0 (secure 0 0 0) 0x7f3540085950 con 0x7f35500c2f50 2024-04-25T00:07:32.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.245+0000 7f3554cb2640 1 --2- 172.21.15.116:0/38542667 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f353c0678c0 0x7f353c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:32.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.249+0000 7f3554cb2640 1 --2- 172.21.15.116:0/38542667 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f353c0678c0 0x7f353c069d80 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f35440ae000 tx=0x7f3544008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:32.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.249+0000 7f354d7fa640 1 -- 172.21.15.116:0/38542667 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f3540055860 con 0x7f35500c2f50 2024-04-25T00:07:32.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.433+0000 7f3556f16640 1 -- 172.21.15.116:0/38542667 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f35500c34f0 con 0x7f35500c2f50 2024-04-25T00:07:32.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.437+0000 7f354d7fa640 1 -- 172.21.15.116:0/38542667 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+5669 (secure 0 0 0) 0x7f3540059b60 con 0x7f35500c2f50 2024-04-25T00:07:32.437 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:07:32.438 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:07:30.726562+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5775":{"gid":5775,"name":"c","rank":0,"incarnation":52,"state":"up:reconnect","state_seq":8,"addr":"172.21.15.116:6833/3006671671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3006671671},{"type":"v1","addr":"172.21.15.116:6833","nonce":3006671671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:07:03.468507+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4829":{"gid":4829,"name":"b","rank":0,"incarnation":47,"state":"up:active","state_seq":10,"addr":"172.21.15.174:6837/3583791822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3583791822},{"type":"v1","addr":"172.21.15.174:6837","nonce":3583791822}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4829,"qdb_cluster":[4829]},"id":2}]} 2024-04-25T00:07:32.438 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 53 2024-04-25T00:07:32.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.437+0000 7f3556f16640 1 -- 172.21.15.116:0/38542667 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f353c0678c0 msgr2=0x7f353c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:32.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.437+0000 7f3556f16640 1 --2- 172.21.15.116:0/38542667 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f353c0678c0 0x7f353c069d80 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f35440ae000 tx=0x7f3544008040 comp rx=0 tx=0).stop 2024-04-25T00:07:32.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.437+0000 7f3556f16640 1 -- 172.21.15.116:0/38542667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35500c2f50 msgr2=0x7f35501415c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:32.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.437+0000 7f3556f16640 1 --2- 172.21.15.116:0/38542667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35500c2f50 0x7f35501415c0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f354000c420 tx=0x7f354000c8e0 comp rx=0 tx=0).stop 2024-04-25T00:07:32.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.437+0000 7f3556f16640 1 -- 172.21.15.116:0/38542667 shutdown_connections 2024-04-25T00:07:32.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.437+0000 7f3556f16640 1 --2- 172.21.15.116:0/38542667 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f353c0678c0 0x7f353c069d80 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:32.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.437+0000 7f3556f16640 1 --2- 172.21.15.116:0/38542667 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3550150d20 0x7f3550141b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:32.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.437+0000 7f3556f16640 1 --2- 172.21.15.116:0/38542667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f35500c2f50 0x7f35501415c0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:32.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.437+0000 7f3556f16640 1 --2- 172.21.15.116:0/38542667 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f35500a94a0 0x7f35500c5ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:32.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.437+0000 7f3556f16640 1 -- 172.21.15.116:0/38542667 >> 172.21.15.116:0/38542667 conn(0x7f35500c3b00 msgr2=0x7f3550147070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:32.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.437+0000 7f3556f16640 1 -- 172.21.15.116:0/38542667 shutdown_connections 2024-04-25T00:07:32.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.437+0000 7f3556f16640 1 -- 172.21.15.116:0/38542667 wait complete. 2024-04-25T00:07:32.455 INFO:tasks.mds_thrash.fs.[a]:mds.a down, removed from mdsmap 2024-04-25T00:07:32.455 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-04-25T00:07:32.455 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:07:32.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.569+0000 7ff008fe6640 1 Processor -- start 2024-04-25T00:07:32.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.569+0000 7ff008fe6640 1 -- start start 2024-04-25T00:07:32.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.569+0000 7ff008fe6640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff00414c720 0x7ff0041563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:32.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.569+0000 7ff008fe6640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff004071680 0x7ff004071ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:32.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.569+0000 7ff008fe6640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff004071220 con 0x7ff0040a94a0 2024-04-25T00:07:32.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.569+0000 7ff008fe6640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7ff004153460 con 0x7ff004071680 2024-04-25T00:07:32.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.569+0000 7ff008fe6640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7ff004152420 con 0x7ff00414c720 2024-04-25T00:07:32.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.569+0000 7ff002d9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff004071680 0x7ff004071ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:32.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.569+0000 7ff002d9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff004071680 0x7ff004071ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51714/0 (socket says 172.21.15.116:51714) 2024-04-25T00:07:32.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.569+0000 7ff002d9d640 1 -- 172.21.15.116:0/262748263 learned_addr learned my addr 172.21.15.116:0/262748263 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:07:32.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.569+0000 7ff00259c640 1 --2- 172.21.15.116:0/262748263 >> v2:172.21.15.174:3300/0 conn(0x7ff00414c720 0x7ff0041563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:32.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.569+0000 7ff002d9d640 1 -- 172.21.15.116:0/262748263 >> v2:172.21.15.174:3300/0 conn(0x7ff00414c720 msgr2=0x7ff0041563d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:07:32.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.569+0000 7ff002d9d640 1 --2- 172.21.15.116:0/262748263 >> v2:172.21.15.174:3300/0 conn(0x7ff00414c720 0x7ff0041563d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:32.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.569+0000 7ff002d9d640 1 -- 172.21.15.116:0/262748263 >> v1:172.21.15.174:6789/0 conn(0x7ff0040a94a0 legacy=0x7ff0040c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:07:32.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.569+0000 7ff002d9d640 1 -- 172.21.15.116:0/262748263 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff00414cb00 con 0x7ff004071680 2024-04-25T00:07:32.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.573+0000 7ff002d9d640 1 --2- 172.21.15.116:0/262748263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff004071680 0x7ff004071ab0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7ff004070190 tx=0x7feffc01c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5edabcacd9c878e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:32.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.573+0000 7ff001d9b640 1 -- 172.21.15.116:0/262748263 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feffc0a57b0 con 0x7ff004071680 2024-04-25T00:07:32.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.573+0000 7ff001d9b640 1 -- 172.21.15.116:0/262748263 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feffc0a5950 con 0x7ff004071680 2024-04-25T00:07:32.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.573+0000 7ff001d9b640 1 -- 172.21.15.116:0/262748263 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feffc0a5d20 con 0x7ff004071680 2024-04-25T00:07:32.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.573+0000 7ff008fe6640 1 -- 172.21.15.116:0/262748263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff004071680 msgr2=0x7ff004071ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:32.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.573+0000 7ff008fe6640 1 --2- 172.21.15.116:0/262748263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff004071680 0x7ff004071ab0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7ff004070190 tx=0x7feffc01c8f0 comp rx=0 tx=0).stop 2024-04-25T00:07:32.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.573+0000 7ff008fe6640 1 -- 172.21.15.116:0/262748263 shutdown_connections 2024-04-25T00:07:32.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.573+0000 7ff008fe6640 1 --2- 172.21.15.116:0/262748263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff004071680 0x7ff004071ab0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:32.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.573+0000 7ff008fe6640 1 --2- 172.21.15.116:0/262748263 >> v2:172.21.15.174:3300/0 conn(0x7ff00414c720 0x7ff0041563d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:32.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.573+0000 7ff008fe6640 1 -- 172.21.15.116:0/262748263 >> 172.21.15.116:0/262748263 conn(0x7ff0040c3b00 msgr2=0x7ff0040c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:32.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.573+0000 7ff008fe6640 1 -- 172.21.15.116:0/262748263 shutdown_connections 2024-04-25T00:07:32.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.573+0000 7ff008fe6640 1 -- 172.21.15.116:0/262748263 wait complete. 2024-04-25T00:07:32.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.573+0000 7ff008fe6640 1 Processor -- start 2024-04-25T00:07:32.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.573+0000 7ff008fe6640 1 -- start start 2024-04-25T00:07:32.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.573+0000 7ff008fe6640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff004071680 0x7ff00406dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:32.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.573+0000 7ff008fe6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0040a94a0 0x7ff004154870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:32.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.573+0000 7ff008fe6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff00414c720 0x7ff004154db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:32.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.573+0000 7ff008fe6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff0040c3120 con 0x7ff0040a94a0 2024-04-25T00:07:32.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.573+0000 7ff008fe6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff004152f40 con 0x7ff00414c720 2024-04-25T00:07:32.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.573+0000 7ff008fe6640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff0040c32a0 con 0x7ff004071680 2024-04-25T00:07:32.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.577+0000 7feff9d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0040a94a0 0x7ff004154870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:32.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.577+0000 7feff9d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0040a94a0 0x7ff004154870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51722/0 (socket says 172.21.15.116:51722) 2024-04-25T00:07:32.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.577+0000 7feff9d9b640 1 -- 172.21.15.116:0/209231792 learned_addr learned my addr 172.21.15.116:0/209231792 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:07:32.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.577+0000 7ff002d9d640 1 --2- 172.21.15.116:0/209231792 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff00414c720 0x7ff004154db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:32.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.577+0000 7ff00259c640 1 --2- 172.21.15.116:0/209231792 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff004071680 0x7ff00406dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:32.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.577+0000 7feff9d9b640 1 -- 172.21.15.116:0/209231792 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff004071680 msgr2=0x7ff00406dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:32.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.577+0000 7feff9d9b640 1 --2- 172.21.15.116:0/209231792 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff004071680 0x7ff00406dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:32.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.577+0000 7feff9d9b640 1 -- 172.21.15.116:0/209231792 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff00414c720 msgr2=0x7ff004154db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:32.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.577+0000 7feff9d9b640 1 --2- 172.21.15.116:0/209231792 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff00414c720 0x7ff004154db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:32.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.577+0000 7feff9d9b640 1 -- 172.21.15.116:0/209231792 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0041ef8d0 con 0x7ff0040a94a0 2024-04-25T00:07:32.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.577+0000 7feff9d9b640 1 --2- 172.21.15.116:0/209231792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0040a94a0 0x7ff004154870 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7feff400c480 tx=0x7feff400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:32.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.577+0000 7feffb7fe640 1 -- 172.21.15.116:0/209231792 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff4012830 con 0x7ff0040a94a0 2024-04-25T00:07:32.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.577+0000 7feffb7fe640 1 -- 172.21.15.116:0/209231792 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feff4017070 con 0x7ff0040a94a0 2024-04-25T00:07:32.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.577+0000 7ff008fe6640 1 -- 172.21.15.116:0/209231792 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0041efbb0 con 0x7ff0040a94a0 2024-04-25T00:07:32.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.577+0000 7feffb7fe640 1 -- 172.21.15.116:0/209231792 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff4012c30 con 0x7ff0040a94a0 2024-04-25T00:07:32.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.577+0000 7ff008fe6640 1 -- 172.21.15.116:0/209231792 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0041f0170 con 0x7ff0040a94a0 2024-04-25T00:07:32.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.577+0000 7ff008fe6640 1 -- 172.21.15.116:0/209231792 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff004068350 con 0x7ff0040a94a0 2024-04-25T00:07:32.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.577+0000 7feffb7fe640 1 -- 172.21.15.116:0/209231792 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7feff401c460 con 0x7ff0040a94a0 2024-04-25T00:07:32.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.577+0000 7feffb7fe640 1 --2- 172.21.15.116:0/209231792 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fefe40678c0 0x7fefe4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:32.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.577+0000 7feffb7fe640 1 -- 172.21.15.116:0/209231792 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7354+0+0 (secure 0 0 0) 0x7feff4086ae0 con 0x7ff0040a94a0 2024-04-25T00:07:32.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.577+0000 7ff00259c640 1 --2- 172.21.15.116:0/209231792 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fefe40678c0 0x7fefe4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:32.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.581+0000 7ff00259c640 1 --2- 172.21.15.116:0/209231792 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fefe40678c0 0x7fefe4069d80 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fefec00b930 tx=0x7fefec008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:32.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.585+0000 7feffb7fe640 1 -- 172.21.15.116:0/209231792 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7feff4057120 con 0x7ff0040a94a0 2024-04-25T00:07:32.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.777+0000 7ff008fe6640 1 -- 172.21.15.116:0/209231792 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff0041561e0 con 0x7ff0040a94a0 2024-04-25T00:07:32.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.781+0000 7feffb7fe640 1 -- 172.21.15.116:0/209231792 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) ==== 94+0+5666 (secure 0 0 0) 0x7feff4056a20 con 0x7ff0040a94a0 2024-04-25T00:07:32.782 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:07:32.782 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:07:31.567508+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5775":{"gid":5775,"name":"c","rank":0,"incarnation":52,"state":"up:rejoin","state_seq":9,"addr":"172.21.15.116:6833/3006671671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3006671671},{"type":"v1","addr":"172.21.15.116:6833","nonce":3006671671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:07:03.468507+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4829":{"gid":4829,"name":"b","rank":0,"incarnation":47,"state":"up:active","state_seq":10,"addr":"172.21.15.174:6837/3583791822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3583791822},{"type":"v1","addr":"172.21.15.174:6837","nonce":3583791822}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4829,"qdb_cluster":[4829]},"id":2}]} 2024-04-25T00:07:32.782 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 54 2024-04-25T00:07:32.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.781+0000 7ff008fe6640 1 -- 172.21.15.116:0/209231792 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fefe40678c0 msgr2=0x7fefe4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:32.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.781+0000 7ff008fe6640 1 --2- 172.21.15.116:0/209231792 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fefe40678c0 0x7fefe4069d80 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fefec00b930 tx=0x7fefec008040 comp rx=0 tx=0).stop 2024-04-25T00:07:32.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.781+0000 7ff008fe6640 1 -- 172.21.15.116:0/209231792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0040a94a0 msgr2=0x7ff004154870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:32.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.781+0000 7ff008fe6640 1 --2- 172.21.15.116:0/209231792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0040a94a0 0x7ff004154870 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7feff400c480 tx=0x7feff400c940 comp rx=0 tx=0).stop 2024-04-25T00:07:32.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.781+0000 7ff008fe6640 1 -- 172.21.15.116:0/209231792 shutdown_connections 2024-04-25T00:07:32.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.781+0000 7ff008fe6640 1 --2- 172.21.15.116:0/209231792 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fefe40678c0 0x7fefe4069d80 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:32.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.781+0000 7ff008fe6640 1 --2- 172.21.15.116:0/209231792 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff00414c720 0x7ff004154db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:32.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.781+0000 7ff008fe6640 1 --2- 172.21.15.116:0/209231792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0040a94a0 0x7ff004154870 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:32.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.781+0000 7ff008fe6640 1 --2- 172.21.15.116:0/209231792 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff004071680 0x7ff00406dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:32.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.781+0000 7ff008fe6640 1 -- 172.21.15.116:0/209231792 >> 172.21.15.116:0/209231792 conn(0x7ff0040c3b00 msgr2=0x7ff004071a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:32.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.781+0000 7ff008fe6640 1 -- 172.21.15.116:0/209231792 shutdown_connections 2024-04-25T00:07:32.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:32.781+0000 7ff008fe6640 1 -- 172.21.15.116:0/209231792 wait complete. 2024-04-25T00:07:32.801 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 54, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:04.678737+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 54, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5775": { "addr": "172.21.15.116:6833/3006671671", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6832", "nonce": 3006671671, "type": "v2" }, { "addr": "172.21.15.116:6833", "nonce": 3006671671, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5775, "incarnation": 52, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:rejoin", "state_seq": 9 } }, "last_failure": 0, "last_failure_osd_epoch": 37, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T00:07:31.567508+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5775 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:18.825887+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 50, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_4829": { "addr": "172.21.15.174:6837/3583791822", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6836", "nonce": 3583791822, "type": "v2" }, { "addr": "172.21.15.174:6837", "nonce": 3583791822, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4829, "incarnation": 47, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 10 } }, "last_failure": 0, "last_failure_osd_epoch": 36, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-04-25T00:07:03.468507+0000", "qdb_cluster": [ 4829 ], "qdb_leader": 4829, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4829 } } } ], "standbys": [ { "addr": "172.21.15.174:6835/2883220846", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6834", "nonce": 2883220846, "type": "v2" }, { "addr": "172.21.15.174:6835", "nonce": 2883220846, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4235, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-25T00:07:34.802 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:07:34.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.921+0000 7f7dd1f4b640 1 Processor -- start 2024-04-25T00:07:34.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.921+0000 7f7dd1f4b640 1 -- start start 2024-04-25T00:07:34.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.921+0000 7f7dd1f4b640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f7dcc14c700 0x7f7dcc0c5090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:34.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.921+0000 7f7dd1f4b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7dcc155b20 0x7f7dcc152c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:34.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.921+0000 7f7dd1f4b640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7dcc0a9e60 con 0x7f7dcc0c55d0 2024-04-25T00:07:34.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.921+0000 7f7dd1f4b640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f7dcc1527b0 con 0x7f7dcc155b20 2024-04-25T00:07:34.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.921+0000 7f7dd1f4b640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f7dcc0c3ef0 con 0x7f7dcc14c700 2024-04-25T00:07:34.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.921+0000 7f7dcaffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7dcc155b20 0x7f7dcc152c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:34.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.921+0000 7f7dcbfff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f7dcc0c55d0 0x7f7dcc071df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:45442/0 (socket says 172.21.15.116:45442) 2024-04-25T00:07:34.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.921+0000 7f7dcbfff640 1 -- 172.21.15.116:0/2830211181 learned_addr learned my addr 172.21.15.116:0/2830211181 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:07:34.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.921+0000 7f7dcb7fe640 1 --2- 172.21.15.116:0/2830211181 >> v2:172.21.15.174:3300/0 conn(0x7f7dcc14c700 0x7f7dcc0c5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:34.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.921+0000 7f7dcaffd640 1 -- 172.21.15.116:0/2830211181 >> v2:172.21.15.174:3300/0 conn(0x7f7dcc14c700 msgr2=0x7f7dcc0c5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:07:34.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.921+0000 7f7dcaffd640 1 --2- 172.21.15.116:0/2830211181 >> v2:172.21.15.174:3300/0 conn(0x7f7dcc14c700 0x7f7dcc0c5090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:34.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.921+0000 7f7dcaffd640 1 -- 172.21.15.116:0/2830211181 >> v1:172.21.15.116:6789/0 conn(0x7f7dcc0c55d0 legacy=0x7f7dcc071df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:34.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.921+0000 7f7dca7fc640 1 -- 172.21.15.116:0/2830211181 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 220605055 0 0) 0x7f7dcc0a9e60 con 0x7f7dcc0c55d0 2024-04-25T00:07:34.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.921+0000 7f7dcaffd640 1 -- 172.21.15.116:0/2830211181 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dcc0c41d0 con 0x7f7dcc155b20 2024-04-25T00:07:34.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.921+0000 7f7dcb7fe640 1 --2- 172.21.15.116:0/2830211181 >> v2:172.21.15.174:3300/0 conn(0x7f7dcc14c700 0x7f7dcc0c5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:07:34.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.925+0000 7f7dcaffd640 1 --2- 172.21.15.116:0/2830211181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7dcc155b20 0x7f7dcc152c40 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc0c90b0 tx=0x7f7dc001c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33bc9ebff9c2dbde server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:34.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.925+0000 7f7dca7fc640 1 -- 172.21.15.116:0/2830211181 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc00b0070 con 0x7f7dcc155b20 2024-04-25T00:07:34.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.925+0000 7f7dca7fc640 1 -- 172.21.15.116:0/2830211181 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7dc00ab600 con 0x7f7dcc155b20 2024-04-25T00:07:34.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.925+0000 7f7dca7fc640 1 -- 172.21.15.116:0/2830211181 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc00ab9d0 con 0x7f7dcc155b20 2024-04-25T00:07:34.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.925+0000 7f7dd1f4b640 1 -- 172.21.15.116:0/2830211181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7dcc155b20 msgr2=0x7f7dcc152c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:34.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.925+0000 7f7dd1f4b640 1 --2- 172.21.15.116:0/2830211181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7dcc155b20 0x7f7dcc152c40 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc0c90b0 tx=0x7f7dc001c360 comp rx=0 tx=0).stop 2024-04-25T00:07:34.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.925+0000 7f7dd1f4b640 1 -- 172.21.15.116:0/2830211181 shutdown_connections 2024-04-25T00:07:34.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.925+0000 7f7dd1f4b640 1 --2- 172.21.15.116:0/2830211181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7dcc155b20 0x7f7dcc152c40 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:34.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.925+0000 7f7dd1f4b640 1 --2- 172.21.15.116:0/2830211181 >> v2:172.21.15.174:3300/0 conn(0x7f7dcc14c700 0x7f7dcc0c5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:34.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.925+0000 7f7dd1f4b640 1 -- 172.21.15.116:0/2830211181 >> 172.21.15.116:0/2830211181 conn(0x7f7dcc06c720 msgr2=0x7f7dcc06cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:34.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.925+0000 7f7dd1f4b640 1 -- 172.21.15.116:0/2830211181 shutdown_connections 2024-04-25T00:07:34.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.925+0000 7f7dd1f4b640 1 -- 172.21.15.116:0/2830211181 wait complete. 2024-04-25T00:07:34.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.929+0000 7f7dd1f4b640 1 Processor -- start 2024-04-25T00:07:34.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.929+0000 7f7dd1f4b640 1 -- start start 2024-04-25T00:07:34.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.929+0000 7f7dd1f4b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7dcc0c55d0 0x7f7dcc1460c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:34.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.929+0000 7f7dd1f4b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7dcc14c700 0x7f7dcc0c1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:34.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.929+0000 7f7dd1f4b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7dcc155b20 0x7f7dcc1f1e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:34.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.929+0000 7f7dd1f4b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7dcc153510 con 0x7f7dcc14c700 2024-04-25T00:07:34.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.929+0000 7f7dd1f4b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7dcc153390 con 0x7f7dcc155b20 2024-04-25T00:07:34.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.929+0000 7f7dd1f4b640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f7dcc15fed0 con 0x7f7dcc0c55d0 2024-04-25T00:07:34.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.929+0000 7f7dcaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7dcc14c700 0x7f7dcc0c1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:34.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.929+0000 7f7dcaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7dcc14c700 0x7f7dcc0c1730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42852/0 (socket says 172.21.15.116:42852) 2024-04-25T00:07:34.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.929+0000 7f7dcaffd640 1 -- 172.21.15.116:0/22018690 learned_addr learned my addr 172.21.15.116:0/22018690 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:07:34.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.929+0000 7f7dcb7fe640 1 --2- 172.21.15.116:0/22018690 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7dcc0c55d0 0x7f7dcc1460c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:34.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.929+0000 7f7dcbfff640 1 --2- 172.21.15.116:0/22018690 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7dcc155b20 0x7f7dcc1f1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:34.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.929+0000 7f7dcaffd640 1 -- 172.21.15.116:0/22018690 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7dcc0c55d0 msgr2=0x7f7dcc1460c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:34.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.929+0000 7f7dcaffd640 1 --2- 172.21.15.116:0/22018690 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7dcc0c55d0 0x7f7dcc1460c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:34.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.929+0000 7f7dcaffd640 1 -- 172.21.15.116:0/22018690 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7dcc155b20 msgr2=0x7f7dcc1f1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:34.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.929+0000 7f7dcaffd640 1 --2- 172.21.15.116:0/22018690 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7dcc155b20 0x7f7dcc1f1e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:34.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.929+0000 7f7dcaffd640 1 -- 172.21.15.116:0/22018690 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dcc146720 con 0x7f7dcc14c700 2024-04-25T00:07:34.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.929+0000 7f7dcaffd640 1 --2- 172.21.15.116:0/22018690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7dcc14c700 0x7f7dcc0c1730 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f7dc0018f70 tx=0x7f7dc0019100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:34.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.929+0000 7f7db3fff640 1 -- 172.21.15.116:0/22018690 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc00ba070 con 0x7f7dcc14c700 2024-04-25T00:07:34.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.929+0000 7f7db3fff640 1 -- 172.21.15.116:0/22018690 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7dc00b5670 con 0x7f7dcc14c700 2024-04-25T00:07:34.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.929+0000 7f7dd1f4b640 1 -- 172.21.15.116:0/22018690 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7dcc146a00 con 0x7f7dcc14c700 2024-04-25T00:07:34.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.929+0000 7f7db3fff640 1 -- 172.21.15.116:0/22018690 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dc00b0040 con 0x7f7dcc14c700 2024-04-25T00:07:34.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.929+0000 7f7dd1f4b640 1 -- 172.21.15.116:0/22018690 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7dcc1458c0 con 0x7f7dcc14c700 2024-04-25T00:07:34.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.933+0000 7f7db3fff640 1 -- 172.21.15.116:0/22018690 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7dc00bd070 con 0x7f7dcc14c700 2024-04-25T00:07:34.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.933+0000 7f7db3fff640 1 --2- 172.21.15.116:0/22018690 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7db80678f0 0x7f7db8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:34.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.933+0000 7f7db3fff640 1 -- 172.21.15.116:0/22018690 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7354+0+0 (secure 0 0 0) 0x7f7dc0120a80 con 0x7f7dcc14c700 2024-04-25T00:07:34.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.933+0000 7f7dcb7fe640 1 --2- 172.21.15.116:0/22018690 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7db80678f0 0x7f7db8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:34.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.933+0000 7f7dd1f4b640 1 -- 172.21.15.116:0/22018690 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7dcc068a60 con 0x7f7dcc14c700 2024-04-25T00:07:34.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.933+0000 7f7dcb7fe640 1 --2- 172.21.15.116:0/22018690 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7db80678f0 0x7f7db8069db0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc00c900 tx=0x7f7dbc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:34.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:34.941+0000 7f7db3fff640 1 -- 172.21.15.116:0/22018690 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f7dc00f5e20 con 0x7f7dcc14c700 2024-04-25T00:07:35.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:35.225+0000 7f7dd1f4b640 1 -- 172.21.15.116:0/22018690 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7dcc0ad630 con 0x7f7dcc14c700 2024-04-25T00:07:35.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:35.225+0000 7f7db3fff640 1 -- 172.21.15.116:0/22018690 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) ==== 94+0+5674 (secure 0 0 0) 0x7f7dc0016070 con 0x7f7dcc14c700 2024-04-25T00:07:35.229 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:07:35.229 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":55,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:07:33.575034+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5775":{"gid":5775,"name":"c","rank":0,"incarnation":52,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6833/3006671671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3006671671},{"type":"v1","addr":"172.21.15.116:6833","nonce":3006671671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5775]},"id":1},{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:07:03.468507+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4829":{"gid":4829,"name":"b","rank":0,"incarnation":47,"state":"up:active","state_seq":10,"addr":"172.21.15.174:6837/3583791822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3583791822},{"type":"v1","addr":"172.21.15.174:6837","nonce":3583791822}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4829,"qdb_cluster":[4829]},"id":2}]} 2024-04-25T00:07:35.229 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 55 2024-04-25T00:07:35.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:35.229+0000 7f7dd1f4b640 1 -- 172.21.15.116:0/22018690 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7db80678f0 msgr2=0x7f7db8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:35.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:35.229+0000 7f7dd1f4b640 1 --2- 172.21.15.116:0/22018690 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7db80678f0 0x7f7db8069db0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc00c900 tx=0x7f7dbc008040 comp rx=0 tx=0).stop 2024-04-25T00:07:35.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:35.229+0000 7f7dd1f4b640 1 -- 172.21.15.116:0/22018690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7dcc14c700 msgr2=0x7f7dcc0c1730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:35.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:35.229+0000 7f7dd1f4b640 1 --2- 172.21.15.116:0/22018690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7dcc14c700 0x7f7dcc0c1730 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f7dc0018f70 tx=0x7f7dc0019100 comp rx=0 tx=0).stop 2024-04-25T00:07:35.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:35.229+0000 7f7dd1f4b640 1 -- 172.21.15.116:0/22018690 shutdown_connections 2024-04-25T00:07:35.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:35.229+0000 7f7dd1f4b640 1 --2- 172.21.15.116:0/22018690 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7db80678f0 0x7f7db8069db0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:35.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:35.229+0000 7f7dd1f4b640 1 --2- 172.21.15.116:0/22018690 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7dcc155b20 0x7f7dcc1f1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:35.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:35.229+0000 7f7dd1f4b640 1 --2- 172.21.15.116:0/22018690 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7dcc14c700 0x7f7dcc0c1730 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:35.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:35.229+0000 7f7dd1f4b640 1 --2- 172.21.15.116:0/22018690 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7dcc0c55d0 0x7f7dcc1460c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:35.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:35.229+0000 7f7dd1f4b640 1 -- 172.21.15.116:0/22018690 >> 172.21.15.116:0/22018690 conn(0x7f7dcc06c720 msgr2=0x7f7dcc0c3230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:35.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:35.229+0000 7f7dd1f4b640 1 -- 172.21.15.116:0/22018690 shutdown_connections 2024-04-25T00:07:35.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:35.229+0000 7f7dd1f4b640 1 -- 172.21.15.116:0/22018690 wait complete. 2024-04-25T00:07:35.255 INFO:tasks.mds_thrash.fs.[a]:mds.c has gained rank=0, replacing gid=5613 2024-04-25T00:07:35.255 INFO:tasks.mds_thrash.fs.[a]:waiting for 4 secs before reviving mds.a 2024-04-25T00:07:35.481 INFO:tasks.ceph.mds.c.smithi116.stderr:2024-04-25T00:07:35.477+0000 7f09c3a18640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T00:07:36.134 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:07:36.136 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:07:39.256 INFO:tasks.mds_thrash.fs.[a]:reviving mds.a 2024-04-25T00:07:39.256 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-25T00:07:39.257 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-25T00:07:39.260 INFO:tasks.ceph.mds.a:Started 2024-04-25T00:07:39.261 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:07:39.338 INFO:tasks.ceph.mds.a.smithi116.stdout:starting mds.a at 2024-04-25T00:07:39.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.393+0000 7fd0c63be640 1 Processor -- start 2024-04-25T00:07:39.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.393+0000 7fd0c63be640 1 -- start start 2024-04-25T00:07:39.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.397+0000 7fd0c63be640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd0c0152e60 0x7fd0c01599a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:39.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.397+0000 7fd0c63be640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0c0073200 con 0x7fd0c01562d0 2024-04-25T00:07:39.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.397+0000 7fd0c63be640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0c0158ec0 con 0x7fd0c00c5010 2024-04-25T00:07:39.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.397+0000 7fd0c63be640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fd0c0153240 con 0x7fd0c0152e60 2024-04-25T00:07:39.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.397+0000 7fd0b7fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fd0c01562d0 0x7fd0c00c36e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:45462/0 (socket says 172.21.15.116:45462) 2024-04-25T00:07:39.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.397+0000 7fd0b7fff640 1 -- 172.21.15.116:0/1309270950 learned_addr learned my addr 172.21.15.116:0/1309270950 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:07:39.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.397+0000 7fd0bffff640 1 --2- 172.21.15.116:0/1309270950 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd0c0152e60 0x7fd0c01599a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:39.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.397+0000 7fd0bf7fe640 1 -- 172.21.15.116:0/1309270950 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1780147976 0 0) 0x7fd0c0073200 con 0x7fd0c01562d0 2024-04-25T00:07:39.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.397+0000 7fd0bf7fe640 1 -- 172.21.15.116:0/1309270950 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd0ac003530 con 0x7fd0c01562d0 2024-04-25T00:07:39.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.397+0000 7fd0bf7fe640 1 -- 172.21.15.116:0/1309270950 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3831953438 0 0) 0x7fd0c0158ec0 con 0x7fd0c00c5010 2024-04-25T00:07:39.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.397+0000 7fd0bf7fe640 1 -- 172.21.15.116:0/1309270950 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd0c0073200 con 0x7fd0c00c5010 2024-04-25T00:07:39.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.397+0000 7fd0bf7fe640 1 -- 172.21.15.116:0/1309270950 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 202326071 0 0) 0x7fd0ac003530 con 0x7fd0c01562d0 2024-04-25T00:07:39.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.397+0000 7fd0bf7fe640 1 -- 172.21.15.116:0/1309270950 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd0c0152e60 msgr2=0x7fd0c01599a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:07:39.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.397+0000 7fd0bf7fe640 1 --2- 172.21.15.116:0/1309270950 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd0c0152e60 0x7fd0c01599a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:39.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.397+0000 7fd0bf7fe640 1 -- 172.21.15.116:0/1309270950 >> v1:172.21.15.174:6789/0 conn(0x7fd0c00c5010 legacy=0x7fd0c0160120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:39.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.397+0000 7fd0bf7fe640 1 -- 172.21.15.116:0/1309270950 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0c01591a0 con 0x7fd0c01562d0 2024-04-25T00:07:39.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.397+0000 7fd0bf7fe640 1 -- 172.21.15.116:0/1309270950 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7fd0a8001680 con 0x7fd0c01562d0 2024-04-25T00:07:39.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.397+0000 7fd0bf7fe640 1 -- 172.21.15.116:0/1309270950 >> v1:172.21.15.116:6789/0 conn(0x7fd0c01562d0 legacy=0x7fd0c00c36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:39.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.397+0000 7fd0bf7fe640 1 --2- 172.21.15.116:0/1309270950 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd0c0152e60 0x7fd0ac003b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:39.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.397+0000 7fd0bf7fe640 1 --2- 172.21.15.116:0/1309270950 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0ac0040e0 0x7fd0ac004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:39.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.397+0000 7fd0bf7fe640 1 --2- 172.21.15.116:0/1309270950 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0ac004d50 0x7fd0ac0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:39.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.397+0000 7fd0bf7fe640 1 -- 172.21.15.116:0/1309270950 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd0c0153240 con 0x7fd0ac004d50 2024-04-25T00:07:39.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.397+0000 7fd0bf7fe640 1 -- 172.21.15.116:0/1309270950 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd0ac00bb60 con 0x7fd0ac0040e0 2024-04-25T00:07:39.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.397+0000 7fd0bf7fe640 1 -- 172.21.15.116:0/1309270950 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fd0ac00bd30 con 0x7fd0c0152e60 2024-04-25T00:07:39.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.397+0000 7fd0c495b640 1 --2- 172.21.15.116:0/1309270950 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0ac004d50 0x7fd0ac0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:39.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.397+0000 7fd0b7fff640 1 --2- 172.21.15.116:0/1309270950 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0ac0040e0 0x7fd0ac004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:39.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.397+0000 7fd0bffff640 1 --2- 172.21.15.116:0/1309270950 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd0c0152e60 0x7fd0ac003b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:39.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.397+0000 7fd0c495b640 1 -- 172.21.15.116:0/1309270950 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd0c0152e60 msgr2=0x7fd0ac003b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:07:39.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.397+0000 7fd0c495b640 1 --2- 172.21.15.116:0/1309270950 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd0c0152e60 0x7fd0ac003b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:39.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.397+0000 7fd0c495b640 1 -- 172.21.15.116:0/1309270950 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0ac0040e0 msgr2=0x7fd0ac004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:07:39.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.397+0000 7fd0c495b640 1 --2- 172.21.15.116:0/1309270950 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0ac0040e0 0x7fd0ac004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:39.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.397+0000 7fd0c495b640 1 -- 172.21.15.116:0/1309270950 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0ac003530 con 0x7fd0ac004d50 2024-04-25T00:07:39.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.397+0000 7fd0bffff640 1 --2- 172.21.15.116:0/1309270950 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd0c0152e60 0x7fd0ac003b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:07:39.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.401+0000 7fd0c495b640 1 --2- 172.21.15.116:0/1309270950 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0ac004d50 0x7fd0ac0051d0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fd0b8001fd0 tx=0x7fd0b801f070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef921992835523b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:39.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.401+0000 7fd0bf7fe640 1 -- 172.21.15.116:0/1309270950 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0b8001c20 con 0x7fd0ac004d50 2024-04-25T00:07:39.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.401+0000 7fd0bf7fe640 1 -- 172.21.15.116:0/1309270950 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0b80a8cb0 con 0x7fd0ac004d50 2024-04-25T00:07:39.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.401+0000 7fd0bf7fe640 1 -- 172.21.15.116:0/1309270950 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0b80a8e60 con 0x7fd0ac004d50 2024-04-25T00:07:39.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.401+0000 7fd0c63be640 1 -- 172.21.15.116:0/1309270950 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0ac004d50 msgr2=0x7fd0ac0051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:39.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.401+0000 7fd0c63be640 1 --2- 172.21.15.116:0/1309270950 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0ac004d50 0x7fd0ac0051d0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fd0b8001fd0 tx=0x7fd0b801f070 comp rx=0 tx=0).stop 2024-04-25T00:07:39.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.401+0000 7fd0c495b640 1 -- 172.21.15.116:0/1309270950 reap_dead start 2024-04-25T00:07:39.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.401+0000 7fd0c63be640 1 -- 172.21.15.116:0/1309270950 shutdown_connections 2024-04-25T00:07:39.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.401+0000 7fd0c63be640 1 -- 172.21.15.116:0/1309270950 >> 172.21.15.116:0/1309270950 conn(0x7fd0c006c720 msgr2=0x7fd0c006cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:39.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.401+0000 7fd0c63be640 1 -- 172.21.15.116:0/1309270950 shutdown_connections 2024-04-25T00:07:39.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.401+0000 7fd0c63be640 1 -- 172.21.15.116:0/1309270950 wait complete. 2024-04-25T00:07:39.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.401+0000 7fd0c63be640 1 Processor -- start 2024-04-25T00:07:39.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.401+0000 7fd0c63be640 1 -- start start 2024-04-25T00:07:39.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.401+0000 7fd0c63be640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0c0151030 0x7fd0c0153760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:39.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.401+0000 7fd0c63be640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd0c014f340 0x7fd0c01ef200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:39.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.401+0000 7fd0c63be640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0c01ebac0 0x7fd0c00ab060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:39.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.401+0000 7fd0c63be640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd0c0156730 con 0x7fd0c0151030 2024-04-25T00:07:39.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.401+0000 7fd0c63be640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd0c00734e0 con 0x7fd0c01ebac0 2024-04-25T00:07:39.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.401+0000 7fd0c63be640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fd0c0072080 con 0x7fd0c014f340 2024-04-25T00:07:39.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.401+0000 7fd0bffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0c0151030 0x7fd0c0153760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:39.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.401+0000 7fd0bffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0c0151030 0x7fd0c0153760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42866/0 (socket says 172.21.15.116:42866) 2024-04-25T00:07:39.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.401+0000 7fd0bffff640 1 -- 172.21.15.116:0/1034475743 learned_addr learned my addr 172.21.15.116:0/1034475743 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:07:39.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.401+0000 7fd0b7fff640 1 --2- 172.21.15.116:0/1034475743 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd0c014f340 0x7fd0c01ef200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:39.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.401+0000 7fd0c495b640 1 --2- 172.21.15.116:0/1034475743 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0c01ebac0 0x7fd0c00ab060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:39.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.401+0000 7fd0bffff640 1 -- 172.21.15.116:0/1034475743 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd0c014f340 msgr2=0x7fd0c01ef200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:39.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.401+0000 7fd0bffff640 1 --2- 172.21.15.116:0/1034475743 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd0c014f340 0x7fd0c01ef200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:39.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.405+0000 7fd0bffff640 1 -- 172.21.15.116:0/1034475743 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0c01ebac0 msgr2=0x7fd0c00ab060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:39.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.405+0000 7fd0bffff640 1 --2- 172.21.15.116:0/1034475743 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0c01ebac0 0x7fd0c00ab060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:39.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.405+0000 7fd0bffff640 1 -- 172.21.15.116:0/1034475743 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0c01519f0 con 0x7fd0c0151030 2024-04-25T00:07:39.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.405+0000 7fd0b7fff640 1 --2- 172.21.15.116:0/1034475743 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd0c014f340 0x7fd0c01ef200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:07:39.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.405+0000 7fd0bffff640 1 --2- 172.21.15.116:0/1034475743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0c0151030 0x7fd0c0153760 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fd0b0009fd0 tx=0x7fd0b0011b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:39.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.405+0000 7fd0bd7fa640 1 -- 172.21.15.116:0/1034475743 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0b0004220 con 0x7fd0c0151030 2024-04-25T00:07:39.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.405+0000 7fd0bd7fa640 1 -- 172.21.15.116:0/1034475743 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0b00043c0 con 0x7fd0c0151030 2024-04-25T00:07:39.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.405+0000 7fd0c63be640 1 -- 172.21.15.116:0/1034475743 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0c00ac7f0 con 0x7fd0c0151030 2024-04-25T00:07:39.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.405+0000 7fd0bd7fa640 1 -- 172.21.15.116:0/1034475743 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0b000f9e0 con 0x7fd0c0151030 2024-04-25T00:07:39.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.405+0000 7fd0c63be640 1 -- 172.21.15.116:0/1034475743 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0c00acdb0 con 0x7fd0c0151030 2024-04-25T00:07:39.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.405+0000 7fd0c63be640 1 -- 172.21.15.116:0/1034475743 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0c0068a60 con 0x7fd0c0151030 2024-04-25T00:07:39.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.405+0000 7fd0bd7fa640 1 -- 172.21.15.116:0/1034475743 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd0b000fb80 con 0x7fd0c0151030 2024-04-25T00:07:39.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.405+0000 7fd0bd7fa640 1 --2- 172.21.15.116:0/1034475743 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd0a00678c0 0x7fd0a0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:39.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.405+0000 7fd0bd7fa640 1 -- 172.21.15.116:0/1034475743 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7354+0+0 (secure 0 0 0) 0x7fd0b00868b0 con 0x7fd0c0151030 2024-04-25T00:07:39.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.405+0000 7fd0b7fff640 1 --2- 172.21.15.116:0/1034475743 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd0a00678c0 0x7fd0a0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:39.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.405+0000 7fd0b7fff640 1 --2- 172.21.15.116:0/1034475743 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd0a00678c0 0x7fd0a0069d80 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd0a8004850 tx=0x7fd0a8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:39.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.413+0000 7fd0bd7fa640 1 -- 172.21.15.116:0/1034475743 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fd0b0019030 con 0x7fd0c0151030 2024-04-25T00:07:39.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.657+0000 7fd0c63be640 1 -- 172.21.15.116:0/1034475743 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd0c0159c00 con 0x7fd0c0151030 2024-04-25T00:07:39.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.657+0000 7fd0bd7fa640 1 -- 172.21.15.116:0/1034475743 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) ==== 94+0+6502 (secure 0 0 0) 0x7fd0b00567c0 con 0x7fd0c0151030 2024-04-25T00:07:39.658 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:07:39.659 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5868,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1670061091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1670061091},{"type":"v1","addr":"172.21.15.116:6835","nonce":1670061091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:07:33.575034+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5775":{"gid":5775,"name":"c","rank":0,"incarnation":52,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6833/3006671671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3006671671},{"type":"v1","addr":"172.21.15.116:6833","nonce":3006671671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5775]},"id":1},{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:07:03.468507+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4829":{"gid":4829,"name":"b","rank":0,"incarnation":47,"state":"up:active","state_seq":10,"addr":"172.21.15.174:6837/3583791822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3583791822},{"type":"v1","addr":"172.21.15.174:6837","nonce":3583791822}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4829,"qdb_cluster":[4829]},"id":2}]} 2024-04-25T00:07:39.659 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 56 2024-04-25T00:07:39.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.661+0000 7fd0c63be640 1 -- 172.21.15.116:0/1034475743 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd0a00678c0 msgr2=0x7fd0a0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:39.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.661+0000 7fd0c63be640 1 --2- 172.21.15.116:0/1034475743 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd0a00678c0 0x7fd0a0069d80 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd0a8004850 tx=0x7fd0a8009040 comp rx=0 tx=0).stop 2024-04-25T00:07:39.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.661+0000 7fd0c63be640 1 -- 172.21.15.116:0/1034475743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0c0151030 msgr2=0x7fd0c0153760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:39.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.661+0000 7fd0c63be640 1 --2- 172.21.15.116:0/1034475743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0c0151030 0x7fd0c0153760 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fd0b0009fd0 tx=0x7fd0b0011b30 comp rx=0 tx=0).stop 2024-04-25T00:07:39.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.661+0000 7fd0c63be640 1 -- 172.21.15.116:0/1034475743 shutdown_connections 2024-04-25T00:07:39.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.661+0000 7fd0c63be640 1 --2- 172.21.15.116:0/1034475743 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd0a00678c0 0x7fd0a0069d80 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:39.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.661+0000 7fd0c63be640 1 --2- 172.21.15.116:0/1034475743 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0c01ebac0 0x7fd0c00ab060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:39.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.661+0000 7fd0c63be640 1 --2- 172.21.15.116:0/1034475743 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd0c014f340 0x7fd0c01ef200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:39.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.661+0000 7fd0c63be640 1 --2- 172.21.15.116:0/1034475743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0c0151030 0x7fd0c0153760 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:39.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.661+0000 7fd0c63be640 1 -- 172.21.15.116:0/1034475743 >> 172.21.15.116:0/1034475743 conn(0x7fd0c006c720 msgr2=0x7fd0c00c8960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:39.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.661+0000 7fd0c63be640 1 -- 172.21.15.116:0/1034475743 shutdown_connections 2024-04-25T00:07:39.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:39.661+0000 7fd0c63be640 1 -- 172.21.15.116:0/1034475743 wait complete. 2024-04-25T00:07:39.680 INFO:tasks.mds_thrash.fs.[a]:mds.a reported in up:standby state 2024-04-25T00:07:39.680 INFO:tasks.mds_thrash.fs.[a]:waiting for 96 secs before thrashing 2024-04-25T00:07:43.687 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:07:43.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.817+0000 7f1627d19640 1 Processor -- start 2024-04-25T00:07:43.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.817+0000 7f1627d19640 1 -- start start 2024-04-25T00:07:43.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.821+0000 7f1627d19640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1620152e40 0x7f16200710b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:43.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.821+0000 7f1627d19640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f16200715f0 0x7f16201557c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:43.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.821+0000 7f1627d19640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1620153220 con 0x7f16200c3f70 2024-04-25T00:07:43.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.821+0000 7f1627d19640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f16200c4c90 con 0x7f1620152e40 2024-04-25T00:07:43.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.821+0000 7f1627d19640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f16200c7a90 con 0x7f16200715f0 2024-04-25T00:07:43.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.821+0000 7f1625ab5640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1620152e40 0x7f16200710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:43.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.821+0000 7f16262b6640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f16200c3f70 0x7f16200c23c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:45478/0 (socket says 172.21.15.116:45478) 2024-04-25T00:07:43.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.821+0000 7f16262b6640 1 -- 172.21.15.116:0/213047748 learned_addr learned my addr 172.21.15.116:0/213047748 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:07:43.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.821+0000 7f16252b4640 1 --2- 172.21.15.116:0/213047748 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f16200715f0 0x7f16201557c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:43.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.821+0000 7f1625ab5640 1 -- 172.21.15.116:0/213047748 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f16200715f0 msgr2=0x7f16201557c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:07:43.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.821+0000 7f1625ab5640 1 --2- 172.21.15.116:0/213047748 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f16200715f0 0x7f16201557c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:43.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.821+0000 7f1625ab5640 1 -- 172.21.15.116:0/213047748 >> v1:172.21.15.116:6789/0 conn(0x7f16200c3f70 legacy=0x7f16200c23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:43.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.821+0000 7f1625ab5640 1 -- 172.21.15.116:0/213047748 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16200c7d70 con 0x7f1620152e40 2024-04-25T00:07:43.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.821+0000 7f1625ab5640 1 --2- 172.21.15.116:0/213047748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1620152e40 0x7f16200710b0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f161c0097c0 tx=0x7f161c01c2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c723519322f72a3f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:43.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.821+0000 7f1624ab3640 1 -- 172.21.15.116:0/213047748 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f161c0b0070 con 0x7f1620152e40 2024-04-25T00:07:43.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.821+0000 7f1624ab3640 1 -- 172.21.15.116:0/213047748 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f161c0ab600 con 0x7f1620152e40 2024-04-25T00:07:43.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.821+0000 7f1624ab3640 1 -- 172.21.15.116:0/213047748 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f161c0ab9d0 con 0x7f1620152e40 2024-04-25T00:07:43.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.821+0000 7f1627d19640 1 -- 172.21.15.116:0/213047748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1620152e40 msgr2=0x7f16200710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:43.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.821+0000 7f1627d19640 1 --2- 172.21.15.116:0/213047748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1620152e40 0x7f16200710b0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f161c0097c0 tx=0x7f161c01c2a0 comp rx=0 tx=0).stop 2024-04-25T00:07:43.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.825+0000 7f1627d19640 1 -- 172.21.15.116:0/213047748 shutdown_connections 2024-04-25T00:07:43.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.825+0000 7f1627d19640 1 --2- 172.21.15.116:0/213047748 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f16200715f0 0x7f16201557c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:43.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.825+0000 7f1627d19640 1 --2- 172.21.15.116:0/213047748 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1620152e40 0x7f16200710b0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:43.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.825+0000 7f1627d19640 1 -- 172.21.15.116:0/213047748 >> 172.21.15.116:0/213047748 conn(0x7f16200c9160 msgr2=0x7f16200c9570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:43.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.825+0000 7f1627d19640 1 -- 172.21.15.116:0/213047748 shutdown_connections 2024-04-25T00:07:43.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.825+0000 7f1627d19640 1 -- 172.21.15.116:0/213047748 wait complete. 2024-04-25T00:07:43.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.825+0000 7f1627d19640 1 Processor -- start 2024-04-25T00:07:43.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.825+0000 7f1627d19640 1 -- start start 2024-04-25T00:07:43.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.825+0000 7f1627d19640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16200715f0 0x7f16200ae900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:43.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.825+0000 7f1627d19640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f16200c3f70 0x7f16201466a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:43.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.825+0000 7f1627d19640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1620152e40 0x7f16200ac220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:43.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.825+0000 7f1627d19640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f16200c5120 con 0x7f16200715f0 2024-04-25T00:07:43.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.825+0000 7f1627d19640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f16200c4920 con 0x7f1620152e40 2024-04-25T00:07:43.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.825+0000 7f1627d19640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f16200c52a0 con 0x7f16200c3f70 2024-04-25T00:07:43.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.825+0000 7f1625ab5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16200715f0 0x7f16200ae900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:43.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.825+0000 7f1625ab5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16200715f0 0x7f16200ae900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42876/0 (socket says 172.21.15.116:42876) 2024-04-25T00:07:43.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.825+0000 7f1625ab5640 1 -- 172.21.15.116:0/3633524597 learned_addr learned my addr 172.21.15.116:0/3633524597 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:07:43.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.825+0000 7f16262b6640 1 --2- 172.21.15.116:0/3633524597 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1620152e40 0x7f16200ac220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:43.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.825+0000 7f16252b4640 1 --2- 172.21.15.116:0/3633524597 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f16200c3f70 0x7f16201466a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:43.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.825+0000 7f1625ab5640 1 -- 172.21.15.116:0/3633524597 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f16200c3f70 msgr2=0x7f16201466a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:43.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.825+0000 7f1625ab5640 1 --2- 172.21.15.116:0/3633524597 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f16200c3f70 0x7f16201466a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:43.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.825+0000 7f1625ab5640 1 -- 172.21.15.116:0/3633524597 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1620152e40 msgr2=0x7f16200ac220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:43.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.825+0000 7f1625ab5640 1 --2- 172.21.15.116:0/3633524597 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1620152e40 0x7f16200ac220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:43.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.825+0000 7f1625ab5640 1 -- 172.21.15.116:0/3633524597 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16200ac760 con 0x7f16200715f0 2024-04-25T00:07:43.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.825+0000 7f16252b4640 1 --2- 172.21.15.116:0/3633524597 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f16200c3f70 0x7f16201466a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:07:43.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.825+0000 7f1625ab5640 1 --2- 172.21.15.116:0/3633524597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16200715f0 0x7f16200ae900 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f161c0ae040 tx=0x7f161c019040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:43.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.825+0000 7f16167fc640 1 -- 172.21.15.116:0/3633524597 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f161c0b8020 con 0x7f16200715f0 2024-04-25T00:07:43.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.825+0000 7f16167fc640 1 -- 172.21.15.116:0/3633524597 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f161c0aa7d0 con 0x7f16200715f0 2024-04-25T00:07:43.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.825+0000 7f1627d19640 1 -- 172.21.15.116:0/3633524597 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16200abf40 con 0x7f16200715f0 2024-04-25T00:07:43.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.825+0000 7f16167fc640 1 -- 172.21.15.116:0/3633524597 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f161c0b0040 con 0x7f16200715f0 2024-04-25T00:07:43.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.825+0000 7f1627d19640 1 -- 172.21.15.116:0/3633524597 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1620156470 con 0x7f16200715f0 2024-04-25T00:07:43.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.829+0000 7f1627d19640 1 -- 172.21.15.116:0/3633524597 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1620068a60 con 0x7f16200715f0 2024-04-25T00:07:43.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.829+0000 7f16167fc640 1 -- 172.21.15.116:0/3633524597 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f161c0bb020 con 0x7f16200715f0 2024-04-25T00:07:43.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.829+0000 7f16167fc640 1 --2- 172.21.15.116:0/3633524597 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f16040678c0 0x7f1604069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:43.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.829+0000 7f16167fc640 1 -- 172.21.15.116:0/3633524597 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7354+0+0 (secure 0 0 0) 0x7f161c1203c0 con 0x7f16200715f0 2024-04-25T00:07:43.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.829+0000 7f16252b4640 1 --2- 172.21.15.116:0/3633524597 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f16040678c0 0x7f1604069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:43.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.829+0000 7f16252b4640 1 --2- 172.21.15.116:0/3633524597 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f16040678c0 0x7f1604069d80 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f1620070fd0 tx=0x7f1610008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:43.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:43.833+0000 7f16167fc640 1 -- 172.21.15.116:0/3633524597 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f161c0c2030 con 0x7f16200715f0 2024-04-25T00:07:44.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.053+0000 7f1627d19640 1 -- 172.21.15.116:0/3633524597 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f162006d180 con 0x7f16200715f0 2024-04-25T00:07:44.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.053+0000 7f16167fc640 1 -- 172.21.15.116:0/3633524597 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) ==== 94+0+6502 (secure 0 0 0) 0x7f16200ac760 con 0x7f16200715f0 2024-04-25T00:07:44.054 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:07:44.054 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5868,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1670061091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1670061091},{"type":"v1","addr":"172.21.15.116:6835","nonce":1670061091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:07:33.575034+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5775":{"gid":5775,"name":"c","rank":0,"incarnation":52,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6833/3006671671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3006671671},{"type":"v1","addr":"172.21.15.116:6833","nonce":3006671671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5775]},"id":1},{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:07:03.468507+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4829":{"gid":4829,"name":"b","rank":0,"incarnation":47,"state":"up:active","state_seq":10,"addr":"172.21.15.174:6837/3583791822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3583791822},{"type":"v1","addr":"172.21.15.174:6837","nonce":3583791822}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4829,"qdb_cluster":[4829]},"id":2}]} 2024-04-25T00:07:44.055 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 56 2024-04-25T00:07:44.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.053+0000 7f1627d19640 1 -- 172.21.15.116:0/3633524597 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f16040678c0 msgr2=0x7f1604069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:44.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.053+0000 7f1627d19640 1 --2- 172.21.15.116:0/3633524597 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f16040678c0 0x7f1604069d80 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f1620070fd0 tx=0x7f1610008040 comp rx=0 tx=0).stop 2024-04-25T00:07:44.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.053+0000 7f1627d19640 1 -- 172.21.15.116:0/3633524597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16200715f0 msgr2=0x7f16200ae900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:44.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.053+0000 7f1627d19640 1 --2- 172.21.15.116:0/3633524597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16200715f0 0x7f16200ae900 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f161c0ae040 tx=0x7f161c019040 comp rx=0 tx=0).stop 2024-04-25T00:07:44.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.053+0000 7f1627d19640 1 -- 172.21.15.116:0/3633524597 shutdown_connections 2024-04-25T00:07:44.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.053+0000 7f1627d19640 1 --2- 172.21.15.116:0/3633524597 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f16040678c0 0x7f1604069d80 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:44.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.053+0000 7f1627d19640 1 --2- 172.21.15.116:0/3633524597 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1620152e40 0x7f16200ac220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:44.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.053+0000 7f1627d19640 1 --2- 172.21.15.116:0/3633524597 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f16200c3f70 0x7f16201466a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:44.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.053+0000 7f1627d19640 1 --2- 172.21.15.116:0/3633524597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16200715f0 0x7f16200ae900 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:44.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.053+0000 7f1627d19640 1 -- 172.21.15.116:0/3633524597 >> 172.21.15.116:0/3633524597 conn(0x7f16200c9160 msgr2=0x7f162014fcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:44.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.053+0000 7f1627d19640 1 -- 172.21.15.116:0/3633524597 shutdown_connections 2024-04-25T00:07:44.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.053+0000 7f1627d19640 1 -- 172.21.15.116:0/3633524597 wait complete. 2024-04-25T00:07:44.075 INFO:tasks.mds_thrash.fs.[b]:kill mds.b (rank=0) 2024-04-25T00:07:44.076 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-25T00:07:44.076 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:07:44.092 INFO:tasks.ceph.mds.b:Stopped 2024-04-25T00:07:44.092 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:07:44.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.201+0000 7fbcb5bc1640 1 Processor -- start 2024-04-25T00:07:44.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.201+0000 7fbcb5bc1640 1 -- start start 2024-04-25T00:07:44.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.201+0000 7fbcb5bc1640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbcb014d7e0 0x7fbcb01540b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:44.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.201+0000 7fbcb5bc1640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fbcb00c3460 0x7fbcb00a94a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:44.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.201+0000 7fbcb5bc1640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcb01545f0 con 0x7fbcb00a99e0 2024-04-25T00:07:44.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.201+0000 7fbcb5bc1640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fbcb00c2c30 con 0x7fbcb014d7e0 2024-04-25T00:07:44.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.201+0000 7fbcb5bc1640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fbcb00702a0 con 0x7fbcb00c3460 2024-04-25T00:07:44.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.201+0000 7fbcaf7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbcb014d7e0 0x7fbcb01540b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:44.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.201+0000 7fbcaffff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fbcb00a99e0 0x7fbcb01572e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:45488/0 (socket says 172.21.15.116:45488) 2024-04-25T00:07:44.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.201+0000 7fbcaffff640 1 -- 172.21.15.116:0/3682831001 learned_addr learned my addr 172.21.15.116:0/3682831001 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:07:44.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.201+0000 7fbcaeffd640 1 --2- 172.21.15.116:0/3682831001 >> v2:172.21.15.174:3300/0 conn(0x7fbcb00c3460 0x7fbcb00a94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:44.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.201+0000 7fbcaf7fe640 1 -- 172.21.15.116:0/3682831001 >> v2:172.21.15.174:3300/0 conn(0x7fbcb00c3460 msgr2=0x7fbcb00a94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:07:44.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.201+0000 7fbcaf7fe640 1 --2- 172.21.15.116:0/3682831001 >> v2:172.21.15.174:3300/0 conn(0x7fbcb00c3460 0x7fbcb00a94a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:44.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.201+0000 7fbcaf7fe640 1 -- 172.21.15.116:0/3682831001 >> v1:172.21.15.116:6789/0 conn(0x7fbcb00a99e0 legacy=0x7fbcb01572e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:44.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.201+0000 7fbcae7fc640 1 -- 172.21.15.116:0/3682831001 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 983166491 0 0) 0x7fbca80014a0 con 0x7fbcb00a99e0 2024-04-25T00:07:44.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.201+0000 7fbcaf7fe640 1 -- 172.21.15.116:0/3682831001 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcb006b210 con 0x7fbcb014d7e0 2024-04-25T00:07:44.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.201+0000 7fbcaeffd640 1 --2- 172.21.15.116:0/3682831001 >> v2:172.21.15.174:3300/0 conn(0x7fbcb00c3460 0x7fbcb00a94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:07:44.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.205+0000 7fbcaf7fe640 1 --2- 172.21.15.116:0/3682831001 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbcb014d7e0 0x7fbcb01540b0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fbcb0073310 tx=0x7fbca001c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77153b624988545e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:44.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.205+0000 7fbcae7fc640 1 -- 172.21.15.116:0/3682831001 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbca00b0070 con 0x7fbcb014d7e0 2024-04-25T00:07:44.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.205+0000 7fbcae7fc640 1 -- 172.21.15.116:0/3682831001 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbca00ab600 con 0x7fbcb014d7e0 2024-04-25T00:07:44.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.205+0000 7fbcae7fc640 1 -- 172.21.15.116:0/3682831001 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbca00ab9d0 con 0x7fbcb014d7e0 2024-04-25T00:07:44.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.205+0000 7fbcb5bc1640 1 -- 172.21.15.116:0/3682831001 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbcb014d7e0 msgr2=0x7fbcb01540b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:44.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.205+0000 7fbcb5bc1640 1 --2- 172.21.15.116:0/3682831001 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbcb014d7e0 0x7fbcb01540b0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fbcb0073310 tx=0x7fbca001c3f0 comp rx=0 tx=0).stop 2024-04-25T00:07:44.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.205+0000 7fbcb5bc1640 1 -- 172.21.15.116:0/3682831001 shutdown_connections 2024-04-25T00:07:44.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.205+0000 7fbcb5bc1640 1 --2- 172.21.15.116:0/3682831001 >> v2:172.21.15.174:3300/0 conn(0x7fbcb00c3460 0x7fbcb00a94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:44.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.205+0000 7fbcb5bc1640 1 --2- 172.21.15.116:0/3682831001 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbcb014d7e0 0x7fbcb01540b0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:44.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.205+0000 7fbcb5bc1640 1 -- 172.21.15.116:0/3682831001 >> 172.21.15.116:0/3682831001 conn(0x7fbcb00c7fa0 msgr2=0x7fbcb00c83b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:44.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.205+0000 7fbcb5bc1640 1 -- 172.21.15.116:0/3682831001 shutdown_connections 2024-04-25T00:07:44.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.205+0000 7fbcb5bc1640 1 -- 172.21.15.116:0/3682831001 wait complete. 2024-04-25T00:07:44.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.205+0000 7fbcb5bc1640 1 Processor -- start 2024-04-25T00:07:44.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.205+0000 7fbcb5bc1640 1 -- start start 2024-04-25T00:07:44.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.205+0000 7fbcb5bc1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbcb00a99e0 0x7fbcb01491b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:44.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.205+0000 7fbcb5bc1640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbcb00c3460 0x7fbcb0144350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:44.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.209+0000 7fbcb5bc1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbcb014d7e0 0x7fbcb0151ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:44.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.209+0000 7fbcb5bc1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbcb01579f0 con 0x7fbcb014d7e0 2024-04-25T00:07:44.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.209+0000 7fbcb5bc1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbcb0156dc0 con 0x7fbcb00a99e0 2024-04-25T00:07:44.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.209+0000 7fbcb5bc1640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fbcb0157b70 con 0x7fbcb00c3460 2024-04-25T00:07:44.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.209+0000 7fbcaffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbcb014d7e0 0x7fbcb0151ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:44.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.209+0000 7fbcaffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbcb014d7e0 0x7fbcb0151ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42898/0 (socket says 172.21.15.116:42898) 2024-04-25T00:07:44.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.209+0000 7fbcaffff640 1 -- 172.21.15.116:0/1265407087 learned_addr learned my addr 172.21.15.116:0/1265407087 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:07:44.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.209+0000 7fbcaeffd640 1 --2- 172.21.15.116:0/1265407087 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbcb00c3460 0x7fbcb0144350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:44.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.209+0000 7fbcaf7fe640 1 --2- 172.21.15.116:0/1265407087 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbcb00a99e0 0x7fbcb01491b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:44.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.209+0000 7fbcaffff640 1 -- 172.21.15.116:0/1265407087 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbcb00c3460 msgr2=0x7fbcb0144350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:44.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.209+0000 7fbcaffff640 1 --2- 172.21.15.116:0/1265407087 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbcb00c3460 0x7fbcb0144350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:44.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.209+0000 7fbcaffff640 1 -- 172.21.15.116:0/1265407087 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbcb00a99e0 msgr2=0x7fbcb01491b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:44.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.209+0000 7fbcaffff640 1 --2- 172.21.15.116:0/1265407087 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbcb00a99e0 0x7fbcb01491b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:44.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.209+0000 7fbcaffff640 1 -- 172.21.15.116:0/1265407087 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcb01f1e00 con 0x7fbcb014d7e0 2024-04-25T00:07:44.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.209+0000 7fbcaf7fe640 1 --2- 172.21.15.116:0/1265407087 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbcb00a99e0 0x7fbcb01491b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:07:44.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.209+0000 7fbcaffff640 1 --2- 172.21.15.116:0/1265407087 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbcb014d7e0 0x7fbcb0151ba0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fbca800c820 tx=0x7fbca800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:44.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.209+0000 7fbc97fff640 1 -- 172.21.15.116:0/1265407087 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbca8018070 con 0x7fbcb014d7e0 2024-04-25T00:07:44.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.209+0000 7fbc97fff640 1 -- 172.21.15.116:0/1265407087 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbca800a640 con 0x7fbcb014d7e0 2024-04-25T00:07:44.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.209+0000 7fbcb5bc1640 1 -- 172.21.15.116:0/1265407087 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcb01f20e0 con 0x7fbcb014d7e0 2024-04-25T00:07:44.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.209+0000 7fbc97fff640 1 -- 172.21.15.116:0/1265407087 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbca800aa30 con 0x7fbcb014d7e0 2024-04-25T00:07:44.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.209+0000 7fbcb5bc1640 1 -- 172.21.15.116:0/1265407087 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcb01f26a0 con 0x7fbcb014d7e0 2024-04-25T00:07:44.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.209+0000 7fbcb5bc1640 1 -- 172.21.15.116:0/1265407087 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbcb0068a60 con 0x7fbcb014d7e0 2024-04-25T00:07:44.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.209+0000 7fbc97fff640 1 -- 172.21.15.116:0/1265407087 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbca801b070 con 0x7fbcb014d7e0 2024-04-25T00:07:44.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.209+0000 7fbc97fff640 1 --2- 172.21.15.116:0/1265407087 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbc9c0678f0 0x7fbc9c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:44.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.209+0000 7fbc97fff640 1 -- 172.21.15.116:0/1265407087 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7354+0+0 (secure 0 0 0) 0x7fbca80858f0 con 0x7fbcb014d7e0 2024-04-25T00:07:44.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.209+0000 7fbcaf7fe640 1 --2- 172.21.15.116:0/1265407087 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbc9c0678f0 0x7fbc9c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:44.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.213+0000 7fbcaf7fe640 1 --2- 172.21.15.116:0/1265407087 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbc9c0678f0 0x7fbc9c069db0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fbca00ad000 tx=0x7fbca00a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:44.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.217+0000 7fbc97fff640 1 -- 172.21.15.116:0/1265407087 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fbcb01545f0 con 0x7fbcb014d7e0 2024-04-25T00:07:44.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.457+0000 7fbcb5bc1640 1 -- 172.21.15.116:0/1265407087 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbcb01577d0 con 0x7fbcb014d7e0 2024-04-25T00:07:44.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.461+0000 7fbc97fff640 1 -- 172.21.15.116:0/1265407087 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) ==== 94+0+6502 (secure 0 0 0) 0x7fbca80556e0 con 0x7fbcb014d7e0 2024-04-25T00:07:44.462 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:07:44.462 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5868,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1670061091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1670061091},{"type":"v1","addr":"172.21.15.116:6835","nonce":1670061091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:07:33.575034+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5775":{"gid":5775,"name":"c","rank":0,"incarnation":52,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6833/3006671671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3006671671},{"type":"v1","addr":"172.21.15.116:6833","nonce":3006671671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5775]},"id":1},{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:07:03.468507+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4829":{"gid":4829,"name":"b","rank":0,"incarnation":47,"state":"up:active","state_seq":10,"addr":"172.21.15.174:6837/3583791822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3583791822},{"type":"v1","addr":"172.21.15.174:6837","nonce":3583791822}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4829,"qdb_cluster":[4829]},"id":2}]} 2024-04-25T00:07:44.462 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 56 2024-04-25T00:07:44.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.461+0000 7fbcb5bc1640 1 -- 172.21.15.116:0/1265407087 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbc9c0678f0 msgr2=0x7fbc9c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:44.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.461+0000 7fbcb5bc1640 1 --2- 172.21.15.116:0/1265407087 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbc9c0678f0 0x7fbc9c069db0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fbca00ad000 tx=0x7fbca00a6000 comp rx=0 tx=0).stop 2024-04-25T00:07:44.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.461+0000 7fbcb5bc1640 1 -- 172.21.15.116:0/1265407087 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbcb014d7e0 msgr2=0x7fbcb0151ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:44.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.461+0000 7fbcb5bc1640 1 --2- 172.21.15.116:0/1265407087 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbcb014d7e0 0x7fbcb0151ba0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fbca800c820 tx=0x7fbca800cce0 comp rx=0 tx=0).stop 2024-04-25T00:07:44.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.461+0000 7fbcb5bc1640 1 -- 172.21.15.116:0/1265407087 shutdown_connections 2024-04-25T00:07:44.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.461+0000 7fbcb5bc1640 1 --2- 172.21.15.116:0/1265407087 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbc9c0678f0 0x7fbc9c069db0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:44.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.461+0000 7fbcb5bc1640 1 --2- 172.21.15.116:0/1265407087 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbcb014d7e0 0x7fbcb0151ba0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:44.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.461+0000 7fbcb5bc1640 1 --2- 172.21.15.116:0/1265407087 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbcb00c3460 0x7fbcb0144350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:44.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.461+0000 7fbcb5bc1640 1 --2- 172.21.15.116:0/1265407087 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbcb00a99e0 0x7fbcb01491b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:44.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.461+0000 7fbcb5bc1640 1 -- 172.21.15.116:0/1265407087 >> 172.21.15.116:0/1265407087 conn(0x7fbcb00c7fa0 msgr2=0x7fbcb00c8380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:44.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.461+0000 7fbcb5bc1640 1 -- 172.21.15.116:0/1265407087 shutdown_connections 2024-04-25T00:07:44.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:44.461+0000 7fbcb5bc1640 1 -- 172.21.15.116:0/1265407087 wait complete. 2024-04-25T00:07:44.482 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T00:07:46.483 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:07:46.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.621+0000 7f846f12a640 1 Processor -- start 2024-04-25T00:07:46.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.621+0000 7f846f12a640 1 -- start start 2024-04-25T00:07:46.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.621+0000 7f846f12a640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f846814fc20 0x7f8468152f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:46.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.621+0000 7f846f12a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f846806f6c0 0x7f846806fb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:46.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.621+0000 7f846f12a640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84680aaca0 con 0x7f84680aa100 2024-04-25T00:07:46.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.621+0000 7f846f12a640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f8468073260 con 0x7f846814fc20 2024-04-25T00:07:46.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.621+0000 7f846f12a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f84680a9620 con 0x7f846806f6c0 2024-04-25T00:07:46.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.621+0000 7f846d6c7640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f84680aa100 0x7f84680aa590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:33674/0 (socket says 172.21.15.116:33674) 2024-04-25T00:07:46.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.621+0000 7f846d6c7640 1 -- 172.21.15.116:0/3038875784 learned_addr learned my addr 172.21.15.116:0/3038875784 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:07:46.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.621+0000 7f846cec6640 1 --2- 172.21.15.116:0/3038875784 >> v2:172.21.15.174:3300/0 conn(0x7f846814fc20 0x7f8468152f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:46.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.621+0000 7f8467fff640 1 --2- 172.21.15.116:0/3038875784 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f846806f6c0 0x7f846806fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:46.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.621+0000 7f84677fe640 1 -- 172.21.15.116:0/3038875784 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4147994161 0 0) 0x7f84680aaca0 con 0x7f84680aa100 2024-04-25T00:07:46.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.621+0000 7f84677fe640 1 -- 172.21.15.116:0/3038875784 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8458003530 con 0x7f84680aa100 2024-04-25T00:07:46.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.621+0000 7f84677fe640 1 -- 172.21.15.116:0/3038875784 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2890387406 0 0) 0x7f8458003530 con 0x7f84680aa100 2024-04-25T00:07:46.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.621+0000 7f84677fe640 1 -- 172.21.15.116:0/3038875784 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f846806f6c0 msgr2=0x7f846806fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:07:46.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.621+0000 7f84677fe640 1 --2- 172.21.15.116:0/3038875784 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f846806f6c0 0x7f846806fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:46.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.621+0000 7f84677fe640 1 -- 172.21.15.116:0/3038875784 >> v2:172.21.15.174:3300/0 conn(0x7f846814fc20 msgr2=0x7f8468152f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:07:46.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.621+0000 7f84677fe640 1 --2- 172.21.15.116:0/3038875784 >> v2:172.21.15.174:3300/0 conn(0x7f846814fc20 0x7f8468152f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:46.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.621+0000 7f84677fe640 1 -- 172.21.15.116:0/3038875784 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84680a9900 con 0x7f84680aa100 2024-04-25T00:07:46.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.621+0000 7f84677fe640 1 -- 172.21.15.116:0/3038875784 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f8460001a00 con 0x7f84680aa100 2024-04-25T00:07:46.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.621+0000 7f84677fe640 1 -- 172.21.15.116:0/3038875784 >> v1:172.21.15.116:6789/0 conn(0x7f84680aa100 legacy=0x7f84680aa590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:46.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.621+0000 7f84677fe640 1 --2- 172.21.15.116:0/3038875784 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f846806f6c0 0x7f84580039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:46.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.621+0000 7f84677fe640 1 --2- 172.21.15.116:0/3038875784 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8458003f40 0x7f8458004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:46.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.621+0000 7f84677fe640 1 --2- 172.21.15.116:0/3038875784 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8458004d50 0x7f84580051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:46.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.621+0000 7f84677fe640 1 -- 172.21.15.116:0/3038875784 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8468073260 con 0x7f8458003f40 2024-04-25T00:07:46.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.621+0000 7f84677fe640 1 -- 172.21.15.116:0/3038875784 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f84680a9620 con 0x7f8458004d50 2024-04-25T00:07:46.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.621+0000 7f84677fe640 1 -- 172.21.15.116:0/3038875784 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f845800bab0 con 0x7f846806f6c0 2024-04-25T00:07:46.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.621+0000 7f8467fff640 1 --2- 172.21.15.116:0/3038875784 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8458003f40 0x7f8458004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:46.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.621+0000 7f846d6c7640 1 --2- 172.21.15.116:0/3038875784 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8458004d50 0x7f84580051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:46.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.625+0000 7f8467fff640 1 -- 172.21.15.116:0/3038875784 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f846806f6c0 msgr2=0x7f84580039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:07:46.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.625+0000 7f846cec6640 1 --2- 172.21.15.116:0/3038875784 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f846806f6c0 0x7f84580039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:46.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.625+0000 7f8467fff640 1 --2- 172.21.15.116:0/3038875784 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f846806f6c0 0x7f84580039c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:46.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.625+0000 7f8467fff640 1 -- 172.21.15.116:0/3038875784 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8458004d50 msgr2=0x7f84580051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:07:46.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.625+0000 7f8467fff640 1 --2- 172.21.15.116:0/3038875784 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8458004d50 0x7f84580051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:46.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.625+0000 7f8467fff640 1 -- 172.21.15.116:0/3038875784 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8458003530 con 0x7f8458003f40 2024-04-25T00:07:46.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.625+0000 7f846d6c7640 1 --2- 172.21.15.116:0/3038875784 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8458004d50 0x7f84580051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:07:46.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.625+0000 7f8467fff640 1 --2- 172.21.15.116:0/3038875784 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8458003f40 0x7f8458004810 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f8454009ae0 tx=0x7f845401ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e24468686937204a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:46.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.625+0000 7f84677fe640 1 -- 172.21.15.116:0/3038875784 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84540096b0 con 0x7f8458003f40 2024-04-25T00:07:46.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.625+0000 7f84677fe640 1 -- 172.21.15.116:0/3038875784 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84540a6450 con 0x7f8458003f40 2024-04-25T00:07:46.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.625+0000 7f84677fe640 1 -- 172.21.15.116:0/3038875784 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84540a6840 con 0x7f8458003f40 2024-04-25T00:07:46.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.625+0000 7f846f12a640 1 -- 172.21.15.116:0/3038875784 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8458003f40 msgr2=0x7f8458004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:46.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.625+0000 7f846f12a640 1 --2- 172.21.15.116:0/3038875784 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8458003f40 0x7f8458004810 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f8454009ae0 tx=0x7f845401ca70 comp rx=0 tx=0).stop 2024-04-25T00:07:46.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.625+0000 7f846d6c7640 1 -- 172.21.15.116:0/3038875784 reap_dead start 2024-04-25T00:07:46.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.625+0000 7f846f12a640 1 -- 172.21.15.116:0/3038875784 shutdown_connections 2024-04-25T00:07:46.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.625+0000 7f846f12a640 1 -- 172.21.15.116:0/3038875784 >> 172.21.15.116:0/3038875784 conn(0x7f84680c4c10 msgr2=0x7f84680c5020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:46.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.625+0000 7f846f12a640 1 -- 172.21.15.116:0/3038875784 shutdown_connections 2024-04-25T00:07:46.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.625+0000 7f846f12a640 1 -- 172.21.15.116:0/3038875784 wait complete. 2024-04-25T00:07:46.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.629+0000 7f846f12a640 1 Processor -- start 2024-04-25T00:07:46.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.629+0000 7f846f12a640 1 -- start start 2024-04-25T00:07:46.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.629+0000 7f846f12a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8458003f40 0x7f84680c3020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:46.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.629+0000 7f846f12a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f84681ef610 0x7f8468151ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:46.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.629+0000 7f846f12a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8468152220 0x7f8468144a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:46.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.629+0000 7f846f12a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8468073000 con 0x7f8468152220 2024-04-25T00:07:46.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.629+0000 7f846f12a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8468072e80 con 0x7f84681ef610 2024-04-25T00:07:46.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.629+0000 7f846f12a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f846815d5d0 con 0x7f8458003f40 2024-04-25T00:07:46.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.629+0000 7f846d6c7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8468152220 0x7f8468144a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:46.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.629+0000 7f846d6c7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8468152220 0x7f8468144a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45686/0 (socket says 172.21.15.116:45686) 2024-04-25T00:07:46.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.629+0000 7f846d6c7640 1 -- 172.21.15.116:0/1220513005 learned_addr learned my addr 172.21.15.116:0/1220513005 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:07:46.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.629+0000 7f846cec6640 1 --2- 172.21.15.116:0/1220513005 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8458003f40 0x7f84680c3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:46.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.629+0000 7f8467fff640 1 --2- 172.21.15.116:0/1220513005 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f84681ef610 0x7f8468151ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:46.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.629+0000 7f846d6c7640 1 -- 172.21.15.116:0/1220513005 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8458003f40 msgr2=0x7f84680c3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:46.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.629+0000 7f846d6c7640 1 --2- 172.21.15.116:0/1220513005 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8458003f40 0x7f84680c3020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:46.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.629+0000 7f846d6c7640 1 -- 172.21.15.116:0/1220513005 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f84681ef610 msgr2=0x7f8468151ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:46.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.629+0000 7f846d6c7640 1 --2- 172.21.15.116:0/1220513005 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f84681ef610 0x7f8468151ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:46.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.629+0000 7f846d6c7640 1 -- 172.21.15.116:0/1220513005 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84681ec240 con 0x7f8468152220 2024-04-25T00:07:46.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.629+0000 7f846d6c7640 1 --2- 172.21.15.116:0/1220513005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8468152220 0x7f8468144a10 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f8468071880 tx=0x7f846000b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:46.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.629+0000 7f84657fa640 1 -- 172.21.15.116:0/1220513005 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8460004d70 con 0x7f8468152220 2024-04-25T00:07:46.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.629+0000 7f846f12a640 1 -- 172.21.15.116:0/1220513005 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84681ec520 con 0x7f8468152220 2024-04-25T00:07:46.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.629+0000 7f84657fa640 1 -- 172.21.15.116:0/1220513005 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f846000bb80 con 0x7f8468152220 2024-04-25T00:07:46.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.629+0000 7f84657fa640 1 -- 172.21.15.116:0/1220513005 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8460007110 con 0x7f8468152220 2024-04-25T00:07:46.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.629+0000 7f846f12a640 1 -- 172.21.15.116:0/1220513005 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84681ecae0 con 0x7f8468152220 2024-04-25T00:07:46.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.629+0000 7f846f12a640 1 -- 172.21.15.116:0/1220513005 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8468068a60 con 0x7f8468152220 2024-04-25T00:07:46.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.633+0000 7f84657fa640 1 -- 172.21.15.116:0/1220513005 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f846001b020 con 0x7f8468152220 2024-04-25T00:07:46.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.633+0000 7f84657fa640 1 --2- 172.21.15.116:0/1220513005 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f844c0678c0 0x7f844c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:46.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.633+0000 7f84657fa640 1 -- 172.21.15.116:0/1220513005 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7354+0+0 (secure 0 0 0) 0x7f84600864c0 con 0x7f8468152220 2024-04-25T00:07:46.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.633+0000 7f846cec6640 1 --2- 172.21.15.116:0/1220513005 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f844c0678c0 0x7f844c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:46.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.633+0000 7f846cec6640 1 --2- 172.21.15.116:0/1220513005 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f844c0678c0 0x7f844c069d80 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f845c009ab0 tx=0x7f845c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:46.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.637+0000 7f84657fa640 1 -- 172.21.15.116:0/1220513005 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f84600569a0 con 0x7f8468152220 2024-04-25T00:07:46.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.877+0000 7f846f12a640 1 -- 172.21.15.116:0/1220513005 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f84680c3560 con 0x7f8468152220 2024-04-25T00:07:46.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.877+0000 7f84657fa640 1 -- 172.21.15.116:0/1220513005 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) ==== 94+0+6502 (secure 0 0 0) 0x7f84600562a0 con 0x7f8468152220 2024-04-25T00:07:46.880 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:07:46.880 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5868,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1670061091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1670061091},{"type":"v1","addr":"172.21.15.116:6835","nonce":1670061091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:07:33.575034+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5775":{"gid":5775,"name":"c","rank":0,"incarnation":52,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6833/3006671671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3006671671},{"type":"v1","addr":"172.21.15.116:6833","nonce":3006671671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5775]},"id":1},{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:07:03.468507+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4829":{"gid":4829,"name":"b","rank":0,"incarnation":47,"state":"up:active","state_seq":10,"addr":"172.21.15.174:6837/3583791822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3583791822},{"type":"v1","addr":"172.21.15.174:6837","nonce":3583791822}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4829,"qdb_cluster":[4829]},"id":2}]} 2024-04-25T00:07:46.881 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 56 2024-04-25T00:07:46.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.881+0000 7f846f12a640 1 -- 172.21.15.116:0/1220513005 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f844c0678c0 msgr2=0x7f844c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:46.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.881+0000 7f846f12a640 1 --2- 172.21.15.116:0/1220513005 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f844c0678c0 0x7f844c069d80 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f845c009ab0 tx=0x7f845c008040 comp rx=0 tx=0).stop 2024-04-25T00:07:46.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.881+0000 7f846f12a640 1 -- 172.21.15.116:0/1220513005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8468152220 msgr2=0x7f8468144a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:46.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.881+0000 7f846f12a640 1 --2- 172.21.15.116:0/1220513005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8468152220 0x7f8468144a10 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f8468071880 tx=0x7f846000b4a0 comp rx=0 tx=0).stop 2024-04-25T00:07:46.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.881+0000 7f846f12a640 1 -- 172.21.15.116:0/1220513005 shutdown_connections 2024-04-25T00:07:46.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.881+0000 7f846f12a640 1 --2- 172.21.15.116:0/1220513005 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f844c0678c0 0x7f844c069d80 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:46.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.881+0000 7f846f12a640 1 --2- 172.21.15.116:0/1220513005 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8468152220 0x7f8468144a10 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:46.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.881+0000 7f846f12a640 1 --2- 172.21.15.116:0/1220513005 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f84681ef610 0x7f8468151ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:46.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.881+0000 7f846f12a640 1 --2- 172.21.15.116:0/1220513005 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8458003f40 0x7f84680c3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:46.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.881+0000 7f846f12a640 1 -- 172.21.15.116:0/1220513005 >> 172.21.15.116:0/1220513005 conn(0x7f84680c4c10 msgr2=0x7f846814b280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:46.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.881+0000 7f846f12a640 1 -- 172.21.15.116:0/1220513005 shutdown_connections 2024-04-25T00:07:46.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:46.881+0000 7f846f12a640 1 -- 172.21.15.116:0/1220513005 wait complete. 2024-04-25T00:07:46.901 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T00:07:48.902 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:07:49.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.053+0000 7f5c4ddec640 1 Processor -- start 2024-04-25T00:07:49.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.053+0000 7f5c4ddec640 1 -- start start 2024-04-25T00:07:49.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.053+0000 7f5c4ddec640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f5c480c3c30 0x7f5c480c6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:49.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.053+0000 7f5c4ddec640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c4806ff40 con 0x7f5c4806fb30 2024-04-25T00:07:49.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.053+0000 7f5c4ddec640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c4806e8f0 con 0x7f5c48152f60 2024-04-25T00:07:49.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.053+0000 7f5c4ddec640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f5c481595b0 con 0x7f5c480c3c30 2024-04-25T00:07:49.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.053+0000 7f5c47fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f5c4806fb30 0x7f5c4806c220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:33688/0 (socket says 172.21.15.116:33688) 2024-04-25T00:07:49.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.053+0000 7f5c47fff640 1 -- 172.21.15.116:0/1604670688 learned_addr learned my addr 172.21.15.116:0/1604670688 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:07:49.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.053+0000 7f5c467fc640 1 -- 172.21.15.116:0/1604670688 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2933234554 0 0) 0x7f5c4806ff40 con 0x7f5c4806fb30 2024-04-25T00:07:49.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.053+0000 7f5c46ffd640 1 --2- 172.21.15.116:0/1604670688 >> v2:172.21.15.174:3300/0 conn(0x7f5c480c3c30 0x7f5c480c6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:49.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.053+0000 7f5c467fc640 1 -- 172.21.15.116:0/1604670688 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c30003530 con 0x7f5c4806fb30 2024-04-25T00:07:49.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.053+0000 7f5c467fc640 1 -- 172.21.15.116:0/1604670688 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4034645959 0 0) 0x7f5c30003530 con 0x7f5c4806fb30 2024-04-25T00:07:49.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.053+0000 7f5c467fc640 1 -- 172.21.15.116:0/1604670688 >> v2:172.21.15.174:3300/0 conn(0x7f5c480c3c30 msgr2=0x7f5c480c6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:07:49.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.053+0000 7f5c467fc640 1 --2- 172.21.15.116:0/1604670688 >> v2:172.21.15.174:3300/0 conn(0x7f5c480c3c30 0x7f5c480c6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:49.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.053+0000 7f5c467fc640 1 -- 172.21.15.116:0/1604670688 >> v1:172.21.15.174:6789/0 conn(0x7f5c48152f60 legacy=0x7f5c48158ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:49.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.053+0000 7f5c467fc640 1 -- 172.21.15.116:0/1604670688 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c4806ebd0 con 0x7f5c4806fb30 2024-04-25T00:07:49.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.053+0000 7f5c467fc640 1 -- 172.21.15.116:0/1604670688 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f5c40001a20 con 0x7f5c4806fb30 2024-04-25T00:07:49.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.053+0000 7f5c467fc640 1 -- 172.21.15.116:0/1604670688 >> v1:172.21.15.116:6789/0 conn(0x7f5c4806fb30 legacy=0x7f5c4806c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:49.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.053+0000 7f5c467fc640 1 --2- 172.21.15.116:0/1604670688 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5c300039c0 0x7f5c30003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:49.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.053+0000 7f5c467fc640 1 --2- 172.21.15.116:0/1604670688 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5c30004810 0x7f5c30004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:49.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.053+0000 7f5c467fc640 1 --2- 172.21.15.116:0/1604670688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c300051d0 0x7f5c3000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:49.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.053+0000 7f5c467fc640 1 -- 172.21.15.116:0/1604670688 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5c481595b0 con 0x7f5c300051d0 2024-04-25T00:07:49.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.053+0000 7f5c467fc640 1 -- 172.21.15.116:0/1604670688 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5c3000c170 con 0x7f5c30004810 2024-04-25T00:07:49.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.053+0000 7f5c467fc640 1 -- 172.21.15.116:0/1604670688 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f5c3000c370 con 0x7f5c300039c0 2024-04-25T00:07:49.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.053+0000 7f5c47fff640 1 --2- 172.21.15.116:0/1604670688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c300051d0 0x7f5c3000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:49.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.053+0000 7f5c477fe640 1 --2- 172.21.15.116:0/1604670688 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5c300039c0 0x7f5c30003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:49.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.053+0000 7f5c47fff640 1 -- 172.21.15.116:0/1604670688 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5c300039c0 msgr2=0x7f5c30003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:07:49.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.053+0000 7f5c47fff640 1 --2- 172.21.15.116:0/1604670688 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5c300039c0 0x7f5c30003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:49.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.053+0000 7f5c47fff640 1 -- 172.21.15.116:0/1604670688 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5c30004810 msgr2=0x7f5c30004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:07:49.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.053+0000 7f5c47fff640 1 --2- 172.21.15.116:0/1604670688 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5c30004810 0x7f5c30004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:49.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.053+0000 7f5c47fff640 1 -- 172.21.15.116:0/1604670688 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c30003530 con 0x7f5c300051d0 2024-04-25T00:07:49.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.057+0000 7f5c47fff640 1 -- 172.21.15.116:0/1604670688 reap_dead start 2024-04-25T00:07:49.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.057+0000 7f5c47fff640 1 --2- 172.21.15.116:0/1604670688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c300051d0 0x7f5c3000b9d0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f5c4000a830 tx=0x7f5c4001faa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c675a9e9941faf0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:49.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.057+0000 7f5c467fc640 1 -- 172.21.15.116:0/1604670688 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c4000a480 con 0x7f5c300051d0 2024-04-25T00:07:49.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.057+0000 7f5c467fc640 1 -- 172.21.15.116:0/1604670688 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c400aa6f0 con 0x7f5c300051d0 2024-04-25T00:07:49.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.057+0000 7f5c467fc640 1 -- 172.21.15.116:0/1604670688 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c400aaae0 con 0x7f5c300051d0 2024-04-25T00:07:49.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.057+0000 7f5c4ddec640 1 -- 172.21.15.116:0/1604670688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c300051d0 msgr2=0x7f5c3000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:49.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.057+0000 7f5c4ddec640 1 --2- 172.21.15.116:0/1604670688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c300051d0 0x7f5c3000b9d0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f5c4000a830 tx=0x7f5c4001faa0 comp rx=0 tx=0).stop 2024-04-25T00:07:49.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.057+0000 7f5c4ddec640 1 -- 172.21.15.116:0/1604670688 shutdown_connections 2024-04-25T00:07:49.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.057+0000 7f5c4ddec640 1 --2- 172.21.15.116:0/1604670688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c300051d0 0x7f5c3000b9d0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:49.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.057+0000 7f5c4ddec640 1 -- 172.21.15.116:0/1604670688 >> 172.21.15.116:0/1604670688 conn(0x7f5c480c4270 msgr2=0x7f5c480c4680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:49.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.057+0000 7f5c4ddec640 1 -- 172.21.15.116:0/1604670688 shutdown_connections 2024-04-25T00:07:49.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.057+0000 7f5c4ddec640 1 -- 172.21.15.116:0/1604670688 wait complete. 2024-04-25T00:07:49.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.057+0000 7f5c4ddec640 1 Processor -- start 2024-04-25T00:07:49.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.057+0000 7f5c4ddec640 1 -- start start 2024-04-25T00:07:49.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.057+0000 7f5c4ddec640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c300051d0 0x7f5c481ee800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:49.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.057+0000 7f5c4ddec640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5c481eed40 0x7f5c481e2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:49.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.057+0000 7f5c4ddec640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5c481e26a0 0x7f5c4814dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:49.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.057+0000 7f5c4ddec640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5c48153340 con 0x7f5c300051d0 2024-04-25T00:07:49.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.057+0000 7f5c4ddec640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5c4806f610 con 0x7f5c481eed40 2024-04-25T00:07:49.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.057+0000 7f5c4ddec640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f5c48074a70 con 0x7f5c481e26a0 2024-04-25T00:07:49.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.057+0000 7f5c477fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c300051d0 0x7f5c481ee800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:49.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.057+0000 7f5c477fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c300051d0 0x7f5c481ee800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45712/0 (socket says 172.21.15.116:45712) 2024-04-25T00:07:49.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.057+0000 7f5c477fe640 1 -- 172.21.15.116:0/1432613016 learned_addr learned my addr 172.21.15.116:0/1432613016 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:07:49.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.061+0000 7f5c46ffd640 1 --2- 172.21.15.116:0/1432613016 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5c481eed40 0x7f5c481e2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:49.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.061+0000 7f5c47fff640 1 --2- 172.21.15.116:0/1432613016 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5c481e26a0 0x7f5c4814dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:49.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.061+0000 7f5c477fe640 1 -- 172.21.15.116:0/1432613016 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5c481e26a0 msgr2=0x7f5c4814dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:49.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.061+0000 7f5c477fe640 1 --2- 172.21.15.116:0/1432613016 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5c481e26a0 0x7f5c4814dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:49.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.061+0000 7f5c477fe640 1 -- 172.21.15.116:0/1432613016 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5c481eed40 msgr2=0x7f5c481e2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:49.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.061+0000 7f5c477fe640 1 --2- 172.21.15.116:0/1432613016 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5c481eed40 0x7f5c481e2160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:49.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.061+0000 7f5c477fe640 1 -- 172.21.15.116:0/1432613016 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c481df050 con 0x7f5c300051d0 2024-04-25T00:07:49.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.061+0000 7f5c477fe640 1 --2- 172.21.15.116:0/1432613016 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c300051d0 0x7f5c481ee800 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f5c3800e4b0 tx=0x7f5c3800e970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:49.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.061+0000 7f5c2ffff640 1 -- 172.21.15.116:0/1432613016 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c380052e0 con 0x7f5c300051d0 2024-04-25T00:07:49.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.061+0000 7f5c2ffff640 1 -- 172.21.15.116:0/1432613016 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c38018070 con 0x7f5c300051d0 2024-04-25T00:07:49.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.061+0000 7f5c2ffff640 1 -- 172.21.15.116:0/1432613016 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c38005480 con 0x7f5c300051d0 2024-04-25T00:07:49.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.061+0000 7f5c4ddec640 1 -- 172.21.15.116:0/1432613016 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c481df330 con 0x7f5c300051d0 2024-04-25T00:07:49.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.061+0000 7f5c4ddec640 1 -- 172.21.15.116:0/1432613016 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c481ec540 con 0x7f5c300051d0 2024-04-25T00:07:49.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.061+0000 7f5c4ddec640 1 -- 172.21.15.116:0/1432613016 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c48068a60 con 0x7f5c300051d0 2024-04-25T00:07:49.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.061+0000 7f5c2ffff640 1 -- 172.21.15.116:0/1432613016 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5c38003840 con 0x7f5c300051d0 2024-04-25T00:07:49.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.061+0000 7f5c2ffff640 1 --2- 172.21.15.116:0/1432613016 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5c340678c0 0x7f5c34069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:49.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.061+0000 7f5c2ffff640 1 -- 172.21.15.116:0/1432613016 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7354+0+0 (secure 0 0 0) 0x7f5c38085f00 con 0x7f5c300051d0 2024-04-25T00:07:49.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.061+0000 7f5c46ffd640 1 --2- 172.21.15.116:0/1432613016 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5c340678c0 0x7f5c34069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:49.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.065+0000 7f5c46ffd640 1 --2- 172.21.15.116:0/1432613016 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5c340678c0 0x7f5c34069d80 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c0047c0 tx=0x7f5c3c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:49.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.069+0000 7f5c2ffff640 1 -- 172.21.15.116:0/1432613016 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f5c38056460 con 0x7f5c300051d0 2024-04-25T00:07:49.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.297+0000 7f5c4ddec640 1 -- 172.21.15.116:0/1432613016 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5c48150df0 con 0x7f5c300051d0 2024-04-25T00:07:49.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.297+0000 7f5c2ffff640 1 -- 172.21.15.116:0/1432613016 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) ==== 94+0+6502 (secure 0 0 0) 0x7f5c38055d60 con 0x7f5c300051d0 2024-04-25T00:07:49.299 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:07:49.299 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5868,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1670061091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1670061091},{"type":"v1","addr":"172.21.15.116:6835","nonce":1670061091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:07:33.575034+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5775":{"gid":5775,"name":"c","rank":0,"incarnation":52,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6833/3006671671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3006671671},{"type":"v1","addr":"172.21.15.116:6833","nonce":3006671671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5775]},"id":1},{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:07:03.468507+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4829":{"gid":4829,"name":"b","rank":0,"incarnation":47,"state":"up:active","state_seq":10,"addr":"172.21.15.174:6837/3583791822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3583791822},{"type":"v1","addr":"172.21.15.174:6837","nonce":3583791822}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4829,"qdb_cluster":[4829]},"id":2}]} 2024-04-25T00:07:49.300 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 56 2024-04-25T00:07:49.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.301+0000 7f5c4ddec640 1 -- 172.21.15.116:0/1432613016 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5c340678c0 msgr2=0x7f5c34069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:49.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.301+0000 7f5c4ddec640 1 --2- 172.21.15.116:0/1432613016 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5c340678c0 0x7f5c34069d80 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c0047c0 tx=0x7f5c3c008040 comp rx=0 tx=0).stop 2024-04-25T00:07:49.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.301+0000 7f5c4ddec640 1 -- 172.21.15.116:0/1432613016 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c300051d0 msgr2=0x7f5c481ee800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:49.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.301+0000 7f5c4ddec640 1 --2- 172.21.15.116:0/1432613016 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c300051d0 0x7f5c481ee800 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f5c3800e4b0 tx=0x7f5c3800e970 comp rx=0 tx=0).stop 2024-04-25T00:07:49.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.301+0000 7f5c4ddec640 1 -- 172.21.15.116:0/1432613016 shutdown_connections 2024-04-25T00:07:49.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.301+0000 7f5c4ddec640 1 --2- 172.21.15.116:0/1432613016 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5c340678c0 0x7f5c34069d80 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:49.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.301+0000 7f5c4ddec640 1 --2- 172.21.15.116:0/1432613016 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5c481e26a0 0x7f5c4814dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:49.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.301+0000 7f5c4ddec640 1 --2- 172.21.15.116:0/1432613016 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5c481eed40 0x7f5c481e2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:49.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.301+0000 7f5c4ddec640 1 --2- 172.21.15.116:0/1432613016 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c300051d0 0x7f5c481ee800 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:49.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.301+0000 7f5c4ddec640 1 -- 172.21.15.116:0/1432613016 >> 172.21.15.116:0/1432613016 conn(0x7f5c480c4270 msgr2=0x7f5c48151750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:49.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.301+0000 7f5c4ddec640 1 -- 172.21.15.116:0/1432613016 shutdown_connections 2024-04-25T00:07:49.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:49.301+0000 7f5c4ddec640 1 -- 172.21.15.116:0/1432613016 wait complete. 2024-04-25T00:07:49.319 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T00:07:51.319 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:07:51.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.441+0000 7f81f85af640 1 Processor -- start 2024-04-25T00:07:51.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.441+0000 7f81f85af640 1 -- start start 2024-04-25T00:07:51.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.441+0000 7f81f85af640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f81f0150ba0 0x7f81f00c5010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:51.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.441+0000 7f81f85af640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f81f0072820 0x7f81f0153770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:51.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.441+0000 7f81f85af640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f81f0072c60 0x7f81f0153cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:51.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.441+0000 7f81f85af640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f81f0157c30 con 0x7f81f0072820 2024-04-25T00:07:51.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.441+0000 7f81f85af640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f81f00c2cb0 con 0x7f81f0150ba0 2024-04-25T00:07:51.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.441+0000 7f81f85af640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f81f0157670 con 0x7f81f0072c60 2024-04-25T00:07:51.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.441+0000 7f81f5b4a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f81f0072820 0x7f81f0153770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:51.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.441+0000 7f81f5b4a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f81f0072820 0x7f81f0153770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45720/0 (socket says 172.21.15.116:45720) 2024-04-25T00:07:51.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.441+0000 7f81f5b4a640 1 -- 172.21.15.116:0/479014676 learned_addr learned my addr 172.21.15.116:0/479014676 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:07:51.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.441+0000 7f81f634b640 1 --2- 172.21.15.116:0/479014676 >> v2:172.21.15.174:3300/0 conn(0x7f81f0150ba0 0x7f81f00c5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:51.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.441+0000 7f81f6b4c640 1 --2- 172.21.15.116:0/479014676 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f81f0072c60 0x7f81f0153cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:51.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.441+0000 7f81f5b4a640 1 -- 172.21.15.116:0/479014676 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f81f0072c60 msgr2=0x7f81f0153cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:07:51.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.441+0000 7f81f5b4a640 1 --2- 172.21.15.116:0/479014676 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f81f0072c60 0x7f81f0153cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:51.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.441+0000 7f81f5b4a640 1 -- 172.21.15.116:0/479014676 >> v2:172.21.15.174:3300/0 conn(0x7f81f0150ba0 msgr2=0x7f81f00c5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:07:51.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.441+0000 7f81f5b4a640 1 --2- 172.21.15.116:0/479014676 >> v2:172.21.15.174:3300/0 conn(0x7f81f0150ba0 0x7f81f00c5010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:51.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.441+0000 7f81f5b4a640 1 -- 172.21.15.116:0/479014676 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81f0157950 con 0x7f81f0072820 2024-04-25T00:07:51.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.441+0000 7f81f634b640 1 --2- 172.21.15.116:0/479014676 >> v2:172.21.15.174:3300/0 conn(0x7f81f0150ba0 0x7f81f00c5010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:07:51.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.441+0000 7f81f6b4c640 1 --2- 172.21.15.116:0/479014676 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f81f0072c60 0x7f81f0153cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:07:51.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.445+0000 7f81f5b4a640 1 --2- 172.21.15.116:0/479014676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81f0072820 0x7f81f0153770 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f81f006c2b0 tx=0x7f81e001c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a62fd54083f65dcd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:51.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.445+0000 7f81f5349640 1 -- 172.21.15.116:0/479014676 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81e00b2070 con 0x7f81f0072820 2024-04-25T00:07:51.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.445+0000 7f81f5349640 1 -- 172.21.15.116:0/479014676 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81e00a9070 con 0x7f81f0072820 2024-04-25T00:07:51.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.445+0000 7f81f5349640 1 -- 172.21.15.116:0/479014676 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81e00ad880 con 0x7f81f0072820 2024-04-25T00:07:51.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.445+0000 7f81f85af640 1 -- 172.21.15.116:0/479014676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81f0072820 msgr2=0x7f81f0153770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:51.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.445+0000 7f81f85af640 1 --2- 172.21.15.116:0/479014676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81f0072820 0x7f81f0153770 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f81f006c2b0 tx=0x7f81e001c490 comp rx=0 tx=0).stop 2024-04-25T00:07:51.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.445+0000 7f81f85af640 1 -- 172.21.15.116:0/479014676 shutdown_connections 2024-04-25T00:07:51.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.445+0000 7f81f85af640 1 --2- 172.21.15.116:0/479014676 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f81f0072c60 0x7f81f0153cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:51.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.445+0000 7f81f85af640 1 --2- 172.21.15.116:0/479014676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81f0072820 0x7f81f0153770 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:51.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.445+0000 7f81f85af640 1 --2- 172.21.15.116:0/479014676 >> v2:172.21.15.174:3300/0 conn(0x7f81f0150ba0 0x7f81f00c5010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:51.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.445+0000 7f81f85af640 1 -- 172.21.15.116:0/479014676 >> 172.21.15.116:0/479014676 conn(0x7f81f006c720 msgr2=0x7f81f006cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:51.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.445+0000 7f81f85af640 1 -- 172.21.15.116:0/479014676 shutdown_connections 2024-04-25T00:07:51.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.445+0000 7f81f85af640 1 -- 172.21.15.116:0/479014676 wait complete. 2024-04-25T00:07:51.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.449+0000 7f81f85af640 1 Processor -- start 2024-04-25T00:07:51.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.449+0000 7f81f85af640 1 -- start start 2024-04-25T00:07:51.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.449+0000 7f81f85af640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f81f0072820 0x7f81f014a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:51.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.449+0000 7f81f85af640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81f0072c60 0x7f81f0151910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:51.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.449+0000 7f81f85af640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81f0150ba0 0x7f81f0152ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:51.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.449+0000 7f81f85af640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f81f006bdf0 con 0x7f81f0072c60 2024-04-25T00:07:51.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.449+0000 7f81f85af640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f81f006bc70 con 0x7f81f0150ba0 2024-04-25T00:07:51.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.449+0000 7f81f85af640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f81f006bf70 con 0x7f81f0072820 2024-04-25T00:07:51.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.449+0000 7f81f5b4a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81f0072c60 0x7f81f0151910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:51.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.449+0000 7f81f5b4a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81f0072c60 0x7f81f0151910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45736/0 (socket says 172.21.15.116:45736) 2024-04-25T00:07:51.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.449+0000 7f81f5b4a640 1 -- 172.21.15.116:0/3902198648 learned_addr learned my addr 172.21.15.116:0/3902198648 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:07:51.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.449+0000 7f81f634b640 1 --2- 172.21.15.116:0/3902198648 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f81f0072820 0x7f81f014a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:51.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.449+0000 7f81f6b4c640 1 --2- 172.21.15.116:0/3902198648 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81f0150ba0 0x7f81f0152ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:51.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.449+0000 7f81f5b4a640 1 -- 172.21.15.116:0/3902198648 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f81f0072820 msgr2=0x7f81f014a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:51.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.449+0000 7f81f5b4a640 1 --2- 172.21.15.116:0/3902198648 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f81f0072820 0x7f81f014a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:51.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.449+0000 7f81f5b4a640 1 -- 172.21.15.116:0/3902198648 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81f0150ba0 msgr2=0x7f81f0152ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:51.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.449+0000 7f81f5b4a640 1 --2- 172.21.15.116:0/3902198648 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81f0150ba0 0x7f81f0152ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:51.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.449+0000 7f81f5b4a640 1 -- 172.21.15.116:0/3902198648 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81f0152af0 con 0x7f81f0072c60 2024-04-25T00:07:51.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.449+0000 7f81f6b4c640 1 --2- 172.21.15.116:0/3902198648 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81f0150ba0 0x7f81f0152ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:07:51.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.449+0000 7f81f5b4a640 1 --2- 172.21.15.116:0/3902198648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81f0072c60 0x7f81f0151910 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f81e001ab40 tx=0x7f81e0019230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:51.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.449+0000 7f81e6ffd640 1 -- 172.21.15.116:0/3902198648 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81e00b2040 con 0x7f81f0072c60 2024-04-25T00:07:51.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.449+0000 7f81e6ffd640 1 -- 172.21.15.116:0/3902198648 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81e00b6620 con 0x7f81f0072c60 2024-04-25T00:07:51.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.449+0000 7f81f85af640 1 -- 172.21.15.116:0/3902198648 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81f0148170 con 0x7f81f0072c60 2024-04-25T00:07:51.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.449+0000 7f81e6ffd640 1 -- 172.21.15.116:0/3902198648 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81e00b6a10 con 0x7f81f0072c60 2024-04-25T00:07:51.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.449+0000 7f81f85af640 1 -- 172.21.15.116:0/3902198648 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81f0147bb0 con 0x7f81f0072c60 2024-04-25T00:07:51.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.449+0000 7f81f85af640 1 -- 172.21.15.116:0/3902198648 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81f0152cc0 con 0x7f81f0072c60 2024-04-25T00:07:51.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.453+0000 7f81e6ffd640 1 -- 172.21.15.116:0/3902198648 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f81e00ab070 con 0x7f81f0072c60 2024-04-25T00:07:51.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.453+0000 7f81e6ffd640 1 --2- 172.21.15.116:0/3902198648 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f81d40678c0 0x7f81d4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:51.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.453+0000 7f81e6ffd640 1 -- 172.21.15.116:0/3902198648 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7354+0+0 (secure 0 0 0) 0x7f81e011ff60 con 0x7f81f0072c60 2024-04-25T00:07:51.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.453+0000 7f81f634b640 1 --2- 172.21.15.116:0/3902198648 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f81d40678c0 0x7f81d4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:51.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.453+0000 7f81f634b640 1 --2- 172.21.15.116:0/3902198648 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f81d40678c0 0x7f81d4069d80 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f81ec00a9a0 tx=0x7f81ec008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:51.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.457+0000 7f81e6ffd640 1 -- 172.21.15.116:0/3902198648 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f81e0016110 con 0x7f81f0072c60 2024-04-25T00:07:51.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.713+0000 7f81f85af640 1 -- 172.21.15.116:0/3902198648 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f81f0068a60 con 0x7f81f0072c60 2024-04-25T00:07:51.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.713+0000 7f81e6ffd640 1 -- 172.21.15.116:0/3902198648 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) ==== 94+0+6502 (secure 0 0 0) 0x7f81e00bd300 con 0x7f81f0072c60 2024-04-25T00:07:51.716 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:07:51.716 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5868,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1670061091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1670061091},{"type":"v1","addr":"172.21.15.116:6835","nonce":1670061091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:07:33.575034+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5775":{"gid":5775,"name":"c","rank":0,"incarnation":52,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6833/3006671671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3006671671},{"type":"v1","addr":"172.21.15.116:6833","nonce":3006671671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5775]},"id":1},{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:07:03.468507+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4829":{"gid":4829,"name":"b","rank":0,"incarnation":47,"state":"up:active","state_seq":10,"addr":"172.21.15.174:6837/3583791822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3583791822},{"type":"v1","addr":"172.21.15.174:6837","nonce":3583791822}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4829,"qdb_cluster":[4829]},"id":2}]} 2024-04-25T00:07:51.717 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 56 2024-04-25T00:07:51.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.717+0000 7f81f85af640 1 -- 172.21.15.116:0/3902198648 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f81d40678c0 msgr2=0x7f81d4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:51.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.717+0000 7f81f85af640 1 --2- 172.21.15.116:0/3902198648 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f81d40678c0 0x7f81d4069d80 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f81ec00a9a0 tx=0x7f81ec008040 comp rx=0 tx=0).stop 2024-04-25T00:07:51.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.717+0000 7f81f85af640 1 -- 172.21.15.116:0/3902198648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81f0072c60 msgr2=0x7f81f0151910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:51.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.717+0000 7f81f85af640 1 --2- 172.21.15.116:0/3902198648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81f0072c60 0x7f81f0151910 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f81e001ab40 tx=0x7f81e0019230 comp rx=0 tx=0).stop 2024-04-25T00:07:51.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.717+0000 7f81f85af640 1 -- 172.21.15.116:0/3902198648 shutdown_connections 2024-04-25T00:07:51.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.717+0000 7f81f85af640 1 --2- 172.21.15.116:0/3902198648 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f81d40678c0 0x7f81d4069d80 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:51.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.717+0000 7f81f85af640 1 --2- 172.21.15.116:0/3902198648 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81f0150ba0 0x7f81f0152ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:51.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.717+0000 7f81f85af640 1 --2- 172.21.15.116:0/3902198648 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81f0072c60 0x7f81f0151910 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:51.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.717+0000 7f81f85af640 1 --2- 172.21.15.116:0/3902198648 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f81f0072820 0x7f81f014a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:51.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.717+0000 7f81f85af640 1 -- 172.21.15.116:0/3902198648 >> 172.21.15.116:0/3902198648 conn(0x7f81f006c720 msgr2=0x7f81f014f5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:51.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.717+0000 7f81f85af640 1 -- 172.21.15.116:0/3902198648 shutdown_connections 2024-04-25T00:07:51.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:51.717+0000 7f81f85af640 1 -- 172.21.15.116:0/3902198648 wait complete. 2024-04-25T00:07:51.736 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T00:07:53.737 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:07:53.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.873+0000 7fcb46933640 1 Processor -- start 2024-04-25T00:07:53.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.873+0000 7fcb46933640 1 -- start start 2024-04-25T00:07:53.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.873+0000 7fcb46933640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fcb40150e50 0x7fcb4006fe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:53.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.873+0000 7fcb46933640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcb40157900 0x7fcb4015e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:53.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.873+0000 7fcb46933640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb40156d10 con 0x7fcb400a9e20 2024-04-25T00:07:53.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.873+0000 7fcb46933640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fcb400c4870 con 0x7fcb40150e50 2024-04-25T00:07:53.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.873+0000 7fcb46933640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fcb400c42b0 con 0x7fcb40157900 2024-04-25T00:07:53.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.873+0000 7fcb3f7fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fcb400a9e20 0x7fcb401571f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:33704/0 (socket says 172.21.15.116:33704) 2024-04-25T00:07:53.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.873+0000 7fcb3f7fe640 1 -- 172.21.15.116:0/3072453604 learned_addr learned my addr 172.21.15.116:0/3072453604 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:07:53.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.873+0000 7fcb3ffff640 1 --2- 172.21.15.116:0/3072453604 >> v2:172.21.15.174:3300/0 conn(0x7fcb40150e50 0x7fcb4006fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:53.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.873+0000 7fcb44ed0640 1 --2- 172.21.15.116:0/3072453604 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcb40157900 0x7fcb4015e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:53.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.873+0000 7fcb3effd640 1 -- 172.21.15.116:0/3072453604 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3550303761 0 0) 0x7fcb40156d10 con 0x7fcb400a9e20 2024-04-25T00:07:53.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.873+0000 7fcb3effd640 1 -- 172.21.15.116:0/3072453604 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcb28003530 con 0x7fcb400a9e20 2024-04-25T00:07:53.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.873+0000 7fcb3effd640 1 -- 172.21.15.116:0/3072453604 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4265371021 0 0) 0x7fcb28003530 con 0x7fcb400a9e20 2024-04-25T00:07:53.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.873+0000 7fcb3effd640 1 -- 172.21.15.116:0/3072453604 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcb40157900 msgr2=0x7fcb4015e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:07:53.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.873+0000 7fcb3effd640 1 --2- 172.21.15.116:0/3072453604 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcb40157900 0x7fcb4015e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:53.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.873+0000 7fcb3effd640 1 -- 172.21.15.116:0/3072453604 >> v2:172.21.15.174:3300/0 conn(0x7fcb40150e50 msgr2=0x7fcb4006fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:07:53.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.873+0000 7fcb3effd640 1 --2- 172.21.15.116:0/3072453604 >> v2:172.21.15.174:3300/0 conn(0x7fcb40150e50 0x7fcb4006fe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:53.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.873+0000 7fcb3effd640 1 -- 172.21.15.116:0/3072453604 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb400c4590 con 0x7fcb400a9e20 2024-04-25T00:07:53.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.873+0000 7fcb3effd640 1 -- 172.21.15.116:0/3072453604 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7fcb340016a0 con 0x7fcb400a9e20 2024-04-25T00:07:53.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.873+0000 7fcb3effd640 1 -- 172.21.15.116:0/3072453604 >> v1:172.21.15.116:6789/0 conn(0x7fcb400a9e20 legacy=0x7fcb401571f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:53.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.873+0000 7fcb3effd640 1 --2- 172.21.15.116:0/3072453604 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb280039c0 0x7fcb28003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:53.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.873+0000 7fcb3effd640 1 --2- 172.21.15.116:0/3072453604 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcb28004810 0x7fcb28004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:53.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.873+0000 7fcb3effd640 1 --2- 172.21.15.116:0/3072453604 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcb40157900 0x7fcb280051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:53.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.873+0000 7fcb3effd640 1 -- 172.21.15.116:0/3072453604 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcb400c4870 con 0x7fcb280039c0 2024-04-25T00:07:53.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.873+0000 7fcb3effd640 1 -- 172.21.15.116:0/3072453604 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcb400c42b0 con 0x7fcb28004810 2024-04-25T00:07:53.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.873+0000 7fcb3effd640 1 -- 172.21.15.116:0/3072453604 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fcb2800b9d0 con 0x7fcb40157900 2024-04-25T00:07:53.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.873+0000 7fcb3ffff640 1 --2- 172.21.15.116:0/3072453604 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb280039c0 0x7fcb28003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:53.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.873+0000 7fcb3f7fe640 1 --2- 172.21.15.116:0/3072453604 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcb28004810 0x7fcb28004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:53.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.877+0000 7fcb3ffff640 1 -- 172.21.15.116:0/3072453604 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcb40157900 msgr2=0x7fcb280051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:07:53.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.877+0000 7fcb44ed0640 1 --2- 172.21.15.116:0/3072453604 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcb40157900 0x7fcb280051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:53.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.877+0000 7fcb3ffff640 1 --2- 172.21.15.116:0/3072453604 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcb40157900 0x7fcb280051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:53.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.877+0000 7fcb3ffff640 1 -- 172.21.15.116:0/3072453604 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcb28004810 msgr2=0x7fcb28004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:07:53.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.877+0000 7fcb3ffff640 1 --2- 172.21.15.116:0/3072453604 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcb28004810 0x7fcb28004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:53.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.877+0000 7fcb3ffff640 1 -- 172.21.15.116:0/3072453604 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb28003530 con 0x7fcb280039c0 2024-04-25T00:07:53.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.877+0000 7fcb3f7fe640 1 --2- 172.21.15.116:0/3072453604 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcb28004810 0x7fcb28004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:07:53.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.877+0000 7fcb44ed0640 1 --2- 172.21.15.116:0/3072453604 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcb40157900 0x7fcb280051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:07:53.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.877+0000 7fcb3ffff640 1 --2- 172.21.15.116:0/3072453604 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb280039c0 0x7fcb28003de0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fcb30009b10 tx=0x7fcb3001cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45e4c10742e7af50 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:53.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.877+0000 7fcb3effd640 1 -- 172.21.15.116:0/3072453604 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb300096b0 con 0x7fcb280039c0 2024-04-25T00:07:53.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.877+0000 7fcb3effd640 1 -- 172.21.15.116:0/3072453604 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb300a8410 con 0x7fcb280039c0 2024-04-25T00:07:53.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.877+0000 7fcb3effd640 1 -- 172.21.15.116:0/3072453604 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb300a8800 con 0x7fcb280039c0 2024-04-25T00:07:53.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.877+0000 7fcb46933640 1 -- 172.21.15.116:0/3072453604 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb280039c0 msgr2=0x7fcb28003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:53.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.877+0000 7fcb46933640 1 --2- 172.21.15.116:0/3072453604 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb280039c0 0x7fcb28003de0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fcb30009b10 tx=0x7fcb3001cab0 comp rx=0 tx=0).stop 2024-04-25T00:07:53.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.877+0000 7fcb44ed0640 1 -- 172.21.15.116:0/3072453604 reap_dead start 2024-04-25T00:07:53.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.877+0000 7fcb46933640 1 -- 172.21.15.116:0/3072453604 shutdown_connections 2024-04-25T00:07:53.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.877+0000 7fcb46933640 1 -- 172.21.15.116:0/3072453604 >> 172.21.15.116:0/3072453604 conn(0x7fcb400c4aa0 msgr2=0x7fcb400c4eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:53.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.877+0000 7fcb46933640 1 -- 172.21.15.116:0/3072453604 shutdown_connections 2024-04-25T00:07:53.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.877+0000 7fcb46933640 1 -- 172.21.15.116:0/3072453604 wait complete. 2024-04-25T00:07:53.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.881+0000 7fcb46933640 1 Processor -- start 2024-04-25T00:07:53.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.881+0000 7fcb46933640 1 -- start start 2024-04-25T00:07:53.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.881+0000 7fcb46933640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb4014ff40 0x7fcb4006e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:53.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.881+0000 7fcb46933640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcb4006dc30 0x7fcb40147d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:53.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.881+0000 7fcb46933640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcb40150320 0x7fcb40149810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:53.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.881+0000 7fcb46933640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcb4015e810 con 0x7fcb4014ff40 2024-04-25T00:07:53.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.881+0000 7fcb46933640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcb4015e990 con 0x7fcb4006dc30 2024-04-25T00:07:53.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.881+0000 7fcb46933640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fcb4015eb10 con 0x7fcb40150320 2024-04-25T00:07:53.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.881+0000 7fcb3ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb4014ff40 0x7fcb4006e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:53.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.881+0000 7fcb3ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb4014ff40 0x7fcb4006e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45746/0 (socket says 172.21.15.116:45746) 2024-04-25T00:07:53.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.881+0000 7fcb3ffff640 1 -- 172.21.15.116:0/3868125972 learned_addr learned my addr 172.21.15.116:0/3868125972 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:07:53.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.881+0000 7fcb3f7fe640 1 --2- 172.21.15.116:0/3868125972 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcb4006dc30 0x7fcb40147d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:53.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.881+0000 7fcb44ed0640 1 --2- 172.21.15.116:0/3868125972 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcb40150320 0x7fcb40149810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:53.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.881+0000 7fcb3ffff640 1 -- 172.21.15.116:0/3868125972 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcb40150320 msgr2=0x7fcb40149810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:53.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.881+0000 7fcb3ffff640 1 --2- 172.21.15.116:0/3868125972 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcb40150320 0x7fcb40149810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:53.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.881+0000 7fcb3ffff640 1 -- 172.21.15.116:0/3868125972 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcb4006dc30 msgr2=0x7fcb40147d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:53.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.881+0000 7fcb3ffff640 1 --2- 172.21.15.116:0/3868125972 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcb4006dc30 0x7fcb40147d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:53.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.881+0000 7fcb3ffff640 1 -- 172.21.15.116:0/3868125972 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb40148250 con 0x7fcb4014ff40 2024-04-25T00:07:53.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.881+0000 7fcb44ed0640 1 --2- 172.21.15.116:0/3868125972 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcb40150320 0x7fcb40149810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:07:53.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.881+0000 7fcb3f7fe640 1 --2- 172.21.15.116:0/3868125972 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcb4006dc30 0x7fcb40147d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:07:53.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.881+0000 7fcb3ffff640 1 --2- 172.21.15.116:0/3868125972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb4014ff40 0x7fcb4006e640 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fcb30009b10 tx=0x7fcb30019850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:53.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.881+0000 7fcb3cff9640 1 -- 172.21.15.116:0/3868125972 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb300a8b40 con 0x7fcb4014ff40 2024-04-25T00:07:53.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.881+0000 7fcb3cff9640 1 -- 172.21.15.116:0/3868125972 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb300a8ce0 con 0x7fcb4014ff40 2024-04-25T00:07:53.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.881+0000 7fcb46933640 1 -- 172.21.15.116:0/3868125972 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb40147a70 con 0x7fcb4014ff40 2024-04-25T00:07:53.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.881+0000 7fcb3cff9640 1 -- 172.21.15.116:0/3868125972 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb300bc690 con 0x7fcb4014ff40 2024-04-25T00:07:53.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.881+0000 7fcb46933640 1 -- 172.21.15.116:0/3868125972 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb401f2ce0 con 0x7fcb4014ff40 2024-04-25T00:07:53.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.885+0000 7fcb46933640 1 -- 172.21.15.116:0/3868125972 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb401f2af0 con 0x7fcb4014ff40 2024-04-25T00:07:53.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.885+0000 7fcb3cff9640 1 -- 172.21.15.116:0/3868125972 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fcb300bc830 con 0x7fcb4014ff40 2024-04-25T00:07:53.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.885+0000 7fcb3cff9640 1 --2- 172.21.15.116:0/3868125972 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcb2c0678c0 0x7fcb2c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:53.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.885+0000 7fcb3cff9640 1 -- 172.21.15.116:0/3868125972 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7354+0+0 (secure 0 0 0) 0x7fcb30120850 con 0x7fcb4014ff40 2024-04-25T00:07:53.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.885+0000 7fcb3f7fe640 1 --2- 172.21.15.116:0/3868125972 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcb2c0678c0 0x7fcb2c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:53.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.885+0000 7fcb3f7fe640 1 --2- 172.21.15.116:0/3868125972 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcb2c0678c0 0x7fcb2c069d80 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fcb4014df40 tx=0x7fcb34009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:53.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:53.889+0000 7fcb3cff9640 1 -- 172.21.15.116:0/3868125972 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fcb40148250 con 0x7fcb4014ff40 2024-04-25T00:07:54.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:54.125+0000 7fcb46933640 1 -- 172.21.15.116:0/3868125972 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcb401490e0 con 0x7fcb4014ff40 2024-04-25T00:07:54.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:54.125+0000 7fcb3cff9640 1 -- 172.21.15.116:0/3868125972 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) ==== 94+0+6502 (secure 0 0 0) 0x7fcb300ac030 con 0x7fcb4014ff40 2024-04-25T00:07:54.126 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:07:54.127 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5868,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1670061091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1670061091},{"type":"v1","addr":"172.21.15.116:6835","nonce":1670061091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:07:33.575034+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5775":{"gid":5775,"name":"c","rank":0,"incarnation":52,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6833/3006671671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3006671671},{"type":"v1","addr":"172.21.15.116:6833","nonce":3006671671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5775]},"id":1},{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:07:03.468507+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4829":{"gid":4829,"name":"b","rank":0,"incarnation":47,"state":"up:active","state_seq":10,"addr":"172.21.15.174:6837/3583791822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3583791822},{"type":"v1","addr":"172.21.15.174:6837","nonce":3583791822}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4829,"qdb_cluster":[4829]},"id":2}]} 2024-04-25T00:07:54.127 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 56 2024-04-25T00:07:54.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:54.125+0000 7fcb46933640 1 -- 172.21.15.116:0/3868125972 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcb2c0678c0 msgr2=0x7fcb2c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:54.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:54.125+0000 7fcb46933640 1 --2- 172.21.15.116:0/3868125972 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcb2c0678c0 0x7fcb2c069d80 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fcb4014df40 tx=0x7fcb34009040 comp rx=0 tx=0).stop 2024-04-25T00:07:54.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:54.125+0000 7fcb46933640 1 -- 172.21.15.116:0/3868125972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb4014ff40 msgr2=0x7fcb4006e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:54.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:54.125+0000 7fcb46933640 1 --2- 172.21.15.116:0/3868125972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb4014ff40 0x7fcb4006e640 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fcb30009b10 tx=0x7fcb30019850 comp rx=0 tx=0).stop 2024-04-25T00:07:54.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:54.125+0000 7fcb46933640 1 -- 172.21.15.116:0/3868125972 shutdown_connections 2024-04-25T00:07:54.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:54.125+0000 7fcb46933640 1 --2- 172.21.15.116:0/3868125972 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcb2c0678c0 0x7fcb2c069d80 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:54.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:54.125+0000 7fcb46933640 1 --2- 172.21.15.116:0/3868125972 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcb40150320 0x7fcb40149810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:54.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:54.125+0000 7fcb46933640 1 --2- 172.21.15.116:0/3868125972 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcb4006dc30 0x7fcb40147d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:54.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:54.125+0000 7fcb46933640 1 --2- 172.21.15.116:0/3868125972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcb4014ff40 0x7fcb4006e640 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:54.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:54.125+0000 7fcb46933640 1 -- 172.21.15.116:0/3868125972 >> 172.21.15.116:0/3868125972 conn(0x7fcb400c4aa0 msgr2=0x7fcb400ad020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:54.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:54.125+0000 7fcb46933640 1 -- 172.21.15.116:0/3868125972 shutdown_connections 2024-04-25T00:07:54.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:54.125+0000 7fcb46933640 1 -- 172.21.15.116:0/3868125972 wait complete. 2024-04-25T00:07:54.147 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T00:07:56.148 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:07:56.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.297+0000 7f180bfe9640 1 Processor -- start 2024-04-25T00:07:56.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.297+0000 7f180bfe9640 1 -- start start 2024-04-25T00:07:56.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.297+0000 7f180bfe9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18040ad180 0x7f18040c83b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:56.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.297+0000 7f180bfe9640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f18040c88f0 0x7f18040c9040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:56.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.297+0000 7f180bfe9640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1804157630 con 0x7f18040adf40 2024-04-25T00:07:56.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.297+0000 7f180bfe9640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f18040ad600 con 0x7f18040ad180 2024-04-25T00:07:56.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.297+0000 7f180bfe9640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f18040c7960 con 0x7f18040c88f0 2024-04-25T00:07:56.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.297+0000 7f1809584640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18040ad180 0x7f18040c83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:56.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.297+0000 7f1809584640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f18040ad180 0x7f18040c83b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47736/0 (socket says 172.21.15.116:47736) 2024-04-25T00:07:56.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.297+0000 7f1809584640 1 -- 172.21.15.116:0/3074563369 learned_addr learned my addr 172.21.15.116:0/3074563369 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:07:56.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.297+0000 7f180a586640 1 --2- 172.21.15.116:0/3074563369 >> v2:172.21.15.174:3300/0 conn(0x7f18040c88f0 0x7f18040c9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:56.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.297+0000 7f1809584640 1 -- 172.21.15.116:0/3074563369 >> v2:172.21.15.174:3300/0 conn(0x7f18040c88f0 msgr2=0x7f18040c9040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:07:56.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.297+0000 7f1809584640 1 --2- 172.21.15.116:0/3074563369 >> v2:172.21.15.174:3300/0 conn(0x7f18040c88f0 0x7f18040c9040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:56.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.297+0000 7f1809584640 1 -- 172.21.15.116:0/3074563369 >> v1:172.21.15.174:6789/0 conn(0x7f18040adf40 legacy=0x7f1804156c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:07:56.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.297+0000 7f1809584640 1 -- 172.21.15.116:0/3074563369 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1804157910 con 0x7f18040ad180 2024-04-25T00:07:56.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.297+0000 7f180a586640 1 --2- 172.21.15.116:0/3074563369 >> v2:172.21.15.174:3300/0 conn(0x7f18040c88f0 0x7f18040c9040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:07:56.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.301+0000 7f1809584640 1 --2- 172.21.15.116:0/3074563369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18040ad180 0x7f18040c83b0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f18041535a0 tx=0x7f17f401c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62f79797ce26060d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:56.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.301+0000 7f1808d83640 1 -- 172.21.15.116:0/3074563369 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f40b0070 con 0x7f18040ad180 2024-04-25T00:07:56.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.301+0000 7f1808d83640 1 -- 172.21.15.116:0/3074563369 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17f40ab600 con 0x7f18040ad180 2024-04-25T00:07:56.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.301+0000 7f1808d83640 1 -- 172.21.15.116:0/3074563369 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f40ab9d0 con 0x7f18040ad180 2024-04-25T00:07:56.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.301+0000 7f180bfe9640 1 -- 172.21.15.116:0/3074563369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18040ad180 msgr2=0x7f18040c83b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:56.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.301+0000 7f180bfe9640 1 --2- 172.21.15.116:0/3074563369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18040ad180 0x7f18040c83b0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f18041535a0 tx=0x7f17f401c4e0 comp rx=0 tx=0).stop 2024-04-25T00:07:56.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.301+0000 7f180bfe9640 1 -- 172.21.15.116:0/3074563369 shutdown_connections 2024-04-25T00:07:56.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.301+0000 7f180bfe9640 1 --2- 172.21.15.116:0/3074563369 >> v2:172.21.15.174:3300/0 conn(0x7f18040c88f0 0x7f18040c9040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:56.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.301+0000 7f180bfe9640 1 --2- 172.21.15.116:0/3074563369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18040ad180 0x7f18040c83b0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:56.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.301+0000 7f180bfe9640 1 -- 172.21.15.116:0/3074563369 >> 172.21.15.116:0/3074563369 conn(0x7f1804074480 msgr2=0x7f1804074890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:56.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.301+0000 7f180bfe9640 1 -- 172.21.15.116:0/3074563369 shutdown_connections 2024-04-25T00:07:56.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.301+0000 7f180bfe9640 1 -- 172.21.15.116:0/3074563369 wait complete. 2024-04-25T00:07:56.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.305+0000 7f180bfe9640 1 Processor -- start 2024-04-25T00:07:56.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.305+0000 7f180bfe9640 1 -- start start 2024-04-25T00:07:56.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.305+0000 7f180bfe9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18040ad180 0x7f18041ec400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:56.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.305+0000 7f180bfe9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f18040adf40 0x7f180414ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:56.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.305+0000 7f180bfe9640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f18040c88f0 0x7f180414ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:56.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.305+0000 7f180bfe9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f18040ae4a0 con 0x7f18040ad180 2024-04-25T00:07:56.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.305+0000 7f180bfe9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f18040ae320 con 0x7f18040adf40 2024-04-25T00:07:56.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.305+0000 7f180bfe9640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f180414d4f0 con 0x7f18040c88f0 2024-04-25T00:07:56.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.305+0000 7f1809d85640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18040ad180 0x7f18041ec400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:56.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.305+0000 7f1809d85640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18040ad180 0x7f18041ec400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47744/0 (socket says 172.21.15.116:47744) 2024-04-25T00:07:56.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.305+0000 7f1809d85640 1 -- 172.21.15.116:0/996040122 learned_addr learned my addr 172.21.15.116:0/996040122 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:07:56.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.305+0000 7f1809584640 1 --2- 172.21.15.116:0/996040122 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f18040adf40 0x7f180414ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:56.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.305+0000 7f180a586640 1 --2- 172.21.15.116:0/996040122 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f18040c88f0 0x7f180414ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:56.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.305+0000 7f1809d85640 1 -- 172.21.15.116:0/996040122 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f18040c88f0 msgr2=0x7f180414ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:56.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.305+0000 7f1809d85640 1 --2- 172.21.15.116:0/996040122 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f18040c88f0 0x7f180414ffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:56.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.305+0000 7f1809d85640 1 -- 172.21.15.116:0/996040122 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f18040adf40 msgr2=0x7f180414ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:56.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.305+0000 7f1809d85640 1 --2- 172.21.15.116:0/996040122 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f18040adf40 0x7f180414ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:56.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.305+0000 7f1809d85640 1 -- 172.21.15.116:0/996040122 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f180414dde0 con 0x7f18040ad180 2024-04-25T00:07:56.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.305+0000 7f180a586640 1 --2- 172.21.15.116:0/996040122 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f18040c88f0 0x7f180414ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:07:56.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.305+0000 7f1809d85640 1 --2- 172.21.15.116:0/996040122 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18040ad180 0x7f18041ec400 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f180000c4c0 tx=0x7f1800001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:56.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.305+0000 7f17fa7fc640 1 -- 172.21.15.116:0/996040122 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1800017070 con 0x7f18040ad180 2024-04-25T00:07:56.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.305+0000 7f17fa7fc640 1 -- 172.21.15.116:0/996040122 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1800003600 con 0x7f18040ad180 2024-04-25T00:07:56.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.305+0000 7f180bfe9640 1 -- 172.21.15.116:0/996040122 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f180414e0c0 con 0x7f18040ad180 2024-04-25T00:07:56.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.305+0000 7f17fa7fc640 1 -- 172.21.15.116:0/996040122 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18000039f0 con 0x7f18040ad180 2024-04-25T00:07:56.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.305+0000 7f180bfe9640 1 -- 172.21.15.116:0/996040122 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f180414e680 con 0x7f18040ad180 2024-04-25T00:07:56.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.305+0000 7f180bfe9640 1 -- 172.21.15.116:0/996040122 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1804068a60 con 0x7f18040ad180 2024-04-25T00:07:56.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.305+0000 7f17fa7fc640 1 -- 172.21.15.116:0/996040122 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f180001a070 con 0x7f18040ad180 2024-04-25T00:07:56.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.309+0000 7f17fa7fc640 1 --2- 172.21.15.116:0/996040122 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f17e80678c0 0x7f17e8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:56.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.309+0000 7f17fa7fc640 1 -- 172.21.15.116:0/996040122 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7354+0+0 (secure 0 0 0) 0x7f1800085690 con 0x7f18040ad180 2024-04-25T00:07:56.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.309+0000 7f1809584640 1 --2- 172.21.15.116:0/996040122 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f17e80678c0 0x7f17e8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:56.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.309+0000 7f1809584640 1 --2- 172.21.15.116:0/996040122 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f17e80678c0 0x7f17e8069d80 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f17f40ad000 tx=0x7f17f4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:56.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.313+0000 7f17fa7fc640 1 -- 172.21.15.116:0/996040122 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f1800055bf0 con 0x7f18040ad180 2024-04-25T00:07:56.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.533+0000 7f180bfe9640 1 -- 172.21.15.116:0/996040122 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1804152a10 con 0x7f18040ad180 2024-04-25T00:07:56.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.537+0000 7f17fa7fc640 1 -- 172.21.15.116:0/996040122 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) ==== 94+0+6502 (secure 0 0 0) 0x7f18000554f0 con 0x7f18040ad180 2024-04-25T00:07:56.538 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:07:56.538 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5868,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1670061091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1670061091},{"type":"v1","addr":"172.21.15.116:6835","nonce":1670061091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:07:33.575034+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5775":{"gid":5775,"name":"c","rank":0,"incarnation":52,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6833/3006671671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3006671671},{"type":"v1","addr":"172.21.15.116:6833","nonce":3006671671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5775]},"id":1},{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:07:03.468507+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4829":{"gid":4829,"name":"b","rank":0,"incarnation":47,"state":"up:active","state_seq":10,"addr":"172.21.15.174:6837/3583791822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3583791822},{"type":"v1","addr":"172.21.15.174:6837","nonce":3583791822}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4829,"qdb_cluster":[4829]},"id":2}]} 2024-04-25T00:07:56.538 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 56 2024-04-25T00:07:56.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.537+0000 7f180bfe9640 1 -- 172.21.15.116:0/996040122 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f17e80678c0 msgr2=0x7f17e8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:56.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.537+0000 7f180bfe9640 1 --2- 172.21.15.116:0/996040122 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f17e80678c0 0x7f17e8069d80 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f17f40ad000 tx=0x7f17f4008040 comp rx=0 tx=0).stop 2024-04-25T00:07:56.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.537+0000 7f180bfe9640 1 -- 172.21.15.116:0/996040122 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18040ad180 msgr2=0x7f18041ec400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:56.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.537+0000 7f180bfe9640 1 --2- 172.21.15.116:0/996040122 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18040ad180 0x7f18041ec400 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f180000c4c0 tx=0x7f1800001e80 comp rx=0 tx=0).stop 2024-04-25T00:07:56.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.537+0000 7f180bfe9640 1 -- 172.21.15.116:0/996040122 shutdown_connections 2024-04-25T00:07:56.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.537+0000 7f180bfe9640 1 --2- 172.21.15.116:0/996040122 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f17e80678c0 0x7f17e8069d80 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:56.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.537+0000 7f180bfe9640 1 --2- 172.21.15.116:0/996040122 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f18040c88f0 0x7f180414ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:56.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.537+0000 7f180bfe9640 1 --2- 172.21.15.116:0/996040122 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f18040adf40 0x7f180414ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:56.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.537+0000 7f180bfe9640 1 --2- 172.21.15.116:0/996040122 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18040ad180 0x7f18041ec400 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:56.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.537+0000 7f180bfe9640 1 -- 172.21.15.116:0/996040122 >> 172.21.15.116:0/996040122 conn(0x7f1804074480 msgr2=0x7f1804151ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:56.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.537+0000 7f180bfe9640 1 -- 172.21.15.116:0/996040122 shutdown_connections 2024-04-25T00:07:56.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:56.537+0000 7f180bfe9640 1 -- 172.21.15.116:0/996040122 wait complete. 2024-04-25T00:07:56.561 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T00:07:58.562 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:07:58.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.689+0000 7efccb0f1640 1 Processor -- start 2024-04-25T00:07:58.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.689+0000 7efccb0f1640 1 -- start start 2024-04-25T00:07:58.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.689+0000 7efccb0f1640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7efcc4150bf0 0x7efcc406c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:58.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.693+0000 7efccb0f1640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efcc406c760 0x7efcc406f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:58.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.693+0000 7efccb0f1640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efcc4153aa0 con 0x7efcc4153570 2024-04-25T00:07:58.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.693+0000 7efccb0f1640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7efcc406e630 con 0x7efcc4150bf0 2024-04-25T00:07:58.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.693+0000 7efccb0f1640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7efcc4150fd0 con 0x7efcc406c760 2024-04-25T00:07:58.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.693+0000 7efcc8e8d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7efcc4150bf0 0x7efcc406c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:58.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.693+0000 7efcc3fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7efcc4153570 0x7efcc406eac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:53770/0 (socket says 172.21.15.116:53770) 2024-04-25T00:07:58.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.693+0000 7efcc3fff640 1 -- 172.21.15.116:0/3823175864 learned_addr learned my addr 172.21.15.116:0/3823175864 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:07:58.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.693+0000 7efcc968e640 1 --2- 172.21.15.116:0/3823175864 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efcc406c760 0x7efcc406f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:58.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.693+0000 7efcc8e8d640 1 -- 172.21.15.116:0/3823175864 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efcc406c760 msgr2=0x7efcc406f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:07:58.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.693+0000 7efcc8e8d640 1 --2- 172.21.15.116:0/3823175864 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efcc406c760 0x7efcc406f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:58.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.693+0000 7efcc8e8d640 1 -- 172.21.15.116:0/3823175864 >> v1:172.21.15.116:6789/0 conn(0x7efcc4153570 legacy=0x7efcc406eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:58.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.693+0000 7efcc8e8d640 1 -- 172.21.15.116:0/3823175864 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcc4153d80 con 0x7efcc4150bf0 2024-04-25T00:07:58.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.693+0000 7efcc8e8d640 1 --2- 172.21.15.116:0/3823175864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efcc4150bf0 0x7efcc406c220 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7efcb80097c0 tx=0x7efcb801c270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbba9ac8c1296d73 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:58.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.693+0000 7efcc37fe640 1 -- 172.21.15.116:0/3823175864 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb80b0070 con 0x7efcc4150bf0 2024-04-25T00:07:58.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.693+0000 7efcc37fe640 1 -- 172.21.15.116:0/3823175864 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcb80ab600 con 0x7efcc4150bf0 2024-04-25T00:07:58.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.693+0000 7efcc37fe640 1 -- 172.21.15.116:0/3823175864 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb80ab9d0 con 0x7efcc4150bf0 2024-04-25T00:07:58.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.693+0000 7efccb0f1640 1 -- 172.21.15.116:0/3823175864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efcc4150bf0 msgr2=0x7efcc406c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:58.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.693+0000 7efccb0f1640 1 --2- 172.21.15.116:0/3823175864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efcc4150bf0 0x7efcc406c220 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7efcb80097c0 tx=0x7efcb801c270 comp rx=0 tx=0).stop 2024-04-25T00:07:58.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.693+0000 7efccb0f1640 1 -- 172.21.15.116:0/3823175864 shutdown_connections 2024-04-25T00:07:58.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.693+0000 7efccb0f1640 1 --2- 172.21.15.116:0/3823175864 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efcc406c760 0x7efcc406f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:58.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.693+0000 7efccb0f1640 1 --2- 172.21.15.116:0/3823175864 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efcc4150bf0 0x7efcc406c220 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:58.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.693+0000 7efccb0f1640 1 -- 172.21.15.116:0/3823175864 >> 172.21.15.116:0/3823175864 conn(0x7efcc40c4270 msgr2=0x7efcc40c4680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:58.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.693+0000 7efccb0f1640 1 -- 172.21.15.116:0/3823175864 shutdown_connections 2024-04-25T00:07:58.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.693+0000 7efccb0f1640 1 -- 172.21.15.116:0/3823175864 wait complete. 2024-04-25T00:07:58.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.697+0000 7efccb0f1640 1 Processor -- start 2024-04-25T00:07:58.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.697+0000 7efccb0f1640 1 -- start start 2024-04-25T00:07:58.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.697+0000 7efccb0f1640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efcc406c760 0x7efcc40ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:58.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.697+0000 7efccb0f1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efcc4150bf0 0x7efcc4156cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:58.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.697+0000 7efccb0f1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efcc4153570 0x7efcc414f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:58.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.697+0000 7efccb0f1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7efcc40a72b0 con 0x7efcc4150bf0 2024-04-25T00:07:58.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.697+0000 7efccb0f1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7efcc40a7130 con 0x7efcc4153570 2024-04-25T00:07:58.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.697+0000 7efccb0f1640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7efcc4160710 con 0x7efcc406c760 2024-04-25T00:07:58.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.697+0000 7efcc3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efcc4150bf0 0x7efcc4156cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:58.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.697+0000 7efcc3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efcc4150bf0 0x7efcc4156cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47752/0 (socket says 172.21.15.116:47752) 2024-04-25T00:07:58.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.697+0000 7efcc3fff640 1 -- 172.21.15.116:0/556581751 learned_addr learned my addr 172.21.15.116:0/556581751 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:07:58.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.697+0000 7efcc8e8d640 1 --2- 172.21.15.116:0/556581751 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efcc406c760 0x7efcc40ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:58.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.697+0000 7efcc3fff640 1 -- 172.21.15.116:0/556581751 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efcc406c760 msgr2=0x7efcc40ac8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:58.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.697+0000 7efcc3fff640 1 --2- 172.21.15.116:0/556581751 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efcc406c760 0x7efcc40ac8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:58.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.697+0000 7efcc3fff640 1 -- 172.21.15.116:0/556581751 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efcc4153570 msgr2=0x7efcc414f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:58.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.697+0000 7efcc968e640 1 --2- 172.21.15.116:0/556581751 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efcc4153570 0x7efcc414f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:58.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.697+0000 7efcc3fff640 1 --2- 172.21.15.116:0/556581751 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efcc4153570 0x7efcc414f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:58.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.697+0000 7efcc3fff640 1 -- 172.21.15.116:0/556581751 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcc4142370 con 0x7efcc4150bf0 2024-04-25T00:07:58.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.697+0000 7efcc3fff640 1 --2- 172.21.15.116:0/556581751 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efcc4150bf0 0x7efcc4156cb0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7efcb000c7b0 tx=0x7efcb000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:58.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.697+0000 7efcc17fa640 1 -- 172.21.15.116:0/556581751 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb0018070 con 0x7efcc4150bf0 2024-04-25T00:07:58.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.697+0000 7efcc17fa640 1 -- 172.21.15.116:0/556581751 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcb00072d0 con 0x7efcc4150bf0 2024-04-25T00:07:58.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.697+0000 7efccb0f1640 1 -- 172.21.15.116:0/556581751 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcc4143300 con 0x7efcc4150bf0 2024-04-25T00:07:58.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.697+0000 7efcc17fa640 1 -- 172.21.15.116:0/556581751 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb00047f0 con 0x7efcc4150bf0 2024-04-25T00:07:58.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.697+0000 7efccb0f1640 1 -- 172.21.15.116:0/556581751 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcc4142040 con 0x7efcc4150bf0 2024-04-25T00:07:58.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.697+0000 7efccb0f1640 1 -- 172.21.15.116:0/556581751 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efcc4068a60 con 0x7efcc4150bf0 2024-04-25T00:07:58.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.701+0000 7efcc17fa640 1 -- 172.21.15.116:0/556581751 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7efcb001b070 con 0x7efcc4150bf0 2024-04-25T00:07:58.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.701+0000 7efcc17fa640 1 --2- 172.21.15.116:0/556581751 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efca80678f0 0x7efca8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:07:58.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.701+0000 7efcc17fa640 1 -- 172.21.15.116:0/556581751 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 7354+0+0 (secure 0 0 0) 0x7efcb0085970 con 0x7efcc4150bf0 2024-04-25T00:07:58.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.701+0000 7efcc8e8d640 1 --2- 172.21.15.116:0/556581751 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efca80678f0 0x7efca8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:07:58.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.701+0000 7efcc8e8d640 1 --2- 172.21.15.116:0/556581751 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efca80678f0 0x7efca8069db0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7efcb80ad000 tx=0x7efcb8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:07:58.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.705+0000 7efcc17fa640 1 -- 172.21.15.116:0/556581751 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7efcb0056070 con 0x7efcc4150bf0 2024-04-25T00:07:58.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.945+0000 7efccb0f1640 1 -- 172.21.15.116:0/556581751 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efcc40ada60 con 0x7efcc4150bf0 2024-04-25T00:07:58.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.949+0000 7efcc17fa640 1 -- 172.21.15.116:0/556581751 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) ==== 94+0+6502 (secure 0 0 0) 0x7efcb0055970 con 0x7efcc4150bf0 2024-04-25T00:07:58.949 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:07:58.949 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5868,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1670061091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1670061091},{"type":"v1","addr":"172.21.15.116:6835","nonce":1670061091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":56}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:07:33.575034+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5775":{"gid":5775,"name":"c","rank":0,"incarnation":52,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6833/3006671671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3006671671},{"type":"v1","addr":"172.21.15.116:6833","nonce":3006671671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5775]},"id":1},{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:07:03.468507+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":36,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4829":{"gid":4829,"name":"b","rank":0,"incarnation":47,"state":"up:active","state_seq":10,"addr":"172.21.15.174:6837/3583791822","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3583791822},{"type":"v1","addr":"172.21.15.174:6837","nonce":3583791822}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4829,"qdb_cluster":[4829]},"id":2}]} 2024-04-25T00:07:58.950 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 56 2024-04-25T00:07:58.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.949+0000 7efccb0f1640 1 -- 172.21.15.116:0/556581751 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efca80678f0 msgr2=0x7efca8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:58.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.949+0000 7efccb0f1640 1 --2- 172.21.15.116:0/556581751 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efca80678f0 0x7efca8069db0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7efcb80ad000 tx=0x7efcb8008040 comp rx=0 tx=0).stop 2024-04-25T00:07:58.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.949+0000 7efccb0f1640 1 -- 172.21.15.116:0/556581751 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efcc4150bf0 msgr2=0x7efcc4156cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:07:58.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.949+0000 7efccb0f1640 1 --2- 172.21.15.116:0/556581751 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efcc4150bf0 0x7efcc4156cb0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7efcb000c7b0 tx=0x7efcb000cc70 comp rx=0 tx=0).stop 2024-04-25T00:07:58.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.949+0000 7efccb0f1640 1 -- 172.21.15.116:0/556581751 shutdown_connections 2024-04-25T00:07:58.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.949+0000 7efccb0f1640 1 --2- 172.21.15.116:0/556581751 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efca80678f0 0x7efca8069db0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:58.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.949+0000 7efccb0f1640 1 --2- 172.21.15.116:0/556581751 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efcc4153570 0x7efcc414f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:58.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.949+0000 7efccb0f1640 1 --2- 172.21.15.116:0/556581751 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efcc4150bf0 0x7efcc4156cb0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:58.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.949+0000 7efccb0f1640 1 --2- 172.21.15.116:0/556581751 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efcc406c760 0x7efcc40ac8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:07:58.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.949+0000 7efccb0f1640 1 -- 172.21.15.116:0/556581751 >> 172.21.15.116:0/556581751 conn(0x7efcc40c4270 msgr2=0x7efcc406e9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:07:58.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.949+0000 7efccb0f1640 1 -- 172.21.15.116:0/556581751 shutdown_connections 2024-04-25T00:07:58.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:07:58.949+0000 7efccb0f1640 1 -- 172.21.15.116:0/556581751 wait complete. 2024-04-25T00:07:58.970 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T00:08:00.971 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:08:01.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.085+0000 7f7b9d921640 1 Processor -- start 2024-04-25T00:08:01.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.085+0000 7f7b9d921640 1 -- start start 2024-04-25T00:08:01.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.085+0000 7f7b9d921640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7b980c3f70 0x7f7b981557c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:08:01.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.085+0000 7f7b9d921640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b980c3a20 0x7f7b98155d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:08:01.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.085+0000 7f7b9d921640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b98153220 con 0x7f7b98152e40 2024-04-25T00:08:01.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.085+0000 7f7b9d921640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f7b980c4c90 con 0x7f7b980c3f70 2024-04-25T00:08:01.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.085+0000 7f7b9d921640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f7b98156240 con 0x7f7b980c3a20 2024-04-25T00:08:01.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.085+0000 7f7b967fc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7b980c3f70 0x7f7b981557c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:08:01.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.085+0000 7f7b96ffd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f7b98152e40 0x7f7b980710b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:53776/0 (socket says 172.21.15.116:53776) 2024-04-25T00:08:01.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.085+0000 7f7b967fc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7b980c3f70 0x7f7b981557c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47762/0 (socket says 172.21.15.116:47762) 2024-04-25T00:08:01.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.085+0000 7f7b96ffd640 1 -- 172.21.15.116:0/4240403566 learned_addr learned my addr 172.21.15.116:0/4240403566 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:08:01.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.085+0000 7f7b977fe640 1 --2- 172.21.15.116:0/4240403566 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b980c3a20 0x7f7b98155d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:08:01.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.085+0000 7f7b967fc640 1 -- 172.21.15.116:0/4240403566 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b980c3a20 msgr2=0x7f7b98155d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:08:01.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.085+0000 7f7b967fc640 1 --2- 172.21.15.116:0/4240403566 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b980c3a20 0x7f7b98155d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:01.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.085+0000 7f7b967fc640 1 -- 172.21.15.116:0/4240403566 >> v1:172.21.15.116:6789/0 conn(0x7f7b98152e40 legacy=0x7f7b980710b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:08:01.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.085+0000 7f7b95ffb640 1 -- 172.21.15.116:0/4240403566 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 601382451 0 0) 0x7f7b98153220 con 0x7f7b98152e40 2024-04-25T00:08:01.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.085+0000 7f7b967fc640 1 -- 172.21.15.116:0/4240403566 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b980c7a90 con 0x7f7b980c3f70 2024-04-25T00:08:01.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.085+0000 7f7b977fe640 1 --2- 172.21.15.116:0/4240403566 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b980c3a20 0x7f7b98155d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:08:01.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.089+0000 7f7b967fc640 1 --2- 172.21.15.116:0/4240403566 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b980c3f70 0x7f7b981557c0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f7b8c0097c0 tx=0x7f7b8c01c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3da55cc97048b147 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:08:01.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.089+0000 7f7b95ffb640 1 -- 172.21.15.116:0/4240403566 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b8c0b1070 con 0x7f7b980c3f70 2024-04-25T00:08:01.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.089+0000 7f7b95ffb640 1 -- 172.21.15.116:0/4240403566 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b8c0a8050 con 0x7f7b980c3f70 2024-04-25T00:08:01.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.089+0000 7f7b95ffb640 1 -- 172.21.15.116:0/4240403566 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b8c0ac860 con 0x7f7b980c3f70 2024-04-25T00:08:01.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.089+0000 7f7b9d921640 1 -- 172.21.15.116:0/4240403566 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b980c3f70 msgr2=0x7f7b981557c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:08:01.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.089+0000 7f7b9d921640 1 --2- 172.21.15.116:0/4240403566 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b980c3f70 0x7f7b981557c0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f7b8c0097c0 tx=0x7f7b8c01c3e0 comp rx=0 tx=0).stop 2024-04-25T00:08:01.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.089+0000 7f7b9d921640 1 -- 172.21.15.116:0/4240403566 shutdown_connections 2024-04-25T00:08:01.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.089+0000 7f7b9d921640 1 --2- 172.21.15.116:0/4240403566 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b980c3a20 0x7f7b98155d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:01.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.089+0000 7f7b9d921640 1 --2- 172.21.15.116:0/4240403566 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b980c3f70 0x7f7b981557c0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:01.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.089+0000 7f7b9d921640 1 -- 172.21.15.116:0/4240403566 >> 172.21.15.116:0/4240403566 conn(0x7f7b980c9160 msgr2=0x7f7b980c9570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:08:01.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.089+0000 7f7b9d921640 1 -- 172.21.15.116:0/4240403566 shutdown_connections 2024-04-25T00:08:01.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.089+0000 7f7b9d921640 1 -- 172.21.15.116:0/4240403566 wait complete. 2024-04-25T00:08:01.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.093+0000 7f7b9d921640 1 Processor -- start 2024-04-25T00:08:01.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.093+0000 7f7b9d921640 1 -- start start 2024-04-25T00:08:01.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.093+0000 7f7b9d921640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b980c3a20 0x7f7b98144f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:08:01.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.093+0000 7f7b9d921640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b980c3f70 0x7f7b9806fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:08:01.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.093+0000 7f7b9d921640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b98152e40 0x7f7b9806ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:08:01.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.093+0000 7f7b9d921640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7b98071940 con 0x7f7b980c3f70 2024-04-25T00:08:01.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.093+0000 7f7b9d921640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7b980717c0 con 0x7f7b98152e40 2024-04-25T00:08:01.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.093+0000 7f7b9d921640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f7b980c5120 con 0x7f7b980c3a20 2024-04-25T00:08:01.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.093+0000 7f7b967fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b980c3f70 0x7f7b9806fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:08:01.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.093+0000 7f7b967fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b980c3f70 0x7f7b9806fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47764/0 (socket says 172.21.15.116:47764) 2024-04-25T00:08:01.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.093+0000 7f7b967fc640 1 -- 172.21.15.116:0/3081983897 learned_addr learned my addr 172.21.15.116:0/3081983897 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:08:01.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.093+0000 7f7b96ffd640 1 --2- 172.21.15.116:0/3081983897 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b980c3a20 0x7f7b98144f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:08:01.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.093+0000 7f7b967fc640 1 -- 172.21.15.116:0/3081983897 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b980c3a20 msgr2=0x7f7b98144f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:08:01.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.093+0000 7f7b967fc640 1 --2- 172.21.15.116:0/3081983897 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b980c3a20 0x7f7b98144f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:01.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.093+0000 7f7b977fe640 1 --2- 172.21.15.116:0/3081983897 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b98152e40 0x7f7b9806ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:08:01.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.093+0000 7f7b967fc640 1 -- 172.21.15.116:0/3081983897 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b98152e40 msgr2=0x7f7b9806ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:08:01.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.093+0000 7f7b967fc640 1 --2- 172.21.15.116:0/3081983897 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b98152e40 0x7f7b9806ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:01.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.093+0000 7f7b967fc640 1 -- 172.21.15.116:0/3081983897 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b98145450 con 0x7f7b980c3f70 2024-04-25T00:08:01.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.093+0000 7f7b977fe640 1 --2- 172.21.15.116:0/3081983897 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b98152e40 0x7f7b9806ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:08:01.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.093+0000 7f7b967fc640 1 --2- 172.21.15.116:0/3081983897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b980c3f70 0x7f7b9806fa10 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f7b8c01a010 tx=0x7f7b8c01ac20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:08:01.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.093+0000 7f7b7ffff640 1 -- 172.21.15.116:0/3081983897 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b8c0bb070 con 0x7f7b980c3f70 2024-04-25T00:08:01.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.093+0000 7f7b7ffff640 1 -- 172.21.15.116:0/3081983897 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b8c004020 con 0x7f7b980c3f70 2024-04-25T00:08:01.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.093+0000 7f7b7ffff640 1 -- 172.21.15.116:0/3081983897 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b8c0b1040 con 0x7f7b980c3f70 2024-04-25T00:08:01.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.093+0000 7f7b9d921640 1 -- 172.21.15.116:0/3081983897 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b981edd50 con 0x7f7b980c3f70 2024-04-25T00:08:01.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.093+0000 7f7b9d921640 1 -- 172.21.15.116:0/3081983897 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b981ee2f0 con 0x7f7b980c3f70 2024-04-25T00:08:01.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.093+0000 7f7b9d921640 1 -- 172.21.15.116:0/3081983897 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b98068a60 con 0x7f7b980c3f70 2024-04-25T00:08:01.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.097+0000 7f7b7ffff640 1 -- 172.21.15.116:0/3081983897 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7b8c003620 con 0x7f7b980c3f70 2024-04-25T00:08:01.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.097+0000 7f7b7ffff640 1 --2- 172.21.15.116:0/3081983897 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7b840678f0 0x7f7b84069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:08:01.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.097+0000 7f7b7ffff640 1 -- 172.21.15.116:0/3081983897 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7440+0+0 (secure 0 0 0) 0x7f7b8c120700 con 0x7f7b980c3f70 2024-04-25T00:08:01.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.097+0000 7f7b96ffd640 1 --2- 172.21.15.116:0/3081983897 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7b840678f0 0x7f7b84069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:08:01.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.097+0000 7f7b96ffd640 1 --2- 172.21.15.116:0/3081983897 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7b840678f0 0x7f7b84069db0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f7b88000f70 tx=0x7f7b88007b80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:08:01.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.101+0000 7f7b7ffff640 1 -- 172.21.15.116:0/3081983897 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f7b98145450 con 0x7f7b980c3f70 2024-04-25T00:08:01.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.345+0000 7f7b9d921640 1 -- 172.21.15.116:0/3081983897 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7b98159c80 con 0x7f7b980c3f70 2024-04-25T00:08:01.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.345+0000 7f7b7ffff640 1 -- 172.21.15.116:0/3081983897 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) ==== 94+0+5666 (secure 0 0 0) 0x7f7b8c016090 con 0x7f7b980c3f70 2024-04-25T00:08:01.347 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:08:01.347 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:07:33.575034+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5775":{"gid":5775,"name":"c","rank":0,"incarnation":52,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6833/3006671671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3006671671},{"type":"v1","addr":"172.21.15.116:6833","nonce":3006671671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5775]},"id":1},{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:08:00.461138+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5868},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5868":{"gid":5868,"name":"a","rank":0,"incarnation":57,"state":"up:replay","state_seq":1,"addr":"172.21.15.116:6835/1670061091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1670061091},{"type":"v1","addr":"172.21.15.116:6835","nonce":1670061091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:08:01.347 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 57 2024-04-25T00:08:01.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.349+0000 7f7b9d921640 1 -- 172.21.15.116:0/3081983897 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7b840678f0 msgr2=0x7f7b84069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:08:01.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.349+0000 7f7b9d921640 1 --2- 172.21.15.116:0/3081983897 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7b840678f0 0x7f7b84069db0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f7b88000f70 tx=0x7f7b88007b80 comp rx=0 tx=0).stop 2024-04-25T00:08:01.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.349+0000 7f7b9d921640 1 -- 172.21.15.116:0/3081983897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b980c3f70 msgr2=0x7f7b9806fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:08:01.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.349+0000 7f7b9d921640 1 --2- 172.21.15.116:0/3081983897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b980c3f70 0x7f7b9806fa10 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f7b8c01a010 tx=0x7f7b8c01ac20 comp rx=0 tx=0).stop 2024-04-25T00:08:01.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.349+0000 7f7b9d921640 1 -- 172.21.15.116:0/3081983897 shutdown_connections 2024-04-25T00:08:01.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.349+0000 7f7b9d921640 1 --2- 172.21.15.116:0/3081983897 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7b840678f0 0x7f7b84069db0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:01.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.349+0000 7f7b9d921640 1 --2- 172.21.15.116:0/3081983897 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b98152e40 0x7f7b9806ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:01.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.349+0000 7f7b9d921640 1 --2- 172.21.15.116:0/3081983897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b980c3f70 0x7f7b9806fa10 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:01.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.349+0000 7f7b9d921640 1 --2- 172.21.15.116:0/3081983897 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b980c3a20 0x7f7b98144f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:01.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.349+0000 7f7b9d921640 1 -- 172.21.15.116:0/3081983897 >> 172.21.15.116:0/3081983897 conn(0x7f7b980c9160 msgr2=0x7f7b980c9540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:08:01.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.349+0000 7f7b9d921640 1 -- 172.21.15.116:0/3081983897 shutdown_connections 2024-04-25T00:08:01.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.349+0000 7f7b9d921640 1 -- 172.21.15.116:0/3081983897 wait complete. 2024-04-25T00:08:01.365 INFO:tasks.mds_thrash.fs.[b]:mds.b down, removed from mdsmap 2024-04-25T00:08:01.365 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-04-25T00:08:01.366 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:08:01.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.473+0000 7f3756c3a640 1 Processor -- start 2024-04-25T00:08:01.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.473+0000 7f3756c3a640 1 -- start start 2024-04-25T00:08:01.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.473+0000 7f3756c3a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f37500aa170 0x7f3750155af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:08:01.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.473+0000 7f3756c3a640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37501526a0 con 0x7f3750156030 2024-04-25T00:08:01.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.473+0000 7f3756c3a640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3750152980 con 0x7f37500c4300 2024-04-25T00:08:01.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.473+0000 7f3756c3a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f3750153460 con 0x7f37500aa170 2024-04-25T00:08:01.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.473+0000 7f37551d7640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f3750156030 0x7f375015ce80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:53792/0 (socket says 172.21.15.116:53792) 2024-04-25T00:08:01.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.473+0000 7f37551d7640 1 -- 172.21.15.116:0/1535338955 learned_addr learned my addr 172.21.15.116:0/1535338955 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:08:01.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.473+0000 7f374f7fe640 1 -- 172.21.15.116:0/1535338955 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 27215190 0 0) 0x7f37501526a0 con 0x7f3750156030 2024-04-25T00:08:01.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.473+0000 7f374f7fe640 1 -- 172.21.15.116:0/1535338955 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3740003530 con 0x7f3750156030 2024-04-25T00:08:01.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.473+0000 7f374ffff640 1 --2- 172.21.15.116:0/1535338955 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f37500aa170 0x7f3750155af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:08:01.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.473+0000 7f374f7fe640 1 -- 172.21.15.116:0/1535338955 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 946481765 0 0) 0x7f3740003530 con 0x7f3750156030 2024-04-25T00:08:01.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.473+0000 7f374f7fe640 1 -- 172.21.15.116:0/1535338955 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f37500aa170 msgr2=0x7f3750155af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:08:01.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.473+0000 7f374f7fe640 1 --2- 172.21.15.116:0/1535338955 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f37500aa170 0x7f3750155af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:01.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.473+0000 7f374f7fe640 1 -- 172.21.15.116:0/1535338955 >> v1:172.21.15.174:6789/0 conn(0x7f37500c4300 legacy=0x7f37500c5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:08:01.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.473+0000 7f374f7fe640 1 -- 172.21.15.116:0/1535338955 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3750152c60 con 0x7f3750156030 2024-04-25T00:08:01.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.473+0000 7f374f7fe640 1 -- 172.21.15.116:0/1535338955 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f3748001a00 con 0x7f3750156030 2024-04-25T00:08:01.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.477+0000 7f374f7fe640 1 -- 172.21.15.116:0/1535338955 >> v1:172.21.15.116:6789/0 conn(0x7f3750156030 legacy=0x7f375015ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:08:01.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.477+0000 7f374f7fe640 1 --2- 172.21.15.116:0/1535338955 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37400039f0 0x7f3740003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:08:01.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.477+0000 7f374f7fe640 1 --2- 172.21.15.116:0/1535338955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3740004810 0x7f3740004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:08:01.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.477+0000 7f374f7fe640 1 --2- 172.21.15.116:0/1535338955 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f37500aa170 0x7f37400051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:08:01.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.477+0000 7f374f7fe640 1 -- 172.21.15.116:0/1535338955 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3750153460 con 0x7f3740004810 2024-04-25T00:08:01.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.477+0000 7f374f7fe640 1 -- 172.21.15.116:0/1535338955 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f374000dc90 con 0x7f37400039f0 2024-04-25T00:08:01.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.477+0000 7f374f7fe640 1 -- 172.21.15.116:0/1535338955 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f374000de60 con 0x7f37500aa170 2024-04-25T00:08:01.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.477+0000 7f374ffff640 1 --2- 172.21.15.116:0/1535338955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3740004810 0x7f3740004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:08:01.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.477+0000 7f37549d6640 1 --2- 172.21.15.116:0/1535338955 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37400039f0 0x7f3740003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:08:01.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.477+0000 7f37551d7640 1 --2- 172.21.15.116:0/1535338955 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f37500aa170 0x7f37400051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:08:01.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.477+0000 7f374ffff640 1 -- 172.21.15.116:0/1535338955 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f37500aa170 msgr2=0x7f37400051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:08:01.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.477+0000 7f374ffff640 1 --2- 172.21.15.116:0/1535338955 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f37500aa170 0x7f37400051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:01.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.477+0000 7f374ffff640 1 -- 172.21.15.116:0/1535338955 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37400039f0 msgr2=0x7f3740003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:08:01.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.477+0000 7f374ffff640 1 --2- 172.21.15.116:0/1535338955 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37400039f0 0x7f3740003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:01.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.477+0000 7f374ffff640 1 -- 172.21.15.116:0/1535338955 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3740003530 con 0x7f3740004810 2024-04-25T00:08:01.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.477+0000 7f37549d6640 1 --2- 172.21.15.116:0/1535338955 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37400039f0 0x7f3740003df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:08:01.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.477+0000 7f374ffff640 1 --2- 172.21.15.116:0/1535338955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3740004810 0x7f3740004c90 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f373c009ae0 tx=0x7f373c01ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac994fdad830e48a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:08:01.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.477+0000 7f374f7fe640 1 -- 172.21.15.116:0/1535338955 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f373c0b2070 con 0x7f3740004810 2024-04-25T00:08:01.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.477+0000 7f374f7fe640 1 -- 172.21.15.116:0/1535338955 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f373c0a5600 con 0x7f3740004810 2024-04-25T00:08:01.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.477+0000 7f374f7fe640 1 -- 172.21.15.116:0/1535338955 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f373c0096b0 con 0x7f3740004810 2024-04-25T00:08:01.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.477+0000 7f3756c3a640 1 -- 172.21.15.116:0/1535338955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3740004810 msgr2=0x7f3740004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:08:01.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.477+0000 7f3756c3a640 1 --2- 172.21.15.116:0/1535338955 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3740004810 0x7f3740004c90 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f373c009ae0 tx=0x7f373c01ca10 comp rx=0 tx=0).stop 2024-04-25T00:08:01.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.477+0000 7f37551d7640 1 -- 172.21.15.116:0/1535338955 reap_dead start 2024-04-25T00:08:01.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.481+0000 7f3756c3a640 1 -- 172.21.15.116:0/1535338955 shutdown_connections 2024-04-25T00:08:01.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.481+0000 7f3756c3a640 1 -- 172.21.15.116:0/1535338955 >> 172.21.15.116:0/1535338955 conn(0x7f375006c720 msgr2=0x7f375006cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:08:01.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.481+0000 7f3756c3a640 1 -- 172.21.15.116:0/1535338955 shutdown_connections 2024-04-25T00:08:01.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.481+0000 7f3756c3a640 1 -- 172.21.15.116:0/1535338955 wait complete. 2024-04-25T00:08:01.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.481+0000 7f3756c3a640 1 Processor -- start 2024-04-25T00:08:01.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.481+0000 7f3756c3a640 1 -- start start 2024-04-25T00:08:01.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.481+0000 7f3756c3a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37501ddde0 0x7f37501eaf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:08:01.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.481+0000 7f3756c3a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37500c77e0 0x7f37501e6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:08:01.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.481+0000 7f3756c3a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f37501e7520 0x7f3750143290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:08:01.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.481+0000 7f3756c3a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f37501564c0 con 0x7f37500c77e0 2024-04-25T00:08:01.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.481+0000 7f3756c3a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f375006ffd0 con 0x7f37501ddde0 2024-04-25T00:08:01.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.481+0000 7f3756c3a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f3750156640 con 0x7f37501e7520 2024-04-25T00:08:01.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.481+0000 7f374ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37500c77e0 0x7f37501e6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:08:01.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.481+0000 7f374ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37500c77e0 0x7f37501e6fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47782/0 (socket says 172.21.15.116:47782) 2024-04-25T00:08:01.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.481+0000 7f374ffff640 1 -- 172.21.15.116:0/2899174590 learned_addr learned my addr 172.21.15.116:0/2899174590 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:08:01.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.481+0000 7f374ffff640 1 -- 172.21.15.116:0/2899174590 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f37501e7520 msgr2=0x7f3750143290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:08:01.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.481+0000 7f374ffff640 1 --2- 172.21.15.116:0/2899174590 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f37501e7520 0x7f3750143290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:01.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.481+0000 7f374ffff640 1 -- 172.21.15.116:0/2899174590 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37501ddde0 msgr2=0x7f37501eaf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:08:01.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.481+0000 7f374ffff640 1 --2- 172.21.15.116:0/2899174590 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37501ddde0 0x7f37501eaf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:01.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.481+0000 7f374ffff640 1 -- 172.21.15.116:0/2899174590 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37501eb470 con 0x7f37500c77e0 2024-04-25T00:08:01.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.481+0000 7f374ffff640 1 --2- 172.21.15.116:0/2899174590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37500c77e0 0x7f37501e6fe0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f373c0b0040 tx=0x7f373c0197b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:08:01.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.481+0000 7f374d7fa640 1 -- 172.21.15.116:0/2899174590 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f373c0b2040 con 0x7f37500c77e0 2024-04-25T00:08:01.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.481+0000 7f374d7fa640 1 -- 172.21.15.116:0/2899174590 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f373c0bd070 con 0x7f37500c77e0 2024-04-25T00:08:01.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.481+0000 7f3756c3a640 1 -- 172.21.15.116:0/2899174590 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37500ab530 con 0x7f37500c77e0 2024-04-25T00:08:01.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.481+0000 7f374d7fa640 1 -- 172.21.15.116:0/2899174590 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f373c0096b0 con 0x7f37500c77e0 2024-04-25T00:08:01.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.481+0000 7f3756c3a640 1 -- 172.21.15.116:0/2899174590 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37500ad1c0 con 0x7f37500c77e0 2024-04-25T00:08:01.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.485+0000 7f3756c3a640 1 -- 172.21.15.116:0/2899174590 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37500c7420 con 0x7f37500c77e0 2024-04-25T00:08:01.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.485+0000 7f374d7fa640 1 -- 172.21.15.116:0/2899174590 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f373c0c2460 con 0x7f37500c77e0 2024-04-25T00:08:01.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.485+0000 7f374d7fa640 1 --2- 172.21.15.116:0/2899174590 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f37340678f0 0x7f3734069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:08:01.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.485+0000 7f374d7fa640 1 -- 172.21.15.116:0/2899174590 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7440+0+0 (secure 0 0 0) 0x7f373c120920 con 0x7f37500c77e0 2024-04-25T00:08:01.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.485+0000 7f37549d6640 1 --2- 172.21.15.116:0/2899174590 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f37340678f0 0x7f3734069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:08:01.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.485+0000 7f37549d6640 1 --2- 172.21.15.116:0/2899174590 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f37340678f0 0x7f3734069db0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f3744003de0 tx=0x7f3744009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:08:01.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.489+0000 7f374d7fa640 1 -- 172.21.15.116:0/2899174590 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f373c016760 con 0x7f37500c77e0 2024-04-25T00:08:01.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.697+0000 7f3756c3a640 1 -- 172.21.15.116:0/2899174590 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3750068a60 con 0x7f37500c77e0 2024-04-25T00:08:01.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.697+0000 7f374d7fa640 1 -- 172.21.15.116:0/2899174590 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) ==== 94+0+5669 (secure 0 0 0) 0x7f373c0f0d20 con 0x7f37500c77e0 2024-04-25T00:08:01.698 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:08:01.699 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:07:33.575034+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5775":{"gid":5775,"name":"c","rank":0,"incarnation":52,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6833/3006671671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3006671671},{"type":"v1","addr":"172.21.15.116:6833","nonce":3006671671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5775]},"id":1},{"mdsmap":{"epoch":58,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:08:00.801682+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5868},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5868":{"gid":5868,"name":"a","rank":0,"incarnation":57,"state":"up:reconnect","state_seq":7,"addr":"172.21.15.116:6835/1670061091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1670061091},{"type":"v1","addr":"172.21.15.116:6835","nonce":1670061091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:08:01.700 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 58 2024-04-25T00:08:01.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.697+0000 7f3756c3a640 1 -- 172.21.15.116:0/2899174590 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f37340678f0 msgr2=0x7f3734069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:08:01.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.697+0000 7f3756c3a640 1 --2- 172.21.15.116:0/2899174590 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f37340678f0 0x7f3734069db0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f3744003de0 tx=0x7f3744009040 comp rx=0 tx=0).stop 2024-04-25T00:08:01.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.697+0000 7f3756c3a640 1 -- 172.21.15.116:0/2899174590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37500c77e0 msgr2=0x7f37501e6fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:08:01.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.697+0000 7f3756c3a640 1 --2- 172.21.15.116:0/2899174590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37500c77e0 0x7f37501e6fe0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f373c0b0040 tx=0x7f373c0197b0 comp rx=0 tx=0).stop 2024-04-25T00:08:01.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.697+0000 7f3756c3a640 1 -- 172.21.15.116:0/2899174590 shutdown_connections 2024-04-25T00:08:01.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.697+0000 7f3756c3a640 1 --2- 172.21.15.116:0/2899174590 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f37340678f0 0x7f3734069db0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:01.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.697+0000 7f3756c3a640 1 --2- 172.21.15.116:0/2899174590 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f37501e7520 0x7f3750143290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:01.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.697+0000 7f3756c3a640 1 --2- 172.21.15.116:0/2899174590 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f37500c77e0 0x7f37501e6fe0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:01.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.697+0000 7f3756c3a640 1 --2- 172.21.15.116:0/2899174590 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f37501ddde0 0x7f37501eaf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:01.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.697+0000 7f3756c3a640 1 -- 172.21.15.116:0/2899174590 >> 172.21.15.116:0/2899174590 conn(0x7f375006c720 msgr2=0x7f37500aa620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:08:01.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.697+0000 7f3756c3a640 1 -- 172.21.15.116:0/2899174590 shutdown_connections 2024-04-25T00:08:01.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:01.697+0000 7f3756c3a640 1 -- 172.21.15.116:0/2899174590 wait complete. 2024-04-25T00:08:01.719 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 58, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:04.678737+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 55, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5775": { "addr": "172.21.15.116:6833/3006671671", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6832", "nonce": 3006671671, "type": "v2" }, { "addr": "172.21.15.116:6833", "nonce": 3006671671, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5775, "incarnation": 52, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 10 } }, "last_failure": 0, "last_failure_osd_epoch": 37, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T00:07:33.575034+0000", "qdb_cluster": [ 5775 ], "qdb_leader": 5775, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5775 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:18.825887+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 58, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5868": { "addr": "172.21.15.116:6835/1670061091", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6834", "nonce": 1670061091, "type": "v2" }, { "addr": "172.21.15.116:6835", "nonce": 1670061091, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5868, "incarnation": 57, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:reconnect", "state_seq": 7 } }, "last_failure": 0, "last_failure_osd_epoch": 38, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-04-25T00:08:00.801682+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5868 } } } ], "standbys": [ { "addr": "172.21.15.174:6835/2883220846", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6834", "nonce": 2883220846, "type": "v2" }, { "addr": "172.21.15.174:6835", "nonce": 2883220846, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4235, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-25T00:08:03.720 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:08:03.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:03.853+0000 7fe197fea640 1 Processor -- start 2024-04-25T00:08:03.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:03.853+0000 7fe197fea640 1 -- start start 2024-04-25T00:08:03.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:03.857+0000 7fe197fea640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe190150d20 0x7fe190153620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:08:03.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:03.857+0000 7fe197fea640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe1900c3240 0x7fe1900c3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:08:03.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:03.857+0000 7fe197fea640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1900c36a0 con 0x7fe1900c4380 2024-04-25T00:08:03.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:03.857+0000 7fe197fea640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fe19006fba0 con 0x7fe1900c3240 2024-04-25T00:08:03.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:03.857+0000 7fe197fea640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe19006f8c0 con 0x7fe190150d20 2024-04-25T00:08:03.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:03.857+0000 7fe195585640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe1900c3240 0x7fe1900c3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:08:03.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:03.857+0000 7fe195585640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe1900c3240 0x7fe1900c3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47792/0 (socket says 172.21.15.116:47792) 2024-04-25T00:08:03.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:03.857+0000 7fe195585640 1 -- 172.21.15.116:0/2477640113 learned_addr learned my addr 172.21.15.116:0/2477640113 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:08:03.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:03.857+0000 7fe195d86640 1 --2- 172.21.15.116:0/2477640113 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe190150d20 0x7fe190153620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:08:03.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:03.857+0000 7fe195585640 1 -- 172.21.15.116:0/2477640113 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe190150d20 msgr2=0x7fe190153620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:08:03.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:03.857+0000 7fe195585640 1 --2- 172.21.15.116:0/2477640113 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe190150d20 0x7fe190153620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:03.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:03.857+0000 7fe195585640 1 -- 172.21.15.116:0/2477640113 >> v1:172.21.15.174:6789/0 conn(0x7fe1900c4380 legacy=0x7fe190071050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:08:03.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:03.857+0000 7fe195585640 1 -- 172.21.15.116:0/2477640113 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1900c5420 con 0x7fe1900c3240 2024-04-25T00:08:03.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:03.857+0000 7fe195585640 1 --2- 172.21.15.116:0/2477640113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1900c3240 0x7fe1900c3e40 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fe184009e50 tx=0x7fe18401ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3aecb7de2a84c18f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:08:03.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:03.857+0000 7fe194d84640 1 -- 172.21.15.116:0/2477640113 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1840a5810 con 0x7fe1900c3240 2024-04-25T00:08:03.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:03.857+0000 7fe194d84640 1 -- 172.21.15.116:0/2477640113 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1840a59b0 con 0x7fe1900c3240 2024-04-25T00:08:03.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:03.857+0000 7fe194d84640 1 -- 172.21.15.116:0/2477640113 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1840a5d80 con 0x7fe1900c3240 2024-04-25T00:08:03.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:03.861+0000 7fe197fea640 1 -- 172.21.15.116:0/2477640113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1900c3240 msgr2=0x7fe1900c3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:08:03.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:03.861+0000 7fe197fea640 1 --2- 172.21.15.116:0/2477640113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1900c3240 0x7fe1900c3e40 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fe184009e50 tx=0x7fe18401ca30 comp rx=0 tx=0).stop 2024-04-25T00:08:03.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:03.861+0000 7fe197fea640 1 -- 172.21.15.116:0/2477640113 shutdown_connections 2024-04-25T00:08:03.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:03.861+0000 7fe197fea640 1 --2- 172.21.15.116:0/2477640113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1900c3240 0x7fe1900c3e40 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:03.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:03.861+0000 7fe197fea640 1 --2- 172.21.15.116:0/2477640113 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe190150d20 0x7fe190153620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:03.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:03.861+0000 7fe197fea640 1 -- 172.21.15.116:0/2477640113 >> 172.21.15.116:0/2477640113 conn(0x7fe1900c9160 msgr2=0x7fe1900c9570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:08:03.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:03.861+0000 7fe197fea640 1 -- 172.21.15.116:0/2477640113 shutdown_connections 2024-04-25T00:08:03.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:03.861+0000 7fe197fea640 1 -- 172.21.15.116:0/2477640113 wait complete. 2024-04-25T00:08:03.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:03.861+0000 7fe197fea640 1 Processor -- start 2024-04-25T00:08:03.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:03.861+0000 7fe197fea640 1 -- start start 2024-04-25T00:08:03.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:03.861+0000 7fe197fea640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1900c3240 0x7fe1901424f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:08:03.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:03.861+0000 7fe197fea640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe1900c4380 0x7fe190140d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:08:03.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:03.861+0000 7fe197fea640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe190150d20 0x7fe1901412c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:08:03.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:03.861+0000 7fe197fea640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe1900718e0 con 0x7fe1900c3240 2024-04-25T00:08:03.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:03.861+0000 7fe197fea640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe190071760 con 0x7fe1900c4380 2024-04-25T00:08:03.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:03.861+0000 7fe197fea640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe190151100 con 0x7fe190150d20 2024-04-25T00:08:03.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:03.861+0000 7fe195d86640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1900c3240 0x7fe1901424f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:08:03.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:03.861+0000 7fe195d86640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1900c3240 0x7fe1901424f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47802/0 (socket says 172.21.15.116:47802) 2024-04-25T00:08:03.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:03.861+0000 7fe195d86640 1 -- 172.21.15.116:0/3387583038 learned_addr learned my addr 172.21.15.116:0/3387583038 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:08:03.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:03.861+0000 7fe195585640 1 --2- 172.21.15.116:0/3387583038 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe1900c4380 0x7fe190140d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:08:03.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:03.861+0000 7fe196587640 1 --2- 172.21.15.116:0/3387583038 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe190150d20 0x7fe1901412c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:08:03.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:03.861+0000 7fe195d86640 1 -- 172.21.15.116:0/3387583038 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe190150d20 msgr2=0x7fe1901412c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:08:03.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:03.861+0000 7fe195d86640 1 --2- 172.21.15.116:0/3387583038 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe190150d20 0x7fe1901412c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:03.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:03.861+0000 7fe195d86640 1 -- 172.21.15.116:0/3387583038 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe1900c4380 msgr2=0x7fe190140d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:08:03.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:03.861+0000 7fe195d86640 1 --2- 172.21.15.116:0/3387583038 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe1900c4380 0x7fe190140d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:03.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:03.861+0000 7fe195d86640 1 -- 172.21.15.116:0/3387583038 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1901490a0 con 0x7fe1900c3240 2024-04-25T00:08:03.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:03.861+0000 7fe195d86640 1 --2- 172.21.15.116:0/3387583038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1900c3240 0x7fe1901424f0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fe18800d5f0 tx=0x7fe18800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:08:03.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:03.861+0000 7fe18e7fc640 1 -- 172.21.15.116:0/3387583038 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1880125f0 con 0x7fe1900c3240 2024-04-25T00:08:03.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:03.861+0000 7fe18e7fc640 1 -- 172.21.15.116:0/3387583038 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe188019070 con 0x7fe1900c3240 2024-04-25T00:08:03.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:03.861+0000 7fe197fea640 1 -- 172.21.15.116:0/3387583038 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe190149380 con 0x7fe1900c3240 2024-04-25T00:08:03.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:03.861+0000 7fe18e7fc640 1 -- 172.21.15.116:0/3387583038 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1880129d0 con 0x7fe1900c3240 2024-04-25T00:08:03.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:03.861+0000 7fe197fea640 1 -- 172.21.15.116:0/3387583038 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe190142a30 con 0x7fe1900c3240 2024-04-25T00:08:03.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:03.865+0000 7fe197fea640 1 -- 172.21.15.116:0/3387583038 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe190068a60 con 0x7fe1900c3240 2024-04-25T00:08:03.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:03.865+0000 7fe18e7fc640 1 -- 172.21.15.116:0/3387583038 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe188012b70 con 0x7fe1900c3240 2024-04-25T00:08:03.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:03.865+0000 7fe18e7fc640 1 --2- 172.21.15.116:0/3387583038 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe17c0678c0 0x7fe17c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:08:03.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:03.865+0000 7fe18e7fc640 1 -- 172.21.15.116:0/3387583038 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7440+0+0 (secure 0 0 0) 0x7fe188086710 con 0x7fe1900c3240 2024-04-25T00:08:03.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:03.865+0000 7fe195585640 1 --2- 172.21.15.116:0/3387583038 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe17c0678c0 0x7fe17c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:08:03.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:03.865+0000 7fe195585640 1 --2- 172.21.15.116:0/3387583038 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe17c0678c0 0x7fe17c069d80 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fe1840ae000 tx=0x7fe1840a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:08:03.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:03.873+0000 7fe18e7fc640 1 -- 172.21.15.116:0/3387583038 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fe188056c20 con 0x7fe1900c3240 2024-04-25T00:08:04.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:04.125+0000 7fe197fea640 1 -- 172.21.15.116:0/3387583038 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe1900c4790 con 0x7fe1900c3240 2024-04-25T00:08:04.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:04.125+0000 7fe18e7fc640 1 -- 172.21.15.116:0/3387583038 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) ==== 94+0+5673 (secure 0 0 0) 0x7fe188056520 con 0x7fe1900c3240 2024-04-25T00:08:04.127 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:08:04.127 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:07:33.575034+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5775":{"gid":5775,"name":"c","rank":0,"incarnation":52,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6833/3006671671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3006671671},{"type":"v1","addr":"172.21.15.116:6833","nonce":3006671671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5775]},"id":1},{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:08:03.679028+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5868},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5868":{"gid":5868,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6835/1670061091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1670061091},{"type":"v1","addr":"172.21.15.116:6835","nonce":1670061091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5868,"qdb_cluster":[5868]},"id":2}]} 2024-04-25T00:08:04.127 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 60 2024-04-25T00:08:04.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:04.129+0000 7fe197fea640 1 -- 172.21.15.116:0/3387583038 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe17c0678c0 msgr2=0x7fe17c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:08:04.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:04.129+0000 7fe197fea640 1 --2- 172.21.15.116:0/3387583038 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe17c0678c0 0x7fe17c069d80 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fe1840ae000 tx=0x7fe1840a7070 comp rx=0 tx=0).stop 2024-04-25T00:08:04.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:04.129+0000 7fe197fea640 1 -- 172.21.15.116:0/3387583038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1900c3240 msgr2=0x7fe1901424f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:08:04.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:04.129+0000 7fe197fea640 1 --2- 172.21.15.116:0/3387583038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1900c3240 0x7fe1901424f0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fe18800d5f0 tx=0x7fe18800dab0 comp rx=0 tx=0).stop 2024-04-25T00:08:04.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:04.129+0000 7fe197fea640 1 -- 172.21.15.116:0/3387583038 shutdown_connections 2024-04-25T00:08:04.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:04.129+0000 7fe197fea640 1 --2- 172.21.15.116:0/3387583038 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe17c0678c0 0x7fe17c069d80 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:04.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:04.129+0000 7fe197fea640 1 --2- 172.21.15.116:0/3387583038 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe190150d20 0x7fe1901412c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:04.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:04.129+0000 7fe197fea640 1 --2- 172.21.15.116:0/3387583038 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe1900c4380 0x7fe190140d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:04.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:04.129+0000 7fe197fea640 1 --2- 172.21.15.116:0/3387583038 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1900c3240 0x7fe1901424f0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:04.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:04.129+0000 7fe197fea640 1 -- 172.21.15.116:0/3387583038 >> 172.21.15.116:0/3387583038 conn(0x7fe1900c9160 msgr2=0x7fe1900c9540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:08:04.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:04.129+0000 7fe197fea640 1 -- 172.21.15.116:0/3387583038 shutdown_connections 2024-04-25T00:08:04.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:04.129+0000 7fe197fea640 1 -- 172.21.15.116:0/3387583038 wait complete. 2024-04-25T00:08:04.147 INFO:tasks.mds_thrash.fs.[b]:mds.a has gained rank=0, replacing gid=4829 2024-04-25T00:08:04.147 INFO:tasks.mds_thrash.fs.[b]:waiting for 1 secs before reviving mds.b 2024-04-25T00:08:04.603 INFO:tasks.ceph.mds.a.smithi116.stderr:2024-04-25T00:08:04.601+0000 7f72e7ca6640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T00:08:05.149 INFO:tasks.mds_thrash.fs.[b]:reviving mds.b 2024-04-25T00:08:05.149 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-25T00:08:05.149 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-25T00:08:05.152 INFO:tasks.ceph.mds.b:Started 2024-04-25T00:08:05.152 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:08:05.230 INFO:tasks.ceph.mds.b.smithi174.stdout:starting mds.b at 2024-04-25T00:08:05.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.293+0000 7fb914eab640 1 Processor -- start 2024-04-25T00:08:05.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.293+0000 7fb914eab640 1 -- start start 2024-04-25T00:08:05.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.297+0000 7fb914eab640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb9100c5200 0x7fb910154040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:08:05.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.297+0000 7fb914eab640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb910156bb0 0x7fb910157020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:08:05.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.297+0000 7fb914eab640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9100c2b40 con 0x7fb9100c4760 2024-04-25T00:08:05.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.297+0000 7fb914eab640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fb910072b80 con 0x7fb9100c5200 2024-04-25T00:08:05.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.297+0000 7fb914eab640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fb910154580 con 0x7fb910156bb0 2024-04-25T00:08:05.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.297+0000 7fb90e59c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb9100c5200 0x7fb910154040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:08:05.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.297+0000 7fb90e59c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb9100c5200 0x7fb910154040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45340/0 (socket says 172.21.15.116:45340) 2024-04-25T00:08:05.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.297+0000 7fb90e59c640 1 -- 172.21.15.116:0/947492094 learned_addr learned my addr 172.21.15.116:0/947492094 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:08:05.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.297+0000 7fb90ed9d640 1 --2- 172.21.15.116:0/947492094 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb910156bb0 0x7fb910157020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:08:05.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.297+0000 7fb90e59c640 1 -- 172.21.15.116:0/947492094 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb910156bb0 msgr2=0x7fb910157020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:08:05.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.297+0000 7fb90e59c640 1 --2- 172.21.15.116:0/947492094 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb910156bb0 0x7fb910157020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:05.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.297+0000 7fb90e59c640 1 -- 172.21.15.116:0/947492094 >> v1:172.21.15.174:6789/0 conn(0x7fb9100c4760 legacy=0x7fb91014d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:08:05.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.297+0000 7fb90e59c640 1 -- 172.21.15.116:0/947492094 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9100c2db0 con 0x7fb9100c5200 2024-04-25T00:08:05.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.297+0000 7fb90ed9d640 1 --2- 172.21.15.116:0/947492094 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb910156bb0 0x7fb910157020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:08:05.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.297+0000 7fb90e59c640 1 --2- 172.21.15.116:0/947492094 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9100c5200 0x7fb910154040 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fb900009870 tx=0x7fb90001c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f605637adf7a810c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:08:05.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.297+0000 7fb90dd9b640 1 -- 172.21.15.116:0/947492094 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9000b1070 con 0x7fb9100c5200 2024-04-25T00:08:05.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.297+0000 7fb90dd9b640 1 -- 172.21.15.116:0/947492094 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9000ac620 con 0x7fb9100c5200 2024-04-25T00:08:05.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.297+0000 7fb90dd9b640 1 -- 172.21.15.116:0/947492094 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9000ac9f0 con 0x7fb9100c5200 2024-04-25T00:08:05.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.301+0000 7fb914eab640 1 -- 172.21.15.116:0/947492094 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9100c5200 msgr2=0x7fb910154040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:08:05.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.301+0000 7fb914eab640 1 --2- 172.21.15.116:0/947492094 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9100c5200 0x7fb910154040 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fb900009870 tx=0x7fb90001c4d0 comp rx=0 tx=0).stop 2024-04-25T00:08:05.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.301+0000 7fb914eab640 1 -- 172.21.15.116:0/947492094 shutdown_connections 2024-04-25T00:08:05.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.301+0000 7fb914eab640 1 --2- 172.21.15.116:0/947492094 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb910156bb0 0x7fb910157020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:05.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.301+0000 7fb914eab640 1 --2- 172.21.15.116:0/947492094 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9100c5200 0x7fb910154040 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:05.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.301+0000 7fb914eab640 1 -- 172.21.15.116:0/947492094 >> 172.21.15.116:0/947492094 conn(0x7fb91006c720 msgr2=0x7fb91006cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:08:05.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.301+0000 7fb914eab640 1 -- 172.21.15.116:0/947492094 shutdown_connections 2024-04-25T00:08:05.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.301+0000 7fb914eab640 1 -- 172.21.15.116:0/947492094 wait complete. 2024-04-25T00:08:05.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.301+0000 7fb914eab640 1 Processor -- start 2024-04-25T00:08:05.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.301+0000 7fb914eab640 1 -- start start 2024-04-25T00:08:05.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.301+0000 7fb914eab640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9100c4760 0x7fb91013f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:08:05.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.301+0000 7fb914eab640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb9100c5200 0x7fb9100acdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:08:05.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.301+0000 7fb914eab640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb910156bb0 0x7fb9100c1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:08:05.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.301+0000 7fb914eab640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb91014dac0 con 0x7fb9100c4760 2024-04-25T00:08:05.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.301+0000 7fb914eab640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb91014d940 con 0x7fb910156bb0 2024-04-25T00:08:05.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.301+0000 7fb914eab640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fb910157560 con 0x7fb9100c5200 2024-04-25T00:08:05.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.301+0000 7fb90e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9100c4760 0x7fb91013f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:08:05.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.301+0000 7fb90e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9100c4760 0x7fb91013f690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45352/0 (socket says 172.21.15.116:45352) 2024-04-25T00:08:05.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.301+0000 7fb90e59c640 1 -- 172.21.15.116:0/3596776241 learned_addr learned my addr 172.21.15.116:0/3596776241 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:08:05.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.301+0000 7fb905d9b640 1 --2- 172.21.15.116:0/3596776241 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb9100c5200 0x7fb9100acdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:08:05.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.301+0000 7fb90ed9d640 1 --2- 172.21.15.116:0/3596776241 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb910156bb0 0x7fb9100c1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:08:05.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.301+0000 7fb90e59c640 1 -- 172.21.15.116:0/3596776241 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb9100c5200 msgr2=0x7fb9100acdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:08:05.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.301+0000 7fb90e59c640 1 --2- 172.21.15.116:0/3596776241 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb9100c5200 0x7fb9100acdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:05.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.301+0000 7fb90e59c640 1 -- 172.21.15.116:0/3596776241 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb910156bb0 msgr2=0x7fb9100c1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:08:05.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.301+0000 7fb90e59c640 1 --2- 172.21.15.116:0/3596776241 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb910156bb0 0x7fb9100c1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:05.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.301+0000 7fb90e59c640 1 -- 172.21.15.116:0/3596776241 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9100ad2f0 con 0x7fb9100c4760 2024-04-25T00:08:05.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.301+0000 7fb90e59c640 1 --2- 172.21.15.116:0/3596776241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9100c4760 0x7fb91013f690 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fb9000af040 tx=0x7fb90001ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:08:05.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.301+0000 7fb9077fe640 1 -- 172.21.15.116:0/3596776241 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9000bb020 con 0x7fb9100c4760 2024-04-25T00:08:05.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.301+0000 7fb9077fe640 1 -- 172.21.15.116:0/3596776241 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9000b7640 con 0x7fb9100c4760 2024-04-25T00:08:05.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.301+0000 7fb9077fe640 1 -- 172.21.15.116:0/3596776241 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9000b1040 con 0x7fb9100c4760 2024-04-25T00:08:05.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.301+0000 7fb914eab640 1 -- 172.21.15.116:0/3596776241 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9100c2360 con 0x7fb9100c4760 2024-04-25T00:08:05.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.301+0000 7fb914eab640 1 -- 172.21.15.116:0/3596776241 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9101f1fd0 con 0x7fb9100c4760 2024-04-25T00:08:05.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.305+0000 7fb914eab640 1 -- 172.21.15.116:0/3596776241 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb910068a60 con 0x7fb9100c4760 2024-04-25T00:08:05.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.305+0000 7fb9077fe640 1 -- 172.21.15.116:0/3596776241 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb9000be020 con 0x7fb9100c4760 2024-04-25T00:08:05.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.305+0000 7fb9077fe640 1 --2- 172.21.15.116:0/3596776241 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb8f00678c0 0x7fb8f0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:08:05.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.305+0000 7fb9077fe640 1 -- 172.21.15.116:0/3596776241 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7440+0+0 (secure 0 0 0) 0x7fb9001205e0 con 0x7fb9100c4760 2024-04-25T00:08:05.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.305+0000 7fb905d9b640 1 --2- 172.21.15.116:0/3596776241 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb8f00678c0 0x7fb8f0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:08:05.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.305+0000 7fb905d9b640 1 --2- 172.21.15.116:0/3596776241 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb8f00678c0 0x7fb8f0069d80 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fb91006fd20 tx=0x7fb8f8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:08:05.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.309+0000 7fb9077fe640 1 -- 172.21.15.116:0/3596776241 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fb9000c5030 con 0x7fb9100c4760 2024-04-25T00:08:05.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.513+0000 7fb914eab640 1 -- 172.21.15.116:0/3596776241 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb9100ab880 con 0x7fb9100c4760 2024-04-25T00:08:05.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.513+0000 7fb9077fe640 1 -- 172.21.15.116:0/3596776241 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) ==== 94+0+5673 (secure 0 0 0) 0x7fb9100ad2f0 con 0x7fb9100c4760 2024-04-25T00:08:05.514 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:08:05.514 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:07:33.575034+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5775":{"gid":5775,"name":"c","rank":0,"incarnation":52,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6833/3006671671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3006671671},{"type":"v1","addr":"172.21.15.116:6833","nonce":3006671671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5775]},"id":1},{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:08:03.679028+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5868},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5868":{"gid":5868,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6835/1670061091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1670061091},{"type":"v1","addr":"172.21.15.116:6835","nonce":1670061091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5868,"qdb_cluster":[5868]},"id":2}]} 2024-04-25T00:08:05.514 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 60 2024-04-25T00:08:05.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.513+0000 7fb914eab640 1 -- 172.21.15.116:0/3596776241 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb8f00678c0 msgr2=0x7fb8f0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:08:05.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.513+0000 7fb914eab640 1 --2- 172.21.15.116:0/3596776241 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb8f00678c0 0x7fb8f0069d80 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fb91006fd20 tx=0x7fb8f8009040 comp rx=0 tx=0).stop 2024-04-25T00:08:05.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.513+0000 7fb914eab640 1 -- 172.21.15.116:0/3596776241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9100c4760 msgr2=0x7fb91013f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:08:05.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.513+0000 7fb914eab640 1 --2- 172.21.15.116:0/3596776241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9100c4760 0x7fb91013f690 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fb9000af040 tx=0x7fb90001ad10 comp rx=0 tx=0).stop 2024-04-25T00:08:05.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.513+0000 7fb914eab640 1 -- 172.21.15.116:0/3596776241 shutdown_connections 2024-04-25T00:08:05.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.513+0000 7fb914eab640 1 --2- 172.21.15.116:0/3596776241 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb8f00678c0 0x7fb8f0069d80 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:05.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.513+0000 7fb914eab640 1 --2- 172.21.15.116:0/3596776241 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb910156bb0 0x7fb9100c1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:05.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.513+0000 7fb914eab640 1 --2- 172.21.15.116:0/3596776241 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb9100c5200 0x7fb9100acdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:05.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.513+0000 7fb914eab640 1 --2- 172.21.15.116:0/3596776241 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9100c4760 0x7fb91013f690 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:05.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.513+0000 7fb914eab640 1 -- 172.21.15.116:0/3596776241 >> 172.21.15.116:0/3596776241 conn(0x7fb91006c720 msgr2=0x7fb910140660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:08:05.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.513+0000 7fb914eab640 1 -- 172.21.15.116:0/3596776241 shutdown_connections 2024-04-25T00:08:05.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:05.513+0000 7fb914eab640 1 -- 172.21.15.116:0/3596776241 wait complete. 2024-04-25T00:08:05.532 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2024-04-25T00:08:06.274 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:08:06.277 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:08:07.533 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:08:07.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.669+0000 7f793483c640 1 Processor -- start 2024-04-25T00:08:07.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.669+0000 7f793483c640 1 -- start start 2024-04-25T00:08:07.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.669+0000 7f793483c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f79300c3a60 0x7f7930152280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:08:07.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.669+0000 7f793483c640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7930151da0 con 0x7f79300c6230 2024-04-25T00:08:07.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.669+0000 7f793483c640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79300c6610 con 0x7f79301527c0 2024-04-25T00:08:07.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.669+0000 7f793483c640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f793014f980 con 0x7f79300c3a60 2024-04-25T00:08:07.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.669+0000 7f792dd9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f79300c3a60 0x7f7930152280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:08:07.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.669+0000 7f792dd9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f79300c3a60 0x7f7930152280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45362/0 (socket says 172.21.15.116:45362) 2024-04-25T00:08:07.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.669+0000 7f792dd9b640 1 -- 172.21.15.116:0/2105880917 learned_addr learned my addr 172.21.15.116:0/2105880917 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:08:07.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.669+0000 7f792dd9b640 1 -- 172.21.15.116:0/2105880917 >> v1:172.21.15.174:6789/0 conn(0x7f79301527c0 legacy=0x7f793015c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:08:07.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.669+0000 7f792dd9b640 1 -- 172.21.15.116:0/2105880917 >> v1:172.21.15.116:6789/0 conn(0x7f79300c6230 legacy=0x7f793006c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:08:07.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.669+0000 7f792d59a640 1 -- 172.21.15.116:0/2105880917 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2696186108 0 0) 0x7f7930151da0 con 0x7f79300c6230 2024-04-25T00:08:07.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.669+0000 7f792dd9b640 1 -- 172.21.15.116:0/2105880917 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79300c6bc0 con 0x7f79300c3a60 2024-04-25T00:08:07.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.673+0000 7f792dd9b640 1 --2- 172.21.15.116:0/2105880917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79300c3a60 0x7f7930152280 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f792400bb10 tx=0x7f792401eb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3a2057955a68daa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:08:07.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.673+0000 7f792d59a640 1 -- 172.21.15.116:0/2105880917 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79240b4070 con 0x7f79300c3a60 2024-04-25T00:08:07.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.673+0000 7f792d59a640 1 -- 172.21.15.116:0/2105880917 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79240a7a50 con 0x7f79300c3a60 2024-04-25T00:08:07.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.673+0000 7f792d59a640 1 -- 172.21.15.116:0/2105880917 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79240a7e20 con 0x7f79300c3a60 2024-04-25T00:08:07.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.673+0000 7f793483c640 1 -- 172.21.15.116:0/2105880917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79300c3a60 msgr2=0x7f7930152280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:08:07.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.673+0000 7f793483c640 1 --2- 172.21.15.116:0/2105880917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79300c3a60 0x7f7930152280 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f792400bb10 tx=0x7f792401eb10 comp rx=0 tx=0).stop 2024-04-25T00:08:07.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.673+0000 7f793483c640 1 -- 172.21.15.116:0/2105880917 shutdown_connections 2024-04-25T00:08:07.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.673+0000 7f793483c640 1 --2- 172.21.15.116:0/2105880917 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79300c3a60 0x7f7930152280 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:07.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.673+0000 7f793483c640 1 -- 172.21.15.116:0/2105880917 >> 172.21.15.116:0/2105880917 conn(0x7f79300c42e0 msgr2=0x7f79300c46f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:08:07.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.673+0000 7f793483c640 1 -- 172.21.15.116:0/2105880917 shutdown_connections 2024-04-25T00:08:07.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.673+0000 7f793483c640 1 -- 172.21.15.116:0/2105880917 wait complete. 2024-04-25T00:08:07.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.677+0000 7f793483c640 1 Processor -- start 2024-04-25T00:08:07.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.677+0000 7f793483c640 1 -- start start 2024-04-25T00:08:07.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.677+0000 7f793483c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79300c3a60 0x7f7930147210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:08:07.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.677+0000 7f793483c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79300c6230 0x7f79300ab780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:08:07.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.677+0000 7f793483c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f79301527c0 0x7f79300abcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:08:07.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.677+0000 7f793483c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f793006c930 con 0x7f79300c6230 2024-04-25T00:08:07.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.677+0000 7f793483c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f79300c1d80 con 0x7f79300c3a60 2024-04-25T00:08:07.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.677+0000 7f793483c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f793006cab0 con 0x7f79301527c0 2024-04-25T00:08:07.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.677+0000 7f792dd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79300c6230 0x7f79300ab780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:08:07.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.677+0000 7f792dd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79300c6230 0x7f79300ab780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45364/0 (socket says 172.21.15.116:45364) 2024-04-25T00:08:07.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.677+0000 7f792dd9b640 1 -- 172.21.15.116:0/22235706 learned_addr learned my addr 172.21.15.116:0/22235706 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:08:07.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.677+0000 7f792e59c640 1 --2- 172.21.15.116:0/22235706 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79300c3a60 0x7f7930147210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:08:07.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.677+0000 7f792ed9d640 1 --2- 172.21.15.116:0/22235706 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f79301527c0 0x7f79300abcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:08:07.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.677+0000 7f792dd9b640 1 -- 172.21.15.116:0/22235706 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f79301527c0 msgr2=0x7f79300abcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:08:07.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.677+0000 7f792dd9b640 1 --2- 172.21.15.116:0/22235706 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f79301527c0 0x7f79300abcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:07.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.677+0000 7f792dd9b640 1 -- 172.21.15.116:0/22235706 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79300c3a60 msgr2=0x7f7930147210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:08:07.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.677+0000 7f792dd9b640 1 --2- 172.21.15.116:0/22235706 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79300c3a60 0x7f7930147210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:07.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.677+0000 7f792dd9b640 1 -- 172.21.15.116:0/22235706 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79301e9db0 con 0x7f79300c6230 2024-04-25T00:08:07.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.677+0000 7f792dd9b640 1 --2- 172.21.15.116:0/22235706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79300c6230 0x7f79300ab780 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f792400b2c0 tx=0x7f792401b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:08:07.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.677+0000 7f791effd640 1 -- 172.21.15.116:0/22235706 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79240be070 con 0x7f79300c6230 2024-04-25T00:08:07.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.677+0000 7f791effd640 1 -- 172.21.15.116:0/22235706 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79240afda0 con 0x7f79300c6230 2024-04-25T00:08:07.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.677+0000 7f793483c640 1 -- 172.21.15.116:0/22235706 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79301ea090 con 0x7f79300c6230 2024-04-25T00:08:07.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.677+0000 7f791effd640 1 -- 172.21.15.116:0/22235706 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79240b4040 con 0x7f79300c6230 2024-04-25T00:08:07.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.677+0000 7f793483c640 1 -- 172.21.15.116:0/22235706 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79301ea650 con 0x7f79300c6230 2024-04-25T00:08:07.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.677+0000 7f793483c640 1 -- 172.21.15.116:0/22235706 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7930068a60 con 0x7f79300c6230 2024-04-25T00:08:07.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.681+0000 7f791effd640 1 -- 172.21.15.116:0/22235706 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f79240a7910 con 0x7f79300c6230 2024-04-25T00:08:07.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.681+0000 7f791effd640 1 --2- 172.21.15.116:0/22235706 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f79100678c0 0x7f7910069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:08:07.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.681+0000 7f791effd640 1 -- 172.21.15.116:0/22235706 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7440+0+0 (secure 0 0 0) 0x7f79241233b0 con 0x7f79300c6230 2024-04-25T00:08:07.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.681+0000 7f792e59c640 1 --2- 172.21.15.116:0/22235706 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f79100678c0 0x7f7910069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:08:07.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.681+0000 7f792e59c640 1 --2- 172.21.15.116:0/22235706 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f79100678c0 0x7f7910069d80 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f7920000f70 tx=0x7f7920009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:08:07.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.685+0000 7f791effd640 1 -- 172.21.15.116:0/22235706 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f7924018820 con 0x7f79300c6230 2024-04-25T00:08:07.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.913+0000 7f793483c640 1 -- 172.21.15.116:0/22235706 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f793006cc30 con 0x7f79300c6230 2024-04-25T00:08:07.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.913+0000 7f791effd640 1 -- 172.21.15.116:0/22235706 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+6501 (secure 0 0 0) 0x7f79240f3270 con 0x7f79300c6230 2024-04-25T00:08:07.915 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:08:07.915 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5236,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/1159002662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1159002662},{"type":"v1","addr":"172.21.15.174:6837","nonce":1159002662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:07:33.575034+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5775":{"gid":5775,"name":"c","rank":0,"incarnation":52,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6833/3006671671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3006671671},{"type":"v1","addr":"172.21.15.116:6833","nonce":3006671671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5775]},"id":1},{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:08:03.679028+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5868},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5868":{"gid":5868,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6835/1670061091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1670061091},{"type":"v1","addr":"172.21.15.116:6835","nonce":1670061091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5868,"qdb_cluster":[5868]},"id":2}]} 2024-04-25T00:08:07.915 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 61 2024-04-25T00:08:07.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.917+0000 7f793483c640 1 -- 172.21.15.116:0/22235706 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f79100678c0 msgr2=0x7f7910069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:08:07.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.917+0000 7f793483c640 1 --2- 172.21.15.116:0/22235706 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f79100678c0 0x7f7910069d80 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f7920000f70 tx=0x7f7920009040 comp rx=0 tx=0).stop 2024-04-25T00:08:07.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.917+0000 7f793483c640 1 -- 172.21.15.116:0/22235706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79300c6230 msgr2=0x7f79300ab780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:08:07.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.917+0000 7f793483c640 1 --2- 172.21.15.116:0/22235706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79300c6230 0x7f79300ab780 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f792400b2c0 tx=0x7f792401b8b0 comp rx=0 tx=0).stop 2024-04-25T00:08:07.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.917+0000 7f793483c640 1 -- 172.21.15.116:0/22235706 shutdown_connections 2024-04-25T00:08:07.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.917+0000 7f793483c640 1 --2- 172.21.15.116:0/22235706 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f79100678c0 0x7f7910069d80 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:07.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.917+0000 7f793483c640 1 --2- 172.21.15.116:0/22235706 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f79301527c0 0x7f79300abcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:07.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.917+0000 7f793483c640 1 --2- 172.21.15.116:0/22235706 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79300c6230 0x7f79300ab780 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:07.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.917+0000 7f793483c640 1 --2- 172.21.15.116:0/22235706 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79300c3a60 0x7f7930147210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:08:07.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.917+0000 7f793483c640 1 -- 172.21.15.116:0/22235706 >> 172.21.15.116:0/22235706 conn(0x7f79300c42e0 msgr2=0x7f793006bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:08:07.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.917+0000 7f793483c640 1 -- 172.21.15.116:0/22235706 shutdown_connections 2024-04-25T00:08:07.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:08:07.917+0000 7f793483c640 1 -- 172.21.15.116:0/22235706 wait complete. 2024-04-25T00:08:07.936 INFO:tasks.mds_thrash.fs.[b]:mds.b reported in up:standby state 2024-04-25T00:08:07.936 INFO:tasks.mds_thrash.fs.[b]:waiting for 68 secs before thrashing 2024-04-25T00:08:36.327 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:08:36.334 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:09:06.743 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:09:06.746 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:09:15.681 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:09:15.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.812+0000 7f91e7511640 1 Processor -- start 2024-04-25T00:09:15.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.812+0000 7f91e7511640 1 -- start start 2024-04-25T00:09:15.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.812+0000 7f91e7511640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f91e00aeb20 0x7f91e00c9310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:15.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.812+0000 7f91e7511640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f91e00c9850 0x7f91e0150870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:15.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.812+0000 7f91e7511640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91e01535d0 con 0x7f91e0150db0 2024-04-25T00:09:15.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.812+0000 7f91e7511640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f91e0153850 con 0x7f91e00c9850 2024-04-25T00:09:15.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.812+0000 7f91e7511640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f91e0153b30 con 0x7f91e00aeb20 2024-04-25T00:09:15.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.812+0000 7f91e4aac640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f91e00c9850 0x7f91e0150870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:15.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.812+0000 7f91e5aae640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f91e0150db0 0x7f91e00c5270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:36054/0 (socket says 172.21.15.116:36054) 2024-04-25T00:09:15.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.812+0000 7f91e4aac640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f91e00c9850 0x7f91e0150870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41130/0 (socket says 172.21.15.116:41130) 2024-04-25T00:09:15.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.812+0000 7f91e5aae640 1 -- 172.21.15.116:0/2782074841 learned_addr learned my addr 172.21.15.116:0/2782074841 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:09:15.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.812+0000 7f91e52ad640 1 --2- 172.21.15.116:0/2782074841 >> v2:172.21.15.174:3300/0 conn(0x7f91e00aeb20 0x7f91e00c9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:15.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.816+0000 7f91e4aac640 1 -- 172.21.15.116:0/2782074841 >> v2:172.21.15.174:3300/0 conn(0x7f91e00aeb20 msgr2=0x7f91e00c9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:09:15.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.816+0000 7f91e4aac640 1 --2- 172.21.15.116:0/2782074841 >> v2:172.21.15.174:3300/0 conn(0x7f91e00aeb20 0x7f91e00c9310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:15.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.816+0000 7f91e4aac640 1 -- 172.21.15.116:0/2782074841 >> v1:172.21.15.116:6789/0 conn(0x7f91e0150db0 legacy=0x7f91e00c5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:15.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.816+0000 7f91e4aac640 1 -- 172.21.15.116:0/2782074841 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91e0153e10 con 0x7f91e00c9850 2024-04-25T00:09:15.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.816+0000 7f91d7fff640 1 -- 172.21.15.116:0/2782074841 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 848483387 0 0) 0x7f91d80014a0 con 0x7f91e0150db0 2024-04-25T00:09:15.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.816+0000 7f91e4aac640 1 --2- 172.21.15.116:0/2782074841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91e00c9850 0x7f91e0150870 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f91e00705f0 tx=0x7f91dc01c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51fe2aed43714bdb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:15.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.816+0000 7f91d7fff640 1 -- 172.21.15.116:0/2782074841 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91dc0b0070 con 0x7f91e00c9850 2024-04-25T00:09:15.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.816+0000 7f91d7fff640 1 -- 172.21.15.116:0/2782074841 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91dc0ab600 con 0x7f91e00c9850 2024-04-25T00:09:15.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.816+0000 7f91d7fff640 1 -- 172.21.15.116:0/2782074841 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91dc0ab9d0 con 0x7f91e00c9850 2024-04-25T00:09:15.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.816+0000 7f91e7511640 1 -- 172.21.15.116:0/2782074841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91e00c9850 msgr2=0x7f91e0150870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:15.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.816+0000 7f91e7511640 1 --2- 172.21.15.116:0/2782074841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91e00c9850 0x7f91e0150870 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f91e00705f0 tx=0x7f91dc01c3f0 comp rx=0 tx=0).stop 2024-04-25T00:09:15.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.820+0000 7f91e7511640 1 -- 172.21.15.116:0/2782074841 shutdown_connections 2024-04-25T00:09:15.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.820+0000 7f91e7511640 1 --2- 172.21.15.116:0/2782074841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91e00c9850 0x7f91e0150870 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:15.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.820+0000 7f91e7511640 1 --2- 172.21.15.116:0/2782074841 >> v2:172.21.15.174:3300/0 conn(0x7f91e00aeb20 0x7f91e00c9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:15.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.820+0000 7f91e7511640 1 -- 172.21.15.116:0/2782074841 >> 172.21.15.116:0/2782074841 conn(0x7f91e0074650 msgr2=0x7f91e0074a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:09:15.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.820+0000 7f91e7511640 1 -- 172.21.15.116:0/2782074841 shutdown_connections 2024-04-25T00:09:15.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.820+0000 7f91e7511640 1 -- 172.21.15.116:0/2782074841 wait complete. 2024-04-25T00:09:15.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.820+0000 7f91e7511640 1 Processor -- start 2024-04-25T00:09:15.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.820+0000 7f91e7511640 1 -- start start 2024-04-25T00:09:15.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.820+0000 7f91e7511640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f91e00aeb20 0x7f91e01efcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:15.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.820+0000 7f91e7511640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91e00c9850 0x7f91e00c8440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:15.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.820+0000 7f91e7511640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f91e0150db0 0x7f91e014b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:15.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.820+0000 7f91e7511640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f91e00c5f10 con 0x7f91e00c9850 2024-04-25T00:09:15.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.820+0000 7f91e7511640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f91e00c5d90 con 0x7f91e00aeb20 2024-04-25T00:09:15.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.820+0000 7f91e7511640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f91e00aef00 con 0x7f91e0150db0 2024-04-25T00:09:15.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.820+0000 7f91e4aac640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91e00c9850 0x7f91e00c8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:15.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.820+0000 7f91e4aac640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91e00c9850 0x7f91e00c8440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41144/0 (socket says 172.21.15.116:41144) 2024-04-25T00:09:15.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.820+0000 7f91e4aac640 1 -- 172.21.15.116:0/580724697 learned_addr learned my addr 172.21.15.116:0/580724697 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:09:15.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.820+0000 7f91e5aae640 1 --2- 172.21.15.116:0/580724697 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f91e0150db0 0x7f91e014b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:15.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.820+0000 7f91e52ad640 1 --2- 172.21.15.116:0/580724697 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f91e00aeb20 0x7f91e01efcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:15.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.820+0000 7f91e4aac640 1 -- 172.21.15.116:0/580724697 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f91e0150db0 msgr2=0x7f91e014b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:15.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.820+0000 7f91e4aac640 1 --2- 172.21.15.116:0/580724697 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f91e0150db0 0x7f91e014b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:15.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.820+0000 7f91e4aac640 1 -- 172.21.15.116:0/580724697 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f91e00aeb20 msgr2=0x7f91e01efcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:15.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.820+0000 7f91e4aac640 1 --2- 172.21.15.116:0/580724697 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f91e00aeb20 0x7f91e01efcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:15.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.820+0000 7f91e4aac640 1 -- 172.21.15.116:0/580724697 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91e01ef220 con 0x7f91e00c9850 2024-04-25T00:09:15.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.820+0000 7f91e5aae640 1 --2- 172.21.15.116:0/580724697 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f91e0150db0 0x7f91e014b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:09:15.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.820+0000 7f91e4aac640 1 --2- 172.21.15.116:0/580724697 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91e00c9850 0x7f91e00c8440 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f91dc0ae040 tx=0x7f91dc01ac30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:15.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.820+0000 7f91d5ffb640 1 -- 172.21.15.116:0/580724697 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91dc0bc070 con 0x7f91e00c9850 2024-04-25T00:09:15.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.820+0000 7f91d5ffb640 1 -- 172.21.15.116:0/580724697 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91dc0b1040 con 0x7f91e00c9850 2024-04-25T00:09:15.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.820+0000 7f91e7511640 1 -- 172.21.15.116:0/580724697 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91e014af60 con 0x7f91e00c9850 2024-04-25T00:09:15.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.820+0000 7f91d5ffb640 1 -- 172.21.15.116:0/580724697 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91dc0b0050 con 0x7f91e00c9850 2024-04-25T00:09:15.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.820+0000 7f91e7511640 1 -- 172.21.15.116:0/580724697 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91e01ef4c0 con 0x7f91e00c9850 2024-04-25T00:09:15.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.824+0000 7f91e7511640 1 -- 172.21.15.116:0/580724697 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91e0068a60 con 0x7f91e00c9850 2024-04-25T00:09:15.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.824+0000 7f91d5ffb640 1 -- 172.21.15.116:0/580724697 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f91dc0bf070 con 0x7f91e00c9850 2024-04-25T00:09:15.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.824+0000 7f91d5ffb640 1 --2- 172.21.15.116:0/580724697 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f91c40678c0 0x7f91c4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:15.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.824+0000 7f91d5ffb640 1 -- 172.21.15.116:0/580724697 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7440+0+0 (secure 0 0 0) 0x7f91dc11fef0 con 0x7f91e00c9850 2024-04-25T00:09:15.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.824+0000 7f91e52ad640 1 --2- 172.21.15.116:0/580724697 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f91c40678c0 0x7f91c4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:15.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.824+0000 7f91e52ad640 1 --2- 172.21.15.116:0/580724697 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f91c40678c0 0x7f91c4069d80 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f91e0067860 tx=0x7f91d0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:15.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:15.828+0000 7f91d5ffb640 1 -- 172.21.15.116:0/580724697 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f91dc016070 con 0x7f91e00c9850 2024-04-25T00:09:15.937 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:09:16.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.048+0000 7f82ae10c640 1 Processor -- start 2024-04-25T00:09:16.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.048+0000 7f82ae10c640 1 -- start start 2024-04-25T00:09:16.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.052+0000 7f82ae10c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f82a806c320 0x7f82a80a8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:16.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.052+0000 7f82ae10c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f82a806c760 0x7f82a814f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:16.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.052+0000 7f82ae10c640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82a8071a30 con 0x7f82a81527b0 2024-04-25T00:09:16.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.052+0000 7f82ae10c640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f82a80c2b10 con 0x7f82a806c760 2024-04-25T00:09:16.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.052+0000 7f82ae10c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f82a80a8fe0 con 0x7f82a806c320 2024-04-25T00:09:16.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.052+0000 7f82a7fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f82a806c760 0x7f82a814f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:16.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.052+0000 7f82a77fe640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f82a81527b0 0x7f82a80c3280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:50176/0 (socket says 172.21.15.116:50176) 2024-04-25T00:09:16.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.052+0000 7f82a77fe640 1 -- 172.21.15.116:0/934736028 learned_addr learned my addr 172.21.15.116:0/934736028 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:09:16.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.052+0000 7f82a6ffd640 1 --2- 172.21.15.116:0/934736028 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f82a806c320 0x7f82a80a8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:16.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.052+0000 7f82a7fff640 1 -- 172.21.15.116:0/934736028 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f82a806c320 msgr2=0x7f82a80a8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:09:16.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.052+0000 7f82a7fff640 1 --2- 172.21.15.116:0/934736028 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f82a806c320 0x7f82a80a8aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:16.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.052+0000 7f82a7fff640 1 -- 172.21.15.116:0/934736028 >> v1:172.21.15.174:6789/0 conn(0x7f82a81527b0 legacy=0x7f82a80c3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:16.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.052+0000 7f82a7fff640 1 -- 172.21.15.116:0/934736028 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82a80c1dc0 con 0x7f82a806c760 2024-04-25T00:09:16.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.052+0000 7f82a6ffd640 1 --2- 172.21.15.116:0/934736028 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f82a806c320 0x7f82a80a8aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:09:16.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.056+0000 7f82a7fff640 1 --2- 172.21.15.116:0/934736028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82a806c760 0x7f82a814f8a0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f82a0009d90 tx=0x7f82a001c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acde355ead8d8753 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:16.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.056+0000 7f82a67fc640 1 -- 172.21.15.116:0/934736028 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82a00b2070 con 0x7f82a806c760 2024-04-25T00:09:16.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.056+0000 7f82a67fc640 1 -- 172.21.15.116:0/934736028 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82a00a5830 con 0x7f82a806c760 2024-04-25T00:09:16.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.056+0000 7f82a67fc640 1 -- 172.21.15.116:0/934736028 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82a00a5c00 con 0x7f82a806c760 2024-04-25T00:09:16.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.056+0000 7f82ae10c640 1 -- 172.21.15.116:0/934736028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82a806c760 msgr2=0x7f82a814f8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:16.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.056+0000 7f82ae10c640 1 --2- 172.21.15.116:0/934736028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82a806c760 0x7f82a814f8a0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f82a0009d90 tx=0x7f82a001c970 comp rx=0 tx=0).stop 2024-04-25T00:09:16.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.056+0000 7f82ae10c640 1 -- 172.21.15.116:0/934736028 shutdown_connections 2024-04-25T00:09:16.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.056+0000 7f82ae10c640 1 --2- 172.21.15.116:0/934736028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82a806c760 0x7f82a814f8a0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:16.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.056+0000 7f82ae10c640 1 --2- 172.21.15.116:0/934736028 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f82a806c320 0x7f82a80a8aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:16.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.056+0000 7f82ae10c640 1 -- 172.21.15.116:0/934736028 >> 172.21.15.116:0/934736028 conn(0x7f82a80c4200 msgr2=0x7f82a80c4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:09:16.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.056+0000 7f82ae10c640 1 -- 172.21.15.116:0/934736028 shutdown_connections 2024-04-25T00:09:16.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.056+0000 7f82ae10c640 1 -- 172.21.15.116:0/934736028 wait complete. 2024-04-25T00:09:16.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.056+0000 7f82ae10c640 1 Processor -- start 2024-04-25T00:09:16.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.056+0000 7f82ae10c640 1 -- start start 2024-04-25T00:09:16.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.060+0000 7f82ae10c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f82a806c320 0x7f82a80c52f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:16.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.060+0000 7f82ae10c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82a806c760 0x7f82a8148000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:16.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.060+0000 7f82ae10c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82a81527b0 0x7f82a81efc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:16.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.060+0000 7f82ae10c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f82a8152b90 con 0x7f82a806c760 2024-04-25T00:09:16.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.060+0000 7f82ae10c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f82a80c2d60 con 0x7f82a81527b0 2024-04-25T00:09:16.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.060+0000 7f82ae10c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f82a8152d10 con 0x7f82a806c320 2024-04-25T00:09:16.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.060+0000 7f82a6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82a806c760 0x7f82a8148000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:16.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.060+0000 7f82a6ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82a806c760 0x7f82a8148000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41152/0 (socket says 172.21.15.116:41152) 2024-04-25T00:09:16.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.060+0000 7f82a6ffd640 1 -- 172.21.15.116:0/3873626080 learned_addr learned my addr 172.21.15.116:0/3873626080 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:09:16.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.060+0000 7f82a7fff640 1 --2- 172.21.15.116:0/3873626080 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82a81527b0 0x7f82a81efc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:16.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.060+0000 7f82a77fe640 1 --2- 172.21.15.116:0/3873626080 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f82a806c320 0x7f82a80c52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:16.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.060+0000 7f82a6ffd640 1 -- 172.21.15.116:0/3873626080 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f82a806c320 msgr2=0x7f82a80c52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:16.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.060+0000 7f82a6ffd640 1 --2- 172.21.15.116:0/3873626080 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f82a806c320 0x7f82a80c52f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:16.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.060+0000 7f82a6ffd640 1 -- 172.21.15.116:0/3873626080 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82a81527b0 msgr2=0x7f82a81efc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:16.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.060+0000 7f82a6ffd640 1 --2- 172.21.15.116:0/3873626080 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82a81527b0 0x7f82a81efc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:16.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.060+0000 7f82a6ffd640 1 -- 172.21.15.116:0/3873626080 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82a814d3e0 con 0x7f82a806c760 2024-04-25T00:09:16.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.060+0000 7f82a6ffd640 1 --2- 172.21.15.116:0/3873626080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82a806c760 0x7f82a8148000 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f829c00e810 tx=0x7f829c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:16.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.060+0000 7f828ffff640 1 -- 172.21.15.116:0/3873626080 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f829c017070 con 0x7f82a806c760 2024-04-25T00:09:16.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.060+0000 7f828ffff640 1 -- 172.21.15.116:0/3873626080 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f829c00c8d0 con 0x7f82a806c760 2024-04-25T00:09:16.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.060+0000 7f828ffff640 1 -- 172.21.15.116:0/3873626080 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f829c00f040 con 0x7f82a806c760 2024-04-25T00:09:16.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.060+0000 7f82ae10c640 1 -- 172.21.15.116:0/3873626080 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82a8144db0 con 0x7f82a806c760 2024-04-25T00:09:16.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.060+0000 7f82ae10c640 1 -- 172.21.15.116:0/3873626080 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82a8148540 con 0x7f82a806c760 2024-04-25T00:09:16.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.060+0000 7f82ae10c640 1 -- 172.21.15.116:0/3873626080 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82a8068a60 con 0x7f82a806c760 2024-04-25T00:09:16.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.060+0000 7f828ffff640 1 -- 172.21.15.116:0/3873626080 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f829c01a070 con 0x7f82a806c760 2024-04-25T00:09:16.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.060+0000 7f828ffff640 1 --2- 172.21.15.116:0/3873626080 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f82940678f0 0x7f8294069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:16.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.060+0000 7f828ffff640 1 -- 172.21.15.116:0/3873626080 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7440+0+0 (secure 0 0 0) 0x7f829c085e60 con 0x7f82a806c760 2024-04-25T00:09:16.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.064+0000 7f82a77fe640 1 --2- 172.21.15.116:0/3873626080 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f82940678f0 0x7f8294069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:16.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.064+0000 7f82a77fe640 1 --2- 172.21.15.116:0/3873626080 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f82940678f0 0x7f8294069db0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f8298005890 tx=0x7f8298009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:16.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.068+0000 7f828ffff640 1 -- 172.21.15.116:0/3873626080 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f829c056370 con 0x7f82a806c760 2024-04-25T00:09:16.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.112+0000 7f91e7511640 1 -- 172.21.15.116:0/580724697 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f91e006e530 con 0x7f91e00c9850 2024-04-25T00:09:16.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.112+0000 7f91d5ffb640 1 -- 172.21.15.116:0/580724697 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+6501 (secure 0 0 0) 0x7f91dc0f0400 con 0x7f91e00c9850 2024-04-25T00:09:16.116 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:09:16.116 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5236,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/1159002662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1159002662},{"type":"v1","addr":"172.21.15.174:6837","nonce":1159002662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:07:33.575034+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5775":{"gid":5775,"name":"c","rank":0,"incarnation":52,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6833/3006671671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3006671671},{"type":"v1","addr":"172.21.15.116:6833","nonce":3006671671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5775]},"id":1},{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:08:03.679028+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5868},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5868":{"gid":5868,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6835/1670061091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1670061091},{"type":"v1","addr":"172.21.15.116:6835","nonce":1670061091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5868,"qdb_cluster":[5868]},"id":2}]} 2024-04-25T00:09:16.116 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 61 2024-04-25T00:09:16.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.116+0000 7f91e7511640 1 -- 172.21.15.116:0/580724697 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f91c40678c0 msgr2=0x7f91c4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:16.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.116+0000 7f91e7511640 1 --2- 172.21.15.116:0/580724697 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f91c40678c0 0x7f91c4069d80 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f91e0067860 tx=0x7f91d0008040 comp rx=0 tx=0).stop 2024-04-25T00:09:16.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.116+0000 7f91e7511640 1 -- 172.21.15.116:0/580724697 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91e00c9850 msgr2=0x7f91e00c8440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:16.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.116+0000 7f91e7511640 1 --2- 172.21.15.116:0/580724697 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91e00c9850 0x7f91e00c8440 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f91dc0ae040 tx=0x7f91dc01ac30 comp rx=0 tx=0).stop 2024-04-25T00:09:16.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.116+0000 7f91e7511640 1 -- 172.21.15.116:0/580724697 shutdown_connections 2024-04-25T00:09:16.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.116+0000 7f91e7511640 1 --2- 172.21.15.116:0/580724697 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f91c40678c0 0x7f91c4069d80 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:16.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.116+0000 7f91e7511640 1 --2- 172.21.15.116:0/580724697 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f91e0150db0 0x7f91e014b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:16.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.116+0000 7f91e7511640 1 --2- 172.21.15.116:0/580724697 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91e00c9850 0x7f91e00c8440 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:16.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.116+0000 7f91e7511640 1 --2- 172.21.15.116:0/580724697 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f91e00aeb20 0x7f91e01efcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:16.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.116+0000 7f91e7511640 1 -- 172.21.15.116:0/580724697 >> 172.21.15.116:0/580724697 conn(0x7f91e0074650 msgr2=0x7f91e014a5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:09:16.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.116+0000 7f91e7511640 1 -- 172.21.15.116:0/580724697 shutdown_connections 2024-04-25T00:09:16.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.116+0000 7f91e7511640 1 -- 172.21.15.116:0/580724697 wait complete. 2024-04-25T00:09:16.136 INFO:tasks.mds_thrash.fs.[a]:kill mds.c (rank=0) 2024-04-25T00:09:16.136 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-25T00:09:16.136 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:09:16.150 INFO:tasks.ceph.mds.c:Stopped 2024-04-25T00:09:16.151 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:09:16.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.316+0000 7f58cb307640 1 Processor -- start 2024-04-25T00:09:16.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.316+0000 7f58cb307640 1 -- start start 2024-04-25T00:09:16.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.316+0000 7f58cb307640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f58c4152d80 0x7f58c4159950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:16.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.316+0000 7f58cb307640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f58c4072080 0x7f58c41562a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:16.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.316+0000 7f58cb307640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58c40730b0 con 0x7f58c40c36b0 2024-04-25T00:09:16.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.316+0000 7f58cb307640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f58c41567e0 con 0x7f58c4072080 2024-04-25T00:09:16.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.316+0000 7f58cb307640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f58c4072e60 con 0x7f58c4152d80 2024-04-25T00:09:16.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.316+0000 7f58c88a2640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f58c4072080 0x7f58c41562a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:16.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.316+0000 7f58c88a2640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f58c4072080 0x7f58c41562a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41162/0 (socket says 172.21.15.116:41162) 2024-04-25T00:09:16.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.316+0000 7f58c88a2640 1 -- 172.21.15.116:0/2956496507 learned_addr learned my addr 172.21.15.116:0/2956496507 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:09:16.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.316+0000 7f58c90a3640 1 --2- 172.21.15.116:0/2956496507 >> v2:172.21.15.174:3300/0 conn(0x7f58c4152d80 0x7f58c4159950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:16.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.316+0000 7f58c88a2640 1 -- 172.21.15.116:0/2956496507 >> v2:172.21.15.174:3300/0 conn(0x7f58c4152d80 msgr2=0x7f58c4159950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:09:16.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.316+0000 7f58c88a2640 1 --2- 172.21.15.116:0/2956496507 >> v2:172.21.15.174:3300/0 conn(0x7f58c4152d80 0x7f58c4159950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:16.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.316+0000 7f58c88a2640 1 -- 172.21.15.116:0/2956496507 >> v1:172.21.15.174:6789/0 conn(0x7f58c40c36b0 legacy=0x7f58c40c5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:09:16.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.316+0000 7f58c88a2640 1 -- 172.21.15.116:0/2956496507 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58c4073390 con 0x7f58c4072080 2024-04-25T00:09:16.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.316+0000 7f58c90a3640 1 --2- 172.21.15.116:0/2956496507 >> v2:172.21.15.174:3300/0 conn(0x7f58c4152d80 0x7f58c4159950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:09:16.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.320+0000 7f58c88a2640 1 --2- 172.21.15.116:0/2956496507 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58c4072080 0x7f58c41562a0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f58c40c7fb0 tx=0x7f58b401c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe1a4087e2a01618 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:16.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.320+0000 7f58bbfff640 1 -- 172.21.15.116:0/2956496507 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58b40b0070 con 0x7f58c4072080 2024-04-25T00:09:16.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.320+0000 7f58bbfff640 1 -- 172.21.15.116:0/2956496507 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58b40ab600 con 0x7f58c4072080 2024-04-25T00:09:16.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.320+0000 7f58bbfff640 1 -- 172.21.15.116:0/2956496507 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58b40ab9d0 con 0x7f58c4072080 2024-04-25T00:09:16.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.320+0000 7f58cb307640 1 -- 172.21.15.116:0/2956496507 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58c4072080 msgr2=0x7f58c41562a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:16.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.320+0000 7f58cb307640 1 --2- 172.21.15.116:0/2956496507 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58c4072080 0x7f58c41562a0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f58c40c7fb0 tx=0x7f58b401c460 comp rx=0 tx=0).stop 2024-04-25T00:09:16.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.320+0000 7f58cb307640 1 -- 172.21.15.116:0/2956496507 shutdown_connections 2024-04-25T00:09:16.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.320+0000 7f58cb307640 1 --2- 172.21.15.116:0/2956496507 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58c4072080 0x7f58c41562a0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:16.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.320+0000 7f58cb307640 1 --2- 172.21.15.116:0/2956496507 >> v2:172.21.15.174:3300/0 conn(0x7f58c4152d80 0x7f58c4159950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:16.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.320+0000 7f58cb307640 1 -- 172.21.15.116:0/2956496507 >> 172.21.15.116:0/2956496507 conn(0x7f58c406c720 msgr2=0x7f58c406cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:09:16.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.320+0000 7f58cb307640 1 -- 172.21.15.116:0/2956496507 shutdown_connections 2024-04-25T00:09:16.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.320+0000 7f58cb307640 1 -- 172.21.15.116:0/2956496507 wait complete. 2024-04-25T00:09:16.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.320+0000 7f58cb307640 1 Processor -- start 2024-04-25T00:09:16.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.320+0000 7f58cb307640 1 -- start start 2024-04-25T00:09:16.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.324+0000 7f58cb307640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58c4072080 0x7f58c414bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:16.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.324+0000 7f58cb307640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f58c40c36b0 0x7f58c414b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:16.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.324+0000 7f58cb307640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58c4152d80 0x7f58c414fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:16.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.324+0000 7f58cb307640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f58c40c58b0 con 0x7f58c4072080 2024-04-25T00:09:16.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.324+0000 7f58cb307640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f58c40c5730 con 0x7f58c4152d80 2024-04-25T00:09:16.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.324+0000 7f58cb307640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f58c4153160 con 0x7f58c40c36b0 2024-04-25T00:09:16.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.324+0000 7f58c90a3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58c4072080 0x7f58c414bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:16.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.324+0000 7f58c90a3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58c4072080 0x7f58c414bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41174/0 (socket says 172.21.15.116:41174) 2024-04-25T00:09:16.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.324+0000 7f58c90a3640 1 -- 172.21.15.116:0/1016638835 learned_addr learned my addr 172.21.15.116:0/1016638835 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:09:16.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.324+0000 7f58c88a2640 1 --2- 172.21.15.116:0/1016638835 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f58c40c36b0 0x7f58c414b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:16.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.324+0000 7f58c98a4640 1 --2- 172.21.15.116:0/1016638835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58c4152d80 0x7f58c414fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:16.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.324+0000 7f58c90a3640 1 -- 172.21.15.116:0/1016638835 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f58c40c36b0 msgr2=0x7f58c414b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:16.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.324+0000 7f58c90a3640 1 --2- 172.21.15.116:0/1016638835 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f58c40c36b0 0x7f58c414b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:16.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.324+0000 7f58c90a3640 1 -- 172.21.15.116:0/1016638835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58c4152d80 msgr2=0x7f58c414fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:16.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.324+0000 7f58c90a3640 1 --2- 172.21.15.116:0/1016638835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58c4152d80 0x7f58c414fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:16.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.324+0000 7f58c90a3640 1 -- 172.21.15.116:0/1016638835 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58c41f2e90 con 0x7f58c4072080 2024-04-25T00:09:16.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.324+0000 7f58c90a3640 1 --2- 172.21.15.116:0/1016638835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58c4072080 0x7f58c414bf00 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f58c000d800 tx=0x7f58c000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:16.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.324+0000 7f58b9ffb640 1 -- 172.21.15.116:0/1016638835 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58c000b6c0 con 0x7f58c4072080 2024-04-25T00:09:16.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.324+0000 7f58b9ffb640 1 -- 172.21.15.116:0/1016638835 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58c0019070 con 0x7f58c4072080 2024-04-25T00:09:16.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.324+0000 7f58cb307640 1 -- 172.21.15.116:0/1016638835 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58c414b3a0 con 0x7f58c4072080 2024-04-25T00:09:16.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.324+0000 7f58b9ffb640 1 -- 172.21.15.116:0/1016638835 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58c000baa0 con 0x7f58c4072080 2024-04-25T00:09:16.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.324+0000 7f58cb307640 1 -- 172.21.15.116:0/1016638835 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58c414cb80 con 0x7f58c4072080 2024-04-25T00:09:16.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.324+0000 7f58cb307640 1 -- 172.21.15.116:0/1016638835 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58a40052a0 con 0x7f58c4072080 2024-04-25T00:09:16.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.328+0000 7f58b9ffb640 1 -- 172.21.15.116:0/1016638835 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f58c000bc40 con 0x7f58c4072080 2024-04-25T00:09:16.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.328+0000 7f58b9ffb640 1 --2- 172.21.15.116:0/1016638835 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f58a80678c0 0x7f58a8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:16.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.328+0000 7f58b9ffb640 1 -- 172.21.15.116:0/1016638835 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7440+0+0 (secure 0 0 0) 0x7f58c00865c0 con 0x7f58c4072080 2024-04-25T00:09:16.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.328+0000 7f58c88a2640 1 --2- 172.21.15.116:0/1016638835 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f58a80678c0 0x7f58a8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:16.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.328+0000 7f58c88a2640 1 --2- 172.21.15.116:0/1016638835 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f58a80678c0 0x7f58a8069d80 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f58b40ad000 tx=0x7f58b4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:16.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.328+0000 7f82ae10c640 1 -- 172.21.15.116:0/3873626080 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f82a80729f0 con 0x7f82a806c760 2024-04-25T00:09:16.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.332+0000 7f58b9ffb640 1 -- 172.21.15.116:0/1016638835 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f58c0056480 con 0x7f58c4072080 2024-04-25T00:09:16.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.332+0000 7f828ffff640 1 -- 172.21.15.116:0/3873626080 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+6501 (secure 0 0 0) 0x7f829c05c030 con 0x7f82a806c760 2024-04-25T00:09:16.334 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:09:16.335 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5236,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/1159002662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1159002662},{"type":"v1","addr":"172.21.15.174:6837","nonce":1159002662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:07:33.575034+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5775":{"gid":5775,"name":"c","rank":0,"incarnation":52,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6833/3006671671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3006671671},{"type":"v1","addr":"172.21.15.116:6833","nonce":3006671671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5775]},"id":1},{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:08:03.679028+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5868},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5868":{"gid":5868,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6835/1670061091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1670061091},{"type":"v1","addr":"172.21.15.116:6835","nonce":1670061091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5868,"qdb_cluster":[5868]},"id":2}]} 2024-04-25T00:09:16.335 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 61 2024-04-25T00:09:16.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.336+0000 7f82ae10c640 1 -- 172.21.15.116:0/3873626080 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f82940678f0 msgr2=0x7f8294069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:16.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.336+0000 7f82ae10c640 1 --2- 172.21.15.116:0/3873626080 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f82940678f0 0x7f8294069db0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f8298005890 tx=0x7f8298009040 comp rx=0 tx=0).stop 2024-04-25T00:09:16.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.336+0000 7f82ae10c640 1 -- 172.21.15.116:0/3873626080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82a806c760 msgr2=0x7f82a8148000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:16.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.336+0000 7f82ae10c640 1 --2- 172.21.15.116:0/3873626080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82a806c760 0x7f82a8148000 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f829c00e810 tx=0x7f829c00ecd0 comp rx=0 tx=0).stop 2024-04-25T00:09:16.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.340+0000 7f82ae10c640 1 -- 172.21.15.116:0/3873626080 shutdown_connections 2024-04-25T00:09:16.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.340+0000 7f82ae10c640 1 --2- 172.21.15.116:0/3873626080 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f82940678f0 0x7f8294069db0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:16.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.340+0000 7f82ae10c640 1 --2- 172.21.15.116:0/3873626080 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82a81527b0 0x7f82a81efc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:16.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.340+0000 7f82ae10c640 1 --2- 172.21.15.116:0/3873626080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82a806c760 0x7f82a8148000 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:16.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.340+0000 7f82ae10c640 1 --2- 172.21.15.116:0/3873626080 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f82a806c320 0x7f82a80c52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:16.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.340+0000 7f82ae10c640 1 -- 172.21.15.116:0/3873626080 >> 172.21.15.116:0/3873626080 conn(0x7f82a80c4200 msgr2=0x7f82a80c3280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:09:16.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.340+0000 7f82ae10c640 1 -- 172.21.15.116:0/3873626080 shutdown_connections 2024-04-25T00:09:16.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.340+0000 7f82ae10c640 1 -- 172.21.15.116:0/3873626080 wait complete. 2024-04-25T00:09:16.358 INFO:tasks.mds_thrash.fs.[b]:kill mds.a (rank=0) 2024-04-25T00:09:16.358 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-25T00:09:16.358 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:09:16.370 INFO:tasks.ceph.mds.a:Stopped 2024-04-25T00:09:16.371 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:09:16.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.520+0000 7fb978ab2640 1 Processor -- start 2024-04-25T00:09:16.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.520+0000 7fb978ab2640 1 -- start start 2024-04-25T00:09:16.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.520+0000 7fb978ab2640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fb9740c3f70 0x7fb9741557c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:16.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.520+0000 7fb978ab2640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb9740c3a20 0x7fb974155d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:16.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.520+0000 7fb978ab2640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb974153220 con 0x7fb974152e40 2024-04-25T00:09:16.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.520+0000 7fb978ab2640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fb9740c4c90 con 0x7fb9740c3f70 2024-04-25T00:09:16.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.520+0000 7fb978ab2640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fb974156240 con 0x7fb9740c3a20 2024-04-25T00:09:16.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.520+0000 7fb97259c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fb974152e40 0x7fb9740710b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:36066/0 (socket says 172.21.15.116:36066) 2024-04-25T00:09:16.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.520+0000 7fb97259c640 1 -- 172.21.15.116:0/3069016952 learned_addr learned my addr 172.21.15.116:0/3069016952 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:09:16.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.520+0000 7fb971d9b640 1 --2- 172.21.15.116:0/3069016952 >> v2:172.21.15.174:3300/0 conn(0x7fb9740c3f70 0x7fb9741557c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:16.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.520+0000 7fb972d9d640 1 --2- 172.21.15.116:0/3069016952 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb9740c3a20 0x7fb974155d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:16.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.520+0000 7fb97159a640 1 -- 172.21.15.116:0/3069016952 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 588531003 0 0) 0x7fb974153220 con 0x7fb974152e40 2024-04-25T00:09:16.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.520+0000 7fb97159a640 1 -- 172.21.15.116:0/3069016952 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb95c003530 con 0x7fb974152e40 2024-04-25T00:09:16.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.520+0000 7fb97159a640 1 -- 172.21.15.116:0/3069016952 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1743887769 0 0) 0x7fb95c003530 con 0x7fb974152e40 2024-04-25T00:09:16.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.520+0000 7fb97159a640 1 -- 172.21.15.116:0/3069016952 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb9740c3a20 msgr2=0x7fb974155d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:09:16.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.520+0000 7fb97159a640 1 --2- 172.21.15.116:0/3069016952 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb9740c3a20 0x7fb974155d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:16.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.520+0000 7fb97159a640 1 -- 172.21.15.116:0/3069016952 >> v2:172.21.15.174:3300/0 conn(0x7fb9740c3f70 msgr2=0x7fb9741557c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:09:16.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.520+0000 7fb97159a640 1 --2- 172.21.15.116:0/3069016952 >> v2:172.21.15.174:3300/0 conn(0x7fb9740c3f70 0x7fb9741557c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:16.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.520+0000 7fb97159a640 1 -- 172.21.15.116:0/3069016952 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9740c7a90 con 0x7fb974152e40 2024-04-25T00:09:16.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.520+0000 7fb97159a640 1 -- 172.21.15.116:0/3069016952 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7fb964001680 con 0x7fb974152e40 2024-04-25T00:09:16.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.520+0000 7fb97159a640 1 -- 172.21.15.116:0/3069016952 >> v1:172.21.15.116:6789/0 conn(0x7fb974152e40 legacy=0x7fb9740710b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:16.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.520+0000 7fb97159a640 1 --2- 172.21.15.116:0/3069016952 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb9740c3a20 0x7fb95c0039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:16.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.520+0000 7fb97159a640 1 --2- 172.21.15.116:0/3069016952 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb95c003f40 0x7fb95c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:16.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.520+0000 7fb97159a640 1 --2- 172.21.15.116:0/3069016952 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb95c004d50 0x7fb95c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:16.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.520+0000 7fb97159a640 1 -- 172.21.15.116:0/3069016952 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb9740c4c90 con 0x7fb95c003f40 2024-04-25T00:09:16.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.520+0000 7fb97159a640 1 -- 172.21.15.116:0/3069016952 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb974156240 con 0x7fb95c004d50 2024-04-25T00:09:16.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.520+0000 7fb97159a640 1 -- 172.21.15.116:0/3069016952 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fb95c00ba20 con 0x7fb9740c3a20 2024-04-25T00:09:16.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.520+0000 7fb971d9b640 1 --2- 172.21.15.116:0/3069016952 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb95c003f40 0x7fb95c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:16.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.520+0000 7fb971d9b640 1 -- 172.21.15.116:0/3069016952 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb9740c3a20 msgr2=0x7fb95c0039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:09:16.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.520+0000 7fb971d9b640 1 --2- 172.21.15.116:0/3069016952 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb9740c3a20 0x7fb95c0039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:16.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.520+0000 7fb972d9d640 1 --2- 172.21.15.116:0/3069016952 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb95c004d50 0x7fb95c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:16.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.520+0000 7fb971d9b640 1 -- 172.21.15.116:0/3069016952 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb95c004d50 msgr2=0x7fb95c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:09:16.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.520+0000 7fb971d9b640 1 --2- 172.21.15.116:0/3069016952 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb95c004d50 0x7fb95c0051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:16.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.524+0000 7fb971d9b640 1 -- 172.21.15.116:0/3069016952 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb95c003530 con 0x7fb95c003f40 2024-04-25T00:09:16.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.524+0000 7fb971d9b640 1 --2- 172.21.15.116:0/3069016952 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb95c003f40 0x7fb95c004810 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fb968009b10 tx=0x7fb96801ca80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b620bce16620c5b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:16.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.524+0000 7fb97159a640 1 -- 172.21.15.116:0/3069016952 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9680b2070 con 0x7fb95c003f40 2024-04-25T00:09:16.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.524+0000 7fb97159a640 1 -- 172.21.15.116:0/3069016952 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9680a5670 con 0x7fb95c003f40 2024-04-25T00:09:16.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.524+0000 7fb97159a640 1 -- 172.21.15.116:0/3069016952 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9680096b0 con 0x7fb95c003f40 2024-04-25T00:09:16.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.524+0000 7fb978ab2640 1 -- 172.21.15.116:0/3069016952 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb95c003f40 msgr2=0x7fb95c004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:16.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.524+0000 7fb978ab2640 1 --2- 172.21.15.116:0/3069016952 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb95c003f40 0x7fb95c004810 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fb968009b10 tx=0x7fb96801ca80 comp rx=0 tx=0).stop 2024-04-25T00:09:16.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.524+0000 7fb972d9d640 1 -- 172.21.15.116:0/3069016952 reap_dead start 2024-04-25T00:09:16.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.524+0000 7fb978ab2640 1 -- 172.21.15.116:0/3069016952 shutdown_connections 2024-04-25T00:09:16.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.524+0000 7fb978ab2640 1 -- 172.21.15.116:0/3069016952 >> 172.21.15.116:0/3069016952 conn(0x7fb9740c9160 msgr2=0x7fb9740c9570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:09:16.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.524+0000 7fb978ab2640 1 -- 172.21.15.116:0/3069016952 shutdown_connections 2024-04-25T00:09:16.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.524+0000 7fb978ab2640 1 -- 172.21.15.116:0/3069016952 wait complete. 2024-04-25T00:09:16.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.528+0000 7fb978ab2640 1 Processor -- start 2024-04-25T00:09:16.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.528+0000 7fb978ab2640 1 -- start start 2024-04-25T00:09:16.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.528+0000 7fb978ab2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb95c003f40 0x7fb9741542d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:16.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.528+0000 7fb978ab2640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb97414a8c0 0x7fb97414ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:16.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.528+0000 7fb978ab2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9741ee240 0x7fb9741ee6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:16.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.528+0000 7fb978ab2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb974071940 con 0x7fb9741ee240 2024-04-25T00:09:16.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.528+0000 7fb978ab2640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb9740717c0 con 0x7fb95c003f40 2024-04-25T00:09:16.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.528+0000 7fb978ab2640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fb9740c5120 con 0x7fb97414a8c0 2024-04-25T00:09:16.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.528+0000 7fb972d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9741ee240 0x7fb9741ee6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:16.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.528+0000 7fb972d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9741ee240 0x7fb9741ee6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41186/0 (socket says 172.21.15.116:41186) 2024-04-25T00:09:16.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.528+0000 7fb972d9d640 1 -- 172.21.15.116:0/1606948428 learned_addr learned my addr 172.21.15.116:0/1606948428 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:09:16.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.528+0000 7fb97259c640 1 --2- 172.21.15.116:0/1606948428 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb95c003f40 0x7fb9741542d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:16.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.528+0000 7fb972d9d640 1 -- 172.21.15.116:0/1606948428 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb97414a8c0 msgr2=0x7fb97414ad40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T00:09:16.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.528+0000 7fb972d9d640 1 --2- 172.21.15.116:0/1606948428 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb97414a8c0 0x7fb97414ad40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:16.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.528+0000 7fb972d9d640 1 -- 172.21.15.116:0/1606948428 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb95c003f40 msgr2=0x7fb9741542d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:16.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.528+0000 7fb972d9d640 1 --2- 172.21.15.116:0/1606948428 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb95c003f40 0x7fb9741542d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:16.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.528+0000 7fb972d9d640 1 -- 172.21.15.116:0/1606948428 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb97415a0b0 con 0x7fb9741ee240 2024-04-25T00:09:16.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.528+0000 7fb972d9d640 1 --2- 172.21.15.116:0/1606948428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9741ee240 0x7fb9741ee6c0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fb9740c83a0 tx=0x7fb96c00f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:16.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.528+0000 7fb962ffd640 1 -- 172.21.15.116:0/1606948428 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb96c0039a0 con 0x7fb9741ee240 2024-04-25T00:09:16.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.528+0000 7fb962ffd640 1 -- 172.21.15.116:0/1606948428 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb96c003b40 con 0x7fb9741ee240 2024-04-25T00:09:16.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.528+0000 7fb962ffd640 1 -- 172.21.15.116:0/1606948428 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb96c004020 con 0x7fb9741ee240 2024-04-25T00:09:16.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.528+0000 7fb978ab2640 1 -- 172.21.15.116:0/1606948428 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb97415a390 con 0x7fb9741ee240 2024-04-25T00:09:16.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.528+0000 7fb978ab2640 1 -- 172.21.15.116:0/1606948428 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb97415a950 con 0x7fb9741ee240 2024-04-25T00:09:16.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.532+0000 7fb978ab2640 1 -- 172.21.15.116:0/1606948428 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb97415ac80 con 0x7fb9741ee240 2024-04-25T00:09:16.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.532+0000 7fb962ffd640 1 -- 172.21.15.116:0/1606948428 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb96c0041c0 con 0x7fb9741ee240 2024-04-25T00:09:16.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.532+0000 7fb962ffd640 1 --2- 172.21.15.116:0/1606948428 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb9540678f0 0x7fb954069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:16.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.532+0000 7fb962ffd640 1 -- 172.21.15.116:0/1606948428 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7440+0+0 (secure 0 0 0) 0x7fb96c086890 con 0x7fb9741ee240 2024-04-25T00:09:16.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.532+0000 7fb97259c640 1 --2- 172.21.15.116:0/1606948428 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb9540678f0 0x7fb954069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:16.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.532+0000 7fb97259c640 1 --2- 172.21.15.116:0/1606948428 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb9540678f0 0x7fb954069db0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fb974146440 tx=0x7fb9640015f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:16.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.536+0000 7fb962ffd640 1 -- 172.21.15.116:0/1606948428 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fb96c012070 con 0x7fb9741ee240 2024-04-25T00:09:16.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.588+0000 7f58cb307640 1 -- 172.21.15.116:0/1016638835 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f58a4005860 con 0x7f58c4072080 2024-04-25T00:09:16.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.588+0000 7f58b9ffb640 1 -- 172.21.15.116:0/1016638835 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+6501 (secure 0 0 0) 0x7f58c005a780 con 0x7f58c4072080 2024-04-25T00:09:16.590 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:09:16.591 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5236,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/1159002662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1159002662},{"type":"v1","addr":"172.21.15.174:6837","nonce":1159002662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:07:33.575034+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5775":{"gid":5775,"name":"c","rank":0,"incarnation":52,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6833/3006671671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3006671671},{"type":"v1","addr":"172.21.15.116:6833","nonce":3006671671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5775]},"id":1},{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:08:03.679028+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5868},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5868":{"gid":5868,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6835/1670061091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1670061091},{"type":"v1","addr":"172.21.15.116:6835","nonce":1670061091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5868,"qdb_cluster":[5868]},"id":2}]} 2024-04-25T00:09:16.591 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 61 2024-04-25T00:09:16.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.588+0000 7f58cb307640 1 -- 172.21.15.116:0/1016638835 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f58a80678c0 msgr2=0x7f58a8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:16.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.588+0000 7f58cb307640 1 --2- 172.21.15.116:0/1016638835 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f58a80678c0 0x7f58a8069d80 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f58b40ad000 tx=0x7f58b4008040 comp rx=0 tx=0).stop 2024-04-25T00:09:16.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.588+0000 7f58cb307640 1 -- 172.21.15.116:0/1016638835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58c4072080 msgr2=0x7f58c414bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:16.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.588+0000 7f58cb307640 1 --2- 172.21.15.116:0/1016638835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58c4072080 0x7f58c414bf00 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f58c000d800 tx=0x7f58c000dcc0 comp rx=0 tx=0).stop 2024-04-25T00:09:16.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.592+0000 7f58cb307640 1 -- 172.21.15.116:0/1016638835 shutdown_connections 2024-04-25T00:09:16.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.592+0000 7f58cb307640 1 --2- 172.21.15.116:0/1016638835 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f58a80678c0 0x7f58a8069d80 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:16.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.592+0000 7f58cb307640 1 --2- 172.21.15.116:0/1016638835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58c4152d80 0x7f58c414fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:16.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.592+0000 7f58cb307640 1 --2- 172.21.15.116:0/1016638835 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f58c40c36b0 0x7f58c414b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:16.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.592+0000 7f58cb307640 1 --2- 172.21.15.116:0/1016638835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58c4072080 0x7f58c414bf00 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:16.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.592+0000 7f58cb307640 1 -- 172.21.15.116:0/1016638835 >> 172.21.15.116:0/1016638835 conn(0x7f58c406c720 msgr2=0x7f58c406e750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:09:16.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.592+0000 7f58cb307640 1 -- 172.21.15.116:0/1016638835 shutdown_connections 2024-04-25T00:09:16.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.592+0000 7f58cb307640 1 -- 172.21.15.116:0/1016638835 wait complete. 2024-04-25T00:09:16.611 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:09:16.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.748+0000 7fb978ab2640 1 -- 172.21.15.116:0/1606948428 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb9740c7870 con 0x7fb9741ee240 2024-04-25T00:09:16.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.748+0000 7fb962ffd640 1 -- 172.21.15.116:0/1606948428 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+6501 (secure 0 0 0) 0x7fb96c056d20 con 0x7fb9741ee240 2024-04-25T00:09:16.750 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:09:16.750 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5236,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/1159002662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1159002662},{"type":"v1","addr":"172.21.15.174:6837","nonce":1159002662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:07:33.575034+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5775":{"gid":5775,"name":"c","rank":0,"incarnation":52,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6833/3006671671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3006671671},{"type":"v1","addr":"172.21.15.116:6833","nonce":3006671671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5775]},"id":1},{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:08:03.679028+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5868},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5868":{"gid":5868,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6835/1670061091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1670061091},{"type":"v1","addr":"172.21.15.116:6835","nonce":1670061091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5868,"qdb_cluster":[5868]},"id":2}]} 2024-04-25T00:09:16.750 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 61 2024-04-25T00:09:16.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.748+0000 7fb978ab2640 1 -- 172.21.15.116:0/1606948428 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb9540678f0 msgr2=0x7fb954069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:16.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.748+0000 7fb978ab2640 1 --2- 172.21.15.116:0/1606948428 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb9540678f0 0x7fb954069db0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fb974146440 tx=0x7fb9640015f0 comp rx=0 tx=0).stop 2024-04-25T00:09:16.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.748+0000 7fb978ab2640 1 -- 172.21.15.116:0/1606948428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9741ee240 msgr2=0x7fb9741ee6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:16.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.748+0000 7fb978ab2640 1 --2- 172.21.15.116:0/1606948428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9741ee240 0x7fb9741ee6c0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fb9740c83a0 tx=0x7fb96c00f4a0 comp rx=0 tx=0).stop 2024-04-25T00:09:16.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.748+0000 7fb978ab2640 1 -- 172.21.15.116:0/1606948428 shutdown_connections 2024-04-25T00:09:16.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.748+0000 7fb978ab2640 1 --2- 172.21.15.116:0/1606948428 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb9540678f0 0x7fb954069db0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:16.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.748+0000 7fb978ab2640 1 --2- 172.21.15.116:0/1606948428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9741ee240 0x7fb9741ee6c0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:16.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.748+0000 7fb978ab2640 1 --2- 172.21.15.116:0/1606948428 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb97414a8c0 0x7fb97414ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:16.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.748+0000 7fb978ab2640 1 --2- 172.21.15.116:0/1606948428 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb95c003f40 0x7fb9741542d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:16.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.748+0000 7fb978ab2640 1 -- 172.21.15.116:0/1606948428 >> 172.21.15.116:0/1606948428 conn(0x7fb9740c9160 msgr2=0x7fb974153220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:09:16.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.748+0000 7fb978ab2640 1 -- 172.21.15.116:0/1606948428 shutdown_connections 2024-04-25T00:09:16.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:16.748+0000 7fb978ab2640 1 -- 172.21.15.116:0/1606948428 wait complete. 2024-04-25T00:09:16.769 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:09:18.611 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:09:18.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.740+0000 7fb9e40ed640 1 Processor -- start 2024-04-25T00:09:18.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.740+0000 7fb9e40ed640 1 -- start start 2024-04-25T00:09:18.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.740+0000 7fb9e40ed640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb9dc150d20 0x7fb9dc0c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:18.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.740+0000 7fb9e40ed640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb9dc0a94a0 0x7fb9dc153ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:18.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.740+0000 7fb9e40ed640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9dc0c3910 con 0x7fb9dc0c2f50 2024-04-25T00:09:18.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.740+0000 7fb9e40ed640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fb9dc154530 con 0x7fb9dc0a94a0 2024-04-25T00:09:18.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.740+0000 7fb9e40ed640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fb9dc0c16e0 con 0x7fb9dc150d20 2024-04-25T00:09:18.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.740+0000 7fb9e268a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb9dc0a94a0 0x7fb9dc153ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:18.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.740+0000 7fb9e268a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb9dc0a94a0 0x7fb9dc153ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41200/0 (socket says 172.21.15.116:41200) 2024-04-25T00:09:18.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.740+0000 7fb9e268a640 1 -- 172.21.15.116:0/4117217312 learned_addr learned my addr 172.21.15.116:0/4117217312 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:09:18.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.740+0000 7fb9e1e89640 1 --2- 172.21.15.116:0/4117217312 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb9dc150d20 0x7fb9dc0c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:18.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.740+0000 7fb9e268a640 1 -- 172.21.15.116:0/4117217312 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb9dc150d20 msgr2=0x7fb9dc0c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:09:18.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.740+0000 7fb9e268a640 1 --2- 172.21.15.116:0/4117217312 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb9dc150d20 0x7fb9dc0c2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:18.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.740+0000 7fb9e268a640 1 -- 172.21.15.116:0/4117217312 >> v1:172.21.15.174:6789/0 conn(0x7fb9dc0c2f50 legacy=0x7fb9dc1538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:18.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.740+0000 7fb9e268a640 1 -- 172.21.15.116:0/4117217312 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9dc0c1400 con 0x7fb9dc0a94a0 2024-04-25T00:09:18.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.740+0000 7fb9e268a640 1 --2- 172.21.15.116:0/4117217312 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9dc0a94a0 0x7fb9dc153ff0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fb9d0009870 tx=0x7fb9d001c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b9144ccee3480d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:18.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.740+0000 7fb9e0e87640 1 -- 172.21.15.116:0/4117217312 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9d00a7070 con 0x7fb9dc0a94a0 2024-04-25T00:09:18.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.740+0000 7fb9e0e87640 1 -- 172.21.15.116:0/4117217312 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9d00b0070 con 0x7fb9dc0a94a0 2024-04-25T00:09:18.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.744+0000 7fb9e0e87640 1 -- 172.21.15.116:0/4117217312 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9d00ab890 con 0x7fb9dc0a94a0 2024-04-25T00:09:18.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.744+0000 7fb9e40ed640 1 -- 172.21.15.116:0/4117217312 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9dc0a94a0 msgr2=0x7fb9dc153ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:18.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.744+0000 7fb9e40ed640 1 --2- 172.21.15.116:0/4117217312 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9dc0a94a0 0x7fb9dc153ff0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fb9d0009870 tx=0x7fb9d001c560 comp rx=0 tx=0).stop 2024-04-25T00:09:18.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.744+0000 7fb9e40ed640 1 -- 172.21.15.116:0/4117217312 shutdown_connections 2024-04-25T00:09:18.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.744+0000 7fb9e40ed640 1 --2- 172.21.15.116:0/4117217312 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9dc0a94a0 0x7fb9dc153ff0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:18.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.744+0000 7fb9e40ed640 1 --2- 172.21.15.116:0/4117217312 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb9dc150d20 0x7fb9dc0c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:18.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.744+0000 7fb9e40ed640 1 -- 172.21.15.116:0/4117217312 >> 172.21.15.116:0/4117217312 conn(0x7fb9dc0c3b00 msgr2=0x7fb9dc0c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:09:18.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.744+0000 7fb9e40ed640 1 -- 172.21.15.116:0/4117217312 shutdown_connections 2024-04-25T00:09:18.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.744+0000 7fb9e40ed640 1 -- 172.21.15.116:0/4117217312 wait complete. 2024-04-25T00:09:18.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.744+0000 7fb9e40ed640 1 Processor -- start 2024-04-25T00:09:18.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.744+0000 7fb9e40ed640 1 -- start start 2024-04-25T00:09:18.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.744+0000 7fb9e40ed640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb9dc0a94a0 0x7fb9dc140bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:18.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.744+0000 7fb9e40ed640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9dc0c2f50 0x7fb9dc154980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:18.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.744+0000 7fb9e40ed640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9dc150d20 0x7fb9dc1efe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:18.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.744+0000 7fb9e40ed640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb9dc1608b0 con 0x7fb9dc0c2f50 2024-04-25T00:09:18.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.744+0000 7fb9e40ed640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb9dc160730 con 0x7fb9dc150d20 2024-04-25T00:09:18.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.744+0000 7fb9e40ed640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fb9dc160a30 con 0x7fb9dc0a94a0 2024-04-25T00:09:18.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.744+0000 7fb9e1688640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9dc0c2f50 0x7fb9dc154980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:18.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.744+0000 7fb9e1688640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9dc0c2f50 0x7fb9dc154980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41208/0 (socket says 172.21.15.116:41208) 2024-04-25T00:09:18.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.744+0000 7fb9e1688640 1 -- 172.21.15.116:0/625972994 learned_addr learned my addr 172.21.15.116:0/625972994 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:09:18.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.744+0000 7fb9e1e89640 1 --2- 172.21.15.116:0/625972994 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb9dc0a94a0 0x7fb9dc140bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:18.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.744+0000 7fb9e268a640 1 --2- 172.21.15.116:0/625972994 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9dc150d20 0x7fb9dc1efe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:18.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.744+0000 7fb9e1688640 1 -- 172.21.15.116:0/625972994 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb9dc0a94a0 msgr2=0x7fb9dc140bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:18.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.744+0000 7fb9e1688640 1 --2- 172.21.15.116:0/625972994 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb9dc0a94a0 0x7fb9dc140bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:18.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.744+0000 7fb9e1688640 1 -- 172.21.15.116:0/625972994 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9dc150d20 msgr2=0x7fb9dc1efe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:18.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.744+0000 7fb9e1688640 1 --2- 172.21.15.116:0/625972994 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9dc150d20 0x7fb9dc1efe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:18.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.744+0000 7fb9e1688640 1 -- 172.21.15.116:0/625972994 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9dc154220 con 0x7fb9dc0c2f50 2024-04-25T00:09:18.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.744+0000 7fb9e268a640 1 --2- 172.21.15.116:0/625972994 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9dc150d20 0x7fb9dc1efe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:09:18.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.744+0000 7fb9e1688640 1 --2- 172.21.15.116:0/625972994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9dc0c2f50 0x7fb9dc154980 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fb9d400c790 tx=0x7fb9d400cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:18.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.744+0000 7fb9ce7fc640 1 -- 172.21.15.116:0/625972994 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9d4016020 con 0x7fb9dc0c2f50 2024-04-25T00:09:18.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.744+0000 7fb9ce7fc640 1 -- 172.21.15.116:0/625972994 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9d4006d60 con 0x7fb9dc0c2f50 2024-04-25T00:09:18.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.744+0000 7fb9ce7fc640 1 -- 172.21.15.116:0/625972994 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9d4003bc0 con 0x7fb9dc0c2f50 2024-04-25T00:09:18.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.744+0000 7fb9e40ed640 1 -- 172.21.15.116:0/625972994 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9dc146cf0 con 0x7fb9dc0c2f50 2024-04-25T00:09:18.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.744+0000 7fb9e40ed640 1 -- 172.21.15.116:0/625972994 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9dc146730 con 0x7fb9dc0c2f50 2024-04-25T00:09:18.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.748+0000 7fb9e40ed640 1 -- 172.21.15.116:0/625972994 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9dc068350 con 0x7fb9dc0c2f50 2024-04-25T00:09:18.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.748+0000 7fb9ce7fc640 1 -- 172.21.15.116:0/625972994 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb9d400a850 con 0x7fb9dc0c2f50 2024-04-25T00:09:18.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.748+0000 7fb9ce7fc640 1 --2- 172.21.15.116:0/625972994 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb9c00678c0 0x7fb9c0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:18.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.748+0000 7fb9ce7fc640 1 -- 172.21.15.116:0/625972994 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7440+0+0 (secure 0 0 0) 0x7fb9d4086000 con 0x7fb9dc0c2f50 2024-04-25T00:09:18.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.748+0000 7fb9e1e89640 1 --2- 172.21.15.116:0/625972994 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb9c00678c0 0x7fb9c0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:18.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.748+0000 7fb9e1e89640 1 --2- 172.21.15.116:0/625972994 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb9c00678c0 0x7fb9c0069d80 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fb9d800b930 tx=0x7fb9d8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:18.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.752+0000 7fb9ce7fc640 1 -- 172.21.15.116:0/625972994 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fb9d4020020 con 0x7fb9dc0c2f50 2024-04-25T00:09:18.770 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:09:18.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.876+0000 7fe8984e5640 1 Processor -- start 2024-04-25T00:09:18.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.876+0000 7fe8984e5640 1 -- start start 2024-04-25T00:09:18.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.876+0000 7fe8984e5640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe8900aeb20 0x7fe8900c9310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:18.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.876+0000 7fe8984e5640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe890150f80 0x7fe8900c5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:18.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.876+0000 7fe8984e5640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8901535d0 con 0x7fe8900c9850 2024-04-25T00:09:18.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.876+0000 7fe8984e5640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fe890153850 con 0x7fe8900aeb20 2024-04-25T00:09:18.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.876+0000 7fe8984e5640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe890153b30 con 0x7fe890150f80 2024-04-25T00:09:18.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.876+0000 7fe896281640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe8900aeb20 0x7fe8900c9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:18.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.876+0000 7fe895a80640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fe8900c9850 0x7fe890150870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:36074/0 (socket says 172.21.15.116:36074) 2024-04-25T00:09:18.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.876+0000 7fe895a80640 1 -- 172.21.15.116:0/682349692 learned_addr learned my addr 172.21.15.116:0/682349692 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:09:18.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.876+0000 7fe89527f640 1 -- 172.21.15.116:0/682349692 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3348631122 0 0) 0x7fe880001120 con 0x7fe8900c9850 2024-04-25T00:09:18.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.876+0000 7fe89527f640 1 -- 172.21.15.116:0/682349692 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe87c003530 con 0x7fe8900c9850 2024-04-25T00:09:18.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.876+0000 7fe896a82640 1 --2- 172.21.15.116:0/682349692 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe890150f80 0x7fe8900c5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:18.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.876+0000 7fe896281640 1 -- 172.21.15.116:0/682349692 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe890150f80 msgr2=0x7fe8900c5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:09:18.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.876+0000 7fe896281640 1 --2- 172.21.15.116:0/682349692 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe890150f80 0x7fe8900c5270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:18.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.876+0000 7fe896281640 1 -- 172.21.15.116:0/682349692 >> v1:172.21.15.116:6789/0 conn(0x7fe8900c9850 legacy=0x7fe890150870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:18.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.876+0000 7fe89527f640 1 -- 172.21.15.116:0/682349692 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3107052624 0 0) 0x7fe87c003530 con 0x7fe8900c9850 2024-04-25T00:09:18.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.876+0000 7fe896281640 1 -- 172.21.15.116:0/682349692 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe890153e10 con 0x7fe8900aeb20 2024-04-25T00:09:18.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.876+0000 7fe896a82640 1 --2- 172.21.15.116:0/682349692 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe890150f80 0x7fe8900c5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:09:18.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.880+0000 7fe896281640 1 --2- 172.21.15.116:0/682349692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8900aeb20 0x7fe8900c9310 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fe88c0097c0 tx=0x7fe88c01c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e8c33d753100cd8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:18.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.880+0000 7fe89527f640 1 -- 172.21.15.116:0/682349692 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe88c0b0070 con 0x7fe8900aeb20 2024-04-25T00:09:18.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.880+0000 7fe89527f640 1 -- 172.21.15.116:0/682349692 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe88c0ab400 con 0x7fe8900aeb20 2024-04-25T00:09:18.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.880+0000 7fe89527f640 1 -- 172.21.15.116:0/682349692 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe880001850 con 0x7fe8900aeb20 2024-04-25T00:09:18.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.880+0000 7fe8984e5640 1 -- 172.21.15.116:0/682349692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8900aeb20 msgr2=0x7fe8900c9310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:18.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.880+0000 7fe8984e5640 1 --2- 172.21.15.116:0/682349692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8900aeb20 0x7fe8900c9310 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fe88c0097c0 tx=0x7fe88c01c330 comp rx=0 tx=0).stop 2024-04-25T00:09:18.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.880+0000 7fe8984e5640 1 -- 172.21.15.116:0/682349692 shutdown_connections 2024-04-25T00:09:18.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.880+0000 7fe8984e5640 1 --2- 172.21.15.116:0/682349692 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe890150f80 0x7fe8900c5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:18.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.880+0000 7fe8984e5640 1 --2- 172.21.15.116:0/682349692 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8900aeb20 0x7fe8900c9310 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:18.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.880+0000 7fe8984e5640 1 -- 172.21.15.116:0/682349692 >> 172.21.15.116:0/682349692 conn(0x7fe890074650 msgr2=0x7fe890074a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:09:18.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.880+0000 7fe8984e5640 1 -- 172.21.15.116:0/682349692 shutdown_connections 2024-04-25T00:09:18.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.880+0000 7fe8984e5640 1 -- 172.21.15.116:0/682349692 wait complete. 2024-04-25T00:09:18.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.880+0000 7fe8984e5640 1 Processor -- start 2024-04-25T00:09:18.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.880+0000 7fe8984e5640 1 -- start start 2024-04-25T00:09:18.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.884+0000 7fe8984e5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8900aeb20 0x7fe89006df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:18.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.884+0000 7fe8984e5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe8900c9850 0x7fe89014e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:18.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.884+0000 7fe8984e5640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe890150f80 0x7fe89014feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:18.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.884+0000 7fe8984e5640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe8901600e0 con 0x7fe8900aeb20 2024-04-25T00:09:18.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.884+0000 7fe8984e5640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe89015ff60 con 0x7fe8900c9850 2024-04-25T00:09:18.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.884+0000 7fe8984e5640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe890160260 con 0x7fe890150f80 2024-04-25T00:09:18.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.884+0000 7fe896281640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8900aeb20 0x7fe89006df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:18.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.884+0000 7fe896281640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8900aeb20 0x7fe89006df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41234/0 (socket says 172.21.15.116:41234) 2024-04-25T00:09:18.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.884+0000 7fe896281640 1 -- 172.21.15.116:0/777670505 learned_addr learned my addr 172.21.15.116:0/777670505 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:09:18.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.884+0000 7fe896281640 1 -- 172.21.15.116:0/777670505 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe890150f80 msgr2=0x7fe89014feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:18.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.884+0000 7fe896281640 1 --2- 172.21.15.116:0/777670505 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe890150f80 0x7fe89014feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:18.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.884+0000 7fe896281640 1 -- 172.21.15.116:0/777670505 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe8900c9850 msgr2=0x7fe89014e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:18.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.884+0000 7fe896281640 1 --2- 172.21.15.116:0/777670505 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe8900c9850 0x7fe89014e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:18.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.884+0000 7fe896281640 1 -- 172.21.15.116:0/777670505 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8901433d0 con 0x7fe8900aeb20 2024-04-25T00:09:18.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.884+0000 7fe896281640 1 --2- 172.21.15.116:0/777670505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8900aeb20 0x7fe89006df50 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fe88c0ae040 tx=0x7fe88c01ab70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:18.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.884+0000 7fe886ffd640 1 -- 172.21.15.116:0/777670505 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe88c0b0050 con 0x7fe8900aeb20 2024-04-25T00:09:18.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.884+0000 7fe886ffd640 1 -- 172.21.15.116:0/777670505 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe88c0bc070 con 0x7fe8900aeb20 2024-04-25T00:09:18.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.884+0000 7fe8984e5640 1 -- 172.21.15.116:0/777670505 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8901f21a0 con 0x7fe8900aeb20 2024-04-25T00:09:18.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.884+0000 7fe886ffd640 1 -- 172.21.15.116:0/777670505 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe88c0b7690 con 0x7fe8900aeb20 2024-04-25T00:09:18.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.884+0000 7fe8984e5640 1 -- 172.21.15.116:0/777670505 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe890143060 con 0x7fe8900aeb20 2024-04-25T00:09:18.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.884+0000 7fe8984e5640 1 -- 172.21.15.116:0/777670505 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe890068a60 con 0x7fe8900aeb20 2024-04-25T00:09:18.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.884+0000 7fe886ffd640 1 -- 172.21.15.116:0/777670505 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe88c0b7830 con 0x7fe8900aeb20 2024-04-25T00:09:18.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.884+0000 7fe886ffd640 1 --2- 172.21.15.116:0/777670505 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe8740678c0 0x7fe874069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:18.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.884+0000 7fe886ffd640 1 -- 172.21.15.116:0/777670505 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7440+0+0 (secure 0 0 0) 0x7fe88c120980 con 0x7fe8900aeb20 2024-04-25T00:09:18.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.884+0000 7fe895a80640 1 --2- 172.21.15.116:0/777670505 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe8740678c0 0x7fe874069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:18.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.888+0000 7fe895a80640 1 --2- 172.21.15.116:0/777670505 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe8740678c0 0x7fe874069d80 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fe89015de30 tx=0x7fe880000f00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:18.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.892+0000 7fe886ffd640 1 -- 172.21.15.116:0/777670505 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fe88c015fb0 con 0x7fe8900aeb20 2024-04-25T00:09:18.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.960+0000 7fb9e40ed640 1 -- 172.21.15.116:0/625972994 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb9dc0c34f0 con 0x7fb9dc0c2f50 2024-04-25T00:09:18.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.960+0000 7fb9ce7fc640 1 -- 172.21.15.116:0/625972994 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+6501 (secure 0 0 0) 0x7fb9dc154220 con 0x7fb9dc0c2f50 2024-04-25T00:09:18.962 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:09:18.962 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5236,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/1159002662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1159002662},{"type":"v1","addr":"172.21.15.174:6837","nonce":1159002662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:07:33.575034+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5775":{"gid":5775,"name":"c","rank":0,"incarnation":52,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6833/3006671671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3006671671},{"type":"v1","addr":"172.21.15.116:6833","nonce":3006671671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5775]},"id":1},{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:08:03.679028+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5868},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5868":{"gid":5868,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6835/1670061091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1670061091},{"type":"v1","addr":"172.21.15.116:6835","nonce":1670061091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5868,"qdb_cluster":[5868]},"id":2}]} 2024-04-25T00:09:18.962 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 61 2024-04-25T00:09:18.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.964+0000 7fb9e40ed640 1 -- 172.21.15.116:0/625972994 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb9c00678c0 msgr2=0x7fb9c0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:18.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.964+0000 7fb9e40ed640 1 --2- 172.21.15.116:0/625972994 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb9c00678c0 0x7fb9c0069d80 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fb9d800b930 tx=0x7fb9d8008040 comp rx=0 tx=0).stop 2024-04-25T00:09:18.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.964+0000 7fb9e40ed640 1 -- 172.21.15.116:0/625972994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9dc0c2f50 msgr2=0x7fb9dc154980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:18.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.964+0000 7fb9e40ed640 1 --2- 172.21.15.116:0/625972994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9dc0c2f50 0x7fb9dc154980 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fb9d400c790 tx=0x7fb9d400cc50 comp rx=0 tx=0).stop 2024-04-25T00:09:18.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.964+0000 7fb9e40ed640 1 -- 172.21.15.116:0/625972994 shutdown_connections 2024-04-25T00:09:18.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.964+0000 7fb9e40ed640 1 --2- 172.21.15.116:0/625972994 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb9c00678c0 0x7fb9c0069d80 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:18.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.964+0000 7fb9e40ed640 1 --2- 172.21.15.116:0/625972994 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9dc150d20 0x7fb9dc1efe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:18.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.964+0000 7fb9e40ed640 1 --2- 172.21.15.116:0/625972994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb9dc0c2f50 0x7fb9dc154980 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:18.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.964+0000 7fb9e40ed640 1 --2- 172.21.15.116:0/625972994 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb9dc0a94a0 0x7fb9dc140bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:18.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.964+0000 7fb9e40ed640 1 -- 172.21.15.116:0/625972994 >> 172.21.15.116:0/625972994 conn(0x7fb9dc0c3b00 msgr2=0x7fb9dc14fbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:09:18.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.964+0000 7fb9e40ed640 1 -- 172.21.15.116:0/625972994 shutdown_connections 2024-04-25T00:09:18.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:18.964+0000 7fb9e40ed640 1 -- 172.21.15.116:0/625972994 wait complete. 2024-04-25T00:09:18.980 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:09:19.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:19.088+0000 7fe8984e5640 1 -- 172.21.15.116:0/777670505 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe8900c34d0 con 0x7fe8900aeb20 2024-04-25T00:09:19.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:19.088+0000 7fe886ffd640 1 -- 172.21.15.116:0/777670505 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+6501 (secure 0 0 0) 0x7fe88c0f0e10 con 0x7fe8900aeb20 2024-04-25T00:09:19.090 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:09:19.090 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5236,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/1159002662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1159002662},{"type":"v1","addr":"172.21.15.174:6837","nonce":1159002662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:07:33.575034+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5775":{"gid":5775,"name":"c","rank":0,"incarnation":52,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6833/3006671671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3006671671},{"type":"v1","addr":"172.21.15.116:6833","nonce":3006671671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5775]},"id":1},{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:08:03.679028+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5868},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5868":{"gid":5868,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6835/1670061091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1670061091},{"type":"v1","addr":"172.21.15.116:6835","nonce":1670061091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5868,"qdb_cluster":[5868]},"id":2}]} 2024-04-25T00:09:19.090 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 61 2024-04-25T00:09:19.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:19.088+0000 7fe8984e5640 1 -- 172.21.15.116:0/777670505 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe8740678c0 msgr2=0x7fe874069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:19.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:19.088+0000 7fe8984e5640 1 --2- 172.21.15.116:0/777670505 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe8740678c0 0x7fe874069d80 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fe89015de30 tx=0x7fe880000f00 comp rx=0 tx=0).stop 2024-04-25T00:09:19.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:19.088+0000 7fe8984e5640 1 -- 172.21.15.116:0/777670505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8900aeb20 msgr2=0x7fe89006df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:19.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:19.088+0000 7fe8984e5640 1 --2- 172.21.15.116:0/777670505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8900aeb20 0x7fe89006df50 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fe88c0ae040 tx=0x7fe88c01ab70 comp rx=0 tx=0).stop 2024-04-25T00:09:19.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:19.088+0000 7fe8984e5640 1 -- 172.21.15.116:0/777670505 shutdown_connections 2024-04-25T00:09:19.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:19.088+0000 7fe8984e5640 1 --2- 172.21.15.116:0/777670505 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe8740678c0 0x7fe874069d80 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:19.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:19.088+0000 7fe8984e5640 1 --2- 172.21.15.116:0/777670505 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe890150f80 0x7fe89014feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:19.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:19.088+0000 7fe8984e5640 1 --2- 172.21.15.116:0/777670505 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe8900c9850 0x7fe89014e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:19.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:19.088+0000 7fe8984e5640 1 --2- 172.21.15.116:0/777670505 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8900aeb20 0x7fe89006df50 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:19.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:19.088+0000 7fe8984e5640 1 -- 172.21.15.116:0/777670505 >> 172.21.15.116:0/777670505 conn(0x7fe890074650 msgr2=0x7fe8900671c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:09:19.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:19.088+0000 7fe8984e5640 1 -- 172.21.15.116:0/777670505 shutdown_connections 2024-04-25T00:09:19.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:19.088+0000 7fe8984e5640 1 -- 172.21.15.116:0/777670505 wait complete. 2024-04-25T00:09:19.106 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:09:20.981 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:09:21.107 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:09:21.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.112+0000 7f22ef3ad640 1 Processor -- start 2024-04-25T00:09:21.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.112+0000 7f22ef3ad640 1 -- start start 2024-04-25T00:09:21.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.116+0000 7f22ef3ad640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f22e814c700 0x7f22e80c5090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:21.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.116+0000 7f22ef3ad640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f22e80c55d0 0x7f22e815ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:21.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.116+0000 7f22ef3ad640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22e80a9e60 con 0x7f22e8155b20 2024-04-25T00:09:21.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.116+0000 7f22ef3ad640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f22e81527b0 con 0x7f22e814c700 2024-04-25T00:09:21.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.116+0000 7f22ef3ad640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f22e80c3ef0 con 0x7f22e80c55d0 2024-04-25T00:09:21.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.116+0000 7f22ed149640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f22e814c700 0x7f22e80c5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:21.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.116+0000 7f22ed149640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f22e814c700 0x7f22e80c5090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41244/0 (socket says 172.21.15.116:41244) 2024-04-25T00:09:21.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.116+0000 7f22ed149640 1 -- 172.21.15.116:0/802269802 learned_addr learned my addr 172.21.15.116:0/802269802 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:09:21.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.116+0000 7f22ed94a640 1 --2- 172.21.15.116:0/802269802 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f22e80c55d0 0x7f22e815ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:21.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.116+0000 7f22ed149640 1 -- 172.21.15.116:0/802269802 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f22e80c55d0 msgr2=0x7f22e815ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:09:21.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.116+0000 7f22ed149640 1 --2- 172.21.15.116:0/802269802 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f22e80c55d0 0x7f22e815ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:21.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.116+0000 7f22dffff640 1 -- 172.21.15.116:0/802269802 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2444360904 0 0) 0x7f22e80a9e60 con 0x7f22e8155b20 2024-04-25T00:09:21.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.116+0000 7f22ed149640 1 -- 172.21.15.116:0/802269802 >> v1:172.21.15.116:6789/0 conn(0x7f22e8155b20 legacy=0x7f22e8152c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:21.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.116+0000 7f22ed149640 1 -- 172.21.15.116:0/802269802 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22e80c41d0 con 0x7f22e814c700 2024-04-25T00:09:21.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.116+0000 7f22ed149640 1 --2- 172.21.15.116:0/802269802 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22e814c700 0x7f22e80c5090 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f22e400bb10 tx=0x7f22e401eb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7c6f60ecf704b92 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:21.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.120+0000 7f22dffff640 1 -- 172.21.15.116:0/802269802 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22e40b4070 con 0x7f22e814c700 2024-04-25T00:09:21.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.120+0000 7f22dffff640 1 -- 172.21.15.116:0/802269802 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22e40a78e0 con 0x7f22e814c700 2024-04-25T00:09:21.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.120+0000 7f22dffff640 1 -- 172.21.15.116:0/802269802 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22e40a7cb0 con 0x7f22e814c700 2024-04-25T00:09:21.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.120+0000 7f22ef3ad640 1 -- 172.21.15.116:0/802269802 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22e814c700 msgr2=0x7f22e80c5090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:21.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.120+0000 7f22ef3ad640 1 --2- 172.21.15.116:0/802269802 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22e814c700 0x7f22e80c5090 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f22e400bb10 tx=0x7f22e401eb10 comp rx=0 tx=0).stop 2024-04-25T00:09:21.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.120+0000 7f22ef3ad640 1 -- 172.21.15.116:0/802269802 shutdown_connections 2024-04-25T00:09:21.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.120+0000 7f22ef3ad640 1 --2- 172.21.15.116:0/802269802 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f22e80c55d0 0x7f22e815ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:21.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.120+0000 7f22ef3ad640 1 --2- 172.21.15.116:0/802269802 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22e814c700 0x7f22e80c5090 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:21.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.120+0000 7f22ef3ad640 1 -- 172.21.15.116:0/802269802 >> 172.21.15.116:0/802269802 conn(0x7f22e806c720 msgr2=0x7f22e806cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:09:21.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.120+0000 7f22ef3ad640 1 -- 172.21.15.116:0/802269802 shutdown_connections 2024-04-25T00:09:21.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.120+0000 7f22ef3ad640 1 -- 172.21.15.116:0/802269802 wait complete. 2024-04-25T00:09:21.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.120+0000 7f22ef3ad640 1 Processor -- start 2024-04-25T00:09:21.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.120+0000 7f22ef3ad640 1 -- start start 2024-04-25T00:09:21.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.120+0000 7f22ef3ad640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22e80c55d0 0x7f22e80c31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:21.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.120+0000 7f22ef3ad640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f22e814c700 0x7f22e8140f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:21.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.120+0000 7f22ef3ad640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f22e8155b20 0x7f22e81414b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:21.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.120+0000 7f22ef3ad640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f22e806bdf0 con 0x7f22e80c55d0 2024-04-25T00:09:21.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.120+0000 7f22ef3ad640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f22e806bc70 con 0x7f22e8155b20 2024-04-25T00:09:21.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.120+0000 7f22ef3ad640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f22e806bf70 con 0x7f22e814c700 2024-04-25T00:09:21.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.120+0000 7f22ed149640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22e80c55d0 0x7f22e80c31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:21.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.120+0000 7f22ed149640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22e80c55d0 0x7f22e80c31a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41258/0 (socket says 172.21.15.116:41258) 2024-04-25T00:09:21.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.120+0000 7f22ed149640 1 -- 172.21.15.116:0/917206625 learned_addr learned my addr 172.21.15.116:0/917206625 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:09:21.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.120+0000 7f22ec948640 1 --2- 172.21.15.116:0/917206625 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f22e814c700 0x7f22e8140f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:21.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.120+0000 7f22ed149640 1 -- 172.21.15.116:0/917206625 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f22e814c700 msgr2=0x7f22e8140f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:21.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.120+0000 7f22ed149640 1 --2- 172.21.15.116:0/917206625 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f22e814c700 0x7f22e8140f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:21.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.120+0000 7f22ed149640 1 -- 172.21.15.116:0/917206625 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f22e8155b20 msgr2=0x7f22e81414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:21.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.120+0000 7f22ed94a640 1 --2- 172.21.15.116:0/917206625 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f22e8155b20 0x7f22e81414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:21.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.120+0000 7f22ed149640 1 --2- 172.21.15.116:0/917206625 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f22e8155b20 0x7f22e81414b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:21.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.120+0000 7f22ed149640 1 -- 172.21.15.116:0/917206625 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22e80c7d50 con 0x7f22e80c55d0 2024-04-25T00:09:21.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.120+0000 7f22ed149640 1 --2- 172.21.15.116:0/917206625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22e80c55d0 0x7f22e80c31a0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f22e400b2c0 tx=0x7f22e401b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:21.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.120+0000 7f22ddffb640 1 -- 172.21.15.116:0/917206625 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22e40b4040 con 0x7f22e80c55d0 2024-04-25T00:09:21.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.120+0000 7f22ddffb640 1 -- 172.21.15.116:0/917206625 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22e40af700 con 0x7f22e80c55d0 2024-04-25T00:09:21.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.120+0000 7f22ef3ad640 1 -- 172.21.15.116:0/917206625 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22e80c7a70 con 0x7f22e80c55d0 2024-04-25T00:09:21.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.120+0000 7f22ddffb640 1 -- 172.21.15.116:0/917206625 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22e40afaf0 con 0x7f22e80c55d0 2024-04-25T00:09:21.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.120+0000 7f22ef3ad640 1 -- 172.21.15.116:0/917206625 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22e81eb4b0 con 0x7f22e80c55d0 2024-04-25T00:09:21.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.124+0000 7f22ef3ad640 1 -- 172.21.15.116:0/917206625 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22e8068a60 con 0x7f22e80c55d0 2024-04-25T00:09:21.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.124+0000 7f22ddffb640 1 -- 172.21.15.116:0/917206625 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f22e40afc90 con 0x7f22e80c55d0 2024-04-25T00:09:21.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.124+0000 7f22ddffb640 1 --2- 172.21.15.116:0/917206625 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f22cc0678c0 0x7f22cc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:21.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.124+0000 7f22ddffb640 1 -- 172.21.15.116:0/917206625 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7440+0+0 (secure 0 0 0) 0x7f22e4122d10 con 0x7f22e80c55d0 2024-04-25T00:09:21.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.124+0000 7f22ec948640 1 --2- 172.21.15.116:0/917206625 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f22cc0678c0 0x7f22cc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:21.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.124+0000 7f22ec948640 1 --2- 172.21.15.116:0/917206625 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f22cc0678c0 0x7f22cc069d80 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f22e81eb100 tx=0x7f22d8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:21.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.128+0000 7f22ddffb640 1 -- 172.21.15.116:0/917206625 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f22e4018820 con 0x7f22e80c55d0 2024-04-25T00:09:21.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.220+0000 7f65c949c640 1 Processor -- start 2024-04-25T00:09:21.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.220+0000 7f65c949c640 1 -- start start 2024-04-25T00:09:21.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.224+0000 7f65c949c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f65c40c5200 0x7f65c4154040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:21.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.224+0000 7f65c949c640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65c40c2b40 con 0x7f65c4156bb0 2024-04-25T00:09:21.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.224+0000 7f65c949c640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65c4154580 con 0x7f65c40c4760 2024-04-25T00:09:21.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.224+0000 7f65c949c640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f65c4072b80 con 0x7f65c40c5200 2024-04-25T00:09:21.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.224+0000 7f65c2ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f65c40c5200 0x7f65c4154040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:21.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.224+0000 7f65c2ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f65c40c5200 0x7f65c4154040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41270/0 (socket says 172.21.15.116:41270) 2024-04-25T00:09:21.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.224+0000 7f65c2ffd640 1 -- 172.21.15.116:0/1638217494 learned_addr learned my addr 172.21.15.116:0/1638217494 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:09:21.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.224+0000 7f65c37fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f65c4156bb0 0x7f65c4156fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:36090/0 (socket says 172.21.15.116:36090) 2024-04-25T00:09:21.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.224+0000 7f65c2ffd640 1 -- 172.21.15.116:0/1638217494 >> v1:172.21.15.174:6789/0 conn(0x7f65c40c4760 legacy=0x7f65c414d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:09:21.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.224+0000 7f65c2ffd640 1 -- 172.21.15.116:0/1638217494 >> v1:172.21.15.116:6789/0 conn(0x7f65c4156bb0 legacy=0x7f65c4156fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:21.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.224+0000 7f65c2ffd640 1 -- 172.21.15.116:0/1638217494 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65c41576f0 con 0x7f65c40c5200 2024-04-25T00:09:21.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.224+0000 7f65c2ffd640 1 --2- 172.21.15.116:0/1638217494 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65c40c5200 0x7f65c4154040 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f65b4009870 tx=0x7f65b401c2b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4466f388f462cd5d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:21.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.224+0000 7f65c27fc640 1 -- 172.21.15.116:0/1638217494 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65b40b0070 con 0x7f65c40c5200 2024-04-25T00:09:21.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.224+0000 7f65c27fc640 1 -- 172.21.15.116:0/1638217494 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65b40ab600 con 0x7f65c40c5200 2024-04-25T00:09:21.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.224+0000 7f65c27fc640 1 -- 172.21.15.116:0/1638217494 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65b40ab9d0 con 0x7f65c40c5200 2024-04-25T00:09:21.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.224+0000 7f65c949c640 1 -- 172.21.15.116:0/1638217494 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65c40c5200 msgr2=0x7f65c4154040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:21.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.224+0000 7f65c949c640 1 --2- 172.21.15.116:0/1638217494 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65c40c5200 0x7f65c4154040 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f65b4009870 tx=0x7f65b401c2b0 comp rx=0 tx=0).stop 2024-04-25T00:09:21.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.228+0000 7f65c949c640 1 -- 172.21.15.116:0/1638217494 shutdown_connections 2024-04-25T00:09:21.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.228+0000 7f65c949c640 1 --2- 172.21.15.116:0/1638217494 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65c40c5200 0x7f65c4154040 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:21.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.228+0000 7f65c949c640 1 -- 172.21.15.116:0/1638217494 >> 172.21.15.116:0/1638217494 conn(0x7f65c406c720 msgr2=0x7f65c406cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:09:21.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.228+0000 7f65c949c640 1 -- 172.21.15.116:0/1638217494 shutdown_connections 2024-04-25T00:09:21.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.228+0000 7f65c949c640 1 -- 172.21.15.116:0/1638217494 wait complete. 2024-04-25T00:09:21.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.228+0000 7f65c949c640 1 Processor -- start 2024-04-25T00:09:21.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.228+0000 7f65c949c640 1 -- start start 2024-04-25T00:09:21.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.228+0000 7f65c949c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65c40c4760 0x7f65c406e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:21.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.228+0000 7f65c949c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f65c40c5200 0x7f65c41ef900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:21.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.228+0000 7f65c949c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65c4156bb0 0x7f65c41efe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:21.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.228+0000 7f65c949c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f65c414d940 con 0x7f65c4156bb0 2024-04-25T00:09:21.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.228+0000 7f65c949c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f65c40c2720 con 0x7f65c40c4760 2024-04-25T00:09:21.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.228+0000 7f65c949c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f65c414dac0 con 0x7f65c40c5200 2024-04-25T00:09:21.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.228+0000 7f65c37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65c4156bb0 0x7f65c41efe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:21.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.228+0000 7f65c37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65c4156bb0 0x7f65c41efe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41280/0 (socket says 172.21.15.116:41280) 2024-04-25T00:09:21.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.228+0000 7f65c37fe640 1 -- 172.21.15.116:0/1195162105 learned_addr learned my addr 172.21.15.116:0/1195162105 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:09:21.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.228+0000 7f65c2ffd640 1 --2- 172.21.15.116:0/1195162105 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65c40c4760 0x7f65c406e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:21.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.228+0000 7f65bbfff640 1 --2- 172.21.15.116:0/1195162105 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f65c40c5200 0x7f65c41ef900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:21.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.228+0000 7f65c37fe640 1 -- 172.21.15.116:0/1195162105 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f65c40c5200 msgr2=0x7f65c41ef900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:21.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.228+0000 7f65c37fe640 1 --2- 172.21.15.116:0/1195162105 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f65c40c5200 0x7f65c41ef900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:21.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.228+0000 7f65c37fe640 1 -- 172.21.15.116:0/1195162105 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65c40c4760 msgr2=0x7f65c406e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:21.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.228+0000 7f65c37fe640 1 --2- 172.21.15.116:0/1195162105 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65c40c4760 0x7f65c406e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:21.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.228+0000 7f65c37fe640 1 -- 172.21.15.116:0/1195162105 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65c40c5d30 con 0x7f65c4156bb0 2024-04-25T00:09:21.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.228+0000 7f65bbfff640 1 --2- 172.21.15.116:0/1195162105 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f65c40c5200 0x7f65c41ef900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:09:21.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.228+0000 7f65c37fe640 1 --2- 172.21.15.116:0/1195162105 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65c4156bb0 0x7f65c41efe40 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f65bc00c7a0 tx=0x7f65bc00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:21.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.228+0000 7f65bb7fe640 1 -- 172.21.15.116:0/1195162105 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65bc017070 con 0x7f65c4156bb0 2024-04-25T00:09:21.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.228+0000 7f65bb7fe640 1 -- 172.21.15.116:0/1195162105 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65bc00a640 con 0x7f65c4156bb0 2024-04-25T00:09:21.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.228+0000 7f65c949c640 1 -- 172.21.15.116:0/1195162105 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65c40c6010 con 0x7f65c4156bb0 2024-04-25T00:09:21.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.228+0000 7f65bb7fe640 1 -- 172.21.15.116:0/1195162105 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65bc00a9f0 con 0x7f65c4156bb0 2024-04-25T00:09:21.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.228+0000 7f65c949c640 1 -- 172.21.15.116:0/1195162105 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65c413ea00 con 0x7f65c4156bb0 2024-04-25T00:09:21.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.228+0000 7f65c949c640 1 -- 172.21.15.116:0/1195162105 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65c4068a60 con 0x7f65c4156bb0 2024-04-25T00:09:21.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.232+0000 7f65bb7fe640 1 -- 172.21.15.116:0/1195162105 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f65bc01a070 con 0x7f65c4156bb0 2024-04-25T00:09:21.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.232+0000 7f65bb7fe640 1 --2- 172.21.15.116:0/1195162105 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f65a40678c0 0x7f65a4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:21.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.232+0000 7f65bb7fe640 1 -- 172.21.15.116:0/1195162105 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7440+0+0 (secure 0 0 0) 0x7f65bc085ba0 con 0x7f65c4156bb0 2024-04-25T00:09:21.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.232+0000 7f65c2ffd640 1 --2- 172.21.15.116:0/1195162105 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f65a40678c0 0x7f65a4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:21.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.232+0000 7f65c2ffd640 1 --2- 172.21.15.116:0/1195162105 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f65a40678c0 0x7f65a4069d80 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f65b40ad000 tx=0x7f65b4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:21.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.236+0000 7f65bb7fe640 1 -- 172.21.15.116:0/1195162105 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f65bc055b60 con 0x7f65c4156bb0 2024-04-25T00:09:21.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.324+0000 7f22ef3ad640 1 -- 172.21.15.116:0/917206625 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f22e80ad630 con 0x7f22e80c55d0 2024-04-25T00:09:21.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.324+0000 7f22ddffb640 1 -- 172.21.15.116:0/917206625 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+6501 (secure 0 0 0) 0x7f22e40f3340 con 0x7f22e80c55d0 2024-04-25T00:09:21.327 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:09:21.327 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5236,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/1159002662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1159002662},{"type":"v1","addr":"172.21.15.174:6837","nonce":1159002662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:07:33.575034+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5775":{"gid":5775,"name":"c","rank":0,"incarnation":52,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6833/3006671671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3006671671},{"type":"v1","addr":"172.21.15.116:6833","nonce":3006671671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5775]},"id":1},{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:08:03.679028+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5868},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5868":{"gid":5868,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6835/1670061091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1670061091},{"type":"v1","addr":"172.21.15.116:6835","nonce":1670061091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5868,"qdb_cluster":[5868]},"id":2}]} 2024-04-25T00:09:21.328 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 61 2024-04-25T00:09:21.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.328+0000 7f22ef3ad640 1 -- 172.21.15.116:0/917206625 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f22cc0678c0 msgr2=0x7f22cc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:21.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.328+0000 7f22ef3ad640 1 --2- 172.21.15.116:0/917206625 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f22cc0678c0 0x7f22cc069d80 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f22e81eb100 tx=0x7f22d8009040 comp rx=0 tx=0).stop 2024-04-25T00:09:21.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.328+0000 7f22ef3ad640 1 -- 172.21.15.116:0/917206625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22e80c55d0 msgr2=0x7f22e80c31a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:21.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.328+0000 7f22ef3ad640 1 --2- 172.21.15.116:0/917206625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22e80c55d0 0x7f22e80c31a0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f22e400b2c0 tx=0x7f22e401b8b0 comp rx=0 tx=0).stop 2024-04-25T00:09:21.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.328+0000 7f22ef3ad640 1 -- 172.21.15.116:0/917206625 shutdown_connections 2024-04-25T00:09:21.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.328+0000 7f22ef3ad640 1 --2- 172.21.15.116:0/917206625 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f22cc0678c0 0x7f22cc069d80 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:21.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.328+0000 7f22ef3ad640 1 --2- 172.21.15.116:0/917206625 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f22e8155b20 0x7f22e81414b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:21.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.328+0000 7f22ef3ad640 1 --2- 172.21.15.116:0/917206625 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f22e814c700 0x7f22e8140f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:21.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.328+0000 7f22ef3ad640 1 --2- 172.21.15.116:0/917206625 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22e80c55d0 0x7f22e80c31a0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:21.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.328+0000 7f22ef3ad640 1 -- 172.21.15.116:0/917206625 >> 172.21.15.116:0/917206625 conn(0x7f22e806c720 msgr2=0x7f22e8152c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:09:21.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.328+0000 7f22ef3ad640 1 -- 172.21.15.116:0/917206625 shutdown_connections 2024-04-25T00:09:21.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.328+0000 7f22ef3ad640 1 -- 172.21.15.116:0/917206625 wait complete. 2024-04-25T00:09:21.345 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:09:21.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.432+0000 7f65c949c640 1 -- 172.21.15.116:0/1195162105 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f65c41511e0 con 0x7f65c4156bb0 2024-04-25T00:09:21.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.436+0000 7f65bb7fe640 1 -- 172.21.15.116:0/1195162105 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+6501 (secure 0 0 0) 0x7f65bc059e60 con 0x7f65c4156bb0 2024-04-25T00:09:21.437 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:09:21.437 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5236,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/1159002662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1159002662},{"type":"v1","addr":"172.21.15.174:6837","nonce":1159002662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:07:33.575034+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5775":{"gid":5775,"name":"c","rank":0,"incarnation":52,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6833/3006671671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3006671671},{"type":"v1","addr":"172.21.15.116:6833","nonce":3006671671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5775]},"id":1},{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:08:03.679028+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5868},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5868":{"gid":5868,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6835/1670061091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1670061091},{"type":"v1","addr":"172.21.15.116:6835","nonce":1670061091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5868,"qdb_cluster":[5868]},"id":2}]} 2024-04-25T00:09:21.437 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 61 2024-04-25T00:09:21.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.436+0000 7f65c949c640 1 -- 172.21.15.116:0/1195162105 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f65a40678c0 msgr2=0x7f65a4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:21.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.436+0000 7f65c949c640 1 --2- 172.21.15.116:0/1195162105 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f65a40678c0 0x7f65a4069d80 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f65b40ad000 tx=0x7f65b4008040 comp rx=0 tx=0).stop 2024-04-25T00:09:21.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.436+0000 7f65c949c640 1 -- 172.21.15.116:0/1195162105 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65c4156bb0 msgr2=0x7f65c41efe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:21.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.436+0000 7f65c949c640 1 --2- 172.21.15.116:0/1195162105 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65c4156bb0 0x7f65c41efe40 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f65bc00c7a0 tx=0x7f65bc00cc60 comp rx=0 tx=0).stop 2024-04-25T00:09:21.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.436+0000 7f65c949c640 1 -- 172.21.15.116:0/1195162105 shutdown_connections 2024-04-25T00:09:21.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.436+0000 7f65c949c640 1 --2- 172.21.15.116:0/1195162105 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f65a40678c0 0x7f65a4069d80 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:21.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.436+0000 7f65c949c640 1 --2- 172.21.15.116:0/1195162105 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65c4156bb0 0x7f65c41efe40 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:21.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.436+0000 7f65c949c640 1 --2- 172.21.15.116:0/1195162105 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f65c40c5200 0x7f65c41ef900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:21.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.436+0000 7f65c949c640 1 --2- 172.21.15.116:0/1195162105 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65c40c4760 0x7f65c406e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:21.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.436+0000 7f65c949c640 1 -- 172.21.15.116:0/1195162105 >> 172.21.15.116:0/1195162105 conn(0x7f65c406c720 msgr2=0x7f65c406cb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:09:21.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.436+0000 7f65c949c640 1 -- 172.21.15.116:0/1195162105 shutdown_connections 2024-04-25T00:09:21.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:21.436+0000 7f65c949c640 1 -- 172.21.15.116:0/1195162105 wait complete. 2024-04-25T00:09:21.458 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:09:23.346 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:09:23.460 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:09:23.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.472+0000 7fef25781640 1 Processor -- start 2024-04-25T00:09:23.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.472+0000 7fef25781640 1 -- start start 2024-04-25T00:09:23.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.472+0000 7fef25781640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fef2014c720 0x7fef201563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:23.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.472+0000 7fef25781640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef20071220 con 0x7fef20071680 2024-04-25T00:09:23.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.472+0000 7fef25781640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef200721e0 con 0x7fef200a94a0 2024-04-25T00:09:23.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.472+0000 7fef25781640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fef20153460 con 0x7fef2014c720 2024-04-25T00:09:23.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.472+0000 7fef1f7fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fef20071680 0x7fef20071ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:36100/0 (socket says 172.21.15.116:36100) 2024-04-25T00:09:23.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.472+0000 7fef1f7fe640 1 -- 172.21.15.116:0/3382230217 learned_addr learned my addr 172.21.15.116:0/3382230217 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:09:23.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.472+0000 7fef1effd640 1 --2- 172.21.15.116:0/3382230217 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fef2014c720 0x7fef201563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:23.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.472+0000 7fef1dffb640 1 -- 172.21.15.116:0/3382230217 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1496654537 0 0) 0x7fef20071220 con 0x7fef20071680 2024-04-25T00:09:23.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.472+0000 7fef1dffb640 1 -- 172.21.15.116:0/3382230217 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef0c003530 con 0x7fef20071680 2024-04-25T00:09:23.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.472+0000 7fef1dffb640 1 -- 172.21.15.116:0/3382230217 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 864156848 0 0) 0x7fef0c003530 con 0x7fef20071680 2024-04-25T00:09:23.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.472+0000 7fef1dffb640 1 -- 172.21.15.116:0/3382230217 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fef2014c720 msgr2=0x7fef201563d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:09:23.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.472+0000 7fef1dffb640 1 --2- 172.21.15.116:0/3382230217 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fef2014c720 0x7fef201563d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:23.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.472+0000 7fef1dffb640 1 -- 172.21.15.116:0/3382230217 >> v1:172.21.15.174:6789/0 conn(0x7fef200a94a0 legacy=0x7fef200c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:23.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.472+0000 7fef1dffb640 1 -- 172.21.15.116:0/3382230217 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef20152420 con 0x7fef20071680 2024-04-25T00:09:23.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.472+0000 7fef1dffb640 1 -- 172.21.15.116:0/3382230217 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7fef18001a00 con 0x7fef20071680 2024-04-25T00:09:23.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.472+0000 7fef1dffb640 1 -- 172.21.15.116:0/3382230217 >> v1:172.21.15.116:6789/0 conn(0x7fef20071680 legacy=0x7fef20071ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:23.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.472+0000 7fef1dffb640 1 --2- 172.21.15.116:0/3382230217 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef0c0039f0 0x7fef0c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:23.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.472+0000 7fef1dffb640 1 --2- 172.21.15.116:0/3382230217 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fef0c004810 0x7fef0c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:23.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.472+0000 7fef1dffb640 1 --2- 172.21.15.116:0/3382230217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef0c0051d0 0x7fef0c00dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:23.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.472+0000 7fef1dffb640 1 -- 172.21.15.116:0/3382230217 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fef20153460 con 0x7fef0c0051d0 2024-04-25T00:09:23.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.472+0000 7fef1dffb640 1 -- 172.21.15.116:0/3382230217 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fef0c00e2b0 con 0x7fef0c0039f0 2024-04-25T00:09:23.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.472+0000 7fef1dffb640 1 -- 172.21.15.116:0/3382230217 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fef0c00e4b0 con 0x7fef0c004810 2024-04-25T00:09:23.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.472+0000 7fef1effd640 1 --2- 172.21.15.116:0/3382230217 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fef2014c720 0x7fef201563d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:09:23.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.476+0000 7fef1f7fe640 1 --2- 172.21.15.116:0/3382230217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef0c0051d0 0x7fef0c00dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:23.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.476+0000 7fef1e7fc640 1 --2- 172.21.15.116:0/3382230217 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fef0c004810 0x7fef0c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:23.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.476+0000 7fef1effd640 1 --2- 172.21.15.116:0/3382230217 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef0c0039f0 0x7fef0c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:23.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.476+0000 7fef1f7fe640 1 -- 172.21.15.116:0/3382230217 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fef0c004810 msgr2=0x7fef0c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:09:23.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.476+0000 7fef1f7fe640 1 --2- 172.21.15.116:0/3382230217 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fef0c004810 0x7fef0c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:23.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.476+0000 7fef1f7fe640 1 -- 172.21.15.116:0/3382230217 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef0c0039f0 msgr2=0x7fef0c003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:09:23.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.476+0000 7fef1f7fe640 1 --2- 172.21.15.116:0/3382230217 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef0c0039f0 0x7fef0c003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:23.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.476+0000 7fef1f7fe640 1 -- 172.21.15.116:0/3382230217 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef0c003530 con 0x7fef0c0051d0 2024-04-25T00:09:23.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.476+0000 7fef1effd640 1 --2- 172.21.15.116:0/3382230217 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef0c0039f0 0x7fef0c003e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:09:23.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.476+0000 7fef1f7fe640 1 --2- 172.21.15.116:0/3382230217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef0c0051d0 0x7fef0c00dbe0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fef1800a850 tx=0x7fef1801f920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b2b2c40322192bb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:23.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.476+0000 7fef1dffb640 1 -- 172.21.15.116:0/3382230217 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef1800a4a0 con 0x7fef0c0051d0 2024-04-25T00:09:23.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.476+0000 7fef1dffb640 1 -- 172.21.15.116:0/3382230217 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef180b2070 con 0x7fef0c0051d0 2024-04-25T00:09:23.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.476+0000 7fef1dffb640 1 -- 172.21.15.116:0/3382230217 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef180ad8c0 con 0x7fef0c0051d0 2024-04-25T00:09:23.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.476+0000 7fef25781640 1 -- 172.21.15.116:0/3382230217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef0c0051d0 msgr2=0x7fef0c00dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:23.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.476+0000 7fef25781640 1 --2- 172.21.15.116:0/3382230217 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef0c0051d0 0x7fef0c00dbe0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fef1800a850 tx=0x7fef1801f920 comp rx=0 tx=0).stop 2024-04-25T00:09:23.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.476+0000 7fef1f7fe640 1 -- 172.21.15.116:0/3382230217 reap_dead start 2024-04-25T00:09:23.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.476+0000 7fef25781640 1 -- 172.21.15.116:0/3382230217 shutdown_connections 2024-04-25T00:09:23.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.476+0000 7fef25781640 1 -- 172.21.15.116:0/3382230217 >> 172.21.15.116:0/3382230217 conn(0x7fef200c3b00 msgr2=0x7fef200c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:09:23.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.476+0000 7fef25781640 1 -- 172.21.15.116:0/3382230217 shutdown_connections 2024-04-25T00:09:23.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.476+0000 7fef25781640 1 -- 172.21.15.116:0/3382230217 wait complete. 2024-04-25T00:09:23.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.480+0000 7fef25781640 1 Processor -- start 2024-04-25T00:09:23.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.480+0000 7fef25781640 1 -- start start 2024-04-25T00:09:23.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.480+0000 7fef25781640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fef2006dcb0 0x7fef200c4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:23.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.480+0000 7fef25781640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef201e7150 0x7fef201e75d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:23.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.480+0000 7fef25781640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef201e7b10 0x7fef201ea6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:23.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.480+0000 7fef25781640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fef20152980 con 0x7fef201e7b10 2024-04-25T00:09:23.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.480+0000 7fef25781640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fef201526a0 con 0x7fef201e7150 2024-04-25T00:09:23.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.480+0000 7fef25781640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fef20152c60 con 0x7fef2006dcb0 2024-04-25T00:09:23.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.480+0000 7fef1f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef201e7b10 0x7fef201ea6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:23.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.480+0000 7fef1f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef201e7b10 0x7fef201ea6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41290/0 (socket says 172.21.15.116:41290) 2024-04-25T00:09:23.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.480+0000 7fef1f7fe640 1 -- 172.21.15.116:0/1164802885 learned_addr learned my addr 172.21.15.116:0/1164802885 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:09:23.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.480+0000 7fef1e7fc640 1 --2- 172.21.15.116:0/1164802885 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef201e7150 0x7fef201e75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:23.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.480+0000 7fef1effd640 1 --2- 172.21.15.116:0/1164802885 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fef2006dcb0 0x7fef200c4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:23.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.480+0000 7fef1f7fe640 1 -- 172.21.15.116:0/1164802885 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fef2006dcb0 msgr2=0x7fef200c4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:23.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.480+0000 7fef1f7fe640 1 --2- 172.21.15.116:0/1164802885 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fef2006dcb0 0x7fef200c4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:23.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.480+0000 7fef1f7fe640 1 -- 172.21.15.116:0/1164802885 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef201e7150 msgr2=0x7fef201e75d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:23.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.480+0000 7fef1f7fe640 1 --2- 172.21.15.116:0/1164802885 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef201e7150 0x7fef201e75d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:23.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.480+0000 7fef1f7fe640 1 -- 172.21.15.116:0/1164802885 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef200acc60 con 0x7fef201e7b10 2024-04-25T00:09:23.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.480+0000 7fef1e7fc640 1 --2- 172.21.15.116:0/1164802885 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef201e7150 0x7fef201e75d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:09:23.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.480+0000 7fef1f7fe640 1 --2- 172.21.15.116:0/1164802885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef201e7b10 0x7fef201ea6c0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fef180af040 tx=0x7fef0c005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:23.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.480+0000 7fef177fe640 1 -- 172.21.15.116:0/1164802885 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef1800a480 con 0x7fef201e7b10 2024-04-25T00:09:23.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.480+0000 7fef177fe640 1 -- 172.21.15.116:0/1164802885 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef180ad460 con 0x7fef201e7b10 2024-04-25T00:09:23.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.480+0000 7fef25781640 1 -- 172.21.15.116:0/1164802885 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef200ac980 con 0x7fef201e7b10 2024-04-25T00:09:23.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.480+0000 7fef177fe640 1 -- 172.21.15.116:0/1164802885 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef180ad810 con 0x7fef201e7b10 2024-04-25T00:09:23.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.480+0000 7fef25781640 1 -- 172.21.15.116:0/1164802885 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef201451d0 con 0x7fef201e7b10 2024-04-25T00:09:23.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.480+0000 7fef25781640 1 -- 172.21.15.116:0/1164802885 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef20068350 con 0x7fef201e7b10 2024-04-25T00:09:23.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.480+0000 7fef177fe640 1 -- 172.21.15.116:0/1164802885 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fef180ad9b0 con 0x7fef201e7b10 2024-04-25T00:09:23.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.484+0000 7fef177fe640 1 --2- 172.21.15.116:0/1164802885 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fef000678c0 0x7fef00069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:23.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.484+0000 7fef177fe640 1 -- 172.21.15.116:0/1164802885 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7440+0+0 (secure 0 0 0) 0x7fef18123f10 con 0x7fef201e7b10 2024-04-25T00:09:23.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.484+0000 7fef1effd640 1 --2- 172.21.15.116:0/1164802885 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fef000678c0 0x7fef00069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:23.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.484+0000 7fef1effd640 1 --2- 172.21.15.116:0/1164802885 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fef000678c0 0x7fef00069d80 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fef20067990 tx=0x7fef08008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:23.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.488+0000 7fef177fe640 1 -- 172.21.15.116:0/1164802885 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fef20152420 con 0x7fef201e7b10 2024-04-25T00:09:23.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.576+0000 7efe20b9a640 1 Processor -- start 2024-04-25T00:09:23.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.576+0000 7efe20b9a640 1 -- start start 2024-04-25T00:09:23.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.580+0000 7efe20b9a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efe1c14fe30 0x7efe1c153040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:23.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.580+0000 7efe20b9a640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7efe1c06f4d0 0x7efe1c06f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:23.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.580+0000 7efe20b9a640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe1c153580 con 0x7efe1c0a8fc0 2024-04-25T00:09:23.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.580+0000 7efe20b9a640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7efe1c0a7610 con 0x7efe1c06f4d0 2024-04-25T00:09:23.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.580+0000 7efe20b9a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7efe1c0a7330 con 0x7efe1c14fe30 2024-04-25T00:09:23.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.580+0000 7efe1a59c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efe1c14fe30 0x7efe1c153040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:23.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.580+0000 7efe19d9b640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7efe1c06f4d0 0x7efe1c06f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:23.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.580+0000 7efe1a59c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efe1c14fe30 0x7efe1c153040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:47052/0 (socket says 172.21.15.116:47052) 2024-04-25T00:09:23.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.580+0000 7efe1a59c640 1 -- 172.21.15.116:0/2898182128 learned_addr learned my addr 172.21.15.116:0/2898182128 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:09:23.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.580+0000 7efe19d9b640 1 -- 172.21.15.116:0/2898182128 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efe1c14fe30 msgr2=0x7efe1c153040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:09:23.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.580+0000 7efe19d9b640 1 --2- 172.21.15.116:0/2898182128 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efe1c14fe30 0x7efe1c153040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:23.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.580+0000 7efe19d9b640 1 -- 172.21.15.116:0/2898182128 >> v1:172.21.15.174:6789/0 conn(0x7efe1c0a8fc0 legacy=0x7efe1c0a9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:23.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.580+0000 7efe19d9b640 1 -- 172.21.15.116:0/2898182128 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe1c0c3b80 con 0x7efe1c06f4d0 2024-04-25T00:09:23.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.580+0000 7efe1a59c640 1 --2- 172.21.15.116:0/2898182128 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efe1c14fe30 0x7efe1c153040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:09:23.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.580+0000 7efe19d9b640 1 --2- 172.21.15.116:0/2898182128 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efe1c06f4d0 0x7efe1c06f930 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7efe0c0097c0 tx=0x7efe0c01c3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68892e49ea58f2dd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:23.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.580+0000 7efe1959a640 1 -- 172.21.15.116:0/2898182128 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe0c0b1070 con 0x7efe1c06f4d0 2024-04-25T00:09:23.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.580+0000 7efe1959a640 1 -- 172.21.15.116:0/2898182128 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe0c0a8050 con 0x7efe1c06f4d0 2024-04-25T00:09:23.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.580+0000 7efe1959a640 1 -- 172.21.15.116:0/2898182128 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe0c0ac860 con 0x7efe1c06f4d0 2024-04-25T00:09:23.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.584+0000 7efe20b9a640 1 -- 172.21.15.116:0/2898182128 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efe1c06f4d0 msgr2=0x7efe1c06f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:23.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.584+0000 7efe20b9a640 1 --2- 172.21.15.116:0/2898182128 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efe1c06f4d0 0x7efe1c06f930 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7efe0c0097c0 tx=0x7efe0c01c3e0 comp rx=0 tx=0).stop 2024-04-25T00:09:23.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.584+0000 7efe20b9a640 1 -- 172.21.15.116:0/2898182128 shutdown_connections 2024-04-25T00:09:23.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.584+0000 7efe20b9a640 1 --2- 172.21.15.116:0/2898182128 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efe1c06f4d0 0x7efe1c06f930 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:23.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.584+0000 7efe20b9a640 1 --2- 172.21.15.116:0/2898182128 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efe1c14fe30 0x7efe1c153040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:23.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.584+0000 7efe20b9a640 1 -- 172.21.15.116:0/2898182128 >> 172.21.15.116:0/2898182128 conn(0x7efe1c06b210 msgr2=0x7efe1c06b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:09:23.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.584+0000 7efe20b9a640 1 -- 172.21.15.116:0/2898182128 shutdown_connections 2024-04-25T00:09:23.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.584+0000 7efe20b9a640 1 -- 172.21.15.116:0/2898182128 wait complete. 2024-04-25T00:09:23.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.584+0000 7efe20b9a640 1 Processor -- start 2024-04-25T00:09:23.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.584+0000 7efe20b9a640 1 -- start start 2024-04-25T00:09:23.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.584+0000 7efe20b9a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efe1c06f4d0 0x7efe1c13fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:23.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.584+0000 7efe20b9a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efe1c0a8fc0 0x7efe1c071470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:23.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.584+0000 7efe20b9a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe1c14fe30 0x7efe1c1517a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:23.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.584+0000 7efe20b9a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7efe1c0a9b60 con 0x7efe1c14fe30 2024-04-25T00:09:23.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.584+0000 7efe20b9a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7efe1c150210 con 0x7efe1c0a8fc0 2024-04-25T00:09:23.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.584+0000 7efe20b9a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7efe1c1ef070 con 0x7efe1c06f4d0 2024-04-25T00:09:23.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.584+0000 7efe1ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe1c14fe30 0x7efe1c1517a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:23.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.584+0000 7efe1ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe1c14fe30 0x7efe1c1517a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41300/0 (socket says 172.21.15.116:41300) 2024-04-25T00:09:23.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.584+0000 7efe1ad9d640 1 -- 172.21.15.116:0/1111627465 learned_addr learned my addr 172.21.15.116:0/1111627465 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:09:23.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.584+0000 7efe1a59c640 1 --2- 172.21.15.116:0/1111627465 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efe1c06f4d0 0x7efe1c13fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:23.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.584+0000 7efe19d9b640 1 --2- 172.21.15.116:0/1111627465 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efe1c0a8fc0 0x7efe1c071470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:23.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.584+0000 7efe1ad9d640 1 -- 172.21.15.116:0/1111627465 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efe1c06f4d0 msgr2=0x7efe1c13fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:23.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.584+0000 7efe1ad9d640 1 --2- 172.21.15.116:0/1111627465 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efe1c06f4d0 0x7efe1c13fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:23.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.584+0000 7efe1ad9d640 1 -- 172.21.15.116:0/1111627465 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efe1c0a8fc0 msgr2=0x7efe1c071470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:23.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.584+0000 7efe1ad9d640 1 --2- 172.21.15.116:0/1111627465 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efe1c0a8fc0 0x7efe1c071470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:23.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.584+0000 7efe1ad9d640 1 -- 172.21.15.116:0/1111627465 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe1c1521a0 con 0x7efe1c14fe30 2024-04-25T00:09:23.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.584+0000 7efe1a59c640 1 --2- 172.21.15.116:0/1111627465 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efe1c06f4d0 0x7efe1c13fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:09:23.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.584+0000 7efe1ad9d640 1 --2- 172.21.15.116:0/1111627465 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe1c14fe30 0x7efe1c1517a0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7efe1400c7a0 tx=0x7efe1400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:23.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.584+0000 7efe12ffd640 1 -- 172.21.15.116:0/1111627465 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe14003b80 con 0x7efe1c14fe30 2024-04-25T00:09:23.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.584+0000 7efe12ffd640 1 -- 172.21.15.116:0/1111627465 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe14003d20 con 0x7efe1c14fe30 2024-04-25T00:09:23.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.584+0000 7efe20b9a640 1 -- 172.21.15.116:0/1111627465 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe1c1eed00 con 0x7efe1c14fe30 2024-04-25T00:09:23.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.584+0000 7efe12ffd640 1 -- 172.21.15.116:0/1111627465 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe1400aa90 con 0x7efe1c14fe30 2024-04-25T00:09:23.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.584+0000 7efe20b9a640 1 -- 172.21.15.116:0/1111627465 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe1c1458a0 con 0x7efe1c14fe30 2024-04-25T00:09:23.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.588+0000 7efe20b9a640 1 -- 172.21.15.116:0/1111627465 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe1c1eeb30 con 0x7efe1c14fe30 2024-04-25T00:09:23.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.588+0000 7efe12ffd640 1 -- 172.21.15.116:0/1111627465 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7efe1400ac30 con 0x7efe1c14fe30 2024-04-25T00:09:23.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.588+0000 7efe12ffd640 1 --2- 172.21.15.116:0/1111627465 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efdfc0678c0 0x7efdfc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:23.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.588+0000 7efe12ffd640 1 -- 172.21.15.116:0/1111627465 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7440+0+0 (secure 0 0 0) 0x7efe140861f0 con 0x7efe1c14fe30 2024-04-25T00:09:23.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.588+0000 7efe1a59c640 1 --2- 172.21.15.116:0/1111627465 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efdfc0678c0 0x7efdfc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:23.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.588+0000 7efe1a59c640 1 --2- 172.21.15.116:0/1111627465 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efdfc0678c0 0x7efdfc069d80 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7efe1c15fd70 tx=0x7efe04008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:23.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.592+0000 7efe12ffd640 1 -- 172.21.15.116:0/1111627465 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7efe14056700 con 0x7efe1c14fe30 2024-04-25T00:09:23.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.684+0000 7fef25781640 1 -- 172.21.15.116:0/1164802885 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fef200c3640 con 0x7fef201e7b10 2024-04-25T00:09:23.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.684+0000 7fef177fe640 1 -- 172.21.15.116:0/1164802885 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+6501 (secure 0 0 0) 0x7fef180f43a0 con 0x7fef201e7b10 2024-04-25T00:09:23.687 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:09:23.687 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5236,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/1159002662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1159002662},{"type":"v1","addr":"172.21.15.174:6837","nonce":1159002662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:07:33.575034+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5775":{"gid":5775,"name":"c","rank":0,"incarnation":52,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6833/3006671671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3006671671},{"type":"v1","addr":"172.21.15.116:6833","nonce":3006671671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5775]},"id":1},{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:08:03.679028+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5868},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5868":{"gid":5868,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6835/1670061091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1670061091},{"type":"v1","addr":"172.21.15.116:6835","nonce":1670061091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5868,"qdb_cluster":[5868]},"id":2}]} 2024-04-25T00:09:23.687 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 61 2024-04-25T00:09:23.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.688+0000 7fef25781640 1 -- 172.21.15.116:0/1164802885 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fef000678c0 msgr2=0x7fef00069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:23.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.688+0000 7fef25781640 1 --2- 172.21.15.116:0/1164802885 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fef000678c0 0x7fef00069d80 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fef20067990 tx=0x7fef08008040 comp rx=0 tx=0).stop 2024-04-25T00:09:23.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.688+0000 7fef25781640 1 -- 172.21.15.116:0/1164802885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef201e7b10 msgr2=0x7fef201ea6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:23.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.688+0000 7fef25781640 1 --2- 172.21.15.116:0/1164802885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef201e7b10 0x7fef201ea6c0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fef180af040 tx=0x7fef0c005840 comp rx=0 tx=0).stop 2024-04-25T00:09:23.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.688+0000 7fef25781640 1 -- 172.21.15.116:0/1164802885 shutdown_connections 2024-04-25T00:09:23.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.688+0000 7fef25781640 1 --2- 172.21.15.116:0/1164802885 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fef000678c0 0x7fef00069d80 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:23.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.688+0000 7fef25781640 1 --2- 172.21.15.116:0/1164802885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef201e7b10 0x7fef201ea6c0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:23.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.688+0000 7fef25781640 1 --2- 172.21.15.116:0/1164802885 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef201e7150 0x7fef201e75d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:23.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.688+0000 7fef25781640 1 --2- 172.21.15.116:0/1164802885 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fef2006dcb0 0x7fef200c4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:23.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.688+0000 7fef25781640 1 -- 172.21.15.116:0/1164802885 >> 172.21.15.116:0/1164802885 conn(0x7fef200c3b00 msgr2=0x7fef20151a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:09:23.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.688+0000 7fef25781640 1 -- 172.21.15.116:0/1164802885 shutdown_connections 2024-04-25T00:09:23.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.688+0000 7fef25781640 1 -- 172.21.15.116:0/1164802885 wait complete. 2024-04-25T00:09:23.704 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:09:23.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.788+0000 7efe20b9a640 1 -- 172.21.15.116:0/1111627465 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efe1c068a60 con 0x7efe1c14fe30 2024-04-25T00:09:23.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.788+0000 7efe12ffd640 1 -- 172.21.15.116:0/1111627465 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+6501 (secure 0 0 0) 0x7efe14056000 con 0x7efe1c14fe30 2024-04-25T00:09:23.792 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:09:23.792 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5236,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/1159002662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1159002662},{"type":"v1","addr":"172.21.15.174:6837","nonce":1159002662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:07:33.575034+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5775":{"gid":5775,"name":"c","rank":0,"incarnation":52,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6833/3006671671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3006671671},{"type":"v1","addr":"172.21.15.116:6833","nonce":3006671671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5775]},"id":1},{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:08:03.679028+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5868},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5868":{"gid":5868,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6835/1670061091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1670061091},{"type":"v1","addr":"172.21.15.116:6835","nonce":1670061091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5868,"qdb_cluster":[5868]},"id":2}]} 2024-04-25T00:09:23.792 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 61 2024-04-25T00:09:23.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.792+0000 7efe20b9a640 1 -- 172.21.15.116:0/1111627465 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efdfc0678c0 msgr2=0x7efdfc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:23.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.792+0000 7efe20b9a640 1 --2- 172.21.15.116:0/1111627465 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efdfc0678c0 0x7efdfc069d80 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7efe1c15fd70 tx=0x7efe04008040 comp rx=0 tx=0).stop 2024-04-25T00:09:23.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.792+0000 7efe20b9a640 1 -- 172.21.15.116:0/1111627465 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe1c14fe30 msgr2=0x7efe1c1517a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:23.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.792+0000 7efe20b9a640 1 --2- 172.21.15.116:0/1111627465 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe1c14fe30 0x7efe1c1517a0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7efe1400c7a0 tx=0x7efe1400cc60 comp rx=0 tx=0).stop 2024-04-25T00:09:23.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.792+0000 7efe20b9a640 1 -- 172.21.15.116:0/1111627465 shutdown_connections 2024-04-25T00:09:23.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.792+0000 7efe20b9a640 1 --2- 172.21.15.116:0/1111627465 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efdfc0678c0 0x7efdfc069d80 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:23.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.792+0000 7efe20b9a640 1 --2- 172.21.15.116:0/1111627465 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe1c14fe30 0x7efe1c1517a0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:23.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.792+0000 7efe20b9a640 1 --2- 172.21.15.116:0/1111627465 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efe1c0a8fc0 0x7efe1c071470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:23.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.792+0000 7efe20b9a640 1 --2- 172.21.15.116:0/1111627465 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efe1c06f4d0 0x7efe1c13fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:23.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.792+0000 7efe20b9a640 1 -- 172.21.15.116:0/1111627465 >> 172.21.15.116:0/1111627465 conn(0x7efe1c06b210 msgr2=0x7efe1c0a9450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:09:23.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.792+0000 7efe20b9a640 1 -- 172.21.15.116:0/1111627465 shutdown_connections 2024-04-25T00:09:23.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:23.792+0000 7efe20b9a640 1 -- 172.21.15.116:0/1111627465 wait complete. 2024-04-25T00:09:23.811 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:09:25.705 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:09:25.812 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:09:25.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.836+0000 7f50448fa640 1 Processor -- start 2024-04-25T00:09:25.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.836+0000 7f50448fa640 1 -- start start 2024-04-25T00:09:25.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.836+0000 7f50448fa640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50400c2bf0 0x7f5040154290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:25.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.836+0000 7f50448fa640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f50400c3140 0x7f504006c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:25.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.836+0000 7f50448fa640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5040153810 con 0x7f50400730a0 2024-04-25T00:09:25.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.836+0000 7f50448fa640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f50401547d0 con 0x7f50400c3140 2024-04-25T00:09:25.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.836+0000 7f50448fa640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f50400c3690 con 0x7f50400c2bf0 2024-04-25T00:09:25.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.836+0000 7f503dd9b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50400c2bf0 0x7f5040154290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:25.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.836+0000 7f503ed9d640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f50400c3140 0x7f504006c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:25.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.836+0000 7f503dd9b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50400c2bf0 0x7f5040154290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:40742/0 (socket says 172.21.15.116:40742) 2024-04-25T00:09:25.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.836+0000 7f503dd9b640 1 -- 172.21.15.116:0/3482123184 learned_addr learned my addr 172.21.15.116:0/3482123184 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:09:25.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.836+0000 7f503dd9b640 1 -- 172.21.15.116:0/3482123184 >> v2:172.21.15.174:3300/0 conn(0x7f50400c3140 msgr2=0x7f504006c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:09:25.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.836+0000 7f503dd9b640 1 --2- 172.21.15.116:0/3482123184 >> v2:172.21.15.174:3300/0 conn(0x7f50400c3140 0x7f504006c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:25.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.836+0000 7f503dd9b640 1 -- 172.21.15.116:0/3482123184 >> v1:172.21.15.174:6789/0 conn(0x7f50400730a0 legacy=0x7f50400723b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:25.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.836+0000 7f503dd9b640 1 -- 172.21.15.116:0/3482123184 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50400c3970 con 0x7f50400c2bf0 2024-04-25T00:09:25.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.836+0000 7f503ed9d640 1 --2- 172.21.15.116:0/3482123184 >> v2:172.21.15.174:3300/0 conn(0x7f50400c3140 0x7f504006c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:09:25.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.840+0000 7f503dd9b640 1 --2- 172.21.15.116:0/3482123184 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50400c2bf0 0x7f5040154290 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5034009870 tx=0x7f503401c560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e3075f3865f1a19 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:25.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.840+0000 7f503d59a640 1 -- 172.21.15.116:0/3482123184 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50340b2070 con 0x7f50400c2bf0 2024-04-25T00:09:25.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.840+0000 7f503d59a640 1 -- 172.21.15.116:0/3482123184 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50340a9070 con 0x7f50400c2bf0 2024-04-25T00:09:25.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.840+0000 7f503d59a640 1 -- 172.21.15.116:0/3482123184 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50340ad880 con 0x7f50400c2bf0 2024-04-25T00:09:25.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.840+0000 7f50448fa640 1 -- 172.21.15.116:0/3482123184 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50400c2bf0 msgr2=0x7f5040154290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:25.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.840+0000 7f50448fa640 1 --2- 172.21.15.116:0/3482123184 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50400c2bf0 0x7f5040154290 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5034009870 tx=0x7f503401c560 comp rx=0 tx=0).stop 2024-04-25T00:09:25.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.840+0000 7f50448fa640 1 -- 172.21.15.116:0/3482123184 shutdown_connections 2024-04-25T00:09:25.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.840+0000 7f50448fa640 1 --2- 172.21.15.116:0/3482123184 >> v2:172.21.15.174:3300/0 conn(0x7f50400c3140 0x7f504006c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:25.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.840+0000 7f50448fa640 1 --2- 172.21.15.116:0/3482123184 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50400c2bf0 0x7f5040154290 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:25.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.840+0000 7f50448fa640 1 -- 172.21.15.116:0/3482123184 >> 172.21.15.116:0/3482123184 conn(0x7f50400c42e0 msgr2=0x7f50400c46f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:09:25.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.840+0000 7f50448fa640 1 -- 172.21.15.116:0/3482123184 shutdown_connections 2024-04-25T00:09:25.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.840+0000 7f50448fa640 1 -- 172.21.15.116:0/3482123184 wait complete. 2024-04-25T00:09:25.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.840+0000 7f50448fa640 1 Processor -- start 2024-04-25T00:09:25.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.840+0000 7f50448fa640 1 -- start start 2024-04-25T00:09:25.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.840+0000 7f50448fa640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50400730a0 0x7f5040149560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:25.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.840+0000 7f50448fa640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50400c2bf0 0x7f50400c0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:25.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.840+0000 7f50448fa640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50400c3140 0x7f504014eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:25.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.840+0000 7f50448fa640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f50400c7400 con 0x7f50400c3140 2024-04-25T00:09:25.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.840+0000 7f50448fa640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f50400c7280 con 0x7f50400c2bf0 2024-04-25T00:09:25.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.840+0000 7f50448fa640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f50401608d0 con 0x7f50400730a0 2024-04-25T00:09:25.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.840+0000 7f503ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50400c3140 0x7f504014eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:25.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.840+0000 7f503ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50400c3140 0x7f504014eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44836/0 (socket says 172.21.15.116:44836) 2024-04-25T00:09:25.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.840+0000 7f503ed9d640 1 -- 172.21.15.116:0/2270870392 learned_addr learned my addr 172.21.15.116:0/2270870392 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:09:25.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.840+0000 7f503e59c640 1 --2- 172.21.15.116:0/2270870392 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50400730a0 0x7f5040149560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:25.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.840+0000 7f503dd9b640 1 --2- 172.21.15.116:0/2270870392 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50400c2bf0 0x7f50400c0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:25.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.840+0000 7f503ed9d640 1 -- 172.21.15.116:0/2270870392 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50400730a0 msgr2=0x7f5040149560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:25.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.840+0000 7f503ed9d640 1 --2- 172.21.15.116:0/2270870392 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50400730a0 0x7f5040149560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:25.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.840+0000 7f503ed9d640 1 -- 172.21.15.116:0/2270870392 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50400c2bf0 msgr2=0x7f50400c0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:25.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.840+0000 7f503ed9d640 1 --2- 172.21.15.116:0/2270870392 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50400c2bf0 0x7f50400c0c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:25.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.840+0000 7f503ed9d640 1 -- 172.21.15.116:0/2270870392 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50401eef30 con 0x7f50400c3140 2024-04-25T00:09:25.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.840+0000 7f503e59c640 1 --2- 172.21.15.116:0/2270870392 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50400730a0 0x7f5040149560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:09:25.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.844+0000 7f503ed9d640 1 --2- 172.21.15.116:0/2270870392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50400c3140 0x7f504014eae0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f5038009f00 tx=0x7f503800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:25.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.844+0000 7f502effd640 1 -- 172.21.15.116:0/2270870392 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5038018030 con 0x7f50400c3140 2024-04-25T00:09:25.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.844+0000 7f502effd640 1 -- 172.21.15.116:0/2270870392 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f503800ce50 con 0x7f50400c3140 2024-04-25T00:09:25.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.844+0000 7f50448fa640 1 -- 172.21.15.116:0/2270870392 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5040154c20 con 0x7f50400c3140 2024-04-25T00:09:25.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.844+0000 7f502effd640 1 -- 172.21.15.116:0/2270870392 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5038003bc0 con 0x7f50400c3140 2024-04-25T00:09:25.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.844+0000 7f50448fa640 1 -- 172.21.15.116:0/2270870392 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5040149aa0 con 0x7f50400c3140 2024-04-25T00:09:25.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.844+0000 7f50448fa640 1 -- 172.21.15.116:0/2270870392 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5040068a60 con 0x7f50400c3140 2024-04-25T00:09:25.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.844+0000 7f502effd640 1 -- 172.21.15.116:0/2270870392 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f503801b070 con 0x7f50400c3140 2024-04-25T00:09:25.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.844+0000 7f502effd640 1 --2- 172.21.15.116:0/2270870392 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f50200678c0 0x7f5020069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:25.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.844+0000 7f502effd640 1 -- 172.21.15.116:0/2270870392 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7440+0+0 (secure 0 0 0) 0x7f5038086d40 con 0x7f50400c3140 2024-04-25T00:09:25.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.844+0000 7f503e59c640 1 --2- 172.21.15.116:0/2270870392 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f50200678c0 0x7f5020069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:25.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.844+0000 7f503e59c640 1 --2- 172.21.15.116:0/2270870392 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f50200678c0 0x7f5020069d80 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f5030001670 tx=0x7f5030009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:25.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.848+0000 7f502effd640 1 -- 172.21.15.116:0/2270870392 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f5038010040 con 0x7f50400c3140 2024-04-25T00:09:25.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.928+0000 7fd6b3450640 1 Processor -- start 2024-04-25T00:09:25.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.928+0000 7fd6b3450640 1 -- start start 2024-04-25T00:09:25.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.928+0000 7fd6b3450640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd6ac152e40 0x7fd6ac0710b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:25.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.928+0000 7fd6b3450640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fd6ac0715f0 0x7fd6ac1557c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:25.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.928+0000 7fd6b3450640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd6ac0c3f70 0x7fd6ac155d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:25.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.928+0000 7fd6b3450640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fd6ac0c4c90 con 0x7fd6ac0c3f70 2024-04-25T00:09:25.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.928+0000 7fd6b3450640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fd6ac156240 con 0x7fd6ac0715f0 2024-04-25T00:09:25.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.928+0000 7fd6b3450640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fd6ac153220 con 0x7fd6ac152e40 2024-04-25T00:09:25.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.928+0000 7fd6b19ed640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd6ac0c3f70 0x7fd6ac155d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:25.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.928+0000 7fd6b19ed640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd6ac0c3f70 0x7fd6ac155d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44844/0 (socket says 172.21.15.116:44844) 2024-04-25T00:09:25.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.928+0000 7fd6b19ed640 1 -- 172.21.15.116:0/1914821144 learned_addr learned my addr 172.21.15.116:0/1914821144 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:09:25.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.932+0000 7fd6b11ec640 1 --2- 172.21.15.116:0/1914821144 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd6ac152e40 0x7fd6ac0710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:25.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.932+0000 7fd6b09eb640 1 --2- 172.21.15.116:0/1914821144 >> v2:172.21.15.174:3300/0 conn(0x7fd6ac0715f0 0x7fd6ac1557c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:25.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.932+0000 7fd6b19ed640 1 -- 172.21.15.116:0/1914821144 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd6ac152e40 msgr2=0x7fd6ac0710b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:09:25.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.932+0000 7fd6b19ed640 1 --2- 172.21.15.116:0/1914821144 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd6ac152e40 0x7fd6ac0710b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:25.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.932+0000 7fd6b19ed640 1 -- 172.21.15.116:0/1914821144 >> v2:172.21.15.174:3300/0 conn(0x7fd6ac0715f0 msgr2=0x7fd6ac1557c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:09:25.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.932+0000 7fd6b19ed640 1 --2- 172.21.15.116:0/1914821144 >> v2:172.21.15.174:3300/0 conn(0x7fd6ac0715f0 0x7fd6ac1557c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:25.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.932+0000 7fd6b19ed640 1 -- 172.21.15.116:0/1914821144 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6ac0c7a90 con 0x7fd6ac0c3f70 2024-04-25T00:09:25.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.932+0000 7fd6b09eb640 1 --2- 172.21.15.116:0/1914821144 >> v2:172.21.15.174:3300/0 conn(0x7fd6ac0715f0 0x7fd6ac1557c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:09:25.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.932+0000 7fd6b19ed640 1 --2- 172.21.15.116:0/1914821144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6ac0c3f70 0x7fd6ac155d00 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac0c34e0 tx=0x7fd6a401c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7524cf3fb2ed9f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:25.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.932+0000 7fd6a3fff640 1 -- 172.21.15.116:0/1914821144 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6a40a57b0 con 0x7fd6ac0c3f70 2024-04-25T00:09:25.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.932+0000 7fd6a3fff640 1 -- 172.21.15.116:0/1914821144 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6a40a5950 con 0x7fd6ac0c3f70 2024-04-25T00:09:25.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.932+0000 7fd6a3fff640 1 -- 172.21.15.116:0/1914821144 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6a40a5d20 con 0x7fd6ac0c3f70 2024-04-25T00:09:25.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.936+0000 7fd6b3450640 1 -- 172.21.15.116:0/1914821144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6ac0c3f70 msgr2=0x7fd6ac155d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:25.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.936+0000 7fd6b3450640 1 --2- 172.21.15.116:0/1914821144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6ac0c3f70 0x7fd6ac155d00 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac0c34e0 tx=0x7fd6a401c930 comp rx=0 tx=0).stop 2024-04-25T00:09:25.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.936+0000 7fd6b3450640 1 -- 172.21.15.116:0/1914821144 shutdown_connections 2024-04-25T00:09:25.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.936+0000 7fd6b3450640 1 --2- 172.21.15.116:0/1914821144 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6ac0c3f70 0x7fd6ac155d00 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:25.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.936+0000 7fd6b3450640 1 --2- 172.21.15.116:0/1914821144 >> v2:172.21.15.174:3300/0 conn(0x7fd6ac0715f0 0x7fd6ac1557c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:25.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.936+0000 7fd6b3450640 1 --2- 172.21.15.116:0/1914821144 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd6ac152e40 0x7fd6ac0710b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:25.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.936+0000 7fd6b3450640 1 -- 172.21.15.116:0/1914821144 >> 172.21.15.116:0/1914821144 conn(0x7fd6ac0c9160 msgr2=0x7fd6ac0c9570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:09:25.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.936+0000 7fd6b3450640 1 -- 172.21.15.116:0/1914821144 shutdown_connections 2024-04-25T00:09:25.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.936+0000 7fd6b3450640 1 -- 172.21.15.116:0/1914821144 wait complete. 2024-04-25T00:09:25.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.936+0000 7fd6b3450640 1 Processor -- start 2024-04-25T00:09:25.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.936+0000 7fd6b3450640 1 -- start start 2024-04-25T00:09:25.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.936+0000 7fd6b3450640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6ac0715f0 0x7fd6ac0abe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:25.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.936+0000 7fd6b3450640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd6ac0c3f70 0x7fd6ac0acbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:25.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.936+0000 7fd6b3450640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd6ac152e40 0x7fd6ac148260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:25.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.936+0000 7fd6b3450640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd6ac0c5280 con 0x7fd6ac0715f0 2024-04-25T00:09:25.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.936+0000 7fd6b3450640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd6ac0c5100 con 0x7fd6ac0c3f70 2024-04-25T00:09:25.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.936+0000 7fd6b3450640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fd6ac0c2bd0 con 0x7fd6ac152e40 2024-04-25T00:09:25.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.936+0000 7fd6b11ec640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6ac0715f0 0x7fd6ac0abe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:25.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.936+0000 7fd6b11ec640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6ac0715f0 0x7fd6ac0abe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44856/0 (socket says 172.21.15.116:44856) 2024-04-25T00:09:25.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.936+0000 7fd6b11ec640 1 -- 172.21.15.116:0/2505790985 learned_addr learned my addr 172.21.15.116:0/2505790985 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:09:25.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.936+0000 7fd6b09eb640 1 --2- 172.21.15.116:0/2505790985 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd6ac0c3f70 0x7fd6ac0acbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:25.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.936+0000 7fd6b19ed640 1 --2- 172.21.15.116:0/2505790985 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd6ac152e40 0x7fd6ac148260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:25.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.936+0000 7fd6b11ec640 1 -- 172.21.15.116:0/2505790985 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd6ac152e40 msgr2=0x7fd6ac148260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:25.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.936+0000 7fd6b11ec640 1 --2- 172.21.15.116:0/2505790985 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd6ac152e40 0x7fd6ac148260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:25.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.936+0000 7fd6b11ec640 1 -- 172.21.15.116:0/2505790985 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd6ac0c3f70 msgr2=0x7fd6ac0acbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:25.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.936+0000 7fd6b11ec640 1 --2- 172.21.15.116:0/2505790985 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd6ac0c3f70 0x7fd6ac0acbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:25.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.936+0000 7fd6b11ec640 1 -- 172.21.15.116:0/2505790985 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6ac1f1fb0 con 0x7fd6ac0715f0 2024-04-25T00:09:25.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.936+0000 7fd6b09eb640 1 --2- 172.21.15.116:0/2505790985 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd6ac0c3f70 0x7fd6ac0acbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:09:25.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.936+0000 7fd6b11ec640 1 --2- 172.21.15.116:0/2505790985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6ac0715f0 0x7fd6ac0abe10 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fd69c00d620 tx=0x7fd69c00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:25.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.936+0000 7fd6a1ffb640 1 -- 172.21.15.116:0/2505790985 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd69c00b620 con 0x7fd6ac0715f0 2024-04-25T00:09:25.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.936+0000 7fd6a1ffb640 1 -- 172.21.15.116:0/2505790985 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd69c019070 con 0x7fd6ac0715f0 2024-04-25T00:09:25.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.936+0000 7fd6b3450640 1 -- 172.21.15.116:0/2505790985 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6ac0ae950 con 0x7fd6ac0715f0 2024-04-25T00:09:25.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.936+0000 7fd6a1ffb640 1 -- 172.21.15.116:0/2505790985 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd69c00ba20 con 0x7fd6ac0715f0 2024-04-25T00:09:25.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.936+0000 7fd6b3450640 1 -- 172.21.15.116:0/2505790985 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6ac15f890 con 0x7fd6ac0715f0 2024-04-25T00:09:25.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.936+0000 7fd6b3450640 1 -- 172.21.15.116:0/2505790985 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6ac068a60 con 0x7fd6ac0715f0 2024-04-25T00:09:25.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.940+0000 7fd6a1ffb640 1 -- 172.21.15.116:0/2505790985 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd69c00bbc0 con 0x7fd6ac0715f0 2024-04-25T00:09:25.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.940+0000 7fd6a1ffb640 1 --2- 172.21.15.116:0/2505790985 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd6900678c0 0x7fd690069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:25.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.940+0000 7fd6a1ffb640 1 -- 172.21.15.116:0/2505790985 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7440+0+0 (secure 0 0 0) 0x7fd69c0866f0 con 0x7fd6ac0715f0 2024-04-25T00:09:25.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.940+0000 7fd6b09eb640 1 --2- 172.21.15.116:0/2505790985 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd6900678c0 0x7fd690069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:25.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.940+0000 7fd6b09eb640 1 --2- 172.21.15.116:0/2505790985 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd6900678c0 0x7fd690069d80 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac15f650 tx=0x7fd6a8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:25.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:25.944+0000 7fd6a1ffb640 1 -- 172.21.15.116:0/2505790985 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fd69c056c00 con 0x7fd6ac0715f0 2024-04-25T00:09:26.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:26.036+0000 7f50448fa640 1 -- 172.21.15.116:0/2270870392 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5040074980 con 0x7f50400c3140 2024-04-25T00:09:26.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:26.036+0000 7f502effd640 1 -- 172.21.15.116:0/2270870392 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+6501 (secure 0 0 0) 0x7f5038057250 con 0x7f50400c3140 2024-04-25T00:09:26.038 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:09:26.038 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5236,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/1159002662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1159002662},{"type":"v1","addr":"172.21.15.174:6837","nonce":1159002662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:07:33.575034+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5775":{"gid":5775,"name":"c","rank":0,"incarnation":52,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6833/3006671671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3006671671},{"type":"v1","addr":"172.21.15.116:6833","nonce":3006671671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5775]},"id":1},{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:08:03.679028+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5868},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5868":{"gid":5868,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6835/1670061091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1670061091},{"type":"v1","addr":"172.21.15.116:6835","nonce":1670061091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5868,"qdb_cluster":[5868]},"id":2}]} 2024-04-25T00:09:26.038 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 61 2024-04-25T00:09:26.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:26.036+0000 7f50448fa640 1 -- 172.21.15.116:0/2270870392 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f50200678c0 msgr2=0x7f5020069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:26.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:26.036+0000 7f50448fa640 1 --2- 172.21.15.116:0/2270870392 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f50200678c0 0x7f5020069d80 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f5030001670 tx=0x7f5030009040 comp rx=0 tx=0).stop 2024-04-25T00:09:26.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:26.036+0000 7f50448fa640 1 -- 172.21.15.116:0/2270870392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50400c3140 msgr2=0x7f504014eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:26.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:26.036+0000 7f50448fa640 1 --2- 172.21.15.116:0/2270870392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50400c3140 0x7f504014eae0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f5038009f00 tx=0x7f503800c490 comp rx=0 tx=0).stop 2024-04-25T00:09:26.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:26.036+0000 7f50448fa640 1 -- 172.21.15.116:0/2270870392 shutdown_connections 2024-04-25T00:09:26.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:26.036+0000 7f50448fa640 1 --2- 172.21.15.116:0/2270870392 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f50200678c0 0x7f5020069d80 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:26.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:26.036+0000 7f50448fa640 1 --2- 172.21.15.116:0/2270870392 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50400c3140 0x7f504014eae0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:26.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:26.036+0000 7f50448fa640 1 --2- 172.21.15.116:0/2270870392 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50400c2bf0 0x7f50400c0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:26.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:26.036+0000 7f50448fa640 1 --2- 172.21.15.116:0/2270870392 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50400730a0 0x7f5040149560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:26.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:26.036+0000 7f50448fa640 1 -- 172.21.15.116:0/2270870392 >> 172.21.15.116:0/2270870392 conn(0x7f50400c42e0 msgr2=0x7f50400c1c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:09:26.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:26.036+0000 7f50448fa640 1 -- 172.21.15.116:0/2270870392 shutdown_connections 2024-04-25T00:09:26.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:26.036+0000 7f50448fa640 1 -- 172.21.15.116:0/2270870392 wait complete. 2024-04-25T00:09:26.056 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:09:26.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:26.136+0000 7fd6b3450640 1 -- 172.21.15.116:0/2505790985 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd6ac0c7870 con 0x7fd6ac0715f0 2024-04-25T00:09:26.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:26.136+0000 7fd6a1ffb640 1 -- 172.21.15.116:0/2505790985 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+6501 (secure 0 0 0) 0x7fd69c056500 con 0x7fd6ac0715f0 2024-04-25T00:09:26.138 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:09:26.138 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5236,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/1159002662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1159002662},{"type":"v1","addr":"172.21.15.174:6837","nonce":1159002662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:07:33.575034+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5775":{"gid":5775,"name":"c","rank":0,"incarnation":52,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6833/3006671671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3006671671},{"type":"v1","addr":"172.21.15.116:6833","nonce":3006671671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5775]},"id":1},{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:08:03.679028+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5868},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5868":{"gid":5868,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6835/1670061091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1670061091},{"type":"v1","addr":"172.21.15.116:6835","nonce":1670061091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5868,"qdb_cluster":[5868]},"id":2}]} 2024-04-25T00:09:26.138 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 61 2024-04-25T00:09:26.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:26.136+0000 7fd6b3450640 1 -- 172.21.15.116:0/2505790985 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd6900678c0 msgr2=0x7fd690069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:26.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:26.136+0000 7fd6b3450640 1 --2- 172.21.15.116:0/2505790985 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd6900678c0 0x7fd690069d80 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac15f650 tx=0x7fd6a8008040 comp rx=0 tx=0).stop 2024-04-25T00:09:26.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:26.136+0000 7fd6b3450640 1 -- 172.21.15.116:0/2505790985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6ac0715f0 msgr2=0x7fd6ac0abe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:26.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:26.136+0000 7fd6b3450640 1 --2- 172.21.15.116:0/2505790985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6ac0715f0 0x7fd6ac0abe10 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fd69c00d620 tx=0x7fd69c00dae0 comp rx=0 tx=0).stop 2024-04-25T00:09:26.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:26.136+0000 7fd6b3450640 1 -- 172.21.15.116:0/2505790985 shutdown_connections 2024-04-25T00:09:26.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:26.136+0000 7fd6b3450640 1 --2- 172.21.15.116:0/2505790985 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd6900678c0 0x7fd690069d80 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:26.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:26.136+0000 7fd6b3450640 1 --2- 172.21.15.116:0/2505790985 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd6ac152e40 0x7fd6ac148260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:26.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:26.136+0000 7fd6b3450640 1 --2- 172.21.15.116:0/2505790985 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd6ac0c3f70 0x7fd6ac0acbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:26.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:26.136+0000 7fd6b3450640 1 --2- 172.21.15.116:0/2505790985 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6ac0715f0 0x7fd6ac0abe10 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:26.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:26.136+0000 7fd6b3450640 1 -- 172.21.15.116:0/2505790985 >> 172.21.15.116:0/2505790985 conn(0x7fd6ac0c9160 msgr2=0x7fd6ac1520a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:09:26.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:26.136+0000 7fd6b3450640 1 -- 172.21.15.116:0/2505790985 shutdown_connections 2024-04-25T00:09:26.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:26.136+0000 7fd6b3450640 1 -- 172.21.15.116:0/2505790985 wait complete. 2024-04-25T00:09:26.156 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:09:28.057 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:09:28.157 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:09:28.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.180+0000 7f82d651d640 1 Processor -- start 2024-04-25T00:09:28.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.180+0000 7f82d651d640 1 -- start start 2024-04-25T00:09:28.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.180+0000 7f82d651d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f82d01598e0 0x7f82d00c6e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:28.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.180+0000 7f82d651d640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f82d00c73b0 0x7f82d006d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:28.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.180+0000 7f82d651d640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82d0155a50 con 0x7f82d00c5c30 2024-04-25T00:09:28.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.180+0000 7f82d651d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f82d00717e0 con 0x7f82d01598e0 2024-04-25T00:09:28.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.180+0000 7f82d651d640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f82d0159d70 con 0x7f82d00c73b0 2024-04-25T00:09:28.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.180+0000 7f82c7fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f82d01598e0 0x7f82d00c6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:28.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.180+0000 7f82c7fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f82d01598e0 0x7f82d00c6e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44866/0 (socket says 172.21.15.116:44866) 2024-04-25T00:09:28.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.180+0000 7f82c7fff640 1 -- 172.21.15.116:0/1525002881 learned_addr learned my addr 172.21.15.116:0/1525002881 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:09:28.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.180+0000 7f82c7fff640 1 -- 172.21.15.116:0/1525002881 >> v2:172.21.15.174:3300/0 conn(0x7f82d00c73b0 msgr2=0x7f82d006d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:09:28.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.180+0000 7f82d4aba640 1 --2- 172.21.15.116:0/1525002881 >> v2:172.21.15.174:3300/0 conn(0x7f82d00c73b0 0x7f82d006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:28.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.180+0000 7f82c7fff640 1 --2- 172.21.15.116:0/1525002881 >> v2:172.21.15.174:3300/0 conn(0x7f82d00c73b0 0x7f82d006d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:28.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.180+0000 7f82c7fff640 1 -- 172.21.15.116:0/1525002881 >> v1:172.21.15.174:6789/0 conn(0x7f82d00c5c30 legacy=0x7f82d0152d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:09:28.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.180+0000 7f82c7fff640 1 -- 172.21.15.116:0/1525002881 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82d0155cc0 con 0x7f82d01598e0 2024-04-25T00:09:28.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.184+0000 7f82c7fff640 1 --2- 172.21.15.116:0/1525002881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82d01598e0 0x7f82d00c6e70 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f82d00c9fb0 tx=0x7f82b801c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baf3ac383b937eca server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:28.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.184+0000 7f82cf7fe640 1 -- 172.21.15.116:0/1525002881 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82b80b0070 con 0x7f82d01598e0 2024-04-25T00:09:28.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.184+0000 7f82cf7fe640 1 -- 172.21.15.116:0/1525002881 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82b80ab600 con 0x7f82d01598e0 2024-04-25T00:09:28.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.184+0000 7f82cf7fe640 1 -- 172.21.15.116:0/1525002881 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82b80ab9d0 con 0x7f82d01598e0 2024-04-25T00:09:28.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.184+0000 7f82d651d640 1 -- 172.21.15.116:0/1525002881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82d01598e0 msgr2=0x7f82d00c6e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:28.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.184+0000 7f82d651d640 1 --2- 172.21.15.116:0/1525002881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82d01598e0 0x7f82d00c6e70 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f82d00c9fb0 tx=0x7f82b801c450 comp rx=0 tx=0).stop 2024-04-25T00:09:28.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.184+0000 7f82d651d640 1 -- 172.21.15.116:0/1525002881 shutdown_connections 2024-04-25T00:09:28.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.184+0000 7f82d651d640 1 --2- 172.21.15.116:0/1525002881 >> v2:172.21.15.174:3300/0 conn(0x7f82d00c73b0 0x7f82d006d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:28.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.184+0000 7f82d651d640 1 --2- 172.21.15.116:0/1525002881 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82d01598e0 0x7f82d00c6e70 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:28.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.184+0000 7f82d651d640 1 -- 172.21.15.116:0/1525002881 >> 172.21.15.116:0/1525002881 conn(0x7f82d0074a80 msgr2=0x7f82d00c8f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:09:28.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.184+0000 7f82d651d640 1 -- 172.21.15.116:0/1525002881 shutdown_connections 2024-04-25T00:09:28.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.184+0000 7f82d651d640 1 -- 172.21.15.116:0/1525002881 wait complete. 2024-04-25T00:09:28.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.184+0000 7f82d651d640 1 Processor -- start 2024-04-25T00:09:28.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.184+0000 7f82d651d640 1 -- start start 2024-04-25T00:09:28.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.184+0000 7f82d651d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82d00c5c30 0x7f82d014a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:28.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.184+0000 7f82d651d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f82d00c73b0 0x7f82d0156fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:28.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.184+0000 7f82d651d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82d01598e0 0x7f82d01574f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:28.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.184+0000 7f82d651d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f82d00c8c20 con 0x7f82d00c5c30 2024-04-25T00:09:28.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.184+0000 7f82d651d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f82d00c77c0 con 0x7f82d01598e0 2024-04-25T00:09:28.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.184+0000 7f82d651d640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f82d00c4410 con 0x7f82d00c73b0 2024-04-25T00:09:28.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.184+0000 7f82cffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82d00c5c30 0x7f82d014a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:28.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.184+0000 7f82cffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82d00c5c30 0x7f82d014a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44870/0 (socket says 172.21.15.116:44870) 2024-04-25T00:09:28.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.184+0000 7f82cffff640 1 -- 172.21.15.116:0/3056920925 learned_addr learned my addr 172.21.15.116:0/3056920925 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:09:28.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.188+0000 7f82d4aba640 1 --2- 172.21.15.116:0/3056920925 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82d01598e0 0x7f82d01574f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:28.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.188+0000 7f82cffff640 1 -- 172.21.15.116:0/3056920925 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f82d00c73b0 msgr2=0x7f82d0156fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:28.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.188+0000 7f82cffff640 1 --2- 172.21.15.116:0/3056920925 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f82d00c73b0 0x7f82d0156fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:28.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.188+0000 7f82cffff640 1 -- 172.21.15.116:0/3056920925 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82d01598e0 msgr2=0x7f82d01574f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:28.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.188+0000 7f82cffff640 1 --2- 172.21.15.116:0/3056920925 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82d01598e0 0x7f82d01574f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:28.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.188+0000 7f82cffff640 1 -- 172.21.15.116:0/3056920925 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82d00acb10 con 0x7f82d00c5c30 2024-04-25T00:09:28.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.188+0000 7f82cffff640 1 --2- 172.21.15.116:0/3056920925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82d00c5c30 0x7f82d014a070 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f82c000c4c0 tx=0x7f82c0001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:28.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.188+0000 7f82cd7fa640 1 -- 172.21.15.116:0/3056920925 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82c0017070 con 0x7f82d00c5c30 2024-04-25T00:09:28.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.188+0000 7f82cd7fa640 1 -- 172.21.15.116:0/3056920925 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82c0003600 con 0x7f82d00c5c30 2024-04-25T00:09:28.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.188+0000 7f82cd7fa640 1 -- 172.21.15.116:0/3056920925 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82c00039f0 con 0x7f82d00c5c30 2024-04-25T00:09:28.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.188+0000 7f82d651d640 1 -- 172.21.15.116:0/3056920925 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82d00ad390 con 0x7f82d00c5c30 2024-04-25T00:09:28.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.188+0000 7f82d651d640 1 -- 172.21.15.116:0/3056920925 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82d00ad950 con 0x7f82d00c5c30 2024-04-25T00:09:28.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.188+0000 7f82d651d640 1 -- 172.21.15.116:0/3056920925 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82d0068a60 con 0x7f82d00c5c30 2024-04-25T00:09:28.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.188+0000 7f82cd7fa640 1 -- 172.21.15.116:0/3056920925 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f82c0003b90 con 0x7f82d00c5c30 2024-04-25T00:09:28.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.188+0000 7f82cd7fa640 1 --2- 172.21.15.116:0/3056920925 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f82b00678f0 0x7f82b0069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:28.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.188+0000 7f82cd7fa640 1 -- 172.21.15.116:0/3056920925 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7440+0+0 (secure 0 0 0) 0x7f82c0085510 con 0x7f82d00c5c30 2024-04-25T00:09:28.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.188+0000 7f82c7fff640 1 --2- 172.21.15.116:0/3056920925 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f82b00678f0 0x7f82b0069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:28.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.188+0000 7f82c7fff640 1 --2- 172.21.15.116:0/3056920925 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f82b00678f0 0x7f82b0069db0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f82d0075460 tx=0x7f82b8004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:28.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.196+0000 7f82cd7fa640 1 -- 172.21.15.116:0/3056920925 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f82c0055990 con 0x7f82d00c5c30 2024-04-25T00:09:28.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.272+0000 7f937d6ab640 1 Processor -- start 2024-04-25T00:09:28.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.272+0000 7f937d6ab640 1 -- start start 2024-04-25T00:09:28.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.276+0000 7f937d6ab640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f9378150d10 0x7f93780c5010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:28.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.276+0000 7f937d6ab640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f93780c5550 0x7f9378153a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:28.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.276+0000 7f937d6ab640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93780c4950 con 0x7f9378072ca0 2024-04-25T00:09:28.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.276+0000 7f937d6ab640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f93781546d0 con 0x7f9378150d10 2024-04-25T00:09:28.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.276+0000 7f937d6ab640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f93781510f0 con 0x7f93780c5550 2024-04-25T00:09:28.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.276+0000 7f93767fc640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f93780c5550 0x7f9378153a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:28.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.276+0000 7f9376ffd640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f9378150d10 0x7f93780c5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:28.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.276+0000 7f93767fc640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f93780c5550 0x7f9378153a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:40780/0 (socket says 172.21.15.116:40780) 2024-04-25T00:09:28.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.276+0000 7f93767fc640 1 -- 172.21.15.116:0/1383897725 learned_addr learned my addr 172.21.15.116:0/1383897725 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:09:28.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.276+0000 7f93767fc640 1 -- 172.21.15.116:0/1383897725 >> v2:172.21.15.174:3300/0 conn(0x7f9378150d10 msgr2=0x7f93780c5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:09:28.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.276+0000 7f93767fc640 1 --2- 172.21.15.116:0/1383897725 >> v2:172.21.15.174:3300/0 conn(0x7f9378150d10 0x7f93780c5010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:28.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.276+0000 7f93767fc640 1 -- 172.21.15.116:0/1383897725 >> v1:172.21.15.174:6789/0 conn(0x7f9378072ca0 legacy=0x7f9378153fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:28.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.276+0000 7f93767fc640 1 -- 172.21.15.116:0/1383897725 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9378157a20 con 0x7f93780c5550 2024-04-25T00:09:28.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.280+0000 7f93767fc640 1 --2- 172.21.15.116:0/1383897725 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f93780c5550 0x7f9378153a80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9378157bf0 tx=0x7f936801c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3cb55a7859b891e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:28.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.280+0000 7f9375ffb640 1 -- 172.21.15.116:0/1383897725 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93680ab620 con 0x7f93780c5550 2024-04-25T00:09:28.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.280+0000 7f9375ffb640 1 -- 172.21.15.116:0/1383897725 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93680b0070 con 0x7f93780c5550 2024-04-25T00:09:28.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.280+0000 7f9375ffb640 1 -- 172.21.15.116:0/1383897725 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93680ab9e0 con 0x7f93780c5550 2024-04-25T00:09:28.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.280+0000 7f937d6ab640 1 -- 172.21.15.116:0/1383897725 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f93780c5550 msgr2=0x7f9378153a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:28.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.280+0000 7f937d6ab640 1 --2- 172.21.15.116:0/1383897725 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f93780c5550 0x7f9378153a80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9378157bf0 tx=0x7f936801c420 comp rx=0 tx=0).stop 2024-04-25T00:09:28.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.280+0000 7f937d6ab640 1 -- 172.21.15.116:0/1383897725 shutdown_connections 2024-04-25T00:09:28.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.280+0000 7f937d6ab640 1 --2- 172.21.15.116:0/1383897725 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f93780c5550 0x7f9378153a80 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:28.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.280+0000 7f937d6ab640 1 --2- 172.21.15.116:0/1383897725 >> v2:172.21.15.174:3300/0 conn(0x7f9378150d10 0x7f93780c5010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:28.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.280+0000 7f937d6ab640 1 -- 172.21.15.116:0/1383897725 >> 172.21.15.116:0/1383897725 conn(0x7f937806c720 msgr2=0x7f937806cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:09:28.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.280+0000 7f937d6ab640 1 -- 172.21.15.116:0/1383897725 shutdown_connections 2024-04-25T00:09:28.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.280+0000 7f937d6ab640 1 -- 172.21.15.116:0/1383897725 wait complete. 2024-04-25T00:09:28.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.280+0000 7f937d6ab640 1 Processor -- start 2024-04-25T00:09:28.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.280+0000 7f937d6ab640 1 -- start start 2024-04-25T00:09:28.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.280+0000 7f937d6ab640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9378072ca0 0x7f9378152a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:28.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.280+0000 7f937d6ab640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93780c5550 0x7f9378154fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:28.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.280+0000 7f937d6ab640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9378150d10 0x7f93781eeaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:28.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.280+0000 7f937d6ab640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f93780c2640 con 0x7f9378072ca0 2024-04-25T00:09:28.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.280+0000 7f937d6ab640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f93781576a0 con 0x7f93780c5550 2024-04-25T00:09:28.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.280+0000 7f937d6ab640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f93780c27c0 con 0x7f9378150d10 2024-04-25T00:09:28.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.280+0000 7f9376ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9378072ca0 0x7f9378152a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:28.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.280+0000 7f9376ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9378072ca0 0x7f9378152a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44876/0 (socket says 172.21.15.116:44876) 2024-04-25T00:09:28.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.280+0000 7f9376ffd640 1 -- 172.21.15.116:0/3972772942 learned_addr learned my addr 172.21.15.116:0/3972772942 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:09:28.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.280+0000 7f93777fe640 1 --2- 172.21.15.116:0/3972772942 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9378150d10 0x7f93781eeaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:28.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.280+0000 7f93767fc640 1 --2- 172.21.15.116:0/3972772942 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93780c5550 0x7f9378154fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:28.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.280+0000 7f9376ffd640 1 -- 172.21.15.116:0/3972772942 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9378150d10 msgr2=0x7f93781eeaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:28.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.280+0000 7f9376ffd640 1 --2- 172.21.15.116:0/3972772942 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9378150d10 0x7f93781eeaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:28.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.280+0000 7f9376ffd640 1 -- 172.21.15.116:0/3972772942 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93780c5550 msgr2=0x7f9378154fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:28.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.280+0000 7f9376ffd640 1 --2- 172.21.15.116:0/3972772942 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93780c5550 0x7f9378154fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:28.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.280+0000 7f9376ffd640 1 -- 172.21.15.116:0/3972772942 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9378153080 con 0x7f9378072ca0 2024-04-25T00:09:28.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.280+0000 7f93767fc640 1 --2- 172.21.15.116:0/3972772942 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93780c5550 0x7f9378154fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:09:28.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.280+0000 7f9376ffd640 1 --2- 172.21.15.116:0/3972772942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9378072ca0 0x7f9378152a30 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f936c00e8a0 tx=0x7f936c00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:28.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.280+0000 7f935f7fe640 1 -- 172.21.15.116:0/3972772942 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f936c017070 con 0x7f9378072ca0 2024-04-25T00:09:28.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.280+0000 7f935f7fe640 1 -- 172.21.15.116:0/3972772942 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f936c00c940 con 0x7f9378072ca0 2024-04-25T00:09:28.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.280+0000 7f935f7fe640 1 -- 172.21.15.116:0/3972772942 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f936c00ccf0 con 0x7f9378072ca0 2024-04-25T00:09:28.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.280+0000 7f937d6ab640 1 -- 172.21.15.116:0/3972772942 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9378153360 con 0x7f9378072ca0 2024-04-25T00:09:28.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.280+0000 7f937d6ab640 1 -- 172.21.15.116:0/3972772942 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9378153640 con 0x7f9378072ca0 2024-04-25T00:09:28.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.284+0000 7f937d6ab640 1 -- 172.21.15.116:0/3972772942 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9378068a60 con 0x7f9378072ca0 2024-04-25T00:09:28.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.284+0000 7f935f7fe640 1 -- 172.21.15.116:0/3972772942 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f936c01a070 con 0x7f9378072ca0 2024-04-25T00:09:28.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.284+0000 7f935f7fe640 1 --2- 172.21.15.116:0/3972772942 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f93640678c0 0x7f9364069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:28.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.284+0000 7f935f7fe640 1 -- 172.21.15.116:0/3972772942 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 7440+0+0 (secure 0 0 0) 0x7f936c086550 con 0x7f9378072ca0 2024-04-25T00:09:28.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.284+0000 7f93767fc640 1 --2- 172.21.15.116:0/3972772942 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f93640678c0 0x7f9364069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:28.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.284+0000 7f93767fc640 1 --2- 172.21.15.116:0/3972772942 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f93640678c0 0x7f9364069d80 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f93680af000 tx=0x7f9368008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:28.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.288+0000 7f935f7fe640 1 -- 172.21.15.116:0/3972772942 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f936c056a60 con 0x7f9378072ca0 2024-04-25T00:09:28.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.384+0000 7f82d651d640 1 -- 172.21.15.116:0/3056920925 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f82d006cf70 con 0x7f82d00c5c30 2024-04-25T00:09:28.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.384+0000 7f82cd7fa640 1 -- 172.21.15.116:0/3056920925 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+6501 (secure 0 0 0) 0x7f82c0055290 con 0x7f82d00c5c30 2024-04-25T00:09:28.386 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:09:28.386 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5236,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/1159002662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1159002662},{"type":"v1","addr":"172.21.15.174:6837","nonce":1159002662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:07:33.575034+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5775":{"gid":5775,"name":"c","rank":0,"incarnation":52,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6833/3006671671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3006671671},{"type":"v1","addr":"172.21.15.116:6833","nonce":3006671671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5775]},"id":1},{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:08:03.679028+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5868},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5868":{"gid":5868,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6835/1670061091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1670061091},{"type":"v1","addr":"172.21.15.116:6835","nonce":1670061091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5868,"qdb_cluster":[5868]},"id":2}]} 2024-04-25T00:09:28.387 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 61 2024-04-25T00:09:28.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.384+0000 7f82d651d640 1 -- 172.21.15.116:0/3056920925 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f82b00678f0 msgr2=0x7f82b0069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:28.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.384+0000 7f82d651d640 1 --2- 172.21.15.116:0/3056920925 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f82b00678f0 0x7f82b0069db0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f82d0075460 tx=0x7f82b8004740 comp rx=0 tx=0).stop 2024-04-25T00:09:28.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.384+0000 7f82d651d640 1 -- 172.21.15.116:0/3056920925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82d00c5c30 msgr2=0x7f82d014a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:28.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.384+0000 7f82d651d640 1 --2- 172.21.15.116:0/3056920925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82d00c5c30 0x7f82d014a070 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f82c000c4c0 tx=0x7f82c0001e80 comp rx=0 tx=0).stop 2024-04-25T00:09:28.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.384+0000 7f82d651d640 1 -- 172.21.15.116:0/3056920925 shutdown_connections 2024-04-25T00:09:28.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.384+0000 7f82d651d640 1 --2- 172.21.15.116:0/3056920925 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f82b00678f0 0x7f82b0069db0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:28.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.384+0000 7f82d651d640 1 --2- 172.21.15.116:0/3056920925 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82d01598e0 0x7f82d01574f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:28.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.384+0000 7f82d651d640 1 --2- 172.21.15.116:0/3056920925 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f82d00c73b0 0x7f82d0156fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:28.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.384+0000 7f82d651d640 1 --2- 172.21.15.116:0/3056920925 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82d00c5c30 0x7f82d014a070 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:28.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.384+0000 7f82d651d640 1 -- 172.21.15.116:0/3056920925 >> 172.21.15.116:0/3056920925 conn(0x7f82d0074a80 msgr2=0x7f82d0070530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:09:28.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.388+0000 7f82d651d640 1 -- 172.21.15.116:0/3056920925 shutdown_connections 2024-04-25T00:09:28.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.388+0000 7f82d651d640 1 -- 172.21.15.116:0/3056920925 wait complete. 2024-04-25T00:09:28.404 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:09:28.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.476+0000 7f937d6ab640 1 -- 172.21.15.116:0/3972772942 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9378072a80 con 0x7f9378072ca0 2024-04-25T00:09:28.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.480+0000 7f935f7fe640 1 -- 172.21.15.116:0/3972772942 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) ==== 94+0+6501 (secure 0 0 0) 0x7f936c056360 con 0x7f9378072ca0 2024-04-25T00:09:28.481 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:09:28.481 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":5236,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/1159002662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1159002662},{"type":"v1","addr":"172.21.15.174:6837","nonce":1159002662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:07:33.575034+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":37,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5775},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5775":{"gid":5775,"name":"c","rank":0,"incarnation":52,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6833/3006671671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3006671671},{"type":"v1","addr":"172.21.15.116:6833","nonce":3006671671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5775,"qdb_cluster":[5775]},"id":1},{"mdsmap":{"epoch":60,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:08:03.679028+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":38,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5868},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5868":{"gid":5868,"name":"a","rank":0,"incarnation":57,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6835/1670061091","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1670061091},{"type":"v1","addr":"172.21.15.116:6835","nonce":1670061091}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5868,"qdb_cluster":[5868]},"id":2}]} 2024-04-25T00:09:28.481 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 61 2024-04-25T00:09:28.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.480+0000 7f937d6ab640 1 -- 172.21.15.116:0/3972772942 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f93640678c0 msgr2=0x7f9364069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:28.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.480+0000 7f937d6ab640 1 --2- 172.21.15.116:0/3972772942 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f93640678c0 0x7f9364069d80 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f93680af000 tx=0x7f9368008040 comp rx=0 tx=0).stop 2024-04-25T00:09:28.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.480+0000 7f937d6ab640 1 -- 172.21.15.116:0/3972772942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9378072ca0 msgr2=0x7f9378152a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:28.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.480+0000 7f937d6ab640 1 --2- 172.21.15.116:0/3972772942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9378072ca0 0x7f9378152a30 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f936c00e8a0 tx=0x7f936c00eca0 comp rx=0 tx=0).stop 2024-04-25T00:09:28.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.480+0000 7f937d6ab640 1 -- 172.21.15.116:0/3972772942 shutdown_connections 2024-04-25T00:09:28.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.480+0000 7f937d6ab640 1 --2- 172.21.15.116:0/3972772942 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f93640678c0 0x7f9364069d80 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:28.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.480+0000 7f937d6ab640 1 --2- 172.21.15.116:0/3972772942 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9378150d10 0x7f93781eeaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:28.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.480+0000 7f937d6ab640 1 --2- 172.21.15.116:0/3972772942 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93780c5550 0x7f9378154fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:28.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.480+0000 7f937d6ab640 1 --2- 172.21.15.116:0/3972772942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9378072ca0 0x7f9378152a30 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:28.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.480+0000 7f937d6ab640 1 -- 172.21.15.116:0/3972772942 >> 172.21.15.116:0/3972772942 conn(0x7f937806c720 msgr2=0x7f93780aa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:09:28.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.480+0000 7f937d6ab640 1 -- 172.21.15.116:0/3972772942 shutdown_connections 2024-04-25T00:09:28.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:28.480+0000 7f937d6ab640 1 -- 172.21.15.116:0/3972772942 wait complete. 2024-04-25T00:09:28.498 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:09:30.405 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:09:30.498 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:09:30.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.548+0000 7f3ab20c5640 1 Processor -- start 2024-04-25T00:09:30.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.548+0000 7f3ab20c5640 1 -- start start 2024-04-25T00:09:30.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.548+0000 7f3ab20c5640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3aac150d20 0x7f3aac153620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:30.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.548+0000 7f3ab20c5640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f3aac071050 0x7f3aac0714e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:30.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.548+0000 7f3ab20c5640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3aac0c36a0 con 0x7f3aac0c3240 2024-04-25T00:09:30.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.548+0000 7f3ab20c5640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f3aac06fba0 con 0x7f3aac071050 2024-04-25T00:09:30.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.548+0000 7f3ab20c5640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f3aac06f8c0 con 0x7f3aac150d20 2024-04-25T00:09:30.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.548+0000 7f3aab7fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3aac150d20 0x7f3aac153620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:30.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.548+0000 7f3aabfff640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f3aac071050 0x7f3aac0714e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:30.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.548+0000 7f3aabfff640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f3aac071050 0x7f3aac0714e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:56106/0 (socket says 172.21.15.116:56106) 2024-04-25T00:09:30.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.548+0000 7f3aabfff640 1 -- 172.21.15.116:0/1986211925 learned_addr learned my addr 172.21.15.116:0/1986211925 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:09:30.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.548+0000 7f3aaaffd640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f3aac0c3240 0x7f3aac0c3e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:46938/0 (socket says 172.21.15.116:46938) 2024-04-25T00:09:30.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.548+0000 7f3aab7fe640 1 -- 172.21.15.116:0/1986211925 >> v2:172.21.15.174:3300/0 conn(0x7f3aac071050 msgr2=0x7f3aac0714e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:09:30.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.548+0000 7f3aab7fe640 1 --2- 172.21.15.116:0/1986211925 >> v2:172.21.15.174:3300/0 conn(0x7f3aac071050 0x7f3aac0714e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:30.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.548+0000 7f3aab7fe640 1 -- 172.21.15.116:0/1986211925 >> v1:172.21.15.174:6789/0 conn(0x7f3aac0c3240 legacy=0x7f3aac0c3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:30.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.548+0000 7f3aab7fe640 1 -- 172.21.15.116:0/1986211925 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3aac0c5420 con 0x7f3aac150d20 2024-04-25T00:09:30.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.548+0000 7f3aabfff640 1 --2- 172.21.15.116:0/1986211925 >> v2:172.21.15.174:3300/0 conn(0x7f3aac071050 0x7f3aac0714e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:09:30.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.552+0000 7f3aab7fe640 1 --2- 172.21.15.116:0/1986211925 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3aac150d20 0x7f3aac153620 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c0097c0 tx=0x7f3a9c01c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7837548b90439661 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:30.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.552+0000 7f3aaa7fc640 1 -- 172.21.15.116:0/1986211925 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a9c0b1070 con 0x7f3aac150d20 2024-04-25T00:09:30.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.552+0000 7f3aaa7fc640 1 -- 172.21.15.116:0/1986211925 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a9c0ac410 con 0x7f3aac150d20 2024-04-25T00:09:30.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.552+0000 7f3aaa7fc640 1 -- 172.21.15.116:0/1986211925 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a9c0ac7e0 con 0x7f3aac150d20 2024-04-25T00:09:30.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.552+0000 7f3ab20c5640 1 -- 172.21.15.116:0/1986211925 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3aac150d20 msgr2=0x7f3aac153620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:30.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.552+0000 7f3ab20c5640 1 --2- 172.21.15.116:0/1986211925 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3aac150d20 0x7f3aac153620 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c0097c0 tx=0x7f3a9c01c4a0 comp rx=0 tx=0).stop 2024-04-25T00:09:30.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.552+0000 7f3ab20c5640 1 -- 172.21.15.116:0/1986211925 shutdown_connections 2024-04-25T00:09:30.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.552+0000 7f3ab20c5640 1 --2- 172.21.15.116:0/1986211925 >> v2:172.21.15.174:3300/0 conn(0x7f3aac071050 0x7f3aac0714e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:30.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.552+0000 7f3ab20c5640 1 --2- 172.21.15.116:0/1986211925 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3aac150d20 0x7f3aac153620 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:30.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.552+0000 7f3ab20c5640 1 -- 172.21.15.116:0/1986211925 >> 172.21.15.116:0/1986211925 conn(0x7f3aac0c9160 msgr2=0x7f3aac0c9570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:09:30.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.552+0000 7f3ab20c5640 1 -- 172.21.15.116:0/1986211925 shutdown_connections 2024-04-25T00:09:30.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.552+0000 7f3ab20c5640 1 -- 172.21.15.116:0/1986211925 wait complete. 2024-04-25T00:09:30.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.552+0000 7f3ab20c5640 1 Processor -- start 2024-04-25T00:09:30.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.552+0000 7f3ab20c5640 1 -- start start 2024-04-25T00:09:30.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.556+0000 7f3ab20c5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3aac071050 0x7f3aac13f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:30.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.556+0000 7f3ab20c5640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3aac0c3240 0x7f3aac13f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:30.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.556+0000 7f3ab20c5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3aac150d20 0x7f3aac1ee1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:30.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.556+0000 7f3ab20c5640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3aac06f640 con 0x7f3aac071050 2024-04-25T00:09:30.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.556+0000 7f3ab20c5640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3aac06f4c0 con 0x7f3aac150d20 2024-04-25T00:09:30.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.556+0000 7f3ab20c5640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f3aac151100 con 0x7f3aac0c3240 2024-04-25T00:09:30.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.556+0000 7f3aab7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3aac071050 0x7f3aac13f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:30.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.556+0000 7f3aab7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3aac071050 0x7f3aac13f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44884/0 (socket says 172.21.15.116:44884) 2024-04-25T00:09:30.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.556+0000 7f3aab7fe640 1 -- 172.21.15.116:0/1586164327 learned_addr learned my addr 172.21.15.116:0/1586164327 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:09:30.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.556+0000 7f3aaaffd640 1 --2- 172.21.15.116:0/1586164327 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3aac0c3240 0x7f3aac13f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:30.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.556+0000 7f3aabfff640 1 --2- 172.21.15.116:0/1586164327 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3aac150d20 0x7f3aac1ee1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:30.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.556+0000 7f3aab7fe640 1 -- 172.21.15.116:0/1586164327 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3aac0c3240 msgr2=0x7f3aac13f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:30.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.556+0000 7f3aab7fe640 1 --2- 172.21.15.116:0/1586164327 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3aac0c3240 0x7f3aac13f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:30.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.556+0000 7f3aab7fe640 1 -- 172.21.15.116:0/1586164327 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3aac150d20 msgr2=0x7f3aac1ee1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:30.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.556+0000 7f3aab7fe640 1 --2- 172.21.15.116:0/1586164327 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3aac150d20 0x7f3aac1ee1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:30.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.556+0000 7f3aab7fe640 1 -- 172.21.15.116:0/1586164327 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3aac153640 con 0x7f3aac071050 2024-04-25T00:09:30.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.556+0000 7f3aab7fe640 1 --2- 172.21.15.116:0/1586164327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3aac071050 0x7f3aac13f1a0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c0097c0 tx=0x7f3a9c019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:30.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.556+0000 7f3a93fff640 1 -- 172.21.15.116:0/1586164327 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a9c0b9020 con 0x7f3aac071050 2024-04-25T00:09:30.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.556+0000 7f3a93fff640 1 -- 172.21.15.116:0/1586164327 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a9c0acd70 con 0x7f3aac071050 2024-04-25T00:09:30.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.556+0000 7f3a93fff640 1 -- 172.21.15.116:0/1586164327 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a9c0b1040 con 0x7f3aac071050 2024-04-25T00:09:30.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.556+0000 7f3ab20c5640 1 -- 172.21.15.116:0/1586164327 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3aac153920 con 0x7f3aac071050 2024-04-25T00:09:30.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.556+0000 7f3ab20c5640 1 -- 172.21.15.116:0/1586164327 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3aac14bb90 con 0x7f3aac071050 2024-04-25T00:09:30.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.556+0000 7f3a91ffb640 1 -- 172.21.15.116:0/1586164327 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a840052a0 con 0x7f3aac071050 2024-04-25T00:09:30.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.556+0000 7f3a93fff640 1 -- 172.21.15.116:0/1586164327 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3a9c0bc020 con 0x7f3aac071050 2024-04-25T00:09:30.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.556+0000 7f3a93fff640 1 --2- 172.21.15.116:0/1586164327 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3a980678c0 0x7f3a98069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:30.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.556+0000 7f3a93fff640 1 -- 172.21.15.116:0/1586164327 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7612+0+0 (secure 0 0 0) 0x7f3a9c11ff80 con 0x7f3aac071050 2024-04-25T00:09:30.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.556+0000 7f3aaaffd640 1 --2- 172.21.15.116:0/1586164327 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3a980678c0 0x7f3a98069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:30.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.560+0000 7f3aaaffd640 1 --2- 172.21.15.116:0/1586164327 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3a980678c0 0x7f3a98069d80 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f3aac0c8dc0 tx=0x7f3aa0007e90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:30.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.564+0000 7f3a93fff640 1 -- 172.21.15.116:0/1586164327 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f3a9c0c3030 con 0x7f3aac071050 2024-04-25T00:09:30.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.620+0000 7f358aa92640 1 Processor -- start 2024-04-25T00:09:30.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.620+0000 7f358aa92640 1 -- start start 2024-04-25T00:09:30.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.620+0000 7f358aa92640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3584152f60 0x7f35840c6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:30.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.620+0000 7f358aa92640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f35840c3c30 0x7f358406fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:30.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.620+0000 7f358aa92640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f358406e8f0 con 0x7f3584158ea0 2024-04-25T00:09:30.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.620+0000 7f358aa92640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f358406ebd0 con 0x7f35840c3c30 2024-04-25T00:09:30.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.620+0000 7f358aa92640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f358406f330 con 0x7f3584152f60 2024-04-25T00:09:30.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.620+0000 7f3583fff640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f35840c3c30 0x7f358406fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:30.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.620+0000 7f358882e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3584152f60 0x7f35840c6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:30.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.620+0000 7f3583fff640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f35840c3c30 0x7f358406fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:56126/0 (socket says 172.21.15.116:56126) 2024-04-25T00:09:30.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.620+0000 7f358882e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3584152f60 0x7f35840c6080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:40806/0 (socket says 172.21.15.116:40806) 2024-04-25T00:09:30.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.620+0000 7f3583fff640 1 -- 172.21.15.116:0/3119583520 learned_addr learned my addr 172.21.15.116:0/3119583520 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:09:30.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.620+0000 7f358882e640 1 -- 172.21.15.116:0/3119583520 >> v2:172.21.15.174:3300/0 conn(0x7f35840c3c30 msgr2=0x7f358406fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:09:30.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.620+0000 7f358882e640 1 --2- 172.21.15.116:0/3119583520 >> v2:172.21.15.174:3300/0 conn(0x7f35840c3c30 0x7f358406fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:30.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.624+0000 7f35837fe640 1 -- 172.21.15.116:0/3119583520 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1149464876 0 0) 0x7f358406e8f0 con 0x7f3584158ea0 2024-04-25T00:09:30.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.624+0000 7f358882e640 1 -- 172.21.15.116:0/3119583520 >> v1:172.21.15.174:6789/0 conn(0x7f3584158ea0 legacy=0x7f358406c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:30.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.624+0000 7f358882e640 1 -- 172.21.15.116:0/3119583520 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f358406f610 con 0x7f3584152f60 2024-04-25T00:09:30.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.624+0000 7f358882e640 1 --2- 172.21.15.116:0/3119583520 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3584152f60 0x7f35840c6080 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f356c0097c0 tx=0x7f356c01c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=25bee9d29ad6f85d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:30.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.624+0000 7f35837fe640 1 -- 172.21.15.116:0/3119583520 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f356c0b0070 con 0x7f3584152f60 2024-04-25T00:09:30.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.624+0000 7f35837fe640 1 -- 172.21.15.116:0/3119583520 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f356c0ab600 con 0x7f3584152f60 2024-04-25T00:09:30.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.624+0000 7f35837fe640 1 -- 172.21.15.116:0/3119583520 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f356c0ab9d0 con 0x7f3584152f60 2024-04-25T00:09:30.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.624+0000 7f358aa92640 1 -- 172.21.15.116:0/3119583520 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3584152f60 msgr2=0x7f35840c6080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:30.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.624+0000 7f358aa92640 1 --2- 172.21.15.116:0/3119583520 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3584152f60 0x7f35840c6080 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f356c0097c0 tx=0x7f356c01c410 comp rx=0 tx=0).stop 2024-04-25T00:09:30.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.624+0000 7f358aa92640 1 -- 172.21.15.116:0/3119583520 shutdown_connections 2024-04-25T00:09:30.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.624+0000 7f358aa92640 1 --2- 172.21.15.116:0/3119583520 >> v2:172.21.15.174:3300/0 conn(0x7f35840c3c30 0x7f358406fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:30.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.624+0000 7f358aa92640 1 --2- 172.21.15.116:0/3119583520 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3584152f60 0x7f35840c6080 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:30.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.624+0000 7f358aa92640 1 -- 172.21.15.116:0/3119583520 >> 172.21.15.116:0/3119583520 conn(0x7f35840c4270 msgr2=0x7f35840c4680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:09:30.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.624+0000 7f358aa92640 1 -- 172.21.15.116:0/3119583520 shutdown_connections 2024-04-25T00:09:30.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.624+0000 7f358aa92640 1 -- 172.21.15.116:0/3119583520 wait complete. 2024-04-25T00:09:30.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.628+0000 7f358aa92640 1 Processor -- start 2024-04-25T00:09:30.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.628+0000 7f358aa92640 1 -- start start 2024-04-25T00:09:30.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.628+0000 7f358aa92640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f35840c3c30 0x7f35841564c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:30.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.628+0000 7f358aa92640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3584152f60 0x7f358414c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:30.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.628+0000 7f358aa92640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3584158ea0 0x7f358414ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:30.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.628+0000 7f358aa92640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3584162b70 con 0x7f3584152f60 2024-04-25T00:09:30.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.628+0000 7f358aa92640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f35841629f0 con 0x7f3584158ea0 2024-04-25T00:09:30.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.628+0000 7f358aa92640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f3584162cf0 con 0x7f35840c3c30 2024-04-25T00:09:30.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.628+0000 7f3583fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3584152f60 0x7f358414c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:30.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.628+0000 7f3583fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3584152f60 0x7f358414c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44886/0 (socket says 172.21.15.116:44886) 2024-04-25T00:09:30.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.628+0000 7f3583fff640 1 -- 172.21.15.116:0/355282647 learned_addr learned my addr 172.21.15.116:0/355282647 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:09:30.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.628+0000 7f358882e640 1 --2- 172.21.15.116:0/355282647 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f35840c3c30 0x7f35841564c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:30.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.628+0000 7f358902f640 1 --2- 172.21.15.116:0/355282647 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3584158ea0 0x7f358414ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:30.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.628+0000 7f3583fff640 1 -- 172.21.15.116:0/355282647 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f35840c3c30 msgr2=0x7f35841564c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:30.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.628+0000 7f3583fff640 1 --2- 172.21.15.116:0/355282647 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f35840c3c30 0x7f35841564c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:30.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.628+0000 7f3583fff640 1 -- 172.21.15.116:0/355282647 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3584158ea0 msgr2=0x7f358414ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:30.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.628+0000 7f3583fff640 1 --2- 172.21.15.116:0/355282647 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3584158ea0 0x7f358414ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:30.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.628+0000 7f3583fff640 1 -- 172.21.15.116:0/355282647 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35840724f0 con 0x7f3584152f60 2024-04-25T00:09:30.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.628+0000 7f3583fff640 1 --2- 172.21.15.116:0/355282647 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3584152f60 0x7f358414c970 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f357400e8a0 tx=0x7f357400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:30.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.628+0000 7f35817fa640 1 -- 172.21.15.116:0/355282647 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3574017070 con 0x7f3584152f60 2024-04-25T00:09:30.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.628+0000 7f358aa92640 1 -- 172.21.15.116:0/355282647 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35840727d0 con 0x7f3584152f60 2024-04-25T00:09:30.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.628+0000 7f35817fa640 1 -- 172.21.15.116:0/355282647 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f357400c940 con 0x7f3584152f60 2024-04-25T00:09:30.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.628+0000 7f35817fa640 1 -- 172.21.15.116:0/355282647 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f357400cd50 con 0x7f3584152f60 2024-04-25T00:09:30.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.628+0000 7f358aa92640 1 -- 172.21.15.116:0/355282647 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3584071580 con 0x7f3584152f60 2024-04-25T00:09:30.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.628+0000 7f358aa92640 1 -- 172.21.15.116:0/355282647 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3584068a60 con 0x7f3584152f60 2024-04-25T00:09:30.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.628+0000 7f35817fa640 1 -- 172.21.15.116:0/355282647 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f357401a070 con 0x7f3584152f60 2024-04-25T00:09:30.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.628+0000 7f35817fa640 1 --2- 172.21.15.116:0/355282647 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f35640678c0 0x7f3564069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:30.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.628+0000 7f35817fa640 1 -- 172.21.15.116:0/355282647 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7612+0+0 (secure 0 0 0) 0x7f3574086680 con 0x7f3584152f60 2024-04-25T00:09:30.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.632+0000 7f358882e640 1 --2- 172.21.15.116:0/355282647 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f35640678c0 0x7f3564069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:30.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.632+0000 7f358882e640 1 --2- 172.21.15.116:0/355282647 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f35640678c0 0x7f3564069d80 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f356c0ad000 tx=0x7f356c0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:30.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.636+0000 7f35817fa640 1 -- 172.21.15.116:0/355282647 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f3574056380 con 0x7f3584152f60 2024-04-25T00:09:30.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.768+0000 7f3a91ffb640 1 -- 172.21.15.116:0/1586164327 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3a84005590 con 0x7f3aac071050 2024-04-25T00:09:30.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.768+0000 7f3a93fff640 1 -- 172.21.15.116:0/1586164327 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) ==== 94+0+4832 (secure 0 0 0) 0x7f3a9c016090 con 0x7f3aac071050 2024-04-25T00:09:30.772 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:09:30.772 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:09:30.490930+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5236":{"gid":5236,"name":"b","rank":0,"incarnation":62,"state":"up:replay","state_seq":1,"addr":"172.21.15.174:6837/1159002662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1159002662},{"type":"v1","addr":"172.21.15.174:6837","nonce":1159002662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:09:30.490931+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"d","rank":0,"incarnation":62,"state":"up:replay","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:09:30.772 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 62 2024-04-25T00:09:30.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.772+0000 7f3a91ffb640 1 -- 172.21.15.116:0/1586164327 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3a980678c0 msgr2=0x7f3a98069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:30.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.772+0000 7f3a91ffb640 1 --2- 172.21.15.116:0/1586164327 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3a980678c0 0x7f3a98069d80 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f3aac0c8dc0 tx=0x7f3aa0007e90 comp rx=0 tx=0).stop 2024-04-25T00:09:30.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.772+0000 7f3a91ffb640 1 -- 172.21.15.116:0/1586164327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3aac071050 msgr2=0x7f3aac13f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:30.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.772+0000 7f3a91ffb640 1 --2- 172.21.15.116:0/1586164327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3aac071050 0x7f3aac13f1a0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c0097c0 tx=0x7f3a9c019240 comp rx=0 tx=0).stop 2024-04-25T00:09:30.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.772+0000 7f3a91ffb640 1 -- 172.21.15.116:0/1586164327 shutdown_connections 2024-04-25T00:09:30.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.772+0000 7f3a91ffb640 1 --2- 172.21.15.116:0/1586164327 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3a980678c0 0x7f3a98069d80 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:30.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.772+0000 7f3a91ffb640 1 --2- 172.21.15.116:0/1586164327 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3aac150d20 0x7f3aac1ee1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:30.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.772+0000 7f3a91ffb640 1 --2- 172.21.15.116:0/1586164327 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3aac0c3240 0x7f3aac13f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:30.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.772+0000 7f3a91ffb640 1 --2- 172.21.15.116:0/1586164327 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3aac071050 0x7f3aac13f1a0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:30.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.772+0000 7f3a91ffb640 1 -- 172.21.15.116:0/1586164327 >> 172.21.15.116:0/1586164327 conn(0x7f3aac0c9160 msgr2=0x7f3aac145950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:09:30.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.772+0000 7f3a91ffb640 1 -- 172.21.15.116:0/1586164327 shutdown_connections 2024-04-25T00:09:30.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.772+0000 7f3a91ffb640 1 -- 172.21.15.116:0/1586164327 wait complete. 2024-04-25T00:09:30.791 INFO:tasks.mds_thrash.fs.[a]:mds.c down, removed from mdsmap 2024-04-25T00:09:30.791 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-04-25T00:09:30.791 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:09:30.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.832+0000 7f358aa92640 1 -- 172.21.15.116:0/355282647 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f35840c51a0 con 0x7f3584152f60 2024-04-25T00:09:30.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.832+0000 7f35817fa640 1 -- 172.21.15.116:0/355282647 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) ==== 94+0+4832 (secure 0 0 0) 0x7f357405a680 con 0x7f3584152f60 2024-04-25T00:09:30.834 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:09:30.835 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:09:30.490930+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5236":{"gid":5236,"name":"b","rank":0,"incarnation":62,"state":"up:replay","state_seq":1,"addr":"172.21.15.174:6837/1159002662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1159002662},{"type":"v1","addr":"172.21.15.174:6837","nonce":1159002662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:09:30.490931+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"d","rank":0,"incarnation":62,"state":"up:replay","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:09:30.835 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 62 2024-04-25T00:09:30.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.836+0000 7f358aa92640 1 -- 172.21.15.116:0/355282647 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f35640678c0 msgr2=0x7f3564069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:30.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.836+0000 7f358aa92640 1 --2- 172.21.15.116:0/355282647 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f35640678c0 0x7f3564069d80 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f356c0ad000 tx=0x7f356c0a6000 comp rx=0 tx=0).stop 2024-04-25T00:09:30.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.836+0000 7f358aa92640 1 -- 172.21.15.116:0/355282647 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3584152f60 msgr2=0x7f358414c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:30.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.836+0000 7f358aa92640 1 --2- 172.21.15.116:0/355282647 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3584152f60 0x7f358414c970 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f357400e8a0 tx=0x7f357400ed60 comp rx=0 tx=0).stop 2024-04-25T00:09:30.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.836+0000 7f358aa92640 1 -- 172.21.15.116:0/355282647 shutdown_connections 2024-04-25T00:09:30.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.836+0000 7f358aa92640 1 --2- 172.21.15.116:0/355282647 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f35640678c0 0x7f3564069d80 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:30.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.836+0000 7f358aa92640 1 --2- 172.21.15.116:0/355282647 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3584158ea0 0x7f358414ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:30.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.836+0000 7f358aa92640 1 --2- 172.21.15.116:0/355282647 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3584152f60 0x7f358414c970 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:30.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.836+0000 7f358aa92640 1 --2- 172.21.15.116:0/355282647 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f35840c3c30 0x7f35841564c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:30.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.836+0000 7f358aa92640 1 -- 172.21.15.116:0/355282647 >> 172.21.15.116:0/355282647 conn(0x7f35840c4270 msgr2=0x7f3584151870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:09:30.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.836+0000 7f358aa92640 1 -- 172.21.15.116:0/355282647 shutdown_connections 2024-04-25T00:09:30.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.836+0000 7f358aa92640 1 -- 172.21.15.116:0/355282647 wait complete. 2024-04-25T00:09:30.853 INFO:tasks.mds_thrash.fs.[b]:mds.a down, removed from mdsmap 2024-04-25T00:09:30.853 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-04-25T00:09:30.853 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:09:30.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.896+0000 7f70fa7c4640 1 Processor -- start 2024-04-25T00:09:30.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.896+0000 7f70fa7c4640 1 -- start start 2024-04-25T00:09:30.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.900+0000 7f70fa7c4640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f70f4150d20 0x7f70f40c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:30.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.900+0000 7f70fa7c4640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70f40c3910 con 0x7f70f40a94a0 2024-04-25T00:09:30.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.900+0000 7f70fa7c4640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70f40c16e0 con 0x7f70f40c2f50 2024-04-25T00:09:30.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.900+0000 7f70fa7c4640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f70f40c1400 con 0x7f70f4150d20 2024-04-25T00:09:30.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.900+0000 7f70f8d61640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f70f40a94a0 0x7f70f4153ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:47782/0 (socket says 172.21.15.116:47782) 2024-04-25T00:09:30.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.900+0000 7f70f8d61640 1 -- 172.21.15.116:0/1116774333 learned_addr learned my addr 172.21.15.116:0/1116774333 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:09:30.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.900+0000 7f70f3fff640 1 --2- 172.21.15.116:0/1116774333 >> v2:172.21.15.174:3300/0 conn(0x7f70f4150d20 0x7f70f40c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:30.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.900+0000 7f70f2ffd640 1 -- 172.21.15.116:0/1116774333 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3213871500 0 0) 0x7f70f40c3910 con 0x7f70f40a94a0 2024-04-25T00:09:30.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.900+0000 7f70f2ffd640 1 -- 172.21.15.116:0/1116774333 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f70dc003530 con 0x7f70f40a94a0 2024-04-25T00:09:30.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.900+0000 7f70f2ffd640 1 -- 172.21.15.116:0/1116774333 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2529356995 0 0) 0x7f70dc003530 con 0x7f70f40a94a0 2024-04-25T00:09:30.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.900+0000 7f70f2ffd640 1 -- 172.21.15.116:0/1116774333 >> v2:172.21.15.174:3300/0 conn(0x7f70f4150d20 msgr2=0x7f70f40c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:09:30.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.900+0000 7f70f2ffd640 1 --2- 172.21.15.116:0/1116774333 >> v2:172.21.15.174:3300/0 conn(0x7f70f4150d20 0x7f70f40c2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:30.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.900+0000 7f70f2ffd640 1 -- 172.21.15.116:0/1116774333 >> v1:172.21.15.174:6789/0 conn(0x7f70f40c2f50 legacy=0x7f70f41538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:30.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.900+0000 7f70f2ffd640 1 -- 172.21.15.116:0/1116774333 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70f406e660 con 0x7f70f40a94a0 2024-04-25T00:09:30.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.900+0000 7f70f2ffd640 1 -- 172.21.15.116:0/1116774333 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f70ec001a00 con 0x7f70f40a94a0 2024-04-25T00:09:30.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.900+0000 7f70f2ffd640 1 -- 172.21.15.116:0/1116774333 >> v1:172.21.15.116:6789/0 conn(0x7f70f40a94a0 legacy=0x7f70f4153ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:30.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.900+0000 7f70f2ffd640 1 --2- 172.21.15.116:0/1116774333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70dc003a10 0x7f70dc003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:30.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.900+0000 7f70f2ffd640 1 --2- 172.21.15.116:0/1116774333 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f70dc004810 0x7f70dc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:30.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.900+0000 7f70f2ffd640 1 --2- 172.21.15.116:0/1116774333 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f70dc0051d0 0x7f70dc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:30.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.900+0000 7f70f2ffd640 1 -- 172.21.15.116:0/1116774333 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f70f40c1400 con 0x7f70dc003a10 2024-04-25T00:09:30.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.900+0000 7f70f2ffd640 1 -- 172.21.15.116:0/1116774333 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f70dc00c0d0 con 0x7f70dc004810 2024-04-25T00:09:30.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.900+0000 7f70f2ffd640 1 -- 172.21.15.116:0/1116774333 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f70dc00c2d0 con 0x7f70dc0051d0 2024-04-25T00:09:30.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.900+0000 7f70f3fff640 1 --2- 172.21.15.116:0/1116774333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70dc003a10 0x7f70dc003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:30.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.900+0000 7f70f37fe640 1 --2- 172.21.15.116:0/1116774333 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f70dc004810 0x7f70dc004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:30.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.900+0000 7f70f8d61640 1 --2- 172.21.15.116:0/1116774333 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f70dc0051d0 0x7f70dc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:30.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.900+0000 7f70f3fff640 1 -- 172.21.15.116:0/1116774333 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f70dc0051d0 msgr2=0x7f70dc00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:09:30.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.900+0000 7f70f3fff640 1 --2- 172.21.15.116:0/1116774333 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f70dc0051d0 0x7f70dc00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:30.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.900+0000 7f70f3fff640 1 -- 172.21.15.116:0/1116774333 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f70dc004810 msgr2=0x7f70dc004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:09:30.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.900+0000 7f70f3fff640 1 --2- 172.21.15.116:0/1116774333 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f70dc004810 0x7f70dc004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:30.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.900+0000 7f70f3fff640 1 -- 172.21.15.116:0/1116774333 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70dc003530 con 0x7f70dc003a10 2024-04-25T00:09:30.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.900+0000 7f70f37fe640 1 --2- 172.21.15.116:0/1116774333 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f70dc004810 0x7f70dc004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:09:30.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.900+0000 7f70f8d61640 1 -- 172.21.15.116:0/1116774333 reap_dead start 2024-04-25T00:09:30.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.900+0000 7f70f3fff640 1 --2- 172.21.15.116:0/1116774333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70dc003a10 0x7f70dc003e30 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f70e4009ae0 tx=0x7f70e401ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6ef996b5cc7d8f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:30.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.900+0000 7f70f2ffd640 1 -- 172.21.15.116:0/1116774333 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70e40b2070 con 0x7f70dc003a10 2024-04-25T00:09:30.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.904+0000 7f70f2ffd640 1 -- 172.21.15.116:0/1116774333 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70e40a5570 con 0x7f70dc003a10 2024-04-25T00:09:30.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.904+0000 7f70f2ffd640 1 -- 172.21.15.116:0/1116774333 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70e40096b0 con 0x7f70dc003a10 2024-04-25T00:09:30.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.904+0000 7f70fa7c4640 1 -- 172.21.15.116:0/1116774333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70dc003a10 msgr2=0x7f70dc003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:30.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.904+0000 7f70fa7c4640 1 --2- 172.21.15.116:0/1116774333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70dc003a10 0x7f70dc003e30 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f70e4009ae0 tx=0x7f70e401ca90 comp rx=0 tx=0).stop 2024-04-25T00:09:30.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.904+0000 7f70fa7c4640 1 -- 172.21.15.116:0/1116774333 shutdown_connections 2024-04-25T00:09:30.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.904+0000 7f70fa7c4640 1 --2- 172.21.15.116:0/1116774333 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70dc003a10 0x7f70dc003e30 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:30.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.904+0000 7f70fa7c4640 1 -- 172.21.15.116:0/1116774333 >> 172.21.15.116:0/1116774333 conn(0x7f70f40c3b00 msgr2=0x7f70f40c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:09:30.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.904+0000 7f70fa7c4640 1 -- 172.21.15.116:0/1116774333 shutdown_connections 2024-04-25T00:09:30.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.904+0000 7f70fa7c4640 1 -- 172.21.15.116:0/1116774333 wait complete. 2024-04-25T00:09:30.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.904+0000 7f70fa7c4640 1 Processor -- start 2024-04-25T00:09:30.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.904+0000 7f70fa7c4640 1 -- start start 2024-04-25T00:09:30.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.904+0000 7f70fa7c4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70dc003a10 0x7f70f4149450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:30.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.904+0000 7f70fa7c4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f70f41ec420 0x7f70f41ec8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:30.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.904+0000 7f70fa7c4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f70f4145340 0x7f70f41457a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:30.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.904+0000 7f70fa7c4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f70f4154480 con 0x7f70dc003a10 2024-04-25T00:09:30.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.904+0000 7f70fa7c4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f70f4154300 con 0x7f70f41ec420 2024-04-25T00:09:30.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.904+0000 7f70fa7c4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f70f4156b30 con 0x7f70f4145340 2024-04-25T00:09:30.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.904+0000 7f70f3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70dc003a10 0x7f70f4149450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:30.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.904+0000 7f70f3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70dc003a10 0x7f70f4149450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44902/0 (socket says 172.21.15.116:44902) 2024-04-25T00:09:30.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.904+0000 7f70f3fff640 1 -- 172.21.15.116:0/369316502 learned_addr learned my addr 172.21.15.116:0/369316502 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:09:30.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.904+0000 7f70f8d61640 1 --2- 172.21.15.116:0/369316502 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f70f4145340 0x7f70f41457a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:30.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.908+0000 7f70f37fe640 1 --2- 172.21.15.116:0/369316502 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f70f41ec420 0x7f70f41ec8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:30.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.908+0000 7f70f3fff640 1 -- 172.21.15.116:0/369316502 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f70f4145340 msgr2=0x7f70f41457a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:30.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.908+0000 7f70f3fff640 1 --2- 172.21.15.116:0/369316502 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f70f4145340 0x7f70f41457a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:30.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.908+0000 7f70f3fff640 1 -- 172.21.15.116:0/369316502 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f70f41ec420 msgr2=0x7f70f41ec8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:30.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.908+0000 7f70f3fff640 1 --2- 172.21.15.116:0/369316502 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f70f41ec420 0x7f70f41ec8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:30.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.908+0000 7f70f3fff640 1 -- 172.21.15.116:0/369316502 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70f41f07f0 con 0x7f70dc003a10 2024-04-25T00:09:30.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.908+0000 7f70f3fff640 1 --2- 172.21.15.116:0/369316502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70dc003a10 0x7f70f4149450 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f70e40b0040 tx=0x7f70e4019830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:30.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.908+0000 7f70f0ff9640 1 -- 172.21.15.116:0/369316502 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70e40b2040 con 0x7f70dc003a10 2024-04-25T00:09:30.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.908+0000 7f70f0ff9640 1 -- 172.21.15.116:0/369316502 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70e40a5de0 con 0x7f70dc003a10 2024-04-25T00:09:30.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.908+0000 7f70fa7c4640 1 -- 172.21.15.116:0/369316502 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70f41efa30 con 0x7f70dc003a10 2024-04-25T00:09:30.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.908+0000 7f70f0ff9640 1 -- 172.21.15.116:0/369316502 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70e4004020 con 0x7f70dc003a10 2024-04-25T00:09:30.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.908+0000 7f70fa7c4640 1 -- 172.21.15.116:0/369316502 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70f41efff0 con 0x7f70dc003a10 2024-04-25T00:09:30.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.908+0000 7f70fa7c4640 1 -- 172.21.15.116:0/369316502 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70f4068350 con 0x7f70dc003a10 2024-04-25T00:09:30.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.908+0000 7f70f0ff9640 1 -- 172.21.15.116:0/369316502 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f70e40bf030 con 0x7f70dc003a10 2024-04-25T00:09:30.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.908+0000 7f70f0ff9640 1 --2- 172.21.15.116:0/369316502 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f70e00678f0 0x7f70e0069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:30.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.908+0000 7f70f0ff9640 1 -- 172.21.15.116:0/369316502 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7612+0+0 (secure 0 0 0) 0x7f70e411fd50 con 0x7f70dc003a10 2024-04-25T00:09:30.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.908+0000 7f70f37fe640 1 --2- 172.21.15.116:0/369316502 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f70e00678f0 0x7f70e0069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:30.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.908+0000 7f70f37fe640 1 --2- 172.21.15.116:0/369316502 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f70e00678f0 0x7f70e0069db0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f70f40c4940 tx=0x7f70e8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:30.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:30.912+0000 7f70f0ff9640 1 -- 172.21.15.116:0/369316502 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f70e4123030 con 0x7f70dc003a10 2024-04-25T00:09:31.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.008+0000 7f7c7b125640 1 Processor -- start 2024-04-25T00:09:31.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.012+0000 7f7c7b125640 1 -- start start 2024-04-25T00:09:31.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.012+0000 7f7c7b125640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7c740c6080 0x7f7c7406fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:31.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.012+0000 7f7c7b125640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c740c6490 con 0x7f7c740c3c30 2024-04-25T00:09:31.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.012+0000 7f7c7b125640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c7406e8f0 con 0x7f7c74152f60 2024-04-25T00:09:31.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.012+0000 7f7c7b125640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f7c741595b0 con 0x7f7c740c6080 2024-04-25T00:09:31.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.012+0000 7f7c796c2640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7c740c6080 0x7f7c7406fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:31.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.012+0000 7f7c796c2640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7c740c6080 0x7f7c7406fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44910/0 (socket says 172.21.15.116:44910) 2024-04-25T00:09:31.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.012+0000 7f7c796c2640 1 -- 172.21.15.116:0/1838344799 learned_addr learned my addr 172.21.15.116:0/1838344799 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:09:31.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.012+0000 7f7c796c2640 1 -- 172.21.15.116:0/1838344799 >> v1:172.21.15.174:6789/0 conn(0x7f7c74152f60 legacy=0x7f7c74158ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:09:31.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.012+0000 7f7c737fe640 1 -- 172.21.15.116:0/1838344799 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3565370470 0 0) 0x7f7c60001120 con 0x7f7c740c3c30 2024-04-25T00:09:31.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.012+0000 7f7c796c2640 1 -- 172.21.15.116:0/1838344799 >> v1:172.21.15.116:6789/0 conn(0x7f7c740c3c30 legacy=0x7f7c7406c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:31.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.012+0000 7f7c796c2640 1 -- 172.21.15.116:0/1838344799 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c7406ebd0 con 0x7f7c740c6080 2024-04-25T00:09:31.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.012+0000 7f7c796c2640 1 --2- 172.21.15.116:0/1838344799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c740c6080 0x7f7c7406fb30 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c009e50 tx=0x7f7c6c01c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77085aadaed302a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:31.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.012+0000 7f7c737fe640 1 -- 172.21.15.116:0/1838344799 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c6c0a5750 con 0x7f7c740c6080 2024-04-25T00:09:31.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.012+0000 7f7c737fe640 1 -- 172.21.15.116:0/1838344799 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c6c0a58f0 con 0x7f7c740c6080 2024-04-25T00:09:31.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.012+0000 7f7c737fe640 1 -- 172.21.15.116:0/1838344799 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c6c0a5cc0 con 0x7f7c740c6080 2024-04-25T00:09:31.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.016+0000 7f7c7b125640 1 -- 172.21.15.116:0/1838344799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c740c6080 msgr2=0x7f7c7406fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:31.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.016+0000 7f7c7b125640 1 --2- 172.21.15.116:0/1838344799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c740c6080 0x7f7c7406fb30 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c009e50 tx=0x7f7c6c01c9a0 comp rx=0 tx=0).stop 2024-04-25T00:09:31.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.016+0000 7f7c7b125640 1 -- 172.21.15.116:0/1838344799 shutdown_connections 2024-04-25T00:09:31.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.016+0000 7f7c7b125640 1 --2- 172.21.15.116:0/1838344799 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c740c6080 0x7f7c7406fb30 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:31.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.016+0000 7f7c7b125640 1 -- 172.21.15.116:0/1838344799 >> 172.21.15.116:0/1838344799 conn(0x7f7c740c4270 msgr2=0x7f7c740c4680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:09:31.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.016+0000 7f7c7b125640 1 -- 172.21.15.116:0/1838344799 shutdown_connections 2024-04-25T00:09:31.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.016+0000 7f7c7b125640 1 -- 172.21.15.116:0/1838344799 wait complete. 2024-04-25T00:09:31.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.016+0000 7f7c7b125640 1 Processor -- start 2024-04-25T00:09:31.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.016+0000 7f7c7b125640 1 -- start start 2024-04-25T00:09:31.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.016+0000 7f7c7b125640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7c740c3c30 0x7f7c7414faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:31.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.016+0000 7f7c7b125640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c740c6080 0x7f7c74158a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:31.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.016+0000 7f7c7b125640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7c74152f60 0x7f7c740aca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:31.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.016+0000 7f7c7b125640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7c7406c930 con 0x7f7c740c6080 2024-04-25T00:09:31.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.016+0000 7f7c7b125640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7c74074a70 con 0x7f7c74152f60 2024-04-25T00:09:31.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.016+0000 7f7c7b125640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f7c7406cab0 con 0x7f7c740c3c30 2024-04-25T00:09:31.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.016+0000 7f7c73fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c740c6080 0x7f7c74158a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:31.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.016+0000 7f7c73fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c740c6080 0x7f7c74158a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44920/0 (socket says 172.21.15.116:44920) 2024-04-25T00:09:31.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.016+0000 7f7c73fff640 1 -- 172.21.15.116:0/3479625020 learned_addr learned my addr 172.21.15.116:0/3479625020 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:09:31.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.016+0000 7f7c796c2640 1 --2- 172.21.15.116:0/3479625020 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7c74152f60 0x7f7c740aca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:31.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.016+0000 7f7c78ec1640 1 --2- 172.21.15.116:0/3479625020 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7c740c3c30 0x7f7c7414faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:31.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.016+0000 7f7c73fff640 1 -- 172.21.15.116:0/3479625020 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7c740c3c30 msgr2=0x7f7c7414faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:31.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.016+0000 7f7c73fff640 1 --2- 172.21.15.116:0/3479625020 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7c740c3c30 0x7f7c7414faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:31.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.016+0000 7f7c73fff640 1 -- 172.21.15.116:0/3479625020 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7c74152f60 msgr2=0x7f7c740aca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:31.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.016+0000 7f7c73fff640 1 --2- 172.21.15.116:0/3479625020 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7c74152f60 0x7f7c740aca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:31.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.016+0000 7f7c73fff640 1 -- 172.21.15.116:0/3479625020 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c741f23c0 con 0x7f7c740c6080 2024-04-25T00:09:31.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.016+0000 7f7c73fff640 1 --2- 172.21.15.116:0/3479625020 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c740c6080 0x7f7c74158a20 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f7c6000c410 tx=0x7f7c6000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:31.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.016+0000 7f7c717fa640 1 -- 172.21.15.116:0/3479625020 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c60017070 con 0x7f7c740c6080 2024-04-25T00:09:31.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.016+0000 7f7c717fa640 1 -- 172.21.15.116:0/3479625020 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c600052d0 con 0x7f7c740c6080 2024-04-25T00:09:31.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.016+0000 7f7c7b125640 1 -- 172.21.15.116:0/3479625020 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c741f26a0 con 0x7f7c740c6080 2024-04-25T00:09:31.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.016+0000 7f7c717fa640 1 -- 172.21.15.116:0/3479625020 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c60003840 con 0x7f7c740c6080 2024-04-25T00:09:31.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.016+0000 7f7c7b125640 1 -- 172.21.15.116:0/3479625020 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c74155220 con 0x7f7c740c6080 2024-04-25T00:09:31.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.020+0000 7f7c7b125640 1 -- 172.21.15.116:0/3479625020 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c74155420 con 0x7f7c740c6080 2024-04-25T00:09:31.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.020+0000 7f7c717fa640 1 -- 172.21.15.116:0/3479625020 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7c6001a070 con 0x7f7c740c6080 2024-04-25T00:09:31.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.020+0000 7f7c717fa640 1 --2- 172.21.15.116:0/3479625020 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7c580678c0 0x7f7c58069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:31.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.020+0000 7f7c717fa640 1 -- 172.21.15.116:0/3479625020 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7612+0+0 (secure 0 0 0) 0x7f7c60085640 con 0x7f7c740c6080 2024-04-25T00:09:31.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.020+0000 7f7c78ec1640 1 --2- 172.21.15.116:0/3479625020 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7c580678c0 0x7f7c58069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:31.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.020+0000 7f7c78ec1640 1 --2- 172.21.15.116:0/3479625020 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7c580678c0 0x7f7c58069d80 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f7c740ad950 tx=0x7f7c68009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:31.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.024+0000 7f7c717fa640 1 -- 172.21.15.116:0/3479625020 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f7c74155420 con 0x7f7c740c6080 2024-04-25T00:09:31.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.112+0000 7f70fa7c4640 1 -- 172.21.15.116:0/369316502 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f70f40c34f0 con 0x7f70dc003a10 2024-04-25T00:09:31.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.112+0000 7f70f0ff9640 1 -- 172.21.15.116:0/369316502 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) ==== 94+0+4832 (secure 0 0 0) 0x7f70e40c6060 con 0x7f70dc003a10 2024-04-25T00:09:31.114 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:09:31.114 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:09:30.490930+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5236":{"gid":5236,"name":"b","rank":0,"incarnation":62,"state":"up:replay","state_seq":1,"addr":"172.21.15.174:6837/1159002662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1159002662},{"type":"v1","addr":"172.21.15.174:6837","nonce":1159002662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:09:30.490931+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"d","rank":0,"incarnation":62,"state":"up:replay","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:09:31.115 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 62 2024-04-25T00:09:31.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.116+0000 7f70fa7c4640 1 -- 172.21.15.116:0/369316502 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f70e00678f0 msgr2=0x7f70e0069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:31.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.116+0000 7f70fa7c4640 1 --2- 172.21.15.116:0/369316502 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f70e00678f0 0x7f70e0069db0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f70f40c4940 tx=0x7f70e8009040 comp rx=0 tx=0).stop 2024-04-25T00:09:31.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.116+0000 7f70fa7c4640 1 -- 172.21.15.116:0/369316502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70dc003a10 msgr2=0x7f70f4149450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:31.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.116+0000 7f70fa7c4640 1 --2- 172.21.15.116:0/369316502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70dc003a10 0x7f70f4149450 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f70e40b0040 tx=0x7f70e4019830 comp rx=0 tx=0).stop 2024-04-25T00:09:31.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.116+0000 7f70fa7c4640 1 -- 172.21.15.116:0/369316502 shutdown_connections 2024-04-25T00:09:31.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.116+0000 7f70fa7c4640 1 --2- 172.21.15.116:0/369316502 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f70e00678f0 0x7f70e0069db0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:31.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.116+0000 7f70fa7c4640 1 --2- 172.21.15.116:0/369316502 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f70f4145340 0x7f70f41457a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:31.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.116+0000 7f70fa7c4640 1 --2- 172.21.15.116:0/369316502 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f70f41ec420 0x7f70f41ec8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:31.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.116+0000 7f70fa7c4640 1 --2- 172.21.15.116:0/369316502 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f70dc003a10 0x7f70f4149450 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:31.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.116+0000 7f70fa7c4640 1 -- 172.21.15.116:0/369316502 >> 172.21.15.116:0/369316502 conn(0x7f70f40c3b00 msgr2=0x7f70f4152e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:09:31.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.116+0000 7f70fa7c4640 1 -- 172.21.15.116:0/369316502 shutdown_connections 2024-04-25T00:09:31.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.116+0000 7f70fa7c4640 1 -- 172.21.15.116:0/369316502 wait complete. 2024-04-25T00:09:31.135 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 62, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:04.678737+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 62, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5236": { "addr": "172.21.15.174:6837/1159002662", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6836", "nonce": 1159002662, "type": "v2" }, { "addr": "172.21.15.174:6837", "nonce": 1159002662, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5236, "incarnation": 62, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 39, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T00:09:30.490930+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5236 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:18.825887+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 62, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_4235": { "addr": "172.21.15.174:6835/2883220846", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6834", "nonce": 2883220846, "type": "v2" }, { "addr": "172.21.15.174:6835", "nonce": 2883220846, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4235, "incarnation": 62, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 39, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-04-25T00:09:30.490931+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4235 } } } ], "standbys": [] } 2024-04-25T00:09:31.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.212+0000 7f7c7b125640 1 -- 172.21.15.116:0/3479625020 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7c74160070 con 0x7f7c740c6080 2024-04-25T00:09:31.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.216+0000 7f7c717fa640 1 -- 172.21.15.116:0/3479625020 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) ==== 94+0+4832 (secure 0 0 0) 0x7f7c740c6490 con 0x7f7c740c6080 2024-04-25T00:09:31.216 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:09:31.217 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:09:30.490930+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5236":{"gid":5236,"name":"b","rank":0,"incarnation":62,"state":"up:replay","state_seq":1,"addr":"172.21.15.174:6837/1159002662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1159002662},{"type":"v1","addr":"172.21.15.174:6837","nonce":1159002662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:09:30.490931+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"d","rank":0,"incarnation":62,"state":"up:replay","state_seq":1,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:09:31.217 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 62 2024-04-25T00:09:31.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.216+0000 7f7c7b125640 1 -- 172.21.15.116:0/3479625020 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7c580678c0 msgr2=0x7f7c58069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:31.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.216+0000 7f7c7b125640 1 --2- 172.21.15.116:0/3479625020 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7c580678c0 0x7f7c58069d80 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f7c740ad950 tx=0x7f7c68009040 comp rx=0 tx=0).stop 2024-04-25T00:09:31.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.216+0000 7f7c7b125640 1 -- 172.21.15.116:0/3479625020 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c740c6080 msgr2=0x7f7c74158a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:31.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.216+0000 7f7c7b125640 1 --2- 172.21.15.116:0/3479625020 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c740c6080 0x7f7c74158a20 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f7c6000c410 tx=0x7f7c6000c8d0 comp rx=0 tx=0).stop 2024-04-25T00:09:31.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.216+0000 7f7c7b125640 1 -- 172.21.15.116:0/3479625020 shutdown_connections 2024-04-25T00:09:31.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.216+0000 7f7c7b125640 1 --2- 172.21.15.116:0/3479625020 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7c580678c0 0x7f7c58069d80 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:31.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.216+0000 7f7c7b125640 1 --2- 172.21.15.116:0/3479625020 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7c74152f60 0x7f7c740aca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:31.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.216+0000 7f7c7b125640 1 --2- 172.21.15.116:0/3479625020 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c740c6080 0x7f7c74158a20 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:31.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.216+0000 7f7c7b125640 1 --2- 172.21.15.116:0/3479625020 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7c740c3c30 0x7f7c7414faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:31.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.216+0000 7f7c7b125640 1 -- 172.21.15.116:0/3479625020 >> 172.21.15.116:0/3479625020 conn(0x7f7c740c4270 msgr2=0x7f7c741520b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:09:31.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.216+0000 7f7c7b125640 1 -- 172.21.15.116:0/3479625020 shutdown_connections 2024-04-25T00:09:31.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:31.216+0000 7f7c7b125640 1 -- 172.21.15.116:0/3479625020 wait complete. 2024-04-25T00:09:31.234 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 62, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:04.678737+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 62, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5236": { "addr": "172.21.15.174:6837/1159002662", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6836", "nonce": 1159002662, "type": "v2" }, { "addr": "172.21.15.174:6837", "nonce": 1159002662, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5236, "incarnation": 62, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 39, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T00:09:30.490930+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5236 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:18.825887+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 62, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_4235": { "addr": "172.21.15.174:6835/2883220846", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6834", "nonce": 2883220846, "type": "v2" }, { "addr": "172.21.15.174:6835", "nonce": 2883220846, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4235, "incarnation": 62, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 39, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-04-25T00:09:30.490931+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4235 } } } ], "standbys": [] } 2024-04-25T00:09:33.136 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:09:33.235 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:09:33.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.264+0000 7f917ad4f640 1 Processor -- start 2024-04-25T00:09:33.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.264+0000 7f917ad4f640 1 -- start start 2024-04-25T00:09:33.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.264+0000 7f917ad4f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f91740c3f70 0x7f91741557c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:33.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.264+0000 7f917ad4f640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f91740c3a20 0x7f9174155d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:33.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.264+0000 7f917ad4f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9174153220 con 0x7f9174152e40 2024-04-25T00:09:33.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.264+0000 7f917ad4f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f91740c4c90 con 0x7f91740c3f70 2024-04-25T00:09:33.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.264+0000 7f917ad4f640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f9174156240 con 0x7f91740c3a20 2024-04-25T00:09:33.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.264+0000 7f9173fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f91740c3f70 0x7f91741557c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:33.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.264+0000 7f9173fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f91740c3f70 0x7f91741557c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44936/0 (socket says 172.21.15.116:44936) 2024-04-25T00:09:33.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.264+0000 7f9173fff640 1 -- 172.21.15.116:0/968293457 learned_addr learned my addr 172.21.15.116:0/968293457 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:09:33.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.264+0000 7f9173fff640 1 -- 172.21.15.116:0/968293457 >> v2:172.21.15.174:3300/0 conn(0x7f91740c3a20 msgr2=0x7f9174155d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:09:33.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.264+0000 7f91792ec640 1 --2- 172.21.15.116:0/968293457 >> v2:172.21.15.174:3300/0 conn(0x7f91740c3a20 0x7f9174155d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:33.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.264+0000 7f9173fff640 1 --2- 172.21.15.116:0/968293457 >> v2:172.21.15.174:3300/0 conn(0x7f91740c3a20 0x7f9174155d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:33.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.264+0000 7f9173fff640 1 -- 172.21.15.116:0/968293457 >> v1:172.21.15.116:6789/0 conn(0x7f9174152e40 legacy=0x7f91740710b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:33.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.264+0000 7f91737fe640 1 -- 172.21.15.116:0/968293457 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 456808498 0 0) 0x7f9174153220 con 0x7f9174152e40 2024-04-25T00:09:33.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.264+0000 7f9173fff640 1 -- 172.21.15.116:0/968293457 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91740c7a90 con 0x7f91740c3f70 2024-04-25T00:09:33.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.264+0000 7f91792ec640 1 --2- 172.21.15.116:0/968293457 >> v2:172.21.15.174:3300/0 conn(0x7f91740c3a20 0x7f9174155d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:09:33.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.268+0000 7f9173fff640 1 --2- 172.21.15.116:0/968293457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91740c3f70 0x7f91741557c0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f91740c6550 tx=0x7f916801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26f49823ec8f11aa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:33.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.268+0000 7f91737fe640 1 -- 172.21.15.116:0/968293457 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91680ab660 con 0x7f91740c3f70 2024-04-25T00:09:33.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.268+0000 7f91737fe640 1 -- 172.21.15.116:0/968293457 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91680b0070 con 0x7f91740c3f70 2024-04-25T00:09:33.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.268+0000 7f91737fe640 1 -- 172.21.15.116:0/968293457 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91680aba20 con 0x7f91740c3f70 2024-04-25T00:09:33.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.268+0000 7f917ad4f640 1 -- 172.21.15.116:0/968293457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91740c3f70 msgr2=0x7f91741557c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:33.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.268+0000 7f917ad4f640 1 --2- 172.21.15.116:0/968293457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91740c3f70 0x7f91741557c0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f91740c6550 tx=0x7f916801c460 comp rx=0 tx=0).stop 2024-04-25T00:09:33.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.268+0000 7f917ad4f640 1 -- 172.21.15.116:0/968293457 shutdown_connections 2024-04-25T00:09:33.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.268+0000 7f917ad4f640 1 --2- 172.21.15.116:0/968293457 >> v2:172.21.15.174:3300/0 conn(0x7f91740c3a20 0x7f9174155d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:33.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.268+0000 7f917ad4f640 1 --2- 172.21.15.116:0/968293457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91740c3f70 0x7f91741557c0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:33.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.268+0000 7f917ad4f640 1 -- 172.21.15.116:0/968293457 >> 172.21.15.116:0/968293457 conn(0x7f91740c9160 msgr2=0x7f91740c9570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:09:33.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.268+0000 7f917ad4f640 1 -- 172.21.15.116:0/968293457 shutdown_connections 2024-04-25T00:09:33.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.268+0000 7f917ad4f640 1 -- 172.21.15.116:0/968293457 wait complete. 2024-04-25T00:09:33.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.268+0000 7f917ad4f640 1 Processor -- start 2024-04-25T00:09:33.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.268+0000 7f917ad4f640 1 -- start start 2024-04-25T00:09:33.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.268+0000 7f917ad4f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91740c3a20 0x7f9174156dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:33.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.268+0000 7f917ad4f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f91740c3f70 0x7f91741f1620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:33.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.268+0000 7f917ad4f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9174152e40 0x7f91741f1b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:33.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.268+0000 7f917ad4f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9174071940 con 0x7f91740c3a20 2024-04-25T00:09:33.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.268+0000 7f917ad4f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f91740717c0 con 0x7f9174152e40 2024-04-25T00:09:33.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.268+0000 7f917ad4f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f91740c5120 con 0x7f91740c3f70 2024-04-25T00:09:33.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.268+0000 7f9178aeb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91740c3a20 0x7f9174156dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:33.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.268+0000 7f9178aeb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91740c3a20 0x7f9174156dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44942/0 (socket says 172.21.15.116:44942) 2024-04-25T00:09:33.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.268+0000 7f9178aeb640 1 -- 172.21.15.116:0/620647654 learned_addr learned my addr 172.21.15.116:0/620647654 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:09:33.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.272+0000 7f9178aeb640 1 -- 172.21.15.116:0/620647654 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f91740c3f70 msgr2=0x7f91741f1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:33.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.272+0000 7f9178aeb640 1 --2- 172.21.15.116:0/620647654 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f91740c3f70 0x7f91741f1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:33.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.272+0000 7f91792ec640 1 --2- 172.21.15.116:0/620647654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9174152e40 0x7f91741f1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:33.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.272+0000 7f9178aeb640 1 -- 172.21.15.116:0/620647654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9174152e40 msgr2=0x7f91741f1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:33.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.272+0000 7f9178aeb640 1 --2- 172.21.15.116:0/620647654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9174152e40 0x7f91741f1b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:33.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.272+0000 7f9178aeb640 1 -- 172.21.15.116:0/620647654 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91741f20a0 con 0x7f91740c3a20 2024-04-25T00:09:33.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.272+0000 7f9178aeb640 1 --2- 172.21.15.116:0/620647654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91740c3a20 0x7f9174156dd0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f916c00c840 tx=0x7f916c00cd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:33.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.272+0000 7f91717fa640 1 -- 172.21.15.116:0/620647654 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f916c003ba0 con 0x7f91740c3a20 2024-04-25T00:09:33.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.272+0000 7f91717fa640 1 -- 172.21.15.116:0/620647654 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f916c003d40 con 0x7f91740c3a20 2024-04-25T00:09:33.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.272+0000 7f917ad4f640 1 -- 172.21.15.116:0/620647654 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91740702e0 con 0x7f91740c3a20 2024-04-25T00:09:33.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.272+0000 7f91717fa640 1 -- 172.21.15.116:0/620647654 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f916c00aa90 con 0x7f91740c3a20 2024-04-25T00:09:33.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.272+0000 7f917ad4f640 1 -- 172.21.15.116:0/620647654 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91740705a0 con 0x7f91740c3a20 2024-04-25T00:09:33.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.272+0000 7f917ad4f640 1 -- 172.21.15.116:0/620647654 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9174068a60 con 0x7f91740c3a20 2024-04-25T00:09:33.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.272+0000 7f91717fa640 1 -- 172.21.15.116:0/620647654 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f916c00ac30 con 0x7f91740c3a20 2024-04-25T00:09:33.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.272+0000 7f91717fa640 1 --2- 172.21.15.116:0/620647654 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f91600678c0 0x7f9160069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:33.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.272+0000 7f91717fa640 1 -- 172.21.15.116:0/620647654 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7612+0+0 (secure 0 0 0) 0x7f916c086370 con 0x7f91740c3a20 2024-04-25T00:09:33.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.272+0000 7f9173fff640 1 --2- 172.21.15.116:0/620647654 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f91600678c0 0x7f9160069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:33.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.272+0000 7f9173fff640 1 --2- 172.21.15.116:0/620647654 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f91600678c0 0x7f9160069d80 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f91680af000 tx=0x7f9168004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:33.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.276+0000 7f91717fa640 1 -- 172.21.15.116:0/620647654 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f916c0567d0 con 0x7f91740c3a20 2024-04-25T00:09:33.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.356+0000 7f95a08fb640 1 Processor -- start 2024-04-25T00:09:33.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.356+0000 7f95a08fb640 1 -- start start 2024-04-25T00:09:33.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.360+0000 7f95a08fb640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f959c0c9040 0x7f959c0c9420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:33.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.360+0000 7f95a08fb640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f959c06da40 0x7f959c14bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:33.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.360+0000 7f95a08fb640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f959c159b60 con 0x7f959c0c6e20 2024-04-25T00:09:33.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.360+0000 7f95a08fb640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f959c0c7860 con 0x7f959c06da40 2024-04-25T00:09:33.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.360+0000 7f95a08fb640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f959c0c3920 con 0x7f959c0c9040 2024-04-25T00:09:33.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.360+0000 7f959a59c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f959c0c9040 0x7f959c0c9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:33.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.360+0000 7f959ad9d640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f959c06da40 0x7f959c14bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:33.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.360+0000 7f959a59c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f959c0c9040 0x7f959c0c9420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:40842/0 (socket says 172.21.15.116:40842) 2024-04-25T00:09:33.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.360+0000 7f959a59c640 1 -- 172.21.15.116:0/1470003158 learned_addr learned my addr 172.21.15.116:0/1470003158 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:09:33.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.360+0000 7f959a59c640 1 -- 172.21.15.116:0/1470003158 >> v2:172.21.15.174:3300/0 conn(0x7f959c06da40 msgr2=0x7f959c14bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:09:33.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.360+0000 7f959a59c640 1 --2- 172.21.15.116:0/1470003158 >> v2:172.21.15.174:3300/0 conn(0x7f959c06da40 0x7f959c14bcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:33.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.360+0000 7f959a59c640 1 -- 172.21.15.116:0/1470003158 >> v1:172.21.15.174:6789/0 conn(0x7f959c0c6e20 legacy=0x7f959c06d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:33.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.360+0000 7f959a59c640 1 -- 172.21.15.116:0/1470003158 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f959c0c3b70 con 0x7f959c0c9040 2024-04-25T00:09:33.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.360+0000 7f959a59c640 1 --2- 172.21.15.116:0/1470003158 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f959c0c9040 0x7f959c0c9420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9594009e50 tx=0x7f959401c9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e0373a0912b4064 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:33.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.360+0000 7f959959a640 1 -- 172.21.15.116:0/1470003158 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95940a5660 con 0x7f959c0c9040 2024-04-25T00:09:33.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.360+0000 7f959959a640 1 -- 172.21.15.116:0/1470003158 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95940a5800 con 0x7f959c0c9040 2024-04-25T00:09:33.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.360+0000 7f959959a640 1 -- 172.21.15.116:0/1470003158 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95940a5bd0 con 0x7f959c0c9040 2024-04-25T00:09:33.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.364+0000 7f95a08fb640 1 -- 172.21.15.116:0/1470003158 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f959c0c9040 msgr2=0x7f959c0c9420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:33.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.364+0000 7f95a08fb640 1 --2- 172.21.15.116:0/1470003158 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f959c0c9040 0x7f959c0c9420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9594009e50 tx=0x7f959401c9a0 comp rx=0 tx=0).stop 2024-04-25T00:09:33.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.364+0000 7f95a08fb640 1 -- 172.21.15.116:0/1470003158 shutdown_connections 2024-04-25T00:09:33.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.364+0000 7f95a08fb640 1 --2- 172.21.15.116:0/1470003158 >> v2:172.21.15.174:3300/0 conn(0x7f959c06da40 0x7f959c14bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:33.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.364+0000 7f95a08fb640 1 --2- 172.21.15.116:0/1470003158 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f959c0c9040 0x7f959c0c9420 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:33.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.364+0000 7f95a08fb640 1 -- 172.21.15.116:0/1470003158 >> 172.21.15.116:0/1470003158 conn(0x7f959c074480 msgr2=0x7f959c074890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:09:33.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.364+0000 7f95a08fb640 1 -- 172.21.15.116:0/1470003158 shutdown_connections 2024-04-25T00:09:33.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.364+0000 7f95a08fb640 1 -- 172.21.15.116:0/1470003158 wait complete. 2024-04-25T00:09:33.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.364+0000 7f95a08fb640 1 Processor -- start 2024-04-25T00:09:33.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.364+0000 7f95a08fb640 1 -- start start 2024-04-25T00:09:33.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.364+0000 7f95a08fb640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f959c06da40 0x7f959c0ad040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:33.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.364+0000 7f95a08fb640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f959c0c6e20 0x7f959c152040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:33.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.364+0000 7f95a08fb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f959c0c9040 0x7f959c150160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:33.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.364+0000 7f95a08fb640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f959c1565a0 con 0x7f959c0c9040 2024-04-25T00:09:33.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.364+0000 7f95a08fb640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f959c156420 con 0x7f959c0c6e20 2024-04-25T00:09:33.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.364+0000 7f95a08fb640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f959c156720 con 0x7f959c06da40 2024-04-25T00:09:33.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.364+0000 7f959ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f959c0c9040 0x7f959c150160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:33.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.364+0000 7f959ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f959c0c9040 0x7f959c150160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44946/0 (socket says 172.21.15.116:44946) 2024-04-25T00:09:33.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.364+0000 7f959ad9d640 1 -- 172.21.15.116:0/1157642044 learned_addr learned my addr 172.21.15.116:0/1157642044 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:09:33.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.364+0000 7f959a59c640 1 --2- 172.21.15.116:0/1157642044 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f959c06da40 0x7f959c0ad040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:33.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.364+0000 7f9599d9b640 1 --2- 172.21.15.116:0/1157642044 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f959c0c6e20 0x7f959c152040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:33.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.364+0000 7f959ad9d640 1 -- 172.21.15.116:0/1157642044 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f959c06da40 msgr2=0x7f959c0ad040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:33.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.364+0000 7f959ad9d640 1 --2- 172.21.15.116:0/1157642044 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f959c06da40 0x7f959c0ad040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:33.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.364+0000 7f959ad9d640 1 -- 172.21.15.116:0/1157642044 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f959c0c6e20 msgr2=0x7f959c152040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:33.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.364+0000 7f959ad9d640 1 --2- 172.21.15.116:0/1157642044 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f959c0c6e20 0x7f959c152040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:33.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.364+0000 7f959ad9d640 1 -- 172.21.15.116:0/1157642044 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f959c14dac0 con 0x7f959c0c9040 2024-04-25T00:09:33.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.364+0000 7f959a59c640 1 --2- 172.21.15.116:0/1157642044 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f959c06da40 0x7f959c0ad040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:09:33.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.364+0000 7f9599d9b640 1 --2- 172.21.15.116:0/1157642044 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f959c0c6e20 0x7f959c152040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:09:33.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.364+0000 7f959ad9d640 1 --2- 172.21.15.116:0/1157642044 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f959c0c9040 0x7f959c150160 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f959000e7f0 tx=0x7f959000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:33.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.364+0000 7f958affd640 1 -- 172.21.15.116:0/1157642044 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9590017070 con 0x7f959c0c9040 2024-04-25T00:09:33.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.364+0000 7f958affd640 1 -- 172.21.15.116:0/1157642044 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f959000c8e0 con 0x7f959c0c9040 2024-04-25T00:09:33.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.364+0000 7f95a08fb640 1 -- 172.21.15.116:0/1157642044 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f959c1ee5b0 con 0x7f959c0c9040 2024-04-25T00:09:33.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.364+0000 7f958affd640 1 -- 172.21.15.116:0/1157642044 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f959000eea0 con 0x7f959c0c9040 2024-04-25T00:09:33.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.364+0000 7f95a08fb640 1 -- 172.21.15.116:0/1157642044 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f959c14dd60 con 0x7f959c0c9040 2024-04-25T00:09:33.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.368+0000 7f958affd640 1 -- 172.21.15.116:0/1157642044 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f959001a070 con 0x7f959c0c9040 2024-04-25T00:09:33.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.368+0000 7f95a08fb640 1 -- 172.21.15.116:0/1157642044 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f959c068a60 con 0x7f959c0c9040 2024-04-25T00:09:33.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.368+0000 7f958affd640 1 --2- 172.21.15.116:0/1157642044 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f957c0678c0 0x7f957c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:33.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.368+0000 7f958affd640 1 -- 172.21.15.116:0/1157642044 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7612+0+0 (secure 0 0 0) 0x7f95900862f0 con 0x7f959c0c9040 2024-04-25T00:09:33.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.368+0000 7f959a59c640 1 --2- 172.21.15.116:0/1157642044 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f957c0678c0 0x7f957c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:33.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.368+0000 7f959a59c640 1 --2- 172.21.15.116:0/1157642044 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f957c0678c0 0x7f957c069d80 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f95940ae000 tx=0x7f95940a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:33.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.372+0000 7f958affd640 1 -- 172.21.15.116:0/1157642044 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f95900566c0 con 0x7f959c0c9040 2024-04-25T00:09:33.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.488+0000 7f917ad4f640 1 -- 172.21.15.116:0/620647654 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f91740c7870 con 0x7f91740c3a20 2024-04-25T00:09:33.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.488+0000 7f91717fa640 1 -- 172.21.15.116:0/620647654 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) ==== 94+0+4835 (secure 0 0 0) 0x7f916c0560d0 con 0x7f91740c3a20 2024-04-25T00:09:33.490 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:09:33.490 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:09:31.593888+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5236":{"gid":5236,"name":"b","rank":0,"incarnation":62,"state":"up:rejoin","state_seq":24,"addr":"172.21.15.174:6837/1159002662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1159002662},{"type":"v1","addr":"172.21.15.174:6837","nonce":1159002662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:09:31.595714+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"d","rank":0,"incarnation":62,"state":"up:rejoin","state_seq":131,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:09:33.490 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 64 2024-04-25T00:09:33.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.488+0000 7f917ad4f640 1 -- 172.21.15.116:0/620647654 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f91600678c0 msgr2=0x7f9160069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:33.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.488+0000 7f917ad4f640 1 --2- 172.21.15.116:0/620647654 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f91600678c0 0x7f9160069d80 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f91680af000 tx=0x7f9168004740 comp rx=0 tx=0).stop 2024-04-25T00:09:33.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.488+0000 7f917ad4f640 1 -- 172.21.15.116:0/620647654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91740c3a20 msgr2=0x7f9174156dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:33.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.488+0000 7f917ad4f640 1 --2- 172.21.15.116:0/620647654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91740c3a20 0x7f9174156dd0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f916c00c840 tx=0x7f916c00cd00 comp rx=0 tx=0).stop 2024-04-25T00:09:33.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.488+0000 7f917ad4f640 1 -- 172.21.15.116:0/620647654 shutdown_connections 2024-04-25T00:09:33.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.488+0000 7f917ad4f640 1 --2- 172.21.15.116:0/620647654 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f91600678c0 0x7f9160069d80 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:33.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.488+0000 7f917ad4f640 1 --2- 172.21.15.116:0/620647654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9174152e40 0x7f91741f1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:33.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.488+0000 7f917ad4f640 1 --2- 172.21.15.116:0/620647654 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f91740c3f70 0x7f91741f1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:33.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.488+0000 7f917ad4f640 1 --2- 172.21.15.116:0/620647654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f91740c3a20 0x7f9174156dd0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:33.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.488+0000 7f917ad4f640 1 -- 172.21.15.116:0/620647654 >> 172.21.15.116:0/620647654 conn(0x7f91740c9160 msgr2=0x7f91740c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:09:33.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.488+0000 7f917ad4f640 1 -- 172.21.15.116:0/620647654 shutdown_connections 2024-04-25T00:09:33.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.488+0000 7f917ad4f640 1 -- 172.21.15.116:0/620647654 wait complete. 2024-04-25T00:09:33.508 INFO:tasks.mds_thrash.fs.[a]:no change 2024-04-25T00:09:33.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.580+0000 7f95a08fb640 1 -- 172.21.15.116:0/1157642044 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f959c0c30f0 con 0x7f959c0c9040 2024-04-25T00:09:33.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.580+0000 7f958affd640 1 -- 172.21.15.116:0/1157642044 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) ==== 94+0+4835 (secure 0 0 0) 0x7f9590055fc0 con 0x7f959c0c9040 2024-04-25T00:09:33.585 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:09:33.585 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:09:31.593888+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5236":{"gid":5236,"name":"b","rank":0,"incarnation":62,"state":"up:rejoin","state_seq":24,"addr":"172.21.15.174:6837/1159002662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1159002662},{"type":"v1","addr":"172.21.15.174:6837","nonce":1159002662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:09:31.595714+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"d","rank":0,"incarnation":62,"state":"up:rejoin","state_seq":131,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:09:33.586 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 64 2024-04-25T00:09:33.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.584+0000 7f95a08fb640 1 -- 172.21.15.116:0/1157642044 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f957c0678c0 msgr2=0x7f957c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:33.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.584+0000 7f95a08fb640 1 --2- 172.21.15.116:0/1157642044 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f957c0678c0 0x7f957c069d80 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f95940ae000 tx=0x7f95940a7070 comp rx=0 tx=0).stop 2024-04-25T00:09:33.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.584+0000 7f95a08fb640 1 -- 172.21.15.116:0/1157642044 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f959c0c9040 msgr2=0x7f959c150160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:33.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.584+0000 7f95a08fb640 1 --2- 172.21.15.116:0/1157642044 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f959c0c9040 0x7f959c150160 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f959000e7f0 tx=0x7f959000ecb0 comp rx=0 tx=0).stop 2024-04-25T00:09:33.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.584+0000 7f95a08fb640 1 -- 172.21.15.116:0/1157642044 shutdown_connections 2024-04-25T00:09:33.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.584+0000 7f95a08fb640 1 --2- 172.21.15.116:0/1157642044 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f957c0678c0 0x7f957c069d80 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:33.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.584+0000 7f95a08fb640 1 --2- 172.21.15.116:0/1157642044 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f959c0c9040 0x7f959c150160 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:33.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.584+0000 7f95a08fb640 1 --2- 172.21.15.116:0/1157642044 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f959c0c6e20 0x7f959c152040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:33.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.584+0000 7f95a08fb640 1 --2- 172.21.15.116:0/1157642044 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f959c06da40 0x7f959c0ad040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:33.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.584+0000 7f95a08fb640 1 -- 172.21.15.116:0/1157642044 >> 172.21.15.116:0/1157642044 conn(0x7f959c074480 msgr2=0x7f959c0710d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:09:33.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.584+0000 7f95a08fb640 1 -- 172.21.15.116:0/1157642044 shutdown_connections 2024-04-25T00:09:33.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:33.584+0000 7f95a08fb640 1 -- 172.21.15.116:0/1157642044 wait complete. 2024-04-25T00:09:33.602 INFO:tasks.mds_thrash.fs.[b]:no change 2024-04-25T00:09:34.754 INFO:tasks.ceph.mds.d.smithi174.stderr:2024-04-25T00:09:34.752+0000 7f6d0e02f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T00:09:35.509 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:09:35.603 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:09:35.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.644+0000 7fbb9b523640 1 Processor -- start 2024-04-25T00:09:35.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.644+0000 7fbb9b523640 1 -- start start 2024-04-25T00:09:35.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.644+0000 7fbb9b523640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fbb94150d20 0x7fbb94153620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:35.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.644+0000 7fbb9b523640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbb940c3240 0x7fbb940c3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:35.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.644+0000 7fbb9b523640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb940c4380 0x7fbb94071050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:35.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.644+0000 7fbb9b523640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fbb9406fba0 con 0x7fbb940c3240 2024-04-25T00:09:35.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.644+0000 7fbb9b523640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fbb940c36a0 con 0x7fbb94150d20 2024-04-25T00:09:35.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.644+0000 7fbb9b523640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fbb9406f8c0 con 0x7fbb940c4380 2024-04-25T00:09:35.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.644+0000 7fbb98abe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbb940c3240 0x7fbb940c3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:35.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.644+0000 7fbb98abe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbb940c3240 0x7fbb940c3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41328/0 (socket says 172.21.15.116:41328) 2024-04-25T00:09:35.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.644+0000 7fbb98abe640 1 -- 172.21.15.116:0/802222983 learned_addr learned my addr 172.21.15.116:0/802222983 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:09:35.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.644+0000 7fbb992bf640 1 --2- 172.21.15.116:0/802222983 >> v2:172.21.15.174:3300/0 conn(0x7fbb94150d20 0x7fbb94153620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:35.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.644+0000 7fbb99ac0640 1 --2- 172.21.15.116:0/802222983 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb940c4380 0x7fbb94071050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:35.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.644+0000 7fbb98abe640 1 -- 172.21.15.116:0/802222983 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb940c4380 msgr2=0x7fbb94071050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:09:35.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.644+0000 7fbb98abe640 1 --2- 172.21.15.116:0/802222983 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb940c4380 0x7fbb94071050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:35.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.644+0000 7fbb98abe640 1 -- 172.21.15.116:0/802222983 >> v2:172.21.15.174:3300/0 conn(0x7fbb94150d20 msgr2=0x7fbb94153620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:09:35.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.644+0000 7fbb98abe640 1 --2- 172.21.15.116:0/802222983 >> v2:172.21.15.174:3300/0 conn(0x7fbb94150d20 0x7fbb94153620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:35.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.644+0000 7fbb98abe640 1 -- 172.21.15.116:0/802222983 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb940c5420 con 0x7fbb940c3240 2024-04-25T00:09:35.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.644+0000 7fbb992bf640 1 --2- 172.21.15.116:0/802222983 >> v2:172.21.15.174:3300/0 conn(0x7fbb94150d20 0x7fbb94153620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:09:35.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.648+0000 7fbb98abe640 1 --2- 172.21.15.116:0/802222983 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb940c3240 0x7fbb940c3e40 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fbb940c6b50 tx=0x7fbb8401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4bc20986312f0b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:35.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.648+0000 7fbb8bfff640 1 -- 172.21.15.116:0/802222983 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb840ab5f0 con 0x7fbb940c3240 2024-04-25T00:09:35.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.648+0000 7fbb8bfff640 1 -- 172.21.15.116:0/802222983 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb840b0070 con 0x7fbb940c3240 2024-04-25T00:09:35.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.648+0000 7fbb8bfff640 1 -- 172.21.15.116:0/802222983 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb840ab9b0 con 0x7fbb940c3240 2024-04-25T00:09:35.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.648+0000 7fbb9b523640 1 -- 172.21.15.116:0/802222983 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb940c3240 msgr2=0x7fbb940c3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:35.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.648+0000 7fbb9b523640 1 --2- 172.21.15.116:0/802222983 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb940c3240 0x7fbb940c3e40 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fbb940c6b50 tx=0x7fbb8401c520 comp rx=0 tx=0).stop 2024-04-25T00:09:35.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.648+0000 7fbb9b523640 1 -- 172.21.15.116:0/802222983 shutdown_connections 2024-04-25T00:09:35.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.648+0000 7fbb9b523640 1 --2- 172.21.15.116:0/802222983 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb940c4380 0x7fbb94071050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:35.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.648+0000 7fbb9b523640 1 --2- 172.21.15.116:0/802222983 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb940c3240 0x7fbb940c3e40 secure :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fbb940c6b50 tx=0x7fbb8401c520 comp rx=0 tx=0).stop 2024-04-25T00:09:35.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.648+0000 7fbb9b523640 1 --2- 172.21.15.116:0/802222983 >> v2:172.21.15.174:3300/0 conn(0x7fbb94150d20 0x7fbb94153620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:35.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.648+0000 7fbb9b523640 1 -- 172.21.15.116:0/802222983 >> 172.21.15.116:0/802222983 conn(0x7fbb940c9160 msgr2=0x7fbb940c9570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:09:35.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.648+0000 7fbb9b523640 1 -- 172.21.15.116:0/802222983 shutdown_connections 2024-04-25T00:09:35.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.648+0000 7fbb9b523640 1 -- 172.21.15.116:0/802222983 wait complete. 2024-04-25T00:09:35.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.648+0000 7fbb9b523640 1 Processor -- start 2024-04-25T00:09:35.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.652+0000 7fbb9b523640 1 -- start start 2024-04-25T00:09:35.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.652+0000 7fbb9b523640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb940c4380 0x7fbb941469f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:35.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.652+0000 7fbb9b523640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb94150d20 0x7fbb94153590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:35.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.652+0000 7fbb9b523640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb941f0370 0x7fbb94149e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:35.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.652+0000 7fbb9b523640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbb94151280 con 0x7fbb941f0370 2024-04-25T00:09:35.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.652+0000 7fbb9b523640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbb94151100 con 0x7fbb94150d20 2024-04-25T00:09:35.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.652+0000 7fbb9b523640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fbb940c76a0 con 0x7fbb940c4380 2024-04-25T00:09:35.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.652+0000 7fbb99ac0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb941f0370 0x7fbb94149e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:35.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.652+0000 7fbb99ac0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb941f0370 0x7fbb94149e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41340/0 (socket says 172.21.15.116:41340) 2024-04-25T00:09:35.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.652+0000 7fbb99ac0640 1 -- 172.21.15.116:0/1299410589 learned_addr learned my addr 172.21.15.116:0/1299410589 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:09:35.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.652+0000 7fbb992bf640 1 --2- 172.21.15.116:0/1299410589 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb940c4380 0x7fbb941469f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:35.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.652+0000 7fbb98abe640 1 --2- 172.21.15.116:0/1299410589 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb94150d20 0x7fbb94153590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:35.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.652+0000 7fbb99ac0640 1 -- 172.21.15.116:0/1299410589 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb940c4380 msgr2=0x7fbb941469f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:35.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.652+0000 7fbb99ac0640 1 --2- 172.21.15.116:0/1299410589 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb940c4380 0x7fbb941469f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:35.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.652+0000 7fbb99ac0640 1 -- 172.21.15.116:0/1299410589 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb94150d20 msgr2=0x7fbb94153590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:35.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.652+0000 7fbb99ac0640 1 --2- 172.21.15.116:0/1299410589 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb94150d20 0x7fbb94153590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:35.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.652+0000 7fbb99ac0640 1 -- 172.21.15.116:0/1299410589 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb941f0820 con 0x7fbb941f0370 2024-04-25T00:09:35.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.652+0000 7fbb992bf640 1 --2- 172.21.15.116:0/1299410589 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb940c4380 0x7fbb941469f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:09:35.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.652+0000 7fbb99ac0640 1 --2- 172.21.15.116:0/1299410589 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb941f0370 0x7fbb94149e00 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c007e10 tx=0x7fbb8c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:35.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.652+0000 7fbb89ffb640 1 -- 172.21.15.116:0/1299410589 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb8c014620 con 0x7fbb941f0370 2024-04-25T00:09:35.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.652+0000 7fbb89ffb640 1 -- 172.21.15.116:0/1299410589 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb8c019070 con 0x7fbb941f0370 2024-04-25T00:09:35.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.652+0000 7fbb9b523640 1 -- 172.21.15.116:0/1299410589 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb94147040 con 0x7fbb941f0370 2024-04-25T00:09:35.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.652+0000 7fbb89ffb640 1 -- 172.21.15.116:0/1299410589 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb8c014a20 con 0x7fbb941f0370 2024-04-25T00:09:35.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.652+0000 7fbb9b523640 1 -- 172.21.15.116:0/1299410589 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb94152d90 con 0x7fbb941f0370 2024-04-25T00:09:35.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.652+0000 7fbb9b523640 1 -- 172.21.15.116:0/1299410589 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb94068a60 con 0x7fbb941f0370 2024-04-25T00:09:35.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.652+0000 7fbb89ffb640 1 -- 172.21.15.116:0/1299410589 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbb8c014bc0 con 0x7fbb941f0370 2024-04-25T00:09:35.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.656+0000 7fbb89ffb640 1 --2- 172.21.15.116:0/1299410589 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbb780678f0 0x7fbb78069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:35.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.656+0000 7fbb89ffb640 1 -- 172.21.15.116:0/1299410589 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7612+0+0 (secure 0 0 0) 0x7fbb8c086b90 con 0x7fbb941f0370 2024-04-25T00:09:35.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.656+0000 7fbb992bf640 1 --2- 172.21.15.116:0/1299410589 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbb780678f0 0x7fbb78069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:35.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.656+0000 7fbb992bf640 1 --2- 172.21.15.116:0/1299410589 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbb780678f0 0x7fbb78069db0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fbb90007950 tx=0x7fbb90008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:35.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.660+0000 7fbb89ffb640 1 -- 172.21.15.116:0/1299410589 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fbb8c056f50 con 0x7fbb941f0370 2024-04-25T00:09:35.694 INFO:tasks.ceph.mds.b.smithi174.stderr:2024-04-25T00:09:35.692+0000 7fbd4310f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T00:09:35.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.728+0000 7f1191fd3640 1 Processor -- start 2024-04-25T00:09:35.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.728+0000 7f1191fd3640 1 -- start start 2024-04-25T00:09:35.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.728+0000 7f1191fd3640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f118c155c00 0x7f118c071520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:35.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.728+0000 7f1191fd3640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f118c0730c0 0x7f118c155fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:35.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.728+0000 7f1191fd3640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f118c072920 con 0x7f118c073540 2024-04-25T00:09:35.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.728+0000 7f1191fd3640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f118c071bb0 con 0x7f118c155c00 2024-04-25T00:09:35.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.728+0000 7f1191fd3640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f118c072ee0 con 0x7f118c0730c0 2024-04-25T00:09:35.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.728+0000 7f118b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f118c155c00 0x7f118c071520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:35.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.728+0000 7f118bfff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f118c073540 0x7f118c14f7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:44196/0 (socket says 172.21.15.116:44196) 2024-04-25T00:09:35.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.728+0000 7f118bfff640 1 -- 172.21.15.116:0/3341703070 learned_addr learned my addr 172.21.15.116:0/3341703070 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:09:35.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.732+0000 7f118affd640 1 --2- 172.21.15.116:0/3341703070 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f118c0730c0 0x7f118c155fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:35.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.732+0000 7f118b7fe640 1 -- 172.21.15.116:0/3341703070 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f118c0730c0 msgr2=0x7f118c155fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:09:35.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.732+0000 7f118b7fe640 1 --2- 172.21.15.116:0/3341703070 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f118c0730c0 0x7f118c155fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:35.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.732+0000 7f118b7fe640 1 -- 172.21.15.116:0/3341703070 >> v1:172.21.15.116:6789/0 conn(0x7f118c073540 legacy=0x7f118c14f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:35.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.732+0000 7f118b7fe640 1 -- 172.21.15.116:0/3341703070 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f118c072c00 con 0x7f118c155c00 2024-04-25T00:09:35.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.732+0000 7f118a7fc640 1 -- 172.21.15.116:0/3341703070 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3955094472 0 0) 0x7f118c072920 con 0x7f118c073540 2024-04-25T00:09:35.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.732+0000 7f118b7fe640 1 --2- 172.21.15.116:0/3341703070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f118c155c00 0x7f118c071520 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f117c0097c0 tx=0x7f117c01c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be79b0cdd4a9561e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:35.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.732+0000 7f118a7fc640 1 -- 172.21.15.116:0/3341703070 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f117c0b0070 con 0x7f118c155c00 2024-04-25T00:09:35.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.732+0000 7f118a7fc640 1 -- 172.21.15.116:0/3341703070 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f117c0ab600 con 0x7f118c155c00 2024-04-25T00:09:35.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.732+0000 7f118a7fc640 1 -- 172.21.15.116:0/3341703070 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f117c0ab9d0 con 0x7f118c155c00 2024-04-25T00:09:35.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.732+0000 7f1191fd3640 1 -- 172.21.15.116:0/3341703070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f118c155c00 msgr2=0x7f118c071520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:35.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.732+0000 7f1191fd3640 1 --2- 172.21.15.116:0/3341703070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f118c155c00 0x7f118c071520 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f117c0097c0 tx=0x7f117c01c380 comp rx=0 tx=0).stop 2024-04-25T00:09:35.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.732+0000 7f1191fd3640 1 -- 172.21.15.116:0/3341703070 shutdown_connections 2024-04-25T00:09:35.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.732+0000 7f1191fd3640 1 --2- 172.21.15.116:0/3341703070 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f118c0730c0 0x7f118c155fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:35.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.732+0000 7f1191fd3640 1 --2- 172.21.15.116:0/3341703070 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f118c155c00 0x7f118c071520 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:35.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.732+0000 7f1191fd3640 1 -- 172.21.15.116:0/3341703070 >> 172.21.15.116:0/3341703070 conn(0x7f118c0c4200 msgr2=0x7f118c0c4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:09:35.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.732+0000 7f1191fd3640 1 -- 172.21.15.116:0/3341703070 shutdown_connections 2024-04-25T00:09:35.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.732+0000 7f1191fd3640 1 -- 172.21.15.116:0/3341703070 wait complete. 2024-04-25T00:09:35.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.736+0000 7f1191fd3640 1 Processor -- start 2024-04-25T00:09:35.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.736+0000 7f1191fd3640 1 -- start start 2024-04-25T00:09:35.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.736+0000 7f1191fd3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f118c0730c0 0x7f118c156ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:35.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.736+0000 7f1191fd3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f118c073540 0x7f118c1f1620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:35.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.736+0000 7f1191fd3640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f118c155c00 0x7f118c1f1b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:35.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.736+0000 7f1191fd3640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f118c0c2280 con 0x7f118c0730c0 2024-04-25T00:09:35.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.736+0000 7f1191fd3640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f118c0c2100 con 0x7f118c073540 2024-04-25T00:09:35.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.736+0000 7f1191fd3640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f118c156520 con 0x7f118c155c00 2024-04-25T00:09:35.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.736+0000 7f118b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f118c0730c0 0x7f118c156ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:35.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.736+0000 7f118b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f118c0730c0 0x7f118c156ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41352/0 (socket says 172.21.15.116:41352) 2024-04-25T00:09:35.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.736+0000 7f118b7fe640 1 -- 172.21.15.116:0/1332869471 learned_addr learned my addr 172.21.15.116:0/1332869471 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:09:35.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.736+0000 7f118bfff640 1 --2- 172.21.15.116:0/1332869471 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f118c155c00 0x7f118c1f1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:35.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.736+0000 7f118affd640 1 --2- 172.21.15.116:0/1332869471 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f118c073540 0x7f118c1f1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:35.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.736+0000 7f118b7fe640 1 -- 172.21.15.116:0/1332869471 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f118c155c00 msgr2=0x7f118c1f1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:35.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.736+0000 7f118b7fe640 1 --2- 172.21.15.116:0/1332869471 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f118c155c00 0x7f118c1f1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:35.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.736+0000 7f118b7fe640 1 -- 172.21.15.116:0/1332869471 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f118c073540 msgr2=0x7f118c1f1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:35.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.736+0000 7f118b7fe640 1 --2- 172.21.15.116:0/1332869471 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f118c073540 0x7f118c1f1620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:35.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.736+0000 7f118b7fe640 1 -- 172.21.15.116:0/1332869471 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f118c1f20a0 con 0x7f118c0730c0 2024-04-25T00:09:35.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.736+0000 7f118affd640 1 --2- 172.21.15.116:0/1332869471 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f118c073540 0x7f118c1f1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:09:35.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.736+0000 7f118b7fe640 1 --2- 172.21.15.116:0/1332869471 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f118c0730c0 0x7f118c156ce0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f117c0ae040 tx=0x7f117c01abc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:35.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.736+0000 7f1173fff640 1 -- 172.21.15.116:0/1332869471 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f117c0b0050 con 0x7f118c0730c0 2024-04-25T00:09:35.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.736+0000 7f1173fff640 1 -- 172.21.15.116:0/1332869471 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f117c0bc070 con 0x7f118c0730c0 2024-04-25T00:09:35.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.736+0000 7f1191fd3640 1 -- 172.21.15.116:0/1332869471 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f118c0c1670 con 0x7f118c0730c0 2024-04-25T00:09:35.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.736+0000 7f1173fff640 1 -- 172.21.15.116:0/1332869471 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f117c0b5850 con 0x7f118c0730c0 2024-04-25T00:09:35.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.736+0000 7f1191fd3640 1 -- 172.21.15.116:0/1332869471 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f118c0aaa00 con 0x7f118c0730c0 2024-04-25T00:09:35.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.736+0000 7f1191fd3640 1 -- 172.21.15.116:0/1332869471 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f118c068a60 con 0x7f118c0730c0 2024-04-25T00:09:35.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.740+0000 7f1173fff640 1 -- 172.21.15.116:0/1332869471 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f117c0b59f0 con 0x7f118c0730c0 2024-04-25T00:09:35.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.740+0000 7f1173fff640 1 --2- 172.21.15.116:0/1332869471 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f11780678f0 0x7f1178069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:35.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.740+0000 7f1173fff640 1 -- 172.21.15.116:0/1332869471 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7612+0+0 (secure 0 0 0) 0x7f117c1209d0 con 0x7f118c0730c0 2024-04-25T00:09:35.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.740+0000 7f118affd640 1 --2- 172.21.15.116:0/1332869471 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f11780678f0 0x7f1178069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:35.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.740+0000 7f118affd640 1 --2- 172.21.15.116:0/1332869471 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f11780678f0 0x7f1178069db0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f118c155610 tx=0x7f1180008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:35.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.744+0000 7f1173fff640 1 -- 172.21.15.116:0/1332869471 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f118c1f20a0 con 0x7f118c0730c0 2024-04-25T00:09:35.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.912+0000 7fbb9b523640 1 -- 172.21.15.116:0/1299410589 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbb940c4790 con 0x7fbb941f0370 2024-04-25T00:09:35.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.912+0000 7fbb89ffb640 1 -- 172.21.15.116:0/1299410589 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) ==== 94+0+4849 (secure 0 0 0) 0x7fbb8c056850 con 0x7fbb941f0370 2024-04-25T00:09:35.916 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:09:35.916 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:09:33.601522+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5236":{"gid":5236,"name":"b","rank":0,"incarnation":62,"state":"up:active","state_seq":25,"addr":"172.21.15.174:6837/1159002662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1159002662},{"type":"v1","addr":"172.21.15.174:6837","nonce":1159002662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5236,"qdb_cluster":[5236]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:09:33.601524+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":132,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4235,"qdb_cluster":[4235]},"id":2}]} 2024-04-25T00:09:35.917 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 65 2024-04-25T00:09:35.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.916+0000 7fbb9b523640 1 -- 172.21.15.116:0/1299410589 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbb780678f0 msgr2=0x7fbb78069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:35.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.916+0000 7fbb9b523640 1 --2- 172.21.15.116:0/1299410589 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbb780678f0 0x7fbb78069db0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fbb90007950 tx=0x7fbb90008040 comp rx=0 tx=0).stop 2024-04-25T00:09:35.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.916+0000 7fbb9b523640 1 -- 172.21.15.116:0/1299410589 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb941f0370 msgr2=0x7fbb94149e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:35.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.916+0000 7fbb9b523640 1 --2- 172.21.15.116:0/1299410589 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb941f0370 0x7fbb94149e00 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c007e10 tx=0x7fbb8c00e4b0 comp rx=0 tx=0).stop 2024-04-25T00:09:35.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.916+0000 7fbb9b523640 1 -- 172.21.15.116:0/1299410589 shutdown_connections 2024-04-25T00:09:35.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.916+0000 7fbb9b523640 1 --2- 172.21.15.116:0/1299410589 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbb780678f0 0x7fbb78069db0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:35.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.916+0000 7fbb9b523640 1 --2- 172.21.15.116:0/1299410589 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb941f0370 0x7fbb94149e00 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:35.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.916+0000 7fbb9b523640 1 --2- 172.21.15.116:0/1299410589 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb94150d20 0x7fbb94153590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:35.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.916+0000 7fbb9b523640 1 --2- 172.21.15.116:0/1299410589 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb940c4380 0x7fbb941469f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:35.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.916+0000 7fbb9b523640 1 -- 172.21.15.116:0/1299410589 >> 172.21.15.116:0/1299410589 conn(0x7fbb940c9160 msgr2=0x7fbb9415d640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:09:35.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.916+0000 7fbb9b523640 1 -- 172.21.15.116:0/1299410589 shutdown_connections 2024-04-25T00:09:35.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.916+0000 7fbb9b523640 1 -- 172.21.15.116:0/1299410589 wait complete. 2024-04-25T00:09:35.938 INFO:tasks.mds_thrash.fs.[a]:mds.b has gained rank=0, replacing gid=5775 2024-04-25T00:09:35.939 INFO:tasks.mds_thrash.fs.[a]:waiting for 5 secs before reviving mds.c 2024-04-25T00:09:35.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.988+0000 7f1191fd3640 1 -- 172.21.15.116:0/1332869471 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f118c0a8f30 con 0x7f118c0730c0 2024-04-25T00:09:35.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.988+0000 7f1173fff640 1 -- 172.21.15.116:0/1332869471 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) ==== 94+0+4849 (secure 0 0 0) 0x7f117c016090 con 0x7f118c0730c0 2024-04-25T00:09:35.992 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:09:35.992 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:09:33.601522+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5236":{"gid":5236,"name":"b","rank":0,"incarnation":62,"state":"up:active","state_seq":25,"addr":"172.21.15.174:6837/1159002662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1159002662},{"type":"v1","addr":"172.21.15.174:6837","nonce":1159002662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5236,"qdb_cluster":[5236]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:09:33.601524+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":132,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4235,"qdb_cluster":[4235]},"id":2}]} 2024-04-25T00:09:35.992 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 65 2024-04-25T00:09:35.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.992+0000 7f1191fd3640 1 -- 172.21.15.116:0/1332869471 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f11780678f0 msgr2=0x7f1178069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:35.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.992+0000 7f1191fd3640 1 --2- 172.21.15.116:0/1332869471 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f11780678f0 0x7f1178069db0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f118c155610 tx=0x7f1180008040 comp rx=0 tx=0).stop 2024-04-25T00:09:35.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.992+0000 7f1191fd3640 1 -- 172.21.15.116:0/1332869471 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f118c0730c0 msgr2=0x7f118c156ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:35.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.992+0000 7f1191fd3640 1 --2- 172.21.15.116:0/1332869471 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f118c0730c0 0x7f118c156ce0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f117c0ae040 tx=0x7f117c01abc0 comp rx=0 tx=0).stop 2024-04-25T00:09:35.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.992+0000 7f1191fd3640 1 -- 172.21.15.116:0/1332869471 shutdown_connections 2024-04-25T00:09:35.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.992+0000 7f1191fd3640 1 --2- 172.21.15.116:0/1332869471 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f11780678f0 0x7f1178069db0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:35.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.992+0000 7f1191fd3640 1 --2- 172.21.15.116:0/1332869471 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f118c155c00 0x7f118c1f1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:35.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.992+0000 7f1191fd3640 1 --2- 172.21.15.116:0/1332869471 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f118c073540 0x7f118c1f1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:35.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.992+0000 7f1191fd3640 1 --2- 172.21.15.116:0/1332869471 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f118c0730c0 0x7f118c156ce0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:35.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.992+0000 7f1191fd3640 1 -- 172.21.15.116:0/1332869471 >> 172.21.15.116:0/1332869471 conn(0x7f118c0c4200 msgr2=0x7f118c0a9230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:09:35.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.992+0000 7f1191fd3640 1 -- 172.21.15.116:0/1332869471 shutdown_connections 2024-04-25T00:09:35.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:35.992+0000 7f1191fd3640 1 -- 172.21.15.116:0/1332869471 wait complete. 2024-04-25T00:09:36.012 INFO:tasks.mds_thrash.fs.[b]:mds.d has gained rank=0, replacing gid=5868 2024-04-25T00:09:36.012 INFO:tasks.mds_thrash.fs.[b]:waiting for 0 secs before reviving mds.a 2024-04-25T00:09:36.012 INFO:tasks.mds_thrash.fs.[b]:reviving mds.a 2024-04-25T00:09:36.012 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-25T00:09:36.012 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-25T00:09:36.014 INFO:tasks.ceph.mds.a:Started 2024-04-25T00:09:36.014 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:09:36.084 INFO:tasks.ceph.mds.a.smithi116.stdout:starting mds.a at 2024-04-25T00:09:36.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.140+0000 7f5cec3a2640 1 Processor -- start 2024-04-25T00:09:36.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.140+0000 7f5cec3a2640 1 -- start start 2024-04-25T00:09:36.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.140+0000 7f5cec3a2640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5ce40c4c20 0x7f5ce4154170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:36.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.140+0000 7f5cec3a2640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f5ce4070770 0x7f5ce40ab200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:36.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.140+0000 7f5cec3a2640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ce4070180 con 0x7f5ce4150d40 2024-04-25T00:09:36.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.140+0000 7f5cec3a2640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f5ce40c7530 con 0x7f5ce4070770 2024-04-25T00:09:36.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.140+0000 7f5cec3a2640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f5ce4071280 con 0x7f5ce40c4c20 2024-04-25T00:09:36.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.140+0000 7f5cea13e640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f5ce4150d40 0x7f5ce40c6e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:44212/0 (socket says 172.21.15.116:44212) 2024-04-25T00:09:36.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.140+0000 7f5cea13e640 1 -- 172.21.15.116:0/3801229105 learned_addr learned my addr 172.21.15.116:0/3801229105 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:09:36.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.140+0000 7f5ce993d640 1 --2- 172.21.15.116:0/3801229105 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5ce40c4c20 0x7f5ce4154170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:36.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.140+0000 7f5cea93f640 1 --2- 172.21.15.116:0/3801229105 >> v2:172.21.15.174:3300/0 conn(0x7f5ce4070770 0x7f5ce40ab200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:36.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.140+0000 7f5ce913c640 1 -- 172.21.15.116:0/3801229105 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1582462035 0 0) 0x7f5ce4070180 con 0x7f5ce4150d40 2024-04-25T00:09:36.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.140+0000 7f5ce913c640 1 -- 172.21.15.116:0/3801229105 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5cd0003530 con 0x7f5ce4150d40 2024-04-25T00:09:36.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.144+0000 7f5ce913c640 1 -- 172.21.15.116:0/3801229105 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1181561421 0 0) 0x7f5cd0003530 con 0x7f5ce4150d40 2024-04-25T00:09:36.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.144+0000 7f5cea93f640 1 -- 172.21.15.116:0/3801229105 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5ce40c4c20 msgr2=0x7f5ce4154170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:09:36.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.144+0000 7f5cea93f640 1 --2- 172.21.15.116:0/3801229105 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5ce40c4c20 0x7f5ce4154170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:36.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.144+0000 7f5cea93f640 1 -- 172.21.15.116:0/3801229105 >> v1:172.21.15.116:6789/0 conn(0x7f5ce4150d40 legacy=0x7f5ce40c6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:36.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.144+0000 7f5cea93f640 1 -- 172.21.15.116:0/3801229105 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ce4070460 con 0x7f5ce4070770 2024-04-25T00:09:36.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.144+0000 7f5cea93f640 1 --2- 172.21.15.116:0/3801229105 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ce4070770 0x7f5ce40ab200 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc009dc0 tx=0x7f5cdc01c7e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d2b975cdd0f0984 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:36.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.144+0000 7f5ce913c640 1 -- 172.21.15.116:0/3801229105 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cdc0b2070 con 0x7f5ce4070770 2024-04-25T00:09:36.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.144+0000 7f5ce913c640 1 -- 172.21.15.116:0/3801229105 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cdc0a5530 con 0x7f5ce4070770 2024-04-25T00:09:36.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.144+0000 7f5ce913c640 1 -- 172.21.15.116:0/3801229105 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce0001680 con 0x7f5ce4070770 2024-04-25T00:09:36.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.144+0000 7f5cec3a2640 1 -- 172.21.15.116:0/3801229105 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ce4070770 msgr2=0x7f5ce40ab200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:36.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.144+0000 7f5cec3a2640 1 --2- 172.21.15.116:0/3801229105 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ce4070770 0x7f5ce40ab200 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc009dc0 tx=0x7f5cdc01c7e0 comp rx=0 tx=0).stop 2024-04-25T00:09:36.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.144+0000 7f5cec3a2640 1 -- 172.21.15.116:0/3801229105 shutdown_connections 2024-04-25T00:09:36.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.144+0000 7f5cec3a2640 1 --2- 172.21.15.116:0/3801229105 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ce4070770 0x7f5ce40ab200 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:36.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.144+0000 7f5cec3a2640 1 --2- 172.21.15.116:0/3801229105 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5ce40c4c20 0x7f5ce4154170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:36.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.144+0000 7f5cec3a2640 1 -- 172.21.15.116:0/3801229105 >> 172.21.15.116:0/3801229105 conn(0x7f5ce406f0c0 msgr2=0x7f5ce406f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:09:36.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.144+0000 7f5cec3a2640 1 -- 172.21.15.116:0/3801229105 shutdown_connections 2024-04-25T00:09:36.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.144+0000 7f5cec3a2640 1 -- 172.21.15.116:0/3801229105 wait complete. 2024-04-25T00:09:36.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.148+0000 7f5cec3a2640 1 Processor -- start 2024-04-25T00:09:36.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.148+0000 7f5cec3a2640 1 -- start start 2024-04-25T00:09:36.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.148+0000 7f5cec3a2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ce4070770 0x7f5ce40ad8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:36.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.148+0000 7f5cec3a2640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5ce40c4c20 0x7f5ce41ee460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:36.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.148+0000 7f5cec3a2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ce4150d40 0x7f5ce41ee9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:36.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.148+0000 7f5cec3a2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5ce4151120 con 0x7f5ce4070770 2024-04-25T00:09:36.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.148+0000 7f5cec3a2640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5ce406ff30 con 0x7f5ce4150d40 2024-04-25T00:09:36.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.148+0000 7f5cec3a2640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f5ce41512a0 con 0x7f5ce40c4c20 2024-04-25T00:09:36.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.148+0000 7f5cea13e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ce4070770 0x7f5ce40ad8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:36.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.148+0000 7f5cea13e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ce4070770 0x7f5ce40ad8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41388/0 (socket says 172.21.15.116:41388) 2024-04-25T00:09:36.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.148+0000 7f5cea13e640 1 -- 172.21.15.116:0/759880268 learned_addr learned my addr 172.21.15.116:0/759880268 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:09:36.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.148+0000 7f5cea93f640 1 --2- 172.21.15.116:0/759880268 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ce4150d40 0x7f5ce41ee9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:36.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.148+0000 7f5ce993d640 1 --2- 172.21.15.116:0/759880268 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5ce40c4c20 0x7f5ce41ee460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:36.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.148+0000 7f5cea13e640 1 -- 172.21.15.116:0/759880268 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5ce40c4c20 msgr2=0x7f5ce41ee460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:36.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.148+0000 7f5cea13e640 1 --2- 172.21.15.116:0/759880268 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5ce40c4c20 0x7f5ce41ee460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:36.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.148+0000 7f5cea13e640 1 -- 172.21.15.116:0/759880268 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ce4150d40 msgr2=0x7f5ce41ee9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:36.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.148+0000 7f5cea13e640 1 --2- 172.21.15.116:0/759880268 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ce4150d40 0x7f5ce41ee9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:36.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.148+0000 7f5cea13e640 1 -- 172.21.15.116:0/759880268 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ce41eeee0 con 0x7f5ce4070770 2024-04-25T00:09:36.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.148+0000 7f5cea93f640 1 --2- 172.21.15.116:0/759880268 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ce4150d40 0x7f5ce41ee9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:09:36.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.148+0000 7f5cea13e640 1 --2- 172.21.15.116:0/759880268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ce4070770 0x7f5ce40ad8d0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f5ce000c750 tx=0x7f5ce000cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:36.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.148+0000 7f5cdaffd640 1 -- 172.21.15.116:0/759880268 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce0005ae0 con 0x7f5ce4070770 2024-04-25T00:09:36.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.148+0000 7f5cdaffd640 1 -- 172.21.15.116:0/759880268 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ce0005ca0 con 0x7f5ce4070770 2024-04-25T00:09:36.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.148+0000 7f5cdaffd640 1 -- 172.21.15.116:0/759880268 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce00072b0 con 0x7f5ce4070770 2024-04-25T00:09:36.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.148+0000 7f5cec3a2640 1 -- 172.21.15.116:0/759880268 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ce40715e0 con 0x7f5ce4070770 2024-04-25T00:09:36.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.148+0000 7f5cec3a2640 1 -- 172.21.15.116:0/759880268 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ce41f2130 con 0x7f5ce4070770 2024-04-25T00:09:36.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.152+0000 7f5cec3a2640 1 -- 172.21.15.116:0/759880268 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ce4068a60 con 0x7f5ce4070770 2024-04-25T00:09:36.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.152+0000 7f5cdaffd640 1 -- 172.21.15.116:0/759880268 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5ce0007450 con 0x7f5ce4070770 2024-04-25T00:09:36.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.152+0000 7f5cdaffd640 1 --2- 172.21.15.116:0/759880268 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5cc80678c0 0x7f5cc8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:36.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.152+0000 7f5cdaffd640 1 -- 172.21.15.116:0/759880268 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7612+0+0 (secure 0 0 0) 0x7f5ce0086340 con 0x7f5ce4070770 2024-04-25T00:09:36.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.152+0000 7f5ce993d640 1 --2- 172.21.15.116:0/759880268 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5cc80678c0 0x7f5cc8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:36.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.152+0000 7f5ce993d640 1 --2- 172.21.15.116:0/759880268 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5cc80678c0 0x7f5cc8069d80 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f5ce4153330 tx=0x7f5cd4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:36.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.156+0000 7f5cdaffd640 1 -- 172.21.15.116:0/759880268 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f5ce41eeee0 con 0x7f5ce4070770 2024-04-25T00:09:36.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.364+0000 7f5cec3a2640 1 -- 172.21.15.116:0/759880268 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5ce4070d40 con 0x7f5ce4070770 2024-04-25T00:09:36.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.364+0000 7f5cdaffd640 1 -- 172.21.15.116:0/759880268 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) ==== 94+0+4849 (secure 0 0 0) 0x7f5ce0056720 con 0x7f5ce4070770 2024-04-25T00:09:36.367 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:09:36.367 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:09:33.601522+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5236":{"gid":5236,"name":"b","rank":0,"incarnation":62,"state":"up:active","state_seq":25,"addr":"172.21.15.174:6837/1159002662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1159002662},{"type":"v1","addr":"172.21.15.174:6837","nonce":1159002662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5236,"qdb_cluster":[5236]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:09:33.601524+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":132,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4235,"qdb_cluster":[4235]},"id":2}]} 2024-04-25T00:09:36.367 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 65 2024-04-25T00:09:36.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.368+0000 7f5cec3a2640 1 -- 172.21.15.116:0/759880268 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5cc80678c0 msgr2=0x7f5cc8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:36.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.368+0000 7f5cec3a2640 1 --2- 172.21.15.116:0/759880268 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5cc80678c0 0x7f5cc8069d80 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f5ce4153330 tx=0x7f5cd4008040 comp rx=0 tx=0).stop 2024-04-25T00:09:36.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.368+0000 7f5cec3a2640 1 -- 172.21.15.116:0/759880268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ce4070770 msgr2=0x7f5ce40ad8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:36.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.368+0000 7f5cec3a2640 1 --2- 172.21.15.116:0/759880268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ce4070770 0x7f5ce40ad8d0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f5ce000c750 tx=0x7f5ce000cc10 comp rx=0 tx=0).stop 2024-04-25T00:09:36.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.368+0000 7f5cec3a2640 1 -- 172.21.15.116:0/759880268 shutdown_connections 2024-04-25T00:09:36.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.368+0000 7f5cec3a2640 1 --2- 172.21.15.116:0/759880268 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5cc80678c0 0x7f5cc8069d80 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:36.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.368+0000 7f5cec3a2640 1 --2- 172.21.15.116:0/759880268 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ce4150d40 0x7f5ce41ee9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:36.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.368+0000 7f5cec3a2640 1 --2- 172.21.15.116:0/759880268 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5ce40c4c20 0x7f5ce41ee460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:36.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.368+0000 7f5cec3a2640 1 --2- 172.21.15.116:0/759880268 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ce4070770 0x7f5ce40ad8d0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:36.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.368+0000 7f5cec3a2640 1 -- 172.21.15.116:0/759880268 >> 172.21.15.116:0/759880268 conn(0x7f5ce406f0c0 msgr2=0x7f5ce40c59b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:09:36.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.368+0000 7f5cec3a2640 1 -- 172.21.15.116:0/759880268 shutdown_connections 2024-04-25T00:09:36.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:36.368+0000 7f5cec3a2640 1 -- 172.21.15.116:0/759880268 wait complete. 2024-04-25T00:09:36.392 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2024-04-25T00:09:36.780 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:09:36.782 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:09:38.393 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:09:38.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.516+0000 7ff8e98ab640 1 Processor -- start 2024-04-25T00:09:38.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.516+0000 7ff8e98ab640 1 -- start start 2024-04-25T00:09:38.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.520+0000 7ff8e98ab640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff8e4072080 0x7ff8e4159950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:38.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.520+0000 7ff8e98ab640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff8e41562a0 0x7ff8e40c5020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:38.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.520+0000 7ff8e98ab640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8e40730b0 con 0x7ff8e4152d80 2024-04-25T00:09:38.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.520+0000 7ff8e98ab640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7ff8e4072e60 con 0x7ff8e4072080 2024-04-25T00:09:38.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.520+0000 7ff8e98ab640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff8e4073390 con 0x7ff8e41562a0 2024-04-25T00:09:38.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.520+0000 7ff8e2ffd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7ff8e4152d80 0x7ff8e40c36b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:44228/0 (socket says 172.21.15.116:44228) 2024-04-25T00:09:38.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.520+0000 7ff8e2ffd640 1 -- 172.21.15.116:0/210448906 learned_addr learned my addr 172.21.15.116:0/210448906 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:09:38.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.520+0000 7ff8e37fe640 1 --2- 172.21.15.116:0/210448906 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff8e41562a0 0x7ff8e40c5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:38.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.520+0000 7ff8dbfff640 1 --2- 172.21.15.116:0/210448906 >> v2:172.21.15.174:3300/0 conn(0x7ff8e4072080 0x7ff8e4159950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:38.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.520+0000 7ff8e27fc640 1 -- 172.21.15.116:0/210448906 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3288499785 0 0) 0x7ff8e40730b0 con 0x7ff8e4152d80 2024-04-25T00:09:38.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.520+0000 7ff8e27fc640 1 -- 172.21.15.116:0/210448906 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff8d0003530 con 0x7ff8e4152d80 2024-04-25T00:09:38.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.520+0000 7ff8e27fc640 1 -- 172.21.15.116:0/210448906 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3573653830 0 0) 0x7ff8d0003530 con 0x7ff8e4152d80 2024-04-25T00:09:38.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.520+0000 7ff8e27fc640 1 -- 172.21.15.116:0/210448906 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff8e41562a0 msgr2=0x7ff8e40c5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:09:38.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.520+0000 7ff8e27fc640 1 --2- 172.21.15.116:0/210448906 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff8e41562a0 0x7ff8e40c5020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:38.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.520+0000 7ff8e27fc640 1 -- 172.21.15.116:0/210448906 >> v2:172.21.15.174:3300/0 conn(0x7ff8e4072080 msgr2=0x7ff8e4159950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:09:38.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.520+0000 7ff8e27fc640 1 --2- 172.21.15.116:0/210448906 >> v2:172.21.15.174:3300/0 conn(0x7ff8e4072080 0x7ff8e4159950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:38.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.520+0000 7ff8e27fc640 1 -- 172.21.15.116:0/210448906 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8e4158e70 con 0x7ff8e4152d80 2024-04-25T00:09:38.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.520+0000 7ff8e27fc640 1 -- 172.21.15.116:0/210448906 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7ff8d4001680 con 0x7ff8e4152d80 2024-04-25T00:09:38.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.520+0000 7ff8e27fc640 1 -- 172.21.15.116:0/210448906 >> v1:172.21.15.116:6789/0 conn(0x7ff8e4152d80 legacy=0x7ff8e40c36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:38.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.520+0000 7ff8e27fc640 1 --2- 172.21.15.116:0/210448906 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff8e41562a0 0x7ff8d00039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:38.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.520+0000 7ff8e27fc640 1 --2- 172.21.15.116:0/210448906 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff8d0003f90 0x7ff8d0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:38.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.520+0000 7ff8e27fc640 1 --2- 172.21.15.116:0/210448906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8d0004d50 0x7ff8d00051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:38.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.520+0000 7ff8e27fc640 1 -- 172.21.15.116:0/210448906 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e4072e60 con 0x7ff8d0004d50 2024-04-25T00:09:38.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.520+0000 7ff8e27fc640 1 -- 172.21.15.116:0/210448906 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e4073390 con 0x7ff8d0003f90 2024-04-25T00:09:38.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.520+0000 7ff8e27fc640 1 -- 172.21.15.116:0/210448906 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff8d000ba80 con 0x7ff8e41562a0 2024-04-25T00:09:38.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.520+0000 7ff8e37fe640 1 --2- 172.21.15.116:0/210448906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8d0004d50 0x7ff8d00051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:38.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.520+0000 7ff8e2ffd640 1 --2- 172.21.15.116:0/210448906 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff8e41562a0 0x7ff8d00039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:38.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.520+0000 7ff8dbfff640 1 --2- 172.21.15.116:0/210448906 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff8d0003f90 0x7ff8d0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:38.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.520+0000 7ff8e37fe640 1 -- 172.21.15.116:0/210448906 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff8e41562a0 msgr2=0x7ff8d00039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:09:38.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.520+0000 7ff8e37fe640 1 --2- 172.21.15.116:0/210448906 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff8e41562a0 0x7ff8d00039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:38.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.520+0000 7ff8e37fe640 1 -- 172.21.15.116:0/210448906 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff8d0003f90 msgr2=0x7ff8d0004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:09:38.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.520+0000 7ff8e37fe640 1 --2- 172.21.15.116:0/210448906 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff8d0003f90 0x7ff8d0004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:38.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.520+0000 7ff8e37fe640 1 -- 172.21.15.116:0/210448906 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8d0003530 con 0x7ff8d0004d50 2024-04-25T00:09:38.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.520+0000 7ff8dbfff640 1 --2- 172.21.15.116:0/210448906 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff8d0003f90 0x7ff8d0004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:09:38.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.524+0000 7ff8e37fe640 1 --2- 172.21.15.116:0/210448906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8d0004d50 0x7ff8d00051d0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc009fc0 tx=0x7ff8dc01cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccb7df2fc45f34b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:38.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.524+0000 7ff8e27fc640 1 -- 172.21.15.116:0/210448906 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8dc0b2070 con 0x7ff8d0004d50 2024-04-25T00:09:38.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.524+0000 7ff8e27fc640 1 -- 172.21.15.116:0/210448906 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8dc0a5c90 con 0x7ff8d0004d50 2024-04-25T00:09:38.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.524+0000 7ff8e27fc640 1 -- 172.21.15.116:0/210448906 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8dc009c10 con 0x7ff8d0004d50 2024-04-25T00:09:38.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.524+0000 7ff8e98ab640 1 -- 172.21.15.116:0/210448906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8d0004d50 msgr2=0x7ff8d00051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:38.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.524+0000 7ff8e98ab640 1 --2- 172.21.15.116:0/210448906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8d0004d50 0x7ff8d00051d0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc009fc0 tx=0x7ff8dc01cc80 comp rx=0 tx=0).stop 2024-04-25T00:09:38.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.524+0000 7ff8e37fe640 1 -- 172.21.15.116:0/210448906 reap_dead start 2024-04-25T00:09:38.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.524+0000 7ff8e98ab640 1 -- 172.21.15.116:0/210448906 shutdown_connections 2024-04-25T00:09:38.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.524+0000 7ff8e98ab640 1 -- 172.21.15.116:0/210448906 >> 172.21.15.116:0/210448906 conn(0x7ff8e406c720 msgr2=0x7ff8e406cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:09:38.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.524+0000 7ff8e98ab640 1 -- 172.21.15.116:0/210448906 shutdown_connections 2024-04-25T00:09:38.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.524+0000 7ff8e98ab640 1 -- 172.21.15.116:0/210448906 wait complete. 2024-04-25T00:09:38.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.524+0000 7ff8e98ab640 1 Processor -- start 2024-04-25T00:09:38.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.524+0000 7ff8e98ab640 1 -- start start 2024-04-25T00:09:38.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.524+0000 7ff8e98ab640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff8e40c6550 0x7ff8e41ebe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:38.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.524+0000 7ff8e98ab640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff8e40c6930 0x7ff8e4142260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:38.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.524+0000 7ff8e98ab640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8e40c6db0 0x7ff8e41427a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:38.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.524+0000 7ff8e98ab640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e4156730 con 0x7ff8e40c6db0 2024-04-25T00:09:38.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.524+0000 7ff8e98ab640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e4159430 con 0x7ff8e40c6930 2024-04-25T00:09:38.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.524+0000 7ff8e98ab640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff8e40c3dc0 con 0x7ff8e40c6550 2024-04-25T00:09:38.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.524+0000 7ff8e37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8e40c6db0 0x7ff8e41427a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:38.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.524+0000 7ff8e37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8e40c6db0 0x7ff8e41427a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41396/0 (socket says 172.21.15.116:41396) 2024-04-25T00:09:38.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.524+0000 7ff8e37fe640 1 -- 172.21.15.116:0/3582253859 learned_addr learned my addr 172.21.15.116:0/3582253859 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:09:38.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.524+0000 7ff8e2ffd640 1 --2- 172.21.15.116:0/3582253859 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff8e40c6550 0x7ff8e41ebe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:38.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.524+0000 7ff8dbfff640 1 --2- 172.21.15.116:0/3582253859 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff8e40c6930 0x7ff8e4142260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:38.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.524+0000 7ff8e37fe640 1 -- 172.21.15.116:0/3582253859 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff8e40c6550 msgr2=0x7ff8e41ebe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:38.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.524+0000 7ff8e37fe640 1 --2- 172.21.15.116:0/3582253859 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff8e40c6550 0x7ff8e41ebe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:38.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.524+0000 7ff8e37fe640 1 -- 172.21.15.116:0/3582253859 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff8e40c6930 msgr2=0x7ff8e4142260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:38.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.524+0000 7ff8e37fe640 1 --2- 172.21.15.116:0/3582253859 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff8e40c6930 0x7ff8e4142260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:38.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.524+0000 7ff8e37fe640 1 -- 172.21.15.116:0/3582253859 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8e4150330 con 0x7ff8e40c6db0 2024-04-25T00:09:38.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.528+0000 7ff8e37fe640 1 --2- 172.21.15.116:0/3582253859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8e40c6db0 0x7ff8e41427a0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc002240 tx=0x7ff8dc009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:38.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.528+0000 7ff8e0ff9640 1 -- 172.21.15.116:0/3582253859 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8dc0bd070 con 0x7ff8e40c6db0 2024-04-25T00:09:38.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.528+0000 7ff8e0ff9640 1 -- 172.21.15.116:0/3582253859 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8dc0a5a90 con 0x7ff8e40c6db0 2024-04-25T00:09:38.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.528+0000 7ff8e98ab640 1 -- 172.21.15.116:0/3582253859 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8e4142ce0 con 0x7ff8e40c6db0 2024-04-25T00:09:38.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.528+0000 7ff8e0ff9640 1 -- 172.21.15.116:0/3582253859 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8dc0b2040 con 0x7ff8e40c6db0 2024-04-25T00:09:38.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.528+0000 7ff8e98ab640 1 -- 172.21.15.116:0/3582253859 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8e4121500 con 0x7ff8e40c6db0 2024-04-25T00:09:38.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.528+0000 7ff8e98ab640 1 -- 172.21.15.116:0/3582253859 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8e4068a60 con 0x7ff8e40c6db0 2024-04-25T00:09:38.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.528+0000 7ff8e0ff9640 1 -- 172.21.15.116:0/3582253859 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff8dc0c0070 con 0x7ff8e40c6db0 2024-04-25T00:09:38.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.528+0000 7ff8e0ff9640 1 --2- 172.21.15.116:0/3582253859 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff8c40678c0 0x7ff8c4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:38.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.528+0000 7ff8e0ff9640 1 -- 172.21.15.116:0/3582253859 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7612+0+0 (secure 0 0 0) 0x7ff8dc11fde0 con 0x7ff8e40c6db0 2024-04-25T00:09:38.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.528+0000 7ff8e2ffd640 1 --2- 172.21.15.116:0/3582253859 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff8c40678c0 0x7ff8c4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:38.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.528+0000 7ff8e2ffd640 1 --2- 172.21.15.116:0/3582253859 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff8c40678c0 0x7ff8c4069d80 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7ff8d4004850 tx=0x7ff8d4001e00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:38.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.532+0000 7ff8e0ff9640 1 -- 172.21.15.116:0/3582253859 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ff8dc016990 con 0x7ff8e40c6db0 2024-04-25T00:09:38.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.780+0000 7ff8e98ab640 1 -- 172.21.15.116:0/3582253859 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff8e40c2c60 con 0x7ff8e40c6db0 2024-04-25T00:09:38.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.780+0000 7ff8e0ff9640 1 -- 172.21.15.116:0/3582253859 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) ==== 94+0+5676 (secure 0 0 0) 0x7ff8dc0efc20 con 0x7ff8e40c6db0 2024-04-25T00:09:38.783 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:09:38.783 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6114,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:09:33.601522+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5236":{"gid":5236,"name":"b","rank":0,"incarnation":62,"state":"up:active","state_seq":25,"addr":"172.21.15.174:6837/1159002662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1159002662},{"type":"v1","addr":"172.21.15.174:6837","nonce":1159002662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5236,"qdb_cluster":[5236]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:09:33.601524+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":132,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4235,"qdb_cluster":[4235]},"id":2}]} 2024-04-25T00:09:38.784 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 66 2024-04-25T00:09:38.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.784+0000 7ff8e98ab640 1 -- 172.21.15.116:0/3582253859 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff8c40678c0 msgr2=0x7ff8c4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:38.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.784+0000 7ff8e98ab640 1 --2- 172.21.15.116:0/3582253859 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff8c40678c0 0x7ff8c4069d80 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7ff8d4004850 tx=0x7ff8d4001e00 comp rx=0 tx=0).stop 2024-04-25T00:09:38.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.784+0000 7ff8e98ab640 1 -- 172.21.15.116:0/3582253859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8e40c6db0 msgr2=0x7ff8e41427a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:38.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.784+0000 7ff8e98ab640 1 --2- 172.21.15.116:0/3582253859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8e40c6db0 0x7ff8e41427a0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc002240 tx=0x7ff8dc009750 comp rx=0 tx=0).stop 2024-04-25T00:09:38.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.784+0000 7ff8e98ab640 1 -- 172.21.15.116:0/3582253859 shutdown_connections 2024-04-25T00:09:38.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.784+0000 7ff8e98ab640 1 --2- 172.21.15.116:0/3582253859 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff8c40678c0 0x7ff8c4069d80 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:38.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.784+0000 7ff8e98ab640 1 --2- 172.21.15.116:0/3582253859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8e40c6db0 0x7ff8e41427a0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:38.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.784+0000 7ff8e98ab640 1 --2- 172.21.15.116:0/3582253859 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff8e40c6930 0x7ff8e4142260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:38.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.784+0000 7ff8e98ab640 1 --2- 172.21.15.116:0/3582253859 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff8e40c6550 0x7ff8e41ebe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:38.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.784+0000 7ff8e98ab640 1 -- 172.21.15.116:0/3582253859 >> 172.21.15.116:0/3582253859 conn(0x7ff8e406c720 msgr2=0x7ff8e40c4fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:09:38.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.784+0000 7ff8e98ab640 1 -- 172.21.15.116:0/3582253859 shutdown_connections 2024-04-25T00:09:38.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:38.784+0000 7ff8e98ab640 1 -- 172.21.15.116:0/3582253859 wait complete. 2024-04-25T00:09:38.803 INFO:tasks.mds_thrash.fs.[b]:mds.a reported in up:standby state 2024-04-25T00:09:38.803 INFO:tasks.mds_thrash.fs.[b]:waiting for 116 secs before thrashing 2024-04-25T00:09:40.940 INFO:tasks.mds_thrash.fs.[a]:reviving mds.c 2024-04-25T00:09:40.940 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-25T00:09:40.940 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-25T00:09:40.943 INFO:tasks.ceph.mds.c:Started 2024-04-25T00:09:40.943 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:09:41.012 INFO:tasks.ceph.mds.c.smithi116.stdout:starting mds.c at 2024-04-25T00:09:41.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.076+0000 7f73ea727640 1 Processor -- start 2024-04-25T00:09:41.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.076+0000 7f73ea727640 1 -- start start 2024-04-25T00:09:41.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.076+0000 7f73ea727640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f73e4150d20 0x7f73e4153620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:41.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.076+0000 7f73ea727640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f73e40c3240 0x7f73e40c3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:41.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.076+0000 7f73ea727640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f73e40c4380 0x7f73e4071050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:41.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.076+0000 7f73ea727640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f73e406fba0 con 0x7f73e40c3240 2024-04-25T00:09:41.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.076+0000 7f73ea727640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f73e40c36a0 con 0x7f73e4150d20 2024-04-25T00:09:41.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.076+0000 7f73ea727640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f73e406f8c0 con 0x7f73e40c4380 2024-04-25T00:09:41.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.076+0000 7f73e37fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f73e40c3240 0x7f73e40c3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:41.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.076+0000 7f73e37fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f73e40c3240 0x7f73e40c3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41426/0 (socket says 172.21.15.116:41426) 2024-04-25T00:09:41.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.076+0000 7f73e37fe640 1 -- 172.21.15.116:0/3603425279 learned_addr learned my addr 172.21.15.116:0/3603425279 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:09:41.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.076+0000 7f73e8cc4640 1 --2- 172.21.15.116:0/3603425279 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f73e40c4380 0x7f73e4071050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:41.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.076+0000 7f73e3fff640 1 --2- 172.21.15.116:0/3603425279 >> v2:172.21.15.174:3300/0 conn(0x7f73e4150d20 0x7f73e4153620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:41.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.076+0000 7f73e37fe640 1 -- 172.21.15.116:0/3603425279 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f73e40c4380 msgr2=0x7f73e4071050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:09:41.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.076+0000 7f73e37fe640 1 --2- 172.21.15.116:0/3603425279 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f73e40c4380 0x7f73e4071050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:41.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.076+0000 7f73e37fe640 1 -- 172.21.15.116:0/3603425279 >> v2:172.21.15.174:3300/0 conn(0x7f73e4150d20 msgr2=0x7f73e4153620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:09:41.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.076+0000 7f73e37fe640 1 --2- 172.21.15.116:0/3603425279 >> v2:172.21.15.174:3300/0 conn(0x7f73e4150d20 0x7f73e4153620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:41.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.076+0000 7f73e37fe640 1 -- 172.21.15.116:0/3603425279 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73e40c5420 con 0x7f73e40c3240 2024-04-25T00:09:41.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.080+0000 7f73e37fe640 1 --2- 172.21.15.116:0/3603425279 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73e40c3240 0x7f73e40c3e40 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f73e40c6b50 tx=0x7f73d801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8aaaecc80c952398 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:41.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.080+0000 7f73e2ffd640 1 -- 172.21.15.116:0/3603425279 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73d80ab5f0 con 0x7f73e40c3240 2024-04-25T00:09:41.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.080+0000 7f73e2ffd640 1 -- 172.21.15.116:0/3603425279 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73d80b0070 con 0x7f73e40c3240 2024-04-25T00:09:41.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.080+0000 7f73e2ffd640 1 -- 172.21.15.116:0/3603425279 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73d80ab9b0 con 0x7f73e40c3240 2024-04-25T00:09:41.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.080+0000 7f73ea727640 1 -- 172.21.15.116:0/3603425279 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73e40c3240 msgr2=0x7f73e40c3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:41.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.080+0000 7f73ea727640 1 --2- 172.21.15.116:0/3603425279 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73e40c3240 0x7f73e40c3e40 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f73e40c6b50 tx=0x7f73d801c520 comp rx=0 tx=0).stop 2024-04-25T00:09:41.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.080+0000 7f73ea727640 1 -- 172.21.15.116:0/3603425279 shutdown_connections 2024-04-25T00:09:41.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.080+0000 7f73ea727640 1 --2- 172.21.15.116:0/3603425279 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f73e40c4380 0x7f73e4071050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:41.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.080+0000 7f73ea727640 1 --2- 172.21.15.116:0/3603425279 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73e40c3240 0x7f73e40c3e40 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:41.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.080+0000 7f73ea727640 1 --2- 172.21.15.116:0/3603425279 >> v2:172.21.15.174:3300/0 conn(0x7f73e4150d20 0x7f73e4153620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:41.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.080+0000 7f73ea727640 1 -- 172.21.15.116:0/3603425279 >> 172.21.15.116:0/3603425279 conn(0x7f73e40c9160 msgr2=0x7f73e40c9570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:09:41.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.080+0000 7f73ea727640 1 -- 172.21.15.116:0/3603425279 shutdown_connections 2024-04-25T00:09:41.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.080+0000 7f73ea727640 1 -- 172.21.15.116:0/3603425279 wait complete. 2024-04-25T00:09:41.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.084+0000 7f73ea727640 1 Processor -- start 2024-04-25T00:09:41.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.084+0000 7f73ea727640 1 -- start start 2024-04-25T00:09:41.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.084+0000 7f73ea727640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f73e40c3240 0x7f73e4152040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:41.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.084+0000 7f73ea727640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73e40c4380 0x7f73e41ecc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:41.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.084+0000 7f73ea727640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f73e4150d20 0x7f73e4152d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:41.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.084+0000 7f73ea727640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f73e4151280 con 0x7f73e40c4380 2024-04-25T00:09:41.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.084+0000 7f73ea727640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f73e4151100 con 0x7f73e4150d20 2024-04-25T00:09:41.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.084+0000 7f73ea727640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f73e40c76a0 con 0x7f73e40c3240 2024-04-25T00:09:41.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.084+0000 7f73e37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73e40c4380 0x7f73e41ecc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:41.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.084+0000 7f73e37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73e40c4380 0x7f73e41ecc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41428/0 (socket says 172.21.15.116:41428) 2024-04-25T00:09:41.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.084+0000 7f73e37fe640 1 -- 172.21.15.116:0/2933565724 learned_addr learned my addr 172.21.15.116:0/2933565724 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:09:41.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.084+0000 7f73e3fff640 1 --2- 172.21.15.116:0/2933565724 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f73e40c3240 0x7f73e4152040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:41.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.084+0000 7f73e8cc4640 1 --2- 172.21.15.116:0/2933565724 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f73e4150d20 0x7f73e4152d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:41.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.084+0000 7f73e37fe640 1 -- 172.21.15.116:0/2933565724 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f73e40c3240 msgr2=0x7f73e4152040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:41.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.084+0000 7f73e37fe640 1 --2- 172.21.15.116:0/2933565724 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f73e40c3240 0x7f73e4152040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:41.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.084+0000 7f73e37fe640 1 -- 172.21.15.116:0/2933565724 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f73e4150d20 msgr2=0x7f73e4152d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:41.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.084+0000 7f73e37fe640 1 --2- 172.21.15.116:0/2933565724 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f73e4150d20 0x7f73e4152d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:41.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.084+0000 7f73e37fe640 1 -- 172.21.15.116:0/2933565724 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73e4153760 con 0x7f73e40c4380 2024-04-25T00:09:41.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.084+0000 7f73e3fff640 1 --2- 172.21.15.116:0/2933565724 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f73e40c3240 0x7f73e4152040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:09:41.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.084+0000 7f73e8cc4640 1 --2- 172.21.15.116:0/2933565724 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f73e4150d20 0x7f73e4152d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:09:41.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.084+0000 7f73e37fe640 1 --2- 172.21.15.116:0/2933565724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73e40c4380 0x7f73e41ecc10 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f73d801abd0 tx=0x7f73d801ad60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:41.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.084+0000 7f73e0ff9640 1 -- 172.21.15.116:0/2933565724 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73d80be5f0 con 0x7f73e40c4380 2024-04-25T00:09:41.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.084+0000 7f73e0ff9640 1 -- 172.21.15.116:0/2933565724 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73d80be790 con 0x7f73e40c4380 2024-04-25T00:09:41.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.084+0000 7f73ea727640 1 -- 172.21.15.116:0/2933565724 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73e4153480 con 0x7f73e40c4380 2024-04-25T00:09:41.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.084+0000 7f73e0ff9640 1 -- 172.21.15.116:0/2933565724 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73d80beb40 con 0x7f73e40c4380 2024-04-25T00:09:41.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.084+0000 7f73ea727640 1 -- 172.21.15.116:0/2933565724 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73e4151840 con 0x7f73e40c4380 2024-04-25T00:09:41.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.084+0000 7f73ea727640 1 -- 172.21.15.116:0/2933565724 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73e4068a60 con 0x7f73e40c4380 2024-04-25T00:09:41.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.088+0000 7f73e0ff9640 1 -- 172.21.15.116:0/2933565724 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f73d80b43f0 con 0x7f73e40c4380 2024-04-25T00:09:41.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.088+0000 7f73e0ff9640 1 --2- 172.21.15.116:0/2933565724 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f73d00678c0 0x7f73d0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:41.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.088+0000 7f73e0ff9640 1 -- 172.21.15.116:0/2933565724 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7612+0+0 (secure 0 0 0) 0x7f73d8120f00 con 0x7f73e40c4380 2024-04-25T00:09:41.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.088+0000 7f73e3fff640 1 --2- 172.21.15.116:0/2933565724 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f73d00678c0 0x7f73d0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:41.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.088+0000 7f73e3fff640 1 --2- 172.21.15.116:0/2933565724 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f73d00678c0 0x7f73d0069d80 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f73d4009720 tx=0x7f73d4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:41.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.092+0000 7f73e0ff9640 1 -- 172.21.15.116:0/2933565724 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f73d80c4060 con 0x7f73e40c4380 2024-04-25T00:09:41.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.308+0000 7f73ea727640 1 -- 172.21.15.116:0/2933565724 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f73e40c4790 con 0x7f73e40c4380 2024-04-25T00:09:41.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.308+0000 7f73e0ff9640 1 -- 172.21.15.116:0/2933565724 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) ==== 94+0+5676 (secure 0 0 0) 0x7f73d8016110 con 0x7f73e40c4380 2024-04-25T00:09:41.311 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:09:41.311 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6114,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:09:33.601522+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5236":{"gid":5236,"name":"b","rank":0,"incarnation":62,"state":"up:active","state_seq":25,"addr":"172.21.15.174:6837/1159002662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1159002662},{"type":"v1","addr":"172.21.15.174:6837","nonce":1159002662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5236,"qdb_cluster":[5236]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:09:33.601524+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":132,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4235,"qdb_cluster":[4235]},"id":2}]} 2024-04-25T00:09:41.311 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 66 2024-04-25T00:09:41.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.312+0000 7f73ea727640 1 -- 172.21.15.116:0/2933565724 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f73d00678c0 msgr2=0x7f73d0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:41.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.312+0000 7f73ea727640 1 --2- 172.21.15.116:0/2933565724 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f73d00678c0 0x7f73d0069d80 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f73d4009720 tx=0x7f73d4008040 comp rx=0 tx=0).stop 2024-04-25T00:09:41.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.312+0000 7f73ea727640 1 -- 172.21.15.116:0/2933565724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73e40c4380 msgr2=0x7f73e41ecc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:41.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.312+0000 7f73ea727640 1 --2- 172.21.15.116:0/2933565724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73e40c4380 0x7f73e41ecc10 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f73d801abd0 tx=0x7f73d801ad60 comp rx=0 tx=0).stop 2024-04-25T00:09:41.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.312+0000 7f73ea727640 1 -- 172.21.15.116:0/2933565724 shutdown_connections 2024-04-25T00:09:41.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.312+0000 7f73ea727640 1 --2- 172.21.15.116:0/2933565724 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f73d00678c0 0x7f73d0069d80 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:41.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.312+0000 7f73ea727640 1 --2- 172.21.15.116:0/2933565724 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f73e4150d20 0x7f73e4152d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:41.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.312+0000 7f73ea727640 1 --2- 172.21.15.116:0/2933565724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f73e40c4380 0x7f73e41ecc10 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:41.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.312+0000 7f73ea727640 1 --2- 172.21.15.116:0/2933565724 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f73e40c3240 0x7f73e4152040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:41.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.312+0000 7f73ea727640 1 -- 172.21.15.116:0/2933565724 >> 172.21.15.116:0/2933565724 conn(0x7f73e40c9160 msgr2=0x7f73e40a9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:09:41.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.312+0000 7f73ea727640 1 -- 172.21.15.116:0/2933565724 shutdown_connections 2024-04-25T00:09:41.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:41.312+0000 7f73ea727640 1 -- 172.21.15.116:0/2933565724 wait complete. 2024-04-25T00:09:41.330 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2024-04-25T00:09:43.334 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:09:43.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.480+0000 7fdd4efd6640 1 Processor -- start 2024-04-25T00:09:43.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.480+0000 7fdd4efd6640 1 -- start start 2024-04-25T00:09:43.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.480+0000 7fdd4efd6640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fdd480aeb20 0x7fdd480c9310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:43.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.480+0000 7fdd4efd6640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd481535d0 con 0x7fdd480c9850 2024-04-25T00:09:43.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.480+0000 7fdd4efd6640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd48153850 con 0x7fdd48150f80 2024-04-25T00:09:43.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.480+0000 7fdd4efd6640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fdd48153b30 con 0x7fdd480aeb20 2024-04-25T00:09:43.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.480+0000 7fdd47fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fdd480c9850 0x7fdd48150870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:44242/0 (socket says 172.21.15.116:44242) 2024-04-25T00:09:43.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.480+0000 7fdd47fff640 1 -- 172.21.15.116:0/2970073119 learned_addr learned my addr 172.21.15.116:0/2970073119 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:09:43.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.480+0000 7fdd4cd72640 1 --2- 172.21.15.116:0/2970073119 >> v2:172.21.15.174:3300/0 conn(0x7fdd480aeb20 0x7fdd480c9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:43.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.480+0000 7fdd477fe640 1 -- 172.21.15.116:0/2970073119 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1407138228 0 0) 0x7fdd34001120 con 0x7fdd480c9850 2024-04-25T00:09:43.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.480+0000 7fdd477fe640 1 -- 172.21.15.116:0/2970073119 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd38003530 con 0x7fdd480c9850 2024-04-25T00:09:43.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.480+0000 7fdd477fe640 1 -- 172.21.15.116:0/2970073119 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3306856237 0 0) 0x7fdd38003530 con 0x7fdd480c9850 2024-04-25T00:09:43.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.480+0000 7fdd477fe640 1 -- 172.21.15.116:0/2970073119 >> v2:172.21.15.174:3300/0 conn(0x7fdd480aeb20 msgr2=0x7fdd480c9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:09:43.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.480+0000 7fdd477fe640 1 --2- 172.21.15.116:0/2970073119 >> v2:172.21.15.174:3300/0 conn(0x7fdd480aeb20 0x7fdd480c9310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:43.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.480+0000 7fdd477fe640 1 -- 172.21.15.116:0/2970073119 >> v1:172.21.15.174:6789/0 conn(0x7fdd48150f80 legacy=0x7fdd480c5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:43.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.480+0000 7fdd477fe640 1 -- 172.21.15.116:0/2970073119 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd48153e10 con 0x7fdd480c9850 2024-04-25T00:09:43.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.480+0000 7fdd477fe640 1 -- 172.21.15.116:0/2970073119 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7fdd34001850 con 0x7fdd480c9850 2024-04-25T00:09:43.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.484+0000 7fdd477fe640 1 -- 172.21.15.116:0/2970073119 >> v1:172.21.15.116:6789/0 conn(0x7fdd480c9850 legacy=0x7fdd48150870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:43.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.484+0000 7fdd477fe640 1 --2- 172.21.15.116:0/2970073119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd380039c0 0x7fdd38003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:43.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.484+0000 7fdd477fe640 1 --2- 172.21.15.116:0/2970073119 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdd38004810 0x7fdd38004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:43.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.484+0000 7fdd477fe640 1 --2- 172.21.15.116:0/2970073119 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdd380052f0 0x7fdd3800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:43.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.484+0000 7fdd477fe640 1 -- 172.21.15.116:0/2970073119 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdd48153b30 con 0x7fdd380039c0 2024-04-25T00:09:43.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.484+0000 7fdd477fe640 1 -- 172.21.15.116:0/2970073119 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fdd3800c1c0 con 0x7fdd38004810 2024-04-25T00:09:43.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.484+0000 7fdd477fe640 1 -- 172.21.15.116:0/2970073119 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fdd3800c3c0 con 0x7fdd380052f0 2024-04-25T00:09:43.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.484+0000 7fdd4cd72640 1 --2- 172.21.15.116:0/2970073119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd380039c0 0x7fdd38003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:43.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.484+0000 7fdd4cd72640 1 -- 172.21.15.116:0/2970073119 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdd380052f0 msgr2=0x7fdd3800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:09:43.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.484+0000 7fdd4cd72640 1 --2- 172.21.15.116:0/2970073119 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdd380052f0 0x7fdd3800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:43.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.484+0000 7fdd4cd72640 1 -- 172.21.15.116:0/2970073119 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdd38004810 msgr2=0x7fdd38004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:09:43.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.484+0000 7fdd4cd72640 1 --2- 172.21.15.116:0/2970073119 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdd38004810 0x7fdd38004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:43.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.484+0000 7fdd4cd72640 1 -- 172.21.15.116:0/2970073119 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd38003530 con 0x7fdd380039c0 2024-04-25T00:09:43.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.484+0000 7fdd4d573640 1 -- 172.21.15.116:0/2970073119 reap_dead start 2024-04-25T00:09:43.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.484+0000 7fdd4cd72640 1 --2- 172.21.15.116:0/2970073119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd380039c0 0x7fdd38003de0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c009ab0 tx=0x7fdd3c01ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7ea3b65a61d7964 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:43.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.484+0000 7fdd477fe640 1 -- 172.21.15.116:0/2970073119 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c0096b0 con 0x7fdd380039c0 2024-04-25T00:09:43.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.484+0000 7fdd477fe640 1 -- 172.21.15.116:0/2970073119 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd3c0a6410 con 0x7fdd380039c0 2024-04-25T00:09:43.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.484+0000 7fdd477fe640 1 -- 172.21.15.116:0/2970073119 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c0a6800 con 0x7fdd380039c0 2024-04-25T00:09:43.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.488+0000 7fdd4efd6640 1 -- 172.21.15.116:0/2970073119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd380039c0 msgr2=0x7fdd38003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:43.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.488+0000 7fdd4efd6640 1 --2- 172.21.15.116:0/2970073119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd380039c0 0x7fdd38003de0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c009ab0 tx=0x7fdd3c01ca30 comp rx=0 tx=0).stop 2024-04-25T00:09:43.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.488+0000 7fdd4efd6640 1 -- 172.21.15.116:0/2970073119 shutdown_connections 2024-04-25T00:09:43.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.488+0000 7fdd4efd6640 1 --2- 172.21.15.116:0/2970073119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd380039c0 0x7fdd38003de0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:43.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.488+0000 7fdd4efd6640 1 -- 172.21.15.116:0/2970073119 >> 172.21.15.116:0/2970073119 conn(0x7fdd48074650 msgr2=0x7fdd48074a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:09:43.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.488+0000 7fdd4efd6640 1 -- 172.21.15.116:0/2970073119 shutdown_connections 2024-04-25T00:09:43.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.488+0000 7fdd4efd6640 1 -- 172.21.15.116:0/2970073119 wait complete. 2024-04-25T00:09:43.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.488+0000 7fdd4efd6640 1 Processor -- start 2024-04-25T00:09:43.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.488+0000 7fdd4efd6640 1 -- start start 2024-04-25T00:09:43.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.488+0000 7fdd4efd6640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdd380039c0 0x7fdd480c2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:43.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.488+0000 7fdd4efd6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdd481468f0 0x7fdd4814f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:43.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.488+0000 7fdd4efd6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd480adbf0 0x7fdd4814fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:43.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.488+0000 7fdd4efd6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdd480af080 con 0x7fdd480adbf0 2024-04-25T00:09:43.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.488+0000 7fdd4efd6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fdd480aef00 con 0x7fdd481468f0 2024-04-25T00:09:43.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.488+0000 7fdd4efd6640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fdd480c5a80 con 0x7fdd380039c0 2024-04-25T00:09:43.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.488+0000 7fdd4d573640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd480adbf0 0x7fdd4814fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:43.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.488+0000 7fdd4d573640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd480adbf0 0x7fdd4814fa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41444/0 (socket says 172.21.15.116:41444) 2024-04-25T00:09:43.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.488+0000 7fdd4d573640 1 -- 172.21.15.116:0/3202263909 learned_addr learned my addr 172.21.15.116:0/3202263909 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:09:43.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.488+0000 7fdd47fff640 1 --2- 172.21.15.116:0/3202263909 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdd481468f0 0x7fdd4814f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:43.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.488+0000 7fdd4cd72640 1 --2- 172.21.15.116:0/3202263909 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdd380039c0 0x7fdd480c2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:43.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.488+0000 7fdd4d573640 1 -- 172.21.15.116:0/3202263909 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdd380039c0 msgr2=0x7fdd480c2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:43.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.488+0000 7fdd4d573640 1 --2- 172.21.15.116:0/3202263909 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdd380039c0 0x7fdd480c2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:43.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.488+0000 7fdd4d573640 1 -- 172.21.15.116:0/3202263909 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdd481468f0 msgr2=0x7fdd4814f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:43.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.488+0000 7fdd4d573640 1 --2- 172.21.15.116:0/3202263909 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdd481468f0 0x7fdd4814f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:43.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.488+0000 7fdd4d573640 1 -- 172.21.15.116:0/3202263909 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd4806c3c0 con 0x7fdd480adbf0 2024-04-25T00:09:43.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.488+0000 7fdd4d573640 1 --2- 172.21.15.116:0/3202263909 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd480adbf0 0x7fdd4814fa80 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fdd48070350 tx=0x7fdd4000cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:43.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.488+0000 7fdd457fa640 1 -- 172.21.15.116:0/3202263909 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd40005d50 con 0x7fdd480adbf0 2024-04-25T00:09:43.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.488+0000 7fdd457fa640 1 -- 172.21.15.116:0/3202263909 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd40003980 con 0x7fdd480adbf0 2024-04-25T00:09:43.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.488+0000 7fdd4efd6640 1 -- 172.21.15.116:0/3202263909 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd481ef730 con 0x7fdd480adbf0 2024-04-25T00:09:43.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.488+0000 7fdd457fa640 1 -- 172.21.15.116:0/3202263909 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd40003d70 con 0x7fdd480adbf0 2024-04-25T00:09:43.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.488+0000 7fdd4efd6640 1 -- 172.21.15.116:0/3202263909 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd480c2bd0 con 0x7fdd480adbf0 2024-04-25T00:09:43.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.492+0000 7fdd4efd6640 1 -- 172.21.15.116:0/3202263909 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd48068a60 con 0x7fdd480adbf0 2024-04-25T00:09:43.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.492+0000 7fdd457fa640 1 -- 172.21.15.116:0/3202263909 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fdd4000a640 con 0x7fdd480adbf0 2024-04-25T00:09:43.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.492+0000 7fdd457fa640 1 --2- 172.21.15.116:0/3202263909 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdd2c0678f0 0x7fdd2c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:09:43.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.492+0000 7fdd457fa640 1 -- 172.21.15.116:0/3202263909 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7612+0+0 (secure 0 0 0) 0x7fdd400857c0 con 0x7fdd480adbf0 2024-04-25T00:09:43.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.492+0000 7fdd4cd72640 1 --2- 172.21.15.116:0/3202263909 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdd2c0678f0 0x7fdd2c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:09:43.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.492+0000 7fdd4cd72640 1 --2- 172.21.15.116:0/3202263909 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdd2c0678f0 0x7fdd2c069db0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c0ad000 tx=0x7fdd3c0046d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:09:43.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.496+0000 7fdd457fa640 1 -- 172.21.15.116:0/3202263909 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fdd40055c20 con 0x7fdd480adbf0 2024-04-25T00:09:43.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.784+0000 7fdd4efd6640 1 -- 172.21.15.116:0/3202263909 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdd480ae0d0 con 0x7fdd480adbf0 2024-04-25T00:09:43.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.784+0000 7fdd457fa640 1 -- 172.21.15.116:0/3202263909 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+6504 (secure 0 0 0) 0x7fdd40055520 con 0x7fdd480adbf0 2024-04-25T00:09:43.785 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:09:43.786 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6114,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6135,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1358527848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1358527848},{"type":"v1","addr":"172.21.15.116:6835","nonce":1358527848}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:09:33.601522+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5236":{"gid":5236,"name":"b","rank":0,"incarnation":62,"state":"up:active","state_seq":25,"addr":"172.21.15.174:6837/1159002662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1159002662},{"type":"v1","addr":"172.21.15.174:6837","nonce":1159002662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5236,"qdb_cluster":[5236]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:09:33.601524+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":132,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4235,"qdb_cluster":[4235]},"id":2}]} 2024-04-25T00:09:43.786 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 67 2024-04-25T00:09:43.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.784+0000 7fdd4efd6640 1 -- 172.21.15.116:0/3202263909 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdd2c0678f0 msgr2=0x7fdd2c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:43.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.784+0000 7fdd4efd6640 1 --2- 172.21.15.116:0/3202263909 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdd2c0678f0 0x7fdd2c069db0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c0ad000 tx=0x7fdd3c0046d0 comp rx=0 tx=0).stop 2024-04-25T00:09:43.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.784+0000 7fdd4efd6640 1 -- 172.21.15.116:0/3202263909 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd480adbf0 msgr2=0x7fdd4814fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:09:43.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.784+0000 7fdd4efd6640 1 --2- 172.21.15.116:0/3202263909 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd480adbf0 0x7fdd4814fa80 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fdd48070350 tx=0x7fdd4000cb90 comp rx=0 tx=0).stop 2024-04-25T00:09:43.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.784+0000 7fdd4efd6640 1 -- 172.21.15.116:0/3202263909 shutdown_connections 2024-04-25T00:09:43.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.784+0000 7fdd4efd6640 1 --2- 172.21.15.116:0/3202263909 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdd2c0678f0 0x7fdd2c069db0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:43.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.784+0000 7fdd4efd6640 1 --2- 172.21.15.116:0/3202263909 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd480adbf0 0x7fdd4814fa80 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:43.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.784+0000 7fdd4efd6640 1 --2- 172.21.15.116:0/3202263909 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdd481468f0 0x7fdd4814f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:43.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.784+0000 7fdd4efd6640 1 --2- 172.21.15.116:0/3202263909 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdd380039c0 0x7fdd480c2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:09:43.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.784+0000 7fdd4efd6640 1 -- 172.21.15.116:0/3202263909 >> 172.21.15.116:0/3202263909 conn(0x7fdd48074650 msgr2=0x7fdd480a8b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:09:43.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.784+0000 7fdd4efd6640 1 -- 172.21.15.116:0/3202263909 shutdown_connections 2024-04-25T00:09:43.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:09:43.784+0000 7fdd4efd6640 1 -- 172.21.15.116:0/3202263909 wait complete. 2024-04-25T00:09:43.807 INFO:tasks.mds_thrash.fs.[a]:mds.c reported in up:standby state 2024-04-25T00:09:43.807 INFO:tasks.mds_thrash.fs.[a]:waiting for 21 secs before thrashing 2024-04-25T00:10:04.809 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:10:04.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.944+0000 7f2eb6cc2640 1 Processor -- start 2024-04-25T00:10:04.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.944+0000 7f2eb6cc2640 1 -- start start 2024-04-25T00:10:04.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.948+0000 7f2eb6cc2640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2eb0152e60 0x7f2eb01599a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:04.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.948+0000 7f2eb6cc2640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f2eb01562d0 0x7f2eb00c36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:04.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.948+0000 7f2eb6cc2640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2eb00c3c20 0x7f2eb00c5010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:04.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.948+0000 7f2eb6cc2640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f2eb0156760 con 0x7f2eb0152e60 2024-04-25T00:10:04.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.948+0000 7f2eb6cc2640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f2eb0153240 con 0x7f2eb01562d0 2024-04-25T00:10:04.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.948+0000 7f2eb6cc2640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f2eb00734e0 con 0x7f2eb00c3c20 2024-04-25T00:10:04.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.948+0000 7f2eb4a5e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2eb0152e60 0x7f2eb01599a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:04.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.948+0000 7f2eb4a5e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2eb0152e60 0x7f2eb01599a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50656/0 (socket says 172.21.15.116:50656) 2024-04-25T00:10:04.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.948+0000 7f2eb4a5e640 1 -- 172.21.15.116:0/2951098856 learned_addr learned my addr 172.21.15.116:0/2951098856 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:10:04.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.948+0000 7f2eaffff640 1 --2- 172.21.15.116:0/2951098856 >> v2:172.21.15.174:3300/0 conn(0x7f2eb01562d0 0x7f2eb00c36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:04.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.948+0000 7f2eb525f640 1 --2- 172.21.15.116:0/2951098856 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2eb00c3c20 0x7f2eb00c5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:04.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.948+0000 7f2eb4a5e640 1 -- 172.21.15.116:0/2951098856 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2eb00c3c20 msgr2=0x7f2eb00c5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:10:04.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.948+0000 7f2eb4a5e640 1 --2- 172.21.15.116:0/2951098856 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2eb00c3c20 0x7f2eb00c5010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:04.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.948+0000 7f2eb4a5e640 1 -- 172.21.15.116:0/2951098856 >> v2:172.21.15.174:3300/0 conn(0x7f2eb01562d0 msgr2=0x7f2eb00c36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:10:04.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.948+0000 7f2eb4a5e640 1 --2- 172.21.15.116:0/2951098856 >> v2:172.21.15.174:3300/0 conn(0x7f2eb01562d0 0x7f2eb00c36e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:04.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.948+0000 7f2eb4a5e640 1 -- 172.21.15.116:0/2951098856 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2eb0073200 con 0x7f2eb0152e60 2024-04-25T00:10:04.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.948+0000 7f2eb4a5e640 1 --2- 172.21.15.116:0/2951098856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2eb0152e60 0x7f2eb01599a0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f2eb00c3100 tx=0x7f2ea401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3e22f655da8cf7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:04.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.948+0000 7f2eaf7fe640 1 -- 172.21.15.116:0/2951098856 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea40b0070 con 0x7f2eb0152e60 2024-04-25T00:10:04.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.948+0000 7f2eaf7fe640 1 -- 172.21.15.116:0/2951098856 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ea40ab600 con 0x7f2eb0152e60 2024-04-25T00:10:04.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.948+0000 7f2eaf7fe640 1 -- 172.21.15.116:0/2951098856 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea40ab9d0 con 0x7f2eb0152e60 2024-04-25T00:10:04.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.952+0000 7f2eb6cc2640 1 -- 172.21.15.116:0/2951098856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2eb0152e60 msgr2=0x7f2eb01599a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:04.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.952+0000 7f2eb6cc2640 1 --2- 172.21.15.116:0/2951098856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2eb0152e60 0x7f2eb01599a0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f2eb00c3100 tx=0x7f2ea401c490 comp rx=0 tx=0).stop 2024-04-25T00:10:04.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.952+0000 7f2eb6cc2640 1 -- 172.21.15.116:0/2951098856 shutdown_connections 2024-04-25T00:10:04.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.952+0000 7f2eb6cc2640 1 --2- 172.21.15.116:0/2951098856 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2eb00c3c20 0x7f2eb00c5010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:04.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.952+0000 7f2eb6cc2640 1 --2- 172.21.15.116:0/2951098856 >> v2:172.21.15.174:3300/0 conn(0x7f2eb01562d0 0x7f2eb00c36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:04.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.952+0000 7f2eb6cc2640 1 --2- 172.21.15.116:0/2951098856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2eb0152e60 0x7f2eb01599a0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:04.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.952+0000 7f2eb6cc2640 1 -- 172.21.15.116:0/2951098856 >> 172.21.15.116:0/2951098856 conn(0x7f2eb006c720 msgr2=0x7f2eb006cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:10:04.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.952+0000 7f2eb6cc2640 1 -- 172.21.15.116:0/2951098856 shutdown_connections 2024-04-25T00:10:04.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.952+0000 7f2eb6cc2640 1 -- 172.21.15.116:0/2951098856 wait complete. 2024-04-25T00:10:04.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.952+0000 7f2eb6cc2640 1 Processor -- start 2024-04-25T00:10:04.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.952+0000 7f2eb6cc2640 1 -- start start 2024-04-25T00:10:04.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.952+0000 7f2eb6cc2640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2eb00c3c20 0x7f2eb014aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:04.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.952+0000 7f2eb6cc2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2eb0152e60 0x7f2eb00ab060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:04.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.952+0000 7f2eb6cc2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2eb01562d0 0x7f2eb00abd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:04.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.952+0000 7f2eb6cc2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2eb0072e20 con 0x7f2eb0152e60 2024-04-25T00:10:04.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.952+0000 7f2eb6cc2640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2eb0072080 con 0x7f2eb01562d0 2024-04-25T00:10:04.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.952+0000 7f2eb6cc2640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f2eb0072fa0 con 0x7f2eb00c3c20 2024-04-25T00:10:04.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.952+0000 7f2eaffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2eb0152e60 0x7f2eb00ab060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:04.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.952+0000 7f2eaffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2eb0152e60 0x7f2eb00ab060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50672/0 (socket says 172.21.15.116:50672) 2024-04-25T00:10:04.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.952+0000 7f2eaffff640 1 -- 172.21.15.116:0/2669389338 learned_addr learned my addr 172.21.15.116:0/2669389338 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:10:04.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.952+0000 7f2eb4a5e640 1 --2- 172.21.15.116:0/2669389338 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2eb00c3c20 0x7f2eb014aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:04.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.952+0000 7f2eb525f640 1 --2- 172.21.15.116:0/2669389338 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2eb01562d0 0x7f2eb00abd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:04.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.952+0000 7f2eaffff640 1 -- 172.21.15.116:0/2669389338 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2eb00c3c20 msgr2=0x7f2eb014aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:04.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.952+0000 7f2eaffff640 1 --2- 172.21.15.116:0/2669389338 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2eb00c3c20 0x7f2eb014aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:04.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.952+0000 7f2eaffff640 1 -- 172.21.15.116:0/2669389338 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2eb01562d0 msgr2=0x7f2eb00abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:04.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.952+0000 7f2eaffff640 1 --2- 172.21.15.116:0/2669389338 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2eb01562d0 0x7f2eb00abd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:04.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.952+0000 7f2eaffff640 1 -- 172.21.15.116:0/2669389338 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2eb0157420 con 0x7f2eb0152e60 2024-04-25T00:10:04.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.952+0000 7f2eb4a5e640 1 --2- 172.21.15.116:0/2669389338 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2eb00c3c20 0x7f2eb014aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:10:04.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.952+0000 7f2eaffff640 1 --2- 172.21.15.116:0/2669389338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2eb0152e60 0x7f2eb00ab060 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c00d830 tx=0x7f2e9c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:04.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.956+0000 7f2ead7fa640 1 -- 172.21.15.116:0/2669389338 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e9c00b6c0 con 0x7f2eb0152e60 2024-04-25T00:10:04.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.956+0000 7f2ead7fa640 1 -- 172.21.15.116:0/2669389338 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e9c019070 con 0x7f2eb0152e60 2024-04-25T00:10:04.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.956+0000 7f2eb6cc2640 1 -- 172.21.15.116:0/2669389338 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2eb0157700 con 0x7f2eb0152e60 2024-04-25T00:10:04.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.956+0000 7f2eb6cc2640 1 -- 172.21.15.116:0/2669389338 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2eb0158670 con 0x7f2eb0152e60 2024-04-25T00:10:04.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.956+0000 7f2ead7fa640 1 -- 172.21.15.116:0/2669389338 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e9c00ba60 con 0x7f2eb0152e60 2024-04-25T00:10:04.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.956+0000 7f2eb6cc2640 1 -- 172.21.15.116:0/2669389338 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2eb0068a60 con 0x7f2eb0152e60 2024-04-25T00:10:04.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.956+0000 7f2ead7fa640 1 -- 172.21.15.116:0/2669389338 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2e9c01e460 con 0x7f2eb0152e60 2024-04-25T00:10:04.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.956+0000 7f2ead7fa640 1 --2- 172.21.15.116:0/2669389338 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2e940678f0 0x7f2e94069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:04.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.956+0000 7f2ead7fa640 1 -- 172.21.15.116:0/2669389338 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7612+0+0 (secure 0 0 0) 0x7f2e9c086a10 con 0x7f2eb0152e60 2024-04-25T00:10:04.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.956+0000 7f2eb4a5e640 1 --2- 172.21.15.116:0/2669389338 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2e940678f0 0x7f2e94069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:04.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.956+0000 7f2eb4a5e640 1 --2- 172.21.15.116:0/2669389338 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2e940678f0 0x7f2e94069db0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f2eb00c2080 tx=0x7f2ea4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:04.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:04.960+0000 7f2ead7fa640 1 -- 172.21.15.116:0/2669389338 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f2e9c056e70 con 0x7f2eb0152e60 2024-04-25T00:10:05.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.204+0000 7f2eb6cc2640 1 -- 172.21.15.116:0/2669389338 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2eb00c24e0 con 0x7f2eb0152e60 2024-04-25T00:10:05.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.204+0000 7f2ead7fa640 1 -- 172.21.15.116:0/2669389338 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+6504 (secure 0 0 0) 0x7f2e9c056770 con 0x7f2eb0152e60 2024-04-25T00:10:05.207 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:10:05.207 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6114,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6135,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1358527848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1358527848},{"type":"v1","addr":"172.21.15.116:6835","nonce":1358527848}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:09:33.601522+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5236":{"gid":5236,"name":"b","rank":0,"incarnation":62,"state":"up:active","state_seq":25,"addr":"172.21.15.174:6837/1159002662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1159002662},{"type":"v1","addr":"172.21.15.174:6837","nonce":1159002662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5236,"qdb_cluster":[5236]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:09:33.601524+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":132,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4235,"qdb_cluster":[4235]},"id":2}]} 2024-04-25T00:10:05.207 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 67 2024-04-25T00:10:05.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.208+0000 7f2eb6cc2640 1 -- 172.21.15.116:0/2669389338 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2e940678f0 msgr2=0x7f2e94069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:05.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.208+0000 7f2eb6cc2640 1 --2- 172.21.15.116:0/2669389338 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2e940678f0 0x7f2e94069db0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f2eb00c2080 tx=0x7f2ea4009040 comp rx=0 tx=0).stop 2024-04-25T00:10:05.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.208+0000 7f2eb6cc2640 1 -- 172.21.15.116:0/2669389338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2eb0152e60 msgr2=0x7f2eb00ab060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:05.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.208+0000 7f2eb6cc2640 1 --2- 172.21.15.116:0/2669389338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2eb0152e60 0x7f2eb00ab060 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c00d830 tx=0x7f2e9c00dcf0 comp rx=0 tx=0).stop 2024-04-25T00:10:05.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.208+0000 7f2eb6cc2640 1 -- 172.21.15.116:0/2669389338 shutdown_connections 2024-04-25T00:10:05.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.208+0000 7f2eb6cc2640 1 --2- 172.21.15.116:0/2669389338 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2e940678f0 0x7f2e94069db0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:05.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.208+0000 7f2eb6cc2640 1 --2- 172.21.15.116:0/2669389338 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2eb01562d0 0x7f2eb00abd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:05.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.208+0000 7f2eb6cc2640 1 --2- 172.21.15.116:0/2669389338 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2eb0152e60 0x7f2eb00ab060 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:05.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.208+0000 7f2eb6cc2640 1 --2- 172.21.15.116:0/2669389338 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2eb00c3c20 0x7f2eb014aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:05.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.208+0000 7f2eb6cc2640 1 -- 172.21.15.116:0/2669389338 >> 172.21.15.116:0/2669389338 conn(0x7f2eb006c720 msgr2=0x7f2eb006f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:10:05.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.208+0000 7f2eb6cc2640 1 -- 172.21.15.116:0/2669389338 shutdown_connections 2024-04-25T00:10:05.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.208+0000 7f2eb6cc2640 1 -- 172.21.15.116:0/2669389338 wait complete. 2024-04-25T00:10:05.226 INFO:tasks.mds_thrash.fs.[a]:kill mds.b (rank=0) 2024-04-25T00:10:05.226 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-25T00:10:05.227 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:10:05.242 INFO:tasks.ceph.mds.b:Stopped 2024-04-25T00:10:05.243 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:10:05.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.376+0000 7fd4e4a70640 1 Processor -- start 2024-04-25T00:10:05.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.380+0000 7fd4e4a70640 1 -- start start 2024-04-25T00:10:05.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.380+0000 7fd4e4a70640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd4e0152d80 0x7fd4e006bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:05.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.380+0000 7fd4e4a70640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd4e006d5f0 0x7fd4e0156c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:05.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.380+0000 7fd4e4a70640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fd4e006c290 0x7fd4e01571c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:05.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.380+0000 7fd4e4a70640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fd4e00c9d60 con 0x7fd4e006d5f0 2024-04-25T00:10:05.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.380+0000 7fd4e4a70640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fd4e00c9150 con 0x7fd4e006c290 2024-04-25T00:10:05.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.380+0000 7fd4e4a70640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fd4e00c9430 con 0x7fd4e0152d80 2024-04-25T00:10:05.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.380+0000 7fd4ddd9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd4e006d5f0 0x7fd4e0156c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:05.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.380+0000 7fd4ddd9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd4e006d5f0 0x7fd4e0156c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50674/0 (socket says 172.21.15.116:50674) 2024-04-25T00:10:05.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.380+0000 7fd4ddd9b640 1 -- 172.21.15.116:0/4035287152 learned_addr learned my addr 172.21.15.116:0/4035287152 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:10:05.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.380+0000 7fd4de59c640 1 --2- 172.21.15.116:0/4035287152 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd4e0152d80 0x7fd4e006bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:05.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.380+0000 7fd4ded9d640 1 --2- 172.21.15.116:0/4035287152 >> v2:172.21.15.174:3300/0 conn(0x7fd4e006c290 0x7fd4e01571c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:05.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.380+0000 7fd4ddd9b640 1 -- 172.21.15.116:0/4035287152 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd4e0152d80 msgr2=0x7fd4e006bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:10:05.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.380+0000 7fd4ddd9b640 1 --2- 172.21.15.116:0/4035287152 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd4e0152d80 0x7fd4e006bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:05.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.380+0000 7fd4ddd9b640 1 -- 172.21.15.116:0/4035287152 >> v2:172.21.15.174:3300/0 conn(0x7fd4e006c290 msgr2=0x7fd4e01571c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:10:05.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.380+0000 7fd4ddd9b640 1 --2- 172.21.15.116:0/4035287152 >> v2:172.21.15.174:3300/0 conn(0x7fd4e006c290 0x7fd4e01571c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:05.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.380+0000 7fd4ddd9b640 1 -- 172.21.15.116:0/4035287152 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4e00c9710 con 0x7fd4e006d5f0 2024-04-25T00:10:05.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.380+0000 7fd4ddd9b640 1 --2- 172.21.15.116:0/4035287152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4e006d5f0 0x7fd4e0156c80 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fd4e00c86f0 tx=0x7fd4d401c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a6b9627f0f09ec9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:05.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.380+0000 7fd4dd59a640 1 -- 172.21.15.116:0/4035287152 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4d40b0070 con 0x7fd4e006d5f0 2024-04-25T00:10:05.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.380+0000 7fd4dd59a640 1 -- 172.21.15.116:0/4035287152 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4d40ab600 con 0x7fd4e006d5f0 2024-04-25T00:10:05.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.380+0000 7fd4dd59a640 1 -- 172.21.15.116:0/4035287152 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4d40ab9d0 con 0x7fd4e006d5f0 2024-04-25T00:10:05.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.384+0000 7fd4e4a70640 1 -- 172.21.15.116:0/4035287152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4e006d5f0 msgr2=0x7fd4e0156c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:05.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.384+0000 7fd4e4a70640 1 --2- 172.21.15.116:0/4035287152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4e006d5f0 0x7fd4e0156c80 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fd4e00c86f0 tx=0x7fd4d401c460 comp rx=0 tx=0).stop 2024-04-25T00:10:05.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.384+0000 7fd4e4a70640 1 -- 172.21.15.116:0/4035287152 shutdown_connections 2024-04-25T00:10:05.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.384+0000 7fd4e4a70640 1 --2- 172.21.15.116:0/4035287152 >> v2:172.21.15.174:3300/0 conn(0x7fd4e006c290 0x7fd4e01571c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:05.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.384+0000 7fd4e4a70640 1 --2- 172.21.15.116:0/4035287152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4e006d5f0 0x7fd4e0156c80 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:05.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.384+0000 7fd4e4a70640 1 --2- 172.21.15.116:0/4035287152 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd4e0152d80 0x7fd4e006bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:05.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.384+0000 7fd4e4a70640 1 -- 172.21.15.116:0/4035287152 >> 172.21.15.116:0/4035287152 conn(0x7fd4e00acdd0 msgr2=0x7fd4e00ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:10:05.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.384+0000 7fd4e4a70640 1 -- 172.21.15.116:0/4035287152 shutdown_connections 2024-04-25T00:10:05.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.384+0000 7fd4e4a70640 1 -- 172.21.15.116:0/4035287152 wait complete. 2024-04-25T00:10:05.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.384+0000 7fd4e4a70640 1 Processor -- start 2024-04-25T00:10:05.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.384+0000 7fd4e4a70640 1 -- start start 2024-04-25T00:10:05.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.384+0000 7fd4e4a70640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd4e006c290 0x7fd4e00ac1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:05.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.384+0000 7fd4e4a70640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd4e006d5f0 0x7fd4e014bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:05.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.384+0000 7fd4e4a70640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4e0152d80 0x7fd4e014b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:05.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.384+0000 7fd4e4a70640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd4e0159710 con 0x7fd4e0152d80 2024-04-25T00:10:05.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.384+0000 7fd4e4a70640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd4e0159590 con 0x7fd4e006c290 2024-04-25T00:10:05.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.384+0000 7fd4e4a70640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fd4e0159890 con 0x7fd4e006d5f0 2024-04-25T00:10:05.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.384+0000 7fd4ded9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4e0152d80 0x7fd4e014b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:05.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.384+0000 7fd4ded9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4e0152d80 0x7fd4e014b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50688/0 (socket says 172.21.15.116:50688) 2024-04-25T00:10:05.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.384+0000 7fd4ded9d640 1 -- 172.21.15.116:0/268923750 learned_addr learned my addr 172.21.15.116:0/268923750 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:10:05.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.384+0000 7fd4de59c640 1 --2- 172.21.15.116:0/268923750 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd4e006c290 0x7fd4e00ac1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:05.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.384+0000 7fd4ddd9b640 1 --2- 172.21.15.116:0/268923750 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd4e006d5f0 0x7fd4e014bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:05.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.384+0000 7fd4ded9d640 1 -- 172.21.15.116:0/268923750 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd4e006d5f0 msgr2=0x7fd4e014bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:05.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.384+0000 7fd4ded9d640 1 --2- 172.21.15.116:0/268923750 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd4e006d5f0 0x7fd4e014bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:05.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.384+0000 7fd4ded9d640 1 -- 172.21.15.116:0/268923750 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd4e006c290 msgr2=0x7fd4e00ac1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:05.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.384+0000 7fd4ded9d640 1 --2- 172.21.15.116:0/268923750 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd4e006c290 0x7fd4e00ac1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:05.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.384+0000 7fd4ded9d640 1 -- 172.21.15.116:0/268923750 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4e01ee9f0 con 0x7fd4e0152d80 2024-04-25T00:10:05.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.384+0000 7fd4ded9d640 1 --2- 172.21.15.116:0/268923750 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4e0152d80 0x7fd4e014b0a0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fd4d8007e60 tx=0x7fd4d800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:05.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.384+0000 7fd4ceffd640 1 -- 172.21.15.116:0/268923750 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4d8014620 con 0x7fd4e0152d80 2024-04-25T00:10:05.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.384+0000 7fd4ceffd640 1 -- 172.21.15.116:0/268923750 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4d8019070 con 0x7fd4e0152d80 2024-04-25T00:10:05.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.384+0000 7fd4e4a70640 1 -- 172.21.15.116:0/268923750 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4e01f29b0 con 0x7fd4e0152d80 2024-04-25T00:10:05.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.384+0000 7fd4ceffd640 1 -- 172.21.15.116:0/268923750 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4d8014a20 con 0x7fd4e0152d80 2024-04-25T00:10:05.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.384+0000 7fd4e4a70640 1 -- 172.21.15.116:0/268923750 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4e0152490 con 0x7fd4e0152d80 2024-04-25T00:10:05.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.388+0000 7fd4e4a70640 1 -- 172.21.15.116:0/268923750 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4e01ee7e0 con 0x7fd4e0152d80 2024-04-25T00:10:05.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.388+0000 7fd4ceffd640 1 -- 172.21.15.116:0/268923750 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd4d8014bc0 con 0x7fd4e0152d80 2024-04-25T00:10:05.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.388+0000 7fd4ceffd640 1 --2- 172.21.15.116:0/268923750 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd4c00678c0 0x7fd4c0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:05.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.388+0000 7fd4ceffd640 1 -- 172.21.15.116:0/268923750 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7612+0+0 (secure 0 0 0) 0x7fd4d8086b80 con 0x7fd4e0152d80 2024-04-25T00:10:05.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.388+0000 7fd4de59c640 1 --2- 172.21.15.116:0/268923750 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd4c00678c0 0x7fd4c0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:05.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.388+0000 7fd4de59c640 1 --2- 172.21.15.116:0/268923750 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd4c00678c0 0x7fd4c0069d80 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fd4d00079c0 tx=0x7fd4d0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:05.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.396+0000 7fd4ceffd640 1 -- 172.21.15.116:0/268923750 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fd4d8056880 con 0x7fd4e0152d80 2024-04-25T00:10:05.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.628+0000 7fd4e4a70640 1 -- 172.21.15.116:0/268923750 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd4e0068350 con 0x7fd4e0152d80 2024-04-25T00:10:05.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.628+0000 7fd4ceffd640 1 -- 172.21.15.116:0/268923750 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+6504 (secure 0 0 0) 0x7fd4d805ab80 con 0x7fd4e0152d80 2024-04-25T00:10:05.630 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:10:05.630 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6114,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6135,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1358527848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1358527848},{"type":"v1","addr":"172.21.15.116:6835","nonce":1358527848}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:09:33.601522+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5236":{"gid":5236,"name":"b","rank":0,"incarnation":62,"state":"up:active","state_seq":25,"addr":"172.21.15.174:6837/1159002662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1159002662},{"type":"v1","addr":"172.21.15.174:6837","nonce":1159002662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5236,"qdb_cluster":[5236]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:09:33.601524+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":132,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4235,"qdb_cluster":[4235]},"id":2}]} 2024-04-25T00:10:05.630 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 67 2024-04-25T00:10:05.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.632+0000 7fd4e4a70640 1 -- 172.21.15.116:0/268923750 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd4c00678c0 msgr2=0x7fd4c0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:05.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.632+0000 7fd4e4a70640 1 --2- 172.21.15.116:0/268923750 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd4c00678c0 0x7fd4c0069d80 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fd4d00079c0 tx=0x7fd4d0008040 comp rx=0 tx=0).stop 2024-04-25T00:10:05.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.632+0000 7fd4e4a70640 1 -- 172.21.15.116:0/268923750 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4e0152d80 msgr2=0x7fd4e014b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:05.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.632+0000 7fd4e4a70640 1 --2- 172.21.15.116:0/268923750 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4e0152d80 0x7fd4e014b0a0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fd4d8007e60 tx=0x7fd4d800e4b0 comp rx=0 tx=0).stop 2024-04-25T00:10:05.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.632+0000 7fd4e4a70640 1 -- 172.21.15.116:0/268923750 shutdown_connections 2024-04-25T00:10:05.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.632+0000 7fd4e4a70640 1 --2- 172.21.15.116:0/268923750 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd4c00678c0 0x7fd4c0069d80 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:05.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.632+0000 7fd4e4a70640 1 --2- 172.21.15.116:0/268923750 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd4e0152d80 0x7fd4e014b0a0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:05.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.632+0000 7fd4e4a70640 1 --2- 172.21.15.116:0/268923750 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd4e006d5f0 0x7fd4e014bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:05.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.632+0000 7fd4e4a70640 1 --2- 172.21.15.116:0/268923750 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd4e006c290 0x7fd4e00ac1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:05.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.632+0000 7fd4e4a70640 1 -- 172.21.15.116:0/268923750 >> 172.21.15.116:0/268923750 conn(0x7fd4e00acdd0 msgr2=0x7fd4e015f950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:10:05.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.632+0000 7fd4e4a70640 1 -- 172.21.15.116:0/268923750 shutdown_connections 2024-04-25T00:10:05.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:05.632+0000 7fd4e4a70640 1 -- 172.21.15.116:0/268923750 wait complete. 2024-04-25T00:10:05.649 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T00:10:07.008 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:10:07.011 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:10:07.651 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:10:07.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.779+0000 7f41cad85640 1 Processor -- start 2024-04-25T00:10:07.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.783+0000 7f41cad85640 1 -- start start 2024-04-25T00:10:07.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.783+0000 7f41cad85640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f41c4150d20 0x7f41c40c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:07.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.783+0000 7f41cad85640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f41c40a94a0 0x7f41c4153ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:07.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.783+0000 7f41cad85640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41c40c3910 con 0x7f41c40c2f50 2024-04-25T00:10:07.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.783+0000 7f41cad85640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f41c4154530 con 0x7f41c4150d20 2024-04-25T00:10:07.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.783+0000 7f41cad85640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f41c40c16e0 con 0x7f41c40a94a0 2024-04-25T00:10:07.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.783+0000 7f41c9322640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f41c40a94a0 0x7f41c4153ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:07.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.783+0000 7f41c8b21640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f41c4150d20 0x7f41c40c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:07.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.783+0000 7f41c9322640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f41c40a94a0 0x7f41c4153ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:59662/0 (socket says 172.21.15.116:59662) 2024-04-25T00:10:07.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.783+0000 7f41c9322640 1 -- 172.21.15.116:0/334266153 learned_addr learned my addr 172.21.15.116:0/334266153 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:10:07.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.783+0000 7f41c9322640 1 -- 172.21.15.116:0/334266153 >> v2:172.21.15.174:3300/0 conn(0x7f41c4150d20 msgr2=0x7f41c40c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:10:07.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.783+0000 7f41c9322640 1 --2- 172.21.15.116:0/334266153 >> v2:172.21.15.174:3300/0 conn(0x7f41c4150d20 0x7f41c40c2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:07.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.783+0000 7f41c9322640 1 -- 172.21.15.116:0/334266153 >> v1:172.21.15.174:6789/0 conn(0x7f41c40c2f50 legacy=0x7f41c41538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:07.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.783+0000 7f41c9322640 1 -- 172.21.15.116:0/334266153 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41c40c1400 con 0x7f41c40a94a0 2024-04-25T00:10:07.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.783+0000 7f41c8b21640 1 --2- 172.21.15.116:0/334266153 >> v2:172.21.15.174:3300/0 conn(0x7f41c4150d20 0x7f41c40c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:10:07.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.787+0000 7f41c9322640 1 --2- 172.21.15.116:0/334266153 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f41c40a94a0 0x7f41c4153ff0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f41bc009e50 tx=0x7f41bc01ca30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49cf8ef30c26fd8d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:07.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.787+0000 7f41c37fe640 1 -- 172.21.15.116:0/334266153 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41bc0a5810 con 0x7f41c40a94a0 2024-04-25T00:10:07.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.787+0000 7f41c37fe640 1 -- 172.21.15.116:0/334266153 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41bc0a59b0 con 0x7f41c40a94a0 2024-04-25T00:10:07.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.787+0000 7f41c37fe640 1 -- 172.21.15.116:0/334266153 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41bc0a5d80 con 0x7f41c40a94a0 2024-04-25T00:10:07.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.787+0000 7f41cad85640 1 -- 172.21.15.116:0/334266153 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f41c40a94a0 msgr2=0x7f41c4153ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:07.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.787+0000 7f41cad85640 1 --2- 172.21.15.116:0/334266153 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f41c40a94a0 0x7f41c4153ff0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f41bc009e50 tx=0x7f41bc01ca30 comp rx=0 tx=0).stop 2024-04-25T00:10:07.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.787+0000 7f41cad85640 1 -- 172.21.15.116:0/334266153 shutdown_connections 2024-04-25T00:10:07.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.787+0000 7f41cad85640 1 --2- 172.21.15.116:0/334266153 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f41c40a94a0 0x7f41c4153ff0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:07.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.787+0000 7f41cad85640 1 --2- 172.21.15.116:0/334266153 >> v2:172.21.15.174:3300/0 conn(0x7f41c4150d20 0x7f41c40c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:07.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.787+0000 7f41cad85640 1 -- 172.21.15.116:0/334266153 >> 172.21.15.116:0/334266153 conn(0x7f41c40c3b00 msgr2=0x7f41c40c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:10:07.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.787+0000 7f41cad85640 1 -- 172.21.15.116:0/334266153 shutdown_connections 2024-04-25T00:10:07.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.787+0000 7f41cad85640 1 -- 172.21.15.116:0/334266153 wait complete. 2024-04-25T00:10:07.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.787+0000 7f41cad85640 1 Processor -- start 2024-04-25T00:10:07.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.787+0000 7f41cad85640 1 -- start start 2024-04-25T00:10:07.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.787+0000 7f41cad85640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f41c40a94a0 0x7f41c4154980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:07.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.787+0000 7f41cad85640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f41c40c2f50 0x7f41c406cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:07.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.787+0000 7f41cad85640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41c4150d20 0x7f41c4143ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:07.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.787+0000 7f41cad85640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f41c41608b0 con 0x7f41c4150d20 2024-04-25T00:10:07.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.787+0000 7f41cad85640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f41c4160730 con 0x7f41c40a94a0 2024-04-25T00:10:07.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.787+0000 7f41cad85640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f41c4160a30 con 0x7f41c40c2f50 2024-04-25T00:10:07.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.787+0000 7f41c9322640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41c4150d20 0x7f41c4143ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:07.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.787+0000 7f41c9322640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41c4150d20 0x7f41c4143ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50704/0 (socket says 172.21.15.116:50704) 2024-04-25T00:10:07.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.787+0000 7f41c9322640 1 -- 172.21.15.116:0/1557459576 learned_addr learned my addr 172.21.15.116:0/1557459576 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:10:07.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.787+0000 7f41c3fff640 1 --2- 172.21.15.116:0/1557459576 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f41c40c2f50 0x7f41c406cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:07.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.791+0000 7f41c8b21640 1 --2- 172.21.15.116:0/1557459576 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f41c40a94a0 0x7f41c4154980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:07.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.791+0000 7f41c9322640 1 -- 172.21.15.116:0/1557459576 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f41c40c2f50 msgr2=0x7f41c406cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:07.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.791+0000 7f41c9322640 1 --2- 172.21.15.116:0/1557459576 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f41c40c2f50 0x7f41c406cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:07.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.791+0000 7f41c9322640 1 -- 172.21.15.116:0/1557459576 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f41c40a94a0 msgr2=0x7f41c4154980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:07.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.791+0000 7f41c9322640 1 --2- 172.21.15.116:0/1557459576 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f41c40a94a0 0x7f41c4154980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:07.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.791+0000 7f41c9322640 1 -- 172.21.15.116:0/1557459576 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41c41575e0 con 0x7f41c4150d20 2024-04-25T00:10:07.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.791+0000 7f41c9322640 1 --2- 172.21.15.116:0/1557459576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41c4150d20 0x7f41c4143ac0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f41bc0af040 tx=0x7f41bc0197d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:07.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.791+0000 7f41c17fa640 1 -- 172.21.15.116:0/1557459576 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41bc0bc020 con 0x7f41c4150d20 2024-04-25T00:10:07.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.791+0000 7f41c17fa640 1 -- 172.21.15.116:0/1557459576 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41bc0b86b0 con 0x7f41c4150d20 2024-04-25T00:10:07.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.791+0000 7f41cad85640 1 -- 172.21.15.116:0/1557459576 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41c4150070 con 0x7f41c4150d20 2024-04-25T00:10:07.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.791+0000 7f41c17fa640 1 -- 172.21.15.116:0/1557459576 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41bc0b8aa0 con 0x7f41c4150d20 2024-04-25T00:10:07.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.791+0000 7f41cad85640 1 -- 172.21.15.116:0/1557459576 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41c406d2f0 con 0x7f41c4150d20 2024-04-25T00:10:07.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.791+0000 7f41cad85640 1 -- 172.21.15.116:0/1557459576 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41c4068350 con 0x7f41c4150d20 2024-04-25T00:10:07.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.791+0000 7f41c17fa640 1 -- 172.21.15.116:0/1557459576 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f41bc0bf020 con 0x7f41c4150d20 2024-04-25T00:10:07.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.791+0000 7f41c17fa640 1 --2- 172.21.15.116:0/1557459576 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f41a80678c0 0x7f41a8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:07.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.791+0000 7f41c17fa640 1 -- 172.21.15.116:0/1557459576 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7612+0+0 (secure 0 0 0) 0x7f41bc121890 con 0x7f41c4150d20 2024-04-25T00:10:07.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.791+0000 7f41c8b21640 1 --2- 172.21.15.116:0/1557459576 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f41a80678c0 0x7f41a8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:07.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.795+0000 7f41c8b21640 1 --2- 172.21.15.116:0/1557459576 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f41a80678c0 0x7f41a8069d80 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f41b8009580 tx=0x7f41b8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:07.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.795+0000 7f41c17fa640 1 -- 172.21.15.116:0/1557459576 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f41bc0c6070 con 0x7f41c4150d20 2024-04-25T00:10:07.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.991+0000 7f41cad85640 1 -- 172.21.15.116:0/1557459576 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f41c41577b0 con 0x7f41c4150d20 2024-04-25T00:10:07.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.991+0000 7f41c17fa640 1 -- 172.21.15.116:0/1557459576 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+6504 (secure 0 0 0) 0x7f41bc016620 con 0x7f41c4150d20 2024-04-25T00:10:07.996 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:10:07.996 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6114,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6135,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1358527848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1358527848},{"type":"v1","addr":"172.21.15.116:6835","nonce":1358527848}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:09:33.601522+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5236":{"gid":5236,"name":"b","rank":0,"incarnation":62,"state":"up:active","state_seq":25,"addr":"172.21.15.174:6837/1159002662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1159002662},{"type":"v1","addr":"172.21.15.174:6837","nonce":1159002662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5236,"qdb_cluster":[5236]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:09:33.601524+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":132,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4235,"qdb_cluster":[4235]},"id":2}]} 2024-04-25T00:10:07.996 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 67 2024-04-25T00:10:07.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.995+0000 7f41cad85640 1 -- 172.21.15.116:0/1557459576 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f41a80678c0 msgr2=0x7f41a8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:07.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.995+0000 7f41cad85640 1 --2- 172.21.15.116:0/1557459576 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f41a80678c0 0x7f41a8069d80 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f41b8009580 tx=0x7f41b8009190 comp rx=0 tx=0).stop 2024-04-25T00:10:07.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.995+0000 7f41cad85640 1 -- 172.21.15.116:0/1557459576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41c4150d20 msgr2=0x7f41c4143ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:07.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.995+0000 7f41cad85640 1 --2- 172.21.15.116:0/1557459576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41c4150d20 0x7f41c4143ac0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f41bc0af040 tx=0x7f41bc0197d0 comp rx=0 tx=0).stop 2024-04-25T00:10:07.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.995+0000 7f41cad85640 1 -- 172.21.15.116:0/1557459576 shutdown_connections 2024-04-25T00:10:07.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.995+0000 7f41cad85640 1 --2- 172.21.15.116:0/1557459576 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f41a80678c0 0x7f41a8069d80 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:07.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.995+0000 7f41cad85640 1 --2- 172.21.15.116:0/1557459576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41c4150d20 0x7f41c4143ac0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:07.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.995+0000 7f41cad85640 1 --2- 172.21.15.116:0/1557459576 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f41c40c2f50 0x7f41c406cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:07.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.995+0000 7f41cad85640 1 --2- 172.21.15.116:0/1557459576 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f41c40a94a0 0x7f41c4154980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:07.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.995+0000 7f41cad85640 1 -- 172.21.15.116:0/1557459576 >> 172.21.15.116:0/1557459576 conn(0x7f41c40c3b00 msgr2=0x7f41c40a9f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:10:07.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.995+0000 7f41cad85640 1 -- 172.21.15.116:0/1557459576 shutdown_connections 2024-04-25T00:10:07.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:07.995+0000 7f41cad85640 1 -- 172.21.15.116:0/1557459576 wait complete. 2024-04-25T00:10:08.014 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T00:10:10.015 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:10:10.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.139+0000 7fa132e32640 1 Processor -- start 2024-04-25T00:10:10.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.139+0000 7fa132e32640 1 -- start start 2024-04-25T00:10:10.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.139+0000 7fa132e32640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fa12c0ae260 0x7fa12c071090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:10.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.139+0000 7fa132e32640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa12c153650 con 0x7fa12c14d810 2024-04-25T00:10:10.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.139+0000 7fa132e32640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa12c153910 con 0x7fa12c0715d0 2024-04-25T00:10:10.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.139+0000 7fa132e32640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fa12c153bf0 con 0x7fa12c0ae260 2024-04-25T00:10:10.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.139+0000 7fa130bce640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fa12c14d810 0x7fa12c0c8340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:51210/0 (socket says 172.21.15.116:51210) 2024-04-25T00:10:10.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.139+0000 7fa130bce640 1 -- 172.21.15.116:0/781554564 learned_addr learned my addr 172.21.15.116:0/781554564 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:10:10.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.139+0000 7fa12bfff640 1 --2- 172.21.15.116:0/781554564 >> v2:172.21.15.174:3300/0 conn(0x7fa12c0ae260 0x7fa12c071090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:10.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.139+0000 7fa12b7fe640 1 -- 172.21.15.116:0/781554564 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1982314595 0 0) 0x7fa12c153650 con 0x7fa12c14d810 2024-04-25T00:10:10.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.139+0000 7fa12b7fe640 1 -- 172.21.15.116:0/781554564 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa114003530 con 0x7fa12c14d810 2024-04-25T00:10:10.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.139+0000 7fa12b7fe640 1 -- 172.21.15.116:0/781554564 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 394479365 0 0) 0x7fa12c153910 con 0x7fa12c0715d0 2024-04-25T00:10:10.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.139+0000 7fa12b7fe640 1 -- 172.21.15.116:0/781554564 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa12c153650 con 0x7fa12c0715d0 2024-04-25T00:10:10.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.139+0000 7fa12bfff640 1 -- 172.21.15.116:0/781554564 >> v1:172.21.15.174:6789/0 conn(0x7fa12c0715d0 legacy=0x7fa12c156c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:10.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.139+0000 7fa12b7fe640 1 -- 172.21.15.116:0/781554564 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4027234607 0 0) 0x7fa114003530 con 0x7fa12c14d810 2024-04-25T00:10:10.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.139+0000 7fa12bfff640 1 -- 172.21.15.116:0/781554564 >> v1:172.21.15.116:6789/0 conn(0x7fa12c14d810 legacy=0x7fa12c0c8340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:10.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.139+0000 7fa12bfff640 1 -- 172.21.15.116:0/781554564 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa12c153ed0 con 0x7fa12c0ae260 2024-04-25T00:10:10.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.143+0000 7fa12bfff640 1 --2- 172.21.15.116:0/781554564 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa12c0ae260 0x7fa12c071090 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa1200097c0 tx=0x7fa12001c2a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3fd03d7c01a9403f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:10.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.143+0000 7fa12b7fe640 1 -- 172.21.15.116:0/781554564 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1200b0070 con 0x7fa12c0ae260 2024-04-25T00:10:10.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.143+0000 7fa12b7fe640 1 -- 172.21.15.116:0/781554564 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1200ab400 con 0x7fa12c0ae260 2024-04-25T00:10:10.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.143+0000 7fa12b7fe640 1 -- 172.21.15.116:0/781554564 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa11c001680 con 0x7fa12c0ae260 2024-04-25T00:10:10.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.143+0000 7fa132e32640 1 -- 172.21.15.116:0/781554564 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa12c0ae260 msgr2=0x7fa12c071090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:10.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.143+0000 7fa132e32640 1 --2- 172.21.15.116:0/781554564 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa12c0ae260 0x7fa12c071090 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa1200097c0 tx=0x7fa12001c2a0 comp rx=0 tx=0).stop 2024-04-25T00:10:10.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.143+0000 7fa132e32640 1 -- 172.21.15.116:0/781554564 shutdown_connections 2024-04-25T00:10:10.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.143+0000 7fa132e32640 1 --2- 172.21.15.116:0/781554564 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa12c0ae260 0x7fa12c071090 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:10.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.143+0000 7fa132e32640 1 -- 172.21.15.116:0/781554564 >> 172.21.15.116:0/781554564 conn(0x7fa12c074a80 msgr2=0x7fa12c0c8f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:10:10.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.143+0000 7fa132e32640 1 -- 172.21.15.116:0/781554564 shutdown_connections 2024-04-25T00:10:10.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.143+0000 7fa132e32640 1 -- 172.21.15.116:0/781554564 wait complete. 2024-04-25T00:10:10.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.143+0000 7fa132e32640 1 Processor -- start 2024-04-25T00:10:10.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.143+0000 7fa132e32640 1 -- start start 2024-04-25T00:10:10.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.143+0000 7fa132e32640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa12c0715d0 0x7fa12c1487e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:10.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.143+0000 7fa132e32640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa12c0ae260 0x7fa12c155640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:10.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.143+0000 7fa132e32640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa12c14d810 0x7fa12c143c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:10.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.143+0000 7fa132e32640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa12c161b90 con 0x7fa12c0715d0 2024-04-25T00:10:10.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.143+0000 7fa132e32640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa12c161a10 con 0x7fa12c0ae260 2024-04-25T00:10:10.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.143+0000 7fa132e32640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa12c161d10 con 0x7fa12c14d810 2024-04-25T00:10:10.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.143+0000 7fa130bce640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa12c0715d0 0x7fa12c1487e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:10.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.143+0000 7fa130bce640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa12c0715d0 0x7fa12c1487e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50718/0 (socket says 172.21.15.116:50718) 2024-04-25T00:10:10.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.143+0000 7fa130bce640 1 -- 172.21.15.116:0/4031691721 learned_addr learned my addr 172.21.15.116:0/4031691721 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:10:10.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.143+0000 7fa12bfff640 1 --2- 172.21.15.116:0/4031691721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa12c0ae260 0x7fa12c155640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:10.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.143+0000 7fa1313cf640 1 --2- 172.21.15.116:0/4031691721 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa12c14d810 0x7fa12c143c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:10.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.143+0000 7fa130bce640 1 -- 172.21.15.116:0/4031691721 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa12c14d810 msgr2=0x7fa12c143c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:10.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.143+0000 7fa130bce640 1 --2- 172.21.15.116:0/4031691721 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa12c14d810 0x7fa12c143c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:10.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.143+0000 7fa130bce640 1 -- 172.21.15.116:0/4031691721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa12c0ae260 msgr2=0x7fa12c155640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:10.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.143+0000 7fa130bce640 1 --2- 172.21.15.116:0/4031691721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa12c0ae260 0x7fa12c155640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:10.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.143+0000 7fa130bce640 1 -- 172.21.15.116:0/4031691721 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa12c1ecee0 con 0x7fa12c0715d0 2024-04-25T00:10:10.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.143+0000 7fa12bfff640 1 --2- 172.21.15.116:0/4031691721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa12c0ae260 0x7fa12c155640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:10:10.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.143+0000 7fa130bce640 1 --2- 172.21.15.116:0/4031691721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa12c0715d0 0x7fa12c1487e0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fa11c00c730 tx=0x7fa11c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:10.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.143+0000 7fa1297fa640 1 -- 172.21.15.116:0/4031691721 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa11c017070 con 0x7fa12c0715d0 2024-04-25T00:10:10.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.143+0000 7fa1297fa640 1 -- 172.21.15.116:0/4031691721 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa11c005b00 con 0x7fa12c0715d0 2024-04-25T00:10:10.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.143+0000 7fa132e32640 1 -- 172.21.15.116:0/4031691721 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa12c14c380 con 0x7fa12c0715d0 2024-04-25T00:10:10.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.143+0000 7fa1297fa640 1 -- 172.21.15.116:0/4031691721 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa11c007070 con 0x7fa12c0715d0 2024-04-25T00:10:10.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.143+0000 7fa132e32640 1 -- 172.21.15.116:0/4031691721 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa12c1ecc00 con 0x7fa12c0715d0 2024-04-25T00:10:10.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.147+0000 7fa132e32640 1 -- 172.21.15.116:0/4031691721 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa12c1ec9f0 con 0x7fa12c0715d0 2024-04-25T00:10:10.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.147+0000 7fa1297fa640 1 -- 172.21.15.116:0/4031691721 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa11c01a070 con 0x7fa12c0715d0 2024-04-25T00:10:10.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.147+0000 7fa1297fa640 1 --2- 172.21.15.116:0/4031691721 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa1180678c0 0x7fa118069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:10.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.147+0000 7fa1297fa640 1 -- 172.21.15.116:0/4031691721 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7612+0+0 (secure 0 0 0) 0x7fa11c0857e0 con 0x7fa12c0715d0 2024-04-25T00:10:10.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.147+0000 7fa12bfff640 1 --2- 172.21.15.116:0/4031691721 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa1180678c0 0x7fa118069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:10.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.147+0000 7fa12bfff640 1 --2- 172.21.15.116:0/4031691721 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa1180678c0 0x7fa118069d80 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fa1200ad000 tx=0x7fa1200a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:10.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.151+0000 7fa1297fa640 1 -- 172.21.15.116:0/4031691721 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fa11c0555f0 con 0x7fa12c0715d0 2024-04-25T00:10:10.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.347+0000 7fa132e32640 1 -- 172.21.15.116:0/4031691721 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa12c068350 con 0x7fa12c0715d0 2024-04-25T00:10:10.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.347+0000 7fa1297fa640 1 -- 172.21.15.116:0/4031691721 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+6504 (secure 0 0 0) 0x7fa11c0598f0 con 0x7fa12c0715d0 2024-04-25T00:10:10.351 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:10:10.351 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6114,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6135,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1358527848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1358527848},{"type":"v1","addr":"172.21.15.116:6835","nonce":1358527848}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:09:33.601522+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5236":{"gid":5236,"name":"b","rank":0,"incarnation":62,"state":"up:active","state_seq":25,"addr":"172.21.15.174:6837/1159002662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1159002662},{"type":"v1","addr":"172.21.15.174:6837","nonce":1159002662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5236,"qdb_cluster":[5236]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:09:33.601524+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":132,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4235,"qdb_cluster":[4235]},"id":2}]} 2024-04-25T00:10:10.352 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 67 2024-04-25T00:10:10.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.351+0000 7fa132e32640 1 -- 172.21.15.116:0/4031691721 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa1180678c0 msgr2=0x7fa118069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:10.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.351+0000 7fa132e32640 1 --2- 172.21.15.116:0/4031691721 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa1180678c0 0x7fa118069d80 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fa1200ad000 tx=0x7fa1200a6000 comp rx=0 tx=0).stop 2024-04-25T00:10:10.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.351+0000 7fa132e32640 1 -- 172.21.15.116:0/4031691721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa12c0715d0 msgr2=0x7fa12c1487e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:10.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.351+0000 7fa132e32640 1 --2- 172.21.15.116:0/4031691721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa12c0715d0 0x7fa12c1487e0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fa11c00c730 tx=0x7fa11c00cbf0 comp rx=0 tx=0).stop 2024-04-25T00:10:10.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.351+0000 7fa132e32640 1 -- 172.21.15.116:0/4031691721 shutdown_connections 2024-04-25T00:10:10.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.351+0000 7fa132e32640 1 --2- 172.21.15.116:0/4031691721 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa1180678c0 0x7fa118069d80 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:10.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.351+0000 7fa132e32640 1 --2- 172.21.15.116:0/4031691721 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa12c14d810 0x7fa12c143c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:10.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.351+0000 7fa132e32640 1 --2- 172.21.15.116:0/4031691721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa12c0ae260 0x7fa12c155640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:10.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.351+0000 7fa132e32640 1 --2- 172.21.15.116:0/4031691721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa12c0715d0 0x7fa12c1487e0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:10.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.351+0000 7fa132e32640 1 -- 172.21.15.116:0/4031691721 >> 172.21.15.116:0/4031691721 conn(0x7fa12c074a80 msgr2=0x7fa12c0c8f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:10:10.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.351+0000 7fa132e32640 1 -- 172.21.15.116:0/4031691721 shutdown_connections 2024-04-25T00:10:10.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:10.351+0000 7fa132e32640 1 -- 172.21.15.116:0/4031691721 wait complete. 2024-04-25T00:10:10.370 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T00:10:12.371 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:10:12.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.499+0000 7f369ca38640 1 Processor -- start 2024-04-25T00:10:12.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.499+0000 7f369ca38640 1 -- start start 2024-04-25T00:10:12.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.499+0000 7f369ca38640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3698152d80 0x7f369806bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:12.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.499+0000 7f369ca38640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36980c9150 con 0x7f369806c290 2024-04-25T00:10:12.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.499+0000 7f369ca38640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36980c9430 con 0x7f369806d5f0 2024-04-25T00:10:12.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.499+0000 7f369ca38640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f36980c9d60 con 0x7f3698152d80 2024-04-25T00:10:12.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.499+0000 7f369659c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3698152d80 0x7f369806bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:12.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.499+0000 7f3696d9d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f369806c290 0x7f3698157390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:51216/0 (socket says 172.21.15.116:51216) 2024-04-25T00:10:12.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.499+0000 7f369659c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3698152d80 0x7f369806bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50730/0 (socket says 172.21.15.116:50730) 2024-04-25T00:10:12.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.499+0000 7f3696d9d640 1 -- 172.21.15.116:0/3126959027 learned_addr learned my addr 172.21.15.116:0/3126959027 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:10:12.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.499+0000 7f369659c640 1 -- 172.21.15.116:0/3126959027 >> v1:172.21.15.174:6789/0 conn(0x7f369806d5f0 legacy=0x7f3698156c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:12.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.499+0000 7f369659c640 1 -- 172.21.15.116:0/3126959027 >> v1:172.21.15.116:6789/0 conn(0x7f369806c290 legacy=0x7f3698157390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:12.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.499+0000 7f369659c640 1 -- 172.21.15.116:0/3126959027 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36980c9710 con 0x7f3698152d80 2024-04-25T00:10:12.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.499+0000 7f3695d9b640 1 -- 172.21.15.116:0/3126959027 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 321356711 0 0) 0x7f36980c9150 con 0x7f369806c290 2024-04-25T00:10:12.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.503+0000 7f369659c640 1 --2- 172.21.15.116:0/3126959027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3698152d80 0x7f369806bd50 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f36880097f0 tx=0x7f368801c350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67194f0ac0c4ea88 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:12.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.503+0000 7f3695d9b640 1 -- 172.21.15.116:0/3126959027 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36880b0070 con 0x7f3698152d80 2024-04-25T00:10:12.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.503+0000 7f3695d9b640 1 -- 172.21.15.116:0/3126959027 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36880ab600 con 0x7f3698152d80 2024-04-25T00:10:12.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.503+0000 7f3695d9b640 1 -- 172.21.15.116:0/3126959027 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36880ab9d0 con 0x7f3698152d80 2024-04-25T00:10:12.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.503+0000 7f369ca38640 1 -- 172.21.15.116:0/3126959027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3698152d80 msgr2=0x7f369806bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:12.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.503+0000 7f369ca38640 1 --2- 172.21.15.116:0/3126959027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3698152d80 0x7f369806bd50 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f36880097f0 tx=0x7f368801c350 comp rx=0 tx=0).stop 2024-04-25T00:10:12.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.503+0000 7f369ca38640 1 -- 172.21.15.116:0/3126959027 shutdown_connections 2024-04-25T00:10:12.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.503+0000 7f369ca38640 1 --2- 172.21.15.116:0/3126959027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3698152d80 0x7f369806bd50 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:12.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.503+0000 7f369ca38640 1 -- 172.21.15.116:0/3126959027 >> 172.21.15.116:0/3126959027 conn(0x7f36980acdd0 msgr2=0x7f36980ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:10:12.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.503+0000 7f369ca38640 1 -- 172.21.15.116:0/3126959027 shutdown_connections 2024-04-25T00:10:12.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.503+0000 7f369ca38640 1 -- 172.21.15.116:0/3126959027 wait complete. 2024-04-25T00:10:12.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.503+0000 7f369ca38640 1 Processor -- start 2024-04-25T00:10:12.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.503+0000 7f369ca38640 1 -- start start 2024-04-25T00:10:12.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.503+0000 7f369ca38640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f369806c290 0x7f36980aaa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:12.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.503+0000 7f369ca38640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f369806d5f0 0x7f3698151e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:12.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.503+0000 7f369ca38640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3698152d80 0x7f3698154740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:12.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.503+0000 7f369ca38640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3698163a60 con 0x7f3698152d80 2024-04-25T00:10:12.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.503+0000 7f369ca38640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f36981638e0 con 0x7f369806d5f0 2024-04-25T00:10:12.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.503+0000 7f369ca38640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f3698163be0 con 0x7f369806c290 2024-04-25T00:10:12.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.503+0000 7f3696d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3698152d80 0x7f3698154740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:12.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.503+0000 7f3696d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3698152d80 0x7f3698154740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50744/0 (socket says 172.21.15.116:50744) 2024-04-25T00:10:12.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.503+0000 7f3696d9d640 1 -- 172.21.15.116:0/1665038701 learned_addr learned my addr 172.21.15.116:0/1665038701 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:10:12.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.503+0000 7f368ffff640 1 --2- 172.21.15.116:0/1665038701 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f369806d5f0 0x7f3698151e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:12.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.507+0000 7f369659c640 1 --2- 172.21.15.116:0/1665038701 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f369806c290 0x7f36980aaa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:12.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.507+0000 7f3696d9d640 1 -- 172.21.15.116:0/1665038701 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f369806c290 msgr2=0x7f36980aaa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:12.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.507+0000 7f3696d9d640 1 --2- 172.21.15.116:0/1665038701 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f369806c290 0x7f36980aaa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:12.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.507+0000 7f3696d9d640 1 -- 172.21.15.116:0/1665038701 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f369806d5f0 msgr2=0x7f3698151e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:12.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.507+0000 7f3696d9d640 1 --2- 172.21.15.116:0/1665038701 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f369806d5f0 0x7f3698151e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:12.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.507+0000 7f3696d9d640 1 -- 172.21.15.116:0/1665038701 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3698153d90 con 0x7f3698152d80 2024-04-25T00:10:12.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.507+0000 7f3696d9d640 1 --2- 172.21.15.116:0/1665038701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3698152d80 0x7f3698154740 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f369000c700 tx=0x7f369000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:12.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.507+0000 7f368effd640 1 -- 172.21.15.116:0/1665038701 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3690018070 con 0x7f3698152d80 2024-04-25T00:10:12.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.507+0000 7f368effd640 1 -- 172.21.15.116:0/1665038701 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f369000ce10 con 0x7f3698152d80 2024-04-25T00:10:12.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.507+0000 7f369ca38640 1 -- 172.21.15.116:0/1665038701 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36981432f0 con 0x7f3698152d80 2024-04-25T00:10:12.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.507+0000 7f368effd640 1 -- 172.21.15.116:0/1665038701 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f369000a880 con 0x7f3698152d80 2024-04-25T00:10:12.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.507+0000 7f369ca38640 1 -- 172.21.15.116:0/1665038701 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3698151650 con 0x7f3698152d80 2024-04-25T00:10:12.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.507+0000 7f369ca38640 1 -- 172.21.15.116:0/1665038701 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3698068350 con 0x7f3698152d80 2024-04-25T00:10:12.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.507+0000 7f368effd640 1 -- 172.21.15.116:0/1665038701 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f369001b070 con 0x7f3698152d80 2024-04-25T00:10:12.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.507+0000 7f368effd640 1 --2- 172.21.15.116:0/1665038701 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f36780678c0 0x7f3678069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:12.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.507+0000 7f368effd640 1 -- 172.21.15.116:0/1665038701 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7612+0+0 (secure 0 0 0) 0x7f36900858a0 con 0x7f3698152d80 2024-04-25T00:10:12.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.507+0000 7f369659c640 1 --2- 172.21.15.116:0/1665038701 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f36780678c0 0x7f3678069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:12.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.507+0000 7f369659c640 1 --2- 172.21.15.116:0/1665038701 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f36780678c0 0x7f3678069d80 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f36880ad000 tx=0x7f36880a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:12.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.511+0000 7f368effd640 1 -- 172.21.15.116:0/1665038701 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f3690055d00 con 0x7f3698152d80 2024-04-25T00:10:12.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.707+0000 7f369ca38640 1 -- 172.21.15.116:0/1665038701 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f36980cb270 con 0x7f3698152d80 2024-04-25T00:10:12.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.707+0000 7f368effd640 1 -- 172.21.15.116:0/1665038701 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+6504 (secure 0 0 0) 0x7f3690055600 con 0x7f3698152d80 2024-04-25T00:10:12.709 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:10:12.709 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6114,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6135,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1358527848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1358527848},{"type":"v1","addr":"172.21.15.116:6835","nonce":1358527848}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:09:33.601522+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5236":{"gid":5236,"name":"b","rank":0,"incarnation":62,"state":"up:active","state_seq":25,"addr":"172.21.15.174:6837/1159002662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1159002662},{"type":"v1","addr":"172.21.15.174:6837","nonce":1159002662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5236,"qdb_cluster":[5236]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:09:33.601524+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":132,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4235,"qdb_cluster":[4235]},"id":2}]} 2024-04-25T00:10:12.710 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 67 2024-04-25T00:10:12.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.707+0000 7f369ca38640 1 -- 172.21.15.116:0/1665038701 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f36780678c0 msgr2=0x7f3678069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:12.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.707+0000 7f369ca38640 1 --2- 172.21.15.116:0/1665038701 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f36780678c0 0x7f3678069d80 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f36880ad000 tx=0x7f36880a6000 comp rx=0 tx=0).stop 2024-04-25T00:10:12.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.707+0000 7f369ca38640 1 -- 172.21.15.116:0/1665038701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3698152d80 msgr2=0x7f3698154740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:12.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.707+0000 7f369ca38640 1 --2- 172.21.15.116:0/1665038701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3698152d80 0x7f3698154740 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f369000c700 tx=0x7f369000cbc0 comp rx=0 tx=0).stop 2024-04-25T00:10:12.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.707+0000 7f369ca38640 1 -- 172.21.15.116:0/1665038701 shutdown_connections 2024-04-25T00:10:12.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.707+0000 7f369ca38640 1 --2- 172.21.15.116:0/1665038701 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f36780678c0 0x7f3678069d80 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:12.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.707+0000 7f369ca38640 1 --2- 172.21.15.116:0/1665038701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3698152d80 0x7f3698154740 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:12.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.707+0000 7f369ca38640 1 --2- 172.21.15.116:0/1665038701 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f369806d5f0 0x7f3698151e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:12.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.707+0000 7f369ca38640 1 --2- 172.21.15.116:0/1665038701 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f369806c290 0x7f36980aaa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:12.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.707+0000 7f369ca38640 1 -- 172.21.15.116:0/1665038701 >> 172.21.15.116:0/1665038701 conn(0x7f36980acdd0 msgr2=0x7f369814da10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:10:12.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.707+0000 7f369ca38640 1 -- 172.21.15.116:0/1665038701 shutdown_connections 2024-04-25T00:10:12.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:12.707+0000 7f369ca38640 1 -- 172.21.15.116:0/1665038701 wait complete. 2024-04-25T00:10:12.727 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T00:10:14.728 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:10:14.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.859+0000 7f2087507640 1 Processor -- start 2024-04-25T00:10:14.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.859+0000 7f2087507640 1 -- start start 2024-04-25T00:10:14.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.859+0000 7f2087507640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2080155b20 0x7f2080152c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:14.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.859+0000 7f2087507640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f20800aacd0 0x7f2080153180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:14.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.859+0000 7f2087507640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20800a9e60 con 0x7f208014c700 2024-04-25T00:10:14.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.859+0000 7f2087507640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f20801527b0 con 0x7f20800aacd0 2024-04-25T00:10:14.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.859+0000 7f2087507640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f20800c3ef0 con 0x7f2080155b20 2024-04-25T00:10:14.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.859+0000 7f20852a3640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f208014c700 0x7f20800c5090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:46156/0 (socket says 172.21.15.116:46156) 2024-04-25T00:10:14.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.859+0000 7f20852a3640 1 -- 172.21.15.116:0/3844153561 learned_addr learned my addr 172.21.15.116:0/3844153561 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:10:14.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.859+0000 7f2084aa2640 1 --2- 172.21.15.116:0/3844153561 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2080155b20 0x7f2080152c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:14.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.859+0000 7f2085aa4640 1 --2- 172.21.15.116:0/3844153561 >> v2:172.21.15.174:3300/0 conn(0x7f20800aacd0 0x7f2080153180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:14.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.859+0000 7f207ffff640 1 -- 172.21.15.116:0/3844153561 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1859127280 0 0) 0x7f20800a9e60 con 0x7f208014c700 2024-04-25T00:10:14.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.859+0000 7f207ffff640 1 -- 172.21.15.116:0/3844153561 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2068005740 con 0x7f208014c700 2024-04-25T00:10:14.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.859+0000 7f207ffff640 1 -- 172.21.15.116:0/3844153561 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3619288594 0 0) 0x7f2068005740 con 0x7f208014c700 2024-04-25T00:10:14.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.859+0000 7f207ffff640 1 -- 172.21.15.116:0/3844153561 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2080155b20 msgr2=0x7f2080152c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:10:14.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.859+0000 7f207ffff640 1 --2- 172.21.15.116:0/3844153561 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2080155b20 0x7f2080152c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:14.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.859+0000 7f207ffff640 1 -- 172.21.15.116:0/3844153561 >> v2:172.21.15.174:3300/0 conn(0x7f20800aacd0 msgr2=0x7f2080153180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:10:14.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.859+0000 7f207ffff640 1 --2- 172.21.15.116:0/3844153561 >> v2:172.21.15.174:3300/0 conn(0x7f20800aacd0 0x7f2080153180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:14.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.859+0000 7f207ffff640 1 -- 172.21.15.116:0/3844153561 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20800c41d0 con 0x7f208014c700 2024-04-25T00:10:14.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.859+0000 7f207ffff640 1 -- 172.21.15.116:0/3844153561 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f2070001a00 con 0x7f208014c700 2024-04-25T00:10:14.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.859+0000 7f207ffff640 1 -- 172.21.15.116:0/3844153561 >> v1:172.21.15.116:6789/0 conn(0x7f208014c700 legacy=0x7f20800c5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:14.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.859+0000 7f207ffff640 1 --2- 172.21.15.116:0/3844153561 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2080155b20 0x7f2068005bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:14.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.859+0000 7f207ffff640 1 --2- 172.21.15.116:0/3844153561 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2068006170 0x7f2068006a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:14.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.859+0000 7f207ffff640 1 --2- 172.21.15.116:0/3844153561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2068006f60 0x7f20680073e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:14.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.859+0000 7f207ffff640 1 -- 172.21.15.116:0/3844153561 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f20801527b0 con 0x7f2068006f60 2024-04-25T00:10:14.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.859+0000 7f207ffff640 1 -- 172.21.15.116:0/3844153561 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f20800c3ef0 con 0x7f2068006170 2024-04-25T00:10:14.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.859+0000 7f207ffff640 1 -- 172.21.15.116:0/3844153561 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f206800dc90 con 0x7f2080155b20 2024-04-25T00:10:14.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.859+0000 7f20852a3640 1 --2- 172.21.15.116:0/3844153561 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2080155b20 0x7f2068005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:14.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.859+0000 7f2085aa4640 1 --2- 172.21.15.116:0/3844153561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2068006f60 0x7f20680073e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:14.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.859+0000 7f2084aa2640 1 --2- 172.21.15.116:0/3844153561 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2068006170 0x7f2068006a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:14.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.859+0000 7f2085aa4640 1 -- 172.21.15.116:0/3844153561 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2080155b20 msgr2=0x7f2068005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:10:14.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.863+0000 7f2085aa4640 1 --2- 172.21.15.116:0/3844153561 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2080155b20 0x7f2068005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:14.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.863+0000 7f2085aa4640 1 -- 172.21.15.116:0/3844153561 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2068006170 msgr2=0x7f2068006a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:10:14.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.863+0000 7f2085aa4640 1 --2- 172.21.15.116:0/3844153561 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2068006170 0x7f2068006a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:14.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.863+0000 7f2085aa4640 1 -- 172.21.15.116:0/3844153561 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20800c41d0 con 0x7f2068006f60 2024-04-25T00:10:14.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.863+0000 7f2085aa4640 1 --2- 172.21.15.116:0/3844153561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2068006f60 0x7f20680073e0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f2078009ae0 tx=0x7f207801cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4e779ef9e518412 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:14.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.863+0000 7f207ffff640 1 -- 172.21.15.116:0/3844153561 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20780096b0 con 0x7f2068006f60 2024-04-25T00:10:14.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.863+0000 7f207ffff640 1 -- 172.21.15.116:0/3844153561 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20780b0400 con 0x7f2068006f60 2024-04-25T00:10:14.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.863+0000 7f207ffff640 1 -- 172.21.15.116:0/3844153561 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20780b07f0 con 0x7f2068006f60 2024-04-25T00:10:14.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.863+0000 7f2087507640 1 -- 172.21.15.116:0/3844153561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2068006f60 msgr2=0x7f20680073e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:14.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.863+0000 7f2087507640 1 --2- 172.21.15.116:0/3844153561 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2068006f60 0x7f20680073e0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f2078009ae0 tx=0x7f207801cab0 comp rx=0 tx=0).stop 2024-04-25T00:10:14.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.863+0000 7f2085aa4640 1 -- 172.21.15.116:0/3844153561 reap_dead start 2024-04-25T00:10:14.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.863+0000 7f2087507640 1 -- 172.21.15.116:0/3844153561 shutdown_connections 2024-04-25T00:10:14.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.863+0000 7f2087507640 1 -- 172.21.15.116:0/3844153561 >> 172.21.15.116:0/3844153561 conn(0x7f208006c720 msgr2=0x7f208006cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:10:14.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.863+0000 7f2087507640 1 -- 172.21.15.116:0/3844153561 shutdown_connections 2024-04-25T00:10:14.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.863+0000 7f2087507640 1 -- 172.21.15.116:0/3844153561 wait complete. 2024-04-25T00:10:14.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.867+0000 7f2087507640 1 Processor -- start 2024-04-25T00:10:14.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.867+0000 7f2087507640 1 -- start start 2024-04-25T00:10:14.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.867+0000 7f2087507640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2068006f60 0x7f20801543a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:14.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.867+0000 7f2087507640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f20800bd0c0 0x7f20801ea7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:14.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.867+0000 7f2087507640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20801548e0 0x7f20801eacf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:14.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.867+0000 7f2087507640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f208006bdb0 con 0x7f20801548e0 2024-04-25T00:10:14.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.867+0000 7f2087507640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f208006bc30 con 0x7f20800bd0c0 2024-04-25T00:10:14.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.867+0000 7f2087507640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f208006bf30 con 0x7f2068006f60 2024-04-25T00:10:14.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.867+0000 7f2085aa4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20801548e0 0x7f20801eacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:14.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.867+0000 7f2085aa4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20801548e0 0x7f20801eacf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48584/0 (socket says 172.21.15.116:48584) 2024-04-25T00:10:14.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.867+0000 7f2085aa4640 1 -- 172.21.15.116:0/1487152793 learned_addr learned my addr 172.21.15.116:0/1487152793 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:10:14.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.867+0000 7f2084aa2640 1 --2- 172.21.15.116:0/1487152793 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f20800bd0c0 0x7f20801ea7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:14.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.867+0000 7f20852a3640 1 --2- 172.21.15.116:0/1487152793 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2068006f60 0x7f20801543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:14.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.867+0000 7f2085aa4640 1 -- 172.21.15.116:0/1487152793 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2068006f60 msgr2=0x7f20801543a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:14.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.867+0000 7f2085aa4640 1 --2- 172.21.15.116:0/1487152793 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2068006f60 0x7f20801543a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:14.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.867+0000 7f2085aa4640 1 -- 172.21.15.116:0/1487152793 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f20800bd0c0 msgr2=0x7f20801ea7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:14.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.867+0000 7f2085aa4640 1 --2- 172.21.15.116:0/1487152793 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f20800bd0c0 0x7f20801ea7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:14.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.867+0000 7f2085aa4640 1 -- 172.21.15.116:0/1487152793 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f208006f5b0 con 0x7f20801548e0 2024-04-25T00:10:14.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.867+0000 7f2085aa4640 1 --2- 172.21.15.116:0/1487152793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20801548e0 0x7f20801eacf0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f207801d350 tx=0x7f2078019850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:14.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.867+0000 7f207dffb640 1 -- 172.21.15.116:0/1487152793 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20780096b0 con 0x7f20801548e0 2024-04-25T00:10:14.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.867+0000 7f207dffb640 1 -- 172.21.15.116:0/1487152793 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20780b1ac0 con 0x7f20801548e0 2024-04-25T00:10:14.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.867+0000 7f2087507640 1 -- 172.21.15.116:0/1487152793 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f208014bd20 con 0x7f20801548e0 2024-04-25T00:10:14.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.867+0000 7f207dffb640 1 -- 172.21.15.116:0/1487152793 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2078004030 con 0x7f20801548e0 2024-04-25T00:10:14.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.867+0000 7f2087507640 1 -- 172.21.15.116:0/1487152793 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f208014bfc0 con 0x7f20801548e0 2024-04-25T00:10:14.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.867+0000 7f2087507640 1 -- 172.21.15.116:0/1487152793 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2080068a60 con 0x7f20801548e0 2024-04-25T00:10:14.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.871+0000 7f207dffb640 1 -- 172.21.15.116:0/1487152793 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f20780041d0 con 0x7f20801548e0 2024-04-25T00:10:14.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.871+0000 7f207dffb640 1 --2- 172.21.15.116:0/1487152793 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f206c0678f0 0x7f206c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:14.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.871+0000 7f207dffb640 1 -- 172.21.15.116:0/1487152793 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7612+0+0 (secure 0 0 0) 0x7f207811ff50 con 0x7f20801548e0 2024-04-25T00:10:14.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.871+0000 7f20852a3640 1 --2- 172.21.15.116:0/1487152793 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f206c0678f0 0x7f206c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:14.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.871+0000 7f20852a3640 1 --2- 172.21.15.116:0/1487152793 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f206c0678f0 0x7f206c069db0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f2070001470 tx=0x7f2070006040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:14.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:14.875+0000 7f207dffb640 1 -- 172.21.15.116:0/1487152793 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f20780ac030 con 0x7f20801548e0 2024-04-25T00:10:15.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:15.063+0000 7f2087507640 1 -- 172.21.15.116:0/1487152793 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f20800ad630 con 0x7f20801548e0 2024-04-25T00:10:15.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:15.063+0000 7f207dffb640 1 -- 172.21.15.116:0/1487152793 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+6504 (secure 0 0 0) 0x7f20780167c0 con 0x7f20801548e0 2024-04-25T00:10:15.067 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:10:15.068 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6114,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6135,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1358527848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1358527848},{"type":"v1","addr":"172.21.15.116:6835","nonce":1358527848}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:09:33.601522+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5236":{"gid":5236,"name":"b","rank":0,"incarnation":62,"state":"up:active","state_seq":25,"addr":"172.21.15.174:6837/1159002662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1159002662},{"type":"v1","addr":"172.21.15.174:6837","nonce":1159002662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5236,"qdb_cluster":[5236]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:09:33.601524+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":132,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4235,"qdb_cluster":[4235]},"id":2}]} 2024-04-25T00:10:15.068 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 67 2024-04-25T00:10:15.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:15.067+0000 7f2087507640 1 -- 172.21.15.116:0/1487152793 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f206c0678f0 msgr2=0x7f206c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:15.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:15.067+0000 7f2087507640 1 --2- 172.21.15.116:0/1487152793 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f206c0678f0 0x7f206c069db0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f2070001470 tx=0x7f2070006040 comp rx=0 tx=0).stop 2024-04-25T00:10:15.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:15.067+0000 7f2087507640 1 -- 172.21.15.116:0/1487152793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20801548e0 msgr2=0x7f20801eacf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:15.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:15.067+0000 7f2087507640 1 --2- 172.21.15.116:0/1487152793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20801548e0 0x7f20801eacf0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f207801d350 tx=0x7f2078019850 comp rx=0 tx=0).stop 2024-04-25T00:10:15.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:15.067+0000 7f2087507640 1 -- 172.21.15.116:0/1487152793 shutdown_connections 2024-04-25T00:10:15.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:15.067+0000 7f2087507640 1 --2- 172.21.15.116:0/1487152793 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f206c0678f0 0x7f206c069db0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:15.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:15.067+0000 7f2087507640 1 --2- 172.21.15.116:0/1487152793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20801548e0 0x7f20801eacf0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:15.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:15.067+0000 7f2087507640 1 --2- 172.21.15.116:0/1487152793 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f20800bd0c0 0x7f20801ea7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:15.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:15.067+0000 7f2087507640 1 --2- 172.21.15.116:0/1487152793 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2068006f60 0x7f20801543a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:15.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:15.067+0000 7f2087507640 1 -- 172.21.15.116:0/1487152793 >> 172.21.15.116:0/1487152793 conn(0x7f208006c720 msgr2=0x7f2080158650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:10:15.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:15.067+0000 7f2087507640 1 -- 172.21.15.116:0/1487152793 shutdown_connections 2024-04-25T00:10:15.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:15.067+0000 7f2087507640 1 -- 172.21.15.116:0/1487152793 wait complete. 2024-04-25T00:10:15.085 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T00:10:17.085 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:10:17.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.215+0000 7f7dc05de640 1 Processor -- start 2024-04-25T00:10:17.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.219+0000 7f7dc05de640 1 -- start start 2024-04-25T00:10:17.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.219+0000 7f7dc05de640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7db814c720 0x7f7db81563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:17.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.219+0000 7f7dc05de640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7db80a94a0 0x7f7db80c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:17.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.219+0000 7f7dc05de640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f7db80c2f50 0x7f7db8071680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:17.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.219+0000 7f7dc05de640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f7db8153460 con 0x7f7db80a94a0 2024-04-25T00:10:17.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.219+0000 7f7dc05de640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f7db8071220 con 0x7f7db80c2f50 2024-04-25T00:10:17.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.219+0000 7f7dc05de640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f7db8152420 con 0x7f7db814c720 2024-04-25T00:10:17.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.219+0000 7f7dbdb79640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7db80a94a0 0x7f7db80c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:17.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.219+0000 7f7dbdb79640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7db80a94a0 0x7f7db80c2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48588/0 (socket says 172.21.15.116:48588) 2024-04-25T00:10:17.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.219+0000 7f7dbdb79640 1 -- 172.21.15.116:0/3807045026 learned_addr learned my addr 172.21.15.116:0/3807045026 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:10:17.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.219+0000 7f7dbeb7b640 1 --2- 172.21.15.116:0/3807045026 >> v2:172.21.15.174:3300/0 conn(0x7f7db80c2f50 0x7f7db8071680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:17.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.219+0000 7f7dbe37a640 1 --2- 172.21.15.116:0/3807045026 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7db814c720 0x7f7db81563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:17.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.219+0000 7f7dbdb79640 1 -- 172.21.15.116:0/3807045026 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7db814c720 msgr2=0x7f7db81563d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:10:17.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.219+0000 7f7dbdb79640 1 --2- 172.21.15.116:0/3807045026 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7db814c720 0x7f7db81563d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:17.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.219+0000 7f7dbdb79640 1 -- 172.21.15.116:0/3807045026 >> v2:172.21.15.174:3300/0 conn(0x7f7db80c2f50 msgr2=0x7f7db8071680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:10:17.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.219+0000 7f7dbdb79640 1 --2- 172.21.15.116:0/3807045026 >> v2:172.21.15.174:3300/0 conn(0x7f7db80c2f50 0x7f7db8071680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:17.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.219+0000 7f7dbdb79640 1 -- 172.21.15.116:0/3807045026 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7db814cb00 con 0x7f7db80a94a0 2024-04-25T00:10:17.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.219+0000 7f7dbdb79640 1 --2- 172.21.15.116:0/3807045026 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7db80a94a0 0x7f7db80c2a10 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f7db80c35f0 tx=0x7f7da801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7c2ec7d3d5f0c67 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:17.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.219+0000 7f7dbd378640 1 -- 172.21.15.116:0/3807045026 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7da80b1070 con 0x7f7db80a94a0 2024-04-25T00:10:17.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.223+0000 7f7dbd378640 1 -- 172.21.15.116:0/3807045026 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7da80ac640 con 0x7f7db80a94a0 2024-04-25T00:10:17.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.223+0000 7f7dbd378640 1 -- 172.21.15.116:0/3807045026 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7da80aca10 con 0x7f7db80a94a0 2024-04-25T00:10:17.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.223+0000 7f7dc05de640 1 -- 172.21.15.116:0/3807045026 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7db80a94a0 msgr2=0x7f7db80c2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:17.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.223+0000 7f7dc05de640 1 --2- 172.21.15.116:0/3807045026 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7db80a94a0 0x7f7db80c2a10 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f7db80c35f0 tx=0x7f7da801c460 comp rx=0 tx=0).stop 2024-04-25T00:10:17.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.223+0000 7f7dc05de640 1 -- 172.21.15.116:0/3807045026 shutdown_connections 2024-04-25T00:10:17.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.223+0000 7f7dc05de640 1 --2- 172.21.15.116:0/3807045026 >> v2:172.21.15.174:3300/0 conn(0x7f7db80c2f50 0x7f7db8071680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:17.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.223+0000 7f7dc05de640 1 --2- 172.21.15.116:0/3807045026 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7db80a94a0 0x7f7db80c2a10 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:17.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.223+0000 7f7dc05de640 1 --2- 172.21.15.116:0/3807045026 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7db814c720 0x7f7db81563d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:17.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.223+0000 7f7dc05de640 1 -- 172.21.15.116:0/3807045026 >> 172.21.15.116:0/3807045026 conn(0x7f7db80c3b00 msgr2=0x7f7db80c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:10:17.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.223+0000 7f7dc05de640 1 -- 172.21.15.116:0/3807045026 shutdown_connections 2024-04-25T00:10:17.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.223+0000 7f7dc05de640 1 -- 172.21.15.116:0/3807045026 wait complete. 2024-04-25T00:10:17.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.223+0000 7f7dc05de640 1 Processor -- start 2024-04-25T00:10:17.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.223+0000 7f7dc05de640 1 -- start start 2024-04-25T00:10:17.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.223+0000 7f7dc05de640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7db80a94a0 0x7f7db8140d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:17.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.223+0000 7f7dc05de640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7db80c2f50 0x7f7db8154860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:17.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.223+0000 7f7dc05de640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7db814c720 0x7f7db8142280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:17.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.223+0000 7f7dc05de640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7db8071fd0 con 0x7f7db80c2f50 2024-04-25T00:10:17.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.223+0000 7f7dc05de640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7db8152f40 con 0x7f7db814c720 2024-04-25T00:10:17.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.223+0000 7f7dc05de640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f7db8072150 con 0x7f7db80a94a0 2024-04-25T00:10:17.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.223+0000 7f7dbdb79640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7db80c2f50 0x7f7db8154860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:17.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.223+0000 7f7dbdb79640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7db80c2f50 0x7f7db8154860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48594/0 (socket says 172.21.15.116:48594) 2024-04-25T00:10:17.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.223+0000 7f7dbdb79640 1 -- 172.21.15.116:0/1023038193 learned_addr learned my addr 172.21.15.116:0/1023038193 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:10:17.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.223+0000 7f7dbeb7b640 1 --2- 172.21.15.116:0/1023038193 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7db814c720 0x7f7db8142280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:17.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.223+0000 7f7dbe37a640 1 --2- 172.21.15.116:0/1023038193 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7db80a94a0 0x7f7db8140d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:17.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.223+0000 7f7dbdb79640 1 -- 172.21.15.116:0/1023038193 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7db80a94a0 msgr2=0x7f7db8140d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:17.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.223+0000 7f7dbdb79640 1 --2- 172.21.15.116:0/1023038193 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7db80a94a0 0x7f7db8140d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:17.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.223+0000 7f7dbdb79640 1 -- 172.21.15.116:0/1023038193 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7db814c720 msgr2=0x7f7db8142280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:17.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.223+0000 7f7dbdb79640 1 --2- 172.21.15.116:0/1023038193 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7db814c720 0x7f7db8142280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:17.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.223+0000 7f7dbdb79640 1 -- 172.21.15.116:0/1023038193 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7db80ab1c0 con 0x7f7db80c2f50 2024-04-25T00:10:17.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.223+0000 7f7dbe37a640 1 --2- 172.21.15.116:0/1023038193 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7db80a94a0 0x7f7db8140d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:10:17.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.223+0000 7f7dbdb79640 1 --2- 172.21.15.116:0/1023038193 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7db80c2f50 0x7f7db8154860 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f7da801ab10 tx=0x7f7da8019200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:17.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.223+0000 7f7daeffd640 1 -- 172.21.15.116:0/1023038193 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7da80b1040 con 0x7f7db80c2f50 2024-04-25T00:10:17.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.223+0000 7f7daeffd640 1 -- 172.21.15.116:0/1023038193 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7da80ba070 con 0x7f7db80c2f50 2024-04-25T00:10:17.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.223+0000 7f7dc05de640 1 -- 172.21.15.116:0/1023038193 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7db80ab4a0 con 0x7f7db80c2f50 2024-04-25T00:10:17.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.223+0000 7f7daeffd640 1 -- 172.21.15.116:0/1023038193 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7da80b5660 con 0x7f7db80c2f50 2024-04-25T00:10:17.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.223+0000 7f7dc05de640 1 -- 172.21.15.116:0/1023038193 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7db80aba60 con 0x7f7db80c2f50 2024-04-25T00:10:17.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.227+0000 7f7dc05de640 1 -- 172.21.15.116:0/1023038193 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7db8068350 con 0x7f7db80c2f50 2024-04-25T00:10:17.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.227+0000 7f7daeffd640 1 -- 172.21.15.116:0/1023038193 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7da80b5800 con 0x7f7db80c2f50 2024-04-25T00:10:17.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.227+0000 7f7daeffd640 1 --2- 172.21.15.116:0/1023038193 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7d9c0678c0 0x7f7d9c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:17.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.227+0000 7f7daeffd640 1 -- 172.21.15.116:0/1023038193 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7612+0+0 (secure 0 0 0) 0x7f7da8120ba0 con 0x7f7db80c2f50 2024-04-25T00:10:17.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.227+0000 7f7dbe37a640 1 --2- 172.21.15.116:0/1023038193 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7d9c0678c0 0x7f7d9c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:17.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.227+0000 7f7dbe37a640 1 --2- 172.21.15.116:0/1023038193 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7d9c0678c0 0x7f7d9c069d80 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f7db4009720 tx=0x7f7db4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:17.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.231+0000 7f7daeffd640 1 -- 172.21.15.116:0/1023038193 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f7da8016110 con 0x7f7db80c2f50 2024-04-25T00:10:17.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.431+0000 7f7dc05de640 1 -- 172.21.15.116:0/1023038193 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7db80c3640 con 0x7f7db80c2f50 2024-04-25T00:10:17.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.431+0000 7f7daeffd640 1 -- 172.21.15.116:0/1023038193 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+6504 (secure 0 0 0) 0x7f7da80f10e0 con 0x7f7db80c2f50 2024-04-25T00:10:17.436 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:10:17.436 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6114,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6135,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1358527848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1358527848},{"type":"v1","addr":"172.21.15.116:6835","nonce":1358527848}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:09:33.601522+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5236":{"gid":5236,"name":"b","rank":0,"incarnation":62,"state":"up:active","state_seq":25,"addr":"172.21.15.174:6837/1159002662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1159002662},{"type":"v1","addr":"172.21.15.174:6837","nonce":1159002662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5236,"qdb_cluster":[5236]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:09:33.601524+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":132,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4235,"qdb_cluster":[4235]},"id":2}]} 2024-04-25T00:10:17.436 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 67 2024-04-25T00:10:17.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.435+0000 7f7dc05de640 1 -- 172.21.15.116:0/1023038193 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7d9c0678c0 msgr2=0x7f7d9c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:17.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.435+0000 7f7dc05de640 1 --2- 172.21.15.116:0/1023038193 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7d9c0678c0 0x7f7d9c069d80 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f7db4009720 tx=0x7f7db4008040 comp rx=0 tx=0).stop 2024-04-25T00:10:17.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.435+0000 7f7dc05de640 1 -- 172.21.15.116:0/1023038193 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7db80c2f50 msgr2=0x7f7db8154860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:17.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.435+0000 7f7dc05de640 1 --2- 172.21.15.116:0/1023038193 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7db80c2f50 0x7f7db8154860 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f7da801ab10 tx=0x7f7da8019200 comp rx=0 tx=0).stop 2024-04-25T00:10:17.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.435+0000 7f7dc05de640 1 -- 172.21.15.116:0/1023038193 shutdown_connections 2024-04-25T00:10:17.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.435+0000 7f7dc05de640 1 --2- 172.21.15.116:0/1023038193 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7d9c0678c0 0x7f7d9c069d80 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:17.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.435+0000 7f7dc05de640 1 --2- 172.21.15.116:0/1023038193 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7db814c720 0x7f7db8142280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:17.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.435+0000 7f7dc05de640 1 --2- 172.21.15.116:0/1023038193 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7db80c2f50 0x7f7db8154860 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:17.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.435+0000 7f7dc05de640 1 --2- 172.21.15.116:0/1023038193 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7db80a94a0 0x7f7db8140d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:17.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.435+0000 7f7dc05de640 1 -- 172.21.15.116:0/1023038193 >> 172.21.15.116:0/1023038193 conn(0x7f7db80c3b00 msgr2=0x7f7db80c2790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:10:17.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.435+0000 7f7dc05de640 1 -- 172.21.15.116:0/1023038193 shutdown_connections 2024-04-25T00:10:17.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:17.435+0000 7f7dc05de640 1 -- 172.21.15.116:0/1023038193 wait complete. 2024-04-25T00:10:17.454 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T00:10:19.455 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:10:19.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.583+0000 7fa21a2f9640 1 Processor -- start 2024-04-25T00:10:19.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.583+0000 7fa21a2f9640 1 -- start start 2024-04-25T00:10:19.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.587+0000 7fa21a2f9640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa214148890 0x7fa214072590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:19.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.587+0000 7fa21a2f9640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fa21406e890 0x7fa214071980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:19.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.587+0000 7fa21a2f9640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa214152c10 con 0x7fa214070dd0 2024-04-25T00:10:19.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.587+0000 7fa21a2f9640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fa214151e50 con 0x7fa21406e890 2024-04-25T00:10:19.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.587+0000 7fa21a2f9640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa214152130 con 0x7fa214148890 2024-04-25T00:10:19.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.587+0000 7fa20b7fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fa214070dd0 0x7fa214071270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:46162/0 (socket says 172.21.15.116:46162) 2024-04-25T00:10:19.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.587+0000 7fa20b7fe640 1 -- 172.21.15.116:0/1184429000 learned_addr learned my addr 172.21.15.116:0/1184429000 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:10:19.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.587+0000 7fa218896640 1 --2- 172.21.15.116:0/1184429000 >> v2:172.21.15.174:3300/0 conn(0x7fa21406e890 0x7fa214071980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:19.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.587+0000 7fa213fff640 1 --2- 172.21.15.116:0/1184429000 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa214148890 0x7fa214072590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:19.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.587+0000 7fa2137fe640 1 -- 172.21.15.116:0/1184429000 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1293445765 0 0) 0x7fa214152c10 con 0x7fa214070dd0 2024-04-25T00:10:19.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.587+0000 7fa2137fe640 1 -- 172.21.15.116:0/1184429000 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa200005740 con 0x7fa214070dd0 2024-04-25T00:10:19.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.587+0000 7fa2137fe640 1 -- 172.21.15.116:0/1184429000 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 733579763 0 0) 0x7fa200005740 con 0x7fa214070dd0 2024-04-25T00:10:19.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.587+0000 7fa2137fe640 1 -- 172.21.15.116:0/1184429000 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa214148890 msgr2=0x7fa214072590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:10:19.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.587+0000 7fa2137fe640 1 --2- 172.21.15.116:0/1184429000 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa214148890 0x7fa214072590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:19.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.587+0000 7fa2137fe640 1 -- 172.21.15.116:0/1184429000 >> v2:172.21.15.174:3300/0 conn(0x7fa21406e890 msgr2=0x7fa214071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:10:19.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.587+0000 7fa2137fe640 1 --2- 172.21.15.116:0/1184429000 >> v2:172.21.15.174:3300/0 conn(0x7fa21406e890 0x7fa214071980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:19.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.587+0000 7fa2137fe640 1 -- 172.21.15.116:0/1184429000 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa214152410 con 0x7fa214070dd0 2024-04-25T00:10:19.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.587+0000 7fa2137fe640 1 -- 172.21.15.116:0/1184429000 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7fa1fc001680 con 0x7fa214070dd0 2024-04-25T00:10:19.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.587+0000 7fa2137fe640 1 -- 172.21.15.116:0/1184429000 >> v1:172.21.15.116:6789/0 conn(0x7fa214070dd0 legacy=0x7fa214071270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:19.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.587+0000 7fa2137fe640 1 --2- 172.21.15.116:0/1184429000 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa200005bd0 0x7fa200005ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:19.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.587+0000 7fa2137fe640 1 --2- 172.21.15.116:0/1184429000 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa200006a20 0x7fa200006ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:19.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.587+0000 7fa2137fe640 1 --2- 172.21.15.116:0/1184429000 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa214148890 0x7fa2000073e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:19.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.587+0000 7fa2137fe640 1 -- 172.21.15.116:0/1184429000 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa214151e50 con 0x7fa200006a20 2024-04-25T00:10:19.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.587+0000 7fa2137fe640 1 -- 172.21.15.116:0/1184429000 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa214152130 con 0x7fa200005bd0 2024-04-25T00:10:19.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.587+0000 7fa2137fe640 1 -- 172.21.15.116:0/1184429000 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa20000dbe0 con 0x7fa214148890 2024-04-25T00:10:19.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.587+0000 7fa20b7fe640 1 --2- 172.21.15.116:0/1184429000 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa200006a20 0x7fa200006ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:19.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.587+0000 7fa213fff640 1 --2- 172.21.15.116:0/1184429000 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa200005bd0 0x7fa200005ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:19.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.587+0000 7fa20b7fe640 1 -- 172.21.15.116:0/1184429000 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa214148890 msgr2=0x7fa2000073e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:10:19.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.587+0000 7fa20b7fe640 1 --2- 172.21.15.116:0/1184429000 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa214148890 0x7fa2000073e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:19.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.587+0000 7fa20b7fe640 1 -- 172.21.15.116:0/1184429000 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa200005bd0 msgr2=0x7fa200005ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:10:19.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.587+0000 7fa20b7fe640 1 --2- 172.21.15.116:0/1184429000 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa200005bd0 0x7fa200005ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:19.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.587+0000 7fa20b7fe640 1 -- 172.21.15.116:0/1184429000 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa200005740 con 0x7fa200006a20 2024-04-25T00:10:19.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.587+0000 7fa20b7fe640 1 --2- 172.21.15.116:0/1184429000 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa200006a20 0x7fa200006ea0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc00c160 tx=0x7fa1fc01f2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4697bfd70f09c828 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:19.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.591+0000 7fa2137fe640 1 -- 172.21.15.116:0/1184429000 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1fc007c90 con 0x7fa200006a20 2024-04-25T00:10:19.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.591+0000 7fa2137fe640 1 -- 172.21.15.116:0/1184429000 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1fc0ac020 con 0x7fa200006a20 2024-04-25T00:10:19.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.591+0000 7fa2137fe640 1 -- 172.21.15.116:0/1184429000 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1fc0a8e50 con 0x7fa200006a20 2024-04-25T00:10:19.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.591+0000 7fa21a2f9640 1 -- 172.21.15.116:0/1184429000 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa200006a20 msgr2=0x7fa200006ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:19.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.591+0000 7fa21a2f9640 1 --2- 172.21.15.116:0/1184429000 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa200006a20 0x7fa200006ea0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc00c160 tx=0x7fa1fc01f2d0 comp rx=0 tx=0).stop 2024-04-25T00:10:19.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.591+0000 7fa218896640 1 -- 172.21.15.116:0/1184429000 reap_dead start 2024-04-25T00:10:19.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.591+0000 7fa21a2f9640 1 -- 172.21.15.116:0/1184429000 shutdown_connections 2024-04-25T00:10:19.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.591+0000 7fa21a2f9640 1 -- 172.21.15.116:0/1184429000 >> 172.21.15.116:0/1184429000 conn(0x7fa2140c3b00 msgr2=0x7fa2140c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:10:19.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.591+0000 7fa21a2f9640 1 -- 172.21.15.116:0/1184429000 shutdown_connections 2024-04-25T00:10:19.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.591+0000 7fa21a2f9640 1 -- 172.21.15.116:0/1184429000 wait complete. 2024-04-25T00:10:19.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.591+0000 7fa21a2f9640 1 Processor -- start 2024-04-25T00:10:19.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.591+0000 7fa21a2f9640 1 -- start start 2024-04-25T00:10:19.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.591+0000 7fa21a2f9640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa200006a20 0x7fa21414ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:19.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.591+0000 7fa21a2f9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa21414ad80 0x7fa214146810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:19.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.591+0000 7fa21a2f9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa214146d50 0x7fa2141e6e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:19.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.591+0000 7fa21a2f9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa2140c3540 con 0x7fa21414ad80 2024-04-25T00:10:19.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.591+0000 7fa21a2f9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa21406ed70 con 0x7fa214146d50 2024-04-25T00:10:19.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.591+0000 7fa21a2f9640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa2140c36c0 con 0x7fa200006a20 2024-04-25T00:10:19.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.591+0000 7fa20b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa21414ad80 0x7fa214146810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:19.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.591+0000 7fa20b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa21414ad80 0x7fa214146810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48610/0 (socket says 172.21.15.116:48610) 2024-04-25T00:10:19.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.591+0000 7fa20b7fe640 1 -- 172.21.15.116:0/3449047264 learned_addr learned my addr 172.21.15.116:0/3449047264 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:10:19.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.591+0000 7fa213fff640 1 --2- 172.21.15.116:0/3449047264 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa200006a20 0x7fa21414ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:19.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.591+0000 7fa218896640 1 --2- 172.21.15.116:0/3449047264 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa214146d50 0x7fa2141e6e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:19.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.595+0000 7fa20b7fe640 1 -- 172.21.15.116:0/3449047264 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa200006a20 msgr2=0x7fa21414ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:19.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.595+0000 7fa20b7fe640 1 --2- 172.21.15.116:0/3449047264 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa200006a20 0x7fa21414ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:19.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.595+0000 7fa20b7fe640 1 -- 172.21.15.116:0/3449047264 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa214146d50 msgr2=0x7fa2141e6e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:19.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.595+0000 7fa20b7fe640 1 --2- 172.21.15.116:0/3449047264 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa214146d50 0x7fa2141e6e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:19.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.595+0000 7fa20b7fe640 1 -- 172.21.15.116:0/3449047264 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa21414efe0 con 0x7fa21414ad80 2024-04-25T00:10:19.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.595+0000 7fa20b7fe640 1 --2- 172.21.15.116:0/3449047264 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa21414ad80 0x7fa214146810 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc00c250 tx=0x7fa200007a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:19.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.595+0000 7fa2117fa640 1 -- 172.21.15.116:0/3449047264 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1fc0047f0 con 0x7fa21414ad80 2024-04-25T00:10:19.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.595+0000 7fa2117fa640 1 -- 172.21.15.116:0/3449047264 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1fc0b6070 con 0x7fa21414ad80 2024-04-25T00:10:19.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.595+0000 7fa21a2f9640 1 -- 172.21.15.116:0/3449047264 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa21414c940 con 0x7fa21414ad80 2024-04-25T00:10:19.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.595+0000 7fa2117fa640 1 -- 172.21.15.116:0/3449047264 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1fc004bb0 con 0x7fa21414ad80 2024-04-25T00:10:19.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.595+0000 7fa21a2f9640 1 -- 172.21.15.116:0/3449047264 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa21414cf00 con 0x7fa21414ad80 2024-04-25T00:10:19.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.595+0000 7fa21a2f9640 1 -- 172.21.15.116:0/3449047264 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa214068350 con 0x7fa21414ad80 2024-04-25T00:10:19.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.595+0000 7fa2117fa640 1 -- 172.21.15.116:0/3449047264 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa1fc007070 con 0x7fa21414ad80 2024-04-25T00:10:19.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.595+0000 7fa2117fa640 1 --2- 172.21.15.116:0/3449047264 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa1f40678f0 0x7fa1f4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:19.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.595+0000 7fa2117fa640 1 -- 172.21.15.116:0/3449047264 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 7612+0+0 (secure 0 0 0) 0x7fa1fc123230 con 0x7fa21414ad80 2024-04-25T00:10:19.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.595+0000 7fa213fff640 1 --2- 172.21.15.116:0/3449047264 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa1f40678f0 0x7fa1f4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:19.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.595+0000 7fa213fff640 1 --2- 172.21.15.116:0/3449047264 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa1f40678f0 0x7fa1f4069db0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fa2140acad0 tx=0x7fa204008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:19.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.599+0000 7fa2117fa640 1 -- 172.21.15.116:0/3449047264 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fa1fc019040 con 0x7fa21414ad80 2024-04-25T00:10:19.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.795+0000 7fa21a2f9640 1 -- 172.21.15.116:0/3449047264 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa214151a30 con 0x7fa21414ad80 2024-04-25T00:10:19.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.795+0000 7fa2117fa640 1 -- 172.21.15.116:0/3449047264 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) ==== 94+0+6504 (secure 0 0 0) 0x7fa1fc0f35f0 con 0x7fa21414ad80 2024-04-25T00:10:19.800 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:10:19.800 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6114,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66},{"gid":6135,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1358527848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1358527848},{"type":"v1","addr":"172.21.15.116:6835","nonce":1358527848}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":67}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:09:33.601522+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5236":{"gid":5236,"name":"b","rank":0,"incarnation":62,"state":"up:active","state_seq":25,"addr":"172.21.15.174:6837/1159002662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1159002662},{"type":"v1","addr":"172.21.15.174:6837","nonce":1159002662}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5236,"qdb_cluster":[5236]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:09:33.601524+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":132,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4235,"qdb_cluster":[4235]},"id":2}]} 2024-04-25T00:10:19.800 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 67 2024-04-25T00:10:19.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.799+0000 7fa21a2f9640 1 -- 172.21.15.116:0/3449047264 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa1f40678f0 msgr2=0x7fa1f4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:19.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.799+0000 7fa21a2f9640 1 --2- 172.21.15.116:0/3449047264 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa1f40678f0 0x7fa1f4069db0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fa2140acad0 tx=0x7fa204008040 comp rx=0 tx=0).stop 2024-04-25T00:10:19.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.799+0000 7fa21a2f9640 1 -- 172.21.15.116:0/3449047264 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa21414ad80 msgr2=0x7fa214146810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:19.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.799+0000 7fa21a2f9640 1 --2- 172.21.15.116:0/3449047264 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa21414ad80 0x7fa214146810 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fa1fc00c250 tx=0x7fa200007a50 comp rx=0 tx=0).stop 2024-04-25T00:10:19.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.799+0000 7fa21a2f9640 1 -- 172.21.15.116:0/3449047264 shutdown_connections 2024-04-25T00:10:19.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.799+0000 7fa21a2f9640 1 --2- 172.21.15.116:0/3449047264 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa1f40678f0 0x7fa1f4069db0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:19.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.799+0000 7fa21a2f9640 1 --2- 172.21.15.116:0/3449047264 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa214146d50 0x7fa2141e6e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:19.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.799+0000 7fa21a2f9640 1 --2- 172.21.15.116:0/3449047264 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa21414ad80 0x7fa214146810 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:19.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.799+0000 7fa21a2f9640 1 --2- 172.21.15.116:0/3449047264 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa200006a20 0x7fa21414ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:19.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.799+0000 7fa21a2f9640 1 -- 172.21.15.116:0/3449047264 >> 172.21.15.116:0/3449047264 conn(0x7fa2140c3b00 msgr2=0x7fa214147d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:10:19.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.799+0000 7fa21a2f9640 1 -- 172.21.15.116:0/3449047264 shutdown_connections 2024-04-25T00:10:19.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:19.799+0000 7fa21a2f9640 1 -- 172.21.15.116:0/3449047264 wait complete. 2024-04-25T00:10:19.819 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T00:10:21.820 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:10:21.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:21.951+0000 7f15c01e8640 1 Processor -- start 2024-04-25T00:10:21.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:21.951+0000 7f15c01e8640 1 -- start start 2024-04-25T00:10:21.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:21.955+0000 7f15c01e8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f15b8152d80 0x7f15b806bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:21.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:21.955+0000 7f15c01e8640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f15b806c290 0x7f15b8157390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:21.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:21.955+0000 7f15c01e8640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15b80c9150 con 0x7f15b806d5f0 2024-04-25T00:10:21.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:21.955+0000 7f15c01e8640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f15b80c9d60 con 0x7f15b8152d80 2024-04-25T00:10:21.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:21.955+0000 7f15c01e8640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f15b80c9430 con 0x7f15b806c290 2024-04-25T00:10:21.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:21.955+0000 7f15bdf84640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f15b8152d80 0x7f15b806bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:21.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:21.955+0000 7f15bd783640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f15b806d5f0 0x7f15b8156c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:46164/0 (socket says 172.21.15.116:46164) 2024-04-25T00:10:21.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:21.955+0000 7f15bd783640 1 -- 172.21.15.116:0/3026643747 learned_addr learned my addr 172.21.15.116:0/3026643747 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:10:21.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:21.955+0000 7f15be785640 1 --2- 172.21.15.116:0/3026643747 >> v2:172.21.15.174:3300/0 conn(0x7f15b806c290 0x7f15b8157390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:21.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:21.955+0000 7f15bdf84640 1 -- 172.21.15.116:0/3026643747 >> v2:172.21.15.174:3300/0 conn(0x7f15b806c290 msgr2=0x7f15b8157390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:10:21.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:21.955+0000 7f15bdf84640 1 --2- 172.21.15.116:0/3026643747 >> v2:172.21.15.174:3300/0 conn(0x7f15b806c290 0x7f15b8157390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:21.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:21.955+0000 7f15bdf84640 1 -- 172.21.15.116:0/3026643747 >> v1:172.21.15.116:6789/0 conn(0x7f15b806d5f0 legacy=0x7f15b8156c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:21.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:21.955+0000 7f15bdf84640 1 -- 172.21.15.116:0/3026643747 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15b80c9710 con 0x7f15b8152d80 2024-04-25T00:10:21.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:21.955+0000 7f15bdf84640 1 --2- 172.21.15.116:0/3026643747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15b8152d80 0x7f15b806bd50 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f15a80097c0 tx=0x7f15a801c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26c8f21f89fcc6b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:21.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:21.955+0000 7f15bcf82640 1 -- 172.21.15.116:0/3026643747 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a80b1070 con 0x7f15b8152d80 2024-04-25T00:10:21.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:21.955+0000 7f15bcf82640 1 -- 172.21.15.116:0/3026643747 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15a80ac620 con 0x7f15b8152d80 2024-04-25T00:10:21.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:21.955+0000 7f15bcf82640 1 -- 172.21.15.116:0/3026643747 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a80ac9f0 con 0x7f15b8152d80 2024-04-25T00:10:21.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:21.955+0000 7f15c01e8640 1 -- 172.21.15.116:0/3026643747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15b8152d80 msgr2=0x7f15b806bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:21.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:21.955+0000 7f15c01e8640 1 --2- 172.21.15.116:0/3026643747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15b8152d80 0x7f15b806bd50 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f15a80097c0 tx=0x7f15a801c3a0 comp rx=0 tx=0).stop 2024-04-25T00:10:21.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:21.955+0000 7f15c01e8640 1 -- 172.21.15.116:0/3026643747 shutdown_connections 2024-04-25T00:10:21.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:21.955+0000 7f15c01e8640 1 --2- 172.21.15.116:0/3026643747 >> v2:172.21.15.174:3300/0 conn(0x7f15b806c290 0x7f15b8157390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:21.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:21.955+0000 7f15c01e8640 1 --2- 172.21.15.116:0/3026643747 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15b8152d80 0x7f15b806bd50 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:21.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:21.955+0000 7f15c01e8640 1 -- 172.21.15.116:0/3026643747 >> 172.21.15.116:0/3026643747 conn(0x7f15b80acdd0 msgr2=0x7f15b80ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:10:21.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:21.955+0000 7f15c01e8640 1 -- 172.21.15.116:0/3026643747 shutdown_connections 2024-04-25T00:10:21.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:21.955+0000 7f15c01e8640 1 -- 172.21.15.116:0/3026643747 wait complete. 2024-04-25T00:10:21.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:21.959+0000 7f15c01e8640 1 Processor -- start 2024-04-25T00:10:21.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:21.959+0000 7f15c01e8640 1 -- start start 2024-04-25T00:10:21.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:21.959+0000 7f15c01e8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f15b806c290 0x7f15b8151fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:21.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:21.959+0000 7f15c01e8640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f15b806d5f0 0x7f15b81f1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:21.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:21.959+0000 7f15c01e8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15b8152d80 0x7f15b814fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:21.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:21.959+0000 7f15c01e8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f15b80caf00 con 0x7f15b8152d80 2024-04-25T00:10:21.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:21.959+0000 7f15c01e8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f15b80cad80 con 0x7f15b806c290 2024-04-25T00:10:21.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:21.959+0000 7f15c01e8640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f15b81628d0 con 0x7f15b806d5f0 2024-04-25T00:10:21.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:21.959+0000 7f15be785640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15b8152d80 0x7f15b814fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:21.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:21.959+0000 7f15be785640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15b8152d80 0x7f15b814fe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48632/0 (socket says 172.21.15.116:48632) 2024-04-25T00:10:21.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:21.959+0000 7f15be785640 1 -- 172.21.15.116:0/2384366102 learned_addr learned my addr 172.21.15.116:0/2384366102 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:10:21.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:21.959+0000 7f15bdf84640 1 --2- 172.21.15.116:0/2384366102 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f15b806c290 0x7f15b8151fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:21.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:21.959+0000 7f15bd783640 1 --2- 172.21.15.116:0/2384366102 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f15b806d5f0 0x7f15b81f1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:21.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:21.959+0000 7f15be785640 1 -- 172.21.15.116:0/2384366102 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f15b806d5f0 msgr2=0x7f15b81f1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:21.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:21.959+0000 7f15be785640 1 --2- 172.21.15.116:0/2384366102 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f15b806d5f0 0x7f15b81f1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:21.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:21.959+0000 7f15be785640 1 -- 172.21.15.116:0/2384366102 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f15b806c290 msgr2=0x7f15b8151fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:21.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:21.959+0000 7f15be785640 1 --2- 172.21.15.116:0/2384366102 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f15b806c290 0x7f15b8151fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:21.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:21.959+0000 7f15be785640 1 -- 172.21.15.116:0/2384366102 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15b8151d60 con 0x7f15b8152d80 2024-04-25T00:10:21.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:21.959+0000 7f15be785640 1 --2- 172.21.15.116:0/2384366102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15b8152d80 0x7f15b814fe00 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f15b000d590 tx=0x7f15b000da50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:21.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:21.959+0000 7f15ae7fc640 1 -- 172.21.15.116:0/2384366102 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15b00125f0 con 0x7f15b8152d80 2024-04-25T00:10:21.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:21.959+0000 7f15ae7fc640 1 -- 172.21.15.116:0/2384366102 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15b0019070 con 0x7f15b8152d80 2024-04-25T00:10:21.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:21.959+0000 7f15c01e8640 1 -- 172.21.15.116:0/2384366102 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15b81f26a0 con 0x7f15b8152d80 2024-04-25T00:10:21.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:21.959+0000 7f15ae7fc640 1 -- 172.21.15.116:0/2384366102 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15b00129f0 con 0x7f15b8152d80 2024-04-25T00:10:21.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:21.959+0000 7f15c01e8640 1 -- 172.21.15.116:0/2384366102 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15b81f2390 con 0x7f15b8152d80 2024-04-25T00:10:21.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:21.959+0000 7f15c01e8640 1 -- 172.21.15.116:0/2384366102 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15b8068350 con 0x7f15b8152d80 2024-04-25T00:10:21.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:21.963+0000 7f15ae7fc640 1 -- 172.21.15.116:0/2384366102 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f15b0012b90 con 0x7f15b8152d80 2024-04-25T00:10:21.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:21.963+0000 7f15ae7fc640 1 --2- 172.21.15.116:0/2384366102 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f159c0678c0 0x7f159c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:21.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:21.963+0000 7f15ae7fc640 1 -- 172.21.15.116:0/2384366102 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7698+0+0 (secure 0 0 0) 0x7f15b00867d0 con 0x7f15b8152d80 2024-04-25T00:10:21.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:21.963+0000 7f15bdf84640 1 --2- 172.21.15.116:0/2384366102 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f159c0678c0 0x7f159c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:21.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:21.963+0000 7f15bdf84640 1 --2- 172.21.15.116:0/2384366102 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f159c0678c0 0x7f159c069d80 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f15a80ae000 tx=0x7f15a8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:21.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:21.967+0000 7f15ae7fc640 1 -- 172.21.15.116:0/2384366102 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f15b0056bd0 con 0x7f15b8152d80 2024-04-25T00:10:22.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.159+0000 7f15c01e8640 1 -- 172.21.15.116:0/2384366102 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f15b80cab60 con 0x7f15b8152d80 2024-04-25T00:10:22.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.163+0000 7f15ae7fc640 1 -- 172.21.15.116:0/2384366102 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) ==== 94+0+5672 (secure 0 0 0) 0x7f15b00564d0 con 0x7f15b8152d80 2024-04-25T00:10:22.164 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:10:22.164 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6114,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:10:20.800226+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6135},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6135":{"gid":6135,"name":"c","rank":0,"incarnation":68,"state":"up:reconnect","state_seq":11,"addr":"172.21.15.116:6835/1358527848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1358527848},{"type":"v1","addr":"172.21.15.116:6835","nonce":1358527848}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:09:33.601524+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":132,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4235,"qdb_cluster":[4235]},"id":2}]} 2024-04-25T00:10:22.165 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 69 2024-04-25T00:10:22.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.163+0000 7f15c01e8640 1 -- 172.21.15.116:0/2384366102 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f159c0678c0 msgr2=0x7f159c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:22.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.163+0000 7f15c01e8640 1 --2- 172.21.15.116:0/2384366102 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f159c0678c0 0x7f159c069d80 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f15a80ae000 tx=0x7f15a8008040 comp rx=0 tx=0).stop 2024-04-25T00:10:22.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.163+0000 7f15c01e8640 1 -- 172.21.15.116:0/2384366102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15b8152d80 msgr2=0x7f15b814fe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:22.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.163+0000 7f15c01e8640 1 --2- 172.21.15.116:0/2384366102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15b8152d80 0x7f15b814fe00 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f15b000d590 tx=0x7f15b000da50 comp rx=0 tx=0).stop 2024-04-25T00:10:22.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.163+0000 7f15c01e8640 1 -- 172.21.15.116:0/2384366102 shutdown_connections 2024-04-25T00:10:22.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.163+0000 7f15c01e8640 1 --2- 172.21.15.116:0/2384366102 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f159c0678c0 0x7f159c069d80 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:22.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.163+0000 7f15c01e8640 1 --2- 172.21.15.116:0/2384366102 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f15b8152d80 0x7f15b814fe00 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:22.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.163+0000 7f15c01e8640 1 --2- 172.21.15.116:0/2384366102 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f15b806d5f0 0x7f15b81f1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:22.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.163+0000 7f15c01e8640 1 --2- 172.21.15.116:0/2384366102 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f15b806c290 0x7f15b8151fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:22.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.163+0000 7f15c01e8640 1 -- 172.21.15.116:0/2384366102 >> 172.21.15.116:0/2384366102 conn(0x7f15b80acdd0 msgr2=0x7f15b8156b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:10:22.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.167+0000 7f15c01e8640 1 -- 172.21.15.116:0/2384366102 shutdown_connections 2024-04-25T00:10:22.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.167+0000 7f15c01e8640 1 -- 172.21.15.116:0/2384366102 wait complete. 2024-04-25T00:10:22.185 INFO:tasks.mds_thrash.fs.[a]:mds.b down, removed from mdsmap 2024-04-25T00:10:22.185 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-04-25T00:10:22.186 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:10:22.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.303+0000 7f3a5f5c6640 1 Processor -- start 2024-04-25T00:10:22.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.303+0000 7f3a5f5c6640 1 -- start start 2024-04-25T00:10:22.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.303+0000 7f3a5f5c6640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f3a58152d80 0x7f3a5806bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:22.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.303+0000 7f3a5f5c6640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3a5806c290 0x7f3a58157390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:22.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.303+0000 7f3a5f5c6640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a580c9150 con 0x7f3a5806d5f0 2024-04-25T00:10:22.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.303+0000 7f3a5f5c6640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f3a580c9d60 con 0x7f3a58152d80 2024-04-25T00:10:22.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.303+0000 7f3a5f5c6640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f3a580c9430 con 0x7f3a5806c290 2024-04-25T00:10:22.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.303+0000 7f3a5db63640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3a5806c290 0x7f3a58157390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:22.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.303+0000 7f3a5db63640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3a5806c290 0x7f3a58157390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:48546/0 (socket says 172.21.15.116:48546) 2024-04-25T00:10:22.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.303+0000 7f3a5db63640 1 -- 172.21.15.116:0/1040812564 learned_addr learned my addr 172.21.15.116:0/1040812564 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:10:22.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.307+0000 7f3a5d362640 1 --2- 172.21.15.116:0/1040812564 >> v2:172.21.15.174:3300/0 conn(0x7f3a58152d80 0x7f3a5806bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:22.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.307+0000 7f3a5db63640 1 -- 172.21.15.116:0/1040812564 >> v2:172.21.15.174:3300/0 conn(0x7f3a58152d80 msgr2=0x7f3a5806bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:10:22.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.307+0000 7f3a5db63640 1 --2- 172.21.15.116:0/1040812564 >> v2:172.21.15.174:3300/0 conn(0x7f3a58152d80 0x7f3a5806bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:22.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.307+0000 7f3a5db63640 1 -- 172.21.15.116:0/1040812564 >> v1:172.21.15.174:6789/0 conn(0x7f3a5806d5f0 legacy=0x7f3a58156c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:22.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.307+0000 7f3a5db63640 1 -- 172.21.15.116:0/1040812564 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a580c9710 con 0x7f3a5806c290 2024-04-25T00:10:22.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.307+0000 7f3a5db63640 1 --2- 172.21.15.116:0/1040812564 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3a5806c290 0x7f3a58157390 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3a50009e50 tx=0x7f3a5001c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8abfda88487ea89 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:22.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.307+0000 7f3a4ffff640 1 -- 172.21.15.116:0/1040812564 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a500a5850 con 0x7f3a5806c290 2024-04-25T00:10:22.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.307+0000 7f3a4ffff640 1 -- 172.21.15.116:0/1040812564 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a500a59f0 con 0x7f3a5806c290 2024-04-25T00:10:22.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.307+0000 7f3a4ffff640 1 -- 172.21.15.116:0/1040812564 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a500a5dc0 con 0x7f3a5806c290 2024-04-25T00:10:22.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.307+0000 7f3a5f5c6640 1 -- 172.21.15.116:0/1040812564 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3a5806c290 msgr2=0x7f3a58157390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:22.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.307+0000 7f3a5f5c6640 1 --2- 172.21.15.116:0/1040812564 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3a5806c290 0x7f3a58157390 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3a50009e50 tx=0x7f3a5001c970 comp rx=0 tx=0).stop 2024-04-25T00:10:22.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.311+0000 7f3a5f5c6640 1 -- 172.21.15.116:0/1040812564 shutdown_connections 2024-04-25T00:10:22.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.311+0000 7f3a5f5c6640 1 --2- 172.21.15.116:0/1040812564 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3a5806c290 0x7f3a58157390 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:22.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.311+0000 7f3a5f5c6640 1 --2- 172.21.15.116:0/1040812564 >> v2:172.21.15.174:3300/0 conn(0x7f3a58152d80 0x7f3a5806bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:22.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.311+0000 7f3a5f5c6640 1 -- 172.21.15.116:0/1040812564 >> 172.21.15.116:0/1040812564 conn(0x7f3a580acdd0 msgr2=0x7f3a580ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:10:22.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.311+0000 7f3a5f5c6640 1 -- 172.21.15.116:0/1040812564 shutdown_connections 2024-04-25T00:10:22.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.311+0000 7f3a5f5c6640 1 -- 172.21.15.116:0/1040812564 wait complete. 2024-04-25T00:10:22.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.311+0000 7f3a5f5c6640 1 Processor -- start 2024-04-25T00:10:22.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.311+0000 7f3a5f5c6640 1 -- start start 2024-04-25T00:10:22.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.311+0000 7f3a5f5c6640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3a5806c290 0x7f3a58151a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:22.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.311+0000 7f3a5f5c6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a5806d5f0 0x7f3a5814ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:22.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.311+0000 7f3a5f5c6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a58152d80 0x7f3a5814b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:22.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.311+0000 7f3a5f5c6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3a580caf00 con 0x7f3a58152d80 2024-04-25T00:10:22.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.311+0000 7f3a5f5c6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3a580cad80 con 0x7f3a5806d5f0 2024-04-25T00:10:22.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.311+0000 7f3a5f5c6640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f3a581628d0 con 0x7f3a5806c290 2024-04-25T00:10:22.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.311+0000 7f3a5db63640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a58152d80 0x7f3a5814b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:22.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.311+0000 7f3a5db63640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a58152d80 0x7f3a5814b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48644/0 (socket says 172.21.15.116:48644) 2024-04-25T00:10:22.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.311+0000 7f3a5db63640 1 -- 172.21.15.116:0/3171020293 learned_addr learned my addr 172.21.15.116:0/3171020293 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:10:22.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.311+0000 7f3a5d362640 1 --2- 172.21.15.116:0/3171020293 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3a5806c290 0x7f3a58151a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:22.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.311+0000 7f3a5cb61640 1 --2- 172.21.15.116:0/3171020293 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a5806d5f0 0x7f3a5814ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:22.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.311+0000 7f3a5db63640 1 -- 172.21.15.116:0/3171020293 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3a5806c290 msgr2=0x7f3a58151a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:22.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.311+0000 7f3a5db63640 1 --2- 172.21.15.116:0/3171020293 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3a5806c290 0x7f3a58151a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:22.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.311+0000 7f3a5db63640 1 -- 172.21.15.116:0/3171020293 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a5806d5f0 msgr2=0x7f3a5814ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:22.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.311+0000 7f3a5db63640 1 --2- 172.21.15.116:0/3171020293 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a5806d5f0 0x7f3a5814ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:22.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.311+0000 7f3a5db63640 1 -- 172.21.15.116:0/3171020293 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a581efeb0 con 0x7f3a58152d80 2024-04-25T00:10:22.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.311+0000 7f3a5db63640 1 --2- 172.21.15.116:0/3171020293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a58152d80 0x7f3a5814b3b0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f3a5001a5a0 tx=0x7f3a50019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:22.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.311+0000 7f3a4dffb640 1 -- 172.21.15.116:0/3171020293 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a500bc020 con 0x7f3a58152d80 2024-04-25T00:10:22.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.311+0000 7f3a4dffb640 1 -- 172.21.15.116:0/3171020293 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a500b8670 con 0x7f3a58152d80 2024-04-25T00:10:22.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.311+0000 7f3a5f5c6640 1 -- 172.21.15.116:0/3171020293 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a581ef110 con 0x7f3a58152d80 2024-04-25T00:10:22.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.311+0000 7f3a4dffb640 1 -- 172.21.15.116:0/3171020293 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a500b8a60 con 0x7f3a58152d80 2024-04-25T00:10:22.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.311+0000 7f3a5f5c6640 1 -- 172.21.15.116:0/3171020293 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a581ef6b0 con 0x7f3a58152d80 2024-04-25T00:10:22.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.311+0000 7f3a5f5c6640 1 -- 172.21.15.116:0/3171020293 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a58068350 con 0x7f3a58152d80 2024-04-25T00:10:22.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.315+0000 7f3a4dffb640 1 -- 172.21.15.116:0/3171020293 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3a500bf020 con 0x7f3a58152d80 2024-04-25T00:10:22.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.315+0000 7f3a4dffb640 1 --2- 172.21.15.116:0/3171020293 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3a3c0678c0 0x7f3a3c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:22.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.315+0000 7f3a4dffb640 1 -- 172.21.15.116:0/3171020293 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7698+0+0 (secure 0 0 0) 0x7f3a50121860 con 0x7f3a58152d80 2024-04-25T00:10:22.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.315+0000 7f3a5d362640 1 --2- 172.21.15.116:0/3171020293 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3a3c0678c0 0x7f3a3c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:22.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.315+0000 7f3a5d362640 1 --2- 172.21.15.116:0/3171020293 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3a3c0678c0 0x7f3a3c069d80 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f3a48009b60 tx=0x7f3a48009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:22.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.319+0000 7f3a4dffb640 1 -- 172.21.15.116:0/3171020293 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f3a500c6030 con 0x7f3a58152d80 2024-04-25T00:10:22.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.515+0000 7f3a5f5c6640 1 -- 172.21.15.116:0/3171020293 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3a580cab60 con 0x7f3a58152d80 2024-04-25T00:10:22.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.515+0000 7f3a4dffb640 1 -- 172.21.15.116:0/3171020293 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) ==== 94+0+5672 (secure 0 0 0) 0x7f3a50016620 con 0x7f3a58152d80 2024-04-25T00:10:22.519 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:10:22.519 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6114,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:10:20.800226+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6135},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6135":{"gid":6135,"name":"c","rank":0,"incarnation":68,"state":"up:reconnect","state_seq":11,"addr":"172.21.15.116:6835/1358527848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1358527848},{"type":"v1","addr":"172.21.15.116:6835","nonce":1358527848}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:09:33.601524+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":132,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4235,"qdb_cluster":[4235]},"id":2}]} 2024-04-25T00:10:22.519 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 69 2024-04-25T00:10:22.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.519+0000 7f3a5f5c6640 1 -- 172.21.15.116:0/3171020293 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3a3c0678c0 msgr2=0x7f3a3c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:22.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.519+0000 7f3a5f5c6640 1 --2- 172.21.15.116:0/3171020293 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3a3c0678c0 0x7f3a3c069d80 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f3a48009b60 tx=0x7f3a48009190 comp rx=0 tx=0).stop 2024-04-25T00:10:22.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.519+0000 7f3a5f5c6640 1 -- 172.21.15.116:0/3171020293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a58152d80 msgr2=0x7f3a5814b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:22.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.519+0000 7f3a5f5c6640 1 --2- 172.21.15.116:0/3171020293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a58152d80 0x7f3a5814b3b0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f3a5001a5a0 tx=0x7f3a50019710 comp rx=0 tx=0).stop 2024-04-25T00:10:22.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.519+0000 7f3a5f5c6640 1 -- 172.21.15.116:0/3171020293 shutdown_connections 2024-04-25T00:10:22.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.519+0000 7f3a5f5c6640 1 --2- 172.21.15.116:0/3171020293 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3a3c0678c0 0x7f3a3c069d80 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:22.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.519+0000 7f3a5f5c6640 1 --2- 172.21.15.116:0/3171020293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a58152d80 0x7f3a5814b3b0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:22.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.519+0000 7f3a5f5c6640 1 --2- 172.21.15.116:0/3171020293 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a5806d5f0 0x7f3a5814ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:22.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.519+0000 7f3a5f5c6640 1 --2- 172.21.15.116:0/3171020293 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3a5806c290 0x7f3a58151a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:22.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.519+0000 7f3a5f5c6640 1 -- 172.21.15.116:0/3171020293 >> 172.21.15.116:0/3171020293 conn(0x7f3a580acdd0 msgr2=0x7f3a5815f6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:10:22.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.519+0000 7f3a5f5c6640 1 -- 172.21.15.116:0/3171020293 shutdown_connections 2024-04-25T00:10:22.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:22.519+0000 7f3a5f5c6640 1 -- 172.21.15.116:0/3171020293 wait complete. 2024-04-25T00:10:22.537 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 69, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:04.678737+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 69, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6135": { "addr": "172.21.15.116:6835/1358527848", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6834", "nonce": 1358527848, "type": "v2" }, { "addr": "172.21.15.116:6835", "nonce": 1358527848, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6135, "incarnation": 68, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:reconnect", "state_seq": 11 } }, "last_failure": 0, "last_failure_osd_epoch": 40, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T00:10:20.800226+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6135 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:18.825887+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 65, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_4235": { "addr": "172.21.15.174:6835/2883220846", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6834", "nonce": 2883220846, "type": "v2" }, { "addr": "172.21.15.174:6835", "nonce": 2883220846, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4235, "incarnation": 62, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 132 } }, "last_failure": 0, "last_failure_osd_epoch": 39, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-04-25T00:09:33.601524+0000", "qdb_cluster": [ 4235 ], "qdb_leader": 4235, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4235 } } } ], "standbys": [ { "addr": "172.21.15.116:6833/3705654249", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6832", "nonce": 3705654249, "type": "v2" }, { "addr": "172.21.15.116:6833", "nonce": 3705654249, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 66, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6114, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-25T00:10:24.538 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:10:24.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.667+0000 7fc2f63fe640 1 Processor -- start 2024-04-25T00:10:24.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.667+0000 7fc2f63fe640 1 -- start start 2024-04-25T00:10:24.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.667+0000 7fc2f63fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc2f00aa1d0 0x7fc2f006fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:24.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.667+0000 7fc2f63fe640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fc2f0155c50 0x7fc2f0156100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:24.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.667+0000 7fc2f63fe640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2f01524b0 con 0x7fc2f014fbb0 2024-04-25T00:10:24.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.667+0000 7fc2f63fe640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fc2f0152770 con 0x7fc2f0155c50 2024-04-25T00:10:24.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.667+0000 7fc2f63fe640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fc2f0152a50 con 0x7fc2f00aa1d0 2024-04-25T00:10:24.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.667+0000 7fc2effff640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fc2f014fbb0 0x7fc2f00c5030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:32904/0 (socket says 172.21.15.116:32904) 2024-04-25T00:10:24.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.667+0000 7fc2e7fff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc2f00aa1d0 0x7fc2f006fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:24.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.667+0000 7fc2effff640 1 -- 172.21.15.116:0/1949558770 learned_addr learned my addr 172.21.15.116:0/1949558770 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:10:24.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.667+0000 7fc2f499b640 1 --2- 172.21.15.116:0/1949558770 >> v2:172.21.15.174:3300/0 conn(0x7fc2f0155c50 0x7fc2f0156100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:24.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.667+0000 7fc2ef7fe640 1 -- 172.21.15.116:0/1949558770 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 885988514 0 0) 0x7fc2f01524b0 con 0x7fc2f014fbb0 2024-04-25T00:10:24.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.667+0000 7fc2ef7fe640 1 -- 172.21.15.116:0/1949558770 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2dc003530 con 0x7fc2f014fbb0 2024-04-25T00:10:24.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.667+0000 7fc2e7fff640 1 -- 172.21.15.116:0/1949558770 >> v2:172.21.15.174:3300/0 conn(0x7fc2f0155c50 msgr2=0x7fc2f0156100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:10:24.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.667+0000 7fc2e7fff640 1 --2- 172.21.15.116:0/1949558770 >> v2:172.21.15.174:3300/0 conn(0x7fc2f0155c50 0x7fc2f0156100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:24.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.667+0000 7fc2e7fff640 1 -- 172.21.15.116:0/1949558770 >> v1:172.21.15.174:6789/0 conn(0x7fc2f014fbb0 legacy=0x7fc2f00c5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:24.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.667+0000 7fc2e7fff640 1 -- 172.21.15.116:0/1949558770 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2f0152d30 con 0x7fc2f00aa1d0 2024-04-25T00:10:24.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.671+0000 7fc2e7fff640 1 --2- 172.21.15.116:0/1949558770 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc2f00aa1d0 0x7fc2f006fbd0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc2d80097c0 tx=0x7fc2d801c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9209ad2d2a9d724 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:24.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.671+0000 7fc2ef7fe640 1 -- 172.21.15.116:0/1949558770 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d80b0070 con 0x7fc2f00aa1d0 2024-04-25T00:10:24.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.671+0000 7fc2ef7fe640 1 -- 172.21.15.116:0/1949558770 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2d80ab600 con 0x7fc2f00aa1d0 2024-04-25T00:10:24.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.671+0000 7fc2ef7fe640 1 -- 172.21.15.116:0/1949558770 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d80ab9b0 con 0x7fc2f00aa1d0 2024-04-25T00:10:24.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.671+0000 7fc2f63fe640 1 -- 172.21.15.116:0/1949558770 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc2f00aa1d0 msgr2=0x7fc2f006fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:24.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.671+0000 7fc2f63fe640 1 --2- 172.21.15.116:0/1949558770 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc2f00aa1d0 0x7fc2f006fbd0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc2d80097c0 tx=0x7fc2d801c410 comp rx=0 tx=0).stop 2024-04-25T00:10:24.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.671+0000 7fc2f63fe640 1 -- 172.21.15.116:0/1949558770 shutdown_connections 2024-04-25T00:10:24.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.671+0000 7fc2f63fe640 1 --2- 172.21.15.116:0/1949558770 >> v2:172.21.15.174:3300/0 conn(0x7fc2f0155c50 0x7fc2f0156100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:24.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.671+0000 7fc2f63fe640 1 --2- 172.21.15.116:0/1949558770 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc2f00aa1d0 0x7fc2f006fbd0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:24.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.671+0000 7fc2f63fe640 1 -- 172.21.15.116:0/1949558770 >> 172.21.15.116:0/1949558770 conn(0x7fc2f006c720 msgr2=0x7fc2f006cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:10:24.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.671+0000 7fc2f63fe640 1 -- 172.21.15.116:0/1949558770 shutdown_connections 2024-04-25T00:10:24.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.671+0000 7fc2f63fe640 1 -- 172.21.15.116:0/1949558770 wait complete. 2024-04-25T00:10:24.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.671+0000 7fc2f63fe640 1 Processor -- start 2024-04-25T00:10:24.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.671+0000 7fc2f63fe640 1 -- start start 2024-04-25T00:10:24.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.675+0000 7fc2f63fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc2f00aa1d0 0x7fc2f014c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:24.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.675+0000 7fc2f63fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2f014fbb0 0x7fc2f015b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:24.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.675+0000 7fc2f63fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc2f0155c50 0x7fc2f00c7680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:24.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.675+0000 7fc2f63fe640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f015f9e0 con 0x7fc2f014fbb0 2024-04-25T00:10:24.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.675+0000 7fc2f63fe640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f015f860 con 0x7fc2f0155c50 2024-04-25T00:10:24.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.675+0000 7fc2f63fe640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fc2f015fb60 con 0x7fc2f00aa1d0 2024-04-25T00:10:24.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.675+0000 7fc2e7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2f014fbb0 0x7fc2f015b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:24.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.675+0000 7fc2e7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2f014fbb0 0x7fc2f015b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48660/0 (socket says 172.21.15.116:48660) 2024-04-25T00:10:24.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.675+0000 7fc2e7fff640 1 -- 172.21.15.116:0/22271016 learned_addr learned my addr 172.21.15.116:0/22271016 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:10:24.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.675+0000 7fc2effff640 1 --2- 172.21.15.116:0/22271016 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc2f00aa1d0 0x7fc2f014c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:24.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.675+0000 7fc2f499b640 1 --2- 172.21.15.116:0/22271016 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc2f0155c50 0x7fc2f00c7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:24.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.675+0000 7fc2e7fff640 1 -- 172.21.15.116:0/22271016 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc2f00aa1d0 msgr2=0x7fc2f014c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:24.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.675+0000 7fc2e7fff640 1 --2- 172.21.15.116:0/22271016 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc2f00aa1d0 0x7fc2f014c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:24.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.675+0000 7fc2e7fff640 1 -- 172.21.15.116:0/22271016 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc2f0155c50 msgr2=0x7fc2f00c7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:24.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.675+0000 7fc2e7fff640 1 --2- 172.21.15.116:0/22271016 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc2f0155c50 0x7fc2f00c7680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:24.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.675+0000 7fc2e7fff640 1 -- 172.21.15.116:0/22271016 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2f014c670 con 0x7fc2f014fbb0 2024-04-25T00:10:24.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.675+0000 7fc2e7fff640 1 --2- 172.21.15.116:0/22271016 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2f014fbb0 0x7fc2f015b210 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fc2d8000c10 tx=0x7fc2d80191b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:24.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.675+0000 7fc2ed7fa640 1 -- 172.21.15.116:0/22271016 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d80b0040 con 0x7fc2f014fbb0 2024-04-25T00:10:24.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.675+0000 7fc2ed7fa640 1 -- 172.21.15.116:0/22271016 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2d80ab9b0 con 0x7fc2f014fbb0 2024-04-25T00:10:24.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.675+0000 7fc2f63fe640 1 -- 172.21.15.116:0/22271016 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2f00c21c0 con 0x7fc2f014fbb0 2024-04-25T00:10:24.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.675+0000 7fc2ed7fa640 1 -- 172.21.15.116:0/22271016 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d80abd60 con 0x7fc2f014fbb0 2024-04-25T00:10:24.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.675+0000 7fc2f63fe640 1 -- 172.21.15.116:0/22271016 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2f00c6e80 con 0x7fc2f014fbb0 2024-04-25T00:10:24.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.675+0000 7fc2f63fe640 1 -- 172.21.15.116:0/22271016 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2f0068a60 con 0x7fc2f014fbb0 2024-04-25T00:10:24.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.675+0000 7fc2ed7fa640 1 -- 172.21.15.116:0/22271016 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc2d80ab4c0 con 0x7fc2f014fbb0 2024-04-25T00:10:24.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.675+0000 7fc2ed7fa640 1 --2- 172.21.15.116:0/22271016 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc2d00678c0 0x7fc2d0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:24.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.679+0000 7fc2ed7fa640 1 -- 172.21.15.116:0/22271016 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7698+0+0 (secure 0 0 0) 0x7fc2d811fe80 con 0x7fc2f014fbb0 2024-04-25T00:10:24.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.679+0000 7fc2effff640 1 --2- 172.21.15.116:0/22271016 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc2d00678c0 0x7fc2d0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:24.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.679+0000 7fc2effff640 1 --2- 172.21.15.116:0/22271016 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc2d00678c0 0x7fc2d0069d80 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fc2e00011f0 tx=0x7fc2e0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:24.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.683+0000 7fc2ed7fa640 1 -- 172.21.15.116:0/22271016 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fc2d8016090 con 0x7fc2f014fbb0 2024-04-25T00:10:24.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.907+0000 7fc2f63fe640 1 -- 172.21.15.116:0/22271016 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc2f00759d0 con 0x7fc2f014fbb0 2024-04-25T00:10:24.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.907+0000 7fc2ed7fa640 1 -- 172.21.15.116:0/22271016 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) ==== 94+0+5676 (secure 0 0 0) 0x7fc2d80f01e0 con 0x7fc2f014fbb0 2024-04-25T00:10:24.912 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:10:24.912 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6114,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:10:23.780120+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6135},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6135":{"gid":6135,"name":"c","rank":0,"incarnation":68,"state":"up:active","state_seq":13,"addr":"172.21.15.116:6835/1358527848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1358527848},{"type":"v1","addr":"172.21.15.116:6835","nonce":1358527848}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6135,"qdb_cluster":[6135]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:09:33.601524+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":132,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4235,"qdb_cluster":[4235]},"id":2}]} 2024-04-25T00:10:24.912 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 71 2024-04-25T00:10:24.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.911+0000 7fc2f63fe640 1 -- 172.21.15.116:0/22271016 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc2d00678c0 msgr2=0x7fc2d0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:24.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.911+0000 7fc2f63fe640 1 --2- 172.21.15.116:0/22271016 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc2d00678c0 0x7fc2d0069d80 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fc2e00011f0 tx=0x7fc2e0009040 comp rx=0 tx=0).stop 2024-04-25T00:10:24.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.911+0000 7fc2f63fe640 1 -- 172.21.15.116:0/22271016 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2f014fbb0 msgr2=0x7fc2f015b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:24.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.911+0000 7fc2f63fe640 1 --2- 172.21.15.116:0/22271016 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2f014fbb0 0x7fc2f015b210 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fc2d8000c10 tx=0x7fc2d80191b0 comp rx=0 tx=0).stop 2024-04-25T00:10:24.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.911+0000 7fc2f63fe640 1 -- 172.21.15.116:0/22271016 shutdown_connections 2024-04-25T00:10:24.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.911+0000 7fc2f63fe640 1 --2- 172.21.15.116:0/22271016 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc2d00678c0 0x7fc2d0069d80 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:24.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.911+0000 7fc2f63fe640 1 --2- 172.21.15.116:0/22271016 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc2f0155c50 0x7fc2f00c7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:24.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.911+0000 7fc2f63fe640 1 --2- 172.21.15.116:0/22271016 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2f014fbb0 0x7fc2f015b210 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:24.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.911+0000 7fc2f63fe640 1 --2- 172.21.15.116:0/22271016 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc2f00aa1d0 0x7fc2f014c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:24.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.911+0000 7fc2f63fe640 1 -- 172.21.15.116:0/22271016 >> 172.21.15.116:0/22271016 conn(0x7fc2f006c720 msgr2=0x7fc2f006cb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:10:24.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.911+0000 7fc2f63fe640 1 -- 172.21.15.116:0/22271016 shutdown_connections 2024-04-25T00:10:24.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:24.911+0000 7fc2f63fe640 1 -- 172.21.15.116:0/22271016 wait complete. 2024-04-25T00:10:24.935 INFO:tasks.mds_thrash.fs.[a]:mds.c has gained rank=0, replacing gid=5236 2024-04-25T00:10:24.936 INFO:tasks.mds_thrash.fs.[a]:waiting for 8 secs before reviving mds.b 2024-04-25T00:10:26.662 INFO:tasks.ceph.mds.c.smithi116.stderr:2024-04-25T00:10:26.659+0000 7fb8c657f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T00:10:32.937 INFO:tasks.mds_thrash.fs.[a]:reviving mds.b 2024-04-25T00:10:32.937 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-25T00:10:32.937 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-25T00:10:32.940 INFO:tasks.ceph.mds.b:Started 2024-04-25T00:10:32.940 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:10:33.022 INFO:tasks.ceph.mds.b.smithi174.stdout:starting mds.b at 2024-04-25T00:10:33.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.091+0000 7f54ef149640 1 Processor -- start 2024-04-25T00:10:33.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.091+0000 7f54ef149640 1 -- start start 2024-04-25T00:10:33.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.095+0000 7f54ef149640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f54e814fe10 0x7f54e806c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:33.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.095+0000 7f54ef149640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f54e815aeb0 0x7f54e815d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:33.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.095+0000 7f54ef149640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54e80a6d50 con 0x7f54e806c760 2024-04-25T00:10:33.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.095+0000 7f54ef149640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f54e8155cf0 con 0x7f54e815aeb0 2024-04-25T00:10:33.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.095+0000 7f54ef149640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f54e80c6e90 con 0x7f54e814fe10 2024-04-25T00:10:33.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.095+0000 7f54ecee5640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f54e814fe10 0x7f54e806c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:33.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.095+0000 7f54ed6e6640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f54e815aeb0 0x7f54e815d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:33.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.095+0000 7f54ecee5640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f54e814fe10 0x7f54e806c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:37294/0 (socket says 172.21.15.116:37294) 2024-04-25T00:10:33.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.095+0000 7f54ecee5640 1 -- 172.21.15.116:0/3774309525 learned_addr learned my addr 172.21.15.116:0/3774309525 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:10:33.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.095+0000 7f54e7fff640 1 --1- 172.21.15.116:0/3774309525 >> v1:172.21.15.174:6789/0 conn(0x7f54e806c760 0x7f54e8156180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:60462/0 (socket says 172.21.15.116:60462) 2024-04-25T00:10:33.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.095+0000 7f54ecee5640 1 -- 172.21.15.116:0/3774309525 >> v2:172.21.15.174:3300/0 conn(0x7f54e815aeb0 msgr2=0x7f54e815d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:10:33.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.095+0000 7f54ecee5640 1 --2- 172.21.15.116:0/3774309525 >> v2:172.21.15.174:3300/0 conn(0x7f54e815aeb0 0x7f54e815d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:33.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.095+0000 7f54ecee5640 1 -- 172.21.15.116:0/3774309525 >> v1:172.21.15.174:6789/0 conn(0x7f54e806c760 legacy=0x7f54e8156180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:33.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.095+0000 7f54ecee5640 1 -- 172.21.15.116:0/3774309525 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54e80c6bb0 con 0x7f54e814fe10 2024-04-25T00:10:33.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.095+0000 7f54ed6e6640 1 --2- 172.21.15.116:0/3774309525 >> v2:172.21.15.174:3300/0 conn(0x7f54e815aeb0 0x7f54e815d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:10:33.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.095+0000 7f54ecee5640 1 --2- 172.21.15.116:0/3774309525 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f54e814fe10 0x7f54e806c220 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f54dc0098a0 tx=0x7f54dc01c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e05a105f706d10b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:33.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.095+0000 7f54e77fe640 1 -- 172.21.15.116:0/3774309525 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54dc0b0070 con 0x7f54e814fe10 2024-04-25T00:10:33.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.099+0000 7f54e77fe640 1 -- 172.21.15.116:0/3774309525 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54dc0ab600 con 0x7f54e814fe10 2024-04-25T00:10:33.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.099+0000 7f54e77fe640 1 -- 172.21.15.116:0/3774309525 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54dc0ab9d0 con 0x7f54e814fe10 2024-04-25T00:10:33.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.099+0000 7f54ef149640 1 -- 172.21.15.116:0/3774309525 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f54e814fe10 msgr2=0x7f54e806c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:33.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.099+0000 7f54ef149640 1 --2- 172.21.15.116:0/3774309525 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f54e814fe10 0x7f54e806c220 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f54dc0098a0 tx=0x7f54dc01c420 comp rx=0 tx=0).stop 2024-04-25T00:10:33.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.099+0000 7f54ef149640 1 -- 172.21.15.116:0/3774309525 shutdown_connections 2024-04-25T00:10:33.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.099+0000 7f54ef149640 1 --2- 172.21.15.116:0/3774309525 >> v2:172.21.15.174:3300/0 conn(0x7f54e815aeb0 0x7f54e815d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:33.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.099+0000 7f54ef149640 1 --2- 172.21.15.116:0/3774309525 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f54e814fe10 0x7f54e806c220 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:33.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.099+0000 7f54ef149640 1 -- 172.21.15.116:0/3774309525 >> 172.21.15.116:0/3774309525 conn(0x7f54e80c42e0 msgr2=0x7f54e80c46f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:10:33.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.099+0000 7f54ef149640 1 -- 172.21.15.116:0/3774309525 shutdown_connections 2024-04-25T00:10:33.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.099+0000 7f54ef149640 1 -- 172.21.15.116:0/3774309525 wait complete. 2024-04-25T00:10:33.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.099+0000 7f54ef149640 1 Processor -- start 2024-04-25T00:10:33.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.099+0000 7f54ef149640 1 -- start start 2024-04-25T00:10:33.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.099+0000 7f54ef149640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54e806c760 0x7f54e815c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:33.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.099+0000 7f54ef149640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f54e814fe10 0x7f54e815c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:33.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.099+0000 7f54ef149640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f54e815aeb0 0x7f54e814e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:33.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.099+0000 7f54ef149640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f54e80a72f0 con 0x7f54e806c760 2024-04-25T00:10:33.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.099+0000 7f54ef149640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f54e80a7170 con 0x7f54e814fe10 2024-04-25T00:10:33.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.099+0000 7f54ef149640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f54e8071590 con 0x7f54e815aeb0 2024-04-25T00:10:33.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.099+0000 7f54ecee5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54e806c760 0x7f54e815c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:33.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.099+0000 7f54ecee5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54e806c760 0x7f54e815c020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48590/0 (socket says 172.21.15.116:48590) 2024-04-25T00:10:33.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.099+0000 7f54ecee5640 1 -- 172.21.15.116:0/1313664419 learned_addr learned my addr 172.21.15.116:0/1313664419 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:10:33.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.099+0000 7f54ed6e6640 1 --2- 172.21.15.116:0/1313664419 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f54e815aeb0 0x7f54e814e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:33.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.099+0000 7f54e7fff640 1 --2- 172.21.15.116:0/1313664419 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f54e814fe10 0x7f54e815c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:33.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.099+0000 7f54ecee5640 1 -- 172.21.15.116:0/1313664419 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f54e815aeb0 msgr2=0x7f54e814e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:33.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.099+0000 7f54ecee5640 1 --2- 172.21.15.116:0/1313664419 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f54e815aeb0 0x7f54e814e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:33.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.099+0000 7f54ecee5640 1 -- 172.21.15.116:0/1313664419 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f54e814fe10 msgr2=0x7f54e815c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:33.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.099+0000 7f54ecee5640 1 --2- 172.21.15.116:0/1313664419 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f54e814fe10 0x7f54e815c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:33.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.099+0000 7f54ecee5640 1 -- 172.21.15.116:0/1313664419 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54e80c62a0 con 0x7f54e806c760 2024-04-25T00:10:33.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.103+0000 7f54ecee5640 1 --2- 172.21.15.116:0/1313664419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54e806c760 0x7f54e815c020 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f54dc01aad0 tx=0x7f54dc0191c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:33.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.103+0000 7f54e57fa640 1 -- 172.21.15.116:0/1313664419 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54dc0b0040 con 0x7f54e806c760 2024-04-25T00:10:33.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.103+0000 7f54ef149640 1 -- 172.21.15.116:0/1313664419 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54e81ec800 con 0x7f54e806c760 2024-04-25T00:10:33.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.103+0000 7f54e57fa640 1 -- 172.21.15.116:0/1313664419 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54dc0b9070 con 0x7f54e806c760 2024-04-25T00:10:33.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.103+0000 7f54e57fa640 1 -- 172.21.15.116:0/1313664419 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54dc0b4640 con 0x7f54e806c760 2024-04-25T00:10:33.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.103+0000 7f54ef149640 1 -- 172.21.15.116:0/1313664419 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54e81ecdc0 con 0x7f54e806c760 2024-04-25T00:10:33.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.103+0000 7f54ef149640 1 -- 172.21.15.116:0/1313664419 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54e8068a60 con 0x7f54e806c760 2024-04-25T00:10:33.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.103+0000 7f54e57fa640 1 -- 172.21.15.116:0/1313664419 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f54dc0b47e0 con 0x7f54e806c760 2024-04-25T00:10:33.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.103+0000 7f54e57fa640 1 --2- 172.21.15.116:0/1313664419 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f54cc0678f0 0x7f54cc069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:33.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.103+0000 7f54e57fa640 1 -- 172.21.15.116:0/1313664419 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7698+0+0 (secure 0 0 0) 0x7f54dc11ff20 con 0x7f54e806c760 2024-04-25T00:10:33.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.103+0000 7f54e7fff640 1 --2- 172.21.15.116:0/1313664419 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f54cc0678f0 0x7f54cc069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:33.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.103+0000 7f54e7fff640 1 --2- 172.21.15.116:0/1313664419 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f54cc0678f0 0x7f54cc069db0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f54e8155500 tx=0x7f54d4007e90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:33.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.107+0000 7f54e57fa640 1 -- 172.21.15.116:0/1313664419 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f54dc016130 con 0x7f54e806c760 2024-04-25T00:10:33.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.331+0000 7f54ef149640 1 -- 172.21.15.116:0/1313664419 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f54e80c6490 con 0x7f54e806c760 2024-04-25T00:10:33.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.331+0000 7f54e57fa640 1 -- 172.21.15.116:0/1313664419 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) ==== 94+0+5676 (secure 0 0 0) 0x7f54dc0efc50 con 0x7f54e806c760 2024-04-25T00:10:33.330 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:10:33.331 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6114,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:10:23.780120+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6135},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6135":{"gid":6135,"name":"c","rank":0,"incarnation":68,"state":"up:active","state_seq":13,"addr":"172.21.15.116:6835/1358527848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1358527848},{"type":"v1","addr":"172.21.15.116:6835","nonce":1358527848}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6135,"qdb_cluster":[6135]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:09:33.601524+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":132,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4235,"qdb_cluster":[4235]},"id":2}]} 2024-04-25T00:10:33.331 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 71 2024-04-25T00:10:33.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.335+0000 7f54ef149640 1 -- 172.21.15.116:0/1313664419 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f54cc0678f0 msgr2=0x7f54cc069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:33.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.335+0000 7f54ef149640 1 --2- 172.21.15.116:0/1313664419 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f54cc0678f0 0x7f54cc069db0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f54e8155500 tx=0x7f54d4007e90 comp rx=0 tx=0).stop 2024-04-25T00:10:33.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.335+0000 7f54ef149640 1 -- 172.21.15.116:0/1313664419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54e806c760 msgr2=0x7f54e815c020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:33.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.335+0000 7f54ef149640 1 --2- 172.21.15.116:0/1313664419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54e806c760 0x7f54e815c020 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f54dc01aad0 tx=0x7f54dc0191c0 comp rx=0 tx=0).stop 2024-04-25T00:10:33.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.335+0000 7f54ef149640 1 -- 172.21.15.116:0/1313664419 shutdown_connections 2024-04-25T00:10:33.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.335+0000 7f54ef149640 1 --2- 172.21.15.116:0/1313664419 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f54cc0678f0 0x7f54cc069db0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:33.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.335+0000 7f54ef149640 1 --2- 172.21.15.116:0/1313664419 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f54e815aeb0 0x7f54e814e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:33.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.335+0000 7f54ef149640 1 --2- 172.21.15.116:0/1313664419 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f54e814fe10 0x7f54e815c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:33.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.335+0000 7f54ef149640 1 --2- 172.21.15.116:0/1313664419 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f54e806c760 0x7f54e815c020 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:33.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.335+0000 7f54ef149640 1 -- 172.21.15.116:0/1313664419 >> 172.21.15.116:0/1313664419 conn(0x7f54e80c42e0 msgr2=0x7f54e80c53d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:10:33.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.335+0000 7f54ef149640 1 -- 172.21.15.116:0/1313664419 shutdown_connections 2024-04-25T00:10:33.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:33.335+0000 7f54ef149640 1 -- 172.21.15.116:0/1313664419 wait complete. 2024-04-25T00:10:33.352 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2024-04-25T00:10:35.353 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:10:35.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.483+0000 7fbe8a1e5640 1 Processor -- start 2024-04-25T00:10:35.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.483+0000 7fbe8a1e5640 1 -- start start 2024-04-25T00:10:35.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.483+0000 7fbe8a1e5640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe840c2b10 0x7fbe841541f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:35.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.483+0000 7fbe8a1e5640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbe8406bc20 0x7fbe840c3260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:35.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.483+0000 7fbe8a1e5640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe840734b0 con 0x7fbe84150d10 2024-04-25T00:10:35.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.483+0000 7fbe8a1e5640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fbe84153750 con 0x7fbe8406bc20 2024-04-25T00:10:35.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.483+0000 7fbe8a1e5640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fbe840a9bb0 con 0x7fbe840c2b10 2024-04-25T00:10:35.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.483+0000 7fbe83fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbe8406bc20 0x7fbe840c3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:35.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.483+0000 7fbe83fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbe8406bc20 0x7fbe840c3260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36036/0 (socket says 172.21.15.116:36036) 2024-04-25T00:10:35.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.483+0000 7fbe83fff640 1 -- 172.21.15.116:0/2659275973 learned_addr learned my addr 172.21.15.116:0/2659275973 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:10:35.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.483+0000 7fbe82ffd640 1 --2- 172.21.15.116:0/2659275973 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe840c2b10 0x7fbe841541f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:35.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.483+0000 7fbe83fff640 1 -- 172.21.15.116:0/2659275973 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe840c2b10 msgr2=0x7fbe841541f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:10:35.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.483+0000 7fbe83fff640 1 --2- 172.21.15.116:0/2659275973 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe840c2b10 0x7fbe841541f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:35.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.483+0000 7fbe83fff640 1 -- 172.21.15.116:0/2659275973 >> v1:172.21.15.174:6789/0 conn(0x7fbe84150d10 legacy=0x7fbe840a94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:35.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.483+0000 7fbe83fff640 1 -- 172.21.15.116:0/2659275973 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe840731d0 con 0x7fbe8406bc20 2024-04-25T00:10:35.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.487+0000 7fbe83fff640 1 --2- 172.21.15.116:0/2659275973 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe8406bc20 0x7fbe840c3260 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c009e50 tx=0x7fbe7c01c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6b5d1a8dadade75 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:35.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.487+0000 7fbe827fc640 1 -- 172.21.15.116:0/2659275973 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe7c0a5750 con 0x7fbe8406bc20 2024-04-25T00:10:35.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.487+0000 7fbe827fc640 1 -- 172.21.15.116:0/2659275973 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe7c0a58f0 con 0x7fbe8406bc20 2024-04-25T00:10:35.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.487+0000 7fbe827fc640 1 -- 172.21.15.116:0/2659275973 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe7c0a5cc0 con 0x7fbe8406bc20 2024-04-25T00:10:35.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.487+0000 7fbe8a1e5640 1 -- 172.21.15.116:0/2659275973 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe8406bc20 msgr2=0x7fbe840c3260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:35.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.487+0000 7fbe8a1e5640 1 --2- 172.21.15.116:0/2659275973 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe8406bc20 0x7fbe840c3260 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c009e50 tx=0x7fbe7c01c9a0 comp rx=0 tx=0).stop 2024-04-25T00:10:35.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.487+0000 7fbe8a1e5640 1 -- 172.21.15.116:0/2659275973 shutdown_connections 2024-04-25T00:10:35.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.487+0000 7fbe8a1e5640 1 --2- 172.21.15.116:0/2659275973 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe8406bc20 0x7fbe840c3260 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:35.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.487+0000 7fbe8a1e5640 1 --2- 172.21.15.116:0/2659275973 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe840c2b10 0x7fbe841541f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:35.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.487+0000 7fbe8a1e5640 1 -- 172.21.15.116:0/2659275973 >> 172.21.15.116:0/2659275973 conn(0x7fbe840c4c10 msgr2=0x7fbe840c5020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:10:35.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.487+0000 7fbe8a1e5640 1 -- 172.21.15.116:0/2659275973 shutdown_connections 2024-04-25T00:10:35.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.487+0000 7fbe8a1e5640 1 -- 172.21.15.116:0/2659275973 wait complete. 2024-04-25T00:10:35.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.491+0000 7fbe8a1e5640 1 Processor -- start 2024-04-25T00:10:35.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.491+0000 7fbe8a1e5640 1 -- start start 2024-04-25T00:10:35.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.491+0000 7fbe8a1e5640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe8406bc20 0x7fbe8406ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:35.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.491+0000 7fbe8a1e5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe840c2b10 0x7fbe84140a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:35.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.491+0000 7fbe8a1e5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe84150d10 0x7fbe841eebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:35.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.491+0000 7fbe8a1e5640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbe84151270 con 0x7fbe840c2b10 2024-04-25T00:10:35.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.491+0000 7fbe8a1e5640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbe841510f0 con 0x7fbe84150d10 2024-04-25T00:10:35.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.491+0000 7fbe8a1e5640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fbe84160750 con 0x7fbe8406bc20 2024-04-25T00:10:35.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.491+0000 7fbe82ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe840c2b10 0x7fbe84140a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:35.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.491+0000 7fbe82ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe840c2b10 0x7fbe84140a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36048/0 (socket says 172.21.15.116:36048) 2024-04-25T00:10:35.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.491+0000 7fbe82ffd640 1 -- 172.21.15.116:0/1778379019 learned_addr learned my addr 172.21.15.116:0/1778379019 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:10:35.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.491+0000 7fbe837fe640 1 --2- 172.21.15.116:0/1778379019 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe8406bc20 0x7fbe8406ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:35.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.491+0000 7fbe83fff640 1 --2- 172.21.15.116:0/1778379019 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe84150d10 0x7fbe841eebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:35.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.491+0000 7fbe82ffd640 1 -- 172.21.15.116:0/1778379019 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe8406bc20 msgr2=0x7fbe8406ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:35.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.491+0000 7fbe82ffd640 1 --2- 172.21.15.116:0/1778379019 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe8406bc20 0x7fbe8406ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:35.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.491+0000 7fbe82ffd640 1 -- 172.21.15.116:0/1778379019 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe84150d10 msgr2=0x7fbe841eebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:35.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.491+0000 7fbe82ffd640 1 --2- 172.21.15.116:0/1778379019 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe84150d10 0x7fbe841eebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:35.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.491+0000 7fbe82ffd640 1 -- 172.21.15.116:0/1778379019 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe84153190 con 0x7fbe840c2b10 2024-04-25T00:10:35.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.491+0000 7fbe83fff640 1 --2- 172.21.15.116:0/1778379019 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe84150d10 0x7fbe841eebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:10:35.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.491+0000 7fbe82ffd640 1 --2- 172.21.15.116:0/1778379019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe840c2b10 0x7fbe84140a70 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fbe7800d620 tx=0x7fbe7800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:35.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.491+0000 7fbe6bfff640 1 -- 172.21.15.116:0/1778379019 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe7800b620 con 0x7fbe840c2b10 2024-04-25T00:10:35.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.491+0000 7fbe6bfff640 1 -- 172.21.15.116:0/1778379019 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe78019070 con 0x7fbe840c2b10 2024-04-25T00:10:35.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.491+0000 7fbe6bfff640 1 -- 172.21.15.116:0/1778379019 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe7800b9e0 con 0x7fbe840c2b10 2024-04-25T00:10:35.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.491+0000 7fbe8a1e5640 1 -- 172.21.15.116:0/1778379019 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe8406dfd0 con 0x7fbe840c2b10 2024-04-25T00:10:35.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.491+0000 7fbe8a1e5640 1 -- 172.21.15.116:0/1778379019 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe84152940 con 0x7fbe840c2b10 2024-04-25T00:10:35.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.495+0000 7fbe8a1e5640 1 -- 172.21.15.116:0/1778379019 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe84068a60 con 0x7fbe840c2b10 2024-04-25T00:10:35.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.495+0000 7fbe6bfff640 1 -- 172.21.15.116:0/1778379019 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbe7800bb80 con 0x7fbe840c2b10 2024-04-25T00:10:35.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.495+0000 7fbe6bfff640 1 --2- 172.21.15.116:0/1778379019 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbe700678c0 0x7fbe70069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:35.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.495+0000 7fbe6bfff640 1 -- 172.21.15.116:0/1778379019 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7698+0+0 (secure 0 0 0) 0x7fbe78087250 con 0x7fbe840c2b10 2024-04-25T00:10:35.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.495+0000 7fbe837fe640 1 --2- 172.21.15.116:0/1778379019 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbe700678c0 0x7fbe70069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:35.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.495+0000 7fbe837fe640 1 --2- 172.21.15.116:0/1778379019 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbe700678c0 0x7fbe70069d80 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fbe74001650 tx=0x7fbe74009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:35.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.499+0000 7fbe6bfff640 1 -- 172.21.15.116:0/1778379019 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fbe78057730 con 0x7fbe840c2b10 2024-04-25T00:10:35.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.731+0000 7fbe8a1e5640 1 -- 172.21.15.116:0/1778379019 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbe840c2540 con 0x7fbe840c2b10 2024-04-25T00:10:35.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.731+0000 7fbe6bfff640 1 -- 172.21.15.116:0/1778379019 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) ==== 94+0+6504 (secure 0 0 0) 0x7fbe78057030 con 0x7fbe840c2b10 2024-04-25T00:10:35.729 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:10:35.729 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5461,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72},{"gid":6114,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:10:23.780120+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6135},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6135":{"gid":6135,"name":"c","rank":0,"incarnation":68,"state":"up:active","state_seq":13,"addr":"172.21.15.116:6835/1358527848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1358527848},{"type":"v1","addr":"172.21.15.116:6835","nonce":1358527848}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6135,"qdb_cluster":[6135]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:09:33.601524+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":132,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4235,"qdb_cluster":[4235]},"id":2}]} 2024-04-25T00:10:35.730 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 72 2024-04-25T00:10:35.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.731+0000 7fbe8a1e5640 1 -- 172.21.15.116:0/1778379019 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbe700678c0 msgr2=0x7fbe70069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:35.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.731+0000 7fbe8a1e5640 1 --2- 172.21.15.116:0/1778379019 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbe700678c0 0x7fbe70069d80 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fbe74001650 tx=0x7fbe74009040 comp rx=0 tx=0).stop 2024-04-25T00:10:35.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.731+0000 7fbe8a1e5640 1 -- 172.21.15.116:0/1778379019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe840c2b10 msgr2=0x7fbe84140a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:35.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.731+0000 7fbe8a1e5640 1 --2- 172.21.15.116:0/1778379019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe840c2b10 0x7fbe84140a70 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fbe7800d620 tx=0x7fbe7800dae0 comp rx=0 tx=0).stop 2024-04-25T00:10:35.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.735+0000 7fbe8a1e5640 1 -- 172.21.15.116:0/1778379019 shutdown_connections 2024-04-25T00:10:35.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.735+0000 7fbe8a1e5640 1 --2- 172.21.15.116:0/1778379019 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbe700678c0 0x7fbe70069d80 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:35.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.735+0000 7fbe8a1e5640 1 --2- 172.21.15.116:0/1778379019 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe84150d10 0x7fbe841eebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:35.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.735+0000 7fbe8a1e5640 1 --2- 172.21.15.116:0/1778379019 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe840c2b10 0x7fbe84140a70 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:35.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.735+0000 7fbe8a1e5640 1 --2- 172.21.15.116:0/1778379019 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe8406bc20 0x7fbe8406ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:35.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.735+0000 7fbe8a1e5640 1 -- 172.21.15.116:0/1778379019 >> 172.21.15.116:0/1778379019 conn(0x7fbe840c4c10 msgr2=0x7fbe840c1d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:10:35.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.735+0000 7fbe8a1e5640 1 -- 172.21.15.116:0/1778379019 shutdown_connections 2024-04-25T00:10:35.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:35.735+0000 7fbe8a1e5640 1 -- 172.21.15.116:0/1778379019 wait complete. 2024-04-25T00:10:35.750 INFO:tasks.mds_thrash.fs.[a]:mds.b reported in up:standby state 2024-04-25T00:10:35.750 INFO:tasks.mds_thrash.fs.[a]:waiting for 7 secs before thrashing 2024-04-25T00:10:37.058 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:10:37.061 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:10:42.751 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:10:42.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.887+0000 7f3c78dfa640 1 Processor -- start 2024-04-25T00:10:42.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.887+0000 7f3c78dfa640 1 -- start start 2024-04-25T00:10:42.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.891+0000 7f3c78dfa640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3c74152d80 0x7f3c74159950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:42.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.891+0000 7f3c78dfa640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c740730b0 con 0x7f3c74072080 2024-04-25T00:10:42.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.891+0000 7f3c78dfa640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c74073390 con 0x7f3c741562a0 2024-04-25T00:10:42.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.891+0000 7f3c78dfa640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f3c74072e60 con 0x7f3c74152d80 2024-04-25T00:10:42.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.891+0000 7f3c6bfff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f3c74072080 0x7f3c740c36b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:39192/0 (socket says 172.21.15.116:39192) 2024-04-25T00:10:42.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.891+0000 7f3c6bfff640 1 -- 172.21.15.116:0/3533232361 learned_addr learned my addr 172.21.15.116:0/3533232361 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:10:42.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.891+0000 7f3c7259c640 1 --2- 172.21.15.116:0/3533232361 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3c74152d80 0x7f3c74159950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:42.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.891+0000 7f3c71d9b640 1 -- 172.21.15.116:0/3533232361 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 209609992 0 0) 0x7f3c740730b0 con 0x7f3c74072080 2024-04-25T00:10:42.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.891+0000 7f3c71d9b640 1 -- 172.21.15.116:0/3533232361 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3c60003530 con 0x7f3c74072080 2024-04-25T00:10:42.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.891+0000 7f3c71d9b640 1 -- 172.21.15.116:0/3533232361 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4239038710 0 0) 0x7f3c60003530 con 0x7f3c74072080 2024-04-25T00:10:42.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.891+0000 7f3c71d9b640 1 -- 172.21.15.116:0/3533232361 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3c74152d80 msgr2=0x7f3c74159950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:10:42.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.891+0000 7f3c71d9b640 1 --2- 172.21.15.116:0/3533232361 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3c74152d80 0x7f3c74159950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:42.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.891+0000 7f3c71d9b640 1 -- 172.21.15.116:0/3533232361 >> v1:172.21.15.174:6789/0 conn(0x7f3c741562a0 legacy=0x7f3c740c5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:42.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.891+0000 7f3c71d9b640 1 -- 172.21.15.116:0/3533232361 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c74158e70 con 0x7f3c74072080 2024-04-25T00:10:42.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.891+0000 7f3c71d9b640 1 -- 172.21.15.116:0/3533232361 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f3c5c001680 con 0x7f3c74072080 2024-04-25T00:10:42.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.891+0000 7f3c71d9b640 1 -- 172.21.15.116:0/3533232361 >> v1:172.21.15.116:6789/0 conn(0x7f3c74072080 legacy=0x7f3c740c36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:42.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.891+0000 7f3c71d9b640 1 --2- 172.21.15.116:0/3533232361 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c600039f0 0x7f3c60003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:42.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.891+0000 7f3c71d9b640 1 --2- 172.21.15.116:0/3533232361 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c60004810 0x7f3c60004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:42.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.891+0000 7f3c71d9b640 1 --2- 172.21.15.116:0/3533232361 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3c74152d80 0x7f3c60005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:42.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.891+0000 7f3c71d9b640 1 -- 172.21.15.116:0/3533232361 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3c74072e60 con 0x7f3c600039f0 2024-04-25T00:10:42.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.891+0000 7f3c71d9b640 1 -- 172.21.15.116:0/3533232361 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3c6000bb00 con 0x7f3c60004810 2024-04-25T00:10:42.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.891+0000 7f3c71d9b640 1 -- 172.21.15.116:0/3533232361 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f3c6000bd00 con 0x7f3c74152d80 2024-04-25T00:10:42.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.891+0000 7f3c7259c640 1 --2- 172.21.15.116:0/3533232361 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c600039f0 0x7f3c60003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:42.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.891+0000 7f3c6bfff640 1 --2- 172.21.15.116:0/3533232361 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c60004810 0x7f3c60004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:42.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.891+0000 7f3c7259c640 1 -- 172.21.15.116:0/3533232361 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3c74152d80 msgr2=0x7f3c60005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:10:42.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.891+0000 7f3c7259c640 1 --2- 172.21.15.116:0/3533232361 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3c74152d80 0x7f3c60005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:42.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.891+0000 7f3c7259c640 1 -- 172.21.15.116:0/3533232361 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c60004810 msgr2=0x7f3c60004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:10:42.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.891+0000 7f3c7259c640 1 --2- 172.21.15.116:0/3533232361 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c60004810 0x7f3c60004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:42.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.891+0000 7f3c7259c640 1 -- 172.21.15.116:0/3533232361 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c60003530 con 0x7f3c600039f0 2024-04-25T00:10:42.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.895+0000 7f3c7259c640 1 --2- 172.21.15.116:0/3533232361 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c600039f0 0x7f3c60003e10 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f3c64009ae0 tx=0x7f3c6401ca80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8df5d64355b6d72c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:42.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.895+0000 7f3c71d9b640 1 -- 172.21.15.116:0/3533232361 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c640b2070 con 0x7f3c600039f0 2024-04-25T00:10:42.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.895+0000 7f3c71d9b640 1 -- 172.21.15.116:0/3533232361 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c640a5560 con 0x7f3c600039f0 2024-04-25T00:10:42.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.895+0000 7f3c71d9b640 1 -- 172.21.15.116:0/3533232361 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c640096b0 con 0x7f3c600039f0 2024-04-25T00:10:42.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.895+0000 7f3c78dfa640 1 -- 172.21.15.116:0/3533232361 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c600039f0 msgr2=0x7f3c60003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:42.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.895+0000 7f3c78dfa640 1 --2- 172.21.15.116:0/3533232361 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c600039f0 0x7f3c60003e10 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f3c64009ae0 tx=0x7f3c6401ca80 comp rx=0 tx=0).stop 2024-04-25T00:10:42.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.895+0000 7f3c72d9d640 1 -- 172.21.15.116:0/3533232361 reap_dead start 2024-04-25T00:10:42.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.895+0000 7f3c78dfa640 1 -- 172.21.15.116:0/3533232361 shutdown_connections 2024-04-25T00:10:42.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.895+0000 7f3c78dfa640 1 -- 172.21.15.116:0/3533232361 >> 172.21.15.116:0/3533232361 conn(0x7f3c7406c720 msgr2=0x7f3c7406cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:10:42.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.895+0000 7f3c78dfa640 1 -- 172.21.15.116:0/3533232361 shutdown_connections 2024-04-25T00:10:42.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.895+0000 7f3c78dfa640 1 -- 172.21.15.116:0/3533232361 wait complete. 2024-04-25T00:10:42.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.895+0000 7f3c78dfa640 1 Processor -- start 2024-04-25T00:10:42.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.895+0000 7f3c78dfa640 1 -- start start 2024-04-25T00:10:42.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.895+0000 7f3c78dfa640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c600039f0 0x7f3c74151540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:42.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.895+0000 7f3c78dfa640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c7414fe50 0x7f3c741f1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:42.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.895+0000 7f3c78dfa640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3c74156a60 0x7f3c741f1e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:42.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.895+0000 7f3c78dfa640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3c740c5730 con 0x7f3c600039f0 2024-04-25T00:10:42.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.895+0000 7f3c78dfa640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3c741566b0 con 0x7f3c7414fe50 2024-04-25T00:10:42.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.895+0000 7f3c78dfa640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f3c740c58b0 con 0x7f3c74156a60 2024-04-25T00:10:42.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.895+0000 7f3c7259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c600039f0 0x7f3c74151540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:42.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.895+0000 7f3c7259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c600039f0 0x7f3c74151540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36064/0 (socket says 172.21.15.116:36064) 2024-04-25T00:10:42.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.895+0000 7f3c7259c640 1 -- 172.21.15.116:0/931230088 learned_addr learned my addr 172.21.15.116:0/931230088 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:10:42.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.895+0000 7f3c6bfff640 1 --2- 172.21.15.116:0/931230088 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c7414fe50 0x7f3c741f1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:42.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.895+0000 7f3c72d9d640 1 --2- 172.21.15.116:0/931230088 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3c74156a60 0x7f3c741f1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:42.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.895+0000 7f3c7259c640 1 -- 172.21.15.116:0/931230088 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3c74156a60 msgr2=0x7f3c741f1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:42.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.895+0000 7f3c7259c640 1 --2- 172.21.15.116:0/931230088 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3c74156a60 0x7f3c741f1e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:42.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.895+0000 7f3c7259c640 1 -- 172.21.15.116:0/931230088 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c7414fe50 msgr2=0x7f3c741f1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:42.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.895+0000 7f3c7259c640 1 --2- 172.21.15.116:0/931230088 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c7414fe50 0x7f3c741f1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:42.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.895+0000 7f3c7259c640 1 -- 172.21.15.116:0/931230088 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c741472d0 con 0x7f3c600039f0 2024-04-25T00:10:42.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.895+0000 7f3c7259c640 1 --2- 172.21.15.116:0/931230088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c600039f0 0x7f3c74151540 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f3c640b0040 tx=0x7f3c60005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:42.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.899+0000 7f3c6affd640 1 -- 172.21.15.116:0/931230088 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c640bc030 con 0x7f3c600039f0 2024-04-25T00:10:42.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.899+0000 7f3c6affd640 1 -- 172.21.15.116:0/931230088 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c6400f000 con 0x7f3c600039f0 2024-04-25T00:10:42.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.899+0000 7f3c6affd640 1 -- 172.21.15.116:0/931230088 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c640b2040 con 0x7f3c600039f0 2024-04-25T00:10:42.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.899+0000 7f3c78dfa640 1 -- 172.21.15.116:0/931230088 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c7414afb0 con 0x7f3c600039f0 2024-04-25T00:10:42.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.899+0000 7f3c78dfa640 1 -- 172.21.15.116:0/931230088 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c74151a80 con 0x7f3c600039f0 2024-04-25T00:10:42.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.899+0000 7f3c78dfa640 1 -- 172.21.15.116:0/931230088 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c74068a60 con 0x7f3c600039f0 2024-04-25T00:10:42.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.899+0000 7f3c6affd640 1 -- 172.21.15.116:0/931230088 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3c640bf070 con 0x7f3c600039f0 2024-04-25T00:10:42.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.899+0000 7f3c6affd640 1 --2- 172.21.15.116:0/931230088 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3c540678c0 0x7f3c54069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:42.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.899+0000 7f3c6affd640 1 -- 172.21.15.116:0/931230088 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7698+0+0 (secure 0 0 0) 0x7f3c64120080 con 0x7f3c600039f0 2024-04-25T00:10:42.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.899+0000 7f3c6bfff640 1 --2- 172.21.15.116:0/931230088 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3c540678c0 0x7f3c54069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:42.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.899+0000 7f3c6bfff640 1 --2- 172.21.15.116:0/931230088 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3c540678c0 0x7f3c54069d80 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c001ec0 tx=0x7f3c5c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:42.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:42.903+0000 7f3c6affd640 1 -- 172.21.15.116:0/931230088 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f3c640c7020 con 0x7f3c600039f0 2024-04-25T00:10:43.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.131+0000 7f3c78dfa640 1 -- 172.21.15.116:0/931230088 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3c74150580 con 0x7f3c600039f0 2024-04-25T00:10:43.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.135+0000 7f3c6affd640 1 -- 172.21.15.116:0/931230088 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) ==== 94+0+6504 (secure 0 0 0) 0x7f3c640167c0 con 0x7f3c600039f0 2024-04-25T00:10:43.132 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:10:43.132 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5461,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72},{"gid":6114,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:10:23.780120+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6135},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6135":{"gid":6135,"name":"c","rank":0,"incarnation":68,"state":"up:active","state_seq":13,"addr":"172.21.15.116:6835/1358527848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1358527848},{"type":"v1","addr":"172.21.15.116:6835","nonce":1358527848}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6135,"qdb_cluster":[6135]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:09:33.601524+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":132,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4235,"qdb_cluster":[4235]},"id":2}]} 2024-04-25T00:10:43.133 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 72 2024-04-25T00:10:43.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.135+0000 7f3c78dfa640 1 -- 172.21.15.116:0/931230088 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3c540678c0 msgr2=0x7f3c54069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:43.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.135+0000 7f3c78dfa640 1 --2- 172.21.15.116:0/931230088 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3c540678c0 0x7f3c54069d80 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c001ec0 tx=0x7f3c5c009040 comp rx=0 tx=0).stop 2024-04-25T00:10:43.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.135+0000 7f3c78dfa640 1 -- 172.21.15.116:0/931230088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c600039f0 msgr2=0x7f3c74151540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:43.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.135+0000 7f3c78dfa640 1 --2- 172.21.15.116:0/931230088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c600039f0 0x7f3c74151540 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f3c640b0040 tx=0x7f3c60005840 comp rx=0 tx=0).stop 2024-04-25T00:10:43.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.135+0000 7f3c78dfa640 1 -- 172.21.15.116:0/931230088 shutdown_connections 2024-04-25T00:10:43.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.135+0000 7f3c78dfa640 1 --2- 172.21.15.116:0/931230088 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3c540678c0 0x7f3c54069d80 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:43.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.135+0000 7f3c78dfa640 1 --2- 172.21.15.116:0/931230088 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3c74156a60 0x7f3c741f1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:43.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.135+0000 7f3c78dfa640 1 --2- 172.21.15.116:0/931230088 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3c7414fe50 0x7f3c741f1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:43.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.135+0000 7f3c78dfa640 1 --2- 172.21.15.116:0/931230088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3c600039f0 0x7f3c74151540 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:43.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.135+0000 7f3c78dfa640 1 -- 172.21.15.116:0/931230088 >> 172.21.15.116:0/931230088 conn(0x7f3c7406c720 msgr2=0x7f3c74150b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:10:43.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.135+0000 7f3c78dfa640 1 -- 172.21.15.116:0/931230088 shutdown_connections 2024-04-25T00:10:43.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.135+0000 7f3c78dfa640 1 -- 172.21.15.116:0/931230088 wait complete. 2024-04-25T00:10:43.152 INFO:tasks.mds_thrash.fs.[a]:kill mds.c (rank=0) 2024-04-25T00:10:43.153 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-25T00:10:43.153 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:10:43.164 INFO:tasks.ceph.mds.c:Stopped 2024-04-25T00:10:43.164 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:10:43.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.335+0000 7f6865bbd640 1 Processor -- start 2024-04-25T00:10:43.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.335+0000 7f6865bbd640 1 -- start start 2024-04-25T00:10:43.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.335+0000 7f6865bbd640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f68600aeb20 0x7f68600c9310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:43.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.335+0000 7f6865bbd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6860150f80 0x7f68600c5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:43.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.335+0000 7f6865bbd640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68601535d0 con 0x7f68600c9850 2024-04-25T00:10:43.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.335+0000 7f6865bbd640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f6860153850 con 0x7f6860150f80 2024-04-25T00:10:43.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.335+0000 7f6865bbd640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f6860153b30 con 0x7f68600aeb20 2024-04-25T00:10:43.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.335+0000 7f685ffff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6860150f80 0x7f68600c5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:43.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.335+0000 7f685ffff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6860150f80 0x7f68600c5270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36076/0 (socket says 172.21.15.116:36076) 2024-04-25T00:10:43.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.335+0000 7f685ffff640 1 -- 172.21.15.116:0/3517636931 learned_addr learned my addr 172.21.15.116:0/3517636931 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:10:43.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.335+0000 7f685f7fe640 1 --2- 172.21.15.116:0/3517636931 >> v2:172.21.15.174:3300/0 conn(0x7f68600aeb20 0x7f68600c9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:43.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.335+0000 7f685ffff640 1 -- 172.21.15.116:0/3517636931 >> v2:172.21.15.174:3300/0 conn(0x7f68600aeb20 msgr2=0x7f68600c9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:10:43.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.335+0000 7f685ffff640 1 --2- 172.21.15.116:0/3517636931 >> v2:172.21.15.174:3300/0 conn(0x7f68600aeb20 0x7f68600c9310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:43.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.335+0000 7f685ffff640 1 -- 172.21.15.116:0/3517636931 >> v1:172.21.15.116:6789/0 conn(0x7f68600c9850 legacy=0x7f6860150870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:43.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.335+0000 7f685effd640 1 -- 172.21.15.116:0/3517636931 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 655309099 0 0) 0x7f6848001120 con 0x7f68600c9850 2024-04-25T00:10:43.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.335+0000 7f685ffff640 1 -- 172.21.15.116:0/3517636931 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6860153e10 con 0x7f6860150f80 2024-04-25T00:10:43.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.339+0000 7f685ffff640 1 --2- 172.21.15.116:0/3517636931 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6860150f80 0x7f68600c5270 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f68600705f0 tx=0x7f685801c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbc41f4322b46866 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:43.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.339+0000 7f685effd640 1 -- 172.21.15.116:0/3517636931 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68580a5680 con 0x7f6860150f80 2024-04-25T00:10:43.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.339+0000 7f685effd640 1 -- 172.21.15.116:0/3517636931 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68580a5820 con 0x7f6860150f80 2024-04-25T00:10:43.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.339+0000 7f685effd640 1 -- 172.21.15.116:0/3517636931 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68580a5bf0 con 0x7f6860150f80 2024-04-25T00:10:43.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.339+0000 7f6865bbd640 1 -- 172.21.15.116:0/3517636931 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6860150f80 msgr2=0x7f68600c5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:43.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.339+0000 7f6865bbd640 1 --2- 172.21.15.116:0/3517636931 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6860150f80 0x7f68600c5270 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f68600705f0 tx=0x7f685801c8d0 comp rx=0 tx=0).stop 2024-04-25T00:10:43.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.339+0000 7f6865bbd640 1 -- 172.21.15.116:0/3517636931 shutdown_connections 2024-04-25T00:10:43.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.339+0000 7f6865bbd640 1 --2- 172.21.15.116:0/3517636931 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6860150f80 0x7f68600c5270 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:43.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.339+0000 7f6865bbd640 1 --2- 172.21.15.116:0/3517636931 >> v2:172.21.15.174:3300/0 conn(0x7f68600aeb20 0x7f68600c9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:43.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.339+0000 7f6865bbd640 1 -- 172.21.15.116:0/3517636931 >> 172.21.15.116:0/3517636931 conn(0x7f6860074650 msgr2=0x7f6860074a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:10:43.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.339+0000 7f6865bbd640 1 -- 172.21.15.116:0/3517636931 shutdown_connections 2024-04-25T00:10:43.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.339+0000 7f6865bbd640 1 -- 172.21.15.116:0/3517636931 wait complete. 2024-04-25T00:10:43.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.339+0000 7f6865bbd640 1 Processor -- start 2024-04-25T00:10:43.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.339+0000 7f6865bbd640 1 -- start start 2024-04-25T00:10:43.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.343+0000 7f6865bbd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68600aeb20 0x7f68600be020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:43.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.343+0000 7f6865bbd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f68600c9850 0x7f68601ef0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:43.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.343+0000 7f6865bbd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6860150f80 0x7f68601ef630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:43.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.343+0000 7f6865bbd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f686015ff60 con 0x7f68600aeb20 2024-04-25T00:10:43.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.343+0000 7f6865bbd640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f68600af150 con 0x7f6860150f80 2024-04-25T00:10:43.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.343+0000 7f6865bbd640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f68601600e0 con 0x7f68600c9850 2024-04-25T00:10:43.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.343+0000 7f685f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68600aeb20 0x7f68600be020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:43.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.343+0000 7f685f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68600aeb20 0x7f68600be020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36092/0 (socket says 172.21.15.116:36092) 2024-04-25T00:10:43.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.343+0000 7f685f7fe640 1 -- 172.21.15.116:0/2843624003 learned_addr learned my addr 172.21.15.116:0/2843624003 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:10:43.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.343+0000 7f685ffff640 1 --2- 172.21.15.116:0/2843624003 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6860150f80 0x7f68601ef630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:43.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.343+0000 7f6856ffd640 1 --2- 172.21.15.116:0/2843624003 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f68600c9850 0x7f68601ef0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:43.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.343+0000 7f685f7fe640 1 -- 172.21.15.116:0/2843624003 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f68600c9850 msgr2=0x7f68601ef0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:43.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.343+0000 7f685f7fe640 1 --2- 172.21.15.116:0/2843624003 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f68600c9850 0x7f68601ef0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:43.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.343+0000 7f685f7fe640 1 -- 172.21.15.116:0/2843624003 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6860150f80 msgr2=0x7f68601ef630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:43.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.343+0000 7f685f7fe640 1 --2- 172.21.15.116:0/2843624003 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6860150f80 0x7f68601ef630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:43.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.343+0000 7f685f7fe640 1 -- 172.21.15.116:0/2843624003 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6860147020 con 0x7f68600aeb20 2024-04-25T00:10:43.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.343+0000 7f685f7fe640 1 --2- 172.21.15.116:0/2843624003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68600aeb20 0x7f68600be020 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f685000d800 tx=0x7f685000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:43.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.343+0000 7f685cff9640 1 -- 172.21.15.116:0/2843624003 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f685000b6c0 con 0x7f68600aeb20 2024-04-25T00:10:43.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.343+0000 7f685cff9640 1 -- 172.21.15.116:0/2843624003 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6850019070 con 0x7f68600aeb20 2024-04-25T00:10:43.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.343+0000 7f6865bbd640 1 -- 172.21.15.116:0/2843624003 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68601efb70 con 0x7f68600aeb20 2024-04-25T00:10:43.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.343+0000 7f685cff9640 1 -- 172.21.15.116:0/2843624003 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f685000ba60 con 0x7f68600aeb20 2024-04-25T00:10:43.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.343+0000 7f6865bbd640 1 -- 172.21.15.116:0/2843624003 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68601efde0 con 0x7f68600aeb20 2024-04-25T00:10:43.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.343+0000 7f6865bbd640 1 -- 172.21.15.116:0/2843624003 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6860068a60 con 0x7f68600aeb20 2024-04-25T00:10:43.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.343+0000 7f685cff9640 1 -- 172.21.15.116:0/2843624003 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f685000bc00 con 0x7f68600aeb20 2024-04-25T00:10:43.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.343+0000 7f685cff9640 1 --2- 172.21.15.116:0/2843624003 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f68400678c0 0x7f6840069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:43.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.343+0000 7f685cff9640 1 -- 172.21.15.116:0/2843624003 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7698+0+0 (secure 0 0 0) 0x7f6850086890 con 0x7f68600aeb20 2024-04-25T00:10:43.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.343+0000 7f6856ffd640 1 --2- 172.21.15.116:0/2843624003 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f68400678c0 0x7f6840069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:43.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.347+0000 7f6856ffd640 1 --2- 172.21.15.116:0/2843624003 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f68400678c0 0x7f6840069d80 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f6860152da0 tx=0x7f6848009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:43.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.347+0000 7f685cff9640 1 -- 172.21.15.116:0/2843624003 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f6850056c90 con 0x7f68600aeb20 2024-04-25T00:10:43.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.551+0000 7f6865bbd640 1 -- 172.21.15.116:0/2843624003 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f68600705d0 con 0x7f68600aeb20 2024-04-25T00:10:43.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.555+0000 7f685cff9640 1 -- 172.21.15.116:0/2843624003 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) ==== 94+0+6504 (secure 0 0 0) 0x7f6850056590 con 0x7f68600aeb20 2024-04-25T00:10:43.552 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:10:43.552 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5461,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72},{"gid":6114,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:10:23.780120+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6135},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6135":{"gid":6135,"name":"c","rank":0,"incarnation":68,"state":"up:active","state_seq":13,"addr":"172.21.15.116:6835/1358527848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1358527848},{"type":"v1","addr":"172.21.15.116:6835","nonce":1358527848}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6135,"qdb_cluster":[6135]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:09:33.601524+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":132,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4235,"qdb_cluster":[4235]},"id":2}]} 2024-04-25T00:10:43.552 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 72 2024-04-25T00:10:43.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.555+0000 7f6865bbd640 1 -- 172.21.15.116:0/2843624003 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f68400678c0 msgr2=0x7f6840069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:43.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.555+0000 7f6865bbd640 1 --2- 172.21.15.116:0/2843624003 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f68400678c0 0x7f6840069d80 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f6860152da0 tx=0x7f6848009040 comp rx=0 tx=0).stop 2024-04-25T00:10:43.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.555+0000 7f6865bbd640 1 -- 172.21.15.116:0/2843624003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68600aeb20 msgr2=0x7f68600be020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:43.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.555+0000 7f6865bbd640 1 --2- 172.21.15.116:0/2843624003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68600aeb20 0x7f68600be020 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f685000d800 tx=0x7f685000dcc0 comp rx=0 tx=0).stop 2024-04-25T00:10:43.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.555+0000 7f6865bbd640 1 -- 172.21.15.116:0/2843624003 shutdown_connections 2024-04-25T00:10:43.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.555+0000 7f6865bbd640 1 --2- 172.21.15.116:0/2843624003 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f68400678c0 0x7f6840069d80 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:43.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.555+0000 7f6865bbd640 1 --2- 172.21.15.116:0/2843624003 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6860150f80 0x7f68601ef630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:43.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.555+0000 7f6865bbd640 1 --2- 172.21.15.116:0/2843624003 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f68600c9850 0x7f68601ef0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:43.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.555+0000 7f6865bbd640 1 --2- 172.21.15.116:0/2843624003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68600aeb20 0x7f68600be020 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:43.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.555+0000 7f6865bbd640 1 -- 172.21.15.116:0/2843624003 >> 172.21.15.116:0/2843624003 conn(0x7f6860074650 msgr2=0x7f6860074a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:10:43.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.555+0000 7f6865bbd640 1 -- 172.21.15.116:0/2843624003 shutdown_connections 2024-04-25T00:10:43.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:43.555+0000 7f6865bbd640 1 -- 172.21.15.116:0/2843624003 wait complete. 2024-04-25T00:10:43.571 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:10:45.572 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:10:45.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.699+0000 7f06162eb640 1 Processor -- start 2024-04-25T00:10:45.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.699+0000 7f06162eb640 1 -- start start 2024-04-25T00:10:45.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.703+0000 7f06162eb640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f0610150bf0 0x7f061006c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:45.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.703+0000 7f06162eb640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0610153570 0x7f061006eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:45.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.703+0000 7f06162eb640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0610153aa0 con 0x7f061006c760 2024-04-25T00:10:45.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.703+0000 7f06162eb640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f061006e630 con 0x7f0610150bf0 2024-04-25T00:10:45.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.703+0000 7f06162eb640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f0610150fd0 con 0x7f0610153570 2024-04-25T00:10:45.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.703+0000 7f060f7fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0610153570 0x7f061006eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:45.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.703+0000 7f060ffff640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f0610150bf0 0x7f061006c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:45.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.703+0000 7f060ffff640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f0610150bf0 0x7f061006c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:55942/0 (socket says 172.21.15.116:55942) 2024-04-25T00:10:45.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.703+0000 7f060ffff640 1 -- 172.21.15.116:0/3822806379 learned_addr learned my addr 172.21.15.116:0/3822806379 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:10:45.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.703+0000 7f0614888640 1 --1- 172.21.15.116:0/3822806379 >> v1:172.21.15.174:6789/0 conn(0x7f061006c760 0x7f061006f000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:36996/0 (socket says 172.21.15.116:36996) 2024-04-25T00:10:45.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.703+0000 7f060ffff640 1 -- 172.21.15.116:0/3822806379 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0610153570 msgr2=0x7f061006eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:10:45.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.703+0000 7f060ffff640 1 --2- 172.21.15.116:0/3822806379 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0610153570 0x7f061006eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:45.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.703+0000 7f060ffff640 1 -- 172.21.15.116:0/3822806379 >> v1:172.21.15.174:6789/0 conn(0x7f061006c760 legacy=0x7f061006f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:45.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.703+0000 7f060ffff640 1 -- 172.21.15.116:0/3822806379 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0610153d80 con 0x7f0610150bf0 2024-04-25T00:10:45.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.707+0000 7f060ffff640 1 --2- 172.21.15.116:0/3822806379 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0610150bf0 0x7f061006c220 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f061006e7b0 tx=0x7f05f801c950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a48e6b7a722d76e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:45.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.707+0000 7f060effd640 1 -- 172.21.15.116:0/3822806379 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f80a5810 con 0x7f0610150bf0 2024-04-25T00:10:45.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.707+0000 7f060effd640 1 -- 172.21.15.116:0/3822806379 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05f80a59b0 con 0x7f0610150bf0 2024-04-25T00:10:45.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.707+0000 7f060effd640 1 -- 172.21.15.116:0/3822806379 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f80a5d80 con 0x7f0610150bf0 2024-04-25T00:10:45.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.707+0000 7f06162eb640 1 -- 172.21.15.116:0/3822806379 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0610150bf0 msgr2=0x7f061006c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:45.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.707+0000 7f06162eb640 1 --2- 172.21.15.116:0/3822806379 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0610150bf0 0x7f061006c220 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f061006e7b0 tx=0x7f05f801c950 comp rx=0 tx=0).stop 2024-04-25T00:10:45.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.707+0000 7f06162eb640 1 -- 172.21.15.116:0/3822806379 shutdown_connections 2024-04-25T00:10:45.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.707+0000 7f06162eb640 1 --2- 172.21.15.116:0/3822806379 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0610153570 0x7f061006eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:45.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.707+0000 7f06162eb640 1 --2- 172.21.15.116:0/3822806379 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0610150bf0 0x7f061006c220 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:45.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.707+0000 7f06162eb640 1 -- 172.21.15.116:0/3822806379 >> 172.21.15.116:0/3822806379 conn(0x7f06100c4270 msgr2=0x7f06100c4680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:10:45.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.707+0000 7f06162eb640 1 -- 172.21.15.116:0/3822806379 shutdown_connections 2024-04-25T00:10:45.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.707+0000 7f06162eb640 1 -- 172.21.15.116:0/3822806379 wait complete. 2024-04-25T00:10:45.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.707+0000 7f06162eb640 1 Processor -- start 2024-04-25T00:10:45.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.707+0000 7f06162eb640 1 -- start start 2024-04-25T00:10:45.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.707+0000 7f06162eb640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f061006c760 0x7f06101476f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:45.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.707+0000 7f06162eb640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0610150bf0 0x7f06100aca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:45.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.707+0000 7f06162eb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0610153570 0x7f06101552b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:45.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.707+0000 7f06162eb640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f06100a72b0 con 0x7f0610153570 2024-04-25T00:10:45.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.707+0000 7f06162eb640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f06100a7130 con 0x7f061006c760 2024-04-25T00:10:45.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.707+0000 7f06162eb640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f0610160710 con 0x7f0610150bf0 2024-04-25T00:10:45.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.707+0000 7f0614888640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0610153570 0x7f06101552b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:45.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.707+0000 7f0614888640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0610153570 0x7f06101552b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50778/0 (socket says 172.21.15.116:50778) 2024-04-25T00:10:45.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.707+0000 7f0614888640 1 -- 172.21.15.116:0/3588664260 learned_addr learned my addr 172.21.15.116:0/3588664260 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:10:45.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.707+0000 7f060ffff640 1 --2- 172.21.15.116:0/3588664260 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f061006c760 0x7f06101476f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:45.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.707+0000 7f060f7fe640 1 --2- 172.21.15.116:0/3588664260 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0610150bf0 0x7f06100aca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:45.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.707+0000 7f0614888640 1 -- 172.21.15.116:0/3588664260 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0610150bf0 msgr2=0x7f06100aca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:45.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.707+0000 7f0614888640 1 --2- 172.21.15.116:0/3588664260 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0610150bf0 0x7f06100aca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:45.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.707+0000 7f0614888640 1 -- 172.21.15.116:0/3588664260 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f061006c760 msgr2=0x7f06101476f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:45.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.707+0000 7f0614888640 1 --2- 172.21.15.116:0/3588664260 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f061006c760 0x7f06101476f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:45.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.707+0000 7f0614888640 1 -- 172.21.15.116:0/3588664260 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0610156f00 con 0x7f0610153570 2024-04-25T00:10:45.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.711+0000 7f0614888640 1 --2- 172.21.15.116:0/3588664260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0610153570 0x7f06101552b0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f060000c3c0 tx=0x7f060000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:45.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.711+0000 7f060cff9640 1 -- 172.21.15.116:0/3588664260 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06000074e0 con 0x7f0610153570 2024-04-25T00:10:45.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.711+0000 7f060cff9640 1 -- 172.21.15.116:0/3588664260 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0600007680 con 0x7f0610153570 2024-04-25T00:10:45.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.711+0000 7f06162eb640 1 -- 172.21.15.116:0/3588664260 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06101571e0 con 0x7f0610153570 2024-04-25T00:10:45.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.711+0000 7f060cff9640 1 -- 172.21.15.116:0/3588664260 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0600004a10 con 0x7f0610153570 2024-04-25T00:10:45.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.711+0000 7f06162eb640 1 -- 172.21.15.116:0/3588664260 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0610154d90 con 0x7f0610153570 2024-04-25T00:10:45.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.711+0000 7f06162eb640 1 -- 172.21.15.116:0/3588664260 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0610154f90 con 0x7f0610153570 2024-04-25T00:10:45.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.711+0000 7f060cff9640 1 -- 172.21.15.116:0/3588664260 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f060001a020 con 0x7f0610153570 2024-04-25T00:10:45.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.711+0000 7f060cff9640 1 --2- 172.21.15.116:0/3588664260 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f05f00678c0 0x7f05f0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:45.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.711+0000 7f060cff9640 1 -- 172.21.15.116:0/3588664260 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7698+0+0 (secure 0 0 0) 0x7f0600085b50 con 0x7f0610153570 2024-04-25T00:10:45.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.711+0000 7f060ffff640 1 --2- 172.21.15.116:0/3588664260 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f05f00678c0 0x7f05f0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:45.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.711+0000 7f060ffff640 1 --2- 172.21.15.116:0/3588664260 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f05f00678c0 0x7f05f0069d80 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f05f80ae000 tx=0x7f05f80a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:45.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.715+0000 7f060cff9640 1 -- 172.21.15.116:0/3588664260 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f0610154f90 con 0x7f0610153570 2024-04-25T00:10:45.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.911+0000 7f06162eb640 1 -- 172.21.15.116:0/3588664260 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0610068a60 con 0x7f0610153570 2024-04-25T00:10:45.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.911+0000 7f060cff9640 1 -- 172.21.15.116:0/3588664260 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) ==== 94+0+6504 (secure 0 0 0) 0x7f0600055eb0 con 0x7f0610153570 2024-04-25T00:10:45.911 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:10:45.911 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5461,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72},{"gid":6114,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:10:23.780120+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6135},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6135":{"gid":6135,"name":"c","rank":0,"incarnation":68,"state":"up:active","state_seq":13,"addr":"172.21.15.116:6835/1358527848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1358527848},{"type":"v1","addr":"172.21.15.116:6835","nonce":1358527848}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6135,"qdb_cluster":[6135]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:09:33.601524+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":132,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4235,"qdb_cluster":[4235]},"id":2}]} 2024-04-25T00:10:45.911 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 72 2024-04-25T00:10:45.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.915+0000 7f06162eb640 1 -- 172.21.15.116:0/3588664260 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f05f00678c0 msgr2=0x7f05f0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:45.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.915+0000 7f06162eb640 1 --2- 172.21.15.116:0/3588664260 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f05f00678c0 0x7f05f0069d80 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f05f80ae000 tx=0x7f05f80a7070 comp rx=0 tx=0).stop 2024-04-25T00:10:45.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.915+0000 7f06162eb640 1 -- 172.21.15.116:0/3588664260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0610153570 msgr2=0x7f06101552b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:45.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.915+0000 7f06162eb640 1 --2- 172.21.15.116:0/3588664260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0610153570 0x7f06101552b0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f060000c3c0 tx=0x7f060000c880 comp rx=0 tx=0).stop 2024-04-25T00:10:45.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.915+0000 7f06162eb640 1 -- 172.21.15.116:0/3588664260 shutdown_connections 2024-04-25T00:10:45.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.915+0000 7f06162eb640 1 --2- 172.21.15.116:0/3588664260 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f05f00678c0 0x7f05f0069d80 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:45.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.915+0000 7f06162eb640 1 --2- 172.21.15.116:0/3588664260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0610153570 0x7f06101552b0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:45.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.915+0000 7f06162eb640 1 --2- 172.21.15.116:0/3588664260 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0610150bf0 0x7f06100aca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:45.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.915+0000 7f06162eb640 1 --2- 172.21.15.116:0/3588664260 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f061006c760 0x7f06101476f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:45.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.915+0000 7f06162eb640 1 -- 172.21.15.116:0/3588664260 >> 172.21.15.116:0/3588664260 conn(0x7f06100c4270 msgr2=0x7f06100a83c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:10:45.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.915+0000 7f06162eb640 1 -- 172.21.15.116:0/3588664260 shutdown_connections 2024-04-25T00:10:45.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:45.915+0000 7f06162eb640 1 -- 172.21.15.116:0/3588664260 wait complete. 2024-04-25T00:10:45.929 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:10:47.931 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:10:48.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.063+0000 7f7921665640 1 Processor -- start 2024-04-25T00:10:48.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.063+0000 7f7921665640 1 -- start start 2024-04-25T00:10:48.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.063+0000 7f7921665640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f791c150d20 0x7f791c0c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:48.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.063+0000 7f7921665640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f791c0a94a0 0x7f791c153ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:48.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.063+0000 7f7921665640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f791c0c3910 con 0x7f791c0c2f50 2024-04-25T00:10:48.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.063+0000 7f7921665640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f791c154530 con 0x7f791c0a94a0 2024-04-25T00:10:48.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.063+0000 7f7921665640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f791c0c16e0 con 0x7f791c150d20 2024-04-25T00:10:48.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.063+0000 7f791b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f791c0a94a0 0x7f791c153ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:48.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.063+0000 7f791b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f791c0a94a0 0x7f791c153ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50782/0 (socket says 172.21.15.116:50782) 2024-04-25T00:10:48.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.063+0000 7f791b7fe640 1 -- 172.21.15.116:0/3196143367 learned_addr learned my addr 172.21.15.116:0/3196143367 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:10:48.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.063+0000 7f791affd640 1 --2- 172.21.15.116:0/3196143367 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f791c150d20 0x7f791c0c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:48.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.063+0000 7f791b7fe640 1 -- 172.21.15.116:0/3196143367 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f791c150d20 msgr2=0x7f791c0c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:10:48.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.063+0000 7f791b7fe640 1 --2- 172.21.15.116:0/3196143367 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f791c150d20 0x7f791c0c2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:48.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.063+0000 7f791b7fe640 1 -- 172.21.15.116:0/3196143367 >> v1:172.21.15.174:6789/0 conn(0x7f791c0c2f50 legacy=0x7f791c1538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:48.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.063+0000 7f791b7fe640 1 -- 172.21.15.116:0/3196143367 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f791c0c1400 con 0x7f791c0a94a0 2024-04-25T00:10:48.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.063+0000 7f791affd640 1 --2- 172.21.15.116:0/3196143367 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f791c150d20 0x7f791c0c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:10:48.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.067+0000 7f791b7fe640 1 --2- 172.21.15.116:0/3196143367 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f791c0a94a0 0x7f791c153ff0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f7914009e50 tx=0x7f791401ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd26fa2d9f4db11b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:48.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.067+0000 7f791a7fc640 1 -- 172.21.15.116:0/3196143367 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79140a5810 con 0x7f791c0a94a0 2024-04-25T00:10:48.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.067+0000 7f791a7fc640 1 -- 172.21.15.116:0/3196143367 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79140a59b0 con 0x7f791c0a94a0 2024-04-25T00:10:48.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.067+0000 7f791a7fc640 1 -- 172.21.15.116:0/3196143367 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79140a5d80 con 0x7f791c0a94a0 2024-04-25T00:10:48.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.067+0000 7f7921665640 1 -- 172.21.15.116:0/3196143367 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f791c0a94a0 msgr2=0x7f791c153ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:48.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.067+0000 7f7921665640 1 --2- 172.21.15.116:0/3196143367 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f791c0a94a0 0x7f791c153ff0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f7914009e50 tx=0x7f791401ca30 comp rx=0 tx=0).stop 2024-04-25T00:10:48.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.067+0000 7f7921665640 1 -- 172.21.15.116:0/3196143367 shutdown_connections 2024-04-25T00:10:48.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.067+0000 7f7921665640 1 --2- 172.21.15.116:0/3196143367 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f791c0a94a0 0x7f791c153ff0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:48.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.067+0000 7f7921665640 1 --2- 172.21.15.116:0/3196143367 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f791c150d20 0x7f791c0c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:48.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.067+0000 7f7921665640 1 -- 172.21.15.116:0/3196143367 >> 172.21.15.116:0/3196143367 conn(0x7f791c0c3b00 msgr2=0x7f791c0c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:10:48.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.067+0000 7f7921665640 1 -- 172.21.15.116:0/3196143367 shutdown_connections 2024-04-25T00:10:48.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.067+0000 7f7921665640 1 -- 172.21.15.116:0/3196143367 wait complete. 2024-04-25T00:10:48.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.067+0000 7f7921665640 1 Processor -- start 2024-04-25T00:10:48.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.067+0000 7f7921665640 1 -- start start 2024-04-25T00:10:48.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.067+0000 7f7921665640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f791c0a94a0 0x7f791c13e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:48.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.067+0000 7f7921665640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f791c0c2f50 0x7f791c14afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:48.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.067+0000 7f7921665640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f791c150d20 0x7f791c1415a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:48.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.067+0000 7f7921665640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f791c1608b0 con 0x7f791c0c2f50 2024-04-25T00:10:48.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.067+0000 7f7921665640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f791c160730 con 0x7f791c0a94a0 2024-04-25T00:10:48.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.067+0000 7f7921665640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f791c160a30 con 0x7f791c150d20 2024-04-25T00:10:48.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.067+0000 7f7913fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f791c0c2f50 0x7f791c14afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:48.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.067+0000 7f7913fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f791c0c2f50 0x7f791c14afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50792/0 (socket says 172.21.15.116:50792) 2024-04-25T00:10:48.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.067+0000 7f7913fff640 1 -- 172.21.15.116:0/53856362 learned_addr learned my addr 172.21.15.116:0/53856362 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:10:48.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.067+0000 7f791affd640 1 --2- 172.21.15.116:0/53856362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f791c0a94a0 0x7f791c13e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:48.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.067+0000 7f791b7fe640 1 --2- 172.21.15.116:0/53856362 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f791c150d20 0x7f791c1415a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:48.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.071+0000 7f7913fff640 1 -- 172.21.15.116:0/53856362 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f791c150d20 msgr2=0x7f791c1415a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:48.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.071+0000 7f7913fff640 1 --2- 172.21.15.116:0/53856362 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f791c150d20 0x7f791c1415a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:48.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.071+0000 7f7913fff640 1 -- 172.21.15.116:0/53856362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f791c0a94a0 msgr2=0x7f791c13e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:48.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.071+0000 7f7913fff640 1 --2- 172.21.15.116:0/53856362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f791c0a94a0 0x7f791c13e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:48.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.071+0000 7f7913fff640 1 -- 172.21.15.116:0/53856362 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f791c14cf90 con 0x7f791c0c2f50 2024-04-25T00:10:48.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.071+0000 7f7913fff640 1 --2- 172.21.15.116:0/53856362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f791c0c2f50 0x7f791c14afd0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f790400c500 tx=0x7f790400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:48.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.071+0000 7f79137fe640 1 -- 172.21.15.116:0/53856362 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7904005220 con 0x7f791c0c2f50 2024-04-25T00:10:48.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.071+0000 7f79137fe640 1 -- 172.21.15.116:0/53856362 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79040053c0 con 0x7f791c0c2f50 2024-04-25T00:10:48.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.071+0000 7f7921665640 1 -- 172.21.15.116:0/53856362 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f791c14d270 con 0x7f791c0c2f50 2024-04-25T00:10:48.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.071+0000 7f79137fe640 1 -- 172.21.15.116:0/53856362 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7904007270 con 0x7f791c0c2f50 2024-04-25T00:10:48.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.071+0000 7f7921665640 1 -- 172.21.15.116:0/53856362 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f791c14d830 con 0x7f791c0c2f50 2024-04-25T00:10:48.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.071+0000 7f7921665640 1 -- 172.21.15.116:0/53856362 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f791c068350 con 0x7f791c0c2f50 2024-04-25T00:10:48.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.071+0000 7f79137fe640 1 -- 172.21.15.116:0/53856362 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7904019020 con 0x7f791c0c2f50 2024-04-25T00:10:48.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.071+0000 7f79137fe640 1 --2- 172.21.15.116:0/53856362 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f78fc0678c0 0x7f78fc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:48.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.071+0000 7f79137fe640 1 -- 172.21.15.116:0/53856362 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7698+0+0 (secure 0 0 0) 0x7f79040857d0 con 0x7f791c0c2f50 2024-04-25T00:10:48.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.071+0000 7f791affd640 1 --2- 172.21.15.116:0/53856362 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f78fc0678c0 0x7f78fc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:48.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.071+0000 7f791affd640 1 --2- 172.21.15.116:0/53856362 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f78fc0678c0 0x7f78fc069d80 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f790c006fd0 tx=0x7f790c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:48.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.075+0000 7f79137fe640 1 -- 172.21.15.116:0/53856362 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f7904055b30 con 0x7f791c0c2f50 2024-04-25T00:10:48.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.267+0000 7f7921665640 1 -- 172.21.15.116:0/53856362 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f791c0c34f0 con 0x7f791c0c2f50 2024-04-25T00:10:48.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.267+0000 7f79137fe640 1 -- 172.21.15.116:0/53856362 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) ==== 94+0+6504 (secure 0 0 0) 0x7f7904055430 con 0x7f791c0c2f50 2024-04-25T00:10:48.267 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:10:48.267 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5461,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72},{"gid":6114,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:10:23.780120+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6135},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6135":{"gid":6135,"name":"c","rank":0,"incarnation":68,"state":"up:active","state_seq":13,"addr":"172.21.15.116:6835/1358527848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1358527848},{"type":"v1","addr":"172.21.15.116:6835","nonce":1358527848}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6135,"qdb_cluster":[6135]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:09:33.601524+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":132,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4235,"qdb_cluster":[4235]},"id":2}]} 2024-04-25T00:10:48.267 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 72 2024-04-25T00:10:48.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.271+0000 7f7921665640 1 -- 172.21.15.116:0/53856362 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f78fc0678c0 msgr2=0x7f78fc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:48.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.271+0000 7f7921665640 1 --2- 172.21.15.116:0/53856362 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f78fc0678c0 0x7f78fc069d80 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f790c006fd0 tx=0x7f790c008040 comp rx=0 tx=0).stop 2024-04-25T00:10:48.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.271+0000 7f7921665640 1 -- 172.21.15.116:0/53856362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f791c0c2f50 msgr2=0x7f791c14afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:48.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.271+0000 7f7921665640 1 --2- 172.21.15.116:0/53856362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f791c0c2f50 0x7f791c14afd0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f790400c500 tx=0x7f790400c9c0 comp rx=0 tx=0).stop 2024-04-25T00:10:48.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.271+0000 7f7921665640 1 -- 172.21.15.116:0/53856362 shutdown_connections 2024-04-25T00:10:48.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.271+0000 7f7921665640 1 --2- 172.21.15.116:0/53856362 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f78fc0678c0 0x7f78fc069d80 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:48.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.271+0000 7f7921665640 1 --2- 172.21.15.116:0/53856362 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f791c150d20 0x7f791c1415a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:48.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.271+0000 7f7921665640 1 --2- 172.21.15.116:0/53856362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f791c0c2f50 0x7f791c14afd0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:48.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.271+0000 7f7921665640 1 --2- 172.21.15.116:0/53856362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f791c0a94a0 0x7f791c13e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:48.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.271+0000 7f7921665640 1 -- 172.21.15.116:0/53856362 >> 172.21.15.116:0/53856362 conn(0x7f791c0c3b00 msgr2=0x7f791c14fae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:10:48.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.271+0000 7f7921665640 1 -- 172.21.15.116:0/53856362 shutdown_connections 2024-04-25T00:10:48.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:48.271+0000 7f7921665640 1 -- 172.21.15.116:0/53856362 wait complete. 2024-04-25T00:10:48.285 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:10:50.286 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:10:50.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.419+0000 7ff8b9bb7640 1 Processor -- start 2024-04-25T00:10:50.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.419+0000 7ff8b9bb7640 1 -- start start 2024-04-25T00:10:50.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.423+0000 7ff8b9bb7640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff8b4150d20 0x7ff8b40c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:50.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.423+0000 7ff8b9bb7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff8b40a94a0 0x7ff8b4153ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:50.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.423+0000 7ff8b9bb7640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8b40c3910 con 0x7ff8b40c2f50 2024-04-25T00:10:50.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.423+0000 7ff8b9bb7640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7ff8b4154530 con 0x7ff8b40a94a0 2024-04-25T00:10:50.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.423+0000 7ff8b9bb7640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff8b40c16e0 con 0x7ff8b4150d20 2024-04-25T00:10:50.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.423+0000 7ff8b3fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff8b40a94a0 0x7ff8b4153ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:50.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.423+0000 7ff8b3fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff8b40a94a0 0x7ff8b4153ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50798/0 (socket says 172.21.15.116:50798) 2024-04-25T00:10:50.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.423+0000 7ff8b3fff640 1 -- 172.21.15.116:0/3758731972 learned_addr learned my addr 172.21.15.116:0/3758731972 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:10:50.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.423+0000 7ff8b37fe640 1 --2- 172.21.15.116:0/3758731972 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff8b4150d20 0x7ff8b40c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:50.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.423+0000 7ff8b3fff640 1 -- 172.21.15.116:0/3758731972 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff8b4150d20 msgr2=0x7ff8b40c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:10:50.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.423+0000 7ff8b3fff640 1 --2- 172.21.15.116:0/3758731972 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff8b4150d20 0x7ff8b40c2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:50.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.423+0000 7ff8b3fff640 1 -- 172.21.15.116:0/3758731972 >> v1:172.21.15.174:6789/0 conn(0x7ff8b40c2f50 legacy=0x7ff8b41538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:10:50.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.423+0000 7ff8b3fff640 1 -- 172.21.15.116:0/3758731972 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8b40c1400 con 0x7ff8b40a94a0 2024-04-25T00:10:50.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.423+0000 7ff8b3fff640 1 --2- 172.21.15.116:0/3758731972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8b40a94a0 0x7ff8b4153ff0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac009e50 tx=0x7ff8ac01c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58e16b970b38c06a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:50.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.423+0000 7ff8b27fc640 1 -- 172.21.15.116:0/3758731972 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac0a5780 con 0x7ff8b40a94a0 2024-04-25T00:10:50.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.423+0000 7ff8b27fc640 1 -- 172.21.15.116:0/3758731972 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8ac0a5920 con 0x7ff8b40a94a0 2024-04-25T00:10:50.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.423+0000 7ff8b27fc640 1 -- 172.21.15.116:0/3758731972 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac0a5cf0 con 0x7ff8b40a94a0 2024-04-25T00:10:50.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.427+0000 7ff8b9bb7640 1 -- 172.21.15.116:0/3758731972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8b40a94a0 msgr2=0x7ff8b4153ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:50.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.427+0000 7ff8b9bb7640 1 --2- 172.21.15.116:0/3758731972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8b40a94a0 0x7ff8b4153ff0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac009e50 tx=0x7ff8ac01c9f0 comp rx=0 tx=0).stop 2024-04-25T00:10:50.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.427+0000 7ff8b9bb7640 1 -- 172.21.15.116:0/3758731972 shutdown_connections 2024-04-25T00:10:50.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.427+0000 7ff8b9bb7640 1 --2- 172.21.15.116:0/3758731972 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8b40a94a0 0x7ff8b4153ff0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:50.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.427+0000 7ff8b9bb7640 1 --2- 172.21.15.116:0/3758731972 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff8b4150d20 0x7ff8b40c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:50.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.427+0000 7ff8b9bb7640 1 -- 172.21.15.116:0/3758731972 >> 172.21.15.116:0/3758731972 conn(0x7ff8b40c3b00 msgr2=0x7ff8b40c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:10:50.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.427+0000 7ff8b9bb7640 1 -- 172.21.15.116:0/3758731972 shutdown_connections 2024-04-25T00:10:50.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.427+0000 7ff8b9bb7640 1 -- 172.21.15.116:0/3758731972 wait complete. 2024-04-25T00:10:50.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.427+0000 7ff8b9bb7640 1 Processor -- start 2024-04-25T00:10:50.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.427+0000 7ff8b9bb7640 1 -- start start 2024-04-25T00:10:50.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.427+0000 7ff8b9bb7640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff8b40a94a0 0x7ff8b4142a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:50.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.427+0000 7ff8b9bb7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff8b40c2f50 0x7ff8b4156e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:50.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.427+0000 7ff8b9bb7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8b4150d20 0x7ff8b4157380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:50.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.427+0000 7ff8b9bb7640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff8b41608b0 con 0x7ff8b4150d20 2024-04-25T00:10:50.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.427+0000 7ff8b9bb7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff8b4160730 con 0x7ff8b40c2f50 2024-04-25T00:10:50.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.427+0000 7ff8b9bb7640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff8b4160a30 con 0x7ff8b40a94a0 2024-04-25T00:10:50.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.427+0000 7ff8b3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8b4150d20 0x7ff8b4157380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:50.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.427+0000 7ff8b3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8b4150d20 0x7ff8b4157380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50802/0 (socket says 172.21.15.116:50802) 2024-04-25T00:10:50.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.427+0000 7ff8b3fff640 1 -- 172.21.15.116:0/3803265289 learned_addr learned my addr 172.21.15.116:0/3803265289 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:10:50.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.427+0000 7ff8b37fe640 1 --2- 172.21.15.116:0/3803265289 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff8b40a94a0 0x7ff8b4142a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:50.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.427+0000 7ff8b2ffd640 1 --2- 172.21.15.116:0/3803265289 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff8b40c2f50 0x7ff8b4156e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:50.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.427+0000 7ff8b3fff640 1 -- 172.21.15.116:0/3803265289 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff8b40a94a0 msgr2=0x7ff8b4142a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:50.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.427+0000 7ff8b3fff640 1 --2- 172.21.15.116:0/3803265289 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff8b40a94a0 0x7ff8b4142a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:50.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.427+0000 7ff8b3fff640 1 -- 172.21.15.116:0/3803265289 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff8b40c2f50 msgr2=0x7ff8b4156e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:50.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.427+0000 7ff8b3fff640 1 --2- 172.21.15.116:0/3803265289 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff8b40c2f50 0x7ff8b4156e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:50.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.427+0000 7ff8b3fff640 1 -- 172.21.15.116:0/3803265289 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8b40abfb0 con 0x7ff8b4150d20 2024-04-25T00:10:50.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.427+0000 7ff8b3fff640 1 --2- 172.21.15.116:0/3803265289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8b4150d20 0x7ff8b4157380 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac0af040 tx=0x7ff8ac019790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:50.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.427+0000 7ff8abfff640 1 -- 172.21.15.116:0/3803265289 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac0bc020 con 0x7ff8b4150d20 2024-04-25T00:10:50.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.427+0000 7ff8abfff640 1 -- 172.21.15.116:0/3803265289 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8ac0b8690 con 0x7ff8b4150d20 2024-04-25T00:10:50.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.427+0000 7ff8b9bb7640 1 -- 172.21.15.116:0/3803265289 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8b41578c0 con 0x7ff8b4150d20 2024-04-25T00:10:50.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.427+0000 7ff8abfff640 1 -- 172.21.15.116:0/3803265289 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac0b8a80 con 0x7ff8b4150d20 2024-04-25T00:10:50.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.427+0000 7ff8b9bb7640 1 -- 172.21.15.116:0/3803265289 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8b414ffd0 con 0x7ff8b4150d20 2024-04-25T00:10:50.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.431+0000 7ff8b9bb7640 1 -- 172.21.15.116:0/3803265289 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8b4068350 con 0x7ff8b4150d20 2024-04-25T00:10:50.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.431+0000 7ff8abfff640 1 -- 172.21.15.116:0/3803265289 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff8ac0bf020 con 0x7ff8b4150d20 2024-04-25T00:10:50.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.431+0000 7ff8abfff640 1 --2- 172.21.15.116:0/3803265289 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff8940678c0 0x7ff894069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:50.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.431+0000 7ff8abfff640 1 -- 172.21.15.116:0/3803265289 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7698+0+0 (secure 0 0 0) 0x7ff8ac1211a0 con 0x7ff8b4150d20 2024-04-25T00:10:50.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.431+0000 7ff8b37fe640 1 --2- 172.21.15.116:0/3803265289 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff8940678c0 0x7ff894069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:50.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.431+0000 7ff8b37fe640 1 --2- 172.21.15.116:0/3803265289 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff8940678c0 0x7ff894069d80 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7ff8a400b9d0 tx=0x7ff8a4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:50.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.435+0000 7ff8abfff640 1 -- 172.21.15.116:0/3803265289 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ff8ac0165e0 con 0x7ff8b4150d20 2024-04-25T00:10:50.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.631+0000 7ff8b9bb7640 1 -- 172.21.15.116:0/3803265289 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff8b4150290 con 0x7ff8b4150d20 2024-04-25T00:10:50.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.631+0000 7ff8abfff640 1 -- 172.21.15.116:0/3803265289 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) ==== 94+0+6504 (secure 0 0 0) 0x7ff8ac0f15a0 con 0x7ff8b4150d20 2024-04-25T00:10:50.629 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:10:50.629 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5461,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72},{"gid":6114,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:10:23.780120+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6135},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6135":{"gid":6135,"name":"c","rank":0,"incarnation":68,"state":"up:active","state_seq":13,"addr":"172.21.15.116:6835/1358527848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1358527848},{"type":"v1","addr":"172.21.15.116:6835","nonce":1358527848}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6135,"qdb_cluster":[6135]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:09:33.601524+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":132,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4235,"qdb_cluster":[4235]},"id":2}]} 2024-04-25T00:10:50.629 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 72 2024-04-25T00:10:50.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.631+0000 7ff8b9bb7640 1 -- 172.21.15.116:0/3803265289 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff8940678c0 msgr2=0x7ff894069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:50.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.631+0000 7ff8b9bb7640 1 --2- 172.21.15.116:0/3803265289 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff8940678c0 0x7ff894069d80 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7ff8a400b9d0 tx=0x7ff8a4008040 comp rx=0 tx=0).stop 2024-04-25T00:10:50.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.631+0000 7ff8b9bb7640 1 -- 172.21.15.116:0/3803265289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8b4150d20 msgr2=0x7ff8b4157380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:50.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.631+0000 7ff8b9bb7640 1 --2- 172.21.15.116:0/3803265289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8b4150d20 0x7ff8b4157380 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac0af040 tx=0x7ff8ac019790 comp rx=0 tx=0).stop 2024-04-25T00:10:50.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.631+0000 7ff8b9bb7640 1 -- 172.21.15.116:0/3803265289 shutdown_connections 2024-04-25T00:10:50.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.631+0000 7ff8b9bb7640 1 --2- 172.21.15.116:0/3803265289 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff8940678c0 0x7ff894069d80 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:50.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.631+0000 7ff8b9bb7640 1 --2- 172.21.15.116:0/3803265289 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8b4150d20 0x7ff8b4157380 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:50.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.631+0000 7ff8b9bb7640 1 --2- 172.21.15.116:0/3803265289 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff8b40c2f50 0x7ff8b4156e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:50.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.631+0000 7ff8b9bb7640 1 --2- 172.21.15.116:0/3803265289 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff8b40a94a0 0x7ff8b4142a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:50.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.631+0000 7ff8b9bb7640 1 -- 172.21.15.116:0/3803265289 >> 172.21.15.116:0/3803265289 conn(0x7ff8b40c3b00 msgr2=0x7ff8b40a9f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:10:50.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.631+0000 7ff8b9bb7640 1 -- 172.21.15.116:0/3803265289 shutdown_connections 2024-04-25T00:10:50.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:50.631+0000 7ff8b9bb7640 1 -- 172.21.15.116:0/3803265289 wait complete. 2024-04-25T00:10:50.648 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:10:52.649 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:10:52.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.787+0000 7fccfe3ee640 1 Processor -- start 2024-04-25T00:10:52.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.787+0000 7fccfe3ee640 1 -- start start 2024-04-25T00:10:52.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.791+0000 7fccfe3ee640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fccf80a94a0 0x7fccf81538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:52.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.791+0000 7fccfe3ee640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fccf80a99f0 0x7fccf8153e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:52.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.791+0000 7fccfe3ee640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fccf80c3910 con 0x7fccf8150d20 2024-04-25T00:10:52.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.791+0000 7fccfe3ee640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fccf80c16e0 con 0x7fccf80a94a0 2024-04-25T00:10:52.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.791+0000 7fccfe3ee640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fccf80c1400 con 0x7fccf80a99f0 2024-04-25T00:10:52.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.791+0000 7fccf77fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fccf80a94a0 0x7fccf81538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:52.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.791+0000 7fccf77fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fccf80a94a0 0x7fccf81538e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50808/0 (socket says 172.21.15.116:50808) 2024-04-25T00:10:52.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.791+0000 7fccf77fe640 1 -- 172.21.15.116:0/3760863271 learned_addr learned my addr 172.21.15.116:0/3760863271 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:10:52.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.791+0000 7fccfc98b640 1 --2- 172.21.15.116:0/3760863271 >> v2:172.21.15.174:3300/0 conn(0x7fccf80a99f0 0x7fccf8153e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:52.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.791+0000 7fccf77fe640 1 -- 172.21.15.116:0/3760863271 >> v2:172.21.15.174:3300/0 conn(0x7fccf80a99f0 msgr2=0x7fccf8153e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:10:52.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.791+0000 7fccf77fe640 1 --2- 172.21.15.116:0/3760863271 >> v2:172.21.15.174:3300/0 conn(0x7fccf80a99f0 0x7fccf8153e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:52.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.791+0000 7fccf77fe640 1 -- 172.21.15.116:0/3760863271 >> v1:172.21.15.174:6789/0 conn(0x7fccf8150d20 legacy=0x7fccf80c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:10:52.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.791+0000 7fccf77fe640 1 -- 172.21.15.116:0/3760863271 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccf806e660 con 0x7fccf80a94a0 2024-04-25T00:10:52.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.791+0000 7fccf77fe640 1 --2- 172.21.15.116:0/3760863271 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccf80a94a0 0x7fccf81538e0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fccf8157b40 tx=0x7fccec01c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdb7b0bdf68db46d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:52.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.791+0000 7fccf6ffd640 1 -- 172.21.15.116:0/3760863271 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccec0ab5f0 con 0x7fccf80a94a0 2024-04-25T00:10:52.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.791+0000 7fccf6ffd640 1 -- 172.21.15.116:0/3760863271 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccec0b0070 con 0x7fccf80a94a0 2024-04-25T00:10:52.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.791+0000 7fccf6ffd640 1 -- 172.21.15.116:0/3760863271 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccec0ab9b0 con 0x7fccf80a94a0 2024-04-25T00:10:52.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.795+0000 7fccfe3ee640 1 -- 172.21.15.116:0/3760863271 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccf80a94a0 msgr2=0x7fccf81538e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:52.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.795+0000 7fccfe3ee640 1 --2- 172.21.15.116:0/3760863271 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccf80a94a0 0x7fccf81538e0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fccf8157b40 tx=0x7fccec01c4e0 comp rx=0 tx=0).stop 2024-04-25T00:10:52.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.795+0000 7fccfe3ee640 1 -- 172.21.15.116:0/3760863271 shutdown_connections 2024-04-25T00:10:52.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.795+0000 7fccfe3ee640 1 --2- 172.21.15.116:0/3760863271 >> v2:172.21.15.174:3300/0 conn(0x7fccf80a99f0 0x7fccf8153e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:52.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.795+0000 7fccfe3ee640 1 --2- 172.21.15.116:0/3760863271 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccf80a94a0 0x7fccf81538e0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:52.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.795+0000 7fccfe3ee640 1 -- 172.21.15.116:0/3760863271 >> 172.21.15.116:0/3760863271 conn(0x7fccf80c3b00 msgr2=0x7fccf80c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:10:52.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.795+0000 7fccfe3ee640 1 -- 172.21.15.116:0/3760863271 shutdown_connections 2024-04-25T00:10:52.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.795+0000 7fccfe3ee640 1 -- 172.21.15.116:0/3760863271 wait complete. 2024-04-25T00:10:52.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.795+0000 7fccfe3ee640 1 Processor -- start 2024-04-25T00:10:52.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.795+0000 7fccfe3ee640 1 -- start start 2024-04-25T00:10:52.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.795+0000 7fccfe3ee640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccf80a94a0 0x7fccf8145df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:52.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.795+0000 7fccfe3ee640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fccf80a99f0 0x7fccf8140760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:52.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.795+0000 7fccfe3ee640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fccf8150d20 0x7fccf8140ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:52.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.795+0000 7fccfe3ee640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fccf81544e0 con 0x7fccf80a94a0 2024-04-25T00:10:52.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.795+0000 7fccfe3ee640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fccf8154360 con 0x7fccf80a99f0 2024-04-25T00:10:52.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.795+0000 7fccfe3ee640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fccf8160730 con 0x7fccf8150d20 2024-04-25T00:10:52.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.795+0000 7fccf7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccf80a94a0 0x7fccf8145df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:52.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.795+0000 7fccf7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccf80a94a0 0x7fccf8145df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50812/0 (socket says 172.21.15.116:50812) 2024-04-25T00:10:52.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.795+0000 7fccf7fff640 1 -- 172.21.15.116:0/1177159695 learned_addr learned my addr 172.21.15.116:0/1177159695 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:10:52.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.795+0000 7fccf77fe640 1 --2- 172.21.15.116:0/1177159695 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fccf80a99f0 0x7fccf8140760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:52.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.795+0000 7fccfc98b640 1 --2- 172.21.15.116:0/1177159695 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fccf8150d20 0x7fccf8140ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:52.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.795+0000 7fccf7fff640 1 -- 172.21.15.116:0/1177159695 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fccf8150d20 msgr2=0x7fccf8140ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:52.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.795+0000 7fccf7fff640 1 --2- 172.21.15.116:0/1177159695 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fccf8150d20 0x7fccf8140ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:52.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.795+0000 7fccf7fff640 1 -- 172.21.15.116:0/1177159695 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fccf80a99f0 msgr2=0x7fccf8140760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:52.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.795+0000 7fccf7fff640 1 --2- 172.21.15.116:0/1177159695 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fccf80a99f0 0x7fccf8140760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:52.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.795+0000 7fccf7fff640 1 -- 172.21.15.116:0/1177159695 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccf81eec00 con 0x7fccf80a94a0 2024-04-25T00:10:52.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.795+0000 7fccf77fe640 1 --2- 172.21.15.116:0/1177159695 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fccf80a99f0 0x7fccf8140760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:10:52.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.795+0000 7fccf7fff640 1 --2- 172.21.15.116:0/1177159695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccf80a94a0 0x7fccf8145df0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fcce800c450 tx=0x7fcce800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:52.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.795+0000 7fccf4ff9640 1 -- 172.21.15.116:0/1177159695 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcce8012830 con 0x7fccf80a94a0 2024-04-25T00:10:52.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.795+0000 7fccf4ff9640 1 -- 172.21.15.116:0/1177159695 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcce8017070 con 0x7fccf80a94a0 2024-04-25T00:10:52.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.795+0000 7fccf4ff9640 1 -- 172.21.15.116:0/1177159695 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcce8012c30 con 0x7fccf80a94a0 2024-04-25T00:10:52.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.795+0000 7fccfe3ee640 1 -- 172.21.15.116:0/1177159695 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccf81eeee0 con 0x7fccf80a94a0 2024-04-25T00:10:52.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.795+0000 7fccfe3ee640 1 -- 172.21.15.116:0/1177159695 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccf806d770 con 0x7fccf80a94a0 2024-04-25T00:10:52.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.799+0000 7fccfe3ee640 1 -- 172.21.15.116:0/1177159695 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccf8068350 con 0x7fccf80a94a0 2024-04-25T00:10:52.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.799+0000 7fccf4ff9640 1 -- 172.21.15.116:0/1177159695 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fcce801c460 con 0x7fccf80a94a0 2024-04-25T00:10:52.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.799+0000 7fccf4ff9640 1 --2- 172.21.15.116:0/1177159695 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcce40678c0 0x7fcce4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:52.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.799+0000 7fccf4ff9640 1 -- 172.21.15.116:0/1177159695 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7698+0+0 (secure 0 0 0) 0x7fcce8086200 con 0x7fccf80a94a0 2024-04-25T00:10:52.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.799+0000 7fccf77fe640 1 --2- 172.21.15.116:0/1177159695 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcce40678c0 0x7fcce4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:52.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.799+0000 7fccf77fe640 1 --2- 172.21.15.116:0/1177159695 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcce40678c0 0x7fcce4069d80 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fccec0af000 tx=0x7fccec0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:52.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:52.803+0000 7fccf4ff9640 1 -- 172.21.15.116:0/1177159695 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fcce8056600 con 0x7fccf80a94a0 2024-04-25T00:10:53.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:53.003+0000 7fccfe3ee640 1 -- 172.21.15.116:0/1177159695 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fccf80c34f0 con 0x7fccf80a94a0 2024-04-25T00:10:53.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:53.003+0000 7fccf4ff9640 1 -- 172.21.15.116:0/1177159695 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) ==== 94+0+6504 (secure 0 0 0) 0x7fcce8055f00 con 0x7fccf80a94a0 2024-04-25T00:10:53.003 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:10:53.003 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5461,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72},{"gid":6114,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:10:23.780120+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6135},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6135":{"gid":6135,"name":"c","rank":0,"incarnation":68,"state":"up:active","state_seq":13,"addr":"172.21.15.116:6835/1358527848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1358527848},{"type":"v1","addr":"172.21.15.116:6835","nonce":1358527848}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6135,"qdb_cluster":[6135]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:09:33.601524+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":132,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4235,"qdb_cluster":[4235]},"id":2}]} 2024-04-25T00:10:53.003 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 72 2024-04-25T00:10:53.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:53.007+0000 7fccfe3ee640 1 -- 172.21.15.116:0/1177159695 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcce40678c0 msgr2=0x7fcce4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:53.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:53.007+0000 7fccfe3ee640 1 --2- 172.21.15.116:0/1177159695 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcce40678c0 0x7fcce4069d80 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fccec0af000 tx=0x7fccec0a7000 comp rx=0 tx=0).stop 2024-04-25T00:10:53.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:53.007+0000 7fccfe3ee640 1 -- 172.21.15.116:0/1177159695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccf80a94a0 msgr2=0x7fccf8145df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:53.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:53.007+0000 7fccfe3ee640 1 --2- 172.21.15.116:0/1177159695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccf80a94a0 0x7fccf8145df0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fcce800c450 tx=0x7fcce800c910 comp rx=0 tx=0).stop 2024-04-25T00:10:53.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:53.007+0000 7fccfe3ee640 1 -- 172.21.15.116:0/1177159695 shutdown_connections 2024-04-25T00:10:53.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:53.007+0000 7fccfe3ee640 1 --2- 172.21.15.116:0/1177159695 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcce40678c0 0x7fcce4069d80 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:53.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:53.007+0000 7fccfe3ee640 1 --2- 172.21.15.116:0/1177159695 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fccf8150d20 0x7fccf8140ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:53.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:53.007+0000 7fccfe3ee640 1 --2- 172.21.15.116:0/1177159695 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fccf80a99f0 0x7fccf8140760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:53.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:53.007+0000 7fccfe3ee640 1 --2- 172.21.15.116:0/1177159695 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fccf80a94a0 0x7fccf8145df0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:53.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:53.007+0000 7fccfe3ee640 1 -- 172.21.15.116:0/1177159695 >> 172.21.15.116:0/1177159695 conn(0x7fccf80c3b00 msgr2=0x7fccf81537e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:10:53.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:53.007+0000 7fccfe3ee640 1 -- 172.21.15.116:0/1177159695 shutdown_connections 2024-04-25T00:10:53.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:53.007+0000 7fccfe3ee640 1 -- 172.21.15.116:0/1177159695 wait complete. 2024-04-25T00:10:53.021 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:10:55.022 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:10:55.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.155+0000 7f8bf536f640 1 Processor -- start 2024-04-25T00:10:55.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.155+0000 7f8bf536f640 1 -- start start 2024-04-25T00:10:55.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.159+0000 7f8bf536f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8bf014c720 0x7f8bf01563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:55.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.159+0000 7f8bf536f640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f8bf00a94a0 0x7f8bf00c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:55.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.159+0000 7f8bf536f640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bf0071220 con 0x7f8bf00c2f50 2024-04-25T00:10:55.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.159+0000 7f8bf536f640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f8bf0153460 con 0x7f8bf00a94a0 2024-04-25T00:10:55.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.159+0000 7f8bf536f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f8bf0152420 con 0x7f8bf014c720 2024-04-25T00:10:55.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.159+0000 7f8bee7fc640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f8bf00a94a0 0x7f8bf00c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:55.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.159+0000 7f8beeffd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8bf014c720 0x7f8bf01563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:55.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.159+0000 7f8bef7fe640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f8bf00c2f50 0x7f8bf0071680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:45888/0 (socket says 172.21.15.116:45888) 2024-04-25T00:10:55.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.159+0000 7f8bef7fe640 1 -- 172.21.15.116:0/2157360741 learned_addr learned my addr 172.21.15.116:0/2157360741 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:10:55.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.159+0000 7f8bee7fc640 1 -- 172.21.15.116:0/2157360741 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8bf014c720 msgr2=0x7f8bf01563d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:10:55.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.159+0000 7f8bee7fc640 1 --2- 172.21.15.116:0/2157360741 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8bf014c720 0x7f8bf01563d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:55.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.159+0000 7f8bee7fc640 1 -- 172.21.15.116:0/2157360741 >> v1:172.21.15.174:6789/0 conn(0x7f8bf00c2f50 legacy=0x7f8bf0071680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:55.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.159+0000 7f8bee7fc640 1 -- 172.21.15.116:0/2157360741 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bf014cb00 con 0x7f8bf00a94a0 2024-04-25T00:10:55.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.163+0000 7f8bee7fc640 1 --2- 172.21.15.116:0/2157360741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8bf00a94a0 0x7f8bf00c2a10 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f8be40097c0 tx=0x7f8be401c3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb0545a6a22e18d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:55.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.163+0000 7f8bedffb640 1 -- 172.21.15.116:0/2157360741 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be40ab600 con 0x7f8bf00a94a0 2024-04-25T00:10:55.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.163+0000 7f8bedffb640 1 -- 172.21.15.116:0/2157360741 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8be40b0070 con 0x7f8bf00a94a0 2024-04-25T00:10:55.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.163+0000 7f8bedffb640 1 -- 172.21.15.116:0/2157360741 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be40ab9c0 con 0x7f8bf00a94a0 2024-04-25T00:10:55.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.163+0000 7f8bf536f640 1 -- 172.21.15.116:0/2157360741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8bf00a94a0 msgr2=0x7f8bf00c2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:55.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.163+0000 7f8bf536f640 1 --2- 172.21.15.116:0/2157360741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8bf00a94a0 0x7f8bf00c2a10 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f8be40097c0 tx=0x7f8be401c3e0 comp rx=0 tx=0).stop 2024-04-25T00:10:55.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.163+0000 7f8bf536f640 1 -- 172.21.15.116:0/2157360741 shutdown_connections 2024-04-25T00:10:55.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.163+0000 7f8bf536f640 1 --2- 172.21.15.116:0/2157360741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8bf00a94a0 0x7f8bf00c2a10 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:55.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.163+0000 7f8bf536f640 1 --2- 172.21.15.116:0/2157360741 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8bf014c720 0x7f8bf01563d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:55.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.163+0000 7f8bf536f640 1 -- 172.21.15.116:0/2157360741 >> 172.21.15.116:0/2157360741 conn(0x7f8bf00c3b00 msgr2=0x7f8bf00c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:10:55.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.163+0000 7f8bf536f640 1 -- 172.21.15.116:0/2157360741 shutdown_connections 2024-04-25T00:10:55.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.163+0000 7f8bf536f640 1 -- 172.21.15.116:0/2157360741 wait complete. 2024-04-25T00:10:55.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.163+0000 7f8bf536f640 1 Processor -- start 2024-04-25T00:10:55.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.163+0000 7f8bf536f640 1 -- start start 2024-04-25T00:10:55.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.163+0000 7f8bf536f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8bf00a94a0 0x7f8bf0140840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:55.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.163+0000 7f8bf536f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bf00c2f50 0x7f8bf00c6060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:55.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.163+0000 7f8bf536f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8bf014c720 0x7f8bf01469d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:55.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.163+0000 7f8bf536f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8bf0152f40 con 0x7f8bf00c2f50 2024-04-25T00:10:55.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.163+0000 7f8bf536f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8bf0152c60 con 0x7f8bf014c720 2024-04-25T00:10:55.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.163+0000 7f8bf536f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f8bf015f5c0 con 0x7f8bf00a94a0 2024-04-25T00:10:55.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.163+0000 7f8bee7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bf00c2f50 0x7f8bf00c6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:55.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.163+0000 7f8bee7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bf00c2f50 0x7f8bf00c6060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40688/0 (socket says 172.21.15.116:40688) 2024-04-25T00:10:55.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.163+0000 7f8bee7fc640 1 -- 172.21.15.116:0/2610794919 learned_addr learned my addr 172.21.15.116:0/2610794919 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:10:55.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.163+0000 7f8beeffd640 1 --2- 172.21.15.116:0/2610794919 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8bf00a94a0 0x7f8bf0140840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:55.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.163+0000 7f8bef7fe640 1 --2- 172.21.15.116:0/2610794919 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8bf014c720 0x7f8bf01469d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:55.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.163+0000 7f8bee7fc640 1 -- 172.21.15.116:0/2610794919 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8bf00a94a0 msgr2=0x7f8bf0140840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:55.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.163+0000 7f8bee7fc640 1 --2- 172.21.15.116:0/2610794919 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8bf00a94a0 0x7f8bf0140840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:55.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.163+0000 7f8bee7fc640 1 -- 172.21.15.116:0/2610794919 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8bf014c720 msgr2=0x7f8bf01469d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:55.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.163+0000 7f8bee7fc640 1 --2- 172.21.15.116:0/2610794919 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8bf014c720 0x7f8bf01469d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:55.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.163+0000 7f8bee7fc640 1 -- 172.21.15.116:0/2610794919 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bf00c52e0 con 0x7f8bf00c2f50 2024-04-25T00:10:55.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.163+0000 7f8bee7fc640 1 --2- 172.21.15.116:0/2610794919 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bf00c2f50 0x7f8bf00c6060 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f8be401cc80 tx=0x7f8be401ac20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:55.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.167+0000 7f8bd77fe640 1 -- 172.21.15.116:0/2610794919 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be4006e20 con 0x7f8bf00c2f50 2024-04-25T00:10:55.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.167+0000 7f8bd77fe640 1 -- 172.21.15.116:0/2610794919 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8be40bf610 con 0x7f8bf00c2f50 2024-04-25T00:10:55.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.167+0000 7f8bf536f640 1 -- 172.21.15.116:0/2610794919 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bf00c65a0 con 0x7f8bf00c2f50 2024-04-25T00:10:55.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.167+0000 7f8bd77fe640 1 -- 172.21.15.116:0/2610794919 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be40bf9e0 con 0x7f8bf00c2f50 2024-04-25T00:10:55.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.167+0000 7f8bf536f640 1 -- 172.21.15.116:0/2610794919 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bf01f06c0 con 0x7f8bf00c2f50 2024-04-25T00:10:55.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.167+0000 7f8bf536f640 1 -- 172.21.15.116:0/2610794919 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bf0068350 con 0x7f8bf00c2f50 2024-04-25T00:10:55.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.167+0000 7f8bd77fe640 1 -- 172.21.15.116:0/2610794919 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8be40bd020 con 0x7f8bf00c2f50 2024-04-25T00:10:55.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.167+0000 7f8bd77fe640 1 --2- 172.21.15.116:0/2610794919 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8bdc0678f0 0x7f8bdc069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:55.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.167+0000 7f8bd77fe640 1 -- 172.21.15.116:0/2610794919 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7698+0+0 (secure 0 0 0) 0x7f8be4120b20 con 0x7f8bf00c2f50 2024-04-25T00:10:55.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.167+0000 7f8beeffd640 1 --2- 172.21.15.116:0/2610794919 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8bdc0678f0 0x7f8bdc069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:55.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.167+0000 7f8beeffd640 1 --2- 172.21.15.116:0/2610794919 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8bdc0678f0 0x7f8bdc069db0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f8bf00c5270 tx=0x7f8be0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:55.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.171+0000 7f8bd77fe640 1 -- 172.21.15.116:0/2610794919 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f8be4016090 con 0x7f8bf00c2f50 2024-04-25T00:10:55.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.359+0000 7f8bf536f640 1 -- 172.21.15.116:0/2610794919 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8bf00c1f50 con 0x7f8bf00c2f50 2024-04-25T00:10:55.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.359+0000 7f8bd77fe640 1 -- 172.21.15.116:0/2610794919 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) ==== 94+0+6504 (secure 0 0 0) 0x7f8be40f0fc0 con 0x7f8bf00c2f50 2024-04-25T00:10:55.359 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:10:55.359 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5461,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72},{"gid":6114,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:10:23.780120+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6135},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6135":{"gid":6135,"name":"c","rank":0,"incarnation":68,"state":"up:active","state_seq":13,"addr":"172.21.15.116:6835/1358527848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1358527848},{"type":"v1","addr":"172.21.15.116:6835","nonce":1358527848}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6135,"qdb_cluster":[6135]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:09:33.601524+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":132,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4235,"qdb_cluster":[4235]},"id":2}]} 2024-04-25T00:10:55.359 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 72 2024-04-25T00:10:55.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.363+0000 7f8bf536f640 1 -- 172.21.15.116:0/2610794919 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8bdc0678f0 msgr2=0x7f8bdc069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:55.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.363+0000 7f8bf536f640 1 --2- 172.21.15.116:0/2610794919 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8bdc0678f0 0x7f8bdc069db0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f8bf00c5270 tx=0x7f8be0008040 comp rx=0 tx=0).stop 2024-04-25T00:10:55.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.363+0000 7f8bf536f640 1 -- 172.21.15.116:0/2610794919 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bf00c2f50 msgr2=0x7f8bf00c6060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:55.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.363+0000 7f8bf536f640 1 --2- 172.21.15.116:0/2610794919 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bf00c2f50 0x7f8bf00c6060 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f8be401cc80 tx=0x7f8be401ac20 comp rx=0 tx=0).stop 2024-04-25T00:10:55.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.363+0000 7f8bf536f640 1 -- 172.21.15.116:0/2610794919 shutdown_connections 2024-04-25T00:10:55.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.363+0000 7f8bf536f640 1 --2- 172.21.15.116:0/2610794919 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8bdc0678f0 0x7f8bdc069db0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:55.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.363+0000 7f8bf536f640 1 --2- 172.21.15.116:0/2610794919 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8bf014c720 0x7f8bf01469d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:55.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.363+0000 7f8bf536f640 1 --2- 172.21.15.116:0/2610794919 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8bf00c2f50 0x7f8bf00c6060 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:55.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.363+0000 7f8bf536f640 1 --2- 172.21.15.116:0/2610794919 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8bf00a94a0 0x7f8bf0140840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:55.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.363+0000 7f8bf536f640 1 -- 172.21.15.116:0/2610794919 >> 172.21.15.116:0/2610794919 conn(0x7f8bf00c3b00 msgr2=0x7f8bf00c2170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:10:55.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.363+0000 7f8bf536f640 1 -- 172.21.15.116:0/2610794919 shutdown_connections 2024-04-25T00:10:55.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:55.363+0000 7f8bf536f640 1 -- 172.21.15.116:0/2610794919 wait complete. 2024-04-25T00:10:55.376 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:10:57.376 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:10:57.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.511+0000 7f4103df6640 1 Processor -- start 2024-04-25T00:10:57.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.511+0000 7f4103df6640 1 -- start start 2024-04-25T00:10:57.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.515+0000 7f4103df6640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f40fc148890 0x7f40fc072590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:57.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.515+0000 7f4103df6640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f40fc06e890 0x7f40fc071980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:57.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.515+0000 7f4103df6640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40fc152c10 con 0x7f40fc070dd0 2024-04-25T00:10:57.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.515+0000 7f4103df6640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f40fc151e50 con 0x7f40fc148890 2024-04-25T00:10:57.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.515+0000 7f4103df6640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f40fc152130 con 0x7f40fc06e890 2024-04-25T00:10:57.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.515+0000 7f4101b92640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f40fc148890 0x7f40fc072590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:57.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.515+0000 7f4101b92640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f40fc148890 0x7f40fc072590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40702/0 (socket says 172.21.15.116:40702) 2024-04-25T00:10:57.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.515+0000 7f4101b92640 1 -- 172.21.15.116:0/1427058281 learned_addr learned my addr 172.21.15.116:0/1427058281 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:10:57.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.515+0000 7f4102393640 1 --2- 172.21.15.116:0/1427058281 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f40fc06e890 0x7f40fc071980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:57.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.515+0000 7f4101b92640 1 -- 172.21.15.116:0/1427058281 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f40fc06e890 msgr2=0x7f40fc071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:10:57.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.515+0000 7f4101b92640 1 --2- 172.21.15.116:0/1427058281 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f40fc06e890 0x7f40fc071980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:57.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.515+0000 7f4101b92640 1 -- 172.21.15.116:0/1427058281 >> v1:172.21.15.116:6789/0 conn(0x7f40fc070dd0 legacy=0x7f40fc071270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:57.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.515+0000 7f4101b92640 1 -- 172.21.15.116:0/1427058281 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40fc152410 con 0x7f40fc148890 2024-04-25T00:10:57.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.515+0000 7f4100b90640 1 -- 172.21.15.116:0/1427058281 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3355154469 0 0) 0x7f40fc152c10 con 0x7f40fc070dd0 2024-04-25T00:10:57.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.515+0000 7f4102393640 1 --2- 172.21.15.116:0/1427058281 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f40fc06e890 0x7f40fc071980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:10:57.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.515+0000 7f4101b92640 1 --2- 172.21.15.116:0/1427058281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40fc148890 0x7f40fc072590 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f40ec006fd0 tx=0x7f40ec01e9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5c5609a0b7f8228 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:57.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.515+0000 7f4100b90640 1 -- 172.21.15.116:0/1427058281 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40ec0ad070 con 0x7f40fc148890 2024-04-25T00:10:57.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.515+0000 7f4100b90640 1 -- 172.21.15.116:0/1427058281 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40ec004230 con 0x7f40fc148890 2024-04-25T00:10:57.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.515+0000 7f4100b90640 1 -- 172.21.15.116:0/1427058281 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40ec0a76c0 con 0x7f40fc148890 2024-04-25T00:10:57.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.519+0000 7f4103df6640 1 -- 172.21.15.116:0/1427058281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40fc148890 msgr2=0x7f40fc072590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:57.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.519+0000 7f4103df6640 1 --2- 172.21.15.116:0/1427058281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40fc148890 0x7f40fc072590 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f40ec006fd0 tx=0x7f40ec01e9f0 comp rx=0 tx=0).stop 2024-04-25T00:10:57.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.519+0000 7f4103df6640 1 -- 172.21.15.116:0/1427058281 shutdown_connections 2024-04-25T00:10:57.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.519+0000 7f4103df6640 1 --2- 172.21.15.116:0/1427058281 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f40fc06e890 0x7f40fc071980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:57.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.519+0000 7f4103df6640 1 --2- 172.21.15.116:0/1427058281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40fc148890 0x7f40fc072590 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:57.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.519+0000 7f4103df6640 1 -- 172.21.15.116:0/1427058281 >> 172.21.15.116:0/1427058281 conn(0x7f40fc0c3b00 msgr2=0x7f40fc0c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:10:57.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.519+0000 7f4103df6640 1 -- 172.21.15.116:0/1427058281 shutdown_connections 2024-04-25T00:10:57.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.519+0000 7f4103df6640 1 -- 172.21.15.116:0/1427058281 wait complete. 2024-04-25T00:10:57.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.519+0000 7f4103df6640 1 Processor -- start 2024-04-25T00:10:57.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.519+0000 7f4103df6640 1 -- start start 2024-04-25T00:10:57.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.519+0000 7f4103df6640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f40fc06e890 0x7f40fc150620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:57.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.519+0000 7f4103df6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f40fc070dd0 0x7f40fc1edd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:57.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.519+0000 7f4103df6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40fc148890 0x7f40fc1470f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:57.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.519+0000 7f4103df6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f40fc0ab920 con 0x7f40fc148890 2024-04-25T00:10:57.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.519+0000 7f4103df6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f40fc0ab7a0 con 0x7f40fc070dd0 2024-04-25T00:10:57.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.519+0000 7f4103df6640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f40fc0abaa0 con 0x7f40fc06e890 2024-04-25T00:10:57.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.519+0000 7f4102393640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40fc148890 0x7f40fc1470f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:57.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.519+0000 7f4102393640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40fc148890 0x7f40fc1470f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40716/0 (socket says 172.21.15.116:40716) 2024-04-25T00:10:57.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.519+0000 7f4102393640 1 -- 172.21.15.116:0/3066754082 learned_addr learned my addr 172.21.15.116:0/3066754082 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:10:57.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.519+0000 7f4101b92640 1 --2- 172.21.15.116:0/3066754082 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f40fc06e890 0x7f40fc150620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:57.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.519+0000 7f4101391640 1 --2- 172.21.15.116:0/3066754082 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f40fc070dd0 0x7f40fc1edd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:57.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.519+0000 7f4102393640 1 -- 172.21.15.116:0/3066754082 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f40fc06e890 msgr2=0x7f40fc150620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:57.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.519+0000 7f4102393640 1 --2- 172.21.15.116:0/3066754082 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f40fc06e890 0x7f40fc150620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:57.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.519+0000 7f4102393640 1 -- 172.21.15.116:0/3066754082 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f40fc070dd0 msgr2=0x7f40fc1edd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:57.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.519+0000 7f4102393640 1 --2- 172.21.15.116:0/3066754082 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f40fc070dd0 0x7f40fc1edd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:57.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.519+0000 7f4102393640 1 -- 172.21.15.116:0/3066754082 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40fc145930 con 0x7f40fc148890 2024-04-25T00:10:57.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.519+0000 7f4101391640 1 --2- 172.21.15.116:0/3066754082 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f40fc070dd0 0x7f40fc1edd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:10:57.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.519+0000 7f4102393640 1 --2- 172.21.15.116:0/3066754082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40fc148890 0x7f40fc1470f0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f40f4007e70 tx=0x7f40f400d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:57.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.519+0000 7f40f27fc640 1 -- 172.21.15.116:0/3066754082 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40f401a070 con 0x7f40fc148890 2024-04-25T00:10:57.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.519+0000 7f40f27fc640 1 -- 172.21.15.116:0/3066754082 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40f4015600 con 0x7f40fc148890 2024-04-25T00:10:57.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.519+0000 7f4103df6640 1 -- 172.21.15.116:0/3066754082 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40fc148260 con 0x7f40fc148890 2024-04-25T00:10:57.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.519+0000 7f40f27fc640 1 -- 172.21.15.116:0/3066754082 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40f40159f0 con 0x7f40fc148890 2024-04-25T00:10:57.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.519+0000 7f4103df6640 1 -- 172.21.15.116:0/3066754082 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40fc1519d0 con 0x7f40fc148890 2024-04-25T00:10:57.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.519+0000 7f4103df6640 1 -- 172.21.15.116:0/3066754082 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40fc068350 con 0x7f40fc148890 2024-04-25T00:10:57.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.523+0000 7f40f27fc640 1 -- 172.21.15.116:0/3066754082 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f40f401d070 con 0x7f40fc148890 2024-04-25T00:10:57.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.523+0000 7f40f27fc640 1 --2- 172.21.15.116:0/3066754082 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f40e00678f0 0x7f40e0069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:57.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.523+0000 7f40f27fc640 1 -- 172.21.15.116:0/3066754082 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7698+0+0 (secure 0 0 0) 0x7f40f4086670 con 0x7f40fc148890 2024-04-25T00:10:57.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.523+0000 7f4101b92640 1 --2- 172.21.15.116:0/3066754082 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f40e00678f0 0x7f40e0069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:57.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.523+0000 7f4101b92640 1 --2- 172.21.15.116:0/3066754082 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f40e00678f0 0x7f40e0069db0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f40ec0b1000 tx=0x7f40ec0b3040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:57.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.527+0000 7f40f27fc640 1 -- 172.21.15.116:0/3066754082 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f40f4010070 con 0x7f40fc148890 2024-04-25T00:10:57.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.719+0000 7f4103df6640 1 -- 172.21.15.116:0/3066754082 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f40fc1ee9d0 con 0x7f40fc148890 2024-04-25T00:10:57.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.719+0000 7f40f27fc640 1 -- 172.21.15.116:0/3066754082 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) ==== 94+0+6504 (secure 0 0 0) 0x7f40f4056390 con 0x7f40fc148890 2024-04-25T00:10:57.717 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:10:57.718 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5461,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72},{"gid":6114,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:10:23.780120+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6135},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6135":{"gid":6135,"name":"c","rank":0,"incarnation":68,"state":"up:active","state_seq":13,"addr":"172.21.15.116:6835/1358527848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1358527848},{"type":"v1","addr":"172.21.15.116:6835","nonce":1358527848}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6135,"qdb_cluster":[6135]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:09:33.601524+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":132,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4235,"qdb_cluster":[4235]},"id":2}]} 2024-04-25T00:10:57.718 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 72 2024-04-25T00:10:57.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.719+0000 7f4103df6640 1 -- 172.21.15.116:0/3066754082 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f40e00678f0 msgr2=0x7f40e0069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:57.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.719+0000 7f4103df6640 1 --2- 172.21.15.116:0/3066754082 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f40e00678f0 0x7f40e0069db0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f40ec0b1000 tx=0x7f40ec0b3040 comp rx=0 tx=0).stop 2024-04-25T00:10:57.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.719+0000 7f4103df6640 1 -- 172.21.15.116:0/3066754082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40fc148890 msgr2=0x7f40fc1470f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:57.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.719+0000 7f4103df6640 1 --2- 172.21.15.116:0/3066754082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40fc148890 0x7f40fc1470f0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f40f4007e70 tx=0x7f40f400d4a0 comp rx=0 tx=0).stop 2024-04-25T00:10:57.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.719+0000 7f4103df6640 1 -- 172.21.15.116:0/3066754082 shutdown_connections 2024-04-25T00:10:57.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.719+0000 7f4103df6640 1 --2- 172.21.15.116:0/3066754082 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f40e00678f0 0x7f40e0069db0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:57.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.719+0000 7f4103df6640 1 --2- 172.21.15.116:0/3066754082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40fc148890 0x7f40fc1470f0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:57.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.719+0000 7f4103df6640 1 --2- 172.21.15.116:0/3066754082 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f40fc070dd0 0x7f40fc1edd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:57.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.719+0000 7f4103df6640 1 --2- 172.21.15.116:0/3066754082 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f40fc06e890 0x7f40fc150620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:57.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.719+0000 7f4103df6640 1 -- 172.21.15.116:0/3066754082 >> 172.21.15.116:0/3066754082 conn(0x7f40fc0c3b00 msgr2=0x7f40fc0c4bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:10:57.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.719+0000 7f4103df6640 1 -- 172.21.15.116:0/3066754082 shutdown_connections 2024-04-25T00:10:57.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:57.719+0000 7f4103df6640 1 -- 172.21.15.116:0/3066754082 wait complete. 2024-04-25T00:10:57.735 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:10:59.736 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:10:59.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.871+0000 7f07b64fc640 1 Processor -- start 2024-04-25T00:10:59.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.875+0000 7f07b64fc640 1 -- start start 2024-04-25T00:10:59.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.875+0000 7f07b64fc640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f07b00a94a0 0x7f07b01563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:59.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.875+0000 7f07b64fc640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f07b0071680 0x7f07b0071ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:59.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.875+0000 7f07b64fc640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07b0071220 con 0x7f07b014c720 2024-04-25T00:10:59.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.875+0000 7f07b64fc640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f07b0153460 con 0x7f07b0071680 2024-04-25T00:10:59.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.875+0000 7f07b64fc640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f07b0152420 con 0x7f07b00a94a0 2024-04-25T00:10:59.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.875+0000 7f07affff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f07b014c720 0x7f07b00c2a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:56762/0 (socket says 172.21.15.116:56762) 2024-04-25T00:10:59.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.875+0000 7f07affff640 1 -- 172.21.15.116:0/2920436432 learned_addr learned my addr 172.21.15.116:0/2920436432 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:10:59.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.875+0000 7f07af7fe640 1 --2- 172.21.15.116:0/2920436432 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f07b00a94a0 0x7f07b01563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:59.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.875+0000 7f07b4a99640 1 --2- 172.21.15.116:0/2920436432 >> v2:172.21.15.174:3300/0 conn(0x7f07b0071680 0x7f07b0071ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:59.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.875+0000 7f07aeffd640 1 -- 172.21.15.116:0/2920436432 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 508172599 0 0) 0x7f07b0071220 con 0x7f07b014c720 2024-04-25T00:10:59.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.875+0000 7f07aeffd640 1 -- 172.21.15.116:0/2920436432 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0798003530 con 0x7f07b014c720 2024-04-25T00:10:59.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.875+0000 7f07aeffd640 1 -- 172.21.15.116:0/2920436432 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2437394300 0 0) 0x7f0798003530 con 0x7f07b014c720 2024-04-25T00:10:59.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.875+0000 7f07aeffd640 1 -- 172.21.15.116:0/2920436432 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f07b00a94a0 msgr2=0x7f07b01563d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:10:59.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.875+0000 7f07aeffd640 1 --2- 172.21.15.116:0/2920436432 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f07b00a94a0 0x7f07b01563d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:59.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.875+0000 7f07aeffd640 1 -- 172.21.15.116:0/2920436432 >> v2:172.21.15.174:3300/0 conn(0x7f07b0071680 msgr2=0x7f07b0071ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:10:59.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.875+0000 7f07aeffd640 1 --2- 172.21.15.116:0/2920436432 >> v2:172.21.15.174:3300/0 conn(0x7f07b0071680 0x7f07b0071ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:59.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.875+0000 7f07aeffd640 1 -- 172.21.15.116:0/2920436432 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07b014cb00 con 0x7f07b014c720 2024-04-25T00:10:59.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.875+0000 7f07aeffd640 1 -- 172.21.15.116:0/2920436432 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f07a0001680 con 0x7f07b014c720 2024-04-25T00:10:59.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.875+0000 7f07aeffd640 1 -- 172.21.15.116:0/2920436432 >> v1:172.21.15.116:6789/0 conn(0x7f07b014c720 legacy=0x7f07b00c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:59.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.875+0000 7f07aeffd640 1 --2- 172.21.15.116:0/2920436432 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f07b00a94a0 0x7f07980039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:59.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.875+0000 7f07aeffd640 1 --2- 172.21.15.116:0/2920436432 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0798003f40 0x7f0798004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:59.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.875+0000 7f07aeffd640 1 --2- 172.21.15.116:0/2920436432 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0798004d50 0x7f07980051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:59.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.875+0000 7f07aeffd640 1 -- 172.21.15.116:0/2920436432 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f07b0153460 con 0x7f0798004d50 2024-04-25T00:10:59.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.875+0000 7f07aeffd640 1 -- 172.21.15.116:0/2920436432 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f07b0152420 con 0x7f0798003f40 2024-04-25T00:10:59.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.875+0000 7f07aeffd640 1 -- 172.21.15.116:0/2920436432 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f079800ba20 con 0x7f07b00a94a0 2024-04-25T00:10:59.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.875+0000 7f07b4a99640 1 --2- 172.21.15.116:0/2920436432 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0798004d50 0x7f07980051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:59.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.875+0000 7f07af7fe640 1 --2- 172.21.15.116:0/2920436432 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0798003f40 0x7f0798004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:59.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.875+0000 7f07affff640 1 --2- 172.21.15.116:0/2920436432 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f07b00a94a0 0x7f07980039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:59.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.875+0000 7f07b4a99640 1 -- 172.21.15.116:0/2920436432 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f07b00a94a0 msgr2=0x7f07980039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:10:59.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.875+0000 7f07b4a99640 1 --2- 172.21.15.116:0/2920436432 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f07b00a94a0 0x7f07980039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:59.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.875+0000 7f07b4a99640 1 -- 172.21.15.116:0/2920436432 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0798003f40 msgr2=0x7f0798004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:10:59.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.875+0000 7f07b4a99640 1 --2- 172.21.15.116:0/2920436432 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0798003f40 0x7f0798004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:59.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.875+0000 7f07b4a99640 1 -- 172.21.15.116:0/2920436432 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0798003530 con 0x7f0798004d50 2024-04-25T00:10:59.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.879+0000 7f07b4a99640 1 --2- 172.21.15.116:0/2920436432 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0798004d50 0x7f07980051d0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f07a800e170 tx=0x7f07a801cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc57f188f208c194 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:59.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.879+0000 7f07aeffd640 1 -- 172.21.15.116:0/2920436432 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07a80b2070 con 0x7f0798004d50 2024-04-25T00:10:59.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.879+0000 7f07aeffd640 1 -- 172.21.15.116:0/2920436432 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07a80a5cb0 con 0x7f0798004d50 2024-04-25T00:10:59.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.879+0000 7f07aeffd640 1 -- 172.21.15.116:0/2920436432 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07a8009c10 con 0x7f0798004d50 2024-04-25T00:10:59.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.879+0000 7f07b64fc640 1 -- 172.21.15.116:0/2920436432 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0798004d50 msgr2=0x7f07980051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:59.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.879+0000 7f07b64fc640 1 --2- 172.21.15.116:0/2920436432 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0798004d50 0x7f07980051d0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f07a800e170 tx=0x7f07a801cca0 comp rx=0 tx=0).stop 2024-04-25T00:10:59.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.879+0000 7f07b4a99640 1 -- 172.21.15.116:0/2920436432 reap_dead start 2024-04-25T00:10:59.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.879+0000 7f07b64fc640 1 -- 172.21.15.116:0/2920436432 shutdown_connections 2024-04-25T00:10:59.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.879+0000 7f07b64fc640 1 -- 172.21.15.116:0/2920436432 >> 172.21.15.116:0/2920436432 conn(0x7f07b00c3b00 msgr2=0x7f07b00c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:10:59.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.879+0000 7f07b64fc640 1 -- 172.21.15.116:0/2920436432 shutdown_connections 2024-04-25T00:10:59.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.879+0000 7f07b64fc640 1 -- 172.21.15.116:0/2920436432 wait complete. 2024-04-25T00:10:59.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.879+0000 7f07b64fc640 1 Processor -- start 2024-04-25T00:10:59.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.879+0000 7f07b64fc640 1 -- start start 2024-04-25T00:10:59.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.879+0000 7f07b64fc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f07b0147eb0 0x7f07b006d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:59.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.879+0000 7f07b64fc640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f07b0148290 0x7f07b006d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:59.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.879+0000 7f07b64fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07b006dc20 0x7f07b013ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:59.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.879+0000 7f07b64fc640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f07b0152c60 con 0x7f07b006dc20 2024-04-25T00:10:59.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.879+0000 7f07b64fc640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f07b0152980 con 0x7f07b0147eb0 2024-04-25T00:10:59.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.879+0000 7f07b64fc640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f07b0152f40 con 0x7f07b0148290 2024-04-25T00:10:59.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.879+0000 7f07b4a99640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07b006dc20 0x7f07b013ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:59.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.879+0000 7f07b4a99640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07b006dc20 0x7f07b013ff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40726/0 (socket says 172.21.15.116:40726) 2024-04-25T00:10:59.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.879+0000 7f07b4a99640 1 -- 172.21.15.116:0/1682389045 learned_addr learned my addr 172.21.15.116:0/1682389045 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:10:59.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.879+0000 7f07affff640 1 --2- 172.21.15.116:0/1682389045 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f07b0147eb0 0x7f07b006d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:59.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.879+0000 7f07af7fe640 1 --2- 172.21.15.116:0/1682389045 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f07b0148290 0x7f07b006d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:59.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.879+0000 7f07b4a99640 1 -- 172.21.15.116:0/1682389045 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f07b0148290 msgr2=0x7f07b006d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:59.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.879+0000 7f07b4a99640 1 --2- 172.21.15.116:0/1682389045 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f07b0148290 0x7f07b006d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:59.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.879+0000 7f07b4a99640 1 -- 172.21.15.116:0/1682389045 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f07b0147eb0 msgr2=0x7f07b006d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:10:59.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.879+0000 7f07b4a99640 1 --2- 172.21.15.116:0/1682389045 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f07b0147eb0 0x7f07b006d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:10:59.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.879+0000 7f07b4a99640 1 -- 172.21.15.116:0/1682389045 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07b00c6c80 con 0x7f07b006dc20 2024-04-25T00:10:59.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.883+0000 7f07b4a99640 1 --2- 172.21.15.116:0/1682389045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07b006dc20 0x7f07b013ff50 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f07a800e170 tx=0x7f07a8009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:59.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.883+0000 7f07acff9640 1 -- 172.21.15.116:0/1682389045 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07a80b2040 con 0x7f07b006dc20 2024-04-25T00:10:59.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.883+0000 7f07acff9640 1 -- 172.21.15.116:0/1682389045 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07a80a5ab0 con 0x7f07b006dc20 2024-04-25T00:10:59.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.883+0000 7f07b64fc640 1 -- 172.21.15.116:0/1682389045 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07b006e450 con 0x7f07b006dc20 2024-04-25T00:10:59.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.883+0000 7f07acff9640 1 -- 172.21.15.116:0/1682389045 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07a80a5e60 con 0x7f07b006dc20 2024-04-25T00:10:59.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.883+0000 7f07b64fc640 1 -- 172.21.15.116:0/1682389045 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07b013b5e0 con 0x7f07b006dc20 2024-04-25T00:10:59.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.883+0000 7f07b64fc640 1 -- 172.21.15.116:0/1682389045 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07b0068350 con 0x7f07b006dc20 2024-04-25T00:10:59.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.883+0000 7f07acff9640 1 -- 172.21.15.116:0/1682389045 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f07a80be020 con 0x7f07b006dc20 2024-04-25T00:10:59.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.883+0000 7f07acff9640 1 --2- 172.21.15.116:0/1682389045 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f079c0678c0 0x7f079c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:10:59.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.883+0000 7f07acff9640 1 -- 172.21.15.116:0/1682389045 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 7698+0+0 (secure 0 0 0) 0x7f07a8120360 con 0x7f07b006dc20 2024-04-25T00:10:59.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.883+0000 7f07affff640 1 --2- 172.21.15.116:0/1682389045 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f079c0678c0 0x7f079c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:10:59.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.883+0000 7f07affff640 1 --2- 172.21.15.116:0/1682389045 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f079c0678c0 0x7f079c069d80 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f07a0000c10 tx=0x7f07a0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:10:59.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:10:59.887+0000 7f07acff9640 1 -- 172.21.15.116:0/1682389045 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f07a80c5020 con 0x7f07b006dc20 2024-04-25T00:11:00.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:00.079+0000 7f07b64fc640 1 -- 172.21.15.116:0/1682389045 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f07b00c3640 con 0x7f07b006dc20 2024-04-25T00:11:00.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:00.083+0000 7f07acff9640 1 -- 172.21.15.116:0/1682389045 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) ==== 94+0+6504 (secure 0 0 0) 0x7f07a8016a30 con 0x7f07b006dc20 2024-04-25T00:11:00.080 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:11:00.080 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5461,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72},{"gid":6114,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:10:23.780120+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":40,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6135},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6135":{"gid":6135,"name":"c","rank":0,"incarnation":68,"state":"up:active","state_seq":13,"addr":"172.21.15.116:6835/1358527848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1358527848},{"type":"v1","addr":"172.21.15.116:6835","nonce":1358527848}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6135,"qdb_cluster":[6135]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:09:33.601524+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":132,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4235,"qdb_cluster":[4235]},"id":2}]} 2024-04-25T00:11:00.080 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 72 2024-04-25T00:11:00.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:00.083+0000 7f07b64fc640 1 -- 172.21.15.116:0/1682389045 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f079c0678c0 msgr2=0x7f079c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:00.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:00.083+0000 7f07b64fc640 1 --2- 172.21.15.116:0/1682389045 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f079c0678c0 0x7f079c069d80 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f07a0000c10 tx=0x7f07a0009040 comp rx=0 tx=0).stop 2024-04-25T00:11:00.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:00.083+0000 7f07b64fc640 1 -- 172.21.15.116:0/1682389045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07b006dc20 msgr2=0x7f07b013ff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:00.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:00.083+0000 7f07b64fc640 1 --2- 172.21.15.116:0/1682389045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07b006dc20 0x7f07b013ff50 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f07a800e170 tx=0x7f07a8009750 comp rx=0 tx=0).stop 2024-04-25T00:11:00.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:00.083+0000 7f07b64fc640 1 -- 172.21.15.116:0/1682389045 shutdown_connections 2024-04-25T00:11:00.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:00.083+0000 7f07b64fc640 1 --2- 172.21.15.116:0/1682389045 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f079c0678c0 0x7f079c069d80 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:00.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:00.083+0000 7f07b64fc640 1 --2- 172.21.15.116:0/1682389045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f07b006dc20 0x7f07b013ff50 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:00.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:00.083+0000 7f07b64fc640 1 --2- 172.21.15.116:0/1682389045 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f07b0148290 0x7f07b006d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:00.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:00.083+0000 7f07b64fc640 1 --2- 172.21.15.116:0/1682389045 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f07b0147eb0 0x7f07b006d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:00.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:00.083+0000 7f07b64fc640 1 -- 172.21.15.116:0/1682389045 >> 172.21.15.116:0/1682389045 conn(0x7f07b00c3b00 msgr2=0x7f07b0150ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:11:00.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:00.083+0000 7f07b64fc640 1 -- 172.21.15.116:0/1682389045 shutdown_connections 2024-04-25T00:11:00.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:00.083+0000 7f07b64fc640 1 -- 172.21.15.116:0/1682389045 wait complete. 2024-04-25T00:11:00.098 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:11:02.099 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:11:02.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.227+0000 7ff0093bd640 1 Processor -- start 2024-04-25T00:11:02.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.227+0000 7ff0093bd640 1 -- start start 2024-04-25T00:11:02.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.231+0000 7ff0093bd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff0040ad180 0x7ff0040c83b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:02.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.231+0000 7ff0093bd640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff0040c88f0 0x7ff0040c9040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:02.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.231+0000 7ff0093bd640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff004157630 con 0x7ff0040adf40 2024-04-25T00:11:02.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.231+0000 7ff0093bd640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7ff0040ad600 con 0x7ff0040c88f0 2024-04-25T00:11:02.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.231+0000 7ff0093bd640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff0040c7960 con 0x7ff0040ad180 2024-04-25T00:11:02.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.231+0000 7ff0027fc640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff0040ad180 0x7ff0040c83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:02.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.231+0000 7ff0037fe640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff0040c88f0 0x7ff0040c9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:02.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.231+0000 7ff0037fe640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff0040c88f0 0x7ff0040c9040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:51188/0 (socket says 172.21.15.116:51188) 2024-04-25T00:11:02.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.231+0000 7ff0037fe640 1 -- 172.21.15.116:0/4235136512 learned_addr learned my addr 172.21.15.116:0/4235136512 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:11:02.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.231+0000 7ff0027fc640 1 -- 172.21.15.116:0/4235136512 >> v2:172.21.15.174:3300/0 conn(0x7ff0040c88f0 msgr2=0x7ff0040c9040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:11:02.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.231+0000 7ff0027fc640 1 --2- 172.21.15.116:0/4235136512 >> v2:172.21.15.174:3300/0 conn(0x7ff0040c88f0 0x7ff0040c9040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:02.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.231+0000 7ff0027fc640 1 -- 172.21.15.116:0/4235136512 >> v1:172.21.15.174:6789/0 conn(0x7ff0040adf40 legacy=0x7ff004156c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:02.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.231+0000 7ff0027fc640 1 -- 172.21.15.116:0/4235136512 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff004157910 con 0x7ff0040ad180 2024-04-25T00:11:02.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.231+0000 7ff0037fe640 1 --2- 172.21.15.116:0/4235136512 >> v2:172.21.15.174:3300/0 conn(0x7ff0040c88f0 0x7ff0040c9040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:11:02.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.231+0000 7ff0027fc640 1 --2- 172.21.15.116:0/4235136512 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff0040ad180 0x7ff0040c83b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7feff80097c0 tx=0x7feff801c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=acd0d27c50154bf2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:11:02.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.235+0000 7ff001ffb640 1 -- 172.21.15.116:0/4235136512 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff80b0070 con 0x7ff0040ad180 2024-04-25T00:11:02.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.235+0000 7ff001ffb640 1 -- 172.21.15.116:0/4235136512 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feff80ab600 con 0x7ff0040ad180 2024-04-25T00:11:02.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.235+0000 7ff001ffb640 1 -- 172.21.15.116:0/4235136512 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff80ab9d0 con 0x7ff0040ad180 2024-04-25T00:11:02.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.235+0000 7ff0093bd640 1 -- 172.21.15.116:0/4235136512 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff0040ad180 msgr2=0x7ff0040c83b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:02.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.235+0000 7ff0093bd640 1 --2- 172.21.15.116:0/4235136512 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff0040ad180 0x7ff0040c83b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7feff80097c0 tx=0x7feff801c4a0 comp rx=0 tx=0).stop 2024-04-25T00:11:02.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.235+0000 7ff0093bd640 1 -- 172.21.15.116:0/4235136512 shutdown_connections 2024-04-25T00:11:02.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.235+0000 7ff0093bd640 1 --2- 172.21.15.116:0/4235136512 >> v2:172.21.15.174:3300/0 conn(0x7ff0040c88f0 0x7ff0040c9040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:02.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.235+0000 7ff0093bd640 1 --2- 172.21.15.116:0/4235136512 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff0040ad180 0x7ff0040c83b0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:02.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.235+0000 7ff0093bd640 1 -- 172.21.15.116:0/4235136512 >> 172.21.15.116:0/4235136512 conn(0x7ff004074480 msgr2=0x7ff004074890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:11:02.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.235+0000 7ff0093bd640 1 -- 172.21.15.116:0/4235136512 shutdown_connections 2024-04-25T00:11:02.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.235+0000 7ff0093bd640 1 -- 172.21.15.116:0/4235136512 wait complete. 2024-04-25T00:11:02.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.235+0000 7ff0093bd640 1 Processor -- start 2024-04-25T00:11:02.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.235+0000 7ff0093bd640 1 -- start start 2024-04-25T00:11:02.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.235+0000 7ff0093bd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff0040ad180 0x7ff004152d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:02.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.235+0000 7ff0093bd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0040adf40 0x7ff00414ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:02.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.235+0000 7ff0093bd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff0040c88f0 0x7ff00414bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:02.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.235+0000 7ff0093bd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff0040ae4a0 con 0x7ff0040adf40 2024-04-25T00:11:02.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.235+0000 7ff0093bd640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff0040ae320 con 0x7ff0040ad180 2024-04-25T00:11:02.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.235+0000 7ff0093bd640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff00414d4f0 con 0x7ff0040c88f0 2024-04-25T00:11:02.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.235+0000 7ff0027fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0040adf40 0x7ff00414ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:02.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.235+0000 7ff0027fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0040adf40 0x7ff00414ff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40732/0 (socket says 172.21.15.116:40732) 2024-04-25T00:11:02.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.235+0000 7ff0027fc640 1 -- 172.21.15.116:0/2509213127 learned_addr learned my addr 172.21.15.116:0/2509213127 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:11:02.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.235+0000 7ff002ffd640 1 --2- 172.21.15.116:0/2509213127 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff0040ad180 0x7ff004152d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:02.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.235+0000 7ff0037fe640 1 --2- 172.21.15.116:0/2509213127 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff0040c88f0 0x7ff00414bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:02.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.235+0000 7ff0027fc640 1 -- 172.21.15.116:0/2509213127 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff0040c88f0 msgr2=0x7ff00414bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:02.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.235+0000 7ff0027fc640 1 --2- 172.21.15.116:0/2509213127 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff0040c88f0 0x7ff00414bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:02.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.235+0000 7ff0027fc640 1 -- 172.21.15.116:0/2509213127 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff0040ad180 msgr2=0x7ff004152d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:02.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.235+0000 7ff0027fc640 1 --2- 172.21.15.116:0/2509213127 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff0040ad180 0x7ff004152d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:02.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.235+0000 7ff0027fc640 1 -- 172.21.15.116:0/2509213127 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff004150910 con 0x7ff0040adf40 2024-04-25T00:11:02.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.235+0000 7ff0027fc640 1 --2- 172.21.15.116:0/2509213127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0040adf40 0x7ff00414ff00 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7feff80ae040 tx=0x7feff8019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:11:02.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.235+0000 7fefeb7fe640 1 -- 172.21.15.116:0/2509213127 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff80b0040 con 0x7ff0040adf40 2024-04-25T00:11:02.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.235+0000 7fefeb7fe640 1 -- 172.21.15.116:0/2509213127 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feff80ab400 con 0x7ff0040adf40 2024-04-25T00:11:02.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.235+0000 7ff0093bd640 1 -- 172.21.15.116:0/2509213127 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0040c1950 con 0x7ff0040adf40 2024-04-25T00:11:02.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.235+0000 7fefeb7fe640 1 -- 172.21.15.116:0/2509213127 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff80b4690 con 0x7ff0040adf40 2024-04-25T00:11:02.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.235+0000 7ff0093bd640 1 -- 172.21.15.116:0/2509213127 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0040c1ed0 con 0x7ff0040adf40 2024-04-25T00:11:02.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.239+0000 7ff0093bd640 1 -- 172.21.15.116:0/2509213127 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff004068a60 con 0x7ff0040adf40 2024-04-25T00:11:02.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.239+0000 7fefeb7fe640 1 -- 172.21.15.116:0/2509213127 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7feff80bb020 con 0x7ff0040adf40 2024-04-25T00:11:02.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.239+0000 7fefeb7fe640 1 --2- 172.21.15.116:0/2509213127 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7feff00678c0 0x7feff0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:02.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.239+0000 7fefeb7fe640 1 -- 172.21.15.116:0/2509213127 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 7784+0+0 (secure 0 0 0) 0x7feff811f8a0 con 0x7ff0040adf40 2024-04-25T00:11:02.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.239+0000 7ff002ffd640 1 --2- 172.21.15.116:0/2509213127 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7feff00678c0 0x7feff0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:02.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.239+0000 7ff002ffd640 1 --2- 172.21.15.116:0/2509213127 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7feff00678c0 0x7feff0069d80 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7ff0040c1510 tx=0x7feffc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:11:02.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.243+0000 7fefeb7fe640 1 -- 172.21.15.116:0/2509213127 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7feff80c2020 con 0x7ff0040adf40 2024-04-25T00:11:02.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.447+0000 7ff0093bd640 1 -- 172.21.15.116:0/2509213127 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff004152600 con 0x7ff0040adf40 2024-04-25T00:11:02.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.447+0000 7fefeb7fe640 1 -- 172.21.15.116:0/2509213127 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) ==== 94+0+5672 (secure 0 0 0) 0x7feff8016090 con 0x7ff0040adf40 2024-04-25T00:11:02.446 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:11:02.446 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":74,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5461,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:11:00.805929+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6114},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6114":{"gid":6114,"name":"a","rank":0,"incarnation":73,"state":"up:reconnect","state_seq":23,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:09:33.601524+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":132,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4235,"qdb_cluster":[4235]},"id":2}]} 2024-04-25T00:11:02.446 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 74 2024-04-25T00:11:02.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.451+0000 7ff0093bd640 1 -- 172.21.15.116:0/2509213127 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7feff00678c0 msgr2=0x7feff0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:02.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.451+0000 7ff0093bd640 1 --2- 172.21.15.116:0/2509213127 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7feff00678c0 0x7feff0069d80 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7ff0040c1510 tx=0x7feffc009040 comp rx=0 tx=0).stop 2024-04-25T00:11:02.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.451+0000 7ff0093bd640 1 -- 172.21.15.116:0/2509213127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0040adf40 msgr2=0x7ff00414ff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:02.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.451+0000 7ff0093bd640 1 --2- 172.21.15.116:0/2509213127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0040adf40 0x7ff00414ff00 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7feff80ae040 tx=0x7feff8019240 comp rx=0 tx=0).stop 2024-04-25T00:11:02.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.451+0000 7ff0093bd640 1 -- 172.21.15.116:0/2509213127 shutdown_connections 2024-04-25T00:11:02.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.451+0000 7ff0093bd640 1 --2- 172.21.15.116:0/2509213127 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7feff00678c0 0x7feff0069d80 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:02.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.451+0000 7ff0093bd640 1 --2- 172.21.15.116:0/2509213127 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff0040c88f0 0x7ff00414bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:02.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.451+0000 7ff0093bd640 1 --2- 172.21.15.116:0/2509213127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff0040adf40 0x7ff00414ff00 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:02.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.451+0000 7ff0093bd640 1 --2- 172.21.15.116:0/2509213127 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff0040ad180 0x7ff004152d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:02.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.451+0000 7ff0093bd640 1 -- 172.21.15.116:0/2509213127 >> 172.21.15.116:0/2509213127 conn(0x7ff004074480 msgr2=0x7ff004151910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:11:02.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.451+0000 7ff0093bd640 1 -- 172.21.15.116:0/2509213127 shutdown_connections 2024-04-25T00:11:02.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.451+0000 7ff0093bd640 1 -- 172.21.15.116:0/2509213127 wait complete. 2024-04-25T00:11:02.464 INFO:tasks.mds_thrash.fs.[a]:mds.c down, removed from mdsmap 2024-04-25T00:11:02.464 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-04-25T00:11:02.465 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:11:02.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.575+0000 7f04c55a0640 1 Processor -- start 2024-04-25T00:11:02.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.575+0000 7f04c55a0640 1 -- start start 2024-04-25T00:11:02.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.579+0000 7f04c55a0640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f04c013e310 0x7f04c00c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:02.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.579+0000 7f04c55a0640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f04c013ed60 0x7f04c01560d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:02.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.579+0000 7f04c55a0640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04c006d200 con 0x7f04c00c2f50 2024-04-25T00:11:02.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.579+0000 7f04c55a0640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f04c0070d40 con 0x7f04c013e310 2024-04-25T00:11:02.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.579+0000 7f04c55a0640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f04c0071020 con 0x7f04c013ed60 2024-04-25T00:11:02.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.579+0000 7f04beffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f04c013e310 0x7f04c00c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:02.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.579+0000 7f04beffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f04c013e310 0x7f04c00c2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40734/0 (socket says 172.21.15.116:40734) 2024-04-25T00:11:02.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.579+0000 7f04beffd640 1 -- 172.21.15.116:0/364852647 learned_addr learned my addr 172.21.15.116:0/364852647 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:11:02.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.579+0000 7f04b7fff640 1 --1- 172.21.15.116:0/364852647 >> v1:172.21.15.116:6789/0 conn(0x7f04c00c2f50 0x7f04c01559c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:56768/0 (socket says 172.21.15.116:56768) 2024-04-25T00:11:02.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.579+0000 7f04bf7fe640 1 --2- 172.21.15.116:0/364852647 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f04c013ed60 0x7f04c01560d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:02.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.579+0000 7f04beffd640 1 -- 172.21.15.116:0/364852647 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f04c013ed60 msgr2=0x7f04c01560d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:11:02.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.579+0000 7f04beffd640 1 --2- 172.21.15.116:0/364852647 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f04c013ed60 0x7f04c01560d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:02.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.579+0000 7f04beffd640 1 -- 172.21.15.116:0/364852647 >> v1:172.21.15.116:6789/0 conn(0x7f04c00c2f50 legacy=0x7f04c01559c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:02.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.579+0000 7f04be7fc640 1 -- 172.21.15.116:0/364852647 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 121494734 0 0) 0x7f04c006d200 con 0x7f04c00c2f50 2024-04-25T00:11:02.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.579+0000 7f04beffd640 1 -- 172.21.15.116:0/364852647 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04c0156610 con 0x7f04c013e310 2024-04-25T00:11:02.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.579+0000 7f04beffd640 1 --2- 172.21.15.116:0/364852647 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04c013e310 0x7f04c00c2a10 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f04b0009870 tx=0x7f04b001c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86879d6048bdf349 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:11:02.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.579+0000 7f04be7fc640 1 -- 172.21.15.116:0/364852647 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b00b0070 con 0x7f04c013e310 2024-04-25T00:11:02.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.579+0000 7f04be7fc640 1 -- 172.21.15.116:0/364852647 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04b0004230 con 0x7f04c013e310 2024-04-25T00:11:02.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.579+0000 7f04be7fc640 1 -- 172.21.15.116:0/364852647 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b0003620 con 0x7f04c013e310 2024-04-25T00:11:02.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.579+0000 7f04c55a0640 1 -- 172.21.15.116:0/364852647 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04c013e310 msgr2=0x7f04c00c2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:02.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.579+0000 7f04c55a0640 1 --2- 172.21.15.116:0/364852647 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04c013e310 0x7f04c00c2a10 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f04b0009870 tx=0x7f04b001c440 comp rx=0 tx=0).stop 2024-04-25T00:11:02.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.579+0000 7f04c55a0640 1 -- 172.21.15.116:0/364852647 shutdown_connections 2024-04-25T00:11:02.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.579+0000 7f04c55a0640 1 --2- 172.21.15.116:0/364852647 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f04c013ed60 0x7f04c01560d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:02.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.579+0000 7f04c55a0640 1 --2- 172.21.15.116:0/364852647 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04c013e310 0x7f04c00c2a10 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:02.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.579+0000 7f04c55a0640 1 -- 172.21.15.116:0/364852647 >> 172.21.15.116:0/364852647 conn(0x7f04c00c3b00 msgr2=0x7f04c00c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:11:02.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.579+0000 7f04c55a0640 1 -- 172.21.15.116:0/364852647 shutdown_connections 2024-04-25T00:11:02.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.583+0000 7f04c55a0640 1 -- 172.21.15.116:0/364852647 wait complete. 2024-04-25T00:11:02.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.583+0000 7f04c55a0640 1 Processor -- start 2024-04-25T00:11:02.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.583+0000 7f04c55a0640 1 -- start start 2024-04-25T00:11:02.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.583+0000 7f04c55a0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04c00c2f50 0x7f04c01300b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:02.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.583+0000 7f04c55a0640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f04c013e310 0x7f04c0152330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:02.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.583+0000 7f04c55a0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f04c013ed60 0x7f04c0152870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:02.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.583+0000 7f04c55a0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f04c0162b70 con 0x7f04c00c2f50 2024-04-25T00:11:02.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.583+0000 7f04c55a0640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f04c01629f0 con 0x7f04c013ed60 2024-04-25T00:11:02.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.583+0000 7f04c55a0640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f04c0162cf0 con 0x7f04c013e310 2024-04-25T00:11:02.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.583+0000 7f04beffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04c00c2f50 0x7f04c01300b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:02.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.583+0000 7f04beffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04c00c2f50 0x7f04c01300b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40746/0 (socket says 172.21.15.116:40746) 2024-04-25T00:11:02.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.583+0000 7f04beffd640 1 -- 172.21.15.116:0/3663413334 learned_addr learned my addr 172.21.15.116:0/3663413334 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:11:02.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.583+0000 7f04b7fff640 1 --2- 172.21.15.116:0/3663413334 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f04c013e310 0x7f04c0152330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:02.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.583+0000 7f04beffd640 1 -- 172.21.15.116:0/3663413334 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f04c013e310 msgr2=0x7f04c0152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:02.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.583+0000 7f04beffd640 1 --2- 172.21.15.116:0/3663413334 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f04c013e310 0x7f04c0152330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:02.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.583+0000 7f04beffd640 1 -- 172.21.15.116:0/3663413334 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f04c013ed60 msgr2=0x7f04c0152870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:02.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.583+0000 7f04beffd640 1 --2- 172.21.15.116:0/3663413334 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f04c013ed60 0x7f04c0152870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:02.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.583+0000 7f04beffd640 1 -- 172.21.15.116:0/3663413334 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04c0130fc0 con 0x7f04c00c2f50 2024-04-25T00:11:02.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.583+0000 7f04beffd640 1 --2- 172.21.15.116:0/3663413334 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04c00c2f50 0x7f04c01300b0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f04b00ae040 tx=0x7f04b00191e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:11:02.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.583+0000 7f04b77fe640 1 -- 172.21.15.116:0/3663413334 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b00b9020 con 0x7f04c00c2f50 2024-04-25T00:11:02.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.583+0000 7f04b77fe640 1 -- 172.21.15.116:0/3663413334 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04b00b1040 con 0x7f04c00c2f50 2024-04-25T00:11:02.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.583+0000 7f04c55a0640 1 -- 172.21.15.116:0/3663413334 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04c01f2380 con 0x7f04c00c2f50 2024-04-25T00:11:02.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.583+0000 7f04b77fe640 1 -- 172.21.15.116:0/3663413334 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b00b0050 con 0x7f04c00c2f50 2024-04-25T00:11:02.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.583+0000 7f04c55a0640 1 -- 172.21.15.116:0/3663413334 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04c01f2940 con 0x7f04c00c2f50 2024-04-25T00:11:02.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.583+0000 7f04c55a0640 1 -- 172.21.15.116:0/3663413334 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04c0068350 con 0x7f04c00c2f50 2024-04-25T00:11:02.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.583+0000 7f04b77fe640 1 -- 172.21.15.116:0/3663413334 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f04b00bc020 con 0x7f04c00c2f50 2024-04-25T00:11:02.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.587+0000 7f04b77fe640 1 --2- 172.21.15.116:0/3663413334 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f04a00678f0 0x7f04a0069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:02.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.587+0000 7f04b77fe640 1 -- 172.21.15.116:0/3663413334 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 7784+0+0 (secure 0 0 0) 0x7f04b0120050 con 0x7f04c00c2f50 2024-04-25T00:11:02.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.587+0000 7f04b7fff640 1 --2- 172.21.15.116:0/3663413334 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f04a00678f0 0x7f04a0069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:02.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.587+0000 7f04b7fff640 1 --2- 172.21.15.116:0/3663413334 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f04a00678f0 0x7f04a0069db0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f04c0130e80 tx=0x7f04a8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:11:02.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.591+0000 7f04b77fe640 1 -- 172.21.15.116:0/3663413334 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f04b00c3030 con 0x7f04c00c2f50 2024-04-25T00:11:02.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.775+0000 7f04c55a0640 1 -- 172.21.15.116:0/3663413334 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f04c00ac760 con 0x7f04c00c2f50 2024-04-25T00:11:02.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.779+0000 7f04b77fe640 1 -- 172.21.15.116:0/3663413334 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) ==== 94+0+5669 (secure 0 0 0) 0x7f04b0016150 con 0x7f04c00c2f50 2024-04-25T00:11:02.776 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:11:02.777 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":75,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5461,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:11:01.633897+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6114},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6114":{"gid":6114,"name":"a","rank":0,"incarnation":73,"state":"up:rejoin","state_seq":24,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:09:33.601524+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":132,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4235,"qdb_cluster":[4235]},"id":2}]} 2024-04-25T00:11:02.777 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 75 2024-04-25T00:11:02.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.779+0000 7f04c55a0640 1 -- 172.21.15.116:0/3663413334 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f04a00678f0 msgr2=0x7f04a0069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:02.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.779+0000 7f04c55a0640 1 --2- 172.21.15.116:0/3663413334 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f04a00678f0 0x7f04a0069db0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f04c0130e80 tx=0x7f04a8009040 comp rx=0 tx=0).stop 2024-04-25T00:11:02.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.779+0000 7f04c55a0640 1 -- 172.21.15.116:0/3663413334 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04c00c2f50 msgr2=0x7f04c01300b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:02.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.779+0000 7f04c55a0640 1 --2- 172.21.15.116:0/3663413334 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04c00c2f50 0x7f04c01300b0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f04b00ae040 tx=0x7f04b00191e0 comp rx=0 tx=0).stop 2024-04-25T00:11:02.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.779+0000 7f04c55a0640 1 -- 172.21.15.116:0/3663413334 shutdown_connections 2024-04-25T00:11:02.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.779+0000 7f04c55a0640 1 --2- 172.21.15.116:0/3663413334 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f04a00678f0 0x7f04a0069db0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:02.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.779+0000 7f04c55a0640 1 --2- 172.21.15.116:0/3663413334 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f04c013ed60 0x7f04c0152870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:02.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.779+0000 7f04c55a0640 1 --2- 172.21.15.116:0/3663413334 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f04c013e310 0x7f04c0152330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:02.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.779+0000 7f04c55a0640 1 --2- 172.21.15.116:0/3663413334 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f04c00c2f50 0x7f04c01300b0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:02.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.779+0000 7f04c55a0640 1 -- 172.21.15.116:0/3663413334 >> 172.21.15.116:0/3663413334 conn(0x7f04c00c3b00 msgr2=0x7f04c006d9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:11:02.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.779+0000 7f04c55a0640 1 -- 172.21.15.116:0/3663413334 shutdown_connections 2024-04-25T00:11:02.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:02.779+0000 7f04c55a0640 1 -- 172.21.15.116:0/3663413334 wait complete. 2024-04-25T00:11:02.794 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 75, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:04.678737+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 75, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6114": { "addr": "172.21.15.116:6833/3705654249", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6832", "nonce": 3705654249, "type": "v2" }, { "addr": "172.21.15.116:6833", "nonce": 3705654249, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6114, "incarnation": 73, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:rejoin", "state_seq": 24 } }, "last_failure": 0, "last_failure_osd_epoch": 41, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T00:11:01.633897+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6114 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:18.825887+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 65, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_4235": { "addr": "172.21.15.174:6835/2883220846", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6834", "nonce": 2883220846, "type": "v2" }, { "addr": "172.21.15.174:6835", "nonce": 2883220846, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 4235, "incarnation": 62, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 132 } }, "last_failure": 0, "last_failure_osd_epoch": 39, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-04-25T00:09:33.601524+0000", "qdb_cluster": [ 4235 ], "qdb_leader": 4235, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4235 } } } ], "standbys": [ { "addr": "172.21.15.174:6837/3398690896", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6836", "nonce": 3398690896, "type": "v2" }, { "addr": "172.21.15.174:6837", "nonce": 3398690896, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 72, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5461, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-25T00:11:04.795 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:11:04.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:04.939+0000 7f3bff3ae640 1 Processor -- start 2024-04-25T00:11:04.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:04.939+0000 7f3bff3ae640 1 -- start start 2024-04-25T00:11:04.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:04.939+0000 7f3bff3ae640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f3bf81541d0 0x7f3bf80ad2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:04.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:04.939+0000 7f3bff3ae640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3bf80c32d0 0x7f3bf80c36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:04.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:04.939+0000 7f3bff3ae640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3bf80a9ef0 con 0x7f3bf80acef0 2024-04-25T00:11:04.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:04.939+0000 7f3bff3ae640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f3bf80c2b60 con 0x7f3bf80c32d0 2024-04-25T00:11:04.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:04.939+0000 7f3bff3ae640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f3bf80c2e40 con 0x7f3bf81541d0 2024-04-25T00:11:04.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:04.943+0000 7f3bfd94b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3bf80c32d0 0x7f3bf80c36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:04.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:04.943+0000 7f3bfd94b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3bf80c32d0 0x7f3bf80c36e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40608/0 (socket says 172.21.15.116:40608) 2024-04-25T00:11:04.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:04.943+0000 7f3bfd94b640 1 -- 172.21.15.116:0/2517504173 learned_addr learned my addr 172.21.15.116:0/2517504173 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:11:04.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:04.943+0000 7f3bfc949640 1 --2- 172.21.15.116:0/2517504173 >> v2:172.21.15.174:3300/0 conn(0x7f3bf81541d0 0x7f3bf80ad2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:04.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:04.943+0000 7f3bfd94b640 1 -- 172.21.15.116:0/2517504173 >> v2:172.21.15.174:3300/0 conn(0x7f3bf81541d0 msgr2=0x7f3bf80ad2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:11:04.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:04.943+0000 7f3bfd94b640 1 --2- 172.21.15.116:0/2517504173 >> v2:172.21.15.174:3300/0 conn(0x7f3bf81541d0 0x7f3bf80ad2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:04.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:04.943+0000 7f3bfd94b640 1 -- 172.21.15.116:0/2517504173 >> v1:172.21.15.174:6789/0 conn(0x7f3bf80acef0 legacy=0x7f3bf80a7d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:11:04.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:04.943+0000 7f3bfd94b640 1 -- 172.21.15.116:0/2517504173 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bf80aa1d0 con 0x7f3bf80c32d0 2024-04-25T00:11:04.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:04.943+0000 7f3bfd94b640 1 --2- 172.21.15.116:0/2517504173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bf80c32d0 0x7f3bf80c36e0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f3bf80748c0 tx=0x7f3bf001c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d3111c44c47725e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:11:04.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:04.943+0000 7f3beffff640 1 -- 172.21.15.116:0/2517504173 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf00a55e0 con 0x7f3bf80c32d0 2024-04-25T00:11:04.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:04.943+0000 7f3beffff640 1 -- 172.21.15.116:0/2517504173 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bf00a5780 con 0x7f3bf80c32d0 2024-04-25T00:11:04.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:04.943+0000 7f3beffff640 1 -- 172.21.15.116:0/2517504173 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf00a5b50 con 0x7f3bf80c32d0 2024-04-25T00:11:04.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:04.947+0000 7f3bff3ae640 1 -- 172.21.15.116:0/2517504173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bf80c32d0 msgr2=0x7f3bf80c36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:04.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:04.947+0000 7f3bff3ae640 1 --2- 172.21.15.116:0/2517504173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bf80c32d0 0x7f3bf80c36e0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f3bf80748c0 tx=0x7f3bf001c920 comp rx=0 tx=0).stop 2024-04-25T00:11:04.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:04.947+0000 7f3bff3ae640 1 -- 172.21.15.116:0/2517504173 shutdown_connections 2024-04-25T00:11:04.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:04.947+0000 7f3bff3ae640 1 --2- 172.21.15.116:0/2517504173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bf80c32d0 0x7f3bf80c36e0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:04.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:04.947+0000 7f3bff3ae640 1 --2- 172.21.15.116:0/2517504173 >> v2:172.21.15.174:3300/0 conn(0x7f3bf81541d0 0x7f3bf80ad2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:04.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:04.947+0000 7f3bff3ae640 1 -- 172.21.15.116:0/2517504173 >> 172.21.15.116:0/2517504173 conn(0x7f3bf80c4200 msgr2=0x7f3bf80c4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:11:04.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:04.947+0000 7f3bff3ae640 1 -- 172.21.15.116:0/2517504173 shutdown_connections 2024-04-25T00:11:04.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:04.947+0000 7f3bff3ae640 1 -- 172.21.15.116:0/2517504173 wait complete. 2024-04-25T00:11:04.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:04.947+0000 7f3bff3ae640 1 Processor -- start 2024-04-25T00:11:04.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:04.947+0000 7f3bff3ae640 1 -- start start 2024-04-25T00:11:04.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:04.947+0000 7f3bff3ae640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bf80acef0 0x7f3bf81eea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:04.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:04.947+0000 7f3bff3ae640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3bf80c32d0 0x7f3bf81f2770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:04.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:04.947+0000 7f3bff3ae640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3bf81541d0 0x7f3bf806c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:04.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:04.947+0000 7f3bff3ae640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3bf80a8610 con 0x7f3bf80acef0 2024-04-25T00:11:04.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:04.947+0000 7f3bff3ae640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3bf80a8490 con 0x7f3bf81541d0 2024-04-25T00:11:04.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:04.947+0000 7f3bff3ae640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f3bf8160860 con 0x7f3bf80c32d0 2024-04-25T00:11:04.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:04.947+0000 7f3bfd14a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bf80acef0 0x7f3bf81eea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:04.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:04.947+0000 7f3bfd14a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bf80acef0 0x7f3bf81eea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40612/0 (socket says 172.21.15.116:40612) 2024-04-25T00:11:04.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:04.947+0000 7f3bfd14a640 1 -- 172.21.15.116:0/453016940 learned_addr learned my addr 172.21.15.116:0/453016940 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:11:04.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:04.947+0000 7f3bfc949640 1 --2- 172.21.15.116:0/453016940 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3bf80c32d0 0x7f3bf81f2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:04.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:04.947+0000 7f3bfd94b640 1 --2- 172.21.15.116:0/453016940 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3bf81541d0 0x7f3bf806c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:04.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:04.947+0000 7f3bfd14a640 1 -- 172.21.15.116:0/453016940 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3bf80c32d0 msgr2=0x7f3bf81f2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:04.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:04.947+0000 7f3bfd14a640 1 --2- 172.21.15.116:0/453016940 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3bf80c32d0 0x7f3bf81f2770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:04.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:04.947+0000 7f3bfd14a640 1 -- 172.21.15.116:0/453016940 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3bf81541d0 msgr2=0x7f3bf806c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:04.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:04.947+0000 7f3bfd14a640 1 --2- 172.21.15.116:0/453016940 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3bf81541d0 0x7f3bf806c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:04.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:04.947+0000 7f3bfd14a640 1 -- 172.21.15.116:0/453016940 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bf8072e40 con 0x7f3bf80acef0 2024-04-25T00:11:04.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:04.947+0000 7f3bfd14a640 1 --2- 172.21.15.116:0/453016940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bf80acef0 0x7f3bf81eea90 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f3bf400c480 tx=0x7f3bf400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:11:04.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:04.947+0000 7f3bedffb640 1 -- 172.21.15.116:0/453016940 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf4012830 con 0x7f3bf80acef0 2024-04-25T00:11:04.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:04.947+0000 7f3bedffb640 1 -- 172.21.15.116:0/453016940 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bf4017070 con 0x7f3bf80acef0 2024-04-25T00:11:04.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:04.947+0000 7f3bff3ae640 1 -- 172.21.15.116:0/453016940 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bf80730e0 con 0x7f3bf80acef0 2024-04-25T00:11:04.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:04.947+0000 7f3bedffb640 1 -- 172.21.15.116:0/453016940 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bf4012c30 con 0x7f3bf80acef0 2024-04-25T00:11:04.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:04.947+0000 7f3bff3ae640 1 -- 172.21.15.116:0/453016940 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bf815cf60 con 0x7f3bf80acef0 2024-04-25T00:11:04.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:04.951+0000 7f3bff3ae640 1 -- 172.21.15.116:0/453016940 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bf8068a60 con 0x7f3bf80acef0 2024-04-25T00:11:04.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:04.951+0000 7f3bedffb640 1 -- 172.21.15.116:0/453016940 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3bf401c460 con 0x7f3bf80acef0 2024-04-25T00:11:04.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:04.951+0000 7f3bedffb640 1 --2- 172.21.15.116:0/453016940 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3bdc0678c0 0x7f3bdc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:04.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:04.951+0000 7f3bedffb640 1 -- 172.21.15.116:0/453016940 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 7784+0+0 (secure 0 0 0) 0x7f3bf4086240 con 0x7f3bf80acef0 2024-04-25T00:11:04.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:04.951+0000 7f3bfc949640 1 --2- 172.21.15.116:0/453016940 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3bdc0678c0 0x7f3bdc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:04.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:04.951+0000 7f3bfc949640 1 --2- 172.21.15.116:0/453016940 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3bdc0678c0 0x7f3bdc069d80 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f3bf815c490 tx=0x7f3be8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:11:04.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:04.955+0000 7f3bedffb640 1 -- 172.21.15.116:0/453016940 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f3bf40565f0 con 0x7f3bf80acef0 2024-04-25T00:11:05.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.163+0000 7f3bff3ae640 1 -- 172.21.15.116:0/453016940 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3bf815de20 con 0x7f3bf80acef0 2024-04-25T00:11:05.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.163+0000 7f3bedffb640 1 -- 172.21.15.116:0/453016940 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) ==== 94+0+5676 (secure 0 0 0) 0x7f3bf4055ef0 con 0x7f3bf80acef0 2024-04-25T00:11:05.163 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:11:05.163 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":76,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5461,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:11:03.640578+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6114},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6114":{"gid":6114,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":25,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6114,"qdb_cluster":[6114]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:09:33.601524+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":132,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4235,"qdb_cluster":[4235]},"id":2}]} 2024-04-25T00:11:05.163 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 76 2024-04-25T00:11:05.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.167+0000 7f3bff3ae640 1 -- 172.21.15.116:0/453016940 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3bdc0678c0 msgr2=0x7f3bdc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:05.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.167+0000 7f3bff3ae640 1 --2- 172.21.15.116:0/453016940 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3bdc0678c0 0x7f3bdc069d80 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f3bf815c490 tx=0x7f3be8008040 comp rx=0 tx=0).stop 2024-04-25T00:11:05.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.167+0000 7f3bff3ae640 1 -- 172.21.15.116:0/453016940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bf80acef0 msgr2=0x7f3bf81eea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:05.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.167+0000 7f3bff3ae640 1 --2- 172.21.15.116:0/453016940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bf80acef0 0x7f3bf81eea90 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f3bf400c480 tx=0x7f3bf400c940 comp rx=0 tx=0).stop 2024-04-25T00:11:05.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.167+0000 7f3bff3ae640 1 -- 172.21.15.116:0/453016940 shutdown_connections 2024-04-25T00:11:05.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.167+0000 7f3bff3ae640 1 --2- 172.21.15.116:0/453016940 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3bdc0678c0 0x7f3bdc069d80 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:05.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.167+0000 7f3bff3ae640 1 --2- 172.21.15.116:0/453016940 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3bf81541d0 0x7f3bf806c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:05.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.167+0000 7f3bff3ae640 1 --2- 172.21.15.116:0/453016940 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3bf80c32d0 0x7f3bf81f2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:05.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.167+0000 7f3bff3ae640 1 --2- 172.21.15.116:0/453016940 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bf80acef0 0x7f3bf81eea90 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:05.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.167+0000 7f3bff3ae640 1 -- 172.21.15.116:0/453016940 >> 172.21.15.116:0/453016940 conn(0x7f3bf80c4200 msgr2=0x7f3bf814c6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:11:05.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.167+0000 7f3bff3ae640 1 -- 172.21.15.116:0/453016940 shutdown_connections 2024-04-25T00:11:05.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.167+0000 7f3bff3ae640 1 -- 172.21.15.116:0/453016940 wait complete. 2024-04-25T00:11:05.183 INFO:tasks.mds_thrash.fs.[a]:mds.a has gained rank=0, replacing gid=6135 2024-04-25T00:11:05.183 INFO:tasks.mds_thrash.fs.[a]:waiting for 0 secs before reviving mds.c 2024-04-25T00:11:05.183 INFO:tasks.mds_thrash.fs.[a]:reviving mds.c 2024-04-25T00:11:05.184 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-25T00:11:05.184 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-25T00:11:05.185 INFO:tasks.ceph.mds.c:Started 2024-04-25T00:11:05.185 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:11:05.249 INFO:tasks.ceph.mds.c.smithi116.stdout:starting mds.c at 2024-04-25T00:11:05.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.303+0000 7f4517259640 1 Processor -- start 2024-04-25T00:11:05.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.303+0000 7f4517259640 1 -- start start 2024-04-25T00:11:05.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.307+0000 7f4517259640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f45100c8740 0x7f45100711e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:05.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.307+0000 7f4517259640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f45100c53b0 0x7f4510152d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:05.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.307+0000 7f4517259640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45100c23f0 con 0x7f45101532a0 2024-04-25T00:11:05.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.307+0000 7f4517259640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f4510158f20 con 0x7f45100c8740 2024-04-25T00:11:05.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.307+0000 7f4517259640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f45100c2690 con 0x7f45100c53b0 2024-04-25T00:11:05.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.307+0000 7f4514ff5640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f45100c8740 0x7f45100711e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:05.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.307+0000 7f4514ff5640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f45100c8740 0x7f45100711e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40644/0 (socket says 172.21.15.116:40644) 2024-04-25T00:11:05.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.307+0000 7f4514ff5640 1 -- 172.21.15.116:0/2436449014 learned_addr learned my addr 172.21.15.116:0/2436449014 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:11:05.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.307+0000 7f45157f6640 1 --1- 172.21.15.116:0/2436449014 >> v1:172.21.15.116:6789/0 conn(0x7f45101532a0 0x7f4510155da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:33882/0 (socket says 172.21.15.116:33882) 2024-04-25T00:11:05.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.307+0000 7f450ffff640 1 --2- 172.21.15.116:0/2436449014 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f45100c53b0 0x7f4510152d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:05.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.307+0000 7f4514ff5640 1 -- 172.21.15.116:0/2436449014 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f45100c53b0 msgr2=0x7f4510152d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:11:05.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.307+0000 7f4514ff5640 1 --2- 172.21.15.116:0/2436449014 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f45100c53b0 0x7f4510152d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:05.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.307+0000 7f4514ff5640 1 -- 172.21.15.116:0/2436449014 >> v1:172.21.15.116:6789/0 conn(0x7f45101532a0 legacy=0x7f4510155da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:05.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.307+0000 7f4514ff5640 1 -- 172.21.15.116:0/2436449014 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45100c2970 con 0x7f45100c8740 2024-04-25T00:11:05.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.307+0000 7f4514ff5640 1 --2- 172.21.15.116:0/2436449014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45100c8740 0x7f45100711e0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f4504009870 tx=0x7f450401c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f9526c033f38042 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:11:05.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.307+0000 7f450f7fe640 1 -- 172.21.15.116:0/2436449014 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45040b0070 con 0x7f45100c8740 2024-04-25T00:11:05.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.307+0000 7f450f7fe640 1 -- 172.21.15.116:0/2436449014 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4504004230 con 0x7f45100c8740 2024-04-25T00:11:05.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.307+0000 7f450f7fe640 1 -- 172.21.15.116:0/2436449014 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4504003620 con 0x7f45100c8740 2024-04-25T00:11:05.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.307+0000 7f4517259640 1 -- 172.21.15.116:0/2436449014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45100c8740 msgr2=0x7f45100711e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:05.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.307+0000 7f4517259640 1 --2- 172.21.15.116:0/2436449014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45100c8740 0x7f45100711e0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f4504009870 tx=0x7f450401c440 comp rx=0 tx=0).stop 2024-04-25T00:11:05.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.307+0000 7f4517259640 1 -- 172.21.15.116:0/2436449014 shutdown_connections 2024-04-25T00:11:05.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.307+0000 7f4517259640 1 --2- 172.21.15.116:0/2436449014 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f45100c53b0 0x7f4510152d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:05.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.307+0000 7f4517259640 1 --2- 172.21.15.116:0/2436449014 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45100c8740 0x7f45100711e0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:05.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.307+0000 7f4517259640 1 -- 172.21.15.116:0/2436449014 >> 172.21.15.116:0/2436449014 conn(0x7f451006f0c0 msgr2=0x7f451006f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:11:05.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.307+0000 7f4517259640 1 -- 172.21.15.116:0/2436449014 shutdown_connections 2024-04-25T00:11:05.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.307+0000 7f4517259640 1 -- 172.21.15.116:0/2436449014 wait complete. 2024-04-25T00:11:05.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.311+0000 7f4517259640 1 Processor -- start 2024-04-25T00:11:05.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.311+0000 7f4517259640 1 -- start start 2024-04-25T00:11:05.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.311+0000 7f4517259640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45100c53b0 0x7f451014b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:05.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.311+0000 7f4517259640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f45100c8740 0x7f45101f20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:05.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.311+0000 7f4517259640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45101532a0 0x7f45101f2600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:05.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.311+0000 7f4517259640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4510071930 con 0x7f45101532a0 2024-04-25T00:11:05.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.311+0000 7f4517259640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f45100c8d70 con 0x7f45100c53b0 2024-04-25T00:11:05.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.311+0000 7f4517259640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f45101568c0 con 0x7f45100c8740 2024-04-25T00:11:05.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.311+0000 7f45157f6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45101532a0 0x7f45101f2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:05.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.311+0000 7f45157f6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45101532a0 0x7f45101f2600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40658/0 (socket says 172.21.15.116:40658) 2024-04-25T00:11:05.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.311+0000 7f45157f6640 1 -- 172.21.15.116:0/1458544705 learned_addr learned my addr 172.21.15.116:0/1458544705 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:11:05.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.311+0000 7f4514ff5640 1 --2- 172.21.15.116:0/1458544705 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45100c53b0 0x7f451014b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:05.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.311+0000 7f450ffff640 1 --2- 172.21.15.116:0/1458544705 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f45100c8740 0x7f45101f20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:05.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.311+0000 7f45157f6640 1 -- 172.21.15.116:0/1458544705 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f45100c8740 msgr2=0x7f45101f20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:05.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.311+0000 7f45157f6640 1 --2- 172.21.15.116:0/1458544705 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f45100c8740 0x7f45101f20c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:05.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.311+0000 7f45157f6640 1 -- 172.21.15.116:0/1458544705 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45100c53b0 msgr2=0x7f451014b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:05.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.311+0000 7f45157f6640 1 --2- 172.21.15.116:0/1458544705 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45100c53b0 0x7f451014b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:05.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.311+0000 7f45157f6640 1 -- 172.21.15.116:0/1458544705 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f451014a1e0 con 0x7f45101532a0 2024-04-25T00:11:05.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.311+0000 7f45157f6640 1 --2- 172.21.15.116:0/1458544705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45101532a0 0x7f45101f2600 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f450800c730 tx=0x7f450800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:11:05.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.311+0000 7f450d7fa640 1 -- 172.21.15.116:0/1458544705 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4508018070 con 0x7f45101532a0 2024-04-25T00:11:05.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.311+0000 7f450d7fa640 1 -- 172.21.15.116:0/1458544705 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4508004d70 con 0x7f45101532a0 2024-04-25T00:11:05.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.311+0000 7f4517259640 1 -- 172.21.15.116:0/1458544705 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4510149940 con 0x7f45101532a0 2024-04-25T00:11:05.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.311+0000 7f450d7fa640 1 -- 172.21.15.116:0/1458544705 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f450800a860 con 0x7f45101532a0 2024-04-25T00:11:05.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.311+0000 7f4517259640 1 -- 172.21.15.116:0/1458544705 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4510149f00 con 0x7f45101532a0 2024-04-25T00:11:05.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.311+0000 7f4517259640 1 -- 172.21.15.116:0/1458544705 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4510068a60 con 0x7f45101532a0 2024-04-25T00:11:05.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.315+0000 7f450d7fa640 1 -- 172.21.15.116:0/1458544705 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f450801b070 con 0x7f45101532a0 2024-04-25T00:11:05.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.315+0000 7f450d7fa640 1 --2- 172.21.15.116:0/1458544705 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f44f40678c0 0x7f44f4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:05.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.315+0000 7f450d7fa640 1 -- 172.21.15.116:0/1458544705 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 7784+0+0 (secure 0 0 0) 0x7f4508085a40 con 0x7f45101532a0 2024-04-25T00:11:05.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.315+0000 7f4514ff5640 1 --2- 172.21.15.116:0/1458544705 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f44f40678c0 0x7f44f4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:05.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.315+0000 7f4514ff5640 1 --2- 172.21.15.116:0/1458544705 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f44f40678c0 0x7f44f4069d80 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f45040ad000 tx=0x7f4504008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:11:05.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.319+0000 7f450d7fa640 1 -- 172.21.15.116:0/1458544705 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f4508055df0 con 0x7f45101532a0 2024-04-25T00:11:05.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.571+0000 7f4517259640 1 -- 172.21.15.116:0/1458544705 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f451014f940 con 0x7f45101532a0 2024-04-25T00:11:05.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.571+0000 7f450d7fa640 1 -- 172.21.15.116:0/1458544705 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) ==== 94+0+5676 (secure 0 0 0) 0x7f45080556f0 con 0x7f45101532a0 2024-04-25T00:11:05.570 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:11:05.571 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":76,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5461,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:11:03.640578+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6114},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6114":{"gid":6114,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":25,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6114,"qdb_cluster":[6114]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:09:33.601524+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":132,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4235,"qdb_cluster":[4235]},"id":2}]} 2024-04-25T00:11:05.571 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 76 2024-04-25T00:11:05.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.575+0000 7f4517259640 1 -- 172.21.15.116:0/1458544705 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f44f40678c0 msgr2=0x7f44f4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:05.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.575+0000 7f4517259640 1 --2- 172.21.15.116:0/1458544705 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f44f40678c0 0x7f44f4069d80 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f45040ad000 tx=0x7f4504008040 comp rx=0 tx=0).stop 2024-04-25T00:11:05.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.575+0000 7f4517259640 1 -- 172.21.15.116:0/1458544705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45101532a0 msgr2=0x7f45101f2600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:05.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.575+0000 7f4517259640 1 --2- 172.21.15.116:0/1458544705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45101532a0 0x7f45101f2600 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f450800c730 tx=0x7f450800cbf0 comp rx=0 tx=0).stop 2024-04-25T00:11:05.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.575+0000 7f4517259640 1 -- 172.21.15.116:0/1458544705 shutdown_connections 2024-04-25T00:11:05.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.575+0000 7f4517259640 1 --2- 172.21.15.116:0/1458544705 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f44f40678c0 0x7f44f4069d80 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:05.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.575+0000 7f4517259640 1 --2- 172.21.15.116:0/1458544705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45101532a0 0x7f45101f2600 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:05.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.575+0000 7f4517259640 1 --2- 172.21.15.116:0/1458544705 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f45100c8740 0x7f45101f20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:05.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.575+0000 7f4517259640 1 --2- 172.21.15.116:0/1458544705 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45100c53b0 0x7f451014b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:05.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.575+0000 7f4517259640 1 -- 172.21.15.116:0/1458544705 >> 172.21.15.116:0/1458544705 conn(0x7f451006f0c0 msgr2=0x7f4510155da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:11:05.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.575+0000 7f4517259640 1 -- 172.21.15.116:0/1458544705 shutdown_connections 2024-04-25T00:11:05.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:05.575+0000 7f4517259640 1 -- 172.21.15.116:0/1458544705 wait complete. 2024-04-25T00:11:05.591 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2024-04-25T00:11:06.628 INFO:tasks.ceph.mds.a.smithi116.stderr:2024-04-25T00:11:06.631+0000 7fe194a32640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T00:11:07.189 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:11:07.192 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:11:07.593 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:11:07.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.727+0000 7fbd461e0640 1 Processor -- start 2024-04-25T00:11:07.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.727+0000 7fbd461e0640 1 -- start start 2024-04-25T00:11:07.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.731+0000 7fbd461e0640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbd40071520 0x7fbd40155fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:07.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.731+0000 7fbd461e0640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbd4014f7a0 0x7fbd4014fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:07.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.731+0000 7fbd461e0640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd400737d0 con 0x7fbd40155c00 2024-04-25T00:11:07.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.731+0000 7fbd461e0640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fbd40071bb0 con 0x7fbd40071520 2024-04-25T00:11:07.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.731+0000 7fbd461e0640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fbd40072ee0 con 0x7fbd4014f7a0 2024-04-25T00:11:07.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.731+0000 7fbd3effd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbd40071520 0x7fbd40155fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:07.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.731+0000 7fbd3effd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbd40071520 0x7fbd40155fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40666/0 (socket says 172.21.15.116:40666) 2024-04-25T00:11:07.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.731+0000 7fbd3f7fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fbd40155c00 0x7fbd400730c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:33896/0 (socket says 172.21.15.116:33896) 2024-04-25T00:11:07.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.731+0000 7fbd3effd640 1 -- 172.21.15.116:0/841168796 learned_addr learned my addr 172.21.15.116:0/841168796 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:11:07.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.731+0000 7fbd3ffff640 1 --2- 172.21.15.116:0/841168796 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbd4014f7a0 0x7fbd4014fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:07.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.731+0000 7fbd3effd640 1 -- 172.21.15.116:0/841168796 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbd4014f7a0 msgr2=0x7fbd4014fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:11:07.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.731+0000 7fbd3effd640 1 --2- 172.21.15.116:0/841168796 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbd4014f7a0 0x7fbd4014fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:07.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.731+0000 7fbd3effd640 1 -- 172.21.15.116:0/841168796 >> v1:172.21.15.116:6789/0 conn(0x7fbd40155c00 legacy=0x7fbd400730c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:07.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.731+0000 7fbd3e7fc640 1 -- 172.21.15.116:0/841168796 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1535311446 0 0) 0x7fbd400737d0 con 0x7fbd40155c00 2024-04-25T00:11:07.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.731+0000 7fbd3effd640 1 -- 172.21.15.116:0/841168796 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd40072920 con 0x7fbd40071520 2024-04-25T00:11:07.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.731+0000 7fbd3effd640 1 --2- 172.21.15.116:0/841168796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd40071520 0x7fbd40155fe0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fbd3400bb40 tx=0x7fbd3401eaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e309c3fba036220 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:11:07.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.731+0000 7fbd3e7fc640 1 -- 172.21.15.116:0/841168796 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd340b4070 con 0x7fbd40071520 2024-04-25T00:11:07.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.731+0000 7fbd3e7fc640 1 -- 172.21.15.116:0/841168796 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd340a78c0 con 0x7fbd40071520 2024-04-25T00:11:07.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.731+0000 7fbd3e7fc640 1 -- 172.21.15.116:0/841168796 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd340a7c90 con 0x7fbd40071520 2024-04-25T00:11:07.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.735+0000 7fbd461e0640 1 -- 172.21.15.116:0/841168796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd40071520 msgr2=0x7fbd40155fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:07.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.735+0000 7fbd461e0640 1 --2- 172.21.15.116:0/841168796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd40071520 0x7fbd40155fe0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fbd3400bb40 tx=0x7fbd3401eaf0 comp rx=0 tx=0).stop 2024-04-25T00:11:07.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.735+0000 7fbd461e0640 1 -- 172.21.15.116:0/841168796 shutdown_connections 2024-04-25T00:11:07.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.735+0000 7fbd461e0640 1 --2- 172.21.15.116:0/841168796 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbd4014f7a0 0x7fbd4014fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:07.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.735+0000 7fbd461e0640 1 --2- 172.21.15.116:0/841168796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd40071520 0x7fbd40155fe0 secure :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fbd3400bb40 tx=0x7fbd3401eaf0 comp rx=0 tx=0).stop 2024-04-25T00:11:07.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.735+0000 7fbd461e0640 1 -- 172.21.15.116:0/841168796 >> 172.21.15.116:0/841168796 conn(0x7fbd400c4200 msgr2=0x7fbd400c4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:11:07.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.735+0000 7fbd461e0640 1 -- 172.21.15.116:0/841168796 shutdown_connections 2024-04-25T00:11:07.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.735+0000 7fbd461e0640 1 -- 172.21.15.116:0/841168796 wait complete. 2024-04-25T00:11:07.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.735+0000 7fbd461e0640 1 Processor -- start 2024-04-25T00:11:07.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.735+0000 7fbd461e0640 1 -- start start 2024-04-25T00:11:07.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.735+0000 7fbd461e0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd4014f7a0 0x7fbd4014add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:07.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.735+0000 7fbd461e0640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbd40155c00 0x7fbd401480a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:07.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.735+0000 7fbd461e0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd400aa7b0 0x7fbd401485e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:07.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.735+0000 7fbd461e0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbd400c2100 con 0x7fbd400aa7b0 2024-04-25T00:11:07.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.735+0000 7fbd461e0640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbd400a7520 con 0x7fbd4014f7a0 2024-04-25T00:11:07.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.735+0000 7fbd461e0640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fbd400c2280 con 0x7fbd40155c00 2024-04-25T00:11:07.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.735+0000 7fbd3ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd400aa7b0 0x7fbd401485e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:07.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.735+0000 7fbd3ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd400aa7b0 0x7fbd401485e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40668/0 (socket says 172.21.15.116:40668) 2024-04-25T00:11:07.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.735+0000 7fbd3ffff640 1 -- 172.21.15.116:0/3459118421 learned_addr learned my addr 172.21.15.116:0/3459118421 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:11:07.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.735+0000 7fbd3effd640 1 --2- 172.21.15.116:0/3459118421 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbd40155c00 0x7fbd401480a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:07.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.735+0000 7fbd3f7fe640 1 --2- 172.21.15.116:0/3459118421 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd4014f7a0 0x7fbd4014add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:07.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.735+0000 7fbd3ffff640 1 -- 172.21.15.116:0/3459118421 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbd40155c00 msgr2=0x7fbd401480a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:07.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.735+0000 7fbd3ffff640 1 --2- 172.21.15.116:0/3459118421 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbd40155c00 0x7fbd401480a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:07.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.735+0000 7fbd3ffff640 1 -- 172.21.15.116:0/3459118421 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd4014f7a0 msgr2=0x7fbd4014add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:07.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.735+0000 7fbd3ffff640 1 --2- 172.21.15.116:0/3459118421 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd4014f7a0 0x7fbd4014add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:07.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.735+0000 7fbd3ffff640 1 -- 172.21.15.116:0/3459118421 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd40149230 con 0x7fbd400aa7b0 2024-04-25T00:11:07.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.735+0000 7fbd3f7fe640 1 --2- 172.21.15.116:0/3459118421 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd4014f7a0 0x7fbd4014add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:11:07.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.735+0000 7fbd3ffff640 1 --2- 172.21.15.116:0/3459118421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd400aa7b0 0x7fbd401485e0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fbd38009dd0 tx=0x7fbd3800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:11:07.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.735+0000 7fbd27fff640 1 -- 172.21.15.116:0/3459118421 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd38018020 con 0x7fbd400aa7b0 2024-04-25T00:11:07.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.735+0000 7fbd27fff640 1 -- 172.21.15.116:0/3459118421 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd38014480 con 0x7fbd400aa7b0 2024-04-25T00:11:07.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.735+0000 7fbd461e0640 1 -- 172.21.15.116:0/3459118421 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd40148df0 con 0x7fbd400aa7b0 2024-04-25T00:11:07.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.735+0000 7fbd27fff640 1 -- 172.21.15.116:0/3459118421 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd38014870 con 0x7fbd400aa7b0 2024-04-25T00:11:07.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.735+0000 7fbd461e0640 1 -- 172.21.15.116:0/3459118421 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd4014e730 con 0x7fbd400aa7b0 2024-04-25T00:11:07.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.739+0000 7fbd461e0640 1 -- 172.21.15.116:0/3459118421 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd40068a60 con 0x7fbd400aa7b0 2024-04-25T00:11:07.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.739+0000 7fbd27fff640 1 -- 172.21.15.116:0/3459118421 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbd3801b020 con 0x7fbd400aa7b0 2024-04-25T00:11:07.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.739+0000 7fbd27fff640 1 --2- 172.21.15.116:0/3459118421 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbd2c0678f0 0x7fbd2c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:07.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.739+0000 7fbd27fff640 1 -- 172.21.15.116:0/3459118421 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 7784+0+0 (secure 0 0 0) 0x7fbd38087100 con 0x7fbd400aa7b0 2024-04-25T00:11:07.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.739+0000 7fbd3f7fe640 1 --2- 172.21.15.116:0/3459118421 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbd2c0678f0 0x7fbd2c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:07.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.739+0000 7fbd3f7fe640 1 --2- 172.21.15.116:0/3459118421 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbd2c0678f0 0x7fbd2c069db0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fbd30000f70 tx=0x7fbd30007cb0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:11:07.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.743+0000 7fbd27fff640 1 -- 172.21.15.116:0/3459118421 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fbd38022030 con 0x7fbd400aa7b0 2024-04-25T00:11:07.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.979+0000 7fbd461e0640 1 -- 172.21.15.116:0/3459118421 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbd4014e220 con 0x7fbd400aa7b0 2024-04-25T00:11:07.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.983+0000 7fbd27fff640 1 -- 172.21.15.116:0/3459118421 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) ==== 94+0+6504 (secure 0 0 0) 0x7fbd38010040 con 0x7fbd400aa7b0 2024-04-25T00:11:07.980 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:11:07.980 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5461,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72},{"gid":6309,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1599708526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1599708526},{"type":"v1","addr":"172.21.15.116:6835","nonce":1599708526}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:11:03.640578+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6114},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6114":{"gid":6114,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":25,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6114,"qdb_cluster":[6114]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:09:33.601524+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":132,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4235,"qdb_cluster":[4235]},"id":2}]} 2024-04-25T00:11:07.980 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 77 2024-04-25T00:11:07.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.983+0000 7fbd461e0640 1 -- 172.21.15.116:0/3459118421 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbd2c0678f0 msgr2=0x7fbd2c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:07.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.983+0000 7fbd461e0640 1 --2- 172.21.15.116:0/3459118421 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbd2c0678f0 0x7fbd2c069db0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fbd30000f70 tx=0x7fbd30007cb0 comp rx=0 tx=0).stop 2024-04-25T00:11:07.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.983+0000 7fbd461e0640 1 -- 172.21.15.116:0/3459118421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd400aa7b0 msgr2=0x7fbd401485e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:07.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.983+0000 7fbd461e0640 1 --2- 172.21.15.116:0/3459118421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd400aa7b0 0x7fbd401485e0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fbd38009dd0 tx=0x7fbd3800c490 comp rx=0 tx=0).stop 2024-04-25T00:11:07.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.983+0000 7fbd461e0640 1 -- 172.21.15.116:0/3459118421 shutdown_connections 2024-04-25T00:11:07.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.983+0000 7fbd461e0640 1 --2- 172.21.15.116:0/3459118421 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbd2c0678f0 0x7fbd2c069db0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:07.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.983+0000 7fbd461e0640 1 --2- 172.21.15.116:0/3459118421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbd400aa7b0 0x7fbd401485e0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:07.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.983+0000 7fbd461e0640 1 --2- 172.21.15.116:0/3459118421 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbd40155c00 0x7fbd401480a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:07.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.983+0000 7fbd461e0640 1 --2- 172.21.15.116:0/3459118421 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbd4014f7a0 0x7fbd4014add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:07.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.983+0000 7fbd461e0640 1 -- 172.21.15.116:0/3459118421 >> 172.21.15.116:0/3459118421 conn(0x7fbd400c4200 msgr2=0x7fbd4014c040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:11:07.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.983+0000 7fbd461e0640 1 -- 172.21.15.116:0/3459118421 shutdown_connections 2024-04-25T00:11:07.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:07.983+0000 7fbd461e0640 1 -- 172.21.15.116:0/3459118421 wait complete. 2024-04-25T00:11:08.003 INFO:tasks.mds_thrash.fs.[a]:mds.c reported in up:standby state 2024-04-25T00:11:08.003 INFO:tasks.mds_thrash.fs.[a]:waiting for 79 secs before thrashing 2024-04-25T00:11:34.804 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:11:34.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:34.935+0000 7fb591e89640 1 Processor -- start 2024-04-25T00:11:34.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:34.935+0000 7fb591e89640 1 -- start start 2024-04-25T00:11:34.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:34.935+0000 7fb591e89640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb58c0c9a20 0x7fb58c150870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:34.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:34.935+0000 7fb591e89640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fb58c150db0 0x7fb58c0c5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:34.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:34.935+0000 7fb591e89640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb58c1535d0 con 0x7fb58c0aeb20 2024-04-25T00:11:34.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:34.935+0000 7fb591e89640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fb58c153850 con 0x7fb58c0c9a20 2024-04-25T00:11:34.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:34.935+0000 7fb591e89640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fb58c153b30 con 0x7fb58c150db0 2024-04-25T00:11:34.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:34.935+0000 7fb58affd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb58c0c9a20 0x7fb58c150870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:34.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:34.935+0000 7fb58affd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb58c0c9a20 0x7fb58c150870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53912/0 (socket says 172.21.15.116:53912) 2024-04-25T00:11:34.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:34.935+0000 7fb58affd640 1 -- 172.21.15.116:0/78315458 learned_addr learned my addr 172.21.15.116:0/78315458 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:11:34.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:34.935+0000 7fb58bfff640 1 --2- 172.21.15.116:0/78315458 >> v2:172.21.15.174:3300/0 conn(0x7fb58c150db0 0x7fb58c0c5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:34.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:34.935+0000 7fb58affd640 1 -- 172.21.15.116:0/78315458 >> v2:172.21.15.174:3300/0 conn(0x7fb58c150db0 msgr2=0x7fb58c0c5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:11:34.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:34.935+0000 7fb58affd640 1 --2- 172.21.15.116:0/78315458 >> v2:172.21.15.174:3300/0 conn(0x7fb58c150db0 0x7fb58c0c5270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:34.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:34.935+0000 7fb58affd640 1 -- 172.21.15.116:0/78315458 >> v1:172.21.15.174:6789/0 conn(0x7fb58c0aeb20 legacy=0x7fb58c0c9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:11:34.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:34.935+0000 7fb58affd640 1 -- 172.21.15.116:0/78315458 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb58c153e10 con 0x7fb58c0c9a20 2024-04-25T00:11:34.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:34.939+0000 7fb58affd640 1 --2- 172.21.15.116:0/78315458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb58c0c9a20 0x7fb58c150870 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fb58c0af340 tx=0x7fb58001c240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5387de60ac1305ac server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:11:34.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:34.939+0000 7fb58a7fc640 1 -- 172.21.15.116:0/78315458 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5800b0070 con 0x7fb58c0c9a20 2024-04-25T00:11:34.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:34.939+0000 7fb58a7fc640 1 -- 172.21.15.116:0/78315458 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5800ab600 con 0x7fb58c0c9a20 2024-04-25T00:11:34.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:34.939+0000 7fb58a7fc640 1 -- 172.21.15.116:0/78315458 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5800ab9d0 con 0x7fb58c0c9a20 2024-04-25T00:11:34.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:34.939+0000 7fb591e89640 1 -- 172.21.15.116:0/78315458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb58c0c9a20 msgr2=0x7fb58c150870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:34.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:34.939+0000 7fb591e89640 1 --2- 172.21.15.116:0/78315458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb58c0c9a20 0x7fb58c150870 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fb58c0af340 tx=0x7fb58001c240 comp rx=0 tx=0).stop 2024-04-25T00:11:34.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:34.939+0000 7fb591e89640 1 -- 172.21.15.116:0/78315458 shutdown_connections 2024-04-25T00:11:34.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:34.939+0000 7fb591e89640 1 --2- 172.21.15.116:0/78315458 >> v2:172.21.15.174:3300/0 conn(0x7fb58c150db0 0x7fb58c0c5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:34.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:34.939+0000 7fb591e89640 1 --2- 172.21.15.116:0/78315458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb58c0c9a20 0x7fb58c150870 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:34.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:34.939+0000 7fb591e89640 1 -- 172.21.15.116:0/78315458 >> 172.21.15.116:0/78315458 conn(0x7fb58c074650 msgr2=0x7fb58c074a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:11:34.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:34.939+0000 7fb591e89640 1 -- 172.21.15.116:0/78315458 shutdown_connections 2024-04-25T00:11:34.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:34.939+0000 7fb591e89640 1 -- 172.21.15.116:0/78315458 wait complete. 2024-04-25T00:11:34.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:34.939+0000 7fb591e89640 1 Processor -- start 2024-04-25T00:11:34.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:34.943+0000 7fb591e89640 1 -- start start 2024-04-25T00:11:34.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:34.943+0000 7fb591e89640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb58c0aeb20 0x7fb58c14c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:34.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:34.943+0000 7fb591e89640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb58c0c9a20 0x7fb58c1f1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:34.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:34.943+0000 7fb591e89640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb58c150db0 0x7fb58c1f1770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:34.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:34.943+0000 7fb591e89640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb58c0af080 con 0x7fb58c0aeb20 2024-04-25T00:11:34.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:34.943+0000 7fb591e89640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb58c0aef00 con 0x7fb58c150db0 2024-04-25T00:11:34.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:34.943+0000 7fb591e89640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fb58c15ff60 con 0x7fb58c0c9a20 2024-04-25T00:11:34.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:34.943+0000 7fb58b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb58c0aeb20 0x7fb58c14c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:34.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:34.943+0000 7fb58b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb58c0aeb20 0x7fb58c14c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53928/0 (socket says 172.21.15.116:53928) 2024-04-25T00:11:34.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:34.943+0000 7fb58b7fe640 1 -- 172.21.15.116:0/2832291219 learned_addr learned my addr 172.21.15.116:0/2832291219 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:11:34.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:34.943+0000 7fb58bfff640 1 --2- 172.21.15.116:0/2832291219 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb58c150db0 0x7fb58c1f1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:34.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:34.943+0000 7fb58affd640 1 --2- 172.21.15.116:0/2832291219 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb58c0c9a20 0x7fb58c1f1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:34.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:34.943+0000 7fb58b7fe640 1 -- 172.21.15.116:0/2832291219 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb58c0c9a20 msgr2=0x7fb58c1f1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:34.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:34.943+0000 7fb58b7fe640 1 --2- 172.21.15.116:0/2832291219 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb58c0c9a20 0x7fb58c1f1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:34.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:34.943+0000 7fb58b7fe640 1 -- 172.21.15.116:0/2832291219 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb58c150db0 msgr2=0x7fb58c1f1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:34.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:34.943+0000 7fb58b7fe640 1 --2- 172.21.15.116:0/2832291219 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb58c150db0 0x7fb58c1f1770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:34.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:34.943+0000 7fb58b7fe640 1 -- 172.21.15.116:0/2832291219 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb58c0add40 con 0x7fb58c0aeb20 2024-04-25T00:11:34.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:34.943+0000 7fb58b7fe640 1 --2- 172.21.15.116:0/2832291219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb58c0aeb20 0x7fb58c14c630 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fb57c00c480 tx=0x7fb57c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:11:34.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:34.943+0000 7fb573fff640 1 -- 172.21.15.116:0/2832291219 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb57c012830 con 0x7fb58c0aeb20 2024-04-25T00:11:34.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:34.943+0000 7fb573fff640 1 -- 172.21.15.116:0/2832291219 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb57c017070 con 0x7fb58c0aeb20 2024-04-25T00:11:34.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:34.943+0000 7fb591e89640 1 -- 172.21.15.116:0/2832291219 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb58c1f1cb0 con 0x7fb58c0aeb20 2024-04-25T00:11:34.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:34.943+0000 7fb573fff640 1 -- 172.21.15.116:0/2832291219 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb57c012c30 con 0x7fb58c0aeb20 2024-04-25T00:11:34.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:34.943+0000 7fb591e89640 1 -- 172.21.15.116:0/2832291219 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb58c14a850 con 0x7fb58c0aeb20 2024-04-25T00:11:34.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:34.943+0000 7fb591e89640 1 -- 172.21.15.116:0/2832291219 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb58c068a60 con 0x7fb58c0aeb20 2024-04-25T00:11:34.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:34.943+0000 7fb573fff640 1 -- 172.21.15.116:0/2832291219 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb57c01c460 con 0x7fb58c0aeb20 2024-04-25T00:11:34.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:34.943+0000 7fb573fff640 1 --2- 172.21.15.116:0/2832291219 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb5780678c0 0x7fb578069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:34.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:34.947+0000 7fb573fff640 1 -- 172.21.15.116:0/2832291219 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 7784+0+0 (secure 0 0 0) 0x7fb57c086200 con 0x7fb58c0aeb20 2024-04-25T00:11:34.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:34.947+0000 7fb58affd640 1 --2- 172.21.15.116:0/2832291219 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb5780678c0 0x7fb578069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:34.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:34.947+0000 7fb58affd640 1 --2- 172.21.15.116:0/2832291219 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb5780678c0 0x7fb578069d80 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fb5800ad000 tx=0x7fb580008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:11:34.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:34.951+0000 7fb573fff640 1 -- 172.21.15.116:0/2832291219 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fb57c0565b0 con 0x7fb58c0aeb20 2024-04-25T00:11:35.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.179+0000 7fb591e89640 1 -- 172.21.15.116:0/2832291219 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb58c06db40 con 0x7fb58c0aeb20 2024-04-25T00:11:35.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.183+0000 7fb573fff640 1 -- 172.21.15.116:0/2832291219 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) ==== 94+0+6504 (secure 0 0 0) 0x7fb57c055eb0 con 0x7fb58c0aeb20 2024-04-25T00:11:35.183 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:11:35.183 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5461,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72},{"gid":6309,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1599708526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1599708526},{"type":"v1","addr":"172.21.15.116:6835","nonce":1599708526}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:11:03.640578+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6114},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6114":{"gid":6114,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":25,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6114,"qdb_cluster":[6114]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:09:33.601524+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":132,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4235,"qdb_cluster":[4235]},"id":2}]} 2024-04-25T00:11:35.183 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 77 2024-04-25T00:11:35.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.187+0000 7fb591e89640 1 -- 172.21.15.116:0/2832291219 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb5780678c0 msgr2=0x7fb578069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:35.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.187+0000 7fb591e89640 1 --2- 172.21.15.116:0/2832291219 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb5780678c0 0x7fb578069d80 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fb5800ad000 tx=0x7fb580008040 comp rx=0 tx=0).stop 2024-04-25T00:11:35.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.187+0000 7fb591e89640 1 -- 172.21.15.116:0/2832291219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb58c0aeb20 msgr2=0x7fb58c14c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:35.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.187+0000 7fb591e89640 1 --2- 172.21.15.116:0/2832291219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb58c0aeb20 0x7fb58c14c630 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fb57c00c480 tx=0x7fb57c00c940 comp rx=0 tx=0).stop 2024-04-25T00:11:35.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.187+0000 7fb591e89640 1 -- 172.21.15.116:0/2832291219 shutdown_connections 2024-04-25T00:11:35.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.187+0000 7fb591e89640 1 --2- 172.21.15.116:0/2832291219 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb5780678c0 0x7fb578069d80 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:35.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.187+0000 7fb591e89640 1 --2- 172.21.15.116:0/2832291219 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb58c150db0 0x7fb58c1f1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:35.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.187+0000 7fb591e89640 1 --2- 172.21.15.116:0/2832291219 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb58c0c9a20 0x7fb58c1f1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:35.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.187+0000 7fb591e89640 1 --2- 172.21.15.116:0/2832291219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb58c0aeb20 0x7fb58c14c630 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:35.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.187+0000 7fb591e89640 1 -- 172.21.15.116:0/2832291219 >> 172.21.15.116:0/2832291219 conn(0x7fb58c074650 msgr2=0x7fb58c147740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:11:35.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.187+0000 7fb591e89640 1 -- 172.21.15.116:0/2832291219 shutdown_connections 2024-04-25T00:11:35.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.187+0000 7fb591e89640 1 -- 172.21.15.116:0/2832291219 wait complete. 2024-04-25T00:11:35.205 INFO:tasks.mds_thrash.fs.[b]:kill mds.d (rank=0) 2024-04-25T00:11:35.206 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-25T00:11:35.206 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:11:35.225 INFO:tasks.ceph.mds.d:Stopped 2024-04-25T00:11:35.225 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:11:35.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.359+0000 7f6e0c1cd640 1 Processor -- start 2024-04-25T00:11:35.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.359+0000 7f6e0c1cd640 1 -- start start 2024-04-25T00:11:35.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.359+0000 7f6e0c1cd640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f6e0414c700 0x7f6e040c5090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:35.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.359+0000 7f6e0c1cd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6e04155b20 0x7f6e04152c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:35.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.359+0000 7f6e0c1cd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6e040c55d0 0x7f6e04153180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:35.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.359+0000 7f6e0c1cd640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f6e041527b0 con 0x7f6e04155b20 2024-04-25T00:11:35.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.359+0000 7f6e0c1cd640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f6e040a9e60 con 0x7f6e0414c700 2024-04-25T00:11:35.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.359+0000 7f6e0c1cd640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6e040c3ef0 con 0x7f6e040c55d0 2024-04-25T00:11:35.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.359+0000 7f6e09768640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6e04155b20 0x7f6e04152c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:35.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.359+0000 7f6e09768640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6e04155b20 0x7f6e04152c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53932/0 (socket says 172.21.15.116:53932) 2024-04-25T00:11:35.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.359+0000 7f6e09768640 1 -- 172.21.15.116:0/1324303231 learned_addr learned my addr 172.21.15.116:0/1324303231 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:11:35.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.359+0000 7f6e0a76a640 1 --2- 172.21.15.116:0/1324303231 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6e040c55d0 0x7f6e04153180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:35.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.359+0000 7f6e09f69640 1 --2- 172.21.15.116:0/1324303231 >> v2:172.21.15.174:3300/0 conn(0x7f6e0414c700 0x7f6e040c5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:35.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.359+0000 7f6e09768640 1 -- 172.21.15.116:0/1324303231 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6e040c55d0 msgr2=0x7f6e04153180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:11:35.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.359+0000 7f6e09768640 1 --2- 172.21.15.116:0/1324303231 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6e040c55d0 0x7f6e04153180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:35.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.359+0000 7f6e09768640 1 -- 172.21.15.116:0/1324303231 >> v2:172.21.15.174:3300/0 conn(0x7f6e0414c700 msgr2=0x7f6e040c5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:11:35.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.359+0000 7f6e09768640 1 --2- 172.21.15.116:0/1324303231 >> v2:172.21.15.174:3300/0 conn(0x7f6e0414c700 0x7f6e040c5090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:35.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.359+0000 7f6e09768640 1 -- 172.21.15.116:0/1324303231 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e040c41d0 con 0x7f6e04155b20 2024-04-25T00:11:35.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.363+0000 7f6e09768640 1 --2- 172.21.15.116:0/1324303231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e04155b20 0x7f6e04152c40 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f6e040c90b0 tx=0x7f6df401c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b09f7bc32a7151ba server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:11:35.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.363+0000 7f6e08f67640 1 -- 172.21.15.116:0/1324303231 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6df40b0070 con 0x7f6e04155b20 2024-04-25T00:11:35.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.363+0000 7f6e08f67640 1 -- 172.21.15.116:0/1324303231 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6df40ab600 con 0x7f6e04155b20 2024-04-25T00:11:35.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.363+0000 7f6e08f67640 1 -- 172.21.15.116:0/1324303231 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6df40ab9d0 con 0x7f6e04155b20 2024-04-25T00:11:35.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.363+0000 7f6e0c1cd640 1 -- 172.21.15.116:0/1324303231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e04155b20 msgr2=0x7f6e04152c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:35.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.363+0000 7f6e0c1cd640 1 --2- 172.21.15.116:0/1324303231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e04155b20 0x7f6e04152c40 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f6e040c90b0 tx=0x7f6df401c400 comp rx=0 tx=0).stop 2024-04-25T00:11:35.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.363+0000 7f6e0c1cd640 1 -- 172.21.15.116:0/1324303231 shutdown_connections 2024-04-25T00:11:35.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.363+0000 7f6e0c1cd640 1 --2- 172.21.15.116:0/1324303231 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6e040c55d0 0x7f6e04153180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:35.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.363+0000 7f6e0c1cd640 1 --2- 172.21.15.116:0/1324303231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e04155b20 0x7f6e04152c40 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:35.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.363+0000 7f6e0c1cd640 1 --2- 172.21.15.116:0/1324303231 >> v2:172.21.15.174:3300/0 conn(0x7f6e0414c700 0x7f6e040c5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:35.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.363+0000 7f6e0c1cd640 1 -- 172.21.15.116:0/1324303231 >> 172.21.15.116:0/1324303231 conn(0x7f6e0406c720 msgr2=0x7f6e0406cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:11:35.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.363+0000 7f6e0c1cd640 1 -- 172.21.15.116:0/1324303231 shutdown_connections 2024-04-25T00:11:35.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.363+0000 7f6e0c1cd640 1 -- 172.21.15.116:0/1324303231 wait complete. 2024-04-25T00:11:35.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.367+0000 7f6e0c1cd640 1 Processor -- start 2024-04-25T00:11:35.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.367+0000 7f6e0c1cd640 1 -- start start 2024-04-25T00:11:35.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.367+0000 7f6e0c1cd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6e040c55d0 0x7f6e0414b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:35.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.367+0000 7f6e0c1cd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e0414c700 0x7f6e04147a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:35.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.367+0000 7f6e0c1cd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6e04155b20 0x7f6e04147fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:35.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.367+0000 7f6e0c1cd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6e04072990 con 0x7f6e0414c700 2024-04-25T00:11:35.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.367+0000 7f6e0c1cd640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6e04072810 con 0x7f6e040c55d0 2024-04-25T00:11:35.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.367+0000 7f6e0c1cd640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6e04072b10 con 0x7f6e04155b20 2024-04-25T00:11:35.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.367+0000 7f6e09768640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e0414c700 0x7f6e04147a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:35.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.367+0000 7f6e09768640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e0414c700 0x7f6e04147a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53942/0 (socket says 172.21.15.116:53942) 2024-04-25T00:11:35.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.367+0000 7f6e09768640 1 -- 172.21.15.116:0/2072759829 learned_addr learned my addr 172.21.15.116:0/2072759829 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:11:35.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.367+0000 7f6e09f69640 1 --2- 172.21.15.116:0/2072759829 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6e040c55d0 0x7f6e0414b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:35.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.367+0000 7f6e0a76a640 1 --2- 172.21.15.116:0/2072759829 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6e04155b20 0x7f6e04147fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:35.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.367+0000 7f6e09768640 1 -- 172.21.15.116:0/2072759829 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6e04155b20 msgr2=0x7f6e04147fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:35.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.367+0000 7f6e09768640 1 --2- 172.21.15.116:0/2072759829 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6e04155b20 0x7f6e04147fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:35.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.367+0000 7f6e09768640 1 -- 172.21.15.116:0/2072759829 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6e040c55d0 msgr2=0x7f6e0414b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:35.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.367+0000 7f6e09768640 1 --2- 172.21.15.116:0/2072759829 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6e040c55d0 0x7f6e0414b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:35.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.367+0000 7f6e09768640 1 -- 172.21.15.116:0/2072759829 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e040ac420 con 0x7f6e0414c700 2024-04-25T00:11:35.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.367+0000 7f6e0a76a640 1 --2- 172.21.15.116:0/2072759829 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6e04155b20 0x7f6e04147fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:11:35.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.367+0000 7f6e09768640 1 --2- 172.21.15.116:0/2072759829 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e0414c700 0x7f6e04147a70 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f6df401cca0 tx=0x7f6df40191a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:11:35.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.367+0000 7f6dfa7fc640 1 -- 172.21.15.116:0/2072759829 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6df40bb020 con 0x7f6e0414c700 2024-04-25T00:11:35.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.367+0000 7f6dfa7fc640 1 -- 172.21.15.116:0/2072759829 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6df40ace20 con 0x7f6e0414c700 2024-04-25T00:11:35.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.367+0000 7f6e0c1cd640 1 -- 172.21.15.116:0/2072759829 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e041f1f20 con 0x7f6e0414c700 2024-04-25T00:11:35.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.367+0000 7f6dfa7fc640 1 -- 172.21.15.116:0/2072759829 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6df40b0040 con 0x7f6e0414c700 2024-04-25T00:11:35.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.367+0000 7f6e0c1cd640 1 -- 172.21.15.116:0/2072759829 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e040ac6e0 con 0x7f6e0414c700 2024-04-25T00:11:35.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.367+0000 7f6e0c1cd640 1 -- 172.21.15.116:0/2072759829 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e04068a60 con 0x7f6e0414c700 2024-04-25T00:11:35.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.367+0000 7f6dfa7fc640 1 -- 172.21.15.116:0/2072759829 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6df40be020 con 0x7f6e0414c700 2024-04-25T00:11:35.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.371+0000 7f6dfa7fc640 1 --2- 172.21.15.116:0/2072759829 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6de80678c0 0x7f6de8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:35.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.371+0000 7f6dfa7fc640 1 -- 172.21.15.116:0/2072759829 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 7784+0+0 (secure 0 0 0) 0x7f6df4128350 con 0x7f6e0414c700 2024-04-25T00:11:35.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.371+0000 7f6e09f69640 1 --2- 172.21.15.116:0/2072759829 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6de80678c0 0x7f6de8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:35.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.371+0000 7f6e09f69640 1 --2- 172.21.15.116:0/2072759829 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6de80678c0 0x7f6de8069d80 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f6e00003ab0 tx=0x7f6e00009830 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:11:35.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.375+0000 7f6dfa7fc640 1 -- 172.21.15.116:0/2072759829 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f6df4016110 con 0x7f6e0414c700 2024-04-25T00:11:35.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.607+0000 7f6e0c1cd640 1 -- 172.21.15.116:0/2072759829 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6e040aa0d0 con 0x7f6e0414c700 2024-04-25T00:11:35.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.611+0000 7f6dfa7fc640 1 -- 172.21.15.116:0/2072759829 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) ==== 94+0+6504 (secure 0 0 0) 0x7f6df40f0660 con 0x7f6e0414c700 2024-04-25T00:11:35.608 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:11:35.609 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5461,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72},{"gid":6309,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1599708526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1599708526},{"type":"v1","addr":"172.21.15.116:6835","nonce":1599708526}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:11:03.640578+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6114},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6114":{"gid":6114,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":25,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6114,"qdb_cluster":[6114]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:09:33.601524+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":132,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4235,"qdb_cluster":[4235]},"id":2}]} 2024-04-25T00:11:35.609 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 77 2024-04-25T00:11:35.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.611+0000 7f6e0c1cd640 1 -- 172.21.15.116:0/2072759829 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6de80678c0 msgr2=0x7f6de8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:35.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.611+0000 7f6e0c1cd640 1 --2- 172.21.15.116:0/2072759829 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6de80678c0 0x7f6de8069d80 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f6e00003ab0 tx=0x7f6e00009830 comp rx=0 tx=0).stop 2024-04-25T00:11:35.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.611+0000 7f6e0c1cd640 1 -- 172.21.15.116:0/2072759829 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e0414c700 msgr2=0x7f6e04147a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:35.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.611+0000 7f6e0c1cd640 1 --2- 172.21.15.116:0/2072759829 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e0414c700 0x7f6e04147a70 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f6df401cca0 tx=0x7f6df40191a0 comp rx=0 tx=0).stop 2024-04-25T00:11:35.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.611+0000 7f6e0c1cd640 1 -- 172.21.15.116:0/2072759829 shutdown_connections 2024-04-25T00:11:35.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.611+0000 7f6e0c1cd640 1 --2- 172.21.15.116:0/2072759829 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6de80678c0 0x7f6de8069d80 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:35.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.611+0000 7f6e0c1cd640 1 --2- 172.21.15.116:0/2072759829 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6e04155b20 0x7f6e04147fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:35.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.611+0000 7f6e0c1cd640 1 --2- 172.21.15.116:0/2072759829 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6e0414c700 0x7f6e04147a70 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:35.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.611+0000 7f6e0c1cd640 1 --2- 172.21.15.116:0/2072759829 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6e040c55d0 0x7f6e0414b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:35.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.611+0000 7f6e0c1cd640 1 -- 172.21.15.116:0/2072759829 >> 172.21.15.116:0/2072759829 conn(0x7f6e0406c720 msgr2=0x7f6e040c3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:11:35.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.611+0000 7f6e0c1cd640 1 -- 172.21.15.116:0/2072759829 shutdown_connections 2024-04-25T00:11:35.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:35.611+0000 7f6e0c1cd640 1 -- 172.21.15.116:0/2072759829 wait complete. 2024-04-25T00:11:35.629 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:11:37.271 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:11:37.274 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:11:37.630 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:11:37.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.774+0000 7f47c2f08640 1 Processor -- start 2024-04-25T00:11:37.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.774+0000 7f47c2f08640 1 -- start start 2024-04-25T00:11:37.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.774+0000 7f47c2f08640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f47bc0acef0 0x7f47bc1541d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:37.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.774+0000 7f47c2f08640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f47bc0ad2d0 0x7f47bc0a7d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:37.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.774+0000 7f47c2f08640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f47bc0a82c0 0x7f47bc0c32d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:37.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.774+0000 7f47c2f08640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f47bc0c2b60 con 0x7f47bc0a82c0 2024-04-25T00:11:37.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.774+0000 7f47c2f08640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f47bc0c2e40 con 0x7f47bc0acef0 2024-04-25T00:11:37.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.774+0000 7f47c2f08640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f47bc0ad750 con 0x7f47bc0ad2d0 2024-04-25T00:11:37.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.774+0000 7f47c14a5640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f47bc0a82c0 0x7f47bc0c32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:37.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.774+0000 7f47c14a5640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f47bc0a82c0 0x7f47bc0c32d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53954/0 (socket says 172.21.15.116:53954) 2024-04-25T00:11:37.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.774+0000 7f47c14a5640 1 -- 172.21.15.116:0/1954389084 learned_addr learned my addr 172.21.15.116:0/1954389084 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:11:37.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.774+0000 7f47bbfff640 1 --2- 172.21.15.116:0/1954389084 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f47bc0ad2d0 0x7f47bc0a7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:37.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.774+0000 7f47c14a5640 1 -- 172.21.15.116:0/1954389084 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f47bc0ad2d0 msgr2=0x7f47bc0a7d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:11:37.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.774+0000 7f47c14a5640 1 --2- 172.21.15.116:0/1954389084 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f47bc0ad2d0 0x7f47bc0a7d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:37.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.774+0000 7f47c14a5640 1 -- 172.21.15.116:0/1954389084 >> v2:172.21.15.174:3300/0 conn(0x7f47bc0acef0 msgr2=0x7f47bc1541d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:11:37.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.774+0000 7f47c0ca4640 1 --2- 172.21.15.116:0/1954389084 >> v2:172.21.15.174:3300/0 conn(0x7f47bc0acef0 0x7f47bc1541d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:37.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.774+0000 7f47c14a5640 1 --2- 172.21.15.116:0/1954389084 >> v2:172.21.15.174:3300/0 conn(0x7f47bc0acef0 0x7f47bc1541d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:37.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.774+0000 7f47c14a5640 1 -- 172.21.15.116:0/1954389084 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47bc0a9ef0 con 0x7f47bc0a82c0 2024-04-25T00:11:37.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.774+0000 7f47bbfff640 1 --2- 172.21.15.116:0/1954389084 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f47bc0ad2d0 0x7f47bc0a7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:11:37.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.774+0000 7f47c0ca4640 1 --2- 172.21.15.116:0/1954389084 >> v2:172.21.15.174:3300/0 conn(0x7f47bc0acef0 0x7f47bc1541d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:11:37.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.778+0000 7f47c14a5640 1 --2- 172.21.15.116:0/1954389084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47bc0a82c0 0x7f47bc0c32d0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f47bc0c8700 tx=0x7f47b401c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af02a7e972b3c6b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:11:37.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.778+0000 7f47bb7fe640 1 -- 172.21.15.116:0/1954389084 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47b40a56f0 con 0x7f47bc0a82c0 2024-04-25T00:11:37.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.778+0000 7f47bb7fe640 1 -- 172.21.15.116:0/1954389084 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47b40a5890 con 0x7f47bc0a82c0 2024-04-25T00:11:37.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.778+0000 7f47bb7fe640 1 -- 172.21.15.116:0/1954389084 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47b40a5c60 con 0x7f47bc0a82c0 2024-04-25T00:11:37.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.778+0000 7f47c2f08640 1 -- 172.21.15.116:0/1954389084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47bc0a82c0 msgr2=0x7f47bc0c32d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:37.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.778+0000 7f47c2f08640 1 --2- 172.21.15.116:0/1954389084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47bc0a82c0 0x7f47bc0c32d0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f47bc0c8700 tx=0x7f47b401c960 comp rx=0 tx=0).stop 2024-04-25T00:11:37.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.778+0000 7f47c2f08640 1 -- 172.21.15.116:0/1954389084 shutdown_connections 2024-04-25T00:11:37.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.778+0000 7f47c2f08640 1 --2- 172.21.15.116:0/1954389084 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47bc0a82c0 0x7f47bc0c32d0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:37.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.778+0000 7f47c2f08640 1 --2- 172.21.15.116:0/1954389084 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f47bc0ad2d0 0x7f47bc0a7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:37.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.778+0000 7f47c2f08640 1 --2- 172.21.15.116:0/1954389084 >> v2:172.21.15.174:3300/0 conn(0x7f47bc0acef0 0x7f47bc1541d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:37.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.778+0000 7f47c2f08640 1 -- 172.21.15.116:0/1954389084 >> 172.21.15.116:0/1954389084 conn(0x7f47bc0c4200 msgr2=0x7f47bc0c4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:11:37.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.778+0000 7f47c2f08640 1 -- 172.21.15.116:0/1954389084 shutdown_connections 2024-04-25T00:11:37.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.778+0000 7f47c2f08640 1 -- 172.21.15.116:0/1954389084 wait complete. 2024-04-25T00:11:37.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.782+0000 7f47c2f08640 1 Processor -- start 2024-04-25T00:11:37.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.782+0000 7f47c2f08640 1 -- start start 2024-04-25T00:11:37.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.782+0000 7f47c2f08640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f47bc0a82c0 0x7f47bc153120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:37.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.782+0000 7f47c2f08640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f47bc0acef0 0x7f47bc0727b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:37.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.782+0000 7f47c2f08640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47bc0ad2d0 0x7f47bc145800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:37.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.782+0000 7f47c2f08640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f47bc156e90 con 0x7f47bc0ad2d0 2024-04-25T00:11:37.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.782+0000 7f47c2f08640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f47bc156d10 con 0x7f47bc0a82c0 2024-04-25T00:11:37.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.782+0000 7f47c2f08640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f47bc157010 con 0x7f47bc0acef0 2024-04-25T00:11:37.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.782+0000 7f47c14a5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47bc0ad2d0 0x7f47bc145800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:37.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.782+0000 7f47c14a5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47bc0ad2d0 0x7f47bc145800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53964/0 (socket says 172.21.15.116:53964) 2024-04-25T00:11:37.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.782+0000 7f47c14a5640 1 -- 172.21.15.116:0/2574011085 learned_addr learned my addr 172.21.15.116:0/2574011085 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:11:37.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.782+0000 7f47c0ca4640 1 --2- 172.21.15.116:0/2574011085 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f47bc0a82c0 0x7f47bc153120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:37.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.782+0000 7f47bbfff640 1 --2- 172.21.15.116:0/2574011085 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f47bc0acef0 0x7f47bc0727b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:37.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.782+0000 7f47c14a5640 1 -- 172.21.15.116:0/2574011085 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f47bc0acef0 msgr2=0x7f47bc0727b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:37.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.782+0000 7f47c14a5640 1 --2- 172.21.15.116:0/2574011085 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f47bc0acef0 0x7f47bc0727b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:37.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.782+0000 7f47c14a5640 1 -- 172.21.15.116:0/2574011085 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f47bc0a82c0 msgr2=0x7f47bc153120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:37.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.782+0000 7f47c14a5640 1 --2- 172.21.15.116:0/2574011085 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f47bc0a82c0 0x7f47bc153120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:37.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.782+0000 7f47c14a5640 1 -- 172.21.15.116:0/2574011085 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47bc1f1a50 con 0x7f47bc0ad2d0 2024-04-25T00:11:37.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.782+0000 7f47c0ca4640 1 --2- 172.21.15.116:0/2574011085 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f47bc0a82c0 0x7f47bc153120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:11:37.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.782+0000 7f47bbfff640 1 --2- 172.21.15.116:0/2574011085 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f47bc0acef0 0x7f47bc0727b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:11:37.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.782+0000 7f47c14a5640 1 --2- 172.21.15.116:0/2574011085 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47bc0ad2d0 0x7f47bc145800 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f47b40af040 tx=0x7f47b4019700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:11:37.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.782+0000 7f47b97fa640 1 -- 172.21.15.116:0/2574011085 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47b40bc020 con 0x7f47bc0ad2d0 2024-04-25T00:11:37.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.782+0000 7f47b97fa640 1 -- 172.21.15.116:0/2574011085 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47b40b8670 con 0x7f47bc0ad2d0 2024-04-25T00:11:37.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.782+0000 7f47c2f08640 1 -- 172.21.15.116:0/2574011085 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47bc1f1d30 con 0x7f47bc0ad2d0 2024-04-25T00:11:37.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.782+0000 7f47b97fa640 1 -- 172.21.15.116:0/2574011085 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47b40b8a60 con 0x7f47bc0ad2d0 2024-04-25T00:11:37.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.782+0000 7f47c2f08640 1 -- 172.21.15.116:0/2574011085 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47bc1f22f0 con 0x7f47bc0ad2d0 2024-04-25T00:11:37.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.782+0000 7f47c2f08640 1 -- 172.21.15.116:0/2574011085 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47bc1f2530 con 0x7f47bc0ad2d0 2024-04-25T00:11:37.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.786+0000 7f47b97fa640 1 -- 172.21.15.116:0/2574011085 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f47b40bf020 con 0x7f47bc0ad2d0 2024-04-25T00:11:37.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.786+0000 7f47b97fa640 1 --2- 172.21.15.116:0/2574011085 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f47a00678c0 0x7f47a0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:37.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.786+0000 7f47b97fa640 1 -- 172.21.15.116:0/2574011085 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 7784+0+0 (secure 0 0 0) 0x7f47b4121690 con 0x7f47bc0ad2d0 2024-04-25T00:11:37.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.786+0000 7f47c0ca4640 1 --2- 172.21.15.116:0/2574011085 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f47a00678c0 0x7f47a0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:37.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.786+0000 7f47c0ca4640 1 --2- 172.21.15.116:0/2574011085 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f47a00678c0 0x7f47a0069d80 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f47b0006fd0 tx=0x7f47b0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:11:37.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:37.790+0000 7f47b97fa640 1 -- 172.21.15.116:0/2574011085 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f47bc1f2530 con 0x7f47bc0ad2d0 2024-04-25T00:11:38.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:38.006+0000 7f47c2f08640 1 -- 172.21.15.116:0/2574011085 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f47bc068a60 con 0x7f47bc0ad2d0 2024-04-25T00:11:38.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:38.010+0000 7f47b97fa640 1 -- 172.21.15.116:0/2574011085 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) ==== 94+0+6504 (secure 0 0 0) 0x7f47b40c6030 con 0x7f47bc0ad2d0 2024-04-25T00:11:38.009 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:11:38.009 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5461,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72},{"gid":6309,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1599708526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1599708526},{"type":"v1","addr":"172.21.15.116:6835","nonce":1599708526}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:11:03.640578+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6114},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6114":{"gid":6114,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":25,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6114,"qdb_cluster":[6114]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:09:33.601524+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":132,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4235,"qdb_cluster":[4235]},"id":2}]} 2024-04-25T00:11:38.009 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 77 2024-04-25T00:11:38.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:38.010+0000 7f47c2f08640 1 -- 172.21.15.116:0/2574011085 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f47a00678c0 msgr2=0x7f47a0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:38.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:38.010+0000 7f47c2f08640 1 --2- 172.21.15.116:0/2574011085 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f47a00678c0 0x7f47a0069d80 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f47b0006fd0 tx=0x7f47b0008040 comp rx=0 tx=0).stop 2024-04-25T00:11:38.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:38.010+0000 7f47c2f08640 1 -- 172.21.15.116:0/2574011085 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47bc0ad2d0 msgr2=0x7f47bc145800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:38.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:38.010+0000 7f47c2f08640 1 --2- 172.21.15.116:0/2574011085 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47bc0ad2d0 0x7f47bc145800 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f47b40af040 tx=0x7f47b4019700 comp rx=0 tx=0).stop 2024-04-25T00:11:38.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:38.010+0000 7f47c2f08640 1 -- 172.21.15.116:0/2574011085 shutdown_connections 2024-04-25T00:11:38.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:38.010+0000 7f47c2f08640 1 --2- 172.21.15.116:0/2574011085 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f47a00678c0 0x7f47a0069d80 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:38.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:38.010+0000 7f47c2f08640 1 --2- 172.21.15.116:0/2574011085 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f47bc0ad2d0 0x7f47bc145800 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:38.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:38.010+0000 7f47c2f08640 1 --2- 172.21.15.116:0/2574011085 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f47bc0acef0 0x7f47bc0727b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:38.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:38.010+0000 7f47c2f08640 1 --2- 172.21.15.116:0/2574011085 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f47bc0a82c0 0x7f47bc153120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:38.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:38.010+0000 7f47c2f08640 1 -- 172.21.15.116:0/2574011085 >> 172.21.15.116:0/2574011085 conn(0x7f47bc0c4200 msgr2=0x7f47bc155fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:11:38.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:38.010+0000 7f47c2f08640 1 -- 172.21.15.116:0/2574011085 shutdown_connections 2024-04-25T00:11:38.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:38.010+0000 7f47c2f08640 1 -- 172.21.15.116:0/2574011085 wait complete. 2024-04-25T00:11:38.030 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:11:40.031 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:11:40.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.170+0000 7f1cae2d2640 1 Processor -- start 2024-04-25T00:11:40.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.170+0000 7f1cae2d2640 1 -- start start 2024-04-25T00:11:40.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.174+0000 7f1cae2d2640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1ca8072080 0x7f1ca8159950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:40.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.174+0000 7f1cae2d2640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ca80730b0 con 0x7f1ca81562a0 2024-04-25T00:11:40.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.174+0000 7f1cae2d2640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ca8073390 con 0x7f1ca8152d80 2024-04-25T00:11:40.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.174+0000 7f1cae2d2640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f1ca8072e60 con 0x7f1ca8072080 2024-04-25T00:11:40.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.174+0000 7f1cac86f640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f1ca81562a0 0x7f1ca80c5020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:33106/0 (socket says 172.21.15.116:33106) 2024-04-25T00:11:40.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.174+0000 7f1cac86f640 1 -- 172.21.15.116:0/331788658 learned_addr learned my addr 172.21.15.116:0/331788658 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:11:40.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.174+0000 7f1c9f7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1ca8072080 0x7f1ca8159950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:40.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.174+0000 7f1c9f7fe640 1 -- 172.21.15.116:0/331788658 >> v1:172.21.15.174:6789/0 conn(0x7f1ca8152d80 legacy=0x7f1ca80c36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:11:40.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.174+0000 7f1c9f7fe640 1 -- 172.21.15.116:0/331788658 >> v1:172.21.15.116:6789/0 conn(0x7f1ca81562a0 legacy=0x7f1ca80c5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:40.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.174+0000 7f1ca77fe640 1 -- 172.21.15.116:0/331788658 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3872311480 0 0) 0x7f1ca80730b0 con 0x7f1ca81562a0 2024-04-25T00:11:40.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.174+0000 7f1c9f7fe640 1 -- 172.21.15.116:0/331788658 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ca8158e70 con 0x7f1ca8072080 2024-04-25T00:11:40.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.174+0000 7f1c9f7fe640 1 --2- 172.21.15.116:0/331788658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ca8072080 0x7f1ca8159950 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f1c900097c0 tx=0x7f1c9001c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4774b6c8e146bf59 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:11:40.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.178+0000 7f1ca77fe640 1 -- 172.21.15.116:0/331788658 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c900b1070 con 0x7f1ca8072080 2024-04-25T00:11:40.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.178+0000 7f1ca77fe640 1 -- 172.21.15.116:0/331788658 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c900ac600 con 0x7f1ca8072080 2024-04-25T00:11:40.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.178+0000 7f1ca77fe640 1 -- 172.21.15.116:0/331788658 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c900ac9d0 con 0x7f1ca8072080 2024-04-25T00:11:40.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.178+0000 7f1cae2d2640 1 -- 172.21.15.116:0/331788658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ca8072080 msgr2=0x7f1ca8159950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:40.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.178+0000 7f1cae2d2640 1 --2- 172.21.15.116:0/331788658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ca8072080 0x7f1ca8159950 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f1c900097c0 tx=0x7f1c9001c3a0 comp rx=0 tx=0).stop 2024-04-25T00:11:40.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.178+0000 7f1cae2d2640 1 -- 172.21.15.116:0/331788658 shutdown_connections 2024-04-25T00:11:40.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.178+0000 7f1cae2d2640 1 --2- 172.21.15.116:0/331788658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ca8072080 0x7f1ca8159950 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:40.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.178+0000 7f1cae2d2640 1 -- 172.21.15.116:0/331788658 >> 172.21.15.116:0/331788658 conn(0x7f1ca806c720 msgr2=0x7f1ca806cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:11:40.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.178+0000 7f1cae2d2640 1 -- 172.21.15.116:0/331788658 shutdown_connections 2024-04-25T00:11:40.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.178+0000 7f1cae2d2640 1 -- 172.21.15.116:0/331788658 wait complete. 2024-04-25T00:11:40.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.178+0000 7f1cae2d2640 1 Processor -- start 2024-04-25T00:11:40.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.178+0000 7f1cae2d2640 1 -- start start 2024-04-25T00:11:40.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.178+0000 7f1cae2d2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ca8072080 0x7f1ca80acbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:40.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.178+0000 7f1cae2d2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1ca8152d80 0x7f1ca814e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:40.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.178+0000 7f1cae2d2640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1ca81562a0 0x7f1ca81487e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:40.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.178+0000 7f1cae2d2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1ca80c5730 con 0x7f1ca8072080 2024-04-25T00:11:40.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.178+0000 7f1cae2d2640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1ca81566b0 con 0x7f1ca8152d80 2024-04-25T00:11:40.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.178+0000 7f1cae2d2640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f1ca80c58b0 con 0x7f1ca81562a0 2024-04-25T00:11:40.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.178+0000 7f1ca7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ca8072080 0x7f1ca80acbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:40.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.178+0000 7f1ca7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ca8072080 0x7f1ca80acbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53972/0 (socket says 172.21.15.116:53972) 2024-04-25T00:11:40.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.178+0000 7f1ca7fff640 1 -- 172.21.15.116:0/3456338927 learned_addr learned my addr 172.21.15.116:0/3456338927 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:11:40.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.178+0000 7f1c9f7fe640 1 --2- 172.21.15.116:0/3456338927 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1ca8152d80 0x7f1ca814e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:40.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.178+0000 7f1ca7fff640 1 -- 172.21.15.116:0/3456338927 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1ca81562a0 msgr2=0x7f1ca81487e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:40.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.178+0000 7f1cac86f640 1 --2- 172.21.15.116:0/3456338927 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1ca81562a0 0x7f1ca81487e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:40.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.178+0000 7f1ca7fff640 1 --2- 172.21.15.116:0/3456338927 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1ca81562a0 0x7f1ca81487e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:40.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.178+0000 7f1ca7fff640 1 -- 172.21.15.116:0/3456338927 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1ca8152d80 msgr2=0x7f1ca814e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:40.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.178+0000 7f1ca7fff640 1 --2- 172.21.15.116:0/3456338927 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1ca8152d80 0x7f1ca814e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:40.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.178+0000 7f1ca7fff640 1 -- 172.21.15.116:0/3456338927 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ca814a9f0 con 0x7f1ca8072080 2024-04-25T00:11:40.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.178+0000 7f1ca7fff640 1 --2- 172.21.15.116:0/3456338927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ca8072080 0x7f1ca80acbc0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f1c9800c450 tx=0x7f1c9800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:11:40.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.178+0000 7f1ca57fa640 1 -- 172.21.15.116:0/3456338927 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c98012830 con 0x7f1ca8072080 2024-04-25T00:11:40.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.178+0000 7f1ca57fa640 1 -- 172.21.15.116:0/3456338927 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c98017070 con 0x7f1ca8072080 2024-04-25T00:11:40.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.178+0000 7f1cae2d2640 1 -- 172.21.15.116:0/3456338927 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ca8153af0 con 0x7f1ca8072080 2024-04-25T00:11:40.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.178+0000 7f1ca57fa640 1 -- 172.21.15.116:0/3456338927 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c98012c30 con 0x7f1ca8072080 2024-04-25T00:11:40.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.178+0000 7f1cae2d2640 1 -- 172.21.15.116:0/3456338927 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ca81540b0 con 0x7f1ca8072080 2024-04-25T00:11:40.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.182+0000 7f1cae2d2640 1 -- 172.21.15.116:0/3456338927 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ca8068a60 con 0x7f1ca8072080 2024-04-25T00:11:40.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.182+0000 7f1ca57fa640 1 -- 172.21.15.116:0/3456338927 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1c9801c460 con 0x7f1ca8072080 2024-04-25T00:11:40.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.182+0000 7f1ca57fa640 1 --2- 172.21.15.116:0/3456338927 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1c880678c0 0x7f1c88069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:40.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.182+0000 7f1ca57fa640 1 -- 172.21.15.116:0/3456338927 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 7784+0+0 (secure 0 0 0) 0x7f1c98086290 con 0x7f1ca8072080 2024-04-25T00:11:40.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.182+0000 7f1c9f7fe640 1 --2- 172.21.15.116:0/3456338927 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1c880678c0 0x7f1c88069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:40.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.182+0000 7f1c9f7fe640 1 --2- 172.21.15.116:0/3456338927 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1c880678c0 0x7f1c88069d80 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f1c900ae000 tx=0x7f1c900a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:11:40.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.186+0000 7f1ca57fa640 1 -- 172.21.15.116:0/3456338927 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f1c980565a0 con 0x7f1ca8072080 2024-04-25T00:11:40.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.402+0000 7f1cae2d2640 1 -- 172.21.15.116:0/3456338927 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1ca8160f50 con 0x7f1ca8072080 2024-04-25T00:11:40.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.402+0000 7f1ca57fa640 1 -- 172.21.15.116:0/3456338927 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) ==== 94+0+6504 (secure 0 0 0) 0x7f1c98055ea0 con 0x7f1ca8072080 2024-04-25T00:11:40.402 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:11:40.402 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5461,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72},{"gid":6309,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1599708526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1599708526},{"type":"v1","addr":"172.21.15.116:6835","nonce":1599708526}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:11:03.640578+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6114},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6114":{"gid":6114,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":25,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6114,"qdb_cluster":[6114]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:09:33.601524+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":132,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4235,"qdb_cluster":[4235]},"id":2}]} 2024-04-25T00:11:40.403 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 77 2024-04-25T00:11:40.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.406+0000 7f1cae2d2640 1 -- 172.21.15.116:0/3456338927 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1c880678c0 msgr2=0x7f1c88069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:40.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.406+0000 7f1cae2d2640 1 --2- 172.21.15.116:0/3456338927 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1c880678c0 0x7f1c88069d80 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f1c900ae000 tx=0x7f1c900a7000 comp rx=0 tx=0).stop 2024-04-25T00:11:40.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.406+0000 7f1cae2d2640 1 -- 172.21.15.116:0/3456338927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ca8072080 msgr2=0x7f1ca80acbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:40.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.406+0000 7f1cae2d2640 1 --2- 172.21.15.116:0/3456338927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ca8072080 0x7f1ca80acbc0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f1c9800c450 tx=0x7f1c9800c910 comp rx=0 tx=0).stop 2024-04-25T00:11:40.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.406+0000 7f1cae2d2640 1 -- 172.21.15.116:0/3456338927 shutdown_connections 2024-04-25T00:11:40.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.406+0000 7f1cae2d2640 1 --2- 172.21.15.116:0/3456338927 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1c880678c0 0x7f1c88069d80 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:40.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.406+0000 7f1cae2d2640 1 --2- 172.21.15.116:0/3456338927 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1ca81562a0 0x7f1ca81487e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:40.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.406+0000 7f1cae2d2640 1 --2- 172.21.15.116:0/3456338927 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1ca8152d80 0x7f1ca814e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:40.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.406+0000 7f1cae2d2640 1 --2- 172.21.15.116:0/3456338927 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ca8072080 0x7f1ca80acbc0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:40.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.406+0000 7f1cae2d2640 1 -- 172.21.15.116:0/3456338927 >> 172.21.15.116:0/3456338927 conn(0x7f1ca806c720 msgr2=0x7f1ca80c5020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:11:40.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.406+0000 7f1cae2d2640 1 -- 172.21.15.116:0/3456338927 shutdown_connections 2024-04-25T00:11:40.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:40.406+0000 7f1cae2d2640 1 -- 172.21.15.116:0/3456338927 wait complete. 2024-04-25T00:11:40.423 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:11:42.424 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:11:42.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.558+0000 7fea186fe640 1 Processor -- start 2024-04-25T00:11:42.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.558+0000 7fea186fe640 1 -- start start 2024-04-25T00:11:42.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.558+0000 7fea186fe640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fea10152d60 0x7fea100ab270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:42.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.558+0000 7fea186fe640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea10155c50 con 0x7fea100ab7b0 2024-04-25T00:11:42.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.558+0000 7fea186fe640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea10155f30 con 0x7fea10153140 2024-04-25T00:11:42.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.558+0000 7fea186fe640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fea10156210 con 0x7fea10152d60 2024-04-25T00:11:42.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.558+0000 7fea15c99640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fea100ab7b0 0x7fea100c8450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:33114/0 (socket says 172.21.15.116:33114) 2024-04-25T00:11:42.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.558+0000 7fea15c99640 1 -- 172.21.15.116:0/237451352 learned_addr learned my addr 172.21.15.116:0/237451352 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:11:42.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.558+0000 7fea1649a640 1 --2- 172.21.15.116:0/237451352 >> v2:172.21.15.174:3300/0 conn(0x7fea10152d60 0x7fea100ab270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:42.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.558+0000 7fea15498640 1 -- 172.21.15.116:0/237451352 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4008834027 0 0) 0x7fea10155c50 con 0x7fea100ab7b0 2024-04-25T00:11:42.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.558+0000 7fea15498640 1 -- 172.21.15.116:0/237451352 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe9fc003530 con 0x7fea100ab7b0 2024-04-25T00:11:42.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.562+0000 7fea15498640 1 -- 172.21.15.116:0/237451352 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1049685704 0 0) 0x7fe9fc003530 con 0x7fea100ab7b0 2024-04-25T00:11:42.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.562+0000 7fea15498640 1 -- 172.21.15.116:0/237451352 >> v2:172.21.15.174:3300/0 conn(0x7fea10152d60 msgr2=0x7fea100ab270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:11:42.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.562+0000 7fea15498640 1 --2- 172.21.15.116:0/237451352 >> v2:172.21.15.174:3300/0 conn(0x7fea10152d60 0x7fea100ab270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:42.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.562+0000 7fea15498640 1 -- 172.21.15.116:0/237451352 >> v1:172.21.15.174:6789/0 conn(0x7fea10153140 legacy=0x7fea100c3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:42.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.562+0000 7fea15498640 1 -- 172.21.15.116:0/237451352 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea101564f0 con 0x7fea100ab7b0 2024-04-25T00:11:42.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.562+0000 7fea15498640 1 -- 172.21.15.116:0/237451352 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7fea00001680 con 0x7fea100ab7b0 2024-04-25T00:11:42.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.562+0000 7fea15498640 1 -- 172.21.15.116:0/237451352 >> v1:172.21.15.116:6789/0 conn(0x7fea100ab7b0 legacy=0x7fea100c8450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:42.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.562+0000 7fea15498640 1 --2- 172.21.15.116:0/237451352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9fc0039c0 0x7fe9fc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:42.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.562+0000 7fea15498640 1 --2- 172.21.15.116:0/237451352 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9fc004810 0x7fe9fc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:42.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.562+0000 7fea15498640 1 --2- 172.21.15.116:0/237451352 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe9fc0051d0 0x7fe9fc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:42.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.562+0000 7fea15498640 1 -- 172.21.15.116:0/237451352 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fea10156210 con 0x7fe9fc0039c0 2024-04-25T00:11:42.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.562+0000 7fea15498640 1 -- 172.21.15.116:0/237451352 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe9fc00c120 con 0x7fe9fc004810 2024-04-25T00:11:42.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.562+0000 7fea15498640 1 -- 172.21.15.116:0/237451352 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe9fc00c320 con 0x7fe9fc0051d0 2024-04-25T00:11:42.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.562+0000 7fea1649a640 1 --2- 172.21.15.116:0/237451352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9fc0039c0 0x7fe9fc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:42.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.562+0000 7fea16c9b640 1 --2- 172.21.15.116:0/237451352 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe9fc0051d0 0x7fe9fc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:42.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.562+0000 7fea15c99640 1 --2- 172.21.15.116:0/237451352 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9fc004810 0x7fe9fc004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:42.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.562+0000 7fea1649a640 1 -- 172.21.15.116:0/237451352 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe9fc0051d0 msgr2=0x7fe9fc00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:11:42.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.562+0000 7fea1649a640 1 --2- 172.21.15.116:0/237451352 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe9fc0051d0 0x7fe9fc00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:42.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.562+0000 7fea1649a640 1 -- 172.21.15.116:0/237451352 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9fc004810 msgr2=0x7fe9fc004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:11:42.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.562+0000 7fea1649a640 1 --2- 172.21.15.116:0/237451352 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9fc004810 0x7fe9fc004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:42.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.562+0000 7fea1649a640 1 -- 172.21.15.116:0/237451352 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9fc003530 con 0x7fe9fc0039c0 2024-04-25T00:11:42.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.562+0000 7fea16c9b640 1 --2- 172.21.15.116:0/237451352 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe9fc0051d0 0x7fe9fc00b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:11:42.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.562+0000 7fea15c99640 1 --2- 172.21.15.116:0/237451352 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9fc004810 0x7fe9fc004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:11:42.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.562+0000 7fea16c9b640 1 -- 172.21.15.116:0/237451352 reap_dead start 2024-04-25T00:11:42.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.562+0000 7fea1649a640 1 --2- 172.21.15.116:0/237451352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9fc0039c0 0x7fe9fc003de0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fea0c00ba30 tx=0x7fea0c01c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8daf5a7d30007c02 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:11:42.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.562+0000 7fea15498640 1 -- 172.21.15.116:0/237451352 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea0c00b600 con 0x7fe9fc0039c0 2024-04-25T00:11:42.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.562+0000 7fea15498640 1 -- 172.21.15.116:0/237451352 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea0c0b1070 con 0x7fe9fc0039c0 2024-04-25T00:11:42.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.562+0000 7fea15498640 1 -- 172.21.15.116:0/237451352 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea0c0ac680 con 0x7fe9fc0039c0 2024-04-25T00:11:42.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.566+0000 7fea186fe640 1 -- 172.21.15.116:0/237451352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9fc0039c0 msgr2=0x7fe9fc003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:42.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.566+0000 7fea186fe640 1 --2- 172.21.15.116:0/237451352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9fc0039c0 0x7fe9fc003de0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fea0c00ba30 tx=0x7fea0c01c6a0 comp rx=0 tx=0).stop 2024-04-25T00:11:42.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.566+0000 7fea186fe640 1 -- 172.21.15.116:0/237451352 shutdown_connections 2024-04-25T00:11:42.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.566+0000 7fea186fe640 1 --2- 172.21.15.116:0/237451352 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9fc0039c0 0x7fe9fc003de0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:42.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.566+0000 7fea186fe640 1 -- 172.21.15.116:0/237451352 >> 172.21.15.116:0/237451352 conn(0x7fea1006f0c0 msgr2=0x7fea1006f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:11:42.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.566+0000 7fea186fe640 1 -- 172.21.15.116:0/237451352 shutdown_connections 2024-04-25T00:11:42.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.566+0000 7fea186fe640 1 -- 172.21.15.116:0/237451352 wait complete. 2024-04-25T00:11:42.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.566+0000 7fea186fe640 1 Processor -- start 2024-04-25T00:11:42.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.566+0000 7fea186fe640 1 -- start start 2024-04-25T00:11:42.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.566+0000 7fea186fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9fc0039c0 0x7fea101595b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:42.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.566+0000 7fea186fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fea1014bd00 0x7fea10158410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:42.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.566+0000 7fea186fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fea10158950 0x7fea101f1810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:42.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.566+0000 7fea186fe640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fea100c4090 con 0x7fe9fc0039c0 2024-04-25T00:11:42.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.566+0000 7fea186fe640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fea100c3f10 con 0x7fea1014bd00 2024-04-25T00:11:42.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.566+0000 7fea186fe640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fea100c4210 con 0x7fea10158950 2024-04-25T00:11:42.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.566+0000 7fea1649a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9fc0039c0 0x7fea101595b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:42.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.566+0000 7fea1649a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9fc0039c0 0x7fea101595b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53988/0 (socket says 172.21.15.116:53988) 2024-04-25T00:11:42.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.566+0000 7fea1649a640 1 -- 172.21.15.116:0/3388731186 learned_addr learned my addr 172.21.15.116:0/3388731186 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:11:42.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.566+0000 7fea16c9b640 1 --2- 172.21.15.116:0/3388731186 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fea10158950 0x7fea101f1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:42.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.566+0000 7fea15c99640 1 --2- 172.21.15.116:0/3388731186 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fea1014bd00 0x7fea10158410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:42.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.566+0000 7fea1649a640 1 -- 172.21.15.116:0/3388731186 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fea10158950 msgr2=0x7fea101f1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:42.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.566+0000 7fea1649a640 1 --2- 172.21.15.116:0/3388731186 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fea10158950 0x7fea101f1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:42.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.566+0000 7fea1649a640 1 -- 172.21.15.116:0/3388731186 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fea1014bd00 msgr2=0x7fea10158410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:42.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.566+0000 7fea1649a640 1 --2- 172.21.15.116:0/3388731186 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fea1014bd00 0x7fea10158410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:42.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.566+0000 7fea1649a640 1 -- 172.21.15.116:0/3388731186 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea101eef70 con 0x7fe9fc0039c0 2024-04-25T00:11:42.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.566+0000 7fea1649a640 1 --2- 172.21.15.116:0/3388731186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9fc0039c0 0x7fea101595b0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fea0c0ae040 tx=0x7fe9fc005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:11:42.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.566+0000 7fea06ffd640 1 -- 172.21.15.116:0/3388731186 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea0c0bb070 con 0x7fe9fc0039c0 2024-04-25T00:11:42.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.566+0000 7fea06ffd640 1 -- 172.21.15.116:0/3388731186 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea0c004220 con 0x7fe9fc0039c0 2024-04-25T00:11:42.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.566+0000 7fea06ffd640 1 -- 172.21.15.116:0/3388731186 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea0c0ac910 con 0x7fe9fc0039c0 2024-04-25T00:11:42.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.566+0000 7fea186fe640 1 -- 172.21.15.116:0/3388731186 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea10156e80 con 0x7fe9fc0039c0 2024-04-25T00:11:42.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.566+0000 7fea186fe640 1 -- 172.21.15.116:0/3388731186 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea100adac0 con 0x7fe9fc0039c0 2024-04-25T00:11:42.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.570+0000 7fea186fe640 1 -- 172.21.15.116:0/3388731186 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea101573a0 con 0x7fe9fc0039c0 2024-04-25T00:11:42.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.570+0000 7fea06ffd640 1 -- 172.21.15.116:0/3388731186 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fea0c0acab0 con 0x7fe9fc0039c0 2024-04-25T00:11:42.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.570+0000 7fea06ffd640 1 --2- 172.21.15.116:0/3388731186 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe9f40678f0 0x7fe9f4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:42.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.570+0000 7fea06ffd640 1 -- 172.21.15.116:0/3388731186 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 7784+0+0 (secure 0 0 0) 0x7fea0c120960 con 0x7fe9fc0039c0 2024-04-25T00:11:42.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.570+0000 7fea15c99640 1 --2- 172.21.15.116:0/3388731186 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe9f40678f0 0x7fe9f4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:42.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.570+0000 7fea15c99640 1 --2- 172.21.15.116:0/3388731186 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe9f40678f0 0x7fe9f4069db0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fea10154d40 tx=0x7fea00009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:11:42.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.574+0000 7fea06ffd640 1 -- 172.21.15.116:0/3388731186 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fea101573a0 con 0x7fe9fc0039c0 2024-04-25T00:11:42.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.822+0000 7fea186fe640 1 -- 172.21.15.116:0/3388731186 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fea10068a60 con 0x7fe9fc0039c0 2024-04-25T00:11:42.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.822+0000 7fea06ffd640 1 -- 172.21.15.116:0/3388731186 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) ==== 94+0+6504 (secure 0 0 0) 0x7fea0c0163b0 con 0x7fe9fc0039c0 2024-04-25T00:11:42.823 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:11:42.823 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5461,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72},{"gid":6309,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1599708526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1599708526},{"type":"v1","addr":"172.21.15.116:6835","nonce":1599708526}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:11:03.640578+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6114},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6114":{"gid":6114,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":25,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6114,"qdb_cluster":[6114]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:09:33.601524+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":132,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4235,"qdb_cluster":[4235]},"id":2}]} 2024-04-25T00:11:42.824 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 77 2024-04-25T00:11:42.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.826+0000 7fea186fe640 1 -- 172.21.15.116:0/3388731186 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe9f40678f0 msgr2=0x7fe9f4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:42.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.826+0000 7fea186fe640 1 --2- 172.21.15.116:0/3388731186 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe9f40678f0 0x7fe9f4069db0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fea10154d40 tx=0x7fea00009040 comp rx=0 tx=0).stop 2024-04-25T00:11:42.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.826+0000 7fea186fe640 1 -- 172.21.15.116:0/3388731186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9fc0039c0 msgr2=0x7fea101595b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:42.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.826+0000 7fea186fe640 1 --2- 172.21.15.116:0/3388731186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9fc0039c0 0x7fea101595b0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fea0c0ae040 tx=0x7fe9fc005840 comp rx=0 tx=0).stop 2024-04-25T00:11:42.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.826+0000 7fea186fe640 1 -- 172.21.15.116:0/3388731186 shutdown_connections 2024-04-25T00:11:42.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.826+0000 7fea186fe640 1 --2- 172.21.15.116:0/3388731186 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe9f40678f0 0x7fe9f4069db0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:42.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.826+0000 7fea186fe640 1 --2- 172.21.15.116:0/3388731186 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fea10158950 0x7fea101f1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:42.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.826+0000 7fea186fe640 1 --2- 172.21.15.116:0/3388731186 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fea1014bd00 0x7fea10158410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:42.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.826+0000 7fea186fe640 1 --2- 172.21.15.116:0/3388731186 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9fc0039c0 0x7fea101595b0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:42.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.826+0000 7fea186fe640 1 -- 172.21.15.116:0/3388731186 >> 172.21.15.116:0/3388731186 conn(0x7fea1006f0c0 msgr2=0x7fea10150060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:11:42.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.826+0000 7fea186fe640 1 -- 172.21.15.116:0/3388731186 shutdown_connections 2024-04-25T00:11:42.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:42.826+0000 7fea186fe640 1 -- 172.21.15.116:0/3388731186 wait complete. 2024-04-25T00:11:42.845 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:11:44.846 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:11:44.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.986+0000 7f897a9b0640 1 Processor -- start 2024-04-25T00:11:44.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.986+0000 7f897a9b0640 1 -- start start 2024-04-25T00:11:44.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.986+0000 7f897a9b0640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f8974152d80 0x7f8974159950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:44.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.986+0000 7f897a9b0640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8974072080 0x7f89741562a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:44.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.986+0000 7f897a9b0640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f89740c36b0 0x7f89740c5020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:44.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.986+0000 7f897a9b0640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f89741567e0 con 0x7f8974072080 2024-04-25T00:11:44.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.986+0000 7f897a9b0640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f8974072e60 con 0x7f8974152d80 2024-04-25T00:11:44.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.986+0000 7f897a9b0640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f89740730b0 con 0x7f89740c36b0 2024-04-25T00:11:44.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.986+0000 7f89737fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8974072080 0x7f89741562a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:44.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.986+0000 7f89737fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8974072080 0x7f89741562a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51472/0 (socket says 172.21.15.116:51472) 2024-04-25T00:11:44.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.986+0000 7f89737fe640 1 -- 172.21.15.116:0/688184228 learned_addr learned my addr 172.21.15.116:0/688184228 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:11:44.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.986+0000 7f8978f4d640 1 --2- 172.21.15.116:0/688184228 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f89740c36b0 0x7f89740c5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:44.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.986+0000 7f8973fff640 1 --2- 172.21.15.116:0/688184228 >> v2:172.21.15.174:3300/0 conn(0x7f8974152d80 0x7f8974159950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:44.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.986+0000 7f89737fe640 1 -- 172.21.15.116:0/688184228 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f89740c36b0 msgr2=0x7f89740c5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:11:44.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.986+0000 7f89737fe640 1 --2- 172.21.15.116:0/688184228 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f89740c36b0 0x7f89740c5020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:44.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.986+0000 7f89737fe640 1 -- 172.21.15.116:0/688184228 >> v2:172.21.15.174:3300/0 conn(0x7f8974152d80 msgr2=0x7f8974159950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:11:44.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.986+0000 7f89737fe640 1 --2- 172.21.15.116:0/688184228 >> v2:172.21.15.174:3300/0 conn(0x7f8974152d80 0x7f8974159950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:44.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.986+0000 7f89737fe640 1 -- 172.21.15.116:0/688184228 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8974073390 con 0x7f8974072080 2024-04-25T00:11:44.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.990+0000 7f89737fe640 1 --2- 172.21.15.116:0/688184228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8974072080 0x7f89741562a0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f89740c7fb0 tx=0x7f896801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21dcfbbbdeb25cbb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:11:44.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.990+0000 7f8972ffd640 1 -- 172.21.15.116:0/688184228 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89680ab620 con 0x7f8974072080 2024-04-25T00:11:44.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.990+0000 7f8972ffd640 1 -- 172.21.15.116:0/688184228 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89680b0070 con 0x7f8974072080 2024-04-25T00:11:44.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.990+0000 7f8972ffd640 1 -- 172.21.15.116:0/688184228 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89680ab9e0 con 0x7f8974072080 2024-04-25T00:11:44.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.990+0000 7f897a9b0640 1 -- 172.21.15.116:0/688184228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8974072080 msgr2=0x7f89741562a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:44.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.990+0000 7f897a9b0640 1 --2- 172.21.15.116:0/688184228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8974072080 0x7f89741562a0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f89740c7fb0 tx=0x7f896801c460 comp rx=0 tx=0).stop 2024-04-25T00:11:44.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.990+0000 7f897a9b0640 1 -- 172.21.15.116:0/688184228 shutdown_connections 2024-04-25T00:11:44.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.990+0000 7f897a9b0640 1 --2- 172.21.15.116:0/688184228 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f89740c36b0 0x7f89740c5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:44.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.990+0000 7f897a9b0640 1 --2- 172.21.15.116:0/688184228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8974072080 0x7f89741562a0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:44.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.990+0000 7f897a9b0640 1 --2- 172.21.15.116:0/688184228 >> v2:172.21.15.174:3300/0 conn(0x7f8974152d80 0x7f8974159950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:44.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.990+0000 7f897a9b0640 1 -- 172.21.15.116:0/688184228 >> 172.21.15.116:0/688184228 conn(0x7f897406c720 msgr2=0x7f897406cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:11:44.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.990+0000 7f897a9b0640 1 -- 172.21.15.116:0/688184228 shutdown_connections 2024-04-25T00:11:44.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.990+0000 7f897a9b0640 1 -- 172.21.15.116:0/688184228 wait complete. 2024-04-25T00:11:44.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.990+0000 7f897a9b0640 1 Processor -- start 2024-04-25T00:11:44.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.990+0000 7f897a9b0640 1 -- start start 2024-04-25T00:11:44.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.994+0000 7f897a9b0640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8974072080 0x7f89740abb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:44.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.994+0000 7f897a9b0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89740c36b0 0x7f897414d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:44.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.994+0000 7f897a9b0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8974152d80 0x7f8974144bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:44.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.994+0000 7f897a9b0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f89741532e0 con 0x7f89740c36b0 2024-04-25T00:11:44.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.994+0000 7f897a9b0640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8974153160 con 0x7f8974152d80 2024-04-25T00:11:44.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.994+0000 7f897a9b0640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f897406ba20 con 0x7f8974072080 2024-04-25T00:11:44.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.994+0000 7f89737fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89740c36b0 0x7f897414d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:44.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.994+0000 7f89737fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89740c36b0 0x7f897414d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51480/0 (socket says 172.21.15.116:51480) 2024-04-25T00:11:44.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.994+0000 7f89737fe640 1 -- 172.21.15.116:0/3755961134 learned_addr learned my addr 172.21.15.116:0/3755961134 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:11:44.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.994+0000 7f8973fff640 1 --2- 172.21.15.116:0/3755961134 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8974072080 0x7f89740abb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:44.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.994+0000 7f8978f4d640 1 --2- 172.21.15.116:0/3755961134 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8974152d80 0x7f8974144bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:44.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.994+0000 7f89737fe640 1 -- 172.21.15.116:0/3755961134 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8974072080 msgr2=0x7f89740abb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:44.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.994+0000 7f89737fe640 1 --2- 172.21.15.116:0/3755961134 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8974072080 0x7f89740abb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:44.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.994+0000 7f89737fe640 1 -- 172.21.15.116:0/3755961134 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8974152d80 msgr2=0x7f8974144bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:44.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.994+0000 7f89737fe640 1 --2- 172.21.15.116:0/3755961134 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8974152d80 0x7f8974144bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:44.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.994+0000 7f89737fe640 1 -- 172.21.15.116:0/3755961134 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89741f1ae0 con 0x7f89740c36b0 2024-04-25T00:11:44.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.994+0000 7f89737fe640 1 --2- 172.21.15.116:0/3755961134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89740c36b0 0x7f897414d540 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f89680ad040 tx=0x7f896801aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:11:44.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.994+0000 7f8970ff9640 1 -- 172.21.15.116:0/3755961134 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89680abdb0 con 0x7f89740c36b0 2024-04-25T00:11:44.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.994+0000 7f8970ff9640 1 -- 172.21.15.116:0/3755961134 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89680be3f0 con 0x7f89740c36b0 2024-04-25T00:11:44.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.994+0000 7f897a9b0640 1 -- 172.21.15.116:0/3755961134 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f897414b9d0 con 0x7f89740c36b0 2024-04-25T00:11:44.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.994+0000 7f8970ff9640 1 -- 172.21.15.116:0/3755961134 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89680be7a0 con 0x7f89740c36b0 2024-04-25T00:11:44.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.994+0000 7f897a9b0640 1 -- 172.21.15.116:0/3755961134 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89741f2080 con 0x7f89740c36b0 2024-04-25T00:11:44.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.994+0000 7f897a9b0640 1 -- 172.21.15.116:0/3755961134 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8974068a60 con 0x7f89740c36b0 2024-04-25T00:11:44.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.994+0000 7f8970ff9640 1 -- 172.21.15.116:0/3755961134 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f89680b43f0 con 0x7f89740c36b0 2024-04-25T00:11:44.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.994+0000 7f8970ff9640 1 --2- 172.21.15.116:0/3755961134 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f89600678c0 0x7f8960069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:44.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.994+0000 7f8970ff9640 1 -- 172.21.15.116:0/3755961134 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 7784+0+0 (secure 0 0 0) 0x7f8968120950 con 0x7f89740c36b0 2024-04-25T00:11:44.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.998+0000 7f8973fff640 1 --2- 172.21.15.116:0/3755961134 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f89600678c0 0x7f8960069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:44.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:44.998+0000 7f8973fff640 1 --2- 172.21.15.116:0/3755961134 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f89600678c0 0x7f8960069d80 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f896400c870 tx=0x7f8964008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:11:44.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:45.002+0000 7f8970ff9640 1 -- 172.21.15.116:0/3755961134 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f89680c4060 con 0x7f89740c36b0 2024-04-25T00:11:45.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:45.226+0000 7f897a9b0640 1 -- 172.21.15.116:0/3755961134 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8974071dd0 con 0x7f89740c36b0 2024-04-25T00:11:45.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:45.226+0000 7f8970ff9640 1 -- 172.21.15.116:0/3755961134 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) ==== 94+0+6504 (secure 0 0 0) 0x7f8968016110 con 0x7f89740c36b0 2024-04-25T00:11:45.225 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:11:45.225 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5461,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72},{"gid":6309,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1599708526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1599708526},{"type":"v1","addr":"172.21.15.116:6835","nonce":1599708526}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:11:03.640578+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6114},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6114":{"gid":6114,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":25,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6114,"qdb_cluster":[6114]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:09:33.601524+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":132,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4235,"qdb_cluster":[4235]},"id":2}]} 2024-04-25T00:11:45.225 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 77 2024-04-25T00:11:45.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:45.226+0000 7f897a9b0640 1 -- 172.21.15.116:0/3755961134 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f89600678c0 msgr2=0x7f8960069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:45.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:45.226+0000 7f897a9b0640 1 --2- 172.21.15.116:0/3755961134 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f89600678c0 0x7f8960069d80 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f896400c870 tx=0x7f8964008040 comp rx=0 tx=0).stop 2024-04-25T00:11:45.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:45.226+0000 7f897a9b0640 1 -- 172.21.15.116:0/3755961134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89740c36b0 msgr2=0x7f897414d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:45.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:45.226+0000 7f897a9b0640 1 --2- 172.21.15.116:0/3755961134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89740c36b0 0x7f897414d540 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f89680ad040 tx=0x7f896801aca0 comp rx=0 tx=0).stop 2024-04-25T00:11:45.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:45.226+0000 7f897a9b0640 1 -- 172.21.15.116:0/3755961134 shutdown_connections 2024-04-25T00:11:45.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:45.226+0000 7f897a9b0640 1 --2- 172.21.15.116:0/3755961134 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f89600678c0 0x7f8960069d80 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:45.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:45.226+0000 7f897a9b0640 1 --2- 172.21.15.116:0/3755961134 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8974152d80 0x7f8974144bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:45.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:45.226+0000 7f897a9b0640 1 --2- 172.21.15.116:0/3755961134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89740c36b0 0x7f897414d540 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:45.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:45.226+0000 7f897a9b0640 1 --2- 172.21.15.116:0/3755961134 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8974072080 0x7f89740abb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:45.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:45.226+0000 7f897a9b0640 1 -- 172.21.15.116:0/3755961134 >> 172.21.15.116:0/3755961134 conn(0x7f897406c720 msgr2=0x7f897414f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:11:45.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:45.226+0000 7f897a9b0640 1 -- 172.21.15.116:0/3755961134 shutdown_connections 2024-04-25T00:11:45.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:45.226+0000 7f897a9b0640 1 -- 172.21.15.116:0/3755961134 wait complete. 2024-04-25T00:11:45.245 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:11:47.247 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:11:47.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.390+0000 7f91a4abd640 1 Processor -- start 2024-04-25T00:11:47.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.390+0000 7f91a4abd640 1 -- start start 2024-04-25T00:11:47.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.394+0000 7f91a4abd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f91a0152d80 0x7f91a0159950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:47.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.394+0000 7f91a4abd640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f91a01562a0 0x7f91a00c5020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:47.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.394+0000 7f91a4abd640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91a00730b0 con 0x7f91a0072080 2024-04-25T00:11:47.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.394+0000 7f91a4abd640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f91a0072e60 con 0x7f91a01562a0 2024-04-25T00:11:47.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.394+0000 7f91a4abd640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f91a0073390 con 0x7f91a0152d80 2024-04-25T00:11:47.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.394+0000 7f919dd9b640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f91a0072080 0x7f91a00c36b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:47974/0 (socket says 172.21.15.116:47974) 2024-04-25T00:11:47.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.394+0000 7f919dd9b640 1 -- 172.21.15.116:0/1821016882 learned_addr learned my addr 172.21.15.116:0/1821016882 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:11:47.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.394+0000 7f919d59a640 1 -- 172.21.15.116:0/1821016882 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4043180912 0 0) 0x7f91a00730b0 con 0x7f91a0072080 2024-04-25T00:11:47.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.394+0000 7f919d59a640 1 -- 172.21.15.116:0/1821016882 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9188003530 con 0x7f91a0072080 2024-04-25T00:11:47.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.394+0000 7f919e59c640 1 --2- 172.21.15.116:0/1821016882 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f91a0152d80 0x7f91a0159950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:47.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.394+0000 7f919ed9d640 1 --2- 172.21.15.116:0/1821016882 >> v2:172.21.15.174:3300/0 conn(0x7f91a01562a0 0x7f91a00c5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:47.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.394+0000 7f919d59a640 1 -- 172.21.15.116:0/1821016882 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1246375124 0 0) 0x7f9188003530 con 0x7f91a0072080 2024-04-25T00:11:47.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.394+0000 7f919d59a640 1 -- 172.21.15.116:0/1821016882 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f91a0152d80 msgr2=0x7f91a0159950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:11:47.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.394+0000 7f919d59a640 1 --2- 172.21.15.116:0/1821016882 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f91a0152d80 0x7f91a0159950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:47.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.394+0000 7f919d59a640 1 -- 172.21.15.116:0/1821016882 >> v2:172.21.15.174:3300/0 conn(0x7f91a01562a0 msgr2=0x7f91a00c5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:11:47.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.394+0000 7f919d59a640 1 --2- 172.21.15.116:0/1821016882 >> v2:172.21.15.174:3300/0 conn(0x7f91a01562a0 0x7f91a00c5020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:47.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.394+0000 7f919d59a640 1 -- 172.21.15.116:0/1821016882 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91a0158e70 con 0x7f91a0072080 2024-04-25T00:11:47.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.394+0000 7f919d59a640 1 -- 172.21.15.116:0/1821016882 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f9194001680 con 0x7f91a0072080 2024-04-25T00:11:47.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.394+0000 7f919d59a640 1 -- 172.21.15.116:0/1821016882 >> v1:172.21.15.116:6789/0 conn(0x7f91a0072080 legacy=0x7f91a00c36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:47.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.394+0000 7f919d59a640 1 --2- 172.21.15.116:0/1821016882 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f91a0152d80 0x7f91880039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:47.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.394+0000 7f919d59a640 1 --2- 172.21.15.116:0/1821016882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9188003f40 0x7f9188004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:47.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.394+0000 7f919d59a640 1 --2- 172.21.15.116:0/1821016882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9188004d50 0x7f91880051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:47.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.394+0000 7f919d59a640 1 -- 172.21.15.116:0/1821016882 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f91a0072e60 con 0x7f9188003f40 2024-04-25T00:11:47.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.394+0000 7f919d59a640 1 -- 172.21.15.116:0/1821016882 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f91a0073390 con 0x7f9188004d50 2024-04-25T00:11:47.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.394+0000 7f919d59a640 1 -- 172.21.15.116:0/1821016882 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f918800ba20 con 0x7f91a0152d80 2024-04-25T00:11:47.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.394+0000 7f919dd9b640 1 --2- 172.21.15.116:0/1821016882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9188003f40 0x7f9188004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:47.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.394+0000 7f919e59c640 1 --2- 172.21.15.116:0/1821016882 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f91a0152d80 0x7f91880039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:47.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.394+0000 7f919ed9d640 1 --2- 172.21.15.116:0/1821016882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9188004d50 0x7f91880051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:47.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.394+0000 7f919dd9b640 1 -- 172.21.15.116:0/1821016882 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f91a0152d80 msgr2=0x7f91880039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:11:47.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.394+0000 7f919dd9b640 1 --2- 172.21.15.116:0/1821016882 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f91a0152d80 0x7f91880039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:47.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.394+0000 7f919dd9b640 1 -- 172.21.15.116:0/1821016882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9188004d50 msgr2=0x7f91880051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:11:47.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.394+0000 7f919dd9b640 1 --2- 172.21.15.116:0/1821016882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9188004d50 0x7f91880051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:47.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.394+0000 7f919dd9b640 1 -- 172.21.15.116:0/1821016882 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9188003530 con 0x7f9188003f40 2024-04-25T00:11:47.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.394+0000 7f919ed9d640 1 --2- 172.21.15.116:0/1821016882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9188004d50 0x7f91880051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:11:47.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.398+0000 7f919dd9b640 1 --2- 172.21.15.116:0/1821016882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9188003f40 0x7f9188004810 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f919400c240 tx=0x7f919401cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=702857fa10b0178e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:11:47.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.398+0000 7f919d59a640 1 -- 172.21.15.116:0/1821016882 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9194007d40 con 0x7f9188003f40 2024-04-25T00:11:47.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.398+0000 7f919d59a640 1 -- 172.21.15.116:0/1821016882 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91940a9020 con 0x7f9188003f40 2024-04-25T00:11:47.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.398+0000 7f919d59a640 1 -- 172.21.15.116:0/1821016882 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91940a5b30 con 0x7f9188003f40 2024-04-25T00:11:47.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.398+0000 7f91a4abd640 1 -- 172.21.15.116:0/1821016882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9188003f40 msgr2=0x7f9188004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:47.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.398+0000 7f91a4abd640 1 --2- 172.21.15.116:0/1821016882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9188003f40 0x7f9188004810 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f919400c240 tx=0x7f919401cc30 comp rx=0 tx=0).stop 2024-04-25T00:11:47.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.398+0000 7f919ed9d640 1 -- 172.21.15.116:0/1821016882 reap_dead start 2024-04-25T00:11:47.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.398+0000 7f91a4abd640 1 -- 172.21.15.116:0/1821016882 shutdown_connections 2024-04-25T00:11:47.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.398+0000 7f91a4abd640 1 -- 172.21.15.116:0/1821016882 >> 172.21.15.116:0/1821016882 conn(0x7f91a006c720 msgr2=0x7f91a006cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:11:47.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.398+0000 7f91a4abd640 1 -- 172.21.15.116:0/1821016882 shutdown_connections 2024-04-25T00:11:47.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.398+0000 7f91a4abd640 1 -- 172.21.15.116:0/1821016882 wait complete. 2024-04-25T00:11:47.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.398+0000 7f91a4abd640 1 Processor -- start 2024-04-25T00:11:47.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.398+0000 7f91a4abd640 1 -- start start 2024-04-25T00:11:47.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.398+0000 7f91a4abd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9188003f40 0x7f91a01eee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:47.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.398+0000 7f91a4abd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f91a014a220 0x7f91a0151d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:47.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.398+0000 7f91a4abd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f91a01487c0 0x7f91a00ac730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:47.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.398+0000 7f91a4abd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f91a0153160 con 0x7f9188003f40 2024-04-25T00:11:47.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.398+0000 7f91a4abd640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f91a01566b0 con 0x7f91a014a220 2024-04-25T00:11:47.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.398+0000 7f91a4abd640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f91a01532e0 con 0x7f91a01487c0 2024-04-25T00:11:47.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.398+0000 7f919e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9188003f40 0x7f91a01eee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:47.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.398+0000 7f919e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9188003f40 0x7f91a01eee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51498/0 (socket says 172.21.15.116:51498) 2024-04-25T00:11:47.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.398+0000 7f919e59c640 1 -- 172.21.15.116:0/2770579429 learned_addr learned my addr 172.21.15.116:0/2770579429 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:11:47.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.398+0000 7f919ed9d640 1 --2- 172.21.15.116:0/2770579429 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f91a01487c0 0x7f91a00ac730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:47.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.398+0000 7f919dd9b640 1 --2- 172.21.15.116:0/2770579429 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f91a014a220 0x7f91a0151d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:47.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.398+0000 7f919e59c640 1 -- 172.21.15.116:0/2770579429 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f91a01487c0 msgr2=0x7f91a00ac730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:47.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.398+0000 7f919e59c640 1 --2- 172.21.15.116:0/2770579429 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f91a01487c0 0x7f91a00ac730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:47.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.398+0000 7f919e59c640 1 -- 172.21.15.116:0/2770579429 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f91a014a220 msgr2=0x7f91a0151d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:47.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.398+0000 7f919e59c640 1 --2- 172.21.15.116:0/2770579429 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f91a014a220 0x7f91a0151d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:47.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.398+0000 7f919e59c640 1 -- 172.21.15.116:0/2770579429 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91a01588c0 con 0x7f9188003f40 2024-04-25T00:11:47.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.402+0000 7f919e59c640 1 --2- 172.21.15.116:0/2770579429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9188003f40 0x7f91a01eee80 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f91900115c0 tx=0x7f9190011a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:11:47.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.402+0000 7f918effd640 1 -- 172.21.15.116:0/2770579429 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f919000f9b0 con 0x7f9188003f40 2024-04-25T00:11:47.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.402+0000 7f918effd640 1 -- 172.21.15.116:0/2770579429 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f919000fb50 con 0x7f9188003f40 2024-04-25T00:11:47.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.402+0000 7f91a4abd640 1 -- 172.21.15.116:0/2770579429 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91a01ee3e0 con 0x7f9188003f40 2024-04-25T00:11:47.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.402+0000 7f918effd640 1 -- 172.21.15.116:0/2770579429 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9190015460 con 0x7f9188003f40 2024-04-25T00:11:47.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.402+0000 7f91a4abd640 1 -- 172.21.15.116:0/2770579429 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91a01ee680 con 0x7f9188003f40 2024-04-25T00:11:47.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.402+0000 7f91a4abd640 1 -- 172.21.15.116:0/2770579429 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91a0068a60 con 0x7f9188003f40 2024-04-25T00:11:47.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.402+0000 7f918effd640 1 -- 172.21.15.116:0/2770579429 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f9190015600 con 0x7f9188003f40 2024-04-25T00:11:47.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.402+0000 7f918effd640 1 --2- 172.21.15.116:0/2770579429 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f91800678f0 0x7f9180069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:47.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.402+0000 7f918effd640 1 -- 172.21.15.116:0/2770579429 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 7784+0+0 (secure 0 0 0) 0x7f9190086c20 con 0x7f9188003f40 2024-04-25T00:11:47.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.402+0000 7f919dd9b640 1 --2- 172.21.15.116:0/2770579429 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f91800678f0 0x7f9180069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:47.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.402+0000 7f919dd9b640 1 --2- 172.21.15.116:0/2770579429 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f91800678f0 0x7f9180069db0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f91940aa000 tx=0x7f91940b0040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:11:47.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.410+0000 7f918effd640 1 -- 172.21.15.116:0/2770579429 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f9190019030 con 0x7f9188003f40 2024-04-25T00:11:47.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.634+0000 7f91a4abd640 1 -- 172.21.15.116:0/2770579429 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f91a00c4310 con 0x7f9188003f40 2024-04-25T00:11:47.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.634+0000 7f918effd640 1 -- 172.21.15.116:0/2770579429 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) ==== 94+0+6504 (secure 0 0 0) 0x7f9190056fd0 con 0x7f9188003f40 2024-04-25T00:11:47.633 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:11:47.633 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5461,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72},{"gid":6309,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1599708526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1599708526},{"type":"v1","addr":"172.21.15.116:6835","nonce":1599708526}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:11:03.640578+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6114},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6114":{"gid":6114,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":25,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6114,"qdb_cluster":[6114]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:09:33.601524+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":132,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4235,"qdb_cluster":[4235]},"id":2}]} 2024-04-25T00:11:47.633 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 77 2024-04-25T00:11:47.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.634+0000 7f91a4abd640 1 -- 172.21.15.116:0/2770579429 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f91800678f0 msgr2=0x7f9180069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:47.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.634+0000 7f91a4abd640 1 --2- 172.21.15.116:0/2770579429 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f91800678f0 0x7f9180069db0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f91940aa000 tx=0x7f91940b0040 comp rx=0 tx=0).stop 2024-04-25T00:11:47.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.634+0000 7f91a4abd640 1 -- 172.21.15.116:0/2770579429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9188003f40 msgr2=0x7f91a01eee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:47.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.634+0000 7f91a4abd640 1 --2- 172.21.15.116:0/2770579429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9188003f40 0x7f91a01eee80 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f91900115c0 tx=0x7f9190011a80 comp rx=0 tx=0).stop 2024-04-25T00:11:47.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.634+0000 7f91a4abd640 1 -- 172.21.15.116:0/2770579429 shutdown_connections 2024-04-25T00:11:47.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.634+0000 7f91a4abd640 1 --2- 172.21.15.116:0/2770579429 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f91800678f0 0x7f9180069db0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:47.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.634+0000 7f91a4abd640 1 --2- 172.21.15.116:0/2770579429 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f91a01487c0 0x7f91a00ac730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:47.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.634+0000 7f91a4abd640 1 --2- 172.21.15.116:0/2770579429 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f91a014a220 0x7f91a0151d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:47.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.634+0000 7f91a4abd640 1 --2- 172.21.15.116:0/2770579429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9188003f40 0x7f91a01eee80 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:47.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.634+0000 7f91a4abd640 1 -- 172.21.15.116:0/2770579429 >> 172.21.15.116:0/2770579429 conn(0x7f91a006c720 msgr2=0x7f91a006cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:11:47.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.634+0000 7f91a4abd640 1 -- 172.21.15.116:0/2770579429 shutdown_connections 2024-04-25T00:11:47.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:47.634+0000 7f91a4abd640 1 -- 172.21.15.116:0/2770579429 wait complete. 2024-04-25T00:11:47.653 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:11:49.653 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:11:49.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.778+0000 7f7698e27640 1 Processor -- start 2024-04-25T00:11:49.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.778+0000 7f7698e27640 1 -- start start 2024-04-25T00:11:49.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.778+0000 7f7698e27640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f769406d5f0 0x7f7694156c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:49.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.778+0000 7f7698e27640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7694155810 0x7f76941571c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:49.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.778+0000 7f7698e27640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76940c9150 con 0x7f7694152d80 2024-04-25T00:11:49.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.778+0000 7f7698e27640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f76940c9d60 con 0x7f7694155810 2024-04-25T00:11:49.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.778+0000 7f7698e27640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f76940c9430 con 0x7f769406d5f0 2024-04-25T00:11:49.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.778+0000 7f769259c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f7694152d80 0x7f769406bd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:47982/0 (socket says 172.21.15.116:47982) 2024-04-25T00:11:49.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.778+0000 7f769259c640 1 -- 172.21.15.116:0/4012334201 learned_addr learned my addr 172.21.15.116:0/4012334201 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:11:49.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.778+0000 7f7692d9d640 1 --2- 172.21.15.116:0/4012334201 >> v2:172.21.15.116:3300/0 conn(0x7f7694155810 0x7f76941571c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:49.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.778+0000 7f7691d9b640 1 --2- 172.21.15.116:0/4012334201 >> v2:172.21.15.174:3300/0 conn(0x7f769406d5f0 0x7f7694156c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:49.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.778+0000 7f7692d9d640 1 -- 172.21.15.116:0/4012334201 >> v2:172.21.15.174:3300/0 conn(0x7f769406d5f0 msgr2=0x7f7694156c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:11:49.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.778+0000 7f7692d9d640 1 --2- 172.21.15.116:0/4012334201 >> v2:172.21.15.174:3300/0 conn(0x7f769406d5f0 0x7f7694156c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:49.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.778+0000 7f7692d9d640 1 -- 172.21.15.116:0/4012334201 >> v1:172.21.15.116:6789/0 conn(0x7f7694152d80 legacy=0x7f769406bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:49.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.778+0000 7f7692d9d640 1 -- 172.21.15.116:0/4012334201 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76940c9710 con 0x7f7694155810 2024-04-25T00:11:49.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.782+0000 7f7692d9d640 1 --2- 172.21.15.116:0/4012334201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7694155810 0x7f76941571c0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f769408ca90 tx=0x7f768401c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6acc0ab3c876a82d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:11:49.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.782+0000 7f769159a640 1 -- 172.21.15.116:0/4012334201 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76840b0070 con 0x7f7694155810 2024-04-25T00:11:49.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.782+0000 7f769159a640 1 -- 172.21.15.116:0/4012334201 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76840ab600 con 0x7f7694155810 2024-04-25T00:11:49.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.782+0000 7f769159a640 1 -- 172.21.15.116:0/4012334201 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76840ab9d0 con 0x7f7694155810 2024-04-25T00:11:49.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.782+0000 7f7698e27640 1 -- 172.21.15.116:0/4012334201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7694155810 msgr2=0x7f76941571c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:49.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.782+0000 7f7698e27640 1 --2- 172.21.15.116:0/4012334201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7694155810 0x7f76941571c0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f769408ca90 tx=0x7f768401c3c0 comp rx=0 tx=0).stop 2024-04-25T00:11:49.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.782+0000 7f7698e27640 1 -- 172.21.15.116:0/4012334201 shutdown_connections 2024-04-25T00:11:49.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.782+0000 7f7698e27640 1 --2- 172.21.15.116:0/4012334201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7694155810 0x7f76941571c0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:49.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.782+0000 7f7698e27640 1 --2- 172.21.15.116:0/4012334201 >> v2:172.21.15.174:3300/0 conn(0x7f769406d5f0 0x7f7694156c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:49.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.782+0000 7f7698e27640 1 -- 172.21.15.116:0/4012334201 >> 172.21.15.116:0/4012334201 conn(0x7f76940acdd0 msgr2=0x7f76940ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:11:49.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.782+0000 7f7698e27640 1 -- 172.21.15.116:0/4012334201 shutdown_connections 2024-04-25T00:11:49.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.782+0000 7f7698e27640 1 -- 172.21.15.116:0/4012334201 wait complete. 2024-04-25T00:11:49.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.782+0000 7f7698e27640 1 Processor -- start 2024-04-25T00:11:49.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.782+0000 7f7698e27640 1 -- start start 2024-04-25T00:11:49.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.782+0000 7f7698e27640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f769406d5f0 0x7f769414c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:49.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.782+0000 7f7698e27640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7694152d80 0x7f76941f1360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:49.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.782+0000 7f7698e27640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7694155810 0x7f7694151c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:49.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.782+0000 7f7698e27640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f76940caf00 con 0x7f769406d5f0 2024-04-25T00:11:49.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.782+0000 7f7698e27640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f76940cad80 con 0x7f7694155810 2024-04-25T00:11:49.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.782+0000 7f7698e27640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f76941628d0 con 0x7f7694152d80 2024-04-25T00:11:49.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.782+0000 7f769259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f769406d5f0 0x7f769414c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:49.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.782+0000 7f769259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f769406d5f0 0x7f769414c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51502/0 (socket says 172.21.15.116:51502) 2024-04-25T00:11:49.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.786+0000 7f769259c640 1 -- 172.21.15.116:0/1312265111 learned_addr learned my addr 172.21.15.116:0/1312265111 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:11:49.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.786+0000 7f7691d9b640 1 --2- 172.21.15.116:0/1312265111 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7694152d80 0x7f76941f1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:49.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.786+0000 7f7692d9d640 1 --2- 172.21.15.116:0/1312265111 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7694155810 0x7f7694151c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:49.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.786+0000 7f769259c640 1 -- 172.21.15.116:0/1312265111 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7694152d80 msgr2=0x7f76941f1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:49.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.786+0000 7f769259c640 1 --2- 172.21.15.116:0/1312265111 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7694152d80 0x7f76941f1360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:49.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.786+0000 7f769259c640 1 -- 172.21.15.116:0/1312265111 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7694155810 msgr2=0x7f7694151c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:49.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.786+0000 7f769259c640 1 --2- 172.21.15.116:0/1312265111 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7694155810 0x7f7694151c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:49.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.786+0000 7f769259c640 1 -- 172.21.15.116:0/1312265111 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76940c2470 con 0x7f769406d5f0 2024-04-25T00:11:49.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.786+0000 7f7691d9b640 1 --2- 172.21.15.116:0/1312265111 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7694152d80 0x7f76941f1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:11:49.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.786+0000 7f769259c640 1 --2- 172.21.15.116:0/1312265111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f769406d5f0 0x7f769414c6d0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f768c00cb40 tx=0x7f768c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:11:49.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.786+0000 7f768affd640 1 -- 172.21.15.116:0/1312265111 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f768c017070 con 0x7f769406d5f0 2024-04-25T00:11:49.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.786+0000 7f768affd640 1 -- 172.21.15.116:0/1312265111 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f768c00aca0 con 0x7f769406d5f0 2024-04-25T00:11:49.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.786+0000 7f7698e27640 1 -- 172.21.15.116:0/1312265111 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f769414c4a0 con 0x7f769406d5f0 2024-04-25T00:11:49.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.786+0000 7f768affd640 1 -- 172.21.15.116:0/1312265111 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f768c012620 con 0x7f769406d5f0 2024-04-25T00:11:49.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.786+0000 7f7698e27640 1 -- 172.21.15.116:0/1312265111 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76941504a0 con 0x7f769406d5f0 2024-04-25T00:11:49.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.786+0000 7f7698e27640 1 -- 172.21.15.116:0/1312265111 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7694068350 con 0x7f769406d5f0 2024-04-25T00:11:49.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.786+0000 7f768affd640 1 -- 172.21.15.116:0/1312265111 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f768c01a070 con 0x7f769406d5f0 2024-04-25T00:11:49.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.786+0000 7f768affd640 1 --2- 172.21.15.116:0/1312265111 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f76740678c0 0x7f7674069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:49.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.786+0000 7f768affd640 1 -- 172.21.15.116:0/1312265111 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 7784+0+0 (secure 0 0 0) 0x7f768c0861d0 con 0x7f769406d5f0 2024-04-25T00:11:49.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.786+0000 7f7691d9b640 1 --2- 172.21.15.116:0/1312265111 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f76740678c0 0x7f7674069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:49.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.790+0000 7f7691d9b640 1 --2- 172.21.15.116:0/1312265111 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f76740678c0 0x7f7674069d80 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f769414d190 tx=0x7f767c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:11:49.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:49.794+0000 7f768affd640 1 -- 172.21.15.116:0/1312265111 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f768c056580 con 0x7f769406d5f0 2024-04-25T00:11:50.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:50.042+0000 7f7698e27640 1 -- 172.21.15.116:0/1312265111 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f76940cab60 con 0x7f769406d5f0 2024-04-25T00:11:50.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:50.042+0000 7f768affd640 1 -- 172.21.15.116:0/1312265111 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) ==== 94+0+6504 (secure 0 0 0) 0x7f768c055e80 con 0x7f769406d5f0 2024-04-25T00:11:50.041 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:11:50.041 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5461,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72},{"gid":6309,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1599708526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1599708526},{"type":"v1","addr":"172.21.15.116:6835","nonce":1599708526}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:11:03.640578+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6114},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6114":{"gid":6114,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":25,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6114,"qdb_cluster":[6114]},"id":1},{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:09:33.601524+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4235":{"gid":4235,"name":"d","rank":0,"incarnation":62,"state":"up:active","state_seq":132,"addr":"172.21.15.174:6835/2883220846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":2883220846},{"type":"v1","addr":"172.21.15.174:6835","nonce":2883220846}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4235,"qdb_cluster":[4235]},"id":2}]} 2024-04-25T00:11:50.042 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 77 2024-04-25T00:11:50.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:50.042+0000 7f7698e27640 1 -- 172.21.15.116:0/1312265111 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f76740678c0 msgr2=0x7f7674069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:50.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:50.042+0000 7f7698e27640 1 --2- 172.21.15.116:0/1312265111 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f76740678c0 0x7f7674069d80 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f769414d190 tx=0x7f767c008040 comp rx=0 tx=0).stop 2024-04-25T00:11:50.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:50.042+0000 7f7698e27640 1 -- 172.21.15.116:0/1312265111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f769406d5f0 msgr2=0x7f769414c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:50.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:50.042+0000 7f7698e27640 1 --2- 172.21.15.116:0/1312265111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f769406d5f0 0x7f769414c6d0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f768c00cb40 tx=0x7f768c00a060 comp rx=0 tx=0).stop 2024-04-25T00:11:50.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:50.042+0000 7f7698e27640 1 -- 172.21.15.116:0/1312265111 shutdown_connections 2024-04-25T00:11:50.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:50.042+0000 7f7698e27640 1 --2- 172.21.15.116:0/1312265111 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f76740678c0 0x7f7674069d80 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:50.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:50.042+0000 7f7698e27640 1 --2- 172.21.15.116:0/1312265111 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7694155810 0x7f7694151c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:50.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:50.042+0000 7f7698e27640 1 --2- 172.21.15.116:0/1312265111 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7694152d80 0x7f76941f1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:50.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:50.042+0000 7f7698e27640 1 --2- 172.21.15.116:0/1312265111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f769406d5f0 0x7f769414c6d0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:50.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:50.042+0000 7f7698e27640 1 -- 172.21.15.116:0/1312265111 >> 172.21.15.116:0/1312265111 conn(0x7f76940acdd0 msgr2=0x7f769414ef40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:11:50.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:50.046+0000 7f7698e27640 1 -- 172.21.15.116:0/1312265111 shutdown_connections 2024-04-25T00:11:50.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:50.046+0000 7f7698e27640 1 -- 172.21.15.116:0/1312265111 wait complete. 2024-04-25T00:11:50.060 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:11:52.060 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:11:52.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.186+0000 7fa5df3e1640 1 Processor -- start 2024-04-25T00:11:52.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.186+0000 7fa5df3e1640 1 -- start start 2024-04-25T00:11:52.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.190+0000 7fa5df3e1640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa5d8148f20 0x7fa5d80c3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:52.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.190+0000 7fa5df3e1640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fa5d8152f70 0x7fa5d806e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:52.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.190+0000 7fa5df3e1640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa5d80c3300 0x7fa5d8071d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:52.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.190+0000 7fa5df3e1640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fa5d80c4790 con 0x7fa5d80c3300 2024-04-25T00:11:52.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.190+0000 7fa5df3e1640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fa5d80c2b40 con 0x7fa5d8152f70 2024-04-25T00:11:52.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.190+0000 7fa5df3e1640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa5d81533f0 con 0x7fa5d8148f20 2024-04-25T00:11:52.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.190+0000 7fa5dd97e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa5d80c3300 0x7fa5d8071d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:52.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.190+0000 7fa5dd97e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa5d80c3300 0x7fa5d8071d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51518/0 (socket says 172.21.15.116:51518) 2024-04-25T00:11:52.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.190+0000 7fa5dd97e640 1 -- 172.21.15.116:0/3348699274 learned_addr learned my addr 172.21.15.116:0/3348699274 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:11:52.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.190+0000 7fa5dc97c640 1 --2- 172.21.15.116:0/3348699274 >> v2:172.21.15.174:3300/0 conn(0x7fa5d8152f70 0x7fa5d806e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:52.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.190+0000 7fa5dd97e640 1 -- 172.21.15.116:0/3348699274 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa5d8148f20 msgr2=0x7fa5d80c3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:11:52.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.190+0000 7fa5dd97e640 1 --2- 172.21.15.116:0/3348699274 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa5d8148f20 0x7fa5d80c3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:52.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.190+0000 7fa5dd97e640 1 -- 172.21.15.116:0/3348699274 >> v2:172.21.15.174:3300/0 conn(0x7fa5d8152f70 msgr2=0x7fa5d806e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:11:52.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.190+0000 7fa5dd97e640 1 --2- 172.21.15.116:0/3348699274 >> v2:172.21.15.174:3300/0 conn(0x7fa5d8152f70 0x7fa5d806e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:52.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.190+0000 7fa5dd97e640 1 -- 172.21.15.116:0/3348699274 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5d80c41d0 con 0x7fa5d80c3300 2024-04-25T00:11:52.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.190+0000 7fa5dd97e640 1 --2- 172.21.15.116:0/3348699274 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5d80c3300 0x7fa5d8071d80 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fa5d80c5b00 tx=0x7fa5d001ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8a7cb207d75c79e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:11:52.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.190+0000 7fa5cffff640 1 -- 172.21.15.116:0/3348699274 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5d000b2d0 con 0x7fa5d80c3300 2024-04-25T00:11:52.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.190+0000 7fa5cffff640 1 -- 172.21.15.116:0/3348699274 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5d000b470 con 0x7fa5d80c3300 2024-04-25T00:11:52.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.190+0000 7fa5cffff640 1 -- 172.21.15.116:0/3348699274 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5d00a7ca0 con 0x7fa5d80c3300 2024-04-25T00:11:52.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.190+0000 7fa5df3e1640 1 -- 172.21.15.116:0/3348699274 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5d80c3300 msgr2=0x7fa5d8071d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:52.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.190+0000 7fa5df3e1640 1 --2- 172.21.15.116:0/3348699274 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5d80c3300 0x7fa5d8071d80 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fa5d80c5b00 tx=0x7fa5d001ebf0 comp rx=0 tx=0).stop 2024-04-25T00:11:52.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.190+0000 7fa5df3e1640 1 -- 172.21.15.116:0/3348699274 shutdown_connections 2024-04-25T00:11:52.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.190+0000 7fa5df3e1640 1 --2- 172.21.15.116:0/3348699274 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5d80c3300 0x7fa5d8071d80 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:52.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.190+0000 7fa5df3e1640 1 --2- 172.21.15.116:0/3348699274 >> v2:172.21.15.174:3300/0 conn(0x7fa5d8152f70 0x7fa5d806e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:52.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.190+0000 7fa5df3e1640 1 --2- 172.21.15.116:0/3348699274 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa5d8148f20 0x7fa5d80c3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:52.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.190+0000 7fa5df3e1640 1 -- 172.21.15.116:0/3348699274 >> 172.21.15.116:0/3348699274 conn(0x7fa5d80c7fa0 msgr2=0x7fa5d80c83b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:11:52.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.194+0000 7fa5df3e1640 1 -- 172.21.15.116:0/3348699274 shutdown_connections 2024-04-25T00:11:52.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.194+0000 7fa5df3e1640 1 -- 172.21.15.116:0/3348699274 wait complete. 2024-04-25T00:11:52.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.194+0000 7fa5df3e1640 1 Processor -- start 2024-04-25T00:11:52.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.194+0000 7fa5df3e1640 1 -- start start 2024-04-25T00:11:52.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.194+0000 7fa5df3e1640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa5d80c3300 0x7fa5d8154ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:52.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.194+0000 7fa5df3e1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5d8148f20 0x7fa5d81513b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:52.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.194+0000 7fa5df3e1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa5d8152f70 0x7fa5d81518f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:52.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.194+0000 7fa5df3e1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d8149300 con 0x7fa5d8148f20 2024-04-25T00:11:52.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.194+0000 7fa5df3e1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d80c3770 con 0x7fa5d8152f70 2024-04-25T00:11:52.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.194+0000 7fa5df3e1640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa5d81ee1e0 con 0x7fa5d80c3300 2024-04-25T00:11:52.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.194+0000 7fa5dc97c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5d8148f20 0x7fa5d81513b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:52.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.194+0000 7fa5dc97c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5d8148f20 0x7fa5d81513b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51522/0 (socket says 172.21.15.116:51522) 2024-04-25T00:11:52.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.194+0000 7fa5dc97c640 1 -- 172.21.15.116:0/2859185332 learned_addr learned my addr 172.21.15.116:0/2859185332 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:11:52.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.194+0000 7fa5dd17d640 1 --2- 172.21.15.116:0/2859185332 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa5d80c3300 0x7fa5d8154ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:52.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.194+0000 7fa5dd97e640 1 --2- 172.21.15.116:0/2859185332 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa5d8152f70 0x7fa5d81518f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:52.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.194+0000 7fa5dc97c640 1 -- 172.21.15.116:0/2859185332 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa5d80c3300 msgr2=0x7fa5d8154ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:52.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.194+0000 7fa5dc97c640 1 --2- 172.21.15.116:0/2859185332 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa5d80c3300 0x7fa5d8154ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:52.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.194+0000 7fa5dc97c640 1 -- 172.21.15.116:0/2859185332 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa5d8152f70 msgr2=0x7fa5d81518f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:52.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.194+0000 7fa5dc97c640 1 --2- 172.21.15.116:0/2859185332 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa5d8152f70 0x7fa5d81518f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:52.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.194+0000 7fa5dc97c640 1 -- 172.21.15.116:0/2859185332 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5d81eacc0 con 0x7fa5d8148f20 2024-04-25T00:11:52.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.194+0000 7fa5dd97e640 1 --2- 172.21.15.116:0/2859185332 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa5d8152f70 0x7fa5d81518f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:11:52.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.194+0000 7fa5dc97c640 1 --2- 172.21.15.116:0/2859185332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5d8148f20 0x7fa5d81513b0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fa5c800c820 tx=0x7fa5c800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:11:52.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.194+0000 7fa5cdffb640 1 -- 172.21.15.116:0/2859185332 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c8015070 con 0x7fa5d8148f20 2024-04-25T00:11:52.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.194+0000 7fa5cdffb640 1 -- 172.21.15.116:0/2859185332 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5c8007a30 con 0x7fa5d8148f20 2024-04-25T00:11:52.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.194+0000 7fa5cdffb640 1 -- 172.21.15.116:0/2859185332 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c8007e20 con 0x7fa5d8148f20 2024-04-25T00:11:52.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.194+0000 7fa5df3e1640 1 -- 172.21.15.116:0/2859185332 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5d8151e30 con 0x7fa5d8148f20 2024-04-25T00:11:52.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.194+0000 7fa5df3e1640 1 -- 172.21.15.116:0/2859185332 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5d81ea1e0 con 0x7fa5d8148f20 2024-04-25T00:11:52.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.198+0000 7fa5df3e1640 1 -- 172.21.15.116:0/2859185332 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5d8068a60 con 0x7fa5d8148f20 2024-04-25T00:11:52.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.198+0000 7fa5cdffb640 1 -- 172.21.15.116:0/2859185332 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa5c8018070 con 0x7fa5d8148f20 2024-04-25T00:11:52.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.198+0000 7fa5cdffb640 1 --2- 172.21.15.116:0/2859185332 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa5bc0678c0 0x7fa5bc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:52.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.198+0000 7fa5cdffb640 1 -- 172.21.15.116:0/2859185332 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7870+0+0 (secure 0 0 0) 0x7fa5c80862a0 con 0x7fa5d8148f20 2024-04-25T00:11:52.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.198+0000 7fa5dd17d640 1 --2- 172.21.15.116:0/2859185332 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa5bc0678c0 0x7fa5bc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:52.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.198+0000 7fa5dd17d640 1 --2- 172.21.15.116:0/2859185332 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa5bc0678c0 0x7fa5bc069d80 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fa5d4006fd0 tx=0x7fa5d4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:11:52.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.202+0000 7fa5cdffb640 1 -- 172.21.15.116:0/2859185332 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fa5c8056600 con 0x7fa5d8148f20 2024-04-25T00:11:52.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.466+0000 7fa5df3e1640 1 -- 172.21.15.116:0/2859185332 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa5d815cc50 con 0x7fa5d8148f20 2024-04-25T00:11:52.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.466+0000 7fa5cdffb640 1 -- 172.21.15.116:0/2859185332 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) ==== 94+0+5671 (secure 0 0 0) 0x7fa5c8055f00 con 0x7fa5d8148f20 2024-04-25T00:11:52.467 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:11:52.467 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5461,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:11:03.640578+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6114},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6114":{"gid":6114,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":25,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6114,"qdb_cluster":[6114]},"id":1},{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:11:50.909463+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6309},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6309":{"gid":6309,"name":"c","rank":0,"incarnation":78,"state":"up:reconnect","state_seq":13,"addr":"172.21.15.116:6835/1599708526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1599708526},{"type":"v1","addr":"172.21.15.116:6835","nonce":1599708526}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:11:52.468 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 79 2024-04-25T00:11:52.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.470+0000 7fa5df3e1640 1 -- 172.21.15.116:0/2859185332 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa5bc0678c0 msgr2=0x7fa5bc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:52.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.470+0000 7fa5df3e1640 1 --2- 172.21.15.116:0/2859185332 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa5bc0678c0 0x7fa5bc069d80 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fa5d4006fd0 tx=0x7fa5d4008040 comp rx=0 tx=0).stop 2024-04-25T00:11:52.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.470+0000 7fa5df3e1640 1 -- 172.21.15.116:0/2859185332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5d8148f20 msgr2=0x7fa5d81513b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:52.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.470+0000 7fa5df3e1640 1 --2- 172.21.15.116:0/2859185332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5d8148f20 0x7fa5d81513b0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fa5c800c820 tx=0x7fa5c800cce0 comp rx=0 tx=0).stop 2024-04-25T00:11:52.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.470+0000 7fa5df3e1640 1 -- 172.21.15.116:0/2859185332 shutdown_connections 2024-04-25T00:11:52.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.470+0000 7fa5df3e1640 1 --2- 172.21.15.116:0/2859185332 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa5bc0678c0 0x7fa5bc069d80 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:52.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.470+0000 7fa5df3e1640 1 --2- 172.21.15.116:0/2859185332 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa5d8152f70 0x7fa5d81518f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:52.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.470+0000 7fa5df3e1640 1 --2- 172.21.15.116:0/2859185332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa5d8148f20 0x7fa5d81513b0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:52.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.470+0000 7fa5df3e1640 1 --2- 172.21.15.116:0/2859185332 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa5d80c3300 0x7fa5d8154ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:52.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.470+0000 7fa5df3e1640 1 -- 172.21.15.116:0/2859185332 >> 172.21.15.116:0/2859185332 conn(0x7fa5d80c7fa0 msgr2=0x7fa5d80c4910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:11:52.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.470+0000 7fa5df3e1640 1 -- 172.21.15.116:0/2859185332 shutdown_connections 2024-04-25T00:11:52.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.470+0000 7fa5df3e1640 1 -- 172.21.15.116:0/2859185332 wait complete. 2024-04-25T00:11:52.487 INFO:tasks.mds_thrash.fs.[b]:mds.d down, removed from mdsmap 2024-04-25T00:11:52.487 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-04-25T00:11:52.487 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:11:52.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.606+0000 7fe193279640 1 Processor -- start 2024-04-25T00:11:52.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.606+0000 7fe193279640 1 -- start start 2024-04-25T00:11:52.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.606+0000 7fe193279640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe18c150d10 0x7fe18c0c5010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:52.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.606+0000 7fe193279640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe18c072ca0 0x7fe18c154190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:52.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.606+0000 7fe193279640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe18c0c4950 con 0x7fe18c0c5550 2024-04-25T00:11:52.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.606+0000 7fe193279640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fe18c1546d0 con 0x7fe18c150d10 2024-04-25T00:11:52.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.606+0000 7fe193279640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe18c1510f0 con 0x7fe18c072ca0 2024-04-25T00:11:52.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.606+0000 7fe191015640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe18c150d10 0x7fe18c0c5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:52.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.606+0000 7fe191015640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe18c150d10 0x7fe18c0c5010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51528/0 (socket says 172.21.15.116:51528) 2024-04-25T00:11:52.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.606+0000 7fe191015640 1 -- 172.21.15.116:0/1024462318 learned_addr learned my addr 172.21.15.116:0/1024462318 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:11:52.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.606+0000 7fe191816640 1 --2- 172.21.15.116:0/1024462318 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe18c072ca0 0x7fe18c154190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:52.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.606+0000 7fe191015640 1 -- 172.21.15.116:0/1024462318 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe18c072ca0 msgr2=0x7fe18c154190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:11:52.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.606+0000 7fe191015640 1 --2- 172.21.15.116:0/1024462318 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe18c072ca0 0x7fe18c154190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:52.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.606+0000 7fe191015640 1 -- 172.21.15.116:0/1024462318 >> v1:172.21.15.174:6789/0 conn(0x7fe18c0c5550 legacy=0x7fe18c153a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:52.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.606+0000 7fe191015640 1 -- 172.21.15.116:0/1024462318 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe18c157a20 con 0x7fe18c150d10 2024-04-25T00:11:52.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.610+0000 7fe191015640 1 --2- 172.21.15.116:0/1024462318 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe18c150d10 0x7fe18c0c5010 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fe188009870 tx=0x7fe18801c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce81169fbfe9588 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:11:52.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.610+0000 7fe183fff640 1 -- 172.21.15.116:0/1024462318 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1880ab640 con 0x7fe18c150d10 2024-04-25T00:11:52.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.610+0000 7fe183fff640 1 -- 172.21.15.116:0/1024462318 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1880b0070 con 0x7fe18c150d10 2024-04-25T00:11:52.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.610+0000 7fe183fff640 1 -- 172.21.15.116:0/1024462318 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1880aba00 con 0x7fe18c150d10 2024-04-25T00:11:52.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.610+0000 7fe193279640 1 -- 172.21.15.116:0/1024462318 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe18c150d10 msgr2=0x7fe18c0c5010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:52.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.610+0000 7fe193279640 1 --2- 172.21.15.116:0/1024462318 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe18c150d10 0x7fe18c0c5010 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fe188009870 tx=0x7fe18801c440 comp rx=0 tx=0).stop 2024-04-25T00:11:52.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.610+0000 7fe193279640 1 -- 172.21.15.116:0/1024462318 shutdown_connections 2024-04-25T00:11:52.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.610+0000 7fe193279640 1 --2- 172.21.15.116:0/1024462318 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe18c072ca0 0x7fe18c154190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:52.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.610+0000 7fe193279640 1 --2- 172.21.15.116:0/1024462318 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe18c150d10 0x7fe18c0c5010 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:52.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.610+0000 7fe193279640 1 -- 172.21.15.116:0/1024462318 >> 172.21.15.116:0/1024462318 conn(0x7fe18c06c720 msgr2=0x7fe18c06cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:11:52.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.610+0000 7fe193279640 1 -- 172.21.15.116:0/1024462318 shutdown_connections 2024-04-25T00:11:52.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.610+0000 7fe193279640 1 -- 172.21.15.116:0/1024462318 wait complete. 2024-04-25T00:11:52.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.610+0000 7fe193279640 1 Processor -- start 2024-04-25T00:11:52.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.610+0000 7fe193279640 1 -- start start 2024-04-25T00:11:52.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.610+0000 7fe193279640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe18c072ca0 0x7fe18c0ac1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:52.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.610+0000 7fe193279640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe18c0c5550 0x7fe18c06f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:52.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.610+0000 7fe193279640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe18c150d10 0x7fe18c146b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:52.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.610+0000 7fe193279640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe18c0c2640 con 0x7fe18c072ca0 2024-04-25T00:11:52.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.610+0000 7fe193279640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe18c1576a0 con 0x7fe18c0c5550 2024-04-25T00:11:52.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.610+0000 7fe193279640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe18c0c27c0 con 0x7fe18c150d10 2024-04-25T00:11:52.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.610+0000 7fe191015640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe18c072ca0 0x7fe18c0ac1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:52.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.610+0000 7fe191015640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe18c072ca0 0x7fe18c0ac1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51538/0 (socket says 172.21.15.116:51538) 2024-04-25T00:11:52.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.610+0000 7fe191015640 1 -- 172.21.15.116:0/2246678936 learned_addr learned my addr 172.21.15.116:0/2246678936 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:11:52.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.614+0000 7fe191816640 1 --2- 172.21.15.116:0/2246678936 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe18c150d10 0x7fe18c146b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:52.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.614+0000 7fe190814640 1 --2- 172.21.15.116:0/2246678936 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe18c0c5550 0x7fe18c06f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:52.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.614+0000 7fe191015640 1 -- 172.21.15.116:0/2246678936 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe18c150d10 msgr2=0x7fe18c146b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:52.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.614+0000 7fe191015640 1 --2- 172.21.15.116:0/2246678936 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe18c150d10 0x7fe18c146b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:52.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.614+0000 7fe191015640 1 -- 172.21.15.116:0/2246678936 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe18c0c5550 msgr2=0x7fe18c06f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:52.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.614+0000 7fe191015640 1 --2- 172.21.15.116:0/2246678936 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe18c0c5550 0x7fe18c06f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:52.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.614+0000 7fe191015640 1 -- 172.21.15.116:0/2246678936 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe18c0ab640 con 0x7fe18c072ca0 2024-04-25T00:11:52.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.614+0000 7fe191015640 1 --2- 172.21.15.116:0/2246678936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe18c072ca0 0x7fe18c0ac1d0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fe1880ad040 tx=0x7fe18801ac80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:11:52.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.614+0000 7fe181ffb640 1 -- 172.21.15.116:0/2246678936 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1880be5f0 con 0x7fe18c072ca0 2024-04-25T00:11:52.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.614+0000 7fe181ffb640 1 -- 172.21.15.116:0/2246678936 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1880be7b0 con 0x7fe18c072ca0 2024-04-25T00:11:52.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.614+0000 7fe193279640 1 -- 172.21.15.116:0/2246678936 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe18c1ef150 con 0x7fe18c072ca0 2024-04-25T00:11:52.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.614+0000 7fe181ffb640 1 -- 172.21.15.116:0/2246678936 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1880beba0 con 0x7fe18c072ca0 2024-04-25T00:11:52.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.614+0000 7fe193279640 1 -- 172.21.15.116:0/2246678936 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe18c1ee390 con 0x7fe18c072ca0 2024-04-25T00:11:52.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.614+0000 7fe193279640 1 -- 172.21.15.116:0/2246678936 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe18c068a60 con 0x7fe18c072ca0 2024-04-25T00:11:52.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.614+0000 7fe181ffb640 1 -- 172.21.15.116:0/2246678936 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe1880b43f0 con 0x7fe18c072ca0 2024-04-25T00:11:52.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.614+0000 7fe181ffb640 1 --2- 172.21.15.116:0/2246678936 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe170067910 0x7fe170069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:52.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.614+0000 7fe181ffb640 1 -- 172.21.15.116:0/2246678936 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7870+0+0 (secure 0 0 0) 0x7fe188121090 con 0x7fe18c072ca0 2024-04-25T00:11:52.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.614+0000 7fe190814640 1 --2- 172.21.15.116:0/2246678936 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe170067910 0x7fe170069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:52.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.618+0000 7fe190814640 1 --2- 172.21.15.116:0/2246678936 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe170067910 0x7fe170069dd0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fe18c157400 tx=0x7fe17c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:11:52.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.622+0000 7fe181ffb640 1 -- 172.21.15.116:0/2246678936 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fe18c0ab640 con 0x7fe18c072ca0 2024-04-25T00:11:52.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.854+0000 7fe193279640 1 -- 172.21.15.116:0/2246678936 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe18c072a80 con 0x7fe18c072ca0 2024-04-25T00:11:52.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.854+0000 7fe181ffb640 1 -- 172.21.15.116:0/2246678936 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) ==== 94+0+5668 (secure 0 0 0) 0x7fe188016150 con 0x7fe18c072ca0 2024-04-25T00:11:52.853 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:11:52.853 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5461,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:11:03.640578+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6114},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6114":{"gid":6114,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":25,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6114,"qdb_cluster":[6114]},"id":1},{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:11:51.787120+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6309},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6309":{"gid":6309,"name":"c","rank":0,"incarnation":78,"state":"up:rejoin","state_seq":14,"addr":"172.21.15.116:6835/1599708526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1599708526},{"type":"v1","addr":"172.21.15.116:6835","nonce":1599708526}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:11:52.853 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 80 2024-04-25T00:11:52.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.854+0000 7fe193279640 1 -- 172.21.15.116:0/2246678936 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe170067910 msgr2=0x7fe170069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:52.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.854+0000 7fe193279640 1 --2- 172.21.15.116:0/2246678936 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe170067910 0x7fe170069dd0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fe18c157400 tx=0x7fe17c009040 comp rx=0 tx=0).stop 2024-04-25T00:11:52.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.854+0000 7fe193279640 1 -- 172.21.15.116:0/2246678936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe18c072ca0 msgr2=0x7fe18c0ac1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:52.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.854+0000 7fe193279640 1 --2- 172.21.15.116:0/2246678936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe18c072ca0 0x7fe18c0ac1d0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fe1880ad040 tx=0x7fe18801ac80 comp rx=0 tx=0).stop 2024-04-25T00:11:52.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.854+0000 7fe193279640 1 -- 172.21.15.116:0/2246678936 shutdown_connections 2024-04-25T00:11:52.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.854+0000 7fe193279640 1 --2- 172.21.15.116:0/2246678936 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe170067910 0x7fe170069dd0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:52.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.854+0000 7fe193279640 1 --2- 172.21.15.116:0/2246678936 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe18c150d10 0x7fe18c146b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:52.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.854+0000 7fe193279640 1 --2- 172.21.15.116:0/2246678936 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe18c0c5550 0x7fe18c06f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:52.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.854+0000 7fe193279640 1 --2- 172.21.15.116:0/2246678936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe18c072ca0 0x7fe18c0ac1d0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:52.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.854+0000 7fe193279640 1 -- 172.21.15.116:0/2246678936 >> 172.21.15.116:0/2246678936 conn(0x7fe18c06c720 msgr2=0x7fe18c15d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:11:52.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.854+0000 7fe193279640 1 -- 172.21.15.116:0/2246678936 shutdown_connections 2024-04-25T00:11:52.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:52.854+0000 7fe193279640 1 -- 172.21.15.116:0/2246678936 wait complete. 2024-04-25T00:11:52.876 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 80, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:04.678737+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 76, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6114": { "addr": "172.21.15.116:6833/3705654249", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6832", "nonce": 3705654249, "type": "v2" }, { "addr": "172.21.15.116:6833", "nonce": 3705654249, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6114, "incarnation": 73, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 25 } }, "last_failure": 0, "last_failure_osd_epoch": 41, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T00:11:03.640578+0000", "qdb_cluster": [ 6114 ], "qdb_leader": 6114, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6114 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:18.825887+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 80, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6309": { "addr": "172.21.15.116:6835/1599708526", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6834", "nonce": 1599708526, "type": "v2" }, { "addr": "172.21.15.116:6835", "nonce": 1599708526, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6309, "incarnation": 78, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:rejoin", "state_seq": 14 } }, "last_failure": 0, "last_failure_osd_epoch": 42, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-04-25T00:11:51.787120+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6309 } } } ], "standbys": [ { "addr": "172.21.15.174:6837/3398690896", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6836", "nonce": 3398690896, "type": "v2" }, { "addr": "172.21.15.174:6837", "nonce": 3398690896, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 72, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5461, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-25T00:11:54.877 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:11:54.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:54.998+0000 7f78fc6e2640 1 Processor -- start 2024-04-25T00:11:54.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:54.998+0000 7f78fc6e2640 1 -- start start 2024-04-25T00:11:54.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:54.998+0000 7f78fc6e2640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f78f4149f60 0x7f78f4153fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:54.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:54.998+0000 7f78fc6e2640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f78f4070ca0 0x7f78f4071110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:54.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:54.998+0000 7f78fc6e2640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78f40c8360 con 0x7f78f4154a60 2024-04-25T00:11:54.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:54.998+0000 7f78fc6e2640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f78f40c8620 con 0x7f78f4149f60 2024-04-25T00:11:54.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:54.998+0000 7f78fc6e2640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f78f40c8900 con 0x7f78f4070ca0 2024-04-25T00:11:54.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:54.998+0000 7f78fac7f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f78f4070ca0 0x7f78f4071110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:54.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:54.998+0000 7f78fa47e640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f78f4149f60 0x7f78f4153fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:55.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:54.998+0000 7f78f9c7d640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f78f4154a60 0x7f78f4157220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:55636/0 (socket says 172.21.15.116:55636) 2024-04-25T00:11:55.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:54.998+0000 7f78f9c7d640 1 -- 172.21.15.116:0/1817978020 learned_addr learned my addr 172.21.15.116:0/1817978020 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:11:55.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.002+0000 7f78fac7f640 1 -- 172.21.15.116:0/1817978020 >> v2:172.21.15.174:3300/0 conn(0x7f78f4149f60 msgr2=0x7f78f4153fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:11:55.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.002+0000 7f78fac7f640 1 --2- 172.21.15.116:0/1817978020 >> v2:172.21.15.174:3300/0 conn(0x7f78f4149f60 0x7f78f4153fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:55.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.002+0000 7f78fac7f640 1 -- 172.21.15.116:0/1817978020 >> v1:172.21.15.174:6789/0 conn(0x7f78f4154a60 legacy=0x7f78f4157220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:55.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.002+0000 7f78fac7f640 1 -- 172.21.15.116:0/1817978020 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78f4156d00 con 0x7f78f4070ca0 2024-04-25T00:11:55.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.002+0000 7f78fa47e640 1 --2- 172.21.15.116:0/1817978020 >> v2:172.21.15.174:3300/0 conn(0x7f78f4149f60 0x7f78f4153fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:11:55.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.002+0000 7f78fac7f640 1 --2- 172.21.15.116:0/1817978020 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f78f4070ca0 0x7f78f4071110 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f78ec009d90 tx=0x7f78ec01c8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83df3b7959f86ec9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:11:55.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.002+0000 7f78f947c640 1 -- 172.21.15.116:0/1817978020 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78ec0b2070 con 0x7f78f4070ca0 2024-04-25T00:11:55.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.002+0000 7f78f947c640 1 -- 172.21.15.116:0/1817978020 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78ec0a58b0 con 0x7f78f4070ca0 2024-04-25T00:11:55.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.002+0000 7f78f947c640 1 -- 172.21.15.116:0/1817978020 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78ec0a5c80 con 0x7f78f4070ca0 2024-04-25T00:11:55.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.002+0000 7f78fc6e2640 1 -- 172.21.15.116:0/1817978020 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f78f4070ca0 msgr2=0x7f78f4071110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:55.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.002+0000 7f78fc6e2640 1 --2- 172.21.15.116:0/1817978020 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f78f4070ca0 0x7f78f4071110 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f78ec009d90 tx=0x7f78ec01c8e0 comp rx=0 tx=0).stop 2024-04-25T00:11:55.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.002+0000 7f78fc6e2640 1 -- 172.21.15.116:0/1817978020 shutdown_connections 2024-04-25T00:11:55.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.002+0000 7f78fc6e2640 1 --2- 172.21.15.116:0/1817978020 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f78f4070ca0 0x7f78f4071110 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:55.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.002+0000 7f78fc6e2640 1 --2- 172.21.15.116:0/1817978020 >> v2:172.21.15.174:3300/0 conn(0x7f78f4149f60 0x7f78f4153fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:55.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.002+0000 7f78fc6e2640 1 -- 172.21.15.116:0/1817978020 >> 172.21.15.116:0/1817978020 conn(0x7f78f4074a80 msgr2=0x7f78f40c8f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:11:55.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.002+0000 7f78fc6e2640 1 -- 172.21.15.116:0/1817978020 shutdown_connections 2024-04-25T00:11:55.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.002+0000 7f78fc6e2640 1 -- 172.21.15.116:0/1817978020 wait complete. 2024-04-25T00:11:55.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.006+0000 7f78fc6e2640 1 Processor -- start 2024-04-25T00:11:55.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.006+0000 7f78fc6e2640 1 -- start start 2024-04-25T00:11:55.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.006+0000 7f78fc6e2640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f78f4070ca0 0x7f78f41ed4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:55.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.006+0000 7f78fc6e2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78f4149f60 0x7f78f4152b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:55.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.006+0000 7f78fc6e2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f78f4154a60 0x7f78f41ec8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:55.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.006+0000 7f78fc6e2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f78f40ae100 con 0x7f78f4149f60 2024-04-25T00:11:55.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.006+0000 7f78fc6e2640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f78f40adf80 con 0x7f78f4154a60 2024-04-25T00:11:55.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.006+0000 7f78fc6e2640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f78f40ae280 con 0x7f78f4070ca0 2024-04-25T00:11:55.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.006+0000 7f78f9c7d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78f4149f60 0x7f78f4152b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:55.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.006+0000 7f78f9c7d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78f4149f60 0x7f78f4152b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57658/0 (socket says 172.21.15.116:57658) 2024-04-25T00:11:55.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.006+0000 7f78f9c7d640 1 -- 172.21.15.116:0/550510245 learned_addr learned my addr 172.21.15.116:0/550510245 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:11:55.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.006+0000 7f78fac7f640 1 --2- 172.21.15.116:0/550510245 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f78f4154a60 0x7f78f41ec8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:55.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.006+0000 7f78fa47e640 1 --2- 172.21.15.116:0/550510245 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f78f4070ca0 0x7f78f41ed4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:55.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.006+0000 7f78f9c7d640 1 -- 172.21.15.116:0/550510245 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f78f4070ca0 msgr2=0x7f78f41ed4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:55.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.006+0000 7f78f9c7d640 1 --2- 172.21.15.116:0/550510245 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f78f4070ca0 0x7f78f41ed4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:55.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.006+0000 7f78f9c7d640 1 -- 172.21.15.116:0/550510245 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f78f4154a60 msgr2=0x7f78f41ec8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:55.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.006+0000 7f78f9c7d640 1 --2- 172.21.15.116:0/550510245 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f78f4154a60 0x7f78f41ec8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:55.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.006+0000 7f78f9c7d640 1 -- 172.21.15.116:0/550510245 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78f40c5c40 con 0x7f78f4149f60 2024-04-25T00:11:55.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.006+0000 7f78f9c7d640 1 --2- 172.21.15.116:0/550510245 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78f4149f60 0x7f78f4152b40 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f78e400c370 tx=0x7f78e400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:11:55.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.006+0000 7f78eaffd640 1 -- 172.21.15.116:0/550510245 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e4017070 con 0x7f78f4149f60 2024-04-25T00:11:55.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.006+0000 7f78eaffd640 1 -- 172.21.15.116:0/550510245 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78e4005cf0 con 0x7f78f4149f60 2024-04-25T00:11:55.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.006+0000 7f78fc6e2640 1 -- 172.21.15.116:0/550510245 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78f40c5f20 con 0x7f78f4149f60 2024-04-25T00:11:55.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.006+0000 7f78eaffd640 1 -- 172.21.15.116:0/550510245 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e40072f0 con 0x7f78f4149f60 2024-04-25T00:11:55.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.006+0000 7f78fc6e2640 1 -- 172.21.15.116:0/550510245 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78f41ec090 con 0x7f78f4149f60 2024-04-25T00:11:55.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.006+0000 7f78fc6e2640 1 -- 172.21.15.116:0/550510245 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78f4068a60 con 0x7f78f4149f60 2024-04-25T00:11:55.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.010+0000 7f78eaffd640 1 -- 172.21.15.116:0/550510245 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f78e401a070 con 0x7f78f4149f60 2024-04-25T00:11:55.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.010+0000 7f78eaffd640 1 --2- 172.21.15.116:0/550510245 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f78d80678f0 0x7f78d8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:55.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.010+0000 7f78eaffd640 1 -- 172.21.15.116:0/550510245 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7870+0+0 (secure 0 0 0) 0x7f78e40857d0 con 0x7f78f4149f60 2024-04-25T00:11:55.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.010+0000 7f78fa47e640 1 --2- 172.21.15.116:0/550510245 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f78d80678f0 0x7f78d8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:55.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.010+0000 7f78fa47e640 1 --2- 172.21.15.116:0/550510245 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f78d80678f0 0x7f78d8069db0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f78f0009580 tx=0x7f78f0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:11:55.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.014+0000 7f78eaffd640 1 -- 172.21.15.116:0/550510245 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f78e4055b30 con 0x7f78f4149f60 2024-04-25T00:11:55.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.246+0000 7f78fc6e2640 1 -- 172.21.15.116:0/550510245 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f78f40c8140 con 0x7f78f4149f60 2024-04-25T00:11:55.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.250+0000 7f78eaffd640 1 -- 172.21.15.116:0/550510245 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) ==== 94+0+5675 (secure 0 0 0) 0x7f78e4055430 con 0x7f78f4149f60 2024-04-25T00:11:55.248 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:11:55.248 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":81,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5461,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:11:03.640578+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6114},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6114":{"gid":6114,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":25,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6114,"qdb_cluster":[6114]},"id":1},{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:11:53.790738+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6309},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6309":{"gid":6309,"name":"c","rank":0,"incarnation":78,"state":"up:active","state_seq":15,"addr":"172.21.15.116:6835/1599708526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1599708526},{"type":"v1","addr":"172.21.15.116:6835","nonce":1599708526}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6309,"qdb_cluster":[6309]},"id":2}]} 2024-04-25T00:11:55.248 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 81 2024-04-25T00:11:55.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.250+0000 7f78fc6e2640 1 -- 172.21.15.116:0/550510245 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f78d80678f0 msgr2=0x7f78d8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:55.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.250+0000 7f78fc6e2640 1 --2- 172.21.15.116:0/550510245 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f78d80678f0 0x7f78d8069db0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f78f0009580 tx=0x7f78f0008040 comp rx=0 tx=0).stop 2024-04-25T00:11:55.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.250+0000 7f78fc6e2640 1 -- 172.21.15.116:0/550510245 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78f4149f60 msgr2=0x7f78f4152b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:55.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.250+0000 7f78fc6e2640 1 --2- 172.21.15.116:0/550510245 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78f4149f60 0x7f78f4152b40 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f78e400c370 tx=0x7f78e400c830 comp rx=0 tx=0).stop 2024-04-25T00:11:55.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.250+0000 7f78fc6e2640 1 -- 172.21.15.116:0/550510245 shutdown_connections 2024-04-25T00:11:55.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.250+0000 7f78fc6e2640 1 --2- 172.21.15.116:0/550510245 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f78d80678f0 0x7f78d8069db0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:55.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.250+0000 7f78fc6e2640 1 --2- 172.21.15.116:0/550510245 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f78f4154a60 0x7f78f41ec8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:55.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.250+0000 7f78fc6e2640 1 --2- 172.21.15.116:0/550510245 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f78f4149f60 0x7f78f4152b40 secure :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f78e400c370 tx=0x7f78e400c830 comp rx=0 tx=0).stop 2024-04-25T00:11:55.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.250+0000 7f78fc6e2640 1 --2- 172.21.15.116:0/550510245 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f78f4070ca0 0x7f78f41ed4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:55.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.250+0000 7f78fc6e2640 1 -- 172.21.15.116:0/550510245 >> 172.21.15.116:0/550510245 conn(0x7f78f4074a80 msgr2=0x7f78f406ec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:11:55.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.250+0000 7f78fc6e2640 1 -- 172.21.15.116:0/550510245 shutdown_connections 2024-04-25T00:11:55.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:55.250+0000 7f78fc6e2640 1 -- 172.21.15.116:0/550510245 wait complete. 2024-04-25T00:11:55.269 INFO:tasks.mds_thrash.fs.[b]:mds.c has gained rank=0, replacing gid=4235 2024-04-25T00:11:55.270 INFO:tasks.mds_thrash.fs.[b]:waiting for 3 secs before reviving mds.d 2024-04-25T00:11:55.664 INFO:tasks.ceph.mds.c.smithi116.stderr:2024-04-25T00:11:55.666+0000 7fabcc9a3640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T00:11:58.270 INFO:tasks.mds_thrash.fs.[b]:reviving mds.d 2024-04-25T00:11:58.270 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-25T00:11:58.271 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-25T00:11:58.273 INFO:tasks.ceph.mds.d:Started 2024-04-25T00:11:58.273 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:11:58.364 INFO:tasks.ceph.mds.d.smithi174.stdout:starting mds.d at 2024-04-25T00:11:58.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.406+0000 7fbb39872640 1 Processor -- start 2024-04-25T00:11:58.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.406+0000 7fbb39872640 1 -- start start 2024-04-25T00:11:58.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.406+0000 7fbb39872640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fbb34150d10 0x7fbb340c5010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:58.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.406+0000 7fbb39872640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb340c5550 0x7fbb34153a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:58.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.406+0000 7fbb39872640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb340c4950 con 0x7fbb34072ca0 2024-04-25T00:11:58.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.406+0000 7fbb39872640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fbb341546d0 con 0x7fbb34150d10 2024-04-25T00:11:58.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.406+0000 7fbb39872640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fbb341510f0 con 0x7fbb340c5550 2024-04-25T00:11:58.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.406+0000 7fbb337fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fbb34072ca0 0x7fbb34153fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:60724/0 (socket says 172.21.15.116:60724) 2024-04-25T00:11:58.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.406+0000 7fbb337fe640 1 -- 172.21.15.116:0/22977810 learned_addr learned my addr 172.21.15.116:0/22977810 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:11:58.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.406+0000 7fbb327fc640 1 --2- 172.21.15.116:0/22977810 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb340c5550 0x7fbb34153a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:58.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.406+0000 7fbb32ffd640 1 --2- 172.21.15.116:0/22977810 >> v2:172.21.15.174:3300/0 conn(0x7fbb34150d10 0x7fbb340c5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:58.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.406+0000 7fbb31ffb640 1 -- 172.21.15.116:0/22977810 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 889759903 0 0) 0x7fbb340c4950 con 0x7fbb34072ca0 2024-04-25T00:11:58.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.406+0000 7fbb31ffb640 1 -- 172.21.15.116:0/22977810 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb1c003530 con 0x7fbb34072ca0 2024-04-25T00:11:58.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.410+0000 7fbb32ffd640 1 -- 172.21.15.116:0/22977810 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb340c5550 msgr2=0x7fbb34153a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:11:58.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.410+0000 7fbb32ffd640 1 --2- 172.21.15.116:0/22977810 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb340c5550 0x7fbb34153a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:58.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.410+0000 7fbb32ffd640 1 -- 172.21.15.116:0/22977810 >> v1:172.21.15.116:6789/0 conn(0x7fbb34072ca0 legacy=0x7fbb34153fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:58.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.410+0000 7fbb32ffd640 1 -- 172.21.15.116:0/22977810 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb34157a20 con 0x7fbb34150d10 2024-04-25T00:11:58.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.410+0000 7fbb327fc640 1 --2- 172.21.15.116:0/22977810 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb340c5550 0x7fbb34153a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:11:58.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.410+0000 7fbb32ffd640 1 --2- 172.21.15.116:0/22977810 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb34150d10 0x7fbb340c5010 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fbb34157bf0 tx=0x7fbb2401c380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68b0501c7bf7715e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:11:58.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.410+0000 7fbb31ffb640 1 -- 172.21.15.116:0/22977810 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb240b2070 con 0x7fbb34150d10 2024-04-25T00:11:58.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.410+0000 7fbb31ffb640 1 -- 172.21.15.116:0/22977810 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb240ad680 con 0x7fbb34150d10 2024-04-25T00:11:58.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.410+0000 7fbb31ffb640 1 -- 172.21.15.116:0/22977810 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb240ada50 con 0x7fbb34150d10 2024-04-25T00:11:58.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.410+0000 7fbb39872640 1 -- 172.21.15.116:0/22977810 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb34150d10 msgr2=0x7fbb340c5010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:58.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.410+0000 7fbb39872640 1 --2- 172.21.15.116:0/22977810 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb34150d10 0x7fbb340c5010 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fbb34157bf0 tx=0x7fbb2401c380 comp rx=0 tx=0).stop 2024-04-25T00:11:58.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.410+0000 7fbb39872640 1 -- 172.21.15.116:0/22977810 shutdown_connections 2024-04-25T00:11:58.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.410+0000 7fbb39872640 1 --2- 172.21.15.116:0/22977810 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb340c5550 0x7fbb34153a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:58.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.410+0000 7fbb39872640 1 --2- 172.21.15.116:0/22977810 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb34150d10 0x7fbb340c5010 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:58.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.410+0000 7fbb39872640 1 -- 172.21.15.116:0/22977810 >> 172.21.15.116:0/22977810 conn(0x7fbb3406c720 msgr2=0x7fbb3406cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:11:58.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.410+0000 7fbb39872640 1 -- 172.21.15.116:0/22977810 shutdown_connections 2024-04-25T00:11:58.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.410+0000 7fbb39872640 1 -- 172.21.15.116:0/22977810 wait complete. 2024-04-25T00:11:58.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.414+0000 7fbb39872640 1 Processor -- start 2024-04-25T00:11:58.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.414+0000 7fbb39872640 1 -- start start 2024-04-25T00:11:58.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.414+0000 7fbb39872640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb34072ca0 0x7fbb341407d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:58.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.414+0000 7fbb39872640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb340c5550 0x7fbb34140d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:58.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.414+0000 7fbb39872640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb34150d10 0x7fbb3406ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:58.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.414+0000 7fbb39872640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbb340c2640 con 0x7fbb34072ca0 2024-04-25T00:11:58.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.414+0000 7fbb39872640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbb341576a0 con 0x7fbb340c5550 2024-04-25T00:11:58.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.414+0000 7fbb39872640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fbb340c27c0 con 0x7fbb34150d10 2024-04-25T00:11:58.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.414+0000 7fbb32ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb34072ca0 0x7fbb341407d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:58.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.414+0000 7fbb32ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb34072ca0 0x7fbb341407d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57674/0 (socket says 172.21.15.116:57674) 2024-04-25T00:11:58.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.414+0000 7fbb32ffd640 1 -- 172.21.15.116:0/3758868722 learned_addr learned my addr 172.21.15.116:0/3758868722 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:11:58.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.414+0000 7fbb327fc640 1 --2- 172.21.15.116:0/3758868722 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb340c5550 0x7fbb34140d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:58.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.414+0000 7fbb337fe640 1 --2- 172.21.15.116:0/3758868722 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb34150d10 0x7fbb3406ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:58.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.414+0000 7fbb32ffd640 1 -- 172.21.15.116:0/3758868722 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb34150d10 msgr2=0x7fbb3406ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:58.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.414+0000 7fbb32ffd640 1 --2- 172.21.15.116:0/3758868722 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb34150d10 0x7fbb3406ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:58.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.414+0000 7fbb32ffd640 1 -- 172.21.15.116:0/3758868722 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb340c5550 msgr2=0x7fbb34140d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:58.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.414+0000 7fbb32ffd640 1 --2- 172.21.15.116:0/3758868722 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb340c5550 0x7fbb34140d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:58.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.414+0000 7fbb32ffd640 1 -- 172.21.15.116:0/3758868722 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb341535f0 con 0x7fbb34072ca0 2024-04-25T00:11:58.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.414+0000 7fbb32ffd640 1 --2- 172.21.15.116:0/3758868722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb34072ca0 0x7fbb341407d0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fbb24019fb0 tx=0x7fbb2401abc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:11:58.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.414+0000 7fbb1bfff640 1 -- 172.21.15.116:0/3758868722 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb240bc020 con 0x7fbb34072ca0 2024-04-25T00:11:58.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.414+0000 7fbb1bfff640 1 -- 172.21.15.116:0/3758868722 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb240b8420 con 0x7fbb34072ca0 2024-04-25T00:11:58.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.414+0000 7fbb39872640 1 -- 172.21.15.116:0/3758868722 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb341553e0 con 0x7fbb34072ca0 2024-04-25T00:11:58.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.414+0000 7fbb1bfff640 1 -- 172.21.15.116:0/3758868722 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb240b2040 con 0x7fbb34072ca0 2024-04-25T00:11:58.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.414+0000 7fbb39872640 1 -- 172.21.15.116:0/3758868722 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb34154d30 con 0x7fbb34072ca0 2024-04-25T00:11:58.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.414+0000 7fbb39872640 1 -- 172.21.15.116:0/3758868722 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb34068a60 con 0x7fbb34072ca0 2024-04-25T00:11:58.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.418+0000 7fbb1bfff640 1 -- 172.21.15.116:0/3758868722 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbb240bf020 con 0x7fbb34072ca0 2024-04-25T00:11:58.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.418+0000 7fbb1bfff640 1 --2- 172.21.15.116:0/3758868722 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbb200678c0 0x7fbb20069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:11:58.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.418+0000 7fbb1bfff640 1 -- 172.21.15.116:0/3758868722 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7870+0+0 (secure 0 0 0) 0x7fbb24120870 con 0x7fbb34072ca0 2024-04-25T00:11:58.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.418+0000 7fbb327fc640 1 --2- 172.21.15.116:0/3758868722 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbb200678c0 0x7fbb20069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:11:58.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.418+0000 7fbb327fc640 1 --2- 172.21.15.116:0/3758868722 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbb200678c0 0x7fbb20069d80 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fbb28009580 tx=0x7fbb28009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:11:58.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.422+0000 7fbb1bfff640 1 -- 172.21.15.116:0/3758868722 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fbb240c6030 con 0x7fbb34072ca0 2024-04-25T00:11:58.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.662+0000 7fbb39872640 1 -- 172.21.15.116:0/3758868722 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbb340c2420 con 0x7fbb34072ca0 2024-04-25T00:11:58.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.662+0000 7fbb1bfff640 1 -- 172.21.15.116:0/3758868722 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) ==== 94+0+5675 (secure 0 0 0) 0x7fbb24016090 con 0x7fbb34072ca0 2024-04-25T00:11:58.662 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:11:58.662 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":81,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5461,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:11:03.640578+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6114},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6114":{"gid":6114,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":25,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6114,"qdb_cluster":[6114]},"id":1},{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:11:53.790738+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6309},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6309":{"gid":6309,"name":"c","rank":0,"incarnation":78,"state":"up:active","state_seq":15,"addr":"172.21.15.116:6835/1599708526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1599708526},{"type":"v1","addr":"172.21.15.116:6835","nonce":1599708526}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6309,"qdb_cluster":[6309]},"id":2}]} 2024-04-25T00:11:58.662 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 81 2024-04-25T00:11:58.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.666+0000 7fbb39872640 1 -- 172.21.15.116:0/3758868722 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbb200678c0 msgr2=0x7fbb20069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:58.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.666+0000 7fbb39872640 1 --2- 172.21.15.116:0/3758868722 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbb200678c0 0x7fbb20069d80 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fbb28009580 tx=0x7fbb28009190 comp rx=0 tx=0).stop 2024-04-25T00:11:58.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.666+0000 7fbb39872640 1 -- 172.21.15.116:0/3758868722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb34072ca0 msgr2=0x7fbb341407d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:11:58.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.666+0000 7fbb39872640 1 --2- 172.21.15.116:0/3758868722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb34072ca0 0x7fbb341407d0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fbb24019fb0 tx=0x7fbb2401abc0 comp rx=0 tx=0).stop 2024-04-25T00:11:58.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.666+0000 7fbb39872640 1 -- 172.21.15.116:0/3758868722 shutdown_connections 2024-04-25T00:11:58.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.666+0000 7fbb39872640 1 --2- 172.21.15.116:0/3758868722 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbb200678c0 0x7fbb20069d80 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:58.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.666+0000 7fbb39872640 1 --2- 172.21.15.116:0/3758868722 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb34150d10 0x7fbb3406ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:58.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.666+0000 7fbb39872640 1 --2- 172.21.15.116:0/3758868722 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb340c5550 0x7fbb34140d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:58.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.666+0000 7fbb39872640 1 --2- 172.21.15.116:0/3758868722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb34072ca0 0x7fbb341407d0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:11:58.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.666+0000 7fbb39872640 1 -- 172.21.15.116:0/3758868722 >> 172.21.15.116:0/3758868722 conn(0x7fbb3406c720 msgr2=0x7fbb3414f860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:11:58.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.666+0000 7fbb39872640 1 -- 172.21.15.116:0/3758868722 shutdown_connections 2024-04-25T00:11:58.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:11:58.666+0000 7fbb39872640 1 -- 172.21.15.116:0/3758868722 wait complete. 2024-04-25T00:11:58.681 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2024-04-25T00:12:00.682 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:12:00.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:00.818+0000 7fb588887640 1 Processor -- start 2024-04-25T00:12:00.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:00.818+0000 7fb588887640 1 -- start start 2024-04-25T00:12:00.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:00.822+0000 7fb588887640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb5840c4c20 0x7fb584154170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:00.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:00.822+0000 7fb588887640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb584070180 con 0x7fb584070770 2024-04-25T00:12:00.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:00.822+0000 7fb588887640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb584070460 con 0x7fb584150d40 2024-04-25T00:12:00.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:00.822+0000 7fb588887640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fb5840c7530 con 0x7fb5840c4c20 2024-04-25T00:12:00.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:00.822+0000 7fb57bfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb5840c4c20 0x7fb584154170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:00.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:00.822+0000 7fb582d9d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fb584070770 0x7fb5840ab200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:60732/0 (socket says 172.21.15.116:60732) 2024-04-25T00:12:00.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:00.822+0000 7fb57bfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb5840c4c20 0x7fb584154170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57680/0 (socket says 172.21.15.116:57680) 2024-04-25T00:12:00.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:00.822+0000 7fb582d9d640 1 -- 172.21.15.116:0/2329325523 learned_addr learned my addr 172.21.15.116:0/2329325523 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:12:00.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:00.822+0000 7fb57bfff640 1 -- 172.21.15.116:0/2329325523 >> v1:172.21.15.174:6789/0 conn(0x7fb584150d40 legacy=0x7fb5840c6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:00.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:00.822+0000 7fb57bfff640 1 -- 172.21.15.116:0/2329325523 >> v1:172.21.15.116:6789/0 conn(0x7fb584070770 legacy=0x7fb5840ab200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:00.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:00.822+0000 7fb581d9b640 1 -- 172.21.15.116:0/2329325523 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2945476941 0 0) 0x7fb584070180 con 0x7fb584070770 2024-04-25T00:12:00.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:00.822+0000 7fb57bfff640 1 -- 172.21.15.116:0/2329325523 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb584153970 con 0x7fb5840c4c20 2024-04-25T00:12:00.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:00.822+0000 7fb57bfff640 1 --2- 172.21.15.116:0/2329325523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5840c4c20 0x7fb584154170 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fb56c0097c0 tx=0x7fb56c01c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ae1f747e6dcb6a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:00.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:00.822+0000 7fb581d9b640 1 -- 172.21.15.116:0/2329325523 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb56c0b0070 con 0x7fb5840c4c20 2024-04-25T00:12:00.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:00.822+0000 7fb581d9b640 1 -- 172.21.15.116:0/2329325523 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb56c0ab600 con 0x7fb5840c4c20 2024-04-25T00:12:00.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:00.822+0000 7fb581d9b640 1 -- 172.21.15.116:0/2329325523 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb56c0ab9d0 con 0x7fb5840c4c20 2024-04-25T00:12:00.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:00.826+0000 7fb588887640 1 -- 172.21.15.116:0/2329325523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5840c4c20 msgr2=0x7fb584154170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:00.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:00.826+0000 7fb588887640 1 --2- 172.21.15.116:0/2329325523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5840c4c20 0x7fb584154170 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fb56c0097c0 tx=0x7fb56c01c410 comp rx=0 tx=0).stop 2024-04-25T00:12:00.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:00.826+0000 7fb588887640 1 -- 172.21.15.116:0/2329325523 shutdown_connections 2024-04-25T00:12:00.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:00.826+0000 7fb588887640 1 --2- 172.21.15.116:0/2329325523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5840c4c20 0x7fb584154170 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:00.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:00.826+0000 7fb588887640 1 -- 172.21.15.116:0/2329325523 >> 172.21.15.116:0/2329325523 conn(0x7fb58406f0c0 msgr2=0x7fb58406f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:00.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:00.826+0000 7fb588887640 1 -- 172.21.15.116:0/2329325523 shutdown_connections 2024-04-25T00:12:00.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:00.826+0000 7fb588887640 1 -- 172.21.15.116:0/2329325523 wait complete. 2024-04-25T00:12:00.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:00.826+0000 7fb588887640 1 Processor -- start 2024-04-25T00:12:00.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:00.826+0000 7fb588887640 1 -- start start 2024-04-25T00:12:00.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:00.826+0000 7fb588887640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb584070770 0x7fb58414cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:00.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:00.826+0000 7fb588887640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5840c4c20 0x7fb5840ad280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:00.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:00.826+0000 7fb588887640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb584150d40 0x7fb58414b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:00.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:00.826+0000 7fb588887640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb584151120 con 0x7fb5840c4c20 2024-04-25T00:12:00.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:00.826+0000 7fb588887640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb58406ff30 con 0x7fb584070770 2024-04-25T00:12:00.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:00.826+0000 7fb588887640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fb5841512a0 con 0x7fb584150d40 2024-04-25T00:12:00.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:00.826+0000 7fb57bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5840c4c20 0x7fb5840ad280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:00.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:00.826+0000 7fb57bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5840c4c20 0x7fb5840ad280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57684/0 (socket says 172.21.15.116:57684) 2024-04-25T00:12:00.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:00.826+0000 7fb57bfff640 1 -- 172.21.15.116:0/2130026191 learned_addr learned my addr 172.21.15.116:0/2130026191 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:12:00.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:00.830+0000 7fb57bfff640 1 -- 172.21.15.116:0/2130026191 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb584150d40 msgr2=0x7fb58414b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:00.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:00.830+0000 7fb57bfff640 1 --2- 172.21.15.116:0/2130026191 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb584150d40 0x7fb58414b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:00.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:00.830+0000 7fb57bfff640 1 -- 172.21.15.116:0/2130026191 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb584070770 msgr2=0x7fb58414cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:00.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:00.830+0000 7fb57bfff640 1 --2- 172.21.15.116:0/2130026191 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb584070770 0x7fb58414cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:00.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:00.830+0000 7fb57bfff640 1 -- 172.21.15.116:0/2130026191 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5841f0200 con 0x7fb5840c4c20 2024-04-25T00:12:00.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:00.830+0000 7fb57bfff640 1 --2- 172.21.15.116:0/2130026191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5840c4c20 0x7fb5840ad280 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fb56c019020 tx=0x7fb56c0191b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:00.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:00.830+0000 7fb57affd640 1 -- 172.21.15.116:0/2130026191 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb56c0b8030 con 0x7fb5840c4c20 2024-04-25T00:12:00.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:00.830+0000 7fb57affd640 1 -- 172.21.15.116:0/2130026191 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb56c0b4410 con 0x7fb5840c4c20 2024-04-25T00:12:00.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:00.830+0000 7fb588887640 1 -- 172.21.15.116:0/2130026191 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5841f04e0 con 0x7fb5840c4c20 2024-04-25T00:12:00.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:00.830+0000 7fb57affd640 1 -- 172.21.15.116:0/2130026191 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb56c0b0040 con 0x7fb5840c4c20 2024-04-25T00:12:00.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:00.830+0000 7fb588887640 1 -- 172.21.15.116:0/2130026191 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb58414faf0 con 0x7fb5840c4c20 2024-04-25T00:12:00.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:00.834+0000 7fb588887640 1 -- 172.21.15.116:0/2130026191 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb584068a60 con 0x7fb5840c4c20 2024-04-25T00:12:00.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:00.834+0000 7fb57affd640 1 -- 172.21.15.116:0/2130026191 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb56c0bb070 con 0x7fb5840c4c20 2024-04-25T00:12:00.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:00.834+0000 7fb57affd640 1 --2- 172.21.15.116:0/2130026191 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb5640678c0 0x7fb564069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:00.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:00.834+0000 7fb57affd640 1 -- 172.21.15.116:0/2130026191 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7870+0+0 (secure 0 0 0) 0x7fb56c1203f0 con 0x7fb5840c4c20 2024-04-25T00:12:00.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:00.834+0000 7fb58259c640 1 --2- 172.21.15.116:0/2130026191 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb5640678c0 0x7fb564069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:00.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:00.834+0000 7fb58259c640 1 --2- 172.21.15.116:0/2130026191 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb5640678c0 0x7fb564069d80 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fb574000f70 tx=0x7fb5740079c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:00.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:00.838+0000 7fb57affd640 1 -- 172.21.15.116:0/2130026191 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fb56c0c2070 con 0x7fb5840c4c20 2024-04-25T00:12:01.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:01.106+0000 7fb588887640 1 -- 172.21.15.116:0/2130026191 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb5841f06b0 con 0x7fb5840c4c20 2024-04-25T00:12:01.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:01.110+0000 7fb57affd640 1 -- 172.21.15.116:0/2130026191 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) ==== 94+0+6503 (secure 0 0 0) 0x7fb56c016090 con 0x7fb5840c4c20 2024-04-25T00:12:01.108 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:12:01.108 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82},{"gid":5461,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:11:03.640578+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6114},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6114":{"gid":6114,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":25,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6114,"qdb_cluster":[6114]},"id":1},{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:11:53.790738+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6309},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6309":{"gid":6309,"name":"c","rank":0,"incarnation":78,"state":"up:active","state_seq":15,"addr":"172.21.15.116:6835/1599708526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1599708526},{"type":"v1","addr":"172.21.15.116:6835","nonce":1599708526}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6309,"qdb_cluster":[6309]},"id":2}]} 2024-04-25T00:12:01.108 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 82 2024-04-25T00:12:01.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:01.110+0000 7fb588887640 1 -- 172.21.15.116:0/2130026191 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb5640678c0 msgr2=0x7fb564069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:01.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:01.110+0000 7fb588887640 1 --2- 172.21.15.116:0/2130026191 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb5640678c0 0x7fb564069d80 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fb574000f70 tx=0x7fb5740079c0 comp rx=0 tx=0).stop 2024-04-25T00:12:01.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:01.110+0000 7fb588887640 1 -- 172.21.15.116:0/2130026191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5840c4c20 msgr2=0x7fb5840ad280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:01.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:01.110+0000 7fb588887640 1 --2- 172.21.15.116:0/2130026191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5840c4c20 0x7fb5840ad280 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fb56c019020 tx=0x7fb56c0191b0 comp rx=0 tx=0).stop 2024-04-25T00:12:01.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:01.110+0000 7fb588887640 1 -- 172.21.15.116:0/2130026191 shutdown_connections 2024-04-25T00:12:01.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:01.110+0000 7fb588887640 1 --2- 172.21.15.116:0/2130026191 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb5640678c0 0x7fb564069d80 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:01.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:01.110+0000 7fb588887640 1 --2- 172.21.15.116:0/2130026191 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb584150d40 0x7fb58414b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:01.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:01.110+0000 7fb588887640 1 --2- 172.21.15.116:0/2130026191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb5840c4c20 0x7fb5840ad280 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:01.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:01.110+0000 7fb588887640 1 --2- 172.21.15.116:0/2130026191 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb584070770 0x7fb58414cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:01.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:01.110+0000 7fb588887640 1 -- 172.21.15.116:0/2130026191 >> 172.21.15.116:0/2130026191 conn(0x7fb58406f0c0 msgr2=0x7fb5840ab150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:01.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:01.110+0000 7fb588887640 1 -- 172.21.15.116:0/2130026191 shutdown_connections 2024-04-25T00:12:01.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:01.110+0000 7fb588887640 1 -- 172.21.15.116:0/2130026191 wait complete. 2024-04-25T00:12:01.128 INFO:tasks.mds_thrash.fs.[b]:mds.d reported in up:standby state 2024-04-25T00:12:01.128 INFO:tasks.mds_thrash.fs.[b]:waiting for 7 secs before thrashing 2024-04-25T00:12:07.377 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:12:07.383 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:12:08.129 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:12:08.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.274+0000 7ff25a5d4640 1 Processor -- start 2024-04-25T00:12:08.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.274+0000 7ff25a5d4640 1 -- start start 2024-04-25T00:12:08.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.274+0000 7ff25a5d4640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff25414c720 0x7ff2541563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:08.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.274+0000 7ff25a5d4640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff2540a94a0 0x7ff2540c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:08.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.274+0000 7ff25a5d4640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff254071220 con 0x7ff2540c2f50 2024-04-25T00:12:08.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.274+0000 7ff25a5d4640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7ff254153460 con 0x7ff25414c720 2024-04-25T00:12:08.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.274+0000 7ff25a5d4640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7ff254152420 con 0x7ff2540a94a0 2024-04-25T00:12:08.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.274+0000 7ff253fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff25414c720 0x7ff2541563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:08.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.274+0000 7ff258b71640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7ff2540c2f50 0x7ff254071680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:56700/0 (socket says 172.21.15.116:56700) 2024-04-25T00:12:08.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.274+0000 7ff258b71640 1 -- 172.21.15.116:0/509377665 learned_addr learned my addr 172.21.15.116:0/509377665 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:12:08.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.274+0000 7ff2537fe640 1 --2- 172.21.15.116:0/509377665 >> v2:172.21.15.174:3300/0 conn(0x7ff2540a94a0 0x7ff2540c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:08.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.274+0000 7ff253fff640 1 -- 172.21.15.116:0/509377665 >> v2:172.21.15.174:3300/0 conn(0x7ff2540a94a0 msgr2=0x7ff2540c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:12:08.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.274+0000 7ff253fff640 1 --2- 172.21.15.116:0/509377665 >> v2:172.21.15.174:3300/0 conn(0x7ff2540a94a0 0x7ff2540c2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:08.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.274+0000 7ff253fff640 1 -- 172.21.15.116:0/509377665 >> v1:172.21.15.116:6789/0 conn(0x7ff2540c2f50 legacy=0x7ff254071680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:08.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.274+0000 7ff252ffd640 1 -- 172.21.15.116:0/509377665 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2099722185 0 0) 0x7ff254071220 con 0x7ff2540c2f50 2024-04-25T00:12:08.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.274+0000 7ff253fff640 1 -- 172.21.15.116:0/509377665 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff25414cb00 con 0x7ff25414c720 2024-04-25T00:12:08.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.278+0000 7ff253fff640 1 --2- 172.21.15.116:0/509377665 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff25414c720 0x7ff2541563d0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7ff2540a9930 tx=0x7ff24401c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dba85562225d87c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:08.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.278+0000 7ff252ffd640 1 -- 172.21.15.116:0/509377665 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2440b1070 con 0x7ff25414c720 2024-04-25T00:12:08.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.278+0000 7ff252ffd640 1 -- 172.21.15.116:0/509377665 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2440a8050 con 0x7ff25414c720 2024-04-25T00:12:08.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.278+0000 7ff252ffd640 1 -- 172.21.15.116:0/509377665 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2440ac860 con 0x7ff25414c720 2024-04-25T00:12:08.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.278+0000 7ff25a5d4640 1 -- 172.21.15.116:0/509377665 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff25414c720 msgr2=0x7ff2541563d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:08.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.278+0000 7ff25a5d4640 1 --2- 172.21.15.116:0/509377665 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff25414c720 0x7ff2541563d0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7ff2540a9930 tx=0x7ff24401c3c0 comp rx=0 tx=0).stop 2024-04-25T00:12:08.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.278+0000 7ff25a5d4640 1 -- 172.21.15.116:0/509377665 shutdown_connections 2024-04-25T00:12:08.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.278+0000 7ff25a5d4640 1 --2- 172.21.15.116:0/509377665 >> v2:172.21.15.174:3300/0 conn(0x7ff2540a94a0 0x7ff2540c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:08.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.278+0000 7ff25a5d4640 1 --2- 172.21.15.116:0/509377665 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff25414c720 0x7ff2541563d0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:08.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.278+0000 7ff25a5d4640 1 -- 172.21.15.116:0/509377665 >> 172.21.15.116:0/509377665 conn(0x7ff2540c3b00 msgr2=0x7ff2540c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:08.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.278+0000 7ff25a5d4640 1 -- 172.21.15.116:0/509377665 shutdown_connections 2024-04-25T00:12:08.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.278+0000 7ff25a5d4640 1 -- 172.21.15.116:0/509377665 wait complete. 2024-04-25T00:12:08.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.282+0000 7ff25a5d4640 1 Processor -- start 2024-04-25T00:12:08.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.282+0000 7ff25a5d4640 1 -- start start 2024-04-25T00:12:08.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.282+0000 7ff25a5d4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2540a94a0 0x7ff25406deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:08.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.282+0000 7ff25a5d4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff2540c2f50 0x7ff25415e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:08.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.282+0000 7ff25a5d4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff25414c720 0x7ff25415e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:08.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.282+0000 7ff25a5d4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff254152f40 con 0x7ff2540a94a0 2024-04-25T00:12:08.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.282+0000 7ff25a5d4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff254152c60 con 0x7ff25414c720 2024-04-25T00:12:08.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.282+0000 7ff25a5d4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff25415d5b0 con 0x7ff2540c2f50 2024-04-25T00:12:08.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.282+0000 7ff253fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2540a94a0 0x7ff25406deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:08.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.282+0000 7ff253fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2540a94a0 0x7ff25406deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58330/0 (socket says 172.21.15.116:58330) 2024-04-25T00:12:08.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.282+0000 7ff253fff640 1 -- 172.21.15.116:0/2512702017 learned_addr learned my addr 172.21.15.116:0/2512702017 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:12:08.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.282+0000 7ff258b71640 1 --2- 172.21.15.116:0/2512702017 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff25414c720 0x7ff25415e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:08.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.282+0000 7ff2537fe640 1 --2- 172.21.15.116:0/2512702017 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff2540c2f50 0x7ff25415e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:08.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.282+0000 7ff253fff640 1 -- 172.21.15.116:0/2512702017 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff2540c2f50 msgr2=0x7ff25415e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:08.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.282+0000 7ff253fff640 1 --2- 172.21.15.116:0/2512702017 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff2540c2f50 0x7ff25415e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:08.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.282+0000 7ff253fff640 1 -- 172.21.15.116:0/2512702017 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff25414c720 msgr2=0x7ff25415e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:08.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.282+0000 7ff253fff640 1 --2- 172.21.15.116:0/2512702017 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff25414c720 0x7ff25415e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:08.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.282+0000 7ff253fff640 1 -- 172.21.15.116:0/2512702017 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2540c6da0 con 0x7ff2540a94a0 2024-04-25T00:12:08.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.282+0000 7ff253fff640 1 --2- 172.21.15.116:0/2512702017 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2540a94a0 0x7ff25406deb0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7ff244019ff0 tx=0x7ff24401ac00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:08.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.282+0000 7ff250ff9640 1 -- 172.21.15.116:0/2512702017 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2440bb020 con 0x7ff2540a94a0 2024-04-25T00:12:08.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.282+0000 7ff250ff9640 1 -- 172.21.15.116:0/2512702017 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2440b7420 con 0x7ff2540a94a0 2024-04-25T00:12:08.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.282+0000 7ff25a5d4640 1 -- 172.21.15.116:0/2512702017 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff25415de50 con 0x7ff2540a94a0 2024-04-25T00:12:08.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.282+0000 7ff250ff9640 1 -- 172.21.15.116:0/2512702017 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2440b1040 con 0x7ff2540a94a0 2024-04-25T00:12:08.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.282+0000 7ff25a5d4640 1 -- 172.21.15.116:0/2512702017 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff25406e3f0 con 0x7ff2540a94a0 2024-04-25T00:12:08.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.282+0000 7ff25a5d4640 1 -- 172.21.15.116:0/2512702017 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff254068350 con 0x7ff2540a94a0 2024-04-25T00:12:08.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.286+0000 7ff250ff9640 1 -- 172.21.15.116:0/2512702017 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff2440be020 con 0x7ff2540a94a0 2024-04-25T00:12:08.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.286+0000 7ff250ff9640 1 --2- 172.21.15.116:0/2512702017 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff2400678c0 0x7ff240069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:08.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.286+0000 7ff250ff9640 1 -- 172.21.15.116:0/2512702017 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7870+0+0 (secure 0 0 0) 0x7ff244120830 con 0x7ff2540a94a0 2024-04-25T00:12:08.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.286+0000 7ff2537fe640 1 --2- 172.21.15.116:0/2512702017 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff2400678c0 0x7ff240069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:08.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.286+0000 7ff2537fe640 1 --2- 172.21.15.116:0/2512702017 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff2400678c0 0x7ff240069d80 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7ff2541561f0 tx=0x7ff248008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:08.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.290+0000 7ff250ff9640 1 -- 172.21.15.116:0/2512702017 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ff2440c5030 con 0x7ff2540a94a0 2024-04-25T00:12:08.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.558+0000 7ff25a5d4640 1 -- 172.21.15.116:0/2512702017 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff2540c3640 con 0x7ff2540a94a0 2024-04-25T00:12:08.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.558+0000 7ff250ff9640 1 -- 172.21.15.116:0/2512702017 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) ==== 94+0+6503 (secure 0 0 0) 0x7ff244016070 con 0x7ff2540a94a0 2024-04-25T00:12:08.559 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:12:08.559 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82},{"gid":5461,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:11:03.640578+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6114},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6114":{"gid":6114,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":25,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6114,"qdb_cluster":[6114]},"id":1},{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:11:53.790738+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6309},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6309":{"gid":6309,"name":"c","rank":0,"incarnation":78,"state":"up:active","state_seq":15,"addr":"172.21.15.116:6835/1599708526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1599708526},{"type":"v1","addr":"172.21.15.116:6835","nonce":1599708526}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6309,"qdb_cluster":[6309]},"id":2}]} 2024-04-25T00:12:08.559 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 82 2024-04-25T00:12:08.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.562+0000 7ff25a5d4640 1 -- 172.21.15.116:0/2512702017 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff2400678c0 msgr2=0x7ff240069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:08.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.562+0000 7ff25a5d4640 1 --2- 172.21.15.116:0/2512702017 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff2400678c0 0x7ff240069d80 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7ff2541561f0 tx=0x7ff248008040 comp rx=0 tx=0).stop 2024-04-25T00:12:08.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.562+0000 7ff25a5d4640 1 -- 172.21.15.116:0/2512702017 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2540a94a0 msgr2=0x7ff25406deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:08.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.562+0000 7ff25a5d4640 1 --2- 172.21.15.116:0/2512702017 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2540a94a0 0x7ff25406deb0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7ff244019ff0 tx=0x7ff24401ac00 comp rx=0 tx=0).stop 2024-04-25T00:12:08.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.562+0000 7ff25a5d4640 1 -- 172.21.15.116:0/2512702017 shutdown_connections 2024-04-25T00:12:08.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.562+0000 7ff25a5d4640 1 --2- 172.21.15.116:0/2512702017 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff2400678c0 0x7ff240069d80 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:08.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.562+0000 7ff25a5d4640 1 --2- 172.21.15.116:0/2512702017 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff25414c720 0x7ff25415e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:08.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.562+0000 7ff25a5d4640 1 --2- 172.21.15.116:0/2512702017 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff2540c2f50 0x7ff25415e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:08.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.562+0000 7ff25a5d4640 1 --2- 172.21.15.116:0/2512702017 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2540a94a0 0x7ff25406deb0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:08.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.562+0000 7ff25a5d4640 1 -- 172.21.15.116:0/2512702017 >> 172.21.15.116:0/2512702017 conn(0x7ff2540c3b00 msgr2=0x7ff2540c2270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:08.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.562+0000 7ff25a5d4640 1 -- 172.21.15.116:0/2512702017 shutdown_connections 2024-04-25T00:12:08.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.562+0000 7ff25a5d4640 1 -- 172.21.15.116:0/2512702017 wait complete. 2024-04-25T00:12:08.582 INFO:tasks.mds_thrash.fs.[b]:kill mds.c (rank=0) 2024-04-25T00:12:08.583 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-25T00:12:08.583 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:12:08.596 INFO:tasks.ceph.mds.c:Stopped 2024-04-25T00:12:08.596 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:12:08.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.762+0000 7f7d0af2f640 1 Processor -- start 2024-04-25T00:12:08.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.762+0000 7f7d0af2f640 1 -- start start 2024-04-25T00:12:08.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.766+0000 7f7d0af2f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7d04152d60 0x7f7d040ab270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:08.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.766+0000 7f7d0af2f640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f7d040ab7b0 0x7f7d040c8450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:08.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.766+0000 7f7d0af2f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d04153140 0x7f7d040c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:08.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.766+0000 7f7d0af2f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f7d04155c50 con 0x7f7d04152d60 2024-04-25T00:12:08.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.766+0000 7f7d0af2f640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f7d04155f30 con 0x7f7d040ab7b0 2024-04-25T00:12:08.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.766+0000 7f7d0af2f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f7d04156210 con 0x7f7d04153140 2024-04-25T00:12:08.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.766+0000 7f7d08ccb640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7d04152d60 0x7f7d040ab270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:08.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.766+0000 7f7d08ccb640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7d04152d60 0x7f7d040ab270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58336/0 (socket says 172.21.15.116:58336) 2024-04-25T00:12:08.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.766+0000 7f7d08ccb640 1 -- 172.21.15.116:0/534208467 learned_addr learned my addr 172.21.15.116:0/534208467 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:12:08.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.766+0000 7f7d03fff640 1 --2- 172.21.15.116:0/534208467 >> v2:172.21.15.174:3300/0 conn(0x7f7d040ab7b0 0x7f7d040c8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:08.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.766+0000 7f7d094cc640 1 --2- 172.21.15.116:0/534208467 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d04153140 0x7f7d040c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:08.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.766+0000 7f7d08ccb640 1 -- 172.21.15.116:0/534208467 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d04153140 msgr2=0x7f7d040c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:12:08.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.766+0000 7f7d08ccb640 1 --2- 172.21.15.116:0/534208467 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d04153140 0x7f7d040c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:08.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.766+0000 7f7d08ccb640 1 -- 172.21.15.116:0/534208467 >> v2:172.21.15.174:3300/0 conn(0x7f7d040ab7b0 msgr2=0x7f7d040c8450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:12:08.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.766+0000 7f7d08ccb640 1 --2- 172.21.15.116:0/534208467 >> v2:172.21.15.174:3300/0 conn(0x7f7d040ab7b0 0x7f7d040c8450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:08.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.766+0000 7f7d08ccb640 1 -- 172.21.15.116:0/534208467 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d041564f0 con 0x7f7d04152d60 2024-04-25T00:12:08.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.766+0000 7f7d094cc640 1 --2- 172.21.15.116:0/534208467 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d04153140 0x7f7d040c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:12:08.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.766+0000 7f7d08ccb640 1 --2- 172.21.15.116:0/534208467 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d04152d60 0x7f7d040ab270 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f7d04156140 tx=0x7f7cf801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdc0139ec3932ada server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:08.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.766+0000 7f7d037fe640 1 -- 172.21.15.116:0/534208467 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf80ab5f0 con 0x7f7d04152d60 2024-04-25T00:12:08.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.766+0000 7f7d037fe640 1 -- 172.21.15.116:0/534208467 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cf80b0070 con 0x7f7d04152d60 2024-04-25T00:12:08.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.766+0000 7f7d037fe640 1 -- 172.21.15.116:0/534208467 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf80ab9b0 con 0x7f7d04152d60 2024-04-25T00:12:08.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.770+0000 7f7d0af2f640 1 -- 172.21.15.116:0/534208467 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d04152d60 msgr2=0x7f7d040ab270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:08.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.770+0000 7f7d0af2f640 1 --2- 172.21.15.116:0/534208467 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d04152d60 0x7f7d040ab270 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f7d04156140 tx=0x7f7cf801c520 comp rx=0 tx=0).stop 2024-04-25T00:12:08.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.770+0000 7f7d0af2f640 1 -- 172.21.15.116:0/534208467 shutdown_connections 2024-04-25T00:12:08.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.770+0000 7f7d0af2f640 1 --2- 172.21.15.116:0/534208467 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d04153140 0x7f7d040c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:08.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.770+0000 7f7d0af2f640 1 --2- 172.21.15.116:0/534208467 >> v2:172.21.15.174:3300/0 conn(0x7f7d040ab7b0 0x7f7d040c8450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:08.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.770+0000 7f7d0af2f640 1 --2- 172.21.15.116:0/534208467 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d04152d60 0x7f7d040ab270 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:08.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.770+0000 7f7d0af2f640 1 -- 172.21.15.116:0/534208467 >> 172.21.15.116:0/534208467 conn(0x7f7d0406f0c0 msgr2=0x7f7d0406f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:08.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.770+0000 7f7d0af2f640 1 -- 172.21.15.116:0/534208467 shutdown_connections 2024-04-25T00:12:08.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.770+0000 7f7d0af2f640 1 -- 172.21.15.116:0/534208467 wait complete. 2024-04-25T00:12:08.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.770+0000 7f7d0af2f640 1 Processor -- start 2024-04-25T00:12:08.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.770+0000 7f7d0af2f640 1 -- start start 2024-04-25T00:12:08.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.770+0000 7f7d0af2f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d040ab7b0 0x7f7d040ad5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:08.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.770+0000 7f7d0af2f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d04152d60 0x7f7d0406d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:08.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.770+0000 7f7d0af2f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d04153140 0x7f7d041f13a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:08.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.770+0000 7f7d0af2f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7d040c48a0 con 0x7f7d04153140 2024-04-25T00:12:08.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.770+0000 7f7d0af2f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7d040c4720 con 0x7f7d04152d60 2024-04-25T00:12:08.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.770+0000 7f7d0af2f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f7d0414ba80 con 0x7f7d040ab7b0 2024-04-25T00:12:08.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.770+0000 7f7d094cc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d04153140 0x7f7d041f13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:08.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.770+0000 7f7d094cc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d04153140 0x7f7d041f13a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58342/0 (socket says 172.21.15.116:58342) 2024-04-25T00:12:08.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.770+0000 7f7d094cc640 1 -- 172.21.15.116:0/152996307 learned_addr learned my addr 172.21.15.116:0/152996307 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:12:08.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.770+0000 7f7d03fff640 1 --2- 172.21.15.116:0/152996307 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d04152d60 0x7f7d0406d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:08.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.770+0000 7f7d08ccb640 1 --2- 172.21.15.116:0/152996307 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d040ab7b0 0x7f7d040ad5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:08.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.770+0000 7f7d094cc640 1 -- 172.21.15.116:0/152996307 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d040ab7b0 msgr2=0x7f7d040ad5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:08.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.770+0000 7f7d094cc640 1 --2- 172.21.15.116:0/152996307 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d040ab7b0 0x7f7d040ad5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:08.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.770+0000 7f7d094cc640 1 -- 172.21.15.116:0/152996307 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d04152d60 msgr2=0x7f7d0406d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:08.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.770+0000 7f7d094cc640 1 --2- 172.21.15.116:0/152996307 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d04152d60 0x7f7d0406d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:08.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.770+0000 7f7d094cc640 1 -- 172.21.15.116:0/152996307 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d041453e0 con 0x7f7d04153140 2024-04-25T00:12:08.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.770+0000 7f7d094cc640 1 --2- 172.21.15.116:0/152996307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d04153140 0x7f7d041f13a0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f7cfc00ce70 tx=0x7f7cfc00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:08.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.770+0000 7f7d017fa640 1 -- 172.21.15.116:0/152996307 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cfc01a070 con 0x7f7d04153140 2024-04-25T00:12:08.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.770+0000 7f7d017fa640 1 -- 172.21.15.116:0/152996307 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cfc00d840 con 0x7f7d04153140 2024-04-25T00:12:08.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.770+0000 7f7d0af2f640 1 -- 172.21.15.116:0/152996307 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d0414e7c0 con 0x7f7d04153140 2024-04-25T00:12:08.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.770+0000 7f7d017fa640 1 -- 172.21.15.116:0/152996307 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cfc00dc30 con 0x7f7d04153140 2024-04-25T00:12:08.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.770+0000 7f7d0af2f640 1 -- 172.21.15.116:0/152996307 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d0406dc20 con 0x7f7d04153140 2024-04-25T00:12:08.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.774+0000 7f7d0af2f640 1 -- 172.21.15.116:0/152996307 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d04068a60 con 0x7f7d04153140 2024-04-25T00:12:08.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.774+0000 7f7d017fa640 1 -- 172.21.15.116:0/152996307 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7cfc01d070 con 0x7f7d04153140 2024-04-25T00:12:08.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.774+0000 7f7d017fa640 1 --2- 172.21.15.116:0/152996307 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7ce80678c0 0x7f7ce8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:08.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.774+0000 7f7d017fa640 1 -- 172.21.15.116:0/152996307 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7870+0+0 (secure 0 0 0) 0x7f7cfc086bb0 con 0x7f7d04153140 2024-04-25T00:12:08.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.774+0000 7f7d08ccb640 1 --2- 172.21.15.116:0/152996307 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7ce80678c0 0x7f7ce8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:08.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.774+0000 7f7d08ccb640 1 --2- 172.21.15.116:0/152996307 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7ce80678c0 0x7f7ce8069d80 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f7cf80af000 tx=0x7f7cf80a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:08.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:08.778+0000 7f7d017fa640 1 -- 172.21.15.116:0/152996307 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f7cfc0568c0 con 0x7f7d04153140 2024-04-25T00:12:09.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:09.018+0000 7f7d0af2f640 1 -- 172.21.15.116:0/152996307 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7d041539a0 con 0x7f7d04153140 2024-04-25T00:12:09.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:09.018+0000 7f7d017fa640 1 -- 172.21.15.116:0/152996307 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) ==== 94+0+6503 (secure 0 0 0) 0x7f7cfc05abc0 con 0x7f7d04153140 2024-04-25T00:12:09.018 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:12:09.018 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82},{"gid":5461,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:11:03.640578+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6114},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6114":{"gid":6114,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":25,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6114,"qdb_cluster":[6114]},"id":1},{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:11:53.790738+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6309},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6309":{"gid":6309,"name":"c","rank":0,"incarnation":78,"state":"up:active","state_seq":15,"addr":"172.21.15.116:6835/1599708526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1599708526},{"type":"v1","addr":"172.21.15.116:6835","nonce":1599708526}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6309,"qdb_cluster":[6309]},"id":2}]} 2024-04-25T00:12:09.018 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 82 2024-04-25T00:12:09.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:09.018+0000 7f7d0af2f640 1 -- 172.21.15.116:0/152996307 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7ce80678c0 msgr2=0x7f7ce8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:09.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:09.018+0000 7f7d0af2f640 1 --2- 172.21.15.116:0/152996307 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7ce80678c0 0x7f7ce8069d80 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f7cf80af000 tx=0x7f7cf80a7000 comp rx=0 tx=0).stop 2024-04-25T00:12:09.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:09.018+0000 7f7d0af2f640 1 -- 172.21.15.116:0/152996307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d04153140 msgr2=0x7f7d041f13a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:09.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:09.018+0000 7f7d0af2f640 1 --2- 172.21.15.116:0/152996307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d04153140 0x7f7d041f13a0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f7cfc00ce70 tx=0x7f7cfc00a710 comp rx=0 tx=0).stop 2024-04-25T00:12:09.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:09.022+0000 7f7d0af2f640 1 -- 172.21.15.116:0/152996307 shutdown_connections 2024-04-25T00:12:09.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:09.022+0000 7f7d0af2f640 1 --2- 172.21.15.116:0/152996307 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7ce80678c0 0x7f7ce8069d80 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:09.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:09.022+0000 7f7d0af2f640 1 --2- 172.21.15.116:0/152996307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d04153140 0x7f7d041f13a0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:09.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:09.022+0000 7f7d0af2f640 1 --2- 172.21.15.116:0/152996307 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d04152d60 0x7f7d0406d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:09.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:09.022+0000 7f7d0af2f640 1 --2- 172.21.15.116:0/152996307 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d040ab7b0 0x7f7d040ad5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:09.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:09.022+0000 7f7d0af2f640 1 -- 172.21.15.116:0/152996307 >> 172.21.15.116:0/152996307 conn(0x7f7d0406f0c0 msgr2=0x7f7d040c83a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:09.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:09.022+0000 7f7d0af2f640 1 -- 172.21.15.116:0/152996307 shutdown_connections 2024-04-25T00:12:09.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:09.022+0000 7f7d0af2f640 1 -- 172.21.15.116:0/152996307 wait complete. 2024-04-25T00:12:09.038 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:12:11.039 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:12:11.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.166+0000 7fc524d35640 1 Processor -- start 2024-04-25T00:12:11.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.166+0000 7fc524d35640 1 -- start start 2024-04-25T00:12:11.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.170+0000 7fc524d35640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fc5200c4300 0x7fc5200aa170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:11.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.170+0000 7fc524d35640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc5200c5100 0x7fc5200c5520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:11.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.170+0000 7fc524d35640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5201526a0 con 0x7fc520155af0 2024-04-25T00:12:11.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.170+0000 7fc524d35640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fc520153460 con 0x7fc5200c4300 2024-04-25T00:12:11.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.170+0000 7fc524d35640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fc520156690 con 0x7fc5200c5100 2024-04-25T00:12:11.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.170+0000 7fc51ed9d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fc520155af0 0x7fc520155f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:56710/0 (socket says 172.21.15.116:56710) 2024-04-25T00:12:11.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.170+0000 7fc51ed9d640 1 -- 172.21.15.116:0/158363475 learned_addr learned my addr 172.21.15.116:0/158363475 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:12:11.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.170+0000 7fc51e59c640 1 --2- 172.21.15.116:0/158363475 >> v2:172.21.15.174:3300/0 conn(0x7fc5200c4300 0x7fc5200aa170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:11.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.170+0000 7fc51dd9b640 1 --2- 172.21.15.116:0/158363475 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc5200c5100 0x7fc5200c5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:11.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.170+0000 7fc51d59a640 1 -- 172.21.15.116:0/158363475 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1100490043 0 0) 0x7fc5201526a0 con 0x7fc520155af0 2024-04-25T00:12:11.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.170+0000 7fc51d59a640 1 -- 172.21.15.116:0/158363475 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc508003530 con 0x7fc520155af0 2024-04-25T00:12:11.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.170+0000 7fc51d59a640 1 -- 172.21.15.116:0/158363475 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4207688656 0 0) 0x7fc508003530 con 0x7fc520155af0 2024-04-25T00:12:11.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.170+0000 7fc51d59a640 1 -- 172.21.15.116:0/158363475 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc5200c5100 msgr2=0x7fc5200c5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:12:11.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.170+0000 7fc51d59a640 1 --2- 172.21.15.116:0/158363475 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc5200c5100 0x7fc5200c5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:11.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.170+0000 7fc51d59a640 1 -- 172.21.15.116:0/158363475 >> v2:172.21.15.174:3300/0 conn(0x7fc5200c4300 msgr2=0x7fc5200aa170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:12:11.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.170+0000 7fc51d59a640 1 --2- 172.21.15.116:0/158363475 >> v2:172.21.15.174:3300/0 conn(0x7fc5200c4300 0x7fc5200aa170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:11.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.170+0000 7fc51d59a640 1 -- 172.21.15.116:0/158363475 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc520152980 con 0x7fc520155af0 2024-04-25T00:12:11.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.170+0000 7fc51d59a640 1 -- 172.21.15.116:0/158363475 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7fc518001a00 con 0x7fc520155af0 2024-04-25T00:12:11.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.170+0000 7fc51d59a640 1 -- 172.21.15.116:0/158363475 >> v1:172.21.15.116:6789/0 conn(0x7fc520155af0 legacy=0x7fc520155f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:11.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.170+0000 7fc51d59a640 1 --2- 172.21.15.116:0/158363475 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5080039c0 0x7fc508003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:11.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.170+0000 7fc51d59a640 1 --2- 172.21.15.116:0/158363475 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc508004810 0x7fc508004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:11.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.170+0000 7fc51d59a640 1 --2- 172.21.15.116:0/158363475 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc5200c5100 0x7fc5080051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:11.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.170+0000 7fc51d59a640 1 -- 172.21.15.116:0/158363475 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc520153460 con 0x7fc5080039c0 2024-04-25T00:12:11.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.170+0000 7fc51d59a640 1 -- 172.21.15.116:0/158363475 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc50800b9d0 con 0x7fc508004810 2024-04-25T00:12:11.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.170+0000 7fc51d59a640 1 -- 172.21.15.116:0/158363475 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fc50800ff60 con 0x7fc5200c5100 2024-04-25T00:12:11.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.170+0000 7fc51e59c640 1 --2- 172.21.15.116:0/158363475 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5080039c0 0x7fc508003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:11.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.170+0000 7fc51dd9b640 1 --2- 172.21.15.116:0/158363475 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc508004810 0x7fc508004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:11.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.170+0000 7fc51ed9d640 1 --2- 172.21.15.116:0/158363475 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc5200c5100 0x7fc5080051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:11.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.170+0000 7fc51e59c640 1 -- 172.21.15.116:0/158363475 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc5200c5100 msgr2=0x7fc5080051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:12:11.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.170+0000 7fc51e59c640 1 --2- 172.21.15.116:0/158363475 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc5200c5100 0x7fc5080051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:11.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.170+0000 7fc51e59c640 1 -- 172.21.15.116:0/158363475 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc508004810 msgr2=0x7fc508004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:12:11.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.170+0000 7fc51e59c640 1 --2- 172.21.15.116:0/158363475 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc508004810 0x7fc508004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:11.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.170+0000 7fc51e59c640 1 -- 172.21.15.116:0/158363475 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc508003530 con 0x7fc5080039c0 2024-04-25T00:12:11.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.170+0000 7fc51ed9d640 1 --2- 172.21.15.116:0/158363475 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc5200c5100 0x7fc5080051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:12:11.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.170+0000 7fc51e59c640 1 --2- 172.21.15.116:0/158363475 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5080039c0 0x7fc508003de0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fc510009ae0 tx=0x7fc51001cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57212f822ae8e332 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:11.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.174+0000 7fc51d59a640 1 -- 172.21.15.116:0/158363475 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5100096b0 con 0x7fc5080039c0 2024-04-25T00:12:11.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.174+0000 7fc51d59a640 1 -- 172.21.15.116:0/158363475 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5100af070 con 0x7fc5080039c0 2024-04-25T00:12:11.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.174+0000 7fc51d59a640 1 -- 172.21.15.116:0/158363475 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5100aa690 con 0x7fc5080039c0 2024-04-25T00:12:11.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.174+0000 7fc524d35640 1 -- 172.21.15.116:0/158363475 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5080039c0 msgr2=0x7fc508003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:11.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.174+0000 7fc524d35640 1 --2- 172.21.15.116:0/158363475 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5080039c0 0x7fc508003de0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fc510009ae0 tx=0x7fc51001cab0 comp rx=0 tx=0).stop 2024-04-25T00:12:11.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.174+0000 7fc51ed9d640 1 -- 172.21.15.116:0/158363475 reap_dead start 2024-04-25T00:12:11.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.174+0000 7fc524d35640 1 -- 172.21.15.116:0/158363475 shutdown_connections 2024-04-25T00:12:11.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.174+0000 7fc524d35640 1 -- 172.21.15.116:0/158363475 >> 172.21.15.116:0/158363475 conn(0x7fc52006c720 msgr2=0x7fc52006cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:11.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.174+0000 7fc524d35640 1 -- 172.21.15.116:0/158363475 shutdown_connections 2024-04-25T00:12:11.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.174+0000 7fc524d35640 1 -- 172.21.15.116:0/158363475 wait complete. 2024-04-25T00:12:11.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.174+0000 7fc524d35640 1 Processor -- start 2024-04-25T00:12:11.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.174+0000 7fc524d35640 1 -- start start 2024-04-25T00:12:11.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.174+0000 7fc524d35640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5080039c0 0x7fc520072130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:11.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.174+0000 7fc524d35640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc5200c5d00 0x7fc5201ed2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:11.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.174+0000 7fc524d35640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc5201ed7e0 0x7fc52013d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:11.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.174+0000 7fc524d35640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc5200aae50 con 0x7fc5080039c0 2024-04-25T00:12:11.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.174+0000 7fc524d35640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc5200aacd0 con 0x7fc5200c5d00 2024-04-25T00:12:11.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.174+0000 7fc524d35640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fc52006fb80 con 0x7fc5201ed7e0 2024-04-25T00:12:11.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.174+0000 7fc51e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5080039c0 0x7fc520072130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:11.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.174+0000 7fc51e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5080039c0 0x7fc520072130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58362/0 (socket says 172.21.15.116:58362) 2024-04-25T00:12:11.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.174+0000 7fc51e59c640 1 -- 172.21.15.116:0/3768981453 learned_addr learned my addr 172.21.15.116:0/3768981453 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:12:11.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.174+0000 7fc51dd9b640 1 --2- 172.21.15.116:0/3768981453 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc5200c5d00 0x7fc5201ed2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:11.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.174+0000 7fc51ed9d640 1 --2- 172.21.15.116:0/3768981453 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc5201ed7e0 0x7fc52013d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:11.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.174+0000 7fc51e59c640 1 -- 172.21.15.116:0/3768981453 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc5201ed7e0 msgr2=0x7fc52013d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:11.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.174+0000 7fc51e59c640 1 --2- 172.21.15.116:0/3768981453 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc5201ed7e0 0x7fc52013d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:11.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.174+0000 7fc51e59c640 1 -- 172.21.15.116:0/3768981453 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc5200c5d00 msgr2=0x7fc5201ed2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:11.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.174+0000 7fc51e59c640 1 --2- 172.21.15.116:0/3768981453 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc5200c5d00 0x7fc5201ed2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:11.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.174+0000 7fc51e59c640 1 -- 172.21.15.116:0/3768981453 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc520146080 con 0x7fc5080039c0 2024-04-25T00:12:11.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.174+0000 7fc51e59c640 1 --2- 172.21.15.116:0/3768981453 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5080039c0 0x7fc520072130 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fc5100ac040 tx=0x7fc508005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:11.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.174+0000 7fc50effd640 1 -- 172.21.15.116:0/3768981453 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc510006450 con 0x7fc5080039c0 2024-04-25T00:12:11.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.174+0000 7fc50effd640 1 -- 172.21.15.116:0/3768981453 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5100a6040 con 0x7fc5080039c0 2024-04-25T00:12:11.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.174+0000 7fc524d35640 1 -- 172.21.15.116:0/3768981453 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc520146310 con 0x7fc5080039c0 2024-04-25T00:12:11.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.174+0000 7fc50effd640 1 -- 172.21.15.116:0/3768981453 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc510006830 con 0x7fc5080039c0 2024-04-25T00:12:11.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.178+0000 7fc524d35640 1 -- 172.21.15.116:0/3768981453 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5200ad520 con 0x7fc5080039c0 2024-04-25T00:12:11.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.178+0000 7fc524d35640 1 -- 172.21.15.116:0/3768981453 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc520068a60 con 0x7fc5080039c0 2024-04-25T00:12:11.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.178+0000 7fc50effd640 1 -- 172.21.15.116:0/3768981453 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc5100aa450 con 0x7fc5080039c0 2024-04-25T00:12:11.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.178+0000 7fc50effd640 1 --2- 172.21.15.116:0/3768981453 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc5000678f0 0x7fc500069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:11.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.178+0000 7fc50effd640 1 -- 172.21.15.116:0/3768981453 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7870+0+0 (secure 0 0 0) 0x7fc510120e70 con 0x7fc5080039c0 2024-04-25T00:12:11.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.178+0000 7fc51dd9b640 1 --2- 172.21.15.116:0/3768981453 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc5000678f0 0x7fc500069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:11.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.178+0000 7fc51dd9b640 1 --2- 172.21.15.116:0/3768981453 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc5000678f0 0x7fc500069db0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fc51400bbb0 tx=0x7fc514008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:11.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.186+0000 7fc50effd640 1 -- 172.21.15.116:0/3768981453 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fc5100167c0 con 0x7fc5080039c0 2024-04-25T00:12:11.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.458+0000 7fc524d35640 1 -- 172.21.15.116:0/3768981453 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc5200c3750 con 0x7fc5080039c0 2024-04-25T00:12:11.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.458+0000 7fc50effd640 1 -- 172.21.15.116:0/3768981453 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) ==== 94+0+6503 (secure 0 0 0) 0x7fc5100f11d0 con 0x7fc5080039c0 2024-04-25T00:12:11.458 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:12:11.458 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82},{"gid":5461,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:11:03.640578+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6114},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6114":{"gid":6114,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":25,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6114,"qdb_cluster":[6114]},"id":1},{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:11:53.790738+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6309},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6309":{"gid":6309,"name":"c","rank":0,"incarnation":78,"state":"up:active","state_seq":15,"addr":"172.21.15.116:6835/1599708526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1599708526},{"type":"v1","addr":"172.21.15.116:6835","nonce":1599708526}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6309,"qdb_cluster":[6309]},"id":2}]} 2024-04-25T00:12:11.458 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 82 2024-04-25T00:12:11.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.462+0000 7fc524d35640 1 -- 172.21.15.116:0/3768981453 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc5000678f0 msgr2=0x7fc500069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:11.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.462+0000 7fc524d35640 1 --2- 172.21.15.116:0/3768981453 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc5000678f0 0x7fc500069db0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fc51400bbb0 tx=0x7fc514008040 comp rx=0 tx=0).stop 2024-04-25T00:12:11.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.462+0000 7fc524d35640 1 -- 172.21.15.116:0/3768981453 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5080039c0 msgr2=0x7fc520072130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:11.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.462+0000 7fc524d35640 1 --2- 172.21.15.116:0/3768981453 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5080039c0 0x7fc520072130 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fc5100ac040 tx=0x7fc508005840 comp rx=0 tx=0).stop 2024-04-25T00:12:11.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.462+0000 7fc524d35640 1 -- 172.21.15.116:0/3768981453 shutdown_connections 2024-04-25T00:12:11.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.462+0000 7fc524d35640 1 --2- 172.21.15.116:0/3768981453 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc5000678f0 0x7fc500069db0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:11.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.462+0000 7fc524d35640 1 --2- 172.21.15.116:0/3768981453 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc5201ed7e0 0x7fc52013d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:11.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.462+0000 7fc524d35640 1 --2- 172.21.15.116:0/3768981453 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc5200c5d00 0x7fc5201ed2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:11.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.462+0000 7fc524d35640 1 --2- 172.21.15.116:0/3768981453 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc5080039c0 0x7fc520072130 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:11.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.462+0000 7fc524d35640 1 -- 172.21.15.116:0/3768981453 >> 172.21.15.116:0/3768981453 conn(0x7fc52006c720 msgr2=0x7fc5200671c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:11.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.462+0000 7fc524d35640 1 -- 172.21.15.116:0/3768981453 shutdown_connections 2024-04-25T00:12:11.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:11.462+0000 7fc524d35640 1 -- 172.21.15.116:0/3768981453 wait complete. 2024-04-25T00:12:11.478 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:12:13.479 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:12:13.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.634+0000 7f8873802640 1 Processor -- start 2024-04-25T00:12:13.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.634+0000 7f8873802640 1 -- start start 2024-04-25T00:12:13.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.634+0000 7f8873802640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f886c0c19f0 0x7f886c06f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:13.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.634+0000 7f8873802640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f886c153aa0 con 0x7f886c153570 2024-04-25T00:12:13.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.634+0000 7f8873802640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f886c153d80 con 0x7f886c150bf0 2024-04-25T00:12:13.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.634+0000 7f8873802640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f886c06e630 con 0x7f886c0c19f0 2024-04-25T00:12:13.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.634+0000 7f8870d9d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f886c153570 0x7f886c06eac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:56720/0 (socket says 172.21.15.116:56720) 2024-04-25T00:12:13.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.634+0000 7f8870d9d640 1 -- 172.21.15.116:0/3557341113 learned_addr learned my addr 172.21.15.116:0/3557341113 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:12:13.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.634+0000 7f8871d9f640 1 --2- 172.21.15.116:0/3557341113 >> v2:172.21.15.174:3300/0 conn(0x7f886c0c19f0 0x7f886c06f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:13.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.634+0000 7f8863fff640 1 -- 172.21.15.116:0/3557341113 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2164034941 0 0) 0x7f886c153aa0 con 0x7f886c153570 2024-04-25T00:12:13.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.634+0000 7f8863fff640 1 -- 172.21.15.116:0/3557341113 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8858003530 con 0x7f886c153570 2024-04-25T00:12:13.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.634+0000 7f8863fff640 1 -- 172.21.15.116:0/3557341113 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1075219044 0 0) 0x7f8858003530 con 0x7f886c153570 2024-04-25T00:12:13.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.634+0000 7f8863fff640 1 -- 172.21.15.116:0/3557341113 >> v2:172.21.15.174:3300/0 conn(0x7f886c0c19f0 msgr2=0x7f886c06f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:12:13.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.634+0000 7f8863fff640 1 --2- 172.21.15.116:0/3557341113 >> v2:172.21.15.174:3300/0 conn(0x7f886c0c19f0 0x7f886c06f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:13.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.634+0000 7f8863fff640 1 -- 172.21.15.116:0/3557341113 >> v1:172.21.15.174:6789/0 conn(0x7f886c150bf0 legacy=0x7f886c06c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:13.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.634+0000 7f8863fff640 1 -- 172.21.15.116:0/3557341113 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f886c154060 con 0x7f886c153570 2024-04-25T00:12:13.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.634+0000 7f8863fff640 1 -- 172.21.15.116:0/3557341113 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f885c001680 con 0x7f886c153570 2024-04-25T00:12:13.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.634+0000 7f8863fff640 1 -- 172.21.15.116:0/3557341113 >> v1:172.21.15.116:6789/0 conn(0x7f886c153570 legacy=0x7f886c06eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:13.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.634+0000 7f8863fff640 1 --2- 172.21.15.116:0/3557341113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88580039c0 0x7f8858003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:13.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.634+0000 7f8863fff640 1 --2- 172.21.15.116:0/3557341113 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8858004810 0x7f8858004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:13.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.634+0000 7f8863fff640 1 --2- 172.21.15.116:0/3557341113 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88580051d0 0x7f885800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:13.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.634+0000 7f8863fff640 1 -- 172.21.15.116:0/3557341113 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f886c06e630 con 0x7f88580039c0 2024-04-25T00:12:13.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.634+0000 7f8863fff640 1 -- 172.21.15.116:0/3557341113 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f885800c170 con 0x7f88580051d0 2024-04-25T00:12:13.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.634+0000 7f8863fff640 1 -- 172.21.15.116:0/3557341113 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f885800c370 con 0x7f8858004810 2024-04-25T00:12:13.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.634+0000 7f887159e640 1 --2- 172.21.15.116:0/3557341113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88580039c0 0x7f8858003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:13.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.634+0000 7f8871d9f640 1 --2- 172.21.15.116:0/3557341113 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88580051d0 0x7f885800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:13.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.634+0000 7f8870d9d640 1 --2- 172.21.15.116:0/3557341113 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8858004810 0x7f8858004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:13.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.634+0000 7f887159e640 1 -- 172.21.15.116:0/3557341113 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8858004810 msgr2=0x7f8858004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:12:13.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.634+0000 7f887159e640 1 --2- 172.21.15.116:0/3557341113 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8858004810 0x7f8858004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:13.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.634+0000 7f887159e640 1 -- 172.21.15.116:0/3557341113 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88580051d0 msgr2=0x7f885800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:12:13.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.634+0000 7f887159e640 1 --2- 172.21.15.116:0/3557341113 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88580051d0 0x7f885800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:13.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.634+0000 7f887159e640 1 -- 172.21.15.116:0/3557341113 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8858003530 con 0x7f88580039c0 2024-04-25T00:12:13.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.634+0000 7f8871d9f640 1 -- 172.21.15.116:0/3557341113 reap_dead start 2024-04-25T00:12:13.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.638+0000 7f887159e640 1 --2- 172.21.15.116:0/3557341113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88580039c0 0x7f8858003de0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f8868001780 tx=0x7f886801c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78a4722a85f683c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:13.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.638+0000 7f8863fff640 1 -- 172.21.15.116:0/3557341113 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88680013a0 con 0x7f88580039c0 2024-04-25T00:12:13.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.638+0000 7f8863fff640 1 -- 172.21.15.116:0/3557341113 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88680050d0 con 0x7f88580039c0 2024-04-25T00:12:13.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.638+0000 7f8863fff640 1 -- 172.21.15.116:0/3557341113 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8868003600 con 0x7f88580039c0 2024-04-25T00:12:13.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.638+0000 7f8873802640 1 -- 172.21.15.116:0/3557341113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88580039c0 msgr2=0x7f8858003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:13.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.638+0000 7f8873802640 1 --2- 172.21.15.116:0/3557341113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88580039c0 0x7f8858003de0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f8868001780 tx=0x7f886801c400 comp rx=0 tx=0).stop 2024-04-25T00:12:13.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.638+0000 7f8873802640 1 -- 172.21.15.116:0/3557341113 shutdown_connections 2024-04-25T00:12:13.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.638+0000 7f8873802640 1 --2- 172.21.15.116:0/3557341113 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88580039c0 0x7f8858003de0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:13.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.638+0000 7f8873802640 1 -- 172.21.15.116:0/3557341113 >> 172.21.15.116:0/3557341113 conn(0x7f886c0c4270 msgr2=0x7f886c0c4680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:13.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.638+0000 7f8873802640 1 -- 172.21.15.116:0/3557341113 shutdown_connections 2024-04-25T00:12:13.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.638+0000 7f8873802640 1 -- 172.21.15.116:0/3557341113 wait complete. 2024-04-25T00:12:13.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.638+0000 7f8873802640 1 Processor -- start 2024-04-25T00:12:13.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.638+0000 7f8873802640 1 -- start start 2024-04-25T00:12:13.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.642+0000 7f8873802640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88580039c0 0x7f886c0c0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:13.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.642+0000 7f8873802640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f886c0ab020 0x7f886c0c6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:13.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.642+0000 7f8873802640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f886c156cd0 0x7f886c073040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:13.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.642+0000 7f8873802640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f886c06ee50 con 0x7f88580039c0 2024-04-25T00:12:13.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.642+0000 7f8873802640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f886c06ecd0 con 0x7f886c156cd0 2024-04-25T00:12:13.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.642+0000 7f8873802640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f886c06efd0 con 0x7f886c0ab020 2024-04-25T00:12:13.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.642+0000 7f887159e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88580039c0 0x7f886c0c0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:13.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.642+0000 7f887159e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88580039c0 0x7f886c0c0ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:58378/0 (socket says 172.21.15.116:58378) 2024-04-25T00:12:13.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.642+0000 7f887159e640 1 -- 172.21.15.116:0/2721996936 learned_addr learned my addr 172.21.15.116:0/2721996936 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:12:13.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.642+0000 7f8870d9d640 1 --2- 172.21.15.116:0/2721996936 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f886c0ab020 0x7f886c0c6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:13.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.642+0000 7f8871d9f640 1 --2- 172.21.15.116:0/2721996936 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f886c156cd0 0x7f886c073040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:13.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.642+0000 7f887159e640 1 -- 172.21.15.116:0/2721996936 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f886c0ab020 msgr2=0x7f886c0c6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:13.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.642+0000 7f887159e640 1 --2- 172.21.15.116:0/2721996936 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f886c0ab020 0x7f886c0c6bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:13.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.642+0000 7f887159e640 1 -- 172.21.15.116:0/2721996936 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f886c156cd0 msgr2=0x7f886c073040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:13.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.642+0000 7f887159e640 1 --2- 172.21.15.116:0/2721996936 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f886c156cd0 0x7f886c073040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:13.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.642+0000 7f887159e640 1 -- 172.21.15.116:0/2721996936 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f886c0aa610 con 0x7f88580039c0 2024-04-25T00:12:13.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.642+0000 7f887159e640 1 --2- 172.21.15.116:0/2721996936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88580039c0 0x7f886c0c0ba0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f886801c8e0 tx=0x7f8858005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:13.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.642+0000 7f8861ffb640 1 -- 172.21.15.116:0/2721996936 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88680072f0 con 0x7f88580039c0 2024-04-25T00:12:13.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.642+0000 7f8861ffb640 1 -- 172.21.15.116:0/2721996936 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88680ba070 con 0x7f88580039c0 2024-04-25T00:12:13.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.642+0000 7f8861ffb640 1 -- 172.21.15.116:0/2721996936 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88680067e0 con 0x7f88580039c0 2024-04-25T00:12:13.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.642+0000 7f8873802640 1 -- 172.21.15.116:0/2721996936 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f886c0ab570 con 0x7f88580039c0 2024-04-25T00:12:13.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.642+0000 7f8873802640 1 -- 172.21.15.116:0/2721996936 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f886c0aab90 con 0x7f88580039c0 2024-04-25T00:12:13.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.642+0000 7f8873802640 1 -- 172.21.15.116:0/2721996936 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f886c068a60 con 0x7f88580039c0 2024-04-25T00:12:13.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.642+0000 7f8861ffb640 1 -- 172.21.15.116:0/2721996936 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8868006980 con 0x7f88580039c0 2024-04-25T00:12:13.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.646+0000 7f8861ffb640 1 --2- 172.21.15.116:0/2721996936 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f88500678f0 0x7f8850069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:13.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.646+0000 7f8861ffb640 1 -- 172.21.15.116:0/2721996936 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7870+0+0 (secure 0 0 0) 0x7f8868120440 con 0x7f88580039c0 2024-04-25T00:12:13.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.646+0000 7f8870d9d640 1 --2- 172.21.15.116:0/2721996936 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f88500678f0 0x7f8850069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:13.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.646+0000 7f8870d9d640 1 --2- 172.21.15.116:0/2721996936 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f88500678f0 0x7f8850069db0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f885c001ec0 tx=0x7f885c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:13.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.650+0000 7f8861ffb640 1 -- 172.21.15.116:0/2721996936 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f8868016110 con 0x7f88580039c0 2024-04-25T00:12:13.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.886+0000 7f8873802640 1 -- 172.21.15.116:0/2721996936 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f886c072c60 con 0x7f88580039c0 2024-04-25T00:12:13.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.890+0000 7f8861ffb640 1 -- 172.21.15.116:0/2721996936 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) ==== 94+0+6503 (secure 0 0 0) 0x7f88680f07a0 con 0x7f88580039c0 2024-04-25T00:12:13.888 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:12:13.888 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82},{"gid":5461,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:11:03.640578+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6114},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6114":{"gid":6114,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":25,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6114,"qdb_cluster":[6114]},"id":1},{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:11:53.790738+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6309},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6309":{"gid":6309,"name":"c","rank":0,"incarnation":78,"state":"up:active","state_seq":15,"addr":"172.21.15.116:6835/1599708526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1599708526},{"type":"v1","addr":"172.21.15.116:6835","nonce":1599708526}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6309,"qdb_cluster":[6309]},"id":2}]} 2024-04-25T00:12:13.888 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 82 2024-04-25T00:12:13.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.890+0000 7f8873802640 1 -- 172.21.15.116:0/2721996936 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f88500678f0 msgr2=0x7f8850069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:13.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.890+0000 7f8873802640 1 --2- 172.21.15.116:0/2721996936 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f88500678f0 0x7f8850069db0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f885c001ec0 tx=0x7f885c009040 comp rx=0 tx=0).stop 2024-04-25T00:12:13.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.890+0000 7f8873802640 1 -- 172.21.15.116:0/2721996936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88580039c0 msgr2=0x7f886c0c0ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:13.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.890+0000 7f8873802640 1 --2- 172.21.15.116:0/2721996936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88580039c0 0x7f886c0c0ba0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f886801c8e0 tx=0x7f8858005840 comp rx=0 tx=0).stop 2024-04-25T00:12:13.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.890+0000 7f8873802640 1 -- 172.21.15.116:0/2721996936 shutdown_connections 2024-04-25T00:12:13.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.890+0000 7f8873802640 1 --2- 172.21.15.116:0/2721996936 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f88500678f0 0x7f8850069db0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:13.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.890+0000 7f8873802640 1 --2- 172.21.15.116:0/2721996936 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f886c156cd0 0x7f886c073040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:13.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.890+0000 7f8873802640 1 --2- 172.21.15.116:0/2721996936 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f886c0ab020 0x7f886c0c6bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:13.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.890+0000 7f8873802640 1 --2- 172.21.15.116:0/2721996936 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88580039c0 0x7f886c0c0ba0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:13.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.890+0000 7f8873802640 1 -- 172.21.15.116:0/2721996936 >> 172.21.15.116:0/2721996936 conn(0x7f886c0c4270 msgr2=0x7f886c1525a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:13.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.890+0000 7f8873802640 1 -- 172.21.15.116:0/2721996936 shutdown_connections 2024-04-25T00:12:13.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:13.890+0000 7f8873802640 1 -- 172.21.15.116:0/2721996936 wait complete. 2024-04-25T00:12:13.909 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:12:15.911 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:12:16.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.054+0000 7fdcbba12640 1 Processor -- start 2024-04-25T00:12:16.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.054+0000 7fdcbba12640 1 -- start start 2024-04-25T00:12:16.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.054+0000 7fdcbba12640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fdcb4152f50 0x7fdcb40aa100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:16.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.054+0000 7fdcbba12640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdcb40aa640 0x7fdcb415cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:16.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.054+0000 7fdcbba12640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdcb4073260 con 0x7fdcb414fc20 2024-04-25T00:12:16.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.054+0000 7fdcbba12640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fdcb40a9620 con 0x7fdcb4152f50 2024-04-25T00:12:16.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.054+0000 7fdcbba12640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fdcb40a9900 con 0x7fdcb40aa640 2024-04-25T00:12:16.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.054+0000 7fdcb97ae640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fdcb414fc20 0x7fdcb406f6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:55884/0 (socket says 172.21.15.116:55884) 2024-04-25T00:12:16.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.054+0000 7fdcb97ae640 1 -- 172.21.15.116:0/2989026078 learned_addr learned my addr 172.21.15.116:0/2989026078 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:12:16.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.054+0000 7fdcb8fad640 1 --2- 172.21.15.116:0/2989026078 >> v2:172.21.15.174:3300/0 conn(0x7fdcb4152f50 0x7fdcb40aa100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:16.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.054+0000 7fdcb9faf640 1 --2- 172.21.15.116:0/2989026078 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdcb40aa640 0x7fdcb415cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:16.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.054+0000 7fdcb3fff640 1 -- 172.21.15.116:0/2989026078 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4249245788 0 0) 0x7fdcb4073260 con 0x7fdcb414fc20 2024-04-25T00:12:16.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.054+0000 7fdcb3fff640 1 -- 172.21.15.116:0/2989026078 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc9c005740 con 0x7fdcb414fc20 2024-04-25T00:12:16.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.054+0000 7fdcb3fff640 1 -- 172.21.15.116:0/2989026078 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2702960974 0 0) 0x7fdc9c005740 con 0x7fdcb414fc20 2024-04-25T00:12:16.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.054+0000 7fdcb8fad640 1 -- 172.21.15.116:0/2989026078 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdcb40aa640 msgr2=0x7fdcb415cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:12:16.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.054+0000 7fdcb8fad640 1 --2- 172.21.15.116:0/2989026078 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdcb40aa640 0x7fdcb415cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:16.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.054+0000 7fdcb8fad640 1 -- 172.21.15.116:0/2989026078 >> v1:172.21.15.116:6789/0 conn(0x7fdcb414fc20 legacy=0x7fdcb406f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:16.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.054+0000 7fdcb8fad640 1 -- 172.21.15.116:0/2989026078 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcb40a9be0 con 0x7fdcb4152f50 2024-04-25T00:12:16.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.058+0000 7fdcb8fad640 1 --2- 172.21.15.116:0/2989026078 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdcb4152f50 0x7fdcb40aa100 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fdca80097c0 tx=0x7fdca801c2a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e5c0709be353c38 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:16.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.058+0000 7fdcb3fff640 1 -- 172.21.15.116:0/2989026078 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdca80b0070 con 0x7fdcb4152f50 2024-04-25T00:12:16.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.058+0000 7fdcb3fff640 1 -- 172.21.15.116:0/2989026078 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdca80ab400 con 0x7fdcb4152f50 2024-04-25T00:12:16.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.058+0000 7fdcb3fff640 1 -- 172.21.15.116:0/2989026078 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdca4001a00 con 0x7fdcb4152f50 2024-04-25T00:12:16.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.058+0000 7fdcbba12640 1 -- 172.21.15.116:0/2989026078 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdcb4152f50 msgr2=0x7fdcb40aa100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:16.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.058+0000 7fdcbba12640 1 --2- 172.21.15.116:0/2989026078 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdcb4152f50 0x7fdcb40aa100 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fdca80097c0 tx=0x7fdca801c2a0 comp rx=0 tx=0).stop 2024-04-25T00:12:16.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.058+0000 7fdcbba12640 1 -- 172.21.15.116:0/2989026078 shutdown_connections 2024-04-25T00:12:16.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.058+0000 7fdcbba12640 1 --2- 172.21.15.116:0/2989026078 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdcb40aa640 0x7fdcb415cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:16.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.058+0000 7fdcbba12640 1 --2- 172.21.15.116:0/2989026078 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdcb4152f50 0x7fdcb40aa100 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:16.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.058+0000 7fdcbba12640 1 -- 172.21.15.116:0/2989026078 >> 172.21.15.116:0/2989026078 conn(0x7fdcb40c4c10 msgr2=0x7fdcb40c5020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:16.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.058+0000 7fdcbba12640 1 -- 172.21.15.116:0/2989026078 shutdown_connections 2024-04-25T00:12:16.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.058+0000 7fdcbba12640 1 -- 172.21.15.116:0/2989026078 wait complete. 2024-04-25T00:12:16.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.058+0000 7fdcbba12640 1 Processor -- start 2024-04-25T00:12:16.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.058+0000 7fdcbba12640 1 -- start start 2024-04-25T00:12:16.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.062+0000 7fdcbba12640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcb40aa640 0x7fdcb4147d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:16.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.062+0000 7fdcbba12640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdcb414fc20 0x7fdcb40c17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:16.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.062+0000 7fdcbba12640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdcb4152f50 0x7fdcb41ee230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:16.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.062+0000 7fdcbba12640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdcb406ff50 con 0x7fdcb40aa640 2024-04-25T00:12:16.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.062+0000 7fdcbba12640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fdcb406fdd0 con 0x7fdcb4152f50 2024-04-25T00:12:16.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.062+0000 7fdcbba12640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fdcb406ba20 con 0x7fdcb414fc20 2024-04-25T00:12:16.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.062+0000 7fdcb97ae640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcb40aa640 0x7fdcb4147d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:16.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.062+0000 7fdcb97ae640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcb40aa640 0x7fdcb4147d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32960/0 (socket says 172.21.15.116:32960) 2024-04-25T00:12:16.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.062+0000 7fdcb97ae640 1 -- 172.21.15.116:0/3065123308 learned_addr learned my addr 172.21.15.116:0/3065123308 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:12:16.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.062+0000 7fdcb97ae640 1 -- 172.21.15.116:0/3065123308 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdcb414fc20 msgr2=0x7fdcb40c17a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T00:12:16.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.062+0000 7fdcb97ae640 1 --2- 172.21.15.116:0/3065123308 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdcb414fc20 0x7fdcb40c17a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:16.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.062+0000 7fdcb97ae640 1 -- 172.21.15.116:0/3065123308 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdcb4152f50 msgr2=0x7fdcb41ee230 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T00:12:16.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.062+0000 7fdcb97ae640 1 --2- 172.21.15.116:0/3065123308 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdcb4152f50 0x7fdcb41ee230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:16.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.062+0000 7fdcb97ae640 1 -- 172.21.15.116:0/3065123308 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcb4075450 con 0x7fdcb40aa640 2024-04-25T00:12:16.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.062+0000 7fdcb97ae640 1 --2- 172.21.15.116:0/3065123308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcb40aa640 0x7fdcb4147d60 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fdca4001f60 tx=0x7fdca400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:16.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.062+0000 7fdcb1ffb640 1 -- 172.21.15.116:0/3065123308 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdca4017070 con 0x7fdcb40aa640 2024-04-25T00:12:16.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.062+0000 7fdcb1ffb640 1 -- 172.21.15.116:0/3065123308 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdca4012400 con 0x7fdcb40aa640 2024-04-25T00:12:16.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.062+0000 7fdcb1ffb640 1 -- 172.21.15.116:0/3065123308 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdca40127d0 con 0x7fdcb40aa640 2024-04-25T00:12:16.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.062+0000 7fdcbba12640 1 -- 172.21.15.116:0/3065123308 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcb41f1840 con 0x7fdcb40aa640 2024-04-25T00:12:16.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.062+0000 7fdcbba12640 1 -- 172.21.15.116:0/3065123308 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcb41f1e00 con 0x7fdcb40aa640 2024-04-25T00:12:16.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.062+0000 7fdcbba12640 1 -- 172.21.15.116:0/3065123308 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcb4068a60 con 0x7fdcb40aa640 2024-04-25T00:12:16.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.062+0000 7fdcb1ffb640 1 -- 172.21.15.116:0/3065123308 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fdca401a070 con 0x7fdcb40aa640 2024-04-25T00:12:16.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.062+0000 7fdcb1ffb640 1 --2- 172.21.15.116:0/3065123308 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdca00678f0 0x7fdca0069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:16.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.062+0000 7fdcb1ffb640 1 -- 172.21.15.116:0/3065123308 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7870+0+0 (secure 0 0 0) 0x7fdca4086450 con 0x7fdcb40aa640 2024-04-25T00:12:16.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.066+0000 7fdcb8fad640 1 --2- 172.21.15.116:0/3065123308 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdca00678f0 0x7fdca0069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:16.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.066+0000 7fdcb8fad640 1 --2- 172.21.15.116:0/3065123308 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdca00678f0 0x7fdca0069db0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fdca80ad000 tx=0x7fdca80a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:16.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.070+0000 7fdcb1ffb640 1 -- 172.21.15.116:0/3065123308 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fdca40567b0 con 0x7fdcb40aa640 2024-04-25T00:12:16.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.314+0000 7fdcbba12640 1 -- 172.21.15.116:0/3065123308 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdcb40c6d10 con 0x7fdcb40aa640 2024-04-25T00:12:16.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.314+0000 7fdcb1ffb640 1 -- 172.21.15.116:0/3065123308 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) ==== 94+0+6503 (secure 0 0 0) 0x7fdca40560b0 con 0x7fdcb40aa640 2024-04-25T00:12:16.314 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:12:16.314 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82},{"gid":5461,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:11:03.640578+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6114},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6114":{"gid":6114,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":25,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6114,"qdb_cluster":[6114]},"id":1},{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:11:53.790738+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6309},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6309":{"gid":6309,"name":"c","rank":0,"incarnation":78,"state":"up:active","state_seq":15,"addr":"172.21.15.116:6835/1599708526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1599708526},{"type":"v1","addr":"172.21.15.116:6835","nonce":1599708526}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6309,"qdb_cluster":[6309]},"id":2}]} 2024-04-25T00:12:16.314 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 82 2024-04-25T00:12:16.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.318+0000 7fdcbba12640 1 -- 172.21.15.116:0/3065123308 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdca00678f0 msgr2=0x7fdca0069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:16.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.318+0000 7fdcbba12640 1 --2- 172.21.15.116:0/3065123308 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdca00678f0 0x7fdca0069db0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fdca80ad000 tx=0x7fdca80a6000 comp rx=0 tx=0).stop 2024-04-25T00:12:16.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.318+0000 7fdcbba12640 1 -- 172.21.15.116:0/3065123308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcb40aa640 msgr2=0x7fdcb4147d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:16.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.318+0000 7fdcbba12640 1 --2- 172.21.15.116:0/3065123308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcb40aa640 0x7fdcb4147d60 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fdca4001f60 tx=0x7fdca400a430 comp rx=0 tx=0).stop 2024-04-25T00:12:16.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.318+0000 7fdcbba12640 1 -- 172.21.15.116:0/3065123308 shutdown_connections 2024-04-25T00:12:16.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.318+0000 7fdcbba12640 1 --2- 172.21.15.116:0/3065123308 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdca00678f0 0x7fdca0069db0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:16.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.318+0000 7fdcbba12640 1 --2- 172.21.15.116:0/3065123308 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdcb4152f50 0x7fdcb41ee230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:16.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.318+0000 7fdcbba12640 1 --2- 172.21.15.116:0/3065123308 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdcb414fc20 0x7fdcb40c17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:16.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.318+0000 7fdcbba12640 1 --2- 172.21.15.116:0/3065123308 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdcb40aa640 0x7fdcb4147d60 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:16.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.318+0000 7fdcbba12640 1 -- 172.21.15.116:0/3065123308 >> 172.21.15.116:0/3065123308 conn(0x7fdcb40c4c10 msgr2=0x7fdcb40c3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:16.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.318+0000 7fdcbba12640 1 -- 172.21.15.116:0/3065123308 shutdown_connections 2024-04-25T00:12:16.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:16.318+0000 7fdcbba12640 1 -- 172.21.15.116:0/3065123308 wait complete. 2024-04-25T00:12:16.336 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:12:18.337 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:12:18.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.474+0000 7f3bd9971640 1 Processor -- start 2024-04-25T00:12:18.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.474+0000 7f3bd9971640 1 -- start start 2024-04-25T00:12:18.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.474+0000 7f3bd9971640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3bd4150d20 0x7f3bd4153620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:18.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.474+0000 7f3bd9971640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3bd4071050 0x7f3bd40714e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:18.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.474+0000 7f3bd9971640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3bd40c36a0 con 0x7f3bd40c3240 2024-04-25T00:12:18.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.474+0000 7f3bd9971640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f3bd406fba0 con 0x7f3bd4071050 2024-04-25T00:12:18.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.474+0000 7f3bd9971640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f3bd406f8c0 con 0x7f3bd4150d20 2024-04-25T00:12:18.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.474+0000 7f3bd37fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3bd4071050 0x7f3bd40714e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:18.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.474+0000 7f3bd37fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3bd4071050 0x7f3bd40714e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32970/0 (socket says 172.21.15.116:32970) 2024-04-25T00:12:18.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.474+0000 7f3bd37fe640 1 -- 172.21.15.116:0/567366727 learned_addr learned my addr 172.21.15.116:0/567366727 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:12:18.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.474+0000 7f3bd2ffd640 1 --2- 172.21.15.116:0/567366727 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3bd4150d20 0x7f3bd4153620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:18.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.474+0000 7f3bd37fe640 1 -- 172.21.15.116:0/567366727 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3bd4150d20 msgr2=0x7f3bd4153620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:12:18.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.474+0000 7f3bd37fe640 1 --2- 172.21.15.116:0/567366727 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3bd4150d20 0x7f3bd4153620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:18.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.474+0000 7f3bd37fe640 1 -- 172.21.15.116:0/567366727 >> v1:172.21.15.174:6789/0 conn(0x7f3bd40c3240 legacy=0x7f3bd40c3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:12:18.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.474+0000 7f3bd37fe640 1 -- 172.21.15.116:0/567366727 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bd40c5420 con 0x7f3bd4071050 2024-04-25T00:12:18.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.478+0000 7f3bd37fe640 1 --2- 172.21.15.116:0/567366727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bd4071050 0x7f3bd40714e0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc009e50 tx=0x7f3bcc01c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cbb3cfcbe5587ab server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:18.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.478+0000 7f3bd1ffb640 1 -- 172.21.15.116:0/567366727 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bcc0a5780 con 0x7f3bd4071050 2024-04-25T00:12:18.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.478+0000 7f3bd1ffb640 1 -- 172.21.15.116:0/567366727 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bcc0a5920 con 0x7f3bd4071050 2024-04-25T00:12:18.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.478+0000 7f3bd1ffb640 1 -- 172.21.15.116:0/567366727 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bcc0a5cf0 con 0x7f3bd4071050 2024-04-25T00:12:18.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.478+0000 7f3bd9971640 1 -- 172.21.15.116:0/567366727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bd4071050 msgr2=0x7f3bd40714e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:18.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.478+0000 7f3bd9971640 1 --2- 172.21.15.116:0/567366727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bd4071050 0x7f3bd40714e0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc009e50 tx=0x7f3bcc01c9f0 comp rx=0 tx=0).stop 2024-04-25T00:12:18.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.478+0000 7f3bd9971640 1 -- 172.21.15.116:0/567366727 shutdown_connections 2024-04-25T00:12:18.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.478+0000 7f3bd9971640 1 --2- 172.21.15.116:0/567366727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bd4071050 0x7f3bd40714e0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:18.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.478+0000 7f3bd9971640 1 --2- 172.21.15.116:0/567366727 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3bd4150d20 0x7f3bd4153620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:18.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.478+0000 7f3bd9971640 1 -- 172.21.15.116:0/567366727 >> 172.21.15.116:0/567366727 conn(0x7f3bd40c9160 msgr2=0x7f3bd40c9570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:18.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.478+0000 7f3bd9971640 1 -- 172.21.15.116:0/567366727 shutdown_connections 2024-04-25T00:12:18.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.478+0000 7f3bd9971640 1 -- 172.21.15.116:0/567366727 wait complete. 2024-04-25T00:12:18.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.478+0000 7f3bd9971640 1 Processor -- start 2024-04-25T00:12:18.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.478+0000 7f3bd9971640 1 -- start start 2024-04-25T00:12:18.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.478+0000 7f3bd9971640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3bd4071050 0x7f3bd414b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:18.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.478+0000 7f3bd9971640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bd40c3240 0x7f3bd4152f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:18.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.478+0000 7f3bd9971640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3bd4150d20 0x7f3bd4140350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:18.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.478+0000 7f3bd9971640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3bd406f640 con 0x7f3bd40c3240 2024-04-25T00:12:18.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.478+0000 7f3bd9971640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3bd406f4c0 con 0x7f3bd4150d20 2024-04-25T00:12:18.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.478+0000 7f3bd9971640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f3bd4151100 con 0x7f3bd4071050 2024-04-25T00:12:18.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.478+0000 7f3bd27fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bd40c3240 0x7f3bd4152f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:18.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.478+0000 7f3bd27fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bd40c3240 0x7f3bd4152f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32972/0 (socket says 172.21.15.116:32972) 2024-04-25T00:12:18.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.478+0000 7f3bd27fc640 1 -- 172.21.15.116:0/51420480 learned_addr learned my addr 172.21.15.116:0/51420480 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:12:18.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.478+0000 7f3bd2ffd640 1 --2- 172.21.15.116:0/51420480 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3bd4071050 0x7f3bd414b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:18.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.478+0000 7f3bd37fe640 1 --2- 172.21.15.116:0/51420480 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3bd4150d20 0x7f3bd4140350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:18.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.478+0000 7f3bd27fc640 1 -- 172.21.15.116:0/51420480 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3bd4071050 msgr2=0x7f3bd414b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:18.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.478+0000 7f3bd27fc640 1 --2- 172.21.15.116:0/51420480 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3bd4071050 0x7f3bd414b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:18.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.478+0000 7f3bd27fc640 1 -- 172.21.15.116:0/51420480 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3bd4150d20 msgr2=0x7f3bd4140350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:18.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.478+0000 7f3bd27fc640 1 --2- 172.21.15.116:0/51420480 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3bd4150d20 0x7f3bd4140350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:18.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.478+0000 7f3bd27fc640 1 -- 172.21.15.116:0/51420480 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bd4151710 con 0x7f3bd40c3240 2024-04-25T00:12:18.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.478+0000 7f3bd2ffd640 1 --2- 172.21.15.116:0/51420480 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3bd4071050 0x7f3bd414b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:12:18.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.478+0000 7f3bd27fc640 1 --2- 172.21.15.116:0/51420480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bd40c3240 0x7f3bd4152f40 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f3bc800c480 tx=0x7f3bc800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:18.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.482+0000 7f3bbbfff640 1 -- 172.21.15.116:0/51420480 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bc8012830 con 0x7f3bd40c3240 2024-04-25T00:12:18.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.482+0000 7f3bbbfff640 1 -- 172.21.15.116:0/51420480 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bc8017070 con 0x7f3bd40c3240 2024-04-25T00:12:18.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.482+0000 7f3bd9971640 1 -- 172.21.15.116:0/51420480 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bd41519f0 con 0x7f3bd40c3240 2024-04-25T00:12:18.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.482+0000 7f3bbbfff640 1 -- 172.21.15.116:0/51420480 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bc8012c30 con 0x7f3bd40c3240 2024-04-25T00:12:18.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.482+0000 7f3bd9971640 1 -- 172.21.15.116:0/51420480 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bd4151fb0 con 0x7f3bd40c3240 2024-04-25T00:12:18.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.482+0000 7f3bd9971640 1 -- 172.21.15.116:0/51420480 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bd4068a60 con 0x7f3bd40c3240 2024-04-25T00:12:18.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.482+0000 7f3bbbfff640 1 -- 172.21.15.116:0/51420480 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3bc801c460 con 0x7f3bd40c3240 2024-04-25T00:12:18.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.482+0000 7f3bbbfff640 1 --2- 172.21.15.116:0/51420480 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3bc00678c0 0x7f3bc0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:18.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.482+0000 7f3bbbfff640 1 -- 172.21.15.116:0/51420480 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 7870+0+0 (secure 0 0 0) 0x7f3bc80862f0 con 0x7f3bd40c3240 2024-04-25T00:12:18.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.482+0000 7f3bd2ffd640 1 --2- 172.21.15.116:0/51420480 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3bc00678c0 0x7f3bc0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:18.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.486+0000 7f3bd2ffd640 1 --2- 172.21.15.116:0/51420480 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3bc00678c0 0x7f3bc0069d80 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f3bc4006fd0 tx=0x7f3bc4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:18.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.486+0000 7f3bbbfff640 1 -- 172.21.15.116:0/51420480 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f3bc8056650 con 0x7f3bd40c3240 2024-04-25T00:12:18.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.726+0000 7f3bd9971640 1 -- 172.21.15.116:0/51420480 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3bd40c4790 con 0x7f3bd40c3240 2024-04-25T00:12:18.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.726+0000 7f3bbbfff640 1 -- 172.21.15.116:0/51420480 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) ==== 94+0+6503 (secure 0 0 0) 0x7f3bc8055f50 con 0x7f3bd40c3240 2024-04-25T00:12:18.727 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:12:18.727 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82},{"gid":5461,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:11:03.640578+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6114},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6114":{"gid":6114,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":25,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6114,"qdb_cluster":[6114]},"id":1},{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:11:53.790738+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6309},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6309":{"gid":6309,"name":"c","rank":0,"incarnation":78,"state":"up:active","state_seq":15,"addr":"172.21.15.116:6835/1599708526","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1599708526},{"type":"v1","addr":"172.21.15.116:6835","nonce":1599708526}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6309,"qdb_cluster":[6309]},"id":2}]} 2024-04-25T00:12:18.727 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 82 2024-04-25T00:12:18.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.730+0000 7f3bd9971640 1 -- 172.21.15.116:0/51420480 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3bc00678c0 msgr2=0x7f3bc0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:18.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.730+0000 7f3bd9971640 1 --2- 172.21.15.116:0/51420480 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3bc00678c0 0x7f3bc0069d80 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f3bc4006fd0 tx=0x7f3bc4008040 comp rx=0 tx=0).stop 2024-04-25T00:12:18.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.730+0000 7f3bd9971640 1 -- 172.21.15.116:0/51420480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bd40c3240 msgr2=0x7f3bd4152f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:18.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.730+0000 7f3bd9971640 1 --2- 172.21.15.116:0/51420480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bd40c3240 0x7f3bd4152f40 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f3bc800c480 tx=0x7f3bc800c940 comp rx=0 tx=0).stop 2024-04-25T00:12:18.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.730+0000 7f3bd9971640 1 -- 172.21.15.116:0/51420480 shutdown_connections 2024-04-25T00:12:18.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.730+0000 7f3bd9971640 1 --2- 172.21.15.116:0/51420480 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3bc00678c0 0x7f3bc0069d80 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:18.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.730+0000 7f3bd9971640 1 --2- 172.21.15.116:0/51420480 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3bd4150d20 0x7f3bd4140350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:18.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.730+0000 7f3bd9971640 1 --2- 172.21.15.116:0/51420480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3bd40c3240 0x7f3bd4152f40 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:18.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.730+0000 7f3bd9971640 1 --2- 172.21.15.116:0/51420480 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3bd4071050 0x7f3bd414b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:18.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.730+0000 7f3bd9971640 1 -- 172.21.15.116:0/51420480 >> 172.21.15.116:0/51420480 conn(0x7f3bd40c9160 msgr2=0x7f3bd4143510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:18.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.730+0000 7f3bd9971640 1 -- 172.21.15.116:0/51420480 shutdown_connections 2024-04-25T00:12:18.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:18.730+0000 7f3bd9971640 1 -- 172.21.15.116:0/51420480 wait complete. 2024-04-25T00:12:18.746 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:12:20.747 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:12:20.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.870+0000 7fb8ecc2a640 1 Processor -- start 2024-04-25T00:12:20.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.870+0000 7fb8ecc2a640 1 -- start start 2024-04-25T00:12:20.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.874+0000 7fb8ecc2a640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fb8e8155c00 0x7fb8e8071520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:20.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.874+0000 7fb8ecc2a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb8e80730c0 0x7fb8e8155fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:20.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.874+0000 7fb8ecc2a640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8e8072920 con 0x7fb8e8073540 2024-04-25T00:12:20.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.874+0000 7fb8ecc2a640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fb8e8071bb0 con 0x7fb8e8155c00 2024-04-25T00:12:20.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.874+0000 7fb8ecc2a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fb8e8072ee0 con 0x7fb8e80730c0 2024-04-25T00:12:20.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.874+0000 7fb8e5d9b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb8e80730c0 0x7fb8e8155fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:20.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.874+0000 7fb8e659c640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fb8e8155c00 0x7fb8e8071520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:20.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.874+0000 7fb8e5d9b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb8e80730c0 0x7fb8e8155fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:57054/0 (socket says 172.21.15.116:57054) 2024-04-25T00:12:20.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.874+0000 7fb8e5d9b640 1 -- 172.21.15.116:0/4209123941 learned_addr learned my addr 172.21.15.116:0/4209123941 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:12:20.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.874+0000 7fb8e6d9d640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fb8e8073540 0x7fb8e814f7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:33224/0 (socket says 172.21.15.116:33224) 2024-04-25T00:12:20.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.874+0000 7fb8e659c640 1 -- 172.21.15.116:0/4209123941 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb8e80730c0 msgr2=0x7fb8e8155fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:12:20.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.874+0000 7fb8e659c640 1 --2- 172.21.15.116:0/4209123941 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb8e80730c0 0x7fb8e8155fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:20.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.874+0000 7fb8e659c640 1 -- 172.21.15.116:0/4209123941 >> v1:172.21.15.174:6789/0 conn(0x7fb8e8073540 legacy=0x7fb8e814f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:20.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.874+0000 7fb8e659c640 1 -- 172.21.15.116:0/4209123941 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8e8072c00 con 0x7fb8e8155c00 2024-04-25T00:12:20.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.874+0000 7fb8e659c640 1 --2- 172.21.15.116:0/4209123941 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb8e8155c00 0x7fb8e8071520 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb8d80097c0 tx=0x7fb8d801c380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=810f0a5500a4cb7f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:20.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.874+0000 7fb8e559a640 1 -- 172.21.15.116:0/4209123941 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d80b0070 con 0x7fb8e8155c00 2024-04-25T00:12:20.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.874+0000 7fb8e559a640 1 -- 172.21.15.116:0/4209123941 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8d80ab600 con 0x7fb8e8155c00 2024-04-25T00:12:20.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.874+0000 7fb8e559a640 1 -- 172.21.15.116:0/4209123941 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d80ab9d0 con 0x7fb8e8155c00 2024-04-25T00:12:20.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.878+0000 7fb8ecc2a640 1 -- 172.21.15.116:0/4209123941 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb8e8155c00 msgr2=0x7fb8e8071520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:20.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.878+0000 7fb8ecc2a640 1 --2- 172.21.15.116:0/4209123941 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb8e8155c00 0x7fb8e8071520 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb8d80097c0 tx=0x7fb8d801c380 comp rx=0 tx=0).stop 2024-04-25T00:12:20.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.878+0000 7fb8ecc2a640 1 -- 172.21.15.116:0/4209123941 shutdown_connections 2024-04-25T00:12:20.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.878+0000 7fb8ecc2a640 1 --2- 172.21.15.116:0/4209123941 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb8e80730c0 0x7fb8e8155fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:20.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.878+0000 7fb8ecc2a640 1 --2- 172.21.15.116:0/4209123941 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb8e8155c00 0x7fb8e8071520 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:20.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.878+0000 7fb8ecc2a640 1 -- 172.21.15.116:0/4209123941 >> 172.21.15.116:0/4209123941 conn(0x7fb8e80c4200 msgr2=0x7fb8e80c4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:20.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.878+0000 7fb8ecc2a640 1 -- 172.21.15.116:0/4209123941 shutdown_connections 2024-04-25T00:12:20.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.878+0000 7fb8ecc2a640 1 -- 172.21.15.116:0/4209123941 wait complete. 2024-04-25T00:12:20.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.878+0000 7fb8ecc2a640 1 Processor -- start 2024-04-25T00:12:20.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.878+0000 7fb8ecc2a640 1 -- start start 2024-04-25T00:12:20.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.878+0000 7fb8ecc2a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8e80730c0 0x7fb8e8153a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:20.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.878+0000 7fb8ecc2a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb8e8073540 0x7fb8e8142640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:20.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.878+0000 7fb8ecc2a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb8e8155c00 0x7fb8e8142b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:20.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.878+0000 7fb8ecc2a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb8e80c2280 con 0x7fb8e80730c0 2024-04-25T00:12:20.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.878+0000 7fb8ecc2a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb8e80c2100 con 0x7fb8e8073540 2024-04-25T00:12:20.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.878+0000 7fb8ecc2a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fb8e8156520 con 0x7fb8e8155c00 2024-04-25T00:12:20.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.878+0000 7fb8e659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8e80730c0 0x7fb8e8153a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:20.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.878+0000 7fb8e659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8e80730c0 0x7fb8e8153a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32974/0 (socket says 172.21.15.116:32974) 2024-04-25T00:12:20.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.878+0000 7fb8e659c640 1 -- 172.21.15.116:0/663348622 learned_addr learned my addr 172.21.15.116:0/663348622 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:12:20.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.878+0000 7fb8e6d9d640 1 --2- 172.21.15.116:0/663348622 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb8e8155c00 0x7fb8e8142b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:20.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.878+0000 7fb8e5d9b640 1 --2- 172.21.15.116:0/663348622 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb8e8073540 0x7fb8e8142640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:20.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.878+0000 7fb8e659c640 1 -- 172.21.15.116:0/663348622 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb8e8155c00 msgr2=0x7fb8e8142b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:20.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.878+0000 7fb8e659c640 1 --2- 172.21.15.116:0/663348622 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb8e8155c00 0x7fb8e8142b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:20.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.878+0000 7fb8e659c640 1 -- 172.21.15.116:0/663348622 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb8e8073540 msgr2=0x7fb8e8142640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:20.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.878+0000 7fb8e659c640 1 --2- 172.21.15.116:0/663348622 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb8e8073540 0x7fb8e8142640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:20.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.878+0000 7fb8e659c640 1 -- 172.21.15.116:0/663348622 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8e8147aa0 con 0x7fb8e80730c0 2024-04-25T00:12:20.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.878+0000 7fb8e5d9b640 1 --2- 172.21.15.116:0/663348622 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb8e8073540 0x7fb8e8142640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:12:20.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.878+0000 7fb8e6d9d640 1 --2- 172.21.15.116:0/663348622 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb8e8155c00 0x7fb8e8142b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:12:20.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.878+0000 7fb8e659c640 1 --2- 172.21.15.116:0/663348622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8e80730c0 0x7fb8e8153a50 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fb8d8019fb0 tx=0x7fb8d801abc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:20.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.878+0000 7fb8d6ffd640 1 -- 172.21.15.116:0/663348622 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d80b0050 con 0x7fb8e80730c0 2024-04-25T00:12:20.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.882+0000 7fb8d6ffd640 1 -- 172.21.15.116:0/663348622 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8d80bc070 con 0x7fb8e80730c0 2024-04-25T00:12:20.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.882+0000 7fb8d6ffd640 1 -- 172.21.15.116:0/663348622 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d8006e20 con 0x7fb8e80730c0 2024-04-25T00:12:20.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.882+0000 7fb8ecc2a640 1 -- 172.21.15.116:0/663348622 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8e81eb390 con 0x7fb8e80730c0 2024-04-25T00:12:20.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.882+0000 7fb8ecc2a640 1 -- 172.21.15.116:0/663348622 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8e80aaf70 con 0x7fb8e80730c0 2024-04-25T00:12:20.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.882+0000 7fb8ecc2a640 1 -- 172.21.15.116:0/663348622 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8e8068a60 con 0x7fb8e80730c0 2024-04-25T00:12:20.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.882+0000 7fb8d6ffd640 1 -- 172.21.15.116:0/663348622 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb8d80b5830 con 0x7fb8e80730c0 2024-04-25T00:12:20.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.882+0000 7fb8d6ffd640 1 --2- 172.21.15.116:0/663348622 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb8c80678c0 0x7fb8c8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:20.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.882+0000 7fb8d6ffd640 1 -- 172.21.15.116:0/663348622 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 7956+0+0 (secure 0 0 0) 0x7fb8d8120bf0 con 0x7fb8e80730c0 2024-04-25T00:12:20.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.882+0000 7fb8e5d9b640 1 --2- 172.21.15.116:0/663348622 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb8c80678c0 0x7fb8c8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:20.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.882+0000 7fb8e5d9b640 1 --2- 172.21.15.116:0/663348622 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb8c80678c0 0x7fb8c8069d80 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fb8e80c7570 tx=0x7fb8dc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:20.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:20.886+0000 7fb8d6ffd640 1 -- 172.21.15.116:0/663348622 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fb8d8016090 con 0x7fb8e80730c0 2024-04-25T00:12:21.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.158+0000 7fb8ecc2a640 1 -- 172.21.15.116:0/663348622 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb8e8075460 con 0x7fb8e80730c0 2024-04-25T00:12:21.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.162+0000 7fb8d6ffd640 1 -- 172.21.15.116:0/663348622 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) ==== 94+0+5667 (secure 0 0 0) 0x7fb8d80f0e50 con 0x7fb8e80730c0 2024-04-25T00:12:21.160 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:12:21.160 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:11:03.640578+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6114},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6114":{"gid":6114,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":25,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6114,"qdb_cluster":[6114]},"id":1},{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:12:20.527188+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5461},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5461":{"gid":5461,"name":"b","rank":0,"incarnation":83,"state":"up:replay","state_seq":1,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:12:21.160 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 83 2024-04-25T00:12:21.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.166+0000 7fb8ecc2a640 1 -- 172.21.15.116:0/663348622 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb8c80678c0 msgr2=0x7fb8c8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:21.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.166+0000 7fb8ecc2a640 1 --2- 172.21.15.116:0/663348622 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb8c80678c0 0x7fb8c8069d80 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fb8e80c7570 tx=0x7fb8dc008040 comp rx=0 tx=0).stop 2024-04-25T00:12:21.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.166+0000 7fb8ecc2a640 1 -- 172.21.15.116:0/663348622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8e80730c0 msgr2=0x7fb8e8153a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:21.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.166+0000 7fb8ecc2a640 1 --2- 172.21.15.116:0/663348622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8e80730c0 0x7fb8e8153a50 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fb8d8019fb0 tx=0x7fb8d801abc0 comp rx=0 tx=0).stop 2024-04-25T00:12:21.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.166+0000 7fb8ecc2a640 1 -- 172.21.15.116:0/663348622 shutdown_connections 2024-04-25T00:12:21.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.166+0000 7fb8ecc2a640 1 --2- 172.21.15.116:0/663348622 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb8c80678c0 0x7fb8c8069d80 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:21.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.166+0000 7fb8ecc2a640 1 --2- 172.21.15.116:0/663348622 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb8e8155c00 0x7fb8e8142b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:21.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.166+0000 7fb8ecc2a640 1 --2- 172.21.15.116:0/663348622 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb8e8073540 0x7fb8e8142640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:21.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.166+0000 7fb8ecc2a640 1 --2- 172.21.15.116:0/663348622 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb8e80730c0 0x7fb8e8153a50 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:21.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.166+0000 7fb8ecc2a640 1 -- 172.21.15.116:0/663348622 >> 172.21.15.116:0/663348622 conn(0x7fb8e80c4200 msgr2=0x7fb8e8155160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:21.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.166+0000 7fb8ecc2a640 1 -- 172.21.15.116:0/663348622 shutdown_connections 2024-04-25T00:12:21.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.166+0000 7fb8ecc2a640 1 -- 172.21.15.116:0/663348622 wait complete. 2024-04-25T00:12:21.183 INFO:tasks.mds_thrash.fs.[b]:mds.c down, removed from mdsmap 2024-04-25T00:12:21.183 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-04-25T00:12:21.183 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:12:21.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.298+0000 7f960ae7c640 1 Processor -- start 2024-04-25T00:12:21.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.298+0000 7f960ae7c640 1 -- start start 2024-04-25T00:12:21.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.298+0000 7f960ae7c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9604156200 0x7f960415de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:21.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.298+0000 7f960ae7c640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96041526a0 con 0x7f96040aa170 2024-04-25T00:12:21.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.298+0000 7f960ae7c640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9604152980 con 0x7f96040c4300 2024-04-25T00:12:21.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.298+0000 7f960ae7c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f9604153460 con 0x7f9604156200 2024-04-25T00:12:21.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.298+0000 7f9603fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f96040aa170 0x7f9604155af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:55890/0 (socket says 172.21.15.116:55890) 2024-04-25T00:12:21.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.298+0000 7f9603fff640 1 -- 172.21.15.116:0/18569149 learned_addr learned my addr 172.21.15.116:0/18569149 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:12:21.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.298+0000 7f9609419640 1 --2- 172.21.15.116:0/18569149 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9604156200 0x7f960415de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:21.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.298+0000 7f96037fe640 1 -- 172.21.15.116:0/18569149 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2230826054 0 0) 0x7f96041526a0 con 0x7f96040aa170 2024-04-25T00:12:21.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.298+0000 7f96037fe640 1 -- 172.21.15.116:0/18569149 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f95f4003530 con 0x7f96040aa170 2024-04-25T00:12:21.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.298+0000 7f96037fe640 1 -- 172.21.15.116:0/18569149 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 728531924 0 0) 0x7f95f4003530 con 0x7f96040aa170 2024-04-25T00:12:21.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.298+0000 7f96037fe640 1 -- 172.21.15.116:0/18569149 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9604156200 msgr2=0x7f960415de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:12:21.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.298+0000 7f96037fe640 1 --2- 172.21.15.116:0/18569149 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9604156200 0x7f960415de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:21.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.298+0000 7f96037fe640 1 -- 172.21.15.116:0/18569149 >> v1:172.21.15.174:6789/0 conn(0x7f96040c4300 legacy=0x7f96040c5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:21.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.298+0000 7f96037fe640 1 -- 172.21.15.116:0/18569149 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9604152c60 con 0x7f96040aa170 2024-04-25T00:12:21.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.298+0000 7f96037fe640 1 -- 172.21.15.116:0/18569149 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f95f0001680 con 0x7f96040aa170 2024-04-25T00:12:21.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.298+0000 7f96037fe640 1 -- 172.21.15.116:0/18569149 >> v1:172.21.15.116:6789/0 conn(0x7f96040aa170 legacy=0x7f9604155af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:21.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.298+0000 7f96037fe640 1 --2- 172.21.15.116:0/18569149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95f40039f0 0x7f95f4003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:21.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.298+0000 7f96037fe640 1 --2- 172.21.15.116:0/18569149 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f95f4004810 0x7f95f4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:21.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.298+0000 7f96037fe640 1 --2- 172.21.15.116:0/18569149 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9604156200 0x7f95f40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:21.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.298+0000 7f96037fe640 1 -- 172.21.15.116:0/18569149 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9604153460 con 0x7f95f40039f0 2024-04-25T00:12:21.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.298+0000 7f96037fe640 1 -- 172.21.15.116:0/18569149 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f95f400bab0 con 0x7f95f4004810 2024-04-25T00:12:21.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.298+0000 7f96037fe640 1 -- 172.21.15.116:0/18569149 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f95f400bcb0 con 0x7f9604156200 2024-04-25T00:12:21.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.298+0000 7f9608c18640 1 --2- 172.21.15.116:0/18569149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95f40039f0 0x7f95f4003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:21.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.298+0000 7f9603fff640 1 --2- 172.21.15.116:0/18569149 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f95f4004810 0x7f95f4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:21.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.298+0000 7f9608c18640 1 -- 172.21.15.116:0/18569149 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9604156200 msgr2=0x7f95f40051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:12:21.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.298+0000 7f9608c18640 1 --2- 172.21.15.116:0/18569149 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9604156200 0x7f95f40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:21.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.298+0000 7f9608c18640 1 -- 172.21.15.116:0/18569149 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f95f4004810 msgr2=0x7f95f4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:12:21.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.298+0000 7f9608c18640 1 --2- 172.21.15.116:0/18569149 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f95f4004810 0x7f95f4004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:21.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.298+0000 7f9608c18640 1 -- 172.21.15.116:0/18569149 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95f4003530 con 0x7f95f40039f0 2024-04-25T00:12:21.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.298+0000 7f9603fff640 1 --2- 172.21.15.116:0/18569149 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f95f4004810 0x7f95f4004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:12:21.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.302+0000 7f9608c18640 1 --2- 172.21.15.116:0/18569149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95f40039f0 0x7f95f4003e10 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f95f8001750 tx=0x7f95f801c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c3b6cc211d76386 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:21.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.302+0000 7f96037fe640 1 -- 172.21.15.116:0/18569149 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95f80013a0 con 0x7f95f40039f0 2024-04-25T00:12:21.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.302+0000 7f96037fe640 1 -- 172.21.15.116:0/18569149 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95f80052f0 con 0x7f95f40039f0 2024-04-25T00:12:21.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.302+0000 7f96037fe640 1 -- 172.21.15.116:0/18569149 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95f8003840 con 0x7f95f40039f0 2024-04-25T00:12:21.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.302+0000 7f960ae7c640 1 -- 172.21.15.116:0/18569149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95f40039f0 msgr2=0x7f95f4003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:21.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.302+0000 7f960ae7c640 1 --2- 172.21.15.116:0/18569149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95f40039f0 0x7f95f4003e10 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f95f8001750 tx=0x7f95f801c3c0 comp rx=0 tx=0).stop 2024-04-25T00:12:21.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.302+0000 7f9609419640 1 -- 172.21.15.116:0/18569149 reap_dead start 2024-04-25T00:12:21.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.302+0000 7f960ae7c640 1 -- 172.21.15.116:0/18569149 shutdown_connections 2024-04-25T00:12:21.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.302+0000 7f960ae7c640 1 -- 172.21.15.116:0/18569149 >> 172.21.15.116:0/18569149 conn(0x7f960406c720 msgr2=0x7f960406cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:21.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.302+0000 7f960ae7c640 1 -- 172.21.15.116:0/18569149 shutdown_connections 2024-04-25T00:12:21.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.302+0000 7f960ae7c640 1 -- 172.21.15.116:0/18569149 wait complete. 2024-04-25T00:12:21.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.302+0000 7f960ae7c640 1 Processor -- start 2024-04-25T00:12:21.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.302+0000 7f960ae7c640 1 -- start start 2024-04-25T00:12:21.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.306+0000 7f960ae7c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95f40039f0 0x7f9604145900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:21.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.306+0000 7f960ae7c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f96040ab150 0x7f96040acc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:21.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.306+0000 7f960ae7c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96041ec380 0x7f960414b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:21.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.306+0000 7f960ae7c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f96040aacd0 con 0x7f95f40039f0 2024-04-25T00:12:21.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.306+0000 7f960ae7c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f960406ffd0 con 0x7f96041ec380 2024-04-25T00:12:21.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.306+0000 7f960ae7c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f96040aae50 con 0x7f96040ab150 2024-04-25T00:12:21.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.306+0000 7f9608c18640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95f40039f0 0x7f9604145900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:21.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.306+0000 7f9608c18640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95f40039f0 0x7f9604145900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33000/0 (socket says 172.21.15.116:33000) 2024-04-25T00:12:21.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.306+0000 7f9608c18640 1 -- 172.21.15.116:0/27025403 learned_addr learned my addr 172.21.15.116:0/27025403 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:12:21.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.306+0000 7f9603fff640 1 --2- 172.21.15.116:0/27025403 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f96040ab150 0x7f96040acc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:21.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.306+0000 7f9609419640 1 --2- 172.21.15.116:0/27025403 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96041ec380 0x7f960414b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:21.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.306+0000 7f9608c18640 1 -- 172.21.15.116:0/27025403 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f96040ab150 msgr2=0x7f96040acc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:21.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.306+0000 7f9608c18640 1 --2- 172.21.15.116:0/27025403 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f96040ab150 0x7f96040acc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:21.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.306+0000 7f9608c18640 1 -- 172.21.15.116:0/27025403 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96041ec380 msgr2=0x7f960414b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:21.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.306+0000 7f9608c18640 1 --2- 172.21.15.116:0/27025403 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96041ec380 0x7f960414b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:21.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.306+0000 7f9608c18640 1 -- 172.21.15.116:0/27025403 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f960406b210 con 0x7f95f40039f0 2024-04-25T00:12:21.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.306+0000 7f9603fff640 1 --2- 172.21.15.116:0/27025403 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f96040ab150 0x7f96040acc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:12:21.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.306+0000 7f9608c18640 1 --2- 172.21.15.116:0/27025403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95f40039f0 0x7f9604145900 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f95f801c8a0 tx=0x7f95f4005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:21.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.306+0000 7f96017fa640 1 -- 172.21.15.116:0/27025403 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95f80b8070 con 0x7f95f40039f0 2024-04-25T00:12:21.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.306+0000 7f96017fa640 1 -- 172.21.15.116:0/27025403 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95f80054a0 con 0x7f95f40039f0 2024-04-25T00:12:21.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.306+0000 7f960ae7c640 1 -- 172.21.15.116:0/27025403 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f960406b4f0 con 0x7f95f40039f0 2024-04-25T00:12:21.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.306+0000 7f96017fa640 1 -- 172.21.15.116:0/27025403 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95f8007a40 con 0x7f95f40039f0 2024-04-25T00:12:21.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.306+0000 7f960ae7c640 1 -- 172.21.15.116:0/27025403 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f960406bab0 con 0x7f95f40039f0 2024-04-25T00:12:21.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.306+0000 7f960ae7c640 1 -- 172.21.15.116:0/27025403 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9604068a60 con 0x7f95f40039f0 2024-04-25T00:12:21.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.306+0000 7f96017fa640 1 -- 172.21.15.116:0/27025403 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f95f80bb070 con 0x7f95f40039f0 2024-04-25T00:12:21.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.310+0000 7f96017fa640 1 --2- 172.21.15.116:0/27025403 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f95e80678f0 0x7f95e8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:21.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.310+0000 7f96017fa640 1 -- 172.21.15.116:0/27025403 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 7956+0+0 (secure 0 0 0) 0x7f95f81204e0 con 0x7f95f40039f0 2024-04-25T00:12:21.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.310+0000 7f9603fff640 1 --2- 172.21.15.116:0/27025403 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f95e80678f0 0x7f95e8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:21.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.310+0000 7f9603fff640 1 --2- 172.21.15.116:0/27025403 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f95e80678f0 0x7f95e8069db0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f95f0001210 tx=0x7f95f0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:21.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.314+0000 7f96017fa640 1 -- 172.21.15.116:0/27025403 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f95f80160d0 con 0x7f95f40039f0 2024-04-25T00:12:21.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.558+0000 7f960ae7c640 1 -- 172.21.15.116:0/27025403 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f96040ab7d0 con 0x7f95f40039f0 2024-04-25T00:12:21.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.558+0000 7f96017fa640 1 -- 172.21.15.116:0/27025403 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) ==== 94+0+5667 (secure 0 0 0) 0x7f95f80f07e0 con 0x7f95f40039f0 2024-04-25T00:12:21.559 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:12:21.559 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:11:03.640578+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6114},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6114":{"gid":6114,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":25,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6114,"qdb_cluster":[6114]},"id":1},{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:12:20.527188+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5461},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5461":{"gid":5461,"name":"b","rank":0,"incarnation":83,"state":"up:replay","state_seq":1,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:12:21.559 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 83 2024-04-25T00:12:21.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.562+0000 7f960ae7c640 1 -- 172.21.15.116:0/27025403 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f95e80678f0 msgr2=0x7f95e8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:21.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.562+0000 7f960ae7c640 1 --2- 172.21.15.116:0/27025403 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f95e80678f0 0x7f95e8069db0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f95f0001210 tx=0x7f95f0009040 comp rx=0 tx=0).stop 2024-04-25T00:12:21.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.562+0000 7f960ae7c640 1 -- 172.21.15.116:0/27025403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95f40039f0 msgr2=0x7f9604145900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:21.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.562+0000 7f960ae7c640 1 --2- 172.21.15.116:0/27025403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95f40039f0 0x7f9604145900 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f95f801c8a0 tx=0x7f95f4005840 comp rx=0 tx=0).stop 2024-04-25T00:12:21.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.562+0000 7f960ae7c640 1 -- 172.21.15.116:0/27025403 shutdown_connections 2024-04-25T00:12:21.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.562+0000 7f960ae7c640 1 --2- 172.21.15.116:0/27025403 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f95e80678f0 0x7f95e8069db0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:21.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.562+0000 7f960ae7c640 1 --2- 172.21.15.116:0/27025403 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96041ec380 0x7f960414b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:21.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.562+0000 7f960ae7c640 1 --2- 172.21.15.116:0/27025403 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f96040ab150 0x7f96040acc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:21.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.562+0000 7f960ae7c640 1 --2- 172.21.15.116:0/27025403 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95f40039f0 0x7f9604145900 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:21.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.562+0000 7f960ae7c640 1 -- 172.21.15.116:0/27025403 >> 172.21.15.116:0/27025403 conn(0x7f960406c720 msgr2=0x7f96040c5100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:21.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.562+0000 7f960ae7c640 1 -- 172.21.15.116:0/27025403 shutdown_connections 2024-04-25T00:12:21.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:21.562+0000 7f960ae7c640 1 -- 172.21.15.116:0/27025403 wait complete. 2024-04-25T00:12:21.579 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 83, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:04.678737+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 76, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6114": { "addr": "172.21.15.116:6833/3705654249", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6832", "nonce": 3705654249, "type": "v2" }, { "addr": "172.21.15.116:6833", "nonce": 3705654249, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6114, "incarnation": 73, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 25 } }, "last_failure": 0, "last_failure_osd_epoch": 41, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T00:11:03.640578+0000", "qdb_cluster": [ 6114 ], "qdb_leader": 6114, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6114 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:18.825887+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 83, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5461": { "addr": "172.21.15.174:6837/3398690896", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6836", "nonce": 3398690896, "type": "v2" }, { "addr": "172.21.15.174:6837", "nonce": 3398690896, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5461, "incarnation": 83, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 43, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-04-25T00:12:20.527188+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5461 } } } ], "standbys": [ { "addr": "172.21.15.174:6835/4215220222", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6834", "nonce": 4215220222, "type": "v2" }, { "addr": "172.21.15.174:6835", "nonce": 4215220222, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 82, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5273, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-25T00:12:23.580 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:12:23.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.714+0000 7fe868c16640 1 Processor -- start 2024-04-25T00:12:23.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.714+0000 7fe868c16640 1 -- start start 2024-04-25T00:12:23.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.714+0000 7fe868c16640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fe8640c3c90 0x7fe8640c4100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:23.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.714+0000 7fe868c16640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe864156760 con 0x7fe864153140 2024-04-25T00:12:23.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.714+0000 7fe868c16640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe864155be0 con 0x7fe864152d60 2024-04-25T00:12:23.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.714+0000 7fe868c16640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fe864155ec0 con 0x7fe8640c3c90 2024-04-25T00:12:23.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.714+0000 7fe861d9b640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fe864153140 0x7fe8640c8450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:55904/0 (socket says 172.21.15.116:55904) 2024-04-25T00:12:23.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.714+0000 7fe861d9b640 1 -- 172.21.15.116:0/764950448 learned_addr learned my addr 172.21.15.116:0/764950448 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:12:23.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.714+0000 7fe862d9d640 1 --2- 172.21.15.116:0/764950448 >> v2:172.21.15.174:3300/0 conn(0x7fe8640c3c90 0x7fe8640c4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:23.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.714+0000 7fe86159a640 1 -- 172.21.15.116:0/764950448 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3890451060 0 0) 0x7fe864156760 con 0x7fe864153140 2024-04-25T00:12:23.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.714+0000 7fe86159a640 1 -- 172.21.15.116:0/764950448 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe84c003530 con 0x7fe864153140 2024-04-25T00:12:23.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.714+0000 7fe86159a640 1 -- 172.21.15.116:0/764950448 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3003433606 0 0) 0x7fe864155be0 con 0x7fe864152d60 2024-04-25T00:12:23.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.714+0000 7fe86159a640 1 -- 172.21.15.116:0/764950448 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe864156760 con 0x7fe864152d60 2024-04-25T00:12:23.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.714+0000 7fe86159a640 1 -- 172.21.15.116:0/764950448 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 113612999 0 0) 0x7fe84c003530 con 0x7fe864153140 2024-04-25T00:12:23.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.714+0000 7fe86159a640 1 -- 172.21.15.116:0/764950448 >> v2:172.21.15.174:3300/0 conn(0x7fe8640c3c90 msgr2=0x7fe8640c4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:12:23.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.714+0000 7fe86159a640 1 --2- 172.21.15.116:0/764950448 >> v2:172.21.15.174:3300/0 conn(0x7fe8640c3c90 0x7fe8640c4100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:23.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.714+0000 7fe86159a640 1 -- 172.21.15.116:0/764950448 >> v1:172.21.15.174:6789/0 conn(0x7fe864152d60 legacy=0x7fe8640ab250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:23.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.714+0000 7fe86159a640 1 -- 172.21.15.116:0/764950448 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8641561a0 con 0x7fe864153140 2024-04-25T00:12:23.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.714+0000 7fe86159a640 1 -- 172.21.15.116:0/764950448 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7fe858001600 con 0x7fe864153140 2024-04-25T00:12:23.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.714+0000 7fe86159a640 1 -- 172.21.15.116:0/764950448 >> v1:172.21.15.116:6789/0 conn(0x7fe864153140 legacy=0x7fe8640c8450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:23.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.714+0000 7fe86159a640 1 --2- 172.21.15.116:0/764950448 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe84c003ba0 0x7fe84c003fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:23.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.714+0000 7fe86159a640 1 --2- 172.21.15.116:0/764950448 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe84c004810 0x7fe84c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:23.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.714+0000 7fe86159a640 1 --2- 172.21.15.116:0/764950448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe84c005220 0x7fe84c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:23.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.714+0000 7fe86159a640 1 -- 172.21.15.116:0/764950448 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe864155ec0 con 0x7fe84c005220 2024-04-25T00:12:23.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.714+0000 7fe86159a640 1 -- 172.21.15.116:0/764950448 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe84c00c150 con 0x7fe84c004810 2024-04-25T00:12:23.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.718+0000 7fe86159a640 1 -- 172.21.15.116:0/764950448 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe84c00c350 con 0x7fe84c003ba0 2024-04-25T00:12:23.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.718+0000 7fe862d9d640 1 --2- 172.21.15.116:0/764950448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe84c005220 0x7fe84c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:23.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.718+0000 7fe861d9b640 1 --2- 172.21.15.116:0/764950448 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe84c004810 0x7fe84c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:23.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.718+0000 7fe86259c640 1 --2- 172.21.15.116:0/764950448 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe84c003ba0 0x7fe84c003fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:23.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.718+0000 7fe862d9d640 1 -- 172.21.15.116:0/764950448 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe84c003ba0 msgr2=0x7fe84c003fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:12:23.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.718+0000 7fe862d9d640 1 --2- 172.21.15.116:0/764950448 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe84c003ba0 0x7fe84c003fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:23.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.718+0000 7fe862d9d640 1 -- 172.21.15.116:0/764950448 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe84c004810 msgr2=0x7fe84c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:12:23.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.718+0000 7fe862d9d640 1 --2- 172.21.15.116:0/764950448 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe84c004810 0x7fe84c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:23.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.718+0000 7fe862d9d640 1 -- 172.21.15.116:0/764950448 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe84c003530 con 0x7fe84c005220 2024-04-25T00:12:23.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.718+0000 7fe862d9d640 1 -- 172.21.15.116:0/764950448 reap_dead start 2024-04-25T00:12:23.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.718+0000 7fe862d9d640 1 --2- 172.21.15.116:0/764950448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe84c005220 0x7fe84c00b9d0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fe85c009fc0 tx=0x7fe85c01ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29987da00e5dde37 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:23.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.718+0000 7fe86159a640 1 -- 172.21.15.116:0/764950448 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe85c009c10 con 0x7fe84c005220 2024-04-25T00:12:23.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.718+0000 7fe86159a640 1 -- 172.21.15.116:0/764950448 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe85c0a5d10 con 0x7fe84c005220 2024-04-25T00:12:23.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.718+0000 7fe86159a640 1 -- 172.21.15.116:0/764950448 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe85c003be0 con 0x7fe84c005220 2024-04-25T00:12:23.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.722+0000 7fe868c16640 1 -- 172.21.15.116:0/764950448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe84c005220 msgr2=0x7fe84c00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:23.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.722+0000 7fe868c16640 1 --2- 172.21.15.116:0/764950448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe84c005220 0x7fe84c00b9d0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fe85c009fc0 tx=0x7fe85c01ccc0 comp rx=0 tx=0).stop 2024-04-25T00:12:23.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.722+0000 7fe868c16640 1 -- 172.21.15.116:0/764950448 shutdown_connections 2024-04-25T00:12:23.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.722+0000 7fe868c16640 1 --2- 172.21.15.116:0/764950448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe84c005220 0x7fe84c00b9d0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:23.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.722+0000 7fe868c16640 1 -- 172.21.15.116:0/764950448 >> 172.21.15.116:0/764950448 conn(0x7fe86406f0c0 msgr2=0x7fe86406f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:23.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.722+0000 7fe868c16640 1 -- 172.21.15.116:0/764950448 shutdown_connections 2024-04-25T00:12:23.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.722+0000 7fe868c16640 1 -- 172.21.15.116:0/764950448 wait complete. 2024-04-25T00:12:23.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.722+0000 7fe868c16640 1 Processor -- start 2024-04-25T00:12:23.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.722+0000 7fe868c16640 1 -- start start 2024-04-25T00:12:23.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.722+0000 7fe868c16640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe84c005220 0x7fe8641eb650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:23.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.722+0000 7fe868c16640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe86414e5b0 0x7fe864154070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:23.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.722+0000 7fe868c16640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe8641ee840 0x7fe8641eecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:23.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.722+0000 7fe868c16640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe864155980 con 0x7fe86414e5b0 2024-04-25T00:12:23.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.722+0000 7fe868c16640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe864155800 con 0x7fe8641ee840 2024-04-25T00:12:23.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.722+0000 7fe868c16640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe8640ab250 con 0x7fe84c005220 2024-04-25T00:12:23.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.722+0000 7fe861d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe86414e5b0 0x7fe864154070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:23.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.722+0000 7fe861d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe86414e5b0 0x7fe864154070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33028/0 (socket says 172.21.15.116:33028) 2024-04-25T00:12:23.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.722+0000 7fe861d9b640 1 -- 172.21.15.116:0/2526605231 learned_addr learned my addr 172.21.15.116:0/2526605231 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:12:23.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.722+0000 7fe86259c640 1 --2- 172.21.15.116:0/2526605231 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe84c005220 0x7fe8641eb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:23.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.722+0000 7fe862d9d640 1 --2- 172.21.15.116:0/2526605231 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe8641ee840 0x7fe8641eecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:23.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.722+0000 7fe861d9b640 1 -- 172.21.15.116:0/2526605231 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe84c005220 msgr2=0x7fe8641eb650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:23.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.722+0000 7fe861d9b640 1 --2- 172.21.15.116:0/2526605231 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe84c005220 0x7fe8641eb650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:23.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.722+0000 7fe861d9b640 1 -- 172.21.15.116:0/2526605231 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe8641ee840 msgr2=0x7fe8641eecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:23.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.722+0000 7fe861d9b640 1 --2- 172.21.15.116:0/2526605231 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe8641ee840 0x7fe8641eecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:23.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.722+0000 7fe861d9b640 1 -- 172.21.15.116:0/2526605231 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe86414d320 con 0x7fe86414e5b0 2024-04-25T00:12:23.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.722+0000 7fe861d9b640 1 --2- 172.21.15.116:0/2526605231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe86414e5b0 0x7fe864154070 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fe858001770 tx=0x7fe858007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:23.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.722+0000 7fe852ffd640 1 -- 172.21.15.116:0/2526605231 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe85800f7b0 con 0x7fe86414e5b0 2024-04-25T00:12:23.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.722+0000 7fe852ffd640 1 -- 172.21.15.116:0/2526605231 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe858018070 con 0x7fe86414e5b0 2024-04-25T00:12:23.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.722+0000 7fe868c16640 1 -- 172.21.15.116:0/2526605231 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe86414d600 con 0x7fe86414e5b0 2024-04-25T00:12:23.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.722+0000 7fe852ffd640 1 -- 172.21.15.116:0/2526605231 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe85800fb90 con 0x7fe86414e5b0 2024-04-25T00:12:23.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.722+0000 7fe868c16640 1 -- 172.21.15.116:0/2526605231 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8641ee320 con 0x7fe86414e5b0 2024-04-25T00:12:23.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.726+0000 7fe868c16640 1 -- 172.21.15.116:0/2526605231 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8641ee520 con 0x7fe86414e5b0 2024-04-25T00:12:23.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.726+0000 7fe852ffd640 1 -- 172.21.15.116:0/2526605231 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe85800fd30 con 0x7fe86414e5b0 2024-04-25T00:12:23.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.726+0000 7fe852ffd640 1 --2- 172.21.15.116:0/2526605231 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe8440678c0 0x7fe844069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:23.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.726+0000 7fe852ffd640 1 -- 172.21.15.116:0/2526605231 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 7956+0+0 (secure 0 0 0) 0x7fe858086340 con 0x7fe86414e5b0 2024-04-25T00:12:23.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.726+0000 7fe86259c640 1 --2- 172.21.15.116:0/2526605231 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe8440678c0 0x7fe844069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:23.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.726+0000 7fe86259c640 1 --2- 172.21.15.116:0/2526605231 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe8440678c0 0x7fe844069d80 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fe854005630 tx=0x7fe854001490 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:23.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.730+0000 7fe852ffd640 1 -- 172.21.15.116:0/2526605231 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fe8641ee520 con 0x7fe86414e5b0 2024-04-25T00:12:23.824 INFO:tasks.ceph.mds.b.smithi174.stderr:2024-04-25T00:12:23.823+0000 7f5e70d81640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T00:12:24.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.966+0000 7fe868c16640 1 -- 172.21.15.116:0/2526605231 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe864068a60 con 0x7fe86414e5b0 2024-04-25T00:12:24.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.966+0000 7fe852ffd640 1 -- 172.21.15.116:0/2526605231 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) ==== 94+0+5675 (secure 0 0 0) 0x7fe858056640 con 0x7fe86414e5b0 2024-04-25T00:12:24.009 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:12:24.009 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:11:03.640578+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6114},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6114":{"gid":6114,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":25,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6114,"qdb_cluster":[6114]},"id":1},{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:12:23.643548+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5461},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5461":{"gid":5461,"name":"b","rank":0,"incarnation":83,"state":"up:active","state_seq":30,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5461,"qdb_cluster":[5461]},"id":2}]} 2024-04-25T00:12:24.010 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 86 2024-04-25T00:12:24.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.970+0000 7fe868c16640 1 -- 172.21.15.116:0/2526605231 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe8440678c0 msgr2=0x7fe844069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:24.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.970+0000 7fe868c16640 1 --2- 172.21.15.116:0/2526605231 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe8440678c0 0x7fe844069d80 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fe854005630 tx=0x7fe854001490 comp rx=0 tx=0).stop 2024-04-25T00:12:24.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.970+0000 7fe868c16640 1 -- 172.21.15.116:0/2526605231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe86414e5b0 msgr2=0x7fe864154070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:24.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.970+0000 7fe868c16640 1 --2- 172.21.15.116:0/2526605231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe86414e5b0 0x7fe864154070 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fe858001770 tx=0x7fe858007430 comp rx=0 tx=0).stop 2024-04-25T00:12:24.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.970+0000 7fe868c16640 1 -- 172.21.15.116:0/2526605231 shutdown_connections 2024-04-25T00:12:24.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.970+0000 7fe868c16640 1 --2- 172.21.15.116:0/2526605231 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe8440678c0 0x7fe844069d80 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:24.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.970+0000 7fe868c16640 1 --2- 172.21.15.116:0/2526605231 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe8641ee840 0x7fe8641eecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:24.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.970+0000 7fe868c16640 1 --2- 172.21.15.116:0/2526605231 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe86414e5b0 0x7fe864154070 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:24.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.970+0000 7fe868c16640 1 --2- 172.21.15.116:0/2526605231 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe84c005220 0x7fe8641eb650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:24.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.970+0000 7fe868c16640 1 -- 172.21.15.116:0/2526605231 >> 172.21.15.116:0/2526605231 conn(0x7fe86406f0c0 msgr2=0x7fe86406c5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:24.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.970+0000 7fe868c16640 1 -- 172.21.15.116:0/2526605231 shutdown_connections 2024-04-25T00:12:24.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:23.970+0000 7fe868c16640 1 -- 172.21.15.116:0/2526605231 wait complete. 2024-04-25T00:12:24.012 INFO:tasks.mds_thrash.fs.[b]:mds.b has gained rank=0, replacing gid=6309 2024-04-25T00:12:24.012 INFO:tasks.mds_thrash.fs.[b]:waiting for 6 secs before reviving mds.c 2024-04-25T00:12:27.004 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:12:27.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.134+0000 7f643d25b640 1 Processor -- start 2024-04-25T00:12:27.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.134+0000 7f643d25b640 1 -- start start 2024-04-25T00:12:27.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.138+0000 7f643d25b640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f643814fc20 0x7f6438152f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:27.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.138+0000 7f643d25b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f643806fb20 0x7f64380aa810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:27.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.138+0000 7f643d25b640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f643806ff90 con 0x7f643806f6c0 2024-04-25T00:12:27.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.138+0000 7f643d25b640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f64380aad50 con 0x7f643814fc20 2024-04-25T00:12:27.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.138+0000 7f643d25b640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6438073260 con 0x7f643806fb20 2024-04-25T00:12:27.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.138+0000 7f643659c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f643806f6c0 0x7f64380aa100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:52180/0 (socket says 172.21.15.116:52180) 2024-04-25T00:12:27.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.138+0000 7f643659c640 1 -- 172.21.15.116:0/2443870642 learned_addr learned my addr 172.21.15.116:0/2443870642 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:12:27.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.138+0000 7f6436d9d640 1 --2- 172.21.15.116:0/2443870642 >> v2:172.21.15.174:3300/0 conn(0x7f643814fc20 0x7f6438152f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:27.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.138+0000 7f643759e640 1 --2- 172.21.15.116:0/2443870642 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f643806fb20 0x7f64380aa810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:27.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.138+0000 7f6435d9b640 1 -- 172.21.15.116:0/2443870642 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 631714531 0 0) 0x7f643806ff90 con 0x7f643806f6c0 2024-04-25T00:12:27.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.138+0000 7f6435d9b640 1 -- 172.21.15.116:0/2443870642 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6420003530 con 0x7f643806f6c0 2024-04-25T00:12:27.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.138+0000 7f6435d9b640 1 -- 172.21.15.116:0/2443870642 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 622137072 0 0) 0x7f6420003530 con 0x7f643806f6c0 2024-04-25T00:12:27.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.138+0000 7f6435d9b640 1 -- 172.21.15.116:0/2443870642 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f643806fb20 msgr2=0x7f64380aa810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:12:27.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.138+0000 7f6435d9b640 1 --2- 172.21.15.116:0/2443870642 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f643806fb20 0x7f64380aa810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:27.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.138+0000 7f6435d9b640 1 -- 172.21.15.116:0/2443870642 >> v2:172.21.15.174:3300/0 conn(0x7f643814fc20 msgr2=0x7f6438152f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:12:27.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.138+0000 7f6435d9b640 1 --2- 172.21.15.116:0/2443870642 >> v2:172.21.15.174:3300/0 conn(0x7f643814fc20 0x7f6438152f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:27.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.138+0000 7f6435d9b640 1 -- 172.21.15.116:0/2443870642 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64380a9620 con 0x7f643806f6c0 2024-04-25T00:12:27.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.138+0000 7f6435d9b640 1 -- 172.21.15.116:0/2443870642 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f642c001680 con 0x7f643806f6c0 2024-04-25T00:12:27.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.138+0000 7f6436d9d640 1 --2- 172.21.15.116:0/2443870642 >> v2:172.21.15.174:3300/0 conn(0x7f643814fc20 0x7f6438152f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:12:27.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.138+0000 7f6435d9b640 1 -- 172.21.15.116:0/2443870642 >> v1:172.21.15.116:6789/0 conn(0x7f643806f6c0 legacy=0x7f64380aa100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:27.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.138+0000 7f6435d9b640 1 --2- 172.21.15.116:0/2443870642 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f643806fb20 0x7f64200039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:27.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.138+0000 7f6435d9b640 1 --2- 172.21.15.116:0/2443870642 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6420003f40 0x7f6420004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:27.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.138+0000 7f6435d9b640 1 --2- 172.21.15.116:0/2443870642 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6420004d50 0x7f64200051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:27.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.138+0000 7f6435d9b640 1 -- 172.21.15.116:0/2443870642 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f64380aad50 con 0x7f6420004d50 2024-04-25T00:12:27.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.138+0000 7f6435d9b640 1 -- 172.21.15.116:0/2443870642 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6438073260 con 0x7f6420003f40 2024-04-25T00:12:27.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.138+0000 7f6435d9b640 1 -- 172.21.15.116:0/2443870642 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f642000bab0 con 0x7f643806fb20 2024-04-25T00:12:27.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.138+0000 7f643759e640 1 --2- 172.21.15.116:0/2443870642 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6420004d50 0x7f64200051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:27.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.138+0000 7f643659c640 1 --2- 172.21.15.116:0/2443870642 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6420003f40 0x7f6420004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:27.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.138+0000 7f6436d9d640 1 --2- 172.21.15.116:0/2443870642 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f643806fb20 0x7f64200039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:27.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.138+0000 7f643759e640 1 -- 172.21.15.116:0/2443870642 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f643806fb20 msgr2=0x7f64200039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:12:27.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.138+0000 7f643759e640 1 --2- 172.21.15.116:0/2443870642 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f643806fb20 0x7f64200039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:27.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.138+0000 7f643759e640 1 -- 172.21.15.116:0/2443870642 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6420003f40 msgr2=0x7f6420004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:12:27.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.138+0000 7f643759e640 1 --2- 172.21.15.116:0/2443870642 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6420003f40 0x7f6420004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:27.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.138+0000 7f643759e640 1 -- 172.21.15.116:0/2443870642 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6420003530 con 0x7f6420004d50 2024-04-25T00:12:27.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.142+0000 7f643759e640 1 --2- 172.21.15.116:0/2443870642 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6420004d50 0x7f64200051d0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f6430009fa0 tx=0x7f643001cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46f90afbf49d703e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:27.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.142+0000 7f6435d9b640 1 -- 172.21.15.116:0/2443870642 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64300b2070 con 0x7f6420004d50 2024-04-25T00:12:27.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.142+0000 7f6435d9b640 1 -- 172.21.15.116:0/2443870642 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64300a5b10 con 0x7f6420004d50 2024-04-25T00:12:27.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.142+0000 7f6435d9b640 1 -- 172.21.15.116:0/2443870642 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6430009bf0 con 0x7f6420004d50 2024-04-25T00:12:27.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.142+0000 7f643d25b640 1 -- 172.21.15.116:0/2443870642 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6420004d50 msgr2=0x7f64200051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:27.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.142+0000 7f643d25b640 1 --2- 172.21.15.116:0/2443870642 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6420004d50 0x7f64200051d0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f6430009fa0 tx=0x7f643001cc60 comp rx=0 tx=0).stop 2024-04-25T00:12:27.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.142+0000 7f643759e640 1 -- 172.21.15.116:0/2443870642 reap_dead start 2024-04-25T00:12:27.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.142+0000 7f643d25b640 1 -- 172.21.15.116:0/2443870642 shutdown_connections 2024-04-25T00:12:27.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.142+0000 7f643d25b640 1 -- 172.21.15.116:0/2443870642 >> 172.21.15.116:0/2443870642 conn(0x7f64380c4c10 msgr2=0x7f64380c5020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:27.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.142+0000 7f643d25b640 1 -- 172.21.15.116:0/2443870642 shutdown_connections 2024-04-25T00:12:27.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.142+0000 7f643d25b640 1 -- 172.21.15.116:0/2443870642 wait complete. 2024-04-25T00:12:27.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.142+0000 7f643d25b640 1 Processor -- start 2024-04-25T00:12:27.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.142+0000 7f643d25b640 1 -- start start 2024-04-25T00:12:27.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.142+0000 7f643d25b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64381eebb0 0x7f643814e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:27.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.142+0000 7f643d25b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f643814ee20 0x7f6438151630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:27.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.142+0000 7f643d25b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64381ef520 0x7f6438151b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:27.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.142+0000 7f643d25b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6438073000 con 0x7f64381ef520 2024-04-25T00:12:27.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.142+0000 7f643d25b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6438072e80 con 0x7f64381eebb0 2024-04-25T00:12:27.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.142+0000 7f643d25b640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f643815d5d0 con 0x7f643814ee20 2024-04-25T00:12:27.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.142+0000 7f643759e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64381ef520 0x7f6438151b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:27.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.142+0000 7f643759e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64381ef520 0x7f6438151b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35532/0 (socket says 172.21.15.116:35532) 2024-04-25T00:12:27.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.142+0000 7f643759e640 1 -- 172.21.15.116:0/3306128079 learned_addr learned my addr 172.21.15.116:0/3306128079 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:12:27.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.142+0000 7f6436d9d640 1 --2- 172.21.15.116:0/3306128079 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64381eebb0 0x7f643814e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:27.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.142+0000 7f643659c640 1 --2- 172.21.15.116:0/3306128079 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f643814ee20 0x7f6438151630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:27.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.142+0000 7f643759e640 1 -- 172.21.15.116:0/3306128079 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f643814ee20 msgr2=0x7f6438151630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:27.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.142+0000 7f643759e640 1 --2- 172.21.15.116:0/3306128079 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f643814ee20 0x7f6438151630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:27.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.142+0000 7f643759e640 1 -- 172.21.15.116:0/3306128079 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64381eebb0 msgr2=0x7f643814e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:27.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.142+0000 7f643759e640 1 --2- 172.21.15.116:0/3306128079 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64381eebb0 0x7f643814e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:27.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.142+0000 7f643759e640 1 -- 172.21.15.116:0/3306128079 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6438142b50 con 0x7f64381ef520 2024-04-25T00:12:27.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.146+0000 7f643759e640 1 --2- 172.21.15.116:0/3306128079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64381ef520 0x7f6438151b70 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f64300b0040 tx=0x7f6430009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:27.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.146+0000 7f641f7fe640 1 -- 172.21.15.116:0/3306128079 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64300bc070 con 0x7f64381ef520 2024-04-25T00:12:27.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.146+0000 7f641f7fe640 1 -- 172.21.15.116:0/3306128079 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64300a5910 con 0x7f64381ef520 2024-04-25T00:12:27.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.146+0000 7f643d25b640 1 -- 172.21.15.116:0/3306128079 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64381502b0 con 0x7f64381ef520 2024-04-25T00:12:27.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.146+0000 7f641f7fe640 1 -- 172.21.15.116:0/3306128079 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64300b2040 con 0x7f64381ef520 2024-04-25T00:12:27.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.146+0000 7f643d25b640 1 -- 172.21.15.116:0/3306128079 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6438150870 con 0x7f64381ef520 2024-04-25T00:12:27.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.146+0000 7f641d7fa640 1 -- 172.21.15.116:0/3306128079 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64100052a0 con 0x7f64381ef520 2024-04-25T00:12:27.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.146+0000 7f641f7fe640 1 -- 172.21.15.116:0/3306128079 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f64300adba0 con 0x7f64381ef520 2024-04-25T00:12:27.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.146+0000 7f641f7fe640 1 --2- 172.21.15.116:0/3306128079 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f64240678c0 0x7f6424069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:27.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.146+0000 7f641f7fe640 1 -- 172.21.15.116:0/3306128079 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 7956+0+0 (secure 0 0 0) 0x7f6430124380 con 0x7f64381ef520 2024-04-25T00:12:27.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.146+0000 7f6436d9d640 1 --2- 172.21.15.116:0/3306128079 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f64240678c0 0x7f6424069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:27.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.146+0000 7f6436d9d640 1 --2- 172.21.15.116:0/3306128079 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f64240678c0 0x7f6424069d80 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f64280093f0 tx=0x7f6428008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:27.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.150+0000 7f641f7fe640 1 -- 172.21.15.116:0/3306128079 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f64300c6060 con 0x7f64381ef520 2024-04-25T00:12:27.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.386+0000 7f641d7fa640 1 -- 172.21.15.116:0/3306128079 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6410005860 con 0x7f64381ef520 2024-04-25T00:12:27.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.386+0000 7f641f7fe640 1 -- 172.21.15.116:0/3306128079 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) ==== 94+0+5675 (secure 0 0 0) 0x7f6430016970 con 0x7f64381ef520 2024-04-25T00:12:27.387 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:12:27.387 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:11:03.640578+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6114},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6114":{"gid":6114,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":25,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6114,"qdb_cluster":[6114]},"id":1},{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:12:23.643548+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5461},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5461":{"gid":5461,"name":"b","rank":0,"incarnation":83,"state":"up:active","state_seq":30,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5461,"qdb_cluster":[5461]},"id":2}]} 2024-04-25T00:12:27.387 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 86 2024-04-25T00:12:27.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.390+0000 7f641d7fa640 1 -- 172.21.15.116:0/3306128079 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f64240678c0 msgr2=0x7f6424069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:27.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.390+0000 7f641d7fa640 1 --2- 172.21.15.116:0/3306128079 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f64240678c0 0x7f6424069d80 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f64280093f0 tx=0x7f6428008040 comp rx=0 tx=0).stop 2024-04-25T00:12:27.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.390+0000 7f641d7fa640 1 -- 172.21.15.116:0/3306128079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64381ef520 msgr2=0x7f6438151b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:27.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.390+0000 7f641d7fa640 1 --2- 172.21.15.116:0/3306128079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64381ef520 0x7f6438151b70 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f64300b0040 tx=0x7f6430009750 comp rx=0 tx=0).stop 2024-04-25T00:12:27.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.390+0000 7f641d7fa640 1 -- 172.21.15.116:0/3306128079 shutdown_connections 2024-04-25T00:12:27.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.390+0000 7f641d7fa640 1 --2- 172.21.15.116:0/3306128079 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f64240678c0 0x7f6424069d80 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:27.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.390+0000 7f641d7fa640 1 --2- 172.21.15.116:0/3306128079 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64381ef520 0x7f6438151b70 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:27.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.390+0000 7f641d7fa640 1 --2- 172.21.15.116:0/3306128079 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f643814ee20 0x7f6438151630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:27.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.390+0000 7f641d7fa640 1 --2- 172.21.15.116:0/3306128079 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64381eebb0 0x7f643814e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:27.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.390+0000 7f641d7fa640 1 -- 172.21.15.116:0/3306128079 >> 172.21.15.116:0/3306128079 conn(0x7f64380c4c10 msgr2=0x7f64380671c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:27.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.390+0000 7f641d7fa640 1 -- 172.21.15.116:0/3306128079 shutdown_connections 2024-04-25T00:12:27.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.390+0000 7f641d7fa640 1 -- 172.21.15.116:0/3306128079 wait complete. 2024-04-25T00:12:27.406 INFO:tasks.mds_thrash.fs.[a]:kill mds.a (rank=0) 2024-04-25T00:12:27.407 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-25T00:12:27.407 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:12:27.423 INFO:tasks.ceph.mds.a:Stopped 2024-04-25T00:12:27.424 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:12:27.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.594+0000 7fe40f49c640 1 Processor -- start 2024-04-25T00:12:27.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.594+0000 7fe40f49c640 1 -- start start 2024-04-25T00:12:27.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.594+0000 7fe40f49c640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fe408158d30 0x7fe40806f270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:27.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.594+0000 7fe40f49c640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4081558e0 con 0x7fe408152ca0 2024-04-25T00:12:27.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.594+0000 7fe40f49c640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe408155bc0 con 0x7fe40806f7b0 2024-04-25T00:12:27.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.594+0000 7fe40f49c640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fe408155ea0 con 0x7fe408158d30 2024-04-25T00:12:27.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.594+0000 7fe40d238640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fe408152ca0 0x7fe40806b210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:52196/0 (socket says 172.21.15.116:52196) 2024-04-25T00:12:27.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.594+0000 7fe40d238640 1 -- 172.21.15.116:0/1125061995 learned_addr learned my addr 172.21.15.116:0/1125061995 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:12:27.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.594+0000 7fe40ca37640 1 --2- 172.21.15.116:0/1125061995 >> v2:172.21.15.174:3300/0 conn(0x7fe408158d30 0x7fe40806f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:27.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.594+0000 7fe3fffff640 1 -- 172.21.15.116:0/1125061995 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 995971163 0 0) 0x7fe4081558e0 con 0x7fe408152ca0 2024-04-25T00:12:27.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.594+0000 7fe3fffff640 1 -- 172.21.15.116:0/1125061995 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe3f4003530 con 0x7fe408152ca0 2024-04-25T00:12:27.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.594+0000 7fe3fffff640 1 -- 172.21.15.116:0/1125061995 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1339891308 0 0) 0x7fe3f4003530 con 0x7fe408152ca0 2024-04-25T00:12:27.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.594+0000 7fe3fffff640 1 -- 172.21.15.116:0/1125061995 >> v2:172.21.15.174:3300/0 conn(0x7fe408158d30 msgr2=0x7fe40806f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:12:27.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.594+0000 7fe3fffff640 1 --2- 172.21.15.116:0/1125061995 >> v2:172.21.15.174:3300/0 conn(0x7fe408158d30 0x7fe40806f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:27.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.594+0000 7fe3fffff640 1 -- 172.21.15.116:0/1125061995 >> v1:172.21.15.174:6789/0 conn(0x7fe40806f7b0 legacy=0x7fe4081600c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:27.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.594+0000 7fe3fffff640 1 -- 172.21.15.116:0/1125061995 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe408156180 con 0x7fe408152ca0 2024-04-25T00:12:27.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.594+0000 7fe3fffff640 1 -- 172.21.15.116:0/1125061995 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7fe404001680 con 0x7fe408152ca0 2024-04-25T00:12:27.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.594+0000 7fe3fffff640 1 -- 172.21.15.116:0/1125061995 >> v1:172.21.15.116:6789/0 conn(0x7fe408152ca0 legacy=0x7fe40806b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:27.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.594+0000 7fe3fffff640 1 --2- 172.21.15.116:0/1125061995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3f40039c0 0x7fe3f4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:27.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.594+0000 7fe3fffff640 1 --2- 172.21.15.116:0/1125061995 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe3f4004810 0x7fe3f4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:27.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.594+0000 7fe3fffff640 1 --2- 172.21.15.116:0/1125061995 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe3f40051d0 0x7fe3f400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:27.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.594+0000 7fe3fffff640 1 -- 172.21.15.116:0/1125061995 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe408155ea0 con 0x7fe3f40039c0 2024-04-25T00:12:27.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.594+0000 7fe3fffff640 1 -- 172.21.15.116:0/1125061995 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f400c170 con 0x7fe3f40051d0 2024-04-25T00:12:27.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.594+0000 7fe3fffff640 1 -- 172.21.15.116:0/1125061995 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe3f400c370 con 0x7fe3f4004810 2024-04-25T00:12:27.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.594+0000 7fe40d238640 1 --2- 172.21.15.116:0/1125061995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3f40039c0 0x7fe3f4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:27.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.594+0000 7fe40da39640 1 --2- 172.21.15.116:0/1125061995 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe3f40051d0 0x7fe3f400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:27.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.594+0000 7fe40ca37640 1 --2- 172.21.15.116:0/1125061995 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe3f4004810 0x7fe3f4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:27.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.594+0000 7fe40d238640 1 -- 172.21.15.116:0/1125061995 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe3f4004810 msgr2=0x7fe3f4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:12:27.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.594+0000 7fe40d238640 1 --2- 172.21.15.116:0/1125061995 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe3f4004810 0x7fe3f4004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:27.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.594+0000 7fe40d238640 1 -- 172.21.15.116:0/1125061995 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe3f40051d0 msgr2=0x7fe3f400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:12:27.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.594+0000 7fe40d238640 1 --2- 172.21.15.116:0/1125061995 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe3f40051d0 0x7fe3f400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:27.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.594+0000 7fe40d238640 1 -- 172.21.15.116:0/1125061995 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3f4003530 con 0x7fe3f40039c0 2024-04-25T00:12:27.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.594+0000 7fe40da39640 1 -- 172.21.15.116:0/1125061995 reap_dead start 2024-04-25T00:12:27.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.598+0000 7fe40d238640 1 --2- 172.21.15.116:0/1125061995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3f40039c0 0x7fe3f4003de0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fe40400c1c0 tx=0x7fe40401cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12889d90441ed1ef server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:27.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.598+0000 7fe3fffff640 1 -- 172.21.15.116:0/1125061995 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe404007d00 con 0x7fe3f40039c0 2024-04-25T00:12:27.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.598+0000 7fe3fffff640 1 -- 172.21.15.116:0/1125061995 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4040a9020 con 0x7fe3f40039c0 2024-04-25T00:12:27.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.598+0000 7fe3fffff640 1 -- 172.21.15.116:0/1125061995 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe404005b00 con 0x7fe3f40039c0 2024-04-25T00:12:27.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.598+0000 7fe40f49c640 1 -- 172.21.15.116:0/1125061995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3f40039c0 msgr2=0x7fe3f4003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:27.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.598+0000 7fe40f49c640 1 --2- 172.21.15.116:0/1125061995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3f40039c0 0x7fe3f4003de0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fe40400c1c0 tx=0x7fe40401cc60 comp rx=0 tx=0).stop 2024-04-25T00:12:27.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.598+0000 7fe40f49c640 1 -- 172.21.15.116:0/1125061995 shutdown_connections 2024-04-25T00:12:27.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.598+0000 7fe40f49c640 1 --2- 172.21.15.116:0/1125061995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe3f40039c0 0x7fe3f4003de0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:27.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.598+0000 7fe40f49c640 1 -- 172.21.15.116:0/1125061995 >> 172.21.15.116:0/1125061995 conn(0x7fe4080c7fa0 msgr2=0x7fe4080c83b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:27.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.598+0000 7fe40f49c640 1 -- 172.21.15.116:0/1125061995 shutdown_connections 2024-04-25T00:12:27.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.598+0000 7fe40f49c640 1 -- 172.21.15.116:0/1125061995 wait complete. 2024-04-25T00:12:27.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.602+0000 7fe40f49c640 1 Processor -- start 2024-04-25T00:12:27.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.602+0000 7fe40f49c640 1 -- start start 2024-04-25T00:12:27.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.602+0000 7fe40f49c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe3f40039c0 0x7fe408157020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:27.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.602+0000 7fe40f49c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe4081eed90 0x7fe40814b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:27.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.602+0000 7fe40f49c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4080c6ea0 0x7fe40814b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:27.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.602+0000 7fe40f49c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe40806ba20 con 0x7fe4080c6ea0 2024-04-25T00:12:27.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.602+0000 7fe40f49c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe40806ffa0 con 0x7fe4081eed90 2024-04-25T00:12:27.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.602+0000 7fe40f49c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe4080c13d0 con 0x7fe3f40039c0 2024-04-25T00:12:27.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.602+0000 7fe40da39640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4080c6ea0 0x7fe40814b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:27.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.602+0000 7fe40da39640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4080c6ea0 0x7fe40814b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35548/0 (socket says 172.21.15.116:35548) 2024-04-25T00:12:27.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.602+0000 7fe40da39640 1 -- 172.21.15.116:0/1476888780 learned_addr learned my addr 172.21.15.116:0/1476888780 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:12:27.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.602+0000 7fe40d238640 1 --2- 172.21.15.116:0/1476888780 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe3f40039c0 0x7fe408157020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:27.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.602+0000 7fe40ca37640 1 --2- 172.21.15.116:0/1476888780 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe4081eed90 0x7fe40814b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:27.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.602+0000 7fe40da39640 1 -- 172.21.15.116:0/1476888780 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe3f40039c0 msgr2=0x7fe408157020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:27.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.602+0000 7fe40da39640 1 --2- 172.21.15.116:0/1476888780 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe3f40039c0 0x7fe408157020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:27.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.602+0000 7fe40da39640 1 -- 172.21.15.116:0/1476888780 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe4081eed90 msgr2=0x7fe40814b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:27.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.602+0000 7fe40da39640 1 --2- 172.21.15.116:0/1476888780 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe4081eed90 0x7fe40814b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:27.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.602+0000 7fe40da39640 1 -- 172.21.15.116:0/1476888780 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4081ef290 con 0x7fe4080c6ea0 2024-04-25T00:12:27.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.602+0000 7fe40da39640 1 --2- 172.21.15.116:0/1476888780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4080c6ea0 0x7fe40814b9a0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fe408159110 tx=0x7fe40000e9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:27.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.602+0000 7fe3fdffb640 1 -- 172.21.15.116:0/1476888780 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe400019070 con 0x7fe4080c6ea0 2024-04-25T00:12:27.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.602+0000 7fe3fdffb640 1 -- 172.21.15.116:0/1476888780 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe400006e30 con 0x7fe4080c6ea0 2024-04-25T00:12:27.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.602+0000 7fe3fdffb640 1 -- 172.21.15.116:0/1476888780 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe400003ba0 con 0x7fe4080c6ea0 2024-04-25T00:12:27.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.602+0000 7fe40f49c640 1 -- 172.21.15.116:0/1476888780 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4081ef570 con 0x7fe4080c6ea0 2024-04-25T00:12:27.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.602+0000 7fe40f49c640 1 -- 172.21.15.116:0/1476888780 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4081efb30 con 0x7fe4080c6ea0 2024-04-25T00:12:27.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.602+0000 7fe40f49c640 1 -- 172.21.15.116:0/1476888780 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe408068a60 con 0x7fe4080c6ea0 2024-04-25T00:12:27.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.606+0000 7fe3fdffb640 1 -- 172.21.15.116:0/1476888780 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe40001c070 con 0x7fe4080c6ea0 2024-04-25T00:12:27.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.606+0000 7fe3fdffb640 1 --2- 172.21.15.116:0/1476888780 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe3ec0678f0 0x7fe3ec069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:27.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.606+0000 7fe3fdffb640 1 -- 172.21.15.116:0/1476888780 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 7956+0+0 (secure 0 0 0) 0x7fe400085a60 con 0x7fe4080c6ea0 2024-04-25T00:12:27.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.606+0000 7fe40d238640 1 --2- 172.21.15.116:0/1476888780 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe3ec0678f0 0x7fe3ec069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:27.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.606+0000 7fe40d238640 1 --2- 172.21.15.116:0/1476888780 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe3ec0678f0 0x7fe3ec069db0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fe4040aa000 tx=0x7fe408155ab0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:27.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.610+0000 7fe3fdffb640 1 -- 172.21.15.116:0/1476888780 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fe400055710 con 0x7fe4080c6ea0 2024-04-25T00:12:27.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.814+0000 7fe40f49c640 1 -- 172.21.15.116:0/1476888780 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe408072de0 con 0x7fe4080c6ea0 2024-04-25T00:12:27.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.814+0000 7fe3fdffb640 1 -- 172.21.15.116:0/1476888780 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) ==== 94+0+5675 (secure 0 0 0) 0x7fe400059a10 con 0x7fe4080c6ea0 2024-04-25T00:12:27.815 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:12:27.815 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:11:03.640578+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6114},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6114":{"gid":6114,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":25,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6114,"qdb_cluster":[6114]},"id":1},{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:12:23.643548+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5461},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5461":{"gid":5461,"name":"b","rank":0,"incarnation":83,"state":"up:active","state_seq":30,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5461,"qdb_cluster":[5461]},"id":2}]} 2024-04-25T00:12:27.816 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 86 2024-04-25T00:12:27.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.818+0000 7fe40f49c640 1 -- 172.21.15.116:0/1476888780 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe3ec0678f0 msgr2=0x7fe3ec069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:27.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.818+0000 7fe40f49c640 1 --2- 172.21.15.116:0/1476888780 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe3ec0678f0 0x7fe3ec069db0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fe4040aa000 tx=0x7fe408155ab0 comp rx=0 tx=0).stop 2024-04-25T00:12:27.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.818+0000 7fe40f49c640 1 -- 172.21.15.116:0/1476888780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4080c6ea0 msgr2=0x7fe40814b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:27.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.818+0000 7fe40f49c640 1 --2- 172.21.15.116:0/1476888780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4080c6ea0 0x7fe40814b9a0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fe408159110 tx=0x7fe40000e9e0 comp rx=0 tx=0).stop 2024-04-25T00:12:27.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.818+0000 7fe40f49c640 1 -- 172.21.15.116:0/1476888780 shutdown_connections 2024-04-25T00:12:27.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.818+0000 7fe40f49c640 1 --2- 172.21.15.116:0/1476888780 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe3ec0678f0 0x7fe3ec069db0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:27.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.818+0000 7fe40f49c640 1 --2- 172.21.15.116:0/1476888780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe4080c6ea0 0x7fe40814b9a0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:27.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.818+0000 7fe40f49c640 1 --2- 172.21.15.116:0/1476888780 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe4081eed90 0x7fe40814b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:27.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.818+0000 7fe40f49c640 1 --2- 172.21.15.116:0/1476888780 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe3f40039c0 0x7fe408157020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:27.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.818+0000 7fe40f49c640 1 -- 172.21.15.116:0/1476888780 >> 172.21.15.116:0/1476888780 conn(0x7fe4080c7fa0 msgr2=0x7fe408148a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:27.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.818+0000 7fe40f49c640 1 -- 172.21.15.116:0/1476888780 shutdown_connections 2024-04-25T00:12:27.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:27.818+0000 7fe40f49c640 1 -- 172.21.15.116:0/1476888780 wait complete. 2024-04-25T00:12:27.833 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:12:29.835 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:12:29.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.962+0000 7f5a752c9640 1 Processor -- start 2024-04-25T00:12:29.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.962+0000 7f5a752c9640 1 -- start start 2024-04-25T00:12:29.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.966+0000 7f5a752c9640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f5a7013ed60 0x7f5a701559c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:29.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.966+0000 7f5a752c9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5a7013e810 0x7f5a70155f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:29.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.966+0000 7f5a752c9640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a7006d200 con 0x7f5a7013e310 2024-04-25T00:12:29.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.966+0000 7f5a752c9640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f5a70070d40 con 0x7f5a7013e810 2024-04-25T00:12:29.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.966+0000 7f5a752c9640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f5a70071020 con 0x7f5a7013ed60 2024-04-25T00:12:29.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.966+0000 7f5a6f7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5a7013e810 0x7f5a70155f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:29.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.966+0000 7f5a6effd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f5a7013e310 0x7f5a700c2a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:52198/0 (socket says 172.21.15.116:52198) 2024-04-25T00:12:29.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.966+0000 7f5a6effd640 1 -- 172.21.15.116:0/1962958582 learned_addr learned my addr 172.21.15.116:0/1962958582 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:12:29.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.966+0000 7f5a6e7fc640 1 --2- 172.21.15.116:0/1962958582 >> v2:172.21.15.174:3300/0 conn(0x7f5a7013ed60 0x7f5a701559c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:29.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.966+0000 7f5a6f7fe640 1 -- 172.21.15.116:0/1962958582 >> v2:172.21.15.174:3300/0 conn(0x7f5a7013ed60 msgr2=0x7f5a701559c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:12:29.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.966+0000 7f5a6f7fe640 1 --2- 172.21.15.116:0/1962958582 >> v2:172.21.15.174:3300/0 conn(0x7f5a7013ed60 0x7f5a701559c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:29.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.966+0000 7f5a6f7fe640 1 -- 172.21.15.116:0/1962958582 >> v1:172.21.15.116:6789/0 conn(0x7f5a7013e310 legacy=0x7f5a700c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:29.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.966+0000 7f5a6dffb640 1 -- 172.21.15.116:0/1962958582 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1115523546 0 0) 0x7f5a7006d200 con 0x7f5a7013e310 2024-04-25T00:12:29.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.966+0000 7f5a6f7fe640 1 -- 172.21.15.116:0/1962958582 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a7006cb50 con 0x7f5a7013e810 2024-04-25T00:12:29.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.966+0000 7f5a6f7fe640 1 --2- 172.21.15.116:0/1962958582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a7013e810 0x7f5a70155f00 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f5a70073980 tx=0x7f5a6801c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bebab9b02f01cf5a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:29.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.966+0000 7f5a6dffb640 1 -- 172.21.15.116:0/1962958582 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a680b2070 con 0x7f5a7013e810 2024-04-25T00:12:29.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.966+0000 7f5a6dffb640 1 -- 172.21.15.116:0/1962958582 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a680a57e0 con 0x7f5a7013e810 2024-04-25T00:12:29.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.966+0000 7f5a6dffb640 1 -- 172.21.15.116:0/1962958582 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a680a5bb0 con 0x7f5a7013e810 2024-04-25T00:12:29.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.966+0000 7f5a752c9640 1 -- 172.21.15.116:0/1962958582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a7013e810 msgr2=0x7f5a70155f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:29.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.966+0000 7f5a752c9640 1 --2- 172.21.15.116:0/1962958582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a7013e810 0x7f5a70155f00 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f5a70073980 tx=0x7f5a6801c810 comp rx=0 tx=0).stop 2024-04-25T00:12:29.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.966+0000 7f5a752c9640 1 -- 172.21.15.116:0/1962958582 shutdown_connections 2024-04-25T00:12:29.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.966+0000 7f5a752c9640 1 --2- 172.21.15.116:0/1962958582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a7013e810 0x7f5a70155f00 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:29.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.966+0000 7f5a752c9640 1 --2- 172.21.15.116:0/1962958582 >> v2:172.21.15.174:3300/0 conn(0x7f5a7013ed60 0x7f5a701559c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:29.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.966+0000 7f5a752c9640 1 -- 172.21.15.116:0/1962958582 >> 172.21.15.116:0/1962958582 conn(0x7f5a700c3b00 msgr2=0x7f5a700c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:29.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.970+0000 7f5a752c9640 1 -- 172.21.15.116:0/1962958582 shutdown_connections 2024-04-25T00:12:29.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.970+0000 7f5a752c9640 1 -- 172.21.15.116:0/1962958582 wait complete. 2024-04-25T00:12:29.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.970+0000 7f5a752c9640 1 Processor -- start 2024-04-25T00:12:29.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.970+0000 7f5a752c9640 1 -- start start 2024-04-25T00:12:29.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.970+0000 7f5a752c9640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5a7013e310 0x7f5a7013a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:29.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.970+0000 7f5a752c9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a7013e810 0x7f5a70132af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:29.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.970+0000 7f5a752c9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5a7013ed60 0x7f5a701f1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:29.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.970+0000 7f5a752c9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5a701565c0 con 0x7f5a7013e810 2024-04-25T00:12:29.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.970+0000 7f5a752c9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5a70156440 con 0x7f5a7013ed60 2024-04-25T00:12:29.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.970+0000 7f5a752c9640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f5a701629f0 con 0x7f5a7013e310 2024-04-25T00:12:29.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.970+0000 7f5a6e7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a7013e810 0x7f5a70132af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:29.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.970+0000 7f5a6e7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a7013e810 0x7f5a70132af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35572/0 (socket says 172.21.15.116:35572) 2024-04-25T00:12:29.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.970+0000 7f5a6e7fc640 1 -- 172.21.15.116:0/1161358275 learned_addr learned my addr 172.21.15.116:0/1161358275 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:12:29.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.970+0000 7f5a6effd640 1 --2- 172.21.15.116:0/1161358275 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5a7013e310 0x7f5a7013a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:29.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.970+0000 7f5a6f7fe640 1 --2- 172.21.15.116:0/1161358275 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5a7013ed60 0x7f5a701f1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:29.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.970+0000 7f5a6e7fc640 1 -- 172.21.15.116:0/1161358275 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5a7013e310 msgr2=0x7f5a7013a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:29.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.970+0000 7f5a6e7fc640 1 --2- 172.21.15.116:0/1161358275 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5a7013e310 0x7f5a7013a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:29.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.970+0000 7f5a6e7fc640 1 -- 172.21.15.116:0/1161358275 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5a7013ed60 msgr2=0x7f5a701f1ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:29.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.970+0000 7f5a6e7fc640 1 --2- 172.21.15.116:0/1161358275 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5a7013ed60 0x7f5a701f1ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:29.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.970+0000 7f5a6e7fc640 1 -- 172.21.15.116:0/1161358275 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a700731a0 con 0x7f5a7013e810 2024-04-25T00:12:29.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.970+0000 7f5a6e7fc640 1 --2- 172.21.15.116:0/1161358275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a7013e810 0x7f5a70132af0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f5a6400d800 tx=0x7f5a6400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:29.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.970+0000 7f5a577fe640 1 -- 172.21.15.116:0/1161358275 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a6400b6c0 con 0x7f5a7013e810 2024-04-25T00:12:29.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.970+0000 7f5a577fe640 1 -- 172.21.15.116:0/1161358275 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a64019070 con 0x7f5a7013e810 2024-04-25T00:12:29.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.970+0000 7f5a752c9640 1 -- 172.21.15.116:0/1161358275 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a70155720 con 0x7f5a7013e810 2024-04-25T00:12:29.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.970+0000 7f5a577fe640 1 -- 172.21.15.116:0/1161358275 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a6400ba60 con 0x7f5a7013e810 2024-04-25T00:12:29.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.970+0000 7f5a752c9640 1 -- 172.21.15.116:0/1161358275 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a700c4eb0 con 0x7f5a7013e810 2024-04-25T00:12:29.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.970+0000 7f5a752c9640 1 -- 172.21.15.116:0/1161358275 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a70068350 con 0x7f5a7013e810 2024-04-25T00:12:29.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.974+0000 7f5a577fe640 1 -- 172.21.15.116:0/1161358275 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5a6400bc00 con 0x7f5a7013e810 2024-04-25T00:12:29.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.974+0000 7f5a577fe640 1 --2- 172.21.15.116:0/1161358275 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5a5c0678c0 0x7f5a5c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:29.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.974+0000 7f5a577fe640 1 -- 172.21.15.116:0/1161358275 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 7956+0+0 (secure 0 0 0) 0x7f5a64086a20 con 0x7f5a7013e810 2024-04-25T00:12:29.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.974+0000 7f5a6effd640 1 --2- 172.21.15.116:0/1161358275 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5a5c0678c0 0x7f5a5c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:29.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.974+0000 7f5a6effd640 1 --2- 172.21.15.116:0/1161358275 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5a5c0678c0 0x7f5a5c069d80 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f5a60000f70 tx=0x7f5a60007df0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:29.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:29.978+0000 7f5a577fe640 1 -- 172.21.15.116:0/1161358275 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f5a64056c00 con 0x7f5a7013e810 2024-04-25T00:12:30.012 INFO:tasks.mds_thrash.fs.[b]:reviving mds.c 2024-04-25T00:12:30.013 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-25T00:12:30.013 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-25T00:12:30.016 INFO:tasks.ceph.mds.c:Started 2024-04-25T00:12:30.016 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:12:30.077 INFO:tasks.ceph.mds.c.smithi116.stdout:starting mds.c at 2024-04-25T00:12:30.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.134+0000 7f10d0c79640 1 Processor -- start 2024-04-25T00:12:30.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.134+0000 7f10d0c79640 1 -- start start 2024-04-25T00:12:30.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.134+0000 7f10d0c79640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f10cc152e60 0x7f10cc1599a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:30.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.134+0000 7f10d0c79640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10cc073200 con 0x7f10cc1562d0 2024-04-25T00:12:30.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.134+0000 7f10d0c79640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10cc158ec0 con 0x7f10cc0c5010 2024-04-25T00:12:30.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.134+0000 7f10d0c79640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f10cc156760 con 0x7f10cc152e60 2024-04-25T00:12:30.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.134+0000 7f10ca59c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f10cc152e60 0x7f10cc1599a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:30.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.134+0000 7f10c1d9b640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f10cc1562d0 0x7f10cc0c36e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:52212/0 (socket says 172.21.15.116:52212) 2024-04-25T00:12:30.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.134+0000 7f10c1d9b640 1 -- 172.21.15.116:0/2780758464 learned_addr learned my addr 172.21.15.116:0/2780758464 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:12:30.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.134+0000 7f10c9d9b640 1 -- 172.21.15.116:0/2780758464 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 281520345 0 0) 0x7f10cc073200 con 0x7f10cc1562d0 2024-04-25T00:12:30.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.134+0000 7f10c9d9b640 1 -- 172.21.15.116:0/2780758464 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f10ac003530 con 0x7f10cc1562d0 2024-04-25T00:12:30.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.134+0000 7f10ca59c640 1 -- 172.21.15.116:0/2780758464 >> v1:172.21.15.174:6789/0 conn(0x7f10cc0c5010 legacy=0x7f10cc160120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:12:30.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.134+0000 7f10ca59c640 1 -- 172.21.15.116:0/2780758464 >> v1:172.21.15.116:6789/0 conn(0x7f10cc1562d0 legacy=0x7f10cc0c36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:30.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.134+0000 7f10ca59c640 1 -- 172.21.15.116:0/2780758464 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10cc1591a0 con 0x7f10cc152e60 2024-04-25T00:12:30.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.138+0000 7f10ca59c640 1 --2- 172.21.15.116:0/2780758464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10cc152e60 0x7f10cc1599a0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f10bc0097c0 tx=0x7f10bc01c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e166ad4e3260bbe3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:30.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.138+0000 7f10c9d9b640 1 -- 172.21.15.116:0/2780758464 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10bc0b0070 con 0x7f10cc152e60 2024-04-25T00:12:30.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.138+0000 7f10c9d9b640 1 -- 172.21.15.116:0/2780758464 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10bc0ab600 con 0x7f10cc152e60 2024-04-25T00:12:30.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.138+0000 7f10c9d9b640 1 -- 172.21.15.116:0/2780758464 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10bc0a4e60 con 0x7f10cc152e60 2024-04-25T00:12:30.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.138+0000 7f10d0c79640 1 -- 172.21.15.116:0/2780758464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10cc152e60 msgr2=0x7f10cc1599a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:30.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.138+0000 7f10d0c79640 1 --2- 172.21.15.116:0/2780758464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10cc152e60 0x7f10cc1599a0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f10bc0097c0 tx=0x7f10bc01c3d0 comp rx=0 tx=0).stop 2024-04-25T00:12:30.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.138+0000 7f10d0c79640 1 -- 172.21.15.116:0/2780758464 shutdown_connections 2024-04-25T00:12:30.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.138+0000 7f10d0c79640 1 --2- 172.21.15.116:0/2780758464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10cc152e60 0x7f10cc1599a0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:30.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.138+0000 7f10d0c79640 1 -- 172.21.15.116:0/2780758464 >> 172.21.15.116:0/2780758464 conn(0x7f10cc06c720 msgr2=0x7f10cc06cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:30.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.138+0000 7f10d0c79640 1 -- 172.21.15.116:0/2780758464 shutdown_connections 2024-04-25T00:12:30.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.138+0000 7f10d0c79640 1 -- 172.21.15.116:0/2780758464 wait complete. 2024-04-25T00:12:30.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.142+0000 7f10d0c79640 1 Processor -- start 2024-04-25T00:12:30.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.142+0000 7f10d0c79640 1 -- start start 2024-04-25T00:12:30.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.142+0000 7f10d0c79640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f10cc0c5010 0x7f10cc145db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:30.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.142+0000 7f10d0c79640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f10cc152e60 0x7f10cc13cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:30.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.142+0000 7f10d0c79640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10cc1562d0 0x7f10cc13d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:30.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.142+0000 7f10d0c79640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f10cc0c5870 con 0x7f10cc1562d0 2024-04-25T00:12:30.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.142+0000 7f10d0c79640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f10cc0734e0 con 0x7f10cc152e60 2024-04-25T00:12:30.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.142+0000 7f10d0c79640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f10cc072e20 con 0x7f10cc0c5010 2024-04-25T00:12:30.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.142+0000 7f10cad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10cc1562d0 0x7f10cc13d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:30.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.142+0000 7f10cad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10cc1562d0 0x7f10cc13d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35616/0 (socket says 172.21.15.116:35616) 2024-04-25T00:12:30.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.142+0000 7f10cad9d640 1 -- 172.21.15.116:0/1296336605 learned_addr learned my addr 172.21.15.116:0/1296336605 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:12:30.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.142+0000 7f10c1d9b640 1 --2- 172.21.15.116:0/1296336605 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f10cc152e60 0x7f10cc13cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:30.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.142+0000 7f10ca59c640 1 --2- 172.21.15.116:0/1296336605 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f10cc0c5010 0x7f10cc145db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:30.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.142+0000 7f10cad9d640 1 -- 172.21.15.116:0/1296336605 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f10cc0c5010 msgr2=0x7f10cc145db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:30.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.142+0000 7f10cad9d640 1 --2- 172.21.15.116:0/1296336605 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f10cc0c5010 0x7f10cc145db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:30.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.142+0000 7f10cad9d640 1 -- 172.21.15.116:0/1296336605 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f10cc152e60 msgr2=0x7f10cc13cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:30.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.142+0000 7f10cad9d640 1 --2- 172.21.15.116:0/1296336605 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f10cc152e60 0x7f10cc13cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:30.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.142+0000 7f10cad9d640 1 -- 172.21.15.116:0/1296336605 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10cc1553b0 con 0x7f10cc1562d0 2024-04-25T00:12:30.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.142+0000 7f10cad9d640 1 --2- 172.21.15.116:0/1296336605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10cc1562d0 0x7f10cc13d220 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f10c400b9a0 tx=0x7f10c400be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:30.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.142+0000 7f10c37fe640 1 -- 172.21.15.116:0/1296336605 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10c4019070 con 0x7f10cc1562d0 2024-04-25T00:12:30.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.142+0000 7f10c37fe640 1 -- 172.21.15.116:0/1296336605 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10c400cba0 con 0x7f10cc1562d0 2024-04-25T00:12:30.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.142+0000 7f10d0c79640 1 -- 172.21.15.116:0/1296336605 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10cc153980 con 0x7f10cc1562d0 2024-04-25T00:12:30.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.142+0000 7f10c37fe640 1 -- 172.21.15.116:0/1296336605 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10c4014440 con 0x7f10cc1562d0 2024-04-25T00:12:30.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.142+0000 7f10d0c79640 1 -- 172.21.15.116:0/1296336605 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10cc153f40 con 0x7f10cc1562d0 2024-04-25T00:12:30.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.142+0000 7f10d0c79640 1 -- 172.21.15.116:0/1296336605 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10cc068a60 con 0x7f10cc1562d0 2024-04-25T00:12:30.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.146+0000 7f10c37fe640 1 -- 172.21.15.116:0/1296336605 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f10c401c070 con 0x7f10cc1562d0 2024-04-25T00:12:30.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.146+0000 7f10c37fe640 1 --2- 172.21.15.116:0/1296336605 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f10b00678f0 0x7f10b0069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:30.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.146+0000 7f10c37fe640 1 -- 172.21.15.116:0/1296336605 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 7956+0+0 (secure 0 0 0) 0x7f10c4086320 con 0x7f10cc1562d0 2024-04-25T00:12:30.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.146+0000 7f10ca59c640 1 --2- 172.21.15.116:0/1296336605 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f10b00678f0 0x7f10b0069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:30.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.146+0000 7f10ca59c640 1 --2- 172.21.15.116:0/1296336605 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f10b00678f0 0x7f10b0069db0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f10bc0ad000 tx=0x7f10bc0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:30.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.150+0000 7f10c37fe640 1 -- 172.21.15.116:0/1296336605 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f10c4023080 con 0x7f10cc1562d0 2024-04-25T00:12:30.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.194+0000 7f5a752c9640 1 -- 172.21.15.116:0/1161358275 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5a700c34f0 con 0x7f5a7013e810 2024-04-25T00:12:30.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.194+0000 7f5a577fe640 1 -- 172.21.15.116:0/1161358275 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) ==== 94+0+5675 (secure 0 0 0) 0x7f5a64056500 con 0x7f5a7013e810 2024-04-25T00:12:30.194 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:12:30.194 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:11:03.640578+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6114},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6114":{"gid":6114,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":25,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6114,"qdb_cluster":[6114]},"id":1},{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:12:23.643548+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5461},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5461":{"gid":5461,"name":"b","rank":0,"incarnation":83,"state":"up:active","state_seq":30,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5461,"qdb_cluster":[5461]},"id":2}]} 2024-04-25T00:12:30.195 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 86 2024-04-25T00:12:30.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.198+0000 7f5a752c9640 1 -- 172.21.15.116:0/1161358275 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5a5c0678c0 msgr2=0x7f5a5c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:30.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.198+0000 7f5a752c9640 1 --2- 172.21.15.116:0/1161358275 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5a5c0678c0 0x7f5a5c069d80 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f5a60000f70 tx=0x7f5a60007df0 comp rx=0 tx=0).stop 2024-04-25T00:12:30.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.198+0000 7f5a752c9640 1 -- 172.21.15.116:0/1161358275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a7013e810 msgr2=0x7f5a70132af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:30.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.198+0000 7f5a752c9640 1 --2- 172.21.15.116:0/1161358275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a7013e810 0x7f5a70132af0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f5a6400d800 tx=0x7f5a6400dcc0 comp rx=0 tx=0).stop 2024-04-25T00:12:30.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.198+0000 7f5a752c9640 1 -- 172.21.15.116:0/1161358275 shutdown_connections 2024-04-25T00:12:30.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.198+0000 7f5a752c9640 1 --2- 172.21.15.116:0/1161358275 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5a5c0678c0 0x7f5a5c069d80 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:30.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.198+0000 7f5a752c9640 1 --2- 172.21.15.116:0/1161358275 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5a7013ed60 0x7f5a701f1ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:30.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.198+0000 7f5a752c9640 1 --2- 172.21.15.116:0/1161358275 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a7013e810 0x7f5a70132af0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:30.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.198+0000 7f5a752c9640 1 --2- 172.21.15.116:0/1161358275 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5a7013e310 0x7f5a7013a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:30.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.198+0000 7f5a752c9640 1 -- 172.21.15.116:0/1161358275 >> 172.21.15.116:0/1161358275 conn(0x7f5a700c3b00 msgr2=0x7f5a7006e3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:30.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.198+0000 7f5a752c9640 1 -- 172.21.15.116:0/1161358275 shutdown_connections 2024-04-25T00:12:30.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.198+0000 7f5a752c9640 1 -- 172.21.15.116:0/1161358275 wait complete. 2024-04-25T00:12:30.213 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:12:30.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.358+0000 7f10d0c79640 1 -- 172.21.15.116:0/1296336605 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f10cc155630 con 0x7f10cc1562d0 2024-04-25T00:12:30.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.362+0000 7f10c37fe640 1 -- 172.21.15.116:0/1296336605 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) ==== 94+0+5675 (secure 0 0 0) 0x7f10c4056700 con 0x7f10cc1562d0 2024-04-25T00:12:30.361 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:12:30.361 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:11:03.640578+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6114},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6114":{"gid":6114,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":25,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6114,"qdb_cluster":[6114]},"id":1},{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:12:23.643548+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5461},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5461":{"gid":5461,"name":"b","rank":0,"incarnation":83,"state":"up:active","state_seq":30,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5461,"qdb_cluster":[5461]},"id":2}]} 2024-04-25T00:12:30.361 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 86 2024-04-25T00:12:30.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.362+0000 7f10d0c79640 1 -- 172.21.15.116:0/1296336605 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f10b00678f0 msgr2=0x7f10b0069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:30.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.362+0000 7f10d0c79640 1 --2- 172.21.15.116:0/1296336605 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f10b00678f0 0x7f10b0069db0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f10bc0ad000 tx=0x7f10bc0a6000 comp rx=0 tx=0).stop 2024-04-25T00:12:30.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.362+0000 7f10d0c79640 1 -- 172.21.15.116:0/1296336605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10cc1562d0 msgr2=0x7f10cc13d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:30.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.362+0000 7f10d0c79640 1 --2- 172.21.15.116:0/1296336605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10cc1562d0 0x7f10cc13d220 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f10c400b9a0 tx=0x7f10c400be60 comp rx=0 tx=0).stop 2024-04-25T00:12:30.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.362+0000 7f10d0c79640 1 -- 172.21.15.116:0/1296336605 shutdown_connections 2024-04-25T00:12:30.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.362+0000 7f10d0c79640 1 --2- 172.21.15.116:0/1296336605 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f10b00678f0 0x7f10b0069db0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:30.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.362+0000 7f10d0c79640 1 --2- 172.21.15.116:0/1296336605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10cc1562d0 0x7f10cc13d220 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:30.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.362+0000 7f10d0c79640 1 --2- 172.21.15.116:0/1296336605 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f10cc152e60 0x7f10cc13cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:30.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.362+0000 7f10d0c79640 1 --2- 172.21.15.116:0/1296336605 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f10cc0c5010 0x7f10cc145db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:30.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.362+0000 7f10d0c79640 1 -- 172.21.15.116:0/1296336605 >> 172.21.15.116:0/1296336605 conn(0x7f10cc06c720 msgr2=0x7f10cc0a9d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:30.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.362+0000 7f10d0c79640 1 -- 172.21.15.116:0/1296336605 shutdown_connections 2024-04-25T00:12:30.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:30.362+0000 7f10d0c79640 1 -- 172.21.15.116:0/1296336605 wait complete. 2024-04-25T00:12:30.381 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2024-04-25T00:12:32.214 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:12:32.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.346+0000 7fe257b83640 1 Processor -- start 2024-04-25T00:12:32.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.346+0000 7fe257b83640 1 -- start start 2024-04-25T00:12:32.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.346+0000 7fe257b83640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe250152d80 0x7fe25006bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:32.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.346+0000 7fe257b83640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe25006d5f0 0x7fe250156c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:32.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.346+0000 7fe257b83640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fe25006c290 0x7fe2501571c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:32.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.346+0000 7fe257b83640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fe2500c9d60 con 0x7fe250152d80 2024-04-25T00:12:32.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.346+0000 7fe257b83640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fe2500c9150 con 0x7fe25006c290 2024-04-25T00:12:32.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.346+0000 7fe257b83640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe2500c9430 con 0x7fe25006d5f0 2024-04-25T00:12:32.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.346+0000 7fe25591f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe250152d80 0x7fe25006bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:32.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.346+0000 7fe25591f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe250152d80 0x7fe25006bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35632/0 (socket says 172.21.15.116:35632) 2024-04-25T00:12:32.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.346+0000 7fe25591f640 1 -- 172.21.15.116:0/2791320839 learned_addr learned my addr 172.21.15.116:0/2791320839 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:12:32.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.346+0000 7fe25511e640 1 --2- 172.21.15.116:0/2791320839 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe25006d5f0 0x7fe250156c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:32.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.346+0000 7fe256120640 1 --2- 172.21.15.116:0/2791320839 >> v2:172.21.15.174:3300/0 conn(0x7fe25006c290 0x7fe2501571c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:32.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.346+0000 7fe25591f640 1 -- 172.21.15.116:0/2791320839 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe25006d5f0 msgr2=0x7fe250156c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:12:32.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.346+0000 7fe25591f640 1 --2- 172.21.15.116:0/2791320839 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe25006d5f0 0x7fe250156c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:32.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.346+0000 7fe25591f640 1 -- 172.21.15.116:0/2791320839 >> v2:172.21.15.174:3300/0 conn(0x7fe25006c290 msgr2=0x7fe2501571c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:12:32.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.346+0000 7fe25591f640 1 --2- 172.21.15.116:0/2791320839 >> v2:172.21.15.174:3300/0 conn(0x7fe25006c290 0x7fe2501571c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:32.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.346+0000 7fe25591f640 1 -- 172.21.15.116:0/2791320839 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2500c9710 con 0x7fe250152d80 2024-04-25T00:12:32.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.346+0000 7fe256120640 1 --2- 172.21.15.116:0/2791320839 >> v2:172.21.15.174:3300/0 conn(0x7fe25006c290 0x7fe2501571c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:12:32.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.350+0000 7fe25591f640 1 --2- 172.21.15.116:0/2791320839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe250152d80 0x7fe25006bd50 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fe2500c86f0 tx=0x7fe24001c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b25f747a6835e41 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:32.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.350+0000 7fe25491d640 1 -- 172.21.15.116:0/2791320839 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2400b0070 con 0x7fe250152d80 2024-04-25T00:12:32.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.350+0000 7fe25491d640 1 -- 172.21.15.116:0/2791320839 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2400ab600 con 0x7fe250152d80 2024-04-25T00:12:32.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.350+0000 7fe25491d640 1 -- 172.21.15.116:0/2791320839 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2400ab9d0 con 0x7fe250152d80 2024-04-25T00:12:32.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.350+0000 7fe257b83640 1 -- 172.21.15.116:0/2791320839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe250152d80 msgr2=0x7fe25006bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:32.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.350+0000 7fe257b83640 1 --2- 172.21.15.116:0/2791320839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe250152d80 0x7fe25006bd50 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fe2500c86f0 tx=0x7fe24001c460 comp rx=0 tx=0).stop 2024-04-25T00:12:32.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.350+0000 7fe257b83640 1 -- 172.21.15.116:0/2791320839 shutdown_connections 2024-04-25T00:12:32.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.350+0000 7fe257b83640 1 --2- 172.21.15.116:0/2791320839 >> v2:172.21.15.174:3300/0 conn(0x7fe25006c290 0x7fe2501571c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:32.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.350+0000 7fe257b83640 1 --2- 172.21.15.116:0/2791320839 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe25006d5f0 0x7fe250156c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:32.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.350+0000 7fe257b83640 1 --2- 172.21.15.116:0/2791320839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe250152d80 0x7fe25006bd50 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:32.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.350+0000 7fe257b83640 1 -- 172.21.15.116:0/2791320839 >> 172.21.15.116:0/2791320839 conn(0x7fe2500acdd0 msgr2=0x7fe2500ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:32.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.350+0000 7fe257b83640 1 -- 172.21.15.116:0/2791320839 shutdown_connections 2024-04-25T00:12:32.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.350+0000 7fe257b83640 1 -- 172.21.15.116:0/2791320839 wait complete. 2024-04-25T00:12:32.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.350+0000 7fe257b83640 1 Processor -- start 2024-04-25T00:12:32.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.350+0000 7fe257b83640 1 -- start start 2024-04-25T00:12:32.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.350+0000 7fe257b83640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe25006c290 0x7fe2501f1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:32.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.350+0000 7fe257b83640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe25006d5f0 0x7fe25014c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:32.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.350+0000 7fe257b83640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe250152d80 0x7fe25014c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:32.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.350+0000 7fe257b83640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe250159710 con 0x7fe25006d5f0 2024-04-25T00:12:32.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.350+0000 7fe257b83640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe250159590 con 0x7fe25006c290 2024-04-25T00:12:32.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.350+0000 7fe257b83640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe250159890 con 0x7fe250152d80 2024-04-25T00:12:32.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.350+0000 7fe25511e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe25006d5f0 0x7fe25014c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:32.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.350+0000 7fe25511e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe25006d5f0 0x7fe25014c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35638/0 (socket says 172.21.15.116:35638) 2024-04-25T00:12:32.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.350+0000 7fe25511e640 1 -- 172.21.15.116:0/1268404616 learned_addr learned my addr 172.21.15.116:0/1268404616 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:12:32.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.350+0000 7fe25591f640 1 --2- 172.21.15.116:0/1268404616 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe25006c290 0x7fe2501f1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:32.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.350+0000 7fe256120640 1 --2- 172.21.15.116:0/1268404616 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe250152d80 0x7fe25014c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:32.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.350+0000 7fe25511e640 1 -- 172.21.15.116:0/1268404616 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe250152d80 msgr2=0x7fe25014c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:32.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.350+0000 7fe25511e640 1 --2- 172.21.15.116:0/1268404616 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe250152d80 0x7fe25014c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:32.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.350+0000 7fe25511e640 1 -- 172.21.15.116:0/1268404616 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe25006c290 msgr2=0x7fe2501f1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:32.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.350+0000 7fe25511e640 1 --2- 172.21.15.116:0/1268404616 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe25006c290 0x7fe2501f1cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:32.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.350+0000 7fe25511e640 1 -- 172.21.15.116:0/1268404616 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2500bf790 con 0x7fe25006d5f0 2024-04-25T00:12:32.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.350+0000 7fe25511e640 1 --2- 172.21.15.116:0/1268404616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe25006d5f0 0x7fe25014c240 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fe24c00d800 tx=0x7fe24c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:32.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.354+0000 7fe2467fc640 1 -- 172.21.15.116:0/1268404616 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe24c00b6c0 con 0x7fe25006d5f0 2024-04-25T00:12:32.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.354+0000 7fe2467fc640 1 -- 172.21.15.116:0/1268404616 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe24c019070 con 0x7fe25006d5f0 2024-04-25T00:12:32.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.354+0000 7fe257b83640 1 -- 172.21.15.116:0/1268404616 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2501f2640 con 0x7fe25006d5f0 2024-04-25T00:12:32.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.354+0000 7fe2467fc640 1 -- 172.21.15.116:0/1268404616 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe24c00bac0 con 0x7fe25006d5f0 2024-04-25T00:12:32.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.354+0000 7fe257b83640 1 -- 172.21.15.116:0/1268404616 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe250150ff0 con 0x7fe25006d5f0 2024-04-25T00:12:32.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.354+0000 7fe257b83640 1 -- 172.21.15.116:0/1268404616 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe250150b20 con 0x7fe25006d5f0 2024-04-25T00:12:32.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.354+0000 7fe2467fc640 1 -- 172.21.15.116:0/1268404616 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe24c00bc60 con 0x7fe25006d5f0 2024-04-25T00:12:32.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.354+0000 7fe2467fc640 1 --2- 172.21.15.116:0/1268404616 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe2340678c0 0x7fe234069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:32.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.354+0000 7fe2467fc640 1 -- 172.21.15.116:0/1268404616 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 7956+0+0 (secure 0 0 0) 0x7fe24c0869e0 con 0x7fe25006d5f0 2024-04-25T00:12:32.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.354+0000 7fe25591f640 1 --2- 172.21.15.116:0/1268404616 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe2340678c0 0x7fe234069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:32.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.354+0000 7fe25591f640 1 --2- 172.21.15.116:0/1268404616 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe2340678c0 0x7fe234069d80 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fe2500caac0 tx=0x7fe240009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:32.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.358+0000 7fe2467fc640 1 -- 172.21.15.116:0/1268404616 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fe24c056ce0 con 0x7fe25006d5f0 2024-04-25T00:12:32.382 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:12:32.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.494+0000 7f41ac72f640 1 Processor -- start 2024-04-25T00:12:32.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.498+0000 7f41ac72f640 1 -- start start 2024-04-25T00:12:32.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.498+0000 7f41ac72f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f41a4152e60 0x7f41a41599a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:32.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.498+0000 7f41ac72f640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f41a41562d0 0x7f41a40c36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:32.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.498+0000 7f41ac72f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41a4073200 con 0x7f41a40c3c20 2024-04-25T00:12:32.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.498+0000 7f41ac72f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f41a4156760 con 0x7f41a4152e60 2024-04-25T00:12:32.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.498+0000 7f41ac72f640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f41a4153240 con 0x7f41a41562d0 2024-04-25T00:12:32.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.498+0000 7f41aa4cb640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f41a4152e60 0x7f41a41599a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:32.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.498+0000 7f41aa4cb640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f41a4152e60 0x7f41a41599a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35648/0 (socket says 172.21.15.116:35648) 2024-04-25T00:12:32.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.498+0000 7f41aa4cb640 1 -- 172.21.15.116:0/550667645 learned_addr learned my addr 172.21.15.116:0/550667645 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:12:32.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.498+0000 7f41a9cca640 1 --2- 172.21.15.116:0/550667645 >> v2:172.21.15.174:3300/0 conn(0x7f41a41562d0 0x7f41a40c36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:32.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.498+0000 7f41aa4cb640 1 -- 172.21.15.116:0/550667645 >> v2:172.21.15.174:3300/0 conn(0x7f41a41562d0 msgr2=0x7f41a40c36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:12:32.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.498+0000 7f41aa4cb640 1 --2- 172.21.15.116:0/550667645 >> v2:172.21.15.174:3300/0 conn(0x7f41a41562d0 0x7f41a40c36e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:32.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.498+0000 7f41aa4cb640 1 -- 172.21.15.116:0/550667645 >> v1:172.21.15.116:6789/0 conn(0x7f41a40c3c20 legacy=0x7f41a40c5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:32.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.498+0000 7f41a94c9640 1 -- 172.21.15.116:0/550667645 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2550193381 0 0) 0x7f41a4073200 con 0x7f41a40c3c20 2024-04-25T00:12:32.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.498+0000 7f41aa4cb640 1 -- 172.21.15.116:0/550667645 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41a4158ec0 con 0x7f41a4152e60 2024-04-25T00:12:32.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.498+0000 7f41aa4cb640 1 --2- 172.21.15.116:0/550667645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41a4152e60 0x7f41a41599a0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f41a40c3100 tx=0x7f419401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76179fb2398e427a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:32.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.498+0000 7f41a94c9640 1 -- 172.21.15.116:0/550667645 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41940b0070 con 0x7f41a4152e60 2024-04-25T00:12:32.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.498+0000 7f41a94c9640 1 -- 172.21.15.116:0/550667645 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41940ab600 con 0x7f41a4152e60 2024-04-25T00:12:32.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.498+0000 7f41a94c9640 1 -- 172.21.15.116:0/550667645 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41940ab9d0 con 0x7f41a4152e60 2024-04-25T00:12:32.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.502+0000 7f41ac72f640 1 -- 172.21.15.116:0/550667645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41a4152e60 msgr2=0x7f41a41599a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:32.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.502+0000 7f41ac72f640 1 --2- 172.21.15.116:0/550667645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41a4152e60 0x7f41a41599a0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f41a40c3100 tx=0x7f419401c490 comp rx=0 tx=0).stop 2024-04-25T00:12:32.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.502+0000 7f41ac72f640 1 -- 172.21.15.116:0/550667645 shutdown_connections 2024-04-25T00:12:32.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.502+0000 7f41ac72f640 1 --2- 172.21.15.116:0/550667645 >> v2:172.21.15.174:3300/0 conn(0x7f41a41562d0 0x7f41a40c36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:32.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.502+0000 7f41ac72f640 1 --2- 172.21.15.116:0/550667645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41a4152e60 0x7f41a41599a0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:32.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.502+0000 7f41ac72f640 1 -- 172.21.15.116:0/550667645 >> 172.21.15.116:0/550667645 conn(0x7f41a406c720 msgr2=0x7f41a406cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:32.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.502+0000 7f41ac72f640 1 -- 172.21.15.116:0/550667645 shutdown_connections 2024-04-25T00:12:32.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.502+0000 7f41ac72f640 1 -- 172.21.15.116:0/550667645 wait complete. 2024-04-25T00:12:32.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.502+0000 7f41ac72f640 1 Processor -- start 2024-04-25T00:12:32.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.502+0000 7f41ac72f640 1 -- start start 2024-04-25T00:12:32.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.502+0000 7f41ac72f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41a40c3c20 0x7f41a4151fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:32.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.502+0000 7f41ac72f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f41a4152e60 0x7f41a40ac6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:32.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.502+0000 7f41ac72f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f41a41562d0 0x7f41a413c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:32.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.502+0000 7f41ac72f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f41a40c58a0 con 0x7f41a40c3c20 2024-04-25T00:12:32.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.502+0000 7f41ac72f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f41a40c5720 con 0x7f41a4152e60 2024-04-25T00:12:32.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.502+0000 7f41ac72f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f41a4072e20 con 0x7f41a41562d0 2024-04-25T00:12:32.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.502+0000 7f41aa4cb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41a40c3c20 0x7f41a4151fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:32.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.502+0000 7f41aa4cb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41a40c3c20 0x7f41a4151fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35650/0 (socket says 172.21.15.116:35650) 2024-04-25T00:12:32.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.502+0000 7f41aa4cb640 1 -- 172.21.15.116:0/3268807877 learned_addr learned my addr 172.21.15.116:0/3268807877 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:12:32.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.502+0000 7f41a9cca640 1 --2- 172.21.15.116:0/3268807877 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f41a4152e60 0x7f41a40ac6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:32.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.502+0000 7f41aaccc640 1 --2- 172.21.15.116:0/3268807877 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f41a41562d0 0x7f41a413c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:32.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.502+0000 7f41aa4cb640 1 -- 172.21.15.116:0/3268807877 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f41a41562d0 msgr2=0x7f41a413c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:32.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.502+0000 7f41aa4cb640 1 --2- 172.21.15.116:0/3268807877 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f41a41562d0 0x7f41a413c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:32.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.502+0000 7f41aa4cb640 1 -- 172.21.15.116:0/3268807877 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f41a4152e60 msgr2=0x7f41a40ac6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:32.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.502+0000 7f41aa4cb640 1 --2- 172.21.15.116:0/3268807877 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f41a4152e60 0x7f41a40ac6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:32.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.502+0000 7f41aa4cb640 1 -- 172.21.15.116:0/3268807877 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41a41505c0 con 0x7f41a40c3c20 2024-04-25T00:12:32.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.502+0000 7f41a9cca640 1 --2- 172.21.15.116:0/3268807877 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f41a4152e60 0x7f41a40ac6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:12:32.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.502+0000 7f41aaccc640 1 --2- 172.21.15.116:0/3268807877 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f41a41562d0 0x7f41a413c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:12:32.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.502+0000 7f41aa4cb640 1 --2- 172.21.15.116:0/3268807877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41a40c3c20 0x7f41a4151fb0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f41940ae040 tx=0x7f419401acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:32.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.502+0000 7f419affd640 1 -- 172.21.15.116:0/3268807877 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41940b0040 con 0x7f41a40c3c20 2024-04-25T00:12:32.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.502+0000 7f419affd640 1 -- 172.21.15.116:0/3268807877 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4194004030 con 0x7f41a40c3c20 2024-04-25T00:12:32.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.502+0000 7f419affd640 1 -- 172.21.15.116:0/3268807877 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41940043e0 con 0x7f41a40c3c20 2024-04-25T00:12:32.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.502+0000 7f41ac72f640 1 -- 172.21.15.116:0/3268807877 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41a41f1340 con 0x7f41a40c3c20 2024-04-25T00:12:32.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.502+0000 7f41ac72f640 1 -- 172.21.15.116:0/3268807877 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41a41f1900 con 0x7f41a40c3c20 2024-04-25T00:12:32.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.506+0000 7f4198ff9640 1 -- 172.21.15.116:0/3268807877 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41800052a0 con 0x7f41a40c3c20 2024-04-25T00:12:32.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.506+0000 7f419affd640 1 -- 172.21.15.116:0/3268807877 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f41940b3070 con 0x7f41a40c3c20 2024-04-25T00:12:32.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.506+0000 7f419affd640 1 --2- 172.21.15.116:0/3268807877 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f41880678c0 0x7f4188069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:32.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.506+0000 7f419affd640 1 -- 172.21.15.116:0/3268807877 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 7956+0+0 (secure 0 0 0) 0x7f4194120710 con 0x7f41a40c3c20 2024-04-25T00:12:32.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.506+0000 7f41a9cca640 1 --2- 172.21.15.116:0/3268807877 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f41880678c0 0x7f4188069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:32.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.506+0000 7f41a9cca640 1 --2- 172.21.15.116:0/3268807877 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f41880678c0 0x7f4188069d80 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f41a41537f0 tx=0x7f41a0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:32.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.510+0000 7f419affd640 1 -- 172.21.15.116:0/3268807877 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f41940ba020 con 0x7f41a40c3c20 2024-04-25T00:12:32.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.566+0000 7fe257b83640 1 -- 172.21.15.116:0/1268404616 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe250068350 con 0x7fe25006d5f0 2024-04-25T00:12:32.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.566+0000 7fe2467fc640 1 -- 172.21.15.116:0/1268404616 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+6500 (secure 0 0 0) 0x7fe24c0565e0 con 0x7fe25006d5f0 2024-04-25T00:12:32.565 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:12:32.565 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82},{"gid":6495,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/365773583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":365773583},{"type":"v1","addr":"172.21.15.116:6833","nonce":365773583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:11:03.640578+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6114},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6114":{"gid":6114,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":25,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6114,"qdb_cluster":[6114]},"id":1},{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:12:23.643548+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5461},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5461":{"gid":5461,"name":"b","rank":0,"incarnation":83,"state":"up:active","state_seq":30,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5461,"qdb_cluster":[5461]},"id":2}]} 2024-04-25T00:12:32.565 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 87 2024-04-25T00:12:32.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.566+0000 7fe257b83640 1 -- 172.21.15.116:0/1268404616 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe2340678c0 msgr2=0x7fe234069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:32.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.566+0000 7fe257b83640 1 --2- 172.21.15.116:0/1268404616 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe2340678c0 0x7fe234069d80 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fe2500caac0 tx=0x7fe240009040 comp rx=0 tx=0).stop 2024-04-25T00:12:32.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.566+0000 7fe257b83640 1 -- 172.21.15.116:0/1268404616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe25006d5f0 msgr2=0x7fe25014c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:32.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.566+0000 7fe257b83640 1 --2- 172.21.15.116:0/1268404616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe25006d5f0 0x7fe25014c240 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fe24c00d800 tx=0x7fe24c00dcc0 comp rx=0 tx=0).stop 2024-04-25T00:12:32.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.566+0000 7fe257b83640 1 -- 172.21.15.116:0/1268404616 shutdown_connections 2024-04-25T00:12:32.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.566+0000 7fe257b83640 1 --2- 172.21.15.116:0/1268404616 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe2340678c0 0x7fe234069d80 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:32.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.566+0000 7fe257b83640 1 --2- 172.21.15.116:0/1268404616 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe250152d80 0x7fe25014c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:32.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.566+0000 7fe257b83640 1 --2- 172.21.15.116:0/1268404616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe25006d5f0 0x7fe25014c240 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:32.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.566+0000 7fe257b83640 1 --2- 172.21.15.116:0/1268404616 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe25006c290 0x7fe2501f1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:32.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.566+0000 7fe257b83640 1 -- 172.21.15.116:0/1268404616 >> 172.21.15.116:0/1268404616 conn(0x7fe2500acdd0 msgr2=0x7fe2500caf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:32.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.566+0000 7fe257b83640 1 -- 172.21.15.116:0/1268404616 shutdown_connections 2024-04-25T00:12:32.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.566+0000 7fe257b83640 1 -- 172.21.15.116:0/1268404616 wait complete. 2024-04-25T00:12:32.584 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:12:32.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.710+0000 7f4198ff9640 1 -- 172.21.15.116:0/3268807877 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4180005860 con 0x7f41a40c3c20 2024-04-25T00:12:32.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.710+0000 7f419affd640 1 -- 172.21.15.116:0/3268807877 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+6500 (secure 0 0 0) 0x7f4194016110 con 0x7f41a40c3c20 2024-04-25T00:12:32.709 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:12:32.709 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82},{"gid":6495,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/365773583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":365773583},{"type":"v1","addr":"172.21.15.116:6833","nonce":365773583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:11:03.640578+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6114},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6114":{"gid":6114,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":25,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6114,"qdb_cluster":[6114]},"id":1},{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:12:23.643548+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5461},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5461":{"gid":5461,"name":"b","rank":0,"incarnation":83,"state":"up:active","state_seq":30,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5461,"qdb_cluster":[5461]},"id":2}]} 2024-04-25T00:12:32.710 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 87 2024-04-25T00:12:32.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.710+0000 7f4198ff9640 1 -- 172.21.15.116:0/3268807877 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f41880678c0 msgr2=0x7f4188069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:32.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.710+0000 7f4198ff9640 1 --2- 172.21.15.116:0/3268807877 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f41880678c0 0x7f4188069d80 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f41a41537f0 tx=0x7f41a0008040 comp rx=0 tx=0).stop 2024-04-25T00:12:32.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.710+0000 7f4198ff9640 1 -- 172.21.15.116:0/3268807877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41a40c3c20 msgr2=0x7f41a4151fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:32.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.710+0000 7f4198ff9640 1 --2- 172.21.15.116:0/3268807877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41a40c3c20 0x7f41a4151fb0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f41940ae040 tx=0x7f419401acd0 comp rx=0 tx=0).stop 2024-04-25T00:12:32.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.714+0000 7f4198ff9640 1 -- 172.21.15.116:0/3268807877 shutdown_connections 2024-04-25T00:12:32.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.714+0000 7f4198ff9640 1 --2- 172.21.15.116:0/3268807877 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f41880678c0 0x7f4188069d80 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:32.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.714+0000 7f4198ff9640 1 --2- 172.21.15.116:0/3268807877 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f41a41562d0 0x7f41a413c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:32.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.714+0000 7f4198ff9640 1 --2- 172.21.15.116:0/3268807877 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f41a4152e60 0x7f41a40ac6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:32.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.714+0000 7f4198ff9640 1 --2- 172.21.15.116:0/3268807877 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f41a40c3c20 0x7f41a4151fb0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:32.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.714+0000 7f4198ff9640 1 -- 172.21.15.116:0/3268807877 >> 172.21.15.116:0/3268807877 conn(0x7f41a406c720 msgr2=0x7f41a415f350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:32.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.714+0000 7f4198ff9640 1 -- 172.21.15.116:0/3268807877 shutdown_connections 2024-04-25T00:12:32.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:32.714+0000 7f4198ff9640 1 -- 172.21.15.116:0/3268807877 wait complete. 2024-04-25T00:12:32.728 INFO:tasks.mds_thrash.fs.[b]:mds.c reported in up:standby state 2024-04-25T00:12:32.728 INFO:tasks.mds_thrash.fs.[b]:waiting for 41 secs before thrashing 2024-04-25T00:12:34.585 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:12:34.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.722+0000 7f28bfd7f640 1 Processor -- start 2024-04-25T00:12:34.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.722+0000 7f28bfd7f640 1 -- start start 2024-04-25T00:12:34.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.722+0000 7f28bfd7f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28b8152d80 0x7f28b806bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:34.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.722+0000 7f28bfd7f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f28b806c290 0x7f28b8157390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:34.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.722+0000 7f28bfd7f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28b80c9150 con 0x7f28b806d5f0 2024-04-25T00:12:34.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.722+0000 7f28bfd7f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f28b80c9d60 con 0x7f28b806c290 2024-04-25T00:12:34.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.722+0000 7f28bfd7f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f28b80c9430 con 0x7f28b8152d80 2024-04-25T00:12:34.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.722+0000 7f28be31c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f28b806c290 0x7f28b8157390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:34.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.722+0000 7f28be31c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f28b806c290 0x7f28b8157390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35654/0 (socket says 172.21.15.116:35654) 2024-04-25T00:12:34.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.722+0000 7f28be31c640 1 -- 172.21.15.116:0/778316921 learned_addr learned my addr 172.21.15.116:0/778316921 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:12:34.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.722+0000 7f28bdb1b640 1 --2- 172.21.15.116:0/778316921 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28b8152d80 0x7f28b806bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:34.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.722+0000 7f28be31c640 1 -- 172.21.15.116:0/778316921 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28b8152d80 msgr2=0x7f28b806bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:12:34.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.722+0000 7f28be31c640 1 --2- 172.21.15.116:0/778316921 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28b8152d80 0x7f28b806bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:34.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.722+0000 7f28be31c640 1 -- 172.21.15.116:0/778316921 >> v1:172.21.15.116:6789/0 conn(0x7f28b806d5f0 legacy=0x7f28b8156c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:34.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.722+0000 7f28be31c640 1 -- 172.21.15.116:0/778316921 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28b80c9710 con 0x7f28b806c290 2024-04-25T00:12:34.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.722+0000 7f28be31c640 1 --2- 172.21.15.116:0/778316921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28b806c290 0x7f28b8157390 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f28b4009870 tx=0x7f28b401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a517c006e0238be9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:34.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.726+0000 7f28bcb19640 1 -- 172.21.15.116:0/778316921 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28b40aa620 con 0x7f28b806c290 2024-04-25T00:12:34.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.726+0000 7f28bcb19640 1 -- 172.21.15.116:0/778316921 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28b40af070 con 0x7f28b806c290 2024-04-25T00:12:34.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.726+0000 7f28bcb19640 1 -- 172.21.15.116:0/778316921 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28b40aa9e0 con 0x7f28b806c290 2024-04-25T00:12:34.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.726+0000 7f28bfd7f640 1 -- 172.21.15.116:0/778316921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28b806c290 msgr2=0x7f28b8157390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:34.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.726+0000 7f28bfd7f640 1 --2- 172.21.15.116:0/778316921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28b806c290 0x7f28b8157390 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f28b4009870 tx=0x7f28b401c4a0 comp rx=0 tx=0).stop 2024-04-25T00:12:34.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.726+0000 7f28bfd7f640 1 -- 172.21.15.116:0/778316921 shutdown_connections 2024-04-25T00:12:34.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.726+0000 7f28bfd7f640 1 --2- 172.21.15.116:0/778316921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28b806c290 0x7f28b8157390 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:34.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.726+0000 7f28bfd7f640 1 --2- 172.21.15.116:0/778316921 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28b8152d80 0x7f28b806bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:34.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.726+0000 7f28bfd7f640 1 -- 172.21.15.116:0/778316921 >> 172.21.15.116:0/778316921 conn(0x7f28b80acdd0 msgr2=0x7f28b80ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:34.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.726+0000 7f28bfd7f640 1 -- 172.21.15.116:0/778316921 shutdown_connections 2024-04-25T00:12:34.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.726+0000 7f28bfd7f640 1 -- 172.21.15.116:0/778316921 wait complete. 2024-04-25T00:12:34.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.726+0000 7f28bfd7f640 1 Processor -- start 2024-04-25T00:12:34.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.726+0000 7f28bfd7f640 1 -- start start 2024-04-25T00:12:34.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.726+0000 7f28bfd7f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28b806c290 0x7f28b80aae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:34.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.726+0000 7f28bfd7f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28b806d5f0 0x7f28b81476c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:34.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.726+0000 7f28bfd7f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28b8152d80 0x7f28b80ac210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:34.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.726+0000 7f28bfd7f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f28b80caf00 con 0x7f28b806c290 2024-04-25T00:12:34.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.726+0000 7f28bfd7f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f28b80cad80 con 0x7f28b806d5f0 2024-04-25T00:12:34.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.726+0000 7f28bfd7f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f28b81628d0 con 0x7f28b8152d80 2024-04-25T00:12:34.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.726+0000 7f28bdb1b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28b806c290 0x7f28b80aae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:34.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.726+0000 7f28bdb1b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28b806c290 0x7f28b80aae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35670/0 (socket says 172.21.15.116:35670) 2024-04-25T00:12:34.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.726+0000 7f28bdb1b640 1 -- 172.21.15.116:0/2756460002 learned_addr learned my addr 172.21.15.116:0/2756460002 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:12:34.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.726+0000 7f28be31c640 1 --2- 172.21.15.116:0/2756460002 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28b8152d80 0x7f28b80ac210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:34.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.726+0000 7f28bdb1b640 1 -- 172.21.15.116:0/2756460002 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28b8152d80 msgr2=0x7f28b80ac210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:34.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.726+0000 7f28bdb1b640 1 --2- 172.21.15.116:0/2756460002 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28b8152d80 0x7f28b80ac210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:34.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.726+0000 7f28bdb1b640 1 -- 172.21.15.116:0/2756460002 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28b806d5f0 msgr2=0x7f28b81476c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:34.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.726+0000 7f28bdb1b640 1 --2- 172.21.15.116:0/2756460002 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28b806d5f0 0x7f28b81476c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:34.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.726+0000 7f28bdb1b640 1 -- 172.21.15.116:0/2756460002 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28b814a5a0 con 0x7f28b806c290 2024-04-25T00:12:34.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.726+0000 7f28bdb1b640 1 --2- 172.21.15.116:0/2756460002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28b806c290 0x7f28b80aae10 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f28b0007ef0 tx=0x7f28b000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:34.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.726+0000 7f28ae7fc640 1 -- 172.21.15.116:0/2756460002 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28b0014620 con 0x7f28b806c290 2024-04-25T00:12:34.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.726+0000 7f28ae7fc640 1 -- 172.21.15.116:0/2756460002 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28b0019070 con 0x7f28b806c290 2024-04-25T00:12:34.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.726+0000 7f28bfd7f640 1 -- 172.21.15.116:0/2756460002 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28b80abdf0 con 0x7f28b806c290 2024-04-25T00:12:34.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.726+0000 7f28ae7fc640 1 -- 172.21.15.116:0/2756460002 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28b0014a00 con 0x7f28b806c290 2024-04-25T00:12:34.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.726+0000 7f28bfd7f640 1 -- 172.21.15.116:0/2756460002 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28b8147c00 con 0x7f28b806c290 2024-04-25T00:12:34.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.730+0000 7f28bfd7f640 1 -- 172.21.15.116:0/2756460002 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28b8068350 con 0x7f28b806c290 2024-04-25T00:12:34.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.730+0000 7f28ae7fc640 1 -- 172.21.15.116:0/2756460002 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f28b0014ba0 con 0x7f28b806c290 2024-04-25T00:12:34.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.730+0000 7f28ae7fc640 1 --2- 172.21.15.116:0/2756460002 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f289c0678f0 0x7f289c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:34.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.730+0000 7f28ae7fc640 1 -- 172.21.15.116:0/2756460002 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 7956+0+0 (secure 0 0 0) 0x7f28b0086df0 con 0x7f28b806c290 2024-04-25T00:12:34.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.730+0000 7f28bd31a640 1 --2- 172.21.15.116:0/2756460002 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f289c0678f0 0x7f289c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:34.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.730+0000 7f28bd31a640 1 --2- 172.21.15.116:0/2756460002 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f289c0678f0 0x7f289c069db0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f28a8000f70 tx=0x7f28a80079c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:34.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.734+0000 7f28ae7fc640 1 -- 172.21.15.116:0/2756460002 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f28b00570f0 con 0x7f28b806c290 2024-04-25T00:12:34.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.926+0000 7f28bfd7f640 1 -- 172.21.15.116:0/2756460002 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f28b80cab60 con 0x7f28b806c290 2024-04-25T00:12:34.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.930+0000 7f28ae7fc640 1 -- 172.21.15.116:0/2756460002 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+6500 (secure 0 0 0) 0x7f28b00569f0 con 0x7f28b806c290 2024-04-25T00:12:34.929 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:12:34.929 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82},{"gid":6495,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/365773583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":365773583},{"type":"v1","addr":"172.21.15.116:6833","nonce":365773583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:11:03.640578+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6114},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6114":{"gid":6114,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":25,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6114,"qdb_cluster":[6114]},"id":1},{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:12:23.643548+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5461},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5461":{"gid":5461,"name":"b","rank":0,"incarnation":83,"state":"up:active","state_seq":30,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5461,"qdb_cluster":[5461]},"id":2}]} 2024-04-25T00:12:34.929 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 87 2024-04-25T00:12:34.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.930+0000 7f28bfd7f640 1 -- 172.21.15.116:0/2756460002 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f289c0678f0 msgr2=0x7f289c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:34.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.930+0000 7f28bfd7f640 1 --2- 172.21.15.116:0/2756460002 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f289c0678f0 0x7f289c069db0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f28a8000f70 tx=0x7f28a80079c0 comp rx=0 tx=0).stop 2024-04-25T00:12:34.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.930+0000 7f28bfd7f640 1 -- 172.21.15.116:0/2756460002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28b806c290 msgr2=0x7f28b80aae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:34.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.930+0000 7f28bfd7f640 1 --2- 172.21.15.116:0/2756460002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28b806c290 0x7f28b80aae10 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f28b0007ef0 tx=0x7f28b000e4b0 comp rx=0 tx=0).stop 2024-04-25T00:12:34.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.930+0000 7f28bfd7f640 1 -- 172.21.15.116:0/2756460002 shutdown_connections 2024-04-25T00:12:34.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.930+0000 7f28bfd7f640 1 --2- 172.21.15.116:0/2756460002 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f289c0678f0 0x7f289c069db0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:34.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.930+0000 7f28bfd7f640 1 --2- 172.21.15.116:0/2756460002 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28b8152d80 0x7f28b80ac210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:34.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.930+0000 7f28bfd7f640 1 --2- 172.21.15.116:0/2756460002 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28b806d5f0 0x7f28b81476c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:34.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.930+0000 7f28bfd7f640 1 --2- 172.21.15.116:0/2756460002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28b806c290 0x7f28b80aae10 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:34.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.930+0000 7f28bfd7f640 1 -- 172.21.15.116:0/2756460002 >> 172.21.15.116:0/2756460002 conn(0x7f28b80acdd0 msgr2=0x7f28b8155d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:34.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.930+0000 7f28bfd7f640 1 -- 172.21.15.116:0/2756460002 shutdown_connections 2024-04-25T00:12:34.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:34.930+0000 7f28bfd7f640 1 -- 172.21.15.116:0/2756460002 wait complete. 2024-04-25T00:12:34.948 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:12:36.949 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:12:37.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.082+0000 7f6c3181c640 1 Processor -- start 2024-04-25T00:12:37.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.082+0000 7f6c3181c640 1 -- start start 2024-04-25T00:12:37.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.082+0000 7f6c3181c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c2c155810 0x7f6c2c157390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:37.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.082+0000 7f6c3181c640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c2c0c9150 con 0x7f6c2c152d80 2024-04-25T00:12:37.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.082+0000 7f6c3181c640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c2c0c9430 con 0x7f6c2c06d5f0 2024-04-25T00:12:37.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.082+0000 7f6c3181c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6c2c0c9d60 con 0x7f6c2c155810 2024-04-25T00:12:37.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.082+0000 7f6c2affd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f6c2c152d80 0x7f6c2c06bd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:55810/0 (socket says 172.21.15.116:55810) 2024-04-25T00:12:37.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.082+0000 7f6c2affd640 1 -- 172.21.15.116:0/2181488624 learned_addr learned my addr 172.21.15.116:0/2181488624 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:12:37.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.082+0000 7f6c2b7fe640 1 --2- 172.21.15.116:0/2181488624 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c2c155810 0x7f6c2c157390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:37.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.082+0000 7f6c29ffb640 1 -- 172.21.15.116:0/2181488624 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2053570592 0 0) 0x7f6c2c0c9150 con 0x7f6c2c152d80 2024-04-25T00:12:37.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.082+0000 7f6c29ffb640 1 -- 172.21.15.116:0/2181488624 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c14003530 con 0x7f6c2c152d80 2024-04-25T00:12:37.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.082+0000 7f6c29ffb640 1 -- 172.21.15.116:0/2181488624 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3602102160 0 0) 0x7f6c14003530 con 0x7f6c2c152d80 2024-04-25T00:12:37.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.082+0000 7f6c29ffb640 1 -- 172.21.15.116:0/2181488624 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c2c155810 msgr2=0x7f6c2c157390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:12:37.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.082+0000 7f6c29ffb640 1 --2- 172.21.15.116:0/2181488624 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c2c155810 0x7f6c2c157390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:37.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.082+0000 7f6c29ffb640 1 -- 172.21.15.116:0/2181488624 >> v1:172.21.15.174:6789/0 conn(0x7f6c2c06d5f0 legacy=0x7f6c2c156c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:37.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.082+0000 7f6c29ffb640 1 -- 172.21.15.116:0/2181488624 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c2c0c9710 con 0x7f6c2c152d80 2024-04-25T00:12:37.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.082+0000 7f6c29ffb640 1 -- 172.21.15.116:0/2181488624 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f6c1c001680 con 0x7f6c2c152d80 2024-04-25T00:12:37.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.082+0000 7f6c29ffb640 1 -- 172.21.15.116:0/2181488624 >> v1:172.21.15.116:6789/0 conn(0x7f6c2c152d80 legacy=0x7f6c2c06bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:37.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.082+0000 7f6c29ffb640 1 --2- 172.21.15.116:0/2181488624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c140039f0 0x7f6c14003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:37.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.082+0000 7f6c29ffb640 1 --2- 172.21.15.116:0/2181488624 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c2c155810 0x7f6c14004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:37.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.082+0000 7f6c29ffb640 1 --2- 172.21.15.116:0/2181488624 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c14004d50 0x7f6c140051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:37.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.082+0000 7f6c29ffb640 1 -- 172.21.15.116:0/2181488624 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6c2c0c9d60 con 0x7f6c140039f0 2024-04-25T00:12:37.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.082+0000 7f6c29ffb640 1 -- 172.21.15.116:0/2181488624 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6c1400bb00 con 0x7f6c14004d50 2024-04-25T00:12:37.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.082+0000 7f6c29ffb640 1 -- 172.21.15.116:0/2181488624 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6c1400bd00 con 0x7f6c2c155810 2024-04-25T00:12:37.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.086+0000 7f6c2affd640 1 --2- 172.21.15.116:0/2181488624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c140039f0 0x7f6c14003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:37.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.086+0000 7f6c2a7fc640 1 --2- 172.21.15.116:0/2181488624 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c2c155810 0x7f6c14004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:37.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.086+0000 7f6c2affd640 1 -- 172.21.15.116:0/2181488624 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c2c155810 msgr2=0x7f6c14004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:12:37.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.086+0000 7f6c2affd640 1 --2- 172.21.15.116:0/2181488624 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c2c155810 0x7f6c14004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:37.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.086+0000 7f6c2b7fe640 1 --2- 172.21.15.116:0/2181488624 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c14004d50 0x7f6c140051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:37.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.086+0000 7f6c2affd640 1 -- 172.21.15.116:0/2181488624 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c14004d50 msgr2=0x7f6c140051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:12:37.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.086+0000 7f6c2affd640 1 --2- 172.21.15.116:0/2181488624 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c14004d50 0x7f6c140051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:37.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.086+0000 7f6c2affd640 1 -- 172.21.15.116:0/2181488624 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c14003530 con 0x7f6c140039f0 2024-04-25T00:12:37.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.086+0000 7f6c2affd640 1 --2- 172.21.15.116:0/2181488624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c140039f0 0x7f6c14003e10 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c00c1c0 tx=0x7f6c1c01cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0b014d28d906e5d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:37.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.086+0000 7f6c29ffb640 1 -- 172.21.15.116:0/2181488624 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c1c007d00 con 0x7f6c140039f0 2024-04-25T00:12:37.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.086+0000 7f6c29ffb640 1 -- 172.21.15.116:0/2181488624 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c1c0a9020 con 0x7f6c140039f0 2024-04-25T00:12:37.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.086+0000 7f6c29ffb640 1 -- 172.21.15.116:0/2181488624 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c1c005b00 con 0x7f6c140039f0 2024-04-25T00:12:37.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.086+0000 7f6c3181c640 1 -- 172.21.15.116:0/2181488624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c140039f0 msgr2=0x7f6c14003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:37.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.086+0000 7f6c3181c640 1 --2- 172.21.15.116:0/2181488624 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c140039f0 0x7f6c14003e10 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c00c1c0 tx=0x7f6c1c01cb70 comp rx=0 tx=0).stop 2024-04-25T00:12:37.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.086+0000 7f6c2b7fe640 1 -- 172.21.15.116:0/2181488624 reap_dead start 2024-04-25T00:12:37.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.090+0000 7f6c3181c640 1 -- 172.21.15.116:0/2181488624 shutdown_connections 2024-04-25T00:12:37.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.090+0000 7f6c3181c640 1 -- 172.21.15.116:0/2181488624 >> 172.21.15.116:0/2181488624 conn(0x7f6c2c0acdd0 msgr2=0x7f6c2c0ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:37.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.090+0000 7f6c3181c640 1 -- 172.21.15.116:0/2181488624 shutdown_connections 2024-04-25T00:12:37.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.090+0000 7f6c3181c640 1 -- 172.21.15.116:0/2181488624 wait complete. 2024-04-25T00:12:37.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.090+0000 7f6c3181c640 1 Processor -- start 2024-04-25T00:12:37.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.090+0000 7f6c3181c640 1 -- start start 2024-04-25T00:12:37.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.090+0000 7f6c3181c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c140039f0 0x7f6c2c0c2540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:37.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.090+0000 7f6c3181c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c2c14c560 0x7f6c2c0c3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:37.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.090+0000 7f6c3181c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c2c1f2490 0x7f6c2c14fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:37.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.090+0000 7f6c3181c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6c2c0caf00 con 0x7f6c2c14c560 2024-04-25T00:12:37.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.090+0000 7f6c3181c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6c2c0cad80 con 0x7f6c140039f0 2024-04-25T00:12:37.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.090+0000 7f6c3181c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6c2c1577a0 con 0x7f6c2c1f2490 2024-04-25T00:12:37.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.090+0000 7f6c2a7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c2c14c560 0x7f6c2c0c3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:37.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.090+0000 7f6c2a7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c2c14c560 0x7f6c2c0c3fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46956/0 (socket says 172.21.15.116:46956) 2024-04-25T00:12:37.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.090+0000 7f6c2a7fc640 1 -- 172.21.15.116:0/2246840147 learned_addr learned my addr 172.21.15.116:0/2246840147 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:12:37.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.090+0000 7f6c2affd640 1 --2- 172.21.15.116:0/2246840147 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c140039f0 0x7f6c2c0c2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:37.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.090+0000 7f6c2b7fe640 1 --2- 172.21.15.116:0/2246840147 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c2c1f2490 0x7f6c2c14fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:37.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.090+0000 7f6c2a7fc640 1 -- 172.21.15.116:0/2246840147 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c2c1f2490 msgr2=0x7f6c2c14fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:37.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.090+0000 7f6c2a7fc640 1 --2- 172.21.15.116:0/2246840147 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c2c1f2490 0x7f6c2c14fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:37.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.090+0000 7f6c2a7fc640 1 -- 172.21.15.116:0/2246840147 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c140039f0 msgr2=0x7f6c2c0c2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:37.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.090+0000 7f6c2a7fc640 1 --2- 172.21.15.116:0/2246840147 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c140039f0 0x7f6c2c0c2540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:37.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.090+0000 7f6c2a7fc640 1 -- 172.21.15.116:0/2246840147 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c2c152270 con 0x7f6c2c14c560 2024-04-25T00:12:37.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.090+0000 7f6c2a7fc640 1 --2- 172.21.15.116:0/2246840147 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c2c14c560 0x7f6c2c0c3fa0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f6c2000e500 tx=0x7f6c2000e9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:37.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.090+0000 7f6c13fff640 1 -- 172.21.15.116:0/2246840147 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c200053a0 con 0x7f6c2c14c560 2024-04-25T00:12:37.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.090+0000 7f6c13fff640 1 -- 172.21.15.116:0/2246840147 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c20018070 con 0x7f6c2c14c560 2024-04-25T00:12:37.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.090+0000 7f6c13fff640 1 -- 172.21.15.116:0/2246840147 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c20003820 con 0x7f6c2c14c560 2024-04-25T00:12:37.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.090+0000 7f6c3181c640 1 -- 172.21.15.116:0/2246840147 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c2c152550 con 0x7f6c2c14c560 2024-04-25T00:12:37.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.090+0000 7f6c3181c640 1 -- 172.21.15.116:0/2246840147 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c2c151270 con 0x7f6c2c14c560 2024-04-25T00:12:37.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.094+0000 7f6c3181c640 1 -- 172.21.15.116:0/2246840147 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c2c068350 con 0x7f6c2c14c560 2024-04-25T00:12:37.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.094+0000 7f6c13fff640 1 -- 172.21.15.116:0/2246840147 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6c200039c0 con 0x7f6c2c14c560 2024-04-25T00:12:37.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.094+0000 7f6c13fff640 1 --2- 172.21.15.116:0/2246840147 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6c180678f0 0x7f6c18069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:37.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.094+0000 7f6c13fff640 1 -- 172.21.15.116:0/2246840147 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 7956+0+0 (secure 0 0 0) 0x7f6c200861d0 con 0x7f6c2c14c560 2024-04-25T00:12:37.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.094+0000 7f6c2affd640 1 --2- 172.21.15.116:0/2246840147 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6c180678f0 0x7f6c18069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:37.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.094+0000 7f6c2affd640 1 --2- 172.21.15.116:0/2246840147 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6c180678f0 0x7f6c18069db0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c0aa000 tx=0x7f6c2c0c9cd0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:37.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.098+0000 7f6c13fff640 1 -- 172.21.15.116:0/2246840147 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f6c200564d0 con 0x7f6c2c14c560 2024-04-25T00:12:37.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.298+0000 7f6c3181c640 1 -- 172.21.15.116:0/2246840147 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6c2c0cab60 con 0x7f6c2c14c560 2024-04-25T00:12:37.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.298+0000 7f6c13fff640 1 -- 172.21.15.116:0/2246840147 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+6500 (secure 0 0 0) 0x7f6c20055dd0 con 0x7f6c2c14c560 2024-04-25T00:12:37.297 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:12:37.297 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82},{"gid":6495,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/365773583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":365773583},{"type":"v1","addr":"172.21.15.116:6833","nonce":365773583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:11:03.640578+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6114},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6114":{"gid":6114,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":25,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6114,"qdb_cluster":[6114]},"id":1},{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:12:23.643548+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5461},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5461":{"gid":5461,"name":"b","rank":0,"incarnation":83,"state":"up:active","state_seq":30,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5461,"qdb_cluster":[5461]},"id":2}]} 2024-04-25T00:12:37.298 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 87 2024-04-25T00:12:37.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.298+0000 7f6c3181c640 1 -- 172.21.15.116:0/2246840147 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6c180678f0 msgr2=0x7f6c18069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:37.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.298+0000 7f6c3181c640 1 --2- 172.21.15.116:0/2246840147 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6c180678f0 0x7f6c18069db0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c0aa000 tx=0x7f6c2c0c9cd0 comp rx=0 tx=0).stop 2024-04-25T00:12:37.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.298+0000 7f6c3181c640 1 -- 172.21.15.116:0/2246840147 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c2c14c560 msgr2=0x7f6c2c0c3fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:37.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.298+0000 7f6c3181c640 1 --2- 172.21.15.116:0/2246840147 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c2c14c560 0x7f6c2c0c3fa0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f6c2000e500 tx=0x7f6c2000e9c0 comp rx=0 tx=0).stop 2024-04-25T00:12:37.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.302+0000 7f6c3181c640 1 -- 172.21.15.116:0/2246840147 shutdown_connections 2024-04-25T00:12:37.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.302+0000 7f6c3181c640 1 --2- 172.21.15.116:0/2246840147 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6c180678f0 0x7f6c18069db0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:37.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.302+0000 7f6c3181c640 1 --2- 172.21.15.116:0/2246840147 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c2c1f2490 0x7f6c2c14fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:37.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.302+0000 7f6c3181c640 1 --2- 172.21.15.116:0/2246840147 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c2c14c560 0x7f6c2c0c3fa0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:37.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.302+0000 7f6c3181c640 1 --2- 172.21.15.116:0/2246840147 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c140039f0 0x7f6c2c0c2540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:37.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.302+0000 7f6c3181c640 1 -- 172.21.15.116:0/2246840147 >> 172.21.15.116:0/2246840147 conn(0x7f6c2c0acdd0 msgr2=0x7f6c2c0ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:37.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.302+0000 7f6c3181c640 1 -- 172.21.15.116:0/2246840147 shutdown_connections 2024-04-25T00:12:37.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:37.302+0000 7f6c3181c640 1 -- 172.21.15.116:0/2246840147 wait complete. 2024-04-25T00:12:37.317 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:12:37.666 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:12:37.669 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:12:39.318 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:12:39.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.450+0000 7fc7d290c640 1 Processor -- start 2024-04-25T00:12:39.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.450+0000 7fc7d290c640 1 -- start start 2024-04-25T00:12:39.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.454+0000 7fc7d290c640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fc7cc14c720 0x7fc7cc1563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:39.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.454+0000 7fc7d290c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc7cc0a94a0 0x7fc7cc0c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:39.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.454+0000 7fc7d290c640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7cc071220 con 0x7fc7cc0c2f50 2024-04-25T00:12:39.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.454+0000 7fc7d290c640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fc7cc153460 con 0x7fc7cc0a94a0 2024-04-25T00:12:39.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.454+0000 7fc7d290c640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fc7cc152420 con 0x7fc7cc14c720 2024-04-25T00:12:39.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.454+0000 7fc7cb7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc7cc0a94a0 0x7fc7cc0c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:39.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.454+0000 7fc7cb7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc7cc0a94a0 0x7fc7cc0c2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46958/0 (socket says 172.21.15.116:46958) 2024-04-25T00:12:39.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.454+0000 7fc7cb7fe640 1 -- 172.21.15.116:0/758854951 learned_addr learned my addr 172.21.15.116:0/758854951 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:12:39.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.454+0000 7fc7cb7fe640 1 -- 172.21.15.116:0/758854951 >> v2:172.21.15.174:3300/0 conn(0x7fc7cc14c720 msgr2=0x7fc7cc1563d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:12:39.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.454+0000 7fc7cbfff640 1 --2- 172.21.15.116:0/758854951 >> v2:172.21.15.174:3300/0 conn(0x7fc7cc14c720 0x7fc7cc1563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:39.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.454+0000 7fc7cb7fe640 1 --2- 172.21.15.116:0/758854951 >> v2:172.21.15.174:3300/0 conn(0x7fc7cc14c720 0x7fc7cc1563d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:39.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.454+0000 7fc7cb7fe640 1 -- 172.21.15.116:0/758854951 >> v1:172.21.15.174:6789/0 conn(0x7fc7cc0c2f50 legacy=0x7fc7cc071680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:12:39.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.454+0000 7fc7cb7fe640 1 -- 172.21.15.116:0/758854951 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7cc14cb00 con 0x7fc7cc0a94a0 2024-04-25T00:12:39.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.458+0000 7fc7cb7fe640 1 --2- 172.21.15.116:0/758854951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7cc0a94a0 0x7fc7cc0c2a10 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc070190 tx=0x7fc7c001eaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24462c706506d0e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:39.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.458+0000 7fc7caffd640 1 -- 172.21.15.116:0/758854951 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c00b4070 con 0x7fc7cc0a94a0 2024-04-25T00:12:39.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.458+0000 7fc7caffd640 1 -- 172.21.15.116:0/758854951 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7c00a7ae0 con 0x7fc7cc0a94a0 2024-04-25T00:12:39.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.458+0000 7fc7caffd640 1 -- 172.21.15.116:0/758854951 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c00af440 con 0x7fc7cc0a94a0 2024-04-25T00:12:39.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.458+0000 7fc7d290c640 1 -- 172.21.15.116:0/758854951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7cc0a94a0 msgr2=0x7fc7cc0c2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:39.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.458+0000 7fc7d290c640 1 --2- 172.21.15.116:0/758854951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7cc0a94a0 0x7fc7cc0c2a10 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc070190 tx=0x7fc7c001eaf0 comp rx=0 tx=0).stop 2024-04-25T00:12:39.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.458+0000 7fc7d290c640 1 -- 172.21.15.116:0/758854951 shutdown_connections 2024-04-25T00:12:39.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.458+0000 7fc7d290c640 1 --2- 172.21.15.116:0/758854951 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7cc0a94a0 0x7fc7cc0c2a10 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:39.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.458+0000 7fc7d290c640 1 --2- 172.21.15.116:0/758854951 >> v2:172.21.15.174:3300/0 conn(0x7fc7cc14c720 0x7fc7cc1563d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:39.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.458+0000 7fc7d290c640 1 -- 172.21.15.116:0/758854951 >> 172.21.15.116:0/758854951 conn(0x7fc7cc0c3b00 msgr2=0x7fc7cc0c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:39.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.458+0000 7fc7d290c640 1 -- 172.21.15.116:0/758854951 shutdown_connections 2024-04-25T00:12:39.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.458+0000 7fc7d290c640 1 -- 172.21.15.116:0/758854951 wait complete. 2024-04-25T00:12:39.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.458+0000 7fc7d290c640 1 Processor -- start 2024-04-25T00:12:39.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.458+0000 7fc7d290c640 1 -- start start 2024-04-25T00:12:39.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.462+0000 7fc7d290c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc7cc0a94a0 0x7fc7cc0c0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:39.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.462+0000 7fc7d290c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7cc0c2f50 0x7fc7cc1f2390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:39.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.462+0000 7fc7d290c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7cc14c720 0x7fc7cc1f28d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:39.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.462+0000 7fc7d290c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc7cc152f40 con 0x7fc7cc14c720 2024-04-25T00:12:39.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.462+0000 7fc7d290c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc7cc152c60 con 0x7fc7cc0c2f50 2024-04-25T00:12:39.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.462+0000 7fc7d290c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fc7cc06d460 con 0x7fc7cc0a94a0 2024-04-25T00:12:39.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.462+0000 7fc7d0ea9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7cc14c720 0x7fc7cc1f28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:39.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.462+0000 7fc7d0ea9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7cc14c720 0x7fc7cc1f28d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46974/0 (socket says 172.21.15.116:46974) 2024-04-25T00:12:39.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.462+0000 7fc7d0ea9640 1 -- 172.21.15.116:0/661531929 learned_addr learned my addr 172.21.15.116:0/661531929 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:12:39.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.462+0000 7fc7cbfff640 1 --2- 172.21.15.116:0/661531929 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc7cc0a94a0 0x7fc7cc0c0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:39.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.462+0000 7fc7cb7fe640 1 --2- 172.21.15.116:0/661531929 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7cc0c2f50 0x7fc7cc1f2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:39.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.462+0000 7fc7d0ea9640 1 -- 172.21.15.116:0/661531929 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc7cc0a94a0 msgr2=0x7fc7cc0c0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:39.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.462+0000 7fc7d0ea9640 1 --2- 172.21.15.116:0/661531929 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc7cc0a94a0 0x7fc7cc0c0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:39.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.462+0000 7fc7d0ea9640 1 -- 172.21.15.116:0/661531929 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7cc0c2f50 msgr2=0x7fc7cc1f2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:39.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.462+0000 7fc7d0ea9640 1 --2- 172.21.15.116:0/661531929 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7cc0c2f50 0x7fc7cc1f2390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:39.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.462+0000 7fc7d0ea9640 1 -- 172.21.15.116:0/661531929 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7cc1f1f00 con 0x7fc7cc14c720 2024-04-25T00:12:39.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.462+0000 7fc7cb7fe640 1 --2- 172.21.15.116:0/661531929 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7cc0c2f50 0x7fc7cc1f2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:12:39.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.462+0000 7fc7d0ea9640 1 --2- 172.21.15.116:0/661531929 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7cc14c720 0x7fc7cc1f28d0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fc7c400c780 tx=0x7fc7c400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:39.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.462+0000 7fc7c8ff9640 1 -- 172.21.15.116:0/661531929 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c4004b70 con 0x7fc7cc14c720 2024-04-25T00:12:39.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.462+0000 7fc7c8ff9640 1 -- 172.21.15.116:0/661531929 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7c4004d10 con 0x7fc7cc14c720 2024-04-25T00:12:39.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.462+0000 7fc7d290c640 1 -- 172.21.15.116:0/661531929 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7cc0c6c20 con 0x7fc7cc14c720 2024-04-25T00:12:39.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.462+0000 7fc7c8ff9640 1 -- 172.21.15.116:0/661531929 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c400aa90 con 0x7fc7cc14c720 2024-04-25T00:12:39.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.462+0000 7fc7d290c640 1 -- 172.21.15.116:0/661531929 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7cc0c4ed0 con 0x7fc7cc14c720 2024-04-25T00:12:39.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.462+0000 7fc7d290c640 1 -- 172.21.15.116:0/661531929 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7cc068350 con 0x7fc7cc14c720 2024-04-25T00:12:39.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.462+0000 7fc7c8ff9640 1 -- 172.21.15.116:0/661531929 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc7c400ac30 con 0x7fc7cc14c720 2024-04-25T00:12:39.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.462+0000 7fc7c8ff9640 1 --2- 172.21.15.116:0/661531929 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc7b80678c0 0x7fc7b8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:39.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.462+0000 7fc7c8ff9640 1 -- 172.21.15.116:0/661531929 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 7956+0+0 (secure 0 0 0) 0x7fc7c40867e0 con 0x7fc7cc14c720 2024-04-25T00:12:39.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.466+0000 7fc7cbfff640 1 --2- 172.21.15.116:0/661531929 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc7b80678c0 0x7fc7b8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:39.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.466+0000 7fc7cbfff640 1 --2- 172.21.15.116:0/661531929 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc7b80678c0 0x7fc7b8069d80 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fc7bc006fd0 tx=0x7fc7bc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:39.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.470+0000 7fc7c8ff9640 1 -- 172.21.15.116:0/661531929 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fc7c4020070 con 0x7fc7cc14c720 2024-04-25T00:12:39.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.666+0000 7fc7d290c640 1 -- 172.21.15.116:0/661531929 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc7cc073360 con 0x7fc7cc14c720 2024-04-25T00:12:39.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.666+0000 7fc7c8ff9640 1 -- 172.21.15.116:0/661531929 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+6500 (secure 0 0 0) 0x7fc7c40569b0 con 0x7fc7cc14c720 2024-04-25T00:12:39.666 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:12:39.667 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82},{"gid":6495,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/365773583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":365773583},{"type":"v1","addr":"172.21.15.116:6833","nonce":365773583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:11:03.640578+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6114},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6114":{"gid":6114,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":25,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6114,"qdb_cluster":[6114]},"id":1},{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:12:23.643548+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5461},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5461":{"gid":5461,"name":"b","rank":0,"incarnation":83,"state":"up:active","state_seq":30,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5461,"qdb_cluster":[5461]},"id":2}]} 2024-04-25T00:12:39.667 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 87 2024-04-25T00:12:39.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.670+0000 7fc7d290c640 1 -- 172.21.15.116:0/661531929 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc7b80678c0 msgr2=0x7fc7b8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:39.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.670+0000 7fc7d290c640 1 --2- 172.21.15.116:0/661531929 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc7b80678c0 0x7fc7b8069d80 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fc7bc006fd0 tx=0x7fc7bc008040 comp rx=0 tx=0).stop 2024-04-25T00:12:39.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.670+0000 7fc7d290c640 1 -- 172.21.15.116:0/661531929 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7cc14c720 msgr2=0x7fc7cc1f28d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:39.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.670+0000 7fc7d290c640 1 --2- 172.21.15.116:0/661531929 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7cc14c720 0x7fc7cc1f28d0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fc7c400c780 tx=0x7fc7c400cc40 comp rx=0 tx=0).stop 2024-04-25T00:12:39.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.670+0000 7fc7d290c640 1 -- 172.21.15.116:0/661531929 shutdown_connections 2024-04-25T00:12:39.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.670+0000 7fc7d290c640 1 --2- 172.21.15.116:0/661531929 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc7b80678c0 0x7fc7b8069d80 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:39.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.670+0000 7fc7d290c640 1 --2- 172.21.15.116:0/661531929 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7cc14c720 0x7fc7cc1f28d0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:39.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.670+0000 7fc7d290c640 1 --2- 172.21.15.116:0/661531929 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7cc0c2f50 0x7fc7cc1f2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:39.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.670+0000 7fc7d290c640 1 --2- 172.21.15.116:0/661531929 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc7cc0a94a0 0x7fc7cc0c0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:39.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.670+0000 7fc7d290c640 1 -- 172.21.15.116:0/661531929 >> 172.21.15.116:0/661531929 conn(0x7fc7cc0c3b00 msgr2=0x7fc7cc0c3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:39.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.670+0000 7fc7d290c640 1 -- 172.21.15.116:0/661531929 shutdown_connections 2024-04-25T00:12:39.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:39.670+0000 7fc7d290c640 1 -- 172.21.15.116:0/661531929 wait complete. 2024-04-25T00:12:39.686 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:12:41.687 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:12:41.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.818+0000 7f45ad8ea640 1 Processor -- start 2024-04-25T00:12:41.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.818+0000 7f45ad8ea640 1 -- start start 2024-04-25T00:12:41.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.818+0000 7f45ad8ea640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f45a8150d20 0x7f45a80c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:41.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.818+0000 7f45ad8ea640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f45a80a94a0 0x7f45a8153ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:41.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.818+0000 7f45ad8ea640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45a80c3910 con 0x7f45a80c2f50 2024-04-25T00:12:41.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.818+0000 7f45ad8ea640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f45a8154530 con 0x7f45a8150d20 2024-04-25T00:12:41.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.818+0000 7f45ad8ea640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f45a80c16e0 con 0x7f45a80a94a0 2024-04-25T00:12:41.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.818+0000 7f45a6ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f45a8150d20 0x7f45a80c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:41.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.818+0000 7f45a6ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f45a8150d20 0x7f45a80c2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46988/0 (socket says 172.21.15.116:46988) 2024-04-25T00:12:41.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.818+0000 7f45a6ffd640 1 -- 172.21.15.116:0/4293771651 learned_addr learned my addr 172.21.15.116:0/4293771651 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:12:41.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.818+0000 7f45a77fe640 1 --2- 172.21.15.116:0/4293771651 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f45a80a94a0 0x7f45a8153ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:41.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.818+0000 7f45a6ffd640 1 -- 172.21.15.116:0/4293771651 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f45a80a94a0 msgr2=0x7f45a8153ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:12:41.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.818+0000 7f45a6ffd640 1 --2- 172.21.15.116:0/4293771651 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f45a80a94a0 0x7f45a8153ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:41.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.818+0000 7f45a6ffd640 1 -- 172.21.15.116:0/4293771651 >> v1:172.21.15.116:6789/0 conn(0x7f45a80c2f50 legacy=0x7f45a81538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:41.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.818+0000 7f45a6ffd640 1 -- 172.21.15.116:0/4293771651 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45a80c1400 con 0x7f45a8150d20 2024-04-25T00:12:41.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.822+0000 7f45a6ffd640 1 --2- 172.21.15.116:0/4293771651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45a8150d20 0x7f45a80c2a10 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f459c009870 tx=0x7f459c01c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e107dd681295f106 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:41.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.822+0000 7f45a5ffb640 1 -- 172.21.15.116:0/4293771651 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f459c0a7070 con 0x7f45a8150d20 2024-04-25T00:12:41.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.822+0000 7f45a5ffb640 1 -- 172.21.15.116:0/4293771651 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f459c0b0070 con 0x7f45a8150d20 2024-04-25T00:12:41.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.822+0000 7f45a5ffb640 1 -- 172.21.15.116:0/4293771651 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f459c0ab890 con 0x7f45a8150d20 2024-04-25T00:12:41.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.822+0000 7f45ad8ea640 1 -- 172.21.15.116:0/4293771651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45a8150d20 msgr2=0x7f45a80c2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:41.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.822+0000 7f45ad8ea640 1 --2- 172.21.15.116:0/4293771651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45a8150d20 0x7f45a80c2a10 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f459c009870 tx=0x7f459c01c560 comp rx=0 tx=0).stop 2024-04-25T00:12:41.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.822+0000 7f45ad8ea640 1 -- 172.21.15.116:0/4293771651 shutdown_connections 2024-04-25T00:12:41.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.822+0000 7f45ad8ea640 1 --2- 172.21.15.116:0/4293771651 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f45a80a94a0 0x7f45a8153ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:41.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.822+0000 7f45ad8ea640 1 --2- 172.21.15.116:0/4293771651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45a8150d20 0x7f45a80c2a10 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:41.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.822+0000 7f45ad8ea640 1 -- 172.21.15.116:0/4293771651 >> 172.21.15.116:0/4293771651 conn(0x7f45a80c3b00 msgr2=0x7f45a80c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:41.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.822+0000 7f45ad8ea640 1 -- 172.21.15.116:0/4293771651 shutdown_connections 2024-04-25T00:12:41.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.822+0000 7f45ad8ea640 1 -- 172.21.15.116:0/4293771651 wait complete. 2024-04-25T00:12:41.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.822+0000 7f45ad8ea640 1 Processor -- start 2024-04-25T00:12:41.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.822+0000 7f45ad8ea640 1 -- start start 2024-04-25T00:12:41.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.822+0000 7f45ad8ea640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f45a80a94a0 0x7f45a81f2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:41.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.822+0000 7f45ad8ea640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45a80c2f50 0x7f45a8145bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:41.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.822+0000 7f45ad8ea640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45a8150d20 0x7f45a80714b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:41.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.822+0000 7f45ad8ea640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f45a81608b0 con 0x7f45a80c2f50 2024-04-25T00:12:41.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.822+0000 7f45ad8ea640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f45a8160730 con 0x7f45a8150d20 2024-04-25T00:12:41.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.822+0000 7f45ad8ea640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f45a8160a30 con 0x7f45a80a94a0 2024-04-25T00:12:41.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.822+0000 7f45a67fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45a80c2f50 0x7f45a8145bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:41.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.822+0000 7f45a67fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45a80c2f50 0x7f45a8145bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47004/0 (socket says 172.21.15.116:47004) 2024-04-25T00:12:41.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.822+0000 7f45a67fc640 1 -- 172.21.15.116:0/3165597534 learned_addr learned my addr 172.21.15.116:0/3165597534 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:12:41.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.822+0000 7f45a6ffd640 1 --2- 172.21.15.116:0/3165597534 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f45a80a94a0 0x7f45a81f2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:41.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.822+0000 7f45a77fe640 1 --2- 172.21.15.116:0/3165597534 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45a8150d20 0x7f45a80714b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:41.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.822+0000 7f45a67fc640 1 -- 172.21.15.116:0/3165597534 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f45a80a94a0 msgr2=0x7f45a81f2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:41.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.822+0000 7f45a67fc640 1 --2- 172.21.15.116:0/3165597534 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f45a80a94a0 0x7f45a81f2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:41.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.822+0000 7f45a67fc640 1 -- 172.21.15.116:0/3165597534 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45a8150d20 msgr2=0x7f45a80714b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:41.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.822+0000 7f45a67fc640 1 --2- 172.21.15.116:0/3165597534 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45a8150d20 0x7f45a80714b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:41.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.822+0000 7f45a67fc640 1 -- 172.21.15.116:0/3165597534 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45a81f2bd0 con 0x7f45a80c2f50 2024-04-25T00:12:41.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.822+0000 7f45a6ffd640 1 --2- 172.21.15.116:0/3165597534 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f45a80a94a0 0x7f45a81f2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:12:41.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.822+0000 7f45a67fc640 1 --2- 172.21.15.116:0/3165597534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45a80c2f50 0x7f45a8145bc0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f459800c420 tx=0x7f459800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:41.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.822+0000 7f458ffff640 1 -- 172.21.15.116:0/3165597534 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45980052d0 con 0x7f45a80c2f50 2024-04-25T00:12:41.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.822+0000 7f458ffff640 1 -- 172.21.15.116:0/3165597534 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4598005470 con 0x7f45a80c2f50 2024-04-25T00:12:41.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.822+0000 7f45ad8ea640 1 -- 172.21.15.116:0/3165597534 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45a8072100 con 0x7f45a80c2f50 2024-04-25T00:12:41.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.822+0000 7f458ffff640 1 -- 172.21.15.116:0/3165597534 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4598007260 con 0x7f45a80c2f50 2024-04-25T00:12:41.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.826+0000 7f45ad8ea640 1 -- 172.21.15.116:0/3165597534 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45a80726c0 con 0x7f45a80c2f50 2024-04-25T00:12:41.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.826+0000 7f45ad8ea640 1 -- 172.21.15.116:0/3165597534 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45a8068350 con 0x7f45a80c2f50 2024-04-25T00:12:41.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.826+0000 7f458ffff640 1 -- 172.21.15.116:0/3165597534 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4598007400 con 0x7f45a80c2f50 2024-04-25T00:12:41.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.826+0000 7f458ffff640 1 --2- 172.21.15.116:0/3165597534 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f45940678c0 0x7f4594069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:41.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.826+0000 7f458ffff640 1 -- 172.21.15.116:0/3165597534 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 7956+0+0 (secure 0 0 0) 0x7f4598085a10 con 0x7f45a80c2f50 2024-04-25T00:12:41.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.826+0000 7f45a6ffd640 1 --2- 172.21.15.116:0/3165597534 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f45940678c0 0x7f4594069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:41.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.826+0000 7f45a6ffd640 1 --2- 172.21.15.116:0/3165597534 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f45940678c0 0x7f4594069d80 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f459c0af000 tx=0x7f459c0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:41.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:41.830+0000 7f458ffff640 1 -- 172.21.15.116:0/3165597534 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f4598055d10 con 0x7f45a80c2f50 2024-04-25T00:12:42.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:42.026+0000 7f45ad8ea640 1 -- 172.21.15.116:0/3165597534 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f45a80c34f0 con 0x7f45a80c2f50 2024-04-25T00:12:42.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:42.030+0000 7f458ffff640 1 -- 172.21.15.116:0/3165597534 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+6500 (secure 0 0 0) 0x7f4598055610 con 0x7f45a80c2f50 2024-04-25T00:12:42.028 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:12:42.028 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82},{"gid":6495,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/365773583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":365773583},{"type":"v1","addr":"172.21.15.116:6833","nonce":365773583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:11:03.640578+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6114},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6114":{"gid":6114,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":25,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6114,"qdb_cluster":[6114]},"id":1},{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:12:23.643548+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5461},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5461":{"gid":5461,"name":"b","rank":0,"incarnation":83,"state":"up:active","state_seq":30,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5461,"qdb_cluster":[5461]},"id":2}]} 2024-04-25T00:12:42.028 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 87 2024-04-25T00:12:42.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:42.030+0000 7f45ad8ea640 1 -- 172.21.15.116:0/3165597534 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f45940678c0 msgr2=0x7f4594069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:42.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:42.030+0000 7f45ad8ea640 1 --2- 172.21.15.116:0/3165597534 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f45940678c0 0x7f4594069d80 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f459c0af000 tx=0x7f459c0a7000 comp rx=0 tx=0).stop 2024-04-25T00:12:42.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:42.030+0000 7f45ad8ea640 1 -- 172.21.15.116:0/3165597534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45a80c2f50 msgr2=0x7f45a8145bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:42.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:42.030+0000 7f45ad8ea640 1 --2- 172.21.15.116:0/3165597534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45a80c2f50 0x7f45a8145bc0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f459800c420 tx=0x7f459800c8e0 comp rx=0 tx=0).stop 2024-04-25T00:12:42.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:42.030+0000 7f45ad8ea640 1 -- 172.21.15.116:0/3165597534 shutdown_connections 2024-04-25T00:12:42.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:42.030+0000 7f45ad8ea640 1 --2- 172.21.15.116:0/3165597534 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f45940678c0 0x7f4594069d80 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:42.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:42.030+0000 7f45ad8ea640 1 --2- 172.21.15.116:0/3165597534 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45a8150d20 0x7f45a80714b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:42.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:42.030+0000 7f45ad8ea640 1 --2- 172.21.15.116:0/3165597534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45a80c2f50 0x7f45a8145bc0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:42.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:42.030+0000 7f45ad8ea640 1 --2- 172.21.15.116:0/3165597534 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f45a80a94a0 0x7f45a81f2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:42.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:42.030+0000 7f45ad8ea640 1 -- 172.21.15.116:0/3165597534 >> 172.21.15.116:0/3165597534 conn(0x7f45a80c3b00 msgr2=0x7f45a80c0430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:42.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:42.030+0000 7f45ad8ea640 1 -- 172.21.15.116:0/3165597534 shutdown_connections 2024-04-25T00:12:42.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:42.030+0000 7f45ad8ea640 1 -- 172.21.15.116:0/3165597534 wait complete. 2024-04-25T00:12:42.047 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:12:44.048 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:12:44.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.182+0000 7ffb8062f640 1 Processor -- start 2024-04-25T00:12:44.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.182+0000 7ffb8062f640 1 -- start start 2024-04-25T00:12:44.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.182+0000 7ffb8062f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffb78148890 0x7ffb78072590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:44.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.182+0000 7ffb8062f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ffb7806e890 0x7ffb78071980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:44.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.182+0000 7ffb8062f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb78152c10 con 0x7ffb78070dd0 2024-04-25T00:12:44.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.182+0000 7ffb8062f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7ffb78151e50 con 0x7ffb7806e890 2024-04-25T00:12:44.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.182+0000 7ffb8062f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ffb78152130 con 0x7ffb78148890 2024-04-25T00:12:44.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.182+0000 7ffb7ebcc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ffb7806e890 0x7ffb78071980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:44.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.182+0000 7ffb7ebcc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ffb7806e890 0x7ffb78071980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47014/0 (socket says 172.21.15.116:47014) 2024-04-25T00:12:44.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.182+0000 7ffb7ebcc640 1 -- 172.21.15.116:0/926541941 learned_addr learned my addr 172.21.15.116:0/926541941 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:12:44.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.182+0000 7ffb7e3cb640 1 --2- 172.21.15.116:0/926541941 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffb78148890 0x7ffb78072590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:44.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.182+0000 7ffb7ebcc640 1 -- 172.21.15.116:0/926541941 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffb78148890 msgr2=0x7ffb78072590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:12:44.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.182+0000 7ffb7ebcc640 1 --2- 172.21.15.116:0/926541941 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffb78148890 0x7ffb78072590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:44.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.182+0000 7ffb7ebcc640 1 -- 172.21.15.116:0/926541941 >> v1:172.21.15.116:6789/0 conn(0x7ffb78070dd0 legacy=0x7ffb78071270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:44.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.182+0000 7ffb7ebcc640 1 -- 172.21.15.116:0/926541941 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb78152410 con 0x7ffb7806e890 2024-04-25T00:12:44.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.186+0000 7ffb7ebcc640 1 --2- 172.21.15.116:0/926541941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb7806e890 0x7ffb78071980 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7ffb70009e50 tx=0x7ffb7001ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f59c0970d7f35bbc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:44.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.186+0000 7ffb7d3c9640 1 -- 172.21.15.116:0/926541941 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb700a5700 con 0x7ffb7806e890 2024-04-25T00:12:44.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.186+0000 7ffb7d3c9640 1 -- 172.21.15.116:0/926541941 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb700a58a0 con 0x7ffb7806e890 2024-04-25T00:12:44.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.186+0000 7ffb7d3c9640 1 -- 172.21.15.116:0/926541941 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb700a5c70 con 0x7ffb7806e890 2024-04-25T00:12:44.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.186+0000 7ffb8062f640 1 -- 172.21.15.116:0/926541941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb7806e890 msgr2=0x7ffb78071980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:44.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.186+0000 7ffb8062f640 1 --2- 172.21.15.116:0/926541941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb7806e890 0x7ffb78071980 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7ffb70009e50 tx=0x7ffb7001ca30 comp rx=0 tx=0).stop 2024-04-25T00:12:44.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.186+0000 7ffb8062f640 1 -- 172.21.15.116:0/926541941 shutdown_connections 2024-04-25T00:12:44.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.186+0000 7ffb8062f640 1 --2- 172.21.15.116:0/926541941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb7806e890 0x7ffb78071980 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:44.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.186+0000 7ffb8062f640 1 --2- 172.21.15.116:0/926541941 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffb78148890 0x7ffb78072590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:44.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.186+0000 7ffb8062f640 1 -- 172.21.15.116:0/926541941 >> 172.21.15.116:0/926541941 conn(0x7ffb780c3b00 msgr2=0x7ffb780c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:44.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.186+0000 7ffb8062f640 1 -- 172.21.15.116:0/926541941 shutdown_connections 2024-04-25T00:12:44.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.186+0000 7ffb8062f640 1 -- 172.21.15.116:0/926541941 wait complete. 2024-04-25T00:12:44.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.186+0000 7ffb8062f640 1 Processor -- start 2024-04-25T00:12:44.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.186+0000 7ffb8062f640 1 -- start start 2024-04-25T00:12:44.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.186+0000 7ffb8062f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffb7806e890 0x7ffb781ec780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:44.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.186+0000 7ffb8062f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb78070dd0 0x7ffb780c62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:44.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.186+0000 7ffb8062f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffb78148890 0x7ffb781426f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:44.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.186+0000 7ffb8062f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ffb780ab920 con 0x7ffb78070dd0 2024-04-25T00:12:44.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.186+0000 7ffb8062f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ffb780ab7a0 con 0x7ffb7806e890 2024-04-25T00:12:44.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.186+0000 7ffb8062f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ffb780abaa0 con 0x7ffb78148890 2024-04-25T00:12:44.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.186+0000 7ffb7dbca640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb78070dd0 0x7ffb780c62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:44.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.186+0000 7ffb7dbca640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb78070dd0 0x7ffb780c62e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47020/0 (socket says 172.21.15.116:47020) 2024-04-25T00:12:44.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.186+0000 7ffb7dbca640 1 -- 172.21.15.116:0/597090556 learned_addr learned my addr 172.21.15.116:0/597090556 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:12:44.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.186+0000 7ffb7e3cb640 1 --2- 172.21.15.116:0/597090556 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffb7806e890 0x7ffb781ec780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:44.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.186+0000 7ffb7ebcc640 1 --2- 172.21.15.116:0/597090556 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffb78148890 0x7ffb781426f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:44.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.186+0000 7ffb7dbca640 1 -- 172.21.15.116:0/597090556 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffb78148890 msgr2=0x7ffb781426f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:44.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.186+0000 7ffb7dbca640 1 --2- 172.21.15.116:0/597090556 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffb78148890 0x7ffb781426f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:44.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.186+0000 7ffb7dbca640 1 -- 172.21.15.116:0/597090556 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffb7806e890 msgr2=0x7ffb781ec780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:44.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.186+0000 7ffb7dbca640 1 --2- 172.21.15.116:0/597090556 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffb7806e890 0x7ffb781ec780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:44.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.186+0000 7ffb7dbca640 1 -- 172.21.15.116:0/597090556 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb78143be0 con 0x7ffb78070dd0 2024-04-25T00:12:44.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.186+0000 7ffb7dbca640 1 --2- 172.21.15.116:0/597090556 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb78070dd0 0x7ffb780c62e0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7ffb6800c430 tx=0x7ffb6800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:44.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.190+0000 7ffb6effd640 1 -- 172.21.15.116:0/597090556 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb680052d0 con 0x7ffb78070dd0 2024-04-25T00:12:44.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.190+0000 7ffb6effd640 1 -- 172.21.15.116:0/597090556 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb68005470 con 0x7ffb78070dd0 2024-04-25T00:12:44.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.190+0000 7ffb8062f640 1 -- 172.21.15.116:0/597090556 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb7814bb40 con 0x7ffb78070dd0 2024-04-25T00:12:44.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.190+0000 7ffb6effd640 1 -- 172.21.15.116:0/597090556 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb68007260 con 0x7ffb78070dd0 2024-04-25T00:12:44.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.190+0000 7ffb8062f640 1 -- 172.21.15.116:0/597090556 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb781448a0 con 0x7ffb78070dd0 2024-04-25T00:12:44.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.190+0000 7ffb8062f640 1 -- 172.21.15.116:0/597090556 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb78144aa0 con 0x7ffb78070dd0 2024-04-25T00:12:44.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.190+0000 7ffb6effd640 1 -- 172.21.15.116:0/597090556 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ffb68007400 con 0x7ffb78070dd0 2024-04-25T00:12:44.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.190+0000 7ffb6effd640 1 --2- 172.21.15.116:0/597090556 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ffb5c0678c0 0x7ffb5c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:44.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.190+0000 7ffb6effd640 1 -- 172.21.15.116:0/597090556 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 7956+0+0 (secure 0 0 0) 0x7ffb680859b0 con 0x7ffb78070dd0 2024-04-25T00:12:44.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.190+0000 7ffb7e3cb640 1 --2- 172.21.15.116:0/597090556 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ffb5c0678c0 0x7ffb5c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:44.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.190+0000 7ffb7e3cb640 1 --2- 172.21.15.116:0/597090556 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ffb5c0678c0 0x7ffb5c069d80 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7ffb740096f0 tx=0x7ffb74008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:44.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.194+0000 7ffb6effd640 1 -- 172.21.15.116:0/597090556 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ffb78144aa0 con 0x7ffb78070dd0 2024-04-25T00:12:44.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.386+0000 7ffb8062f640 1 -- 172.21.15.116:0/597090556 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffb78068350 con 0x7ffb78070dd0 2024-04-25T00:12:44.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.386+0000 7ffb6effd640 1 -- 172.21.15.116:0/597090556 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) ==== 94+0+6500 (secure 0 0 0) 0x7ffb68055cb0 con 0x7ffb78070dd0 2024-04-25T00:12:44.388 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:12:44.388 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82},{"gid":6495,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/365773583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":365773583},{"type":"v1","addr":"172.21.15.116:6833","nonce":365773583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:11:03.640578+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6114},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6114":{"gid":6114,"name":"a","rank":0,"incarnation":73,"state":"up:active","state_seq":25,"addr":"172.21.15.116:6833/3705654249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3705654249},{"type":"v1","addr":"172.21.15.116:6833","nonce":3705654249}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6114,"qdb_cluster":[6114]},"id":1},{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:12:23.643548+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5461},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5461":{"gid":5461,"name":"b","rank":0,"incarnation":83,"state":"up:active","state_seq":30,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5461,"qdb_cluster":[5461]},"id":2}]} 2024-04-25T00:12:44.389 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 87 2024-04-25T00:12:44.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.390+0000 7ffb8062f640 1 -- 172.21.15.116:0/597090556 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ffb5c0678c0 msgr2=0x7ffb5c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:44.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.390+0000 7ffb8062f640 1 --2- 172.21.15.116:0/597090556 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ffb5c0678c0 0x7ffb5c069d80 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7ffb740096f0 tx=0x7ffb74008040 comp rx=0 tx=0).stop 2024-04-25T00:12:44.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.390+0000 7ffb8062f640 1 -- 172.21.15.116:0/597090556 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb78070dd0 msgr2=0x7ffb780c62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:44.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.390+0000 7ffb8062f640 1 --2- 172.21.15.116:0/597090556 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb78070dd0 0x7ffb780c62e0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7ffb6800c430 tx=0x7ffb6800c8f0 comp rx=0 tx=0).stop 2024-04-25T00:12:44.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.390+0000 7ffb8062f640 1 -- 172.21.15.116:0/597090556 shutdown_connections 2024-04-25T00:12:44.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.390+0000 7ffb8062f640 1 --2- 172.21.15.116:0/597090556 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ffb5c0678c0 0x7ffb5c069d80 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:44.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.390+0000 7ffb8062f640 1 --2- 172.21.15.116:0/597090556 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffb78148890 0x7ffb781426f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:44.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.390+0000 7ffb8062f640 1 --2- 172.21.15.116:0/597090556 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffb78070dd0 0x7ffb780c62e0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:44.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.390+0000 7ffb8062f640 1 --2- 172.21.15.116:0/597090556 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffb7806e890 0x7ffb781ec780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:44.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.390+0000 7ffb8062f640 1 -- 172.21.15.116:0/597090556 >> 172.21.15.116:0/597090556 conn(0x7ffb780c3b00 msgr2=0x7ffb7806d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:44.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.390+0000 7ffb8062f640 1 -- 172.21.15.116:0/597090556 shutdown_connections 2024-04-25T00:12:44.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:44.390+0000 7ffb8062f640 1 -- 172.21.15.116:0/597090556 wait complete. 2024-04-25T00:12:44.405 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:12:46.406 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:12:46.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.538+0000 7f6f12368640 1 Processor -- start 2024-04-25T00:12:46.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.538+0000 7f6f12368640 1 -- start start 2024-04-25T00:12:46.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.542+0000 7f6f12368640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6f0c06d5f0 0x7f6f0c156c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:46.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.542+0000 7f6f12368640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f0c0c9150 con 0x7f6f0c155810 2024-04-25T00:12:46.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.542+0000 7f6f12368640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f0c0c9430 con 0x7f6f0c152d80 2024-04-25T00:12:46.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.542+0000 7f6f12368640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f6f0c0c9d60 con 0x7f6f0c06d5f0 2024-04-25T00:12:46.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.542+0000 7f6f0b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6f0c06d5f0 0x7f6f0c156c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:46.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.542+0000 7f6f0b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6f0c06d5f0 0x7f6f0c156c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49256/0 (socket says 172.21.15.116:49256) 2024-04-25T00:12:46.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.542+0000 7f6f0b7fe640 1 -- 172.21.15.116:0/671915603 learned_addr learned my addr 172.21.15.116:0/671915603 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:12:46.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.542+0000 7f6f0b7fe640 1 -- 172.21.15.116:0/671915603 >> v1:172.21.15.174:6789/0 conn(0x7f6f0c152d80 legacy=0x7f6f0c06bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:46.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.542+0000 7f6f0b7fe640 1 -- 172.21.15.116:0/671915603 >> v1:172.21.15.116:6789/0 conn(0x7f6f0c155810 legacy=0x7f6f0c1571c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:46.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.542+0000 7f6f0b7fe640 1 -- 172.21.15.116:0/671915603 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f0c0c9710 con 0x7f6f0c06d5f0 2024-04-25T00:12:46.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.542+0000 7f6f0b7fe640 1 --2- 172.21.15.116:0/671915603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f0c06d5f0 0x7f6f0c156c80 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f6f00009870 tx=0x7f6f0001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=809ccf46eb674d58 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:46.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.542+0000 7f6f0affd640 1 -- 172.21.15.116:0/671915603 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f000aa620 con 0x7f6f0c06d5f0 2024-04-25T00:12:46.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.542+0000 7f6f0affd640 1 -- 172.21.15.116:0/671915603 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f000af070 con 0x7f6f0c06d5f0 2024-04-25T00:12:46.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.542+0000 7f6f0affd640 1 -- 172.21.15.116:0/671915603 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f000aa9e0 con 0x7f6f0c06d5f0 2024-04-25T00:12:46.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.546+0000 7f6f12368640 1 -- 172.21.15.116:0/671915603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f0c06d5f0 msgr2=0x7f6f0c156c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:46.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.546+0000 7f6f12368640 1 --2- 172.21.15.116:0/671915603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f0c06d5f0 0x7f6f0c156c80 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f6f00009870 tx=0x7f6f0001c4a0 comp rx=0 tx=0).stop 2024-04-25T00:12:46.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.546+0000 7f6f12368640 1 -- 172.21.15.116:0/671915603 shutdown_connections 2024-04-25T00:12:46.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.546+0000 7f6f12368640 1 --2- 172.21.15.116:0/671915603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f0c06d5f0 0x7f6f0c156c80 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:46.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.546+0000 7f6f12368640 1 -- 172.21.15.116:0/671915603 >> 172.21.15.116:0/671915603 conn(0x7f6f0c0acdd0 msgr2=0x7f6f0c0ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:46.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.546+0000 7f6f12368640 1 -- 172.21.15.116:0/671915603 shutdown_connections 2024-04-25T00:12:46.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.546+0000 7f6f12368640 1 -- 172.21.15.116:0/671915603 wait complete. 2024-04-25T00:12:46.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.546+0000 7f6f12368640 1 Processor -- start 2024-04-25T00:12:46.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.546+0000 7f6f12368640 1 -- start start 2024-04-25T00:12:46.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.546+0000 7f6f12368640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f0c06d5f0 0x7f6f0c143560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:46.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.546+0000 7f6f12368640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6f0c152d80 0x7f6f0c154d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:46.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.546+0000 7f6f12368640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6f0c155810 0x7f6f0c145430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:46.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.546+0000 7f6f12368640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6f0c153160 con 0x7f6f0c06d5f0 2024-04-25T00:12:46.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.546+0000 7f6f12368640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6f0c0cad80 con 0x7f6f0c155810 2024-04-25T00:12:46.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.546+0000 7f6f12368640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6f0c0abc70 con 0x7f6f0c152d80 2024-04-25T00:12:46.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.546+0000 7f6f0bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f0c06d5f0 0x7f6f0c143560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:46.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.546+0000 7f6f0bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f0c06d5f0 0x7f6f0c143560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49266/0 (socket says 172.21.15.116:49266) 2024-04-25T00:12:46.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.546+0000 7f6f0bfff640 1 -- 172.21.15.116:0/1114345098 learned_addr learned my addr 172.21.15.116:0/1114345098 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:12:46.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.546+0000 7f6f0b7fe640 1 --2- 172.21.15.116:0/1114345098 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6f0c152d80 0x7f6f0c154d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:46.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.546+0000 7f6f10905640 1 --2- 172.21.15.116:0/1114345098 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6f0c155810 0x7f6f0c145430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:46.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.546+0000 7f6f0bfff640 1 -- 172.21.15.116:0/1114345098 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6f0c152d80 msgr2=0x7f6f0c154d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:46.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.546+0000 7f6f0bfff640 1 --2- 172.21.15.116:0/1114345098 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6f0c152d80 0x7f6f0c154d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:46.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.546+0000 7f6f0bfff640 1 -- 172.21.15.116:0/1114345098 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6f0c155810 msgr2=0x7f6f0c145430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:46.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.546+0000 7f6f0bfff640 1 --2- 172.21.15.116:0/1114345098 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6f0c155810 0x7f6f0c145430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:46.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.546+0000 7f6f0bfff640 1 -- 172.21.15.116:0/1114345098 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f0c1f2b20 con 0x7f6f0c06d5f0 2024-04-25T00:12:46.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.546+0000 7f6f0bfff640 1 --2- 172.21.15.116:0/1114345098 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f0c06d5f0 0x7f6f0c143560 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f6efc00c370 tx=0x7f6efc00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:46.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.546+0000 7f6f08ff9640 1 -- 172.21.15.116:0/1114345098 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6efc0052d0 con 0x7f6f0c06d5f0 2024-04-25T00:12:46.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.546+0000 7f6f08ff9640 1 -- 172.21.15.116:0/1114345098 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6efc005470 con 0x7f6f0c06d5f0 2024-04-25T00:12:46.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.546+0000 7f6f12368640 1 -- 172.21.15.116:0/1114345098 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f0c1f1d60 con 0x7f6f0c06d5f0 2024-04-25T00:12:46.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.546+0000 7f6f08ff9640 1 -- 172.21.15.116:0/1114345098 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6efc007260 con 0x7f6f0c06d5f0 2024-04-25T00:12:46.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.546+0000 7f6f12368640 1 -- 172.21.15.116:0/1114345098 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f0c1f2320 con 0x7f6f0c06d5f0 2024-04-25T00:12:46.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.550+0000 7f6f12368640 1 -- 172.21.15.116:0/1114345098 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f0c068350 con 0x7f6f0c06d5f0 2024-04-25T00:12:46.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.550+0000 7f6f08ff9640 1 -- 172.21.15.116:0/1114345098 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6efc007400 con 0x7f6f0c06d5f0 2024-04-25T00:12:46.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.550+0000 7f6f08ff9640 1 --2- 172.21.15.116:0/1114345098 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6ef80678f0 0x7f6ef8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:46.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.550+0000 7f6f08ff9640 1 -- 172.21.15.116:0/1114345098 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 8042+0+0 (secure 0 0 0) 0x7f6efc085a00 con 0x7f6f0c06d5f0 2024-04-25T00:12:46.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.550+0000 7f6f0b7fe640 1 --2- 172.21.15.116:0/1114345098 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6ef80678f0 0x7f6ef8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:46.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.550+0000 7f6f0b7fe640 1 --2- 172.21.15.116:0/1114345098 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6ef80678f0 0x7f6ef8069db0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f6f000ae000 tx=0x7f6f000a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:46.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.554+0000 7f6f08ff9640 1 -- 172.21.15.116:0/1114345098 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f6efc055c20 con 0x7f6f0c06d5f0 2024-04-25T00:12:46.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.750+0000 7f6f12368640 1 -- 172.21.15.116:0/1114345098 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6f0c0c8a30 con 0x7f6f0c06d5f0 2024-04-25T00:12:46.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.754+0000 7f6f08ff9640 1 -- 172.21.15.116:0/1114345098 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) ==== 94+0+5664 (secure 0 0 0) 0x7f6efc055520 con 0x7f6f0c06d5f0 2024-04-25T00:12:46.754 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:12:46.755 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":88,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:12:45.533679+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":88,"state":"up:replay","state_seq":1,"addr":"172.21.15.116:6833/365773583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":365773583},{"type":"v1","addr":"172.21.15.116:6833","nonce":365773583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:12:23.643548+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5461},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5461":{"gid":5461,"name":"b","rank":0,"incarnation":83,"state":"up:active","state_seq":30,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5461,"qdb_cluster":[5461]},"id":2}]} 2024-04-25T00:12:46.755 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 88 2024-04-25T00:12:46.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.758+0000 7f6f12368640 1 -- 172.21.15.116:0/1114345098 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6ef80678f0 msgr2=0x7f6ef8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:46.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.758+0000 7f6f12368640 1 --2- 172.21.15.116:0/1114345098 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6ef80678f0 0x7f6ef8069db0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f6f000ae000 tx=0x7f6f000a7000 comp rx=0 tx=0).stop 2024-04-25T00:12:46.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.758+0000 7f6f12368640 1 -- 172.21.15.116:0/1114345098 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f0c06d5f0 msgr2=0x7f6f0c143560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:46.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.758+0000 7f6f12368640 1 --2- 172.21.15.116:0/1114345098 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f0c06d5f0 0x7f6f0c143560 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f6efc00c370 tx=0x7f6efc00c830 comp rx=0 tx=0).stop 2024-04-25T00:12:46.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.758+0000 7f6f12368640 1 -- 172.21.15.116:0/1114345098 shutdown_connections 2024-04-25T00:12:46.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.758+0000 7f6f12368640 1 --2- 172.21.15.116:0/1114345098 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6ef80678f0 0x7f6ef8069db0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:46.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.758+0000 7f6f12368640 1 --2- 172.21.15.116:0/1114345098 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6f0c155810 0x7f6f0c145430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:46.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.758+0000 7f6f12368640 1 --2- 172.21.15.116:0/1114345098 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6f0c152d80 0x7f6f0c154d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:46.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.758+0000 7f6f12368640 1 --2- 172.21.15.116:0/1114345098 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f0c06d5f0 0x7f6f0c143560 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:46.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.758+0000 7f6f12368640 1 -- 172.21.15.116:0/1114345098 >> 172.21.15.116:0/1114345098 conn(0x7f6f0c0acdd0 msgr2=0x7f6f0c160510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:46.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.758+0000 7f6f12368640 1 -- 172.21.15.116:0/1114345098 shutdown_connections 2024-04-25T00:12:46.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.758+0000 7f6f12368640 1 -- 172.21.15.116:0/1114345098 wait complete. 2024-04-25T00:12:46.775 INFO:tasks.mds_thrash.fs.[a]:mds.a down, removed from mdsmap 2024-04-25T00:12:46.775 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-04-25T00:12:46.775 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:12:46.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.894+0000 7f2bcb8fa640 1 Processor -- start 2024-04-25T00:12:46.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.894+0000 7f2bcb8fa640 1 -- start start 2024-04-25T00:12:46.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.898+0000 7f2bcb8fa640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2bc414c720 0x7f2bc41563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:46.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.898+0000 7f2bcb8fa640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f2bc40a94a0 0x7f2bc40c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:46.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.898+0000 7f2bcb8fa640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2bc40c2f50 0x7f2bc4071680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:46.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.898+0000 7f2bcb8fa640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f2bc4153460 con 0x7f2bc40c2f50 2024-04-25T00:12:46.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.898+0000 7f2bcb8fa640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f2bc4071220 con 0x7f2bc40a94a0 2024-04-25T00:12:46.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.898+0000 7f2bcb8fa640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f2bc4152420 con 0x7f2bc414c720 2024-04-25T00:12:46.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.898+0000 7f2bc9e97640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2bc40c2f50 0x7f2bc4071680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:46.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.898+0000 7f2bc9e97640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2bc40c2f50 0x7f2bc4071680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49268/0 (socket says 172.21.15.116:49268) 2024-04-25T00:12:46.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.898+0000 7f2bc9e97640 1 -- 172.21.15.116:0/3919754076 learned_addr learned my addr 172.21.15.116:0/3919754076 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:12:46.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.898+0000 7f2bc9696640 1 --2- 172.21.15.116:0/3919754076 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2bc414c720 0x7f2bc41563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:46.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.898+0000 7f2bc8e95640 1 --2- 172.21.15.116:0/3919754076 >> v2:172.21.15.174:3300/0 conn(0x7f2bc40a94a0 0x7f2bc40c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:46.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.898+0000 7f2bc9e97640 1 -- 172.21.15.116:0/3919754076 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2bc414c720 msgr2=0x7f2bc41563d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:12:46.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.898+0000 7f2bc9e97640 1 --2- 172.21.15.116:0/3919754076 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2bc414c720 0x7f2bc41563d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:46.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.898+0000 7f2bc9e97640 1 -- 172.21.15.116:0/3919754076 >> v2:172.21.15.174:3300/0 conn(0x7f2bc40a94a0 msgr2=0x7f2bc40c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:12:46.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.898+0000 7f2bc9e97640 1 --2- 172.21.15.116:0/3919754076 >> v2:172.21.15.174:3300/0 conn(0x7f2bc40a94a0 0x7f2bc40c2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:46.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.898+0000 7f2bc9e97640 1 -- 172.21.15.116:0/3919754076 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bc414cb00 con 0x7f2bc40c2f50 2024-04-25T00:12:46.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.898+0000 7f2bc9e97640 1 --2- 172.21.15.116:0/3919754076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bc40c2f50 0x7f2bc4071680 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f2bc40c4bc0 tx=0x7f2bbc01c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26b8937da4d485cb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:46.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.898+0000 7f2bbbfff640 1 -- 172.21.15.116:0/3919754076 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc0a57b0 con 0x7f2bc40c2f50 2024-04-25T00:12:46.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.898+0000 7f2bbbfff640 1 -- 172.21.15.116:0/3919754076 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2bbc0a5950 con 0x7f2bc40c2f50 2024-04-25T00:12:46.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.898+0000 7f2bbbfff640 1 -- 172.21.15.116:0/3919754076 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc0a5d20 con 0x7f2bc40c2f50 2024-04-25T00:12:46.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.902+0000 7f2bcb8fa640 1 -- 172.21.15.116:0/3919754076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bc40c2f50 msgr2=0x7f2bc4071680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:46.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.902+0000 7f2bcb8fa640 1 --2- 172.21.15.116:0/3919754076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bc40c2f50 0x7f2bc4071680 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f2bc40c4bc0 tx=0x7f2bbc01c930 comp rx=0 tx=0).stop 2024-04-25T00:12:46.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.902+0000 7f2bcb8fa640 1 -- 172.21.15.116:0/3919754076 shutdown_connections 2024-04-25T00:12:46.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.902+0000 7f2bcb8fa640 1 --2- 172.21.15.116:0/3919754076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bc40c2f50 0x7f2bc4071680 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:46.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.902+0000 7f2bcb8fa640 1 --2- 172.21.15.116:0/3919754076 >> v2:172.21.15.174:3300/0 conn(0x7f2bc40a94a0 0x7f2bc40c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:46.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.902+0000 7f2bcb8fa640 1 --2- 172.21.15.116:0/3919754076 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2bc414c720 0x7f2bc41563d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:46.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.902+0000 7f2bcb8fa640 1 -- 172.21.15.116:0/3919754076 >> 172.21.15.116:0/3919754076 conn(0x7f2bc40c3b00 msgr2=0x7f2bc40c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:46.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.902+0000 7f2bcb8fa640 1 -- 172.21.15.116:0/3919754076 shutdown_connections 2024-04-25T00:12:46.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.902+0000 7f2bcb8fa640 1 -- 172.21.15.116:0/3919754076 wait complete. 2024-04-25T00:12:46.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.902+0000 7f2bcb8fa640 1 Processor -- start 2024-04-25T00:12:46.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.902+0000 7f2bcb8fa640 1 -- start start 2024-04-25T00:12:46.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.902+0000 7f2bcb8fa640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2bc40a94a0 0x7f2bc413ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:46.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.902+0000 7f2bcb8fa640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bc40c2f50 0x7f2bc4143a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:46.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.902+0000 7f2bcb8fa640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2bc414c720 0x7f2bc41397f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:46.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.902+0000 7f2bcb8fa640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc4071fd0 con 0x7f2bc40c2f50 2024-04-25T00:12:46.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.902+0000 7f2bcb8fa640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc4152f40 con 0x7f2bc414c720 2024-04-25T00:12:46.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.902+0000 7f2bcb8fa640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f2bc4072150 con 0x7f2bc40a94a0 2024-04-25T00:12:46.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.902+0000 7f2bc8e95640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bc40c2f50 0x7f2bc4143a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:46.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.902+0000 7f2bc8e95640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bc40c2f50 0x7f2bc4143a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49280/0 (socket says 172.21.15.116:49280) 2024-04-25T00:12:46.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.902+0000 7f2bc8e95640 1 -- 172.21.15.116:0/546446109 learned_addr learned my addr 172.21.15.116:0/546446109 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:12:46.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.902+0000 7f2bc9696640 1 --2- 172.21.15.116:0/546446109 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2bc40a94a0 0x7f2bc413ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:46.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.902+0000 7f2bc9e97640 1 --2- 172.21.15.116:0/546446109 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2bc414c720 0x7f2bc41397f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:46.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.902+0000 7f2bc8e95640 1 -- 172.21.15.116:0/546446109 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2bc40a94a0 msgr2=0x7f2bc413ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:46.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.902+0000 7f2bc8e95640 1 --2- 172.21.15.116:0/546446109 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2bc40a94a0 0x7f2bc413ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:46.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.902+0000 7f2bc8e95640 1 -- 172.21.15.116:0/546446109 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2bc414c720 msgr2=0x7f2bc41397f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:46.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.902+0000 7f2bc8e95640 1 --2- 172.21.15.116:0/546446109 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2bc414c720 0x7f2bc41397f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:46.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.902+0000 7f2bc8e95640 1 -- 172.21.15.116:0/546446109 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bc4138750 con 0x7f2bc40c2f50 2024-04-25T00:12:46.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.902+0000 7f2bc8e95640 1 --2- 172.21.15.116:0/546446109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bc40c2f50 0x7f2bc4143a20 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f2bb400d800 tx=0x7f2bb400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:46.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.902+0000 7f2bb9ffb640 1 -- 172.21.15.116:0/546446109 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bb400b6c0 con 0x7f2bc40c2f50 2024-04-25T00:12:46.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.902+0000 7f2bb9ffb640 1 -- 172.21.15.116:0/546446109 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2bb4019070 con 0x7f2bc40c2f50 2024-04-25T00:12:46.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.902+0000 7f2bcb8fa640 1 -- 172.21.15.116:0/546446109 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bc4138a30 con 0x7f2bc40c2f50 2024-04-25T00:12:46.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.902+0000 7f2bb9ffb640 1 -- 172.21.15.116:0/546446109 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bb400ba60 con 0x7f2bc40c2f50 2024-04-25T00:12:46.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.902+0000 7f2bcb8fa640 1 -- 172.21.15.116:0/546446109 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bc4138ff0 con 0x7f2bc40c2f50 2024-04-25T00:12:46.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.902+0000 7f2bcb8fa640 1 -- 172.21.15.116:0/546446109 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2bc4068350 con 0x7f2bc40c2f50 2024-04-25T00:12:46.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.906+0000 7f2bb9ffb640 1 -- 172.21.15.116:0/546446109 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2bb400bc00 con 0x7f2bc40c2f50 2024-04-25T00:12:46.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.906+0000 7f2bb9ffb640 1 --2- 172.21.15.116:0/546446109 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2ba80678c0 0x7f2ba8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:46.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.906+0000 7f2bb9ffb640 1 -- 172.21.15.116:0/546446109 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 8042+0+0 (secure 0 0 0) 0x7f2bb4086a00 con 0x7f2bc40c2f50 2024-04-25T00:12:46.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.906+0000 7f2bc9696640 1 --2- 172.21.15.116:0/546446109 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2ba80678c0 0x7f2ba8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:46.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.906+0000 7f2bc9696640 1 --2- 172.21.15.116:0/546446109 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2ba80678c0 0x7f2ba8069d80 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f2bc000d8d0 tx=0x7f2bc0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:46.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:46.910+0000 7f2bb9ffb640 1 -- 172.21.15.116:0/546446109 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f2bb4056cb0 con 0x7f2bc40c2f50 2024-04-25T00:12:47.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:47.105+0000 7f2bcb8fa640 1 -- 172.21.15.116:0/546446109 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2bc40c3640 con 0x7f2bc40c2f50 2024-04-25T00:12:47.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:47.109+0000 7f2bb9ffb640 1 -- 172.21.15.116:0/546446109 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) ==== 94+0+5667 (secure 0 0 0) 0x7f2bb40565b0 con 0x7f2bc40c2f50 2024-04-25T00:12:47.108 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:12:47.108 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":89,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:12:45.853225+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":88,"state":"up:reconnect","state_seq":5,"addr":"172.21.15.116:6833/365773583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":365773583},{"type":"v1","addr":"172.21.15.116:6833","nonce":365773583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:12:23.643548+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5461},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5461":{"gid":5461,"name":"b","rank":0,"incarnation":83,"state":"up:active","state_seq":30,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5461,"qdb_cluster":[5461]},"id":2}]} 2024-04-25T00:12:47.109 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 89 2024-04-25T00:12:47.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:47.109+0000 7f2bcb8fa640 1 -- 172.21.15.116:0/546446109 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2ba80678c0 msgr2=0x7f2ba8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:47.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:47.109+0000 7f2bcb8fa640 1 --2- 172.21.15.116:0/546446109 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2ba80678c0 0x7f2ba8069d80 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f2bc000d8d0 tx=0x7f2bc0008040 comp rx=0 tx=0).stop 2024-04-25T00:12:47.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:47.109+0000 7f2bcb8fa640 1 -- 172.21.15.116:0/546446109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bc40c2f50 msgr2=0x7f2bc4143a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:47.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:47.109+0000 7f2bcb8fa640 1 --2- 172.21.15.116:0/546446109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bc40c2f50 0x7f2bc4143a20 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f2bb400d800 tx=0x7f2bb400dcc0 comp rx=0 tx=0).stop 2024-04-25T00:12:47.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:47.109+0000 7f2bcb8fa640 1 -- 172.21.15.116:0/546446109 shutdown_connections 2024-04-25T00:12:47.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:47.109+0000 7f2bcb8fa640 1 --2- 172.21.15.116:0/546446109 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2ba80678c0 0x7f2ba8069d80 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:47.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:47.109+0000 7f2bcb8fa640 1 --2- 172.21.15.116:0/546446109 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2bc414c720 0x7f2bc41397f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:47.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:47.109+0000 7f2bcb8fa640 1 --2- 172.21.15.116:0/546446109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bc40c2f50 0x7f2bc4143a20 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:47.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:47.109+0000 7f2bcb8fa640 1 --2- 172.21.15.116:0/546446109 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2bc40a94a0 0x7f2bc413ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:47.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:47.109+0000 7f2bcb8fa640 1 -- 172.21.15.116:0/546446109 >> 172.21.15.116:0/546446109 conn(0x7f2bc40c3b00 msgr2=0x7f2bc40671c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:47.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:47.109+0000 7f2bcb8fa640 1 -- 172.21.15.116:0/546446109 shutdown_connections 2024-04-25T00:12:47.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:47.109+0000 7f2bcb8fa640 1 -- 172.21.15.116:0/546446109 wait complete. 2024-04-25T00:12:47.128 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 89, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:04.678737+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 89, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6495": { "addr": "172.21.15.116:6833/365773583", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6832", "nonce": 365773583, "type": "v2" }, { "addr": "172.21.15.116:6833", "nonce": 365773583, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6495, "incarnation": 88, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:reconnect", "state_seq": 5 } }, "last_failure": 0, "last_failure_osd_epoch": 44, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T00:12:45.853225+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6495 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:18.825887+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 86, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5461": { "addr": "172.21.15.174:6837/3398690896", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6836", "nonce": 3398690896, "type": "v2" }, { "addr": "172.21.15.174:6837", "nonce": 3398690896, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5461, "incarnation": 83, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 30 } }, "last_failure": 0, "last_failure_osd_epoch": 43, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-04-25T00:12:23.643548+0000", "qdb_cluster": [ 5461 ], "qdb_leader": 5461, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5461 } } } ], "standbys": [ { "addr": "172.21.15.174:6835/4215220222", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6834", "nonce": 4215220222, "type": "v2" }, { "addr": "172.21.15.174:6835", "nonce": 4215220222, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 82, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5273, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-25T00:12:49.128 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:12:49.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.269+0000 7fbb59ad2640 1 Processor -- start 2024-04-25T00:12:49.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.269+0000 7fbb59ad2640 1 -- start start 2024-04-25T00:12:49.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.273+0000 7fbb59ad2640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fbb54150d40 0x7fbb54154170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:49.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.273+0000 7fbb59ad2640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb540c4c20 0x7fbb540c6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:49.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.273+0000 7fbb59ad2640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbb54070770 0x7fbb540ab200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:49.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.273+0000 7fbb59ad2640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fbb54071280 con 0x7fbb54070770 2024-04-25T00:12:49.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.273+0000 7fbb59ad2640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fbb5406ff30 con 0x7fbb54150d40 2024-04-25T00:12:49.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.273+0000 7fbb59ad2640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fbb54070180 con 0x7fbb540c4c20 2024-04-25T00:12:49.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.273+0000 7fbb53fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbb54070770 0x7fbb540ab200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:49.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.273+0000 7fbb53fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbb54070770 0x7fbb540ab200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49290/0 (socket says 172.21.15.116:49290) 2024-04-25T00:12:49.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.273+0000 7fbb53fff640 1 -- 172.21.15.116:0/1970146944 learned_addr learned my addr 172.21.15.116:0/1970146944 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:12:49.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.273+0000 7fbb537fe640 1 --2- 172.21.15.116:0/1970146944 >> v2:172.21.15.174:3300/0 conn(0x7fbb54150d40 0x7fbb54154170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:49.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.273+0000 7fbb52ffd640 1 --2- 172.21.15.116:0/1970146944 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb540c4c20 0x7fbb540c6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:49.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.273+0000 7fbb53fff640 1 -- 172.21.15.116:0/1970146944 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb540c4c20 msgr2=0x7fbb540c6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:12:49.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.273+0000 7fbb53fff640 1 --2- 172.21.15.116:0/1970146944 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb540c4c20 0x7fbb540c6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:49.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.273+0000 7fbb53fff640 1 -- 172.21.15.116:0/1970146944 >> v2:172.21.15.174:3300/0 conn(0x7fbb54150d40 msgr2=0x7fbb54154170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:12:49.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.273+0000 7fbb53fff640 1 --2- 172.21.15.116:0/1970146944 >> v2:172.21.15.174:3300/0 conn(0x7fbb54150d40 0x7fbb54154170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:49.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.273+0000 7fbb53fff640 1 -- 172.21.15.116:0/1970146944 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb54070460 con 0x7fbb54070770 2024-04-25T00:12:49.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.273+0000 7fbb53fff640 1 --2- 172.21.15.116:0/1970146944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb54070770 0x7fbb540ab200 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c009e50 tx=0x7fbb4c01c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d05878e27f723f2c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:49.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.273+0000 7fbb527fc640 1 -- 172.21.15.116:0/1970146944 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb4c0a56f0 con 0x7fbb54070770 2024-04-25T00:12:49.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.273+0000 7fbb527fc640 1 -- 172.21.15.116:0/1970146944 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb4c0a5890 con 0x7fbb54070770 2024-04-25T00:12:49.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.273+0000 7fbb527fc640 1 -- 172.21.15.116:0/1970146944 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb4c0a5c60 con 0x7fbb54070770 2024-04-25T00:12:49.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.277+0000 7fbb59ad2640 1 -- 172.21.15.116:0/1970146944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb54070770 msgr2=0x7fbb540ab200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:49.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.277+0000 7fbb59ad2640 1 --2- 172.21.15.116:0/1970146944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb54070770 0x7fbb540ab200 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c009e50 tx=0x7fbb4c01c960 comp rx=0 tx=0).stop 2024-04-25T00:12:49.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.277+0000 7fbb59ad2640 1 -- 172.21.15.116:0/1970146944 shutdown_connections 2024-04-25T00:12:49.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.277+0000 7fbb59ad2640 1 --2- 172.21.15.116:0/1970146944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb54070770 0x7fbb540ab200 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:49.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.277+0000 7fbb59ad2640 1 --2- 172.21.15.116:0/1970146944 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb540c4c20 0x7fbb540c6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:49.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.277+0000 7fbb59ad2640 1 --2- 172.21.15.116:0/1970146944 >> v2:172.21.15.174:3300/0 conn(0x7fbb54150d40 0x7fbb54154170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:49.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.277+0000 7fbb59ad2640 1 -- 172.21.15.116:0/1970146944 >> 172.21.15.116:0/1970146944 conn(0x7fbb5406f0c0 msgr2=0x7fbb5406f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:49.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.277+0000 7fbb59ad2640 1 -- 172.21.15.116:0/1970146944 shutdown_connections 2024-04-25T00:12:49.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.277+0000 7fbb59ad2640 1 -- 172.21.15.116:0/1970146944 wait complete. 2024-04-25T00:12:49.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.277+0000 7fbb59ad2640 1 Processor -- start 2024-04-25T00:12:49.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.277+0000 7fbb59ad2640 1 -- start start 2024-04-25T00:12:49.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.277+0000 7fbb59ad2640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb54070770 0x7fbb54145350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:49.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.277+0000 7fbb59ad2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb540c4c20 0x7fbb541f1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:49.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.277+0000 7fbb59ad2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb54150d40 0x7fbb541f1c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:49.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.277+0000 7fbb59ad2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbb540c74e0 con 0x7fbb540c4c20 2024-04-25T00:12:49.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.277+0000 7fbb59ad2640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbb540c7360 con 0x7fbb54150d40 2024-04-25T00:12:49.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.277+0000 7fbb59ad2640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fbb54151120 con 0x7fbb54070770 2024-04-25T00:12:49.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.277+0000 7fbb52ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb540c4c20 0x7fbb541f1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:49.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.277+0000 7fbb52ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb540c4c20 0x7fbb541f1720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49300/0 (socket says 172.21.15.116:49300) 2024-04-25T00:12:49.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.277+0000 7fbb52ffd640 1 -- 172.21.15.116:0/3944126161 learned_addr learned my addr 172.21.15.116:0/3944126161 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:12:49.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.277+0000 7fbb537fe640 1 --2- 172.21.15.116:0/3944126161 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb54070770 0x7fbb54145350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:49.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.277+0000 7fbb53fff640 1 --2- 172.21.15.116:0/3944126161 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb54150d40 0x7fbb541f1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:49.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.277+0000 7fbb52ffd640 1 -- 172.21.15.116:0/3944126161 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb54070770 msgr2=0x7fbb54145350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:49.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.277+0000 7fbb52ffd640 1 --2- 172.21.15.116:0/3944126161 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb54070770 0x7fbb54145350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:49.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.277+0000 7fbb52ffd640 1 -- 172.21.15.116:0/3944126161 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb54150d40 msgr2=0x7fbb541f1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:49.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.277+0000 7fbb52ffd640 1 --2- 172.21.15.116:0/3944126161 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb54150d40 0x7fbb541f1c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:49.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.277+0000 7fbb52ffd640 1 -- 172.21.15.116:0/3944126161 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb54152100 con 0x7fbb540c4c20 2024-04-25T00:12:49.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.277+0000 7fbb52ffd640 1 --2- 172.21.15.116:0/3944126161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb540c4c20 0x7fbb541f1720 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fbb4800d5f0 tx=0x7fbb4800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:49.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.277+0000 7fbb3bfff640 1 -- 172.21.15.116:0/3944126161 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb480125f0 con 0x7fbb540c4c20 2024-04-25T00:12:49.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.277+0000 7fbb3bfff640 1 -- 172.21.15.116:0/3944126161 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb48019070 con 0x7fbb540c4c20 2024-04-25T00:12:49.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.277+0000 7fbb3bfff640 1 -- 172.21.15.116:0/3944126161 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb480129d0 con 0x7fbb540c4c20 2024-04-25T00:12:49.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.281+0000 7fbb59ad2640 1 -- 172.21.15.116:0/3944126161 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb54152830 con 0x7fbb540c4c20 2024-04-25T00:12:49.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.281+0000 7fbb59ad2640 1 -- 172.21.15.116:0/3944126161 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb54152df0 con 0x7fbb540c4c20 2024-04-25T00:12:49.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.281+0000 7fbb59ad2640 1 -- 172.21.15.116:0/3944126161 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb54068a60 con 0x7fbb540c4c20 2024-04-25T00:12:49.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.281+0000 7fbb3bfff640 1 -- 172.21.15.116:0/3944126161 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbb48012b70 con 0x7fbb540c4c20 2024-04-25T00:12:49.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.281+0000 7fbb3bfff640 1 --2- 172.21.15.116:0/3944126161 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbb400678c0 0x7fbb40069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:49.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.281+0000 7fbb3bfff640 1 -- 172.21.15.116:0/3944126161 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 8042+0+0 (secure 0 0 0) 0x7fbb480869f0 con 0x7fbb540c4c20 2024-04-25T00:12:49.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.281+0000 7fbb537fe640 1 --2- 172.21.15.116:0/3944126161 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbb400678c0 0x7fbb40069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:49.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.281+0000 7fbb537fe640 1 --2- 172.21.15.116:0/3944126161 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbb400678c0 0x7fbb40069d80 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fbb4400b950 tx=0x7fbb44008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:49.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.285+0000 7fbb3bfff640 1 -- 172.21.15.116:0/3944126161 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fbb48056ca0 con 0x7fbb540c4c20 2024-04-25T00:12:49.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.501+0000 7fbb59ad2640 1 -- 172.21.15.116:0/3944126161 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbb540c5cf0 con 0x7fbb540c4c20 2024-04-25T00:12:49.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.505+0000 7fbb3bfff640 1 -- 172.21.15.116:0/3944126161 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) ==== 94+0+5671 (secure 0 0 0) 0x7fbb480565a0 con 0x7fbb540c4c20 2024-04-25T00:12:49.504 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:12:49.504 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:12:48.771611+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":88,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6833/365773583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":365773583},{"type":"v1","addr":"172.21.15.116:6833","nonce":365773583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":1},{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:12:23.643548+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5461},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5461":{"gid":5461,"name":"b","rank":0,"incarnation":83,"state":"up:active","state_seq":30,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5461,"qdb_cluster":[5461]},"id":2}]} 2024-04-25T00:12:49.504 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 91 2024-04-25T00:12:49.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.505+0000 7fbb59ad2640 1 -- 172.21.15.116:0/3944126161 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbb400678c0 msgr2=0x7fbb40069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:49.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.505+0000 7fbb59ad2640 1 --2- 172.21.15.116:0/3944126161 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbb400678c0 0x7fbb40069d80 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fbb4400b950 tx=0x7fbb44008040 comp rx=0 tx=0).stop 2024-04-25T00:12:49.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.505+0000 7fbb59ad2640 1 -- 172.21.15.116:0/3944126161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb540c4c20 msgr2=0x7fbb541f1720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:49.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.505+0000 7fbb59ad2640 1 --2- 172.21.15.116:0/3944126161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb540c4c20 0x7fbb541f1720 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fbb4800d5f0 tx=0x7fbb4800dab0 comp rx=0 tx=0).stop 2024-04-25T00:12:49.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.505+0000 7fbb59ad2640 1 -- 172.21.15.116:0/3944126161 shutdown_connections 2024-04-25T00:12:49.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.505+0000 7fbb59ad2640 1 --2- 172.21.15.116:0/3944126161 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbb400678c0 0x7fbb40069d80 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:49.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.505+0000 7fbb59ad2640 1 --2- 172.21.15.116:0/3944126161 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb54150d40 0x7fbb541f1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:49.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.505+0000 7fbb59ad2640 1 --2- 172.21.15.116:0/3944126161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb540c4c20 0x7fbb541f1720 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:49.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.505+0000 7fbb59ad2640 1 --2- 172.21.15.116:0/3944126161 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb54070770 0x7fbb54145350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:49.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.505+0000 7fbb59ad2640 1 -- 172.21.15.116:0/3944126161 >> 172.21.15.116:0/3944126161 conn(0x7fbb5406f0c0 msgr2=0x7fbb540a9630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:49.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.505+0000 7fbb59ad2640 1 -- 172.21.15.116:0/3944126161 shutdown_connections 2024-04-25T00:12:49.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.505+0000 7fbb59ad2640 1 -- 172.21.15.116:0/3944126161 wait complete. 2024-04-25T00:12:49.525 INFO:tasks.mds_thrash.fs.[a]:mds.c has gained rank=0, replacing gid=6114 2024-04-25T00:12:49.526 INFO:tasks.mds_thrash.fs.[a]:waiting for 0 secs before reviving mds.a 2024-04-25T00:12:49.526 INFO:tasks.mds_thrash.fs.[a]:reviving mds.a 2024-04-25T00:12:49.526 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-25T00:12:49.526 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-25T00:12:49.528 INFO:tasks.ceph.mds.a:Started 2024-04-25T00:12:49.528 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:12:49.596 INFO:tasks.ceph.mds.a.smithi116.stdout:starting mds.a at 2024-04-25T00:12:49.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.649+0000 7f50da7d6640 1 Processor -- start 2024-04-25T00:12:49.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.649+0000 7f50da7d6640 1 -- start start 2024-04-25T00:12:49.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.649+0000 7f50da7d6640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50d4152d80 0x7f50d406bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:49.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.649+0000 7f50da7d6640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f50d406c290 0x7f50d4157390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:49.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.649+0000 7f50da7d6640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50d40c9150 con 0x7f50d406d5f0 2024-04-25T00:12:49.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.649+0000 7f50da7d6640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f50d40c9d60 con 0x7f50d406c290 2024-04-25T00:12:49.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.649+0000 7f50da7d6640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f50d40c9430 con 0x7f50d4152d80 2024-04-25T00:12:49.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.649+0000 7f50cbfff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f50d406d5f0 0x7f50d4156c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:59674/0 (socket says 172.21.15.116:59674) 2024-04-25T00:12:49.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.649+0000 7f50cbfff640 1 -- 172.21.15.116:0/383358901 learned_addr learned my addr 172.21.15.116:0/383358901 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:12:49.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.649+0000 7f50d8d73640 1 --2- 172.21.15.116:0/383358901 >> v2:172.21.15.174:3300/0 conn(0x7f50d406c290 0x7f50d4157390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:49.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.649+0000 7f50d3fff640 1 --2- 172.21.15.116:0/383358901 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50d4152d80 0x7f50d406bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:49.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.649+0000 7f50d37fe640 1 -- 172.21.15.116:0/383358901 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3574354023 0 0) 0x7f50d40c9150 con 0x7f50d406d5f0 2024-04-25T00:12:49.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.649+0000 7f50d37fe640 1 -- 172.21.15.116:0/383358901 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f50c0003530 con 0x7f50d406d5f0 2024-04-25T00:12:49.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.649+0000 7f50d37fe640 1 -- 172.21.15.116:0/383358901 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 877445361 0 0) 0x7f50c0003530 con 0x7f50d406d5f0 2024-04-25T00:12:49.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.649+0000 7f50d37fe640 1 -- 172.21.15.116:0/383358901 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50d4152d80 msgr2=0x7f50d406bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:12:49.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.649+0000 7f50d37fe640 1 --2- 172.21.15.116:0/383358901 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50d4152d80 0x7f50d406bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:49.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.649+0000 7f50d37fe640 1 -- 172.21.15.116:0/383358901 >> v2:172.21.15.174:3300/0 conn(0x7f50d406c290 msgr2=0x7f50d4157390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:12:49.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.649+0000 7f50d37fe640 1 --2- 172.21.15.116:0/383358901 >> v2:172.21.15.174:3300/0 conn(0x7f50d406c290 0x7f50d4157390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:49.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.649+0000 7f50d37fe640 1 -- 172.21.15.116:0/383358901 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50d40c9710 con 0x7f50d406d5f0 2024-04-25T00:12:49.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.649+0000 7f50d37fe640 1 -- 172.21.15.116:0/383358901 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f50bc001680 con 0x7f50d406d5f0 2024-04-25T00:12:49.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.649+0000 7f50d37fe640 1 -- 172.21.15.116:0/383358901 >> v1:172.21.15.116:6789/0 conn(0x7f50d406d5f0 legacy=0x7f50d4156c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:49.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.649+0000 7f50d37fe640 1 --2- 172.21.15.116:0/383358901 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50c00039c0 0x7f50c0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:49.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.649+0000 7f50d37fe640 1 --2- 172.21.15.116:0/383358901 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50c0004810 0x7f50c0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:49.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.649+0000 7f50d37fe640 1 --2- 172.21.15.116:0/383358901 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50d4152d80 0x7f50c00051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:49.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.649+0000 7f50d37fe640 1 -- 172.21.15.116:0/383358901 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f50d40c9d60 con 0x7f50c0004810 2024-04-25T00:12:49.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.649+0000 7f50d37fe640 1 -- 172.21.15.116:0/383358901 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f50d40c9430 con 0x7f50c00039c0 2024-04-25T00:12:49.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.649+0000 7f50d37fe640 1 -- 172.21.15.116:0/383358901 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f50c000b9d0 con 0x7f50d4152d80 2024-04-25T00:12:49.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.649+0000 7f50cbfff640 1 --2- 172.21.15.116:0/383358901 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50c0004810 0x7f50c0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:49.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.649+0000 7f50d3fff640 1 --2- 172.21.15.116:0/383358901 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50c00039c0 0x7f50c0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:49.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.649+0000 7f50d8d73640 1 --2- 172.21.15.116:0/383358901 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50d4152d80 0x7f50c00051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:49.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.649+0000 7f50cbfff640 1 -- 172.21.15.116:0/383358901 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50d4152d80 msgr2=0x7f50c00051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:12:49.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.649+0000 7f50cbfff640 1 --2- 172.21.15.116:0/383358901 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50d4152d80 0x7f50c00051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:49.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.649+0000 7f50cbfff640 1 -- 172.21.15.116:0/383358901 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50c00039c0 msgr2=0x7f50c0003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:12:49.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.649+0000 7f50cbfff640 1 --2- 172.21.15.116:0/383358901 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50c00039c0 0x7f50c0003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:49.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.649+0000 7f50cbfff640 1 -- 172.21.15.116:0/383358901 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50c0003530 con 0x7f50c0004810 2024-04-25T00:12:49.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.653+0000 7f50cbfff640 1 --2- 172.21.15.116:0/383358901 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50c0004810 0x7f50c0004c90 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f50bc00c240 tx=0x7f50bc01cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a880cb8b9d65f2db server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:49.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.653+0000 7f50d37fe640 1 -- 172.21.15.116:0/383358901 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50bc007d40 con 0x7f50c0004810 2024-04-25T00:12:49.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.653+0000 7f50d37fe640 1 -- 172.21.15.116:0/383358901 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50bc0a9020 con 0x7f50c0004810 2024-04-25T00:12:49.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.653+0000 7f50d37fe640 1 -- 172.21.15.116:0/383358901 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50bc005b00 con 0x7f50c0004810 2024-04-25T00:12:49.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.653+0000 7f50da7d6640 1 -- 172.21.15.116:0/383358901 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50c0004810 msgr2=0x7f50c0004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:49.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.653+0000 7f50da7d6640 1 --2- 172.21.15.116:0/383358901 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50c0004810 0x7f50c0004c90 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f50bc00c240 tx=0x7f50bc01cc30 comp rx=0 tx=0).stop 2024-04-25T00:12:49.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.653+0000 7f50d8d73640 1 -- 172.21.15.116:0/383358901 reap_dead start 2024-04-25T00:12:49.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.653+0000 7f50da7d6640 1 -- 172.21.15.116:0/383358901 shutdown_connections 2024-04-25T00:12:49.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.653+0000 7f50da7d6640 1 -- 172.21.15.116:0/383358901 >> 172.21.15.116:0/383358901 conn(0x7f50d40acdd0 msgr2=0x7f50d40ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:49.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.653+0000 7f50da7d6640 1 -- 172.21.15.116:0/383358901 shutdown_connections 2024-04-25T00:12:49.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.653+0000 7f50da7d6640 1 -- 172.21.15.116:0/383358901 wait complete. 2024-04-25T00:12:49.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.653+0000 7f50da7d6640 1 Processor -- start 2024-04-25T00:12:49.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.653+0000 7f50da7d6640 1 -- start start 2024-04-25T00:12:49.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.653+0000 7f50da7d6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50d4154ca0 0x7f50d4155080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:49.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.653+0000 7f50da7d6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50d41526e0 0x7f50d41f0120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:49.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.653+0000 7f50da7d6640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50d41f0660 0x7f50d414c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:49.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.653+0000 7f50da7d6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f50d40caf00 con 0x7f50d4154ca0 2024-04-25T00:12:49.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.653+0000 7f50da7d6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f50d40cad80 con 0x7f50d41526e0 2024-04-25T00:12:49.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.653+0000 7f50da7d6640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f50d41608c0 con 0x7f50d41f0660 2024-04-25T00:12:49.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.653+0000 7f50d3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50d4154ca0 0x7f50d4155080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:49.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.653+0000 7f50d3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50d4154ca0 0x7f50d4155080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49336/0 (socket says 172.21.15.116:49336) 2024-04-25T00:12:49.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.653+0000 7f50d3fff640 1 -- 172.21.15.116:0/800596757 learned_addr learned my addr 172.21.15.116:0/800596757 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:12:49.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.657+0000 7f50cbfff640 1 --2- 172.21.15.116:0/800596757 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50d41526e0 0x7f50d41f0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:49.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.657+0000 7f50d3fff640 1 -- 172.21.15.116:0/800596757 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50d41f0660 msgr2=0x7f50d414c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:49.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.657+0000 7f50d8d73640 1 --2- 172.21.15.116:0/800596757 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50d41f0660 0x7f50d414c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:49.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.657+0000 7f50d3fff640 1 --2- 172.21.15.116:0/800596757 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50d41f0660 0x7f50d414c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:49.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.657+0000 7f50d3fff640 1 -- 172.21.15.116:0/800596757 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50d41526e0 msgr2=0x7f50d41f0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:49.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.657+0000 7f50d3fff640 1 --2- 172.21.15.116:0/800596757 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50d41526e0 0x7f50d41f0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:49.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.657+0000 7f50d3fff640 1 -- 172.21.15.116:0/800596757 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50d41eecb0 con 0x7f50d4154ca0 2024-04-25T00:12:49.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.657+0000 7f50d8d73640 1 --2- 172.21.15.116:0/800596757 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50d41f0660 0x7f50d414c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:12:49.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.657+0000 7f50d3fff640 1 --2- 172.21.15.116:0/800596757 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50d4154ca0 0x7f50d4155080 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f50c4011560 tx=0x7f50c4011a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:49.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.657+0000 7f50d17fa640 1 -- 172.21.15.116:0/800596757 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50c4015930 con 0x7f50d4154ca0 2024-04-25T00:12:49.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.657+0000 7f50d17fa640 1 -- 172.21.15.116:0/800596757 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50c4015ad0 con 0x7f50d4154ca0 2024-04-25T00:12:49.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.657+0000 7f50da7d6640 1 -- 172.21.15.116:0/800596757 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50d41f23c0 con 0x7f50d4154ca0 2024-04-25T00:12:49.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.657+0000 7f50d17fa640 1 -- 172.21.15.116:0/800596757 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50c400f460 con 0x7f50d4154ca0 2024-04-25T00:12:49.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.657+0000 7f50da7d6640 1 -- 172.21.15.116:0/800596757 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50d41ef250 con 0x7f50d4154ca0 2024-04-25T00:12:49.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.657+0000 7f50da7d6640 1 -- 172.21.15.116:0/800596757 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50d41f0a40 con 0x7f50d4154ca0 2024-04-25T00:12:49.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.657+0000 7f50d17fa640 1 -- 172.21.15.116:0/800596757 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f50c400f600 con 0x7f50d4154ca0 2024-04-25T00:12:49.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.657+0000 7f50d17fa640 1 --2- 172.21.15.116:0/800596757 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f50b40678f0 0x7f50b4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:12:49.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.657+0000 7f50d17fa640 1 -- 172.21.15.116:0/800596757 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 8042+0+0 (secure 0 0 0) 0x7f50c4086d80 con 0x7f50d4154ca0 2024-04-25T00:12:49.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.657+0000 7f50cbfff640 1 --2- 172.21.15.116:0/800596757 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f50b40678f0 0x7f50b4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:12:49.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.661+0000 7f50cbfff640 1 --2- 172.21.15.116:0/800596757 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f50b40678f0 0x7f50b4069db0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f50bc0aa000 tx=0x7f50d40c9cd0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:12:49.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.661+0000 7f50d17fa640 1 -- 172.21.15.116:0/800596757 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f50d41f0a40 con 0x7f50d4154ca0 2024-04-25T00:12:49.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.893+0000 7f50da7d6640 1 -- 172.21.15.116:0/800596757 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f50d414cdb0 con 0x7f50d4154ca0 2024-04-25T00:12:49.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.897+0000 7f50d17fa640 1 -- 172.21.15.116:0/800596757 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) ==== 94+0+6496 (secure 0 0 0) 0x7f50d414cdb0 con 0x7f50d4154ca0 2024-04-25T00:12:49.896 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:12:49.896 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82},{"gid":6582,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/462190523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":462190523},{"type":"v1","addr":"172.21.15.116:6835","nonce":462190523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:12:48.771611+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":88,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6833/365773583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":365773583},{"type":"v1","addr":"172.21.15.116:6833","nonce":365773583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":1},{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:12:23.643548+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5461},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5461":{"gid":5461,"name":"b","rank":0,"incarnation":83,"state":"up:active","state_seq":30,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5461,"qdb_cluster":[5461]},"id":2}]} 2024-04-25T00:12:49.897 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 92 2024-04-25T00:12:49.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.897+0000 7f50da7d6640 1 -- 172.21.15.116:0/800596757 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f50b40678f0 msgr2=0x7f50b4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:49.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.897+0000 7f50da7d6640 1 --2- 172.21.15.116:0/800596757 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f50b40678f0 0x7f50b4069db0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f50bc0aa000 tx=0x7f50d40c9cd0 comp rx=0 tx=0).stop 2024-04-25T00:12:49.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.897+0000 7f50da7d6640 1 -- 172.21.15.116:0/800596757 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50d4154ca0 msgr2=0x7f50d4155080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:12:49.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.897+0000 7f50da7d6640 1 --2- 172.21.15.116:0/800596757 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50d4154ca0 0x7f50d4155080 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f50c4011560 tx=0x7f50c4011a20 comp rx=0 tx=0).stop 2024-04-25T00:12:49.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.897+0000 7f50da7d6640 1 -- 172.21.15.116:0/800596757 shutdown_connections 2024-04-25T00:12:49.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.897+0000 7f50da7d6640 1 --2- 172.21.15.116:0/800596757 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f50b40678f0 0x7f50b4069db0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:49.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.897+0000 7f50da7d6640 1 --2- 172.21.15.116:0/800596757 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f50d41f0660 0x7f50d414c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:49.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.897+0000 7f50da7d6640 1 --2- 172.21.15.116:0/800596757 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50d41526e0 0x7f50d41f0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:49.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.897+0000 7f50da7d6640 1 --2- 172.21.15.116:0/800596757 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50d4154ca0 0x7f50d4155080 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:12:49.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.897+0000 7f50da7d6640 1 -- 172.21.15.116:0/800596757 >> 172.21.15.116:0/800596757 conn(0x7f50d40acdd0 msgr2=0x7f50d4155cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:12:49.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.897+0000 7f50da7d6640 1 -- 172.21.15.116:0/800596757 shutdown_connections 2024-04-25T00:12:49.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:12:49.897+0000 7f50da7d6640 1 -- 172.21.15.116:0/800596757 wait complete. 2024-04-25T00:12:49.919 INFO:tasks.mds_thrash.fs.[a]:mds.a reported in up:standby state 2024-04-25T00:12:49.919 INFO:tasks.mds_thrash.fs.[a]:waiting for 117 secs before thrashing 2024-04-25T00:12:50.691 INFO:tasks.ceph.mds.c.smithi116.stderr:2024-04-25T00:12:50.689+0000 7f4de08d3640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T00:13:07.780 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:13:07.783 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:13:13.729 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:13:13.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.877+0000 7f3525ba1640 1 Processor -- start 2024-04-25T00:13:13.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.877+0000 7f3525ba1640 1 -- start start 2024-04-25T00:13:13.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.881+0000 7f3525ba1640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f352014fa90 0x7f3520153970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:13.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.881+0000 7f3525ba1640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f352014fe70 con 0x7f3520072d70 2024-04-25T00:13:13.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.881+0000 7f3525ba1640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35200c4770 con 0x7f3520155a70 2024-04-25T00:13:13.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.881+0000 7f3525ba1640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f35200c3570 con 0x7f352014fa90 2024-04-25T00:13:13.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.881+0000 7f3517fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f3520072d70 0x7f3520153eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:35610/0 (socket says 172.21.15.116:35610) 2024-04-25T00:13:13.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.881+0000 7f3517fff640 1 -- 172.21.15.116:0/2205497698 learned_addr learned my addr 172.21.15.116:0/2205497698 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:13:13.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.881+0000 7f351f7fe640 1 --2- 172.21.15.116:0/2205497698 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f352014fa90 0x7f3520153970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:13.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.881+0000 7f351effd640 1 -- 172.21.15.116:0/2205497698 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 654204865 0 0) 0x7f352014fe70 con 0x7f3520072d70 2024-04-25T00:13:13.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.881+0000 7f351effd640 1 -- 172.21.15.116:0/2205497698 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f350c003530 con 0x7f3520072d70 2024-04-25T00:13:13.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.881+0000 7f351effd640 1 -- 172.21.15.116:0/2205497698 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3620863696 0 0) 0x7f350c003530 con 0x7f3520072d70 2024-04-25T00:13:13.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.881+0000 7f351effd640 1 -- 172.21.15.116:0/2205497698 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f352014fa90 msgr2=0x7f3520153970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:13:13.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.881+0000 7f351effd640 1 --2- 172.21.15.116:0/2205497698 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f352014fa90 0x7f3520153970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:13.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.881+0000 7f351effd640 1 -- 172.21.15.116:0/2205497698 >> v1:172.21.15.174:6789/0 conn(0x7f3520155a70 legacy=0x7f3520155ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:13.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.881+0000 7f351effd640 1 -- 172.21.15.116:0/2205497698 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35200c3850 con 0x7f3520072d70 2024-04-25T00:13:13.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.881+0000 7f351effd640 1 -- 172.21.15.116:0/2205497698 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f3508001680 con 0x7f3520072d70 2024-04-25T00:13:13.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.881+0000 7f351f7fe640 1 --2- 172.21.15.116:0/2205497698 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f352014fa90 0x7f3520153970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:13:13.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.881+0000 7f351effd640 1 -- 172.21.15.116:0/2205497698 >> v1:172.21.15.116:6789/0 conn(0x7f3520072d70 legacy=0x7f3520153eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:13.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.881+0000 7f351effd640 1 --2- 172.21.15.116:0/2205497698 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f352014fa90 0x7f350c0039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:13.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.881+0000 7f351effd640 1 --2- 172.21.15.116:0/2205497698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f350c003f30 0x7f350c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:13.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.881+0000 7f351effd640 1 --2- 172.21.15.116:0/2205497698 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f350c004d50 0x7f350c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:13.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.881+0000 7f351effd640 1 -- 172.21.15.116:0/2205497698 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f35200c3570 con 0x7f350c003f30 2024-04-25T00:13:13.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.881+0000 7f351effd640 1 -- 172.21.15.116:0/2205497698 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f350c00b9d0 con 0x7f350c004d50 2024-04-25T00:13:13.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.881+0000 7f351effd640 1 -- 172.21.15.116:0/2205497698 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f350c00bbd0 con 0x7f352014fa90 2024-04-25T00:13:13.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.881+0000 7f3517fff640 1 --2- 172.21.15.116:0/2205497698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f350c003f30 0x7f350c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:13.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.881+0000 7f351f7fe640 1 --2- 172.21.15.116:0/2205497698 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f352014fa90 0x7f350c0039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:13.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.881+0000 7f351ffff640 1 --2- 172.21.15.116:0/2205497698 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f350c004d50 0x7f350c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:13.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.881+0000 7f3517fff640 1 -- 172.21.15.116:0/2205497698 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f352014fa90 msgr2=0x7f350c0039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:13:13.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.881+0000 7f3517fff640 1 --2- 172.21.15.116:0/2205497698 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f352014fa90 0x7f350c0039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:13.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.881+0000 7f3517fff640 1 -- 172.21.15.116:0/2205497698 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f350c004d50 msgr2=0x7f350c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:13:13.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.881+0000 7f3517fff640 1 --2- 172.21.15.116:0/2205497698 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f350c004d50 0x7f350c0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:13.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.881+0000 7f3517fff640 1 -- 172.21.15.116:0/2205497698 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f350c003530 con 0x7f350c003f30 2024-04-25T00:13:13.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.885+0000 7f3517fff640 1 --2- 172.21.15.116:0/2205497698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f350c003f30 0x7f350c004810 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f350800c1c0 tx=0x7f350801cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23c7a44a80c84509 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:13.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.885+0000 7f351effd640 1 -- 172.21.15.116:0/2205497698 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3508007d00 con 0x7f350c003f30 2024-04-25T00:13:13.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.885+0000 7f351effd640 1 -- 172.21.15.116:0/2205497698 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35080a9020 con 0x7f350c003f30 2024-04-25T00:13:13.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.885+0000 7f351effd640 1 -- 172.21.15.116:0/2205497698 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3508005ae0 con 0x7f350c003f30 2024-04-25T00:13:13.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.885+0000 7f3525ba1640 1 -- 172.21.15.116:0/2205497698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f350c003f30 msgr2=0x7f350c004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:13.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.885+0000 7f3525ba1640 1 --2- 172.21.15.116:0/2205497698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f350c003f30 0x7f350c004810 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f350800c1c0 tx=0x7f350801cc60 comp rx=0 tx=0).stop 2024-04-25T00:13:13.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.885+0000 7f351ffff640 1 -- 172.21.15.116:0/2205497698 reap_dead start 2024-04-25T00:13:13.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.885+0000 7f3525ba1640 1 -- 172.21.15.116:0/2205497698 shutdown_connections 2024-04-25T00:13:13.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.885+0000 7f3525ba1640 1 -- 172.21.15.116:0/2205497698 >> 172.21.15.116:0/2205497698 conn(0x7f35200c4aa0 msgr2=0x7f35200c4eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:13.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.885+0000 7f3525ba1640 1 -- 172.21.15.116:0/2205497698 shutdown_connections 2024-04-25T00:13:13.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.885+0000 7f3525ba1640 1 -- 172.21.15.116:0/2205497698 wait complete. 2024-04-25T00:13:13.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.885+0000 7f3525ba1640 1 Processor -- start 2024-04-25T00:13:13.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.885+0000 7f3525ba1640 1 -- start start 2024-04-25T00:13:13.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.885+0000 7f3525ba1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f350c003f30 0x7f35200ab450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:13.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.885+0000 7f3525ba1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f352014ba10 0x7f352014a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:13.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.885+0000 7f3525ba1640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f352014be10 0x7f352014aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:13.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.885+0000 7f3525ba1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f352006bba0 con 0x7f350c003f30 2024-04-25T00:13:13.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.885+0000 7f3525ba1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f352006ba20 con 0x7f352014ba10 2024-04-25T00:13:13.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.885+0000 7f3525ba1640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f352006bd20 con 0x7f352014be10 2024-04-25T00:13:13.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.885+0000 7f351f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f350c003f30 0x7f35200ab450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:13.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.885+0000 7f351f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f350c003f30 0x7f35200ab450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59882/0 (socket says 172.21.15.116:59882) 2024-04-25T00:13:13.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.885+0000 7f351f7fe640 1 -- 172.21.15.116:0/2627698307 learned_addr learned my addr 172.21.15.116:0/2627698307 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:13:13.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.885+0000 7f3517fff640 1 --2- 172.21.15.116:0/2627698307 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f352014ba10 0x7f352014a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:13.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.885+0000 7f351ffff640 1 --2- 172.21.15.116:0/2627698307 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f352014be10 0x7f352014aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:13.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.885+0000 7f351f7fe640 1 -- 172.21.15.116:0/2627698307 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f352014be10 msgr2=0x7f352014aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:13.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.885+0000 7f351f7fe640 1 --2- 172.21.15.116:0/2627698307 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f352014be10 0x7f352014aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:13.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.885+0000 7f351f7fe640 1 -- 172.21.15.116:0/2627698307 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f352014ba10 msgr2=0x7f352014a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:13.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.885+0000 7f351f7fe640 1 --2- 172.21.15.116:0/2627698307 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f352014ba10 0x7f352014a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:13.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.885+0000 7f351f7fe640 1 -- 172.21.15.116:0/2627698307 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f352014b680 con 0x7f350c003f30 2024-04-25T00:13:13.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.885+0000 7f351f7fe640 1 --2- 172.21.15.116:0/2627698307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f350c003f30 0x7f35200ab450 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f35100119d0 tx=0x7f3510011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:13.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.889+0000 7f351cff9640 1 -- 172.21.15.116:0/2627698307 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f351000fb50 con 0x7f350c003f30 2024-04-25T00:13:13.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.889+0000 7f351cff9640 1 -- 172.21.15.116:0/2627698307 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f351000fcf0 con 0x7f350c003f30 2024-04-25T00:13:13.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.889+0000 7f3525ba1640 1 -- 172.21.15.116:0/2627698307 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35200acff0 con 0x7f350c003f30 2024-04-25T00:13:13.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.889+0000 7f351cff9640 1 -- 172.21.15.116:0/2627698307 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f351000fea0 con 0x7f350c003f30 2024-04-25T00:13:13.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.889+0000 7f3525ba1640 1 -- 172.21.15.116:0/2627698307 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35200c24d0 con 0x7f350c003f30 2024-04-25T00:13:13.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.889+0000 7f3525ba1640 1 -- 172.21.15.116:0/2627698307 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35200ad4a0 con 0x7f350c003f30 2024-04-25T00:13:13.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.889+0000 7f351cff9640 1 -- 172.21.15.116:0/2627698307 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3510012040 con 0x7f350c003f30 2024-04-25T00:13:13.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.889+0000 7f351cff9640 1 --2- 172.21.15.116:0/2627698307 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f35000678c0 0x7f3500069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:13.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.889+0000 7f351cff9640 1 -- 172.21.15.116:0/2627698307 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 8042+0+0 (secure 0 0 0) 0x7f3510086990 con 0x7f350c003f30 2024-04-25T00:13:13.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.889+0000 7f3517fff640 1 --2- 172.21.15.116:0/2627698307 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f35000678c0 0x7f3500069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:13.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.889+0000 7f3517fff640 1 --2- 172.21.15.116:0/2627698307 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f35000678c0 0x7f3500069d80 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f35080aa000 tx=0x7f3520153490 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:13.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:13.897+0000 7f351cff9640 1 -- 172.21.15.116:0/2627698307 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f3510019070 con 0x7f350c003f30 2024-04-25T00:13:14.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.189+0000 7f3525ba1640 1 -- 172.21.15.116:0/2627698307 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3520068a60 con 0x7f350c003f30 2024-04-25T00:13:14.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.189+0000 7f351cff9640 1 -- 172.21.15.116:0/2627698307 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) ==== 94+0+6496 (secure 0 0 0) 0x7f3510056c40 con 0x7f350c003f30 2024-04-25T00:13:14.189 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:13:14.189 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82},{"gid":6582,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/462190523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":462190523},{"type":"v1","addr":"172.21.15.116:6835","nonce":462190523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:12:48.771611+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":88,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6833/365773583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":365773583},{"type":"v1","addr":"172.21.15.116:6833","nonce":365773583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":1},{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:12:23.643548+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5461},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5461":{"gid":5461,"name":"b","rank":0,"incarnation":83,"state":"up:active","state_seq":30,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5461,"qdb_cluster":[5461]},"id":2}]} 2024-04-25T00:13:14.189 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 92 2024-04-25T00:13:14.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.193+0000 7f3525ba1640 1 -- 172.21.15.116:0/2627698307 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f35000678c0 msgr2=0x7f3500069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:14.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.193+0000 7f3525ba1640 1 --2- 172.21.15.116:0/2627698307 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f35000678c0 0x7f3500069d80 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f35080aa000 tx=0x7f3520153490 comp rx=0 tx=0).stop 2024-04-25T00:13:14.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.193+0000 7f3525ba1640 1 -- 172.21.15.116:0/2627698307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f350c003f30 msgr2=0x7f35200ab450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:14.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.193+0000 7f3525ba1640 1 --2- 172.21.15.116:0/2627698307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f350c003f30 0x7f35200ab450 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f35100119d0 tx=0x7f3510011e90 comp rx=0 tx=0).stop 2024-04-25T00:13:14.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.193+0000 7f3525ba1640 1 -- 172.21.15.116:0/2627698307 shutdown_connections 2024-04-25T00:13:14.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.193+0000 7f3525ba1640 1 --2- 172.21.15.116:0/2627698307 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f35000678c0 0x7f3500069d80 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:14.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.193+0000 7f3525ba1640 1 --2- 172.21.15.116:0/2627698307 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f352014be10 0x7f352014aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:14.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.193+0000 7f3525ba1640 1 --2- 172.21.15.116:0/2627698307 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f352014ba10 0x7f352014a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:14.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.193+0000 7f3525ba1640 1 --2- 172.21.15.116:0/2627698307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f350c003f30 0x7f35200ab450 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:14.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.193+0000 7f3525ba1640 1 -- 172.21.15.116:0/2627698307 >> 172.21.15.116:0/2627698307 conn(0x7f35200c4aa0 msgr2=0x7f352014c870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:14.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.193+0000 7f3525ba1640 1 -- 172.21.15.116:0/2627698307 shutdown_connections 2024-04-25T00:13:14.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.193+0000 7f3525ba1640 1 -- 172.21.15.116:0/2627698307 wait complete. 2024-04-25T00:13:14.209 INFO:tasks.mds_thrash.fs.[b]:kill mds.b (rank=0) 2024-04-25T00:13:14.209 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-25T00:13:14.209 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:13:14.223 INFO:tasks.ceph.mds.b:Stopped 2024-04-25T00:13:14.224 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:13:14.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.369+0000 7fd0931ad640 1 Processor -- start 2024-04-25T00:13:14.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.369+0000 7fd0931ad640 1 -- start start 2024-04-25T00:13:14.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.369+0000 7fd0931ad640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd08c0c1c30 0x7fd08c0c3060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:14.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.369+0000 7fd0931ad640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fd08c153170 0x7fd08c14fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:14.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.369+0000 7fd0931ad640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd08c0c22a0 con 0x7fd08c06c480 2024-04-25T00:13:14.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.369+0000 7fd0931ad640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fd08c0c35a0 con 0x7fd08c153170 2024-04-25T00:13:14.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.369+0000 7fd0931ad640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fd08c153670 con 0x7fd08c0c1c30 2024-04-25T00:13:14.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.369+0000 7fd090f49640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fd08c06c480 0x7fd08c071e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:35626/0 (socket says 172.21.15.116:35626) 2024-04-25T00:13:14.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.369+0000 7fd090f49640 1 -- 172.21.15.116:0/1824588594 learned_addr learned my addr 172.21.15.116:0/1824588594 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:13:14.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.369+0000 7fd08bfff640 1 --2- 172.21.15.116:0/1824588594 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd08c0c1c30 0x7fd08c0c3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:14.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.369+0000 7fd09174a640 1 --2- 172.21.15.116:0/1824588594 >> v2:172.21.15.174:3300/0 conn(0x7fd08c153170 0x7fd08c14fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:14.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.369+0000 7fd08b7fe640 1 -- 172.21.15.116:0/1824588594 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 478578084 0 0) 0x7fd08c0c22a0 con 0x7fd08c06c480 2024-04-25T00:13:14.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.369+0000 7fd08b7fe640 1 -- 172.21.15.116:0/1824588594 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd07c003530 con 0x7fd08c06c480 2024-04-25T00:13:14.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.369+0000 7fd08b7fe640 1 -- 172.21.15.116:0/1824588594 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1908642001 0 0) 0x7fd07c003530 con 0x7fd08c06c480 2024-04-25T00:13:14.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.369+0000 7fd08b7fe640 1 -- 172.21.15.116:0/1824588594 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd08c0c1c30 msgr2=0x7fd08c0c3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:13:14.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.369+0000 7fd08b7fe640 1 --2- 172.21.15.116:0/1824588594 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd08c0c1c30 0x7fd08c0c3060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:14.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.369+0000 7fd08b7fe640 1 -- 172.21.15.116:0/1824588594 >> v2:172.21.15.174:3300/0 conn(0x7fd08c153170 msgr2=0x7fd08c14fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:13:14.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.369+0000 7fd08b7fe640 1 --2- 172.21.15.116:0/1824588594 >> v2:172.21.15.174:3300/0 conn(0x7fd08c153170 0x7fd08c14fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:14.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.369+0000 7fd08b7fe640 1 -- 172.21.15.116:0/1824588594 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd08c0c2580 con 0x7fd08c06c480 2024-04-25T00:13:14.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.369+0000 7fd08b7fe640 1 -- 172.21.15.116:0/1824588594 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7fd080001680 con 0x7fd08c06c480 2024-04-25T00:13:14.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.369+0000 7fd08b7fe640 1 -- 172.21.15.116:0/1824588594 >> v1:172.21.15.116:6789/0 conn(0x7fd08c06c480 legacy=0x7fd08c071e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:14.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.369+0000 7fd08b7fe640 1 --2- 172.21.15.116:0/1824588594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd07c0039c0 0x7fd07c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:14.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.369+0000 7fd08b7fe640 1 --2- 172.21.15.116:0/1824588594 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd07c004810 0x7fd07c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:14.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.369+0000 7fd08b7fe640 1 --2- 172.21.15.116:0/1824588594 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd08c0c1c30 0x7fd07c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:14.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.369+0000 7fd08b7fe640 1 -- 172.21.15.116:0/1824588594 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd08c0c35a0 con 0x7fd07c0039c0 2024-04-25T00:13:14.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.369+0000 7fd08b7fe640 1 -- 172.21.15.116:0/1824588594 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd08c153670 con 0x7fd07c004810 2024-04-25T00:13:14.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.369+0000 7fd08b7fe640 1 -- 172.21.15.116:0/1824588594 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fd07c00b9d0 con 0x7fd08c0c1c30 2024-04-25T00:13:14.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.369+0000 7fd090f49640 1 --2- 172.21.15.116:0/1824588594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd07c0039c0 0x7fd07c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:14.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.369+0000 7fd08bfff640 1 --2- 172.21.15.116:0/1824588594 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd07c004810 0x7fd07c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:14.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.369+0000 7fd09174a640 1 --2- 172.21.15.116:0/1824588594 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd08c0c1c30 0x7fd07c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:14.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.369+0000 7fd090f49640 1 -- 172.21.15.116:0/1824588594 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd08c0c1c30 msgr2=0x7fd07c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:13:14.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.369+0000 7fd090f49640 1 --2- 172.21.15.116:0/1824588594 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd08c0c1c30 0x7fd07c0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:14.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.369+0000 7fd090f49640 1 -- 172.21.15.116:0/1824588594 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd07c004810 msgr2=0x7fd07c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:13:14.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.369+0000 7fd090f49640 1 --2- 172.21.15.116:0/1824588594 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd07c004810 0x7fd07c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:14.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.369+0000 7fd090f49640 1 -- 172.21.15.116:0/1824588594 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd07c003530 con 0x7fd07c0039c0 2024-04-25T00:13:14.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.369+0000 7fd08bfff640 1 --2- 172.21.15.116:0/1824588594 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd07c004810 0x7fd07c004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:13:14.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.373+0000 7fd090f49640 1 --2- 172.21.15.116:0/1824588594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd07c0039c0 0x7fd07c003de0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fd08000c190 tx=0x7fd08001cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c0ea29c9971b6d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:14.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.373+0000 7fd08b7fe640 1 -- 172.21.15.116:0/1824588594 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd080007c70 con 0x7fd07c0039c0 2024-04-25T00:13:14.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.373+0000 7fd08b7fe640 1 -- 172.21.15.116:0/1824588594 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0800a9020 con 0x7fd07c0039c0 2024-04-25T00:13:14.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.373+0000 7fd08b7fe640 1 -- 172.21.15.116:0/1824588594 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd080005b00 con 0x7fd07c0039c0 2024-04-25T00:13:14.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.373+0000 7fd0931ad640 1 -- 172.21.15.116:0/1824588594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd07c0039c0 msgr2=0x7fd07c003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:14.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.373+0000 7fd0931ad640 1 --2- 172.21.15.116:0/1824588594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd07c0039c0 0x7fd07c003de0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fd08000c190 tx=0x7fd08001cbe0 comp rx=0 tx=0).stop 2024-04-25T00:13:14.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.373+0000 7fd09174a640 1 -- 172.21.15.116:0/1824588594 reap_dead start 2024-04-25T00:13:14.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.373+0000 7fd0931ad640 1 -- 172.21.15.116:0/1824588594 shutdown_connections 2024-04-25T00:13:14.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.373+0000 7fd0931ad640 1 -- 172.21.15.116:0/1824588594 >> 172.21.15.116:0/1824588594 conn(0x7fd08c0c4270 msgr2=0x7fd08c0c4680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:14.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.373+0000 7fd0931ad640 1 -- 172.21.15.116:0/1824588594 shutdown_connections 2024-04-25T00:13:14.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.373+0000 7fd0931ad640 1 -- 172.21.15.116:0/1824588594 wait complete. 2024-04-25T00:13:14.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.373+0000 7fd0931ad640 1 Processor -- start 2024-04-25T00:13:14.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.373+0000 7fd0931ad640 1 -- start start 2024-04-25T00:13:14.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.377+0000 7fd0931ad640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd07c0039c0 0x7fd08c147060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:14.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.377+0000 7fd0931ad640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd08c0a90c0 0x7fd08c1eb540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:14.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.377+0000 7fd0931ad640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd08c1441e0 0x7fd08c1eba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:14.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.377+0000 7fd0931ad640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd08c150350 con 0x7fd08c0a90c0 2024-04-25T00:13:14.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.377+0000 7fd0931ad640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd08c1501d0 con 0x7fd07c0039c0 2024-04-25T00:13:14.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.377+0000 7fd0931ad640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fd08c06ca70 con 0x7fd08c1441e0 2024-04-25T00:13:14.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.377+0000 7fd08bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd08c0a90c0 0x7fd08c1eb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:14.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.377+0000 7fd08bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd08c0a90c0 0x7fd08c1eb540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59888/0 (socket says 172.21.15.116:59888) 2024-04-25T00:13:14.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.377+0000 7fd08bfff640 1 -- 172.21.15.116:0/1904172579 learned_addr learned my addr 172.21.15.116:0/1904172579 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:13:14.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.377+0000 7fd090f49640 1 --2- 172.21.15.116:0/1904172579 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd07c0039c0 0x7fd08c147060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:14.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.377+0000 7fd09174a640 1 --2- 172.21.15.116:0/1904172579 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd08c1441e0 0x7fd08c1eba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:14.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.377+0000 7fd08bfff640 1 -- 172.21.15.116:0/1904172579 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd08c1441e0 msgr2=0x7fd08c1eba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:14.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.377+0000 7fd08bfff640 1 --2- 172.21.15.116:0/1904172579 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd08c1441e0 0x7fd08c1eba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:14.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.377+0000 7fd08bfff640 1 -- 172.21.15.116:0/1904172579 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd07c0039c0 msgr2=0x7fd08c147060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:14.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.377+0000 7fd08bfff640 1 --2- 172.21.15.116:0/1904172579 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd07c0039c0 0x7fd08c147060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:14.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.377+0000 7fd08bfff640 1 -- 172.21.15.116:0/1904172579 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd08c158110 con 0x7fd08c0a90c0 2024-04-25T00:13:14.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.377+0000 7fd09174a640 1 --2- 172.21.15.116:0/1904172579 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd08c1441e0 0x7fd08c1eba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:13:14.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.377+0000 7fd08bfff640 1 --2- 172.21.15.116:0/1904172579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd08c0a90c0 0x7fd08c1eb540 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fd078009160 tx=0x7fd078011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:14.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.377+0000 7fd0897fa640 1 -- 172.21.15.116:0/1904172579 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd078004220 con 0x7fd08c0a90c0 2024-04-25T00:13:14.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.377+0000 7fd0897fa640 1 -- 172.21.15.116:0/1904172579 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0780043c0 con 0x7fd08c0a90c0 2024-04-25T00:13:14.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.377+0000 7fd0931ad640 1 -- 172.21.15.116:0/1904172579 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd08c1583f0 con 0x7fd08c0a90c0 2024-04-25T00:13:14.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.377+0000 7fd0897fa640 1 -- 172.21.15.116:0/1904172579 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd078015890 con 0x7fd08c0a90c0 2024-04-25T00:13:14.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.377+0000 7fd0931ad640 1 -- 172.21.15.116:0/1904172579 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd08c1589b0 con 0x7fd08c0a90c0 2024-04-25T00:13:14.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.377+0000 7fd0931ad640 1 -- 172.21.15.116:0/1904172579 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd08c068a60 con 0x7fd08c0a90c0 2024-04-25T00:13:14.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.377+0000 7fd0897fa640 1 -- 172.21.15.116:0/1904172579 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd078015a30 con 0x7fd08c0a90c0 2024-04-25T00:13:14.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.377+0000 7fd0897fa640 1 --2- 172.21.15.116:0/1904172579 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd0700678c0 0x7fd070069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:14.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.377+0000 7fd0897fa640 1 -- 172.21.15.116:0/1904172579 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 8042+0+0 (secure 0 0 0) 0x7fd078086b70 con 0x7fd08c0a90c0 2024-04-25T00:13:14.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.377+0000 7fd090f49640 1 --2- 172.21.15.116:0/1904172579 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd0700678c0 0x7fd070069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:14.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.381+0000 7fd090f49640 1 --2- 172.21.15.116:0/1904172579 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd0700678c0 0x7fd070069d80 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fd0800aa000 tx=0x7fd08c071d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:14.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.385+0000 7fd0897fa640 1 -- 172.21.15.116:0/1904172579 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fd078019030 con 0x7fd08c0a90c0 2024-04-25T00:13:14.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.621+0000 7fd0931ad640 1 -- 172.21.15.116:0/1904172579 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd08c0c4b20 con 0x7fd08c0a90c0 2024-04-25T00:13:14.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.621+0000 7fd0897fa640 1 -- 172.21.15.116:0/1904172579 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) ==== 94+0+6496 (secure 0 0 0) 0x7fd078056740 con 0x7fd08c0a90c0 2024-04-25T00:13:14.623 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:13:14.623 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82},{"gid":6582,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/462190523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":462190523},{"type":"v1","addr":"172.21.15.116:6835","nonce":462190523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:12:48.771611+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":88,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6833/365773583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":365773583},{"type":"v1","addr":"172.21.15.116:6833","nonce":365773583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":1},{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:12:23.643548+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5461},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5461":{"gid":5461,"name":"b","rank":0,"incarnation":83,"state":"up:active","state_seq":30,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5461,"qdb_cluster":[5461]},"id":2}]} 2024-04-25T00:13:14.624 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 92 2024-04-25T00:13:14.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.625+0000 7fd0931ad640 1 -- 172.21.15.116:0/1904172579 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd0700678c0 msgr2=0x7fd070069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:14.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.625+0000 7fd0931ad640 1 --2- 172.21.15.116:0/1904172579 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd0700678c0 0x7fd070069d80 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fd0800aa000 tx=0x7fd08c071d50 comp rx=0 tx=0).stop 2024-04-25T00:13:14.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.625+0000 7fd0931ad640 1 -- 172.21.15.116:0/1904172579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd08c0a90c0 msgr2=0x7fd08c1eb540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:14.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.625+0000 7fd0931ad640 1 --2- 172.21.15.116:0/1904172579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd08c0a90c0 0x7fd08c1eb540 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fd078009160 tx=0x7fd078011a60 comp rx=0 tx=0).stop 2024-04-25T00:13:14.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.625+0000 7fd0931ad640 1 -- 172.21.15.116:0/1904172579 shutdown_connections 2024-04-25T00:13:14.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.625+0000 7fd0931ad640 1 --2- 172.21.15.116:0/1904172579 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd0700678c0 0x7fd070069d80 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:14.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.625+0000 7fd0931ad640 1 --2- 172.21.15.116:0/1904172579 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd08c1441e0 0x7fd08c1eba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:14.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.625+0000 7fd0931ad640 1 --2- 172.21.15.116:0/1904172579 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd08c0a90c0 0x7fd08c1eb540 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:14.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.625+0000 7fd0931ad640 1 --2- 172.21.15.116:0/1904172579 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd07c0039c0 0x7fd08c147060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:14.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.625+0000 7fd0931ad640 1 -- 172.21.15.116:0/1904172579 >> 172.21.15.116:0/1904172579 conn(0x7fd08c0c4270 msgr2=0x7fd08c15af80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:14.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.625+0000 7fd0931ad640 1 -- 172.21.15.116:0/1904172579 shutdown_connections 2024-04-25T00:13:14.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:14.625+0000 7fd0931ad640 1 -- 172.21.15.116:0/1904172579 wait complete. 2024-04-25T00:13:14.648 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T00:13:16.649 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:13:16.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:16.793+0000 7f4470814640 1 Processor -- start 2024-04-25T00:13:16.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:16.793+0000 7f4470814640 1 -- start start 2024-04-25T00:13:16.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:16.793+0000 7f4470814640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f446c155810 0x7f446c157390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:16.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:16.793+0000 7f4470814640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f446c0c9150 con 0x7f446c06d5f0 2024-04-25T00:13:16.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:16.793+0000 7f4470814640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f446c0c9430 con 0x7f446c152d80 2024-04-25T00:13:16.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:16.793+0000 7f4470814640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f446c0c9d60 con 0x7f446c155810 2024-04-25T00:13:16.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:16.793+0000 7f4463fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f446c06d5f0 0x7f446c156c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:59612/0 (socket says 172.21.15.116:59612) 2024-04-25T00:13:16.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:16.793+0000 7f4463fff640 1 -- 172.21.15.116:0/3643412315 learned_addr learned my addr 172.21.15.116:0/3643412315 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:13:16.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:16.793+0000 7f446ad9d640 1 --2- 172.21.15.116:0/3643412315 >> v2:172.21.15.116:3300/0 conn(0x7f446c155810 0x7f446c157390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:16.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:16.793+0000 7f446ad9d640 1 -- 172.21.15.116:0/3643412315 >> v1:172.21.15.174:6789/0 conn(0x7f446c152d80 legacy=0x7f446c06bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:16.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:16.793+0000 7f446ad9d640 1 -- 172.21.15.116:0/3643412315 >> v1:172.21.15.116:6789/0 conn(0x7f446c06d5f0 legacy=0x7f446c156c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:16.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:16.793+0000 7f446ad9d640 1 -- 172.21.15.116:0/3643412315 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f446c0c9710 con 0x7f446c155810 2024-04-25T00:13:16.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:16.793+0000 7f4469d9b640 1 -- 172.21.15.116:0/3643412315 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3471132919 0 0) 0x7f446c0c9150 con 0x7f446c06d5f0 2024-04-25T00:13:16.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:16.797+0000 7f446ad9d640 1 --2- 172.21.15.116:0/3643412315 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f446c155810 0x7f446c157390 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f4464009d90 tx=0x7f446401c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=225e104211831fa1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:16.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:16.797+0000 7f4469d9b640 1 -- 172.21.15.116:0/3643412315 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44640b2070 con 0x7f446c155810 2024-04-25T00:13:16.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:16.797+0000 7f4469d9b640 1 -- 172.21.15.116:0/3643412315 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44640a5990 con 0x7f446c155810 2024-04-25T00:13:16.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:16.797+0000 7f4469d9b640 1 -- 172.21.15.116:0/3643412315 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44640a5d60 con 0x7f446c155810 2024-04-25T00:13:16.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:16.797+0000 7f4470814640 1 -- 172.21.15.116:0/3643412315 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f446c155810 msgr2=0x7f446c157390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:16.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:16.797+0000 7f4470814640 1 --2- 172.21.15.116:0/3643412315 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f446c155810 0x7f446c157390 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f4464009d90 tx=0x7f446401c8b0 comp rx=0 tx=0).stop 2024-04-25T00:13:16.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:16.797+0000 7f4470814640 1 -- 172.21.15.116:0/3643412315 shutdown_connections 2024-04-25T00:13:16.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:16.797+0000 7f4470814640 1 --2- 172.21.15.116:0/3643412315 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f446c155810 0x7f446c157390 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:16.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:16.797+0000 7f4470814640 1 -- 172.21.15.116:0/3643412315 >> 172.21.15.116:0/3643412315 conn(0x7f446c0acdd0 msgr2=0x7f446c0ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:16.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:16.797+0000 7f4470814640 1 -- 172.21.15.116:0/3643412315 shutdown_connections 2024-04-25T00:13:16.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:16.797+0000 7f4470814640 1 -- 172.21.15.116:0/3643412315 wait complete. 2024-04-25T00:13:16.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:16.797+0000 7f4470814640 1 Processor -- start 2024-04-25T00:13:16.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:16.797+0000 7f4470814640 1 -- start start 2024-04-25T00:13:16.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:16.801+0000 7f4470814640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f446c06d5f0 0x7f446c154930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:16.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:16.801+0000 7f4470814640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f446c152d80 0x7f446c0c5930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:16.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:16.801+0000 7f4470814640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f446c155810 0x7f446c0c5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:16.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:16.801+0000 7f4470814640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f446c153160 con 0x7f446c152d80 2024-04-25T00:13:16.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:16.801+0000 7f4470814640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f446c0cad80 con 0x7f446c155810 2024-04-25T00:13:16.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:16.801+0000 7f4470814640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f446c1edd20 con 0x7f446c06d5f0 2024-04-25T00:13:16.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:16.801+0000 7f4463fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f446c152d80 0x7f446c0c5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:16.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:16.801+0000 7f4463fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f446c152d80 0x7f446c0c5930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36428/0 (socket says 172.21.15.116:36428) 2024-04-25T00:13:16.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:16.801+0000 7f4463fff640 1 -- 172.21.15.116:0/3979863482 learned_addr learned my addr 172.21.15.116:0/3979863482 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:13:16.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:16.801+0000 7f446a59c640 1 --2- 172.21.15.116:0/3979863482 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f446c06d5f0 0x7f446c154930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:16.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:16.801+0000 7f446ad9d640 1 --2- 172.21.15.116:0/3979863482 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f446c155810 0x7f446c0c5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:16.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:16.801+0000 7f4463fff640 1 -- 172.21.15.116:0/3979863482 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f446c06d5f0 msgr2=0x7f446c154930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:16.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:16.801+0000 7f4463fff640 1 --2- 172.21.15.116:0/3979863482 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f446c06d5f0 0x7f446c154930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:16.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:16.801+0000 7f4463fff640 1 -- 172.21.15.116:0/3979863482 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f446c155810 msgr2=0x7f446c0c5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:16.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:16.801+0000 7f4463fff640 1 --2- 172.21.15.116:0/3979863482 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f446c155810 0x7f446c0c5e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:16.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:16.801+0000 7f4463fff640 1 -- 172.21.15.116:0/3979863482 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f446c0c76b0 con 0x7f446c152d80 2024-04-25T00:13:16.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:16.801+0000 7f446ad9d640 1 --2- 172.21.15.116:0/3979863482 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f446c155810 0x7f446c0c5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:13:16.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:16.801+0000 7f4463fff640 1 --2- 172.21.15.116:0/3979863482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f446c152d80 0x7f446c0c5930 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f445400c410 tx=0x7f445400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:16.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:16.801+0000 7f4462ffd640 1 -- 172.21.15.116:0/3979863482 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44540074e0 con 0x7f446c152d80 2024-04-25T00:13:16.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:16.801+0000 7f4462ffd640 1 -- 172.21.15.116:0/3979863482 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44540047f0 con 0x7f446c152d80 2024-04-25T00:13:16.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:16.801+0000 7f4470814640 1 -- 172.21.15.116:0/3979863482 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f446c1edfe0 con 0x7f446c152d80 2024-04-25T00:13:16.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:16.801+0000 7f4462ffd640 1 -- 172.21.15.116:0/3979863482 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4454007680 con 0x7f446c152d80 2024-04-25T00:13:16.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:16.801+0000 7f4470814640 1 -- 172.21.15.116:0/3979863482 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f446c1ee5a0 con 0x7f446c152d80 2024-04-25T00:13:16.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:16.801+0000 7f4470814640 1 -- 172.21.15.116:0/3979863482 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f446c068350 con 0x7f446c152d80 2024-04-25T00:13:16.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:16.801+0000 7f4462ffd640 1 -- 172.21.15.116:0/3979863482 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4454011070 con 0x7f446c152d80 2024-04-25T00:13:16.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:16.805+0000 7f4462ffd640 1 --2- 172.21.15.116:0/3979863482 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f444c0678c0 0x7f444c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:16.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:16.805+0000 7f4462ffd640 1 -- 172.21.15.116:0/3979863482 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 8042+0+0 (secure 0 0 0) 0x7f4454086150 con 0x7f446c152d80 2024-04-25T00:13:16.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:16.805+0000 7f446a59c640 1 --2- 172.21.15.116:0/3979863482 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f444c0678c0 0x7f444c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:16.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:16.805+0000 7f446a59c640 1 --2- 172.21.15.116:0/3979863482 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f444c0678c0 0x7f444c069d80 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f445c000f70 tx=0x7f445c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:16.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:16.809+0000 7f4462ffd640 1 -- 172.21.15.116:0/3979863482 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f44540563e0 con 0x7f446c152d80 2024-04-25T00:13:17.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:17.065+0000 7f4470814640 1 -- 172.21.15.116:0/3979863482 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f446c0cab60 con 0x7f446c152d80 2024-04-25T00:13:17.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:17.065+0000 7f4462ffd640 1 -- 172.21.15.116:0/3979863482 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) ==== 94+0+6496 (secure 0 0 0) 0x7f4454055ce0 con 0x7f446c152d80 2024-04-25T00:13:17.066 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:13:17.066 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82},{"gid":6582,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/462190523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":462190523},{"type":"v1","addr":"172.21.15.116:6835","nonce":462190523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:12:48.771611+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":88,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6833/365773583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":365773583},{"type":"v1","addr":"172.21.15.116:6833","nonce":365773583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":1},{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:12:23.643548+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5461},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5461":{"gid":5461,"name":"b","rank":0,"incarnation":83,"state":"up:active","state_seq":30,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5461,"qdb_cluster":[5461]},"id":2}]} 2024-04-25T00:13:17.067 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 92 2024-04-25T00:13:17.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:17.069+0000 7f4470814640 1 -- 172.21.15.116:0/3979863482 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f444c0678c0 msgr2=0x7f444c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:17.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:17.069+0000 7f4470814640 1 --2- 172.21.15.116:0/3979863482 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f444c0678c0 0x7f444c069d80 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f445c000f70 tx=0x7f445c009040 comp rx=0 tx=0).stop 2024-04-25T00:13:17.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:17.069+0000 7f4470814640 1 -- 172.21.15.116:0/3979863482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f446c152d80 msgr2=0x7f446c0c5930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:17.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:17.069+0000 7f4470814640 1 --2- 172.21.15.116:0/3979863482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f446c152d80 0x7f446c0c5930 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f445400c410 tx=0x7f445400c8d0 comp rx=0 tx=0).stop 2024-04-25T00:13:17.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:17.069+0000 7f4470814640 1 -- 172.21.15.116:0/3979863482 shutdown_connections 2024-04-25T00:13:17.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:17.069+0000 7f4470814640 1 --2- 172.21.15.116:0/3979863482 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f444c0678c0 0x7f444c069d80 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:17.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:17.069+0000 7f4470814640 1 --2- 172.21.15.116:0/3979863482 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f446c155810 0x7f446c0c5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:17.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:17.069+0000 7f4470814640 1 --2- 172.21.15.116:0/3979863482 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f446c152d80 0x7f446c0c5930 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:17.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:17.069+0000 7f4470814640 1 --2- 172.21.15.116:0/3979863482 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f446c06d5f0 0x7f446c154930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:17.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:17.069+0000 7f4470814640 1 -- 172.21.15.116:0/3979863482 >> 172.21.15.116:0/3979863482 conn(0x7f446c0acdd0 msgr2=0x7f446c1602f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:17.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:17.069+0000 7f4470814640 1 -- 172.21.15.116:0/3979863482 shutdown_connections 2024-04-25T00:13:17.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:17.069+0000 7f4470814640 1 -- 172.21.15.116:0/3979863482 wait complete. 2024-04-25T00:13:17.088 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T00:13:19.089 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:13:19.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.237+0000 7feb480fa640 1 Processor -- start 2024-04-25T00:13:19.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.237+0000 7feb480fa640 1 -- start start 2024-04-25T00:13:19.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.237+0000 7feb480fa640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7feb400c5280 0x7feb40150840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:19.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.237+0000 7feb480fa640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7feb40072c90 0x7feb40150d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:19.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.237+0000 7feb480fa640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7feb40072810 0x7feb40153760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:19.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.237+0000 7feb480fa640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7feb40157c20 con 0x7feb400c5280 2024-04-25T00:13:19.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.237+0000 7feb480fa640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7feb400c2f70 con 0x7feb40072810 2024-04-25T00:13:19.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.237+0000 7feb480fa640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7feb400c2c40 con 0x7feb40072c90 2024-04-25T00:13:19.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.237+0000 7feb45e96640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7feb400c5280 0x7feb40150840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:19.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.237+0000 7feb45e96640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7feb400c5280 0x7feb40150840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36440/0 (socket says 172.21.15.116:36440) 2024-04-25T00:13:19.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.237+0000 7feb45e96640 1 -- 172.21.15.116:0/9446278 learned_addr learned my addr 172.21.15.116:0/9446278 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:13:19.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.237+0000 7feb45695640 1 --2- 172.21.15.116:0/9446278 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7feb40072c90 0x7feb40150d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:19.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.237+0000 7feb46697640 1 --2- 172.21.15.116:0/9446278 >> v2:172.21.15.174:3300/0 conn(0x7feb40072810 0x7feb40153760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:19.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.237+0000 7feb45e96640 1 -- 172.21.15.116:0/9446278 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7feb40072c90 msgr2=0x7feb40150d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:13:19.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.237+0000 7feb45e96640 1 --2- 172.21.15.116:0/9446278 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7feb40072c90 0x7feb40150d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:19.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.237+0000 7feb45e96640 1 -- 172.21.15.116:0/9446278 >> v2:172.21.15.174:3300/0 conn(0x7feb40072810 msgr2=0x7feb40153760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:13:19.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.237+0000 7feb45e96640 1 --2- 172.21.15.116:0/9446278 >> v2:172.21.15.174:3300/0 conn(0x7feb40072810 0x7feb40153760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:19.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.237+0000 7feb45e96640 1 -- 172.21.15.116:0/9446278 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb401512c0 con 0x7feb400c5280 2024-04-25T00:13:19.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.237+0000 7feb46697640 1 --2- 172.21.15.116:0/9446278 >> v2:172.21.15.174:3300/0 conn(0x7feb40072810 0x7feb40153760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:13:19.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.241+0000 7feb45e96640 1 --2- 172.21.15.116:0/9446278 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb400c5280 0x7feb40150840 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7feb400c2940 tx=0x7feb3c01c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1113135f326d89a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:19.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.241+0000 7feb44e94640 1 -- 172.21.15.116:0/9446278 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb3c0ab620 con 0x7feb400c5280 2024-04-25T00:13:19.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.241+0000 7feb44e94640 1 -- 172.21.15.116:0/9446278 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb3c0b0070 con 0x7feb400c5280 2024-04-25T00:13:19.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.241+0000 7feb44e94640 1 -- 172.21.15.116:0/9446278 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb3c0ab9e0 con 0x7feb400c5280 2024-04-25T00:13:19.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.241+0000 7feb480fa640 1 -- 172.21.15.116:0/9446278 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb400c5280 msgr2=0x7feb40150840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:19.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.241+0000 7feb480fa640 1 --2- 172.21.15.116:0/9446278 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb400c5280 0x7feb40150840 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7feb400c2940 tx=0x7feb3c01c460 comp rx=0 tx=0).stop 2024-04-25T00:13:19.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.241+0000 7feb480fa640 1 -- 172.21.15.116:0/9446278 shutdown_connections 2024-04-25T00:13:19.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.241+0000 7feb480fa640 1 --2- 172.21.15.116:0/9446278 >> v2:172.21.15.174:3300/0 conn(0x7feb40072810 0x7feb40153760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:19.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.241+0000 7feb480fa640 1 --2- 172.21.15.116:0/9446278 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7feb40072c90 0x7feb40150d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:19.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.241+0000 7feb480fa640 1 --2- 172.21.15.116:0/9446278 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb400c5280 0x7feb40150840 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:19.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.241+0000 7feb480fa640 1 -- 172.21.15.116:0/9446278 >> 172.21.15.116:0/9446278 conn(0x7feb4006c720 msgr2=0x7feb4006cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:19.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.241+0000 7feb480fa640 1 -- 172.21.15.116:0/9446278 shutdown_connections 2024-04-25T00:13:19.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.241+0000 7feb480fa640 1 -- 172.21.15.116:0/9446278 wait complete. 2024-04-25T00:13:19.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.241+0000 7feb480fa640 1 Processor -- start 2024-04-25T00:13:19.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.241+0000 7feb480fa640 1 -- start start 2024-04-25T00:13:19.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.245+0000 7feb480fa640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb40072810 0x7feb401ef820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:19.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.245+0000 7feb480fa640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb40072c90 0x7feb401efd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:19.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.245+0000 7feb480fa640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7feb400c5280 0x7feb400c6850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:19.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.245+0000 7feb480fa640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7feb4006b210 con 0x7feb40072810 2024-04-25T00:13:19.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.245+0000 7feb480fa640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7feb400c5870 con 0x7feb40072c90 2024-04-25T00:13:19.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.245+0000 7feb480fa640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7feb4006b390 con 0x7feb400c5280 2024-04-25T00:13:19.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.245+0000 7feb45e96640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb40072810 0x7feb401ef820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:19.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.245+0000 7feb45e96640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb40072810 0x7feb401ef820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36442/0 (socket says 172.21.15.116:36442) 2024-04-25T00:13:19.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.245+0000 7feb45e96640 1 -- 172.21.15.116:0/2525716796 learned_addr learned my addr 172.21.15.116:0/2525716796 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:13:19.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.245+0000 7feb45695640 1 --2- 172.21.15.116:0/2525716796 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb40072c90 0x7feb401efd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:19.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.245+0000 7feb46697640 1 --2- 172.21.15.116:0/2525716796 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7feb400c5280 0x7feb400c6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:19.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.245+0000 7feb45e96640 1 -- 172.21.15.116:0/2525716796 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7feb400c5280 msgr2=0x7feb400c6850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:19.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.245+0000 7feb45e96640 1 --2- 172.21.15.116:0/2525716796 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7feb400c5280 0x7feb400c6850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:19.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.245+0000 7feb45e96640 1 -- 172.21.15.116:0/2525716796 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb40072c90 msgr2=0x7feb401efd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:19.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.245+0000 7feb45e96640 1 --2- 172.21.15.116:0/2525716796 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb40072c90 0x7feb401efd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:19.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.245+0000 7feb45e96640 1 -- 172.21.15.116:0/2525716796 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb400c5d00 con 0x7feb40072810 2024-04-25T00:13:19.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.245+0000 7feb45e96640 1 --2- 172.21.15.116:0/2525716796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb40072810 0x7feb401ef820 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7feb3c01ab10 tx=0x7feb3c01aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:19.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.245+0000 7feb367fc640 1 -- 172.21.15.116:0/2525716796 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb3c0abdd0 con 0x7feb40072810 2024-04-25T00:13:19.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.245+0000 7feb367fc640 1 -- 172.21.15.116:0/2525716796 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb3c0be3f0 con 0x7feb40072810 2024-04-25T00:13:19.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.245+0000 7feb480fa640 1 -- 172.21.15.116:0/2525716796 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb400c6370 con 0x7feb40072810 2024-04-25T00:13:19.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.245+0000 7feb367fc640 1 -- 172.21.15.116:0/2525716796 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb3c0be7e0 con 0x7feb40072810 2024-04-25T00:13:19.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.245+0000 7feb480fa640 1 -- 172.21.15.116:0/2525716796 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb401557c0 con 0x7feb40072810 2024-04-25T00:13:19.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.245+0000 7feb480fa640 1 -- 172.21.15.116:0/2525716796 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb40068a60 con 0x7feb40072810 2024-04-25T00:13:19.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.245+0000 7feb367fc640 1 -- 172.21.15.116:0/2525716796 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7feb3c0c7440 con 0x7feb40072810 2024-04-25T00:13:19.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.245+0000 7feb367fc640 1 --2- 172.21.15.116:0/2525716796 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7feb240678f0 0x7feb24069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:19.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.245+0000 7feb367fc640 1 -- 172.21.15.116:0/2525716796 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 8042+0+0 (secure 0 0 0) 0x7feb3c120a20 con 0x7feb40072810 2024-04-25T00:13:19.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.249+0000 7feb45695640 1 --2- 172.21.15.116:0/2525716796 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7feb240678f0 0x7feb24069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:19.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.249+0000 7feb45695640 1 --2- 172.21.15.116:0/2525716796 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7feb240678f0 0x7feb24069db0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7feb4014bbf0 tx=0x7feb30008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:19.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.253+0000 7feb367fc640 1 -- 172.21.15.116:0/2525716796 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7feb3c016110 con 0x7feb40072810 2024-04-25T00:13:19.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.493+0000 7feb480fa640 1 -- 172.21.15.116:0/2525716796 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feb4014e700 con 0x7feb40072810 2024-04-25T00:13:19.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.493+0000 7feb367fc640 1 -- 172.21.15.116:0/2525716796 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) ==== 94+0+6496 (secure 0 0 0) 0x7feb3c0f0cd0 con 0x7feb40072810 2024-04-25T00:13:19.494 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:13:19.494 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82},{"gid":6582,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/462190523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":462190523},{"type":"v1","addr":"172.21.15.116:6835","nonce":462190523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:12:48.771611+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":88,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6833/365773583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":365773583},{"type":"v1","addr":"172.21.15.116:6833","nonce":365773583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":1},{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:12:23.643548+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5461},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5461":{"gid":5461,"name":"b","rank":0,"incarnation":83,"state":"up:active","state_seq":30,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5461,"qdb_cluster":[5461]},"id":2}]} 2024-04-25T00:13:19.495 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 92 2024-04-25T00:13:19.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.497+0000 7feb480fa640 1 -- 172.21.15.116:0/2525716796 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7feb240678f0 msgr2=0x7feb24069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:19.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.497+0000 7feb480fa640 1 --2- 172.21.15.116:0/2525716796 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7feb240678f0 0x7feb24069db0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7feb4014bbf0 tx=0x7feb30008040 comp rx=0 tx=0).stop 2024-04-25T00:13:19.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.497+0000 7feb480fa640 1 -- 172.21.15.116:0/2525716796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb40072810 msgr2=0x7feb401ef820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:19.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.497+0000 7feb480fa640 1 --2- 172.21.15.116:0/2525716796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb40072810 0x7feb401ef820 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7feb3c01ab10 tx=0x7feb3c01aca0 comp rx=0 tx=0).stop 2024-04-25T00:13:19.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.497+0000 7feb480fa640 1 -- 172.21.15.116:0/2525716796 shutdown_connections 2024-04-25T00:13:19.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.497+0000 7feb480fa640 1 --2- 172.21.15.116:0/2525716796 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7feb240678f0 0x7feb24069db0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:19.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.497+0000 7feb480fa640 1 --2- 172.21.15.116:0/2525716796 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7feb400c5280 0x7feb400c6850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:19.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.497+0000 7feb480fa640 1 --2- 172.21.15.116:0/2525716796 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feb40072c90 0x7feb401efd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:19.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.497+0000 7feb480fa640 1 --2- 172.21.15.116:0/2525716796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feb40072810 0x7feb401ef820 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:19.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.497+0000 7feb480fa640 1 -- 172.21.15.116:0/2525716796 >> 172.21.15.116:0/2525716796 conn(0x7feb4006c720 msgr2=0x7feb4014f5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:19.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.497+0000 7feb480fa640 1 -- 172.21.15.116:0/2525716796 shutdown_connections 2024-04-25T00:13:19.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:19.497+0000 7feb480fa640 1 -- 172.21.15.116:0/2525716796 wait complete. 2024-04-25T00:13:19.516 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T00:13:21.518 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:13:21.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.649+0000 7fa873e87640 1 Processor -- start 2024-04-25T00:13:21.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.649+0000 7fa873e87640 1 -- start start 2024-04-25T00:13:21.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.653+0000 7fa873e87640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa86c14c700 0x7fa86c0c5090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:21.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.653+0000 7fa873e87640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa86c0a9e60 con 0x7fa86c155b20 2024-04-25T00:13:21.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.653+0000 7fa873e87640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa86c0c3ef0 con 0x7fa86c0c55d0 2024-04-25T00:13:21.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.653+0000 7fa873e87640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fa86c1527b0 con 0x7fa86c14c700 2024-04-25T00:13:21.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.653+0000 7fa871c23640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa86c14c700 0x7fa86c0c5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:21.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.653+0000 7fa871c23640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa86c14c700 0x7fa86c0c5090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36458/0 (socket says 172.21.15.116:36458) 2024-04-25T00:13:21.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.653+0000 7fa871c23640 1 -- 172.21.15.116:0/2780397018 learned_addr learned my addr 172.21.15.116:0/2780397018 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:13:21.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.653+0000 7fa871c23640 1 -- 172.21.15.116:0/2780397018 >> v1:172.21.15.174:6789/0 conn(0x7fa86c0c55d0 legacy=0x7fa86c15ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:13:21.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.653+0000 7fa871c23640 1 -- 172.21.15.116:0/2780397018 >> v1:172.21.15.116:6789/0 conn(0x7fa86c155b20 legacy=0x7fa86c152c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:21.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.653+0000 7fa870c21640 1 -- 172.21.15.116:0/2780397018 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3511692871 0 0) 0x7fa86c0a9e60 con 0x7fa86c155b20 2024-04-25T00:13:21.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.653+0000 7fa871c23640 1 -- 172.21.15.116:0/2780397018 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa86c0c41d0 con 0x7fa86c14c700 2024-04-25T00:13:21.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.657+0000 7fa871c23640 1 --2- 172.21.15.116:0/2780397018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa86c14c700 0x7fa86c0c5090 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fa868009870 tx=0x7fa86801c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c72c66af4b887f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:21.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.657+0000 7fa870c21640 1 -- 172.21.15.116:0/2780397018 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8680b0070 con 0x7fa86c14c700 2024-04-25T00:13:21.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.657+0000 7fa870c21640 1 -- 172.21.15.116:0/2780397018 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8680ab600 con 0x7fa86c14c700 2024-04-25T00:13:21.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.657+0000 7fa870c21640 1 -- 172.21.15.116:0/2780397018 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8680ab9d0 con 0x7fa86c14c700 2024-04-25T00:13:21.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.657+0000 7fa873e87640 1 -- 172.21.15.116:0/2780397018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa86c14c700 msgr2=0x7fa86c0c5090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:21.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.657+0000 7fa873e87640 1 --2- 172.21.15.116:0/2780397018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa86c14c700 0x7fa86c0c5090 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fa868009870 tx=0x7fa86801c400 comp rx=0 tx=0).stop 2024-04-25T00:13:21.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.657+0000 7fa873e87640 1 -- 172.21.15.116:0/2780397018 shutdown_connections 2024-04-25T00:13:21.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.657+0000 7fa873e87640 1 --2- 172.21.15.116:0/2780397018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa86c14c700 0x7fa86c0c5090 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:21.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.657+0000 7fa873e87640 1 -- 172.21.15.116:0/2780397018 >> 172.21.15.116:0/2780397018 conn(0x7fa86c06c720 msgr2=0x7fa86c06cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:21.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.657+0000 7fa873e87640 1 -- 172.21.15.116:0/2780397018 shutdown_connections 2024-04-25T00:13:21.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.657+0000 7fa873e87640 1 -- 172.21.15.116:0/2780397018 wait complete. 2024-04-25T00:13:21.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.657+0000 7fa873e87640 1 Processor -- start 2024-04-25T00:13:21.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.657+0000 7fa873e87640 1 -- start start 2024-04-25T00:13:21.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.657+0000 7fa873e87640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa86c0c55d0 0x7fa86c1f1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:21.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.657+0000 7fa873e87640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa86c14c700 0x7fa86c0acdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:21.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.657+0000 7fa873e87640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa86c155b20 0x7fa86c1ef0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:21.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.657+0000 7fa873e87640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa86c161180 con 0x7fa86c14c700 2024-04-25T00:13:21.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.657+0000 7fa873e87640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa86c161000 con 0x7fa86c155b20 2024-04-25T00:13:21.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.657+0000 7fa873e87640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa86c161300 con 0x7fa86c0c55d0 2024-04-25T00:13:21.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.657+0000 7fa871422640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa86c14c700 0x7fa86c0acdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:21.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.657+0000 7fa871422640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa86c14c700 0x7fa86c0acdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36468/0 (socket says 172.21.15.116:36468) 2024-04-25T00:13:21.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.657+0000 7fa871422640 1 -- 172.21.15.116:0/749866146 learned_addr learned my addr 172.21.15.116:0/749866146 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:13:21.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.661+0000 7fa871422640 1 -- 172.21.15.116:0/749866146 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa86c0c55d0 msgr2=0x7fa86c1f1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:21.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.661+0000 7fa871422640 1 --2- 172.21.15.116:0/749866146 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa86c0c55d0 0x7fa86c1f1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:21.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.661+0000 7fa871422640 1 -- 172.21.15.116:0/749866146 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa86c155b20 msgr2=0x7fa86c1ef0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:21.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.661+0000 7fa871422640 1 --2- 172.21.15.116:0/749866146 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa86c155b20 0x7fa86c1ef0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:21.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.661+0000 7fa871422640 1 -- 172.21.15.116:0/749866146 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa86c146f90 con 0x7fa86c14c700 2024-04-25T00:13:21.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.661+0000 7fa871422640 1 --2- 172.21.15.116:0/749866146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa86c14c700 0x7fa86c0acdf0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fa85c00c4d0 tx=0x7fa85c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:21.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.661+0000 7fa8627fc640 1 -- 172.21.15.116:0/749866146 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa85c017070 con 0x7fa86c14c700 2024-04-25T00:13:21.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.661+0000 7fa8627fc640 1 -- 172.21.15.116:0/749866146 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa85c005480 con 0x7fa86c14c700 2024-04-25T00:13:21.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.661+0000 7fa873e87640 1 -- 172.21.15.116:0/749866146 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa86c148210 con 0x7fa86c14c700 2024-04-25T00:13:21.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.661+0000 7fa8627fc640 1 -- 172.21.15.116:0/749866146 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa85c003840 con 0x7fa86c14c700 2024-04-25T00:13:21.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.661+0000 7fa873e87640 1 -- 172.21.15.116:0/749866146 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa86c0ab460 con 0x7fa86c14c700 2024-04-25T00:13:21.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.661+0000 7fa873e87640 1 -- 172.21.15.116:0/749866146 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa86c068a60 con 0x7fa86c14c700 2024-04-25T00:13:21.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.661+0000 7fa8627fc640 1 -- 172.21.15.116:0/749866146 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa85c01a070 con 0x7fa86c14c700 2024-04-25T00:13:21.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.661+0000 7fa8627fc640 1 --2- 172.21.15.116:0/749866146 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa8500678f0 0x7fa850069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:21.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.661+0000 7fa8627fc640 1 -- 172.21.15.116:0/749866146 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 8042+0+0 (secure 0 0 0) 0x7fa85c0859c0 con 0x7fa86c14c700 2024-04-25T00:13:21.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.661+0000 7fa871c23640 1 --2- 172.21.15.116:0/749866146 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa8500678f0 0x7fa850069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:21.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.661+0000 7fa871c23640 1 --2- 172.21.15.116:0/749866146 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa8500678f0 0x7fa850069db0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fa8680ad000 tx=0x7fa868008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:21.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.669+0000 7fa8627fc640 1 -- 172.21.15.116:0/749866146 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fa85c055c90 con 0x7fa86c14c700 2024-04-25T00:13:21.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.933+0000 7fa873e87640 1 -- 172.21.15.116:0/749866146 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa86c0aa0d0 con 0x7fa86c14c700 2024-04-25T00:13:21.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.937+0000 7fa8627fc640 1 -- 172.21.15.116:0/749866146 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) ==== 94+0+6496 (secure 0 0 0) 0x7fa85c055590 con 0x7fa86c14c700 2024-04-25T00:13:21.936 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:13:21.936 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82},{"gid":6582,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/462190523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":462190523},{"type":"v1","addr":"172.21.15.116:6835","nonce":462190523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:12:48.771611+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":88,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6833/365773583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":365773583},{"type":"v1","addr":"172.21.15.116:6833","nonce":365773583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":1},{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:12:23.643548+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5461},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5461":{"gid":5461,"name":"b","rank":0,"incarnation":83,"state":"up:active","state_seq":30,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5461,"qdb_cluster":[5461]},"id":2}]} 2024-04-25T00:13:21.936 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 92 2024-04-25T00:13:21.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.937+0000 7fa873e87640 1 -- 172.21.15.116:0/749866146 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa8500678f0 msgr2=0x7fa850069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:21.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.937+0000 7fa873e87640 1 --2- 172.21.15.116:0/749866146 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa8500678f0 0x7fa850069db0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fa8680ad000 tx=0x7fa868008040 comp rx=0 tx=0).stop 2024-04-25T00:13:21.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.937+0000 7fa873e87640 1 -- 172.21.15.116:0/749866146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa86c14c700 msgr2=0x7fa86c0acdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:21.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.937+0000 7fa873e87640 1 --2- 172.21.15.116:0/749866146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa86c14c700 0x7fa86c0acdf0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fa85c00c4d0 tx=0x7fa85c00c990 comp rx=0 tx=0).stop 2024-04-25T00:13:21.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.937+0000 7fa873e87640 1 -- 172.21.15.116:0/749866146 shutdown_connections 2024-04-25T00:13:21.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.937+0000 7fa873e87640 1 --2- 172.21.15.116:0/749866146 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa8500678f0 0x7fa850069db0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:21.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.937+0000 7fa873e87640 1 --2- 172.21.15.116:0/749866146 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa86c155b20 0x7fa86c1ef0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:21.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.937+0000 7fa873e87640 1 --2- 172.21.15.116:0/749866146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa86c14c700 0x7fa86c0acdf0 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:21.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.937+0000 7fa873e87640 1 --2- 172.21.15.116:0/749866146 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa86c0c55d0 0x7fa86c1f1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:21.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.937+0000 7fa873e87640 1 -- 172.21.15.116:0/749866146 >> 172.21.15.116:0/749866146 conn(0x7fa86c06c720 msgr2=0x7fa86c152c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:21.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.937+0000 7fa873e87640 1 -- 172.21.15.116:0/749866146 shutdown_connections 2024-04-25T00:13:21.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:21.937+0000 7fa873e87640 1 -- 172.21.15.116:0/749866146 wait complete. 2024-04-25T00:13:21.955 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T00:13:23.957 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:13:24.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.097+0000 7f9eeb730640 1 Processor -- start 2024-04-25T00:13:24.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.097+0000 7f9eeb730640 1 -- start start 2024-04-25T00:13:24.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.097+0000 7f9eeb730640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f9ee40c5280 0x7f9ee4150840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:24.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.097+0000 7f9eeb730640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9ee4072c90 0x7f9ee4150d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:24.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.097+0000 7f9eeb730640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9ee4072810 0x7f9ee4153760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:24.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.097+0000 7f9eeb730640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f9ee4157c20 con 0x7f9ee4072810 2024-04-25T00:13:24.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.097+0000 7f9eeb730640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f9ee40c2f70 con 0x7f9ee40c5280 2024-04-25T00:13:24.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.097+0000 7f9eeb730640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f9ee40c2c40 con 0x7f9ee4072c90 2024-04-25T00:13:24.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.097+0000 7f9ee9ccd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9ee4072810 0x7f9ee4153760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:24.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.097+0000 7f9ee9ccd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9ee4072810 0x7f9ee4153760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36470/0 (socket says 172.21.15.116:36470) 2024-04-25T00:13:24.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.097+0000 7f9ee9ccd640 1 -- 172.21.15.116:0/3567263104 learned_addr learned my addr 172.21.15.116:0/3567263104 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:13:24.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.097+0000 7f9ee8ccb640 1 --2- 172.21.15.116:0/3567263104 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9ee4072c90 0x7f9ee4150d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:24.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.097+0000 7f9ee94cc640 1 --2- 172.21.15.116:0/3567263104 >> v2:172.21.15.174:3300/0 conn(0x7f9ee40c5280 0x7f9ee4150840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:24.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.097+0000 7f9ee9ccd640 1 -- 172.21.15.116:0/3567263104 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9ee4072c90 msgr2=0x7f9ee4150d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:13:24.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.097+0000 7f9ee9ccd640 1 --2- 172.21.15.116:0/3567263104 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9ee4072c90 0x7f9ee4150d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:24.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.097+0000 7f9ee9ccd640 1 -- 172.21.15.116:0/3567263104 >> v2:172.21.15.174:3300/0 conn(0x7f9ee40c5280 msgr2=0x7f9ee4150840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:13:24.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.097+0000 7f9ee9ccd640 1 --2- 172.21.15.116:0/3567263104 >> v2:172.21.15.174:3300/0 conn(0x7f9ee40c5280 0x7f9ee4150840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:24.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.097+0000 7f9ee9ccd640 1 -- 172.21.15.116:0/3567263104 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ee41512c0 con 0x7f9ee4072810 2024-04-25T00:13:24.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.097+0000 7f9ee8ccb640 1 --2- 172.21.15.116:0/3567263104 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9ee4072c90 0x7f9ee4150d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:13:24.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.101+0000 7f9ee9ccd640 1 --2- 172.21.15.116:0/3567263104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ee4072810 0x7f9ee4153760 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f9ee40727a0 tx=0x7f9edc01c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50b1e8c85d8d3cd4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:24.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.101+0000 7f9edbfff640 1 -- 172.21.15.116:0/3567263104 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9edc0a57b0 con 0x7f9ee4072810 2024-04-25T00:13:24.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.101+0000 7f9edbfff640 1 -- 172.21.15.116:0/3567263104 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9edc0a5950 con 0x7f9ee4072810 2024-04-25T00:13:24.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.101+0000 7f9edbfff640 1 -- 172.21.15.116:0/3567263104 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9edc0a5d20 con 0x7f9ee4072810 2024-04-25T00:13:24.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.101+0000 7f9eeb730640 1 -- 172.21.15.116:0/3567263104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ee4072810 msgr2=0x7f9ee4153760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:24.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.101+0000 7f9eeb730640 1 --2- 172.21.15.116:0/3567263104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ee4072810 0x7f9ee4153760 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f9ee40727a0 tx=0x7f9edc01c930 comp rx=0 tx=0).stop 2024-04-25T00:13:24.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.101+0000 7f9eeb730640 1 -- 172.21.15.116:0/3567263104 shutdown_connections 2024-04-25T00:13:24.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.101+0000 7f9eeb730640 1 --2- 172.21.15.116:0/3567263104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ee4072810 0x7f9ee4153760 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:24.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.101+0000 7f9eeb730640 1 --2- 172.21.15.116:0/3567263104 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9ee4072c90 0x7f9ee4150d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:24.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.101+0000 7f9eeb730640 1 --2- 172.21.15.116:0/3567263104 >> v2:172.21.15.174:3300/0 conn(0x7f9ee40c5280 0x7f9ee4150840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:24.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.101+0000 7f9eeb730640 1 -- 172.21.15.116:0/3567263104 >> 172.21.15.116:0/3567263104 conn(0x7f9ee406c720 msgr2=0x7f9ee406cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:24.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.101+0000 7f9eeb730640 1 -- 172.21.15.116:0/3567263104 shutdown_connections 2024-04-25T00:13:24.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.101+0000 7f9eeb730640 1 -- 172.21.15.116:0/3567263104 wait complete. 2024-04-25T00:13:24.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.101+0000 7f9eeb730640 1 Processor -- start 2024-04-25T00:13:24.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.101+0000 7f9eeb730640 1 -- start start 2024-04-25T00:13:24.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.105+0000 7f9eeb730640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ee4072810 0x7f9ee4152740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:24.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.105+0000 7f9eeb730640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9ee4072c90 0x7f9ee4156b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:24.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.105+0000 7f9eeb730640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ee40c5280 0x7f9ee41570b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:24.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.105+0000 7f9eeb730640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9ee406b210 con 0x7f9ee40c5280 2024-04-25T00:13:24.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.105+0000 7f9eeb730640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9ee40c5870 con 0x7f9ee4072810 2024-04-25T00:13:24.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.105+0000 7f9eeb730640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f9ee406b390 con 0x7f9ee4072c90 2024-04-25T00:13:24.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.105+0000 7f9ee9ccd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ee40c5280 0x7f9ee41570b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:24.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.105+0000 7f9ee9ccd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ee40c5280 0x7f9ee41570b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36478/0 (socket says 172.21.15.116:36478) 2024-04-25T00:13:24.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.105+0000 7f9ee9ccd640 1 -- 172.21.15.116:0/105078825 learned_addr learned my addr 172.21.15.116:0/105078825 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:13:24.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.105+0000 7f9ee94cc640 1 --2- 172.21.15.116:0/105078825 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ee4072810 0x7f9ee4152740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:24.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.105+0000 7f9ee8ccb640 1 --2- 172.21.15.116:0/105078825 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9ee4072c90 0x7f9ee4156b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:24.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.105+0000 7f9ee9ccd640 1 -- 172.21.15.116:0/105078825 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9ee4072c90 msgr2=0x7f9ee4156b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:24.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.105+0000 7f9ee9ccd640 1 --2- 172.21.15.116:0/105078825 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9ee4072c90 0x7f9ee4156b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:24.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.105+0000 7f9ee9ccd640 1 -- 172.21.15.116:0/105078825 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ee4072810 msgr2=0x7f9ee4152740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:24.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.105+0000 7f9ee9ccd640 1 --2- 172.21.15.116:0/105078825 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ee4072810 0x7f9ee4152740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:24.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.105+0000 7f9ee9ccd640 1 -- 172.21.15.116:0/105078825 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ee41ecdd0 con 0x7f9ee40c5280 2024-04-25T00:13:24.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.105+0000 7f9ee9ccd640 1 --2- 172.21.15.116:0/105078825 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ee40c5280 0x7f9ee41570b0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f9edc0af040 tx=0x7f9edc0196d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:24.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.105+0000 7f9ed9ffb640 1 -- 172.21.15.116:0/105078825 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9edc0bc020 con 0x7f9ee40c5280 2024-04-25T00:13:24.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.105+0000 7f9ed9ffb640 1 -- 172.21.15.116:0/105078825 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9edc0b8670 con 0x7f9ee40c5280 2024-04-25T00:13:24.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.105+0000 7f9ed9ffb640 1 -- 172.21.15.116:0/105078825 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9edc0b8a60 con 0x7f9ee40c5280 2024-04-25T00:13:24.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.105+0000 7f9eeb730640 1 -- 172.21.15.116:0/105078825 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ee41ed0b0 con 0x7f9ee40c5280 2024-04-25T00:13:24.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.105+0000 7f9eeb730640 1 -- 172.21.15.116:0/105078825 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ee4148910 con 0x7f9ee40c5280 2024-04-25T00:13:24.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.105+0000 7f9eeb730640 1 -- 172.21.15.116:0/105078825 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ee414ce60 con 0x7f9ee40c5280 2024-04-25T00:13:24.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.105+0000 7f9ed9ffb640 1 -- 172.21.15.116:0/105078825 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f9edc0bf020 con 0x7f9ee40c5280 2024-04-25T00:13:24.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.105+0000 7f9ed9ffb640 1 --2- 172.21.15.116:0/105078825 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9ec80678c0 0x7f9ec8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:24.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.105+0000 7f9ed9ffb640 1 -- 172.21.15.116:0/105078825 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 8042+0+0 (secure 0 0 0) 0x7f9edc1216a0 con 0x7f9ee40c5280 2024-04-25T00:13:24.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.105+0000 7f9ee94cc640 1 --2- 172.21.15.116:0/105078825 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9ec80678c0 0x7f9ec8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:24.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.109+0000 7f9ee94cc640 1 --2- 172.21.15.116:0/105078825 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9ec80678c0 0x7f9ec8069d80 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f9ee0007940 tx=0x7f9ee0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:24.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.109+0000 7f9ed9ffb640 1 -- 172.21.15.116:0/105078825 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f9edc0c6030 con 0x7f9ee40c5280 2024-04-25T00:13:24.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.353+0000 7f9eeb730640 1 -- 172.21.15.116:0/105078825 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9ee4068a60 con 0x7f9ee40c5280 2024-04-25T00:13:24.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.353+0000 7f9ed9ffb640 1 -- 172.21.15.116:0/105078825 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) ==== 94+0+6496 (secure 0 0 0) 0x7f9edc0165e0 con 0x7f9ee40c5280 2024-04-25T00:13:24.354 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:13:24.354 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82},{"gid":6582,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/462190523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":462190523},{"type":"v1","addr":"172.21.15.116:6835","nonce":462190523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:12:48.771611+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":88,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6833/365773583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":365773583},{"type":"v1","addr":"172.21.15.116:6833","nonce":365773583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":1},{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:12:23.643548+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5461},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5461":{"gid":5461,"name":"b","rank":0,"incarnation":83,"state":"up:active","state_seq":30,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5461,"qdb_cluster":[5461]},"id":2}]} 2024-04-25T00:13:24.354 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 92 2024-04-25T00:13:24.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.357+0000 7f9eeb730640 1 -- 172.21.15.116:0/105078825 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9ec80678c0 msgr2=0x7f9ec8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:24.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.357+0000 7f9eeb730640 1 --2- 172.21.15.116:0/105078825 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9ec80678c0 0x7f9ec8069d80 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f9ee0007940 tx=0x7f9ee0008040 comp rx=0 tx=0).stop 2024-04-25T00:13:24.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.357+0000 7f9eeb730640 1 -- 172.21.15.116:0/105078825 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ee40c5280 msgr2=0x7f9ee41570b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:24.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.357+0000 7f9eeb730640 1 --2- 172.21.15.116:0/105078825 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ee40c5280 0x7f9ee41570b0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f9edc0af040 tx=0x7f9edc0196d0 comp rx=0 tx=0).stop 2024-04-25T00:13:24.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.357+0000 7f9eeb730640 1 -- 172.21.15.116:0/105078825 shutdown_connections 2024-04-25T00:13:24.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.357+0000 7f9eeb730640 1 --2- 172.21.15.116:0/105078825 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9ec80678c0 0x7f9ec8069d80 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:24.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.357+0000 7f9eeb730640 1 --2- 172.21.15.116:0/105078825 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ee40c5280 0x7f9ee41570b0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:24.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.357+0000 7f9eeb730640 1 --2- 172.21.15.116:0/105078825 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9ee4072c90 0x7f9ee4156b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:24.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.357+0000 7f9eeb730640 1 --2- 172.21.15.116:0/105078825 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ee4072810 0x7f9ee4152740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:24.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.357+0000 7f9eeb730640 1 -- 172.21.15.116:0/105078825 >> 172.21.15.116:0/105078825 conn(0x7f9ee406c720 msgr2=0x7f9ee4150610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:24.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.357+0000 7f9eeb730640 1 -- 172.21.15.116:0/105078825 shutdown_connections 2024-04-25T00:13:24.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:24.357+0000 7f9eeb730640 1 -- 172.21.15.116:0/105078825 wait complete. 2024-04-25T00:13:24.375 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T00:13:26.377 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:13:26.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.497+0000 7f4d1969f640 1 Processor -- start 2024-04-25T00:13:26.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.497+0000 7f4d1969f640 1 -- start start 2024-04-25T00:13:26.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.497+0000 7f4d1969f640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f4d14155b20 0x7f4d14152c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:26.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.497+0000 7f4d1969f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d140a9e60 con 0x7f4d1414c700 2024-04-25T00:13:26.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.497+0000 7f4d1969f640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d140c3ef0 con 0x7f4d140aacd0 2024-04-25T00:13:26.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.497+0000 7f4d1969f640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f4d141527b0 con 0x7f4d14155b20 2024-04-25T00:13:26.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.497+0000 7f4d12ffd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f4d1414c700 0x7f4d140c5090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:45268/0 (socket says 172.21.15.116:45268) 2024-04-25T00:13:26.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.497+0000 7f4d12ffd640 1 -- 172.21.15.116:0/2897129033 learned_addr learned my addr 172.21.15.116:0/2897129033 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:13:26.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.497+0000 7f4d127fc640 1 --2- 172.21.15.116:0/2897129033 >> v2:172.21.15.174:3300/0 conn(0x7f4d14155b20 0x7f4d14152c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:26.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.497+0000 7f4d11ffb640 1 -- 172.21.15.116:0/2897129033 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3637061240 0 0) 0x7f4d140a9e60 con 0x7f4d1414c700 2024-04-25T00:13:26.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.497+0000 7f4d11ffb640 1 -- 172.21.15.116:0/2897129033 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4cfc003530 con 0x7f4d1414c700 2024-04-25T00:13:26.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.497+0000 7f4d11ffb640 1 -- 172.21.15.116:0/2897129033 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1644791646 0 0) 0x7f4cfc003530 con 0x7f4d1414c700 2024-04-25T00:13:26.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.497+0000 7f4d11ffb640 1 -- 172.21.15.116:0/2897129033 >> v2:172.21.15.174:3300/0 conn(0x7f4d14155b20 msgr2=0x7f4d14152c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:13:26.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.497+0000 7f4d11ffb640 1 --2- 172.21.15.116:0/2897129033 >> v2:172.21.15.174:3300/0 conn(0x7f4d14155b20 0x7f4d14152c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:26.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.497+0000 7f4d11ffb640 1 -- 172.21.15.116:0/2897129033 >> v1:172.21.15.174:6789/0 conn(0x7f4d140aacd0 legacy=0x7f4d1415ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:26.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.497+0000 7f4d11ffb640 1 -- 172.21.15.116:0/2897129033 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d140c41d0 con 0x7f4d1414c700 2024-04-25T00:13:26.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.497+0000 7f4d11ffb640 1 -- 172.21.15.116:0/2897129033 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f4d04001680 con 0x7f4d1414c700 2024-04-25T00:13:26.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.497+0000 7f4d11ffb640 1 -- 172.21.15.116:0/2897129033 >> v1:172.21.15.116:6789/0 conn(0x7f4d1414c700 legacy=0x7f4d140c5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:26.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.497+0000 7f4d11ffb640 1 --2- 172.21.15.116:0/2897129033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4cfc0039c0 0x7f4cfc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:26.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.497+0000 7f4d11ffb640 1 --2- 172.21.15.116:0/2897129033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cfc004810 0x7f4cfc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:26.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.497+0000 7f4d11ffb640 1 --2- 172.21.15.116:0/2897129033 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4cfc0052f0 0x7f4cfc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:26.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.497+0000 7f4d11ffb640 1 -- 172.21.15.116:0/2897129033 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4d141527b0 con 0x7f4cfc004810 2024-04-25T00:13:26.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.497+0000 7f4d11ffb640 1 -- 172.21.15.116:0/2897129033 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4cfc00c1c0 con 0x7f4cfc0039c0 2024-04-25T00:13:26.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.497+0000 7f4d11ffb640 1 -- 172.21.15.116:0/2897129033 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f4cfc00c3c0 con 0x7f4cfc0052f0 2024-04-25T00:13:26.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.497+0000 7f4d127fc640 1 --2- 172.21.15.116:0/2897129033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cfc004810 0x7f4cfc004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:26.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.497+0000 7f4d12ffd640 1 --2- 172.21.15.116:0/2897129033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4cfc0039c0 0x7f4cfc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:26.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.497+0000 7f4d127fc640 1 -- 172.21.15.116:0/2897129033 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4cfc0052f0 msgr2=0x7f4cfc00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:13:26.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.497+0000 7f4d127fc640 1 --2- 172.21.15.116:0/2897129033 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4cfc0052f0 0x7f4cfc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:26.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.497+0000 7f4d127fc640 1 -- 172.21.15.116:0/2897129033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4cfc0039c0 msgr2=0x7f4cfc003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:13:26.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.497+0000 7f4d127fc640 1 --2- 172.21.15.116:0/2897129033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4cfc0039c0 0x7f4cfc003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:26.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.497+0000 7f4d127fc640 1 -- 172.21.15.116:0/2897129033 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cfc003530 con 0x7f4cfc004810 2024-04-25T00:13:26.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.497+0000 7f4d137fe640 1 -- 172.21.15.116:0/2897129033 reap_dead start 2024-04-25T00:13:26.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.497+0000 7f4d12ffd640 1 --2- 172.21.15.116:0/2897129033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4cfc0039c0 0x7f4cfc003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:13:26.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.501+0000 7f4d127fc640 1 --2- 172.21.15.116:0/2897129033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cfc004810 0x7f4cfc004c90 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f4d08009ab0 tx=0x7f4d0801ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98ee8746c42ed7f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:26.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.501+0000 7f4d11ffb640 1 -- 172.21.15.116:0/2897129033 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d080096b0 con 0x7f4cfc004810 2024-04-25T00:13:26.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.501+0000 7f4d11ffb640 1 -- 172.21.15.116:0/2897129033 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d080a9440 con 0x7f4cfc004810 2024-04-25T00:13:26.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.501+0000 7f4d11ffb640 1 -- 172.21.15.116:0/2897129033 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d080a9630 con 0x7f4cfc004810 2024-04-25T00:13:26.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.501+0000 7f4d1969f640 1 -- 172.21.15.116:0/2897129033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cfc004810 msgr2=0x7f4cfc004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:26.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.501+0000 7f4d1969f640 1 --2- 172.21.15.116:0/2897129033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cfc004810 0x7f4cfc004c90 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f4d08009ab0 tx=0x7f4d0801ca70 comp rx=0 tx=0).stop 2024-04-25T00:13:26.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.501+0000 7f4d1969f640 1 -- 172.21.15.116:0/2897129033 shutdown_connections 2024-04-25T00:13:26.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.501+0000 7f4d1969f640 1 --2- 172.21.15.116:0/2897129033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4cfc004810 0x7f4cfc004c90 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:26.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.501+0000 7f4d1969f640 1 -- 172.21.15.116:0/2897129033 >> 172.21.15.116:0/2897129033 conn(0x7f4d1406c720 msgr2=0x7f4d1406cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:26.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.501+0000 7f4d1969f640 1 -- 172.21.15.116:0/2897129033 shutdown_connections 2024-04-25T00:13:26.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.501+0000 7f4d1969f640 1 -- 172.21.15.116:0/2897129033 wait complete. 2024-04-25T00:13:26.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.501+0000 7f4d1969f640 1 Processor -- start 2024-04-25T00:13:26.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.501+0000 7f4d1969f640 1 -- start start 2024-04-25T00:13:26.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.505+0000 7f4d1969f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4cfc004810 0x7f4d141ee5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:26.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.505+0000 7f4d1969f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d1406b470 0x7f4d1413f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:26.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.505+0000 7f4d1969f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4d1413fc70 0x7f4d14151420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:26.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.505+0000 7f4d1969f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4d14152fd0 con 0x7f4d1406b470 2024-04-25T00:13:26.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.505+0000 7f4d1969f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4d14152e50 con 0x7f4cfc004810 2024-04-25T00:13:26.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.505+0000 7f4d1969f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f4d14153150 con 0x7f4d1413fc70 2024-04-25T00:13:26.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.505+0000 7f4d127fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d1406b470 0x7f4d1413f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:26.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.505+0000 7f4d127fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d1406b470 0x7f4d1413f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55376/0 (socket says 172.21.15.116:55376) 2024-04-25T00:13:26.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.505+0000 7f4d127fc640 1 -- 172.21.15.116:0/831593890 learned_addr learned my addr 172.21.15.116:0/831593890 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:13:26.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.505+0000 7f4d12ffd640 1 --2- 172.21.15.116:0/831593890 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4cfc004810 0x7f4d141ee5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:26.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.505+0000 7f4d137fe640 1 --2- 172.21.15.116:0/831593890 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4d1413fc70 0x7f4d14151420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:26.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.505+0000 7f4d127fc640 1 -- 172.21.15.116:0/831593890 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4d1413fc70 msgr2=0x7f4d14151420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:26.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.505+0000 7f4d127fc640 1 --2- 172.21.15.116:0/831593890 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4d1413fc70 0x7f4d14151420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:26.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.505+0000 7f4d127fc640 1 -- 172.21.15.116:0/831593890 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4cfc004810 msgr2=0x7f4d141ee5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:26.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.505+0000 7f4d127fc640 1 --2- 172.21.15.116:0/831593890 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4cfc004810 0x7f4d141ee5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:26.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.505+0000 7f4d127fc640 1 -- 172.21.15.116:0/831593890 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d141f1df0 con 0x7f4d1406b470 2024-04-25T00:13:26.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.505+0000 7f4d137fe640 1 --2- 172.21.15.116:0/831593890 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4d1413fc70 0x7f4d14151420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:13:26.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.505+0000 7f4d127fc640 1 --2- 172.21.15.116:0/831593890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d1406b470 0x7f4d1413f730 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f4d080ab040 tx=0x7f4d08019810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:26.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.505+0000 7f4cfb7fe640 1 -- 172.21.15.116:0/831593890 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d080c1070 con 0x7f4d1406b470 2024-04-25T00:13:26.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.505+0000 7f4cfb7fe640 1 -- 172.21.15.116:0/831593890 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d0800f000 con 0x7f4d1406b470 2024-04-25T00:13:26.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.505+0000 7f4d1969f640 1 -- 172.21.15.116:0/831593890 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d141f20d0 con 0x7f4d1406b470 2024-04-25T00:13:26.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.505+0000 7f4cfb7fe640 1 -- 172.21.15.116:0/831593890 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d080a9db0 con 0x7f4d1406b470 2024-04-25T00:13:26.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.505+0000 7f4d1969f640 1 -- 172.21.15.116:0/831593890 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d141f2690 con 0x7f4d1406b470 2024-04-25T00:13:26.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.505+0000 7f4d1969f640 1 -- 172.21.15.116:0/831593890 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d14068a60 con 0x7f4d1406b470 2024-04-25T00:13:26.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.505+0000 7f4cfb7fe640 1 -- 172.21.15.116:0/831593890 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4d080c4070 con 0x7f4d1406b470 2024-04-25T00:13:26.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.505+0000 7f4cfb7fe640 1 --2- 172.21.15.116:0/831593890 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4d000678f0 0x7f4d00069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:26.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.505+0000 7f4cfb7fe640 1 -- 172.21.15.116:0/831593890 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 8042+0+0 (secure 0 0 0) 0x7f4d08120920 con 0x7f4d1406b470 2024-04-25T00:13:26.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.505+0000 7f4d12ffd640 1 --2- 172.21.15.116:0/831593890 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4d000678f0 0x7f4d00069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:26.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.509+0000 7f4d12ffd640 1 --2- 172.21.15.116:0/831593890 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4d000678f0 0x7f4d00069db0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f4d04001210 tx=0x7f4d04009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:26.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.513+0000 7f4cfb7fe640 1 -- 172.21.15.116:0/831593890 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f4d080ad020 con 0x7f4d1406b470 2024-04-25T00:13:26.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.757+0000 7f4d1969f640 1 -- 172.21.15.116:0/831593890 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4d140ad630 con 0x7f4d1406b470 2024-04-25T00:13:26.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.757+0000 7f4cfb7fe640 1 -- 172.21.15.116:0/831593890 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) ==== 94+0+6496 (secure 0 0 0) 0x7f4d08016780 con 0x7f4d1406b470 2024-04-25T00:13:26.759 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:13:26.759 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82},{"gid":6582,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/462190523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":462190523},{"type":"v1","addr":"172.21.15.116:6835","nonce":462190523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:12:48.771611+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":88,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6833/365773583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":365773583},{"type":"v1","addr":"172.21.15.116:6833","nonce":365773583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":1},{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:12:23.643548+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5461},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5461":{"gid":5461,"name":"b","rank":0,"incarnation":83,"state":"up:active","state_seq":30,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5461,"qdb_cluster":[5461]},"id":2}]} 2024-04-25T00:13:26.760 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 92 2024-04-25T00:13:26.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.761+0000 7f4d1969f640 1 -- 172.21.15.116:0/831593890 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4d000678f0 msgr2=0x7f4d00069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:26.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.761+0000 7f4d1969f640 1 --2- 172.21.15.116:0/831593890 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4d000678f0 0x7f4d00069db0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f4d04001210 tx=0x7f4d04009040 comp rx=0 tx=0).stop 2024-04-25T00:13:26.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.761+0000 7f4d1969f640 1 -- 172.21.15.116:0/831593890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d1406b470 msgr2=0x7f4d1413f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:26.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.761+0000 7f4d1969f640 1 --2- 172.21.15.116:0/831593890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d1406b470 0x7f4d1413f730 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f4d080ab040 tx=0x7f4d08019810 comp rx=0 tx=0).stop 2024-04-25T00:13:26.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.761+0000 7f4d1969f640 1 -- 172.21.15.116:0/831593890 shutdown_connections 2024-04-25T00:13:26.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.761+0000 7f4d1969f640 1 --2- 172.21.15.116:0/831593890 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4d000678f0 0x7f4d00069db0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:26.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.761+0000 7f4d1969f640 1 --2- 172.21.15.116:0/831593890 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4d1413fc70 0x7f4d14151420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:26.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.761+0000 7f4d1969f640 1 --2- 172.21.15.116:0/831593890 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4d1406b470 0x7f4d1413f730 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:26.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.761+0000 7f4d1969f640 1 --2- 172.21.15.116:0/831593890 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4cfc004810 0x7f4d141ee5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:26.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.761+0000 7f4d1969f640 1 -- 172.21.15.116:0/831593890 >> 172.21.15.116:0/831593890 conn(0x7f4d1406c720 msgr2=0x7f4d1414bd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:26.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.761+0000 7f4d1969f640 1 -- 172.21.15.116:0/831593890 shutdown_connections 2024-04-25T00:13:26.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:26.761+0000 7f4d1969f640 1 -- 172.21.15.116:0/831593890 wait complete. 2024-04-25T00:13:26.780 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T00:13:28.781 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:13:28.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.921+0000 7f51f7777640 1 Processor -- start 2024-04-25T00:13:28.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.921+0000 7f51f7777640 1 -- start start 2024-04-25T00:13:28.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.921+0000 7f51f7777640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f51f00c8740 0x7f51f00711e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:28.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.921+0000 7f51f7777640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f51f00c53b0 0x7f51f0152d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:28.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.921+0000 7f51f7777640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51f00c23f0 con 0x7f51f01532a0 2024-04-25T00:13:28.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.921+0000 7f51f7777640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f51f0158f20 con 0x7f51f00c53b0 2024-04-25T00:13:28.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.921+0000 7f51f7777640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f51f00c2690 con 0x7f51f00c8740 2024-04-25T00:13:28.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.921+0000 7f51f5d14640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f51f01532a0 0x7f51f0155da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:45276/0 (socket says 172.21.15.116:45276) 2024-04-25T00:13:28.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.921+0000 7f51f5d14640 1 -- 172.21.15.116:0/80810658 learned_addr learned my addr 172.21.15.116:0/80810658 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:13:28.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.921+0000 7f51f4d12640 1 --2- 172.21.15.116:0/80810658 >> v2:172.21.15.116:3300/0 conn(0x7f51f00c53b0 0x7f51f0152d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:28.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.921+0000 7f51f5513640 1 --2- 172.21.15.116:0/80810658 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f51f00c8740 0x7f51f00711e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:28.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.921+0000 7f51f4d12640 1 -- 172.21.15.116:0/80810658 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f51f00c8740 msgr2=0x7f51f00711e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:13:28.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.921+0000 7f51f4d12640 1 --2- 172.21.15.116:0/80810658 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f51f00c8740 0x7f51f00711e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:28.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.921+0000 7f51f4d12640 1 -- 172.21.15.116:0/80810658 >> v1:172.21.15.116:6789/0 conn(0x7f51f01532a0 legacy=0x7f51f0155da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:28.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.921+0000 7f51e7fff640 1 -- 172.21.15.116:0/80810658 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4130055619 0 0) 0x7f51f00c23f0 con 0x7f51f01532a0 2024-04-25T00:13:28.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.921+0000 7f51f4d12640 1 -- 172.21.15.116:0/80810658 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51f00c2970 con 0x7f51f00c53b0 2024-04-25T00:13:28.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.925+0000 7f51f4d12640 1 --2- 172.21.15.116:0/80810658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51f00c53b0 0x7f51f0152d60 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f51e00097c0 tx=0x7f51e001c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=520b83129b67b59a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:28.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.925+0000 7f51e7fff640 1 -- 172.21.15.116:0/80810658 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e00b0070 con 0x7f51f00c53b0 2024-04-25T00:13:28.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.925+0000 7f51e7fff640 1 -- 172.21.15.116:0/80810658 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51e00ab600 con 0x7f51f00c53b0 2024-04-25T00:13:28.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.925+0000 7f51e7fff640 1 -- 172.21.15.116:0/80810658 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e00ab9d0 con 0x7f51f00c53b0 2024-04-25T00:13:28.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.925+0000 7f51f7777640 1 -- 172.21.15.116:0/80810658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51f00c53b0 msgr2=0x7f51f0152d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:28.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.925+0000 7f51f7777640 1 --2- 172.21.15.116:0/80810658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51f00c53b0 0x7f51f0152d60 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f51e00097c0 tx=0x7f51e001c380 comp rx=0 tx=0).stop 2024-04-25T00:13:28.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.925+0000 7f51f7777640 1 -- 172.21.15.116:0/80810658 shutdown_connections 2024-04-25T00:13:28.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.925+0000 7f51f7777640 1 --2- 172.21.15.116:0/80810658 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51f00c53b0 0x7f51f0152d60 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:28.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.925+0000 7f51f7777640 1 --2- 172.21.15.116:0/80810658 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f51f00c8740 0x7f51f00711e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:28.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.925+0000 7f51f7777640 1 -- 172.21.15.116:0/80810658 >> 172.21.15.116:0/80810658 conn(0x7f51f006f0c0 msgr2=0x7f51f006f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:28.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.925+0000 7f51f7777640 1 -- 172.21.15.116:0/80810658 shutdown_connections 2024-04-25T00:13:28.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.925+0000 7f51f7777640 1 -- 172.21.15.116:0/80810658 wait complete. 2024-04-25T00:13:28.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.925+0000 7f51f7777640 1 Processor -- start 2024-04-25T00:13:28.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.925+0000 7f51f7777640 1 -- start start 2024-04-25T00:13:28.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.929+0000 7f51f7777640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f51f00c53b0 0x7f51f014a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:28.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.929+0000 7f51f7777640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51f00c8740 0x7f51f01f1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:28.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.929+0000 7f51f7777640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f51f01532a0 0x7f51f006c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:28.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.929+0000 7f51f7777640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f51f0071930 con 0x7f51f00c8740 2024-04-25T00:13:28.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.929+0000 7f51f7777640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f51f00c8d30 con 0x7f51f00c53b0 2024-04-25T00:13:28.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.929+0000 7f51f7777640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f51f01568c0 con 0x7f51f01532a0 2024-04-25T00:13:28.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.929+0000 7f51f4d12640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51f00c8740 0x7f51f01f1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:28.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.929+0000 7f51f4d12640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51f00c8740 0x7f51f01f1a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55402/0 (socket says 172.21.15.116:55402) 2024-04-25T00:13:28.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.929+0000 7f51f4d12640 1 -- 172.21.15.116:0/3262623209 learned_addr learned my addr 172.21.15.116:0/3262623209 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:13:28.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.929+0000 7f51f5d14640 1 --2- 172.21.15.116:0/3262623209 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f51f01532a0 0x7f51f006c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:28.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.929+0000 7f51f5513640 1 --2- 172.21.15.116:0/3262623209 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f51f00c53b0 0x7f51f014a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:28.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.929+0000 7f51f4d12640 1 -- 172.21.15.116:0/3262623209 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f51f01532a0 msgr2=0x7f51f006c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:28.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.929+0000 7f51f4d12640 1 --2- 172.21.15.116:0/3262623209 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f51f01532a0 0x7f51f006c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:28.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.929+0000 7f51f4d12640 1 -- 172.21.15.116:0/3262623209 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f51f00c53b0 msgr2=0x7f51f014a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:28.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.929+0000 7f51f4d12640 1 --2- 172.21.15.116:0/3262623209 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f51f00c53b0 0x7f51f014a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:28.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.929+0000 7f51f4d12640 1 -- 172.21.15.116:0/3262623209 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51f00ac2b0 con 0x7f51f00c8740 2024-04-25T00:13:28.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.929+0000 7f51f5513640 1 --2- 172.21.15.116:0/3262623209 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f51f00c53b0 0x7f51f014a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:13:28.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.929+0000 7f51f4d12640 1 --2- 172.21.15.116:0/3262623209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51f00c8740 0x7f51f01f1a60 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f51e00ae040 tx=0x7f51e001abc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:28.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.929+0000 7f51e5ffb640 1 -- 172.21.15.116:0/3262623209 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e00b0050 con 0x7f51f00c8740 2024-04-25T00:13:28.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.929+0000 7f51e5ffb640 1 -- 172.21.15.116:0/3262623209 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51e00b1040 con 0x7f51f00c8740 2024-04-25T00:13:28.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.929+0000 7f51f7777640 1 -- 172.21.15.116:0/3262623209 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51f00abfd0 con 0x7f51f00c8740 2024-04-25T00:13:28.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.929+0000 7f51e5ffb640 1 -- 172.21.15.116:0/3262623209 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e0004220 con 0x7f51f00c8740 2024-04-25T00:13:28.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.929+0000 7f51f7777640 1 -- 172.21.15.116:0/3262623209 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51f014f020 con 0x7f51f00c8740 2024-04-25T00:13:28.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.929+0000 7f51f7777640 1 -- 172.21.15.116:0/3262623209 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51f0068a60 con 0x7f51f00c8740 2024-04-25T00:13:28.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.929+0000 7f51e5ffb640 1 -- 172.21.15.116:0/3262623209 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f51e00be060 con 0x7f51f00c8740 2024-04-25T00:13:28.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.929+0000 7f51e5ffb640 1 --2- 172.21.15.116:0/3262623209 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f51d40678c0 0x7f51d4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:28.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.929+0000 7f51e5ffb640 1 -- 172.21.15.116:0/3262623209 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 8042+0+0 (secure 0 0 0) 0x7f51e011fd90 con 0x7f51f00c8740 2024-04-25T00:13:28.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.929+0000 7f51f5513640 1 --2- 172.21.15.116:0/3262623209 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f51d40678c0 0x7f51d4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:28.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.933+0000 7f51f5513640 1 --2- 172.21.15.116:0/3262623209 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f51d40678c0 0x7f51d4069d80 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f51f014b120 tx=0x7f51ec008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:28.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:28.937+0000 7f51e5ffb640 1 -- 172.21.15.116:0/3262623209 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f51e0016090 con 0x7f51f00c8740 2024-04-25T00:13:29.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:29.141+0000 7f51f7777640 1 -- 172.21.15.116:0/3262623209 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f51f00c95e0 con 0x7f51f00c8740 2024-04-25T00:13:29.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:29.141+0000 7f51e5ffb640 1 -- 172.21.15.116:0/3262623209 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) ==== 94+0+6496 (secure 0 0 0) 0x7f51e00f0040 con 0x7f51f00c8740 2024-04-25T00:13:29.142 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:13:29.142 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82},{"gid":6582,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/462190523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":462190523},{"type":"v1","addr":"172.21.15.116:6835","nonce":462190523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":92}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:12:48.771611+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":88,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6833/365773583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":365773583},{"type":"v1","addr":"172.21.15.116:6833","nonce":365773583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":1},{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:12:23.643548+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5461},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5461":{"gid":5461,"name":"b","rank":0,"incarnation":83,"state":"up:active","state_seq":30,"addr":"172.21.15.174:6837/3398690896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3398690896},{"type":"v1","addr":"172.21.15.174:6837","nonce":3398690896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5461,"qdb_cluster":[5461]},"id":2}]} 2024-04-25T00:13:29.142 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 92 2024-04-25T00:13:29.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:29.145+0000 7f51f7777640 1 -- 172.21.15.116:0/3262623209 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f51d40678c0 msgr2=0x7f51d4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:29.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:29.145+0000 7f51f7777640 1 --2- 172.21.15.116:0/3262623209 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f51d40678c0 0x7f51d4069d80 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f51f014b120 tx=0x7f51ec008040 comp rx=0 tx=0).stop 2024-04-25T00:13:29.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:29.145+0000 7f51f7777640 1 -- 172.21.15.116:0/3262623209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51f00c8740 msgr2=0x7f51f01f1a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:29.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:29.145+0000 7f51f7777640 1 --2- 172.21.15.116:0/3262623209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51f00c8740 0x7f51f01f1a60 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f51e00ae040 tx=0x7f51e001abc0 comp rx=0 tx=0).stop 2024-04-25T00:13:29.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:29.145+0000 7f51f7777640 1 -- 172.21.15.116:0/3262623209 shutdown_connections 2024-04-25T00:13:29.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:29.145+0000 7f51f7777640 1 --2- 172.21.15.116:0/3262623209 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f51d40678c0 0x7f51d4069d80 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:29.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:29.145+0000 7f51f7777640 1 --2- 172.21.15.116:0/3262623209 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f51f01532a0 0x7f51f006c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:29.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:29.145+0000 7f51f7777640 1 --2- 172.21.15.116:0/3262623209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51f00c8740 0x7f51f01f1a60 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:29.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:29.145+0000 7f51f7777640 1 --2- 172.21.15.116:0/3262623209 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f51f00c53b0 0x7f51f014a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:29.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:29.145+0000 7f51f7777640 1 -- 172.21.15.116:0/3262623209 >> 172.21.15.116:0/3262623209 conn(0x7f51f006f0c0 msgr2=0x7f51f00c6040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:29.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:29.145+0000 7f51f7777640 1 -- 172.21.15.116:0/3262623209 shutdown_connections 2024-04-25T00:13:29.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:29.145+0000 7f51f7777640 1 -- 172.21.15.116:0/3262623209 wait complete. 2024-04-25T00:13:29.162 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T00:13:31.163 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:13:31.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.297+0000 7ff8c865b640 1 Processor -- start 2024-04-25T00:13:31.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.297+0000 7ff8c865b640 1 -- start start 2024-04-25T00:13:31.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.301+0000 7ff8c865b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff8c0150d20 0x7ff8c0153620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:31.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.301+0000 7ff8c865b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff8c00c3240 0x7ff8c00c3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:31.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.301+0000 7ff8c865b640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8c00c36a0 con 0x7ff8c00c4380 2024-04-25T00:13:31.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.301+0000 7ff8c865b640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7ff8c006fba0 con 0x7ff8c00c3240 2024-04-25T00:13:31.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.301+0000 7ff8c865b640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff8c006f8c0 con 0x7ff8c0150d20 2024-04-25T00:13:31.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.301+0000 7ff8c5bf6640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff8c00c3240 0x7ff8c00c3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:31.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.301+0000 7ff8c6bf8640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7ff8c00c4380 0x7ff8c0071050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:45282/0 (socket says 172.21.15.116:45282) 2024-04-25T00:13:31.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.301+0000 7ff8c5bf6640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff8c00c3240 0x7ff8c00c3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55406/0 (socket says 172.21.15.116:55406) 2024-04-25T00:13:31.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.301+0000 7ff8c6bf8640 1 -- 172.21.15.116:0/4283417831 learned_addr learned my addr 172.21.15.116:0/4283417831 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:13:31.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.301+0000 7ff8c63f7640 1 --2- 172.21.15.116:0/4283417831 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff8c0150d20 0x7ff8c0153620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:31.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.301+0000 7ff8c53f5640 1 -- 172.21.15.116:0/4283417831 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1979704006 0 0) 0x7ff8c00c36a0 con 0x7ff8c00c4380 2024-04-25T00:13:31.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.301+0000 7ff8c53f5640 1 -- 172.21.15.116:0/4283417831 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff8ac003530 con 0x7ff8c00c4380 2024-04-25T00:13:31.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.301+0000 7ff8c5bf6640 1 -- 172.21.15.116:0/4283417831 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff8c0150d20 msgr2=0x7ff8c0153620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:13:31.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.301+0000 7ff8c5bf6640 1 --2- 172.21.15.116:0/4283417831 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff8c0150d20 0x7ff8c0153620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:31.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.301+0000 7ff8c5bf6640 1 -- 172.21.15.116:0/4283417831 >> v1:172.21.15.116:6789/0 conn(0x7ff8c00c4380 legacy=0x7ff8c0071050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:31.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.301+0000 7ff8c5bf6640 1 -- 172.21.15.116:0/4283417831 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8c00c5420 con 0x7ff8c00c3240 2024-04-25T00:13:31.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.305+0000 7ff8c5bf6640 1 --2- 172.21.15.116:0/4283417831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8c00c3240 0x7ff8c00c3e40 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7ff8b00097c0 tx=0x7ff8b001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eec33b55ebcfefe8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:31.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.305+0000 7ff8c53f5640 1 -- 172.21.15.116:0/4283417831 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8b00b1070 con 0x7ff8c00c3240 2024-04-25T00:13:31.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.305+0000 7ff8c53f5640 1 -- 172.21.15.116:0/4283417831 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8b00ac410 con 0x7ff8c00c3240 2024-04-25T00:13:31.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.305+0000 7ff8c53f5640 1 -- 172.21.15.116:0/4283417831 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8b00ac7c0 con 0x7ff8c00c3240 2024-04-25T00:13:31.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.305+0000 7ff8c865b640 1 -- 172.21.15.116:0/4283417831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8c00c3240 msgr2=0x7ff8c00c3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:31.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.305+0000 7ff8c865b640 1 --2- 172.21.15.116:0/4283417831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8c00c3240 0x7ff8c00c3e40 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7ff8b00097c0 tx=0x7ff8b001c4a0 comp rx=0 tx=0).stop 2024-04-25T00:13:31.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.305+0000 7ff8c865b640 1 -- 172.21.15.116:0/4283417831 shutdown_connections 2024-04-25T00:13:31.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.305+0000 7ff8c865b640 1 --2- 172.21.15.116:0/4283417831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8c00c3240 0x7ff8c00c3e40 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:31.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.305+0000 7ff8c865b640 1 --2- 172.21.15.116:0/4283417831 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff8c0150d20 0x7ff8c0153620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:31.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.305+0000 7ff8c865b640 1 -- 172.21.15.116:0/4283417831 >> 172.21.15.116:0/4283417831 conn(0x7ff8c00c9160 msgr2=0x7ff8c00c9570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:31.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.305+0000 7ff8c865b640 1 -- 172.21.15.116:0/4283417831 shutdown_connections 2024-04-25T00:13:31.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.305+0000 7ff8c865b640 1 -- 172.21.15.116:0/4283417831 wait complete. 2024-04-25T00:13:31.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.305+0000 7ff8c865b640 1 Processor -- start 2024-04-25T00:13:31.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.305+0000 7ff8c865b640 1 -- start start 2024-04-25T00:13:31.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.305+0000 7ff8c865b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8c00c3240 0x7ff8c00a71f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:31.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.305+0000 7ff8c865b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff8c00c4380 0x7ff8c00a6b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:31.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.305+0000 7ff8c865b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff8c0150d20 0x7ff8c01f1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:31.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.305+0000 7ff8c865b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff8c00718e0 con 0x7ff8c00c3240 2024-04-25T00:13:31.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.305+0000 7ff8c865b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff8c0071760 con 0x7ff8c0150d20 2024-04-25T00:13:31.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.305+0000 7ff8c865b640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff8c0151100 con 0x7ff8c00c4380 2024-04-25T00:13:31.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.305+0000 7ff8c63f7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8c00c3240 0x7ff8c00a71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:31.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.305+0000 7ff8c63f7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8c00c3240 0x7ff8c00a71f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55408/0 (socket says 172.21.15.116:55408) 2024-04-25T00:13:31.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.305+0000 7ff8c63f7640 1 -- 172.21.15.116:0/47007170 learned_addr learned my addr 172.21.15.116:0/47007170 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:13:31.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.305+0000 7ff8c6bf8640 1 --2- 172.21.15.116:0/47007170 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff8c0150d20 0x7ff8c01f1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:31.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.305+0000 7ff8c5bf6640 1 --2- 172.21.15.116:0/47007170 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff8c00c4380 0x7ff8c00a6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:31.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.305+0000 7ff8c63f7640 1 -- 172.21.15.116:0/47007170 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff8c00c4380 msgr2=0x7ff8c00a6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:31.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.305+0000 7ff8c63f7640 1 --2- 172.21.15.116:0/47007170 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff8c00c4380 0x7ff8c00a6b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:31.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.305+0000 7ff8c63f7640 1 -- 172.21.15.116:0/47007170 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff8c0150d20 msgr2=0x7ff8c01f1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:31.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.305+0000 7ff8c63f7640 1 --2- 172.21.15.116:0/47007170 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff8c0150d20 0x7ff8c01f1f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:31.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.305+0000 7ff8c63f7640 1 -- 172.21.15.116:0/47007170 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8c00a6060 con 0x7ff8c00c3240 2024-04-25T00:13:31.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.305+0000 7ff8c6bf8640 1 --2- 172.21.15.116:0/47007170 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff8c0150d20 0x7ff8c01f1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:13:31.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.305+0000 7ff8c5bf6640 1 --2- 172.21.15.116:0/47007170 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff8c00c4380 0x7ff8c00a6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:13:31.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.305+0000 7ff8c63f7640 1 --2- 172.21.15.116:0/47007170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8c00c3240 0x7ff8c00a71f0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7ff8bc00d620 tx=0x7ff8bc00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:31.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.305+0000 7ff8b6ffd640 1 -- 172.21.15.116:0/47007170 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8bc00b620 con 0x7ff8c00c3240 2024-04-25T00:13:31.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.309+0000 7ff8b6ffd640 1 -- 172.21.15.116:0/47007170 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8bc019070 con 0x7ff8c00c3240 2024-04-25T00:13:31.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.309+0000 7ff8c865b640 1 -- 172.21.15.116:0/47007170 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8c00a7840 con 0x7ff8c00c3240 2024-04-25T00:13:31.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.309+0000 7ff8b6ffd640 1 -- 172.21.15.116:0/47007170 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8bc00b9e0 con 0x7ff8c00c3240 2024-04-25T00:13:31.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.309+0000 7ff8c865b640 1 -- 172.21.15.116:0/47007170 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8c00a6300 con 0x7ff8c00c3240 2024-04-25T00:13:31.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.309+0000 7ff8c865b640 1 -- 172.21.15.116:0/47007170 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8c0068a60 con 0x7ff8c00c3240 2024-04-25T00:13:31.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.309+0000 7ff8b6ffd640 1 -- 172.21.15.116:0/47007170 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff8bc00bb80 con 0x7ff8c00c3240 2024-04-25T00:13:31.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.309+0000 7ff8b6ffd640 1 --2- 172.21.15.116:0/47007170 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff8a40678c0 0x7ff8a4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:31.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.309+0000 7ff8b6ffd640 1 -- 172.21.15.116:0/47007170 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 8128+0+0 (secure 0 0 0) 0x7ff8bc086a00 con 0x7ff8c00c3240 2024-04-25T00:13:31.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.309+0000 7ff8c5bf6640 1 --2- 172.21.15.116:0/47007170 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff8a40678c0 0x7ff8a4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:31.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.309+0000 7ff8c5bf6640 1 --2- 172.21.15.116:0/47007170 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff8a40678c0 0x7ff8a4069d80 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7ff8b00ae000 tx=0x7ff8b00a6040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:31.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.313+0000 7ff8b6ffd640 1 -- 172.21.15.116:0/47007170 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ff8bc056bd0 con 0x7ff8c00c3240 2024-04-25T00:13:31.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.553+0000 7ff8c865b640 1 -- 172.21.15.116:0/47007170 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff8c00c4790 con 0x7ff8c00c3240 2024-04-25T00:13:31.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.553+0000 7ff8b6ffd640 1 -- 172.21.15.116:0/47007170 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) ==== 94+0+5660 (secure 0 0 0) 0x7ff8bc0564d0 con 0x7ff8c00c3240 2024-04-25T00:13:31.554 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:13:31.554 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:12:48.771611+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":88,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6833/365773583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":365773583},{"type":"v1","addr":"172.21.15.116:6833","nonce":365773583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":1},{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:13:30.544146+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6582},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6582":{"gid":6582,"name":"a","rank":0,"incarnation":93,"state":"up:replay","state_seq":1,"addr":"172.21.15.116:6835/462190523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":462190523},{"type":"v1","addr":"172.21.15.116:6835","nonce":462190523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:13:31.554 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 93 2024-04-25T00:13:31.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.557+0000 7ff8c865b640 1 -- 172.21.15.116:0/47007170 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff8a40678c0 msgr2=0x7ff8a4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:31.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.557+0000 7ff8c865b640 1 --2- 172.21.15.116:0/47007170 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff8a40678c0 0x7ff8a4069d80 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7ff8b00ae000 tx=0x7ff8b00a6040 comp rx=0 tx=0).stop 2024-04-25T00:13:31.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.557+0000 7ff8c865b640 1 -- 172.21.15.116:0/47007170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8c00c3240 msgr2=0x7ff8c00a71f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:31.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.557+0000 7ff8c865b640 1 --2- 172.21.15.116:0/47007170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8c00c3240 0x7ff8c00a71f0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7ff8bc00d620 tx=0x7ff8bc00dae0 comp rx=0 tx=0).stop 2024-04-25T00:13:31.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.557+0000 7ff8c865b640 1 -- 172.21.15.116:0/47007170 shutdown_connections 2024-04-25T00:13:31.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.557+0000 7ff8c865b640 1 --2- 172.21.15.116:0/47007170 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff8a40678c0 0x7ff8a4069d80 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:31.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.557+0000 7ff8c865b640 1 --2- 172.21.15.116:0/47007170 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff8c0150d20 0x7ff8c01f1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:31.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.557+0000 7ff8c865b640 1 --2- 172.21.15.116:0/47007170 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff8c00c4380 0x7ff8c00a6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:31.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.557+0000 7ff8c865b640 1 --2- 172.21.15.116:0/47007170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff8c00c3240 0x7ff8c00a71f0 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:31.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.557+0000 7ff8c865b640 1 -- 172.21.15.116:0/47007170 >> 172.21.15.116:0/47007170 conn(0x7ff8c00c9160 msgr2=0x7ff8c014d7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:31.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.557+0000 7ff8c865b640 1 -- 172.21.15.116:0/47007170 shutdown_connections 2024-04-25T00:13:31.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.557+0000 7ff8c865b640 1 -- 172.21.15.116:0/47007170 wait complete. 2024-04-25T00:13:31.572 INFO:tasks.mds_thrash.fs.[b]:mds.b down, removed from mdsmap 2024-04-25T00:13:31.572 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-04-25T00:13:31.572 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:13:31.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.685+0000 7f7928ac9640 1 Processor -- start 2024-04-25T00:13:31.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.685+0000 7f7928ac9640 1 -- start start 2024-04-25T00:13:31.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.685+0000 7f7928ac9640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f79240c4300 0x7f79240aa170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:31.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.685+0000 7f7928ac9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f79240c5550 0x7f7924156200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:31.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.685+0000 7f7928ac9640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79241526a0 con 0x7f79240c5100 2024-04-25T00:13:31.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.685+0000 7f7928ac9640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f7924153460 con 0x7f79240c5550 2024-04-25T00:13:31.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.685+0000 7f7928ac9640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f792406ffd0 con 0x7f79240c4300 2024-04-25T00:13:31.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.685+0000 7f7922d9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f79240c5550 0x7f7924156200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:31.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.685+0000 7f791bfff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f79240c5100 0x7f7924155af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:45296/0 (socket says 172.21.15.116:45296) 2024-04-25T00:13:31.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.685+0000 7f791bfff640 1 -- 172.21.15.116:0/1246138550 learned_addr learned my addr 172.21.15.116:0/1246138550 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:13:31.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.685+0000 7f7922d9d640 1 -- 172.21.15.116:0/1246138550 >> v2:172.21.15.174:3300/0 conn(0x7f79240c4300 msgr2=0x7f79240aa170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:13:31.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.685+0000 7f792259c640 1 --2- 172.21.15.116:0/1246138550 >> v2:172.21.15.174:3300/0 conn(0x7f79240c4300 0x7f79240aa170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:31.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.685+0000 7f7922d9d640 1 --2- 172.21.15.116:0/1246138550 >> v2:172.21.15.174:3300/0 conn(0x7f79240c4300 0x7f79240aa170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:31.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.685+0000 7f7921d9b640 1 -- 172.21.15.116:0/1246138550 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2906509666 0 0) 0x7f79241526a0 con 0x7f79240c5100 2024-04-25T00:13:31.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.685+0000 7f7922d9d640 1 -- 172.21.15.116:0/1246138550 >> v1:172.21.15.116:6789/0 conn(0x7f79240c5100 legacy=0x7f7924155af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:31.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.685+0000 7f7922d9d640 1 -- 172.21.15.116:0/1246138550 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7924152980 con 0x7f79240c5550 2024-04-25T00:13:31.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.689+0000 7f7922d9d640 1 --2- 172.21.15.116:0/1246138550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79240c5550 0x7f7924156200 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f7924070150 tx=0x7f791c01c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53881fda4da760f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:31.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.689+0000 7f7921d9b640 1 -- 172.21.15.116:0/1246138550 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f791c0a6700 con 0x7f79240c5550 2024-04-25T00:13:31.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.689+0000 7f7921d9b640 1 -- 172.21.15.116:0/1246138550 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f791c0a68a0 con 0x7f79240c5550 2024-04-25T00:13:31.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.689+0000 7f7921d9b640 1 -- 172.21.15.116:0/1246138550 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f791c0a6c70 con 0x7f79240c5550 2024-04-25T00:13:31.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.689+0000 7f7928ac9640 1 -- 172.21.15.116:0/1246138550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79240c5550 msgr2=0x7f7924156200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:31.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.689+0000 7f7928ac9640 1 --2- 172.21.15.116:0/1246138550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79240c5550 0x7f7924156200 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f7924070150 tx=0x7f791c01c930 comp rx=0 tx=0).stop 2024-04-25T00:13:31.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.689+0000 7f7928ac9640 1 -- 172.21.15.116:0/1246138550 shutdown_connections 2024-04-25T00:13:31.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.689+0000 7f7928ac9640 1 --2- 172.21.15.116:0/1246138550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79240c5550 0x7f7924156200 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:31.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.689+0000 7f7928ac9640 1 --2- 172.21.15.116:0/1246138550 >> v2:172.21.15.174:3300/0 conn(0x7f79240c4300 0x7f79240aa170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:31.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.689+0000 7f7928ac9640 1 -- 172.21.15.116:0/1246138550 >> 172.21.15.116:0/1246138550 conn(0x7f792406c720 msgr2=0x7f792406cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:31.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.689+0000 7f7928ac9640 1 -- 172.21.15.116:0/1246138550 shutdown_connections 2024-04-25T00:13:31.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.689+0000 7f7928ac9640 1 -- 172.21.15.116:0/1246138550 wait complete. 2024-04-25T00:13:31.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.689+0000 7f7928ac9640 1 Processor -- start 2024-04-25T00:13:31.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.689+0000 7f7928ac9640 1 -- start start 2024-04-25T00:13:31.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.693+0000 7f7928ac9640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f79240c4300 0x7f792414ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:31.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.693+0000 7f7928ac9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79240c5100 0x7f79240ac400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:31.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.693+0000 7f7928ac9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79240c5550 0x7f79241eea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:31.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.693+0000 7f7928ac9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f792406fb80 con 0x7f79240c5550 2024-04-25T00:13:31.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.693+0000 7f7928ac9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f792415ee90 con 0x7f79240c5100 2024-04-25T00:13:31.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.693+0000 7f7928ac9640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f792406fd00 con 0x7f79240c4300 2024-04-25T00:13:31.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.693+0000 7f7922d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79240c5550 0x7f79241eea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:31.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.693+0000 7f7922d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79240c5550 0x7f79241eea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55426/0 (socket says 172.21.15.116:55426) 2024-04-25T00:13:31.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.693+0000 7f7922d9d640 1 -- 172.21.15.116:0/35107723 learned_addr learned my addr 172.21.15.116:0/35107723 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:13:31.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.693+0000 7f792259c640 1 --2- 172.21.15.116:0/35107723 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f79240c4300 0x7f792414ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:31.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.693+0000 7f791bfff640 1 --2- 172.21.15.116:0/35107723 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79240c5100 0x7f79240ac400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:31.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.693+0000 7f7922d9d640 1 -- 172.21.15.116:0/35107723 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f79240c4300 msgr2=0x7f792414ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:31.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.693+0000 7f7922d9d640 1 --2- 172.21.15.116:0/35107723 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f79240c4300 0x7f792414ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:31.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.693+0000 7f7922d9d640 1 -- 172.21.15.116:0/35107723 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79240c5100 msgr2=0x7f79240ac400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:31.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.693+0000 7f7922d9d640 1 --2- 172.21.15.116:0/35107723 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79240c5100 0x7f79240ac400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:31.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.693+0000 7f7922d9d640 1 -- 172.21.15.116:0/35107723 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79241ee280 con 0x7f79240c5550 2024-04-25T00:13:31.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.693+0000 7f7922d9d640 1 --2- 172.21.15.116:0/35107723 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79240c5550 0x7f79241eea60 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f791c0b0040 tx=0x7f791c0196d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:31.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.693+0000 7f791affd640 1 -- 172.21.15.116:0/35107723 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f791c0b8840 con 0x7f79240c5550 2024-04-25T00:13:31.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.693+0000 7f791affd640 1 -- 172.21.15.116:0/35107723 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f791c0c1070 con 0x7f79240c5550 2024-04-25T00:13:31.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.693+0000 7f7928ac9640 1 -- 172.21.15.116:0/35107723 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7924148df0 con 0x7f79240c5550 2024-04-25T00:13:31.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.693+0000 7f791affd640 1 -- 172.21.15.116:0/35107723 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f791c0b8c40 con 0x7f79240c5550 2024-04-25T00:13:31.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.693+0000 7f7928ac9640 1 -- 172.21.15.116:0/35107723 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79240ac940 con 0x7f79240c5550 2024-04-25T00:13:31.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.693+0000 7f7928ac9640 1 -- 172.21.15.116:0/35107723 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79240c2820 con 0x7f79240c5550 2024-04-25T00:13:31.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.697+0000 7f791affd640 1 -- 172.21.15.116:0/35107723 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f791c0a6760 con 0x7f79240c5550 2024-04-25T00:13:31.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.697+0000 7f791affd640 1 --2- 172.21.15.116:0/35107723 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f79040678c0 0x7f7904069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:31.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.697+0000 7f791affd640 1 -- 172.21.15.116:0/35107723 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 8128+0+0 (secure 0 0 0) 0x7f791c122420 con 0x7f79240c5550 2024-04-25T00:13:31.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.697+0000 7f792259c640 1 --2- 172.21.15.116:0/35107723 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f79040678c0 0x7f7904069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:31.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.697+0000 7f792259c640 1 --2- 172.21.15.116:0/35107723 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f79040678c0 0x7f7904069d80 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f79240c2480 tx=0x7f7914008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:31.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.701+0000 7f791affd640 1 -- 172.21.15.116:0/35107723 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f791c016520 con 0x7f79240c5550 2024-04-25T00:13:31.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.913+0000 7f7928ac9640 1 -- 172.21.15.116:0/35107723 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f792414c070 con 0x7f79240c5550 2024-04-25T00:13:31.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.913+0000 7f791affd640 1 -- 172.21.15.116:0/35107723 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) ==== 94+0+5664 (secure 0 0 0) 0x7f791c0f26f0 con 0x7f79240c5550 2024-04-25T00:13:31.913 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:13:31.913 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:12:48.771611+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":88,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6833/365773583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":365773583},{"type":"v1","addr":"172.21.15.116:6833","nonce":365773583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":1},{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:13:30.930141+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6582},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6582":{"gid":6582,"name":"a","rank":0,"incarnation":93,"state":"up:reconnect","state_seq":12,"addr":"172.21.15.116:6835/462190523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":462190523},{"type":"v1","addr":"172.21.15.116:6835","nonce":462190523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:13:31.914 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 94 2024-04-25T00:13:31.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.913+0000 7f7928ac9640 1 -- 172.21.15.116:0/35107723 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f79040678c0 msgr2=0x7f7904069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:31.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.913+0000 7f7928ac9640 1 --2- 172.21.15.116:0/35107723 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f79040678c0 0x7f7904069d80 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f79240c2480 tx=0x7f7914008040 comp rx=0 tx=0).stop 2024-04-25T00:13:31.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.913+0000 7f7928ac9640 1 -- 172.21.15.116:0/35107723 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79240c5550 msgr2=0x7f79241eea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:31.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.913+0000 7f7928ac9640 1 --2- 172.21.15.116:0/35107723 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79240c5550 0x7f79241eea60 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f791c0b0040 tx=0x7f791c0196d0 comp rx=0 tx=0).stop 2024-04-25T00:13:31.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.913+0000 7f7928ac9640 1 -- 172.21.15.116:0/35107723 shutdown_connections 2024-04-25T00:13:31.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.913+0000 7f7928ac9640 1 --2- 172.21.15.116:0/35107723 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f79040678c0 0x7f7904069d80 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:31.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.913+0000 7f7928ac9640 1 --2- 172.21.15.116:0/35107723 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79240c5550 0x7f79241eea60 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:31.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.913+0000 7f7928ac9640 1 --2- 172.21.15.116:0/35107723 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79240c5100 0x7f79240ac400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:31.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.913+0000 7f7928ac9640 1 --2- 172.21.15.116:0/35107723 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f79240c4300 0x7f792414ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:31.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.913+0000 7f7928ac9640 1 -- 172.21.15.116:0/35107723 >> 172.21.15.116:0/35107723 conn(0x7f792406c720 msgr2=0x7f79240c7140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:31.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.913+0000 7f7928ac9640 1 -- 172.21.15.116:0/35107723 shutdown_connections 2024-04-25T00:13:31.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:31.913+0000 7f7928ac9640 1 -- 172.21.15.116:0/35107723 wait complete. 2024-04-25T00:13:31.933 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 94, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:04.678737+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 91, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6495": { "addr": "172.21.15.116:6833/365773583", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6832", "nonce": 365773583, "type": "v2" }, { "addr": "172.21.15.116:6833", "nonce": 365773583, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6495, "incarnation": 88, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 7 } }, "last_failure": 0, "last_failure_osd_epoch": 44, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T00:12:48.771611+0000", "qdb_cluster": [ 6495 ], "qdb_leader": 6495, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6495 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:18.825887+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 94, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6582": { "addr": "172.21.15.116:6835/462190523", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6834", "nonce": 462190523, "type": "v2" }, { "addr": "172.21.15.116:6835", "nonce": 462190523, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6582, "incarnation": 93, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:reconnect", "state_seq": 12 } }, "last_failure": 0, "last_failure_osd_epoch": 45, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-04-25T00:13:30.930141+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6582 } } } ], "standbys": [ { "addr": "172.21.15.174:6835/4215220222", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6834", "nonce": 4215220222, "type": "v2" }, { "addr": "172.21.15.174:6835", "nonce": 4215220222, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 82, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5273, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-25T00:13:33.935 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:13:34.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.081+0000 7fd6cecd8640 1 Processor -- start 2024-04-25T00:13:34.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.081+0000 7fd6cecd8640 1 -- start start 2024-04-25T00:13:34.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.085+0000 7fd6cecd8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd6c8152db0 0x7fd6c80c7af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:34.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.085+0000 7fd6cecd8640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd6c80c8030 0x7fd6c80c5a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:34.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.085+0000 7fd6cecd8640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fd6c80c5fa0 0x7fd6c81558c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:34.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.085+0000 7fd6cecd8640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fd6c8158eb0 con 0x7fd6c8152db0 2024-04-25T00:13:34.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.085+0000 7fd6cecd8640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fd6c8159190 con 0x7fd6c80c5fa0 2024-04-25T00:13:34.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.085+0000 7fd6cecd8640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fd6c8159470 con 0x7fd6c80c8030 2024-04-25T00:13:34.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.085+0000 7fd6cca74640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd6c8152db0 0x7fd6c80c7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:34.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.085+0000 7fd6cca74640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd6c8152db0 0x7fd6c80c7af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55430/0 (socket says 172.21.15.116:55430) 2024-04-25T00:13:34.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.085+0000 7fd6cca74640 1 -- 172.21.15.116:0/3393243912 learned_addr learned my addr 172.21.15.116:0/3393243912 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:13:34.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.085+0000 7fd6c7fff640 1 --2- 172.21.15.116:0/3393243912 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd6c80c8030 0x7fd6c80c5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:34.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.085+0000 7fd6cd275640 1 --2- 172.21.15.116:0/3393243912 >> v2:172.21.15.174:3300/0 conn(0x7fd6c80c5fa0 0x7fd6c81558c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:34.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.085+0000 7fd6cca74640 1 -- 172.21.15.116:0/3393243912 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd6c80c8030 msgr2=0x7fd6c80c5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:13:34.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.085+0000 7fd6cca74640 1 --2- 172.21.15.116:0/3393243912 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd6c80c8030 0x7fd6c80c5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:34.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.085+0000 7fd6cca74640 1 -- 172.21.15.116:0/3393243912 >> v2:172.21.15.174:3300/0 conn(0x7fd6c80c5fa0 msgr2=0x7fd6c81558c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:13:34.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.085+0000 7fd6cca74640 1 --2- 172.21.15.116:0/3393243912 >> v2:172.21.15.174:3300/0 conn(0x7fd6c80c5fa0 0x7fd6c81558c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:34.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.085+0000 7fd6cca74640 1 -- 172.21.15.116:0/3393243912 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6c8159750 con 0x7fd6c8152db0 2024-04-25T00:13:34.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.085+0000 7fd6cd275640 1 --2- 172.21.15.116:0/3393243912 >> v2:172.21.15.174:3300/0 conn(0x7fd6c80c5fa0 0x7fd6c81558c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:13:34.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.085+0000 7fd6cca74640 1 --2- 172.21.15.116:0/3393243912 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6c8152db0 0x7fd6c80c7af0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fd6c80c8c10 tx=0x7fd6bc01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed64857e8f6217b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:34.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.085+0000 7fd6c77fe640 1 -- 172.21.15.116:0/3393243912 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6bc0ab660 con 0x7fd6c8152db0 2024-04-25T00:13:34.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.085+0000 7fd6c77fe640 1 -- 172.21.15.116:0/3393243912 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6bc0b0070 con 0x7fd6c8152db0 2024-04-25T00:13:34.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.085+0000 7fd6c77fe640 1 -- 172.21.15.116:0/3393243912 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6bc0aba60 con 0x7fd6c8152db0 2024-04-25T00:13:34.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.089+0000 7fd6cecd8640 1 -- 172.21.15.116:0/3393243912 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6c8152db0 msgr2=0x7fd6c80c7af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:34.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.089+0000 7fd6cecd8640 1 --2- 172.21.15.116:0/3393243912 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6c8152db0 0x7fd6c80c7af0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fd6c80c8c10 tx=0x7fd6bc01c480 comp rx=0 tx=0).stop 2024-04-25T00:13:34.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.089+0000 7fd6cecd8640 1 -- 172.21.15.116:0/3393243912 shutdown_connections 2024-04-25T00:13:34.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.089+0000 7fd6cecd8640 1 --2- 172.21.15.116:0/3393243912 >> v2:172.21.15.174:3300/0 conn(0x7fd6c80c5fa0 0x7fd6c81558c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:34.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.089+0000 7fd6cecd8640 1 --2- 172.21.15.116:0/3393243912 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd6c80c8030 0x7fd6c80c5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:34.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.089+0000 7fd6cecd8640 1 --2- 172.21.15.116:0/3393243912 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6c8152db0 0x7fd6c80c7af0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:34.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.089+0000 7fd6cecd8640 1 -- 172.21.15.116:0/3393243912 >> 172.21.15.116:0/3393243912 conn(0x7fd6c80acc10 msgr2=0x7fd6c80ad020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:34.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.089+0000 7fd6cecd8640 1 -- 172.21.15.116:0/3393243912 shutdown_connections 2024-04-25T00:13:34.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.089+0000 7fd6cecd8640 1 -- 172.21.15.116:0/3393243912 wait complete. 2024-04-25T00:13:34.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.089+0000 7fd6cecd8640 1 Processor -- start 2024-04-25T00:13:34.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.089+0000 7fd6cecd8640 1 -- start start 2024-04-25T00:13:34.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.093+0000 7fd6cecd8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd6c80c5fa0 0x7fd6c80aaf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:34.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.093+0000 7fd6cecd8640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd6c80c8030 0x7fd6c80c3260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:34.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.093+0000 7fd6cecd8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6c8152db0 0x7fd6c80c37a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:34.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.093+0000 7fd6cecd8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c80c9160 con 0x7fd6c8152db0 2024-04-25T00:13:34.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.093+0000 7fd6cecd8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd6c80c8fe0 con 0x7fd6c80c5fa0 2024-04-25T00:13:34.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.093+0000 7fd6cecd8640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fd6c80c92e0 con 0x7fd6c80c8030 2024-04-25T00:13:34.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.093+0000 7fd6cd275640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6c8152db0 0x7fd6c80c37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:34.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.093+0000 7fd6cd275640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6c8152db0 0x7fd6c80c37a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:55444/0 (socket says 172.21.15.116:55444) 2024-04-25T00:13:34.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.093+0000 7fd6cd275640 1 -- 172.21.15.116:0/694517304 learned_addr learned my addr 172.21.15.116:0/694517304 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:13:34.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.093+0000 7fd6cca74640 1 --2- 172.21.15.116:0/694517304 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd6c80c5fa0 0x7fd6c80aaf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:34.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.093+0000 7fd6c7fff640 1 --2- 172.21.15.116:0/694517304 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd6c80c8030 0x7fd6c80c3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:34.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.093+0000 7fd6cd275640 1 -- 172.21.15.116:0/694517304 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd6c80c8030 msgr2=0x7fd6c80c3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:34.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.093+0000 7fd6cd275640 1 --2- 172.21.15.116:0/694517304 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd6c80c8030 0x7fd6c80c3260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:34.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.093+0000 7fd6cd275640 1 -- 172.21.15.116:0/694517304 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd6c80c5fa0 msgr2=0x7fd6c80aaf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:34.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.093+0000 7fd6cd275640 1 --2- 172.21.15.116:0/694517304 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd6c80c5fa0 0x7fd6c80aaf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:34.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.093+0000 7fd6cd275640 1 -- 172.21.15.116:0/694517304 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6c8151b80 con 0x7fd6c8152db0 2024-04-25T00:13:34.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.093+0000 7fd6cd275640 1 --2- 172.21.15.116:0/694517304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6c8152db0 0x7fd6c80c37a0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fd6c000cef0 tx=0x7fd6c000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:34.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.093+0000 7fd6c57fa640 1 -- 172.21.15.116:0/694517304 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c001a070 con 0x7fd6c8152db0 2024-04-25T00:13:34.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.093+0000 7fd6c57fa640 1 -- 172.21.15.116:0/694517304 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6c000d840 con 0x7fd6c8152db0 2024-04-25T00:13:34.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.093+0000 7fd6cecd8640 1 -- 172.21.15.116:0/694517304 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6c81507a0 con 0x7fd6c8152db0 2024-04-25T00:13:34.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.093+0000 7fd6c57fa640 1 -- 172.21.15.116:0/694517304 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c000dc30 con 0x7fd6c8152db0 2024-04-25T00:13:34.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.093+0000 7fd6cecd8640 1 -- 172.21.15.116:0/694517304 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6c8150d60 con 0x7fd6c8152db0 2024-04-25T00:13:34.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.097+0000 7fd6c57fa640 1 -- 172.21.15.116:0/694517304 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd6c001d070 con 0x7fd6c8152db0 2024-04-25T00:13:34.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.097+0000 7fd6cecd8640 1 -- 172.21.15.116:0/694517304 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6c8068a60 con 0x7fd6c8152db0 2024-04-25T00:13:34.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.097+0000 7fd6c57fa640 1 --2- 172.21.15.116:0/694517304 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd6ac0678c0 0x7fd6ac069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:34.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.097+0000 7fd6c57fa640 1 -- 172.21.15.116:0/694517304 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 8128+0+0 (secure 0 0 0) 0x7fd6c008b060 con 0x7fd6c8152db0 2024-04-25T00:13:34.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.097+0000 7fd6cca74640 1 --2- 172.21.15.116:0/694517304 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd6ac0678c0 0x7fd6ac069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:34.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.097+0000 7fd6cca74640 1 --2- 172.21.15.116:0/694517304 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd6ac0678c0 0x7fd6ac069d80 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc0af000 tx=0x7fd6bc0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:34.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.101+0000 7fd6c57fa640 1 -- 172.21.15.116:0/694517304 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fd6c0056730 con 0x7fd6c8152db0 2024-04-25T00:13:34.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.329+0000 7fd6cecd8640 1 -- 172.21.15.116:0/694517304 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd6c80c9ff0 con 0x7fd6c8152db0 2024-04-25T00:13:34.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.329+0000 7fd6c57fa640 1 -- 172.21.15.116:0/694517304 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) ==== 94+0+5668 (secure 0 0 0) 0x7fd6c005aa30 con 0x7fd6c8152db0 2024-04-25T00:13:34.332 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:13:34.332 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:12:48.771611+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":88,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6833/365773583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":365773583},{"type":"v1","addr":"172.21.15.116:6833","nonce":365773583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":1},{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:13:33.916554+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6582},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6582":{"gid":6582,"name":"a","rank":0,"incarnation":93,"state":"up:active","state_seq":14,"addr":"172.21.15.116:6835/462190523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":462190523},{"type":"v1","addr":"172.21.15.116:6835","nonce":462190523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6582,"qdb_cluster":[6582]},"id":2}]} 2024-04-25T00:13:34.333 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 96 2024-04-25T00:13:34.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.329+0000 7fd6cecd8640 1 -- 172.21.15.116:0/694517304 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd6ac0678c0 msgr2=0x7fd6ac069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:34.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.329+0000 7fd6cecd8640 1 --2- 172.21.15.116:0/694517304 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd6ac0678c0 0x7fd6ac069d80 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc0af000 tx=0x7fd6bc0a7000 comp rx=0 tx=0).stop 2024-04-25T00:13:34.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.329+0000 7fd6cecd8640 1 -- 172.21.15.116:0/694517304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6c8152db0 msgr2=0x7fd6c80c37a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:34.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.329+0000 7fd6cecd8640 1 --2- 172.21.15.116:0/694517304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6c8152db0 0x7fd6c80c37a0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fd6c000cef0 tx=0x7fd6c000a710 comp rx=0 tx=0).stop 2024-04-25T00:13:34.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.329+0000 7fd6cecd8640 1 -- 172.21.15.116:0/694517304 shutdown_connections 2024-04-25T00:13:34.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.329+0000 7fd6cecd8640 1 --2- 172.21.15.116:0/694517304 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd6ac0678c0 0x7fd6ac069d80 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:34.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.329+0000 7fd6cecd8640 1 --2- 172.21.15.116:0/694517304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6c8152db0 0x7fd6c80c37a0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:34.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.329+0000 7fd6cecd8640 1 --2- 172.21.15.116:0/694517304 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd6c80c8030 0x7fd6c80c3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:34.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.329+0000 7fd6cecd8640 1 --2- 172.21.15.116:0/694517304 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd6c80c5fa0 0x7fd6c80aaf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:34.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.329+0000 7fd6cecd8640 1 -- 172.21.15.116:0/694517304 >> 172.21.15.116:0/694517304 conn(0x7fd6c80acc10 msgr2=0x7fd6c814c250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:34.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.333+0000 7fd6cecd8640 1 -- 172.21.15.116:0/694517304 shutdown_connections 2024-04-25T00:13:34.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:34.333+0000 7fd6cecd8640 1 -- 172.21.15.116:0/694517304 wait complete. 2024-04-25T00:13:34.350 INFO:tasks.mds_thrash.fs.[b]:mds.a has gained rank=0, replacing gid=5461 2024-04-25T00:13:34.350 INFO:tasks.mds_thrash.fs.[b]:waiting for 3 secs before reviving mds.b 2024-04-25T00:13:34.790 INFO:tasks.ceph.mds.a.smithi116.stderr:2024-04-25T00:13:34.789+0000 7f5add9b4640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T00:13:37.351 INFO:tasks.mds_thrash.fs.[b]:reviving mds.b 2024-04-25T00:13:37.351 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-25T00:13:37.351 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-25T00:13:37.355 INFO:tasks.ceph.mds.b:Started 2024-04-25T00:13:37.355 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:13:37.437 INFO:tasks.ceph.mds.b.smithi174.stdout:starting mds.b at 2024-04-25T00:13:37.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.505+0000 7f3db763c640 1 Processor -- start 2024-04-25T00:13:37.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.505+0000 7f3db763c640 1 -- start start 2024-04-25T00:13:37.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.509+0000 7f3db763c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3db00aeb20 0x7f3db00c9310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:37.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.509+0000 7f3db763c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3db00c9850 0x7f3db0150870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:37.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.509+0000 7f3db763c640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f3db0150db0 0x7f3db00c5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:37.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.509+0000 7f3db763c640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f3db00c9cb0 con 0x7f3db00c9850 2024-04-25T00:13:37.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.509+0000 7f3db763c640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f3db01535d0 con 0x7f3db0150db0 2024-04-25T00:13:37.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.509+0000 7f3db763c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f3db0151220 con 0x7f3db00aeb20 2024-04-25T00:13:37.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.509+0000 7f3db4bd7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3db00c9850 0x7f3db0150870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:37.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.509+0000 7f3db4bd7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3db00c9850 0x7f3db0150870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52274/0 (socket says 172.21.15.116:52274) 2024-04-25T00:13:37.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.509+0000 7f3db4bd7640 1 -- 172.21.15.116:0/684900570 learned_addr learned my addr 172.21.15.116:0/684900570 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:13:37.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.509+0000 7f3db5bd9640 1 --2- 172.21.15.116:0/684900570 >> v2:172.21.15.174:3300/0 conn(0x7f3db0150db0 0x7f3db00c5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:37.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.509+0000 7f3db53d8640 1 --2- 172.21.15.116:0/684900570 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3db00aeb20 0x7f3db00c9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:37.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.509+0000 7f3db4bd7640 1 -- 172.21.15.116:0/684900570 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3db00aeb20 msgr2=0x7f3db00c9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:13:37.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.509+0000 7f3db4bd7640 1 --2- 172.21.15.116:0/684900570 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3db00aeb20 0x7f3db00c9310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:37.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.509+0000 7f3db4bd7640 1 -- 172.21.15.116:0/684900570 >> v2:172.21.15.174:3300/0 conn(0x7f3db0150db0 msgr2=0x7f3db00c5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:13:37.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.509+0000 7f3db4bd7640 1 --2- 172.21.15.116:0/684900570 >> v2:172.21.15.174:3300/0 conn(0x7f3db0150db0 0x7f3db00c5270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:37.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.509+0000 7f3db4bd7640 1 -- 172.21.15.116:0/684900570 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3db0153850 con 0x7f3db00c9850 2024-04-25T00:13:37.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.509+0000 7f3db4bd7640 1 --2- 172.21.15.116:0/684900570 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3db00c9850 0x7f3db0150870 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f3db0072d00 tx=0x7f3da001c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c13d638a45d310f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:37.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.509+0000 7f3da7fff640 1 -- 172.21.15.116:0/684900570 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da00ab620 con 0x7f3db00c9850 2024-04-25T00:13:37.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.509+0000 7f3da7fff640 1 -- 172.21.15.116:0/684900570 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3da00b0070 con 0x7f3db00c9850 2024-04-25T00:13:37.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.509+0000 7f3da7fff640 1 -- 172.21.15.116:0/684900570 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da00ab9e0 con 0x7f3db00c9850 2024-04-25T00:13:37.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.513+0000 7f3db763c640 1 -- 172.21.15.116:0/684900570 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3db00c9850 msgr2=0x7f3db0150870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:37.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.513+0000 7f3db763c640 1 --2- 172.21.15.116:0/684900570 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3db00c9850 0x7f3db0150870 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f3db0072d00 tx=0x7f3da001c460 comp rx=0 tx=0).stop 2024-04-25T00:13:37.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.513+0000 7f3db763c640 1 -- 172.21.15.116:0/684900570 shutdown_connections 2024-04-25T00:13:37.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.513+0000 7f3db763c640 1 --2- 172.21.15.116:0/684900570 >> v2:172.21.15.174:3300/0 conn(0x7f3db0150db0 0x7f3db00c5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:37.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.513+0000 7f3db763c640 1 --2- 172.21.15.116:0/684900570 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3db00c9850 0x7f3db0150870 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:37.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.513+0000 7f3db763c640 1 --2- 172.21.15.116:0/684900570 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3db00aeb20 0x7f3db00c9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:37.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.513+0000 7f3db763c640 1 -- 172.21.15.116:0/684900570 >> 172.21.15.116:0/684900570 conn(0x7f3db0074650 msgr2=0x7f3db0074a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:37.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.513+0000 7f3db763c640 1 -- 172.21.15.116:0/684900570 shutdown_connections 2024-04-25T00:13:37.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.513+0000 7f3db763c640 1 -- 172.21.15.116:0/684900570 wait complete. 2024-04-25T00:13:37.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.513+0000 7f3db763c640 1 Processor -- start 2024-04-25T00:13:37.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.513+0000 7f3db763c640 1 -- start start 2024-04-25T00:13:37.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.513+0000 7f3db763c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3db00aeb20 0x7f3db01f1c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:37.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.513+0000 7f3db763c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3db00c9850 0x7f3db00c2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:37.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.513+0000 7f3db763c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3db0150db0 0x7f3db0148210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:37.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.513+0000 7f3db763c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3db00aef00 con 0x7f3db0150db0 2024-04-25T00:13:37.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.513+0000 7f3db763c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3db01543d0 con 0x7f3db00c9850 2024-04-25T00:13:37.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.513+0000 7f3db763c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f3db00af080 con 0x7f3db00aeb20 2024-04-25T00:13:37.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.513+0000 7f3db5bd9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3db0150db0 0x7f3db0148210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:37.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.513+0000 7f3db5bd9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3db0150db0 0x7f3db0148210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52290/0 (socket says 172.21.15.116:52290) 2024-04-25T00:13:37.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.513+0000 7f3db5bd9640 1 -- 172.21.15.116:0/3553969345 learned_addr learned my addr 172.21.15.116:0/3553969345 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:13:37.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.513+0000 7f3db53d8640 1 --2- 172.21.15.116:0/3553969345 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3db00aeb20 0x7f3db01f1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:37.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.513+0000 7f3db5bd9640 1 -- 172.21.15.116:0/3553969345 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3db00aeb20 msgr2=0x7f3db01f1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:37.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.513+0000 7f3db4bd7640 1 --2- 172.21.15.116:0/3553969345 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3db00c9850 0x7f3db00c2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:37.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.513+0000 7f3db5bd9640 1 --2- 172.21.15.116:0/3553969345 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3db00aeb20 0x7f3db01f1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:37.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.513+0000 7f3db5bd9640 1 -- 172.21.15.116:0/3553969345 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3db00c9850 msgr2=0x7f3db00c2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:37.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.513+0000 7f3db5bd9640 1 --2- 172.21.15.116:0/3553969345 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3db00c9850 0x7f3db00c2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:37.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.513+0000 7f3db5bd9640 1 -- 172.21.15.116:0/3553969345 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3db006c460 con 0x7f3db0150db0 2024-04-25T00:13:37.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.513+0000 7f3db4bd7640 1 --2- 172.21.15.116:0/3553969345 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3db00c9850 0x7f3db00c2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:13:37.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.513+0000 7f3db5bd9640 1 --2- 172.21.15.116:0/3553969345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3db0150db0 0x7f3db0148210 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f3da8009d30 tx=0x7f3da800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:37.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.513+0000 7f3da5ffb640 1 -- 172.21.15.116:0/3553969345 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da8018020 con 0x7f3db0150db0 2024-04-25T00:13:37.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.513+0000 7f3da5ffb640 1 -- 172.21.15.116:0/3553969345 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3da8014480 con 0x7f3db0150db0 2024-04-25T00:13:37.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.513+0000 7f3db763c640 1 -- 172.21.15.116:0/3553969345 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3db006dfe0 con 0x7f3db0150db0 2024-04-25T00:13:37.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.513+0000 7f3da5ffb640 1 -- 172.21.15.116:0/3553969345 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da8014870 con 0x7f3db0150db0 2024-04-25T00:13:37.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.513+0000 7f3db763c640 1 -- 172.21.15.116:0/3553969345 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3db006cfe0 con 0x7f3db0150db0 2024-04-25T00:13:37.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.517+0000 7f3db763c640 1 -- 172.21.15.116:0/3553969345 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3db0068a60 con 0x7f3db0150db0 2024-04-25T00:13:37.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.517+0000 7f3da5ffb640 1 -- 172.21.15.116:0/3553969345 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3da801b020 con 0x7f3db0150db0 2024-04-25T00:13:37.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.517+0000 7f3da5ffb640 1 --2- 172.21.15.116:0/3553969345 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3d940678f0 0x7f3d94069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:13:37.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.517+0000 7f3da5ffb640 1 -- 172.21.15.116:0/3553969345 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 8128+0+0 (secure 0 0 0) 0x7f3da8086e90 con 0x7f3db0150db0 2024-04-25T00:13:37.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.517+0000 7f3db53d8640 1 --2- 172.21.15.116:0/3553969345 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3d940678f0 0x7f3d94069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:13:37.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.517+0000 7f3db53d8640 1 --2- 172.21.15.116:0/3553969345 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3d940678f0 0x7f3d94069db0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f3db00c2eb0 tx=0x7f3dac009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:13:37.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.521+0000 7f3da5ffb640 1 -- 172.21.15.116:0/3553969345 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f3da8022070 con 0x7f3db0150db0 2024-04-25T00:13:37.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.729+0000 7f3db763c640 1 -- 172.21.15.116:0/3553969345 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3db0151820 con 0x7f3db0150db0 2024-04-25T00:13:37.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.729+0000 7f3da5ffb640 1 -- 172.21.15.116:0/3553969345 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) ==== 94+0+6496 (secure 0 0 0) 0x7f3da8010040 con 0x7f3db0150db0 2024-04-25T00:13:37.731 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:13:37.731 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82},{"gid":5791,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3180024481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3180024481},{"type":"v1","addr":"172.21.15.174:6837","nonce":3180024481}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:12:48.771611+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":88,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6833/365773583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":365773583},{"type":"v1","addr":"172.21.15.116:6833","nonce":365773583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":1},{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:13:33.916554+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6582},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6582":{"gid":6582,"name":"a","rank":0,"incarnation":93,"state":"up:active","state_seq":14,"addr":"172.21.15.116:6835/462190523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":462190523},{"type":"v1","addr":"172.21.15.116:6835","nonce":462190523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6582,"qdb_cluster":[6582]},"id":2}]} 2024-04-25T00:13:37.731 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 97 2024-04-25T00:13:37.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.733+0000 7f3db763c640 1 -- 172.21.15.116:0/3553969345 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3d940678f0 msgr2=0x7f3d94069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:37.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.733+0000 7f3db763c640 1 --2- 172.21.15.116:0/3553969345 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3d940678f0 0x7f3d94069db0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f3db00c2eb0 tx=0x7f3dac009190 comp rx=0 tx=0).stop 2024-04-25T00:13:37.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.733+0000 7f3db763c640 1 -- 172.21.15.116:0/3553969345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3db0150db0 msgr2=0x7f3db0148210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:13:37.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.733+0000 7f3db763c640 1 --2- 172.21.15.116:0/3553969345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3db0150db0 0x7f3db0148210 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f3da8009d30 tx=0x7f3da800c490 comp rx=0 tx=0).stop 2024-04-25T00:13:37.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.733+0000 7f3db763c640 1 -- 172.21.15.116:0/3553969345 shutdown_connections 2024-04-25T00:13:37.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.733+0000 7f3db763c640 1 --2- 172.21.15.116:0/3553969345 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3d940678f0 0x7f3d94069db0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:37.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.733+0000 7f3db763c640 1 --2- 172.21.15.116:0/3553969345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3db0150db0 0x7f3db0148210 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:37.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.733+0000 7f3db763c640 1 --2- 172.21.15.116:0/3553969345 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3db00c9850 0x7f3db00c2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:37.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.733+0000 7f3db763c640 1 --2- 172.21.15.116:0/3553969345 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3db00aeb20 0x7f3db01f1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:13:37.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.733+0000 7f3db763c640 1 -- 172.21.15.116:0/3553969345 >> 172.21.15.116:0/3553969345 conn(0x7f3db0074650 msgr2=0x7f3db0152e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:13:37.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.733+0000 7f3db763c640 1 -- 172.21.15.116:0/3553969345 shutdown_connections 2024-04-25T00:13:37.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:13:37.733+0000 7f3db763c640 1 -- 172.21.15.116:0/3553969345 wait complete. 2024-04-25T00:13:37.752 INFO:tasks.mds_thrash.fs.[b]:mds.b reported in up:standby state 2024-04-25T00:13:37.752 INFO:tasks.mds_thrash.fs.[b]:waiting for 59 secs before thrashing 2024-04-25T00:13:37.892 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:13:37.894 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:14:08.007 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:14:08.012 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:14:36.754 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:14:36.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:36.892+0000 7f868f68b640 1 Processor -- start 2024-04-25T00:14:36.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:36.892+0000 7f868f68b640 1 -- start start 2024-04-25T00:14:36.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:36.892+0000 7f868f68b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8688150ba0 0x7f86880c5010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:36.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:36.892+0000 7f868f68b640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f8688072c60 0x7f8688153e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:36.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:36.892+0000 7f868f68b640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86880c2cb0 con 0x7f8688072820 2024-04-25T00:14:36.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:36.892+0000 7f868f68b640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f8688157c30 con 0x7f8688150ba0 2024-04-25T00:14:36.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:36.892+0000 7f868f68b640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f8688157670 con 0x7f8688072c60 2024-04-25T00:14:36.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:36.892+0000 7f868d427640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8688150ba0 0x7f86880c5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:36.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:36.892+0000 7f868d427640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8688150ba0 0x7f86880c5010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60820/0 (socket says 172.21.15.116:60820) 2024-04-25T00:14:36.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:36.892+0000 7f868d427640 1 -- 172.21.15.116:0/4014297438 learned_addr learned my addr 172.21.15.116:0/4014297438 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:14:36.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:36.892+0000 7f868dc28640 1 --2- 172.21.15.116:0/4014297438 >> v2:172.21.15.174:3300/0 conn(0x7f8688072c60 0x7f8688153e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:36.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:36.892+0000 7f868d427640 1 -- 172.21.15.116:0/4014297438 >> v2:172.21.15.174:3300/0 conn(0x7f8688072c60 msgr2=0x7f8688153e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:14:36.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:36.892+0000 7f868d427640 1 --2- 172.21.15.116:0/4014297438 >> v2:172.21.15.174:3300/0 conn(0x7f8688072c60 0x7f8688153e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:36.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:36.892+0000 7f868d427640 1 -- 172.21.15.116:0/4014297438 >> v1:172.21.15.174:6789/0 conn(0x7f8688072820 legacy=0x7f8688153770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:36.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:36.892+0000 7f868d427640 1 -- 172.21.15.116:0/4014297438 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8688157950 con 0x7f8688150ba0 2024-04-25T00:14:36.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:36.896+0000 7f868d427640 1 --2- 172.21.15.116:0/4014297438 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8688150ba0 0x7f86880c5010 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f86880a9980 tx=0x7f868401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=711b660db72624aa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:36.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:36.896+0000 7f867ffff640 1 -- 172.21.15.116:0/4014297438 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86840b0070 con 0x7f8688150ba0 2024-04-25T00:14:36.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:36.896+0000 7f867ffff640 1 -- 172.21.15.116:0/4014297438 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86840ab600 con 0x7f8688150ba0 2024-04-25T00:14:36.896 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:36.896+0000 7f867ffff640 1 -- 172.21.15.116:0/4014297438 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86840ab9d0 con 0x7f8688150ba0 2024-04-25T00:14:36.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:36.896+0000 7f868f68b640 1 -- 172.21.15.116:0/4014297438 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8688150ba0 msgr2=0x7f86880c5010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:36.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:36.896+0000 7f868f68b640 1 --2- 172.21.15.116:0/4014297438 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8688150ba0 0x7f86880c5010 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f86880a9980 tx=0x7f868401c490 comp rx=0 tx=0).stop 2024-04-25T00:14:36.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:36.896+0000 7f868f68b640 1 -- 172.21.15.116:0/4014297438 shutdown_connections 2024-04-25T00:14:36.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:36.896+0000 7f868f68b640 1 --2- 172.21.15.116:0/4014297438 >> v2:172.21.15.174:3300/0 conn(0x7f8688072c60 0x7f8688153e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:36.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:36.896+0000 7f868f68b640 1 --2- 172.21.15.116:0/4014297438 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8688150ba0 0x7f86880c5010 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:36.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:36.896+0000 7f868f68b640 1 -- 172.21.15.116:0/4014297438 >> 172.21.15.116:0/4014297438 conn(0x7f868806c720 msgr2=0x7f868806cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:36.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:36.900+0000 7f868f68b640 1 -- 172.21.15.116:0/4014297438 shutdown_connections 2024-04-25T00:14:36.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:36.900+0000 7f868f68b640 1 -- 172.21.15.116:0/4014297438 wait complete. 2024-04-25T00:14:36.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:36.900+0000 7f868f68b640 1 Processor -- start 2024-04-25T00:14:36.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:36.900+0000 7f868f68b640 1 -- start start 2024-04-25T00:14:36.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:36.900+0000 7f868f68b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8688072820 0x7f868815c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:36.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:36.900+0000 7f868f68b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8688072c60 0x7f8688155ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:36.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:36.900+0000 7f868f68b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8688150ba0 0x7f8688156420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:36.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:36.900+0000 7f868f68b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8688160990 con 0x7f8688150ba0 2024-04-25T00:14:36.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:36.900+0000 7f868f68b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8688160810 con 0x7f8688072820 2024-04-25T00:14:36.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:36.900+0000 7f868f68b640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f8688160b10 con 0x7f8688072c60 2024-04-25T00:14:36.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:36.900+0000 7f868dc28640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8688150ba0 0x7f8688156420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:36.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:36.900+0000 7f868dc28640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8688150ba0 0x7f8688156420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60832/0 (socket says 172.21.15.116:60832) 2024-04-25T00:14:36.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:36.900+0000 7f868dc28640 1 -- 172.21.15.116:0/3731143677 learned_addr learned my addr 172.21.15.116:0/3731143677 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:14:36.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:36.900+0000 7f868dc28640 1 -- 172.21.15.116:0/3731143677 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8688072c60 msgr2=0x7f8688155ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:36.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:36.900+0000 7f868dc28640 1 --2- 172.21.15.116:0/3731143677 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8688072c60 0x7f8688155ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:36.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:36.900+0000 7f868dc28640 1 -- 172.21.15.116:0/3731143677 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8688072820 msgr2=0x7f868815c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:36.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:36.900+0000 7f868dc28640 1 --2- 172.21.15.116:0/3731143677 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8688072820 0x7f868815c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:36.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:36.900+0000 7f868dc28640 1 -- 172.21.15.116:0/3731143677 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86880ab4f0 con 0x7f8688150ba0 2024-04-25T00:14:36.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:36.900+0000 7f868dc28640 1 --2- 172.21.15.116:0/3731143677 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8688150ba0 0x7f8688156420 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f8680007e30 tx=0x7f868000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:36.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:36.900+0000 7f867dffb640 1 -- 172.21.15.116:0/3731143677 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8680014620 con 0x7f8688150ba0 2024-04-25T00:14:36.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:36.900+0000 7f867dffb640 1 -- 172.21.15.116:0/3731143677 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8680019070 con 0x7f8688150ba0 2024-04-25T00:14:36.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:36.900+0000 7f868f68b640 1 -- 172.21.15.116:0/3731143677 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86880ad0b0 con 0x7f8688150ba0 2024-04-25T00:14:36.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:36.900+0000 7f867dffb640 1 -- 172.21.15.116:0/3731143677 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8680014a00 con 0x7f8688150ba0 2024-04-25T00:14:36.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:36.900+0000 7f868f68b640 1 -- 172.21.15.116:0/3731143677 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86880acdb0 con 0x7f8688150ba0 2024-04-25T00:14:36.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:36.904+0000 7f868f68b640 1 -- 172.21.15.116:0/3731143677 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8688143570 con 0x7f8688150ba0 2024-04-25T00:14:36.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:36.904+0000 7f867dffb640 1 -- 172.21.15.116:0/3731143677 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8680014ba0 con 0x7f8688150ba0 2024-04-25T00:14:36.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:36.904+0000 7f867dffb640 1 --2- 172.21.15.116:0/3731143677 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f866c0678c0 0x7f866c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:36.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:36.904+0000 7f867dffb640 1 -- 172.21.15.116:0/3731143677 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 8128+0+0 (secure 0 0 0) 0x7f8680086ed0 con 0x7f8688150ba0 2024-04-25T00:14:36.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:36.904+0000 7f868d427640 1 --2- 172.21.15.116:0/3731143677 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f866c0678c0 0x7f866c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:36.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:36.904+0000 7f868d427640 1 --2- 172.21.15.116:0/3731143677 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f866c0678c0 0x7f866c069d80 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f86840ad000 tx=0x7f8684008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:36.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:36.908+0000 7f867dffb640 1 -- 172.21.15.116:0/3731143677 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f86800570a0 con 0x7f8688150ba0 2024-04-25T00:14:37.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.136+0000 7f868f68b640 1 -- 172.21.15.116:0/3731143677 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8688142610 con 0x7f8688150ba0 2024-04-25T00:14:37.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.136+0000 7f867dffb640 1 -- 172.21.15.116:0/3731143677 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) ==== 94+0+6496 (secure 0 0 0) 0x7f8688142610 con 0x7f8688150ba0 2024-04-25T00:14:37.139 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:14:37.139 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82},{"gid":5791,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3180024481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3180024481},{"type":"v1","addr":"172.21.15.174:6837","nonce":3180024481}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:12:48.771611+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":88,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6833/365773583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":365773583},{"type":"v1","addr":"172.21.15.116:6833","nonce":365773583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":1},{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:13:33.916554+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6582},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6582":{"gid":6582,"name":"a","rank":0,"incarnation":93,"state":"up:active","state_seq":14,"addr":"172.21.15.116:6835/462190523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":462190523},{"type":"v1","addr":"172.21.15.116:6835","nonce":462190523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6582,"qdb_cluster":[6582]},"id":2}]} 2024-04-25T00:14:37.139 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 97 2024-04-25T00:14:37.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.140+0000 7f868f68b640 1 -- 172.21.15.116:0/3731143677 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f866c0678c0 msgr2=0x7f866c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:37.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.140+0000 7f868f68b640 1 --2- 172.21.15.116:0/3731143677 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f866c0678c0 0x7f866c069d80 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f86840ad000 tx=0x7f8684008040 comp rx=0 tx=0).stop 2024-04-25T00:14:37.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.140+0000 7f868f68b640 1 -- 172.21.15.116:0/3731143677 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8688150ba0 msgr2=0x7f8688156420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:37.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.140+0000 7f868f68b640 1 --2- 172.21.15.116:0/3731143677 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8688150ba0 0x7f8688156420 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f8680007e30 tx=0x7f868000e4b0 comp rx=0 tx=0).stop 2024-04-25T00:14:37.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.140+0000 7f868f68b640 1 -- 172.21.15.116:0/3731143677 shutdown_connections 2024-04-25T00:14:37.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.140+0000 7f868f68b640 1 --2- 172.21.15.116:0/3731143677 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f866c0678c0 0x7f866c069d80 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:37.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.140+0000 7f868f68b640 1 --2- 172.21.15.116:0/3731143677 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8688150ba0 0x7f8688156420 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:37.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.140+0000 7f868f68b640 1 --2- 172.21.15.116:0/3731143677 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8688072c60 0x7f8688155ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:37.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.140+0000 7f868f68b640 1 --2- 172.21.15.116:0/3731143677 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8688072820 0x7f868815c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:37.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.140+0000 7f868f68b640 1 -- 172.21.15.116:0/3731143677 >> 172.21.15.116:0/3731143677 conn(0x7f868806c720 msgr2=0x7f86880aa8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:37.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.140+0000 7f868f68b640 1 -- 172.21.15.116:0/3731143677 shutdown_connections 2024-04-25T00:14:37.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.140+0000 7f868f68b640 1 -- 172.21.15.116:0/3731143677 wait complete. 2024-04-25T00:14:37.158 INFO:tasks.mds_thrash.fs.[b]:kill mds.a (rank=0) 2024-04-25T00:14:37.159 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-25T00:14:37.159 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:14:37.168 INFO:tasks.ceph.mds.a:Stopped 2024-04-25T00:14:37.169 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:14:37.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.352+0000 7f6c12e37640 1 Processor -- start 2024-04-25T00:14:37.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.352+0000 7f6c12e37640 1 -- start start 2024-04-25T00:14:37.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.352+0000 7f6c12e37640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c0c150d20 0x7f6c0c153620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:37.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.352+0000 7f6c12e37640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c0c0c36a0 con 0x7f6c0c0c3240 2024-04-25T00:14:37.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.352+0000 7f6c12e37640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c0c06f8c0 con 0x7f6c0c071050 2024-04-25T00:14:37.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.352+0000 7f6c12e37640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6c0c06fba0 con 0x7f6c0c150d20 2024-04-25T00:14:37.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.352+0000 7f6c0bfff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f6c0c0c3240 0x7f6c0c0c3e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:36246/0 (socket says 172.21.15.116:36246) 2024-04-25T00:14:37.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.352+0000 7f6c0bfff640 1 -- 172.21.15.116:0/1294891032 learned_addr learned my addr 172.21.15.116:0/1294891032 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:14:37.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.352+0000 7f6c10bd3640 1 --2- 172.21.15.116:0/1294891032 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c0c150d20 0x7f6c0c153620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:37.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.352+0000 7f6c0b7fe640 1 -- 172.21.15.116:0/1294891032 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1646553320 0 0) 0x7f6c0c0c36a0 con 0x7f6c0c0c3240 2024-04-25T00:14:37.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.352+0000 7f6c0b7fe640 1 -- 172.21.15.116:0/1294891032 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6bfc003530 con 0x7f6c0c0c3240 2024-04-25T00:14:37.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.352+0000 7f6c0b7fe640 1 -- 172.21.15.116:0/1294891032 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1551083211 0 0) 0x7f6bfc003530 con 0x7f6c0c0c3240 2024-04-25T00:14:37.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.352+0000 7f6c0b7fe640 1 -- 172.21.15.116:0/1294891032 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c0c150d20 msgr2=0x7f6c0c153620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:14:37.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.352+0000 7f6c0b7fe640 1 --2- 172.21.15.116:0/1294891032 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c0c150d20 0x7f6c0c153620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:37.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.352+0000 7f6c0b7fe640 1 -- 172.21.15.116:0/1294891032 >> v1:172.21.15.174:6789/0 conn(0x7f6c0c071050 legacy=0x7f6c0c15df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:37.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.352+0000 7f6c0b7fe640 1 -- 172.21.15.116:0/1294891032 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c0c0c5420 con 0x7f6c0c0c3240 2024-04-25T00:14:37.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.352+0000 7f6c0b7fe640 1 -- 172.21.15.116:0/1294891032 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f6bf8001680 con 0x7f6c0c0c3240 2024-04-25T00:14:37.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.352+0000 7f6c0b7fe640 1 -- 172.21.15.116:0/1294891032 >> v1:172.21.15.116:6789/0 conn(0x7f6c0c0c3240 legacy=0x7f6c0c0c3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:37.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.352+0000 7f6c0b7fe640 1 --2- 172.21.15.116:0/1294891032 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c0c150d20 0x7f6bfc0039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:37.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.352+0000 7f6c0b7fe640 1 --2- 172.21.15.116:0/1294891032 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6bfc003f70 0x7f6bfc004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:37.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.352+0000 7f6c0b7fe640 1 --2- 172.21.15.116:0/1294891032 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6bfc004d50 0x7f6bfc0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:37.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.352+0000 7f6c0b7fe640 1 -- 172.21.15.116:0/1294891032 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6c0c06fba0 con 0x7f6bfc004d50 2024-04-25T00:14:37.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.352+0000 7f6c0b7fe640 1 -- 172.21.15.116:0/1294891032 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6bfc00bab0 con 0x7f6bfc003f70 2024-04-25T00:14:37.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.352+0000 7f6c0b7fe640 1 -- 172.21.15.116:0/1294891032 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6bfc00bcb0 con 0x7f6c0c150d20 2024-04-25T00:14:37.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.352+0000 7f6c113d4640 1 --2- 172.21.15.116:0/1294891032 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6bfc004d50 0x7f6bfc0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:37.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.352+0000 7f6c0bfff640 1 --2- 172.21.15.116:0/1294891032 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6bfc003f70 0x7f6bfc004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:37.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.352+0000 7f6c113d4640 1 -- 172.21.15.116:0/1294891032 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c0c150d20 msgr2=0x7f6bfc0039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:14:37.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.352+0000 7f6c113d4640 1 --2- 172.21.15.116:0/1294891032 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c0c150d20 0x7f6bfc0039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:37.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.352+0000 7f6c113d4640 1 -- 172.21.15.116:0/1294891032 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6bfc003f70 msgr2=0x7f6bfc004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:14:37.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.352+0000 7f6c113d4640 1 --2- 172.21.15.116:0/1294891032 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6bfc003f70 0x7f6bfc004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:37.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.352+0000 7f6c113d4640 1 -- 172.21.15.116:0/1294891032 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bfc003530 con 0x7f6bfc004d50 2024-04-25T00:14:37.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.356+0000 7f6c113d4640 1 --2- 172.21.15.116:0/1294891032 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6bfc004d50 0x7f6bfc0051d0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f6c04001e70 tx=0x7f6c0401c780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b08eb776112946d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:37.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.356+0000 7f6c0b7fe640 1 -- 172.21.15.116:0/1294891032 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c04001a90 con 0x7f6bfc004d50 2024-04-25T00:14:37.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.356+0000 7f6c0b7fe640 1 -- 172.21.15.116:0/1294891032 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c04003b80 con 0x7f6bfc004d50 2024-04-25T00:14:37.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.356+0000 7f6c0b7fe640 1 -- 172.21.15.116:0/1294891032 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c04005080 con 0x7f6bfc004d50 2024-04-25T00:14:37.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.356+0000 7f6c12e37640 1 -- 172.21.15.116:0/1294891032 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6bfc004d50 msgr2=0x7f6bfc0051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:37.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.356+0000 7f6c12e37640 1 --2- 172.21.15.116:0/1294891032 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6bfc004d50 0x7f6bfc0051d0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f6c04001e70 tx=0x7f6c0401c780 comp rx=0 tx=0).stop 2024-04-25T00:14:37.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.356+0000 7f6c113d4640 1 -- 172.21.15.116:0/1294891032 reap_dead start 2024-04-25T00:14:37.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.356+0000 7f6c12e37640 1 -- 172.21.15.116:0/1294891032 shutdown_connections 2024-04-25T00:14:37.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.356+0000 7f6c12e37640 1 -- 172.21.15.116:0/1294891032 >> 172.21.15.116:0/1294891032 conn(0x7f6c0c0c9160 msgr2=0x7f6c0c0c9570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:37.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.356+0000 7f6c12e37640 1 -- 172.21.15.116:0/1294891032 shutdown_connections 2024-04-25T00:14:37.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.356+0000 7f6c12e37640 1 -- 172.21.15.116:0/1294891032 wait complete. 2024-04-25T00:14:37.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.360+0000 7f6c12e37640 1 Processor -- start 2024-04-25T00:14:37.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.360+0000 7f6c12e37640 1 -- start start 2024-04-25T00:14:37.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.360+0000 7f6c12e37640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6bfc004d50 0x7f6c0c1ec520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:37.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.360+0000 7f6c12e37640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c0c0ad590 0x7f6c0c1420d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:37.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.360+0000 7f6c12e37640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c0c143350 0x7f6c0c142610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:37.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.360+0000 7f6c12e37640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6c0c0c4bd0 con 0x7f6c0c0ad590 2024-04-25T00:14:37.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.360+0000 7f6c12e37640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6c0c0c4550 con 0x7f6c0c143350 2024-04-25T00:14:37.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.360+0000 7f6c12e37640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6c0c0c4d50 con 0x7f6bfc004d50 2024-04-25T00:14:37.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.360+0000 7f6c0bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c0c0ad590 0x7f6c0c1420d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:37.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.360+0000 7f6c0bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c0c0ad590 0x7f6c0c1420d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60854/0 (socket says 172.21.15.116:60854) 2024-04-25T00:14:37.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.360+0000 7f6c0bfff640 1 -- 172.21.15.116:0/4145950054 learned_addr learned my addr 172.21.15.116:0/4145950054 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:14:37.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.360+0000 7f6c113d4640 1 --2- 172.21.15.116:0/4145950054 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c0c143350 0x7f6c0c142610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:37.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.360+0000 7f6c10bd3640 1 --2- 172.21.15.116:0/4145950054 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6bfc004d50 0x7f6c0c1ec520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:37.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.360+0000 7f6c0bfff640 1 -- 172.21.15.116:0/4145950054 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6bfc004d50 msgr2=0x7f6c0c1ec520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:37.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.360+0000 7f6c0bfff640 1 --2- 172.21.15.116:0/4145950054 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6bfc004d50 0x7f6c0c1ec520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:37.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.360+0000 7f6c0bfff640 1 -- 172.21.15.116:0/4145950054 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c0c143350 msgr2=0x7f6c0c142610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:37.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.360+0000 7f6c0bfff640 1 --2- 172.21.15.116:0/4145950054 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c0c143350 0x7f6c0c142610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:37.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.360+0000 7f6c0bfff640 1 -- 172.21.15.116:0/4145950054 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c0c143100 con 0x7f6c0c0ad590 2024-04-25T00:14:37.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.360+0000 7f6c113d4640 1 --2- 172.21.15.116:0/4145950054 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c0c143350 0x7f6c0c142610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:14:37.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.360+0000 7f6c0bfff640 1 --2- 172.21.15.116:0/4145950054 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c0c0ad590 0x7f6c0c1420d0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f6bf800e880 tx=0x7f6bf800ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:37.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.360+0000 7f6c097fa640 1 -- 172.21.15.116:0/4145950054 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bf800c8d0 con 0x7f6c0c0ad590 2024-04-25T00:14:37.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.360+0000 7f6c097fa640 1 -- 172.21.15.116:0/4145950054 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6bf8018070 con 0x7f6c0c0ad590 2024-04-25T00:14:37.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.360+0000 7f6c12e37640 1 -- 172.21.15.116:0/4145950054 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c0c142e20 con 0x7f6c0c0ad590 2024-04-25T00:14:37.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.360+0000 7f6c097fa640 1 -- 172.21.15.116:0/4145950054 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bf800ccd0 con 0x7f6c0c0ad590 2024-04-25T00:14:37.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.360+0000 7f6c12e37640 1 -- 172.21.15.116:0/4145950054 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c0c1ed050 con 0x7f6c0c0ad590 2024-04-25T00:14:37.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.360+0000 7f6c12e37640 1 -- 172.21.15.116:0/4145950054 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c0c068a60 con 0x7f6c0c0ad590 2024-04-25T00:14:37.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.360+0000 7f6c097fa640 1 -- 172.21.15.116:0/4145950054 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6bf801d460 con 0x7f6c0c0ad590 2024-04-25T00:14:37.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.364+0000 7f6c097fa640 1 --2- 172.21.15.116:0/4145950054 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6bf00678c0 0x7f6bf0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:37.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.364+0000 7f6c097fa640 1 -- 172.21.15.116:0/4145950054 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 8128+0+0 (secure 0 0 0) 0x7f6bf8086580 con 0x7f6c0c0ad590 2024-04-25T00:14:37.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.364+0000 7f6c10bd3640 1 --2- 172.21.15.116:0/4145950054 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6bf00678c0 0x7f6bf0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:37.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.364+0000 7f6c10bd3640 1 --2- 172.21.15.116:0/4145950054 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6bf00678c0 0x7f6bf0069d80 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f6c000047c0 tx=0x7f6c00008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:37.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.368+0000 7f6c097fa640 1 -- 172.21.15.116:0/4145950054 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f6bf80567e0 con 0x7f6c0c0ad590 2024-04-25T00:14:37.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.620+0000 7f6c12e37640 1 -- 172.21.15.116:0/4145950054 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6c0c0c4790 con 0x7f6c0c0ad590 2024-04-25T00:14:37.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.620+0000 7f6c097fa640 1 -- 172.21.15.116:0/4145950054 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) ==== 94+0+6496 (secure 0 0 0) 0x7f6bf80560e0 con 0x7f6c0c0ad590 2024-04-25T00:14:37.623 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:14:37.623 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82},{"gid":5791,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3180024481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3180024481},{"type":"v1","addr":"172.21.15.174:6837","nonce":3180024481}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:12:48.771611+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":88,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6833/365773583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":365773583},{"type":"v1","addr":"172.21.15.116:6833","nonce":365773583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":1},{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:13:33.916554+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6582},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6582":{"gid":6582,"name":"a","rank":0,"incarnation":93,"state":"up:active","state_seq":14,"addr":"172.21.15.116:6835/462190523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":462190523},{"type":"v1","addr":"172.21.15.116:6835","nonce":462190523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6582,"qdb_cluster":[6582]},"id":2}]} 2024-04-25T00:14:37.623 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 97 2024-04-25T00:14:37.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.624+0000 7f6c12e37640 1 -- 172.21.15.116:0/4145950054 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6bf00678c0 msgr2=0x7f6bf0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:37.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.624+0000 7f6c12e37640 1 --2- 172.21.15.116:0/4145950054 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6bf00678c0 0x7f6bf0069d80 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f6c000047c0 tx=0x7f6c00008040 comp rx=0 tx=0).stop 2024-04-25T00:14:37.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.624+0000 7f6c12e37640 1 -- 172.21.15.116:0/4145950054 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c0c0ad590 msgr2=0x7f6c0c1420d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:37.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.624+0000 7f6c12e37640 1 --2- 172.21.15.116:0/4145950054 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c0c0ad590 0x7f6c0c1420d0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f6bf800e880 tx=0x7f6bf800ed40 comp rx=0 tx=0).stop 2024-04-25T00:14:37.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.624+0000 7f6c12e37640 1 -- 172.21.15.116:0/4145950054 shutdown_connections 2024-04-25T00:14:37.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.624+0000 7f6c12e37640 1 --2- 172.21.15.116:0/4145950054 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6bf00678c0 0x7f6bf0069d80 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:37.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.624+0000 7f6c12e37640 1 --2- 172.21.15.116:0/4145950054 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c0c143350 0x7f6c0c142610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:37.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.624+0000 7f6c12e37640 1 --2- 172.21.15.116:0/4145950054 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c0c0ad590 0x7f6c0c1420d0 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:37.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.624+0000 7f6c12e37640 1 --2- 172.21.15.116:0/4145950054 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6bfc004d50 0x7f6c0c1ec520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:37.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.624+0000 7f6c12e37640 1 -- 172.21.15.116:0/4145950054 >> 172.21.15.116:0/4145950054 conn(0x7f6c0c0c9160 msgr2=0x7f6c0c14ff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:37.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.624+0000 7f6c12e37640 1 -- 172.21.15.116:0/4145950054 shutdown_connections 2024-04-25T00:14:37.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:37.624+0000 7f6c12e37640 1 -- 172.21.15.116:0/4145950054 wait complete. 2024-04-25T00:14:37.644 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:14:38.080 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:14:38.082 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:14:39.645 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:14:39.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.776+0000 7f3b9f6bb640 1 Processor -- start 2024-04-25T00:14:39.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.776+0000 7f3b9f6bb640 1 -- start start 2024-04-25T00:14:39.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.776+0000 7f3b9f6bb640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f3b9814c700 0x7f3b980c5090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:39.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.776+0000 7f3b9f6bb640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b980a9e60 con 0x7f3b980c55d0 2024-04-25T00:14:39.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.776+0000 7f3b9f6bb640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b980c3ef0 con 0x7f3b98155b20 2024-04-25T00:14:39.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.776+0000 7f3b9f6bb640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f3b981527b0 con 0x7f3b9814c700 2024-04-25T00:14:39.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.776+0000 7f3b9dc58640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f3b980c55d0 0x7f3b9815ceb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:36256/0 (socket says 172.21.15.116:36256) 2024-04-25T00:14:39.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.776+0000 7f3b9dc58640 1 -- 172.21.15.116:0/3718361509 learned_addr learned my addr 172.21.15.116:0/3718361509 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:14:39.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.776+0000 7f3b9d457640 1 --2- 172.21.15.116:0/3718361509 >> v2:172.21.15.174:3300/0 conn(0x7f3b9814c700 0x7f3b980c5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:39.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.776+0000 7f3b8ffff640 1 -- 172.21.15.116:0/3718361509 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3825778760 0 0) 0x7f3b980a9e60 con 0x7f3b980c55d0 2024-04-25T00:14:39.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.776+0000 7f3b8ffff640 1 -- 172.21.15.116:0/3718361509 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3b84003530 con 0x7f3b980c55d0 2024-04-25T00:14:39.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.776+0000 7f3b8ffff640 1 -- 172.21.15.116:0/3718361509 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 49449675 0 0) 0x7f3b84003530 con 0x7f3b980c55d0 2024-04-25T00:14:39.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.776+0000 7f3b8ffff640 1 -- 172.21.15.116:0/3718361509 >> v2:172.21.15.174:3300/0 conn(0x7f3b9814c700 msgr2=0x7f3b980c5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:14:39.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.776+0000 7f3b8ffff640 1 --2- 172.21.15.116:0/3718361509 >> v2:172.21.15.174:3300/0 conn(0x7f3b9814c700 0x7f3b980c5090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:39.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.776+0000 7f3b8ffff640 1 -- 172.21.15.116:0/3718361509 >> v1:172.21.15.174:6789/0 conn(0x7f3b98155b20 legacy=0x7f3b98152c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:39.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.776+0000 7f3b8ffff640 1 -- 172.21.15.116:0/3718361509 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b980c41d0 con 0x7f3b980c55d0 2024-04-25T00:14:39.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.776+0000 7f3b8ffff640 1 -- 172.21.15.116:0/3718361509 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f3b90001a00 con 0x7f3b980c55d0 2024-04-25T00:14:39.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.776+0000 7f3b8ffff640 1 -- 172.21.15.116:0/3718361509 >> v1:172.21.15.116:6789/0 conn(0x7f3b980c55d0 legacy=0x7f3b9815ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:39.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.776+0000 7f3b8ffff640 1 --2- 172.21.15.116:0/3718361509 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b840039c0 0x7f3b84003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:39.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.776+0000 7f3b8ffff640 1 --2- 172.21.15.116:0/3718361509 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3b84004810 0x7f3b84004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:39.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.776+0000 7f3b8ffff640 1 --2- 172.21.15.116:0/3718361509 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3b840052f0 0x7f3b8400dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:39.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.776+0000 7f3b8ffff640 1 -- 172.21.15.116:0/3718361509 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3b981527b0 con 0x7f3b840039c0 2024-04-25T00:14:39.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.776+0000 7f3b8ffff640 1 -- 172.21.15.116:0/3718361509 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3b8400e3d0 con 0x7f3b840052f0 2024-04-25T00:14:39.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.776+0000 7f3b9d457640 1 --2- 172.21.15.116:0/3718361509 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b840039c0 0x7f3b84003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:39.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.776+0000 7f3b8ffff640 1 -- 172.21.15.116:0/3718361509 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f3b8400e5d0 con 0x7f3b84004810 2024-04-25T00:14:39.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.776+0000 7f3b9d457640 1 -- 172.21.15.116:0/3718361509 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3b84004810 msgr2=0x7f3b84004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:14:39.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.776+0000 7f3b9d457640 1 --2- 172.21.15.116:0/3718361509 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3b84004810 0x7f3b84004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:39.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.776+0000 7f3b9d457640 1 -- 172.21.15.116:0/3718361509 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3b840052f0 msgr2=0x7f3b8400dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:14:39.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.776+0000 7f3b9d457640 1 --2- 172.21.15.116:0/3718361509 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3b840052f0 0x7f3b8400dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:39.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.776+0000 7f3b9d457640 1 -- 172.21.15.116:0/3718361509 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b84003530 con 0x7f3b840039c0 2024-04-25T00:14:39.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.780+0000 7f3b9dc58640 1 -- 172.21.15.116:0/3718361509 reap_dead start 2024-04-25T00:14:39.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.780+0000 7f3b9d457640 1 --2- 172.21.15.116:0/3718361509 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b840039c0 0x7f3b84003de0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f3b94009ab0 tx=0x7f3b9401ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87aca56d1090ef38 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:39.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.780+0000 7f3b8ffff640 1 -- 172.21.15.116:0/3718361509 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b940096b0 con 0x7f3b840039c0 2024-04-25T00:14:39.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.780+0000 7f3b8ffff640 1 -- 172.21.15.116:0/3718361509 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b940a9440 con 0x7f3b840039c0 2024-04-25T00:14:39.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.780+0000 7f3b8ffff640 1 -- 172.21.15.116:0/3718361509 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b940a9830 con 0x7f3b840039c0 2024-04-25T00:14:39.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.780+0000 7f3b9f6bb640 1 -- 172.21.15.116:0/3718361509 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b840039c0 msgr2=0x7f3b84003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:39.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.780+0000 7f3b9f6bb640 1 --2- 172.21.15.116:0/3718361509 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b840039c0 0x7f3b84003de0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f3b94009ab0 tx=0x7f3b9401ca70 comp rx=0 tx=0).stop 2024-04-25T00:14:39.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.780+0000 7f3b9f6bb640 1 -- 172.21.15.116:0/3718361509 shutdown_connections 2024-04-25T00:14:39.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.780+0000 7f3b9f6bb640 1 --2- 172.21.15.116:0/3718361509 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b840039c0 0x7f3b84003de0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:39.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.780+0000 7f3b9f6bb640 1 -- 172.21.15.116:0/3718361509 >> 172.21.15.116:0/3718361509 conn(0x7f3b9806c720 msgr2=0x7f3b9806cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:39.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.780+0000 7f3b9f6bb640 1 -- 172.21.15.116:0/3718361509 shutdown_connections 2024-04-25T00:14:39.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.780+0000 7f3b9f6bb640 1 -- 172.21.15.116:0/3718361509 wait complete. 2024-04-25T00:14:39.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.784+0000 7f3b9f6bb640 1 Processor -- start 2024-04-25T00:14:39.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.784+0000 7f3b9f6bb640 1 -- start start 2024-04-25T00:14:39.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.784+0000 7f3b9f6bb640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3b840039c0 0x7f3b981e03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:39.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.784+0000 7f3b9f6bb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b9814ea40 0x7f3b980c2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:39.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.784+0000 7f3b9f6bb640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3b98149870 0x7f3b98149cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:39.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.784+0000 7f3b9f6bb640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3b980c5210 con 0x7f3b9814ea40 2024-04-25T00:14:39.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.784+0000 7f3b9f6bb640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3b980c5090 con 0x7f3b840039c0 2024-04-25T00:14:39.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.784+0000 7f3b9f6bb640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f3b980c5390 con 0x7f3b98149870 2024-04-25T00:14:39.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.784+0000 7f3b9cc56640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b9814ea40 0x7f3b980c2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:39.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.784+0000 7f3b9cc56640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b9814ea40 0x7f3b980c2bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60860/0 (socket says 172.21.15.116:60860) 2024-04-25T00:14:39.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.784+0000 7f3b9cc56640 1 -- 172.21.15.116:0/49679145 learned_addr learned my addr 172.21.15.116:0/49679145 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:14:39.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.784+0000 7f3b9dc58640 1 --2- 172.21.15.116:0/49679145 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3b98149870 0x7f3b98149cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:39.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.784+0000 7f3b9d457640 1 --2- 172.21.15.116:0/49679145 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3b840039c0 0x7f3b981e03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:39.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.784+0000 7f3b9cc56640 1 -- 172.21.15.116:0/49679145 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3b98149870 msgr2=0x7f3b98149cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:39.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.784+0000 7f3b9cc56640 1 --2- 172.21.15.116:0/49679145 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3b98149870 0x7f3b98149cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:39.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.784+0000 7f3b9cc56640 1 -- 172.21.15.116:0/49679145 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3b840039c0 msgr2=0x7f3b981e03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:39.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.784+0000 7f3b9cc56640 1 --2- 172.21.15.116:0/49679145 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3b840039c0 0x7f3b981e03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:39.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.784+0000 7f3b9cc56640 1 -- 172.21.15.116:0/49679145 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b981f2260 con 0x7f3b9814ea40 2024-04-25T00:14:39.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.784+0000 7f3b9d457640 1 --2- 172.21.15.116:0/49679145 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3b840039c0 0x7f3b981e03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:14:39.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.784+0000 7f3b9cc56640 1 --2- 172.21.15.116:0/49679145 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b9814ea40 0x7f3b980c2bc0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f3b8800c500 tx=0x7f3b8800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:39.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.784+0000 7f3b8dffb640 1 -- 172.21.15.116:0/49679145 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b880052d0 con 0x7f3b9814ea40 2024-04-25T00:14:39.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.784+0000 7f3b8dffb640 1 -- 172.21.15.116:0/49679145 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b88005490 con 0x7f3b9814ea40 2024-04-25T00:14:39.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.784+0000 7f3b9f6bb640 1 -- 172.21.15.116:0/49679145 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b981f2540 con 0x7f3b9814ea40 2024-04-25T00:14:39.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.784+0000 7f3b8dffb640 1 -- 172.21.15.116:0/49679145 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b88012850 con 0x7f3b9814ea40 2024-04-25T00:14:39.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.784+0000 7f3b9f6bb640 1 -- 172.21.15.116:0/49679145 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b981f2b00 con 0x7f3b9814ea40 2024-04-25T00:14:39.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.784+0000 7f3b9f6bb640 1 -- 172.21.15.116:0/49679145 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b98068a60 con 0x7f3b9814ea40 2024-04-25T00:14:39.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.788+0000 7f3b8dffb640 1 -- 172.21.15.116:0/49679145 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3b880129f0 con 0x7f3b9814ea40 2024-04-25T00:14:39.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.788+0000 7f3b8dffb640 1 --2- 172.21.15.116:0/49679145 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3b7c0678c0 0x7f3b7c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:39.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.788+0000 7f3b8dffb640 1 -- 172.21.15.116:0/49679145 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 8128+0+0 (secure 0 0 0) 0x7f3b88086200 con 0x7f3b9814ea40 2024-04-25T00:14:39.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.788+0000 7f3b9d457640 1 --2- 172.21.15.116:0/49679145 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3b7c0678c0 0x7f3b7c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:39.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.788+0000 7f3b9d457640 1 --2- 172.21.15.116:0/49679145 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3b7c0678c0 0x7f3b7c069d80 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f3b940ae000 tx=0x7f3b940046d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:39.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:39.792+0000 7f3b8dffb640 1 -- 172.21.15.116:0/49679145 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f3b88055e10 con 0x7f3b9814ea40 2024-04-25T00:14:40.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:40.028+0000 7f3b9f6bb640 1 -- 172.21.15.116:0/49679145 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3b980ad630 con 0x7f3b9814ea40 2024-04-25T00:14:40.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:40.028+0000 7f3b8dffb640 1 -- 172.21.15.116:0/49679145 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) ==== 94+0+6496 (secure 0 0 0) 0x7f3b8805a110 con 0x7f3b9814ea40 2024-04-25T00:14:40.030 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:14:40.030 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82},{"gid":5791,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3180024481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3180024481},{"type":"v1","addr":"172.21.15.174:6837","nonce":3180024481}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:12:48.771611+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":88,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6833/365773583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":365773583},{"type":"v1","addr":"172.21.15.116:6833","nonce":365773583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":1},{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:13:33.916554+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6582},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6582":{"gid":6582,"name":"a","rank":0,"incarnation":93,"state":"up:active","state_seq":14,"addr":"172.21.15.116:6835/462190523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":462190523},{"type":"v1","addr":"172.21.15.116:6835","nonce":462190523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6582,"qdb_cluster":[6582]},"id":2}]} 2024-04-25T00:14:40.031 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 97 2024-04-25T00:14:40.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:40.032+0000 7f3b9f6bb640 1 -- 172.21.15.116:0/49679145 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3b7c0678c0 msgr2=0x7f3b7c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:40.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:40.032+0000 7f3b9f6bb640 1 --2- 172.21.15.116:0/49679145 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3b7c0678c0 0x7f3b7c069d80 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f3b940ae000 tx=0x7f3b940046d0 comp rx=0 tx=0).stop 2024-04-25T00:14:40.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:40.032+0000 7f3b9f6bb640 1 -- 172.21.15.116:0/49679145 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b9814ea40 msgr2=0x7f3b980c2bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:40.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:40.032+0000 7f3b9f6bb640 1 --2- 172.21.15.116:0/49679145 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b9814ea40 0x7f3b980c2bc0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f3b8800c500 tx=0x7f3b8800c9c0 comp rx=0 tx=0).stop 2024-04-25T00:14:40.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:40.032+0000 7f3b9f6bb640 1 -- 172.21.15.116:0/49679145 shutdown_connections 2024-04-25T00:14:40.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:40.032+0000 7f3b9f6bb640 1 --2- 172.21.15.116:0/49679145 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3b7c0678c0 0x7f3b7c069d80 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:40.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:40.032+0000 7f3b9f6bb640 1 --2- 172.21.15.116:0/49679145 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3b98149870 0x7f3b98149cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:40.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:40.032+0000 7f3b9f6bb640 1 --2- 172.21.15.116:0/49679145 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b9814ea40 0x7f3b980c2bc0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:40.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:40.032+0000 7f3b9f6bb640 1 --2- 172.21.15.116:0/49679145 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3b840039c0 0x7f3b981e03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:40.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:40.032+0000 7f3b9f6bb640 1 -- 172.21.15.116:0/49679145 >> 172.21.15.116:0/49679145 conn(0x7f3b9806c720 msgr2=0x7f3b9806baa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:40.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:40.032+0000 7f3b9f6bb640 1 -- 172.21.15.116:0/49679145 shutdown_connections 2024-04-25T00:14:40.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:40.032+0000 7f3b9f6bb640 1 -- 172.21.15.116:0/49679145 wait complete. 2024-04-25T00:14:40.050 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:14:42.052 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:14:42.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.208+0000 7fdf96163640 1 Processor -- start 2024-04-25T00:14:42.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.208+0000 7fdf96163640 1 -- start start 2024-04-25T00:14:42.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.212+0000 7fdf96163640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdf900c56e0 0x7fdf900a6d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:42.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.212+0000 7fdf96163640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf90074770 con 0x7fdf900c3cd0 2024-04-25T00:14:42.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.212+0000 7fdf96163640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf90074a30 con 0x7fdf900c19d0 2024-04-25T00:14:42.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.212+0000 7fdf96163640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fdf900c21e0 con 0x7fdf900c56e0 2024-04-25T00:14:42.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.212+0000 7fdf8ffff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fdf900c3cd0 0x7fdf9006c220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:36262/0 (socket says 172.21.15.116:36262) 2024-04-25T00:14:42.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.212+0000 7fdf8ffff640 1 -- 172.21.15.116:0/1169543628 learned_addr learned my addr 172.21.15.116:0/1169543628 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:14:42.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.212+0000 7fdf8effd640 1 --2- 172.21.15.116:0/1169543628 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdf900c56e0 0x7fdf900a6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:42.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.212+0000 7fdf8e7fc640 1 -- 172.21.15.116:0/1169543628 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3689033635 0 0) 0x7fdf90074770 con 0x7fdf900c3cd0 2024-04-25T00:14:42.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.212+0000 7fdf8e7fc640 1 -- 172.21.15.116:0/1169543628 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf78003530 con 0x7fdf900c3cd0 2024-04-25T00:14:42.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.212+0000 7fdf8e7fc640 1 -- 172.21.15.116:0/1169543628 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3904971439 0 0) 0x7fdf78003530 con 0x7fdf900c3cd0 2024-04-25T00:14:42.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.212+0000 7fdf8e7fc640 1 -- 172.21.15.116:0/1169543628 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdf900c56e0 msgr2=0x7fdf900a6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:14:42.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.212+0000 7fdf8e7fc640 1 --2- 172.21.15.116:0/1169543628 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdf900c56e0 0x7fdf900a6d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:42.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.212+0000 7fdf8e7fc640 1 -- 172.21.15.116:0/1169543628 >> v1:172.21.15.174:6789/0 conn(0x7fdf900c19d0 legacy=0x7fdf900c0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:42.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.212+0000 7fdf8e7fc640 1 -- 172.21.15.116:0/1169543628 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf90159970 con 0x7fdf900c3cd0 2024-04-25T00:14:42.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.212+0000 7fdf8e7fc640 1 -- 172.21.15.116:0/1169543628 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7fdf88001a00 con 0x7fdf900c3cd0 2024-04-25T00:14:42.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.212+0000 7fdf8e7fc640 1 -- 172.21.15.116:0/1169543628 >> v1:172.21.15.116:6789/0 conn(0x7fdf900c3cd0 legacy=0x7fdf9006c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:42.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.212+0000 7fdf8e7fc640 1 --2- 172.21.15.116:0/1169543628 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdf900c56e0 0x7fdf780039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:42.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.212+0000 7fdf8e7fc640 1 --2- 172.21.15.116:0/1169543628 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdf78003f70 0x7fdf78004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:42.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.212+0000 7fdf8e7fc640 1 --2- 172.21.15.116:0/1169543628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf78004d50 0x7fdf780051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:42.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.212+0000 7fdf8e7fc640 1 -- 172.21.15.116:0/1169543628 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdf900c21e0 con 0x7fdf78004d50 2024-04-25T00:14:42.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.212+0000 7fdf8e7fc640 1 -- 172.21.15.116:0/1169543628 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fdf7800bb60 con 0x7fdf78003f70 2024-04-25T00:14:42.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.212+0000 7fdf8e7fc640 1 -- 172.21.15.116:0/1169543628 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fdf7800bd60 con 0x7fdf900c56e0 2024-04-25T00:14:42.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.212+0000 7fdf8ffff640 1 --2- 172.21.15.116:0/1169543628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf78004d50 0x7fdf780051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:42.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.212+0000 7fdf8effd640 1 --2- 172.21.15.116:0/1169543628 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdf78003f70 0x7fdf78004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:42.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.212+0000 7fdf8f7fe640 1 --2- 172.21.15.116:0/1169543628 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdf900c56e0 0x7fdf780039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:42.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.212+0000 7fdf8ffff640 1 -- 172.21.15.116:0/1169543628 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdf900c56e0 msgr2=0x7fdf780039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:14:42.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.212+0000 7fdf8ffff640 1 --2- 172.21.15.116:0/1169543628 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdf900c56e0 0x7fdf780039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:42.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.212+0000 7fdf8ffff640 1 -- 172.21.15.116:0/1169543628 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdf78003f70 msgr2=0x7fdf78004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:14:42.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.212+0000 7fdf8ffff640 1 --2- 172.21.15.116:0/1169543628 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdf78003f70 0x7fdf78004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:42.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.212+0000 7fdf8ffff640 1 -- 172.21.15.116:0/1169543628 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf78003530 con 0x7fdf78004d50 2024-04-25T00:14:42.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.212+0000 7fdf8effd640 1 --2- 172.21.15.116:0/1169543628 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdf78003f70 0x7fdf78004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:14:42.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.212+0000 7fdf8f7fe640 1 --2- 172.21.15.116:0/1169543628 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdf900c56e0 0x7fdf780039f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:14:42.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.216+0000 7fdf8ffff640 1 --2- 172.21.15.116:0/1169543628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf78004d50 0x7fdf780051d0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fdf8800a850 tx=0x7fdf8801fb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de858140a3bf1488 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:42.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.216+0000 7fdf8e7fc640 1 -- 172.21.15.116:0/1169543628 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf8800a4a0 con 0x7fdf78004d50 2024-04-25T00:14:42.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.216+0000 7fdf8e7fc640 1 -- 172.21.15.116:0/1169543628 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf880ad070 con 0x7fdf78004d50 2024-04-25T00:14:42.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.216+0000 7fdf8e7fc640 1 -- 172.21.15.116:0/1169543628 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf880a9980 con 0x7fdf78004d50 2024-04-25T00:14:42.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.216+0000 7fdf96163640 1 -- 172.21.15.116:0/1169543628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf78004d50 msgr2=0x7fdf780051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:42.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.216+0000 7fdf96163640 1 --2- 172.21.15.116:0/1169543628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf78004d50 0x7fdf780051d0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fdf8800a850 tx=0x7fdf8801fb00 comp rx=0 tx=0).stop 2024-04-25T00:14:42.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.216+0000 7fdf8ffff640 1 -- 172.21.15.116:0/1169543628 reap_dead start 2024-04-25T00:14:42.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.216+0000 7fdf96163640 1 -- 172.21.15.116:0/1169543628 shutdown_connections 2024-04-25T00:14:42.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.216+0000 7fdf96163640 1 -- 172.21.15.116:0/1169543628 >> 172.21.15.116:0/1169543628 conn(0x7fdf900c42e0 msgr2=0x7fdf900c46f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:42.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.216+0000 7fdf96163640 1 -- 172.21.15.116:0/1169543628 shutdown_connections 2024-04-25T00:14:42.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.216+0000 7fdf96163640 1 -- 172.21.15.116:0/1169543628 wait complete. 2024-04-25T00:14:42.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.216+0000 7fdf96163640 1 Processor -- start 2024-04-25T00:14:42.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.216+0000 7fdf96163640 1 -- start start 2024-04-25T00:14:42.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.220+0000 7fdf96163640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf90150130 0x7fdf90155e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:42.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.220+0000 7fdf96163640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdf901eba50 0x7fdf901ec250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:42.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.220+0000 7fdf96163640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdf901563b0 0x7fdf901ec790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:42.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.220+0000 7fdf96163640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdf9006c930 con 0x7fdf90150130 2024-04-25T00:14:42.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.220+0000 7fdf96163640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fdf900c5dc0 con 0x7fdf901eba50 2024-04-25T00:14:42.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.220+0000 7fdf96163640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fdf9006cab0 con 0x7fdf901563b0 2024-04-25T00:14:42.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.220+0000 7fdf8f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf90150130 0x7fdf90155e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:42.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.220+0000 7fdf8f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf90150130 0x7fdf90155e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60886/0 (socket says 172.21.15.116:60886) 2024-04-25T00:14:42.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.220+0000 7fdf8f7fe640 1 -- 172.21.15.116:0/596170615 learned_addr learned my addr 172.21.15.116:0/596170615 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:14:42.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.220+0000 7fdf8ffff640 1 --2- 172.21.15.116:0/596170615 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdf901563b0 0x7fdf901ec790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:42.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.220+0000 7fdf8effd640 1 --2- 172.21.15.116:0/596170615 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdf901eba50 0x7fdf901ec250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:42.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.220+0000 7fdf8f7fe640 1 -- 172.21.15.116:0/596170615 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdf901563b0 msgr2=0x7fdf901ec790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:42.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.220+0000 7fdf8f7fe640 1 --2- 172.21.15.116:0/596170615 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdf901563b0 0x7fdf901ec790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:42.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.220+0000 7fdf8f7fe640 1 -- 172.21.15.116:0/596170615 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdf901eba50 msgr2=0x7fdf901ec250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:42.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.220+0000 7fdf8f7fe640 1 --2- 172.21.15.116:0/596170615 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdf901eba50 0x7fdf901ec250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:42.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.220+0000 7fdf8f7fe640 1 -- 172.21.15.116:0/596170615 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf9009c680 con 0x7fdf90150130 2024-04-25T00:14:42.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.220+0000 7fdf8ffff640 1 --2- 172.21.15.116:0/596170615 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdf901563b0 0x7fdf901ec790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:14:42.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.220+0000 7fdf8f7fe640 1 --2- 172.21.15.116:0/596170615 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf90150130 0x7fdf90155e70 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fdf8000e7b0 tx=0x7fdf8000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:42.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.220+0000 7fdf77fff640 1 -- 172.21.15.116:0/596170615 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf800050d0 con 0x7fdf90150130 2024-04-25T00:14:42.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.220+0000 7fdf77fff640 1 -- 172.21.15.116:0/596170615 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf80018070 con 0x7fdf90150130 2024-04-25T00:14:42.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.220+0000 7fdf96163640 1 -- 172.21.15.116:0/596170615 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf9009acf0 con 0x7fdf90150130 2024-04-25T00:14:42.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.220+0000 7fdf77fff640 1 -- 172.21.15.116:0/596170615 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf8000c660 con 0x7fdf90150130 2024-04-25T00:14:42.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.220+0000 7fdf96163640 1 -- 172.21.15.116:0/596170615 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf901eb250 con 0x7fdf90150130 2024-04-25T00:14:42.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.220+0000 7fdf96163640 1 -- 172.21.15.116:0/596170615 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf901ead60 con 0x7fdf90150130 2024-04-25T00:14:42.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.220+0000 7fdf77fff640 1 -- 172.21.15.116:0/596170615 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fdf8000c800 con 0x7fdf90150130 2024-04-25T00:14:42.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.220+0000 7fdf77fff640 1 --2- 172.21.15.116:0/596170615 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdf7c0678f0 0x7fdf7c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:42.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.220+0000 7fdf77fff640 1 -- 172.21.15.116:0/596170615 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 8128+0+0 (secure 0 0 0) 0x7fdf80086290 con 0x7fdf90150130 2024-04-25T00:14:42.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.220+0000 7fdf8effd640 1 --2- 172.21.15.116:0/596170615 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdf7c0678f0 0x7fdf7c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:42.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.224+0000 7fdf8effd640 1 --2- 172.21.15.116:0/596170615 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdf7c0678f0 0x7fdf7c069db0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fdf901ee2b0 tx=0x7fdf84008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:42.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.228+0000 7fdf77fff640 1 -- 172.21.15.116:0/596170615 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fdf800565b0 con 0x7fdf90150130 2024-04-25T00:14:42.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.476+0000 7fdf96163640 1 -- 172.21.15.116:0/596170615 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdf90068a60 con 0x7fdf90150130 2024-04-25T00:14:42.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.476+0000 7fdf77fff640 1 -- 172.21.15.116:0/596170615 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) ==== 94+0+6496 (secure 0 0 0) 0x7fdf80055eb0 con 0x7fdf90150130 2024-04-25T00:14:42.476 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:14:42.476 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82},{"gid":5791,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3180024481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3180024481},{"type":"v1","addr":"172.21.15.174:6837","nonce":3180024481}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:12:48.771611+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":88,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6833/365773583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":365773583},{"type":"v1","addr":"172.21.15.116:6833","nonce":365773583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":1},{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:13:33.916554+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6582},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6582":{"gid":6582,"name":"a","rank":0,"incarnation":93,"state":"up:active","state_seq":14,"addr":"172.21.15.116:6835/462190523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":462190523},{"type":"v1","addr":"172.21.15.116:6835","nonce":462190523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6582,"qdb_cluster":[6582]},"id":2}]} 2024-04-25T00:14:42.477 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 97 2024-04-25T00:14:42.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.476+0000 7fdf96163640 1 -- 172.21.15.116:0/596170615 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdf7c0678f0 msgr2=0x7fdf7c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:42.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.476+0000 7fdf96163640 1 --2- 172.21.15.116:0/596170615 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdf7c0678f0 0x7fdf7c069db0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fdf901ee2b0 tx=0x7fdf84008040 comp rx=0 tx=0).stop 2024-04-25T00:14:42.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.476+0000 7fdf96163640 1 -- 172.21.15.116:0/596170615 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf90150130 msgr2=0x7fdf90155e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:42.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.476+0000 7fdf96163640 1 --2- 172.21.15.116:0/596170615 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf90150130 0x7fdf90155e70 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fdf8000e7b0 tx=0x7fdf8000ec70 comp rx=0 tx=0).stop 2024-04-25T00:14:42.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.476+0000 7fdf96163640 1 -- 172.21.15.116:0/596170615 shutdown_connections 2024-04-25T00:14:42.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.476+0000 7fdf96163640 1 --2- 172.21.15.116:0/596170615 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdf7c0678f0 0x7fdf7c069db0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:42.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.476+0000 7fdf96163640 1 --2- 172.21.15.116:0/596170615 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdf901563b0 0x7fdf901ec790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:42.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.476+0000 7fdf96163640 1 --2- 172.21.15.116:0/596170615 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdf901eba50 0x7fdf901ec250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:42.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.476+0000 7fdf96163640 1 --2- 172.21.15.116:0/596170615 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdf90150130 0x7fdf90155e70 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:42.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.476+0000 7fdf96163640 1 -- 172.21.15.116:0/596170615 >> 172.21.15.116:0/596170615 conn(0x7fdf900c42e0 msgr2=0x7fdf90071590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:42.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.476+0000 7fdf96163640 1 -- 172.21.15.116:0/596170615 shutdown_connections 2024-04-25T00:14:42.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:42.476+0000 7fdf96163640 1 -- 172.21.15.116:0/596170615 wait complete. 2024-04-25T00:14:42.495 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:14:44.496 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:14:44.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.632+0000 7f8a4d9c9640 1 Processor -- start 2024-04-25T00:14:44.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.632+0000 7f8a4d9c9640 1 -- start start 2024-04-25T00:14:44.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.632+0000 7f8a4d9c9640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a48150d40 0x7f8a48154170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:44.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.632+0000 7f8a4d9c9640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a48070180 con 0x7f8a480c4c20 2024-04-25T00:14:44.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.632+0000 7f8a4d9c9640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a48070460 con 0x7f8a48070770 2024-04-25T00:14:44.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.632+0000 7f8a4d9c9640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f8a480c7530 con 0x7f8a48150d40 2024-04-25T00:14:44.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.632+0000 7f8a467fc640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f8a480c4c20 0x7f8a480c6e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:36276/0 (socket says 172.21.15.116:36276) 2024-04-25T00:14:44.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.632+0000 7f8a467fc640 1 -- 172.21.15.116:0/1370283457 learned_addr learned my addr 172.21.15.116:0/1370283457 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:14:44.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.632+0000 7f8a46ffd640 1 --2- 172.21.15.116:0/1370283457 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a48150d40 0x7f8a48154170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:44.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.632+0000 7f8a45ffb640 1 -- 172.21.15.116:0/1370283457 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1410258422 0 0) 0x7f8a48070180 con 0x7f8a480c4c20 2024-04-25T00:14:44.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.632+0000 7f8a45ffb640 1 -- 172.21.15.116:0/1370283457 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8a30003530 con 0x7f8a480c4c20 2024-04-25T00:14:44.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.632+0000 7f8a45ffb640 1 -- 172.21.15.116:0/1370283457 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3018689829 0 0) 0x7f8a30003530 con 0x7f8a480c4c20 2024-04-25T00:14:44.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.632+0000 7f8a45ffb640 1 -- 172.21.15.116:0/1370283457 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a48150d40 msgr2=0x7f8a48154170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:14:44.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.632+0000 7f8a45ffb640 1 --2- 172.21.15.116:0/1370283457 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a48150d40 0x7f8a48154170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:44.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.632+0000 7f8a45ffb640 1 -- 172.21.15.116:0/1370283457 >> v1:172.21.15.174:6789/0 conn(0x7f8a48070770 legacy=0x7f8a480ab200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:44.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.632+0000 7f8a45ffb640 1 -- 172.21.15.116:0/1370283457 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a48153970 con 0x7f8a480c4c20 2024-04-25T00:14:44.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.632+0000 7f8a45ffb640 1 -- 172.21.15.116:0/1370283457 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f8a3c001680 con 0x7f8a480c4c20 2024-04-25T00:14:44.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.632+0000 7f8a45ffb640 1 -- 172.21.15.116:0/1370283457 >> v1:172.21.15.116:6789/0 conn(0x7f8a480c4c20 legacy=0x7f8a480c6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:44.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.632+0000 7f8a45ffb640 1 --2- 172.21.15.116:0/1370283457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a300039f0 0x7f8a30003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:44.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.632+0000 7f8a45ffb640 1 --2- 172.21.15.116:0/1370283457 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a30004810 0x7f8a30004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:44.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.632+0000 7f8a45ffb640 1 --2- 172.21.15.116:0/1370283457 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a48150d40 0x7f8a30005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:44.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.632+0000 7f8a45ffb640 1 -- 172.21.15.116:0/1370283457 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8a480c7530 con 0x7f8a300039f0 2024-04-25T00:14:44.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.632+0000 7f8a45ffb640 1 -- 172.21.15.116:0/1370283457 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8a3000bb00 con 0x7f8a30004810 2024-04-25T00:14:44.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.632+0000 7f8a45ffb640 1 -- 172.21.15.116:0/1370283457 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f8a3000bd00 con 0x7f8a48150d40 2024-04-25T00:14:44.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.632+0000 7f8a46ffd640 1 --2- 172.21.15.116:0/1370283457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a300039f0 0x7f8a30003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:44.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.632+0000 7f8a467fc640 1 --2- 172.21.15.116:0/1370283457 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a30004810 0x7f8a30004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:44.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.632+0000 7f8a46ffd640 1 -- 172.21.15.116:0/1370283457 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a48150d40 msgr2=0x7f8a30005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:14:44.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.632+0000 7f8a477fe640 1 --2- 172.21.15.116:0/1370283457 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a48150d40 0x7f8a30005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:44.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.632+0000 7f8a46ffd640 1 --2- 172.21.15.116:0/1370283457 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a48150d40 0x7f8a30005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:44.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.632+0000 7f8a46ffd640 1 -- 172.21.15.116:0/1370283457 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a30004810 msgr2=0x7f8a30004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:14:44.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.632+0000 7f8a46ffd640 1 --2- 172.21.15.116:0/1370283457 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a30004810 0x7f8a30004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:44.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.632+0000 7f8a46ffd640 1 -- 172.21.15.116:0/1370283457 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a30003530 con 0x7f8a300039f0 2024-04-25T00:14:44.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.632+0000 7f8a477fe640 1 --2- 172.21.15.116:0/1370283457 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a48150d40 0x7f8a30005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:14:44.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.636+0000 7f8a46ffd640 1 --2- 172.21.15.116:0/1370283457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a300039f0 0x7f8a30003e10 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f8a38009ae0 tx=0x7f8a3801ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff06a4809e1d6805 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:44.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.636+0000 7f8a45ffb640 1 -- 172.21.15.116:0/1370283457 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a380b2070 con 0x7f8a300039f0 2024-04-25T00:14:44.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.636+0000 7f8a45ffb640 1 -- 172.21.15.116:0/1370283457 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a380a5550 con 0x7f8a300039f0 2024-04-25T00:14:44.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.636+0000 7f8a45ffb640 1 -- 172.21.15.116:0/1370283457 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a380096b0 con 0x7f8a300039f0 2024-04-25T00:14:44.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.636+0000 7f8a4d9c9640 1 -- 172.21.15.116:0/1370283457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a300039f0 msgr2=0x7f8a30003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:44.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.636+0000 7f8a4d9c9640 1 --2- 172.21.15.116:0/1370283457 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a300039f0 0x7f8a30003e10 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f8a38009ae0 tx=0x7f8a3801ca70 comp rx=0 tx=0).stop 2024-04-25T00:14:44.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.636+0000 7f8a477fe640 1 -- 172.21.15.116:0/1370283457 reap_dead start 2024-04-25T00:14:44.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.636+0000 7f8a4d9c9640 1 -- 172.21.15.116:0/1370283457 shutdown_connections 2024-04-25T00:14:44.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.636+0000 7f8a4d9c9640 1 -- 172.21.15.116:0/1370283457 >> 172.21.15.116:0/1370283457 conn(0x7f8a4806f0c0 msgr2=0x7f8a4806f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:44.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.636+0000 7f8a4d9c9640 1 -- 172.21.15.116:0/1370283457 shutdown_connections 2024-04-25T00:14:44.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.636+0000 7f8a4d9c9640 1 -- 172.21.15.116:0/1370283457 wait complete. 2024-04-25T00:14:44.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.640+0000 7f8a4d9c9640 1 Processor -- start 2024-04-25T00:14:44.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.640+0000 7f8a4d9c9640 1 -- start start 2024-04-25T00:14:44.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.640+0000 7f8a4d9c9640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a480c25b0 0x7f8a481eeca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:44.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.640+0000 7f8a4d9c9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a480ade00 0x7f8a481f1850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:44.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.640+0000 7f8a4d9c9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a4806d940 0x7f8a4806ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:44.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.640+0000 7f8a4d9c9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8a480ab910 con 0x7f8a480ade00 2024-04-25T00:14:44.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.640+0000 7f8a4d9c9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8a4806ff30 con 0x7f8a4806d940 2024-04-25T00:14:44.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.640+0000 7f8a4d9c9640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f8a480aba90 con 0x7f8a480c25b0 2024-04-25T00:14:44.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.640+0000 7f8a467fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a480ade00 0x7f8a481f1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:44.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.640+0000 7f8a467fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a480ade00 0x7f8a481f1850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60900/0 (socket says 172.21.15.116:60900) 2024-04-25T00:14:44.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.640+0000 7f8a467fc640 1 -- 172.21.15.116:0/1103373484 learned_addr learned my addr 172.21.15.116:0/1103373484 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:14:44.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.640+0000 7f8a46ffd640 1 --2- 172.21.15.116:0/1103373484 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a480c25b0 0x7f8a481eeca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:44.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.640+0000 7f8a477fe640 1 --2- 172.21.15.116:0/1103373484 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a4806d940 0x7f8a4806ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:44.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.640+0000 7f8a467fc640 1 -- 172.21.15.116:0/1103373484 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a480c25b0 msgr2=0x7f8a481eeca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:44.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.640+0000 7f8a467fc640 1 --2- 172.21.15.116:0/1103373484 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a480c25b0 0x7f8a481eeca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:44.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.640+0000 7f8a467fc640 1 -- 172.21.15.116:0/1103373484 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a4806d940 msgr2=0x7f8a4806ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:44.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.640+0000 7f8a467fc640 1 --2- 172.21.15.116:0/1103373484 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a4806d940 0x7f8a4806ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:44.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.640+0000 7f8a467fc640 1 -- 172.21.15.116:0/1103373484 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a480acf40 con 0x7f8a480ade00 2024-04-25T00:14:44.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.640+0000 7f8a46ffd640 1 --2- 172.21.15.116:0/1103373484 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a480c25b0 0x7f8a481eeca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:14:44.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.640+0000 7f8a467fc640 1 --2- 172.21.15.116:0/1103373484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a480ade00 0x7f8a481f1850 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c00eb20 tx=0x7f8a3c001750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:44.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.640+0000 7f8a2ffff640 1 -- 172.21.15.116:0/1103373484 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c00c870 con 0x7f8a480ade00 2024-04-25T00:14:44.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.640+0000 7f8a2ffff640 1 -- 172.21.15.116:0/1103373484 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a3c018070 con 0x7f8a480ade00 2024-04-25T00:14:44.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.640+0000 7f8a4d9c9640 1 -- 172.21.15.116:0/1103373484 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a481ef1e0 con 0x7f8a480ade00 2024-04-25T00:14:44.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.640+0000 7f8a2ffff640 1 -- 172.21.15.116:0/1103373484 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c00cc50 con 0x7f8a480ade00 2024-04-25T00:14:44.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.640+0000 7f8a4d9c9640 1 -- 172.21.15.116:0/1103373484 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a4814e890 con 0x7f8a480ade00 2024-04-25T00:14:44.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.640+0000 7f8a4d9c9640 1 -- 172.21.15.116:0/1103373484 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a4814ead0 con 0x7f8a480ade00 2024-04-25T00:14:44.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.644+0000 7f8a2ffff640 1 -- 172.21.15.116:0/1103373484 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8a3c01d460 con 0x7f8a480ade00 2024-04-25T00:14:44.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.644+0000 7f8a2ffff640 1 --2- 172.21.15.116:0/1103373484 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8a340678c0 0x7f8a34069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:44.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.644+0000 7f8a2ffff640 1 -- 172.21.15.116:0/1103373484 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 8128+0+0 (secure 0 0 0) 0x7f8a3c0865e0 con 0x7f8a480ade00 2024-04-25T00:14:44.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.644+0000 7f8a46ffd640 1 --2- 172.21.15.116:0/1103373484 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8a340678c0 0x7f8a34069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:44.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.644+0000 7f8a46ffd640 1 --2- 172.21.15.116:0/1103373484 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8a340678c0 0x7f8a34069d80 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f8a380af000 tx=0x7f8a38008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:44.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.648+0000 7f8a2ffff640 1 -- 172.21.15.116:0/1103373484 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f8a4814ead0 con 0x7f8a480ade00 2024-04-25T00:14:44.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.896+0000 7f8a4d9c9640 1 -- 172.21.15.116:0/1103373484 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8a48068a60 con 0x7f8a480ade00 2024-04-25T00:14:44.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.900+0000 7f8a2ffff640 1 -- 172.21.15.116:0/1103373484 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) ==== 94+0+6496 (secure 0 0 0) 0x7f8a3c056840 con 0x7f8a480ade00 2024-04-25T00:14:44.899 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:14:44.900 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82},{"gid":5791,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3180024481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3180024481},{"type":"v1","addr":"172.21.15.174:6837","nonce":3180024481}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:12:48.771611+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":88,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6833/365773583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":365773583},{"type":"v1","addr":"172.21.15.116:6833","nonce":365773583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":1},{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:13:33.916554+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6582},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6582":{"gid":6582,"name":"a","rank":0,"incarnation":93,"state":"up:active","state_seq":14,"addr":"172.21.15.116:6835/462190523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":462190523},{"type":"v1","addr":"172.21.15.116:6835","nonce":462190523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6582,"qdb_cluster":[6582]},"id":2}]} 2024-04-25T00:14:44.900 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 97 2024-04-25T00:14:44.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.900+0000 7f8a4d9c9640 1 -- 172.21.15.116:0/1103373484 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8a340678c0 msgr2=0x7f8a34069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:44.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.900+0000 7f8a4d9c9640 1 --2- 172.21.15.116:0/1103373484 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8a340678c0 0x7f8a34069d80 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f8a380af000 tx=0x7f8a38008040 comp rx=0 tx=0).stop 2024-04-25T00:14:44.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.900+0000 7f8a4d9c9640 1 -- 172.21.15.116:0/1103373484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a480ade00 msgr2=0x7f8a481f1850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:44.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.900+0000 7f8a4d9c9640 1 --2- 172.21.15.116:0/1103373484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a480ade00 0x7f8a481f1850 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c00eb20 tx=0x7f8a3c001750 comp rx=0 tx=0).stop 2024-04-25T00:14:44.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.900+0000 7f8a4d9c9640 1 -- 172.21.15.116:0/1103373484 shutdown_connections 2024-04-25T00:14:44.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.900+0000 7f8a4d9c9640 1 --2- 172.21.15.116:0/1103373484 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8a340678c0 0x7f8a34069d80 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:44.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.900+0000 7f8a4d9c9640 1 --2- 172.21.15.116:0/1103373484 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a4806d940 0x7f8a4806ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:44.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.900+0000 7f8a4d9c9640 1 --2- 172.21.15.116:0/1103373484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a480ade00 0x7f8a481f1850 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:44.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.900+0000 7f8a4d9c9640 1 --2- 172.21.15.116:0/1103373484 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a480c25b0 0x7f8a481eeca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:44.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.900+0000 7f8a4d9c9640 1 -- 172.21.15.116:0/1103373484 >> 172.21.15.116:0/1103373484 conn(0x7f8a4806f0c0 msgr2=0x7f8a480671c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:44.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.900+0000 7f8a4d9c9640 1 -- 172.21.15.116:0/1103373484 shutdown_connections 2024-04-25T00:14:44.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:44.900+0000 7f8a4d9c9640 1 -- 172.21.15.116:0/1103373484 wait complete. 2024-04-25T00:14:44.925 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:14:46.920 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:14:46.926 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:14:47.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.076+0000 7f757d6de640 1 Processor -- start 2024-04-25T00:14:47.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.076+0000 7f757d6de640 1 -- start start 2024-04-25T00:14:47.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.076+0000 7f757d6de640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f75780c4c20 0x7f7578154170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:47.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.076+0000 7f757d6de640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f7578070770 0x7f75780ab200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:47.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.076+0000 7f757d6de640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7578070180 con 0x7f7578150d40 2024-04-25T00:14:47.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.076+0000 7f757d6de640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f75780c7530 con 0x7f7578070770 2024-04-25T00:14:47.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.076+0000 7f757d6de640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f7578071280 con 0x7f75780c4c20 2024-04-25T00:14:47.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.080+0000 7f75767fc640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f75780c4c20 0x7f7578154170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:47.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.080+0000 7f75777fe640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f7578070770 0x7f75780ab200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:47.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.080+0000 7f75767fc640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f75780c4c20 0x7f7578154170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:36612/0 (socket says 172.21.15.116:36612) 2024-04-25T00:14:47.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.080+0000 7f75767fc640 1 -- 172.21.15.116:0/1106654921 learned_addr learned my addr 172.21.15.116:0/1106654921 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:14:47.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.080+0000 7f75767fc640 1 -- 172.21.15.116:0/1106654921 >> v2:172.21.15.174:3300/0 conn(0x7f7578070770 msgr2=0x7f75780ab200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:14:47.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.080+0000 7f75767fc640 1 --2- 172.21.15.116:0/1106654921 >> v2:172.21.15.174:3300/0 conn(0x7f7578070770 0x7f75780ab200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:47.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.080+0000 7f75767fc640 1 -- 172.21.15.116:0/1106654921 >> v1:172.21.15.174:6789/0 conn(0x7f7578150d40 legacy=0x7f75780c6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:47.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.080+0000 7f75767fc640 1 -- 172.21.15.116:0/1106654921 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7578070460 con 0x7f75780c4c20 2024-04-25T00:14:47.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.080+0000 7f75767fc640 1 --2- 172.21.15.116:0/1106654921 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f75780c4c20 0x7f7578154170 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f756c0098a0 tx=0x7f756c01c4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6992145c44a7702 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:47.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.080+0000 7f7575ffb640 1 -- 172.21.15.116:0/1106654921 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f756c0b1070 con 0x7f75780c4c20 2024-04-25T00:14:47.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.080+0000 7f7575ffb640 1 -- 172.21.15.116:0/1106654921 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f756c0ac620 con 0x7f75780c4c20 2024-04-25T00:14:47.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.080+0000 7f7575ffb640 1 -- 172.21.15.116:0/1106654921 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f756c0ac9f0 con 0x7f75780c4c20 2024-04-25T00:14:47.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.084+0000 7f757d6de640 1 -- 172.21.15.116:0/1106654921 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f75780c4c20 msgr2=0x7f7578154170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:47.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.084+0000 7f757d6de640 1 --2- 172.21.15.116:0/1106654921 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f75780c4c20 0x7f7578154170 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f756c0098a0 tx=0x7f756c01c4b0 comp rx=0 tx=0).stop 2024-04-25T00:14:47.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.084+0000 7f757d6de640 1 -- 172.21.15.116:0/1106654921 shutdown_connections 2024-04-25T00:14:47.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.084+0000 7f757d6de640 1 --2- 172.21.15.116:0/1106654921 >> v2:172.21.15.174:3300/0 conn(0x7f7578070770 0x7f75780ab200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:47.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.084+0000 7f757d6de640 1 --2- 172.21.15.116:0/1106654921 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f75780c4c20 0x7f7578154170 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:47.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.084+0000 7f757d6de640 1 -- 172.21.15.116:0/1106654921 >> 172.21.15.116:0/1106654921 conn(0x7f757806f0c0 msgr2=0x7f757806f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:47.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.084+0000 7f757d6de640 1 -- 172.21.15.116:0/1106654921 shutdown_connections 2024-04-25T00:14:47.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.084+0000 7f757d6de640 1 -- 172.21.15.116:0/1106654921 wait complete. 2024-04-25T00:14:47.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.084+0000 7f757d6de640 1 Processor -- start 2024-04-25T00:14:47.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.084+0000 7f757d6de640 1 -- start start 2024-04-25T00:14:47.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.084+0000 7f757d6de640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7578070770 0x7f7578155340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:47.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.084+0000 7f757d6de640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75780c4c20 0x7f7578149ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:47.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.084+0000 7f757d6de640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7578150d40 0x7f75780adef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:47.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.084+0000 7f757d6de640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7578151120 con 0x7f75780c4c20 2024-04-25T00:14:47.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.084+0000 7f757d6de640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f757806ff30 con 0x7f7578070770 2024-04-25T00:14:47.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.084+0000 7f757d6de640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f75781512a0 con 0x7f7578150d40 2024-04-25T00:14:47.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.084+0000 7f75767fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75780c4c20 0x7f7578149ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:47.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.084+0000 7f75767fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75780c4c20 0x7f7578149ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38208/0 (socket says 172.21.15.116:38208) 2024-04-25T00:14:47.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.084+0000 7f75767fc640 1 -- 172.21.15.116:0/3976467107 learned_addr learned my addr 172.21.15.116:0/3976467107 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:14:47.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.084+0000 7f7576ffd640 1 --2- 172.21.15.116:0/3976467107 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7578070770 0x7f7578155340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:47.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.084+0000 7f75777fe640 1 --2- 172.21.15.116:0/3976467107 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7578150d40 0x7f75780adef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:47.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.084+0000 7f75767fc640 1 -- 172.21.15.116:0/3976467107 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7578150d40 msgr2=0x7f75780adef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:47.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.084+0000 7f75767fc640 1 --2- 172.21.15.116:0/3976467107 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7578150d40 0x7f75780adef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:47.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.084+0000 7f75767fc640 1 -- 172.21.15.116:0/3976467107 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7578070770 msgr2=0x7f7578155340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:47.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.084+0000 7f75767fc640 1 --2- 172.21.15.116:0/3976467107 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7578070770 0x7f7578155340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:47.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.084+0000 7f75767fc640 1 -- 172.21.15.116:0/3976467107 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75780a9e40 con 0x7f75780c4c20 2024-04-25T00:14:47.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.084+0000 7f75777fe640 1 --2- 172.21.15.116:0/3976467107 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7578150d40 0x7f75780adef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:14:47.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.084+0000 7f75767fc640 1 --2- 172.21.15.116:0/3976467107 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75780c4c20 0x7f7578149ed0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f756c01a0e0 tx=0x7f756c019250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:47.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.084+0000 7f755f7fe640 1 -- 172.21.15.116:0/3976467107 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f756c0b9020 con 0x7f75780c4c20 2024-04-25T00:14:47.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.084+0000 7f755f7fe640 1 -- 172.21.15.116:0/3976467107 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f756c0b5410 con 0x7f75780c4c20 2024-04-25T00:14:47.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.084+0000 7f755f7fe640 1 -- 172.21.15.116:0/3976467107 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f756c0b1040 con 0x7f75780c4c20 2024-04-25T00:14:47.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.084+0000 7f757d6de640 1 -- 172.21.15.116:0/3976467107 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75780ac100 con 0x7f75780c4c20 2024-04-25T00:14:47.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.084+0000 7f757d6de640 1 -- 172.21.15.116:0/3976467107 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75780ac6c0 con 0x7f75780c4c20 2024-04-25T00:14:47.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.088+0000 7f757d6de640 1 -- 172.21.15.116:0/3976467107 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7578068a60 con 0x7f75780c4c20 2024-04-25T00:14:47.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.088+0000 7f755f7fe640 1 -- 172.21.15.116:0/3976467107 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f756c0bc020 con 0x7f75780c4c20 2024-04-25T00:14:47.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.088+0000 7f755f7fe640 1 --2- 172.21.15.116:0/3976467107 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f75640678c0 0x7f7564069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:47.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.088+0000 7f755f7fe640 1 -- 172.21.15.116:0/3976467107 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 8128+0+0 (secure 0 0 0) 0x7f756c11ffd0 con 0x7f75780c4c20 2024-04-25T00:14:47.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.088+0000 7f7576ffd640 1 --2- 172.21.15.116:0/3976467107 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f75640678c0 0x7f7564069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:47.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.088+0000 7f7576ffd640 1 --2- 172.21.15.116:0/3976467107 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f75640678c0 0x7f7564069d80 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f7568001810 tx=0x7f7568009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:47.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.092+0000 7f755f7fe640 1 -- 172.21.15.116:0/3976467107 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f756c0c3030 con 0x7f75780c4c20 2024-04-25T00:14:47.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.112+0000 7f51a2787640 1 Processor -- start 2024-04-25T00:14:47.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.112+0000 7f51a2787640 1 -- start start 2024-04-25T00:14:47.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.112+0000 7f51a2787640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f519c152f60 0x7f519c0c6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:47.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.112+0000 7f51a2787640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f519c0c3c30 0x7f519c06fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:47.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.112+0000 7f51a2787640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f519c158ea0 0x7f519c06c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:47.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.112+0000 7f51a2787640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f519c06e8f0 con 0x7f519c152f60 2024-04-25T00:14:47.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.112+0000 7f51a2787640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f519c06ebd0 con 0x7f519c158ea0 2024-04-25T00:14:47.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.112+0000 7f51a2787640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f519c06f330 con 0x7f519c0c3c30 2024-04-25T00:14:47.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.112+0000 7f519bfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f519c152f60 0x7f519c0c6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:47.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.112+0000 7f519bfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f519c152f60 0x7f519c0c6080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38210/0 (socket says 172.21.15.116:38210) 2024-04-25T00:14:47.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.112+0000 7f519bfff640 1 -- 172.21.15.116:0/2056113045 learned_addr learned my addr 172.21.15.116:0/2056113045 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:14:47.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.112+0000 7f519b7fe640 1 --2- 172.21.15.116:0/2056113045 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f519c0c3c30 0x7f519c06fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:47.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.112+0000 7f51a0d24640 1 --2- 172.21.15.116:0/2056113045 >> v2:172.21.15.174:3300/0 conn(0x7f519c158ea0 0x7f519c06c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:47.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.112+0000 7f519bfff640 1 -- 172.21.15.116:0/2056113045 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f519c0c3c30 msgr2=0x7f519c06fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:14:47.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.112+0000 7f519bfff640 1 --2- 172.21.15.116:0/2056113045 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f519c0c3c30 0x7f519c06fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:47.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.112+0000 7f519bfff640 1 -- 172.21.15.116:0/2056113045 >> v2:172.21.15.174:3300/0 conn(0x7f519c158ea0 msgr2=0x7f519c06c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:14:47.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.112+0000 7f519bfff640 1 --2- 172.21.15.116:0/2056113045 >> v2:172.21.15.174:3300/0 conn(0x7f519c158ea0 0x7f519c06c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:47.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.112+0000 7f519bfff640 1 -- 172.21.15.116:0/2056113045 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f519c06f610 con 0x7f519c152f60 2024-04-25T00:14:47.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.116+0000 7f519bfff640 1 --2- 172.21.15.116:0/2056113045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f519c152f60 0x7f519c0c6080 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f519c0a7420 tx=0x7f518c01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdd09328999b54bf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:47.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.116+0000 7f519affd640 1 -- 172.21.15.116:0/2056113045 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f518c0b2070 con 0x7f519c152f60 2024-04-25T00:14:47.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.116+0000 7f519affd640 1 -- 172.21.15.116:0/2056113045 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f518c0a9070 con 0x7f519c152f60 2024-04-25T00:14:47.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.116+0000 7f519affd640 1 -- 172.21.15.116:0/2056113045 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f518c0ad880 con 0x7f519c152f60 2024-04-25T00:14:47.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.116+0000 7f51a2787640 1 -- 172.21.15.116:0/2056113045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f519c152f60 msgr2=0x7f519c0c6080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:47.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.116+0000 7f51a2787640 1 --2- 172.21.15.116:0/2056113045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f519c152f60 0x7f519c0c6080 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f519c0a7420 tx=0x7f518c01c490 comp rx=0 tx=0).stop 2024-04-25T00:14:47.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.116+0000 7f51a2787640 1 -- 172.21.15.116:0/2056113045 shutdown_connections 2024-04-25T00:14:47.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.116+0000 7f51a2787640 1 --2- 172.21.15.116:0/2056113045 >> v2:172.21.15.174:3300/0 conn(0x7f519c158ea0 0x7f519c06c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:47.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.116+0000 7f51a2787640 1 --2- 172.21.15.116:0/2056113045 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f519c0c3c30 0x7f519c06fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:47.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.116+0000 7f51a2787640 1 --2- 172.21.15.116:0/2056113045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f519c152f60 0x7f519c0c6080 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:47.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.116+0000 7f51a2787640 1 -- 172.21.15.116:0/2056113045 >> 172.21.15.116:0/2056113045 conn(0x7f519c0c4270 msgr2=0x7f519c0c4680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:47.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.116+0000 7f51a2787640 1 -- 172.21.15.116:0/2056113045 shutdown_connections 2024-04-25T00:14:47.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.116+0000 7f51a2787640 1 -- 172.21.15.116:0/2056113045 wait complete. 2024-04-25T00:14:47.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.120+0000 7f51a2787640 1 Processor -- start 2024-04-25T00:14:47.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.120+0000 7f51a2787640 1 -- start start 2024-04-25T00:14:47.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.120+0000 7f51a2787640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f519c0c3c30 0x7f519c14a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:47.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.120+0000 7f51a2787640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f519c152f60 0x7f519c0aab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:47.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.120+0000 7f51a2787640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f519c158ea0 0x7f519c1f0550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:47.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.120+0000 7f51a2787640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f519c0c1740 con 0x7f519c0c3c30 2024-04-25T00:14:47.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.120+0000 7f51a2787640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f519c0c15c0 con 0x7f519c152f60 2024-04-25T00:14:47.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.120+0000 7f51a2787640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f519c0c18c0 con 0x7f519c158ea0 2024-04-25T00:14:47.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.120+0000 7f519bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f519c0c3c30 0x7f519c14a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:47.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.120+0000 7f519bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f519c0c3c30 0x7f519c14a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38212/0 (socket says 172.21.15.116:38212) 2024-04-25T00:14:47.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.120+0000 7f519bfff640 1 -- 172.21.15.116:0/2798179617 learned_addr learned my addr 172.21.15.116:0/2798179617 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:14:47.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.120+0000 7f519b7fe640 1 --2- 172.21.15.116:0/2798179617 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f519c152f60 0x7f519c0aab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:47.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.120+0000 7f51a0d24640 1 --2- 172.21.15.116:0/2798179617 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f519c158ea0 0x7f519c1f0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:47.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.120+0000 7f519bfff640 1 -- 172.21.15.116:0/2798179617 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f519c158ea0 msgr2=0x7f519c1f0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:47.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.120+0000 7f519bfff640 1 --2- 172.21.15.116:0/2798179617 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f519c158ea0 0x7f519c1f0550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:47.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.120+0000 7f519bfff640 1 -- 172.21.15.116:0/2798179617 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f519c152f60 msgr2=0x7f519c0aab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:47.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.120+0000 7f519bfff640 1 --2- 172.21.15.116:0/2798179617 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f519c152f60 0x7f519c0aab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:47.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.120+0000 7f519bfff640 1 -- 172.21.15.116:0/2798179617 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f519c06cec0 con 0x7f519c0c3c30 2024-04-25T00:14:47.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.120+0000 7f519bfff640 1 --2- 172.21.15.116:0/2798179617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f519c0c3c30 0x7f519c14a800 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f518c01ab40 tx=0x7f518c01acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:47.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.120+0000 7f5198ff9640 1 -- 172.21.15.116:0/2798179617 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f518c0bc020 con 0x7f519c0c3c30 2024-04-25T00:14:47.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.120+0000 7f5198ff9640 1 -- 172.21.15.116:0/2798179617 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f518c0ade10 con 0x7f519c0c3c30 2024-04-25T00:14:47.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.120+0000 7f51a2787640 1 -- 172.21.15.116:0/2798179617 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f519c1f31f0 con 0x7f519c0c3c30 2024-04-25T00:14:47.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.120+0000 7f5198ff9640 1 -- 172.21.15.116:0/2798179617 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f518c0b2040 con 0x7f519c0c3c30 2024-04-25T00:14:47.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.120+0000 7f51a2787640 1 -- 172.21.15.116:0/2798179617 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f519c1f2c30 con 0x7f519c0c3c30 2024-04-25T00:14:47.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.120+0000 7f51a2787640 1 -- 172.21.15.116:0/2798179617 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f519c068a60 con 0x7f519c0c3c30 2024-04-25T00:14:47.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.120+0000 7f5198ff9640 1 -- 172.21.15.116:0/2798179617 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f518c0bf020 con 0x7f519c0c3c30 2024-04-25T00:14:47.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.124+0000 7f5198ff9640 1 --2- 172.21.15.116:0/2798179617 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f51880678c0 0x7f5188069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:47.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.124+0000 7f5198ff9640 1 -- 172.21.15.116:0/2798179617 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 8128+0+0 (secure 0 0 0) 0x7f518c1206d0 con 0x7f519c0c3c30 2024-04-25T00:14:47.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.124+0000 7f519b7fe640 1 --2- 172.21.15.116:0/2798179617 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f51880678c0 0x7f5188069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:47.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.124+0000 7f519b7fe640 1 --2- 172.21.15.116:0/2798179617 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f51880678c0 0x7f5188069d80 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f519c151f90 tx=0x7f5190008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:47.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.128+0000 7f5198ff9640 1 -- 172.21.15.116:0/2798179617 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f518c0c6030 con 0x7f519c0c3c30 2024-04-25T00:14:47.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.324+0000 7f757d6de640 1 -- 172.21.15.116:0/3976467107 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f757814db20 con 0x7f75780c4c20 2024-04-25T00:14:47.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.324+0000 7f755f7fe640 1 -- 172.21.15.116:0/3976467107 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) ==== 94+0+6496 (secure 0 0 0) 0x7f756c016130 con 0x7f75780c4c20 2024-04-25T00:14:47.324 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:14:47.324 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82},{"gid":5791,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3180024481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3180024481},{"type":"v1","addr":"172.21.15.174:6837","nonce":3180024481}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:12:48.771611+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":88,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6833/365773583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":365773583},{"type":"v1","addr":"172.21.15.116:6833","nonce":365773583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":1},{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:13:33.916554+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6582},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6582":{"gid":6582,"name":"a","rank":0,"incarnation":93,"state":"up:active","state_seq":14,"addr":"172.21.15.116:6835/462190523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":462190523},{"type":"v1","addr":"172.21.15.116:6835","nonce":462190523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6582,"qdb_cluster":[6582]},"id":2}]} 2024-04-25T00:14:47.324 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 97 2024-04-25T00:14:47.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.324+0000 7f757d6de640 1 -- 172.21.15.116:0/3976467107 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f75640678c0 msgr2=0x7f7564069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:47.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.324+0000 7f757d6de640 1 --2- 172.21.15.116:0/3976467107 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f75640678c0 0x7f7564069d80 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f7568001810 tx=0x7f7568009040 comp rx=0 tx=0).stop 2024-04-25T00:14:47.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.324+0000 7f757d6de640 1 -- 172.21.15.116:0/3976467107 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75780c4c20 msgr2=0x7f7578149ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:47.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.324+0000 7f757d6de640 1 --2- 172.21.15.116:0/3976467107 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75780c4c20 0x7f7578149ed0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f756c01a0e0 tx=0x7f756c019250 comp rx=0 tx=0).stop 2024-04-25T00:14:47.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.324+0000 7f757d6de640 1 -- 172.21.15.116:0/3976467107 shutdown_connections 2024-04-25T00:14:47.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.324+0000 7f757d6de640 1 --2- 172.21.15.116:0/3976467107 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f75640678c0 0x7f7564069d80 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:47.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.324+0000 7f757d6de640 1 --2- 172.21.15.116:0/3976467107 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7578150d40 0x7f75780adef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:47.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.324+0000 7f757d6de640 1 --2- 172.21.15.116:0/3976467107 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75780c4c20 0x7f7578149ed0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:47.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.324+0000 7f757d6de640 1 --2- 172.21.15.116:0/3976467107 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7578070770 0x7f7578155340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:47.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.324+0000 7f757d6de640 1 -- 172.21.15.116:0/3976467107 >> 172.21.15.116:0/3976467107 conn(0x7f757806f0c0 msgr2=0x7f75780a8c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:47.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.324+0000 7f757d6de640 1 -- 172.21.15.116:0/3976467107 shutdown_connections 2024-04-25T00:14:47.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.324+0000 7f757d6de640 1 -- 172.21.15.116:0/3976467107 wait complete. 2024-04-25T00:14:47.346 INFO:tasks.mds_thrash.fs.[a]:kill mds.c (rank=0) 2024-04-25T00:14:47.346 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-25T00:14:47.346 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:14:47.363 INFO:tasks.ceph.mds.c:Stopped 2024-04-25T00:14:47.363 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:14:47.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.376+0000 7f51a2787640 1 -- 172.21.15.116:0/2798179617 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f519c151ce0 con 0x7f519c0c3c30 2024-04-25T00:14:47.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.376+0000 7f5198ff9640 1 -- 172.21.15.116:0/2798179617 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) ==== 94+0+6496 (secure 0 0 0) 0x7f518c016110 con 0x7f519c0c3c30 2024-04-25T00:14:47.378 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:14:47.378 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82},{"gid":5791,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3180024481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3180024481},{"type":"v1","addr":"172.21.15.174:6837","nonce":3180024481}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:12:48.771611+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":88,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6833/365773583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":365773583},{"type":"v1","addr":"172.21.15.116:6833","nonce":365773583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":1},{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:13:33.916554+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6582},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6582":{"gid":6582,"name":"a","rank":0,"incarnation":93,"state":"up:active","state_seq":14,"addr":"172.21.15.116:6835/462190523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":462190523},{"type":"v1","addr":"172.21.15.116:6835","nonce":462190523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6582,"qdb_cluster":[6582]},"id":2}]} 2024-04-25T00:14:47.378 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 97 2024-04-25T00:14:47.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.380+0000 7f51a2787640 1 -- 172.21.15.116:0/2798179617 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f51880678c0 msgr2=0x7f5188069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:47.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.380+0000 7f51a2787640 1 --2- 172.21.15.116:0/2798179617 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f51880678c0 0x7f5188069d80 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f519c151f90 tx=0x7f5190008040 comp rx=0 tx=0).stop 2024-04-25T00:14:47.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.380+0000 7f51a2787640 1 -- 172.21.15.116:0/2798179617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f519c0c3c30 msgr2=0x7f519c14a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:47.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.380+0000 7f51a2787640 1 --2- 172.21.15.116:0/2798179617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f519c0c3c30 0x7f519c14a800 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f518c01ab40 tx=0x7f518c01acd0 comp rx=0 tx=0).stop 2024-04-25T00:14:47.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.380+0000 7f51a2787640 1 -- 172.21.15.116:0/2798179617 shutdown_connections 2024-04-25T00:14:47.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.380+0000 7f51a2787640 1 --2- 172.21.15.116:0/2798179617 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f51880678c0 0x7f5188069d80 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:47.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.380+0000 7f51a2787640 1 --2- 172.21.15.116:0/2798179617 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f519c158ea0 0x7f519c1f0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:47.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.380+0000 7f51a2787640 1 --2- 172.21.15.116:0/2798179617 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f519c152f60 0x7f519c0aab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:47.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.380+0000 7f51a2787640 1 --2- 172.21.15.116:0/2798179617 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f519c0c3c30 0x7f519c14a800 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:47.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.380+0000 7f51a2787640 1 -- 172.21.15.116:0/2798179617 >> 172.21.15.116:0/2798179617 conn(0x7f519c0c4270 msgr2=0x7f519c152200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:47.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.380+0000 7f51a2787640 1 -- 172.21.15.116:0/2798179617 shutdown_connections 2024-04-25T00:14:47.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.380+0000 7f51a2787640 1 -- 172.21.15.116:0/2798179617 wait complete. 2024-04-25T00:14:47.399 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:14:47.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.496+0000 7f53886ee640 1 Processor -- start 2024-04-25T00:14:47.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.496+0000 7f53886ee640 1 -- start start 2024-04-25T00:14:47.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.496+0000 7f53886ee640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5380152ca0 0x7f538006b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:47.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.496+0000 7f53886ee640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f538006b750 0x7f538006f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:47.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.496+0000 7f53886ee640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53801558e0 con 0x7f5380158d30 2024-04-25T00:14:47.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.496+0000 7f53886ee640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f5380155bc0 con 0x7f538006b750 2024-04-25T00:14:47.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.496+0000 7f53886ee640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f5380155ea0 con 0x7f5380152ca0 2024-04-25T00:14:47.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.496+0000 7f5386c8b640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f538006b750 0x7f538006f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:47.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.496+0000 7f538648a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5380152ca0 0x7f538006b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:47.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.496+0000 7f538648a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5380152ca0 0x7f538006b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:36646/0 (socket says 172.21.15.116:36646) 2024-04-25T00:14:47.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.496+0000 7f538648a640 1 -- 172.21.15.116:0/355086510 learned_addr learned my addr 172.21.15.116:0/355086510 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:14:47.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.500+0000 7f538648a640 1 -- 172.21.15.116:0/355086510 >> v2:172.21.15.174:3300/0 conn(0x7f538006b750 msgr2=0x7f538006f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:14:47.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.500+0000 7f538648a640 1 --2- 172.21.15.116:0/355086510 >> v2:172.21.15.174:3300/0 conn(0x7f538006b750 0x7f538006f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:47.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.500+0000 7f538648a640 1 -- 172.21.15.116:0/355086510 >> v1:172.21.15.174:6789/0 conn(0x7f5380158d30 legacy=0x7f538006f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:47.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.500+0000 7f538648a640 1 -- 172.21.15.116:0/355086510 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5380156180 con 0x7f5380152ca0 2024-04-25T00:14:47.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.500+0000 7f538648a640 1 --2- 172.21.15.116:0/355086510 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5380152ca0 0x7f538006b210 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f537c009870 tx=0x7f537c01c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be442d3962dc4f7a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:47.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.500+0000 7f5385488640 1 -- 172.21.15.116:0/355086510 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f537c0b1070 con 0x7f5380152ca0 2024-04-25T00:14:47.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.500+0000 7f5385488640 1 -- 172.21.15.116:0/355086510 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f537c0a8050 con 0x7f5380152ca0 2024-04-25T00:14:47.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.500+0000 7f5385488640 1 -- 172.21.15.116:0/355086510 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f537c0ac860 con 0x7f5380152ca0 2024-04-25T00:14:47.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.504+0000 7f53886ee640 1 -- 172.21.15.116:0/355086510 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5380152ca0 msgr2=0x7f538006b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:47.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.504+0000 7f53886ee640 1 --2- 172.21.15.116:0/355086510 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5380152ca0 0x7f538006b210 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f537c009870 tx=0x7f537c01c4d0 comp rx=0 tx=0).stop 2024-04-25T00:14:47.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.504+0000 7f53886ee640 1 -- 172.21.15.116:0/355086510 shutdown_connections 2024-04-25T00:14:47.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.504+0000 7f53886ee640 1 --2- 172.21.15.116:0/355086510 >> v2:172.21.15.174:3300/0 conn(0x7f538006b750 0x7f538006f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:47.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.504+0000 7f53886ee640 1 --2- 172.21.15.116:0/355086510 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5380152ca0 0x7f538006b210 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:47.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.504+0000 7f53886ee640 1 -- 172.21.15.116:0/355086510 >> 172.21.15.116:0/355086510 conn(0x7f53800c7fa0 msgr2=0x7f53800c83b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:47.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.504+0000 7f53886ee640 1 -- 172.21.15.116:0/355086510 shutdown_connections 2024-04-25T00:14:47.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.504+0000 7f53886ee640 1 -- 172.21.15.116:0/355086510 wait complete. 2024-04-25T00:14:47.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.504+0000 7f53886ee640 1 Processor -- start 2024-04-25T00:14:47.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.504+0000 7f53886ee640 1 -- start start 2024-04-25T00:14:47.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.504+0000 7f53886ee640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f538006b750 0x7f53801584f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:47.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.504+0000 7f53886ee640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5380152ca0 0x7f53800acf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:47.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.504+0000 7f53886ee640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5380158d30 0x7f5380153380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:47.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.504+0000 7f53886ee640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5380153200 con 0x7f5380152ca0 2024-04-25T00:14:47.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.504+0000 7f53886ee640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5380153080 con 0x7f5380158d30 2024-04-25T00:14:47.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.504+0000 7f53886ee640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f53801628e0 con 0x7f538006b750 2024-04-25T00:14:47.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.504+0000 7f5385c89640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5380152ca0 0x7f53800acf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:47.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.504+0000 7f5385c89640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5380152ca0 0x7f53800acf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38228/0 (socket says 172.21.15.116:38228) 2024-04-25T00:14:47.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.504+0000 7f5385c89640 1 -- 172.21.15.116:0/2089052129 learned_addr learned my addr 172.21.15.116:0/2089052129 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:14:47.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.504+0000 7f5386c8b640 1 --2- 172.21.15.116:0/2089052129 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5380158d30 0x7f5380153380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:47.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.504+0000 7f538648a640 1 --2- 172.21.15.116:0/2089052129 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f538006b750 0x7f53801584f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:47.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.504+0000 7f5385c89640 1 -- 172.21.15.116:0/2089052129 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f538006b750 msgr2=0x7f53801584f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:47.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.504+0000 7f5385c89640 1 --2- 172.21.15.116:0/2089052129 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f538006b750 0x7f53801584f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:47.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.504+0000 7f5385c89640 1 -- 172.21.15.116:0/2089052129 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5380158d30 msgr2=0x7f5380153380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:47.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.504+0000 7f5385c89640 1 --2- 172.21.15.116:0/2089052129 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5380158d30 0x7f5380153380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:47.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.504+0000 7f5385c89640 1 -- 172.21.15.116:0/2089052129 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5380156d40 con 0x7f5380152ca0 2024-04-25T00:14:47.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.504+0000 7f538648a640 1 --2- 172.21.15.116:0/2089052129 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f538006b750 0x7f53801584f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:14:47.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.504+0000 7f5386c8b640 1 --2- 172.21.15.116:0/2089052129 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5380158d30 0x7f5380153380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:14:47.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.504+0000 7f5385c89640 1 --2- 172.21.15.116:0/2089052129 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5380152ca0 0x7f53800acf20 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f537000c450 tx=0x7f537000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:47.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.504+0000 7f5376ffd640 1 -- 172.21.15.116:0/2089052129 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5370016020 con 0x7f5380152ca0 2024-04-25T00:14:47.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.504+0000 7f5376ffd640 1 -- 172.21.15.116:0/2089052129 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53700052d0 con 0x7f5380152ca0 2024-04-25T00:14:47.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.504+0000 7f53886ee640 1 -- 172.21.15.116:0/2089052129 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53801547b0 con 0x7f5380152ca0 2024-04-25T00:14:47.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.504+0000 7f5376ffd640 1 -- 172.21.15.116:0/2089052129 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5370007260 con 0x7f5380152ca0 2024-04-25T00:14:47.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.504+0000 7f53886ee640 1 -- 172.21.15.116:0/2089052129 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53800a9640 con 0x7f5380152ca0 2024-04-25T00:14:47.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.508+0000 7f53886ee640 1 -- 172.21.15.116:0/2089052129 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5380068a60 con 0x7f5380152ca0 2024-04-25T00:14:47.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.508+0000 7f5376ffd640 1 -- 172.21.15.116:0/2089052129 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5370007400 con 0x7f5380152ca0 2024-04-25T00:14:47.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.508+0000 7f5376ffd640 1 --2- 172.21.15.116:0/2089052129 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f53640678c0 0x7f5364069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:47.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.508+0000 7f5376ffd640 1 -- 172.21.15.116:0/2089052129 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 8128+0+0 (secure 0 0 0) 0x7f5370085fb0 con 0x7f5380152ca0 2024-04-25T00:14:47.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.508+0000 7f538648a640 1 --2- 172.21.15.116:0/2089052129 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f53640678c0 0x7f5364069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:47.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.508+0000 7f538648a640 1 --2- 172.21.15.116:0/2089052129 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f53640678c0 0x7f5364069d80 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f537c0ae000 tx=0x7f537c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:47.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.512+0000 7f5376ffd640 1 -- 172.21.15.116:0/2089052129 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f5370020020 con 0x7f5380152ca0 2024-04-25T00:14:47.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.724+0000 7f53886ee640 1 -- 172.21.15.116:0/2089052129 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f538015fea0 con 0x7f5380152ca0 2024-04-25T00:14:47.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.724+0000 7f5376ffd640 1 -- 172.21.15.116:0/2089052129 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) ==== 94+0+6496 (secure 0 0 0) 0x7f53801547b0 con 0x7f5380152ca0 2024-04-25T00:14:47.727 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:14:47.727 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82},{"gid":5791,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3180024481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3180024481},{"type":"v1","addr":"172.21.15.174:6837","nonce":3180024481}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:12:48.771611+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":88,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6833/365773583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":365773583},{"type":"v1","addr":"172.21.15.116:6833","nonce":365773583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":1},{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:13:33.916554+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6582},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6582":{"gid":6582,"name":"a","rank":0,"incarnation":93,"state":"up:active","state_seq":14,"addr":"172.21.15.116:6835/462190523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":462190523},{"type":"v1","addr":"172.21.15.116:6835","nonce":462190523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6582,"qdb_cluster":[6582]},"id":2}]} 2024-04-25T00:14:47.727 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 97 2024-04-25T00:14:47.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.728+0000 7f53886ee640 1 -- 172.21.15.116:0/2089052129 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f53640678c0 msgr2=0x7f5364069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:47.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.728+0000 7f53886ee640 1 --2- 172.21.15.116:0/2089052129 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f53640678c0 0x7f5364069d80 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f537c0ae000 tx=0x7f537c008040 comp rx=0 tx=0).stop 2024-04-25T00:14:47.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.728+0000 7f53886ee640 1 -- 172.21.15.116:0/2089052129 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5380152ca0 msgr2=0x7f53800acf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:47.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.728+0000 7f53886ee640 1 --2- 172.21.15.116:0/2089052129 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5380152ca0 0x7f53800acf20 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f537000c450 tx=0x7f537000c910 comp rx=0 tx=0).stop 2024-04-25T00:14:47.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.728+0000 7f53886ee640 1 -- 172.21.15.116:0/2089052129 shutdown_connections 2024-04-25T00:14:47.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.728+0000 7f53886ee640 1 --2- 172.21.15.116:0/2089052129 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f53640678c0 0x7f5364069d80 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:47.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.728+0000 7f53886ee640 1 --2- 172.21.15.116:0/2089052129 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5380158d30 0x7f5380153380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:47.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.728+0000 7f53886ee640 1 --2- 172.21.15.116:0/2089052129 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5380152ca0 0x7f53800acf20 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:47.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.728+0000 7f53886ee640 1 --2- 172.21.15.116:0/2089052129 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f538006b750 0x7f53801584f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:47.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.728+0000 7f53886ee640 1 -- 172.21.15.116:0/2089052129 >> 172.21.15.116:0/2089052129 conn(0x7f53800c7fa0 msgr2=0x7f53800c2d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:47.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.728+0000 7f53886ee640 1 -- 172.21.15.116:0/2089052129 shutdown_connections 2024-04-25T00:14:47.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:47.728+0000 7f53886ee640 1 -- 172.21.15.116:0/2089052129 wait complete. 2024-04-25T00:14:47.748 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:14:49.400 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:14:49.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.524+0000 7f3835b33640 1 Processor -- start 2024-04-25T00:14:49.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.528+0000 7f3835b33640 1 -- start start 2024-04-25T00:14:49.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.528+0000 7f3835b33640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f38300a94a0 0x7f38301538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:49.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.528+0000 7f3835b33640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38300c3910 con 0x7f3830150d20 2024-04-25T00:14:49.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.528+0000 7f3835b33640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3830154530 con 0x7f38300a99f0 2024-04-25T00:14:49.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.528+0000 7f3835b33640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f38300c16e0 con 0x7f38300a94a0 2024-04-25T00:14:49.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.528+0000 7f3826ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f38300a94a0 0x7f38301538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:49.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.528+0000 7f3826ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f38300a94a0 0x7f38301538e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38244/0 (socket says 172.21.15.116:38244) 2024-04-25T00:14:49.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.528+0000 7f3826ffd640 1 -- 172.21.15.116:0/379379537 learned_addr learned my addr 172.21.15.116:0/379379537 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:14:49.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.528+0000 7f3826ffd640 1 -- 172.21.15.116:0/379379537 >> v1:172.21.15.174:6789/0 conn(0x7f38300a99f0 legacy=0x7f3830153e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:14:49.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.528+0000 7f3826ffd640 1 -- 172.21.15.116:0/379379537 >> v1:172.21.15.116:6789/0 conn(0x7f3830150d20 legacy=0x7f38300c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:49.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.528+0000 7f3826ffd640 1 -- 172.21.15.116:0/379379537 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38300c1400 con 0x7f38300a94a0 2024-04-25T00:14:49.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.528+0000 7f3826ffd640 1 --2- 172.21.15.116:0/379379537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38300a94a0 0x7f38301538e0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f3818009870 tx=0x7f381801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7183f856d83c0ad5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:49.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.528+0000 7f382effd640 1 -- 172.21.15.116:0/379379537 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38180b0070 con 0x7f38300a94a0 2024-04-25T00:14:49.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.528+0000 7f382effd640 1 -- 172.21.15.116:0/379379537 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38180ab600 con 0x7f38300a94a0 2024-04-25T00:14:49.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.528+0000 7f382effd640 1 -- 172.21.15.116:0/379379537 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38180ab9d0 con 0x7f38300a94a0 2024-04-25T00:14:49.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.532+0000 7f3835b33640 1 -- 172.21.15.116:0/379379537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38300a94a0 msgr2=0x7f38301538e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:49.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.532+0000 7f3835b33640 1 --2- 172.21.15.116:0/379379537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38300a94a0 0x7f38301538e0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f3818009870 tx=0x7f381801c520 comp rx=0 tx=0).stop 2024-04-25T00:14:49.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.532+0000 7f3835b33640 1 -- 172.21.15.116:0/379379537 shutdown_connections 2024-04-25T00:14:49.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.532+0000 7f3835b33640 1 --2- 172.21.15.116:0/379379537 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38300a94a0 0x7f38301538e0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:49.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.532+0000 7f3835b33640 1 -- 172.21.15.116:0/379379537 >> 172.21.15.116:0/379379537 conn(0x7f38300c3b00 msgr2=0x7f38300c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:49.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.532+0000 7f3835b33640 1 -- 172.21.15.116:0/379379537 shutdown_connections 2024-04-25T00:14:49.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.532+0000 7f3835b33640 1 -- 172.21.15.116:0/379379537 wait complete. 2024-04-25T00:14:49.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.532+0000 7f3835b33640 1 Processor -- start 2024-04-25T00:14:49.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.532+0000 7f3835b33640 1 -- start start 2024-04-25T00:14:49.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.532+0000 7f3835b33640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f38300a94a0 0x7f383013f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:49.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.532+0000 7f3835b33640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f38300a99f0 0x7f383013f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:49.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.532+0000 7f3835b33640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3830150d20 0x7f38301efa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:49.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.532+0000 7f3835b33640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f38300c32a0 con 0x7f3830150d20 2024-04-25T00:14:49.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.532+0000 7f3835b33640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f38300c3120 con 0x7f38300a99f0 2024-04-25T00:14:49.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.532+0000 7f3835b33640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f3830161740 con 0x7f38300a94a0 2024-04-25T00:14:49.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.532+0000 7f382ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3830150d20 0x7f38301efa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:49.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.532+0000 7f382ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3830150d20 0x7f38301efa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38248/0 (socket says 172.21.15.116:38248) 2024-04-25T00:14:49.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.532+0000 7f382ffff640 1 -- 172.21.15.116:0/1622328553 learned_addr learned my addr 172.21.15.116:0/1622328553 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:14:49.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.532+0000 7f382f7fe640 1 --2- 172.21.15.116:0/1622328553 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f38300a94a0 0x7f383013f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:49.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.532+0000 7f3826ffd640 1 --2- 172.21.15.116:0/1622328553 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f38300a99f0 0x7f383013f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:49.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.532+0000 7f382ffff640 1 -- 172.21.15.116:0/1622328553 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f38300a94a0 msgr2=0x7f383013f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:49.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.532+0000 7f382ffff640 1 --2- 172.21.15.116:0/1622328553 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f38300a94a0 0x7f383013f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:49.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.532+0000 7f382ffff640 1 -- 172.21.15.116:0/1622328553 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f38300a99f0 msgr2=0x7f383013f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:49.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.532+0000 7f382ffff640 1 --2- 172.21.15.116:0/1622328553 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f38300a99f0 0x7f383013f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:49.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.532+0000 7f382ffff640 1 -- 172.21.15.116:0/1622328553 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38300ac720 con 0x7f3830150d20 2024-04-25T00:14:49.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.532+0000 7f382ffff640 1 --2- 172.21.15.116:0/1622328553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3830150d20 0x7f38301efa50 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f382800c7b0 tx=0x7f382800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:49.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.532+0000 7f382cff9640 1 -- 172.21.15.116:0/1622328553 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3828004b70 con 0x7f3830150d20 2024-04-25T00:14:49.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.532+0000 7f382cff9640 1 -- 172.21.15.116:0/1622328553 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3828004d10 con 0x7f3830150d20 2024-04-25T00:14:49.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.532+0000 7f3835b33640 1 -- 172.21.15.116:0/1622328553 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3830143bd0 con 0x7f3830150d20 2024-04-25T00:14:49.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.532+0000 7f382cff9640 1 -- 172.21.15.116:0/1622328553 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f382800aa90 con 0x7f3830150d20 2024-04-25T00:14:49.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.532+0000 7f3835b33640 1 -- 172.21.15.116:0/1622328553 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3830144190 con 0x7f3830150d20 2024-04-25T00:14:49.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.536+0000 7f3835b33640 1 -- 172.21.15.116:0/1622328553 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3830068350 con 0x7f3830150d20 2024-04-25T00:14:49.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.536+0000 7f382cff9640 1 -- 172.21.15.116:0/1622328553 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f382800ac30 con 0x7f3830150d20 2024-04-25T00:14:49.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.536+0000 7f382cff9640 1 --2- 172.21.15.116:0/1622328553 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f38100678c0 0x7f3810069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:49.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.536+0000 7f382cff9640 1 -- 172.21.15.116:0/1622328553 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 8128+0+0 (secure 0 0 0) 0x7f38280867f0 con 0x7f3830150d20 2024-04-25T00:14:49.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.536+0000 7f382f7fe640 1 --2- 172.21.15.116:0/1622328553 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f38100678c0 0x7f3810069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:49.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.536+0000 7f382f7fe640 1 --2- 172.21.15.116:0/1622328553 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f38100678c0 0x7f3810069d80 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f38200015b0 tx=0x7f3820009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:49.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.540+0000 7f382cff9640 1 -- 172.21.15.116:0/1622328553 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f3828020070 con 0x7f3830150d20 2024-04-25T00:14:49.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.732+0000 7f3835b33640 1 -- 172.21.15.116:0/1622328553 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f38301432a0 con 0x7f3830150d20 2024-04-25T00:14:49.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.732+0000 7f382cff9640 1 -- 172.21.15.116:0/1622328553 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) ==== 94+0+6496 (secure 0 0 0) 0x7f3828056a50 con 0x7f3830150d20 2024-04-25T00:14:49.734 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:14:49.734 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82},{"gid":5791,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3180024481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3180024481},{"type":"v1","addr":"172.21.15.174:6837","nonce":3180024481}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:12:48.771611+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":88,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6833/365773583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":365773583},{"type":"v1","addr":"172.21.15.116:6833","nonce":365773583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":1},{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:13:33.916554+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6582},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6582":{"gid":6582,"name":"a","rank":0,"incarnation":93,"state":"up:active","state_seq":14,"addr":"172.21.15.116:6835/462190523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":462190523},{"type":"v1","addr":"172.21.15.116:6835","nonce":462190523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6582,"qdb_cluster":[6582]},"id":2}]} 2024-04-25T00:14:49.735 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 97 2024-04-25T00:14:49.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.736+0000 7f3835b33640 1 -- 172.21.15.116:0/1622328553 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f38100678c0 msgr2=0x7f3810069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:49.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.736+0000 7f3835b33640 1 --2- 172.21.15.116:0/1622328553 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f38100678c0 0x7f3810069d80 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f38200015b0 tx=0x7f3820009040 comp rx=0 tx=0).stop 2024-04-25T00:14:49.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.736+0000 7f3835b33640 1 -- 172.21.15.116:0/1622328553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3830150d20 msgr2=0x7f38301efa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:49.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.736+0000 7f3835b33640 1 --2- 172.21.15.116:0/1622328553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3830150d20 0x7f38301efa50 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f382800c7b0 tx=0x7f382800cc70 comp rx=0 tx=0).stop 2024-04-25T00:14:49.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.736+0000 7f3835b33640 1 -- 172.21.15.116:0/1622328553 shutdown_connections 2024-04-25T00:14:49.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.736+0000 7f3835b33640 1 --2- 172.21.15.116:0/1622328553 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f38100678c0 0x7f3810069d80 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:49.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.736+0000 7f3835b33640 1 --2- 172.21.15.116:0/1622328553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3830150d20 0x7f38301efa50 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:49.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.736+0000 7f3835b33640 1 --2- 172.21.15.116:0/1622328553 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f38300a99f0 0x7f383013f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:49.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.736+0000 7f3835b33640 1 --2- 172.21.15.116:0/1622328553 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f38300a94a0 0x7f383013f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:49.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.736+0000 7f3835b33640 1 -- 172.21.15.116:0/1622328553 >> 172.21.15.116:0/1622328553 conn(0x7f38300c3b00 msgr2=0x7f38301538e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:49.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.736+0000 7f3835b33640 1 -- 172.21.15.116:0/1622328553 shutdown_connections 2024-04-25T00:14:49.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.736+0000 7f3835b33640 1 -- 172.21.15.116:0/1622328553 wait complete. 2024-04-25T00:14:49.748 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:14:49.753 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:14:49.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.868+0000 7fd368de0640 1 Processor -- start 2024-04-25T00:14:49.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.868+0000 7fd368de0640 1 -- start start 2024-04-25T00:14:49.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.872+0000 7fd368de0640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd364152d20 0x7fd3640c40b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:49.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.872+0000 7fd368de0640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd3640ab8c0 0x7fd364159240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:49.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.872+0000 7fd368de0640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fd3640c25d0 0x7fd364159780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:49.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.872+0000 7fd368de0640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fd364159cc0 con 0x7fd3640ab8c0 2024-04-25T00:14:49.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.872+0000 7fd368de0640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fd3640c2ad0 con 0x7fd3640c25d0 2024-04-25T00:14:49.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.872+0000 7fd368de0640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fd3640c35b0 con 0x7fd364152d20 2024-04-25T00:14:49.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.872+0000 7fd361d9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd3640ab8c0 0x7fd364159240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:49.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.872+0000 7fd361d9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd3640ab8c0 0x7fd364159240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38260/0 (socket says 172.21.15.116:38260) 2024-04-25T00:14:49.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.872+0000 7fd361d9b640 1 -- 172.21.15.116:0/156160860 learned_addr learned my addr 172.21.15.116:0/156160860 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:14:49.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.872+0000 7fd362d9d640 1 --2- 172.21.15.116:0/156160860 >> v2:172.21.15.174:3300/0 conn(0x7fd3640c25d0 0x7fd364159780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:49.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.872+0000 7fd36259c640 1 --2- 172.21.15.116:0/156160860 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd364152d20 0x7fd3640c40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:49.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.872+0000 7fd361d9b640 1 -- 172.21.15.116:0/156160860 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd364152d20 msgr2=0x7fd3640c40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:14:49.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.872+0000 7fd361d9b640 1 --2- 172.21.15.116:0/156160860 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd364152d20 0x7fd3640c40b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:49.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.872+0000 7fd361d9b640 1 -- 172.21.15.116:0/156160860 >> v2:172.21.15.174:3300/0 conn(0x7fd3640c25d0 msgr2=0x7fd364159780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:14:49.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.872+0000 7fd361d9b640 1 --2- 172.21.15.116:0/156160860 >> v2:172.21.15.174:3300/0 conn(0x7fd3640c25d0 0x7fd364159780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:49.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.872+0000 7fd361d9b640 1 -- 172.21.15.116:0/156160860 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3640c3890 con 0x7fd3640ab8c0 2024-04-25T00:14:49.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.872+0000 7fd361d9b640 1 --2- 172.21.15.116:0/156160860 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3640ab8c0 0x7fd364159240 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fd3640c1f10 tx=0x7fd34c01c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d9f484292069300 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:49.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.872+0000 7fd36159a640 1 -- 172.21.15.116:0/156160860 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd34c0ab620 con 0x7fd3640ab8c0 2024-04-25T00:14:49.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.872+0000 7fd36159a640 1 -- 172.21.15.116:0/156160860 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd34c0b0070 con 0x7fd3640ab8c0 2024-04-25T00:14:49.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.872+0000 7fd36159a640 1 -- 172.21.15.116:0/156160860 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd34c0ab9e0 con 0x7fd3640ab8c0 2024-04-25T00:14:49.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.876+0000 7fd368de0640 1 -- 172.21.15.116:0/156160860 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3640ab8c0 msgr2=0x7fd364159240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:49.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.876+0000 7fd368de0640 1 --2- 172.21.15.116:0/156160860 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3640ab8c0 0x7fd364159240 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fd3640c1f10 tx=0x7fd34c01c460 comp rx=0 tx=0).stop 2024-04-25T00:14:49.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.876+0000 7fd368de0640 1 -- 172.21.15.116:0/156160860 shutdown_connections 2024-04-25T00:14:49.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.876+0000 7fd368de0640 1 --2- 172.21.15.116:0/156160860 >> v2:172.21.15.174:3300/0 conn(0x7fd3640c25d0 0x7fd364159780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:49.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.876+0000 7fd368de0640 1 --2- 172.21.15.116:0/156160860 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3640ab8c0 0x7fd364159240 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:49.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.876+0000 7fd368de0640 1 --2- 172.21.15.116:0/156160860 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd364152d20 0x7fd3640c40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:49.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.876+0000 7fd368de0640 1 -- 172.21.15.116:0/156160860 >> 172.21.15.116:0/156160860 conn(0x7fd36406b3e0 msgr2=0x7fd36406b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:49.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.876+0000 7fd368de0640 1 -- 172.21.15.116:0/156160860 shutdown_connections 2024-04-25T00:14:49.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.876+0000 7fd368de0640 1 -- 172.21.15.116:0/156160860 wait complete. 2024-04-25T00:14:49.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.876+0000 7fd368de0640 1 Processor -- start 2024-04-25T00:14:49.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.876+0000 7fd368de0640 1 -- start start 2024-04-25T00:14:49.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.876+0000 7fd368de0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3640c25d0 0x7fd364146dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:49.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.876+0000 7fd368de0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd364152d20 0x7fd364144c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:49.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.876+0000 7fd368de0640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3640c6dc0 0x7fd3641451d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:49.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.876+0000 7fd368de0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd364153100 con 0x7fd3640c25d0 2024-04-25T00:14:49.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.876+0000 7fd368de0640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd364155fc0 con 0x7fd364152d20 2024-04-25T00:14:49.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.876+0000 7fd368de0640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fd36415f2e0 con 0x7fd3640c6dc0 2024-04-25T00:14:49.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.876+0000 7fd36259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3640c25d0 0x7fd364146dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:49.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.876+0000 7fd36259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3640c25d0 0x7fd364146dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38268/0 (socket says 172.21.15.116:38268) 2024-04-25T00:14:49.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.876+0000 7fd36259c640 1 -- 172.21.15.116:0/3535694707 learned_addr learned my addr 172.21.15.116:0/3535694707 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:14:49.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.876+0000 7fd361d9b640 1 --2- 172.21.15.116:0/3535694707 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd364152d20 0x7fd364144c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:49.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.876+0000 7fd362d9d640 1 --2- 172.21.15.116:0/3535694707 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3640c6dc0 0x7fd3641451d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:49.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.876+0000 7fd36259c640 1 -- 172.21.15.116:0/3535694707 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3640c6dc0 msgr2=0x7fd3641451d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:49.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.876+0000 7fd36259c640 1 --2- 172.21.15.116:0/3535694707 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3640c6dc0 0x7fd3641451d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:49.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.876+0000 7fd36259c640 1 -- 172.21.15.116:0/3535694707 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd364152d20 msgr2=0x7fd364144c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:49.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.876+0000 7fd36259c640 1 --2- 172.21.15.116:0/3535694707 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd364152d20 0x7fd364144c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:49.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.876+0000 7fd36259c640 1 -- 172.21.15.116:0/3535694707 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd36415fa60 con 0x7fd3640c25d0 2024-04-25T00:14:49.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.876+0000 7fd36259c640 1 --2- 172.21.15.116:0/3535694707 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3640c25d0 0x7fd364146dc0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fd35400d5f0 tx=0x7fd35400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:49.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.876+0000 7fd35affd640 1 -- 172.21.15.116:0/3535694707 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3540125f0 con 0x7fd3640c25d0 2024-04-25T00:14:49.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.876+0000 7fd35affd640 1 -- 172.21.15.116:0/3535694707 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd354019070 con 0x7fd3640c25d0 2024-04-25T00:14:49.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.876+0000 7fd368de0640 1 -- 172.21.15.116:0/3535694707 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd36415f530 con 0x7fd3640c25d0 2024-04-25T00:14:49.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.876+0000 7fd35affd640 1 -- 172.21.15.116:0/3535694707 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3540129b0 con 0x7fd3640c25d0 2024-04-25T00:14:49.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.876+0000 7fd368de0640 1 -- 172.21.15.116:0/3535694707 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd364147300 con 0x7fd3640c25d0 2024-04-25T00:14:49.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.880+0000 7fd368de0640 1 -- 172.21.15.116:0/3535694707 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd364068350 con 0x7fd3640c25d0 2024-04-25T00:14:49.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.880+0000 7fd35affd640 1 -- 172.21.15.116:0/3535694707 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd354012b50 con 0x7fd3640c25d0 2024-04-25T00:14:49.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.880+0000 7fd35affd640 1 --2- 172.21.15.116:0/3535694707 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd3440678f0 0x7fd344069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:49.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.880+0000 7fd35affd640 1 -- 172.21.15.116:0/3535694707 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 8128+0+0 (secure 0 0 0) 0x7fd354086860 con 0x7fd3640c25d0 2024-04-25T00:14:49.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.880+0000 7fd361d9b640 1 --2- 172.21.15.116:0/3535694707 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd3440678f0 0x7fd344069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:49.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.880+0000 7fd361d9b640 1 --2- 172.21.15.116:0/3535694707 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd3440678f0 0x7fd344069db0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fd34c0af000 tx=0x7fd34c004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:49.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:49.884+0000 7fd35affd640 1 -- 172.21.15.116:0/3535694707 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fd354011340 con 0x7fd3640c25d0 2024-04-25T00:14:50.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:50.080+0000 7fd368de0640 1 -- 172.21.15.116:0/3535694707 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd3640c23b0 con 0x7fd3640c25d0 2024-04-25T00:14:50.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:50.080+0000 7fd35affd640 1 -- 172.21.15.116:0/3535694707 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) ==== 94+0+6496 (secure 0 0 0) 0x7fd354056ac0 con 0x7fd3640c25d0 2024-04-25T00:14:50.083 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:14:50.083 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82},{"gid":5791,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3180024481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3180024481},{"type":"v1","addr":"172.21.15.174:6837","nonce":3180024481}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:12:48.771611+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":88,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6833/365773583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":365773583},{"type":"v1","addr":"172.21.15.116:6833","nonce":365773583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":1},{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:13:33.916554+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6582},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6582":{"gid":6582,"name":"a","rank":0,"incarnation":93,"state":"up:active","state_seq":14,"addr":"172.21.15.116:6835/462190523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":462190523},{"type":"v1","addr":"172.21.15.116:6835","nonce":462190523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6582,"qdb_cluster":[6582]},"id":2}]} 2024-04-25T00:14:50.083 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 97 2024-04-25T00:14:50.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:50.084+0000 7fd368de0640 1 -- 172.21.15.116:0/3535694707 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd3440678f0 msgr2=0x7fd344069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:50.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:50.084+0000 7fd368de0640 1 --2- 172.21.15.116:0/3535694707 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd3440678f0 0x7fd344069db0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fd34c0af000 tx=0x7fd34c004740 comp rx=0 tx=0).stop 2024-04-25T00:14:50.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:50.084+0000 7fd368de0640 1 -- 172.21.15.116:0/3535694707 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3640c25d0 msgr2=0x7fd364146dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:50.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:50.084+0000 7fd368de0640 1 --2- 172.21.15.116:0/3535694707 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3640c25d0 0x7fd364146dc0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fd35400d5f0 tx=0x7fd35400dab0 comp rx=0 tx=0).stop 2024-04-25T00:14:50.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:50.084+0000 7fd368de0640 1 -- 172.21.15.116:0/3535694707 shutdown_connections 2024-04-25T00:14:50.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:50.084+0000 7fd368de0640 1 --2- 172.21.15.116:0/3535694707 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd3440678f0 0x7fd344069db0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:50.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:50.084+0000 7fd368de0640 1 --2- 172.21.15.116:0/3535694707 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3640c6dc0 0x7fd3641451d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:50.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:50.084+0000 7fd368de0640 1 --2- 172.21.15.116:0/3535694707 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd364152d20 0x7fd364144c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:50.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:50.084+0000 7fd368de0640 1 --2- 172.21.15.116:0/3535694707 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3640c25d0 0x7fd364146dc0 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:50.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:50.084+0000 7fd368de0640 1 -- 172.21.15.116:0/3535694707 >> 172.21.15.116:0/3535694707 conn(0x7fd36406b3e0 msgr2=0x7fd3640c12c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:50.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:50.084+0000 7fd368de0640 1 -- 172.21.15.116:0/3535694707 shutdown_connections 2024-04-25T00:14:50.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:50.084+0000 7fd368de0640 1 -- 172.21.15.116:0/3535694707 wait complete. 2024-04-25T00:14:50.102 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:14:51.754 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:14:51.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.880+0000 7f8b7a1af640 1 Processor -- start 2024-04-25T00:14:51.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.880+0000 7f8b7a1af640 1 -- start start 2024-04-25T00:14:51.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.884+0000 7f8b7a1af640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8b7413e310 0x7f8b740c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:51.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.884+0000 7f8b7a1af640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f8b7413ed60 0x7f8b741560d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:51.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.884+0000 7f8b7a1af640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b7406d200 con 0x7f8b740c2f50 2024-04-25T00:14:51.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.884+0000 7f8b7a1af640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f8b74070d40 con 0x7f8b7413ed60 2024-04-25T00:14:51.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.884+0000 7f8b7a1af640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f8b74071020 con 0x7f8b7413e310 2024-04-25T00:14:51.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.884+0000 7f8b737fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8b7413e310 0x7f8b740c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:51.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.884+0000 7f8b73fff640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f8b7413ed60 0x7f8b741560d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:51.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.884+0000 7f8b737fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8b7413e310 0x7f8b740c2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:36690/0 (socket says 172.21.15.116:36690) 2024-04-25T00:14:51.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.884+0000 7f8b72ffd640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f8b740c2f50 0x7f8b741559c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:50978/0 (socket says 172.21.15.116:50978) 2024-04-25T00:14:51.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.884+0000 7f8b737fe640 1 -- 172.21.15.116:0/973091191 learned_addr learned my addr 172.21.15.116:0/973091191 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:14:51.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.884+0000 7f8b737fe640 1 -- 172.21.15.116:0/973091191 >> v2:172.21.15.174:3300/0 conn(0x7f8b7413ed60 msgr2=0x7f8b741560d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:14:51.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.884+0000 7f8b737fe640 1 --2- 172.21.15.116:0/973091191 >> v2:172.21.15.174:3300/0 conn(0x7f8b7413ed60 0x7f8b741560d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:51.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.884+0000 7f8b737fe640 1 -- 172.21.15.116:0/973091191 >> v1:172.21.15.174:6789/0 conn(0x7f8b740c2f50 legacy=0x7f8b741559c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:51.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.884+0000 7f8b737fe640 1 -- 172.21.15.116:0/973091191 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b74156610 con 0x7f8b7413e310 2024-04-25T00:14:51.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.884+0000 7f8b73fff640 1 --2- 172.21.15.116:0/973091191 >> v2:172.21.15.174:3300/0 conn(0x7f8b7413ed60 0x7f8b741560d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:14:51.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.888+0000 7f8b737fe640 1 --2- 172.21.15.116:0/973091191 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8b7413e310 0x7f8b740c2a10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8b68009870 tx=0x7f8b6801c440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2912e6498b4edb2d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:51.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.888+0000 7f8b727fc640 1 -- 172.21.15.116:0/973091191 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b680b0070 con 0x7f8b7413e310 2024-04-25T00:14:51.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.888+0000 7f8b727fc640 1 -- 172.21.15.116:0/973091191 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b68004230 con 0x7f8b7413e310 2024-04-25T00:14:51.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.888+0000 7f8b727fc640 1 -- 172.21.15.116:0/973091191 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b68003620 con 0x7f8b7413e310 2024-04-25T00:14:51.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.888+0000 7f8b7a1af640 1 -- 172.21.15.116:0/973091191 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8b7413e310 msgr2=0x7f8b740c2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:51.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.888+0000 7f8b7a1af640 1 --2- 172.21.15.116:0/973091191 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8b7413e310 0x7f8b740c2a10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8b68009870 tx=0x7f8b6801c440 comp rx=0 tx=0).stop 2024-04-25T00:14:51.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.888+0000 7f8b7a1af640 1 -- 172.21.15.116:0/973091191 shutdown_connections 2024-04-25T00:14:51.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.888+0000 7f8b7a1af640 1 --2- 172.21.15.116:0/973091191 >> v2:172.21.15.174:3300/0 conn(0x7f8b7413ed60 0x7f8b741560d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:51.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.888+0000 7f8b7a1af640 1 --2- 172.21.15.116:0/973091191 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8b7413e310 0x7f8b740c2a10 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:51.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.888+0000 7f8b7a1af640 1 -- 172.21.15.116:0/973091191 >> 172.21.15.116:0/973091191 conn(0x7f8b740c3b00 msgr2=0x7f8b740c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:51.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.888+0000 7f8b7a1af640 1 -- 172.21.15.116:0/973091191 shutdown_connections 2024-04-25T00:14:51.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.888+0000 7f8b7a1af640 1 -- 172.21.15.116:0/973091191 wait complete. 2024-04-25T00:14:51.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.888+0000 7f8b7a1af640 1 Processor -- start 2024-04-25T00:14:51.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.888+0000 7f8b7a1af640 1 -- start start 2024-04-25T00:14:51.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.888+0000 7f8b7a1af640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8b740c2f50 0x7f8b740a9be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:51.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.888+0000 7f8b7a1af640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b7413e310 0x7f8b74136c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:51.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.888+0000 7f8b7a1af640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8b7413ed60 0x7f8b74134360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:51.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.888+0000 7f8b7a1af640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8b74162b70 con 0x7f8b7413e310 2024-04-25T00:14:51.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.888+0000 7f8b7a1af640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8b741629f0 con 0x7f8b740c2f50 2024-04-25T00:14:51.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.888+0000 7f8b7a1af640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f8b74162cf0 con 0x7f8b7413ed60 2024-04-25T00:14:51.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.888+0000 7f8b72ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b7413e310 0x7f8b74136c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:51.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.888+0000 7f8b72ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b7413e310 0x7f8b74136c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38274/0 (socket says 172.21.15.116:38274) 2024-04-25T00:14:51.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.888+0000 7f8b72ffd640 1 -- 172.21.15.116:0/2983913440 learned_addr learned my addr 172.21.15.116:0/2983913440 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:14:51.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.888+0000 7f8b73fff640 1 --2- 172.21.15.116:0/2983913440 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8b7413ed60 0x7f8b74134360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:51.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.888+0000 7f8b737fe640 1 --2- 172.21.15.116:0/2983913440 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8b740c2f50 0x7f8b740a9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:51.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.888+0000 7f8b72ffd640 1 -- 172.21.15.116:0/2983913440 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8b7413ed60 msgr2=0x7f8b74134360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:51.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.888+0000 7f8b72ffd640 1 --2- 172.21.15.116:0/2983913440 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8b7413ed60 0x7f8b74134360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:51.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.888+0000 7f8b72ffd640 1 -- 172.21.15.116:0/2983913440 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8b740c2f50 msgr2=0x7f8b740a9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:51.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.888+0000 7f8b72ffd640 1 --2- 172.21.15.116:0/2983913440 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8b740c2f50 0x7f8b740a9be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:51.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.888+0000 7f8b72ffd640 1 -- 172.21.15.116:0/2983913440 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b740aa2e0 con 0x7f8b7413e310 2024-04-25T00:14:51.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.888+0000 7f8b72ffd640 1 --2- 172.21.15.116:0/2983913440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b7413e310 0x7f8b74136c70 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f8b6400c370 tx=0x7f8b6400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:51.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.888+0000 7f8b5bfff640 1 -- 172.21.15.116:0/2983913440 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b640074e0 con 0x7f8b7413e310 2024-04-25T00:14:51.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.892+0000 7f8b7a1af640 1 -- 172.21.15.116:0/2983913440 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b74133880 con 0x7f8b7413e310 2024-04-25T00:14:51.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.892+0000 7f8b5bfff640 1 -- 172.21.15.116:0/2983913440 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b64007680 con 0x7f8b7413e310 2024-04-25T00:14:51.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.892+0000 7f8b5bfff640 1 -- 172.21.15.116:0/2983913440 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b64004a30 con 0x7f8b7413e310 2024-04-25T00:14:51.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.892+0000 7f8b7a1af640 1 -- 172.21.15.116:0/2983913440 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b74137910 con 0x7f8b7413e310 2024-04-25T00:14:51.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.892+0000 7f8b7a1af640 1 -- 172.21.15.116:0/2983913440 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b74068350 con 0x7f8b7413e310 2024-04-25T00:14:51.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.892+0000 7f8b5bfff640 1 -- 172.21.15.116:0/2983913440 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8b6401a020 con 0x7f8b7413e310 2024-04-25T00:14:51.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.892+0000 7f8b5bfff640 1 --2- 172.21.15.116:0/2983913440 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8b600678c0 0x7f8b60069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:51.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.892+0000 7f8b5bfff640 1 -- 172.21.15.116:0/2983913440 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 8128+0+0 (secure 0 0 0) 0x7f8b64085e20 con 0x7f8b7413e310 2024-04-25T00:14:51.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.892+0000 7f8b737fe640 1 --2- 172.21.15.116:0/2983913440 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8b600678c0 0x7f8b60069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:51.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.892+0000 7f8b737fe640 1 --2- 172.21.15.116:0/2983913440 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8b600678c0 0x7f8b60069d80 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f8b680ad000 tx=0x7f8b68009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:51.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:51.896+0000 7f8b5bfff640 1 -- 172.21.15.116:0/2983913440 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f8b64055f80 con 0x7f8b7413e310 2024-04-25T00:14:52.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.084+0000 7f8b7a1af640 1 -- 172.21.15.116:0/2983913440 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8b740c34f0 con 0x7f8b7413e310 2024-04-25T00:14:52.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.084+0000 7f8b5bfff640 1 -- 172.21.15.116:0/2983913440 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) ==== 94+0+6496 (secure 0 0 0) 0x7f8b64055880 con 0x7f8b7413e310 2024-04-25T00:14:52.084 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:14:52.085 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82},{"gid":5791,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3180024481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3180024481},{"type":"v1","addr":"172.21.15.174:6837","nonce":3180024481}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:12:48.771611+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":88,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6833/365773583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":365773583},{"type":"v1","addr":"172.21.15.116:6833","nonce":365773583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":1},{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:13:33.916554+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6582},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6582":{"gid":6582,"name":"a","rank":0,"incarnation":93,"state":"up:active","state_seq":14,"addr":"172.21.15.116:6835/462190523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":462190523},{"type":"v1","addr":"172.21.15.116:6835","nonce":462190523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6582,"qdb_cluster":[6582]},"id":2}]} 2024-04-25T00:14:52.085 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 97 2024-04-25T00:14:52.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.084+0000 7f8b7a1af640 1 -- 172.21.15.116:0/2983913440 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8b600678c0 msgr2=0x7f8b60069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:52.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.084+0000 7f8b7a1af640 1 --2- 172.21.15.116:0/2983913440 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8b600678c0 0x7f8b60069d80 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f8b680ad000 tx=0x7f8b68009040 comp rx=0 tx=0).stop 2024-04-25T00:14:52.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.084+0000 7f8b7a1af640 1 -- 172.21.15.116:0/2983913440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b7413e310 msgr2=0x7f8b74136c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:52.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.084+0000 7f8b7a1af640 1 --2- 172.21.15.116:0/2983913440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b7413e310 0x7f8b74136c70 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f8b6400c370 tx=0x7f8b6400c830 comp rx=0 tx=0).stop 2024-04-25T00:14:52.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.084+0000 7f8b7a1af640 1 -- 172.21.15.116:0/2983913440 shutdown_connections 2024-04-25T00:14:52.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.084+0000 7f8b7a1af640 1 --2- 172.21.15.116:0/2983913440 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8b600678c0 0x7f8b60069d80 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:52.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.084+0000 7f8b7a1af640 1 --2- 172.21.15.116:0/2983913440 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8b7413ed60 0x7f8b74134360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:52.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.084+0000 7f8b7a1af640 1 --2- 172.21.15.116:0/2983913440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b7413e310 0x7f8b74136c70 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:52.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.084+0000 7f8b7a1af640 1 --2- 172.21.15.116:0/2983913440 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8b740c2f50 0x7f8b740a9be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:52.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.084+0000 7f8b7a1af640 1 -- 172.21.15.116:0/2983913440 >> 172.21.15.116:0/2983913440 conn(0x7f8b740c3b00 msgr2=0x7f8b740ace70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:52.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.084+0000 7f8b7a1af640 1 -- 172.21.15.116:0/2983913440 shutdown_connections 2024-04-25T00:14:52.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.084+0000 7f8b7a1af640 1 -- 172.21.15.116:0/2983913440 wait complete. 2024-04-25T00:14:52.101 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:14:52.102 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:14:52.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.224+0000 7f651f138640 1 Processor -- start 2024-04-25T00:14:52.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.224+0000 7f651f138640 1 -- start start 2024-04-25T00:14:52.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.224+0000 7f651f138640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f65180c40e0 0x7f65180c1540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:52.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.224+0000 7f651f138640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f651806bd50 con 0x7f65180730a0 2024-04-25T00:14:52.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.224+0000 7f651f138640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f651806f990 con 0x7f651806fcd0 2024-04-25T00:14:52.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.224+0000 7f651f138640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f6518156780 con 0x7f65180c40e0 2024-04-25T00:14:52.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.224+0000 7f651ced4640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f65180c40e0 0x7f65180c1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:52.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.224+0000 7f651ced4640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f65180c40e0 0x7f65180c1540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38284/0 (socket says 172.21.15.116:38284) 2024-04-25T00:14:52.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.224+0000 7f6517fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f65180730a0 0x7f6518156070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54986/0 (socket says 172.21.15.116:54986) 2024-04-25T00:14:52.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.224+0000 7f651ced4640 1 -- 172.21.15.116:0/1955206608 learned_addr learned my addr 172.21.15.116:0/1955206608 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:14:52.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.224+0000 7f651ced4640 1 -- 172.21.15.116:0/1955206608 >> v1:172.21.15.174:6789/0 conn(0x7f651806fcd0 legacy=0x7f65181525a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:52.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.224+0000 7f651ced4640 1 -- 172.21.15.116:0/1955206608 >> v1:172.21.15.116:6789/0 conn(0x7f65180730a0 legacy=0x7f6518156070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:52.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.224+0000 7f651ced4640 1 -- 172.21.15.116:0/1955206608 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6518155b50 con 0x7f65180c40e0 2024-04-25T00:14:52.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.228+0000 7f651ced4640 1 --2- 172.21.15.116:0/1955206608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65180c40e0 0x7f65180c1540 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f6508009870 tx=0x7f650801c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a86bb6684d7cac02 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:52.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.228+0000 7f65177fe640 1 -- 172.21.15.116:0/1955206608 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65080a5700 con 0x7f65180c40e0 2024-04-25T00:14:52.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.228+0000 7f65177fe640 1 -- 172.21.15.116:0/1955206608 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65080a58a0 con 0x7f65180c40e0 2024-04-25T00:14:52.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.228+0000 7f65177fe640 1 -- 172.21.15.116:0/1955206608 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65080a5c70 con 0x7f65180c40e0 2024-04-25T00:14:52.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.228+0000 7f651f138640 1 -- 172.21.15.116:0/1955206608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65180c40e0 msgr2=0x7f65180c1540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:52.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.228+0000 7f651f138640 1 --2- 172.21.15.116:0/1955206608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65180c40e0 0x7f65180c1540 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f6508009870 tx=0x7f650801c040 comp rx=0 tx=0).stop 2024-04-25T00:14:52.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.228+0000 7f651f138640 1 -- 172.21.15.116:0/1955206608 shutdown_connections 2024-04-25T00:14:52.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.228+0000 7f651f138640 1 --2- 172.21.15.116:0/1955206608 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65180c40e0 0x7f65180c1540 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:52.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.228+0000 7f651f138640 1 -- 172.21.15.116:0/1955206608 >> 172.21.15.116:0/1955206608 conn(0x7f65180c4c10 msgr2=0x7f65180c5020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:52.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.228+0000 7f651f138640 1 -- 172.21.15.116:0/1955206608 shutdown_connections 2024-04-25T00:14:52.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.228+0000 7f651f138640 1 -- 172.21.15.116:0/1955206608 wait complete. 2024-04-25T00:14:52.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.232+0000 7f651f138640 1 Processor -- start 2024-04-25T00:14:52.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.232+0000 7f651f138640 1 -- start start 2024-04-25T00:14:52.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.232+0000 7f651f138640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f651806fcd0 0x7f65180aa790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:52.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.232+0000 7f651f138640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65180730a0 0x7f6518143640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:52.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.232+0000 7f651f138640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f65180c40e0 0x7f65181f42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:52.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.232+0000 7f651f138640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6518164e90 con 0x7f651806fcd0 2024-04-25T00:14:52.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.232+0000 7f651f138640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6518164d10 con 0x7f65180730a0 2024-04-25T00:14:52.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.232+0000 7f651f138640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6518165010 con 0x7f65180c40e0 2024-04-25T00:14:52.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.232+0000 7f651ced4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f651806fcd0 0x7f65180aa790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:52.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.232+0000 7f651ced4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f651806fcd0 0x7f65180aa790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38294/0 (socket says 172.21.15.116:38294) 2024-04-25T00:14:52.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.232+0000 7f651ced4640 1 -- 172.21.15.116:0/1040428754 learned_addr learned my addr 172.21.15.116:0/1040428754 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:14:52.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.232+0000 7f651ced4640 1 -- 172.21.15.116:0/1040428754 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f65180c40e0 msgr2=0x7f65181f42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:52.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.232+0000 7f651ced4640 1 --2- 172.21.15.116:0/1040428754 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f65180c40e0 0x7f65181f42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:52.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.232+0000 7f651ced4640 1 -- 172.21.15.116:0/1040428754 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65180730a0 msgr2=0x7f6518143640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:52.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.232+0000 7f651ced4640 1 --2- 172.21.15.116:0/1040428754 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65180730a0 0x7f6518143640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:52.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.232+0000 7f651ced4640 1 -- 172.21.15.116:0/1040428754 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65181f3ae0 con 0x7f651806fcd0 2024-04-25T00:14:52.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.232+0000 7f651ced4640 1 --2- 172.21.15.116:0/1040428754 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f651806fcd0 0x7f65180aa790 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f65080af040 tx=0x7f650801a880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:52.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.232+0000 7f65157fa640 1 -- 172.21.15.116:0/1040428754 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65080bc020 con 0x7f651806fcd0 2024-04-25T00:14:52.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.232+0000 7f65157fa640 1 -- 172.21.15.116:0/1040428754 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65080ad9f0 con 0x7f651806fcd0 2024-04-25T00:14:52.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.232+0000 7f651f138640 1 -- 172.21.15.116:0/1040428754 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65180c7870 con 0x7f651806fcd0 2024-04-25T00:14:52.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.232+0000 7f65157fa640 1 -- 172.21.15.116:0/1040428754 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65080adde0 con 0x7f651806fcd0 2024-04-25T00:14:52.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.232+0000 7f651f138640 1 -- 172.21.15.116:0/1040428754 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65181f3da0 con 0x7f651806fcd0 2024-04-25T00:14:52.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.232+0000 7f651f138640 1 -- 172.21.15.116:0/1040428754 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6518068350 con 0x7f651806fcd0 2024-04-25T00:14:52.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.232+0000 7f65157fa640 1 -- 172.21.15.116:0/1040428754 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f65080bf020 con 0x7f651806fcd0 2024-04-25T00:14:52.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.232+0000 7f65157fa640 1 --2- 172.21.15.116:0/1040428754 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f65040678c0 0x7f6504069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:52.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.236+0000 7f65157fa640 1 -- 172.21.15.116:0/1040428754 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 8128+0+0 (secure 0 0 0) 0x7f65081204f0 con 0x7f651806fcd0 2024-04-25T00:14:52.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.236+0000 7f6517fff640 1 --2- 172.21.15.116:0/1040428754 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f65040678c0 0x7f6504069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:52.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.236+0000 7f6517fff640 1 --2- 172.21.15.116:0/1040428754 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f65040678c0 0x7f6504069d80 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f65180c2890 tx=0x7f650c007990 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:52.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.240+0000 7f65157fa640 1 -- 172.21.15.116:0/1040428754 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f6508016070 con 0x7f651806fcd0 2024-04-25T00:14:52.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.432+0000 7f651f138640 1 -- 172.21.15.116:0/1040428754 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f65180759d0 con 0x7f651806fcd0 2024-04-25T00:14:52.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.432+0000 7f65157fa640 1 -- 172.21.15.116:0/1040428754 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) ==== 94+0+6496 (secure 0 0 0) 0x7f65080f0870 con 0x7f651806fcd0 2024-04-25T00:14:52.434 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:14:52.434 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82},{"gid":5791,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3180024481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3180024481},{"type":"v1","addr":"172.21.15.174:6837","nonce":3180024481}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:12:48.771611+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":88,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6833/365773583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":365773583},{"type":"v1","addr":"172.21.15.116:6833","nonce":365773583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":1},{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:13:33.916554+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6582},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6582":{"gid":6582,"name":"a","rank":0,"incarnation":93,"state":"up:active","state_seq":14,"addr":"172.21.15.116:6835/462190523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":462190523},{"type":"v1","addr":"172.21.15.116:6835","nonce":462190523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6582,"qdb_cluster":[6582]},"id":2}]} 2024-04-25T00:14:52.434 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 97 2024-04-25T00:14:52.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.436+0000 7f651f138640 1 -- 172.21.15.116:0/1040428754 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f65040678c0 msgr2=0x7f6504069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:52.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.436+0000 7f651f138640 1 --2- 172.21.15.116:0/1040428754 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f65040678c0 0x7f6504069d80 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f65180c2890 tx=0x7f650c007990 comp rx=0 tx=0).stop 2024-04-25T00:14:52.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.436+0000 7f651f138640 1 -- 172.21.15.116:0/1040428754 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f651806fcd0 msgr2=0x7f65180aa790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:52.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.436+0000 7f651f138640 1 --2- 172.21.15.116:0/1040428754 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f651806fcd0 0x7f65180aa790 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f65080af040 tx=0x7f650801a880 comp rx=0 tx=0).stop 2024-04-25T00:14:52.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.436+0000 7f651f138640 1 -- 172.21.15.116:0/1040428754 shutdown_connections 2024-04-25T00:14:52.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.436+0000 7f651f138640 1 --2- 172.21.15.116:0/1040428754 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f65040678c0 0x7f6504069d80 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:52.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.436+0000 7f651f138640 1 --2- 172.21.15.116:0/1040428754 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f65180c40e0 0x7f65181f42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:52.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.436+0000 7f651f138640 1 --2- 172.21.15.116:0/1040428754 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65180730a0 0x7f6518143640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:52.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.436+0000 7f651f138640 1 --2- 172.21.15.116:0/1040428754 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f651806fcd0 0x7f65180aa790 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:52.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.436+0000 7f651f138640 1 -- 172.21.15.116:0/1040428754 >> 172.21.15.116:0/1040428754 conn(0x7f65180c4c10 msgr2=0x7f651806b210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:52.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.436+0000 7f651f138640 1 -- 172.21.15.116:0/1040428754 shutdown_connections 2024-04-25T00:14:52.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:52.436+0000 7f651f138640 1 -- 172.21.15.116:0/1040428754 wait complete. 2024-04-25T00:14:52.452 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:14:54.101 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:14:54.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.228+0000 7fc983cc3640 1 Processor -- start 2024-04-25T00:14:54.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.228+0000 7fc983cc3640 1 -- start start 2024-04-25T00:14:54.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.228+0000 7fc983cc3640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc97c06e890 0x7fc97c148890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:54.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.228+0000 7fc983cc3640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc97c152c10 con 0x7fc97c0c3540 2024-04-25T00:14:54.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.228+0000 7fc983cc3640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc97c151e50 con 0x7fc97c070dd0 2024-04-25T00:14:54.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.228+0000 7fc983cc3640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fc97c0c3120 con 0x7fc97c06e890 2024-04-25T00:14:54.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.228+0000 7fc982260640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fc97c0c3540 0x7fc97c072590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54988/0 (socket says 172.21.15.116:54988) 2024-04-25T00:14:54.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.228+0000 7fc982260640 1 -- 172.21.15.116:0/4178676632 learned_addr learned my addr 172.21.15.116:0/4178676632 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:14:54.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.228+0000 7fc98125e640 1 --2- 172.21.15.116:0/4178676632 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc97c06e890 0x7fc97c148890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:54.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.228+0000 7fc980a5d640 1 -- 172.21.15.116:0/4178676632 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 713979057 0 0) 0x7fc97c152c10 con 0x7fc97c0c3540 2024-04-25T00:14:54.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.228+0000 7fc980a5d640 1 -- 172.21.15.116:0/4178676632 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc968003530 con 0x7fc97c0c3540 2024-04-25T00:14:54.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.228+0000 7fc980a5d640 1 -- 172.21.15.116:0/4178676632 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3452677365 0 0) 0x7fc968003530 con 0x7fc97c0c3540 2024-04-25T00:14:54.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.228+0000 7fc980a5d640 1 -- 172.21.15.116:0/4178676632 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc97c06e890 msgr2=0x7fc97c148890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:14:54.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.228+0000 7fc980a5d640 1 --2- 172.21.15.116:0/4178676632 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc97c06e890 0x7fc97c148890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:54.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.228+0000 7fc980a5d640 1 -- 172.21.15.116:0/4178676632 >> v1:172.21.15.174:6789/0 conn(0x7fc97c070dd0 legacy=0x7fc97c0c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:54.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.228+0000 7fc980a5d640 1 -- 172.21.15.116:0/4178676632 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc97c152130 con 0x7fc97c0c3540 2024-04-25T00:14:54.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.228+0000 7fc980a5d640 1 -- 172.21.15.116:0/4178676632 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7fc978001a00 con 0x7fc97c0c3540 2024-04-25T00:14:54.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.228+0000 7fc980a5d640 1 -- 172.21.15.116:0/4178676632 >> v1:172.21.15.116:6789/0 conn(0x7fc97c0c3540 legacy=0x7fc97c072590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:54.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.228+0000 7fc980a5d640 1 --2- 172.21.15.116:0/4178676632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9680039f0 0x7fc968003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:54.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.228+0000 7fc980a5d640 1 --2- 172.21.15.116:0/4178676632 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc97c06e890 0x7fc968004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:54.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.228+0000 7fc980a5d640 1 --2- 172.21.15.116:0/4178676632 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc968004d50 0x7fc9680051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:54.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.228+0000 7fc980a5d640 1 -- 172.21.15.116:0/4178676632 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc97c0c3120 con 0x7fc9680039f0 2024-04-25T00:14:54.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.228+0000 7fc980a5d640 1 -- 172.21.15.116:0/4178676632 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc96800bab0 con 0x7fc968004d50 2024-04-25T00:14:54.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.228+0000 7fc980a5d640 1 -- 172.21.15.116:0/4178676632 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fc96800bcb0 con 0x7fc97c06e890 2024-04-25T00:14:54.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.228+0000 7fc981a5f640 1 --2- 172.21.15.116:0/4178676632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9680039f0 0x7fc968003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:54.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.228+0000 7fc98125e640 1 --2- 172.21.15.116:0/4178676632 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc97c06e890 0x7fc968004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:54.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.228+0000 7fc982260640 1 --2- 172.21.15.116:0/4178676632 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc968004d50 0x7fc9680051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:54.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.228+0000 7fc981a5f640 1 -- 172.21.15.116:0/4178676632 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc97c06e890 msgr2=0x7fc968004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:14:54.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.228+0000 7fc981a5f640 1 --2- 172.21.15.116:0/4178676632 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc97c06e890 0x7fc968004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:54.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.228+0000 7fc981a5f640 1 -- 172.21.15.116:0/4178676632 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc968004d50 msgr2=0x7fc9680051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:14:54.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.228+0000 7fc981a5f640 1 --2- 172.21.15.116:0/4178676632 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc968004d50 0x7fc9680051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:54.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.228+0000 7fc981a5f640 1 -- 172.21.15.116:0/4178676632 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc968003530 con 0x7fc9680039f0 2024-04-25T00:14:54.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.232+0000 7fc981a5f640 1 --2- 172.21.15.116:0/4178676632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9680039f0 0x7fc968003e10 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fc974001750 tx=0x7fc97401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37e2a8b623dae4e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:54.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.232+0000 7fc980a5d640 1 -- 172.21.15.116:0/4178676632 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9740013a0 con 0x7fc9680039f0 2024-04-25T00:14:54.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.232+0000 7fc980a5d640 1 -- 172.21.15.116:0/4178676632 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9740052f0 con 0x7fc9680039f0 2024-04-25T00:14:54.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.232+0000 7fc980a5d640 1 -- 172.21.15.116:0/4178676632 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc974003840 con 0x7fc9680039f0 2024-04-25T00:14:54.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.232+0000 7fc983cc3640 1 -- 172.21.15.116:0/4178676632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9680039f0 msgr2=0x7fc968003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:54.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.232+0000 7fc983cc3640 1 --2- 172.21.15.116:0/4178676632 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9680039f0 0x7fc968003e10 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fc974001750 tx=0x7fc97401c3e0 comp rx=0 tx=0).stop 2024-04-25T00:14:54.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.232+0000 7fc982260640 1 -- 172.21.15.116:0/4178676632 reap_dead start 2024-04-25T00:14:54.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.232+0000 7fc983cc3640 1 -- 172.21.15.116:0/4178676632 shutdown_connections 2024-04-25T00:14:54.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.232+0000 7fc983cc3640 1 -- 172.21.15.116:0/4178676632 >> 172.21.15.116:0/4178676632 conn(0x7fc97c0c3b00 msgr2=0x7fc97c0c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:54.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.232+0000 7fc983cc3640 1 -- 172.21.15.116:0/4178676632 shutdown_connections 2024-04-25T00:14:54.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.232+0000 7fc983cc3640 1 -- 172.21.15.116:0/4178676632 wait complete. 2024-04-25T00:14:54.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.232+0000 7fc983cc3640 1 Processor -- start 2024-04-25T00:14:54.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.232+0000 7fc983cc3640 1 -- start start 2024-04-25T00:14:54.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.232+0000 7fc983cc3640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc9680039f0 0x7fc97c13fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:54.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.232+0000 7fc983cc3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc97c161810 0x7fc97c134d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:54.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.232+0000 7fc983cc3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc97c135ab0 0x7fc97c1352a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:54.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.232+0000 7fc983cc3640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc97c072e20 con 0x7fc97c161810 2024-04-25T00:14:54.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.232+0000 7fc983cc3640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc97c072ca0 con 0x7fc97c135ab0 2024-04-25T00:14:54.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.232+0000 7fc983cc3640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fc97c071a80 con 0x7fc9680039f0 2024-04-25T00:14:54.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.236+0000 7fc98125e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc97c161810 0x7fc97c134d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:54.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.236+0000 7fc98125e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc97c161810 0x7fc97c134d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38302/0 (socket says 172.21.15.116:38302) 2024-04-25T00:14:54.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.236+0000 7fc98125e640 1 -- 172.21.15.116:0/1822773405 learned_addr learned my addr 172.21.15.116:0/1822773405 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:14:54.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.236+0000 7fc982260640 1 --2- 172.21.15.116:0/1822773405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc97c135ab0 0x7fc97c1352a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:54.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.236+0000 7fc981a5f640 1 --2- 172.21.15.116:0/1822773405 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc9680039f0 0x7fc97c13fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:54.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.236+0000 7fc98125e640 1 -- 172.21.15.116:0/1822773405 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc9680039f0 msgr2=0x7fc97c13fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:54.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.236+0000 7fc98125e640 1 --2- 172.21.15.116:0/1822773405 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc9680039f0 0x7fc97c13fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:54.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.236+0000 7fc98125e640 1 -- 172.21.15.116:0/1822773405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc97c135ab0 msgr2=0x7fc97c1352a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:54.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.236+0000 7fc98125e640 1 --2- 172.21.15.116:0/1822773405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc97c135ab0 0x7fc97c1352a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:54.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.236+0000 7fc98125e640 1 -- 172.21.15.116:0/1822773405 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc97c13f470 con 0x7fc97c161810 2024-04-25T00:14:54.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.236+0000 7fc98125e640 1 --2- 172.21.15.116:0/1822773405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc97c161810 0x7fc97c134d60 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fc96c011970 tx=0x7fc96c011e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:54.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.236+0000 7fc9727fc640 1 -- 172.21.15.116:0/1822773405 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc96c019070 con 0x7fc97c161810 2024-04-25T00:14:54.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.236+0000 7fc9727fc640 1 -- 172.21.15.116:0/1822773405 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc96c004240 con 0x7fc97c161810 2024-04-25T00:14:54.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.236+0000 7fc983cc3640 1 -- 172.21.15.116:0/1822773405 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc97c13eeb0 con 0x7fc97c161810 2024-04-25T00:14:54.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.236+0000 7fc9727fc640 1 -- 172.21.15.116:0/1822773405 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc96c00f880 con 0x7fc97c161810 2024-04-25T00:14:54.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.236+0000 7fc983cc3640 1 -- 172.21.15.116:0/1822773405 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc97c0aa590 con 0x7fc97c161810 2024-04-25T00:14:54.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.236+0000 7fc983cc3640 1 -- 172.21.15.116:0/1822773405 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc97c068350 con 0x7fc97c161810 2024-04-25T00:14:54.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.236+0000 7fc9727fc640 1 -- 172.21.15.116:0/1822773405 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc96c01c070 con 0x7fc97c161810 2024-04-25T00:14:54.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.236+0000 7fc9727fc640 1 --2- 172.21.15.116:0/1822773405 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc9600678f0 0x7fc960069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:54.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.236+0000 7fc9727fc640 1 -- 172.21.15.116:0/1822773405 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 8128+0+0 (secure 0 0 0) 0x7fc96c086920 con 0x7fc97c161810 2024-04-25T00:14:54.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.236+0000 7fc981a5f640 1 --2- 172.21.15.116:0/1822773405 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc9600678f0 0x7fc960069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:54.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.236+0000 7fc981a5f640 1 --2- 172.21.15.116:0/1822773405 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc9600678f0 0x7fc960069db0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fc9680057f0 tx=0x7fc9740011a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:54.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.240+0000 7fc9727fc640 1 -- 172.21.15.116:0/1822773405 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fc96c056b00 con 0x7fc97c161810 2024-04-25T00:14:54.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.436+0000 7fc983cc3640 1 -- 172.21.15.116:0/1822773405 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc97c06e670 con 0x7fc97c161810 2024-04-25T00:14:54.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.436+0000 7fc9727fc640 1 -- 172.21.15.116:0/1822773405 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) ==== 94+0+6496 (secure 0 0 0) 0x7fc96c056400 con 0x7fc97c161810 2024-04-25T00:14:54.438 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:14:54.438 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82},{"gid":5791,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3180024481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3180024481},{"type":"v1","addr":"172.21.15.174:6837","nonce":3180024481}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:12:48.771611+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":88,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6833/365773583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":365773583},{"type":"v1","addr":"172.21.15.116:6833","nonce":365773583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":1},{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:13:33.916554+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6582},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6582":{"gid":6582,"name":"a","rank":0,"incarnation":93,"state":"up:active","state_seq":14,"addr":"172.21.15.116:6835/462190523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":462190523},{"type":"v1","addr":"172.21.15.116:6835","nonce":462190523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6582,"qdb_cluster":[6582]},"id":2}]} 2024-04-25T00:14:54.438 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 97 2024-04-25T00:14:54.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.440+0000 7fc983cc3640 1 -- 172.21.15.116:0/1822773405 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc9600678f0 msgr2=0x7fc960069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:54.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.440+0000 7fc983cc3640 1 --2- 172.21.15.116:0/1822773405 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc9600678f0 0x7fc960069db0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fc9680057f0 tx=0x7fc9740011a0 comp rx=0 tx=0).stop 2024-04-25T00:14:54.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.440+0000 7fc983cc3640 1 -- 172.21.15.116:0/1822773405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc97c161810 msgr2=0x7fc97c134d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:54.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.440+0000 7fc983cc3640 1 --2- 172.21.15.116:0/1822773405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc97c161810 0x7fc97c134d60 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fc96c011970 tx=0x7fc96c011e30 comp rx=0 tx=0).stop 2024-04-25T00:14:54.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.440+0000 7fc983cc3640 1 -- 172.21.15.116:0/1822773405 shutdown_connections 2024-04-25T00:14:54.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.440+0000 7fc983cc3640 1 --2- 172.21.15.116:0/1822773405 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc9600678f0 0x7fc960069db0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:54.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.440+0000 7fc983cc3640 1 --2- 172.21.15.116:0/1822773405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc97c135ab0 0x7fc97c1352a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:54.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.440+0000 7fc983cc3640 1 --2- 172.21.15.116:0/1822773405 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc97c161810 0x7fc97c134d60 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:54.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.440+0000 7fc983cc3640 1 --2- 172.21.15.116:0/1822773405 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc9680039f0 0x7fc97c13fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:54.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.440+0000 7fc983cc3640 1 -- 172.21.15.116:0/1822773405 >> 172.21.15.116:0/1822773405 conn(0x7fc97c0c3b00 msgr2=0x7fc97c072590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:54.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.440+0000 7fc983cc3640 1 -- 172.21.15.116:0/1822773405 shutdown_connections 2024-04-25T00:14:54.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.440+0000 7fc983cc3640 1 -- 172.21.15.116:0/1822773405 wait complete. 2024-04-25T00:14:54.453 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:14:54.457 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:14:54.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.564+0000 7f96ad06b640 1 Processor -- start 2024-04-25T00:14:54.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.564+0000 7f96ad06b640 1 -- start start 2024-04-25T00:14:54.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.568+0000 7f96ad06b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f96a8070dd0 0x7f96a8071270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:54.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.568+0000 7f96ad06b640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96a8152c10 con 0x7f96a8148890 2024-04-25T00:14:54.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.568+0000 7f96ad06b640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96a8151e50 con 0x7f96a806e890 2024-04-25T00:14:54.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.568+0000 7f96ad06b640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f96a8152130 con 0x7f96a8070dd0 2024-04-25T00:14:54.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.568+0000 7f969e59c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f96a8070dd0 0x7f96a8071270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:54.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.568+0000 7f96a6d9d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f96a8148890 0x7f96a8072590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54992/0 (socket says 172.21.15.116:54992) 2024-04-25T00:14:54.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.568+0000 7f96a6d9d640 1 -- 172.21.15.116:0/3820365339 learned_addr learned my addr 172.21.15.116:0/3820365339 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:14:54.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.568+0000 7f969e59c640 1 -- 172.21.15.116:0/3820365339 >> v1:172.21.15.174:6789/0 conn(0x7f96a806e890 legacy=0x7f96a80717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:14:54.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.568+0000 7f969e59c640 1 -- 172.21.15.116:0/3820365339 >> v1:172.21.15.116:6789/0 conn(0x7f96a8148890 legacy=0x7f96a8072590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:54.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.568+0000 7f969e59c640 1 -- 172.21.15.116:0/3820365339 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96a8152410 con 0x7f96a8070dd0 2024-04-25T00:14:54.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.568+0000 7f96a659c640 1 -- 172.21.15.116:0/3820365339 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3808532634 0 0) 0x7f96a8152c10 con 0x7f96a8148890 2024-04-25T00:14:54.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.568+0000 7f969e59c640 1 --2- 172.21.15.116:0/3820365339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96a8070dd0 0x7f96a8071270 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f969000ba80 tx=0x7f969001e8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98c717348a6efaee server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:54.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.568+0000 7f96a659c640 1 -- 172.21.15.116:0/3820365339 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96900ad070 con 0x7f96a8070dd0 2024-04-25T00:14:54.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.568+0000 7f96a659c640 1 -- 172.21.15.116:0/3820365339 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96900a7740 con 0x7f96a8070dd0 2024-04-25T00:14:54.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.568+0000 7f96a659c640 1 -- 172.21.15.116:0/3820365339 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96900a7b10 con 0x7f96a8070dd0 2024-04-25T00:14:54.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.568+0000 7f96ad06b640 1 -- 172.21.15.116:0/3820365339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96a8070dd0 msgr2=0x7f96a8071270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:54.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.568+0000 7f96ad06b640 1 --2- 172.21.15.116:0/3820365339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96a8070dd0 0x7f96a8071270 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f969000ba80 tx=0x7f969001e8f0 comp rx=0 tx=0).stop 2024-04-25T00:14:54.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.572+0000 7f96ad06b640 1 -- 172.21.15.116:0/3820365339 shutdown_connections 2024-04-25T00:14:54.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.572+0000 7f96ad06b640 1 --2- 172.21.15.116:0/3820365339 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96a8070dd0 0x7f96a8071270 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:54.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.572+0000 7f96ad06b640 1 -- 172.21.15.116:0/3820365339 >> 172.21.15.116:0/3820365339 conn(0x7f96a80c3b00 msgr2=0x7f96a80c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:54.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.572+0000 7f96ad06b640 1 -- 172.21.15.116:0/3820365339 shutdown_connections 2024-04-25T00:14:54.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.572+0000 7f96ad06b640 1 -- 172.21.15.116:0/3820365339 wait complete. 2024-04-25T00:14:54.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.572+0000 7f96ad06b640 1 Processor -- start 2024-04-25T00:14:54.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.572+0000 7f96ad06b640 1 -- start start 2024-04-25T00:14:54.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.572+0000 7f96ad06b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96a806e890 0x7f96a80c5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:54.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.572+0000 7f96ad06b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f96a8070dd0 0x7f96a81ebc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:54.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.572+0000 7f96ad06b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96a8148890 0x7f96a81ec1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:54.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.572+0000 7f96ad06b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f96a80ab710 con 0x7f96a806e890 2024-04-25T00:14:54.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.572+0000 7f96ad06b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f96a80ab590 con 0x7f96a8148890 2024-04-25T00:14:54.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.572+0000 7f96ad06b640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f96a80ab890 con 0x7f96a8070dd0 2024-04-25T00:14:54.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.572+0000 7f96a6d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96a806e890 0x7f96a80c5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:54.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.572+0000 7f96a6d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96a806e890 0x7f96a80c5ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38312/0 (socket says 172.21.15.116:38312) 2024-04-25T00:14:54.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.572+0000 7f96a6d9d640 1 -- 172.21.15.116:0/1740682542 learned_addr learned my addr 172.21.15.116:0/1740682542 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:14:54.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.572+0000 7f969e59c640 1 --2- 172.21.15.116:0/1740682542 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f96a8070dd0 0x7f96a81ebc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:54.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.572+0000 7f96a759e640 1 --2- 172.21.15.116:0/1740682542 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96a8148890 0x7f96a81ec1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:54.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.572+0000 7f96a6d9d640 1 -- 172.21.15.116:0/1740682542 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f96a8070dd0 msgr2=0x7f96a81ebc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:54.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.572+0000 7f96a6d9d640 1 --2- 172.21.15.116:0/1740682542 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f96a8070dd0 0x7f96a81ebc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:54.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.572+0000 7f96a6d9d640 1 -- 172.21.15.116:0/1740682542 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96a8148890 msgr2=0x7f96a81ec1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:54.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.572+0000 7f96a6d9d640 1 --2- 172.21.15.116:0/1740682542 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96a8148890 0x7f96a81ec1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:54.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.572+0000 7f96a6d9d640 1 -- 172.21.15.116:0/1740682542 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96a813f800 con 0x7f96a806e890 2024-04-25T00:14:54.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.572+0000 7f96a6d9d640 1 --2- 172.21.15.116:0/1740682542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96a806e890 0x7f96a80c5ea0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f969800c4b0 tx=0x7f969800c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:54.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.572+0000 7f969ffff640 1 -- 172.21.15.116:0/1740682542 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9698018070 con 0x7f96a806e890 2024-04-25T00:14:54.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.572+0000 7f969ffff640 1 -- 172.21.15.116:0/1740682542 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9698007500 con 0x7f96a806e890 2024-04-25T00:14:54.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.572+0000 7f96ad06b640 1 -- 172.21.15.116:0/1740682542 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96a806d690 con 0x7f96a806e890 2024-04-25T00:14:54.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.572+0000 7f969ffff640 1 -- 172.21.15.116:0/1740682542 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9698004a30 con 0x7f96a806e890 2024-04-25T00:14:54.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.572+0000 7f96ad06b640 1 -- 172.21.15.116:0/1740682542 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96a813f240 con 0x7f96a806e890 2024-04-25T00:14:54.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.572+0000 7f96ad06b640 1 -- 172.21.15.116:0/1740682542 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96a8068350 con 0x7f96a806e890 2024-04-25T00:14:54.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.572+0000 7f969ffff640 1 -- 172.21.15.116:0/1740682542 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f969801b070 con 0x7f96a806e890 2024-04-25T00:14:54.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.576+0000 7f969ffff640 1 --2- 172.21.15.116:0/1740682542 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f96880678c0 0x7f9688069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:54.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.576+0000 7f969ffff640 1 -- 172.21.15.116:0/1740682542 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 8128+0+0 (secure 0 0 0) 0x7f96980857d0 con 0x7f96a806e890 2024-04-25T00:14:54.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.576+0000 7f969e59c640 1 --2- 172.21.15.116:0/1740682542 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f96880678c0 0x7f9688069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:54.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.576+0000 7f969e59c640 1 --2- 172.21.15.116:0/1740682542 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f96880678c0 0x7f9688069d80 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f96900b2000 tx=0x7f96900a9070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:54.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.580+0000 7f969ffff640 1 -- 172.21.15.116:0/1740682542 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f9698055a30 con 0x7f96a806e890 2024-04-25T00:14:54.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.768+0000 7f96ad06b640 1 -- 172.21.15.116:0/1740682542 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f96a806e670 con 0x7f96a806e890 2024-04-25T00:14:54.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.768+0000 7f969ffff640 1 -- 172.21.15.116:0/1740682542 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) ==== 94+0+6496 (secure 0 0 0) 0x7f9698055330 con 0x7f96a806e890 2024-04-25T00:14:54.769 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:14:54.769 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82},{"gid":5791,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3180024481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3180024481},{"type":"v1","addr":"172.21.15.174:6837","nonce":3180024481}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":97}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:12:48.771611+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":88,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6833/365773583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":365773583},{"type":"v1","addr":"172.21.15.116:6833","nonce":365773583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":1},{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:13:33.916554+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":45,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6582},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6582":{"gid":6582,"name":"a","rank":0,"incarnation":93,"state":"up:active","state_seq":14,"addr":"172.21.15.116:6835/462190523","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":462190523},{"type":"v1","addr":"172.21.15.116:6835","nonce":462190523}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6582,"qdb_cluster":[6582]},"id":2}]} 2024-04-25T00:14:54.770 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 97 2024-04-25T00:14:54.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.772+0000 7f96ad06b640 1 -- 172.21.15.116:0/1740682542 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f96880678c0 msgr2=0x7f9688069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:54.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.772+0000 7f96ad06b640 1 --2- 172.21.15.116:0/1740682542 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f96880678c0 0x7f9688069d80 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f96900b2000 tx=0x7f96900a9070 comp rx=0 tx=0).stop 2024-04-25T00:14:54.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.772+0000 7f96ad06b640 1 -- 172.21.15.116:0/1740682542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96a806e890 msgr2=0x7f96a80c5ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:54.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.772+0000 7f96ad06b640 1 --2- 172.21.15.116:0/1740682542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96a806e890 0x7f96a80c5ea0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f969800c4b0 tx=0x7f969800c970 comp rx=0 tx=0).stop 2024-04-25T00:14:54.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.772+0000 7f96ad06b640 1 -- 172.21.15.116:0/1740682542 shutdown_connections 2024-04-25T00:14:54.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.772+0000 7f96ad06b640 1 --2- 172.21.15.116:0/1740682542 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f96880678c0 0x7f9688069d80 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:54.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.772+0000 7f96ad06b640 1 --2- 172.21.15.116:0/1740682542 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f96a8148890 0x7f96a81ec1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:54.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.772+0000 7f96ad06b640 1 --2- 172.21.15.116:0/1740682542 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f96a8070dd0 0x7f96a81ebc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:54.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.772+0000 7f96ad06b640 1 --2- 172.21.15.116:0/1740682542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f96a806e890 0x7f96a80c5ea0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:54.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.772+0000 7f96ad06b640 1 -- 172.21.15.116:0/1740682542 >> 172.21.15.116:0/1740682542 conn(0x7f96a80c3b00 msgr2=0x7f96a814f680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:54.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.772+0000 7f96ad06b640 1 -- 172.21.15.116:0/1740682542 shutdown_connections 2024-04-25T00:14:54.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:54.772+0000 7f96ad06b640 1 -- 172.21.15.116:0/1740682542 wait complete. 2024-04-25T00:14:54.786 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:14:56.458 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:14:56.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.588+0000 7fa45c40f640 1 Processor -- start 2024-04-25T00:14:56.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.588+0000 7fa45c40f640 1 -- start start 2024-04-25T00:14:56.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.588+0000 7fa45c40f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa454070dd0 0x7fa454148890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:56.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.588+0000 7fa45c40f640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fa45406e890 0x7fa4540c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:56.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.588+0000 7fa45c40f640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa454152c10 con 0x7fa4540c3540 2024-04-25T00:14:56.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.588+0000 7fa45c40f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fa454151e50 con 0x7fa454070dd0 2024-04-25T00:14:56.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.588+0000 7fa45c40f640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fa454152130 con 0x7fa45406e890 2024-04-25T00:14:56.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.588+0000 7fa45a1ab640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa454070dd0 0x7fa454148890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:56.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.588+0000 7fa45a1ab640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa454070dd0 0x7fa454148890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39998/0 (socket says 172.21.15.116:39998) 2024-04-25T00:14:56.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.588+0000 7fa45a1ab640 1 -- 172.21.15.116:0/489457906 learned_addr learned my addr 172.21.15.116:0/489457906 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:14:56.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.588+0000 7fa45a1ab640 1 -- 172.21.15.116:0/489457906 >> v2:172.21.15.174:3300/0 conn(0x7fa45406e890 msgr2=0x7fa4540c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:14:56.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.588+0000 7fa45a1ab640 1 --2- 172.21.15.116:0/489457906 >> v2:172.21.15.174:3300/0 conn(0x7fa45406e890 0x7fa4540c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:56.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.588+0000 7fa45a1ab640 1 -- 172.21.15.116:0/489457906 >> v1:172.21.15.174:6789/0 conn(0x7fa4540c3540 legacy=0x7fa454072590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:14:56.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.588+0000 7fa45a1ab640 1 -- 172.21.15.116:0/489457906 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa454152410 con 0x7fa454070dd0 2024-04-25T00:14:56.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.588+0000 7fa45a1ab640 1 --2- 172.21.15.116:0/489457906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa454070dd0 0x7fa454148890 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fa45000bb10 tx=0x7fa45001ebb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe0acfc0f7746c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:56.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.592+0000 7fa4591a9640 1 -- 172.21.15.116:0/489457906 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4500b4070 con 0x7fa454070dd0 2024-04-25T00:14:56.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.592+0000 7fa4591a9640 1 -- 172.21.15.116:0/489457906 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4500a7a80 con 0x7fa454070dd0 2024-04-25T00:14:56.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.592+0000 7fa4591a9640 1 -- 172.21.15.116:0/489457906 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4500a7e50 con 0x7fa454070dd0 2024-04-25T00:14:56.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.592+0000 7fa45c40f640 1 -- 172.21.15.116:0/489457906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa454070dd0 msgr2=0x7fa454148890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:56.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.592+0000 7fa45c40f640 1 --2- 172.21.15.116:0/489457906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa454070dd0 0x7fa454148890 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fa45000bb10 tx=0x7fa45001ebb0 comp rx=0 tx=0).stop 2024-04-25T00:14:56.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.592+0000 7fa45c40f640 1 -- 172.21.15.116:0/489457906 shutdown_connections 2024-04-25T00:14:56.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.592+0000 7fa45c40f640 1 --2- 172.21.15.116:0/489457906 >> v2:172.21.15.174:3300/0 conn(0x7fa45406e890 0x7fa4540c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:56.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.592+0000 7fa45c40f640 1 --2- 172.21.15.116:0/489457906 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa454070dd0 0x7fa454148890 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:56.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.592+0000 7fa45c40f640 1 -- 172.21.15.116:0/489457906 >> 172.21.15.116:0/489457906 conn(0x7fa4540c3b00 msgr2=0x7fa4540c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:56.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.592+0000 7fa45c40f640 1 -- 172.21.15.116:0/489457906 shutdown_connections 2024-04-25T00:14:56.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.592+0000 7fa45c40f640 1 -- 172.21.15.116:0/489457906 wait complete. 2024-04-25T00:14:56.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.592+0000 7fa45c40f640 1 Processor -- start 2024-04-25T00:14:56.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.592+0000 7fa45c40f640 1 -- start start 2024-04-25T00:14:56.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.592+0000 7fa45c40f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa45406e890 0x7fa454154200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:56.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.592+0000 7fa45c40f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa454070dd0 0x7fa4540ab7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:56.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.592+0000 7fa45c40f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa4540c3540 0x7fa454149e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:56.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.592+0000 7fa45c40f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa4540c30d0 con 0x7fa45406e890 2024-04-25T00:14:56.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.592+0000 7fa45c40f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa4540c2f50 con 0x7fa454070dd0 2024-04-25T00:14:56.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.592+0000 7fa45c40f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa454071a80 con 0x7fa4540c3540 2024-04-25T00:14:56.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.592+0000 7fa45a1ab640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa45406e890 0x7fa454154200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:56.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.592+0000 7fa45a1ab640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa45406e890 0x7fa454154200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40000/0 (socket says 172.21.15.116:40000) 2024-04-25T00:14:56.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.592+0000 7fa45a1ab640 1 -- 172.21.15.116:0/494789592 learned_addr learned my addr 172.21.15.116:0/494789592 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:14:56.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.592+0000 7fa4599aa640 1 --2- 172.21.15.116:0/494789592 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa454070dd0 0x7fa4540ab7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:56.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.592+0000 7fa45a1ab640 1 -- 172.21.15.116:0/494789592 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa4540c3540 msgr2=0x7fa454149e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:56.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.592+0000 7fa45a9ac640 1 --2- 172.21.15.116:0/494789592 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa4540c3540 0x7fa454149e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:56.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.592+0000 7fa45a1ab640 1 --2- 172.21.15.116:0/494789592 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa4540c3540 0x7fa454149e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:56.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.592+0000 7fa45a1ab640 1 -- 172.21.15.116:0/494789592 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa454070dd0 msgr2=0x7fa4540ab7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:56.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.592+0000 7fa45a1ab640 1 --2- 172.21.15.116:0/494789592 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa454070dd0 0x7fa4540ab7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:56.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.592+0000 7fa45a1ab640 1 -- 172.21.15.116:0/494789592 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa45413aa90 con 0x7fa45406e890 2024-04-25T00:14:56.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.592+0000 7fa4599aa640 1 --2- 172.21.15.116:0/494789592 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa454070dd0 0x7fa4540ab7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:14:56.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.592+0000 7fa45a1ab640 1 --2- 172.21.15.116:0/494789592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa45406e890 0x7fa454154200 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fa45000b2c0 tx=0x7fa45001b950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:56.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.592+0000 7fa44affd640 1 -- 172.21.15.116:0/494789592 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4500be070 con 0x7fa45406e890 2024-04-25T00:14:56.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.592+0000 7fa44affd640 1 -- 172.21.15.116:0/494789592 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4500b9460 con 0x7fa45406e890 2024-04-25T00:14:56.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.592+0000 7fa45c40f640 1 -- 172.21.15.116:0/494789592 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa454155ce0 con 0x7fa45406e890 2024-04-25T00:14:56.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.592+0000 7fa44affd640 1 -- 172.21.15.116:0/494789592 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4500b4040 con 0x7fa45406e890 2024-04-25T00:14:56.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.592+0000 7fa45c40f640 1 -- 172.21.15.116:0/494789592 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4540ac020 con 0x7fa45406e890 2024-04-25T00:14:56.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.596+0000 7fa45c40f640 1 -- 172.21.15.116:0/494789592 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa454068350 con 0x7fa45406e890 2024-04-25T00:14:56.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.596+0000 7fa44affd640 1 -- 172.21.15.116:0/494789592 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa4500c1070 con 0x7fa45406e890 2024-04-25T00:14:56.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.596+0000 7fa44affd640 1 --2- 172.21.15.116:0/494789592 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa4380678f0 0x7fa438069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:56.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.596+0000 7fa44affd640 1 -- 172.21.15.116:0/494789592 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 8214+0+0 (secure 0 0 0) 0x7fa450123e50 con 0x7fa45406e890 2024-04-25T00:14:56.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.596+0000 7fa4599aa640 1 --2- 172.21.15.116:0/494789592 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa4380678f0 0x7fa438069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:56.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.596+0000 7fa4599aa640 1 --2- 172.21.15.116:0/494789592 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa4380678f0 0x7fa438069db0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fa4541503d0 tx=0x7fa444008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:56.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.600+0000 7fa44affd640 1 -- 172.21.15.116:0/494789592 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fa4500187a0 con 0x7fa45406e890 2024-04-25T00:14:56.787 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:14:56.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.796+0000 7fa45c40f640 1 -- 172.21.15.116:0/494789592 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa4540ac7e0 con 0x7fa45406e890 2024-04-25T00:14:56.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.796+0000 7fa44affd640 1 -- 172.21.15.116:0/494789592 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) ==== 94+0+5663 (secure 0 0 0) 0x7fa4500f4080 con 0x7fa45406e890 2024-04-25T00:14:56.796 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:14:56.797 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:12:48.771611+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":88,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6833/365773583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":365773583},{"type":"v1","addr":"172.21.15.116:6833","nonce":365773583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":1},{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:14:55.561272+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5791},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5791":{"gid":5791,"name":"b","rank":0,"incarnation":98,"state":"up:replay","state_seq":1,"addr":"172.21.15.174:6837/3180024481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3180024481},{"type":"v1","addr":"172.21.15.174:6837","nonce":3180024481}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:14:56.797 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 98 2024-04-25T00:14:56.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.796+0000 7fa45c40f640 1 -- 172.21.15.116:0/494789592 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa4380678f0 msgr2=0x7fa438069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:56.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.796+0000 7fa45c40f640 1 --2- 172.21.15.116:0/494789592 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa4380678f0 0x7fa438069db0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fa4541503d0 tx=0x7fa444008040 comp rx=0 tx=0).stop 2024-04-25T00:14:56.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.796+0000 7fa45c40f640 1 -- 172.21.15.116:0/494789592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa45406e890 msgr2=0x7fa454154200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:56.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.796+0000 7fa45c40f640 1 --2- 172.21.15.116:0/494789592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa45406e890 0x7fa454154200 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fa45000b2c0 tx=0x7fa45001b950 comp rx=0 tx=0).stop 2024-04-25T00:14:56.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.796+0000 7fa45c40f640 1 -- 172.21.15.116:0/494789592 shutdown_connections 2024-04-25T00:14:56.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.796+0000 7fa45c40f640 1 --2- 172.21.15.116:0/494789592 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa4380678f0 0x7fa438069db0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:56.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.796+0000 7fa45c40f640 1 --2- 172.21.15.116:0/494789592 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa4540c3540 0x7fa454149e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:56.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.796+0000 7fa45c40f640 1 --2- 172.21.15.116:0/494789592 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa454070dd0 0x7fa4540ab7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:56.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.796+0000 7fa45c40f640 1 --2- 172.21.15.116:0/494789592 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa45406e890 0x7fa454154200 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:56.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.796+0000 7fa45c40f640 1 -- 172.21.15.116:0/494789592 >> 172.21.15.116:0/494789592 conn(0x7fa4540c3b00 msgr2=0x7fa4540c67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:56.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.796+0000 7fa45c40f640 1 -- 172.21.15.116:0/494789592 shutdown_connections 2024-04-25T00:14:56.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.796+0000 7fa45c40f640 1 -- 172.21.15.116:0/494789592 wait complete. 2024-04-25T00:14:56.816 INFO:tasks.mds_thrash.fs.[b]:mds.a down, removed from mdsmap 2024-04-25T00:14:56.816 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-04-25T00:14:56.817 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:14:56.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.916+0000 7f00bdd09640 1 Processor -- start 2024-04-25T00:14:56.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.916+0000 7f00bdd09640 1 -- start start 2024-04-25T00:14:56.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.916+0000 7f00bdd09640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f00b8152d80 0x7f00b8159950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:56.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.916+0000 7f00bdd09640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f00b81562a0 0x7f00b80c5020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:56.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.916+0000 7f00bdd09640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00b80730b0 con 0x7f00b8072080 2024-04-25T00:14:56.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.916+0000 7f00bdd09640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f00b8072e60 con 0x7f00b81562a0 2024-04-25T00:14:56.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.916+0000 7f00bdd09640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f00b8073390 con 0x7f00b8152d80 2024-04-25T00:14:56.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.916+0000 7f00b77fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f00b8152d80 0x7f00b8159950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:56.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.916+0000 7f00b7fff640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f00b81562a0 0x7f00b80c5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:56.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.916+0000 7f00b77fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f00b8152d80 0x7f00b8159950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:43808/0 (socket says 172.21.15.116:43808) 2024-04-25T00:14:56.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.916+0000 7f00b77fe640 1 -- 172.21.15.116:0/3160100387 learned_addr learned my addr 172.21.15.116:0/3160100387 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:14:56.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.916+0000 7f00affff640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f00b8072080 0x7f00b80c36b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:33628/0 (socket says 172.21.15.116:33628) 2024-04-25T00:14:56.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.916+0000 7f00b6ffd640 1 -- 172.21.15.116:0/3160100387 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2075084612 0 0) 0x7f00b80730b0 con 0x7f00b8072080 2024-04-25T00:14:56.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.916+0000 7f00b77fe640 1 -- 172.21.15.116:0/3160100387 >> v2:172.21.15.174:3300/0 conn(0x7f00b81562a0 msgr2=0x7f00b80c5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:14:56.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.916+0000 7f00b77fe640 1 --2- 172.21.15.116:0/3160100387 >> v2:172.21.15.174:3300/0 conn(0x7f00b81562a0 0x7f00b80c5020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:56.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.916+0000 7f00b77fe640 1 -- 172.21.15.116:0/3160100387 >> v1:172.21.15.174:6789/0 conn(0x7f00b8072080 legacy=0x7f00b80c36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:56.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.916+0000 7f00b77fe640 1 -- 172.21.15.116:0/3160100387 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00b8158e70 con 0x7f00b8152d80 2024-04-25T00:14:56.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.920+0000 7f00b77fe640 1 --2- 172.21.15.116:0/3160100387 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f00b8152d80 0x7f00b8159950 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f00a8009870 tx=0x7f00a801c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f70e255aa7382cdc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:56.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.920+0000 7f00b6ffd640 1 -- 172.21.15.116:0/3160100387 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00a80aa600 con 0x7f00b8152d80 2024-04-25T00:14:56.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.920+0000 7f00b6ffd640 1 -- 172.21.15.116:0/3160100387 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00a80af070 con 0x7f00b8152d80 2024-04-25T00:14:56.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.920+0000 7f00b6ffd640 1 -- 172.21.15.116:0/3160100387 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00a80aa9c0 con 0x7f00b8152d80 2024-04-25T00:14:56.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.920+0000 7f00bdd09640 1 -- 172.21.15.116:0/3160100387 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f00b8152d80 msgr2=0x7f00b8159950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:56.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.920+0000 7f00bdd09640 1 --2- 172.21.15.116:0/3160100387 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f00b8152d80 0x7f00b8159950 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f00a8009870 tx=0x7f00a801c4a0 comp rx=0 tx=0).stop 2024-04-25T00:14:56.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.920+0000 7f00bdd09640 1 -- 172.21.15.116:0/3160100387 shutdown_connections 2024-04-25T00:14:56.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.920+0000 7f00bdd09640 1 --2- 172.21.15.116:0/3160100387 >> v2:172.21.15.174:3300/0 conn(0x7f00b81562a0 0x7f00b80c5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:56.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.920+0000 7f00bdd09640 1 --2- 172.21.15.116:0/3160100387 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f00b8152d80 0x7f00b8159950 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:56.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.920+0000 7f00bdd09640 1 -- 172.21.15.116:0/3160100387 >> 172.21.15.116:0/3160100387 conn(0x7f00b806c720 msgr2=0x7f00b806cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:56.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.920+0000 7f00bdd09640 1 -- 172.21.15.116:0/3160100387 shutdown_connections 2024-04-25T00:14:56.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.920+0000 7f00bdd09640 1 -- 172.21.15.116:0/3160100387 wait complete. 2024-04-25T00:14:56.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.920+0000 7f00bdd09640 1 Processor -- start 2024-04-25T00:14:56.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.920+0000 7f00bdd09640 1 -- start start 2024-04-25T00:14:56.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.920+0000 7f00bdd09640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00b8072080 0x7f00b81f2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:56.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.920+0000 7f00bdd09640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f00b8152d80 0x7f00b81512b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:56.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.920+0000 7f00bdd09640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f00b81562a0 0x7f00b814e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:56.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.924+0000 7f00bdd09640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f00b8153160 con 0x7f00b8072080 2024-04-25T00:14:56.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.924+0000 7f00bdd09640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f00b81566b0 con 0x7f00b8152d80 2024-04-25T00:14:56.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.924+0000 7f00bdd09640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f00b81532e0 con 0x7f00b81562a0 2024-04-25T00:14:56.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.924+0000 7f00b77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00b8072080 0x7f00b81f2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:56.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.924+0000 7f00b77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00b8072080 0x7f00b81f2070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40014/0 (socket says 172.21.15.116:40014) 2024-04-25T00:14:56.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.924+0000 7f00b77fe640 1 -- 172.21.15.116:0/1618567790 learned_addr learned my addr 172.21.15.116:0/1618567790 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:14:56.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.924+0000 7f00affff640 1 --2- 172.21.15.116:0/1618567790 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f00b8152d80 0x7f00b81512b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:56.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.924+0000 7f00b7fff640 1 --2- 172.21.15.116:0/1618567790 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f00b81562a0 0x7f00b814e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:56.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.924+0000 7f00b77fe640 1 -- 172.21.15.116:0/1618567790 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f00b81562a0 msgr2=0x7f00b814e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:56.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.924+0000 7f00b77fe640 1 --2- 172.21.15.116:0/1618567790 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f00b81562a0 0x7f00b814e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:56.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.924+0000 7f00b77fe640 1 -- 172.21.15.116:0/1618567790 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f00b8152d80 msgr2=0x7f00b81512b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:56.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.924+0000 7f00b77fe640 1 --2- 172.21.15.116:0/1618567790 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f00b8152d80 0x7f00b81512b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:56.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.924+0000 7f00b77fe640 1 -- 172.21.15.116:0/1618567790 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00b80c8fa0 con 0x7f00b8072080 2024-04-25T00:14:56.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.924+0000 7f00b7fff640 1 --2- 172.21.15.116:0/1618567790 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f00b81562a0 0x7f00b814e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:14:56.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.924+0000 7f00b77fe640 1 --2- 172.21.15.116:0/1618567790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00b8072080 0x7f00b81f2070 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f00a801a0d0 tx=0x7f00a801ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:56.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.924+0000 7f00b4ff9640 1 -- 172.21.15.116:0/1618567790 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00a80b9070 con 0x7f00b8072080 2024-04-25T00:14:56.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.924+0000 7f00b4ff9640 1 -- 172.21.15.116:0/1618567790 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00a80aabc0 con 0x7f00b8072080 2024-04-25T00:14:56.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.924+0000 7f00bdd09640 1 -- 172.21.15.116:0/1618567790 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00b814dd20 con 0x7f00b8072080 2024-04-25T00:14:56.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.924+0000 7f00b4ff9640 1 -- 172.21.15.116:0/1618567790 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00a8004020 con 0x7f00b8072080 2024-04-25T00:14:56.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.924+0000 7f00bdd09640 1 -- 172.21.15.116:0/1618567790 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00b814dfc0 con 0x7f00b8072080 2024-04-25T00:14:56.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.924+0000 7f00bdd09640 1 -- 172.21.15.116:0/1618567790 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00b8068a60 con 0x7f00b8072080 2024-04-25T00:14:56.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.924+0000 7f00b4ff9640 1 -- 172.21.15.116:0/1618567790 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f00a80a9070 con 0x7f00b8072080 2024-04-25T00:14:56.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.924+0000 7f00b4ff9640 1 --2- 172.21.15.116:0/1618567790 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f00980678c0 0x7f0098069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:56.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.924+0000 7f00b4ff9640 1 -- 172.21.15.116:0/1618567790 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 8214+0+0 (secure 0 0 0) 0x7f00a81207e0 con 0x7f00b8072080 2024-04-25T00:14:56.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.924+0000 7f00affff640 1 --2- 172.21.15.116:0/1618567790 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f00980678c0 0x7f0098069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:56.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.928+0000 7f00affff640 1 --2- 172.21.15.116:0/1618567790 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f00980678c0 0x7f0098069d80 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f00b80c5470 tx=0x7f00a0007b60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:56.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.928+0000 7f00b4ff9640 1 -- 172.21.15.116:0/1618567790 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f00a80c2060 con 0x7f00b8072080 2024-04-25T00:14:56.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.976+0000 7f79fb578640 1 Processor -- start 2024-04-25T00:14:56.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.976+0000 7f79fb578640 1 -- start start 2024-04-25T00:14:56.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.976+0000 7f79fb578640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f79f414c720 0x7f79f41563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:56.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.976+0000 7f79fb578640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f79f40a94a0 0x7f79f40c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:56.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.976+0000 7f79fb578640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79f4071220 con 0x7f79f40c2f50 2024-04-25T00:14:56.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.976+0000 7f79fb578640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f79f4153460 con 0x7f79f40a94a0 2024-04-25T00:14:56.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.976+0000 7f79fb578640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f79f4152420 con 0x7f79f414c720 2024-04-25T00:14:56.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.976+0000 7f79f9314640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f79f414c720 0x7f79f41563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:56.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.976+0000 7f79f9314640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f79f414c720 0x7f79f41563d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:43824/0 (socket says 172.21.15.116:43824) 2024-04-25T00:14:56.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.976+0000 7f79f8b13640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f79f40a94a0 0x7f79f40c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:56.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.976+0000 7f79f9314640 1 -- 172.21.15.116:0/2713830943 learned_addr learned my addr 172.21.15.116:0/2713830943 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:14:56.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.980+0000 7f79f9314640 1 -- 172.21.15.116:0/2713830943 >> v2:172.21.15.174:3300/0 conn(0x7f79f40a94a0 msgr2=0x7f79f40c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:14:56.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.980+0000 7f79f9314640 1 --2- 172.21.15.116:0/2713830943 >> v2:172.21.15.174:3300/0 conn(0x7f79f40a94a0 0x7f79f40c2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:56.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.980+0000 7f79f9314640 1 -- 172.21.15.116:0/2713830943 >> v1:172.21.15.174:6789/0 conn(0x7f79f40c2f50 legacy=0x7f79f4071680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:56.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.980+0000 7f79f9314640 1 -- 172.21.15.116:0/2713830943 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79f414cb00 con 0x7f79f414c720 2024-04-25T00:14:56.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.980+0000 7f79f9314640 1 --2- 172.21.15.116:0/2713830943 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f79f414c720 0x7f79f41563d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f79e4009870 tx=0x7f79e401c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bff96b6ca014a69e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:56.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.980+0000 7f79f3fff640 1 -- 172.21.15.116:0/2713830943 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e40aa620 con 0x7f79f414c720 2024-04-25T00:14:56.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.980+0000 7f79f3fff640 1 -- 172.21.15.116:0/2713830943 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79e40af070 con 0x7f79f414c720 2024-04-25T00:14:56.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.980+0000 7f79f3fff640 1 -- 172.21.15.116:0/2713830943 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e40aa9e0 con 0x7f79f414c720 2024-04-25T00:14:56.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.980+0000 7f79fb578640 1 -- 172.21.15.116:0/2713830943 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f79f414c720 msgr2=0x7f79f41563d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:56.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.980+0000 7f79fb578640 1 --2- 172.21.15.116:0/2713830943 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f79f414c720 0x7f79f41563d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f79e4009870 tx=0x7f79e401c4a0 comp rx=0 tx=0).stop 2024-04-25T00:14:56.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.980+0000 7f79fb578640 1 -- 172.21.15.116:0/2713830943 shutdown_connections 2024-04-25T00:14:56.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.980+0000 7f79fb578640 1 --2- 172.21.15.116:0/2713830943 >> v2:172.21.15.174:3300/0 conn(0x7f79f40a94a0 0x7f79f40c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:56.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.980+0000 7f79fb578640 1 --2- 172.21.15.116:0/2713830943 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f79f414c720 0x7f79f41563d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:56.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.980+0000 7f79fb578640 1 -- 172.21.15.116:0/2713830943 >> 172.21.15.116:0/2713830943 conn(0x7f79f40c3b00 msgr2=0x7f79f40c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:56.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.980+0000 7f79fb578640 1 -- 172.21.15.116:0/2713830943 shutdown_connections 2024-04-25T00:14:56.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.980+0000 7f79fb578640 1 -- 172.21.15.116:0/2713830943 wait complete. 2024-04-25T00:14:56.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.984+0000 7f79fb578640 1 Processor -- start 2024-04-25T00:14:56.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.984+0000 7f79fb578640 1 -- start start 2024-04-25T00:14:56.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.984+0000 7f79fb578640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79f40a94a0 0x7f79f4154e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:56.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.984+0000 7f79fb578640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79f40c2f50 0x7f79f4149d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:56.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.984+0000 7f79fb578640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f79f414c720 0x7f79f414a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:56.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.984+0000 7f79fb578640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f79f4152f40 con 0x7f79f40c2f50 2024-04-25T00:14:56.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.984+0000 7f79fb578640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f79f4152c60 con 0x7f79f40a94a0 2024-04-25T00:14:56.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.984+0000 7f79fb578640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f79f415f5c0 con 0x7f79f414c720 2024-04-25T00:14:56.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.984+0000 7f79f8b13640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79f40c2f50 0x7f79f4149d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:56.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.984+0000 7f79f8b13640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79f40c2f50 0x7f79f4149d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40030/0 (socket says 172.21.15.116:40030) 2024-04-25T00:14:56.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.984+0000 7f79f8b13640 1 -- 172.21.15.116:0/2101700805 learned_addr learned my addr 172.21.15.116:0/2101700805 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:14:56.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.984+0000 7f79f9314640 1 --2- 172.21.15.116:0/2101700805 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79f40a94a0 0x7f79f4154e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:56.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.984+0000 7f79f9b15640 1 --2- 172.21.15.116:0/2101700805 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f79f414c720 0x7f79f414a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:56.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.984+0000 7f79f8b13640 1 -- 172.21.15.116:0/2101700805 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f79f414c720 msgr2=0x7f79f414a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:56.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.984+0000 7f79f8b13640 1 --2- 172.21.15.116:0/2101700805 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f79f414c720 0x7f79f414a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:56.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.984+0000 7f79f8b13640 1 -- 172.21.15.116:0/2101700805 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79f40a94a0 msgr2=0x7f79f4154e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:56.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.984+0000 7f79f8b13640 1 --2- 172.21.15.116:0/2101700805 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79f40a94a0 0x7f79f4154e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:56.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.984+0000 7f79f8b13640 1 -- 172.21.15.116:0/2101700805 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79f4154190 con 0x7f79f40c2f50 2024-04-25T00:14:56.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.984+0000 7f79f8b13640 1 --2- 172.21.15.116:0/2101700805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79f40c2f50 0x7f79f4149d80 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f79e800e930 tx=0x7f79e800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:56.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.984+0000 7f79f1ffb640 1 -- 172.21.15.116:0/2101700805 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e8017070 con 0x7f79f40c2f50 2024-04-25T00:14:56.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.984+0000 7f79f1ffb640 1 -- 172.21.15.116:0/2101700805 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79e800ca00 con 0x7f79f40c2f50 2024-04-25T00:14:56.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.984+0000 7f79fb578640 1 -- 172.21.15.116:0/2101700805 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79f40c5550 con 0x7f79f40c2f50 2024-04-25T00:14:56.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.984+0000 7f79f1ffb640 1 -- 172.21.15.116:0/2101700805 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79e800cdb0 con 0x7f79f40c2f50 2024-04-25T00:14:56.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.984+0000 7f79fb578640 1 -- 172.21.15.116:0/2101700805 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79f4154430 con 0x7f79f40c2f50 2024-04-25T00:14:56.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.984+0000 7f79fb578640 1 -- 172.21.15.116:0/2101700805 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79f4068350 con 0x7f79f40c2f50 2024-04-25T00:14:56.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.984+0000 7f79f1ffb640 1 -- 172.21.15.116:0/2101700805 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f79e801a070 con 0x7f79f40c2f50 2024-04-25T00:14:56.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.988+0000 7f79f1ffb640 1 --2- 172.21.15.116:0/2101700805 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f79e00678f0 0x7f79e0069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:56.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.988+0000 7f79f1ffb640 1 -- 172.21.15.116:0/2101700805 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 8214+0+0 (secure 0 0 0) 0x7f79e8086610 con 0x7f79f40c2f50 2024-04-25T00:14:56.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.988+0000 7f79f9314640 1 --2- 172.21.15.116:0/2101700805 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f79e00678f0 0x7f79e0069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:56.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.988+0000 7f79f9314640 1 --2- 172.21.15.116:0/2101700805 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f79e00678f0 0x7f79e0069db0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f79e40ae000 tx=0x7f79e40a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:56.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:56.992+0000 7f79f1ffb640 1 -- 172.21.15.116:0/2101700805 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f79e80561d0 con 0x7f79f40c2f50 2024-04-25T00:14:57.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:57.120+0000 7f00bdd09640 1 -- 172.21.15.116:0/1618567790 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f00b80ad120 con 0x7f00b8072080 2024-04-25T00:14:57.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:57.120+0000 7f00b4ff9640 1 -- 172.21.15.116:0/1618567790 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) ==== 94+0+5667 (secure 0 0 0) 0x7f00a8016150 con 0x7f00b8072080 2024-04-25T00:14:57.120 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:14:57.121 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":99,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:12:48.771611+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":88,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6833/365773583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":365773583},{"type":"v1","addr":"172.21.15.116:6833","nonce":365773583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":1},{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:14:55.893217+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5791},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5791":{"gid":5791,"name":"b","rank":0,"incarnation":98,"state":"up:reconnect","state_seq":21,"addr":"172.21.15.174:6837/3180024481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3180024481},{"type":"v1","addr":"172.21.15.174:6837","nonce":3180024481}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:14:57.121 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 99 2024-04-25T00:14:57.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:57.120+0000 7f00bdd09640 1 -- 172.21.15.116:0/1618567790 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f00980678c0 msgr2=0x7f0098069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:57.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:57.120+0000 7f00bdd09640 1 --2- 172.21.15.116:0/1618567790 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f00980678c0 0x7f0098069d80 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f00b80c5470 tx=0x7f00a0007b60 comp rx=0 tx=0).stop 2024-04-25T00:14:57.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:57.120+0000 7f00bdd09640 1 -- 172.21.15.116:0/1618567790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00b8072080 msgr2=0x7f00b81f2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:57.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:57.120+0000 7f00bdd09640 1 --2- 172.21.15.116:0/1618567790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00b8072080 0x7f00b81f2070 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f00a801a0d0 tx=0x7f00a801ace0 comp rx=0 tx=0).stop 2024-04-25T00:14:57.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:57.120+0000 7f00bdd09640 1 -- 172.21.15.116:0/1618567790 shutdown_connections 2024-04-25T00:14:57.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:57.120+0000 7f00bdd09640 1 --2- 172.21.15.116:0/1618567790 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f00980678c0 0x7f0098069d80 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:57.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:57.120+0000 7f00bdd09640 1 --2- 172.21.15.116:0/1618567790 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f00b81562a0 0x7f00b814e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:57.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:57.120+0000 7f00bdd09640 1 --2- 172.21.15.116:0/1618567790 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f00b8152d80 0x7f00b81512b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:57.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:57.120+0000 7f00bdd09640 1 --2- 172.21.15.116:0/1618567790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00b8072080 0x7f00b81f2070 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:57.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:57.120+0000 7f00bdd09640 1 -- 172.21.15.116:0/1618567790 >> 172.21.15.116:0/1618567790 conn(0x7f00b806c720 msgr2=0x7f00b806cb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:57.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:57.120+0000 7f00bdd09640 1 -- 172.21.15.116:0/1618567790 shutdown_connections 2024-04-25T00:14:57.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:57.120+0000 7f00bdd09640 1 -- 172.21.15.116:0/1618567790 wait complete. 2024-04-25T00:14:57.137 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:14:57.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:57.184+0000 7f79fb578640 1 -- 172.21.15.116:0/2101700805 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f79f40c3640 con 0x7f79f40c2f50 2024-04-25T00:14:57.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:57.184+0000 7f79f1ffb640 1 -- 172.21.15.116:0/2101700805 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) ==== 94+0+5667 (secure 0 0 0) 0x7f79e805a4d0 con 0x7f79f40c2f50 2024-04-25T00:14:57.185 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:14:57.185 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":99,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":82}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:12:48.771611+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":44,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6495},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6495":{"gid":6495,"name":"c","rank":0,"incarnation":88,"state":"up:active","state_seq":7,"addr":"172.21.15.116:6833/365773583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":365773583},{"type":"v1","addr":"172.21.15.116:6833","nonce":365773583}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6495,"qdb_cluster":[6495]},"id":1},{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:14:55.893217+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5791},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5791":{"gid":5791,"name":"b","rank":0,"incarnation":98,"state":"up:reconnect","state_seq":21,"addr":"172.21.15.174:6837/3180024481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3180024481},{"type":"v1","addr":"172.21.15.174:6837","nonce":3180024481}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:14:57.185 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 99 2024-04-25T00:14:57.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:57.184+0000 7f79fb578640 1 -- 172.21.15.116:0/2101700805 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f79e00678f0 msgr2=0x7f79e0069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:57.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:57.184+0000 7f79fb578640 1 --2- 172.21.15.116:0/2101700805 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f79e00678f0 0x7f79e0069db0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f79e40ae000 tx=0x7f79e40a7000 comp rx=0 tx=0).stop 2024-04-25T00:14:57.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:57.184+0000 7f79fb578640 1 -- 172.21.15.116:0/2101700805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79f40c2f50 msgr2=0x7f79f4149d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:57.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:57.184+0000 7f79fb578640 1 --2- 172.21.15.116:0/2101700805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79f40c2f50 0x7f79f4149d80 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f79e800e930 tx=0x7f79e800edf0 comp rx=0 tx=0).stop 2024-04-25T00:14:57.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:57.184+0000 7f79fb578640 1 -- 172.21.15.116:0/2101700805 shutdown_connections 2024-04-25T00:14:57.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:57.184+0000 7f79fb578640 1 --2- 172.21.15.116:0/2101700805 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f79e00678f0 0x7f79e0069db0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:57.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:57.184+0000 7f79fb578640 1 --2- 172.21.15.116:0/2101700805 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f79f414c720 0x7f79f414a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:57.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:57.184+0000 7f79fb578640 1 --2- 172.21.15.116:0/2101700805 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f79f40c2f50 0x7f79f4149d80 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:57.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:57.184+0000 7f79fb578640 1 --2- 172.21.15.116:0/2101700805 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f79f40a94a0 0x7f79f4154e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:57.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:57.184+0000 7f79fb578640 1 -- 172.21.15.116:0/2101700805 >> 172.21.15.116:0/2101700805 conn(0x7f79f40c3b00 msgr2=0x7f79f4144c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:57.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:57.184+0000 7f79fb578640 1 -- 172.21.15.116:0/2101700805 shutdown_connections 2024-04-25T00:14:57.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:57.184+0000 7f79fb578640 1 -- 172.21.15.116:0/2101700805 wait complete. 2024-04-25T00:14:57.201 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 99, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:04.678737+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 91, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6495": { "addr": "172.21.15.116:6833/365773583", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6832", "nonce": 365773583, "type": "v2" }, { "addr": "172.21.15.116:6833", "nonce": 365773583, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6495, "incarnation": 88, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 7 } }, "last_failure": 0, "last_failure_osd_epoch": 44, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T00:12:48.771611+0000", "qdb_cluster": [ 6495 ], "qdb_leader": 6495, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6495 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:18.825887+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 99, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5791": { "addr": "172.21.15.174:6837/3180024481", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6836", "nonce": 3180024481, "type": "v2" }, { "addr": "172.21.15.174:6837", "nonce": 3180024481, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5791, "incarnation": 98, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:reconnect", "state_seq": 21 } }, "last_failure": 0, "last_failure_osd_epoch": 46, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-04-25T00:14:55.893217+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5791 } } } ], "standbys": [ { "addr": "172.21.15.174:6835/4215220222", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6834", "nonce": 4215220222, "type": "v2" }, { "addr": "172.21.15.174:6835", "nonce": 4215220222, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 82, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5273, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-25T00:14:59.139 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:14:59.202 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:14:59.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.268+0000 7f1777d56640 1 Processor -- start 2024-04-25T00:14:59.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.268+0000 7f1777d56640 1 -- start start 2024-04-25T00:14:59.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.272+0000 7f1777d56640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f177014c720 0x7f17701563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:59.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.272+0000 7f1777d56640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1770071220 con 0x7f1770071680 2024-04-25T00:14:59.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.272+0000 7f1777d56640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17700721e0 con 0x7f17700a94a0 2024-04-25T00:14:59.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.272+0000 7f1777d56640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f1770153460 con 0x7f177014c720 2024-04-25T00:14:59.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.272+0000 7f1775af2640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f177014c720 0x7f17701563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:59.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.272+0000 7f17762f3640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f1770071680 0x7f1770071ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54622/0 (socket says 172.21.15.116:54622) 2024-04-25T00:14:59.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.272+0000 7f17762f3640 1 -- 172.21.15.116:0/3784213361 learned_addr learned my addr 172.21.15.116:0/3784213361 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:14:59.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.272+0000 7f1775af2640 1 -- 172.21.15.116:0/3784213361 >> v1:172.21.15.174:6789/0 conn(0x7f17700a94a0 legacy=0x7f17700c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:59.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.272+0000 7f1775af2640 1 -- 172.21.15.116:0/3784213361 >> v1:172.21.15.116:6789/0 conn(0x7f1770071680 legacy=0x7f1770071ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:59.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.272+0000 7f1775af2640 1 -- 172.21.15.116:0/3784213361 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1770152420 con 0x7f177014c720 2024-04-25T00:14:59.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.272+0000 7f1775af2640 1 --2- 172.21.15.116:0/3784213361 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f177014c720 0x7f17701563d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f17680097f0 tx=0x7f176801c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da5c5c92e2156bf7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:59.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.272+0000 7f1774af0640 1 -- 172.21.15.116:0/3784213361 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17680b0070 con 0x7f177014c720 2024-04-25T00:14:59.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.272+0000 7f1774af0640 1 -- 172.21.15.116:0/3784213361 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1768004230 con 0x7f177014c720 2024-04-25T00:14:59.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.272+0000 7f1774af0640 1 -- 172.21.15.116:0/3784213361 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1768003620 con 0x7f177014c720 2024-04-25T00:14:59.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.276+0000 7f1777d56640 1 -- 172.21.15.116:0/3784213361 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f177014c720 msgr2=0x7f17701563d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:59.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.276+0000 7f1777d56640 1 --2- 172.21.15.116:0/3784213361 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f177014c720 0x7f17701563d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f17680097f0 tx=0x7f176801c450 comp rx=0 tx=0).stop 2024-04-25T00:14:59.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.276+0000 7f1777d56640 1 -- 172.21.15.116:0/3784213361 shutdown_connections 2024-04-25T00:14:59.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.276+0000 7f1777d56640 1 --2- 172.21.15.116:0/3784213361 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f177014c720 0x7f17701563d0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:59.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.276+0000 7f1777d56640 1 -- 172.21.15.116:0/3784213361 >> 172.21.15.116:0/3784213361 conn(0x7f17700c3b00 msgr2=0x7f17700c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:59.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.276+0000 7f1777d56640 1 -- 172.21.15.116:0/3784213361 shutdown_connections 2024-04-25T00:14:59.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.276+0000 7f1777d56640 1 -- 172.21.15.116:0/3784213361 wait complete. 2024-04-25T00:14:59.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.276+0000 7f1777d56640 1 Processor -- start 2024-04-25T00:14:59.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.276+0000 7f1777d56640 1 -- start start 2024-04-25T00:14:59.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.276+0000 7f1777d56640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1770071680 0x7f17700c64f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:59.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.276+0000 7f1777d56640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17700a94a0 0x7f17700c0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:59.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.276+0000 7f1777d56640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f177014c720 0x7f177013f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:59.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.276+0000 7f1777d56640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1770152f40 con 0x7f17700a94a0 2024-04-25T00:14:59.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.276+0000 7f1777d56640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1770152c60 con 0x7f1770071680 2024-04-25T00:14:59.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.276+0000 7f1777d56640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f17700c3120 con 0x7f177014c720 2024-04-25T00:14:59.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.276+0000 7f17752f1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17700a94a0 0x7f17700c0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:59.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.276+0000 7f17752f1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17700a94a0 0x7f17700c0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40040/0 (socket says 172.21.15.116:40040) 2024-04-25T00:14:59.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.276+0000 7f17762f3640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f177014c720 0x7f177013f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:59.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.276+0000 7f17752f1640 1 -- 172.21.15.116:0/2982466464 learned_addr learned my addr 172.21.15.116:0/2982466464 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:14:59.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.276+0000 7f1775af2640 1 --2- 172.21.15.116:0/2982466464 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1770071680 0x7f17700c64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:59.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.276+0000 7f17752f1640 1 -- 172.21.15.116:0/2982466464 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f177014c720 msgr2=0x7f177013f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:59.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.276+0000 7f17752f1640 1 --2- 172.21.15.116:0/2982466464 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f177014c720 0x7f177013f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:59.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.276+0000 7f17752f1640 1 -- 172.21.15.116:0/2982466464 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1770071680 msgr2=0x7f17700c64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:59.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.276+0000 7f17752f1640 1 --2- 172.21.15.116:0/2982466464 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1770071680 0x7f17700c64f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:59.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.276+0000 7f17752f1640 1 -- 172.21.15.116:0/2982466464 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17700c6a30 con 0x7f17700a94a0 2024-04-25T00:14:59.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.276+0000 7f17752f1640 1 --2- 172.21.15.116:0/2982466464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17700a94a0 0x7f17700c0690 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f176000c420 tx=0x7f176000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:59.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.276+0000 7f17667fc640 1 -- 172.21.15.116:0/2982466464 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17600052d0 con 0x7f17700a94a0 2024-04-25T00:14:59.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.276+0000 7f17667fc640 1 -- 172.21.15.116:0/2982466464 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1760005470 con 0x7f17700a94a0 2024-04-25T00:14:59.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.276+0000 7f17667fc640 1 -- 172.21.15.116:0/2982466464 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1760007260 con 0x7f17700a94a0 2024-04-25T00:14:59.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.280+0000 7f1777d56640 1 -- 172.21.15.116:0/2982466464 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17700c6d00 con 0x7f17700a94a0 2024-04-25T00:14:59.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.280+0000 7f1777d56640 1 -- 172.21.15.116:0/2982466464 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17701616e0 con 0x7f17700a94a0 2024-04-25T00:14:59.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.280+0000 7f1777d56640 1 -- 172.21.15.116:0/2982466464 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1770068350 con 0x7f17700a94a0 2024-04-25T00:14:59.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.280+0000 7f17667fc640 1 -- 172.21.15.116:0/2982466464 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1760007400 con 0x7f17700a94a0 2024-04-25T00:14:59.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.280+0000 7f17667fc640 1 --2- 172.21.15.116:0/2982466464 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f17540678c0 0x7f1754069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:59.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.280+0000 7f17667fc640 1 -- 172.21.15.116:0/2982466464 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 8300+0+0 (secure 0 0 0) 0x7f1760085b00 con 0x7f17700a94a0 2024-04-25T00:14:59.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.280+0000 7f1775af2640 1 --2- 172.21.15.116:0/2982466464 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f17540678c0 0x7f1754069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:59.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.280+0000 7f1775af2640 1 --2- 172.21.15.116:0/2982466464 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f17540678c0 0x7f1754069d80 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f17680ad000 tx=0x7f17680a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:59.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.284+0000 7f17667fc640 1 -- 172.21.15.116:0/2982466464 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f1760055590 con 0x7f17700a94a0 2024-04-25T00:14:59.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.320+0000 7fabcca99640 1 Processor -- start 2024-04-25T00:14:59.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.320+0000 7fabcca99640 1 -- start start 2024-04-25T00:14:59.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.320+0000 7fabcca99640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fabc814f400 0x7fabc80c2170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:59.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.320+0000 7fabcca99640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fabc80c6080 0x7fabc806c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:59.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.320+0000 7fabcca99640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabc814f7e0 con 0x7fabc806c760 2024-04-25T00:14:59.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.320+0000 7fabcca99640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fabc8152990 con 0x7fabc80c6080 2024-04-25T00:14:59.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.320+0000 7fabcca99640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fabc8152060 con 0x7fabc814f400 2024-04-25T00:14:59.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.320+0000 7fabc6d9d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fabc806c760 0x7fabc806e630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54630/0 (socket says 172.21.15.116:54630) 2024-04-25T00:14:59.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.320+0000 7fabc6d9d640 1 -- 172.21.15.116:0/402643817 learned_addr learned my addr 172.21.15.116:0/402643817 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:14:59.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.320+0000 7fabc5d9b640 1 --2- 172.21.15.116:0/402643817 >> v2:172.21.15.174:3300/0 conn(0x7fabc80c6080 0x7fabc806c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:59.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.320+0000 7fabc659c640 1 --2- 172.21.15.116:0/402643817 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fabc814f400 0x7fabc80c2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:59.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.320+0000 7fabc559a640 1 -- 172.21.15.116:0/402643817 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 399190600 0 0) 0x7fabc814f7e0 con 0x7fabc806c760 2024-04-25T00:14:59.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.320+0000 7fabc559a640 1 -- 172.21.15.116:0/402643817 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fabb4003530 con 0x7fabc806c760 2024-04-25T00:14:59.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.324+0000 7fabc659c640 1 -- 172.21.15.116:0/402643817 >> v2:172.21.15.174:3300/0 conn(0x7fabc80c6080 msgr2=0x7fabc806c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:14:59.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.324+0000 7fabc659c640 1 --2- 172.21.15.116:0/402643817 >> v2:172.21.15.174:3300/0 conn(0x7fabc80c6080 0x7fabc806c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:59.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.324+0000 7fabc659c640 1 -- 172.21.15.116:0/402643817 >> v1:172.21.15.116:6789/0 conn(0x7fabc806c760 legacy=0x7fabc806e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:59.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.324+0000 7fabc659c640 1 -- 172.21.15.116:0/402643817 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabc8152340 con 0x7fabc814f400 2024-04-25T00:14:59.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.324+0000 7fabc5d9b640 1 --2- 172.21.15.116:0/402643817 >> v2:172.21.15.174:3300/0 conn(0x7fabc80c6080 0x7fabc806c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:14:59.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.324+0000 7fabc659c640 1 --2- 172.21.15.116:0/402643817 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fabc814f400 0x7fabc80c2170 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fabb800a960 tx=0x7fabb801d7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa40f717153c047b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:59.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.324+0000 7fabc559a640 1 -- 172.21.15.116:0/402643817 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb80b3070 con 0x7fabc814f400 2024-04-25T00:14:59.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.324+0000 7fabc559a640 1 -- 172.21.15.116:0/402643817 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabb80a67f0 con 0x7fabc814f400 2024-04-25T00:14:59.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.324+0000 7fabc559a640 1 -- 172.21.15.116:0/402643817 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb800b040 con 0x7fabc814f400 2024-04-25T00:14:59.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.324+0000 7fabcca99640 1 -- 172.21.15.116:0/402643817 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fabc814f400 msgr2=0x7fabc80c2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:59.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.324+0000 7fabcca99640 1 --2- 172.21.15.116:0/402643817 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fabc814f400 0x7fabc80c2170 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fabb800a960 tx=0x7fabb801d7e0 comp rx=0 tx=0).stop 2024-04-25T00:14:59.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.324+0000 7fabcca99640 1 -- 172.21.15.116:0/402643817 shutdown_connections 2024-04-25T00:14:59.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.324+0000 7fabcca99640 1 --2- 172.21.15.116:0/402643817 >> v2:172.21.15.174:3300/0 conn(0x7fabc80c6080 0x7fabc806c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:59.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.324+0000 7fabcca99640 1 --2- 172.21.15.116:0/402643817 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fabc814f400 0x7fabc80c2170 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:59.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.324+0000 7fabcca99640 1 -- 172.21.15.116:0/402643817 >> 172.21.15.116:0/402643817 conn(0x7fabc80c4270 msgr2=0x7fabc80c4680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:59.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.324+0000 7fabcca99640 1 -- 172.21.15.116:0/402643817 shutdown_connections 2024-04-25T00:14:59.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.324+0000 7fabcca99640 1 -- 172.21.15.116:0/402643817 wait complete. 2024-04-25T00:14:59.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.328+0000 7fabcca99640 1 Processor -- start 2024-04-25T00:14:59.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.328+0000 7fabcca99640 1 -- start start 2024-04-25T00:14:59.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.328+0000 7fabcca99640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fabc806c760 0x7fabc81ef9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:59.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.328+0000 7fabcca99640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fabc80c6080 0x7fabc81eef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:59.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.328+0000 7fabcca99640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabc814f400 0x7fabc815ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:59.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.328+0000 7fabcca99640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fabc806f2d0 con 0x7fabc814f400 2024-04-25T00:14:59.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.328+0000 7fabcca99640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fabc806f150 con 0x7fabc806c760 2024-04-25T00:14:59.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.328+0000 7fabcca99640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fabc815e620 con 0x7fabc80c6080 2024-04-25T00:14:59.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.328+0000 7fabc6d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabc814f400 0x7fabc815ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:59.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.328+0000 7fabc6d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabc814f400 0x7fabc815ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40044/0 (socket says 172.21.15.116:40044) 2024-04-25T00:14:59.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.328+0000 7fabc6d9d640 1 -- 172.21.15.116:0/2101627460 learned_addr learned my addr 172.21.15.116:0/2101627460 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:14:59.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.328+0000 7fabc5d9b640 1 --2- 172.21.15.116:0/2101627460 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fabc80c6080 0x7fabc81eef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:59.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.328+0000 7fabc659c640 1 --2- 172.21.15.116:0/2101627460 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fabc806c760 0x7fabc81ef9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:59.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.328+0000 7fabc6d9d640 1 -- 172.21.15.116:0/2101627460 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fabc80c6080 msgr2=0x7fabc81eef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:59.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.328+0000 7fabc6d9d640 1 --2- 172.21.15.116:0/2101627460 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fabc80c6080 0x7fabc81eef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:59.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.328+0000 7fabc6d9d640 1 -- 172.21.15.116:0/2101627460 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fabc806c760 msgr2=0x7fabc81ef9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:59.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.328+0000 7fabc6d9d640 1 --2- 172.21.15.116:0/2101627460 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fabc806c760 0x7fabc81ef9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:59.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.328+0000 7fabc6d9d640 1 -- 172.21.15.116:0/2101627460 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabc8073190 con 0x7fabc814f400 2024-04-25T00:14:59.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.328+0000 7fabc6d9d640 1 --2- 172.21.15.116:0/2101627460 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabc814f400 0x7fabc815ac50 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fabc000cb40 tx=0x7fabc000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:59.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.328+0000 7fabbeffd640 1 -- 172.21.15.116:0/2101627460 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabc0017070 con 0x7fabc814f400 2024-04-25T00:14:59.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.328+0000 7fabbeffd640 1 -- 172.21.15.116:0/2101627460 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabc000aca0 con 0x7fabc814f400 2024-04-25T00:14:59.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.328+0000 7fabbeffd640 1 -- 172.21.15.116:0/2101627460 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabc0012620 con 0x7fabc814f400 2024-04-25T00:14:59.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.328+0000 7fabcca99640 1 -- 172.21.15.116:0/2101627460 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabc8073470 con 0x7fabc814f400 2024-04-25T00:14:59.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.328+0000 7fabcca99640 1 -- 172.21.15.116:0/2101627460 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabc815b190 con 0x7fabc814f400 2024-04-25T00:14:59.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.328+0000 7fabcca99640 1 -- 172.21.15.116:0/2101627460 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabc8068a60 con 0x7fabc814f400 2024-04-25T00:14:59.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.328+0000 7fabbeffd640 1 -- 172.21.15.116:0/2101627460 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fabc001a070 con 0x7fabc814f400 2024-04-25T00:14:59.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.332+0000 7fabbeffd640 1 --2- 172.21.15.116:0/2101627460 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faba80678c0 0x7faba8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:59.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.332+0000 7fabbeffd640 1 -- 172.21.15.116:0/2101627460 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 8300+0+0 (secure 0 0 0) 0x7fabc00867c0 con 0x7fabc814f400 2024-04-25T00:14:59.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.332+0000 7fabc659c640 1 --2- 172.21.15.116:0/2101627460 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faba80678c0 0x7faba8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:59.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.332+0000 7fabc659c640 1 --2- 172.21.15.116:0/2101627460 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faba80678c0 0x7faba8069d80 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fabb80b0000 tx=0x7fabb800c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:59.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.336+0000 7fabbeffd640 1 -- 172.21.15.116:0/2101627460 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fabc0021020 con 0x7fabc814f400 2024-04-25T00:14:59.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.484+0000 7f1777d56640 1 -- 172.21.15.116:0/2982466464 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f17700c3640 con 0x7f17700a94a0 2024-04-25T00:14:59.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.484+0000 7f17667fc640 1 -- 172.21.15.116:0/2982466464 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) ==== 95+0+4844 (secure 0 0 0) 0x7f1760059890 con 0x7f17700a94a0 2024-04-25T00:14:59.485 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:14:59.485 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:14:58.858430+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5273":{"gid":5273,"name":"d","rank":0,"incarnation":102,"state":"up:replay","state_seq":1,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:14:58.851744+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5791},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5791":{"gid":5791,"name":"b","rank":0,"incarnation":98,"state":"up:active","state_seq":23,"addr":"172.21.15.174:6837/3180024481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3180024481},{"type":"v1","addr":"172.21.15.174:6837","nonce":3180024481}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5791,"qdb_cluster":[5791]},"id":2}]} 2024-04-25T00:14:59.486 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 102 2024-04-25T00:14:59.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.488+0000 7f1777d56640 1 -- 172.21.15.116:0/2982466464 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f17540678c0 msgr2=0x7f1754069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:59.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.488+0000 7f1777d56640 1 --2- 172.21.15.116:0/2982466464 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f17540678c0 0x7f1754069d80 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f17680ad000 tx=0x7f17680a6000 comp rx=0 tx=0).stop 2024-04-25T00:14:59.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.488+0000 7f1777d56640 1 -- 172.21.15.116:0/2982466464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17700a94a0 msgr2=0x7f17700c0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:59.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.488+0000 7f1777d56640 1 --2- 172.21.15.116:0/2982466464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17700a94a0 0x7f17700c0690 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f176000c420 tx=0x7f176000c8e0 comp rx=0 tx=0).stop 2024-04-25T00:14:59.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.488+0000 7f1777d56640 1 -- 172.21.15.116:0/2982466464 shutdown_connections 2024-04-25T00:14:59.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.488+0000 7f1777d56640 1 --2- 172.21.15.116:0/2982466464 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f17540678c0 0x7f1754069d80 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:59.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.488+0000 7f1777d56640 1 --2- 172.21.15.116:0/2982466464 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f177014c720 0x7f177013f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:59.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.488+0000 7f1777d56640 1 --2- 172.21.15.116:0/2982466464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f17700a94a0 0x7f17700c0690 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:59.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.488+0000 7f1777d56640 1 --2- 172.21.15.116:0/2982466464 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1770071680 0x7f17700c64f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:59.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.488+0000 7f1777d56640 1 -- 172.21.15.116:0/2982466464 >> 172.21.15.116:0/2982466464 conn(0x7f17700c3b00 msgr2=0x7f1770150290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:59.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.488+0000 7f1777d56640 1 -- 172.21.15.116:0/2982466464 shutdown_connections 2024-04-25T00:14:59.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.488+0000 7f1777d56640 1 -- 172.21.15.116:0/2982466464 wait complete. 2024-04-25T00:14:59.509 INFO:tasks.mds_thrash.fs.[a]:mds.c down, removed from mdsmap 2024-04-25T00:14:59.509 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-04-25T00:14:59.509 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:14:59.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.540+0000 7fabcca99640 1 -- 172.21.15.116:0/2101627460 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fabc8073640 con 0x7fabc814f400 2024-04-25T00:14:59.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.540+0000 7fabbeffd640 1 -- 172.21.15.116:0/2101627460 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) ==== 95+0+4844 (secure 0 0 0) 0x7fabc0056350 con 0x7fabc814f400 2024-04-25T00:14:59.542 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:14:59.542 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:14:58.858430+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5273":{"gid":5273,"name":"d","rank":0,"incarnation":102,"state":"up:replay","state_seq":1,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:14:58.851744+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5791},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5791":{"gid":5791,"name":"b","rank":0,"incarnation":98,"state":"up:active","state_seq":23,"addr":"172.21.15.174:6837/3180024481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3180024481},{"type":"v1","addr":"172.21.15.174:6837","nonce":3180024481}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5791,"qdb_cluster":[5791]},"id":2}]} 2024-04-25T00:14:59.542 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 102 2024-04-25T00:14:59.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.544+0000 7fabcca99640 1 -- 172.21.15.116:0/2101627460 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faba80678c0 msgr2=0x7faba8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:59.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.544+0000 7fabcca99640 1 --2- 172.21.15.116:0/2101627460 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faba80678c0 0x7faba8069d80 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fabb80b0000 tx=0x7fabb800c040 comp rx=0 tx=0).stop 2024-04-25T00:14:59.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.544+0000 7fabcca99640 1 -- 172.21.15.116:0/2101627460 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabc814f400 msgr2=0x7fabc815ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:59.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.544+0000 7fabcca99640 1 --2- 172.21.15.116:0/2101627460 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabc814f400 0x7fabc815ac50 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fabc000cb40 tx=0x7fabc000a060 comp rx=0 tx=0).stop 2024-04-25T00:14:59.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.544+0000 7fabcca99640 1 -- 172.21.15.116:0/2101627460 shutdown_connections 2024-04-25T00:14:59.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.544+0000 7fabcca99640 1 --2- 172.21.15.116:0/2101627460 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faba80678c0 0x7faba8069d80 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:59.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.544+0000 7fabcca99640 1 --2- 172.21.15.116:0/2101627460 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fabc814f400 0x7fabc815ac50 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:59.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.544+0000 7fabcca99640 1 --2- 172.21.15.116:0/2101627460 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fabc80c6080 0x7fabc81eef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:59.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.544+0000 7fabcca99640 1 --2- 172.21.15.116:0/2101627460 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fabc806c760 0x7fabc81ef9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:59.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.544+0000 7fabcca99640 1 -- 172.21.15.116:0/2101627460 >> 172.21.15.116:0/2101627460 conn(0x7fabc80c4270 msgr2=0x7fabc80c4650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:59.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.544+0000 7fabcca99640 1 -- 172.21.15.116:0/2101627460 shutdown_connections 2024-04-25T00:14:59.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.544+0000 7fabcca99640 1 -- 172.21.15.116:0/2101627460 wait complete. 2024-04-25T00:14:59.560 INFO:tasks.mds_thrash.fs.[b]:mds.b has gained rank=0, replacing gid=6582 2024-04-25T00:14:59.560 INFO:tasks.mds_thrash.fs.[b]:waiting for 8 secs before reviving mds.a 2024-04-25T00:14:59.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.624+0000 7f9eac756640 1 Processor -- start 2024-04-25T00:14:59.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.624+0000 7f9eac756640 1 -- start start 2024-04-25T00:14:59.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.628+0000 7f9eac756640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9ea40acef0 0x7f9ea41541d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:59.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.628+0000 7f9eac756640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9ea40c32d0 0x7f9ea40c3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:59.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.628+0000 7f9eac756640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ea40a9ef0 con 0x7f9ea40ad2d0 2024-04-25T00:14:59.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.628+0000 7f9eac756640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f9ea40c2b60 con 0x7f9ea40c32d0 2024-04-25T00:14:59.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.628+0000 7f9eac756640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f9ea40c2e40 con 0x7f9ea40acef0 2024-04-25T00:14:59.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.628+0000 7f9ea9cf1640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f9ea40ad2d0 0x7f9ea40a7d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54638/0 (socket says 172.21.15.116:54638) 2024-04-25T00:14:59.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.628+0000 7f9ea9cf1640 1 -- 172.21.15.116:0/1833423090 learned_addr learned my addr 172.21.15.116:0/1833423090 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:14:59.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.628+0000 7f9eaacf3640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9ea40c32d0 0x7f9ea40c3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:59.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.628+0000 7f9eaa4f2640 1 --2- 172.21.15.116:0/1833423090 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9ea40acef0 0x7f9ea41541d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:59.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.628+0000 7f9ea94f0640 1 -- 172.21.15.116:0/1833423090 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4129834502 0 0) 0x7f9ea40a9ef0 con 0x7f9ea40ad2d0 2024-04-25T00:14:59.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.628+0000 7f9eaa4f2640 1 -- 172.21.15.116:0/1833423090 >> v2:172.21.15.116:3300/0 conn(0x7f9ea40c32d0 msgr2=0x7f9ea40c3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:14:59.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.628+0000 7f9eaa4f2640 1 --2- 172.21.15.116:0/1833423090 >> v2:172.21.15.116:3300/0 conn(0x7f9ea40c32d0 0x7f9ea40c3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:59.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.628+0000 7f9eaa4f2640 1 -- 172.21.15.116:0/1833423090 >> v1:172.21.15.116:6789/0 conn(0x7f9ea40ad2d0 legacy=0x7f9ea40a7d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:59.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.628+0000 7f9eaa4f2640 1 -- 172.21.15.116:0/1833423090 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ea40aa1d0 con 0x7f9ea40acef0 2024-04-25T00:14:59.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.628+0000 7f9eaa4f2640 1 --2- 172.21.15.116:0/1833423090 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9ea40acef0 0x7f9ea41541d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9e940097c0 tx=0x7f9e9401c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3339deb7ebb34108 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:59.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.628+0000 7f9ea94f0640 1 -- 172.21.15.116:0/1833423090 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e940b0070 con 0x7f9ea40acef0 2024-04-25T00:14:59.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.632+0000 7f9ea94f0640 1 -- 172.21.15.116:0/1833423090 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e940ab600 con 0x7f9ea40acef0 2024-04-25T00:14:59.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.632+0000 7f9ea94f0640 1 -- 172.21.15.116:0/1833423090 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e940ab9d0 con 0x7f9ea40acef0 2024-04-25T00:14:59.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.632+0000 7f9eac756640 1 -- 172.21.15.116:0/1833423090 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9ea40acef0 msgr2=0x7f9ea41541d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:59.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.632+0000 7f9eac756640 1 --2- 172.21.15.116:0/1833423090 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9ea40acef0 0x7f9ea41541d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9e940097c0 tx=0x7f9e9401c410 comp rx=0 tx=0).stop 2024-04-25T00:14:59.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.632+0000 7f9eac756640 1 -- 172.21.15.116:0/1833423090 shutdown_connections 2024-04-25T00:14:59.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.632+0000 7f9eac756640 1 --2- 172.21.15.116:0/1833423090 >> v2:172.21.15.116:3300/0 conn(0x7f9ea40c32d0 0x7f9ea40c3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:59.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.632+0000 7f9eac756640 1 --2- 172.21.15.116:0/1833423090 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9ea40acef0 0x7f9ea41541d0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:59.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.632+0000 7f9eac756640 1 -- 172.21.15.116:0/1833423090 >> 172.21.15.116:0/1833423090 conn(0x7f9ea40c4200 msgr2=0x7f9ea40c4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:59.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.632+0000 7f9eac756640 1 -- 172.21.15.116:0/1833423090 shutdown_connections 2024-04-25T00:14:59.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.632+0000 7f9eac756640 1 -- 172.21.15.116:0/1833423090 wait complete. 2024-04-25T00:14:59.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.636+0000 7f9eac756640 1 Processor -- start 2024-04-25T00:14:59.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.636+0000 7f9eac756640 1 -- start start 2024-04-25T00:14:59.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.636+0000 7f9eac756640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ea40acef0 0x7f9ea41ee040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:59.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.636+0000 7f9eac756640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9ea40ad2d0 0x7f9ea415c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:59.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.636+0000 7f9eac756640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ea40c32d0 0x7f9ea4072b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:59.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.636+0000 7f9eac756640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9ea41609e0 con 0x7f9ea40acef0 2024-04-25T00:14:59.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.636+0000 7f9eac756640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9ea4160860 con 0x7f9ea40c32d0 2024-04-25T00:14:59.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.636+0000 7f9eac756640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f9ea4160b60 con 0x7f9ea40ad2d0 2024-04-25T00:14:59.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.636+0000 7f9eaa4f2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ea40acef0 0x7f9ea41ee040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:59.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.636+0000 7f9eaa4f2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ea40acef0 0x7f9ea41ee040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40060/0 (socket says 172.21.15.116:40060) 2024-04-25T00:14:59.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.636+0000 7f9ea9cf1640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9ea40ad2d0 0x7f9ea415c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:59.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.636+0000 7f9eaa4f2640 1 -- 172.21.15.116:0/2802860382 learned_addr learned my addr 172.21.15.116:0/2802860382 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:14:59.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.636+0000 7f9eaacf3640 1 --2- 172.21.15.116:0/2802860382 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ea40c32d0 0x7f9ea4072b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:59.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.636+0000 7f9ea9cf1640 1 -- 172.21.15.116:0/2802860382 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ea40c32d0 msgr2=0x7f9ea4072b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:59.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.636+0000 7f9ea9cf1640 1 --2- 172.21.15.116:0/2802860382 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ea40c32d0 0x7f9ea4072b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:59.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.636+0000 7f9ea9cf1640 1 -- 172.21.15.116:0/2802860382 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ea40acef0 msgr2=0x7f9ea41ee040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:59.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.636+0000 7f9ea9cf1640 1 --2- 172.21.15.116:0/2802860382 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ea40acef0 0x7f9ea41ee040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:59.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.636+0000 7f9ea9cf1640 1 -- 172.21.15.116:0/2802860382 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ea41574c0 con 0x7f9ea40ad2d0 2024-04-25T00:14:59.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.636+0000 7f9ea9cf1640 1 --2- 172.21.15.116:0/2802860382 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9ea40ad2d0 0x7f9ea415c680 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9ea000c440 tx=0x7f9ea000c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:59.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.636+0000 7f9e9affd640 1 -- 172.21.15.116:0/2802860382 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea00074e0 con 0x7f9ea40ad2d0 2024-04-25T00:14:59.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.636+0000 7f9eac756640 1 -- 172.21.15.116:0/2802860382 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ea41577a0 con 0x7f9ea40ad2d0 2024-04-25T00:14:59.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.636+0000 7f9e9affd640 1 -- 172.21.15.116:0/2802860382 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ea00047f0 con 0x7f9ea40ad2d0 2024-04-25T00:14:59.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.636+0000 7f9e9affd640 1 -- 172.21.15.116:0/2802860382 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ea0004be0 con 0x7f9ea40ad2d0 2024-04-25T00:14:59.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.636+0000 7f9eac756640 1 -- 172.21.15.116:0/2802860382 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ea415cbc0 con 0x7f9ea40ad2d0 2024-04-25T00:14:59.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.636+0000 7f9eac756640 1 -- 172.21.15.116:0/2802860382 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ea4068a60 con 0x7f9ea40ad2d0 2024-04-25T00:14:59.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.636+0000 7f9e9affd640 1 -- 172.21.15.116:0/2802860382 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f9ea0011070 con 0x7f9ea40ad2d0 2024-04-25T00:14:59.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.640+0000 7f9e9affd640 1 --2- 172.21.15.116:0/2802860382 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9e880678c0 0x7f9e88069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:14:59.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.640+0000 7f9e9affd640 1 -- 172.21.15.116:0/2802860382 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(47..47 src has 1..47) ==== 8300+0+0 (secure 0 0 0) 0x7f9ea00862e0 con 0x7f9ea40ad2d0 2024-04-25T00:14:59.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.640+0000 7f9eaa4f2640 1 --2- 172.21.15.116:0/2802860382 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9e880678c0 0x7f9e88069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:14:59.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.640+0000 7f9eaa4f2640 1 --2- 172.21.15.116:0/2802860382 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9e880678c0 0x7f9e88069d80 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f9e9401a040 tx=0x7f9e940a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:14:59.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.644+0000 7f9e9affd640 1 -- 172.21.15.116:0/2802860382 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f9ea0055e50 con 0x7f9ea40ad2d0 2024-04-25T00:14:59.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.832+0000 7f9eac756640 1 -- 172.21.15.116:0/2802860382 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9ea40a9f20 con 0x7f9ea40ad2d0 2024-04-25T00:14:59.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.836+0000 7f9e9affd640 1 -- 172.21.15.116:0/2802860382 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) ==== 95+0+4844 (secure 0 0 0) 0x7f9ea005a150 con 0x7f9ea40ad2d0 2024-04-25T00:14:59.835 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:14:59.835 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:14:58.858430+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5273":{"gid":5273,"name":"d","rank":0,"incarnation":102,"state":"up:replay","state_seq":1,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:14:58.851744+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5791},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5791":{"gid":5791,"name":"b","rank":0,"incarnation":98,"state":"up:active","state_seq":23,"addr":"172.21.15.174:6837/3180024481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3180024481},{"type":"v1","addr":"172.21.15.174:6837","nonce":3180024481}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5791,"qdb_cluster":[5791]},"id":2}]} 2024-04-25T00:14:59.835 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 102 2024-04-25T00:14:59.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.836+0000 7f9eac756640 1 -- 172.21.15.116:0/2802860382 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9e880678c0 msgr2=0x7f9e88069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:59.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.836+0000 7f9eac756640 1 --2- 172.21.15.116:0/2802860382 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9e880678c0 0x7f9e88069d80 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f9e9401a040 tx=0x7f9e940a6000 comp rx=0 tx=0).stop 2024-04-25T00:14:59.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.836+0000 7f9eac756640 1 -- 172.21.15.116:0/2802860382 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9ea40ad2d0 msgr2=0x7f9ea415c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:14:59.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.836+0000 7f9eac756640 1 --2- 172.21.15.116:0/2802860382 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9ea40ad2d0 0x7f9ea415c680 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9ea000c440 tx=0x7f9ea000c900 comp rx=0 tx=0).stop 2024-04-25T00:14:59.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.836+0000 7f9eac756640 1 -- 172.21.15.116:0/2802860382 shutdown_connections 2024-04-25T00:14:59.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.836+0000 7f9eac756640 1 --2- 172.21.15.116:0/2802860382 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9e880678c0 0x7f9e88069d80 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:59.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.836+0000 7f9eac756640 1 --2- 172.21.15.116:0/2802860382 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9ea40c32d0 0x7f9ea4072b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:59.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.836+0000 7f9eac756640 1 --2- 172.21.15.116:0/2802860382 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9ea40ad2d0 0x7f9ea415c680 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:59.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.836+0000 7f9eac756640 1 --2- 172.21.15.116:0/2802860382 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ea40acef0 0x7f9ea41ee040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:14:59.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.836+0000 7f9eac756640 1 -- 172.21.15.116:0/2802860382 >> 172.21.15.116:0/2802860382 conn(0x7f9ea40c4200 msgr2=0x7f9ea40aae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:14:59.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.836+0000 7f9eac756640 1 -- 172.21.15.116:0/2802860382 shutdown_connections 2024-04-25T00:14:59.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:14:59.836+0000 7f9eac756640 1 -- 172.21.15.116:0/2802860382 wait complete. 2024-04-25T00:14:59.854 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 102, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:04.678737+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 102, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5273": { "addr": "172.21.15.174:6835/4215220222", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6834", "nonce": 4215220222, "type": "v2" }, { "addr": "172.21.15.174:6835", "nonce": 4215220222, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5273, "incarnation": 102, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 47, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T00:14:58.858430+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5273 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:18.825887+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 101, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5791": { "addr": "172.21.15.174:6837/3180024481", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6836", "nonce": 3180024481, "type": "v2" }, { "addr": "172.21.15.174:6837", "nonce": 3180024481, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5791, "incarnation": 98, "join_fscid": -1, "name": "b", "rank": 0, "state": "up:active", "state_seq": 23 } }, "last_failure": 0, "last_failure_osd_epoch": 46, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-04-25T00:14:58.851744+0000", "qdb_cluster": [ 5791 ], "qdb_leader": 5791, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5791 } } } ], "standbys": [] } 2024-04-25T00:15:01.855 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:15:01.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:01.992+0000 7f31e5582640 1 Processor -- start 2024-04-25T00:15:01.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:01.992+0000 7f31e5582640 1 -- start start 2024-04-25T00:15:01.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:01.992+0000 7f31e5582640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f31e0155b20 0x7f31e0152c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:01.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:01.992+0000 7f31e5582640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31e00a9e60 con 0x7f31e014c700 2024-04-25T00:15:01.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:01.992+0000 7f31e5582640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31e00c3ef0 con 0x7f31e00aacd0 2024-04-25T00:15:01.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:01.992+0000 7f31e5582640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f31e01527b0 con 0x7f31e0155b20 2024-04-25T00:15:01.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:01.992+0000 7f31deffd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f31e014c700 0x7f31e00c5090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54640/0 (socket says 172.21.15.116:54640) 2024-04-25T00:15:01.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:01.992+0000 7f31deffd640 1 -- 172.21.15.116:0/1038195338 learned_addr learned my addr 172.21.15.116:0/1038195338 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:15:01.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:01.992+0000 7f31d67fc640 1 --2- 172.21.15.116:0/1038195338 >> v2:172.21.15.174:3300/0 conn(0x7f31e0155b20 0x7f31e0152c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:01.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:01.992+0000 7f31de7fc640 1 -- 172.21.15.116:0/1038195338 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1619383071 0 0) 0x7f31e00a9e60 con 0x7f31e014c700 2024-04-25T00:15:01.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:01.992+0000 7f31de7fc640 1 -- 172.21.15.116:0/1038195338 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f31cc003530 con 0x7f31e014c700 2024-04-25T00:15:01.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:01.992+0000 7f31de7fc640 1 -- 172.21.15.116:0/1038195338 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3510464661 0 0) 0x7f31e00c3ef0 con 0x7f31e00aacd0 2024-04-25T00:15:01.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:01.992+0000 7f31de7fc640 1 -- 172.21.15.116:0/1038195338 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f31e00a9e60 con 0x7f31e00aacd0 2024-04-25T00:15:01.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:01.992+0000 7f31d67fc640 1 -- 172.21.15.116:0/1038195338 >> v1:172.21.15.174:6789/0 conn(0x7f31e00aacd0 legacy=0x7f31e015ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:01.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:01.992+0000 7f31d67fc640 1 -- 172.21.15.116:0/1038195338 >> v1:172.21.15.116:6789/0 conn(0x7f31e014c700 legacy=0x7f31e00c5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:01.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:01.992+0000 7f31d67fc640 1 -- 172.21.15.116:0/1038195338 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31e00c41d0 con 0x7f31e0155b20 2024-04-25T00:15:01.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:01.996+0000 7f31d67fc640 1 --2- 172.21.15.116:0/1038195338 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31e0155b20 0x7f31e0152c40 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f31c80097c0 tx=0x7f31c801c380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c734f1cd2d0d2915 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:01.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:01.996+0000 7f31de7fc640 1 -- 172.21.15.116:0/1038195338 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31c80b0070 con 0x7f31e0155b20 2024-04-25T00:15:01.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:01.996+0000 7f31de7fc640 1 -- 172.21.15.116:0/1038195338 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31c80ab600 con 0x7f31e0155b20 2024-04-25T00:15:01.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:01.996+0000 7f31de7fc640 1 -- 172.21.15.116:0/1038195338 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31c80ab9d0 con 0x7f31e0155b20 2024-04-25T00:15:01.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:01.996+0000 7f31e5582640 1 -- 172.21.15.116:0/1038195338 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31e0155b20 msgr2=0x7f31e0152c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:01.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:01.996+0000 7f31e5582640 1 --2- 172.21.15.116:0/1038195338 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31e0155b20 0x7f31e0152c40 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f31c80097c0 tx=0x7f31c801c380 comp rx=0 tx=0).stop 2024-04-25T00:15:01.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:01.996+0000 7f31e5582640 1 -- 172.21.15.116:0/1038195338 shutdown_connections 2024-04-25T00:15:01.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:01.996+0000 7f31e5582640 1 --2- 172.21.15.116:0/1038195338 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31e0155b20 0x7f31e0152c40 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:01.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:01.996+0000 7f31e5582640 1 -- 172.21.15.116:0/1038195338 >> 172.21.15.116:0/1038195338 conn(0x7f31e006c720 msgr2=0x7f31e006cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:01.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:01.996+0000 7f31e5582640 1 -- 172.21.15.116:0/1038195338 shutdown_connections 2024-04-25T00:15:01.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:01.996+0000 7f31e5582640 1 -- 172.21.15.116:0/1038195338 wait complete. 2024-04-25T00:15:01.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:02.000+0000 7f31e5582640 1 Processor -- start 2024-04-25T00:15:01.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:02.000+0000 7f31e5582640 1 -- start start 2024-04-25T00:15:02.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:02.000+0000 7f31e5582640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31e00aacd0 0x7f31e014aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:02.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:02.000+0000 7f31e5582640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f31e014c700 0x7f31e00ac470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:02.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:02.000+0000 7f31e5582640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31e0155b20 0x7f31e01eece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:02.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:02.000+0000 7f31e5582640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f31e006be30 con 0x7f31e0155b20 2024-04-25T00:15:02.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:02.000+0000 7f31e5582640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f31e006bcb0 con 0x7f31e00aacd0 2024-04-25T00:15:02.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:02.000+0000 7f31e5582640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f31e006bfb0 con 0x7f31e014c700 2024-04-25T00:15:02.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:02.000+0000 7f31df7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31e0155b20 0x7f31e01eece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:02.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:02.000+0000 7f31df7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31e0155b20 0x7f31e01eece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40066/0 (socket says 172.21.15.116:40066) 2024-04-25T00:15:02.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:02.000+0000 7f31df7fe640 1 -- 172.21.15.116:0/2823550593 learned_addr learned my addr 172.21.15.116:0/2823550593 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:15:02.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:02.000+0000 7f31deffd640 1 --2- 172.21.15.116:0/2823550593 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31e00aacd0 0x7f31e014aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:02.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:02.000+0000 7f31d67fc640 1 --2- 172.21.15.116:0/2823550593 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f31e014c700 0x7f31e00ac470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:02.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:02.000+0000 7f31df7fe640 1 -- 172.21.15.116:0/2823550593 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f31e014c700 msgr2=0x7f31e00ac470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:02.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:02.000+0000 7f31df7fe640 1 --2- 172.21.15.116:0/2823550593 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f31e014c700 0x7f31e00ac470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:02.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:02.000+0000 7f31df7fe640 1 -- 172.21.15.116:0/2823550593 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31e00aacd0 msgr2=0x7f31e014aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:02.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:02.000+0000 7f31df7fe640 1 --2- 172.21.15.116:0/2823550593 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31e00aacd0 0x7f31e014aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:02.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:02.000+0000 7f31df7fe640 1 -- 172.21.15.116:0/2823550593 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31e0145310 con 0x7f31e0155b20 2024-04-25T00:15:02.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:02.000+0000 7f31df7fe640 1 --2- 172.21.15.116:0/2823550593 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31e0155b20 0x7f31e01eece0 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f31d800c3c0 tx=0x7f31d800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:02.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:02.000+0000 7f31d7fff640 1 -- 172.21.15.116:0/2823550593 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31d8017070 con 0x7f31e0155b20 2024-04-25T00:15:02.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:02.000+0000 7f31d7fff640 1 -- 172.21.15.116:0/2823550593 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31d80052d0 con 0x7f31e0155b20 2024-04-25T00:15:02.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:02.000+0000 7f31e5582640 1 -- 172.21.15.116:0/2823550593 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31e01455b0 con 0x7f31e0155b20 2024-04-25T00:15:02.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:02.000+0000 7f31d7fff640 1 -- 172.21.15.116:0/2823550593 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31d8003820 con 0x7f31e0155b20 2024-04-25T00:15:02.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:02.000+0000 7f31e5582640 1 -- 172.21.15.116:0/2823550593 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31e0146fa0 con 0x7f31e0155b20 2024-04-25T00:15:02.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:02.000+0000 7f31e5582640 1 -- 172.21.15.116:0/2823550593 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31e0068a60 con 0x7f31e0155b20 2024-04-25T00:15:02.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:02.004+0000 7f31d7fff640 1 -- 172.21.15.116:0/2823550593 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f31d801a070 con 0x7f31e0155b20 2024-04-25T00:15:02.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:02.004+0000 7f31d7fff640 1 --2- 172.21.15.116:0/2823550593 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f31c00678c0 0x7f31c0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:02.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:02.004+0000 7f31d7fff640 1 -- 172.21.15.116:0/2823550593 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 8300+0+0 (secure 0 0 0) 0x7f31d8085900 con 0x7f31e0155b20 2024-04-25T00:15:02.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:02.004+0000 7f31deffd640 1 --2- 172.21.15.116:0/2823550593 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f31c00678c0 0x7f31c0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:02.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:02.004+0000 7f31deffd640 1 --2- 172.21.15.116:0/2823550593 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f31c00678c0 0x7f31c0069d80 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f31d00012f0 tx=0x7f31d0001aa0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:02.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:02.012+0000 7f31d7fff640 1 -- 172.21.15.116:0/2823550593 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f31d8005480 con 0x7f31e0155b20 2024-04-25T00:15:02.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:02.244+0000 7f31e5582640 1 -- 172.21.15.116:0/2823550593 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f31e00759d0 con 0x7f31e0155b20 2024-04-25T00:15:02.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:02.244+0000 7f31d7fff640 1 -- 172.21.15.116:0/2823550593 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) ==== 95+0+4852 (secure 0 0 0) 0x7f31d8055410 con 0x7f31e0155b20 2024-04-25T00:15:02.245 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:15:02.245 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":105,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:15:01.877414+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5273":{"gid":5273,"name":"d","rank":0,"incarnation":102,"state":"up:active","state_seq":49,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5273,"qdb_cluster":[5273]},"id":1},{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:14:58.851744+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5791},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5791":{"gid":5791,"name":"b","rank":0,"incarnation":98,"state":"up:active","state_seq":23,"addr":"172.21.15.174:6837/3180024481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3180024481},{"type":"v1","addr":"172.21.15.174:6837","nonce":3180024481}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5791,"qdb_cluster":[5791]},"id":2}]} 2024-04-25T00:15:02.245 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 105 2024-04-25T00:15:02.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:02.244+0000 7f31e5582640 1 -- 172.21.15.116:0/2823550593 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f31c00678c0 msgr2=0x7f31c0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:02.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:02.244+0000 7f31e5582640 1 --2- 172.21.15.116:0/2823550593 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f31c00678c0 0x7f31c0069d80 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f31d00012f0 tx=0x7f31d0001aa0 comp rx=0 tx=0).stop 2024-04-25T00:15:02.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:02.244+0000 7f31e5582640 1 -- 172.21.15.116:0/2823550593 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31e0155b20 msgr2=0x7f31e01eece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:02.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:02.244+0000 7f31e5582640 1 --2- 172.21.15.116:0/2823550593 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31e0155b20 0x7f31e01eece0 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f31d800c3c0 tx=0x7f31d800c880 comp rx=0 tx=0).stop 2024-04-25T00:15:02.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:02.244+0000 7f31e5582640 1 -- 172.21.15.116:0/2823550593 shutdown_connections 2024-04-25T00:15:02.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:02.248+0000 7f31e5582640 1 --2- 172.21.15.116:0/2823550593 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f31c00678c0 0x7f31c0069d80 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:02.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:02.248+0000 7f31e5582640 1 --2- 172.21.15.116:0/2823550593 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31e0155b20 0x7f31e01eece0 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:02.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:02.248+0000 7f31e5582640 1 --2- 172.21.15.116:0/2823550593 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f31e014c700 0x7f31e00ac470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:02.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:02.248+0000 7f31e5582640 1 --2- 172.21.15.116:0/2823550593 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31e00aacd0 0x7f31e014aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:02.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:02.248+0000 7f31e5582640 1 -- 172.21.15.116:0/2823550593 >> 172.21.15.116:0/2823550593 conn(0x7f31e006c720 msgr2=0x7f31e00c34d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:02.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:02.248+0000 7f31e5582640 1 -- 172.21.15.116:0/2823550593 shutdown_connections 2024-04-25T00:15:02.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:02.248+0000 7f31e5582640 1 -- 172.21.15.116:0/2823550593 wait complete. 2024-04-25T00:15:02.267 INFO:tasks.mds_thrash.fs.[a]:mds.d has gained rank=0, replacing gid=6495 2024-04-25T00:15:02.267 INFO:tasks.mds_thrash.fs.[a]:waiting for 1 secs before reviving mds.c 2024-04-25T00:15:02.520 INFO:tasks.ceph.mds.b.smithi174.stderr:2024-04-25T00:15:02.521+0000 7f6ca4125640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T00:15:03.269 INFO:tasks.mds_thrash.fs.[a]:reviving mds.c 2024-04-25T00:15:03.269 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-25T00:15:03.269 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-25T00:15:03.271 INFO:tasks.ceph.mds.c:Started 2024-04-25T00:15:03.272 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:15:03.337 INFO:tasks.ceph.mds.c.smithi116.stdout:starting mds.c at 2024-04-25T00:15:03.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.404+0000 7f2e3c65a640 1 Processor -- start 2024-04-25T00:15:03.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.404+0000 7f2e3c65a640 1 -- start start 2024-04-25T00:15:03.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.404+0000 7f2e3c65a640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f2e3414f400 0x7f2e340c2170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:03.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.404+0000 7f2e3c65a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2e3406e630 0x7f2e3406ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:03.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.404+0000 7f2e3c65a640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e3414f7e0 con 0x7f2e340c6080 2024-04-25T00:15:03.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.404+0000 7f2e3c65a640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f2e34152990 con 0x7f2e3406e630 2024-04-25T00:15:03.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.404+0000 7f2e3c65a640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f2e34152060 con 0x7f2e3414f400 2024-04-25T00:15:03.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.404+0000 7f2e3abf7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2e3406e630 0x7f2e3406ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:03.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.404+0000 7f2e3abf7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2e3406e630 0x7f2e3406ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40092/0 (socket says 172.21.15.116:40092) 2024-04-25T00:15:03.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.404+0000 7f2e3a3f6640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f2e3414f400 0x7f2e340c2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:03.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.404+0000 7f2e3abf7640 1 -- 172.21.15.116:0/1715553992 learned_addr learned my addr 172.21.15.116:0/1715553992 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:15:03.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.404+0000 7f2e3abf7640 1 -- 172.21.15.116:0/1715553992 >> v2:172.21.15.174:3300/0 conn(0x7f2e3414f400 msgr2=0x7f2e340c2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:03.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.404+0000 7f2e3abf7640 1 --2- 172.21.15.116:0/1715553992 >> v2:172.21.15.174:3300/0 conn(0x7f2e3414f400 0x7f2e340c2170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:03.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.404+0000 7f2e3abf7640 1 -- 172.21.15.116:0/1715553992 >> v1:172.21.15.174:6789/0 conn(0x7f2e340c6080 legacy=0x7f2e3406c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:03.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.404+0000 7f2e3abf7640 1 -- 172.21.15.116:0/1715553992 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e34152340 con 0x7f2e3406e630 2024-04-25T00:15:03.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.408+0000 7f2e3abf7640 1 --2- 172.21.15.116:0/1715553992 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e3406e630 0x7f2e3406ea80 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f2e340680c0 tx=0x7f2e2401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d57e2722448162f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:03.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.408+0000 7f2e393f4640 1 -- 172.21.15.116:0/1715553992 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e240b0070 con 0x7f2e3406e630 2024-04-25T00:15:03.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.408+0000 7f2e393f4640 1 -- 172.21.15.116:0/1715553992 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e240ab600 con 0x7f2e3406e630 2024-04-25T00:15:03.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.408+0000 7f2e393f4640 1 -- 172.21.15.116:0/1715553992 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e240ab9d0 con 0x7f2e3406e630 2024-04-25T00:15:03.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.408+0000 7f2e3c65a640 1 -- 172.21.15.116:0/1715553992 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e3406e630 msgr2=0x7f2e3406ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:03.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.408+0000 7f2e3c65a640 1 --2- 172.21.15.116:0/1715553992 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e3406e630 0x7f2e3406ea80 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f2e340680c0 tx=0x7f2e2401c490 comp rx=0 tx=0).stop 2024-04-25T00:15:03.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.408+0000 7f2e3c65a640 1 -- 172.21.15.116:0/1715553992 shutdown_connections 2024-04-25T00:15:03.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.408+0000 7f2e3c65a640 1 --2- 172.21.15.116:0/1715553992 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e3406e630 0x7f2e3406ea80 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:03.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.408+0000 7f2e3c65a640 1 --2- 172.21.15.116:0/1715553992 >> v2:172.21.15.174:3300/0 conn(0x7f2e3414f400 0x7f2e340c2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:03.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.408+0000 7f2e3c65a640 1 -- 172.21.15.116:0/1715553992 >> 172.21.15.116:0/1715553992 conn(0x7f2e340c4270 msgr2=0x7f2e340c4680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:03.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.408+0000 7f2e3c65a640 1 -- 172.21.15.116:0/1715553992 shutdown_connections 2024-04-25T00:15:03.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.408+0000 7f2e3c65a640 1 -- 172.21.15.116:0/1715553992 wait complete. 2024-04-25T00:15:03.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.408+0000 7f2e3c65a640 1 Processor -- start 2024-04-25T00:15:03.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.408+0000 7f2e3c65a640 1 -- start start 2024-04-25T00:15:03.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.412+0000 7f2e3c65a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e3406e630 0x7f2e3415a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:03.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.412+0000 7f2e3c65a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e340c6080 0x7f2e341ef760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:03.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.412+0000 7f2e3c65a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e3414f400 0x7f2e341eb530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:03.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.412+0000 7f2e3c65a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2e3406cab0 con 0x7f2e340c6080 2024-04-25T00:15:03.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.412+0000 7f2e3c65a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2e3406c930 con 0x7f2e3406e630 2024-04-25T00:15:03.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.412+0000 7f2e3c65a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f2e3415e620 con 0x7f2e3414f400 2024-04-25T00:15:03.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.412+0000 7f2e39bf5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e340c6080 0x7f2e341ef760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:03.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.412+0000 7f2e39bf5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e340c6080 0x7f2e341ef760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40096/0 (socket says 172.21.15.116:40096) 2024-04-25T00:15:03.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.412+0000 7f2e39bf5640 1 -- 172.21.15.116:0/3763798450 learned_addr learned my addr 172.21.15.116:0/3763798450 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:15:03.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.412+0000 7f2e3a3f6640 1 --2- 172.21.15.116:0/3763798450 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e3406e630 0x7f2e3415a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:03.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.412+0000 7f2e3abf7640 1 --2- 172.21.15.116:0/3763798450 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e3414f400 0x7f2e341eb530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:03.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.412+0000 7f2e39bf5640 1 -- 172.21.15.116:0/3763798450 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e3414f400 msgr2=0x7f2e341eb530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:03.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.412+0000 7f2e39bf5640 1 --2- 172.21.15.116:0/3763798450 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e3414f400 0x7f2e341eb530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:03.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.412+0000 7f2e39bf5640 1 -- 172.21.15.116:0/3763798450 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e3406e630 msgr2=0x7f2e3415a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:03.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.412+0000 7f2e39bf5640 1 --2- 172.21.15.116:0/3763798450 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e3406e630 0x7f2e3415a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:03.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.412+0000 7f2e39bf5640 1 -- 172.21.15.116:0/3763798450 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e341ef220 con 0x7f2e340c6080 2024-04-25T00:15:03.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.412+0000 7f2e3abf7640 1 --2- 172.21.15.116:0/3763798450 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e3414f400 0x7f2e341eb530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:15:03.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.412+0000 7f2e39bf5640 1 --2- 172.21.15.116:0/3763798450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e340c6080 0x7f2e341ef760 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f2e3000c450 tx=0x7f2e3000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:03.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.412+0000 7f2e2affd640 1 -- 172.21.15.116:0/3763798450 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e30016020 con 0x7f2e340c6080 2024-04-25T00:15:03.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.412+0000 7f2e2affd640 1 -- 172.21.15.116:0/3763798450 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e300052d0 con 0x7f2e340c6080 2024-04-25T00:15:03.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.412+0000 7f2e3c65a640 1 -- 172.21.15.116:0/3763798450 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e34071820 con 0x7f2e340c6080 2024-04-25T00:15:03.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.412+0000 7f2e2affd640 1 -- 172.21.15.116:0/3763798450 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e30007260 con 0x7f2e340c6080 2024-04-25T00:15:03.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.412+0000 7f2e3c65a640 1 -- 172.21.15.116:0/3763798450 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e34072080 con 0x7f2e340c6080 2024-04-25T00:15:03.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.412+0000 7f2e3c65a640 1 -- 172.21.15.116:0/3763798450 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e340719f0 con 0x7f2e340c6080 2024-04-25T00:15:03.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.412+0000 7f2e2affd640 1 -- 172.21.15.116:0/3763798450 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2e30007400 con 0x7f2e340c6080 2024-04-25T00:15:03.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.412+0000 7f2e2affd640 1 --2- 172.21.15.116:0/3763798450 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2e180678c0 0x7f2e18069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:03.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.412+0000 7f2e2affd640 1 -- 172.21.15.116:0/3763798450 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 8300+0+0 (secure 0 0 0) 0x7f2e30085fa0 con 0x7f2e340c6080 2024-04-25T00:15:03.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.412+0000 7f2e3a3f6640 1 --2- 172.21.15.116:0/3763798450 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2e180678c0 0x7f2e18069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:03.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.416+0000 7f2e3a3f6640 1 --2- 172.21.15.116:0/3763798450 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2e180678c0 0x7f2e18069d80 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c00a490 tx=0x7f2e2c00a860 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:03.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.416+0000 7f2e2affd640 1 -- 172.21.15.116:0/3763798450 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f2e340719f0 con 0x7f2e340c6080 2024-04-25T00:15:03.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.664+0000 7f2e3c65a640 1 -- 172.21.15.116:0/3763798450 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2e34068a60 con 0x7f2e340c6080 2024-04-25T00:15:03.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.664+0000 7f2e2affd640 1 -- 172.21.15.116:0/3763798450 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) ==== 95+0+4852 (secure 0 0 0) 0x7f2e30020020 con 0x7f2e340c6080 2024-04-25T00:15:03.665 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:15:03.665 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":105,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:15:01.877414+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5273":{"gid":5273,"name":"d","rank":0,"incarnation":102,"state":"up:active","state_seq":49,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5273,"qdb_cluster":[5273]},"id":1},{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:14:58.851744+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5791},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5791":{"gid":5791,"name":"b","rank":0,"incarnation":98,"state":"up:active","state_seq":23,"addr":"172.21.15.174:6837/3180024481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3180024481},{"type":"v1","addr":"172.21.15.174:6837","nonce":3180024481}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5791,"qdb_cluster":[5791]},"id":2}]} 2024-04-25T00:15:03.665 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 105 2024-04-25T00:15:03.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.664+0000 7f2e3c65a640 1 -- 172.21.15.116:0/3763798450 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2e180678c0 msgr2=0x7f2e18069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:03.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.664+0000 7f2e3c65a640 1 --2- 172.21.15.116:0/3763798450 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2e180678c0 0x7f2e18069d80 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f2e2c00a490 tx=0x7f2e2c00a860 comp rx=0 tx=0).stop 2024-04-25T00:15:03.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.664+0000 7f2e3c65a640 1 -- 172.21.15.116:0/3763798450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e340c6080 msgr2=0x7f2e341ef760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:03.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.664+0000 7f2e3c65a640 1 --2- 172.21.15.116:0/3763798450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e340c6080 0x7f2e341ef760 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f2e3000c450 tx=0x7f2e3000c910 comp rx=0 tx=0).stop 2024-04-25T00:15:03.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.664+0000 7f2e3c65a640 1 -- 172.21.15.116:0/3763798450 shutdown_connections 2024-04-25T00:15:03.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.664+0000 7f2e3c65a640 1 --2- 172.21.15.116:0/3763798450 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2e180678c0 0x7f2e18069d80 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:03.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.664+0000 7f2e3c65a640 1 --2- 172.21.15.116:0/3763798450 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e3414f400 0x7f2e341eb530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:03.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.664+0000 7f2e3c65a640 1 --2- 172.21.15.116:0/3763798450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e340c6080 0x7f2e341ef760 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:03.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.664+0000 7f2e3c65a640 1 --2- 172.21.15.116:0/3763798450 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e3406e630 0x7f2e3415a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:03.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.664+0000 7f2e3c65a640 1 -- 172.21.15.116:0/3763798450 >> 172.21.15.116:0/3763798450 conn(0x7f2e340c4270 msgr2=0x7f2e341500a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:03.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.664+0000 7f2e3c65a640 1 -- 172.21.15.116:0/3763798450 shutdown_connections 2024-04-25T00:15:03.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:03.664+0000 7f2e3c65a640 1 -- 172.21.15.116:0/3763798450 wait complete. 2024-04-25T00:15:03.683 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2024-04-25T00:15:03.826 INFO:tasks.ceph.mds.d.smithi174.stderr:2024-04-25T00:15:03.825+0000 7f0903533640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T00:15:05.684 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:15:05.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.804+0000 7fe02b1a7640 1 Processor -- start 2024-04-25T00:15:05.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.804+0000 7fe02b1a7640 1 -- start start 2024-04-25T00:15:05.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.804+0000 7fe02b1a7640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe024150d10 0x7fe0240c5010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:05.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.804+0000 7fe02b1a7640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fe0240c5550 0x7fe024153a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:05.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.804+0000 7fe02b1a7640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0240c4950 con 0x7fe024072ca0 2024-04-25T00:15:05.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.804+0000 7fe02b1a7640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fe0241546d0 con 0x7fe0240c5550 2024-04-25T00:15:05.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.804+0000 7fe02b1a7640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe0241510f0 con 0x7fe024150d10 2024-04-25T00:15:05.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.808+0000 7fe029744640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fe024072ca0 0x7fe024153fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:42262/0 (socket says 172.21.15.116:42262) 2024-04-25T00:15:05.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.808+0000 7fe029744640 1 -- 172.21.15.116:0/187328610 learned_addr learned my addr 172.21.15.116:0/187328610 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:15:05.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.808+0000 7fe023fff640 1 --2- 172.21.15.116:0/187328610 >> v2:172.21.15.174:3300/0 conn(0x7fe0240c5550 0x7fe024153a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:05.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.808+0000 7fe028f43640 1 --2- 172.21.15.116:0/187328610 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe024150d10 0x7fe0240c5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:05.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.808+0000 7fe0237fe640 1 -- 172.21.15.116:0/187328610 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2562029121 0 0) 0x7fe0240c4950 con 0x7fe024072ca0 2024-04-25T00:15:05.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.808+0000 7fe0237fe640 1 -- 172.21.15.116:0/187328610 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe014003530 con 0x7fe024072ca0 2024-04-25T00:15:05.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.808+0000 7fe023fff640 1 -- 172.21.15.116:0/187328610 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe024150d10 msgr2=0x7fe0240c5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:05.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.808+0000 7fe023fff640 1 --2- 172.21.15.116:0/187328610 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe024150d10 0x7fe0240c5010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:05.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.808+0000 7fe023fff640 1 -- 172.21.15.116:0/187328610 >> v1:172.21.15.174:6789/0 conn(0x7fe024072ca0 legacy=0x7fe024153fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:05.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.808+0000 7fe023fff640 1 -- 172.21.15.116:0/187328610 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe024157a20 con 0x7fe0240c5550 2024-04-25T00:15:05.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.808+0000 7fe023fff640 1 --2- 172.21.15.116:0/187328610 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe0240c5550 0x7fe024153a80 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fe0100097c0 tx=0x7fe01001c380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3dcb7353f8d414f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:05.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.808+0000 7fe0237fe640 1 -- 172.21.15.116:0/187328610 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0100b2070 con 0x7fe0240c5550 2024-04-25T00:15:05.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.808+0000 7fe0237fe640 1 -- 172.21.15.116:0/187328610 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0100ad680 con 0x7fe0240c5550 2024-04-25T00:15:05.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.808+0000 7fe0237fe640 1 -- 172.21.15.116:0/187328610 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0100ada30 con 0x7fe0240c5550 2024-04-25T00:15:05.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.808+0000 7fe02b1a7640 1 -- 172.21.15.116:0/187328610 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe0240c5550 msgr2=0x7fe024153a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:05.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.808+0000 7fe02b1a7640 1 --2- 172.21.15.116:0/187328610 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe0240c5550 0x7fe024153a80 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fe0100097c0 tx=0x7fe01001c380 comp rx=0 tx=0).stop 2024-04-25T00:15:05.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.812+0000 7fe02b1a7640 1 -- 172.21.15.116:0/187328610 shutdown_connections 2024-04-25T00:15:05.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.812+0000 7fe02b1a7640 1 --2- 172.21.15.116:0/187328610 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe0240c5550 0x7fe024153a80 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:05.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.812+0000 7fe02b1a7640 1 --2- 172.21.15.116:0/187328610 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe024150d10 0x7fe0240c5010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:05.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.812+0000 7fe02b1a7640 1 -- 172.21.15.116:0/187328610 >> 172.21.15.116:0/187328610 conn(0x7fe02406c720 msgr2=0x7fe02406cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:05.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.812+0000 7fe02b1a7640 1 -- 172.21.15.116:0/187328610 shutdown_connections 2024-04-25T00:15:05.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.812+0000 7fe02b1a7640 1 -- 172.21.15.116:0/187328610 wait complete. 2024-04-25T00:15:05.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.812+0000 7fe02b1a7640 1 Processor -- start 2024-04-25T00:15:05.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.812+0000 7fe02b1a7640 1 -- start start 2024-04-25T00:15:05.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.812+0000 7fe02b1a7640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe024072ca0 0x7fe0241eeaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:05.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.812+0000 7fe02b1a7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0240c5550 0x7fe0241f1e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:05.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.812+0000 7fe02b1a7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe024150d10 0x7fe02415c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:05.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.812+0000 7fe02b1a7640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe0240c2640 con 0x7fe0240c5550 2024-04-25T00:15:05.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.812+0000 7fe02b1a7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe0241576a0 con 0x7fe024150d10 2024-04-25T00:15:05.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.812+0000 7fe02b1a7640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe0240c27c0 con 0x7fe024072ca0 2024-04-25T00:15:05.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.812+0000 7fe023fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0240c5550 0x7fe0241f1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:05.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.812+0000 7fe023fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0240c5550 0x7fe0241f1e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44978/0 (socket says 172.21.15.116:44978) 2024-04-25T00:15:05.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.812+0000 7fe023fff640 1 -- 172.21.15.116:0/390339071 learned_addr learned my addr 172.21.15.116:0/390339071 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:15:05.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.812+0000 7fe029744640 1 --2- 172.21.15.116:0/390339071 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe024150d10 0x7fe02415c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:05.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.812+0000 7fe028f43640 1 --2- 172.21.15.116:0/390339071 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe024072ca0 0x7fe0241eeaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:05.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.812+0000 7fe023fff640 1 -- 172.21.15.116:0/390339071 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe024072ca0 msgr2=0x7fe0241eeaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:05.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.812+0000 7fe023fff640 1 --2- 172.21.15.116:0/390339071 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe024072ca0 0x7fe0241eeaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:05.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.812+0000 7fe023fff640 1 -- 172.21.15.116:0/390339071 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe024150d10 msgr2=0x7fe02415c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:05.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.812+0000 7fe023fff640 1 --2- 172.21.15.116:0/390339071 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe024150d10 0x7fe02415c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:05.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.812+0000 7fe023fff640 1 -- 172.21.15.116:0/390339071 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0240acef0 con 0x7fe0240c5550 2024-04-25T00:15:05.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.812+0000 7fe023fff640 1 --2- 172.21.15.116:0/390339071 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0240c5550 0x7fe0241f1e80 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fe0100097c0 tx=0x7fe010019120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:05.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.812+0000 7fe0217fa640 1 -- 172.21.15.116:0/390339071 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0100b2040 con 0x7fe0240c5550 2024-04-25T00:15:05.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.812+0000 7fe0217fa640 1 -- 172.21.15.116:0/390339071 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0100bb070 con 0x7fe0240c5550 2024-04-25T00:15:05.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.812+0000 7fe0217fa640 1 -- 172.21.15.116:0/390339071 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0100ad830 con 0x7fe0240c5550 2024-04-25T00:15:05.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.812+0000 7fe02b1a7640 1 -- 172.21.15.116:0/390339071 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0241ee010 con 0x7fe0240c5550 2024-04-25T00:15:05.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.812+0000 7fe02b1a7640 1 -- 172.21.15.116:0/390339071 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0241ee5d0 con 0x7fe0240c5550 2024-04-25T00:15:05.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.812+0000 7fe02b1a7640 1 -- 172.21.15.116:0/390339071 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe024068a60 con 0x7fe0240c5550 2024-04-25T00:15:05.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.816+0000 7fe0217fa640 1 -- 172.21.15.116:0/390339071 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe0100ad9d0 con 0x7fe0240c5550 2024-04-25T00:15:05.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.816+0000 7fe0217fa640 1 --2- 172.21.15.116:0/390339071 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe0080678f0 0x7fe008069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:05.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.816+0000 7fe0217fa640 1 -- 172.21.15.116:0/390339071 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 8300+0+0 (secure 0 0 0) 0x7fe010120270 con 0x7fe0240c5550 2024-04-25T00:15:05.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.816+0000 7fe028f43640 1 --2- 172.21.15.116:0/390339071 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe0080678f0 0x7fe008069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:05.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.816+0000 7fe028f43640 1 --2- 172.21.15.116:0/390339071 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe0080678f0 0x7fe008069db0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fe024072a00 tx=0x7fe018008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:05.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:05.820+0000 7fe0217fa640 1 -- 172.21.15.116:0/390339071 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fe010016090 con 0x7fe0240c5550 2024-04-25T00:15:06.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:06.048+0000 7fe02b1a7640 1 -- 172.21.15.116:0/390339071 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe024072a80 con 0x7fe0240c5550 2024-04-25T00:15:06.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:06.052+0000 7fe0217fa640 1 -- 172.21.15.116:0/390339071 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) ==== 95+0+5680 (secure 0 0 0) 0x7fe0100f03d0 con 0x7fe0240c5550 2024-04-25T00:15:06.051 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:15:06.052 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":106,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6810,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/2340581953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2340581953},{"type":"v1","addr":"172.21.15.116:6833","nonce":2340581953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:15:01.877414+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5273":{"gid":5273,"name":"d","rank":0,"incarnation":102,"state":"up:active","state_seq":49,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5273,"qdb_cluster":[5273]},"id":1},{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:14:58.851744+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5791},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5791":{"gid":5791,"name":"b","rank":0,"incarnation":98,"state":"up:active","state_seq":23,"addr":"172.21.15.174:6837/3180024481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3180024481},{"type":"v1","addr":"172.21.15.174:6837","nonce":3180024481}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5791,"qdb_cluster":[5791]},"id":2}]} 2024-04-25T00:15:06.052 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 106 2024-04-25T00:15:06.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:06.052+0000 7fe02b1a7640 1 -- 172.21.15.116:0/390339071 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe0080678f0 msgr2=0x7fe008069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:06.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:06.052+0000 7fe02b1a7640 1 --2- 172.21.15.116:0/390339071 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe0080678f0 0x7fe008069db0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fe024072a00 tx=0x7fe018008040 comp rx=0 tx=0).stop 2024-04-25T00:15:06.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:06.052+0000 7fe02b1a7640 1 -- 172.21.15.116:0/390339071 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0240c5550 msgr2=0x7fe0241f1e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:06.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:06.052+0000 7fe02b1a7640 1 --2- 172.21.15.116:0/390339071 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0240c5550 0x7fe0241f1e80 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fe0100097c0 tx=0x7fe010019120 comp rx=0 tx=0).stop 2024-04-25T00:15:06.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:06.052+0000 7fe02b1a7640 1 -- 172.21.15.116:0/390339071 shutdown_connections 2024-04-25T00:15:06.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:06.052+0000 7fe02b1a7640 1 --2- 172.21.15.116:0/390339071 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe0080678f0 0x7fe008069db0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:06.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:06.052+0000 7fe02b1a7640 1 --2- 172.21.15.116:0/390339071 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe024150d10 0x7fe02415c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:06.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:06.052+0000 7fe02b1a7640 1 --2- 172.21.15.116:0/390339071 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0240c5550 0x7fe0241f1e80 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:06.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:06.052+0000 7fe02b1a7640 1 --2- 172.21.15.116:0/390339071 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe024072ca0 0x7fe0241eeaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:06.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:06.052+0000 7fe02b1a7640 1 -- 172.21.15.116:0/390339071 >> 172.21.15.116:0/390339071 conn(0x7fe02406c720 msgr2=0x7fe02414e540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:06.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:06.052+0000 7fe02b1a7640 1 -- 172.21.15.116:0/390339071 shutdown_connections 2024-04-25T00:15:06.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:06.052+0000 7fe02b1a7640 1 -- 172.21.15.116:0/390339071 wait complete. 2024-04-25T00:15:06.073 INFO:tasks.mds_thrash.fs.[a]:mds.c reported in up:standby state 2024-04-25T00:15:06.073 INFO:tasks.mds_thrash.fs.[a]:waiting for 22 secs before thrashing 2024-04-25T00:15:07.561 INFO:tasks.mds_thrash.fs.[b]:reviving mds.a 2024-04-25T00:15:07.561 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-25T00:15:07.561 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-25T00:15:07.564 INFO:tasks.ceph.mds.a:Started 2024-04-25T00:15:07.564 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:15:07.637 INFO:tasks.ceph.mds.a.smithi116.stdout:starting mds.a at 2024-04-25T00:15:07.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.696+0000 7f4e822b7640 1 Processor -- start 2024-04-25T00:15:07.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.696+0000 7f4e822b7640 1 -- start start 2024-04-25T00:15:07.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.696+0000 7f4e822b7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4e7c14bf10 0x7f4e7c14c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:07.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.696+0000 7f4e822b7640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f4e7c0cacc0 0x7f4e7c06be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:07.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.696+0000 7f4e822b7640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e7c156720 con 0x7f4e7c0c5ac0 2024-04-25T00:15:07.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.696+0000 7f4e822b7640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f4e7c0c8460 con 0x7f4e7c14bf10 2024-04-25T00:15:07.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.696+0000 7f4e822b7640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f4e7c156440 con 0x7f4e7c0cacc0 2024-04-25T00:15:07.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.696+0000 7f4e7bfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4e7c14bf10 0x7f4e7c14c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:07.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.696+0000 7f4e80854640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f4e7c0c5ac0 0x7f4e7c0c9040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57306/0 (socket says 172.21.15.116:57306) 2024-04-25T00:15:07.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.696+0000 7f4e7bfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4e7c14bf10 0x7f4e7c14c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45008/0 (socket says 172.21.15.116:45008) 2024-04-25T00:15:07.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.696+0000 7f4e80854640 1 -- 172.21.15.116:0/2896824201 learned_addr learned my addr 172.21.15.116:0/2896824201 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:15:07.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.696+0000 7f4e73fff640 1 --2- 172.21.15.116:0/2896824201 >> v2:172.21.15.174:3300/0 conn(0x7f4e7c0cacc0 0x7f4e7c06be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:07.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.696+0000 7f4e7bfff640 1 -- 172.21.15.116:0/2896824201 >> v2:172.21.15.174:3300/0 conn(0x7f4e7c0cacc0 msgr2=0x7f4e7c06be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:07.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.696+0000 7f4e7bfff640 1 --2- 172.21.15.116:0/2896824201 >> v2:172.21.15.174:3300/0 conn(0x7f4e7c0cacc0 0x7f4e7c06be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:07.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.696+0000 7f4e7bfff640 1 -- 172.21.15.116:0/2896824201 >> v1:172.21.15.116:6789/0 conn(0x7f4e7c0c5ac0 legacy=0x7f4e7c0c9040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:07.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.696+0000 7f4e7b7fe640 1 -- 172.21.15.116:0/2896824201 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2500500162 0 0) 0x7f4e7c156720 con 0x7f4e7c0c5ac0 2024-04-25T00:15:07.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.696+0000 7f4e7bfff640 1 -- 172.21.15.116:0/2896824201 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e7c0c7c60 con 0x7f4e7c14bf10 2024-04-25T00:15:07.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.696+0000 7f4e73fff640 1 --2- 172.21.15.116:0/2896824201 >> v2:172.21.15.174:3300/0 conn(0x7f4e7c0cacc0 0x7f4e7c06be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:15:07.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.700+0000 7f4e7bfff640 1 --2- 172.21.15.116:0/2896824201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e7c14bf10 0x7f4e7c14c2f0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c0097f0 tx=0x7f4e6c01c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c57933c41a899db4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:07.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.700+0000 7f4e7b7fe640 1 -- 172.21.15.116:0/2896824201 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e6c0b0070 con 0x7f4e7c14bf10 2024-04-25T00:15:07.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.700+0000 7f4e7b7fe640 1 -- 172.21.15.116:0/2896824201 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e6c0ab600 con 0x7f4e7c14bf10 2024-04-25T00:15:07.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.700+0000 7f4e7b7fe640 1 -- 172.21.15.116:0/2896824201 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e6c0ab9d0 con 0x7f4e7c14bf10 2024-04-25T00:15:07.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.700+0000 7f4e822b7640 1 -- 172.21.15.116:0/2896824201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e7c14bf10 msgr2=0x7f4e7c14c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:07.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.700+0000 7f4e822b7640 1 --2- 172.21.15.116:0/2896824201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e7c14bf10 0x7f4e7c14c2f0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c0097f0 tx=0x7f4e6c01c460 comp rx=0 tx=0).stop 2024-04-25T00:15:07.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.700+0000 7f4e822b7640 1 -- 172.21.15.116:0/2896824201 shutdown_connections 2024-04-25T00:15:07.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.700+0000 7f4e822b7640 1 --2- 172.21.15.116:0/2896824201 >> v2:172.21.15.174:3300/0 conn(0x7f4e7c0cacc0 0x7f4e7c06be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:07.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.700+0000 7f4e822b7640 1 --2- 172.21.15.116:0/2896824201 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e7c14bf10 0x7f4e7c14c2f0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:07.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.700+0000 7f4e822b7640 1 -- 172.21.15.116:0/2896824201 >> 172.21.15.116:0/2896824201 conn(0x7f4e7c0ace40 msgr2=0x7f4e7c0ad250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:07.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.700+0000 7f4e822b7640 1 -- 172.21.15.116:0/2896824201 shutdown_connections 2024-04-25T00:15:07.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.700+0000 7f4e822b7640 1 -- 172.21.15.116:0/2896824201 wait complete. 2024-04-25T00:15:07.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.700+0000 7f4e822b7640 1 Processor -- start 2024-04-25T00:15:07.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.700+0000 7f4e822b7640 1 -- start start 2024-04-25T00:15:07.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.704+0000 7f4e822b7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4e7c0c5ac0 0x7f4e7c14aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:07.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.704+0000 7f4e822b7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e7c0cacc0 0x7f4e7c145280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:07.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.704+0000 7f4e822b7640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4e7c14bf10 0x7f4e7c1457c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:07.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.704+0000 7f4e822b7640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4e7c0b0940 con 0x7f4e7c0cacc0 2024-04-25T00:15:07.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.704+0000 7f4e822b7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4e7c0b07c0 con 0x7f4e7c0c5ac0 2024-04-25T00:15:07.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.704+0000 7f4e822b7640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f4e7c0b0ac0 con 0x7f4e7c14bf10 2024-04-25T00:15:07.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.704+0000 7f4e73fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e7c0cacc0 0x7f4e7c145280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:07.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.704+0000 7f4e73fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e7c0cacc0 0x7f4e7c145280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45016/0 (socket says 172.21.15.116:45016) 2024-04-25T00:15:07.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.704+0000 7f4e73fff640 1 -- 172.21.15.116:0/3097267488 learned_addr learned my addr 172.21.15.116:0/3097267488 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:15:07.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.704+0000 7f4e7bfff640 1 --2- 172.21.15.116:0/3097267488 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4e7c0c5ac0 0x7f4e7c14aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:07.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.704+0000 7f4e80854640 1 --2- 172.21.15.116:0/3097267488 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4e7c14bf10 0x7f4e7c1457c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:07.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.704+0000 7f4e73fff640 1 -- 172.21.15.116:0/3097267488 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4e7c14bf10 msgr2=0x7f4e7c1457c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:07.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.704+0000 7f4e73fff640 1 --2- 172.21.15.116:0/3097267488 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4e7c14bf10 0x7f4e7c1457c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:07.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.704+0000 7f4e73fff640 1 -- 172.21.15.116:0/3097267488 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4e7c0c5ac0 msgr2=0x7f4e7c14aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:07.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.704+0000 7f4e73fff640 1 --2- 172.21.15.116:0/3097267488 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4e7c0c5ac0 0x7f4e7c14aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:07.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.704+0000 7f4e73fff640 1 -- 172.21.15.116:0/3097267488 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e7c145d00 con 0x7f4e7c0cacc0 2024-04-25T00:15:07.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.704+0000 7f4e73fff640 1 --2- 172.21.15.116:0/3097267488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e7c0cacc0 0x7f4e7c145280 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f4e6400d830 tx=0x7f4e6400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:07.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.704+0000 7f4e797fa640 1 -- 172.21.15.116:0/3097267488 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e640096a0 con 0x7f4e7c0cacc0 2024-04-25T00:15:07.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.704+0000 7f4e797fa640 1 -- 172.21.15.116:0/3097267488 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e64019070 con 0x7f4e7c0cacc0 2024-04-25T00:15:07.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.704+0000 7f4e822b7640 1 -- 172.21.15.116:0/3097267488 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e7c0c1690 con 0x7f4e7c0cacc0 2024-04-25T00:15:07.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.704+0000 7f4e797fa640 1 -- 172.21.15.116:0/3097267488 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e64009a40 con 0x7f4e7c0cacc0 2024-04-25T00:15:07.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.704+0000 7f4e822b7640 1 -- 172.21.15.116:0/3097267488 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e7c0c3820 con 0x7f4e7c0cacc0 2024-04-25T00:15:07.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.704+0000 7f4e822b7640 1 -- 172.21.15.116:0/3097267488 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e7c0c1bb0 con 0x7f4e7c0cacc0 2024-04-25T00:15:07.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.704+0000 7f4e797fa640 1 -- 172.21.15.116:0/3097267488 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4e6401e460 con 0x7f4e7c0cacc0 2024-04-25T00:15:07.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.704+0000 7f4e797fa640 1 --2- 172.21.15.116:0/3097267488 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4e5c0678f0 0x7f4e5c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:07.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.704+0000 7f4e797fa640 1 -- 172.21.15.116:0/3097267488 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 8300+0+0 (secure 0 0 0) 0x7f4e64086b10 con 0x7f4e7c0cacc0 2024-04-25T00:15:07.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.704+0000 7f4e7bfff640 1 --2- 172.21.15.116:0/3097267488 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4e5c0678f0 0x7f4e5c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:07.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.708+0000 7f4e7bfff640 1 --2- 172.21.15.116:0/3097267488 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4e5c0678f0 0x7f4e5c069db0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c0ad000 tx=0x7f4e6c0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:07.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.708+0000 7f4e797fa640 1 -- 172.21.15.116:0/3097267488 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f4e64056620 con 0x7f4e7c0cacc0 2024-04-25T00:15:07.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.948+0000 7f4e822b7640 1 -- 172.21.15.116:0/3097267488 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4e7c068350 con 0x7f4e7c0cacc0 2024-04-25T00:15:07.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.948+0000 7f4e797fa640 1 -- 172.21.15.116:0/3097267488 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) ==== 95+0+5680 (secure 0 0 0) 0x7f4e6405a920 con 0x7f4e7c0cacc0 2024-04-25T00:15:07.948 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:15:07.948 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":106,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6810,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/2340581953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2340581953},{"type":"v1","addr":"172.21.15.116:6833","nonce":2340581953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:15:01.877414+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5273":{"gid":5273,"name":"d","rank":0,"incarnation":102,"state":"up:active","state_seq":49,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5273,"qdb_cluster":[5273]},"id":1},{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:14:58.851744+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5791},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5791":{"gid":5791,"name":"b","rank":0,"incarnation":98,"state":"up:active","state_seq":23,"addr":"172.21.15.174:6837/3180024481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3180024481},{"type":"v1","addr":"172.21.15.174:6837","nonce":3180024481}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5791,"qdb_cluster":[5791]},"id":2}]} 2024-04-25T00:15:07.949 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 106 2024-04-25T00:15:07.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.952+0000 7f4e822b7640 1 -- 172.21.15.116:0/3097267488 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4e5c0678f0 msgr2=0x7f4e5c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:07.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.952+0000 7f4e822b7640 1 --2- 172.21.15.116:0/3097267488 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4e5c0678f0 0x7f4e5c069db0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c0ad000 tx=0x7f4e6c0a6000 comp rx=0 tx=0).stop 2024-04-25T00:15:07.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.952+0000 7f4e822b7640 1 -- 172.21.15.116:0/3097267488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e7c0cacc0 msgr2=0x7f4e7c145280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:07.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.952+0000 7f4e822b7640 1 --2- 172.21.15.116:0/3097267488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e7c0cacc0 0x7f4e7c145280 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f4e6400d830 tx=0x7f4e6400dcf0 comp rx=0 tx=0).stop 2024-04-25T00:15:07.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.952+0000 7f4e822b7640 1 -- 172.21.15.116:0/3097267488 shutdown_connections 2024-04-25T00:15:07.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.952+0000 7f4e822b7640 1 --2- 172.21.15.116:0/3097267488 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4e5c0678f0 0x7f4e5c069db0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:07.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.952+0000 7f4e822b7640 1 --2- 172.21.15.116:0/3097267488 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4e7c14bf10 0x7f4e7c1457c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:07.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.952+0000 7f4e822b7640 1 --2- 172.21.15.116:0/3097267488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e7c0cacc0 0x7f4e7c145280 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:07.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.952+0000 7f4e822b7640 1 --2- 172.21.15.116:0/3097267488 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4e7c0c5ac0 0x7f4e7c14aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:07.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.952+0000 7f4e822b7640 1 -- 172.21.15.116:0/3097267488 >> 172.21.15.116:0/3097267488 conn(0x7f4e7c0ace40 msgr2=0x7f4e7c0c9e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:07.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.952+0000 7f4e822b7640 1 -- 172.21.15.116:0/3097267488 shutdown_connections 2024-04-25T00:15:07.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:07.952+0000 7f4e822b7640 1 -- 172.21.15.116:0/3097267488 wait complete. 2024-04-25T00:15:07.972 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2024-04-25T00:15:08.303 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:15:08.305 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:15:09.973 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:15:10.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.100+0000 7f57f7b67640 1 Processor -- start 2024-04-25T00:15:10.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.100+0000 7f57f7b67640 1 -- start start 2024-04-25T00:15:10.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.104+0000 7f57f7b67640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f57f0152ca0 0x7f57f006b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.104+0000 7f57f7b67640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f57f006b750 0x7f57f006f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.104+0000 7f57f7b67640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57f01558e0 con 0x7f57f0158d30 2024-04-25T00:15:10.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.104+0000 7f57f7b67640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f57f0155bc0 con 0x7f57f0152ca0 2024-04-25T00:15:10.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.104+0000 7f57f7b67640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f57f0155ea0 con 0x7f57f006b750 2024-04-25T00:15:10.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.104+0000 7f57f5903640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f57f0152ca0 0x7f57f006b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.104+0000 7f57f5102640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f57f0158d30 0x7f57f006f270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57308/0 (socket says 172.21.15.116:57308) 2024-04-25T00:15:10.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.104+0000 7f57f5102640 1 -- 172.21.15.116:0/1141111974 learned_addr learned my addr 172.21.15.116:0/1141111974 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:15:10.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.104+0000 7f57f5903640 1 --2- 172.21.15.116:0/1141111974 >> v2:172.21.15.116:3300/0 conn(0x7f57f0152ca0 0x7f57f006b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45030/0 (socket says 172.21.15.116:45030) 2024-04-25T00:15:10.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.104+0000 7f57f6104640 1 --2- 172.21.15.116:0/1141111974 >> v2:172.21.15.174:3300/0 conn(0x7f57f006b750 0x7f57f006f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.104+0000 7f57f5903640 1 -- 172.21.15.116:0/1141111974 >> v2:172.21.15.174:3300/0 conn(0x7f57f006b750 msgr2=0x7f57f006f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:10.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.104+0000 7f57f5903640 1 --2- 172.21.15.116:0/1141111974 >> v2:172.21.15.174:3300/0 conn(0x7f57f006b750 0x7f57f006f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.104+0000 7f57f5903640 1 -- 172.21.15.116:0/1141111974 >> v1:172.21.15.116:6789/0 conn(0x7f57f0158d30 legacy=0x7f57f006f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:10.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.104+0000 7f57f5903640 1 -- 172.21.15.116:0/1141111974 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57f0156180 con 0x7f57f0152ca0 2024-04-25T00:15:10.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.104+0000 7f57f4901640 1 -- 172.21.15.116:0/1141111974 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2563736347 0 0) 0x7f57f01558e0 con 0x7f57f0158d30 2024-04-25T00:15:10.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.104+0000 7f57f5903640 1 --2- 172.21.15.116:0/1141111974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57f0152ca0 0x7f57f006b210 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f57f00ad700 tx=0x7f57ec01c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce3615e548da28da server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:10.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.108+0000 7f57f4901640 1 -- 172.21.15.116:0/1141111974 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57ec0b0070 con 0x7f57f0152ca0 2024-04-25T00:15:10.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.108+0000 7f57f4901640 1 -- 172.21.15.116:0/1141111974 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57ec0ab600 con 0x7f57f0152ca0 2024-04-25T00:15:10.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.108+0000 7f57f4901640 1 -- 172.21.15.116:0/1141111974 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57ec0ab9d0 con 0x7f57f0152ca0 2024-04-25T00:15:10.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.108+0000 7f57f7b67640 1 -- 172.21.15.116:0/1141111974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57f0152ca0 msgr2=0x7f57f006b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:10.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.108+0000 7f57f7b67640 1 --2- 172.21.15.116:0/1141111974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57f0152ca0 0x7f57f006b210 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f57f00ad700 tx=0x7f57ec01c3f0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.108+0000 7f57f7b67640 1 -- 172.21.15.116:0/1141111974 shutdown_connections 2024-04-25T00:15:10.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.108+0000 7f57f7b67640 1 --2- 172.21.15.116:0/1141111974 >> v2:172.21.15.174:3300/0 conn(0x7f57f006b750 0x7f57f006f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.108+0000 7f57f7b67640 1 --2- 172.21.15.116:0/1141111974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57f0152ca0 0x7f57f006b210 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.108+0000 7f57f7b67640 1 -- 172.21.15.116:0/1141111974 >> 172.21.15.116:0/1141111974 conn(0x7f57f00c7fa0 msgr2=0x7f57f00c83b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:10.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.108+0000 7f57f7b67640 1 -- 172.21.15.116:0/1141111974 shutdown_connections 2024-04-25T00:15:10.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.108+0000 7f57f7b67640 1 -- 172.21.15.116:0/1141111974 wait complete. 2024-04-25T00:15:10.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.108+0000 7f57f7b67640 1 Processor -- start 2024-04-25T00:15:10.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.108+0000 7f57f7b67640 1 -- start start 2024-04-25T00:15:10.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.108+0000 7f57f7b67640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f57f006b750 0x7f57f00c70e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.108+0000 7f57f7b67640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57f0152ca0 0x7f57f01543d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.108+0000 7f57f7b67640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f57f0158d30 0x7f57f01485c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.108+0000 7f57f7b67640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f57f0153200 con 0x7f57f0152ca0 2024-04-25T00:15:10.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.108+0000 7f57f7b67640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f57f0153080 con 0x7f57f0158d30 2024-04-25T00:15:10.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.108+0000 7f57f7b67640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f57f01628e0 con 0x7f57f006b750 2024-04-25T00:15:10.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.108+0000 7f57f5102640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57f0152ca0 0x7f57f01543d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.108+0000 7f57f5102640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57f0152ca0 0x7f57f01543d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45040/0 (socket says 172.21.15.116:45040) 2024-04-25T00:15:10.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.108+0000 7f57f5102640 1 -- 172.21.15.116:0/4288210828 learned_addr learned my addr 172.21.15.116:0/4288210828 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:15:10.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.108+0000 7f57f5903640 1 --2- 172.21.15.116:0/4288210828 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f57f006b750 0x7f57f00c70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.108+0000 7f57f6104640 1 --2- 172.21.15.116:0/4288210828 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f57f0158d30 0x7f57f01485c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.108+0000 7f57f5102640 1 -- 172.21.15.116:0/4288210828 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f57f006b750 msgr2=0x7f57f00c70e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:10.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.108+0000 7f57f5102640 1 --2- 172.21.15.116:0/4288210828 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f57f006b750 0x7f57f00c70e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.108+0000 7f57f5102640 1 -- 172.21.15.116:0/4288210828 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f57f0158d30 msgr2=0x7f57f01485c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:10.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.108+0000 7f57f5102640 1 --2- 172.21.15.116:0/4288210828 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f57f0158d30 0x7f57f01485c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.108+0000 7f57f5102640 1 -- 172.21.15.116:0/4288210828 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57f0154cd0 con 0x7f57f0152ca0 2024-04-25T00:15:10.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.108+0000 7f57f6104640 1 --2- 172.21.15.116:0/4288210828 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f57f0158d30 0x7f57f01485c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:15:10.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.108+0000 7f57f5102640 1 --2- 172.21.15.116:0/4288210828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57f0152ca0 0x7f57f01543d0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f57e000c4b0 tx=0x7f57e000c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:10.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.108+0000 7f57e67fc640 1 -- 172.21.15.116:0/4288210828 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e0018070 con 0x7f57f0152ca0 2024-04-25T00:15:10.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.108+0000 7f57e67fc640 1 -- 172.21.15.116:0/4288210828 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57e0007500 con 0x7f57f0152ca0 2024-04-25T00:15:10.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.108+0000 7f57f7b67640 1 -- 172.21.15.116:0/4288210828 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57f01508d0 con 0x7f57f0152ca0 2024-04-25T00:15:10.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.108+0000 7f57e67fc640 1 -- 172.21.15.116:0/4288210828 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e0004a30 con 0x7f57f0152ca0 2024-04-25T00:15:10.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.108+0000 7f57f7b67640 1 -- 172.21.15.116:0/4288210828 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57f00c7620 con 0x7f57f0152ca0 2024-04-25T00:15:10.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.112+0000 7f57f7b67640 1 -- 172.21.15.116:0/4288210828 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57f0068a60 con 0x7f57f0152ca0 2024-04-25T00:15:10.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.112+0000 7f57e67fc640 1 -- 172.21.15.116:0/4288210828 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f57e0004bd0 con 0x7f57f0152ca0 2024-04-25T00:15:10.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.112+0000 7f57e67fc640 1 --2- 172.21.15.116:0/4288210828 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f57d40678c0 0x7f57d4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:10.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.112+0000 7f57e67fc640 1 -- 172.21.15.116:0/4288210828 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 8300+0+0 (secure 0 0 0) 0x7f57e0085980 con 0x7f57f0152ca0 2024-04-25T00:15:10.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.112+0000 7f57f5903640 1 --2- 172.21.15.116:0/4288210828 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f57d40678c0 0x7f57d4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:10.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.112+0000 7f57f5903640 1 --2- 172.21.15.116:0/4288210828 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f57d40678c0 0x7f57d4069d80 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f57ec0ad000 tx=0x7f57ec008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:10.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.120+0000 7f57e67fc640 1 -- 172.21.15.116:0/4288210828 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f57e0055380 con 0x7f57f0152ca0 2024-04-25T00:15:10.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.344+0000 7f57f7b67640 1 -- 172.21.15.116:0/4288210828 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f57f0072de0 con 0x7f57f0152ca0 2024-04-25T00:15:10.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.344+0000 7f57e67fc640 1 -- 172.21.15.116:0/4288210828 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) ==== 95+0+6509 (secure 0 0 0) 0x7f57e0059680 con 0x7f57f0152ca0 2024-04-25T00:15:10.344 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:15:10.344 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":107,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6810,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/2340581953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2340581953},{"type":"v1","addr":"172.21.15.116:6833","nonce":2340581953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106},{"gid":6828,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/2012470875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2012470875},{"type":"v1","addr":"172.21.15.116:6835","nonce":2012470875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:15:01.877414+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5273":{"gid":5273,"name":"d","rank":0,"incarnation":102,"state":"up:active","state_seq":49,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5273,"qdb_cluster":[5273]},"id":1},{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:14:58.851744+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5791},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5791":{"gid":5791,"name":"b","rank":0,"incarnation":98,"state":"up:active","state_seq":23,"addr":"172.21.15.174:6837/3180024481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3180024481},{"type":"v1","addr":"172.21.15.174:6837","nonce":3180024481}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5791,"qdb_cluster":[5791]},"id":2}]} 2024-04-25T00:15:10.344 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 107 2024-04-25T00:15:10.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.344+0000 7f57f7b67640 1 -- 172.21.15.116:0/4288210828 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f57d40678c0 msgr2=0x7f57d4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:10.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.344+0000 7f57f7b67640 1 --2- 172.21.15.116:0/4288210828 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f57d40678c0 0x7f57d4069d80 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f57ec0ad000 tx=0x7f57ec008040 comp rx=0 tx=0).stop 2024-04-25T00:15:10.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.344+0000 7f57f7b67640 1 -- 172.21.15.116:0/4288210828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57f0152ca0 msgr2=0x7f57f01543d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:10.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.344+0000 7f57f7b67640 1 --2- 172.21.15.116:0/4288210828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57f0152ca0 0x7f57f01543d0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f57e000c4b0 tx=0x7f57e000c970 comp rx=0 tx=0).stop 2024-04-25T00:15:10.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.344+0000 7f57f7b67640 1 -- 172.21.15.116:0/4288210828 shutdown_connections 2024-04-25T00:15:10.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.344+0000 7f57f7b67640 1 --2- 172.21.15.116:0/4288210828 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f57d40678c0 0x7f57d4069d80 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.344+0000 7f57f7b67640 1 --2- 172.21.15.116:0/4288210828 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f57f0158d30 0x7f57f01485c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.344+0000 7f57f7b67640 1 --2- 172.21.15.116:0/4288210828 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57f0152ca0 0x7f57f01543d0 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.344+0000 7f57f7b67640 1 --2- 172.21.15.116:0/4288210828 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f57f006b750 0x7f57f00c70e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:10.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.344+0000 7f57f7b67640 1 -- 172.21.15.116:0/4288210828 >> 172.21.15.116:0/4288210828 conn(0x7f57f00c7fa0 msgr2=0x7f57f006f270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:10.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.344+0000 7f57f7b67640 1 -- 172.21.15.116:0/4288210828 shutdown_connections 2024-04-25T00:15:10.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:10.344+0000 7f57f7b67640 1 -- 172.21.15.116:0/4288210828 wait complete. 2024-04-25T00:15:10.365 INFO:tasks.mds_thrash.fs.[b]:mds.a reported in up:standby state 2024-04-25T00:15:10.365 INFO:tasks.mds_thrash.fs.[b]:waiting for 3 secs before thrashing 2024-04-25T00:15:13.366 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:15:13.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.508+0000 7f8931bfa640 1 Processor -- start 2024-04-25T00:15:13.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.508+0000 7f8931bfa640 1 -- start start 2024-04-25T00:15:13.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.508+0000 7f8931bfa640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f892c159b00 0x7f892c1611b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:13.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.508+0000 7f8931bfa640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f892c06e340 con 0x7f892c153050 2024-04-25T00:15:13.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.508+0000 7f8931bfa640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f892c0c5580 con 0x7f892c06bc20 2024-04-25T00:15:13.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.508+0000 7f8931bfa640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f892c06e060 con 0x7f892c159b00 2024-04-25T00:15:13.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.508+0000 7f8930bf8640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f892c153050 0x7f892c06e6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57316/0 (socket says 172.21.15.116:57316) 2024-04-25T00:15:13.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.508+0000 7f8930bf8640 1 -- 172.21.15.116:0/1969951741 learned_addr learned my addr 172.21.15.116:0/1969951741 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:15:13.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.508+0000 7f89313f9640 1 --2- 172.21.15.116:0/1969951741 >> v2:172.21.15.174:3300/0 conn(0x7f892c159b00 0x7f892c1611b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:13.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.508+0000 7f892b7fe640 1 -- 172.21.15.116:0/1969951741 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1910521829 0 0) 0x7f892c06e340 con 0x7f892c153050 2024-04-25T00:15:13.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.508+0000 7f892b7fe640 1 -- 172.21.15.116:0/1969951741 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8914003530 con 0x7f892c153050 2024-04-25T00:15:13.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.508+0000 7f892b7fe640 1 -- 172.21.15.116:0/1969951741 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 335979439 0 0) 0x7f8914003530 con 0x7f892c153050 2024-04-25T00:15:13.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.508+0000 7f892b7fe640 1 -- 172.21.15.116:0/1969951741 >> v2:172.21.15.174:3300/0 conn(0x7f892c159b00 msgr2=0x7f892c1611b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:13.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.508+0000 7f892b7fe640 1 --2- 172.21.15.116:0/1969951741 >> v2:172.21.15.174:3300/0 conn(0x7f892c159b00 0x7f892c1611b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:13.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.508+0000 7f892b7fe640 1 -- 172.21.15.116:0/1969951741 >> v1:172.21.15.174:6789/0 conn(0x7f892c06bc20 legacy=0x7f892c1593f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:13.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.508+0000 7f892b7fe640 1 -- 172.21.15.116:0/1969951741 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f892c158ed0 con 0x7f892c153050 2024-04-25T00:15:13.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.508+0000 7f892b7fe640 1 -- 172.21.15.116:0/1969951741 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f891c001a00 con 0x7f892c153050 2024-04-25T00:15:13.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.508+0000 7f892b7fe640 1 -- 172.21.15.116:0/1969951741 >> v1:172.21.15.116:6789/0 conn(0x7f892c153050 legacy=0x7f892c06e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:13.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.508+0000 7f892b7fe640 1 --2- 172.21.15.116:0/1969951741 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f89140039c0 0x7f8914003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:13.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.508+0000 7f892b7fe640 1 --2- 172.21.15.116:0/1969951741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8914004810 0x7f8914004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:13.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.508+0000 7f892b7fe640 1 --2- 172.21.15.116:0/1969951741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89140051d0 0x7f891400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:13.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.508+0000 7f892b7fe640 1 -- 172.21.15.116:0/1969951741 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f892c06e060 con 0x7f89140051d0 2024-04-25T00:15:13.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.508+0000 7f892b7fe640 1 -- 172.21.15.116:0/1969951741 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f891400c170 con 0x7f8914004810 2024-04-25T00:15:13.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.508+0000 7f892b7fe640 1 -- 172.21.15.116:0/1969951741 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f891400c370 con 0x7f89140039c0 2024-04-25T00:15:13.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.508+0000 7f89313f9640 1 --2- 172.21.15.116:0/1969951741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89140051d0 0x7f891400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:13.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.508+0000 7f892bfff640 1 --2- 172.21.15.116:0/1969951741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8914004810 0x7f8914004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:13.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.508+0000 7f8930bf8640 1 --2- 172.21.15.116:0/1969951741 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f89140039c0 0x7f8914003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:13.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.508+0000 7f89313f9640 1 -- 172.21.15.116:0/1969951741 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f89140039c0 msgr2=0x7f8914003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:13.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.508+0000 7f89313f9640 1 --2- 172.21.15.116:0/1969951741 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f89140039c0 0x7f8914003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:13.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.508+0000 7f89313f9640 1 -- 172.21.15.116:0/1969951741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8914004810 msgr2=0x7f8914004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:13.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.508+0000 7f89313f9640 1 --2- 172.21.15.116:0/1969951741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8914004810 0x7f8914004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:13.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.508+0000 7f89313f9640 1 -- 172.21.15.116:0/1969951741 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8914003530 con 0x7f89140051d0 2024-04-25T00:15:13.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.508+0000 7f892bfff640 1 --2- 172.21.15.116:0/1969951741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8914004810 0x7f8914004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:15:13.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.512+0000 7f89313f9640 1 -- 172.21.15.116:0/1969951741 reap_dead start 2024-04-25T00:15:13.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.512+0000 7f89313f9640 1 --2- 172.21.15.116:0/1969951741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89140051d0 0x7f891400b9d0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f8924009ab0 tx=0x7f892401ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93be02637027fe5a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:13.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.512+0000 7f892b7fe640 1 -- 172.21.15.116:0/1969951741 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89240096b0 con 0x7f89140051d0 2024-04-25T00:15:13.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.512+0000 7f892b7fe640 1 -- 172.21.15.116:0/1969951741 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89240a5570 con 0x7f89140051d0 2024-04-25T00:15:13.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.512+0000 7f892b7fe640 1 -- 172.21.15.116:0/1969951741 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89240a5920 con 0x7f89140051d0 2024-04-25T00:15:13.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.512+0000 7f8931bfa640 1 -- 172.21.15.116:0/1969951741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89140051d0 msgr2=0x7f891400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:13.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.512+0000 7f8931bfa640 1 --2- 172.21.15.116:0/1969951741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89140051d0 0x7f891400b9d0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f8924009ab0 tx=0x7f892401ca90 comp rx=0 tx=0).stop 2024-04-25T00:15:13.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.512+0000 7f8931bfa640 1 -- 172.21.15.116:0/1969951741 shutdown_connections 2024-04-25T00:15:13.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.512+0000 7f8931bfa640 1 --2- 172.21.15.116:0/1969951741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89140051d0 0x7f891400b9d0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:13.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.512+0000 7f8931bfa640 1 -- 172.21.15.116:0/1969951741 >> 172.21.15.116:0/1969951741 conn(0x7f892c0c49c0 msgr2=0x7f892c0c4dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:13.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.512+0000 7f8931bfa640 1 -- 172.21.15.116:0/1969951741 shutdown_connections 2024-04-25T00:15:13.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.512+0000 7f8931bfa640 1 -- 172.21.15.116:0/1969951741 wait complete. 2024-04-25T00:15:13.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.512+0000 7f8931bfa640 1 Processor -- start 2024-04-25T00:15:13.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.512+0000 7f8931bfa640 1 -- start start 2024-04-25T00:15:13.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.516+0000 7f8931bfa640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89140051d0 0x7f892c151750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:13.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.516+0000 7f8931bfa640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f892c14d300 0x7f892c0c62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:13.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.516+0000 7f8931bfa640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f892c150880 0x7f892c1eb7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:13.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.516+0000 7f8931bfa640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f892c073750 con 0x7f89140051d0 2024-04-25T00:15:13.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.516+0000 7f8931bfa640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f892c070220 con 0x7f892c150880 2024-04-25T00:15:13.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.516+0000 7f8931bfa640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f892c153430 con 0x7f892c14d300 2024-04-25T00:15:13.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.516+0000 7f8930bf8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89140051d0 0x7f892c151750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:13.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.516+0000 7f8930bf8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89140051d0 0x7f892c151750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45062/0 (socket says 172.21.15.116:45062) 2024-04-25T00:15:13.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.516+0000 7f8930bf8640 1 -- 172.21.15.116:0/882440604 learned_addr learned my addr 172.21.15.116:0/882440604 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:15:13.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.516+0000 7f892bfff640 1 --2- 172.21.15.116:0/882440604 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f892c14d300 0x7f892c0c62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:13.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.516+0000 7f89313f9640 1 --2- 172.21.15.116:0/882440604 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f892c150880 0x7f892c1eb7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:13.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.516+0000 7f8930bf8640 1 -- 172.21.15.116:0/882440604 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f892c14d300 msgr2=0x7f892c0c62f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:13.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.516+0000 7f8930bf8640 1 --2- 172.21.15.116:0/882440604 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f892c14d300 0x7f892c0c62f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:13.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.516+0000 7f8930bf8640 1 -- 172.21.15.116:0/882440604 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f892c150880 msgr2=0x7f892c1eb7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:13.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.516+0000 7f8930bf8640 1 --2- 172.21.15.116:0/882440604 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f892c150880 0x7f892c1eb7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:13.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.516+0000 7f8930bf8640 1 -- 172.21.15.116:0/882440604 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f892c0c6b40 con 0x7f89140051d0 2024-04-25T00:15:13.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.516+0000 7f8930bf8640 1 --2- 172.21.15.116:0/882440604 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89140051d0 0x7f892c151750 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f891c011850 tx=0x7f891c011d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:13.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.516+0000 7f89297fa640 1 -- 172.21.15.116:0/882440604 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f891c010040 con 0x7f89140051d0 2024-04-25T00:15:13.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.516+0000 7f89297fa640 1 -- 172.21.15.116:0/882440604 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f891c01a070 con 0x7f89140051d0 2024-04-25T00:15:13.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.516+0000 7f8931bfa640 1 -- 172.21.15.116:0/882440604 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f892c0c7370 con 0x7f89140051d0 2024-04-25T00:15:13.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.516+0000 7f89297fa640 1 -- 172.21.15.116:0/882440604 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f891c00fdc0 con 0x7f89140051d0 2024-04-25T00:15:13.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.516+0000 7f8931bfa640 1 -- 172.21.15.116:0/882440604 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f892c0c6db0 con 0x7f89140051d0 2024-04-25T00:15:13.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.516+0000 7f8931bfa640 1 -- 172.21.15.116:0/882440604 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f892c04f510 con 0x7f89140051d0 2024-04-25T00:15:13.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.516+0000 7f89297fa640 1 -- 172.21.15.116:0/882440604 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f891c01f460 con 0x7f89140051d0 2024-04-25T00:15:13.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.516+0000 7f89297fa640 1 --2- 172.21.15.116:0/882440604 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f89180678c0 0x7f8918069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:13.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.516+0000 7f89297fa640 1 -- 172.21.15.116:0/882440604 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 8300+0+0 (secure 0 0 0) 0x7f891c089750 con 0x7f89140051d0 2024-04-25T00:15:13.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.516+0000 7f892bfff640 1 --2- 172.21.15.116:0/882440604 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f89180678c0 0x7f8918069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:13.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.520+0000 7f892bfff640 1 --2- 172.21.15.116:0/882440604 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f89180678c0 0x7f8918069d80 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f892c14bcf0 tx=0x7f8920009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:13.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.524+0000 7f89297fa640 1 -- 172.21.15.116:0/882440604 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f891c059260 con 0x7f89140051d0 2024-04-25T00:15:13.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.724+0000 7f8931bfa640 1 -- 172.21.15.116:0/882440604 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f892c0c3170 con 0x7f89140051d0 2024-04-25T00:15:13.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.724+0000 7f89297fa640 1 -- 172.21.15.116:0/882440604 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) ==== 95+0+6509 (secure 0 0 0) 0x7f891c05d560 con 0x7f89140051d0 2024-04-25T00:15:13.725 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:15:13.726 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":107,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6810,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/2340581953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2340581953},{"type":"v1","addr":"172.21.15.116:6833","nonce":2340581953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106},{"gid":6828,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/2012470875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2012470875},{"type":"v1","addr":"172.21.15.116:6835","nonce":2012470875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:15:01.877414+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5273":{"gid":5273,"name":"d","rank":0,"incarnation":102,"state":"up:active","state_seq":49,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5273,"qdb_cluster":[5273]},"id":1},{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:14:58.851744+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5791},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5791":{"gid":5791,"name":"b","rank":0,"incarnation":98,"state":"up:active","state_seq":23,"addr":"172.21.15.174:6837/3180024481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3180024481},{"type":"v1","addr":"172.21.15.174:6837","nonce":3180024481}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5791,"qdb_cluster":[5791]},"id":2}]} 2024-04-25T00:15:13.726 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 107 2024-04-25T00:15:13.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.728+0000 7f8931bfa640 1 -- 172.21.15.116:0/882440604 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f89180678c0 msgr2=0x7f8918069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:13.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.728+0000 7f8931bfa640 1 --2- 172.21.15.116:0/882440604 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f89180678c0 0x7f8918069d80 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f892c14bcf0 tx=0x7f8920009040 comp rx=0 tx=0).stop 2024-04-25T00:15:13.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.728+0000 7f8931bfa640 1 -- 172.21.15.116:0/882440604 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89140051d0 msgr2=0x7f892c151750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:13.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.728+0000 7f8931bfa640 1 --2- 172.21.15.116:0/882440604 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89140051d0 0x7f892c151750 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f891c011850 tx=0x7f891c011d10 comp rx=0 tx=0).stop 2024-04-25T00:15:13.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.728+0000 7f8931bfa640 1 -- 172.21.15.116:0/882440604 shutdown_connections 2024-04-25T00:15:13.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.728+0000 7f8931bfa640 1 --2- 172.21.15.116:0/882440604 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f89180678c0 0x7f8918069d80 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:13.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.728+0000 7f8931bfa640 1 --2- 172.21.15.116:0/882440604 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f892c150880 0x7f892c1eb7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:13.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.728+0000 7f8931bfa640 1 --2- 172.21.15.116:0/882440604 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f892c14d300 0x7f892c0c62f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:13.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.728+0000 7f8931bfa640 1 --2- 172.21.15.116:0/882440604 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89140051d0 0x7f892c151750 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:13.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.728+0000 7f8931bfa640 1 -- 172.21.15.116:0/882440604 >> 172.21.15.116:0/882440604 conn(0x7f892c0c49c0 msgr2=0x7f892c14c770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:13.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.728+0000 7f8931bfa640 1 -- 172.21.15.116:0/882440604 shutdown_connections 2024-04-25T00:15:13.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.728+0000 7f8931bfa640 1 -- 172.21.15.116:0/882440604 wait complete. 2024-04-25T00:15:13.750 INFO:tasks.mds_thrash.fs.[b]:kill mds.b (rank=0) 2024-04-25T00:15:13.751 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-25T00:15:13.751 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:15:13.764 INFO:tasks.ceph.mds.b:Stopped 2024-04-25T00:15:13.764 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:15:13.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.908+0000 7fc77276b640 1 Processor -- start 2024-04-25T00:15:13.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.908+0000 7fc77276b640 1 -- start start 2024-04-25T00:15:13.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.908+0000 7fc77276b640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fc76c06c9a0 0x7fc76c0c8030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:13.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.908+0000 7fc77276b640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc76c0c5c50 con 0x7fc76c152fd0 2024-04-25T00:15:13.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.908+0000 7fc77276b640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc76c0c69a0 con 0x7fc76c0c8570 2024-04-25T00:15:13.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.908+0000 7fc77276b640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fc76c0c5ec0 con 0x7fc76c06c9a0 2024-04-25T00:15:13.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.908+0000 7fc76bfff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fc76c152fd0 0x7fc76c0c6f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57330/0 (socket says 172.21.15.116:57330) 2024-04-25T00:15:13.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.908+0000 7fc76bfff640 1 -- 172.21.15.116:0/1793761959 learned_addr learned my addr 172.21.15.116:0/1793761959 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:15:13.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.908+0000 7fc76b7fe640 1 --2- 172.21.15.116:0/1793761959 >> v2:172.21.15.174:3300/0 conn(0x7fc76c06c9a0 0x7fc76c0c8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:13.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.908+0000 7fc76affd640 1 -- 172.21.15.116:0/1793761959 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2479812581 0 0) 0x7fc76c0c5c50 con 0x7fc76c152fd0 2024-04-25T00:15:13.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.908+0000 7fc76affd640 1 -- 172.21.15.116:0/1793761959 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc754003530 con 0x7fc76c152fd0 2024-04-25T00:15:13.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.908+0000 7fc76affd640 1 -- 172.21.15.116:0/1793761959 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1447984907 0 0) 0x7fc754003530 con 0x7fc76c152fd0 2024-04-25T00:15:13.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.908+0000 7fc76affd640 1 -- 172.21.15.116:0/1793761959 >> v2:172.21.15.174:3300/0 conn(0x7fc76c06c9a0 msgr2=0x7fc76c0c8030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:13.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.908+0000 7fc76affd640 1 --2- 172.21.15.116:0/1793761959 >> v2:172.21.15.174:3300/0 conn(0x7fc76c06c9a0 0x7fc76c0c8030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:13.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.908+0000 7fc76affd640 1 -- 172.21.15.116:0/1793761959 >> v1:172.21.15.174:6789/0 conn(0x7fc76c0c8570 legacy=0x7fc76c158e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:13.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.908+0000 7fc76affd640 1 -- 172.21.15.116:0/1793761959 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc76c0c61a0 con 0x7fc76c152fd0 2024-04-25T00:15:13.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.908+0000 7fc76affd640 1 -- 172.21.15.116:0/1793761959 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7fc75c001680 con 0x7fc76c152fd0 2024-04-25T00:15:13.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.908+0000 7fc76affd640 1 -- 172.21.15.116:0/1793761959 >> v1:172.21.15.116:6789/0 conn(0x7fc76c152fd0 legacy=0x7fc76c0c6f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:13.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.908+0000 7fc76affd640 1 --2- 172.21.15.116:0/1793761959 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc7540039c0 0x7fc754003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:13.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.908+0000 7fc76affd640 1 --2- 172.21.15.116:0/1793761959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7540043c0 0x7fc754004890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:13.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.908+0000 7fc76affd640 1 --2- 172.21.15.116:0/1793761959 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc754005130 0x7fc75400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:13.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.908+0000 7fc76affd640 1 -- 172.21.15.116:0/1793761959 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc76c0c5ec0 con 0x7fc7540043c0 2024-04-25T00:15:13.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.908+0000 7fc76affd640 1 -- 172.21.15.116:0/1793761959 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc75400c070 con 0x7fc754005130 2024-04-25T00:15:13.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.908+0000 7fc76affd640 1 -- 172.21.15.116:0/1793761959 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fc75400c270 con 0x7fc7540039c0 2024-04-25T00:15:13.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.912+0000 7fc76b7fe640 1 --2- 172.21.15.116:0/1793761959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7540043c0 0x7fc754004890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:13.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.912+0000 7fc770d08640 1 --2- 172.21.15.116:0/1793761959 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc754005130 0x7fc75400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:13.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.912+0000 7fc76bfff640 1 --2- 172.21.15.116:0/1793761959 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc7540039c0 0x7fc754003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:13.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.912+0000 7fc76b7fe640 1 -- 172.21.15.116:0/1793761959 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc7540039c0 msgr2=0x7fc754003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:13.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.912+0000 7fc76b7fe640 1 --2- 172.21.15.116:0/1793761959 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc7540039c0 0x7fc754003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:13.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.912+0000 7fc76b7fe640 1 -- 172.21.15.116:0/1793761959 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc754005130 msgr2=0x7fc75400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:13.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.912+0000 7fc76b7fe640 1 --2- 172.21.15.116:0/1793761959 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc754005130 0x7fc75400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:13.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.912+0000 7fc76b7fe640 1 -- 172.21.15.116:0/1793761959 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc754003530 con 0x7fc7540043c0 2024-04-25T00:15:13.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.912+0000 7fc770d08640 1 --2- 172.21.15.116:0/1793761959 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc754005130 0x7fc75400b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:15:13.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.912+0000 7fc770d08640 1 -- 172.21.15.116:0/1793761959 reap_dead start 2024-04-25T00:15:13.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.912+0000 7fc76b7fe640 1 --2- 172.21.15.116:0/1793761959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7540043c0 0x7fc754004890 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fc760009ae0 tx=0x7fc76001ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=552be58e15296302 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:13.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.912+0000 7fc76affd640 1 -- 172.21.15.116:0/1793761959 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7600b2070 con 0x7fc7540043c0 2024-04-25T00:15:13.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.912+0000 7fc76affd640 1 -- 172.21.15.116:0/1793761959 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7600a5660 con 0x7fc7540043c0 2024-04-25T00:15:13.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.912+0000 7fc76affd640 1 -- 172.21.15.116:0/1793761959 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7600096b0 con 0x7fc7540043c0 2024-04-25T00:15:13.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.912+0000 7fc77276b640 1 -- 172.21.15.116:0/1793761959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7540043c0 msgr2=0x7fc754004890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:13.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.912+0000 7fc77276b640 1 --2- 172.21.15.116:0/1793761959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7540043c0 0x7fc754004890 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fc760009ae0 tx=0x7fc76001ca70 comp rx=0 tx=0).stop 2024-04-25T00:15:13.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.912+0000 7fc77276b640 1 -- 172.21.15.116:0/1793761959 shutdown_connections 2024-04-25T00:15:13.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.912+0000 7fc77276b640 1 --2- 172.21.15.116:0/1793761959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7540043c0 0x7fc754004890 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:13.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.912+0000 7fc77276b640 1 -- 172.21.15.116:0/1793761959 >> 172.21.15.116:0/1793761959 conn(0x7fc76c0ad010 msgr2=0x7fc76c0ad420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:13.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.912+0000 7fc77276b640 1 -- 172.21.15.116:0/1793761959 shutdown_connections 2024-04-25T00:15:13.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.912+0000 7fc77276b640 1 -- 172.21.15.116:0/1793761959 wait complete. 2024-04-25T00:15:13.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.916+0000 7fc77276b640 1 Processor -- start 2024-04-25T00:15:13.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.916+0000 7fc77276b640 1 -- start start 2024-04-25T00:15:13.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.916+0000 7fc77276b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc7540043c0 0x7fc76c0c1430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:13.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.916+0000 7fc77276b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc76c149600 0x7fc76c149a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:13.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.916+0000 7fc77276b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc76c1ee530 0x7fc76c1ee9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:13.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.916+0000 7fc77276b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc76c0c7990 con 0x7fc76c149600 2024-04-25T00:15:13.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.916+0000 7fc77276b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc76c0c7810 con 0x7fc76c1ee530 2024-04-25T00:15:13.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.916+0000 7fc77276b640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fc76c1533b0 con 0x7fc7540043c0 2024-04-25T00:15:13.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.916+0000 7fc76b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc76c149600 0x7fc76c149a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:13.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.916+0000 7fc76b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc76c149600 0x7fc76c149a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45078/0 (socket says 172.21.15.116:45078) 2024-04-25T00:15:13.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.916+0000 7fc76b7fe640 1 -- 172.21.15.116:0/988225464 learned_addr learned my addr 172.21.15.116:0/988225464 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:15:13.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.916+0000 7fc76bfff640 1 --2- 172.21.15.116:0/988225464 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc7540043c0 0x7fc76c0c1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:13.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.916+0000 7fc770d08640 1 --2- 172.21.15.116:0/988225464 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc76c1ee530 0x7fc76c1ee9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:13.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.916+0000 7fc76b7fe640 1 -- 172.21.15.116:0/988225464 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc7540043c0 msgr2=0x7fc76c0c1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:13.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.916+0000 7fc76b7fe640 1 --2- 172.21.15.116:0/988225464 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc7540043c0 0x7fc76c0c1430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:13.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.916+0000 7fc76b7fe640 1 -- 172.21.15.116:0/988225464 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc76c1ee530 msgr2=0x7fc76c1ee9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:13.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.916+0000 7fc76b7fe640 1 --2- 172.21.15.116:0/988225464 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc76c1ee530 0x7fc76c1ee9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:13.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.916+0000 7fc76b7fe640 1 -- 172.21.15.116:0/988225464 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc76c157a00 con 0x7fc76c149600 2024-04-25T00:15:13.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.916+0000 7fc76b7fe640 1 --2- 172.21.15.116:0/988225464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc76c149600 0x7fc76c149a60 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fc76001c8a0 tx=0x7fc760019810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:13.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.916+0000 7fc768ff9640 1 -- 172.21.15.116:0/988225464 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7600b2040 con 0x7fc76c149600 2024-04-25T00:15:13.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.916+0000 7fc768ff9640 1 -- 172.21.15.116:0/988225464 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc760004260 con 0x7fc76c149600 2024-04-25T00:15:13.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.916+0000 7fc77276b640 1 -- 172.21.15.116:0/988225464 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc76c157ce0 con 0x7fc76c149600 2024-04-25T00:15:13.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.916+0000 7fc768ff9640 1 -- 172.21.15.116:0/988225464 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc760003620 con 0x7fc76c149600 2024-04-25T00:15:13.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.916+0000 7fc77276b640 1 -- 172.21.15.116:0/988225464 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc76c0c0810 con 0x7fc76c149600 2024-04-25T00:15:13.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.920+0000 7fc77276b640 1 -- 172.21.15.116:0/988225464 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc76c068a60 con 0x7fc76c149600 2024-04-25T00:15:13.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.920+0000 7fc768ff9640 1 -- 172.21.15.116:0/988225464 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc7600bf060 con 0x7fc76c149600 2024-04-25T00:15:13.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.920+0000 7fc768ff9640 1 --2- 172.21.15.116:0/988225464 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc7580678c0 0x7fc758069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:13.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.920+0000 7fc768ff9640 1 -- 172.21.15.116:0/988225464 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 8300+0+0 (secure 0 0 0) 0x7fc760120300 con 0x7fc76c149600 2024-04-25T00:15:13.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.920+0000 7fc76bfff640 1 --2- 172.21.15.116:0/988225464 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc7580678c0 0x7fc758069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:13.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.920+0000 7fc76bfff640 1 --2- 172.21.15.116:0/988225464 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc7580678c0 0x7fc758069d80 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fc75c001cf0 tx=0x7fc75c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:13.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:13.924+0000 7fc768ff9640 1 -- 172.21.15.116:0/988225464 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fc760016780 con 0x7fc76c149600 2024-04-25T00:15:14.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:14.216+0000 7fc77276b640 1 -- 172.21.15.116:0/988225464 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc76c0c5510 con 0x7fc76c149600 2024-04-25T00:15:14.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:14.220+0000 7fc768ff9640 1 -- 172.21.15.116:0/988225464 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) ==== 95+0+6509 (secure 0 0 0) 0x7fc7600f03d0 con 0x7fc76c149600 2024-04-25T00:15:14.219 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:15:14.220 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":107,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6810,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/2340581953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2340581953},{"type":"v1","addr":"172.21.15.116:6833","nonce":2340581953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106},{"gid":6828,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/2012470875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2012470875},{"type":"v1","addr":"172.21.15.116:6835","nonce":2012470875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:15:01.877414+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5273":{"gid":5273,"name":"d","rank":0,"incarnation":102,"state":"up:active","state_seq":49,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5273,"qdb_cluster":[5273]},"id":1},{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:14:58.851744+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5791},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5791":{"gid":5791,"name":"b","rank":0,"incarnation":98,"state":"up:active","state_seq":23,"addr":"172.21.15.174:6837/3180024481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3180024481},{"type":"v1","addr":"172.21.15.174:6837","nonce":3180024481}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5791,"qdb_cluster":[5791]},"id":2}]} 2024-04-25T00:15:14.220 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 107 2024-04-25T00:15:14.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:14.220+0000 7fc77276b640 1 -- 172.21.15.116:0/988225464 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc7580678c0 msgr2=0x7fc758069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:14.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:14.220+0000 7fc77276b640 1 --2- 172.21.15.116:0/988225464 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc7580678c0 0x7fc758069d80 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fc75c001cf0 tx=0x7fc75c009040 comp rx=0 tx=0).stop 2024-04-25T00:15:14.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:14.220+0000 7fc77276b640 1 -- 172.21.15.116:0/988225464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc76c149600 msgr2=0x7fc76c149a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:14.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:14.220+0000 7fc77276b640 1 --2- 172.21.15.116:0/988225464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc76c149600 0x7fc76c149a60 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fc76001c8a0 tx=0x7fc760019810 comp rx=0 tx=0).stop 2024-04-25T00:15:14.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:14.220+0000 7fc77276b640 1 -- 172.21.15.116:0/988225464 shutdown_connections 2024-04-25T00:15:14.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:14.220+0000 7fc77276b640 1 --2- 172.21.15.116:0/988225464 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc7580678c0 0x7fc758069d80 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:14.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:14.220+0000 7fc77276b640 1 --2- 172.21.15.116:0/988225464 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc76c1ee530 0x7fc76c1ee9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:14.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:14.220+0000 7fc77276b640 1 --2- 172.21.15.116:0/988225464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc76c149600 0x7fc76c149a60 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:14.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:14.220+0000 7fc77276b640 1 --2- 172.21.15.116:0/988225464 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc7540043c0 0x7fc76c0c1430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:14.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:14.220+0000 7fc77276b640 1 -- 172.21.15.116:0/988225464 >> 172.21.15.116:0/988225464 conn(0x7fc76c0ad010 msgr2=0x7fc76c154f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:14.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:14.220+0000 7fc77276b640 1 -- 172.21.15.116:0/988225464 shutdown_connections 2024-04-25T00:15:14.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:14.220+0000 7fc77276b640 1 -- 172.21.15.116:0/988225464 wait complete. 2024-04-25T00:15:14.239 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T00:15:16.240 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:15:16.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.380+0000 7f77d696c640 1 Processor -- start 2024-04-25T00:15:16.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.380+0000 7f77d696c640 1 -- start start 2024-04-25T00:15:16.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.380+0000 7f77d696c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f77d0150d10 0x7f77d00c3890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:16.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.380+0000 7f77d696c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f77d00a8280 0x7f77d0156d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:16.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.380+0000 7f77d696c640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77d0157630 con 0x7f77d01510f0 2024-04-25T00:15:16.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.380+0000 7f77d696c640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f77d0154760 con 0x7f77d00a8280 2024-04-25T00:15:16.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.380+0000 7f77d696c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f77d0157910 con 0x7f77d0150d10 2024-04-25T00:15:16.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.384+0000 7f77cf7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f77d00a8280 0x7f77d0156d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:16.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.384+0000 7f77cf7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f77d00a8280 0x7f77d0156d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39406/0 (socket says 172.21.15.116:39406) 2024-04-25T00:15:16.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.384+0000 7f77cf7fe640 1 -- 172.21.15.116:0/3084797133 learned_addr learned my addr 172.21.15.116:0/3084797133 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:15:16.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.384+0000 7f77cffff640 1 --2- 172.21.15.116:0/3084797133 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f77d0150d10 0x7f77d00c3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:16.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.384+0000 7f77cf7fe640 1 -- 172.21.15.116:0/3084797133 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f77d0150d10 msgr2=0x7f77d00c3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:16.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.384+0000 7f77cf7fe640 1 --2- 172.21.15.116:0/3084797133 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f77d0150d10 0x7f77d00c3890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:16.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.384+0000 7f77cf7fe640 1 -- 172.21.15.116:0/3084797133 >> v1:172.21.15.174:6789/0 conn(0x7f77d01510f0 legacy=0x7f77d0154050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:16.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.384+0000 7f77cf7fe640 1 -- 172.21.15.116:0/3084797133 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77d0157bf0 con 0x7f77d00a8280 2024-04-25T00:15:16.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.384+0000 7f77cf7fe640 1 --2- 172.21.15.116:0/3084797133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77d00a8280 0x7f77d0156d30 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f77c4009870 tx=0x7f77c401c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=659b8cf37819095b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:16.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.384+0000 7f77ceffd640 1 -- 172.21.15.116:0/3084797133 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77c40b1070 con 0x7f77d00a8280 2024-04-25T00:15:16.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.384+0000 7f77ceffd640 1 -- 172.21.15.116:0/3084797133 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77c40ac420 con 0x7f77d00a8280 2024-04-25T00:15:16.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.384+0000 7f77ceffd640 1 -- 172.21.15.116:0/3084797133 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77c40ac7f0 con 0x7f77d00a8280 2024-04-25T00:15:16.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.384+0000 7f77d696c640 1 -- 172.21.15.116:0/3084797133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77d00a8280 msgr2=0x7f77d0156d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:16.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.384+0000 7f77d696c640 1 --2- 172.21.15.116:0/3084797133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77d00a8280 0x7f77d0156d30 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f77c4009870 tx=0x7f77c401c400 comp rx=0 tx=0).stop 2024-04-25T00:15:16.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.384+0000 7f77d696c640 1 -- 172.21.15.116:0/3084797133 shutdown_connections 2024-04-25T00:15:16.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.384+0000 7f77d696c640 1 --2- 172.21.15.116:0/3084797133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77d00a8280 0x7f77d0156d30 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:16.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.384+0000 7f77d696c640 1 --2- 172.21.15.116:0/3084797133 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f77d0150d10 0x7f77d00c3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:16.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.384+0000 7f77d696c640 1 -- 172.21.15.116:0/3084797133 >> 172.21.15.116:0/3084797133 conn(0x7f77d0074650 msgr2=0x7f77d0074a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:16.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.384+0000 7f77d696c640 1 -- 172.21.15.116:0/3084797133 shutdown_connections 2024-04-25T00:15:16.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.384+0000 7f77d696c640 1 -- 172.21.15.116:0/3084797133 wait complete. 2024-04-25T00:15:16.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.388+0000 7f77d696c640 1 Processor -- start 2024-04-25T00:15:16.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.388+0000 7f77d696c640 1 -- start start 2024-04-25T00:15:16.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.388+0000 7f77d696c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77d00a8280 0x7f77d014d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:16.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.388+0000 7f77d696c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f77d0150d10 0x7f77d0147eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:16.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.388+0000 7f77d696c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f77d01510f0 0x7f77d01483f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:16.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.388+0000 7f77d696c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f77d0160250 con 0x7f77d00a8280 2024-04-25T00:15:16.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.388+0000 7f77d696c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f77d01600d0 con 0x7f77d0150d10 2024-04-25T00:15:16.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.388+0000 7f77d696c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f77d01603d0 con 0x7f77d01510f0 2024-04-25T00:15:16.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.388+0000 7f77cffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77d00a8280 0x7f77d014d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:16.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.388+0000 7f77cffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77d00a8280 0x7f77d014d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39410/0 (socket says 172.21.15.116:39410) 2024-04-25T00:15:16.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.388+0000 7f77cffff640 1 -- 172.21.15.116:0/1990906843 learned_addr learned my addr 172.21.15.116:0/1990906843 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:15:16.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.388+0000 7f77d4f09640 1 --2- 172.21.15.116:0/1990906843 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f77d01510f0 0x7f77d01483f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:16.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.388+0000 7f77cf7fe640 1 --2- 172.21.15.116:0/1990906843 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f77d0150d10 0x7f77d0147eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:16.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.388+0000 7f77cffff640 1 -- 172.21.15.116:0/1990906843 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f77d01510f0 msgr2=0x7f77d01483f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:16.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.388+0000 7f77cffff640 1 --2- 172.21.15.116:0/1990906843 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f77d01510f0 0x7f77d01483f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:16.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.388+0000 7f77cffff640 1 -- 172.21.15.116:0/1990906843 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f77d0150d10 msgr2=0x7f77d0147eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:16.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.388+0000 7f77cffff640 1 --2- 172.21.15.116:0/1990906843 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f77d0150d10 0x7f77d0147eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:16.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.388+0000 7f77cffff640 1 -- 172.21.15.116:0/1990906843 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77d014abc0 con 0x7f77d00a8280 2024-04-25T00:15:16.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.388+0000 7f77cffff640 1 --2- 172.21.15.116:0/1990906843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77d00a8280 0x7f77d014d2c0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f77c000c7f0 tx=0x7f77c000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:16.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.388+0000 7f77ccff9640 1 -- 172.21.15.116:0/1990906843 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77c0015070 con 0x7f77d00a8280 2024-04-25T00:15:16.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.388+0000 7f77ccff9640 1 -- 172.21.15.116:0/1990906843 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77c0004240 con 0x7f77d00a8280 2024-04-25T00:15:16.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.388+0000 7f77d696c640 1 -- 172.21.15.116:0/1990906843 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77d0149a80 con 0x7f77d00a8280 2024-04-25T00:15:16.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.388+0000 7f77ccff9640 1 -- 172.21.15.116:0/1990906843 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77c0003620 con 0x7f77d00a8280 2024-04-25T00:15:16.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.388+0000 7f77d696c640 1 -- 172.21.15.116:0/1990906843 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77d014ae30 con 0x7f77d00a8280 2024-04-25T00:15:16.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.388+0000 7f77d696c640 1 -- 172.21.15.116:0/1990906843 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77d0068350 con 0x7f77d00a8280 2024-04-25T00:15:16.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.388+0000 7f77ccff9640 1 -- 172.21.15.116:0/1990906843 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f77c00037c0 con 0x7f77d00a8280 2024-04-25T00:15:16.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.392+0000 7f77ccff9640 1 --2- 172.21.15.116:0/1990906843 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f77bc0678c0 0x7f77bc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:16.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.392+0000 7f77ccff9640 1 -- 172.21.15.116:0/1990906843 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 8300+0+0 (secure 0 0 0) 0x7f77c0086540 con 0x7f77d00a8280 2024-04-25T00:15:16.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.392+0000 7f77cf7fe640 1 --2- 172.21.15.116:0/1990906843 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f77bc0678c0 0x7f77bc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:16.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.392+0000 7f77cf7fe640 1 --2- 172.21.15.116:0/1990906843 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f77bc0678c0 0x7f77bc069d80 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f77c40ae000 tx=0x7f77c4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:16.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.396+0000 7f77ccff9640 1 -- 172.21.15.116:0/1990906843 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f77c0056610 con 0x7f77d00a8280 2024-04-25T00:15:16.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.648+0000 7f77d696c640 1 -- 172.21.15.116:0/1990906843 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f77d00a7fd0 con 0x7f77d00a8280 2024-04-25T00:15:16.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.648+0000 7f77ccff9640 1 -- 172.21.15.116:0/1990906843 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) ==== 95+0+6509 (secure 0 0 0) 0x7f77c0055f10 con 0x7f77d00a8280 2024-04-25T00:15:16.650 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:15:16.651 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":107,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6810,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/2340581953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2340581953},{"type":"v1","addr":"172.21.15.116:6833","nonce":2340581953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106},{"gid":6828,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/2012470875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2012470875},{"type":"v1","addr":"172.21.15.116:6835","nonce":2012470875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:15:01.877414+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5273":{"gid":5273,"name":"d","rank":0,"incarnation":102,"state":"up:active","state_seq":49,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5273,"qdb_cluster":[5273]},"id":1},{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:14:58.851744+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5791},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5791":{"gid":5791,"name":"b","rank":0,"incarnation":98,"state":"up:active","state_seq":23,"addr":"172.21.15.174:6837/3180024481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3180024481},{"type":"v1","addr":"172.21.15.174:6837","nonce":3180024481}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5791,"qdb_cluster":[5791]},"id":2}]} 2024-04-25T00:15:16.651 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 107 2024-04-25T00:15:16.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.652+0000 7f77d696c640 1 -- 172.21.15.116:0/1990906843 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f77bc0678c0 msgr2=0x7f77bc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:16.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.652+0000 7f77d696c640 1 --2- 172.21.15.116:0/1990906843 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f77bc0678c0 0x7f77bc069d80 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f77c40ae000 tx=0x7f77c4008040 comp rx=0 tx=0).stop 2024-04-25T00:15:16.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.652+0000 7f77d696c640 1 -- 172.21.15.116:0/1990906843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77d00a8280 msgr2=0x7f77d014d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:16.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.652+0000 7f77d696c640 1 --2- 172.21.15.116:0/1990906843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77d00a8280 0x7f77d014d2c0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f77c000c7f0 tx=0x7f77c000ccb0 comp rx=0 tx=0).stop 2024-04-25T00:15:16.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.652+0000 7f77d696c640 1 -- 172.21.15.116:0/1990906843 shutdown_connections 2024-04-25T00:15:16.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.652+0000 7f77d696c640 1 --2- 172.21.15.116:0/1990906843 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f77bc0678c0 0x7f77bc069d80 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:16.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.652+0000 7f77d696c640 1 --2- 172.21.15.116:0/1990906843 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f77d01510f0 0x7f77d01483f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:16.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.652+0000 7f77d696c640 1 --2- 172.21.15.116:0/1990906843 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f77d0150d10 0x7f77d0147eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:16.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.652+0000 7f77d696c640 1 --2- 172.21.15.116:0/1990906843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f77d00a8280 0x7f77d014d2c0 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:16.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.652+0000 7f77d696c640 1 -- 172.21.15.116:0/1990906843 >> 172.21.15.116:0/1990906843 conn(0x7f77d0074650 msgr2=0x7f77d0152940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:16.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.652+0000 7f77d696c640 1 -- 172.21.15.116:0/1990906843 shutdown_connections 2024-04-25T00:15:16.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:16.652+0000 7f77d696c640 1 -- 172.21.15.116:0/1990906843 wait complete. 2024-04-25T00:15:16.674 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T00:15:18.675 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:15:18.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.812+0000 7fa48406b640 1 Processor -- start 2024-04-25T00:15:18.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.812+0000 7fa48406b640 1 -- start start 2024-04-25T00:15:18.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.812+0000 7fa48406b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa47c0c5a60 0x7fa47c0c5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:18.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.812+0000 7fa48406b640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fa47c1558c0 0x7fa47c155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:18.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.812+0000 7fa48406b640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa47c158eb0 con 0x7fa47c152db0 2024-04-25T00:15:18.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.812+0000 7fa48406b640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fa47c159190 con 0x7fa47c0c5a60 2024-04-25T00:15:18.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.812+0000 7fa48406b640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fa47c159470 con 0x7fa47c1558c0 2024-04-25T00:15:18.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.812+0000 7fa481606640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa47c0c5a60 0x7fa47c0c5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:18.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.812+0000 7fa481606640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa47c0c5a60 0x7fa47c0c5ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39418/0 (socket says 172.21.15.116:39418) 2024-04-25T00:15:18.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.812+0000 7fa481e07640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fa47c152db0 0x7fa47c0c7af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:43518/0 (socket says 172.21.15.116:43518) 2024-04-25T00:15:18.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.812+0000 7fa481606640 1 -- 172.21.15.116:0/1787158152 learned_addr learned my addr 172.21.15.116:0/1787158152 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:15:18.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.812+0000 7fa481606640 1 -- 172.21.15.116:0/1787158152 >> v2:172.21.15.174:3300/0 conn(0x7fa47c1558c0 msgr2=0x7fa47c155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:18.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.812+0000 7fa481606640 1 --2- 172.21.15.116:0/1787158152 >> v2:172.21.15.174:3300/0 conn(0x7fa47c1558c0 0x7fa47c155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:18.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.812+0000 7fa481606640 1 -- 172.21.15.116:0/1787158152 >> v1:172.21.15.116:6789/0 conn(0x7fa47c152db0 legacy=0x7fa47c0c7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:18.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.812+0000 7fa481606640 1 -- 172.21.15.116:0/1787158152 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa47c159750 con 0x7fa47c0c5a60 2024-04-25T00:15:18.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.816+0000 7fa481606640 1 --2- 172.21.15.116:0/1787158152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa47c0c5a60 0x7fa47c0c5ef0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fa47c0b1230 tx=0x7fa46c01cec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59ded59701bafc9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:18.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.816+0000 7fa480e05640 1 -- 172.21.15.116:0/1787158152 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa46c0ab680 con 0x7fa47c0c5a60 2024-04-25T00:15:18.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.816+0000 7fa480e05640 1 -- 172.21.15.116:0/1787158152 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa46c0b0070 con 0x7fa47c0c5a60 2024-04-25T00:15:18.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.816+0000 7fa480e05640 1 -- 172.21.15.116:0/1787158152 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa46c0aba40 con 0x7fa47c0c5a60 2024-04-25T00:15:18.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.816+0000 7fa48406b640 1 -- 172.21.15.116:0/1787158152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa47c0c5a60 msgr2=0x7fa47c0c5ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:18.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.816+0000 7fa48406b640 1 --2- 172.21.15.116:0/1787158152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa47c0c5a60 0x7fa47c0c5ef0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fa47c0b1230 tx=0x7fa46c01cec0 comp rx=0 tx=0).stop 2024-04-25T00:15:18.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.816+0000 7fa48406b640 1 -- 172.21.15.116:0/1787158152 shutdown_connections 2024-04-25T00:15:18.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.816+0000 7fa48406b640 1 --2- 172.21.15.116:0/1787158152 >> v2:172.21.15.174:3300/0 conn(0x7fa47c1558c0 0x7fa47c155cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:18.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.816+0000 7fa48406b640 1 --2- 172.21.15.116:0/1787158152 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa47c0c5a60 0x7fa47c0c5ef0 secure :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fa47c0b1230 tx=0x7fa46c01cec0 comp rx=0 tx=0).stop 2024-04-25T00:15:18.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.816+0000 7fa48406b640 1 -- 172.21.15.116:0/1787158152 >> 172.21.15.116:0/1787158152 conn(0x7fa47c0acc10 msgr2=0x7fa47c0ad020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:18.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.820+0000 7fa48406b640 1 -- 172.21.15.116:0/1787158152 shutdown_connections 2024-04-25T00:15:18.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.820+0000 7fa48406b640 1 -- 172.21.15.116:0/1787158152 wait complete. 2024-04-25T00:15:18.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.820+0000 7fa48406b640 1 Processor -- start 2024-04-25T00:15:18.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.820+0000 7fa48406b640 1 -- start start 2024-04-25T00:15:18.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.820+0000 7fa48406b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa47c152db0 0x7fa47c14ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:18.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.820+0000 7fa48406b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa47c1558c0 0x7fa47c153560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:18.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.820+0000 7fa48406b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa47c0ac5f0 0x7fa47c1f2400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:18.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.820+0000 7fa48406b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa47c153310 con 0x7fa47c152db0 2024-04-25T00:15:18.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.820+0000 7fa48406b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa47c153190 con 0x7fa47c1558c0 2024-04-25T00:15:18.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.820+0000 7fa48406b640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa47c162a60 con 0x7fa47c0ac5f0 2024-04-25T00:15:18.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.820+0000 7fa481e07640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa47c152db0 0x7fa47c14ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:18.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.820+0000 7fa481e07640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa47c152db0 0x7fa47c14ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39428/0 (socket says 172.21.15.116:39428) 2024-04-25T00:15:18.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.820+0000 7fa481e07640 1 -- 172.21.15.116:0/846415944 learned_addr learned my addr 172.21.15.116:0/846415944 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:15:18.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.824+0000 7fa481606640 1 --2- 172.21.15.116:0/846415944 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa47c1558c0 0x7fa47c153560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:18.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.824+0000 7fa482608640 1 --2- 172.21.15.116:0/846415944 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa47c0ac5f0 0x7fa47c1f2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:18.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.824+0000 7fa481e07640 1 -- 172.21.15.116:0/846415944 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa47c0ac5f0 msgr2=0x7fa47c1f2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:18.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.824+0000 7fa481e07640 1 --2- 172.21.15.116:0/846415944 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa47c0ac5f0 0x7fa47c1f2400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:18.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.824+0000 7fa481e07640 1 -- 172.21.15.116:0/846415944 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa47c1558c0 msgr2=0x7fa47c153560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:18.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.824+0000 7fa481e07640 1 --2- 172.21.15.116:0/846415944 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa47c1558c0 0x7fa47c153560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:18.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.824+0000 7fa481e07640 1 -- 172.21.15.116:0/846415944 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa47c14c6d0 con 0x7fa47c152db0 2024-04-25T00:15:18.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.824+0000 7fa481606640 1 --2- 172.21.15.116:0/846415944 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa47c1558c0 0x7fa47c153560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:15:18.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.824+0000 7fa481e07640 1 --2- 172.21.15.116:0/846415944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa47c152db0 0x7fa47c14ab00 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fa47800c790 tx=0x7fa47800cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:18.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.824+0000 7fa4727fc640 1 -- 172.21.15.116:0/846415944 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa478017070 con 0x7fa47c152db0 2024-04-25T00:15:18.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.824+0000 7fa4727fc640 1 -- 172.21.15.116:0/846415944 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa47800a640 con 0x7fa47c152db0 2024-04-25T00:15:18.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.824+0000 7fa48406b640 1 -- 172.21.15.116:0/846415944 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa47c14c970 con 0x7fa47c152db0 2024-04-25T00:15:18.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.824+0000 7fa4727fc640 1 -- 172.21.15.116:0/846415944 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa47800a9f0 con 0x7fa47c152db0 2024-04-25T00:15:18.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.824+0000 7fa48406b640 1 -- 172.21.15.116:0/846415944 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa47c1ef830 con 0x7fa47c152db0 2024-04-25T00:15:18.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.824+0000 7fa48406b640 1 -- 172.21.15.116:0/846415944 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa47c1f2040 con 0x7fa47c152db0 2024-04-25T00:15:18.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.824+0000 7fa4727fc640 1 -- 172.21.15.116:0/846415944 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa47801a070 con 0x7fa47c152db0 2024-04-25T00:15:18.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.824+0000 7fa4727fc640 1 --2- 172.21.15.116:0/846415944 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa4600678c0 0x7fa460069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:18.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.824+0000 7fa4727fc640 1 -- 172.21.15.116:0/846415944 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 8300+0+0 (secure 0 0 0) 0x7fa478085bf0 con 0x7fa47c152db0 2024-04-25T00:15:18.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.824+0000 7fa481606640 1 --2- 172.21.15.116:0/846415944 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa4600678c0 0x7fa460069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:18.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.828+0000 7fa481606640 1 --2- 172.21.15.116:0/846415944 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa4600678c0 0x7fa460069d80 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fa46c0af000 tx=0x7fa46c004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:18.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:18.828+0000 7fa4727fc640 1 -- 172.21.15.116:0/846415944 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fa478055700 con 0x7fa47c152db0 2024-04-25T00:15:19.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:19.056+0000 7fa48406b640 1 -- 172.21.15.116:0/846415944 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa47c068a60 con 0x7fa47c152db0 2024-04-25T00:15:19.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:19.056+0000 7fa4727fc640 1 -- 172.21.15.116:0/846415944 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) ==== 95+0+6509 (secure 0 0 0) 0x7fa478059a00 con 0x7fa47c152db0 2024-04-25T00:15:19.056 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:15:19.056 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":107,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6810,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/2340581953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2340581953},{"type":"v1","addr":"172.21.15.116:6833","nonce":2340581953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106},{"gid":6828,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/2012470875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2012470875},{"type":"v1","addr":"172.21.15.116:6835","nonce":2012470875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:15:01.877414+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5273":{"gid":5273,"name":"d","rank":0,"incarnation":102,"state":"up:active","state_seq":49,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5273,"qdb_cluster":[5273]},"id":1},{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:14:58.851744+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5791},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5791":{"gid":5791,"name":"b","rank":0,"incarnation":98,"state":"up:active","state_seq":23,"addr":"172.21.15.174:6837/3180024481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3180024481},{"type":"v1","addr":"172.21.15.174:6837","nonce":3180024481}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5791,"qdb_cluster":[5791]},"id":2}]} 2024-04-25T00:15:19.057 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 107 2024-04-25T00:15:19.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:19.056+0000 7fa48406b640 1 -- 172.21.15.116:0/846415944 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa4600678c0 msgr2=0x7fa460069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:19.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:19.056+0000 7fa48406b640 1 --2- 172.21.15.116:0/846415944 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa4600678c0 0x7fa460069d80 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fa46c0af000 tx=0x7fa46c004740 comp rx=0 tx=0).stop 2024-04-25T00:15:19.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:19.056+0000 7fa48406b640 1 -- 172.21.15.116:0/846415944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa47c152db0 msgr2=0x7fa47c14ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:19.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:19.056+0000 7fa48406b640 1 --2- 172.21.15.116:0/846415944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa47c152db0 0x7fa47c14ab00 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fa47800c790 tx=0x7fa47800cc50 comp rx=0 tx=0).stop 2024-04-25T00:15:19.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:19.056+0000 7fa48406b640 1 -- 172.21.15.116:0/846415944 shutdown_connections 2024-04-25T00:15:19.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:19.056+0000 7fa48406b640 1 --2- 172.21.15.116:0/846415944 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa4600678c0 0x7fa460069d80 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:19.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:19.056+0000 7fa48406b640 1 --2- 172.21.15.116:0/846415944 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa47c0ac5f0 0x7fa47c1f2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:19.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:19.056+0000 7fa48406b640 1 --2- 172.21.15.116:0/846415944 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa47c1558c0 0x7fa47c153560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:19.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:19.056+0000 7fa48406b640 1 --2- 172.21.15.116:0/846415944 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa47c152db0 0x7fa47c14ab00 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:19.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:19.060+0000 7fa48406b640 1 -- 172.21.15.116:0/846415944 >> 172.21.15.116:0/846415944 conn(0x7fa47c0acc10 msgr2=0x7fa47c0c7af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:19.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:19.060+0000 7fa48406b640 1 -- 172.21.15.116:0/846415944 shutdown_connections 2024-04-25T00:15:19.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:19.060+0000 7fa48406b640 1 -- 172.21.15.116:0/846415944 wait complete. 2024-04-25T00:15:19.075 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T00:15:21.077 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:15:21.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.216+0000 7fdffa71d640 1 Processor -- start 2024-04-25T00:15:21.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.216+0000 7fdffa71d640 1 -- start start 2024-04-25T00:15:21.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.220+0000 7fdffa71d640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fdff4152e60 0x7fdff41599a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:21.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.220+0000 7fdffa71d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdff41562d0 0x7fdff40c36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:21.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.220+0000 7fdffa71d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdff4073200 con 0x7fdff40c3c20 2024-04-25T00:15:21.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.220+0000 7fdffa71d640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fdff4153240 con 0x7fdff4152e60 2024-04-25T00:15:21.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.220+0000 7fdffa71d640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fdff40734e0 con 0x7fdff41562d0 2024-04-25T00:15:21.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.220+0000 7fdff8cba640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fdff40c3c20 0x7fdff40c5010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:43520/0 (socket says 172.21.15.116:43520) 2024-04-25T00:15:21.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.220+0000 7fdff8cba640 1 -- 172.21.15.116:0/3091494850 learned_addr learned my addr 172.21.15.116:0/3091494850 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:15:21.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.220+0000 7fdff3fff640 1 --2- 172.21.15.116:0/3091494850 >> v2:172.21.15.174:3300/0 conn(0x7fdff4152e60 0x7fdff41599a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:21.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.220+0000 7fdff37fe640 1 --2- 172.21.15.116:0/3091494850 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdff41562d0 0x7fdff40c36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:21.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.220+0000 7fdff2ffd640 1 -- 172.21.15.116:0/3091494850 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 12552043 0 0) 0x7fdff4073200 con 0x7fdff40c3c20 2024-04-25T00:15:21.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.220+0000 7fdff2ffd640 1 -- 172.21.15.116:0/3091494850 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdfdc003530 con 0x7fdff40c3c20 2024-04-25T00:15:21.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.220+0000 7fdff2ffd640 1 -- 172.21.15.116:0/3091494850 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1665876255 0 0) 0x7fdfdc003530 con 0x7fdff40c3c20 2024-04-25T00:15:21.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.220+0000 7fdff2ffd640 1 -- 172.21.15.116:0/3091494850 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdff41562d0 msgr2=0x7fdff40c36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:21.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.220+0000 7fdff2ffd640 1 --2- 172.21.15.116:0/3091494850 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdff41562d0 0x7fdff40c36e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:21.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.220+0000 7fdff2ffd640 1 -- 172.21.15.116:0/3091494850 >> v2:172.21.15.174:3300/0 conn(0x7fdff4152e60 msgr2=0x7fdff41599a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:21.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.220+0000 7fdff2ffd640 1 --2- 172.21.15.116:0/3091494850 >> v2:172.21.15.174:3300/0 conn(0x7fdff4152e60 0x7fdff41599a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:21.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.220+0000 7fdff2ffd640 1 -- 172.21.15.116:0/3091494850 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdff4158ec0 con 0x7fdff40c3c20 2024-04-25T00:15:21.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.220+0000 7fdff2ffd640 1 -- 172.21.15.116:0/3091494850 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7fdfec001a00 con 0x7fdff40c3c20 2024-04-25T00:15:21.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.220+0000 7fdff2ffd640 1 -- 172.21.15.116:0/3091494850 >> v1:172.21.15.116:6789/0 conn(0x7fdff40c3c20 legacy=0x7fdff40c5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:21.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.220+0000 7fdff2ffd640 1 --2- 172.21.15.116:0/3091494850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfdc0039c0 0x7fdfdc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:21.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.220+0000 7fdff2ffd640 1 --2- 172.21.15.116:0/3091494850 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdfdc004810 0x7fdfdc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:21.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.220+0000 7fdff2ffd640 1 --2- 172.21.15.116:0/3091494850 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdff41562d0 0x7fdfdc0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:21.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.220+0000 7fdff2ffd640 1 -- 172.21.15.116:0/3091494850 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdff4153240 con 0x7fdfdc0039c0 2024-04-25T00:15:21.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.220+0000 7fdff2ffd640 1 -- 172.21.15.116:0/3091494850 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fdff40734e0 con 0x7fdfdc004810 2024-04-25T00:15:21.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.220+0000 7fdff2ffd640 1 -- 172.21.15.116:0/3091494850 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fdfdc00ba20 con 0x7fdff41562d0 2024-04-25T00:15:21.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.220+0000 7fdff3fff640 1 --2- 172.21.15.116:0/3091494850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfdc0039c0 0x7fdfdc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:21.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.220+0000 7fdff37fe640 1 --2- 172.21.15.116:0/3091494850 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdfdc004810 0x7fdfdc004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:21.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.220+0000 7fdff3fff640 1 -- 172.21.15.116:0/3091494850 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdff41562d0 msgr2=0x7fdfdc0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:21.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.220+0000 7fdff3fff640 1 --2- 172.21.15.116:0/3091494850 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdff41562d0 0x7fdfdc0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:21.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.220+0000 7fdff3fff640 1 -- 172.21.15.116:0/3091494850 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdfdc004810 msgr2=0x7fdfdc004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:21.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.220+0000 7fdff3fff640 1 --2- 172.21.15.116:0/3091494850 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdfdc004810 0x7fdfdc004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:21.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.220+0000 7fdff3fff640 1 -- 172.21.15.116:0/3091494850 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfdc003530 con 0x7fdfdc0039c0 2024-04-25T00:15:21.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.224+0000 7fdff3fff640 1 --2- 172.21.15.116:0/3091494850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfdc0039c0 0x7fdfdc003de0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fdfe4009b10 tx=0x7fdfe401cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0e56e5309c91fa2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:21.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.224+0000 7fdff2ffd640 1 -- 172.21.15.116:0/3091494850 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfe40096b0 con 0x7fdfdc0039c0 2024-04-25T00:15:21.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.224+0000 7fdff2ffd640 1 -- 172.21.15.116:0/3091494850 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfe40a8410 con 0x7fdfdc0039c0 2024-04-25T00:15:21.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.224+0000 7fdff2ffd640 1 -- 172.21.15.116:0/3091494850 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfe40a8800 con 0x7fdfdc0039c0 2024-04-25T00:15:21.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.224+0000 7fdffa71d640 1 -- 172.21.15.116:0/3091494850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfdc0039c0 msgr2=0x7fdfdc003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:21.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.224+0000 7fdffa71d640 1 --2- 172.21.15.116:0/3091494850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdfdc0039c0 0x7fdfdc003de0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fdfe4009b10 tx=0x7fdfe401cab0 comp rx=0 tx=0).stop 2024-04-25T00:15:21.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.224+0000 7fdff8cba640 1 -- 172.21.15.116:0/3091494850 reap_dead start 2024-04-25T00:15:21.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.224+0000 7fdffa71d640 1 -- 172.21.15.116:0/3091494850 shutdown_connections 2024-04-25T00:15:21.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.224+0000 7fdffa71d640 1 -- 172.21.15.116:0/3091494850 >> 172.21.15.116:0/3091494850 conn(0x7fdff406c720 msgr2=0x7fdff406cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:21.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.224+0000 7fdffa71d640 1 -- 172.21.15.116:0/3091494850 shutdown_connections 2024-04-25T00:15:21.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.224+0000 7fdffa71d640 1 -- 172.21.15.116:0/3091494850 wait complete. 2024-04-25T00:15:21.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.224+0000 7fdffa71d640 1 Processor -- start 2024-04-25T00:15:21.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.224+0000 7fdffa71d640 1 -- start start 2024-04-25T00:15:21.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.224+0000 7fdffa71d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdfdc0039c0 0x7fdff40acb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:21.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.224+0000 7fdffa71d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdff40c6d50 0x7fdff4157880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:21.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.224+0000 7fdffa71d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdff40ab8e0 0x7fdff414c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:21.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.224+0000 7fdffa71d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdff40c5720 con 0x7fdff40ab8e0 2024-04-25T00:15:21.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.224+0000 7fdffa71d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fdff4072080 con 0x7fdff40c6d50 2024-04-25T00:15:21.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.224+0000 7fdffa71d640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fdff40c58a0 con 0x7fdfdc0039c0 2024-04-25T00:15:21.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.224+0000 7fdff8cba640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdff40ab8e0 0x7fdff414c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:21.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.224+0000 7fdff8cba640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdff40ab8e0 0x7fdff414c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39450/0 (socket says 172.21.15.116:39450) 2024-04-25T00:15:21.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.224+0000 7fdff8cba640 1 -- 172.21.15.116:0/1380647303 learned_addr learned my addr 172.21.15.116:0/1380647303 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:15:21.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.224+0000 7fdff3fff640 1 --2- 172.21.15.116:0/1380647303 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdfdc0039c0 0x7fdff40acb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:21.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.228+0000 7fdff37fe640 1 --2- 172.21.15.116:0/1380647303 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdff40c6d50 0x7fdff4157880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:21.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.228+0000 7fdff8cba640 1 -- 172.21.15.116:0/1380647303 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdfdc0039c0 msgr2=0x7fdff40acb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:21.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.228+0000 7fdff8cba640 1 --2- 172.21.15.116:0/1380647303 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdfdc0039c0 0x7fdff40acb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:21.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.228+0000 7fdff8cba640 1 -- 172.21.15.116:0/1380647303 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdff40c6d50 msgr2=0x7fdff4157880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:21.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.228+0000 7fdff8cba640 1 --2- 172.21.15.116:0/1380647303 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdff40c6d50 0x7fdff4157880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:21.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.228+0000 7fdff8cba640 1 -- 172.21.15.116:0/1380647303 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdff40c7e90 con 0x7fdff40ab8e0 2024-04-25T00:15:21.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.228+0000 7fdff3fff640 1 --2- 172.21.15.116:0/1380647303 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdfdc0039c0 0x7fdff40acb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:15:21.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.228+0000 7fdff8cba640 1 --2- 172.21.15.116:0/1380647303 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdff40ab8e0 0x7fdff414c990 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fdff40c5c20 tx=0x7fdfec00ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:21.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.228+0000 7fdff0ff9640 1 -- 172.21.15.116:0/1380647303 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfec00a5b0 con 0x7fdff40ab8e0 2024-04-25T00:15:21.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.228+0000 7fdff0ff9640 1 -- 172.21.15.116:0/1380647303 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfec00a750 con 0x7fdff40ab8e0 2024-04-25T00:15:21.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.228+0000 7fdff0ff9640 1 -- 172.21.15.116:0/1380647303 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfec00ab40 con 0x7fdff40ab8e0 2024-04-25T00:15:21.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.228+0000 7fdffa71d640 1 -- 172.21.15.116:0/1380647303 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdff4150e10 con 0x7fdff40ab8e0 2024-04-25T00:15:21.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.228+0000 7fdffa71d640 1 -- 172.21.15.116:0/1380647303 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdff41513d0 con 0x7fdff40ab8e0 2024-04-25T00:15:21.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.228+0000 7fdffa71d640 1 -- 172.21.15.116:0/1380647303 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdff4068a60 con 0x7fdff40ab8e0 2024-04-25T00:15:21.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.228+0000 7fdff0ff9640 1 -- 172.21.15.116:0/1380647303 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fdfec00ace0 con 0x7fdff40ab8e0 2024-04-25T00:15:21.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.228+0000 7fdff0ff9640 1 --2- 172.21.15.116:0/1380647303 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdfe00678c0 0x7fdfe0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:21.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.228+0000 7fdff0ff9640 1 -- 172.21.15.116:0/1380647303 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 8300+0+0 (secure 0 0 0) 0x7fdfec085e70 con 0x7fdff40ab8e0 2024-04-25T00:15:21.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.228+0000 7fdff3fff640 1 --2- 172.21.15.116:0/1380647303 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdfe00678c0 0x7fdfe0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:21.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.228+0000 7fdff3fff640 1 --2- 172.21.15.116:0/1380647303 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdfe00678c0 0x7fdfe0069d80 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fdfe40af000 tx=0x7fdfe40b2040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:21.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.236+0000 7fdff0ff9640 1 -- 172.21.15.116:0/1380647303 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fdfec055f40 con 0x7fdff40ab8e0 2024-04-25T00:15:21.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.468+0000 7fdffa71d640 1 -- 172.21.15.116:0/1380647303 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdff40c24e0 con 0x7fdff40ab8e0 2024-04-25T00:15:21.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.468+0000 7fdff0ff9640 1 -- 172.21.15.116:0/1380647303 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) ==== 95+0+6509 (secure 0 0 0) 0x7fdfec055840 con 0x7fdff40ab8e0 2024-04-25T00:15:21.470 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:15:21.471 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":107,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6810,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/2340581953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2340581953},{"type":"v1","addr":"172.21.15.116:6833","nonce":2340581953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106},{"gid":6828,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/2012470875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2012470875},{"type":"v1","addr":"172.21.15.116:6835","nonce":2012470875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:15:01.877414+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5273":{"gid":5273,"name":"d","rank":0,"incarnation":102,"state":"up:active","state_seq":49,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5273,"qdb_cluster":[5273]},"id":1},{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:14:58.851744+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5791},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5791":{"gid":5791,"name":"b","rank":0,"incarnation":98,"state":"up:active","state_seq":23,"addr":"172.21.15.174:6837/3180024481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3180024481},{"type":"v1","addr":"172.21.15.174:6837","nonce":3180024481}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5791,"qdb_cluster":[5791]},"id":2}]} 2024-04-25T00:15:21.471 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 107 2024-04-25T00:15:21.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.472+0000 7fdffa71d640 1 -- 172.21.15.116:0/1380647303 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdfe00678c0 msgr2=0x7fdfe0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:21.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.472+0000 7fdffa71d640 1 --2- 172.21.15.116:0/1380647303 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdfe00678c0 0x7fdfe0069d80 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fdfe40af000 tx=0x7fdfe40b2040 comp rx=0 tx=0).stop 2024-04-25T00:15:21.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.472+0000 7fdffa71d640 1 -- 172.21.15.116:0/1380647303 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdff40ab8e0 msgr2=0x7fdff414c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:21.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.472+0000 7fdffa71d640 1 --2- 172.21.15.116:0/1380647303 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdff40ab8e0 0x7fdff414c990 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fdff40c5c20 tx=0x7fdfec00ccd0 comp rx=0 tx=0).stop 2024-04-25T00:15:21.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.472+0000 7fdffa71d640 1 -- 172.21.15.116:0/1380647303 shutdown_connections 2024-04-25T00:15:21.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.472+0000 7fdffa71d640 1 --2- 172.21.15.116:0/1380647303 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdfe00678c0 0x7fdfe0069d80 secure :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fdfe40af000 tx=0x7fdfe40b2040 comp rx=0 tx=0).stop 2024-04-25T00:15:21.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.472+0000 7fdffa71d640 1 --2- 172.21.15.116:0/1380647303 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdff40ab8e0 0x7fdff414c990 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:21.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.472+0000 7fdffa71d640 1 --2- 172.21.15.116:0/1380647303 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdff40c6d50 0x7fdff4157880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:21.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.472+0000 7fdffa71d640 1 --2- 172.21.15.116:0/1380647303 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdfdc0039c0 0x7fdff40acb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:21.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.472+0000 7fdffa71d640 1 -- 172.21.15.116:0/1380647303 >> 172.21.15.116:0/1380647303 conn(0x7fdff406c720 msgr2=0x7fdff415f500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:21.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.472+0000 7fdffa71d640 1 -- 172.21.15.116:0/1380647303 shutdown_connections 2024-04-25T00:15:21.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:21.472+0000 7fdffa71d640 1 -- 172.21.15.116:0/1380647303 wait complete. 2024-04-25T00:15:21.490 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T00:15:23.491 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:15:23.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.624+0000 7f6ccd64f640 1 Processor -- start 2024-04-25T00:15:23.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.624+0000 7f6ccd64f640 1 -- start start 2024-04-25T00:15:23.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.628+0000 7f6ccd64f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6cc8152f60 0x7f6cc80c6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:23.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.628+0000 7f6ccd64f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6cc806fb30 0x7f6cc806c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:23.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.628+0000 7f6ccd64f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6cc806ff80 con 0x7f6cc80c3c30 2024-04-25T00:15:23.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.628+0000 7f6ccd64f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f6cc81595b0 con 0x7f6cc806fb30 2024-04-25T00:15:23.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.628+0000 7f6ccd64f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6cc806e8f0 con 0x7f6cc8152f60 2024-04-25T00:15:23.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.628+0000 7f6cc67fc640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f6cc80c3c30 0x7f6cc8158ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:43526/0 (socket says 172.21.15.116:43526) 2024-04-25T00:15:23.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.628+0000 7f6cc67fc640 1 -- 172.21.15.116:0/2206976994 learned_addr learned my addr 172.21.15.116:0/2206976994 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:15:23.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.628+0000 7f6cc77fe640 1 --2- 172.21.15.116:0/2206976994 >> v2:172.21.15.116:3300/0 conn(0x7f6cc806fb30 0x7f6cc806c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:23.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.628+0000 7f6cc6ffd640 1 --2- 172.21.15.116:0/2206976994 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6cc8152f60 0x7f6cc80c6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:23.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.628+0000 7f6cc5ffb640 1 -- 172.21.15.116:0/2206976994 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1219664401 0 0) 0x7f6cc806ff80 con 0x7f6cc80c3c30 2024-04-25T00:15:23.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.628+0000 7f6cc5ffb640 1 -- 172.21.15.116:0/2206976994 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6cb0003530 con 0x7f6cc80c3c30 2024-04-25T00:15:23.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.628+0000 7f6cc77fe640 1 -- 172.21.15.116:0/2206976994 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6cc8152f60 msgr2=0x7f6cc80c6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:23.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.628+0000 7f6cc77fe640 1 --2- 172.21.15.116:0/2206976994 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6cc8152f60 0x7f6cc80c6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:23.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.628+0000 7f6cc77fe640 1 -- 172.21.15.116:0/2206976994 >> v1:172.21.15.116:6789/0 conn(0x7f6cc80c3c30 legacy=0x7f6cc8158ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:23.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.628+0000 7f6cc77fe640 1 -- 172.21.15.116:0/2206976994 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cc806ebd0 con 0x7f6cc806fb30 2024-04-25T00:15:23.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.628+0000 7f6cc77fe640 1 --2- 172.21.15.116:0/2206976994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cc806fb30 0x7f6cc806c220 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f6cc0009d90 tx=0x7f6cc001c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3514c5f7794f99ca server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:23.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.628+0000 7f6cc5ffb640 1 -- 172.21.15.116:0/2206976994 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc00b2070 con 0x7f6cc806fb30 2024-04-25T00:15:23.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.628+0000 7f6cc5ffb640 1 -- 172.21.15.116:0/2206976994 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cc00a5890 con 0x7f6cc806fb30 2024-04-25T00:15:23.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.628+0000 7f6cc5ffb640 1 -- 172.21.15.116:0/2206976994 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc00a5c60 con 0x7f6cc806fb30 2024-04-25T00:15:23.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.632+0000 7f6ccd64f640 1 -- 172.21.15.116:0/2206976994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cc806fb30 msgr2=0x7f6cc806c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:23.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.632+0000 7f6ccd64f640 1 --2- 172.21.15.116:0/2206976994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cc806fb30 0x7f6cc806c220 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f6cc0009d90 tx=0x7f6cc001c8e0 comp rx=0 tx=0).stop 2024-04-25T00:15:23.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.632+0000 7f6ccd64f640 1 -- 172.21.15.116:0/2206976994 shutdown_connections 2024-04-25T00:15:23.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.632+0000 7f6ccd64f640 1 --2- 172.21.15.116:0/2206976994 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cc806fb30 0x7f6cc806c220 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:23.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.632+0000 7f6ccd64f640 1 --2- 172.21.15.116:0/2206976994 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6cc8152f60 0x7f6cc80c6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:23.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.632+0000 7f6ccd64f640 1 -- 172.21.15.116:0/2206976994 >> 172.21.15.116:0/2206976994 conn(0x7f6cc80c4270 msgr2=0x7f6cc80c4680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:23.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.632+0000 7f6ccd64f640 1 -- 172.21.15.116:0/2206976994 shutdown_connections 2024-04-25T00:15:23.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.632+0000 7f6ccd64f640 1 -- 172.21.15.116:0/2206976994 wait complete. 2024-04-25T00:15:23.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.632+0000 7f6ccd64f640 1 Processor -- start 2024-04-25T00:15:23.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.632+0000 7f6ccd64f640 1 -- start start 2024-04-25T00:15:23.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.632+0000 7f6ccd64f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6cc806fb30 0x7f6cc80a8a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:23.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.632+0000 7f6ccd64f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cc80c3c30 0x7f6cc814f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:23.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.632+0000 7f6ccd64f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6cc8152f60 0x7f6cc814fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:23.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.632+0000 7f6ccd64f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6cc81629f0 con 0x7f6cc80c3c30 2024-04-25T00:15:23.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.632+0000 7f6ccd64f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6cc8074a70 con 0x7f6cc8152f60 2024-04-25T00:15:23.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.632+0000 7f6ccd64f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6cc8162b70 con 0x7f6cc806fb30 2024-04-25T00:15:23.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.632+0000 7f6cc67fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cc80c3c30 0x7f6cc814f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:23.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.632+0000 7f6cc67fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cc80c3c30 0x7f6cc814f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39458/0 (socket says 172.21.15.116:39458) 2024-04-25T00:15:23.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.632+0000 7f6cc67fc640 1 -- 172.21.15.116:0/203504108 learned_addr learned my addr 172.21.15.116:0/203504108 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:15:23.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.632+0000 7f6cc6ffd640 1 --2- 172.21.15.116:0/203504108 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6cc806fb30 0x7f6cc80a8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:23.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.632+0000 7f6cc77fe640 1 --2- 172.21.15.116:0/203504108 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6cc8152f60 0x7f6cc814fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:23.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.632+0000 7f6cc67fc640 1 -- 172.21.15.116:0/203504108 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6cc806fb30 msgr2=0x7f6cc80a8a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:23.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.632+0000 7f6cc67fc640 1 --2- 172.21.15.116:0/203504108 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6cc806fb30 0x7f6cc80a8a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:23.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.632+0000 7f6cc67fc640 1 -- 172.21.15.116:0/203504108 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6cc8152f60 msgr2=0x7f6cc814fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:23.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.632+0000 7f6cc67fc640 1 --2- 172.21.15.116:0/203504108 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6cc8152f60 0x7f6cc814fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:23.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.632+0000 7f6cc67fc640 1 -- 172.21.15.116:0/203504108 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cc815b4b0 con 0x7f6cc80c3c30 2024-04-25T00:15:23.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.632+0000 7f6cc77fe640 1 --2- 172.21.15.116:0/203504108 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6cc8152f60 0x7f6cc814fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:15:23.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.632+0000 7f6cc67fc640 1 --2- 172.21.15.116:0/203504108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cc80c3c30 0x7f6cc814f780 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc00c3c0 tx=0x7f6cbc00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:23.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.632+0000 7f6caf7fe640 1 -- 172.21.15.116:0/203504108 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cbc017070 con 0x7f6cc80c3c30 2024-04-25T00:15:23.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.632+0000 7f6caf7fe640 1 -- 172.21.15.116:0/203504108 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cbc005cf0 con 0x7f6cc80c3c30 2024-04-25T00:15:23.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.632+0000 7f6ccd64f640 1 -- 172.21.15.116:0/203504108 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cc815a240 con 0x7f6cc80c3c30 2024-04-25T00:15:23.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.632+0000 7f6caf7fe640 1 -- 172.21.15.116:0/203504108 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cbc0072f0 con 0x7f6cc80c3c30 2024-04-25T00:15:23.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.632+0000 7f6ccd64f640 1 -- 172.21.15.116:0/203504108 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cc81efcb0 con 0x7f6cc80c3c30 2024-04-25T00:15:23.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.636+0000 7f6ccd64f640 1 -- 172.21.15.116:0/203504108 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cc8158560 con 0x7f6cc80c3c30 2024-04-25T00:15:23.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.636+0000 7f6caf7fe640 1 -- 172.21.15.116:0/203504108 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6cbc01a070 con 0x7f6cc80c3c30 2024-04-25T00:15:23.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.636+0000 7f6caf7fe640 1 --2- 172.21.15.116:0/203504108 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6cb40678c0 0x7f6cb4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:23.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.636+0000 7f6caf7fe640 1 -- 172.21.15.116:0/203504108 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 8300+0+0 (secure 0 0 0) 0x7f6cbc085a20 con 0x7f6cc80c3c30 2024-04-25T00:15:23.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.636+0000 7f6cc6ffd640 1 --2- 172.21.15.116:0/203504108 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6cb40678c0 0x7f6cb4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:23.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.636+0000 7f6cc6ffd640 1 --2- 172.21.15.116:0/203504108 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6cb40678c0 0x7f6cb4069d80 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f6cb800b9d0 tx=0x7f6cb8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:23.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.640+0000 7f6caf7fe640 1 -- 172.21.15.116:0/203504108 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f6cc8158560 con 0x7f6cc80c3c30 2024-04-25T00:15:23.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.871+0000 7f6ccd64f640 1 -- 172.21.15.116:0/203504108 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6cc8068a60 con 0x7f6cc80c3c30 2024-04-25T00:15:23.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.871+0000 7f6caf7fe640 1 -- 172.21.15.116:0/203504108 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) ==== 95+0+6509 (secure 0 0 0) 0x7f6cbc055b80 con 0x7f6cc80c3c30 2024-04-25T00:15:23.872 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:15:23.872 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":107,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6810,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/2340581953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2340581953},{"type":"v1","addr":"172.21.15.116:6833","nonce":2340581953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106},{"gid":6828,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/2012470875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2012470875},{"type":"v1","addr":"172.21.15.116:6835","nonce":2012470875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":107}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:15:01.877414+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5273":{"gid":5273,"name":"d","rank":0,"incarnation":102,"state":"up:active","state_seq":49,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5273,"qdb_cluster":[5273]},"id":1},{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:14:58.851744+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5791},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5791":{"gid":5791,"name":"b","rank":0,"incarnation":98,"state":"up:active","state_seq":23,"addr":"172.21.15.174:6837/3180024481","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3180024481},{"type":"v1","addr":"172.21.15.174:6837","nonce":3180024481}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5791,"qdb_cluster":[5791]},"id":2}]} 2024-04-25T00:15:23.873 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 107 2024-04-25T00:15:23.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.875+0000 7f6ccd64f640 1 -- 172.21.15.116:0/203504108 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6cb40678c0 msgr2=0x7f6cb4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:23.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.875+0000 7f6ccd64f640 1 --2- 172.21.15.116:0/203504108 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6cb40678c0 0x7f6cb4069d80 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f6cb800b9d0 tx=0x7f6cb8008040 comp rx=0 tx=0).stop 2024-04-25T00:15:23.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.875+0000 7f6ccd64f640 1 -- 172.21.15.116:0/203504108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cc80c3c30 msgr2=0x7f6cc814f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:23.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.875+0000 7f6ccd64f640 1 --2- 172.21.15.116:0/203504108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cc80c3c30 0x7f6cc814f780 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc00c3c0 tx=0x7f6cbc00c880 comp rx=0 tx=0).stop 2024-04-25T00:15:23.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.875+0000 7f6ccd64f640 1 -- 172.21.15.116:0/203504108 shutdown_connections 2024-04-25T00:15:23.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.875+0000 7f6ccd64f640 1 --2- 172.21.15.116:0/203504108 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6cb40678c0 0x7f6cb4069d80 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:23.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.875+0000 7f6ccd64f640 1 --2- 172.21.15.116:0/203504108 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6cc8152f60 0x7f6cc814fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:23.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.875+0000 7f6ccd64f640 1 --2- 172.21.15.116:0/203504108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6cc80c3c30 0x7f6cc814f780 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:23.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.875+0000 7f6ccd64f640 1 --2- 172.21.15.116:0/203504108 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6cc806fb30 0x7f6cc80a8a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:23.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.875+0000 7f6ccd64f640 1 -- 172.21.15.116:0/203504108 >> 172.21.15.116:0/203504108 conn(0x7f6cc80c4270 msgr2=0x7f6cc80c5370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:23.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.875+0000 7f6ccd64f640 1 -- 172.21.15.116:0/203504108 shutdown_connections 2024-04-25T00:15:23.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:23.875+0000 7f6ccd64f640 1 -- 172.21.15.116:0/203504108 wait complete. 2024-04-25T00:15:23.892 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2024-04-25T00:15:25.894 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:15:26.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.023+0000 7f491d67d640 1 Processor -- start 2024-04-25T00:15:26.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.023+0000 7f491d67d640 1 -- start start 2024-04-25T00:15:26.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.023+0000 7f491d67d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f491814c700 0x7f49180c5090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:26.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.023+0000 7f491d67d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4918155b20 0x7f4918152c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:26.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.023+0000 7f491d67d640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f49180c55d0 0x7f4918153180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:26.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.023+0000 7f491d67d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f49181527b0 con 0x7f4918155b20 2024-04-25T00:15:26.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.023+0000 7f491d67d640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f49180a9e60 con 0x7f49180c55d0 2024-04-25T00:15:26.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.023+0000 7f491d67d640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f49180c3ef0 con 0x7f491814c700 2024-04-25T00:15:26.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.023+0000 7f490ffff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4918155b20 0x7f4918152c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:26.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.023+0000 7f490ffff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4918155b20 0x7f4918152c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33302/0 (socket says 172.21.15.116:33302) 2024-04-25T00:15:26.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.023+0000 7f490ffff640 1 -- 172.21.15.116:0/2405941006 learned_addr learned my addr 172.21.15.116:0/2405941006 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:15:26.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.023+0000 7f4916ffd640 1 --2- 172.21.15.116:0/2405941006 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f491814c700 0x7f49180c5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:26.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.023+0000 7f49177fe640 1 --2- 172.21.15.116:0/2405941006 >> v2:172.21.15.174:3300/0 conn(0x7f49180c55d0 0x7f4918153180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:26.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.023+0000 7f490ffff640 1 -- 172.21.15.116:0/2405941006 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f491814c700 msgr2=0x7f49180c5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:26.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.023+0000 7f490ffff640 1 --2- 172.21.15.116:0/2405941006 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f491814c700 0x7f49180c5090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:26.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.023+0000 7f490ffff640 1 -- 172.21.15.116:0/2405941006 >> v2:172.21.15.174:3300/0 conn(0x7f49180c55d0 msgr2=0x7f4918153180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:26.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.023+0000 7f490ffff640 1 --2- 172.21.15.116:0/2405941006 >> v2:172.21.15.174:3300/0 conn(0x7f49180c55d0 0x7f4918153180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:26.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.023+0000 7f490ffff640 1 -- 172.21.15.116:0/2405941006 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49180c41d0 con 0x7f4918155b20 2024-04-25T00:15:26.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.023+0000 7f490ffff640 1 --2- 172.21.15.116:0/2405941006 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4918155b20 0x7f4918152c40 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f4918073280 tx=0x7f490001c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=298fc5b095b92473 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:26.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.027+0000 7f49167fc640 1 -- 172.21.15.116:0/2405941006 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49000b0070 con 0x7f4918155b20 2024-04-25T00:15:26.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.027+0000 7f49167fc640 1 -- 172.21.15.116:0/2405941006 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49000ab600 con 0x7f4918155b20 2024-04-25T00:15:26.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.027+0000 7f49167fc640 1 -- 172.21.15.116:0/2405941006 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49000ab9d0 con 0x7f4918155b20 2024-04-25T00:15:26.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.027+0000 7f491d67d640 1 -- 172.21.15.116:0/2405941006 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4918155b20 msgr2=0x7f4918152c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:26.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.027+0000 7f491d67d640 1 --2- 172.21.15.116:0/2405941006 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4918155b20 0x7f4918152c40 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f4918073280 tx=0x7f490001c400 comp rx=0 tx=0).stop 2024-04-25T00:15:26.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.027+0000 7f491d67d640 1 -- 172.21.15.116:0/2405941006 shutdown_connections 2024-04-25T00:15:26.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.027+0000 7f491d67d640 1 --2- 172.21.15.116:0/2405941006 >> v2:172.21.15.174:3300/0 conn(0x7f49180c55d0 0x7f4918153180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:26.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.027+0000 7f491d67d640 1 --2- 172.21.15.116:0/2405941006 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4918155b20 0x7f4918152c40 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:26.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.027+0000 7f491d67d640 1 --2- 172.21.15.116:0/2405941006 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f491814c700 0x7f49180c5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:26.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.027+0000 7f491d67d640 1 -- 172.21.15.116:0/2405941006 >> 172.21.15.116:0/2405941006 conn(0x7f491806c720 msgr2=0x7f491806cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:26.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.027+0000 7f491d67d640 1 -- 172.21.15.116:0/2405941006 shutdown_connections 2024-04-25T00:15:26.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.027+0000 7f491d67d640 1 -- 172.21.15.116:0/2405941006 wait complete. 2024-04-25T00:15:26.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.027+0000 7f491d67d640 1 Processor -- start 2024-04-25T00:15:26.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.027+0000 7f491d67d640 1 -- start start 2024-04-25T00:15:26.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.027+0000 7f491d67d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49180c55d0 0x7f49181ec060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:26.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.027+0000 7f491d67d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f491814c700 0x7f49181547b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:26.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.027+0000 7f491d67d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4918155b20 0x7f4918146c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:26.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.027+0000 7f491d67d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4918072990 con 0x7f49180c55d0 2024-04-25T00:15:26.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.027+0000 7f491d67d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4918072810 con 0x7f4918155b20 2024-04-25T00:15:26.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.027+0000 7f491d67d640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f4918072b10 con 0x7f491814c700 2024-04-25T00:15:26.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.027+0000 7f4916ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49180c55d0 0x7f49181ec060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:26.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.027+0000 7f4916ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49180c55d0 0x7f49181ec060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33304/0 (socket says 172.21.15.116:33304) 2024-04-25T00:15:26.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.027+0000 7f4916ffd640 1 -- 172.21.15.116:0/3193824522 learned_addr learned my addr 172.21.15.116:0/3193824522 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:15:26.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.027+0000 7f490ffff640 1 --2- 172.21.15.116:0/3193824522 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f491814c700 0x7f49181547b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:26.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.027+0000 7f49177fe640 1 --2- 172.21.15.116:0/3193824522 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4918155b20 0x7f4918146c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:26.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.031+0000 7f4916ffd640 1 -- 172.21.15.116:0/3193824522 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f491814c700 msgr2=0x7f49181547b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:26.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.031+0000 7f4916ffd640 1 --2- 172.21.15.116:0/3193824522 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f491814c700 0x7f49181547b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:26.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.031+0000 7f4916ffd640 1 -- 172.21.15.116:0/3193824522 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4918155b20 msgr2=0x7f4918146c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:26.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.031+0000 7f4916ffd640 1 --2- 172.21.15.116:0/3193824522 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4918155b20 0x7f4918146c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:26.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.031+0000 7f4916ffd640 1 -- 172.21.15.116:0/3193824522 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f491814ce40 con 0x7f49180c55d0 2024-04-25T00:15:26.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.031+0000 7f4916ffd640 1 --2- 172.21.15.116:0/3193824522 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49180c55d0 0x7f49181ec060 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f490800d620 tx=0x7f490800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:26.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.031+0000 7f490f7fe640 1 -- 172.21.15.116:0/3193824522 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4908011920 con 0x7f49180c55d0 2024-04-25T00:15:26.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.031+0000 7f490f7fe640 1 -- 172.21.15.116:0/3193824522 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4908011ae0 con 0x7f49180c55d0 2024-04-25T00:15:26.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.031+0000 7f490f7fe640 1 -- 172.21.15.116:0/3193824522 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4908010440 con 0x7f49180c55d0 2024-04-25T00:15:26.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.031+0000 7f491d67d640 1 -- 172.21.15.116:0/3193824522 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49180ac8f0 con 0x7f49180c55d0 2024-04-25T00:15:26.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.031+0000 7f491d67d640 1 -- 172.21.15.116:0/3193824522 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49180aceb0 con 0x7f49180c55d0 2024-04-25T00:15:26.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.031+0000 7f491d67d640 1 -- 172.21.15.116:0/3193824522 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4918068a60 con 0x7f49180c55d0 2024-04-25T00:15:26.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.031+0000 7f490f7fe640 1 -- 172.21.15.116:0/3193824522 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4908019020 con 0x7f49180c55d0 2024-04-25T00:15:26.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.031+0000 7f490f7fe640 1 --2- 172.21.15.116:0/3193824522 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f48f80678c0 0x7f48f8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:26.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.031+0000 7f490f7fe640 1 -- 172.21.15.116:0/3193824522 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 8386+0+0 (secure 0 0 0) 0x7f49080885e0 con 0x7f49180c55d0 2024-04-25T00:15:26.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.031+0000 7f490ffff640 1 --2- 172.21.15.116:0/3193824522 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f48f80678c0 0x7f48f8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:26.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.031+0000 7f490ffff640 1 --2- 172.21.15.116:0/3193824522 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f48f80678c0 0x7f48f8069d80 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f49000ad000 tx=0x7f4900008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:26.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.039+0000 7f490f7fe640 1 -- 172.21.15.116:0/3193824522 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f49080586e0 con 0x7f49180c55d0 2024-04-25T00:15:26.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.295+0000 7f491d67d640 1 -- 172.21.15.116:0/3193824522 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f49180ad630 con 0x7f49180c55d0 2024-04-25T00:15:26.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.295+0000 7f490f7fe640 1 -- 172.21.15.116:0/3193824522 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) ==== 95+0+5673 (secure 0 0 0) 0x7f4908057fe0 con 0x7f49180c55d0 2024-04-25T00:15:26.297 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:15:26.297 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6810,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/2340581953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2340581953},{"type":"v1","addr":"172.21.15.116:6833","nonce":2340581953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:15:01.877414+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5273":{"gid":5273,"name":"d","rank":0,"incarnation":102,"state":"up:active","state_seq":49,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5273,"qdb_cluster":[5273]},"id":1},{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:15:25.568468+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6828":{"gid":6828,"name":"a","rank":0,"incarnation":108,"state":"up:replay","state_seq":1,"addr":"172.21.15.116:6835/2012470875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2012470875},{"type":"v1","addr":"172.21.15.116:6835","nonce":2012470875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:15:26.298 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 108 2024-04-25T00:15:26.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.299+0000 7f491d67d640 1 -- 172.21.15.116:0/3193824522 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f48f80678c0 msgr2=0x7f48f8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:26.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.299+0000 7f491d67d640 1 --2- 172.21.15.116:0/3193824522 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f48f80678c0 0x7f48f8069d80 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f49000ad000 tx=0x7f4900008040 comp rx=0 tx=0).stop 2024-04-25T00:15:26.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.299+0000 7f491d67d640 1 -- 172.21.15.116:0/3193824522 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49180c55d0 msgr2=0x7f49181ec060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:26.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.299+0000 7f491d67d640 1 --2- 172.21.15.116:0/3193824522 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49180c55d0 0x7f49181ec060 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f490800d620 tx=0x7f490800dae0 comp rx=0 tx=0).stop 2024-04-25T00:15:26.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.299+0000 7f491d67d640 1 -- 172.21.15.116:0/3193824522 shutdown_connections 2024-04-25T00:15:26.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.299+0000 7f491d67d640 1 --2- 172.21.15.116:0/3193824522 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f48f80678c0 0x7f48f8069d80 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:26.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.299+0000 7f491d67d640 1 --2- 172.21.15.116:0/3193824522 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4918155b20 0x7f4918146c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:26.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.299+0000 7f491d67d640 1 --2- 172.21.15.116:0/3193824522 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f491814c700 0x7f49181547b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:26.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.299+0000 7f491d67d640 1 --2- 172.21.15.116:0/3193824522 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49180c55d0 0x7f49181ec060 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:26.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.299+0000 7f491d67d640 1 -- 172.21.15.116:0/3193824522 >> 172.21.15.116:0/3193824522 conn(0x7f491806c720 msgr2=0x7f4918152c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:26.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.299+0000 7f491d67d640 1 -- 172.21.15.116:0/3193824522 shutdown_connections 2024-04-25T00:15:26.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.299+0000 7f491d67d640 1 -- 172.21.15.116:0/3193824522 wait complete. 2024-04-25T00:15:26.317 INFO:tasks.mds_thrash.fs.[b]:mds.b down, removed from mdsmap 2024-04-25T00:15:26.317 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-04-25T00:15:26.317 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:15:26.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.415+0000 7f0d3fe99640 1 Processor -- start 2024-04-25T00:15:26.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.419+0000 7f0d3fe99640 1 -- start start 2024-04-25T00:15:26.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.419+0000 7f0d3fe99640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0d38072d70 0x7f0d38154080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:26.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.419+0000 7f0d3fe99640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d3814fe70 con 0x7f0d3814fa90 2024-04-25T00:15:26.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.419+0000 7f0d3fe99640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d380c4770 con 0x7f0d38155a70 2024-04-25T00:15:26.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.419+0000 7f0d3fe99640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f0d380c3570 con 0x7f0d38072d70 2024-04-25T00:15:26.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.419+0000 7f0d3dc35640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f0d3814fa90 0x7f0d38153970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:51280/0 (socket says 172.21.15.116:51280) 2024-04-25T00:15:26.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.419+0000 7f0d3dc35640 1 -- 172.21.15.116:0/1057283143 learned_addr learned my addr 172.21.15.116:0/1057283143 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:15:26.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.419+0000 7f0d3d434640 1 --2- 172.21.15.116:0/1057283143 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0d38072d70 0x7f0d38154080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:26.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.419+0000 7f0d3cc33640 1 -- 172.21.15.116:0/1057283143 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 57858015 0 0) 0x7f0d3814fe70 con 0x7f0d3814fa90 2024-04-25T00:15:26.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.419+0000 7f0d3cc33640 1 -- 172.21.15.116:0/1057283143 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d24003530 con 0x7f0d3814fa90 2024-04-25T00:15:26.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.419+0000 7f0d3cc33640 1 -- 172.21.15.116:0/1057283143 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 561479590 0 0) 0x7f0d24003530 con 0x7f0d3814fa90 2024-04-25T00:15:26.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.419+0000 7f0d3cc33640 1 -- 172.21.15.116:0/1057283143 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0d38072d70 msgr2=0x7f0d38154080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:26.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.419+0000 7f0d3cc33640 1 --2- 172.21.15.116:0/1057283143 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0d38072d70 0x7f0d38154080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:26.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.419+0000 7f0d3cc33640 1 -- 172.21.15.116:0/1057283143 >> v1:172.21.15.174:6789/0 conn(0x7f0d38155a70 legacy=0x7f0d38155ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:26.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.419+0000 7f0d3cc33640 1 -- 172.21.15.116:0/1057283143 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d380c3850 con 0x7f0d3814fa90 2024-04-25T00:15:26.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.419+0000 7f0d3cc33640 1 -- 172.21.15.116:0/1057283143 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f0d34001680 con 0x7f0d3814fa90 2024-04-25T00:15:26.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.419+0000 7f0d3cc33640 1 -- 172.21.15.116:0/1057283143 >> v1:172.21.15.116:6789/0 conn(0x7f0d3814fa90 legacy=0x7f0d38153970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:26.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.419+0000 7f0d3cc33640 1 --2- 172.21.15.116:0/1057283143 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0d240039f0 0x7f0d24003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:26.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.419+0000 7f0d3cc33640 1 --2- 172.21.15.116:0/1057283143 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0d38072d70 0x7f0d24004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:26.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.419+0000 7f0d3cc33640 1 --2- 172.21.15.116:0/1057283143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d24004d50 0x7f0d240051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:26.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.419+0000 7f0d3cc33640 1 -- 172.21.15.116:0/1057283143 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0d380c3570 con 0x7f0d24004d50 2024-04-25T00:15:26.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.419+0000 7f0d3cc33640 1 -- 172.21.15.116:0/1057283143 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0d2400bb30 con 0x7f0d240039f0 2024-04-25T00:15:26.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.419+0000 7f0d3cc33640 1 -- 172.21.15.116:0/1057283143 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f0d2400df40 con 0x7f0d38072d70 2024-04-25T00:15:26.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.419+0000 7f0d3e436640 1 --2- 172.21.15.116:0/1057283143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d24004d50 0x7f0d240051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:26.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.419+0000 7f0d3dc35640 1 --2- 172.21.15.116:0/1057283143 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0d240039f0 0x7f0d24003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:26.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.419+0000 7f0d3d434640 1 --2- 172.21.15.116:0/1057283143 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0d38072d70 0x7f0d24004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:26.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.419+0000 7f0d3e436640 1 -- 172.21.15.116:0/1057283143 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0d38072d70 msgr2=0x7f0d24004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:26.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.419+0000 7f0d3e436640 1 --2- 172.21.15.116:0/1057283143 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0d38072d70 0x7f0d24004810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:26.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.419+0000 7f0d3e436640 1 -- 172.21.15.116:0/1057283143 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0d240039f0 msgr2=0x7f0d24003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:26.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.419+0000 7f0d3e436640 1 --2- 172.21.15.116:0/1057283143 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0d240039f0 0x7f0d24003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:26.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.419+0000 7f0d3e436640 1 -- 172.21.15.116:0/1057283143 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d24003530 con 0x7f0d24004d50 2024-04-25T00:15:26.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.419+0000 7f0d3dc35640 1 --2- 172.21.15.116:0/1057283143 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0d240039f0 0x7f0d24003e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:15:26.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.423+0000 7f0d3e436640 1 --2- 172.21.15.116:0/1057283143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d24004d50 0x7f0d240051d0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f0d30001e70 tx=0x7f0d3001c780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8062a61997c57c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:26.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.423+0000 7f0d3cc33640 1 -- 172.21.15.116:0/1057283143 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d30001a90 con 0x7f0d24004d50 2024-04-25T00:15:26.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.423+0000 7f0d3cc33640 1 -- 172.21.15.116:0/1057283143 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d30003b80 con 0x7f0d24004d50 2024-04-25T00:15:26.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.423+0000 7f0d3cc33640 1 -- 172.21.15.116:0/1057283143 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d30005080 con 0x7f0d24004d50 2024-04-25T00:15:26.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.423+0000 7f0d3fe99640 1 -- 172.21.15.116:0/1057283143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d24004d50 msgr2=0x7f0d240051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:26.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.423+0000 7f0d3fe99640 1 --2- 172.21.15.116:0/1057283143 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d24004d50 0x7f0d240051d0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f0d30001e70 tx=0x7f0d3001c780 comp rx=0 tx=0).stop 2024-04-25T00:15:26.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.423+0000 7f0d3e436640 1 -- 172.21.15.116:0/1057283143 reap_dead start 2024-04-25T00:15:26.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.423+0000 7f0d3fe99640 1 -- 172.21.15.116:0/1057283143 shutdown_connections 2024-04-25T00:15:26.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.423+0000 7f0d3fe99640 1 -- 172.21.15.116:0/1057283143 >> 172.21.15.116:0/1057283143 conn(0x7f0d380c4aa0 msgr2=0x7f0d380c4eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:26.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.423+0000 7f0d3fe99640 1 -- 172.21.15.116:0/1057283143 shutdown_connections 2024-04-25T00:15:26.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.423+0000 7f0d3fe99640 1 -- 172.21.15.116:0/1057283143 wait complete. 2024-04-25T00:15:26.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.423+0000 7f0d3fe99640 1 Processor -- start 2024-04-25T00:15:26.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.423+0000 7f0d3fe99640 1 -- start start 2024-04-25T00:15:26.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.423+0000 7f0d3fe99640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0d24004d50 0x7f0d380c1630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:26.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.423+0000 7f0d3fe99640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0d381499f0 0x7f0d381458b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:26.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.423+0000 7f0d3fe99640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d381500f0 0x7f0d38145df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:26.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.423+0000 7f0d3fe99640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0d381565d0 con 0x7f0d381500f0 2024-04-25T00:15:26.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.423+0000 7f0d3fe99640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0d380c3e10 con 0x7f0d381499f0 2024-04-25T00:15:26.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.423+0000 7f0d3fe99640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f0d38156750 con 0x7f0d24004d50 2024-04-25T00:15:26.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.427+0000 7f0d3e436640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d381500f0 0x7f0d38145df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:26.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.427+0000 7f0d3e436640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d381500f0 0x7f0d38145df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33322/0 (socket says 172.21.15.116:33322) 2024-04-25T00:15:26.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.427+0000 7f0d3e436640 1 -- 172.21.15.116:0/3020826051 learned_addr learned my addr 172.21.15.116:0/3020826051 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:15:26.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.427+0000 7f0d3dc35640 1 --2- 172.21.15.116:0/3020826051 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0d24004d50 0x7f0d380c1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:26.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.427+0000 7f0d3d434640 1 --2- 172.21.15.116:0/3020826051 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0d381499f0 0x7f0d381458b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:26.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.427+0000 7f0d3e436640 1 -- 172.21.15.116:0/3020826051 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0d24004d50 msgr2=0x7f0d380c1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:26.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.427+0000 7f0d3e436640 1 --2- 172.21.15.116:0/3020826051 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0d24004d50 0x7f0d380c1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:26.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.427+0000 7f0d3e436640 1 -- 172.21.15.116:0/3020826051 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0d381499f0 msgr2=0x7f0d381458b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:26.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.427+0000 7f0d3e436640 1 --2- 172.21.15.116:0/3020826051 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0d381499f0 0x7f0d381458b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:26.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.427+0000 7f0d3e436640 1 -- 172.21.15.116:0/3020826051 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d380c1f80 con 0x7f0d381500f0 2024-04-25T00:15:26.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.427+0000 7f0d3d434640 1 --2- 172.21.15.116:0/3020826051 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0d381499f0 0x7f0d381458b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:15:26.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.427+0000 7f0d3dc35640 1 --2- 172.21.15.116:0/3020826051 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0d24004d50 0x7f0d380c1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:15:26.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.427+0000 7f0d3e436640 1 --2- 172.21.15.116:0/3020826051 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d381500f0 0x7f0d38145df0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f0d3001cc60 tx=0x7f0d24005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:26.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.427+0000 7f0d2e7fc640 1 -- 172.21.15.116:0/3020826051 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d30007dc0 con 0x7f0d381500f0 2024-04-25T00:15:26.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.427+0000 7f0d2e7fc640 1 -- 172.21.15.116:0/3020826051 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d300bc070 con 0x7f0d381500f0 2024-04-25T00:15:26.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.427+0000 7f0d3fe99640 1 -- 172.21.15.116:0/3020826051 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d381ebc80 con 0x7f0d381500f0 2024-04-25T00:15:26.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.427+0000 7f0d2e7fc640 1 -- 172.21.15.116:0/3020826051 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d30005460 con 0x7f0d381500f0 2024-04-25T00:15:26.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.427+0000 7f0d3fe99640 1 -- 172.21.15.116:0/3020826051 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d3814c980 con 0x7f0d381500f0 2024-04-25T00:15:26.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.427+0000 7f0d3fe99640 1 -- 172.21.15.116:0/3020826051 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d3814c770 con 0x7f0d381500f0 2024-04-25T00:15:26.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.427+0000 7f0d2e7fc640 1 -- 172.21.15.116:0/3020826051 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f0d300b1040 con 0x7f0d381500f0 2024-04-25T00:15:26.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.427+0000 7f0d2e7fc640 1 --2- 172.21.15.116:0/3020826051 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0d1c0678c0 0x7f0d1c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:26.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.427+0000 7f0d2e7fc640 1 -- 172.21.15.116:0/3020826051 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 8386+0+0 (secure 0 0 0) 0x7f0d30120f70 con 0x7f0d381500f0 2024-04-25T00:15:26.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.427+0000 7f0d3dc35640 1 --2- 172.21.15.116:0/3020826051 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0d1c0678c0 0x7f0d1c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:26.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.427+0000 7f0d3dc35640 1 --2- 172.21.15.116:0/3020826051 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0d1c0678c0 0x7f0d1c069d80 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f0d34000c10 tx=0x7f0d34009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:26.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.435+0000 7f0d2e7fc640 1 -- 172.21.15.116:0/3020826051 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f0d300ad020 con 0x7f0d381500f0 2024-04-25T00:15:26.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.695+0000 7f0d3fe99640 1 -- 172.21.15.116:0/3020826051 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0d38068a60 con 0x7f0d381500f0 2024-04-25T00:15:26.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.695+0000 7f0d2e7fc640 1 -- 172.21.15.116:0/3020826051 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) ==== 95+0+5673 (secure 0 0 0) 0x7f0d30016490 con 0x7f0d381500f0 2024-04-25T00:15:26.697 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:15:26.697 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6810,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/2340581953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2340581953},{"type":"v1","addr":"172.21.15.116:6833","nonce":2340581953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:15:01.877414+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5273":{"gid":5273,"name":"d","rank":0,"incarnation":102,"state":"up:active","state_seq":49,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5273,"qdb_cluster":[5273]},"id":1},{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:15:25.568468+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6828":{"gid":6828,"name":"a","rank":0,"incarnation":108,"state":"up:replay","state_seq":1,"addr":"172.21.15.116:6835/2012470875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2012470875},{"type":"v1","addr":"172.21.15.116:6835","nonce":2012470875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:15:26.698 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 108 2024-04-25T00:15:26.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.695+0000 7f0d3fe99640 1 -- 172.21.15.116:0/3020826051 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0d1c0678c0 msgr2=0x7f0d1c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:26.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.695+0000 7f0d3fe99640 1 --2- 172.21.15.116:0/3020826051 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0d1c0678c0 0x7f0d1c069d80 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f0d34000c10 tx=0x7f0d34009040 comp rx=0 tx=0).stop 2024-04-25T00:15:26.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.695+0000 7f0d3fe99640 1 -- 172.21.15.116:0/3020826051 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d381500f0 msgr2=0x7f0d38145df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:26.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.695+0000 7f0d3fe99640 1 --2- 172.21.15.116:0/3020826051 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d381500f0 0x7f0d38145df0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f0d3001cc60 tx=0x7f0d24005840 comp rx=0 tx=0).stop 2024-04-25T00:15:26.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.695+0000 7f0d3fe99640 1 -- 172.21.15.116:0/3020826051 shutdown_connections 2024-04-25T00:15:26.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.695+0000 7f0d3fe99640 1 --2- 172.21.15.116:0/3020826051 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0d1c0678c0 0x7f0d1c069d80 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:26.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.695+0000 7f0d3fe99640 1 --2- 172.21.15.116:0/3020826051 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0d381500f0 0x7f0d38145df0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:26.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.695+0000 7f0d3fe99640 1 --2- 172.21.15.116:0/3020826051 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0d381499f0 0x7f0d381458b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:26.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.695+0000 7f0d3fe99640 1 --2- 172.21.15.116:0/3020826051 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0d24004d50 0x7f0d380c1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:26.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.695+0000 7f0d3fe99640 1 -- 172.21.15.116:0/3020826051 >> 172.21.15.116:0/3020826051 conn(0x7f0d380c4aa0 msgr2=0x7f0d38155ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:26.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.695+0000 7f0d3fe99640 1 -- 172.21.15.116:0/3020826051 shutdown_connections 2024-04-25T00:15:26.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:26.695+0000 7f0d3fe99640 1 -- 172.21.15.116:0/3020826051 wait complete. 2024-04-25T00:15:26.721 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 108, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:04.678737+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 105, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_5273": { "addr": "172.21.15.174:6835/4215220222", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6834", "nonce": 4215220222, "type": "v2" }, { "addr": "172.21.15.174:6835", "nonce": 4215220222, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5273, "incarnation": 102, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 49 } }, "last_failure": 0, "last_failure_osd_epoch": 47, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T00:15:01.877414+0000", "qdb_cluster": [ 5273 ], "qdb_leader": 5273, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5273 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:18.825887+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 108, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6828": { "addr": "172.21.15.116:6835/2012470875", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6834", "nonce": 2012470875, "type": "v2" }, { "addr": "172.21.15.116:6835", "nonce": 2012470875, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6828, "incarnation": 108, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 48, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-04-25T00:15:25.568468+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6828 } } } ], "standbys": [ { "addr": "172.21.15.116:6833/2340581953", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6832", "nonce": 2340581953, "type": "v2" }, { "addr": "172.21.15.116:6833", "nonce": 2340581953, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 106, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6810, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-25T00:15:28.074 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:15:28.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.207+0000 7fba8f720640 1 Processor -- start 2024-04-25T00:15:28.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.207+0000 7fba8f720640 1 -- start start 2024-04-25T00:15:28.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.207+0000 7fba8f720640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fba88152d80 0x7fba88159950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:28.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.207+0000 7fba8f720640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fba881562a0 0x7fba880c5020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:28.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.207+0000 7fba8f720640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba880730b0 con 0x7fba88072080 2024-04-25T00:15:28.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.207+0000 7fba8f720640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fba88072e60 con 0x7fba88152d80 2024-04-25T00:15:28.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.207+0000 7fba8f720640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fba88073390 con 0x7fba881562a0 2024-04-25T00:15:28.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.207+0000 7fba8d4bc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fba88152d80 0x7fba88159950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:28.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.207+0000 7fba8ccbb640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fba88072080 0x7fba880c36b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:51282/0 (socket says 172.21.15.116:51282) 2024-04-25T00:15:28.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.207+0000 7fba8ccbb640 1 -- 172.21.15.116:0/3677028897 learned_addr learned my addr 172.21.15.116:0/3677028897 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:15:28.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.207+0000 7fba8dcbd640 1 --2- 172.21.15.116:0/3677028897 >> v2:172.21.15.174:3300/0 conn(0x7fba881562a0 0x7fba880c5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:28.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.207+0000 7fba8d4bc640 1 -- 172.21.15.116:0/3677028897 >> v2:172.21.15.174:3300/0 conn(0x7fba881562a0 msgr2=0x7fba880c5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:28.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.207+0000 7fba8d4bc640 1 --2- 172.21.15.116:0/3677028897 >> v2:172.21.15.174:3300/0 conn(0x7fba881562a0 0x7fba880c5020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:28.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.207+0000 7fba8d4bc640 1 -- 172.21.15.116:0/3677028897 >> v1:172.21.15.116:6789/0 conn(0x7fba88072080 legacy=0x7fba880c36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:28.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.207+0000 7fba8d4bc640 1 -- 172.21.15.116:0/3677028897 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba88158e70 con 0x7fba88152d80 2024-04-25T00:15:28.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.207+0000 7fba7ffff640 1 -- 172.21.15.116:0/3677028897 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 832829839 0 0) 0x7fba880730b0 con 0x7fba88072080 2024-04-25T00:15:28.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.211+0000 7fba8d4bc640 1 --2- 172.21.15.116:0/3677028897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba88152d80 0x7fba88159950 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fba880c21e0 tx=0x7fba8401c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba385a8baaa29cad server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:28.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.211+0000 7fba7ffff640 1 -- 172.21.15.116:0/3677028897 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba840b1070 con 0x7fba88152d80 2024-04-25T00:15:28.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.211+0000 7fba7ffff640 1 -- 172.21.15.116:0/3677028897 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba840ac620 con 0x7fba88152d80 2024-04-25T00:15:28.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.211+0000 7fba7ffff640 1 -- 172.21.15.116:0/3677028897 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba840ac9f0 con 0x7fba88152d80 2024-04-25T00:15:28.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.211+0000 7fba8f720640 1 -- 172.21.15.116:0/3677028897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba88152d80 msgr2=0x7fba88159950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:28.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.211+0000 7fba8f720640 1 --2- 172.21.15.116:0/3677028897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba88152d80 0x7fba88159950 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fba880c21e0 tx=0x7fba8401c3c0 comp rx=0 tx=0).stop 2024-04-25T00:15:28.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.211+0000 7fba8f720640 1 -- 172.21.15.116:0/3677028897 shutdown_connections 2024-04-25T00:15:28.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.211+0000 7fba8f720640 1 --2- 172.21.15.116:0/3677028897 >> v2:172.21.15.174:3300/0 conn(0x7fba881562a0 0x7fba880c5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:28.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.211+0000 7fba8f720640 1 --2- 172.21.15.116:0/3677028897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba88152d80 0x7fba88159950 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:28.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.211+0000 7fba8f720640 1 -- 172.21.15.116:0/3677028897 >> 172.21.15.116:0/3677028897 conn(0x7fba8806c720 msgr2=0x7fba8806cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:28.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.211+0000 7fba8f720640 1 -- 172.21.15.116:0/3677028897 shutdown_connections 2024-04-25T00:15:28.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.211+0000 7fba8f720640 1 -- 172.21.15.116:0/3677028897 wait complete. 2024-04-25T00:15:28.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.215+0000 7fba8f720640 1 Processor -- start 2024-04-25T00:15:28.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.215+0000 7fba8f720640 1 -- start start 2024-04-25T00:15:28.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.215+0000 7fba8f720640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fba88072080 0x7fba880abed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:28.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.215+0000 7fba8f720640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fba88152d80 0x7fba88142810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:28.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.215+0000 7fba8f720640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba881562a0 0x7fba88142d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:28.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.215+0000 7fba8f720640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fba88153160 con 0x7fba881562a0 2024-04-25T00:15:28.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.215+0000 7fba8f720640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fba881566b0 con 0x7fba88072080 2024-04-25T00:15:28.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.215+0000 7fba8f720640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fba881532e0 con 0x7fba88152d80 2024-04-25T00:15:28.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.215+0000 7fba8dcbd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba881562a0 0x7fba88142d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:28.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.215+0000 7fba8dcbd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba881562a0 0x7fba88142d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33340/0 (socket says 172.21.15.116:33340) 2024-04-25T00:15:28.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.215+0000 7fba8dcbd640 1 -- 172.21.15.116:0/2370935377 learned_addr learned my addr 172.21.15.116:0/2370935377 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:15:28.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.215+0000 7fba8d4bc640 1 --2- 172.21.15.116:0/2370935377 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fba88072080 0x7fba880abed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:28.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.215+0000 7fba8ccbb640 1 --2- 172.21.15.116:0/2370935377 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fba88152d80 0x7fba88142810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:28.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.215+0000 7fba8dcbd640 1 -- 172.21.15.116:0/2370935377 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fba88152d80 msgr2=0x7fba88142810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:28.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.215+0000 7fba8dcbd640 1 --2- 172.21.15.116:0/2370935377 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fba88152d80 0x7fba88142810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:28.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.215+0000 7fba8dcbd640 1 -- 172.21.15.116:0/2370935377 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fba88072080 msgr2=0x7fba880abed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:28.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.215+0000 7fba8dcbd640 1 --2- 172.21.15.116:0/2370935377 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fba88072080 0x7fba880abed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:28.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.215+0000 7fba8dcbd640 1 -- 172.21.15.116:0/2370935377 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba881ee050 con 0x7fba881562a0 2024-04-25T00:15:28.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.215+0000 7fba8d4bc640 1 --2- 172.21.15.116:0/2370935377 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fba88072080 0x7fba880abed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:15:28.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.215+0000 7fba8dcbd640 1 --2- 172.21.15.116:0/2370935377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba881562a0 0x7fba88142d50 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fba80007e60 tx=0x7fba8000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:28.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.215+0000 7fba7dffb640 1 -- 172.21.15.116:0/2370935377 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba80014420 con 0x7fba881562a0 2024-04-25T00:15:28.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.215+0000 7fba7dffb640 1 -- 172.21.15.116:0/2370935377 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba80019070 con 0x7fba881562a0 2024-04-25T00:15:28.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.215+0000 7fba8f720640 1 -- 172.21.15.116:0/2370935377 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba881ee330 con 0x7fba881562a0 2024-04-25T00:15:28.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.215+0000 7fba7dffb640 1 -- 172.21.15.116:0/2370935377 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba80014800 con 0x7fba881562a0 2024-04-25T00:15:28.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.215+0000 7fba8f720640 1 -- 172.21.15.116:0/2370935377 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba881ee8f0 con 0x7fba881562a0 2024-04-25T00:15:28.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.215+0000 7fba8f720640 1 -- 172.21.15.116:0/2370935377 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba88068a60 con 0x7fba881562a0 2024-04-25T00:15:28.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.215+0000 7fba7dffb640 1 -- 172.21.15.116:0/2370935377 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fba800149a0 con 0x7fba881562a0 2024-04-25T00:15:28.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.219+0000 7fba7dffb640 1 --2- 172.21.15.116:0/2370935377 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fba6c0678f0 0x7fba6c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:28.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.219+0000 7fba7dffb640 1 -- 172.21.15.116:0/2370935377 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 8386+0+0 (secure 0 0 0) 0x7fba80086f30 con 0x7fba881562a0 2024-04-25T00:15:28.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.219+0000 7fba8d4bc640 1 --2- 172.21.15.116:0/2370935377 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fba6c0678f0 0x7fba6c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:28.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.219+0000 7fba8d4bc640 1 --2- 172.21.15.116:0/2370935377 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fba6c0678f0 0x7fba6c069db0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fba840ae000 tx=0x7fba840a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:28.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.223+0000 7fba7dffb640 1 -- 172.21.15.116:0/2370935377 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fba80056fd0 con 0x7fba881562a0 2024-04-25T00:15:28.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.475+0000 7fba8f720640 1 -- 172.21.15.116:0/2370935377 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fba8815fd30 con 0x7fba881562a0 2024-04-25T00:15:28.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.475+0000 7fba7dffb640 1 -- 172.21.15.116:0/2370935377 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) ==== 95+0+5673 (secure 0 0 0) 0x7fba800568d0 con 0x7fba881562a0 2024-04-25T00:15:28.478 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:15:28.478 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":110,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6810,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/2340581953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2340581953},{"type":"v1","addr":"172.21.15.116:6833","nonce":2340581953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:15:01.877414+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5273":{"gid":5273,"name":"d","rank":0,"incarnation":102,"state":"up:active","state_seq":49,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5273,"qdb_cluster":[5273]},"id":1},{"mdsmap":{"epoch":110,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:15:26.730303+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6828":{"gid":6828,"name":"a","rank":0,"incarnation":108,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.116:6835/2012470875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2012470875},{"type":"v1","addr":"172.21.15.116:6835","nonce":2012470875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:15:28.478 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 110 2024-04-25T00:15:28.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.479+0000 7fba8f720640 1 -- 172.21.15.116:0/2370935377 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fba6c0678f0 msgr2=0x7fba6c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:28.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.479+0000 7fba8f720640 1 --2- 172.21.15.116:0/2370935377 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fba6c0678f0 0x7fba6c069db0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fba840ae000 tx=0x7fba840a7000 comp rx=0 tx=0).stop 2024-04-25T00:15:28.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.479+0000 7fba8f720640 1 -- 172.21.15.116:0/2370935377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba881562a0 msgr2=0x7fba88142d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:28.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.479+0000 7fba8f720640 1 --2- 172.21.15.116:0/2370935377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba881562a0 0x7fba88142d50 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fba80007e60 tx=0x7fba8000e4b0 comp rx=0 tx=0).stop 2024-04-25T00:15:28.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.479+0000 7fba8f720640 1 -- 172.21.15.116:0/2370935377 shutdown_connections 2024-04-25T00:15:28.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.479+0000 7fba8f720640 1 --2- 172.21.15.116:0/2370935377 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fba6c0678f0 0x7fba6c069db0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:28.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.479+0000 7fba8f720640 1 --2- 172.21.15.116:0/2370935377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fba881562a0 0x7fba88142d50 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:28.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.479+0000 7fba8f720640 1 --2- 172.21.15.116:0/2370935377 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fba88152d80 0x7fba88142810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:28.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.479+0000 7fba8f720640 1 --2- 172.21.15.116:0/2370935377 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fba88072080 0x7fba880abed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:28.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.479+0000 7fba8f720640 1 -- 172.21.15.116:0/2370935377 >> 172.21.15.116:0/2370935377 conn(0x7fba8806c720 msgr2=0x7fba880a94a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:28.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.479+0000 7fba8f720640 1 -- 172.21.15.116:0/2370935377 shutdown_connections 2024-04-25T00:15:28.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.479+0000 7fba8f720640 1 -- 172.21.15.116:0/2370935377 wait complete. 2024-04-25T00:15:28.495 INFO:tasks.mds_thrash.fs.[a]:kill mds.d (rank=0) 2024-04-25T00:15:28.495 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-25T00:15:28.495 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:15:28.511 INFO:tasks.ceph.mds.d:Stopped 2024-04-25T00:15:28.511 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:15:28.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.631+0000 7f1305124640 1 Processor -- start 2024-04-25T00:15:28.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.631+0000 7f1305124640 1 -- start start 2024-04-25T00:15:28.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.631+0000 7f1305124640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1300152d80 0x7f130006bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:28.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.631+0000 7f1305124640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f130006d5f0 0x7f1300156c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:28.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.631+0000 7f1305124640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13000c9150 con 0x7f130006c290 2024-04-25T00:15:28.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.631+0000 7f1305124640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f13000c9d60 con 0x7f1300152d80 2024-04-25T00:15:28.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.631+0000 7f1305124640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f13000c9430 con 0x7f130006d5f0 2024-04-25T00:15:28.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.631+0000 7f12fed9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1300152d80 0x7f130006bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:28.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.631+0000 7f12fed9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1300152d80 0x7f130006bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33348/0 (socket says 172.21.15.116:33348) 2024-04-25T00:15:28.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.631+0000 7f12fed9d640 1 -- 172.21.15.116:0/1273892410 learned_addr learned my addr 172.21.15.116:0/1273892410 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:15:28.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.631+0000 7f12fed9d640 1 -- 172.21.15.116:0/1273892410 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f130006d5f0 msgr2=0x7f1300156c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:28.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.631+0000 7f12fed9d640 1 --2- 172.21.15.116:0/1273892410 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f130006d5f0 0x7f1300156c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:28.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.631+0000 7f12fed9d640 1 -- 172.21.15.116:0/1273892410 >> v1:172.21.15.174:6789/0 conn(0x7f130006c290 legacy=0x7f13001571c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:28.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.631+0000 7f12fed9d640 1 -- 172.21.15.116:0/1273892410 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13000c9710 con 0x7f1300152d80 2024-04-25T00:15:28.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.635+0000 7f12fed9d640 1 --2- 172.21.15.116:0/1273892410 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1300152d80 0x7f130006bd50 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f12f0009870 tx=0x7f12f001c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a2c486539db4787 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:28.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.635+0000 7f12fdd9b640 1 -- 172.21.15.116:0/1273892410 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12f00ab640 con 0x7f1300152d80 2024-04-25T00:15:28.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.635+0000 7f12fdd9b640 1 -- 172.21.15.116:0/1273892410 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12f00b0070 con 0x7f1300152d80 2024-04-25T00:15:28.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.635+0000 7f12fdd9b640 1 -- 172.21.15.116:0/1273892410 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12f00aba00 con 0x7f1300152d80 2024-04-25T00:15:28.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.635+0000 7f1305124640 1 -- 172.21.15.116:0/1273892410 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1300152d80 msgr2=0x7f130006bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:28.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.635+0000 7f1305124640 1 --2- 172.21.15.116:0/1273892410 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1300152d80 0x7f130006bd50 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f12f0009870 tx=0x7f12f001c460 comp rx=0 tx=0).stop 2024-04-25T00:15:28.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.635+0000 7f1305124640 1 -- 172.21.15.116:0/1273892410 shutdown_connections 2024-04-25T00:15:28.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.635+0000 7f1305124640 1 --2- 172.21.15.116:0/1273892410 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f130006d5f0 0x7f1300156c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:28.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.635+0000 7f1305124640 1 --2- 172.21.15.116:0/1273892410 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1300152d80 0x7f130006bd50 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:28.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.635+0000 7f1305124640 1 -- 172.21.15.116:0/1273892410 >> 172.21.15.116:0/1273892410 conn(0x7f13000acdd0 msgr2=0x7f13000ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:28.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.635+0000 7f1305124640 1 -- 172.21.15.116:0/1273892410 shutdown_connections 2024-04-25T00:15:28.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.635+0000 7f1305124640 1 -- 172.21.15.116:0/1273892410 wait complete. 2024-04-25T00:15:28.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.635+0000 7f1305124640 1 Processor -- start 2024-04-25T00:15:28.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.635+0000 7f1305124640 1 -- start start 2024-04-25T00:15:28.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.635+0000 7f1305124640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f130006c290 0x7f13000c7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:28.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.635+0000 7f1305124640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f130006d5f0 0x7f13000c39a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:28.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.635+0000 7f1305124640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1300152d80 0x7f13000c3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:28.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.635+0000 7f1305124640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f13000caf00 con 0x7f1300152d80 2024-04-25T00:15:28.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.635+0000 7f1305124640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f13000cad80 con 0x7f130006c290 2024-04-25T00:15:28.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.635+0000 7f1305124640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f13001628d0 con 0x7f130006d5f0 2024-04-25T00:15:28.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.635+0000 7f12ff59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1300152d80 0x7f13000c3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:28.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.635+0000 7f12ff59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1300152d80 0x7f13000c3ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33362/0 (socket says 172.21.15.116:33362) 2024-04-25T00:15:28.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.635+0000 7f12ff59e640 1 -- 172.21.15.116:0/690164432 learned_addr learned my addr 172.21.15.116:0/690164432 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:15:28.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.635+0000 7f12fed9d640 1 --2- 172.21.15.116:0/690164432 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f130006c290 0x7f13000c7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:28.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.635+0000 7f12ff59e640 1 -- 172.21.15.116:0/690164432 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f130006d5f0 msgr2=0x7f13000c39a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:28.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.635+0000 7f12fe59c640 1 --2- 172.21.15.116:0/690164432 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f130006d5f0 0x7f13000c39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:28.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.635+0000 7f12ff59e640 1 --2- 172.21.15.116:0/690164432 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f130006d5f0 0x7f13000c39a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:28.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.635+0000 7f12ff59e640 1 -- 172.21.15.116:0/690164432 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f130006c290 msgr2=0x7f13000c7040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:28.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.635+0000 7f12ff59e640 1 --2- 172.21.15.116:0/690164432 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f130006c290 0x7f13000c7040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:28.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.635+0000 7f12ff59e640 1 -- 172.21.15.116:0/690164432 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13001f2b60 con 0x7f1300152d80 2024-04-25T00:15:28.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.635+0000 7f12ff59e640 1 --2- 172.21.15.116:0/690164432 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1300152d80 0x7f13000c3ee0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f12f800b9a0 tx=0x7f12f800be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:28.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.635+0000 7f12e77fe640 1 -- 172.21.15.116:0/690164432 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12f8019070 con 0x7f1300152d80 2024-04-25T00:15:28.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.635+0000 7f12e77fe640 1 -- 172.21.15.116:0/690164432 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12f800cba0 con 0x7f1300152d80 2024-04-25T00:15:28.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.635+0000 7f1305124640 1 -- 172.21.15.116:0/690164432 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13001f2e00 con 0x7f1300152d80 2024-04-25T00:15:28.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.635+0000 7f12e77fe640 1 -- 172.21.15.116:0/690164432 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12f8014440 con 0x7f1300152d80 2024-04-25T00:15:28.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.639+0000 7f1305124640 1 -- 172.21.15.116:0/690164432 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13000c28e0 con 0x7f1300152d80 2024-04-25T00:15:28.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.639+0000 7f1305124640 1 -- 172.21.15.116:0/690164432 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13000c34d0 con 0x7f1300152d80 2024-04-25T00:15:28.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.639+0000 7f12e77fe640 1 -- 172.21.15.116:0/690164432 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f12f801c070 con 0x7f1300152d80 2024-04-25T00:15:28.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.639+0000 7f12e77fe640 1 --2- 172.21.15.116:0/690164432 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f12ec0678c0 0x7f12ec069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:28.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.639+0000 7f12e77fe640 1 -- 172.21.15.116:0/690164432 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 8386+0+0 (secure 0 0 0) 0x7f12f80862b0 con 0x7f1300152d80 2024-04-25T00:15:28.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.639+0000 7f12fed9d640 1 --2- 172.21.15.116:0/690164432 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f12ec0678c0 0x7f12ec069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:28.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.639+0000 7f12fed9d640 1 --2- 172.21.15.116:0/690164432 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f12ec0678c0 0x7f12ec069d80 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f12f00af000 tx=0x7f12f00a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:28.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.643+0000 7f12e77fe640 1 -- 172.21.15.116:0/690164432 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f12f80563b0 con 0x7f1300152d80 2024-04-25T00:15:28.722 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:15:28.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.851+0000 7ffa8df07640 1 Processor -- start 2024-04-25T00:15:28.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.851+0000 7ffa8df07640 1 -- start start 2024-04-25T00:15:28.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.851+0000 7ffa8df07640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ffa88152d60 0x7ffa880ab250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:28.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.851+0000 7ffa8df07640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ffa88153140 0x7ffa880c3c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:28.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.851+0000 7ffa8df07640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa88156760 con 0x7ffa880ab790 2024-04-25T00:15:28.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.851+0000 7ffa8df07640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7ffa88155be0 con 0x7ffa88153140 2024-04-25T00:15:28.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.851+0000 7ffa8df07640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7ffa88155ec0 con 0x7ffa88152d60 2024-04-25T00:15:28.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.851+0000 7ffa87fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ffa88153140 0x7ffa880c3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:28.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.851+0000 7ffa87fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ffa88153140 0x7ffa880c3c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33376/0 (socket says 172.21.15.116:33376) 2024-04-25T00:15:28.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.851+0000 7ffa87fff640 1 -- 172.21.15.116:0/1917097467 learned_addr learned my addr 172.21.15.116:0/1917097467 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:15:28.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.851+0000 7ffa86ffd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7ffa880ab790 0x7ffa880c8450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:51292/0 (socket says 172.21.15.116:51292) 2024-04-25T00:15:28.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.851+0000 7ffa877fe640 1 --2- 172.21.15.116:0/1917097467 >> v2:172.21.15.174:3300/0 conn(0x7ffa88152d60 0x7ffa880ab250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:28.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.851+0000 7ffa87fff640 1 -- 172.21.15.116:0/1917097467 >> v2:172.21.15.174:3300/0 conn(0x7ffa88152d60 msgr2=0x7ffa880ab250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:28.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.851+0000 7ffa87fff640 1 --2- 172.21.15.116:0/1917097467 >> v2:172.21.15.174:3300/0 conn(0x7ffa88152d60 0x7ffa880ab250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:28.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.851+0000 7ffa87fff640 1 -- 172.21.15.116:0/1917097467 >> v1:172.21.15.116:6789/0 conn(0x7ffa880ab790 legacy=0x7ffa880c8450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:28.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.851+0000 7ffa867fc640 1 -- 172.21.15.116:0/1917097467 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3190413019 0 0) 0x7ffa88156760 con 0x7ffa880ab790 2024-04-25T00:15:28.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.851+0000 7ffa87fff640 1 -- 172.21.15.116:0/1917097467 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa881561a0 con 0x7ffa88153140 2024-04-25T00:15:28.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.855+0000 7ffa87fff640 1 --2- 172.21.15.116:0/1917097467 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa88153140 0x7ffa880c3c90 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7ffa880c35e0 tx=0x7ffa8001c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c916ac123a9e51f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:28.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.855+0000 7ffa867fc640 1 -- 172.21.15.116:0/1917097467 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa800a5840 con 0x7ffa88153140 2024-04-25T00:15:28.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.855+0000 7ffa867fc640 1 -- 172.21.15.116:0/1917097467 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa800a59e0 con 0x7ffa88153140 2024-04-25T00:15:28.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.855+0000 7ffa867fc640 1 -- 172.21.15.116:0/1917097467 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa800a5db0 con 0x7ffa88153140 2024-04-25T00:15:28.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.855+0000 7ffa8df07640 1 -- 172.21.15.116:0/1917097467 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa88153140 msgr2=0x7ffa880c3c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:28.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.855+0000 7ffa8df07640 1 --2- 172.21.15.116:0/1917097467 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa88153140 0x7ffa880c3c90 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7ffa880c35e0 tx=0x7ffa8001c980 comp rx=0 tx=0).stop 2024-04-25T00:15:28.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.855+0000 7ffa8df07640 1 -- 172.21.15.116:0/1917097467 shutdown_connections 2024-04-25T00:15:28.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.855+0000 7ffa8df07640 1 --2- 172.21.15.116:0/1917097467 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa88153140 0x7ffa880c3c90 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:28.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.855+0000 7ffa8df07640 1 --2- 172.21.15.116:0/1917097467 >> v2:172.21.15.174:3300/0 conn(0x7ffa88152d60 0x7ffa880ab250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:28.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.855+0000 7ffa8df07640 1 -- 172.21.15.116:0/1917097467 >> 172.21.15.116:0/1917097467 conn(0x7ffa8806f0c0 msgr2=0x7ffa8806f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:28.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.855+0000 7ffa8df07640 1 -- 172.21.15.116:0/1917097467 shutdown_connections 2024-04-25T00:15:28.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.855+0000 7ffa8df07640 1 -- 172.21.15.116:0/1917097467 wait complete. 2024-04-25T00:15:28.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.855+0000 7ffa8df07640 1 Processor -- start 2024-04-25T00:15:28.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.855+0000 7ffa8df07640 1 -- start start 2024-04-25T00:15:28.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.855+0000 7ffa8df07640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa880ab790 0x7ffa880ad280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:28.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.855+0000 7ffa8df07640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa88152d60 0x7ffa8815f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:28.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.855+0000 7ffa8df07640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffa88153140 0x7ffa88156b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:28.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.855+0000 7ffa8df07640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ffa88162310 con 0x7ffa880ab790 2024-04-25T00:15:28.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.855+0000 7ffa8df07640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ffa88162190 con 0x7ffa88152d60 2024-04-25T00:15:28.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.855+0000 7ffa8df07640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ffa88162490 con 0x7ffa88153140 2024-04-25T00:15:28.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.855+0000 7ffa877fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa880ab790 0x7ffa880ad280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:28.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.855+0000 7ffa877fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa880ab790 0x7ffa880ad280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33380/0 (socket says 172.21.15.116:33380) 2024-04-25T00:15:28.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.855+0000 7ffa877fe640 1 -- 172.21.15.116:0/2744318260 learned_addr learned my addr 172.21.15.116:0/2744318260 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:15:28.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.855+0000 7ffa86ffd640 1 --2- 172.21.15.116:0/2744318260 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa88152d60 0x7ffa8815f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:28.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.855+0000 7ffa877fe640 1 -- 172.21.15.116:0/2744318260 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffa88153140 msgr2=0x7ffa88156b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:28.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.855+0000 7ffa877fe640 1 --2- 172.21.15.116:0/2744318260 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffa88153140 0x7ffa88156b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:28.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.855+0000 7ffa877fe640 1 -- 172.21.15.116:0/2744318260 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa88152d60 msgr2=0x7ffa8815f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:28.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.855+0000 7ffa877fe640 1 --2- 172.21.15.116:0/2744318260 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa88152d60 0x7ffa8815f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:28.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.855+0000 7ffa877fe640 1 -- 172.21.15.116:0/2744318260 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa880c33d0 con 0x7ffa880ab790 2024-04-25T00:15:28.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.855+0000 7ffa877fe640 1 --2- 172.21.15.116:0/2744318260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa880ab790 0x7ffa880ad280 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7ffa7800d5f0 tx=0x7ffa7800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:28.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.855+0000 7ffa6ffff640 1 -- 172.21.15.116:0/2744318260 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa780125f0 con 0x7ffa880ab790 2024-04-25T00:15:28.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.855+0000 7ffa6ffff640 1 -- 172.21.15.116:0/2744318260 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa78019070 con 0x7ffa880ab790 2024-04-25T00:15:28.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.855+0000 7ffa8df07640 1 -- 172.21.15.116:0/2744318260 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa881eec20 con 0x7ffa880ab790 2024-04-25T00:15:28.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.855+0000 7ffa6ffff640 1 -- 172.21.15.116:0/2744318260 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa780129b0 con 0x7ffa880ab790 2024-04-25T00:15:28.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.855+0000 7ffa8df07640 1 -- 172.21.15.116:0/2744318260 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa881ede60 con 0x7ffa880ab790 2024-04-25T00:15:28.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.859+0000 7ffa6dffb640 1 -- 172.21.15.116:0/2744318260 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa600052a0 con 0x7ffa880ab790 2024-04-25T00:15:28.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.859+0000 7ffa6ffff640 1 -- 172.21.15.116:0/2744318260 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ffa78012b50 con 0x7ffa880ab790 2024-04-25T00:15:28.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.859+0000 7ffa6ffff640 1 --2- 172.21.15.116:0/2744318260 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ffa740678f0 0x7ffa74069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:28.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.859+0000 7ffa6ffff640 1 -- 172.21.15.116:0/2744318260 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 8386+0+0 (secure 0 0 0) 0x7ffa78086a90 con 0x7ffa880ab790 2024-04-25T00:15:28.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.859+0000 7ffa86ffd640 1 --2- 172.21.15.116:0/2744318260 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ffa740678f0 0x7ffa74069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:28.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.859+0000 7ffa86ffd640 1 --2- 172.21.15.116:0/2744318260 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ffa740678f0 0x7ffa74069db0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c000f70 tx=0x7ffa7c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:28.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.863+0000 7ffa6ffff640 1 -- 172.21.15.116:0/2744318260 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ffa78011340 con 0x7ffa880ab790 2024-04-25T00:15:28.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.867+0000 7f1305124640 1 -- 172.21.15.116:0/690164432 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1300068350 con 0x7f1300152d80 2024-04-25T00:15:28.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.867+0000 7f12e77fe640 1 -- 172.21.15.116:0/690164432 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) ==== 95+0+5680 (secure 0 0 0) 0x7f12f8055cb0 con 0x7f1300152d80 2024-04-25T00:15:28.869 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:15:28.870 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":111,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6810,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/2340581953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2340581953},{"type":"v1","addr":"172.21.15.116:6833","nonce":2340581953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:15:01.877414+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5273":{"gid":5273,"name":"d","rank":0,"incarnation":102,"state":"up:active","state_seq":49,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5273,"qdb_cluster":[5273]},"id":1},{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:15:28.737458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6828":{"gid":6828,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":8,"addr":"172.21.15.116:6835/2012470875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2012470875},{"type":"v1","addr":"172.21.15.116:6835","nonce":2012470875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6828,"qdb_cluster":[6828]},"id":2}]} 2024-04-25T00:15:28.871 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 111 2024-04-25T00:15:28.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.867+0000 7f1305124640 1 -- 172.21.15.116:0/690164432 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f12ec0678c0 msgr2=0x7f12ec069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:28.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.867+0000 7f1305124640 1 --2- 172.21.15.116:0/690164432 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f12ec0678c0 0x7f12ec069d80 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f12f00af000 tx=0x7f12f00a7000 comp rx=0 tx=0).stop 2024-04-25T00:15:28.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.867+0000 7f1305124640 1 -- 172.21.15.116:0/690164432 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1300152d80 msgr2=0x7f13000c3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:28.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.867+0000 7f1305124640 1 --2- 172.21.15.116:0/690164432 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1300152d80 0x7f13000c3ee0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f12f800b9a0 tx=0x7f12f800be60 comp rx=0 tx=0).stop 2024-04-25T00:15:28.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.867+0000 7f1305124640 1 -- 172.21.15.116:0/690164432 shutdown_connections 2024-04-25T00:15:28.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.867+0000 7f1305124640 1 --2- 172.21.15.116:0/690164432 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f12ec0678c0 0x7f12ec069d80 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:28.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.867+0000 7f1305124640 1 --2- 172.21.15.116:0/690164432 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1300152d80 0x7f13000c3ee0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:28.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.867+0000 7f1305124640 1 --2- 172.21.15.116:0/690164432 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f130006d5f0 0x7f13000c39a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:28.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.867+0000 7f1305124640 1 --2- 172.21.15.116:0/690164432 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f130006c290 0x7f13000c7040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:28.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.867+0000 7f1305124640 1 -- 172.21.15.116:0/690164432 >> 172.21.15.116:0/690164432 conn(0x7f13000acdd0 msgr2=0x7f1300158d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:28.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.867+0000 7f1305124640 1 -- 172.21.15.116:0/690164432 shutdown_connections 2024-04-25T00:15:28.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:28.867+0000 7f1305124640 1 -- 172.21.15.116:0/690164432 wait complete. 2024-04-25T00:15:28.886 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:15:28.962 INFO:tasks.ceph.mds.a.smithi116.stderr:2024-04-25T00:15:28.959+0000 7fbfefa48640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T00:15:29.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:29.059+0000 7ffa6dffb640 1 -- 172.21.15.116:0/2744318260 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffa60005590 con 0x7ffa880ab790 2024-04-25T00:15:29.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:29.063+0000 7ffa6ffff640 1 -- 172.21.15.116:0/2744318260 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) ==== 95+0+5680 (secure 0 0 0) 0x7ffa78056b90 con 0x7ffa880ab790 2024-04-25T00:15:29.064 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:15:29.064 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":111,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6810,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/2340581953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2340581953},{"type":"v1","addr":"172.21.15.116:6833","nonce":2340581953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:15:01.877414+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5273":{"gid":5273,"name":"d","rank":0,"incarnation":102,"state":"up:active","state_seq":49,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5273,"qdb_cluster":[5273]},"id":1},{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:15:28.737458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6828":{"gid":6828,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":8,"addr":"172.21.15.116:6835/2012470875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2012470875},{"type":"v1","addr":"172.21.15.116:6835","nonce":2012470875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6828,"qdb_cluster":[6828]},"id":2}]} 2024-04-25T00:15:29.064 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 111 2024-04-25T00:15:29.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:29.063+0000 7ffa6dffb640 1 -- 172.21.15.116:0/2744318260 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ffa740678f0 msgr2=0x7ffa74069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:29.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:29.063+0000 7ffa6dffb640 1 --2- 172.21.15.116:0/2744318260 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ffa740678f0 0x7ffa74069db0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c000f70 tx=0x7ffa7c009040 comp rx=0 tx=0).stop 2024-04-25T00:15:29.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:29.063+0000 7ffa6dffb640 1 -- 172.21.15.116:0/2744318260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa880ab790 msgr2=0x7ffa880ad280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:29.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:29.063+0000 7ffa6dffb640 1 --2- 172.21.15.116:0/2744318260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa880ab790 0x7ffa880ad280 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7ffa7800d5f0 tx=0x7ffa7800dab0 comp rx=0 tx=0).stop 2024-04-25T00:15:29.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:29.063+0000 7ffa6dffb640 1 -- 172.21.15.116:0/2744318260 shutdown_connections 2024-04-25T00:15:29.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:29.063+0000 7ffa6dffb640 1 --2- 172.21.15.116:0/2744318260 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ffa740678f0 0x7ffa74069db0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:29.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:29.063+0000 7ffa6dffb640 1 --2- 172.21.15.116:0/2744318260 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffa88153140 0x7ffa88156b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:29.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:29.063+0000 7ffa6dffb640 1 --2- 172.21.15.116:0/2744318260 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa88152d60 0x7ffa8815f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:29.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:29.063+0000 7ffa6dffb640 1 --2- 172.21.15.116:0/2744318260 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa880ab790 0x7ffa880ad280 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:29.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:29.063+0000 7ffa6dffb640 1 -- 172.21.15.116:0/2744318260 >> 172.21.15.116:0/2744318260 conn(0x7ffa8806f0c0 msgr2=0x7ffa8806cc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:29.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:29.063+0000 7ffa6dffb640 1 -- 172.21.15.116:0/2744318260 shutdown_connections 2024-04-25T00:15:29.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:29.063+0000 7ffa6dffb640 1 -- 172.21.15.116:0/2744318260 wait complete. 2024-04-25T00:15:29.082 INFO:tasks.mds_thrash.fs.[b]:mds.a has gained rank=0, replacing gid=5791 2024-04-25T00:15:29.082 INFO:tasks.mds_thrash.fs.[b]:waiting for 3 secs before reviving mds.b 2024-04-25T00:15:30.887 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:15:31.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.011+0000 7f6f1106c640 1 Processor -- start 2024-04-25T00:15:31.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.011+0000 7f6f1106c640 1 -- start start 2024-04-25T00:15:31.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.015+0000 7f6f1106c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6f0c150d20 0x7f6f0c0c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:31.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.015+0000 7f6f1106c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6f0c0c2f50 0x7f6f0c1538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:31.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.015+0000 7f6f1106c640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f0c0c3910 con 0x7f6f0c0a94a0 2024-04-25T00:15:31.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.015+0000 7f6f1106c640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f6f0c154530 con 0x7f6f0c150d20 2024-04-25T00:15:31.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.015+0000 7f6f1106c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6f0c0c16e0 con 0x7f6f0c0c2f50 2024-04-25T00:15:31.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.015+0000 7f6f0ad9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6f0c150d20 0x7f6f0c0c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:31.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.015+0000 7f6f0ad9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6f0c150d20 0x7f6f0c0c2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33392/0 (socket says 172.21.15.116:33392) 2024-04-25T00:15:31.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.015+0000 7f6f0ad9d640 1 -- 172.21.15.116:0/730265120 learned_addr learned my addr 172.21.15.116:0/730265120 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:15:31.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.015+0000 7f6f0b59e640 1 --1- 172.21.15.116:0/730265120 >> v1:172.21.15.116:6789/0 conn(0x7f6f0c0a94a0 0x7f6f0c153e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:51304/0 (socket says 172.21.15.116:51304) 2024-04-25T00:15:31.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.015+0000 7f6f0a59c640 1 --2- 172.21.15.116:0/730265120 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6f0c0c2f50 0x7f6f0c1538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:31.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.015+0000 7f6f0ad9d640 1 -- 172.21.15.116:0/730265120 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6f0c0c2f50 msgr2=0x7f6f0c1538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:31.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.015+0000 7f6f0ad9d640 1 --2- 172.21.15.116:0/730265120 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6f0c0c2f50 0x7f6f0c1538e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:31.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.015+0000 7f6f0ad9d640 1 -- 172.21.15.116:0/730265120 >> v1:172.21.15.116:6789/0 conn(0x7f6f0c0a94a0 legacy=0x7f6f0c153e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:31.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.015+0000 7f6f09d9b640 1 -- 172.21.15.116:0/730265120 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3885443536 0 0) 0x7f6f0c0c3910 con 0x7f6f0c0a94a0 2024-04-25T00:15:31.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.015+0000 7f6f0ad9d640 1 -- 172.21.15.116:0/730265120 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f0c0c1400 con 0x7f6f0c150d20 2024-04-25T00:15:31.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.015+0000 7f6f0ad9d640 1 --2- 172.21.15.116:0/730265120 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f0c150d20 0x7f6f0c0c2a10 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f6f04009e50 tx=0x7f6f0401ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc3a715190e5ad7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:31.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.015+0000 7f6f09d9b640 1 -- 172.21.15.116:0/730265120 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f040a5810 con 0x7f6f0c150d20 2024-04-25T00:15:31.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.019+0000 7f6f09d9b640 1 -- 172.21.15.116:0/730265120 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f040a59b0 con 0x7f6f0c150d20 2024-04-25T00:15:31.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.019+0000 7f6f09d9b640 1 -- 172.21.15.116:0/730265120 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f040a5d80 con 0x7f6f0c150d20 2024-04-25T00:15:31.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.019+0000 7f6f1106c640 1 -- 172.21.15.116:0/730265120 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f0c150d20 msgr2=0x7f6f0c0c2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:31.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.019+0000 7f6f1106c640 1 --2- 172.21.15.116:0/730265120 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f0c150d20 0x7f6f0c0c2a10 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f6f04009e50 tx=0x7f6f0401ca30 comp rx=0 tx=0).stop 2024-04-25T00:15:31.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.019+0000 7f6f1106c640 1 -- 172.21.15.116:0/730265120 shutdown_connections 2024-04-25T00:15:31.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.019+0000 7f6f1106c640 1 --2- 172.21.15.116:0/730265120 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6f0c0c2f50 0x7f6f0c1538e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:31.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.019+0000 7f6f1106c640 1 --2- 172.21.15.116:0/730265120 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f0c150d20 0x7f6f0c0c2a10 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:31.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.019+0000 7f6f1106c640 1 -- 172.21.15.116:0/730265120 >> 172.21.15.116:0/730265120 conn(0x7f6f0c0c3b00 msgr2=0x7f6f0c0c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:31.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.019+0000 7f6f1106c640 1 -- 172.21.15.116:0/730265120 shutdown_connections 2024-04-25T00:15:31.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.019+0000 7f6f1106c640 1 -- 172.21.15.116:0/730265120 wait complete. 2024-04-25T00:15:31.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.019+0000 7f6f1106c640 1 Processor -- start 2024-04-25T00:15:31.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.019+0000 7f6f1106c640 1 -- start start 2024-04-25T00:15:31.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.019+0000 7f6f1106c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f0c0a94a0 0x7f6f0c15c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:31.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.019+0000 7f6f1106c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6f0c0c2f50 0x7f6f0c143670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:31.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.019+0000 7f6f1106c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6f0c150d20 0x7f6f0c143bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:31.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.019+0000 7f6f1106c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6f0c1608b0 con 0x7f6f0c0a94a0 2024-04-25T00:15:31.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.019+0000 7f6f1106c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6f0c160730 con 0x7f6f0c150d20 2024-04-25T00:15:31.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.019+0000 7f6f1106c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6f0c160a30 con 0x7f6f0c0c2f50 2024-04-25T00:15:31.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.019+0000 7f6f0ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f0c0a94a0 0x7f6f0c15c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:31.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.019+0000 7f6f0ad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f0c0a94a0 0x7f6f0c15c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33394/0 (socket says 172.21.15.116:33394) 2024-04-25T00:15:31.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.019+0000 7f6f0ad9d640 1 -- 172.21.15.116:0/1089564477 learned_addr learned my addr 172.21.15.116:0/1089564477 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:15:31.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.019+0000 7f6f0a59c640 1 --2- 172.21.15.116:0/1089564477 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6f0c0c2f50 0x7f6f0c143670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:31.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.019+0000 7f6f0b59e640 1 --2- 172.21.15.116:0/1089564477 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6f0c150d20 0x7f6f0c143bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:31.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.019+0000 7f6f0ad9d640 1 -- 172.21.15.116:0/1089564477 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6f0c0c2f50 msgr2=0x7f6f0c143670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:31.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.019+0000 7f6f0ad9d640 1 --2- 172.21.15.116:0/1089564477 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6f0c0c2f50 0x7f6f0c143670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:31.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.019+0000 7f6f0ad9d640 1 -- 172.21.15.116:0/1089564477 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6f0c150d20 msgr2=0x7f6f0c143bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:31.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.019+0000 7f6f0ad9d640 1 --2- 172.21.15.116:0/1089564477 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6f0c150d20 0x7f6f0c143bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:31.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.019+0000 7f6f0ad9d640 1 -- 172.21.15.116:0/1089564477 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f0c1f2750 con 0x7f6f0c0a94a0 2024-04-25T00:15:31.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.019+0000 7f6f0ad9d640 1 --2- 172.21.15.116:0/1089564477 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f0c0a94a0 0x7f6f0c15c550 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f6f0401a660 tx=0x7f6f040197d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:31.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.019+0000 7f6ef37fe640 1 -- 172.21.15.116:0/1089564477 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f040bb020 con 0x7f6f0c0a94a0 2024-04-25T00:15:31.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.019+0000 7f6ef37fe640 1 -- 172.21.15.116:0/1089564477 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f040b6660 con 0x7f6f0c0a94a0 2024-04-25T00:15:31.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.019+0000 7f6f1106c640 1 -- 172.21.15.116:0/1089564477 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f0c070e70 con 0x7f6f0c0a94a0 2024-04-25T00:15:31.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.019+0000 7f6ef37fe640 1 -- 172.21.15.116:0/1089564477 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f040b6a50 con 0x7f6f0c0a94a0 2024-04-25T00:15:31.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.019+0000 7f6f1106c640 1 -- 172.21.15.116:0/1089564477 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f0c1f2470 con 0x7f6f0c0a94a0 2024-04-25T00:15:31.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.023+0000 7f6f1106c640 1 -- 172.21.15.116:0/1089564477 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f0c154c80 con 0x7f6f0c0a94a0 2024-04-25T00:15:31.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.023+0000 7f6ef37fe640 1 -- 172.21.15.116:0/1089564477 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6f040be020 con 0x7f6f0c0a94a0 2024-04-25T00:15:31.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.023+0000 7f6ef37fe640 1 --2- 172.21.15.116:0/1089564477 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6ef80678c0 0x7f6ef8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:31.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.023+0000 7f6ef37fe640 1 -- 172.21.15.116:0/1089564477 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 8386+0+0 (secure 0 0 0) 0x7f6f04121600 con 0x7f6f0c0a94a0 2024-04-25T00:15:31.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.023+0000 7f6f0a59c640 1 --2- 172.21.15.116:0/1089564477 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6ef80678c0 0x7f6ef8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:31.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.023+0000 7f6f0a59c640 1 --2- 172.21.15.116:0/1089564477 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6ef80678c0 0x7f6ef8069d80 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c1ec190 tx=0x7f6f00008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:31.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.027+0000 7f6ef37fe640 1 -- 172.21.15.116:0/1089564477 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f6f040c5030 con 0x7f6f0c0a94a0 2024-04-25T00:15:31.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.207+0000 7f6f1106c640 1 -- 172.21.15.116:0/1089564477 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6f0c068350 con 0x7f6f0c0a94a0 2024-04-25T00:15:31.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.207+0000 7f6ef37fe640 1 -- 172.21.15.116:0/1089564477 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) ==== 95+0+5680 (secure 0 0 0) 0x7f6f04016620 con 0x7f6f0c0a94a0 2024-04-25T00:15:31.211 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:15:31.211 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":111,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6810,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/2340581953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2340581953},{"type":"v1","addr":"172.21.15.116:6833","nonce":2340581953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:15:01.877414+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5273":{"gid":5273,"name":"d","rank":0,"incarnation":102,"state":"up:active","state_seq":49,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5273,"qdb_cluster":[5273]},"id":1},{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:15:28.737458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6828":{"gid":6828,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":8,"addr":"172.21.15.116:6835/2012470875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2012470875},{"type":"v1","addr":"172.21.15.116:6835","nonce":2012470875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6828,"qdb_cluster":[6828]},"id":2}]} 2024-04-25T00:15:31.211 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 111 2024-04-25T00:15:31.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.211+0000 7f6f1106c640 1 -- 172.21.15.116:0/1089564477 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6ef80678c0 msgr2=0x7f6ef8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:31.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.211+0000 7f6f1106c640 1 --2- 172.21.15.116:0/1089564477 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6ef80678c0 0x7f6ef8069d80 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c1ec190 tx=0x7f6f00008040 comp rx=0 tx=0).stop 2024-04-25T00:15:31.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.211+0000 7f6f1106c640 1 -- 172.21.15.116:0/1089564477 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f0c0a94a0 msgr2=0x7f6f0c15c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:31.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.211+0000 7f6f1106c640 1 --2- 172.21.15.116:0/1089564477 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f0c0a94a0 0x7f6f0c15c550 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f6f0401a660 tx=0x7f6f040197d0 comp rx=0 tx=0).stop 2024-04-25T00:15:31.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.211+0000 7f6f1106c640 1 -- 172.21.15.116:0/1089564477 shutdown_connections 2024-04-25T00:15:31.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.211+0000 7f6f1106c640 1 --2- 172.21.15.116:0/1089564477 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6ef80678c0 0x7f6ef8069d80 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:31.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.211+0000 7f6f1106c640 1 --2- 172.21.15.116:0/1089564477 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6f0c150d20 0x7f6f0c143bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:31.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.211+0000 7f6f1106c640 1 --2- 172.21.15.116:0/1089564477 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6f0c0c2f50 0x7f6f0c143670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:31.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.211+0000 7f6f1106c640 1 --2- 172.21.15.116:0/1089564477 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f0c0a94a0 0x7f6f0c15c550 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:31.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.211+0000 7f6f1106c640 1 -- 172.21.15.116:0/1089564477 >> 172.21.15.116:0/1089564477 conn(0x7f6f0c0c3b00 msgr2=0x7f6f0c146fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:31.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.211+0000 7f6f1106c640 1 -- 172.21.15.116:0/1089564477 shutdown_connections 2024-04-25T00:15:31.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:31.211+0000 7f6f1106c640 1 -- 172.21.15.116:0/1089564477 wait complete. 2024-04-25T00:15:31.228 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:15:32.083 INFO:tasks.mds_thrash.fs.[b]:reviving mds.b 2024-04-25T00:15:32.084 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-25T00:15:32.084 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-25T00:15:32.087 INFO:tasks.ceph.mds.b:Started 2024-04-25T00:15:32.087 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:15:32.164 INFO:tasks.ceph.mds.b.smithi174.stdout:starting mds.b at 2024-04-25T00:15:32.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.215+0000 7fc64c3b8640 1 Processor -- start 2024-04-25T00:15:32.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.215+0000 7fc64c3b8640 1 -- start start 2024-04-25T00:15:32.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.215+0000 7fc64c3b8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc6440a94a0 0x7fc6441538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:32.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.215+0000 7fc64c3b8640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6440c3910 con 0x7fc6440a99f0 2024-04-25T00:15:32.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.215+0000 7fc64c3b8640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc644154530 con 0x7fc644150d20 2024-04-25T00:15:32.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.215+0000 7fc64c3b8640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fc6440c16e0 con 0x7fc6440a94a0 2024-04-25T00:15:32.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.215+0000 7fc649953640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc6440a94a0 0x7fc6441538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:32.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.215+0000 7fc64a955640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fc6440a99f0 0x7fc644153e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:51312/0 (socket says 172.21.15.116:51312) 2024-04-25T00:15:32.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.215+0000 7fc64a955640 1 -- 172.21.15.116:0/726911733 learned_addr learned my addr 172.21.15.116:0/726911733 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:15:32.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.215+0000 7fc649152640 1 -- 172.21.15.116:0/726911733 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1932573065 0 0) 0x7fc6440c3910 con 0x7fc6440a99f0 2024-04-25T00:15:32.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.215+0000 7fc649152640 1 -- 172.21.15.116:0/726911733 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc630003530 con 0x7fc6440a99f0 2024-04-25T00:15:32.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.215+0000 7fc649152640 1 -- 172.21.15.116:0/726911733 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3829631986 0 0) 0x7fc644154530 con 0x7fc644150d20 2024-04-25T00:15:32.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.215+0000 7fc649152640 1 -- 172.21.15.116:0/726911733 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc6440c3910 con 0x7fc644150d20 2024-04-25T00:15:32.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.215+0000 7fc649953640 1 -- 172.21.15.116:0/726911733 >> v1:172.21.15.174:6789/0 conn(0x7fc644150d20 legacy=0x7fc6440c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:32.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.215+0000 7fc649953640 1 -- 172.21.15.116:0/726911733 >> v1:172.21.15.116:6789/0 conn(0x7fc6440a99f0 legacy=0x7fc644153e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:32.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.215+0000 7fc649953640 1 -- 172.21.15.116:0/726911733 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6440c1400 con 0x7fc6440a94a0 2024-04-25T00:15:32.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.219+0000 7fc649953640 1 --2- 172.21.15.116:0/726911733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6440a94a0 0x7fc6441538e0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fc6340097c0 tx=0x7fc63401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9e2f88dc0d00372 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:32.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.219+0000 7fc649152640 1 -- 172.21.15.116:0/726911733 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6340b1070 con 0x7fc6440a94a0 2024-04-25T00:15:32.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.219+0000 7fc649152640 1 -- 172.21.15.116:0/726911733 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6340ac410 con 0x7fc6440a94a0 2024-04-25T00:15:32.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.219+0000 7fc649152640 1 -- 172.21.15.116:0/726911733 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6340ac7e0 con 0x7fc6440a94a0 2024-04-25T00:15:32.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.219+0000 7fc64c3b8640 1 -- 172.21.15.116:0/726911733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6440a94a0 msgr2=0x7fc6441538e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:32.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.219+0000 7fc64c3b8640 1 --2- 172.21.15.116:0/726911733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6440a94a0 0x7fc6441538e0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fc6340097c0 tx=0x7fc63401c4a0 comp rx=0 tx=0).stop 2024-04-25T00:15:32.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.219+0000 7fc64c3b8640 1 -- 172.21.15.116:0/726911733 shutdown_connections 2024-04-25T00:15:32.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.219+0000 7fc64c3b8640 1 --2- 172.21.15.116:0/726911733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6440a94a0 0x7fc6441538e0 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:32.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.219+0000 7fc64c3b8640 1 -- 172.21.15.116:0/726911733 >> 172.21.15.116:0/726911733 conn(0x7fc6440c3b00 msgr2=0x7fc6440c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:32.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.219+0000 7fc64c3b8640 1 -- 172.21.15.116:0/726911733 shutdown_connections 2024-04-25T00:15:32.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.219+0000 7fc64c3b8640 1 -- 172.21.15.116:0/726911733 wait complete. 2024-04-25T00:15:32.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.219+0000 7fc64c3b8640 1 Processor -- start 2024-04-25T00:15:32.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.219+0000 7fc64c3b8640 1 -- start start 2024-04-25T00:15:32.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.219+0000 7fc64c3b8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6440a94a0 0x7fc644145d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:32.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.219+0000 7fc64c3b8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc6440a99f0 0x7fc6441422c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:32.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.219+0000 7fc64c3b8640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc644150d20 0x7fc64413f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:32.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.219+0000 7fc64c3b8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc6440c32a0 con 0x7fc6440a94a0 2024-04-25T00:15:32.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.219+0000 7fc64c3b8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc6440c3120 con 0x7fc6440a99f0 2024-04-25T00:15:32.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.219+0000 7fc64c3b8640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fc644161740 con 0x7fc644150d20 2024-04-25T00:15:32.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.219+0000 7fc64a154640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6440a94a0 0x7fc644145d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:32.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.219+0000 7fc64a154640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6440a94a0 0x7fc644145d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33402/0 (socket says 172.21.15.116:33402) 2024-04-25T00:15:32.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.219+0000 7fc64a154640 1 -- 172.21.15.116:0/854582065 learned_addr learned my addr 172.21.15.116:0/854582065 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:15:32.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.219+0000 7fc64a955640 1 --2- 172.21.15.116:0/854582065 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc644150d20 0x7fc64413f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:32.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.219+0000 7fc649953640 1 --2- 172.21.15.116:0/854582065 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc6440a99f0 0x7fc6441422c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:32.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.219+0000 7fc64a154640 1 -- 172.21.15.116:0/854582065 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc644150d20 msgr2=0x7fc64413f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:32.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.219+0000 7fc64a154640 1 --2- 172.21.15.116:0/854582065 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc644150d20 0x7fc64413f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:32.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.219+0000 7fc64a154640 1 -- 172.21.15.116:0/854582065 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc6440a99f0 msgr2=0x7fc6441422c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:32.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.219+0000 7fc64a154640 1 --2- 172.21.15.116:0/854582065 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc6440a99f0 0x7fc6441422c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:32.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.219+0000 7fc64a154640 1 -- 172.21.15.116:0/854582065 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6441ec880 con 0x7fc6440a94a0 2024-04-25T00:15:32.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.219+0000 7fc64a154640 1 --2- 172.21.15.116:0/854582065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6440a94a0 0x7fc644145d60 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fc64000c460 tx=0x7fc64000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:32.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.219+0000 7fc63affd640 1 -- 172.21.15.116:0/854582065 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc640017070 con 0x7fc6440a94a0 2024-04-25T00:15:32.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.219+0000 7fc63affd640 1 -- 172.21.15.116:0/854582065 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6400052f0 con 0x7fc6440a94a0 2024-04-25T00:15:32.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.219+0000 7fc64c3b8640 1 -- 172.21.15.116:0/854582065 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6441ecb60 con 0x7fc6440a94a0 2024-04-25T00:15:32.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.219+0000 7fc63affd640 1 -- 172.21.15.116:0/854582065 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc640003840 con 0x7fc6440a94a0 2024-04-25T00:15:32.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.219+0000 7fc64c3b8640 1 -- 172.21.15.116:0/854582065 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc644143cd0 con 0x7fc6440a94a0 2024-04-25T00:15:32.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.223+0000 7fc64c3b8640 1 -- 172.21.15.116:0/854582065 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc644068350 con 0x7fc6440a94a0 2024-04-25T00:15:32.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.223+0000 7fc63affd640 1 -- 172.21.15.116:0/854582065 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc64001a070 con 0x7fc6440a94a0 2024-04-25T00:15:32.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.223+0000 7fc63affd640 1 --2- 172.21.15.116:0/854582065 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc6280678f0 0x7fc628069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:32.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.223+0000 7fc63affd640 1 -- 172.21.15.116:0/854582065 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 8386+0+0 (secure 0 0 0) 0x7fc6400859b0 con 0x7fc6440a94a0 2024-04-25T00:15:32.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.223+0000 7fc649953640 1 --2- 172.21.15.116:0/854582065 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc6280678f0 0x7fc628069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:32.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.223+0000 7fc649953640 1 --2- 172.21.15.116:0/854582065 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc6280678f0 0x7fc628069db0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fc6340ae000 tx=0x7fc634008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:32.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.227+0000 7fc63affd640 1 -- 172.21.15.116:0/854582065 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fc640055460 con 0x7fc6440a94a0 2024-04-25T00:15:32.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.423+0000 7fc64c3b8640 1 -- 172.21.15.116:0/854582065 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc6440c34f0 con 0x7fc6440a94a0 2024-04-25T00:15:32.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.427+0000 7fc63affd640 1 -- 172.21.15.116:0/854582065 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) ==== 95+0+5680 (secure 0 0 0) 0x7fc640059760 con 0x7fc6440a94a0 2024-04-25T00:15:32.427 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:15:32.427 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":111,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6810,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/2340581953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2340581953},{"type":"v1","addr":"172.21.15.116:6833","nonce":2340581953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:15:01.877414+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5273":{"gid":5273,"name":"d","rank":0,"incarnation":102,"state":"up:active","state_seq":49,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5273,"qdb_cluster":[5273]},"id":1},{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:15:28.737458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6828":{"gid":6828,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":8,"addr":"172.21.15.116:6835/2012470875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2012470875},{"type":"v1","addr":"172.21.15.116:6835","nonce":2012470875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6828,"qdb_cluster":[6828]},"id":2}]} 2024-04-25T00:15:32.427 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 111 2024-04-25T00:15:32.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.431+0000 7fc64c3b8640 1 -- 172.21.15.116:0/854582065 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc6280678f0 msgr2=0x7fc628069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:32.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.431+0000 7fc64c3b8640 1 --2- 172.21.15.116:0/854582065 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc6280678f0 0x7fc628069db0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fc6340ae000 tx=0x7fc634008040 comp rx=0 tx=0).stop 2024-04-25T00:15:32.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.431+0000 7fc64c3b8640 1 -- 172.21.15.116:0/854582065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6440a94a0 msgr2=0x7fc644145d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:32.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.431+0000 7fc64c3b8640 1 --2- 172.21.15.116:0/854582065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6440a94a0 0x7fc644145d60 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fc64000c460 tx=0x7fc64000c920 comp rx=0 tx=0).stop 2024-04-25T00:15:32.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.431+0000 7fc64c3b8640 1 -- 172.21.15.116:0/854582065 shutdown_connections 2024-04-25T00:15:32.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.431+0000 7fc64c3b8640 1 --2- 172.21.15.116:0/854582065 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc6280678f0 0x7fc628069db0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:32.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.431+0000 7fc64c3b8640 1 --2- 172.21.15.116:0/854582065 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc644150d20 0x7fc64413f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:32.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.431+0000 7fc64c3b8640 1 --2- 172.21.15.116:0/854582065 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc6440a99f0 0x7fc6441422c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:32.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.431+0000 7fc64c3b8640 1 --2- 172.21.15.116:0/854582065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6440a94a0 0x7fc644145d60 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:32.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.431+0000 7fc64c3b8640 1 -- 172.21.15.116:0/854582065 >> 172.21.15.116:0/854582065 conn(0x7fc6440c3b00 msgr2=0x7fc644146f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:32.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.431+0000 7fc64c3b8640 1 -- 172.21.15.116:0/854582065 shutdown_connections 2024-04-25T00:15:32.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:32.431+0000 7fc64c3b8640 1 -- 172.21.15.116:0/854582065 wait complete. 2024-04-25T00:15:32.448 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2024-04-25T00:15:33.229 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:15:33.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.359+0000 7fb409e3e640 1 Processor -- start 2024-04-25T00:15:33.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.359+0000 7fb409e3e640 1 -- start start 2024-04-25T00:15:33.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.359+0000 7fb409e3e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb404152d80 0x7fb40406bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:33.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.359+0000 7fb409e3e640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fb40406d5f0 0x7fb404156c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:33.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.359+0000 7fb409e3e640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4040c9150 con 0x7fb40406c290 2024-04-25T00:15:33.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.359+0000 7fb409e3e640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fb4040c9d60 con 0x7fb404152d80 2024-04-25T00:15:33.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.359+0000 7fb409e3e640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fb4040c9430 con 0x7fb40406d5f0 2024-04-25T00:15:33.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.359+0000 7fb3fbfff640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fb40406d5f0 0x7fb404156c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:33.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.359+0000 7fb4037fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb404152d80 0x7fb40406bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:33.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.359+0000 7fb3fbfff640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fb40406d5f0 0x7fb404156c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:49284/0 (socket says 172.21.15.116:49284) 2024-04-25T00:15:33.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.359+0000 7fb4037fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb404152d80 0x7fb40406bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33406/0 (socket says 172.21.15.116:33406) 2024-04-25T00:15:33.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.359+0000 7fb3fbfff640 1 -- 172.21.15.116:0/3316970599 learned_addr learned my addr 172.21.15.116:0/3316970599 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:15:33.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.359+0000 7fb403fff640 1 --1- 172.21.15.116:0/3316970599 >> v1:172.21.15.174:6789/0 conn(0x7fb40406c290 0x7fb4041571c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:48156/0 (socket says 172.21.15.116:48156) 2024-04-25T00:15:33.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.359+0000 7fb4037fe640 1 -- 172.21.15.116:0/3316970599 >> v2:172.21.15.174:3300/0 conn(0x7fb40406d5f0 msgr2=0x7fb404156c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:33.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.359+0000 7fb4037fe640 1 --2- 172.21.15.116:0/3316970599 >> v2:172.21.15.174:3300/0 conn(0x7fb40406d5f0 0x7fb404156c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:33.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.359+0000 7fb4037fe640 1 -- 172.21.15.116:0/3316970599 >> v1:172.21.15.174:6789/0 conn(0x7fb40406c290 legacy=0x7fb4041571c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:33.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.359+0000 7fb4037fe640 1 -- 172.21.15.116:0/3316970599 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4040c9710 con 0x7fb404152d80 2024-04-25T00:15:33.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.363+0000 7fb4037fe640 1 --2- 172.21.15.116:0/3316970599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb404152d80 0x7fb40406bd50 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fb3f40097c0 tx=0x7fb3f401c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=946192afd246da9f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:33.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.363+0000 7fb402ffd640 1 -- 172.21.15.116:0/3316970599 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f40b1070 con 0x7fb404152d80 2024-04-25T00:15:33.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.363+0000 7fb402ffd640 1 -- 172.21.15.116:0/3316970599 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3f40ac620 con 0x7fb404152d80 2024-04-25T00:15:33.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.363+0000 7fb402ffd640 1 -- 172.21.15.116:0/3316970599 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f40ac9f0 con 0x7fb404152d80 2024-04-25T00:15:33.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.363+0000 7fb409e3e640 1 -- 172.21.15.116:0/3316970599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb404152d80 msgr2=0x7fb40406bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:33.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.363+0000 7fb409e3e640 1 --2- 172.21.15.116:0/3316970599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb404152d80 0x7fb40406bd50 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fb3f40097c0 tx=0x7fb3f401c3a0 comp rx=0 tx=0).stop 2024-04-25T00:15:33.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.363+0000 7fb409e3e640 1 -- 172.21.15.116:0/3316970599 shutdown_connections 2024-04-25T00:15:33.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.363+0000 7fb409e3e640 1 --2- 172.21.15.116:0/3316970599 >> v2:172.21.15.174:3300/0 conn(0x7fb40406d5f0 0x7fb404156c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:33.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.363+0000 7fb409e3e640 1 --2- 172.21.15.116:0/3316970599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb404152d80 0x7fb40406bd50 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:33.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.363+0000 7fb409e3e640 1 -- 172.21.15.116:0/3316970599 >> 172.21.15.116:0/3316970599 conn(0x7fb4040acdd0 msgr2=0x7fb4040ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:33.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.363+0000 7fb409e3e640 1 -- 172.21.15.116:0/3316970599 shutdown_connections 2024-04-25T00:15:33.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.363+0000 7fb409e3e640 1 -- 172.21.15.116:0/3316970599 wait complete. 2024-04-25T00:15:33.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.363+0000 7fb409e3e640 1 Processor -- start 2024-04-25T00:15:33.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.363+0000 7fb409e3e640 1 -- start start 2024-04-25T00:15:33.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.367+0000 7fb409e3e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb40406c290 0x7fb4040c7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:33.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.367+0000 7fb409e3e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb40406d5f0 0x7fb404151860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:33.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.367+0000 7fb409e3e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb404152d80 0x7fb4040adcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:33.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.367+0000 7fb409e3e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb4040caf00 con 0x7fb40406d5f0 2024-04-25T00:15:33.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.367+0000 7fb409e3e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb4040cad80 con 0x7fb40406c290 2024-04-25T00:15:33.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.367+0000 7fb409e3e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fb4041628d0 con 0x7fb404152d80 2024-04-25T00:15:33.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.367+0000 7fb3fbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb40406d5f0 0x7fb404151860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:33.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.367+0000 7fb3fbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb40406d5f0 0x7fb404151860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33408/0 (socket says 172.21.15.116:33408) 2024-04-25T00:15:33.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.367+0000 7fb3fbfff640 1 -- 172.21.15.116:0/4084581325 learned_addr learned my addr 172.21.15.116:0/4084581325 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:15:33.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.367+0000 7fb4037fe640 1 --2- 172.21.15.116:0/4084581325 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb40406c290 0x7fb4040c7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:33.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.367+0000 7fb403fff640 1 --2- 172.21.15.116:0/4084581325 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb404152d80 0x7fb4040adcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:33.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.367+0000 7fb3fbfff640 1 -- 172.21.15.116:0/4084581325 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb404152d80 msgr2=0x7fb4040adcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:33.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.367+0000 7fb3fbfff640 1 --2- 172.21.15.116:0/4084581325 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb404152d80 0x7fb4040adcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:33.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.367+0000 7fb3fbfff640 1 -- 172.21.15.116:0/4084581325 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb40406c290 msgr2=0x7fb4040c7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:33.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.367+0000 7fb3fbfff640 1 --2- 172.21.15.116:0/4084581325 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb40406c290 0x7fb4040c7870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:33.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.367+0000 7fb3fbfff640 1 -- 172.21.15.116:0/4084581325 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4041f2800 con 0x7fb40406d5f0 2024-04-25T00:15:33.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.367+0000 7fb403fff640 1 --2- 172.21.15.116:0/4084581325 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb404152d80 0x7fb4040adcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:15:33.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.367+0000 7fb3fbfff640 1 --2- 172.21.15.116:0/4084581325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb40406d5f0 0x7fb404151860 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec00e890 tx=0x7fb3ec00ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:33.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.367+0000 7fb400ff9640 1 -- 172.21.15.116:0/4084581325 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3ec017070 con 0x7fb40406d5f0 2024-04-25T00:15:33.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.367+0000 7fb400ff9640 1 -- 172.21.15.116:0/4084581325 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3ec00c940 con 0x7fb40406d5f0 2024-04-25T00:15:33.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.367+0000 7fb400ff9640 1 -- 172.21.15.116:0/4084581325 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3ec00cd30 con 0x7fb40406d5f0 2024-04-25T00:15:33.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.367+0000 7fb409e3e640 1 -- 172.21.15.116:0/4084581325 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4040c6bd0 con 0x7fb40406d5f0 2024-04-25T00:15:33.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.367+0000 7fb409e3e640 1 -- 172.21.15.116:0/4084581325 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb404151da0 con 0x7fb40406d5f0 2024-04-25T00:15:33.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.367+0000 7fb409e3e640 1 -- 172.21.15.116:0/4084581325 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb404068350 con 0x7fb40406d5f0 2024-04-25T00:15:33.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.367+0000 7fb400ff9640 1 -- 172.21.15.116:0/4084581325 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb3ec01a070 con 0x7fb40406d5f0 2024-04-25T00:15:33.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.367+0000 7fb400ff9640 1 --2- 172.21.15.116:0/4084581325 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb3e40678f0 0x7fb3e4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:33.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.367+0000 7fb400ff9640 1 -- 172.21.15.116:0/4084581325 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 8386+0+0 (secure 0 0 0) 0x7fb3ec086600 con 0x7fb40406d5f0 2024-04-25T00:15:33.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.371+0000 7fb4037fe640 1 --2- 172.21.15.116:0/4084581325 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb3e40678f0 0x7fb3e4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:33.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.371+0000 7fb4037fe640 1 --2- 172.21.15.116:0/4084581325 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb3e40678f0 0x7fb3e4069db0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fb3f40ae000 tx=0x7fb3f4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:33.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.375+0000 7fb400ff9640 1 -- 172.21.15.116:0/4084581325 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fb3ec056020 con 0x7fb40406d5f0 2024-04-25T00:15:33.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.551+0000 7fb409e3e640 1 -- 172.21.15.116:0/4084581325 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb4040c75d0 con 0x7fb40406d5f0 2024-04-25T00:15:33.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.555+0000 7fb400ff9640 1 -- 172.21.15.116:0/4084581325 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) ==== 95+0+6509 (secure 0 0 0) 0x7fb3ec05a320 con 0x7fb40406d5f0 2024-04-25T00:15:33.555 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:15:33.555 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":112,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6810,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/2340581953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2340581953},{"type":"v1","addr":"172.21.15.116:6833","nonce":2340581953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:15:01.877414+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5273":{"gid":5273,"name":"d","rank":0,"incarnation":102,"state":"up:active","state_seq":49,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5273,"qdb_cluster":[5273]},"id":1},{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:15:28.737458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6828":{"gid":6828,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":8,"addr":"172.21.15.116:6835/2012470875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2012470875},{"type":"v1","addr":"172.21.15.116:6835","nonce":2012470875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6828,"qdb_cluster":[6828]},"id":2}]} 2024-04-25T00:15:33.555 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 112 2024-04-25T00:15:33.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.555+0000 7fb409e3e640 1 -- 172.21.15.116:0/4084581325 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb3e40678f0 msgr2=0x7fb3e4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:33.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.555+0000 7fb409e3e640 1 --2- 172.21.15.116:0/4084581325 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb3e40678f0 0x7fb3e4069db0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fb3f40ae000 tx=0x7fb3f4008040 comp rx=0 tx=0).stop 2024-04-25T00:15:33.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.555+0000 7fb409e3e640 1 -- 172.21.15.116:0/4084581325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb40406d5f0 msgr2=0x7fb404151860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:33.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.555+0000 7fb409e3e640 1 --2- 172.21.15.116:0/4084581325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb40406d5f0 0x7fb404151860 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec00e890 tx=0x7fb3ec00ed50 comp rx=0 tx=0).stop 2024-04-25T00:15:33.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.555+0000 7fb409e3e640 1 -- 172.21.15.116:0/4084581325 shutdown_connections 2024-04-25T00:15:33.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.555+0000 7fb409e3e640 1 --2- 172.21.15.116:0/4084581325 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb3e40678f0 0x7fb3e4069db0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:33.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.555+0000 7fb409e3e640 1 --2- 172.21.15.116:0/4084581325 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb404152d80 0x7fb4040adcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:33.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.555+0000 7fb409e3e640 1 --2- 172.21.15.116:0/4084581325 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb40406d5f0 0x7fb404151860 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:33.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.555+0000 7fb409e3e640 1 --2- 172.21.15.116:0/4084581325 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb40406c290 0x7fb4040c7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:33.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.555+0000 7fb409e3e640 1 -- 172.21.15.116:0/4084581325 >> 172.21.15.116:0/4084581325 conn(0x7fb4040acdd0 msgr2=0x7fb40414df60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:33.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.555+0000 7fb409e3e640 1 -- 172.21.15.116:0/4084581325 shutdown_connections 2024-04-25T00:15:33.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:33.555+0000 7fb409e3e640 1 -- 172.21.15.116:0/4084581325 wait complete. 2024-04-25T00:15:33.573 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:15:34.449 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:15:34.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.579+0000 7f88f9666640 1 Processor -- start 2024-04-25T00:15:34.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.579+0000 7f88f9666640 1 -- start start 2024-04-25T00:15:34.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.579+0000 7f88f9666640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f88f4150d40 0x7f88f4154170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:34.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.579+0000 7f88f9666640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88f4070770 0x7f88f40ab200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:34.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.579+0000 7f88f9666640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88f4070180 con 0x7f88f40c4c20 2024-04-25T00:15:34.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.579+0000 7f88f9666640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f88f40c7530 con 0x7f88f4150d40 2024-04-25T00:15:34.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.579+0000 7f88f9666640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f88f4071280 con 0x7f88f4070770 2024-04-25T00:15:34.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.579+0000 7f88f27fc640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f88f40c4c20 0x7f88f40c6e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:51316/0 (socket says 172.21.15.116:51316) 2024-04-25T00:15:34.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.579+0000 7f88f27fc640 1 -- 172.21.15.116:0/2240907061 learned_addr learned my addr 172.21.15.116:0/2240907061 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:15:34.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.579+0000 7f88f2ffd640 1 --2- 172.21.15.116:0/2240907061 >> v2:172.21.15.174:3300/0 conn(0x7f88f4150d40 0x7f88f4154170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:34.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.579+0000 7f88f37fe640 1 --2- 172.21.15.116:0/2240907061 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88f4070770 0x7f88f40ab200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:34.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.579+0000 7f88f1ffb640 1 -- 172.21.15.116:0/2240907061 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1647922680 0 0) 0x7f88f4070180 con 0x7f88f40c4c20 2024-04-25T00:15:34.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.579+0000 7f88f1ffb640 1 -- 172.21.15.116:0/2240907061 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88dc003530 con 0x7f88f40c4c20 2024-04-25T00:15:34.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.579+0000 7f88f1ffb640 1 -- 172.21.15.116:0/2240907061 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2413666366 0 0) 0x7f88dc003530 con 0x7f88f40c4c20 2024-04-25T00:15:34.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.579+0000 7f88f1ffb640 1 -- 172.21.15.116:0/2240907061 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88f4070770 msgr2=0x7f88f40ab200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:34.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.579+0000 7f88f1ffb640 1 --2- 172.21.15.116:0/2240907061 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88f4070770 0x7f88f40ab200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:34.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.579+0000 7f88f1ffb640 1 -- 172.21.15.116:0/2240907061 >> v2:172.21.15.174:3300/0 conn(0x7f88f4150d40 msgr2=0x7f88f4154170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:34.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.579+0000 7f88f1ffb640 1 --2- 172.21.15.116:0/2240907061 >> v2:172.21.15.174:3300/0 conn(0x7f88f4150d40 0x7f88f4154170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:34.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.579+0000 7f88f1ffb640 1 -- 172.21.15.116:0/2240907061 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88f4070460 con 0x7f88f40c4c20 2024-04-25T00:15:34.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.579+0000 7f88f1ffb640 1 -- 172.21.15.116:0/2240907061 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f88e8001680 con 0x7f88f40c4c20 2024-04-25T00:15:34.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.579+0000 7f88f1ffb640 1 -- 172.21.15.116:0/2240907061 >> v1:172.21.15.116:6789/0 conn(0x7f88f40c4c20 legacy=0x7f88f40c6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:34.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.579+0000 7f88f1ffb640 1 --2- 172.21.15.116:0/2240907061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88dc0039c0 0x7f88dc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:34.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.579+0000 7f88f1ffb640 1 --2- 172.21.15.116:0/2240907061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88dc004810 0x7f88dc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:34.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.579+0000 7f88f1ffb640 1 --2- 172.21.15.116:0/2240907061 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88f4070770 0x7f88dc0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:34.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.579+0000 7f88f1ffb640 1 -- 172.21.15.116:0/2240907061 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f88f40c7530 con 0x7f88dc0039c0 2024-04-25T00:15:34.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.579+0000 7f88f1ffb640 1 -- 172.21.15.116:0/2240907061 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f88f4071280 con 0x7f88dc004810 2024-04-25T00:15:34.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.579+0000 7f88f1ffb640 1 -- 172.21.15.116:0/2240907061 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f88dc00ba00 con 0x7f88f4070770 2024-04-25T00:15:34.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.579+0000 7f88f2ffd640 1 --2- 172.21.15.116:0/2240907061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88dc0039c0 0x7f88dc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:34.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.579+0000 7f88f27fc640 1 --2- 172.21.15.116:0/2240907061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88dc004810 0x7f88dc004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:34.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.579+0000 7f88f2ffd640 1 -- 172.21.15.116:0/2240907061 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88f4070770 msgr2=0x7f88dc0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:34.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.579+0000 7f88f2ffd640 1 --2- 172.21.15.116:0/2240907061 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88f4070770 0x7f88dc0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:34.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.579+0000 7f88f2ffd640 1 -- 172.21.15.116:0/2240907061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88dc004810 msgr2=0x7f88dc004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:34.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.579+0000 7f88f2ffd640 1 --2- 172.21.15.116:0/2240907061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88dc004810 0x7f88dc004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:34.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.579+0000 7f88f2ffd640 1 -- 172.21.15.116:0/2240907061 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88dc003530 con 0x7f88dc0039c0 2024-04-25T00:15:34.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.583+0000 7f88f2ffd640 1 --2- 172.21.15.116:0/2240907061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88dc0039c0 0x7f88dc003de0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f88e4009ae0 tx=0x7f88e401ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22ff5b66499644ae server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:34.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.583+0000 7f88f1ffb640 1 -- 172.21.15.116:0/2240907061 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e40096b0 con 0x7f88dc0039c0 2024-04-25T00:15:34.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.583+0000 7f88f1ffb640 1 -- 172.21.15.116:0/2240907061 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88e40a64c0 con 0x7f88dc0039c0 2024-04-25T00:15:34.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.583+0000 7f88f1ffb640 1 -- 172.21.15.116:0/2240907061 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e40a66b0 con 0x7f88dc0039c0 2024-04-25T00:15:34.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.583+0000 7f88f9666640 1 -- 172.21.15.116:0/2240907061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88dc0039c0 msgr2=0x7f88dc003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:34.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.583+0000 7f88f9666640 1 --2- 172.21.15.116:0/2240907061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88dc0039c0 0x7f88dc003de0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f88e4009ae0 tx=0x7f88e401ca50 comp rx=0 tx=0).stop 2024-04-25T00:15:34.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.583+0000 7f88f37fe640 1 -- 172.21.15.116:0/2240907061 reap_dead start 2024-04-25T00:15:34.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.583+0000 7f88f9666640 1 -- 172.21.15.116:0/2240907061 shutdown_connections 2024-04-25T00:15:34.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.583+0000 7f88f9666640 1 -- 172.21.15.116:0/2240907061 >> 172.21.15.116:0/2240907061 conn(0x7f88f406f0c0 msgr2=0x7f88f406f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:34.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.583+0000 7f88f9666640 1 -- 172.21.15.116:0/2240907061 shutdown_connections 2024-04-25T00:15:34.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.583+0000 7f88f9666640 1 -- 172.21.15.116:0/2240907061 wait complete. 2024-04-25T00:15:34.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.583+0000 7f88f9666640 1 Processor -- start 2024-04-25T00:15:34.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.583+0000 7f88f9666640 1 -- start start 2024-04-25T00:15:34.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.583+0000 7f88f9666640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88dc0039c0 0x7f88f41ecd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:34.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.583+0000 7f88f9666640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88f4156ef0 0x7f88f414e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:34.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.583+0000 7f88f9666640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88f414ec80 0x7f88f414da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:34.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.583+0000 7f88f9666640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f88f4151120 con 0x7f88f4156ef0 2024-04-25T00:15:34.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.583+0000 7f88f9666640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f88f406ff30 con 0x7f88dc0039c0 2024-04-25T00:15:34.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.583+0000 7f88f9666640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f88f41512a0 con 0x7f88f414ec80 2024-04-25T00:15:34.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.587+0000 7f88f27fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88f4156ef0 0x7f88f414e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:34.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.587+0000 7f88f27fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88f4156ef0 0x7f88f414e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33420/0 (socket says 172.21.15.116:33420) 2024-04-25T00:15:34.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.587+0000 7f88f27fc640 1 -- 172.21.15.116:0/1077929730 learned_addr learned my addr 172.21.15.116:0/1077929730 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:15:34.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.587+0000 7f88f2ffd640 1 --2- 172.21.15.116:0/1077929730 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88dc0039c0 0x7f88f41ecd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:34.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.587+0000 7f88f37fe640 1 --2- 172.21.15.116:0/1077929730 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88f414ec80 0x7f88f414da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:34.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.587+0000 7f88f27fc640 1 -- 172.21.15.116:0/1077929730 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88f414ec80 msgr2=0x7f88f414da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:34.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.587+0000 7f88f27fc640 1 --2- 172.21.15.116:0/1077929730 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88f414ec80 0x7f88f414da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:34.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.587+0000 7f88f27fc640 1 -- 172.21.15.116:0/1077929730 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88dc0039c0 msgr2=0x7f88f41ecd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:34.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.587+0000 7f88f27fc640 1 --2- 172.21.15.116:0/1077929730 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88dc0039c0 0x7f88f41ecd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:34.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.587+0000 7f88f27fc640 1 -- 172.21.15.116:0/1077929730 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88f41490e0 con 0x7f88f4156ef0 2024-04-25T00:15:34.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.587+0000 7f88f2ffd640 1 --2- 172.21.15.116:0/1077929730 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88dc0039c0 0x7f88f41ecd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:15:34.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.587+0000 7f88f27fc640 1 --2- 172.21.15.116:0/1077929730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88f4156ef0 0x7f88f414e740 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f88e800e7f0 tx=0x7f88e800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:34.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.587+0000 7f88db7fe640 1 -- 172.21.15.116:0/1077929730 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e800c8d0 con 0x7f88f4156ef0 2024-04-25T00:15:34.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.587+0000 7f88db7fe640 1 -- 172.21.15.116:0/1077929730 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88e8018070 con 0x7f88f4156ef0 2024-04-25T00:15:34.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.587+0000 7f88f9666640 1 -- 172.21.15.116:0/1077929730 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88f4154b50 con 0x7f88f4156ef0 2024-04-25T00:15:34.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.587+0000 7f88db7fe640 1 -- 172.21.15.116:0/1077929730 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e800ccb0 con 0x7f88f4156ef0 2024-04-25T00:15:34.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.587+0000 7f88f9666640 1 -- 172.21.15.116:0/1077929730 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88f41ec270 con 0x7f88f4156ef0 2024-04-25T00:15:34.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.587+0000 7f88f9666640 1 -- 172.21.15.116:0/1077929730 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88f4068a60 con 0x7f88f4156ef0 2024-04-25T00:15:34.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.587+0000 7f88db7fe640 1 -- 172.21.15.116:0/1077929730 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f88e801d460 con 0x7f88f4156ef0 2024-04-25T00:15:34.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.587+0000 7f88db7fe640 1 --2- 172.21.15.116:0/1077929730 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f88e00678c0 0x7f88e0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:34.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.587+0000 7f88db7fe640 1 -- 172.21.15.116:0/1077929730 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 8386+0+0 (secure 0 0 0) 0x7f88e8086600 con 0x7f88f4156ef0 2024-04-25T00:15:34.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.587+0000 7f88f2ffd640 1 --2- 172.21.15.116:0/1077929730 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f88e00678c0 0x7f88e0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:34.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.587+0000 7f88f2ffd640 1 --2- 172.21.15.116:0/1077929730 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f88e00678c0 0x7f88e0069d80 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f88e40af000 tx=0x7f88e40046d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:34.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.591+0000 7f88db7fe640 1 -- 172.21.15.116:0/1077929730 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f88e8056700 con 0x7f88f4156ef0 2024-04-25T00:15:34.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.791+0000 7f88f9666640 1 -- 172.21.15.116:0/1077929730 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f88f41506e0 con 0x7f88f4156ef0 2024-04-25T00:15:34.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.791+0000 7f88db7fe640 1 -- 172.21.15.116:0/1077929730 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) ==== 95+0+6509 (secure 0 0 0) 0x7f88e8056000 con 0x7f88f4156ef0 2024-04-25T00:15:34.794 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:15:34.794 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":112,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6810,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/2340581953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2340581953},{"type":"v1","addr":"172.21.15.116:6833","nonce":2340581953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:15:01.877414+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5273":{"gid":5273,"name":"d","rank":0,"incarnation":102,"state":"up:active","state_seq":49,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5273,"qdb_cluster":[5273]},"id":1},{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:15:28.737458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6828":{"gid":6828,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":8,"addr":"172.21.15.116:6835/2012470875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2012470875},{"type":"v1","addr":"172.21.15.116:6835","nonce":2012470875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6828,"qdb_cluster":[6828]},"id":2}]} 2024-04-25T00:15:34.794 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 112 2024-04-25T00:15:34.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.795+0000 7f88f9666640 1 -- 172.21.15.116:0/1077929730 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f88e00678c0 msgr2=0x7f88e0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:34.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.795+0000 7f88f9666640 1 --2- 172.21.15.116:0/1077929730 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f88e00678c0 0x7f88e0069d80 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f88e40af000 tx=0x7f88e40046d0 comp rx=0 tx=0).stop 2024-04-25T00:15:34.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.795+0000 7f88f9666640 1 -- 172.21.15.116:0/1077929730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88f4156ef0 msgr2=0x7f88f414e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:34.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.795+0000 7f88f9666640 1 --2- 172.21.15.116:0/1077929730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88f4156ef0 0x7f88f414e740 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f88e800e7f0 tx=0x7f88e800ecb0 comp rx=0 tx=0).stop 2024-04-25T00:15:34.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.795+0000 7f88f9666640 1 -- 172.21.15.116:0/1077929730 shutdown_connections 2024-04-25T00:15:34.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.795+0000 7f88f9666640 1 --2- 172.21.15.116:0/1077929730 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f88e00678c0 0x7f88e0069d80 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:34.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.795+0000 7f88f9666640 1 --2- 172.21.15.116:0/1077929730 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f88f414ec80 0x7f88f414da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:34.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.795+0000 7f88f9666640 1 --2- 172.21.15.116:0/1077929730 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f88f4156ef0 0x7f88f414e740 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:34.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.795+0000 7f88f9666640 1 --2- 172.21.15.116:0/1077929730 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f88dc0039c0 0x7f88f41ecd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:34.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.795+0000 7f88f9666640 1 -- 172.21.15.116:0/1077929730 >> 172.21.15.116:0/1077929730 conn(0x7f88f406f0c0 msgr2=0x7f88f40c5120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:34.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.795+0000 7f88f9666640 1 -- 172.21.15.116:0/1077929730 shutdown_connections 2024-04-25T00:15:34.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:34.795+0000 7f88f9666640 1 -- 172.21.15.116:0/1077929730 wait complete. 2024-04-25T00:15:34.813 INFO:tasks.mds_thrash.fs.[b]:mds.b reported in up:standby state 2024-04-25T00:15:34.813 INFO:tasks.mds_thrash.fs.[b]:waiting for 47 secs before thrashing 2024-04-25T00:15:35.574 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:15:35.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.703+0000 7f401c517640 1 Processor -- start 2024-04-25T00:15:35.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.703+0000 7f401c517640 1 -- start start 2024-04-25T00:15:35.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.707+0000 7f401c517640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f401414c720 0x7f40141563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:35.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.707+0000 7f401c517640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f40140a94a0 0x7f40140c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:35.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.707+0000 7f401c517640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4014071220 con 0x7f40140c2f50 2024-04-25T00:15:35.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.707+0000 7f401c517640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f4014153460 con 0x7f40140a94a0 2024-04-25T00:15:35.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.707+0000 7f401c517640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f4014152420 con 0x7f401414c720 2024-04-25T00:15:35.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.707+0000 7f401a2b3640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f401414c720 0x7f40141563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:35.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.707+0000 7f4019ab2640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f40140a94a0 0x7f40140c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:35.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.707+0000 7f401aab4640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f40140c2f50 0x7f4014071680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:49162/0 (socket says 172.21.15.116:49162) 2024-04-25T00:15:35.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.707+0000 7f401aab4640 1 -- 172.21.15.116:0/2678189601 learned_addr learned my addr 172.21.15.116:0/2678189601 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:15:35.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.707+0000 7f401a2b3640 1 -- 172.21.15.116:0/2678189601 >> v2:172.21.15.174:3300/0 conn(0x7f40140a94a0 msgr2=0x7f40140c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:35.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.707+0000 7f401a2b3640 1 --2- 172.21.15.116:0/2678189601 >> v2:172.21.15.174:3300/0 conn(0x7f40140a94a0 0x7f40140c2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.707+0000 7f401a2b3640 1 -- 172.21.15.116:0/2678189601 >> v1:172.21.15.174:6789/0 conn(0x7f40140c2f50 legacy=0x7f4014071680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:35.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.707+0000 7f401a2b3640 1 -- 172.21.15.116:0/2678189601 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f401414cb00 con 0x7f401414c720 2024-04-25T00:15:35.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.711+0000 7f401a2b3640 1 --2- 172.21.15.116:0/2678189601 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f401414c720 0x7f40141563d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4010009d90 tx=0x7f401001c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0962443fff89db1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:35.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.711+0000 7f40192b1640 1 -- 172.21.15.116:0/2678189601 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40100b2070 con 0x7f401414c720 2024-04-25T00:15:35.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.711+0000 7f40192b1640 1 -- 172.21.15.116:0/2678189601 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40100a5990 con 0x7f401414c720 2024-04-25T00:15:35.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.711+0000 7f40192b1640 1 -- 172.21.15.116:0/2678189601 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40100a5d60 con 0x7f401414c720 2024-04-25T00:15:35.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.711+0000 7f401c517640 1 -- 172.21.15.116:0/2678189601 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f401414c720 msgr2=0x7f40141563d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:35.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.711+0000 7f401c517640 1 --2- 172.21.15.116:0/2678189601 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f401414c720 0x7f40141563d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4010009d90 tx=0x7f401001c8b0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.711+0000 7f401c517640 1 -- 172.21.15.116:0/2678189601 shutdown_connections 2024-04-25T00:15:35.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.711+0000 7f401c517640 1 --2- 172.21.15.116:0/2678189601 >> v2:172.21.15.174:3300/0 conn(0x7f40140a94a0 0x7f40140c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.711+0000 7f401c517640 1 --2- 172.21.15.116:0/2678189601 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f401414c720 0x7f40141563d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.711+0000 7f401c517640 1 -- 172.21.15.116:0/2678189601 >> 172.21.15.116:0/2678189601 conn(0x7f40140c3b00 msgr2=0x7f40140c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:35.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.711+0000 7f401c517640 1 -- 172.21.15.116:0/2678189601 shutdown_connections 2024-04-25T00:15:35.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.711+0000 7f401c517640 1 -- 172.21.15.116:0/2678189601 wait complete. 2024-04-25T00:15:35.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.711+0000 7f401c517640 1 Processor -- start 2024-04-25T00:15:35.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.711+0000 7f401c517640 1 -- start start 2024-04-25T00:15:35.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.711+0000 7f401c517640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f40140a94a0 0x7f4014155260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:35.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.711+0000 7f401c517640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f40140c2f50 0x7f401414cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:35.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.715+0000 7f401c517640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f401414c720 0x7f40141ee7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:35.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.715+0000 7f401c517640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4014152f40 con 0x7f401414c720 2024-04-25T00:15:35.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.715+0000 7f401c517640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4014152c60 con 0x7f40140c2f50 2024-04-25T00:15:35.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.715+0000 7f401c517640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f401415f5c0 con 0x7f40140a94a0 2024-04-25T00:15:35.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.715+0000 7f401aab4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f401414c720 0x7f40141ee7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:35.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.715+0000 7f401aab4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f401414c720 0x7f40141ee7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60870/0 (socket says 172.21.15.116:60870) 2024-04-25T00:15:35.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.715+0000 7f401aab4640 1 -- 172.21.15.116:0/2145897089 learned_addr learned my addr 172.21.15.116:0/2145897089 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:15:35.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.715+0000 7f401a2b3640 1 --2- 172.21.15.116:0/2145897089 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f40140a94a0 0x7f4014155260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:35.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.715+0000 7f4019ab2640 1 --2- 172.21.15.116:0/2145897089 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f40140c2f50 0x7f401414cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:35.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.715+0000 7f401aab4640 1 -- 172.21.15.116:0/2145897089 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f40140a94a0 msgr2=0x7f4014155260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:35.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.715+0000 7f401aab4640 1 --2- 172.21.15.116:0/2145897089 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f40140a94a0 0x7f4014155260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.715+0000 7f401aab4640 1 -- 172.21.15.116:0/2145897089 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f40140c2f50 msgr2=0x7f401414cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:35.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.715+0000 7f401aab4640 1 --2- 172.21.15.116:0/2145897089 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f40140c2f50 0x7f401414cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.715+0000 7f401aab4640 1 -- 172.21.15.116:0/2145897089 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40141557a0 con 0x7f401414c720 2024-04-25T00:15:35.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.715+0000 7f401aab4640 1 --2- 172.21.15.116:0/2145897089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f401414c720 0x7f40141ee7c0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f400c00c430 tx=0x7f400c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:35.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.715+0000 7f400affd640 1 -- 172.21.15.116:0/2145897089 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f400c017070 con 0x7f401414c720 2024-04-25T00:15:35.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.715+0000 7f400affd640 1 -- 172.21.15.116:0/2145897089 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f400c005cf0 con 0x7f401414c720 2024-04-25T00:15:35.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.715+0000 7f401c517640 1 -- 172.21.15.116:0/2145897089 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4014145b90 con 0x7f401414c720 2024-04-25T00:15:35.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.715+0000 7f400affd640 1 -- 172.21.15.116:0/2145897089 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f400c0072d0 con 0x7f401414c720 2024-04-25T00:15:35.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.715+0000 7f401c517640 1 -- 172.21.15.116:0/2145897089 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4014153970 con 0x7f401414c720 2024-04-25T00:15:35.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.715+0000 7f401c517640 1 -- 172.21.15.116:0/2145897089 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4014068350 con 0x7f401414c720 2024-04-25T00:15:35.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.715+0000 7f400affd640 1 -- 172.21.15.116:0/2145897089 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f400c01a070 con 0x7f401414c720 2024-04-25T00:15:35.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.715+0000 7f400affd640 1 --2- 172.21.15.116:0/2145897089 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3ff80678f0 0x7f3ff8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:35.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.715+0000 7f400affd640 1 -- 172.21.15.116:0/2145897089 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 8386+0+0 (secure 0 0 0) 0x7f400c085960 con 0x7f401414c720 2024-04-25T00:15:35.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.715+0000 7f401a2b3640 1 --2- 172.21.15.116:0/2145897089 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3ff80678f0 0x7f3ff8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:35.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.719+0000 7f401a2b3640 1 --2- 172.21.15.116:0/2145897089 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3ff80678f0 0x7f3ff8069db0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f40100af000 tx=0x7f40100a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:35.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.719+0000 7f400affd640 1 -- 172.21.15.116:0/2145897089 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f400c055410 con 0x7f401414c720 2024-04-25T00:15:35.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.923+0000 7f401c517640 1 -- 172.21.15.116:0/2145897089 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f40140c3640 con 0x7f401414c720 2024-04-25T00:15:35.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.927+0000 7f400affd640 1 -- 172.21.15.116:0/2145897089 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) ==== 95+0+6509 (secure 0 0 0) 0x7f400c059710 con 0x7f401414c720 2024-04-25T00:15:35.928 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:15:35.928 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":112,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6810,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/2340581953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2340581953},{"type":"v1","addr":"172.21.15.116:6833","nonce":2340581953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:15:01.877414+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5273":{"gid":5273,"name":"d","rank":0,"incarnation":102,"state":"up:active","state_seq":49,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5273,"qdb_cluster":[5273]},"id":1},{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:15:28.737458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6828":{"gid":6828,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":8,"addr":"172.21.15.116:6835/2012470875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2012470875},{"type":"v1","addr":"172.21.15.116:6835","nonce":2012470875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6828,"qdb_cluster":[6828]},"id":2}]} 2024-04-25T00:15:35.928 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 112 2024-04-25T00:15:35.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.927+0000 7f401c517640 1 -- 172.21.15.116:0/2145897089 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3ff80678f0 msgr2=0x7f3ff8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:35.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.927+0000 7f401c517640 1 --2- 172.21.15.116:0/2145897089 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3ff80678f0 0x7f3ff8069db0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f40100af000 tx=0x7f40100a7070 comp rx=0 tx=0).stop 2024-04-25T00:15:35.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.927+0000 7f401c517640 1 -- 172.21.15.116:0/2145897089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f401414c720 msgr2=0x7f40141ee7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:35.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.927+0000 7f401c517640 1 --2- 172.21.15.116:0/2145897089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f401414c720 0x7f40141ee7c0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f400c00c430 tx=0x7f400c00c8f0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.927+0000 7f401c517640 1 -- 172.21.15.116:0/2145897089 shutdown_connections 2024-04-25T00:15:35.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.927+0000 7f401c517640 1 --2- 172.21.15.116:0/2145897089 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3ff80678f0 0x7f3ff8069db0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.927+0000 7f401c517640 1 --2- 172.21.15.116:0/2145897089 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f401414c720 0x7f40141ee7c0 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.927+0000 7f401c517640 1 --2- 172.21.15.116:0/2145897089 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f40140c2f50 0x7f401414cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.927+0000 7f401c517640 1 --2- 172.21.15.116:0/2145897089 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f40140a94a0 0x7f4014155260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:35.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.927+0000 7f401c517640 1 -- 172.21.15.116:0/2145897089 >> 172.21.15.116:0/2145897089 conn(0x7f40140c3b00 msgr2=0x7f40140c3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:35.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.927+0000 7f401c517640 1 -- 172.21.15.116:0/2145897089 shutdown_connections 2024-04-25T00:15:35.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:35.927+0000 7f401c517640 1 -- 172.21.15.116:0/2145897089 wait complete. 2024-04-25T00:15:35.947 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:15:37.948 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:15:38.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.079+0000 7fa2850ca640 1 Processor -- start 2024-04-25T00:15:38.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.079+0000 7fa2850ca640 1 -- start start 2024-04-25T00:15:38.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.079+0000 7fa2850ca640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa28006e890 0x7fa280071980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:38.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.079+0000 7fa2850ca640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa280152c10 con 0x7fa280070dd0 2024-04-25T00:15:38.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.079+0000 7fa2850ca640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa280151e50 con 0x7fa280148890 2024-04-25T00:15:38.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.079+0000 7fa2850ca640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa280152130 con 0x7fa28006e890 2024-04-25T00:15:38.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.079+0000 7fa27659c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fa280070dd0 0x7fa280071270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:45376/0 (socket says 172.21.15.116:45376) 2024-04-25T00:15:38.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.079+0000 7fa27659c640 1 -- 172.21.15.116:0/1159385689 learned_addr learned my addr 172.21.15.116:0/1159385689 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:15:38.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.079+0000 7fa27f59e640 1 --2- 172.21.15.116:0/1159385689 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa28006e890 0x7fa280071980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:38.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.079+0000 7fa27e59c640 1 -- 172.21.15.116:0/1159385689 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2459464249 0 0) 0x7fa280152c10 con 0x7fa280070dd0 2024-04-25T00:15:38.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.079+0000 7fa27e59c640 1 -- 172.21.15.116:0/1159385689 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa26c003530 con 0x7fa280070dd0 2024-04-25T00:15:38.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.079+0000 7fa27e59c640 1 -- 172.21.15.116:0/1159385689 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 442585980 0 0) 0x7fa26c003530 con 0x7fa280070dd0 2024-04-25T00:15:38.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.079+0000 7fa27e59c640 1 -- 172.21.15.116:0/1159385689 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa28006e890 msgr2=0x7fa280071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:38.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.079+0000 7fa27e59c640 1 --2- 172.21.15.116:0/1159385689 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa28006e890 0x7fa280071980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:38.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.079+0000 7fa27e59c640 1 -- 172.21.15.116:0/1159385689 >> v1:172.21.15.174:6789/0 conn(0x7fa280148890 legacy=0x7fa280072590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:38.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.079+0000 7fa27e59c640 1 -- 172.21.15.116:0/1159385689 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa280152410 con 0x7fa280070dd0 2024-04-25T00:15:38.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.079+0000 7fa27e59c640 1 -- 172.21.15.116:0/1159385689 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7fa268001680 con 0x7fa280070dd0 2024-04-25T00:15:38.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.079+0000 7fa27e59c640 1 -- 172.21.15.116:0/1159385689 >> v1:172.21.15.116:6789/0 conn(0x7fa280070dd0 legacy=0x7fa280071270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:38.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.079+0000 7fa27e59c640 1 --2- 172.21.15.116:0/1159385689 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa26c0039f0 0x7fa26c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:38.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.079+0000 7fa27e59c640 1 --2- 172.21.15.116:0/1159385689 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa28006e890 0x7fa26c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:38.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.079+0000 7fa27e59c640 1 --2- 172.21.15.116:0/1159385689 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa26c004d50 0x7fa26c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:38.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.079+0000 7fa27e59c640 1 -- 172.21.15.116:0/1159385689 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa280152130 con 0x7fa26c004d50 2024-04-25T00:15:38.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.079+0000 7fa27e59c640 1 -- 172.21.15.116:0/1159385689 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa26c00bb60 con 0x7fa26c0039f0 2024-04-25T00:15:38.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.079+0000 7fa27e59c640 1 -- 172.21.15.116:0/1159385689 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa26c00df40 con 0x7fa28006e890 2024-04-25T00:15:38.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.079+0000 7fa27f59e640 1 --2- 172.21.15.116:0/1159385689 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa26c004d50 0x7fa26c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:38.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.079+0000 7fa27ed9d640 1 --2- 172.21.15.116:0/1159385689 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa26c0039f0 0x7fa26c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:38.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.079+0000 7fa27659c640 1 --2- 172.21.15.116:0/1159385689 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa28006e890 0x7fa26c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:38.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.079+0000 7fa27f59e640 1 -- 172.21.15.116:0/1159385689 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa28006e890 msgr2=0x7fa26c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:38.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.079+0000 7fa27f59e640 1 --2- 172.21.15.116:0/1159385689 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa28006e890 0x7fa26c004810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:38.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.079+0000 7fa27f59e640 1 -- 172.21.15.116:0/1159385689 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa26c0039f0 msgr2=0x7fa26c003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:38.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.079+0000 7fa27f59e640 1 --2- 172.21.15.116:0/1159385689 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa26c0039f0 0x7fa26c003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:38.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.079+0000 7fa27f59e640 1 -- 172.21.15.116:0/1159385689 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa26c003530 con 0x7fa26c004d50 2024-04-25T00:15:38.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.083+0000 7fa27f59e640 1 --2- 172.21.15.116:0/1159385689 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa26c004d50 0x7fa26c0051d0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fa278007330 tx=0x7fa27801cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bdd8f336e542437 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:38.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.083+0000 7fa27e59c640 1 -- 172.21.15.116:0/1159385689 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2780b2070 con 0x7fa26c004d50 2024-04-25T00:15:38.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.083+0000 7fa27e59c640 1 -- 172.21.15.116:0/1159385689 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2780a5ce0 con 0x7fa26c004d50 2024-04-25T00:15:38.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.083+0000 7fa27e59c640 1 -- 172.21.15.116:0/1159385689 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa278009be0 con 0x7fa26c004d50 2024-04-25T00:15:38.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.083+0000 7fa2850ca640 1 -- 172.21.15.116:0/1159385689 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa26c004d50 msgr2=0x7fa26c0051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:38.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.083+0000 7fa2850ca640 1 --2- 172.21.15.116:0/1159385689 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa26c004d50 0x7fa26c0051d0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fa278007330 tx=0x7fa27801cde0 comp rx=0 tx=0).stop 2024-04-25T00:15:38.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.083+0000 7fa27f59e640 1 -- 172.21.15.116:0/1159385689 reap_dead start 2024-04-25T00:15:38.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.083+0000 7fa2850ca640 1 -- 172.21.15.116:0/1159385689 shutdown_connections 2024-04-25T00:15:38.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.083+0000 7fa2850ca640 1 -- 172.21.15.116:0/1159385689 >> 172.21.15.116:0/1159385689 conn(0x7fa2800c3b00 msgr2=0x7fa2800c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:38.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.083+0000 7fa2850ca640 1 -- 172.21.15.116:0/1159385689 shutdown_connections 2024-04-25T00:15:38.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.083+0000 7fa2850ca640 1 -- 172.21.15.116:0/1159385689 wait complete. 2024-04-25T00:15:38.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.083+0000 7fa2850ca640 1 Processor -- start 2024-04-25T00:15:38.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.083+0000 7fa2850ca640 1 -- start start 2024-04-25T00:15:38.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.083+0000 7fa2850ca640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa26c004d50 0x7fa2801459f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:38.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.083+0000 7fa2850ca640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa280155ad0 0x7fa280153130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:38.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.083+0000 7fa2850ca640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa280155f50 0x7fa280153670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:38.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.083+0000 7fa2850ca640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa280072ca0 con 0x7fa280155ad0 2024-04-25T00:15:38.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.083+0000 7fa2850ca640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa28006ed70 con 0x7fa280155f50 2024-04-25T00:15:38.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.083+0000 7fa2850ca640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa280072e20 con 0x7fa26c004d50 2024-04-25T00:15:38.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.083+0000 7fa27659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa280155ad0 0x7fa280153130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:38.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.083+0000 7fa27659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa280155ad0 0x7fa280153130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60876/0 (socket says 172.21.15.116:60876) 2024-04-25T00:15:38.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.083+0000 7fa27659c640 1 -- 172.21.15.116:0/1612922914 learned_addr learned my addr 172.21.15.116:0/1612922914 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:15:38.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.083+0000 7fa27ed9d640 1 --2- 172.21.15.116:0/1612922914 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa26c004d50 0x7fa2801459f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:38.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.083+0000 7fa27f59e640 1 --2- 172.21.15.116:0/1612922914 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa280155f50 0x7fa280153670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:38.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.083+0000 7fa27659c640 1 -- 172.21.15.116:0/1612922914 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa26c004d50 msgr2=0x7fa2801459f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:38.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.083+0000 7fa27659c640 1 --2- 172.21.15.116:0/1612922914 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa26c004d50 0x7fa2801459f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:38.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.083+0000 7fa27659c640 1 -- 172.21.15.116:0/1612922914 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa280155f50 msgr2=0x7fa280153670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:38.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.083+0000 7fa27659c640 1 --2- 172.21.15.116:0/1612922914 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa280155f50 0x7fa280153670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:38.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.083+0000 7fa27659c640 1 -- 172.21.15.116:0/1612922914 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa28014f360 con 0x7fa280155ad0 2024-04-25T00:15:38.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.083+0000 7fa27659c640 1 --2- 172.21.15.116:0/1612922914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa280155ad0 0x7fa280153130 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fa26800d7e0 tx=0x7fa26800dca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:38.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.083+0000 7fa277fff640 1 -- 172.21.15.116:0/1612922914 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa268019070 con 0x7fa280155ad0 2024-04-25T00:15:38.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.087+0000 7fa277fff640 1 -- 172.21.15.116:0/1612922914 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2680074e0 con 0x7fa280155ad0 2024-04-25T00:15:38.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.087+0000 7fa2850ca640 1 -- 172.21.15.116:0/1612922914 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2801ea8a0 con 0x7fa280155ad0 2024-04-25T00:15:38.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.087+0000 7fa277fff640 1 -- 172.21.15.116:0/1612922914 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2680049f0 con 0x7fa280155ad0 2024-04-25T00:15:38.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.087+0000 7fa2850ca640 1 -- 172.21.15.116:0/1612922914 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa28014efe0 con 0x7fa280155ad0 2024-04-25T00:15:38.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.087+0000 7fa2850ca640 1 -- 172.21.15.116:0/1612922914 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa280068350 con 0x7fa280155ad0 2024-04-25T00:15:38.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.087+0000 7fa277fff640 1 -- 172.21.15.116:0/1612922914 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa268004b90 con 0x7fa280155ad0 2024-04-25T00:15:38.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.087+0000 7fa277fff640 1 --2- 172.21.15.116:0/1612922914 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa2600678f0 0x7fa260069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:38.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.087+0000 7fa277fff640 1 -- 172.21.15.116:0/1612922914 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 8386+0+0 (secure 0 0 0) 0x7fa268086090 con 0x7fa280155ad0 2024-04-25T00:15:38.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.087+0000 7fa27ed9d640 1 --2- 172.21.15.116:0/1612922914 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa2600678f0 0x7fa260069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:38.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.087+0000 7fa27ed9d640 1 --2- 172.21.15.116:0/1612922914 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa2600678f0 0x7fa260069db0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fa270005510 tx=0x7fa270009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:38.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.091+0000 7fa277fff640 1 -- 172.21.15.116:0/1612922914 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fa268056230 con 0x7fa280155ad0 2024-04-25T00:15:38.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.287+0000 7fa2850ca640 1 -- 172.21.15.116:0/1612922914 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa28006e670 con 0x7fa280155ad0 2024-04-25T00:15:38.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.287+0000 7fa277fff640 1 -- 172.21.15.116:0/1612922914 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) ==== 95+0+6509 (secure 0 0 0) 0x7fa268055b30 con 0x7fa280155ad0 2024-04-25T00:15:38.290 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:15:38.290 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":112,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6810,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/2340581953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2340581953},{"type":"v1","addr":"172.21.15.116:6833","nonce":2340581953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":106}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:15:01.877414+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5273":{"gid":5273,"name":"d","rank":0,"incarnation":102,"state":"up:active","state_seq":49,"addr":"172.21.15.174:6835/4215220222","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4215220222},{"type":"v1","addr":"172.21.15.174:6835","nonce":4215220222}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5273,"qdb_cluster":[5273]},"id":1},{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:15:28.737458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6828":{"gid":6828,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":8,"addr":"172.21.15.116:6835/2012470875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2012470875},{"type":"v1","addr":"172.21.15.116:6835","nonce":2012470875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6828,"qdb_cluster":[6828]},"id":2}]} 2024-04-25T00:15:38.290 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 112 2024-04-25T00:15:38.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.291+0000 7fa2850ca640 1 -- 172.21.15.116:0/1612922914 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa2600678f0 msgr2=0x7fa260069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:38.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.291+0000 7fa2850ca640 1 --2- 172.21.15.116:0/1612922914 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa2600678f0 0x7fa260069db0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fa270005510 tx=0x7fa270009040 comp rx=0 tx=0).stop 2024-04-25T00:15:38.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.291+0000 7fa2850ca640 1 -- 172.21.15.116:0/1612922914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa280155ad0 msgr2=0x7fa280153130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:38.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.291+0000 7fa2850ca640 1 --2- 172.21.15.116:0/1612922914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa280155ad0 0x7fa280153130 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fa26800d7e0 tx=0x7fa26800dca0 comp rx=0 tx=0).stop 2024-04-25T00:15:38.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.291+0000 7fa2850ca640 1 -- 172.21.15.116:0/1612922914 shutdown_connections 2024-04-25T00:15:38.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.291+0000 7fa2850ca640 1 --2- 172.21.15.116:0/1612922914 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa2600678f0 0x7fa260069db0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:38.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.291+0000 7fa2850ca640 1 --2- 172.21.15.116:0/1612922914 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa280155f50 0x7fa280153670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:38.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.291+0000 7fa2850ca640 1 --2- 172.21.15.116:0/1612922914 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa280155ad0 0x7fa280153130 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:38.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.291+0000 7fa2850ca640 1 --2- 172.21.15.116:0/1612922914 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa26c004d50 0x7fa2801459f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:38.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.291+0000 7fa2850ca640 1 -- 172.21.15.116:0/1612922914 >> 172.21.15.116:0/1612922914 conn(0x7fa2800c3b00 msgr2=0x7fa280150830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:38.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.291+0000 7fa2850ca640 1 -- 172.21.15.116:0/1612922914 shutdown_connections 2024-04-25T00:15:38.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:38.291+0000 7fa2850ca640 1 -- 172.21.15.116:0/1612922914 wait complete. 2024-04-25T00:15:38.309 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:15:38.368 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:15:38.371 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:15:40.310 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:15:40.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.443+0000 7fd27594f640 1 Processor -- start 2024-04-25T00:15:40.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.443+0000 7fd27594f640 1 -- start start 2024-04-25T00:15:40.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.443+0000 7fd27594f640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fd270071d90 0x7fd2701563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:40.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.443+0000 7fd27594f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd270071220 con 0x7fd27014c720 2024-04-25T00:15:40.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.443+0000 7fd27594f640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd270152420 con 0x7fd2700a94a0 2024-04-25T00:15:40.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.443+0000 7fd27594f640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fd270153460 con 0x7fd270071d90 2024-04-25T00:15:40.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.443+0000 7fd26effd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fd27014c720 0x7fd2700c2a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:45378/0 (socket says 172.21.15.116:45378) 2024-04-25T00:15:40.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.443+0000 7fd26effd640 1 -- 172.21.15.116:0/3648266790 learned_addr learned my addr 172.21.15.116:0/3648266790 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:15:40.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.443+0000 7fd26f7fe640 1 --2- 172.21.15.116:0/3648266790 >> v2:172.21.15.174:3300/0 conn(0x7fd270071d90 0x7fd2701563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:40.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.443+0000 7fd26dffb640 1 -- 172.21.15.116:0/3648266790 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1286892638 0 0) 0x7fd270071220 con 0x7fd27014c720 2024-04-25T00:15:40.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.443+0000 7fd26dffb640 1 -- 172.21.15.116:0/3648266790 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd258005740 con 0x7fd27014c720 2024-04-25T00:15:40.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.443+0000 7fd26dffb640 1 -- 172.21.15.116:0/3648266790 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1573434077 0 0) 0x7fd258005740 con 0x7fd27014c720 2024-04-25T00:15:40.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.443+0000 7fd26dffb640 1 -- 172.21.15.116:0/3648266790 >> v2:172.21.15.174:3300/0 conn(0x7fd270071d90 msgr2=0x7fd2701563d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:40.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.443+0000 7fd26dffb640 1 --2- 172.21.15.116:0/3648266790 >> v2:172.21.15.174:3300/0 conn(0x7fd270071d90 0x7fd2701563d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:40.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.443+0000 7fd26dffb640 1 -- 172.21.15.116:0/3648266790 >> v1:172.21.15.174:6789/0 conn(0x7fd2700a94a0 legacy=0x7fd270071680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:40.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.443+0000 7fd26dffb640 1 -- 172.21.15.116:0/3648266790 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2700721c0 con 0x7fd27014c720 2024-04-25T00:15:40.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.443+0000 7fd26dffb640 1 -- 172.21.15.116:0/3648266790 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7fd260001680 con 0x7fd27014c720 2024-04-25T00:15:40.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.443+0000 7fd26dffb640 1 -- 172.21.15.116:0/3648266790 >> v1:172.21.15.116:6789/0 conn(0x7fd27014c720 legacy=0x7fd2700c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:40.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.443+0000 7fd26dffb640 1 --2- 172.21.15.116:0/3648266790 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd258005bd0 0x7fd258005ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:40.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.443+0000 7fd26dffb640 1 --2- 172.21.15.116:0/3648266790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd258006a20 0x7fd258006ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:40.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.443+0000 7fd26dffb640 1 --2- 172.21.15.116:0/3648266790 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd2580073e0 0x7fd25800dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:40.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.443+0000 7fd26dffb640 1 -- 172.21.15.116:0/3648266790 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd270153460 con 0x7fd258006a20 2024-04-25T00:15:40.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.443+0000 7fd26dffb640 1 -- 172.21.15.116:0/3648266790 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd25800e380 con 0x7fd2580073e0 2024-04-25T00:15:40.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.443+0000 7fd26dffb640 1 -- 172.21.15.116:0/3648266790 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fd25800e580 con 0x7fd258005bd0 2024-04-25T00:15:40.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.443+0000 7fd26e7fc640 1 --2- 172.21.15.116:0/3648266790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd258006a20 0x7fd258006ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:40.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.443+0000 7fd26f7fe640 1 --2- 172.21.15.116:0/3648266790 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd2580073e0 0x7fd25800dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:40.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.443+0000 7fd26effd640 1 --2- 172.21.15.116:0/3648266790 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd258005bd0 0x7fd258005ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:40.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.443+0000 7fd26e7fc640 1 -- 172.21.15.116:0/3648266790 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd258005bd0 msgr2=0x7fd258005ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:40.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.443+0000 7fd26e7fc640 1 --2- 172.21.15.116:0/3648266790 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd258005bd0 0x7fd258005ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:40.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.443+0000 7fd26e7fc640 1 -- 172.21.15.116:0/3648266790 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd2580073e0 msgr2=0x7fd25800dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:40.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.443+0000 7fd26e7fc640 1 --2- 172.21.15.116:0/3648266790 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd2580073e0 0x7fd25800dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:40.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.443+0000 7fd26e7fc640 1 -- 172.21.15.116:0/3648266790 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd258005740 con 0x7fd258006a20 2024-04-25T00:15:40.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.443+0000 7fd26f7fe640 1 -- 172.21.15.116:0/3648266790 reap_dead start 2024-04-25T00:15:40.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.447+0000 7fd26e7fc640 1 --2- 172.21.15.116:0/3648266790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd258006a20 0x7fd258006ea0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fd264001780 tx=0x7fd26401c220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=621f30ea530f5a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:40.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.447+0000 7fd26dffb640 1 -- 172.21.15.116:0/3648266790 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2640013a0 con 0x7fd258006a20 2024-04-25T00:15:40.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.447+0000 7fd26dffb640 1 -- 172.21.15.116:0/3648266790 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2640052f0 con 0x7fd258006a20 2024-04-25T00:15:40.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.447+0000 7fd26dffb640 1 -- 172.21.15.116:0/3648266790 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd264003840 con 0x7fd258006a20 2024-04-25T00:15:40.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.447+0000 7fd27594f640 1 -- 172.21.15.116:0/3648266790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd258006a20 msgr2=0x7fd258006ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:40.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.447+0000 7fd27594f640 1 --2- 172.21.15.116:0/3648266790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd258006a20 0x7fd258006ea0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fd264001780 tx=0x7fd26401c220 comp rx=0 tx=0).stop 2024-04-25T00:15:40.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.447+0000 7fd27594f640 1 -- 172.21.15.116:0/3648266790 shutdown_connections 2024-04-25T00:15:40.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.447+0000 7fd27594f640 1 --2- 172.21.15.116:0/3648266790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd258006a20 0x7fd258006ea0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:40.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.447+0000 7fd27594f640 1 -- 172.21.15.116:0/3648266790 >> 172.21.15.116:0/3648266790 conn(0x7fd2700c3b00 msgr2=0x7fd2700c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:40.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.447+0000 7fd27594f640 1 -- 172.21.15.116:0/3648266790 shutdown_connections 2024-04-25T00:15:40.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.447+0000 7fd27594f640 1 -- 172.21.15.116:0/3648266790 wait complete. 2024-04-25T00:15:40.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.447+0000 7fd27594f640 1 Processor -- start 2024-04-25T00:15:40.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.447+0000 7fd27594f640 1 -- start start 2024-04-25T00:15:40.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.447+0000 7fd27594f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd258006a20 0x7fd2700ab910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:40.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.447+0000 7fd27594f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2701557b0 0x7fd2701ddaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:40.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.447+0000 7fd27594f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd270140720 0x7fd2700abee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:40.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.447+0000 7fd27594f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd270152980 con 0x7fd2701557b0 2024-04-25T00:15:40.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.447+0000 7fd27594f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd2700c3220 con 0x7fd270140720 2024-04-25T00:15:40.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.447+0000 7fd27594f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fd270152c60 con 0x7fd258006a20 2024-04-25T00:15:40.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.447+0000 7fd26e7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2701557b0 0x7fd2701ddaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:40.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.447+0000 7fd26e7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2701557b0 0x7fd2701ddaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60892/0 (socket says 172.21.15.116:60892) 2024-04-25T00:15:40.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.447+0000 7fd26e7fc640 1 -- 172.21.15.116:0/95721209 learned_addr learned my addr 172.21.15.116:0/95721209 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:15:40.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.447+0000 7fd26effd640 1 --2- 172.21.15.116:0/95721209 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd258006a20 0x7fd2700ab910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:40.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.447+0000 7fd26f7fe640 1 --2- 172.21.15.116:0/95721209 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd270140720 0x7fd2700abee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:40.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.451+0000 7fd26e7fc640 1 -- 172.21.15.116:0/95721209 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd258006a20 msgr2=0x7fd2700ab910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:40.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.451+0000 7fd26e7fc640 1 --2- 172.21.15.116:0/95721209 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd258006a20 0x7fd2700ab910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:40.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.451+0000 7fd26e7fc640 1 -- 172.21.15.116:0/95721209 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd270140720 msgr2=0x7fd2700abee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:40.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.451+0000 7fd26e7fc640 1 --2- 172.21.15.116:0/95721209 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd270140720 0x7fd2700abee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:40.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.451+0000 7fd26e7fc640 1 -- 172.21.15.116:0/95721209 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2700ab010 con 0x7fd2701557b0 2024-04-25T00:15:40.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.451+0000 7fd26e7fc640 1 --2- 172.21.15.116:0/95721209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2701557b0 0x7fd2701ddaa0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fd264005040 tx=0x7fd258007a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:40.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.451+0000 7fd257fff640 1 -- 172.21.15.116:0/95721209 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2640ba020 con 0x7fd2701557b0 2024-04-25T00:15:40.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.451+0000 7fd257fff640 1 -- 172.21.15.116:0/95721209 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2640054a0 con 0x7fd2701557b0 2024-04-25T00:15:40.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.451+0000 7fd27594f640 1 -- 172.21.15.116:0/95721209 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2701eaa80 con 0x7fd2701557b0 2024-04-25T00:15:40.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.451+0000 7fd257fff640 1 -- 172.21.15.116:0/95721209 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd264007a40 con 0x7fd2701557b0 2024-04-25T00:15:40.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.451+0000 7fd27594f640 1 -- 172.21.15.116:0/95721209 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2701ddfe0 con 0x7fd2701557b0 2024-04-25T00:15:40.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.451+0000 7fd27594f640 1 -- 172.21.15.116:0/95721209 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd270068350 con 0x7fd2701557b0 2024-04-25T00:15:40.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.451+0000 7fd257fff640 1 -- 172.21.15.116:0/95721209 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd2640bd020 con 0x7fd2701557b0 2024-04-25T00:15:40.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.451+0000 7fd257fff640 1 --2- 172.21.15.116:0/95721209 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd25c0678f0 0x7fd25c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:40.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.451+0000 7fd257fff640 1 -- 172.21.15.116:0/95721209 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 8386+0+0 (secure 0 0 0) 0x7fd264120910 con 0x7fd2701557b0 2024-04-25T00:15:40.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.451+0000 7fd26effd640 1 --2- 172.21.15.116:0/95721209 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd25c0678f0 0x7fd25c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:40.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.451+0000 7fd26effd640 1 --2- 172.21.15.116:0/95721209 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd25c0678f0 0x7fd25c069db0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fd2600017f0 tx=0x7fd2600014a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:40.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.455+0000 7fd257fff640 1 -- 172.21.15.116:0/95721209 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fd2640c4030 con 0x7fd2701557b0 2024-04-25T00:15:40.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.683+0000 7fd27594f640 1 -- 172.21.15.116:0/95721209 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd2700c3640 con 0x7fd2701557b0 2024-04-25T00:15:40.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.683+0000 7fd257fff640 1 -- 172.21.15.116:0/95721209 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) ==== 95+0+5672 (secure 0 0 0) 0x7fd270152420 con 0x7fd2701557b0 2024-04-25T00:15:40.681 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:15:40.681 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":113,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:15:40.572986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6810":{"gid":6810,"name":"c","rank":0,"incarnation":113,"state":"up:replay","state_seq":1,"addr":"172.21.15.116:6833/2340581953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2340581953},{"type":"v1","addr":"172.21.15.116:6833","nonce":2340581953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:15:28.737458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6828":{"gid":6828,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":8,"addr":"172.21.15.116:6835/2012470875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2012470875},{"type":"v1","addr":"172.21.15.116:6835","nonce":2012470875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6828,"qdb_cluster":[6828]},"id":2}]} 2024-04-25T00:15:40.682 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 113 2024-04-25T00:15:40.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.687+0000 7fd27594f640 1 -- 172.21.15.116:0/95721209 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd25c0678f0 msgr2=0x7fd25c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:40.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.687+0000 7fd27594f640 1 --2- 172.21.15.116:0/95721209 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd25c0678f0 0x7fd25c069db0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fd2600017f0 tx=0x7fd2600014a0 comp rx=0 tx=0).stop 2024-04-25T00:15:40.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.687+0000 7fd27594f640 1 -- 172.21.15.116:0/95721209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2701557b0 msgr2=0x7fd2701ddaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:40.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.687+0000 7fd27594f640 1 --2- 172.21.15.116:0/95721209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2701557b0 0x7fd2701ddaa0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fd264005040 tx=0x7fd258007a50 comp rx=0 tx=0).stop 2024-04-25T00:15:40.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.687+0000 7fd27594f640 1 -- 172.21.15.116:0/95721209 shutdown_connections 2024-04-25T00:15:40.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.687+0000 7fd27594f640 1 --2- 172.21.15.116:0/95721209 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd25c0678f0 0x7fd25c069db0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:40.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.687+0000 7fd27594f640 1 --2- 172.21.15.116:0/95721209 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd270140720 0x7fd2700abee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:40.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.687+0000 7fd27594f640 1 --2- 172.21.15.116:0/95721209 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2701557b0 0x7fd2701ddaa0 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:40.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.687+0000 7fd27594f640 1 --2- 172.21.15.116:0/95721209 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd258006a20 0x7fd2700ab910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:40.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.687+0000 7fd27594f640 1 -- 172.21.15.116:0/95721209 >> 172.21.15.116:0/95721209 conn(0x7fd2700c3b00 msgr2=0x7fd270145a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:40.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.687+0000 7fd27594f640 1 -- 172.21.15.116:0/95721209 shutdown_connections 2024-04-25T00:15:40.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.687+0000 7fd27594f640 1 -- 172.21.15.116:0/95721209 wait complete. 2024-04-25T00:15:40.705 INFO:tasks.mds_thrash.fs.[a]:mds.d down, removed from mdsmap 2024-04-25T00:15:40.706 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-04-25T00:15:40.706 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:15:40.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.827+0000 7fcf92ca7640 1 Processor -- start 2024-04-25T00:15:40.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.827+0000 7fcf92ca7640 1 -- start start 2024-04-25T00:15:40.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.827+0000 7fcf92ca7640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fcf8c150d10 0x7fcf8c0c5010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:40.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.827+0000 7fcf92ca7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcf8c072ca0 0x7fcf8c154190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:40.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.827+0000 7fcf92ca7640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf8c0c4950 con 0x7fcf8c0c5550 2024-04-25T00:15:40.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.827+0000 7fcf92ca7640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fcf8c1546d0 con 0x7fcf8c072ca0 2024-04-25T00:15:40.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.827+0000 7fcf92ca7640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fcf8c1510f0 con 0x7fcf8c150d10 2024-04-25T00:15:40.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.827+0000 7fcf91244640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcf8c072ca0 0x7fcf8c154190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:40.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.827+0000 7fcf8bfff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fcf8c0c5550 0x7fcf8c153a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:45384/0 (socket says 172.21.15.116:45384) 2024-04-25T00:15:40.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.827+0000 7fcf8bfff640 1 -- 172.21.15.116:0/3986223594 learned_addr learned my addr 172.21.15.116:0/3986223594 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:15:40.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.827+0000 7fcf90a43640 1 --2- 172.21.15.116:0/3986223594 >> v2:172.21.15.174:3300/0 conn(0x7fcf8c150d10 0x7fcf8c0c5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:40.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.827+0000 7fcf91244640 1 -- 172.21.15.116:0/3986223594 >> v2:172.21.15.174:3300/0 conn(0x7fcf8c150d10 msgr2=0x7fcf8c0c5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:40.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.827+0000 7fcf91244640 1 --2- 172.21.15.116:0/3986223594 >> v2:172.21.15.174:3300/0 conn(0x7fcf8c150d10 0x7fcf8c0c5010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:40.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.827+0000 7fcf91244640 1 -- 172.21.15.116:0/3986223594 >> v1:172.21.15.116:6789/0 conn(0x7fcf8c0c5550 legacy=0x7fcf8c153a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:40.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.827+0000 7fcf91244640 1 -- 172.21.15.116:0/3986223594 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf8c157a20 con 0x7fcf8c072ca0 2024-04-25T00:15:40.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.831+0000 7fcf91244640 1 --2- 172.21.15.116:0/3986223594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf8c072ca0 0x7fcf8c154190 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c0ad5e0 tx=0x7fcf8401c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f76242ed4aaab10b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:40.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.831+0000 7fcf8b7fe640 1 -- 172.21.15.116:0/3986223594 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf840b2070 con 0x7fcf8c072ca0 2024-04-25T00:15:40.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.831+0000 7fcf8b7fe640 1 -- 172.21.15.116:0/3986223594 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf840a58d0 con 0x7fcf8c072ca0 2024-04-25T00:15:40.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.831+0000 7fcf8b7fe640 1 -- 172.21.15.116:0/3986223594 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf840a5ca0 con 0x7fcf8c072ca0 2024-04-25T00:15:40.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.831+0000 7fcf92ca7640 1 -- 172.21.15.116:0/3986223594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf8c072ca0 msgr2=0x7fcf8c154190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:40.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.831+0000 7fcf92ca7640 1 --2- 172.21.15.116:0/3986223594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf8c072ca0 0x7fcf8c154190 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c0ad5e0 tx=0x7fcf8401c810 comp rx=0 tx=0).stop 2024-04-25T00:15:40.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.831+0000 7fcf92ca7640 1 -- 172.21.15.116:0/3986223594 shutdown_connections 2024-04-25T00:15:40.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.831+0000 7fcf92ca7640 1 --2- 172.21.15.116:0/3986223594 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf8c072ca0 0x7fcf8c154190 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:40.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.831+0000 7fcf92ca7640 1 --2- 172.21.15.116:0/3986223594 >> v2:172.21.15.174:3300/0 conn(0x7fcf8c150d10 0x7fcf8c0c5010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:40.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.831+0000 7fcf92ca7640 1 -- 172.21.15.116:0/3986223594 >> 172.21.15.116:0/3986223594 conn(0x7fcf8c06c720 msgr2=0x7fcf8c06cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:40.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.831+0000 7fcf92ca7640 1 -- 172.21.15.116:0/3986223594 shutdown_connections 2024-04-25T00:15:40.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.831+0000 7fcf92ca7640 1 -- 172.21.15.116:0/3986223594 wait complete. 2024-04-25T00:15:40.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.831+0000 7fcf92ca7640 1 Processor -- start 2024-04-25T00:15:40.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.831+0000 7fcf92ca7640 1 -- start start 2024-04-25T00:15:40.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.835+0000 7fcf92ca7640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcf8c072ca0 0x7fcf8c1efad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:40.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.835+0000 7fcf92ca7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf8c0c5550 0x7fcf8c141d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:40.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.835+0000 7fcf92ca7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf8c150d10 0x7fcf8c14afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:40.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.835+0000 7fcf92ca7640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcf8c0c2640 con 0x7fcf8c0c5550 2024-04-25T00:15:40.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.835+0000 7fcf92ca7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcf8c1576a0 con 0x7fcf8c150d10 2024-04-25T00:15:40.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.835+0000 7fcf92ca7640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fcf8c0c27c0 con 0x7fcf8c072ca0 2024-04-25T00:15:40.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.835+0000 7fcf8bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf8c0c5550 0x7fcf8c141d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:40.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.835+0000 7fcf8bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf8c0c5550 0x7fcf8c141d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60908/0 (socket says 172.21.15.116:60908) 2024-04-25T00:15:40.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.835+0000 7fcf8bfff640 1 -- 172.21.15.116:0/39834653 learned_addr learned my addr 172.21.15.116:0/39834653 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:15:40.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.835+0000 7fcf91244640 1 --2- 172.21.15.116:0/39834653 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf8c150d10 0x7fcf8c14afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:40.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.835+0000 7fcf90a43640 1 --2- 172.21.15.116:0/39834653 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcf8c072ca0 0x7fcf8c1efad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:40.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.835+0000 7fcf8bfff640 1 -- 172.21.15.116:0/39834653 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcf8c072ca0 msgr2=0x7fcf8c1efad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:40.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.835+0000 7fcf8bfff640 1 --2- 172.21.15.116:0/39834653 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcf8c072ca0 0x7fcf8c1efad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:40.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.835+0000 7fcf8bfff640 1 -- 172.21.15.116:0/39834653 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf8c150d10 msgr2=0x7fcf8c14afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:40.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.835+0000 7fcf8bfff640 1 --2- 172.21.15.116:0/39834653 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf8c150d10 0x7fcf8c14afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:40.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.835+0000 7fcf8bfff640 1 -- 172.21.15.116:0/39834653 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf8c14e0a0 con 0x7fcf8c0c5550 2024-04-25T00:15:40.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.835+0000 7fcf8bfff640 1 --2- 172.21.15.116:0/39834653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf8c0c5550 0x7fcf8c141d00 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7fcf7800c420 tx=0x7fcf7800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:40.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.835+0000 7fcf897fa640 1 -- 172.21.15.116:0/39834653 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf78017070 con 0x7fcf8c0c5550 2024-04-25T00:15:40.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.835+0000 7fcf897fa640 1 -- 172.21.15.116:0/39834653 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf78005cf0 con 0x7fcf8c0c5550 2024-04-25T00:15:40.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.835+0000 7fcf92ca7640 1 -- 172.21.15.116:0/39834653 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf8c1f0010 con 0x7fcf8c0c5550 2024-04-25T00:15:40.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.835+0000 7fcf897fa640 1 -- 172.21.15.116:0/39834653 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf780072f0 con 0x7fcf8c0c5550 2024-04-25T00:15:40.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.835+0000 7fcf92ca7640 1 -- 172.21.15.116:0/39834653 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf8c14d1b0 con 0x7fcf8c0c5550 2024-04-25T00:15:40.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.835+0000 7fcf92ca7640 1 -- 172.21.15.116:0/39834653 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf8c068a60 con 0x7fcf8c0c5550 2024-04-25T00:15:40.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.835+0000 7fcf897fa640 1 -- 172.21.15.116:0/39834653 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fcf7801a070 con 0x7fcf8c0c5550 2024-04-25T00:15:40.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.839+0000 7fcf897fa640 1 --2- 172.21.15.116:0/39834653 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcf700678c0 0x7fcf70069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:40.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.839+0000 7fcf897fa640 1 -- 172.21.15.116:0/39834653 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 8472+0+0 (secure 0 0 0) 0x7fcf78086260 con 0x7fcf8c0c5550 2024-04-25T00:15:40.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.839+0000 7fcf90a43640 1 --2- 172.21.15.116:0/39834653 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcf700678c0 0x7fcf70069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:40.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.839+0000 7fcf90a43640 1 --2- 172.21.15.116:0/39834653 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcf700678c0 0x7fcf70069d80 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fcf8000b930 tx=0x7fcf80008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:40.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:40.839+0000 7fcf897fa640 1 -- 172.21.15.116:0/39834653 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fcf78021070 con 0x7fcf8c0c5550 2024-04-25T00:15:41.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:41.051+0000 7fcf92ca7640 1 -- 172.21.15.116:0/39834653 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcf8c072a80 con 0x7fcf8c0c5550 2024-04-25T00:15:41.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:41.051+0000 7fcf897fa640 1 -- 172.21.15.116:0/39834653 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) ==== 95+0+5672 (secure 0 0 0) 0x7fcf8c14e0a0 con 0x7fcf8c0c5550 2024-04-25T00:15:41.049 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:15:41.049 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":113,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:15:40.572986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6810":{"gid":6810,"name":"c","rank":0,"incarnation":113,"state":"up:replay","state_seq":1,"addr":"172.21.15.116:6833/2340581953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2340581953},{"type":"v1","addr":"172.21.15.116:6833","nonce":2340581953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:15:28.737458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6828":{"gid":6828,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":8,"addr":"172.21.15.116:6835/2012470875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2012470875},{"type":"v1","addr":"172.21.15.116:6835","nonce":2012470875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6828,"qdb_cluster":[6828]},"id":2}]} 2024-04-25T00:15:41.049 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 113 2024-04-25T00:15:41.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:41.051+0000 7fcf92ca7640 1 -- 172.21.15.116:0/39834653 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcf700678c0 msgr2=0x7fcf70069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:41.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:41.051+0000 7fcf92ca7640 1 --2- 172.21.15.116:0/39834653 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcf700678c0 0x7fcf70069d80 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fcf8000b930 tx=0x7fcf80008040 comp rx=0 tx=0).stop 2024-04-25T00:15:41.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:41.051+0000 7fcf92ca7640 1 -- 172.21.15.116:0/39834653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf8c0c5550 msgr2=0x7fcf8c141d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:41.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:41.051+0000 7fcf92ca7640 1 --2- 172.21.15.116:0/39834653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf8c0c5550 0x7fcf8c141d00 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7fcf7800c420 tx=0x7fcf7800c8e0 comp rx=0 tx=0).stop 2024-04-25T00:15:41.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:41.051+0000 7fcf92ca7640 1 -- 172.21.15.116:0/39834653 shutdown_connections 2024-04-25T00:15:41.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:41.051+0000 7fcf92ca7640 1 --2- 172.21.15.116:0/39834653 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcf700678c0 0x7fcf70069d80 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:41.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:41.051+0000 7fcf92ca7640 1 --2- 172.21.15.116:0/39834653 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf8c150d10 0x7fcf8c14afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:41.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:41.051+0000 7fcf92ca7640 1 --2- 172.21.15.116:0/39834653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf8c0c5550 0x7fcf8c141d00 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:41.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:41.051+0000 7fcf92ca7640 1 --2- 172.21.15.116:0/39834653 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcf8c072ca0 0x7fcf8c1efad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:41.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:41.051+0000 7fcf92ca7640 1 -- 172.21.15.116:0/39834653 >> 172.21.15.116:0/39834653 conn(0x7fcf8c06c720 msgr2=0x7fcf8c152f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:41.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:41.051+0000 7fcf92ca7640 1 -- 172.21.15.116:0/39834653 shutdown_connections 2024-04-25T00:15:41.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:41.055+0000 7fcf92ca7640 1 -- 172.21.15.116:0/39834653 wait complete. 2024-04-25T00:15:41.068 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 113, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:04.678737+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 113, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6810": { "addr": "172.21.15.116:6833/2340581953", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6832", "nonce": 2340581953, "type": "v2" }, { "addr": "172.21.15.116:6833", "nonce": 2340581953, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6810, "incarnation": 113, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 49, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T00:15:40.572986+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6810 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:18.825887+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 111, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6828": { "addr": "172.21.15.116:6835/2012470875", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6834", "nonce": 2012470875, "type": "v2" }, { "addr": "172.21.15.116:6835", "nonce": 2012470875, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6828, "incarnation": 108, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 8 } }, "last_failure": 0, "last_failure_osd_epoch": 48, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-04-25T00:15:28.737458+0000", "qdb_cluster": [ 6828 ], "qdb_leader": 6828, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6828 } } } ], "standbys": [ { "addr": "172.21.15.174:6835/4112145782", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6834", "nonce": 4112145782, "type": "v2" }, { "addr": "172.21.15.174:6835", "nonce": 4112145782, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 112, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5582, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-25T00:15:43.069 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:15:43.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.199+0000 7f0c99bd8640 1 Processor -- start 2024-04-25T00:15:43.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.203+0000 7f0c99bd8640 1 -- start start 2024-04-25T00:15:43.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.203+0000 7f0c99bd8640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f0c940a94a0 0x7f0c941538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:43.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.203+0000 7f0c99bd8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0c940a99f0 0x7f0c94153e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:43.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.203+0000 7f0c99bd8640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c940c3910 con 0x7f0c94150d20 2024-04-25T00:15:43.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.203+0000 7f0c99bd8640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f0c940c16e0 con 0x7f0c940a99f0 2024-04-25T00:15:43.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.203+0000 7f0c99bd8640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f0c940c1400 con 0x7f0c940a94a0 2024-04-25T00:15:43.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.203+0000 7f0c93fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0c940a99f0 0x7f0c94153e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:43.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.203+0000 7f0c93fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0c940a99f0 0x7f0c94153e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60912/0 (socket says 172.21.15.116:60912) 2024-04-25T00:15:43.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.203+0000 7f0c93fff640 1 -- 172.21.15.116:0/1953828792 learned_addr learned my addr 172.21.15.116:0/1953828792 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:15:43.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.203+0000 7f0c92ffd640 1 --2- 172.21.15.116:0/1953828792 >> v2:172.21.15.174:3300/0 conn(0x7f0c940a94a0 0x7f0c941538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:43.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.203+0000 7f0c93fff640 1 -- 172.21.15.116:0/1953828792 >> v2:172.21.15.174:3300/0 conn(0x7f0c940a94a0 msgr2=0x7f0c941538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:43.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.203+0000 7f0c93fff640 1 --2- 172.21.15.116:0/1953828792 >> v2:172.21.15.174:3300/0 conn(0x7f0c940a94a0 0x7f0c941538e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:43.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.203+0000 7f0c93fff640 1 -- 172.21.15.116:0/1953828792 >> v1:172.21.15.174:6789/0 conn(0x7f0c94150d20 legacy=0x7f0c940c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:43.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.203+0000 7f0c93fff640 1 -- 172.21.15.116:0/1953828792 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c9406e660 con 0x7f0c940a99f0 2024-04-25T00:15:43.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.207+0000 7f0c93fff640 1 --2- 172.21.15.116:0/1953828792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c940a99f0 0x7f0c94153e20 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f0c94070710 tx=0x7f0c8c01c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4547cbc9c43ee46 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:43.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.207+0000 7f0c927fc640 1 -- 172.21.15.116:0/1953828792 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c8c0a56c0 con 0x7f0c940a99f0 2024-04-25T00:15:43.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.207+0000 7f0c927fc640 1 -- 172.21.15.116:0/1953828792 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c8c0a5860 con 0x7f0c940a99f0 2024-04-25T00:15:43.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.207+0000 7f0c927fc640 1 -- 172.21.15.116:0/1953828792 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c8c0a5c30 con 0x7f0c940a99f0 2024-04-25T00:15:43.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.207+0000 7f0c99bd8640 1 -- 172.21.15.116:0/1953828792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c940a99f0 msgr2=0x7f0c94153e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:43.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.207+0000 7f0c99bd8640 1 --2- 172.21.15.116:0/1953828792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c940a99f0 0x7f0c94153e20 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f0c94070710 tx=0x7f0c8c01c9f0 comp rx=0 tx=0).stop 2024-04-25T00:15:43.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.207+0000 7f0c99bd8640 1 -- 172.21.15.116:0/1953828792 shutdown_connections 2024-04-25T00:15:43.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.207+0000 7f0c99bd8640 1 --2- 172.21.15.116:0/1953828792 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c940a99f0 0x7f0c94153e20 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:43.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.207+0000 7f0c99bd8640 1 --2- 172.21.15.116:0/1953828792 >> v2:172.21.15.174:3300/0 conn(0x7f0c940a94a0 0x7f0c941538e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:43.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.207+0000 7f0c99bd8640 1 -- 172.21.15.116:0/1953828792 >> 172.21.15.116:0/1953828792 conn(0x7f0c940c3b00 msgr2=0x7f0c940c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:43.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.207+0000 7f0c99bd8640 1 -- 172.21.15.116:0/1953828792 shutdown_connections 2024-04-25T00:15:43.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.207+0000 7f0c99bd8640 1 -- 172.21.15.116:0/1953828792 wait complete. 2024-04-25T00:15:43.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.207+0000 7f0c99bd8640 1 Processor -- start 2024-04-25T00:15:43.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.207+0000 7f0c99bd8640 1 -- start start 2024-04-25T00:15:43.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.207+0000 7f0c99bd8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c940a94a0 0x7f0c941f2950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:43.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.207+0000 7f0c99bd8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0c940a99f0 0x7f0c940c0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:43.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.207+0000 7f0c99bd8640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0c94150d20 0x7f0c941ee820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:43.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.207+0000 7f0c99bd8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0c941544e0 con 0x7f0c940a94a0 2024-04-25T00:15:43.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.207+0000 7f0c99bd8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0c94154360 con 0x7f0c940a99f0 2024-04-25T00:15:43.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.207+0000 7f0c99bd8640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f0c94160730 con 0x7f0c94150d20 2024-04-25T00:15:43.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.207+0000 7f0c937fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c940a94a0 0x7f0c941f2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:43.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.207+0000 7f0c937fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c940a94a0 0x7f0c941f2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60928/0 (socket says 172.21.15.116:60928) 2024-04-25T00:15:43.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.207+0000 7f0c937fe640 1 -- 172.21.15.116:0/1200470688 learned_addr learned my addr 172.21.15.116:0/1200470688 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:15:43.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.207+0000 7f0c93fff640 1 --2- 172.21.15.116:0/1200470688 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0c94150d20 0x7f0c941ee820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:43.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.207+0000 7f0c92ffd640 1 --2- 172.21.15.116:0/1200470688 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0c940a99f0 0x7f0c940c0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:43.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.207+0000 7f0c937fe640 1 -- 172.21.15.116:0/1200470688 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0c94150d20 msgr2=0x7f0c941ee820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:43.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.207+0000 7f0c937fe640 1 --2- 172.21.15.116:0/1200470688 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0c94150d20 0x7f0c941ee820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:43.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.207+0000 7f0c937fe640 1 -- 172.21.15.116:0/1200470688 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0c940a99f0 msgr2=0x7f0c940c0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:43.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.207+0000 7f0c937fe640 1 --2- 172.21.15.116:0/1200470688 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0c940a99f0 0x7f0c940c0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:43.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.207+0000 7f0c937fe640 1 -- 172.21.15.116:0/1200470688 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c941423b0 con 0x7f0c940a94a0 2024-04-25T00:15:43.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.211+0000 7f0c937fe640 1 --2- 172.21.15.116:0/1200470688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c940a94a0 0x7f0c941f2950 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f0c8800c450 tx=0x7f0c8800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:43.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.211+0000 7f0c7bfff640 1 -- 172.21.15.116:0/1200470688 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c88016020 con 0x7f0c940a94a0 2024-04-25T00:15:43.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.211+0000 7f0c7bfff640 1 -- 172.21.15.116:0/1200470688 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c880052d0 con 0x7f0c940a94a0 2024-04-25T00:15:43.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.211+0000 7f0c99bd8640 1 -- 172.21.15.116:0/1200470688 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c941420d0 con 0x7f0c940a94a0 2024-04-25T00:15:43.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.211+0000 7f0c7bfff640 1 -- 172.21.15.116:0/1200470688 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c88007260 con 0x7f0c940a94a0 2024-04-25T00:15:43.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.211+0000 7f0c99bd8640 1 -- 172.21.15.116:0/1200470688 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c941f2150 con 0x7f0c940a94a0 2024-04-25T00:15:43.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.211+0000 7f0c99bd8640 1 -- 172.21.15.116:0/1200470688 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c94068350 con 0x7f0c940a94a0 2024-04-25T00:15:43.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.211+0000 7f0c7bfff640 1 -- 172.21.15.116:0/1200470688 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f0c88007400 con 0x7f0c940a94a0 2024-04-25T00:15:43.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.211+0000 7f0c7bfff640 1 --2- 172.21.15.116:0/1200470688 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0c800678c0 0x7f0c80069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:43.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.211+0000 7f0c7bfff640 1 -- 172.21.15.116:0/1200470688 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 8472+0+0 (secure 0 0 0) 0x7f0c880860f0 con 0x7f0c940a94a0 2024-04-25T00:15:43.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.211+0000 7f0c92ffd640 1 --2- 172.21.15.116:0/1200470688 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0c800678c0 0x7f0c80069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:43.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.211+0000 7f0c92ffd640 1 --2- 172.21.15.116:0/1200470688 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0c800678c0 0x7f0c80069d80 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f0c9414d540 tx=0x7f0c84008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:43.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.215+0000 7f0c7bfff640 1 -- 172.21.15.116:0/1200470688 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f0c88020020 con 0x7f0c940a94a0 2024-04-25T00:15:43.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.415+0000 7f0c99bd8640 1 -- 172.21.15.116:0/1200470688 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0c9415dcf0 con 0x7f0c940a94a0 2024-04-25T00:15:43.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.415+0000 7f0c7bfff640 1 -- 172.21.15.116:0/1200470688 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) ==== 95+0+5673 (secure 0 0 0) 0x7f0c88005480 con 0x7f0c940a94a0 2024-04-25T00:15:43.414 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:15:43.414 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":115,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:15:41.814150+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6810":{"gid":6810,"name":"c","rank":0,"incarnation":113,"state":"up:rejoin","state_seq":12,"addr":"172.21.15.116:6833/2340581953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2340581953},{"type":"v1","addr":"172.21.15.116:6833","nonce":2340581953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:15:28.737458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6828":{"gid":6828,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":8,"addr":"172.21.15.116:6835/2012470875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2012470875},{"type":"v1","addr":"172.21.15.116:6835","nonce":2012470875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6828,"qdb_cluster":[6828]},"id":2}]} 2024-04-25T00:15:43.414 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 115 2024-04-25T00:15:43.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.419+0000 7f0c99bd8640 1 -- 172.21.15.116:0/1200470688 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0c800678c0 msgr2=0x7f0c80069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:43.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.419+0000 7f0c99bd8640 1 --2- 172.21.15.116:0/1200470688 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0c800678c0 0x7f0c80069d80 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f0c9414d540 tx=0x7f0c84008040 comp rx=0 tx=0).stop 2024-04-25T00:15:43.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.419+0000 7f0c99bd8640 1 -- 172.21.15.116:0/1200470688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c940a94a0 msgr2=0x7f0c941f2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:43.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.419+0000 7f0c99bd8640 1 --2- 172.21.15.116:0/1200470688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c940a94a0 0x7f0c941f2950 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f0c8800c450 tx=0x7f0c8800c910 comp rx=0 tx=0).stop 2024-04-25T00:15:43.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.419+0000 7f0c99bd8640 1 -- 172.21.15.116:0/1200470688 shutdown_connections 2024-04-25T00:15:43.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.419+0000 7f0c99bd8640 1 --2- 172.21.15.116:0/1200470688 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0c800678c0 0x7f0c80069d80 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:43.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.419+0000 7f0c99bd8640 1 --2- 172.21.15.116:0/1200470688 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0c94150d20 0x7f0c941ee820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:43.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.419+0000 7f0c99bd8640 1 --2- 172.21.15.116:0/1200470688 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0c940a99f0 0x7f0c940c0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:43.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.419+0000 7f0c99bd8640 1 --2- 172.21.15.116:0/1200470688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c940a94a0 0x7f0c941f2950 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:43.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.419+0000 7f0c99bd8640 1 -- 172.21.15.116:0/1200470688 >> 172.21.15.116:0/1200470688 conn(0x7f0c940c3b00 msgr2=0x7f0c940c2a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:43.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.419+0000 7f0c99bd8640 1 -- 172.21.15.116:0/1200470688 shutdown_connections 2024-04-25T00:15:43.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:43.419+0000 7f0c99bd8640 1 -- 172.21.15.116:0/1200470688 wait complete. 2024-04-25T00:15:43.433 INFO:tasks.mds_thrash.fs.[a]:no change 2024-04-25T00:15:44.897 INFO:tasks.ceph.mds.c.smithi116.stderr:2024-04-25T00:15:44.899+0000 7f0cf6b2a640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T00:15:45.434 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:15:45.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.575+0000 7ff001ea2640 1 Processor -- start 2024-04-25T00:15:45.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.575+0000 7ff001ea2640 1 -- start start 2024-04-25T00:15:45.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.575+0000 7ff001ea2640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7feffc1573e0 0x7feffc15ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:45.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.575+0000 7ff001ea2640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feffc073200 con 0x7feffc072c40 2024-04-25T00:15:45.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.575+0000 7ff001ea2640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feffc154090 con 0x7feffc14a1a0 2024-04-25T00:15:45.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.575+0000 7ff001ea2640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7feffc0c30f0 con 0x7feffc1573e0 2024-04-25T00:15:45.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.575+0000 7feffaffd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7feffc072c40 0x7feffc156cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:42426/0 (socket says 172.21.15.116:42426) 2024-04-25T00:15:45.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.575+0000 7feffaffd640 1 -- 172.21.15.116:0/3676217251 learned_addr learned my addr 172.21.15.116:0/3676217251 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:15:45.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.575+0000 7feffbfff640 1 --2- 172.21.15.116:0/3676217251 >> v2:172.21.15.116:3300/0 conn(0x7feffc1573e0 0x7feffc15ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:45.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.575+0000 7feffbfff640 1 -- 172.21.15.116:0/3676217251 >> v1:172.21.15.174:6789/0 conn(0x7feffc14a1a0 legacy=0x7feffc0c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:45.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.575+0000 7feffa7fc640 1 -- 172.21.15.116:0/3676217251 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1729744391 0 0) 0x7feffc073200 con 0x7feffc072c40 2024-04-25T00:15:45.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.575+0000 7feffbfff640 1 -- 172.21.15.116:0/3676217251 >> v1:172.21.15.116:6789/0 conn(0x7feffc072c40 legacy=0x7feffc156cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:45.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.575+0000 7feffbfff640 1 -- 172.21.15.116:0/3676217251 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feffc154370 con 0x7feffc1573e0 2024-04-25T00:15:45.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.579+0000 7feffbfff640 1 --2- 172.21.15.116:0/3676217251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feffc1573e0 0x7feffc15ef00 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7feff4009d90 tx=0x7feff401c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d171e208d19958ee server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:45.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.579+0000 7feffa7fc640 1 -- 172.21.15.116:0/3676217251 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff40b2070 con 0x7feffc1573e0 2024-04-25T00:15:45.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.579+0000 7feffa7fc640 1 -- 172.21.15.116:0/3676217251 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feff40a5890 con 0x7feffc1573e0 2024-04-25T00:15:45.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.579+0000 7feffa7fc640 1 -- 172.21.15.116:0/3676217251 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff40a5c60 con 0x7feffc1573e0 2024-04-25T00:15:45.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.579+0000 7ff001ea2640 1 -- 172.21.15.116:0/3676217251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feffc1573e0 msgr2=0x7feffc15ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:45.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.579+0000 7ff001ea2640 1 --2- 172.21.15.116:0/3676217251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feffc1573e0 0x7feffc15ef00 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7feff4009d90 tx=0x7feff401c8e0 comp rx=0 tx=0).stop 2024-04-25T00:15:45.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.579+0000 7ff001ea2640 1 -- 172.21.15.116:0/3676217251 shutdown_connections 2024-04-25T00:15:45.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.579+0000 7ff001ea2640 1 --2- 172.21.15.116:0/3676217251 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feffc1573e0 0x7feffc15ef00 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:45.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.579+0000 7ff001ea2640 1 -- 172.21.15.116:0/3676217251 >> 172.21.15.116:0/3676217251 conn(0x7feffc06c720 msgr2=0x7feffc06cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:45.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.579+0000 7ff001ea2640 1 -- 172.21.15.116:0/3676217251 shutdown_connections 2024-04-25T00:15:45.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.579+0000 7ff001ea2640 1 -- 172.21.15.116:0/3676217251 wait complete. 2024-04-25T00:15:45.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.583+0000 7ff001ea2640 1 Processor -- start 2024-04-25T00:15:45.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.583+0000 7ff001ea2640 1 -- start start 2024-04-25T00:15:45.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.583+0000 7ff001ea2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feffc072c40 0x7feffc1ef040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:45.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.583+0000 7ff001ea2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feffc14a1a0 0x7feffc142d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:45.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.583+0000 7ff001ea2640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7feffc1573e0 0x7feffc143280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:45.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.583+0000 7ff001ea2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7feffc0c58d0 con 0x7feffc14a1a0 2024-04-25T00:15:45.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.583+0000 7ff001ea2640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7feffc0c5750 con 0x7feffc072c40 2024-04-25T00:15:45.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.583+0000 7ff001ea2640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7feffc0c25b0 con 0x7feffc1573e0 2024-04-25T00:15:45.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.583+0000 7feffaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feffc14a1a0 0x7feffc142d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:45.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.583+0000 7feffaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feffc14a1a0 0x7feffc142d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52170/0 (socket says 172.21.15.116:52170) 2024-04-25T00:15:45.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.583+0000 7feffaffd640 1 -- 172.21.15.116:0/143268318 learned_addr learned my addr 172.21.15.116:0/143268318 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:15:45.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.583+0000 7feffb7fe640 1 --2- 172.21.15.116:0/143268318 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feffc072c40 0x7feffc1ef040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:45.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.583+0000 7feffbfff640 1 --2- 172.21.15.116:0/143268318 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7feffc1573e0 0x7feffc143280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:45.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.583+0000 7feffaffd640 1 -- 172.21.15.116:0/143268318 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7feffc1573e0 msgr2=0x7feffc143280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:45.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.583+0000 7feffaffd640 1 --2- 172.21.15.116:0/143268318 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7feffc1573e0 0x7feffc143280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:45.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.583+0000 7feffaffd640 1 -- 172.21.15.116:0/143268318 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feffc072c40 msgr2=0x7feffc1ef040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:45.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.583+0000 7feffaffd640 1 --2- 172.21.15.116:0/143268318 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feffc072c40 0x7feffc1ef040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:45.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.583+0000 7feffaffd640 1 -- 172.21.15.116:0/143268318 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feffc144cd0 con 0x7feffc14a1a0 2024-04-25T00:15:45.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.583+0000 7feffbfff640 1 --2- 172.21.15.116:0/143268318 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7feffc1573e0 0x7feffc143280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:15:45.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.583+0000 7feffaffd640 1 --2- 172.21.15.116:0/143268318 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feffc14a1a0 0x7feffc142d40 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7feff000c4a0 tx=0x7feff000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:45.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.583+0000 7fefe3fff640 1 -- 172.21.15.116:0/143268318 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff0018070 con 0x7feffc14a1a0 2024-04-25T00:15:45.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.583+0000 7fefe3fff640 1 -- 172.21.15.116:0/143268318 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feff00074e0 con 0x7feffc14a1a0 2024-04-25T00:15:45.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.583+0000 7fefe3fff640 1 -- 172.21.15.116:0/143268318 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff0004a30 con 0x7feffc14a1a0 2024-04-25T00:15:45.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.583+0000 7ff001ea2640 1 -- 172.21.15.116:0/143268318 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feffc1ef990 con 0x7feffc14a1a0 2024-04-25T00:15:45.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.583+0000 7ff001ea2640 1 -- 172.21.15.116:0/143268318 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feffc145250 con 0x7feffc14a1a0 2024-04-25T00:15:45.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.583+0000 7ff001ea2640 1 -- 172.21.15.116:0/143268318 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feffc068a60 con 0x7feffc14a1a0 2024-04-25T00:15:45.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.583+0000 7fefe3fff640 1 -- 172.21.15.116:0/143268318 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7feff001b070 con 0x7feffc14a1a0 2024-04-25T00:15:45.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.587+0000 7fefe3fff640 1 --2- 172.21.15.116:0/143268318 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fefe80678c0 0x7fefe8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:45.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.587+0000 7fefe3fff640 1 -- 172.21.15.116:0/143268318 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 8472+0+0 (secure 0 0 0) 0x7feff0085940 con 0x7feffc14a1a0 2024-04-25T00:15:45.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.587+0000 7feffb7fe640 1 --2- 172.21.15.116:0/143268318 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fefe80678c0 0x7fefe8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:45.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.587+0000 7feffb7fe640 1 --2- 172.21.15.116:0/143268318 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fefe80678c0 0x7fefe8069d80 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fefec007fd0 tx=0x7fefec009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:45.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.591+0000 7fefe3fff640 1 -- 172.21.15.116:0/143268318 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7feff00553a0 con 0x7feffc14a1a0 2024-04-25T00:15:45.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.843+0000 7ff001ea2640 1 -- 172.21.15.116:0/143268318 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feffc072a20 con 0x7feffc14a1a0 2024-04-25T00:15:45.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.843+0000 7fefe3fff640 1 -- 172.21.15.116:0/143268318 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) ==== 95+0+5680 (secure 0 0 0) 0x7feff00596a0 con 0x7feffc14a1a0 2024-04-25T00:15:45.841 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:15:45.841 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":116,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:15:43.819845+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6810":{"gid":6810,"name":"c","rank":0,"incarnation":113,"state":"up:active","state_seq":13,"addr":"172.21.15.116:6833/2340581953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2340581953},{"type":"v1","addr":"172.21.15.116:6833","nonce":2340581953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6810,"qdb_cluster":[6810]},"id":1},{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:15:28.737458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6828":{"gid":6828,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":8,"addr":"172.21.15.116:6835/2012470875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2012470875},{"type":"v1","addr":"172.21.15.116:6835","nonce":2012470875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6828,"qdb_cluster":[6828]},"id":2}]} 2024-04-25T00:15:45.842 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 116 2024-04-25T00:15:45.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.847+0000 7ff001ea2640 1 -- 172.21.15.116:0/143268318 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fefe80678c0 msgr2=0x7fefe8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:45.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.847+0000 7ff001ea2640 1 --2- 172.21.15.116:0/143268318 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fefe80678c0 0x7fefe8069d80 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fefec007fd0 tx=0x7fefec009040 comp rx=0 tx=0).stop 2024-04-25T00:15:45.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.847+0000 7ff001ea2640 1 -- 172.21.15.116:0/143268318 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feffc14a1a0 msgr2=0x7feffc142d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:45.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.847+0000 7ff001ea2640 1 --2- 172.21.15.116:0/143268318 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feffc14a1a0 0x7feffc142d40 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7feff000c4a0 tx=0x7feff000c960 comp rx=0 tx=0).stop 2024-04-25T00:15:45.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.847+0000 7ff001ea2640 1 -- 172.21.15.116:0/143268318 shutdown_connections 2024-04-25T00:15:45.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.847+0000 7ff001ea2640 1 --2- 172.21.15.116:0/143268318 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fefe80678c0 0x7fefe8069d80 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:45.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.847+0000 7ff001ea2640 1 --2- 172.21.15.116:0/143268318 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7feffc1573e0 0x7feffc143280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:45.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.847+0000 7ff001ea2640 1 --2- 172.21.15.116:0/143268318 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feffc14a1a0 0x7feffc142d40 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:45.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.847+0000 7ff001ea2640 1 --2- 172.21.15.116:0/143268318 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feffc072c40 0x7feffc1ef040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:45.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.847+0000 7ff001ea2640 1 -- 172.21.15.116:0/143268318 >> 172.21.15.116:0/143268318 conn(0x7feffc06c720 msgr2=0x7feffc0c5020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:45.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.847+0000 7ff001ea2640 1 -- 172.21.15.116:0/143268318 shutdown_connections 2024-04-25T00:15:45.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:45.847+0000 7ff001ea2640 1 -- 172.21.15.116:0/143268318 wait complete. 2024-04-25T00:15:45.861 INFO:tasks.mds_thrash.fs.[a]:mds.c has gained rank=0, replacing gid=5273 2024-04-25T00:15:45.861 INFO:tasks.mds_thrash.fs.[a]:waiting for 3 secs before reviving mds.d 2024-04-25T00:15:48.862 INFO:tasks.mds_thrash.fs.[a]:reviving mds.d 2024-04-25T00:15:48.862 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-25T00:15:48.863 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-25T00:15:48.866 INFO:tasks.ceph.mds.d:Started 2024-04-25T00:15:48.867 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:15:48.941 INFO:tasks.ceph.mds.d.smithi174.stdout:starting mds.d at 2024-04-25T00:15:49.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.007+0000 7f768dc2e640 1 Processor -- start 2024-04-25T00:15:49.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.007+0000 7f768dc2e640 1 -- start start 2024-04-25T00:15:49.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.007+0000 7f768dc2e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f768814fa60 0x7f7688073490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:49.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.007+0000 7f768dc2e640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f76880c0b30 0x7f76881560a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:49.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.007+0000 7f768dc2e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f76881565e0 0x7f76880c3780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:49.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.007+0000 7f768dc2e640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f7688155c10 con 0x7f768814fa60 2024-04-25T00:15:49.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.007+0000 7f768dc2e640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f768814fe40 con 0x7f76880c0b30 2024-04-25T00:15:49.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.007+0000 7f768dc2e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f76881558c0 con 0x7f76881565e0 2024-04-25T00:15:49.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.007+0000 7f76877fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f768814fa60 0x7f7688073490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:49.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.007+0000 7f76877fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f768814fa60 0x7f7688073490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52180/0 (socket says 172.21.15.116:52180) 2024-04-25T00:15:49.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.007+0000 7f76877fe640 1 -- 172.21.15.116:0/656266778 learned_addr learned my addr 172.21.15.116:0/656266778 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:15:49.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.007+0000 7f7687fff640 1 --2- 172.21.15.116:0/656266778 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f76881565e0 0x7f76880c3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:49.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.007+0000 7f7686ffd640 1 --2- 172.21.15.116:0/656266778 >> v2:172.21.15.174:3300/0 conn(0x7f76880c0b30 0x7f76881560a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:49.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.011+0000 7f76877fe640 1 -- 172.21.15.116:0/656266778 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f76881565e0 msgr2=0x7f76880c3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:49.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.011+0000 7f76877fe640 1 --2- 172.21.15.116:0/656266778 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f76881565e0 0x7f76880c3780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:49.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.011+0000 7f76877fe640 1 -- 172.21.15.116:0/656266778 >> v2:172.21.15.174:3300/0 conn(0x7f76880c0b30 msgr2=0x7f76881560a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:15:49.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.011+0000 7f76877fe640 1 --2- 172.21.15.116:0/656266778 >> v2:172.21.15.174:3300/0 conn(0x7f76880c0b30 0x7f76881560a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:49.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.011+0000 7f76877fe640 1 -- 172.21.15.116:0/656266778 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76880c1040 con 0x7f768814fa60 2024-04-25T00:15:49.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.011+0000 7f76877fe640 1 --2- 172.21.15.116:0/656266778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f768814fa60 0x7f7688073490 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f76880aa570 tx=0x7f767801c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b85ad11743424196 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:49.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.011+0000 7f76867fc640 1 -- 172.21.15.116:0/656266778 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76780b2070 con 0x7f768814fa60 2024-04-25T00:15:49.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.011+0000 7f76867fc640 1 -- 172.21.15.116:0/656266778 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76780ad640 con 0x7f768814fa60 2024-04-25T00:15:49.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.011+0000 7f76867fc640 1 -- 172.21.15.116:0/656266778 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76780ada10 con 0x7f768814fa60 2024-04-25T00:15:49.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.011+0000 7f768dc2e640 1 -- 172.21.15.116:0/656266778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f768814fa60 msgr2=0x7f7688073490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:49.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.011+0000 7f768dc2e640 1 --2- 172.21.15.116:0/656266778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f768814fa60 0x7f7688073490 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f76880aa570 tx=0x7f767801c3d0 comp rx=0 tx=0).stop 2024-04-25T00:15:49.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.011+0000 7f768dc2e640 1 -- 172.21.15.116:0/656266778 shutdown_connections 2024-04-25T00:15:49.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.011+0000 7f768dc2e640 1 --2- 172.21.15.116:0/656266778 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f76881565e0 0x7f76880c3780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:49.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.011+0000 7f768dc2e640 1 --2- 172.21.15.116:0/656266778 >> v2:172.21.15.174:3300/0 conn(0x7f76880c0b30 0x7f76881560a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:49.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.011+0000 7f768dc2e640 1 --2- 172.21.15.116:0/656266778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f768814fa60 0x7f7688073490 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:49.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.011+0000 7f768dc2e640 1 -- 172.21.15.116:0/656266778 >> 172.21.15.116:0/656266778 conn(0x7f76880c4200 msgr2=0x7f76880c4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:49.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.011+0000 7f768dc2e640 1 -- 172.21.15.116:0/656266778 shutdown_connections 2024-04-25T00:15:49.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.011+0000 7f768dc2e640 1 -- 172.21.15.116:0/656266778 wait complete. 2024-04-25T00:15:49.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.015+0000 7f768dc2e640 1 Processor -- start 2024-04-25T00:15:49.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.015+0000 7f768dc2e640 1 -- start start 2024-04-25T00:15:49.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.015+0000 7f768dc2e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f76880c0b30 0x7f76880c65a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:49.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.015+0000 7f768dc2e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f768814fa60 0x7f768814e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:49.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.015+0000 7f768dc2e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76881565e0 0x7f768814e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:49.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.015+0000 7f768dc2e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f768813ef90 con 0x7f76881565e0 2024-04-25T00:15:49.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.015+0000 7f768dc2e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f768813ee10 con 0x7f76880c0b30 2024-04-25T00:15:49.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.015+0000 7f768dc2e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f768813f110 con 0x7f768814fa60 2024-04-25T00:15:49.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.015+0000 7f7687fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76881565e0 0x7f768814e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:49.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.015+0000 7f7687fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76881565e0 0x7f768814e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52194/0 (socket says 172.21.15.116:52194) 2024-04-25T00:15:49.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.015+0000 7f7687fff640 1 -- 172.21.15.116:0/1461433764 learned_addr learned my addr 172.21.15.116:0/1461433764 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:15:49.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.015+0000 7f76877fe640 1 --2- 172.21.15.116:0/1461433764 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f76880c0b30 0x7f76880c65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:49.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.015+0000 7f7686ffd640 1 --2- 172.21.15.116:0/1461433764 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f768814fa60 0x7f768814e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:49.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.015+0000 7f7687fff640 1 -- 172.21.15.116:0/1461433764 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f768814fa60 msgr2=0x7f768814e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:49.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.015+0000 7f7687fff640 1 --2- 172.21.15.116:0/1461433764 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f768814fa60 0x7f768814e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:49.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.015+0000 7f7687fff640 1 -- 172.21.15.116:0/1461433764 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f76880c0b30 msgr2=0x7f76880c65a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:49.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.015+0000 7f7687fff640 1 --2- 172.21.15.116:0/1461433764 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f76880c0b30 0x7f76880c65a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:49.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.015+0000 7f7687fff640 1 -- 172.21.15.116:0/1461433764 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f768814d850 con 0x7f76881565e0 2024-04-25T00:15:49.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.015+0000 7f76877fe640 1 --2- 172.21.15.116:0/1461433764 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f76880c0b30 0x7f76880c65a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:15:49.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.015+0000 7f7687fff640 1 --2- 172.21.15.116:0/1461433764 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76881565e0 0x7f768814e550 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f7680009e30 tx=0x7f768000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:49.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.015+0000 7f766ffff640 1 -- 172.21.15.116:0/1461433764 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7680018020 con 0x7f76881565e0 2024-04-25T00:15:49.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.015+0000 7f766ffff640 1 -- 172.21.15.116:0/1461433764 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7680014480 con 0x7f76881565e0 2024-04-25T00:15:49.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.015+0000 7f768dc2e640 1 -- 172.21.15.116:0/1461433764 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7688150910 con 0x7f76881565e0 2024-04-25T00:15:49.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.015+0000 7f766ffff640 1 -- 172.21.15.116:0/1461433764 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7680014870 con 0x7f76881565e0 2024-04-25T00:15:49.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.015+0000 7f768dc2e640 1 -- 172.21.15.116:0/1461433764 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76880c6ae0 con 0x7f76881565e0 2024-04-25T00:15:49.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.015+0000 7f768dc2e640 1 -- 172.21.15.116:0/1461433764 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7688068a60 con 0x7f76881565e0 2024-04-25T00:15:49.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.019+0000 7f766ffff640 1 -- 172.21.15.116:0/1461433764 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f768001b020 con 0x7f76881565e0 2024-04-25T00:15:49.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.019+0000 7f766ffff640 1 --2- 172.21.15.116:0/1461433764 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f76740678c0 0x7f7674069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:49.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.019+0000 7f766ffff640 1 -- 172.21.15.116:0/1461433764 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 8472+0+0 (secure 0 0 0) 0x7f7680087520 con 0x7f76881565e0 2024-04-25T00:15:49.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.019+0000 7f76877fe640 1 --2- 172.21.15.116:0/1461433764 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f76740678c0 0x7f7674069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:49.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.019+0000 7f76877fe640 1 --2- 172.21.15.116:0/1461433764 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f76740678c0 0x7f7674069d80 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f76780af000 tx=0x7f7678008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:49.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.023+0000 7f766ffff640 1 -- 172.21.15.116:0/1461433764 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f7680022020 con 0x7f76881565e0 2024-04-25T00:15:49.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.255+0000 7f768dc2e640 1 -- 172.21.15.116:0/1461433764 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f768806c790 con 0x7f76881565e0 2024-04-25T00:15:49.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.255+0000 7f766ffff640 1 -- 172.21.15.116:0/1461433764 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) ==== 95+0+5680 (secure 0 0 0) 0x7f7680010040 con 0x7f76881565e0 2024-04-25T00:15:49.254 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:15:49.254 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":116,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:15:43.819845+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6810":{"gid":6810,"name":"c","rank":0,"incarnation":113,"state":"up:active","state_seq":13,"addr":"172.21.15.116:6833/2340581953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2340581953},{"type":"v1","addr":"172.21.15.116:6833","nonce":2340581953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6810,"qdb_cluster":[6810]},"id":1},{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:15:28.737458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6828":{"gid":6828,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":8,"addr":"172.21.15.116:6835/2012470875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2012470875},{"type":"v1","addr":"172.21.15.116:6835","nonce":2012470875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6828,"qdb_cluster":[6828]},"id":2}]} 2024-04-25T00:15:49.254 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 116 2024-04-25T00:15:49.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.259+0000 7f768dc2e640 1 -- 172.21.15.116:0/1461433764 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f76740678c0 msgr2=0x7f7674069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:49.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.259+0000 7f768dc2e640 1 --2- 172.21.15.116:0/1461433764 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f76740678c0 0x7f7674069d80 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f76780af000 tx=0x7f7678008040 comp rx=0 tx=0).stop 2024-04-25T00:15:49.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.259+0000 7f768dc2e640 1 -- 172.21.15.116:0/1461433764 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76881565e0 msgr2=0x7f768814e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:49.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.259+0000 7f768dc2e640 1 --2- 172.21.15.116:0/1461433764 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76881565e0 0x7f768814e550 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f7680009e30 tx=0x7f768000c490 comp rx=0 tx=0).stop 2024-04-25T00:15:49.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.259+0000 7f768dc2e640 1 -- 172.21.15.116:0/1461433764 shutdown_connections 2024-04-25T00:15:49.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.259+0000 7f768dc2e640 1 --2- 172.21.15.116:0/1461433764 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f76740678c0 0x7f7674069d80 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:49.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.259+0000 7f768dc2e640 1 --2- 172.21.15.116:0/1461433764 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f76881565e0 0x7f768814e550 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:49.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.259+0000 7f768dc2e640 1 --2- 172.21.15.116:0/1461433764 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f768814fa60 0x7f768814e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:49.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.259+0000 7f768dc2e640 1 --2- 172.21.15.116:0/1461433764 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f76880c0b30 0x7f76880c65a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:49.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.259+0000 7f768dc2e640 1 -- 172.21.15.116:0/1461433764 >> 172.21.15.116:0/1461433764 conn(0x7f76880c4200 msgr2=0x7f7688158690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:49.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.259+0000 7f768dc2e640 1 -- 172.21.15.116:0/1461433764 shutdown_connections 2024-04-25T00:15:49.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:49.259+0000 7f768dc2e640 1 -- 172.21.15.116:0/1461433764 wait complete. 2024-04-25T00:15:49.273 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2024-04-25T00:15:51.274 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:15:51.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.407+0000 7f8597d37640 1 Processor -- start 2024-04-25T00:15:51.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.407+0000 7f8597d37640 1 -- start start 2024-04-25T00:15:51.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.407+0000 7f8597d37640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f85900c19f0 0x7f859006f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:51.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.407+0000 7f8597d37640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8590153aa0 con 0x7f8590150bf0 2024-04-25T00:15:51.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.407+0000 7f8597d37640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8590153d80 con 0x7f8590153570 2024-04-25T00:15:51.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.407+0000 7f8597d37640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f859006e630 con 0x7f85900c19f0 2024-04-25T00:15:51.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.407+0000 7f8595ad3640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f8590150bf0 0x7f859006c220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:42442/0 (socket says 172.21.15.116:42442) 2024-04-25T00:15:51.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.407+0000 7f8595ad3640 1 -- 172.21.15.116:0/1078694261 learned_addr learned my addr 172.21.15.116:0/1078694261 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:15:51.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.407+0000 7f85962d4640 1 --2- 172.21.15.116:0/1078694261 >> v2:172.21.15.174:3300/0 conn(0x7f85900c19f0 0x7f859006f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:51.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.407+0000 7f8594ad1640 1 -- 172.21.15.116:0/1078694261 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3910442818 0 0) 0x7f8590153aa0 con 0x7f8590150bf0 2024-04-25T00:15:51.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.407+0000 7f8594ad1640 1 -- 172.21.15.116:0/1078694261 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f857c003530 con 0x7f8590150bf0 2024-04-25T00:15:51.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.407+0000 7f8594ad1640 1 -- 172.21.15.116:0/1078694261 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 679104107 0 0) 0x7f8590153d80 con 0x7f8590153570 2024-04-25T00:15:51.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.407+0000 7f8594ad1640 1 -- 172.21.15.116:0/1078694261 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8590153aa0 con 0x7f8590153570 2024-04-25T00:15:51.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.411+0000 7f85962d4640 1 -- 172.21.15.116:0/1078694261 >> v1:172.21.15.174:6789/0 conn(0x7f8590153570 legacy=0x7f859006eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:51.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.411+0000 7f85962d4640 1 -- 172.21.15.116:0/1078694261 >> v1:172.21.15.116:6789/0 conn(0x7f8590150bf0 legacy=0x7f859006c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:51.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.411+0000 7f85962d4640 1 -- 172.21.15.116:0/1078694261 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8590154060 con 0x7f85900c19f0 2024-04-25T00:15:51.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.411+0000 7f85962d4640 1 --2- 172.21.15.116:0/1078694261 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f85900c19f0 0x7f859006f1d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f8588009d90 tx=0x7f858801c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=86868a083b8853d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:51.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.411+0000 7f8594ad1640 1 -- 172.21.15.116:0/1078694261 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85880b2070 con 0x7f85900c19f0 2024-04-25T00:15:51.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.411+0000 7f8594ad1640 1 -- 172.21.15.116:0/1078694261 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85880a57a0 con 0x7f85900c19f0 2024-04-25T00:15:51.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.411+0000 7f8594ad1640 1 -- 172.21.15.116:0/1078694261 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85880a5b70 con 0x7f85900c19f0 2024-04-25T00:15:51.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.411+0000 7f8597d37640 1 -- 172.21.15.116:0/1078694261 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f85900c19f0 msgr2=0x7f859006f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:51.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.411+0000 7f8597d37640 1 --2- 172.21.15.116:0/1078694261 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f85900c19f0 0x7f859006f1d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f8588009d90 tx=0x7f858801c8e0 comp rx=0 tx=0).stop 2024-04-25T00:15:51.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.411+0000 7f8597d37640 1 -- 172.21.15.116:0/1078694261 shutdown_connections 2024-04-25T00:15:51.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.411+0000 7f8597d37640 1 --2- 172.21.15.116:0/1078694261 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f85900c19f0 0x7f859006f1d0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:51.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.411+0000 7f8597d37640 1 -- 172.21.15.116:0/1078694261 >> 172.21.15.116:0/1078694261 conn(0x7f85900c4270 msgr2=0x7f85900c4680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:51.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.411+0000 7f8597d37640 1 -- 172.21.15.116:0/1078694261 shutdown_connections 2024-04-25T00:15:51.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.411+0000 7f8597d37640 1 -- 172.21.15.116:0/1078694261 wait complete. 2024-04-25T00:15:51.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.415+0000 7f8597d37640 1 Processor -- start 2024-04-25T00:15:51.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.415+0000 7f8597d37640 1 -- start start 2024-04-25T00:15:51.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.415+0000 7f8597d37640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f85900c19f0 0x7f85901f1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:51.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.415+0000 7f8597d37640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8590150bf0 0x7f85901ede00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:51.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.415+0000 7f8597d37640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8590153570 0x7f859015e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:51.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.415+0000 7f8597d37640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f85900a72b0 con 0x7f8590153570 2024-04-25T00:15:51.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.415+0000 7f8597d37640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f85900a7130 con 0x7f8590150bf0 2024-04-25T00:15:51.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.415+0000 7f8597d37640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f8590161720 con 0x7f85900c19f0 2024-04-25T00:15:51.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.415+0000 7f85962d4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8590153570 0x7f859015e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:51.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.415+0000 7f85962d4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8590153570 0x7f859015e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52206/0 (socket says 172.21.15.116:52206) 2024-04-25T00:15:51.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.415+0000 7f85962d4640 1 -- 172.21.15.116:0/2837211847 learned_addr learned my addr 172.21.15.116:0/2837211847 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:15:51.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.415+0000 7f85952d2640 1 --2- 172.21.15.116:0/2837211847 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8590150bf0 0x7f85901ede00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:51.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.415+0000 7f8595ad3640 1 --2- 172.21.15.116:0/2837211847 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f85900c19f0 0x7f85901f1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:51.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.415+0000 7f85962d4640 1 -- 172.21.15.116:0/2837211847 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f85900c19f0 msgr2=0x7f85901f1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:51.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.415+0000 7f85962d4640 1 --2- 172.21.15.116:0/2837211847 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f85900c19f0 0x7f85901f1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:51.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.415+0000 7f85962d4640 1 -- 172.21.15.116:0/2837211847 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8590150bf0 msgr2=0x7f85901ede00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:51.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.415+0000 7f85962d4640 1 --2- 172.21.15.116:0/2837211847 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8590150bf0 0x7f85901ede00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:51.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.415+0000 7f85962d4640 1 -- 172.21.15.116:0/2837211847 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85901f1910 con 0x7f8590153570 2024-04-25T00:15:51.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.415+0000 7f85952d2640 1 --2- 172.21.15.116:0/2837211847 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8590150bf0 0x7f85901ede00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:15:51.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.415+0000 7f85962d4640 1 --2- 172.21.15.116:0/2837211847 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8590153570 0x7f859015e390 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f858801a510 tx=0x7f8588019680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:51.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.415+0000 7f85867fc640 1 -- 172.21.15.116:0/2837211847 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85880b2050 con 0x7f8590153570 2024-04-25T00:15:51.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.415+0000 7f85867fc640 1 -- 172.21.15.116:0/2837211847 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85880be070 con 0x7f8590153570 2024-04-25T00:15:51.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.415+0000 7f8597d37640 1 -- 172.21.15.116:0/2837211847 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85901f1bf0 con 0x7f8590153570 2024-04-25T00:15:51.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.415+0000 7f85867fc640 1 -- 172.21.15.116:0/2837211847 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85880ad870 con 0x7f8590153570 2024-04-25T00:15:51.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.415+0000 7f8597d37640 1 -- 172.21.15.116:0/2837211847 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f859015e8d0 con 0x7f8590153570 2024-04-25T00:15:51.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.415+0000 7f8597d37640 1 -- 172.21.15.116:0/2837211847 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8590068a60 con 0x7f8590153570 2024-04-25T00:15:51.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.419+0000 7f85867fc640 1 -- 172.21.15.116:0/2837211847 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f85880ada10 con 0x7f8590153570 2024-04-25T00:15:51.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.419+0000 7f85867fc640 1 --2- 172.21.15.116:0/2837211847 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f85740678c0 0x7f8574069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:15:51.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.419+0000 7f85867fc640 1 -- 172.21.15.116:0/2837211847 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 8472+0+0 (secure 0 0 0) 0x7f8588121750 con 0x7f8590153570 2024-04-25T00:15:51.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.419+0000 7f8595ad3640 1 --2- 172.21.15.116:0/2837211847 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f85740678c0 0x7f8574069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:15:51.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.419+0000 7f8595ad3640 1 --2- 172.21.15.116:0/2837211847 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f85740678c0 0x7f8574069d80 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f858c0011f0 tx=0x7f858c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:15:51.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.423+0000 7f85867fc640 1 -- 172.21.15.116:0/2837211847 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f8588016560 con 0x7f8590153570 2024-04-25T00:15:51.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.659+0000 7f8597d37640 1 -- 172.21.15.116:0/2837211847 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8590074520 con 0x7f8590153570 2024-04-25T00:15:51.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.663+0000 7f85867fc640 1 -- 172.21.15.116:0/2837211847 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) ==== 95+0+6509 (secure 0 0 0) 0x7f85880f18a0 con 0x7f8590153570 2024-04-25T00:15:51.659 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:15:51.660 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":117,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":5621,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3081407040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3081407040},{"type":"v1","addr":"172.21.15.174:6837","nonce":3081407040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:15:43.819845+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6810":{"gid":6810,"name":"c","rank":0,"incarnation":113,"state":"up:active","state_seq":13,"addr":"172.21.15.116:6833/2340581953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2340581953},{"type":"v1","addr":"172.21.15.116:6833","nonce":2340581953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6810,"qdb_cluster":[6810]},"id":1},{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:15:28.737458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6828":{"gid":6828,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":8,"addr":"172.21.15.116:6835/2012470875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2012470875},{"type":"v1","addr":"172.21.15.116:6835","nonce":2012470875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6828,"qdb_cluster":[6828]},"id":2}]} 2024-04-25T00:15:51.660 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 117 2024-04-25T00:15:51.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.663+0000 7f8597d37640 1 -- 172.21.15.116:0/2837211847 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f85740678c0 msgr2=0x7f8574069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:51.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.663+0000 7f8597d37640 1 --2- 172.21.15.116:0/2837211847 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f85740678c0 0x7f8574069d80 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f858c0011f0 tx=0x7f858c009040 comp rx=0 tx=0).stop 2024-04-25T00:15:51.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.663+0000 7f8597d37640 1 -- 172.21.15.116:0/2837211847 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8590153570 msgr2=0x7f859015e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:15:51.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.663+0000 7f8597d37640 1 --2- 172.21.15.116:0/2837211847 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8590153570 0x7f859015e390 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f858801a510 tx=0x7f8588019680 comp rx=0 tx=0).stop 2024-04-25T00:15:51.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.663+0000 7f8597d37640 1 -- 172.21.15.116:0/2837211847 shutdown_connections 2024-04-25T00:15:51.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.663+0000 7f8597d37640 1 --2- 172.21.15.116:0/2837211847 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f85740678c0 0x7f8574069d80 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:51.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.663+0000 7f8597d37640 1 --2- 172.21.15.116:0/2837211847 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8590153570 0x7f859015e390 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:51.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.663+0000 7f8597d37640 1 --2- 172.21.15.116:0/2837211847 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8590150bf0 0x7f85901ede00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:51.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.663+0000 7f8597d37640 1 --2- 172.21.15.116:0/2837211847 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f85900c19f0 0x7f85901f1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:15:51.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.663+0000 7f8597d37640 1 -- 172.21.15.116:0/2837211847 >> 172.21.15.116:0/2837211847 conn(0x7f85900c4270 msgr2=0x7f85900a8a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:15:51.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.663+0000 7f8597d37640 1 -- 172.21.15.116:0/2837211847 shutdown_connections 2024-04-25T00:15:51.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:15:51.663+0000 7f8597d37640 1 -- 172.21.15.116:0/2837211847 wait complete. 2024-04-25T00:15:51.680 INFO:tasks.mds_thrash.fs.[a]:mds.d reported in up:standby state 2024-04-25T00:15:51.681 INFO:tasks.mds_thrash.fs.[a]:waiting for 38 secs before thrashing 2024-04-25T00:16:08.460 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:16:08.463 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:16:21.814 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:16:21.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.962+0000 7f65634d0640 1 Processor -- start 2024-04-25T00:16:21.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.962+0000 7f65634d0640 1 -- start start 2024-04-25T00:16:21.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.962+0000 7f65634d0640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f655c0c6fc0 0x7f655c158ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:21.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.962+0000 7f65634d0640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f655c159400 0x7f655c1601f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:21.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.962+0000 7f65634d0640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f655c06e330 con 0x7f655c14fe80 2024-04-25T00:16:21.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.962+0000 7f65634d0640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f655c06cca0 con 0x7f655c0c6fc0 2024-04-25T00:16:21.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.962+0000 7f65634d0640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f655c06c810 con 0x7f655c159400 2024-04-25T00:16:21.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.962+0000 7f656126c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f655c14fe80 0x7f655c0cb860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:59200/0 (socket says 172.21.15.116:59200) 2024-04-25T00:16:21.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.962+0000 7f656126c640 1 -- 172.21.15.116:0/3710011034 learned_addr learned my addr 172.21.15.116:0/3710011034 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:16:21.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.962+0000 7f6560a6b640 1 --2- 172.21.15.116:0/3710011034 >> v2:172.21.15.174:3300/0 conn(0x7f655c0c6fc0 0x7f655c158ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:21.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.962+0000 7f6561a6d640 1 --2- 172.21.15.116:0/3710011034 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f655c159400 0x7f655c1601f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:21.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.962+0000 7f6553fff640 1 -- 172.21.15.116:0/3710011034 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1573202335 0 0) 0x7f655c06e330 con 0x7f655c14fe80 2024-04-25T00:16:21.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.962+0000 7f6553fff640 1 -- 172.21.15.116:0/3710011034 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6548003530 con 0x7f655c14fe80 2024-04-25T00:16:21.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.966+0000 7f6553fff640 1 -- 172.21.15.116:0/3710011034 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1268887973 0 0) 0x7f6548003530 con 0x7f655c14fe80 2024-04-25T00:16:21.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.966+0000 7f6553fff640 1 -- 172.21.15.116:0/3710011034 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f655c159400 msgr2=0x7f655c1601f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:21.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.966+0000 7f6553fff640 1 --2- 172.21.15.116:0/3710011034 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f655c159400 0x7f655c1601f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:21.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.966+0000 7f6553fff640 1 -- 172.21.15.116:0/3710011034 >> v2:172.21.15.174:3300/0 conn(0x7f655c0c6fc0 msgr2=0x7f655c158ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:21.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.966+0000 7f6553fff640 1 --2- 172.21.15.116:0/3710011034 >> v2:172.21.15.174:3300/0 conn(0x7f655c0c6fc0 0x7f655c158ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:21.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.966+0000 7f6553fff640 1 -- 172.21.15.116:0/3710011034 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f655c0c78f0 con 0x7f655c14fe80 2024-04-25T00:16:21.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.966+0000 7f6553fff640 1 -- 172.21.15.116:0/3710011034 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f6558001680 con 0x7f655c14fe80 2024-04-25T00:16:21.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.966+0000 7f6553fff640 1 -- 172.21.15.116:0/3710011034 >> v1:172.21.15.116:6789/0 conn(0x7f655c14fe80 legacy=0x7f655c0cb860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:21.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.966+0000 7f6553fff640 1 --2- 172.21.15.116:0/3710011034 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65480039c0 0x7f6548003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:21.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.966+0000 7f6553fff640 1 --2- 172.21.15.116:0/3710011034 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f655c159400 0x7f6548004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:21.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.966+0000 7f6553fff640 1 --2- 172.21.15.116:0/3710011034 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6548004d50 0x7f65480051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:21.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.966+0000 7f6553fff640 1 -- 172.21.15.116:0/3710011034 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f655c06cca0 con 0x7f6548004d50 2024-04-25T00:16:21.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.966+0000 7f6553fff640 1 -- 172.21.15.116:0/3710011034 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f655c06c810 con 0x7f65480039c0 2024-04-25T00:16:21.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.966+0000 7f6553fff640 1 -- 172.21.15.116:0/3710011034 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f654800ba50 con 0x7f655c159400 2024-04-25T00:16:21.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.966+0000 7f6561a6d640 1 --2- 172.21.15.116:0/3710011034 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6548004d50 0x7f65480051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:21.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.966+0000 7f656126c640 1 --2- 172.21.15.116:0/3710011034 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65480039c0 0x7f6548003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:21.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.966+0000 7f6560a6b640 1 --2- 172.21.15.116:0/3710011034 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f655c159400 0x7f6548004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:21.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.966+0000 7f6561a6d640 1 -- 172.21.15.116:0/3710011034 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f655c159400 msgr2=0x7f6548004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:21.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.966+0000 7f6561a6d640 1 --2- 172.21.15.116:0/3710011034 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f655c159400 0x7f6548004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:21.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.966+0000 7f6561a6d640 1 -- 172.21.15.116:0/3710011034 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65480039c0 msgr2=0x7f6548003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:21.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.966+0000 7f6561a6d640 1 --2- 172.21.15.116:0/3710011034 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65480039c0 0x7f6548003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:21.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.966+0000 7f6561a6d640 1 -- 172.21.15.116:0/3710011034 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6548003530 con 0x7f6548004d50 2024-04-25T00:16:21.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.966+0000 7f6561a6d640 1 --2- 172.21.15.116:0/3710011034 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6548004d50 0x7f65480051d0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f6554009fc0 tx=0x7f655401cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c18e724dd7b1e2b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:21.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.966+0000 7f6553fff640 1 -- 172.21.15.116:0/3710011034 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65540b2070 con 0x7f6548004d50 2024-04-25T00:16:21.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.966+0000 7f6553fff640 1 -- 172.21.15.116:0/3710011034 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65540a5b60 con 0x7f6548004d50 2024-04-25T00:16:21.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.966+0000 7f6553fff640 1 -- 172.21.15.116:0/3710011034 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6554009c10 con 0x7f6548004d50 2024-04-25T00:16:21.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.966+0000 7f65634d0640 1 -- 172.21.15.116:0/3710011034 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6548004d50 msgr2=0x7f65480051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:21.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.966+0000 7f65634d0640 1 --2- 172.21.15.116:0/3710011034 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6548004d50 0x7f65480051d0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f6554009fc0 tx=0x7f655401cc80 comp rx=0 tx=0).stop 2024-04-25T00:16:21.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.966+0000 7f6561a6d640 1 -- 172.21.15.116:0/3710011034 reap_dead start 2024-04-25T00:16:21.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.970+0000 7f65634d0640 1 -- 172.21.15.116:0/3710011034 shutdown_connections 2024-04-25T00:16:21.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.970+0000 7f65634d0640 1 -- 172.21.15.116:0/3710011034 >> 172.21.15.116:0/3710011034 conn(0x7f655c0ad280 msgr2=0x7f655c0cb150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:21.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.970+0000 7f65634d0640 1 -- 172.21.15.116:0/3710011034 shutdown_connections 2024-04-25T00:16:21.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.970+0000 7f65634d0640 1 -- 172.21.15.116:0/3710011034 wait complete. 2024-04-25T00:16:21.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.970+0000 7f65634d0640 1 Processor -- start 2024-04-25T00:16:21.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.970+0000 7f65634d0640 1 -- start start 2024-04-25T00:16:21.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.970+0000 7f65634d0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f655c14eeb0 0x7f655c0ae060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:21.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.970+0000 7f65634d0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f655c15cdd0 0x7f655c1eea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:21.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.970+0000 7f65634d0640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f655c1f1fa0 0x7f655c149120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:21.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.970+0000 7f65634d0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f655c0cbf70 con 0x7f655c14eeb0 2024-04-25T00:16:21.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.970+0000 7f65634d0640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f655c0cada0 con 0x7f655c15cdd0 2024-04-25T00:16:21.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.970+0000 7f65634d0640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f655c15c240 con 0x7f655c1f1fa0 2024-04-25T00:16:21.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.970+0000 7f656126c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f655c14eeb0 0x7f655c0ae060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:21.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.970+0000 7f656126c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f655c14eeb0 0x7f655c0ae060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41182/0 (socket says 172.21.15.116:41182) 2024-04-25T00:16:21.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.970+0000 7f656126c640 1 -- 172.21.15.116:0/3013909815 learned_addr learned my addr 172.21.15.116:0/3013909815 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:16:21.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.970+0000 7f6560a6b640 1 --2- 172.21.15.116:0/3013909815 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f655c15cdd0 0x7f655c1eea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:21.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.970+0000 7f6561a6d640 1 --2- 172.21.15.116:0/3013909815 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f655c1f1fa0 0x7f655c149120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:21.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.970+0000 7f656126c640 1 -- 172.21.15.116:0/3013909815 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f655c1f1fa0 msgr2=0x7f655c149120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:21.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.970+0000 7f656126c640 1 --2- 172.21.15.116:0/3013909815 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f655c1f1fa0 0x7f655c149120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:21.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.970+0000 7f656126c640 1 -- 172.21.15.116:0/3013909815 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f655c15cdd0 msgr2=0x7f655c1eea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:21.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.970+0000 7f656126c640 1 --2- 172.21.15.116:0/3013909815 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f655c15cdd0 0x7f655c1eea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:21.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.970+0000 7f656126c640 1 -- 172.21.15.116:0/3013909815 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f655c0ae5a0 con 0x7f655c14eeb0 2024-04-25T00:16:21.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.970+0000 7f6561a6d640 1 --2- 172.21.15.116:0/3013909815 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f655c1f1fa0 0x7f655c149120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:16:21.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.970+0000 7f656126c640 1 --2- 172.21.15.116:0/3013909815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f655c14eeb0 0x7f655c0ae060 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f655800eb20 tx=0x7f6558001640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:21.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.970+0000 7f6551ffb640 1 -- 172.21.15.116:0/3013909815 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f655800c8d0 con 0x7f655c14eeb0 2024-04-25T00:16:21.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.970+0000 7f6551ffb640 1 -- 172.21.15.116:0/3013909815 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6558018070 con 0x7f655c14eeb0 2024-04-25T00:16:21.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.970+0000 7f6551ffb640 1 -- 172.21.15.116:0/3013909815 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f655800ccb0 con 0x7f655c14eeb0 2024-04-25T00:16:21.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.970+0000 7f65634d0640 1 -- 172.21.15.116:0/3013909815 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f655c1eefa0 con 0x7f655c14eeb0 2024-04-25T00:16:21.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.970+0000 7f65634d0640 1 -- 172.21.15.116:0/3013909815 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f655c15c770 con 0x7f655c14eeb0 2024-04-25T00:16:21.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.974+0000 7f65634d0640 1 -- 172.21.15.116:0/3013909815 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f655c068a60 con 0x7f655c14eeb0 2024-04-25T00:16:21.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.974+0000 7f6551ffb640 1 -- 172.21.15.116:0/3013909815 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f655801d460 con 0x7f655c14eeb0 2024-04-25T00:16:21.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.974+0000 7f6551ffb640 1 --2- 172.21.15.116:0/3013909815 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f65400678c0 0x7f6540069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:21.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.974+0000 7f6551ffb640 1 -- 172.21.15.116:0/3013909815 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 8472+0+0 (secure 0 0 0) 0x7f65580865b0 con 0x7f655c14eeb0 2024-04-25T00:16:21.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.974+0000 7f6560a6b640 1 --2- 172.21.15.116:0/3013909815 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f65400678c0 0x7f6540069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:21.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.974+0000 7f6560a6b640 1 --2- 172.21.15.116:0/3013909815 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f65400678c0 0x7f6540069d80 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f655c15d4d0 tx=0x7f654c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:21.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:21.978+0000 7f6551ffb640 1 -- 172.21.15.116:0/3013909815 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f655c1eefa0 con 0x7f655c14eeb0 2024-04-25T00:16:22.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.190+0000 7f65634d0640 1 -- 172.21.15.116:0/3013909815 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f655c0c5c80 con 0x7f655c14eeb0 2024-04-25T00:16:22.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.194+0000 7f6551ffb640 1 -- 172.21.15.116:0/3013909815 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) ==== 95+0+6509 (secure 0 0 0) 0x7f6558056660 con 0x7f655c14eeb0 2024-04-25T00:16:22.192 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:16:22.192 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":117,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":5621,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3081407040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3081407040},{"type":"v1","addr":"172.21.15.174:6837","nonce":3081407040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:15:43.819845+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6810":{"gid":6810,"name":"c","rank":0,"incarnation":113,"state":"up:active","state_seq":13,"addr":"172.21.15.116:6833/2340581953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2340581953},{"type":"v1","addr":"172.21.15.116:6833","nonce":2340581953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6810,"qdb_cluster":[6810]},"id":1},{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:15:28.737458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6828":{"gid":6828,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":8,"addr":"172.21.15.116:6835/2012470875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2012470875},{"type":"v1","addr":"172.21.15.116:6835","nonce":2012470875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6828,"qdb_cluster":[6828]},"id":2}]} 2024-04-25T00:16:22.192 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 117 2024-04-25T00:16:22.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.194+0000 7f65634d0640 1 -- 172.21.15.116:0/3013909815 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f65400678c0 msgr2=0x7f6540069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:22.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.194+0000 7f65634d0640 1 --2- 172.21.15.116:0/3013909815 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f65400678c0 0x7f6540069d80 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f655c15d4d0 tx=0x7f654c008040 comp rx=0 tx=0).stop 2024-04-25T00:16:22.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.194+0000 7f65634d0640 1 -- 172.21.15.116:0/3013909815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f655c14eeb0 msgr2=0x7f655c0ae060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:22.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.194+0000 7f65634d0640 1 --2- 172.21.15.116:0/3013909815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f655c14eeb0 0x7f655c0ae060 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f655800eb20 tx=0x7f6558001640 comp rx=0 tx=0).stop 2024-04-25T00:16:22.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.194+0000 7f65634d0640 1 -- 172.21.15.116:0/3013909815 shutdown_connections 2024-04-25T00:16:22.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.194+0000 7f65634d0640 1 --2- 172.21.15.116:0/3013909815 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f65400678c0 0x7f6540069d80 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:22.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.194+0000 7f65634d0640 1 --2- 172.21.15.116:0/3013909815 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f655c1f1fa0 0x7f655c149120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:22.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.194+0000 7f65634d0640 1 --2- 172.21.15.116:0/3013909815 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f655c15cdd0 0x7f655c1eea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:22.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.194+0000 7f65634d0640 1 --2- 172.21.15.116:0/3013909815 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f655c14eeb0 0x7f655c0ae060 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:22.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.194+0000 7f65634d0640 1 -- 172.21.15.116:0/3013909815 >> 172.21.15.116:0/3013909815 conn(0x7f655c0ad280 msgr2=0x7f655c0cb7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:22.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.194+0000 7f65634d0640 1 -- 172.21.15.116:0/3013909815 shutdown_connections 2024-04-25T00:16:22.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.194+0000 7f65634d0640 1 -- 172.21.15.116:0/3013909815 wait complete. 2024-04-25T00:16:22.212 INFO:tasks.mds_thrash.fs.[b]:kill mds.a (rank=0) 2024-04-25T00:16:22.213 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-25T00:16:22.213 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:16:22.224 INFO:tasks.ceph.mds.a:Stopped 2024-04-25T00:16:22.225 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:16:22.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.398+0000 7f4647cc9640 1 Processor -- start 2024-04-25T00:16:22.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.398+0000 7f4647cc9640 1 -- start start 2024-04-25T00:16:22.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.402+0000 7f4647cc9640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f46400acef0 0x7f46401541d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:22.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.402+0000 7f4647cc9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f46400ad2d0 0x7f46400a7d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:22.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.402+0000 7f4647cc9640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46400a9ef0 con 0x7f46400a82c0 2024-04-25T00:16:22.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.402+0000 7f4647cc9640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f46400c2b60 con 0x7f46400ad2d0 2024-04-25T00:16:22.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.402+0000 7f4647cc9640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f46400c2e40 con 0x7f46400acef0 2024-04-25T00:16:22.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.402+0000 7f4645264640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f46400ad2d0 0x7f46400a7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:22.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.402+0000 7f4646266640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f46400a82c0 0x7f46400c32d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:59208/0 (socket says 172.21.15.116:59208) 2024-04-25T00:16:22.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.402+0000 7f4646266640 1 -- 172.21.15.116:0/2271267996 learned_addr learned my addr 172.21.15.116:0/2271267996 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:16:22.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.402+0000 7f4645a65640 1 --2- 172.21.15.116:0/2271267996 >> v2:172.21.15.174:3300/0 conn(0x7f46400acef0 0x7f46401541d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:22.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.402+0000 7f4645264640 1 -- 172.21.15.116:0/2271267996 >> v2:172.21.15.174:3300/0 conn(0x7f46400acef0 msgr2=0x7f46401541d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:22.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.402+0000 7f4645264640 1 --2- 172.21.15.116:0/2271267996 >> v2:172.21.15.174:3300/0 conn(0x7f46400acef0 0x7f46401541d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:22.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.402+0000 7f4645264640 1 -- 172.21.15.116:0/2271267996 >> v1:172.21.15.116:6789/0 conn(0x7f46400a82c0 legacy=0x7f46400c32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:22.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.402+0000 7f4645264640 1 -- 172.21.15.116:0/2271267996 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46400aa1d0 con 0x7f46400ad2d0 2024-04-25T00:16:22.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.402+0000 7f4644a63640 1 -- 172.21.15.116:0/2271267996 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3842278275 0 0) 0x7f46400a9ef0 con 0x7f46400a82c0 2024-04-25T00:16:22.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.406+0000 7f4645264640 1 --2- 172.21.15.116:0/2271267996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46400ad2d0 0x7f46400a7d80 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f46400c8700 tx=0x7f463001c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27dfe4ad18bd4d35 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:22.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.406+0000 7f4644a63640 1 -- 172.21.15.116:0/2271267996 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46300b0070 con 0x7f46400ad2d0 2024-04-25T00:16:22.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.406+0000 7f4644a63640 1 -- 172.21.15.116:0/2271267996 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46300ab600 con 0x7f46400ad2d0 2024-04-25T00:16:22.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.406+0000 7f4644a63640 1 -- 172.21.15.116:0/2271267996 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46300ab9d0 con 0x7f46400ad2d0 2024-04-25T00:16:22.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.406+0000 7f4647cc9640 1 -- 172.21.15.116:0/2271267996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46400ad2d0 msgr2=0x7f46400a7d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:22.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.406+0000 7f4647cc9640 1 --2- 172.21.15.116:0/2271267996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46400ad2d0 0x7f46400a7d80 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f46400c8700 tx=0x7f463001c3f0 comp rx=0 tx=0).stop 2024-04-25T00:16:22.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.406+0000 7f4647cc9640 1 -- 172.21.15.116:0/2271267996 shutdown_connections 2024-04-25T00:16:22.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.406+0000 7f4647cc9640 1 --2- 172.21.15.116:0/2271267996 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46400ad2d0 0x7f46400a7d80 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:22.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.406+0000 7f4647cc9640 1 --2- 172.21.15.116:0/2271267996 >> v2:172.21.15.174:3300/0 conn(0x7f46400acef0 0x7f46401541d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:22.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.406+0000 7f4647cc9640 1 -- 172.21.15.116:0/2271267996 >> 172.21.15.116:0/2271267996 conn(0x7f46400c4200 msgr2=0x7f46400c4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:22.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.406+0000 7f4647cc9640 1 -- 172.21.15.116:0/2271267996 shutdown_connections 2024-04-25T00:16:22.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.406+0000 7f4647cc9640 1 -- 172.21.15.116:0/2271267996 wait complete. 2024-04-25T00:16:22.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.406+0000 7f4647cc9640 1 Processor -- start 2024-04-25T00:16:22.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.406+0000 7f4647cc9640 1 -- start start 2024-04-25T00:16:22.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.406+0000 7f4647cc9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46400a82c0 0x7f464015c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:22.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.406+0000 7f4647cc9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f46400acef0 0x7f464014b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:22.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.406+0000 7f4647cc9640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f46400ad2d0 0x7f464014be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:22.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.406+0000 7f4647cc9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f46401609e0 con 0x7f46400a82c0 2024-04-25T00:16:22.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.406+0000 7f4647cc9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4640160860 con 0x7f46400acef0 2024-04-25T00:16:22.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.406+0000 7f4647cc9640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f4640160b60 con 0x7f46400ad2d0 2024-04-25T00:16:22.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.406+0000 7f4645a65640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46400a82c0 0x7f464015c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:22.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.406+0000 7f4645a65640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46400a82c0 0x7f464015c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41204/0 (socket says 172.21.15.116:41204) 2024-04-25T00:16:22.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.406+0000 7f4645a65640 1 -- 172.21.15.116:0/3001402163 learned_addr learned my addr 172.21.15.116:0/3001402163 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:16:22.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.410+0000 7f4646266640 1 --2- 172.21.15.116:0/3001402163 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f46400ad2d0 0x7f464014be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:22.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.410+0000 7f4645264640 1 --2- 172.21.15.116:0/3001402163 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f46400acef0 0x7f464014b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:22.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.410+0000 7f4645a65640 1 -- 172.21.15.116:0/3001402163 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f46400ad2d0 msgr2=0x7f464014be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:22.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.410+0000 7f4645a65640 1 --2- 172.21.15.116:0/3001402163 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f46400ad2d0 0x7f464014be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:22.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.410+0000 7f4645a65640 1 -- 172.21.15.116:0/3001402163 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f46400acef0 msgr2=0x7f464014b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:22.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.410+0000 7f4645a65640 1 --2- 172.21.15.116:0/3001402163 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f46400acef0 0x7f464014b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:22.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.410+0000 7f4645a65640 1 -- 172.21.15.116:0/3001402163 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f464014c350 con 0x7f46400a82c0 2024-04-25T00:16:22.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.410+0000 7f4645a65640 1 --2- 172.21.15.116:0/3001402163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46400a82c0 0x7f464015c420 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f463c00d800 tx=0x7f463c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:22.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.410+0000 7f46367fc640 1 -- 172.21.15.116:0/3001402163 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f463c00b6c0 con 0x7f46400a82c0 2024-04-25T00:16:22.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.410+0000 7f46367fc640 1 -- 172.21.15.116:0/3001402163 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f463c019070 con 0x7f46400a82c0 2024-04-25T00:16:22.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.410+0000 7f4647cc9640 1 -- 172.21.15.116:0/3001402163 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46401eeae0 con 0x7f46400a82c0 2024-04-25T00:16:22.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.410+0000 7f46367fc640 1 -- 172.21.15.116:0/3001402163 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f463c00ba60 con 0x7f46400a82c0 2024-04-25T00:16:22.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.410+0000 7f4647cc9640 1 -- 172.21.15.116:0/3001402163 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f464014cd50 con 0x7f46400a82c0 2024-04-25T00:16:22.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.410+0000 7f4647cc9640 1 -- 172.21.15.116:0/3001402163 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4640068a60 con 0x7f46400a82c0 2024-04-25T00:16:22.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.410+0000 7f46367fc640 1 -- 172.21.15.116:0/3001402163 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f463c00bc00 con 0x7f46400a82c0 2024-04-25T00:16:22.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.410+0000 7f46367fc640 1 --2- 172.21.15.116:0/3001402163 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f46240678c0 0x7f4624069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:22.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.410+0000 7f46367fc640 1 -- 172.21.15.116:0/3001402163 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 8472+0+0 (secure 0 0 0) 0x7f463c086c00 con 0x7f46400a82c0 2024-04-25T00:16:22.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.410+0000 7f4645264640 1 --2- 172.21.15.116:0/3001402163 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f46240678c0 0x7f4624069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:22.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.414+0000 7f4645264640 1 --2- 172.21.15.116:0/3001402163 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f46240678c0 0x7f4624069d80 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f46300ad000 tx=0x7f46300a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:22.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.414+0000 7f46367fc640 1 -- 172.21.15.116:0/3001402163 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f463c08d080 con 0x7f46400a82c0 2024-04-25T00:16:22.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.662+0000 7f4647cc9640 1 -- 172.21.15.116:0/3001402163 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f46400c23f0 con 0x7f46400a82c0 2024-04-25T00:16:22.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.662+0000 7f46367fc640 1 -- 172.21.15.116:0/3001402163 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) ==== 95+0+6509 (secure 0 0 0) 0x7f463c056cb0 con 0x7f46400a82c0 2024-04-25T00:16:22.662 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:16:22.662 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":117,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":5621,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3081407040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3081407040},{"type":"v1","addr":"172.21.15.174:6837","nonce":3081407040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:15:43.819845+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6810":{"gid":6810,"name":"c","rank":0,"incarnation":113,"state":"up:active","state_seq":13,"addr":"172.21.15.116:6833/2340581953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2340581953},{"type":"v1","addr":"172.21.15.116:6833","nonce":2340581953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6810,"qdb_cluster":[6810]},"id":1},{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:15:28.737458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6828":{"gid":6828,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":8,"addr":"172.21.15.116:6835/2012470875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2012470875},{"type":"v1","addr":"172.21.15.116:6835","nonce":2012470875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6828,"qdb_cluster":[6828]},"id":2}]} 2024-04-25T00:16:22.662 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 117 2024-04-25T00:16:22.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.666+0000 7f4647cc9640 1 -- 172.21.15.116:0/3001402163 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f46240678c0 msgr2=0x7f4624069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:22.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.666+0000 7f4647cc9640 1 --2- 172.21.15.116:0/3001402163 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f46240678c0 0x7f4624069d80 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f46300ad000 tx=0x7f46300a6000 comp rx=0 tx=0).stop 2024-04-25T00:16:22.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.666+0000 7f4647cc9640 1 -- 172.21.15.116:0/3001402163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46400a82c0 msgr2=0x7f464015c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:22.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.666+0000 7f4647cc9640 1 --2- 172.21.15.116:0/3001402163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46400a82c0 0x7f464015c420 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f463c00d800 tx=0x7f463c00dcc0 comp rx=0 tx=0).stop 2024-04-25T00:16:22.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.666+0000 7f4647cc9640 1 -- 172.21.15.116:0/3001402163 shutdown_connections 2024-04-25T00:16:22.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.666+0000 7f4647cc9640 1 --2- 172.21.15.116:0/3001402163 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f46240678c0 0x7f4624069d80 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:22.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.666+0000 7f4647cc9640 1 --2- 172.21.15.116:0/3001402163 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f46400ad2d0 0x7f464014be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:22.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.666+0000 7f4647cc9640 1 --2- 172.21.15.116:0/3001402163 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f46400acef0 0x7f464014b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:22.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.666+0000 7f4647cc9640 1 --2- 172.21.15.116:0/3001402163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f46400a82c0 0x7f464015c420 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:22.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.666+0000 7f4647cc9640 1 -- 172.21.15.116:0/3001402163 >> 172.21.15.116:0/3001402163 conn(0x7f46400c4200 msgr2=0x7f4640152af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:22.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.666+0000 7f4647cc9640 1 -- 172.21.15.116:0/3001402163 shutdown_connections 2024-04-25T00:16:22.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:22.666+0000 7f4647cc9640 1 -- 172.21.15.116:0/3001402163 wait complete. 2024-04-25T00:16:22.684 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:16:24.684 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:16:24.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.810+0000 7f0ef0493640 1 Processor -- start 2024-04-25T00:16:24.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.810+0000 7f0ef0493640 1 -- start start 2024-04-25T00:16:24.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.810+0000 7f0ef0493640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0ee8154190 0x7f0ee815f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.810+0000 7f0ef0493640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ee80c4950 con 0x7f0ee8150d10 2024-04-25T00:16:24.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.810+0000 7f0ef0493640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ee81510f0 con 0x7f0ee8072ca0 2024-04-25T00:16:24.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.810+0000 7f0ef0493640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f0ee8157a20 con 0x7f0ee8154190 2024-04-25T00:16:24.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.810+0000 7f0eee22f640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f0ee8150d10 0x7f0ee80c5010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:47600/0 (socket says 172.21.15.116:47600) 2024-04-25T00:16:24.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.810+0000 7f0eee22f640 1 -- 172.21.15.116:0/840127168 learned_addr learned my addr 172.21.15.116:0/840127168 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:16:24.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.810+0000 7f0eeea30640 1 --2- 172.21.15.116:0/840127168 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0ee8154190 0x7f0ee815f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.810+0000 7f0eed22d640 1 -- 172.21.15.116:0/840127168 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3450084373 0 0) 0x7f0ee80c4950 con 0x7f0ee8150d10 2024-04-25T00:16:24.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.810+0000 7f0eed22d640 1 -- 172.21.15.116:0/840127168 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ed4003530 con 0x7f0ee8150d10 2024-04-25T00:16:24.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.810+0000 7f0eed22d640 1 -- 172.21.15.116:0/840127168 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 933367889 0 0) 0x7f0ed4003530 con 0x7f0ee8150d10 2024-04-25T00:16:24.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.810+0000 7f0eed22d640 1 -- 172.21.15.116:0/840127168 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0ee8154190 msgr2=0x7f0ee815f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:24.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.810+0000 7f0eed22d640 1 --2- 172.21.15.116:0/840127168 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0ee8154190 0x7f0ee815f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.810+0000 7f0eed22d640 1 -- 172.21.15.116:0/840127168 >> v1:172.21.15.174:6789/0 conn(0x7f0ee8072ca0 legacy=0x7f0ee8153a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:24.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.810+0000 7f0eed22d640 1 -- 172.21.15.116:0/840127168 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ee80c2c90 con 0x7f0ee8150d10 2024-04-25T00:16:24.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.814+0000 7f0eed22d640 1 -- 172.21.15.116:0/840127168 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f0ee40016a0 con 0x7f0ee8150d10 2024-04-25T00:16:24.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.814+0000 7f0eed22d640 1 -- 172.21.15.116:0/840127168 >> v1:172.21.15.116:6789/0 conn(0x7f0ee8150d10 legacy=0x7f0ee80c5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:24.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.814+0000 7f0eed22d640 1 --2- 172.21.15.116:0/840127168 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0ed40039f0 0x7f0ed4003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.814+0000 7f0eed22d640 1 --2- 172.21.15.116:0/840127168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ed4004810 0x7f0ed4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.814+0000 7f0eed22d640 1 --2- 172.21.15.116:0/840127168 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0ee8154190 0x7f0ed400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.814+0000 7f0eed22d640 1 -- 172.21.15.116:0/840127168 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee8157a20 con 0x7f0ed4004810 2024-04-25T00:16:24.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.814+0000 7f0eed22d640 1 -- 172.21.15.116:0/840127168 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed4005560 con 0x7f0ed40039f0 2024-04-25T00:16:24.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.814+0000 7f0eed22d640 1 -- 172.21.15.116:0/840127168 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f0ed400bf10 con 0x7f0ee8154190 2024-04-25T00:16:24.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.814+0000 7f0eeda2e640 1 --2- 172.21.15.116:0/840127168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ed4004810 0x7f0ed4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.814+0000 7f0eee22f640 1 --2- 172.21.15.116:0/840127168 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0ed40039f0 0x7f0ed4003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.814+0000 7f0eeea30640 1 --2- 172.21.15.116:0/840127168 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0ee8154190 0x7f0ed400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.814+0000 7f0eeda2e640 1 -- 172.21.15.116:0/840127168 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0ee8154190 msgr2=0x7f0ed400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:24.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.814+0000 7f0eeda2e640 1 --2- 172.21.15.116:0/840127168 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0ee8154190 0x7f0ed400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.814+0000 7f0eeda2e640 1 -- 172.21.15.116:0/840127168 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0ed40039f0 msgr2=0x7f0ed4003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:24.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.814+0000 7f0eeda2e640 1 --2- 172.21.15.116:0/840127168 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0ed40039f0 0x7f0ed4003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.814+0000 7f0eeda2e640 1 -- 172.21.15.116:0/840127168 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ed4003530 con 0x7f0ed4004810 2024-04-25T00:16:24.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.814+0000 7f0eee22f640 1 --2- 172.21.15.116:0/840127168 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0ed40039f0 0x7f0ed4003e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:16:24.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.814+0000 7f0eeda2e640 1 --2- 172.21.15.116:0/840127168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ed4004810 0x7f0ed4004c90 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f0ed8001780 tx=0x7f0ed801c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=391110c3ba83e27e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:24.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.814+0000 7f0eed22d640 1 -- 172.21.15.116:0/840127168 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed80013a0 con 0x7f0ed4004810 2024-04-25T00:16:24.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.814+0000 7f0eed22d640 1 -- 172.21.15.116:0/840127168 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ed80a9070 con 0x7f0ed4004810 2024-04-25T00:16:24.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.814+0000 7f0eed22d640 1 -- 172.21.15.116:0/840127168 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ed8003600 con 0x7f0ed4004810 2024-04-25T00:16:24.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.814+0000 7f0ef0493640 1 -- 172.21.15.116:0/840127168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ed4004810 msgr2=0x7f0ed4004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:24.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.814+0000 7f0ef0493640 1 --2- 172.21.15.116:0/840127168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ed4004810 0x7f0ed4004c90 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f0ed8001780 tx=0x7f0ed801c420 comp rx=0 tx=0).stop 2024-04-25T00:16:24.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.814+0000 7f0eeea30640 1 -- 172.21.15.116:0/840127168 reap_dead start 2024-04-25T00:16:24.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.814+0000 7f0ef0493640 1 -- 172.21.15.116:0/840127168 shutdown_connections 2024-04-25T00:16:24.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.818+0000 7f0ef0493640 1 -- 172.21.15.116:0/840127168 >> 172.21.15.116:0/840127168 conn(0x7f0ee806c720 msgr2=0x7f0ee806cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:24.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.818+0000 7f0ef0493640 1 -- 172.21.15.116:0/840127168 shutdown_connections 2024-04-25T00:16:24.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.818+0000 7f0ef0493640 1 -- 172.21.15.116:0/840127168 wait complete. 2024-04-25T00:16:24.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.818+0000 7f0ef0493640 1 Processor -- start 2024-04-25T00:16:24.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.818+0000 7f0ef0493640 1 -- start start 2024-04-25T00:16:24.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.818+0000 7f0ef0493640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0ed4004810 0x7f0ee81f1e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.818+0000 7f0ef0493640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0ee8156110 0x7f0ee80c75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.818+0000 7f0ef0493640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ee8155a20 0x7f0ee81edf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.818+0000 7f0ef0493640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee80c5720 con 0x7f0ee8155a20 2024-04-25T00:16:24.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.818+0000 7f0ef0493640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee8154570 con 0x7f0ed4004810 2024-04-25T00:16:24.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.818+0000 7f0ef0493640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f0ee80c58a0 con 0x7f0ee8156110 2024-04-25T00:16:24.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.818+0000 7f0eeea30640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ee8155a20 0x7f0ee81edf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.818+0000 7f0eeea30640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ee8155a20 0x7f0ee81edf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57784/0 (socket says 172.21.15.116:57784) 2024-04-25T00:16:24.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.818+0000 7f0eeea30640 1 -- 172.21.15.116:0/4188087389 learned_addr learned my addr 172.21.15.116:0/4188087389 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:16:24.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.818+0000 7f0eee22f640 1 --2- 172.21.15.116:0/4188087389 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0ed4004810 0x7f0ee81f1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.818+0000 7f0eeda2e640 1 --2- 172.21.15.116:0/4188087389 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0ee8156110 0x7f0ee80c75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.818+0000 7f0eeea30640 1 -- 172.21.15.116:0/4188087389 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0ee8156110 msgr2=0x7f0ee80c75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:24.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.818+0000 7f0eeea30640 1 --2- 172.21.15.116:0/4188087389 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0ee8156110 0x7f0ee80c75b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.818+0000 7f0eeea30640 1 -- 172.21.15.116:0/4188087389 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0ed4004810 msgr2=0x7f0ee81f1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:24.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.818+0000 7f0eeea30640 1 --2- 172.21.15.116:0/4188087389 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0ed4004810 0x7f0ee81f1e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:24.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.818+0000 7f0eeea30640 1 -- 172.21.15.116:0/4188087389 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ee81569d0 con 0x7f0ee8155a20 2024-04-25T00:16:24.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.818+0000 7f0eeda2e640 1 --2- 172.21.15.116:0/4188087389 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0ee8156110 0x7f0ee80c75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:16:24.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.818+0000 7f0eeea30640 1 --2- 172.21.15.116:0/4188087389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ee8155a20 0x7f0ee81edf80 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f0ee0009e90 tx=0x7f0ee00115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:24.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.818+0000 7f0edeffd640 1 -- 172.21.15.116:0/4188087389 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee0011e20 con 0x7f0ee8155a20 2024-04-25T00:16:24.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.818+0000 7f0edeffd640 1 -- 172.21.15.116:0/4188087389 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ee001b070 con 0x7f0ee8155a20 2024-04-25T00:16:24.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.818+0000 7f0ef0493640 1 -- 172.21.15.116:0/4188087389 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ee81f1640 con 0x7f0ee8155a20 2024-04-25T00:16:24.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.818+0000 7f0edeffd640 1 -- 172.21.15.116:0/4188087389 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee00039a0 con 0x7f0ee8155a20 2024-04-25T00:16:24.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.818+0000 7f0ef0493640 1 -- 172.21.15.116:0/4188087389 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ee81f2380 con 0x7f0ee8155a20 2024-04-25T00:16:24.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.818+0000 7f0ef0493640 1 -- 172.21.15.116:0/4188087389 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ee8068a60 con 0x7f0ee8155a20 2024-04-25T00:16:24.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.822+0000 7f0edeffd640 1 -- 172.21.15.116:0/4188087389 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f0ee0003b40 con 0x7f0ee8155a20 2024-04-25T00:16:24.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.822+0000 7f0edeffd640 1 --2- 172.21.15.116:0/4188087389 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0ecc0678f0 0x7f0ecc069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:24.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.822+0000 7f0edeffd640 1 -- 172.21.15.116:0/4188087389 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 8472+0+0 (secure 0 0 0) 0x7f0ee0086ff0 con 0x7f0ee8155a20 2024-04-25T00:16:24.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.822+0000 7f0eee22f640 1 --2- 172.21.15.116:0/4188087389 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0ecc0678f0 0x7f0ecc069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:24.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.822+0000 7f0eee22f640 1 --2- 172.21.15.116:0/4188087389 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0ecc0678f0 0x7f0ecc069db0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8073140 tx=0x7f0ee4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:24.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:24.826+0000 7f0edeffd640 1 -- 172.21.15.116:0/4188087389 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f0ee0025020 con 0x7f0ee8155a20 2024-04-25T00:16:25.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:25.130+0000 7f0ef0493640 1 -- 172.21.15.116:0/4188087389 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0ee8150860 con 0x7f0ee8155a20 2024-04-25T00:16:25.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:25.130+0000 7f0edeffd640 1 -- 172.21.15.116:0/4188087389 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) ==== 95+0+6509 (secure 0 0 0) 0x7f0ee00570a0 con 0x7f0ee8155a20 2024-04-25T00:16:25.129 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:16:25.129 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":117,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":5621,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3081407040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3081407040},{"type":"v1","addr":"172.21.15.174:6837","nonce":3081407040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:15:43.819845+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6810":{"gid":6810,"name":"c","rank":0,"incarnation":113,"state":"up:active","state_seq":13,"addr":"172.21.15.116:6833/2340581953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2340581953},{"type":"v1","addr":"172.21.15.116:6833","nonce":2340581953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6810,"qdb_cluster":[6810]},"id":1},{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:15:28.737458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6828":{"gid":6828,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":8,"addr":"172.21.15.116:6835/2012470875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2012470875},{"type":"v1","addr":"172.21.15.116:6835","nonce":2012470875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6828,"qdb_cluster":[6828]},"id":2}]} 2024-04-25T00:16:25.129 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 117 2024-04-25T00:16:25.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:25.130+0000 7f0ef0493640 1 -- 172.21.15.116:0/4188087389 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0ecc0678f0 msgr2=0x7f0ecc069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:25.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:25.130+0000 7f0ef0493640 1 --2- 172.21.15.116:0/4188087389 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0ecc0678f0 0x7f0ecc069db0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8073140 tx=0x7f0ee4009040 comp rx=0 tx=0).stop 2024-04-25T00:16:25.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:25.130+0000 7f0ef0493640 1 -- 172.21.15.116:0/4188087389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ee8155a20 msgr2=0x7f0ee81edf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:25.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:25.130+0000 7f0ef0493640 1 --2- 172.21.15.116:0/4188087389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ee8155a20 0x7f0ee81edf80 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f0ee0009e90 tx=0x7f0ee00115a0 comp rx=0 tx=0).stop 2024-04-25T00:16:25.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:25.130+0000 7f0ef0493640 1 -- 172.21.15.116:0/4188087389 shutdown_connections 2024-04-25T00:16:25.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:25.130+0000 7f0ef0493640 1 --2- 172.21.15.116:0/4188087389 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0ecc0678f0 0x7f0ecc069db0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:25.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:25.130+0000 7f0ef0493640 1 --2- 172.21.15.116:0/4188087389 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0ee8155a20 0x7f0ee81edf80 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:25.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:25.130+0000 7f0ef0493640 1 --2- 172.21.15.116:0/4188087389 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0ee8156110 0x7f0ee80c75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:25.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:25.130+0000 7f0ef0493640 1 --2- 172.21.15.116:0/4188087389 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0ed4004810 0x7f0ee81f1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:25.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:25.130+0000 7f0ef0493640 1 -- 172.21.15.116:0/4188087389 >> 172.21.15.116:0/4188087389 conn(0x7f0ee806c720 msgr2=0x7f0ee814b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:25.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:25.130+0000 7f0ef0493640 1 -- 172.21.15.116:0/4188087389 shutdown_connections 2024-04-25T00:16:25.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:25.134+0000 7f0ef0493640 1 -- 172.21.15.116:0/4188087389 wait complete. 2024-04-25T00:16:25.148 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:16:27.149 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:16:27.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.286+0000 7f58ab451640 1 Processor -- start 2024-04-25T00:16:27.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.290+0000 7f58ab451640 1 -- start start 2024-04-25T00:16:27.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.290+0000 7f58ab451640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f58a414c700 0x7f58a40c5090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:27.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.290+0000 7f58ab451640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58a40a9e60 con 0x7f58a4155b20 2024-04-25T00:16:27.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.290+0000 7f58ab451640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58a40c3ef0 con 0x7f58a40c55d0 2024-04-25T00:16:27.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.290+0000 7f58ab451640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f58a41527b0 con 0x7f58a414c700 2024-04-25T00:16:27.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.290+0000 7f58a89ec640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f58a4155b20 0x7f58a4152c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:47608/0 (socket says 172.21.15.116:47608) 2024-04-25T00:16:27.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.290+0000 7f58a89ec640 1 -- 172.21.15.116:0/1838692939 learned_addr learned my addr 172.21.15.116:0/1838692939 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:16:27.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.290+0000 7f58a91ed640 1 --2- 172.21.15.116:0/1838692939 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f58a414c700 0x7f58a40c5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:27.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.290+0000 7f589bfff640 1 -- 172.21.15.116:0/1838692939 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3698741560 0 0) 0x7f58a40a9e60 con 0x7f58a4155b20 2024-04-25T00:16:27.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.290+0000 7f589bfff640 1 -- 172.21.15.116:0/1838692939 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5890003530 con 0x7f58a4155b20 2024-04-25T00:16:27.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.290+0000 7f589bfff640 1 -- 172.21.15.116:0/1838692939 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3912038896 0 0) 0x7f5890003530 con 0x7f58a4155b20 2024-04-25T00:16:27.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.290+0000 7f589bfff640 1 -- 172.21.15.116:0/1838692939 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f58a414c700 msgr2=0x7f58a40c5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:27.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.290+0000 7f589bfff640 1 --2- 172.21.15.116:0/1838692939 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f58a414c700 0x7f58a40c5090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:27.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.290+0000 7f589bfff640 1 -- 172.21.15.116:0/1838692939 >> v1:172.21.15.174:6789/0 conn(0x7f58a40c55d0 legacy=0x7f58a415ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:27.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.290+0000 7f589bfff640 1 -- 172.21.15.116:0/1838692939 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58a40c41d0 con 0x7f58a4155b20 2024-04-25T00:16:27.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.290+0000 7f589bfff640 1 -- 172.21.15.116:0/1838692939 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f5894001680 con 0x7f58a4155b20 2024-04-25T00:16:27.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.290+0000 7f589bfff640 1 -- 172.21.15.116:0/1838692939 >> v1:172.21.15.116:6789/0 conn(0x7f58a4155b20 legacy=0x7f58a4152c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:27.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.290+0000 7f589bfff640 1 --2- 172.21.15.116:0/1838692939 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58900039f0 0x7f5890003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:27.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.290+0000 7f589bfff640 1 --2- 172.21.15.116:0/1838692939 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f58a414c700 0x7f5890004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:27.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.290+0000 7f589bfff640 1 --2- 172.21.15.116:0/1838692939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5890004d50 0x7f58900051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:27.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.290+0000 7f589bfff640 1 -- 172.21.15.116:0/1838692939 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f58a41527b0 con 0x7f5890004d50 2024-04-25T00:16:27.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.290+0000 7f589bfff640 1 -- 172.21.15.116:0/1838692939 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f589000bb90 con 0x7f58900039f0 2024-04-25T00:16:27.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.290+0000 7f589bfff640 1 -- 172.21.15.116:0/1838692939 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f589000bd90 con 0x7f58a414c700 2024-04-25T00:16:27.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.290+0000 7f58a99ee640 1 --2- 172.21.15.116:0/1838692939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5890004d50 0x7f58900051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:27.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.290+0000 7f58a91ed640 1 --2- 172.21.15.116:0/1838692939 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58900039f0 0x7f5890003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:27.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.290+0000 7f58a89ec640 1 --2- 172.21.15.116:0/1838692939 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f58a414c700 0x7f5890004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:27.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.290+0000 7f58a99ee640 1 -- 172.21.15.116:0/1838692939 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f58a414c700 msgr2=0x7f5890004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:27.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.290+0000 7f58a99ee640 1 --2- 172.21.15.116:0/1838692939 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f58a414c700 0x7f5890004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:27.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.290+0000 7f58a99ee640 1 -- 172.21.15.116:0/1838692939 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58900039f0 msgr2=0x7f5890003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:27.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.290+0000 7f58a99ee640 1 --2- 172.21.15.116:0/1838692939 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58900039f0 0x7f5890003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:27.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.290+0000 7f58a99ee640 1 -- 172.21.15.116:0/1838692939 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5890003530 con 0x7f5890004d50 2024-04-25T00:16:27.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.294+0000 7f58a99ee640 1 --2- 172.21.15.116:0/1838692939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5890004d50 0x7f58900051d0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f589c001e70 tx=0x7f589c01c780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27f99d4449a1edf5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:27.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.294+0000 7f589bfff640 1 -- 172.21.15.116:0/1838692939 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f589c0af070 con 0x7f5890004d50 2024-04-25T00:16:27.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.294+0000 7f589bfff640 1 -- 172.21.15.116:0/1838692939 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f589c003b80 con 0x7f5890004d50 2024-04-25T00:16:27.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.294+0000 7f589bfff640 1 -- 172.21.15.116:0/1838692939 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f589c001a90 con 0x7f5890004d50 2024-04-25T00:16:27.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.294+0000 7f58ab451640 1 -- 172.21.15.116:0/1838692939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5890004d50 msgr2=0x7f58900051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:27.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.294+0000 7f58ab451640 1 --2- 172.21.15.116:0/1838692939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5890004d50 0x7f58900051d0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f589c001e70 tx=0x7f589c01c780 comp rx=0 tx=0).stop 2024-04-25T00:16:27.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.294+0000 7f58a99ee640 1 -- 172.21.15.116:0/1838692939 reap_dead start 2024-04-25T00:16:27.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.294+0000 7f58ab451640 1 -- 172.21.15.116:0/1838692939 shutdown_connections 2024-04-25T00:16:27.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.294+0000 7f58ab451640 1 -- 172.21.15.116:0/1838692939 >> 172.21.15.116:0/1838692939 conn(0x7f58a406c720 msgr2=0x7f58a406cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:27.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.294+0000 7f58ab451640 1 -- 172.21.15.116:0/1838692939 shutdown_connections 2024-04-25T00:16:27.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.294+0000 7f58ab451640 1 -- 172.21.15.116:0/1838692939 wait complete. 2024-04-25T00:16:27.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.294+0000 7f58ab451640 1 Processor -- start 2024-04-25T00:16:27.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.294+0000 7f58ab451640 1 -- start start 2024-04-25T00:16:27.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.298+0000 7f58ab451640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58a40c8750 0x7f58a41ee830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:27.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.298+0000 7f58ab451640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58a40c5c90 0x7f58a40ac0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:27.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.298+0000 7f58ab451640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f58a40c6c10 0x7f58a40ac630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:27.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.298+0000 7f58ab451640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f58a40c5420 con 0x7f58a40c8750 2024-04-25T00:16:27.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.298+0000 7f58ab451640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f58a40c52a0 con 0x7f58a40c5c90 2024-04-25T00:16:27.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.298+0000 7f58ab451640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f58a415ede0 con 0x7f58a40c6c10 2024-04-25T00:16:27.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.298+0000 7f58a91ed640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58a40c8750 0x7f58a41ee830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:27.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.298+0000 7f58a91ed640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58a40c8750 0x7f58a41ee830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57808/0 (socket says 172.21.15.116:57808) 2024-04-25T00:16:27.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.298+0000 7f58a91ed640 1 -- 172.21.15.116:0/4213342454 learned_addr learned my addr 172.21.15.116:0/4213342454 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:16:27.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.298+0000 7f58a89ec640 1 --2- 172.21.15.116:0/4213342454 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58a40c5c90 0x7f58a40ac0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:27.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.298+0000 7f58a99ee640 1 --2- 172.21.15.116:0/4213342454 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f58a40c6c10 0x7f58a40ac630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:27.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.298+0000 7f58a91ed640 1 -- 172.21.15.116:0/4213342454 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f58a40c6c10 msgr2=0x7f58a40ac630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:27.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.298+0000 7f58a91ed640 1 --2- 172.21.15.116:0/4213342454 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f58a40c6c10 0x7f58a40ac630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:27.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.298+0000 7f58a91ed640 1 -- 172.21.15.116:0/4213342454 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58a40c5c90 msgr2=0x7f58a40ac0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:27.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.298+0000 7f58a91ed640 1 --2- 172.21.15.116:0/4213342454 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58a40c5c90 0x7f58a40ac0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:27.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.298+0000 7f58a91ed640 1 -- 172.21.15.116:0/4213342454 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58a40c6130 con 0x7f58a40c8750 2024-04-25T00:16:27.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.298+0000 7f58a89ec640 1 --2- 172.21.15.116:0/4213342454 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58a40c5c90 0x7f58a40ac0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:16:27.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.298+0000 7f58a91ed640 1 --2- 172.21.15.116:0/4213342454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58a40c8750 0x7f58a41ee830 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f58a0011600 tx=0x7f58a0011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:27.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.298+0000 7f5899ffb640 1 -- 172.21.15.116:0/4213342454 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58a000f920 con 0x7f58a40c8750 2024-04-25T00:16:27.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.298+0000 7f5899ffb640 1 -- 172.21.15.116:0/4213342454 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58a000fac0 con 0x7f58a40c8750 2024-04-25T00:16:27.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.298+0000 7f5899ffb640 1 -- 172.21.15.116:0/4213342454 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58a0015460 con 0x7f58a40c8750 2024-04-25T00:16:27.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.298+0000 7f58ab451640 1 -- 172.21.15.116:0/4213342454 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58a40c6410 con 0x7f58a40c8750 2024-04-25T00:16:27.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.298+0000 7f58ab451640 1 -- 172.21.15.116:0/4213342454 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58a40ace90 con 0x7f58a40c8750 2024-04-25T00:16:27.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.298+0000 7f58ab451640 1 -- 172.21.15.116:0/4213342454 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58a4068a60 con 0x7f58a40c8750 2024-04-25T00:16:27.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.298+0000 7f5899ffb640 1 -- 172.21.15.116:0/4213342454 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f58a0015600 con 0x7f58a40c8750 2024-04-25T00:16:27.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.302+0000 7f5899ffb640 1 --2- 172.21.15.116:0/4213342454 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f58880678c0 0x7f5888069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:27.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.302+0000 7f5899ffb640 1 -- 172.21.15.116:0/4213342454 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 8472+0+0 (secure 0 0 0) 0x7f58a0087110 con 0x7f58a40c8750 2024-04-25T00:16:27.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.302+0000 7f58a89ec640 1 --2- 172.21.15.116:0/4213342454 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f58880678c0 0x7f5888069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:27.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.302+0000 7f58a89ec640 1 --2- 172.21.15.116:0/4213342454 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f58880678c0 0x7f5888069d80 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f5894000c10 tx=0x7f5894000ee0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:27.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.306+0000 7f5899ffb640 1 -- 172.21.15.116:0/4213342454 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f58a0019030 con 0x7f58a40c8750 2024-04-25T00:16:27.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.534+0000 7f58ab451640 1 -- 172.21.15.116:0/4213342454 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f58a40ad630 con 0x7f58a40c8750 2024-04-25T00:16:27.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.534+0000 7f5899ffb640 1 -- 172.21.15.116:0/4213342454 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) ==== 95+0+6509 (secure 0 0 0) 0x7f58a00571c0 con 0x7f58a40c8750 2024-04-25T00:16:27.534 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:16:27.534 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":117,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":5621,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3081407040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3081407040},{"type":"v1","addr":"172.21.15.174:6837","nonce":3081407040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:15:43.819845+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6810":{"gid":6810,"name":"c","rank":0,"incarnation":113,"state":"up:active","state_seq":13,"addr":"172.21.15.116:6833/2340581953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2340581953},{"type":"v1","addr":"172.21.15.116:6833","nonce":2340581953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6810,"qdb_cluster":[6810]},"id":1},{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:15:28.737458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6828":{"gid":6828,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":8,"addr":"172.21.15.116:6835/2012470875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2012470875},{"type":"v1","addr":"172.21.15.116:6835","nonce":2012470875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6828,"qdb_cluster":[6828]},"id":2}]} 2024-04-25T00:16:27.534 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 117 2024-04-25T00:16:27.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.538+0000 7f58ab451640 1 -- 172.21.15.116:0/4213342454 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f58880678c0 msgr2=0x7f5888069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:27.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.538+0000 7f58ab451640 1 --2- 172.21.15.116:0/4213342454 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f58880678c0 0x7f5888069d80 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f5894000c10 tx=0x7f5894000ee0 comp rx=0 tx=0).stop 2024-04-25T00:16:27.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.538+0000 7f58ab451640 1 -- 172.21.15.116:0/4213342454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58a40c8750 msgr2=0x7f58a41ee830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:27.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.538+0000 7f58ab451640 1 --2- 172.21.15.116:0/4213342454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58a40c8750 0x7f58a41ee830 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f58a0011600 tx=0x7f58a0011ac0 comp rx=0 tx=0).stop 2024-04-25T00:16:27.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.538+0000 7f58ab451640 1 -- 172.21.15.116:0/4213342454 shutdown_connections 2024-04-25T00:16:27.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.538+0000 7f58ab451640 1 --2- 172.21.15.116:0/4213342454 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f58880678c0 0x7f5888069d80 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:27.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.538+0000 7f58ab451640 1 --2- 172.21.15.116:0/4213342454 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f58a40c6c10 0x7f58a40ac630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:27.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.538+0000 7f58ab451640 1 --2- 172.21.15.116:0/4213342454 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f58a40c5c90 0x7f58a40ac0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:27.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.538+0000 7f58ab451640 1 --2- 172.21.15.116:0/4213342454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f58a40c8750 0x7f58a41ee830 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:27.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.538+0000 7f58ab451640 1 -- 172.21.15.116:0/4213342454 >> 172.21.15.116:0/4213342454 conn(0x7f58a406c720 msgr2=0x7f58a414bcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:27.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.538+0000 7f58ab451640 1 -- 172.21.15.116:0/4213342454 shutdown_connections 2024-04-25T00:16:27.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:27.538+0000 7f58ab451640 1 -- 172.21.15.116:0/4213342454 wait complete. 2024-04-25T00:16:27.557 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:16:29.558 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:16:29.682 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:16:29.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.710+0000 7f3e6759e640 1 Processor -- start 2024-04-25T00:16:29.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.710+0000 7f3e6759e640 1 -- start start 2024-04-25T00:16:29.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.710+0000 7f3e6759e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3e680c5ac0 0x7f3e68153c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:29.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.710+0000 7f3e6759e640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f3e680ab370 0x7f3e6815de30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:29.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.710+0000 7f3e6759e640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e68153780 con 0x7f3e68150dc0 2024-04-25T00:16:29.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.710+0000 7f3e6759e640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f3e680c20f0 con 0x7f3e680c5ac0 2024-04-25T00:16:29.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.710+0000 7f3e6759e640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f3e681511a0 con 0x7f3e680ab370 2024-04-25T00:16:29.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.710+0000 7f3e65d9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3e680c5ac0 0x7f3e68153c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:29.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.710+0000 7f3e65d9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3e680c5ac0 0x7f3e68153c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57822/0 (socket says 172.21.15.116:57822) 2024-04-25T00:16:29.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.710+0000 7f3e65d9b640 1 -- 172.21.15.116:0/22549002 learned_addr learned my addr 172.21.15.116:0/22549002 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:16:29.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.710+0000 7f3e65d9b640 1 -- 172.21.15.116:0/22549002 >> v2:172.21.15.174:3300/0 conn(0x7f3e680ab370 msgr2=0x7f3e6815de30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T00:16:29.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.710+0000 7f3e65d9b640 1 --2- 172.21.15.116:0/22549002 >> v2:172.21.15.174:3300/0 conn(0x7f3e680ab370 0x7f3e6815de30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:29.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.710+0000 7f3e65d9b640 1 -- 172.21.15.116:0/22549002 >> v1:172.21.15.174:6789/0 conn(0x7f3e68150dc0 legacy=0x7f3e680c2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:29.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.710+0000 7f3e65d9b640 1 -- 172.21.15.116:0/22549002 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e680c2340 con 0x7f3e680c5ac0 2024-04-25T00:16:29.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.714+0000 7f3e65d9b640 1 --2- 172.21.15.116:0/22549002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e680c5ac0 0x7f3e68153c80 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f3e680c57b0 tx=0x7f3e5c01c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8521f03759bde900 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:29.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.714+0000 7f3e6559a640 1 -- 172.21.15.116:0/22549002 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e5c0ab650 con 0x7f3e680c5ac0 2024-04-25T00:16:29.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.714+0000 7f3e6559a640 1 -- 172.21.15.116:0/22549002 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e5c0b0070 con 0x7f3e680c5ac0 2024-04-25T00:16:29.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.714+0000 7f3e6559a640 1 -- 172.21.15.116:0/22549002 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e5c0aba10 con 0x7f3e680c5ac0 2024-04-25T00:16:29.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.714+0000 7f3e6759e640 1 -- 172.21.15.116:0/22549002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e680c5ac0 msgr2=0x7f3e68153c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:29.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.714+0000 7f3e6759e640 1 --2- 172.21.15.116:0/22549002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e680c5ac0 0x7f3e68153c80 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f3e680c57b0 tx=0x7f3e5c01c540 comp rx=0 tx=0).stop 2024-04-25T00:16:29.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.714+0000 7f3e6759e640 1 -- 172.21.15.116:0/22549002 shutdown_connections 2024-04-25T00:16:29.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.714+0000 7f3e6759e640 1 --2- 172.21.15.116:0/22549002 >> v2:172.21.15.174:3300/0 conn(0x7f3e680ab370 0x7f3e6815de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:29.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.714+0000 7f3e6759e640 1 --2- 172.21.15.116:0/22549002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e680c5ac0 0x7f3e68153c80 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:29.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.714+0000 7f3e6759e640 1 -- 172.21.15.116:0/22549002 >> 172.21.15.116:0/22549002 conn(0x7f3e680a94e0 msgr2=0x7f3e680a98f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:29.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.714+0000 7f3e6759e640 1 -- 172.21.15.116:0/22549002 shutdown_connections 2024-04-25T00:16:29.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.714+0000 7f3e6759e640 1 -- 172.21.15.116:0/22549002 wait complete. 2024-04-25T00:16:29.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.718+0000 7f3e6759e640 1 Processor -- start 2024-04-25T00:16:29.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.718+0000 7f3e6759e640 1 -- start start 2024-04-25T00:16:29.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.718+0000 7f3e6759e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e680ab370 0x7f3e68148cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:29.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.718+0000 7f3e6759e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3e680c5ac0 0x7f3e680a7e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:29.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.718+0000 7f3e6759e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e68150dc0 0x7f3e68072060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:29.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.718+0000 7f3e6759e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3e680c33a0 con 0x7f3e680ab370 2024-04-25T00:16:29.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.718+0000 7f3e6759e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3e6806f3b0 con 0x7f3e68150dc0 2024-04-25T00:16:29.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.718+0000 7f3e6759e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f3e680c3520 con 0x7f3e680c5ac0 2024-04-25T00:16:29.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.718+0000 7f3e6659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e680ab370 0x7f3e68148cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:29.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.718+0000 7f3e6659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e680ab370 0x7f3e68148cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57838/0 (socket says 172.21.15.116:57838) 2024-04-25T00:16:29.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.718+0000 7f3e6659c640 1 -- 172.21.15.116:0/784547135 learned_addr learned my addr 172.21.15.116:0/784547135 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:16:29.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.718+0000 7f3e66d9d640 1 --2- 172.21.15.116:0/784547135 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e68150dc0 0x7f3e68072060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:29.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.718+0000 7f3e65d9b640 1 --2- 172.21.15.116:0/784547135 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3e680c5ac0 0x7f3e680a7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:29.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.718+0000 7f3e6659c640 1 -- 172.21.15.116:0/784547135 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3e680c5ac0 msgr2=0x7f3e680a7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:29.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.718+0000 7f3e6659c640 1 --2- 172.21.15.116:0/784547135 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3e680c5ac0 0x7f3e680a7e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:29.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.718+0000 7f3e6659c640 1 -- 172.21.15.116:0/784547135 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e68150dc0 msgr2=0x7f3e68072060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:29.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.718+0000 7f3e6659c640 1 --2- 172.21.15.116:0/784547135 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e68150dc0 0x7f3e68072060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:29.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.718+0000 7f3e6659c640 1 -- 172.21.15.116:0/784547135 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e680a7110 con 0x7f3e680ab370 2024-04-25T00:16:29.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.718+0000 7f3e66d9d640 1 --2- 172.21.15.116:0/784547135 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e68150dc0 0x7f3e68072060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:16:29.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.718+0000 7f3e6659c640 1 --2- 172.21.15.116:0/784547135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e680ab370 0x7f3e68148cd0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f3e5800c3c0 tx=0x7f3e5800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:29.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.718+0000 7f3e56ffd640 1 -- 172.21.15.116:0/784547135 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e58017070 con 0x7f3e680ab370 2024-04-25T00:16:29.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.718+0000 7f3e56ffd640 1 -- 172.21.15.116:0/784547135 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e580052d0 con 0x7f3e680ab370 2024-04-25T00:16:29.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.718+0000 7f3e6759e640 1 -- 172.21.15.116:0/784547135 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e681f2ca0 con 0x7f3e680ab370 2024-04-25T00:16:29.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.718+0000 7f3e56ffd640 1 -- 172.21.15.116:0/784547135 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e58003840 con 0x7f3e680ab370 2024-04-25T00:16:29.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.718+0000 7f3e6759e640 1 -- 172.21.15.116:0/784547135 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e680a8370 con 0x7f3e680ab370 2024-04-25T00:16:29.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.722+0000 7f3e56ffd640 1 -- 172.21.15.116:0/784547135 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3e5801a070 con 0x7f3e680ab370 2024-04-25T00:16:29.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.722+0000 7f3e6759e640 1 -- 172.21.15.116:0/784547135 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e680a6b90 con 0x7f3e680ab370 2024-04-25T00:16:29.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.722+0000 7f3e56ffd640 1 --2- 172.21.15.116:0/784547135 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3e480678c0 0x7f3e48069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:29.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.722+0000 7f3e56ffd640 1 -- 172.21.15.116:0/784547135 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 8472+0+0 (secure 0 0 0) 0x7f3e58089170 con 0x7f3e680ab370 2024-04-25T00:16:29.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.722+0000 7f3e65d9b640 1 --2- 172.21.15.116:0/784547135 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3e480678c0 0x7f3e48069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:29.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.722+0000 7f3e65d9b640 1 --2- 172.21.15.116:0/784547135 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3e480678c0 0x7f3e48069d80 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f3e5c0af000 tx=0x7f3e5c0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:29.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.726+0000 7f3e56ffd640 1 -- 172.21.15.116:0/784547135 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f3e58005480 con 0x7f3e680ab370 2024-04-25T00:16:29.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.814+0000 7f4b43988640 1 Processor -- start 2024-04-25T00:16:29.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.814+0000 7f4b43988640 1 -- start start 2024-04-25T00:16:29.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.814+0000 7f4b43988640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f4b3c153140 0x7f4b3c0c8450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:29.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.814+0000 7f4b43988640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4b3c0c8990 0x7f4b3c0c3d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:29.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.814+0000 7f4b43988640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b3c155c50 con 0x7f4b3c152d60 2024-04-25T00:16:29.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.814+0000 7f4b43988640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f4b3c155f30 con 0x7f4b3c0c8990 2024-04-25T00:16:29.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.814+0000 7f4b43988640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f4b3c156210 con 0x7f4b3c153140 2024-04-25T00:16:29.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.814+0000 7f4b41f25640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4b3c0c8990 0x7f4b3c0c3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:29.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.814+0000 7f4b41f25640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4b3c0c8990 0x7f4b3c0c3d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57854/0 (socket says 172.21.15.116:57854) 2024-04-25T00:16:29.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.814+0000 7f4b41724640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f4b3c152d60 0x7f4b3c0ab270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:47624/0 (socket says 172.21.15.116:47624) 2024-04-25T00:16:29.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.814+0000 7f4b41f25640 1 -- 172.21.15.116:0/3758100195 learned_addr learned my addr 172.21.15.116:0/3758100195 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:16:29.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.814+0000 7f4b40f23640 1 --2- 172.21.15.116:0/3758100195 >> v2:172.21.15.174:3300/0 conn(0x7f4b3c153140 0x7f4b3c0c8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:29.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.818+0000 7f4b2ffff640 1 -- 172.21.15.116:0/3758100195 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 628445892 0 0) 0x7f4b3c155c50 con 0x7f4b3c152d60 2024-04-25T00:16:29.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.818+0000 7f4b2ffff640 1 -- 172.21.15.116:0/3758100195 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b28003530 con 0x7f4b3c152d60 2024-04-25T00:16:29.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.818+0000 7f4b41f25640 1 -- 172.21.15.116:0/3758100195 >> v2:172.21.15.174:3300/0 conn(0x7f4b3c153140 msgr2=0x7f4b3c0c8450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:29.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.818+0000 7f4b41f25640 1 --2- 172.21.15.116:0/3758100195 >> v2:172.21.15.174:3300/0 conn(0x7f4b3c153140 0x7f4b3c0c8450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:29.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.818+0000 7f4b41f25640 1 -- 172.21.15.116:0/3758100195 >> v1:172.21.15.116:6789/0 conn(0x7f4b3c152d60 legacy=0x7f4b3c0ab270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:29.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.818+0000 7f4b41f25640 1 -- 172.21.15.116:0/3758100195 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b3c1564f0 con 0x7f4b3c0c8990 2024-04-25T00:16:29.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.818+0000 7f4b41f25640 1 --2- 172.21.15.116:0/3758100195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b3c0c8990 0x7f4b3c0c3d00 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c0c6ac0 tx=0x7f4b3801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e2f41f5a86ab632 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:29.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.818+0000 7f4b2ffff640 1 -- 172.21.15.116:0/3758100195 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b380b0070 con 0x7f4b3c0c8990 2024-04-25T00:16:29.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.818+0000 7f4b2ffff640 1 -- 172.21.15.116:0/3758100195 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b380ab600 con 0x7f4b3c0c8990 2024-04-25T00:16:29.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.818+0000 7f4b2ffff640 1 -- 172.21.15.116:0/3758100195 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b380ab9d0 con 0x7f4b3c0c8990 2024-04-25T00:16:29.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.818+0000 7f4b43988640 1 -- 172.21.15.116:0/3758100195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b3c0c8990 msgr2=0x7f4b3c0c3d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:29.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.818+0000 7f4b43988640 1 --2- 172.21.15.116:0/3758100195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b3c0c8990 0x7f4b3c0c3d00 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c0c6ac0 tx=0x7f4b3801c520 comp rx=0 tx=0).stop 2024-04-25T00:16:29.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.818+0000 7f4b43988640 1 -- 172.21.15.116:0/3758100195 shutdown_connections 2024-04-25T00:16:29.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.818+0000 7f4b43988640 1 --2- 172.21.15.116:0/3758100195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b3c0c8990 0x7f4b3c0c3d00 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:29.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.818+0000 7f4b43988640 1 --2- 172.21.15.116:0/3758100195 >> v2:172.21.15.174:3300/0 conn(0x7f4b3c153140 0x7f4b3c0c8450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:29.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.822+0000 7f4b43988640 1 -- 172.21.15.116:0/3758100195 >> 172.21.15.116:0/3758100195 conn(0x7f4b3c06f0c0 msgr2=0x7f4b3c06f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:29.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.822+0000 7f4b43988640 1 -- 172.21.15.116:0/3758100195 shutdown_connections 2024-04-25T00:16:29.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.822+0000 7f4b43988640 1 -- 172.21.15.116:0/3758100195 wait complete. 2024-04-25T00:16:29.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.822+0000 7f4b43988640 1 Processor -- start 2024-04-25T00:16:29.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.822+0000 7f4b43988640 1 -- start start 2024-04-25T00:16:29.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.822+0000 7f4b43988640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4b3c0c8990 0x7f4b3c0ad510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:29.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.822+0000 7f4b43988640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b3c152d60 0x7f4b3c14a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:29.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.822+0000 7f4b43988640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4b3c153140 0x7f4b3c14a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:29.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.822+0000 7f4b43988640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4b3c1623a0 con 0x7f4b3c152d60 2024-04-25T00:16:29.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.822+0000 7f4b43988640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4b3c155a10 con 0x7f4b3c153140 2024-04-25T00:16:29.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.822+0000 7f4b43988640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f4b3c162520 con 0x7f4b3c0c8990 2024-04-25T00:16:29.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.822+0000 7f4b40f23640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b3c152d60 0x7f4b3c14a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:29.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.822+0000 7f4b40f23640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b3c152d60 0x7f4b3c14a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57856/0 (socket says 172.21.15.116:57856) 2024-04-25T00:16:29.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.822+0000 7f4b40f23640 1 -- 172.21.15.116:0/848159288 learned_addr learned my addr 172.21.15.116:0/848159288 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:16:29.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.822+0000 7f4b41724640 1 --2- 172.21.15.116:0/848159288 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4b3c0c8990 0x7f4b3c0ad510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:29.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.822+0000 7f4b41f25640 1 --2- 172.21.15.116:0/848159288 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4b3c153140 0x7f4b3c14a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:29.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.822+0000 7f4b40f23640 1 -- 172.21.15.116:0/848159288 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4b3c0c8990 msgr2=0x7f4b3c0ad510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:29.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.822+0000 7f4b40f23640 1 --2- 172.21.15.116:0/848159288 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4b3c0c8990 0x7f4b3c0ad510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:29.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.822+0000 7f4b40f23640 1 -- 172.21.15.116:0/848159288 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4b3c153140 msgr2=0x7f4b3c14a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:29.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.822+0000 7f4b40f23640 1 --2- 172.21.15.116:0/848159288 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4b3c153140 0x7f4b3c14a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:29.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.822+0000 7f4b40f23640 1 -- 172.21.15.116:0/848159288 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b3c149c70 con 0x7f4b3c152d60 2024-04-25T00:16:29.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.822+0000 7f4b40f23640 1 --2- 172.21.15.116:0/848159288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b3c152d60 0x7f4b3c14a190 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f4b3000e930 tx=0x7f4b3000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:29.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.822+0000 7f4b2dffb640 1 -- 172.21.15.116:0/848159288 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b30017070 con 0x7f4b3c152d60 2024-04-25T00:16:29.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.822+0000 7f4b2dffb640 1 -- 172.21.15.116:0/848159288 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b3000ca00 con 0x7f4b3c152d60 2024-04-25T00:16:29.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.822+0000 7f4b43988640 1 -- 172.21.15.116:0/848159288 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b3c1f1c40 con 0x7f4b3c152d60 2024-04-25T00:16:29.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.822+0000 7f4b2dffb640 1 -- 172.21.15.116:0/848159288 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b3000cdb0 con 0x7f4b3c152d60 2024-04-25T00:16:29.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.822+0000 7f4b43988640 1 -- 172.21.15.116:0/848159288 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b3c1f2200 con 0x7f4b3c152d60 2024-04-25T00:16:29.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.826+0000 7f4b43988640 1 -- 172.21.15.116:0/848159288 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b3c068a60 con 0x7f4b3c152d60 2024-04-25T00:16:29.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.826+0000 7f4b2dffb640 1 -- 172.21.15.116:0/848159288 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4b3001a070 con 0x7f4b3c152d60 2024-04-25T00:16:29.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.826+0000 7f4b2dffb640 1 --2- 172.21.15.116:0/848159288 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4b20067910 0x7f4b20069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:29.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.826+0000 7f4b2dffb640 1 -- 172.21.15.116:0/848159288 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 8472+0+0 (secure 0 0 0) 0x7f4b30086790 con 0x7f4b3c152d60 2024-04-25T00:16:29.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.826+0000 7f4b41724640 1 --2- 172.21.15.116:0/848159288 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4b20067910 0x7f4b20069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:29.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.826+0000 7f4b41724640 1 --2- 172.21.15.116:0/848159288 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4b20067910 0x7f4b20069dd0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f4b34007fd0 tx=0x7f4b34007ea0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:29.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.830+0000 7f4b2dffb640 1 -- 172.21.15.116:0/848159288 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f4b300568d0 con 0x7f4b3c152d60 2024-04-25T00:16:29.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.966+0000 7f3e6759e640 1 -- 172.21.15.116:0/784547135 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3e6804f510 con 0x7f3e680ab370 2024-04-25T00:16:29.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.966+0000 7f3e56ffd640 1 -- 172.21.15.116:0/784547135 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) ==== 95+0+6509 (secure 0 0 0) 0x7f3e580557f0 con 0x7f3e680ab370 2024-04-25T00:16:29.964 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:16:29.964 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":117,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":5621,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3081407040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3081407040},{"type":"v1","addr":"172.21.15.174:6837","nonce":3081407040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:15:43.819845+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6810":{"gid":6810,"name":"c","rank":0,"incarnation":113,"state":"up:active","state_seq":13,"addr":"172.21.15.116:6833/2340581953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2340581953},{"type":"v1","addr":"172.21.15.116:6833","nonce":2340581953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6810,"qdb_cluster":[6810]},"id":1},{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:15:28.737458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6828":{"gid":6828,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":8,"addr":"172.21.15.116:6835/2012470875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2012470875},{"type":"v1","addr":"172.21.15.116:6835","nonce":2012470875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6828,"qdb_cluster":[6828]},"id":2}]} 2024-04-25T00:16:29.965 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 117 2024-04-25T00:16:29.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.966+0000 7f3e6759e640 1 -- 172.21.15.116:0/784547135 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3e480678c0 msgr2=0x7f3e48069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:29.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.966+0000 7f3e6759e640 1 --2- 172.21.15.116:0/784547135 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3e480678c0 0x7f3e48069d80 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f3e5c0af000 tx=0x7f3e5c0a7000 comp rx=0 tx=0).stop 2024-04-25T00:16:29.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.966+0000 7f3e6759e640 1 -- 172.21.15.116:0/784547135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e680ab370 msgr2=0x7f3e68148cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:29.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.966+0000 7f3e6759e640 1 --2- 172.21.15.116:0/784547135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e680ab370 0x7f3e68148cd0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f3e5800c3c0 tx=0x7f3e5800c880 comp rx=0 tx=0).stop 2024-04-25T00:16:29.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.966+0000 7f3e6759e640 1 -- 172.21.15.116:0/784547135 shutdown_connections 2024-04-25T00:16:29.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.966+0000 7f3e6759e640 1 --2- 172.21.15.116:0/784547135 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3e480678c0 0x7f3e48069d80 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:29.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.966+0000 7f3e6759e640 1 --2- 172.21.15.116:0/784547135 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3e68150dc0 0x7f3e68072060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:29.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.966+0000 7f3e6759e640 1 --2- 172.21.15.116:0/784547135 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3e680c5ac0 0x7f3e680a7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:29.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.966+0000 7f3e6759e640 1 --2- 172.21.15.116:0/784547135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3e680ab370 0x7f3e68148cd0 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:29.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.966+0000 7f3e6759e640 1 -- 172.21.15.116:0/784547135 >> 172.21.15.116:0/784547135 conn(0x7f3e680a94e0 msgr2=0x7f3e6806c450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:29.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.966+0000 7f3e6759e640 1 -- 172.21.15.116:0/784547135 shutdown_connections 2024-04-25T00:16:29.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:29.966+0000 7f3e6759e640 1 -- 172.21.15.116:0/784547135 wait complete. 2024-04-25T00:16:29.985 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:16:30.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.054+0000 7f4b43988640 1 -- 172.21.15.116:0/848159288 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4b3c067540 con 0x7f4b3c152d60 2024-04-25T00:16:30.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.054+0000 7f4b2dffb640 1 -- 172.21.15.116:0/848159288 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) ==== 95+0+6509 (secure 0 0 0) 0x7f4b300561d0 con 0x7f4b3c152d60 2024-04-25T00:16:30.055 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:16:30.055 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":117,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":5621,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3081407040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3081407040},{"type":"v1","addr":"172.21.15.174:6837","nonce":3081407040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:15:43.819845+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6810":{"gid":6810,"name":"c","rank":0,"incarnation":113,"state":"up:active","state_seq":13,"addr":"172.21.15.116:6833/2340581953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2340581953},{"type":"v1","addr":"172.21.15.116:6833","nonce":2340581953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6810,"qdb_cluster":[6810]},"id":1},{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:15:28.737458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6828":{"gid":6828,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":8,"addr":"172.21.15.116:6835/2012470875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2012470875},{"type":"v1","addr":"172.21.15.116:6835","nonce":2012470875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6828,"qdb_cluster":[6828]},"id":2}]} 2024-04-25T00:16:30.055 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 117 2024-04-25T00:16:30.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.058+0000 7f4b43988640 1 -- 172.21.15.116:0/848159288 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4b20067910 msgr2=0x7f4b20069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:30.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.058+0000 7f4b43988640 1 --2- 172.21.15.116:0/848159288 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4b20067910 0x7f4b20069dd0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f4b34007fd0 tx=0x7f4b34007ea0 comp rx=0 tx=0).stop 2024-04-25T00:16:30.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.058+0000 7f4b43988640 1 -- 172.21.15.116:0/848159288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b3c152d60 msgr2=0x7f4b3c14a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:30.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.058+0000 7f4b43988640 1 --2- 172.21.15.116:0/848159288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b3c152d60 0x7f4b3c14a190 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f4b3000e930 tx=0x7f4b3000edf0 comp rx=0 tx=0).stop 2024-04-25T00:16:30.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.058+0000 7f4b43988640 1 -- 172.21.15.116:0/848159288 shutdown_connections 2024-04-25T00:16:30.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.058+0000 7f4b43988640 1 --2- 172.21.15.116:0/848159288 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4b20067910 0x7f4b20069dd0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:30.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.058+0000 7f4b43988640 1 --2- 172.21.15.116:0/848159288 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4b3c153140 0x7f4b3c14a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:30.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.058+0000 7f4b43988640 1 --2- 172.21.15.116:0/848159288 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b3c152d60 0x7f4b3c14a190 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:30.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.058+0000 7f4b43988640 1 --2- 172.21.15.116:0/848159288 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4b3c0c8990 0x7f4b3c0ad510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:30.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.058+0000 7f4b43988640 1 -- 172.21.15.116:0/848159288 >> 172.21.15.116:0/848159288 conn(0x7f4b3c06f0c0 msgr2=0x7f4b3c154ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:30.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.058+0000 7f4b43988640 1 -- 172.21.15.116:0/848159288 shutdown_connections 2024-04-25T00:16:30.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.058+0000 7f4b43988640 1 -- 172.21.15.116:0/848159288 wait complete. 2024-04-25T00:16:30.074 INFO:tasks.mds_thrash.fs.[a]:kill mds.c (rank=0) 2024-04-25T00:16:30.074 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-25T00:16:30.074 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:16:30.085 INFO:tasks.ceph.mds.c:Stopped 2024-04-25T00:16:30.085 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:16:30.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.242+0000 7fe996ac1640 1 Processor -- start 2024-04-25T00:16:30.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.242+0000 7fe996ac1640 1 -- start start 2024-04-25T00:16:30.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.242+0000 7fe996ac1640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fe990150bf0 0x7fe99006c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:30.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.242+0000 7fe996ac1640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe99006c760 0x7fe99006f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:30.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.242+0000 7fe996ac1640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe990153aa0 con 0x7fe990153570 2024-04-25T00:16:30.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.242+0000 7fe996ac1640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fe99006e630 con 0x7fe990150bf0 2024-04-25T00:16:30.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.242+0000 7fe996ac1640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe990150fd0 con 0x7fe99006c760 2024-04-25T00:16:30.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.242+0000 7fe99485d640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fe990150bf0 0x7fe99006c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:30.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.242+0000 7fe99485d640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fe990150bf0 0x7fe99006c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:46956/0 (socket says 172.21.15.116:46956) 2024-04-25T00:16:30.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.242+0000 7fe98ffff640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fe990153570 0x7fe99006eac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:39136/0 (socket says 172.21.15.116:39136) 2024-04-25T00:16:30.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.242+0000 7fe99485d640 1 -- 172.21.15.116:0/4152731762 learned_addr learned my addr 172.21.15.116:0/4152731762 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:16:30.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.242+0000 7fe99505e640 1 --2- 172.21.15.116:0/4152731762 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe99006c760 0x7fe99006f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:30.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.242+0000 7fe99485d640 1 -- 172.21.15.116:0/4152731762 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe99006c760 msgr2=0x7fe99006f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:30.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.242+0000 7fe99485d640 1 --2- 172.21.15.116:0/4152731762 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe99006c760 0x7fe99006f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:30.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.242+0000 7fe99485d640 1 -- 172.21.15.116:0/4152731762 >> v1:172.21.15.174:6789/0 conn(0x7fe990153570 legacy=0x7fe99006eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:30.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.242+0000 7fe99485d640 1 -- 172.21.15.116:0/4152731762 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe990153d80 con 0x7fe990150bf0 2024-04-25T00:16:30.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.246+0000 7fe99485d640 1 --2- 172.21.15.116:0/4152731762 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe990150bf0 0x7fe99006c220 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fe984009870 tx=0x7fe98401c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9cfb49d2f76cbc8f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:30.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.246+0000 7fe98f7fe640 1 -- 172.21.15.116:0/4152731762 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9840aa600 con 0x7fe990150bf0 2024-04-25T00:16:30.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.246+0000 7fe98f7fe640 1 -- 172.21.15.116:0/4152731762 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9840af070 con 0x7fe990150bf0 2024-04-25T00:16:30.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.246+0000 7fe98f7fe640 1 -- 172.21.15.116:0/4152731762 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9840aa9c0 con 0x7fe990150bf0 2024-04-25T00:16:30.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.246+0000 7fe996ac1640 1 -- 172.21.15.116:0/4152731762 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe990150bf0 msgr2=0x7fe99006c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:30.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.246+0000 7fe996ac1640 1 --2- 172.21.15.116:0/4152731762 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe990150bf0 0x7fe99006c220 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fe984009870 tx=0x7fe98401c4a0 comp rx=0 tx=0).stop 2024-04-25T00:16:30.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.246+0000 7fe996ac1640 1 -- 172.21.15.116:0/4152731762 shutdown_connections 2024-04-25T00:16:30.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.246+0000 7fe996ac1640 1 --2- 172.21.15.116:0/4152731762 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe99006c760 0x7fe99006f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:30.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.246+0000 7fe996ac1640 1 --2- 172.21.15.116:0/4152731762 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe990150bf0 0x7fe99006c220 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:30.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.246+0000 7fe996ac1640 1 -- 172.21.15.116:0/4152731762 >> 172.21.15.116:0/4152731762 conn(0x7fe9900c4270 msgr2=0x7fe9900c4680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:30.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.246+0000 7fe996ac1640 1 -- 172.21.15.116:0/4152731762 shutdown_connections 2024-04-25T00:16:30.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.246+0000 7fe996ac1640 1 -- 172.21.15.116:0/4152731762 wait complete. 2024-04-25T00:16:30.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.250+0000 7fe996ac1640 1 Processor -- start 2024-04-25T00:16:30.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.250+0000 7fe996ac1640 1 -- start start 2024-04-25T00:16:30.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.250+0000 7fe996ac1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe99006c760 0x7fe9901ef330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:30.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.250+0000 7fe996ac1640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe990150bf0 0x7fe99015c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:30.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.250+0000 7fe996ac1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe990153570 0x7fe99014f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:30.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.250+0000 7fe996ac1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe9900a72b0 con 0x7fe99006c760 2024-04-25T00:16:30.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.250+0000 7fe996ac1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe9900a7130 con 0x7fe990153570 2024-04-25T00:16:30.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.250+0000 7fe996ac1640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe990160710 con 0x7fe990150bf0 2024-04-25T00:16:30.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.250+0000 7fe99485d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe99006c760 0x7fe9901ef330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:30.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.250+0000 7fe99485d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe99006c760 0x7fe9901ef330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57860/0 (socket says 172.21.15.116:57860) 2024-04-25T00:16:30.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.250+0000 7fe99485d640 1 -- 172.21.15.116:0/4232079243 learned_addr learned my addr 172.21.15.116:0/4232079243 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:16:30.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.250+0000 7fe99505e640 1 --2- 172.21.15.116:0/4232079243 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe990153570 0x7fe99014f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:30.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.250+0000 7fe98ffff640 1 --2- 172.21.15.116:0/4232079243 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe990150bf0 0x7fe99015c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:30.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.250+0000 7fe99485d640 1 -- 172.21.15.116:0/4232079243 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe990150bf0 msgr2=0x7fe99015c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:30.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.250+0000 7fe99485d640 1 --2- 172.21.15.116:0/4232079243 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe990150bf0 0x7fe99015c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:30.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.250+0000 7fe99485d640 1 -- 172.21.15.116:0/4232079243 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe990153570 msgr2=0x7fe99014f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:30.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.250+0000 7fe99485d640 1 --2- 172.21.15.116:0/4232079243 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe990153570 0x7fe99014f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:30.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.250+0000 7fe99485d640 1 -- 172.21.15.116:0/4232079243 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe99014c710 con 0x7fe99006c760 2024-04-25T00:16:30.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.250+0000 7fe98ffff640 1 --2- 172.21.15.116:0/4232079243 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe990150bf0 0x7fe99015c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:16:30.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.250+0000 7fe99505e640 1 --2- 172.21.15.116:0/4232079243 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe990153570 0x7fe99014f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:16:30.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.250+0000 7fe99485d640 1 --2- 172.21.15.116:0/4232079243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe99006c760 0x7fe9901ef330 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fe9840ac040 tx=0x7fe98401ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:30.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.250+0000 7fe98d7fa640 1 -- 172.21.15.116:0/4232079243 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9840bd5f0 con 0x7fe99006c760 2024-04-25T00:16:30.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.250+0000 7fe98d7fa640 1 -- 172.21.15.116:0/4232079243 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9840bd7b0 con 0x7fe99006c760 2024-04-25T00:16:30.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.250+0000 7fe996ac1640 1 -- 172.21.15.116:0/4232079243 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe99014c150 con 0x7fe99006c760 2024-04-25T00:16:30.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.250+0000 7fe98d7fa640 1 -- 172.21.15.116:0/4232079243 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9840bdba0 con 0x7fe99006c760 2024-04-25T00:16:30.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.250+0000 7fe996ac1640 1 -- 172.21.15.116:0/4232079243 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9901eeb30 con 0x7fe99006c760 2024-04-25T00:16:30.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.250+0000 7fe996ac1640 1 -- 172.21.15.116:0/4232079243 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe990068a60 con 0x7fe99006c760 2024-04-25T00:16:30.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.254+0000 7fe98d7fa640 1 -- 172.21.15.116:0/4232079243 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe9840a9070 con 0x7fe99006c760 2024-04-25T00:16:30.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.254+0000 7fe98d7fa640 1 --2- 172.21.15.116:0/4232079243 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe9740678f0 0x7fe974069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:30.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.254+0000 7fe98d7fa640 1 -- 172.21.15.116:0/4232079243 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 8472+0+0 (secure 0 0 0) 0x7fe984120e20 con 0x7fe99006c760 2024-04-25T00:16:30.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.254+0000 7fe98ffff640 1 --2- 172.21.15.116:0/4232079243 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe9740678f0 0x7fe974069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:30.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.254+0000 7fe98ffff640 1 --2- 172.21.15.116:0/4232079243 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe9740678f0 0x7fe974069db0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fe97c007fd0 tx=0x7fe97c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:30.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.258+0000 7fe98d7fa640 1 -- 172.21.15.116:0/4232079243 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fe99014c710 con 0x7fe99006c760 2024-04-25T00:16:30.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.478+0000 7fe996ac1640 1 -- 172.21.15.116:0/4232079243 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe990074aa0 con 0x7fe99006c760 2024-04-25T00:16:30.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.478+0000 7fe98d7fa640 1 -- 172.21.15.116:0/4232079243 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) ==== 95+0+6509 (secure 0 0 0) 0x7fe984016150 con 0x7fe99006c760 2024-04-25T00:16:30.477 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:16:30.478 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":117,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":5621,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3081407040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3081407040},{"type":"v1","addr":"172.21.15.174:6837","nonce":3081407040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:15:43.819845+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6810":{"gid":6810,"name":"c","rank":0,"incarnation":113,"state":"up:active","state_seq":13,"addr":"172.21.15.116:6833/2340581953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2340581953},{"type":"v1","addr":"172.21.15.116:6833","nonce":2340581953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6810,"qdb_cluster":[6810]},"id":1},{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:15:28.737458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6828":{"gid":6828,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":8,"addr":"172.21.15.116:6835/2012470875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2012470875},{"type":"v1","addr":"172.21.15.116:6835","nonce":2012470875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6828,"qdb_cluster":[6828]},"id":2}]} 2024-04-25T00:16:30.478 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 117 2024-04-25T00:16:30.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.478+0000 7fe996ac1640 1 -- 172.21.15.116:0/4232079243 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe9740678f0 msgr2=0x7fe974069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:30.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.478+0000 7fe996ac1640 1 --2- 172.21.15.116:0/4232079243 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe9740678f0 0x7fe974069db0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fe97c007fd0 tx=0x7fe97c009040 comp rx=0 tx=0).stop 2024-04-25T00:16:30.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.478+0000 7fe996ac1640 1 -- 172.21.15.116:0/4232079243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe99006c760 msgr2=0x7fe9901ef330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:30.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.478+0000 7fe996ac1640 1 --2- 172.21.15.116:0/4232079243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe99006c760 0x7fe9901ef330 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fe9840ac040 tx=0x7fe98401ace0 comp rx=0 tx=0).stop 2024-04-25T00:16:30.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.478+0000 7fe996ac1640 1 -- 172.21.15.116:0/4232079243 shutdown_connections 2024-04-25T00:16:30.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.478+0000 7fe996ac1640 1 --2- 172.21.15.116:0/4232079243 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe9740678f0 0x7fe974069db0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:30.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.478+0000 7fe996ac1640 1 --2- 172.21.15.116:0/4232079243 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe990153570 0x7fe99014f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:30.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.482+0000 7fe996ac1640 1 --2- 172.21.15.116:0/4232079243 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe990150bf0 0x7fe99015c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:30.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.482+0000 7fe996ac1640 1 --2- 172.21.15.116:0/4232079243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe99006c760 0x7fe9901ef330 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:30.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.482+0000 7fe996ac1640 1 -- 172.21.15.116:0/4232079243 >> 172.21.15.116:0/4232079243 conn(0x7fe9900c4270 msgr2=0x7fe990146e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:30.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.482+0000 7fe996ac1640 1 -- 172.21.15.116:0/4232079243 shutdown_connections 2024-04-25T00:16:30.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:30.482+0000 7fe996ac1640 1 -- 172.21.15.116:0/4232079243 wait complete. 2024-04-25T00:16:30.497 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:16:31.985 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:16:32.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.114+0000 7f10531a6640 1 Processor -- start 2024-04-25T00:16:32.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.114+0000 7f10531a6640 1 -- start start 2024-04-25T00:16:32.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.118+0000 7f10531a6640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f104c13e310 0x7f104c0c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:32.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.118+0000 7f10531a6640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f104c0c2f50 0x7f104c1559c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:32.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.118+0000 7f10531a6640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f104c13ed60 0x7f104c155f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:32.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.118+0000 7f10531a6640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f104c070d40 con 0x7f104c0c2f50 2024-04-25T00:16:32.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.118+0000 7f10531a6640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f104c071020 con 0x7f104c13e310 2024-04-25T00:16:32.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.118+0000 7f10531a6640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f104c06d200 con 0x7f104c13ed60 2024-04-25T00:16:32.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.118+0000 7f104bfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f104c0c2f50 0x7f104c1559c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:32.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.118+0000 7f104bfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f104c0c2f50 0x7f104c1559c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57876/0 (socket says 172.21.15.116:57876) 2024-04-25T00:16:32.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.118+0000 7f104bfff640 1 -- 172.21.15.116:0/2969970385 learned_addr learned my addr 172.21.15.116:0/2969970385 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:16:32.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.118+0000 7f1050f42640 1 --2- 172.21.15.116:0/2969970385 >> v2:172.21.15.174:3300/0 conn(0x7f104c13e310 0x7f104c0c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:32.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.118+0000 7f1051743640 1 --2- 172.21.15.116:0/2969970385 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f104c13ed60 0x7f104c155f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:32.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.118+0000 7f104bfff640 1 -- 172.21.15.116:0/2969970385 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f104c13ed60 msgr2=0x7f104c155f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:32.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.118+0000 7f104bfff640 1 --2- 172.21.15.116:0/2969970385 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f104c13ed60 0x7f104c155f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:32.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.118+0000 7f104bfff640 1 -- 172.21.15.116:0/2969970385 >> v2:172.21.15.174:3300/0 conn(0x7f104c13e310 msgr2=0x7f104c0c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:32.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.118+0000 7f104bfff640 1 --2- 172.21.15.116:0/2969970385 >> v2:172.21.15.174:3300/0 conn(0x7f104c13e310 0x7f104c0c2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:32.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.118+0000 7f104bfff640 1 -- 172.21.15.116:0/2969970385 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f104c06cb50 con 0x7f104c0c2f50 2024-04-25T00:16:32.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.118+0000 7f104bfff640 1 --2- 172.21.15.116:0/2969970385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f104c0c2f50 0x7f104c1559c0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f104c070620 tx=0x7f104001c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b9407bbd8c1ebf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:32.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.118+0000 7f104b7fe640 1 -- 172.21.15.116:0/2969970385 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10400b0070 con 0x7f104c0c2f50 2024-04-25T00:16:32.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.118+0000 7f104b7fe640 1 -- 172.21.15.116:0/2969970385 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10400ab600 con 0x7f104c0c2f50 2024-04-25T00:16:32.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.118+0000 7f104b7fe640 1 -- 172.21.15.116:0/2969970385 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10400ab9d0 con 0x7f104c0c2f50 2024-04-25T00:16:32.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.122+0000 7f10531a6640 1 -- 172.21.15.116:0/2969970385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f104c0c2f50 msgr2=0x7f104c1559c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:32.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.122+0000 7f10531a6640 1 --2- 172.21.15.116:0/2969970385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f104c0c2f50 0x7f104c1559c0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f104c070620 tx=0x7f104001c400 comp rx=0 tx=0).stop 2024-04-25T00:16:32.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.122+0000 7f10531a6640 1 -- 172.21.15.116:0/2969970385 shutdown_connections 2024-04-25T00:16:32.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.122+0000 7f10531a6640 1 --2- 172.21.15.116:0/2969970385 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f104c13ed60 0x7f104c155f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:32.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.122+0000 7f10531a6640 1 --2- 172.21.15.116:0/2969970385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f104c0c2f50 0x7f104c1559c0 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:32.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.122+0000 7f10531a6640 1 --2- 172.21.15.116:0/2969970385 >> v2:172.21.15.174:3300/0 conn(0x7f104c13e310 0x7f104c0c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:32.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.122+0000 7f10531a6640 1 -- 172.21.15.116:0/2969970385 >> 172.21.15.116:0/2969970385 conn(0x7f104c0c3b00 msgr2=0x7f104c0c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:32.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.122+0000 7f10531a6640 1 -- 172.21.15.116:0/2969970385 shutdown_connections 2024-04-25T00:16:32.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.122+0000 7f10531a6640 1 -- 172.21.15.116:0/2969970385 wait complete. 2024-04-25T00:16:32.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.122+0000 7f10531a6640 1 Processor -- start 2024-04-25T00:16:32.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.122+0000 7f10531a6640 1 -- start start 2024-04-25T00:16:32.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.122+0000 7f10531a6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f104c0c2f50 0x7f104c0aa080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:32.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.122+0000 7f10531a6640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f104c13e310 0x7f104c152920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:32.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.122+0000 7f10531a6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f104c13ed60 0x7f104c152e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:32.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.122+0000 7f10531a6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f104c159650 con 0x7f104c13ed60 2024-04-25T00:16:32.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.122+0000 7f10531a6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f104c156650 con 0x7f104c0c2f50 2024-04-25T00:16:32.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.122+0000 7f10531a6640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f104c1597d0 con 0x7f104c13e310 2024-04-25T00:16:32.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.122+0000 7f1051743640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f104c13ed60 0x7f104c152e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:32.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.122+0000 7f1051743640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f104c13ed60 0x7f104c152e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57886/0 (socket says 172.21.15.116:57886) 2024-04-25T00:16:32.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.122+0000 7f1051743640 1 -- 172.21.15.116:0/812596272 learned_addr learned my addr 172.21.15.116:0/812596272 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:16:32.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.122+0000 7f1050f42640 1 --2- 172.21.15.116:0/812596272 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f104c0c2f50 0x7f104c0aa080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:32.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.122+0000 7f104bfff640 1 --2- 172.21.15.116:0/812596272 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f104c13e310 0x7f104c152920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:32.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.122+0000 7f1051743640 1 -- 172.21.15.116:0/812596272 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f104c13e310 msgr2=0x7f104c152920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:32.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.122+0000 7f1051743640 1 --2- 172.21.15.116:0/812596272 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f104c13e310 0x7f104c152920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:32.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.122+0000 7f1051743640 1 -- 172.21.15.116:0/812596272 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f104c0c2f50 msgr2=0x7f104c0aa080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:32.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.122+0000 7f1051743640 1 --2- 172.21.15.116:0/812596272 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f104c0c2f50 0x7f104c0aa080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:32.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.122+0000 7f1051743640 1 -- 172.21.15.116:0/812596272 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f104c0c67a0 con 0x7f104c13ed60 2024-04-25T00:16:32.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.122+0000 7f104bfff640 1 --2- 172.21.15.116:0/812596272 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f104c13e310 0x7f104c152920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:16:32.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.122+0000 7f1051743640 1 --2- 172.21.15.116:0/812596272 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f104c13ed60 0x7f104c152e60 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f1044007e90 tx=0x7f104400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:32.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.122+0000 7f10497fa640 1 -- 172.21.15.116:0/812596272 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1044014420 con 0x7f104c13ed60 2024-04-25T00:16:32.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.122+0000 7f10497fa640 1 -- 172.21.15.116:0/812596272 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1044019070 con 0x7f104c13ed60 2024-04-25T00:16:32.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.122+0000 7f10531a6640 1 -- 172.21.15.116:0/812596272 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f104c0c6a80 con 0x7f104c13ed60 2024-04-25T00:16:32.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.122+0000 7f10497fa640 1 -- 172.21.15.116:0/812596272 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1044014800 con 0x7f104c13ed60 2024-04-25T00:16:32.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.122+0000 7f10531a6640 1 -- 172.21.15.116:0/812596272 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f104c0c5a10 con 0x7f104c13ed60 2024-04-25T00:16:32.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.122+0000 7f10531a6640 1 -- 172.21.15.116:0/812596272 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f104c068350 con 0x7f104c13ed60 2024-04-25T00:16:32.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.126+0000 7f10497fa640 1 -- 172.21.15.116:0/812596272 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f10440149a0 con 0x7f104c13ed60 2024-04-25T00:16:32.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.126+0000 7f10497fa640 1 --2- 172.21.15.116:0/812596272 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f10380678c0 0x7f1038069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:32.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.126+0000 7f10497fa640 1 -- 172.21.15.116:0/812596272 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 8472+0+0 (secure 0 0 0) 0x7f1044086fc0 con 0x7f104c13ed60 2024-04-25T00:16:32.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.126+0000 7f1050f42640 1 --2- 172.21.15.116:0/812596272 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f10380678c0 0x7f1038069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:32.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.126+0000 7f1050f42640 1 --2- 172.21.15.116:0/812596272 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f10380678c0 0x7f1038069d80 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f103c00ba30 tx=0x7f103c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:32.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.130+0000 7f10497fa640 1 -- 172.21.15.116:0/812596272 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f1044057010 con 0x7f104c13ed60 2024-04-25T00:16:32.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.322+0000 7f10531a6640 1 -- 172.21.15.116:0/812596272 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f104c0c34f0 con 0x7f104c13ed60 2024-04-25T00:16:32.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.326+0000 7f10497fa640 1 -- 172.21.15.116:0/812596272 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) ==== 95+0+6509 (secure 0 0 0) 0x7f1044056910 con 0x7f104c13ed60 2024-04-25T00:16:32.324 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:16:32.324 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":117,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":5621,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3081407040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3081407040},{"type":"v1","addr":"172.21.15.174:6837","nonce":3081407040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:15:43.819845+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6810":{"gid":6810,"name":"c","rank":0,"incarnation":113,"state":"up:active","state_seq":13,"addr":"172.21.15.116:6833/2340581953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2340581953},{"type":"v1","addr":"172.21.15.116:6833","nonce":2340581953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6810,"qdb_cluster":[6810]},"id":1},{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:15:28.737458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6828":{"gid":6828,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":8,"addr":"172.21.15.116:6835/2012470875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2012470875},{"type":"v1","addr":"172.21.15.116:6835","nonce":2012470875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6828,"qdb_cluster":[6828]},"id":2}]} 2024-04-25T00:16:32.325 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 117 2024-04-25T00:16:32.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.326+0000 7f10531a6640 1 -- 172.21.15.116:0/812596272 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f10380678c0 msgr2=0x7f1038069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:32.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.326+0000 7f10531a6640 1 --2- 172.21.15.116:0/812596272 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f10380678c0 0x7f1038069d80 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f103c00ba30 tx=0x7f103c008040 comp rx=0 tx=0).stop 2024-04-25T00:16:32.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.326+0000 7f10531a6640 1 -- 172.21.15.116:0/812596272 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f104c13ed60 msgr2=0x7f104c152e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:32.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.326+0000 7f10531a6640 1 --2- 172.21.15.116:0/812596272 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f104c13ed60 0x7f104c152e60 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f1044007e90 tx=0x7f104400e4b0 comp rx=0 tx=0).stop 2024-04-25T00:16:32.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.326+0000 7f10531a6640 1 -- 172.21.15.116:0/812596272 shutdown_connections 2024-04-25T00:16:32.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.326+0000 7f10531a6640 1 --2- 172.21.15.116:0/812596272 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f10380678c0 0x7f1038069d80 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:32.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.326+0000 7f10531a6640 1 --2- 172.21.15.116:0/812596272 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f104c13ed60 0x7f104c152e60 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:32.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.326+0000 7f10531a6640 1 --2- 172.21.15.116:0/812596272 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f104c13e310 0x7f104c152920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:32.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.326+0000 7f10531a6640 1 --2- 172.21.15.116:0/812596272 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f104c0c2f50 0x7f104c0aa080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:32.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.326+0000 7f10531a6640 1 -- 172.21.15.116:0/812596272 >> 172.21.15.116:0/812596272 conn(0x7f104c0c3b00 msgr2=0x7f104c06d950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:32.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.326+0000 7f10531a6640 1 -- 172.21.15.116:0/812596272 shutdown_connections 2024-04-25T00:16:32.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.326+0000 7f10531a6640 1 -- 172.21.15.116:0/812596272 wait complete. 2024-04-25T00:16:32.343 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:16:32.498 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:16:32.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.626+0000 7f56c35cc640 1 Processor -- start 2024-04-25T00:16:32.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.626+0000 7f56c35cc640 1 -- start start 2024-04-25T00:16:32.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.626+0000 7f56c35cc640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f56bc14d6e0 0x7f56bc153ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:32.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.626+0000 7f56c35cc640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56bc06b210 0x7f56bc06b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:32.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.626+0000 7f56c35cc640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56bc156c30 con 0x7f56bc1570e0 2024-04-25T00:16:32.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.626+0000 7f56c35cc640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f56bc154530 con 0x7f56bc14d6e0 2024-04-25T00:16:32.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.626+0000 7f56c35cc640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f56bc0702a0 con 0x7f56bc06b210 2024-04-25T00:16:32.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.626+0000 7f56c1368640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f56bc14d6e0 0x7f56bc153ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:32.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.626+0000 7f56c1b69640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f56bc1570e0 0x7f56bc071d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:39142/0 (socket says 172.21.15.116:39142) 2024-04-25T00:16:32.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.626+0000 7f56c0b67640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56bc06b210 0x7f56bc06b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:32.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.626+0000 7f56c1b69640 1 -- 172.21.15.116:0/2811207909 learned_addr learned my addr 172.21.15.116:0/2811207909 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:16:32.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.626+0000 7f56c0b67640 1 -- 172.21.15.116:0/2811207909 >> v2:172.21.15.174:3300/0 conn(0x7f56bc14d6e0 msgr2=0x7f56bc153ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:32.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.626+0000 7f56c0b67640 1 --2- 172.21.15.116:0/2811207909 >> v2:172.21.15.174:3300/0 conn(0x7f56bc14d6e0 0x7f56bc153ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:32.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.626+0000 7f56c0b67640 1 -- 172.21.15.116:0/2811207909 >> v1:172.21.15.174:6789/0 conn(0x7f56bc1570e0 legacy=0x7f56bc071d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:32.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.626+0000 7f56c0b67640 1 -- 172.21.15.116:0/2811207909 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56bc0a95b0 con 0x7f56bc06b210 2024-04-25T00:16:32.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.626+0000 7f56c1368640 1 --2- 172.21.15.116:0/2811207909 >> v2:172.21.15.174:3300/0 conn(0x7f56bc14d6e0 0x7f56bc153ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:16:32.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.630+0000 7f56c0b67640 1 --2- 172.21.15.116:0/2811207909 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56bc06b210 0x7f56bc06b690 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f56ac0097c0 tx=0x7f56ac01c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c307724d2fa7c9cc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:32.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.630+0000 7f56b3fff640 1 -- 172.21.15.116:0/2811207909 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56ac0b2070 con 0x7f56bc06b210 2024-04-25T00:16:32.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.630+0000 7f56b3fff640 1 -- 172.21.15.116:0/2811207909 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56ac0ad640 con 0x7f56bc06b210 2024-04-25T00:16:32.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.630+0000 7f56b3fff640 1 -- 172.21.15.116:0/2811207909 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56ac0ada10 con 0x7f56bc06b210 2024-04-25T00:16:32.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.630+0000 7f56c35cc640 1 -- 172.21.15.116:0/2811207909 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56bc06b210 msgr2=0x7f56bc06b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:32.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.630+0000 7f56c35cc640 1 --2- 172.21.15.116:0/2811207909 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56bc06b210 0x7f56bc06b690 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f56ac0097c0 tx=0x7f56ac01c4a0 comp rx=0 tx=0).stop 2024-04-25T00:16:32.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.630+0000 7f56c35cc640 1 -- 172.21.15.116:0/2811207909 shutdown_connections 2024-04-25T00:16:32.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.630+0000 7f56c35cc640 1 --2- 172.21.15.116:0/2811207909 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56bc06b210 0x7f56bc06b690 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:32.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.630+0000 7f56c35cc640 1 --2- 172.21.15.116:0/2811207909 >> v2:172.21.15.174:3300/0 conn(0x7f56bc14d6e0 0x7f56bc153ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:32.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.630+0000 7f56c35cc640 1 -- 172.21.15.116:0/2811207909 >> 172.21.15.116:0/2811207909 conn(0x7f56bc0c7fa0 msgr2=0x7f56bc0c83b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:32.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.630+0000 7f56c35cc640 1 -- 172.21.15.116:0/2811207909 shutdown_connections 2024-04-25T00:16:32.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.630+0000 7f56c35cc640 1 -- 172.21.15.116:0/2811207909 wait complete. 2024-04-25T00:16:32.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.630+0000 7f56c35cc640 1 Processor -- start 2024-04-25T00:16:32.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.634+0000 7f56c35cc640 1 -- start start 2024-04-25T00:16:32.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.634+0000 7f56c35cc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56bc06b210 0x7f56bc140fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:32.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.634+0000 7f56c35cc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f56bc14d6e0 0x7f56bc145ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:32.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.634+0000 7f56c35cc640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56bc1570e0 0x7f56bc1eff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:32.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.634+0000 7f56c35cc640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f56bc0a9fd0 con 0x7f56bc06b210 2024-04-25T00:16:32.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.634+0000 7f56c35cc640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f56bc0a9e50 con 0x7f56bc14d6e0 2024-04-25T00:16:32.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.634+0000 7f56c35cc640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f56bc0731e0 con 0x7f56bc1570e0 2024-04-25T00:16:32.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.634+0000 7f56c1368640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56bc06b210 0x7f56bc140fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:32.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.634+0000 7f56c1368640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56bc06b210 0x7f56bc140fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57900/0 (socket says 172.21.15.116:57900) 2024-04-25T00:16:32.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.634+0000 7f56c1368640 1 -- 172.21.15.116:0/555545733 learned_addr learned my addr 172.21.15.116:0/555545733 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:16:32.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.634+0000 7f56c0b67640 1 --2- 172.21.15.116:0/555545733 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f56bc14d6e0 0x7f56bc145ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:32.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.634+0000 7f56c1b69640 1 --2- 172.21.15.116:0/555545733 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56bc1570e0 0x7f56bc1eff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:32.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.634+0000 7f56c1368640 1 -- 172.21.15.116:0/555545733 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56bc1570e0 msgr2=0x7f56bc1eff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:32.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.634+0000 7f56c1368640 1 --2- 172.21.15.116:0/555545733 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56bc1570e0 0x7f56bc1eff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:32.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.634+0000 7f56c1368640 1 -- 172.21.15.116:0/555545733 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f56bc14d6e0 msgr2=0x7f56bc145ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:32.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.634+0000 7f56c1368640 1 --2- 172.21.15.116:0/555545733 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f56bc14d6e0 0x7f56bc145ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:32.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.634+0000 7f56c1368640 1 -- 172.21.15.116:0/555545733 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56bc1551c0 con 0x7f56bc06b210 2024-04-25T00:16:32.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.634+0000 7f56c1b69640 1 --2- 172.21.15.116:0/555545733 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56bc1570e0 0x7f56bc1eff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:16:32.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.634+0000 7f56c1368640 1 --2- 172.21.15.116:0/555545733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56bc06b210 0x7f56bc140fa0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f56b800ebe0 tx=0x7f56b800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:32.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.634+0000 7f56b1ffb640 1 -- 172.21.15.116:0/555545733 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56b8017070 con 0x7f56bc06b210 2024-04-25T00:16:32.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.634+0000 7f56b1ffb640 1 -- 172.21.15.116:0/555545733 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56b800cc30 con 0x7f56bc06b210 2024-04-25T00:16:32.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.634+0000 7f56c35cc640 1 -- 172.21.15.116:0/555545733 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56bc1ed420 con 0x7f56bc06b210 2024-04-25T00:16:32.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.634+0000 7f56b1ffb640 1 -- 172.21.15.116:0/555545733 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56b8013470 con 0x7f56bc06b210 2024-04-25T00:16:32.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.634+0000 7f56c35cc640 1 -- 172.21.15.116:0/555545733 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56bc14c390 con 0x7f56bc06b210 2024-04-25T00:16:32.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.634+0000 7f56c35cc640 1 -- 172.21.15.116:0/555545733 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56bc068a60 con 0x7f56bc06b210 2024-04-25T00:16:32.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.634+0000 7f56b1ffb640 1 -- 172.21.15.116:0/555545733 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f56b801a070 con 0x7f56bc06b210 2024-04-25T00:16:32.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.638+0000 7f56b1ffb640 1 --2- 172.21.15.116:0/555545733 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f56a00678c0 0x7f56a0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:32.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.638+0000 7f56b1ffb640 1 -- 172.21.15.116:0/555545733 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 8472+0+0 (secure 0 0 0) 0x7f56b8086ef0 con 0x7f56bc06b210 2024-04-25T00:16:32.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.638+0000 7f56c0b67640 1 --2- 172.21.15.116:0/555545733 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f56a00678c0 0x7f56a0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:32.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.638+0000 7f56c0b67640 1 --2- 172.21.15.116:0/555545733 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f56a00678c0 0x7f56a0069d80 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f56ac0af000 tx=0x7f56ac008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:32.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.638+0000 7f56b1ffb640 1 -- 172.21.15.116:0/555545733 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f56b80b35c0 con 0x7f56bc06b210 2024-04-25T00:16:32.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.834+0000 7f56c35cc640 1 -- 172.21.15.116:0/555545733 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f56bc1efb50 con 0x7f56bc06b210 2024-04-25T00:16:32.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.834+0000 7f56b1ffb640 1 -- 172.21.15.116:0/555545733 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) ==== 95+0+6509 (secure 0 0 0) 0x7f56b8056f90 con 0x7f56bc06b210 2024-04-25T00:16:32.833 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:16:32.833 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":117,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":5621,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3081407040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3081407040},{"type":"v1","addr":"172.21.15.174:6837","nonce":3081407040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:15:43.819845+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6810":{"gid":6810,"name":"c","rank":0,"incarnation":113,"state":"up:active","state_seq":13,"addr":"172.21.15.116:6833/2340581953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2340581953},{"type":"v1","addr":"172.21.15.116:6833","nonce":2340581953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6810,"qdb_cluster":[6810]},"id":1},{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:15:28.737458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6828":{"gid":6828,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":8,"addr":"172.21.15.116:6835/2012470875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2012470875},{"type":"v1","addr":"172.21.15.116:6835","nonce":2012470875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6828,"qdb_cluster":[6828]},"id":2}]} 2024-04-25T00:16:32.834 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 117 2024-04-25T00:16:32.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.834+0000 7f56c35cc640 1 -- 172.21.15.116:0/555545733 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f56a00678c0 msgr2=0x7f56a0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:32.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.834+0000 7f56c35cc640 1 --2- 172.21.15.116:0/555545733 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f56a00678c0 0x7f56a0069d80 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f56ac0af000 tx=0x7f56ac008040 comp rx=0 tx=0).stop 2024-04-25T00:16:32.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.834+0000 7f56c35cc640 1 -- 172.21.15.116:0/555545733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56bc06b210 msgr2=0x7f56bc140fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:32.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.834+0000 7f56c35cc640 1 --2- 172.21.15.116:0/555545733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56bc06b210 0x7f56bc140fa0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f56b800ebe0 tx=0x7f56b800c520 comp rx=0 tx=0).stop 2024-04-25T00:16:32.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.834+0000 7f56c35cc640 1 -- 172.21.15.116:0/555545733 shutdown_connections 2024-04-25T00:16:32.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.834+0000 7f56c35cc640 1 --2- 172.21.15.116:0/555545733 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f56a00678c0 0x7f56a0069d80 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:32.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.834+0000 7f56c35cc640 1 --2- 172.21.15.116:0/555545733 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56bc1570e0 0x7f56bc1eff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:32.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.834+0000 7f56c35cc640 1 --2- 172.21.15.116:0/555545733 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f56bc14d6e0 0x7f56bc145ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:32.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.834+0000 7f56c35cc640 1 --2- 172.21.15.116:0/555545733 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56bc06b210 0x7f56bc140fa0 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:32.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.834+0000 7f56c35cc640 1 -- 172.21.15.116:0/555545733 >> 172.21.15.116:0/555545733 conn(0x7f56bc0c7fa0 msgr2=0x7f56bc151e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:32.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.838+0000 7f56c35cc640 1 -- 172.21.15.116:0/555545733 shutdown_connections 2024-04-25T00:16:32.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:32.838+0000 7f56c35cc640 1 -- 172.21.15.116:0/555545733 wait complete. 2024-04-25T00:16:32.851 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:16:34.344 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:16:34.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.478+0000 7f9164617640 1 Processor -- start 2024-04-25T00:16:34.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.478+0000 7f9164617640 1 -- start start 2024-04-25T00:16:34.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.478+0000 7f9164617640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f915c14c720 0x7f915c1563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:34.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.478+0000 7f9164617640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f915c071680 0x7f915c071ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:34.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.478+0000 7f9164617640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f915c071220 con 0x7f915c0a94a0 2024-04-25T00:16:34.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.478+0000 7f9164617640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f915c153460 con 0x7f915c14c720 2024-04-25T00:16:34.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.478+0000 7f9164617640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f915c152420 con 0x7f915c071680 2024-04-25T00:16:34.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.478+0000 7f9161bb2640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f915c0a94a0 0x7f915c0c2a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:47640/0 (socket says 172.21.15.116:47640) 2024-04-25T00:16:34.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.478+0000 7f9161bb2640 1 -- 172.21.15.116:0/2972849393 learned_addr learned my addr 172.21.15.116:0/2972849393 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:16:34.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.478+0000 7f9162bb4640 1 --2- 172.21.15.116:0/2972849393 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f915c071680 0x7f915c071ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:34.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.478+0000 7f91623b3640 1 --2- 172.21.15.116:0/2972849393 >> v2:172.21.15.174:3300/0 conn(0x7f915c14c720 0x7f915c1563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:34.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.482+0000 7f91613b1640 1 -- 172.21.15.116:0/2972849393 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4200685244 0 0) 0x7f915c071220 con 0x7f915c0a94a0 2024-04-25T00:16:34.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.482+0000 7f91613b1640 1 -- 172.21.15.116:0/2972849393 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9148003530 con 0x7f915c0a94a0 2024-04-25T00:16:34.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.482+0000 7f91623b3640 1 -- 172.21.15.116:0/2972849393 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f915c071680 msgr2=0x7f915c071ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:34.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.482+0000 7f91623b3640 1 --2- 172.21.15.116:0/2972849393 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f915c071680 0x7f915c071ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:34.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.482+0000 7f91623b3640 1 -- 172.21.15.116:0/2972849393 >> v1:172.21.15.116:6789/0 conn(0x7f915c0a94a0 legacy=0x7f915c0c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:34.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.482+0000 7f91623b3640 1 -- 172.21.15.116:0/2972849393 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f915c14cb00 con 0x7f915c14c720 2024-04-25T00:16:34.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.482+0000 7f91623b3640 1 --2- 172.21.15.116:0/2972849393 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f915c14c720 0x7f915c1563d0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f9158009d90 tx=0x7f915801c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ff89ec5f71ea046 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:34.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.482+0000 7f91613b1640 1 -- 172.21.15.116:0/2972849393 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91580b2070 con 0x7f915c14c720 2024-04-25T00:16:34.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.482+0000 7f91613b1640 1 -- 172.21.15.116:0/2972849393 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91580a5970 con 0x7f915c14c720 2024-04-25T00:16:34.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.482+0000 7f91613b1640 1 -- 172.21.15.116:0/2972849393 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91580a5d20 con 0x7f915c14c720 2024-04-25T00:16:34.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.482+0000 7f9164617640 1 -- 172.21.15.116:0/2972849393 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f915c14c720 msgr2=0x7f915c1563d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:34.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.482+0000 7f9164617640 1 --2- 172.21.15.116:0/2972849393 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f915c14c720 0x7f915c1563d0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f9158009d90 tx=0x7f915801c8b0 comp rx=0 tx=0).stop 2024-04-25T00:16:34.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.482+0000 7f9164617640 1 -- 172.21.15.116:0/2972849393 shutdown_connections 2024-04-25T00:16:34.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.482+0000 7f9164617640 1 --2- 172.21.15.116:0/2972849393 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f915c071680 0x7f915c071ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:34.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.482+0000 7f9164617640 1 --2- 172.21.15.116:0/2972849393 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f915c14c720 0x7f915c1563d0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:34.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.482+0000 7f9164617640 1 -- 172.21.15.116:0/2972849393 >> 172.21.15.116:0/2972849393 conn(0x7f915c0c3b00 msgr2=0x7f915c0c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:34.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.482+0000 7f9164617640 1 -- 172.21.15.116:0/2972849393 shutdown_connections 2024-04-25T00:16:34.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.482+0000 7f9164617640 1 -- 172.21.15.116:0/2972849393 wait complete. 2024-04-25T00:16:34.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.486+0000 7f9164617640 1 Processor -- start 2024-04-25T00:16:34.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.486+0000 7f9164617640 1 -- start start 2024-04-25T00:16:34.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.486+0000 7f9164617640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f915c071680 0x7f915c0c5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:34.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.486+0000 7f9164617640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f915c0a94a0 0x7f915c150b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:34.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.486+0000 7f9164617640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f915c14c720 0x7f915c14fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:34.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.486+0000 7f9164617640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f915c0c3120 con 0x7f915c0a94a0 2024-04-25T00:16:34.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.486+0000 7f9164617640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f915c152f40 con 0x7f915c14c720 2024-04-25T00:16:34.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.486+0000 7f9164617640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f915c0c32a0 con 0x7f915c071680 2024-04-25T00:16:34.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.486+0000 7f9161bb2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f915c0a94a0 0x7f915c150b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:34.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.486+0000 7f9161bb2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f915c0a94a0 0x7f915c150b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57914/0 (socket says 172.21.15.116:57914) 2024-04-25T00:16:34.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.486+0000 7f9161bb2640 1 -- 172.21.15.116:0/2460665583 learned_addr learned my addr 172.21.15.116:0/2460665583 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:16:34.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.486+0000 7f91623b3640 1 --2- 172.21.15.116:0/2460665583 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f915c071680 0x7f915c0c5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:34.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.486+0000 7f9162bb4640 1 --2- 172.21.15.116:0/2460665583 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f915c14c720 0x7f915c14fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:34.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.486+0000 7f9161bb2640 1 -- 172.21.15.116:0/2460665583 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f915c071680 msgr2=0x7f915c0c5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:34.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.486+0000 7f9161bb2640 1 --2- 172.21.15.116:0/2460665583 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f915c071680 0x7f915c0c5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:34.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.486+0000 7f9161bb2640 1 -- 172.21.15.116:0/2460665583 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f915c14c720 msgr2=0x7f915c14fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:34.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.486+0000 7f9161bb2640 1 --2- 172.21.15.116:0/2460665583 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f915c14c720 0x7f915c14fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:34.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.486+0000 7f9161bb2640 1 -- 172.21.15.116:0/2460665583 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f915c1f1400 con 0x7f915c0a94a0 2024-04-25T00:16:34.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.486+0000 7f91623b3640 1 --2- 172.21.15.116:0/2460665583 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f915c071680 0x7f915c0c5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:16:34.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.486+0000 7f9162bb4640 1 --2- 172.21.15.116:0/2460665583 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f915c14c720 0x7f915c14fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:16:34.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.486+0000 7f9161bb2640 1 --2- 172.21.15.116:0/2460665583 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f915c0a94a0 0x7f915c150b90 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f914c00c450 tx=0x7f914c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:34.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.486+0000 7f9152ffd640 1 -- 172.21.15.116:0/2460665583 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f914c017070 con 0x7f915c0a94a0 2024-04-25T00:16:34.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.486+0000 7f9152ffd640 1 -- 172.21.15.116:0/2460665583 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f914c005cf0 con 0x7f915c0a94a0 2024-04-25T00:16:34.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.486+0000 7f9164617640 1 -- 172.21.15.116:0/2460665583 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f915c1f16a0 con 0x7f915c0a94a0 2024-04-25T00:16:34.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.486+0000 7f9152ffd640 1 -- 172.21.15.116:0/2460665583 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f914c007290 con 0x7f915c0a94a0 2024-04-25T00:16:34.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.486+0000 7f9164617640 1 -- 172.21.15.116:0/2460665583 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f915c1f1c60 con 0x7f915c0a94a0 2024-04-25T00:16:34.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.486+0000 7f9164617640 1 -- 172.21.15.116:0/2460665583 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f915c068350 con 0x7f915c0a94a0 2024-04-25T00:16:34.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.490+0000 7f9152ffd640 1 -- 172.21.15.116:0/2460665583 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f914c01a070 con 0x7f915c0a94a0 2024-04-25T00:16:34.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.490+0000 7f9152ffd640 1 --2- 172.21.15.116:0/2460665583 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f91400678c0 0x7f9140069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:34.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.490+0000 7f9152ffd640 1 -- 172.21.15.116:0/2460665583 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 8472+0+0 (secure 0 0 0) 0x7f914c085b50 con 0x7f915c0a94a0 2024-04-25T00:16:34.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.490+0000 7f91623b3640 1 --2- 172.21.15.116:0/2460665583 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f91400678c0 0x7f9140069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:34.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.490+0000 7f91623b3640 1 --2- 172.21.15.116:0/2460665583 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f91400678c0 0x7f9140069d80 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f91580af000 tx=0x7f91580a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:34.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.494+0000 7f9152ffd640 1 -- 172.21.15.116:0/2460665583 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f914c055b80 con 0x7f915c0a94a0 2024-04-25T00:16:34.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.690+0000 7f9164617640 1 -- 172.21.15.116:0/2460665583 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f915c0c3640 con 0x7f915c0a94a0 2024-04-25T00:16:34.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.690+0000 7f9152ffd640 1 -- 172.21.15.116:0/2460665583 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) ==== 95+0+6509 (secure 0 0 0) 0x7f914c055480 con 0x7f915c0a94a0 2024-04-25T00:16:34.689 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:16:34.690 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":117,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":5621,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3081407040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3081407040},{"type":"v1","addr":"172.21.15.174:6837","nonce":3081407040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:15:43.819845+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6810":{"gid":6810,"name":"c","rank":0,"incarnation":113,"state":"up:active","state_seq":13,"addr":"172.21.15.116:6833/2340581953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2340581953},{"type":"v1","addr":"172.21.15.116:6833","nonce":2340581953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6810,"qdb_cluster":[6810]},"id":1},{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:15:28.737458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6828":{"gid":6828,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":8,"addr":"172.21.15.116:6835/2012470875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2012470875},{"type":"v1","addr":"172.21.15.116:6835","nonce":2012470875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6828,"qdb_cluster":[6828]},"id":2}]} 2024-04-25T00:16:34.690 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 117 2024-04-25T00:16:34.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.690+0000 7f9164617640 1 -- 172.21.15.116:0/2460665583 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f91400678c0 msgr2=0x7f9140069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:34.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.690+0000 7f9164617640 1 --2- 172.21.15.116:0/2460665583 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f91400678c0 0x7f9140069d80 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f91580af000 tx=0x7f91580a7070 comp rx=0 tx=0).stop 2024-04-25T00:16:34.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.690+0000 7f9164617640 1 -- 172.21.15.116:0/2460665583 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f915c0a94a0 msgr2=0x7f915c150b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:34.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.690+0000 7f9164617640 1 --2- 172.21.15.116:0/2460665583 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f915c0a94a0 0x7f915c150b90 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f914c00c450 tx=0x7f914c00c910 comp rx=0 tx=0).stop 2024-04-25T00:16:34.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.690+0000 7f9164617640 1 -- 172.21.15.116:0/2460665583 shutdown_connections 2024-04-25T00:16:34.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.690+0000 7f9164617640 1 --2- 172.21.15.116:0/2460665583 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f91400678c0 0x7f9140069d80 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:34.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.690+0000 7f9164617640 1 --2- 172.21.15.116:0/2460665583 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f915c14c720 0x7f915c14fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:34.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.690+0000 7f9164617640 1 --2- 172.21.15.116:0/2460665583 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f915c0a94a0 0x7f915c150b90 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:34.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.690+0000 7f9164617640 1 --2- 172.21.15.116:0/2460665583 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f915c071680 0x7f915c0c5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:34.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.690+0000 7f9164617640 1 -- 172.21.15.116:0/2460665583 >> 172.21.15.116:0/2460665583 conn(0x7f915c0c3b00 msgr2=0x7f915c0ac870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:34.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.694+0000 7f9164617640 1 -- 172.21.15.116:0/2460665583 shutdown_connections 2024-04-25T00:16:34.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.694+0000 7f9164617640 1 -- 172.21.15.116:0/2460665583 wait complete. 2024-04-25T00:16:34.708 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:16:34.852 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:16:34.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.982+0000 7f400898c640 1 Processor -- start 2024-04-25T00:16:34.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.982+0000 7f400898c640 1 -- start start 2024-04-25T00:16:34.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.982+0000 7f400898c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f400414f950 0x7f400406f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:34.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.982+0000 7f400898c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f40040733c0 0x7f40040c0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:34.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.982+0000 7f400898c640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4004155ca0 con 0x7f40041557e0 2024-04-25T00:16:34.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.982+0000 7f400898c640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f4004155f80 con 0x7f400414f950 2024-04-25T00:16:34.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.982+0000 7f400898c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f4004156260 con 0x7f40040733c0 2024-04-25T00:16:34.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.982+0000 7f400259c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f400414f950 0x7f400406f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:34.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.982+0000 7f400259c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f400414f950 0x7f400406f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52436/0 (socket says 172.21.15.116:52436) 2024-04-25T00:16:34.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.982+0000 7f400259c640 1 -- 172.21.15.116:0/535853525 learned_addr learned my addr 172.21.15.116:0/535853525 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:16:34.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.982+0000 7f4001d9b640 1 --2- 172.21.15.116:0/535853525 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f40040733c0 0x7f40040c0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:34.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.982+0000 7f400259c640 1 -- 172.21.15.116:0/535853525 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f40040733c0 msgr2=0x7f40040c0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:34.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.982+0000 7f400259c640 1 --2- 172.21.15.116:0/535853525 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f40040733c0 0x7f40040c0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:34.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.982+0000 7f400259c640 1 -- 172.21.15.116:0/535853525 >> v1:172.21.15.174:6789/0 conn(0x7f40041557e0 legacy=0x7f40040c1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:34.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.982+0000 7f400259c640 1 -- 172.21.15.116:0/535853525 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4004156540 con 0x7f400414f950 2024-04-25T00:16:34.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.982+0000 7f4001d9b640 1 --2- 172.21.15.116:0/535853525 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f40040733c0 0x7f40040c0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:16:34.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.986+0000 7f400259c640 1 --2- 172.21.15.116:0/535853525 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f400414f950 0x7f400406f5e0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f3ff4009870 tx=0x7f3ff401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d43b63683d0c9fe5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:34.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.986+0000 7f400159a640 1 -- 172.21.15.116:0/535853525 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff40b0070 con 0x7f400414f950 2024-04-25T00:16:34.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.986+0000 7f400159a640 1 -- 172.21.15.116:0/535853525 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ff40ab600 con 0x7f400414f950 2024-04-25T00:16:34.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.986+0000 7f400159a640 1 -- 172.21.15.116:0/535853525 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff40ab9d0 con 0x7f400414f950 2024-04-25T00:16:34.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.986+0000 7f400898c640 1 -- 172.21.15.116:0/535853525 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f400414f950 msgr2=0x7f400406f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:34.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.986+0000 7f400898c640 1 --2- 172.21.15.116:0/535853525 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f400414f950 0x7f400406f5e0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f3ff4009870 tx=0x7f3ff401c490 comp rx=0 tx=0).stop 2024-04-25T00:16:34.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.986+0000 7f400898c640 1 -- 172.21.15.116:0/535853525 shutdown_connections 2024-04-25T00:16:34.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.986+0000 7f400898c640 1 --2- 172.21.15.116:0/535853525 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f40040733c0 0x7f40040c0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:34.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.986+0000 7f400898c640 1 --2- 172.21.15.116:0/535853525 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f400414f950 0x7f400406f5e0 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:34.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.986+0000 7f400898c640 1 -- 172.21.15.116:0/535853525 >> 172.21.15.116:0/535853525 conn(0x7f400406b210 msgr2=0x7f400406b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:34.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.986+0000 7f400898c640 1 -- 172.21.15.116:0/535853525 shutdown_connections 2024-04-25T00:16:34.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.986+0000 7f400898c640 1 -- 172.21.15.116:0/535853525 wait complete. 2024-04-25T00:16:34.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.986+0000 7f400898c640 1 Processor -- start 2024-04-25T00:16:34.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.986+0000 7f400898c640 1 -- start start 2024-04-25T00:16:34.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.986+0000 7f400898c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f40040733c0 0x7f40041521c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:34.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.986+0000 7f400898c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f400414f950 0x7f4004149000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:34.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.986+0000 7f400898c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f40041557e0 0x7f400414fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:34.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.986+0000 7f400898c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4004162d10 con 0x7f400414f950 2024-04-25T00:16:34.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.986+0000 7f400898c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4004162b90 con 0x7f40040733c0 2024-04-25T00:16:34.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.986+0000 7f400898c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f4004162e90 con 0x7f40041557e0 2024-04-25T00:16:34.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.986+0000 7f4001d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f400414f950 0x7f4004149000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:34.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.986+0000 7f4001d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f400414f950 0x7f4004149000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52452/0 (socket says 172.21.15.116:52452) 2024-04-25T00:16:34.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.986+0000 7f4001d9b640 1 -- 172.21.15.116:0/204678279 learned_addr learned my addr 172.21.15.116:0/204678279 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:16:34.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.990+0000 7f400259c640 1 --2- 172.21.15.116:0/204678279 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f40040733c0 0x7f40041521c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:34.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.990+0000 7f4002d9d640 1 --2- 172.21.15.116:0/204678279 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f40041557e0 0x7f400414fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:34.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.990+0000 7f4001d9b640 1 -- 172.21.15.116:0/204678279 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f40041557e0 msgr2=0x7f400414fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:34.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.990+0000 7f4001d9b640 1 --2- 172.21.15.116:0/204678279 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f40041557e0 0x7f400414fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:34.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.990+0000 7f4001d9b640 1 -- 172.21.15.116:0/204678279 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f40040733c0 msgr2=0x7f40041521c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:34.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.990+0000 7f4001d9b640 1 --2- 172.21.15.116:0/204678279 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f40040733c0 0x7f40041521c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:34.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.990+0000 7f4001d9b640 1 -- 172.21.15.116:0/204678279 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4004149540 con 0x7f400414f950 2024-04-25T00:16:34.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.990+0000 7f400259c640 1 --2- 172.21.15.116:0/204678279 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f40040733c0 0x7f40041521c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:16:34.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.990+0000 7f4002d9d640 1 --2- 172.21.15.116:0/204678279 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f40041557e0 0x7f400414fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:16:34.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.990+0000 7f4001d9b640 1 --2- 172.21.15.116:0/204678279 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f400414f950 0x7f4004149000 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f3ff800c7c0 tx=0x7f3ff800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:34.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.990+0000 7f3ff2ffd640 1 -- 172.21.15.116:0/204678279 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff8007a30 con 0x7f400414f950 2024-04-25T00:16:34.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.990+0000 7f3ff2ffd640 1 -- 172.21.15.116:0/204678279 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ff8007bd0 con 0x7f400414f950 2024-04-25T00:16:34.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.990+0000 7f3ff2ffd640 1 -- 172.21.15.116:0/204678279 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff80113f0 con 0x7f400414f950 2024-04-25T00:16:34.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.990+0000 7f400898c640 1 -- 172.21.15.116:0/204678279 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4004150270 con 0x7f400414f950 2024-04-25T00:16:34.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.990+0000 7f400898c640 1 -- 172.21.15.116:0/204678279 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40040c3710 con 0x7f400414f950 2024-04-25T00:16:34.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.990+0000 7f400898c640 1 -- 172.21.15.116:0/204678279 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4004068350 con 0x7f400414f950 2024-04-25T00:16:34.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.990+0000 7f3ff2ffd640 1 -- 172.21.15.116:0/204678279 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3ff8011590 con 0x7f400414f950 2024-04-25T00:16:34.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.990+0000 7f3ff2ffd640 1 --2- 172.21.15.116:0/204678279 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3fe40678c0 0x7f3fe4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:34.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.990+0000 7f3ff2ffd640 1 -- 172.21.15.116:0/204678279 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 8472+0+0 (secure 0 0 0) 0x7f3ff8086a20 con 0x7f400414f950 2024-04-25T00:16:34.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.990+0000 7f400259c640 1 --2- 172.21.15.116:0/204678279 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3fe40678c0 0x7f3fe4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:34.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.990+0000 7f400259c640 1 --2- 172.21.15.116:0/204678279 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3fe40678c0 0x7f3fe4069d80 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f3ff40ad000 tx=0x7f3ff40a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:34.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:34.994+0000 7f3ff2ffd640 1 -- 172.21.15.116:0/204678279 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f4004149540 con 0x7f400414f950 2024-04-25T00:16:35.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:35.186+0000 7f400898c640 1 -- 172.21.15.116:0/204678279 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f400406bd30 con 0x7f400414f950 2024-04-25T00:16:35.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:35.186+0000 7f3ff2ffd640 1 -- 172.21.15.116:0/204678279 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) ==== 95+0+6509 (secure 0 0 0) 0x7f3ff8015020 con 0x7f400414f950 2024-04-25T00:16:35.185 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:16:35.186 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":117,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":5621,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3081407040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3081407040},{"type":"v1","addr":"172.21.15.174:6837","nonce":3081407040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:15:43.819845+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6810":{"gid":6810,"name":"c","rank":0,"incarnation":113,"state":"up:active","state_seq":13,"addr":"172.21.15.116:6833/2340581953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2340581953},{"type":"v1","addr":"172.21.15.116:6833","nonce":2340581953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6810,"qdb_cluster":[6810]},"id":1},{"mdsmap":{"epoch":111,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:15:28.737458+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":48,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6828":{"gid":6828,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":8,"addr":"172.21.15.116:6835/2012470875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2012470875},{"type":"v1","addr":"172.21.15.116:6835","nonce":2012470875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6828,"qdb_cluster":[6828]},"id":2}]} 2024-04-25T00:16:35.186 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 117 2024-04-25T00:16:35.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:35.186+0000 7f400898c640 1 -- 172.21.15.116:0/204678279 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3fe40678c0 msgr2=0x7f3fe4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:35.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:35.186+0000 7f400898c640 1 --2- 172.21.15.116:0/204678279 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3fe40678c0 0x7f3fe4069d80 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f3ff40ad000 tx=0x7f3ff40a6000 comp rx=0 tx=0).stop 2024-04-25T00:16:35.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:35.186+0000 7f400898c640 1 -- 172.21.15.116:0/204678279 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f400414f950 msgr2=0x7f4004149000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:35.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:35.186+0000 7f400898c640 1 --2- 172.21.15.116:0/204678279 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f400414f950 0x7f4004149000 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f3ff800c7c0 tx=0x7f3ff800cc80 comp rx=0 tx=0).stop 2024-04-25T00:16:35.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:35.186+0000 7f400898c640 1 -- 172.21.15.116:0/204678279 shutdown_connections 2024-04-25T00:16:35.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:35.186+0000 7f400898c640 1 --2- 172.21.15.116:0/204678279 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3fe40678c0 0x7f3fe4069d80 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:35.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:35.186+0000 7f400898c640 1 --2- 172.21.15.116:0/204678279 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f40041557e0 0x7f400414fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:35.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:35.186+0000 7f400898c640 1 --2- 172.21.15.116:0/204678279 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f400414f950 0x7f4004149000 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:35.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:35.186+0000 7f400898c640 1 --2- 172.21.15.116:0/204678279 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f40040733c0 0x7f40041521c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:35.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:35.186+0000 7f400898c640 1 -- 172.21.15.116:0/204678279 >> 172.21.15.116:0/204678279 conn(0x7f400406b210 msgr2=0x7f4004153e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:35.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:35.186+0000 7f400898c640 1 -- 172.21.15.116:0/204678279 shutdown_connections 2024-04-25T00:16:35.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:35.186+0000 7f400898c640 1 -- 172.21.15.116:0/204678279 wait complete. 2024-04-25T00:16:35.203 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:16:36.709 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:16:36.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.842+0000 7f7c73413640 1 Processor -- start 2024-04-25T00:16:36.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.842+0000 7f7c73413640 1 -- start start 2024-04-25T00:16:36.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.846+0000 7f7c73413640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f7c6c0a99f0 0x7f7c6c153ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:36.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.846+0000 7f7c73413640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c6c0c3910 con 0x7f7c6c0a94a0 2024-04-25T00:16:36.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.846+0000 7f7c73413640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c6c154530 con 0x7f7c6c150d20 2024-04-25T00:16:36.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.846+0000 7f7c73413640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f7c6c0c16e0 con 0x7f7c6c0a99f0 2024-04-25T00:16:36.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.846+0000 7f7c709ae640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f7c6c0a94a0 0x7f7c6c1538e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:44734/0 (socket says 172.21.15.116:44734) 2024-04-25T00:16:36.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.846+0000 7f7c709ae640 1 -- 172.21.15.116:0/2720198988 learned_addr learned my addr 172.21.15.116:0/2720198988 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:16:36.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.846+0000 7f7c719b0640 1 --2- 172.21.15.116:0/2720198988 >> v2:172.21.15.174:3300/0 conn(0x7f7c6c0a99f0 0x7f7c6c153ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:36.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.846+0000 7f7c63fff640 1 -- 172.21.15.116:0/2720198988 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3304322294 0 0) 0x7f7c6c0c3910 con 0x7f7c6c0a94a0 2024-04-25T00:16:36.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.846+0000 7f7c63fff640 1 -- 172.21.15.116:0/2720198988 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c58003530 con 0x7f7c6c0a94a0 2024-04-25T00:16:36.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.846+0000 7f7c63fff640 1 -- 172.21.15.116:0/2720198988 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3378752485 0 0) 0x7f7c58003530 con 0x7f7c6c0a94a0 2024-04-25T00:16:36.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.846+0000 7f7c63fff640 1 -- 172.21.15.116:0/2720198988 >> v2:172.21.15.174:3300/0 conn(0x7f7c6c0a99f0 msgr2=0x7f7c6c153ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:36.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.846+0000 7f7c63fff640 1 --2- 172.21.15.116:0/2720198988 >> v2:172.21.15.174:3300/0 conn(0x7f7c6c0a99f0 0x7f7c6c153ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:36.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.846+0000 7f7c63fff640 1 -- 172.21.15.116:0/2720198988 >> v1:172.21.15.174:6789/0 conn(0x7f7c6c150d20 legacy=0x7f7c6c0c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:36.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.846+0000 7f7c63fff640 1 -- 172.21.15.116:0/2720198988 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c6c0c1400 con 0x7f7c6c0a94a0 2024-04-25T00:16:36.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.846+0000 7f7c63fff640 1 -- 172.21.15.116:0/2720198988 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f7c5c001680 con 0x7f7c6c0a94a0 2024-04-25T00:16:36.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.846+0000 7f7c63fff640 1 -- 172.21.15.116:0/2720198988 >> v1:172.21.15.116:6789/0 conn(0x7f7c6c0a94a0 legacy=0x7f7c6c1538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:36.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.846+0000 7f7c63fff640 1 --2- 172.21.15.116:0/2720198988 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7c580039c0 0x7f7c58003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:36.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.846+0000 7f7c63fff640 1 --2- 172.21.15.116:0/2720198988 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7c58004810 0x7f7c58004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:36.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.846+0000 7f7c63fff640 1 --2- 172.21.15.116:0/2720198988 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c580051d0 0x7f7c5800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:36.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.846+0000 7f7c63fff640 1 -- 172.21.15.116:0/2720198988 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7c6c0c16e0 con 0x7f7c580051d0 2024-04-25T00:16:36.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.846+0000 7f7c63fff640 1 -- 172.21.15.116:0/2720198988 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7c5800c070 con 0x7f7c58004810 2024-04-25T00:16:36.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.846+0000 7f7c63fff640 1 -- 172.21.15.116:0/2720198988 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f7c5800c270 con 0x7f7c580039c0 2024-04-25T00:16:36.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.846+0000 7f7c719b0640 1 --2- 172.21.15.116:0/2720198988 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c580051d0 0x7f7c5800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:36.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.846+0000 7f7c711af640 1 --2- 172.21.15.116:0/2720198988 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7c580039c0 0x7f7c58003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:36.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.846+0000 7f7c709ae640 1 --2- 172.21.15.116:0/2720198988 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7c58004810 0x7f7c58004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:36.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.846+0000 7f7c719b0640 1 -- 172.21.15.116:0/2720198988 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7c580039c0 msgr2=0x7f7c58003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:36.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.846+0000 7f7c719b0640 1 --2- 172.21.15.116:0/2720198988 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7c580039c0 0x7f7c58003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:36.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.846+0000 7f7c719b0640 1 -- 172.21.15.116:0/2720198988 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7c58004810 msgr2=0x7f7c58004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:36.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.846+0000 7f7c719b0640 1 --2- 172.21.15.116:0/2720198988 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7c58004810 0x7f7c58004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:36.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.846+0000 7f7c719b0640 1 -- 172.21.15.116:0/2720198988 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c58003530 con 0x7f7c580051d0 2024-04-25T00:16:36.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.846+0000 7f7c719b0640 1 -- 172.21.15.116:0/2720198988 reap_dead start 2024-04-25T00:16:36.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.846+0000 7f7c719b0640 1 --2- 172.21.15.116:0/2720198988 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c580051d0 0x7f7c5800b9d0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f7c64009fa0 tx=0x7f7c6401cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74ae0e854bc04305 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:36.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.850+0000 7f7c63fff640 1 -- 172.21.15.116:0/2720198988 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c640b2070 con 0x7f7c580051d0 2024-04-25T00:16:36.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.850+0000 7f7c63fff640 1 -- 172.21.15.116:0/2720198988 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c640a5d50 con 0x7f7c580051d0 2024-04-25T00:16:36.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.850+0000 7f7c63fff640 1 -- 172.21.15.116:0/2720198988 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c64009bf0 con 0x7f7c580051d0 2024-04-25T00:16:36.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.850+0000 7f7c73413640 1 -- 172.21.15.116:0/2720198988 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c580051d0 msgr2=0x7f7c5800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:36.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.850+0000 7f7c73413640 1 --2- 172.21.15.116:0/2720198988 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c580051d0 0x7f7c5800b9d0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f7c64009fa0 tx=0x7f7c6401cc80 comp rx=0 tx=0).stop 2024-04-25T00:16:36.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.850+0000 7f7c73413640 1 -- 172.21.15.116:0/2720198988 shutdown_connections 2024-04-25T00:16:36.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.850+0000 7f7c73413640 1 --2- 172.21.15.116:0/2720198988 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c580051d0 0x7f7c5800b9d0 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:36.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.850+0000 7f7c73413640 1 -- 172.21.15.116:0/2720198988 >> 172.21.15.116:0/2720198988 conn(0x7f7c6c0c3b00 msgr2=0x7f7c6c0c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:36.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.850+0000 7f7c73413640 1 -- 172.21.15.116:0/2720198988 shutdown_connections 2024-04-25T00:16:36.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.850+0000 7f7c73413640 1 -- 172.21.15.116:0/2720198988 wait complete. 2024-04-25T00:16:36.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.850+0000 7f7c73413640 1 Processor -- start 2024-04-25T00:16:36.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.850+0000 7f7c73413640 1 -- start start 2024-04-25T00:16:36.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.850+0000 7f7c73413640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c580051d0 0x7f7c6c1ef270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:36.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.850+0000 7f7c73413640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7c6c152790 0x7f7c6c0c5af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:36.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.850+0000 7f7c73413640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7c6c0c0430 0x7f7c6c0c08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:36.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.850+0000 7f7c73413640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7c6c06e940 con 0x7f7c580051d0 2024-04-25T00:16:36.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.850+0000 7f7c73413640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7c6c0c1280 con 0x7f7c6c0c0430 2024-04-25T00:16:36.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.850+0000 7f7c73413640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f7c6c06ec20 con 0x7f7c6c152790 2024-04-25T00:16:36.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.850+0000 7f7c711af640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c580051d0 0x7f7c6c1ef270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:36.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.850+0000 7f7c711af640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c580051d0 0x7f7c6c1ef270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52472/0 (socket says 172.21.15.116:52472) 2024-04-25T00:16:36.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.850+0000 7f7c711af640 1 -- 172.21.15.116:0/1265720701 learned_addr learned my addr 172.21.15.116:0/1265720701 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:16:36.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.850+0000 7f7c719b0640 1 --2- 172.21.15.116:0/1265720701 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7c6c0c0430 0x7f7c6c0c08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:36.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.850+0000 7f7c709ae640 1 --2- 172.21.15.116:0/1265720701 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7c6c152790 0x7f7c6c0c5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:36.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.850+0000 7f7c711af640 1 -- 172.21.15.116:0/1265720701 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7c6c152790 msgr2=0x7f7c6c0c5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:36.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.850+0000 7f7c711af640 1 --2- 172.21.15.116:0/1265720701 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7c6c152790 0x7f7c6c0c5af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:36.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.850+0000 7f7c711af640 1 -- 172.21.15.116:0/1265720701 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7c6c0c0430 msgr2=0x7f7c6c0c08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:36.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.850+0000 7f7c711af640 1 --2- 172.21.15.116:0/1265720701 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7c6c0c0430 0x7f7c6c0c08c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:36.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.850+0000 7f7c711af640 1 -- 172.21.15.116:0/1265720701 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c6c0c6490 con 0x7f7c580051d0 2024-04-25T00:16:36.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.850+0000 7f7c709ae640 1 --2- 172.21.15.116:0/1265720701 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7c6c152790 0x7f7c6c0c5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:16:36.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.850+0000 7f7c711af640 1 --2- 172.21.15.116:0/1265720701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c580051d0 0x7f7c6c1ef270 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f7c6800e4d0 tx=0x7f7c6800e990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:36.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.850+0000 7f7c61ffb640 1 -- 172.21.15.116:0/1265720701 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c680052e0 con 0x7f7c580051d0 2024-04-25T00:16:36.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.850+0000 7f7c61ffb640 1 -- 172.21.15.116:0/1265720701 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c68018070 con 0x7f7c580051d0 2024-04-25T00:16:36.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.850+0000 7f7c73413640 1 -- 172.21.15.116:0/1265720701 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c6c152c20 con 0x7f7c580051d0 2024-04-25T00:16:36.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.850+0000 7f7c61ffb640 1 -- 172.21.15.116:0/1265720701 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c68003820 con 0x7f7c580051d0 2024-04-25T00:16:36.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.850+0000 7f7c73413640 1 -- 172.21.15.116:0/1265720701 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c6c0710c0 con 0x7f7c580051d0 2024-04-25T00:16:36.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.854+0000 7f7c73413640 1 -- 172.21.15.116:0/1265720701 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c6c068350 con 0x7f7c580051d0 2024-04-25T00:16:36.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.854+0000 7f7c61ffb640 1 -- 172.21.15.116:0/1265720701 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7c680039c0 con 0x7f7c580051d0 2024-04-25T00:16:36.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.854+0000 7f7c61ffb640 1 --2- 172.21.15.116:0/1265720701 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7c500678f0 0x7f7c50069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:36.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.854+0000 7f7c61ffb640 1 -- 172.21.15.116:0/1265720701 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 8558+0+0 (secure 0 0 0) 0x7f7c68086590 con 0x7f7c580051d0 2024-04-25T00:16:36.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.854+0000 7f7c709ae640 1 --2- 172.21.15.116:0/1265720701 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7c500678f0 0x7f7c50069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:36.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.854+0000 7f7c709ae640 1 --2- 172.21.15.116:0/1265720701 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7c500678f0 0x7f7c50069db0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c001ec0 tx=0x7f7c5c001120 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:36.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:36.858+0000 7f7c61ffb640 1 -- 172.21.15.116:0/1265720701 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f7c68056470 con 0x7f7c580051d0 2024-04-25T00:16:37.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.050+0000 7f7c73413640 1 -- 172.21.15.116:0/1265720701 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7c6c0c34f0 con 0x7f7c580051d0 2024-04-25T00:16:37.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.050+0000 7f7c61ffb640 1 -- 172.21.15.116:0/1265720701 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) ==== 95+0+5677 (secure 0 0 0) 0x7f7c68005480 con 0x7f7c580051d0 2024-04-25T00:16:37.049 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:16:37.049 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":119,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:15:43.819845+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6810":{"gid":6810,"name":"c","rank":0,"incarnation":113,"state":"up:active","state_seq":13,"addr":"172.21.15.116:6833/2340581953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2340581953},{"type":"v1","addr":"172.21.15.116:6833","nonce":2340581953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6810,"qdb_cluster":[6810]},"id":1},{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:16:35.918171+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5621},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5621":{"gid":5621,"name":"d","rank":0,"incarnation":118,"state":"up:reconnect","state_seq":13,"addr":"172.21.15.174:6837/3081407040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3081407040},{"type":"v1","addr":"172.21.15.174:6837","nonce":3081407040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:16:37.050 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 119 2024-04-25T00:16:37.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.050+0000 7f7c73413640 1 -- 172.21.15.116:0/1265720701 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7c500678f0 msgr2=0x7f7c50069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:37.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.050+0000 7f7c73413640 1 --2- 172.21.15.116:0/1265720701 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7c500678f0 0x7f7c50069db0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c001ec0 tx=0x7f7c5c001120 comp rx=0 tx=0).stop 2024-04-25T00:16:37.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.050+0000 7f7c73413640 1 -- 172.21.15.116:0/1265720701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c580051d0 msgr2=0x7f7c6c1ef270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:37.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.050+0000 7f7c73413640 1 --2- 172.21.15.116:0/1265720701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c580051d0 0x7f7c6c1ef270 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f7c6800e4d0 tx=0x7f7c6800e990 comp rx=0 tx=0).stop 2024-04-25T00:16:37.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.050+0000 7f7c73413640 1 -- 172.21.15.116:0/1265720701 shutdown_connections 2024-04-25T00:16:37.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.050+0000 7f7c73413640 1 --2- 172.21.15.116:0/1265720701 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7c500678f0 0x7f7c50069db0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:37.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.050+0000 7f7c73413640 1 --2- 172.21.15.116:0/1265720701 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7c6c0c0430 0x7f7c6c0c08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:37.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.050+0000 7f7c73413640 1 --2- 172.21.15.116:0/1265720701 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7c6c152790 0x7f7c6c0c5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:37.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.050+0000 7f7c73413640 1 --2- 172.21.15.116:0/1265720701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c580051d0 0x7f7c6c1ef270 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:37.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.050+0000 7f7c73413640 1 -- 172.21.15.116:0/1265720701 >> 172.21.15.116:0/1265720701 conn(0x7f7c6c0c3b00 msgr2=0x7f7c6c0ac640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:37.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.054+0000 7f7c73413640 1 -- 172.21.15.116:0/1265720701 shutdown_connections 2024-04-25T00:16:37.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.054+0000 7f7c73413640 1 -- 172.21.15.116:0/1265720701 wait complete. 2024-04-25T00:16:37.067 INFO:tasks.mds_thrash.fs.[b]:mds.a down, removed from mdsmap 2024-04-25T00:16:37.067 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-04-25T00:16:37.068 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:16:37.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.174+0000 7fc4120b9640 1 Processor -- start 2024-04-25T00:16:37.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.174+0000 7fc4120b9640 1 -- start start 2024-04-25T00:16:37.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.178+0000 7fc4120b9640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc40c14c720 0x7fc40c1563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:37.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.178+0000 7fc4120b9640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc40c071220 con 0x7fc40c071680 2024-04-25T00:16:37.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.178+0000 7fc4120b9640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc40c0721e0 con 0x7fc40c0a94a0 2024-04-25T00:16:37.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.178+0000 7fc4120b9640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fc40c153460 con 0x7fc40c14c720 2024-04-25T00:16:37.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.178+0000 7fc40bfff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fc40c071680 0x7fc40c071ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:44736/0 (socket says 172.21.15.116:44736) 2024-04-25T00:16:37.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.178+0000 7fc40bfff640 1 -- 172.21.15.116:0/2076556661 learned_addr learned my addr 172.21.15.116:0/2076556661 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:16:37.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.178+0000 7fc40b7fe640 1 --2- 172.21.15.116:0/2076556661 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc40c14c720 0x7fc40c1563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:37.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.178+0000 7fc40a7fc640 1 -- 172.21.15.116:0/2076556661 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1418571213 0 0) 0x7fc40c071220 con 0x7fc40c071680 2024-04-25T00:16:37.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.178+0000 7fc40a7fc640 1 -- 172.21.15.116:0/2076556661 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3f8003530 con 0x7fc40c071680 2024-04-25T00:16:37.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.178+0000 7fc40a7fc640 1 -- 172.21.15.116:0/2076556661 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3323607300 0 0) 0x7fc3f8003530 con 0x7fc40c071680 2024-04-25T00:16:37.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.178+0000 7fc40a7fc640 1 -- 172.21.15.116:0/2076556661 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc40c14c720 msgr2=0x7fc40c1563d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:37.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.178+0000 7fc40a7fc640 1 --2- 172.21.15.116:0/2076556661 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc40c14c720 0x7fc40c1563d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:37.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.178+0000 7fc40a7fc640 1 -- 172.21.15.116:0/2076556661 >> v1:172.21.15.174:6789/0 conn(0x7fc40c0a94a0 legacy=0x7fc40c0c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:37.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.178+0000 7fc40a7fc640 1 -- 172.21.15.116:0/2076556661 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc40c152420 con 0x7fc40c071680 2024-04-25T00:16:37.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.178+0000 7fc40a7fc640 1 -- 172.21.15.116:0/2076556661 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7fc404001a00 con 0x7fc40c071680 2024-04-25T00:16:37.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.178+0000 7fc40a7fc640 1 -- 172.21.15.116:0/2076556661 >> v1:172.21.15.116:6789/0 conn(0x7fc40c071680 legacy=0x7fc40c071ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:37.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.178+0000 7fc40a7fc640 1 --2- 172.21.15.116:0/2076556661 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc3f80039f0 0x7fc3f8003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:37.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.178+0000 7fc40a7fc640 1 --2- 172.21.15.116:0/2076556661 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc40c14c720 0x7fc3f8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:37.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.178+0000 7fc40a7fc640 1 --2- 172.21.15.116:0/2076556661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3f8004d50 0x7fc3f80051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:37.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.178+0000 7fc40a7fc640 1 -- 172.21.15.116:0/2076556661 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc40c153460 con 0x7fc3f8004d50 2024-04-25T00:16:37.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.178+0000 7fc40a7fc640 1 -- 172.21.15.116:0/2076556661 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc3f800bb00 con 0x7fc3f80039f0 2024-04-25T00:16:37.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.178+0000 7fc40a7fc640 1 -- 172.21.15.116:0/2076556661 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fc3f800bd00 con 0x7fc40c14c720 2024-04-25T00:16:37.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.178+0000 7fc40bfff640 1 --2- 172.21.15.116:0/2076556661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3f8004d50 0x7fc3f80051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:37.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.178+0000 7fc40b7fe640 1 --2- 172.21.15.116:0/2076556661 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc3f80039f0 0x7fc3f8003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:37.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.178+0000 7fc40affd640 1 --2- 172.21.15.116:0/2076556661 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc40c14c720 0x7fc3f8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:37.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.178+0000 7fc40bfff640 1 -- 172.21.15.116:0/2076556661 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc40c14c720 msgr2=0x7fc3f8004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:37.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.178+0000 7fc40bfff640 1 --2- 172.21.15.116:0/2076556661 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc40c14c720 0x7fc3f8004810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:37.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.178+0000 7fc40bfff640 1 -- 172.21.15.116:0/2076556661 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc3f80039f0 msgr2=0x7fc3f8003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:37.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.178+0000 7fc40bfff640 1 --2- 172.21.15.116:0/2076556661 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc3f80039f0 0x7fc3f8003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:37.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.178+0000 7fc40bfff640 1 -- 172.21.15.116:0/2076556661 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3f8003530 con 0x7fc3f8004d50 2024-04-25T00:16:37.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.178+0000 7fc40bfff640 1 --2- 172.21.15.116:0/2076556661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3f8004d50 0x7fc3f80051d0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fc40400a850 tx=0x7fc40401f920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b8f4b86eb13a099 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:37.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.182+0000 7fc40a7fc640 1 -- 172.21.15.116:0/2076556661 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc40400a4a0 con 0x7fc3f8004d50 2024-04-25T00:16:37.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.182+0000 7fc40a7fc640 1 -- 172.21.15.116:0/2076556661 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4040b2070 con 0x7fc3f8004d50 2024-04-25T00:16:37.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.182+0000 7fc40a7fc640 1 -- 172.21.15.116:0/2076556661 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4040ad8c0 con 0x7fc3f8004d50 2024-04-25T00:16:37.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.182+0000 7fc4120b9640 1 -- 172.21.15.116:0/2076556661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3f8004d50 msgr2=0x7fc3f80051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:37.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.182+0000 7fc4120b9640 1 --2- 172.21.15.116:0/2076556661 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3f8004d50 0x7fc3f80051d0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fc40400a850 tx=0x7fc40401f920 comp rx=0 tx=0).stop 2024-04-25T00:16:37.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.182+0000 7fc40bfff640 1 -- 172.21.15.116:0/2076556661 reap_dead start 2024-04-25T00:16:37.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.182+0000 7fc4120b9640 1 -- 172.21.15.116:0/2076556661 shutdown_connections 2024-04-25T00:16:37.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.182+0000 7fc4120b9640 1 -- 172.21.15.116:0/2076556661 >> 172.21.15.116:0/2076556661 conn(0x7fc40c0c3b00 msgr2=0x7fc40c0c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:37.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.182+0000 7fc4120b9640 1 -- 172.21.15.116:0/2076556661 shutdown_connections 2024-04-25T00:16:37.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.182+0000 7fc4120b9640 1 -- 172.21.15.116:0/2076556661 wait complete. 2024-04-25T00:16:37.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.182+0000 7fc4120b9640 1 Processor -- start 2024-04-25T00:16:37.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.182+0000 7fc4120b9640 1 -- start start 2024-04-25T00:16:37.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.182+0000 7fc4120b9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3f8004d50 0x7fc40c14aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:37.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.182+0000 7fc4120b9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc40c156410 0x7fc40c13e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:37.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.182+0000 7fc4120b9640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc40c0c68d0 0x7fc40c142db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:37.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.182+0000 7fc4120b9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc40c152980 con 0x7fc3f8004d50 2024-04-25T00:16:37.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.182+0000 7fc4120b9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc40c1526a0 con 0x7fc40c156410 2024-04-25T00:16:37.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.182+0000 7fc4120b9640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fc40c152c60 con 0x7fc40c0c68d0 2024-04-25T00:16:37.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.182+0000 7fc40b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3f8004d50 0x7fc40c14aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:37.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.182+0000 7fc40b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3f8004d50 0x7fc40c14aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52492/0 (socket says 172.21.15.116:52492) 2024-04-25T00:16:37.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.182+0000 7fc40b7fe640 1 -- 172.21.15.116:0/3343777126 learned_addr learned my addr 172.21.15.116:0/3343777126 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:16:37.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.182+0000 7fc40affd640 1 --2- 172.21.15.116:0/3343777126 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc40c156410 0x7fc40c13e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:37.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.182+0000 7fc40b7fe640 1 -- 172.21.15.116:0/3343777126 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc40c0c68d0 msgr2=0x7fc40c142db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:37.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.182+0000 7fc40bfff640 1 --2- 172.21.15.116:0/3343777126 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc40c0c68d0 0x7fc40c142db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:37.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.182+0000 7fc40b7fe640 1 --2- 172.21.15.116:0/3343777126 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc40c0c68d0 0x7fc40c142db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:37.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.182+0000 7fc40b7fe640 1 -- 172.21.15.116:0/3343777126 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc40c156410 msgr2=0x7fc40c13e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:37.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.182+0000 7fc40b7fe640 1 --2- 172.21.15.116:0/3343777126 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc40c156410 0x7fc40c13e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:37.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.182+0000 7fc40b7fe640 1 -- 172.21.15.116:0/3343777126 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc40c1e9020 con 0x7fc3f8004d50 2024-04-25T00:16:37.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.182+0000 7fc40b7fe640 1 --2- 172.21.15.116:0/3343777126 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3f8004d50 0x7fc40c14aa70 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fc3fc011610 tx=0x7fc3fc011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:37.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.182+0000 7fc403fff640 1 -- 172.21.15.116:0/3343777126 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3fc00f8f0 con 0x7fc3f8004d50 2024-04-25T00:16:37.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.182+0000 7fc403fff640 1 -- 172.21.15.116:0/3343777126 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3fc00fa90 con 0x7fc3f8004d50 2024-04-25T00:16:37.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.182+0000 7fc403fff640 1 -- 172.21.15.116:0/3343777126 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3fc00fe40 con 0x7fc3f8004d50 2024-04-25T00:16:37.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.182+0000 7fc4120b9640 1 -- 172.21.15.116:0/3343777126 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc40c1e9300 con 0x7fc3f8004d50 2024-04-25T00:16:37.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.182+0000 7fc4120b9640 1 -- 172.21.15.116:0/3343777126 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc40c0c6db0 con 0x7fc3f8004d50 2024-04-25T00:16:37.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.186+0000 7fc4120b9640 1 -- 172.21.15.116:0/3343777126 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc40c068350 con 0x7fc3f8004d50 2024-04-25T00:16:37.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.186+0000 7fc403fff640 1 -- 172.21.15.116:0/3343777126 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc3fc015460 con 0x7fc3f8004d50 2024-04-25T00:16:37.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.186+0000 7fc403fff640 1 --2- 172.21.15.116:0/3343777126 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc3ec0678c0 0x7fc3ec069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:37.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.186+0000 7fc403fff640 1 -- 172.21.15.116:0/3343777126 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 8558+0+0 (secure 0 0 0) 0x7fc3fc087160 con 0x7fc3f8004d50 2024-04-25T00:16:37.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.186+0000 7fc40affd640 1 --2- 172.21.15.116:0/3343777126 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc3ec0678c0 0x7fc3ec069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:37.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.186+0000 7fc40affd640 1 --2- 172.21.15.116:0/3343777126 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc3ec0678c0 0x7fc3ec069d80 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fc40c14b320 tx=0x7fc3f4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:37.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.190+0000 7fc403fff640 1 -- 172.21.15.116:0/3343777126 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fc3fc019030 con 0x7fc3f8004d50 2024-04-25T00:16:37.204 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:16:37.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.318+0000 7f758e22b640 1 Processor -- start 2024-04-25T00:16:37.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.318+0000 7f758e22b640 1 -- start start 2024-04-25T00:16:37.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.318+0000 7f758e22b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f75880c1c30 0x7f75880c3060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:37.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.318+0000 7f758e22b640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f7588153170 0x7f758814fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:37.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.322+0000 7f758e22b640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75880c22a0 con 0x7f758806c480 2024-04-25T00:16:37.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.322+0000 7f758e22b640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f75880c35a0 con 0x7f75880c1c30 2024-04-25T00:16:37.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.322+0000 7f758e22b640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f7588153670 con 0x7f7588153170 2024-04-25T00:16:37.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.322+0000 7f7586ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f75880c1c30 0x7f75880c3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:37.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.322+0000 7f7586ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f75880c1c30 0x7f75880c3060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52502/0 (socket says 172.21.15.116:52502) 2024-04-25T00:16:37.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.322+0000 7f7586ffd640 1 -- 172.21.15.116:0/94691853 learned_addr learned my addr 172.21.15.116:0/94691853 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:16:37.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.322+0000 7f7587fff640 1 --2- 172.21.15.116:0/94691853 >> v2:172.21.15.174:3300/0 conn(0x7f7588153170 0x7f758814fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:37.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.322+0000 7f7586ffd640 1 -- 172.21.15.116:0/94691853 >> v2:172.21.15.174:3300/0 conn(0x7f7588153170 msgr2=0x7f758814fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:37.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.322+0000 7f7586ffd640 1 --2- 172.21.15.116:0/94691853 >> v2:172.21.15.174:3300/0 conn(0x7f7588153170 0x7f758814fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:37.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.322+0000 7f7586ffd640 1 -- 172.21.15.116:0/94691853 >> v1:172.21.15.174:6789/0 conn(0x7f758806c480 legacy=0x7f7588071e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:37.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.322+0000 7f7586ffd640 1 -- 172.21.15.116:0/94691853 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75880c2580 con 0x7f75880c1c30 2024-04-25T00:16:37.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.322+0000 7f7586ffd640 1 --2- 172.21.15.116:0/94691853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75880c1c30 0x7f75880c3060 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f7588156d00 tx=0x7f757801ebb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf507c279ccf7f0f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:37.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.322+0000 7f75867fc640 1 -- 172.21.15.116:0/94691853 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75780b4070 con 0x7f75880c1c30 2024-04-25T00:16:37.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.322+0000 7f75867fc640 1 -- 172.21.15.116:0/94691853 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75780a7980 con 0x7f75880c1c30 2024-04-25T00:16:37.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.322+0000 7f75867fc640 1 -- 172.21.15.116:0/94691853 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75780a7d50 con 0x7f75880c1c30 2024-04-25T00:16:37.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.322+0000 7f758e22b640 1 -- 172.21.15.116:0/94691853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75880c1c30 msgr2=0x7f75880c3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:37.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.322+0000 7f758e22b640 1 --2- 172.21.15.116:0/94691853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75880c1c30 0x7f75880c3060 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f7588156d00 tx=0x7f757801ebb0 comp rx=0 tx=0).stop 2024-04-25T00:16:37.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.322+0000 7f758e22b640 1 -- 172.21.15.116:0/94691853 shutdown_connections 2024-04-25T00:16:37.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.322+0000 7f758e22b640 1 --2- 172.21.15.116:0/94691853 >> v2:172.21.15.174:3300/0 conn(0x7f7588153170 0x7f758814fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:37.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.322+0000 7f758e22b640 1 --2- 172.21.15.116:0/94691853 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f75880c1c30 0x7f75880c3060 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:37.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.322+0000 7f758e22b640 1 -- 172.21.15.116:0/94691853 >> 172.21.15.116:0/94691853 conn(0x7f75880c4270 msgr2=0x7f75880c4680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:37.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.322+0000 7f758e22b640 1 -- 172.21.15.116:0/94691853 shutdown_connections 2024-04-25T00:16:37.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.322+0000 7f758e22b640 1 -- 172.21.15.116:0/94691853 wait complete. 2024-04-25T00:16:37.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.326+0000 7f758e22b640 1 Processor -- start 2024-04-25T00:16:37.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.326+0000 7f758e22b640 1 -- start start 2024-04-25T00:16:37.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.326+0000 7f758e22b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f758806c480 0x7f75881453d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:37.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.326+0000 7f758e22b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f75880c1c30 0x7f75880c63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:37.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.326+0000 7f758e22b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7588153170 0x7f7588141830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:37.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.326+0000 7f758e22b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f758806f1d0 con 0x7f758806c480 2024-04-25T00:16:37.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.326+0000 7f758e22b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f758806f050 con 0x7f7588153170 2024-04-25T00:16:37.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.326+0000 7f758e22b640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f758806f350 con 0x7f75880c1c30 2024-04-25T00:16:37.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.326+0000 7f75877fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f758806c480 0x7f75881453d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:37.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.326+0000 7f75877fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f758806c480 0x7f75881453d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52504/0 (socket says 172.21.15.116:52504) 2024-04-25T00:16:37.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.326+0000 7f75877fe640 1 -- 172.21.15.116:0/2780694595 learned_addr learned my addr 172.21.15.116:0/2780694595 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:16:37.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.326+0000 7f7587fff640 1 --2- 172.21.15.116:0/2780694595 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7588153170 0x7f7588141830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:37.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.326+0000 7f7586ffd640 1 --2- 172.21.15.116:0/2780694595 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f75880c1c30 0x7f75880c63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:37.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.326+0000 7f75877fe640 1 -- 172.21.15.116:0/2780694595 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f75880c1c30 msgr2=0x7f75880c63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:37.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.326+0000 7f75877fe640 1 --2- 172.21.15.116:0/2780694595 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f75880c1c30 0x7f75880c63b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:37.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.326+0000 7f75877fe640 1 -- 172.21.15.116:0/2780694595 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7588153170 msgr2=0x7f7588141830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:37.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.326+0000 7f75877fe640 1 --2- 172.21.15.116:0/2780694595 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7588153170 0x7f7588141830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:37.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.326+0000 7f75877fe640 1 -- 172.21.15.116:0/2780694595 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75881ea560 con 0x7f758806c480 2024-04-25T00:16:37.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.326+0000 7f75877fe640 1 --2- 172.21.15.116:0/2780694595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f758806c480 0x7f75881453d0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f757c00c450 tx=0x7f757c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:37.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.326+0000 7f756ffff640 1 -- 172.21.15.116:0/2780694595 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f757c012830 con 0x7f758806c480 2024-04-25T00:16:37.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.326+0000 7f756ffff640 1 -- 172.21.15.116:0/2780694595 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f757c017070 con 0x7f758806c480 2024-04-25T00:16:37.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.326+0000 7f758e22b640 1 -- 172.21.15.116:0/2780694595 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75881ea840 con 0x7f758806c480 2024-04-25T00:16:37.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.326+0000 7f756ffff640 1 -- 172.21.15.116:0/2780694595 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f757c012c30 con 0x7f758806c480 2024-04-25T00:16:37.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.326+0000 7f758e22b640 1 -- 172.21.15.116:0/2780694595 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7588145910 con 0x7f758806c480 2024-04-25T00:16:37.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.326+0000 7f758e22b640 1 -- 172.21.15.116:0/2780694595 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7588068a60 con 0x7f758806c480 2024-04-25T00:16:37.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.326+0000 7f756ffff640 1 -- 172.21.15.116:0/2780694595 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f757c01c460 con 0x7f758806c480 2024-04-25T00:16:37.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.326+0000 7f756ffff640 1 --2- 172.21.15.116:0/2780694595 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f75740678f0 0x7f7574069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:37.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.326+0000 7f756ffff640 1 -- 172.21.15.116:0/2780694595 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 8558+0+0 (secure 0 0 0) 0x7f757c086570 con 0x7f758806c480 2024-04-25T00:16:37.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.330+0000 7f7586ffd640 1 --2- 172.21.15.116:0/2780694595 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f75740678f0 0x7f7574069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:37.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.330+0000 7f7586ffd640 1 --2- 172.21.15.116:0/2780694595 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f75740678f0 0x7f7574069db0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f75780b1000 tx=0x7f75780a9070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:37.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.334+0000 7f756ffff640 1 -- 172.21.15.116:0/2780694595 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f757c0565d0 con 0x7f758806c480 2024-04-25T00:16:37.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.378+0000 7fc4120b9640 1 -- 172.21.15.116:0/3343777126 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc40c1567f0 con 0x7fc3f8004d50 2024-04-25T00:16:37.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.378+0000 7fc403fff640 1 -- 172.21.15.116:0/3343777126 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) ==== 95+0+5677 (secure 0 0 0) 0x7fc3fc0571c0 con 0x7fc3f8004d50 2024-04-25T00:16:37.379 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:16:37.379 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":119,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:15:43.819845+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6810":{"gid":6810,"name":"c","rank":0,"incarnation":113,"state":"up:active","state_seq":13,"addr":"172.21.15.116:6833/2340581953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2340581953},{"type":"v1","addr":"172.21.15.116:6833","nonce":2340581953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6810,"qdb_cluster":[6810]},"id":1},{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:16:35.918171+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5621},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5621":{"gid":5621,"name":"d","rank":0,"incarnation":118,"state":"up:reconnect","state_seq":13,"addr":"172.21.15.174:6837/3081407040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3081407040},{"type":"v1","addr":"172.21.15.174:6837","nonce":3081407040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:16:37.379 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 119 2024-04-25T00:16:37.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.382+0000 7fc4120b9640 1 -- 172.21.15.116:0/3343777126 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc3ec0678c0 msgr2=0x7fc3ec069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:37.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.382+0000 7fc4120b9640 1 --2- 172.21.15.116:0/3343777126 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc3ec0678c0 0x7fc3ec069d80 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fc40c14b320 tx=0x7fc3f4009040 comp rx=0 tx=0).stop 2024-04-25T00:16:37.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.382+0000 7fc4120b9640 1 -- 172.21.15.116:0/3343777126 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3f8004d50 msgr2=0x7fc40c14aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:37.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.382+0000 7fc4120b9640 1 --2- 172.21.15.116:0/3343777126 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3f8004d50 0x7fc40c14aa70 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fc3fc011610 tx=0x7fc3fc011ad0 comp rx=0 tx=0).stop 2024-04-25T00:16:37.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.382+0000 7fc4120b9640 1 -- 172.21.15.116:0/3343777126 shutdown_connections 2024-04-25T00:16:37.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.382+0000 7fc4120b9640 1 --2- 172.21.15.116:0/3343777126 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc3ec0678c0 0x7fc3ec069d80 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:37.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.382+0000 7fc4120b9640 1 --2- 172.21.15.116:0/3343777126 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc40c0c68d0 0x7fc40c142db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:37.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.382+0000 7fc4120b9640 1 --2- 172.21.15.116:0/3343777126 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc40c156410 0x7fc40c13e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:37.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.382+0000 7fc4120b9640 1 --2- 172.21.15.116:0/3343777126 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc3f8004d50 0x7fc40c14aa70 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:37.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.382+0000 7fc4120b9640 1 -- 172.21.15.116:0/3343777126 >> 172.21.15.116:0/3343777126 conn(0x7fc40c0c3b00 msgr2=0x7fc40c0723b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:37.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.382+0000 7fc4120b9640 1 -- 172.21.15.116:0/3343777126 shutdown_connections 2024-04-25T00:16:37.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.382+0000 7fc4120b9640 1 -- 172.21.15.116:0/3343777126 wait complete. 2024-04-25T00:16:37.397 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 119, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:04.678737+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 116, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6810": { "addr": "172.21.15.116:6833/2340581953", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6832", "nonce": 2340581953, "type": "v2" }, { "addr": "172.21.15.116:6833", "nonce": 2340581953, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6810, "incarnation": 113, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 13 } }, "last_failure": 0, "last_failure_osd_epoch": 49, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T00:15:43.819845+0000", "qdb_cluster": [ 6810 ], "qdb_leader": 6810, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6810 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:18.825887+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 119, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5621": { "addr": "172.21.15.174:6837/3081407040", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6836", "nonce": 3081407040, "type": "v2" }, { "addr": "172.21.15.174:6837", "nonce": 3081407040, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5621, "incarnation": 118, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:reconnect", "state_seq": 13 } }, "last_failure": 0, "last_failure_osd_epoch": 50, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-04-25T00:16:35.918171+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5621 } } } ], "standbys": [ { "addr": "172.21.15.174:6835/4112145782", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6834", "nonce": 4112145782, "type": "v2" }, { "addr": "172.21.15.174:6835", "nonce": 4112145782, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 112, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5582, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-25T00:16:37.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.530+0000 7f758e22b640 1 -- 172.21.15.116:0/2780694595 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f758814d0c0 con 0x7f758806c480 2024-04-25T00:16:37.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.534+0000 7f756ffff640 1 -- 172.21.15.116:0/2780694595 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) ==== 95+0+5677 (secure 0 0 0) 0x7f757c055ed0 con 0x7f758806c480 2024-04-25T00:16:37.532 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:16:37.532 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":119,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:15:43.819845+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6810":{"gid":6810,"name":"c","rank":0,"incarnation":113,"state":"up:active","state_seq":13,"addr":"172.21.15.116:6833/2340581953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2340581953},{"type":"v1","addr":"172.21.15.116:6833","nonce":2340581953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6810,"qdb_cluster":[6810]},"id":1},{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:16:35.918171+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5621},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5621":{"gid":5621,"name":"d","rank":0,"incarnation":118,"state":"up:reconnect","state_seq":13,"addr":"172.21.15.174:6837/3081407040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3081407040},{"type":"v1","addr":"172.21.15.174:6837","nonce":3081407040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:16:37.533 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 119 2024-04-25T00:16:37.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.534+0000 7f758e22b640 1 -- 172.21.15.116:0/2780694595 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f75740678f0 msgr2=0x7f7574069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:37.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.534+0000 7f758e22b640 1 --2- 172.21.15.116:0/2780694595 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f75740678f0 0x7f7574069db0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f75780b1000 tx=0x7f75780a9070 comp rx=0 tx=0).stop 2024-04-25T00:16:37.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.534+0000 7f758e22b640 1 -- 172.21.15.116:0/2780694595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f758806c480 msgr2=0x7f75881453d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:37.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.534+0000 7f758e22b640 1 --2- 172.21.15.116:0/2780694595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f758806c480 0x7f75881453d0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f757c00c450 tx=0x7f757c00c910 comp rx=0 tx=0).stop 2024-04-25T00:16:37.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.534+0000 7f758e22b640 1 -- 172.21.15.116:0/2780694595 shutdown_connections 2024-04-25T00:16:37.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.534+0000 7f758e22b640 1 --2- 172.21.15.116:0/2780694595 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f75740678f0 0x7f7574069db0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:37.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.534+0000 7f758e22b640 1 --2- 172.21.15.116:0/2780694595 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7588153170 0x7f7588141830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:37.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.534+0000 7f758e22b640 1 --2- 172.21.15.116:0/2780694595 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f75880c1c30 0x7f75880c63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:37.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.534+0000 7f758e22b640 1 --2- 172.21.15.116:0/2780694595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f758806c480 0x7f75881453d0 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:37.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.534+0000 7f758e22b640 1 -- 172.21.15.116:0/2780694595 >> 172.21.15.116:0/2780694595 conn(0x7f75880c4270 msgr2=0x7f7588149660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:37.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.534+0000 7f758e22b640 1 -- 172.21.15.116:0/2780694595 shutdown_connections 2024-04-25T00:16:37.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:37.534+0000 7f758e22b640 1 -- 172.21.15.116:0/2780694595 wait complete. 2024-04-25T00:16:37.550 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:16:38.785 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:16:38.787 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:16:39.398 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:16:39.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.530+0000 7faa78a46640 1 Processor -- start 2024-04-25T00:16:39.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.530+0000 7faa78a46640 1 -- start start 2024-04-25T00:16:39.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.530+0000 7faa78a46640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa741500e0 0x7faa74072b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:39.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.530+0000 7faa78a46640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7faa740c1c10 0x7faa740c2170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:39.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.530+0000 7faa78a46640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa74073040 con 0x7faa7406c220 2024-04-25T00:16:39.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.530+0000 7faa78a46640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7faa7406fc20 con 0x7faa740c1c10 2024-04-25T00:16:39.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.530+0000 7faa78a46640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7faa74152870 con 0x7faa741500e0 2024-04-25T00:16:39.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.530+0000 7faa71d9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7faa740c1c10 0x7faa740c2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:39.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.530+0000 7faa71d9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7faa740c1c10 0x7faa740c2170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52518/0 (socket says 172.21.15.116:52518) 2024-04-25T00:16:39.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.530+0000 7faa71d9b640 1 -- 172.21.15.116:0/3766531738 learned_addr learned my addr 172.21.15.116:0/3766531738 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:16:39.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.530+0000 7faa7259c640 1 --2- 172.21.15.116:0/3766531738 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa741500e0 0x7faa74072b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:39.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.530+0000 7faa71d9b640 1 -- 172.21.15.116:0/3766531738 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa741500e0 msgr2=0x7faa74072b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:39.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.530+0000 7faa71d9b640 1 --2- 172.21.15.116:0/3766531738 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa741500e0 0x7faa74072b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:39.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.530+0000 7faa71d9b640 1 -- 172.21.15.116:0/3766531738 >> v1:172.21.15.174:6789/0 conn(0x7faa7406c220 legacy=0x7faa74155c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:39.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.530+0000 7faa71d9b640 1 -- 172.21.15.116:0/3766531738 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa74072020 con 0x7faa740c1c10 2024-04-25T00:16:39.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.530+0000 7faa7259c640 1 --2- 172.21.15.116:0/3766531738 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa741500e0 0x7faa74072b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:16:39.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.534+0000 7faa71d9b640 1 --2- 172.21.15.116:0/3766531738 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa740c1c10 0x7faa740c2170 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7faa68009870 tx=0x7faa6801ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17237518d52bfd9d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:39.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.534+0000 7faa7159a640 1 -- 172.21.15.116:0/3766531738 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa680b5070 con 0x7faa740c1c10 2024-04-25T00:16:39.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.534+0000 7faa7159a640 1 -- 172.21.15.116:0/3766531738 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa680a8a50 con 0x7faa740c1c10 2024-04-25T00:16:39.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.534+0000 7faa7159a640 1 -- 172.21.15.116:0/3766531738 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa680a8e20 con 0x7faa740c1c10 2024-04-25T00:16:39.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.534+0000 7faa78a46640 1 -- 172.21.15.116:0/3766531738 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa740c1c10 msgr2=0x7faa740c2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:39.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.534+0000 7faa78a46640 1 --2- 172.21.15.116:0/3766531738 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa740c1c10 0x7faa740c2170 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7faa68009870 tx=0x7faa6801ed50 comp rx=0 tx=0).stop 2024-04-25T00:16:39.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.534+0000 7faa78a46640 1 -- 172.21.15.116:0/3766531738 shutdown_connections 2024-04-25T00:16:39.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.534+0000 7faa78a46640 1 --2- 172.21.15.116:0/3766531738 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa740c1c10 0x7faa740c2170 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:39.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.534+0000 7faa78a46640 1 --2- 172.21.15.116:0/3766531738 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa741500e0 0x7faa74072b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:39.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.534+0000 7faa78a46640 1 -- 172.21.15.116:0/3766531738 >> 172.21.15.116:0/3766531738 conn(0x7faa740a9b20 msgr2=0x7faa740c4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:39.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.534+0000 7faa78a46640 1 -- 172.21.15.116:0/3766531738 shutdown_connections 2024-04-25T00:16:39.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.534+0000 7faa78a46640 1 -- 172.21.15.116:0/3766531738 wait complete. 2024-04-25T00:16:39.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.538+0000 7faa78a46640 1 Processor -- start 2024-04-25T00:16:39.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.538+0000 7faa78a46640 1 -- start start 2024-04-25T00:16:39.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.538+0000 7faa78a46640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa7406c220 0x7faa74148df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:39.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.538+0000 7faa78a46640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa740c1c10 0x7faa741dcab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:39.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.538+0000 7faa78a46640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa741500e0 0x7faa7414cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:39.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.538+0000 7faa78a46640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7faa740c26b0 con 0x7faa7406c220 2024-04-25T00:16:39.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.538+0000 7faa78a46640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7faa740725e0 con 0x7faa740c1c10 2024-04-25T00:16:39.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.538+0000 7faa78a46640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7faa7406fe50 con 0x7faa741500e0 2024-04-25T00:16:39.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.538+0000 7faa7259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa7406c220 0x7faa74148df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:39.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.538+0000 7faa7259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa7406c220 0x7faa74148df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52522/0 (socket says 172.21.15.116:52522) 2024-04-25T00:16:39.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.538+0000 7faa7259c640 1 -- 172.21.15.116:0/1037047635 learned_addr learned my addr 172.21.15.116:0/1037047635 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:16:39.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.538+0000 7faa71d9b640 1 --2- 172.21.15.116:0/1037047635 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa740c1c10 0x7faa741dcab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:39.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.538+0000 7faa72d9d640 1 --2- 172.21.15.116:0/1037047635 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa741500e0 0x7faa7414cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:39.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.538+0000 7faa7259c640 1 -- 172.21.15.116:0/1037047635 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa741500e0 msgr2=0x7faa7414cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:39.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.538+0000 7faa7259c640 1 --2- 172.21.15.116:0/1037047635 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa741500e0 0x7faa7414cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:39.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.538+0000 7faa7259c640 1 -- 172.21.15.116:0/1037047635 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa740c1c10 msgr2=0x7faa741dcab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:39.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.538+0000 7faa7259c640 1 --2- 172.21.15.116:0/1037047635 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa740c1c10 0x7faa741dcab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:39.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.538+0000 7faa7259c640 1 -- 172.21.15.116:0/1037047635 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa741dcff0 con 0x7faa7406c220 2024-04-25T00:16:39.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.538+0000 7faa71d9b640 1 --2- 172.21.15.116:0/1037047635 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa740c1c10 0x7faa741dcab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:16:39.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.538+0000 7faa7259c640 1 --2- 172.21.15.116:0/1037047635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa7406c220 0x7faa74148df0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7faa6400d9c0 tx=0x7faa6400de80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:39.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.538+0000 7faa62ffd640 1 -- 172.21.15.116:0/1037047635 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa6400ab80 con 0x7faa7406c220 2024-04-25T00:16:39.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.538+0000 7faa62ffd640 1 -- 172.21.15.116:0/1037047635 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa6400ad20 con 0x7faa7406c220 2024-04-25T00:16:39.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.538+0000 7faa78a46640 1 -- 172.21.15.116:0/1037047635 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa7414bbc0 con 0x7faa7406c220 2024-04-25T00:16:39.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.538+0000 7faa62ffd640 1 -- 172.21.15.116:0/1037047635 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa640106b0 con 0x7faa7406c220 2024-04-25T00:16:39.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.538+0000 7faa78a46640 1 -- 172.21.15.116:0/1037047635 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa741493d0 con 0x7faa7406c220 2024-04-25T00:16:39.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.538+0000 7faa78a46640 1 -- 172.21.15.116:0/1037047635 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa74068a60 con 0x7faa7406c220 2024-04-25T00:16:39.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.538+0000 7faa62ffd640 1 -- 172.21.15.116:0/1037047635 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7faa64010850 con 0x7faa7406c220 2024-04-25T00:16:39.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.542+0000 7faa62ffd640 1 --2- 172.21.15.116:0/1037047635 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faa540678c0 0x7faa54069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:39.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.542+0000 7faa62ffd640 1 -- 172.21.15.116:0/1037047635 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 8558+0+0 (secure 0 0 0) 0x7faa64086cc0 con 0x7faa7406c220 2024-04-25T00:16:39.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.542+0000 7faa71d9b640 1 --2- 172.21.15.116:0/1037047635 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faa540678c0 0x7faa54069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:39.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.542+0000 7faa71d9b640 1 --2- 172.21.15.116:0/1037047635 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faa540678c0 0x7faa54069d80 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7faa680b2000 tx=0x7faa68008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:39.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.546+0000 7faa62ffd640 1 -- 172.21.15.116:0/1037047635 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7faa64015030 con 0x7faa7406c220 2024-04-25T00:16:39.551 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:16:39.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.666+0000 7f367b7f4640 1 Processor -- start 2024-04-25T00:16:39.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.666+0000 7f367b7f4640 1 -- start start 2024-04-25T00:16:39.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.666+0000 7f367b7f4640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f367414fe10 0x7f367406c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:39.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.666+0000 7f367b7f4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f367406c760 0x7f3674156180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:39.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.666+0000 7f367b7f4640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36740a6d50 con 0x7f36741566c0 2024-04-25T00:16:39.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.666+0000 7f367b7f4640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f3674155cf0 con 0x7f367414fe10 2024-04-25T00:16:39.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.666+0000 7f367b7f4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f36740c6e90 con 0x7f367406c760 2024-04-25T00:16:39.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.666+0000 7f3679d91640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f36741566c0 0x7f36741525c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:44752/0 (socket says 172.21.15.116:44752) 2024-04-25T00:16:39.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.666+0000 7f3679d91640 1 -- 172.21.15.116:0/489750149 learned_addr learned my addr 172.21.15.116:0/489750149 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:16:39.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.666+0000 7f3679590640 1 --2- 172.21.15.116:0/489750149 >> v2:172.21.15.116:3300/0 conn(0x7f367414fe10 0x7f367406c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:39.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.666+0000 7f3678d8f640 1 --2- 172.21.15.116:0/489750149 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f367406c760 0x7f3674156180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:39.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.666+0000 7f366bfff640 1 -- 172.21.15.116:0/489750149 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3730725439 0 0) 0x7f36740a6d50 con 0x7f36741566c0 2024-04-25T00:16:39.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.666+0000 7f366bfff640 1 -- 172.21.15.116:0/489750149 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3660003530 con 0x7f36741566c0 2024-04-25T00:16:39.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.666+0000 7f3679590640 1 -- 172.21.15.116:0/489750149 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f367406c760 msgr2=0x7f3674156180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:39.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.666+0000 7f3679590640 1 --2- 172.21.15.116:0/489750149 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f367406c760 0x7f3674156180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:39.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.666+0000 7f3679590640 1 -- 172.21.15.116:0/489750149 >> v1:172.21.15.116:6789/0 conn(0x7f36741566c0 legacy=0x7f36741525c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:39.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.666+0000 7f3679590640 1 -- 172.21.15.116:0/489750149 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36740c6bb0 con 0x7f367414fe10 2024-04-25T00:16:39.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.666+0000 7f3678d8f640 1 --2- 172.21.15.116:0/489750149 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f367406c760 0x7f3674156180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:16:39.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.670+0000 7f3679590640 1 --2- 172.21.15.116:0/489750149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f367414fe10 0x7f367406c220 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f36700097c0 tx=0x7f367001e590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f50b2ce00d35663d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:39.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.670+0000 7f366bfff640 1 -- 172.21.15.116:0/489750149 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36700b3070 con 0x7f367414fe10 2024-04-25T00:16:39.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.670+0000 7f366bfff640 1 -- 172.21.15.116:0/489750149 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36700ae600 con 0x7f367414fe10 2024-04-25T00:16:39.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.670+0000 7f366bfff640 1 -- 172.21.15.116:0/489750149 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36700ae9d0 con 0x7f367414fe10 2024-04-25T00:16:39.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.670+0000 7f367b7f4640 1 -- 172.21.15.116:0/489750149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f367414fe10 msgr2=0x7f367406c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:39.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.670+0000 7f367b7f4640 1 --2- 172.21.15.116:0/489750149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f367414fe10 0x7f367406c220 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f36700097c0 tx=0x7f367001e590 comp rx=0 tx=0).stop 2024-04-25T00:16:39.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.670+0000 7f367b7f4640 1 -- 172.21.15.116:0/489750149 shutdown_connections 2024-04-25T00:16:39.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.670+0000 7f367b7f4640 1 --2- 172.21.15.116:0/489750149 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f367406c760 0x7f3674156180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:39.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.670+0000 7f367b7f4640 1 --2- 172.21.15.116:0/489750149 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f367414fe10 0x7f367406c220 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:39.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.670+0000 7f367b7f4640 1 -- 172.21.15.116:0/489750149 >> 172.21.15.116:0/489750149 conn(0x7f36740c42e0 msgr2=0x7f36740c46f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:39.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.670+0000 7f367b7f4640 1 -- 172.21.15.116:0/489750149 shutdown_connections 2024-04-25T00:16:39.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.670+0000 7f367b7f4640 1 -- 172.21.15.116:0/489750149 wait complete. 2024-04-25T00:16:39.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.670+0000 7f367b7f4640 1 Processor -- start 2024-04-25T00:16:39.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.674+0000 7f367b7f4640 1 -- start start 2024-04-25T00:16:39.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.674+0000 7f367b7f4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f367406c760 0x7f36741515d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:39.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.674+0000 7f367b7f4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f367414fe10 0x7f36741ea0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:39.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.674+0000 7f367b7f4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36741566c0 0x7f36741ea620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:39.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.674+0000 7f367b7f4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f36740a72f0 con 0x7f36741566c0 2024-04-25T00:16:39.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.674+0000 7f367b7f4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f36740a7170 con 0x7f367406c760 2024-04-25T00:16:39.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.674+0000 7f367b7f4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f3674071590 con 0x7f367414fe10 2024-04-25T00:16:39.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.674+0000 7f3679d91640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36741566c0 0x7f36741ea620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:39.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.674+0000 7f3679d91640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36741566c0 0x7f36741ea620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52528/0 (socket says 172.21.15.116:52528) 2024-04-25T00:16:39.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.674+0000 7f3679d91640 1 -- 172.21.15.116:0/756309212 learned_addr learned my addr 172.21.15.116:0/756309212 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:16:39.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.674+0000 7f3678d8f640 1 --2- 172.21.15.116:0/756309212 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f367414fe10 0x7f36741ea0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:39.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.674+0000 7f3679590640 1 --2- 172.21.15.116:0/756309212 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f367406c760 0x7f36741515d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:39.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.674+0000 7f3679d91640 1 -- 172.21.15.116:0/756309212 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f367414fe10 msgr2=0x7f36741ea0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:39.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.674+0000 7f3679d91640 1 --2- 172.21.15.116:0/756309212 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f367414fe10 0x7f36741ea0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:39.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.674+0000 7f3679d91640 1 -- 172.21.15.116:0/756309212 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f367406c760 msgr2=0x7f36741515d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:39.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.674+0000 7f3679d91640 1 --2- 172.21.15.116:0/756309212 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f367406c760 0x7f36741515d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:39.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.674+0000 7f3679d91640 1 -- 172.21.15.116:0/756309212 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36740abaa0 con 0x7f36741566c0 2024-04-25T00:16:39.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.674+0000 7f3678d8f640 1 --2- 172.21.15.116:0/756309212 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f367414fe10 0x7f36741ea0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:16:39.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.674+0000 7f3679590640 1 --2- 172.21.15.116:0/756309212 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f367406c760 0x7f36741515d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:16:39.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.674+0000 7f3679d91640 1 --2- 172.21.15.116:0/756309212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36741566c0 0x7f36741ea620 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f366c00caa0 tx=0x7f366c00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:39.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.674+0000 7f3669ffb640 1 -- 172.21.15.116:0/756309212 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f366c017070 con 0x7f36741566c0 2024-04-25T00:16:39.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.674+0000 7f3669ffb640 1 -- 172.21.15.116:0/756309212 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f366c00ac00 con 0x7f36741566c0 2024-04-25T00:16:39.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.674+0000 7f3669ffb640 1 -- 172.21.15.116:0/756309212 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f366c012400 con 0x7f36741566c0 2024-04-25T00:16:39.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.674+0000 7f367b7f4640 1 -- 172.21.15.116:0/756309212 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36741edee0 con 0x7f36741566c0 2024-04-25T00:16:39.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.674+0000 7f367b7f4640 1 -- 172.21.15.116:0/756309212 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36740ab7c0 con 0x7f36741566c0 2024-04-25T00:16:39.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.674+0000 7f367b7f4640 1 -- 172.21.15.116:0/756309212 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3674068a60 con 0x7f36741566c0 2024-04-25T00:16:39.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.674+0000 7f3669ffb640 1 -- 172.21.15.116:0/756309212 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f366c01a070 con 0x7f36741566c0 2024-04-25T00:16:39.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.674+0000 7f3669ffb640 1 --2- 172.21.15.116:0/756309212 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f36580678c0 0x7f3658069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:39.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.674+0000 7f3669ffb640 1 -- 172.21.15.116:0/756309212 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 8558+0+0 (secure 0 0 0) 0x7f366c0864f0 con 0x7f36741566c0 2024-04-25T00:16:39.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.674+0000 7f3679590640 1 --2- 172.21.15.116:0/756309212 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f36580678c0 0x7f3658069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:39.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.678+0000 7f3679590640 1 --2- 172.21.15.116:0/756309212 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f36580678c0 0x7f3658069d80 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f36700b0000 tx=0x7f36700a9000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:39.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.678+0000 7f3669ffb640 1 -- 172.21.15.116:0/756309212 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f366c056550 con 0x7f36741566c0 2024-04-25T00:16:39.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.746+0000 7faa78a46640 1 -- 172.21.15.116:0/1037047635 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faa740c17e0 con 0x7faa7406c220 2024-04-25T00:16:39.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.746+0000 7faa62ffd640 1 -- 172.21.15.116:0/1037047635 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) ==== 95+0+5681 (secure 0 0 0) 0x7faa64056c90 con 0x7faa7406c220 2024-04-25T00:16:39.747 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:16:39.747 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":121,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:15:43.819845+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6810":{"gid":6810,"name":"c","rank":0,"incarnation":113,"state":"up:active","state_seq":13,"addr":"172.21.15.116:6833/2340581953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2340581953},{"type":"v1","addr":"172.21.15.116:6833","nonce":2340581953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6810,"qdb_cluster":[6810]},"id":1},{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:16:38.790885+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5621},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5621":{"gid":5621,"name":"d","rank":0,"incarnation":118,"state":"up:active","state_seq":15,"addr":"172.21.15.174:6837/3081407040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3081407040},{"type":"v1","addr":"172.21.15.174:6837","nonce":3081407040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5621,"qdb_cluster":[5621]},"id":2}]} 2024-04-25T00:16:39.747 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 121 2024-04-25T00:16:39.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.750+0000 7faa78a46640 1 -- 172.21.15.116:0/1037047635 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faa540678c0 msgr2=0x7faa54069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:39.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.750+0000 7faa78a46640 1 --2- 172.21.15.116:0/1037047635 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faa540678c0 0x7faa54069d80 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7faa680b2000 tx=0x7faa68008040 comp rx=0 tx=0).stop 2024-04-25T00:16:39.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.750+0000 7faa78a46640 1 -- 172.21.15.116:0/1037047635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa7406c220 msgr2=0x7faa74148df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:39.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.750+0000 7faa78a46640 1 --2- 172.21.15.116:0/1037047635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa7406c220 0x7faa74148df0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7faa6400d9c0 tx=0x7faa6400de80 comp rx=0 tx=0).stop 2024-04-25T00:16:39.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.750+0000 7faa78a46640 1 -- 172.21.15.116:0/1037047635 shutdown_connections 2024-04-25T00:16:39.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.750+0000 7faa78a46640 1 --2- 172.21.15.116:0/1037047635 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faa540678c0 0x7faa54069d80 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:39.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.750+0000 7faa78a46640 1 --2- 172.21.15.116:0/1037047635 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa741500e0 0x7faa7414cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:39.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.750+0000 7faa78a46640 1 --2- 172.21.15.116:0/1037047635 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa740c1c10 0x7faa741dcab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:39.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.750+0000 7faa78a46640 1 --2- 172.21.15.116:0/1037047635 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa7406c220 0x7faa74148df0 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:39.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.750+0000 7faa78a46640 1 -- 172.21.15.116:0/1037047635 >> 172.21.15.116:0/1037047635 conn(0x7faa740a9b20 msgr2=0x7faa7414ef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:39.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.750+0000 7faa78a46640 1 -- 172.21.15.116:0/1037047635 shutdown_connections 2024-04-25T00:16:39.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.750+0000 7faa78a46640 1 -- 172.21.15.116:0/1037047635 wait complete. 2024-04-25T00:16:39.765 INFO:tasks.mds_thrash.fs.[b]:mds.d has gained rank=0, replacing gid=6828 2024-04-25T00:16:39.765 INFO:tasks.mds_thrash.fs.[b]:waiting for 1 secs before reviving mds.a 2024-04-25T00:16:39.847 INFO:tasks.ceph.mds.d.smithi174.stderr:2024-04-25T00:16:39.848+0000 7f9ce4987640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T00:16:39.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.870+0000 7f367b7f4640 1 -- 172.21.15.116:0/756309212 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3674074980 con 0x7f36741566c0 2024-04-25T00:16:39.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.870+0000 7f3669ffb640 1 -- 172.21.15.116:0/756309212 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) ==== 95+0+5681 (secure 0 0 0) 0x7f366c055e50 con 0x7f36741566c0 2024-04-25T00:16:39.870 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:16:39.870 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":121,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:15:43.819845+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6810":{"gid":6810,"name":"c","rank":0,"incarnation":113,"state":"up:active","state_seq":13,"addr":"172.21.15.116:6833/2340581953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2340581953},{"type":"v1","addr":"172.21.15.116:6833","nonce":2340581953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6810,"qdb_cluster":[6810]},"id":1},{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:16:38.790885+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5621},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5621":{"gid":5621,"name":"d","rank":0,"incarnation":118,"state":"up:active","state_seq":15,"addr":"172.21.15.174:6837/3081407040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3081407040},{"type":"v1","addr":"172.21.15.174:6837","nonce":3081407040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5621,"qdb_cluster":[5621]},"id":2}]} 2024-04-25T00:16:39.870 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 121 2024-04-25T00:16:39.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.874+0000 7f367b7f4640 1 -- 172.21.15.116:0/756309212 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f36580678c0 msgr2=0x7f3658069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:39.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.874+0000 7f367b7f4640 1 --2- 172.21.15.116:0/756309212 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f36580678c0 0x7f3658069d80 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f36700b0000 tx=0x7f36700a9000 comp rx=0 tx=0).stop 2024-04-25T00:16:39.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.874+0000 7f367b7f4640 1 -- 172.21.15.116:0/756309212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36741566c0 msgr2=0x7f36741ea620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:39.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.874+0000 7f367b7f4640 1 --2- 172.21.15.116:0/756309212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36741566c0 0x7f36741ea620 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f366c00caa0 tx=0x7f366c00cf60 comp rx=0 tx=0).stop 2024-04-25T00:16:39.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.874+0000 7f367b7f4640 1 -- 172.21.15.116:0/756309212 shutdown_connections 2024-04-25T00:16:39.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.874+0000 7f367b7f4640 1 --2- 172.21.15.116:0/756309212 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f36580678c0 0x7f3658069d80 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:39.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.874+0000 7f367b7f4640 1 --2- 172.21.15.116:0/756309212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36741566c0 0x7f36741ea620 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:39.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.874+0000 7f367b7f4640 1 --2- 172.21.15.116:0/756309212 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f367414fe10 0x7f36741ea0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:39.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.874+0000 7f367b7f4640 1 --2- 172.21.15.116:0/756309212 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f367406c760 0x7f36741515d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:39.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.874+0000 7f367b7f4640 1 -- 172.21.15.116:0/756309212 >> 172.21.15.116:0/756309212 conn(0x7f36740c42e0 msgr2=0x7f36740a7500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:39.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.874+0000 7f367b7f4640 1 -- 172.21.15.116:0/756309212 shutdown_connections 2024-04-25T00:16:39.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:39.874+0000 7f367b7f4640 1 -- 172.21.15.116:0/756309212 wait complete. 2024-04-25T00:16:39.887 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:16:40.766 INFO:tasks.mds_thrash.fs.[b]:reviving mds.a 2024-04-25T00:16:40.766 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-25T00:16:40.766 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-25T00:16:40.769 INFO:tasks.ceph.mds.a:Started 2024-04-25T00:16:40.769 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:16:40.835 INFO:tasks.ceph.mds.a.smithi116.stdout:starting mds.a at 2024-04-25T00:16:40.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:40.898+0000 7f8ad1068640 1 Processor -- start 2024-04-25T00:16:40.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:40.898+0000 7f8ad1068640 1 -- start start 2024-04-25T00:16:40.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:40.898+0000 7f8ad1068640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8acc0cacc0 0x7f8acc14c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:40.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:40.898+0000 7f8ad1068640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8acc156720 con 0x7f8acc14bf10 2024-04-25T00:16:40.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:40.898+0000 7f8ad1068640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8acc156440 con 0x7f8acc0c5ac0 2024-04-25T00:16:40.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:40.898+0000 7f8ad1068640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f8acc0c8460 con 0x7f8acc0cacc0 2024-04-25T00:16:40.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:40.898+0000 7f8ac3fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8acc0cacc0 0x7f8acc14c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:40.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:40.898+0000 7f8acad9d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f8acc14bf10 0x7f8acc06be70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:44774/0 (socket says 172.21.15.116:44774) 2024-04-25T00:16:40.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:40.898+0000 7f8acad9d640 1 -- 172.21.15.116:0/58378480 learned_addr learned my addr 172.21.15.116:0/58378480 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:16:40.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:40.902+0000 7f8ac3fff640 1 -- 172.21.15.116:0/58378480 >> v1:172.21.15.174:6789/0 conn(0x7f8acc0c5ac0 legacy=0x7f8acc0c9040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:40.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:40.902+0000 7f8ac3fff640 1 -- 172.21.15.116:0/58378480 >> v1:172.21.15.116:6789/0 conn(0x7f8acc14bf10 legacy=0x7f8acc06be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:40.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:40.902+0000 7f8ac3fff640 1 -- 172.21.15.116:0/58378480 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8acc0c7c60 con 0x7f8acc0cacc0 2024-04-25T00:16:40.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:40.902+0000 7f8aca59c640 1 -- 172.21.15.116:0/58378480 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3464222593 0 0) 0x7f8acc156720 con 0x7f8acc14bf10 2024-04-25T00:16:40.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:40.902+0000 7f8ac3fff640 1 --2- 172.21.15.116:0/58378480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8acc0cacc0 0x7f8acc14c2f0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f8ab40097f0 tx=0x7f8ab401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c232e77b2f2e323e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:40.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:40.902+0000 7f8aca59c640 1 -- 172.21.15.116:0/58378480 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ab40b1070 con 0x7f8acc0cacc0 2024-04-25T00:16:40.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:40.902+0000 7f8aca59c640 1 -- 172.21.15.116:0/58378480 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ab40ac410 con 0x7f8acc0cacc0 2024-04-25T00:16:40.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:40.902+0000 7f8aca59c640 1 -- 172.21.15.116:0/58378480 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ab40ac7e0 con 0x7f8acc0cacc0 2024-04-25T00:16:40.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:40.902+0000 7f8ad1068640 1 -- 172.21.15.116:0/58378480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8acc0cacc0 msgr2=0x7f8acc14c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:40.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:40.902+0000 7f8ad1068640 1 --2- 172.21.15.116:0/58378480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8acc0cacc0 0x7f8acc14c2f0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f8ab40097f0 tx=0x7f8ab401c4a0 comp rx=0 tx=0).stop 2024-04-25T00:16:40.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:40.902+0000 7f8ad1068640 1 -- 172.21.15.116:0/58378480 shutdown_connections 2024-04-25T00:16:40.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:40.902+0000 7f8ad1068640 1 --2- 172.21.15.116:0/58378480 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8acc0cacc0 0x7f8acc14c2f0 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:40.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:40.902+0000 7f8ad1068640 1 -- 172.21.15.116:0/58378480 >> 172.21.15.116:0/58378480 conn(0x7f8acc0ace40 msgr2=0x7f8acc0ad250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:40.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:40.902+0000 7f8ad1068640 1 -- 172.21.15.116:0/58378480 shutdown_connections 2024-04-25T00:16:40.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:40.902+0000 7f8ad1068640 1 -- 172.21.15.116:0/58378480 wait complete. 2024-04-25T00:16:40.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:40.906+0000 7f8ad1068640 1 Processor -- start 2024-04-25T00:16:40.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:40.906+0000 7f8ad1068640 1 -- start start 2024-04-25T00:16:40.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:40.906+0000 7f8ad1068640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8acc0c5ac0 0x7f8acc14e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:40.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:40.906+0000 7f8ad1068640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8acc0cacc0 0x7f8acc0c0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:40.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:40.906+0000 7f8ad1068640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8acc14bf10 0x7f8acc0c0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:40.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:40.906+0000 7f8ad1068640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8acc06c700 con 0x7f8acc0cacc0 2024-04-25T00:16:40.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:40.906+0000 7f8ad1068640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8acc06c580 con 0x7f8acc0c5ac0 2024-04-25T00:16:40.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:40.906+0000 7f8ad1068640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f8acc0b07c0 con 0x7f8acc14bf10 2024-04-25T00:16:40.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:40.906+0000 7f8ac3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8acc0cacc0 0x7f8acc0c0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:40.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:40.906+0000 7f8ac3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8acc0cacc0 0x7f8acc0c0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52544/0 (socket says 172.21.15.116:52544) 2024-04-25T00:16:40.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:40.906+0000 7f8ac3fff640 1 -- 172.21.15.116:0/932117385 learned_addr learned my addr 172.21.15.116:0/932117385 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:16:40.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:40.906+0000 7f8acad9d640 1 --2- 172.21.15.116:0/932117385 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8acc0c5ac0 0x7f8acc14e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:40.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:40.906+0000 7f8acb59e640 1 --2- 172.21.15.116:0/932117385 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8acc14bf10 0x7f8acc0c0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:40.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:40.906+0000 7f8ac3fff640 1 -- 172.21.15.116:0/932117385 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8acc14bf10 msgr2=0x7f8acc0c0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:40.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:40.906+0000 7f8ac3fff640 1 --2- 172.21.15.116:0/932117385 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8acc14bf10 0x7f8acc0c0840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:40.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:40.906+0000 7f8ac3fff640 1 -- 172.21.15.116:0/932117385 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8acc0c5ac0 msgr2=0x7f8acc14e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:40.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:40.906+0000 7f8ac3fff640 1 --2- 172.21.15.116:0/932117385 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8acc0c5ac0 0x7f8acc14e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:40.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:40.906+0000 7f8ac3fff640 1 -- 172.21.15.116:0/932117385 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8acc0c27d0 con 0x7f8acc0cacc0 2024-04-25T00:16:40.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:40.906+0000 7f8acb59e640 1 --2- 172.21.15.116:0/932117385 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8acc14bf10 0x7f8acc0c0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:16:40.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:40.906+0000 7f8acad9d640 1 --2- 172.21.15.116:0/932117385 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8acc0c5ac0 0x7f8acc14e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:16:40.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:40.906+0000 7f8ac3fff640 1 --2- 172.21.15.116:0/932117385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8acc0cacc0 0x7f8acc0c0300 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f8ab40097f0 tx=0x7f8ab4019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:40.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:40.906+0000 7f8ac37fe640 1 -- 172.21.15.116:0/932117385 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ab40b1040 con 0x7f8acc0cacc0 2024-04-25T00:16:40.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:40.906+0000 7f8ac37fe640 1 -- 172.21.15.116:0/932117385 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ab40bb070 con 0x7f8acc0cacc0 2024-04-25T00:16:40.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:40.906+0000 7f8ad1068640 1 -- 172.21.15.116:0/932117385 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8acc1f2e20 con 0x7f8acc0cacc0 2024-04-25T00:16:40.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:40.906+0000 7f8ac37fe640 1 -- 172.21.15.116:0/932117385 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ab4006e20 con 0x7f8acc0cacc0 2024-04-25T00:16:40.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:40.906+0000 7f8ad1068640 1 -- 172.21.15.116:0/932117385 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8acc0c24f0 con 0x7f8acc0cacc0 2024-04-25T00:16:40.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:40.906+0000 7f8ad1068640 1 -- 172.21.15.116:0/932117385 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8acc068350 con 0x7f8acc0cacc0 2024-04-25T00:16:40.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:40.906+0000 7f8ac37fe640 1 -- 172.21.15.116:0/932117385 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8ab40b5650 con 0x7f8acc0cacc0 2024-04-25T00:16:40.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:40.906+0000 7f8ac37fe640 1 --2- 172.21.15.116:0/932117385 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8aac0678f0 0x7f8aac069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:40.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:40.906+0000 7f8ac37fe640 1 -- 172.21.15.116:0/932117385 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 8558+0+0 (secure 0 0 0) 0x7f8ab41203a0 con 0x7f8acc0cacc0 2024-04-25T00:16:40.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:40.910+0000 7f8acad9d640 1 --2- 172.21.15.116:0/932117385 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8aac0678f0 0x7f8aac069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:40.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:40.910+0000 7f8acad9d640 1 --2- 172.21.15.116:0/932117385 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8aac0678f0 0x7f8aac069db0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f8abc000f70 tx=0x7f8abc007dd0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:40.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:40.914+0000 7f8ac37fe640 1 -- 172.21.15.116:0/932117385 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f8ab4016090 con 0x7f8acc0cacc0 2024-04-25T00:16:41.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:41.106+0000 7f8ad1068640 1 -- 172.21.15.116:0/932117385 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8acc0caaa0 con 0x7f8acc0cacc0 2024-04-25T00:16:41.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:41.110+0000 7f8ac37fe640 1 -- 172.21.15.116:0/932117385 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) ==== 95+0+5681 (secure 0 0 0) 0x7f8ab40f0310 con 0x7f8acc0cacc0 2024-04-25T00:16:41.108 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:16:41.109 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":121,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:15:43.819845+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6810":{"gid":6810,"name":"c","rank":0,"incarnation":113,"state":"up:active","state_seq":13,"addr":"172.21.15.116:6833/2340581953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2340581953},{"type":"v1","addr":"172.21.15.116:6833","nonce":2340581953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6810,"qdb_cluster":[6810]},"id":1},{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:16:38.790885+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5621},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5621":{"gid":5621,"name":"d","rank":0,"incarnation":118,"state":"up:active","state_seq":15,"addr":"172.21.15.174:6837/3081407040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3081407040},{"type":"v1","addr":"172.21.15.174:6837","nonce":3081407040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5621,"qdb_cluster":[5621]},"id":2}]} 2024-04-25T00:16:41.109 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 121 2024-04-25T00:16:41.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:41.110+0000 7f8ad1068640 1 -- 172.21.15.116:0/932117385 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8aac0678f0 msgr2=0x7f8aac069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:41.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:41.110+0000 7f8ad1068640 1 --2- 172.21.15.116:0/932117385 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8aac0678f0 0x7f8aac069db0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f8abc000f70 tx=0x7f8abc007dd0 comp rx=0 tx=0).stop 2024-04-25T00:16:41.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:41.110+0000 7f8ad1068640 1 -- 172.21.15.116:0/932117385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8acc0cacc0 msgr2=0x7f8acc0c0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:41.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:41.110+0000 7f8ad1068640 1 --2- 172.21.15.116:0/932117385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8acc0cacc0 0x7f8acc0c0300 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f8ab40097f0 tx=0x7f8ab4019240 comp rx=0 tx=0).stop 2024-04-25T00:16:41.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:41.110+0000 7f8ad1068640 1 -- 172.21.15.116:0/932117385 shutdown_connections 2024-04-25T00:16:41.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:41.110+0000 7f8ad1068640 1 --2- 172.21.15.116:0/932117385 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8aac0678f0 0x7f8aac069db0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:41.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:41.110+0000 7f8ad1068640 1 --2- 172.21.15.116:0/932117385 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8acc14bf10 0x7f8acc0c0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:41.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:41.110+0000 7f8ad1068640 1 --2- 172.21.15.116:0/932117385 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8acc0cacc0 0x7f8acc0c0300 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:41.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:41.110+0000 7f8ad1068640 1 --2- 172.21.15.116:0/932117385 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8acc0c5ac0 0x7f8acc14e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:41.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:41.110+0000 7f8ad1068640 1 -- 172.21.15.116:0/932117385 >> 172.21.15.116:0/932117385 conn(0x7f8acc0ace40 msgr2=0x7f8acc154c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:41.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:41.110+0000 7f8ad1068640 1 -- 172.21.15.116:0/932117385 shutdown_connections 2024-04-25T00:16:41.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:41.110+0000 7f8ad1068640 1 -- 172.21.15.116:0/932117385 wait complete. 2024-04-25T00:16:41.127 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2024-04-25T00:16:41.888 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:16:42.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.022+0000 7fd3d1c33640 1 Processor -- start 2024-04-25T00:16:42.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.022+0000 7fd3d1c33640 1 -- start start 2024-04-25T00:16:42.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.022+0000 7fd3d1c33640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fd3cc0a94a0 0x7fd3cc1538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:42.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.022+0000 7fd3d1c33640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3cc0c3910 con 0x7fd3cc0a99f0 2024-04-25T00:16:42.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.022+0000 7fd3d1c33640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3cc154530 con 0x7fd3cc150d20 2024-04-25T00:16:42.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.022+0000 7fd3d1c33640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fd3cc0c16e0 con 0x7fd3cc0a94a0 2024-04-25T00:16:42.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.022+0000 7fd3cbfff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fd3cc0a99f0 0x7fd3cc153e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:44782/0 (socket says 172.21.15.116:44782) 2024-04-25T00:16:42.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.022+0000 7fd3cbfff640 1 -- 172.21.15.116:0/998235672 learned_addr learned my addr 172.21.15.116:0/998235672 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:16:42.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.022+0000 7fd3c2ffd640 1 --2- 172.21.15.116:0/998235672 >> v2:172.21.15.174:3300/0 conn(0x7fd3cc0a94a0 0x7fd3cc1538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:42.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.022+0000 7fd3caffd640 1 -- 172.21.15.116:0/998235672 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3822152668 0 0) 0x7fd3cc0c3910 con 0x7fd3cc0a99f0 2024-04-25T00:16:42.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.022+0000 7fd3caffd640 1 -- 172.21.15.116:0/998235672 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd3b8003530 con 0x7fd3cc0a99f0 2024-04-25T00:16:42.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.022+0000 7fd3caffd640 1 -- 172.21.15.116:0/998235672 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2585927884 0 0) 0x7fd3b8003530 con 0x7fd3cc0a99f0 2024-04-25T00:16:42.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.022+0000 7fd3caffd640 1 -- 172.21.15.116:0/998235672 >> v2:172.21.15.174:3300/0 conn(0x7fd3cc0a94a0 msgr2=0x7fd3cc1538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:42.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.022+0000 7fd3caffd640 1 --2- 172.21.15.116:0/998235672 >> v2:172.21.15.174:3300/0 conn(0x7fd3cc0a94a0 0x7fd3cc1538e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:42.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.022+0000 7fd3caffd640 1 -- 172.21.15.116:0/998235672 >> v1:172.21.15.174:6789/0 conn(0x7fd3cc150d20 legacy=0x7fd3cc0c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:42.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.022+0000 7fd3caffd640 1 -- 172.21.15.116:0/998235672 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3cc0c1400 con 0x7fd3cc0a99f0 2024-04-25T00:16:42.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.022+0000 7fd3caffd640 1 -- 172.21.15.116:0/998235672 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7fd3c4001a00 con 0x7fd3cc0a99f0 2024-04-25T00:16:42.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.022+0000 7fd3caffd640 1 -- 172.21.15.116:0/998235672 >> v1:172.21.15.116:6789/0 conn(0x7fd3cc0a99f0 legacy=0x7fd3cc153e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:42.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.022+0000 7fd3caffd640 1 --2- 172.21.15.116:0/998235672 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd3b80039c0 0x7fd3b8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:42.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.022+0000 7fd3caffd640 1 --2- 172.21.15.116:0/998235672 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3b8004810 0x7fd3b8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:42.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.022+0000 7fd3caffd640 1 --2- 172.21.15.116:0/998235672 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3b80051d0 0x7fd3b800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:42.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.022+0000 7fd3caffd640 1 -- 172.21.15.116:0/998235672 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd3cc0c16e0 con 0x7fd3b80051d0 2024-04-25T00:16:42.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.022+0000 7fd3caffd640 1 -- 172.21.15.116:0/998235672 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd3b800c070 con 0x7fd3b80039c0 2024-04-25T00:16:42.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.022+0000 7fd3caffd640 1 -- 172.21.15.116:0/998235672 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fd3b800c270 con 0x7fd3b8004810 2024-04-25T00:16:42.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.022+0000 7fd3cbfff640 1 --2- 172.21.15.116:0/998235672 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3b80051d0 0x7fd3b800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:42.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.022+0000 7fd3cb7fe640 1 --2- 172.21.15.116:0/998235672 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd3b80039c0 0x7fd3b8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:42.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.022+0000 7fd3cbfff640 1 -- 172.21.15.116:0/998235672 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3b8004810 msgr2=0x7fd3b8004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:42.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.022+0000 7fd3cbfff640 1 --2- 172.21.15.116:0/998235672 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3b8004810 0x7fd3b8004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:42.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.022+0000 7fd3cbfff640 1 -- 172.21.15.116:0/998235672 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd3b80039c0 msgr2=0x7fd3b8003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:42.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.022+0000 7fd3cbfff640 1 --2- 172.21.15.116:0/998235672 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd3b80039c0 0x7fd3b8003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:42.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.022+0000 7fd3cbfff640 1 -- 172.21.15.116:0/998235672 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3b8003530 con 0x7fd3b80051d0 2024-04-25T00:16:42.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.026+0000 7fd3cbfff640 1 -- 172.21.15.116:0/998235672 reap_dead start 2024-04-25T00:16:42.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.026+0000 7fd3cbfff640 1 --2- 172.21.15.116:0/998235672 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3b80051d0 0x7fd3b800b9d0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fd3c400a850 tx=0x7fd3c401d3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a96a5c8e6627616 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:42.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.026+0000 7fd3caffd640 1 -- 172.21.15.116:0/998235672 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3c400a4a0 con 0x7fd3b80051d0 2024-04-25T00:16:42.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.026+0000 7fd3caffd640 1 -- 172.21.15.116:0/998235672 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3c40ac640 con 0x7fd3b80051d0 2024-04-25T00:16:42.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.026+0000 7fd3caffd640 1 -- 172.21.15.116:0/998235672 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3c40aca30 con 0x7fd3b80051d0 2024-04-25T00:16:42.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.026+0000 7fd3d1c33640 1 -- 172.21.15.116:0/998235672 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3b80051d0 msgr2=0x7fd3b800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:42.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.026+0000 7fd3d1c33640 1 --2- 172.21.15.116:0/998235672 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3b80051d0 0x7fd3b800b9d0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fd3c400a850 tx=0x7fd3c401d3a0 comp rx=0 tx=0).stop 2024-04-25T00:16:42.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.026+0000 7fd3d1c33640 1 -- 172.21.15.116:0/998235672 shutdown_connections 2024-04-25T00:16:42.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.026+0000 7fd3d1c33640 1 --2- 172.21.15.116:0/998235672 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3b80051d0 0x7fd3b800b9d0 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:42.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.026+0000 7fd3d1c33640 1 -- 172.21.15.116:0/998235672 >> 172.21.15.116:0/998235672 conn(0x7fd3cc0c3b00 msgr2=0x7fd3cc0c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:42.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.026+0000 7fd3d1c33640 1 -- 172.21.15.116:0/998235672 shutdown_connections 2024-04-25T00:16:42.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.026+0000 7fd3d1c33640 1 -- 172.21.15.116:0/998235672 wait complete. 2024-04-25T00:16:42.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.026+0000 7fd3d1c33640 1 Processor -- start 2024-04-25T00:16:42.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.026+0000 7fd3d1c33640 1 -- start start 2024-04-25T00:16:42.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.026+0000 7fd3d1c33640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3b80051d0 0x7fd3cc14d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:42.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.026+0000 7fd3d1c33640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3cc1ecbd0 0x7fd3cc154be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:42.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.026+0000 7fd3d1c33640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd3cc14dc30 0x7fd3cc0c4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:42.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.026+0000 7fd3d1c33640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd3cc06e940 con 0x7fd3b80051d0 2024-04-25T00:16:42.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.026+0000 7fd3d1c33640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd3cc0c1280 con 0x7fd3cc14dc30 2024-04-25T00:16:42.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.026+0000 7fd3d1c33640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fd3cc06ec20 con 0x7fd3cc1ecbd0 2024-04-25T00:16:42.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.026+0000 7fd3cb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3b80051d0 0x7fd3cc14d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:42.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.026+0000 7fd3cb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3b80051d0 0x7fd3cc14d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52560/0 (socket says 172.21.15.116:52560) 2024-04-25T00:16:42.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.026+0000 7fd3cb7fe640 1 -- 172.21.15.116:0/3747930894 learned_addr learned my addr 172.21.15.116:0/3747930894 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:16:42.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.026+0000 7fd3cbfff640 1 --2- 172.21.15.116:0/3747930894 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd3cc14dc30 0x7fd3cc0c4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:42.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.026+0000 7fd3c2ffd640 1 --2- 172.21.15.116:0/3747930894 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3cc1ecbd0 0x7fd3cc154be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:42.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.026+0000 7fd3cb7fe640 1 -- 172.21.15.116:0/3747930894 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3cc1ecbd0 msgr2=0x7fd3cc154be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:42.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.026+0000 7fd3cb7fe640 1 --2- 172.21.15.116:0/3747930894 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3cc1ecbd0 0x7fd3cc154be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:42.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.026+0000 7fd3cb7fe640 1 -- 172.21.15.116:0/3747930894 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd3cc14dc30 msgr2=0x7fd3cc0c4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:42.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.026+0000 7fd3cb7fe640 1 --2- 172.21.15.116:0/3747930894 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd3cc14dc30 0x7fd3cc0c4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:42.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.026+0000 7fd3cb7fe640 1 -- 172.21.15.116:0/3747930894 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3cc14c7d0 con 0x7fd3b80051d0 2024-04-25T00:16:42.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.026+0000 7fd3c2ffd640 1 --2- 172.21.15.116:0/3747930894 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3cc1ecbd0 0x7fd3cc154be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:16:42.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.026+0000 7fd3cb7fe640 1 --2- 172.21.15.116:0/3747930894 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3b80051d0 0x7fd3cc14d590 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc00e500 tx=0x7fd3bc00e9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:42.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.026+0000 7fd3c8ff9640 1 -- 172.21.15.116:0/3747930894 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3bc0052e0 con 0x7fd3b80051d0 2024-04-25T00:16:42.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.030+0000 7fd3c8ff9640 1 -- 172.21.15.116:0/3747930894 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3bc018070 con 0x7fd3b80051d0 2024-04-25T00:16:42.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.030+0000 7fd3d1c33640 1 -- 172.21.15.116:0/3747930894 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3cc14cab0 con 0x7fd3b80051d0 2024-04-25T00:16:42.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.030+0000 7fd3c8ff9640 1 -- 172.21.15.116:0/3747930894 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3bc003820 con 0x7fd3b80051d0 2024-04-25T00:16:42.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.030+0000 7fd3d1c33640 1 -- 172.21.15.116:0/3747930894 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3cc155120 con 0x7fd3b80051d0 2024-04-25T00:16:42.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.030+0000 7fd3d1c33640 1 -- 172.21.15.116:0/3747930894 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3cc068350 con 0x7fd3b80051d0 2024-04-25T00:16:42.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.030+0000 7fd3c8ff9640 1 -- 172.21.15.116:0/3747930894 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd3bc0039c0 con 0x7fd3b80051d0 2024-04-25T00:16:42.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.030+0000 7fd3c8ff9640 1 --2- 172.21.15.116:0/3747930894 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd3ac0678f0 0x7fd3ac069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:42.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.030+0000 7fd3c8ff9640 1 -- 172.21.15.116:0/3747930894 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 8558+0+0 (secure 0 0 0) 0x7fd3bc0860c0 con 0x7fd3b80051d0 2024-04-25T00:16:42.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.030+0000 7fd3c2ffd640 1 --2- 172.21.15.116:0/3747930894 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd3ac0678f0 0x7fd3ac069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:42.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.030+0000 7fd3c2ffd640 1 --2- 172.21.15.116:0/3747930894 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd3ac0678f0 0x7fd3ac069db0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fd3b40047c0 tx=0x7fd3b4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:42.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.034+0000 7fd3c8ff9640 1 -- 172.21.15.116:0/3747930894 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fd3bc005480 con 0x7fd3b80051d0 2024-04-25T00:16:42.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.226+0000 7fd3d1c33640 1 -- 172.21.15.116:0/3747930894 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd3cc0c34f0 con 0x7fd3b80051d0 2024-04-25T00:16:42.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.226+0000 7fd3c8ff9640 1 -- 172.21.15.116:0/3747930894 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) ==== 95+0+6510 (secure 0 0 0) 0x7fd3bc0560a0 con 0x7fd3b80051d0 2024-04-25T00:16:42.227 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:16:42.228 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":122,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7119,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/3645553116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3645553116},{"type":"v1","addr":"172.21.15.116:6833","nonce":3645553116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":122}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:15:43.819845+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6810":{"gid":6810,"name":"c","rank":0,"incarnation":113,"state":"up:active","state_seq":13,"addr":"172.21.15.116:6833/2340581953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2340581953},{"type":"v1","addr":"172.21.15.116:6833","nonce":2340581953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6810,"qdb_cluster":[6810]},"id":1},{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:16:38.790885+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5621},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5621":{"gid":5621,"name":"d","rank":0,"incarnation":118,"state":"up:active","state_seq":15,"addr":"172.21.15.174:6837/3081407040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3081407040},{"type":"v1","addr":"172.21.15.174:6837","nonce":3081407040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5621,"qdb_cluster":[5621]},"id":2}]} 2024-04-25T00:16:42.228 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 122 2024-04-25T00:16:42.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.230+0000 7fd3d1c33640 1 -- 172.21.15.116:0/3747930894 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd3ac0678f0 msgr2=0x7fd3ac069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:42.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.230+0000 7fd3d1c33640 1 --2- 172.21.15.116:0/3747930894 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd3ac0678f0 0x7fd3ac069db0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fd3b40047c0 tx=0x7fd3b4008040 comp rx=0 tx=0).stop 2024-04-25T00:16:42.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.230+0000 7fd3d1c33640 1 -- 172.21.15.116:0/3747930894 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3b80051d0 msgr2=0x7fd3cc14d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:42.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.230+0000 7fd3d1c33640 1 --2- 172.21.15.116:0/3747930894 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3b80051d0 0x7fd3cc14d590 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc00e500 tx=0x7fd3bc00e9c0 comp rx=0 tx=0).stop 2024-04-25T00:16:42.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.230+0000 7fd3d1c33640 1 -- 172.21.15.116:0/3747930894 shutdown_connections 2024-04-25T00:16:42.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.230+0000 7fd3d1c33640 1 --2- 172.21.15.116:0/3747930894 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd3ac0678f0 0x7fd3ac069db0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:42.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.230+0000 7fd3d1c33640 1 --2- 172.21.15.116:0/3747930894 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd3cc14dc30 0x7fd3cc0c4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:42.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.230+0000 7fd3d1c33640 1 --2- 172.21.15.116:0/3747930894 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3cc1ecbd0 0x7fd3cc154be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:42.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.230+0000 7fd3d1c33640 1 --2- 172.21.15.116:0/3747930894 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3b80051d0 0x7fd3cc14d590 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:42.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.230+0000 7fd3d1c33640 1 -- 172.21.15.116:0/3747930894 >> 172.21.15.116:0/3747930894 conn(0x7fd3cc0c3b00 msgr2=0x7fd3cc06d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:42.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.230+0000 7fd3d1c33640 1 -- 172.21.15.116:0/3747930894 shutdown_connections 2024-04-25T00:16:42.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:42.230+0000 7fd3d1c33640 1 -- 172.21.15.116:0/3747930894 wait complete. 2024-04-25T00:16:42.245 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:16:43.128 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:16:43.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.262+0000 7f40a1f59640 1 Processor -- start 2024-04-25T00:16:43.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.262+0000 7f40a1f59640 1 -- start start 2024-04-25T00:16:43.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.262+0000 7f40a1f59640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f409c0a94a0 0x7f409c1538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:43.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.262+0000 7f40a1f59640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f409c0c3910 con 0x7f409c150d20 2024-04-25T00:16:43.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.262+0000 7f40a1f59640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f409c154530 con 0x7f409c0a99f0 2024-04-25T00:16:43.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.262+0000 7f40a1f59640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f409c0c16e0 con 0x7f409c0a94a0 2024-04-25T00:16:43.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.262+0000 7f409b7fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f409c150d20 0x7f409c0c2a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:44790/0 (socket says 172.21.15.116:44790) 2024-04-25T00:16:43.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.262+0000 7f409b7fe640 1 -- 172.21.15.116:0/3147177464 learned_addr learned my addr 172.21.15.116:0/3147177464 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:16:43.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.262+0000 7f409affd640 1 --2- 172.21.15.116:0/3147177464 >> v2:172.21.15.174:3300/0 conn(0x7f409c0a94a0 0x7f409c1538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:43.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.262+0000 7f409a7fc640 1 -- 172.21.15.116:0/3147177464 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3874239278 0 0) 0x7f409c0c3910 con 0x7f409c150d20 2024-04-25T00:16:43.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.262+0000 7f409a7fc640 1 -- 172.21.15.116:0/3147177464 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4084003530 con 0x7f409c150d20 2024-04-25T00:16:43.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.262+0000 7f409a7fc640 1 -- 172.21.15.116:0/3147177464 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 861397900 0 0) 0x7f4084003530 con 0x7f409c150d20 2024-04-25T00:16:43.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.262+0000 7f409a7fc640 1 -- 172.21.15.116:0/3147177464 >> v2:172.21.15.174:3300/0 conn(0x7f409c0a94a0 msgr2=0x7f409c1538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:43.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.262+0000 7f409a7fc640 1 --2- 172.21.15.116:0/3147177464 >> v2:172.21.15.174:3300/0 conn(0x7f409c0a94a0 0x7f409c1538e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:43.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.262+0000 7f409a7fc640 1 -- 172.21.15.116:0/3147177464 >> v1:172.21.15.174:6789/0 conn(0x7f409c0a99f0 legacy=0x7f409c153e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:43.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.262+0000 7f409a7fc640 1 -- 172.21.15.116:0/3147177464 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f409c0c1400 con 0x7f409c150d20 2024-04-25T00:16:43.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.262+0000 7f409a7fc640 1 -- 172.21.15.116:0/3147177464 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f408c001680 con 0x7f409c150d20 2024-04-25T00:16:43.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.262+0000 7f409a7fc640 1 -- 172.21.15.116:0/3147177464 >> v1:172.21.15.116:6789/0 conn(0x7f409c150d20 legacy=0x7f409c0c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:43.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.262+0000 7f409a7fc640 1 --2- 172.21.15.116:0/3147177464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40840039c0 0x7f4084003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:43.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.262+0000 7f409a7fc640 1 --2- 172.21.15.116:0/3147177464 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4084004810 0x7f4084004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:43.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.262+0000 7f409a7fc640 1 --2- 172.21.15.116:0/3147177464 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f40840051d0 0x7f408400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:43.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.262+0000 7f409a7fc640 1 -- 172.21.15.116:0/3147177464 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f409c0c16e0 con 0x7f40840039c0 2024-04-25T00:16:43.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.262+0000 7f409a7fc640 1 -- 172.21.15.116:0/3147177464 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f408400c070 con 0x7f4084004810 2024-04-25T00:16:43.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.262+0000 7f409a7fc640 1 -- 172.21.15.116:0/3147177464 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f408400c270 con 0x7f40840051d0 2024-04-25T00:16:43.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.262+0000 7f409b7fe640 1 --2- 172.21.15.116:0/3147177464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40840039c0 0x7f4084003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:43.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.262+0000 7f409b7fe640 1 -- 172.21.15.116:0/3147177464 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f40840051d0 msgr2=0x7f408400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:43.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.262+0000 7f409b7fe640 1 --2- 172.21.15.116:0/3147177464 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f40840051d0 0x7f408400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:43.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.262+0000 7f409b7fe640 1 -- 172.21.15.116:0/3147177464 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4084004810 msgr2=0x7f4084004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:43.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.262+0000 7f409b7fe640 1 --2- 172.21.15.116:0/3147177464 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4084004810 0x7f4084004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:43.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.262+0000 7f409bfff640 1 -- 172.21.15.116:0/3147177464 reap_dead start 2024-04-25T00:16:43.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.262+0000 7f409b7fe640 1 -- 172.21.15.116:0/3147177464 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4084003530 con 0x7f40840039c0 2024-04-25T00:16:43.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.266+0000 7f409b7fe640 1 --2- 172.21.15.116:0/3147177464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40840039c0 0x7f4084003de0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f408c00c210 tx=0x7f408c01cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99cb37e5725b0223 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:43.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.266+0000 7f409a7fc640 1 -- 172.21.15.116:0/3147177464 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f408c007d40 con 0x7f40840039c0 2024-04-25T00:16:43.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.266+0000 7f409a7fc640 1 -- 172.21.15.116:0/3147177464 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f408c0a9070 con 0x7f40840039c0 2024-04-25T00:16:43.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.266+0000 7f409a7fc640 1 -- 172.21.15.116:0/3147177464 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f408c005ae0 con 0x7f40840039c0 2024-04-25T00:16:43.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.266+0000 7f40a1f59640 1 -- 172.21.15.116:0/3147177464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40840039c0 msgr2=0x7f4084003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:43.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.266+0000 7f40a1f59640 1 --2- 172.21.15.116:0/3147177464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40840039c0 0x7f4084003de0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f408c00c210 tx=0x7f408c01cca0 comp rx=0 tx=0).stop 2024-04-25T00:16:43.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.266+0000 7f40a1f59640 1 -- 172.21.15.116:0/3147177464 shutdown_connections 2024-04-25T00:16:43.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.266+0000 7f40a1f59640 1 --2- 172.21.15.116:0/3147177464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f40840039c0 0x7f4084003de0 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:43.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.266+0000 7f40a1f59640 1 -- 172.21.15.116:0/3147177464 >> 172.21.15.116:0/3147177464 conn(0x7f409c0c3b00 msgr2=0x7f409c0c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:43.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.266+0000 7f40a1f59640 1 -- 172.21.15.116:0/3147177464 shutdown_connections 2024-04-25T00:16:43.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.266+0000 7f40a1f59640 1 -- 172.21.15.116:0/3147177464 wait complete. 2024-04-25T00:16:43.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.266+0000 7f40a1f59640 1 Processor -- start 2024-04-25T00:16:43.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.266+0000 7f40a1f59640 1 -- start start 2024-04-25T00:16:43.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.266+0000 7f40a1f59640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f40840039c0 0x7f409c14d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:43.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.266+0000 7f40a1f59640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f409c14c760 0x7f409c14d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:43.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.266+0000 7f40a1f59640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f409c155840 0x7f409c1ec360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:43.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.266+0000 7f40a1f59640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f409c153a60 con 0x7f409c155840 2024-04-25T00:16:43.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.266+0000 7f40a1f59640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f409c1538e0 con 0x7f409c14c760 2024-04-25T00:16:43.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.266+0000 7f40a1f59640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f409c153be0 con 0x7f40840039c0 2024-04-25T00:16:43.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.270+0000 7f409bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f409c155840 0x7f409c1ec360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:43.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.270+0000 7f409bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f409c155840 0x7f409c1ec360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52578/0 (socket says 172.21.15.116:52578) 2024-04-25T00:16:43.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.270+0000 7f409bfff640 1 -- 172.21.15.116:0/385380848 learned_addr learned my addr 172.21.15.116:0/385380848 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:16:43.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.270+0000 7f409b7fe640 1 --2- 172.21.15.116:0/385380848 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f40840039c0 0x7f409c14d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:43.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.270+0000 7f409affd640 1 --2- 172.21.15.116:0/385380848 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f409c14c760 0x7f409c14d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:43.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.270+0000 7f409bfff640 1 -- 172.21.15.116:0/385380848 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f40840039c0 msgr2=0x7f409c14d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:43.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.270+0000 7f409bfff640 1 --2- 172.21.15.116:0/385380848 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f40840039c0 0x7f409c14d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:43.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.270+0000 7f409bfff640 1 -- 172.21.15.116:0/385380848 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f409c14c760 msgr2=0x7f409c14d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:43.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.270+0000 7f409bfff640 1 --2- 172.21.15.116:0/385380848 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f409c14c760 0x7f409c14d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:43.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.270+0000 7f409bfff640 1 -- 172.21.15.116:0/385380848 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f409c14dd90 con 0x7f409c155840 2024-04-25T00:16:43.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.270+0000 7f409bfff640 1 --2- 172.21.15.116:0/385380848 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f409c155840 0x7f409c1ec360 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f409c0c4aa0 tx=0x7f409400cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:43.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.270+0000 7f4083fff640 1 -- 172.21.15.116:0/385380848 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4094017070 con 0x7f409c155840 2024-04-25T00:16:43.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.270+0000 7f4083fff640 1 -- 172.21.15.116:0/385380848 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4094005d50 con 0x7f409c155840 2024-04-25T00:16:43.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.270+0000 7f40a1f59640 1 -- 172.21.15.116:0/385380848 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f409c1ef890 con 0x7f409c155840 2024-04-25T00:16:43.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.270+0000 7f4083fff640 1 -- 172.21.15.116:0/385380848 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f409400a880 con 0x7f409c155840 2024-04-25T00:16:43.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.270+0000 7f40a1f59640 1 -- 172.21.15.116:0/385380848 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f409c1efe50 con 0x7f409c155840 2024-04-25T00:16:43.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.270+0000 7f40a1f59640 1 -- 172.21.15.116:0/385380848 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f409c068350 con 0x7f409c155840 2024-04-25T00:16:43.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.270+0000 7f4083fff640 1 -- 172.21.15.116:0/385380848 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f409401a070 con 0x7f409c155840 2024-04-25T00:16:43.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.270+0000 7f4083fff640 1 --2- 172.21.15.116:0/385380848 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f40880678c0 0x7f4088069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:43.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.270+0000 7f4083fff640 1 -- 172.21.15.116:0/385380848 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 8558+0+0 (secure 0 0 0) 0x7f4094085980 con 0x7f409c155840 2024-04-25T00:16:43.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.270+0000 7f409b7fe640 1 --2- 172.21.15.116:0/385380848 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f40880678c0 0x7f4088069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:43.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.270+0000 7f409b7fe640 1 --2- 172.21.15.116:0/385380848 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f40880678c0 0x7f4088069d80 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f408c0ab000 tx=0x7f408c0b0040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:43.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.274+0000 7f4083fff640 1 -- 172.21.15.116:0/385380848 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f40940559e0 con 0x7f409c155840 2024-04-25T00:16:43.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.466+0000 7f40a1f59640 1 -- 172.21.15.116:0/385380848 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f409c0c34f0 con 0x7f409c155840 2024-04-25T00:16:43.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.466+0000 7f4083fff640 1 -- 172.21.15.116:0/385380848 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) ==== 95+0+6510 (secure 0 0 0) 0x7f40940552e0 con 0x7f409c155840 2024-04-25T00:16:43.467 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:16:43.467 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":122,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7119,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/3645553116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3645553116},{"type":"v1","addr":"172.21.15.116:6833","nonce":3645553116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":122}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:15:43.819845+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6810":{"gid":6810,"name":"c","rank":0,"incarnation":113,"state":"up:active","state_seq":13,"addr":"172.21.15.116:6833/2340581953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2340581953},{"type":"v1","addr":"172.21.15.116:6833","nonce":2340581953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6810,"qdb_cluster":[6810]},"id":1},{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:16:38.790885+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5621},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5621":{"gid":5621,"name":"d","rank":0,"incarnation":118,"state":"up:active","state_seq":15,"addr":"172.21.15.174:6837/3081407040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3081407040},{"type":"v1","addr":"172.21.15.174:6837","nonce":3081407040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5621,"qdb_cluster":[5621]},"id":2}]} 2024-04-25T00:16:43.468 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 122 2024-04-25T00:16:43.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.470+0000 7f40a1f59640 1 -- 172.21.15.116:0/385380848 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f40880678c0 msgr2=0x7f4088069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:43.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.470+0000 7f40a1f59640 1 --2- 172.21.15.116:0/385380848 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f40880678c0 0x7f4088069d80 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f408c0ab000 tx=0x7f408c0b0040 comp rx=0 tx=0).stop 2024-04-25T00:16:43.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.470+0000 7f40a1f59640 1 -- 172.21.15.116:0/385380848 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f409c155840 msgr2=0x7f409c1ec360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:43.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.470+0000 7f40a1f59640 1 --2- 172.21.15.116:0/385380848 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f409c155840 0x7f409c1ec360 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f409c0c4aa0 tx=0x7f409400cb90 comp rx=0 tx=0).stop 2024-04-25T00:16:43.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.470+0000 7f40a1f59640 1 -- 172.21.15.116:0/385380848 shutdown_connections 2024-04-25T00:16:43.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.470+0000 7f40a1f59640 1 --2- 172.21.15.116:0/385380848 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f40880678c0 0x7f4088069d80 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:43.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.470+0000 7f40a1f59640 1 --2- 172.21.15.116:0/385380848 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f409c155840 0x7f409c1ec360 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:43.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.470+0000 7f40a1f59640 1 --2- 172.21.15.116:0/385380848 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f409c14c760 0x7f409c14d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:43.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.470+0000 7f40a1f59640 1 --2- 172.21.15.116:0/385380848 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f40840039c0 0x7f409c14d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:43.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.470+0000 7f40a1f59640 1 -- 172.21.15.116:0/385380848 >> 172.21.15.116:0/385380848 conn(0x7f409c0c3b00 msgr2=0x7f409c06cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:43.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.470+0000 7f40a1f59640 1 -- 172.21.15.116:0/385380848 shutdown_connections 2024-04-25T00:16:43.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:43.470+0000 7f40a1f59640 1 -- 172.21.15.116:0/385380848 wait complete. 2024-04-25T00:16:43.485 INFO:tasks.mds_thrash.fs.[b]:mds.a reported in up:standby state 2024-04-25T00:16:43.485 INFO:tasks.mds_thrash.fs.[b]:waiting for 23 secs before thrashing 2024-04-25T00:16:44.245 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:16:44.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.378+0000 7fa9c2e51640 1 Processor -- start 2024-04-25T00:16:44.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.378+0000 7fa9c2e51640 1 -- start start 2024-04-25T00:16:44.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.382+0000 7fa9c2e51640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fa9bc150d20 0x7fa9bc0c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:44.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.382+0000 7fa9c2e51640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa9bc0c2f50 0x7fa9bc1538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:44.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.382+0000 7fa9c2e51640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9bc0c3910 con 0x7fa9bc0a94a0 2024-04-25T00:16:44.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.382+0000 7fa9c2e51640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fa9bc154530 con 0x7fa9bc150d20 2024-04-25T00:16:44.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.382+0000 7fa9c2e51640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa9bc0c16e0 con 0x7fa9bc0c2f50 2024-04-25T00:16:44.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.382+0000 7fa9c13ee640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fa9bc0a94a0 0x7fa9bc153e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:44794/0 (socket says 172.21.15.116:44794) 2024-04-25T00:16:44.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.382+0000 7fa9c13ee640 1 -- 172.21.15.116:0/2946638470 learned_addr learned my addr 172.21.15.116:0/2946638470 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:16:44.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.382+0000 7fa9c0bed640 1 --2- 172.21.15.116:0/2946638470 >> v2:172.21.15.174:3300/0 conn(0x7fa9bc150d20 0x7fa9bc0c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:44.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.382+0000 7fa9bb7fe640 1 -- 172.21.15.116:0/2946638470 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 68307714 0 0) 0x7fa9bc0c3910 con 0x7fa9bc0a94a0 2024-04-25T00:16:44.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.382+0000 7fa9bb7fe640 1 -- 172.21.15.116:0/2946638470 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa9a4003530 con 0x7fa9bc0a94a0 2024-04-25T00:16:44.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.382+0000 7fa9bbfff640 1 --2- 172.21.15.116:0/2946638470 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa9bc0c2f50 0x7fa9bc1538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:44.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.382+0000 7fa9bb7fe640 1 -- 172.21.15.116:0/2946638470 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1259628349 0 0) 0x7fa9a4003530 con 0x7fa9bc0a94a0 2024-04-25T00:16:44.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.382+0000 7fa9bb7fe640 1 -- 172.21.15.116:0/2946638470 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa9bc0c2f50 msgr2=0x7fa9bc1538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:44.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.382+0000 7fa9bb7fe640 1 --2- 172.21.15.116:0/2946638470 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa9bc0c2f50 0x7fa9bc1538e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:44.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.382+0000 7fa9bb7fe640 1 -- 172.21.15.116:0/2946638470 >> v2:172.21.15.174:3300/0 conn(0x7fa9bc150d20 msgr2=0x7fa9bc0c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:44.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.382+0000 7fa9bb7fe640 1 --2- 172.21.15.116:0/2946638470 >> v2:172.21.15.174:3300/0 conn(0x7fa9bc150d20 0x7fa9bc0c2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:44.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.382+0000 7fa9bb7fe640 1 -- 172.21.15.116:0/2946638470 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9bc0c1400 con 0x7fa9bc0a94a0 2024-04-25T00:16:44.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.382+0000 7fa9bb7fe640 1 -- 172.21.15.116:0/2946638470 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7fa9b4001a00 con 0x7fa9bc0a94a0 2024-04-25T00:16:44.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.382+0000 7fa9bb7fe640 1 -- 172.21.15.116:0/2946638470 >> v1:172.21.15.116:6789/0 conn(0x7fa9bc0a94a0 legacy=0x7fa9bc153e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:44.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.382+0000 7fa9bb7fe640 1 --2- 172.21.15.116:0/2946638470 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9a40039c0 0x7fa9a4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:44.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.382+0000 7fa9bb7fe640 1 --2- 172.21.15.116:0/2946638470 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa9bc0c2f50 0x7fa9a4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:44.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.382+0000 7fa9bb7fe640 1 --2- 172.21.15.116:0/2946638470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9a4004d50 0x7fa9a40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:44.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.382+0000 7fa9bb7fe640 1 -- 172.21.15.116:0/2946638470 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa9bc154530 con 0x7fa9a4004d50 2024-04-25T00:16:44.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.382+0000 7fa9bb7fe640 1 -- 172.21.15.116:0/2946638470 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa9bc0c16e0 con 0x7fa9a40039c0 2024-04-25T00:16:44.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.382+0000 7fa9bb7fe640 1 -- 172.21.15.116:0/2946638470 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa9a400ba50 con 0x7fa9bc0c2f50 2024-04-25T00:16:44.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.382+0000 7fa9c13ee640 1 --2- 172.21.15.116:0/2946638470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9a4004d50 0x7fa9a40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:44.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.382+0000 7fa9bbfff640 1 --2- 172.21.15.116:0/2946638470 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa9bc0c2f50 0x7fa9a4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:44.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.382+0000 7fa9c13ee640 1 -- 172.21.15.116:0/2946638470 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa9bc0c2f50 msgr2=0x7fa9a4004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:44.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.382+0000 7fa9c13ee640 1 --2- 172.21.15.116:0/2946638470 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa9bc0c2f50 0x7fa9a4004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:44.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.382+0000 7fa9c0bed640 1 --2- 172.21.15.116:0/2946638470 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9a40039c0 0x7fa9a4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:44.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.382+0000 7fa9c13ee640 1 -- 172.21.15.116:0/2946638470 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9a40039c0 msgr2=0x7fa9a4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:44.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.382+0000 7fa9c13ee640 1 --2- 172.21.15.116:0/2946638470 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9a40039c0 0x7fa9a4003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:44.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.382+0000 7fa9c13ee640 1 -- 172.21.15.116:0/2946638470 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9a4003530 con 0x7fa9a4004d50 2024-04-25T00:16:44.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.382+0000 7fa9c13ee640 1 --2- 172.21.15.116:0/2946638470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9a4004d50 0x7fa9a40051d0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fa9b40013e0 tx=0x7fa9b401d2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1645d8c92e201f3f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:44.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.382+0000 7fa9bb7fe640 1 -- 172.21.15.116:0/2946638470 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b400a410 con 0x7fa9a4004d50 2024-04-25T00:16:44.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.382+0000 7fa9bb7fe640 1 -- 172.21.15.116:0/2946638470 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9b40ad630 con 0x7fa9a4004d50 2024-04-25T00:16:44.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.382+0000 7fa9bb7fe640 1 -- 172.21.15.116:0/2946638470 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b40ada20 con 0x7fa9a4004d50 2024-04-25T00:16:44.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.386+0000 7fa9c2e51640 1 -- 172.21.15.116:0/2946638470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9a4004d50 msgr2=0x7fa9a40051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:44.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.386+0000 7fa9c2e51640 1 --2- 172.21.15.116:0/2946638470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9a4004d50 0x7fa9a40051d0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fa9b40013e0 tx=0x7fa9b401d2e0 comp rx=0 tx=0).stop 2024-04-25T00:16:44.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.386+0000 7fa9c13ee640 1 -- 172.21.15.116:0/2946638470 reap_dead start 2024-04-25T00:16:44.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.386+0000 7fa9c2e51640 1 -- 172.21.15.116:0/2946638470 shutdown_connections 2024-04-25T00:16:44.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.386+0000 7fa9c2e51640 1 -- 172.21.15.116:0/2946638470 >> 172.21.15.116:0/2946638470 conn(0x7fa9bc0c3b00 msgr2=0x7fa9bc0c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:44.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.386+0000 7fa9c2e51640 1 -- 172.21.15.116:0/2946638470 shutdown_connections 2024-04-25T00:16:44.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.386+0000 7fa9c2e51640 1 -- 172.21.15.116:0/2946638470 wait complete. 2024-04-25T00:16:44.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.386+0000 7fa9c2e51640 1 Processor -- start 2024-04-25T00:16:44.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.386+0000 7fa9c2e51640 1 -- start start 2024-04-25T00:16:44.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.386+0000 7fa9c2e51640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9a4004d50 0x7fa9bc0c58e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:44.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.386+0000 7fa9c2e51640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9bc0c0690 0x7fa9bc13f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:44.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.386+0000 7fa9c2e51640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa9bc13ff30 0x7fa9bc1ee7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:44.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.386+0000 7fa9c2e51640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa9bc06ec20 con 0x7fa9bc0c0690 2024-04-25T00:16:44.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.386+0000 7fa9c2e51640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa9bc06e940 con 0x7fa9a4004d50 2024-04-25T00:16:44.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.386+0000 7fa9c2e51640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa9bc0a99f0 con 0x7fa9bc13ff30 2024-04-25T00:16:44.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.386+0000 7fa9bbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9bc0c0690 0x7fa9bc13f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:44.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.386+0000 7fa9bbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9bc0c0690 0x7fa9bc13f9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52604/0 (socket says 172.21.15.116:52604) 2024-04-25T00:16:44.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.386+0000 7fa9bbfff640 1 -- 172.21.15.116:0/11637450 learned_addr learned my addr 172.21.15.116:0/11637450 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:16:44.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.386+0000 7fa9c0bed640 1 --2- 172.21.15.116:0/11637450 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9a4004d50 0x7fa9bc0c58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:44.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.386+0000 7fa9c13ee640 1 --2- 172.21.15.116:0/11637450 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa9bc13ff30 0x7fa9bc1ee7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:44.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.386+0000 7fa9bbfff640 1 -- 172.21.15.116:0/11637450 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa9bc13ff30 msgr2=0x7fa9bc1ee7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:44.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.386+0000 7fa9bbfff640 1 --2- 172.21.15.116:0/11637450 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa9bc13ff30 0x7fa9bc1ee7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:44.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.386+0000 7fa9bbfff640 1 -- 172.21.15.116:0/11637450 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9a4004d50 msgr2=0x7fa9bc0c58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:44.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.386+0000 7fa9bbfff640 1 --2- 172.21.15.116:0/11637450 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9a4004d50 0x7fa9bc0c58e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:44.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.386+0000 7fa9bbfff640 1 -- 172.21.15.116:0/11637450 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9bc070e40 con 0x7fa9bc0c0690 2024-04-25T00:16:44.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.386+0000 7fa9bbfff640 1 --2- 172.21.15.116:0/11637450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9bc0c0690 0x7fa9bc13f9f0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fa9b000f5b0 tx=0x7fa9b000fa70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:44.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.386+0000 7fa9b97fa640 1 -- 172.21.15.116:0/11637450 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b0019020 con 0x7fa9bc0c0690 2024-04-25T00:16:44.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.386+0000 7fa9b97fa640 1 -- 172.21.15.116:0/11637450 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9b0014870 con 0x7fa9bc0c0690 2024-04-25T00:16:44.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.386+0000 7fa9b97fa640 1 -- 172.21.15.116:0/11637450 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9b0014c40 con 0x7fa9bc0c0690 2024-04-25T00:16:44.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.386+0000 7fa9c2e51640 1 -- 172.21.15.116:0/11637450 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9bc1eb2f0 con 0x7fa9bc0c0690 2024-04-25T00:16:44.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.386+0000 7fa9c2e51640 1 -- 172.21.15.116:0/11637450 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9bc1eb8b0 con 0x7fa9bc0c0690 2024-04-25T00:16:44.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.386+0000 7fa9c2e51640 1 -- 172.21.15.116:0/11637450 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9bc068350 con 0x7fa9bc0c0690 2024-04-25T00:16:44.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.390+0000 7fa9b97fa640 1 -- 172.21.15.116:0/11637450 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa9b001c020 con 0x7fa9bc0c0690 2024-04-25T00:16:44.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.390+0000 7fa9b97fa640 1 --2- 172.21.15.116:0/11637450 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa9a80678c0 0x7fa9a8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:44.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.390+0000 7fa9b97fa640 1 -- 172.21.15.116:0/11637450 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 8558+0+0 (secure 0 0 0) 0x7fa9b0086f60 con 0x7fa9bc0c0690 2024-04-25T00:16:44.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.390+0000 7fa9c0bed640 1 --2- 172.21.15.116:0/11637450 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa9a80678c0 0x7fa9a8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:44.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.390+0000 7fa9c0bed640 1 --2- 172.21.15.116:0/11637450 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa9a80678c0 0x7fa9a8069d80 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc146e50 tx=0x7fa9ac008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:44.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.394+0000 7fa9b97fa640 1 -- 172.21.15.116:0/11637450 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fa9b0023020 con 0x7fa9bc0c0690 2024-04-25T00:16:44.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.574+0000 7fa9c2e51640 1 -- 172.21.15.116:0/11637450 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa9bc0c34f0 con 0x7fa9bc0c0690 2024-04-25T00:16:44.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.574+0000 7fa9b97fa640 1 -- 172.21.15.116:0/11637450 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) ==== 95+0+6510 (secure 0 0 0) 0x7fa9b0056ed0 con 0x7fa9bc0c0690 2024-04-25T00:16:44.575 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:16:44.575 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":122,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7119,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/3645553116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3645553116},{"type":"v1","addr":"172.21.15.116:6833","nonce":3645553116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":122}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:15:43.819845+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":49,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6810},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6810":{"gid":6810,"name":"c","rank":0,"incarnation":113,"state":"up:active","state_seq":13,"addr":"172.21.15.116:6833/2340581953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2340581953},{"type":"v1","addr":"172.21.15.116:6833","nonce":2340581953}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6810,"qdb_cluster":[6810]},"id":1},{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:16:38.790885+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5621},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5621":{"gid":5621,"name":"d","rank":0,"incarnation":118,"state":"up:active","state_seq":15,"addr":"172.21.15.174:6837/3081407040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3081407040},{"type":"v1","addr":"172.21.15.174:6837","nonce":3081407040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5621,"qdb_cluster":[5621]},"id":2}]} 2024-04-25T00:16:44.576 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 122 2024-04-25T00:16:44.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.578+0000 7fa9c2e51640 1 -- 172.21.15.116:0/11637450 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa9a80678c0 msgr2=0x7fa9a8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:44.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.578+0000 7fa9c2e51640 1 --2- 172.21.15.116:0/11637450 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa9a80678c0 0x7fa9a8069d80 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc146e50 tx=0x7fa9ac008040 comp rx=0 tx=0).stop 2024-04-25T00:16:44.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.578+0000 7fa9c2e51640 1 -- 172.21.15.116:0/11637450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9bc0c0690 msgr2=0x7fa9bc13f9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:44.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.578+0000 7fa9c2e51640 1 --2- 172.21.15.116:0/11637450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9bc0c0690 0x7fa9bc13f9f0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fa9b000f5b0 tx=0x7fa9b000fa70 comp rx=0 tx=0).stop 2024-04-25T00:16:44.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.578+0000 7fa9c2e51640 1 -- 172.21.15.116:0/11637450 shutdown_connections 2024-04-25T00:16:44.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.578+0000 7fa9c2e51640 1 --2- 172.21.15.116:0/11637450 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa9a80678c0 0x7fa9a8069d80 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:44.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.578+0000 7fa9c2e51640 1 --2- 172.21.15.116:0/11637450 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa9bc13ff30 0x7fa9bc1ee7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:44.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.578+0000 7fa9c2e51640 1 --2- 172.21.15.116:0/11637450 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa9bc0c0690 0x7fa9bc13f9f0 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:44.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.578+0000 7fa9c2e51640 1 --2- 172.21.15.116:0/11637450 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9a4004d50 0x7fa9bc0c58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:44.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.578+0000 7fa9c2e51640 1 -- 172.21.15.116:0/11637450 >> 172.21.15.116:0/11637450 conn(0x7fa9bc0c3b00 msgr2=0x7fa9bc153730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:44.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.578+0000 7fa9c2e51640 1 -- 172.21.15.116:0/11637450 shutdown_connections 2024-04-25T00:16:44.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:44.578+0000 7fa9c2e51640 1 -- 172.21.15.116:0/11637450 wait complete. 2024-04-25T00:16:44.599 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:16:46.600 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:16:46.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.730+0000 7fde19956640 1 Processor -- start 2024-04-25T00:16:46.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.730+0000 7fde19956640 1 -- start start 2024-04-25T00:16:46.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.730+0000 7fde19956640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fde14150d20 0x7fde140c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:46.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.730+0000 7fde19956640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fde140c2f50 0x7fde141538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:46.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.730+0000 7fde19956640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde140c3910 con 0x7fde140a94a0 2024-04-25T00:16:46.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.730+0000 7fde19956640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fde14154530 con 0x7fde14150d20 2024-04-25T00:16:46.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.730+0000 7fde19956640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fde140c16e0 con 0x7fde140c2f50 2024-04-25T00:16:46.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.730+0000 7fde12ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fde14150d20 0x7fde140c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:46.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.730+0000 7fde12ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fde14150d20 0x7fde140c2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59122/0 (socket says 172.21.15.116:59122) 2024-04-25T00:16:46.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.730+0000 7fde12ffd640 1 -- 172.21.15.116:0/4013925454 learned_addr learned my addr 172.21.15.116:0/4013925454 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:16:46.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.730+0000 7fde0bfff640 1 --2- 172.21.15.116:0/4013925454 >> v2:172.21.15.174:3300/0 conn(0x7fde140c2f50 0x7fde141538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:46.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.730+0000 7fde12ffd640 1 -- 172.21.15.116:0/4013925454 >> v2:172.21.15.174:3300/0 conn(0x7fde140c2f50 msgr2=0x7fde141538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:46.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.730+0000 7fde12ffd640 1 --2- 172.21.15.116:0/4013925454 >> v2:172.21.15.174:3300/0 conn(0x7fde140c2f50 0x7fde141538e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:46.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.730+0000 7fde12ffd640 1 -- 172.21.15.116:0/4013925454 >> v1:172.21.15.174:6789/0 conn(0x7fde140a94a0 legacy=0x7fde14153e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:46.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.730+0000 7fde12ffd640 1 -- 172.21.15.116:0/4013925454 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde140c1400 con 0x7fde14150d20 2024-04-25T00:16:46.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.734+0000 7fde12ffd640 1 --2- 172.21.15.116:0/4013925454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde14150d20 0x7fde140c2a10 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fde14070710 tx=0x7fde0401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bee2c703b9676426 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:46.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.734+0000 7fde127fc640 1 -- 172.21.15.116:0/4013925454 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde040ab430 con 0x7fde14150d20 2024-04-25T00:16:46.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.734+0000 7fde127fc640 1 -- 172.21.15.116:0/4013925454 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde040b0070 con 0x7fde14150d20 2024-04-25T00:16:46.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.734+0000 7fde127fc640 1 -- 172.21.15.116:0/4013925454 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde040ab7f0 con 0x7fde14150d20 2024-04-25T00:16:46.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.734+0000 7fde19956640 1 -- 172.21.15.116:0/4013925454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde14150d20 msgr2=0x7fde140c2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:46.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.734+0000 7fde19956640 1 --2- 172.21.15.116:0/4013925454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde14150d20 0x7fde140c2a10 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fde14070710 tx=0x7fde0401c520 comp rx=0 tx=0).stop 2024-04-25T00:16:46.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.734+0000 7fde19956640 1 -- 172.21.15.116:0/4013925454 shutdown_connections 2024-04-25T00:16:46.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.734+0000 7fde19956640 1 --2- 172.21.15.116:0/4013925454 >> v2:172.21.15.174:3300/0 conn(0x7fde140c2f50 0x7fde141538e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:46.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.734+0000 7fde19956640 1 --2- 172.21.15.116:0/4013925454 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde14150d20 0x7fde140c2a10 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:46.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.734+0000 7fde19956640 1 -- 172.21.15.116:0/4013925454 >> 172.21.15.116:0/4013925454 conn(0x7fde140c3b00 msgr2=0x7fde140c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:46.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.734+0000 7fde19956640 1 -- 172.21.15.116:0/4013925454 shutdown_connections 2024-04-25T00:16:46.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.734+0000 7fde19956640 1 -- 172.21.15.116:0/4013925454 wait complete. 2024-04-25T00:16:46.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.734+0000 7fde19956640 1 Processor -- start 2024-04-25T00:16:46.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.734+0000 7fde19956640 1 -- start start 2024-04-25T00:16:46.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.734+0000 7fde19956640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde140a94a0 0x7fde140c0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:46.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.734+0000 7fde19956640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fde140c2f50 0x7fde141f2480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:46.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.734+0000 7fde19956640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde14150d20 0x7fde140c5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:46.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.734+0000 7fde19956640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fde141608b0 con 0x7fde140a94a0 2024-04-25T00:16:46.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.734+0000 7fde19956640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fde14160730 con 0x7fde14150d20 2024-04-25T00:16:46.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.734+0000 7fde19956640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fde14160a30 con 0x7fde140c2f50 2024-04-25T00:16:46.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.734+0000 7fde12ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde140a94a0 0x7fde140c0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:46.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.734+0000 7fde12ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde140a94a0 0x7fde140c0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59138/0 (socket says 172.21.15.116:59138) 2024-04-25T00:16:46.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.734+0000 7fde12ffd640 1 -- 172.21.15.116:0/956763053 learned_addr learned my addr 172.21.15.116:0/956763053 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:16:46.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.734+0000 7fde137fe640 1 --2- 172.21.15.116:0/956763053 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde14150d20 0x7fde140c5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:46.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.734+0000 7fde0bfff640 1 --2- 172.21.15.116:0/956763053 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fde140c2f50 0x7fde141f2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:46.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.738+0000 7fde12ffd640 1 -- 172.21.15.116:0/956763053 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fde140c2f50 msgr2=0x7fde141f2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:46.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.738+0000 7fde12ffd640 1 --2- 172.21.15.116:0/956763053 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fde140c2f50 0x7fde141f2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:46.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.738+0000 7fde12ffd640 1 -- 172.21.15.116:0/956763053 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde14150d20 msgr2=0x7fde140c5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:46.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.738+0000 7fde12ffd640 1 --2- 172.21.15.116:0/956763053 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde14150d20 0x7fde140c5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:46.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.738+0000 7fde12ffd640 1 -- 172.21.15.116:0/956763053 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde14143480 con 0x7fde140a94a0 2024-04-25T00:16:46.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.738+0000 7fde137fe640 1 --2- 172.21.15.116:0/956763053 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde14150d20 0x7fde140c5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:16:46.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.738+0000 7fde0bfff640 1 --2- 172.21.15.116:0/956763053 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fde140c2f50 0x7fde141f2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:16:46.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.738+0000 7fde12ffd640 1 --2- 172.21.15.116:0/956763053 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde140a94a0 0x7fde140c0690 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fde04019130 tx=0x7fde0401ad60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:46.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.738+0000 7fde10ff9640 1 -- 172.21.15.116:0/956763053 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde040be5f0 con 0x7fde140a94a0 2024-04-25T00:16:46.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.738+0000 7fde10ff9640 1 -- 172.21.15.116:0/956763053 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde040be790 con 0x7fde140a94a0 2024-04-25T00:16:46.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.738+0000 7fde10ff9640 1 -- 172.21.15.116:0/956763053 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde040beb80 con 0x7fde140a94a0 2024-04-25T00:16:46.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.738+0000 7fde19956640 1 -- 172.21.15.116:0/956763053 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde14143d20 con 0x7fde140a94a0 2024-04-25T00:16:46.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.738+0000 7fde19956640 1 -- 172.21.15.116:0/956763053 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde141442e0 con 0x7fde140a94a0 2024-04-25T00:16:46.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.738+0000 7fde19956640 1 -- 172.21.15.116:0/956763053 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde14068350 con 0x7fde140a94a0 2024-04-25T00:16:46.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.738+0000 7fde10ff9640 1 -- 172.21.15.116:0/956763053 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fde040b43f0 con 0x7fde140a94a0 2024-04-25T00:16:46.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.738+0000 7fde10ff9640 1 --2- 172.21.15.116:0/956763053 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fddf40678c0 0x7fddf4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:46.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.738+0000 7fde10ff9640 1 -- 172.21.15.116:0/956763053 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 8644+0+0 (secure 0 0 0) 0x7fde040a8030 con 0x7fde140a94a0 2024-04-25T00:16:46.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.738+0000 7fde0bfff640 1 --2- 172.21.15.116:0/956763053 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fddf40678c0 0x7fddf4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:46.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.738+0000 7fde0bfff640 1 --2- 172.21.15.116:0/956763053 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fddf40678c0 0x7fddf4069d80 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fde1406d9c0 tx=0x7fddfc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:46.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.742+0000 7fde10ff9640 1 -- 172.21.15.116:0/956763053 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fde04127080 con 0x7fde140a94a0 2024-04-25T00:16:46.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.938+0000 7fde19956640 1 -- 172.21.15.116:0/956763053 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fde140aa8a0 con 0x7fde140a94a0 2024-04-25T00:16:46.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.938+0000 7fde10ff9640 1 -- 172.21.15.116:0/956763053 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 124 v124) ==== 95+0+5676 (secure 0 0 0) 0x7fde04016110 con 0x7fde140a94a0 2024-04-25T00:16:46.939 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:16:46.940 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":124,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:16:45.946028+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7119},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7119":{"gid":7119,"name":"a","rank":0,"incarnation":123,"state":"up:reconnect","state_seq":3,"addr":"172.21.15.116:6833/3645553116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3645553116},{"type":"v1","addr":"172.21.15.116:6833","nonce":3645553116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:16:38.790885+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5621},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5621":{"gid":5621,"name":"d","rank":0,"incarnation":118,"state":"up:active","state_seq":15,"addr":"172.21.15.174:6837/3081407040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3081407040},{"type":"v1","addr":"172.21.15.174:6837","nonce":3081407040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5621,"qdb_cluster":[5621]},"id":2}]} 2024-04-25T00:16:46.942 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 124 2024-04-25T00:16:46.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.942+0000 7fde19956640 1 -- 172.21.15.116:0/956763053 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fddf40678c0 msgr2=0x7fddf4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:46.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.942+0000 7fde19956640 1 --2- 172.21.15.116:0/956763053 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fddf40678c0 0x7fddf4069d80 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fde1406d9c0 tx=0x7fddfc008040 comp rx=0 tx=0).stop 2024-04-25T00:16:46.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.942+0000 7fde19956640 1 -- 172.21.15.116:0/956763053 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde140a94a0 msgr2=0x7fde140c0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:46.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.942+0000 7fde19956640 1 --2- 172.21.15.116:0/956763053 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde140a94a0 0x7fde140c0690 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fde04019130 tx=0x7fde0401ad60 comp rx=0 tx=0).stop 2024-04-25T00:16:46.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.942+0000 7fde19956640 1 -- 172.21.15.116:0/956763053 shutdown_connections 2024-04-25T00:16:46.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.942+0000 7fde19956640 1 --2- 172.21.15.116:0/956763053 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fddf40678c0 0x7fddf4069d80 secure :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fde1406d9c0 tx=0x7fddfc008040 comp rx=0 tx=0).stop 2024-04-25T00:16:46.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.942+0000 7fde19956640 1 --2- 172.21.15.116:0/956763053 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde14150d20 0x7fde140c5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:46.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.942+0000 7fde19956640 1 --2- 172.21.15.116:0/956763053 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fde140c2f50 0x7fde141f2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:46.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.942+0000 7fde19956640 1 --2- 172.21.15.116:0/956763053 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde140a94a0 0x7fde140c0690 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:46.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.942+0000 7fde19956640 1 -- 172.21.15.116:0/956763053 >> 172.21.15.116:0/956763053 conn(0x7fde140c3b00 msgr2=0x7fde140ac9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:46.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.942+0000 7fde19956640 1 -- 172.21.15.116:0/956763053 shutdown_connections 2024-04-25T00:16:46.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:46.942+0000 7fde19956640 1 -- 172.21.15.116:0/956763053 wait complete. 2024-04-25T00:16:46.963 INFO:tasks.mds_thrash.fs.[a]:mds.c down, removed from mdsmap 2024-04-25T00:16:46.963 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-04-25T00:16:46.963 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:16:47.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.074+0000 7fc0e498b640 1 Processor -- start 2024-04-25T00:16:47.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.074+0000 7fc0e498b640 1 -- start start 2024-04-25T00:16:47.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.078+0000 7fc0e498b640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fc0e0070dd0 0x7fc0e0071270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:47.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.078+0000 7fc0e498b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc0e006e890 0x7fc0e00717b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:47.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.078+0000 7fc0e498b640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0e0152c10 con 0x7fc0e0148890 2024-04-25T00:16:47.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.078+0000 7fc0e498b640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fc0e0071cf0 con 0x7fc0e0070dd0 2024-04-25T00:16:47.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.078+0000 7fc0e498b640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fc0e0151e50 con 0x7fc0e006e890 2024-04-25T00:16:47.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.078+0000 7fc0ded9d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc0e006e890 0x7fc0e00717b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:47.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.078+0000 7fc0ddd9b640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fc0e0070dd0 0x7fc0e0071270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:47.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.078+0000 7fc0de59c640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fc0e0148890 0x7fc0e0072590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:39752/0 (socket says 172.21.15.116:39752) 2024-04-25T00:16:47.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.078+0000 7fc0de59c640 1 -- 172.21.15.116:0/1630114799 learned_addr learned my addr 172.21.15.116:0/1630114799 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:16:47.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.078+0000 7fc0ddd9b640 1 -- 172.21.15.116:0/1630114799 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc0e006e890 msgr2=0x7fc0e00717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:47.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.078+0000 7fc0ddd9b640 1 --2- 172.21.15.116:0/1630114799 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc0e006e890 0x7fc0e00717b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:47.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.078+0000 7fc0ddd9b640 1 -- 172.21.15.116:0/1630114799 >> v1:172.21.15.174:6789/0 conn(0x7fc0e0148890 legacy=0x7fc0e0072590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:47.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.078+0000 7fc0ddd9b640 1 -- 172.21.15.116:0/1630114799 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0e0152130 con 0x7fc0e0070dd0 2024-04-25T00:16:47.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.078+0000 7fc0ded9d640 1 --2- 172.21.15.116:0/1630114799 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc0e006e890 0x7fc0e00717b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:16:47.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.078+0000 7fc0ddd9b640 1 --2- 172.21.15.116:0/1630114799 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc0e0070dd0 0x7fc0e0071270 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fc0d40097c0 tx=0x7fc0d401c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bfb9d93c05e592d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:47.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.078+0000 7fc0dd59a640 1 -- 172.21.15.116:0/1630114799 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d40b0070 con 0x7fc0e0070dd0 2024-04-25T00:16:47.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.078+0000 7fc0dd59a640 1 -- 172.21.15.116:0/1630114799 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0d40ab600 con 0x7fc0e0070dd0 2024-04-25T00:16:47.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.078+0000 7fc0dd59a640 1 -- 172.21.15.116:0/1630114799 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d40ab9d0 con 0x7fc0e0070dd0 2024-04-25T00:16:47.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.082+0000 7fc0e498b640 1 -- 172.21.15.116:0/1630114799 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc0e0070dd0 msgr2=0x7fc0e0071270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:47.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.082+0000 7fc0e498b640 1 --2- 172.21.15.116:0/1630114799 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc0e0070dd0 0x7fc0e0071270 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fc0d40097c0 tx=0x7fc0d401c4a0 comp rx=0 tx=0).stop 2024-04-25T00:16:47.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.082+0000 7fc0e498b640 1 -- 172.21.15.116:0/1630114799 shutdown_connections 2024-04-25T00:16:47.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.082+0000 7fc0e498b640 1 --2- 172.21.15.116:0/1630114799 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc0e006e890 0x7fc0e00717b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:47.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.082+0000 7fc0e498b640 1 --2- 172.21.15.116:0/1630114799 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc0e0070dd0 0x7fc0e0071270 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:47.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.082+0000 7fc0e498b640 1 -- 172.21.15.116:0/1630114799 >> 172.21.15.116:0/1630114799 conn(0x7fc0e00c3b00 msgr2=0x7fc0e00c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:47.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.082+0000 7fc0e498b640 1 -- 172.21.15.116:0/1630114799 shutdown_connections 2024-04-25T00:16:47.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.082+0000 7fc0e498b640 1 -- 172.21.15.116:0/1630114799 wait complete. 2024-04-25T00:16:47.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.082+0000 7fc0e498b640 1 Processor -- start 2024-04-25T00:16:47.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.082+0000 7fc0e498b640 1 -- start start 2024-04-25T00:16:47.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.082+0000 7fc0e498b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0e006e890 0x7fc0e01eeb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:47.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.082+0000 7fc0e498b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc0e0070dd0 0x7fc0e0146640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:47.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.082+0000 7fc0e498b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc0e0148890 0x7fc0e01f1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:47.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.082+0000 7fc0e498b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e0072e20 con 0x7fc0e006e890 2024-04-25T00:16:47.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.082+0000 7fc0e498b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e0072ca0 con 0x7fc0e0148890 2024-04-25T00:16:47.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.082+0000 7fc0e498b640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fc0e00abda0 con 0x7fc0e0070dd0 2024-04-25T00:16:47.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.082+0000 7fc0de59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0e006e890 0x7fc0e01eeb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:47.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.082+0000 7fc0de59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0e006e890 0x7fc0e01eeb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59146/0 (socket says 172.21.15.116:59146) 2024-04-25T00:16:47.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.082+0000 7fc0de59c640 1 -- 172.21.15.116:0/2866487638 learned_addr learned my addr 172.21.15.116:0/2866487638 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:16:47.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.082+0000 7fc0ded9d640 1 --2- 172.21.15.116:0/2866487638 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc0e0148890 0x7fc0e01f1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:47.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.082+0000 7fc0ddd9b640 1 --2- 172.21.15.116:0/2866487638 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc0e0070dd0 0x7fc0e0146640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:47.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.082+0000 7fc0de59c640 1 -- 172.21.15.116:0/2866487638 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc0e0070dd0 msgr2=0x7fc0e0146640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:47.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.082+0000 7fc0de59c640 1 --2- 172.21.15.116:0/2866487638 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc0e0070dd0 0x7fc0e0146640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:47.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.082+0000 7fc0de59c640 1 -- 172.21.15.116:0/2866487638 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc0e0148890 msgr2=0x7fc0e01f1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:47.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.082+0000 7fc0de59c640 1 --2- 172.21.15.116:0/2866487638 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc0e0148890 0x7fc0e01f1bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:47.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.082+0000 7fc0de59c640 1 -- 172.21.15.116:0/2866487638 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0e00c6c30 con 0x7fc0e006e890 2024-04-25T00:16:47.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.082+0000 7fc0de59c640 1 --2- 172.21.15.116:0/2866487638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0e006e890 0x7fc0e01eeb20 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fc0d000c370 tx=0x7fc0d000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:47.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.082+0000 7fc0ceffd640 1 -- 172.21.15.116:0/2866487638 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d0017070 con 0x7fc0e006e890 2024-04-25T00:16:47.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.082+0000 7fc0ceffd640 1 -- 172.21.15.116:0/2866487638 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0d0005cf0 con 0x7fc0e006e890 2024-04-25T00:16:47.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.082+0000 7fc0ceffd640 1 -- 172.21.15.116:0/2866487638 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d00072f0 con 0x7fc0e006e890 2024-04-25T00:16:47.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.082+0000 7fc0e498b640 1 -- 172.21.15.116:0/2866487638 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0e00c6390 con 0x7fc0e006e890 2024-04-25T00:16:47.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.086+0000 7fc0e498b640 1 -- 172.21.15.116:0/2866487638 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0e01f16d0 con 0x7fc0e006e890 2024-04-25T00:16:47.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.086+0000 7fc0e498b640 1 -- 172.21.15.116:0/2866487638 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0e0068350 con 0x7fc0e006e890 2024-04-25T00:16:47.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.086+0000 7fc0ceffd640 1 -- 172.21.15.116:0/2866487638 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc0d001a070 con 0x7fc0e006e890 2024-04-25T00:16:47.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.086+0000 7fc0ceffd640 1 --2- 172.21.15.116:0/2866487638 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc0c00678c0 0x7fc0c0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:47.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.086+0000 7fc0ceffd640 1 -- 172.21.15.116:0/2866487638 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 8644+0+0 (secure 0 0 0) 0x7fc0d0085ba0 con 0x7fc0e006e890 2024-04-25T00:16:47.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.086+0000 7fc0ddd9b640 1 --2- 172.21.15.116:0/2866487638 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc0c00678c0 0x7fc0c0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:47.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.086+0000 7fc0ddd9b640 1 --2- 172.21.15.116:0/2866487638 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc0c00678c0 0x7fc0c0069d80 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fc0d40ad000 tx=0x7fc0d4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:47.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.090+0000 7fc0ceffd640 1 -- 172.21.15.116:0/2866487638 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fc0d0055ba0 con 0x7fc0e006e890 2024-04-25T00:16:47.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.274+0000 7fc0e498b640 1 -- 172.21.15.116:0/2866487638 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc0e006e670 con 0x7fc0e006e890 2024-04-25T00:16:47.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.274+0000 7fc0ceffd640 1 -- 172.21.15.116:0/2866487638 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 124 v124) ==== 95+0+5676 (secure 0 0 0) 0x7fc0d00554a0 con 0x7fc0e006e890 2024-04-25T00:16:47.273 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:16:47.274 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":124,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:16:45.946028+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7119},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7119":{"gid":7119,"name":"a","rank":0,"incarnation":123,"state":"up:reconnect","state_seq":3,"addr":"172.21.15.116:6833/3645553116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3645553116},{"type":"v1","addr":"172.21.15.116:6833","nonce":3645553116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:16:38.790885+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5621},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5621":{"gid":5621,"name":"d","rank":0,"incarnation":118,"state":"up:active","state_seq":15,"addr":"172.21.15.174:6837/3081407040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3081407040},{"type":"v1","addr":"172.21.15.174:6837","nonce":3081407040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5621,"qdb_cluster":[5621]},"id":2}]} 2024-04-25T00:16:47.274 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 124 2024-04-25T00:16:47.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.274+0000 7fc0e498b640 1 -- 172.21.15.116:0/2866487638 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc0c00678c0 msgr2=0x7fc0c0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:47.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.274+0000 7fc0e498b640 1 --2- 172.21.15.116:0/2866487638 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc0c00678c0 0x7fc0c0069d80 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fc0d40ad000 tx=0x7fc0d4008040 comp rx=0 tx=0).stop 2024-04-25T00:16:47.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.274+0000 7fc0e498b640 1 -- 172.21.15.116:0/2866487638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0e006e890 msgr2=0x7fc0e01eeb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:47.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.274+0000 7fc0e498b640 1 --2- 172.21.15.116:0/2866487638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0e006e890 0x7fc0e01eeb20 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fc0d000c370 tx=0x7fc0d000c830 comp rx=0 tx=0).stop 2024-04-25T00:16:47.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.274+0000 7fc0e498b640 1 -- 172.21.15.116:0/2866487638 shutdown_connections 2024-04-25T00:16:47.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.274+0000 7fc0e498b640 1 --2- 172.21.15.116:0/2866487638 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc0c00678c0 0x7fc0c0069d80 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:47.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.274+0000 7fc0e498b640 1 --2- 172.21.15.116:0/2866487638 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc0e0148890 0x7fc0e01f1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:47.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.274+0000 7fc0e498b640 1 --2- 172.21.15.116:0/2866487638 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc0e0070dd0 0x7fc0e0146640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:47.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.274+0000 7fc0e498b640 1 --2- 172.21.15.116:0/2866487638 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc0e006e890 0x7fc0e01eeb20 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:47.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.274+0000 7fc0e498b640 1 -- 172.21.15.116:0/2866487638 >> 172.21.15.116:0/2866487638 conn(0x7fc0e00c3b00 msgr2=0x7fc0e0072590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:47.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.274+0000 7fc0e498b640 1 -- 172.21.15.116:0/2866487638 shutdown_connections 2024-04-25T00:16:47.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:47.274+0000 7fc0e498b640 1 -- 172.21.15.116:0/2866487638 wait complete. 2024-04-25T00:16:47.292 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 124, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:04.678737+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 124, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_7119": { "addr": "172.21.15.116:6833/3645553116", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6832", "nonce": 3645553116, "type": "v2" }, { "addr": "172.21.15.116:6833", "nonce": 3645553116, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7119, "incarnation": 123, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:reconnect", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 51, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T00:16:45.946028+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7119 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:18.825887+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 121, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5621": { "addr": "172.21.15.174:6837/3081407040", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6836", "nonce": 3081407040, "type": "v2" }, { "addr": "172.21.15.174:6837", "nonce": 3081407040, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5621, "incarnation": 118, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 15 } }, "last_failure": 0, "last_failure_osd_epoch": 50, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-04-25T00:16:38.790885+0000", "qdb_cluster": [ 5621 ], "qdb_leader": 5621, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5621 } } } ], "standbys": [ { "addr": "172.21.15.174:6835/4112145782", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6834", "nonce": 4112145782, "type": "v2" }, { "addr": "172.21.15.174:6835", "nonce": 4112145782, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 112, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5582, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-25T00:16:49.292 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:16:49.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.426+0000 7fd642d19640 1 Processor -- start 2024-04-25T00:16:49.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.426+0000 7fd642d19640 1 -- start start 2024-04-25T00:16:49.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.430+0000 7fd642d19640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd63c152e60 0x7fd63c1599a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:49.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.430+0000 7fd642d19640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd63c073200 con 0x7fd63c1562d0 2024-04-25T00:16:49.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.430+0000 7fd642d19640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd63c158ec0 con 0x7fd63c0c5010 2024-04-25T00:16:49.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.430+0000 7fd642d19640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fd63c153240 con 0x7fd63c152e60 2024-04-25T00:16:49.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.430+0000 7fd63bfff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fd63c1562d0 0x7fd63c0c36e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:53708/0 (socket says 172.21.15.116:53708) 2024-04-25T00:16:49.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.430+0000 7fd63bfff640 1 -- 172.21.15.116:0/967623429 learned_addr learned my addr 172.21.15.116:0/967623429 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:16:49.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.430+0000 7fd640ab5640 1 --2- 172.21.15.116:0/967623429 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd63c152e60 0x7fd63c1599a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:49.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.430+0000 7fd63b7fe640 1 -- 172.21.15.116:0/967623429 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 270780436 0 0) 0x7fd63c073200 con 0x7fd63c1562d0 2024-04-25T00:16:49.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.430+0000 7fd63b7fe640 1 -- 172.21.15.116:0/967623429 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd62c003530 con 0x7fd63c1562d0 2024-04-25T00:16:49.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.430+0000 7fd63b7fe640 1 -- 172.21.15.116:0/967623429 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 266733171 0 0) 0x7fd62c003530 con 0x7fd63c1562d0 2024-04-25T00:16:49.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.430+0000 7fd63b7fe640 1 -- 172.21.15.116:0/967623429 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd63c152e60 msgr2=0x7fd63c1599a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:49.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.430+0000 7fd63b7fe640 1 --2- 172.21.15.116:0/967623429 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd63c152e60 0x7fd63c1599a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:49.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.430+0000 7fd63b7fe640 1 -- 172.21.15.116:0/967623429 >> v1:172.21.15.174:6789/0 conn(0x7fd63c0c5010 legacy=0x7fd63c160120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:49.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.430+0000 7fd63b7fe640 1 -- 172.21.15.116:0/967623429 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd63c1591a0 con 0x7fd63c1562d0 2024-04-25T00:16:49.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.430+0000 7fd63b7fe640 1 -- 172.21.15.116:0/967623429 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7fd628001680 con 0x7fd63c1562d0 2024-04-25T00:16:49.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.430+0000 7fd63b7fe640 1 -- 172.21.15.116:0/967623429 >> v1:172.21.15.116:6789/0 conn(0x7fd63c1562d0 legacy=0x7fd63c0c36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:49.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.430+0000 7fd63b7fe640 1 --2- 172.21.15.116:0/967623429 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd63c152e60 0x7fd62c0039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:49.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.430+0000 7fd63b7fe640 1 --2- 172.21.15.116:0/967623429 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd62c004020 0x7fd62c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:49.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.430+0000 7fd63b7fe640 1 --2- 172.21.15.116:0/967623429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd62c004d50 0x7fd62c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:49.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.430+0000 7fd63b7fe640 1 -- 172.21.15.116:0/967623429 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd63c153240 con 0x7fd62c004d50 2024-04-25T00:16:49.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.430+0000 7fd63b7fe640 1 -- 172.21.15.116:0/967623429 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd62c00bc10 con 0x7fd62c004020 2024-04-25T00:16:49.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.430+0000 7fd63b7fe640 1 -- 172.21.15.116:0/967623429 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fd62c00be10 con 0x7fd63c152e60 2024-04-25T00:16:49.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.430+0000 7fd6412b6640 1 --2- 172.21.15.116:0/967623429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd62c004d50 0x7fd62c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:49.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.430+0000 7fd640ab5640 1 --2- 172.21.15.116:0/967623429 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd63c152e60 0x7fd62c0039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:49.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.430+0000 7fd63bfff640 1 --2- 172.21.15.116:0/967623429 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd62c004020 0x7fd62c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:49.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.430+0000 7fd6412b6640 1 -- 172.21.15.116:0/967623429 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd63c152e60 msgr2=0x7fd62c0039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:49.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.430+0000 7fd6412b6640 1 --2- 172.21.15.116:0/967623429 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd63c152e60 0x7fd62c0039f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:49.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.430+0000 7fd6412b6640 1 -- 172.21.15.116:0/967623429 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd62c004020 msgr2=0x7fd62c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:49.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.430+0000 7fd6412b6640 1 --2- 172.21.15.116:0/967623429 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd62c004020 0x7fd62c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:49.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.430+0000 7fd6412b6640 1 -- 172.21.15.116:0/967623429 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd62c003530 con 0x7fd62c004d50 2024-04-25T00:16:49.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.434+0000 7fd6412b6640 1 --2- 172.21.15.116:0/967623429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd62c004d50 0x7fd62c0051d0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fd634001e70 tx=0x7fd63401c780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=437c6db12e1a93fc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:49.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.434+0000 7fd63b7fe640 1 -- 172.21.15.116:0/967623429 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd634001a90 con 0x7fd62c004d50 2024-04-25T00:16:49.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.434+0000 7fd63b7fe640 1 -- 172.21.15.116:0/967623429 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd634003b80 con 0x7fd62c004d50 2024-04-25T00:16:49.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.434+0000 7fd63b7fe640 1 -- 172.21.15.116:0/967623429 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd634005080 con 0x7fd62c004d50 2024-04-25T00:16:49.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.434+0000 7fd642d19640 1 -- 172.21.15.116:0/967623429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd62c004d50 msgr2=0x7fd62c0051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:49.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.434+0000 7fd642d19640 1 --2- 172.21.15.116:0/967623429 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd62c004d50 0x7fd62c0051d0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fd634001e70 tx=0x7fd63401c780 comp rx=0 tx=0).stop 2024-04-25T00:16:49.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.434+0000 7fd6412b6640 1 -- 172.21.15.116:0/967623429 reap_dead start 2024-04-25T00:16:49.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.434+0000 7fd642d19640 1 -- 172.21.15.116:0/967623429 shutdown_connections 2024-04-25T00:16:49.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.434+0000 7fd642d19640 1 -- 172.21.15.116:0/967623429 >> 172.21.15.116:0/967623429 conn(0x7fd63c06c720 msgr2=0x7fd63c06cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:49.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.434+0000 7fd642d19640 1 -- 172.21.15.116:0/967623429 shutdown_connections 2024-04-25T00:16:49.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.434+0000 7fd642d19640 1 -- 172.21.15.116:0/967623429 wait complete. 2024-04-25T00:16:49.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.434+0000 7fd642d19640 1 Processor -- start 2024-04-25T00:16:49.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.434+0000 7fd642d19640 1 -- start start 2024-04-25T00:16:49.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.438+0000 7fd642d19640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd62c004d50 0x7fd63c1eede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:49.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.438+0000 7fd642d19640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd63c0abb00 0x7fd63c153d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:49.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.438+0000 7fd642d19640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd63c0c6f70 0x7fd63c144510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:49.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.438+0000 7fd642d19640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd63c156730 con 0x7fd63c0c6f70 2024-04-25T00:16:49.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.438+0000 7fd642d19640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd63c0734e0 con 0x7fd62c004d50 2024-04-25T00:16:49.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.438+0000 7fd642d19640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fd63c072080 con 0x7fd63c0abb00 2024-04-25T00:16:49.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.438+0000 7fd6412b6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd63c0c6f70 0x7fd63c144510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:49.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.438+0000 7fd6412b6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd63c0c6f70 0x7fd63c144510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59150/0 (socket says 172.21.15.116:59150) 2024-04-25T00:16:49.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.438+0000 7fd6412b6640 1 -- 172.21.15.116:0/987378808 learned_addr learned my addr 172.21.15.116:0/987378808 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:16:49.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.438+0000 7fd63bfff640 1 --2- 172.21.15.116:0/987378808 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd63c0abb00 0x7fd63c153d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:49.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.438+0000 7fd640ab5640 1 --2- 172.21.15.116:0/987378808 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd62c004d50 0x7fd63c1eede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:49.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.438+0000 7fd6412b6640 1 -- 172.21.15.116:0/987378808 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd63c0abb00 msgr2=0x7fd63c153d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:49.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.438+0000 7fd6412b6640 1 --2- 172.21.15.116:0/987378808 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd63c0abb00 0x7fd63c153d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:49.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.438+0000 7fd6412b6640 1 -- 172.21.15.116:0/987378808 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd62c004d50 msgr2=0x7fd63c1eede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:49.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.438+0000 7fd6412b6640 1 --2- 172.21.15.116:0/987378808 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd62c004d50 0x7fd63c1eede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:49.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.438+0000 7fd6412b6640 1 -- 172.21.15.116:0/987378808 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd63c1ef320 con 0x7fd63c0c6f70 2024-04-25T00:16:49.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.438+0000 7fd6412b6640 1 --2- 172.21.15.116:0/987378808 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd63c0c6f70 0x7fd63c144510 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fd6340ab040 tx=0x7fd62c005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:49.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.438+0000 7fd6397fa640 1 -- 172.21.15.116:0/987378808 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd634007dc0 con 0x7fd63c0c6f70 2024-04-25T00:16:49.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.438+0000 7fd6397fa640 1 -- 172.21.15.116:0/987378808 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd634003980 con 0x7fd63c0c6f70 2024-04-25T00:16:49.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.438+0000 7fd642d19640 1 -- 172.21.15.116:0/987378808 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd63c0c7d30 con 0x7fd63c0c6f70 2024-04-25T00:16:49.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.438+0000 7fd6397fa640 1 -- 172.21.15.116:0/987378808 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd634005460 con 0x7fd63c0c6f70 2024-04-25T00:16:49.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.438+0000 7fd642d19640 1 -- 172.21.15.116:0/987378808 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd63c0acf80 con 0x7fd63c0c6f70 2024-04-25T00:16:49.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.438+0000 7fd642d19640 1 -- 172.21.15.116:0/987378808 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd63c068a60 con 0x7fd63c0c6f70 2024-04-25T00:16:49.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.438+0000 7fd6397fa640 1 -- 172.21.15.116:0/987378808 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd6340be020 con 0x7fd63c0c6f70 2024-04-25T00:16:49.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.442+0000 7fd6397fa640 1 --2- 172.21.15.116:0/987378808 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd6200678c0 0x7fd620069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:49.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.442+0000 7fd6397fa640 1 -- 172.21.15.116:0/987378808 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 8644+0+0 (secure 0 0 0) 0x7fd634120820 con 0x7fd63c0c6f70 2024-04-25T00:16:49.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.442+0000 7fd640ab5640 1 --2- 172.21.15.116:0/987378808 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd6200678c0 0x7fd620069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:49.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.442+0000 7fd640ab5640 1 --2- 172.21.15.116:0/987378808 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd6200678c0 0x7fd620069d80 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fd630006f10 tx=0x7fd630008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:49.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.446+0000 7fd6397fa640 1 -- 172.21.15.116:0/987378808 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fd6340c5020 con 0x7fd63c0c6f70 2024-04-25T00:16:49.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.650+0000 7fd642d19640 1 -- 172.21.15.116:0/987378808 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd63c0c24e0 con 0x7fd63c0c6f70 2024-04-25T00:16:49.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.654+0000 7fd6397fa640 1 -- 172.21.15.116:0/987378808 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 126 v126) ==== 95+0+5680 (secure 0 0 0) 0x7fd63c1ef320 con 0x7fd63c0c6f70 2024-04-25T00:16:49.652 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:16:49.652 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":126,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:16:48.854104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7119},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7119":{"gid":7119,"name":"a","rank":0,"incarnation":123,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6833/3645553116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3645553116},{"type":"v1","addr":"172.21.15.116:6833","nonce":3645553116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7119,"qdb_cluster":[7119]},"id":1},{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:16:38.790885+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5621},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5621":{"gid":5621,"name":"d","rank":0,"incarnation":118,"state":"up:active","state_seq":15,"addr":"172.21.15.174:6837/3081407040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3081407040},{"type":"v1","addr":"172.21.15.174:6837","nonce":3081407040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5621,"qdb_cluster":[5621]},"id":2}]} 2024-04-25T00:16:49.652 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 126 2024-04-25T00:16:49.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.654+0000 7fd642d19640 1 -- 172.21.15.116:0/987378808 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd6200678c0 msgr2=0x7fd620069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:49.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.654+0000 7fd642d19640 1 --2- 172.21.15.116:0/987378808 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd6200678c0 0x7fd620069d80 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fd630006f10 tx=0x7fd630008040 comp rx=0 tx=0).stop 2024-04-25T00:16:49.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.654+0000 7fd642d19640 1 -- 172.21.15.116:0/987378808 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd63c0c6f70 msgr2=0x7fd63c144510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:49.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.654+0000 7fd642d19640 1 --2- 172.21.15.116:0/987378808 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd63c0c6f70 0x7fd63c144510 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fd6340ab040 tx=0x7fd62c005840 comp rx=0 tx=0).stop 2024-04-25T00:16:49.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.654+0000 7fd642d19640 1 -- 172.21.15.116:0/987378808 shutdown_connections 2024-04-25T00:16:49.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.654+0000 7fd642d19640 1 --2- 172.21.15.116:0/987378808 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd6200678c0 0x7fd620069d80 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:49.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.654+0000 7fd642d19640 1 --2- 172.21.15.116:0/987378808 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd63c0c6f70 0x7fd63c144510 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:49.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.654+0000 7fd642d19640 1 --2- 172.21.15.116:0/987378808 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd63c0abb00 0x7fd63c153d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:49.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.654+0000 7fd642d19640 1 --2- 172.21.15.116:0/987378808 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd62c004d50 0x7fd63c1eede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:49.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.654+0000 7fd642d19640 1 -- 172.21.15.116:0/987378808 >> 172.21.15.116:0/987378808 conn(0x7fd63c06c720 msgr2=0x7fd63c0aa040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:49.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.654+0000 7fd642d19640 1 -- 172.21.15.116:0/987378808 shutdown_connections 2024-04-25T00:16:49.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:49.654+0000 7fd642d19640 1 -- 172.21.15.116:0/987378808 wait complete. 2024-04-25T00:16:49.671 INFO:tasks.mds_thrash.fs.[a]:mds.a has gained rank=0, replacing gid=6810 2024-04-25T00:16:49.671 INFO:tasks.mds_thrash.fs.[a]:waiting for 3 secs before reviving mds.c 2024-04-25T00:16:51.832 INFO:tasks.ceph.mds.a.smithi116.stderr:2024-04-25T00:16:51.834+0000 7f9137050640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T00:16:52.671 INFO:tasks.mds_thrash.fs.[a]:reviving mds.c 2024-04-25T00:16:52.671 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-25T00:16:52.672 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-25T00:16:52.674 INFO:tasks.ceph.mds.c:Started 2024-04-25T00:16:52.674 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:16:52.757 INFO:tasks.ceph.mds.c.smithi116.stdout:starting mds.c at 2024-04-25T00:16:52.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.806+0000 7ff9ad644640 1 Processor -- start 2024-04-25T00:16:52.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.806+0000 7ff9ad644640 1 -- start start 2024-04-25T00:16:52.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.806+0000 7ff9ad644640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff9a8150d40 0x7ff9a8154170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:52.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.806+0000 7ff9ad644640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff9a80c4c20 0x7ff9a80c6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:52.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.806+0000 7ff9ad644640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9a8070180 con 0x7ff9a8070770 2024-04-25T00:16:52.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.806+0000 7ff9ad644640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7ff9a8071280 con 0x7ff9a8150d40 2024-04-25T00:16:52.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.806+0000 7ff9ad644640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff9a806ff30 con 0x7ff9a80c4c20 2024-04-25T00:16:52.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.806+0000 7ff9a6ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff9a8150d40 0x7ff9a8154170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:52.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.810+0000 7ff9a6ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff9a8150d40 0x7ff9a8154170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59166/0 (socket says 172.21.15.116:59166) 2024-04-25T00:16:52.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.810+0000 7ff9a6ffd640 1 -- 172.21.15.116:0/2332343819 learned_addr learned my addr 172.21.15.116:0/2332343819 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:16:52.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.810+0000 7ff99ffff640 1 --2- 172.21.15.116:0/2332343819 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff9a80c4c20 0x7ff9a80c6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:52.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.810+0000 7ff9a6ffd640 1 -- 172.21.15.116:0/2332343819 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff9a80c4c20 msgr2=0x7ff9a80c6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:16:52.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.810+0000 7ff9a6ffd640 1 --2- 172.21.15.116:0/2332343819 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff9a80c4c20 0x7ff9a80c6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:52.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.810+0000 7ff9a6ffd640 1 -- 172.21.15.116:0/2332343819 >> v1:172.21.15.174:6789/0 conn(0x7ff9a8070770 legacy=0x7ff9a80ab200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:52.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.810+0000 7ff9a6ffd640 1 -- 172.21.15.116:0/2332343819 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9a8070460 con 0x7ff9a8150d40 2024-04-25T00:16:52.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.810+0000 7ff99ffff640 1 --2- 172.21.15.116:0/2332343819 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff9a80c4c20 0x7ff9a80c6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:16:52.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.810+0000 7ff9a6ffd640 1 --2- 172.21.15.116:0/2332343819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff9a8150d40 0x7ff9a8154170 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7ff9980098a0 tx=0x7ff99801c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e21303d694799989 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:52.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.810+0000 7ff9a67fc640 1 -- 172.21.15.116:0/2332343819 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9980b1070 con 0x7ff9a8150d40 2024-04-25T00:16:52.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.810+0000 7ff9a67fc640 1 -- 172.21.15.116:0/2332343819 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9980ac620 con 0x7ff9a8150d40 2024-04-25T00:16:52.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.810+0000 7ff9a67fc640 1 -- 172.21.15.116:0/2332343819 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9980ac9f0 con 0x7ff9a8150d40 2024-04-25T00:16:52.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.810+0000 7ff9ad644640 1 -- 172.21.15.116:0/2332343819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff9a8150d40 msgr2=0x7ff9a8154170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:52.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.810+0000 7ff9ad644640 1 --2- 172.21.15.116:0/2332343819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff9a8150d40 0x7ff9a8154170 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7ff9980098a0 tx=0x7ff99801c4b0 comp rx=0 tx=0).stop 2024-04-25T00:16:52.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.810+0000 7ff9ad644640 1 -- 172.21.15.116:0/2332343819 shutdown_connections 2024-04-25T00:16:52.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.810+0000 7ff9ad644640 1 --2- 172.21.15.116:0/2332343819 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff9a80c4c20 0x7ff9a80c6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:52.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.810+0000 7ff9ad644640 1 --2- 172.21.15.116:0/2332343819 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff9a8150d40 0x7ff9a8154170 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:52.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.810+0000 7ff9ad644640 1 -- 172.21.15.116:0/2332343819 >> 172.21.15.116:0/2332343819 conn(0x7ff9a806f0c0 msgr2=0x7ff9a806f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:52.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.810+0000 7ff9ad644640 1 -- 172.21.15.116:0/2332343819 shutdown_connections 2024-04-25T00:16:52.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.810+0000 7ff9ad644640 1 -- 172.21.15.116:0/2332343819 wait complete. 2024-04-25T00:16:52.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.814+0000 7ff9ad644640 1 Processor -- start 2024-04-25T00:16:52.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.814+0000 7ff9ad644640 1 -- start start 2024-04-25T00:16:52.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.814+0000 7ff9ad644640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff9a8070770 0x7ff9a8155ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:52.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.814+0000 7ff9ad644640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff9a80c4c20 0x7ff9a8156660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:52.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.814+0000 7ff9ad644640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff9a8150d40 0x7ff9a8156ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:52.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.814+0000 7ff9ad644640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff9a80aba90 con 0x7ff9a8150d40 2024-04-25T00:16:52.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.814+0000 7ff9ad644640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff9a80ab910 con 0x7ff9a8070770 2024-04-25T00:16:52.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.814+0000 7ff9ad644640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff9a80c7360 con 0x7ff9a80c4c20 2024-04-25T00:16:52.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.814+0000 7ff9a77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff9a8150d40 0x7ff9a8156ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:52.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.814+0000 7ff9a77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff9a8150d40 0x7ff9a8156ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59170/0 (socket says 172.21.15.116:59170) 2024-04-25T00:16:52.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.814+0000 7ff9a77fe640 1 -- 172.21.15.116:0/3346366426 learned_addr learned my addr 172.21.15.116:0/3346366426 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:16:52.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.814+0000 7ff99ffff640 1 --2- 172.21.15.116:0/3346366426 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff9a80c4c20 0x7ff9a8156660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:52.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.814+0000 7ff9a6ffd640 1 --2- 172.21.15.116:0/3346366426 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff9a8070770 0x7ff9a8155ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:52.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.814+0000 7ff9a77fe640 1 -- 172.21.15.116:0/3346366426 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff9a80c4c20 msgr2=0x7ff9a8156660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:52.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.814+0000 7ff9a77fe640 1 --2- 172.21.15.116:0/3346366426 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff9a80c4c20 0x7ff9a8156660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:52.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.814+0000 7ff9a77fe640 1 -- 172.21.15.116:0/3346366426 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff9a8070770 msgr2=0x7ff9a8155ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:52.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.814+0000 7ff9a77fe640 1 --2- 172.21.15.116:0/3346366426 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff9a8070770 0x7ff9a8155ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:52.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.814+0000 7ff9a77fe640 1 -- 172.21.15.116:0/3346366426 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9a81570e0 con 0x7ff9a8150d40 2024-04-25T00:16:52.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.814+0000 7ff9a6ffd640 1 --2- 172.21.15.116:0/3346366426 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff9a8070770 0x7ff9a8155ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:16:52.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.814+0000 7ff9a77fe640 1 --2- 172.21.15.116:0/3346366426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff9a8150d40 0x7ff9a8156ba0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7ff9a000c760 tx=0x7ff9a000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:52.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.814+0000 7ff99f7fe640 1 -- 172.21.15.116:0/3346366426 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9a0003b80 con 0x7ff9a8150d40 2024-04-25T00:16:52.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.814+0000 7ff99f7fe640 1 -- 172.21.15.116:0/3346366426 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9a0003d20 con 0x7ff9a8150d40 2024-04-25T00:16:52.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.814+0000 7ff9ad644640 1 -- 172.21.15.116:0/3346366426 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9a814eee0 con 0x7ff9a8150d40 2024-04-25T00:16:52.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.814+0000 7ff99f7fe640 1 -- 172.21.15.116:0/3346366426 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9a000aa90 con 0x7ff9a8150d40 2024-04-25T00:16:52.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.814+0000 7ff9ad644640 1 -- 172.21.15.116:0/3346366426 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9a81547e0 con 0x7ff9a8150d40 2024-04-25T00:16:52.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.818+0000 7ff9ad644640 1 -- 172.21.15.116:0/3346366426 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9a8068a60 con 0x7ff9a8150d40 2024-04-25T00:16:52.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.818+0000 7ff99f7fe640 1 -- 172.21.15.116:0/3346366426 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff9a000ac30 con 0x7ff9a8150d40 2024-04-25T00:16:52.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.818+0000 7ff99f7fe640 1 --2- 172.21.15.116:0/3346366426 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff9880678c0 0x7ff988069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:16:52.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.818+0000 7ff99f7fe640 1 -- 172.21.15.116:0/3346366426 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 8644+0+0 (secure 0 0 0) 0x7ff9a00865e0 con 0x7ff9a8150d40 2024-04-25T00:16:52.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.818+0000 7ff9a6ffd640 1 --2- 172.21.15.116:0/3346366426 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff9880678c0 0x7ff988069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:16:52.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.818+0000 7ff9a6ffd640 1 --2- 172.21.15.116:0/3346366426 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff9880678c0 0x7ff988069d80 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7ff9980ae000 tx=0x7ff9980a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:16:52.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:52.826+0000 7ff99f7fe640 1 -- 172.21.15.116:0/3346366426 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ff9a00565e0 con 0x7ff9a8150d40 2024-04-25T00:16:53.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:53.074+0000 7ff9ad644640 1 -- 172.21.15.116:0/3346366426 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff9a80ae4b0 con 0x7ff9a8150d40 2024-04-25T00:16:53.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:53.074+0000 7ff99f7fe640 1 -- 172.21.15.116:0/3346366426 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) ==== 95+0+6509 (secure 0 0 0) 0x7ff9a0055ee0 con 0x7ff9a8150d40 2024-04-25T00:16:53.074 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:16:53.074 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":127,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7164,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/3336339376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3336339376},{"type":"v1","addr":"172.21.15.116:6835","nonce":3336339376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":127}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:16:48.854104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7119},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7119":{"gid":7119,"name":"a","rank":0,"incarnation":123,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6833/3645553116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3645553116},{"type":"v1","addr":"172.21.15.116:6833","nonce":3645553116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7119,"qdb_cluster":[7119]},"id":1},{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:16:38.790885+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5621},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5621":{"gid":5621,"name":"d","rank":0,"incarnation":118,"state":"up:active","state_seq":15,"addr":"172.21.15.174:6837/3081407040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3081407040},{"type":"v1","addr":"172.21.15.174:6837","nonce":3081407040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5621,"qdb_cluster":[5621]},"id":2}]} 2024-04-25T00:16:53.075 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 127 2024-04-25T00:16:53.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:53.074+0000 7ff9ad644640 1 -- 172.21.15.116:0/3346366426 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff9880678c0 msgr2=0x7ff988069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:53.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:53.074+0000 7ff9ad644640 1 --2- 172.21.15.116:0/3346366426 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff9880678c0 0x7ff988069d80 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7ff9980ae000 tx=0x7ff9980a7000 comp rx=0 tx=0).stop 2024-04-25T00:16:53.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:53.074+0000 7ff9ad644640 1 -- 172.21.15.116:0/3346366426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff9a8150d40 msgr2=0x7ff9a8156ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:16:53.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:53.074+0000 7ff9ad644640 1 --2- 172.21.15.116:0/3346366426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff9a8150d40 0x7ff9a8156ba0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7ff9a000c760 tx=0x7ff9a000cc20 comp rx=0 tx=0).stop 2024-04-25T00:16:53.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:53.074+0000 7ff9ad644640 1 -- 172.21.15.116:0/3346366426 shutdown_connections 2024-04-25T00:16:53.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:53.074+0000 7ff9ad644640 1 --2- 172.21.15.116:0/3346366426 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff9880678c0 0x7ff988069d80 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:53.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:53.074+0000 7ff9ad644640 1 --2- 172.21.15.116:0/3346366426 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff9a8150d40 0x7ff9a8156ba0 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:53.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:53.074+0000 7ff9ad644640 1 --2- 172.21.15.116:0/3346366426 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff9a80c4c20 0x7ff9a8156660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:53.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:53.074+0000 7ff9ad644640 1 --2- 172.21.15.116:0/3346366426 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff9a8070770 0x7ff9a8155ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:16:53.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:53.074+0000 7ff9ad644640 1 -- 172.21.15.116:0/3346366426 >> 172.21.15.116:0/3346366426 conn(0x7ff9a806f0c0 msgr2=0x7ff9a8151ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:16:53.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:53.074+0000 7ff9ad644640 1 -- 172.21.15.116:0/3346366426 shutdown_connections 2024-04-25T00:16:53.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:16:53.074+0000 7ff9ad644640 1 -- 172.21.15.116:0/3346366426 wait complete. 2024-04-25T00:16:53.091 INFO:tasks.mds_thrash.fs.[a]:mds.c reported in up:standby state 2024-04-25T00:16:53.092 INFO:tasks.mds_thrash.fs.[a]:waiting for 28 secs before thrashing 2024-04-25T00:17:06.486 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:17:06.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.625+0000 7f738433b640 1 Processor -- start 2024-04-25T00:17:06.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.625+0000 7f738433b640 1 -- start start 2024-04-25T00:17:06.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.625+0000 7f738433b640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f737c152ce0 0x7f737c06bc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:06.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.625+0000 7f738433b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f737c159970 0x7f737c155720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:06.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.625+0000 7f738433b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f737c155c60 0x7f737c0c3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:06.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.625+0000 7f738433b640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f737c1594e0 con 0x7f737c159970 2024-04-25T00:17:06.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.625+0000 7f738433b640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f737c06c160 con 0x7f737c152ce0 2024-04-25T00:17:06.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.625+0000 7f738433b640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f737c073260 con 0x7f737c155c60 2024-04-25T00:17:06.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.625+0000 7f73818d6640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f737c159970 0x7f737c155720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:06.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.625+0000 7f73818d6640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f737c159970 0x7f737c155720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41422/0 (socket says 172.21.15.116:41422) 2024-04-25T00:17:06.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.625+0000 7f73818d6640 1 -- 172.21.15.116:0/1728286699 learned_addr learned my addr 172.21.15.116:0/1728286699 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:17:06.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.625+0000 7f73820d7640 1 --2- 172.21.15.116:0/1728286699 >> v2:172.21.15.174:3300/0 conn(0x7f737c152ce0 0x7f737c06bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:06.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.629+0000 7f73828d8640 1 --2- 172.21.15.116:0/1728286699 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f737c155c60 0x7f737c0c3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:06.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.629+0000 7f73818d6640 1 -- 172.21.15.116:0/1728286699 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f737c155c60 msgr2=0x7f737c0c3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:06.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.629+0000 7f73818d6640 1 --2- 172.21.15.116:0/1728286699 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f737c155c60 0x7f737c0c3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:06.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.629+0000 7f73818d6640 1 -- 172.21.15.116:0/1728286699 >> v2:172.21.15.174:3300/0 conn(0x7f737c152ce0 msgr2=0x7f737c06bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:06.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.629+0000 7f73818d6640 1 --2- 172.21.15.116:0/1728286699 >> v2:172.21.15.174:3300/0 conn(0x7f737c152ce0 0x7f737c06bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:06.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.629+0000 7f73818d6640 1 -- 172.21.15.116:0/1728286699 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f737c158e90 con 0x7f737c159970 2024-04-25T00:17:06.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.629+0000 7f73820d7640 1 --2- 172.21.15.116:0/1728286699 >> v2:172.21.15.174:3300/0 conn(0x7f737c152ce0 0x7f737c06bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:17:06.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.629+0000 7f73818d6640 1 --2- 172.21.15.116:0/1728286699 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f737c159970 0x7f737c155720 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f737c071800 tx=0x7f736c01c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70d9790467cc04cd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:06.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.629+0000 7f73810d5640 1 -- 172.21.15.116:0/1728286699 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f736c0b0070 con 0x7f737c159970 2024-04-25T00:17:06.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.629+0000 7f73810d5640 1 -- 172.21.15.116:0/1728286699 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f736c0ab600 con 0x7f737c159970 2024-04-25T00:17:06.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.629+0000 7f73810d5640 1 -- 172.21.15.116:0/1728286699 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f736c0ab9d0 con 0x7f737c159970 2024-04-25T00:17:06.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.629+0000 7f738433b640 1 -- 172.21.15.116:0/1728286699 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f737c159970 msgr2=0x7f737c155720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:06.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.629+0000 7f738433b640 1 --2- 172.21.15.116:0/1728286699 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f737c159970 0x7f737c155720 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f737c071800 tx=0x7f736c01c460 comp rx=0 tx=0).stop 2024-04-25T00:17:06.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.629+0000 7f738433b640 1 -- 172.21.15.116:0/1728286699 shutdown_connections 2024-04-25T00:17:06.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.629+0000 7f738433b640 1 --2- 172.21.15.116:0/1728286699 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f737c155c60 0x7f737c0c3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:06.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.629+0000 7f738433b640 1 --2- 172.21.15.116:0/1728286699 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f737c159970 0x7f737c155720 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:06.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.629+0000 7f738433b640 1 --2- 172.21.15.116:0/1728286699 >> v2:172.21.15.174:3300/0 conn(0x7f737c152ce0 0x7f737c06bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:06.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.629+0000 7f738433b640 1 -- 172.21.15.116:0/1728286699 >> 172.21.15.116:0/1728286699 conn(0x7f737c0c4c10 msgr2=0x7f737c0c5020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:06.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.629+0000 7f738433b640 1 -- 172.21.15.116:0/1728286699 shutdown_connections 2024-04-25T00:17:06.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.629+0000 7f738433b640 1 -- 172.21.15.116:0/1728286699 wait complete. 2024-04-25T00:17:06.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.633+0000 7f738433b640 1 Processor -- start 2024-04-25T00:17:06.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.633+0000 7f738433b640 1 -- start start 2024-04-25T00:17:06.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.633+0000 7f738433b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f737c152ce0 0x7f737c14fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:06.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.633+0000 7f738433b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f737c155c60 0x7f737c06ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:06.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.633+0000 7f738433b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f737c159970 0x7f737c13c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:06.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.633+0000 7f738433b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f737c073000 con 0x7f737c159970 2024-04-25T00:17:06.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.633+0000 7f738433b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f737c072e80 con 0x7f737c155c60 2024-04-25T00:17:06.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.633+0000 7f738433b640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f737c0c1d50 con 0x7f737c152ce0 2024-04-25T00:17:06.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.633+0000 7f73828d8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f737c159970 0x7f737c13c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:06.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.633+0000 7f73828d8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f737c159970 0x7f737c13c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41428/0 (socket says 172.21.15.116:41428) 2024-04-25T00:17:06.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.633+0000 7f73828d8640 1 -- 172.21.15.116:0/1619306083 learned_addr learned my addr 172.21.15.116:0/1619306083 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:17:06.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.633+0000 7f73820d7640 1 --2- 172.21.15.116:0/1619306083 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f737c152ce0 0x7f737c14fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:06.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.633+0000 7f73818d6640 1 --2- 172.21.15.116:0/1619306083 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f737c155c60 0x7f737c06ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:06.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.633+0000 7f73828d8640 1 -- 172.21.15.116:0/1619306083 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f737c152ce0 msgr2=0x7f737c14fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:06.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.633+0000 7f73828d8640 1 --2- 172.21.15.116:0/1619306083 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f737c152ce0 0x7f737c14fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:06.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.633+0000 7f73828d8640 1 -- 172.21.15.116:0/1619306083 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f737c155c60 msgr2=0x7f737c06ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:06.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.633+0000 7f73828d8640 1 --2- 172.21.15.116:0/1619306083 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f737c155c60 0x7f737c06ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:06.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.633+0000 7f73828d8640 1 -- 172.21.15.116:0/1619306083 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f737c0c60d0 con 0x7f737c159970 2024-04-25T00:17:06.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.633+0000 7f73818d6640 1 --2- 172.21.15.116:0/1619306083 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f737c155c60 0x7f737c06ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:17:06.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.633+0000 7f73820d7640 1 --2- 172.21.15.116:0/1619306083 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f737c152ce0 0x7f737c14fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:17:06.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.633+0000 7f73828d8640 1 --2- 172.21.15.116:0/1619306083 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f737c159970 0x7f737c13c670 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f7374007e40 tx=0x7f737400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:06.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.633+0000 7f7372ffd640 1 -- 172.21.15.116:0/1619306083 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7374014620 con 0x7f737c159970 2024-04-25T00:17:06.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.633+0000 7f7372ffd640 1 -- 172.21.15.116:0/1619306083 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7374019070 con 0x7f737c159970 2024-04-25T00:17:06.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.633+0000 7f7372ffd640 1 -- 172.21.15.116:0/1619306083 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7374014a20 con 0x7f737c159970 2024-04-25T00:17:06.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.633+0000 7f738433b640 1 -- 172.21.15.116:0/1619306083 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f737c06de70 con 0x7f737c159970 2024-04-25T00:17:06.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.633+0000 7f738433b640 1 -- 172.21.15.116:0/1619306083 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f737c06e430 con 0x7f737c159970 2024-04-25T00:17:06.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.637+0000 7f738433b640 1 -- 172.21.15.116:0/1619306083 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f737c068a60 con 0x7f737c159970 2024-04-25T00:17:06.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.637+0000 7f7372ffd640 1 -- 172.21.15.116:0/1619306083 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7374014bc0 con 0x7f737c159970 2024-04-25T00:17:06.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.637+0000 7f7372ffd640 1 --2- 172.21.15.116:0/1619306083 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f73600678c0 0x7f7360069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:06.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.637+0000 7f7372ffd640 1 -- 172.21.15.116:0/1619306083 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 8644+0+0 (secure 0 0 0) 0x7f7374087040 con 0x7f737c159970 2024-04-25T00:17:06.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.637+0000 7f73820d7640 1 --2- 172.21.15.116:0/1619306083 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f73600678c0 0x7f7360069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:06.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.637+0000 7f73820d7640 1 --2- 172.21.15.116:0/1619306083 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f73600678c0 0x7f7360069d80 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f7378009970 tx=0x7f7378008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:06.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.641+0000 7f7372ffd640 1 -- 172.21.15.116:0/1619306083 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f7374057040 con 0x7f737c159970 2024-04-25T00:17:06.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.889+0000 7f738433b640 1 -- 172.21.15.116:0/1619306083 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f737c0ad8a0 con 0x7f737c159970 2024-04-25T00:17:06.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.889+0000 7f7372ffd640 1 -- 172.21.15.116:0/1619306083 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) ==== 95+0+6509 (secure 0 0 0) 0x7f7374056940 con 0x7f737c159970 2024-04-25T00:17:06.890 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:17:06.890 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":127,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7164,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/3336339376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3336339376},{"type":"v1","addr":"172.21.15.116:6835","nonce":3336339376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":127}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:16:48.854104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7119},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7119":{"gid":7119,"name":"a","rank":0,"incarnation":123,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6833/3645553116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3645553116},{"type":"v1","addr":"172.21.15.116:6833","nonce":3645553116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7119,"qdb_cluster":[7119]},"id":1},{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:16:38.790885+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5621},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5621":{"gid":5621,"name":"d","rank":0,"incarnation":118,"state":"up:active","state_seq":15,"addr":"172.21.15.174:6837/3081407040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3081407040},{"type":"v1","addr":"172.21.15.174:6837","nonce":3081407040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5621,"qdb_cluster":[5621]},"id":2}]} 2024-04-25T00:17:06.890 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 127 2024-04-25T00:17:06.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.893+0000 7f738433b640 1 -- 172.21.15.116:0/1619306083 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f73600678c0 msgr2=0x7f7360069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:06.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.893+0000 7f738433b640 1 --2- 172.21.15.116:0/1619306083 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f73600678c0 0x7f7360069d80 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f7378009970 tx=0x7f7378008040 comp rx=0 tx=0).stop 2024-04-25T00:17:06.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.893+0000 7f738433b640 1 -- 172.21.15.116:0/1619306083 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f737c159970 msgr2=0x7f737c13c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:06.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.893+0000 7f738433b640 1 --2- 172.21.15.116:0/1619306083 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f737c159970 0x7f737c13c670 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f7374007e40 tx=0x7f737400e4b0 comp rx=0 tx=0).stop 2024-04-25T00:17:06.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.893+0000 7f738433b640 1 -- 172.21.15.116:0/1619306083 shutdown_connections 2024-04-25T00:17:06.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.893+0000 7f738433b640 1 --2- 172.21.15.116:0/1619306083 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f73600678c0 0x7f7360069d80 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:06.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.893+0000 7f738433b640 1 --2- 172.21.15.116:0/1619306083 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f737c159970 0x7f737c13c670 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:06.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.893+0000 7f738433b640 1 --2- 172.21.15.116:0/1619306083 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f737c155c60 0x7f737c06ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:06.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.893+0000 7f738433b640 1 --2- 172.21.15.116:0/1619306083 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f737c152ce0 0x7f737c14fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:06.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.893+0000 7f738433b640 1 -- 172.21.15.116:0/1619306083 >> 172.21.15.116:0/1619306083 conn(0x7f737c0c4c10 msgr2=0x7f737c0c74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:06.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.893+0000 7f738433b640 1 -- 172.21.15.116:0/1619306083 shutdown_connections 2024-04-25T00:17:06.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:06.893+0000 7f738433b640 1 -- 172.21.15.116:0/1619306083 wait complete. 2024-04-25T00:17:06.910 INFO:tasks.mds_thrash.fs.[b]:kill mds.d (rank=0) 2024-04-25T00:17:06.911 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-25T00:17:06.911 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:17:06.930 INFO:tasks.ceph.mds.d:Stopped 2024-04-25T00:17:06.930 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:17:07.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.057+0000 7ff164890640 1 Processor -- start 2024-04-25T00:17:07.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.057+0000 7ff164890640 1 -- start start 2024-04-25T00:17:07.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.061+0000 7ff164890640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff1600c5200 0x7ff160154040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:07.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.061+0000 7ff164890640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff160156bb0 0x7ff160157020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:07.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.061+0000 7ff164890640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1600c2b40 con 0x7ff1600c4760 2024-04-25T00:17:07.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.061+0000 7ff164890640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7ff160072b80 con 0x7ff1600c5200 2024-04-25T00:17:07.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.061+0000 7ff164890640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff160154580 con 0x7ff160156bb0 2024-04-25T00:17:07.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.061+0000 7ff15ed9d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff160156bb0 0x7ff160157020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:07.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.061+0000 7ff15e59c640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff1600c5200 0x7ff160154040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:07.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.061+0000 7ff15e59c640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff1600c5200 0x7ff160154040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:51412/0 (socket says 172.21.15.116:51412) 2024-04-25T00:17:07.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.061+0000 7ff15ed9d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff160156bb0 0x7ff160157020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:51614/0 (socket says 172.21.15.116:51614) 2024-04-25T00:17:07.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.061+0000 7ff15e59c640 1 -- 172.21.15.116:0/2686124405 learned_addr learned my addr 172.21.15.116:0/2686124405 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:17:07.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.061+0000 7ff15e59c640 1 -- 172.21.15.116:0/2686124405 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff160156bb0 msgr2=0x7ff160157020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:07.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.061+0000 7ff15e59c640 1 --2- 172.21.15.116:0/2686124405 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff160156bb0 0x7ff160157020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:07.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.061+0000 7ff15e59c640 1 -- 172.21.15.116:0/2686124405 >> v1:172.21.15.174:6789/0 conn(0x7ff1600c4760 legacy=0x7ff16014d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:07.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.061+0000 7ff15e59c640 1 -- 172.21.15.116:0/2686124405 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1600c2db0 con 0x7ff1600c5200 2024-04-25T00:17:07.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.061+0000 7ff15ed9d640 1 --2- 172.21.15.116:0/2686124405 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff160156bb0 0x7ff160157020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:17:07.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.061+0000 7ff15e59c640 1 --2- 172.21.15.116:0/2686124405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1600c5200 0x7ff160154040 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7ff150009870 tx=0x7ff15001c4d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f005c3ef54f1fa12 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:07.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.061+0000 7ff15d59a640 1 -- 172.21.15.116:0/2686124405 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1500b1070 con 0x7ff1600c5200 2024-04-25T00:17:07.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.065+0000 7ff15d59a640 1 -- 172.21.15.116:0/2686124405 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1500ac620 con 0x7ff1600c5200 2024-04-25T00:17:07.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.065+0000 7ff15d59a640 1 -- 172.21.15.116:0/2686124405 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1500ac9f0 con 0x7ff1600c5200 2024-04-25T00:17:07.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.065+0000 7ff164890640 1 -- 172.21.15.116:0/2686124405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1600c5200 msgr2=0x7ff160154040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:07.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.065+0000 7ff164890640 1 --2- 172.21.15.116:0/2686124405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1600c5200 0x7ff160154040 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7ff150009870 tx=0x7ff15001c4d0 comp rx=0 tx=0).stop 2024-04-25T00:17:07.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.065+0000 7ff164890640 1 -- 172.21.15.116:0/2686124405 shutdown_connections 2024-04-25T00:17:07.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.065+0000 7ff164890640 1 --2- 172.21.15.116:0/2686124405 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff160156bb0 0x7ff160157020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:07.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.065+0000 7ff164890640 1 --2- 172.21.15.116:0/2686124405 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1600c5200 0x7ff160154040 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:07.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.065+0000 7ff164890640 1 -- 172.21.15.116:0/2686124405 >> 172.21.15.116:0/2686124405 conn(0x7ff16006c720 msgr2=0x7ff16006cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:07.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.065+0000 7ff164890640 1 -- 172.21.15.116:0/2686124405 shutdown_connections 2024-04-25T00:17:07.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.065+0000 7ff164890640 1 -- 172.21.15.116:0/2686124405 wait complete. 2024-04-25T00:17:07.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.065+0000 7ff164890640 1 Processor -- start 2024-04-25T00:17:07.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.065+0000 7ff164890640 1 -- start start 2024-04-25T00:17:07.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.065+0000 7ff164890640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1600c4760 0x7ff1601ee830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:07.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.065+0000 7ff164890640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff1600c5200 0x7ff160149140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:07.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.065+0000 7ff164890640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff160156bb0 0x7ff16013f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:07.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.065+0000 7ff164890640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff16014dac0 con 0x7ff160156bb0 2024-04-25T00:17:07.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.065+0000 7ff164890640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff16014d940 con 0x7ff1600c4760 2024-04-25T00:17:07.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.065+0000 7ff164890640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff160157560 con 0x7ff1600c5200 2024-04-25T00:17:07.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.065+0000 7ff15ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff160156bb0 0x7ff16013f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:07.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.065+0000 7ff15ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff160156bb0 0x7ff16013f7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41442/0 (socket says 172.21.15.116:41442) 2024-04-25T00:17:07.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.065+0000 7ff15ed9d640 1 -- 172.21.15.116:0/4214082313 learned_addr learned my addr 172.21.15.116:0/4214082313 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:17:07.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.065+0000 7ff15e59c640 1 --2- 172.21.15.116:0/4214082313 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1600c4760 0x7ff1601ee830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:07.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.065+0000 7ff15dd9b640 1 --2- 172.21.15.116:0/4214082313 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff1600c5200 0x7ff160149140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:07.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.065+0000 7ff15ed9d640 1 -- 172.21.15.116:0/4214082313 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff1600c5200 msgr2=0x7ff160149140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:07.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.065+0000 7ff15ed9d640 1 --2- 172.21.15.116:0/4214082313 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff1600c5200 0x7ff160149140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:07.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.065+0000 7ff15ed9d640 1 -- 172.21.15.116:0/4214082313 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1600c4760 msgr2=0x7ff1601ee830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:07.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.065+0000 7ff15ed9d640 1 --2- 172.21.15.116:0/4214082313 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1600c4760 0x7ff1601ee830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:07.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.065+0000 7ff15ed9d640 1 -- 172.21.15.116:0/4214082313 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1600ad1c0 con 0x7ff160156bb0 2024-04-25T00:17:07.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.065+0000 7ff15dd9b640 1 --2- 172.21.15.116:0/4214082313 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff1600c5200 0x7ff160149140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:17:07.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.065+0000 7ff15ed9d640 1 --2- 172.21.15.116:0/4214082313 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff160156bb0 0x7ff16013f7c0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7ff15800efc0 tx=0x7ff158009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:07.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.065+0000 7ff14effd640 1 -- 172.21.15.116:0/4214082313 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff158018030 con 0x7ff160156bb0 2024-04-25T00:17:07.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.065+0000 7ff14effd640 1 -- 172.21.15.116:0/4214082313 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff15800cde0 con 0x7ff160156bb0 2024-04-25T00:17:07.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.065+0000 7ff164890640 1 -- 172.21.15.116:0/4214082313 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1600c5ec0 con 0x7ff160156bb0 2024-04-25T00:17:07.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.065+0000 7ff14effd640 1 -- 172.21.15.116:0/4214082313 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff158003bc0 con 0x7ff160156bb0 2024-04-25T00:17:07.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.065+0000 7ff164890640 1 -- 172.21.15.116:0/4214082313 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1600c6480 con 0x7ff160156bb0 2024-04-25T00:17:07.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.069+0000 7ff164890640 1 -- 172.21.15.116:0/4214082313 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff160068a60 con 0x7ff160156bb0 2024-04-25T00:17:07.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.069+0000 7ff14effd640 1 -- 172.21.15.116:0/4214082313 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff15801b070 con 0x7ff160156bb0 2024-04-25T00:17:07.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.069+0000 7ff14effd640 1 --2- 172.21.15.116:0/4214082313 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff1400678c0 0x7ff140069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:07.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.069+0000 7ff14effd640 1 -- 172.21.15.116:0/4214082313 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 8644+0+0 (secure 0 0 0) 0x7ff158087260 con 0x7ff160156bb0 2024-04-25T00:17:07.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.069+0000 7ff15e59c640 1 --2- 172.21.15.116:0/4214082313 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff1400678c0 0x7ff140069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:07.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.069+0000 7ff15e59c640 1 --2- 172.21.15.116:0/4214082313 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff1400678c0 0x7ff140069d80 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7ff1500ae000 tx=0x7ff1500a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:07.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.077+0000 7ff14effd640 1 -- 172.21.15.116:0/4214082313 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ff158022070 con 0x7ff160156bb0 2024-04-25T00:17:07.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.301+0000 7ff164890640 1 -- 172.21.15.116:0/4214082313 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff16014c160 con 0x7ff160156bb0 2024-04-25T00:17:07.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.301+0000 7ff14effd640 1 -- 172.21.15.116:0/4214082313 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) ==== 95+0+6509 (secure 0 0 0) 0x7ff158010040 con 0x7ff160156bb0 2024-04-25T00:17:07.302 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:17:07.302 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":127,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7164,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/3336339376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3336339376},{"type":"v1","addr":"172.21.15.116:6835","nonce":3336339376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":127}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:16:48.854104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7119},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7119":{"gid":7119,"name":"a","rank":0,"incarnation":123,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6833/3645553116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3645553116},{"type":"v1","addr":"172.21.15.116:6833","nonce":3645553116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7119,"qdb_cluster":[7119]},"id":1},{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:16:38.790885+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5621},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5621":{"gid":5621,"name":"d","rank":0,"incarnation":118,"state":"up:active","state_seq":15,"addr":"172.21.15.174:6837/3081407040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3081407040},{"type":"v1","addr":"172.21.15.174:6837","nonce":3081407040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5621,"qdb_cluster":[5621]},"id":2}]} 2024-04-25T00:17:07.302 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 127 2024-04-25T00:17:07.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.305+0000 7ff164890640 1 -- 172.21.15.116:0/4214082313 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff1400678c0 msgr2=0x7ff140069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:07.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.305+0000 7ff164890640 1 --2- 172.21.15.116:0/4214082313 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff1400678c0 0x7ff140069d80 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7ff1500ae000 tx=0x7ff1500a7000 comp rx=0 tx=0).stop 2024-04-25T00:17:07.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.305+0000 7ff164890640 1 -- 172.21.15.116:0/4214082313 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff160156bb0 msgr2=0x7ff16013f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:07.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.305+0000 7ff164890640 1 --2- 172.21.15.116:0/4214082313 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff160156bb0 0x7ff16013f7c0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7ff15800efc0 tx=0x7ff158009f90 comp rx=0 tx=0).stop 2024-04-25T00:17:07.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.305+0000 7ff164890640 1 -- 172.21.15.116:0/4214082313 shutdown_connections 2024-04-25T00:17:07.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.305+0000 7ff164890640 1 --2- 172.21.15.116:0/4214082313 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff1400678c0 0x7ff140069d80 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:07.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.305+0000 7ff164890640 1 --2- 172.21.15.116:0/4214082313 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff160156bb0 0x7ff16013f7c0 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:07.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.305+0000 7ff164890640 1 --2- 172.21.15.116:0/4214082313 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff1600c5200 0x7ff160149140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:07.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.305+0000 7ff164890640 1 --2- 172.21.15.116:0/4214082313 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1600c4760 0x7ff1601ee830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:07.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.305+0000 7ff164890640 1 -- 172.21.15.116:0/4214082313 >> 172.21.15.116:0/4214082313 conn(0x7ff16006c720 msgr2=0x7ff160142b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:07.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.305+0000 7ff164890640 1 -- 172.21.15.116:0/4214082313 shutdown_connections 2024-04-25T00:17:07.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:07.305+0000 7ff164890640 1 -- 172.21.15.116:0/4214082313 wait complete. 2024-04-25T00:17:07.323 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:17:09.036 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:17:09.039 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:17:09.324 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:17:09.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.453+0000 7fc862abd640 1 Processor -- start 2024-04-25T00:17:09.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.453+0000 7fc862abd640 1 -- start start 2024-04-25T00:17:09.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.457+0000 7fc862abd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc85c14f300 0x7fc85c06c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:09.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.457+0000 7fc862abd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc85c06c720 0x7fc85c155870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:09.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.457+0000 7fc862abd640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc85c155e50 con 0x7fc85c0c2d10 2024-04-25T00:17:09.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.457+0000 7fc862abd640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fc85c0c3780 con 0x7fc85c06c720 2024-04-25T00:17:09.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.457+0000 7fc862abd640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fc85c0c3a60 con 0x7fc85c14f300 2024-04-25T00:17:09.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.457+0000 7fc86105a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc85c06c720 0x7fc85c155870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:09.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.457+0000 7fc85bfff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fc85c0c2d10 0x7fc85c155160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:60768/0 (socket says 172.21.15.116:60768) 2024-04-25T00:17:09.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.457+0000 7fc85bfff640 1 -- 172.21.15.116:0/501686320 learned_addr learned my addr 172.21.15.116:0/501686320 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:17:09.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.457+0000 7fc86105a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc85c06c720 0x7fc85c155870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41452/0 (socket says 172.21.15.116:41452) 2024-04-25T00:17:09.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.457+0000 7fc860859640 1 --2- 172.21.15.116:0/501686320 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc85c14f300 0x7fc85c06c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:09.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.457+0000 7fc86105a640 1 -- 172.21.15.116:0/501686320 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc85c14f300 msgr2=0x7fc85c06c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:09.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.457+0000 7fc86105a640 1 --2- 172.21.15.116:0/501686320 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc85c14f300 0x7fc85c06c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:09.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.457+0000 7fc86105a640 1 -- 172.21.15.116:0/501686320 >> v1:172.21.15.116:6789/0 conn(0x7fc85c0c2d10 legacy=0x7fc85c155160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:09.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.457+0000 7fc86105a640 1 -- 172.21.15.116:0/501686320 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc85c06bcc0 con 0x7fc85c06c720 2024-04-25T00:17:09.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.457+0000 7fc860859640 1 --2- 172.21.15.116:0/501686320 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc85c14f300 0x7fc85c06c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:17:09.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.457+0000 7fc86105a640 1 --2- 172.21.15.116:0/501686320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc85c06c720 0x7fc85c155870 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fc854009d90 tx=0x7fc85401c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec098fa39f47f403 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:09.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.461+0000 7fc85b7fe640 1 -- 172.21.15.116:0/501686320 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8540b2070 con 0x7fc85c06c720 2024-04-25T00:17:09.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.461+0000 7fc85b7fe640 1 -- 172.21.15.116:0/501686320 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8540a5870 con 0x7fc85c06c720 2024-04-25T00:17:09.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.461+0000 7fc85b7fe640 1 -- 172.21.15.116:0/501686320 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8540a5c40 con 0x7fc85c06c720 2024-04-25T00:17:09.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.461+0000 7fc862abd640 1 -- 172.21.15.116:0/501686320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc85c06c720 msgr2=0x7fc85c155870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:09.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.461+0000 7fc862abd640 1 --2- 172.21.15.116:0/501686320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc85c06c720 0x7fc85c155870 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fc854009d90 tx=0x7fc85401c850 comp rx=0 tx=0).stop 2024-04-25T00:17:09.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.461+0000 7fc862abd640 1 -- 172.21.15.116:0/501686320 shutdown_connections 2024-04-25T00:17:09.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.461+0000 7fc862abd640 1 --2- 172.21.15.116:0/501686320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc85c06c720 0x7fc85c155870 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:09.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.461+0000 7fc862abd640 1 --2- 172.21.15.116:0/501686320 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc85c14f300 0x7fc85c06c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:09.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.461+0000 7fc862abd640 1 -- 172.21.15.116:0/501686320 >> 172.21.15.116:0/501686320 conn(0x7fc85c0c4200 msgr2=0x7fc85c0c4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:09.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.461+0000 7fc862abd640 1 -- 172.21.15.116:0/501686320 shutdown_connections 2024-04-25T00:17:09.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.461+0000 7fc862abd640 1 -- 172.21.15.116:0/501686320 wait complete. 2024-04-25T00:17:09.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.461+0000 7fc862abd640 1 Processor -- start 2024-04-25T00:17:09.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.461+0000 7fc862abd640 1 -- start start 2024-04-25T00:17:09.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.461+0000 7fc862abd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc85c06c720 0x7fc85c0bd770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:09.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.461+0000 7fc862abd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc85c0c2d10 0x7fc85c0c69b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:09.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.461+0000 7fc862abd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc85c14f300 0x7fc85c1ec920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:09.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.461+0000 7fc862abd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc85c0716a0 con 0x7fc85c14f300 2024-04-25T00:17:09.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.461+0000 7fc862abd640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc85c071520 con 0x7fc85c06c720 2024-04-25T00:17:09.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.461+0000 7fc862abd640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fc85c15ed10 con 0x7fc85c0c2d10 2024-04-25T00:17:09.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.461+0000 7fc86105a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc85c14f300 0x7fc85c1ec920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:09.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.461+0000 7fc86105a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc85c14f300 0x7fc85c1ec920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41468/0 (socket says 172.21.15.116:41468) 2024-04-25T00:17:09.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.461+0000 7fc86105a640 1 -- 172.21.15.116:0/1950746921 learned_addr learned my addr 172.21.15.116:0/1950746921 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:17:09.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.461+0000 7fc85bfff640 1 --2- 172.21.15.116:0/1950746921 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc85c0c2d10 0x7fc85c0c69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:09.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.461+0000 7fc860859640 1 --2- 172.21.15.116:0/1950746921 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc85c06c720 0x7fc85c0bd770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:09.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.461+0000 7fc86105a640 1 -- 172.21.15.116:0/1950746921 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc85c0c2d10 msgr2=0x7fc85c0c69b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:09.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.461+0000 7fc86105a640 1 --2- 172.21.15.116:0/1950746921 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc85c0c2d10 0x7fc85c0c69b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:09.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.461+0000 7fc86105a640 1 -- 172.21.15.116:0/1950746921 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc85c06c720 msgr2=0x7fc85c0bd770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:09.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.461+0000 7fc86105a640 1 --2- 172.21.15.116:0/1950746921 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc85c06c720 0x7fc85c0bd770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:09.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.461+0000 7fc86105a640 1 -- 172.21.15.116:0/1950746921 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc85c156a90 con 0x7fc85c14f300 2024-04-25T00:17:09.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.461+0000 7fc85bfff640 1 --2- 172.21.15.116:0/1950746921 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc85c0c2d10 0x7fc85c0c69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:17:09.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.461+0000 7fc86105a640 1 --2- 172.21.15.116:0/1950746921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc85c14f300 0x7fc85c1ec920 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fc854009d90 tx=0x7fc8540195f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:09.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.461+0000 7fc8597fa640 1 -- 172.21.15.116:0/1950746921 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8540b2050 con 0x7fc85c14f300 2024-04-25T00:17:09.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.461+0000 7fc8597fa640 1 -- 172.21.15.116:0/1950746921 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8540be070 con 0x7fc85c14f300 2024-04-25T00:17:09.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.461+0000 7fc862abd640 1 -- 172.21.15.116:0/1950746921 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc85c156d70 con 0x7fc85c14f300 2024-04-25T00:17:09.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.461+0000 7fc8597fa640 1 -- 172.21.15.116:0/1950746921 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8540ada70 con 0x7fc85c14f300 2024-04-25T00:17:09.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.461+0000 7fc862abd640 1 -- 172.21.15.116:0/1950746921 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc85c1ec060 con 0x7fc85c14f300 2024-04-25T00:17:09.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.465+0000 7fc862abd640 1 -- 172.21.15.116:0/1950746921 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc85c156f40 con 0x7fc85c14f300 2024-04-25T00:17:09.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.465+0000 7fc8597fa640 1 -- 172.21.15.116:0/1950746921 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc8540adc10 con 0x7fc85c14f300 2024-04-25T00:17:09.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.465+0000 7fc8597fa640 1 --2- 172.21.15.116:0/1950746921 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc8400678c0 0x7fc840069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:09.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.465+0000 7fc8597fa640 1 -- 172.21.15.116:0/1950746921 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 8644+0+0 (secure 0 0 0) 0x7fc854122470 con 0x7fc85c14f300 2024-04-25T00:17:09.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.465+0000 7fc860859640 1 --2- 172.21.15.116:0/1950746921 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc8400678c0 0x7fc840069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:09.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.469+0000 7fc860859640 1 --2- 172.21.15.116:0/1950746921 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc8400678c0 0x7fc840069d80 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fc850006fd0 tx=0x7fc850008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:09.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.469+0000 7fc8597fa640 1 -- 172.21.15.116:0/1950746921 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fc85c156f40 con 0x7fc85c14f300 2024-04-25T00:17:09.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.689+0000 7fc862abd640 1 -- 172.21.15.116:0/1950746921 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc85c068a60 con 0x7fc85c14f300 2024-04-25T00:17:09.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.689+0000 7fc8597fa640 1 -- 172.21.15.116:0/1950746921 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) ==== 95+0+6509 (secure 0 0 0) 0x7fc854016560 con 0x7fc85c14f300 2024-04-25T00:17:09.689 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:17:09.689 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":127,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7164,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/3336339376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3336339376},{"type":"v1","addr":"172.21.15.116:6835","nonce":3336339376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":127}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:16:48.854104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7119},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7119":{"gid":7119,"name":"a","rank":0,"incarnation":123,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6833/3645553116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3645553116},{"type":"v1","addr":"172.21.15.116:6833","nonce":3645553116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7119,"qdb_cluster":[7119]},"id":1},{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:16:38.790885+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5621},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5621":{"gid":5621,"name":"d","rank":0,"incarnation":118,"state":"up:active","state_seq":15,"addr":"172.21.15.174:6837/3081407040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3081407040},{"type":"v1","addr":"172.21.15.174:6837","nonce":3081407040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5621,"qdb_cluster":[5621]},"id":2}]} 2024-04-25T00:17:09.689 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 127 2024-04-25T00:17:09.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.689+0000 7fc862abd640 1 -- 172.21.15.116:0/1950746921 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc8400678c0 msgr2=0x7fc840069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:09.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.689+0000 7fc862abd640 1 --2- 172.21.15.116:0/1950746921 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc8400678c0 0x7fc840069d80 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fc850006fd0 tx=0x7fc850008040 comp rx=0 tx=0).stop 2024-04-25T00:17:09.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.689+0000 7fc862abd640 1 -- 172.21.15.116:0/1950746921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc85c14f300 msgr2=0x7fc85c1ec920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:09.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.689+0000 7fc862abd640 1 --2- 172.21.15.116:0/1950746921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc85c14f300 0x7fc85c1ec920 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fc854009d90 tx=0x7fc8540195f0 comp rx=0 tx=0).stop 2024-04-25T00:17:09.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.689+0000 7fc862abd640 1 -- 172.21.15.116:0/1950746921 shutdown_connections 2024-04-25T00:17:09.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.689+0000 7fc862abd640 1 --2- 172.21.15.116:0/1950746921 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc8400678c0 0x7fc840069d80 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:09.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.689+0000 7fc862abd640 1 --2- 172.21.15.116:0/1950746921 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc85c14f300 0x7fc85c1ec920 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:09.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.689+0000 7fc862abd640 1 --2- 172.21.15.116:0/1950746921 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc85c0c2d10 0x7fc85c0c69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:09.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.689+0000 7fc862abd640 1 --2- 172.21.15.116:0/1950746921 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc85c06c720 0x7fc85c0bd770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:09.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.689+0000 7fc862abd640 1 -- 172.21.15.116:0/1950746921 >> 172.21.15.116:0/1950746921 conn(0x7fc85c0c4200 msgr2=0x7fc85c155270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:09.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.689+0000 7fc862abd640 1 -- 172.21.15.116:0/1950746921 shutdown_connections 2024-04-25T00:17:09.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:09.689+0000 7fc862abd640 1 -- 172.21.15.116:0/1950746921 wait complete. 2024-04-25T00:17:09.707 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:17:11.708 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:17:11.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.841+0000 7fa269c44640 1 Processor -- start 2024-04-25T00:17:11.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.841+0000 7fa269c44640 1 -- start start 2024-04-25T00:17:11.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.845+0000 7fa269c44640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa264152d80 0x7fa264159950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:11.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.845+0000 7fa269c44640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa264072080 0x7fa2641562a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:11.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.845+0000 7fa269c44640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fa2640c36b0 0x7fa2640c5020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:11.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.845+0000 7fa269c44640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fa2641567e0 con 0x7fa264152d80 2024-04-25T00:17:11.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.845+0000 7fa269c44640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fa264072e60 con 0x7fa2640c36b0 2024-04-25T00:17:11.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.845+0000 7fa269c44640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa2640730b0 con 0x7fa264072080 2024-04-25T00:17:11.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.845+0000 7fa2637fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa264152d80 0x7fa264159950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:11.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.845+0000 7fa2637fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa264152d80 0x7fa264159950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41482/0 (socket says 172.21.15.116:41482) 2024-04-25T00:17:11.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.845+0000 7fa2637fe640 1 -- 172.21.15.116:0/209389527 learned_addr learned my addr 172.21.15.116:0/209389527 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:17:11.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.845+0000 7fa2637fe640 1 -- 172.21.15.116:0/209389527 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa264072080 msgr2=0x7fa2641562a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:11.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.845+0000 7fa263fff640 1 --2- 172.21.15.116:0/209389527 >> v2:172.21.15.174:3300/0 conn(0x7fa2640c36b0 0x7fa2640c5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:11.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.845+0000 7fa262ffd640 1 --2- 172.21.15.116:0/209389527 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa264072080 0x7fa2641562a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:11.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.845+0000 7fa2637fe640 1 --2- 172.21.15.116:0/209389527 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa264072080 0x7fa2641562a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:11.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.845+0000 7fa2637fe640 1 -- 172.21.15.116:0/209389527 >> v2:172.21.15.174:3300/0 conn(0x7fa2640c36b0 msgr2=0x7fa2640c5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:11.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.845+0000 7fa2637fe640 1 --2- 172.21.15.116:0/209389527 >> v2:172.21.15.174:3300/0 conn(0x7fa2640c36b0 0x7fa2640c5020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:11.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.845+0000 7fa2637fe640 1 -- 172.21.15.116:0/209389527 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa264073390 con 0x7fa264152d80 2024-04-25T00:17:11.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.845+0000 7fa262ffd640 1 --2- 172.21.15.116:0/209389527 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa264072080 0x7fa2641562a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:17:11.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.845+0000 7fa263fff640 1 --2- 172.21.15.116:0/209389527 >> v2:172.21.15.174:3300/0 conn(0x7fa2640c36b0 0x7fa2640c5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:17:11.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.845+0000 7fa2637fe640 1 --2- 172.21.15.116:0/209389527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa264152d80 0x7fa264159950 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fa2640c19c0 tx=0x7fa25401c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e14da392db28f916 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:11.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.845+0000 7fa2627fc640 1 -- 172.21.15.116:0/209389527 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2540ab620 con 0x7fa264152d80 2024-04-25T00:17:11.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.845+0000 7fa2627fc640 1 -- 172.21.15.116:0/209389527 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2540b0070 con 0x7fa264152d80 2024-04-25T00:17:11.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.845+0000 7fa2627fc640 1 -- 172.21.15.116:0/209389527 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2540ab9e0 con 0x7fa264152d80 2024-04-25T00:17:11.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.849+0000 7fa269c44640 1 -- 172.21.15.116:0/209389527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa264152d80 msgr2=0x7fa264159950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:11.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.849+0000 7fa269c44640 1 --2- 172.21.15.116:0/209389527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa264152d80 0x7fa264159950 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fa2640c19c0 tx=0x7fa25401c460 comp rx=0 tx=0).stop 2024-04-25T00:17:11.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.849+0000 7fa269c44640 1 -- 172.21.15.116:0/209389527 shutdown_connections 2024-04-25T00:17:11.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.849+0000 7fa269c44640 1 --2- 172.21.15.116:0/209389527 >> v2:172.21.15.174:3300/0 conn(0x7fa2640c36b0 0x7fa2640c5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:11.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.849+0000 7fa269c44640 1 --2- 172.21.15.116:0/209389527 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa264072080 0x7fa2641562a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:11.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.849+0000 7fa269c44640 1 --2- 172.21.15.116:0/209389527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa264152d80 0x7fa264159950 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:11.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.849+0000 7fa269c44640 1 -- 172.21.15.116:0/209389527 >> 172.21.15.116:0/209389527 conn(0x7fa26406c720 msgr2=0x7fa26406cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:11.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.849+0000 7fa269c44640 1 -- 172.21.15.116:0/209389527 shutdown_connections 2024-04-25T00:17:11.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.849+0000 7fa269c44640 1 -- 172.21.15.116:0/209389527 wait complete. 2024-04-25T00:17:11.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.849+0000 7fa269c44640 1 Processor -- start 2024-04-25T00:17:11.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.849+0000 7fa269c44640 1 -- start start 2024-04-25T00:17:11.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.849+0000 7fa269c44640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa264072080 0x7fa264155b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:11.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.849+0000 7fa269c44640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2640c36b0 0x7fa2640c61a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:11.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.849+0000 7fa269c44640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa264152d80 0x7fa2640c6d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:11.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.849+0000 7fa269c44640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa2641532e0 con 0x7fa2640c36b0 2024-04-25T00:17:11.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.849+0000 7fa269c44640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa264153160 con 0x7fa264152d80 2024-04-25T00:17:11.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.849+0000 7fa269c44640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa26406ba20 con 0x7fa264072080 2024-04-25T00:17:11.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.849+0000 7fa262ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2640c36b0 0x7fa2640c61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:11.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.849+0000 7fa262ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2640c36b0 0x7fa2640c61a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41490/0 (socket says 172.21.15.116:41490) 2024-04-25T00:17:11.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.849+0000 7fa262ffd640 1 -- 172.21.15.116:0/2556002518 learned_addr learned my addr 172.21.15.116:0/2556002518 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:17:11.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.849+0000 7fa263fff640 1 --2- 172.21.15.116:0/2556002518 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa264152d80 0x7fa2640c6d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:11.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.849+0000 7fa2637fe640 1 --2- 172.21.15.116:0/2556002518 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa264072080 0x7fa264155b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:11.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.849+0000 7fa262ffd640 1 -- 172.21.15.116:0/2556002518 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa264072080 msgr2=0x7fa264155b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:11.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.849+0000 7fa262ffd640 1 --2- 172.21.15.116:0/2556002518 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa264072080 0x7fa264155b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:11.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.849+0000 7fa262ffd640 1 -- 172.21.15.116:0/2556002518 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa264152d80 msgr2=0x7fa2640c6d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:11.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.849+0000 7fa262ffd640 1 --2- 172.21.15.116:0/2556002518 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa264152d80 0x7fa2640c6d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:11.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.849+0000 7fa262ffd640 1 -- 172.21.15.116:0/2556002518 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2641eec80 con 0x7fa2640c36b0 2024-04-25T00:17:11.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.849+0000 7fa262ffd640 1 --2- 172.21.15.116:0/2556002518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2640c36b0 0x7fa2640c61a0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7fa25800c7f0 tx=0x7fa25800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:11.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.849+0000 7fa24bfff640 1 -- 172.21.15.116:0/2556002518 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa258015070 con 0x7fa2640c36b0 2024-04-25T00:17:11.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.849+0000 7fa24bfff640 1 -- 172.21.15.116:0/2556002518 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa258007a30 con 0x7fa2640c36b0 2024-04-25T00:17:11.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.849+0000 7fa269c44640 1 -- 172.21.15.116:0/2556002518 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2641eef60 con 0x7fa2640c36b0 2024-04-25T00:17:11.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.853+0000 7fa24bfff640 1 -- 172.21.15.116:0/2556002518 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa258007e00 con 0x7fa2640c36b0 2024-04-25T00:17:11.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.853+0000 7fa269c44640 1 -- 172.21.15.116:0/2556002518 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa264154400 con 0x7fa2640c36b0 2024-04-25T00:17:11.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.853+0000 7fa269c44640 1 -- 172.21.15.116:0/2556002518 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa264068a60 con 0x7fa2640c36b0 2024-04-25T00:17:11.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.853+0000 7fa24bfff640 1 -- 172.21.15.116:0/2556002518 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa258018070 con 0x7fa2640c36b0 2024-04-25T00:17:11.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.853+0000 7fa24bfff640 1 --2- 172.21.15.116:0/2556002518 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa2500678c0 0x7fa250069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:11.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.853+0000 7fa24bfff640 1 -- 172.21.15.116:0/2556002518 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 8644+0+0 (secure 0 0 0) 0x7fa2580866e0 con 0x7fa2640c36b0 2024-04-25T00:17:11.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.853+0000 7fa2637fe640 1 --2- 172.21.15.116:0/2556002518 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa2500678c0 0x7fa250069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:11.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.853+0000 7fa2637fe640 1 --2- 172.21.15.116:0/2556002518 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa2500678c0 0x7fa250069d80 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fa2540af000 tx=0x7fa2540a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:11.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:11.857+0000 7fa24bfff640 1 -- 172.21.15.116:0/2556002518 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fa258056090 con 0x7fa2640c36b0 2024-04-25T00:17:12.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:12.109+0000 7fa269c44640 1 -- 172.21.15.116:0/2556002518 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa2641ef130 con 0x7fa2640c36b0 2024-04-25T00:17:12.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:12.109+0000 7fa24bfff640 1 -- 172.21.15.116:0/2556002518 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) ==== 95+0+6509 (secure 0 0 0) 0x7fa25805a390 con 0x7fa2640c36b0 2024-04-25T00:17:12.111 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:17:12.111 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":127,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7164,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/3336339376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3336339376},{"type":"v1","addr":"172.21.15.116:6835","nonce":3336339376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":127}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:16:48.854104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7119},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7119":{"gid":7119,"name":"a","rank":0,"incarnation":123,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6833/3645553116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3645553116},{"type":"v1","addr":"172.21.15.116:6833","nonce":3645553116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7119,"qdb_cluster":[7119]},"id":1},{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:16:38.790885+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5621},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5621":{"gid":5621,"name":"d","rank":0,"incarnation":118,"state":"up:active","state_seq":15,"addr":"172.21.15.174:6837/3081407040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3081407040},{"type":"v1","addr":"172.21.15.174:6837","nonce":3081407040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5621,"qdb_cluster":[5621]},"id":2}]} 2024-04-25T00:17:12.111 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 127 2024-04-25T00:17:12.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:12.113+0000 7fa269c44640 1 -- 172.21.15.116:0/2556002518 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa2500678c0 msgr2=0x7fa250069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:12.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:12.113+0000 7fa269c44640 1 --2- 172.21.15.116:0/2556002518 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa2500678c0 0x7fa250069d80 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fa2540af000 tx=0x7fa2540a7000 comp rx=0 tx=0).stop 2024-04-25T00:17:12.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:12.113+0000 7fa269c44640 1 -- 172.21.15.116:0/2556002518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2640c36b0 msgr2=0x7fa2640c61a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:12.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:12.113+0000 7fa269c44640 1 --2- 172.21.15.116:0/2556002518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2640c36b0 0x7fa2640c61a0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7fa25800c7f0 tx=0x7fa25800ccb0 comp rx=0 tx=0).stop 2024-04-25T00:17:12.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:12.113+0000 7fa269c44640 1 -- 172.21.15.116:0/2556002518 shutdown_connections 2024-04-25T00:17:12.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:12.113+0000 7fa269c44640 1 --2- 172.21.15.116:0/2556002518 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa2500678c0 0x7fa250069d80 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:12.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:12.113+0000 7fa269c44640 1 --2- 172.21.15.116:0/2556002518 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa264152d80 0x7fa2640c6d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:12.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:12.113+0000 7fa269c44640 1 --2- 172.21.15.116:0/2556002518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa2640c36b0 0x7fa2640c61a0 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:12.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:12.113+0000 7fa269c44640 1 --2- 172.21.15.116:0/2556002518 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa264072080 0x7fa264155b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:12.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:12.113+0000 7fa269c44640 1 -- 172.21.15.116:0/2556002518 >> 172.21.15.116:0/2556002518 conn(0x7fa26406c720 msgr2=0x7fa2640acb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:12.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:12.113+0000 7fa269c44640 1 -- 172.21.15.116:0/2556002518 shutdown_connections 2024-04-25T00:17:12.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:12.113+0000 7fa269c44640 1 -- 172.21.15.116:0/2556002518 wait complete. 2024-04-25T00:17:12.132 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:17:14.133 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:17:14.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.273+0000 7fcdca718640 1 Processor -- start 2024-04-25T00:17:14.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.273+0000 7fcdca718640 1 -- start start 2024-04-25T00:17:14.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.273+0000 7fcdca718640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcdc4156310 0x7fcdc415df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:14.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.273+0000 7fcdca718640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcdc4072480 con 0x7fcdc406f9b0 2024-04-25T00:17:14.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.273+0000 7fcdca718640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcdc4072760 con 0x7fcdc414fec0 2024-04-25T00:17:14.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.273+0000 7fcdca718640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fcdc41502a0 con 0x7fcdc4156310 2024-04-25T00:17:14.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.273+0000 7fcdc37fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fcdc406f9b0 0x7fcdc4155c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:60782/0 (socket says 172.21.15.116:60782) 2024-04-25T00:17:14.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.273+0000 7fcdc37fe640 1 -- 172.21.15.116:0/921130346 learned_addr learned my addr 172.21.15.116:0/921130346 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:17:14.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.273+0000 7fcdc8cb5640 1 --2- 172.21.15.116:0/921130346 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcdc4156310 0x7fcdc415df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:14.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.273+0000 7fcdc2ffd640 1 -- 172.21.15.116:0/921130346 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3530542134 0 0) 0x7fcdc4072480 con 0x7fcdc406f9b0 2024-04-25T00:17:14.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.277+0000 7fcdc2ffd640 1 -- 172.21.15.116:0/921130346 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcdac003530 con 0x7fcdc406f9b0 2024-04-25T00:17:14.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.277+0000 7fcdc2ffd640 1 -- 172.21.15.116:0/921130346 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3963429077 0 0) 0x7fcdac003530 con 0x7fcdc406f9b0 2024-04-25T00:17:14.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.277+0000 7fcdc2ffd640 1 -- 172.21.15.116:0/921130346 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcdc4156310 msgr2=0x7fcdc415df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:14.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.277+0000 7fcdc2ffd640 1 --2- 172.21.15.116:0/921130346 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcdc4156310 0x7fcdc415df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:14.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.277+0000 7fcdc2ffd640 1 -- 172.21.15.116:0/921130346 >> v1:172.21.15.174:6789/0 conn(0x7fcdc414fec0 legacy=0x7fcdc406c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:14.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.277+0000 7fcdc2ffd640 1 -- 172.21.15.116:0/921130346 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdc4071610 con 0x7fcdc406f9b0 2024-04-25T00:17:14.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.277+0000 7fcdc2ffd640 1 -- 172.21.15.116:0/921130346 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7fcdb80016a0 con 0x7fcdc406f9b0 2024-04-25T00:17:14.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.277+0000 7fcdc2ffd640 1 -- 172.21.15.116:0/921130346 >> v1:172.21.15.116:6789/0 conn(0x7fcdc406f9b0 legacy=0x7fcdc4155c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:14.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.277+0000 7fcdc2ffd640 1 --2- 172.21.15.116:0/921130346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdac0039f0 0x7fcdac003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:14.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.277+0000 7fcdc2ffd640 1 --2- 172.21.15.116:0/921130346 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcdc4156310 0x7fcdac004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:14.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.277+0000 7fcdc2ffd640 1 --2- 172.21.15.116:0/921130346 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcdac004d50 0x7fcdac0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:14.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.277+0000 7fcdc2ffd640 1 -- 172.21.15.116:0/921130346 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc41502a0 con 0x7fcdac0039f0 2024-04-25T00:17:14.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.277+0000 7fcdc2ffd640 1 -- 172.21.15.116:0/921130346 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcdac00dda0 con 0x7fcdac004d50 2024-04-25T00:17:14.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.277+0000 7fcdc2ffd640 1 -- 172.21.15.116:0/921130346 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fcdac00dfa0 con 0x7fcdc4156310 2024-04-25T00:17:14.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.277+0000 7fcdc3fff640 1 --2- 172.21.15.116:0/921130346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdac0039f0 0x7fcdac003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:14.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.277+0000 7fcdc37fe640 1 --2- 172.21.15.116:0/921130346 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcdc4156310 0x7fcdac004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:14.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.277+0000 7fcdc8cb5640 1 --2- 172.21.15.116:0/921130346 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcdac004d50 0x7fcdac0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:14.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.277+0000 7fcdc3fff640 1 -- 172.21.15.116:0/921130346 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcdc4156310 msgr2=0x7fcdac004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:14.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.277+0000 7fcdc3fff640 1 --2- 172.21.15.116:0/921130346 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcdc4156310 0x7fcdac004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:14.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.277+0000 7fcdc3fff640 1 -- 172.21.15.116:0/921130346 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcdac004d50 msgr2=0x7fcdac0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:14.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.277+0000 7fcdc3fff640 1 --2- 172.21.15.116:0/921130346 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcdac004d50 0x7fcdac0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:14.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.277+0000 7fcdc3fff640 1 -- 172.21.15.116:0/921130346 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdac003530 con 0x7fcdac0039f0 2024-04-25T00:17:14.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.277+0000 7fcdc37fe640 1 --2- 172.21.15.116:0/921130346 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcdc4156310 0x7fcdac004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:17:14.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.277+0000 7fcdc8cb5640 1 --2- 172.21.15.116:0/921130346 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcdac004d50 0x7fcdac0051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:17:14.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.277+0000 7fcdc3fff640 1 --2- 172.21.15.116:0/921130346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdac0039f0 0x7fcdac003e10 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7fcdb4001780 tx=0x7fcdb401c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d9d56f192f957fc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:14.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.277+0000 7fcdc2ffd640 1 -- 172.21.15.116:0/921130346 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdb40013a0 con 0x7fcdac0039f0 2024-04-25T00:17:14.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.277+0000 7fcdc2ffd640 1 -- 172.21.15.116:0/921130346 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdb40052f0 con 0x7fcdac0039f0 2024-04-25T00:17:14.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.277+0000 7fcdc2ffd640 1 -- 172.21.15.116:0/921130346 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdb4003840 con 0x7fcdac0039f0 2024-04-25T00:17:14.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.281+0000 7fcdca718640 1 -- 172.21.15.116:0/921130346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdac0039f0 msgr2=0x7fcdac003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:14.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.281+0000 7fcdca718640 1 --2- 172.21.15.116:0/921130346 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdac0039f0 0x7fcdac003e10 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7fcdb4001780 tx=0x7fcdb401c420 comp rx=0 tx=0).stop 2024-04-25T00:17:14.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.281+0000 7fcdc8cb5640 1 -- 172.21.15.116:0/921130346 reap_dead start 2024-04-25T00:17:14.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.281+0000 7fcdca718640 1 -- 172.21.15.116:0/921130346 shutdown_connections 2024-04-25T00:17:14.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.281+0000 7fcdca718640 1 -- 172.21.15.116:0/921130346 >> 172.21.15.116:0/921130346 conn(0x7fcdc40c4170 msgr2=0x7fcdc40c4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:14.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.281+0000 7fcdca718640 1 -- 172.21.15.116:0/921130346 shutdown_connections 2024-04-25T00:17:14.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.281+0000 7fcdca718640 1 -- 172.21.15.116:0/921130346 wait complete. 2024-04-25T00:17:14.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.281+0000 7fcdca718640 1 Processor -- start 2024-04-25T00:17:14.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.281+0000 7fcdca718640 1 -- start start 2024-04-25T00:17:14.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.281+0000 7fcdca718640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcdac0039f0 0x7fcdc40acc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:14.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.281+0000 7fcdca718640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcdc40c6510 0x7fcdc41ea190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:14.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.281+0000 7fcdca718640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdc41ea6d0 0x7fcdc40aaee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:14.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.281+0000 7fcdca718640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc406c930 con 0x7fcdc41ea6d0 2024-04-25T00:17:14.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.281+0000 7fcdca718640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc4071e10 con 0x7fcdc40c6510 2024-04-25T00:17:14.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.281+0000 7fcdca718640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fcdc406cab0 con 0x7fcdac0039f0 2024-04-25T00:17:14.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.281+0000 7fcdc8cb5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdc41ea6d0 0x7fcdc40aaee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:14.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.281+0000 7fcdc8cb5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdc41ea6d0 0x7fcdc40aaee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41520/0 (socket says 172.21.15.116:41520) 2024-04-25T00:17:14.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.281+0000 7fcdc8cb5640 1 -- 172.21.15.116:0/2255036605 learned_addr learned my addr 172.21.15.116:0/2255036605 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:17:14.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.281+0000 7fcdc37fe640 1 --2- 172.21.15.116:0/2255036605 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcdc40c6510 0x7fcdc41ea190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:14.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.281+0000 7fcdc3fff640 1 --2- 172.21.15.116:0/2255036605 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcdac0039f0 0x7fcdc40acc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:14.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.281+0000 7fcdc8cb5640 1 -- 172.21.15.116:0/2255036605 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcdac0039f0 msgr2=0x7fcdc40acc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:14.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.281+0000 7fcdc8cb5640 1 --2- 172.21.15.116:0/2255036605 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcdac0039f0 0x7fcdc40acc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:14.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.281+0000 7fcdc8cb5640 1 -- 172.21.15.116:0/2255036605 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcdc40c6510 msgr2=0x7fcdc41ea190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:14.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.281+0000 7fcdc8cb5640 1 --2- 172.21.15.116:0/2255036605 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcdc40c6510 0x7fcdc41ea190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:14.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.281+0000 7fcdc8cb5640 1 -- 172.21.15.116:0/2255036605 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdc41ee080 con 0x7fcdc41ea6d0 2024-04-25T00:17:14.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.281+0000 7fcdc8cb5640 1 --2- 172.21.15.116:0/2255036605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdc41ea6d0 0x7fcdc40aaee0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fcdc41567a0 tx=0x7fcdbc00f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:14.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.281+0000 7fcdc0ff9640 1 -- 172.21.15.116:0/2255036605 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdbc01c070 con 0x7fcdc41ea6d0 2024-04-25T00:17:14.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.281+0000 7fcdc0ff9640 1 -- 172.21.15.116:0/2255036605 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdbc017400 con 0x7fcdc41ea6d0 2024-04-25T00:17:14.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.281+0000 7fcdca718640 1 -- 172.21.15.116:0/2255036605 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdc41ee360 con 0x7fcdc41ea6d0 2024-04-25T00:17:14.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.281+0000 7fcdc0ff9640 1 -- 172.21.15.116:0/2255036605 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdbc0177d0 con 0x7fcdc41ea6d0 2024-04-25T00:17:14.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.281+0000 7fcdca718640 1 -- 172.21.15.116:0/2255036605 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdc40ac460 con 0x7fcdc41ea6d0 2024-04-25T00:17:14.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.285+0000 7fcdca718640 1 -- 172.21.15.116:0/2255036605 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdc4068a60 con 0x7fcdc41ea6d0 2024-04-25T00:17:14.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.285+0000 7fcdc0ff9640 1 -- 172.21.15.116:0/2255036605 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fcdbc01f070 con 0x7fcdc41ea6d0 2024-04-25T00:17:14.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.285+0000 7fcdc0ff9640 1 --2- 172.21.15.116:0/2255036605 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcdb00678c0 0x7fcdb0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:14.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.285+0000 7fcdc0ff9640 1 -- 172.21.15.116:0/2255036605 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 8644+0+0 (secure 0 0 0) 0x7fcdbc086810 con 0x7fcdc41ea6d0 2024-04-25T00:17:14.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.285+0000 7fcdc3fff640 1 --2- 172.21.15.116:0/2255036605 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcdb00678c0 0x7fcdb0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:14.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.285+0000 7fcdc3fff640 1 --2- 172.21.15.116:0/2255036605 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcdb00678c0 0x7fcdb0069d80 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fcdb4001750 tx=0x7fcdb40011a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:14.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.289+0000 7fcdc0ff9640 1 -- 172.21.15.116:0/2255036605 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fcdbc012070 con 0x7fcdc41ea6d0 2024-04-25T00:17:14.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.517+0000 7fcdca718640 1 -- 172.21.15.116:0/2255036605 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcdc41559e0 con 0x7fcdc41ea6d0 2024-04-25T00:17:14.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.517+0000 7fcdc0ff9640 1 -- 172.21.15.116:0/2255036605 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) ==== 95+0+6509 (secure 0 0 0) 0x7fcdbc0560d0 con 0x7fcdc41ea6d0 2024-04-25T00:17:14.518 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:17:14.518 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":127,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7164,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/3336339376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3336339376},{"type":"v1","addr":"172.21.15.116:6835","nonce":3336339376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":127}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:16:48.854104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7119},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7119":{"gid":7119,"name":"a","rank":0,"incarnation":123,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6833/3645553116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3645553116},{"type":"v1","addr":"172.21.15.116:6833","nonce":3645553116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7119,"qdb_cluster":[7119]},"id":1},{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:16:38.790885+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5621},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5621":{"gid":5621,"name":"d","rank":0,"incarnation":118,"state":"up:active","state_seq":15,"addr":"172.21.15.174:6837/3081407040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3081407040},{"type":"v1","addr":"172.21.15.174:6837","nonce":3081407040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5621,"qdb_cluster":[5621]},"id":2}]} 2024-04-25T00:17:14.518 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 127 2024-04-25T00:17:14.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.521+0000 7fcdca718640 1 -- 172.21.15.116:0/2255036605 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcdb00678c0 msgr2=0x7fcdb0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:14.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.521+0000 7fcdca718640 1 --2- 172.21.15.116:0/2255036605 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcdb00678c0 0x7fcdb0069d80 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fcdb4001750 tx=0x7fcdb40011a0 comp rx=0 tx=0).stop 2024-04-25T00:17:14.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.521+0000 7fcdca718640 1 -- 172.21.15.116:0/2255036605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdc41ea6d0 msgr2=0x7fcdc40aaee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:14.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.521+0000 7fcdca718640 1 --2- 172.21.15.116:0/2255036605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdc41ea6d0 0x7fcdc40aaee0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fcdc41567a0 tx=0x7fcdbc00f4a0 comp rx=0 tx=0).stop 2024-04-25T00:17:14.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.521+0000 7fcdca718640 1 -- 172.21.15.116:0/2255036605 shutdown_connections 2024-04-25T00:17:14.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.521+0000 7fcdca718640 1 --2- 172.21.15.116:0/2255036605 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcdb00678c0 0x7fcdb0069d80 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:14.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.521+0000 7fcdca718640 1 --2- 172.21.15.116:0/2255036605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcdc41ea6d0 0x7fcdc40aaee0 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:14.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.521+0000 7fcdca718640 1 --2- 172.21.15.116:0/2255036605 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcdc40c6510 0x7fcdc41ea190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:14.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.521+0000 7fcdca718640 1 --2- 172.21.15.116:0/2255036605 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcdac0039f0 0x7fcdc40acc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:14.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.521+0000 7fcdca718640 1 -- 172.21.15.116:0/2255036605 >> 172.21.15.116:0/2255036605 conn(0x7fcdc40c4170 msgr2=0x7fcdc40671c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:14.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.521+0000 7fcdca718640 1 -- 172.21.15.116:0/2255036605 shutdown_connections 2024-04-25T00:17:14.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:14.521+0000 7fcdca718640 1 -- 172.21.15.116:0/2255036605 wait complete. 2024-04-25T00:17:14.537 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:17:16.538 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:17:16.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.673+0000 7fa88da3b640 1 Processor -- start 2024-04-25T00:17:16.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.673+0000 7fa88da3b640 1 -- start start 2024-04-25T00:17:16.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.673+0000 7fa88da3b640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fa888072a10 0x7fa8880c1240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:16.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.673+0000 7fa88da3b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa88806b310 0x7fa88806b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:16.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.673+0000 7fa88da3b640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8880c1780 con 0x7fa88814fc00 2024-04-25T00:17:16.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.673+0000 7fa88da3b640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fa8881566a0 con 0x7fa88806b310 2024-04-25T00:17:16.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.673+0000 7fa88da3b640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fa888155ad0 con 0x7fa888072a10 2024-04-25T00:17:16.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.673+0000 7fa8877fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa88806b310 0x7fa88806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:16.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.673+0000 7fa8877fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa88806b310 0x7fa88806b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42230/0 (socket says 172.21.15.116:42230) 2024-04-25T00:17:16.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.673+0000 7fa8877fe640 1 -- 172.21.15.116:0/1135145042 learned_addr learned my addr 172.21.15.116:0/1135145042 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:17:16.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.677+0000 7fa8867fc640 1 --2- 172.21.15.116:0/1135145042 >> v2:172.21.15.174:3300/0 conn(0x7fa888072a10 0x7fa8880c1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:16.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.677+0000 7fa8877fe640 1 -- 172.21.15.116:0/1135145042 >> v2:172.21.15.174:3300/0 conn(0x7fa888072a10 msgr2=0x7fa8880c1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:16.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.677+0000 7fa8877fe640 1 --2- 172.21.15.116:0/1135145042 >> v2:172.21.15.174:3300/0 conn(0x7fa888072a10 0x7fa8880c1240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:16.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.677+0000 7fa8877fe640 1 -- 172.21.15.116:0/1135145042 >> v1:172.21.15.174:6789/0 conn(0x7fa88814fc00 legacy=0x7fa8880c0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:16.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.677+0000 7fa8877fe640 1 -- 172.21.15.116:0/1135145042 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa888155d70 con 0x7fa88806b310 2024-04-25T00:17:16.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.677+0000 7fa8867fc640 1 --2- 172.21.15.116:0/1135145042 >> v2:172.21.15.174:3300/0 conn(0x7fa888072a10 0x7fa8880c1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:17:16.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.677+0000 7fa8877fe640 1 --2- 172.21.15.116:0/1135145042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa88806b310 0x7fa88806b760 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7fa8880c8670 tx=0x7fa88001c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d2d2d804d58aab2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:16.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.677+0000 7fa885ffb640 1 -- 172.21.15.116:0/1135145042 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8800a56c0 con 0x7fa88806b310 2024-04-25T00:17:16.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.677+0000 7fa885ffb640 1 -- 172.21.15.116:0/1135145042 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8800a5860 con 0x7fa88806b310 2024-04-25T00:17:16.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.677+0000 7fa885ffb640 1 -- 172.21.15.116:0/1135145042 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8800a5c30 con 0x7fa88806b310 2024-04-25T00:17:16.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.677+0000 7fa88da3b640 1 -- 172.21.15.116:0/1135145042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa88806b310 msgr2=0x7fa88806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:16.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.677+0000 7fa88da3b640 1 --2- 172.21.15.116:0/1135145042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa88806b310 0x7fa88806b760 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7fa8880c8670 tx=0x7fa88001c9f0 comp rx=0 tx=0).stop 2024-04-25T00:17:16.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.677+0000 7fa88da3b640 1 -- 172.21.15.116:0/1135145042 shutdown_connections 2024-04-25T00:17:16.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.677+0000 7fa88da3b640 1 --2- 172.21.15.116:0/1135145042 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa88806b310 0x7fa88806b760 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:16.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.677+0000 7fa88da3b640 1 --2- 172.21.15.116:0/1135145042 >> v2:172.21.15.174:3300/0 conn(0x7fa888072a10 0x7fa8880c1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:16.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.677+0000 7fa88da3b640 1 -- 172.21.15.116:0/1135145042 >> 172.21.15.116:0/1135145042 conn(0x7fa8880c4200 msgr2=0x7fa8880c4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:16.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.677+0000 7fa88da3b640 1 -- 172.21.15.116:0/1135145042 shutdown_connections 2024-04-25T00:17:16.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.677+0000 7fa88da3b640 1 -- 172.21.15.116:0/1135145042 wait complete. 2024-04-25T00:17:16.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.681+0000 7fa88da3b640 1 Processor -- start 2024-04-25T00:17:16.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.681+0000 7fa88da3b640 1 -- start start 2024-04-25T00:17:16.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.681+0000 7fa88da3b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa88806b310 0x7fa8880acc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:16.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.681+0000 7fa88da3b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa888072a10 0x7fa8881f2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:16.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.681+0000 7fa88da3b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa88814fc00 0x7fa8880aba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:16.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.681+0000 7fa88da3b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa888162b00 con 0x7fa888072a10 2024-04-25T00:17:16.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.681+0000 7fa88da3b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa888162980 con 0x7fa88814fc00 2024-04-25T00:17:16.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.681+0000 7fa88da3b640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa888162c80 con 0x7fa88806b310 2024-04-25T00:17:16.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.681+0000 7fa8867fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa888072a10 0x7fa8881f2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:16.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.681+0000 7fa8867fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa888072a10 0x7fa8881f2090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42240/0 (socket says 172.21.15.116:42240) 2024-04-25T00:17:16.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.681+0000 7fa8867fc640 1 -- 172.21.15.116:0/3527137328 learned_addr learned my addr 172.21.15.116:0/3527137328 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:17:16.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.681+0000 7fa8877fe640 1 --2- 172.21.15.116:0/3527137328 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa88814fc00 0x7fa8880aba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:16.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.681+0000 7fa886ffd640 1 --2- 172.21.15.116:0/3527137328 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa88806b310 0x7fa8880acc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:16.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.681+0000 7fa8867fc640 1 -- 172.21.15.116:0/3527137328 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa88806b310 msgr2=0x7fa8880acc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:16.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.681+0000 7fa8867fc640 1 --2- 172.21.15.116:0/3527137328 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa88806b310 0x7fa8880acc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:16.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.681+0000 7fa8867fc640 1 -- 172.21.15.116:0/3527137328 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa88814fc00 msgr2=0x7fa8880aba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:16.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.681+0000 7fa8867fc640 1 --2- 172.21.15.116:0/3527137328 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa88814fc00 0x7fa8880aba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:16.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.681+0000 7fa8867fc640 1 -- 172.21.15.116:0/3527137328 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8880ad140 con 0x7fa888072a10 2024-04-25T00:17:16.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.681+0000 7fa8867fc640 1 --2- 172.21.15.116:0/3527137328 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa888072a10 0x7fa8881f2090 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7fa87c00c820 tx=0x7fa87c00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:16.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.681+0000 7fa86ffff640 1 -- 172.21.15.116:0/3527137328 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa87c015070 con 0x7fa888072a10 2024-04-25T00:17:16.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.681+0000 7fa86ffff640 1 -- 172.21.15.116:0/3527137328 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa87c007a30 con 0x7fa888072a10 2024-04-25T00:17:16.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.681+0000 7fa88da3b640 1 -- 172.21.15.116:0/3527137328 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa88813da40 con 0x7fa888072a10 2024-04-25T00:17:16.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.681+0000 7fa86ffff640 1 -- 172.21.15.116:0/3527137328 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa87c007e40 con 0x7fa888072a10 2024-04-25T00:17:16.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.681+0000 7fa88da3b640 1 -- 172.21.15.116:0/3527137328 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa88814aa70 con 0x7fa888072a10 2024-04-25T00:17:16.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.681+0000 7fa88da3b640 1 -- 172.21.15.116:0/3527137328 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa888068a60 con 0x7fa888072a10 2024-04-25T00:17:16.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.685+0000 7fa86ffff640 1 -- 172.21.15.116:0/3527137328 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa87c018070 con 0x7fa888072a10 2024-04-25T00:17:16.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.685+0000 7fa86ffff640 1 --2- 172.21.15.116:0/3527137328 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa8740678c0 0x7fa874069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:16.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.685+0000 7fa86ffff640 1 -- 172.21.15.116:0/3527137328 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 8644+0+0 (secure 0 0 0) 0x7fa87c0866e0 con 0x7fa888072a10 2024-04-25T00:17:16.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.685+0000 7fa886ffd640 1 --2- 172.21.15.116:0/3527137328 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa8740678c0 0x7fa874069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:16.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.685+0000 7fa886ffd640 1 --2- 172.21.15.116:0/3527137328 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa8740678c0 0x7fa874069d80 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fa8780015b0 tx=0x7fa878009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:16.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.693+0000 7fa86ffff640 1 -- 172.21.15.116:0/3527137328 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fa87c056680 con 0x7fa888072a10 2024-04-25T00:17:16.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.929+0000 7fa88da3b640 1 -- 172.21.15.116:0/3527137328 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa8880c4840 con 0x7fa888072a10 2024-04-25T00:17:16.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.929+0000 7fa86ffff640 1 -- 172.21.15.116:0/3527137328 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) ==== 95+0+6509 (secure 0 0 0) 0x7fa87c055f80 con 0x7fa888072a10 2024-04-25T00:17:16.932 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:17:16.932 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":127,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7164,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/3336339376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3336339376},{"type":"v1","addr":"172.21.15.116:6835","nonce":3336339376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":127}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:16:48.854104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7119},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7119":{"gid":7119,"name":"a","rank":0,"incarnation":123,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6833/3645553116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3645553116},{"type":"v1","addr":"172.21.15.116:6833","nonce":3645553116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7119,"qdb_cluster":[7119]},"id":1},{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:16:38.790885+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5621},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5621":{"gid":5621,"name":"d","rank":0,"incarnation":118,"state":"up:active","state_seq":15,"addr":"172.21.15.174:6837/3081407040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3081407040},{"type":"v1","addr":"172.21.15.174:6837","nonce":3081407040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5621,"qdb_cluster":[5621]},"id":2}]} 2024-04-25T00:17:16.932 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 127 2024-04-25T00:17:16.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.933+0000 7fa88da3b640 1 -- 172.21.15.116:0/3527137328 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa8740678c0 msgr2=0x7fa874069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:16.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.933+0000 7fa88da3b640 1 --2- 172.21.15.116:0/3527137328 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa8740678c0 0x7fa874069d80 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fa8780015b0 tx=0x7fa878009040 comp rx=0 tx=0).stop 2024-04-25T00:17:16.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.933+0000 7fa88da3b640 1 -- 172.21.15.116:0/3527137328 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa888072a10 msgr2=0x7fa8881f2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:16.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.933+0000 7fa88da3b640 1 --2- 172.21.15.116:0/3527137328 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa888072a10 0x7fa8881f2090 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7fa87c00c820 tx=0x7fa87c00cce0 comp rx=0 tx=0).stop 2024-04-25T00:17:16.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.933+0000 7fa88da3b640 1 -- 172.21.15.116:0/3527137328 shutdown_connections 2024-04-25T00:17:16.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.933+0000 7fa88da3b640 1 --2- 172.21.15.116:0/3527137328 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa8740678c0 0x7fa874069d80 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:16.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.933+0000 7fa88da3b640 1 --2- 172.21.15.116:0/3527137328 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa88814fc00 0x7fa8880aba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:16.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.933+0000 7fa88da3b640 1 --2- 172.21.15.116:0/3527137328 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa888072a10 0x7fa8881f2090 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:16.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.933+0000 7fa88da3b640 1 --2- 172.21.15.116:0/3527137328 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa88806b310 0x7fa8880acc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:16.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.933+0000 7fa88da3b640 1 -- 172.21.15.116:0/3527137328 >> 172.21.15.116:0/3527137328 conn(0x7fa8880c4200 msgr2=0x7fa8880c0b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:16.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.933+0000 7fa88da3b640 1 -- 172.21.15.116:0/3527137328 shutdown_connections 2024-04-25T00:17:16.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:16.933+0000 7fa88da3b640 1 -- 172.21.15.116:0/3527137328 wait complete. 2024-04-25T00:17:16.951 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:17:18.952 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:17:19.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.089+0000 7f5f03605640 1 Processor -- start 2024-04-25T00:17:19.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.089+0000 7f5f03605640 1 -- start start 2024-04-25T00:17:19.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.093+0000 7f5f03605640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5efc153570 0x7f5efc06eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:19.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.093+0000 7f5f03605640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f5efc0c19f0 0x7f5efc06f000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:19.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.093+0000 7f5f03605640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5efc153aa0 con 0x7f5efc150bf0 2024-04-25T00:17:19.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.093+0000 7f5f03605640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f5efc06e630 con 0x7f5efc0c19f0 2024-04-25T00:17:19.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.093+0000 7f5f03605640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f5efc150fd0 con 0x7f5efc153570 2024-04-25T00:17:19.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.093+0000 7f5f013a1640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f5efc150bf0 0x7f5efc06c220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:48144/0 (socket says 172.21.15.116:48144) 2024-04-25T00:17:19.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.093+0000 7f5f013a1640 1 -- 172.21.15.116:0/749800285 learned_addr learned my addr 172.21.15.116:0/749800285 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:17:19.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.093+0000 7f5f00ba0640 1 --2- 172.21.15.116:0/749800285 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5efc153570 0x7f5efc06eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:19.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.093+0000 7f5f01ba2640 1 --2- 172.21.15.116:0/749800285 >> v2:172.21.15.174:3300/0 conn(0x7f5efc0c19f0 0x7f5efc06f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:19.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.093+0000 7f5ef3fff640 1 -- 172.21.15.116:0/749800285 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2088983907 0 0) 0x7f5efc153aa0 con 0x7f5efc150bf0 2024-04-25T00:17:19.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.093+0000 7f5ef3fff640 1 -- 172.21.15.116:0/749800285 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ee8003530 con 0x7f5efc150bf0 2024-04-25T00:17:19.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.093+0000 7f5ef3fff640 1 -- 172.21.15.116:0/749800285 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 661889874 0 0) 0x7f5ee8003530 con 0x7f5efc150bf0 2024-04-25T00:17:19.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.093+0000 7f5ef3fff640 1 -- 172.21.15.116:0/749800285 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5efc153570 msgr2=0x7f5efc06eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:19.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.093+0000 7f5ef3fff640 1 --2- 172.21.15.116:0/749800285 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5efc153570 0x7f5efc06eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:19.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.093+0000 7f5ef3fff640 1 -- 172.21.15.116:0/749800285 >> v2:172.21.15.174:3300/0 conn(0x7f5efc0c19f0 msgr2=0x7f5efc06f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:19.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.093+0000 7f5ef3fff640 1 --2- 172.21.15.116:0/749800285 >> v2:172.21.15.174:3300/0 conn(0x7f5efc0c19f0 0x7f5efc06f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:19.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.093+0000 7f5ef3fff640 1 -- 172.21.15.116:0/749800285 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5efc153d80 con 0x7f5efc150bf0 2024-04-25T00:17:19.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.093+0000 7f5ef3fff640 1 -- 172.21.15.116:0/749800285 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f5ef8001680 con 0x7f5efc150bf0 2024-04-25T00:17:19.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.093+0000 7f5ef3fff640 1 -- 172.21.15.116:0/749800285 >> v1:172.21.15.116:6789/0 conn(0x7f5efc150bf0 legacy=0x7f5efc06c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:19.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.093+0000 7f5ef3fff640 1 --2- 172.21.15.116:0/749800285 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5efc153570 0x7f5ee80039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:19.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.093+0000 7f5ef3fff640 1 --2- 172.21.15.116:0/749800285 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ee8003f60 0x7f5ee8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:19.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.093+0000 7f5ef3fff640 1 --2- 172.21.15.116:0/749800285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ee8004d50 0x7f5ee80051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:19.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.093+0000 7f5ef3fff640 1 -- 172.21.15.116:0/749800285 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5efc06e630 con 0x7f5ee8004d50 2024-04-25T00:17:19.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.093+0000 7f5ef3fff640 1 -- 172.21.15.116:0/749800285 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5efc150fd0 con 0x7f5ee8003f60 2024-04-25T00:17:19.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.093+0000 7f5ef3fff640 1 -- 172.21.15.116:0/749800285 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f5ee800ba50 con 0x7f5efc153570 2024-04-25T00:17:19.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.093+0000 7f5f01ba2640 1 --2- 172.21.15.116:0/749800285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ee8004d50 0x7f5ee80051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:19.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.093+0000 7f5f00ba0640 1 --2- 172.21.15.116:0/749800285 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ee8003f60 0x7f5ee8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:19.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.093+0000 7f5f013a1640 1 --2- 172.21.15.116:0/749800285 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5efc153570 0x7f5ee80039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:19.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.093+0000 7f5f01ba2640 1 -- 172.21.15.116:0/749800285 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5efc153570 msgr2=0x7f5ee80039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:19.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.093+0000 7f5f01ba2640 1 --2- 172.21.15.116:0/749800285 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5efc153570 0x7f5ee80039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:19.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.093+0000 7f5f01ba2640 1 -- 172.21.15.116:0/749800285 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ee8003f60 msgr2=0x7f5ee8004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:19.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.093+0000 7f5f01ba2640 1 --2- 172.21.15.116:0/749800285 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ee8003f60 0x7f5ee8004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:19.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.093+0000 7f5f01ba2640 1 -- 172.21.15.116:0/749800285 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ee8003530 con 0x7f5ee8004d50 2024-04-25T00:17:19.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.097+0000 7f5f01ba2640 1 --2- 172.21.15.116:0/749800285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ee8004d50 0x7f5ee80051d0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f5ef4009fc0 tx=0x7f5ef401ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b807c48f2bd709f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:19.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.097+0000 7f5ef3fff640 1 -- 172.21.15.116:0/749800285 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ef40b2070 con 0x7f5ee8004d50 2024-04-25T00:17:19.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.097+0000 7f5ef3fff640 1 -- 172.21.15.116:0/749800285 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ef40a5bc0 con 0x7f5ee8004d50 2024-04-25T00:17:19.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.097+0000 7f5ef3fff640 1 -- 172.21.15.116:0/749800285 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ef4009c10 con 0x7f5ee8004d50 2024-04-25T00:17:19.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.097+0000 7f5f03605640 1 -- 172.21.15.116:0/749800285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ee8004d50 msgr2=0x7f5ee80051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:19.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.097+0000 7f5f03605640 1 --2- 172.21.15.116:0/749800285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ee8004d50 0x7f5ee80051d0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f5ef4009fc0 tx=0x7f5ef401ccc0 comp rx=0 tx=0).stop 2024-04-25T00:17:19.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.097+0000 7f5f01ba2640 1 -- 172.21.15.116:0/749800285 reap_dead start 2024-04-25T00:17:19.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.097+0000 7f5f03605640 1 -- 172.21.15.116:0/749800285 shutdown_connections 2024-04-25T00:17:19.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.097+0000 7f5f03605640 1 -- 172.21.15.116:0/749800285 >> 172.21.15.116:0/749800285 conn(0x7f5efc0c4270 msgr2=0x7f5efc0c4680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:19.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.097+0000 7f5f03605640 1 -- 172.21.15.116:0/749800285 shutdown_connections 2024-04-25T00:17:19.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.097+0000 7f5f03605640 1 -- 172.21.15.116:0/749800285 wait complete. 2024-04-25T00:17:19.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.101+0000 7f5f03605640 1 Processor -- start 2024-04-25T00:17:19.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.101+0000 7f5f03605640 1 -- start start 2024-04-25T00:17:19.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.101+0000 7f5f03605640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5ee8004d50 0x7f5efc1ead90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:19.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.101+0000 7f5f03605640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5efc146bc0 0x7f5efc0727a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:19.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.101+0000 7f5f03605640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5efc146180 0x7f5efc146600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:19.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.101+0000 7f5f03605640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5efc06c930 con 0x7f5efc146bc0 2024-04-25T00:17:19.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.101+0000 7f5f03605640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5efc154340 con 0x7f5efc146180 2024-04-25T00:17:19.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.101+0000 7f5f03605640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f5efc06cab0 con 0x7f5ee8004d50 2024-04-25T00:17:19.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.101+0000 7f5f00ba0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5efc146bc0 0x7f5efc0727a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:19.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.101+0000 7f5f00ba0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5efc146bc0 0x7f5efc0727a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42260/0 (socket says 172.21.15.116:42260) 2024-04-25T00:17:19.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.101+0000 7f5f00ba0640 1 -- 172.21.15.116:0/276821045 learned_addr learned my addr 172.21.15.116:0/276821045 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:17:19.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.101+0000 7f5f013a1640 1 --2- 172.21.15.116:0/276821045 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5ee8004d50 0x7f5efc1ead90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:19.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.101+0000 7f5f01ba2640 1 --2- 172.21.15.116:0/276821045 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5efc146180 0x7f5efc146600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:19.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.101+0000 7f5f00ba0640 1 -- 172.21.15.116:0/276821045 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5ee8004d50 msgr2=0x7f5efc1ead90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:19.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.101+0000 7f5f00ba0640 1 --2- 172.21.15.116:0/276821045 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5ee8004d50 0x7f5efc1ead90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:19.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.101+0000 7f5f00ba0640 1 -- 172.21.15.116:0/276821045 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5efc146180 msgr2=0x7f5efc146600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:19.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.101+0000 7f5f00ba0640 1 --2- 172.21.15.116:0/276821045 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5efc146180 0x7f5efc146600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:19.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.101+0000 7f5f00ba0640 1 -- 172.21.15.116:0/276821045 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5efc0c6d90 con 0x7f5efc146bc0 2024-04-25T00:17:19.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.101+0000 7f5f00ba0640 1 --2- 172.21.15.116:0/276821045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5efc146bc0 0x7f5efc0727a0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f5eec0119c0 tx=0x7f5eec011e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:19.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.101+0000 7f5ef1ffb640 1 -- 172.21.15.116:0/276821045 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5eec00faf0 con 0x7f5efc146bc0 2024-04-25T00:17:19.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.101+0000 7f5ef1ffb640 1 -- 172.21.15.116:0/276821045 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5eec00fc90 con 0x7f5efc146bc0 2024-04-25T00:17:19.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.101+0000 7f5f03605640 1 -- 172.21.15.116:0/276821045 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5efc0c7070 con 0x7f5efc146bc0 2024-04-25T00:17:19.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.101+0000 7f5ef1ffb640 1 -- 172.21.15.116:0/276821045 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5eec014440 con 0x7f5efc146bc0 2024-04-25T00:17:19.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.101+0000 7f5f03605640 1 -- 172.21.15.116:0/276821045 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5efc0abe60 con 0x7f5efc146bc0 2024-04-25T00:17:19.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.101+0000 7f5f03605640 1 -- 172.21.15.116:0/276821045 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5efc068a60 con 0x7f5efc146bc0 2024-04-25T00:17:19.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.101+0000 7f5ef1ffb640 1 -- 172.21.15.116:0/276821045 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5eec012040 con 0x7f5efc146bc0 2024-04-25T00:17:19.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.105+0000 7f5ef1ffb640 1 --2- 172.21.15.116:0/276821045 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5ee00678c0 0x7f5ee0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:19.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.105+0000 7f5ef1ffb640 1 -- 172.21.15.116:0/276821045 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 8644+0+0 (secure 0 0 0) 0x7f5eec086df0 con 0x7f5efc146bc0 2024-04-25T00:17:19.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.105+0000 7f5f013a1640 1 --2- 172.21.15.116:0/276821045 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5ee00678c0 0x7f5ee0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:19.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.105+0000 7f5f013a1640 1 --2- 172.21.15.116:0/276821045 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5ee00678c0 0x7f5ee0069d80 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f5ef8001060 tx=0x7f5ef8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:19.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.109+0000 7f5ef1ffb640 1 -- 172.21.15.116:0/276821045 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f5eec019030 con 0x7f5efc146bc0 2024-04-25T00:17:19.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.325+0000 7f5f03605640 1 -- 172.21.15.116:0/276821045 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5efc157a00 con 0x7f5efc146bc0 2024-04-25T00:17:19.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.325+0000 7f5ef1ffb640 1 -- 172.21.15.116:0/276821045 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) ==== 95+0+6509 (secure 0 0 0) 0x7f5eec056df0 con 0x7f5efc146bc0 2024-04-25T00:17:19.327 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:17:19.327 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":127,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7164,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/3336339376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3336339376},{"type":"v1","addr":"172.21.15.116:6835","nonce":3336339376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":127}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:16:48.854104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7119},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7119":{"gid":7119,"name":"a","rank":0,"incarnation":123,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6833/3645553116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3645553116},{"type":"v1","addr":"172.21.15.116:6833","nonce":3645553116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7119,"qdb_cluster":[7119]},"id":1},{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:16:38.790885+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5621},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5621":{"gid":5621,"name":"d","rank":0,"incarnation":118,"state":"up:active","state_seq":15,"addr":"172.21.15.174:6837/3081407040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3081407040},{"type":"v1","addr":"172.21.15.174:6837","nonce":3081407040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5621,"qdb_cluster":[5621]},"id":2}]} 2024-04-25T00:17:19.327 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 127 2024-04-25T00:17:19.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.329+0000 7f5f03605640 1 -- 172.21.15.116:0/276821045 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5ee00678c0 msgr2=0x7f5ee0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:19.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.329+0000 7f5f03605640 1 --2- 172.21.15.116:0/276821045 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5ee00678c0 0x7f5ee0069d80 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f5ef8001060 tx=0x7f5ef8009040 comp rx=0 tx=0).stop 2024-04-25T00:17:19.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.329+0000 7f5f03605640 1 -- 172.21.15.116:0/276821045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5efc146bc0 msgr2=0x7f5efc0727a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:19.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.329+0000 7f5f03605640 1 --2- 172.21.15.116:0/276821045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5efc146bc0 0x7f5efc0727a0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f5eec0119c0 tx=0x7f5eec011e80 comp rx=0 tx=0).stop 2024-04-25T00:17:19.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.329+0000 7f5f03605640 1 -- 172.21.15.116:0/276821045 shutdown_connections 2024-04-25T00:17:19.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.329+0000 7f5f03605640 1 --2- 172.21.15.116:0/276821045 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5ee00678c0 0x7f5ee0069d80 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:19.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.329+0000 7f5f03605640 1 --2- 172.21.15.116:0/276821045 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5efc146180 0x7f5efc146600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:19.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.329+0000 7f5f03605640 1 --2- 172.21.15.116:0/276821045 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5efc146bc0 0x7f5efc0727a0 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:19.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.329+0000 7f5f03605640 1 --2- 172.21.15.116:0/276821045 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5ee8004d50 0x7f5efc1ead90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:19.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.329+0000 7f5f03605640 1 -- 172.21.15.116:0/276821045 >> 172.21.15.116:0/276821045 conn(0x7f5efc0c4270 msgr2=0x7f5efc06fa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:19.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.329+0000 7f5f03605640 1 -- 172.21.15.116:0/276821045 shutdown_connections 2024-04-25T00:17:19.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:19.329+0000 7f5f03605640 1 -- 172.21.15.116:0/276821045 wait complete. 2024-04-25T00:17:19.346 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:17:21.092 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:17:21.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.217+0000 7f89e125c640 1 Processor -- start 2024-04-25T00:17:21.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.217+0000 7f89e125c640 1 -- start start 2024-04-25T00:17:21.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.217+0000 7f89e125c640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f89dc152f60 0x7f89dc0c6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:21.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.217+0000 7f89e125c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f89dc06fb30 0x7f89dc06c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:21.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.217+0000 7f89e125c640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89dc06ff40 con 0x7f89dc0c3c30 2024-04-25T00:17:21.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.217+0000 7f89e125c640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f89dc1595b0 con 0x7f89dc06fb30 2024-04-25T00:17:21.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.217+0000 7f89e125c640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f89dc06e8f0 con 0x7f89dc152f60 2024-04-25T00:17:21.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.217+0000 7f89d3fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f89dc0c3c30 0x7f89dc158ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:48146/0 (socket says 172.21.15.116:48146) 2024-04-25T00:17:21.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.217+0000 7f89d3fff640 1 -- 172.21.15.116:0/745928057 learned_addr learned my addr 172.21.15.116:0/745928057 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:17:21.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.217+0000 7f89db59e640 1 --2- 172.21.15.116:0/745928057 >> v2:172.21.15.116:3300/0 conn(0x7f89dc06fb30 0x7f89dc06c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:21.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.217+0000 7f89da59c640 1 -- 172.21.15.116:0/745928057 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2373314516 0 0) 0x7f89dc06ff40 con 0x7f89dc0c3c30 2024-04-25T00:17:21.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.217+0000 7f89dad9d640 1 --2- 172.21.15.116:0/745928057 >> v2:172.21.15.174:3300/0 conn(0x7f89dc152f60 0x7f89dc0c6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:21.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.217+0000 7f89da59c640 1 -- 172.21.15.116:0/745928057 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89c8003530 con 0x7f89dc0c3c30 2024-04-25T00:17:21.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.217+0000 7f89db59e640 1 -- 172.21.15.116:0/745928057 >> v2:172.21.15.174:3300/0 conn(0x7f89dc152f60 msgr2=0x7f89dc0c6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:21.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.217+0000 7f89db59e640 1 --2- 172.21.15.116:0/745928057 >> v2:172.21.15.174:3300/0 conn(0x7f89dc152f60 0x7f89dc0c6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:21.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.217+0000 7f89db59e640 1 -- 172.21.15.116:0/745928057 >> v1:172.21.15.116:6789/0 conn(0x7f89dc0c3c30 legacy=0x7f89dc158ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:21.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.217+0000 7f89db59e640 1 -- 172.21.15.116:0/745928057 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89dc06ebd0 con 0x7f89dc06fb30 2024-04-25T00:17:21.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.221+0000 7f89db59e640 1 --2- 172.21.15.116:0/745928057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89dc06fb30 0x7f89dc06c220 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f89dc067fa0 tx=0x7f89d401c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b36699c75c5c4f61 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:21.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.221+0000 7f89da59c640 1 -- 172.21.15.116:0/745928057 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d40b2070 con 0x7f89dc06fb30 2024-04-25T00:17:21.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.221+0000 7f89da59c640 1 -- 172.21.15.116:0/745928057 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89d40a5850 con 0x7f89dc06fb30 2024-04-25T00:17:21.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.221+0000 7f89da59c640 1 -- 172.21.15.116:0/745928057 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d40a5c20 con 0x7f89dc06fb30 2024-04-25T00:17:21.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.221+0000 7f89e125c640 1 -- 172.21.15.116:0/745928057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89dc06fb30 msgr2=0x7f89dc06c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:21.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.221+0000 7f89e125c640 1 --2- 172.21.15.116:0/745928057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89dc06fb30 0x7f89dc06c220 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f89dc067fa0 tx=0x7f89d401c8a0 comp rx=0 tx=0).stop 2024-04-25T00:17:21.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.221+0000 7f89e125c640 1 -- 172.21.15.116:0/745928057 shutdown_connections 2024-04-25T00:17:21.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.221+0000 7f89e125c640 1 --2- 172.21.15.116:0/745928057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89dc06fb30 0x7f89dc06c220 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:21.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.221+0000 7f89e125c640 1 --2- 172.21.15.116:0/745928057 >> v2:172.21.15.174:3300/0 conn(0x7f89dc152f60 0x7f89dc0c6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:21.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.221+0000 7f89e125c640 1 -- 172.21.15.116:0/745928057 >> 172.21.15.116:0/745928057 conn(0x7f89dc0c4270 msgr2=0x7f89dc0c4680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:21.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.221+0000 7f89e125c640 1 -- 172.21.15.116:0/745928057 shutdown_connections 2024-04-25T00:17:21.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.221+0000 7f89e125c640 1 -- 172.21.15.116:0/745928057 wait complete. 2024-04-25T00:17:21.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.221+0000 7f89e125c640 1 Processor -- start 2024-04-25T00:17:21.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.221+0000 7f89e125c640 1 -- start start 2024-04-25T00:17:21.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.225+0000 7f89e125c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89dc06fb30 0x7f89dc15e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:21.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.225+0000 7f89e125c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f89dc0c3c30 0x7f89dc1582d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:21.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.225+0000 7f89e125c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f89dc152f60 0x7f89dc0ac730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:21.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.225+0000 7f89e125c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f89dc1629f0 con 0x7f89dc06fb30 2024-04-25T00:17:21.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.225+0000 7f89e125c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f89dc074a70 con 0x7f89dc0c3c30 2024-04-25T00:17:21.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.225+0000 7f89e125c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f89dc162b70 con 0x7f89dc152f60 2024-04-25T00:17:21.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.225+0000 7f89dad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89dc06fb30 0x7f89dc15e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:21.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.225+0000 7f89dad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89dc06fb30 0x7f89dc15e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42270/0 (socket says 172.21.15.116:42270) 2024-04-25T00:17:21.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.225+0000 7f89dad9d640 1 -- 172.21.15.116:0/2513807044 learned_addr learned my addr 172.21.15.116:0/2513807044 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:17:21.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.225+0000 7f89d3fff640 1 --2- 172.21.15.116:0/2513807044 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f89dc0c3c30 0x7f89dc1582d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:21.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.225+0000 7f89dad9d640 1 -- 172.21.15.116:0/2513807044 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f89dc152f60 msgr2=0x7f89dc0ac730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:21.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.225+0000 7f89db59e640 1 --2- 172.21.15.116:0/2513807044 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f89dc152f60 0x7f89dc0ac730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:21.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.225+0000 7f89dad9d640 1 --2- 172.21.15.116:0/2513807044 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f89dc152f60 0x7f89dc0ac730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:21.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.225+0000 7f89dad9d640 1 -- 172.21.15.116:0/2513807044 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f89dc0c3c30 msgr2=0x7f89dc1582d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:21.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.225+0000 7f89dad9d640 1 --2- 172.21.15.116:0/2513807044 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f89dc0c3c30 0x7f89dc1582d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:21.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.225+0000 7f89dad9d640 1 -- 172.21.15.116:0/2513807044 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89dc15f000 con 0x7f89dc06fb30 2024-04-25T00:17:21.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.225+0000 7f89d3fff640 1 --2- 172.21.15.116:0/2513807044 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f89dc0c3c30 0x7f89dc1582d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:17:21.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.225+0000 7f89dad9d640 1 --2- 172.21.15.116:0/2513807044 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89dc06fb30 0x7f89dc15e870 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f89cc00d620 tx=0x7f89cc00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:21.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.225+0000 7f89d37fe640 1 -- 172.21.15.116:0/2513807044 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89cc00b620 con 0x7f89dc06fb30 2024-04-25T00:17:21.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.225+0000 7f89d37fe640 1 -- 172.21.15.116:0/2513807044 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89cc019070 con 0x7f89dc06fb30 2024-04-25T00:17:21.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.225+0000 7f89e125c640 1 -- 172.21.15.116:0/2513807044 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89dc1ef0a0 con 0x7f89dc06fb30 2024-04-25T00:17:21.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.225+0000 7f89d37fe640 1 -- 172.21.15.116:0/2513807044 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89cc00b9e0 con 0x7f89dc06fb30 2024-04-25T00:17:21.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.225+0000 7f89e125c640 1 -- 172.21.15.116:0/2513807044 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89dc1ee5c0 con 0x7f89dc06fb30 2024-04-25T00:17:21.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.225+0000 7f89e125c640 1 -- 172.21.15.116:0/2513807044 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89dc068a60 con 0x7f89dc06fb30 2024-04-25T00:17:21.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.225+0000 7f89d37fe640 1 -- 172.21.15.116:0/2513807044 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f89cc00bb80 con 0x7f89dc06fb30 2024-04-25T00:17:21.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.225+0000 7f89d37fe640 1 --2- 172.21.15.116:0/2513807044 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f89bc0678f0 0x7f89bc069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:21.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.225+0000 7f89d37fe640 1 -- 172.21.15.116:0/2513807044 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 8644+0+0 (secure 0 0 0) 0x7f89cc086bb0 con 0x7f89dc06fb30 2024-04-25T00:17:21.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.229+0000 7f89d3fff640 1 --2- 172.21.15.116:0/2513807044 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f89bc0678f0 0x7f89bc069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:21.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.229+0000 7f89d3fff640 1 --2- 172.21.15.116:0/2513807044 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f89bc0678f0 0x7f89bc069db0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f89dc15e5f0 tx=0x7f89c4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:21.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.233+0000 7f89d37fe640 1 -- 172.21.15.116:0/2513807044 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f89cc056bb0 con 0x7f89dc06fb30 2024-04-25T00:17:21.347 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:17:21.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.465+0000 7f7ca2093640 1 Processor -- start 2024-04-25T00:17:21.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.465+0000 7f7ca2093640 1 -- start start 2024-04-25T00:17:21.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.469+0000 7f7ca2093640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7c9c0aeb20 0x7f7c9c0c9310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:21.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.469+0000 7f7ca2093640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f7c9c150f80 0x7f7c9c0c5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:21.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.469+0000 7f7ca2093640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c9c1535d0 con 0x7f7c9c0c9850 2024-04-25T00:17:21.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.469+0000 7f7ca2093640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f7c9c0c9cb0 con 0x7f7c9c150f80 2024-04-25T00:17:21.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.469+0000 7f7ca2093640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f7c9c153850 con 0x7f7c9c0aeb20 2024-04-25T00:17:21.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.469+0000 7f7c9affd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f7c9c0c9850 0x7f7c9c150870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:48152/0 (socket says 172.21.15.116:48152) 2024-04-25T00:17:21.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.469+0000 7f7c9affd640 1 -- 172.21.15.116:0/2925438255 learned_addr learned my addr 172.21.15.116:0/2925438255 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:17:21.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.469+0000 7f7c9a7fc640 1 -- 172.21.15.116:0/2925438255 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 412562268 0 0) 0x7f7c90001120 con 0x7f7c9c0c9850 2024-04-25T00:17:21.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.469+0000 7f7c9bfff640 1 --2- 172.21.15.116:0/2925438255 >> v2:172.21.15.174:3300/0 conn(0x7f7c9c150f80 0x7f7c9c0c5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:21.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.469+0000 7f7c9a7fc640 1 -- 172.21.15.116:0/2925438255 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c84003530 con 0x7f7c9c0c9850 2024-04-25T00:17:21.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.469+0000 7f7c9b7fe640 1 --2- 172.21.15.116:0/2925438255 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7c9c0aeb20 0x7f7c9c0c9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:21.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.469+0000 7f7c9a7fc640 1 -- 172.21.15.116:0/2925438255 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3073898708 0 0) 0x7f7c84003530 con 0x7f7c9c0c9850 2024-04-25T00:17:21.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.469+0000 7f7c9a7fc640 1 -- 172.21.15.116:0/2925438255 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7c9c0aeb20 msgr2=0x7f7c9c0c9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:21.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.469+0000 7f7c9a7fc640 1 --2- 172.21.15.116:0/2925438255 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7c9c0aeb20 0x7f7c9c0c9310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:21.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.469+0000 7f7c9a7fc640 1 -- 172.21.15.116:0/2925438255 >> v2:172.21.15.174:3300/0 conn(0x7f7c9c150f80 msgr2=0x7f7c9c0c5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:21.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.469+0000 7f7c9a7fc640 1 --2- 172.21.15.116:0/2925438255 >> v2:172.21.15.174:3300/0 conn(0x7f7c9c150f80 0x7f7c9c0c5270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:21.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.469+0000 7f7c9a7fc640 1 -- 172.21.15.116:0/2925438255 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c9c153b30 con 0x7f7c9c0c9850 2024-04-25T00:17:21.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.469+0000 7f7c9a7fc640 1 -- 172.21.15.116:0/2925438255 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f7c90001850 con 0x7f7c9c0c9850 2024-04-25T00:17:21.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.469+0000 7f7c9a7fc640 1 -- 172.21.15.116:0/2925438255 >> v1:172.21.15.116:6789/0 conn(0x7f7c9c0c9850 legacy=0x7f7c9c150870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:21.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.469+0000 7f7c9a7fc640 1 --2- 172.21.15.116:0/2925438255 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7c840039c0 0x7f7c84003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:21.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.469+0000 7f7c9a7fc640 1 --2- 172.21.15.116:0/2925438255 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c84004810 0x7f7c84004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:21.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.469+0000 7f7c9a7fc640 1 --2- 172.21.15.116:0/2925438255 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7c9c0aeb20 0x7f7c840051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:21.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.469+0000 7f7c9a7fc640 1 -- 172.21.15.116:0/2925438255 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7c9c153850 con 0x7f7c84004810 2024-04-25T00:17:21.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.469+0000 7f7c9a7fc640 1 -- 172.21.15.116:0/2925438255 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7c8400b9d0 con 0x7f7c840039c0 2024-04-25T00:17:21.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.469+0000 7f7c9a7fc640 1 -- 172.21.15.116:0/2925438255 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f7c8400bb50 con 0x7f7c9c0aeb20 2024-04-25T00:17:21.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.469+0000 7f7c9affd640 1 --2- 172.21.15.116:0/2925438255 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c84004810 0x7f7c84004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:21.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.469+0000 7f7c9b7fe640 1 --2- 172.21.15.116:0/2925438255 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7c840039c0 0x7f7c84003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:21.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.469+0000 7f7c9bfff640 1 --2- 172.21.15.116:0/2925438255 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7c9c0aeb20 0x7f7c840051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:21.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.469+0000 7f7c9affd640 1 -- 172.21.15.116:0/2925438255 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7c9c0aeb20 msgr2=0x7f7c840051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:21.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.469+0000 7f7c9affd640 1 --2- 172.21.15.116:0/2925438255 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7c9c0aeb20 0x7f7c840051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:21.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.469+0000 7f7c9affd640 1 -- 172.21.15.116:0/2925438255 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7c840039c0 msgr2=0x7f7c84003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:21.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.469+0000 7f7c9affd640 1 --2- 172.21.15.116:0/2925438255 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7c840039c0 0x7f7c84003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:21.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.469+0000 7f7c9affd640 1 -- 172.21.15.116:0/2925438255 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c84003530 con 0x7f7c84004810 2024-04-25T00:17:21.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.473+0000 7f7c9affd640 1 --2- 172.21.15.116:0/2925438255 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c84004810 0x7f7c84004c90 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f7c9000c410 tx=0x7f7c9001cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ccd3bc658ea3a13 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:21.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.473+0000 7f7c9a7fc640 1 -- 172.21.15.116:0/2925438255 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c9000c0e0 con 0x7f7c84004810 2024-04-25T00:17:21.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.473+0000 7f7c9a7fc640 1 -- 172.21.15.116:0/2925438255 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c900a9020 con 0x7f7c84004810 2024-04-25T00:17:21.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.473+0000 7f7c9a7fc640 1 -- 172.21.15.116:0/2925438255 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c90005b20 con 0x7f7c84004810 2024-04-25T00:17:21.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.473+0000 7f7ca2093640 1 -- 172.21.15.116:0/2925438255 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c84004810 msgr2=0x7f7c84004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:21.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.473+0000 7f7ca2093640 1 --2- 172.21.15.116:0/2925438255 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c84004810 0x7f7c84004c90 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f7c9000c410 tx=0x7f7c9001cc70 comp rx=0 tx=0).stop 2024-04-25T00:17:21.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.473+0000 7f7c9bfff640 1 -- 172.21.15.116:0/2925438255 reap_dead start 2024-04-25T00:17:21.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.473+0000 7f7ca2093640 1 -- 172.21.15.116:0/2925438255 shutdown_connections 2024-04-25T00:17:21.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.473+0000 7f7ca2093640 1 -- 172.21.15.116:0/2925438255 >> 172.21.15.116:0/2925438255 conn(0x7f7c9c074650 msgr2=0x7f7c9c074a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:21.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.473+0000 7f7ca2093640 1 -- 172.21.15.116:0/2925438255 shutdown_connections 2024-04-25T00:17:21.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.473+0000 7f7ca2093640 1 -- 172.21.15.116:0/2925438255 wait complete. 2024-04-25T00:17:21.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.473+0000 7f7ca2093640 1 Processor -- start 2024-04-25T00:17:21.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.473+0000 7f7ca2093640 1 -- start start 2024-04-25T00:17:21.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.473+0000 7f7ca2093640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c84004810 0x7f7c9c1ee460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:21.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.473+0000 7f7ca2093640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7c9c14b090 0x7f7c9c0c2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:21.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.473+0000 7f7ca2093640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7c9c06d390 0x7f7c9c146320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:21.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.477+0000 7f7ca2093640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7c9c0af080 con 0x7f7c84004810 2024-04-25T00:17:21.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.477+0000 7f7ca2093640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7c9c0aef00 con 0x7f7c9c06d390 2024-04-25T00:17:21.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.477+0000 7f7ca2093640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f7c9c15df50 con 0x7f7c9c14b090 2024-04-25T00:17:21.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.477+0000 7f7c9b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c84004810 0x7f7c9c1ee460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:21.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.477+0000 7f7c9b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c84004810 0x7f7c9c1ee460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42284/0 (socket says 172.21.15.116:42284) 2024-04-25T00:17:21.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.477+0000 7f7c9b7fe640 1 -- 172.21.15.116:0/2450849264 learned_addr learned my addr 172.21.15.116:0/2450849264 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:17:21.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.477+0000 7f7c9bfff640 1 --2- 172.21.15.116:0/2450849264 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7c9c06d390 0x7f7c9c146320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:21.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.477+0000 7f7c9affd640 1 --2- 172.21.15.116:0/2450849264 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7c9c14b090 0x7f7c9c0c2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:21.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.477+0000 7f7c9b7fe640 1 -- 172.21.15.116:0/2450849264 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7c9c14b090 msgr2=0x7f7c9c0c2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:21.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.477+0000 7f7c9b7fe640 1 --2- 172.21.15.116:0/2450849264 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7c9c14b090 0x7f7c9c0c2690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:21.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.477+0000 7f7c9b7fe640 1 -- 172.21.15.116:0/2450849264 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7c9c06d390 msgr2=0x7f7c9c146320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:21.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.477+0000 7f7c9b7fe640 1 --2- 172.21.15.116:0/2450849264 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7c9c06d390 0x7f7c9c146320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:21.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.477+0000 7f7c9b7fe640 1 -- 172.21.15.116:0/2450849264 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c9c14d690 con 0x7f7c84004810 2024-04-25T00:17:21.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.477+0000 7f7c9b7fe640 1 --2- 172.21.15.116:0/2450849264 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c84004810 0x7f7c9c1ee460 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c0115c0 tx=0x7f7c8c011a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:21.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.477+0000 7f7c83fff640 1 -- 172.21.15.116:0/2450849264 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c8c015930 con 0x7f7c84004810 2024-04-25T00:17:21.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.477+0000 7f7c83fff640 1 -- 172.21.15.116:0/2450849264 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c8c015ad0 con 0x7f7c84004810 2024-04-25T00:17:21.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.477+0000 7f7ca2093640 1 -- 172.21.15.116:0/2450849264 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c9c14b510 con 0x7f7c84004810 2024-04-25T00:17:21.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.477+0000 7f7c83fff640 1 -- 172.21.15.116:0/2450849264 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c8c01d400 con 0x7f7c84004810 2024-04-25T00:17:21.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.477+0000 7f7ca2093640 1 -- 172.21.15.116:0/2450849264 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c9c14dbe0 con 0x7f7c84004810 2024-04-25T00:17:21.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.477+0000 7f7ca2093640 1 -- 172.21.15.116:0/2450849264 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c9c068a60 con 0x7f7c84004810 2024-04-25T00:17:21.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.477+0000 7f7c83fff640 1 -- 172.21.15.116:0/2450849264 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7c8c01d5a0 con 0x7f7c84004810 2024-04-25T00:17:21.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.477+0000 7f7c83fff640 1 --2- 172.21.15.116:0/2450849264 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7c880678c0 0x7f7c88069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:21.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.477+0000 7f7c83fff640 1 -- 172.21.15.116:0/2450849264 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 8644+0+0 (secure 0 0 0) 0x7f7c8c086f80 con 0x7f7c84004810 2024-04-25T00:17:21.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.477+0000 7f7c9affd640 1 --2- 172.21.15.116:0/2450849264 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7c880678c0 0x7f7c88069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:21.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.481+0000 7f7c9affd640 1 --2- 172.21.15.116:0/2450849264 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7c880678c0 0x7f7c88069d80 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f7c9000c3e0 tx=0x7f7c90001c20 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:21.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.481+0000 7f7c83fff640 1 -- 172.21.15.116:0/2450849264 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f7c8c019030 con 0x7f7c84004810 2024-04-25T00:17:21.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.517+0000 7f89e125c640 1 -- 172.21.15.116:0/2513807044 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f89dc0c62c0 con 0x7f89dc06fb30 2024-04-25T00:17:21.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.517+0000 7f89d37fe640 1 -- 172.21.15.116:0/2513807044 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) ==== 95+0+6509 (secure 0 0 0) 0x7f89cc0564b0 con 0x7f89dc06fb30 2024-04-25T00:17:21.517 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:17:21.517 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":127,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7164,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/3336339376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3336339376},{"type":"v1","addr":"172.21.15.116:6835","nonce":3336339376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":127}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:16:48.854104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7119},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7119":{"gid":7119,"name":"a","rank":0,"incarnation":123,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6833/3645553116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3645553116},{"type":"v1","addr":"172.21.15.116:6833","nonce":3645553116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7119,"qdb_cluster":[7119]},"id":1},{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:16:38.790885+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5621},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5621":{"gid":5621,"name":"d","rank":0,"incarnation":118,"state":"up:active","state_seq":15,"addr":"172.21.15.174:6837/3081407040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3081407040},{"type":"v1","addr":"172.21.15.174:6837","nonce":3081407040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5621,"qdb_cluster":[5621]},"id":2}]} 2024-04-25T00:17:21.518 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 127 2024-04-25T00:17:21.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.517+0000 7f89e125c640 1 -- 172.21.15.116:0/2513807044 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f89bc0678f0 msgr2=0x7f89bc069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:21.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.517+0000 7f89e125c640 1 --2- 172.21.15.116:0/2513807044 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f89bc0678f0 0x7f89bc069db0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f89dc15e5f0 tx=0x7f89c4009040 comp rx=0 tx=0).stop 2024-04-25T00:17:21.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.517+0000 7f89e125c640 1 -- 172.21.15.116:0/2513807044 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89dc06fb30 msgr2=0x7f89dc15e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:21.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.517+0000 7f89e125c640 1 --2- 172.21.15.116:0/2513807044 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89dc06fb30 0x7f89dc15e870 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f89cc00d620 tx=0x7f89cc00dae0 comp rx=0 tx=0).stop 2024-04-25T00:17:21.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.517+0000 7f89e125c640 1 -- 172.21.15.116:0/2513807044 shutdown_connections 2024-04-25T00:17:21.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.517+0000 7f89e125c640 1 --2- 172.21.15.116:0/2513807044 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f89bc0678f0 0x7f89bc069db0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:21.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.517+0000 7f89e125c640 1 --2- 172.21.15.116:0/2513807044 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f89dc152f60 0x7f89dc0ac730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:21.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.517+0000 7f89e125c640 1 --2- 172.21.15.116:0/2513807044 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f89dc0c3c30 0x7f89dc1582d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:21.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.517+0000 7f89e125c640 1 --2- 172.21.15.116:0/2513807044 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f89dc06fb30 0x7f89dc15e870 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:21.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.521+0000 7f89e125c640 1 -- 172.21.15.116:0/2513807044 >> 172.21.15.116:0/2513807044 conn(0x7f89dc0c4270 msgr2=0x7f89dc15f850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:21.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.521+0000 7f89e125c640 1 -- 172.21.15.116:0/2513807044 shutdown_connections 2024-04-25T00:17:21.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.521+0000 7f89e125c640 1 -- 172.21.15.116:0/2513807044 wait complete. 2024-04-25T00:17:21.537 INFO:tasks.mds_thrash.fs.[a]:kill mds.a (rank=0) 2024-04-25T00:17:21.537 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-25T00:17:21.538 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:17:21.549 INFO:tasks.ceph.mds.a:Stopped 2024-04-25T00:17:21.550 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:17:21.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.701+0000 7f7ca2093640 1 -- 172.21.15.116:0/2450849264 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7c9c0a92c0 con 0x7f7c84004810 2024-04-25T00:17:21.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.701+0000 7f7c83fff640 1 -- 172.21.15.116:0/2450849264 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) ==== 95+0+6509 (secure 0 0 0) 0x7f7c8c056f80 con 0x7f7c84004810 2024-04-25T00:17:21.703 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:17:21.703 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":127,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7164,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/3336339376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3336339376},{"type":"v1","addr":"172.21.15.116:6835","nonce":3336339376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":127}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:16:48.854104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7119},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7119":{"gid":7119,"name":"a","rank":0,"incarnation":123,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6833/3645553116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3645553116},{"type":"v1","addr":"172.21.15.116:6833","nonce":3645553116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7119,"qdb_cluster":[7119]},"id":1},{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:16:38.790885+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5621},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5621":{"gid":5621,"name":"d","rank":0,"incarnation":118,"state":"up:active","state_seq":15,"addr":"172.21.15.174:6837/3081407040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3081407040},{"type":"v1","addr":"172.21.15.174:6837","nonce":3081407040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5621,"qdb_cluster":[5621]},"id":2}]} 2024-04-25T00:17:21.703 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 127 2024-04-25T00:17:21.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.701+0000 7f7f2965b640 1 Processor -- start 2024-04-25T00:17:21.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.701+0000 7f7f2965b640 1 -- start start 2024-04-25T00:17:21.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.705+0000 7f7f2965b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7f24152f50 0x7f7f240aa100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:21.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.705+0000 7f7f2965b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7f240aa640 0x7f7f2415cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:21.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.705+0000 7f7f2965b640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f24073260 con 0x7f7f2414fc20 2024-04-25T00:17:21.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.705+0000 7f7f2965b640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f7f240a9620 con 0x7f7f240aa640 2024-04-25T00:17:21.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.705+0000 7f7f2965b640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f7f240a9900 con 0x7f7f24152f50 2024-04-25T00:17:21.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.705+0000 7f7f22ffd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f7f2414fc20 0x7f7f2406f6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:48162/0 (socket says 172.21.15.116:48162) 2024-04-25T00:17:21.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.705+0000 7f7f22ffd640 1 -- 172.21.15.116:0/3681439440 learned_addr learned my addr 172.21.15.116:0/3681439440 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:17:21.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.705+0000 7f7f237fe640 1 --2- 172.21.15.116:0/3681439440 >> v2:172.21.15.116:3300/0 conn(0x7f7f240aa640 0x7f7f2415cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:21.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.705+0000 7f7f227fc640 1 --2- 172.21.15.116:0/3681439440 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7f24152f50 0x7f7f240aa100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:21.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.705+0000 7f7f237fe640 1 -- 172.21.15.116:0/3681439440 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7f24152f50 msgr2=0x7f7f240aa100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:21.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.705+0000 7f7f237fe640 1 --2- 172.21.15.116:0/3681439440 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7f24152f50 0x7f7f240aa100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:21.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.705+0000 7f7f237fe640 1 -- 172.21.15.116:0/3681439440 >> v1:172.21.15.116:6789/0 conn(0x7f7f2414fc20 legacy=0x7f7f2406f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:21.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.705+0000 7f7f237fe640 1 -- 172.21.15.116:0/3681439440 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f240a9be0 con 0x7f7f240aa640 2024-04-25T00:17:21.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.705+0000 7f7ca2093640 1 -- 172.21.15.116:0/2450849264 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7c880678c0 msgr2=0x7f7c88069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:21.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.705+0000 7f7ca2093640 1 --2- 172.21.15.116:0/2450849264 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7c880678c0 0x7f7c88069d80 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f7c9000c3e0 tx=0x7f7c90001c20 comp rx=0 tx=0).stop 2024-04-25T00:17:21.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.705+0000 7f7ca2093640 1 -- 172.21.15.116:0/2450849264 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c84004810 msgr2=0x7f7c9c1ee460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:21.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.705+0000 7f7ca2093640 1 --2- 172.21.15.116:0/2450849264 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c84004810 0x7f7c9c1ee460 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f7c8c0115c0 tx=0x7f7c8c011a80 comp rx=0 tx=0).stop 2024-04-25T00:17:21.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.705+0000 7f7ca2093640 1 -- 172.21.15.116:0/2450849264 shutdown_connections 2024-04-25T00:17:21.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.705+0000 7f7ca2093640 1 --2- 172.21.15.116:0/2450849264 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7c880678c0 0x7f7c88069d80 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:21.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.705+0000 7f7ca2093640 1 --2- 172.21.15.116:0/2450849264 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7c9c06d390 0x7f7c9c146320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:21.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.705+0000 7f7ca2093640 1 --2- 172.21.15.116:0/2450849264 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7c9c14b090 0x7f7c9c0c2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:21.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.705+0000 7f7ca2093640 1 --2- 172.21.15.116:0/2450849264 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c84004810 0x7f7c9c1ee460 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:21.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.705+0000 7f7ca2093640 1 -- 172.21.15.116:0/2450849264 >> 172.21.15.116:0/2450849264 conn(0x7f7c9c074650 msgr2=0x7f7c9c1517d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:21.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.705+0000 7f7ca2093640 1 -- 172.21.15.116:0/2450849264 shutdown_connections 2024-04-25T00:17:21.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.705+0000 7f7ca2093640 1 -- 172.21.15.116:0/2450849264 wait complete. 2024-04-25T00:17:21.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.705+0000 7f7f237fe640 1 --2- 172.21.15.116:0/3681439440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f240aa640 0x7f7f2415cdc0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f7f1c009d90 tx=0x7f7f1c01c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2fc69d68722b400 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:21.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.705+0000 7f7f21ffb640 1 -- 172.21.15.116:0/3681439440 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f1c0b2070 con 0x7f7f240aa640 2024-04-25T00:17:21.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.705+0000 7f7f21ffb640 1 -- 172.21.15.116:0/3681439440 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f1c0a5820 con 0x7f7f240aa640 2024-04-25T00:17:21.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.705+0000 7f7f21ffb640 1 -- 172.21.15.116:0/3681439440 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f1c0a5bf0 con 0x7f7f240aa640 2024-04-25T00:17:21.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.705+0000 7f7f2965b640 1 -- 172.21.15.116:0/3681439440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f240aa640 msgr2=0x7f7f2415cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:21.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.705+0000 7f7f2965b640 1 --2- 172.21.15.116:0/3681439440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f240aa640 0x7f7f2415cdc0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f7f1c009d90 tx=0x7f7f1c01c850 comp rx=0 tx=0).stop 2024-04-25T00:17:21.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.705+0000 7f7f2965b640 1 -- 172.21.15.116:0/3681439440 shutdown_connections 2024-04-25T00:17:21.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.705+0000 7f7f2965b640 1 --2- 172.21.15.116:0/3681439440 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f240aa640 0x7f7f2415cdc0 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:21.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.705+0000 7f7f2965b640 1 --2- 172.21.15.116:0/3681439440 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7f24152f50 0x7f7f240aa100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:21.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.705+0000 7f7f2965b640 1 -- 172.21.15.116:0/3681439440 >> 172.21.15.116:0/3681439440 conn(0x7f7f240c4c10 msgr2=0x7f7f240c5020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:21.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.705+0000 7f7f2965b640 1 -- 172.21.15.116:0/3681439440 shutdown_connections 2024-04-25T00:17:21.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.705+0000 7f7f2965b640 1 -- 172.21.15.116:0/3681439440 wait complete. 2024-04-25T00:17:21.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.709+0000 7f7f2965b640 1 Processor -- start 2024-04-25T00:17:21.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.709+0000 7f7f2965b640 1 -- start start 2024-04-25T00:17:21.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.709+0000 7f7f2965b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7f240aa640 0x7f7f241eba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:21.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.709+0000 7f7f2965b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7f2414fc20 0x7f7f241ec4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:21.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.709+0000 7f7f2965b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f24152f50 0x7f7f2414a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:21.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.709+0000 7f7f2965b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7f2406ff50 con 0x7f7f24152f50 2024-04-25T00:17:21.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.709+0000 7f7f2965b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7f2406fdd0 con 0x7f7f2414fc20 2024-04-25T00:17:21.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.709+0000 7f7f2965b640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f7f240aaad0 con 0x7f7f240aa640 2024-04-25T00:17:21.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.709+0000 7f7f237fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f24152f50 0x7f7f2414a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:21.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.709+0000 7f7f237fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f24152f50 0x7f7f2414a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42306/0 (socket says 172.21.15.116:42306) 2024-04-25T00:17:21.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.709+0000 7f7f237fe640 1 -- 172.21.15.116:0/2438336322 learned_addr learned my addr 172.21.15.116:0/2438336322 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:17:21.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.709+0000 7f7f22ffd640 1 --2- 172.21.15.116:0/2438336322 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7f240aa640 0x7f7f241eba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:21.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.709+0000 7f7f227fc640 1 --2- 172.21.15.116:0/2438336322 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7f2414fc20 0x7f7f241ec4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:21.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.709+0000 7f7f237fe640 1 -- 172.21.15.116:0/2438336322 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7f240aa640 msgr2=0x7f7f241eba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:21.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.709+0000 7f7f237fe640 1 --2- 172.21.15.116:0/2438336322 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7f240aa640 0x7f7f241eba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:21.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.709+0000 7f7f237fe640 1 -- 172.21.15.116:0/2438336322 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7f2414fc20 msgr2=0x7f7f241ec4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:21.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.709+0000 7f7f237fe640 1 --2- 172.21.15.116:0/2438336322 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7f2414fc20 0x7f7f241ec4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:21.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.709+0000 7f7f237fe640 1 -- 172.21.15.116:0/2438336322 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f241ebfc0 con 0x7f7f24152f50 2024-04-25T00:17:21.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.709+0000 7f7f227fc640 1 --2- 172.21.15.116:0/2438336322 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7f2414fc20 0x7f7f241ec4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:17:21.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.709+0000 7f7f237fe640 1 --2- 172.21.15.116:0/2438336322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f24152f50 0x7f7f2414a270 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f7f1c0b0040 tx=0x7f7f1c0195f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:21.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.709+0000 7f7f0b7fe640 1 -- 172.21.15.116:0/2438336322 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f1c0b2050 con 0x7f7f24152f50 2024-04-25T00:17:21.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.709+0000 7f7f0b7fe640 1 -- 172.21.15.116:0/2438336322 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f1c0be070 con 0x7f7f24152f50 2024-04-25T00:17:21.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.709+0000 7f7f2965b640 1 -- 172.21.15.116:0/2438336322 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f241eca20 con 0x7f7f24152f50 2024-04-25T00:17:21.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.709+0000 7f7f0b7fe640 1 -- 172.21.15.116:0/2438336322 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f1c0ada70 con 0x7f7f24152f50 2024-04-25T00:17:21.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.709+0000 7f7f2965b640 1 -- 172.21.15.116:0/2438336322 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f2414c3c0 con 0x7f7f24152f50 2024-04-25T00:17:21.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.709+0000 7f7f2965b640 1 -- 172.21.15.116:0/2438336322 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f24068a60 con 0x7f7f24152f50 2024-04-25T00:17:21.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.713+0000 7f7f0b7fe640 1 -- 172.21.15.116:0/2438336322 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7f1c0adc10 con 0x7f7f24152f50 2024-04-25T00:17:21.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.713+0000 7f7f0b7fe640 1 --2- 172.21.15.116:0/2438336322 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7f100678c0 0x7f7f10069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:21.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.713+0000 7f7f0b7fe640 1 -- 172.21.15.116:0/2438336322 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 8644+0+0 (secure 0 0 0) 0x7f7f1c1223e0 con 0x7f7f24152f50 2024-04-25T00:17:21.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.713+0000 7f7f22ffd640 1 --2- 172.21.15.116:0/2438336322 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7f100678c0 0x7f7f10069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:21.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.713+0000 7f7f22ffd640 1 --2- 172.21.15.116:0/2438336322 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7f100678c0 0x7f7f10069d80 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f7f14000f70 tx=0x7f7f14009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:21.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.721+0000 7f7f0b7fe640 1 -- 172.21.15.116:0/2438336322 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f7f1c016560 con 0x7f7f24152f50 2024-04-25T00:17:21.723 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:17:21.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.905+0000 7f7f2965b640 1 -- 172.21.15.116:0/2438336322 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7f240c7ae0 con 0x7f7f24152f50 2024-04-25T00:17:21.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.905+0000 7f7f0b7fe640 1 -- 172.21.15.116:0/2438336322 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) ==== 95+0+6509 (secure 0 0 0) 0x7f7f1c0f2460 con 0x7f7f24152f50 2024-04-25T00:17:21.907 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:17:21.908 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":127,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7164,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/3336339376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3336339376},{"type":"v1","addr":"172.21.15.116:6835","nonce":3336339376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":127}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:16:48.854104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7119},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7119":{"gid":7119,"name":"a","rank":0,"incarnation":123,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6833/3645553116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3645553116},{"type":"v1","addr":"172.21.15.116:6833","nonce":3645553116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7119,"qdb_cluster":[7119]},"id":1},{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:16:38.790885+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5621},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5621":{"gid":5621,"name":"d","rank":0,"incarnation":118,"state":"up:active","state_seq":15,"addr":"172.21.15.174:6837/3081407040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3081407040},{"type":"v1","addr":"172.21.15.174:6837","nonce":3081407040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5621,"qdb_cluster":[5621]},"id":2}]} 2024-04-25T00:17:21.908 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 127 2024-04-25T00:17:21.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.909+0000 7f7f2965b640 1 -- 172.21.15.116:0/2438336322 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7f100678c0 msgr2=0x7f7f10069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:21.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.909+0000 7f7f2965b640 1 --2- 172.21.15.116:0/2438336322 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7f100678c0 0x7f7f10069d80 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f7f14000f70 tx=0x7f7f14009040 comp rx=0 tx=0).stop 2024-04-25T00:17:21.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.909+0000 7f7f2965b640 1 -- 172.21.15.116:0/2438336322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f24152f50 msgr2=0x7f7f2414a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:21.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.909+0000 7f7f2965b640 1 --2- 172.21.15.116:0/2438336322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f24152f50 0x7f7f2414a270 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f7f1c0b0040 tx=0x7f7f1c0195f0 comp rx=0 tx=0).stop 2024-04-25T00:17:21.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.909+0000 7f7f2965b640 1 -- 172.21.15.116:0/2438336322 shutdown_connections 2024-04-25T00:17:21.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.909+0000 7f7f2965b640 1 --2- 172.21.15.116:0/2438336322 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7f100678c0 0x7f7f10069d80 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:21.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.909+0000 7f7f2965b640 1 --2- 172.21.15.116:0/2438336322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f24152f50 0x7f7f2414a270 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:21.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.909+0000 7f7f2965b640 1 --2- 172.21.15.116:0/2438336322 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7f2414fc20 0x7f7f241ec4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:21.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.909+0000 7f7f2965b640 1 --2- 172.21.15.116:0/2438336322 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7f240aa640 0x7f7f241eba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:21.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.909+0000 7f7f2965b640 1 -- 172.21.15.116:0/2438336322 >> 172.21.15.116:0/2438336322 conn(0x7f7f240c4c10 msgr2=0x7f7f240c3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:21.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.909+0000 7f7f2965b640 1 -- 172.21.15.116:0/2438336322 shutdown_connections 2024-04-25T00:17:21.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:21.909+0000 7f7f2965b640 1 -- 172.21.15.116:0/2438336322 wait complete. 2024-04-25T00:17:21.926 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:17:23.724 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:17:23.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.853+0000 7f5bbb00d640 1 Processor -- start 2024-04-25T00:17:23.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.853+0000 7f5bbb00d640 1 -- start start 2024-04-25T00:17:23.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.853+0000 7f5bbb00d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5bb4148890 0x7f5bb4072590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:23.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.853+0000 7f5bbb00d640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f5bb4070dd0 0x7f5bb4071270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:23.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.853+0000 7f5bbb00d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5bb406e890 0x7f5bb40717b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:23.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.853+0000 7f5bbb00d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f5bb4152c10 con 0x7f5bb4148890 2024-04-25T00:17:23.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.853+0000 7f5bbb00d640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f5bb4071cf0 con 0x7f5bb4070dd0 2024-04-25T00:17:23.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.853+0000 7f5bbb00d640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f5bb4151e50 con 0x7f5bb406e890 2024-04-25T00:17:23.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.853+0000 7f5bb8da9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5bb4148890 0x7f5bb4072590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:23.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.853+0000 7f5bb8da9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5bb4148890 0x7f5bb4072590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42320/0 (socket says 172.21.15.116:42320) 2024-04-25T00:17:23.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.853+0000 7f5bb8da9640 1 -- 172.21.15.116:0/3150881257 learned_addr learned my addr 172.21.15.116:0/3150881257 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:17:23.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.853+0000 7f5bb95aa640 1 --2- 172.21.15.116:0/3150881257 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5bb406e890 0x7f5bb40717b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:23.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.853+0000 7f5bb3fff640 1 --2- 172.21.15.116:0/3150881257 >> v2:172.21.15.174:3300/0 conn(0x7f5bb4070dd0 0x7f5bb4071270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:23.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.853+0000 7f5bb8da9640 1 -- 172.21.15.116:0/3150881257 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5bb406e890 msgr2=0x7f5bb40717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:23.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.853+0000 7f5bb8da9640 1 --2- 172.21.15.116:0/3150881257 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5bb406e890 0x7f5bb40717b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:23.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.853+0000 7f5bb8da9640 1 -- 172.21.15.116:0/3150881257 >> v2:172.21.15.174:3300/0 conn(0x7f5bb4070dd0 msgr2=0x7f5bb4071270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:23.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.853+0000 7f5bb8da9640 1 --2- 172.21.15.116:0/3150881257 >> v2:172.21.15.174:3300/0 conn(0x7f5bb4070dd0 0x7f5bb4071270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:23.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.853+0000 7f5bb8da9640 1 -- 172.21.15.116:0/3150881257 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bb4152130 con 0x7f5bb4148890 2024-04-25T00:17:23.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.853+0000 7f5bb3fff640 1 --2- 172.21.15.116:0/3150881257 >> v2:172.21.15.174:3300/0 conn(0x7f5bb4070dd0 0x7f5bb4071270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:17:23.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.853+0000 7f5bb8da9640 1 --2- 172.21.15.116:0/3150881257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5bb4148890 0x7f5bb4072590 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f5bb40711b0 tx=0x7f5ba801e9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8617dc70f8ef106 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:23.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.857+0000 7f5bb37fe640 1 -- 172.21.15.116:0/3150881257 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ba80ad070 con 0x7f5bb4148890 2024-04-25T00:17:23.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.857+0000 7f5bb37fe640 1 -- 172.21.15.116:0/3150881257 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ba8004230 con 0x7f5bb4148890 2024-04-25T00:17:23.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.857+0000 7f5bb37fe640 1 -- 172.21.15.116:0/3150881257 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ba80a7660 con 0x7f5bb4148890 2024-04-25T00:17:23.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.857+0000 7f5bbb00d640 1 -- 172.21.15.116:0/3150881257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5bb4148890 msgr2=0x7f5bb4072590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:23.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.857+0000 7f5bbb00d640 1 --2- 172.21.15.116:0/3150881257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5bb4148890 0x7f5bb4072590 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f5bb40711b0 tx=0x7f5ba801e9d0 comp rx=0 tx=0).stop 2024-04-25T00:17:23.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.857+0000 7f5bbb00d640 1 -- 172.21.15.116:0/3150881257 shutdown_connections 2024-04-25T00:17:23.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.857+0000 7f5bbb00d640 1 --2- 172.21.15.116:0/3150881257 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5bb406e890 0x7f5bb40717b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:23.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.857+0000 7f5bbb00d640 1 --2- 172.21.15.116:0/3150881257 >> v2:172.21.15.174:3300/0 conn(0x7f5bb4070dd0 0x7f5bb4071270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:23.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.857+0000 7f5bbb00d640 1 --2- 172.21.15.116:0/3150881257 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5bb4148890 0x7f5bb4072590 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:23.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.857+0000 7f5bbb00d640 1 -- 172.21.15.116:0/3150881257 >> 172.21.15.116:0/3150881257 conn(0x7f5bb40c3b00 msgr2=0x7f5bb40c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:23.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.857+0000 7f5bbb00d640 1 -- 172.21.15.116:0/3150881257 shutdown_connections 2024-04-25T00:17:23.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.857+0000 7f5bbb00d640 1 -- 172.21.15.116:0/3150881257 wait complete. 2024-04-25T00:17:23.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.857+0000 7f5bbb00d640 1 Processor -- start 2024-04-25T00:17:23.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.857+0000 7f5bbb00d640 1 -- start start 2024-04-25T00:17:23.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.857+0000 7f5bbb00d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5bb406e890 0x7f5bb41423a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:23.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.857+0000 7f5bbb00d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5bb4070dd0 0x7f5bb41f2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:23.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.857+0000 7f5bbb00d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5bb4148890 0x7f5bb41f2780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:23.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.857+0000 7f5bbb00d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb40ab710 con 0x7f5bb4070dd0 2024-04-25T00:17:23.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.857+0000 7f5bbb00d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5bb40ab590 con 0x7f5bb406e890 2024-04-25T00:17:23.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.857+0000 7f5bbb00d640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f5bb40ab890 con 0x7f5bb4148890 2024-04-25T00:17:23.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.857+0000 7f5bb3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5bb4070dd0 0x7f5bb41f2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:23.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.857+0000 7f5bb3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5bb4070dd0 0x7f5bb41f2240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42330/0 (socket says 172.21.15.116:42330) 2024-04-25T00:17:23.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.857+0000 7f5bb3fff640 1 -- 172.21.15.116:0/128197175 learned_addr learned my addr 172.21.15.116:0/128197175 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:17:23.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.857+0000 7f5bb8da9640 1 --2- 172.21.15.116:0/128197175 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5bb406e890 0x7f5bb41423a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:23.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.857+0000 7f5bb95aa640 1 --2- 172.21.15.116:0/128197175 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5bb4148890 0x7f5bb41f2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:23.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.857+0000 7f5bb3fff640 1 -- 172.21.15.116:0/128197175 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5bb4148890 msgr2=0x7f5bb41f2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:23.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.857+0000 7f5bb3fff640 1 --2- 172.21.15.116:0/128197175 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5bb4148890 0x7f5bb41f2780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:23.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.857+0000 7f5bb3fff640 1 -- 172.21.15.116:0/128197175 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5bb406e890 msgr2=0x7f5bb41423a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:23.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.857+0000 7f5bb3fff640 1 --2- 172.21.15.116:0/128197175 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5bb406e890 0x7f5bb41423a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:23.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.857+0000 7f5bb3fff640 1 -- 172.21.15.116:0/128197175 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bb40c0510 con 0x7f5bb4070dd0 2024-04-25T00:17:23.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.857+0000 7f5bb95aa640 1 --2- 172.21.15.116:0/128197175 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5bb4148890 0x7f5bb41f2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:17:23.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.857+0000 7f5bb3fff640 1 --2- 172.21.15.116:0/128197175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5bb4070dd0 0x7f5bb41f2240 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f5ba000c820 tx=0x7f5ba000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:23.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.857+0000 7f5bb17fa640 1 -- 172.21.15.116:0/128197175 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ba0015070 con 0x7f5bb4070dd0 2024-04-25T00:17:23.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.857+0000 7f5bb17fa640 1 -- 172.21.15.116:0/128197175 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ba0007a30 con 0x7f5bb4070dd0 2024-04-25T00:17:23.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.857+0000 7f5bbb00d640 1 -- 172.21.15.116:0/128197175 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bb40c07f0 con 0x7f5bb4070dd0 2024-04-25T00:17:23.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.857+0000 7f5bb17fa640 1 -- 172.21.15.116:0/128197175 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ba0007e00 con 0x7f5bb4070dd0 2024-04-25T00:17:23.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.857+0000 7f5bbb00d640 1 -- 172.21.15.116:0/128197175 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bb41428e0 con 0x7f5bb4070dd0 2024-04-25T00:17:23.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.861+0000 7f5bbb00d640 1 -- 172.21.15.116:0/128197175 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5bb41ef6a0 con 0x7f5bb4070dd0 2024-04-25T00:17:23.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.861+0000 7f5bb17fa640 1 -- 172.21.15.116:0/128197175 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5ba0018070 con 0x7f5bb4070dd0 2024-04-25T00:17:23.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.861+0000 7f5bb17fa640 1 --2- 172.21.15.116:0/128197175 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5b980678c0 0x7f5b98069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:23.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.861+0000 7f5bb17fa640 1 -- 172.21.15.116:0/128197175 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 8644+0+0 (secure 0 0 0) 0x7f5ba00867b0 con 0x7f5bb4070dd0 2024-04-25T00:17:23.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.861+0000 7f5bb8da9640 1 --2- 172.21.15.116:0/128197175 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5b980678c0 0x7f5b98069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:23.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.861+0000 7f5bb8da9640 1 --2- 172.21.15.116:0/128197175 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5b980678c0 0x7f5b98069d80 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f5ba80b1000 tx=0x7f5ba80a9070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:23.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:23.865+0000 7f5bb17fa640 1 -- 172.21.15.116:0/128197175 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f5ba00567b0 con 0x7f5bb4070dd0 2024-04-25T00:17:23.927 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:17:24.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.045+0000 7fa7b492b640 1 Processor -- start 2024-04-25T00:17:24.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.045+0000 7fa7b492b640 1 -- start start 2024-04-25T00:17:24.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.045+0000 7fa7b492b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa7b0150bf0 0x7fa7b006c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:24.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.045+0000 7fa7b492b640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7b0153aa0 con 0x7fa7b0153570 2024-04-25T00:17:24.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.045+0000 7fa7b492b640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7b0153d80 con 0x7fa7b006c760 2024-04-25T00:17:24.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.045+0000 7fa7b492b640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa7b006e630 con 0x7fa7b0150bf0 2024-04-25T00:17:24.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.045+0000 7fa7add9b640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fa7b0153570 0x7fa7b006eac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:48172/0 (socket says 172.21.15.116:48172) 2024-04-25T00:17:24.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.045+0000 7fa7add9b640 1 -- 172.21.15.116:0/584730444 learned_addr learned my addr 172.21.15.116:0/584730444 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:17:24.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.045+0000 7fa7ae59c640 1 --2- 172.21.15.116:0/584730444 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa7b0150bf0 0x7fa7b006c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:24.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.045+0000 7fa7ad59a640 1 -- 172.21.15.116:0/584730444 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3048693346 0 0) 0x7fa7b0153aa0 con 0x7fa7b0153570 2024-04-25T00:17:24.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.045+0000 7fa7ad59a640 1 -- 172.21.15.116:0/584730444 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa798003530 con 0x7fa7b0153570 2024-04-25T00:17:24.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.045+0000 7fa7ad59a640 1 -- 172.21.15.116:0/584730444 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2699184257 0 0) 0x7fa798003530 con 0x7fa7b0153570 2024-04-25T00:17:24.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.045+0000 7fa7ad59a640 1 -- 172.21.15.116:0/584730444 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa7b0150bf0 msgr2=0x7fa7b006c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:24.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.045+0000 7fa7ad59a640 1 --2- 172.21.15.116:0/584730444 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa7b0150bf0 0x7fa7b006c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:24.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.045+0000 7fa7ad59a640 1 -- 172.21.15.116:0/584730444 >> v1:172.21.15.174:6789/0 conn(0x7fa7b006c760 legacy=0x7fa7b006f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:24.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.045+0000 7fa7ad59a640 1 -- 172.21.15.116:0/584730444 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7b0154060 con 0x7fa7b0153570 2024-04-25T00:17:24.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.045+0000 7fa7ad59a640 1 -- 172.21.15.116:0/584730444 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7fa7a0001680 con 0x7fa7b0153570 2024-04-25T00:17:24.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.045+0000 7fa7ad59a640 1 -- 172.21.15.116:0/584730444 >> v1:172.21.15.116:6789/0 conn(0x7fa7b0153570 legacy=0x7fa7b006eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:24.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.045+0000 7fa7ad59a640 1 --2- 172.21.15.116:0/584730444 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7980039f0 0x7fa798003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:24.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.045+0000 7fa7ad59a640 1 --2- 172.21.15.116:0/584730444 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa798004810 0x7fa798004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:24.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.045+0000 7fa7ad59a640 1 --2- 172.21.15.116:0/584730444 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa7b0150bf0 0x7fa798005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:24.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.045+0000 7fa7ad59a640 1 -- 172.21.15.116:0/584730444 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa7b006e630 con 0x7fa798004810 2024-04-25T00:17:24.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.045+0000 7fa7ad59a640 1 -- 172.21.15.116:0/584730444 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa79800bb00 con 0x7fa7980039f0 2024-04-25T00:17:24.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.045+0000 7fa7ad59a640 1 -- 172.21.15.116:0/584730444 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa79800bd00 con 0x7fa7b0150bf0 2024-04-25T00:17:24.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.045+0000 7fa7add9b640 1 --2- 172.21.15.116:0/584730444 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa798004810 0x7fa798004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:24.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.045+0000 7fa7ae59c640 1 --2- 172.21.15.116:0/584730444 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7980039f0 0x7fa798003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:24.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.045+0000 7fa7aed9d640 1 --2- 172.21.15.116:0/584730444 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa7b0150bf0 0x7fa798005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:24.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.045+0000 7fa7add9b640 1 -- 172.21.15.116:0/584730444 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa7b0150bf0 msgr2=0x7fa798005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:24.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.045+0000 7fa7add9b640 1 --2- 172.21.15.116:0/584730444 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa7b0150bf0 0x7fa798005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:24.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.045+0000 7fa7add9b640 1 -- 172.21.15.116:0/584730444 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7980039f0 msgr2=0x7fa798003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:24.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.045+0000 7fa7add9b640 1 --2- 172.21.15.116:0/584730444 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7980039f0 0x7fa798003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:24.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.045+0000 7fa7add9b640 1 -- 172.21.15.116:0/584730444 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa798003530 con 0x7fa798004810 2024-04-25T00:17:24.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.045+0000 7fa7aed9d640 1 --2- 172.21.15.116:0/584730444 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa7b0150bf0 0x7fa798005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:17:24.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.045+0000 7fa7ae59c640 1 --2- 172.21.15.116:0/584730444 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7980039f0 0x7fa798003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:17:24.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.049+0000 7fa7add9b640 1 --2- 172.21.15.116:0/584730444 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa798004810 0x7fa798004c90 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7fa7b01539d0 tx=0x7fa7a001cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a3aa2b2563624b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:24.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.049+0000 7fa7ad59a640 1 -- 172.21.15.116:0/584730444 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7a0007cf0 con 0x7fa798004810 2024-04-25T00:17:24.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.049+0000 7fa7ad59a640 1 -- 172.21.15.116:0/584730444 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7a00a9070 con 0x7fa798004810 2024-04-25T00:17:24.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.049+0000 7fa7ad59a640 1 -- 172.21.15.116:0/584730444 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7a0005d40 con 0x7fa798004810 2024-04-25T00:17:24.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.049+0000 7fa7b492b640 1 -- 172.21.15.116:0/584730444 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa798004810 msgr2=0x7fa798004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:24.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.049+0000 7fa7b492b640 1 --2- 172.21.15.116:0/584730444 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa798004810 0x7fa798004c90 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7fa7b01539d0 tx=0x7fa7a001cbe0 comp rx=0 tx=0).stop 2024-04-25T00:17:24.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.049+0000 7fa7aed9d640 1 -- 172.21.15.116:0/584730444 reap_dead start 2024-04-25T00:17:24.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.049+0000 7fa7b492b640 1 -- 172.21.15.116:0/584730444 shutdown_connections 2024-04-25T00:17:24.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.049+0000 7fa7b492b640 1 -- 172.21.15.116:0/584730444 >> 172.21.15.116:0/584730444 conn(0x7fa7b00c4270 msgr2=0x7fa7b00c4680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:24.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.049+0000 7fa7b492b640 1 -- 172.21.15.116:0/584730444 shutdown_connections 2024-04-25T00:17:24.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.049+0000 7fa7b492b640 1 -- 172.21.15.116:0/584730444 wait complete. 2024-04-25T00:17:24.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.049+0000 7fa7b492b640 1 Processor -- start 2024-04-25T00:17:24.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.049+0000 7fa7b492b640 1 -- start start 2024-04-25T00:17:24.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.049+0000 7fa7b492b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa798004810 0x7fa7b014b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:24.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.049+0000 7fa7b492b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa7b014c4f0 0x7fa7b0148c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:24.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.049+0000 7fa7b492b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7b014d670 0x7fa7b0154960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:24.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.049+0000 7fa7b492b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa7b015faa0 con 0x7fa798004810 2024-04-25T00:17:24.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.049+0000 7fa7b492b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa7b015f920 con 0x7fa7b014d670 2024-04-25T00:17:24.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.049+0000 7fa7b492b640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa7b015fc20 con 0x7fa7b014c4f0 2024-04-25T00:17:24.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.049+0000 7fa7ae59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa798004810 0x7fa7b014b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:24.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.049+0000 7fa7ae59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa798004810 0x7fa7b014b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42348/0 (socket says 172.21.15.116:42348) 2024-04-25T00:17:24.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.049+0000 7fa7ae59c640 1 -- 172.21.15.116:0/481479351 learned_addr learned my addr 172.21.15.116:0/481479351 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:17:24.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.053+0000 7fa7add9b640 1 --2- 172.21.15.116:0/481479351 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa7b014c4f0 0x7fa7b0148c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:24.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.053+0000 7fa7aed9d640 1 --2- 172.21.15.116:0/481479351 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7b014d670 0x7fa7b0154960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:24.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.053+0000 7fa7ae59c640 1 -- 172.21.15.116:0/481479351 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa7b014c4f0 msgr2=0x7fa7b0148c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:24.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.053+0000 7fa7ae59c640 1 --2- 172.21.15.116:0/481479351 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa7b014c4f0 0x7fa7b0148c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:24.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.053+0000 7fa7ae59c640 1 -- 172.21.15.116:0/481479351 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7b014d670 msgr2=0x7fa7b0154960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:24.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.053+0000 7fa7ae59c640 1 --2- 172.21.15.116:0/481479351 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7b014d670 0x7fa7b0154960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:24.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.053+0000 7fa7ae59c640 1 -- 172.21.15.116:0/481479351 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7b014b710 con 0x7fa798004810 2024-04-25T00:17:24.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.053+0000 7fa7aed9d640 1 --2- 172.21.15.116:0/481479351 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7b014d670 0x7fa7b0154960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:17:24.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.053+0000 7fa7ae59c640 1 --2- 172.21.15.116:0/481479351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa798004810 0x7fa7b014b1d0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fa7a40116d0 tx=0x7fa7a4011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:24.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.053+0000 7fa79effd640 1 -- 172.21.15.116:0/481479351 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7a4004220 con 0x7fa798004810 2024-04-25T00:17:24.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.053+0000 7fa79effd640 1 -- 172.21.15.116:0/481479351 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7a40043c0 con 0x7fa798004810 2024-04-25T00:17:24.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.053+0000 7fa7b492b640 1 -- 172.21.15.116:0/481479351 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7b01494e0 con 0x7fa798004810 2024-04-25T00:17:24.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.053+0000 7fa79effd640 1 -- 172.21.15.116:0/481479351 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7a400f9e0 con 0x7fa798004810 2024-04-25T00:17:24.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.053+0000 7fa7b492b640 1 -- 172.21.15.116:0/481479351 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7b01491d0 con 0x7fa798004810 2024-04-25T00:17:24.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.053+0000 7fa7b492b640 1 -- 172.21.15.116:0/481479351 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7b0068a60 con 0x7fa798004810 2024-04-25T00:17:24.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.053+0000 7fa79effd640 1 -- 172.21.15.116:0/481479351 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa7a400fb80 con 0x7fa798004810 2024-04-25T00:17:24.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.053+0000 7fa79effd640 1 --2- 172.21.15.116:0/481479351 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa7900678c0 0x7fa790069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:24.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.053+0000 7fa79effd640 1 -- 172.21.15.116:0/481479351 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 8644+0+0 (secure 0 0 0) 0x7fa7a4086ec0 con 0x7fa798004810 2024-04-25T00:17:24.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.053+0000 7f5bbb00d640 1 -- 172.21.15.116:0/128197175 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5bb406e670 con 0x7f5bb4070dd0 2024-04-25T00:17:24.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.053+0000 7fa7add9b640 1 --2- 172.21.15.116:0/481479351 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa7900678c0 0x7fa790069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:24.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.053+0000 7fa7add9b640 1 --2- 172.21.15.116:0/481479351 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa7900678c0 0x7fa790069d80 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fa7a00ab000 tx=0x7fa7a00b1040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:24.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.057+0000 7fa79effd640 1 -- 172.21.15.116:0/481479351 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fa7a4019030 con 0x7fa798004810 2024-04-25T00:17:24.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.057+0000 7f5bb17fa640 1 -- 172.21.15.116:0/128197175 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) ==== 95+0+6509 (secure 0 0 0) 0x7f5ba00560b0 con 0x7f5bb4070dd0 2024-04-25T00:17:24.059 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:17:24.059 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":127,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7164,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/3336339376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3336339376},{"type":"v1","addr":"172.21.15.116:6835","nonce":3336339376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":127}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:16:48.854104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7119},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7119":{"gid":7119,"name":"a","rank":0,"incarnation":123,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6833/3645553116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3645553116},{"type":"v1","addr":"172.21.15.116:6833","nonce":3645553116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7119,"qdb_cluster":[7119]},"id":1},{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:16:38.790885+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5621},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5621":{"gid":5621,"name":"d","rank":0,"incarnation":118,"state":"up:active","state_seq":15,"addr":"172.21.15.174:6837/3081407040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3081407040},{"type":"v1","addr":"172.21.15.174:6837","nonce":3081407040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5621,"qdb_cluster":[5621]},"id":2}]} 2024-04-25T00:17:24.060 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 127 2024-04-25T00:17:24.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.061+0000 7f5bbb00d640 1 -- 172.21.15.116:0/128197175 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5b980678c0 msgr2=0x7f5b98069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:24.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.061+0000 7f5bbb00d640 1 --2- 172.21.15.116:0/128197175 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5b980678c0 0x7f5b98069d80 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f5ba80b1000 tx=0x7f5ba80a9070 comp rx=0 tx=0).stop 2024-04-25T00:17:24.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.061+0000 7f5bbb00d640 1 -- 172.21.15.116:0/128197175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5bb4070dd0 msgr2=0x7f5bb41f2240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:24.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.061+0000 7f5bbb00d640 1 --2- 172.21.15.116:0/128197175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5bb4070dd0 0x7f5bb41f2240 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f5ba000c820 tx=0x7f5ba000cce0 comp rx=0 tx=0).stop 2024-04-25T00:17:24.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.061+0000 7f5bbb00d640 1 -- 172.21.15.116:0/128197175 shutdown_connections 2024-04-25T00:17:24.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.061+0000 7f5bbb00d640 1 --2- 172.21.15.116:0/128197175 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5b980678c0 0x7f5b98069d80 secure :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f5ba80b1000 tx=0x7f5ba80a9070 comp rx=0 tx=0).stop 2024-04-25T00:17:24.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.061+0000 7f5bbb00d640 1 --2- 172.21.15.116:0/128197175 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5bb4148890 0x7f5bb41f2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:24.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.061+0000 7f5bbb00d640 1 --2- 172.21.15.116:0/128197175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5bb4070dd0 0x7f5bb41f2240 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:24.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.061+0000 7f5bbb00d640 1 --2- 172.21.15.116:0/128197175 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5bb406e890 0x7f5bb41423a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:24.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.061+0000 7f5bbb00d640 1 -- 172.21.15.116:0/128197175 >> 172.21.15.116:0/128197175 conn(0x7f5bb40c3b00 msgr2=0x7f5bb4071200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:24.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.061+0000 7f5bbb00d640 1 -- 172.21.15.116:0/128197175 shutdown_connections 2024-04-25T00:17:24.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.061+0000 7f5bbb00d640 1 -- 172.21.15.116:0/128197175 wait complete. 2024-04-25T00:17:24.075 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:17:24.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.249+0000 7fa7b492b640 1 -- 172.21.15.116:0/481479351 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa7b0074aa0 con 0x7fa798004810 2024-04-25T00:17:24.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.249+0000 7fa79effd640 1 -- 172.21.15.116:0/481479351 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) ==== 95+0+6509 (secure 0 0 0) 0x7fa7a4056ec0 con 0x7fa798004810 2024-04-25T00:17:24.251 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:17:24.252 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":127,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7164,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/3336339376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3336339376},{"type":"v1","addr":"172.21.15.116:6835","nonce":3336339376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":127}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:16:48.854104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7119},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7119":{"gid":7119,"name":"a","rank":0,"incarnation":123,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6833/3645553116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3645553116},{"type":"v1","addr":"172.21.15.116:6833","nonce":3645553116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7119,"qdb_cluster":[7119]},"id":1},{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:16:38.790885+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":50,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5621},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5621":{"gid":5621,"name":"d","rank":0,"incarnation":118,"state":"up:active","state_seq":15,"addr":"172.21.15.174:6837/3081407040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3081407040},{"type":"v1","addr":"172.21.15.174:6837","nonce":3081407040}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5621,"qdb_cluster":[5621]},"id":2}]} 2024-04-25T00:17:24.254 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 127 2024-04-25T00:17:24.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.249+0000 7fa7b492b640 1 -- 172.21.15.116:0/481479351 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa7900678c0 msgr2=0x7fa790069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:24.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.249+0000 7fa7b492b640 1 --2- 172.21.15.116:0/481479351 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa7900678c0 0x7fa790069d80 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fa7a00ab000 tx=0x7fa7a00b1040 comp rx=0 tx=0).stop 2024-04-25T00:17:24.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.249+0000 7fa7b492b640 1 -- 172.21.15.116:0/481479351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa798004810 msgr2=0x7fa7b014b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:24.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.249+0000 7fa7b492b640 1 --2- 172.21.15.116:0/481479351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa798004810 0x7fa7b014b1d0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fa7a40116d0 tx=0x7fa7a4011b90 comp rx=0 tx=0).stop 2024-04-25T00:17:24.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.249+0000 7fa7b492b640 1 -- 172.21.15.116:0/481479351 shutdown_connections 2024-04-25T00:17:24.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.249+0000 7fa7b492b640 1 --2- 172.21.15.116:0/481479351 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa7900678c0 0x7fa790069d80 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:24.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.249+0000 7fa7b492b640 1 --2- 172.21.15.116:0/481479351 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa7b014d670 0x7fa7b0154960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:24.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.249+0000 7fa7b492b640 1 --2- 172.21.15.116:0/481479351 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa7b014c4f0 0x7fa7b0148c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:24.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.249+0000 7fa7b492b640 1 --2- 172.21.15.116:0/481479351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa798004810 0x7fa7b014b1d0 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:24.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.249+0000 7fa7b492b640 1 -- 172.21.15.116:0/481479351 >> 172.21.15.116:0/481479351 conn(0x7fa7b00c4270 msgr2=0x7fa7b014a710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:24.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.249+0000 7fa7b492b640 1 -- 172.21.15.116:0/481479351 shutdown_connections 2024-04-25T00:17:24.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:24.249+0000 7fa7b492b640 1 -- 172.21.15.116:0/481479351 wait complete. 2024-04-25T00:17:24.266 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:17:26.076 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:17:26.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.201+0000 7f5778b91640 1 Processor -- start 2024-04-25T00:17:26.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.201+0000 7f5778b91640 1 -- start start 2024-04-25T00:17:26.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.201+0000 7f5778b91640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f577414c720 0x7f57741563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:26.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.201+0000 7f5778b91640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5774071680 0x7f5774071ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:26.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.201+0000 7f5778b91640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5774071220 con 0x7f57740a94a0 2024-04-25T00:17:26.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.201+0000 7f5778b91640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f5774153460 con 0x7f5774071680 2024-04-25T00:17:26.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.201+0000 7f5778b91640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f5774152420 con 0x7f577414c720 2024-04-25T00:17:26.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.201+0000 7f5772d9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5774071680 0x7f5774071ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:26.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.201+0000 7f5772d9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5774071680 0x7f5774071ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42298/0 (socket says 172.21.15.116:42298) 2024-04-25T00:17:26.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.201+0000 7f5772d9d640 1 -- 172.21.15.116:0/2551315597 learned_addr learned my addr 172.21.15.116:0/2551315597 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:17:26.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.201+0000 7f577259c640 1 --2- 172.21.15.116:0/2551315597 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f577414c720 0x7f57741563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:26.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.201+0000 7f5772d9d640 1 -- 172.21.15.116:0/2551315597 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f577414c720 msgr2=0x7f57741563d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:26.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.201+0000 7f5772d9d640 1 --2- 172.21.15.116:0/2551315597 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f577414c720 0x7f57741563d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:26.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.201+0000 7f5772d9d640 1 -- 172.21.15.116:0/2551315597 >> v1:172.21.15.174:6789/0 conn(0x7f57740a94a0 legacy=0x7f57740c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:26.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.201+0000 7f5772d9d640 1 -- 172.21.15.116:0/2551315597 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f577414cb00 con 0x7f5774071680 2024-04-25T00:17:26.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.201+0000 7f5772d9d640 1 --2- 172.21.15.116:0/2551315597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5774071680 0x7f5774071ad0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f576c009e50 tx=0x7f576c01c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5de3f6b94b27a507 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:26.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.205+0000 7f577159a640 1 -- 172.21.15.116:0/2551315597 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f576c0a5730 con 0x7f5774071680 2024-04-25T00:17:26.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.205+0000 7f577159a640 1 -- 172.21.15.116:0/2551315597 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f576c0a58d0 con 0x7f5774071680 2024-04-25T00:17:26.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.205+0000 7f577159a640 1 -- 172.21.15.116:0/2551315597 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f576c0a5ca0 con 0x7f5774071680 2024-04-25T00:17:26.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.205+0000 7f5778b91640 1 -- 172.21.15.116:0/2551315597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5774071680 msgr2=0x7f5774071ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:26.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.205+0000 7f5778b91640 1 --2- 172.21.15.116:0/2551315597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5774071680 0x7f5774071ad0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f576c009e50 tx=0x7f576c01c970 comp rx=0 tx=0).stop 2024-04-25T00:17:26.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.205+0000 7f5778b91640 1 -- 172.21.15.116:0/2551315597 shutdown_connections 2024-04-25T00:17:26.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.205+0000 7f5778b91640 1 --2- 172.21.15.116:0/2551315597 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5774071680 0x7f5774071ad0 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:26.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.205+0000 7f5778b91640 1 --2- 172.21.15.116:0/2551315597 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f577414c720 0x7f57741563d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:26.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.205+0000 7f5778b91640 1 -- 172.21.15.116:0/2551315597 >> 172.21.15.116:0/2551315597 conn(0x7f57740c3b00 msgr2=0x7f57740c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:26.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.205+0000 7f5778b91640 1 -- 172.21.15.116:0/2551315597 shutdown_connections 2024-04-25T00:17:26.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.205+0000 7f5778b91640 1 -- 172.21.15.116:0/2551315597 wait complete. 2024-04-25T00:17:26.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.205+0000 7f5778b91640 1 Processor -- start 2024-04-25T00:17:26.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.205+0000 7f5778b91640 1 -- start start 2024-04-25T00:17:26.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.205+0000 7f5778b91640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5774071680 0x7f57740c5db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:26.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.205+0000 7f5778b91640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f57740a94a0 0x7f57740c62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:26.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.205+0000 7f5778b91640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f577414c720 0x7f577413d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:26.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.205+0000 7f5778b91640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f57740c3120 con 0x7f577414c720 2024-04-25T00:17:26.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.205+0000 7f5778b91640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5774152f40 con 0x7f57740a94a0 2024-04-25T00:17:26.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.205+0000 7f5778b91640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f57740c32a0 con 0x7f5774071680 2024-04-25T00:17:26.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.205+0000 7f5772d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f577414c720 0x7f577413d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:26.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.205+0000 7f5772d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f577414c720 0x7f577413d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42300/0 (socket says 172.21.15.116:42300) 2024-04-25T00:17:26.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.205+0000 7f5772d9d640 1 -- 172.21.15.116:0/913931641 learned_addr learned my addr 172.21.15.116:0/913931641 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:17:26.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.205+0000 7f5772d9d640 1 -- 172.21.15.116:0/913931641 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5774071680 msgr2=0x7f57740c5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:26.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.205+0000 7f5772d9d640 1 --2- 172.21.15.116:0/913931641 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5774071680 0x7f57740c5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:26.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.205+0000 7f5772d9d640 1 -- 172.21.15.116:0/913931641 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f57740a94a0 msgr2=0x7f57740c62f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:26.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.205+0000 7f5772d9d640 1 --2- 172.21.15.116:0/913931641 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f57740a94a0 0x7f57740c62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:26.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.205+0000 7f5772d9d640 1 -- 172.21.15.116:0/913931641 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57740ac540 con 0x7f577414c720 2024-04-25T00:17:26.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.205+0000 7f5772d9d640 1 --2- 172.21.15.116:0/913931641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f577414c720 0x7f577413d4f0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f576c0af040 tx=0x7f576c019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:26.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.205+0000 7f5762ffd640 1 -- 172.21.15.116:0/913931641 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f576c0bc020 con 0x7f577414c720 2024-04-25T00:17:26.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.205+0000 7f5762ffd640 1 -- 172.21.15.116:0/913931641 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f576c0b8670 con 0x7f577414c720 2024-04-25T00:17:26.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.205+0000 7f5778b91640 1 -- 172.21.15.116:0/913931641 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57740ac820 con 0x7f577414c720 2024-04-25T00:17:26.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.205+0000 7f5762ffd640 1 -- 172.21.15.116:0/913931641 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f576c0b8a60 con 0x7f577414c720 2024-04-25T00:17:26.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.205+0000 7f5778b91640 1 -- 172.21.15.116:0/913931641 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f577413cfd0 con 0x7f577414c720 2024-04-25T00:17:26.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.209+0000 7f5778b91640 1 -- 172.21.15.116:0/913931641 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57740acd40 con 0x7f577414c720 2024-04-25T00:17:26.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.209+0000 7f5762ffd640 1 -- 172.21.15.116:0/913931641 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f576c0bf020 con 0x7f577414c720 2024-04-25T00:17:26.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.209+0000 7f5762ffd640 1 --2- 172.21.15.116:0/913931641 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f57540678c0 0x7f5754069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:26.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.209+0000 7f5762ffd640 1 -- 172.21.15.116:0/913931641 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 8730+0+0 (secure 0 0 0) 0x7f576c121980 con 0x7f577414c720 2024-04-25T00:17:26.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.209+0000 7f577259c640 1 --2- 172.21.15.116:0/913931641 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f57540678c0 0x7f5754069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:26.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.209+0000 7f577259c640 1 --2- 172.21.15.116:0/913931641 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f57540678c0 0x7f5754069d80 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f576400b850 tx=0x7f5764008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:26.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.213+0000 7f5762ffd640 1 -- 172.21.15.116:0/913931641 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f57740acd40 con 0x7f577414c720 2024-04-25T00:17:26.267 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:17:26.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.377+0000 7fd393dd7640 1 Processor -- start 2024-04-25T00:17:26.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.377+0000 7fd393dd7640 1 -- start start 2024-04-25T00:17:26.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.381+0000 7fd393dd7640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fd38c1562d0 0x7fd38c161130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:26.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.381+0000 7fd393dd7640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd38c073200 con 0x7fd38c152e60 2024-04-25T00:17:26.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.381+0000 7fd393dd7640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd38c158ec0 con 0x7fd38c1599a0 2024-04-25T00:17:26.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.381+0000 7fd393dd7640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fd38c153240 con 0x7fd38c1562d0 2024-04-25T00:17:26.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.381+0000 7fd391b73640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fd38c152e60 0x7fd38c0c36e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:39628/0 (socket says 172.21.15.116:39628) 2024-04-25T00:17:26.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.381+0000 7fd391b73640 1 -- 172.21.15.116:0/2316490970 learned_addr learned my addr 172.21.15.116:0/2316490970 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:17:26.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.381+0000 7fd390b71640 1 -- 172.21.15.116:0/2316490970 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2067356449 0 0) 0x7fd38c073200 con 0x7fd38c152e60 2024-04-25T00:17:26.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.381+0000 7fd392374640 1 --2- 172.21.15.116:0/2316490970 >> v2:172.21.15.174:3300/0 conn(0x7fd38c1562d0 0x7fd38c161130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:26.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.381+0000 7fd390b71640 1 -- 172.21.15.116:0/2316490970 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd378003530 con 0x7fd38c152e60 2024-04-25T00:17:26.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.381+0000 7fd390b71640 1 -- 172.21.15.116:0/2316490970 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 670759070 0 0) 0x7fd378003530 con 0x7fd38c152e60 2024-04-25T00:17:26.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.381+0000 7fd390b71640 1 -- 172.21.15.116:0/2316490970 >> v2:172.21.15.174:3300/0 conn(0x7fd38c1562d0 msgr2=0x7fd38c161130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:26.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.381+0000 7fd390b71640 1 --2- 172.21.15.116:0/2316490970 >> v2:172.21.15.174:3300/0 conn(0x7fd38c1562d0 0x7fd38c161130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:26.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.381+0000 7fd390b71640 1 -- 172.21.15.116:0/2316490970 >> v1:172.21.15.174:6789/0 conn(0x7fd38c1599a0 legacy=0x7fd38c0c5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:26.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.381+0000 7fd390b71640 1 -- 172.21.15.116:0/2316490970 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd38c1591a0 con 0x7fd38c152e60 2024-04-25T00:17:26.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.381+0000 7fd390b71640 1 -- 172.21.15.116:0/2316490970 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7fd388001680 con 0x7fd38c152e60 2024-04-25T00:17:26.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.381+0000 7fd390b71640 1 -- 172.21.15.116:0/2316490970 >> v1:172.21.15.116:6789/0 conn(0x7fd38c152e60 legacy=0x7fd38c0c36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:26.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.381+0000 7fd390b71640 1 --2- 172.21.15.116:0/2316490970 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3780039c0 0x7fd378003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:26.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.381+0000 7fd390b71640 1 --2- 172.21.15.116:0/2316490970 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd378004810 0x7fd378004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:26.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.381+0000 7fd390b71640 1 --2- 172.21.15.116:0/2316490970 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd3780051d0 0x7fd37800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:26.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.381+0000 7fd390b71640 1 -- 172.21.15.116:0/2316490970 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd38c153240 con 0x7fd3780039c0 2024-04-25T00:17:26.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.381+0000 7fd390b71640 1 -- 172.21.15.116:0/2316490970 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd37800c170 con 0x7fd3780051d0 2024-04-25T00:17:26.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.381+0000 7fd390b71640 1 -- 172.21.15.116:0/2316490970 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fd37800c370 con 0x7fd378004810 2024-04-25T00:17:26.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.381+0000 7fd391b73640 1 --2- 172.21.15.116:0/2316490970 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3780039c0 0x7fd378003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:26.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.381+0000 7fd391b73640 1 -- 172.21.15.116:0/2316490970 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd378004810 msgr2=0x7fd378004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:26.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.381+0000 7fd391b73640 1 --2- 172.21.15.116:0/2316490970 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd378004810 0x7fd378004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:26.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.381+0000 7fd391b73640 1 -- 172.21.15.116:0/2316490970 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd3780051d0 msgr2=0x7fd37800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:26.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.381+0000 7fd391b73640 1 --2- 172.21.15.116:0/2316490970 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd3780051d0 0x7fd37800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:26.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.381+0000 7fd391b73640 1 -- 172.21.15.116:0/2316490970 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd378003530 con 0x7fd3780039c0 2024-04-25T00:17:26.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.381+0000 7fd392374640 1 -- 172.21.15.116:0/2316490970 reap_dead start 2024-04-25T00:17:26.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.381+0000 7fd391b73640 1 --2- 172.21.15.116:0/2316490970 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3780039c0 0x7fd378003de0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7fd38800c210 tx=0x7fd38801cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dbb8aacf02ef9ef server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:26.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.381+0000 7fd390b71640 1 -- 172.21.15.116:0/2316490970 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd388007d40 con 0x7fd3780039c0 2024-04-25T00:17:26.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.381+0000 7fd390b71640 1 -- 172.21.15.116:0/2316490970 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3880a9070 con 0x7fd3780039c0 2024-04-25T00:17:26.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.381+0000 7fd390b71640 1 -- 172.21.15.116:0/2316490970 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd388005d80 con 0x7fd3780039c0 2024-04-25T00:17:26.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.385+0000 7fd393dd7640 1 -- 172.21.15.116:0/2316490970 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3780039c0 msgr2=0x7fd378003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:26.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.385+0000 7fd393dd7640 1 --2- 172.21.15.116:0/2316490970 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3780039c0 0x7fd378003de0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7fd38800c210 tx=0x7fd38801cca0 comp rx=0 tx=0).stop 2024-04-25T00:17:26.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.385+0000 7fd393dd7640 1 -- 172.21.15.116:0/2316490970 shutdown_connections 2024-04-25T00:17:26.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.385+0000 7fd393dd7640 1 --2- 172.21.15.116:0/2316490970 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3780039c0 0x7fd378003de0 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:26.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.385+0000 7fd393dd7640 1 -- 172.21.15.116:0/2316490970 >> 172.21.15.116:0/2316490970 conn(0x7fd38c06c720 msgr2=0x7fd38c06cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:26.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.385+0000 7fd393dd7640 1 -- 172.21.15.116:0/2316490970 shutdown_connections 2024-04-25T00:17:26.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.385+0000 7fd393dd7640 1 -- 172.21.15.116:0/2316490970 wait complete. 2024-04-25T00:17:26.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.385+0000 7fd393dd7640 1 Processor -- start 2024-04-25T00:17:26.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.385+0000 7fd393dd7640 1 -- start start 2024-04-25T00:17:26.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.385+0000 7fd393dd7640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3780039c0 0x7fd38c1ef040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:26.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.385+0000 7fd393dd7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd38c06e6a0 0x7fd38c06eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:26.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.385+0000 7fd393dd7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd38c1436e0 0x7fd38c143b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:26.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.385+0000 7fd393dd7640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd38c1566e0 con 0x7fd38c06e6a0 2024-04-25T00:17:26.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.385+0000 7fd393dd7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd38c0734e0 con 0x7fd38c1436e0 2024-04-25T00:17:26.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.385+0000 7fd393dd7640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fd38c0c5010 con 0x7fd3780039c0 2024-04-25T00:17:26.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.385+0000 7fd391372640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd38c06e6a0 0x7fd38c06eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:26.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.385+0000 7fd391372640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd38c06e6a0 0x7fd38c06eb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42326/0 (socket says 172.21.15.116:42326) 2024-04-25T00:17:26.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.385+0000 7fd391372640 1 -- 172.21.15.116:0/986764001 learned_addr learned my addr 172.21.15.116:0/986764001 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:17:26.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.385+0000 7fd391b73640 1 --2- 172.21.15.116:0/986764001 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3780039c0 0x7fd38c1ef040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:26.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.385+0000 7fd392374640 1 --2- 172.21.15.116:0/986764001 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd38c1436e0 0x7fd38c143b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:26.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.385+0000 7fd391372640 1 -- 172.21.15.116:0/986764001 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3780039c0 msgr2=0x7fd38c1ef040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:26.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.385+0000 7fd391372640 1 --2- 172.21.15.116:0/986764001 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3780039c0 0x7fd38c1ef040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:26.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.385+0000 7fd391372640 1 -- 172.21.15.116:0/986764001 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd38c1436e0 msgr2=0x7fd38c143b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:26.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.385+0000 7fd391372640 1 --2- 172.21.15.116:0/986764001 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd38c1436e0 0x7fd38c143b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:26.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.385+0000 7fd391372640 1 -- 172.21.15.116:0/986764001 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd38c0ad310 con 0x7fd38c06e6a0 2024-04-25T00:17:26.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.385+0000 7fd391372640 1 --2- 172.21.15.116:0/986764001 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd38c06e6a0 0x7fd38c06eb20 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7fd37c00c430 tx=0x7fd37c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:26.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.385+0000 7fd3827fc640 1 -- 172.21.15.116:0/986764001 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd37c0052d0 con 0x7fd38c06e6a0 2024-04-25T00:17:26.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.385+0000 7fd3827fc640 1 -- 172.21.15.116:0/986764001 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd37c005470 con 0x7fd38c06e6a0 2024-04-25T00:17:26.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.385+0000 7fd393dd7640 1 -- 172.21.15.116:0/986764001 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd38c1e16f0 con 0x7fd38c06e6a0 2024-04-25T00:17:26.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.385+0000 7fd3827fc640 1 -- 172.21.15.116:0/986764001 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd37c007260 con 0x7fd38c06e6a0 2024-04-25T00:17:26.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.385+0000 7fd393dd7640 1 -- 172.21.15.116:0/986764001 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd38c1e1cb0 con 0x7fd38c06e6a0 2024-04-25T00:17:26.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.389+0000 7fd393dd7640 1 -- 172.21.15.116:0/986764001 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd38c072b20 con 0x7fd38c06e6a0 2024-04-25T00:17:26.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.389+0000 7fd3827fc640 1 -- 172.21.15.116:0/986764001 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd37c007400 con 0x7fd38c06e6a0 2024-04-25T00:17:26.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.389+0000 7fd3827fc640 1 --2- 172.21.15.116:0/986764001 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd3700678f0 0x7fd370069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:26.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.389+0000 7fd3827fc640 1 -- 172.21.15.116:0/986764001 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 8730+0+0 (secure 0 0 0) 0x7fd37c085b90 con 0x7fd38c06e6a0 2024-04-25T00:17:26.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.389+0000 7fd391b73640 1 --2- 172.21.15.116:0/986764001 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd3700678f0 0x7fd370069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:26.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.389+0000 7fd391b73640 1 --2- 172.21.15.116:0/986764001 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd3700678f0 0x7fd370069db0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fd3880ab000 tx=0x7fd3880b0040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:26.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.393+0000 7fd3827fc640 1 -- 172.21.15.116:0/986764001 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fd37c055ca0 con 0x7fd38c06e6a0 2024-04-25T00:17:26.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.397+0000 7f5778b91640 1 -- 172.21.15.116:0/913931641 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5774068350 con 0x7f577414c720 2024-04-25T00:17:26.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.397+0000 7f5762ffd640 1 -- 172.21.15.116:0/913931641 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) ==== 95+0+5672 (secure 0 0 0) 0x7f576c0c6030 con 0x7f577414c720 2024-04-25T00:17:26.398 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:17:26.399 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":128,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:16:48.854104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7119},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7119":{"gid":7119,"name":"a","rank":0,"incarnation":123,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6833/3645553116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3645553116},{"type":"v1","addr":"172.21.15.116:6833","nonce":3645553116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7119,"qdb_cluster":[7119]},"id":1},{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:17:25.599032+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7164},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7164":{"gid":7164,"name":"c","rank":0,"incarnation":128,"state":"up:replay","state_seq":1,"addr":"172.21.15.116:6835/3336339376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3336339376},{"type":"v1","addr":"172.21.15.116:6835","nonce":3336339376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:17:26.399 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 128 2024-04-25T00:17:26.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.401+0000 7f5778b91640 1 -- 172.21.15.116:0/913931641 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f57540678c0 msgr2=0x7f5754069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:26.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.401+0000 7f5778b91640 1 --2- 172.21.15.116:0/913931641 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f57540678c0 0x7f5754069d80 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f576400b850 tx=0x7f5764008040 comp rx=0 tx=0).stop 2024-04-25T00:17:26.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.401+0000 7f5778b91640 1 -- 172.21.15.116:0/913931641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f577414c720 msgr2=0x7f577413d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:26.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.401+0000 7f5778b91640 1 --2- 172.21.15.116:0/913931641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f577414c720 0x7f577413d4f0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f576c0af040 tx=0x7f576c019710 comp rx=0 tx=0).stop 2024-04-25T00:17:26.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.401+0000 7f5778b91640 1 -- 172.21.15.116:0/913931641 shutdown_connections 2024-04-25T00:17:26.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.401+0000 7f5778b91640 1 --2- 172.21.15.116:0/913931641 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f57540678c0 0x7f5754069d80 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:26.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.401+0000 7f5778b91640 1 --2- 172.21.15.116:0/913931641 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f577414c720 0x7f577413d4f0 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:26.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.401+0000 7f5778b91640 1 --2- 172.21.15.116:0/913931641 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f57740a94a0 0x7f57740c62f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:26.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.401+0000 7f5778b91640 1 --2- 172.21.15.116:0/913931641 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5774071680 0x7f57740c5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:26.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.401+0000 7f5778b91640 1 -- 172.21.15.116:0/913931641 >> 172.21.15.116:0/913931641 conn(0x7f57740c3b00 msgr2=0x7f57740c2480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:26.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.401+0000 7f5778b91640 1 -- 172.21.15.116:0/913931641 shutdown_connections 2024-04-25T00:17:26.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.401+0000 7f5778b91640 1 -- 172.21.15.116:0/913931641 wait complete. 2024-04-25T00:17:26.415 INFO:tasks.mds_thrash.fs.[b]:mds.d down, removed from mdsmap 2024-04-25T00:17:26.415 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-04-25T00:17:26.415 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:17:26.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.521+0000 7fa7799e7640 1 Processor -- start 2024-04-25T00:17:26.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.521+0000 7fa7799e7640 1 -- start start 2024-04-25T00:17:26.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.521+0000 7fa7799e7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa774154a60 0x7fa774153fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:26.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.521+0000 7fa7799e7640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7740c8360 con 0x7fa774070ca0 2024-04-25T00:17:26.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.521+0000 7fa7799e7640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7740c8620 con 0x7fa774149f60 2024-04-25T00:17:26.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.521+0000 7fa7799e7640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fa7740c8900 con 0x7fa774154a60 2024-04-25T00:17:26.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.521+0000 7fa7727fc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa774154a60 0x7fa774153fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:26.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.521+0000 7fa7727fc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa774154a60 0x7fa774153fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42332/0 (socket says 172.21.15.116:42332) 2024-04-25T00:17:26.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.521+0000 7fa7727fc640 1 -- 172.21.15.116:0/3978550573 learned_addr learned my addr 172.21.15.116:0/3978550573 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:17:26.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.521+0000 7fa7727fc640 1 -- 172.21.15.116:0/3978550573 >> v1:172.21.15.174:6789/0 conn(0x7fa774149f60 legacy=0x7fa774157220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:26.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.521+0000 7fa7727fc640 1 -- 172.21.15.116:0/3978550573 >> v1:172.21.15.116:6789/0 conn(0x7fa774070ca0 legacy=0x7fa774071110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:26.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.521+0000 7fa771ffb640 1 -- 172.21.15.116:0/3978550573 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1770726377 0 0) 0x7fa7740c8360 con 0x7fa774070ca0 2024-04-25T00:17:26.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.521+0000 7fa7727fc640 1 -- 172.21.15.116:0/3978550573 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa774156d00 con 0x7fa774154a60 2024-04-25T00:17:26.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.525+0000 7fa7727fc640 1 --2- 172.21.15.116:0/3978550573 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa774154a60 0x7fa774153fc0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7fa768009870 tx=0x7fa76801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=699d0fa7e66b3324 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:26.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.525+0000 7fa771ffb640 1 -- 172.21.15.116:0/3978550573 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7680b1070 con 0x7fa774154a60 2024-04-25T00:17:26.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.525+0000 7fa771ffb640 1 -- 172.21.15.116:0/3978550573 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7680ac600 con 0x7fa774154a60 2024-04-25T00:17:26.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.525+0000 7fa771ffb640 1 -- 172.21.15.116:0/3978550573 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7680ac9d0 con 0x7fa774154a60 2024-04-25T00:17:26.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.525+0000 7fa7799e7640 1 -- 172.21.15.116:0/3978550573 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa774154a60 msgr2=0x7fa774153fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:26.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.525+0000 7fa7799e7640 1 --2- 172.21.15.116:0/3978550573 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa774154a60 0x7fa774153fc0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7fa768009870 tx=0x7fa76801c490 comp rx=0 tx=0).stop 2024-04-25T00:17:26.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.525+0000 7fa7799e7640 1 -- 172.21.15.116:0/3978550573 shutdown_connections 2024-04-25T00:17:26.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.525+0000 7fa7799e7640 1 --2- 172.21.15.116:0/3978550573 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa774154a60 0x7fa774153fc0 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:26.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.525+0000 7fa7799e7640 1 -- 172.21.15.116:0/3978550573 >> 172.21.15.116:0/3978550573 conn(0x7fa774074a80 msgr2=0x7fa7740c8f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:26.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.525+0000 7fa7799e7640 1 -- 172.21.15.116:0/3978550573 shutdown_connections 2024-04-25T00:17:26.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.525+0000 7fa7799e7640 1 -- 172.21.15.116:0/3978550573 wait complete. 2024-04-25T00:17:26.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.525+0000 7fa7799e7640 1 Processor -- start 2024-04-25T00:17:26.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.525+0000 7fa7799e7640 1 -- start start 2024-04-25T00:17:26.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.529+0000 7fa7799e7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa774070ca0 0x7fa7740c2590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:26.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.529+0000 7fa7799e7640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa774149f60 0x7fa77414cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:26.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.529+0000 7fa7799e7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa774154a60 0x7fa77414d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:26.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.529+0000 7fa7799e7640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa7740719a0 con 0x7fa774070ca0 2024-04-25T00:17:26.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.529+0000 7fa7799e7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa774071820 con 0x7fa774154a60 2024-04-25T00:17:26.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.529+0000 7fa7799e7640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa7740adf80 con 0x7fa774149f60 2024-04-25T00:17:26.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.529+0000 7fa772ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa774070ca0 0x7fa7740c2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:26.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.529+0000 7fa772ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa774070ca0 0x7fa7740c2590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42346/0 (socket says 172.21.15.116:42346) 2024-04-25T00:17:26.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.529+0000 7fa772ffd640 1 -- 172.21.15.116:0/3454292738 learned_addr learned my addr 172.21.15.116:0/3454292738 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:17:26.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.529+0000 7fa7727fc640 1 --2- 172.21.15.116:0/3454292738 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa774149f60 0x7fa77414cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:26.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.529+0000 7fa7737fe640 1 --2- 172.21.15.116:0/3454292738 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa774154a60 0x7fa77414d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:26.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.529+0000 7fa772ffd640 1 -- 172.21.15.116:0/3454292738 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa774149f60 msgr2=0x7fa77414cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:26.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.529+0000 7fa772ffd640 1 --2- 172.21.15.116:0/3454292738 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa774149f60 0x7fa77414cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:26.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.529+0000 7fa772ffd640 1 -- 172.21.15.116:0/3454292738 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa774154a60 msgr2=0x7fa77414d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:26.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.529+0000 7fa772ffd640 1 --2- 172.21.15.116:0/3454292738 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa774154a60 0x7fa77414d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:26.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.529+0000 7fa772ffd640 1 -- 172.21.15.116:0/3454292738 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa77406e2c0 con 0x7fa774070ca0 2024-04-25T00:17:26.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.529+0000 7fa7727fc640 1 --2- 172.21.15.116:0/3454292738 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa774149f60 0x7fa77414cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:17:26.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.529+0000 7fa7737fe640 1 --2- 172.21.15.116:0/3454292738 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa774154a60 0x7fa77414d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:17:26.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.529+0000 7fa772ffd640 1 --2- 172.21.15.116:0/3454292738 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa774070ca0 0x7fa7740c2590 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7fa76400c4c0 tx=0x7fa764001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:26.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.529+0000 7fa75bfff640 1 -- 172.21.15.116:0/3454292738 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa764017070 con 0x7fa774070ca0 2024-04-25T00:17:26.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.529+0000 7fa75bfff640 1 -- 172.21.15.116:0/3454292738 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa764003600 con 0x7fa774070ca0 2024-04-25T00:17:26.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.529+0000 7fa7799e7640 1 -- 172.21.15.116:0/3454292738 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa774147430 con 0x7fa774070ca0 2024-04-25T00:17:26.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.529+0000 7fa75bfff640 1 -- 172.21.15.116:0/3454292738 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7640039f0 con 0x7fa774070ca0 2024-04-25T00:17:26.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.529+0000 7fa7799e7640 1 -- 172.21.15.116:0/3454292738 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7740c31c0 con 0x7fa774070ca0 2024-04-25T00:17:26.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.529+0000 7fa7799e7640 1 -- 172.21.15.116:0/3454292738 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa774068a60 con 0x7fa774070ca0 2024-04-25T00:17:26.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.529+0000 7fa75bfff640 1 -- 172.21.15.116:0/3454292738 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa76401a070 con 0x7fa774070ca0 2024-04-25T00:17:26.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.529+0000 7fa75bfff640 1 --2- 172.21.15.116:0/3454292738 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa7600678c0 0x7fa760069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:26.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.529+0000 7fa75bfff640 1 -- 172.21.15.116:0/3454292738 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 8730+0+0 (secure 0 0 0) 0x7fa764085c00 con 0x7fa774070ca0 2024-04-25T00:17:26.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.529+0000 7fa7727fc640 1 --2- 172.21.15.116:0/3454292738 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa7600678c0 0x7fa760069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:26.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.533+0000 7fa7727fc640 1 --2- 172.21.15.116:0/3454292738 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa7600678c0 0x7fa760069d80 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fa7680ae000 tx=0x7fa7680a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:26.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.533+0000 7fa75bfff640 1 -- 172.21.15.116:0/3454292738 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fa76408c080 con 0x7fa774070ca0 2024-04-25T00:17:26.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.589+0000 7fd393dd7640 1 -- 172.21.15.116:0/986764001 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd38c068a60 con 0x7fd38c06e6a0 2024-04-25T00:17:26.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.589+0000 7fd3827fc640 1 -- 172.21.15.116:0/986764001 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) ==== 95+0+5672 (secure 0 0 0) 0x7fd37c0555a0 con 0x7fd38c06e6a0 2024-04-25T00:17:26.590 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:17:26.590 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":128,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:16:48.854104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7119},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7119":{"gid":7119,"name":"a","rank":0,"incarnation":123,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6833/3645553116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3645553116},{"type":"v1","addr":"172.21.15.116:6833","nonce":3645553116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7119,"qdb_cluster":[7119]},"id":1},{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:17:25.599032+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7164},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7164":{"gid":7164,"name":"c","rank":0,"incarnation":128,"state":"up:replay","state_seq":1,"addr":"172.21.15.116:6835/3336339376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3336339376},{"type":"v1","addr":"172.21.15.116:6835","nonce":3336339376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:17:26.590 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 128 2024-04-25T00:17:26.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.593+0000 7fd393dd7640 1 -- 172.21.15.116:0/986764001 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd3700678f0 msgr2=0x7fd370069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:26.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.593+0000 7fd393dd7640 1 --2- 172.21.15.116:0/986764001 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd3700678f0 0x7fd370069db0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fd3880ab000 tx=0x7fd3880b0040 comp rx=0 tx=0).stop 2024-04-25T00:17:26.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.593+0000 7fd393dd7640 1 -- 172.21.15.116:0/986764001 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd38c06e6a0 msgr2=0x7fd38c06eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:26.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.593+0000 7fd393dd7640 1 --2- 172.21.15.116:0/986764001 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd38c06e6a0 0x7fd38c06eb20 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7fd37c00c430 tx=0x7fd37c00c8f0 comp rx=0 tx=0).stop 2024-04-25T00:17:26.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.593+0000 7fd393dd7640 1 -- 172.21.15.116:0/986764001 shutdown_connections 2024-04-25T00:17:26.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.593+0000 7fd393dd7640 1 --2- 172.21.15.116:0/986764001 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd3700678f0 0x7fd370069db0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:26.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.593+0000 7fd393dd7640 1 --2- 172.21.15.116:0/986764001 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd38c1436e0 0x7fd38c143b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:26.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.593+0000 7fd393dd7640 1 --2- 172.21.15.116:0/986764001 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd38c06e6a0 0x7fd38c06eb20 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:26.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.593+0000 7fd393dd7640 1 --2- 172.21.15.116:0/986764001 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3780039c0 0x7fd38c1ef040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:26.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.593+0000 7fd393dd7640 1 -- 172.21.15.116:0/986764001 >> 172.21.15.116:0/986764001 conn(0x7fd38c06c720 msgr2=0x7fd38c0a94a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:26.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.593+0000 7fd393dd7640 1 -- 172.21.15.116:0/986764001 shutdown_connections 2024-04-25T00:17:26.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.593+0000 7fd393dd7640 1 -- 172.21.15.116:0/986764001 wait complete. 2024-04-25T00:17:26.608 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:17:26.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.717+0000 7fa7799e7640 1 -- 172.21.15.116:0/3454292738 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa77414d710 con 0x7fa774070ca0 2024-04-25T00:17:26.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.717+0000 7fa75bfff640 1 -- 172.21.15.116:0/3454292738 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) ==== 95+0+5672 (secure 0 0 0) 0x7fa764055bb0 con 0x7fa774070ca0 2024-04-25T00:17:26.717 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:17:26.717 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":128,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:16:48.854104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7119},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7119":{"gid":7119,"name":"a","rank":0,"incarnation":123,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6833/3645553116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3645553116},{"type":"v1","addr":"172.21.15.116:6833","nonce":3645553116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7119,"qdb_cluster":[7119]},"id":1},{"mdsmap":{"epoch":128,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:17:25.599032+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7164},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7164":{"gid":7164,"name":"c","rank":0,"incarnation":128,"state":"up:replay","state_seq":1,"addr":"172.21.15.116:6835/3336339376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3336339376},{"type":"v1","addr":"172.21.15.116:6835","nonce":3336339376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:17:26.717 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 128 2024-04-25T00:17:26.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.717+0000 7fa7799e7640 1 -- 172.21.15.116:0/3454292738 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa7600678c0 msgr2=0x7fa760069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:26.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.717+0000 7fa7799e7640 1 --2- 172.21.15.116:0/3454292738 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa7600678c0 0x7fa760069d80 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fa7680ae000 tx=0x7fa7680a7000 comp rx=0 tx=0).stop 2024-04-25T00:17:26.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.717+0000 7fa7799e7640 1 -- 172.21.15.116:0/3454292738 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa774070ca0 msgr2=0x7fa7740c2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:26.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.717+0000 7fa7799e7640 1 --2- 172.21.15.116:0/3454292738 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa774070ca0 0x7fa7740c2590 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7fa76400c4c0 tx=0x7fa764001e80 comp rx=0 tx=0).stop 2024-04-25T00:17:26.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.717+0000 7fa7799e7640 1 -- 172.21.15.116:0/3454292738 shutdown_connections 2024-04-25T00:17:26.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.717+0000 7fa7799e7640 1 --2- 172.21.15.116:0/3454292738 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa7600678c0 0x7fa760069d80 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:26.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.717+0000 7fa7799e7640 1 --2- 172.21.15.116:0/3454292738 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa774154a60 0x7fa77414d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:26.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.717+0000 7fa7799e7640 1 --2- 172.21.15.116:0/3454292738 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa774149f60 0x7fa77414cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:26.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.717+0000 7fa7799e7640 1 --2- 172.21.15.116:0/3454292738 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa774070ca0 0x7fa7740c2590 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:26.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.717+0000 7fa7799e7640 1 -- 172.21.15.116:0/3454292738 >> 172.21.15.116:0/3454292738 conn(0x7fa774074a80 msgr2=0x7fa774151b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:26.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.717+0000 7fa7799e7640 1 -- 172.21.15.116:0/3454292738 shutdown_connections 2024-04-25T00:17:26.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:26.717+0000 7fa7799e7640 1 -- 172.21.15.116:0/3454292738 wait complete. 2024-04-25T00:17:26.734 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 128, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:04.678737+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 126, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_7119": { "addr": "172.21.15.116:6833/3645553116", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6832", "nonce": 3645553116, "type": "v2" }, { "addr": "172.21.15.116:6833", "nonce": 3645553116, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7119, "incarnation": 123, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 5 } }, "last_failure": 0, "last_failure_osd_epoch": 51, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T00:16:48.854104+0000", "qdb_cluster": [ 7119 ], "qdb_leader": 7119, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7119 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:18.825887+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 128, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_7164": { "addr": "172.21.15.116:6835/3336339376", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6834", "nonce": 3336339376, "type": "v2" }, { "addr": "172.21.15.116:6835", "nonce": 3336339376, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7164, "incarnation": 128, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 52, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-04-25T00:17:25.599032+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7164 } } } ], "standbys": [ { "addr": "172.21.15.174:6835/4112145782", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6834", "nonce": 4112145782, "type": "v2" }, { "addr": "172.21.15.174:6835", "nonce": 4112145782, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 112, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5582, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-25T00:17:28.609 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:17:28.735 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:17:28.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.741+0000 7f9046bf1640 1 Processor -- start 2024-04-25T00:17:28.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.741+0000 7f9046bf1640 1 -- start start 2024-04-25T00:17:28.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.741+0000 7f9046bf1640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f904014d690 0x7f90400c5010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:28.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.741+0000 7f9046bf1640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9040073200 0x7f9040156c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:28.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.741+0000 7f9046bf1640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f90400c5550 0x7f90401571d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:28.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.741+0000 7f9046bf1640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f90400c2b10 con 0x7f904014d690 2024-04-25T00:17:28.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.741+0000 7f9046bf1640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f904014da70 con 0x7f90400c5550 2024-04-25T00:17:28.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.741+0000 7f9046bf1640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f9040154150 con 0x7f9040073200 2024-04-25T00:17:28.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.741+0000 7f904498d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f904014d690 0x7f90400c5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:28.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.741+0000 7f904498d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f904014d690 0x7f90400c5010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42350/0 (socket says 172.21.15.116:42350) 2024-04-25T00:17:28.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.741+0000 7f904498d640 1 -- 172.21.15.116:0/1442100252 learned_addr learned my addr 172.21.15.116:0/1442100252 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:17:28.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.741+0000 7f904518e640 1 --2- 172.21.15.116:0/1442100252 >> v2:172.21.15.174:3300/0 conn(0x7f90400c5550 0x7f90401571d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:28.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.741+0000 7f903ffff640 1 --2- 172.21.15.116:0/1442100252 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9040073200 0x7f9040156c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:28.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.741+0000 7f904498d640 1 -- 172.21.15.116:0/1442100252 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9040073200 msgr2=0x7f9040156c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:28.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.741+0000 7f904498d640 1 --2- 172.21.15.116:0/1442100252 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9040073200 0x7f9040156c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:28.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.741+0000 7f904498d640 1 -- 172.21.15.116:0/1442100252 >> v2:172.21.15.174:3300/0 conn(0x7f90400c5550 msgr2=0x7f90401571d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:28.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.741+0000 7f904498d640 1 --2- 172.21.15.116:0/1442100252 >> v2:172.21.15.174:3300/0 conn(0x7f90400c5550 0x7f90401571d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:28.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.741+0000 7f904498d640 1 -- 172.21.15.116:0/1442100252 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90401543f0 con 0x7f904014d690 2024-04-25T00:17:28.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.745+0000 7f904498d640 1 --2- 172.21.15.116:0/1442100252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f904014d690 0x7f90400c5010 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f9040157c80 tx=0x7f903401c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58cde2bd868be4f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:28.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.745+0000 7f903f7fe640 1 -- 172.21.15.116:0/1442100252 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90340b0070 con 0x7f904014d690 2024-04-25T00:17:28.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.745+0000 7f903f7fe640 1 -- 172.21.15.116:0/1442100252 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90340ab600 con 0x7f904014d690 2024-04-25T00:17:28.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.745+0000 7f903f7fe640 1 -- 172.21.15.116:0/1442100252 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90340ab9d0 con 0x7f904014d690 2024-04-25T00:17:28.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.745+0000 7f9046bf1640 1 -- 172.21.15.116:0/1442100252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f904014d690 msgr2=0x7f90400c5010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:28.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.745+0000 7f9046bf1640 1 --2- 172.21.15.116:0/1442100252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f904014d690 0x7f90400c5010 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f9040157c80 tx=0x7f903401c3d0 comp rx=0 tx=0).stop 2024-04-25T00:17:28.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.745+0000 7f9046bf1640 1 -- 172.21.15.116:0/1442100252 shutdown_connections 2024-04-25T00:17:28.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.745+0000 7f9046bf1640 1 --2- 172.21.15.116:0/1442100252 >> v2:172.21.15.174:3300/0 conn(0x7f90400c5550 0x7f90401571d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:28.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.745+0000 7f9046bf1640 1 --2- 172.21.15.116:0/1442100252 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9040073200 0x7f9040156c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:28.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.745+0000 7f9046bf1640 1 --2- 172.21.15.116:0/1442100252 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f904014d690 0x7f90400c5010 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:28.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.745+0000 7f9046bf1640 1 -- 172.21.15.116:0/1442100252 >> 172.21.15.116:0/1442100252 conn(0x7f904006c720 msgr2=0x7f904006cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:28.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.745+0000 7f9046bf1640 1 -- 172.21.15.116:0/1442100252 shutdown_connections 2024-04-25T00:17:28.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.745+0000 7f9046bf1640 1 -- 172.21.15.116:0/1442100252 wait complete. 2024-04-25T00:17:28.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.745+0000 7f9046bf1640 1 Processor -- start 2024-04-25T00:17:28.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.745+0000 7f9046bf1640 1 -- start start 2024-04-25T00:17:28.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.745+0000 7f9046bf1640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9040073200 0x7f90401ee890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:28.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.745+0000 7f9046bf1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90400c5550 0x7f90401f24c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:28.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.745+0000 7f9046bf1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f904014d690 0x7f9040142c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:28.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.745+0000 7f9046bf1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f904006bba0 con 0x7f90400c5550 2024-04-25T00:17:28.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.745+0000 7f9046bf1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f904006ba20 con 0x7f904014d690 2024-04-25T00:17:28.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.745+0000 7f9046bf1640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f904006bd20 con 0x7f9040073200 2024-04-25T00:17:28.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.745+0000 7f903ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90400c5550 0x7f90401f24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:28.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.745+0000 7f903ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90400c5550 0x7f90401f24c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42364/0 (socket says 172.21.15.116:42364) 2024-04-25T00:17:28.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.745+0000 7f903ffff640 1 -- 172.21.15.116:0/4281965125 learned_addr learned my addr 172.21.15.116:0/4281965125 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:17:28.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.745+0000 7f904518e640 1 --2- 172.21.15.116:0/4281965125 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f904014d690 0x7f9040142c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:28.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.745+0000 7f904498d640 1 --2- 172.21.15.116:0/4281965125 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9040073200 0x7f90401ee890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:28.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.745+0000 7f903ffff640 1 -- 172.21.15.116:0/4281965125 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9040073200 msgr2=0x7f90401ee890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:28.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.745+0000 7f903ffff640 1 --2- 172.21.15.116:0/4281965125 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9040073200 0x7f90401ee890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:28.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.745+0000 7f903ffff640 1 -- 172.21.15.116:0/4281965125 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f904014d690 msgr2=0x7f9040142c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:28.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.745+0000 7f903ffff640 1 --2- 172.21.15.116:0/4281965125 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f904014d690 0x7f9040142c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:28.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.745+0000 7f903ffff640 1 -- 172.21.15.116:0/4281965125 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9040156230 con 0x7f90400c5550 2024-04-25T00:17:28.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.745+0000 7f903ffff640 1 --2- 172.21.15.116:0/4281965125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90400c5550 0x7f90401f24c0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f902c00d800 tx=0x7f902c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:28.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.749+0000 7f903d7fa640 1 -- 172.21.15.116:0/4281965125 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f902c00b6c0 con 0x7f90400c5550 2024-04-25T00:17:28.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.749+0000 7f903d7fa640 1 -- 172.21.15.116:0/4281965125 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f902c019070 con 0x7f90400c5550 2024-04-25T00:17:28.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.749+0000 7f903d7fa640 1 -- 172.21.15.116:0/4281965125 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f902c00bac0 con 0x7f90400c5550 2024-04-25T00:17:28.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.749+0000 7f9046bf1640 1 -- 172.21.15.116:0/4281965125 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9040156510 con 0x7f90400c5550 2024-04-25T00:17:28.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.749+0000 7f9046bf1640 1 -- 172.21.15.116:0/4281965125 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90401f1fa0 con 0x7f90400c5550 2024-04-25T00:17:28.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.749+0000 7f9046bf1640 1 -- 172.21.15.116:0/4281965125 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90401f21a0 con 0x7f90400c5550 2024-04-25T00:17:28.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.749+0000 7f903d7fa640 1 -- 172.21.15.116:0/4281965125 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f902c00bc60 con 0x7f90400c5550 2024-04-25T00:17:28.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.749+0000 7f903d7fa640 1 --2- 172.21.15.116:0/4281965125 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f90240678c0 0x7f9024069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:28.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.749+0000 7f903d7fa640 1 -- 172.21.15.116:0/4281965125 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 8730+0+0 (secure 0 0 0) 0x7f902c086d10 con 0x7f90400c5550 2024-04-25T00:17:28.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.749+0000 7f904498d640 1 --2- 172.21.15.116:0/4281965125 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f90240678c0 0x7f9024069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:28.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.753+0000 7f904498d640 1 --2- 172.21.15.116:0/4281965125 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f90240678c0 0x7f9024069d80 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f90340ad000 tx=0x7f90340a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:28.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.753+0000 7f903d7fa640 1 -- 172.21.15.116:0/4281965125 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f90401f21a0 con 0x7f90400c5550 2024-04-25T00:17:28.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.849+0000 7f9247d89640 1 Processor -- start 2024-04-25T00:17:28.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.849+0000 7f9247d89640 1 -- start start 2024-04-25T00:17:28.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.849+0000 7f9247d89640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9240152d80 0x7f9240159950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:28.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.849+0000 7f9247d89640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f9240072080 0x7f92401562a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:28.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.849+0000 7f9247d89640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92400730b0 con 0x7f92400c36b0 2024-04-25T00:17:28.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.849+0000 7f9247d89640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f92401567e0 con 0x7f9240072080 2024-04-25T00:17:28.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.849+0000 7f9247d89640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f9240072e60 con 0x7f9240152d80 2024-04-25T00:17:28.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.849+0000 7f9246326640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f92400c36b0 0x7f92400c5020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:39648/0 (socket says 172.21.15.116:39648) 2024-04-25T00:17:28.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.849+0000 7f9246326640 1 -- 172.21.15.116:0/3830244127 learned_addr learned my addr 172.21.15.116:0/3830244127 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:17:28.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.853+0000 7f9245b25640 1 --2- 172.21.15.116:0/3830244127 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9240152d80 0x7f9240159950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:28.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.853+0000 7f9244b23640 1 -- 172.21.15.116:0/3830244127 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3034108595 0 0) 0x7f92400730b0 con 0x7f92400c36b0 2024-04-25T00:17:28.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.853+0000 7f9244b23640 1 -- 172.21.15.116:0/3830244127 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f922c003530 con 0x7f92400c36b0 2024-04-25T00:17:28.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.853+0000 7f9245324640 1 --2- 172.21.15.116:0/3830244127 >> v2:172.21.15.174:3300/0 conn(0x7f9240072080 0x7f92401562a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:28.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.853+0000 7f9244b23640 1 -- 172.21.15.116:0/3830244127 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1264037263 0 0) 0x7f922c003530 con 0x7f92400c36b0 2024-04-25T00:17:28.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.853+0000 7f9244b23640 1 -- 172.21.15.116:0/3830244127 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9240152d80 msgr2=0x7f9240159950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:28.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.853+0000 7f9244b23640 1 --2- 172.21.15.116:0/3830244127 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9240152d80 0x7f9240159950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:28.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.853+0000 7f9244b23640 1 -- 172.21.15.116:0/3830244127 >> v2:172.21.15.174:3300/0 conn(0x7f9240072080 msgr2=0x7f92401562a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:28.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.853+0000 7f9244b23640 1 --2- 172.21.15.116:0/3830244127 >> v2:172.21.15.174:3300/0 conn(0x7f9240072080 0x7f92401562a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:28.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.853+0000 7f9244b23640 1 -- 172.21.15.116:0/3830244127 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9240073390 con 0x7f92400c36b0 2024-04-25T00:17:28.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.853+0000 7f9244b23640 1 -- 172.21.15.116:0/3830244127 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f9238001a00 con 0x7f92400c36b0 2024-04-25T00:17:28.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.853+0000 7f9244b23640 1 -- 172.21.15.116:0/3830244127 >> v1:172.21.15.116:6789/0 conn(0x7f92400c36b0 legacy=0x7f92400c5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:28.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.853+0000 7f9244b23640 1 --2- 172.21.15.116:0/3830244127 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f922c0039c0 0x7f922c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:28.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.853+0000 7f9244b23640 1 --2- 172.21.15.116:0/3830244127 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9240152d80 0x7f922c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:28.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.853+0000 7f9244b23640 1 --2- 172.21.15.116:0/3830244127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f922c004d50 0x7f922c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:28.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.853+0000 7f9244b23640 1 -- 172.21.15.116:0/3830244127 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f92401567e0 con 0x7f922c004d50 2024-04-25T00:17:28.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.853+0000 7f9244b23640 1 -- 172.21.15.116:0/3830244127 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9240072e60 con 0x7f922c0039c0 2024-04-25T00:17:28.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.853+0000 7f9244b23640 1 -- 172.21.15.116:0/3830244127 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f922c00ba80 con 0x7f9240152d80 2024-04-25T00:17:28.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.853+0000 7f9246326640 1 --2- 172.21.15.116:0/3830244127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f922c004d50 0x7f922c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:28.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.853+0000 7f9246326640 1 -- 172.21.15.116:0/3830244127 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9240152d80 msgr2=0x7f922c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:28.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.853+0000 7f9246326640 1 --2- 172.21.15.116:0/3830244127 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9240152d80 0x7f922c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:28.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.853+0000 7f9246326640 1 -- 172.21.15.116:0/3830244127 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f922c0039c0 msgr2=0x7f922c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:28.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.853+0000 7f9246326640 1 --2- 172.21.15.116:0/3830244127 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f922c0039c0 0x7f922c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:28.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.853+0000 7f9246326640 1 -- 172.21.15.116:0/3830244127 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f922c003530 con 0x7f922c004d50 2024-04-25T00:17:28.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.853+0000 7f9246326640 1 --2- 172.21.15.116:0/3830244127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f922c004d50 0x7f922c0051d0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f923800a880 tx=0x7f923801fb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8008616acc47f728 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:28.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.853+0000 7f9244b23640 1 -- 172.21.15.116:0/3830244127 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f923800a4a0 con 0x7f922c004d50 2024-04-25T00:17:28.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.853+0000 7f9244b23640 1 -- 172.21.15.116:0/3830244127 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92380ad020 con 0x7f922c004d50 2024-04-25T00:17:28.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.853+0000 7f9244b23640 1 -- 172.21.15.116:0/3830244127 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92380a9740 con 0x7f922c004d50 2024-04-25T00:17:28.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.857+0000 7f9247d89640 1 -- 172.21.15.116:0/3830244127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f922c004d50 msgr2=0x7f922c0051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:28.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.857+0000 7f9247d89640 1 --2- 172.21.15.116:0/3830244127 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f922c004d50 0x7f922c0051d0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f923800a880 tx=0x7f923801fb00 comp rx=0 tx=0).stop 2024-04-25T00:17:28.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.857+0000 7f9246326640 1 -- 172.21.15.116:0/3830244127 reap_dead start 2024-04-25T00:17:28.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.857+0000 7f9247d89640 1 -- 172.21.15.116:0/3830244127 shutdown_connections 2024-04-25T00:17:28.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.857+0000 7f9247d89640 1 -- 172.21.15.116:0/3830244127 >> 172.21.15.116:0/3830244127 conn(0x7f924006c720 msgr2=0x7f924006cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:28.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.857+0000 7f9247d89640 1 -- 172.21.15.116:0/3830244127 shutdown_connections 2024-04-25T00:17:28.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.857+0000 7f9247d89640 1 -- 172.21.15.116:0/3830244127 wait complete. 2024-04-25T00:17:28.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.857+0000 7f9247d89640 1 Processor -- start 2024-04-25T00:17:28.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.857+0000 7f9247d89640 1 -- start start 2024-04-25T00:17:28.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.857+0000 7f9247d89640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9240157240 0x7f9240149e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:28.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.857+0000 7f9247d89640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9240158730 0x7f924014a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:28.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.857+0000 7f9247d89640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92401ee530 0x7f92401ee9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:28.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.857+0000 7f9247d89640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9240159430 con 0x7f9240158730 2024-04-25T00:17:28.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.857+0000 7f9247d89640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9240159150 con 0x7f92401ee530 2024-04-25T00:17:28.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.857+0000 7f9247d89640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f92400c5730 con 0x7f9240157240 2024-04-25T00:17:28.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.857+0000 7f9245324640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9240158730 0x7f924014a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:28.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.857+0000 7f9245324640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9240158730 0x7f924014a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42382/0 (socket says 172.21.15.116:42382) 2024-04-25T00:17:28.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.857+0000 7f9245324640 1 -- 172.21.15.116:0/2863430942 learned_addr learned my addr 172.21.15.116:0/2863430942 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:17:28.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.857+0000 7f9246326640 1 --2- 172.21.15.116:0/2863430942 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92401ee530 0x7f92401ee9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:28.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.857+0000 7f9245b25640 1 --2- 172.21.15.116:0/2863430942 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9240157240 0x7f9240149e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:28.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.857+0000 7f9245324640 1 -- 172.21.15.116:0/2863430942 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9240157240 msgr2=0x7f9240149e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:28.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.857+0000 7f9245324640 1 --2- 172.21.15.116:0/2863430942 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9240157240 0x7f9240149e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:28.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.857+0000 7f9245324640 1 -- 172.21.15.116:0/2863430942 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92401ee530 msgr2=0x7f92401ee9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:28.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.857+0000 7f9245324640 1 --2- 172.21.15.116:0/2863430942 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92401ee530 0x7f92401ee9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:28.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.857+0000 7f9245324640 1 -- 172.21.15.116:0/2863430942 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f924014b1a0 con 0x7f9240158730 2024-04-25T00:17:28.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.857+0000 7f9246326640 1 --2- 172.21.15.116:0/2863430942 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92401ee530 0x7f92401ee9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:17:28.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.857+0000 7f9245324640 1 --2- 172.21.15.116:0/2863430942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9240158730 0x7f924014a360 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f923000d610 tx=0x7f923000dad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:28.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.857+0000 7f92367fc640 1 -- 172.21.15.116:0/2863430942 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92300125f0 con 0x7f9240158730 2024-04-25T00:17:28.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.857+0000 7f92367fc640 1 -- 172.21.15.116:0/2863430942 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9230019070 con 0x7f9240158730 2024-04-25T00:17:28.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.857+0000 7f9247d89640 1 -- 172.21.15.116:0/2863430942 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f924014ad00 con 0x7f9240158730 2024-04-25T00:17:28.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.857+0000 7f92367fc640 1 -- 172.21.15.116:0/2863430942 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92300129f0 con 0x7f9240158730 2024-04-25T00:17:28.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.857+0000 7f9247d89640 1 -- 172.21.15.116:0/2863430942 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f924014f800 con 0x7f9240158730 2024-04-25T00:17:28.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.861+0000 7f9223fff640 1 -- 172.21.15.116:0/2863430942 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f921c0052a0 con 0x7f9240158730 2024-04-25T00:17:28.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.861+0000 7f92367fc640 1 -- 172.21.15.116:0/2863430942 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f9230012b90 con 0x7f9240158730 2024-04-25T00:17:28.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.861+0000 7f92367fc640 1 --2- 172.21.15.116:0/2863430942 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f92240678c0 0x7f9224069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:28.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.861+0000 7f92367fc640 1 -- 172.21.15.116:0/2863430942 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 8730+0+0 (secure 0 0 0) 0x7f9230086ba0 con 0x7f9240158730 2024-04-25T00:17:28.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.861+0000 7f9245b25640 1 --2- 172.21.15.116:0/2863430942 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f92240678c0 0x7f9224069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:28.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.861+0000 7f9245b25640 1 --2- 172.21.15.116:0/2863430942 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f92240678c0 0x7f9224069d80 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f923c0047c0 tx=0x7f923c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:28.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.865+0000 7f92367fc640 1 -- 172.21.15.116:0/2863430942 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f9230056b50 con 0x7f9240158730 2024-04-25T00:17:28.885 INFO:tasks.ceph.mds.c.smithi116.stderr:2024-04-25T00:17:28.885+0000 7ffa2e6f1640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T00:17:28.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.945+0000 7f9046bf1640 1 -- 172.21.15.116:0/4281965125 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9040068a60 con 0x7f90400c5550 2024-04-25T00:17:28.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.945+0000 7f903d7fa640 1 -- 172.21.15.116:0/4281965125 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) ==== 95+0+5680 (secure 0 0 0) 0x7f902c056cc0 con 0x7f90400c5550 2024-04-25T00:17:28.947 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:17:28.947 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":131,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:16:48.854104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7119},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7119":{"gid":7119,"name":"a","rank":0,"incarnation":123,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6833/3645553116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3645553116},{"type":"v1","addr":"172.21.15.116:6833","nonce":3645553116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7119,"qdb_cluster":[7119]},"id":1},{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:17:28.837517+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7164},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7164":{"gid":7164,"name":"c","rank":0,"incarnation":128,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6835/3336339376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3336339376},{"type":"v1","addr":"172.21.15.116:6835","nonce":3336339376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7164,"qdb_cluster":[7164]},"id":2}]} 2024-04-25T00:17:28.947 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 131 2024-04-25T00:17:28.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.949+0000 7f9046bf1640 1 -- 172.21.15.116:0/4281965125 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f90240678c0 msgr2=0x7f9024069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:28.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.949+0000 7f9046bf1640 1 --2- 172.21.15.116:0/4281965125 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f90240678c0 0x7f9024069d80 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f90340ad000 tx=0x7f90340a6000 comp rx=0 tx=0).stop 2024-04-25T00:17:28.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.949+0000 7f9046bf1640 1 -- 172.21.15.116:0/4281965125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90400c5550 msgr2=0x7f90401f24c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:28.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.949+0000 7f9046bf1640 1 --2- 172.21.15.116:0/4281965125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90400c5550 0x7f90401f24c0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f902c00d800 tx=0x7f902c00dcc0 comp rx=0 tx=0).stop 2024-04-25T00:17:28.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.949+0000 7f9046bf1640 1 -- 172.21.15.116:0/4281965125 shutdown_connections 2024-04-25T00:17:28.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.949+0000 7f9046bf1640 1 --2- 172.21.15.116:0/4281965125 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f90240678c0 0x7f9024069d80 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:28.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.949+0000 7f9046bf1640 1 --2- 172.21.15.116:0/4281965125 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f904014d690 0x7f9040142c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:28.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.949+0000 7f9046bf1640 1 --2- 172.21.15.116:0/4281965125 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90400c5550 0x7f90401f24c0 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:28.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.949+0000 7f9046bf1640 1 --2- 172.21.15.116:0/4281965125 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9040073200 0x7f90401ee890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:28.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.949+0000 7f9046bf1640 1 -- 172.21.15.116:0/4281965125 >> 172.21.15.116:0/4281965125 conn(0x7f904006c720 msgr2=0x7f9040146f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:28.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.949+0000 7f9046bf1640 1 -- 172.21.15.116:0/4281965125 shutdown_connections 2024-04-25T00:17:28.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:28.949+0000 7f9046bf1640 1 -- 172.21.15.116:0/4281965125 wait complete. 2024-04-25T00:17:28.964 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:17:29.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:29.061+0000 7f9223fff640 1 -- 172.21.15.116:0/2863430942 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f921c005590 con 0x7f9240158730 2024-04-25T00:17:29.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:29.061+0000 7f92367fc640 1 -- 172.21.15.116:0/2863430942 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) ==== 95+0+5680 (secure 0 0 0) 0x7f9230056450 con 0x7f9240158730 2024-04-25T00:17:29.061 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:17:29.061 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":131,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:16:48.854104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7119},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7119":{"gid":7119,"name":"a","rank":0,"incarnation":123,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6833/3645553116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3645553116},{"type":"v1","addr":"172.21.15.116:6833","nonce":3645553116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7119,"qdb_cluster":[7119]},"id":1},{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:17:28.837517+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7164},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7164":{"gid":7164,"name":"c","rank":0,"incarnation":128,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6835/3336339376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3336339376},{"type":"v1","addr":"172.21.15.116:6835","nonce":3336339376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7164,"qdb_cluster":[7164]},"id":2}]} 2024-04-25T00:17:29.062 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 131 2024-04-25T00:17:29.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:29.061+0000 7f9223fff640 1 -- 172.21.15.116:0/2863430942 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f92240678c0 msgr2=0x7f9224069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:29.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:29.061+0000 7f9223fff640 1 --2- 172.21.15.116:0/2863430942 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f92240678c0 0x7f9224069d80 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f923c0047c0 tx=0x7f923c008040 comp rx=0 tx=0).stop 2024-04-25T00:17:29.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:29.061+0000 7f9223fff640 1 -- 172.21.15.116:0/2863430942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9240158730 msgr2=0x7f924014a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:29.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:29.061+0000 7f9223fff640 1 --2- 172.21.15.116:0/2863430942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9240158730 0x7f924014a360 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f923000d610 tx=0x7f923000dad0 comp rx=0 tx=0).stop 2024-04-25T00:17:29.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:29.061+0000 7f9223fff640 1 -- 172.21.15.116:0/2863430942 shutdown_connections 2024-04-25T00:17:29.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:29.061+0000 7f9223fff640 1 --2- 172.21.15.116:0/2863430942 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f92240678c0 0x7f9224069d80 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:29.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:29.061+0000 7f9223fff640 1 --2- 172.21.15.116:0/2863430942 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92401ee530 0x7f92401ee9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:29.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:29.061+0000 7f9223fff640 1 --2- 172.21.15.116:0/2863430942 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9240158730 0x7f924014a360 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:29.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:29.061+0000 7f9223fff640 1 --2- 172.21.15.116:0/2863430942 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9240157240 0x7f9240149e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:29.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:29.061+0000 7f9223fff640 1 -- 172.21.15.116:0/2863430942 >> 172.21.15.116:0/2863430942 conn(0x7f924006c720 msgr2=0x7f92400c3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:29.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:29.065+0000 7f9223fff640 1 -- 172.21.15.116:0/2863430942 shutdown_connections 2024-04-25T00:17:29.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:29.065+0000 7f9223fff640 1 -- 172.21.15.116:0/2863430942 wait complete. 2024-04-25T00:17:29.079 INFO:tasks.mds_thrash.fs.[b]:mds.c has gained rank=0, replacing gid=5621 2024-04-25T00:17:29.080 INFO:tasks.mds_thrash.fs.[b]:waiting for 4 secs before reviving mds.d 2024-04-25T00:17:30.965 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:17:31.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.093+0000 7feeee873640 1 Processor -- start 2024-04-25T00:17:31.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.097+0000 7feeee873640 1 -- start start 2024-04-25T00:17:31.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.097+0000 7feeee873640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7feee814c720 0x7feee81563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:31.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.097+0000 7feeee873640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feee8071220 con 0x7feee8071680 2024-04-25T00:17:31.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.097+0000 7feeee873640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feee8152420 con 0x7feee80a94a0 2024-04-25T00:17:31.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.097+0000 7feeee873640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7feee8153460 con 0x7feee814c720 2024-04-25T00:17:31.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.097+0000 7feeece10640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7feee8071680 0x7feee8071ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:39650/0 (socket says 172.21.15.116:39650) 2024-04-25T00:17:31.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.097+0000 7feeece10640 1 -- 172.21.15.116:0/266739666 learned_addr learned my addr 172.21.15.116:0/266739666 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:17:31.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.097+0000 7feee7fff640 1 --2- 172.21.15.116:0/266739666 >> v2:172.21.15.174:3300/0 conn(0x7feee814c720 0x7feee81563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:31.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.097+0000 7feee77fe640 1 -- 172.21.15.116:0/266739666 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4197684931 0 0) 0x7feee8071220 con 0x7feee8071680 2024-04-25T00:17:31.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.097+0000 7feee77fe640 1 -- 172.21.15.116:0/266739666 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feed4003530 con 0x7feee8071680 2024-04-25T00:17:31.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.097+0000 7feee77fe640 1 -- 172.21.15.116:0/266739666 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 614902043 0 0) 0x7feed4003530 con 0x7feee8071680 2024-04-25T00:17:31.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.097+0000 7feee77fe640 1 -- 172.21.15.116:0/266739666 >> v2:172.21.15.174:3300/0 conn(0x7feee814c720 msgr2=0x7feee81563d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:31.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.097+0000 7feee77fe640 1 --2- 172.21.15.116:0/266739666 >> v2:172.21.15.174:3300/0 conn(0x7feee814c720 0x7feee81563d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:31.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.097+0000 7feee77fe640 1 -- 172.21.15.116:0/266739666 >> v1:172.21.15.174:6789/0 conn(0x7feee80a94a0 legacy=0x7feee80c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:31.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.097+0000 7feee77fe640 1 -- 172.21.15.116:0/266739666 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feee80721c0 con 0x7feee8071680 2024-04-25T00:17:31.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.097+0000 7feee77fe640 1 -- 172.21.15.116:0/266739666 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7feee0001a00 con 0x7feee8071680 2024-04-25T00:17:31.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.097+0000 7feee77fe640 1 -- 172.21.15.116:0/266739666 >> v1:172.21.15.116:6789/0 conn(0x7feee8071680 legacy=0x7feee8071ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:31.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.097+0000 7feee77fe640 1 --2- 172.21.15.116:0/266739666 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7feed40039c0 0x7feed4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:31.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.097+0000 7feee77fe640 1 --2- 172.21.15.116:0/266739666 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feed4004810 0x7feed4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:31.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.097+0000 7feee77fe640 1 --2- 172.21.15.116:0/266739666 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feed40051d0 0x7feed400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:31.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.097+0000 7feee77fe640 1 -- 172.21.15.116:0/266739666 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7feee8153460 con 0x7feed40051d0 2024-04-25T00:17:31.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.097+0000 7feee77fe640 1 -- 172.21.15.116:0/266739666 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7feed400c170 con 0x7feed4004810 2024-04-25T00:17:31.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.097+0000 7feee77fe640 1 -- 172.21.15.116:0/266739666 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7feed400e580 con 0x7feed40039c0 2024-04-25T00:17:31.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.097+0000 7feeece10640 1 --2- 172.21.15.116:0/266739666 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feed40051d0 0x7feed400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:31.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.097+0000 7feee7fff640 1 --2- 172.21.15.116:0/266739666 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7feed40039c0 0x7feed4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:31.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.097+0000 7feedf7fe640 1 --2- 172.21.15.116:0/266739666 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feed4004810 0x7feed4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:31.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.097+0000 7feeece10640 1 -- 172.21.15.116:0/266739666 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7feed40039c0 msgr2=0x7feed4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:31.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.097+0000 7feeece10640 1 --2- 172.21.15.116:0/266739666 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7feed40039c0 0x7feed4003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:31.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.097+0000 7feeece10640 1 -- 172.21.15.116:0/266739666 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feed4004810 msgr2=0x7feed4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:31.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.097+0000 7feeece10640 1 --2- 172.21.15.116:0/266739666 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feed4004810 0x7feed4004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:31.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.097+0000 7feeece10640 1 -- 172.21.15.116:0/266739666 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feed4003530 con 0x7feed40051d0 2024-04-25T00:17:31.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.097+0000 7feedf7fe640 1 --2- 172.21.15.116:0/266739666 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feed4004810 0x7feed4004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:17:31.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.101+0000 7feeece10640 1 -- 172.21.15.116:0/266739666 reap_dead start 2024-04-25T00:17:31.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.101+0000 7feeece10640 1 --2- 172.21.15.116:0/266739666 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feed40051d0 0x7feed400b9d0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7feee000a850 tx=0x7feee001fb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8df9764531a3dc1e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:31.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.101+0000 7feee77fe640 1 -- 172.21.15.116:0/266739666 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee000a4a0 con 0x7feed40051d0 2024-04-25T00:17:31.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.101+0000 7feee77fe640 1 -- 172.21.15.116:0/266739666 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feee00ad070 con 0x7feed40051d0 2024-04-25T00:17:31.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.101+0000 7feee77fe640 1 -- 172.21.15.116:0/266739666 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee00a9980 con 0x7feed40051d0 2024-04-25T00:17:31.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.101+0000 7feeee873640 1 -- 172.21.15.116:0/266739666 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feed40051d0 msgr2=0x7feed400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:31.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.101+0000 7feeee873640 1 --2- 172.21.15.116:0/266739666 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feed40051d0 0x7feed400b9d0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7feee000a850 tx=0x7feee001fb00 comp rx=0 tx=0).stop 2024-04-25T00:17:31.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.101+0000 7feeee873640 1 -- 172.21.15.116:0/266739666 shutdown_connections 2024-04-25T00:17:31.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.101+0000 7feeee873640 1 --2- 172.21.15.116:0/266739666 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feed40051d0 0x7feed400b9d0 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:31.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.101+0000 7feeee873640 1 -- 172.21.15.116:0/266739666 >> 172.21.15.116:0/266739666 conn(0x7feee80c3b00 msgr2=0x7feee80c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:31.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.101+0000 7feeee873640 1 -- 172.21.15.116:0/266739666 shutdown_connections 2024-04-25T00:17:31.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.101+0000 7feeee873640 1 -- 172.21.15.116:0/266739666 wait complete. 2024-04-25T00:17:31.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.101+0000 7feeee873640 1 Processor -- start 2024-04-25T00:17:31.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.101+0000 7feeee873640 1 -- start start 2024-04-25T00:17:31.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.101+0000 7feeee873640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7feed40051d0 0x7feee80c0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:31.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.101+0000 7feeee873640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feee80c52e0 0x7feee8142a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:31.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.101+0000 7feeee873640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feee80c5760 0x7feee8142f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:31.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.101+0000 7feeee873640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7feee8152980 con 0x7feee80c5760 2024-04-25T00:17:31.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.101+0000 7feeee873640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7feee81526a0 con 0x7feee80c52e0 2024-04-25T00:17:31.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.101+0000 7feeee873640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7feee8152c60 con 0x7feed40051d0 2024-04-25T00:17:31.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.101+0000 7feeece10640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feee80c5760 0x7feee8142f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:31.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.105+0000 7feeece10640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feee80c5760 0x7feee8142f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42392/0 (socket says 172.21.15.116:42392) 2024-04-25T00:17:31.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.105+0000 7feeece10640 1 -- 172.21.15.116:0/1104923780 learned_addr learned my addr 172.21.15.116:0/1104923780 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:17:31.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.105+0000 7feee7fff640 1 --2- 172.21.15.116:0/1104923780 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7feed40051d0 0x7feee80c0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:31.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.105+0000 7feedf7fe640 1 --2- 172.21.15.116:0/1104923780 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feee80c52e0 0x7feee8142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:31.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.105+0000 7feeece10640 1 -- 172.21.15.116:0/1104923780 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7feed40051d0 msgr2=0x7feee80c0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:31.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.105+0000 7feeece10640 1 --2- 172.21.15.116:0/1104923780 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7feed40051d0 0x7feee80c0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:31.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.105+0000 7feeece10640 1 -- 172.21.15.116:0/1104923780 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feee80c52e0 msgr2=0x7feee8142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:31.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.105+0000 7feeece10640 1 --2- 172.21.15.116:0/1104923780 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feee80c52e0 0x7feee8142a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:31.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.105+0000 7feeece10640 1 -- 172.21.15.116:0/1104923780 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feee81ebb60 con 0x7feee80c5760 2024-04-25T00:17:31.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.105+0000 7feeece10640 1 --2- 172.21.15.116:0/1104923780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feee80c5760 0x7feee8142f60 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7feee001f930 tx=0x7feed4005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:31.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.105+0000 7feee57fa640 1 -- 172.21.15.116:0/1104923780 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee0001260 con 0x7feee80c5760 2024-04-25T00:17:31.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.105+0000 7feee57fa640 1 -- 172.21.15.116:0/1104923780 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feee000a480 con 0x7feee80c5760 2024-04-25T00:17:31.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.105+0000 7feeee873640 1 -- 172.21.15.116:0/1104923780 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feee80c61f0 con 0x7feee80c5760 2024-04-25T00:17:31.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.105+0000 7feee57fa640 1 -- 172.21.15.116:0/1104923780 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee00a9720 con 0x7feee80c5760 2024-04-25T00:17:31.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.105+0000 7feeee873640 1 -- 172.21.15.116:0/1104923780 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feee80c67b0 con 0x7feee80c5760 2024-04-25T00:17:31.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.105+0000 7feeee873640 1 -- 172.21.15.116:0/1104923780 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feee8068350 con 0x7feee80c5760 2024-04-25T00:17:31.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.105+0000 7feee57fa640 1 -- 172.21.15.116:0/1104923780 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7feee00af030 con 0x7feee80c5760 2024-04-25T00:17:31.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.105+0000 7feee57fa640 1 --2- 172.21.15.116:0/1104923780 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7feec8067a90 0x7feec8069f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:31.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.105+0000 7feee57fa640 1 -- 172.21.15.116:0/1104923780 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 8730+0+0 (secure 0 0 0) 0x7feee0122e30 con 0x7feee80c5760 2024-04-25T00:17:31.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.105+0000 7feee7fff640 1 --2- 172.21.15.116:0/1104923780 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7feec8067a90 0x7feec8069f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:31.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.105+0000 7feee7fff640 1 --2- 172.21.15.116:0/1104923780 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7feec8067a90 0x7feec8069f50 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7feee8149ab0 tx=0x7feed8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:31.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.109+0000 7feee57fa640 1 -- 172.21.15.116:0/1104923780 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7feee0019810 con 0x7feee80c5760 2024-04-25T00:17:31.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.309+0000 7feeee873640 1 -- 172.21.15.116:0/1104923780 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feee80c3640 con 0x7feee80c5760 2024-04-25T00:17:31.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.313+0000 7feee57fa640 1 -- 172.21.15.116:0/1104923780 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) ==== 95+0+5680 (secure 0 0 0) 0x7feee00f2d60 con 0x7feee80c5760 2024-04-25T00:17:31.312 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:17:31.312 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":131,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:16:48.854104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7119},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7119":{"gid":7119,"name":"a","rank":0,"incarnation":123,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6833/3645553116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3645553116},{"type":"v1","addr":"172.21.15.116:6833","nonce":3645553116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7119,"qdb_cluster":[7119]},"id":1},{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:17:28.837517+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7164},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7164":{"gid":7164,"name":"c","rank":0,"incarnation":128,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6835/3336339376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3336339376},{"type":"v1","addr":"172.21.15.116:6835","nonce":3336339376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7164,"qdb_cluster":[7164]},"id":2}]} 2024-04-25T00:17:31.312 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 131 2024-04-25T00:17:31.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.313+0000 7feeee873640 1 -- 172.21.15.116:0/1104923780 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7feec8067a90 msgr2=0x7feec8069f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:31.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.313+0000 7feeee873640 1 --2- 172.21.15.116:0/1104923780 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7feec8067a90 0x7feec8069f50 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7feee8149ab0 tx=0x7feed8008040 comp rx=0 tx=0).stop 2024-04-25T00:17:31.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.313+0000 7feeee873640 1 -- 172.21.15.116:0/1104923780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feee80c5760 msgr2=0x7feee8142f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:31.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.313+0000 7feeee873640 1 --2- 172.21.15.116:0/1104923780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feee80c5760 0x7feee8142f60 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7feee001f930 tx=0x7feed4005840 comp rx=0 tx=0).stop 2024-04-25T00:17:31.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.313+0000 7feeee873640 1 -- 172.21.15.116:0/1104923780 shutdown_connections 2024-04-25T00:17:31.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.313+0000 7feeee873640 1 --2- 172.21.15.116:0/1104923780 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7feec8067a90 0x7feec8069f50 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:31.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.313+0000 7feeee873640 1 --2- 172.21.15.116:0/1104923780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7feee80c5760 0x7feee8142f60 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:31.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.313+0000 7feeee873640 1 --2- 172.21.15.116:0/1104923780 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7feee80c52e0 0x7feee8142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:31.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.313+0000 7feeee873640 1 --2- 172.21.15.116:0/1104923780 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7feed40051d0 0x7feee80c0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:31.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.313+0000 7feeee873640 1 -- 172.21.15.116:0/1104923780 >> 172.21.15.116:0/1104923780 conn(0x7feee80c3b00 msgr2=0x7feee814bb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:31.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.313+0000 7feeee873640 1 -- 172.21.15.116:0/1104923780 shutdown_connections 2024-04-25T00:17:31.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:31.313+0000 7feeee873640 1 -- 172.21.15.116:0/1104923780 wait complete. 2024-04-25T00:17:31.331 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:17:33.081 INFO:tasks.mds_thrash.fs.[b]:reviving mds.d 2024-04-25T00:17:33.081 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-25T00:17:33.081 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-25T00:17:33.084 INFO:tasks.ceph.mds.d:Started 2024-04-25T00:17:33.084 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:17:33.169 INFO:tasks.ceph.mds.d.smithi174.stdout:starting mds.d at 2024-04-25T00:17:33.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.217+0000 7fcf4be08640 1 Processor -- start 2024-04-25T00:17:33.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.217+0000 7fcf4be08640 1 -- start start 2024-04-25T00:17:33.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.217+0000 7fcf4be08640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcf4414c720 0x7fcf441563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:33.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.217+0000 7fcf4be08640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fcf44071680 0x7fcf44071ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:33.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.217+0000 7fcf4be08640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf44071220 con 0x7fcf440a94a0 2024-04-25T00:17:33.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.217+0000 7fcf4be08640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fcf44153460 con 0x7fcf4414c720 2024-04-25T00:17:33.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.217+0000 7fcf4be08640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fcf44152420 con 0x7fcf44071680 2024-04-25T00:17:33.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.217+0000 7fcf49ba4640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcf4414c720 0x7fcf441563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:33.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.217+0000 7fcf49ba4640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcf4414c720 0x7fcf441563d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42394/0 (socket says 172.21.15.116:42394) 2024-04-25T00:17:33.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.217+0000 7fcf49ba4640 1 -- 172.21.15.116:0/4254721057 learned_addr learned my addr 172.21.15.116:0/4254721057 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:17:33.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.217+0000 7fcf4a3a5640 1 --2- 172.21.15.116:0/4254721057 >> v2:172.21.15.174:3300/0 conn(0x7fcf44071680 0x7fcf44071ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:33.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.217+0000 7fcf49ba4640 1 -- 172.21.15.116:0/4254721057 >> v2:172.21.15.174:3300/0 conn(0x7fcf44071680 msgr2=0x7fcf44071ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:33.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.217+0000 7fcf49ba4640 1 --2- 172.21.15.116:0/4254721057 >> v2:172.21.15.174:3300/0 conn(0x7fcf44071680 0x7fcf44071ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:33.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.217+0000 7fcf49ba4640 1 -- 172.21.15.116:0/4254721057 >> v1:172.21.15.116:6789/0 conn(0x7fcf440a94a0 legacy=0x7fcf440c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:33.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.217+0000 7fcf49ba4640 1 -- 172.21.15.116:0/4254721057 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf4414cb00 con 0x7fcf4414c720 2024-04-25T00:17:33.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.217+0000 7fcf4a3a5640 1 --2- 172.21.15.116:0/4254721057 >> v2:172.21.15.174:3300/0 conn(0x7fcf44071680 0x7fcf44071ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:17:33.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.221+0000 7fcf49ba4640 1 --2- 172.21.15.116:0/4254721057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf4414c720 0x7fcf441563d0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7fcf440ad370 tx=0x7fcf4001c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47dd1c526b69b7a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:33.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.221+0000 7fcf48ba2640 1 -- 172.21.15.116:0/4254721057 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf400ab660 con 0x7fcf4414c720 2024-04-25T00:17:33.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.221+0000 7fcf48ba2640 1 -- 172.21.15.116:0/4254721057 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf400b0070 con 0x7fcf4414c720 2024-04-25T00:17:33.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.221+0000 7fcf48ba2640 1 -- 172.21.15.116:0/4254721057 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf400aba20 con 0x7fcf4414c720 2024-04-25T00:17:33.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.221+0000 7fcf4be08640 1 -- 172.21.15.116:0/4254721057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf4414c720 msgr2=0x7fcf441563d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:33.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.221+0000 7fcf4be08640 1 --2- 172.21.15.116:0/4254721057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf4414c720 0x7fcf441563d0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7fcf440ad370 tx=0x7fcf4001c460 comp rx=0 tx=0).stop 2024-04-25T00:17:33.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.221+0000 7fcf4be08640 1 -- 172.21.15.116:0/4254721057 shutdown_connections 2024-04-25T00:17:33.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.221+0000 7fcf4be08640 1 --2- 172.21.15.116:0/4254721057 >> v2:172.21.15.174:3300/0 conn(0x7fcf44071680 0x7fcf44071ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:33.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.221+0000 7fcf4be08640 1 --2- 172.21.15.116:0/4254721057 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf4414c720 0x7fcf441563d0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:33.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.221+0000 7fcf4be08640 1 -- 172.21.15.116:0/4254721057 >> 172.21.15.116:0/4254721057 conn(0x7fcf440c3b00 msgr2=0x7fcf440c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:33.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.221+0000 7fcf4be08640 1 -- 172.21.15.116:0/4254721057 shutdown_connections 2024-04-25T00:17:33.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.221+0000 7fcf4be08640 1 -- 172.21.15.116:0/4254721057 wait complete. 2024-04-25T00:17:33.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.221+0000 7fcf4be08640 1 Processor -- start 2024-04-25T00:17:33.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.221+0000 7fcf4be08640 1 -- start start 2024-04-25T00:17:33.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.221+0000 7fcf4be08640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcf44071680 0x7fcf44151670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:33.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.221+0000 7fcf4be08640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf440a94a0 0x7fcf4414a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:33.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.221+0000 7fcf4be08640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf4414c720 0x7fcf4414a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:33.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.221+0000 7fcf4be08640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcf440c3120 con 0x7fcf4414c720 2024-04-25T00:17:33.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.221+0000 7fcf4be08640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcf44152f40 con 0x7fcf440a94a0 2024-04-25T00:17:33.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.221+0000 7fcf4be08640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fcf440c32a0 con 0x7fcf44071680 2024-04-25T00:17:33.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.221+0000 7fcf4a3a5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf4414c720 0x7fcf4414a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:33.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.221+0000 7fcf4a3a5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf4414c720 0x7fcf4414a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42406/0 (socket says 172.21.15.116:42406) 2024-04-25T00:17:33.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.221+0000 7fcf4a3a5640 1 -- 172.21.15.116:0/2089498963 learned_addr learned my addr 172.21.15.116:0/2089498963 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:17:33.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.221+0000 7fcf49ba4640 1 --2- 172.21.15.116:0/2089498963 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcf44071680 0x7fcf44151670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:33.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.221+0000 7fcf493a3640 1 --2- 172.21.15.116:0/2089498963 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf440a94a0 0x7fcf4414a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:33.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.225+0000 7fcf4a3a5640 1 -- 172.21.15.116:0/2089498963 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcf44071680 msgr2=0x7fcf44151670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:33.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.225+0000 7fcf4a3a5640 1 --2- 172.21.15.116:0/2089498963 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcf44071680 0x7fcf44151670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:33.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.225+0000 7fcf4a3a5640 1 -- 172.21.15.116:0/2089498963 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf440a94a0 msgr2=0x7fcf4414a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:33.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.225+0000 7fcf4a3a5640 1 --2- 172.21.15.116:0/2089498963 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf440a94a0 0x7fcf4414a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:33.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.225+0000 7fcf4a3a5640 1 -- 172.21.15.116:0/2089498963 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf441ef840 con 0x7fcf4414c720 2024-04-25T00:17:33.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.225+0000 7fcf4a3a5640 1 --2- 172.21.15.116:0/2089498963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf4414c720 0x7fcf4414a7f0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fcf3c009fd0 tx=0x7fcf3c00d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:33.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.225+0000 7fcf3a7fc640 1 -- 172.21.15.116:0/2089498963 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf3c01a070 con 0x7fcf4414c720 2024-04-25T00:17:33.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.225+0000 7fcf3a7fc640 1 -- 172.21.15.116:0/2089498963 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf3c003bc0 con 0x7fcf4414c720 2024-04-25T00:17:33.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.225+0000 7fcf4be08640 1 -- 172.21.15.116:0/2089498963 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf441ef560 con 0x7fcf4414c720 2024-04-25T00:17:33.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.225+0000 7fcf3a7fc640 1 -- 172.21.15.116:0/2089498963 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf3c004030 con 0x7fcf4414c720 2024-04-25T00:17:33.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.225+0000 7fcf4be08640 1 -- 172.21.15.116:0/2089498963 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf44153b90 con 0x7fcf4414c720 2024-04-25T00:17:33.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.225+0000 7fcf4be08640 1 -- 172.21.15.116:0/2089498963 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf44068350 con 0x7fcf4414c720 2024-04-25T00:17:33.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.225+0000 7fcf3a7fc640 1 -- 172.21.15.116:0/2089498963 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fcf3c01d070 con 0x7fcf4414c720 2024-04-25T00:17:33.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.225+0000 7fcf3a7fc640 1 --2- 172.21.15.116:0/2089498963 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcf280678c0 0x7fcf28069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:33.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.225+0000 7fcf3a7fc640 1 -- 172.21.15.116:0/2089498963 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 8730+0+0 (secure 0 0 0) 0x7fcf3c086b20 con 0x7fcf4414c720 2024-04-25T00:17:33.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.225+0000 7fcf49ba4640 1 --2- 172.21.15.116:0/2089498963 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcf280678c0 0x7fcf28069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:33.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.225+0000 7fcf49ba4640 1 --2- 172.21.15.116:0/2089498963 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcf280678c0 0x7fcf28069d80 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fcf400af000 tx=0x7fcf400a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:33.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.229+0000 7fcf3a7fc640 1 -- 172.21.15.116:0/2089498963 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fcf3c010070 con 0x7fcf4414c720 2024-04-25T00:17:33.331 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:17:33.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.421+0000 7fcf4be08640 1 -- 172.21.15.116:0/2089498963 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcf4406f310 con 0x7fcf4414c720 2024-04-25T00:17:33.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.421+0000 7fcf3a7fc640 1 -- 172.21.15.116:0/2089498963 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) ==== 95+0+5680 (secure 0 0 0) 0x7fcf3c0563f0 con 0x7fcf4414c720 2024-04-25T00:17:33.423 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:17:33.423 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":131,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:16:48.854104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7119},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7119":{"gid":7119,"name":"a","rank":0,"incarnation":123,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6833/3645553116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3645553116},{"type":"v1","addr":"172.21.15.116:6833","nonce":3645553116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7119,"qdb_cluster":[7119]},"id":1},{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:17:28.837517+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7164},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7164":{"gid":7164,"name":"c","rank":0,"incarnation":128,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6835/3336339376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3336339376},{"type":"v1","addr":"172.21.15.116:6835","nonce":3336339376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7164,"qdb_cluster":[7164]},"id":2}]} 2024-04-25T00:17:33.423 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 131 2024-04-25T00:17:33.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.425+0000 7fcf4be08640 1 -- 172.21.15.116:0/2089498963 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcf280678c0 msgr2=0x7fcf28069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:33.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.425+0000 7fcf4be08640 1 --2- 172.21.15.116:0/2089498963 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcf280678c0 0x7fcf28069d80 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fcf400af000 tx=0x7fcf400a7000 comp rx=0 tx=0).stop 2024-04-25T00:17:33.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.425+0000 7fcf4be08640 1 -- 172.21.15.116:0/2089498963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf4414c720 msgr2=0x7fcf4414a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:33.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.425+0000 7fcf4be08640 1 --2- 172.21.15.116:0/2089498963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf4414c720 0x7fcf4414a7f0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fcf3c009fd0 tx=0x7fcf3c00d560 comp rx=0 tx=0).stop 2024-04-25T00:17:33.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.425+0000 7fcf4be08640 1 -- 172.21.15.116:0/2089498963 shutdown_connections 2024-04-25T00:17:33.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.425+0000 7fcf4be08640 1 --2- 172.21.15.116:0/2089498963 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcf280678c0 0x7fcf28069d80 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:33.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.425+0000 7fcf4be08640 1 --2- 172.21.15.116:0/2089498963 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcf4414c720 0x7fcf4414a7f0 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:33.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.425+0000 7fcf4be08640 1 --2- 172.21.15.116:0/2089498963 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcf440a94a0 0x7fcf4414a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:33.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.425+0000 7fcf4be08640 1 --2- 172.21.15.116:0/2089498963 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcf44071680 0x7fcf44151670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:33.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.425+0000 7fcf4be08640 1 -- 172.21.15.116:0/2089498963 >> 172.21.15.116:0/2089498963 conn(0x7fcf440c3b00 msgr2=0x7fcf440c2630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:33.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.425+0000 7fcf4be08640 1 -- 172.21.15.116:0/2089498963 shutdown_connections 2024-04-25T00:17:33.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.425+0000 7fcf4be08640 1 -- 172.21.15.116:0/2089498963 wait complete. 2024-04-25T00:17:33.440 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2024-04-25T00:17:33.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.449+0000 7f7a43cce640 1 Processor -- start 2024-04-25T00:17:33.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.449+0000 7f7a43cce640 1 -- start start 2024-04-25T00:17:33.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.449+0000 7f7a43cce640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7a3c14c700 0x7f7a3c0c5090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:33.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.449+0000 7f7a43cce640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7a3c155b20 0x7f7a3c152c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:33.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.449+0000 7f7a43cce640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a3c0a9e60 con 0x7f7a3c0c55d0 2024-04-25T00:17:33.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.449+0000 7f7a43cce640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f7a3c1527b0 con 0x7f7a3c14c700 2024-04-25T00:17:33.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.449+0000 7f7a43cce640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f7a3c0c3ef0 con 0x7f7a3c155b20 2024-04-25T00:17:33.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.449+0000 7f7a41a6a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7a3c14c700 0x7f7a3c0c5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:33.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.449+0000 7f7a41a6a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7a3c14c700 0x7f7a3c0c5090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42418/0 (socket says 172.21.15.116:42418) 2024-04-25T00:17:33.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.449+0000 7f7a41a6a640 1 -- 172.21.15.116:0/3644660088 learned_addr learned my addr 172.21.15.116:0/3644660088 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:17:33.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.449+0000 7f7a41269640 1 --2- 172.21.15.116:0/3644660088 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7a3c155b20 0x7f7a3c152c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:33.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.449+0000 7f7a41a6a640 1 -- 172.21.15.116:0/3644660088 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7a3c155b20 msgr2=0x7f7a3c152c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:33.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.449+0000 7f7a41a6a640 1 --2- 172.21.15.116:0/3644660088 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7a3c155b20 0x7f7a3c152c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:33.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.449+0000 7f7a41a6a640 1 -- 172.21.15.116:0/3644660088 >> v1:172.21.15.116:6789/0 conn(0x7f7a3c0c55d0 legacy=0x7f7a3c071df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:33.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.449+0000 7f7a41a6a640 1 -- 172.21.15.116:0/3644660088 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a3c0c41d0 con 0x7f7a3c14c700 2024-04-25T00:17:33.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.453+0000 7f7a41a6a640 1 --2- 172.21.15.116:0/3644660088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a3c14c700 0x7f7a3c0c5090 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c152930 tx=0x7f7a2c01c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb0a3b552d5568cc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:33.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.453+0000 7f7a40a68640 1 -- 172.21.15.116:0/3644660088 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a2c0a64b0 con 0x7f7a3c14c700 2024-04-25T00:17:33.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.453+0000 7f7a40a68640 1 -- 172.21.15.116:0/3644660088 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a2c0a6650 con 0x7f7a3c14c700 2024-04-25T00:17:33.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.453+0000 7f7a40a68640 1 -- 172.21.15.116:0/3644660088 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a2c0a6a20 con 0x7f7a3c14c700 2024-04-25T00:17:33.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.453+0000 7f7a43cce640 1 -- 172.21.15.116:0/3644660088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a3c14c700 msgr2=0x7f7a3c0c5090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:33.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.453+0000 7f7a43cce640 1 --2- 172.21.15.116:0/3644660088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a3c14c700 0x7f7a3c0c5090 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c152930 tx=0x7f7a2c01c8f0 comp rx=0 tx=0).stop 2024-04-25T00:17:33.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.453+0000 7f7a43cce640 1 -- 172.21.15.116:0/3644660088 shutdown_connections 2024-04-25T00:17:33.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.453+0000 7f7a43cce640 1 --2- 172.21.15.116:0/3644660088 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7a3c155b20 0x7f7a3c152c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:33.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.453+0000 7f7a43cce640 1 --2- 172.21.15.116:0/3644660088 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a3c14c700 0x7f7a3c0c5090 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:33.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.453+0000 7f7a43cce640 1 -- 172.21.15.116:0/3644660088 >> 172.21.15.116:0/3644660088 conn(0x7f7a3c06c720 msgr2=0x7f7a3c06cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:33.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.453+0000 7f7a43cce640 1 -- 172.21.15.116:0/3644660088 shutdown_connections 2024-04-25T00:17:33.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.453+0000 7f7a43cce640 1 -- 172.21.15.116:0/3644660088 wait complete. 2024-04-25T00:17:33.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.453+0000 7f7a43cce640 1 Processor -- start 2024-04-25T00:17:33.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.453+0000 7f7a43cce640 1 -- start start 2024-04-25T00:17:33.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.453+0000 7f7a43cce640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a3c0c55d0 0x7f7a3c144f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:33.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.453+0000 7f7a43cce640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7a3c14c700 0x7f7a3c142bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:33.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.453+0000 7f7a43cce640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7a3c155b20 0x7f7a3c1430f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:33.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.453+0000 7f7a43cce640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7a3c153510 con 0x7f7a3c0c55d0 2024-04-25T00:17:33.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.453+0000 7f7a43cce640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7a3c153390 con 0x7f7a3c14c700 2024-04-25T00:17:33.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.453+0000 7f7a43cce640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f7a3c15dec0 con 0x7f7a3c155b20 2024-04-25T00:17:33.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.453+0000 7f7a41a6a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a3c0c55d0 0x7f7a3c144f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:33.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.453+0000 7f7a41a6a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a3c0c55d0 0x7f7a3c144f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42422/0 (socket says 172.21.15.116:42422) 2024-04-25T00:17:33.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.453+0000 7f7a41a6a640 1 -- 172.21.15.116:0/1494344332 learned_addr learned my addr 172.21.15.116:0/1494344332 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:17:33.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.453+0000 7f7a4226b640 1 --2- 172.21.15.116:0/1494344332 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7a3c155b20 0x7f7a3c1430f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:33.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.453+0000 7f7a41269640 1 --2- 172.21.15.116:0/1494344332 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7a3c14c700 0x7f7a3c142bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:33.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.453+0000 7f7a41a6a640 1 -- 172.21.15.116:0/1494344332 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7a3c155b20 msgr2=0x7f7a3c1430f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:33.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.453+0000 7f7a41a6a640 1 --2- 172.21.15.116:0/1494344332 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7a3c155b20 0x7f7a3c1430f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:33.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.453+0000 7f7a41a6a640 1 -- 172.21.15.116:0/1494344332 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7a3c14c700 msgr2=0x7f7a3c142bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:33.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.453+0000 7f7a41a6a640 1 --2- 172.21.15.116:0/1494344332 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7a3c14c700 0x7f7a3c142bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:33.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.453+0000 7f7a41a6a640 1 -- 172.21.15.116:0/1494344332 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a3c141ee0 con 0x7f7a3c0c55d0 2024-04-25T00:17:33.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.453+0000 7f7a4226b640 1 --2- 172.21.15.116:0/1494344332 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7a3c155b20 0x7f7a3c1430f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:17:33.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.453+0000 7f7a41a6a640 1 --2- 172.21.15.116:0/1494344332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a3c0c55d0 0x7f7a3c144f70 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f7a2c01a520 tx=0x7f7a2c019690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:33.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.453+0000 7f7a3a7fc640 1 -- 172.21.15.116:0/1494344332 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a2c0bf070 con 0x7f7a3c0c55d0 2024-04-25T00:17:33.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.453+0000 7f7a3a7fc640 1 -- 172.21.15.116:0/1494344332 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a2c0a6880 con 0x7f7a3c0c55d0 2024-04-25T00:17:33.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.453+0000 7f7a43cce640 1 -- 172.21.15.116:0/1494344332 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a3c141c00 con 0x7f7a3c0c55d0 2024-04-25T00:17:33.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.453+0000 7f7a3a7fc640 1 -- 172.21.15.116:0/1494344332 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a2c0a6c70 con 0x7f7a3c0c55d0 2024-04-25T00:17:33.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.453+0000 7f7a43cce640 1 -- 172.21.15.116:0/1494344332 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a3c143ed0 con 0x7f7a3c0c55d0 2024-04-25T00:17:33.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.457+0000 7f7a43cce640 1 -- 172.21.15.116:0/1494344332 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a3c068a60 con 0x7f7a3c0c55d0 2024-04-25T00:17:33.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.457+0000 7f7a3a7fc640 1 -- 172.21.15.116:0/1494344332 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7a2c0c2070 con 0x7f7a3c0c55d0 2024-04-25T00:17:33.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.457+0000 7f7a3a7fc640 1 --2- 172.21.15.116:0/1494344332 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7a280678c0 0x7f7a28069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:33.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.457+0000 7f7a3a7fc640 1 -- 172.21.15.116:0/1494344332 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 8730+0+0 (secure 0 0 0) 0x7f7a2c120960 con 0x7f7a3c0c55d0 2024-04-25T00:17:33.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.457+0000 7f7a41269640 1 --2- 172.21.15.116:0/1494344332 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7a280678c0 0x7f7a28069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:33.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.457+0000 7f7a41269640 1 --2- 172.21.15.116:0/1494344332 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7a280678c0 0x7f7a28069d80 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c147d90 tx=0x7f7a30008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:33.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.461+0000 7f7a3a7fc640 1 -- 172.21.15.116:0/1494344332 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f7a2c016600 con 0x7f7a3c0c55d0 2024-04-25T00:17:33.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.657+0000 7f7a43cce640 1 -- 172.21.15.116:0/1494344332 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7a3c0ad630 con 0x7f7a3c0c55d0 2024-04-25T00:17:33.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.661+0000 7f7a3a7fc640 1 -- 172.21.15.116:0/1494344332 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) ==== 95+0+5680 (secure 0 0 0) 0x7f7a2c0f08b0 con 0x7f7a3c0c55d0 2024-04-25T00:17:33.660 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:17:33.660 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":131,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:16:48.854104+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":51,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7119},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7119":{"gid":7119,"name":"a","rank":0,"incarnation":123,"state":"up:active","state_seq":5,"addr":"172.21.15.116:6833/3645553116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3645553116},{"type":"v1","addr":"172.21.15.116:6833","nonce":3645553116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7119,"qdb_cluster":[7119]},"id":1},{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:17:28.837517+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7164},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7164":{"gid":7164,"name":"c","rank":0,"incarnation":128,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6835/3336339376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3336339376},{"type":"v1","addr":"172.21.15.116:6835","nonce":3336339376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7164,"qdb_cluster":[7164]},"id":2}]} 2024-04-25T00:17:33.660 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 131 2024-04-25T00:17:33.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.661+0000 7f7a43cce640 1 -- 172.21.15.116:0/1494344332 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7a280678c0 msgr2=0x7f7a28069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:33.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.661+0000 7f7a43cce640 1 --2- 172.21.15.116:0/1494344332 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7a280678c0 0x7f7a28069d80 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c147d90 tx=0x7f7a30008040 comp rx=0 tx=0).stop 2024-04-25T00:17:33.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.661+0000 7f7a43cce640 1 -- 172.21.15.116:0/1494344332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a3c0c55d0 msgr2=0x7f7a3c144f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:33.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.661+0000 7f7a43cce640 1 --2- 172.21.15.116:0/1494344332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a3c0c55d0 0x7f7a3c144f70 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f7a2c01a520 tx=0x7f7a2c019690 comp rx=0 tx=0).stop 2024-04-25T00:17:33.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.661+0000 7f7a43cce640 1 -- 172.21.15.116:0/1494344332 shutdown_connections 2024-04-25T00:17:33.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.661+0000 7f7a43cce640 1 --2- 172.21.15.116:0/1494344332 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7a280678c0 0x7f7a28069d80 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:33.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.661+0000 7f7a43cce640 1 --2- 172.21.15.116:0/1494344332 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7a3c155b20 0x7f7a3c1430f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:33.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.661+0000 7f7a43cce640 1 --2- 172.21.15.116:0/1494344332 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7a3c14c700 0x7f7a3c142bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:33.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.661+0000 7f7a43cce640 1 --2- 172.21.15.116:0/1494344332 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7a3c0c55d0 0x7f7a3c144f70 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:33.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.661+0000 7f7a43cce640 1 -- 172.21.15.116:0/1494344332 >> 172.21.15.116:0/1494344332 conn(0x7f7a3c06c720 msgr2=0x7f7a3c06cb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:33.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.661+0000 7f7a43cce640 1 -- 172.21.15.116:0/1494344332 shutdown_connections 2024-04-25T00:17:33.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:33.661+0000 7f7a43cce640 1 -- 172.21.15.116:0/1494344332 wait complete. 2024-04-25T00:17:33.679 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:17:35.441 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:17:35.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.573+0000 7fdb85f46640 1 Processor -- start 2024-04-25T00:17:35.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.573+0000 7fdb85f46640 1 -- start start 2024-04-25T00:17:35.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.577+0000 7fdb85f46640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fdb80152d80 0x7fdb8006bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:35.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.577+0000 7fdb85f46640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb800c9150 con 0x7fdb8006d5f0 2024-04-25T00:17:35.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.577+0000 7fdb85f46640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb800c9430 con 0x7fdb8006c290 2024-04-25T00:17:35.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.577+0000 7fdb85f46640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fdb800c9d60 con 0x7fdb80152d80 2024-04-25T00:17:35.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.577+0000 7fdb7f7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fdb80152d80 0x7fdb8006bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:35.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.577+0000 7fdb7f7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fdb80152d80 0x7fdb8006bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36892/0 (socket says 172.21.15.116:36892) 2024-04-25T00:17:35.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.577+0000 7fdb7f7fe640 1 -- 172.21.15.116:0/1502454514 learned_addr learned my addr 172.21.15.116:0/1502454514 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:17:35.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.577+0000 7fdb7effd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fdb8006d5f0 0x7fdb80156c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54944/0 (socket says 172.21.15.116:54944) 2024-04-25T00:17:35.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.577+0000 7fdb7f7fe640 1 -- 172.21.15.116:0/1502454514 >> v1:172.21.15.174:6789/0 conn(0x7fdb8006c290 legacy=0x7fdb80157390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:35.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.577+0000 7fdb7f7fe640 1 -- 172.21.15.116:0/1502454514 >> v1:172.21.15.116:6789/0 conn(0x7fdb8006d5f0 legacy=0x7fdb80156c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:35.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.577+0000 7fdb7f7fe640 1 -- 172.21.15.116:0/1502454514 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb800c9710 con 0x7fdb80152d80 2024-04-25T00:17:35.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.577+0000 7fdb7f7fe640 1 --2- 172.21.15.116:0/1502454514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb80152d80 0x7fdb8006bd50 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7fdb700098a0 tx=0x7fdb7001c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1d15ab4c7f23f08 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:35.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.577+0000 7fdb7e7fc640 1 -- 172.21.15.116:0/1502454514 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb700b1070 con 0x7fdb80152d80 2024-04-25T00:17:35.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.577+0000 7fdb7e7fc640 1 -- 172.21.15.116:0/1502454514 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb700ac640 con 0x7fdb80152d80 2024-04-25T00:17:35.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.577+0000 7fdb7e7fc640 1 -- 172.21.15.116:0/1502454514 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb700aca10 con 0x7fdb80152d80 2024-04-25T00:17:35.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.577+0000 7fdb85f46640 1 -- 172.21.15.116:0/1502454514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb80152d80 msgr2=0x7fdb8006bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:35.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.577+0000 7fdb85f46640 1 --2- 172.21.15.116:0/1502454514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb80152d80 0x7fdb8006bd50 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7fdb700098a0 tx=0x7fdb7001c3f0 comp rx=0 tx=0).stop 2024-04-25T00:17:35.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.577+0000 7fdb85f46640 1 -- 172.21.15.116:0/1502454514 shutdown_connections 2024-04-25T00:17:35.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.581+0000 7fdb85f46640 1 --2- 172.21.15.116:0/1502454514 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb80152d80 0x7fdb8006bd50 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:35.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.581+0000 7fdb85f46640 1 -- 172.21.15.116:0/1502454514 >> 172.21.15.116:0/1502454514 conn(0x7fdb800acdd0 msgr2=0x7fdb800ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:35.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.581+0000 7fdb85f46640 1 -- 172.21.15.116:0/1502454514 shutdown_connections 2024-04-25T00:17:35.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.581+0000 7fdb85f46640 1 -- 172.21.15.116:0/1502454514 wait complete. 2024-04-25T00:17:35.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.581+0000 7fdb85f46640 1 Processor -- start 2024-04-25T00:17:35.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.581+0000 7fdb85f46640 1 -- start start 2024-04-25T00:17:35.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.581+0000 7fdb85f46640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb8006c290 0x7fdb800abdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:35.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.581+0000 7fdb85f46640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdb8006d5f0 0x7fdb800ac2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:35.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.581+0000 7fdb85f46640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdb80152d80 0x7fdb80147980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:35.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.581+0000 7fdb85f46640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdb80163a60 con 0x7fdb8006c290 2024-04-25T00:17:35.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.581+0000 7fdb85f46640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fdb801638e0 con 0x7fdb80152d80 2024-04-25T00:17:35.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.581+0000 7fdb85f46640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fdb80163be0 con 0x7fdb8006d5f0 2024-04-25T00:17:35.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.581+0000 7fdb7f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb8006c290 0x7fdb800abdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:35.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.581+0000 7fdb7f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb8006c290 0x7fdb800abdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36900/0 (socket says 172.21.15.116:36900) 2024-04-25T00:17:35.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.581+0000 7fdb7f7fe640 1 -- 172.21.15.116:0/3341221607 learned_addr learned my addr 172.21.15.116:0/3341221607 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:17:35.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.581+0000 7fdb7ffff640 1 --2- 172.21.15.116:0/3341221607 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdb80152d80 0x7fdb80147980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:35.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.581+0000 7fdb7effd640 1 --2- 172.21.15.116:0/3341221607 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdb8006d5f0 0x7fdb800ac2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:35.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.581+0000 7fdb7f7fe640 1 -- 172.21.15.116:0/3341221607 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdb8006d5f0 msgr2=0x7fdb800ac2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:35.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.581+0000 7fdb7f7fe640 1 --2- 172.21.15.116:0/3341221607 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdb8006d5f0 0x7fdb800ac2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:35.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.581+0000 7fdb7f7fe640 1 -- 172.21.15.116:0/3341221607 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdb80152d80 msgr2=0x7fdb80147980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:35.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.581+0000 7fdb7f7fe640 1 --2- 172.21.15.116:0/3341221607 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdb80152d80 0x7fdb80147980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:35.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.581+0000 7fdb7f7fe640 1 -- 172.21.15.116:0/3341221607 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb801457a0 con 0x7fdb8006c290 2024-04-25T00:17:35.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.581+0000 7fdb7effd640 1 --2- 172.21.15.116:0/3341221607 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdb8006d5f0 0x7fdb800ac2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:17:35.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.581+0000 7fdb7f7fe640 1 --2- 172.21.15.116:0/3341221607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb8006c290 0x7fdb800abdb0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7fdb7001a020 tx=0x7fdb7001ac30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:35.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.581+0000 7fdb67fff640 1 -- 172.21.15.116:0/3341221607 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb700bb020 con 0x7fdb8006c290 2024-04-25T00:17:35.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.581+0000 7fdb67fff640 1 -- 172.21.15.116:0/3341221607 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb700b7620 con 0x7fdb8006c290 2024-04-25T00:17:35.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.581+0000 7fdb85f46640 1 -- 172.21.15.116:0/3341221607 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb801438b0 con 0x7fdb8006c290 2024-04-25T00:17:35.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.581+0000 7fdb67fff640 1 -- 172.21.15.116:0/3341221607 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb700b1040 con 0x7fdb8006c290 2024-04-25T00:17:35.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.581+0000 7fdb85f46640 1 -- 172.21.15.116:0/3341221607 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb80145d20 con 0x7fdb8006c290 2024-04-25T00:17:35.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.581+0000 7fdb85f46640 1 -- 172.21.15.116:0/3341221607 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb80068350 con 0x7fdb8006c290 2024-04-25T00:17:35.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.585+0000 7fdb67fff640 1 -- 172.21.15.116:0/3341221607 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fdb700be020 con 0x7fdb8006c290 2024-04-25T00:17:35.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.585+0000 7fdb67fff640 1 --2- 172.21.15.116:0/3341221607 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdb6c0678f0 0x7fdb6c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:35.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.585+0000 7fdb67fff640 1 -- 172.21.15.116:0/3341221607 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 8730+0+0 (secure 0 0 0) 0x7fdb70120b90 con 0x7fdb8006c290 2024-04-25T00:17:35.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.585+0000 7fdb7effd640 1 --2- 172.21.15.116:0/3341221607 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdb6c0678f0 0x7fdb6c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:35.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.585+0000 7fdb7effd640 1 --2- 172.21.15.116:0/3341221607 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdb6c0678f0 0x7fdb6c069db0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fdb800bea70 tx=0x7fdb74009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:35.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.589+0000 7fdb67fff640 1 -- 172.21.15.116:0/3341221607 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fdb700c5030 con 0x7fdb8006c290 2024-04-25T00:17:35.680 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:17:35.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.801+0000 7fa19d5ef640 1 Processor -- start 2024-04-25T00:17:35.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.801+0000 7fa19d5ef640 1 -- start start 2024-04-25T00:17:35.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.801+0000 7fdb85f46640 1 -- 172.21.15.116:0/3341221607 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdb800cab60 con 0x7fdb8006c290 2024-04-25T00:17:35.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.801+0000 7fa19d5ef640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa198150d20 0x7fa198153620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:35.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.801+0000 7fdb67fff640 1 -- 172.21.15.116:0/3341221607 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 133 v133) ==== 95+0+5673 (secure 0 0 0) 0x7fdb70016130 con 0x7fdb8006c290 2024-04-25T00:17:35.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.801+0000 7fa19d5ef640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa198071050 0x7fa1980714c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:35.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.801+0000 7fa19d5ef640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1980c36a0 con 0x7fa1980c3240 2024-04-25T00:17:35.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.801+0000 7fa19d5ef640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fa19806fba0 con 0x7fa198150d20 2024-04-25T00:17:35.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.801+0000 7fa19d5ef640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa19806f8c0 con 0x7fa198071050 2024-04-25T00:17:35.802 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:17:35.802 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":133,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:17:35.601854+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6343":{"gid":6343,"name":"d","rank":0,"incarnation":133,"state":"up:replay","state_seq":1,"addr":"172.21.15.174:6837/1281466397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1281466397},{"type":"v1","addr":"172.21.15.174:6837","nonce":1281466397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:17:28.837517+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7164},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7164":{"gid":7164,"name":"c","rank":0,"incarnation":128,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6835/3336339376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3336339376},{"type":"v1","addr":"172.21.15.116:6835","nonce":3336339376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7164,"qdb_cluster":[7164]},"id":2}]} 2024-04-25T00:17:35.803 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 133 2024-04-25T00:17:35.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.801+0000 7fa196ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa198150d20 0x7fa198153620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:35.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.801+0000 7fa196ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa198150d20 0x7fa198153620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36906/0 (socket says 172.21.15.116:36906) 2024-04-25T00:17:35.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.801+0000 7fa196ffd640 1 -- 172.21.15.116:0/648884672 learned_addr learned my addr 172.21.15.116:0/648884672 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:17:35.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.801+0000 7fa1977fe640 1 --2- 172.21.15.116:0/648884672 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa198071050 0x7fa1980714c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:35.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.801+0000 7fa196ffd640 1 -- 172.21.15.116:0/648884672 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa198071050 msgr2=0x7fa1980714c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:35.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.801+0000 7fa196ffd640 1 --2- 172.21.15.116:0/648884672 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa198071050 0x7fa1980714c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:35.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.801+0000 7fa196ffd640 1 -- 172.21.15.116:0/648884672 >> v1:172.21.15.116:6789/0 conn(0x7fa1980c3240 legacy=0x7fa1980c3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:35.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.801+0000 7fa195ffb640 1 -- 172.21.15.116:0/648884672 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3359168386 0 0) 0x7fa1980c36a0 con 0x7fa1980c3240 2024-04-25T00:17:35.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.801+0000 7fa196ffd640 1 -- 172.21.15.116:0/648884672 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1980c5420 con 0x7fa198150d20 2024-04-25T00:17:35.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.801+0000 7fdb85f46640 1 -- 172.21.15.116:0/3341221607 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdb6c0678f0 msgr2=0x7fdb6c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:35.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.801+0000 7fdb85f46640 1 --2- 172.21.15.116:0/3341221607 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdb6c0678f0 0x7fdb6c069db0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fdb800bea70 tx=0x7fdb74009040 comp rx=0 tx=0).stop 2024-04-25T00:17:35.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.801+0000 7fdb85f46640 1 -- 172.21.15.116:0/3341221607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb8006c290 msgr2=0x7fdb800abdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:35.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.801+0000 7fdb85f46640 1 --2- 172.21.15.116:0/3341221607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb8006c290 0x7fdb800abdb0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7fdb7001a020 tx=0x7fdb7001ac30 comp rx=0 tx=0).stop 2024-04-25T00:17:35.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.801+0000 7fdb85f46640 1 -- 172.21.15.116:0/3341221607 shutdown_connections 2024-04-25T00:17:35.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.801+0000 7fdb85f46640 1 --2- 172.21.15.116:0/3341221607 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdb6c0678f0 0x7fdb6c069db0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:35.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.801+0000 7fdb85f46640 1 --2- 172.21.15.116:0/3341221607 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdb80152d80 0x7fdb80147980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:35.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.801+0000 7fdb85f46640 1 --2- 172.21.15.116:0/3341221607 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdb8006d5f0 0x7fdb800ac2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:35.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.801+0000 7fdb85f46640 1 --2- 172.21.15.116:0/3341221607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb8006c290 0x7fdb800abdb0 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:35.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.801+0000 7fdb85f46640 1 -- 172.21.15.116:0/3341221607 >> 172.21.15.116:0/3341221607 conn(0x7fdb800acdd0 msgr2=0x7fdb80160510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:35.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.801+0000 7fdb85f46640 1 -- 172.21.15.116:0/3341221607 shutdown_connections 2024-04-25T00:17:35.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.801+0000 7fdb85f46640 1 -- 172.21.15.116:0/3341221607 wait complete. 2024-04-25T00:17:35.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.805+0000 7fa196ffd640 1 --2- 172.21.15.116:0/648884672 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa198150d20 0x7fa198153620 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7fa188009870 tx=0x7fa18801c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7cbcd404af3cdce server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:35.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.805+0000 7fa195ffb640 1 -- 172.21.15.116:0/648884672 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1880a7070 con 0x7fa198150d20 2024-04-25T00:17:35.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.805+0000 7fa195ffb640 1 -- 172.21.15.116:0/648884672 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1880b0070 con 0x7fa198150d20 2024-04-25T00:17:35.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.805+0000 7fa195ffb640 1 -- 172.21.15.116:0/648884672 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1880ab890 con 0x7fa198150d20 2024-04-25T00:17:35.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.805+0000 7fa19d5ef640 1 -- 172.21.15.116:0/648884672 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa198150d20 msgr2=0x7fa198153620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:35.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.805+0000 7fa19d5ef640 1 --2- 172.21.15.116:0/648884672 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa198150d20 0x7fa198153620 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7fa188009870 tx=0x7fa18801c560 comp rx=0 tx=0).stop 2024-04-25T00:17:35.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.805+0000 7fa19d5ef640 1 -- 172.21.15.116:0/648884672 shutdown_connections 2024-04-25T00:17:35.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.805+0000 7fa19d5ef640 1 --2- 172.21.15.116:0/648884672 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa198071050 0x7fa1980714c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:35.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.805+0000 7fa19d5ef640 1 --2- 172.21.15.116:0/648884672 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa198150d20 0x7fa198153620 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:35.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.805+0000 7fa19d5ef640 1 -- 172.21.15.116:0/648884672 >> 172.21.15.116:0/648884672 conn(0x7fa1980c9160 msgr2=0x7fa1980c9570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:35.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.805+0000 7fa19d5ef640 1 -- 172.21.15.116:0/648884672 shutdown_connections 2024-04-25T00:17:35.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.805+0000 7fa19d5ef640 1 -- 172.21.15.116:0/648884672 wait complete. 2024-04-25T00:17:35.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.805+0000 7fa19d5ef640 1 Processor -- start 2024-04-25T00:17:35.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.805+0000 7fa19d5ef640 1 -- start start 2024-04-25T00:17:35.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.805+0000 7fa19d5ef640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa198071050 0x7fa1981eb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:35.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.805+0000 7fa19d5ef640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1980c3240 0x7fa1981ec3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:35.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.805+0000 7fa19d5ef640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa198150d20 0x7fa1981407b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:35.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.805+0000 7fa19d5ef640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa19806f640 con 0x7fa1980c3240 2024-04-25T00:17:35.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.805+0000 7fa19d5ef640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa19806f4c0 con 0x7fa198150d20 2024-04-25T00:17:35.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.805+0000 7fa19d5ef640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa198151100 con 0x7fa198071050 2024-04-25T00:17:35.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.805+0000 7fa1967fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1980c3240 0x7fa1981ec3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:35.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.805+0000 7fa1967fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1980c3240 0x7fa1981ec3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36916/0 (socket says 172.21.15.116:36916) 2024-04-25T00:17:35.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.805+0000 7fa1967fc640 1 -- 172.21.15.116:0/210292839 learned_addr learned my addr 172.21.15.116:0/210292839 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:17:35.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.805+0000 7fa196ffd640 1 --2- 172.21.15.116:0/210292839 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa198071050 0x7fa1981eb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:35.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.805+0000 7fa1977fe640 1 --2- 172.21.15.116:0/210292839 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa198150d20 0x7fa1981407b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:35.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.809+0000 7fa196ffd640 1 -- 172.21.15.116:0/210292839 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa198150d20 msgr2=0x7fa1981407b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:35.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.809+0000 7fa196ffd640 1 --2- 172.21.15.116:0/210292839 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa198150d20 0x7fa1981407b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:35.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.809+0000 7fa196ffd640 1 -- 172.21.15.116:0/210292839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1980c3240 msgr2=0x7fa1981ec3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:35.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.809+0000 7fa196ffd640 1 --2- 172.21.15.116:0/210292839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1980c3240 0x7fa1981ec3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:35.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.809+0000 7fa196ffd640 1 -- 172.21.15.116:0/210292839 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa198152020 con 0x7fa198071050 2024-04-25T00:17:35.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.809+0000 7fa1967fc640 1 --2- 172.21.15.116:0/210292839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1980c3240 0x7fa1981ec3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:17:35.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.809+0000 7fa196ffd640 1 --2- 172.21.15.116:0/210292839 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa198071050 0x7fa1981eb9e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa1880ac000 tx=0x7fa18801ada0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:35.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.809+0000 7fa17f7fe640 1 -- 172.21.15.116:0/210292839 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1880be5f0 con 0x7fa198071050 2024-04-25T00:17:35.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.809+0000 7fa17f7fe640 1 -- 172.21.15.116:0/210292839 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1880be7d0 con 0x7fa198071050 2024-04-25T00:17:35.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.809+0000 7fa19d5ef640 1 -- 172.21.15.116:0/210292839 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa198152300 con 0x7fa198071050 2024-04-25T00:17:35.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.809+0000 7fa17f7fe640 1 -- 172.21.15.116:0/210292839 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1880bebc0 con 0x7fa198071050 2024-04-25T00:17:35.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.809+0000 7fa19d5ef640 1 -- 172.21.15.116:0/210292839 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1981528c0 con 0x7fa198071050 2024-04-25T00:17:35.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.809+0000 7fa19d5ef640 1 -- 172.21.15.116:0/210292839 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa198068a60 con 0x7fa198071050 2024-04-25T00:17:35.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.809+0000 7fa17f7fe640 1 -- 172.21.15.116:0/210292839 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa1880b43f0 con 0x7fa198071050 2024-04-25T00:17:35.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.809+0000 7fa17f7fe640 1 --2- 172.21.15.116:0/210292839 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa1840678c0 0x7fa184069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:35.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.809+0000 7fa17f7fe640 1 -- 172.21.15.116:0/210292839 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(53..53 src has 1..53) ==== 8816+0+0 (secure 0 0 0) 0x7fa188120fe0 con 0x7fa198071050 2024-04-25T00:17:35.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.809+0000 7fa1967fc640 1 --2- 172.21.15.116:0/210292839 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa1840678c0 0x7fa184069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:35.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.809+0000 7fa1967fc640 1 --2- 172.21.15.116:0/210292839 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa1840678c0 0x7fa184069d80 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fa1981e9820 tx=0x7fa18c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:35.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:35.821+0000 7fa17f7fe640 1 -- 172.21.15.116:0/210292839 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fa188016150 con 0x7fa198071050 2024-04-25T00:17:35.821 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2024-04-25T00:17:36.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.021+0000 7fa19d5ef640 1 -- 172.21.15.116:0/210292839 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa1980c4790 con 0x7fa198071050 2024-04-25T00:17:36.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.021+0000 7fa17f7fe640 1 -- 172.21.15.116:0/210292839 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 133 v133) ==== 95+0+5673 (secure 0 0 0) 0x7fa1880f0900 con 0x7fa198071050 2024-04-25T00:17:36.023 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:17:36.024 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":133,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:17:35.601854+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6343":{"gid":6343,"name":"d","rank":0,"incarnation":133,"state":"up:replay","state_seq":1,"addr":"172.21.15.174:6837/1281466397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1281466397},{"type":"v1","addr":"172.21.15.174:6837","nonce":1281466397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:17:28.837517+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7164},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7164":{"gid":7164,"name":"c","rank":0,"incarnation":128,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6835/3336339376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3336339376},{"type":"v1","addr":"172.21.15.116:6835","nonce":3336339376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7164,"qdb_cluster":[7164]},"id":2}]} 2024-04-25T00:17:36.024 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 133 2024-04-25T00:17:36.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.025+0000 7fa19d5ef640 1 -- 172.21.15.116:0/210292839 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa1840678c0 msgr2=0x7fa184069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:36.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.025+0000 7fa19d5ef640 1 --2- 172.21.15.116:0/210292839 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa1840678c0 0x7fa184069d80 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fa1981e9820 tx=0x7fa18c009040 comp rx=0 tx=0).stop 2024-04-25T00:17:36.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.025+0000 7fa19d5ef640 1 -- 172.21.15.116:0/210292839 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa198071050 msgr2=0x7fa1981eb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:36.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.025+0000 7fa19d5ef640 1 --2- 172.21.15.116:0/210292839 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa198071050 0x7fa1981eb9e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa1880ac000 tx=0x7fa18801ada0 comp rx=0 tx=0).stop 2024-04-25T00:17:36.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.025+0000 7fa19d5ef640 1 -- 172.21.15.116:0/210292839 shutdown_connections 2024-04-25T00:17:36.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.025+0000 7fa19d5ef640 1 --2- 172.21.15.116:0/210292839 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa1840678c0 0x7fa184069d80 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:36.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.025+0000 7fa19d5ef640 1 --2- 172.21.15.116:0/210292839 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa198150d20 0x7fa1981407b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:36.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.025+0000 7fa19d5ef640 1 --2- 172.21.15.116:0/210292839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1980c3240 0x7fa1981ec3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:36.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.025+0000 7fa19d5ef640 1 --2- 172.21.15.116:0/210292839 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa198071050 0x7fa1981eb9e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:36.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.025+0000 7fa19d5ef640 1 -- 172.21.15.116:0/210292839 >> 172.21.15.116:0/210292839 conn(0x7fa1980c9160 msgr2=0x7fa1980c3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:36.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.025+0000 7fa19d5ef640 1 -- 172.21.15.116:0/210292839 shutdown_connections 2024-04-25T00:17:36.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.025+0000 7fa19d5ef640 1 -- 172.21.15.116:0/210292839 wait complete. 2024-04-25T00:17:36.043 INFO:tasks.mds_thrash.fs.[a]:mds.a down, removed from mdsmap 2024-04-25T00:17:36.043 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-04-25T00:17:36.044 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:17:36.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.161+0000 7f03b5a75640 1 Processor -- start 2024-04-25T00:17:36.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.161+0000 7f03b5a75640 1 -- start start 2024-04-25T00:17:36.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.165+0000 7f03b5a75640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f03b006fcd0 0x7f03b00c1540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:36.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.165+0000 7f03b5a75640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03b006bd50 con 0x7f03b00c40e0 2024-04-25T00:17:36.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.165+0000 7f03b5a75640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03b006f990 con 0x7f03b00730a0 2024-04-25T00:17:36.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.165+0000 7f03b5a75640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f03b0156780 con 0x7f03b006fcd0 2024-04-25T00:17:36.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.165+0000 7f03af7fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f03b00c40e0 0x7f03b0156070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54966/0 (socket says 172.21.15.116:54966) 2024-04-25T00:17:36.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.165+0000 7f03af7fe640 1 -- 172.21.15.116:0/58394472 learned_addr learned my addr 172.21.15.116:0/58394472 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:17:36.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.165+0000 7f03affff640 1 --2- 172.21.15.116:0/58394472 >> v2:172.21.15.116:3300/0 conn(0x7f03b006fcd0 0x7f03b00c1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:36.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.165+0000 7f03affff640 1 -- 172.21.15.116:0/58394472 >> v1:172.21.15.174:6789/0 conn(0x7f03b00730a0 legacy=0x7f03b01525a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:36.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.165+0000 7f03affff640 1 -- 172.21.15.116:0/58394472 >> v1:172.21.15.116:6789/0 conn(0x7f03b00c40e0 legacy=0x7f03b0156070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:36.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.165+0000 7f03ae7fc640 1 -- 172.21.15.116:0/58394472 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3784499160 0 0) 0x7f03b006f990 con 0x7f03b00730a0 2024-04-25T00:17:36.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.165+0000 7f03affff640 1 -- 172.21.15.116:0/58394472 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03b0155b50 con 0x7f03b006fcd0 2024-04-25T00:17:36.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.165+0000 7f03affff640 1 --2- 172.21.15.116:0/58394472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03b006fcd0 0x7f03b00c1540 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f03a8009d90 tx=0x7f03a801c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bc4bb3832ad6c90 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:36.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.165+0000 7f03ae7fc640 1 -- 172.21.15.116:0/58394472 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03a80b2070 con 0x7f03b006fcd0 2024-04-25T00:17:36.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.165+0000 7f03ae7fc640 1 -- 172.21.15.116:0/58394472 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03a80a5820 con 0x7f03b006fcd0 2024-04-25T00:17:36.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.165+0000 7f03ae7fc640 1 -- 172.21.15.116:0/58394472 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03a80a5bf0 con 0x7f03b006fcd0 2024-04-25T00:17:36.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.169+0000 7f03b5a75640 1 -- 172.21.15.116:0/58394472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03b006fcd0 msgr2=0x7f03b00c1540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:36.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.169+0000 7f03b5a75640 1 --2- 172.21.15.116:0/58394472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03b006fcd0 0x7f03b00c1540 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f03a8009d90 tx=0x7f03a801c850 comp rx=0 tx=0).stop 2024-04-25T00:17:36.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.169+0000 7f03b5a75640 1 -- 172.21.15.116:0/58394472 shutdown_connections 2024-04-25T00:17:36.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.169+0000 7f03b5a75640 1 --2- 172.21.15.116:0/58394472 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03b006fcd0 0x7f03b00c1540 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:36.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.169+0000 7f03b5a75640 1 -- 172.21.15.116:0/58394472 >> 172.21.15.116:0/58394472 conn(0x7f03b00c4c10 msgr2=0x7f03b00c5020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:36.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.169+0000 7f03b5a75640 1 -- 172.21.15.116:0/58394472 shutdown_connections 2024-04-25T00:17:36.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.169+0000 7f03b5a75640 1 -- 172.21.15.116:0/58394472 wait complete. 2024-04-25T00:17:36.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.169+0000 7f03b5a75640 1 Processor -- start 2024-04-25T00:17:36.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.169+0000 7f03b5a75640 1 -- start start 2024-04-25T00:17:36.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.169+0000 7f03b5a75640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03b006fcd0 0x7f03b00c7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:36.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.169+0000 7f03b5a75640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f03b00730a0 0x7f03b00aaaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:36.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.169+0000 7f03b5a75640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f03b00c40e0 0x7f03b00aafe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:36.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.169+0000 7f03b5a75640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f03b00ad0c0 con 0x7f03b006fcd0 2024-04-25T00:17:36.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.169+0000 7f03b5a75640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f03b00acf40 con 0x7f03b00c40e0 2024-04-25T00:17:36.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.169+0000 7f03b5a75640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f03b01530c0 con 0x7f03b00730a0 2024-04-25T00:17:36.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.169+0000 7f03af7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03b006fcd0 0x7f03b00c7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:36.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.169+0000 7f03af7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03b006fcd0 0x7f03b00c7760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36932/0 (socket says 172.21.15.116:36932) 2024-04-25T00:17:36.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.169+0000 7f03af7fe640 1 -- 172.21.15.116:0/2735712986 learned_addr learned my addr 172.21.15.116:0/2735712986 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:17:36.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.169+0000 7f03affff640 1 --2- 172.21.15.116:0/2735712986 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f03b00c40e0 0x7f03b00aafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:36.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.169+0000 7f03aeffd640 1 --2- 172.21.15.116:0/2735712986 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f03b00730a0 0x7f03b00aaaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:36.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.169+0000 7f03af7fe640 1 -- 172.21.15.116:0/2735712986 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f03b00730a0 msgr2=0x7f03b00aaaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:36.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.169+0000 7f03af7fe640 1 --2- 172.21.15.116:0/2735712986 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f03b00730a0 0x7f03b00aaaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:36.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.169+0000 7f03af7fe640 1 -- 172.21.15.116:0/2735712986 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f03b00c40e0 msgr2=0x7f03b00aafe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:36.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.169+0000 7f03af7fe640 1 --2- 172.21.15.116:0/2735712986 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f03b00c40e0 0x7f03b00aafe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:36.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.169+0000 7f03af7fe640 1 -- 172.21.15.116:0/2735712986 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03b00ab520 con 0x7f03b006fcd0 2024-04-25T00:17:36.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.169+0000 7f03aeffd640 1 --2- 172.21.15.116:0/2735712986 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f03b00730a0 0x7f03b00aaaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:17:36.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.169+0000 7f03affff640 1 --2- 172.21.15.116:0/2735712986 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f03b00c40e0 0x7f03b00aafe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:17:36.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.169+0000 7f03af7fe640 1 --2- 172.21.15.116:0/2735712986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03b006fcd0 0x7f03b00c7760 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f03a000c420 tx=0x7f03a000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:36.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.169+0000 7f03a7fff640 1 -- 172.21.15.116:0/2735712986 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03a0017070 con 0x7f03b006fcd0 2024-04-25T00:17:36.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.169+0000 7f03a7fff640 1 -- 172.21.15.116:0/2735712986 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03a0005cf0 con 0x7f03b006fcd0 2024-04-25T00:17:36.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.169+0000 7f03b5a75640 1 -- 172.21.15.116:0/2735712986 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03b01f2990 con 0x7f03b006fcd0 2024-04-25T00:17:36.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.169+0000 7f03a7fff640 1 -- 172.21.15.116:0/2735712986 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03a00072f0 con 0x7f03b006fcd0 2024-04-25T00:17:36.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.169+0000 7f03b5a75640 1 -- 172.21.15.116:0/2735712986 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03b0157fc0 con 0x7f03b006fcd0 2024-04-25T00:17:36.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.173+0000 7f03b5a75640 1 -- 172.21.15.116:0/2735712986 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03b0068350 con 0x7f03b006fcd0 2024-04-25T00:17:36.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.173+0000 7f03a7fff640 1 -- 172.21.15.116:0/2735712986 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f03a001a070 con 0x7f03b006fcd0 2024-04-25T00:17:36.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.173+0000 7f03a7fff640 1 --2- 172.21.15.116:0/2735712986 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f03900678f0 0x7f0390069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:36.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.173+0000 7f03a7fff640 1 -- 172.21.15.116:0/2735712986 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 8816+0+0 (secure 0 0 0) 0x7f03a0085b70 con 0x7f03b006fcd0 2024-04-25T00:17:36.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.173+0000 7f03aeffd640 1 --2- 172.21.15.116:0/2735712986 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f03900678f0 0x7f0390069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:36.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.173+0000 7f03aeffd640 1 --2- 172.21.15.116:0/2735712986 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f03900678f0 0x7f0390069db0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f03b00ac510 tx=0x7f0398009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:36.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.177+0000 7f03a7fff640 1 -- 172.21.15.116:0/2735712986 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f03a0055480 con 0x7f03b006fcd0 2024-04-25T00:17:36.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.361+0000 7f03b5a75640 1 -- 172.21.15.116:0/2735712986 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f03b00c5ae0 con 0x7f03b006fcd0 2024-04-25T00:17:36.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.365+0000 7f03a7fff640 1 -- 172.21.15.116:0/2735712986 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 133 v133) ==== 95+0+5673 (secure 0 0 0) 0x7f03a0059780 con 0x7f03b006fcd0 2024-04-25T00:17:36.364 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:17:36.365 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":133,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":133,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:17:35.601854+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6343":{"gid":6343,"name":"d","rank":0,"incarnation":133,"state":"up:replay","state_seq":1,"addr":"172.21.15.174:6837/1281466397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1281466397},{"type":"v1","addr":"172.21.15.174:6837","nonce":1281466397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:17:28.837517+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7164},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7164":{"gid":7164,"name":"c","rank":0,"incarnation":128,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6835/3336339376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3336339376},{"type":"v1","addr":"172.21.15.116:6835","nonce":3336339376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7164,"qdb_cluster":[7164]},"id":2}]} 2024-04-25T00:17:36.365 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 133 2024-04-25T00:17:36.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.365+0000 7f03b5a75640 1 -- 172.21.15.116:0/2735712986 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f03900678f0 msgr2=0x7f0390069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:36.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.365+0000 7f03b5a75640 1 --2- 172.21.15.116:0/2735712986 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f03900678f0 0x7f0390069db0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f03b00ac510 tx=0x7f0398009040 comp rx=0 tx=0).stop 2024-04-25T00:17:36.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.365+0000 7f03b5a75640 1 -- 172.21.15.116:0/2735712986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03b006fcd0 msgr2=0x7f03b00c7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:36.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.365+0000 7f03b5a75640 1 --2- 172.21.15.116:0/2735712986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03b006fcd0 0x7f03b00c7760 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f03a000c420 tx=0x7f03a000c8e0 comp rx=0 tx=0).stop 2024-04-25T00:17:36.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.365+0000 7f03b5a75640 1 -- 172.21.15.116:0/2735712986 shutdown_connections 2024-04-25T00:17:36.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.365+0000 7f03b5a75640 1 --2- 172.21.15.116:0/2735712986 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f03900678f0 0x7f0390069db0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:36.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.365+0000 7f03b5a75640 1 --2- 172.21.15.116:0/2735712986 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f03b00c40e0 0x7f03b00aafe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:36.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.365+0000 7f03b5a75640 1 --2- 172.21.15.116:0/2735712986 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f03b00730a0 0x7f03b00aaaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:36.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.365+0000 7f03b5a75640 1 --2- 172.21.15.116:0/2735712986 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03b006fcd0 0x7f03b00c7760 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:36.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.365+0000 7f03b5a75640 1 -- 172.21.15.116:0/2735712986 >> 172.21.15.116:0/2735712986 conn(0x7f03b00c4c10 msgr2=0x7f03b0153640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:36.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.365+0000 7f03b5a75640 1 -- 172.21.15.116:0/2735712986 shutdown_connections 2024-04-25T00:17:36.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:36.365+0000 7f03b5a75640 1 -- 172.21.15.116:0/2735712986 wait complete. 2024-04-25T00:17:36.384 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 133, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:04.678737+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 133, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6343": { "addr": "172.21.15.174:6837/1281466397", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6836", "nonce": 1281466397, "type": "v2" }, { "addr": "172.21.15.174:6837", "nonce": 1281466397, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6343, "incarnation": 133, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 53, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T00:17:35.601854+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6343 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:18.825887+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 131, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_7164": { "addr": "172.21.15.116:6835/3336339376", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6834", "nonce": 3336339376, "type": "v2" }, { "addr": "172.21.15.116:6835", "nonce": 3336339376, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7164, "incarnation": 128, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 12 } }, "last_failure": 0, "last_failure_osd_epoch": 52, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-04-25T00:17:28.837517+0000", "qdb_cluster": [ 7164 ], "qdb_leader": 7164, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7164 } } } ], "standbys": [ { "addr": "172.21.15.174:6835/4112145782", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6834", "nonce": 4112145782, "type": "v2" }, { "addr": "172.21.15.174:6835", "nonce": 4112145782, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 112, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5582, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-25T00:17:37.822 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:17:37.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.953+0000 7f3a97c63640 1 Processor -- start 2024-04-25T00:17:37.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.953+0000 7f3a97c63640 1 -- start start 2024-04-25T00:17:37.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.957+0000 7f3a97c63640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3a9014c720 0x7f3a901563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:37.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.957+0000 7f3a97c63640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3a900a94a0 0x7f3a900c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:37.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.957+0000 7f3a97c63640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f3a900c2f50 0x7f3a90071680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:37.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.957+0000 7f3a97c63640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f3a90153460 con 0x7f3a900a94a0 2024-04-25T00:17:37.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.957+0000 7f3a97c63640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f3a90071220 con 0x7f3a900c2f50 2024-04-25T00:17:37.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.957+0000 7f3a97c63640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f3a90152420 con 0x7f3a9014c720 2024-04-25T00:17:37.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.957+0000 7f3a951fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3a900a94a0 0x7f3a900c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:37.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.957+0000 7f3a951fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3a900a94a0 0x7f3a900c2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36938/0 (socket says 172.21.15.116:36938) 2024-04-25T00:17:37.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.957+0000 7f3a951fe640 1 -- 172.21.15.116:0/1500942851 learned_addr learned my addr 172.21.15.116:0/1500942851 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:17:37.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.957+0000 7f3a959ff640 1 --2- 172.21.15.116:0/1500942851 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3a9014c720 0x7f3a901563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:37.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.957+0000 7f3a96200640 1 --2- 172.21.15.116:0/1500942851 >> v2:172.21.15.174:3300/0 conn(0x7f3a900c2f50 0x7f3a90071680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:37.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.957+0000 7f3a951fe640 1 -- 172.21.15.116:0/1500942851 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3a9014c720 msgr2=0x7f3a901563d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:37.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.957+0000 7f3a951fe640 1 --2- 172.21.15.116:0/1500942851 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3a9014c720 0x7f3a901563d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:37.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.957+0000 7f3a951fe640 1 -- 172.21.15.116:0/1500942851 >> v2:172.21.15.174:3300/0 conn(0x7f3a900c2f50 msgr2=0x7f3a90071680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:37.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.957+0000 7f3a951fe640 1 --2- 172.21.15.116:0/1500942851 >> v2:172.21.15.174:3300/0 conn(0x7f3a900c2f50 0x7f3a90071680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:37.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.957+0000 7f3a951fe640 1 -- 172.21.15.116:0/1500942851 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a9014cb00 con 0x7f3a900a94a0 2024-04-25T00:17:37.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.957+0000 7f3a951fe640 1 --2- 172.21.15.116:0/1500942851 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a900a94a0 0x7f3a900c2a10 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f3a900c35f0 tx=0x7f3a8001c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=427e385fcf5674e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:37.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.957+0000 7f3a949fd640 1 -- 172.21.15.116:0/1500942851 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a800b1070 con 0x7f3a900a94a0 2024-04-25T00:17:37.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.957+0000 7f3a949fd640 1 -- 172.21.15.116:0/1500942851 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a800ac640 con 0x7f3a900a94a0 2024-04-25T00:17:37.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.957+0000 7f3a949fd640 1 -- 172.21.15.116:0/1500942851 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a800aca10 con 0x7f3a900a94a0 2024-04-25T00:17:37.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.961+0000 7f3a97c63640 1 -- 172.21.15.116:0/1500942851 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a900a94a0 msgr2=0x7f3a900c2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:37.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.961+0000 7f3a97c63640 1 --2- 172.21.15.116:0/1500942851 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a900a94a0 0x7f3a900c2a10 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f3a900c35f0 tx=0x7f3a8001c460 comp rx=0 tx=0).stop 2024-04-25T00:17:37.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.961+0000 7f3a97c63640 1 -- 172.21.15.116:0/1500942851 shutdown_connections 2024-04-25T00:17:37.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.961+0000 7f3a97c63640 1 --2- 172.21.15.116:0/1500942851 >> v2:172.21.15.174:3300/0 conn(0x7f3a900c2f50 0x7f3a90071680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:37.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.961+0000 7f3a97c63640 1 --2- 172.21.15.116:0/1500942851 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a900a94a0 0x7f3a900c2a10 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:37.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.961+0000 7f3a97c63640 1 --2- 172.21.15.116:0/1500942851 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3a9014c720 0x7f3a901563d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:37.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.961+0000 7f3a97c63640 1 -- 172.21.15.116:0/1500942851 >> 172.21.15.116:0/1500942851 conn(0x7f3a900c3b00 msgr2=0x7f3a900c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:37.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.961+0000 7f3a97c63640 1 -- 172.21.15.116:0/1500942851 shutdown_connections 2024-04-25T00:17:37.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.961+0000 7f3a97c63640 1 -- 172.21.15.116:0/1500942851 wait complete. 2024-04-25T00:17:37.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.961+0000 7f3a97c63640 1 Processor -- start 2024-04-25T00:17:37.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.961+0000 7f3a97c63640 1 -- start start 2024-04-25T00:17:37.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.961+0000 7f3a97c63640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a900a94a0 0x7f3a900c0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:37.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.961+0000 7f3a97c63640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3a900c2f50 0x7f3a9013fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:37.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.961+0000 7f3a97c63640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a9014c720 0x7f3a901401c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:37.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.961+0000 7f3a97c63640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3a90071fd0 con 0x7f3a9014c720 2024-04-25T00:17:37.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.961+0000 7f3a97c63640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3a90152f40 con 0x7f3a900a94a0 2024-04-25T00:17:37.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.961+0000 7f3a97c63640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f3a90072150 con 0x7f3a900c2f50 2024-04-25T00:17:37.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.961+0000 7f3a96200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a9014c720 0x7f3a901401c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:37.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.961+0000 7f3a96200640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a9014c720 0x7f3a901401c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36954/0 (socket says 172.21.15.116:36954) 2024-04-25T00:17:37.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.961+0000 7f3a96200640 1 -- 172.21.15.116:0/556994521 learned_addr learned my addr 172.21.15.116:0/556994521 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:17:37.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.961+0000 7f3a959ff640 1 --2- 172.21.15.116:0/556994521 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a900a94a0 0x7f3a900c0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:37.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.961+0000 7f3a951fe640 1 --2- 172.21.15.116:0/556994521 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3a900c2f50 0x7f3a9013fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:37.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.961+0000 7f3a96200640 1 -- 172.21.15.116:0/556994521 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3a900c2f50 msgr2=0x7f3a9013fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:37.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.961+0000 7f3a96200640 1 --2- 172.21.15.116:0/556994521 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3a900c2f50 0x7f3a9013fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:37.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.961+0000 7f3a96200640 1 -- 172.21.15.116:0/556994521 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a900a94a0 msgr2=0x7f3a900c0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:37.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.961+0000 7f3a96200640 1 --2- 172.21.15.116:0/556994521 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a900a94a0 0x7f3a900c0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:37.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.961+0000 7f3a96200640 1 -- 172.21.15.116:0/556994521 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a901461f0 con 0x7f3a9014c720 2024-04-25T00:17:37.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.961+0000 7f3a951fe640 1 --2- 172.21.15.116:0/556994521 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3a900c2f50 0x7f3a9013fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:17:37.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.961+0000 7f3a96200640 1 --2- 172.21.15.116:0/556994521 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a9014c720 0x7f3a901401c0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f3a88007e40 tx=0x7f3a8800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:37.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.961+0000 7f3a867fc640 1 -- 172.21.15.116:0/556994521 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a88014620 con 0x7f3a9014c720 2024-04-25T00:17:37.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.961+0000 7f3a867fc640 1 -- 172.21.15.116:0/556994521 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a88019070 con 0x7f3a9014c720 2024-04-25T00:17:37.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.961+0000 7f3a97c63640 1 -- 172.21.15.116:0/556994521 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a901ef830 con 0x7f3a9014c720 2024-04-25T00:17:37.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.961+0000 7f3a867fc640 1 -- 172.21.15.116:0/556994521 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a88014a20 con 0x7f3a9014c720 2024-04-25T00:17:37.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.961+0000 7f3a97c63640 1 -- 172.21.15.116:0/556994521 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a901ef550 con 0x7f3a9014c720 2024-04-25T00:17:37.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.961+0000 7f3a97c63640 1 -- 172.21.15.116:0/556994521 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a90068350 con 0x7f3a9014c720 2024-04-25T00:17:37.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.965+0000 7f3a867fc640 1 -- 172.21.15.116:0/556994521 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3a88014bc0 con 0x7f3a9014c720 2024-04-25T00:17:37.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.965+0000 7f3a867fc640 1 --2- 172.21.15.116:0/556994521 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3a740678c0 0x7f3a74069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:37.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.965+0000 7f3a867fc640 1 -- 172.21.15.116:0/556994521 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 8816+0+0 (secure 0 0 0) 0x7f3a88087210 con 0x7f3a9014c720 2024-04-25T00:17:37.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.965+0000 7f3a959ff640 1 --2- 172.21.15.116:0/556994521 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3a740678c0 0x7f3a74069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:37.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.969+0000 7f3a959ff640 1 --2- 172.21.15.116:0/556994521 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3a740678c0 0x7f3a74069d80 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f3a9014b760 tx=0x7f3a8c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:37.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:37.969+0000 7f3a867fc640 1 -- 172.21.15.116:0/556994521 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f3a901461f0 con 0x7f3a9014c720 2024-04-25T00:17:38.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.161+0000 7f3a97c63640 1 -- 172.21.15.116:0/556994521 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3a900c3640 con 0x7f3a9014c720 2024-04-25T00:17:38.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.165+0000 7f3a867fc640 1 -- 172.21.15.116:0/556994521 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) ==== 95+0+5673 (secure 0 0 0) 0x7f3a880570e0 con 0x7f3a9014c720 2024-04-25T00:17:38.167 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:17:38.167 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":135,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:17:36.892730+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6343":{"gid":6343,"name":"d","rank":0,"incarnation":133,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.174:6837/1281466397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1281466397},{"type":"v1","addr":"172.21.15.174:6837","nonce":1281466397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:17:28.837517+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7164},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7164":{"gid":7164,"name":"c","rank":0,"incarnation":128,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6835/3336339376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3336339376},{"type":"v1","addr":"172.21.15.116:6835","nonce":3336339376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7164,"qdb_cluster":[7164]},"id":2}]} 2024-04-25T00:17:38.168 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 135 2024-04-25T00:17:38.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.165+0000 7f3a97c63640 1 -- 172.21.15.116:0/556994521 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3a740678c0 msgr2=0x7f3a74069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:38.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.165+0000 7f3a97c63640 1 --2- 172.21.15.116:0/556994521 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3a740678c0 0x7f3a74069d80 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f3a9014b760 tx=0x7f3a8c008040 comp rx=0 tx=0).stop 2024-04-25T00:17:38.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.165+0000 7f3a97c63640 1 -- 172.21.15.116:0/556994521 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a9014c720 msgr2=0x7f3a901401c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:38.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.165+0000 7f3a97c63640 1 --2- 172.21.15.116:0/556994521 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a9014c720 0x7f3a901401c0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f3a88007e40 tx=0x7f3a8800e4b0 comp rx=0 tx=0).stop 2024-04-25T00:17:38.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.165+0000 7f3a97c63640 1 -- 172.21.15.116:0/556994521 shutdown_connections 2024-04-25T00:17:38.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.165+0000 7f3a97c63640 1 --2- 172.21.15.116:0/556994521 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3a740678c0 0x7f3a74069d80 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:38.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.165+0000 7f3a97c63640 1 --2- 172.21.15.116:0/556994521 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a9014c720 0x7f3a901401c0 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:38.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.165+0000 7f3a97c63640 1 --2- 172.21.15.116:0/556994521 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3a900c2f50 0x7f3a9013fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:38.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.165+0000 7f3a97c63640 1 --2- 172.21.15.116:0/556994521 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a900a94a0 0x7f3a900c0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:38.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.165+0000 7f3a97c63640 1 -- 172.21.15.116:0/556994521 >> 172.21.15.116:0/556994521 conn(0x7f3a900c3b00 msgr2=0x7f3a900c3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:38.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.165+0000 7f3a97c63640 1 -- 172.21.15.116:0/556994521 shutdown_connections 2024-04-25T00:17:38.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.165+0000 7f3a97c63640 1 -- 172.21.15.116:0/556994521 wait complete. 2024-04-25T00:17:38.184 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2024-04-25T00:17:38.385 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:17:38.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.513+0000 7fdd1dca2640 1 Processor -- start 2024-04-25T00:17:38.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.513+0000 7fdd1dca2640 1 -- start start 2024-04-25T00:17:38.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.513+0000 7fdd1dca2640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fdd18148890 0x7fdd18072590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:38.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.513+0000 7fdd1dca2640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fdd18070dd0 0x7fdd18071270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:38.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.517+0000 7fdd1dca2640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd18152c10 con 0x7fdd1806e890 2024-04-25T00:17:38.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.517+0000 7fdd1dca2640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fdd18151e50 con 0x7fdd18148890 2024-04-25T00:17:38.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.517+0000 7fdd1dca2640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fdd18152130 con 0x7fdd18070dd0 2024-04-25T00:17:38.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.517+0000 7fdd177fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fdd18148890 0x7fdd18072590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:38.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.517+0000 7fdd177fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fdd18148890 0x7fdd18072590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36962/0 (socket says 172.21.15.116:36962) 2024-04-25T00:17:38.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.517+0000 7fdd177fe640 1 -- 172.21.15.116:0/3667644158 learned_addr learned my addr 172.21.15.116:0/3667644158 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:17:38.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.517+0000 7fdd0ffff640 1 --2- 172.21.15.116:0/3667644158 >> v2:172.21.15.174:3300/0 conn(0x7fdd18070dd0 0x7fdd18071270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:38.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.517+0000 7fdd177fe640 1 -- 172.21.15.116:0/3667644158 >> v2:172.21.15.174:3300/0 conn(0x7fdd18070dd0 msgr2=0x7fdd18071270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:38.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.517+0000 7fdd177fe640 1 --2- 172.21.15.116:0/3667644158 >> v2:172.21.15.174:3300/0 conn(0x7fdd18070dd0 0x7fdd18071270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:38.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.517+0000 7fdd177fe640 1 -- 172.21.15.116:0/3667644158 >> v1:172.21.15.174:6789/0 conn(0x7fdd1806e890 legacy=0x7fdd180717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:38.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.517+0000 7fdd177fe640 1 -- 172.21.15.116:0/3667644158 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd18152410 con 0x7fdd18148890 2024-04-25T00:17:38.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.517+0000 7fdd177fe640 1 --2- 172.21.15.116:0/3667644158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd18148890 0x7fdd18072590 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7fdd180711b0 tx=0x7fdd0801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f6bccb8deec8c6b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:38.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.517+0000 7fdd16ffd640 1 -- 172.21.15.116:0/3667644158 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd080b0070 con 0x7fdd18148890 2024-04-25T00:17:38.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.517+0000 7fdd16ffd640 1 -- 172.21.15.116:0/3667644158 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd080ab600 con 0x7fdd18148890 2024-04-25T00:17:38.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.517+0000 7fdd16ffd640 1 -- 172.21.15.116:0/3667644158 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd080ab9d0 con 0x7fdd18148890 2024-04-25T00:17:38.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.517+0000 7fdd1dca2640 1 -- 172.21.15.116:0/3667644158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd18148890 msgr2=0x7fdd18072590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:38.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.517+0000 7fdd1dca2640 1 --2- 172.21.15.116:0/3667644158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd18148890 0x7fdd18072590 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7fdd180711b0 tx=0x7fdd0801c520 comp rx=0 tx=0).stop 2024-04-25T00:17:38.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.517+0000 7fdd1dca2640 1 -- 172.21.15.116:0/3667644158 shutdown_connections 2024-04-25T00:17:38.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.517+0000 7fdd1dca2640 1 --2- 172.21.15.116:0/3667644158 >> v2:172.21.15.174:3300/0 conn(0x7fdd18070dd0 0x7fdd18071270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:38.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.517+0000 7fdd1dca2640 1 --2- 172.21.15.116:0/3667644158 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd18148890 0x7fdd18072590 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:38.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.517+0000 7fdd1dca2640 1 -- 172.21.15.116:0/3667644158 >> 172.21.15.116:0/3667644158 conn(0x7fdd180c3b00 msgr2=0x7fdd180c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:38.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.517+0000 7fdd1dca2640 1 -- 172.21.15.116:0/3667644158 shutdown_connections 2024-04-25T00:17:38.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.517+0000 7fdd1dca2640 1 -- 172.21.15.116:0/3667644158 wait complete. 2024-04-25T00:17:38.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.521+0000 7fdd1dca2640 1 Processor -- start 2024-04-25T00:17:38.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.521+0000 7fdd1dca2640 1 -- start start 2024-04-25T00:17:38.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.521+0000 7fdd1dca2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdd1806e890 0x7fdd181f2810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:38.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.521+0000 7fdd1dca2640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdd18070dd0 0x7fdd181ee890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:38.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.521+0000 7fdd1dca2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd18148890 0x7fdd18160c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:38.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.521+0000 7fdd1dca2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdd180ab920 con 0x7fdd18148890 2024-04-25T00:17:38.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.521+0000 7fdd1dca2640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fdd180ab7a0 con 0x7fdd1806e890 2024-04-25T00:17:38.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.521+0000 7fdd1dca2640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fdd180abaa0 con 0x7fdd18070dd0 2024-04-25T00:17:38.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.521+0000 7fdd17fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd18148890 0x7fdd18160c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:38.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.521+0000 7fdd17fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd18148890 0x7fdd18160c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36966/0 (socket says 172.21.15.116:36966) 2024-04-25T00:17:38.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.521+0000 7fdd17fff640 1 -- 172.21.15.116:0/1460074271 learned_addr learned my addr 172.21.15.116:0/1460074271 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:17:38.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.521+0000 7fdd177fe640 1 --2- 172.21.15.116:0/1460074271 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdd1806e890 0x7fdd181f2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:38.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.521+0000 7fdd0ffff640 1 --2- 172.21.15.116:0/1460074271 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdd18070dd0 0x7fdd181ee890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:38.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.521+0000 7fdd17fff640 1 -- 172.21.15.116:0/1460074271 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdd18070dd0 msgr2=0x7fdd181ee890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:38.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.521+0000 7fdd17fff640 1 --2- 172.21.15.116:0/1460074271 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdd18070dd0 0x7fdd181ee890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:38.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.521+0000 7fdd17fff640 1 -- 172.21.15.116:0/1460074271 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdd1806e890 msgr2=0x7fdd181f2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:38.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.521+0000 7fdd17fff640 1 --2- 172.21.15.116:0/1460074271 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdd1806e890 0x7fdd181f2810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:38.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.521+0000 7fdd17fff640 1 -- 172.21.15.116:0/1460074271 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd1813b950 con 0x7fdd18148890 2024-04-25T00:17:38.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.521+0000 7fdd17fff640 1 --2- 172.21.15.116:0/1460074271 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd18148890 0x7fdd18160c50 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7fdd1000c7a0 tx=0x7fdd1000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:38.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.521+0000 7fdd14ff9640 1 -- 172.21.15.116:0/1460074271 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd10003b80 con 0x7fdd18148890 2024-04-25T00:17:38.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.521+0000 7fdd14ff9640 1 -- 172.21.15.116:0/1460074271 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd10003d20 con 0x7fdd18148890 2024-04-25T00:17:38.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.521+0000 7fdd14ff9640 1 -- 172.21.15.116:0/1460074271 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd1000aa30 con 0x7fdd18148890 2024-04-25T00:17:38.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.521+0000 7fdd1dca2640 1 -- 172.21.15.116:0/1460074271 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd1813bd40 con 0x7fdd18148890 2024-04-25T00:17:38.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.521+0000 7fdd1dca2640 1 -- 172.21.15.116:0/1460074271 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd181f2010 con 0x7fdd18148890 2024-04-25T00:17:38.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.521+0000 7fdd1dca2640 1 -- 172.21.15.116:0/1460074271 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd181f1e00 con 0x7fdd18148890 2024-04-25T00:17:38.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.525+0000 7fdd14ff9640 1 -- 172.21.15.116:0/1460074271 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fdd1000abd0 con 0x7fdd18148890 2024-04-25T00:17:38.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.525+0000 7fdd14ff9640 1 --2- 172.21.15.116:0/1460074271 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdcf80678c0 0x7fdcf8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:38.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.525+0000 7fdd14ff9640 1 -- 172.21.15.116:0/1460074271 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 8816+0+0 (secure 0 0 0) 0x7fdd10086790 con 0x7fdd18148890 2024-04-25T00:17:38.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.525+0000 7fdd177fe640 1 --2- 172.21.15.116:0/1460074271 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdcf80678c0 0x7fdcf8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:38.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.525+0000 7fdd177fe640 1 --2- 172.21.15.116:0/1460074271 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdcf80678c0 0x7fdcf8069d80 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fdd080ad000 tx=0x7fdd08008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:38.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.529+0000 7fdd14ff9640 1 -- 172.21.15.116:0/1460074271 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fdd100566f0 con 0x7fdd18148890 2024-04-25T00:17:38.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.725+0000 7fdd1dca2640 1 -- 172.21.15.116:0/1460074271 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdd18068350 con 0x7fdd18148890 2024-04-25T00:17:38.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.725+0000 7fdd14ff9640 1 -- 172.21.15.116:0/1460074271 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) ==== 95+0+5673 (secure 0 0 0) 0x7fdd10055ff0 con 0x7fdd18148890 2024-04-25T00:17:38.726 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:17:38.727 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":135,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:17:36.892730+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6343":{"gid":6343,"name":"d","rank":0,"incarnation":133,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.174:6837/1281466397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1281466397},{"type":"v1","addr":"172.21.15.174:6837","nonce":1281466397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:17:28.837517+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7164},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7164":{"gid":7164,"name":"c","rank":0,"incarnation":128,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6835/3336339376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3336339376},{"type":"v1","addr":"172.21.15.116:6835","nonce":3336339376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7164,"qdb_cluster":[7164]},"id":2}]} 2024-04-25T00:17:38.727 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 135 2024-04-25T00:17:38.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.729+0000 7fdd1dca2640 1 -- 172.21.15.116:0/1460074271 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdcf80678c0 msgr2=0x7fdcf8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:38.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.729+0000 7fdd1dca2640 1 --2- 172.21.15.116:0/1460074271 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdcf80678c0 0x7fdcf8069d80 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fdd080ad000 tx=0x7fdd08008040 comp rx=0 tx=0).stop 2024-04-25T00:17:38.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.729+0000 7fdd1dca2640 1 -- 172.21.15.116:0/1460074271 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd18148890 msgr2=0x7fdd18160c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:38.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.729+0000 7fdd1dca2640 1 --2- 172.21.15.116:0/1460074271 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd18148890 0x7fdd18160c50 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7fdd1000c7a0 tx=0x7fdd1000cc60 comp rx=0 tx=0).stop 2024-04-25T00:17:38.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.729+0000 7fdd1dca2640 1 -- 172.21.15.116:0/1460074271 shutdown_connections 2024-04-25T00:17:38.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.729+0000 7fdd1dca2640 1 --2- 172.21.15.116:0/1460074271 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdcf80678c0 0x7fdcf8069d80 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:38.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.729+0000 7fdd1dca2640 1 --2- 172.21.15.116:0/1460074271 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdd18148890 0x7fdd18160c50 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:38.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.729+0000 7fdd1dca2640 1 --2- 172.21.15.116:0/1460074271 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdd18070dd0 0x7fdd181ee890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:38.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.729+0000 7fdd1dca2640 1 --2- 172.21.15.116:0/1460074271 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdd1806e890 0x7fdd181f2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:38.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.729+0000 7fdd1dca2640 1 -- 172.21.15.116:0/1460074271 >> 172.21.15.116:0/1460074271 conn(0x7fdd180c3b00 msgr2=0x7fdd180aa560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:38.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.729+0000 7fdd1dca2640 1 -- 172.21.15.116:0/1460074271 shutdown_connections 2024-04-25T00:17:38.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:38.729+0000 7fdd1dca2640 1 -- 172.21.15.116:0/1460074271 wait complete. 2024-04-25T00:17:38.746 INFO:tasks.mds_thrash.fs.[a]:no change 2024-04-25T00:17:39.150 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:17:39.153 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:17:40.185 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:17:40.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.325+0000 7f028359e640 1 Processor -- start 2024-04-25T00:17:40.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.325+0000 7f028359e640 1 -- start start 2024-04-25T00:17:40.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.325+0000 7f028359e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0284152df0 0x7f0284156170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:40.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.325+0000 7f028359e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f02840c8e70 0x7f02840c58f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:40.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.325+0000 7f028359e640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f02840c5e30 0x7f028406c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:40.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.325+0000 7f028359e640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f028406c250 con 0x7f02840c8e70 2024-04-25T00:17:40.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.325+0000 7f028359e640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f02840c7980 con 0x7f02840c5e30 2024-04-25T00:17:40.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.325+0000 7f028359e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f02840c7c60 con 0x7f0284152df0 2024-04-25T00:17:40.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.325+0000 7f0281d9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f02840c8e70 0x7f02840c58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:40.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.325+0000 7f0281d9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f02840c8e70 0x7f02840c58f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36976/0 (socket says 172.21.15.116:36976) 2024-04-25T00:17:40.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.325+0000 7f0281d9b640 1 -- 172.21.15.116:0/1718684582 learned_addr learned my addr 172.21.15.116:0/1718684582 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:17:40.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.325+0000 7f028259c640 1 --2- 172.21.15.116:0/1718684582 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0284152df0 0x7f0284156170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:40.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.325+0000 7f0282d9d640 1 --2- 172.21.15.116:0/1718684582 >> v2:172.21.15.174:3300/0 conn(0x7f02840c5e30 0x7f028406c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:40.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.325+0000 7f0281d9b640 1 -- 172.21.15.116:0/1718684582 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0284152df0 msgr2=0x7f0284156170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:40.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.325+0000 7f0281d9b640 1 --2- 172.21.15.116:0/1718684582 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0284152df0 0x7f0284156170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:40.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.325+0000 7f0281d9b640 1 -- 172.21.15.116:0/1718684582 >> v2:172.21.15.174:3300/0 conn(0x7f02840c5e30 msgr2=0x7f028406c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:40.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.325+0000 7f0281d9b640 1 --2- 172.21.15.116:0/1718684582 >> v2:172.21.15.174:3300/0 conn(0x7f02840c5e30 0x7f028406c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:40.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.325+0000 7f0281d9b640 1 -- 172.21.15.116:0/1718684582 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02840c7f40 con 0x7f02840c8e70 2024-04-25T00:17:40.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.329+0000 7f0281d9b640 1 --2- 172.21.15.116:0/1718684582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02840c8e70 0x7f02840c58f0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f028406bc90 tx=0x7f027801c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e6993ea71367431 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:40.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.329+0000 7f028159a640 1 -- 172.21.15.116:0/1718684582 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02780b0070 con 0x7f02840c8e70 2024-04-25T00:17:40.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.329+0000 7f028159a640 1 -- 172.21.15.116:0/1718684582 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02780ab600 con 0x7f02840c8e70 2024-04-25T00:17:40.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.329+0000 7f028159a640 1 -- 172.21.15.116:0/1718684582 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02780ab9d0 con 0x7f02840c8e70 2024-04-25T00:17:40.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.329+0000 7f028359e640 1 -- 172.21.15.116:0/1718684582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02840c8e70 msgr2=0x7f02840c58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:40.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.329+0000 7f028359e640 1 --2- 172.21.15.116:0/1718684582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02840c8e70 0x7f02840c58f0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f028406bc90 tx=0x7f027801c400 comp rx=0 tx=0).stop 2024-04-25T00:17:40.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.329+0000 7f028359e640 1 -- 172.21.15.116:0/1718684582 shutdown_connections 2024-04-25T00:17:40.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.329+0000 7f028359e640 1 --2- 172.21.15.116:0/1718684582 >> v2:172.21.15.174:3300/0 conn(0x7f02840c5e30 0x7f028406c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:40.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.329+0000 7f028359e640 1 --2- 172.21.15.116:0/1718684582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02840c8e70 0x7f02840c58f0 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:40.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.329+0000 7f028359e640 1 --2- 172.21.15.116:0/1718684582 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0284152df0 0x7f0284156170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:40.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.329+0000 7f028359e640 1 -- 172.21.15.116:0/1718684582 >> 172.21.15.116:0/1718684582 conn(0x7f0284074e60 msgr2=0x7f02840caf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:40.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.329+0000 7f028359e640 1 -- 172.21.15.116:0/1718684582 shutdown_connections 2024-04-25T00:17:40.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.329+0000 7f028359e640 1 -- 172.21.15.116:0/1718684582 wait complete. 2024-04-25T00:17:40.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.329+0000 7f028359e640 1 Processor -- start 2024-04-25T00:17:40.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.329+0000 7f028359e640 1 -- start start 2024-04-25T00:17:40.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.333+0000 7f028359e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f02840c5e30 0x7f02840c2040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:40.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.333+0000 7f028359e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02840c8e70 0x7f02840abb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:40.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.333+0000 7f028359e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0284152df0 0x7f028414b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:40.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.333+0000 7f028359e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0284159070 con 0x7f02840c8e70 2024-04-25T00:17:40.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.333+0000 7f028359e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0284158ef0 con 0x7f0284152df0 2024-04-25T00:17:40.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.333+0000 7f028359e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f02841591f0 con 0x7f02840c5e30 2024-04-25T00:17:40.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.333+0000 7f0281d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02840c8e70 0x7f02840abb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:40.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.333+0000 7f0281d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02840c8e70 0x7f02840abb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36988/0 (socket says 172.21.15.116:36988) 2024-04-25T00:17:40.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.333+0000 7f0281d9b640 1 -- 172.21.15.116:0/3340881971 learned_addr learned my addr 172.21.15.116:0/3340881971 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:17:40.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.333+0000 7f028259c640 1 --2- 172.21.15.116:0/3340881971 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f02840c5e30 0x7f02840c2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:40.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.333+0000 7f0282d9d640 1 --2- 172.21.15.116:0/3340881971 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0284152df0 0x7f028414b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:40.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.333+0000 7f0281d9b640 1 -- 172.21.15.116:0/3340881971 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f02840c5e30 msgr2=0x7f02840c2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:40.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.333+0000 7f0281d9b640 1 --2- 172.21.15.116:0/3340881971 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f02840c5e30 0x7f02840c2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:40.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.333+0000 7f0281d9b640 1 -- 172.21.15.116:0/3340881971 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0284152df0 msgr2=0x7f028414b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:40.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.333+0000 7f0281d9b640 1 --2- 172.21.15.116:0/3340881971 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0284152df0 0x7f028414b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:40.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.333+0000 7f0281d9b640 1 -- 172.21.15.116:0/3340881971 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02840c1db0 con 0x7f02840c8e70 2024-04-25T00:17:40.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.333+0000 7f0281d9b640 1 --2- 172.21.15.116:0/3340881971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02840c8e70 0x7f02840abb80 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f027801aab0 tx=0x7f02780191a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:40.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.333+0000 7f0272ffd640 1 -- 172.21.15.116:0/3340881971 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02780b0040 con 0x7f02840c8e70 2024-04-25T00:17:40.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.333+0000 7f0272ffd640 1 -- 172.21.15.116:0/3340881971 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02780b4420 con 0x7f02840c8e70 2024-04-25T00:17:40.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.333+0000 7f028359e640 1 -- 172.21.15.116:0/3340881971 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f028414cb90 con 0x7f02840c8e70 2024-04-25T00:17:40.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.333+0000 7f0272ffd640 1 -- 172.21.15.116:0/3340881971 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02780b47f0 con 0x7f02840c8e70 2024-04-25T00:17:40.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.333+0000 7f028359e640 1 -- 172.21.15.116:0/3340881971 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f028414c8b0 con 0x7f02840c8e70 2024-04-25T00:17:40.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.333+0000 7f028359e640 1 -- 172.21.15.116:0/3340881971 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f028404f510 con 0x7f02840c8e70 2024-04-25T00:17:40.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.333+0000 7f0272ffd640 1 -- 172.21.15.116:0/3340881971 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f02780b4990 con 0x7f02840c8e70 2024-04-25T00:17:40.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.333+0000 7f0272ffd640 1 --2- 172.21.15.116:0/3340881971 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f02640678c0 0x7f0264069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:40.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.333+0000 7f0272ffd640 1 -- 172.21.15.116:0/3340881971 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 8816+0+0 (secure 0 0 0) 0x7f02781203f0 con 0x7f02840c8e70 2024-04-25T00:17:40.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.333+0000 7f028259c640 1 --2- 172.21.15.116:0/3340881971 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f02640678c0 0x7f0264069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:40.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.337+0000 7f028259c640 1 --2- 172.21.15.116:0/3340881971 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f02640678c0 0x7f0264069d80 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f02841557a0 tx=0x7f0274009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:40.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.341+0000 7f0272ffd640 1 -- 172.21.15.116:0/3340881971 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f0278016110 con 0x7f02840c8e70 2024-04-25T00:17:40.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.573+0000 7f028359e640 1 -- 172.21.15.116:0/3340881971 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f02840b0ca0 con 0x7f02840c8e70 2024-04-25T00:17:40.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.577+0000 7f0272ffd640 1 -- 172.21.15.116:0/3340881971 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 136 v136) ==== 95+0+5680 (secure 0 0 0) 0x7f02780efd00 con 0x7f02840c8e70 2024-04-25T00:17:40.576 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:17:40.576 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":136,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:17:38.902992+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6343":{"gid":6343,"name":"d","rank":0,"incarnation":133,"state":"up:active","state_seq":4,"addr":"172.21.15.174:6837/1281466397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1281466397},{"type":"v1","addr":"172.21.15.174:6837","nonce":1281466397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6343,"qdb_cluster":[6343]},"id":1},{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:17:28.837517+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7164},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7164":{"gid":7164,"name":"c","rank":0,"incarnation":128,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6835/3336339376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3336339376},{"type":"v1","addr":"172.21.15.116:6835","nonce":3336339376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7164,"qdb_cluster":[7164]},"id":2}]} 2024-04-25T00:17:40.577 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 136 2024-04-25T00:17:40.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.577+0000 7f028359e640 1 -- 172.21.15.116:0/3340881971 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f02640678c0 msgr2=0x7f0264069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:40.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.577+0000 7f028359e640 1 --2- 172.21.15.116:0/3340881971 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f02640678c0 0x7f0264069d80 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f02841557a0 tx=0x7f0274009190 comp rx=0 tx=0).stop 2024-04-25T00:17:40.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.577+0000 7f028359e640 1 -- 172.21.15.116:0/3340881971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02840c8e70 msgr2=0x7f02840abb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:40.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.577+0000 7f028359e640 1 --2- 172.21.15.116:0/3340881971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02840c8e70 0x7f02840abb80 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f027801aab0 tx=0x7f02780191a0 comp rx=0 tx=0).stop 2024-04-25T00:17:40.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.577+0000 7f028359e640 1 -- 172.21.15.116:0/3340881971 shutdown_connections 2024-04-25T00:17:40.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.577+0000 7f028359e640 1 --2- 172.21.15.116:0/3340881971 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f02640678c0 0x7f0264069d80 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:40.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.577+0000 7f028359e640 1 --2- 172.21.15.116:0/3340881971 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0284152df0 0x7f028414b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:40.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.577+0000 7f028359e640 1 --2- 172.21.15.116:0/3340881971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02840c8e70 0x7f02840abb80 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:40.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.577+0000 7f028359e640 1 --2- 172.21.15.116:0/3340881971 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f02840c5e30 0x7f02840c2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:40.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.577+0000 7f028359e640 1 -- 172.21.15.116:0/3340881971 >> 172.21.15.116:0/3340881971 conn(0x7f0284074e60 msgr2=0x7f02841519d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:40.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.577+0000 7f028359e640 1 -- 172.21.15.116:0/3340881971 shutdown_connections 2024-04-25T00:17:40.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.577+0000 7f028359e640 1 -- 172.21.15.116:0/3340881971 wait complete. 2024-04-25T00:17:40.597 INFO:tasks.mds_thrash.fs.[b]:mds.d reported in up:active state 2024-04-25T00:17:40.597 INFO:tasks.mds_thrash.fs.[b]:waiting for 38 secs before thrashing 2024-04-25T00:17:40.748 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:17:40.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.865+0000 7f8ca6341640 1 Processor -- start 2024-04-25T00:17:40.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.865+0000 7f8ca6341640 1 -- start start 2024-04-25T00:17:40.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.865+0000 7f8ca6341640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8ca00acef0 0x7f8ca01541d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:40.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.865+0000 7f8ca6341640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8ca00c32d0 0x7f8ca00c3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:40.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.865+0000 7f8ca6341640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ca00a9ef0 con 0x7f8ca00ad2d0 2024-04-25T00:17:40.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.865+0000 7f8ca6341640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f8ca00c2b60 con 0x7f8ca00c32d0 2024-04-25T00:17:40.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.865+0000 7f8ca6341640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f8ca00c2e40 con 0x7f8ca00acef0 2024-04-25T00:17:40.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.865+0000 7f8ca48de640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8ca00c32d0 0x7f8ca00c3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:40.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.865+0000 7f8ca48de640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8ca00c32d0 0x7f8ca00c3740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36990/0 (socket says 172.21.15.116:36990) 2024-04-25T00:17:40.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.865+0000 7f8ca48de640 1 -- 172.21.15.116:0/3540262796 learned_addr learned my addr 172.21.15.116:0/3540262796 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:17:40.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.865+0000 7f8c9ffff640 1 --2- 172.21.15.116:0/3540262796 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8ca00acef0 0x7f8ca01541d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:40.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.869+0000 7f8ca48de640 1 -- 172.21.15.116:0/3540262796 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8ca00acef0 msgr2=0x7f8ca01541d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:40.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.869+0000 7f8ca48de640 1 --2- 172.21.15.116:0/3540262796 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8ca00acef0 0x7f8ca01541d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:40.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.869+0000 7f8ca48de640 1 -- 172.21.15.116:0/3540262796 >> v1:172.21.15.174:6789/0 conn(0x7f8ca00ad2d0 legacy=0x7f8ca00a7d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:40.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.869+0000 7f8ca48de640 1 -- 172.21.15.116:0/3540262796 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ca00aa1d0 con 0x7f8ca00c32d0 2024-04-25T00:17:40.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.869+0000 7f8ca48de640 1 --2- 172.21.15.116:0/3540262796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ca00c32d0 0x7f8ca00c3740 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f8c98009e50 tx=0x7f8c9801c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cd6a2f02b6d90d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:40.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.869+0000 7f8c9f7fe640 1 -- 172.21.15.116:0/3540262796 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c980a5750 con 0x7f8ca00c32d0 2024-04-25T00:17:40.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.869+0000 7f8c9f7fe640 1 -- 172.21.15.116:0/3540262796 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c980a58f0 con 0x7f8ca00c32d0 2024-04-25T00:17:40.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.869+0000 7f8c9f7fe640 1 -- 172.21.15.116:0/3540262796 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c980a5cc0 con 0x7f8ca00c32d0 2024-04-25T00:17:40.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.869+0000 7f8ca6341640 1 -- 172.21.15.116:0/3540262796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ca00c32d0 msgr2=0x7f8ca00c3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:40.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.869+0000 7f8ca6341640 1 --2- 172.21.15.116:0/3540262796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ca00c32d0 0x7f8ca00c3740 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f8c98009e50 tx=0x7f8c9801c9a0 comp rx=0 tx=0).stop 2024-04-25T00:17:40.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.869+0000 7f8ca6341640 1 -- 172.21.15.116:0/3540262796 shutdown_connections 2024-04-25T00:17:40.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.869+0000 7f8ca6341640 1 --2- 172.21.15.116:0/3540262796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ca00c32d0 0x7f8ca00c3740 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:40.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.869+0000 7f8ca6341640 1 --2- 172.21.15.116:0/3540262796 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8ca00acef0 0x7f8ca01541d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:40.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.869+0000 7f8ca6341640 1 -- 172.21.15.116:0/3540262796 >> 172.21.15.116:0/3540262796 conn(0x7f8ca00c4200 msgr2=0x7f8ca00c4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:40.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.869+0000 7f8ca6341640 1 -- 172.21.15.116:0/3540262796 shutdown_connections 2024-04-25T00:17:40.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.869+0000 7f8ca6341640 1 -- 172.21.15.116:0/3540262796 wait complete. 2024-04-25T00:17:40.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.873+0000 7f8ca6341640 1 Processor -- start 2024-04-25T00:17:40.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.873+0000 7f8ca6341640 1 -- start start 2024-04-25T00:17:40.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.873+0000 7f8ca6341640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8ca00acef0 0x7f8ca01507c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:40.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.873+0000 7f8ca6341640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ca00ad2d0 0x7f8ca014e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:40.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.873+0000 7f8ca6341640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ca00c32d0 0x7f8ca014ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:40.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.873+0000 7f8ca6341640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8ca00a8610 con 0x7f8ca00c32d0 2024-04-25T00:17:40.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.873+0000 7f8ca6341640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8ca00a8490 con 0x7f8ca00ad2d0 2024-04-25T00:17:40.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.873+0000 7f8ca6341640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f8ca0160860 con 0x7f8ca00acef0 2024-04-25T00:17:40.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.873+0000 7f8ca48de640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ca00c32d0 0x7f8ca014ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:40.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.873+0000 7f8ca48de640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ca00c32d0 0x7f8ca014ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36998/0 (socket says 172.21.15.116:36998) 2024-04-25T00:17:40.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.873+0000 7f8ca48de640 1 -- 172.21.15.116:0/772778922 learned_addr learned my addr 172.21.15.116:0/772778922 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:17:40.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.873+0000 7f8c9ffff640 1 --2- 172.21.15.116:0/772778922 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8ca00acef0 0x7f8ca01507c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:40.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.873+0000 7f8c97fff640 1 --2- 172.21.15.116:0/772778922 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ca00ad2d0 0x7f8ca014e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:40.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.873+0000 7f8ca48de640 1 -- 172.21.15.116:0/772778922 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8ca00acef0 msgr2=0x7f8ca01507c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:40.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.873+0000 7f8ca48de640 1 --2- 172.21.15.116:0/772778922 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8ca00acef0 0x7f8ca01507c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:40.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.873+0000 7f8ca48de640 1 -- 172.21.15.116:0/772778922 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ca00ad2d0 msgr2=0x7f8ca014e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:40.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.873+0000 7f8ca48de640 1 --2- 172.21.15.116:0/772778922 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ca00ad2d0 0x7f8ca014e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:40.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.873+0000 7f8ca48de640 1 -- 172.21.15.116:0/772778922 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ca015ce80 con 0x7f8ca00c32d0 2024-04-25T00:17:40.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.873+0000 7f8ca48de640 1 --2- 172.21.15.116:0/772778922 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ca00c32d0 0x7f8ca014ea10 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f8c980af040 tx=0x7f8c98019740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:40.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.873+0000 7f8c9d7fa640 1 -- 172.21.15.116:0/772778922 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c980bc020 con 0x7f8ca00c32d0 2024-04-25T00:17:40.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.873+0000 7f8c9d7fa640 1 -- 172.21.15.116:0/772778922 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c980b8670 con 0x7f8ca00c32d0 2024-04-25T00:17:40.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.873+0000 7f8ca6341640 1 -- 172.21.15.116:0/772778922 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ca01f2da0 con 0x7f8ca00c32d0 2024-04-25T00:17:40.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.873+0000 7f8c9d7fa640 1 -- 172.21.15.116:0/772778922 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c98009510 con 0x7f8ca00c32d0 2024-04-25T00:17:40.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.873+0000 7f8ca6341640 1 -- 172.21.15.116:0/772778922 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ca0154e90 con 0x7f8ca00c32d0 2024-04-25T00:17:40.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.873+0000 7f8ca6341640 1 -- 172.21.15.116:0/772778922 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ca0068a60 con 0x7f8ca00c32d0 2024-04-25T00:17:40.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.877+0000 7f8c9d7fa640 1 -- 172.21.15.116:0/772778922 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8c980bf020 con 0x7f8ca00c32d0 2024-04-25T00:17:40.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.877+0000 7f8c9d7fa640 1 --2- 172.21.15.116:0/772778922 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8c800678f0 0x7f8c80069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:40.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.877+0000 7f8c9d7fa640 1 -- 172.21.15.116:0/772778922 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 8816+0+0 (secure 0 0 0) 0x7f8c98121720 con 0x7f8ca00c32d0 2024-04-25T00:17:40.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.877+0000 7f8c9ffff640 1 --2- 172.21.15.116:0/772778922 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8c800678f0 0x7f8c80069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:40.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.877+0000 7f8c9ffff640 1 --2- 172.21.15.116:0/772778922 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8c800678f0 0x7f8c80069db0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f8ca0153770 tx=0x7f8c90009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:40.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:40.881+0000 7f8c9d7fa640 1 -- 172.21.15.116:0/772778922 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f8ca015ce80 con 0x7f8ca00c32d0 2024-04-25T00:17:41.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:41.137+0000 7f8ca6341640 1 -- 172.21.15.116:0/772778922 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8ca0074920 con 0x7f8ca00c32d0 2024-04-25T00:17:41.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:41.137+0000 7f8c9d7fa640 1 -- 172.21.15.116:0/772778922 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 136 v136) ==== 95+0+5680 (secure 0 0 0) 0x7f8c98016620 con 0x7f8ca00c32d0 2024-04-25T00:17:41.138 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:17:41.138 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":136,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:17:38.902992+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6343":{"gid":6343,"name":"d","rank":0,"incarnation":133,"state":"up:active","state_seq":4,"addr":"172.21.15.174:6837/1281466397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1281466397},{"type":"v1","addr":"172.21.15.174:6837","nonce":1281466397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6343,"qdb_cluster":[6343]},"id":1},{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:17:28.837517+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7164},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7164":{"gid":7164,"name":"c","rank":0,"incarnation":128,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6835/3336339376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3336339376},{"type":"v1","addr":"172.21.15.116:6835","nonce":3336339376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7164,"qdb_cluster":[7164]},"id":2}]} 2024-04-25T00:17:41.138 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 136 2024-04-25T00:17:41.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:41.141+0000 7f8ca6341640 1 -- 172.21.15.116:0/772778922 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8c800678f0 msgr2=0x7f8c80069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:41.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:41.141+0000 7f8ca6341640 1 --2- 172.21.15.116:0/772778922 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8c800678f0 0x7f8c80069db0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f8ca0153770 tx=0x7f8c90009190 comp rx=0 tx=0).stop 2024-04-25T00:17:41.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:41.141+0000 7f8ca6341640 1 -- 172.21.15.116:0/772778922 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ca00c32d0 msgr2=0x7f8ca014ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:41.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:41.141+0000 7f8ca6341640 1 --2- 172.21.15.116:0/772778922 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ca00c32d0 0x7f8ca014ea10 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f8c980af040 tx=0x7f8c98019740 comp rx=0 tx=0).stop 2024-04-25T00:17:41.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:41.141+0000 7f8ca6341640 1 -- 172.21.15.116:0/772778922 shutdown_connections 2024-04-25T00:17:41.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:41.141+0000 7f8ca6341640 1 --2- 172.21.15.116:0/772778922 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8c800678f0 0x7f8c80069db0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:41.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:41.141+0000 7f8ca6341640 1 --2- 172.21.15.116:0/772778922 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ca00c32d0 0x7f8ca014ea10 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:41.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:41.141+0000 7f8ca6341640 1 --2- 172.21.15.116:0/772778922 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8ca00ad2d0 0x7f8ca014e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:41.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:41.141+0000 7f8ca6341640 1 --2- 172.21.15.116:0/772778922 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8ca00acef0 0x7f8ca01507c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:41.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:41.141+0000 7f8ca6341640 1 -- 172.21.15.116:0/772778922 >> 172.21.15.116:0/772778922 conn(0x7f8ca00c4200 msgr2=0x7f8ca006c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:41.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:41.141+0000 7f8ca6341640 1 -- 172.21.15.116:0/772778922 shutdown_connections 2024-04-25T00:17:41.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:41.141+0000 7f8ca6341640 1 -- 172.21.15.116:0/772778922 wait complete. 2024-04-25T00:17:41.159 INFO:tasks.mds_thrash.fs.[a]:mds.d has gained rank=0, replacing gid=7119 2024-04-25T00:17:41.159 INFO:tasks.mds_thrash.fs.[a]:waiting for 7 secs before reviving mds.a 2024-04-25T00:17:43.893 INFO:tasks.ceph.mds.d.smithi174.stderr:2024-04-25T00:17:43.891+0000 7fab2cd17640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T00:17:48.160 INFO:tasks.mds_thrash.fs.[a]:reviving mds.a 2024-04-25T00:17:48.160 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-25T00:17:48.160 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-25T00:17:48.163 INFO:tasks.ceph.mds.a:Started 2024-04-25T00:17:48.163 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:17:48.247 INFO:tasks.ceph.mds.a.smithi116.stdout:starting mds.a at 2024-04-25T00:17:48.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.300+0000 7f1cd9e25640 1 Processor -- start 2024-04-25T00:17:48.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.300+0000 7f1cd9e25640 1 -- start start 2024-04-25T00:17:48.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.300+0000 7f1cd9e25640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1cd41599a0 0x7f1cd41562d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:48.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.300+0000 7f1cd9e25640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1cd4073200 con 0x7f1cd4152e60 2024-04-25T00:17:48.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.300+0000 7f1cd9e25640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1cd4158ec0 con 0x7f1cd40c5010 2024-04-25T00:17:48.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.300+0000 7f1cd9e25640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f1cd4153240 con 0x7f1cd41599a0 2024-04-25T00:17:48.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.300+0000 7f1ccbfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1cd41599a0 0x7f1cd41562d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:48.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.300+0000 7f1cd37fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f1cd4152e60 0x7f1cd40c36e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:41542/0 (socket says 172.21.15.116:41542) 2024-04-25T00:17:48.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.300+0000 7f1cd37fe640 1 -- 172.21.15.116:0/1075053767 learned_addr learned my addr 172.21.15.116:0/1075053767 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:17:48.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.300+0000 7f1ccbfff640 1 -- 172.21.15.116:0/1075053767 >> v1:172.21.15.174:6789/0 conn(0x7f1cd40c5010 legacy=0x7f1cd4160120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:48.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.300+0000 7f1ccbfff640 1 -- 172.21.15.116:0/1075053767 >> v1:172.21.15.116:6789/0 conn(0x7f1cd4152e60 legacy=0x7f1cd40c36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:48.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.300+0000 7f1cd2ffd640 1 -- 172.21.15.116:0/1075053767 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3007067446 0 0) 0x7f1cd4073200 con 0x7f1cd4152e60 2024-04-25T00:17:48.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.300+0000 7f1ccbfff640 1 -- 172.21.15.116:0/1075053767 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cd41591a0 con 0x7f1cd41599a0 2024-04-25T00:17:48.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.304+0000 7f1ccbfff640 1 --2- 172.21.15.116:0/1075053767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cd41599a0 0x7f1cd41562d0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f1cbc0097c0 tx=0x7f1cbc01c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a482968de0cca356 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:48.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.304+0000 7f1cd2ffd640 1 -- 172.21.15.116:0/1075053767 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cbc0b0070 con 0x7f1cd41599a0 2024-04-25T00:17:48.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.304+0000 7f1cd2ffd640 1 -- 172.21.15.116:0/1075053767 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1cbc0ab600 con 0x7f1cd41599a0 2024-04-25T00:17:48.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.304+0000 7f1cd2ffd640 1 -- 172.21.15.116:0/1075053767 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cbc0a4e60 con 0x7f1cd41599a0 2024-04-25T00:17:48.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.304+0000 7f1cd9e25640 1 -- 172.21.15.116:0/1075053767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cd41599a0 msgr2=0x7f1cd41562d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:48.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.304+0000 7f1cd9e25640 1 --2- 172.21.15.116:0/1075053767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cd41599a0 0x7f1cd41562d0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f1cbc0097c0 tx=0x7f1cbc01c3d0 comp rx=0 tx=0).stop 2024-04-25T00:17:48.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.304+0000 7f1cd9e25640 1 -- 172.21.15.116:0/1075053767 shutdown_connections 2024-04-25T00:17:48.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.304+0000 7f1cd9e25640 1 --2- 172.21.15.116:0/1075053767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cd41599a0 0x7f1cd41562d0 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:48.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.304+0000 7f1cd9e25640 1 -- 172.21.15.116:0/1075053767 >> 172.21.15.116:0/1075053767 conn(0x7f1cd406c720 msgr2=0x7f1cd406cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:48.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.304+0000 7f1cd9e25640 1 -- 172.21.15.116:0/1075053767 shutdown_connections 2024-04-25T00:17:48.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.304+0000 7f1cd9e25640 1 -- 172.21.15.116:0/1075053767 wait complete. 2024-04-25T00:17:48.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.304+0000 7f1cd9e25640 1 Processor -- start 2024-04-25T00:17:48.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.308+0000 7f1cd9e25640 1 -- start start 2024-04-25T00:17:48.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.308+0000 7f1cd9e25640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1cd40c5010 0x7f1cd41ef0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:48.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.308+0000 7f1cd9e25640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cd4152e60 0x7f1cd40ac2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:48.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.308+0000 7f1cd9e25640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1cd41599a0 0x7f1cd4153bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:48.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.308+0000 7f1cd9e25640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1cd4072e20 con 0x7f1cd4152e60 2024-04-25T00:17:48.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.308+0000 7f1cd9e25640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1cd40c5870 con 0x7f1cd41599a0 2024-04-25T00:17:48.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.308+0000 7f1cd9e25640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f1cd4072fa0 con 0x7f1cd40c5010 2024-04-25T00:17:48.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.308+0000 7f1ccbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cd4152e60 0x7f1cd40ac2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:48.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.308+0000 7f1ccbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cd4152e60 0x7f1cd40ac2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39076/0 (socket says 172.21.15.116:39076) 2024-04-25T00:17:48.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.308+0000 7f1ccbfff640 1 -- 172.21.15.116:0/3081565943 learned_addr learned my addr 172.21.15.116:0/3081565943 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:17:48.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.308+0000 7f1cd37fe640 1 --2- 172.21.15.116:0/3081565943 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1cd40c5010 0x7f1cd41ef0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:48.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.308+0000 7f1cd3fff640 1 --2- 172.21.15.116:0/3081565943 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1cd41599a0 0x7f1cd4153bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:48.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.308+0000 7f1ccbfff640 1 -- 172.21.15.116:0/3081565943 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1cd40c5010 msgr2=0x7f1cd41ef0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:48.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.308+0000 7f1ccbfff640 1 --2- 172.21.15.116:0/3081565943 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1cd40c5010 0x7f1cd41ef0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:48.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.308+0000 7f1ccbfff640 1 -- 172.21.15.116:0/3081565943 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1cd41599a0 msgr2=0x7f1cd4153bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:48.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.308+0000 7f1ccbfff640 1 --2- 172.21.15.116:0/3081565943 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1cd41599a0 0x7f1cd4153bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:48.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.308+0000 7f1ccbfff640 1 -- 172.21.15.116:0/3081565943 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cd41ef5f0 con 0x7f1cd4152e60 2024-04-25T00:17:48.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.308+0000 7f1cd37fe640 1 --2- 172.21.15.116:0/3081565943 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1cd40c5010 0x7f1cd41ef0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:17:48.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.308+0000 7f1ccbfff640 1 --2- 172.21.15.116:0/3081565943 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cd4152e60 0x7f1cd40ac2d0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f1cbc018fe0 tx=0x7f1cbc019170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:48.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.308+0000 7f1cd0ff9640 1 -- 172.21.15.116:0/3081565943 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cbc0bb070 con 0x7f1cd4152e60 2024-04-25T00:17:48.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.308+0000 7f1cd0ff9640 1 -- 172.21.15.116:0/3081565943 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1cbc0b7460 con 0x7f1cd4152e60 2024-04-25T00:17:48.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.308+0000 7f1cd9e25640 1 -- 172.21.15.116:0/3081565943 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cd4154b80 con 0x7f1cd4152e60 2024-04-25T00:17:48.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.308+0000 7f1cd0ff9640 1 -- 172.21.15.116:0/3081565943 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cbc0b0040 con 0x7f1cd4152e60 2024-04-25T00:17:48.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.308+0000 7f1cd9e25640 1 -- 172.21.15.116:0/3081565943 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cd4155140 con 0x7f1cd4152e60 2024-04-25T00:17:48.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.308+0000 7f1cd9e25640 1 -- 172.21.15.116:0/3081565943 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cd4068a60 con 0x7f1cd4152e60 2024-04-25T00:17:48.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.308+0000 7f1cd0ff9640 1 -- 172.21.15.116:0/3081565943 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1cbc0be070 con 0x7f1cd4152e60 2024-04-25T00:17:48.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.312+0000 7f1cd0ff9640 1 --2- 172.21.15.116:0/3081565943 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1cb40678f0 0x7f1cb4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:48.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.312+0000 7f1cd0ff9640 1 -- 172.21.15.116:0/3081565943 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 8816+0+0 (secure 0 0 0) 0x7f1cbc120e00 con 0x7f1cd4152e60 2024-04-25T00:17:48.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.312+0000 7f1cd37fe640 1 --2- 172.21.15.116:0/3081565943 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1cb40678f0 0x7f1cb4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:48.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.312+0000 7f1cd37fe640 1 --2- 172.21.15.116:0/3081565943 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1cb40678f0 0x7f1cb4069db0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f1cc4007fd0 tx=0x7f1cc4007c20 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:48.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.316+0000 7f1cd0ff9640 1 -- 172.21.15.116:0/3081565943 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f1cbc016050 con 0x7f1cd4152e60 2024-04-25T00:17:48.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.536+0000 7f1cd9e25640 1 -- 172.21.15.116:0/3081565943 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1cd40c24e0 con 0x7f1cd4152e60 2024-04-25T00:17:48.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.536+0000 7f1cd0ff9640 1 -- 172.21.15.116:0/3081565943 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 136 v136) ==== 95+0+5680 (secure 0 0 0) 0x7f1cbc0f0e80 con 0x7f1cd4152e60 2024-04-25T00:17:48.540 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:17:48.540 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":136,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:17:38.902992+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6343":{"gid":6343,"name":"d","rank":0,"incarnation":133,"state":"up:active","state_seq":4,"addr":"172.21.15.174:6837/1281466397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1281466397},{"type":"v1","addr":"172.21.15.174:6837","nonce":1281466397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6343,"qdb_cluster":[6343]},"id":1},{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:17:28.837517+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7164},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7164":{"gid":7164,"name":"c","rank":0,"incarnation":128,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6835/3336339376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3336339376},{"type":"v1","addr":"172.21.15.116:6835","nonce":3336339376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7164,"qdb_cluster":[7164]},"id":2}]} 2024-04-25T00:17:48.540 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 136 2024-04-25T00:17:48.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.540+0000 7f1cd9e25640 1 -- 172.21.15.116:0/3081565943 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1cb40678f0 msgr2=0x7f1cb4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:48.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.540+0000 7f1cd9e25640 1 --2- 172.21.15.116:0/3081565943 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1cb40678f0 0x7f1cb4069db0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f1cc4007fd0 tx=0x7f1cc4007c20 comp rx=0 tx=0).stop 2024-04-25T00:17:48.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.540+0000 7f1cd9e25640 1 -- 172.21.15.116:0/3081565943 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cd4152e60 msgr2=0x7f1cd40ac2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:48.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.540+0000 7f1cd9e25640 1 --2- 172.21.15.116:0/3081565943 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cd4152e60 0x7f1cd40ac2d0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f1cbc018fe0 tx=0x7f1cbc019170 comp rx=0 tx=0).stop 2024-04-25T00:17:48.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.540+0000 7f1cd9e25640 1 -- 172.21.15.116:0/3081565943 shutdown_connections 2024-04-25T00:17:48.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.540+0000 7f1cd9e25640 1 --2- 172.21.15.116:0/3081565943 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1cb40678f0 0x7f1cb4069db0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:48.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.540+0000 7f1cd9e25640 1 --2- 172.21.15.116:0/3081565943 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1cd41599a0 0x7f1cd4153bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:48.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.540+0000 7f1cd9e25640 1 --2- 172.21.15.116:0/3081565943 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1cd4152e60 0x7f1cd40ac2d0 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:48.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.540+0000 7f1cd9e25640 1 --2- 172.21.15.116:0/3081565943 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1cd40c5010 0x7f1cd41ef0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:48.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.540+0000 7f1cd9e25640 1 -- 172.21.15.116:0/3081565943 >> 172.21.15.116:0/3081565943 conn(0x7f1cd406c720 msgr2=0x7f1cd406cb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:48.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.540+0000 7f1cd9e25640 1 -- 172.21.15.116:0/3081565943 shutdown_connections 2024-04-25T00:17:48.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:48.540+0000 7f1cd9e25640 1 -- 172.21.15.116:0/3081565943 wait complete. 2024-04-25T00:17:48.565 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2024-04-25T00:17:50.566 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:17:50.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.712+0000 7f9ad245b640 1 Processor -- start 2024-04-25T00:17:50.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.712+0000 7f9ad245b640 1 -- start start 2024-04-25T00:17:50.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.712+0000 7f9ad245b640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f9acc152e60 0x7f9acc1599a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:50.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.712+0000 7f9ad245b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9acc0c5010 0x7f9acc0c54a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:50.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.712+0000 7f9ad245b640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9acc073200 con 0x7f9acc1562d0 2024-04-25T00:17:50.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.712+0000 7f9ad245b640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f9acc156760 con 0x7f9acc152e60 2024-04-25T00:17:50.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.712+0000 7f9ad245b640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f9acc153240 con 0x7f9acc0c5010 2024-04-25T00:17:50.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.712+0000 7f9ad09f8640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9acc0c5010 0x7f9acc0c54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:50.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.712+0000 7f9ad09f8640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9acc0c5010 0x7f9acc0c54a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:47690/0 (socket says 172.21.15.116:47690) 2024-04-25T00:17:50.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.712+0000 7f9ad09f8640 1 -- 172.21.15.116:0/1244311449 learned_addr learned my addr 172.21.15.116:0/1244311449 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:17:50.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.712+0000 7f9acbfff640 1 --2- 172.21.15.116:0/1244311449 >> v2:172.21.15.174:3300/0 conn(0x7f9acc152e60 0x7f9acc1599a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:50.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.712+0000 7f9acbfff640 1 -- 172.21.15.116:0/1244311449 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9acc0c5010 msgr2=0x7f9acc0c54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:17:50.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.712+0000 7f9acbfff640 1 --2- 172.21.15.116:0/1244311449 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9acc0c5010 0x7f9acc0c54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:50.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.712+0000 7f9acbfff640 1 -- 172.21.15.116:0/1244311449 >> v1:172.21.15.174:6789/0 conn(0x7f9acc1562d0 legacy=0x7f9acc0c36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:50.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.712+0000 7f9acbfff640 1 -- 172.21.15.116:0/1244311449 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9acc158ec0 con 0x7f9acc152e60 2024-04-25T00:17:50.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.716+0000 7f9acbfff640 1 --2- 172.21.15.116:0/1244311449 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9acc152e60 0x7f9acc1599a0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f9abc0097c0 tx=0x7f9abc01c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=718e2f25953b8e10 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:50.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.716+0000 7f9acaffd640 1 -- 172.21.15.116:0/1244311449 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9abc0b0070 con 0x7f9acc152e60 2024-04-25T00:17:50.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.716+0000 7f9acaffd640 1 -- 172.21.15.116:0/1244311449 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9abc0ab600 con 0x7f9acc152e60 2024-04-25T00:17:50.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.716+0000 7f9acaffd640 1 -- 172.21.15.116:0/1244311449 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9abc0ab9d0 con 0x7f9acc152e60 2024-04-25T00:17:50.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.716+0000 7f9ad245b640 1 -- 172.21.15.116:0/1244311449 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9acc152e60 msgr2=0x7f9acc1599a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:50.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.716+0000 7f9ad245b640 1 --2- 172.21.15.116:0/1244311449 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9acc152e60 0x7f9acc1599a0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f9abc0097c0 tx=0x7f9abc01c410 comp rx=0 tx=0).stop 2024-04-25T00:17:50.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.716+0000 7f9ad245b640 1 -- 172.21.15.116:0/1244311449 shutdown_connections 2024-04-25T00:17:50.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.716+0000 7f9ad245b640 1 --2- 172.21.15.116:0/1244311449 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9acc0c5010 0x7f9acc0c54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:50.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.720+0000 7f9ad245b640 1 --2- 172.21.15.116:0/1244311449 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9acc152e60 0x7f9acc1599a0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:50.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.720+0000 7f9ad245b640 1 -- 172.21.15.116:0/1244311449 >> 172.21.15.116:0/1244311449 conn(0x7f9acc06c720 msgr2=0x7f9acc06cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:50.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.720+0000 7f9ad245b640 1 -- 172.21.15.116:0/1244311449 shutdown_connections 2024-04-25T00:17:50.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.720+0000 7f9ad245b640 1 -- 172.21.15.116:0/1244311449 wait complete. 2024-04-25T00:17:50.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.720+0000 7f9ad245b640 1 Processor -- start 2024-04-25T00:17:50.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.720+0000 7f9ad245b640 1 -- start start 2024-04-25T00:17:50.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.720+0000 7f9ad245b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9acc0c5010 0x7f9acc1516b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:50.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.720+0000 7f9ad245b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9acc152e60 0x7f9acc0ab420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:50.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.720+0000 7f9ad245b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9acc1562d0 0x7f9acc147720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:50.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.720+0000 7f9ad245b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9acc072e20 con 0x7f9acc0c5010 2024-04-25T00:17:50.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.720+0000 7f9ad245b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9acc0c3df0 con 0x7f9acc152e60 2024-04-25T00:17:50.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.720+0000 7f9ad245b640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f9acc072fa0 con 0x7f9acc1562d0 2024-04-25T00:17:50.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.720+0000 7f9acbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9acc0c5010 0x7f9acc1516b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:50.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.720+0000 7f9acbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9acc0c5010 0x7f9acc1516b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39092/0 (socket says 172.21.15.116:39092) 2024-04-25T00:17:50.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.720+0000 7f9acbfff640 1 -- 172.21.15.116:0/4291818717 learned_addr learned my addr 172.21.15.116:0/4291818717 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:17:50.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.720+0000 7f9ad09f8640 1 --2- 172.21.15.116:0/4291818717 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9acc1562d0 0x7f9acc147720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:50.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.720+0000 7f9acb7fe640 1 --2- 172.21.15.116:0/4291818717 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9acc152e60 0x7f9acc0ab420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:50.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.720+0000 7f9acbfff640 1 -- 172.21.15.116:0/4291818717 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9acc1562d0 msgr2=0x7f9acc147720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:50.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.720+0000 7f9acbfff640 1 --2- 172.21.15.116:0/4291818717 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9acc1562d0 0x7f9acc147720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:50.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.720+0000 7f9acbfff640 1 -- 172.21.15.116:0/4291818717 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9acc152e60 msgr2=0x7f9acc0ab420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:50.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.720+0000 7f9acbfff640 1 --2- 172.21.15.116:0/4291818717 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9acc152e60 0x7f9acc0ab420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:50.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.720+0000 7f9acbfff640 1 -- 172.21.15.116:0/4291818717 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9acc1f24c0 con 0x7f9acc0c5010 2024-04-25T00:17:50.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.720+0000 7f9acbfff640 1 --2- 172.21.15.116:0/4291818717 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9acc0c5010 0x7f9acc1516b0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f9abc0097c0 tx=0x7f9abc0191b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:50.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.724+0000 7f9ac8ff9640 1 -- 172.21.15.116:0/4291818717 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9abc0b0040 con 0x7f9acc0c5010 2024-04-25T00:17:50.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.724+0000 7f9ac8ff9640 1 -- 172.21.15.116:0/4291818717 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9abc0b9070 con 0x7f9acc0c5010 2024-04-25T00:17:50.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.724+0000 7f9ad245b640 1 -- 172.21.15.116:0/4291818717 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9acc14bbf0 con 0x7f9acc0c5010 2024-04-25T00:17:50.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.724+0000 7f9ac8ff9640 1 -- 172.21.15.116:0/4291818717 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9abc006e20 con 0x7f9acc0c5010 2024-04-25T00:17:50.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.724+0000 7f9ad245b640 1 -- 172.21.15.116:0/4291818717 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9acc0ab960 con 0x7f9acc0c5010 2024-04-25T00:17:50.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.724+0000 7f9ad245b640 1 -- 172.21.15.116:0/4291818717 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9acc068a60 con 0x7f9acc0c5010 2024-04-25T00:17:50.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.724+0000 7f9ac8ff9640 1 -- 172.21.15.116:0/4291818717 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f9abc0b4630 con 0x7f9acc0c5010 2024-04-25T00:17:50.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.724+0000 7f9ac8ff9640 1 --2- 172.21.15.116:0/4291818717 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9ab80678f0 0x7f9ab8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:17:50.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.724+0000 7f9ac8ff9640 1 -- 172.21.15.116:0/4291818717 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 8816+0+0 (secure 0 0 0) 0x7f9abc121000 con 0x7f9acc0c5010 2024-04-25T00:17:50.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.724+0000 7f9acb7fe640 1 --2- 172.21.15.116:0/4291818717 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9ab80678f0 0x7f9ab8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:17:50.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.728+0000 7f9acb7fe640 1 --2- 172.21.15.116:0/4291818717 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9ab80678f0 0x7f9ab8069db0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f9acc14a6f0 tx=0x7f9ac0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:17:50.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.732+0000 7f9ac8ff9640 1 -- 172.21.15.116:0/4291818717 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f9abc016090 con 0x7f9acc0c5010 2024-04-25T00:17:50.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.968+0000 7f9ad245b640 1 -- 172.21.15.116:0/4291818717 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9acc0c24e0 con 0x7f9acc0c5010 2024-04-25T00:17:50.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.968+0000 7f9ac8ff9640 1 -- 172.21.15.116:0/4291818717 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) ==== 95+0+6509 (secure 0 0 0) 0x7f9abc0f5e20 con 0x7f9acc0c5010 2024-04-25T00:17:50.969 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:17:50.970 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":137,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7362,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/1655959433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1655959433},{"type":"v1","addr":"172.21.15.116:6833","nonce":1655959433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:17:38.902992+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6343":{"gid":6343,"name":"d","rank":0,"incarnation":133,"state":"up:active","state_seq":4,"addr":"172.21.15.174:6837/1281466397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1281466397},{"type":"v1","addr":"172.21.15.174:6837","nonce":1281466397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6343,"qdb_cluster":[6343]},"id":1},{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:17:28.837517+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7164},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7164":{"gid":7164,"name":"c","rank":0,"incarnation":128,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6835/3336339376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3336339376},{"type":"v1","addr":"172.21.15.116:6835","nonce":3336339376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7164,"qdb_cluster":[7164]},"id":2}]} 2024-04-25T00:17:50.970 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 137 2024-04-25T00:17:50.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.972+0000 7f9ad245b640 1 -- 172.21.15.116:0/4291818717 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9ab80678f0 msgr2=0x7f9ab8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:50.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.972+0000 7f9ad245b640 1 --2- 172.21.15.116:0/4291818717 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9ab80678f0 0x7f9ab8069db0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f9acc14a6f0 tx=0x7f9ac0009040 comp rx=0 tx=0).stop 2024-04-25T00:17:50.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.972+0000 7f9ad245b640 1 -- 172.21.15.116:0/4291818717 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9acc0c5010 msgr2=0x7f9acc1516b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:17:50.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.972+0000 7f9ad245b640 1 --2- 172.21.15.116:0/4291818717 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9acc0c5010 0x7f9acc1516b0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f9abc0097c0 tx=0x7f9abc0191b0 comp rx=0 tx=0).stop 2024-04-25T00:17:50.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.972+0000 7f9ad245b640 1 -- 172.21.15.116:0/4291818717 shutdown_connections 2024-04-25T00:17:50.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.972+0000 7f9ad245b640 1 --2- 172.21.15.116:0/4291818717 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9ab80678f0 0x7f9ab8069db0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:50.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.972+0000 7f9ad245b640 1 --2- 172.21.15.116:0/4291818717 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9acc1562d0 0x7f9acc147720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:50.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.972+0000 7f9ad245b640 1 --2- 172.21.15.116:0/4291818717 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9acc152e60 0x7f9acc0ab420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:50.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.972+0000 7f9ad245b640 1 --2- 172.21.15.116:0/4291818717 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9acc0c5010 0x7f9acc1516b0 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:17:50.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.972+0000 7f9ad245b640 1 -- 172.21.15.116:0/4291818717 >> 172.21.15.116:0/4291818717 conn(0x7f9acc06c720 msgr2=0x7f9acc06edb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:17:50.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.972+0000 7f9ad245b640 1 -- 172.21.15.116:0/4291818717 shutdown_connections 2024-04-25T00:17:50.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:17:50.972+0000 7f9ad245b640 1 -- 172.21.15.116:0/4291818717 wait complete. 2024-04-25T00:17:50.989 INFO:tasks.mds_thrash.fs.[a]:mds.a reported in up:standby state 2024-04-25T00:17:50.990 INFO:tasks.mds_thrash.fs.[a]:waiting for 14 secs before thrashing 2024-04-25T00:18:04.991 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:18:05.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.124+0000 7f0f0f5bc640 1 Processor -- start 2024-04-25T00:18:05.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.124+0000 7f0f0f5bc640 1 -- start start 2024-04-25T00:18:05.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.124+0000 7f0f0f5bc640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f0f0806d5f0 0x7f0f08156c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.124+0000 7f0f0f5bc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0f08155810 0x7f0f081571c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.124+0000 7f0f0f5bc640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f080c9150 con 0x7f0f08152d80 2024-04-25T00:18:05.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.124+0000 7f0f0f5bc640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f0f080c9d60 con 0x7f0f08155810 2024-04-25T00:18:05.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.124+0000 7f0f0f5bc640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f0f080c9430 con 0x7f0f0806d5f0 2024-04-25T00:18:05.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.124+0000 7f0f0db59640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0f08155810 0x7f0f081571c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.124+0000 7f0f0db59640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0f08155810 0x7f0f081571c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41852/0 (socket says 172.21.15.116:41852) 2024-04-25T00:18:05.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.124+0000 7f0f0db59640 1 -- 172.21.15.116:0/814035458 learned_addr learned my addr 172.21.15.116:0/814035458 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:18:05.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.124+0000 7f0f0cb57640 1 --2- 172.21.15.116:0/814035458 >> v2:172.21.15.174:3300/0 conn(0x7f0f0806d5f0 0x7f0f08156c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.124+0000 7f0f0db59640 1 -- 172.21.15.116:0/814035458 >> v2:172.21.15.174:3300/0 conn(0x7f0f0806d5f0 msgr2=0x7f0f08156c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:05.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.124+0000 7f0f0db59640 1 --2- 172.21.15.116:0/814035458 >> v2:172.21.15.174:3300/0 conn(0x7f0f0806d5f0 0x7f0f08156c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.124+0000 7f0f0db59640 1 -- 172.21.15.116:0/814035458 >> v1:172.21.15.174:6789/0 conn(0x7f0f08152d80 legacy=0x7f0f0806bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:05.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.124+0000 7f0f0db59640 1 -- 172.21.15.116:0/814035458 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f080c9710 con 0x7f0f08155810 2024-04-25T00:18:05.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.128+0000 7f0f0db59640 1 --2- 172.21.15.116:0/814035458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f08155810 0x7f0f081571c0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f0f0808ca90 tx=0x7f0f0001c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bffb335309eeecf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:05.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.128+0000 7f0f07fff640 1 -- 172.21.15.116:0/814035458 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f000b0070 con 0x7f0f08155810 2024-04-25T00:18:05.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.128+0000 7f0f07fff640 1 -- 172.21.15.116:0/814035458 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f00004230 con 0x7f0f08155810 2024-04-25T00:18:05.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.128+0000 7f0f07fff640 1 -- 172.21.15.116:0/814035458 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f00003620 con 0x7f0f08155810 2024-04-25T00:18:05.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.128+0000 7f0f0f5bc640 1 -- 172.21.15.116:0/814035458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f08155810 msgr2=0x7f0f081571c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:05.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.128+0000 7f0f0f5bc640 1 --2- 172.21.15.116:0/814035458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f08155810 0x7f0f081571c0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f0f0808ca90 tx=0x7f0f0001c420 comp rx=0 tx=0).stop 2024-04-25T00:18:05.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.128+0000 7f0f0f5bc640 1 -- 172.21.15.116:0/814035458 shutdown_connections 2024-04-25T00:18:05.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.128+0000 7f0f0f5bc640 1 --2- 172.21.15.116:0/814035458 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f08155810 0x7f0f081571c0 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.128+0000 7f0f0f5bc640 1 --2- 172.21.15.116:0/814035458 >> v2:172.21.15.174:3300/0 conn(0x7f0f0806d5f0 0x7f0f08156c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.128+0000 7f0f0f5bc640 1 -- 172.21.15.116:0/814035458 >> 172.21.15.116:0/814035458 conn(0x7f0f080acdd0 msgr2=0x7f0f080ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:05.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.128+0000 7f0f0f5bc640 1 -- 172.21.15.116:0/814035458 shutdown_connections 2024-04-25T00:18:05.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.128+0000 7f0f0f5bc640 1 -- 172.21.15.116:0/814035458 wait complete. 2024-04-25T00:18:05.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.128+0000 7f0f0f5bc640 1 Processor -- start 2024-04-25T00:18:05.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.128+0000 7f0f0f5bc640 1 -- start start 2024-04-25T00:18:05.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.132+0000 7f0f0f5bc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f0806d5f0 0x7f0f081518b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.132+0000 7f0f0f5bc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0f08152d80 0x7f0f080c5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.132+0000 7f0f0f5bc640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0f08155810 0x7f0f080c6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.132+0000 7f0f0f5bc640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0f080caf00 con 0x7f0f0806d5f0 2024-04-25T00:18:05.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.132+0000 7f0f0f5bc640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0f080cad80 con 0x7f0f08152d80 2024-04-25T00:18:05.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.132+0000 7f0f0f5bc640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f0f081628d0 con 0x7f0f08155810 2024-04-25T00:18:05.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.132+0000 7f0f0d358640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f0806d5f0 0x7f0f081518b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.132+0000 7f0f0d358640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f0806d5f0 0x7f0f081518b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41864/0 (socket says 172.21.15.116:41864) 2024-04-25T00:18:05.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.132+0000 7f0f0d358640 1 -- 172.21.15.116:0/4266275854 learned_addr learned my addr 172.21.15.116:0/4266275854 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:18:05.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.132+0000 7f0f0cb57640 1 --2- 172.21.15.116:0/4266275854 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0f08152d80 0x7f0f080c5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.132+0000 7f0f0db59640 1 --2- 172.21.15.116:0/4266275854 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0f08155810 0x7f0f080c6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.132+0000 7f0f0d358640 1 -- 172.21.15.116:0/4266275854 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0f08155810 msgr2=0x7f0f080c6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:05.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.132+0000 7f0f0d358640 1 --2- 172.21.15.116:0/4266275854 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0f08155810 0x7f0f080c6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.132+0000 7f0f0d358640 1 -- 172.21.15.116:0/4266275854 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0f08152d80 msgr2=0x7f0f080c5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:05.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.132+0000 7f0f0d358640 1 --2- 172.21.15.116:0/4266275854 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0f08152d80 0x7f0f080c5e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.132+0000 7f0f0d358640 1 -- 172.21.15.116:0/4266275854 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f081ef4a0 con 0x7f0f0806d5f0 2024-04-25T00:18:05.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.132+0000 7f0f0db59640 1 --2- 172.21.15.116:0/4266275854 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0f08155810 0x7f0f080c6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:18:05.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.132+0000 7f0f0d358640 1 --2- 172.21.15.116:0/4266275854 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f0806d5f0 0x7f0f081518b0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f0ef800c7f0 tx=0x7f0ef800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:05.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.132+0000 7f0f05ffb640 1 -- 172.21.15.116:0/4266275854 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef8016020 con 0x7f0f0806d5f0 2024-04-25T00:18:05.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.132+0000 7f0f05ffb640 1 -- 172.21.15.116:0/4266275854 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ef800aa50 con 0x7f0f0806d5f0 2024-04-25T00:18:05.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.132+0000 7f0f0f5bc640 1 -- 172.21.15.116:0/4266275854 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f080c7690 con 0x7f0f0806d5f0 2024-04-25T00:18:05.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.132+0000 7f0f05ffb640 1 -- 172.21.15.116:0/4266275854 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef800ae40 con 0x7f0f0806d5f0 2024-04-25T00:18:05.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.132+0000 7f0f0f5bc640 1 -- 172.21.15.116:0/4266275854 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f08151df0 con 0x7f0f0806d5f0 2024-04-25T00:18:05.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.132+0000 7f0f0f5bc640 1 -- 172.21.15.116:0/4266275854 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f08068350 con 0x7f0f0806d5f0 2024-04-25T00:18:05.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.132+0000 7f0f05ffb640 1 -- 172.21.15.116:0/4266275854 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f0ef8019020 con 0x7f0f0806d5f0 2024-04-25T00:18:05.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.132+0000 7f0f05ffb640 1 --2- 172.21.15.116:0/4266275854 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0ef40678f0 0x7f0ef4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.132+0000 7f0f05ffb640 1 -- 172.21.15.116:0/4266275854 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 8816+0+0 (secure 0 0 0) 0x7f0ef80868a0 con 0x7f0f0806d5f0 2024-04-25T00:18:05.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.132+0000 7f0f0cb57640 1 --2- 172.21.15.116:0/4266275854 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0ef40678f0 0x7f0ef4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.136+0000 7f0f0cb57640 1 --2- 172.21.15.116:0/4266275854 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0ef40678f0 0x7f0ef4069db0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f0f08152380 tx=0x7f0efc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:05.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.140+0000 7f0f05ffb640 1 -- 172.21.15.116:0/4266275854 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f0ef8020030 con 0x7f0f0806d5f0 2024-04-25T00:18:05.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.360+0000 7f0f0f5bc640 1 -- 172.21.15.116:0/4266275854 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0f080759d0 con 0x7f0f0806d5f0 2024-04-25T00:18:05.364 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.364+0000 7f0f05ffb640 1 -- 172.21.15.116:0/4266275854 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) ==== 95+0+6509 (secure 0 0 0) 0x7f0ef8056800 con 0x7f0f0806d5f0 2024-04-25T00:18:05.364 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:18:05.364 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":137,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7362,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/1655959433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1655959433},{"type":"v1","addr":"172.21.15.116:6833","nonce":1655959433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:17:38.902992+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6343":{"gid":6343,"name":"d","rank":0,"incarnation":133,"state":"up:active","state_seq":4,"addr":"172.21.15.174:6837/1281466397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1281466397},{"type":"v1","addr":"172.21.15.174:6837","nonce":1281466397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6343,"qdb_cluster":[6343]},"id":1},{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:17:28.837517+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7164},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7164":{"gid":7164,"name":"c","rank":0,"incarnation":128,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6835/3336339376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3336339376},{"type":"v1","addr":"172.21.15.116:6835","nonce":3336339376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7164,"qdb_cluster":[7164]},"id":2}]} 2024-04-25T00:18:05.364 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 137 2024-04-25T00:18:05.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.364+0000 7f0f0f5bc640 1 -- 172.21.15.116:0/4266275854 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0ef40678f0 msgr2=0x7f0ef4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:05.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.364+0000 7f0f0f5bc640 1 --2- 172.21.15.116:0/4266275854 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0ef40678f0 0x7f0ef4069db0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f0f08152380 tx=0x7f0efc008040 comp rx=0 tx=0).stop 2024-04-25T00:18:05.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.364+0000 7f0f0f5bc640 1 -- 172.21.15.116:0/4266275854 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f0806d5f0 msgr2=0x7f0f081518b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:05.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.364+0000 7f0f0f5bc640 1 --2- 172.21.15.116:0/4266275854 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f0806d5f0 0x7f0f081518b0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f0ef800c7f0 tx=0x7f0ef800ccb0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.364+0000 7f0f0f5bc640 1 -- 172.21.15.116:0/4266275854 shutdown_connections 2024-04-25T00:18:05.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.364+0000 7f0f0f5bc640 1 --2- 172.21.15.116:0/4266275854 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0ef40678f0 0x7f0ef4069db0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.364+0000 7f0f0f5bc640 1 --2- 172.21.15.116:0/4266275854 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0f08155810 0x7f0f080c6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.364+0000 7f0f0f5bc640 1 --2- 172.21.15.116:0/4266275854 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0f08152d80 0x7f0f080c5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.364+0000 7f0f0f5bc640 1 --2- 172.21.15.116:0/4266275854 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0f0806d5f0 0x7f0f081518b0 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.364+0000 7f0f0f5bc640 1 -- 172.21.15.116:0/4266275854 >> 172.21.15.116:0/4266275854 conn(0x7f0f080acdd0 msgr2=0x7f0f0815f600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:05.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.364+0000 7f0f0f5bc640 1 -- 172.21.15.116:0/4266275854 shutdown_connections 2024-04-25T00:18:05.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.364+0000 7f0f0f5bc640 1 -- 172.21.15.116:0/4266275854 wait complete. 2024-04-25T00:18:05.384 INFO:tasks.mds_thrash.fs.[a]:kill mds.d (rank=0) 2024-04-25T00:18:05.384 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-25T00:18:05.384 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:18:05.400 INFO:tasks.ceph.mds.d:Stopped 2024-04-25T00:18:05.400 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:18:05.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.520+0000 7fa82d169640 1 Processor -- start 2024-04-25T00:18:05.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.520+0000 7fa82d169640 1 -- start start 2024-04-25T00:18:05.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.520+0000 7fa82d169640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa828152f60 0x7fa8280c6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.520+0000 7fa82d169640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fa8280c3c30 0x7fa82806fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.520+0000 7fa82d169640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa82806e8f0 con 0x7fa828158ea0 2024-04-25T00:18:05.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.520+0000 7fa82d169640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fa82806ebd0 con 0x7fa828152f60 2024-04-25T00:18:05.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.520+0000 7fa82d169640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fa82806f330 con 0x7fa8280c3c30 2024-04-25T00:18:05.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.520+0000 7fa826d9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa828152f60 0x7fa8280c6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.520+0000 7fa826d9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa828152f60 0x7fa8280c6080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41872/0 (socket says 172.21.15.116:41872) 2024-04-25T00:18:05.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.520+0000 7fa826d9d640 1 -- 172.21.15.116:0/2527198018 learned_addr learned my addr 172.21.15.116:0/2527198018 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:18:05.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.520+0000 7fa81ffff640 1 --2- 172.21.15.116:0/2527198018 >> v2:172.21.15.174:3300/0 conn(0x7fa8280c3c30 0x7fa82806fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.520+0000 7fa826d9d640 1 -- 172.21.15.116:0/2527198018 >> v2:172.21.15.174:3300/0 conn(0x7fa8280c3c30 msgr2=0x7fa82806fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:05.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.520+0000 7fa826d9d640 1 --2- 172.21.15.116:0/2527198018 >> v2:172.21.15.174:3300/0 conn(0x7fa8280c3c30 0x7fa82806fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.520+0000 7fa826d9d640 1 -- 172.21.15.116:0/2527198018 >> v1:172.21.15.174:6789/0 conn(0x7fa828158ea0 legacy=0x7fa82806c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:05.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.520+0000 7fa826d9d640 1 -- 172.21.15.116:0/2527198018 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa82806f610 con 0x7fa828152f60 2024-04-25T00:18:05.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.524+0000 7fa826d9d640 1 --2- 172.21.15.116:0/2527198018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa828152f60 0x7fa8280c6080 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7fa8280c2040 tx=0x7fa81801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e523b125800f232 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:05.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.524+0000 7fa82659c640 1 -- 172.21.15.116:0/2527198018 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8180b2070 con 0x7fa828152f60 2024-04-25T00:18:05.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.524+0000 7fa82659c640 1 -- 172.21.15.116:0/2527198018 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8180a9070 con 0x7fa828152f60 2024-04-25T00:18:05.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.524+0000 7fa82659c640 1 -- 172.21.15.116:0/2527198018 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8180ad880 con 0x7fa828152f60 2024-04-25T00:18:05.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.524+0000 7fa82d169640 1 -- 172.21.15.116:0/2527198018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa828152f60 msgr2=0x7fa8280c6080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:05.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.524+0000 7fa82d169640 1 --2- 172.21.15.116:0/2527198018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa828152f60 0x7fa8280c6080 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7fa8280c2040 tx=0x7fa81801c490 comp rx=0 tx=0).stop 2024-04-25T00:18:05.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.524+0000 7fa82d169640 1 -- 172.21.15.116:0/2527198018 shutdown_connections 2024-04-25T00:18:05.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.524+0000 7fa82d169640 1 --2- 172.21.15.116:0/2527198018 >> v2:172.21.15.174:3300/0 conn(0x7fa8280c3c30 0x7fa82806fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.524+0000 7fa82d169640 1 --2- 172.21.15.116:0/2527198018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa828152f60 0x7fa8280c6080 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.524+0000 7fa82d169640 1 -- 172.21.15.116:0/2527198018 >> 172.21.15.116:0/2527198018 conn(0x7fa8280c4270 msgr2=0x7fa8280c4680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:05.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.524+0000 7fa82d169640 1 -- 172.21.15.116:0/2527198018 shutdown_connections 2024-04-25T00:18:05.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.524+0000 7fa82d169640 1 -- 172.21.15.116:0/2527198018 wait complete. 2024-04-25T00:18:05.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.524+0000 7fa82d169640 1 Processor -- start 2024-04-25T00:18:05.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.524+0000 7fa82d169640 1 -- start start 2024-04-25T00:18:05.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.524+0000 7fa82d169640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8280c3c30 0x7fa8280ad700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.524+0000 7fa82d169640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa828152f60 0x7fa82815e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.524+0000 7fa82d169640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa828158ea0 0x7fa8281f18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.524+0000 7fa82d169640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa828162b70 con 0x7fa828152f60 2024-04-25T00:18:05.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.524+0000 7fa82d169640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa8281629f0 con 0x7fa8280c3c30 2024-04-25T00:18:05.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.524+0000 7fa82d169640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa828162cf0 con 0x7fa828158ea0 2024-04-25T00:18:05.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.524+0000 7fa81ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa828152f60 0x7fa82815e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.524+0000 7fa81ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa828152f60 0x7fa82815e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41882/0 (socket says 172.21.15.116:41882) 2024-04-25T00:18:05.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.524+0000 7fa81ffff640 1 -- 172.21.15.116:0/3332581576 learned_addr learned my addr 172.21.15.116:0/3332581576 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:18:05.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.524+0000 7fa826d9d640 1 --2- 172.21.15.116:0/3332581576 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8280c3c30 0x7fa8280ad700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.524+0000 7fa82759e640 1 --2- 172.21.15.116:0/3332581576 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa828158ea0 0x7fa8281f18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.524+0000 7fa81ffff640 1 -- 172.21.15.116:0/3332581576 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa828158ea0 msgr2=0x7fa8281f18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:05.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.524+0000 7fa81ffff640 1 --2- 172.21.15.116:0/3332581576 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa828158ea0 0x7fa8281f18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.524+0000 7fa81ffff640 1 -- 172.21.15.116:0/3332581576 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8280c3c30 msgr2=0x7fa8280ad700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:05.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.524+0000 7fa81ffff640 1 --2- 172.21.15.116:0/3332581576 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8280c3c30 0x7fa8280ad700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.524+0000 7fa81ffff640 1 -- 172.21.15.116:0/3332581576 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8280a8aa0 con 0x7fa828152f60 2024-04-25T00:18:05.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.524+0000 7fa81ffff640 1 --2- 172.21.15.116:0/3332581576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa828152f60 0x7fa82815e2e0 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7fa81000d5f0 tx=0x7fa81000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:05.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.524+0000 7fa81f7fe640 1 -- 172.21.15.116:0/3332581576 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8100125f0 con 0x7fa828152f60 2024-04-25T00:18:05.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.524+0000 7fa81f7fe640 1 -- 172.21.15.116:0/3332581576 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa810019070 con 0x7fa828152f60 2024-04-25T00:18:05.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.524+0000 7fa82d169640 1 -- 172.21.15.116:0/3332581576 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8281f2620 con 0x7fa828152f60 2024-04-25T00:18:05.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.524+0000 7fa81f7fe640 1 -- 172.21.15.116:0/3332581576 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8100129d0 con 0x7fa828152f60 2024-04-25T00:18:05.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.524+0000 7fa82d169640 1 -- 172.21.15.116:0/3332581576 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8280a9020 con 0x7fa828152f60 2024-04-25T00:18:05.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.528+0000 7fa82d169640 1 -- 172.21.15.116:0/3332581576 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa828068a60 con 0x7fa828152f60 2024-04-25T00:18:05.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.528+0000 7fa81f7fe640 1 -- 172.21.15.116:0/3332581576 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa810012b70 con 0x7fa828152f60 2024-04-25T00:18:05.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.528+0000 7fa81f7fe640 1 --2- 172.21.15.116:0/3332581576 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa8080678c0 0x7fa808069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:05.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.528+0000 7fa81f7fe640 1 -- 172.21.15.116:0/3332581576 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 8816+0+0 (secure 0 0 0) 0x7fa810086cc0 con 0x7fa828152f60 2024-04-25T00:18:05.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.528+0000 7fa826d9d640 1 --2- 172.21.15.116:0/3332581576 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa8080678c0 0x7fa808069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:05.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.528+0000 7fa826d9d640 1 --2- 172.21.15.116:0/3332581576 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa8080678c0 0x7fa808069d80 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fa8180af000 tx=0x7fa8180a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:05.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.532+0000 7fa81f7fe640 1 -- 172.21.15.116:0/3332581576 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fa810056c20 con 0x7fa828152f60 2024-04-25T00:18:05.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.760+0000 7fa82d169640 1 -- 172.21.15.116:0/3332581576 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa82806c5c0 con 0x7fa828152f60 2024-04-25T00:18:05.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.764+0000 7fa81f7fe640 1 -- 172.21.15.116:0/3332581576 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) ==== 95+0+6509 (secure 0 0 0) 0x7fa810056520 con 0x7fa828152f60 2024-04-25T00:18:05.764 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:18:05.764 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":137,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7362,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/1655959433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1655959433},{"type":"v1","addr":"172.21.15.116:6833","nonce":1655959433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:17:38.902992+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6343":{"gid":6343,"name":"d","rank":0,"incarnation":133,"state":"up:active","state_seq":4,"addr":"172.21.15.174:6837/1281466397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1281466397},{"type":"v1","addr":"172.21.15.174:6837","nonce":1281466397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6343,"qdb_cluster":[6343]},"id":1},{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:17:28.837517+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7164},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7164":{"gid":7164,"name":"c","rank":0,"incarnation":128,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6835/3336339376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3336339376},{"type":"v1","addr":"172.21.15.116:6835","nonce":3336339376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7164,"qdb_cluster":[7164]},"id":2}]} 2024-04-25T00:18:05.765 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 137 2024-04-25T00:18:05.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.764+0000 7fa82d169640 1 -- 172.21.15.116:0/3332581576 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa8080678c0 msgr2=0x7fa808069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:05.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.764+0000 7fa82d169640 1 --2- 172.21.15.116:0/3332581576 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa8080678c0 0x7fa808069d80 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fa8180af000 tx=0x7fa8180a7000 comp rx=0 tx=0).stop 2024-04-25T00:18:05.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.764+0000 7fa82d169640 1 -- 172.21.15.116:0/3332581576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa828152f60 msgr2=0x7fa82815e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:05.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.764+0000 7fa82d169640 1 --2- 172.21.15.116:0/3332581576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa828152f60 0x7fa82815e2e0 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7fa81000d5f0 tx=0x7fa81000dab0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.764+0000 7fa82d169640 1 -- 172.21.15.116:0/3332581576 shutdown_connections 2024-04-25T00:18:05.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.764+0000 7fa82d169640 1 --2- 172.21.15.116:0/3332581576 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa8080678c0 0x7fa808069d80 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.764+0000 7fa82d169640 1 --2- 172.21.15.116:0/3332581576 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa828158ea0 0x7fa8281f18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.764+0000 7fa82d169640 1 --2- 172.21.15.116:0/3332581576 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa828152f60 0x7fa82815e2e0 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.764+0000 7fa82d169640 1 --2- 172.21.15.116:0/3332581576 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8280c3c30 0x7fa8280ad700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:05.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.764+0000 7fa82d169640 1 -- 172.21.15.116:0/3332581576 >> 172.21.15.116:0/3332581576 conn(0x7fa8280c4270 msgr2=0x7fa82806b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:05.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.764+0000 7fa82d169640 1 -- 172.21.15.116:0/3332581576 shutdown_connections 2024-04-25T00:18:05.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:05.764+0000 7fa82d169640 1 -- 172.21.15.116:0/3332581576 wait complete. 2024-04-25T00:18:05.790 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:18:07.791 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:18:07.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.924+0000 7f6c22fd7640 1 Processor -- start 2024-04-25T00:18:07.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.924+0000 7f6c22fd7640 1 -- start start 2024-04-25T00:18:07.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.924+0000 7f6c22fd7640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c1c152d60 0x7f6c1c0ab250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:07.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.924+0000 7f6c22fd7640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c1c156760 con 0x7f6c1c0ab790 2024-04-25T00:18:07.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.924+0000 7f6c22fd7640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c1c155be0 con 0x7f6c1c153140 2024-04-25T00:18:07.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.924+0000 7f6c22fd7640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6c1c155ec0 con 0x7f6c1c152d60 2024-04-25T00:18:07.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.924+0000 7f6c1bfff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f6c1c0ab790 0x7f6c1c0c8450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:42258/0 (socket says 172.21.15.116:42258) 2024-04-25T00:18:07.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.924+0000 7f6c1bfff640 1 -- 172.21.15.116:0/4815696 learned_addr learned my addr 172.21.15.116:0/4815696 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:18:07.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.924+0000 7f6c20d73640 1 --2- 172.21.15.116:0/4815696 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c1c152d60 0x7f6c1c0ab250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:07.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.924+0000 7f6c1b7fe640 1 -- 172.21.15.116:0/4815696 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1911073362 0 0) 0x7f6c1c156760 con 0x7f6c1c0ab790 2024-04-25T00:18:07.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.924+0000 7f6c1b7fe640 1 -- 172.21.15.116:0/4815696 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c0c003530 con 0x7f6c1c0ab790 2024-04-25T00:18:07.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.924+0000 7f6c1b7fe640 1 -- 172.21.15.116:0/4815696 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1690273315 0 0) 0x7f6c0c003530 con 0x7f6c1c0ab790 2024-04-25T00:18:07.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.924+0000 7f6c1b7fe640 1 -- 172.21.15.116:0/4815696 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c1c152d60 msgr2=0x7f6c1c0ab250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:07.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.924+0000 7f6c1b7fe640 1 --2- 172.21.15.116:0/4815696 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c1c152d60 0x7f6c1c0ab250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.924+0000 7f6c1b7fe640 1 -- 172.21.15.116:0/4815696 >> v1:172.21.15.174:6789/0 conn(0x7f6c1c153140 legacy=0x7f6c1c0c3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:07.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.924+0000 7f6c1b7fe640 1 -- 172.21.15.116:0/4815696 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c1c1561a0 con 0x7f6c1c0ab790 2024-04-25T00:18:07.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.924+0000 7f6c1b7fe640 1 -- 172.21.15.116:0/4815696 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f6c08001680 con 0x7f6c1c0ab790 2024-04-25T00:18:07.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.924+0000 7f6c1b7fe640 1 -- 172.21.15.116:0/4815696 >> v1:172.21.15.116:6789/0 conn(0x7f6c1c0ab790 legacy=0x7f6c1c0c8450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:07.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.924+0000 7f6c1b7fe640 1 --2- 172.21.15.116:0/4815696 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c0c0039f0 0x7f6c0c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:07.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.924+0000 7f6c1b7fe640 1 --2- 172.21.15.116:0/4815696 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c0c004810 0x7f6c0c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:07.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.924+0000 7f6c1b7fe640 1 --2- 172.21.15.116:0/4815696 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c1c152d60 0x7f6c0c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:07.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.924+0000 7f6c1b7fe640 1 -- 172.21.15.116:0/4815696 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6c1c155ec0 con 0x7f6c0c0039f0 2024-04-25T00:18:07.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.924+0000 7f6c1b7fe640 1 -- 172.21.15.116:0/4815696 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6c0c00bb00 con 0x7f6c0c004810 2024-04-25T00:18:07.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.924+0000 7f6c1b7fe640 1 -- 172.21.15.116:0/4815696 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6c0c00bd00 con 0x7f6c1c152d60 2024-04-25T00:18:07.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.924+0000 7f6c20d73640 1 --2- 172.21.15.116:0/4815696 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c0c0039f0 0x7f6c0c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:07.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.924+0000 7f6c21574640 1 --2- 172.21.15.116:0/4815696 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c1c152d60 0x7f6c0c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:07.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.924+0000 7f6c20d73640 1 -- 172.21.15.116:0/4815696 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c1c152d60 msgr2=0x7f6c0c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:07.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.924+0000 7f6c20d73640 1 --2- 172.21.15.116:0/4815696 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c1c152d60 0x7f6c0c005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.924+0000 7f6c20d73640 1 -- 172.21.15.116:0/4815696 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c0c004810 msgr2=0x7f6c0c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:07.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.924+0000 7f6c20d73640 1 --2- 172.21.15.116:0/4815696 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c0c004810 0x7f6c0c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.924+0000 7f6c20d73640 1 -- 172.21.15.116:0/4815696 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c0c003530 con 0x7f6c0c0039f0 2024-04-25T00:18:07.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.928+0000 7f6c20d73640 1 --2- 172.21.15.116:0/4815696 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c0c0039f0 0x7f6c0c003e10 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f6c10009ae0 tx=0x7f6c1001ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f807e41cbc0697f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:07.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.928+0000 7f6c1b7fe640 1 -- 172.21.15.116:0/4815696 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c100b2070 con 0x7f6c0c0039f0 2024-04-25T00:18:07.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.928+0000 7f6c1b7fe640 1 -- 172.21.15.116:0/4815696 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c100a5660 con 0x7f6c0c0039f0 2024-04-25T00:18:07.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.928+0000 7f6c1b7fe640 1 -- 172.21.15.116:0/4815696 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c100096b0 con 0x7f6c0c0039f0 2024-04-25T00:18:07.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.928+0000 7f6c22fd7640 1 -- 172.21.15.116:0/4815696 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c0c0039f0 msgr2=0x7f6c0c003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:07.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.928+0000 7f6c22fd7640 1 --2- 172.21.15.116:0/4815696 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c0c0039f0 0x7f6c0c003e10 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f6c10009ae0 tx=0x7f6c1001ca70 comp rx=0 tx=0).stop 2024-04-25T00:18:07.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.928+0000 7f6c21574640 1 -- 172.21.15.116:0/4815696 reap_dead start 2024-04-25T00:18:07.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.928+0000 7f6c22fd7640 1 -- 172.21.15.116:0/4815696 shutdown_connections 2024-04-25T00:18:07.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.928+0000 7f6c22fd7640 1 -- 172.21.15.116:0/4815696 >> 172.21.15.116:0/4815696 conn(0x7f6c1c06f0c0 msgr2=0x7f6c1c06f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:07.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.928+0000 7f6c22fd7640 1 -- 172.21.15.116:0/4815696 shutdown_connections 2024-04-25T00:18:07.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.928+0000 7f6c22fd7640 1 -- 172.21.15.116:0/4815696 wait complete. 2024-04-25T00:18:07.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.928+0000 7f6c22fd7640 1 Processor -- start 2024-04-25T00:18:07.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.928+0000 7f6c22fd7640 1 -- start start 2024-04-25T00:18:07.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.928+0000 7f6c22fd7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c0c0039f0 0x7f6c1c0ad910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:07.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.928+0000 7f6c22fd7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c1c0ade50 0x7f6c1c145f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:07.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.928+0000 7f6c22fd7640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c1c146490 0x7f6c1c1f0090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:07.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.928+0000 7f6c22fd7640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6c1c0c4930 con 0x7f6c1c0ade50 2024-04-25T00:18:07.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.928+0000 7f6c22fd7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6c1c0c47b0 con 0x7f6c0c0039f0 2024-04-25T00:18:07.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.928+0000 7f6c22fd7640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6c1c1619a0 con 0x7f6c1c146490 2024-04-25T00:18:07.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.928+0000 7f6c1bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c1c0ade50 0x7f6c1c145f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:07.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.928+0000 7f6c1bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c1c0ade50 0x7f6c1c145f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41910/0 (socket says 172.21.15.116:41910) 2024-04-25T00:18:07.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.928+0000 7f6c1bfff640 1 -- 172.21.15.116:0/2134687660 learned_addr learned my addr 172.21.15.116:0/2134687660 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:18:07.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.928+0000 7f6c20d73640 1 --2- 172.21.15.116:0/2134687660 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c0c0039f0 0x7f6c1c0ad910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:07.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.928+0000 7f6c21574640 1 --2- 172.21.15.116:0/2134687660 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c1c146490 0x7f6c1c1f0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:07.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.932+0000 7f6c1bfff640 1 -- 172.21.15.116:0/2134687660 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c1c146490 msgr2=0x7f6c1c1f0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:07.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.932+0000 7f6c1bfff640 1 --2- 172.21.15.116:0/2134687660 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c1c146490 0x7f6c1c1f0090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.932+0000 7f6c1bfff640 1 -- 172.21.15.116:0/2134687660 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c0c0039f0 msgr2=0x7f6c1c0ad910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:07.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.932+0000 7f6c1bfff640 1 --2- 172.21.15.116:0/2134687660 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c0c0039f0 0x7f6c1c0ad910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:07.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.932+0000 7f6c1bfff640 1 -- 172.21.15.116:0/2134687660 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c1c14af50 con 0x7f6c1c0ade50 2024-04-25T00:18:07.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.932+0000 7f6c1bfff640 1 --2- 172.21.15.116:0/2134687660 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c1c0ade50 0x7f6c1c145f50 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f6c0800c7a0 tx=0x7f6c0800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:07.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.932+0000 7f6c197fa640 1 -- 172.21.15.116:0/2134687660 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c08018070 con 0x7f6c1c0ade50 2024-04-25T00:18:07.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.932+0000 7f6c197fa640 1 -- 172.21.15.116:0/2134687660 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c080074e0 con 0x7f6c1c0ade50 2024-04-25T00:18:07.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.932+0000 7f6c22fd7640 1 -- 172.21.15.116:0/2134687660 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c1c14b1d0 con 0x7f6c1c0ade50 2024-04-25T00:18:07.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.932+0000 7f6c197fa640 1 -- 172.21.15.116:0/2134687660 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c08004a30 con 0x7f6c1c0ade50 2024-04-25T00:18:07.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.932+0000 7f6c22fd7640 1 -- 172.21.15.116:0/2134687660 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c1c14b9c0 con 0x7f6c1c0ade50 2024-04-25T00:18:07.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.932+0000 7f6c22fd7640 1 -- 172.21.15.116:0/2134687660 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c1c068a60 con 0x7f6c1c0ade50 2024-04-25T00:18:07.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.932+0000 7f6c197fa640 1 -- 172.21.15.116:0/2134687660 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6c0801b070 con 0x7f6c1c0ade50 2024-04-25T00:18:07.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.932+0000 7f6c197fa640 1 --2- 172.21.15.116:0/2134687660 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6c000678c0 0x7f6c00069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:07.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.932+0000 7f6c197fa640 1 -- 172.21.15.116:0/2134687660 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 8816+0+0 (secure 0 0 0) 0x7f6c08085e60 con 0x7f6c1c0ade50 2024-04-25T00:18:07.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.932+0000 7f6c20d73640 1 --2- 172.21.15.116:0/2134687660 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6c000678c0 0x7f6c00069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:07.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.932+0000 7f6c20d73640 1 --2- 172.21.15.116:0/2134687660 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6c000678c0 0x7f6c00069d80 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f6c100af000 tx=0x7f6c100046d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:07.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:07.936+0000 7f6c197fa640 1 -- 172.21.15.116:0/2134687660 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f6c1c14af50 con 0x7f6c1c0ade50 2024-04-25T00:18:08.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:08.132+0000 7f6c22fd7640 1 -- 172.21.15.116:0/2134687660 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6c1c159b90 con 0x7f6c1c0ade50 2024-04-25T00:18:08.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:08.132+0000 7f6c197fa640 1 -- 172.21.15.116:0/2134687660 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) ==== 95+0+6509 (secure 0 0 0) 0x7f6c08055650 con 0x7f6c1c0ade50 2024-04-25T00:18:08.134 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:18:08.134 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":137,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7362,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/1655959433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1655959433},{"type":"v1","addr":"172.21.15.116:6833","nonce":1655959433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:17:38.902992+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6343":{"gid":6343,"name":"d","rank":0,"incarnation":133,"state":"up:active","state_seq":4,"addr":"172.21.15.174:6837/1281466397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1281466397},{"type":"v1","addr":"172.21.15.174:6837","nonce":1281466397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6343,"qdb_cluster":[6343]},"id":1},{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:17:28.837517+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7164},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7164":{"gid":7164,"name":"c","rank":0,"incarnation":128,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6835/3336339376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3336339376},{"type":"v1","addr":"172.21.15.116:6835","nonce":3336339376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7164,"qdb_cluster":[7164]},"id":2}]} 2024-04-25T00:18:08.134 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 137 2024-04-25T00:18:08.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:08.132+0000 7f6c22fd7640 1 -- 172.21.15.116:0/2134687660 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6c000678c0 msgr2=0x7f6c00069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:08.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:08.132+0000 7f6c22fd7640 1 --2- 172.21.15.116:0/2134687660 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6c000678c0 0x7f6c00069d80 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f6c100af000 tx=0x7f6c100046d0 comp rx=0 tx=0).stop 2024-04-25T00:18:08.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:08.132+0000 7f6c22fd7640 1 -- 172.21.15.116:0/2134687660 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c1c0ade50 msgr2=0x7f6c1c145f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:08.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:08.132+0000 7f6c22fd7640 1 --2- 172.21.15.116:0/2134687660 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c1c0ade50 0x7f6c1c145f50 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f6c0800c7a0 tx=0x7f6c0800cc60 comp rx=0 tx=0).stop 2024-04-25T00:18:08.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:08.132+0000 7f6c22fd7640 1 -- 172.21.15.116:0/2134687660 shutdown_connections 2024-04-25T00:18:08.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:08.132+0000 7f6c22fd7640 1 --2- 172.21.15.116:0/2134687660 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6c000678c0 0x7f6c00069d80 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:08.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:08.132+0000 7f6c22fd7640 1 --2- 172.21.15.116:0/2134687660 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c1c146490 0x7f6c1c1f0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:08.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:08.132+0000 7f6c22fd7640 1 --2- 172.21.15.116:0/2134687660 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c1c0ade50 0x7f6c1c145f50 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:08.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:08.132+0000 7f6c22fd7640 1 --2- 172.21.15.116:0/2134687660 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c0c0039f0 0x7f6c1c0ad910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:08.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:08.132+0000 7f6c22fd7640 1 -- 172.21.15.116:0/2134687660 >> 172.21.15.116:0/2134687660 conn(0x7f6c1c06f0c0 msgr2=0x7f6c1c06f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:08.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:08.132+0000 7f6c22fd7640 1 -- 172.21.15.116:0/2134687660 shutdown_connections 2024-04-25T00:18:08.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:08.132+0000 7f6c22fd7640 1 -- 172.21.15.116:0/2134687660 wait complete. 2024-04-25T00:18:08.152 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:18:09.222 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:18:09.225 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:18:10.153 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:18:10.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.276+0000 7f39917e1640 1 Processor -- start 2024-04-25T00:18:10.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.276+0000 7f39917e1640 1 -- start start 2024-04-25T00:18:10.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.280+0000 7f39917e1640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f398c0a94a0 0x7f398c1563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:10.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.280+0000 7f39917e1640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f398c071680 0x7f398c071ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:10.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.280+0000 7f39917e1640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f398c071220 con 0x7f398c14c720 2024-04-25T00:18:10.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.280+0000 7f39917e1640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f398c153460 con 0x7f398c071680 2024-04-25T00:18:10.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.280+0000 7f39917e1640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f398c152420 con 0x7f398c0a94a0 2024-04-25T00:18:10.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.280+0000 7f398a7fc640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f398c0a94a0 0x7f398c1563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:10.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.280+0000 7f398b7fe640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f398c071680 0x7f398c071ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:10.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.280+0000 7f398affd640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f398c14c720 0x7f398c0c2a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:46146/0 (socket says 172.21.15.116:46146) 2024-04-25T00:18:10.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.280+0000 7f398a7fc640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f398c0a94a0 0x7f398c1563d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:32804/0 (socket says 172.21.15.116:32804) 2024-04-25T00:18:10.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.280+0000 7f398affd640 1 -- 172.21.15.116:0/1410397597 learned_addr learned my addr 172.21.15.116:0/1410397597 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:18:10.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.280+0000 7f3989ffb640 1 -- 172.21.15.116:0/1410397597 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2793992707 0 0) 0x7f398c071220 con 0x7f398c14c720 2024-04-25T00:18:10.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.280+0000 7f3989ffb640 1 -- 172.21.15.116:0/1410397597 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3974003530 con 0x7f398c14c720 2024-04-25T00:18:10.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.280+0000 7f398a7fc640 1 -- 172.21.15.116:0/1410397597 >> v2:172.21.15.174:3300/0 conn(0x7f398c071680 msgr2=0x7f398c071ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:10.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.280+0000 7f398a7fc640 1 --2- 172.21.15.116:0/1410397597 >> v2:172.21.15.174:3300/0 conn(0x7f398c071680 0x7f398c071ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:10.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.280+0000 7f398a7fc640 1 -- 172.21.15.116:0/1410397597 >> v1:172.21.15.174:6789/0 conn(0x7f398c14c720 legacy=0x7f398c0c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:10.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.280+0000 7f398a7fc640 1 -- 172.21.15.116:0/1410397597 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f398c14cb00 con 0x7f398c0a94a0 2024-04-25T00:18:10.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.284+0000 7f398a7fc640 1 --2- 172.21.15.116:0/1410397597 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f398c0a94a0 0x7f398c1563d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f397c00ba60 tx=0x7f397c01eab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba46e64c205a83e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:10.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.284+0000 7f3989ffb640 1 -- 172.21.15.116:0/1410397597 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f397c0b4070 con 0x7f398c0a94a0 2024-04-25T00:18:10.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.284+0000 7f3989ffb640 1 -- 172.21.15.116:0/1410397597 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f397c0a7a80 con 0x7f398c0a94a0 2024-04-25T00:18:10.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.284+0000 7f3989ffb640 1 -- 172.21.15.116:0/1410397597 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f397c0a7e50 con 0x7f398c0a94a0 2024-04-25T00:18:10.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.284+0000 7f39917e1640 1 -- 172.21.15.116:0/1410397597 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f398c0a94a0 msgr2=0x7f398c1563d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:10.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.284+0000 7f39917e1640 1 --2- 172.21.15.116:0/1410397597 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f398c0a94a0 0x7f398c1563d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f397c00ba60 tx=0x7f397c01eab0 comp rx=0 tx=0).stop 2024-04-25T00:18:10.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.284+0000 7f39917e1640 1 -- 172.21.15.116:0/1410397597 shutdown_connections 2024-04-25T00:18:10.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.284+0000 7f39917e1640 1 --2- 172.21.15.116:0/1410397597 >> v2:172.21.15.174:3300/0 conn(0x7f398c071680 0x7f398c071ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:10.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.284+0000 7f39917e1640 1 --2- 172.21.15.116:0/1410397597 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f398c0a94a0 0x7f398c1563d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:10.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.284+0000 7f39917e1640 1 -- 172.21.15.116:0/1410397597 >> 172.21.15.116:0/1410397597 conn(0x7f398c0c3b00 msgr2=0x7f398c0c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:10.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.284+0000 7f39917e1640 1 -- 172.21.15.116:0/1410397597 shutdown_connections 2024-04-25T00:18:10.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.284+0000 7f39917e1640 1 -- 172.21.15.116:0/1410397597 wait complete. 2024-04-25T00:18:10.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.284+0000 7f39917e1640 1 Processor -- start 2024-04-25T00:18:10.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.284+0000 7f39917e1640 1 -- start start 2024-04-25T00:18:10.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.284+0000 7f39917e1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f398c071680 0x7f398c1402f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:10.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.284+0000 7f39917e1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f398c0a94a0 0x7f398c140830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:10.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.284+0000 7f39917e1640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f398c14c720 0x7f398c1537f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:10.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.284+0000 7f39917e1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f398c0c3120 con 0x7f398c071680 2024-04-25T00:18:10.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.284+0000 7f39917e1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f398c152f40 con 0x7f398c0a94a0 2024-04-25T00:18:10.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.284+0000 7f39917e1640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f398c0c32a0 con 0x7f398c14c720 2024-04-25T00:18:10.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.284+0000 7f398affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f398c071680 0x7f398c1402f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:10.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.284+0000 7f398affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f398c071680 0x7f398c1402f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41912/0 (socket says 172.21.15.116:41912) 2024-04-25T00:18:10.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.284+0000 7f398affd640 1 -- 172.21.15.116:0/2598129971 learned_addr learned my addr 172.21.15.116:0/2598129971 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:18:10.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.284+0000 7f398a7fc640 1 --2- 172.21.15.116:0/2598129971 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f398c0a94a0 0x7f398c140830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:10.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.284+0000 7f398b7fe640 1 --2- 172.21.15.116:0/2598129971 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f398c14c720 0x7f398c1537f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:10.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.284+0000 7f398affd640 1 -- 172.21.15.116:0/2598129971 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f398c14c720 msgr2=0x7f398c1537f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:10.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.284+0000 7f398affd640 1 --2- 172.21.15.116:0/2598129971 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f398c14c720 0x7f398c1537f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:10.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.284+0000 7f398affd640 1 -- 172.21.15.116:0/2598129971 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f398c0a94a0 msgr2=0x7f398c140830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:10.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.284+0000 7f398affd640 1 --2- 172.21.15.116:0/2598129971 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f398c0a94a0 0x7f398c140830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:10.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.284+0000 7f398affd640 1 -- 172.21.15.116:0/2598129971 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f398c141ec0 con 0x7f398c071680 2024-04-25T00:18:10.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.288+0000 7f398affd640 1 --2- 172.21.15.116:0/2598129971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f398c071680 0x7f398c1402f0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f398000c420 tx=0x7f398000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:10.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.288+0000 7f39737fe640 1 -- 172.21.15.116:0/2598129971 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3980017070 con 0x7f398c071680 2024-04-25T00:18:10.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.288+0000 7f39737fe640 1 -- 172.21.15.116:0/2598129971 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3980005cf0 con 0x7f398c071680 2024-04-25T00:18:10.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.288+0000 7f39917e1640 1 -- 172.21.15.116:0/2598129971 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f398c1421a0 con 0x7f398c071680 2024-04-25T00:18:10.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.288+0000 7f39737fe640 1 -- 172.21.15.116:0/2598129971 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39800072d0 con 0x7f398c071680 2024-04-25T00:18:10.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.288+0000 7f39917e1640 1 -- 172.21.15.116:0/2598129971 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f398c0c6850 con 0x7f398c071680 2024-04-25T00:18:10.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.288+0000 7f39917e1640 1 -- 172.21.15.116:0/2598129971 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f398c068350 con 0x7f398c071680 2024-04-25T00:18:10.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.288+0000 7f39737fe640 1 -- 172.21.15.116:0/2598129971 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f398001a070 con 0x7f398c071680 2024-04-25T00:18:10.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.288+0000 7f39737fe640 1 --2- 172.21.15.116:0/2598129971 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f39780678c0 0x7f3978069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:10.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.288+0000 7f39737fe640 1 -- 172.21.15.116:0/2598129971 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 8816+0+0 (secure 0 0 0) 0x7f3980085be0 con 0x7f398c071680 2024-04-25T00:18:10.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.288+0000 7f398a7fc640 1 --2- 172.21.15.116:0/2598129971 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f39780678c0 0x7f3978069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:10.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.288+0000 7f398a7fc640 1 --2- 172.21.15.116:0/2598129971 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f39780678c0 0x7f3978069d80 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f397c0b1000 tx=0x7f397c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:10.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.292+0000 7f39737fe640 1 -- 172.21.15.116:0/2598129971 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f39800554f0 con 0x7f398c071680 2024-04-25T00:18:10.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.492+0000 7f39917e1640 1 -- 172.21.15.116:0/2598129971 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f398c0c3640 con 0x7f398c071680 2024-04-25T00:18:10.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.496+0000 7f39737fe640 1 -- 172.21.15.116:0/2598129971 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) ==== 95+0+6509 (secure 0 0 0) 0x7f39800597f0 con 0x7f398c071680 2024-04-25T00:18:10.497 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:18:10.497 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":137,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7362,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/1655959433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1655959433},{"type":"v1","addr":"172.21.15.116:6833","nonce":1655959433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:17:38.902992+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6343":{"gid":6343,"name":"d","rank":0,"incarnation":133,"state":"up:active","state_seq":4,"addr":"172.21.15.174:6837/1281466397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1281466397},{"type":"v1","addr":"172.21.15.174:6837","nonce":1281466397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6343,"qdb_cluster":[6343]},"id":1},{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:17:28.837517+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7164},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7164":{"gid":7164,"name":"c","rank":0,"incarnation":128,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6835/3336339376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3336339376},{"type":"v1","addr":"172.21.15.116:6835","nonce":3336339376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7164,"qdb_cluster":[7164]},"id":2}]} 2024-04-25T00:18:10.497 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 137 2024-04-25T00:18:10.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.496+0000 7f39917e1640 1 -- 172.21.15.116:0/2598129971 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f39780678c0 msgr2=0x7f3978069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:10.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.496+0000 7f39917e1640 1 --2- 172.21.15.116:0/2598129971 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f39780678c0 0x7f3978069d80 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f397c0b1000 tx=0x7f397c008040 comp rx=0 tx=0).stop 2024-04-25T00:18:10.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.496+0000 7f39917e1640 1 -- 172.21.15.116:0/2598129971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f398c071680 msgr2=0x7f398c1402f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:10.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.496+0000 7f39917e1640 1 --2- 172.21.15.116:0/2598129971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f398c071680 0x7f398c1402f0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f398000c420 tx=0x7f398000c8e0 comp rx=0 tx=0).stop 2024-04-25T00:18:10.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.496+0000 7f39917e1640 1 -- 172.21.15.116:0/2598129971 shutdown_connections 2024-04-25T00:18:10.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.496+0000 7f39917e1640 1 --2- 172.21.15.116:0/2598129971 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f39780678c0 0x7f3978069d80 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:10.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.496+0000 7f39917e1640 1 --2- 172.21.15.116:0/2598129971 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f398c14c720 0x7f398c1537f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:10.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.496+0000 7f39917e1640 1 --2- 172.21.15.116:0/2598129971 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f398c0a94a0 0x7f398c140830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:10.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.496+0000 7f39917e1640 1 --2- 172.21.15.116:0/2598129971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f398c071680 0x7f398c1402f0 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:10.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.496+0000 7f39917e1640 1 -- 172.21.15.116:0/2598129971 >> 172.21.15.116:0/2598129971 conn(0x7f398c0c3b00 msgr2=0x7f398c0c25f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:10.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.496+0000 7f39917e1640 1 -- 172.21.15.116:0/2598129971 shutdown_connections 2024-04-25T00:18:10.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:10.496+0000 7f39917e1640 1 -- 172.21.15.116:0/2598129971 wait complete. 2024-04-25T00:18:10.515 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:18:12.517 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:18:12.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.648+0000 7f95da6c1640 1 Processor -- start 2024-04-25T00:18:12.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.648+0000 7f95da6c1640 1 -- start start 2024-04-25T00:18:12.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.648+0000 7f95da6c1640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f95d40a94a0 0x7f95d41538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:12.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.648+0000 7f95da6c1640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f95d40a99f0 0x7f95d4153e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:12.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.648+0000 7f95da6c1640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95d40c3910 con 0x7f95d4150d20 2024-04-25T00:18:12.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.648+0000 7f95da6c1640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f95d40c16e0 con 0x7f95d40a99f0 2024-04-25T00:18:12.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.648+0000 7f95da6c1640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f95d40c1400 con 0x7f95d40a94a0 2024-04-25T00:18:12.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.648+0000 7f95d8c5e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f95d40a99f0 0x7f95d4153e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:12.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.648+0000 7f95d8c5e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f95d40a99f0 0x7f95d4153e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41918/0 (socket says 172.21.15.116:41918) 2024-04-25T00:18:12.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.648+0000 7f95d8c5e640 1 -- 172.21.15.116:0/1812245534 learned_addr learned my addr 172.21.15.116:0/1812245534 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:18:12.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.648+0000 7f95d37fe640 1 --2- 172.21.15.116:0/1812245534 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f95d40a94a0 0x7f95d41538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:12.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.648+0000 7f95d8c5e640 1 -- 172.21.15.116:0/1812245534 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f95d40a94a0 msgr2=0x7f95d41538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:12.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.648+0000 7f95d8c5e640 1 --2- 172.21.15.116:0/1812245534 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f95d40a94a0 0x7f95d41538e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:12.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.648+0000 7f95d8c5e640 1 -- 172.21.15.116:0/1812245534 >> v1:172.21.15.174:6789/0 conn(0x7f95d4150d20 legacy=0x7f95d40c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:12.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.648+0000 7f95d8c5e640 1 -- 172.21.15.116:0/1812245534 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95d406e660 con 0x7f95d40a99f0 2024-04-25T00:18:12.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.652+0000 7f95d8c5e640 1 --2- 172.21.15.116:0/1812245534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95d40a99f0 0x7f95d4153e20 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f95cc009e50 tx=0x7f95cc01ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c5f379d0cfd21ca server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:12.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.652+0000 7f95d2ffd640 1 -- 172.21.15.116:0/1812245534 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95cc0a5700 con 0x7f95d40a99f0 2024-04-25T00:18:12.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.652+0000 7f95d2ffd640 1 -- 172.21.15.116:0/1812245534 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95cc0a58a0 con 0x7f95d40a99f0 2024-04-25T00:18:12.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.652+0000 7f95d2ffd640 1 -- 172.21.15.116:0/1812245534 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95cc0a5c70 con 0x7f95d40a99f0 2024-04-25T00:18:12.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.652+0000 7f95da6c1640 1 -- 172.21.15.116:0/1812245534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95d40a99f0 msgr2=0x7f95d4153e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:12.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.652+0000 7f95da6c1640 1 --2- 172.21.15.116:0/1812245534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95d40a99f0 0x7f95d4153e20 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f95cc009e50 tx=0x7f95cc01ca30 comp rx=0 tx=0).stop 2024-04-25T00:18:12.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.652+0000 7f95da6c1640 1 -- 172.21.15.116:0/1812245534 shutdown_connections 2024-04-25T00:18:12.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.652+0000 7f95da6c1640 1 --2- 172.21.15.116:0/1812245534 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95d40a99f0 0x7f95d4153e20 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:12.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.652+0000 7f95da6c1640 1 --2- 172.21.15.116:0/1812245534 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f95d40a94a0 0x7f95d41538e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:12.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.652+0000 7f95da6c1640 1 -- 172.21.15.116:0/1812245534 >> 172.21.15.116:0/1812245534 conn(0x7f95d40c3b00 msgr2=0x7f95d40c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:12.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.652+0000 7f95da6c1640 1 -- 172.21.15.116:0/1812245534 shutdown_connections 2024-04-25T00:18:12.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.652+0000 7f95da6c1640 1 -- 172.21.15.116:0/1812245534 wait complete. 2024-04-25T00:18:12.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.652+0000 7f95da6c1640 1 Processor -- start 2024-04-25T00:18:12.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.652+0000 7f95da6c1640 1 -- start start 2024-04-25T00:18:12.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.656+0000 7f95da6c1640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f95d40a94a0 0x7f95d4072750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:12.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.656+0000 7f95da6c1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95d40a99f0 0x7f95d414f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:12.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.656+0000 7f95da6c1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f95d4150d20 0x7f95d41416e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:12.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.656+0000 7f95da6c1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f95d41544e0 con 0x7f95d40a99f0 2024-04-25T00:18:12.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.656+0000 7f95da6c1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f95d4154360 con 0x7f95d4150d20 2024-04-25T00:18:12.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.656+0000 7f95da6c1640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f95d4160730 con 0x7f95d40a94a0 2024-04-25T00:18:12.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.656+0000 7f95d37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95d40a99f0 0x7f95d414f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:12.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.656+0000 7f95d37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95d40a99f0 0x7f95d414f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41926/0 (socket says 172.21.15.116:41926) 2024-04-25T00:18:12.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.656+0000 7f95d37fe640 1 -- 172.21.15.116:0/998833221 learned_addr learned my addr 172.21.15.116:0/998833221 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:18:12.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.656+0000 7f95d8c5e640 1 --2- 172.21.15.116:0/998833221 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f95d4150d20 0x7f95d41416e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:12.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.656+0000 7f95d3fff640 1 --2- 172.21.15.116:0/998833221 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f95d40a94a0 0x7f95d4072750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:12.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.656+0000 7f95d37fe640 1 -- 172.21.15.116:0/998833221 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f95d40a94a0 msgr2=0x7f95d4072750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:12.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.656+0000 7f95d37fe640 1 --2- 172.21.15.116:0/998833221 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f95d40a94a0 0x7f95d4072750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:12.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.656+0000 7f95d37fe640 1 -- 172.21.15.116:0/998833221 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f95d4150d20 msgr2=0x7f95d41416e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:12.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.656+0000 7f95d37fe640 1 --2- 172.21.15.116:0/998833221 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f95d4150d20 0x7f95d41416e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:12.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.656+0000 7f95d37fe640 1 -- 172.21.15.116:0/998833221 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95d41ee360 con 0x7f95d40a99f0 2024-04-25T00:18:12.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.656+0000 7f95d37fe640 1 --2- 172.21.15.116:0/998833221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95d40a99f0 0x7f95d414f0d0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f95c400d5f0 tx=0x7f95c400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:12.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.656+0000 7f95d0ff9640 1 -- 172.21.15.116:0/998833221 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95c40125f0 con 0x7f95d40a99f0 2024-04-25T00:18:12.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.656+0000 7f95d0ff9640 1 -- 172.21.15.116:0/998833221 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95c4019070 con 0x7f95d40a99f0 2024-04-25T00:18:12.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.656+0000 7f95da6c1640 1 -- 172.21.15.116:0/998833221 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95d41ee620 con 0x7f95d40a99f0 2024-04-25T00:18:12.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.656+0000 7f95d0ff9640 1 -- 172.21.15.116:0/998833221 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95c40129d0 con 0x7f95d40a99f0 2024-04-25T00:18:12.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.656+0000 7f95da6c1640 1 -- 172.21.15.116:0/998833221 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95d41eebe0 con 0x7f95d40a99f0 2024-04-25T00:18:12.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.656+0000 7f95da6c1640 1 -- 172.21.15.116:0/998833221 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95d4068350 con 0x7f95d40a99f0 2024-04-25T00:18:12.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.656+0000 7f95d0ff9640 1 -- 172.21.15.116:0/998833221 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f95c4012b70 con 0x7f95d40a99f0 2024-04-25T00:18:12.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.656+0000 7f95d0ff9640 1 --2- 172.21.15.116:0/998833221 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f95c00678c0 0x7f95c0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:12.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.656+0000 7f95d0ff9640 1 -- 172.21.15.116:0/998833221 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 8816+0+0 (secure 0 0 0) 0x7f95c4086d00 con 0x7f95d40a99f0 2024-04-25T00:18:12.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.660+0000 7f95d3fff640 1 --2- 172.21.15.116:0/998833221 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f95c00678c0 0x7f95c0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:12.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.660+0000 7f95d3fff640 1 --2- 172.21.15.116:0/998833221 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f95c00678c0 0x7f95c0069d80 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f95c8000f70 tx=0x7f95c8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:12.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.664+0000 7f95d0ff9640 1 -- 172.21.15.116:0/998833221 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f95c4056c60 con 0x7f95d40a99f0 2024-04-25T00:18:12.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.860+0000 7f95da6c1640 1 -- 172.21.15.116:0/998833221 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f95d406e0c0 con 0x7f95d40a99f0 2024-04-25T00:18:12.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.860+0000 7f95d0ff9640 1 -- 172.21.15.116:0/998833221 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) ==== 95+0+6509 (secure 0 0 0) 0x7f95c4056560 con 0x7f95d40a99f0 2024-04-25T00:18:12.862 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:18:12.862 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":137,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7362,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/1655959433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1655959433},{"type":"v1","addr":"172.21.15.116:6833","nonce":1655959433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:17:38.902992+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6343":{"gid":6343,"name":"d","rank":0,"incarnation":133,"state":"up:active","state_seq":4,"addr":"172.21.15.174:6837/1281466397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1281466397},{"type":"v1","addr":"172.21.15.174:6837","nonce":1281466397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6343,"qdb_cluster":[6343]},"id":1},{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:17:28.837517+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7164},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7164":{"gid":7164,"name":"c","rank":0,"incarnation":128,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6835/3336339376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3336339376},{"type":"v1","addr":"172.21.15.116:6835","nonce":3336339376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7164,"qdb_cluster":[7164]},"id":2}]} 2024-04-25T00:18:12.863 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 137 2024-04-25T00:18:12.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.864+0000 7f95da6c1640 1 -- 172.21.15.116:0/998833221 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f95c00678c0 msgr2=0x7f95c0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:12.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.864+0000 7f95da6c1640 1 --2- 172.21.15.116:0/998833221 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f95c00678c0 0x7f95c0069d80 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f95c8000f70 tx=0x7f95c8009040 comp rx=0 tx=0).stop 2024-04-25T00:18:12.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.864+0000 7f95da6c1640 1 -- 172.21.15.116:0/998833221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95d40a99f0 msgr2=0x7f95d414f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:12.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.864+0000 7f95da6c1640 1 --2- 172.21.15.116:0/998833221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95d40a99f0 0x7f95d414f0d0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f95c400d5f0 tx=0x7f95c400dab0 comp rx=0 tx=0).stop 2024-04-25T00:18:12.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.864+0000 7f95da6c1640 1 -- 172.21.15.116:0/998833221 shutdown_connections 2024-04-25T00:18:12.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.864+0000 7f95da6c1640 1 --2- 172.21.15.116:0/998833221 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f95c00678c0 0x7f95c0069d80 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:12.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.864+0000 7f95da6c1640 1 --2- 172.21.15.116:0/998833221 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f95d4150d20 0x7f95d41416e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:12.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.864+0000 7f95da6c1640 1 --2- 172.21.15.116:0/998833221 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f95d40a99f0 0x7f95d414f0d0 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:12.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.864+0000 7f95da6c1640 1 --2- 172.21.15.116:0/998833221 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f95d40a94a0 0x7f95d4072750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:12.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.864+0000 7f95da6c1640 1 -- 172.21.15.116:0/998833221 >> 172.21.15.116:0/998833221 conn(0x7f95d40c3b00 msgr2=0x7f95d4146f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:12.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.864+0000 7f95da6c1640 1 -- 172.21.15.116:0/998833221 shutdown_connections 2024-04-25T00:18:12.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:12.864+0000 7f95da6c1640 1 -- 172.21.15.116:0/998833221 wait complete. 2024-04-25T00:18:12.881 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:18:14.882 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:18:15.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.008+0000 7f39f3d6d640 1 Processor -- start 2024-04-25T00:18:15.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.008+0000 7f39f3d6d640 1 -- start start 2024-04-25T00:18:15.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.012+0000 7f39f3d6d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f39ec13e310 0x7f39ec0c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:15.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.012+0000 7f39f3d6d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f39ec0c2f50 0x7f39ec1559c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:15.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.012+0000 7f39f3d6d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39ec06d200 con 0x7f39ec13ed60 2024-04-25T00:18:15.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.012+0000 7f39f3d6d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f39ec070d40 con 0x7f39ec0c2f50 2024-04-25T00:18:15.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.012+0000 7f39f3d6d640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f39ec071020 con 0x7f39ec13e310 2024-04-25T00:18:15.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.012+0000 7f39f1308640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f39ec0c2f50 0x7f39ec1559c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:15.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.012+0000 7f39f1308640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f39ec0c2f50 0x7f39ec1559c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49246/0 (socket says 172.21.15.116:49246) 2024-04-25T00:18:15.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.012+0000 7f39f230a640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f39ec13ed60 0x7f39ec155f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:56666/0 (socket says 172.21.15.116:56666) 2024-04-25T00:18:15.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.012+0000 7f39f1308640 1 -- 172.21.15.116:0/2260387837 learned_addr learned my addr 172.21.15.116:0/2260387837 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:18:15.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.012+0000 7f39f1b09640 1 --2- 172.21.15.116:0/2260387837 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f39ec13e310 0x7f39ec0c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:15.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.012+0000 7f39f1308640 1 -- 172.21.15.116:0/2260387837 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f39ec13e310 msgr2=0x7f39ec0c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:15.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.012+0000 7f39f1308640 1 --2- 172.21.15.116:0/2260387837 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f39ec13e310 0x7f39ec0c2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:15.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.012+0000 7f39f1308640 1 -- 172.21.15.116:0/2260387837 >> v1:172.21.15.116:6789/0 conn(0x7f39ec13ed60 legacy=0x7f39ec155f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:15.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.012+0000 7f39f0b07640 1 -- 172.21.15.116:0/2260387837 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 984422952 0 0) 0x7f39ec06d200 con 0x7f39ec13ed60 2024-04-25T00:18:15.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.012+0000 7f39f1308640 1 -- 172.21.15.116:0/2260387837 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39ec156610 con 0x7f39ec0c2f50 2024-04-25T00:18:15.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.012+0000 7f39f1308640 1 --2- 172.21.15.116:0/2260387837 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39ec0c2f50 0x7f39ec1559c0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f39e4009e50 tx=0x7f39e401c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3624d55fffd68896 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:15.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.012+0000 7f39f0b07640 1 -- 172.21.15.116:0/2260387837 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39e40a56e0 con 0x7f39ec0c2f50 2024-04-25T00:18:15.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.012+0000 7f39f0b07640 1 -- 172.21.15.116:0/2260387837 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39e40a5880 con 0x7f39ec0c2f50 2024-04-25T00:18:15.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.012+0000 7f39f0b07640 1 -- 172.21.15.116:0/2260387837 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39e40a5c50 con 0x7f39ec0c2f50 2024-04-25T00:18:15.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.016+0000 7f39f3d6d640 1 -- 172.21.15.116:0/2260387837 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39ec0c2f50 msgr2=0x7f39ec1559c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:15.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.016+0000 7f39f3d6d640 1 --2- 172.21.15.116:0/2260387837 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39ec0c2f50 0x7f39ec1559c0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f39e4009e50 tx=0x7f39e401c910 comp rx=0 tx=0).stop 2024-04-25T00:18:15.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.016+0000 7f39f3d6d640 1 -- 172.21.15.116:0/2260387837 shutdown_connections 2024-04-25T00:18:15.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.016+0000 7f39f3d6d640 1 --2- 172.21.15.116:0/2260387837 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39ec0c2f50 0x7f39ec1559c0 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:15.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.016+0000 7f39f3d6d640 1 --2- 172.21.15.116:0/2260387837 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f39ec13e310 0x7f39ec0c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:15.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.016+0000 7f39f3d6d640 1 -- 172.21.15.116:0/2260387837 >> 172.21.15.116:0/2260387837 conn(0x7f39ec0c3b00 msgr2=0x7f39ec0c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:15.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.016+0000 7f39f3d6d640 1 -- 172.21.15.116:0/2260387837 shutdown_connections 2024-04-25T00:18:15.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.016+0000 7f39f3d6d640 1 -- 172.21.15.116:0/2260387837 wait complete. 2024-04-25T00:18:15.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.016+0000 7f39f3d6d640 1 Processor -- start 2024-04-25T00:18:15.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.016+0000 7f39f3d6d640 1 -- start start 2024-04-25T00:18:15.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.016+0000 7f39f3d6d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39ec0c2f50 0x7f39ec134c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:15.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.016+0000 7f39f3d6d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f39ec13e310 0x7f39ec0c55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:15.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.016+0000 7f39f3d6d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f39ec13ed60 0x7f39ec0c5b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:15.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.016+0000 7f39f3d6d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f39ec162b70 con 0x7f39ec0c2f50 2024-04-25T00:18:15.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.016+0000 7f39f3d6d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f39ec1629f0 con 0x7f39ec13e310 2024-04-25T00:18:15.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.016+0000 7f39f3d6d640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f39ec162cf0 con 0x7f39ec13ed60 2024-04-25T00:18:15.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.016+0000 7f39f1b09640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39ec0c2f50 0x7f39ec134c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:15.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.016+0000 7f39f1b09640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39ec0c2f50 0x7f39ec134c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49248/0 (socket says 172.21.15.116:49248) 2024-04-25T00:18:15.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.016+0000 7f39f1b09640 1 -- 172.21.15.116:0/1133956025 learned_addr learned my addr 172.21.15.116:0/1133956025 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:18:15.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.016+0000 7f39f230a640 1 --2- 172.21.15.116:0/1133956025 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f39ec13ed60 0x7f39ec0c5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:15.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.016+0000 7f39f1308640 1 --2- 172.21.15.116:0/1133956025 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f39ec13e310 0x7f39ec0c55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:15.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.016+0000 7f39f1b09640 1 -- 172.21.15.116:0/1133956025 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f39ec13ed60 msgr2=0x7f39ec0c5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:15.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.016+0000 7f39f1b09640 1 --2- 172.21.15.116:0/1133956025 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f39ec13ed60 0x7f39ec0c5b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:15.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.016+0000 7f39f1b09640 1 -- 172.21.15.116:0/1133956025 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f39ec13e310 msgr2=0x7f39ec0c55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:15.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.016+0000 7f39f1b09640 1 --2- 172.21.15.116:0/1133956025 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f39ec13e310 0x7f39ec0c55e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:15.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.016+0000 7f39f1b09640 1 -- 172.21.15.116:0/1133956025 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39ec154870 con 0x7f39ec0c2f50 2024-04-25T00:18:15.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.016+0000 7f39f1308640 1 --2- 172.21.15.116:0/1133956025 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f39ec13e310 0x7f39ec0c55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:18:15.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.016+0000 7f39f1b09640 1 --2- 172.21.15.116:0/1133956025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39ec0c2f50 0x7f39ec134c10 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f39e000d800 tx=0x7f39e000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:15.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.016+0000 7f39de7fc640 1 -- 172.21.15.116:0/1133956025 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39e000b6c0 con 0x7f39ec0c2f50 2024-04-25T00:18:15.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.016+0000 7f39f3d6d640 1 -- 172.21.15.116:0/1133956025 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39ec154b50 con 0x7f39ec0c2f50 2024-04-25T00:18:15.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.016+0000 7f39de7fc640 1 -- 172.21.15.116:0/1133956025 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39e0019070 con 0x7f39ec0c2f50 2024-04-25T00:18:15.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.016+0000 7f39de7fc640 1 -- 172.21.15.116:0/1133956025 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39e000ba60 con 0x7f39ec0c2f50 2024-04-25T00:18:15.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.016+0000 7f39f3d6d640 1 -- 172.21.15.116:0/1133956025 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39ec0c50c0 con 0x7f39ec0c2f50 2024-04-25T00:18:15.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.020+0000 7f39f3d6d640 1 -- 172.21.15.116:0/1133956025 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39ec068350 con 0x7f39ec0c2f50 2024-04-25T00:18:15.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.020+0000 7f39de7fc640 1 -- 172.21.15.116:0/1133956025 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f39e000bc00 con 0x7f39ec0c2f50 2024-04-25T00:18:15.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.020+0000 7f39de7fc640 1 --2- 172.21.15.116:0/1133956025 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f39d00678c0 0x7f39d0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:15.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.020+0000 7f39de7fc640 1 -- 172.21.15.116:0/1133956025 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 8816+0+0 (secure 0 0 0) 0x7f39e0086dd0 con 0x7f39ec0c2f50 2024-04-25T00:18:15.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.020+0000 7f39f1308640 1 --2- 172.21.15.116:0/1133956025 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f39d00678c0 0x7f39d0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:15.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.020+0000 7f39f1308640 1 --2- 172.21.15.116:0/1133956025 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f39d00678c0 0x7f39d0069d80 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f39e40ae000 tx=0x7f39e40a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:15.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.024+0000 7f39de7fc640 1 -- 172.21.15.116:0/1133956025 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f39e008a030 con 0x7f39ec0c2f50 2024-04-25T00:18:15.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.224+0000 7f39f3d6d640 1 -- 172.21.15.116:0/1133956025 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f39ec0c34f0 con 0x7f39ec0c2f50 2024-04-25T00:18:15.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.224+0000 7f39de7fc640 1 -- 172.21.15.116:0/1133956025 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) ==== 95+0+6509 (secure 0 0 0) 0x7f39e0056c70 con 0x7f39ec0c2f50 2024-04-25T00:18:15.227 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:18:15.227 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":137,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7362,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/1655959433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1655959433},{"type":"v1","addr":"172.21.15.116:6833","nonce":1655959433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:17:38.902992+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6343":{"gid":6343,"name":"d","rank":0,"incarnation":133,"state":"up:active","state_seq":4,"addr":"172.21.15.174:6837/1281466397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1281466397},{"type":"v1","addr":"172.21.15.174:6837","nonce":1281466397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6343,"qdb_cluster":[6343]},"id":1},{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:17:28.837517+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7164},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7164":{"gid":7164,"name":"c","rank":0,"incarnation":128,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6835/3336339376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3336339376},{"type":"v1","addr":"172.21.15.116:6835","nonce":3336339376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7164,"qdb_cluster":[7164]},"id":2}]} 2024-04-25T00:18:15.227 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 137 2024-04-25T00:18:15.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.228+0000 7f39f3d6d640 1 -- 172.21.15.116:0/1133956025 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f39d00678c0 msgr2=0x7f39d0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:15.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.228+0000 7f39f3d6d640 1 --2- 172.21.15.116:0/1133956025 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f39d00678c0 0x7f39d0069d80 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f39e40ae000 tx=0x7f39e40a7070 comp rx=0 tx=0).stop 2024-04-25T00:18:15.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.228+0000 7f39f3d6d640 1 -- 172.21.15.116:0/1133956025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39ec0c2f50 msgr2=0x7f39ec134c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:15.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.228+0000 7f39f3d6d640 1 --2- 172.21.15.116:0/1133956025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39ec0c2f50 0x7f39ec134c10 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f39e000d800 tx=0x7f39e000dcc0 comp rx=0 tx=0).stop 2024-04-25T00:18:15.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.228+0000 7f39f3d6d640 1 -- 172.21.15.116:0/1133956025 shutdown_connections 2024-04-25T00:18:15.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.228+0000 7f39f3d6d640 1 --2- 172.21.15.116:0/1133956025 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f39d00678c0 0x7f39d0069d80 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:15.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.228+0000 7f39f3d6d640 1 --2- 172.21.15.116:0/1133956025 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f39ec13ed60 0x7f39ec0c5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:15.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.228+0000 7f39f3d6d640 1 --2- 172.21.15.116:0/1133956025 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f39ec13e310 0x7f39ec0c55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:15.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.228+0000 7f39f3d6d640 1 --2- 172.21.15.116:0/1133956025 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f39ec0c2f50 0x7f39ec134c10 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:15.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.228+0000 7f39f3d6d640 1 -- 172.21.15.116:0/1133956025 >> 172.21.15.116:0/1133956025 conn(0x7f39ec0c3b00 msgr2=0x7f39ec137bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:15.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.228+0000 7f39f3d6d640 1 -- 172.21.15.116:0/1133956025 shutdown_connections 2024-04-25T00:18:15.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:15.228+0000 7f39f3d6d640 1 -- 172.21.15.116:0/1133956025 wait complete. 2024-04-25T00:18:15.246 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:18:17.247 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:18:17.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.380+0000 7f9b22bd3640 1 Processor -- start 2024-04-25T00:18:17.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.380+0000 7f9b22bd3640 1 -- start start 2024-04-25T00:18:17.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.380+0000 7f9b22bd3640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9b1c071d90 0x7f9b1c1563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:17.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.380+0000 7f9b22bd3640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b1c071220 con 0x7f9b1c0a94a0 2024-04-25T00:18:17.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.380+0000 7f9b22bd3640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b1c152420 con 0x7f9b1c14c720 2024-04-25T00:18:17.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.380+0000 7f9b22bd3640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f9b1c153460 con 0x7f9b1c071d90 2024-04-25T00:18:17.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.380+0000 7f9b13fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f9b1c0a94a0 0x7f9b1c071680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:56674/0 (socket says 172.21.15.116:56674) 2024-04-25T00:18:17.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.380+0000 7f9b13fff640 1 -- 172.21.15.116:0/1477901486 learned_addr learned my addr 172.21.15.116:0/1477901486 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:18:17.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.380+0000 7f9b21170640 1 --2- 172.21.15.116:0/1477901486 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9b1c071d90 0x7f9b1c1563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:17.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.380+0000 7f9b137fe640 1 -- 172.21.15.116:0/1477901486 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 181243317 0 0) 0x7f9b1c071220 con 0x7f9b1c0a94a0 2024-04-25T00:18:17.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.380+0000 7f9b137fe640 1 -- 172.21.15.116:0/1477901486 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b04003530 con 0x7f9b1c0a94a0 2024-04-25T00:18:17.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.380+0000 7f9b137fe640 1 -- 172.21.15.116:0/1477901486 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2298997472 0 0) 0x7f9b04003530 con 0x7f9b1c0a94a0 2024-04-25T00:18:17.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.380+0000 7f9b137fe640 1 -- 172.21.15.116:0/1477901486 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9b1c071d90 msgr2=0x7f9b1c1563d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:17.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.380+0000 7f9b137fe640 1 --2- 172.21.15.116:0/1477901486 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9b1c071d90 0x7f9b1c1563d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:17.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.380+0000 7f9b137fe640 1 -- 172.21.15.116:0/1477901486 >> v1:172.21.15.174:6789/0 conn(0x7f9b1c14c720 legacy=0x7f9b1c0c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:17.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.380+0000 7f9b137fe640 1 -- 172.21.15.116:0/1477901486 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b1c0721c0 con 0x7f9b1c0a94a0 2024-04-25T00:18:17.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.380+0000 7f9b137fe640 1 -- 172.21.15.116:0/1477901486 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f9b0c001680 con 0x7f9b1c0a94a0 2024-04-25T00:18:17.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.380+0000 7f9b137fe640 1 -- 172.21.15.116:0/1477901486 >> v1:172.21.15.116:6789/0 conn(0x7f9b1c0a94a0 legacy=0x7f9b1c071680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:17.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.380+0000 7f9b137fe640 1 --2- 172.21.15.116:0/1477901486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b040039f0 0x7f9b04003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:17.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.380+0000 7f9b137fe640 1 --2- 172.21.15.116:0/1477901486 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9b040043a0 0x7f9b04004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:17.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.380+0000 7f9b137fe640 1 --2- 172.21.15.116:0/1477901486 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9b1c071d90 0x7f9b04005170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:17.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.380+0000 7f9b137fe640 1 -- 172.21.15.116:0/1477901486 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9b1c153460 con 0x7f9b040039f0 2024-04-25T00:18:17.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.380+0000 7f9b137fe640 1 -- 172.21.15.116:0/1477901486 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9b0400bab0 con 0x7f9b040043a0 2024-04-25T00:18:17.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.380+0000 7f9b137fe640 1 -- 172.21.15.116:0/1477901486 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f9b0400bcb0 con 0x7f9b1c071d90 2024-04-25T00:18:17.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.380+0000 7f9b2096f640 1 --2- 172.21.15.116:0/1477901486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b040039f0 0x7f9b04003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:17.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.380+0000 7f9b13fff640 1 --2- 172.21.15.116:0/1477901486 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9b040043a0 0x7f9b04004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:17.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.380+0000 7f9b21170640 1 --2- 172.21.15.116:0/1477901486 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9b1c071d90 0x7f9b04005170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:17.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.380+0000 7f9b2096f640 1 -- 172.21.15.116:0/1477901486 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9b1c071d90 msgr2=0x7f9b04005170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:17.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.380+0000 7f9b2096f640 1 --2- 172.21.15.116:0/1477901486 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9b1c071d90 0x7f9b04005170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:17.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.380+0000 7f9b2096f640 1 -- 172.21.15.116:0/1477901486 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9b040043a0 msgr2=0x7f9b04004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:17.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.380+0000 7f9b2096f640 1 --2- 172.21.15.116:0/1477901486 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9b040043a0 0x7f9b04004870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:17.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.380+0000 7f9b2096f640 1 -- 172.21.15.116:0/1477901486 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b04003530 con 0x7f9b040039f0 2024-04-25T00:18:17.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.384+0000 7f9b2096f640 1 --2- 172.21.15.116:0/1477901486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b040039f0 0x7f9b04003dd0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f9b18001a30 tx=0x7f9b1801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fb8b42bef975d61 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:17.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.384+0000 7f9b137fe640 1 -- 172.21.15.116:0/1477901486 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b18001630 con 0x7f9b040039f0 2024-04-25T00:18:17.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.384+0000 7f9b137fe640 1 -- 172.21.15.116:0/1477901486 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b180b0070 con 0x7f9b040039f0 2024-04-25T00:18:17.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.384+0000 7f9b137fe640 1 -- 172.21.15.116:0/1477901486 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b180ab880 con 0x7f9b040039f0 2024-04-25T00:18:17.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.384+0000 7f9b22bd3640 1 -- 172.21.15.116:0/1477901486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b040039f0 msgr2=0x7f9b04003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:17.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.384+0000 7f9b22bd3640 1 --2- 172.21.15.116:0/1477901486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b040039f0 0x7f9b04003dd0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f9b18001a30 tx=0x7f9b1801c520 comp rx=0 tx=0).stop 2024-04-25T00:18:17.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.384+0000 7f9b21170640 1 -- 172.21.15.116:0/1477901486 reap_dead start 2024-04-25T00:18:17.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.384+0000 7f9b22bd3640 1 -- 172.21.15.116:0/1477901486 shutdown_connections 2024-04-25T00:18:17.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.384+0000 7f9b22bd3640 1 -- 172.21.15.116:0/1477901486 >> 172.21.15.116:0/1477901486 conn(0x7f9b1c0c3b00 msgr2=0x7f9b1c0c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:17.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.384+0000 7f9b22bd3640 1 -- 172.21.15.116:0/1477901486 shutdown_connections 2024-04-25T00:18:17.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.384+0000 7f9b22bd3640 1 -- 172.21.15.116:0/1477901486 wait complete. 2024-04-25T00:18:17.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.384+0000 7f9b22bd3640 1 Processor -- start 2024-04-25T00:18:17.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.384+0000 7f9b22bd3640 1 -- start start 2024-04-25T00:18:17.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.384+0000 7f9b22bd3640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9b040039f0 0x7f9b1c0c52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:17.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.388+0000 7f9b22bd3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b1c0c6bb0 0x7f9b1c153dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:17.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.388+0000 7f9b22bd3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9b1c154c80 0x7f9b1c139410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:17.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.388+0000 7f9b22bd3640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9b1c152c60 con 0x7f9b1c0c6bb0 2024-04-25T00:18:17.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.388+0000 7f9b22bd3640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9b1c152980 con 0x7f9b1c154c80 2024-04-25T00:18:17.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.388+0000 7f9b22bd3640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f9b1c152f40 con 0x7f9b040039f0 2024-04-25T00:18:17.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.388+0000 7f9b13fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b1c0c6bb0 0x7f9b1c153dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:17.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.388+0000 7f9b13fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b1c0c6bb0 0x7f9b1c153dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49276/0 (socket says 172.21.15.116:49276) 2024-04-25T00:18:17.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.388+0000 7f9b13fff640 1 -- 172.21.15.116:0/4274217765 learned_addr learned my addr 172.21.15.116:0/4274217765 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:18:17.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.388+0000 7f9b21170640 1 --2- 172.21.15.116:0/4274217765 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9b1c154c80 0x7f9b1c139410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:17.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.388+0000 7f9b2096f640 1 --2- 172.21.15.116:0/4274217765 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9b040039f0 0x7f9b1c0c52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:17.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.388+0000 7f9b13fff640 1 -- 172.21.15.116:0/4274217765 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9b040039f0 msgr2=0x7f9b1c0c52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:17.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.388+0000 7f9b13fff640 1 --2- 172.21.15.116:0/4274217765 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9b040039f0 0x7f9b1c0c52b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:17.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.388+0000 7f9b13fff640 1 -- 172.21.15.116:0/4274217765 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9b1c154c80 msgr2=0x7f9b1c139410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:17.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.388+0000 7f9b13fff640 1 --2- 172.21.15.116:0/4274217765 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9b1c154c80 0x7f9b1c139410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:17.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.388+0000 7f9b13fff640 1 -- 172.21.15.116:0/4274217765 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b1c154830 con 0x7f9b1c0c6bb0 2024-04-25T00:18:17.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.388+0000 7f9b13fff640 1 --2- 172.21.15.116:0/4274217765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b1c0c6bb0 0x7f9b1c153dc0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f9b0c00e780 tx=0x7f9b0c00ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:17.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.388+0000 7f9b117fa640 1 -- 172.21.15.116:0/4274217765 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b0c00c660 con 0x7f9b1c0c6bb0 2024-04-25T00:18:17.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.388+0000 7f9b117fa640 1 -- 172.21.15.116:0/4274217765 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b0c018070 con 0x7f9b1c0c6bb0 2024-04-25T00:18:17.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.388+0000 7f9b22bd3640 1 -- 172.21.15.116:0/4274217765 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b1c144820 con 0x7f9b1c0c6bb0 2024-04-25T00:18:17.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.388+0000 7f9b117fa640 1 -- 172.21.15.116:0/4274217765 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b0c00ca60 con 0x7f9b1c0c6bb0 2024-04-25T00:18:17.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.388+0000 7f9b22bd3640 1 -- 172.21.15.116:0/4274217765 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b1c13a5c0 con 0x7f9b1c0c6bb0 2024-04-25T00:18:17.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.388+0000 7f9b22bd3640 1 -- 172.21.15.116:0/4274217765 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b1c068350 con 0x7f9b1c0c6bb0 2024-04-25T00:18:17.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.388+0000 7f9b117fa640 1 -- 172.21.15.116:0/4274217765 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f9b0c00cc00 con 0x7f9b1c0c6bb0 2024-04-25T00:18:17.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.388+0000 7f9b117fa640 1 --2- 172.21.15.116:0/4274217765 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9b080678c0 0x7f9b08069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:17.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.388+0000 7f9b117fa640 1 -- 172.21.15.116:0/4274217765 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 8816+0+0 (secure 0 0 0) 0x7f9b0c086610 con 0x7f9b1c0c6bb0 2024-04-25T00:18:17.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.388+0000 7f9b2096f640 1 --2- 172.21.15.116:0/4274217765 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9b080678c0 0x7f9b08069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:17.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.392+0000 7f9b2096f640 1 --2- 172.21.15.116:0/4274217765 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9b080678c0 0x7f9b08069d80 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f9b180af000 tx=0x7f9b180a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:17.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.392+0000 7f9b117fa640 1 -- 172.21.15.116:0/4274217765 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f9b0c056570 con 0x7f9b1c0c6bb0 2024-04-25T00:18:17.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.592+0000 7f9b22bd3640 1 -- 172.21.15.116:0/4274217765 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9b1c0c4bf0 con 0x7f9b1c0c6bb0 2024-04-25T00:18:17.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.592+0000 7f9b117fa640 1 -- 172.21.15.116:0/4274217765 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) ==== 95+0+6509 (secure 0 0 0) 0x7f9b0c055e70 con 0x7f9b1c0c6bb0 2024-04-25T00:18:17.596 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:18:17.596 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":137,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7362,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/1655959433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1655959433},{"type":"v1","addr":"172.21.15.116:6833","nonce":1655959433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:17:38.902992+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6343":{"gid":6343,"name":"d","rank":0,"incarnation":133,"state":"up:active","state_seq":4,"addr":"172.21.15.174:6837/1281466397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1281466397},{"type":"v1","addr":"172.21.15.174:6837","nonce":1281466397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6343,"qdb_cluster":[6343]},"id":1},{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:17:28.837517+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7164},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7164":{"gid":7164,"name":"c","rank":0,"incarnation":128,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6835/3336339376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3336339376},{"type":"v1","addr":"172.21.15.116:6835","nonce":3336339376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7164,"qdb_cluster":[7164]},"id":2}]} 2024-04-25T00:18:17.596 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 137 2024-04-25T00:18:17.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.596+0000 7f9b22bd3640 1 -- 172.21.15.116:0/4274217765 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9b080678c0 msgr2=0x7f9b08069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:17.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.596+0000 7f9b22bd3640 1 --2- 172.21.15.116:0/4274217765 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9b080678c0 0x7f9b08069d80 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f9b180af000 tx=0x7f9b180a7000 comp rx=0 tx=0).stop 2024-04-25T00:18:17.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.596+0000 7f9b22bd3640 1 -- 172.21.15.116:0/4274217765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b1c0c6bb0 msgr2=0x7f9b1c153dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:17.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.596+0000 7f9b22bd3640 1 --2- 172.21.15.116:0/4274217765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b1c0c6bb0 0x7f9b1c153dc0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f9b0c00e780 tx=0x7f9b0c00ec40 comp rx=0 tx=0).stop 2024-04-25T00:18:17.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.596+0000 7f9b22bd3640 1 -- 172.21.15.116:0/4274217765 shutdown_connections 2024-04-25T00:18:17.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.596+0000 7f9b22bd3640 1 --2- 172.21.15.116:0/4274217765 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9b080678c0 0x7f9b08069d80 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:17.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.596+0000 7f9b22bd3640 1 --2- 172.21.15.116:0/4274217765 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9b1c154c80 0x7f9b1c139410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:17.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.596+0000 7f9b22bd3640 1 --2- 172.21.15.116:0/4274217765 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9b1c0c6bb0 0x7f9b1c153dc0 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:17.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.596+0000 7f9b22bd3640 1 --2- 172.21.15.116:0/4274217765 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9b040039f0 0x7f9b1c0c52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:17.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.596+0000 7f9b22bd3640 1 -- 172.21.15.116:0/4274217765 >> 172.21.15.116:0/4274217765 conn(0x7f9b1c0c3b00 msgr2=0x7f9b1c1415e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:17.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.596+0000 7f9b22bd3640 1 -- 172.21.15.116:0/4274217765 shutdown_connections 2024-04-25T00:18:17.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:17.596+0000 7f9b22bd3640 1 -- 172.21.15.116:0/4274217765 wait complete. 2024-04-25T00:18:17.614 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:18:18.598 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:18:18.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.724+0000 7f3b0402a640 1 Processor -- start 2024-04-25T00:18:18.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.724+0000 7f3b0402a640 1 -- start start 2024-04-25T00:18:18.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.724+0000 7f3b0402a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3afc155da0 0x7f3afc156250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:18.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.724+0000 7f3b0402a640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3afc0718f0 con 0x7f3afc0c53b0 2024-04-25T00:18:18.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.724+0000 7f3b0402a640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3afc0c23f0 con 0x7f3afc0c8740 2024-04-25T00:18:18.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.724+0000 7f3b0402a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f3afc158f20 con 0x7f3afc155da0 2024-04-25T00:18:18.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.724+0000 7f3b015c5640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f3afc0c53b0 0x7f3afc152d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:56690/0 (socket says 172.21.15.116:56690) 2024-04-25T00:18:18.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.724+0000 7f3b015c5640 1 -- 172.21.15.116:0/219319835 learned_addr learned my addr 172.21.15.116:0/219319835 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:18:18.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.724+0000 7f3b025c7640 1 --2- 172.21.15.116:0/219319835 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3afc155da0 0x7f3afc156250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:18.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.724+0000 7f3b00dc4640 1 -- 172.21.15.116:0/219319835 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 454612844 0 0) 0x7f3af8001120 con 0x7f3afc0c53b0 2024-04-25T00:18:18.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.724+0000 7f3b00dc4640 1 -- 172.21.15.116:0/219319835 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ae8003530 con 0x7f3afc0c53b0 2024-04-25T00:18:18.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.724+0000 7f3b00dc4640 1 -- 172.21.15.116:0/219319835 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3780562193 0 0) 0x7f3ae8003530 con 0x7f3afc0c53b0 2024-04-25T00:18:18.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.724+0000 7f3b00dc4640 1 -- 172.21.15.116:0/219319835 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3afc155da0 msgr2=0x7f3afc156250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:18.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.724+0000 7f3b00dc4640 1 --2- 172.21.15.116:0/219319835 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3afc155da0 0x7f3afc156250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:18.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.724+0000 7f3b00dc4640 1 -- 172.21.15.116:0/219319835 >> v1:172.21.15.174:6789/0 conn(0x7f3afc0c8740 legacy=0x7f3afc0711e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:18.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.724+0000 7f3b00dc4640 1 -- 172.21.15.116:0/219319835 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3afc0c2690 con 0x7f3afc0c53b0 2024-04-25T00:18:18.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.724+0000 7f3b00dc4640 1 -- 172.21.15.116:0/219319835 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f3af8001850 con 0x7f3afc0c53b0 2024-04-25T00:18:18.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.724+0000 7f3b00dc4640 1 -- 172.21.15.116:0/219319835 >> v1:172.21.15.116:6789/0 conn(0x7f3afc0c53b0 legacy=0x7f3afc152d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:18.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.724+0000 7f3b00dc4640 1 --2- 172.21.15.116:0/219319835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ae80039f0 0x7f3ae8003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:18.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.724+0000 7f3b00dc4640 1 --2- 172.21.15.116:0/219319835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ae8004810 0x7f3ae8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:18.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.724+0000 7f3b00dc4640 1 --2- 172.21.15.116:0/219319835 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3afc155da0 0x7f3ae8005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:18.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.724+0000 7f3b00dc4640 1 -- 172.21.15.116:0/219319835 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3ae800bb00 con 0x7f3ae80039f0 2024-04-25T00:18:18.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.724+0000 7f3b00dc4640 1 -- 172.21.15.116:0/219319835 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3ae800bcb0 con 0x7f3ae8004810 2024-04-25T00:18:18.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.724+0000 7f3b00dc4640 1 -- 172.21.15.116:0/219319835 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f3ae800beb0 con 0x7f3afc155da0 2024-04-25T00:18:18.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.724+0000 7f3b01dc6640 1 --2- 172.21.15.116:0/219319835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ae80039f0 0x7f3ae8003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:18.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.724+0000 7f3b015c5640 1 --2- 172.21.15.116:0/219319835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ae8004810 0x7f3ae8004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:18.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.724+0000 7f3b01dc6640 1 -- 172.21.15.116:0/219319835 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3afc155da0 msgr2=0x7f3ae8005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:18.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.724+0000 7f3b025c7640 1 --2- 172.21.15.116:0/219319835 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3afc155da0 0x7f3ae8005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:18.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.724+0000 7f3b01dc6640 1 --2- 172.21.15.116:0/219319835 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3afc155da0 0x7f3ae8005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:18.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.724+0000 7f3b01dc6640 1 -- 172.21.15.116:0/219319835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ae8004810 msgr2=0x7f3ae8004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:18.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.724+0000 7f3b01dc6640 1 --2- 172.21.15.116:0/219319835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ae8004810 0x7f3ae8004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:18.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.724+0000 7f3b01dc6640 1 -- 172.21.15.116:0/219319835 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ae8003530 con 0x7f3ae80039f0 2024-04-25T00:18:18.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.724+0000 7f3b015c5640 1 --2- 172.21.15.116:0/219319835 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ae8004810 0x7f3ae8004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:18:18.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.724+0000 7f3b025c7640 1 --2- 172.21.15.116:0/219319835 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3afc155da0 0x7f3ae8005210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:18:18.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.728+0000 7f3b01dc6640 1 --2- 172.21.15.116:0/219319835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ae80039f0 0x7f3ae8003e10 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f3aec001780 tx=0x7f3aec01c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32595a8bc72a876a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:18.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.728+0000 7f3b00dc4640 1 -- 172.21.15.116:0/219319835 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aec0013a0 con 0x7f3ae80039f0 2024-04-25T00:18:18.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.728+0000 7f3b00dc4640 1 -- 172.21.15.116:0/219319835 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3aec0052f0 con 0x7f3ae80039f0 2024-04-25T00:18:18.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.728+0000 7f3b00dc4640 1 -- 172.21.15.116:0/219319835 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aec003840 con 0x7f3ae80039f0 2024-04-25T00:18:18.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.728+0000 7f3b0402a640 1 -- 172.21.15.116:0/219319835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ae80039f0 msgr2=0x7f3ae8003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:18.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.728+0000 7f3b0402a640 1 --2- 172.21.15.116:0/219319835 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ae80039f0 0x7f3ae8003e10 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f3aec001780 tx=0x7f3aec01c420 comp rx=0 tx=0).stop 2024-04-25T00:18:18.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.728+0000 7f3b025c7640 1 -- 172.21.15.116:0/219319835 reap_dead start 2024-04-25T00:18:18.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.728+0000 7f3b0402a640 1 -- 172.21.15.116:0/219319835 shutdown_connections 2024-04-25T00:18:18.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.728+0000 7f3b0402a640 1 -- 172.21.15.116:0/219319835 >> 172.21.15.116:0/219319835 conn(0x7f3afc06f0c0 msgr2=0x7f3afc06f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:18.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.728+0000 7f3b0402a640 1 -- 172.21.15.116:0/219319835 shutdown_connections 2024-04-25T00:18:18.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.728+0000 7f3b0402a640 1 -- 172.21.15.116:0/219319835 wait complete. 2024-04-25T00:18:18.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.728+0000 7f3b0402a640 1 Processor -- start 2024-04-25T00:18:18.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.728+0000 7f3b0402a640 1 -- start start 2024-04-25T00:18:18.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.728+0000 7f3b0402a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3ae80039f0 0x7f3afc1f1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:18.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.728+0000 7f3b0402a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3afc158a90 0x7f3afc06d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:18.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.728+0000 7f3b0402a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3afc14e460 0x7f3afc14e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:18.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.728+0000 7f3b0402a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3afc0c8ca0 con 0x7f3afc14e460 2024-04-25T00:18:18.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.728+0000 7f3b0402a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3afc0c8b20 con 0x7f3afc158a90 2024-04-25T00:18:18.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.728+0000 7f3b0402a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f3afc1612c0 con 0x7f3ae80039f0 2024-04-25T00:18:18.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.728+0000 7f3b025c7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3afc14e460 0x7f3afc14e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:18.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.728+0000 7f3b025c7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3afc14e460 0x7f3afc14e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49290/0 (socket says 172.21.15.116:49290) 2024-04-25T00:18:18.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.728+0000 7f3b025c7640 1 -- 172.21.15.116:0/691748642 learned_addr learned my addr 172.21.15.116:0/691748642 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:18:18.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.728+0000 7f3b01dc6640 1 --2- 172.21.15.116:0/691748642 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3ae80039f0 0x7f3afc1f1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:18.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.728+0000 7f3b015c5640 1 --2- 172.21.15.116:0/691748642 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3afc158a90 0x7f3afc06d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:18.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.728+0000 7f3b025c7640 1 -- 172.21.15.116:0/691748642 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3ae80039f0 msgr2=0x7f3afc1f1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:18.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.728+0000 7f3b025c7640 1 --2- 172.21.15.116:0/691748642 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3ae80039f0 0x7f3afc1f1fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:18.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.728+0000 7f3b025c7640 1 -- 172.21.15.116:0/691748642 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3afc158a90 msgr2=0x7f3afc06d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:18.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.728+0000 7f3b025c7640 1 --2- 172.21.15.116:0/691748642 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3afc158a90 0x7f3afc06d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:18.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.728+0000 7f3b025c7640 1 -- 172.21.15.116:0/691748642 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3afc15a3a0 con 0x7f3afc14e460 2024-04-25T00:18:18.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.728+0000 7f3b025c7640 1 --2- 172.21.15.116:0/691748642 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3afc14e460 0x7f3afc14e8e0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f3afc0c7e70 tx=0x7f3af400f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:18.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.732+0000 7f3af27fc640 1 -- 172.21.15.116:0/691748642 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3af401c070 con 0x7f3afc14e460 2024-04-25T00:18:18.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.732+0000 7f3af27fc640 1 -- 172.21.15.116:0/691748642 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3af400fe90 con 0x7f3afc14e460 2024-04-25T00:18:18.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.732+0000 7f3b0402a640 1 -- 172.21.15.116:0/691748642 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3afc157f80 con 0x7f3afc14e460 2024-04-25T00:18:18.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.732+0000 7f3af27fc640 1 -- 172.21.15.116:0/691748642 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3af4017600 con 0x7f3afc14e460 2024-04-25T00:18:18.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.732+0000 7f3b0402a640 1 -- 172.21.15.116:0/691748642 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3afc158540 con 0x7f3afc14e460 2024-04-25T00:18:18.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.732+0000 7f3b0402a640 1 -- 172.21.15.116:0/691748642 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3afc068a60 con 0x7f3afc14e460 2024-04-25T00:18:18.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.732+0000 7f3af27fc640 1 -- 172.21.15.116:0/691748642 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3af401f070 con 0x7f3afc14e460 2024-04-25T00:18:18.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.732+0000 7f3af27fc640 1 --2- 172.21.15.116:0/691748642 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3ae00678c0 0x7f3ae0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:18.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.732+0000 7f3af27fc640 1 -- 172.21.15.116:0/691748642 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 8816+0+0 (secure 0 0 0) 0x7f3af4086b50 con 0x7f3afc14e460 2024-04-25T00:18:18.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.732+0000 7f3b01dc6640 1 --2- 172.21.15.116:0/691748642 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3ae00678c0 0x7f3ae0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:18.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.732+0000 7f3b01dc6640 1 --2- 172.21.15.116:0/691748642 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3ae00678c0 0x7f3ae0069d80 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f3aec001750 tx=0x7f3aec0011a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:18.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.736+0000 7f3af27fc640 1 -- 172.21.15.116:0/691748642 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f3afc15a3a0 con 0x7f3afc14e460 2024-04-25T00:18:18.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.928+0000 7f3b0402a640 1 -- 172.21.15.116:0/691748642 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3afc06e180 con 0x7f3afc14e460 2024-04-25T00:18:18.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.932+0000 7f3af27fc640 1 -- 172.21.15.116:0/691748642 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) ==== 95+0+6509 (secure 0 0 0) 0x7f3af4012070 con 0x7f3afc14e460 2024-04-25T00:18:18.933 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:18:18.933 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":137,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7362,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/1655959433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1655959433},{"type":"v1","addr":"172.21.15.116:6833","nonce":1655959433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:17:38.902992+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6343":{"gid":6343,"name":"d","rank":0,"incarnation":133,"state":"up:active","state_seq":4,"addr":"172.21.15.174:6837/1281466397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1281466397},{"type":"v1","addr":"172.21.15.174:6837","nonce":1281466397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6343,"qdb_cluster":[6343]},"id":1},{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:17:28.837517+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7164},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7164":{"gid":7164,"name":"c","rank":0,"incarnation":128,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6835/3336339376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3336339376},{"type":"v1","addr":"172.21.15.116:6835","nonce":3336339376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7164,"qdb_cluster":[7164]},"id":2}]} 2024-04-25T00:18:18.933 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 137 2024-04-25T00:18:18.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.932+0000 7f3b0402a640 1 -- 172.21.15.116:0/691748642 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3ae00678c0 msgr2=0x7f3ae0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:18.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.932+0000 7f3b0402a640 1 --2- 172.21.15.116:0/691748642 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3ae00678c0 0x7f3ae0069d80 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f3aec001750 tx=0x7f3aec0011a0 comp rx=0 tx=0).stop 2024-04-25T00:18:18.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.932+0000 7f3b0402a640 1 -- 172.21.15.116:0/691748642 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3afc14e460 msgr2=0x7f3afc14e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:18.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.932+0000 7f3b0402a640 1 --2- 172.21.15.116:0/691748642 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3afc14e460 0x7f3afc14e8e0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f3afc0c7e70 tx=0x7f3af400f4a0 comp rx=0 tx=0).stop 2024-04-25T00:18:18.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.932+0000 7f3b0402a640 1 -- 172.21.15.116:0/691748642 shutdown_connections 2024-04-25T00:18:18.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.932+0000 7f3b0402a640 1 --2- 172.21.15.116:0/691748642 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3ae00678c0 0x7f3ae0069d80 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:18.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.932+0000 7f3b0402a640 1 --2- 172.21.15.116:0/691748642 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3afc14e460 0x7f3afc14e8e0 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:18.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.932+0000 7f3b0402a640 1 --2- 172.21.15.116:0/691748642 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3afc158a90 0x7f3afc06d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:18.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.932+0000 7f3b0402a640 1 --2- 172.21.15.116:0/691748642 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3ae80039f0 0x7f3afc1f1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:18.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.932+0000 7f3b0402a640 1 -- 172.21.15.116:0/691748642 >> 172.21.15.116:0/691748642 conn(0x7f3afc06f0c0 msgr2=0x7f3afc06f4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:18.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.932+0000 7f3b0402a640 1 -- 172.21.15.116:0/691748642 shutdown_connections 2024-04-25T00:18:18.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:18.932+0000 7f3b0402a640 1 -- 172.21.15.116:0/691748642 wait complete. 2024-04-25T00:18:18.951 INFO:tasks.mds_thrash.fs.[b]:kill mds.c (rank=0) 2024-04-25T00:18:18.951 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-25T00:18:18.951 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:18:18.964 INFO:tasks.ceph.mds.c:Stopped 2024-04-25T00:18:18.964 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:18:19.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.132+0000 7f032d31d640 1 Processor -- start 2024-04-25T00:18:19.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.132+0000 7f032d31d640 1 -- start start 2024-04-25T00:18:19.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.132+0000 7f032d31d640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f0328148890 0x7f0328072590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:19.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.132+0000 7f032d31d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f032806e890 0x7f0328071980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:19.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.132+0000 7f032d31d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0328152c10 con 0x7f0328070dd0 2024-04-25T00:18:19.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.132+0000 7f032d31d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f0328151e50 con 0x7f032806e890 2024-04-25T00:18:19.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.132+0000 7f032d31d640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f0328152130 con 0x7f0328148890 2024-04-25T00:18:19.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.132+0000 7f03277fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f032806e890 0x7f0328071980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:19.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.132+0000 7f031ffff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f0328070dd0 0x7f0328071270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:56706/0 (socket says 172.21.15.116:56706) 2024-04-25T00:18:19.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.132+0000 7f031ffff640 1 -- 172.21.15.116:0/2272075724 learned_addr learned my addr 172.21.15.116:0/2272075724 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:18:19.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.132+0000 7f0326ffd640 1 --2- 172.21.15.116:0/2272075724 >> v2:172.21.15.174:3300/0 conn(0x7f0328148890 0x7f0328072590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:19.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.132+0000 7f03277fe640 1 -- 172.21.15.116:0/2272075724 >> v2:172.21.15.174:3300/0 conn(0x7f0328148890 msgr2=0x7f0328072590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:19.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.132+0000 7f03277fe640 1 --2- 172.21.15.116:0/2272075724 >> v2:172.21.15.174:3300/0 conn(0x7f0328148890 0x7f0328072590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:19.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.132+0000 7f03277fe640 1 -- 172.21.15.116:0/2272075724 >> v1:172.21.15.116:6789/0 conn(0x7f0328070dd0 legacy=0x7f0328071270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:19.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.132+0000 7f03267fc640 1 -- 172.21.15.116:0/2272075724 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1740664654 0 0) 0x7f0328152c10 con 0x7f0328070dd0 2024-04-25T00:18:19.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.132+0000 7f03277fe640 1 -- 172.21.15.116:0/2272075724 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0328152410 con 0x7f032806e890 2024-04-25T00:18:19.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.132+0000 7f0326ffd640 1 --2- 172.21.15.116:0/2272075724 >> v2:172.21.15.174:3300/0 conn(0x7f0328148890 0x7f0328072590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:18:19.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.136+0000 7f03277fe640 1 --2- 172.21.15.116:0/2272075724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f032806e890 0x7f0328071980 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f0320009d90 tx=0x7f032001eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9076eeff9eb03415 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:19.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.136+0000 7f03267fc640 1 -- 172.21.15.116:0/2272075724 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03200b4070 con 0x7f032806e890 2024-04-25T00:18:19.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.136+0000 7f03267fc640 1 -- 172.21.15.116:0/2272075724 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03200a7a20 con 0x7f032806e890 2024-04-25T00:18:19.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.136+0000 7f03267fc640 1 -- 172.21.15.116:0/2272075724 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03200a7df0 con 0x7f032806e890 2024-04-25T00:18:19.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.136+0000 7f032d31d640 1 -- 172.21.15.116:0/2272075724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f032806e890 msgr2=0x7f0328071980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:19.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.136+0000 7f032d31d640 1 --2- 172.21.15.116:0/2272075724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f032806e890 0x7f0328071980 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f0320009d90 tx=0x7f032001eb50 comp rx=0 tx=0).stop 2024-04-25T00:18:19.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.136+0000 7f032d31d640 1 -- 172.21.15.116:0/2272075724 shutdown_connections 2024-04-25T00:18:19.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.136+0000 7f032d31d640 1 --2- 172.21.15.116:0/2272075724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f032806e890 0x7f0328071980 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:19.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.136+0000 7f032d31d640 1 --2- 172.21.15.116:0/2272075724 >> v2:172.21.15.174:3300/0 conn(0x7f0328148890 0x7f0328072590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:19.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.136+0000 7f032d31d640 1 -- 172.21.15.116:0/2272075724 >> 172.21.15.116:0/2272075724 conn(0x7f03280c3b00 msgr2=0x7f03280c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:19.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.136+0000 7f032d31d640 1 -- 172.21.15.116:0/2272075724 shutdown_connections 2024-04-25T00:18:19.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.136+0000 7f032d31d640 1 -- 172.21.15.116:0/2272075724 wait complete. 2024-04-25T00:18:19.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.136+0000 7f032d31d640 1 Processor -- start 2024-04-25T00:18:19.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.136+0000 7f032d31d640 1 -- start start 2024-04-25T00:18:19.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.136+0000 7f032d31d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f032806e890 0x7f032814eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:19.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.136+0000 7f032d31d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0328070dd0 0x7f0328140b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:19.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.136+0000 7f032d31d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0328148890 0x7f0328141d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:19.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.136+0000 7f032d31d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f03280ab920 con 0x7f032806e890 2024-04-25T00:18:19.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.136+0000 7f032d31d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f03280ab7a0 con 0x7f0328148890 2024-04-25T00:18:19.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.136+0000 7f032d31d640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f03280abaa0 con 0x7f0328070dd0 2024-04-25T00:18:19.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.136+0000 7f0326ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f032806e890 0x7f032814eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:19.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.136+0000 7f0326ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f032806e890 0x7f032814eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49314/0 (socket says 172.21.15.116:49314) 2024-04-25T00:18:19.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.136+0000 7f0326ffd640 1 -- 172.21.15.116:0/2042045341 learned_addr learned my addr 172.21.15.116:0/2042045341 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:18:19.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.136+0000 7f031ffff640 1 --2- 172.21.15.116:0/2042045341 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0328070dd0 0x7f0328140b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:19.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.136+0000 7f03277fe640 1 --2- 172.21.15.116:0/2042045341 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0328148890 0x7f0328141d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:19.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.136+0000 7f0326ffd640 1 -- 172.21.15.116:0/2042045341 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0328070dd0 msgr2=0x7f0328140b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:19.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.136+0000 7f0326ffd640 1 --2- 172.21.15.116:0/2042045341 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0328070dd0 0x7f0328140b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:19.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.136+0000 7f0326ffd640 1 -- 172.21.15.116:0/2042045341 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0328148890 msgr2=0x7f0328141d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:19.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.136+0000 7f0326ffd640 1 --2- 172.21.15.116:0/2042045341 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0328148890 0x7f0328141d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:19.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.136+0000 7f0326ffd640 1 -- 172.21.15.116:0/2042045341 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0328146140 con 0x7f032806e890 2024-04-25T00:18:19.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.136+0000 7f0326ffd640 1 --2- 172.21.15.116:0/2042045341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f032806e890 0x7f032814eda0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f031800c7f0 tx=0x7f031800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:19.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.136+0000 7f031f7fe640 1 -- 172.21.15.116:0/2042045341 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0318015070 con 0x7f032806e890 2024-04-25T00:18:19.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.136+0000 7f031f7fe640 1 -- 172.21.15.116:0/2042045341 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0318007820 con 0x7f032806e890 2024-04-25T00:18:19.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.136+0000 7f032d31d640 1 -- 172.21.15.116:0/2042045341 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0328146420 con 0x7f032806e890 2024-04-25T00:18:19.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.136+0000 7f031f7fe640 1 -- 172.21.15.116:0/2042045341 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0318007c10 con 0x7f032806e890 2024-04-25T00:18:19.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.136+0000 7f032d31d640 1 -- 172.21.15.116:0/2042045341 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03280c4e60 con 0x7f032806e890 2024-04-25T00:18:19.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.140+0000 7f032d31d640 1 -- 172.21.15.116:0/2042045341 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0328068350 con 0x7f032806e890 2024-04-25T00:18:19.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.140+0000 7f031f7fe640 1 -- 172.21.15.116:0/2042045341 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f0318018070 con 0x7f032806e890 2024-04-25T00:18:19.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.140+0000 7f031f7fe640 1 --2- 172.21.15.116:0/2042045341 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f03080678f0 0x7f0308069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:19.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.140+0000 7f031f7fe640 1 -- 172.21.15.116:0/2042045341 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 8816+0+0 (secure 0 0 0) 0x7f03180869b0 con 0x7f032806e890 2024-04-25T00:18:19.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.140+0000 7f031ffff640 1 --2- 172.21.15.116:0/2042045341 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f03080678f0 0x7f0308069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:19.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.140+0000 7f031ffff640 1 --2- 172.21.15.116:0/2042045341 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f03080678f0 0x7f0308069db0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f0328072210 tx=0x7f0310007b60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:19.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.144+0000 7f031f7fe640 1 -- 172.21.15.116:0/2042045341 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f03180568d0 con 0x7f032806e890 2024-04-25T00:18:19.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.324+0000 7f032d31d640 1 -- 172.21.15.116:0/2042045341 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f032806e670 con 0x7f032806e890 2024-04-25T00:18:19.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.324+0000 7f031f7fe640 1 -- 172.21.15.116:0/2042045341 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) ==== 95+0+6509 (secure 0 0 0) 0x7f03180561d0 con 0x7f032806e890 2024-04-25T00:18:19.326 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:18:19.326 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":137,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7362,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/1655959433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1655959433},{"type":"v1","addr":"172.21.15.116:6833","nonce":1655959433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:17:38.902992+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6343":{"gid":6343,"name":"d","rank":0,"incarnation":133,"state":"up:active","state_seq":4,"addr":"172.21.15.174:6837/1281466397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1281466397},{"type":"v1","addr":"172.21.15.174:6837","nonce":1281466397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6343,"qdb_cluster":[6343]},"id":1},{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:17:28.837517+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7164},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7164":{"gid":7164,"name":"c","rank":0,"incarnation":128,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6835/3336339376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3336339376},{"type":"v1","addr":"172.21.15.116:6835","nonce":3336339376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7164,"qdb_cluster":[7164]},"id":2}]} 2024-04-25T00:18:19.326 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 137 2024-04-25T00:18:19.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.328+0000 7f032d31d640 1 -- 172.21.15.116:0/2042045341 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f03080678f0 msgr2=0x7f0308069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:19.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.328+0000 7f032d31d640 1 --2- 172.21.15.116:0/2042045341 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f03080678f0 0x7f0308069db0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f0328072210 tx=0x7f0310007b60 comp rx=0 tx=0).stop 2024-04-25T00:18:19.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.328+0000 7f032d31d640 1 -- 172.21.15.116:0/2042045341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f032806e890 msgr2=0x7f032814eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:19.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.328+0000 7f032d31d640 1 --2- 172.21.15.116:0/2042045341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f032806e890 0x7f032814eda0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f031800c7f0 tx=0x7f031800ccb0 comp rx=0 tx=0).stop 2024-04-25T00:18:19.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.328+0000 7f032d31d640 1 -- 172.21.15.116:0/2042045341 shutdown_connections 2024-04-25T00:18:19.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.328+0000 7f032d31d640 1 --2- 172.21.15.116:0/2042045341 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f03080678f0 0x7f0308069db0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:19.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.328+0000 7f032d31d640 1 --2- 172.21.15.116:0/2042045341 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0328148890 0x7f0328141d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:19.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.328+0000 7f032d31d640 1 --2- 172.21.15.116:0/2042045341 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0328070dd0 0x7f0328140b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:19.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.328+0000 7f032d31d640 1 --2- 172.21.15.116:0/2042045341 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f032806e890 0x7f032814eda0 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:19.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.328+0000 7f032d31d640 1 -- 172.21.15.116:0/2042045341 >> 172.21.15.116:0/2042045341 conn(0x7f03280c3b00 msgr2=0x7f03280abe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:19.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.328+0000 7f032d31d640 1 -- 172.21.15.116:0/2042045341 shutdown_connections 2024-04-25T00:18:19.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.328+0000 7f032d31d640 1 -- 172.21.15.116:0/2042045341 wait complete. 2024-04-25T00:18:19.344 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:18:19.615 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:18:19.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.740+0000 7f25bf46f640 1 Processor -- start 2024-04-25T00:18:19.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.740+0000 7f25bf46f640 1 -- start start 2024-04-25T00:18:19.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.744+0000 7f25bf46f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f25b80a94a0 0x7f25b81538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:19.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.744+0000 7f25bf46f640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f25b80a99f0 0x7f25b8153e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:19.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.744+0000 7f25bf46f640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25b80c3910 con 0x7f25b8150d20 2024-04-25T00:18:19.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.744+0000 7f25bf46f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f25b80c16e0 con 0x7f25b80a94a0 2024-04-25T00:18:19.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.744+0000 7f25bf46f640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f25b80c1400 con 0x7f25b80a99f0 2024-04-25T00:18:19.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.744+0000 7f25bca0a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f25b80a94a0 0x7f25b81538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:19.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.744+0000 7f25bca0a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f25b80a94a0 0x7f25b81538e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49328/0 (socket says 172.21.15.116:49328) 2024-04-25T00:18:19.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.744+0000 7f25bca0a640 1 -- 172.21.15.116:0/24569509 learned_addr learned my addr 172.21.15.116:0/24569509 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:18:19.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.744+0000 7f25bda0c640 1 --2- 172.21.15.116:0/24569509 >> v2:172.21.15.174:3300/0 conn(0x7f25b80a99f0 0x7f25b8153e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:19.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.744+0000 7f25bca0a640 1 -- 172.21.15.116:0/24569509 >> v2:172.21.15.174:3300/0 conn(0x7f25b80a99f0 msgr2=0x7f25b8153e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:19.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.744+0000 7f25bca0a640 1 --2- 172.21.15.116:0/24569509 >> v2:172.21.15.174:3300/0 conn(0x7f25b80a99f0 0x7f25b8153e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:19.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.744+0000 7f25bca0a640 1 -- 172.21.15.116:0/24569509 >> v1:172.21.15.174:6789/0 conn(0x7f25b8150d20 legacy=0x7f25b80c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:19.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.744+0000 7f25bca0a640 1 -- 172.21.15.116:0/24569509 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25b806e660 con 0x7f25b80a94a0 2024-04-25T00:18:19.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.744+0000 7f25bca0a640 1 --2- 172.21.15.116:0/24569509 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25b80a94a0 0x7f25b81538e0 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f25b8157b40 tx=0x7f25a801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5c2ad05a8d3a250 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:19.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.748+0000 7f25affff640 1 -- 172.21.15.116:0/24569509 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25a80ab430 con 0x7f25b80a94a0 2024-04-25T00:18:19.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.748+0000 7f25affff640 1 -- 172.21.15.116:0/24569509 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25a80b0070 con 0x7f25b80a94a0 2024-04-25T00:18:19.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.748+0000 7f25affff640 1 -- 172.21.15.116:0/24569509 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25a80ab7f0 con 0x7f25b80a94a0 2024-04-25T00:18:19.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.748+0000 7f25bf46f640 1 -- 172.21.15.116:0/24569509 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25b80a94a0 msgr2=0x7f25b81538e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:19.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.748+0000 7f25bf46f640 1 --2- 172.21.15.116:0/24569509 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25b80a94a0 0x7f25b81538e0 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f25b8157b40 tx=0x7f25a801c520 comp rx=0 tx=0).stop 2024-04-25T00:18:19.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.748+0000 7f25bf46f640 1 -- 172.21.15.116:0/24569509 shutdown_connections 2024-04-25T00:18:19.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.748+0000 7f25bf46f640 1 --2- 172.21.15.116:0/24569509 >> v2:172.21.15.174:3300/0 conn(0x7f25b80a99f0 0x7f25b8153e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:19.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.748+0000 7f25bf46f640 1 --2- 172.21.15.116:0/24569509 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25b80a94a0 0x7f25b81538e0 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:19.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.748+0000 7f25bf46f640 1 -- 172.21.15.116:0/24569509 >> 172.21.15.116:0/24569509 conn(0x7f25b80c3b00 msgr2=0x7f25b80c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:19.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.748+0000 7f25bf46f640 1 -- 172.21.15.116:0/24569509 shutdown_connections 2024-04-25T00:18:19.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.748+0000 7f25bf46f640 1 -- 172.21.15.116:0/24569509 wait complete. 2024-04-25T00:18:19.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.748+0000 7f25bf46f640 1 Processor -- start 2024-04-25T00:18:19.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.748+0000 7f25bf46f640 1 -- start start 2024-04-25T00:18:19.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.748+0000 7f25bf46f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25b80a94a0 0x7f25b81f15b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:19.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.748+0000 7f25bf46f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25b80a99f0 0x7f25b81f1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:19.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.748+0000 7f25bf46f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f25b8150d20 0x7f25b814c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:19.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.748+0000 7f25bf46f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f25b8154360 con 0x7f25b80a94a0 2024-04-25T00:18:19.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.748+0000 7f25bf46f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f25b8072d50 con 0x7f25b80a99f0 2024-04-25T00:18:19.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.748+0000 7f25bf46f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f25b81544e0 con 0x7f25b8150d20 2024-04-25T00:18:19.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.748+0000 7f25bd20b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25b80a94a0 0x7f25b81f15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:19.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.748+0000 7f25bd20b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25b80a94a0 0x7f25b81f15b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49344/0 (socket says 172.21.15.116:49344) 2024-04-25T00:18:19.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.748+0000 7f25bd20b640 1 -- 172.21.15.116:0/2939838444 learned_addr learned my addr 172.21.15.116:0/2939838444 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:18:19.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.748+0000 7f25bca0a640 1 --2- 172.21.15.116:0/2939838444 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25b80a99f0 0x7f25b81f1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:19.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.748+0000 7f25bda0c640 1 --2- 172.21.15.116:0/2939838444 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f25b8150d20 0x7f25b814c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:19.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.748+0000 7f25bd20b640 1 -- 172.21.15.116:0/2939838444 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f25b8150d20 msgr2=0x7f25b814c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:19.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.748+0000 7f25bd20b640 1 --2- 172.21.15.116:0/2939838444 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f25b8150d20 0x7f25b814c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:19.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.748+0000 7f25bd20b640 1 -- 172.21.15.116:0/2939838444 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25b80a99f0 msgr2=0x7f25b81f1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:19.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.748+0000 7f25bd20b640 1 --2- 172.21.15.116:0/2939838444 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25b80a99f0 0x7f25b81f1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:19.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.748+0000 7f25bd20b640 1 -- 172.21.15.116:0/2939838444 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25b814b070 con 0x7f25b80a94a0 2024-04-25T00:18:19.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.748+0000 7f25bca0a640 1 --2- 172.21.15.116:0/2939838444 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25b80a99f0 0x7f25b81f1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:18:19.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.748+0000 7f25bd20b640 1 --2- 172.21.15.116:0/2939838444 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25b80a94a0 0x7f25b81f15b0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f25b400c530 tx=0x7f25b400c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:19.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.748+0000 7f25adffb640 1 -- 172.21.15.116:0/2939838444 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b4016020 con 0x7f25b80a94a0 2024-04-25T00:18:19.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.748+0000 7f25adffb640 1 -- 172.21.15.116:0/2939838444 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25b4005220 con 0x7f25b80a94a0 2024-04-25T00:18:19.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.748+0000 7f25bf46f640 1 -- 172.21.15.116:0/2939838444 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25b81f2030 con 0x7f25b80a94a0 2024-04-25T00:18:19.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.748+0000 7f25adffb640 1 -- 172.21.15.116:0/2939838444 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b4007030 con 0x7f25b80a94a0 2024-04-25T00:18:19.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.748+0000 7f25bf46f640 1 -- 172.21.15.116:0/2939838444 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25b814b310 con 0x7f25b80a94a0 2024-04-25T00:18:19.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.752+0000 7f25bf46f640 1 -- 172.21.15.116:0/2939838444 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25b8068350 con 0x7f25b80a94a0 2024-04-25T00:18:19.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.752+0000 7f25adffb640 1 -- 172.21.15.116:0/2939838444 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f25b40071d0 con 0x7f25b80a94a0 2024-04-25T00:18:19.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.752+0000 7f25adffb640 1 --2- 172.21.15.116:0/2939838444 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f259c0678c0 0x7f259c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:19.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.752+0000 7f25adffb640 1 -- 172.21.15.116:0/2939838444 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 8816+0+0 (secure 0 0 0) 0x7f25b4086240 con 0x7f25b80a94a0 2024-04-25T00:18:19.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.752+0000 7f25bca0a640 1 --2- 172.21.15.116:0/2939838444 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f259c0678c0 0x7f259c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:19.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.752+0000 7f25bca0a640 1 --2- 172.21.15.116:0/2939838444 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f259c0678c0 0x7f259c069d80 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f25a80af000 tx=0x7f25a8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:19.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.756+0000 7f25adffb640 1 -- 172.21.15.116:0/2939838444 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f25b4020020 con 0x7f25b80a94a0 2024-04-25T00:18:19.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.952+0000 7f25bf46f640 1 -- 172.21.15.116:0/2939838444 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f25b80c34f0 con 0x7f25b80a94a0 2024-04-25T00:18:19.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.952+0000 7f25adffb640 1 -- 172.21.15.116:0/2939838444 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) ==== 95+0+6509 (secure 0 0 0) 0x7f25b40074c0 con 0x7f25b80a94a0 2024-04-25T00:18:19.954 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:18:19.955 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":137,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7362,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/1655959433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1655959433},{"type":"v1","addr":"172.21.15.116:6833","nonce":1655959433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":137}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:17:38.902992+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6343":{"gid":6343,"name":"d","rank":0,"incarnation":133,"state":"up:active","state_seq":4,"addr":"172.21.15.174:6837/1281466397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":1281466397},{"type":"v1","addr":"172.21.15.174:6837","nonce":1281466397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6343,"qdb_cluster":[6343]},"id":1},{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:17:28.837517+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7164},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7164":{"gid":7164,"name":"c","rank":0,"incarnation":128,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6835/3336339376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3336339376},{"type":"v1","addr":"172.21.15.116:6835","nonce":3336339376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7164,"qdb_cluster":[7164]},"id":2}]} 2024-04-25T00:18:19.955 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 137 2024-04-25T00:18:19.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.956+0000 7f25bf46f640 1 -- 172.21.15.116:0/2939838444 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f259c0678c0 msgr2=0x7f259c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:19.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.956+0000 7f25bf46f640 1 --2- 172.21.15.116:0/2939838444 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f259c0678c0 0x7f259c069d80 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f25a80af000 tx=0x7f25a8008040 comp rx=0 tx=0).stop 2024-04-25T00:18:19.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.956+0000 7f25bf46f640 1 -- 172.21.15.116:0/2939838444 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25b80a94a0 msgr2=0x7f25b81f15b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:19.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.956+0000 7f25bf46f640 1 --2- 172.21.15.116:0/2939838444 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25b80a94a0 0x7f25b81f15b0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f25b400c530 tx=0x7f25b400c9f0 comp rx=0 tx=0).stop 2024-04-25T00:18:19.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.956+0000 7f25bf46f640 1 -- 172.21.15.116:0/2939838444 shutdown_connections 2024-04-25T00:18:19.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.956+0000 7f25bf46f640 1 --2- 172.21.15.116:0/2939838444 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f259c0678c0 0x7f259c069d80 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:19.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.956+0000 7f25bf46f640 1 --2- 172.21.15.116:0/2939838444 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f25b8150d20 0x7f25b814c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:19.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.956+0000 7f25bf46f640 1 --2- 172.21.15.116:0/2939838444 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25b80a99f0 0x7f25b81f1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:19.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.956+0000 7f25bf46f640 1 --2- 172.21.15.116:0/2939838444 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25b80a94a0 0x7f25b81f15b0 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:19.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.956+0000 7f25bf46f640 1 -- 172.21.15.116:0/2939838444 >> 172.21.15.116:0/2939838444 conn(0x7f25b80c3b00 msgr2=0x7f25b81537e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:19.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.956+0000 7f25bf46f640 1 -- 172.21.15.116:0/2939838444 shutdown_connections 2024-04-25T00:18:19.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:19.956+0000 7f25bf46f640 1 -- 172.21.15.116:0/2939838444 wait complete. 2024-04-25T00:18:19.973 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:18:21.345 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:18:21.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.476+0000 7f74ebfbf640 1 Processor -- start 2024-04-25T00:18:21.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.476+0000 7f74ebfbf640 1 -- start start 2024-04-25T00:18:21.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.476+0000 7f74ebfbf640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f74e4148890 0x7f74e4072590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:21.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.476+0000 7f74ebfbf640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f74e406e890 0x7f74e4071980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:21.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.476+0000 7f74ebfbf640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74e4152c10 con 0x7f74e4070dd0 2024-04-25T00:18:21.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.476+0000 7f74ebfbf640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f74e4151e50 con 0x7f74e4148890 2024-04-25T00:18:21.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.476+0000 7f74ebfbf640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f74e4152130 con 0x7f74e406e890 2024-04-25T00:18:21.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.476+0000 7f74ea55c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f74e406e890 0x7f74e4071980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:21.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.476+0000 7f74e9d5b640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f74e4148890 0x7f74e4072590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:21.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.476+0000 7f74ea55c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f74e406e890 0x7f74e4071980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:58098/0 (socket says 172.21.15.116:58098) 2024-04-25T00:18:21.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.476+0000 7f74ea55c640 1 -- 172.21.15.116:0/4136708713 learned_addr learned my addr 172.21.15.116:0/4136708713 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:18:21.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.476+0000 7f74e9d5b640 1 -- 172.21.15.116:0/4136708713 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f74e406e890 msgr2=0x7f74e4071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:21.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.476+0000 7f74e9d5b640 1 --2- 172.21.15.116:0/4136708713 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f74e406e890 0x7f74e4071980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:21.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.476+0000 7f74e9d5b640 1 -- 172.21.15.116:0/4136708713 >> v1:172.21.15.174:6789/0 conn(0x7f74e4070dd0 legacy=0x7f74e4071270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:21.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.476+0000 7f74e8d59640 1 -- 172.21.15.116:0/4136708713 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1859476778 0 0) 0x7f74e4152c10 con 0x7f74e4070dd0 2024-04-25T00:18:21.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.476+0000 7f74e9d5b640 1 -- 172.21.15.116:0/4136708713 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74e4152410 con 0x7f74e4148890 2024-04-25T00:18:21.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.476+0000 7f74ea55c640 1 --2- 172.21.15.116:0/4136708713 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f74e406e890 0x7f74e4071980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:18:21.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.480+0000 7f74e9d5b640 1 --2- 172.21.15.116:0/4136708713 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f74e4148890 0x7f74e4072590 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f74e00097c0 tx=0x7f74e001c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3cb038e2380d1a86 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:21.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.480+0000 7f74e8d59640 1 -- 172.21.15.116:0/4136708713 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74e00b0070 con 0x7f74e4148890 2024-04-25T00:18:21.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.480+0000 7f74e8d59640 1 -- 172.21.15.116:0/4136708713 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74e00ab600 con 0x7f74e4148890 2024-04-25T00:18:21.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.480+0000 7f74e8d59640 1 -- 172.21.15.116:0/4136708713 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74e00ab9d0 con 0x7f74e4148890 2024-04-25T00:18:21.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.480+0000 7f74ebfbf640 1 -- 172.21.15.116:0/4136708713 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f74e4148890 msgr2=0x7f74e4072590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:21.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.480+0000 7f74ebfbf640 1 --2- 172.21.15.116:0/4136708713 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f74e4148890 0x7f74e4072590 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f74e00097c0 tx=0x7f74e001c4a0 comp rx=0 tx=0).stop 2024-04-25T00:18:21.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.480+0000 7f74ebfbf640 1 -- 172.21.15.116:0/4136708713 shutdown_connections 2024-04-25T00:18:21.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.480+0000 7f74ebfbf640 1 --2- 172.21.15.116:0/4136708713 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f74e406e890 0x7f74e4071980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:21.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.480+0000 7f74ebfbf640 1 --2- 172.21.15.116:0/4136708713 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f74e4148890 0x7f74e4072590 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:21.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.480+0000 7f74ebfbf640 1 -- 172.21.15.116:0/4136708713 >> 172.21.15.116:0/4136708713 conn(0x7f74e40c3b00 msgr2=0x7f74e40c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:21.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.480+0000 7f74ebfbf640 1 -- 172.21.15.116:0/4136708713 shutdown_connections 2024-04-25T00:18:21.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.480+0000 7f74ebfbf640 1 -- 172.21.15.116:0/4136708713 wait complete. 2024-04-25T00:18:21.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.480+0000 7f74ebfbf640 1 Processor -- start 2024-04-25T00:18:21.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.480+0000 7f74ebfbf640 1 -- start start 2024-04-25T00:18:21.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.480+0000 7f74ebfbf640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f74e406e890 0x7f74e40c6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:21.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.480+0000 7f74ebfbf640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f74e4070dd0 0x7f74e413b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:21.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.480+0000 7f74ebfbf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74e4148890 0x7f74e413bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:21.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.480+0000 7f74ebfbf640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f74e40ab920 con 0x7f74e4148890 2024-04-25T00:18:21.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.480+0000 7f74ebfbf640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f74e40ab7a0 con 0x7f74e406e890 2024-04-25T00:18:21.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.480+0000 7f74ebfbf640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f74e40abaa0 con 0x7f74e4070dd0 2024-04-25T00:18:21.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.484+0000 7f74ea55c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74e4148890 0x7f74e413bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:21.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.484+0000 7f74ea55c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74e4148890 0x7f74e413bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49348/0 (socket says 172.21.15.116:49348) 2024-04-25T00:18:21.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.484+0000 7f74ea55c640 1 -- 172.21.15.116:0/3964474662 learned_addr learned my addr 172.21.15.116:0/3964474662 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:18:21.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.484+0000 7f74e9d5b640 1 --2- 172.21.15.116:0/3964474662 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f74e406e890 0x7f74e40c6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:21.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.484+0000 7f74e955a640 1 --2- 172.21.15.116:0/3964474662 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f74e4070dd0 0x7f74e413b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:21.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.484+0000 7f74ea55c640 1 -- 172.21.15.116:0/3964474662 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f74e4070dd0 msgr2=0x7f74e413b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:21.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.484+0000 7f74ea55c640 1 --2- 172.21.15.116:0/3964474662 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f74e4070dd0 0x7f74e413b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:21.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.484+0000 7f74ea55c640 1 -- 172.21.15.116:0/3964474662 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f74e406e890 msgr2=0x7f74e40c6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:21.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.484+0000 7f74ea55c640 1 --2- 172.21.15.116:0/3964474662 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f74e406e890 0x7f74e40c6130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:21.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.484+0000 7f74ea55c640 1 -- 172.21.15.116:0/3964474662 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74e41530f0 con 0x7f74e4148890 2024-04-25T00:18:21.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.484+0000 7f74e955a640 1 --2- 172.21.15.116:0/3964474662 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f74e4070dd0 0x7f74e413b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:18:21.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.484+0000 7f74ea55c640 1 --2- 172.21.15.116:0/3964474662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74e4148890 0x7f74e413bd30 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f74dc009e50 tx=0x7f74dc0075c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:21.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.484+0000 7f74da7fc640 1 -- 172.21.15.116:0/3964474662 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74dc007e60 con 0x7f74e4148890 2024-04-25T00:18:21.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.484+0000 7f74da7fc640 1 -- 172.21.15.116:0/3964474662 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74dc019070 con 0x7f74e4148890 2024-04-25T00:18:21.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.484+0000 7f74ebfbf640 1 -- 172.21.15.116:0/3964474662 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74e41538a0 con 0x7f74e4148890 2024-04-25T00:18:21.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.484+0000 7f74da7fc640 1 -- 172.21.15.116:0/3964474662 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74dc014850 con 0x7f74e4148890 2024-04-25T00:18:21.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.484+0000 7f74ebfbf640 1 -- 172.21.15.116:0/3964474662 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74e40c6670 con 0x7f74e4148890 2024-04-25T00:18:21.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.484+0000 7f74ebfbf640 1 -- 172.21.15.116:0/3964474662 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74e4068350 con 0x7f74e4148890 2024-04-25T00:18:21.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.484+0000 7f74da7fc640 1 -- 172.21.15.116:0/3964474662 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f74dc0149f0 con 0x7f74e4148890 2024-04-25T00:18:21.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.484+0000 7f74da7fc640 1 --2- 172.21.15.116:0/3964474662 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f74c80678c0 0x7f74c8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:21.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.484+0000 7f74da7fc640 1 -- 172.21.15.116:0/3964474662 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 8902+0+0 (secure 0 0 0) 0x7f74dc0870a0 con 0x7f74e4148890 2024-04-25T00:18:21.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.484+0000 7f74e9d5b640 1 --2- 172.21.15.116:0/3964474662 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f74c80678c0 0x7f74c8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:21.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.488+0000 7f74e9d5b640 1 --2- 172.21.15.116:0/3964474662 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f74c80678c0 0x7f74c8069d80 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f74e00ad000 tx=0x7f74e00a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:21.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.488+0000 7f74da7fc640 1 -- 172.21.15.116:0/3964474662 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f74dc011070 con 0x7f74e4148890 2024-04-25T00:18:21.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.684+0000 7f74ebfbf640 1 -- 172.21.15.116:0/3964474662 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f74e406e670 con 0x7f74e4148890 2024-04-25T00:18:21.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.684+0000 7f74da7fc640 1 -- 172.21.15.116:0/3964474662 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) ==== 95+0+5673 (secure 0 0 0) 0x7f74dc056e80 con 0x7f74e4148890 2024-04-25T00:18:21.688 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:18:21.688 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":138,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:18:20.613896+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7362},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7362":{"gid":7362,"name":"a","rank":0,"incarnation":138,"state":"up:replay","state_seq":1,"addr":"172.21.15.116:6833/1655959433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1655959433},{"type":"v1","addr":"172.21.15.116:6833","nonce":1655959433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:17:28.837517+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7164},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7164":{"gid":7164,"name":"c","rank":0,"incarnation":128,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6835/3336339376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3336339376},{"type":"v1","addr":"172.21.15.116:6835","nonce":3336339376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7164,"qdb_cluster":[7164]},"id":2}]} 2024-04-25T00:18:21.688 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 138 2024-04-25T00:18:21.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.688+0000 7f74ebfbf640 1 -- 172.21.15.116:0/3964474662 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f74c80678c0 msgr2=0x7f74c8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:21.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.688+0000 7f74ebfbf640 1 --2- 172.21.15.116:0/3964474662 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f74c80678c0 0x7f74c8069d80 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f74e00ad000 tx=0x7f74e00a6000 comp rx=0 tx=0).stop 2024-04-25T00:18:21.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.688+0000 7f74ebfbf640 1 -- 172.21.15.116:0/3964474662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74e4148890 msgr2=0x7f74e413bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:21.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.688+0000 7f74ebfbf640 1 --2- 172.21.15.116:0/3964474662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74e4148890 0x7f74e413bd30 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f74dc009e50 tx=0x7f74dc0075c0 comp rx=0 tx=0).stop 2024-04-25T00:18:21.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.688+0000 7f74ebfbf640 1 -- 172.21.15.116:0/3964474662 shutdown_connections 2024-04-25T00:18:21.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.688+0000 7f74ebfbf640 1 --2- 172.21.15.116:0/3964474662 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f74c80678c0 0x7f74c8069d80 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:21.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.688+0000 7f74ebfbf640 1 --2- 172.21.15.116:0/3964474662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f74e4148890 0x7f74e413bd30 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:21.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.688+0000 7f74ebfbf640 1 --2- 172.21.15.116:0/3964474662 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f74e4070dd0 0x7f74e413b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:21.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.688+0000 7f74ebfbf640 1 --2- 172.21.15.116:0/3964474662 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f74e406e890 0x7f74e40c6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:21.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.688+0000 7f74ebfbf640 1 -- 172.21.15.116:0/3964474662 >> 172.21.15.116:0/3964474662 conn(0x7f74e40c3b00 msgr2=0x7f74e406cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:21.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.688+0000 7f74ebfbf640 1 -- 172.21.15.116:0/3964474662 shutdown_connections 2024-04-25T00:18:21.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:21.688+0000 7f74ebfbf640 1 -- 172.21.15.116:0/3964474662 wait complete. 2024-04-25T00:18:21.707 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:18:21.974 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:18:22.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.100+0000 7fd6d6d30640 1 Processor -- start 2024-04-25T00:18:22.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.104+0000 7fd6d6d30640 1 -- start start 2024-04-25T00:18:22.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.104+0000 7fd6d6d30640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd6d01541d0 0x7fd6d00ad2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:22.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.104+0000 7fd6d6d30640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fd6d00c32d0 0x7fd6d00c36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:22.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.104+0000 7fd6d6d30640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6d00a9ef0 con 0x7fd6d00acef0 2024-04-25T00:18:22.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.104+0000 7fd6d6d30640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fd6d00c2b60 con 0x7fd6d01541d0 2024-04-25T00:18:22.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.104+0000 7fd6d6d30640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fd6d00c2e40 con 0x7fd6d00c32d0 2024-04-25T00:18:22.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.104+0000 7fd6cffff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd6d01541d0 0x7fd6d00ad2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:22.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.104+0000 7fd6cffff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd6d01541d0 0x7fd6d00ad2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49360/0 (socket says 172.21.15.116:49360) 2024-04-25T00:18:22.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.104+0000 7fd6cffff640 1 -- 172.21.15.116:0/3559282795 learned_addr learned my addr 172.21.15.116:0/3559282795 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:18:22.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.104+0000 7fd6d52cd640 1 --2- 172.21.15.116:0/3559282795 >> v2:172.21.15.174:3300/0 conn(0x7fd6d00c32d0 0x7fd6d00c36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:22.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.104+0000 7fd6cffff640 1 -- 172.21.15.116:0/3559282795 >> v2:172.21.15.174:3300/0 conn(0x7fd6d00c32d0 msgr2=0x7fd6d00c36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:22.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.104+0000 7fd6cffff640 1 --2- 172.21.15.116:0/3559282795 >> v2:172.21.15.174:3300/0 conn(0x7fd6d00c32d0 0x7fd6d00c36e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:22.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.104+0000 7fd6cffff640 1 -- 172.21.15.116:0/3559282795 >> v1:172.21.15.174:6789/0 conn(0x7fd6d00acef0 legacy=0x7fd6d00a7d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:22.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.104+0000 7fd6cffff640 1 -- 172.21.15.116:0/3559282795 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6d00aa1d0 con 0x7fd6d01541d0 2024-04-25T00:18:22.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.104+0000 7fd6cffff640 1 --2- 172.21.15.116:0/3559282795 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6d01541d0 0x7fd6d00ad2d0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7fd6d00a7970 tx=0x7fd6c001c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ab11907d700b158 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:22.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.104+0000 7fd6cf7fe640 1 -- 172.21.15.116:0/3559282795 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c00b0070 con 0x7fd6d01541d0 2024-04-25T00:18:22.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.104+0000 7fd6cf7fe640 1 -- 172.21.15.116:0/3559282795 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6c00ab600 con 0x7fd6d01541d0 2024-04-25T00:18:22.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.104+0000 7fd6cf7fe640 1 -- 172.21.15.116:0/3559282795 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c00ab9d0 con 0x7fd6d01541d0 2024-04-25T00:18:22.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.108+0000 7fd6d6d30640 1 -- 172.21.15.116:0/3559282795 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6d01541d0 msgr2=0x7fd6d00ad2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:22.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.108+0000 7fd6d6d30640 1 --2- 172.21.15.116:0/3559282795 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6d01541d0 0x7fd6d00ad2d0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7fd6d00a7970 tx=0x7fd6c001c450 comp rx=0 tx=0).stop 2024-04-25T00:18:22.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.108+0000 7fd6d6d30640 1 -- 172.21.15.116:0/3559282795 shutdown_connections 2024-04-25T00:18:22.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.108+0000 7fd6d6d30640 1 --2- 172.21.15.116:0/3559282795 >> v2:172.21.15.174:3300/0 conn(0x7fd6d00c32d0 0x7fd6d00c36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:22.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.108+0000 7fd6d6d30640 1 --2- 172.21.15.116:0/3559282795 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6d01541d0 0x7fd6d00ad2d0 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:22.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.108+0000 7fd6d6d30640 1 -- 172.21.15.116:0/3559282795 >> 172.21.15.116:0/3559282795 conn(0x7fd6d00c4200 msgr2=0x7fd6d00c4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:22.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.108+0000 7fd6d6d30640 1 -- 172.21.15.116:0/3559282795 shutdown_connections 2024-04-25T00:18:22.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.108+0000 7fd6d6d30640 1 -- 172.21.15.116:0/3559282795 wait complete. 2024-04-25T00:18:22.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.108+0000 7fd6d6d30640 1 Processor -- start 2024-04-25T00:18:22.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.108+0000 7fd6d6d30640 1 -- start start 2024-04-25T00:18:22.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.108+0000 7fd6d6d30640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd6d00acef0 0x7fd6d014ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:22.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.108+0000 7fd6d6d30640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd6d00c32d0 0x7fd6d01f2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:22.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.108+0000 7fd6d6d30640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6d01541d0 0x7fd6d01f2950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:22.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.108+0000 7fd6d6d30640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d0160860 con 0x7fd6d01541d0 2024-04-25T00:18:22.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.108+0000 7fd6d6d30640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d0154650 con 0x7fd6d00c32d0 2024-04-25T00:18:22.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.108+0000 7fd6d6d30640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fd6d01609e0 con 0x7fd6d00acef0 2024-04-25T00:18:22.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.108+0000 7fd6d52cd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6d01541d0 0x7fd6d01f2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:22.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.108+0000 7fd6d52cd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6d01541d0 0x7fd6d01f2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49376/0 (socket says 172.21.15.116:49376) 2024-04-25T00:18:22.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.108+0000 7fd6d52cd640 1 -- 172.21.15.116:0/1739410528 learned_addr learned my addr 172.21.15.116:0/1739410528 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:18:22.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.108+0000 7fd6d4acc640 1 --2- 172.21.15.116:0/1739410528 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd6d00acef0 0x7fd6d014ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:22.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.108+0000 7fd6d52cd640 1 -- 172.21.15.116:0/1739410528 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd6d00acef0 msgr2=0x7fd6d014ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:22.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.108+0000 7fd6d52cd640 1 --2- 172.21.15.116:0/1739410528 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd6d00acef0 0x7fd6d014ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:22.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.108+0000 7fd6d52cd640 1 -- 172.21.15.116:0/1739410528 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd6d00c32d0 msgr2=0x7fd6d01f2410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:22.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.108+0000 7fd6d52cd640 1 --2- 172.21.15.116:0/1739410528 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd6d00c32d0 0x7fd6d01f2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:22.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.108+0000 7fd6d52cd640 1 -- 172.21.15.116:0/1739410528 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6d0157a70 con 0x7fd6d01541d0 2024-04-25T00:18:22.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.108+0000 7fd6d52cd640 1 --2- 172.21.15.116:0/1739410528 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6d01541d0 0x7fd6d01f2950 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7fd6c800cd70 tx=0x7fd6c800a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:22.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.108+0000 7fd6cd7fa640 1 -- 172.21.15.116:0/1739410528 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c8003ba0 con 0x7fd6d01541d0 2024-04-25T00:18:22.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.108+0000 7fd6cd7fa640 1 -- 172.21.15.116:0/1739410528 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6c8003d40 con 0x7fd6d01541d0 2024-04-25T00:18:22.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.108+0000 7fd6d6d30640 1 -- 172.21.15.116:0/1739410528 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6d014dbd0 con 0x7fd6d01541d0 2024-04-25T00:18:22.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.108+0000 7fd6cd7fa640 1 -- 172.21.15.116:0/1739410528 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c8004240 con 0x7fd6d01541d0 2024-04-25T00:18:22.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.108+0000 7fd6d6d30640 1 -- 172.21.15.116:0/1739410528 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6d00c1c10 con 0x7fd6d01541d0 2024-04-25T00:18:22.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.112+0000 7fd6d6d30640 1 -- 172.21.15.116:0/1739410528 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6d0068a60 con 0x7fd6d01541d0 2024-04-25T00:18:22.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.112+0000 7fd6cd7fa640 1 -- 172.21.15.116:0/1739410528 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd6c80043e0 con 0x7fd6d01541d0 2024-04-25T00:18:22.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.112+0000 7fd6cd7fa640 1 --2- 172.21.15.116:0/1739410528 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd6bc0678c0 0x7fd6bc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:22.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.112+0000 7fd6cd7fa640 1 -- 172.21.15.116:0/1739410528 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 8902+0+0 (secure 0 0 0) 0x7fd6c80873d0 con 0x7fd6d01541d0 2024-04-25T00:18:22.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.112+0000 7fd6d4acc640 1 --2- 172.21.15.116:0/1739410528 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd6bc0678c0 0x7fd6bc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:22.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.112+0000 7fd6d4acc640 1 --2- 172.21.15.116:0/1739410528 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd6bc0678c0 0x7fd6bc069d80 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fd6c4003d60 tx=0x7fd6c4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:22.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.116+0000 7fd6cd7fa640 1 -- 172.21.15.116:0/1739410528 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fd6c801d340 con 0x7fd6d01541d0 2024-04-25T00:18:22.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.308+0000 7fd6d6d30640 1 -- 172.21.15.116:0/1739410528 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd6d0153d80 con 0x7fd6d01541d0 2024-04-25T00:18:22.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.308+0000 7fd6cd7fa640 1 -- 172.21.15.116:0/1739410528 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) ==== 95+0+5677 (secure 0 0 0) 0x7fd6c80572d0 con 0x7fd6d01541d0 2024-04-25T00:18:22.309 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:18:22.309 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":139,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:18:20.954694+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7362},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7362":{"gid":7362,"name":"a","rank":0,"incarnation":138,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.116:6833/1655959433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1655959433},{"type":"v1","addr":"172.21.15.116:6833","nonce":1655959433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:17:28.837517+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7164},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7164":{"gid":7164,"name":"c","rank":0,"incarnation":128,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6835/3336339376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3336339376},{"type":"v1","addr":"172.21.15.116:6835","nonce":3336339376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7164,"qdb_cluster":[7164]},"id":2}]} 2024-04-25T00:18:22.309 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 139 2024-04-25T00:18:22.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.308+0000 7fd6d6d30640 1 -- 172.21.15.116:0/1739410528 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd6bc0678c0 msgr2=0x7fd6bc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:22.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.308+0000 7fd6d6d30640 1 --2- 172.21.15.116:0/1739410528 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd6bc0678c0 0x7fd6bc069d80 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fd6c4003d60 tx=0x7fd6c4009040 comp rx=0 tx=0).stop 2024-04-25T00:18:22.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.308+0000 7fd6d6d30640 1 -- 172.21.15.116:0/1739410528 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6d01541d0 msgr2=0x7fd6d01f2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:22.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.308+0000 7fd6d6d30640 1 --2- 172.21.15.116:0/1739410528 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6d01541d0 0x7fd6d01f2950 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7fd6c800cd70 tx=0x7fd6c800a660 comp rx=0 tx=0).stop 2024-04-25T00:18:22.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.308+0000 7fd6d6d30640 1 -- 172.21.15.116:0/1739410528 shutdown_connections 2024-04-25T00:18:22.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.308+0000 7fd6d6d30640 1 --2- 172.21.15.116:0/1739410528 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd6bc0678c0 0x7fd6bc069d80 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:22.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.308+0000 7fd6d6d30640 1 --2- 172.21.15.116:0/1739410528 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6d01541d0 0x7fd6d01f2950 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:22.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.308+0000 7fd6d6d30640 1 --2- 172.21.15.116:0/1739410528 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd6d00c32d0 0x7fd6d01f2410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:22.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.308+0000 7fd6d6d30640 1 --2- 172.21.15.116:0/1739410528 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd6d00acef0 0x7fd6d014ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:22.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.308+0000 7fd6d6d30640 1 -- 172.21.15.116:0/1739410528 >> 172.21.15.116:0/1739410528 conn(0x7fd6d00c4200 msgr2=0x7fd6d00c45e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:22.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.308+0000 7fd6d6d30640 1 -- 172.21.15.116:0/1739410528 shutdown_connections 2024-04-25T00:18:22.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.308+0000 7fd6d6d30640 1 -- 172.21.15.116:0/1739410528 wait complete. 2024-04-25T00:18:22.327 INFO:tasks.mds_thrash.fs.[a]:mds.d down, removed from mdsmap 2024-04-25T00:18:22.327 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-04-25T00:18:22.328 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:18:22.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.432+0000 7fdc6aa9f640 1 Processor -- start 2024-04-25T00:18:22.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.432+0000 7fdc6aa9f640 1 -- start start 2024-04-25T00:18:22.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.436+0000 7fdc6aa9f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdc64150d20 0x7fdc640c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:22.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.436+0000 7fdc6aa9f640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fdc640c2f50 0x7fdc641538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:22.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.436+0000 7fdc6aa9f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc640c3910 con 0x7fdc640a94a0 2024-04-25T00:18:22.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.436+0000 7fdc6aa9f640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fdc64154530 con 0x7fdc640c2f50 2024-04-25T00:18:22.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.436+0000 7fdc6aa9f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fdc640c16e0 con 0x7fdc64150d20 2024-04-25T00:18:22.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.436+0000 7fdc6903c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fdc640a94a0 0x7fdc64153e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:56712/0 (socket says 172.21.15.116:56712) 2024-04-25T00:18:22.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.436+0000 7fdc6903c640 1 -- 172.21.15.116:0/1578697449 learned_addr learned my addr 172.21.15.116:0/1578697449 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:18:22.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.436+0000 7fdc63fff640 1 --2- 172.21.15.116:0/1578697449 >> v2:172.21.15.174:3300/0 conn(0x7fdc640c2f50 0x7fdc641538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:22.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.436+0000 7fdc6883b640 1 --2- 172.21.15.116:0/1578697449 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdc64150d20 0x7fdc640c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:22.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.436+0000 7fdc637fe640 1 -- 172.21.15.116:0/1578697449 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4001233429 0 0) 0x7fdc640c3910 con 0x7fdc640a94a0 2024-04-25T00:18:22.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.436+0000 7fdc637fe640 1 -- 172.21.15.116:0/1578697449 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc4c003530 con 0x7fdc640a94a0 2024-04-25T00:18:22.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.436+0000 7fdc637fe640 1 -- 172.21.15.116:0/1578697449 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3607379606 0 0) 0x7fdc4c003530 con 0x7fdc640a94a0 2024-04-25T00:18:22.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.436+0000 7fdc637fe640 1 -- 172.21.15.116:0/1578697449 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdc64150d20 msgr2=0x7fdc640c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:22.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.436+0000 7fdc637fe640 1 --2- 172.21.15.116:0/1578697449 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdc64150d20 0x7fdc640c2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:22.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.436+0000 7fdc637fe640 1 -- 172.21.15.116:0/1578697449 >> v2:172.21.15.174:3300/0 conn(0x7fdc640c2f50 msgr2=0x7fdc641538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:22.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.436+0000 7fdc637fe640 1 --2- 172.21.15.116:0/1578697449 >> v2:172.21.15.174:3300/0 conn(0x7fdc640c2f50 0x7fdc641538e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:22.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.436+0000 7fdc637fe640 1 -- 172.21.15.116:0/1578697449 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc640c1400 con 0x7fdc640a94a0 2024-04-25T00:18:22.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.436+0000 7fdc637fe640 1 -- 172.21.15.116:0/1578697449 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7fdc5c001a00 con 0x7fdc640a94a0 2024-04-25T00:18:22.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.436+0000 7fdc637fe640 1 -- 172.21.15.116:0/1578697449 >> v1:172.21.15.116:6789/0 conn(0x7fdc640a94a0 legacy=0x7fdc64153e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:22.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.436+0000 7fdc637fe640 1 --2- 172.21.15.116:0/1578697449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc4c0039c0 0x7fdc4c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:22.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.436+0000 7fdc637fe640 1 --2- 172.21.15.116:0/1578697449 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdc4c004810 0x7fdc4c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:22.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.436+0000 7fdc637fe640 1 --2- 172.21.15.116:0/1578697449 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdc64150d20 0x7fdc4c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:22.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.436+0000 7fdc637fe640 1 -- 172.21.15.116:0/1578697449 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdc64154530 con 0x7fdc4c0039c0 2024-04-25T00:18:22.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.436+0000 7fdc637fe640 1 -- 172.21.15.116:0/1578697449 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fdc640c16e0 con 0x7fdc4c004810 2024-04-25T00:18:22.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.436+0000 7fdc637fe640 1 -- 172.21.15.116:0/1578697449 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fdc4c00b9d0 con 0x7fdc64150d20 2024-04-25T00:18:22.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.436+0000 7fdc6883b640 1 --2- 172.21.15.116:0/1578697449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc4c0039c0 0x7fdc4c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:22.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.436+0000 7fdc63fff640 1 --2- 172.21.15.116:0/1578697449 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdc4c004810 0x7fdc4c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:22.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.436+0000 7fdc6883b640 1 -- 172.21.15.116:0/1578697449 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdc64150d20 msgr2=0x7fdc4c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:22.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.436+0000 7fdc6883b640 1 --2- 172.21.15.116:0/1578697449 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdc64150d20 0x7fdc4c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:22.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.436+0000 7fdc6883b640 1 -- 172.21.15.116:0/1578697449 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdc4c004810 msgr2=0x7fdc4c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:22.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.436+0000 7fdc6883b640 1 --2- 172.21.15.116:0/1578697449 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdc4c004810 0x7fdc4c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:22.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.436+0000 7fdc6883b640 1 -- 172.21.15.116:0/1578697449 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc4c003530 con 0x7fdc4c0039c0 2024-04-25T00:18:22.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.436+0000 7fdc6883b640 1 --2- 172.21.15.116:0/1578697449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc4c0039c0 0x7fdc4c003de0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fdc54009ae0 tx=0x7fdc5401cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=124bc0fd0da7948d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:22.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.436+0000 7fdc637fe640 1 -- 172.21.15.116:0/1578697449 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc540096b0 con 0x7fdc4c0039c0 2024-04-25T00:18:22.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.436+0000 7fdc637fe640 1 -- 172.21.15.116:0/1578697449 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc540a64c0 con 0x7fdc4c0039c0 2024-04-25T00:18:22.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.436+0000 7fdc637fe640 1 -- 172.21.15.116:0/1578697449 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc540a68b0 con 0x7fdc4c0039c0 2024-04-25T00:18:22.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.440+0000 7fdc6aa9f640 1 -- 172.21.15.116:0/1578697449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc4c0039c0 msgr2=0x7fdc4c003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:22.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.440+0000 7fdc6aa9f640 1 --2- 172.21.15.116:0/1578697449 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc4c0039c0 0x7fdc4c003de0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fdc54009ae0 tx=0x7fdc5401cab0 comp rx=0 tx=0).stop 2024-04-25T00:18:22.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.440+0000 7fdc6903c640 1 -- 172.21.15.116:0/1578697449 reap_dead start 2024-04-25T00:18:22.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.440+0000 7fdc6aa9f640 1 -- 172.21.15.116:0/1578697449 shutdown_connections 2024-04-25T00:18:22.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.440+0000 7fdc6aa9f640 1 -- 172.21.15.116:0/1578697449 >> 172.21.15.116:0/1578697449 conn(0x7fdc640c3b00 msgr2=0x7fdc640c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:22.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.440+0000 7fdc6aa9f640 1 -- 172.21.15.116:0/1578697449 shutdown_connections 2024-04-25T00:18:22.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.440+0000 7fdc6aa9f640 1 -- 172.21.15.116:0/1578697449 wait complete. 2024-04-25T00:18:22.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.440+0000 7fdc6aa9f640 1 Processor -- start 2024-04-25T00:18:22.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.440+0000 7fdc6aa9f640 1 -- start start 2024-04-25T00:18:22.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.440+0000 7fdc6aa9f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdc4c0039c0 0x7fdc641432d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:22.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.440+0000 7fdc6aa9f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc64143f70 0x7fdc640ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:22.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.440+0000 7fdc6aa9f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdc64143a20 0x7fdc640ace00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:22.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.440+0000 7fdc6aa9f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdc640716d0 con 0x7fdc64143f70 2024-04-25T00:18:22.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.440+0000 7fdc6aa9f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fdc64071550 con 0x7fdc64143a20 2024-04-25T00:18:22.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.440+0000 7fdc6aa9f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fdc64071850 con 0x7fdc4c0039c0 2024-04-25T00:18:22.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.440+0000 7fdc63fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc64143f70 0x7fdc640ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:22.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.440+0000 7fdc63fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc64143f70 0x7fdc640ac8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49400/0 (socket says 172.21.15.116:49400) 2024-04-25T00:18:22.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.440+0000 7fdc63fff640 1 -- 172.21.15.116:0/94637060 learned_addr learned my addr 172.21.15.116:0/94637060 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:18:22.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.440+0000 7fdc6883b640 1 --2- 172.21.15.116:0/94637060 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdc4c0039c0 0x7fdc641432d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:22.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.440+0000 7fdc6903c640 1 --2- 172.21.15.116:0/94637060 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdc64143a20 0x7fdc640ace00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:22.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.440+0000 7fdc63fff640 1 -- 172.21.15.116:0/94637060 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdc4c0039c0 msgr2=0x7fdc641432d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:22.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.440+0000 7fdc63fff640 1 --2- 172.21.15.116:0/94637060 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdc4c0039c0 0x7fdc641432d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:22.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.440+0000 7fdc63fff640 1 -- 172.21.15.116:0/94637060 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdc64143a20 msgr2=0x7fdc640ace00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:22.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.440+0000 7fdc63fff640 1 --2- 172.21.15.116:0/94637060 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdc64143a20 0x7fdc640ace00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:22.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.440+0000 7fdc63fff640 1 -- 172.21.15.116:0/94637060 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc64143810 con 0x7fdc64143f70 2024-04-25T00:18:22.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.440+0000 7fdc63fff640 1 --2- 172.21.15.116:0/94637060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc64143f70 0x7fdc640ac8c0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7fdc58011660 tx=0x7fdc58011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:22.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.440+0000 7fdc617fa640 1 -- 172.21.15.116:0/94637060 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc5800f9c0 con 0x7fdc64143f70 2024-04-25T00:18:22.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.440+0000 7fdc617fa640 1 -- 172.21.15.116:0/94637060 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc5800fb60 con 0x7fdc64143f70 2024-04-25T00:18:22.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.440+0000 7fdc6aa9f640 1 -- 172.21.15.116:0/94637060 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc640c4d90 con 0x7fdc64143f70 2024-04-25T00:18:22.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.440+0000 7fdc617fa640 1 -- 172.21.15.116:0/94637060 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc58015460 con 0x7fdc64143f70 2024-04-25T00:18:22.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.440+0000 7fdc6aa9f640 1 -- 172.21.15.116:0/94637060 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc64147ef0 con 0x7fdc64143f70 2024-04-25T00:18:22.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.444+0000 7fdc6aa9f640 1 -- 172.21.15.116:0/94637060 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc640c4af0 con 0x7fdc64143f70 2024-04-25T00:18:22.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.444+0000 7fdc617fa640 1 -- 172.21.15.116:0/94637060 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fdc58015600 con 0x7fdc64143f70 2024-04-25T00:18:22.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.444+0000 7fdc617fa640 1 --2- 172.21.15.116:0/94637060 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdc500678c0 0x7fdc50069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:22.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.444+0000 7fdc617fa640 1 -- 172.21.15.116:0/94637060 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 8902+0+0 (secure 0 0 0) 0x7fdc58087140 con 0x7fdc64143f70 2024-04-25T00:18:22.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.444+0000 7fdc6883b640 1 --2- 172.21.15.116:0/94637060 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdc500678c0 0x7fdc50069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:22.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.444+0000 7fdc6883b640 1 --2- 172.21.15.116:0/94637060 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdc500678c0 0x7fdc50069d80 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fdc540af000 tx=0x7fdc540046d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:22.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.448+0000 7fdc617fa640 1 -- 172.21.15.116:0/94637060 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fdc58019030 con 0x7fdc64143f70 2024-04-25T00:18:22.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.636+0000 7fdc6aa9f640 1 -- 172.21.15.116:0/94637060 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdc64068350 con 0x7fdc64143f70 2024-04-25T00:18:22.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.636+0000 7fdc617fa640 1 -- 172.21.15.116:0/94637060 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) ==== 95+0+5677 (secure 0 0 0) 0x7fdc58057040 con 0x7fdc64143f70 2024-04-25T00:18:22.638 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:18:22.638 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":139,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:18:20.954694+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7362},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7362":{"gid":7362,"name":"a","rank":0,"incarnation":138,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.116:6833/1655959433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1655959433},{"type":"v1","addr":"172.21.15.116:6833","nonce":1655959433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:17:28.837517+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7164},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7164":{"gid":7164,"name":"c","rank":0,"incarnation":128,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6835/3336339376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3336339376},{"type":"v1","addr":"172.21.15.116:6835","nonce":3336339376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7164,"qdb_cluster":[7164]},"id":2}]} 2024-04-25T00:18:22.639 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 139 2024-04-25T00:18:22.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.640+0000 7fdc6aa9f640 1 -- 172.21.15.116:0/94637060 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdc500678c0 msgr2=0x7fdc50069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:22.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.640+0000 7fdc6aa9f640 1 --2- 172.21.15.116:0/94637060 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdc500678c0 0x7fdc50069d80 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fdc540af000 tx=0x7fdc540046d0 comp rx=0 tx=0).stop 2024-04-25T00:18:22.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.640+0000 7fdc6aa9f640 1 -- 172.21.15.116:0/94637060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc64143f70 msgr2=0x7fdc640ac8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:22.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.640+0000 7fdc6aa9f640 1 --2- 172.21.15.116:0/94637060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc64143f70 0x7fdc640ac8c0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7fdc58011660 tx=0x7fdc58011b20 comp rx=0 tx=0).stop 2024-04-25T00:18:22.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.640+0000 7fdc6aa9f640 1 -- 172.21.15.116:0/94637060 shutdown_connections 2024-04-25T00:18:22.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.640+0000 7fdc6aa9f640 1 --2- 172.21.15.116:0/94637060 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdc500678c0 0x7fdc50069d80 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:22.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.640+0000 7fdc6aa9f640 1 --2- 172.21.15.116:0/94637060 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdc64143a20 0x7fdc640ace00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:22.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.640+0000 7fdc6aa9f640 1 --2- 172.21.15.116:0/94637060 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc64143f70 0x7fdc640ac8c0 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:22.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.640+0000 7fdc6aa9f640 1 --2- 172.21.15.116:0/94637060 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdc4c0039c0 0x7fdc641432d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:22.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.640+0000 7fdc6aa9f640 1 -- 172.21.15.116:0/94637060 >> 172.21.15.116:0/94637060 conn(0x7fdc640c3b00 msgr2=0x7fdc6414a190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:22.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.640+0000 7fdc6aa9f640 1 -- 172.21.15.116:0/94637060 shutdown_connections 2024-04-25T00:18:22.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:22.640+0000 7fdc6aa9f640 1 -- 172.21.15.116:0/94637060 wait complete. 2024-04-25T00:18:22.657 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 139, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:04.678737+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 139, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_7362": { "addr": "172.21.15.116:6833/1655959433", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6832", "nonce": 1655959433, "type": "v2" }, { "addr": "172.21.15.116:6833", "nonce": 1655959433, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7362, "incarnation": 138, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:reconnect", "state_seq": 10 } }, "last_failure": 0, "last_failure_osd_epoch": 54, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T00:18:20.954694+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7362 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:18.825887+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 131, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_7164": { "addr": "172.21.15.116:6835/3336339376", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6834", "nonce": 3336339376, "type": "v2" }, { "addr": "172.21.15.116:6835", "nonce": 3336339376, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7164, "incarnation": 128, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 12 } }, "last_failure": 0, "last_failure_osd_epoch": 52, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-04-25T00:17:28.837517+0000", "qdb_cluster": [ 7164 ], "qdb_leader": 7164, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7164 } } } ], "standbys": [ { "addr": "172.21.15.174:6835/4112145782", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6834", "nonce": 4112145782, "type": "v2" }, { "addr": "172.21.15.174:6835", "nonce": 4112145782, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 112, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5582, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-25T00:18:23.708 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:18:23.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.836+0000 7f8b92d2c640 1 Processor -- start 2024-04-25T00:18:23.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.836+0000 7f8b92d2c640 1 -- start start 2024-04-25T00:18:23.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.836+0000 7f8b92d2c640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f8b8c150d40 0x7f8b8c154170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:23.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.836+0000 7f8b92d2c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8b8c0c4c20 0x7f8b8c0c6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:23.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.836+0000 7f8b92d2c640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b8c070180 con 0x7f8b8c070770 2024-04-25T00:18:23.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.836+0000 7f8b92d2c640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f8b8c071280 con 0x7f8b8c0c4c20 2024-04-25T00:18:23.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.836+0000 7f8b92d2c640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f8b8c06ff30 con 0x7f8b8c150d40 2024-04-25T00:18:23.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.836+0000 7f8b8bfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8b8c0c4c20 0x7f8b8c0c6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:23.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.836+0000 7f8b8bfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8b8c0c4c20 0x7f8b8c0c6e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49402/0 (socket says 172.21.15.116:49402) 2024-04-25T00:18:23.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.836+0000 7f8b8bfff640 1 -- 172.21.15.116:0/1833981899 learned_addr learned my addr 172.21.15.116:0/1833981899 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:18:23.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.836+0000 7f8b912c9640 1 --1- 172.21.15.116:0/1833981899 >> v1:172.21.15.116:6789/0 conn(0x7f8b8c070770 0x7f8b8c0ab200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:56726/0 (socket says 172.21.15.116:56726) 2024-04-25T00:18:23.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.836+0000 7f8b90ac8640 1 --2- 172.21.15.116:0/1833981899 >> v2:172.21.15.174:3300/0 conn(0x7f8b8c150d40 0x7f8b8c154170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:23.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.836+0000 7f8b8bfff640 1 -- 172.21.15.116:0/1833981899 >> v2:172.21.15.174:3300/0 conn(0x7f8b8c150d40 msgr2=0x7f8b8c154170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:23.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.836+0000 7f8b8bfff640 1 --2- 172.21.15.116:0/1833981899 >> v2:172.21.15.174:3300/0 conn(0x7f8b8c150d40 0x7f8b8c154170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:23.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.836+0000 7f8b8bfff640 1 -- 172.21.15.116:0/1833981899 >> v1:172.21.15.116:6789/0 conn(0x7f8b8c070770 legacy=0x7f8b8c0ab200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:23.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.836+0000 7f8b8b7fe640 1 -- 172.21.15.116:0/1833981899 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2897552079 0 0) 0x7f8b8c070180 con 0x7f8b8c070770 2024-04-25T00:18:23.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.836+0000 7f8b8bfff640 1 -- 172.21.15.116:0/1833981899 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b8c070460 con 0x7f8b8c0c4c20 2024-04-25T00:18:23.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.840+0000 7f8b8bfff640 1 --2- 172.21.15.116:0/1833981899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b8c0c4c20 0x7f8b8c0c6e20 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c0c93d0 tx=0x7f8b7801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d34ad5ac5efc7296 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:23.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.840+0000 7f8b8b7fe640 1 -- 172.21.15.116:0/1833981899 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b780b2070 con 0x7f8b8c0c4c20 2024-04-25T00:18:23.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.840+0000 7f8b8b7fe640 1 -- 172.21.15.116:0/1833981899 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b780ad680 con 0x7f8b8c0c4c20 2024-04-25T00:18:23.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.840+0000 7f8b8b7fe640 1 -- 172.21.15.116:0/1833981899 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b780ada50 con 0x7f8b8c0c4c20 2024-04-25T00:18:23.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.840+0000 7f8b92d2c640 1 -- 172.21.15.116:0/1833981899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b8c0c4c20 msgr2=0x7f8b8c0c6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:23.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.840+0000 7f8b92d2c640 1 --2- 172.21.15.116:0/1833981899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b8c0c4c20 0x7f8b8c0c6e20 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c0c93d0 tx=0x7f8b7801c490 comp rx=0 tx=0).stop 2024-04-25T00:18:23.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.840+0000 7f8b92d2c640 1 -- 172.21.15.116:0/1833981899 shutdown_connections 2024-04-25T00:18:23.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.840+0000 7f8b92d2c640 1 --2- 172.21.15.116:0/1833981899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b8c0c4c20 0x7f8b8c0c6e20 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:23.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.840+0000 7f8b92d2c640 1 --2- 172.21.15.116:0/1833981899 >> v2:172.21.15.174:3300/0 conn(0x7f8b8c150d40 0x7f8b8c154170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:23.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.840+0000 7f8b92d2c640 1 -- 172.21.15.116:0/1833981899 >> 172.21.15.116:0/1833981899 conn(0x7f8b8c06f0c0 msgr2=0x7f8b8c06f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:23.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.840+0000 7f8b92d2c640 1 -- 172.21.15.116:0/1833981899 shutdown_connections 2024-04-25T00:18:23.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.840+0000 7f8b92d2c640 1 -- 172.21.15.116:0/1833981899 wait complete. 2024-04-25T00:18:23.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.840+0000 7f8b92d2c640 1 Processor -- start 2024-04-25T00:18:23.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.840+0000 7f8b92d2c640 1 -- start start 2024-04-25T00:18:23.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.840+0000 7f8b92d2c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8b8c070770 0x7f8b8c06deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:23.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.840+0000 7f8b92d2c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8b8c0c4c20 0x7f8b8c06c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:23.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.840+0000 7f8b92d2c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b8c150d40 0x7f8b8c0c25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:23.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.840+0000 7f8b92d2c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8b8c0aba90 con 0x7f8b8c150d40 2024-04-25T00:18:23.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.840+0000 7f8b92d2c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8b8c0ab910 con 0x7f8b8c0c4c20 2024-04-25T00:18:23.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.840+0000 7f8b92d2c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f8b8c0c7360 con 0x7f8b8c070770 2024-04-25T00:18:23.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.840+0000 7f8b912c9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b8c150d40 0x7f8b8c0c25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:23.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.840+0000 7f8b912c9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b8c150d40 0x7f8b8c0c25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49414/0 (socket says 172.21.15.116:49414) 2024-04-25T00:18:23.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.840+0000 7f8b912c9640 1 -- 172.21.15.116:0/820543513 learned_addr learned my addr 172.21.15.116:0/820543513 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:18:23.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.840+0000 7f8b8bfff640 1 --2- 172.21.15.116:0/820543513 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8b8c0c4c20 0x7f8b8c06c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:23.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.840+0000 7f8b90ac8640 1 --2- 172.21.15.116:0/820543513 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8b8c070770 0x7f8b8c06deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:23.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.840+0000 7f8b912c9640 1 -- 172.21.15.116:0/820543513 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8b8c070770 msgr2=0x7f8b8c06deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:23.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.840+0000 7f8b912c9640 1 --2- 172.21.15.116:0/820543513 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8b8c070770 0x7f8b8c06deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:23.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.840+0000 7f8b912c9640 1 -- 172.21.15.116:0/820543513 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8b8c0c4c20 msgr2=0x7f8b8c06c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:23.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.840+0000 7f8b912c9640 1 --2- 172.21.15.116:0/820543513 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8b8c0c4c20 0x7f8b8c06c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:23.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.840+0000 7f8b912c9640 1 -- 172.21.15.116:0/820543513 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b8c152250 con 0x7f8b8c150d40 2024-04-25T00:18:23.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.840+0000 7f8b8bfff640 1 --2- 172.21.15.116:0/820543513 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8b8c0c4c20 0x7f8b8c06c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:18:23.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.840+0000 7f8b912c9640 1 --2- 172.21.15.116:0/820543513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b8c150d40 0x7f8b8c0c25b0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f8b8400c780 tx=0x7f8b8400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:23.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.840+0000 7f8b897fa640 1 -- 172.21.15.116:0/820543513 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b84018070 con 0x7f8b8c150d40 2024-04-25T00:18:23.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.840+0000 7f8b897fa640 1 -- 172.21.15.116:0/820543513 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b84004d70 con 0x7f8b8c150d40 2024-04-25T00:18:23.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.840+0000 7f8b92d2c640 1 -- 172.21.15.116:0/820543513 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b8c0add40 con 0x7f8b8c150d40 2024-04-25T00:18:23.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.840+0000 7f8b897fa640 1 -- 172.21.15.116:0/820543513 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b8400a820 con 0x7f8b8c150d40 2024-04-25T00:18:23.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.844+0000 7f8b92d2c640 1 -- 172.21.15.116:0/820543513 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b8c0ae300 con 0x7f8b8c150d40 2024-04-25T00:18:23.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.844+0000 7f8b92d2c640 1 -- 172.21.15.116:0/820543513 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b8c068a60 con 0x7f8b8c150d40 2024-04-25T00:18:23.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.844+0000 7f8b897fa640 1 -- 172.21.15.116:0/820543513 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8b8401b070 con 0x7f8b8c150d40 2024-04-25T00:18:23.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.844+0000 7f8b897fa640 1 --2- 172.21.15.116:0/820543513 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8b700678c0 0x7f8b70069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:23.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.844+0000 7f8b897fa640 1 -- 172.21.15.116:0/820543513 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 8902+0+0 (secure 0 0 0) 0x7f8b84085d80 con 0x7f8b8c150d40 2024-04-25T00:18:23.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.844+0000 7f8b90ac8640 1 --2- 172.21.15.116:0/820543513 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8b700678c0 0x7f8b70069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:23.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.844+0000 7f8b90ac8640 1 --2- 172.21.15.116:0/820543513 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8b700678c0 0x7f8b70069d80 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f8b80009790 tx=0x7f8b80008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:23.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:23.848+0000 7f8b897fa640 1 -- 172.21.15.116:0/820543513 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f8b84055bf0 con 0x7f8b8c150d40 2024-04-25T00:18:24.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.088+0000 7f8b92d2c640 1 -- 172.21.15.116:0/820543513 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8b8c14d4b0 con 0x7f8b8c150d40 2024-04-25T00:18:24.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.088+0000 7f8b897fa640 1 -- 172.21.15.116:0/820543513 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) ==== 95+0+5681 (secure 0 0 0) 0x7f8b840554f0 con 0x7f8b8c150d40 2024-04-25T00:18:24.091 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:18:24.091 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":141,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:18:23.899096+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7362},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7362":{"gid":7362,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6833/1655959433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1655959433},{"type":"v1","addr":"172.21.15.116:6833","nonce":1655959433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7362,"qdb_cluster":[7362]},"id":1},{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:17:28.837517+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7164},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7164":{"gid":7164,"name":"c","rank":0,"incarnation":128,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6835/3336339376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3336339376},{"type":"v1","addr":"172.21.15.116:6835","nonce":3336339376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7164,"qdb_cluster":[7164]},"id":2}]} 2024-04-25T00:18:24.092 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 141 2024-04-25T00:18:24.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.092+0000 7f8b92d2c640 1 -- 172.21.15.116:0/820543513 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8b700678c0 msgr2=0x7f8b70069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:24.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.092+0000 7f8b92d2c640 1 --2- 172.21.15.116:0/820543513 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8b700678c0 0x7f8b70069d80 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f8b80009790 tx=0x7f8b80008040 comp rx=0 tx=0).stop 2024-04-25T00:18:24.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.092+0000 7f8b92d2c640 1 -- 172.21.15.116:0/820543513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b8c150d40 msgr2=0x7f8b8c0c25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:24.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.092+0000 7f8b92d2c640 1 --2- 172.21.15.116:0/820543513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b8c150d40 0x7f8b8c0c25b0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f8b8400c780 tx=0x7f8b8400cc40 comp rx=0 tx=0).stop 2024-04-25T00:18:24.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.092+0000 7f8b92d2c640 1 -- 172.21.15.116:0/820543513 shutdown_connections 2024-04-25T00:18:24.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.092+0000 7f8b92d2c640 1 --2- 172.21.15.116:0/820543513 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8b700678c0 0x7f8b70069d80 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:24.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.092+0000 7f8b92d2c640 1 --2- 172.21.15.116:0/820543513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b8c150d40 0x7f8b8c0c25b0 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:24.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.092+0000 7f8b92d2c640 1 --2- 172.21.15.116:0/820543513 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8b8c0c4c20 0x7f8b8c06c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:24.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.092+0000 7f8b92d2c640 1 --2- 172.21.15.116:0/820543513 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8b8c070770 0x7f8b8c06deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:24.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.092+0000 7f8b92d2c640 1 -- 172.21.15.116:0/820543513 >> 172.21.15.116:0/820543513 conn(0x7f8b8c06f0c0 msgr2=0x7f8b8c0c5b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:24.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.092+0000 7f8b92d2c640 1 -- 172.21.15.116:0/820543513 shutdown_connections 2024-04-25T00:18:24.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.092+0000 7f8b92d2c640 1 -- 172.21.15.116:0/820543513 wait complete. 2024-04-25T00:18:24.110 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:18:24.658 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:18:24.768 INFO:tasks.ceph.mds.a.smithi116.stderr:2024-04-25T00:18:24.768+0000 7f8c9527b640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T00:18:24.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.780+0000 7fa155146640 1 Processor -- start 2024-04-25T00:18:24.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.780+0000 7fa155146640 1 -- start start 2024-04-25T00:18:24.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.780+0000 7fa155146640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa15006c480 0x7fa1500c3060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:24.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.780+0000 7fa155146640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa1500c1c30 0x7fa150071e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:24.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.780+0000 7fa155146640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1500c22a0 con 0x7fa150153170 2024-04-25T00:18:24.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.780+0000 7fa155146640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fa1500c35a0 con 0x7fa1500c1c30 2024-04-25T00:18:24.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.780+0000 7fa155146640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa150153670 con 0x7fa15006c480 2024-04-25T00:18:24.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.784+0000 7fa147fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa1500c1c30 0x7fa150071e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:24.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.784+0000 7fa147fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa1500c1c30 0x7fa150071e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50596/0 (socket says 172.21.15.116:50596) 2024-04-25T00:18:24.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.784+0000 7fa147fff640 1 -- 172.21.15.116:0/1480948431 learned_addr learned my addr 172.21.15.116:0/1480948431 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:18:24.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.784+0000 7fa14ed9d640 1 --2- 172.21.15.116:0/1480948431 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa15006c480 0x7fa1500c3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:24.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.784+0000 7fa147fff640 1 -- 172.21.15.116:0/1480948431 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa15006c480 msgr2=0x7fa1500c3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:24.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.784+0000 7fa147fff640 1 --2- 172.21.15.116:0/1480948431 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa15006c480 0x7fa1500c3060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:24.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.784+0000 7fa147fff640 1 -- 172.21.15.116:0/1480948431 >> v1:172.21.15.116:6789/0 conn(0x7fa150153170 legacy=0x7fa15014fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:24.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.784+0000 7fa147fff640 1 -- 172.21.15.116:0/1480948431 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1500c2580 con 0x7fa1500c1c30 2024-04-25T00:18:24.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.784+0000 7fa14e59c640 1 -- 172.21.15.116:0/1480948431 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 526924073 0 0) 0x7fa1500c22a0 con 0x7fa150153170 2024-04-25T00:18:24.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.784+0000 7fa147fff640 1 --2- 172.21.15.116:0/1480948431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1500c1c30 0x7fa150071e70 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7fa138009870 tx=0x7fa13801c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f874585d5add3484 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:24.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.784+0000 7fa14e59c640 1 -- 172.21.15.116:0/1480948431 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1380b2070 con 0x7fa1500c1c30 2024-04-25T00:18:24.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.784+0000 7fa14e59c640 1 -- 172.21.15.116:0/1480948431 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1380a9070 con 0x7fa1500c1c30 2024-04-25T00:18:24.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.784+0000 7fa14e59c640 1 -- 172.21.15.116:0/1480948431 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1380ad880 con 0x7fa1500c1c30 2024-04-25T00:18:24.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.784+0000 7fa155146640 1 -- 172.21.15.116:0/1480948431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1500c1c30 msgr2=0x7fa150071e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:24.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.784+0000 7fa155146640 1 --2- 172.21.15.116:0/1480948431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1500c1c30 0x7fa150071e70 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7fa138009870 tx=0x7fa13801c560 comp rx=0 tx=0).stop 2024-04-25T00:18:24.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.784+0000 7fa155146640 1 -- 172.21.15.116:0/1480948431 shutdown_connections 2024-04-25T00:18:24.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.784+0000 7fa155146640 1 --2- 172.21.15.116:0/1480948431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1500c1c30 0x7fa150071e70 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:24.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.784+0000 7fa155146640 1 --2- 172.21.15.116:0/1480948431 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa15006c480 0x7fa1500c3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:24.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.784+0000 7fa155146640 1 -- 172.21.15.116:0/1480948431 >> 172.21.15.116:0/1480948431 conn(0x7fa1500c4270 msgr2=0x7fa1500c4680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:24.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.784+0000 7fa155146640 1 -- 172.21.15.116:0/1480948431 shutdown_connections 2024-04-25T00:18:24.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.784+0000 7fa155146640 1 -- 172.21.15.116:0/1480948431 wait complete. 2024-04-25T00:18:24.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.788+0000 7fa155146640 1 Processor -- start 2024-04-25T00:18:24.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.788+0000 7fa155146640 1 -- start start 2024-04-25T00:18:24.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.788+0000 7fa155146640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa15006c480 0x7fa1501f25c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:24.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.788+0000 7fa155146640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa1500c1c30 0x7fa1500c0e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:24.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.788+0000 7fa155146640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa150153170 0x7fa1500a6b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:24.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.788+0000 7fa155146640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa150150190 con 0x7fa15006c480 2024-04-25T00:18:24.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.788+0000 7fa155146640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa15006ca70 con 0x7fa150153170 2024-04-25T00:18:24.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.788+0000 7fa155146640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa150150310 con 0x7fa1500c1c30 2024-04-25T00:18:24.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.788+0000 7fa14ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa15006c480 0x7fa1501f25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:24.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.788+0000 7fa14ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa15006c480 0x7fa1501f25c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50608/0 (socket says 172.21.15.116:50608) 2024-04-25T00:18:24.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.788+0000 7fa14ed9d640 1 -- 172.21.15.116:0/213447488 learned_addr learned my addr 172.21.15.116:0/213447488 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:18:24.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.788+0000 7fa147fff640 1 --2- 172.21.15.116:0/213447488 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa1500c1c30 0x7fa1500c0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:24.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.788+0000 7fa14ed9d640 1 -- 172.21.15.116:0/213447488 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa1500c1c30 msgr2=0x7fa1500c0e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:24.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.788+0000 7fa14ed9d640 1 --2- 172.21.15.116:0/213447488 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa1500c1c30 0x7fa1500c0e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:24.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.788+0000 7fa14f59e640 1 --2- 172.21.15.116:0/213447488 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa150153170 0x7fa1500a6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:24.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.788+0000 7fa14ed9d640 1 -- 172.21.15.116:0/213447488 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa150153170 msgr2=0x7fa1500a6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:24.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.788+0000 7fa14ed9d640 1 --2- 172.21.15.116:0/213447488 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa150153170 0x7fa1500a6b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:24.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.788+0000 7fa14ed9d640 1 -- 172.21.15.116:0/213447488 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1501f1b20 con 0x7fa15006c480 2024-04-25T00:18:24.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.788+0000 7fa147fff640 1 --2- 172.21.15.116:0/213447488 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa1500c1c30 0x7fa1500c0e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:18:24.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.788+0000 7fa14ed9d640 1 --2- 172.21.15.116:0/213447488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa15006c480 0x7fa1501f25c0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fa14000c820 tx=0x7fa14000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:24.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.788+0000 7fa1477fe640 1 -- 172.21.15.116:0/213447488 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa140015070 con 0x7fa15006c480 2024-04-25T00:18:24.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.788+0000 7fa1477fe640 1 -- 172.21.15.116:0/213447488 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1400079d0 con 0x7fa15006c480 2024-04-25T00:18:24.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.788+0000 7fa1477fe640 1 -- 172.21.15.116:0/213447488 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa140007dc0 con 0x7fa15006c480 2024-04-25T00:18:24.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.788+0000 7fa155146640 1 -- 172.21.15.116:0/213447488 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1501ef130 con 0x7fa15006c480 2024-04-25T00:18:24.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.788+0000 7fa155146640 1 -- 172.21.15.116:0/213447488 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1501f20a0 con 0x7fa15006c480 2024-04-25T00:18:24.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.788+0000 7fa155146640 1 -- 172.21.15.116:0/213447488 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa150068a60 con 0x7fa15006c480 2024-04-25T00:18:24.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.788+0000 7fa1477fe640 1 -- 172.21.15.116:0/213447488 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa140018070 con 0x7fa15006c480 2024-04-25T00:18:24.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.792+0000 7fa1477fe640 1 --2- 172.21.15.116:0/213447488 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa1300678c0 0x7fa130069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:24.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.792+0000 7fa1477fe640 1 -- 172.21.15.116:0/213447488 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 8902+0+0 (secure 0 0 0) 0x7fa140086690 con 0x7fa15006c480 2024-04-25T00:18:24.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.792+0000 7fa147fff640 1 --2- 172.21.15.116:0/213447488 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa1300678c0 0x7fa130069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:24.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.792+0000 7fa147fff640 1 --2- 172.21.15.116:0/213447488 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa1300678c0 0x7fa130069d80 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fa1380af000 tx=0x7fa138008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:24.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:24.796+0000 7fa1477fe640 1 -- 172.21.15.116:0/213447488 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fa140056590 con 0x7fa15006c480 2024-04-25T00:18:25.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.040+0000 7fa155146640 1 -- 172.21.15.116:0/213447488 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa150074480 con 0x7fa15006c480 2024-04-25T00:18:25.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.040+0000 7fa1477fe640 1 -- 172.21.15.116:0/213447488 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) ==== 95+0+5681 (secure 0 0 0) 0x7fa140055e90 con 0x7fa15006c480 2024-04-25T00:18:25.043 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:18:25.043 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":141,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:18:23.899096+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7362},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7362":{"gid":7362,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6833/1655959433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1655959433},{"type":"v1","addr":"172.21.15.116:6833","nonce":1655959433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7362,"qdb_cluster":[7362]},"id":1},{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:17:28.837517+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7164},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7164":{"gid":7164,"name":"c","rank":0,"incarnation":128,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6835/3336339376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3336339376},{"type":"v1","addr":"172.21.15.116:6835","nonce":3336339376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7164,"qdb_cluster":[7164]},"id":2}]} 2024-04-25T00:18:25.043 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 141 2024-04-25T00:18:25.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.044+0000 7fa155146640 1 -- 172.21.15.116:0/213447488 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa1300678c0 msgr2=0x7fa130069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:25.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.044+0000 7fa155146640 1 --2- 172.21.15.116:0/213447488 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa1300678c0 0x7fa130069d80 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fa1380af000 tx=0x7fa138008040 comp rx=0 tx=0).stop 2024-04-25T00:18:25.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.044+0000 7fa155146640 1 -- 172.21.15.116:0/213447488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa15006c480 msgr2=0x7fa1501f25c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:25.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.044+0000 7fa155146640 1 --2- 172.21.15.116:0/213447488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa15006c480 0x7fa1501f25c0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fa14000c820 tx=0x7fa14000cce0 comp rx=0 tx=0).stop 2024-04-25T00:18:25.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.044+0000 7fa155146640 1 -- 172.21.15.116:0/213447488 shutdown_connections 2024-04-25T00:18:25.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.044+0000 7fa155146640 1 --2- 172.21.15.116:0/213447488 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa1300678c0 0x7fa130069d80 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:25.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.044+0000 7fa155146640 1 --2- 172.21.15.116:0/213447488 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa150153170 0x7fa1500a6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:25.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.044+0000 7fa155146640 1 --2- 172.21.15.116:0/213447488 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa1500c1c30 0x7fa1500c0e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:25.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.044+0000 7fa155146640 1 --2- 172.21.15.116:0/213447488 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa15006c480 0x7fa1501f25c0 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:25.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.044+0000 7fa155146640 1 -- 172.21.15.116:0/213447488 >> 172.21.15.116:0/213447488 conn(0x7fa1500c4270 msgr2=0x7fa1500c4650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:25.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.044+0000 7fa155146640 1 -- 172.21.15.116:0/213447488 shutdown_connections 2024-04-25T00:18:25.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.044+0000 7fa155146640 1 -- 172.21.15.116:0/213447488 wait complete. 2024-04-25T00:18:25.061 INFO:tasks.mds_thrash.fs.[a]:mds.a has gained rank=0, replacing gid=6343 2024-04-25T00:18:25.061 INFO:tasks.mds_thrash.fs.[a]:waiting for 0 secs before reviving mds.d 2024-04-25T00:18:25.061 INFO:tasks.mds_thrash.fs.[a]:reviving mds.d 2024-04-25T00:18:25.061 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-25T00:18:25.062 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-25T00:18:25.064 INFO:tasks.ceph.mds.d:Started 2024-04-25T00:18:25.064 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:18:25.142 INFO:tasks.ceph.mds.d.smithi174.stdout:starting mds.d at 2024-04-25T00:18:25.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.192+0000 7fe88c829640 1 Processor -- start 2024-04-25T00:18:25.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.192+0000 7fe88c829640 1 -- start start 2024-04-25T00:18:25.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.196+0000 7fe88c829640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe88806c7b0 0x7fe888155ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:25.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.196+0000 7fe88c829640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe888071520 0x7fe888155ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:25.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.196+0000 7fe88c829640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe888073830 con 0x7fe88814fc90 2024-04-25T00:18:25.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.196+0000 7fe88c829640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fe88806c320 con 0x7fe88806c7b0 2024-04-25T00:18:25.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.196+0000 7fe88c829640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe888072810 con 0x7fe888071520 2024-04-25T00:18:25.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.196+0000 7fe87ffff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe88806c7b0 0x7fe888155ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:25.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.196+0000 7fe87ffff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe88806c7b0 0x7fe888155ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50618/0 (socket says 172.21.15.116:50618) 2024-04-25T00:18:25.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.196+0000 7fe88659c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fe88814fc90 0x7fe888073120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54630/0 (socket says 172.21.15.116:54630) 2024-04-25T00:18:25.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.196+0000 7fe87ffff640 1 -- 172.21.15.116:0/2836497545 learned_addr learned my addr 172.21.15.116:0/2836497545 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:18:25.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.196+0000 7fe886d9d640 1 --2- 172.21.15.116:0/2836497545 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe888071520 0x7fe888155ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:25.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.196+0000 7fe87ffff640 1 -- 172.21.15.116:0/2836497545 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe888071520 msgr2=0x7fe888155ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:25.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.196+0000 7fe87ffff640 1 --2- 172.21.15.116:0/2836497545 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe888071520 0x7fe888155ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:25.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.196+0000 7fe87ffff640 1 -- 172.21.15.116:0/2836497545 >> v1:172.21.15.116:6789/0 conn(0x7fe88814fc90 legacy=0x7fe888073120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:25.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.196+0000 7fe87ffff640 1 -- 172.21.15.116:0/2836497545 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8880a8a70 con 0x7fe88806c7b0 2024-04-25T00:18:25.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.196+0000 7fe886d9d640 1 --2- 172.21.15.116:0/2836497545 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe888071520 0x7fe888155ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:18:25.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.196+0000 7fe87ffff640 1 --2- 172.21.15.116:0/2836497545 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe88806c7b0 0x7fe888155ab0 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7fe870009870 tx=0x7fe87001c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a617f719c662316e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:25.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.196+0000 7fe885d9b640 1 -- 172.21.15.116:0/2836497545 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8700a7070 con 0x7fe88806c7b0 2024-04-25T00:18:25.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.196+0000 7fe885d9b640 1 -- 172.21.15.116:0/2836497545 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8700b0070 con 0x7fe88806c7b0 2024-04-25T00:18:25.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.196+0000 7fe885d9b640 1 -- 172.21.15.116:0/2836497545 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8700ab890 con 0x7fe88806c7b0 2024-04-25T00:18:25.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.196+0000 7fe88c829640 1 -- 172.21.15.116:0/2836497545 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe88806c7b0 msgr2=0x7fe888155ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:25.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.196+0000 7fe88c829640 1 --2- 172.21.15.116:0/2836497545 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe88806c7b0 0x7fe888155ab0 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7fe870009870 tx=0x7fe87001c560 comp rx=0 tx=0).stop 2024-04-25T00:18:25.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.196+0000 7fe88c829640 1 -- 172.21.15.116:0/2836497545 shutdown_connections 2024-04-25T00:18:25.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.196+0000 7fe88c829640 1 --2- 172.21.15.116:0/2836497545 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe888071520 0x7fe888155ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:25.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.196+0000 7fe88c829640 1 --2- 172.21.15.116:0/2836497545 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe88806c7b0 0x7fe888155ab0 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:25.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.196+0000 7fe88c829640 1 -- 172.21.15.116:0/2836497545 >> 172.21.15.116:0/2836497545 conn(0x7fe8880c4200 msgr2=0x7fe8880c4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:25.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.196+0000 7fe88c829640 1 -- 172.21.15.116:0/2836497545 shutdown_connections 2024-04-25T00:18:25.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.196+0000 7fe88c829640 1 -- 172.21.15.116:0/2836497545 wait complete. 2024-04-25T00:18:25.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.200+0000 7fe88c829640 1 Processor -- start 2024-04-25T00:18:25.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.200+0000 7fe88c829640 1 -- start start 2024-04-25T00:18:25.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.200+0000 7fe88c829640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe88806c7b0 0x7fe8881f0220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:25.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.200+0000 7fe88c829640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe888071520 0x7fe8880c65a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:25.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.200+0000 7fe88c829640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe88814fc90 0x7fe888146e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:25.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.200+0000 7fe88c829640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe8880c3bb0 con 0x7fe88814fc90 2024-04-25T00:18:25.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.200+0000 7fe88c829640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe888156530 con 0x7fe88806c7b0 2024-04-25T00:18:25.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.200+0000 7fe88c829640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe8880c3d30 con 0x7fe888071520 2024-04-25T00:18:25.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.200+0000 7fe886d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe88814fc90 0x7fe888146e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:25.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.200+0000 7fe886d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe88814fc90 0x7fe888146e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50626/0 (socket says 172.21.15.116:50626) 2024-04-25T00:18:25.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.200+0000 7fe886d9d640 1 -- 172.21.15.116:0/275586138 learned_addr learned my addr 172.21.15.116:0/275586138 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:18:25.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.200+0000 7fe88659c640 1 --2- 172.21.15.116:0/275586138 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe88806c7b0 0x7fe8881f0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:25.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.200+0000 7fe886d9d640 1 -- 172.21.15.116:0/275586138 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe888071520 msgr2=0x7fe8880c65a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:25.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.200+0000 7fe87ffff640 1 --2- 172.21.15.116:0/275586138 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe888071520 0x7fe8880c65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:25.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.200+0000 7fe886d9d640 1 --2- 172.21.15.116:0/275586138 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe888071520 0x7fe8880c65a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:25.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.200+0000 7fe886d9d640 1 -- 172.21.15.116:0/275586138 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe88806c7b0 msgr2=0x7fe8881f0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:25.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.200+0000 7fe886d9d640 1 --2- 172.21.15.116:0/275586138 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe88806c7b0 0x7fe8881f0220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:25.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.200+0000 7fe886d9d640 1 -- 172.21.15.116:0/275586138 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe888146900 con 0x7fe88814fc90 2024-04-25T00:18:25.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.200+0000 7fe88659c640 1 --2- 172.21.15.116:0/275586138 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe88806c7b0 0x7fe8881f0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:18:25.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.200+0000 7fe87ffff640 1 --2- 172.21.15.116:0/275586138 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe888071520 0x7fe8880c65a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:18:25.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.200+0000 7fe886d9d640 1 --2- 172.21.15.116:0/275586138 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe88814fc90 0x7fe888146e00 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7fe88000cef0 tx=0x7fe88000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:25.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.200+0000 7fe87effd640 1 -- 172.21.15.116:0/275586138 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe88001a070 con 0x7fe88814fc90 2024-04-25T00:18:25.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.200+0000 7fe87effd640 1 -- 172.21.15.116:0/275586138 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe88000d860 con 0x7fe88814fc90 2024-04-25T00:18:25.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.200+0000 7fe88c829640 1 -- 172.21.15.116:0/275586138 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe888152c70 con 0x7fe88814fc90 2024-04-25T00:18:25.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.200+0000 7fe87effd640 1 -- 172.21.15.116:0/275586138 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe88000dc50 con 0x7fe88814fc90 2024-04-25T00:18:25.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.200+0000 7fe88c829640 1 -- 172.21.15.116:0/275586138 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe888153230 con 0x7fe88814fc90 2024-04-25T00:18:25.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.204+0000 7fe88c829640 1 -- 172.21.15.116:0/275586138 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe888068a60 con 0x7fe88814fc90 2024-04-25T00:18:25.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.204+0000 7fe87effd640 1 -- 172.21.15.116:0/275586138 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe88001d070 con 0x7fe88814fc90 2024-04-25T00:18:25.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.204+0000 7fe87effd640 1 --2- 172.21.15.116:0/275586138 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe8680678f0 0x7fe868069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:25.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.204+0000 7fe87effd640 1 -- 172.21.15.116:0/275586138 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 8902+0+0 (secure 0 0 0) 0x7fe880087130 con 0x7fe88814fc90 2024-04-25T00:18:25.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.204+0000 7fe88659c640 1 --2- 172.21.15.116:0/275586138 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe8680678f0 0x7fe868069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:25.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.204+0000 7fe88659c640 1 --2- 172.21.15.116:0/275586138 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe8680678f0 0x7fe868069db0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fe878007fd0 tx=0x7fe878009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:25.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.208+0000 7fe87effd640 1 -- 172.21.15.116:0/275586138 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fe8800570f0 con 0x7fe88814fc90 2024-04-25T00:18:25.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.444+0000 7fe88c829640 1 -- 172.21.15.116:0/275586138 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe8880c1a10 con 0x7fe88814fc90 2024-04-25T00:18:25.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.444+0000 7fe87effd640 1 -- 172.21.15.116:0/275586138 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) ==== 95+0+5681 (secure 0 0 0) 0x7fe8800569f0 con 0x7fe88814fc90 2024-04-25T00:18:25.446 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:18:25.446 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":141,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:18:23.899096+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7362},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7362":{"gid":7362,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6833/1655959433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1655959433},{"type":"v1","addr":"172.21.15.116:6833","nonce":1655959433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7362,"qdb_cluster":[7362]},"id":1},{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:17:28.837517+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7164},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7164":{"gid":7164,"name":"c","rank":0,"incarnation":128,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6835/3336339376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3336339376},{"type":"v1","addr":"172.21.15.116:6835","nonce":3336339376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7164,"qdb_cluster":[7164]},"id":2}]} 2024-04-25T00:18:25.446 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 141 2024-04-25T00:18:25.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.448+0000 7fe88c829640 1 -- 172.21.15.116:0/275586138 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe8680678f0 msgr2=0x7fe868069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:25.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.448+0000 7fe88c829640 1 --2- 172.21.15.116:0/275586138 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe8680678f0 0x7fe868069db0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fe878007fd0 tx=0x7fe878009040 comp rx=0 tx=0).stop 2024-04-25T00:18:25.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.448+0000 7fe88c829640 1 -- 172.21.15.116:0/275586138 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe88814fc90 msgr2=0x7fe888146e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:25.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.448+0000 7fe88c829640 1 --2- 172.21.15.116:0/275586138 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe88814fc90 0x7fe888146e00 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7fe88000cef0 tx=0x7fe88000a710 comp rx=0 tx=0).stop 2024-04-25T00:18:25.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.448+0000 7fe88c829640 1 -- 172.21.15.116:0/275586138 shutdown_connections 2024-04-25T00:18:25.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.448+0000 7fe88c829640 1 --2- 172.21.15.116:0/275586138 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe8680678f0 0x7fe868069db0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:25.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.448+0000 7fe88c829640 1 --2- 172.21.15.116:0/275586138 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe88814fc90 0x7fe888146e00 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:25.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.448+0000 7fe88c829640 1 --2- 172.21.15.116:0/275586138 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe888071520 0x7fe8880c65a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:25.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.448+0000 7fe88c829640 1 --2- 172.21.15.116:0/275586138 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe88806c7b0 0x7fe8881f0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:25.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.448+0000 7fe88c829640 1 -- 172.21.15.116:0/275586138 >> 172.21.15.116:0/275586138 conn(0x7fe8880c4200 msgr2=0x7fe88806b210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:25.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.448+0000 7fe88c829640 1 -- 172.21.15.116:0/275586138 shutdown_connections 2024-04-25T00:18:25.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:25.448+0000 7fe88c829640 1 -- 172.21.15.116:0/275586138 wait complete. 2024-04-25T00:18:25.464 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2024-04-25T00:18:26.111 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:18:26.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.248+0000 7f3b0b27b640 1 Processor -- start 2024-04-25T00:18:26.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.248+0000 7f3b0b27b640 1 -- start start 2024-04-25T00:18:26.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.248+0000 7f3b0b27b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3b04150d20 0x7f3b04153620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:26.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.248+0000 7f3b0b27b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3b04071050 0x7f3b040714c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:26.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.252+0000 7f3b0b27b640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b040c36a0 con 0x7f3b040c3240 2024-04-25T00:18:26.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.252+0000 7f3b0b27b640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f3b0406fba0 con 0x7f3b04150d20 2024-04-25T00:18:26.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.252+0000 7f3b0b27b640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f3b0406f8c0 con 0x7f3b04071050 2024-04-25T00:18:26.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.252+0000 7f3b09818640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3b04071050 0x7f3b040714c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:26.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.252+0000 7f3b09818640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3b04071050 0x7f3b040714c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:59950/0 (socket says 172.21.15.116:59950) 2024-04-25T00:18:26.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.252+0000 7f3b09818640 1 -- 172.21.15.116:0/4210388532 learned_addr learned my addr 172.21.15.116:0/4210388532 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:18:26.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.252+0000 7f3b09818640 1 -- 172.21.15.116:0/4210388532 >> v2:172.21.15.116:3300/0 conn(0x7f3b04150d20 msgr2=0x7f3b04153620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:26.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.252+0000 7f3b09818640 1 --2- 172.21.15.116:0/4210388532 >> v2:172.21.15.116:3300/0 conn(0x7f3b04150d20 0x7f3b04153620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:26.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.252+0000 7f3b09818640 1 -- 172.21.15.116:0/4210388532 >> v1:172.21.15.116:6789/0 conn(0x7f3b040c3240 legacy=0x7f3b040c3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:26.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.252+0000 7f3b09818640 1 -- 172.21.15.116:0/4210388532 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b040c5420 con 0x7f3b04071050 2024-04-25T00:18:26.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.256+0000 7f3b09818640 1 --2- 172.21.15.116:0/4210388532 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3b04071050 0x7f3b040714c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3afc009e50 tx=0x7f3afc01c9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e5f9430b07a22c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:26.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.256+0000 7f3afbfff640 1 -- 172.21.15.116:0/4210388532 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3afc0a5780 con 0x7f3b04071050 2024-04-25T00:18:26.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.256+0000 7f3afbfff640 1 -- 172.21.15.116:0/4210388532 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3afc0a5920 con 0x7f3b04071050 2024-04-25T00:18:26.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.256+0000 7f3afbfff640 1 -- 172.21.15.116:0/4210388532 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3afc0a5cf0 con 0x7f3b04071050 2024-04-25T00:18:26.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.256+0000 7f3b0b27b640 1 -- 172.21.15.116:0/4210388532 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3b04071050 msgr2=0x7f3b040714c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:26.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.256+0000 7f3b0b27b640 1 --2- 172.21.15.116:0/4210388532 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3b04071050 0x7f3b040714c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3afc009e50 tx=0x7f3afc01c9f0 comp rx=0 tx=0).stop 2024-04-25T00:18:26.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.256+0000 7f3b0b27b640 1 -- 172.21.15.116:0/4210388532 shutdown_connections 2024-04-25T00:18:26.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.256+0000 7f3b0b27b640 1 --2- 172.21.15.116:0/4210388532 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3b04071050 0x7f3b040714c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:26.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.256+0000 7f3b0b27b640 1 --2- 172.21.15.116:0/4210388532 >> v2:172.21.15.116:3300/0 conn(0x7f3b04150d20 0x7f3b04153620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:26.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.256+0000 7f3b0b27b640 1 -- 172.21.15.116:0/4210388532 >> 172.21.15.116:0/4210388532 conn(0x7f3b040c9160 msgr2=0x7f3b040c9570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:26.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.256+0000 7f3b0b27b640 1 -- 172.21.15.116:0/4210388532 shutdown_connections 2024-04-25T00:18:26.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.256+0000 7f3b0b27b640 1 -- 172.21.15.116:0/4210388532 wait complete. 2024-04-25T00:18:26.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.256+0000 7f3b0b27b640 1 Processor -- start 2024-04-25T00:18:26.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.256+0000 7f3b0b27b640 1 -- start start 2024-04-25T00:18:26.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.260+0000 7f3b0b27b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3b04071050 0x7f3b041ee560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:26.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.260+0000 7f3b0b27b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3b040c3240 0x7f3b0414eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:26.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.260+0000 7f3b0b27b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b04150d20 0x7f3b0414f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:26.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.260+0000 7f3b0b27b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3b0406f640 con 0x7f3b04150d20 2024-04-25T00:18:26.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.260+0000 7f3b0b27b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3b0406f4c0 con 0x7f3b040c3240 2024-04-25T00:18:26.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.260+0000 7f3b0b27b640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f3b04151100 con 0x7f3b04071050 2024-04-25T00:18:26.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.260+0000 7f3b09818640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b04150d20 0x7f3b0414f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:26.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.260+0000 7f3b09818640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b04150d20 0x7f3b0414f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50634/0 (socket says 172.21.15.116:50634) 2024-04-25T00:18:26.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.260+0000 7f3b09818640 1 -- 172.21.15.116:0/4196297153 learned_addr learned my addr 172.21.15.116:0/4196297153 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:18:26.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.260+0000 7f3b08816640 1 --2- 172.21.15.116:0/4196297153 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3b040c3240 0x7f3b0414eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:26.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.260+0000 7f3b09017640 1 --2- 172.21.15.116:0/4196297153 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3b04071050 0x7f3b041ee560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:26.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.260+0000 7f3b09818640 1 -- 172.21.15.116:0/4196297153 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3b04071050 msgr2=0x7f3b041ee560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:26.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.260+0000 7f3b09818640 1 --2- 172.21.15.116:0/4196297153 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3b04071050 0x7f3b041ee560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:26.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.260+0000 7f3b09818640 1 -- 172.21.15.116:0/4196297153 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3b040c3240 msgr2=0x7f3b0414eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:26.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.260+0000 7f3b09818640 1 --2- 172.21.15.116:0/4196297153 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3b040c3240 0x7f3b0414eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:26.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.260+0000 7f3b09818640 1 -- 172.21.15.116:0/4196297153 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b0414dc30 con 0x7f3b04150d20 2024-04-25T00:18:26.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.260+0000 7f3b09017640 1 --2- 172.21.15.116:0/4196297153 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3b04071050 0x7f3b041ee560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:18:26.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.260+0000 7f3b09818640 1 --2- 172.21.15.116:0/4196297153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b04150d20 0x7f3b0414f0e0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f3afc0af040 tx=0x7f3afc019790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:26.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.260+0000 7f3af9ffb640 1 -- 172.21.15.116:0/4196297153 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3afc0bc020 con 0x7f3b04150d20 2024-04-25T00:18:26.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.260+0000 7f3af9ffb640 1 -- 172.21.15.116:0/4196297153 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3afc0b8670 con 0x7f3b04150d20 2024-04-25T00:18:26.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.260+0000 7f3af9ffb640 1 -- 172.21.15.116:0/4196297153 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3afc0b8a40 con 0x7f3b04150d20 2024-04-25T00:18:26.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.260+0000 7f3b0b27b640 1 -- 172.21.15.116:0/4196297153 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b041eeba0 con 0x7f3b04150d20 2024-04-25T00:18:26.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.260+0000 7f3b0b27b640 1 -- 172.21.15.116:0/4196297153 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b0414b780 con 0x7f3b04150d20 2024-04-25T00:18:26.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.260+0000 7f3b0b27b640 1 -- 172.21.15.116:0/4196297153 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b04068a60 con 0x7f3b04150d20 2024-04-25T00:18:26.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.264+0000 7f3af9ffb640 1 -- 172.21.15.116:0/4196297153 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3afc0bf020 con 0x7f3b04150d20 2024-04-25T00:18:26.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.264+0000 7f3af9ffb640 1 --2- 172.21.15.116:0/4196297153 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3ae80678c0 0x7f3ae8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:26.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.264+0000 7f3af9ffb640 1 -- 172.21.15.116:0/4196297153 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 8902+0+0 (secure 0 0 0) 0x7f3afc121960 con 0x7f3b04150d20 2024-04-25T00:18:26.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.264+0000 7f3b09017640 1 --2- 172.21.15.116:0/4196297153 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3ae80678c0 0x7f3ae8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:26.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.264+0000 7f3b09017640 1 --2- 172.21.15.116:0/4196297153 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3ae80678c0 0x7f3ae8069d80 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f3b0000a8b0 tx=0x7f3b00008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:26.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.268+0000 7f3af9ffb640 1 -- 172.21.15.116:0/4196297153 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f3afc0c6030 con 0x7f3b04150d20 2024-04-25T00:18:26.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.500+0000 7f3b0b27b640 1 -- 172.21.15.116:0/4196297153 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3b0414cfd0 con 0x7f3b04150d20 2024-04-25T00:18:26.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.504+0000 7f3af9ffb640 1 -- 172.21.15.116:0/4196297153 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 142 v142) ==== 95+0+6510 (secure 0 0 0) 0x7f3afc0165e0 con 0x7f3b04150d20 2024-04-25T00:18:26.504 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:18:26.505 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":142,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":5879,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/2276529656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2276529656},{"type":"v1","addr":"172.21.15.174:6837","nonce":2276529656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":142}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:18:23.899096+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7362},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7362":{"gid":7362,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6833/1655959433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1655959433},{"type":"v1","addr":"172.21.15.116:6833","nonce":1655959433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7362,"qdb_cluster":[7362]},"id":1},{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:17:28.837517+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7164},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7164":{"gid":7164,"name":"c","rank":0,"incarnation":128,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6835/3336339376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3336339376},{"type":"v1","addr":"172.21.15.116:6835","nonce":3336339376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7164,"qdb_cluster":[7164]},"id":2}]} 2024-04-25T00:18:26.505 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 142 2024-04-25T00:18:26.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.504+0000 7f3b0b27b640 1 -- 172.21.15.116:0/4196297153 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3ae80678c0 msgr2=0x7f3ae8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:26.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.504+0000 7f3b0b27b640 1 --2- 172.21.15.116:0/4196297153 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3ae80678c0 0x7f3ae8069d80 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f3b0000a8b0 tx=0x7f3b00008040 comp rx=0 tx=0).stop 2024-04-25T00:18:26.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.504+0000 7f3b0b27b640 1 -- 172.21.15.116:0/4196297153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b04150d20 msgr2=0x7f3b0414f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:26.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.504+0000 7f3b0b27b640 1 --2- 172.21.15.116:0/4196297153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b04150d20 0x7f3b0414f0e0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f3afc0af040 tx=0x7f3afc019790 comp rx=0 tx=0).stop 2024-04-25T00:18:26.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.504+0000 7f3b0b27b640 1 -- 172.21.15.116:0/4196297153 shutdown_connections 2024-04-25T00:18:26.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.504+0000 7f3b0b27b640 1 --2- 172.21.15.116:0/4196297153 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3ae80678c0 0x7f3ae8069d80 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:26.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.504+0000 7f3b0b27b640 1 --2- 172.21.15.116:0/4196297153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3b04150d20 0x7f3b0414f0e0 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:26.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.504+0000 7f3b0b27b640 1 --2- 172.21.15.116:0/4196297153 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3b040c3240 0x7f3b0414eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:26.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.504+0000 7f3b0b27b640 1 --2- 172.21.15.116:0/4196297153 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3b04071050 0x7f3b041ee560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:26.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.504+0000 7f3b0b27b640 1 -- 172.21.15.116:0/4196297153 >> 172.21.15.116:0/4196297153 conn(0x7f3b040c9160 msgr2=0x7f3b04145e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:26.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.504+0000 7f3b0b27b640 1 -- 172.21.15.116:0/4196297153 shutdown_connections 2024-04-25T00:18:26.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:26.504+0000 7f3b0b27b640 1 -- 172.21.15.116:0/4196297153 wait complete. 2024-04-25T00:18:26.527 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:18:27.466 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:18:27.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.592+0000 7f59b213f640 1 Processor -- start 2024-04-25T00:18:27.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.592+0000 7f59b213f640 1 -- start start 2024-04-25T00:18:27.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.596+0000 7f59b213f640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f59ac071520 0x7f59ac155fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:27.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.596+0000 7f59b213f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f59ac14f7a0 0x7f59ac14fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:27.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.596+0000 7f59b213f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59ac0737d0 con 0x7f59ac155c00 2024-04-25T00:18:27.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.596+0000 7f59b213f640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f59ac071bb0 con 0x7f59ac071520 2024-04-25T00:18:27.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.596+0000 7f59b213f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f59ac072ee0 con 0x7f59ac14f7a0 2024-04-25T00:18:27.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.596+0000 7f59ab7fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f59ac155c00 0x7f59ac0730c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54648/0 (socket says 172.21.15.116:54648) 2024-04-25T00:18:27.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.596+0000 7f59ab7fe640 1 -- 172.21.15.116:0/1940240183 learned_addr learned my addr 172.21.15.116:0/1940240183 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:18:27.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.596+0000 7f59aaffd640 1 -- 172.21.15.116:0/1940240183 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2837036037 0 0) 0x7f59ac0737d0 con 0x7f59ac155c00 2024-04-25T00:18:27.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.596+0000 7f59a3fff640 1 --2- 172.21.15.116:0/1940240183 >> v2:172.21.15.174:3300/0 conn(0x7f59ac071520 0x7f59ac155fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:27.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.596+0000 7f59aaffd640 1 -- 172.21.15.116:0/1940240183 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5998005740 con 0x7f59ac155c00 2024-04-25T00:18:27.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.596+0000 7f59abfff640 1 --2- 172.21.15.116:0/1940240183 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f59ac14f7a0 0x7f59ac14fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:27.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.596+0000 7f59aaffd640 1 -- 172.21.15.116:0/1940240183 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 715398997 0 0) 0x7f5998005740 con 0x7f59ac155c00 2024-04-25T00:18:27.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.596+0000 7f59aaffd640 1 -- 172.21.15.116:0/1940240183 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f59ac14f7a0 msgr2=0x7f59ac14fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:27.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.596+0000 7f59aaffd640 1 --2- 172.21.15.116:0/1940240183 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f59ac14f7a0 0x7f59ac14fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:27.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.596+0000 7f59aaffd640 1 -- 172.21.15.116:0/1940240183 >> v2:172.21.15.174:3300/0 conn(0x7f59ac071520 msgr2=0x7f59ac155fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:27.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.596+0000 7f59aaffd640 1 --2- 172.21.15.116:0/1940240183 >> v2:172.21.15.174:3300/0 conn(0x7f59ac071520 0x7f59ac155fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:27.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.596+0000 7f59aaffd640 1 -- 172.21.15.116:0/1940240183 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59ac072920 con 0x7f59ac155c00 2024-04-25T00:18:27.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.596+0000 7f59aaffd640 1 -- 172.21.15.116:0/1940240183 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f599c001680 con 0x7f59ac155c00 2024-04-25T00:18:27.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.596+0000 7f59aaffd640 1 -- 172.21.15.116:0/1940240183 >> v1:172.21.15.116:6789/0 conn(0x7f59ac155c00 legacy=0x7f59ac0730c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:27.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.596+0000 7f59aaffd640 1 --2- 172.21.15.116:0/1940240183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5998005bd0 0x7f5998005ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:27.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.596+0000 7f59aaffd640 1 --2- 172.21.15.116:0/1940240183 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5998006a20 0x7f5998006ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:27.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.596+0000 7f59aaffd640 1 --2- 172.21.15.116:0/1940240183 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59980073e0 0x7f599800dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:27.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.596+0000 7f59aaffd640 1 -- 172.21.15.116:0/1940240183 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f59ac071bb0 con 0x7f5998005bd0 2024-04-25T00:18:27.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.596+0000 7f59aaffd640 1 -- 172.21.15.116:0/1940240183 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f59ac072ee0 con 0x7f59980073e0 2024-04-25T00:18:27.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.596+0000 7f59aaffd640 1 -- 172.21.15.116:0/1940240183 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f599800e360 con 0x7f5998006a20 2024-04-25T00:18:27.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.596+0000 7f59a3fff640 1 --2- 172.21.15.116:0/1940240183 >> v2:172.21.15.174:3300/0 conn(0x7f59ac071520 0x7f59ac155fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:18:27.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.596+0000 7f59abfff640 1 --2- 172.21.15.116:0/1940240183 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f59ac14f7a0 0x7f59ac14fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:18:27.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.596+0000 7f59ab7fe640 1 --2- 172.21.15.116:0/1940240183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5998005bd0 0x7f5998005ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:27.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.596+0000 7f59ab7fe640 1 -- 172.21.15.116:0/1940240183 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5998006a20 msgr2=0x7f5998006ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:27.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.596+0000 7f59ab7fe640 1 --2- 172.21.15.116:0/1940240183 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5998006a20 0x7f5998006ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:27.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.596+0000 7f59ab7fe640 1 -- 172.21.15.116:0/1940240183 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59980073e0 msgr2=0x7f599800dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:27.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.596+0000 7f59ab7fe640 1 --2- 172.21.15.116:0/1940240183 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59980073e0 0x7f599800dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:27.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.596+0000 7f59ab7fe640 1 -- 172.21.15.116:0/1940240183 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5998005740 con 0x7f5998005bd0 2024-04-25T00:18:27.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.596+0000 7f59ab7fe640 1 --2- 172.21.15.116:0/1940240183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5998005bd0 0x7f5998005ff0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f599c00c210 tx=0x7f599c01cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c66a975be646eef6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:27.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.600+0000 7f59aaffd640 1 -- 172.21.15.116:0/1940240183 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f599c007d40 con 0x7f5998005bd0 2024-04-25T00:18:27.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.600+0000 7f59aaffd640 1 -- 172.21.15.116:0/1940240183 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f599c0a9070 con 0x7f5998005bd0 2024-04-25T00:18:27.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.600+0000 7f59aaffd640 1 -- 172.21.15.116:0/1940240183 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f599c005d40 con 0x7f5998005bd0 2024-04-25T00:18:27.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.600+0000 7f59b213f640 1 -- 172.21.15.116:0/1940240183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5998005bd0 msgr2=0x7f5998005ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:27.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.600+0000 7f59b213f640 1 --2- 172.21.15.116:0/1940240183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5998005bd0 0x7f5998005ff0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f599c00c210 tx=0x7f599c01cca0 comp rx=0 tx=0).stop 2024-04-25T00:18:27.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.600+0000 7f59abfff640 1 -- 172.21.15.116:0/1940240183 reap_dead start 2024-04-25T00:18:27.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.600+0000 7f59b213f640 1 -- 172.21.15.116:0/1940240183 shutdown_connections 2024-04-25T00:18:27.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.600+0000 7f59b213f640 1 -- 172.21.15.116:0/1940240183 >> 172.21.15.116:0/1940240183 conn(0x7f59ac0c4200 msgr2=0x7f59ac0c4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:27.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.600+0000 7f59b213f640 1 -- 172.21.15.116:0/1940240183 shutdown_connections 2024-04-25T00:18:27.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.600+0000 7f59b213f640 1 -- 172.21.15.116:0/1940240183 wait complete. 2024-04-25T00:18:27.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.600+0000 7f59b213f640 1 Processor -- start 2024-04-25T00:18:27.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.600+0000 7f59b213f640 1 -- start start 2024-04-25T00:18:27.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.600+0000 7f59b213f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5998005bd0 0x7f59ac14ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:27.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.600+0000 7f59b213f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59ac150520 0x7f59ac1489c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:27.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.600+0000 7f59b213f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f59ac148f00 0x7f59ac0ac8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:27.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.600+0000 7f59b213f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f59ac14ffc0 con 0x7f59ac150520 2024-04-25T00:18:27.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.600+0000 7f59b213f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f59ac0a7520 con 0x7f5998005bd0 2024-04-25T00:18:27.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.600+0000 7f59b213f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f59ac0c2100 con 0x7f59ac148f00 2024-04-25T00:18:27.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.600+0000 7f59a3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59ac150520 0x7f59ac1489c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:27.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.600+0000 7f59a3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59ac150520 0x7f59ac1489c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50664/0 (socket says 172.21.15.116:50664) 2024-04-25T00:18:27.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.600+0000 7f59a3fff640 1 -- 172.21.15.116:0/2266480035 learned_addr learned my addr 172.21.15.116:0/2266480035 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:18:27.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.600+0000 7f59ab7fe640 1 --2- 172.21.15.116:0/2266480035 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5998005bd0 0x7f59ac14ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:27.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.600+0000 7f59abfff640 1 --2- 172.21.15.116:0/2266480035 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f59ac148f00 0x7f59ac0ac8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:27.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.600+0000 7f59a3fff640 1 -- 172.21.15.116:0/2266480035 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f59ac148f00 msgr2=0x7f59ac0ac8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:27.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.600+0000 7f59a3fff640 1 --2- 172.21.15.116:0/2266480035 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f59ac148f00 0x7f59ac0ac8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:27.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.600+0000 7f59a3fff640 1 -- 172.21.15.116:0/2266480035 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5998005bd0 msgr2=0x7f59ac14ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:27.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.600+0000 7f59a3fff640 1 --2- 172.21.15.116:0/2266480035 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5998005bd0 0x7f59ac14ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:27.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.600+0000 7f59a3fff640 1 -- 172.21.15.116:0/2266480035 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59ac1f28b0 con 0x7f59ac150520 2024-04-25T00:18:27.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.600+0000 7f59a3fff640 1 --2- 172.21.15.116:0/2266480035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59ac150520 0x7f59ac1489c0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f599400e840 tx=0x7f599400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:27.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.600+0000 7f59a8ff9640 1 -- 172.21.15.116:0/2266480035 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5994017070 con 0x7f59ac150520 2024-04-25T00:18:27.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.600+0000 7f59a8ff9640 1 -- 172.21.15.116:0/2266480035 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f599400c870 con 0x7f59ac150520 2024-04-25T00:18:27.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.600+0000 7f59a8ff9640 1 -- 172.21.15.116:0/2266480035 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f599400f040 con 0x7f59ac150520 2024-04-25T00:18:27.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.604+0000 7f59b213f640 1 -- 172.21.15.116:0/2266480035 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59ac1f2b50 con 0x7f59ac150520 2024-04-25T00:18:27.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.604+0000 7f59b213f640 1 -- 172.21.15.116:0/2266480035 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59ac0ac0e0 con 0x7f59ac150520 2024-04-25T00:18:27.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.604+0000 7f59b213f640 1 -- 172.21.15.116:0/2266480035 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59ac068a60 con 0x7f59ac150520 2024-04-25T00:18:27.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.604+0000 7f59a8ff9640 1 -- 172.21.15.116:0/2266480035 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f599401a070 con 0x7f59ac150520 2024-04-25T00:18:27.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.604+0000 7f59a8ff9640 1 --2- 172.21.15.116:0/2266480035 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f598c0678c0 0x7f598c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:27.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.604+0000 7f59a8ff9640 1 -- 172.21.15.116:0/2266480035 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 8902+0+0 (secure 0 0 0) 0x7f5994086520 con 0x7f59ac150520 2024-04-25T00:18:27.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.604+0000 7f59ab7fe640 1 --2- 172.21.15.116:0/2266480035 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f598c0678c0 0x7f598c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:27.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.604+0000 7f59ab7fe640 1 --2- 172.21.15.116:0/2266480035 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f598c0678c0 0x7f598c069d80 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f599c0ab000 tx=0x7f599c0b1040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:27.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.608+0000 7f59a8ff9640 1 -- 172.21.15.116:0/2266480035 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f5994056390 con 0x7f59ac150520 2024-04-25T00:18:27.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.836+0000 7f59b213f640 1 -- 172.21.15.116:0/2266480035 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f59ac156870 con 0x7f59ac150520 2024-04-25T00:18:27.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.836+0000 7f59a8ff9640 1 -- 172.21.15.116:0/2266480035 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 142 v142) ==== 95+0+6510 (secure 0 0 0) 0x7f5994055c90 con 0x7f59ac150520 2024-04-25T00:18:27.837 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:18:27.837 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":142,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":5879,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/2276529656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2276529656},{"type":"v1","addr":"172.21.15.174:6837","nonce":2276529656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":142}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:18:23.899096+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7362},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7362":{"gid":7362,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6833/1655959433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1655959433},{"type":"v1","addr":"172.21.15.116:6833","nonce":1655959433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7362,"qdb_cluster":[7362]},"id":1},{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:17:28.837517+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7164},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7164":{"gid":7164,"name":"c","rank":0,"incarnation":128,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6835/3336339376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3336339376},{"type":"v1","addr":"172.21.15.116:6835","nonce":3336339376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7164,"qdb_cluster":[7164]},"id":2}]} 2024-04-25T00:18:27.837 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 142 2024-04-25T00:18:27.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.836+0000 7f59b213f640 1 -- 172.21.15.116:0/2266480035 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f598c0678c0 msgr2=0x7f598c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:27.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.836+0000 7f59b213f640 1 --2- 172.21.15.116:0/2266480035 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f598c0678c0 0x7f598c069d80 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f599c0ab000 tx=0x7f599c0b1040 comp rx=0 tx=0).stop 2024-04-25T00:18:27.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.836+0000 7f59b213f640 1 -- 172.21.15.116:0/2266480035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59ac150520 msgr2=0x7f59ac1489c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:27.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.836+0000 7f59b213f640 1 --2- 172.21.15.116:0/2266480035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59ac150520 0x7f59ac1489c0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f599400e840 tx=0x7f599400ed00 comp rx=0 tx=0).stop 2024-04-25T00:18:27.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.836+0000 7f59b213f640 1 -- 172.21.15.116:0/2266480035 shutdown_connections 2024-04-25T00:18:27.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.836+0000 7f59b213f640 1 --2- 172.21.15.116:0/2266480035 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f598c0678c0 0x7f598c069d80 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:27.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.836+0000 7f59b213f640 1 --2- 172.21.15.116:0/2266480035 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f59ac148f00 0x7f59ac0ac8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:27.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.836+0000 7f59b213f640 1 --2- 172.21.15.116:0/2266480035 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59ac150520 0x7f59ac1489c0 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:27.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.836+0000 7f59b213f640 1 --2- 172.21.15.116:0/2266480035 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5998005bd0 0x7f59ac14ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:27.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.836+0000 7f59b213f640 1 -- 172.21.15.116:0/2266480035 >> 172.21.15.116:0/2266480035 conn(0x7f59ac0c4200 msgr2=0x7f59ac071d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:27.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.836+0000 7f59b213f640 1 -- 172.21.15.116:0/2266480035 shutdown_connections 2024-04-25T00:18:27.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:27.836+0000 7f59b213f640 1 -- 172.21.15.116:0/2266480035 wait complete. 2024-04-25T00:18:27.858 INFO:tasks.mds_thrash.fs.[a]:mds.d reported in up:standby state 2024-04-25T00:18:27.858 INFO:tasks.mds_thrash.fs.[a]:waiting for 50 secs before thrashing 2024-04-25T00:18:28.527 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:18:28.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.648+0000 7f3343dcf640 1 Processor -- start 2024-04-25T00:18:28.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.648+0000 7f3343dcf640 1 -- start start 2024-04-25T00:18:28.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.648+0000 7f3343dcf640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f333c153130 0x7f333c0712d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:28.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.648+0000 7f3343dcf640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f333c0c6ff0 0x7f333c0c5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:28.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.648+0000 7f3343dcf640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f333c0c6100 0x7f333c158f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:28.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.648+0000 7f3343dcf640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f333c155b10 con 0x7f333c153130 2024-04-25T00:18:28.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.648+0000 7f3343dcf640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f333c155df0 con 0x7f333c0c6ff0 2024-04-25T00:18:28.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.648+0000 7f3343dcf640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f333c1560d0 con 0x7f333c0c6100 2024-04-25T00:18:28.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.648+0000 7f3341b6b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f333c153130 0x7f333c0712d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:28.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.648+0000 7f3341b6b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f333c153130 0x7f333c0712d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50668/0 (socket says 172.21.15.116:50668) 2024-04-25T00:18:28.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.648+0000 7f3341b6b640 1 -- 172.21.15.116:0/1952870983 learned_addr learned my addr 172.21.15.116:0/1952870983 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:18:28.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.648+0000 7f3341b6b640 1 -- 172.21.15.116:0/1952870983 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f333c0c6100 msgr2=0x7f333c158f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:28.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.648+0000 7f3341b6b640 1 --2- 172.21.15.116:0/1952870983 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f333c0c6100 0x7f333c158f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:28.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.648+0000 7f3341b6b640 1 -- 172.21.15.116:0/1952870983 >> v2:172.21.15.174:3300/0 conn(0x7f333c0c6ff0 msgr2=0x7f333c0c5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:28.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.648+0000 7f334136a640 1 --2- 172.21.15.116:0/1952870983 >> v2:172.21.15.174:3300/0 conn(0x7f333c0c6ff0 0x7f333c0c5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:28.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.648+0000 7f3341b6b640 1 --2- 172.21.15.116:0/1952870983 >> v2:172.21.15.174:3300/0 conn(0x7f333c0c6ff0 0x7f333c0c5bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:28.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.648+0000 7f3341b6b640 1 -- 172.21.15.116:0/1952870983 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f333c1563b0 con 0x7f333c153130 2024-04-25T00:18:28.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.648+0000 7f334136a640 1 --2- 172.21.15.116:0/1952870983 >> v2:172.21.15.174:3300/0 conn(0x7f333c0c6ff0 0x7f333c0c5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:18:28.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.652+0000 7f3341b6b640 1 --2- 172.21.15.116:0/1952870983 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f333c153130 0x7f333c0712d0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f333c0c9f70 tx=0x7f333801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55733f5b2cbd8133 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:28.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.652+0000 7f3340b69640 1 -- 172.21.15.116:0/1952870983 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33380b0070 con 0x7f333c153130 2024-04-25T00:18:28.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.652+0000 7f3340b69640 1 -- 172.21.15.116:0/1952870983 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33380ab600 con 0x7f333c153130 2024-04-25T00:18:28.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.652+0000 7f3340b69640 1 -- 172.21.15.116:0/1952870983 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33380ab9d0 con 0x7f333c153130 2024-04-25T00:18:28.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.652+0000 7f3343dcf640 1 -- 172.21.15.116:0/1952870983 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f333c153130 msgr2=0x7f333c0712d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:28.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.652+0000 7f3343dcf640 1 --2- 172.21.15.116:0/1952870983 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f333c153130 0x7f333c0712d0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f333c0c9f70 tx=0x7f333801c490 comp rx=0 tx=0).stop 2024-04-25T00:18:28.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.652+0000 7f3343dcf640 1 -- 172.21.15.116:0/1952870983 shutdown_connections 2024-04-25T00:18:28.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.652+0000 7f3343dcf640 1 --2- 172.21.15.116:0/1952870983 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f333c0c6100 0x7f333c158f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:28.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.652+0000 7f3343dcf640 1 --2- 172.21.15.116:0/1952870983 >> v2:172.21.15.174:3300/0 conn(0x7f333c0c6ff0 0x7f333c0c5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:28.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.652+0000 7f3343dcf640 1 --2- 172.21.15.116:0/1952870983 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f333c153130 0x7f333c0712d0 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:28.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.652+0000 7f3343dcf640 1 -- 172.21.15.116:0/1952870983 >> 172.21.15.116:0/1952870983 conn(0x7f333c074650 msgr2=0x7f333c074a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:28.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.652+0000 7f3343dcf640 1 -- 172.21.15.116:0/1952870983 shutdown_connections 2024-04-25T00:18:28.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.652+0000 7f3343dcf640 1 -- 172.21.15.116:0/1952870983 wait complete. 2024-04-25T00:18:28.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.652+0000 7f3343dcf640 1 Processor -- start 2024-04-25T00:18:28.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.652+0000 7f3343dcf640 1 -- start start 2024-04-25T00:18:28.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.656+0000 7f3343dcf640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f333c0c6100 0x7f333c0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:28.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.656+0000 7f3343dcf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f333c0c6ff0 0x7f333c1f1c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:28.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.656+0000 7f3343dcf640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f333c153130 0x7f333c06f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:28.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.656+0000 7f3343dcf640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f333c0c9a00 con 0x7f333c0c6ff0 2024-04-25T00:18:28.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.656+0000 7f3343dcf640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f333c0c9880 con 0x7f333c0c6100 2024-04-25T00:18:28.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.656+0000 7f3343dcf640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f333c0c9b80 con 0x7f333c153130 2024-04-25T00:18:28.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.656+0000 7f334136a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f333c0c6ff0 0x7f333c1f1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:28.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.656+0000 7f334136a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f333c0c6ff0 0x7f333c1f1c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50676/0 (socket says 172.21.15.116:50676) 2024-04-25T00:18:28.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.656+0000 7f334136a640 1 -- 172.21.15.116:0/1678074486 learned_addr learned my addr 172.21.15.116:0/1678074486 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:18:28.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.656+0000 7f334136a640 1 -- 172.21.15.116:0/1678074486 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f333c153130 msgr2=0x7f333c06f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:28.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.656+0000 7f334136a640 1 --2- 172.21.15.116:0/1678074486 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f333c153130 0x7f333c06f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:28.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.656+0000 7f334136a640 1 -- 172.21.15.116:0/1678074486 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f333c0c6100 msgr2=0x7f333c0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:28.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.656+0000 7f334136a640 1 --2- 172.21.15.116:0/1678074486 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f333c0c6100 0x7f333c0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:28.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.656+0000 7f334136a640 1 -- 172.21.15.116:0/1678074486 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f333c070fe0 con 0x7f333c0c6ff0 2024-04-25T00:18:28.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.656+0000 7f334136a640 1 --2- 172.21.15.116:0/1678074486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f333c0c6ff0 0x7f333c1f1c10 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f332c00c7f0 tx=0x7f332c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:28.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.656+0000 7f33327fc640 1 -- 172.21.15.116:0/1678074486 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f332c007a30 con 0x7f333c0c6ff0 2024-04-25T00:18:28.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.656+0000 7f33327fc640 1 -- 172.21.15.116:0/1678074486 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f332c007bd0 con 0x7f333c0c6ff0 2024-04-25T00:18:28.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.656+0000 7f3343dcf640 1 -- 172.21.15.116:0/1678074486 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f333c158f40 con 0x7f333c0c6ff0 2024-04-25T00:18:28.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.656+0000 7f33327fc640 1 -- 172.21.15.116:0/1678074486 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f332c0113f0 con 0x7f333c0c6ff0 2024-04-25T00:18:28.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.656+0000 7f3343dcf640 1 -- 172.21.15.116:0/1678074486 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f333c071280 con 0x7f333c0c6ff0 2024-04-25T00:18:28.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.656+0000 7f3343dcf640 1 -- 172.21.15.116:0/1678074486 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f333c068350 con 0x7f333c0c6ff0 2024-04-25T00:18:28.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.656+0000 7f33327fc640 1 -- 172.21.15.116:0/1678074486 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f332c011590 con 0x7f333c0c6ff0 2024-04-25T00:18:28.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.656+0000 7f33327fc640 1 --2- 172.21.15.116:0/1678074486 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f33200678c0 0x7f3320069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:28.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.656+0000 7f33327fc640 1 -- 172.21.15.116:0/1678074486 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 8902+0+0 (secure 0 0 0) 0x7f332c086ad0 con 0x7f333c0c6ff0 2024-04-25T00:18:28.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.656+0000 7f3341b6b640 1 --2- 172.21.15.116:0/1678074486 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f33200678c0 0x7f3320069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:28.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.660+0000 7f3341b6b640 1 --2- 172.21.15.116:0/1678074486 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f33200678c0 0x7f3320069d80 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f33380ad000 tx=0x7f3338008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:28.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.664+0000 7f33327fc640 1 -- 172.21.15.116:0/1678074486 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f332c015020 con 0x7f333c0c6ff0 2024-04-25T00:18:28.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.904+0000 7f3343dcf640 1 -- 172.21.15.116:0/1678074486 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f333c0c6dd0 con 0x7f333c0c6ff0 2024-04-25T00:18:28.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.904+0000 7f33327fc640 1 -- 172.21.15.116:0/1678074486 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 142 v142) ==== 95+0+6510 (secure 0 0 0) 0x7f332c0569d0 con 0x7f333c0c6ff0 2024-04-25T00:18:28.905 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:18:28.905 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":142,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":5879,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/2276529656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2276529656},{"type":"v1","addr":"172.21.15.174:6837","nonce":2276529656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":142}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:18:23.899096+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7362},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7362":{"gid":7362,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6833/1655959433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1655959433},{"type":"v1","addr":"172.21.15.116:6833","nonce":1655959433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7362,"qdb_cluster":[7362]},"id":1},{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:17:28.837517+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7164},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7164":{"gid":7164,"name":"c","rank":0,"incarnation":128,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6835/3336339376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3336339376},{"type":"v1","addr":"172.21.15.116:6835","nonce":3336339376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7164,"qdb_cluster":[7164]},"id":2}]} 2024-04-25T00:18:28.905 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 142 2024-04-25T00:18:28.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.904+0000 7f3343dcf640 1 -- 172.21.15.116:0/1678074486 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f33200678c0 msgr2=0x7f3320069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:28.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.904+0000 7f3343dcf640 1 --2- 172.21.15.116:0/1678074486 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f33200678c0 0x7f3320069d80 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f33380ad000 tx=0x7f3338008040 comp rx=0 tx=0).stop 2024-04-25T00:18:28.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.904+0000 7f3343dcf640 1 -- 172.21.15.116:0/1678074486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f333c0c6ff0 msgr2=0x7f333c1f1c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:28.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.904+0000 7f3343dcf640 1 --2- 172.21.15.116:0/1678074486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f333c0c6ff0 0x7f333c1f1c10 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f332c00c7f0 tx=0x7f332c00ccb0 comp rx=0 tx=0).stop 2024-04-25T00:18:28.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.904+0000 7f3343dcf640 1 -- 172.21.15.116:0/1678074486 shutdown_connections 2024-04-25T00:18:28.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.904+0000 7f3343dcf640 1 --2- 172.21.15.116:0/1678074486 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f33200678c0 0x7f3320069d80 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:28.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.904+0000 7f3343dcf640 1 --2- 172.21.15.116:0/1678074486 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f333c153130 0x7f333c06f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:28.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.904+0000 7f3343dcf640 1 --2- 172.21.15.116:0/1678074486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f333c0c6ff0 0x7f333c1f1c10 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:28.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.904+0000 7f3343dcf640 1 --2- 172.21.15.116:0/1678074486 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f333c0c6100 0x7f333c0ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:28.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.904+0000 7f3343dcf640 1 -- 172.21.15.116:0/1678074486 >> 172.21.15.116:0/1678074486 conn(0x7f333c074650 msgr2=0x7f333c14d4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:28.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.904+0000 7f3343dcf640 1 -- 172.21.15.116:0/1678074486 shutdown_connections 2024-04-25T00:18:28.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:28.904+0000 7f3343dcf640 1 -- 172.21.15.116:0/1678074486 wait complete. 2024-04-25T00:18:28.924 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:18:30.925 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:18:31.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.052+0000 7f14e3f63640 1 Processor -- start 2024-04-25T00:18:31.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.052+0000 7f14e3f63640 1 -- start start 2024-04-25T00:18:31.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.056+0000 7f14e3f63640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f14dc14c700 0x7f14dc0c5090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:31.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.056+0000 7f14e3f63640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f14dc155b20 0x7f14dc152c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:31.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.056+0000 7f14e3f63640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14dc0a9e60 con 0x7f14dc0c55d0 2024-04-25T00:18:31.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.056+0000 7f14e3f63640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f14dc1527b0 con 0x7f14dc14c700 2024-04-25T00:18:31.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.056+0000 7f14e3f63640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f14dc0c3ef0 con 0x7f14dc155b20 2024-04-25T00:18:31.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.056+0000 7f14e1cff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f14dc14c700 0x7f14dc0c5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:31.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.056+0000 7f14e1cff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f14dc14c700 0x7f14dc0c5090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50690/0 (socket says 172.21.15.116:50690) 2024-04-25T00:18:31.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.056+0000 7f14e1cff640 1 -- 172.21.15.116:0/2643258128 learned_addr learned my addr 172.21.15.116:0/2643258128 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:18:31.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.056+0000 7f14e1cff640 1 -- 172.21.15.116:0/2643258128 >> v2:172.21.15.174:3300/0 conn(0x7f14dc155b20 msgr2=0x7f14dc152c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:31.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.056+0000 7f14e1cff640 1 --2- 172.21.15.116:0/2643258128 >> v2:172.21.15.174:3300/0 conn(0x7f14dc155b20 0x7f14dc152c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:31.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.056+0000 7f14e1cff640 1 -- 172.21.15.116:0/2643258128 >> v1:172.21.15.174:6789/0 conn(0x7f14dc0c55d0 legacy=0x7f14dc071df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:31.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.056+0000 7f14e1cff640 1 -- 172.21.15.116:0/2643258128 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14dc0c41d0 con 0x7f14dc14c700 2024-04-25T00:18:31.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.060+0000 7f14e1cff640 1 --2- 172.21.15.116:0/2643258128 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14dc14c700 0x7f14dc0c5090 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f14dc0a9bc0 tx=0x7f14d801e5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc9579ac807b3ffe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:31.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.060+0000 7f14e0cfd640 1 -- 172.21.15.116:0/2643258128 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14d80b2070 con 0x7f14dc14c700 2024-04-25T00:18:31.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.060+0000 7f14e0cfd640 1 -- 172.21.15.116:0/2643258128 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14d80ad600 con 0x7f14dc14c700 2024-04-25T00:18:31.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.060+0000 7f14e0cfd640 1 -- 172.21.15.116:0/2643258128 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14d80ad9d0 con 0x7f14dc14c700 2024-04-25T00:18:31.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.060+0000 7f14e3f63640 1 -- 172.21.15.116:0/2643258128 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14dc14c700 msgr2=0x7f14dc0c5090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:31.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.060+0000 7f14e3f63640 1 --2- 172.21.15.116:0/2643258128 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14dc14c700 0x7f14dc0c5090 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f14dc0a9bc0 tx=0x7f14d801e5d0 comp rx=0 tx=0).stop 2024-04-25T00:18:31.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.060+0000 7f14e3f63640 1 -- 172.21.15.116:0/2643258128 shutdown_connections 2024-04-25T00:18:31.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.060+0000 7f14e3f63640 1 --2- 172.21.15.116:0/2643258128 >> v2:172.21.15.174:3300/0 conn(0x7f14dc155b20 0x7f14dc152c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:31.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.060+0000 7f14e3f63640 1 --2- 172.21.15.116:0/2643258128 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14dc14c700 0x7f14dc0c5090 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:31.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.060+0000 7f14e3f63640 1 -- 172.21.15.116:0/2643258128 >> 172.21.15.116:0/2643258128 conn(0x7f14dc06c720 msgr2=0x7f14dc06cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:31.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.060+0000 7f14e3f63640 1 -- 172.21.15.116:0/2643258128 shutdown_connections 2024-04-25T00:18:31.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.060+0000 7f14e3f63640 1 -- 172.21.15.116:0/2643258128 wait complete. 2024-04-25T00:18:31.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.060+0000 7f14e3f63640 1 Processor -- start 2024-04-25T00:18:31.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.060+0000 7f14e3f63640 1 -- start start 2024-04-25T00:18:31.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.060+0000 7f14e3f63640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14dc0c55d0 0x7f14dc14a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:31.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.060+0000 7f14e3f63640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f14dc14c700 0x7f14dc1ed7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:31.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.060+0000 7f14e3f63640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f14dc155b20 0x7f14dc1eac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:31.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.060+0000 7f14e3f63640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f14dc153510 con 0x7f14dc0c55d0 2024-04-25T00:18:31.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.060+0000 7f14e3f63640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f14dc153390 con 0x7f14dc14c700 2024-04-25T00:18:31.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.060+0000 7f14e3f63640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f14dc06b210 con 0x7f14dc155b20 2024-04-25T00:18:31.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.064+0000 7f14e1cff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14dc0c55d0 0x7f14dc14a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:31.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.064+0000 7f14e1cff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14dc0c55d0 0x7f14dc14a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50692/0 (socket says 172.21.15.116:50692) 2024-04-25T00:18:31.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.064+0000 7f14e1cff640 1 -- 172.21.15.116:0/1972353028 learned_addr learned my addr 172.21.15.116:0/1972353028 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:18:31.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.064+0000 7f14e14fe640 1 --2- 172.21.15.116:0/1972353028 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f14dc14c700 0x7f14dc1ed7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:31.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.064+0000 7f14e2500640 1 --2- 172.21.15.116:0/1972353028 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f14dc155b20 0x7f14dc1eac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:31.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.064+0000 7f14e1cff640 1 -- 172.21.15.116:0/1972353028 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f14dc155b20 msgr2=0x7f14dc1eac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:31.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.064+0000 7f14e1cff640 1 --2- 172.21.15.116:0/1972353028 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f14dc155b20 0x7f14dc1eac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:31.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.064+0000 7f14e1cff640 1 -- 172.21.15.116:0/1972353028 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f14dc14c700 msgr2=0x7f14dc1ed7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:31.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.064+0000 7f14e1cff640 1 --2- 172.21.15.116:0/1972353028 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f14dc14c700 0x7f14dc1ed7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:31.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.064+0000 7f14e1cff640 1 -- 172.21.15.116:0/1972353028 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14dc146710 con 0x7f14dc0c55d0 2024-04-25T00:18:31.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.064+0000 7f14e14fe640 1 --2- 172.21.15.116:0/1972353028 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f14dc14c700 0x7f14dc1ed7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:18:31.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.064+0000 7f14e1cff640 1 --2- 172.21.15.116:0/1972353028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14dc0c55d0 0x7f14dc14a270 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f14d80b0040 tx=0x7f14d801ce10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:31.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.064+0000 7f14d27fc640 1 -- 172.21.15.116:0/1972353028 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14d80bc020 con 0x7f14dc0c55d0 2024-04-25T00:18:31.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.064+0000 7f14d27fc640 1 -- 172.21.15.116:0/1972353028 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14d80ac740 con 0x7f14dc0c55d0 2024-04-25T00:18:31.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.064+0000 7f14e3f63640 1 -- 172.21.15.116:0/1972353028 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14dc145130 con 0x7f14dc0c55d0 2024-04-25T00:18:31.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.064+0000 7f14d27fc640 1 -- 172.21.15.116:0/1972353028 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14d80b2040 con 0x7f14dc0c55d0 2024-04-25T00:18:31.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.064+0000 7f14e3f63640 1 -- 172.21.15.116:0/1972353028 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14dc1ea410 con 0x7f14dc0c55d0 2024-04-25T00:18:31.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.064+0000 7f14e3f63640 1 -- 172.21.15.116:0/1972353028 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14dc068a60 con 0x7f14dc0c55d0 2024-04-25T00:18:31.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.064+0000 7f14d27fc640 1 -- 172.21.15.116:0/1972353028 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f14d80bf020 con 0x7f14dc0c55d0 2024-04-25T00:18:31.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.064+0000 7f14d27fc640 1 --2- 172.21.15.116:0/1972353028 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f14c00678c0 0x7f14c0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:31.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.064+0000 7f14d27fc640 1 -- 172.21.15.116:0/1972353028 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 8902+0+0 (secure 0 0 0) 0x7f14d8122b70 con 0x7f14dc0c55d0 2024-04-25T00:18:31.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.064+0000 7f14e14fe640 1 --2- 172.21.15.116:0/1972353028 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f14c00678c0 0x7f14c0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:31.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.068+0000 7f14e14fe640 1 --2- 172.21.15.116:0/1972353028 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f14c00678c0 0x7f14c0069d80 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f14dc155370 tx=0x7f14cc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:31.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.072+0000 7f14d27fc640 1 -- 172.21.15.116:0/1972353028 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f14d80c6070 con 0x7f14dc0c55d0 2024-04-25T00:18:31.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.320+0000 7f14e3f63640 1 -- 172.21.15.116:0/1972353028 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f14dc0ad630 con 0x7f14dc0c55d0 2024-04-25T00:18:31.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.320+0000 7f14d27fc640 1 -- 172.21.15.116:0/1972353028 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 142 v142) ==== 95+0+6510 (secure 0 0 0) 0x7f14d80182e0 con 0x7f14dc0c55d0 2024-04-25T00:18:31.321 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:18:31.321 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":142,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":5879,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/2276529656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2276529656},{"type":"v1","addr":"172.21.15.174:6837","nonce":2276529656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":142}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:18:23.899096+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7362},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7362":{"gid":7362,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6833/1655959433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1655959433},{"type":"v1","addr":"172.21.15.116:6833","nonce":1655959433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7362,"qdb_cluster":[7362]},"id":1},{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:17:28.837517+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7164},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7164":{"gid":7164,"name":"c","rank":0,"incarnation":128,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6835/3336339376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3336339376},{"type":"v1","addr":"172.21.15.116:6835","nonce":3336339376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7164,"qdb_cluster":[7164]},"id":2}]} 2024-04-25T00:18:31.321 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 142 2024-04-25T00:18:31.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.320+0000 7f14e3f63640 1 -- 172.21.15.116:0/1972353028 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f14c00678c0 msgr2=0x7f14c0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:31.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.320+0000 7f14e3f63640 1 --2- 172.21.15.116:0/1972353028 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f14c00678c0 0x7f14c0069d80 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f14dc155370 tx=0x7f14cc008040 comp rx=0 tx=0).stop 2024-04-25T00:18:31.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.320+0000 7f14e3f63640 1 -- 172.21.15.116:0/1972353028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14dc0c55d0 msgr2=0x7f14dc14a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:31.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.320+0000 7f14e3f63640 1 --2- 172.21.15.116:0/1972353028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14dc0c55d0 0x7f14dc14a270 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f14d80b0040 tx=0x7f14d801ce10 comp rx=0 tx=0).stop 2024-04-25T00:18:31.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.320+0000 7f14e3f63640 1 -- 172.21.15.116:0/1972353028 shutdown_connections 2024-04-25T00:18:31.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.320+0000 7f14e3f63640 1 --2- 172.21.15.116:0/1972353028 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f14c00678c0 0x7f14c0069d80 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:31.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.320+0000 7f14e3f63640 1 --2- 172.21.15.116:0/1972353028 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f14dc155b20 0x7f14dc1eac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:31.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.320+0000 7f14e3f63640 1 --2- 172.21.15.116:0/1972353028 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f14dc14c700 0x7f14dc1ed7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:31.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.320+0000 7f14e3f63640 1 --2- 172.21.15.116:0/1972353028 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f14dc0c55d0 0x7f14dc14a270 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:31.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.320+0000 7f14e3f63640 1 -- 172.21.15.116:0/1972353028 >> 172.21.15.116:0/1972353028 conn(0x7f14dc06c720 msgr2=0x7f14dc071df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:31.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.320+0000 7f14e3f63640 1 -- 172.21.15.116:0/1972353028 shutdown_connections 2024-04-25T00:18:31.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:31.320+0000 7f14e3f63640 1 -- 172.21.15.116:0/1972353028 wait complete. 2024-04-25T00:18:31.340 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:18:33.342 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:18:33.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.484+0000 7f5f12569640 1 Processor -- start 2024-04-25T00:18:33.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.484+0000 7f5f12569640 1 -- start start 2024-04-25T00:18:33.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.484+0000 7f5f12569640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5f0c14c700 0x7f5f0c0c5090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:33.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.484+0000 7f5f12569640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5f0c155b20 0x7f5f0c152c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:33.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.484+0000 7f5f12569640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f0c0a9e60 con 0x7f5f0c0c55d0 2024-04-25T00:18:33.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.484+0000 7f5f12569640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f5f0c1527b0 con 0x7f5f0c155b20 2024-04-25T00:18:33.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.484+0000 7f5f12569640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f5f0c0c3ef0 con 0x7f5f0c14c700 2024-04-25T00:18:33.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.484+0000 7f5f03fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5f0c155b20 0x7f5f0c152c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:33.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.484+0000 7f5f03fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5f0c155b20 0x7f5f0c152c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50696/0 (socket says 172.21.15.116:50696) 2024-04-25T00:18:33.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.484+0000 7f5f03fff640 1 -- 172.21.15.116:0/1186742896 learned_addr learned my addr 172.21.15.116:0/1186742896 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:18:33.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.484+0000 7f5f0bfff640 1 --2- 172.21.15.116:0/1186742896 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5f0c14c700 0x7f5f0c0c5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:33.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.484+0000 7f5f03fff640 1 -- 172.21.15.116:0/1186742896 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5f0c14c700 msgr2=0x7f5f0c0c5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:33.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.484+0000 7f5f03fff640 1 --2- 172.21.15.116:0/1186742896 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5f0c14c700 0x7f5f0c0c5090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:33.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.484+0000 7f5f03fff640 1 -- 172.21.15.116:0/1186742896 >> v1:172.21.15.174:6789/0 conn(0x7f5f0c0c55d0 legacy=0x7f5f0c071df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:33.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.484+0000 7f5f03fff640 1 -- 172.21.15.116:0/1186742896 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f0c0c41d0 con 0x7f5f0c155b20 2024-04-25T00:18:33.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.484+0000 7f5f0bfff640 1 --2- 172.21.15.116:0/1186742896 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5f0c14c700 0x7f5f0c0c5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:18:33.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.488+0000 7f5f03fff640 1 --2- 172.21.15.116:0/1186742896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f0c155b20 0x7f5f0c152c40 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f5f0c152930 tx=0x7f5ef401c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=655f581d17a8415 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:33.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.488+0000 7f5f0b7fe640 1 -- 172.21.15.116:0/1186742896 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ef40b0070 con 0x7f5f0c155b20 2024-04-25T00:18:33.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.488+0000 7f5f0b7fe640 1 -- 172.21.15.116:0/1186742896 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ef40ab600 con 0x7f5f0c155b20 2024-04-25T00:18:33.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.488+0000 7f5f0b7fe640 1 -- 172.21.15.116:0/1186742896 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ef40ab9d0 con 0x7f5f0c155b20 2024-04-25T00:18:33.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.488+0000 7f5f12569640 1 -- 172.21.15.116:0/1186742896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f0c155b20 msgr2=0x7f5f0c152c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:33.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.488+0000 7f5f12569640 1 --2- 172.21.15.116:0/1186742896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f0c155b20 0x7f5f0c152c40 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f5f0c152930 tx=0x7f5ef401c420 comp rx=0 tx=0).stop 2024-04-25T00:18:33.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.488+0000 7f5f12569640 1 -- 172.21.15.116:0/1186742896 shutdown_connections 2024-04-25T00:18:33.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.488+0000 7f5f12569640 1 --2- 172.21.15.116:0/1186742896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f0c155b20 0x7f5f0c152c40 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:33.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.488+0000 7f5f12569640 1 --2- 172.21.15.116:0/1186742896 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5f0c14c700 0x7f5f0c0c5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:33.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.488+0000 7f5f12569640 1 -- 172.21.15.116:0/1186742896 >> 172.21.15.116:0/1186742896 conn(0x7f5f0c06c720 msgr2=0x7f5f0c06cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:33.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.488+0000 7f5f12569640 1 -- 172.21.15.116:0/1186742896 shutdown_connections 2024-04-25T00:18:33.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.488+0000 7f5f12569640 1 -- 172.21.15.116:0/1186742896 wait complete. 2024-04-25T00:18:33.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.492+0000 7f5f12569640 1 Processor -- start 2024-04-25T00:18:33.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.492+0000 7f5f12569640 1 -- start start 2024-04-25T00:18:33.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.492+0000 7f5f12569640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5f0c0c55d0 0x7f5f0c1eec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:33.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.492+0000 7f5f12569640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5f0c14c700 0x7f5f0c14b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:33.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.492+0000 7f5f12569640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f0c155b20 0x7f5f0c1f29d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:33.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.492+0000 7f5f12569640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5f0c153510 con 0x7f5f0c155b20 2024-04-25T00:18:33.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.492+0000 7f5f12569640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5f0c153390 con 0x7f5f0c14c700 2024-04-25T00:18:33.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.492+0000 7f5f12569640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f5f0c15fed0 con 0x7f5f0c0c55d0 2024-04-25T00:18:33.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.492+0000 7f5f10b06640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f0c155b20 0x7f5f0c1f29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:33.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.492+0000 7f5f10b06640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f0c155b20 0x7f5f0c1f29d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50708/0 (socket says 172.21.15.116:50708) 2024-04-25T00:18:33.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.492+0000 7f5f10b06640 1 -- 172.21.15.116:0/632354834 learned_addr learned my addr 172.21.15.116:0/632354834 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:18:33.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.492+0000 7f5f03fff640 1 --2- 172.21.15.116:0/632354834 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5f0c14c700 0x7f5f0c14b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:33.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.492+0000 7f5f0bfff640 1 --2- 172.21.15.116:0/632354834 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5f0c0c55d0 0x7f5f0c1eec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:33.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.492+0000 7f5f10b06640 1 -- 172.21.15.116:0/632354834 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5f0c0c55d0 msgr2=0x7f5f0c1eec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:33.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.492+0000 7f5f10b06640 1 --2- 172.21.15.116:0/632354834 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5f0c0c55d0 0x7f5f0c1eec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:33.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.492+0000 7f5f10b06640 1 -- 172.21.15.116:0/632354834 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5f0c14c700 msgr2=0x7f5f0c14b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:33.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.492+0000 7f5f10b06640 1 --2- 172.21.15.116:0/632354834 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5f0c14c700 0x7f5f0c14b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:33.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.492+0000 7f5f10b06640 1 -- 172.21.15.116:0/632354834 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f0c147f20 con 0x7f5f0c155b20 2024-04-25T00:18:33.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.492+0000 7f5f0bfff640 1 --2- 172.21.15.116:0/632354834 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5f0c0c55d0 0x7f5f0c1eec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:18:33.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.492+0000 7f5f10b06640 1 --2- 172.21.15.116:0/632354834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f0c155b20 0x7f5f0c1f29d0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f5f0400c760 tx=0x7f5f0400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:33.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.492+0000 7f5f097fa640 1 -- 172.21.15.116:0/632354834 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f04003b80 con 0x7f5f0c155b20 2024-04-25T00:18:33.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.492+0000 7f5f097fa640 1 -- 172.21.15.116:0/632354834 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f04003d20 con 0x7f5f0c155b20 2024-04-25T00:18:33.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.492+0000 7f5f12569640 1 -- 172.21.15.116:0/632354834 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f0c148520 con 0x7f5f0c155b20 2024-04-25T00:18:33.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.492+0000 7f5f097fa640 1 -- 172.21.15.116:0/632354834 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f0400aa30 con 0x7f5f0c155b20 2024-04-25T00:18:33.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.492+0000 7f5f12569640 1 -- 172.21.15.116:0/632354834 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f0c14bbe0 con 0x7f5f0c155b20 2024-04-25T00:18:33.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.492+0000 7f5f12569640 1 -- 172.21.15.116:0/632354834 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f0c068a60 con 0x7f5f0c155b20 2024-04-25T00:18:33.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.492+0000 7f5f097fa640 1 -- 172.21.15.116:0/632354834 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5f0400abd0 con 0x7f5f0c155b20 2024-04-25T00:18:33.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.496+0000 7f5f097fa640 1 --2- 172.21.15.116:0/632354834 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5eec0678c0 0x7f5eec069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:33.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.496+0000 7f5f097fa640 1 -- 172.21.15.116:0/632354834 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 8902+0+0 (secure 0 0 0) 0x7f5f04086790 con 0x7f5f0c155b20 2024-04-25T00:18:33.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.496+0000 7f5f0bfff640 1 --2- 172.21.15.116:0/632354834 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5eec0678c0 0x7f5eec069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:33.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.496+0000 7f5f0bfff640 1 --2- 172.21.15.116:0/632354834 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5eec0678c0 0x7f5eec069d80 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f5efc008930 tx=0x7f5efc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:33.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.500+0000 7f5f097fa640 1 -- 172.21.15.116:0/632354834 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f5f04056690 con 0x7f5f0c155b20 2024-04-25T00:18:33.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.712+0000 7f5f12569640 1 -- 172.21.15.116:0/632354834 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5f0c0ad630 con 0x7f5f0c155b20 2024-04-25T00:18:33.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.712+0000 7f5f097fa640 1 -- 172.21.15.116:0/632354834 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 142 v142) ==== 95+0+6510 (secure 0 0 0) 0x7f5f04055f90 con 0x7f5f0c155b20 2024-04-25T00:18:33.715 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:18:33.716 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":142,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":5879,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/2276529656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2276529656},{"type":"v1","addr":"172.21.15.174:6837","nonce":2276529656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":142}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:18:23.899096+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7362},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7362":{"gid":7362,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6833/1655959433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1655959433},{"type":"v1","addr":"172.21.15.116:6833","nonce":1655959433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7362,"qdb_cluster":[7362]},"id":1},{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:17:28.837517+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":52,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7164},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7164":{"gid":7164,"name":"c","rank":0,"incarnation":128,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6835/3336339376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3336339376},{"type":"v1","addr":"172.21.15.116:6835","nonce":3336339376}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7164,"qdb_cluster":[7164]},"id":2}]} 2024-04-25T00:18:33.716 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 142 2024-04-25T00:18:33.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.716+0000 7f5f12569640 1 -- 172.21.15.116:0/632354834 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5eec0678c0 msgr2=0x7f5eec069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:33.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.716+0000 7f5f12569640 1 --2- 172.21.15.116:0/632354834 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5eec0678c0 0x7f5eec069d80 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f5efc008930 tx=0x7f5efc008040 comp rx=0 tx=0).stop 2024-04-25T00:18:33.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.716+0000 7f5f12569640 1 -- 172.21.15.116:0/632354834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f0c155b20 msgr2=0x7f5f0c1f29d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:33.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.716+0000 7f5f12569640 1 --2- 172.21.15.116:0/632354834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f0c155b20 0x7f5f0c1f29d0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f5f0400c760 tx=0x7f5f0400cc20 comp rx=0 tx=0).stop 2024-04-25T00:18:33.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.716+0000 7f5f12569640 1 -- 172.21.15.116:0/632354834 shutdown_connections 2024-04-25T00:18:33.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.716+0000 7f5f12569640 1 --2- 172.21.15.116:0/632354834 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5eec0678c0 0x7f5eec069d80 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:33.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.716+0000 7f5f12569640 1 --2- 172.21.15.116:0/632354834 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f0c155b20 0x7f5f0c1f29d0 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:33.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.716+0000 7f5f12569640 1 --2- 172.21.15.116:0/632354834 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5f0c14c700 0x7f5f0c14b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:33.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.716+0000 7f5f12569640 1 --2- 172.21.15.116:0/632354834 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5f0c0c55d0 0x7f5f0c1eec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:33.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.716+0000 7f5f12569640 1 -- 172.21.15.116:0/632354834 >> 172.21.15.116:0/632354834 conn(0x7f5f0c06c720 msgr2=0x7f5f0c071df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:33.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.716+0000 7f5f12569640 1 -- 172.21.15.116:0/632354834 shutdown_connections 2024-04-25T00:18:33.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:33.716+0000 7f5f12569640 1 -- 172.21.15.116:0/632354834 wait complete. 2024-04-25T00:18:33.735 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:18:35.735 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:18:35.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.859+0000 7f9ec77d2640 1 Processor -- start 2024-04-25T00:18:35.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.859+0000 7f9ec77d2640 1 -- start start 2024-04-25T00:18:35.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.859+0000 7f9ec77d2640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f9ec0152fd0 0x7f9ec006c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:35.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.859+0000 7f9ec77d2640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ec00c5c50 con 0x7f9ec00c6f10 2024-04-25T00:18:35.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.859+0000 7f9ec77d2640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ec00c69a0 con 0x7f9ec0158e50 2024-04-25T00:18:35.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.859+0000 7f9ec77d2640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f9ec0159940 con 0x7f9ec0152fd0 2024-04-25T00:18:35.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.859+0000 7f9ec4d6d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f9ec00c6f10 0x7f9ec00c8030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:35490/0 (socket says 172.21.15.116:35490) 2024-04-25T00:18:35.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.859+0000 7f9ec4d6d640 1 -- 172.21.15.116:0/2392167620 learned_addr learned my addr 172.21.15.116:0/2392167620 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:18:35.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.859+0000 7f9ec556e640 1 --2- 172.21.15.116:0/2392167620 >> v2:172.21.15.174:3300/0 conn(0x7f9ec0152fd0 0x7f9ec006c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:35.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.859+0000 7f9eb7fff640 1 -- 172.21.15.116:0/2392167620 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1899200432 0 0) 0x7f9ec00c5c50 con 0x7f9ec00c6f10 2024-04-25T00:18:35.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.863+0000 7f9eb7fff640 1 -- 172.21.15.116:0/2392167620 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9eac003530 con 0x7f9ec00c6f10 2024-04-25T00:18:35.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.863+0000 7f9eb7fff640 1 -- 172.21.15.116:0/2392167620 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3320729037 0 0) 0x7f9ec00c69a0 con 0x7f9ec0158e50 2024-04-25T00:18:35.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.863+0000 7f9eb7fff640 1 -- 172.21.15.116:0/2392167620 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ec00c5c50 con 0x7f9ec0158e50 2024-04-25T00:18:35.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.863+0000 7f9eb7fff640 1 -- 172.21.15.116:0/2392167620 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2330394721 0 0) 0x7f9eac003530 con 0x7f9ec00c6f10 2024-04-25T00:18:35.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.863+0000 7f9eb7fff640 1 -- 172.21.15.116:0/2392167620 >> v2:172.21.15.174:3300/0 conn(0x7f9ec0152fd0 msgr2=0x7f9ec006c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:35.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.863+0000 7f9eb7fff640 1 --2- 172.21.15.116:0/2392167620 >> v2:172.21.15.174:3300/0 conn(0x7f9ec0152fd0 0x7f9ec006c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:35.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.863+0000 7f9eb7fff640 1 -- 172.21.15.116:0/2392167620 >> v1:172.21.15.174:6789/0 conn(0x7f9ec0158e50 legacy=0x7f9ec0159230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:35.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.863+0000 7f9eb7fff640 1 -- 172.21.15.116:0/2392167620 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ec00c5ec0 con 0x7f9ec00c6f10 2024-04-25T00:18:35.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.863+0000 7f9eb7fff640 1 -- 172.21.15.116:0/2392167620 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f9eb0001680 con 0x7f9ec00c6f10 2024-04-25T00:18:35.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.863+0000 7f9eb7fff640 1 -- 172.21.15.116:0/2392167620 >> v1:172.21.15.116:6789/0 conn(0x7f9ec00c6f10 legacy=0x7f9ec00c8030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:35.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.863+0000 7f9eb7fff640 1 --2- 172.21.15.116:0/2392167620 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9eac003ba0 0x7f9eac003fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:35.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.863+0000 7f9eb7fff640 1 --2- 172.21.15.116:0/2392167620 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9eac004810 0x7f9eac004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:35.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.863+0000 7f9eb7fff640 1 --2- 172.21.15.116:0/2392167620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9eac005220 0x7f9eac00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:35.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.863+0000 7f9eb7fff640 1 -- 172.21.15.116:0/2392167620 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9eac00c120 con 0x7f9eac005220 2024-04-25T00:18:35.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.863+0000 7f9eb7fff640 1 -- 172.21.15.116:0/2392167620 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9eac00c2d0 con 0x7f9eac003ba0 2024-04-25T00:18:35.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.863+0000 7f9eb7fff640 1 -- 172.21.15.116:0/2392167620 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f9eac00c4d0 con 0x7f9eac004810 2024-04-25T00:18:35.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.863+0000 7f9ec5d6f640 1 --2- 172.21.15.116:0/2392167620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9eac005220 0x7f9eac00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:35.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.863+0000 7f9ec4d6d640 1 --2- 172.21.15.116:0/2392167620 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9eac004810 0x7f9eac004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:35.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.863+0000 7f9ec556e640 1 --2- 172.21.15.116:0/2392167620 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9eac003ba0 0x7f9eac003fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:35.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.863+0000 7f9ec5d6f640 1 -- 172.21.15.116:0/2392167620 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9eac004810 msgr2=0x7f9eac004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:35.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.863+0000 7f9ec5d6f640 1 --2- 172.21.15.116:0/2392167620 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9eac004810 0x7f9eac004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:35.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.863+0000 7f9ec5d6f640 1 -- 172.21.15.116:0/2392167620 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9eac003ba0 msgr2=0x7f9eac003fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:35.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.863+0000 7f9ec5d6f640 1 --2- 172.21.15.116:0/2392167620 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9eac003ba0 0x7f9eac003fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:35.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.863+0000 7f9ec5d6f640 1 -- 172.21.15.116:0/2392167620 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9eac003530 con 0x7f9eac005220 2024-04-25T00:18:35.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.863+0000 7f9ec4d6d640 1 --2- 172.21.15.116:0/2392167620 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9eac004810 0x7f9eac004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:18:35.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.863+0000 7f9ec556e640 1 --2- 172.21.15.116:0/2392167620 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9eac003ba0 0x7f9eac003fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:18:35.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.863+0000 7f9ec5d6f640 1 -- 172.21.15.116:0/2392167620 reap_dead start 2024-04-25T00:18:35.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.863+0000 7f9ec5d6f640 1 --2- 172.21.15.116:0/2392167620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9eac005220 0x7f9eac00b9d0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f9eb8001fd0 tx=0x7f9eb801f070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c577f8bc78f9b708 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:35.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.863+0000 7f9eb7fff640 1 -- 172.21.15.116:0/2392167620 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eb8001c20 con 0x7f9eac005220 2024-04-25T00:18:35.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.863+0000 7f9eb7fff640 1 -- 172.21.15.116:0/2392167620 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9eb80a8c90 con 0x7f9eac005220 2024-04-25T00:18:35.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.863+0000 7f9eb7fff640 1 -- 172.21.15.116:0/2392167620 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eb8003bc0 con 0x7f9eac005220 2024-04-25T00:18:35.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.867+0000 7f9ec77d2640 1 -- 172.21.15.116:0/2392167620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9eac005220 msgr2=0x7f9eac00b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:35.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.867+0000 7f9ec77d2640 1 --2- 172.21.15.116:0/2392167620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9eac005220 0x7f9eac00b9d0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f9eb8001fd0 tx=0x7f9eb801f070 comp rx=0 tx=0).stop 2024-04-25T00:18:35.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.867+0000 7f9ec77d2640 1 -- 172.21.15.116:0/2392167620 shutdown_connections 2024-04-25T00:18:35.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.867+0000 7f9ec77d2640 1 --2- 172.21.15.116:0/2392167620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9eac005220 0x7f9eac00b9d0 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:35.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.867+0000 7f9ec77d2640 1 -- 172.21.15.116:0/2392167620 >> 172.21.15.116:0/2392167620 conn(0x7f9ec00ad010 msgr2=0x7f9ec00ad420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:35.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.867+0000 7f9ec77d2640 1 -- 172.21.15.116:0/2392167620 shutdown_connections 2024-04-25T00:18:35.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.867+0000 7f9ec77d2640 1 -- 172.21.15.116:0/2392167620 wait complete. 2024-04-25T00:18:35.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.867+0000 7f9ec77d2640 1 Processor -- start 2024-04-25T00:18:35.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.867+0000 7f9ec77d2640 1 -- start start 2024-04-25T00:18:35.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.867+0000 7f9ec77d2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9eac005220 0x7f9ec014de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:35.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.867+0000 7f9ec77d2640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9ec0148bf0 0x7f9ec0147e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:35.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.867+0000 7f9ec77d2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ec01486c0 0x7f9ec014e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:35.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.867+0000 7f9ec77d2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9ec01533b0 con 0x7f9ec01486c0 2024-04-25T00:18:35.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.867+0000 7f9ec77d2640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9ec00c6480 con 0x7f9eac005220 2024-04-25T00:18:35.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.867+0000 7f9ec77d2640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f9ec0153530 con 0x7f9ec0148bf0 2024-04-25T00:18:35.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.867+0000 7f9ec5d6f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ec01486c0 0x7f9ec014e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:35.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.867+0000 7f9ec5d6f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ec01486c0 0x7f9ec014e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60618/0 (socket says 172.21.15.116:60618) 2024-04-25T00:18:35.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.867+0000 7f9ec5d6f640 1 -- 172.21.15.116:0/3599507519 learned_addr learned my addr 172.21.15.116:0/3599507519 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:18:35.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.867+0000 7f9ec556e640 1 --2- 172.21.15.116:0/3599507519 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9eac005220 0x7f9ec014de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:35.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.867+0000 7f9ec4d6d640 1 --2- 172.21.15.116:0/3599507519 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9ec0148bf0 0x7f9ec0147e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:35.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.867+0000 7f9ec5d6f640 1 -- 172.21.15.116:0/3599507519 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9ec0148bf0 msgr2=0x7f9ec0147e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:35.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.867+0000 7f9ec5d6f640 1 --2- 172.21.15.116:0/3599507519 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9ec0148bf0 0x7f9ec0147e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:35.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.867+0000 7f9ec5d6f640 1 -- 172.21.15.116:0/3599507519 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9eac005220 msgr2=0x7f9ec014de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:35.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.867+0000 7f9ec5d6f640 1 --2- 172.21.15.116:0/3599507519 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9eac005220 0x7f9ec014de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:35.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.867+0000 7f9ec5d6f640 1 -- 172.21.15.116:0/3599507519 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ec01ec0b0 con 0x7f9ec01486c0 2024-04-25T00:18:35.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.867+0000 7f9ec5d6f640 1 --2- 172.21.15.116:0/3599507519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ec01486c0 0x7f9ec014e4b0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f9eb801eea0 tx=0x7f9eac005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:35.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.871+0000 7f9eb5ffb640 1 -- 172.21.15.116:0/3599507519 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eb8001c00 con 0x7f9ec01486c0 2024-04-25T00:18:35.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.871+0000 7f9eb5ffb640 1 -- 172.21.15.116:0/3599507519 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9eb80050d0 con 0x7f9ec01486c0 2024-04-25T00:18:35.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.871+0000 7f9ec77d2640 1 -- 172.21.15.116:0/3599507519 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ec00afd50 con 0x7f9ec01486c0 2024-04-25T00:18:35.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.871+0000 7f9eb5ffb640 1 -- 172.21.15.116:0/3599507519 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eb80054a0 con 0x7f9ec01486c0 2024-04-25T00:18:35.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.871+0000 7f9ec77d2640 1 -- 172.21.15.116:0/3599507519 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ec01ebdd0 con 0x7f9ec01486c0 2024-04-25T00:18:35.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.871+0000 7f9ec77d2640 1 -- 172.21.15.116:0/3599507519 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ec0068a60 con 0x7f9ec01486c0 2024-04-25T00:18:35.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.871+0000 7f9eb5ffb640 1 -- 172.21.15.116:0/3599507519 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f9eb8005640 con 0x7f9ec01486c0 2024-04-25T00:18:35.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.871+0000 7f9eb5ffb640 1 --2- 172.21.15.116:0/3599507519 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9ea40678f0 0x7f9ea4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:35.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.871+0000 7f9eb5ffb640 1 -- 172.21.15.116:0/3599507519 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 8988+0+0 (secure 0 0 0) 0x7f9eb8123950 con 0x7f9ec01486c0 2024-04-25T00:18:35.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.871+0000 7f9ec556e640 1 --2- 172.21.15.116:0/3599507519 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9ea40678f0 0x7f9ea4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:35.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.871+0000 7f9ec556e640 1 --2- 172.21.15.116:0/3599507519 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9ea40678f0 0x7f9ea4069db0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc009ac0 tx=0x7f9ebc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:35.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:35.875+0000 7f9eb5ffb640 1 -- 172.21.15.116:0/3599507519 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f9eb80bf060 con 0x7f9ec01486c0 2024-04-25T00:18:36.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.111+0000 7f9ec77d2640 1 -- 172.21.15.116:0/3599507519 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9ec00aab60 con 0x7f9ec01486c0 2024-04-25T00:18:36.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.111+0000 7f9eb5ffb640 1 -- 172.21.15.116:0/3599507519 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 143 v143) ==== 95+0+5673 (secure 0 0 0) 0x7f9eb8018d80 con 0x7f9ec01486c0 2024-04-25T00:18:36.113 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:18:36.114 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":143,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:18:23.899096+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7362},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7362":{"gid":7362,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6833/1655959433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1655959433},{"type":"v1","addr":"172.21.15.116:6833","nonce":1655959433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7362,"qdb_cluster":[7362]},"id":1},{"mdsmap":{"epoch":143,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:18:35.617345+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5879},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5879":{"gid":5879,"name":"d","rank":0,"incarnation":143,"state":"up:replay","state_seq":1,"addr":"172.21.15.174:6837/2276529656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2276529656},{"type":"v1","addr":"172.21.15.174:6837","nonce":2276529656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:18:36.114 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 143 2024-04-25T00:18:36.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.111+0000 7f9ec77d2640 1 -- 172.21.15.116:0/3599507519 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9ea40678f0 msgr2=0x7f9ea4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:36.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.111+0000 7f9ec77d2640 1 --2- 172.21.15.116:0/3599507519 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9ea40678f0 0x7f9ea4069db0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc009ac0 tx=0x7f9ebc008040 comp rx=0 tx=0).stop 2024-04-25T00:18:36.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.111+0000 7f9ec77d2640 1 -- 172.21.15.116:0/3599507519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ec01486c0 msgr2=0x7f9ec014e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:36.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.111+0000 7f9ec77d2640 1 --2- 172.21.15.116:0/3599507519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ec01486c0 0x7f9ec014e4b0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f9eb801eea0 tx=0x7f9eac005840 comp rx=0 tx=0).stop 2024-04-25T00:18:36.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.111+0000 7f9ec77d2640 1 -- 172.21.15.116:0/3599507519 shutdown_connections 2024-04-25T00:18:36.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.111+0000 7f9ec77d2640 1 --2- 172.21.15.116:0/3599507519 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9ea40678f0 0x7f9ea4069db0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:36.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.111+0000 7f9ec77d2640 1 --2- 172.21.15.116:0/3599507519 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9ec01486c0 0x7f9ec014e4b0 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:36.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.111+0000 7f9ec77d2640 1 --2- 172.21.15.116:0/3599507519 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9ec0148bf0 0x7f9ec0147e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:36.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.111+0000 7f9ec77d2640 1 --2- 172.21.15.116:0/3599507519 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9eac005220 0x7f9ec014de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:36.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.111+0000 7f9ec77d2640 1 -- 172.21.15.116:0/3599507519 >> 172.21.15.116:0/3599507519 conn(0x7f9ec00ad010 msgr2=0x7f9ec006be70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:36.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.115+0000 7f9ec77d2640 1 -- 172.21.15.116:0/3599507519 shutdown_connections 2024-04-25T00:18:36.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.115+0000 7f9ec77d2640 1 -- 172.21.15.116:0/3599507519 wait complete. 2024-04-25T00:18:36.133 INFO:tasks.mds_thrash.fs.[b]:mds.c down, removed from mdsmap 2024-04-25T00:18:36.133 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-04-25T00:18:36.133 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:18:36.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.259+0000 7f85ccc64640 1 Processor -- start 2024-04-25T00:18:36.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.259+0000 7f85ccc64640 1 -- start start 2024-04-25T00:18:36.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.259+0000 7f85ccc64640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f85c8072c90 0x7f85c8150f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:36.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.259+0000 7f85ccc64640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f85c8072810 0x7f85c8153760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:36.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.259+0000 7f85ccc64640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85c80c2c40 con 0x7f85c80c5280 2024-04-25T00:18:36.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.259+0000 7f85ccc64640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f85c8157c20 con 0x7f85c8072810 2024-04-25T00:18:36.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.259+0000 7f85ccc64640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f85c80c2f70 con 0x7f85c8072c90 2024-04-25T00:18:36.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.259+0000 7f85c6d9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f85c8072810 0x7f85c8153760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:36.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.259+0000 7f85c659c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f85c80c5280 0x7f85c8150840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:35494/0 (socket says 172.21.15.116:35494) 2024-04-25T00:18:36.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.259+0000 7f85c659c640 1 -- 172.21.15.116:0/324906780 learned_addr learned my addr 172.21.15.116:0/324906780 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:18:36.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.259+0000 7f85bffff640 1 --2- 172.21.15.116:0/324906780 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f85c8072c90 0x7f85c8150f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:36.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.259+0000 7f85c6d9d640 1 -- 172.21.15.116:0/324906780 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f85c8072c90 msgr2=0x7f85c8150f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:36.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.259+0000 7f85c6d9d640 1 --2- 172.21.15.116:0/324906780 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f85c8072c90 0x7f85c8150f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:36.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.259+0000 7f85c5d9b640 1 -- 172.21.15.116:0/324906780 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1654605917 0 0) 0x7f85c80c2c40 con 0x7f85c80c5280 2024-04-25T00:18:36.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.259+0000 7f85c6d9d640 1 -- 172.21.15.116:0/324906780 >> v1:172.21.15.116:6789/0 conn(0x7f85c80c5280 legacy=0x7f85c8150840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:36.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.259+0000 7f85c6d9d640 1 -- 172.21.15.116:0/324906780 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85c8157660 con 0x7f85c8072810 2024-04-25T00:18:36.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.259+0000 7f85bffff640 1 --2- 172.21.15.116:0/324906780 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f85c8072c90 0x7f85c8150f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:18:36.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.263+0000 7f85c6d9d640 1 --2- 172.21.15.116:0/324906780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85c8072810 0x7f85c8153760 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f85c0009d90 tx=0x7f85c001c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d11463f5df17e62 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:36.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.263+0000 7f85c5d9b640 1 -- 172.21.15.116:0/324906780 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85c00b2070 con 0x7f85c8072810 2024-04-25T00:18:36.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.263+0000 7f85c5d9b640 1 -- 172.21.15.116:0/324906780 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85c00a5990 con 0x7f85c8072810 2024-04-25T00:18:36.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.263+0000 7f85c5d9b640 1 -- 172.21.15.116:0/324906780 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85c00a5d60 con 0x7f85c8072810 2024-04-25T00:18:36.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.263+0000 7f85ccc64640 1 -- 172.21.15.116:0/324906780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85c8072810 msgr2=0x7f85c8153760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:36.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.263+0000 7f85ccc64640 1 --2- 172.21.15.116:0/324906780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85c8072810 0x7f85c8153760 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f85c0009d90 tx=0x7f85c001c8b0 comp rx=0 tx=0).stop 2024-04-25T00:18:36.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.263+0000 7f85ccc64640 1 -- 172.21.15.116:0/324906780 shutdown_connections 2024-04-25T00:18:36.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.263+0000 7f85ccc64640 1 --2- 172.21.15.116:0/324906780 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85c8072810 0x7f85c8153760 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:36.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.263+0000 7f85ccc64640 1 --2- 172.21.15.116:0/324906780 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f85c8072c90 0x7f85c8150f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:36.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.263+0000 7f85ccc64640 1 -- 172.21.15.116:0/324906780 >> 172.21.15.116:0/324906780 conn(0x7f85c806c720 msgr2=0x7f85c806cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:36.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.263+0000 7f85ccc64640 1 -- 172.21.15.116:0/324906780 shutdown_connections 2024-04-25T00:18:36.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.263+0000 7f85ccc64640 1 -- 172.21.15.116:0/324906780 wait complete. 2024-04-25T00:18:36.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.263+0000 7f85ccc64640 1 Processor -- start 2024-04-25T00:18:36.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.263+0000 7f85ccc64640 1 -- start start 2024-04-25T00:18:36.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.267+0000 7f85ccc64640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f85c8072810 0x7f85c8147fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:36.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.267+0000 7f85ccc64640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85c8072c90 0x7f85c8155d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:36.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.267+0000 7f85ccc64640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f85c80c5280 0x7f85c81efc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:36.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.267+0000 7f85ccc64640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f85c8160380 con 0x7f85c8072c90 2024-04-25T00:18:36.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.267+0000 7f85ccc64640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f85c8160200 con 0x7f85c80c5280 2024-04-25T00:18:36.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.267+0000 7f85ccc64640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f85c8160500 con 0x7f85c8072810 2024-04-25T00:18:36.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.267+0000 7f85bffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85c8072c90 0x7f85c8155d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:36.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.267+0000 7f85bffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85c8072c90 0x7f85c8155d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60636/0 (socket says 172.21.15.116:60636) 2024-04-25T00:18:36.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.267+0000 7f85bffff640 1 -- 172.21.15.116:0/297087421 learned_addr learned my addr 172.21.15.116:0/297087421 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:18:36.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.267+0000 7f85c659c640 1 --2- 172.21.15.116:0/297087421 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f85c8072810 0x7f85c8147fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:36.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.267+0000 7f85c6d9d640 1 --2- 172.21.15.116:0/297087421 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f85c80c5280 0x7f85c81efc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:36.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.267+0000 7f85bffff640 1 -- 172.21.15.116:0/297087421 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f85c8072810 msgr2=0x7f85c8147fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:36.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.267+0000 7f85bffff640 1 --2- 172.21.15.116:0/297087421 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f85c8072810 0x7f85c8147fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:36.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.267+0000 7f85bffff640 1 -- 172.21.15.116:0/297087421 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f85c80c5280 msgr2=0x7f85c81efc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:36.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.267+0000 7f85bffff640 1 --2- 172.21.15.116:0/297087421 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f85c80c5280 0x7f85c81efc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:36.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.267+0000 7f85bffff640 1 -- 172.21.15.116:0/297087421 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85c81ecd70 con 0x7f85c8072c90 2024-04-25T00:18:36.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.267+0000 7f85c6d9d640 1 --2- 172.21.15.116:0/297087421 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f85c80c5280 0x7f85c81efc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:18:36.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.267+0000 7f85bffff640 1 --2- 172.21.15.116:0/297087421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85c8072c90 0x7f85c8155d00 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f85b000c7f0 tx=0x7f85b000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:36.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.267+0000 7f85beffd640 1 -- 172.21.15.116:0/297087421 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b0007a30 con 0x7f85c8072c90 2024-04-25T00:18:36.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.267+0000 7f85beffd640 1 -- 172.21.15.116:0/297087421 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85b0007bd0 con 0x7f85c8072c90 2024-04-25T00:18:36.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.267+0000 7f85ccc64640 1 -- 172.21.15.116:0/297087421 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85c81ec7b0 con 0x7f85c8072c90 2024-04-25T00:18:36.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.267+0000 7f85ccc64640 1 -- 172.21.15.116:0/297087421 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85c81ef470 con 0x7f85c8072c90 2024-04-25T00:18:36.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.267+0000 7f85beffd640 1 -- 172.21.15.116:0/297087421 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85b00113f0 con 0x7f85c8072c90 2024-04-25T00:18:36.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.267+0000 7f85ccc64640 1 -- 172.21.15.116:0/297087421 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85ac0052a0 con 0x7f85c8072c90 2024-04-25T00:18:36.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.267+0000 7f85beffd640 1 -- 172.21.15.116:0/297087421 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f85b0011590 con 0x7f85c8072c90 2024-04-25T00:18:36.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.267+0000 7f85beffd640 1 --2- 172.21.15.116:0/297087421 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f85a80678f0 0x7f85a8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:36.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.267+0000 7f85beffd640 1 -- 172.21.15.116:0/297087421 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 8988+0+0 (secure 0 0 0) 0x7f85b0086ee0 con 0x7f85c8072c90 2024-04-25T00:18:36.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.267+0000 7f85c659c640 1 --2- 172.21.15.116:0/297087421 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f85a80678f0 0x7f85a8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:36.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.271+0000 7f85c659c640 1 --2- 172.21.15.116:0/297087421 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f85a80678f0 0x7f85a8069db0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f85b8000f70 tx=0x7f85b8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:36.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.271+0000 7f85beffd640 1 -- 172.21.15.116:0/297087421 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f85b0015020 con 0x7f85c8072c90 2024-04-25T00:18:36.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.483+0000 7f85ccc64640 1 -- 172.21.15.116:0/297087421 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f85ac005590 con 0x7f85c8072c90 2024-04-25T00:18:36.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.487+0000 7f85beffd640 1 -- 172.21.15.116:0/297087421 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 143 v143) ==== 95+0+5673 (secure 0 0 0) 0x7f85b0056e10 con 0x7f85c8072c90 2024-04-25T00:18:36.488 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:18:36.488 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":143,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:18:23.899096+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7362},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7362":{"gid":7362,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6833/1655959433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1655959433},{"type":"v1","addr":"172.21.15.116:6833","nonce":1655959433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7362,"qdb_cluster":[7362]},"id":1},{"mdsmap":{"epoch":143,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:18:35.617345+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5879},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5879":{"gid":5879,"name":"d","rank":0,"incarnation":143,"state":"up:replay","state_seq":1,"addr":"172.21.15.174:6837/2276529656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2276529656},{"type":"v1","addr":"172.21.15.174:6837","nonce":2276529656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:18:36.488 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 143 2024-04-25T00:18:36.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.487+0000 7f85ccc64640 1 -- 172.21.15.116:0/297087421 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f85a80678f0 msgr2=0x7f85a8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:36.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.487+0000 7f85ccc64640 1 --2- 172.21.15.116:0/297087421 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f85a80678f0 0x7f85a8069db0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f85b8000f70 tx=0x7f85b8009040 comp rx=0 tx=0).stop 2024-04-25T00:18:36.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.487+0000 7f85ccc64640 1 -- 172.21.15.116:0/297087421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85c8072c90 msgr2=0x7f85c8155d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:36.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.487+0000 7f85ccc64640 1 --2- 172.21.15.116:0/297087421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85c8072c90 0x7f85c8155d00 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f85b000c7f0 tx=0x7f85b000ccb0 comp rx=0 tx=0).stop 2024-04-25T00:18:36.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.487+0000 7f85ccc64640 1 -- 172.21.15.116:0/297087421 shutdown_connections 2024-04-25T00:18:36.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.487+0000 7f85ccc64640 1 --2- 172.21.15.116:0/297087421 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f85a80678f0 0x7f85a8069db0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:36.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.487+0000 7f85ccc64640 1 --2- 172.21.15.116:0/297087421 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f85c80c5280 0x7f85c81efc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:36.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.487+0000 7f85ccc64640 1 --2- 172.21.15.116:0/297087421 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f85c8072c90 0x7f85c8155d00 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:36.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.487+0000 7f85ccc64640 1 --2- 172.21.15.116:0/297087421 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f85c8072810 0x7f85c8147fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:36.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.487+0000 7f85ccc64640 1 -- 172.21.15.116:0/297087421 >> 172.21.15.116:0/297087421 conn(0x7f85c806c720 msgr2=0x7f85c8156d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:36.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.487+0000 7f85ccc64640 1 -- 172.21.15.116:0/297087421 shutdown_connections 2024-04-25T00:18:36.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:36.487+0000 7f85ccc64640 1 -- 172.21.15.116:0/297087421 wait complete. 2024-04-25T00:18:36.507 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 143, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:04.678737+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 141, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_7362": { "addr": "172.21.15.116:6833/1655959433", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6832", "nonce": 1655959433, "type": "v2" }, { "addr": "172.21.15.116:6833", "nonce": 1655959433, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7362, "incarnation": 138, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 12 } }, "last_failure": 0, "last_failure_osd_epoch": 54, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T00:18:23.899096+0000", "qdb_cluster": [ 7362 ], "qdb_leader": 7362, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7362 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:18.825887+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 143, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_5879": { "addr": "172.21.15.174:6837/2276529656", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6836", "nonce": 2276529656, "type": "v2" }, { "addr": "172.21.15.174:6837", "nonce": 2276529656, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5879, "incarnation": 143, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 55, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-04-25T00:18:35.617345+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5879 } } } ], "standbys": [ { "addr": "172.21.15.174:6835/4112145782", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6834", "nonce": 4112145782, "type": "v2" }, { "addr": "172.21.15.174:6835", "nonce": 4112145782, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 112, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5582, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-25T00:18:38.508 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:18:38.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.643+0000 7f5304108640 1 Processor -- start 2024-04-25T00:18:38.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.643+0000 7f5304108640 1 -- start start 2024-04-25T00:18:38.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.647+0000 7f5304108640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f52fc0acef0 0x7f52fc1541d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:38.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.647+0000 7f5304108640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f52fc0c32d0 0x7f52fc0c3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:38.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.647+0000 7f5304108640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52fc0a9ef0 con 0x7f52fc0ad2d0 2024-04-25T00:18:38.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.647+0000 7f5304108640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f52fc0c2b60 con 0x7f52fc0c32d0 2024-04-25T00:18:38.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.647+0000 7f5304108640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f52fc0c2e40 con 0x7f52fc0acef0 2024-04-25T00:18:38.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.647+0000 7f53026a5640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f52fc0c32d0 0x7f52fc0c3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:38.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.647+0000 7f53016a3640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f52fc0ad2d0 0x7f52fc0a7d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:35498/0 (socket says 172.21.15.116:35498) 2024-04-25T00:18:38.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.647+0000 7f53016a3640 1 -- 172.21.15.116:0/2331720272 learned_addr learned my addr 172.21.15.116:0/2331720272 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:18:38.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.647+0000 7f5301ea4640 1 --2- 172.21.15.116:0/2331720272 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f52fc0acef0 0x7f52fc1541d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:38.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.647+0000 7f53026a5640 1 -- 172.21.15.116:0/2331720272 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f52fc0acef0 msgr2=0x7f52fc1541d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:38.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.647+0000 7f5300ea2640 1 -- 172.21.15.116:0/2331720272 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2232501778 0 0) 0x7f52fc0a9ef0 con 0x7f52fc0ad2d0 2024-04-25T00:18:38.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.647+0000 7f53026a5640 1 --2- 172.21.15.116:0/2331720272 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f52fc0acef0 0x7f52fc1541d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:38.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.647+0000 7f53026a5640 1 -- 172.21.15.116:0/2331720272 >> v1:172.21.15.116:6789/0 conn(0x7f52fc0ad2d0 legacy=0x7f52fc0a7d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:38.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.647+0000 7f53026a5640 1 -- 172.21.15.116:0/2331720272 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52fc0aa1d0 con 0x7f52fc0c32d0 2024-04-25T00:18:38.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.647+0000 7f5301ea4640 1 --2- 172.21.15.116:0/2331720272 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f52fc0acef0 0x7f52fc1541d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:18:38.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.647+0000 7f53026a5640 1 --2- 172.21.15.116:0/2331720272 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52fc0c32d0 0x7f52fc0c3740 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f52f4009d90 tx=0x7f52f401c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fb9f18ed25e6fb8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:38.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.647+0000 7f5300ea2640 1 -- 172.21.15.116:0/2331720272 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f40b2070 con 0x7f52fc0c32d0 2024-04-25T00:18:38.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.647+0000 7f5300ea2640 1 -- 172.21.15.116:0/2331720272 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52f40a57a0 con 0x7f52fc0c32d0 2024-04-25T00:18:38.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.647+0000 7f5300ea2640 1 -- 172.21.15.116:0/2331720272 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f40a5b70 con 0x7f52fc0c32d0 2024-04-25T00:18:38.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.647+0000 7f5304108640 1 -- 172.21.15.116:0/2331720272 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52fc0c32d0 msgr2=0x7f52fc0c3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:38.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.647+0000 7f5304108640 1 --2- 172.21.15.116:0/2331720272 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52fc0c32d0 0x7f52fc0c3740 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f52f4009d90 tx=0x7f52f401c8e0 comp rx=0 tx=0).stop 2024-04-25T00:18:38.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.647+0000 7f5304108640 1 -- 172.21.15.116:0/2331720272 shutdown_connections 2024-04-25T00:18:38.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.647+0000 7f5304108640 1 --2- 172.21.15.116:0/2331720272 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52fc0c32d0 0x7f52fc0c3740 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:38.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.647+0000 7f5304108640 1 --2- 172.21.15.116:0/2331720272 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f52fc0acef0 0x7f52fc1541d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:38.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.647+0000 7f5304108640 1 -- 172.21.15.116:0/2331720272 >> 172.21.15.116:0/2331720272 conn(0x7f52fc0c4200 msgr2=0x7f52fc0c4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:38.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.651+0000 7f5304108640 1 -- 172.21.15.116:0/2331720272 shutdown_connections 2024-04-25T00:18:38.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.651+0000 7f5304108640 1 -- 172.21.15.116:0/2331720272 wait complete. 2024-04-25T00:18:38.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.651+0000 7f5304108640 1 Processor -- start 2024-04-25T00:18:38.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.651+0000 7f5304108640 1 -- start start 2024-04-25T00:18:38.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.651+0000 7f5304108640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52fc0acef0 0x7f52fc152760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:38.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.651+0000 7f5304108640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52fc0ad2d0 0x7f52fc14cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:38.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.651+0000 7f5304108640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f52fc0c32d0 0x7f52fc145440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:38.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.651+0000 7f5304108640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f52fc1609e0 con 0x7f52fc0acef0 2024-04-25T00:18:38.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.651+0000 7f5304108640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f52fc160860 con 0x7f52fc0ad2d0 2024-04-25T00:18:38.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.651+0000 7f5304108640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f52fc160b60 con 0x7f52fc0c32d0 2024-04-25T00:18:38.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.651+0000 7f5301ea4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52fc0acef0 0x7f52fc152760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:38.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.651+0000 7f5301ea4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52fc0acef0 0x7f52fc152760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60656/0 (socket says 172.21.15.116:60656) 2024-04-25T00:18:38.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.651+0000 7f5301ea4640 1 -- 172.21.15.116:0/1797867495 learned_addr learned my addr 172.21.15.116:0/1797867495 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:18:38.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.651+0000 7f53026a5640 1 --2- 172.21.15.116:0/1797867495 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f52fc0c32d0 0x7f52fc145440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:38.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.651+0000 7f53016a3640 1 --2- 172.21.15.116:0/1797867495 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52fc0ad2d0 0x7f52fc14cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:38.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.651+0000 7f5301ea4640 1 -- 172.21.15.116:0/1797867495 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f52fc0c32d0 msgr2=0x7f52fc145440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:38.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.651+0000 7f5301ea4640 1 --2- 172.21.15.116:0/1797867495 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f52fc0c32d0 0x7f52fc145440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:38.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.651+0000 7f5301ea4640 1 -- 172.21.15.116:0/1797867495 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52fc0ad2d0 msgr2=0x7f52fc14cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:38.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.651+0000 7f5301ea4640 1 --2- 172.21.15.116:0/1797867495 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52fc0ad2d0 0x7f52fc14cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:38.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.651+0000 7f5301ea4640 1 -- 172.21.15.116:0/1797867495 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52fc152d40 con 0x7f52fc0acef0 2024-04-25T00:18:38.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.651+0000 7f53016a3640 1 --2- 172.21.15.116:0/1797867495 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52fc0ad2d0 0x7f52fc14cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:18:38.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.651+0000 7f53026a5640 1 --2- 172.21.15.116:0/1797867495 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f52fc0c32d0 0x7f52fc145440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:18:38.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.651+0000 7f5301ea4640 1 --2- 172.21.15.116:0/1797867495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52fc0acef0 0x7f52fc152760 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f52f800c7f0 tx=0x7f52f800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:38.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.651+0000 7f52f27fc640 1 -- 172.21.15.116:0/1797867495 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f8007a30 con 0x7f52fc0acef0 2024-04-25T00:18:38.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.651+0000 7f52f27fc640 1 -- 172.21.15.116:0/1797867495 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52f8007bd0 con 0x7f52fc0acef0 2024-04-25T00:18:38.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.651+0000 7f5304108640 1 -- 172.21.15.116:0/1797867495 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52fc153020 con 0x7f52fc0acef0 2024-04-25T00:18:38.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.651+0000 7f52f27fc640 1 -- 172.21.15.116:0/1797867495 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f80113f0 con 0x7f52fc0acef0 2024-04-25T00:18:38.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.651+0000 7f5304108640 1 -- 172.21.15.116:0/1797867495 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52fc1535e0 con 0x7f52fc0acef0 2024-04-25T00:18:38.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.655+0000 7f5304108640 1 -- 172.21.15.116:0/1797867495 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52fc068a60 con 0x7f52fc0acef0 2024-04-25T00:18:38.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.655+0000 7f52f27fc640 1 -- 172.21.15.116:0/1797867495 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f52f8011590 con 0x7f52fc0acef0 2024-04-25T00:18:38.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.655+0000 7f52f27fc640 1 --2- 172.21.15.116:0/1797867495 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f52e00678c0 0x7f52e0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:38.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.655+0000 7f52f27fc640 1 -- 172.21.15.116:0/1797867495 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 8988+0+0 (secure 0 0 0) 0x7f52f8086bb0 con 0x7f52fc0acef0 2024-04-25T00:18:38.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.655+0000 7f53016a3640 1 --2- 172.21.15.116:0/1797867495 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f52e00678c0 0x7f52e0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:38.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.655+0000 7f53016a3640 1 --2- 172.21.15.116:0/1797867495 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f52e00678c0 0x7f52e0069d80 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f52fc06cd50 tx=0x7f52ec009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:38.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.659+0000 7f52f27fc640 1 -- 172.21.15.116:0/1797867495 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f52f8015020 con 0x7f52fc0acef0 2024-04-25T00:18:38.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.907+0000 7f5304108640 1 -- 172.21.15.116:0/1797867495 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f52fc06c320 con 0x7f52fc0acef0 2024-04-25T00:18:38.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.907+0000 7f52f27fc640 1 -- 172.21.15.116:0/1797867495 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) ==== 95+0+5680 (secure 0 0 0) 0x7f52f80569d0 con 0x7f52fc0acef0 2024-04-25T00:18:38.909 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:18:38.909 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":146,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:18:23.899096+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7362},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7362":{"gid":7362,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6833/1655959433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1655959433},{"type":"v1","addr":"172.21.15.116:6833","nonce":1655959433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7362,"qdb_cluster":[7362]},"id":1},{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:18:38.800778+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5879},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5879":{"gid":5879,"name":"d","rank":0,"incarnation":143,"state":"up:active","state_seq":6,"addr":"172.21.15.174:6837/2276529656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2276529656},{"type":"v1","addr":"172.21.15.174:6837","nonce":2276529656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5879,"qdb_cluster":[5879]},"id":2}]} 2024-04-25T00:18:38.909 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 146 2024-04-25T00:18:38.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.907+0000 7f5304108640 1 -- 172.21.15.116:0/1797867495 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f52e00678c0 msgr2=0x7f52e0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:38.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.907+0000 7f5304108640 1 --2- 172.21.15.116:0/1797867495 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f52e00678c0 0x7f52e0069d80 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f52fc06cd50 tx=0x7f52ec009040 comp rx=0 tx=0).stop 2024-04-25T00:18:38.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.907+0000 7f5304108640 1 -- 172.21.15.116:0/1797867495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52fc0acef0 msgr2=0x7f52fc152760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:38.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.907+0000 7f5304108640 1 --2- 172.21.15.116:0/1797867495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52fc0acef0 0x7f52fc152760 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f52f800c7f0 tx=0x7f52f800ccb0 comp rx=0 tx=0).stop 2024-04-25T00:18:38.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.907+0000 7f5304108640 1 -- 172.21.15.116:0/1797867495 shutdown_connections 2024-04-25T00:18:38.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.907+0000 7f5304108640 1 --2- 172.21.15.116:0/1797867495 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f52e00678c0 0x7f52e0069d80 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:38.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.907+0000 7f5304108640 1 --2- 172.21.15.116:0/1797867495 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f52fc0c32d0 0x7f52fc145440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:38.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.907+0000 7f5304108640 1 --2- 172.21.15.116:0/1797867495 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52fc0ad2d0 0x7f52fc14cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:38.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.907+0000 7f5304108640 1 --2- 172.21.15.116:0/1797867495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52fc0acef0 0x7f52fc152760 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:38.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.907+0000 7f5304108640 1 -- 172.21.15.116:0/1797867495 >> 172.21.15.116:0/1797867495 conn(0x7f52fc0c4200 msgr2=0x7f52fc0c5d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:38.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.907+0000 7f5304108640 1 -- 172.21.15.116:0/1797867495 shutdown_connections 2024-04-25T00:18:38.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:38.907+0000 7f5304108640 1 -- 172.21.15.116:0/1797867495 wait complete. 2024-04-25T00:18:38.932 INFO:tasks.mds_thrash.fs.[b]:mds.d has gained rank=0, replacing gid=7164 2024-04-25T00:18:38.932 INFO:tasks.mds_thrash.fs.[b]:waiting for 6 secs before reviving mds.c 2024-04-25T00:18:39.280 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:18:39.282 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:18:40.929 INFO:tasks.ceph.mds.d.smithi174.stderr:2024-04-25T00:18:40.926+0000 7f7d3a230640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T00:18:44.933 INFO:tasks.mds_thrash.fs.[b]:reviving mds.c 2024-04-25T00:18:44.933 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-25T00:18:44.933 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-25T00:18:44.936 INFO:tasks.ceph.mds.c:Started 2024-04-25T00:18:44.937 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:18:45.016 INFO:tasks.ceph.mds.c.smithi116.stdout:starting mds.c at 2024-04-25T00:18:45.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.063+0000 7f4ac6398640 1 Processor -- start 2024-04-25T00:18:45.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.063+0000 7f4ac6398640 1 -- start start 2024-04-25T00:18:45.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.067+0000 7f4ac6398640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f4ac0152d60 0x7f4ac00ab250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:45.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.067+0000 7f4ac6398640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ac00ab790 0x7f4ac00c8450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:45.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.067+0000 7f4ac6398640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ac0156760 con 0x7f4ac0153140 2024-04-25T00:18:45.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.067+0000 7f4ac6398640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f4ac0155be0 con 0x7f4ac0152d60 2024-04-25T00:18:45.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.067+0000 7f4ac6398640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f4ac0155ec0 con 0x7f4ac00ab790 2024-04-25T00:18:45.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.067+0000 7f4ac4935640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f4ac0153140 0x7f4ac00c3c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:37366/0 (socket says 172.21.15.116:37366) 2024-04-25T00:18:45.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.067+0000 7f4ac4935640 1 -- 172.21.15.116:0/1246344270 learned_addr learned my addr 172.21.15.116:0/1246344270 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:18:45.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.067+0000 7f4abf7fe640 1 --2- 172.21.15.116:0/1246344270 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ac00ab790 0x7f4ac00c8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:45.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.067+0000 7f4abffff640 1 --2- 172.21.15.116:0/1246344270 >> v2:172.21.15.174:3300/0 conn(0x7f4ac0152d60 0x7f4ac00ab250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:45.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.067+0000 7f4abeffd640 1 -- 172.21.15.116:0/1246344270 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1719675018 0 0) 0x7f4ac0156760 con 0x7f4ac0153140 2024-04-25T00:18:45.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.067+0000 7f4abeffd640 1 -- 172.21.15.116:0/1246344270 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4aa8003530 con 0x7f4ac0153140 2024-04-25T00:18:45.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.067+0000 7f4abeffd640 1 -- 172.21.15.116:0/1246344270 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2715432237 0 0) 0x7f4aa8003530 con 0x7f4ac0153140 2024-04-25T00:18:45.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.067+0000 7f4abeffd640 1 -- 172.21.15.116:0/1246344270 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ac00ab790 msgr2=0x7f4ac00c8450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:45.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.067+0000 7f4abeffd640 1 --2- 172.21.15.116:0/1246344270 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ac00ab790 0x7f4ac00c8450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:45.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.067+0000 7f4abeffd640 1 -- 172.21.15.116:0/1246344270 >> v2:172.21.15.174:3300/0 conn(0x7f4ac0152d60 msgr2=0x7f4ac00ab250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:45.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.067+0000 7f4abeffd640 1 --2- 172.21.15.116:0/1246344270 >> v2:172.21.15.174:3300/0 conn(0x7f4ac0152d60 0x7f4ac00ab250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:45.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.067+0000 7f4abeffd640 1 -- 172.21.15.116:0/1246344270 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ac01561a0 con 0x7f4ac0153140 2024-04-25T00:18:45.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.067+0000 7f4abeffd640 1 -- 172.21.15.116:0/1246344270 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f4ab8001a00 con 0x7f4ac0153140 2024-04-25T00:18:45.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.067+0000 7f4abeffd640 1 -- 172.21.15.116:0/1246344270 >> v1:172.21.15.116:6789/0 conn(0x7f4ac0153140 legacy=0x7f4ac00c3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:45.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.067+0000 7f4abeffd640 1 --2- 172.21.15.116:0/1246344270 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ac00ab790 0x7f4aa80039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:45.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.067+0000 7f4abeffd640 1 --2- 172.21.15.116:0/1246344270 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4aa8003f40 0x7f4aa8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:45.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.067+0000 7f4abeffd640 1 --2- 172.21.15.116:0/1246344270 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4aa8004d50 0x7f4aa80051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:45.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.067+0000 7f4abeffd640 1 -- 172.21.15.116:0/1246344270 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4ac0155be0 con 0x7f4aa8004d50 2024-04-25T00:18:45.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.067+0000 7f4abeffd640 1 -- 172.21.15.116:0/1246344270 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4ac0155ec0 con 0x7f4aa8003f40 2024-04-25T00:18:45.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.067+0000 7f4abeffd640 1 -- 172.21.15.116:0/1246344270 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f4aa800ba20 con 0x7f4ac00ab790 2024-04-25T00:18:45.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.067+0000 7f4ac4935640 1 --2- 172.21.15.116:0/1246344270 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4aa8004d50 0x7f4aa80051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:45.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.067+0000 7f4abffff640 1 --2- 172.21.15.116:0/1246344270 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ac00ab790 0x7f4aa80039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:45.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.067+0000 7f4abf7fe640 1 --2- 172.21.15.116:0/1246344270 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4aa8003f40 0x7f4aa8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:45.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.067+0000 7f4ac4935640 1 -- 172.21.15.116:0/1246344270 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ac00ab790 msgr2=0x7f4aa80039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:45.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.067+0000 7f4ac4935640 1 --2- 172.21.15.116:0/1246344270 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ac00ab790 0x7f4aa80039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:45.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.067+0000 7f4ac4935640 1 -- 172.21.15.116:0/1246344270 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4aa8003f40 msgr2=0x7f4aa8004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:45.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.067+0000 7f4ac4935640 1 --2- 172.21.15.116:0/1246344270 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4aa8003f40 0x7f4aa8004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:45.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.067+0000 7f4ac4935640 1 -- 172.21.15.116:0/1246344270 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4aa8003530 con 0x7f4aa8004d50 2024-04-25T00:18:45.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.067+0000 7f4abffff640 1 --2- 172.21.15.116:0/1246344270 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ac00ab790 0x7f4aa80039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:18:45.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.067+0000 7f4abf7fe640 1 --2- 172.21.15.116:0/1246344270 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4aa8003f40 0x7f4aa8004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:18:45.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.071+0000 7f4ac4935640 1 --2- 172.21.15.116:0/1246344270 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4aa8004d50 0x7f4aa80051d0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f4ab800a8b0 tx=0x7f4ab801fb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb189ef04e77ac2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:45.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.071+0000 7f4abeffd640 1 -- 172.21.15.116:0/1246344270 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ab800a4a0 con 0x7f4aa8004d50 2024-04-25T00:18:45.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.071+0000 7f4abeffd640 1 -- 172.21.15.116:0/1246344270 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ab80ad020 con 0x7f4aa8004d50 2024-04-25T00:18:45.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.071+0000 7f4abeffd640 1 -- 172.21.15.116:0/1246344270 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ab80a9740 con 0x7f4aa8004d50 2024-04-25T00:18:45.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.071+0000 7f4ac6398640 1 -- 172.21.15.116:0/1246344270 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4aa8004d50 msgr2=0x7f4aa80051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:45.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.071+0000 7f4ac6398640 1 --2- 172.21.15.116:0/1246344270 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4aa8004d50 0x7f4aa80051d0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f4ab800a8b0 tx=0x7f4ab801fb00 comp rx=0 tx=0).stop 2024-04-25T00:18:45.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.071+0000 7f4ac4935640 1 -- 172.21.15.116:0/1246344270 reap_dead start 2024-04-25T00:18:45.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.071+0000 7f4ac6398640 1 -- 172.21.15.116:0/1246344270 shutdown_connections 2024-04-25T00:18:45.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.071+0000 7f4ac6398640 1 -- 172.21.15.116:0/1246344270 >> 172.21.15.116:0/1246344270 conn(0x7f4ac006f0c0 msgr2=0x7f4ac006f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:45.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.071+0000 7f4ac6398640 1 -- 172.21.15.116:0/1246344270 shutdown_connections 2024-04-25T00:18:45.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.071+0000 7f4ac6398640 1 -- 172.21.15.116:0/1246344270 wait complete. 2024-04-25T00:18:45.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.075+0000 7f4ac6398640 1 Processor -- start 2024-04-25T00:18:45.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.075+0000 7f4ac6398640 1 -- start start 2024-04-25T00:18:45.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.075+0000 7f4ac6398640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ac00ad280 0x7f4ac014c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:45.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.075+0000 7f4ac6398640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ac014ab50 0x7f4ac014f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:45.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.075+0000 7f4ac6398640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ac014b080 0x7f4ac0149e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:45.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.075+0000 7f4ac6398640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4ac0155980 con 0x7f4ac00ad280 2024-04-25T00:18:45.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.075+0000 7f4ac6398640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4ac0155800 con 0x7f4ac014b080 2024-04-25T00:18:45.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.075+0000 7f4ac6398640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f4ac00c43a0 con 0x7f4ac014ab50 2024-04-25T00:18:45.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.075+0000 7f4abffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ac00ad280 0x7f4ac014c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:45.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.075+0000 7f4abffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ac00ad280 0x7f4ac014c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37642/0 (socket says 172.21.15.116:37642) 2024-04-25T00:18:45.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.075+0000 7f4abffff640 1 -- 172.21.15.116:0/403883355 learned_addr learned my addr 172.21.15.116:0/403883355 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:18:45.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.075+0000 7f4abf7fe640 1 --2- 172.21.15.116:0/403883355 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ac014ab50 0x7f4ac014f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:45.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.075+0000 7f4ac4935640 1 --2- 172.21.15.116:0/403883355 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ac014b080 0x7f4ac0149e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:45.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.075+0000 7f4abffff640 1 -- 172.21.15.116:0/403883355 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ac014ab50 msgr2=0x7f4ac014f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:45.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.075+0000 7f4abffff640 1 --2- 172.21.15.116:0/403883355 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ac014ab50 0x7f4ac014f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:45.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.075+0000 7f4abffff640 1 -- 172.21.15.116:0/403883355 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ac014b080 msgr2=0x7f4ac0149e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:45.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.075+0000 7f4abffff640 1 --2- 172.21.15.116:0/403883355 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ac014b080 0x7f4ac0149e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:45.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.075+0000 7f4abffff640 1 -- 172.21.15.116:0/403883355 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ac01e24b0 con 0x7f4ac00ad280 2024-04-25T00:18:45.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.075+0000 7f4abffff640 1 --2- 172.21.15.116:0/403883355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ac00ad280 0x7f4ac014c090 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f4ab00116a0 tx=0x7f4ab0011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:45.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.075+0000 7f4abcff9640 1 -- 172.21.15.116:0/403883355 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ab0004220 con 0x7f4ac00ad280 2024-04-25T00:18:45.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.075+0000 7f4abcff9640 1 -- 172.21.15.116:0/403883355 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ab00043c0 con 0x7f4ac00ad280 2024-04-25T00:18:45.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.075+0000 7f4ac6398640 1 -- 172.21.15.116:0/403883355 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ac01e2790 con 0x7f4ac00ad280 2024-04-25T00:18:45.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.075+0000 7f4abcff9640 1 -- 172.21.15.116:0/403883355 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ab000f980 con 0x7f4ac00ad280 2024-04-25T00:18:45.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.075+0000 7f4ac6398640 1 -- 172.21.15.116:0/403883355 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ac014fd20 con 0x7f4ac00ad280 2024-04-25T00:18:45.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.075+0000 7f4ac6398640 1 -- 172.21.15.116:0/403883355 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ac0068a60 con 0x7f4ac00ad280 2024-04-25T00:18:45.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.075+0000 7f4abcff9640 1 -- 172.21.15.116:0/403883355 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4ab000fb20 con 0x7f4ac00ad280 2024-04-25T00:18:45.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.075+0000 7f4abcff9640 1 --2- 172.21.15.116:0/403883355 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4aac0678c0 0x7f4aac069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:45.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.079+0000 7f4abcff9640 1 -- 172.21.15.116:0/403883355 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 8988+0+0 (secure 0 0 0) 0x7f4ab0086f20 con 0x7f4ac00ad280 2024-04-25T00:18:45.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.079+0000 7f4abf7fe640 1 --2- 172.21.15.116:0/403883355 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4aac0678c0 0x7f4aac069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:45.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.079+0000 7f4abf7fe640 1 --2- 172.21.15.116:0/403883355 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4aac0678c0 0x7f4aac069d80 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f4ac01ec040 tx=0x7f4ab4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:45.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.083+0000 7f4abcff9640 1 -- 172.21.15.116:0/403883355 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f4ab0019030 con 0x7f4ac00ad280 2024-04-25T00:18:45.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.347+0000 7f4ac6398640 1 -- 172.21.15.116:0/403883355 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4ac0152940 con 0x7f4ac00ad280 2024-04-25T00:18:45.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.347+0000 7f4abcff9640 1 -- 172.21.15.116:0/403883355 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) ==== 95+0+5680 (secure 0 0 0) 0x7f4ab0056dd0 con 0x7f4ac00ad280 2024-04-25T00:18:45.351 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:18:45.351 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":146,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:18:23.899096+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7362},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7362":{"gid":7362,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6833/1655959433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1655959433},{"type":"v1","addr":"172.21.15.116:6833","nonce":1655959433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7362,"qdb_cluster":[7362]},"id":1},{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:18:38.800778+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5879},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5879":{"gid":5879,"name":"d","rank":0,"incarnation":143,"state":"up:active","state_seq":6,"addr":"172.21.15.174:6837/2276529656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2276529656},{"type":"v1","addr":"172.21.15.174:6837","nonce":2276529656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5879,"qdb_cluster":[5879]},"id":2}]} 2024-04-25T00:18:45.351 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 146 2024-04-25T00:18:45.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.351+0000 7f4ac6398640 1 -- 172.21.15.116:0/403883355 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4aac0678c0 msgr2=0x7f4aac069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:45.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.351+0000 7f4ac6398640 1 --2- 172.21.15.116:0/403883355 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4aac0678c0 0x7f4aac069d80 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f4ac01ec040 tx=0x7f4ab4008040 comp rx=0 tx=0).stop 2024-04-25T00:18:45.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.351+0000 7f4ac6398640 1 -- 172.21.15.116:0/403883355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ac00ad280 msgr2=0x7f4ac014c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:45.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.351+0000 7f4ac6398640 1 --2- 172.21.15.116:0/403883355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ac00ad280 0x7f4ac014c090 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f4ab00116a0 tx=0x7f4ab0011b60 comp rx=0 tx=0).stop 2024-04-25T00:18:45.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.351+0000 7f4ac6398640 1 -- 172.21.15.116:0/403883355 shutdown_connections 2024-04-25T00:18:45.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.351+0000 7f4ac6398640 1 --2- 172.21.15.116:0/403883355 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4aac0678c0 0x7f4aac069d80 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:45.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.351+0000 7f4ac6398640 1 --2- 172.21.15.116:0/403883355 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4ac014b080 0x7f4ac0149e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:45.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.351+0000 7f4ac6398640 1 --2- 172.21.15.116:0/403883355 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4ac014ab50 0x7f4ac014f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:45.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.351+0000 7f4ac6398640 1 --2- 172.21.15.116:0/403883355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4ac00ad280 0x7f4ac014c090 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:45.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.351+0000 7f4ac6398640 1 -- 172.21.15.116:0/403883355 >> 172.21.15.116:0/403883355 conn(0x7f4ac006f0c0 msgr2=0x7f4ac006f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:45.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.351+0000 7f4ac6398640 1 -- 172.21.15.116:0/403883355 shutdown_connections 2024-04-25T00:18:45.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:45.351+0000 7f4ac6398640 1 -- 172.21.15.116:0/403883355 wait complete. 2024-04-25T00:18:45.373 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2024-04-25T00:18:47.374 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:18:47.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.511+0000 7f1c0d752640 1 Processor -- start 2024-04-25T00:18:47.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.511+0000 7f1c0d752640 1 -- start start 2024-04-25T00:18:47.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.511+0000 7f1c0d752640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1c08148f20 0x7f1c080c3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:47.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.511+0000 7f1c0d752640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c080c2b40 con 0x7f1c080c3300 2024-04-25T00:18:47.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.511+0000 7f1c0d752640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c081533f0 con 0x7f1c08152f70 2024-04-25T00:18:47.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.511+0000 7f1c0d752640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f1c080c4790 con 0x7f1c08148f20 2024-04-25T00:18:47.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.511+0000 7f1c06ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1c08148f20 0x7f1c080c3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:47.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.511+0000 7f1c06ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1c08148f20 0x7f1c080c3ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37650/0 (socket says 172.21.15.116:37650) 2024-04-25T00:18:47.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.511+0000 7f1c067fc640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f1c08152f70 0x7f1c0806e480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:53860/0 (socket says 172.21.15.116:53860) 2024-04-25T00:18:47.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.511+0000 7f1c077fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f1c080c3300 0x7f1c08071d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:37368/0 (socket says 172.21.15.116:37368) 2024-04-25T00:18:47.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.511+0000 7f1c06ffd640 1 -- 172.21.15.116:0/2254587425 learned_addr learned my addr 172.21.15.116:0/2254587425 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:18:47.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.511+0000 7f1c06ffd640 1 -- 172.21.15.116:0/2254587425 >> v1:172.21.15.174:6789/0 conn(0x7f1c08152f70 legacy=0x7f1c0806e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:47.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.511+0000 7f1c06ffd640 1 -- 172.21.15.116:0/2254587425 >> v1:172.21.15.116:6789/0 conn(0x7f1c080c3300 legacy=0x7f1c08071d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:47.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.511+0000 7f1c06ffd640 1 -- 172.21.15.116:0/2254587425 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c080c41d0 con 0x7f1c08148f20 2024-04-25T00:18:47.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.515+0000 7f1c06ffd640 1 --2- 172.21.15.116:0/2254587425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c08148f20 0x7f1c080c3ad0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f1bf8009870 tx=0x7f1bf801c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=646efe13d41b1b66 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:47.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.515+0000 7f1c05ffb640 1 -- 172.21.15.116:0/2254587425 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bf80a9620 con 0x7f1c08148f20 2024-04-25T00:18:47.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.515+0000 7f1c05ffb640 1 -- 172.21.15.116:0/2254587425 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bf80a97c0 con 0x7f1c08148f20 2024-04-25T00:18:47.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.515+0000 7f1c05ffb640 1 -- 172.21.15.116:0/2254587425 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bf80a9b90 con 0x7f1c08148f20 2024-04-25T00:18:47.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.515+0000 7f1c0d752640 1 -- 172.21.15.116:0/2254587425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c08148f20 msgr2=0x7f1c080c3ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:47.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.515+0000 7f1c0d752640 1 --2- 172.21.15.116:0/2254587425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c08148f20 0x7f1c080c3ad0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f1bf8009870 tx=0x7f1bf801c560 comp rx=0 tx=0).stop 2024-04-25T00:18:47.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.515+0000 7f1c0d752640 1 -- 172.21.15.116:0/2254587425 shutdown_connections 2024-04-25T00:18:47.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.515+0000 7f1c0d752640 1 --2- 172.21.15.116:0/2254587425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c08148f20 0x7f1c080c3ad0 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:47.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.515+0000 7f1c0d752640 1 -- 172.21.15.116:0/2254587425 >> 172.21.15.116:0/2254587425 conn(0x7f1c080c7fa0 msgr2=0x7f1c080c83b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:47.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.515+0000 7f1c0d752640 1 -- 172.21.15.116:0/2254587425 shutdown_connections 2024-04-25T00:18:47.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.515+0000 7f1c0d752640 1 -- 172.21.15.116:0/2254587425 wait complete. 2024-04-25T00:18:47.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.515+0000 7f1c0d752640 1 Processor -- start 2024-04-25T00:18:47.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.515+0000 7f1c0d752640 1 -- start start 2024-04-25T00:18:47.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.515+0000 7f1c0d752640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1c080c3300 0x7f1c08151370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:47.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.515+0000 7f1c0d752640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1c08148f20 0x7f1c081518b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:47.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.515+0000 7f1c0d752640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c08152f70 0x7f1c080c7300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:47.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.515+0000 7f1c0d752640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1c08149300 con 0x7f1c08152f70 2024-04-25T00:18:47.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.515+0000 7f1c0d752640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1c080c3710 con 0x7f1c08148f20 2024-04-25T00:18:47.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.515+0000 7f1c0d752640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f1c081337c0 con 0x7f1c080c3300 2024-04-25T00:18:47.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.519+0000 7f1c077fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c08152f70 0x7f1c080c7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:47.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.519+0000 7f1c077fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c08152f70 0x7f1c080c7300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37664/0 (socket says 172.21.15.116:37664) 2024-04-25T00:18:47.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.519+0000 7f1c077fe640 1 -- 172.21.15.116:0/2995220119 learned_addr learned my addr 172.21.15.116:0/2995220119 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:18:47.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.519+0000 7f1c06ffd640 1 --2- 172.21.15.116:0/2995220119 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1c080c3300 0x7f1c08151370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:47.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.519+0000 7f1c067fc640 1 --2- 172.21.15.116:0/2995220119 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1c08148f20 0x7f1c081518b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:47.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.519+0000 7f1c077fe640 1 -- 172.21.15.116:0/2995220119 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1c080c3300 msgr2=0x7f1c08151370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:47.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.519+0000 7f1c077fe640 1 --2- 172.21.15.116:0/2995220119 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1c080c3300 0x7f1c08151370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:47.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.519+0000 7f1c077fe640 1 -- 172.21.15.116:0/2995220119 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1c08148f20 msgr2=0x7f1c081518b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:47.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.519+0000 7f1c077fe640 1 --2- 172.21.15.116:0/2995220119 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1c08148f20 0x7f1c081518b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:47.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.519+0000 7f1c077fe640 1 -- 172.21.15.116:0/2995220119 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c08070280 con 0x7f1c08152f70 2024-04-25T00:18:47.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.519+0000 7f1c077fe640 1 --2- 172.21.15.116:0/2995220119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c08152f70 0x7f1c080c7300 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f1c0000c700 tx=0x7f1c0000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:47.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.519+0000 7f1bef7fe640 1 -- 172.21.15.116:0/2995220119 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c00017070 con 0x7f1c08152f70 2024-04-25T00:18:47.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.519+0000 7f1bef7fe640 1 -- 172.21.15.116:0/2995220119 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c0000ce10 con 0x7f1c08152f70 2024-04-25T00:18:47.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.519+0000 7f1c0d752640 1 -- 172.21.15.116:0/2995220119 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c081ec8c0 con 0x7f1c08152f70 2024-04-25T00:18:47.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.519+0000 7f1bef7fe640 1 -- 172.21.15.116:0/2995220119 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c0000a840 con 0x7f1c08152f70 2024-04-25T00:18:47.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.519+0000 7f1c0d752640 1 -- 172.21.15.116:0/2995220119 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c080c6b00 con 0x7f1c08152f70 2024-04-25T00:18:47.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.519+0000 7f1c0d752640 1 -- 172.21.15.116:0/2995220119 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c08068a60 con 0x7f1c08152f70 2024-04-25T00:18:47.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.519+0000 7f1bef7fe640 1 -- 172.21.15.116:0/2995220119 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1c0001a070 con 0x7f1c08152f70 2024-04-25T00:18:47.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.519+0000 7f1bef7fe640 1 --2- 172.21.15.116:0/2995220119 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1bf40678c0 0x7f1bf4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:47.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.519+0000 7f1bef7fe640 1 -- 172.21.15.116:0/2995220119 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 8988+0+0 (secure 0 0 0) 0x7f1c00085df0 con 0x7f1c08152f70 2024-04-25T00:18:47.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.519+0000 7f1c06ffd640 1 --2- 172.21.15.116:0/2995220119 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1bf40678c0 0x7f1bf4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:47.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.523+0000 7f1c06ffd640 1 --2- 172.21.15.116:0/2995220119 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1bf40678c0 0x7f1bf4069d80 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f1bf80ae000 tx=0x7f1bf80b3040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:47.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.527+0000 7f1bef7fe640 1 -- 172.21.15.116:0/2995220119 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f1c00055650 con 0x7f1c08152f70 2024-04-25T00:18:47.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.803+0000 7f1c0d752640 1 -- 172.21.15.116:0/2995220119 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1c0813e9a0 con 0x7f1c08152f70 2024-04-25T00:18:47.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.803+0000 7f1bef7fe640 1 -- 172.21.15.116:0/2995220119 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 147 v147) ==== 95+0+6506 (secure 0 0 0) 0x7f1c00059950 con 0x7f1c08152f70 2024-04-25T00:18:47.803 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:18:47.803 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":147,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7542,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/879531976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":879531976},{"type":"v1","addr":"172.21.15.116:6835","nonce":879531976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":147}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:18:23.899096+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7362},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7362":{"gid":7362,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6833/1655959433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1655959433},{"type":"v1","addr":"172.21.15.116:6833","nonce":1655959433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7362,"qdb_cluster":[7362]},"id":1},{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:18:38.800778+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5879},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5879":{"gid":5879,"name":"d","rank":0,"incarnation":143,"state":"up:active","state_seq":6,"addr":"172.21.15.174:6837/2276529656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2276529656},{"type":"v1","addr":"172.21.15.174:6837","nonce":2276529656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5879,"qdb_cluster":[5879]},"id":2}]} 2024-04-25T00:18:47.804 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 147 2024-04-25T00:18:47.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.807+0000 7f1c0d752640 1 -- 172.21.15.116:0/2995220119 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1bf40678c0 msgr2=0x7f1bf4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:47.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.807+0000 7f1c0d752640 1 --2- 172.21.15.116:0/2995220119 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1bf40678c0 0x7f1bf4069d80 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f1bf80ae000 tx=0x7f1bf80b3040 comp rx=0 tx=0).stop 2024-04-25T00:18:47.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.807+0000 7f1c0d752640 1 -- 172.21.15.116:0/2995220119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c08152f70 msgr2=0x7f1c080c7300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:47.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.807+0000 7f1c0d752640 1 --2- 172.21.15.116:0/2995220119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c08152f70 0x7f1c080c7300 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f1c0000c700 tx=0x7f1c0000cbc0 comp rx=0 tx=0).stop 2024-04-25T00:18:47.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.807+0000 7f1c0d752640 1 -- 172.21.15.116:0/2995220119 shutdown_connections 2024-04-25T00:18:47.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.807+0000 7f1c0d752640 1 --2- 172.21.15.116:0/2995220119 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1bf40678c0 0x7f1bf4069d80 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:47.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.807+0000 7f1c0d752640 1 --2- 172.21.15.116:0/2995220119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1c08152f70 0x7f1c080c7300 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:47.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.807+0000 7f1c0d752640 1 --2- 172.21.15.116:0/2995220119 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1c08148f20 0x7f1c081518b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:47.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.807+0000 7f1c0d752640 1 --2- 172.21.15.116:0/2995220119 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1c080c3300 0x7f1c08151370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:47.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.807+0000 7f1c0d752640 1 -- 172.21.15.116:0/2995220119 >> 172.21.15.116:0/2995220119 conn(0x7f1c080c7fa0 msgr2=0x7f1c080c8380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:47.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.807+0000 7f1c0d752640 1 -- 172.21.15.116:0/2995220119 shutdown_connections 2024-04-25T00:18:47.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:47.807+0000 7f1c0d752640 1 -- 172.21.15.116:0/2995220119 wait complete. 2024-04-25T00:18:47.822 INFO:tasks.mds_thrash.fs.[b]:mds.c reported in up:standby state 2024-04-25T00:18:47.822 INFO:tasks.mds_thrash.fs.[b]:waiting for 5 secs before thrashing 2024-04-25T00:18:52.823 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:18:52.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.975+0000 7ff598112640 1 Processor -- start 2024-04-25T00:18:52.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.975+0000 7ff598112640 1 -- start start 2024-04-25T00:18:52.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.979+0000 7ff598112640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff590072ca0 0x7ff590153a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:52.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.979+0000 7ff598112640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff590153fc0 0x7ff59015e020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:52.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.979+0000 7ff598112640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5900c4950 con 0x7ff590150d10 2024-04-25T00:18:52.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.979+0000 7ff598112640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7ff5901510f0 con 0x7ff590072ca0 2024-04-25T00:18:52.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.979+0000 7ff598112640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7ff590157a20 con 0x7ff590153fc0 2024-04-25T00:18:52.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.979+0000 7ff5956ad640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff590072ca0 0x7ff590153a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:52.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.979+0000 7ff595eae640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7ff590150d10 0x7ff5900c5010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:37372/0 (socket says 172.21.15.116:37372) 2024-04-25T00:18:52.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.979+0000 7ff595eae640 1 -- 172.21.15.116:0/1059857212 learned_addr learned my addr 172.21.15.116:0/1059857212 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:18:52.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.979+0000 7ff5966af640 1 --2- 172.21.15.116:0/1059857212 >> v2:172.21.15.174:3300/0 conn(0x7ff590153fc0 0x7ff59015e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:52.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.979+0000 7ff5956ad640 1 -- 172.21.15.116:0/1059857212 >> v2:172.21.15.174:3300/0 conn(0x7ff590153fc0 msgr2=0x7ff59015e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:52.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.979+0000 7ff5956ad640 1 --2- 172.21.15.116:0/1059857212 >> v2:172.21.15.174:3300/0 conn(0x7ff590153fc0 0x7ff59015e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:52.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.979+0000 7ff5956ad640 1 -- 172.21.15.116:0/1059857212 >> v1:172.21.15.116:6789/0 conn(0x7ff590150d10 legacy=0x7ff5900c5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:52.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.979+0000 7ff5956ad640 1 -- 172.21.15.116:0/1059857212 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5900c2c90 con 0x7ff590072ca0 2024-04-25T00:18:52.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.979+0000 7ff5956ad640 1 --2- 172.21.15.116:0/1059857212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff590072ca0 0x7ff590153a80 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7ff5900a9670 tx=0x7ff58001bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3671a0d7e227c195 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:52.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.979+0000 7ff594eac640 1 -- 172.21.15.116:0/1059857212 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5800ab070 con 0x7ff590072ca0 2024-04-25T00:18:52.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.979+0000 7ff594eac640 1 -- 172.21.15.116:0/1059857212 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5800a5640 con 0x7ff590072ca0 2024-04-25T00:18:52.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.979+0000 7ff594eac640 1 -- 172.21.15.116:0/1059857212 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5800a5a10 con 0x7ff590072ca0 2024-04-25T00:18:52.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.983+0000 7ff598112640 1 -- 172.21.15.116:0/1059857212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff590072ca0 msgr2=0x7ff590153a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:52.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.983+0000 7ff598112640 1 --2- 172.21.15.116:0/1059857212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff590072ca0 0x7ff590153a80 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7ff5900a9670 tx=0x7ff58001bf60 comp rx=0 tx=0).stop 2024-04-25T00:18:52.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.983+0000 7ff598112640 1 -- 172.21.15.116:0/1059857212 shutdown_connections 2024-04-25T00:18:52.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.983+0000 7ff598112640 1 --2- 172.21.15.116:0/1059857212 >> v2:172.21.15.174:3300/0 conn(0x7ff590153fc0 0x7ff59015e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:52.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.983+0000 7ff598112640 1 --2- 172.21.15.116:0/1059857212 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff590072ca0 0x7ff590153a80 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:52.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.983+0000 7ff598112640 1 -- 172.21.15.116:0/1059857212 >> 172.21.15.116:0/1059857212 conn(0x7ff59006c720 msgr2=0x7ff59006cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:52.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.983+0000 7ff598112640 1 -- 172.21.15.116:0/1059857212 shutdown_connections 2024-04-25T00:18:52.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.983+0000 7ff598112640 1 -- 172.21.15.116:0/1059857212 wait complete. 2024-04-25T00:18:52.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.983+0000 7ff598112640 1 Processor -- start 2024-04-25T00:18:52.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.983+0000 7ff598112640 1 -- start start 2024-04-25T00:18:52.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.983+0000 7ff598112640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff590072ca0 0x7ff5900c76b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:52.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.983+0000 7ff598112640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff590150d10 0x7ff590141490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:52.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.983+0000 7ff598112640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff590153fc0 0x7ff590155ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:52.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.983+0000 7ff598112640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff5901615a0 con 0x7ff590150d10 2024-04-25T00:18:52.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.983+0000 7ff598112640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff5900c2850 con 0x7ff590072ca0 2024-04-25T00:18:52.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.983+0000 7ff598112640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff590161720 con 0x7ff590153fc0 2024-04-25T00:18:52.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.983+0000 7ff5956ad640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff590150d10 0x7ff590141490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:52.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.983+0000 7ff5956ad640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff590150d10 0x7ff590141490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37690/0 (socket says 172.21.15.116:37690) 2024-04-25T00:18:52.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.983+0000 7ff5956ad640 1 -- 172.21.15.116:0/2911636777 learned_addr learned my addr 172.21.15.116:0/2911636777 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:18:52.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.983+0000 7ff595eae640 1 --2- 172.21.15.116:0/2911636777 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff590072ca0 0x7ff5900c76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:52.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.983+0000 7ff5966af640 1 --2- 172.21.15.116:0/2911636777 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff590153fc0 0x7ff590155ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:52.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.983+0000 7ff5956ad640 1 -- 172.21.15.116:0/2911636777 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff590153fc0 msgr2=0x7ff590155ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:52.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.983+0000 7ff5956ad640 1 --2- 172.21.15.116:0/2911636777 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff590153fc0 0x7ff590155ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:52.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.983+0000 7ff5956ad640 1 -- 172.21.15.116:0/2911636777 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff590072ca0 msgr2=0x7ff5900c76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:52.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.983+0000 7ff5956ad640 1 --2- 172.21.15.116:0/2911636777 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff590072ca0 0x7ff5900c76b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:52.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.983+0000 7ff5956ad640 1 -- 172.21.15.116:0/2911636777 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff590149ad0 con 0x7ff590150d10 2024-04-25T00:18:52.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.983+0000 7ff595eae640 1 --2- 172.21.15.116:0/2911636777 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff590072ca0 0x7ff5900c76b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:18:52.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.983+0000 7ff5956ad640 1 --2- 172.21.15.116:0/2911636777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff590150d10 0x7ff590141490 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7ff58001c440 tx=0x7ff580019020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:52.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.983+0000 7ff5867fc640 1 -- 172.21.15.116:0/2911636777 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5800ab070 con 0x7ff590150d10 2024-04-25T00:18:52.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.983+0000 7ff5867fc640 1 -- 172.21.15.116:0/2911636777 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5800a5dd0 con 0x7ff590150d10 2024-04-25T00:18:52.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.983+0000 7ff598112640 1 -- 172.21.15.116:0/2911636777 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff590149db0 con 0x7ff590150d10 2024-04-25T00:18:52.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.983+0000 7ff5867fc640 1 -- 172.21.15.116:0/2911636777 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5800b0670 con 0x7ff590150d10 2024-04-25T00:18:52.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.983+0000 7ff598112640 1 -- 172.21.15.116:0/2911636777 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff590156660 con 0x7ff590150d10 2024-04-25T00:18:52.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.987+0000 7ff598112640 1 -- 172.21.15.116:0/2911636777 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff590068a60 con 0x7ff590150d10 2024-04-25T00:18:52.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.987+0000 7ff5867fc640 1 -- 172.21.15.116:0/2911636777 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff5800ae070 con 0x7ff590150d10 2024-04-25T00:18:52.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.987+0000 7ff5867fc640 1 --2- 172.21.15.116:0/2911636777 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff5740678c0 0x7ff574069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:52.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.987+0000 7ff5867fc640 1 -- 172.21.15.116:0/2911636777 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 8988+0+0 (secure 0 0 0) 0x7ff58011f390 con 0x7ff590150d10 2024-04-25T00:18:52.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.987+0000 7ff595eae640 1 --2- 172.21.15.116:0/2911636777 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff5740678c0 0x7ff574069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:52.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.987+0000 7ff595eae640 1 --2- 172.21.15.116:0/2911636777 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff5740678c0 0x7ff574069d80 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7ff58c000f70 tx=0x7ff58c007d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:52.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:52.991+0000 7ff5867fc640 1 -- 172.21.15.116:0/2911636777 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ff580015f90 con 0x7ff590150d10 2024-04-25T00:18:53.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.243+0000 7ff598112640 1 -- 172.21.15.116:0/2911636777 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff5900727e0 con 0x7ff590150d10 2024-04-25T00:18:53.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.243+0000 7ff5867fc640 1 -- 172.21.15.116:0/2911636777 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 147 v147) ==== 95+0+6506 (secure 0 0 0) 0x7ff5800ef1b0 con 0x7ff590150d10 2024-04-25T00:18:53.244 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:18:53.244 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":147,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7542,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/879531976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":879531976},{"type":"v1","addr":"172.21.15.116:6835","nonce":879531976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":147}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:18:23.899096+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7362},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7362":{"gid":7362,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6833/1655959433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1655959433},{"type":"v1","addr":"172.21.15.116:6833","nonce":1655959433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7362,"qdb_cluster":[7362]},"id":1},{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:18:38.800778+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5879},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5879":{"gid":5879,"name":"d","rank":0,"incarnation":143,"state":"up:active","state_seq":6,"addr":"172.21.15.174:6837/2276529656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2276529656},{"type":"v1","addr":"172.21.15.174:6837","nonce":2276529656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5879,"qdb_cluster":[5879]},"id":2}]} 2024-04-25T00:18:53.244 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 147 2024-04-25T00:18:53.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.247+0000 7ff598112640 1 -- 172.21.15.116:0/2911636777 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff5740678c0 msgr2=0x7ff574069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:53.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.247+0000 7ff598112640 1 --2- 172.21.15.116:0/2911636777 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff5740678c0 0x7ff574069d80 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7ff58c000f70 tx=0x7ff58c007d30 comp rx=0 tx=0).stop 2024-04-25T00:18:53.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.247+0000 7ff598112640 1 -- 172.21.15.116:0/2911636777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff590150d10 msgr2=0x7ff590141490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:53.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.247+0000 7ff598112640 1 --2- 172.21.15.116:0/2911636777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff590150d10 0x7ff590141490 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7ff58001c440 tx=0x7ff580019020 comp rx=0 tx=0).stop 2024-04-25T00:18:53.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.247+0000 7ff598112640 1 -- 172.21.15.116:0/2911636777 shutdown_connections 2024-04-25T00:18:53.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.247+0000 7ff598112640 1 --2- 172.21.15.116:0/2911636777 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff5740678c0 0x7ff574069d80 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:53.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.247+0000 7ff598112640 1 --2- 172.21.15.116:0/2911636777 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff590153fc0 0x7ff590155ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:53.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.247+0000 7ff598112640 1 --2- 172.21.15.116:0/2911636777 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff590150d10 0x7ff590141490 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:53.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.247+0000 7ff598112640 1 --2- 172.21.15.116:0/2911636777 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff590072ca0 0x7ff5900c76b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:53.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.247+0000 7ff598112640 1 -- 172.21.15.116:0/2911636777 >> 172.21.15.116:0/2911636777 conn(0x7ff59006c720 msgr2=0x7ff59006f970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:53.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.247+0000 7ff598112640 1 -- 172.21.15.116:0/2911636777 shutdown_connections 2024-04-25T00:18:53.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.247+0000 7ff598112640 1 -- 172.21.15.116:0/2911636777 wait complete. 2024-04-25T00:18:53.263 INFO:tasks.mds_thrash.fs.[b]:kill mds.d (rank=0) 2024-04-25T00:18:53.263 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-25T00:18:53.264 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:18:53.283 INFO:tasks.ceph.mds.d:Stopped 2024-04-25T00:18:53.283 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:18:53.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.411+0000 7fa45c468640 1 Processor -- start 2024-04-25T00:18:53.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.411+0000 7fa45c468640 1 -- start start 2024-04-25T00:18:53.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.411+0000 7fa45c468640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa454156200 0x7fa45415de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:53.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.411+0000 7fa45c468640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa454156650 con 0x7fa4540aa170 2024-04-25T00:18:53.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.411+0000 7fa45c468640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4541526a0 con 0x7fa4540c4300 2024-04-25T00:18:53.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.411+0000 7fa45c468640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fa454153460 con 0x7fa454156200 2024-04-25T00:18:53.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.411+0000 7fa459a03640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fa4540aa170 0x7fa454155af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:37380/0 (socket says 172.21.15.116:37380) 2024-04-25T00:18:53.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.411+0000 7fa459a03640 1 -- 172.21.15.116:0/632990714 learned_addr learned my addr 172.21.15.116:0/632990714 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:18:53.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.411+0000 7fa45aa05640 1 --2- 172.21.15.116:0/632990714 >> v2:172.21.15.116:3300/0 conn(0x7fa454156200 0x7fa45415de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:53.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.411+0000 7fa459202640 1 -- 172.21.15.116:0/632990714 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 632052668 0 0) 0x7fa454156650 con 0x7fa4540aa170 2024-04-25T00:18:53.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.411+0000 7fa459202640 1 -- 172.21.15.116:0/632990714 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa440003530 con 0x7fa4540aa170 2024-04-25T00:18:53.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.411+0000 7fa45aa05640 1 -- 172.21.15.116:0/632990714 >> v1:172.21.15.174:6789/0 conn(0x7fa4540c4300 legacy=0x7fa4540c5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:53.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.411+0000 7fa45aa05640 1 -- 172.21.15.116:0/632990714 >> v1:172.21.15.116:6789/0 conn(0x7fa4540aa170 legacy=0x7fa454155af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:53.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.411+0000 7fa45aa05640 1 -- 172.21.15.116:0/632990714 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa454152980 con 0x7fa454156200 2024-04-25T00:18:53.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.415+0000 7fa45aa05640 1 --2- 172.21.15.116:0/632990714 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa454156200 0x7fa45415de90 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7fa44c009d90 tx=0x7fa44c01c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d8ae57cdd960f8f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:53.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.415+0000 7fa459202640 1 -- 172.21.15.116:0/632990714 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa44c0b2070 con 0x7fa454156200 2024-04-25T00:18:53.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.415+0000 7fa459202640 1 -- 172.21.15.116:0/632990714 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa44c0a5830 con 0x7fa454156200 2024-04-25T00:18:53.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.415+0000 7fa459202640 1 -- 172.21.15.116:0/632990714 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa44c0a5c00 con 0x7fa454156200 2024-04-25T00:18:53.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.415+0000 7fa45c468640 1 -- 172.21.15.116:0/632990714 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa454156200 msgr2=0x7fa45415de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:53.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.415+0000 7fa45c468640 1 --2- 172.21.15.116:0/632990714 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa454156200 0x7fa45415de90 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7fa44c009d90 tx=0x7fa44c01c970 comp rx=0 tx=0).stop 2024-04-25T00:18:53.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.415+0000 7fa45c468640 1 -- 172.21.15.116:0/632990714 shutdown_connections 2024-04-25T00:18:53.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.415+0000 7fa45c468640 1 --2- 172.21.15.116:0/632990714 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa454156200 0x7fa45415de90 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:53.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.415+0000 7fa45c468640 1 -- 172.21.15.116:0/632990714 >> 172.21.15.116:0/632990714 conn(0x7fa45406c720 msgr2=0x7fa45406cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:53.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.415+0000 7fa45c468640 1 -- 172.21.15.116:0/632990714 shutdown_connections 2024-04-25T00:18:53.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.415+0000 7fa45c468640 1 -- 172.21.15.116:0/632990714 wait complete. 2024-04-25T00:18:53.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.415+0000 7fa45c468640 1 Processor -- start 2024-04-25T00:18:53.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.415+0000 7fa45c468640 1 -- start start 2024-04-25T00:18:53.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.415+0000 7fa45c468640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa4540aa170 0x7fa45413c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:53.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.415+0000 7fa45c468640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4540c4300 0x7fa454142ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:53.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.415+0000 7fa45c468640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa454156200 0x7fa4541ef290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:53.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.415+0000 7fa45c468640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa4540aacd0 con 0x7fa4540c4300 2024-04-25T00:18:53.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.415+0000 7fa45c468640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa45406ffd0 con 0x7fa454156200 2024-04-25T00:18:53.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.415+0000 7fa45c468640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa4540aae50 con 0x7fa4540aa170 2024-04-25T00:18:53.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.419+0000 7fa459a03640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4540c4300 0x7fa454142ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:53.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.419+0000 7fa459a03640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4540c4300 0x7fa454142ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37710/0 (socket says 172.21.15.116:37710) 2024-04-25T00:18:53.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.419+0000 7fa459a03640 1 -- 172.21.15.116:0/186349161 learned_addr learned my addr 172.21.15.116:0/186349161 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:18:53.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.419+0000 7fa45a204640 1 --2- 172.21.15.116:0/186349161 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa4540aa170 0x7fa45413c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:53.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.419+0000 7fa45aa05640 1 --2- 172.21.15.116:0/186349161 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa454156200 0x7fa4541ef290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:53.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.419+0000 7fa459a03640 1 -- 172.21.15.116:0/186349161 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa4540aa170 msgr2=0x7fa45413c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:53.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.419+0000 7fa459a03640 1 --2- 172.21.15.116:0/186349161 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa4540aa170 0x7fa45413c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:53.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.419+0000 7fa459a03640 1 -- 172.21.15.116:0/186349161 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa454156200 msgr2=0x7fa4541ef290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:53.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.419+0000 7fa459a03640 1 --2- 172.21.15.116:0/186349161 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa454156200 0x7fa4541ef290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:53.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.419+0000 7fa459a03640 1 -- 172.21.15.116:0/186349161 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa454142cc0 con 0x7fa4540c4300 2024-04-25T00:18:53.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.419+0000 7fa459a03640 1 --2- 172.21.15.116:0/186349161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4540c4300 0x7fa454142ed0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7fa44400c450 tx=0x7fa44400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:53.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.419+0000 7fa44affd640 1 -- 172.21.15.116:0/186349161 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa444017070 con 0x7fa4540c4300 2024-04-25T00:18:53.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.419+0000 7fa44affd640 1 -- 172.21.15.116:0/186349161 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa444005cf0 con 0x7fa4540c4300 2024-04-25T00:18:53.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.419+0000 7fa45c468640 1 -- 172.21.15.116:0/186349161 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4540c2390 con 0x7fa4540c4300 2024-04-25T00:18:53.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.419+0000 7fa44affd640 1 -- 172.21.15.116:0/186349161 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa444007270 con 0x7fa4540c4300 2024-04-25T00:18:53.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.419+0000 7fa45c468640 1 -- 172.21.15.116:0/186349161 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4541ecb50 con 0x7fa4540c4300 2024-04-25T00:18:53.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.419+0000 7fa45c468640 1 -- 172.21.15.116:0/186349161 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4541427d0 con 0x7fa4540c4300 2024-04-25T00:18:53.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.419+0000 7fa44affd640 1 -- 172.21.15.116:0/186349161 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa44401a070 con 0x7fa4540c4300 2024-04-25T00:18:53.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.419+0000 7fa44affd640 1 --2- 172.21.15.116:0/186349161 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa4380678f0 0x7fa438069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:53.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.419+0000 7fa44affd640 1 -- 172.21.15.116:0/186349161 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 8988+0+0 (secure 0 0 0) 0x7fa444085ae0 con 0x7fa4540c4300 2024-04-25T00:18:53.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.419+0000 7fa45a204640 1 --2- 172.21.15.116:0/186349161 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa4380678f0 0x7fa438069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:53.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.423+0000 7fa45a204640 1 --2- 172.21.15.116:0/186349161 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa4380678f0 0x7fa438069db0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fa450000f70 tx=0x7fa450009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:53.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.423+0000 7fa44affd640 1 -- 172.21.15.116:0/186349161 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fa454142cc0 con 0x7fa4540c4300 2024-04-25T00:18:53.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.667+0000 7fa45c468640 1 -- 172.21.15.116:0/186349161 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa454068a60 con 0x7fa4540c4300 2024-04-25T00:18:53.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.667+0000 7fa44affd640 1 -- 172.21.15.116:0/186349161 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 147 v147) ==== 95+0+6506 (secure 0 0 0) 0x7fa444055340 con 0x7fa4540c4300 2024-04-25T00:18:53.667 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:18:53.667 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":147,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7542,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/879531976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":879531976},{"type":"v1","addr":"172.21.15.116:6835","nonce":879531976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":147}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:18:23.899096+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7362},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7362":{"gid":7362,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6833/1655959433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1655959433},{"type":"v1","addr":"172.21.15.116:6833","nonce":1655959433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7362,"qdb_cluster":[7362]},"id":1},{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:18:38.800778+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5879},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5879":{"gid":5879,"name":"d","rank":0,"incarnation":143,"state":"up:active","state_seq":6,"addr":"172.21.15.174:6837/2276529656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2276529656},{"type":"v1","addr":"172.21.15.174:6837","nonce":2276529656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5879,"qdb_cluster":[5879]},"id":2}]} 2024-04-25T00:18:53.668 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 147 2024-04-25T00:18:53.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.671+0000 7fa45c468640 1 -- 172.21.15.116:0/186349161 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa4380678f0 msgr2=0x7fa438069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:53.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.671+0000 7fa45c468640 1 --2- 172.21.15.116:0/186349161 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa4380678f0 0x7fa438069db0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fa450000f70 tx=0x7fa450009040 comp rx=0 tx=0).stop 2024-04-25T00:18:53.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.671+0000 7fa45c468640 1 -- 172.21.15.116:0/186349161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4540c4300 msgr2=0x7fa454142ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:53.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.671+0000 7fa45c468640 1 --2- 172.21.15.116:0/186349161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4540c4300 0x7fa454142ed0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7fa44400c450 tx=0x7fa44400c910 comp rx=0 tx=0).stop 2024-04-25T00:18:53.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.671+0000 7fa45c468640 1 -- 172.21.15.116:0/186349161 shutdown_connections 2024-04-25T00:18:53.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.671+0000 7fa45c468640 1 --2- 172.21.15.116:0/186349161 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa4380678f0 0x7fa438069db0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:53.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.671+0000 7fa45c468640 1 --2- 172.21.15.116:0/186349161 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa454156200 0x7fa4541ef290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:53.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.671+0000 7fa45c468640 1 --2- 172.21.15.116:0/186349161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4540c4300 0x7fa454142ed0 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:53.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.671+0000 7fa45c468640 1 --2- 172.21.15.116:0/186349161 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa4540aa170 0x7fa45413c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:53.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.671+0000 7fa45c468640 1 -- 172.21.15.116:0/186349161 >> 172.21.15.116:0/186349161 conn(0x7fa45406c720 msgr2=0x7fa45406cb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:53.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.671+0000 7fa45c468640 1 -- 172.21.15.116:0/186349161 shutdown_connections 2024-04-25T00:18:53.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:53.671+0000 7fa45c468640 1 -- 172.21.15.116:0/186349161 wait complete. 2024-04-25T00:18:53.689 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:18:55.690 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:18:55.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.827+0000 7f93dcdcc640 1 Processor -- start 2024-04-25T00:18:55.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.827+0000 7f93dcdcc640 1 -- start start 2024-04-25T00:18:55.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.827+0000 7f93dcdcc640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f93d80c3ad0 0x7f93d8152f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:55.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.827+0000 7f93dcdcc640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f93d80c3300 0x7f93d8071d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:55.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.827+0000 7f93dcdcc640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93d80c3f50 con 0x7f93d8148f20 2024-04-25T00:18:55.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.827+0000 7f93dcdcc640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f93d80c4790 con 0x7f93d80c3ad0 2024-04-25T00:18:55.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.827+0000 7f93dcdcc640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f93d80c2b40 con 0x7f93d80c3300 2024-04-25T00:18:55.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.827+0000 7f93d659c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f93d8148f20 0x7f93d806e480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:36196/0 (socket says 172.21.15.116:36196) 2024-04-25T00:18:55.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.827+0000 7f93d659c640 1 -- 172.21.15.116:0/1206218640 learned_addr learned my addr 172.21.15.116:0/1206218640 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:18:55.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.827+0000 7f93cffff640 1 --2- 172.21.15.116:0/1206218640 >> v2:172.21.15.174:3300/0 conn(0x7f93d80c3ad0 0x7f93d8152f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:55.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.827+0000 7f93d5d9b640 1 -- 172.21.15.116:0/1206218640 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 118497563 0 0) 0x7f93c8001120 con 0x7f93d8148f20 2024-04-25T00:18:55.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.827+0000 7f93d5d9b640 1 -- 172.21.15.116:0/1206218640 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93c4003530 con 0x7f93d8148f20 2024-04-25T00:18:55.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.827+0000 7f93d6d9d640 1 --2- 172.21.15.116:0/1206218640 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f93d80c3300 0x7f93d8071d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:55.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.827+0000 7f93d5d9b640 1 -- 172.21.15.116:0/1206218640 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3741257691 0 0) 0x7f93c4003530 con 0x7f93d8148f20 2024-04-25T00:18:55.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.827+0000 7f93d5d9b640 1 -- 172.21.15.116:0/1206218640 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f93d80c3300 msgr2=0x7f93d8071d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:55.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.827+0000 7f93d5d9b640 1 --2- 172.21.15.116:0/1206218640 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f93d80c3300 0x7f93d8071d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:55.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.827+0000 7f93d5d9b640 1 -- 172.21.15.116:0/1206218640 >> v2:172.21.15.174:3300/0 conn(0x7f93d80c3ad0 msgr2=0x7f93d8152f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:55.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.827+0000 7f93d5d9b640 1 --2- 172.21.15.116:0/1206218640 >> v2:172.21.15.174:3300/0 conn(0x7f93d80c3ad0 0x7f93d8152f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:55.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.827+0000 7f93d5d9b640 1 -- 172.21.15.116:0/1206218640 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93d80c41d0 con 0x7f93d8148f20 2024-04-25T00:18:55.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.827+0000 7f93d5d9b640 1 -- 172.21.15.116:0/1206218640 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f93c8001850 con 0x7f93d8148f20 2024-04-25T00:18:55.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.827+0000 7f93d5d9b640 1 -- 172.21.15.116:0/1206218640 >> v1:172.21.15.116:6789/0 conn(0x7f93d8148f20 legacy=0x7f93d806e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:55.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.827+0000 7f93d5d9b640 1 --2- 172.21.15.116:0/1206218640 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93c40039c0 0x7f93c4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:55.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.827+0000 7f93d5d9b640 1 --2- 172.21.15.116:0/1206218640 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93c4004810 0x7f93c4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:55.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.827+0000 7f93d5d9b640 1 --2- 172.21.15.116:0/1206218640 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f93d80c3300 0x7f93c40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:55.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.827+0000 7f93d5d9b640 1 -- 172.21.15.116:0/1206218640 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f93d80c4790 con 0x7f93c40039c0 2024-04-25T00:18:55.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.827+0000 7f93d5d9b640 1 -- 172.21.15.116:0/1206218640 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f93d80c2b40 con 0x7f93c4004810 2024-04-25T00:18:55.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.827+0000 7f93d5d9b640 1 -- 172.21.15.116:0/1206218640 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f93c400b9d0 con 0x7f93d80c3300 2024-04-25T00:18:55.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.827+0000 7f93d659c640 1 --2- 172.21.15.116:0/1206218640 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93c40039c0 0x7f93c4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:55.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.827+0000 7f93cffff640 1 --2- 172.21.15.116:0/1206218640 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93c4004810 0x7f93c4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:55.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.827+0000 7f93d6d9d640 1 --2- 172.21.15.116:0/1206218640 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f93d80c3300 0x7f93c40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:55.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.827+0000 7f93d659c640 1 -- 172.21.15.116:0/1206218640 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f93d80c3300 msgr2=0x7f93c40051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:55.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.827+0000 7f93d659c640 1 --2- 172.21.15.116:0/1206218640 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f93d80c3300 0x7f93c40051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:55.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.827+0000 7f93d659c640 1 -- 172.21.15.116:0/1206218640 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93c4004810 msgr2=0x7f93c4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:55.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.827+0000 7f93d659c640 1 --2- 172.21.15.116:0/1206218640 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93c4004810 0x7f93c4004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:55.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.827+0000 7f93d659c640 1 -- 172.21.15.116:0/1206218640 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93c4003530 con 0x7f93c40039c0 2024-04-25T00:18:55.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.831+0000 7f93d659c640 1 --2- 172.21.15.116:0/1206218640 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93c40039c0 0x7f93c4003de0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f93c800c490 tx=0x7f93c801cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d127df45dccac344 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:55.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.831+0000 7f93d5d9b640 1 -- 172.21.15.116:0/1206218640 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c800c0e0 con 0x7f93c40039c0 2024-04-25T00:18:55.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.831+0000 7f93d5d9b640 1 -- 172.21.15.116:0/1206218640 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93c80a9020 con 0x7f93c40039c0 2024-04-25T00:18:55.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.831+0000 7f93d5d9b640 1 -- 172.21.15.116:0/1206218640 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c80058e0 con 0x7f93c40039c0 2024-04-25T00:18:55.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.831+0000 7f93dcdcc640 1 -- 172.21.15.116:0/1206218640 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93c40039c0 msgr2=0x7f93c4003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:55.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.831+0000 7f93dcdcc640 1 --2- 172.21.15.116:0/1206218640 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93c40039c0 0x7f93c4003de0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f93c800c490 tx=0x7f93c801cca0 comp rx=0 tx=0).stop 2024-04-25T00:18:55.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.831+0000 7f93d6d9d640 1 -- 172.21.15.116:0/1206218640 reap_dead start 2024-04-25T00:18:55.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.831+0000 7f93dcdcc640 1 -- 172.21.15.116:0/1206218640 shutdown_connections 2024-04-25T00:18:55.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.831+0000 7f93dcdcc640 1 -- 172.21.15.116:0/1206218640 >> 172.21.15.116:0/1206218640 conn(0x7f93d80c7fa0 msgr2=0x7f93d80c83b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:55.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.831+0000 7f93dcdcc640 1 -- 172.21.15.116:0/1206218640 shutdown_connections 2024-04-25T00:18:55.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.831+0000 7f93dcdcc640 1 -- 172.21.15.116:0/1206218640 wait complete. 2024-04-25T00:18:55.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.835+0000 7f93dcdcc640 1 Processor -- start 2024-04-25T00:18:55.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.835+0000 7f93dcdcc640 1 -- start start 2024-04-25T00:18:55.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.835+0000 7f93dcdcc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93c40039c0 0x7f93d814f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:55.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.835+0000 7f93dcdcc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93d814e150 0x7f93d80aa230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:55.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.835+0000 7f93dcdcc640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f93d814cbb0 0x7f93d80aa770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:55.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.835+0000 7f93dcdcc640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f93d80c3890 con 0x7f93d814e150 2024-04-25T00:18:55.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.835+0000 7f93dcdcc640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f93d80c3710 con 0x7f93c40039c0 2024-04-25T00:18:55.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.835+0000 7f93dcdcc640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f93d806b210 con 0x7f93d814cbb0 2024-04-25T00:18:55.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.835+0000 7f93cffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93d814e150 0x7f93d80aa230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:55.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.835+0000 7f93cffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93d814e150 0x7f93d80aa230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33262/0 (socket says 172.21.15.116:33262) 2024-04-25T00:18:55.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.835+0000 7f93cffff640 1 -- 172.21.15.116:0/754326087 learned_addr learned my addr 172.21.15.116:0/754326087 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:18:55.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.835+0000 7f93d659c640 1 --2- 172.21.15.116:0/754326087 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93c40039c0 0x7f93d814f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:55.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.835+0000 7f93d6d9d640 1 --2- 172.21.15.116:0/754326087 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f93d814cbb0 0x7f93d80aa770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:55.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.835+0000 7f93cffff640 1 -- 172.21.15.116:0/754326087 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f93d814cbb0 msgr2=0x7f93d80aa770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:55.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.835+0000 7f93cffff640 1 --2- 172.21.15.116:0/754326087 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f93d814cbb0 0x7f93d80aa770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:55.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.835+0000 7f93cffff640 1 -- 172.21.15.116:0/754326087 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93c40039c0 msgr2=0x7f93d814f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:55.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.835+0000 7f93cffff640 1 --2- 172.21.15.116:0/754326087 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93c40039c0 0x7f93d814f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:55.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.835+0000 7f93cffff640 1 -- 172.21.15.116:0/754326087 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93d814d0b0 con 0x7f93d814e150 2024-04-25T00:18:55.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.835+0000 7f93cffff640 1 --2- 172.21.15.116:0/754326087 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93d814e150 0x7f93d80aa230 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f93c0011610 tx=0x7f93c0011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:55.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.835+0000 7f93ceffd640 1 -- 172.21.15.116:0/754326087 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c000f8f0 con 0x7f93d814e150 2024-04-25T00:18:55.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.835+0000 7f93ceffd640 1 -- 172.21.15.116:0/754326087 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93c000fa90 con 0x7f93d814e150 2024-04-25T00:18:55.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.835+0000 7f93ceffd640 1 -- 172.21.15.116:0/754326087 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93c000fe40 con 0x7f93d814e150 2024-04-25T00:18:55.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.835+0000 7f93dcdcc640 1 -- 172.21.15.116:0/754326087 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93d814d390 con 0x7f93d814e150 2024-04-25T00:18:55.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.835+0000 7f93dcdcc640 1 -- 172.21.15.116:0/754326087 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93d814d950 con 0x7f93d814e150 2024-04-25T00:18:55.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.839+0000 7f93dcdcc640 1 -- 172.21.15.116:0/754326087 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93d8068a60 con 0x7f93d814e150 2024-04-25T00:18:55.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.839+0000 7f93ceffd640 1 -- 172.21.15.116:0/754326087 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f93c0015460 con 0x7f93d814e150 2024-04-25T00:18:55.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.839+0000 7f93ceffd640 1 --2- 172.21.15.116:0/754326087 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f93b8067860 0x7f93b8069d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:55.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.839+0000 7f93ceffd640 1 -- 172.21.15.116:0/754326087 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 8988+0+0 (secure 0 0 0) 0x7f93c0087190 con 0x7f93d814e150 2024-04-25T00:18:55.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.839+0000 7f93d659c640 1 --2- 172.21.15.116:0/754326087 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f93b8067860 0x7f93b8069d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:55.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.839+0000 7f93d659c640 1 --2- 172.21.15.116:0/754326087 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f93b8067860 0x7f93b8069d20 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f93c80aa000 tx=0x7f93c8000ff0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:55.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:55.843+0000 7f93ceffd640 1 -- 172.21.15.116:0/754326087 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f93c0019030 con 0x7f93d814e150 2024-04-25T00:18:56.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:56.067+0000 7f93dcdcc640 1 -- 172.21.15.116:0/754326087 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f93d80c7170 con 0x7f93d814e150 2024-04-25T00:18:56.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:56.067+0000 7f93ceffd640 1 -- 172.21.15.116:0/754326087 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 147 v147) ==== 95+0+6506 (secure 0 0 0) 0x7f93c0056960 con 0x7f93d814e150 2024-04-25T00:18:56.066 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:18:56.066 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":147,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7542,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/879531976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":879531976},{"type":"v1","addr":"172.21.15.116:6835","nonce":879531976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":147}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:18:23.899096+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7362},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7362":{"gid":7362,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6833/1655959433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1655959433},{"type":"v1","addr":"172.21.15.116:6833","nonce":1655959433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7362,"qdb_cluster":[7362]},"id":1},{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:18:38.800778+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5879},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5879":{"gid":5879,"name":"d","rank":0,"incarnation":143,"state":"up:active","state_seq":6,"addr":"172.21.15.174:6837/2276529656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2276529656},{"type":"v1","addr":"172.21.15.174:6837","nonce":2276529656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5879,"qdb_cluster":[5879]},"id":2}]} 2024-04-25T00:18:56.067 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 147 2024-04-25T00:18:56.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:56.071+0000 7f93dcdcc640 1 -- 172.21.15.116:0/754326087 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f93b8067860 msgr2=0x7f93b8069d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:56.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:56.071+0000 7f93dcdcc640 1 --2- 172.21.15.116:0/754326087 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f93b8067860 0x7f93b8069d20 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f93c80aa000 tx=0x7f93c8000ff0 comp rx=0 tx=0).stop 2024-04-25T00:18:56.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:56.071+0000 7f93dcdcc640 1 -- 172.21.15.116:0/754326087 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93d814e150 msgr2=0x7f93d80aa230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:56.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:56.071+0000 7f93dcdcc640 1 --2- 172.21.15.116:0/754326087 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93d814e150 0x7f93d80aa230 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f93c0011610 tx=0x7f93c0011ad0 comp rx=0 tx=0).stop 2024-04-25T00:18:56.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:56.071+0000 7f93dcdcc640 1 -- 172.21.15.116:0/754326087 shutdown_connections 2024-04-25T00:18:56.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:56.071+0000 7f93dcdcc640 1 --2- 172.21.15.116:0/754326087 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f93b8067860 0x7f93b8069d20 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:56.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:56.071+0000 7f93dcdcc640 1 --2- 172.21.15.116:0/754326087 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f93d814cbb0 0x7f93d80aa770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:56.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:56.071+0000 7f93dcdcc640 1 --2- 172.21.15.116:0/754326087 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93d814e150 0x7f93d80aa230 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:56.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:56.071+0000 7f93dcdcc640 1 --2- 172.21.15.116:0/754326087 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f93c40039c0 0x7f93d814f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:56.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:56.071+0000 7f93dcdcc640 1 -- 172.21.15.116:0/754326087 >> 172.21.15.116:0/754326087 conn(0x7f93d80c7fa0 msgr2=0x7f93d80c7760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:56.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:56.071+0000 7f93dcdcc640 1 -- 172.21.15.116:0/754326087 shutdown_connections 2024-04-25T00:18:56.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:56.071+0000 7f93dcdcc640 1 -- 172.21.15.116:0/754326087 wait complete. 2024-04-25T00:18:56.086 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:18:58.087 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:18:58.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.227+0000 7fbfeb427640 1 Processor -- start 2024-04-25T00:18:58.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.227+0000 7fbfeb427640 1 -- start start 2024-04-25T00:18:58.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.227+0000 7fbfeb427640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fbfe4152d80 0x7fbfe406bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:58.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.227+0000 7fbfeb427640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbfe406d5f0 0x7fbfe4156c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:58.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.227+0000 7fbfeb427640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbfe406c290 0x7fbfe41571c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:58.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.227+0000 7fbfeb427640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fbfe40c9d60 con 0x7fbfe406c290 2024-04-25T00:18:58.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.227+0000 7fbfeb427640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fbfe40c9150 con 0x7fbfe4152d80 2024-04-25T00:18:58.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.227+0000 7fbfeb427640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fbfe40c9430 con 0x7fbfe406d5f0 2024-04-25T00:18:58.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.227+0000 7fbfe99c4640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbfe406c290 0x7fbfe41571c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:58.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.227+0000 7fbfe99c4640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbfe406c290 0x7fbfe41571c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33268/0 (socket says 172.21.15.116:33268) 2024-04-25T00:18:58.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.227+0000 7fbfe99c4640 1 -- 172.21.15.116:0/1044217802 learned_addr learned my addr 172.21.15.116:0/1044217802 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:18:58.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.227+0000 7fbfe91c3640 1 --2- 172.21.15.116:0/1044217802 >> v2:172.21.15.174:3300/0 conn(0x7fbfe4152d80 0x7fbfe406bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:58.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.227+0000 7fbfe89c2640 1 --2- 172.21.15.116:0/1044217802 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbfe406d5f0 0x7fbfe4156c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:58.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.227+0000 7fbfe99c4640 1 -- 172.21.15.116:0/1044217802 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbfe406d5f0 msgr2=0x7fbfe4156c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:58.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.227+0000 7fbfe99c4640 1 --2- 172.21.15.116:0/1044217802 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbfe406d5f0 0x7fbfe4156c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:58.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.227+0000 7fbfe99c4640 1 -- 172.21.15.116:0/1044217802 >> v2:172.21.15.174:3300/0 conn(0x7fbfe4152d80 msgr2=0x7fbfe406bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:18:58.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.227+0000 7fbfe99c4640 1 --2- 172.21.15.116:0/1044217802 >> v2:172.21.15.174:3300/0 conn(0x7fbfe4152d80 0x7fbfe406bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:58.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.227+0000 7fbfe99c4640 1 -- 172.21.15.116:0/1044217802 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfe40c9710 con 0x7fbfe406c290 2024-04-25T00:18:58.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.231+0000 7fbfe99c4640 1 --2- 172.21.15.116:0/1044217802 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbfe406c290 0x7fbfe41571c0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7fbfe40caac0 tx=0x7fbfdc01c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23a9b922a8ccd35a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:58.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.231+0000 7fbfdbfff640 1 -- 172.21.15.116:0/1044217802 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfdc0a56d0 con 0x7fbfe406c290 2024-04-25T00:18:58.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.231+0000 7fbfdbfff640 1 -- 172.21.15.116:0/1044217802 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfdc0a5870 con 0x7fbfe406c290 2024-04-25T00:18:58.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.231+0000 7fbfdbfff640 1 -- 172.21.15.116:0/1044217802 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfdc0a5c40 con 0x7fbfe406c290 2024-04-25T00:18:58.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.231+0000 7fbfeb427640 1 -- 172.21.15.116:0/1044217802 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbfe406c290 msgr2=0x7fbfe41571c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:58.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.231+0000 7fbfeb427640 1 --2- 172.21.15.116:0/1044217802 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbfe406c290 0x7fbfe41571c0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7fbfe40caac0 tx=0x7fbfdc01c930 comp rx=0 tx=0).stop 2024-04-25T00:18:58.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.231+0000 7fbfeb427640 1 -- 172.21.15.116:0/1044217802 shutdown_connections 2024-04-25T00:18:58.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.231+0000 7fbfeb427640 1 --2- 172.21.15.116:0/1044217802 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbfe406c290 0x7fbfe41571c0 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:58.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.231+0000 7fbfeb427640 1 --2- 172.21.15.116:0/1044217802 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbfe406d5f0 0x7fbfe4156c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:58.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.231+0000 7fbfeb427640 1 --2- 172.21.15.116:0/1044217802 >> v2:172.21.15.174:3300/0 conn(0x7fbfe4152d80 0x7fbfe406bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:58.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.231+0000 7fbfeb427640 1 -- 172.21.15.116:0/1044217802 >> 172.21.15.116:0/1044217802 conn(0x7fbfe40acdd0 msgr2=0x7fbfe40ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:58.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.231+0000 7fbfeb427640 1 -- 172.21.15.116:0/1044217802 shutdown_connections 2024-04-25T00:18:58.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.231+0000 7fbfeb427640 1 -- 172.21.15.116:0/1044217802 wait complete. 2024-04-25T00:18:58.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.235+0000 7fbfeb427640 1 Processor -- start 2024-04-25T00:18:58.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.235+0000 7fbfeb427640 1 -- start start 2024-04-25T00:18:58.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.235+0000 7fbfeb427640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbfe406c290 0x7fbfe41f2250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:58.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.235+0000 7fbfeb427640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbfe406d5f0 0x7fbfe414bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:58.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.235+0000 7fbfeb427640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbfe4152d80 0x7fbfe414c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:58.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.235+0000 7fbfeb427640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbfe4159710 con 0x7fbfe406c290 2024-04-25T00:18:58.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.235+0000 7fbfeb427640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbfe4159590 con 0x7fbfe4152d80 2024-04-25T00:18:58.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.235+0000 7fbfeb427640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fbfe4159890 con 0x7fbfe406d5f0 2024-04-25T00:18:58.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.235+0000 7fbfe91c3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbfe406c290 0x7fbfe41f2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:58.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.235+0000 7fbfe91c3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbfe406c290 0x7fbfe41f2250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33274/0 (socket says 172.21.15.116:33274) 2024-04-25T00:18:58.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.235+0000 7fbfe91c3640 1 -- 172.21.15.116:0/2551999734 learned_addr learned my addr 172.21.15.116:0/2551999734 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:18:58.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.235+0000 7fbfe89c2640 1 --2- 172.21.15.116:0/2551999734 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbfe406d5f0 0x7fbfe414bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:58.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.235+0000 7fbfe99c4640 1 --2- 172.21.15.116:0/2551999734 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbfe4152d80 0x7fbfe414c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:58.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.235+0000 7fbfe91c3640 1 -- 172.21.15.116:0/2551999734 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbfe406d5f0 msgr2=0x7fbfe414bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:58.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.235+0000 7fbfe91c3640 1 --2- 172.21.15.116:0/2551999734 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbfe406d5f0 0x7fbfe414bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:58.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.235+0000 7fbfe91c3640 1 -- 172.21.15.116:0/2551999734 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbfe4152d80 msgr2=0x7fbfe414c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:58.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.235+0000 7fbfe91c3640 1 --2- 172.21.15.116:0/2551999734 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbfe4152d80 0x7fbfe414c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:58.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.235+0000 7fbfe91c3640 1 -- 172.21.15.116:0/2551999734 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfe40be8e0 con 0x7fbfe406c290 2024-04-25T00:18:58.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.235+0000 7fbfe89c2640 1 --2- 172.21.15.116:0/2551999734 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbfe406d5f0 0x7fbfe414bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:18:58.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.235+0000 7fbfe91c3640 1 --2- 172.21.15.116:0/2551999734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbfe406c290 0x7fbfe41f2250 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7fbfe000d5f0 tx=0x7fbfe000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:58.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.235+0000 7fbfd9ffb640 1 -- 172.21.15.116:0/2551999734 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe00125f0 con 0x7fbfe406c290 2024-04-25T00:18:58.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.235+0000 7fbfd9ffb640 1 -- 172.21.15.116:0/2551999734 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfe0019070 con 0x7fbfe406c290 2024-04-25T00:18:58.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.235+0000 7fbfeb427640 1 -- 172.21.15.116:0/2551999734 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfe414c6e0 con 0x7fbfe406c290 2024-04-25T00:18:58.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.235+0000 7fbfd9ffb640 1 -- 172.21.15.116:0/2551999734 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe00129b0 con 0x7fbfe406c290 2024-04-25T00:18:58.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.235+0000 7fbfeb427640 1 -- 172.21.15.116:0/2551999734 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfe41f0100 con 0x7fbfe406c290 2024-04-25T00:18:58.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.235+0000 7fbfeb427640 1 -- 172.21.15.116:0/2551999734 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfe4068350 con 0x7fbfe406c290 2024-04-25T00:18:58.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.239+0000 7fbfd9ffb640 1 -- 172.21.15.116:0/2551999734 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbfe0012b50 con 0x7fbfe406c290 2024-04-25T00:18:58.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.239+0000 7fbfd9ffb640 1 --2- 172.21.15.116:0/2551999734 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbfc80678c0 0x7fbfc8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:18:58.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.239+0000 7fbfd9ffb640 1 -- 172.21.15.116:0/2551999734 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 8988+0+0 (secure 0 0 0) 0x7fbfe0086d60 con 0x7fbfe406c290 2024-04-25T00:18:58.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.239+0000 7fbfe89c2640 1 --2- 172.21.15.116:0/2551999734 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbfc80678c0 0x7fbfc8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:18:58.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.239+0000 7fbfe89c2640 1 --2- 172.21.15.116:0/2551999734 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbfc80678c0 0x7fbfc8069d80 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fbfe40c1ba0 tx=0x7fbfd4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:18:58.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.243+0000 7fbfd9ffb640 1 -- 172.21.15.116:0/2551999734 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fbfe0056c10 con 0x7fbfe406c290 2024-04-25T00:18:58.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.499+0000 7fbfeb427640 1 -- 172.21.15.116:0/2551999734 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbfe40c1bd0 con 0x7fbfe406c290 2024-04-25T00:18:58.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.499+0000 7fbfd9ffb640 1 -- 172.21.15.116:0/2551999734 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 147 v147) ==== 95+0+6506 (secure 0 0 0) 0x7fbfe0056510 con 0x7fbfe406c290 2024-04-25T00:18:58.499 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:18:58.499 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":147,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7542,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/879531976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":879531976},{"type":"v1","addr":"172.21.15.116:6835","nonce":879531976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":147}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:18:23.899096+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7362},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7362":{"gid":7362,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6833/1655959433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1655959433},{"type":"v1","addr":"172.21.15.116:6833","nonce":1655959433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7362,"qdb_cluster":[7362]},"id":1},{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:18:38.800778+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5879},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5879":{"gid":5879,"name":"d","rank":0,"incarnation":143,"state":"up:active","state_seq":6,"addr":"172.21.15.174:6837/2276529656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2276529656},{"type":"v1","addr":"172.21.15.174:6837","nonce":2276529656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5879,"qdb_cluster":[5879]},"id":2}]} 2024-04-25T00:18:58.499 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 147 2024-04-25T00:18:58.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.503+0000 7fbfeb427640 1 -- 172.21.15.116:0/2551999734 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbfc80678c0 msgr2=0x7fbfc8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:58.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.503+0000 7fbfeb427640 1 --2- 172.21.15.116:0/2551999734 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbfc80678c0 0x7fbfc8069d80 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fbfe40c1ba0 tx=0x7fbfd4008040 comp rx=0 tx=0).stop 2024-04-25T00:18:58.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.503+0000 7fbfeb427640 1 -- 172.21.15.116:0/2551999734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbfe406c290 msgr2=0x7fbfe41f2250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:18:58.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.503+0000 7fbfeb427640 1 --2- 172.21.15.116:0/2551999734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbfe406c290 0x7fbfe41f2250 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7fbfe000d5f0 tx=0x7fbfe000dab0 comp rx=0 tx=0).stop 2024-04-25T00:18:58.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.503+0000 7fbfeb427640 1 -- 172.21.15.116:0/2551999734 shutdown_connections 2024-04-25T00:18:58.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.503+0000 7fbfeb427640 1 --2- 172.21.15.116:0/2551999734 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbfc80678c0 0x7fbfc8069d80 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:58.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.503+0000 7fbfeb427640 1 --2- 172.21.15.116:0/2551999734 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbfe4152d80 0x7fbfe414c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:58.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.503+0000 7fbfeb427640 1 --2- 172.21.15.116:0/2551999734 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbfe406d5f0 0x7fbfe414bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:58.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.503+0000 7fbfeb427640 1 --2- 172.21.15.116:0/2551999734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbfe406c290 0x7fbfe41f2250 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:18:58.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.503+0000 7fbfeb427640 1 -- 172.21.15.116:0/2551999734 >> 172.21.15.116:0/2551999734 conn(0x7fbfe40acdd0 msgr2=0x7fbfe415f710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:18:58.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.503+0000 7fbfeb427640 1 -- 172.21.15.116:0/2551999734 shutdown_connections 2024-04-25T00:18:58.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:18:58.503+0000 7fbfeb427640 1 -- 172.21.15.116:0/2551999734 wait complete. 2024-04-25T00:18:58.522 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:19:00.523 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:19:00.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.667+0000 7f33924f9640 1 Processor -- start 2024-04-25T00:19:00.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.667+0000 7f33924f9640 1 -- start start 2024-04-25T00:19:00.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.667+0000 7f33924f9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f338c0c4300 0x7f338c0aa170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:00.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.667+0000 7f33924f9640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f338c0c5550 0x7f338c156200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:00.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.667+0000 7f33924f9640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f338c1526a0 con 0x7f338c0c5100 2024-04-25T00:19:00.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.667+0000 7f33924f9640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f338c153460 con 0x7f338c0c4300 2024-04-25T00:19:00.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.667+0000 7f33924f9640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f338c06ffd0 con 0x7f338c0c5550 2024-04-25T00:19:00.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.667+0000 7f338bfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f338c0c4300 0x7f338c0aa170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:00.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.667+0000 7f338bfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f338c0c4300 0x7f338c0aa170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33278/0 (socket says 172.21.15.116:33278) 2024-04-25T00:19:00.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.667+0000 7f338b7fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f338c0c5100 0x7f338c155af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:36208/0 (socket says 172.21.15.116:36208) 2024-04-25T00:19:00.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.667+0000 7f338bfff640 1 -- 172.21.15.116:0/1933996874 learned_addr learned my addr 172.21.15.116:0/1933996874 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:19:00.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.667+0000 7f3390a96640 1 --2- 172.21.15.116:0/1933996874 >> v2:172.21.15.174:3300/0 conn(0x7f338c0c5550 0x7f338c156200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:00.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.667+0000 7f338bfff640 1 -- 172.21.15.116:0/1933996874 >> v2:172.21.15.174:3300/0 conn(0x7f338c0c5550 msgr2=0x7f338c156200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:19:00.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.667+0000 7f338bfff640 1 --2- 172.21.15.116:0/1933996874 >> v2:172.21.15.174:3300/0 conn(0x7f338c0c5550 0x7f338c156200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:00.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.667+0000 7f338bfff640 1 -- 172.21.15.116:0/1933996874 >> v1:172.21.15.116:6789/0 conn(0x7f338c0c5100 legacy=0x7f338c155af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:00.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.667+0000 7f338bfff640 1 -- 172.21.15.116:0/1933996874 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f338c152980 con 0x7f338c0c4300 2024-04-25T00:19:00.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.667+0000 7f338affd640 1 -- 172.21.15.116:0/1933996874 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 792984501 0 0) 0x7f338c1526a0 con 0x7f338c0c5100 2024-04-25T00:19:00.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.671+0000 7f338bfff640 1 --2- 172.21.15.116:0/1933996874 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f338c0c4300 0x7f338c0aa170 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f338c070150 tx=0x7f337c01ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62a1506e8ecb78db server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:00.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.671+0000 7f338affd640 1 -- 172.21.15.116:0/1933996874 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f337c0b4070 con 0x7f338c0c4300 2024-04-25T00:19:00.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.671+0000 7f338affd640 1 -- 172.21.15.116:0/1933996874 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f337c0a79a0 con 0x7f338c0c4300 2024-04-25T00:19:00.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.671+0000 7f338affd640 1 -- 172.21.15.116:0/1933996874 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f337c0a7d70 con 0x7f338c0c4300 2024-04-25T00:19:00.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.671+0000 7f33924f9640 1 -- 172.21.15.116:0/1933996874 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f338c0c4300 msgr2=0x7f338c0aa170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:00.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.671+0000 7f33924f9640 1 --2- 172.21.15.116:0/1933996874 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f338c0c4300 0x7f338c0aa170 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f338c070150 tx=0x7f337c01ebd0 comp rx=0 tx=0).stop 2024-04-25T00:19:00.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.671+0000 7f33924f9640 1 -- 172.21.15.116:0/1933996874 shutdown_connections 2024-04-25T00:19:00.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.671+0000 7f33924f9640 1 --2- 172.21.15.116:0/1933996874 >> v2:172.21.15.174:3300/0 conn(0x7f338c0c5550 0x7f338c156200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:00.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.671+0000 7f33924f9640 1 --2- 172.21.15.116:0/1933996874 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f338c0c4300 0x7f338c0aa170 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:00.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.671+0000 7f33924f9640 1 -- 172.21.15.116:0/1933996874 >> 172.21.15.116:0/1933996874 conn(0x7f338c06c720 msgr2=0x7f338c06cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:19:00.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.671+0000 7f33924f9640 1 -- 172.21.15.116:0/1933996874 shutdown_connections 2024-04-25T00:19:00.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.671+0000 7f33924f9640 1 -- 172.21.15.116:0/1933996874 wait complete. 2024-04-25T00:19:00.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.671+0000 7f33924f9640 1 Processor -- start 2024-04-25T00:19:00.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.671+0000 7f33924f9640 1 -- start start 2024-04-25T00:19:00.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.675+0000 7f33924f9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f338c0c4300 0x7f338c1eeed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:00.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.675+0000 7f33924f9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f338c0c5100 0x7f338c15b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:00.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.675+0000 7f33924f9640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f338c0c5550 0x7f338c155490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:00.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.675+0000 7f33924f9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f338c06fd00 con 0x7f338c0c5100 2024-04-25T00:19:00.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.675+0000 7f33924f9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f338c06fb80 con 0x7f338c0c4300 2024-04-25T00:19:00.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.675+0000 7f33924f9640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f338c06ba20 con 0x7f338c0c5550 2024-04-25T00:19:00.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.675+0000 7f338b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f338c0c5100 0x7f338c15b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:00.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.675+0000 7f338b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f338c0c5100 0x7f338c15b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33282/0 (socket says 172.21.15.116:33282) 2024-04-25T00:19:00.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.675+0000 7f338b7fe640 1 -- 172.21.15.116:0/1174467094 learned_addr learned my addr 172.21.15.116:0/1174467094 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:19:00.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.675+0000 7f338bfff640 1 --2- 172.21.15.116:0/1174467094 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f338c0c4300 0x7f338c1eeed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:00.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.675+0000 7f3390a96640 1 --2- 172.21.15.116:0/1174467094 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f338c0c5550 0x7f338c155490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:00.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.675+0000 7f338b7fe640 1 -- 172.21.15.116:0/1174467094 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f338c0c5550 msgr2=0x7f338c155490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:00.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.675+0000 7f338b7fe640 1 --2- 172.21.15.116:0/1174467094 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f338c0c5550 0x7f338c155490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:00.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.675+0000 7f338b7fe640 1 -- 172.21.15.116:0/1174467094 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f338c0c4300 msgr2=0x7f338c1eeed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:00.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.675+0000 7f338b7fe640 1 --2- 172.21.15.116:0/1174467094 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f338c0c4300 0x7f338c1eeed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:00.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.675+0000 7f338b7fe640 1 -- 172.21.15.116:0/1174467094 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f338c14f840 con 0x7f338c0c5100 2024-04-25T00:19:00.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.675+0000 7f338b7fe640 1 --2- 172.21.15.116:0/1174467094 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f338c0c5100 0x7f338c15b260 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f33800011f0 tx=0x7f338000c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:00.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.675+0000 7f3388ff9640 1 -- 172.21.15.116:0/1174467094 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3380018070 con 0x7f338c0c5100 2024-04-25T00:19:00.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.675+0000 7f3388ff9640 1 -- 172.21.15.116:0/1174467094 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33800074e0 con 0x7f338c0c5100 2024-04-25T00:19:00.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.675+0000 7f3388ff9640 1 -- 172.21.15.116:0/1174467094 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3380004a10 con 0x7f338c0c5100 2024-04-25T00:19:00.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.675+0000 7f33924f9640 1 -- 172.21.15.116:0/1174467094 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f338c153980 con 0x7f338c0c5100 2024-04-25T00:19:00.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.675+0000 7f33924f9640 1 -- 172.21.15.116:0/1174467094 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f338c156450 con 0x7f338c0c5100 2024-04-25T00:19:00.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.675+0000 7f33924f9640 1 -- 172.21.15.116:0/1174467094 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f338c068a60 con 0x7f338c0c5100 2024-04-25T00:19:00.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.675+0000 7f3388ff9640 1 -- 172.21.15.116:0/1174467094 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3380004bb0 con 0x7f338c0c5100 2024-04-25T00:19:00.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.675+0000 7f3388ff9640 1 --2- 172.21.15.116:0/1174467094 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f33780678c0 0x7f3378069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:00.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.679+0000 7f3388ff9640 1 -- 172.21.15.116:0/1174467094 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 8988+0+0 (secure 0 0 0) 0x7f3380085d10 con 0x7f338c0c5100 2024-04-25T00:19:00.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.679+0000 7f338bfff640 1 --2- 172.21.15.116:0/1174467094 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f33780678c0 0x7f3378069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:00.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.679+0000 7f338bfff640 1 --2- 172.21.15.116:0/1174467094 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f33780678c0 0x7f3378069d80 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f337c0b1000 tx=0x7f337c0a9070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:00.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.683+0000 7f3388ff9640 1 -- 172.21.15.116:0/1174467094 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f338c14f840 con 0x7f338c0c5100 2024-04-25T00:19:00.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.919+0000 7f33924f9640 1 -- 172.21.15.116:0/1174467094 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f338c0759d0 con 0x7f338c0c5100 2024-04-25T00:19:00.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.919+0000 7f3388ff9640 1 -- 172.21.15.116:0/1174467094 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 147 v147) ==== 95+0+6506 (secure 0 0 0) 0x7f3380055bc0 con 0x7f338c0c5100 2024-04-25T00:19:00.921 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:19:00.921 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":147,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7542,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/879531976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":879531976},{"type":"v1","addr":"172.21.15.116:6835","nonce":879531976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":147}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:18:23.899096+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7362},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7362":{"gid":7362,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6833/1655959433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1655959433},{"type":"v1","addr":"172.21.15.116:6833","nonce":1655959433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7362,"qdb_cluster":[7362]},"id":1},{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:18:38.800778+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5879},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5879":{"gid":5879,"name":"d","rank":0,"incarnation":143,"state":"up:active","state_seq":6,"addr":"172.21.15.174:6837/2276529656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2276529656},{"type":"v1","addr":"172.21.15.174:6837","nonce":2276529656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5879,"qdb_cluster":[5879]},"id":2}]} 2024-04-25T00:19:00.922 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 147 2024-04-25T00:19:00.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.923+0000 7f33924f9640 1 -- 172.21.15.116:0/1174467094 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f33780678c0 msgr2=0x7f3378069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:00.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.923+0000 7f33924f9640 1 --2- 172.21.15.116:0/1174467094 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f33780678c0 0x7f3378069d80 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f337c0b1000 tx=0x7f337c0a9070 comp rx=0 tx=0).stop 2024-04-25T00:19:00.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.923+0000 7f33924f9640 1 -- 172.21.15.116:0/1174467094 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f338c0c5100 msgr2=0x7f338c15b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:00.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.923+0000 7f33924f9640 1 --2- 172.21.15.116:0/1174467094 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f338c0c5100 0x7f338c15b260 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f33800011f0 tx=0x7f338000c9b0 comp rx=0 tx=0).stop 2024-04-25T00:19:00.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.923+0000 7f33924f9640 1 -- 172.21.15.116:0/1174467094 shutdown_connections 2024-04-25T00:19:00.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.923+0000 7f33924f9640 1 --2- 172.21.15.116:0/1174467094 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f33780678c0 0x7f3378069d80 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:00.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.923+0000 7f33924f9640 1 --2- 172.21.15.116:0/1174467094 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f338c0c5550 0x7f338c155490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:00.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.923+0000 7f33924f9640 1 --2- 172.21.15.116:0/1174467094 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f338c0c5100 0x7f338c15b260 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:00.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.923+0000 7f33924f9640 1 --2- 172.21.15.116:0/1174467094 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f338c0c4300 0x7f338c1eeed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:00.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.923+0000 7f33924f9640 1 -- 172.21.15.116:0/1174467094 >> 172.21.15.116:0/1174467094 conn(0x7f338c06c720 msgr2=0x7f338c0ad160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:19:00.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.923+0000 7f33924f9640 1 -- 172.21.15.116:0/1174467094 shutdown_connections 2024-04-25T00:19:00.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:00.923+0000 7f33924f9640 1 -- 172.21.15.116:0/1174467094 wait complete. 2024-04-25T00:19:00.939 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:19:02.940 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:19:03.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.091+0000 7f342d8e6640 1 Processor -- start 2024-04-25T00:19:03.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.091+0000 7f342d8e6640 1 -- start start 2024-04-25T00:19:03.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.091+0000 7f342d8e6640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f342814d6e0 0x7f3428153ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:03.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.091+0000 7f342d8e6640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3428156c30 con 0x7f342806b210 2024-04-25T00:19:03.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.091+0000 7f342d8e6640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34280702a0 con 0x7f34281577f0 2024-04-25T00:19:03.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.091+0000 7f342d8e6640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f3428154530 con 0x7f342814d6e0 2024-04-25T00:19:03.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.091+0000 7f341ffff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f342806b210 0x7f34281570e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:36212/0 (socket says 172.21.15.116:36212) 2024-04-25T00:19:03.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.091+0000 7f341ffff640 1 -- 172.21.15.116:0/2288698477 learned_addr learned my addr 172.21.15.116:0/2288698477 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:19:03.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.091+0000 7f3426ffd640 1 --2- 172.21.15.116:0/2288698477 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f342814d6e0 0x7f3428153ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:03.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.091+0000 7f34267fc640 1 -- 172.21.15.116:0/2288698477 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1667585732 0 0) 0x7f3428156c30 con 0x7f342806b210 2024-04-25T00:19:03.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.091+0000 7f34267fc640 1 -- 172.21.15.116:0/2288698477 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3414003530 con 0x7f342806b210 2024-04-25T00:19:03.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.091+0000 7f34267fc640 1 -- 172.21.15.116:0/2288698477 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 402839469 0 0) 0x7f3414003530 con 0x7f342806b210 2024-04-25T00:19:03.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.091+0000 7f34267fc640 1 -- 172.21.15.116:0/2288698477 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f342814d6e0 msgr2=0x7f3428153ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:19:03.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.091+0000 7f34267fc640 1 --2- 172.21.15.116:0/2288698477 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f342814d6e0 0x7f3428153ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:03.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.091+0000 7f34267fc640 1 -- 172.21.15.116:0/2288698477 >> v1:172.21.15.174:6789/0 conn(0x7f34281577f0 legacy=0x7f342815df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:03.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.091+0000 7f34267fc640 1 -- 172.21.15.116:0/2288698477 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34280a95b0 con 0x7f342806b210 2024-04-25T00:19:03.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.091+0000 7f34267fc640 1 -- 172.21.15.116:0/2288698477 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f3410001680 con 0x7f342806b210 2024-04-25T00:19:03.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.091+0000 7f34267fc640 1 -- 172.21.15.116:0/2288698477 >> v1:172.21.15.116:6789/0 conn(0x7f342806b210 legacy=0x7f34281570e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:03.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.091+0000 7f34267fc640 1 --2- 172.21.15.116:0/2288698477 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34140039f0 0x7f3414003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:03.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.091+0000 7f34267fc640 1 --2- 172.21.15.116:0/2288698477 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3414004810 0x7f3414004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:03.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.091+0000 7f34267fc640 1 --2- 172.21.15.116:0/2288698477 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f342814d6e0 0x7f34140051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:03.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.091+0000 7f34267fc640 1 -- 172.21.15.116:0/2288698477 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3428154530 con 0x7f34140039f0 2024-04-25T00:19:03.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.091+0000 7f34267fc640 1 -- 172.21.15.116:0/2288698477 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f341400ba80 con 0x7f3414004810 2024-04-25T00:19:03.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.091+0000 7f34267fc640 1 -- 172.21.15.116:0/2288698477 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f341400bc80 con 0x7f342814d6e0 2024-04-25T00:19:03.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.091+0000 7f3426ffd640 1 --2- 172.21.15.116:0/2288698477 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34140039f0 0x7f3414003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:03.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.091+0000 7f341ffff640 1 --2- 172.21.15.116:0/2288698477 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3414004810 0x7f3414004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:03.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.091+0000 7f34277fe640 1 --2- 172.21.15.116:0/2288698477 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f342814d6e0 0x7f34140051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:03.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.091+0000 7f3426ffd640 1 -- 172.21.15.116:0/2288698477 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f342814d6e0 msgr2=0x7f34140051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:19:03.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.091+0000 7f3426ffd640 1 --2- 172.21.15.116:0/2288698477 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f342814d6e0 0x7f34140051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:03.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.091+0000 7f3426ffd640 1 -- 172.21.15.116:0/2288698477 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3414004810 msgr2=0x7f3414004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:19:03.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.091+0000 7f3426ffd640 1 --2- 172.21.15.116:0/2288698477 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3414004810 0x7f3414004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:03.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.091+0000 7f3426ffd640 1 -- 172.21.15.116:0/2288698477 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3414003530 con 0x7f34140039f0 2024-04-25T00:19:03.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.091+0000 7f34277fe640 1 --2- 172.21.15.116:0/2288698477 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f342814d6e0 0x7f34140051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:19:03.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.091+0000 7f341ffff640 1 --2- 172.21.15.116:0/2288698477 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3414004810 0x7f3414004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:19:03.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.095+0000 7f3426ffd640 1 --2- 172.21.15.116:0/2288698477 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34140039f0 0x7f3414003e10 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f3418009ab0 tx=0x7f341801ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2023fd48b503f4b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:03.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.095+0000 7f34267fc640 1 -- 172.21.15.116:0/2288698477 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34180096b0 con 0x7f34140039f0 2024-04-25T00:19:03.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.095+0000 7f34267fc640 1 -- 172.21.15.116:0/2288698477 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34180a6410 con 0x7f34140039f0 2024-04-25T00:19:03.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.095+0000 7f34267fc640 1 -- 172.21.15.116:0/2288698477 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34180a6800 con 0x7f34140039f0 2024-04-25T00:19:03.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.095+0000 7f342d8e6640 1 -- 172.21.15.116:0/2288698477 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34140039f0 msgr2=0x7f3414003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:03.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.095+0000 7f342d8e6640 1 --2- 172.21.15.116:0/2288698477 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34140039f0 0x7f3414003e10 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f3418009ab0 tx=0x7f341801ca30 comp rx=0 tx=0).stop 2024-04-25T00:19:03.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.095+0000 7f34277fe640 1 -- 172.21.15.116:0/2288698477 reap_dead start 2024-04-25T00:19:03.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.095+0000 7f342d8e6640 1 -- 172.21.15.116:0/2288698477 shutdown_connections 2024-04-25T00:19:03.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.095+0000 7f342d8e6640 1 -- 172.21.15.116:0/2288698477 >> 172.21.15.116:0/2288698477 conn(0x7f34280c7fa0 msgr2=0x7f34280c83b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:19:03.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.095+0000 7f342d8e6640 1 -- 172.21.15.116:0/2288698477 shutdown_connections 2024-04-25T00:19:03.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.095+0000 7f342d8e6640 1 -- 172.21.15.116:0/2288698477 wait complete. 2024-04-25T00:19:03.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.095+0000 7f342d8e6640 1 Processor -- start 2024-04-25T00:19:03.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.095+0000 7f342d8e6640 1 -- start start 2024-04-25T00:19:03.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.099+0000 7f342d8e6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f34140039f0 0x7f34280ac230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:03.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.099+0000 7f342d8e6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34280c75f0 0x7f34281f28e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:03.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.099+0000 7f342d8e6640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f342815e570 0x7f34280c5f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:03.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.099+0000 7f342d8e6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f34280731e0 con 0x7f34280c75f0 2024-04-25T00:19:03.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.099+0000 7f342d8e6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f342806ba40 con 0x7f34140039f0 2024-04-25T00:19:03.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.099+0000 7f342d8e6640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f3428073360 con 0x7f342815e570 2024-04-25T00:19:03.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.099+0000 7f341ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34280c75f0 0x7f34281f28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:03.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.099+0000 7f341ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34280c75f0 0x7f34281f28e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33300/0 (socket says 172.21.15.116:33300) 2024-04-25T00:19:03.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.099+0000 7f341ffff640 1 -- 172.21.15.116:0/115111523 learned_addr learned my addr 172.21.15.116:0/115111523 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:19:03.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.099+0000 7f34277fe640 1 --2- 172.21.15.116:0/115111523 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f342815e570 0x7f34280c5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:03.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.099+0000 7f3426ffd640 1 --2- 172.21.15.116:0/115111523 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f34140039f0 0x7f34280ac230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:03.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.099+0000 7f341ffff640 1 -- 172.21.15.116:0/115111523 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f342815e570 msgr2=0x7f34280c5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:03.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.099+0000 7f341ffff640 1 --2- 172.21.15.116:0/115111523 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f342815e570 0x7f34280c5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:03.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.099+0000 7f341ffff640 1 -- 172.21.15.116:0/115111523 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f34140039f0 msgr2=0x7f34280ac230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:03.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.099+0000 7f341ffff640 1 --2- 172.21.15.116:0/115111523 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f34140039f0 0x7f34280ac230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:03.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.099+0000 7f341ffff640 1 -- 172.21.15.116:0/115111523 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34281f2e20 con 0x7f34280c75f0 2024-04-25T00:19:03.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.099+0000 7f34277fe640 1 --2- 172.21.15.116:0/115111523 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f342815e570 0x7f34280c5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:19:03.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.099+0000 7f341ffff640 1 --2- 172.21.15.116:0/115111523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34280c75f0 0x7f34281f28e0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f341000d910 tx=0x7f341000ddd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:03.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.099+0000 7f3424ff9640 1 -- 172.21.15.116:0/115111523 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3410019070 con 0x7f34280c75f0 2024-04-25T00:19:03.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.099+0000 7f3424ff9640 1 -- 172.21.15.116:0/115111523 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34100074e0 con 0x7f34280c75f0 2024-04-25T00:19:03.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.099+0000 7f342d8e6640 1 -- 172.21.15.116:0/115111523 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f342814c4f0 con 0x7f34280c75f0 2024-04-25T00:19:03.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.099+0000 7f3424ff9640 1 -- 172.21.15.116:0/115111523 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34100049f0 con 0x7f34280c75f0 2024-04-25T00:19:03.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.099+0000 7f342d8e6640 1 -- 172.21.15.116:0/115111523 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34280ab750 con 0x7f34280c75f0 2024-04-25T00:19:03.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.099+0000 7f342d8e6640 1 -- 172.21.15.116:0/115111523 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3428068a60 con 0x7f34280c75f0 2024-04-25T00:19:03.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.099+0000 7f3424ff9640 1 -- 172.21.15.116:0/115111523 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f341001c070 con 0x7f34280c75f0 2024-04-25T00:19:03.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.099+0000 7f3424ff9640 1 --2- 172.21.15.116:0/115111523 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f34080678c0 0x7f3408069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:03.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.099+0000 7f3424ff9640 1 -- 172.21.15.116:0/115111523 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 8988+0+0 (secure 0 0 0) 0x7f3410086420 con 0x7f34280c75f0 2024-04-25T00:19:03.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.099+0000 7f3426ffd640 1 --2- 172.21.15.116:0/115111523 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f34080678c0 0x7f3408069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:03.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.103+0000 7f3426ffd640 1 --2- 172.21.15.116:0/115111523 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f34080678c0 0x7f3408069d80 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f34180ad000 tx=0x7f34180046d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:03.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.107+0000 7f3424ff9640 1 -- 172.21.15.116:0/115111523 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f34100562d0 con 0x7f34280c75f0 2024-04-25T00:19:03.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.327+0000 7f342d8e6640 1 -- 172.21.15.116:0/115111523 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3428070500 con 0x7f34280c75f0 2024-04-25T00:19:03.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.327+0000 7f3424ff9640 1 -- 172.21.15.116:0/115111523 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 147 v147) ==== 95+0+6506 (secure 0 0 0) 0x7f3410055bd0 con 0x7f34280c75f0 2024-04-25T00:19:03.325 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:19:03.325 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":147,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7542,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/879531976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":879531976},{"type":"v1","addr":"172.21.15.116:6835","nonce":879531976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":147}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:18:23.899096+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7362},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7362":{"gid":7362,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6833/1655959433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1655959433},{"type":"v1","addr":"172.21.15.116:6833","nonce":1655959433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7362,"qdb_cluster":[7362]},"id":1},{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:18:38.800778+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5879},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5879":{"gid":5879,"name":"d","rank":0,"incarnation":143,"state":"up:active","state_seq":6,"addr":"172.21.15.174:6837/2276529656","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2276529656},{"type":"v1","addr":"172.21.15.174:6837","nonce":2276529656}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5879,"qdb_cluster":[5879]},"id":2}]} 2024-04-25T00:19:03.325 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 147 2024-04-25T00:19:03.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.331+0000 7f342d8e6640 1 -- 172.21.15.116:0/115111523 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f34080678c0 msgr2=0x7f3408069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:03.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.331+0000 7f342d8e6640 1 --2- 172.21.15.116:0/115111523 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f34080678c0 0x7f3408069d80 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f34180ad000 tx=0x7f34180046d0 comp rx=0 tx=0).stop 2024-04-25T00:19:03.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.331+0000 7f342d8e6640 1 -- 172.21.15.116:0/115111523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34280c75f0 msgr2=0x7f34281f28e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:03.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.331+0000 7f342d8e6640 1 --2- 172.21.15.116:0/115111523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34280c75f0 0x7f34281f28e0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f341000d910 tx=0x7f341000ddd0 comp rx=0 tx=0).stop 2024-04-25T00:19:03.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.331+0000 7f342d8e6640 1 -- 172.21.15.116:0/115111523 shutdown_connections 2024-04-25T00:19:03.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.331+0000 7f342d8e6640 1 --2- 172.21.15.116:0/115111523 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f34080678c0 0x7f3408069d80 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:03.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.331+0000 7f342d8e6640 1 --2- 172.21.15.116:0/115111523 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f342815e570 0x7f34280c5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:03.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.331+0000 7f342d8e6640 1 --2- 172.21.15.116:0/115111523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34280c75f0 0x7f34281f28e0 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:03.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.331+0000 7f342d8e6640 1 --2- 172.21.15.116:0/115111523 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f34140039f0 0x7f34280ac230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:03.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.331+0000 7f342d8e6640 1 -- 172.21.15.116:0/115111523 >> 172.21.15.116:0/115111523 conn(0x7f34280c7fa0 msgr2=0x7f3428144e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:19:03.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.331+0000 7f342d8e6640 1 -- 172.21.15.116:0/115111523 shutdown_connections 2024-04-25T00:19:03.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:03.331+0000 7f342d8e6640 1 -- 172.21.15.116:0/115111523 wait complete. 2024-04-25T00:19:03.349 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:19:05.350 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:19:05.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.487+0000 7f25fc04f640 1 Processor -- start 2024-04-25T00:19:05.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.487+0000 7f25fc04f640 1 -- start start 2024-04-25T00:19:05.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.487+0000 7f25fc04f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f25f4158d30 0x7f25f406f270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:05.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.487+0000 7f25fc04f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25f41558e0 con 0x7f25f406f7b0 2024-04-25T00:19:05.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.487+0000 7f25fc04f640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25f4155bc0 con 0x7f25f4152ca0 2024-04-25T00:19:05.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.487+0000 7f25fc04f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f25f4155ea0 con 0x7f25f4158d30 2024-04-25T00:19:05.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.487+0000 7f25f95ea640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f25f4158d30 0x7f25f406f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:05.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.487+0000 7f25fa5ec640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f25f406f7b0 0x7f25f41600c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:47822/0 (socket says 172.21.15.116:47822) 2024-04-25T00:19:05.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.487+0000 7f25f95ea640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f25f4158d30 0x7f25f406f270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49104/0 (socket says 172.21.15.116:49104) 2024-04-25T00:19:05.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.487+0000 7f25fa5ec640 1 -- 172.21.15.116:0/3196479851 learned_addr learned my addr 172.21.15.116:0/3196479851 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:19:05.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.491+0000 7f25f95ea640 1 -- 172.21.15.116:0/3196479851 >> v1:172.21.15.174:6789/0 conn(0x7f25f4152ca0 legacy=0x7f25f406b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:19:05.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.491+0000 7f25f95ea640 1 -- 172.21.15.116:0/3196479851 >> v1:172.21.15.116:6789/0 conn(0x7f25f406f7b0 legacy=0x7f25f41600c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:05.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.491+0000 7f25f8de9640 1 -- 172.21.15.116:0/3196479851 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3994940617 0 0) 0x7f25f41558e0 con 0x7f25f406f7b0 2024-04-25T00:19:05.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.491+0000 7f25f95ea640 1 -- 172.21.15.116:0/3196479851 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25f4156180 con 0x7f25f4158d30 2024-04-25T00:19:05.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.491+0000 7f25f95ea640 1 --2- 172.21.15.116:0/3196479851 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25f4158d30 0x7f25f406f270 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f25e40097c0 tx=0x7f25e401c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5327cf2168aaa98 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:05.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.491+0000 7f25f8de9640 1 -- 172.21.15.116:0/3196479851 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25e40b0070 con 0x7f25f4158d30 2024-04-25T00:19:05.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.491+0000 7f25f8de9640 1 -- 172.21.15.116:0/3196479851 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25e40ab600 con 0x7f25f4158d30 2024-04-25T00:19:05.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.491+0000 7f25f8de9640 1 -- 172.21.15.116:0/3196479851 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25e40ab9d0 con 0x7f25f4158d30 2024-04-25T00:19:05.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.491+0000 7f25fc04f640 1 -- 172.21.15.116:0/3196479851 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25f4158d30 msgr2=0x7f25f406f270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:05.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.491+0000 7f25fc04f640 1 --2- 172.21.15.116:0/3196479851 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25f4158d30 0x7f25f406f270 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f25e40097c0 tx=0x7f25e401c3d0 comp rx=0 tx=0).stop 2024-04-25T00:19:05.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.491+0000 7f25fc04f640 1 -- 172.21.15.116:0/3196479851 shutdown_connections 2024-04-25T00:19:05.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.491+0000 7f25fc04f640 1 --2- 172.21.15.116:0/3196479851 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25f4158d30 0x7f25f406f270 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:05.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.491+0000 7f25fc04f640 1 -- 172.21.15.116:0/3196479851 >> 172.21.15.116:0/3196479851 conn(0x7f25f40c7fa0 msgr2=0x7f25f40c83b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:19:05.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.491+0000 7f25fc04f640 1 -- 172.21.15.116:0/3196479851 shutdown_connections 2024-04-25T00:19:05.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.491+0000 7f25fc04f640 1 -- 172.21.15.116:0/3196479851 wait complete. 2024-04-25T00:19:05.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.495+0000 7f25fc04f640 1 Processor -- start 2024-04-25T00:19:05.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.495+0000 7f25fc04f640 1 -- start start 2024-04-25T00:19:05.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.495+0000 7f25fc04f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25f406f7b0 0x7f25f4154d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:05.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.495+0000 7f25fc04f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25f4152ca0 0x7f25f413d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:05.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.495+0000 7f25fc04f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f25f4158d30 0x7f25f413dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:05.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.495+0000 7f25fc04f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f25f406baa0 con 0x7f25f4152ca0 2024-04-25T00:19:05.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.495+0000 7f25fc04f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f25f406b920 con 0x7f25f406f7b0 2024-04-25T00:19:05.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.495+0000 7f25fc04f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f25f4164410 con 0x7f25f4158d30 2024-04-25T00:19:05.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.495+0000 7f25f95ea640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25f4152ca0 0x7f25f413d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:05.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.495+0000 7f25f95ea640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25f4152ca0 0x7f25f413d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49112/0 (socket says 172.21.15.116:49112) 2024-04-25T00:19:05.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.495+0000 7f25f95ea640 1 -- 172.21.15.116:0/940535220 learned_addr learned my addr 172.21.15.116:0/940535220 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:19:05.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.495+0000 7f25fa5ec640 1 --2- 172.21.15.116:0/940535220 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f25f4158d30 0x7f25f413dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:05.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.495+0000 7f25f9deb640 1 --2- 172.21.15.116:0/940535220 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25f406f7b0 0x7f25f4154d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:05.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.495+0000 7f25f95ea640 1 -- 172.21.15.116:0/940535220 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f25f4158d30 msgr2=0x7f25f413dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:05.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.495+0000 7f25f95ea640 1 --2- 172.21.15.116:0/940535220 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f25f4158d30 0x7f25f413dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:05.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.495+0000 7f25f95ea640 1 -- 172.21.15.116:0/940535220 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25f406f7b0 msgr2=0x7f25f4154d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:05.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.495+0000 7f25f95ea640 1 --2- 172.21.15.116:0/940535220 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25f406f7b0 0x7f25f4154d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:05.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.495+0000 7f25f95ea640 1 -- 172.21.15.116:0/940535220 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25f4143a80 con 0x7f25f4152ca0 2024-04-25T00:19:05.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.495+0000 7f25fa5ec640 1 --2- 172.21.15.116:0/940535220 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f25f4158d30 0x7f25f413dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:19:05.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.495+0000 7f25f95ea640 1 --2- 172.21.15.116:0/940535220 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25f4152ca0 0x7f25f413d840 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f25e401a000 tx=0x7f25e4019170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:05.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.495+0000 7f25ea7fc640 1 -- 172.21.15.116:0/940535220 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25e40bb070 con 0x7f25f4152ca0 2024-04-25T00:19:05.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.495+0000 7f25ea7fc640 1 -- 172.21.15.116:0/940535220 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25e40b7460 con 0x7f25f4152ca0 2024-04-25T00:19:05.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.495+0000 7f25fc04f640 1 -- 172.21.15.116:0/940535220 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25f414d720 con 0x7f25f4152ca0 2024-04-25T00:19:05.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.495+0000 7f25ea7fc640 1 -- 172.21.15.116:0/940535220 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25e40b0040 con 0x7f25f4152ca0 2024-04-25T00:19:05.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.495+0000 7f25fc04f640 1 -- 172.21.15.116:0/940535220 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25f4143280 con 0x7f25f4152ca0 2024-04-25T00:19:05.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.495+0000 7f25fc04f640 1 -- 172.21.15.116:0/940535220 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25f4068a60 con 0x7f25f4152ca0 2024-04-25T00:19:05.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.499+0000 7f25ea7fc640 1 -- 172.21.15.116:0/940535220 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f25e40be070 con 0x7f25f4152ca0 2024-04-25T00:19:05.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.499+0000 7f25ea7fc640 1 --2- 172.21.15.116:0/940535220 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f25d80678c0 0x7f25d8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:05.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.499+0000 7f25ea7fc640 1 -- 172.21.15.116:0/940535220 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 8988+0+0 (secure 0 0 0) 0x7f25e4120d60 con 0x7f25f4152ca0 2024-04-25T00:19:05.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.499+0000 7f25f9deb640 1 --2- 172.21.15.116:0/940535220 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f25d80678c0 0x7f25d8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:05.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.499+0000 7f25f9deb640 1 --2- 172.21.15.116:0/940535220 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f25d80678c0 0x7f25d8069d80 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f25f0004e30 tx=0x7f25f0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:05.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.503+0000 7f25ea7fc640 1 -- 172.21.15.116:0/940535220 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f25e4016050 con 0x7f25f4152ca0 2024-04-25T00:19:05.770 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:19:05.770 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":148,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:18:23.899096+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7362},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7362":{"gid":7362,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6833/1655959433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1655959433},{"type":"v1","addr":"172.21.15.116:6833","nonce":1655959433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7362,"qdb_cluster":[7362]},"id":1},{"mdsmap":{"epoch":148,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:19:05.629360+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7542},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7542":{"gid":7542,"name":"c","rank":0,"incarnation":148,"state":"up:replay","state_seq":1,"addr":"172.21.15.116:6835/879531976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":879531976},{"type":"v1","addr":"172.21.15.116:6835","nonce":879531976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:19:05.771 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 148 2024-04-25T00:19:05.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.771+0000 7f25fc04f640 1 -- 172.21.15.116:0/940535220 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f25f4072de0 con 0x7f25f4152ca0 2024-04-25T00:19:05.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.771+0000 7f25ea7fc640 1 -- 172.21.15.116:0/940535220 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 148 v148) ==== 95+0+5670 (secure 0 0 0) 0x7f25e40f0c10 con 0x7f25f4152ca0 2024-04-25T00:19:05.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.775+0000 7f25fc04f640 1 -- 172.21.15.116:0/940535220 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f25d80678c0 msgr2=0x7f25d8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:05.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.775+0000 7f25fc04f640 1 --2- 172.21.15.116:0/940535220 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f25d80678c0 0x7f25d8069d80 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f25f0004e30 tx=0x7f25f0009040 comp rx=0 tx=0).stop 2024-04-25T00:19:05.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.775+0000 7f25fc04f640 1 -- 172.21.15.116:0/940535220 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25f4152ca0 msgr2=0x7f25f413d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:05.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.775+0000 7f25fc04f640 1 --2- 172.21.15.116:0/940535220 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25f4152ca0 0x7f25f413d840 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f25e401a000 tx=0x7f25e4019170 comp rx=0 tx=0).stop 2024-04-25T00:19:05.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.775+0000 7f25fc04f640 1 -- 172.21.15.116:0/940535220 shutdown_connections 2024-04-25T00:19:05.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.775+0000 7f25fc04f640 1 --2- 172.21.15.116:0/940535220 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f25d80678c0 0x7f25d8069d80 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:05.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.775+0000 7f25fc04f640 1 --2- 172.21.15.116:0/940535220 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f25f4158d30 0x7f25f413dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:05.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.775+0000 7f25fc04f640 1 --2- 172.21.15.116:0/940535220 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25f4152ca0 0x7f25f413d840 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:05.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.775+0000 7f25fc04f640 1 --2- 172.21.15.116:0/940535220 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25f406f7b0 0x7f25f4154d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:05.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.775+0000 7f25fc04f640 1 -- 172.21.15.116:0/940535220 >> 172.21.15.116:0/940535220 conn(0x7f25f40c7fa0 msgr2=0x7f25f4156cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:19:05.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.775+0000 7f25fc04f640 1 -- 172.21.15.116:0/940535220 shutdown_connections 2024-04-25T00:19:05.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.775+0000 7f25fc04f640 1 -- 172.21.15.116:0/940535220 wait complete. 2024-04-25T00:19:05.797 INFO:tasks.mds_thrash.fs.[b]:mds.d down, removed from mdsmap 2024-04-25T00:19:05.797 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-04-25T00:19:05.798 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:19:05.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.919+0000 7fa1b9703640 1 Processor -- start 2024-04-25T00:19:05.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.919+0000 7fa1b9703640 1 -- start start 2024-04-25T00:19:05.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.919+0000 7fa1b9703640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa1b40c6080 0x7fa1b406c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:05.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.919+0000 7fa1b9703640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1b414f7e0 con 0x7fa1b406c760 2024-04-25T00:19:05.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.919+0000 7fa1b9703640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1b4152060 con 0x7fa1b414f400 2024-04-25T00:19:05.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.919+0000 7fa1b9703640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa1b4152990 con 0x7fa1b40c6080 2024-04-25T00:19:05.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.923+0000 7fa1b37fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fa1b406c760 0x7fa1b406e630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:47838/0 (socket says 172.21.15.116:47838) 2024-04-25T00:19:05.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.923+0000 7fa1b37fe640 1 -- 172.21.15.116:0/1336181220 learned_addr learned my addr 172.21.15.116:0/1336181220 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:19:05.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.923+0000 7fa1abfff640 1 --2- 172.21.15.116:0/1336181220 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa1b40c6080 0x7fa1b406c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:05.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.923+0000 7fa1b27fc640 1 -- 172.21.15.116:0/1336181220 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1942498906 0 0) 0x7fa1b414f7e0 con 0x7fa1b406c760 2024-04-25T00:19:05.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.923+0000 7fa1b27fc640 1 -- 172.21.15.116:0/1336181220 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa1a0003530 con 0x7fa1b406c760 2024-04-25T00:19:05.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.923+0000 7fa1b27fc640 1 -- 172.21.15.116:0/1336181220 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 827935024 0 0) 0x7fa1b4152060 con 0x7fa1b414f400 2024-04-25T00:19:05.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.923+0000 7fa1b27fc640 1 -- 172.21.15.116:0/1336181220 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa1b414f7e0 con 0x7fa1b414f400 2024-04-25T00:19:05.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.923+0000 7fa1abfff640 1 -- 172.21.15.116:0/1336181220 >> v1:172.21.15.174:6789/0 conn(0x7fa1b414f400 legacy=0x7fa1b40c2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:05.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.923+0000 7fa1abfff640 1 -- 172.21.15.116:0/1336181220 >> v1:172.21.15.116:6789/0 conn(0x7fa1b406c760 legacy=0x7fa1b406e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:05.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.923+0000 7fa1b27fc640 1 -- 172.21.15.116:0/1336181220 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3784716661 0 0) 0x7fa1a0003530 con 0x7fa1b406c760 2024-04-25T00:19:05.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.923+0000 7fa1abfff640 1 -- 172.21.15.116:0/1336181220 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1b4152340 con 0x7fa1b40c6080 2024-04-25T00:19:05.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.923+0000 7fa1abfff640 1 --2- 172.21.15.116:0/1336181220 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa1b40c6080 0x7fa1b406c220 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa19c0097f0 tx=0x7fa19c01c370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=edd006fdb1b983d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:05.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.923+0000 7fa1b27fc640 1 -- 172.21.15.116:0/1336181220 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa19c0b0070 con 0x7fa1b40c6080 2024-04-25T00:19:05.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.927+0000 7fa1b27fc640 1 -- 172.21.15.116:0/1336181220 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa19c0ab400 con 0x7fa1b40c6080 2024-04-25T00:19:05.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.927+0000 7fa1b27fc640 1 -- 172.21.15.116:0/1336181220 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1ac001a00 con 0x7fa1b40c6080 2024-04-25T00:19:05.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.927+0000 7fa1b9703640 1 -- 172.21.15.116:0/1336181220 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa1b40c6080 msgr2=0x7fa1b406c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:05.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.927+0000 7fa1b9703640 1 --2- 172.21.15.116:0/1336181220 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa1b40c6080 0x7fa1b406c220 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa19c0097f0 tx=0x7fa19c01c370 comp rx=0 tx=0).stop 2024-04-25T00:19:05.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.927+0000 7fa1b9703640 1 -- 172.21.15.116:0/1336181220 shutdown_connections 2024-04-25T00:19:05.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.927+0000 7fa1b9703640 1 --2- 172.21.15.116:0/1336181220 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa1b40c6080 0x7fa1b406c220 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:05.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.927+0000 7fa1b9703640 1 -- 172.21.15.116:0/1336181220 >> 172.21.15.116:0/1336181220 conn(0x7fa1b40c4270 msgr2=0x7fa1b40c4680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:19:05.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.927+0000 7fa1b9703640 1 -- 172.21.15.116:0/1336181220 shutdown_connections 2024-04-25T00:19:05.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.927+0000 7fa1b9703640 1 -- 172.21.15.116:0/1336181220 wait complete. 2024-04-25T00:19:05.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.927+0000 7fa1b9703640 1 Processor -- start 2024-04-25T00:19:05.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.927+0000 7fa1b9703640 1 -- start start 2024-04-25T00:19:05.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.931+0000 7fa1b9703640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa1b406c760 0x7fa1b4153050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:05.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.931+0000 7fa1b9703640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa1b40c6080 0x7fa1b4072a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:05.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.931+0000 7fa1b9703640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1b414f400 0x7fa1b41f1f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:05.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.931+0000 7fa1b9703640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa1b415ffc0 con 0x7fa1b414f400 2024-04-25T00:19:05.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.931+0000 7fa1b9703640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa1b415fe40 con 0x7fa1b406c760 2024-04-25T00:19:05.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.931+0000 7fa1b9703640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa1b4160140 con 0x7fa1b40c6080 2024-04-25T00:19:05.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.931+0000 7fa1b37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1b414f400 0x7fa1b41f1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:05.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.931+0000 7fa1b37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1b414f400 0x7fa1b41f1f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49128/0 (socket says 172.21.15.116:49128) 2024-04-25T00:19:05.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.931+0000 7fa1b2ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa1b406c760 0x7fa1b4153050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:05.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.931+0000 7fa1b37fe640 1 -- 172.21.15.116:0/915221867 learned_addr learned my addr 172.21.15.116:0/915221867 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:19:05.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.931+0000 7fa1abfff640 1 --2- 172.21.15.116:0/915221867 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa1b40c6080 0x7fa1b4072a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:05.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.931+0000 7fa1b37fe640 1 -- 172.21.15.116:0/915221867 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa1b40c6080 msgr2=0x7fa1b4072a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:05.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.931+0000 7fa1b37fe640 1 --2- 172.21.15.116:0/915221867 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa1b40c6080 0x7fa1b4072a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:05.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.931+0000 7fa1b37fe640 1 -- 172.21.15.116:0/915221867 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa1b406c760 msgr2=0x7fa1b4153050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:05.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.931+0000 7fa1b37fe640 1 --2- 172.21.15.116:0/915221867 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa1b406c760 0x7fa1b4153050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:05.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.931+0000 7fa1b37fe640 1 -- 172.21.15.116:0/915221867 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1b41559b0 con 0x7fa1b414f400 2024-04-25T00:19:05.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.931+0000 7fa1b37fe640 1 --2- 172.21.15.116:0/915221867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1b414f400 0x7fa1b41f1f00 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac001570 tx=0x7fa1ac00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:05.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.931+0000 7fa1ab7fe640 1 -- 172.21.15.116:0/915221867 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1ac012400 con 0x7fa1b414f400 2024-04-25T00:19:05.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.931+0000 7fa1ab7fe640 1 -- 172.21.15.116:0/915221867 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1ac0125c0 con 0x7fa1b414f400 2024-04-25T00:19:05.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.931+0000 7fa1ab7fe640 1 -- 172.21.15.116:0/915221867 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1ac0129b0 con 0x7fa1b414f400 2024-04-25T00:19:05.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.931+0000 7fa1b9703640 1 -- 172.21.15.116:0/915221867 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1b4155c20 con 0x7fa1b414f400 2024-04-25T00:19:05.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.931+0000 7fa1b9703640 1 -- 172.21.15.116:0/915221867 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1b41f1700 con 0x7fa1b414f400 2024-04-25T00:19:05.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.931+0000 7fa1b9703640 1 -- 172.21.15.116:0/915221867 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1b4068a60 con 0x7fa1b414f400 2024-04-25T00:19:05.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.931+0000 7fa1ab7fe640 1 -- 172.21.15.116:0/915221867 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa1ac012b50 con 0x7fa1b414f400 2024-04-25T00:19:05.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.935+0000 7fa1ab7fe640 1 --2- 172.21.15.116:0/915221867 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa1940678c0 0x7fa194069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:05.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.935+0000 7fa1ab7fe640 1 -- 172.21.15.116:0/915221867 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 9074+0+0 (secure 0 0 0) 0x7fa1ac0870d0 con 0x7fa1b414f400 2024-04-25T00:19:05.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.935+0000 7fa1b2ffd640 1 --2- 172.21.15.116:0/915221867 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa1940678c0 0x7fa194069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:05.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.935+0000 7fa1b2ffd640 1 --2- 172.21.15.116:0/915221867 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa1940678c0 0x7fa194069d80 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fa1a40011f0 tx=0x7fa1a4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:05.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:05.939+0000 7fa1ab7fe640 1 -- 172.21.15.116:0/915221867 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fa1ac056870 con 0x7fa1b414f400 2024-04-25T00:19:06.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:06.175+0000 7fa1b9703640 1 -- 172.21.15.116:0/915221867 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa1b406fee0 con 0x7fa1b414f400 2024-04-25T00:19:06.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:06.175+0000 7fa1ab7fe640 1 -- 172.21.15.116:0/915221867 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 148 v148) ==== 95+0+5670 (secure 0 0 0) 0x7fa1ac05ab70 con 0x7fa1b414f400 2024-04-25T00:19:06.175 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:19:06.176 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":148,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:18:23.899096+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7362},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7362":{"gid":7362,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6833/1655959433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1655959433},{"type":"v1","addr":"172.21.15.116:6833","nonce":1655959433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7362,"qdb_cluster":[7362]},"id":1},{"mdsmap":{"epoch":148,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:19:05.629360+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7542},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7542":{"gid":7542,"name":"c","rank":0,"incarnation":148,"state":"up:replay","state_seq":1,"addr":"172.21.15.116:6835/879531976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":879531976},{"type":"v1","addr":"172.21.15.116:6835","nonce":879531976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:19:06.176 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 148 2024-04-25T00:19:06.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:06.179+0000 7fa1b9703640 1 -- 172.21.15.116:0/915221867 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa1940678c0 msgr2=0x7fa194069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:06.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:06.179+0000 7fa1b9703640 1 --2- 172.21.15.116:0/915221867 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa1940678c0 0x7fa194069d80 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fa1a40011f0 tx=0x7fa1a4009040 comp rx=0 tx=0).stop 2024-04-25T00:19:06.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:06.179+0000 7fa1b9703640 1 -- 172.21.15.116:0/915221867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1b414f400 msgr2=0x7fa1b41f1f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:06.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:06.179+0000 7fa1b9703640 1 --2- 172.21.15.116:0/915221867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1b414f400 0x7fa1b41f1f00 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac001570 tx=0x7fa1ac00a430 comp rx=0 tx=0).stop 2024-04-25T00:19:06.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:06.179+0000 7fa1b9703640 1 -- 172.21.15.116:0/915221867 shutdown_connections 2024-04-25T00:19:06.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:06.179+0000 7fa1b9703640 1 --2- 172.21.15.116:0/915221867 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa1940678c0 0x7fa194069d80 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:06.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:06.179+0000 7fa1b9703640 1 --2- 172.21.15.116:0/915221867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa1b414f400 0x7fa1b41f1f00 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:06.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:06.179+0000 7fa1b9703640 1 --2- 172.21.15.116:0/915221867 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa1b40c6080 0x7fa1b4072a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:06.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:06.179+0000 7fa1b9703640 1 --2- 172.21.15.116:0/915221867 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa1b406c760 0x7fa1b4153050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:06.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:06.179+0000 7fa1b9703640 1 -- 172.21.15.116:0/915221867 >> 172.21.15.116:0/915221867 conn(0x7fa1b40c4270 msgr2=0x7fa1b40c6db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:19:06.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:06.179+0000 7fa1b9703640 1 -- 172.21.15.116:0/915221867 shutdown_connections 2024-04-25T00:19:06.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:06.179+0000 7fa1b9703640 1 -- 172.21.15.116:0/915221867 wait complete. 2024-04-25T00:19:06.193 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 148, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:04.678737+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 141, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_7362": { "addr": "172.21.15.116:6833/1655959433", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6832", "nonce": 1655959433, "type": "v2" }, { "addr": "172.21.15.116:6833", "nonce": 1655959433, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7362, "incarnation": 138, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 12 } }, "last_failure": 0, "last_failure_osd_epoch": 54, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T00:18:23.899096+0000", "qdb_cluster": [ 7362 ], "qdb_leader": 7362, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7362 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:18.825887+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 148, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_7542": { "addr": "172.21.15.116:6835/879531976", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6834", "nonce": 879531976, "type": "v2" }, { "addr": "172.21.15.116:6835", "nonce": 879531976, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7542, "incarnation": 148, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 56, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-04-25T00:19:05.629360+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7542 } } } ], "standbys": [ { "addr": "172.21.15.174:6835/4112145782", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6834", "nonce": 4112145782, "type": "v2" }, { "addr": "172.21.15.174:6835", "nonce": 4112145782, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 112, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5582, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-25T00:19:08.195 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:19:08.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.343+0000 7f3867520640 1 Processor -- start 2024-04-25T00:19:08.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.343+0000 7f3867520640 1 -- start start 2024-04-25T00:19:08.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.343+0000 7f3867520640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3860071440 0x7f3860071820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:08.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.343+0000 7f3867520640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f38600aca30 0x7f38600c2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:08.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.343+0000 7f3867520640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3860153160 0x7f38601559f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:08.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.343+0000 7f3867520640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f38600a91b0 con 0x7f3860071440 2024-04-25T00:19:08.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.343+0000 7f3867520640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f38600c5040 con 0x7f38600aca30 2024-04-25T00:19:08.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.343+0000 7f3867520640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f38600a8ed0 con 0x7f3860153160 2024-04-25T00:19:08.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.343+0000 7f386651e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3860071440 0x7f3860071820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:08.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.343+0000 7f386651e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3860071440 0x7f3860071820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49144/0 (socket says 172.21.15.116:49144) 2024-04-25T00:19:08.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.343+0000 7f386651e640 1 -- 172.21.15.116:0/640595170 learned_addr learned my addr 172.21.15.116:0/640595170 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:19:08.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.343+0000 7f3866d1f640 1 --2- 172.21.15.116:0/640595170 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3860153160 0x7f38601559f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:08.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.343+0000 7f3865d1d640 1 --2- 172.21.15.116:0/640595170 >> v2:172.21.15.174:3300/0 conn(0x7f38600aca30 0x7f38600c2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:08.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.343+0000 7f386651e640 1 -- 172.21.15.116:0/640595170 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3860153160 msgr2=0x7f38601559f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:19:08.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.343+0000 7f386651e640 1 --2- 172.21.15.116:0/640595170 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3860153160 0x7f38601559f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:08.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.343+0000 7f386651e640 1 -- 172.21.15.116:0/640595170 >> v2:172.21.15.174:3300/0 conn(0x7f38600aca30 msgr2=0x7f38600c2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:19:08.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.343+0000 7f386651e640 1 --2- 172.21.15.116:0/640595170 >> v2:172.21.15.174:3300/0 conn(0x7f38600aca30 0x7f38600c2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:08.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.343+0000 7f386651e640 1 -- 172.21.15.116:0/640595170 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3860073470 con 0x7f3860071440 2024-04-25T00:19:08.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.347+0000 7f386651e640 1 --2- 172.21.15.116:0/640595170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3860071440 0x7f3860071820 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f386006e2f0 tx=0x7f385c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fed76e49e7b9d8a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:08.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.347+0000 7f386551c640 1 -- 172.21.15.116:0/640595170 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f385c0ab5f0 con 0x7f3860071440 2024-04-25T00:19:08.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.347+0000 7f386551c640 1 -- 172.21.15.116:0/640595170 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f385c0b0070 con 0x7f3860071440 2024-04-25T00:19:08.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.347+0000 7f386551c640 1 -- 172.21.15.116:0/640595170 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f385c0ab9b0 con 0x7f3860071440 2024-04-25T00:19:08.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.347+0000 7f3867520640 1 -- 172.21.15.116:0/640595170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3860071440 msgr2=0x7f3860071820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:08.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.347+0000 7f3867520640 1 --2- 172.21.15.116:0/640595170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3860071440 0x7f3860071820 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f386006e2f0 tx=0x7f385c01c520 comp rx=0 tx=0).stop 2024-04-25T00:19:08.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.347+0000 7f3867520640 1 -- 172.21.15.116:0/640595170 shutdown_connections 2024-04-25T00:19:08.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.347+0000 7f3867520640 1 --2- 172.21.15.116:0/640595170 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3860153160 0x7f38601559f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:08.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.347+0000 7f3867520640 1 --2- 172.21.15.116:0/640595170 >> v2:172.21.15.174:3300/0 conn(0x7f38600aca30 0x7f38600c2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:08.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.347+0000 7f3867520640 1 --2- 172.21.15.116:0/640595170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3860071440 0x7f3860071820 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:08.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.347+0000 7f3867520640 1 -- 172.21.15.116:0/640595170 >> 172.21.15.116:0/640595170 conn(0x7f38600a9a20 msgr2=0x7f38600c4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:19:08.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.347+0000 7f3867520640 1 -- 172.21.15.116:0/640595170 shutdown_connections 2024-04-25T00:19:08.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.347+0000 7f3867520640 1 -- 172.21.15.116:0/640595170 wait complete. 2024-04-25T00:19:08.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.351+0000 7f3867520640 1 Processor -- start 2024-04-25T00:19:08.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.351+0000 7f3867520640 1 -- start start 2024-04-25T00:19:08.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.351+0000 7f3867520640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3860071440 0x7f386006b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:08.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.351+0000 7f3867520640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3860153160 0x7f38601483e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:08.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.351+0000 7f3867520640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f386006bc60 0x7f3860148920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:08.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.351+0000 7f3867520640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f386014fe20 con 0x7f3860153160 2024-04-25T00:19:08.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.351+0000 7f3867520640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f386014fca0 con 0x7f386006bc60 2024-04-25T00:19:08.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.351+0000 7f3867520640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f386015ed90 con 0x7f3860071440 2024-04-25T00:19:08.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.351+0000 7f3865d1d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3860153160 0x7f38601483e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:08.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.351+0000 7f3865d1d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3860153160 0x7f38601483e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49152/0 (socket says 172.21.15.116:49152) 2024-04-25T00:19:08.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.351+0000 7f3865d1d640 1 -- 172.21.15.116:0/576339225 learned_addr learned my addr 172.21.15.116:0/576339225 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:19:08.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.351+0000 7f386651e640 1 --2- 172.21.15.116:0/576339225 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3860071440 0x7f386006b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:08.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.351+0000 7f3866d1f640 1 --2- 172.21.15.116:0/576339225 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f386006bc60 0x7f3860148920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:08.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.351+0000 7f3865d1d640 1 -- 172.21.15.116:0/576339225 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3860071440 msgr2=0x7f386006b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:08.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.351+0000 7f3865d1d640 1 --2- 172.21.15.116:0/576339225 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3860071440 0x7f386006b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:08.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.351+0000 7f3865d1d640 1 -- 172.21.15.116:0/576339225 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f386006bc60 msgr2=0x7f3860148920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:08.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.351+0000 7f3865d1d640 1 --2- 172.21.15.116:0/576339225 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f386006bc60 0x7f3860148920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:08.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.351+0000 7f3865d1d640 1 -- 172.21.15.116:0/576339225 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38601f03b0 con 0x7f3860153160 2024-04-25T00:19:08.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.351+0000 7f3865d1d640 1 --2- 172.21.15.116:0/576339225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3860153160 0x7f38601483e0 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f385000d830 tx=0x7f385000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:08.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.351+0000 7f3856ffd640 1 -- 172.21.15.116:0/576339225 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3850018020 con 0x7f3860153160 2024-04-25T00:19:08.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.351+0000 7f3856ffd640 1 -- 172.21.15.116:0/576339225 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38500096a0 con 0x7f3860153160 2024-04-25T00:19:08.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.351+0000 7f3867520640 1 -- 172.21.15.116:0/576339225 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f386014b250 con 0x7f3860153160 2024-04-25T00:19:08.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.351+0000 7f3856ffd640 1 -- 172.21.15.116:0/576339225 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3850009a50 con 0x7f3860153160 2024-04-25T00:19:08.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.351+0000 7f3867520640 1 -- 172.21.15.116:0/576339225 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f386014b810 con 0x7f3860153160 2024-04-25T00:19:08.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.355+0000 7f3867520640 1 -- 172.21.15.116:0/576339225 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38601f0140 con 0x7f3860153160 2024-04-25T00:19:08.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.355+0000 7f3856ffd640 1 -- 172.21.15.116:0/576339225 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f385001b020 con 0x7f3860153160 2024-04-25T00:19:08.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.355+0000 7f3856ffd640 1 --2- 172.21.15.116:0/576339225 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f38440678f0 0x7f3844069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:08.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.355+0000 7f3856ffd640 1 -- 172.21.15.116:0/576339225 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 9074+0+0 (secure 0 0 0) 0x7f3850087670 con 0x7f3860153160 2024-04-25T00:19:08.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.355+0000 7f386651e640 1 --2- 172.21.15.116:0/576339225 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f38440678f0 0x7f3844069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:08.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.359+0000 7f386651e640 1 --2- 172.21.15.116:0/576339225 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f38440678f0 0x7f3844069db0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f385c0af000 tx=0x7f385c0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:08.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.359+0000 7f3856ffd640 1 -- 172.21.15.116:0/576339225 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f3850022020 con 0x7f3860153160 2024-04-25T00:19:08.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.619+0000 7f3867520640 1 -- 172.21.15.116:0/576339225 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f386004f510 con 0x7f3860153160 2024-04-25T00:19:08.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.619+0000 7f3856ffd640 1 -- 172.21.15.116:0/576339225 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 150 v150) ==== 95+0+5670 (secure 0 0 0) 0x7f38500574e0 con 0x7f3860153160 2024-04-25T00:19:08.618 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:19:08.618 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":150,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:18:23.899096+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7362},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7362":{"gid":7362,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6833/1655959433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1655959433},{"type":"v1","addr":"172.21.15.116:6833","nonce":1655959433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7362,"qdb_cluster":[7362]},"id":1},{"mdsmap":{"epoch":150,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:19:06.836759+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7542},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7542":{"gid":7542,"name":"c","rank":0,"incarnation":148,"state":"up:rejoin","state_seq":8,"addr":"172.21.15.116:6835/879531976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":879531976},{"type":"v1","addr":"172.21.15.116:6835","nonce":879531976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:19:08.619 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 150 2024-04-25T00:19:08.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.623+0000 7f3867520640 1 -- 172.21.15.116:0/576339225 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f38440678f0 msgr2=0x7f3844069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:08.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.623+0000 7f3867520640 1 --2- 172.21.15.116:0/576339225 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f38440678f0 0x7f3844069db0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f385c0af000 tx=0x7f385c0a7000 comp rx=0 tx=0).stop 2024-04-25T00:19:08.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.623+0000 7f3867520640 1 -- 172.21.15.116:0/576339225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3860153160 msgr2=0x7f38601483e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:08.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.623+0000 7f3867520640 1 --2- 172.21.15.116:0/576339225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3860153160 0x7f38601483e0 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f385000d830 tx=0x7f385000dcf0 comp rx=0 tx=0).stop 2024-04-25T00:19:08.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.623+0000 7f3867520640 1 -- 172.21.15.116:0/576339225 shutdown_connections 2024-04-25T00:19:08.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.623+0000 7f3867520640 1 --2- 172.21.15.116:0/576339225 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f38440678f0 0x7f3844069db0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:08.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.623+0000 7f3867520640 1 --2- 172.21.15.116:0/576339225 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f386006bc60 0x7f3860148920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:08.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.623+0000 7f3867520640 1 --2- 172.21.15.116:0/576339225 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3860153160 0x7f38601483e0 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:08.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.623+0000 7f3867520640 1 --2- 172.21.15.116:0/576339225 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3860071440 0x7f386006b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:08.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.623+0000 7f3867520640 1 -- 172.21.15.116:0/576339225 >> 172.21.15.116:0/576339225 conn(0x7f38600a9a20 msgr2=0x7f38600c4280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:19:08.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.623+0000 7f3867520640 1 -- 172.21.15.116:0/576339225 shutdown_connections 2024-04-25T00:19:08.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:08.623+0000 7f3867520640 1 -- 172.21.15.116:0/576339225 wait complete. 2024-04-25T00:19:08.641 INFO:tasks.mds_thrash.fs.[b]:no change 2024-04-25T00:19:09.325 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:19:09.328 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:19:10.642 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:19:10.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:10.775+0000 7ff073c4d640 1 Processor -- start 2024-04-25T00:19:10.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:10.775+0000 7ff073c4d640 1 -- start start 2024-04-25T00:19:10.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:10.775+0000 7ff073c4d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff06c0c56e0 0x7ff06c0a6d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:10.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:10.775+0000 7ff073c4d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff06c074770 con 0x7ff06c0c3cd0 2024-04-25T00:19:10.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:10.775+0000 7ff073c4d640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff06c074a30 con 0x7ff06c0c19d0 2024-04-25T00:19:10.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:10.775+0000 7ff073c4d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7ff06c0c21e0 con 0x7ff06c0c56e0 2024-04-25T00:19:10.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:10.775+0000 7ff0711e8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff06c0c56e0 0x7ff06c0a6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:10.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:10.775+0000 7ff0721ea640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7ff06c0c3cd0 0x7ff06c06c220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:47852/0 (socket says 172.21.15.116:47852) 2024-04-25T00:19:10.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:10.775+0000 7ff0711e8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff06c0c56e0 0x7ff06c0a6d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49158/0 (socket says 172.21.15.116:49158) 2024-04-25T00:19:10.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:10.775+0000 7ff0721ea640 1 -- 172.21.15.116:0/2796015882 learned_addr learned my addr 172.21.15.116:0/2796015882 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:19:10.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:10.775+0000 7ff0711e8640 1 -- 172.21.15.116:0/2796015882 >> v1:172.21.15.174:6789/0 conn(0x7ff06c0c19d0 legacy=0x7ff06c0c0c10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T00:19:10.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:10.775+0000 7ff0711e8640 1 -- 172.21.15.116:0/2796015882 >> v1:172.21.15.116:6789/0 conn(0x7ff06c0c3cd0 legacy=0x7ff06c06c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:10.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:10.775+0000 7ff0709e7640 1 -- 172.21.15.116:0/2796015882 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 575679482 0 0) 0x7ff06c074770 con 0x7ff06c0c3cd0 2024-04-25T00:19:10.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:10.775+0000 7ff0711e8640 1 -- 172.21.15.116:0/2796015882 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff06c159970 con 0x7ff06c0c56e0 2024-04-25T00:19:10.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:10.779+0000 7ff0711e8640 1 --2- 172.21.15.116:0/2796015882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff06c0c56e0 0x7ff06c0a6d10 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7ff05c0097c0 tx=0x7ff05c01c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adfe6dfc7dd6c027 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:10.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:10.779+0000 7ff0709e7640 1 -- 172.21.15.116:0/2796015882 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff05c0b0070 con 0x7ff06c0c56e0 2024-04-25T00:19:10.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:10.779+0000 7ff0709e7640 1 -- 172.21.15.116:0/2796015882 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff05c0ab600 con 0x7ff06c0c56e0 2024-04-25T00:19:10.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:10.779+0000 7ff0709e7640 1 -- 172.21.15.116:0/2796015882 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff05c0ab9d0 con 0x7ff06c0c56e0 2024-04-25T00:19:10.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:10.779+0000 7ff073c4d640 1 -- 172.21.15.116:0/2796015882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff06c0c56e0 msgr2=0x7ff06c0a6d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:10.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:10.779+0000 7ff073c4d640 1 --2- 172.21.15.116:0/2796015882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff06c0c56e0 0x7ff06c0a6d10 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7ff05c0097c0 tx=0x7ff05c01c3d0 comp rx=0 tx=0).stop 2024-04-25T00:19:10.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:10.779+0000 7ff073c4d640 1 -- 172.21.15.116:0/2796015882 shutdown_connections 2024-04-25T00:19:10.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:10.779+0000 7ff073c4d640 1 --2- 172.21.15.116:0/2796015882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff06c0c56e0 0x7ff06c0a6d10 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:10.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:10.779+0000 7ff073c4d640 1 -- 172.21.15.116:0/2796015882 >> 172.21.15.116:0/2796015882 conn(0x7ff06c0c42e0 msgr2=0x7ff06c0c46f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:19:10.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:10.779+0000 7ff073c4d640 1 -- 172.21.15.116:0/2796015882 shutdown_connections 2024-04-25T00:19:10.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:10.779+0000 7ff073c4d640 1 -- 172.21.15.116:0/2796015882 wait complete. 2024-04-25T00:19:10.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:10.779+0000 7ff073c4d640 1 Processor -- start 2024-04-25T00:19:10.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:10.779+0000 7ff073c4d640 1 -- start start 2024-04-25T00:19:10.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:10.783+0000 7ff073c4d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff06c0c19d0 0x7ff06c1eeea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:10.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:10.783+0000 7ff073c4d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff06c0c3cd0 0x7ff06c151140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:10.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:10.783+0000 7ff073c4d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff06c0c56e0 0x7ff06c0ab930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:10.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:10.783+0000 7ff073c4d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff06c152e00 con 0x7ff06c0c19d0 2024-04-25T00:19:10.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:10.783+0000 7ff073c4d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff06c0c5dc0 con 0x7ff06c0c3cd0 2024-04-25T00:19:10.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:10.783+0000 7ff073c4d640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff06c152f80 con 0x7ff06c0c56e0 2024-04-25T00:19:10.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:10.783+0000 7ff0719e9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff06c0c19d0 0x7ff06c1eeea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:10.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:10.783+0000 7ff0719e9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff06c0c19d0 0x7ff06c1eeea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49170/0 (socket says 172.21.15.116:49170) 2024-04-25T00:19:10.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:10.783+0000 7ff0719e9640 1 -- 172.21.15.116:0/148995554 learned_addr learned my addr 172.21.15.116:0/148995554 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:19:10.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:10.783+0000 7ff0711e8640 1 --2- 172.21.15.116:0/148995554 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff06c0c3cd0 0x7ff06c151140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:10.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:10.783+0000 7ff0721ea640 1 --2- 172.21.15.116:0/148995554 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff06c0c56e0 0x7ff06c0ab930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:10.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:10.783+0000 7ff0719e9640 1 -- 172.21.15.116:0/148995554 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff06c0c56e0 msgr2=0x7ff06c0ab930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:10.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:10.783+0000 7ff0719e9640 1 --2- 172.21.15.116:0/148995554 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff06c0c56e0 0x7ff06c0ab930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:10.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:10.783+0000 7ff0719e9640 1 -- 172.21.15.116:0/148995554 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff06c0c3cd0 msgr2=0x7ff06c151140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:10.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:10.783+0000 7ff0719e9640 1 --2- 172.21.15.116:0/148995554 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff06c0c3cd0 0x7ff06c151140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:10.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:10.783+0000 7ff0719e9640 1 -- 172.21.15.116:0/148995554 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff06c1ee400 con 0x7ff06c0c19d0 2024-04-25T00:19:10.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:10.783+0000 7ff0721ea640 1 --2- 172.21.15.116:0/148995554 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff06c0c56e0 0x7ff06c0ab930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:19:10.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:10.783+0000 7ff0719e9640 1 --2- 172.21.15.116:0/148995554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff06c0c19d0 0x7ff06c1eeea0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7ff06800b4a0 tx=0x7ff06800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:10.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:10.783+0000 7ff0627fc640 1 -- 172.21.15.116:0/148995554 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff068014070 con 0x7ff06c0c19d0 2024-04-25T00:19:10.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:10.783+0000 7ff0627fc640 1 -- 172.21.15.116:0/148995554 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff068010610 con 0x7ff06c0c19d0 2024-04-25T00:19:10.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:10.783+0000 7ff0627fc640 1 -- 172.21.15.116:0/148995554 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0680109e0 con 0x7ff06c0c19d0 2024-04-25T00:19:10.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:10.783+0000 7ff073c4d640 1 -- 172.21.15.116:0/148995554 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff06c1543d0 con 0x7ff06c0c19d0 2024-04-25T00:19:10.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:10.783+0000 7ff073c4d640 1 -- 172.21.15.116:0/148995554 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff06c1ee980 con 0x7ff06c0c19d0 2024-04-25T00:19:10.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:10.783+0000 7ff073c4d640 1 -- 172.21.15.116:0/148995554 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff06c068a60 con 0x7ff06c0c19d0 2024-04-25T00:19:10.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:10.783+0000 7ff0627fc640 1 -- 172.21.15.116:0/148995554 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff068017070 con 0x7ff06c0c19d0 2024-04-25T00:19:10.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:10.783+0000 7ff0627fc640 1 --2- 172.21.15.116:0/148995554 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff0500678c0 0x7ff050069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:10.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:10.787+0000 7ff0627fc640 1 -- 172.21.15.116:0/148995554 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 9074+0+0 (secure 0 0 0) 0x7ff068086470 con 0x7ff06c0c19d0 2024-04-25T00:19:10.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:10.787+0000 7ff0711e8640 1 --2- 172.21.15.116:0/148995554 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff0500678c0 0x7ff050069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:10.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:10.787+0000 7ff0711e8640 1 --2- 172.21.15.116:0/148995554 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff0500678c0 0x7ff050069d80 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7ff05c0ad000 tx=0x7ff05c0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:10.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:10.791+0000 7ff0627fc640 1 -- 172.21.15.116:0/148995554 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ff068056230 con 0x7ff06c0c19d0 2024-04-25T00:19:11.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:11.043+0000 7ff073c4d640 1 -- 172.21.15.116:0/148995554 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff06c074550 con 0x7ff06c0c19d0 2024-04-25T00:19:11.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:11.043+0000 7ff0627fc640 1 -- 172.21.15.116:0/148995554 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) ==== 95+0+5677 (secure 0 0 0) 0x7ff068055b30 con 0x7ff06c0c19d0 2024-04-25T00:19:11.043 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:19:11.043 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":151,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:18:23.899096+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7362},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7362":{"gid":7362,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6833/1655959433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1655959433},{"type":"v1","addr":"172.21.15.116:6833","nonce":1655959433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7362,"qdb_cluster":[7362]},"id":1},{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:19:08.844382+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7542},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7542":{"gid":7542,"name":"c","rank":0,"incarnation":148,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6835/879531976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":879531976},{"type":"v1","addr":"172.21.15.116:6835","nonce":879531976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7542,"qdb_cluster":[7542]},"id":2}]} 2024-04-25T00:19:11.043 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 151 2024-04-25T00:19:11.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:11.047+0000 7ff073c4d640 1 -- 172.21.15.116:0/148995554 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff0500678c0 msgr2=0x7ff050069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:11.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:11.047+0000 7ff073c4d640 1 --2- 172.21.15.116:0/148995554 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff0500678c0 0x7ff050069d80 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7ff05c0ad000 tx=0x7ff05c0a6000 comp rx=0 tx=0).stop 2024-04-25T00:19:11.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:11.047+0000 7ff073c4d640 1 -- 172.21.15.116:0/148995554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff06c0c19d0 msgr2=0x7ff06c1eeea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:11.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:11.047+0000 7ff073c4d640 1 --2- 172.21.15.116:0/148995554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff06c0c19d0 0x7ff06c1eeea0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7ff06800b4a0 tx=0x7ff06800b960 comp rx=0 tx=0).stop 2024-04-25T00:19:11.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:11.047+0000 7ff073c4d640 1 -- 172.21.15.116:0/148995554 shutdown_connections 2024-04-25T00:19:11.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:11.047+0000 7ff073c4d640 1 --2- 172.21.15.116:0/148995554 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff0500678c0 0x7ff050069d80 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:11.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:11.047+0000 7ff073c4d640 1 --2- 172.21.15.116:0/148995554 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff06c0c56e0 0x7ff06c0ab930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:11.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:11.047+0000 7ff073c4d640 1 --2- 172.21.15.116:0/148995554 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff06c0c3cd0 0x7ff06c151140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:11.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:11.047+0000 7ff073c4d640 1 --2- 172.21.15.116:0/148995554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff06c0c19d0 0x7ff06c1eeea0 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:11.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:11.047+0000 7ff073c4d640 1 -- 172.21.15.116:0/148995554 >> 172.21.15.116:0/148995554 conn(0x7ff06c0c42e0 msgr2=0x7ff06c0c46c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:19:11.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:11.047+0000 7ff073c4d640 1 -- 172.21.15.116:0/148995554 shutdown_connections 2024-04-25T00:19:11.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:11.047+0000 7ff073c4d640 1 -- 172.21.15.116:0/148995554 wait complete. 2024-04-25T00:19:11.060 INFO:tasks.mds_thrash.fs.[b]:mds.c has gained rank=0, replacing gid=5879 2024-04-25T00:19:11.060 INFO:tasks.mds_thrash.fs.[b]:waiting for 5 secs before reviving mds.d 2024-04-25T00:19:11.827 INFO:tasks.ceph.mds.c.smithi116.stderr:2024-04-25T00:19:11.827+0000 7f970ccc3640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T00:19:16.061 INFO:tasks.mds_thrash.fs.[b]:reviving mds.d 2024-04-25T00:19:16.061 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-25T00:19:16.061 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-25T00:19:16.064 INFO:tasks.ceph.mds.d:Started 2024-04-25T00:19:16.064 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:19:16.138 INFO:tasks.ceph.mds.d.smithi174.stdout:starting mds.d at 2024-04-25T00:19:16.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.203+0000 7f1f55b28640 1 Processor -- start 2024-04-25T00:19:16.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.203+0000 7f1f55b28640 1 -- start start 2024-04-25T00:19:16.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.207+0000 7f1f55b28640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f1f50150d20 0x7f1f500c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:16.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.207+0000 7f1f55b28640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1f500c2f50 0x7f1f501538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:16.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.207+0000 7f1f55b28640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1f500a94a0 0x7f1f50153e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:16.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.207+0000 7f1f55b28640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f1f500c3910 con 0x7f1f500a94a0 2024-04-25T00:19:16.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.207+0000 7f1f55b28640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f1f500c16e0 con 0x7f1f50150d20 2024-04-25T00:19:16.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.207+0000 7f1f55b28640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f1f500c1400 con 0x7f1f500c2f50 2024-04-25T00:19:16.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.207+0000 7f1f4ffff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1f500a94a0 0x7f1f50153e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:16.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.207+0000 7f1f4ffff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1f500a94a0 0x7f1f50153e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35734/0 (socket says 172.21.15.116:35734) 2024-04-25T00:19:16.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.207+0000 7f1f4ffff640 1 -- 172.21.15.116:0/3445758743 learned_addr learned my addr 172.21.15.116:0/3445758743 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:19:16.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.207+0000 7f1f4f7fe640 1 --2- 172.21.15.116:0/3445758743 >> v2:172.21.15.174:3300/0 conn(0x7f1f50150d20 0x7f1f500c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:16.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.207+0000 7f1f4effd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1f500c2f50 0x7f1f501538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:16.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.207+0000 7f1f4ffff640 1 -- 172.21.15.116:0/3445758743 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1f500c2f50 msgr2=0x7f1f501538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:19:16.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.207+0000 7f1f4ffff640 1 --2- 172.21.15.116:0/3445758743 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1f500c2f50 0x7f1f501538e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:16.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.207+0000 7f1f4ffff640 1 -- 172.21.15.116:0/3445758743 >> v2:172.21.15.174:3300/0 conn(0x7f1f50150d20 msgr2=0x7f1f500c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:19:16.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.207+0000 7f1f4ffff640 1 --2- 172.21.15.116:0/3445758743 >> v2:172.21.15.174:3300/0 conn(0x7f1f50150d20 0x7f1f500c2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:16.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.207+0000 7f1f4ffff640 1 -- 172.21.15.116:0/3445758743 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f5006e660 con 0x7f1f500a94a0 2024-04-25T00:19:16.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.207+0000 7f1f4ffff640 1 --2- 172.21.15.116:0/3445758743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f500a94a0 0x7f1f50153e20 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f1f50073550 tx=0x7f1f4801c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe5ebbc4017b69b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:16.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.207+0000 7f1f4e7fc640 1 -- 172.21.15.116:0/3445758743 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f480a5740 con 0x7f1f500a94a0 2024-04-25T00:19:16.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.207+0000 7f1f4e7fc640 1 -- 172.21.15.116:0/3445758743 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f480a58e0 con 0x7f1f500a94a0 2024-04-25T00:19:16.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.207+0000 7f1f4e7fc640 1 -- 172.21.15.116:0/3445758743 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f480a5cb0 con 0x7f1f500a94a0 2024-04-25T00:19:16.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.211+0000 7f1f55b28640 1 -- 172.21.15.116:0/3445758743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f500a94a0 msgr2=0x7f1f50153e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:16.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.211+0000 7f1f55b28640 1 --2- 172.21.15.116:0/3445758743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f500a94a0 0x7f1f50153e20 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f1f50073550 tx=0x7f1f4801c9f0 comp rx=0 tx=0).stop 2024-04-25T00:19:16.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.211+0000 7f1f55b28640 1 -- 172.21.15.116:0/3445758743 shutdown_connections 2024-04-25T00:19:16.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.211+0000 7f1f55b28640 1 --2- 172.21.15.116:0/3445758743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f500a94a0 0x7f1f50153e20 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:16.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.211+0000 7f1f55b28640 1 --2- 172.21.15.116:0/3445758743 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1f500c2f50 0x7f1f501538e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:16.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.211+0000 7f1f55b28640 1 --2- 172.21.15.116:0/3445758743 >> v2:172.21.15.174:3300/0 conn(0x7f1f50150d20 0x7f1f500c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:16.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.211+0000 7f1f55b28640 1 -- 172.21.15.116:0/3445758743 >> 172.21.15.116:0/3445758743 conn(0x7f1f500c3b00 msgr2=0x7f1f500c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:19:16.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.211+0000 7f1f55b28640 1 -- 172.21.15.116:0/3445758743 shutdown_connections 2024-04-25T00:19:16.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.211+0000 7f1f55b28640 1 -- 172.21.15.116:0/3445758743 wait complete. 2024-04-25T00:19:16.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.211+0000 7f1f55b28640 1 Processor -- start 2024-04-25T00:19:16.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.211+0000 7f1f55b28640 1 -- start start 2024-04-25T00:19:16.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.211+0000 7f1f55b28640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f500a94a0 0x7f1f500c6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:16.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.211+0000 7f1f55b28640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1f500c2f50 0x7f1f500c6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:16.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.211+0000 7f1f55b28640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1f50150d20 0x7f1f50145950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:16.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.211+0000 7f1f55b28640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1f50157340 con 0x7f1f500a94a0 2024-04-25T00:19:16.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.211+0000 7f1f55b28640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1f50154570 con 0x7f1f500c2f50 2024-04-25T00:19:16.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.211+0000 7f1f55b28640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f1f501574c0 con 0x7f1f50150d20 2024-04-25T00:19:16.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.211+0000 7f1f4f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f500a94a0 0x7f1f500c6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:16.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.211+0000 7f1f4f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f500a94a0 0x7f1f500c6350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35742/0 (socket says 172.21.15.116:35742) 2024-04-25T00:19:16.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.211+0000 7f1f4f7fe640 1 -- 172.21.15.116:0/356536236 learned_addr learned my addr 172.21.15.116:0/356536236 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:19:16.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.211+0000 7f1f4effd640 1 --2- 172.21.15.116:0/356536236 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1f500c2f50 0x7f1f500c6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:16.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.211+0000 7f1f4ffff640 1 --2- 172.21.15.116:0/356536236 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1f50150d20 0x7f1f50145950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:16.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.211+0000 7f1f4f7fe640 1 -- 172.21.15.116:0/356536236 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1f50150d20 msgr2=0x7f1f50145950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:16.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.211+0000 7f1f4f7fe640 1 --2- 172.21.15.116:0/356536236 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1f50150d20 0x7f1f50145950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:16.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.211+0000 7f1f4f7fe640 1 -- 172.21.15.116:0/356536236 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1f500c2f50 msgr2=0x7f1f500c6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:16.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.211+0000 7f1f4f7fe640 1 --2- 172.21.15.116:0/356536236 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1f500c2f50 0x7f1f500c6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:16.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.211+0000 7f1f4f7fe640 1 -- 172.21.15.116:0/356536236 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f500c6dd0 con 0x7f1f500a94a0 2024-04-25T00:19:16.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.211+0000 7f1f4f7fe640 1 --2- 172.21.15.116:0/356536236 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f500a94a0 0x7f1f500c6350 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f1f4000e870 tx=0x7f1f4000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:16.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.211+0000 7f1f37fff640 1 -- 172.21.15.116:0/356536236 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f40017070 con 0x7f1f500a94a0 2024-04-25T00:19:16.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.211+0000 7f1f37fff640 1 -- 172.21.15.116:0/356536236 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f4000c940 con 0x7f1f500a94a0 2024-04-25T00:19:16.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.211+0000 7f1f37fff640 1 -- 172.21.15.116:0/356536236 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f4000ccf0 con 0x7f1f500a94a0 2024-04-25T00:19:16.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.211+0000 7f1f55b28640 1 -- 172.21.15.116:0/356536236 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f50144050 con 0x7f1f500a94a0 2024-04-25T00:19:16.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.211+0000 7f1f55b28640 1 -- 172.21.15.116:0/356536236 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f501ec9a0 con 0x7f1f500a94a0 2024-04-25T00:19:16.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.215+0000 7f1f55b28640 1 -- 172.21.15.116:0/356536236 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f50145590 con 0x7f1f500a94a0 2024-04-25T00:19:16.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.215+0000 7f1f37fff640 1 -- 172.21.15.116:0/356536236 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1f4001a070 con 0x7f1f500a94a0 2024-04-25T00:19:16.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.215+0000 7f1f37fff640 1 --2- 172.21.15.116:0/356536236 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1f3c0678f0 0x7f1f3c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:16.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.215+0000 7f1f37fff640 1 -- 172.21.15.116:0/356536236 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 9074+0+0 (secure 0 0 0) 0x7f1f400867b0 con 0x7f1f500a94a0 2024-04-25T00:19:16.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.215+0000 7f1f4effd640 1 --2- 172.21.15.116:0/356536236 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1f3c0678f0 0x7f1f3c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:16.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.215+0000 7f1f4effd640 1 --2- 172.21.15.116:0/356536236 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1f3c0678f0 0x7f1f3c069db0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f1f500c2970 tx=0x7f1f44009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:16.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.219+0000 7f1f37fff640 1 -- 172.21.15.116:0/356536236 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f1f40055fb0 con 0x7f1f500a94a0 2024-04-25T00:19:16.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.463+0000 7f1f55b28640 1 -- 172.21.15.116:0/356536236 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1f50068350 con 0x7f1f500a94a0 2024-04-25T00:19:16.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.467+0000 7f1f37fff640 1 -- 172.21.15.116:0/356536236 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) ==== 95+0+5677 (secure 0 0 0) 0x7f1f4005a2b0 con 0x7f1f500a94a0 2024-04-25T00:19:16.464 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:19:16.465 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":151,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:18:23.899096+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7362},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7362":{"gid":7362,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6833/1655959433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1655959433},{"type":"v1","addr":"172.21.15.116:6833","nonce":1655959433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7362,"qdb_cluster":[7362]},"id":1},{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:19:08.844382+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7542},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7542":{"gid":7542,"name":"c","rank":0,"incarnation":148,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6835/879531976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":879531976},{"type":"v1","addr":"172.21.15.116:6835","nonce":879531976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7542,"qdb_cluster":[7542]},"id":2}]} 2024-04-25T00:19:16.465 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 151 2024-04-25T00:19:16.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.467+0000 7f1f55b28640 1 -- 172.21.15.116:0/356536236 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1f3c0678f0 msgr2=0x7f1f3c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:16.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.467+0000 7f1f55b28640 1 --2- 172.21.15.116:0/356536236 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1f3c0678f0 0x7f1f3c069db0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f1f500c2970 tx=0x7f1f44009190 comp rx=0 tx=0).stop 2024-04-25T00:19:16.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.467+0000 7f1f55b28640 1 -- 172.21.15.116:0/356536236 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f500a94a0 msgr2=0x7f1f500c6350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:16.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.467+0000 7f1f55b28640 1 --2- 172.21.15.116:0/356536236 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f500a94a0 0x7f1f500c6350 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f1f4000e870 tx=0x7f1f4000ec70 comp rx=0 tx=0).stop 2024-04-25T00:19:16.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.467+0000 7f1f55b28640 1 -- 172.21.15.116:0/356536236 shutdown_connections 2024-04-25T00:19:16.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.467+0000 7f1f55b28640 1 --2- 172.21.15.116:0/356536236 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1f3c0678f0 0x7f1f3c069db0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:16.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.467+0000 7f1f55b28640 1 --2- 172.21.15.116:0/356536236 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1f50150d20 0x7f1f50145950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:16.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.467+0000 7f1f55b28640 1 --2- 172.21.15.116:0/356536236 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1f500c2f50 0x7f1f500c6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:16.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.467+0000 7f1f55b28640 1 --2- 172.21.15.116:0/356536236 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1f500a94a0 0x7f1f500c6350 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:16.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.467+0000 7f1f55b28640 1 -- 172.21.15.116:0/356536236 >> 172.21.15.116:0/356536236 conn(0x7f1f500c3b00 msgr2=0x7f1f500a9f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:19:16.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.467+0000 7f1f55b28640 1 -- 172.21.15.116:0/356536236 shutdown_connections 2024-04-25T00:19:16.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:16.467+0000 7f1f55b28640 1 -- 172.21.15.116:0/356536236 wait complete. 2024-04-25T00:19:16.485 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2024-04-25T00:19:17.859 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:19:18.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.007+0000 7fdb38841640 1 Processor -- start 2024-04-25T00:19:18.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.007+0000 7fdb38841640 1 -- start start 2024-04-25T00:19:18.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.011+0000 7fdb38841640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdb3406bc20 0x7fdb341610b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:18.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.011+0000 7fdb38841640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb34073260 con 0x7fdb34152ce0 2024-04-25T00:19:18.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.011+0000 7fdb38841640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb34158e90 con 0x7fdb34159970 2024-04-25T00:19:18.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.011+0000 7fdb38841640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fdb341594e0 con 0x7fdb3406bc20 2024-04-25T00:19:18.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.011+0000 7fdb3259c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fdb34152ce0 0x7fdb34155720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:46566/0 (socket says 172.21.15.116:46566) 2024-04-25T00:19:18.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.011+0000 7fdb3259c640 1 -- 172.21.15.116:0/1712446603 learned_addr learned my addr 172.21.15.116:0/1712446603 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:19:18.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.011+0000 7fdb32d9d640 1 --2- 172.21.15.116:0/1712446603 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdb3406bc20 0x7fdb341610b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:18.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.011+0000 7fdb3159a640 1 -- 172.21.15.116:0/1712446603 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3933755883 0 0) 0x7fdb34073260 con 0x7fdb34152ce0 2024-04-25T00:19:18.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.011+0000 7fdb3159a640 1 -- 172.21.15.116:0/1712446603 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdb1c003530 con 0x7fdb34152ce0 2024-04-25T00:19:18.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.011+0000 7fdb3159a640 1 -- 172.21.15.116:0/1712446603 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 495966745 0 0) 0x7fdb34158e90 con 0x7fdb34159970 2024-04-25T00:19:18.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.011+0000 7fdb3159a640 1 -- 172.21.15.116:0/1712446603 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdb34073260 con 0x7fdb34159970 2024-04-25T00:19:18.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.011+0000 7fdb3159a640 1 -- 172.21.15.116:0/1712446603 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2529768378 0 0) 0x7fdb1c003530 con 0x7fdb34152ce0 2024-04-25T00:19:18.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.011+0000 7fdb3159a640 1 -- 172.21.15.116:0/1712446603 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdb3406bc20 msgr2=0x7fdb341610b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:19:18.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.011+0000 7fdb3159a640 1 --2- 172.21.15.116:0/1712446603 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdb3406bc20 0x7fdb341610b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:18.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.011+0000 7fdb3159a640 1 -- 172.21.15.116:0/1712446603 >> v1:172.21.15.174:6789/0 conn(0x7fdb34159970 legacy=0x7fdb340c3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:18.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.011+0000 7fdb3159a640 1 -- 172.21.15.116:0/1712446603 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb34159170 con 0x7fdb34152ce0 2024-04-25T00:19:18.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.011+0000 7fdb3159a640 1 -- 172.21.15.116:0/1712446603 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7fdb24001680 con 0x7fdb34152ce0 2024-04-25T00:19:18.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.011+0000 7fdb3159a640 1 -- 172.21.15.116:0/1712446603 >> v1:172.21.15.116:6789/0 conn(0x7fdb34152ce0 legacy=0x7fdb34155720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:18.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.011+0000 7fdb3159a640 1 --2- 172.21.15.116:0/1712446603 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdb1c003b80 0x7fdb1c003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:18.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.011+0000 7fdb3159a640 1 --2- 172.21.15.116:0/1712446603 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdb3406bc20 0x7fdb1c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:18.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.011+0000 7fdb3159a640 1 --2- 172.21.15.116:0/1712446603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb1c004d50 0x7fdb1c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:18.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.011+0000 7fdb3159a640 1 -- 172.21.15.116:0/1712446603 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdb341594e0 con 0x7fdb1c004d50 2024-04-25T00:19:18.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.011+0000 7fdb3159a640 1 -- 172.21.15.116:0/1712446603 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fdb1c00bad0 con 0x7fdb1c003b80 2024-04-25T00:19:18.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.011+0000 7fdb3159a640 1 -- 172.21.15.116:0/1712446603 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fdb1c00bcd0 con 0x7fdb3406bc20 2024-04-25T00:19:18.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.011+0000 7fdb32d9d640 1 --2- 172.21.15.116:0/1712446603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb1c004d50 0x7fdb1c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:18.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.011+0000 7fdb3259c640 1 --2- 172.21.15.116:0/1712446603 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdb1c003b80 0x7fdb1c003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:18.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.011+0000 7fdb31d9b640 1 --2- 172.21.15.116:0/1712446603 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdb3406bc20 0x7fdb1c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:18.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.011+0000 7fdb32d9d640 1 -- 172.21.15.116:0/1712446603 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdb3406bc20 msgr2=0x7fdb1c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:19:18.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.011+0000 7fdb32d9d640 1 --2- 172.21.15.116:0/1712446603 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdb3406bc20 0x7fdb1c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:18.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.011+0000 7fdb32d9d640 1 -- 172.21.15.116:0/1712446603 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdb1c003b80 msgr2=0x7fdb1c003f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:19:18.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.011+0000 7fdb32d9d640 1 --2- 172.21.15.116:0/1712446603 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdb1c003b80 0x7fdb1c003f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:18.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.011+0000 7fdb32d9d640 1 -- 172.21.15.116:0/1712446603 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb1c003530 con 0x7fdb1c004d50 2024-04-25T00:19:18.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.015+0000 7fdb32d9d640 1 --2- 172.21.15.116:0/1712446603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb1c004d50 0x7fdb1c0051d0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c009fc0 tx=0x7fdb2c01cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b76a2c36595d74e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:18.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.015+0000 7fdb3159a640 1 -- 172.21.15.116:0/1712446603 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb2c0b2070 con 0x7fdb1c004d50 2024-04-25T00:19:18.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.015+0000 7fdb3159a640 1 -- 172.21.15.116:0/1712446603 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb2c0a5c90 con 0x7fdb1c004d50 2024-04-25T00:19:18.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.015+0000 7fdb3159a640 1 -- 172.21.15.116:0/1712446603 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb2c009c10 con 0x7fdb1c004d50 2024-04-25T00:19:18.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.015+0000 7fdb38841640 1 -- 172.21.15.116:0/1712446603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb1c004d50 msgr2=0x7fdb1c0051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:18.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.015+0000 7fdb38841640 1 --2- 172.21.15.116:0/1712446603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb1c004d50 0x7fdb1c0051d0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c009fc0 tx=0x7fdb2c01cc80 comp rx=0 tx=0).stop 2024-04-25T00:19:18.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.015+0000 7fdb32d9d640 1 -- 172.21.15.116:0/1712446603 reap_dead start 2024-04-25T00:19:18.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.015+0000 7fdb38841640 1 -- 172.21.15.116:0/1712446603 shutdown_connections 2024-04-25T00:19:18.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.015+0000 7fdb38841640 1 -- 172.21.15.116:0/1712446603 >> 172.21.15.116:0/1712446603 conn(0x7fdb340c4c10 msgr2=0x7fdb340c5020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:19:18.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.015+0000 7fdb38841640 1 -- 172.21.15.116:0/1712446603 shutdown_connections 2024-04-25T00:19:18.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.015+0000 7fdb38841640 1 -- 172.21.15.116:0/1712446603 wait complete. 2024-04-25T00:19:18.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.015+0000 7fdb38841640 1 Processor -- start 2024-04-25T00:19:18.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.015+0000 7fdb38841640 1 -- start start 2024-04-25T00:19:18.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.019+0000 7fdb38841640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb1c004d50 0x7fdb3414b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:18.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.019+0000 7fdb38841640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdb340ac9f0 0x7fdb341545e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:18.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.019+0000 7fdb38841640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdb3414ba70 0x7fdb340c5de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:18.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.019+0000 7fdb38841640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdb340c59e0 con 0x7fdb1c004d50 2024-04-25T00:19:18.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.019+0000 7fdb38841640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fdb341530c0 con 0x7fdb3414ba70 2024-04-25T00:19:18.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.019+0000 7fdb38841640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fdb34072e80 con 0x7fdb340ac9f0 2024-04-25T00:19:18.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.019+0000 7fdb3259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb1c004d50 0x7fdb3414b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:18.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.019+0000 7fdb3259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb1c004d50 0x7fdb3414b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35766/0 (socket says 172.21.15.116:35766) 2024-04-25T00:19:18.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.019+0000 7fdb3259c640 1 -- 172.21.15.116:0/1942732108 learned_addr learned my addr 172.21.15.116:0/1942732108 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:19:18.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.019+0000 7fdb31d9b640 1 --2- 172.21.15.116:0/1942732108 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdb340ac9f0 0x7fdb341545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:18.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.019+0000 7fdb32d9d640 1 --2- 172.21.15.116:0/1942732108 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdb3414ba70 0x7fdb340c5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:18.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.019+0000 7fdb3259c640 1 -- 172.21.15.116:0/1942732108 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdb340ac9f0 msgr2=0x7fdb341545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:18.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.019+0000 7fdb3259c640 1 --2- 172.21.15.116:0/1942732108 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdb340ac9f0 0x7fdb341545e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:18.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.019+0000 7fdb3259c640 1 -- 172.21.15.116:0/1942732108 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdb3414ba70 msgr2=0x7fdb340c5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:18.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.019+0000 7fdb3259c640 1 --2- 172.21.15.116:0/1942732108 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdb3414ba70 0x7fdb340c5de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:18.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.019+0000 7fdb3259c640 1 -- 172.21.15.116:0/1942732108 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb341ebe60 con 0x7fdb1c004d50 2024-04-25T00:19:18.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.019+0000 7fdb3259c640 1 --2- 172.21.15.116:0/1942732108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb1c004d50 0x7fdb3414b530 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7fdb2400e780 tx=0x7fdb2400ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:18.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.019+0000 7fdb22ffd640 1 -- 172.21.15.116:0/1942732108 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb2400c870 con 0x7fdb1c004d50 2024-04-25T00:19:18.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.019+0000 7fdb22ffd640 1 -- 172.21.15.116:0/1942732108 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb24018070 con 0x7fdb1c004d50 2024-04-25T00:19:18.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.019+0000 7fdb38841640 1 -- 172.21.15.116:0/1942732108 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb341ec140 con 0x7fdb1c004d50 2024-04-25T00:19:18.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.019+0000 7fdb22ffd640 1 -- 172.21.15.116:0/1942732108 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb2400cc10 con 0x7fdb1c004d50 2024-04-25T00:19:18.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.019+0000 7fdb38841640 1 -- 172.21.15.116:0/1942732108 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb340c75f0 con 0x7fdb1c004d50 2024-04-25T00:19:18.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.019+0000 7fdb38841640 1 -- 172.21.15.116:0/1942732108 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb34068a60 con 0x7fdb1c004d50 2024-04-25T00:19:18.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.019+0000 7fdb22ffd640 1 -- 172.21.15.116:0/1942732108 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fdb2401d460 con 0x7fdb1c004d50 2024-04-25T00:19:18.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.019+0000 7fdb22ffd640 1 --2- 172.21.15.116:0/1942732108 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdb140678c0 0x7fdb14069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:18.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.019+0000 7fdb22ffd640 1 -- 172.21.15.116:0/1942732108 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 9074+0+0 (secure 0 0 0) 0x7fdb24086660 con 0x7fdb1c004d50 2024-04-25T00:19:18.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.019+0000 7fdb31d9b640 1 --2- 172.21.15.116:0/1942732108 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdb140678c0 0x7fdb14069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:18.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.023+0000 7fdb31d9b640 1 --2- 172.21.15.116:0/1942732108 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdb140678c0 0x7fdb14069d80 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fdb3406f210 tx=0x7fdb28007250 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:18.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.027+0000 7fdb22ffd640 1 -- 172.21.15.116:0/1942732108 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fdb24055e60 con 0x7fdb1c004d50 2024-04-25T00:19:18.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.235+0000 7fdb38841640 1 -- 172.21.15.116:0/1942732108 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdb34070000 con 0x7fdb1c004d50 2024-04-25T00:19:18.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.235+0000 7fdb22ffd640 1 -- 172.21.15.116:0/1942732108 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) ==== 95+0+6503 (secure 0 0 0) 0x7fdb2405a160 con 0x7fdb1c004d50 2024-04-25T00:19:18.233 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:19:18.233 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":152,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/991183935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":991183935},{"type":"v1","addr":"172.21.15.174:6837","nonce":991183935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":152}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:18:23.899096+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7362},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7362":{"gid":7362,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6833/1655959433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1655959433},{"type":"v1","addr":"172.21.15.116:6833","nonce":1655959433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7362,"qdb_cluster":[7362]},"id":1},{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:19:08.844382+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7542},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7542":{"gid":7542,"name":"c","rank":0,"incarnation":148,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6835/879531976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":879531976},{"type":"v1","addr":"172.21.15.116:6835","nonce":879531976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7542,"qdb_cluster":[7542]},"id":2}]} 2024-04-25T00:19:18.233 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 152 2024-04-25T00:19:18.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.235+0000 7fdb38841640 1 -- 172.21.15.116:0/1942732108 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdb140678c0 msgr2=0x7fdb14069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:18.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.235+0000 7fdb38841640 1 --2- 172.21.15.116:0/1942732108 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdb140678c0 0x7fdb14069d80 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fdb3406f210 tx=0x7fdb28007250 comp rx=0 tx=0).stop 2024-04-25T00:19:18.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.235+0000 7fdb38841640 1 -- 172.21.15.116:0/1942732108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb1c004d50 msgr2=0x7fdb3414b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:18.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.235+0000 7fdb38841640 1 --2- 172.21.15.116:0/1942732108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb1c004d50 0x7fdb3414b530 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7fdb2400e780 tx=0x7fdb2400ec40 comp rx=0 tx=0).stop 2024-04-25T00:19:18.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.235+0000 7fdb38841640 1 -- 172.21.15.116:0/1942732108 shutdown_connections 2024-04-25T00:19:18.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.235+0000 7fdb38841640 1 --2- 172.21.15.116:0/1942732108 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdb140678c0 0x7fdb14069d80 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:18.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.235+0000 7fdb38841640 1 --2- 172.21.15.116:0/1942732108 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdb3414ba70 0x7fdb340c5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:18.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.235+0000 7fdb38841640 1 --2- 172.21.15.116:0/1942732108 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdb340ac9f0 0x7fdb341545e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:18.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.235+0000 7fdb38841640 1 --2- 172.21.15.116:0/1942732108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdb1c004d50 0x7fdb3414b530 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:18.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.235+0000 7fdb38841640 1 -- 172.21.15.116:0/1942732108 >> 172.21.15.116:0/1942732108 conn(0x7fdb340c4c10 msgr2=0x7fdb340c1e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:19:18.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.235+0000 7fdb38841640 1 -- 172.21.15.116:0/1942732108 shutdown_connections 2024-04-25T00:19:18.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.235+0000 7fdb38841640 1 -- 172.21.15.116:0/1942732108 wait complete. 2024-04-25T00:19:18.252 INFO:tasks.mds_thrash.fs.[a]:kill mds.a (rank=0) 2024-04-25T00:19:18.252 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-25T00:19:18.253 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:19:18.263 INFO:tasks.ceph.mds.a:Stopped 2024-04-25T00:19:18.264 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:19:18.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.427+0000 7fb961ef2640 1 Processor -- start 2024-04-25T00:19:18.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.427+0000 7fb961ef2640 1 -- start start 2024-04-25T00:19:18.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.427+0000 7fb961ef2640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb95c0c4c20 0x7fb95c154170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:18.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.427+0000 7fb961ef2640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fb95c070770 0x7fb95c0ab200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:18.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.427+0000 7fb961ef2640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb95c070180 con 0x7fb95c150d40 2024-04-25T00:19:18.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.427+0000 7fb961ef2640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fb95c0c7530 con 0x7fb95c0c4c20 2024-04-25T00:19:18.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.427+0000 7fb961ef2640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fb95c071280 con 0x7fb95c070770 2024-04-25T00:19:18.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.427+0000 7fb953fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb95c0c4c20 0x7fb95c154170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:18.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.427+0000 7fb953fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb95c0c4c20 0x7fb95c154170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35780/0 (socket says 172.21.15.116:35780) 2024-04-25T00:19:18.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.427+0000 7fb953fff640 1 -- 172.21.15.116:0/2753946033 learned_addr learned my addr 172.21.15.116:0/2753946033 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:19:18.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.427+0000 7fb95bfff640 1 --2- 172.21.15.116:0/2753946033 >> v2:172.21.15.174:3300/0 conn(0x7fb95c070770 0x7fb95c0ab200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:18.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.427+0000 7fb953fff640 1 -- 172.21.15.116:0/2753946033 >> v2:172.21.15.174:3300/0 conn(0x7fb95c070770 msgr2=0x7fb95c0ab200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:19:18.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.427+0000 7fb953fff640 1 --2- 172.21.15.116:0/2753946033 >> v2:172.21.15.174:3300/0 conn(0x7fb95c070770 0x7fb95c0ab200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:18.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.427+0000 7fb953fff640 1 -- 172.21.15.116:0/2753946033 >> v1:172.21.15.116:6789/0 conn(0x7fb95c150d40 legacy=0x7fb95c0c6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:18.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.427+0000 7fb953fff640 1 -- 172.21.15.116:0/2753946033 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb95c070460 con 0x7fb95c0c4c20 2024-04-25T00:19:18.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.431+0000 7fb953fff640 1 --2- 172.21.15.116:0/2753946033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb95c0c4c20 0x7fb95c154170 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7fb95c0c92d0 tx=0x7fb94401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd3269177c5e857 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:18.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.431+0000 7fb95affd640 1 -- 172.21.15.116:0/2753946033 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9440b1070 con 0x7fb95c0c4c20 2024-04-25T00:19:18.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.431+0000 7fb95affd640 1 -- 172.21.15.116:0/2753946033 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9440ac600 con 0x7fb95c0c4c20 2024-04-25T00:19:18.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.431+0000 7fb95affd640 1 -- 172.21.15.116:0/2753946033 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9440ac9d0 con 0x7fb95c0c4c20 2024-04-25T00:19:18.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.431+0000 7fb961ef2640 1 -- 172.21.15.116:0/2753946033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb95c0c4c20 msgr2=0x7fb95c154170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:18.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.431+0000 7fb961ef2640 1 --2- 172.21.15.116:0/2753946033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb95c0c4c20 0x7fb95c154170 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7fb95c0c92d0 tx=0x7fb94401c490 comp rx=0 tx=0).stop 2024-04-25T00:19:18.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.431+0000 7fb961ef2640 1 -- 172.21.15.116:0/2753946033 shutdown_connections 2024-04-25T00:19:18.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.431+0000 7fb961ef2640 1 --2- 172.21.15.116:0/2753946033 >> v2:172.21.15.174:3300/0 conn(0x7fb95c070770 0x7fb95c0ab200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:18.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.431+0000 7fb961ef2640 1 --2- 172.21.15.116:0/2753946033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb95c0c4c20 0x7fb95c154170 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:18.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.431+0000 7fb961ef2640 1 -- 172.21.15.116:0/2753946033 >> 172.21.15.116:0/2753946033 conn(0x7fb95c06f0c0 msgr2=0x7fb95c06f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:19:18.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.431+0000 7fb961ef2640 1 -- 172.21.15.116:0/2753946033 shutdown_connections 2024-04-25T00:19:18.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.431+0000 7fb961ef2640 1 -- 172.21.15.116:0/2753946033 wait complete. 2024-04-25T00:19:18.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.431+0000 7fb961ef2640 1 Processor -- start 2024-04-25T00:19:18.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.431+0000 7fb961ef2640 1 -- start start 2024-04-25T00:19:18.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.435+0000 7fb961ef2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb95c070770 0x7fb95c0add90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:18.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.435+0000 7fb961ef2640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb95c0c4c20 0x7fb95c145ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:18.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.435+0000 7fb961ef2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb95c150d40 0x7fb95c146400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:18.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.435+0000 7fb961ef2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb95c151120 con 0x7fb95c150d40 2024-04-25T00:19:18.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.435+0000 7fb961ef2640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb95c06ff30 con 0x7fb95c070770 2024-04-25T00:19:18.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.435+0000 7fb961ef2640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fb95c1512a0 con 0x7fb95c0c4c20 2024-04-25T00:19:18.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.435+0000 7fb95bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb95c150d40 0x7fb95c146400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:18.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.435+0000 7fb95bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb95c150d40 0x7fb95c146400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35790/0 (socket says 172.21.15.116:35790) 2024-04-25T00:19:18.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.435+0000 7fb95bfff640 1 -- 172.21.15.116:0/132044399 learned_addr learned my addr 172.21.15.116:0/132044399 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:19:18.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.435+0000 7fb95b7fe640 1 --2- 172.21.15.116:0/132044399 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb95c070770 0x7fb95c0add90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:18.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.435+0000 7fb953fff640 1 --2- 172.21.15.116:0/132044399 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb95c0c4c20 0x7fb95c145ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:18.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.435+0000 7fb95bfff640 1 -- 172.21.15.116:0/132044399 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb95c0c4c20 msgr2=0x7fb95c145ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:18.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.435+0000 7fb95bfff640 1 --2- 172.21.15.116:0/132044399 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb95c0c4c20 0x7fb95c145ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:18.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.435+0000 7fb95bfff640 1 -- 172.21.15.116:0/132044399 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb95c070770 msgr2=0x7fb95c0add90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:18.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.435+0000 7fb95bfff640 1 --2- 172.21.15.116:0/132044399 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb95c070770 0x7fb95c0add90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:18.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.435+0000 7fb95bfff640 1 -- 172.21.15.116:0/132044399 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb95c14fe10 con 0x7fb95c150d40 2024-04-25T00:19:18.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.435+0000 7fb95bfff640 1 --2- 172.21.15.116:0/132044399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb95c150d40 0x7fb95c146400 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7fb954007cf0 tx=0x7fb95400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:18.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.435+0000 7fb958ff9640 1 -- 172.21.15.116:0/132044399 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb954014420 con 0x7fb95c150d40 2024-04-25T00:19:18.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.435+0000 7fb958ff9640 1 -- 172.21.15.116:0/132044399 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb954019070 con 0x7fb95c150d40 2024-04-25T00:19:18.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.435+0000 7fb961ef2640 1 -- 172.21.15.116:0/132044399 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb95c14e140 con 0x7fb95c150d40 2024-04-25T00:19:18.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.435+0000 7fb958ff9640 1 -- 172.21.15.116:0/132044399 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb954014820 con 0x7fb95c150d40 2024-04-25T00:19:18.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.435+0000 7fb961ef2640 1 -- 172.21.15.116:0/132044399 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb95c14e700 con 0x7fb95c150d40 2024-04-25T00:19:18.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.435+0000 7fb961ef2640 1 -- 172.21.15.116:0/132044399 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb95c068a60 con 0x7fb95c150d40 2024-04-25T00:19:18.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.435+0000 7fb958ff9640 1 -- 172.21.15.116:0/132044399 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb9540149c0 con 0x7fb95c150d40 2024-04-25T00:19:18.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.435+0000 7fb958ff9640 1 --2- 172.21.15.116:0/132044399 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb93c0678f0 0x7fb93c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:18.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.435+0000 7fb958ff9640 1 -- 172.21.15.116:0/132044399 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 9074+0+0 (secure 0 0 0) 0x7fb954087050 con 0x7fb95c150d40 2024-04-25T00:19:18.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.435+0000 7fb95b7fe640 1 --2- 172.21.15.116:0/132044399 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb93c0678f0 0x7fb93c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:18.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.439+0000 7fb95b7fe640 1 --2- 172.21.15.116:0/132044399 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb93c0678f0 0x7fb93c069db0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fb94c001810 tx=0x7fb94c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:18.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.443+0000 7fb958ff9640 1 -- 172.21.15.116:0/132044399 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fb9540567c0 con 0x7fb95c150d40 2024-04-25T00:19:18.486 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:19:18.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.607+0000 7f3d711db640 1 Processor -- start 2024-04-25T00:19:18.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.607+0000 7f3d711db640 1 -- start start 2024-04-25T00:19:18.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.611+0000 7f3d711db640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3d6c152d80 0x7f3d6c06bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:18.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.611+0000 7f3d711db640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d6c0c9150 con 0x7f3d6c06c290 2024-04-25T00:19:18.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.611+0000 7f3d711db640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d6c0c9430 con 0x7f3d6c06d5f0 2024-04-25T00:19:18.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.611+0000 7f3d711db640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f3d6c0c9d60 con 0x7f3d6c152d80 2024-04-25T00:19:18.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.611+0000 7f3d6ad9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3d6c152d80 0x7f3d6c06bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:18.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.611+0000 7f3d6b59e640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f3d6c06c290 0x7f3d6c157390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:46580/0 (socket says 172.21.15.116:46580) 2024-04-25T00:19:18.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.611+0000 7f3d6ad9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3d6c152d80 0x7f3d6c06bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35806/0 (socket says 172.21.15.116:35806) 2024-04-25T00:19:18.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.611+0000 7f3d6b59e640 1 -- 172.21.15.116:0/3713845124 learned_addr learned my addr 172.21.15.116:0/3713845124 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:19:18.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.611+0000 7f3d6ad9d640 1 -- 172.21.15.116:0/3713845124 >> v1:172.21.15.174:6789/0 conn(0x7f3d6c06d5f0 legacy=0x7f3d6c156c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:18.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.611+0000 7f3d6ad9d640 1 -- 172.21.15.116:0/3713845124 >> v1:172.21.15.116:6789/0 conn(0x7f3d6c06c290 legacy=0x7f3d6c157390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:18.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.611+0000 7f3d6a59c640 1 -- 172.21.15.116:0/3713845124 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1478642927 0 0) 0x7f3d6c0c9150 con 0x7f3d6c06c290 2024-04-25T00:19:18.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.611+0000 7f3d6ad9d640 1 -- 172.21.15.116:0/3713845124 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d6c0c9710 con 0x7f3d6c152d80 2024-04-25T00:19:18.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.611+0000 7f3d6ad9d640 1 --2- 172.21.15.116:0/3713845124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d6c152d80 0x7f3d6c06bd50 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f3d5c0097f0 tx=0x7f3d5c01c350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54f7813a36ada82a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:18.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.611+0000 7f3d6a59c640 1 -- 172.21.15.116:0/3713845124 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d5c0b0070 con 0x7f3d6c152d80 2024-04-25T00:19:18.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.611+0000 7f3d6a59c640 1 -- 172.21.15.116:0/3713845124 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d5c0ab600 con 0x7f3d6c152d80 2024-04-25T00:19:18.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.611+0000 7f3d6a59c640 1 -- 172.21.15.116:0/3713845124 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d5c0ab9d0 con 0x7f3d6c152d80 2024-04-25T00:19:18.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.615+0000 7f3d711db640 1 -- 172.21.15.116:0/3713845124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d6c152d80 msgr2=0x7f3d6c06bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:18.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.615+0000 7f3d711db640 1 --2- 172.21.15.116:0/3713845124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d6c152d80 0x7f3d6c06bd50 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f3d5c0097f0 tx=0x7f3d5c01c350 comp rx=0 tx=0).stop 2024-04-25T00:19:18.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.615+0000 7f3d711db640 1 -- 172.21.15.116:0/3713845124 shutdown_connections 2024-04-25T00:19:18.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.615+0000 7f3d711db640 1 --2- 172.21.15.116:0/3713845124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d6c152d80 0x7f3d6c06bd50 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:18.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.615+0000 7f3d711db640 1 -- 172.21.15.116:0/3713845124 >> 172.21.15.116:0/3713845124 conn(0x7f3d6c0acdd0 msgr2=0x7f3d6c0ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:19:18.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.615+0000 7f3d711db640 1 -- 172.21.15.116:0/3713845124 shutdown_connections 2024-04-25T00:19:18.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.615+0000 7f3d711db640 1 -- 172.21.15.116:0/3713845124 wait complete. 2024-04-25T00:19:18.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.615+0000 7f3d711db640 1 Processor -- start 2024-04-25T00:19:18.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.615+0000 7f3d711db640 1 -- start start 2024-04-25T00:19:18.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.615+0000 7f3d711db640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3d6c06c290 0x7f3d6c0aaa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:18.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.615+0000 7f3d711db640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d6c06d5f0 0x7f3d6c151e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:18.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.615+0000 7f3d711db640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3d6c152d80 0x7f3d6c154740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:18.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.615+0000 7f3d711db640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3d6c163a60 con 0x7f3d6c06d5f0 2024-04-25T00:19:18.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.615+0000 7f3d711db640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3d6c1638e0 con 0x7f3d6c06c290 2024-04-25T00:19:18.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.615+0000 7f3d711db640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f3d6c163be0 con 0x7f3d6c152d80 2024-04-25T00:19:18.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.615+0000 7f3d63fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d6c06d5f0 0x7f3d6c151e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:18.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.615+0000 7f3d63fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d6c06d5f0 0x7f3d6c151e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35822/0 (socket says 172.21.15.116:35822) 2024-04-25T00:19:18.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.615+0000 7f3d63fff640 1 -- 172.21.15.116:0/3082088425 learned_addr learned my addr 172.21.15.116:0/3082088425 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:19:18.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.615+0000 7f3d6ad9d640 1 --2- 172.21.15.116:0/3082088425 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3d6c06c290 0x7f3d6c0aaa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:18.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.615+0000 7f3d6b59e640 1 --2- 172.21.15.116:0/3082088425 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3d6c152d80 0x7f3d6c154740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:18.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.615+0000 7f3d63fff640 1 -- 172.21.15.116:0/3082088425 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3d6c152d80 msgr2=0x7f3d6c154740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:18.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.615+0000 7f3d63fff640 1 --2- 172.21.15.116:0/3082088425 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3d6c152d80 0x7f3d6c154740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:18.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.615+0000 7f3d63fff640 1 -- 172.21.15.116:0/3082088425 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3d6c06c290 msgr2=0x7f3d6c0aaa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:18.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.615+0000 7f3d63fff640 1 --2- 172.21.15.116:0/3082088425 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3d6c06c290 0x7f3d6c0aaa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:18.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.615+0000 7f3d63fff640 1 -- 172.21.15.116:0/3082088425 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d6c153d90 con 0x7f3d6c06d5f0 2024-04-25T00:19:18.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.615+0000 7f3d63fff640 1 --2- 172.21.15.116:0/3082088425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d6c06d5f0 0x7f3d6c151e50 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f3d5400c3c0 tx=0x7f3d5400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:18.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.615+0000 7f3d637fe640 1 -- 172.21.15.116:0/3082088425 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d54017070 con 0x7f3d6c06d5f0 2024-04-25T00:19:18.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.619+0000 7f3d637fe640 1 -- 172.21.15.116:0/3082088425 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d540052d0 con 0x7f3d6c06d5f0 2024-04-25T00:19:18.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.619+0000 7f3d637fe640 1 -- 172.21.15.116:0/3082088425 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d54003840 con 0x7f3d6c06d5f0 2024-04-25T00:19:18.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.619+0000 7f3d711db640 1 -- 172.21.15.116:0/3082088425 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d6c1432f0 con 0x7f3d6c06d5f0 2024-04-25T00:19:18.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.619+0000 7f3d711db640 1 -- 172.21.15.116:0/3082088425 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d6c151650 con 0x7f3d6c06d5f0 2024-04-25T00:19:18.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.619+0000 7f3d711db640 1 -- 172.21.15.116:0/3082088425 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d6c068350 con 0x7f3d6c06d5f0 2024-04-25T00:19:18.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.619+0000 7f3d637fe640 1 -- 172.21.15.116:0/3082088425 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3d5401a070 con 0x7f3d6c06d5f0 2024-04-25T00:19:18.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.619+0000 7f3d637fe640 1 --2- 172.21.15.116:0/3082088425 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3d4c067910 0x7f3d4c069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:18.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.619+0000 7f3d637fe640 1 -- 172.21.15.116:0/3082088425 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 9074+0+0 (secure 0 0 0) 0x7f3d54085ba0 con 0x7f3d6c06d5f0 2024-04-25T00:19:18.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.619+0000 7f3d6ad9d640 1 --2- 172.21.15.116:0/3082088425 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3d4c067910 0x7f3d4c069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:18.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.619+0000 7f3d6ad9d640 1 --2- 172.21.15.116:0/3082088425 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3d4c067910 0x7f3d4c069dd0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f3d5c0ad000 tx=0x7f3d5c0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:18.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.623+0000 7f3d637fe640 1 -- 172.21.15.116:0/3082088425 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f3d54005480 con 0x7f3d6c06d5f0 2024-04-25T00:19:18.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.703+0000 7fb961ef2640 1 -- 172.21.15.116:0/132044399 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb95c14d990 con 0x7fb95c150d40 2024-04-25T00:19:18.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.703+0000 7fb958ff9640 1 -- 172.21.15.116:0/132044399 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) ==== 95+0+6503 (secure 0 0 0) 0x7fb95405aac0 con 0x7fb95c150d40 2024-04-25T00:19:18.703 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:19:18.703 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":152,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/991183935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":991183935},{"type":"v1","addr":"172.21.15.174:6837","nonce":991183935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":152}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:18:23.899096+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7362},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7362":{"gid":7362,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6833/1655959433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1655959433},{"type":"v1","addr":"172.21.15.116:6833","nonce":1655959433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7362,"qdb_cluster":[7362]},"id":1},{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:19:08.844382+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7542},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7542":{"gid":7542,"name":"c","rank":0,"incarnation":148,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6835/879531976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":879531976},{"type":"v1","addr":"172.21.15.116:6835","nonce":879531976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7542,"qdb_cluster":[7542]},"id":2}]} 2024-04-25T00:19:18.703 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 152 2024-04-25T00:19:18.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.707+0000 7fb961ef2640 1 -- 172.21.15.116:0/132044399 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb93c0678f0 msgr2=0x7fb93c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:18.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.707+0000 7fb961ef2640 1 --2- 172.21.15.116:0/132044399 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb93c0678f0 0x7fb93c069db0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fb94c001810 tx=0x7fb94c009040 comp rx=0 tx=0).stop 2024-04-25T00:19:18.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.707+0000 7fb961ef2640 1 -- 172.21.15.116:0/132044399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb95c150d40 msgr2=0x7fb95c146400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:18.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.707+0000 7fb961ef2640 1 --2- 172.21.15.116:0/132044399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb95c150d40 0x7fb95c146400 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7fb954007cf0 tx=0x7fb95400e4b0 comp rx=0 tx=0).stop 2024-04-25T00:19:18.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.707+0000 7fb961ef2640 1 -- 172.21.15.116:0/132044399 shutdown_connections 2024-04-25T00:19:18.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.707+0000 7fb961ef2640 1 --2- 172.21.15.116:0/132044399 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb93c0678f0 0x7fb93c069db0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:18.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.707+0000 7fb961ef2640 1 --2- 172.21.15.116:0/132044399 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb95c150d40 0x7fb95c146400 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:18.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.707+0000 7fb961ef2640 1 --2- 172.21.15.116:0/132044399 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb95c0c4c20 0x7fb95c145ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:18.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.707+0000 7fb961ef2640 1 --2- 172.21.15.116:0/132044399 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb95c070770 0x7fb95c0add90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:18.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.707+0000 7fb961ef2640 1 -- 172.21.15.116:0/132044399 >> 172.21.15.116:0/132044399 conn(0x7fb95c06f0c0 msgr2=0x7fb95c06f4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:19:18.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.707+0000 7fb961ef2640 1 -- 172.21.15.116:0/132044399 shutdown_connections 2024-04-25T00:19:18.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.707+0000 7fb961ef2640 1 -- 172.21.15.116:0/132044399 wait complete. 2024-04-25T00:19:18.723 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:19:18.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.907+0000 7f3d711db640 1 -- 172.21.15.116:0/3082088425 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3d6c0cb270 con 0x7f3d6c06d5f0 2024-04-25T00:19:18.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.907+0000 7f3d637fe640 1 -- 172.21.15.116:0/3082088425 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) ==== 95+0+6503 (secure 0 0 0) 0x7f3d54055420 con 0x7f3d6c06d5f0 2024-04-25T00:19:18.907 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:19:18.907 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":152,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/991183935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":991183935},{"type":"v1","addr":"172.21.15.174:6837","nonce":991183935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":152}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:18:23.899096+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7362},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7362":{"gid":7362,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6833/1655959433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1655959433},{"type":"v1","addr":"172.21.15.116:6833","nonce":1655959433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7362,"qdb_cluster":[7362]},"id":1},{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:19:08.844382+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7542},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7542":{"gid":7542,"name":"c","rank":0,"incarnation":148,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6835/879531976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":879531976},{"type":"v1","addr":"172.21.15.116:6835","nonce":879531976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7542,"qdb_cluster":[7542]},"id":2}]} 2024-04-25T00:19:18.908 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 152 2024-04-25T00:19:18.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.911+0000 7f3d711db640 1 -- 172.21.15.116:0/3082088425 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3d4c067910 msgr2=0x7f3d4c069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:18.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.911+0000 7f3d711db640 1 --2- 172.21.15.116:0/3082088425 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3d4c067910 0x7f3d4c069dd0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f3d5c0ad000 tx=0x7f3d5c0a6000 comp rx=0 tx=0).stop 2024-04-25T00:19:18.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.911+0000 7f3d711db640 1 -- 172.21.15.116:0/3082088425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d6c06d5f0 msgr2=0x7f3d6c151e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:18.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.911+0000 7f3d711db640 1 --2- 172.21.15.116:0/3082088425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d6c06d5f0 0x7f3d6c151e50 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f3d5400c3c0 tx=0x7f3d5400c880 comp rx=0 tx=0).stop 2024-04-25T00:19:18.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.911+0000 7f3d711db640 1 -- 172.21.15.116:0/3082088425 shutdown_connections 2024-04-25T00:19:18.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.911+0000 7f3d711db640 1 --2- 172.21.15.116:0/3082088425 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3d4c067910 0x7f3d4c069dd0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:18.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.911+0000 7f3d711db640 1 --2- 172.21.15.116:0/3082088425 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3d6c152d80 0x7f3d6c154740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:18.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.911+0000 7f3d711db640 1 --2- 172.21.15.116:0/3082088425 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3d6c06d5f0 0x7f3d6c151e50 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:18.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.911+0000 7f3d711db640 1 --2- 172.21.15.116:0/3082088425 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3d6c06c290 0x7f3d6c0aaa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:18.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.911+0000 7f3d711db640 1 -- 172.21.15.116:0/3082088425 >> 172.21.15.116:0/3082088425 conn(0x7f3d6c0acdd0 msgr2=0x7f3d6c14da10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:19:18.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.911+0000 7f3d711db640 1 -- 172.21.15.116:0/3082088425 shutdown_connections 2024-04-25T00:19:18.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:18.911+0000 7f3d711db640 1 -- 172.21.15.116:0/3082088425 wait complete. 2024-04-25T00:19:18.924 INFO:tasks.mds_thrash.fs.[b]:mds.d reported in up:standby state 2024-04-25T00:19:18.924 INFO:tasks.mds_thrash.fs.[b]:waiting for 35 secs before thrashing 2024-04-25T00:19:20.724 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:19:20.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:20.855+0000 7f4712782640 1 Processor -- start 2024-04-25T00:19:20.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:20.855+0000 7f4712782640 1 -- start start 2024-04-25T00:19:20.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:20.859+0000 7f4712782640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f470c153f40 0x7f470c06e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:20.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:20.859+0000 7f4712782640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f470c0c2e70 0x7f470c15a610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:20.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:20.859+0000 7f4712782640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f470c071a30 con 0x7f470c0c2a10 2024-04-25T00:19:20.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:20.859+0000 7f4712782640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f470c0725d0 con 0x7f470c0c2e70 2024-04-25T00:19:20.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:20.859+0000 7f4712782640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f470c071ca0 con 0x7f470c153f40 2024-04-25T00:19:20.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:20.859+0000 7f4710d1f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f470c0c2e70 0x7f470c15a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:20.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:20.859+0000 7f4710d1f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f470c0c2e70 0x7f470c15a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35832/0 (socket says 172.21.15.116:35832) 2024-04-25T00:19:20.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:20.859+0000 7f4710d1f640 1 -- 172.21.15.116:0/483498888 learned_addr learned my addr 172.21.15.116:0/483498888 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:19:20.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:20.859+0000 7f470bfff640 1 --2- 172.21.15.116:0/483498888 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f470c153f40 0x7f470c06e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:20.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:20.859+0000 7f4710d1f640 1 -- 172.21.15.116:0/483498888 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f470c153f40 msgr2=0x7f470c06e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:19:20.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:20.859+0000 7f4710d1f640 1 --2- 172.21.15.116:0/483498888 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f470c153f40 0x7f470c06e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:20.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:20.859+0000 7f4710d1f640 1 -- 172.21.15.116:0/483498888 >> v1:172.21.15.174:6789/0 conn(0x7f470c0c2a10 legacy=0x7f470c159f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:19:20.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:20.859+0000 7f4710d1f640 1 -- 172.21.15.116:0/483498888 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f470c071f80 con 0x7f470c0c2e70 2024-04-25T00:19:20.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:20.859+0000 7f4710d1f640 1 --2- 172.21.15.116:0/483498888 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f470c0c2e70 0x7f470c15a610 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f4704009e50 tx=0x7f470401c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e401d5a7d8e034ee server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:20.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:20.863+0000 7f470affd640 1 -- 172.21.15.116:0/483498888 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47040a56f0 con 0x7f470c0c2e70 2024-04-25T00:19:20.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:20.863+0000 7f470affd640 1 -- 172.21.15.116:0/483498888 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47040a5890 con 0x7f470c0c2e70 2024-04-25T00:19:20.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:20.863+0000 7f470affd640 1 -- 172.21.15.116:0/483498888 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47040a5c60 con 0x7f470c0c2e70 2024-04-25T00:19:20.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:20.863+0000 7f4712782640 1 -- 172.21.15.116:0/483498888 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f470c0c2e70 msgr2=0x7f470c15a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:20.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:20.863+0000 7f4712782640 1 --2- 172.21.15.116:0/483498888 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f470c0c2e70 0x7f470c15a610 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f4704009e50 tx=0x7f470401c960 comp rx=0 tx=0).stop 2024-04-25T00:19:20.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:20.863+0000 7f4712782640 1 -- 172.21.15.116:0/483498888 shutdown_connections 2024-04-25T00:19:20.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:20.863+0000 7f4712782640 1 --2- 172.21.15.116:0/483498888 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f470c0c2e70 0x7f470c15a610 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:20.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:20.863+0000 7f4712782640 1 --2- 172.21.15.116:0/483498888 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f470c153f40 0x7f470c06e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:20.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:20.863+0000 7f4712782640 1 -- 172.21.15.116:0/483498888 >> 172.21.15.116:0/483498888 conn(0x7f470c0c3b00 msgr2=0x7f470c0c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:19:20.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:20.863+0000 7f4712782640 1 -- 172.21.15.116:0/483498888 shutdown_connections 2024-04-25T00:19:20.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:20.863+0000 7f4712782640 1 -- 172.21.15.116:0/483498888 wait complete. 2024-04-25T00:19:20.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:20.863+0000 7f4712782640 1 Processor -- start 2024-04-25T00:19:20.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:20.863+0000 7f4712782640 1 -- start start 2024-04-25T00:19:20.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:20.863+0000 7f4712782640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f470c0c2a10 0x7f470c1f37a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:20.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:20.863+0000 7f4712782640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f470c0c2e70 0x7f470c15d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:20.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:20.863+0000 7f4712782640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f470c153f40 0x7f470c15b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:20.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:20.863+0000 7f4712782640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f470c0c36c0 con 0x7f470c0c2e70 2024-04-25T00:19:20.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:20.863+0000 7f4712782640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f470c06eee0 con 0x7f470c0c2a10 2024-04-25T00:19:20.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:20.863+0000 7f4712782640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f470c0c3840 con 0x7f470c153f40 2024-04-25T00:19:20.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:20.863+0000 7f470b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f470c0c2e70 0x7f470c15d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:20.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:20.863+0000 7f470b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f470c0c2e70 0x7f470c15d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35844/0 (socket says 172.21.15.116:35844) 2024-04-25T00:19:20.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:20.863+0000 7f470b7fe640 1 -- 172.21.15.116:0/3825450749 learned_addr learned my addr 172.21.15.116:0/3825450749 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:19:20.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:20.863+0000 7f4710d1f640 1 --2- 172.21.15.116:0/3825450749 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f470c153f40 0x7f470c15b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:20.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:20.863+0000 7f470bfff640 1 --2- 172.21.15.116:0/3825450749 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f470c0c2a10 0x7f470c1f37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:20.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:20.863+0000 7f470b7fe640 1 -- 172.21.15.116:0/3825450749 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f470c153f40 msgr2=0x7f470c15b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:20.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:20.863+0000 7f470b7fe640 1 --2- 172.21.15.116:0/3825450749 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f470c153f40 0x7f470c15b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:20.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:20.863+0000 7f470b7fe640 1 -- 172.21.15.116:0/3825450749 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f470c0c2a10 msgr2=0x7f470c1f37a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:20.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:20.863+0000 7f470b7fe640 1 --2- 172.21.15.116:0/3825450749 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f470c0c2a10 0x7f470c1f37a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:20.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:20.863+0000 7f470b7fe640 1 -- 172.21.15.116:0/3825450749 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f470c1591e0 con 0x7f470c0c2e70 2024-04-25T00:19:20.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:20.863+0000 7f470b7fe640 1 --2- 172.21.15.116:0/3825450749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f470c0c2e70 0x7f470c15d680 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f470000b410 tx=0x7f470000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:20.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:20.863+0000 7f4708ff9640 1 -- 172.21.15.116:0/3825450749 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4700015030 con 0x7f470c0c2e70 2024-04-25T00:19:20.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:20.863+0000 7f4708ff9640 1 -- 172.21.15.116:0/3825450749 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4700004020 con 0x7f470c0c2e70 2024-04-25T00:19:20.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:20.863+0000 7f4712782640 1 -- 172.21.15.116:0/3825450749 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f470c14aeb0 con 0x7f470c0c2e70 2024-04-25T00:19:20.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:20.863+0000 7f4708ff9640 1 -- 172.21.15.116:0/3825450749 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4700004450 con 0x7f470c0c2e70 2024-04-25T00:19:20.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:20.863+0000 7f4712782640 1 -- 172.21.15.116:0/3825450749 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f470c14b470 con 0x7f470c0c2e70 2024-04-25T00:19:20.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:20.867+0000 7f4712782640 1 -- 172.21.15.116:0/3825450749 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f470c068350 con 0x7f470c0c2e70 2024-04-25T00:19:20.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:20.867+0000 7f4708ff9640 1 -- 172.21.15.116:0/3825450749 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4700018070 con 0x7f470c0c2e70 2024-04-25T00:19:20.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:20.867+0000 7f4708ff9640 1 --2- 172.21.15.116:0/3825450749 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f46f80678c0 0x7f46f8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:20.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:20.867+0000 7f4708ff9640 1 -- 172.21.15.116:0/3825450749 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 9074+0+0 (secure 0 0 0) 0x7f4700086400 con 0x7f470c0c2e70 2024-04-25T00:19:20.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:20.867+0000 7f470bfff640 1 --2- 172.21.15.116:0/3825450749 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f46f80678c0 0x7f46f8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:20.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:20.867+0000 7f470bfff640 1 --2- 172.21.15.116:0/3825450749 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f46f80678c0 0x7f46f8069d80 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f46fc00b9d0 tx=0x7f46fc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:20.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:20.871+0000 7f4708ff9640 1 -- 172.21.15.116:0/3825450749 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f4700056250 con 0x7f470c0c2e70 2024-04-25T00:19:21.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:21.063+0000 7f4712782640 1 -- 172.21.15.116:0/3825450749 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f470c14e820 con 0x7f470c0c2e70 2024-04-25T00:19:21.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:21.063+0000 7f4708ff9640 1 -- 172.21.15.116:0/3825450749 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) ==== 95+0+6503 (secure 0 0 0) 0x7f4700055b50 con 0x7f470c0c2e70 2024-04-25T00:19:21.063 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:19:21.063 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":152,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/991183935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":991183935},{"type":"v1","addr":"172.21.15.174:6837","nonce":991183935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":152}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:18:23.899096+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7362},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7362":{"gid":7362,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6833/1655959433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1655959433},{"type":"v1","addr":"172.21.15.116:6833","nonce":1655959433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7362,"qdb_cluster":[7362]},"id":1},{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:19:08.844382+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7542},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7542":{"gid":7542,"name":"c","rank":0,"incarnation":148,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6835/879531976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":879531976},{"type":"v1","addr":"172.21.15.116:6835","nonce":879531976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7542,"qdb_cluster":[7542]},"id":2}]} 2024-04-25T00:19:21.064 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 152 2024-04-25T00:19:21.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:21.067+0000 7f4712782640 1 -- 172.21.15.116:0/3825450749 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f46f80678c0 msgr2=0x7f46f8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:21.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:21.067+0000 7f4712782640 1 --2- 172.21.15.116:0/3825450749 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f46f80678c0 0x7f46f8069d80 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f46fc00b9d0 tx=0x7f46fc008040 comp rx=0 tx=0).stop 2024-04-25T00:19:21.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:21.067+0000 7f4712782640 1 -- 172.21.15.116:0/3825450749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f470c0c2e70 msgr2=0x7f470c15d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:21.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:21.067+0000 7f4712782640 1 --2- 172.21.15.116:0/3825450749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f470c0c2e70 0x7f470c15d680 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f470000b410 tx=0x7f470000b8d0 comp rx=0 tx=0).stop 2024-04-25T00:19:21.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:21.067+0000 7f4712782640 1 -- 172.21.15.116:0/3825450749 shutdown_connections 2024-04-25T00:19:21.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:21.067+0000 7f4712782640 1 --2- 172.21.15.116:0/3825450749 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f46f80678c0 0x7f46f8069d80 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:21.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:21.067+0000 7f4712782640 1 --2- 172.21.15.116:0/3825450749 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f470c153f40 0x7f470c15b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:21.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:21.067+0000 7f4712782640 1 --2- 172.21.15.116:0/3825450749 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f470c0c2e70 0x7f470c15d680 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:21.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:21.067+0000 7f4712782640 1 --2- 172.21.15.116:0/3825450749 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f470c0c2a10 0x7f470c1f37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:21.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:21.067+0000 7f4712782640 1 -- 172.21.15.116:0/3825450749 >> 172.21.15.116:0/3825450749 conn(0x7f470c0c3b00 msgr2=0x7f470c14ed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:19:21.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:21.067+0000 7f4712782640 1 -- 172.21.15.116:0/3825450749 shutdown_connections 2024-04-25T00:19:21.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:21.067+0000 7f4712782640 1 -- 172.21.15.116:0/3825450749 wait complete. 2024-04-25T00:19:21.080 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:19:23.081 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:19:23.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.218+0000 7fe871ae2640 1 Processor -- start 2024-04-25T00:19:23.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.218+0000 7fe871ae2640 1 -- start start 2024-04-25T00:19:23.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.218+0000 7fe871ae2640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe86c071d90 0x7fe86c1563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:23.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.218+0000 7fe871ae2640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe86c071220 con 0x7fe86c14c720 2024-04-25T00:19:23.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.218+0000 7fe871ae2640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe86c152420 con 0x7fe86c0a94a0 2024-04-25T00:19:23.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.218+0000 7fe871ae2640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fe86c153460 con 0x7fe86c071d90 2024-04-25T00:19:23.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.222+0000 7fe86bfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe86c071d90 0x7fe86c1563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:23.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.222+0000 7fe86bfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe86c071d90 0x7fe86c1563d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35846/0 (socket says 172.21.15.116:35846) 2024-04-25T00:19:23.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.222+0000 7fe86bfff640 1 -- 172.21.15.116:0/1843928686 learned_addr learned my addr 172.21.15.116:0/1843928686 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:19:23.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.222+0000 7fe86bfff640 1 -- 172.21.15.116:0/1843928686 >> v1:172.21.15.174:6789/0 conn(0x7fe86c0a94a0 legacy=0x7fe86c071680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:23.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.222+0000 7fe86bfff640 1 -- 172.21.15.116:0/1843928686 >> v1:172.21.15.116:6789/0 conn(0x7fe86c14c720 legacy=0x7fe86c0c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:23.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.222+0000 7fe86bfff640 1 -- 172.21.15.116:0/1843928686 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe86c0721c0 con 0x7fe86c071d90 2024-04-25T00:19:23.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.222+0000 7fe86bfff640 1 --2- 172.21.15.116:0/1843928686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe86c071d90 0x7fe86c1563d0 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7fe86400d150 tx=0x7fe86401eba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf52cb5a4c017cd0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:23.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.222+0000 7fe86a7fc640 1 -- 172.21.15.116:0/1843928686 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86400b2d0 con 0x7fe86c071d90 2024-04-25T00:19:23.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.222+0000 7fe86a7fc640 1 -- 172.21.15.116:0/1843928686 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe86400b470 con 0x7fe86c071d90 2024-04-25T00:19:23.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.222+0000 7fe86a7fc640 1 -- 172.21.15.116:0/1843928686 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8640a7b90 con 0x7fe86c071d90 2024-04-25T00:19:23.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.222+0000 7fe871ae2640 1 -- 172.21.15.116:0/1843928686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe86c071d90 msgr2=0x7fe86c1563d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:23.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.222+0000 7fe871ae2640 1 --2- 172.21.15.116:0/1843928686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe86c071d90 0x7fe86c1563d0 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7fe86400d150 tx=0x7fe86401eba0 comp rx=0 tx=0).stop 2024-04-25T00:19:23.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.222+0000 7fe871ae2640 1 -- 172.21.15.116:0/1843928686 shutdown_connections 2024-04-25T00:19:23.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.222+0000 7fe871ae2640 1 --2- 172.21.15.116:0/1843928686 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe86c071d90 0x7fe86c1563d0 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:23.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.222+0000 7fe871ae2640 1 -- 172.21.15.116:0/1843928686 >> 172.21.15.116:0/1843928686 conn(0x7fe86c0c3b00 msgr2=0x7fe86c0c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:19:23.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.222+0000 7fe871ae2640 1 -- 172.21.15.116:0/1843928686 shutdown_connections 2024-04-25T00:19:23.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.222+0000 7fe871ae2640 1 -- 172.21.15.116:0/1843928686 wait complete. 2024-04-25T00:19:23.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.226+0000 7fe871ae2640 1 Processor -- start 2024-04-25T00:19:23.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.226+0000 7fe871ae2640 1 -- start start 2024-04-25T00:19:23.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.226+0000 7fe871ae2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe86c071d90 0x7fe86c14a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:23.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.226+0000 7fe871ae2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe86c0a94a0 0x7fe86c0aae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:23.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.226+0000 7fe871ae2640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe86c14c720 0x7fe86c153810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:23.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.226+0000 7fe871ae2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe86c152f40 con 0x7fe86c0a94a0 2024-04-25T00:19:23.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.226+0000 7fe871ae2640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe86c152c60 con 0x7fe86c071d90 2024-04-25T00:19:23.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.226+0000 7fe871ae2640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe86c0c3120 con 0x7fe86c14c720 2024-04-25T00:19:23.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.226+0000 7fe86affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe86c0a94a0 0x7fe86c0aae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:23.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.226+0000 7fe86affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe86c0a94a0 0x7fe86c0aae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35862/0 (socket says 172.21.15.116:35862) 2024-04-25T00:19:23.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.226+0000 7fe86affd640 1 -- 172.21.15.116:0/2960271017 learned_addr learned my addr 172.21.15.116:0/2960271017 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:19:23.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.226+0000 7fe86b7fe640 1 --2- 172.21.15.116:0/2960271017 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe86c071d90 0x7fe86c14a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:23.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.226+0000 7fe86bfff640 1 --2- 172.21.15.116:0/2960271017 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe86c14c720 0x7fe86c153810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:23.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.226+0000 7fe86affd640 1 -- 172.21.15.116:0/2960271017 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe86c14c720 msgr2=0x7fe86c153810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:23.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.226+0000 7fe86affd640 1 --2- 172.21.15.116:0/2960271017 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe86c14c720 0x7fe86c153810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:23.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.226+0000 7fe86affd640 1 -- 172.21.15.116:0/2960271017 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe86c071d90 msgr2=0x7fe86c14a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:23.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.226+0000 7fe86affd640 1 --2- 172.21.15.116:0/2960271017 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe86c071d90 0x7fe86c14a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:23.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.226+0000 7fe86affd640 1 -- 172.21.15.116:0/2960271017 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe86c141020 con 0x7fe86c0a94a0 2024-04-25T00:19:23.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.226+0000 7fe86b7fe640 1 --2- 172.21.15.116:0/2960271017 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe86c071d90 0x7fe86c14a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:19:23.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.226+0000 7fe86affd640 1 --2- 172.21.15.116:0/2960271017 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe86c0a94a0 0x7fe86c0aae70 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7fe86000c420 tx=0x7fe86000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:23.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.226+0000 7fe853fff640 1 -- 172.21.15.116:0/2960271017 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8600052d0 con 0x7fe86c0a94a0 2024-04-25T00:19:23.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.226+0000 7fe853fff640 1 -- 172.21.15.116:0/2960271017 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe860005470 con 0x7fe86c0a94a0 2024-04-25T00:19:23.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.226+0000 7fe871ae2640 1 -- 172.21.15.116:0/2960271017 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe86c14b420 con 0x7fe86c0a94a0 2024-04-25T00:19:23.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.226+0000 7fe853fff640 1 -- 172.21.15.116:0/2960271017 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe860007260 con 0x7fe86c0a94a0 2024-04-25T00:19:23.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.226+0000 7fe871ae2640 1 -- 172.21.15.116:0/2960271017 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe86c14aee0 con 0x7fe86c0a94a0 2024-04-25T00:19:23.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.226+0000 7fe871ae2640 1 -- 172.21.15.116:0/2960271017 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe86c068350 con 0x7fe86c0a94a0 2024-04-25T00:19:23.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.226+0000 7fe853fff640 1 -- 172.21.15.116:0/2960271017 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe860007400 con 0x7fe86c0a94a0 2024-04-25T00:19:23.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.230+0000 7fe853fff640 1 --2- 172.21.15.116:0/2960271017 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe8580678c0 0x7fe858069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:23.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.230+0000 7fe853fff640 1 -- 172.21.15.116:0/2960271017 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 9074+0+0 (secure 0 0 0) 0x7fe860085e10 con 0x7fe86c0a94a0 2024-04-25T00:19:23.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.230+0000 7fe86b7fe640 1 --2- 172.21.15.116:0/2960271017 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe8580678c0 0x7fe858069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:23.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.230+0000 7fe86b7fe640 1 --2- 172.21.15.116:0/2960271017 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe8580678c0 0x7fe858069d80 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fe85c000f70 tx=0x7fe85c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:23.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.234+0000 7fe853fff640 1 -- 172.21.15.116:0/2960271017 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fe86c152420 con 0x7fe86c0a94a0 2024-04-25T00:19:23.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.430+0000 7fe871ae2640 1 -- 172.21.15.116:0/2960271017 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe86c0c3640 con 0x7fe86c0a94a0 2024-04-25T00:19:23.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.430+0000 7fe853fff640 1 -- 172.21.15.116:0/2960271017 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) ==== 95+0+6503 (secure 0 0 0) 0x7fe860055580 con 0x7fe86c0a94a0 2024-04-25T00:19:23.431 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:19:23.431 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":152,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/991183935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":991183935},{"type":"v1","addr":"172.21.15.174:6837","nonce":991183935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":152}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:18:23.899096+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7362},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7362":{"gid":7362,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6833/1655959433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1655959433},{"type":"v1","addr":"172.21.15.116:6833","nonce":1655959433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7362,"qdb_cluster":[7362]},"id":1},{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:19:08.844382+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7542},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7542":{"gid":7542,"name":"c","rank":0,"incarnation":148,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6835/879531976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":879531976},{"type":"v1","addr":"172.21.15.116:6835","nonce":879531976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7542,"qdb_cluster":[7542]},"id":2}]} 2024-04-25T00:19:23.431 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 152 2024-04-25T00:19:23.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.434+0000 7fe871ae2640 1 -- 172.21.15.116:0/2960271017 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe8580678c0 msgr2=0x7fe858069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:23.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.434+0000 7fe871ae2640 1 --2- 172.21.15.116:0/2960271017 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe8580678c0 0x7fe858069d80 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fe85c000f70 tx=0x7fe85c009040 comp rx=0 tx=0).stop 2024-04-25T00:19:23.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.434+0000 7fe871ae2640 1 -- 172.21.15.116:0/2960271017 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe86c0a94a0 msgr2=0x7fe86c0aae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:23.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.434+0000 7fe871ae2640 1 --2- 172.21.15.116:0/2960271017 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe86c0a94a0 0x7fe86c0aae70 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7fe86000c420 tx=0x7fe86000c8e0 comp rx=0 tx=0).stop 2024-04-25T00:19:23.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.434+0000 7fe871ae2640 1 -- 172.21.15.116:0/2960271017 shutdown_connections 2024-04-25T00:19:23.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.434+0000 7fe871ae2640 1 --2- 172.21.15.116:0/2960271017 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe8580678c0 0x7fe858069d80 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:23.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.434+0000 7fe871ae2640 1 --2- 172.21.15.116:0/2960271017 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe86c14c720 0x7fe86c153810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:23.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.434+0000 7fe871ae2640 1 --2- 172.21.15.116:0/2960271017 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe86c0a94a0 0x7fe86c0aae70 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:23.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.434+0000 7fe871ae2640 1 --2- 172.21.15.116:0/2960271017 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe86c071d90 0x7fe86c14a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:23.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.434+0000 7fe871ae2640 1 -- 172.21.15.116:0/2960271017 >> 172.21.15.116:0/2960271017 conn(0x7fe86c0c3b00 msgr2=0x7fe86c071470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:19:23.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.434+0000 7fe871ae2640 1 -- 172.21.15.116:0/2960271017 shutdown_connections 2024-04-25T00:19:23.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:23.434+0000 7fe871ae2640 1 -- 172.21.15.116:0/2960271017 wait complete. 2024-04-25T00:19:23.450 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:19:25.450 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:19:25.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.582+0000 7efeeea1a640 1 Processor -- start 2024-04-25T00:19:25.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.582+0000 7efeeea1a640 1 -- start start 2024-04-25T00:19:25.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.586+0000 7efeeea1a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efee814c720 0x7efee81563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:25.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.586+0000 7efeeea1a640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efee8071220 con 0x7efee80a94a0 2024-04-25T00:19:25.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.586+0000 7efeeea1a640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efee80721e0 con 0x7efee8071680 2024-04-25T00:19:25.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.586+0000 7efeeea1a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7efee8153460 con 0x7efee814c720 2024-04-25T00:19:25.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.586+0000 7efee77fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7efee80a94a0 0x7efee80c2a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:48384/0 (socket says 172.21.15.116:48384) 2024-04-25T00:19:25.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.586+0000 7efee77fe640 1 -- 172.21.15.116:0/2613322553 learned_addr learned my addr 172.21.15.116:0/2613322553 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:19:25.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.586+0000 7efee7fff640 1 --2- 172.21.15.116:0/2613322553 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efee814c720 0x7efee81563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:25.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.586+0000 7efee6ffd640 1 -- 172.21.15.116:0/2613322553 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3293050583 0 0) 0x7efee8071220 con 0x7efee80a94a0 2024-04-25T00:19:25.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.586+0000 7efee6ffd640 1 -- 172.21.15.116:0/2613322553 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efed0003530 con 0x7efee80a94a0 2024-04-25T00:19:25.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.586+0000 7efee6ffd640 1 -- 172.21.15.116:0/2613322553 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3345833027 0 0) 0x7efed0003530 con 0x7efee80a94a0 2024-04-25T00:19:25.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.586+0000 7efee6ffd640 1 -- 172.21.15.116:0/2613322553 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efee814c720 msgr2=0x7efee81563d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:19:25.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.586+0000 7efee6ffd640 1 --2- 172.21.15.116:0/2613322553 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efee814c720 0x7efee81563d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:25.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.586+0000 7efee6ffd640 1 -- 172.21.15.116:0/2613322553 >> v1:172.21.15.174:6789/0 conn(0x7efee8071680 legacy=0x7efee8071ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:25.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.586+0000 7efee6ffd640 1 -- 172.21.15.116:0/2613322553 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efee8152420 con 0x7efee80a94a0 2024-04-25T00:19:25.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.586+0000 7efee6ffd640 1 -- 172.21.15.116:0/2613322553 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7efedc001680 con 0x7efee80a94a0 2024-04-25T00:19:25.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.586+0000 7efee6ffd640 1 -- 172.21.15.116:0/2613322553 >> v1:172.21.15.116:6789/0 conn(0x7efee80a94a0 legacy=0x7efee80c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:25.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.586+0000 7efee6ffd640 1 --2- 172.21.15.116:0/2613322553 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efed00039f0 0x7efed0003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:25.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.586+0000 7efee6ffd640 1 --2- 172.21.15.116:0/2613322553 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efee814c720 0x7efed0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:25.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.586+0000 7efee6ffd640 1 --2- 172.21.15.116:0/2613322553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efed0004d50 0x7efed00051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:25.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.586+0000 7efee6ffd640 1 -- 172.21.15.116:0/2613322553 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7efee8153460 con 0x7efed0004d50 2024-04-25T00:19:25.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.586+0000 7efee6ffd640 1 -- 172.21.15.116:0/2613322553 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7efed000bb00 con 0x7efed00039f0 2024-04-25T00:19:25.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.586+0000 7efee6ffd640 1 -- 172.21.15.116:0/2613322553 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7efed000bd00 con 0x7efee814c720 2024-04-25T00:19:25.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.586+0000 7efeecfb7640 1 --2- 172.21.15.116:0/2613322553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efed0004d50 0x7efed00051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:25.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.586+0000 7efee7fff640 1 --2- 172.21.15.116:0/2613322553 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efed00039f0 0x7efed0003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:25.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.586+0000 7efeecfb7640 1 -- 172.21.15.116:0/2613322553 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efee814c720 msgr2=0x7efed0004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:19:25.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.586+0000 7efeecfb7640 1 --2- 172.21.15.116:0/2613322553 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efee814c720 0x7efed0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:25.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.586+0000 7efeecfb7640 1 -- 172.21.15.116:0/2613322553 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efed00039f0 msgr2=0x7efed0003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:19:25.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.586+0000 7efeecfb7640 1 --2- 172.21.15.116:0/2613322553 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efed00039f0 0x7efed0003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:25.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.586+0000 7efeecfb7640 1 -- 172.21.15.116:0/2613322553 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efed0003530 con 0x7efed0004d50 2024-04-25T00:19:25.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.586+0000 7efeecfb7640 1 --2- 172.21.15.116:0/2613322553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efed0004d50 0x7efed00051d0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7efee0001e70 tx=0x7efee001c780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ef5639464e51e04 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:25.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.590+0000 7efee6ffd640 1 -- 172.21.15.116:0/2613322553 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee0001a90 con 0x7efed0004d50 2024-04-25T00:19:25.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.590+0000 7efee6ffd640 1 -- 172.21.15.116:0/2613322553 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efee0003b80 con 0x7efed0004d50 2024-04-25T00:19:25.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.590+0000 7efee6ffd640 1 -- 172.21.15.116:0/2613322553 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee0005080 con 0x7efed0004d50 2024-04-25T00:19:25.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.590+0000 7efeeea1a640 1 -- 172.21.15.116:0/2613322553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efed0004d50 msgr2=0x7efed00051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:25.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.590+0000 7efeeea1a640 1 --2- 172.21.15.116:0/2613322553 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efed0004d50 0x7efed00051d0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7efee0001e70 tx=0x7efee001c780 comp rx=0 tx=0).stop 2024-04-25T00:19:25.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.590+0000 7efeecfb7640 1 -- 172.21.15.116:0/2613322553 reap_dead start 2024-04-25T00:19:25.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.590+0000 7efeeea1a640 1 -- 172.21.15.116:0/2613322553 shutdown_connections 2024-04-25T00:19:25.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.590+0000 7efeeea1a640 1 -- 172.21.15.116:0/2613322553 >> 172.21.15.116:0/2613322553 conn(0x7efee80c3b00 msgr2=0x7efee80c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:19:25.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.590+0000 7efeeea1a640 1 -- 172.21.15.116:0/2613322553 shutdown_connections 2024-04-25T00:19:25.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.590+0000 7efeeea1a640 1 -- 172.21.15.116:0/2613322553 wait complete. 2024-04-25T00:19:25.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.590+0000 7efeeea1a640 1 Processor -- start 2024-04-25T00:19:25.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.590+0000 7efeeea1a640 1 -- start start 2024-04-25T00:19:25.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.590+0000 7efeeea1a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efee80c1d10 0x7efee80aae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:25.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.590+0000 7efeeea1a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efee80c4bf0 0x7efee8143ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:25.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.590+0000 7efeeea1a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efee80c5310 0x7efee813e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:25.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.590+0000 7efeeea1a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7efee8152980 con 0x7efee80c1d10 2024-04-25T00:19:25.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.590+0000 7efeeea1a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7efee81526a0 con 0x7efee80c4bf0 2024-04-25T00:19:25.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.590+0000 7efeeea1a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7efee8152c60 con 0x7efee80c5310 2024-04-25T00:19:25.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.590+0000 7efee7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efee80c1d10 0x7efee80aae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:25.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.590+0000 7efee7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efee80c1d10 0x7efee80aae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59102/0 (socket says 172.21.15.116:59102) 2024-04-25T00:19:25.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.590+0000 7efee7fff640 1 -- 172.21.15.116:0/3934455816 learned_addr learned my addr 172.21.15.116:0/3934455816 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:19:25.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.590+0000 7efee77fe640 1 --2- 172.21.15.116:0/3934455816 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efee80c4bf0 0x7efee8143ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:25.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.590+0000 7efeecfb7640 1 --2- 172.21.15.116:0/3934455816 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efee80c5310 0x7efee813e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:25.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.590+0000 7efee7fff640 1 -- 172.21.15.116:0/3934455816 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efee80c5310 msgr2=0x7efee813e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:25.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.590+0000 7efee7fff640 1 --2- 172.21.15.116:0/3934455816 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efee80c5310 0x7efee813e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:25.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.590+0000 7efee7fff640 1 -- 172.21.15.116:0/3934455816 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efee80c4bf0 msgr2=0x7efee8143ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:25.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.590+0000 7efee7fff640 1 --2- 172.21.15.116:0/3934455816 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efee80c4bf0 0x7efee8143ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:25.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.590+0000 7efee7fff640 1 -- 172.21.15.116:0/3934455816 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efee8144d50 con 0x7efee80c1d10 2024-04-25T00:19:25.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.590+0000 7efee77fe640 1 --2- 172.21.15.116:0/3934455816 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efee80c4bf0 0x7efee8143ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:19:25.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.590+0000 7efee7fff640 1 --2- 172.21.15.116:0/3934455816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efee80c1d10 0x7efee80aae70 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7efed8009f70 tx=0x7efed8011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:25.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.590+0000 7efee4ff9640 1 -- 172.21.15.116:0/3934455816 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efed800f960 con 0x7efee80c1d10 2024-04-25T00:19:25.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.590+0000 7efee4ff9640 1 -- 172.21.15.116:0/3934455816 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efed8018070 con 0x7efee80c1d10 2024-04-25T00:19:25.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.590+0000 7efee4ff9640 1 -- 172.21.15.116:0/3934455816 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efed800fd40 con 0x7efee80c1d10 2024-04-25T00:19:25.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.594+0000 7efeeea1a640 1 -- 172.21.15.116:0/3934455816 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efee8144790 con 0x7efee80c1d10 2024-04-25T00:19:25.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.594+0000 7efeeea1a640 1 -- 172.21.15.116:0/3934455816 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efee8142fe0 con 0x7efee80c1d10 2024-04-25T00:19:25.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.594+0000 7efeeea1a640 1 -- 172.21.15.116:0/3934455816 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efee8068350 con 0x7efee80c1d10 2024-04-25T00:19:25.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.594+0000 7efee4ff9640 1 -- 172.21.15.116:0/3934455816 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7efed801a030 con 0x7efee80c1d10 2024-04-25T00:19:25.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.594+0000 7efee4ff9640 1 --2- 172.21.15.116:0/3934455816 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efed40678c0 0x7efed4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:25.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.594+0000 7efee4ff9640 1 -- 172.21.15.116:0/3934455816 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 9074+0+0 (secure 0 0 0) 0x7efed8088970 con 0x7efee80c1d10 2024-04-25T00:19:25.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.594+0000 7efee77fe640 1 --2- 172.21.15.116:0/3934455816 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efed40678c0 0x7efed4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:25.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.594+0000 7efee77fe640 1 --2- 172.21.15.116:0/3934455816 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efed40678c0 0x7efed4069d80 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7efedc0017f0 tx=0x7efedc0016a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:25.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.598+0000 7efee4ff9640 1 -- 172.21.15.116:0/3934455816 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7efed80586d0 con 0x7efee80c1d10 2024-04-25T00:19:25.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.790+0000 7efeeea1a640 1 -- 172.21.15.116:0/3934455816 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efee80c4290 con 0x7efee80c1d10 2024-04-25T00:19:25.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.790+0000 7efee4ff9640 1 -- 172.21.15.116:0/3934455816 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) ==== 95+0+6503 (secure 0 0 0) 0x7efed8057fd0 con 0x7efee80c1d10 2024-04-25T00:19:25.795 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:19:25.795 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":152,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/991183935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":991183935},{"type":"v1","addr":"172.21.15.174:6837","nonce":991183935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":152}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:18:23.899096+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7362},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7362":{"gid":7362,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6833/1655959433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1655959433},{"type":"v1","addr":"172.21.15.116:6833","nonce":1655959433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7362,"qdb_cluster":[7362]},"id":1},{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:19:08.844382+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7542},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7542":{"gid":7542,"name":"c","rank":0,"incarnation":148,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6835/879531976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":879531976},{"type":"v1","addr":"172.21.15.116:6835","nonce":879531976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7542,"qdb_cluster":[7542]},"id":2}]} 2024-04-25T00:19:25.795 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 152 2024-04-25T00:19:25.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.794+0000 7efeeea1a640 1 -- 172.21.15.116:0/3934455816 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efed40678c0 msgr2=0x7efed4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:25.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.794+0000 7efeeea1a640 1 --2- 172.21.15.116:0/3934455816 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efed40678c0 0x7efed4069d80 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7efedc0017f0 tx=0x7efedc0016a0 comp rx=0 tx=0).stop 2024-04-25T00:19:25.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.794+0000 7efeeea1a640 1 -- 172.21.15.116:0/3934455816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efee80c1d10 msgr2=0x7efee80aae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:25.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.794+0000 7efeeea1a640 1 --2- 172.21.15.116:0/3934455816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efee80c1d10 0x7efee80aae70 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7efed8009f70 tx=0x7efed8011b60 comp rx=0 tx=0).stop 2024-04-25T00:19:25.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.794+0000 7efeeea1a640 1 -- 172.21.15.116:0/3934455816 shutdown_connections 2024-04-25T00:19:25.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.794+0000 7efeeea1a640 1 --2- 172.21.15.116:0/3934455816 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efed40678c0 0x7efed4069d80 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:25.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.794+0000 7efeeea1a640 1 --2- 172.21.15.116:0/3934455816 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efee80c5310 0x7efee813e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:25.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.794+0000 7efeeea1a640 1 --2- 172.21.15.116:0/3934455816 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efee80c4bf0 0x7efee8143ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:25.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.794+0000 7efeeea1a640 1 --2- 172.21.15.116:0/3934455816 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efee80c1d10 0x7efee80aae70 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:25.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.794+0000 7efeeea1a640 1 -- 172.21.15.116:0/3934455816 >> 172.21.15.116:0/3934455816 conn(0x7efee80c3b00 msgr2=0x7efee80723b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:19:25.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.794+0000 7efeeea1a640 1 -- 172.21.15.116:0/3934455816 shutdown_connections 2024-04-25T00:19:25.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:25.794+0000 7efeeea1a640 1 -- 172.21.15.116:0/3934455816 wait complete. 2024-04-25T00:19:25.808 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:19:27.809 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:19:27.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.942+0000 7f9bd3380640 1 Processor -- start 2024-04-25T00:19:27.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.942+0000 7f9bd3380640 1 -- start start 2024-04-25T00:19:27.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.942+0000 7f9bd3380640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9bcc14c720 0x7f9bcc1563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:27.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.942+0000 7f9bd3380640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f9bcc0a94a0 0x7f9bcc0c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:27.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.942+0000 7f9bd3380640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9bcc071220 con 0x7f9bcc0c2f50 2024-04-25T00:19:27.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.942+0000 7f9bd3380640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f9bcc153460 con 0x7f9bcc14c720 2024-04-25T00:19:27.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.942+0000 7f9bd3380640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f9bcc152420 con 0x7f9bcc0a94a0 2024-04-25T00:19:27.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.942+0000 7f9bd111c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9bcc14c720 0x7f9bcc1563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:27.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.942+0000 7f9bd111c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9bcc14c720 0x7f9bcc1563d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59118/0 (socket says 172.21.15.116:59118) 2024-04-25T00:19:27.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.942+0000 7f9bd111c640 1 -- 172.21.15.116:0/2559397496 learned_addr learned my addr 172.21.15.116:0/2559397496 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:19:27.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.942+0000 7f9bd091b640 1 --2- 172.21.15.116:0/2559397496 >> v2:172.21.15.174:3300/0 conn(0x7f9bcc0a94a0 0x7f9bcc0c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:27.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.942+0000 7f9bd111c640 1 -- 172.21.15.116:0/2559397496 >> v2:172.21.15.174:3300/0 conn(0x7f9bcc0a94a0 msgr2=0x7f9bcc0c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:19:27.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.942+0000 7f9bd111c640 1 --2- 172.21.15.116:0/2559397496 >> v2:172.21.15.174:3300/0 conn(0x7f9bcc0a94a0 0x7f9bcc0c2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:27.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.942+0000 7f9bd111c640 1 -- 172.21.15.116:0/2559397496 >> v1:172.21.15.174:6789/0 conn(0x7f9bcc0c2f50 legacy=0x7f9bcc071680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:19:27.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.942+0000 7f9bd111c640 1 -- 172.21.15.116:0/2559397496 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bcc14cb00 con 0x7f9bcc14c720 2024-04-25T00:19:27.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.942+0000 7f9bd091b640 1 --2- 172.21.15.116:0/2559397496 >> v2:172.21.15.174:3300/0 conn(0x7f9bcc0a94a0 0x7f9bcc0c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:19:27.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.942+0000 7f9bd111c640 1 --2- 172.21.15.116:0/2559397496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9bcc14c720 0x7f9bcc1563d0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc0a9930 tx=0x7f9bc801c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83df8cde83b6e6a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:27.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.946+0000 7f9bc3fff640 1 -- 172.21.15.116:0/2559397496 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc80ab640 con 0x7f9bcc14c720 2024-04-25T00:19:27.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.946+0000 7f9bc3fff640 1 -- 172.21.15.116:0/2559397496 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bc80b0070 con 0x7f9bcc14c720 2024-04-25T00:19:27.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.946+0000 7f9bc3fff640 1 -- 172.21.15.116:0/2559397496 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc80aba00 con 0x7f9bcc14c720 2024-04-25T00:19:27.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.946+0000 7f9bd3380640 1 -- 172.21.15.116:0/2559397496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9bcc14c720 msgr2=0x7f9bcc1563d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:27.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.946+0000 7f9bd3380640 1 --2- 172.21.15.116:0/2559397496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9bcc14c720 0x7f9bcc1563d0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc0a9930 tx=0x7f9bc801c420 comp rx=0 tx=0).stop 2024-04-25T00:19:27.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.946+0000 7f9bd3380640 1 -- 172.21.15.116:0/2559397496 shutdown_connections 2024-04-25T00:19:27.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.946+0000 7f9bd3380640 1 --2- 172.21.15.116:0/2559397496 >> v2:172.21.15.174:3300/0 conn(0x7f9bcc0a94a0 0x7f9bcc0c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:27.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.946+0000 7f9bd3380640 1 --2- 172.21.15.116:0/2559397496 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9bcc14c720 0x7f9bcc1563d0 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:27.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.946+0000 7f9bd3380640 1 -- 172.21.15.116:0/2559397496 >> 172.21.15.116:0/2559397496 conn(0x7f9bcc0c3b00 msgr2=0x7f9bcc0c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:19:27.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.946+0000 7f9bd3380640 1 -- 172.21.15.116:0/2559397496 shutdown_connections 2024-04-25T00:19:27.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.946+0000 7f9bd3380640 1 -- 172.21.15.116:0/2559397496 wait complete. 2024-04-25T00:19:27.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.946+0000 7f9bd3380640 1 Processor -- start 2024-04-25T00:19:27.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.946+0000 7f9bd3380640 1 -- start start 2024-04-25T00:19:27.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.950+0000 7f9bd3380640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9bcc0a94a0 0x7f9bcc0ab3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:27.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.950+0000 7f9bd3380640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9bcc0c2f50 0x7f9bcc13c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:27.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.950+0000 7f9bd3380640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9bcc14c720 0x7f9bcc0c58e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:27.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.950+0000 7f9bd3380640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9bcc152f40 con 0x7f9bcc0c2f50 2024-04-25T00:19:27.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.950+0000 7f9bd3380640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9bcc152c60 con 0x7f9bcc14c720 2024-04-25T00:19:27.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.950+0000 7f9bd3380640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f9bcc15f5c0 con 0x7f9bcc0a94a0 2024-04-25T00:19:27.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.950+0000 7f9bd091b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9bcc0c2f50 0x7f9bcc13c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:27.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.950+0000 7f9bd091b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9bcc0c2f50 0x7f9bcc13c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59124/0 (socket says 172.21.15.116:59124) 2024-04-25T00:19:27.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.950+0000 7f9bd091b640 1 -- 172.21.15.116:0/401413946 learned_addr learned my addr 172.21.15.116:0/401413946 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:19:27.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.950+0000 7f9bd111c640 1 --2- 172.21.15.116:0/401413946 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9bcc0a94a0 0x7f9bcc0ab3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:27.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.950+0000 7f9bd191d640 1 --2- 172.21.15.116:0/401413946 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9bcc14c720 0x7f9bcc0c58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:27.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.950+0000 7f9bd091b640 1 -- 172.21.15.116:0/401413946 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9bcc0a94a0 msgr2=0x7f9bcc0ab3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:27.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.950+0000 7f9bd091b640 1 --2- 172.21.15.116:0/401413946 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9bcc0a94a0 0x7f9bcc0ab3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:27.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.950+0000 7f9bd091b640 1 -- 172.21.15.116:0/401413946 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9bcc14c720 msgr2=0x7f9bcc0c58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:27.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.950+0000 7f9bd091b640 1 --2- 172.21.15.116:0/401413946 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9bcc14c720 0x7f9bcc0c58e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:27.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.950+0000 7f9bd091b640 1 -- 172.21.15.116:0/401413946 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bcc0c5120 con 0x7f9bcc0c2f50 2024-04-25T00:19:27.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.950+0000 7f9bd111c640 1 --2- 172.21.15.116:0/401413946 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9bcc0a94a0 0x7f9bcc0ab3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:19:27.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.950+0000 7f9bd191d640 1 --2- 172.21.15.116:0/401413946 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9bcc14c720 0x7f9bcc0c58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:19:27.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.950+0000 7f9bd091b640 1 --2- 172.21.15.116:0/401413946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9bcc0c2f50 0x7f9bcc13c340 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc00c7c0 tx=0x7f9bbc00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:27.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.950+0000 7f9bc1ffb640 1 -- 172.21.15.116:0/401413946 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bbc0079d0 con 0x7f9bcc0c2f50 2024-04-25T00:19:27.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.950+0000 7f9bc1ffb640 1 -- 172.21.15.116:0/401413946 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bbc007b70 con 0x7f9bcc0c2f50 2024-04-25T00:19:27.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.950+0000 7f9bc1ffb640 1 -- 172.21.15.116:0/401413946 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bbc0113f0 con 0x7f9bcc0c2f50 2024-04-25T00:19:27.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.950+0000 7f9bd3380640 1 -- 172.21.15.116:0/401413946 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bcc147cb0 con 0x7f9bcc0c2f50 2024-04-25T00:19:27.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.950+0000 7f9bd3380640 1 -- 172.21.15.116:0/401413946 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bcc0c53c0 con 0x7f9bcc0c2f50 2024-04-25T00:19:27.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.950+0000 7f9bd3380640 1 -- 172.21.15.116:0/401413946 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bcc0c55c0 con 0x7f9bcc0c2f50 2024-04-25T00:19:27.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.950+0000 7f9bc1ffb640 1 -- 172.21.15.116:0/401413946 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f9bbc011590 con 0x7f9bcc0c2f50 2024-04-25T00:19:27.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.954+0000 7f9bc1ffb640 1 --2- 172.21.15.116:0/401413946 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9bb00678f0 0x7f9bb0069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:27.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.954+0000 7f9bc1ffb640 1 -- 172.21.15.116:0/401413946 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 9074+0+0 (secure 0 0 0) 0x7f9bbc086af0 con 0x7f9bcc0c2f50 2024-04-25T00:19:27.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.954+0000 7f9bd111c640 1 --2- 172.21.15.116:0/401413946 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9bb00678f0 0x7f9bb0069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:27.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.954+0000 7f9bd111c640 1 --2- 172.21.15.116:0/401413946 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9bb00678f0 0x7f9bb0069db0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f9bc80af000 tx=0x7f9bc80a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:27.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:27.958+0000 7f9bc1ffb640 1 -- 172.21.15.116:0/401413946 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f9bcc0c55c0 con 0x7f9bcc0c2f50 2024-04-25T00:19:28.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:28.150+0000 7f9bd3380640 1 -- 172.21.15.116:0/401413946 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9bcc1517e0 con 0x7f9bcc0c2f50 2024-04-25T00:19:28.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:28.154+0000 7f9bc1ffb640 1 -- 172.21.15.116:0/401413946 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) ==== 95+0+6503 (secure 0 0 0) 0x7f9bbc015020 con 0x7f9bcc0c2f50 2024-04-25T00:19:28.153 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:19:28.153 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":152,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/991183935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":991183935},{"type":"v1","addr":"172.21.15.174:6837","nonce":991183935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":152}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:18:23.899096+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7362},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7362":{"gid":7362,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6833/1655959433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1655959433},{"type":"v1","addr":"172.21.15.116:6833","nonce":1655959433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7362,"qdb_cluster":[7362]},"id":1},{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:19:08.844382+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7542},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7542":{"gid":7542,"name":"c","rank":0,"incarnation":148,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6835/879531976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":879531976},{"type":"v1","addr":"172.21.15.116:6835","nonce":879531976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7542,"qdb_cluster":[7542]},"id":2}]} 2024-04-25T00:19:28.153 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 152 2024-04-25T00:19:28.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:28.154+0000 7f9bd3380640 1 -- 172.21.15.116:0/401413946 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9bb00678f0 msgr2=0x7f9bb0069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:28.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:28.154+0000 7f9bd3380640 1 --2- 172.21.15.116:0/401413946 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9bb00678f0 0x7f9bb0069db0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f9bc80af000 tx=0x7f9bc80a7000 comp rx=0 tx=0).stop 2024-04-25T00:19:28.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:28.154+0000 7f9bd3380640 1 -- 172.21.15.116:0/401413946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9bcc0c2f50 msgr2=0x7f9bcc13c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:28.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:28.154+0000 7f9bd3380640 1 --2- 172.21.15.116:0/401413946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9bcc0c2f50 0x7f9bcc13c340 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc00c7c0 tx=0x7f9bbc00cc80 comp rx=0 tx=0).stop 2024-04-25T00:19:28.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:28.154+0000 7f9bd3380640 1 -- 172.21.15.116:0/401413946 shutdown_connections 2024-04-25T00:19:28.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:28.154+0000 7f9bd3380640 1 --2- 172.21.15.116:0/401413946 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9bb00678f0 0x7f9bb0069db0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:28.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:28.154+0000 7f9bd3380640 1 --2- 172.21.15.116:0/401413946 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9bcc14c720 0x7f9bcc0c58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:28.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:28.154+0000 7f9bd3380640 1 --2- 172.21.15.116:0/401413946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9bcc0c2f50 0x7f9bcc13c340 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:28.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:28.154+0000 7f9bd3380640 1 --2- 172.21.15.116:0/401413946 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9bcc0a94a0 0x7f9bcc0ab3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:28.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:28.154+0000 7f9bd3380640 1 -- 172.21.15.116:0/401413946 >> 172.21.15.116:0/401413946 conn(0x7f9bcc0c3b00 msgr2=0x7f9bcc0723b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:19:28.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:28.154+0000 7f9bd3380640 1 -- 172.21.15.116:0/401413946 shutdown_connections 2024-04-25T00:19:28.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:28.154+0000 7f9bd3380640 1 -- 172.21.15.116:0/401413946 wait complete. 2024-04-25T00:19:28.171 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:19:30.173 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:19:30.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.310+0000 7f7225428640 1 Processor -- start 2024-04-25T00:19:30.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.310+0000 7f7225428640 1 -- start start 2024-04-25T00:19:30.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.310+0000 7f7225428640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f7220150d20 0x7f72200c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:30.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.310+0000 7f7225428640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f72200c2f50 0x7f72201538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:30.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.310+0000 7f7225428640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f72200a94a0 0x7f7220153e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:30.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.310+0000 7f7225428640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f72200c3910 con 0x7f72200a94a0 2024-04-25T00:19:30.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.310+0000 7f7225428640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f72200c16e0 con 0x7f7220150d20 2024-04-25T00:19:30.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.310+0000 7f7225428640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f72200c1400 con 0x7f72200c2f50 2024-04-25T00:19:30.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.310+0000 7f721f7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f72200a94a0 0x7f7220153e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:30.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.310+0000 7f721f7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f72200a94a0 0x7f7220153e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59132/0 (socket says 172.21.15.116:59132) 2024-04-25T00:19:30.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.310+0000 7f721f7fe640 1 -- 172.21.15.116:0/1338458435 learned_addr learned my addr 172.21.15.116:0/1338458435 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:19:30.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.310+0000 7f7217fff640 1 --2- 172.21.15.116:0/1338458435 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f72200c2f50 0x7f72201538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:30.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.310+0000 7f721effd640 1 --2- 172.21.15.116:0/1338458435 >> v2:172.21.15.174:3300/0 conn(0x7f7220150d20 0x7f72200c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:30.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.310+0000 7f721f7fe640 1 -- 172.21.15.116:0/1338458435 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f72200c2f50 msgr2=0x7f72201538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:19:30.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.310+0000 7f721f7fe640 1 --2- 172.21.15.116:0/1338458435 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f72200c2f50 0x7f72201538e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:30.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.310+0000 7f721f7fe640 1 -- 172.21.15.116:0/1338458435 >> v2:172.21.15.174:3300/0 conn(0x7f7220150d20 msgr2=0x7f72200c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:19:30.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.310+0000 7f721f7fe640 1 --2- 172.21.15.116:0/1338458435 >> v2:172.21.15.174:3300/0 conn(0x7f7220150d20 0x7f72200c2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:30.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.310+0000 7f721f7fe640 1 -- 172.21.15.116:0/1338458435 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f722006e660 con 0x7f72200a94a0 2024-04-25T00:19:30.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.314+0000 7f721f7fe640 1 --2- 172.21.15.116:0/1338458435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72200a94a0 0x7f7220153e20 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f7220073550 tx=0x7f721801c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca0074599faece10 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:30.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.314+0000 7f721e7fc640 1 -- 172.21.15.116:0/1338458435 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72180a5740 con 0x7f72200a94a0 2024-04-25T00:19:30.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.314+0000 7f721e7fc640 1 -- 172.21.15.116:0/1338458435 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72180a58e0 con 0x7f72200a94a0 2024-04-25T00:19:30.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.314+0000 7f721e7fc640 1 -- 172.21.15.116:0/1338458435 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72180a5cb0 con 0x7f72200a94a0 2024-04-25T00:19:30.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.314+0000 7f7225428640 1 -- 172.21.15.116:0/1338458435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72200a94a0 msgr2=0x7f7220153e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:30.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.314+0000 7f7225428640 1 --2- 172.21.15.116:0/1338458435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72200a94a0 0x7f7220153e20 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f7220073550 tx=0x7f721801c9f0 comp rx=0 tx=0).stop 2024-04-25T00:19:30.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.314+0000 7f7225428640 1 -- 172.21.15.116:0/1338458435 shutdown_connections 2024-04-25T00:19:30.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.314+0000 7f7225428640 1 --2- 172.21.15.116:0/1338458435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72200a94a0 0x7f7220153e20 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:30.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.314+0000 7f7225428640 1 --2- 172.21.15.116:0/1338458435 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f72200c2f50 0x7f72201538e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:30.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.314+0000 7f7225428640 1 --2- 172.21.15.116:0/1338458435 >> v2:172.21.15.174:3300/0 conn(0x7f7220150d20 0x7f72200c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:30.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.314+0000 7f7225428640 1 -- 172.21.15.116:0/1338458435 >> 172.21.15.116:0/1338458435 conn(0x7f72200c3b00 msgr2=0x7f72200c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:19:30.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.314+0000 7f7225428640 1 -- 172.21.15.116:0/1338458435 shutdown_connections 2024-04-25T00:19:30.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.314+0000 7f7225428640 1 -- 172.21.15.116:0/1338458435 wait complete. 2024-04-25T00:19:30.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.314+0000 7f7225428640 1 Processor -- start 2024-04-25T00:19:30.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.314+0000 7f7225428640 1 -- start start 2024-04-25T00:19:30.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.314+0000 7f7225428640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72200a94a0 0x7f72200aa350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:30.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.314+0000 7f7225428640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f72200c2f50 0x7f72201528b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:30.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.314+0000 7f7225428640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7220150d20 0x7f72200724a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:30.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.314+0000 7f7225428640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7220157340 con 0x7f72200a94a0 2024-04-25T00:19:30.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.314+0000 7f7225428640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7220154570 con 0x7f7220150d20 2024-04-25T00:19:30.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.314+0000 7f7225428640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f72201574c0 con 0x7f72200c2f50 2024-04-25T00:19:30.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.314+0000 7f721effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72200a94a0 0x7f72200aa350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:30.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.314+0000 7f721effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72200a94a0 0x7f72200aa350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59142/0 (socket says 172.21.15.116:59142) 2024-04-25T00:19:30.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.314+0000 7f721effd640 1 -- 172.21.15.116:0/2673205529 learned_addr learned my addr 172.21.15.116:0/2673205529 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:19:30.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.314+0000 7f7217fff640 1 --2- 172.21.15.116:0/2673205529 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f72200c2f50 0x7f72201528b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:30.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.314+0000 7f721f7fe640 1 --2- 172.21.15.116:0/2673205529 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7220150d20 0x7f72200724a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:30.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.314+0000 7f721effd640 1 -- 172.21.15.116:0/2673205529 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f72200c2f50 msgr2=0x7f72201528b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:30.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.314+0000 7f721effd640 1 --2- 172.21.15.116:0/2673205529 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f72200c2f50 0x7f72201528b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:30.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.314+0000 7f721effd640 1 -- 172.21.15.116:0/2673205529 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7220150d20 msgr2=0x7f72200724a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:30.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.314+0000 7f721effd640 1 --2- 172.21.15.116:0/2673205529 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7220150d20 0x7f72200724a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:30.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.314+0000 7f721effd640 1 -- 172.21.15.116:0/2673205529 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f722014b150 con 0x7f72200a94a0 2024-04-25T00:19:30.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.314+0000 7f721effd640 1 --2- 172.21.15.116:0/2673205529 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72200a94a0 0x7f72200aa350 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f721000d5f0 tx=0x7f721000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:30.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.314+0000 7f72177fe640 1 -- 172.21.15.116:0/2673205529 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72100125f0 con 0x7f72200a94a0 2024-04-25T00:19:30.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.314+0000 7f72177fe640 1 -- 172.21.15.116:0/2673205529 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7210019070 con 0x7f72200a94a0 2024-04-25T00:19:30.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.314+0000 7f7225428640 1 -- 172.21.15.116:0/2673205529 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f722014b410 con 0x7f72200a94a0 2024-04-25T00:19:30.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.314+0000 7f72177fe640 1 -- 172.21.15.116:0/2673205529 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72100129b0 con 0x7f72200a94a0 2024-04-25T00:19:30.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.314+0000 7f7225428640 1 -- 172.21.15.116:0/2673205529 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72201f2420 con 0x7f72200a94a0 2024-04-25T00:19:30.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.318+0000 7f7225428640 1 -- 172.21.15.116:0/2673205529 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7220068350 con 0x7f72200a94a0 2024-04-25T00:19:30.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.318+0000 7f72177fe640 1 -- 172.21.15.116:0/2673205529 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7210012b50 con 0x7f72200a94a0 2024-04-25T00:19:30.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.318+0000 7f72177fe640 1 --2- 172.21.15.116:0/2673205529 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f72000678c0 0x7f7200069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:30.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.318+0000 7f72177fe640 1 -- 172.21.15.116:0/2673205529 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 9074+0+0 (secure 0 0 0) 0x7f7210086d80 con 0x7f72200a94a0 2024-04-25T00:19:30.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.318+0000 7f7217fff640 1 --2- 172.21.15.116:0/2673205529 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f72000678c0 0x7f7200069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:30.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.318+0000 7f7217fff640 1 --2- 172.21.15.116:0/2673205529 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f72000678c0 0x7f7200069d80 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f7220144920 tx=0x7f7208008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:30.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.322+0000 7f72177fe640 1 -- 172.21.15.116:0/2673205529 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f7210056bd0 con 0x7f72200a94a0 2024-04-25T00:19:30.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.518+0000 7f7225428640 1 -- 172.21.15.116:0/2673205529 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f72200c34f0 con 0x7f72200a94a0 2024-04-25T00:19:30.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.518+0000 7f72177fe640 1 -- 172.21.15.116:0/2673205529 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) ==== 95+0+6503 (secure 0 0 0) 0x7f72100564d0 con 0x7f72200a94a0 2024-04-25T00:19:30.519 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:19:30.519 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":152,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6586,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/991183935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":991183935},{"type":"v1","addr":"172.21.15.174:6837","nonce":991183935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":152}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:18:23.899096+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7362},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7362":{"gid":7362,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6833/1655959433","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":1655959433},{"type":"v1","addr":"172.21.15.116:6833","nonce":1655959433}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7362,"qdb_cluster":[7362]},"id":1},{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:19:08.844382+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7542},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7542":{"gid":7542,"name":"c","rank":0,"incarnation":148,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6835/879531976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":879531976},{"type":"v1","addr":"172.21.15.116:6835","nonce":879531976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7542,"qdb_cluster":[7542]},"id":2}]} 2024-04-25T00:19:30.520 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 152 2024-04-25T00:19:30.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.522+0000 7f7225428640 1 -- 172.21.15.116:0/2673205529 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f72000678c0 msgr2=0x7f7200069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:30.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.522+0000 7f7225428640 1 --2- 172.21.15.116:0/2673205529 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f72000678c0 0x7f7200069d80 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f7220144920 tx=0x7f7208008040 comp rx=0 tx=0).stop 2024-04-25T00:19:30.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.522+0000 7f7225428640 1 -- 172.21.15.116:0/2673205529 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72200a94a0 msgr2=0x7f72200aa350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:30.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.522+0000 7f7225428640 1 --2- 172.21.15.116:0/2673205529 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72200a94a0 0x7f72200aa350 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f721000d5f0 tx=0x7f721000dab0 comp rx=0 tx=0).stop 2024-04-25T00:19:30.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.522+0000 7f7225428640 1 -- 172.21.15.116:0/2673205529 shutdown_connections 2024-04-25T00:19:30.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.522+0000 7f7225428640 1 --2- 172.21.15.116:0/2673205529 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f72000678c0 0x7f7200069d80 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:30.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.522+0000 7f7225428640 1 --2- 172.21.15.116:0/2673205529 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7220150d20 0x7f72200724a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:30.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.522+0000 7f7225428640 1 --2- 172.21.15.116:0/2673205529 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f72200c2f50 0x7f72201528b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:30.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.522+0000 7f7225428640 1 --2- 172.21.15.116:0/2673205529 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f72200a94a0 0x7f72200aa350 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:30.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.522+0000 7f7225428640 1 -- 172.21.15.116:0/2673205529 >> 172.21.15.116:0/2673205529 conn(0x7f72200c3b00 msgr2=0x7f72200abe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:19:30.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.522+0000 7f7225428640 1 -- 172.21.15.116:0/2673205529 shutdown_connections 2024-04-25T00:19:30.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:30.522+0000 7f7225428640 1 -- 172.21.15.116:0/2673205529 wait complete. 2024-04-25T00:19:30.539 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:19:32.540 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:19:32.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.670+0000 7f6283bec640 1 Processor -- start 2024-04-25T00:19:32.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.670+0000 7f6283bec640 1 -- start start 2024-04-25T00:19:32.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.674+0000 7f6283bec640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f627c13e310 0x7f627c0c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:32.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.674+0000 7f6283bec640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f627c0c2f50 0x7f627c1559c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:32.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.674+0000 7f6283bec640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f627c13ed60 0x7f627c155f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:32.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.674+0000 7f6283bec640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f627c070d40 con 0x7f627c13ed60 2024-04-25T00:19:32.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.674+0000 7f6283bec640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f627c071020 con 0x7f627c0c2f50 2024-04-25T00:19:32.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.674+0000 7f6283bec640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f627c06d200 con 0x7f627c13e310 2024-04-25T00:19:32.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.674+0000 7f6282189640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f627c13ed60 0x7f627c155f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:32.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.674+0000 7f6282189640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f627c13ed60 0x7f627c155f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59152/0 (socket says 172.21.15.116:59152) 2024-04-25T00:19:32.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.674+0000 7f6282189640 1 -- 172.21.15.116:0/2880167721 learned_addr learned my addr 172.21.15.116:0/2880167721 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:19:32.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.674+0000 7f6281988640 1 --2- 172.21.15.116:0/2880167721 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f627c13e310 0x7f627c0c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:32.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.674+0000 7f6281187640 1 --2- 172.21.15.116:0/2880167721 >> v2:172.21.15.174:3300/0 conn(0x7f627c0c2f50 0x7f627c1559c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:32.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.674+0000 7f6282189640 1 -- 172.21.15.116:0/2880167721 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f627c13e310 msgr2=0x7f627c0c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:19:32.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.674+0000 7f6282189640 1 --2- 172.21.15.116:0/2880167721 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f627c13e310 0x7f627c0c2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:32.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.674+0000 7f6282189640 1 -- 172.21.15.116:0/2880167721 >> v2:172.21.15.174:3300/0 conn(0x7f627c0c2f50 msgr2=0x7f627c1559c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:19:32.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.674+0000 7f6282189640 1 --2- 172.21.15.116:0/2880167721 >> v2:172.21.15.174:3300/0 conn(0x7f627c0c2f50 0x7f627c1559c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:32.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.674+0000 7f6282189640 1 -- 172.21.15.116:0/2880167721 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f627c06cb50 con 0x7f627c13ed60 2024-04-25T00:19:32.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.674+0000 7f6281988640 1 --2- 172.21.15.116:0/2880167721 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f627c13e310 0x7f627c0c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:19:32.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.674+0000 7f6281187640 1 --2- 172.21.15.116:0/2880167721 >> v2:172.21.15.174:3300/0 conn(0x7f627c0c2f50 0x7f627c1559c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:19:32.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.674+0000 7f6282189640 1 --2- 172.21.15.116:0/2880167721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f627c13ed60 0x7f627c155f00 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f627c070710 tx=0x7f627401c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2be501f76034e85e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:32.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.674+0000 7f6280986640 1 -- 172.21.15.116:0/2880167721 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62740a5660 con 0x7f627c13ed60 2024-04-25T00:19:32.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.674+0000 7f6280986640 1 -- 172.21.15.116:0/2880167721 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62740a5800 con 0x7f627c13ed60 2024-04-25T00:19:32.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.674+0000 7f6280986640 1 -- 172.21.15.116:0/2880167721 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62740a5bd0 con 0x7f627c13ed60 2024-04-25T00:19:32.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.678+0000 7f6283bec640 1 -- 172.21.15.116:0/2880167721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f627c13ed60 msgr2=0x7f627c155f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:32.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.678+0000 7f6283bec640 1 --2- 172.21.15.116:0/2880167721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f627c13ed60 0x7f627c155f00 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f627c070710 tx=0x7f627401c8d0 comp rx=0 tx=0).stop 2024-04-25T00:19:32.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.678+0000 7f6283bec640 1 -- 172.21.15.116:0/2880167721 shutdown_connections 2024-04-25T00:19:32.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.678+0000 7f6283bec640 1 --2- 172.21.15.116:0/2880167721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f627c13ed60 0x7f627c155f00 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:32.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.678+0000 7f6283bec640 1 --2- 172.21.15.116:0/2880167721 >> v2:172.21.15.174:3300/0 conn(0x7f627c0c2f50 0x7f627c1559c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:32.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.678+0000 7f6283bec640 1 --2- 172.21.15.116:0/2880167721 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f627c13e310 0x7f627c0c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:32.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.678+0000 7f6283bec640 1 -- 172.21.15.116:0/2880167721 >> 172.21.15.116:0/2880167721 conn(0x7f627c0c3b00 msgr2=0x7f627c0c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:19:32.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.678+0000 7f6283bec640 1 -- 172.21.15.116:0/2880167721 shutdown_connections 2024-04-25T00:19:32.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.678+0000 7f6283bec640 1 -- 172.21.15.116:0/2880167721 wait complete. 2024-04-25T00:19:32.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.678+0000 7f6283bec640 1 Processor -- start 2024-04-25T00:19:32.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.678+0000 7f6283bec640 1 -- start start 2024-04-25T00:19:32.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.678+0000 7f6283bec640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f627c0c2f50 0x7f627c131cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:32.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.678+0000 7f6283bec640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f627c13e310 0x7f627c132210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:32.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.678+0000 7f6283bec640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f627c13ed60 0x7f627c1527f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:32.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.678+0000 7f6283bec640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f627c159650 con 0x7f627c13e310 2024-04-25T00:19:32.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.678+0000 7f6283bec640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f627c156650 con 0x7f627c0c2f50 2024-04-25T00:19:32.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.678+0000 7f6283bec640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f627c1597d0 con 0x7f627c13ed60 2024-04-25T00:19:32.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.678+0000 7f6281187640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f627c13e310 0x7f627c132210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:32.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.678+0000 7f6281187640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f627c13e310 0x7f627c132210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59166/0 (socket says 172.21.15.116:59166) 2024-04-25T00:19:32.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.678+0000 7f6281187640 1 -- 172.21.15.116:0/3152204782 learned_addr learned my addr 172.21.15.116:0/3152204782 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:19:32.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.678+0000 7f6281988640 1 --2- 172.21.15.116:0/3152204782 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f627c0c2f50 0x7f627c131cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:32.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.678+0000 7f6282189640 1 --2- 172.21.15.116:0/3152204782 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f627c13ed60 0x7f627c1527f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:32.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.678+0000 7f6281187640 1 -- 172.21.15.116:0/3152204782 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f627c13ed60 msgr2=0x7f627c1527f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:32.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.678+0000 7f6281187640 1 --2- 172.21.15.116:0/3152204782 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f627c13ed60 0x7f627c1527f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:32.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.678+0000 7f6281187640 1 -- 172.21.15.116:0/3152204782 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f627c0c2f50 msgr2=0x7f627c131cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:32.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.678+0000 7f6281187640 1 --2- 172.21.15.116:0/3152204782 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f627c0c2f50 0x7f627c131cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:32.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.678+0000 7f6281187640 1 -- 172.21.15.116:0/3152204782 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f627c157900 con 0x7f627c13e310 2024-04-25T00:19:32.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.678+0000 7f6281187640 1 --2- 172.21.15.116:0/3152204782 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f627c13e310 0x7f627c132210 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f626c00c7c0 tx=0x7f626c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:32.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.678+0000 7f62727fc640 1 -- 172.21.15.116:0/3152204782 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f626c015070 con 0x7f627c13e310 2024-04-25T00:19:32.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.678+0000 7f62727fc640 1 -- 172.21.15.116:0/3152204782 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f626c007a30 con 0x7f627c13e310 2024-04-25T00:19:32.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.678+0000 7f62727fc640 1 -- 172.21.15.116:0/3152204782 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f626c007e00 con 0x7f627c13e310 2024-04-25T00:19:32.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.682+0000 7f6283bec640 1 -- 172.21.15.116:0/3152204782 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f627c156a10 con 0x7f627c13e310 2024-04-25T00:19:32.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.682+0000 7f6283bec640 1 -- 172.21.15.116:0/3152204782 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f627c156fd0 con 0x7f627c13e310 2024-04-25T00:19:32.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.682+0000 7f6283bec640 1 -- 172.21.15.116:0/3152204782 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f627c068350 con 0x7f627c13e310 2024-04-25T00:19:32.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.682+0000 7f62727fc640 1 -- 172.21.15.116:0/3152204782 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f626c018070 con 0x7f627c13e310 2024-04-25T00:19:32.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.682+0000 7f62727fc640 1 --2- 172.21.15.116:0/3152204782 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f62600678c0 0x7f6260069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:32.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.682+0000 7f62727fc640 1 -- 172.21.15.116:0/3152204782 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 9160+0+0 (secure 0 0 0) 0x7f626c086930 con 0x7f627c13e310 2024-04-25T00:19:32.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.682+0000 7f6281988640 1 --2- 172.21.15.116:0/3152204782 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f62600678c0 0x7f6260069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:32.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.682+0000 7f6281988640 1 --2- 172.21.15.116:0/3152204782 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f62600678c0 0x7f6260069d80 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f627800ab60 tx=0x7f6278008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:32.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.686+0000 7f62727fc640 1 -- 172.21.15.116:0/3152204782 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f626c055ff0 con 0x7f627c13e310 2024-04-25T00:19:32.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.882+0000 7f6283bec640 1 -- 172.21.15.116:0/3152204782 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f627c132750 con 0x7f627c13e310 2024-04-25T00:19:32.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.882+0000 7f62727fc640 1 -- 172.21.15.116:0/3152204782 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) ==== 95+0+5669 (secure 0 0 0) 0x7f626c05a2f0 con 0x7f627c13e310 2024-04-25T00:19:32.882 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:19:32.882 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":154,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:19:31.107899+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6586":{"gid":6586,"name":"d","rank":0,"incarnation":153,"state":"up:reconnect","state_seq":5,"addr":"172.21.15.174:6837/991183935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":991183935},{"type":"v1","addr":"172.21.15.174:6837","nonce":991183935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:19:08.844382+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7542},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7542":{"gid":7542,"name":"c","rank":0,"incarnation":148,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6835/879531976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":879531976},{"type":"v1","addr":"172.21.15.116:6835","nonce":879531976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7542,"qdb_cluster":[7542]},"id":2}]} 2024-04-25T00:19:32.882 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 154 2024-04-25T00:19:32.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.882+0000 7f6283bec640 1 -- 172.21.15.116:0/3152204782 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f62600678c0 msgr2=0x7f6260069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:32.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.882+0000 7f6283bec640 1 --2- 172.21.15.116:0/3152204782 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f62600678c0 0x7f6260069d80 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f627800ab60 tx=0x7f6278008040 comp rx=0 tx=0).stop 2024-04-25T00:19:32.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.882+0000 7f6283bec640 1 -- 172.21.15.116:0/3152204782 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f627c13e310 msgr2=0x7f627c132210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:32.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.882+0000 7f6283bec640 1 --2- 172.21.15.116:0/3152204782 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f627c13e310 0x7f627c132210 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f626c00c7c0 tx=0x7f626c00cc80 comp rx=0 tx=0).stop 2024-04-25T00:19:32.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.882+0000 7f6283bec640 1 -- 172.21.15.116:0/3152204782 shutdown_connections 2024-04-25T00:19:32.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.882+0000 7f6283bec640 1 --2- 172.21.15.116:0/3152204782 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f62600678c0 0x7f6260069d80 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:32.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.882+0000 7f6283bec640 1 --2- 172.21.15.116:0/3152204782 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f627c13ed60 0x7f627c1527f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:32.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.882+0000 7f6283bec640 1 --2- 172.21.15.116:0/3152204782 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f627c13e310 0x7f627c132210 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:32.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.882+0000 7f6283bec640 1 --2- 172.21.15.116:0/3152204782 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f627c0c2f50 0x7f627c131cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:32.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.882+0000 7f6283bec640 1 -- 172.21.15.116:0/3152204782 >> 172.21.15.116:0/3152204782 conn(0x7f627c0c3b00 msgr2=0x7f627c0c1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:19:32.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.886+0000 7f6283bec640 1 -- 172.21.15.116:0/3152204782 shutdown_connections 2024-04-25T00:19:32.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:32.886+0000 7f6283bec640 1 -- 172.21.15.116:0/3152204782 wait complete. 2024-04-25T00:19:32.901 INFO:tasks.mds_thrash.fs.[a]:mds.a down, removed from mdsmap 2024-04-25T00:19:32.901 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-04-25T00:19:32.901 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:19:33.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.010+0000 7f2e47f99640 1 Processor -- start 2024-04-25T00:19:33.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.010+0000 7f2e47f99640 1 -- start start 2024-04-25T00:19:33.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.014+0000 7f2e47f99640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f2e40150d20 0x7f2e400c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:33.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.014+0000 7f2e47f99640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e400c2f50 0x7f2e401538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:33.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.014+0000 7f2e47f99640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2e400a94a0 0x7f2e40153e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:33.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.014+0000 7f2e47f99640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f2e400c3910 con 0x7f2e400a94a0 2024-04-25T00:19:33.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.014+0000 7f2e47f99640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f2e400c16e0 con 0x7f2e40150d20 2024-04-25T00:19:33.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.014+0000 7f2e47f99640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f2e400c1400 con 0x7f2e400c2f50 2024-04-25T00:19:33.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.014+0000 7f2e46536640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2e400a94a0 0x7f2e40153e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:33.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.014+0000 7f2e46536640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2e400a94a0 0x7f2e40153e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59174/0 (socket says 172.21.15.116:59174) 2024-04-25T00:19:33.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.014+0000 7f2e46536640 1 -- 172.21.15.116:0/3914331718 learned_addr learned my addr 172.21.15.116:0/3914331718 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:19:33.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.014+0000 7f2e45534640 1 --2- 172.21.15.116:0/3914331718 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e400c2f50 0x7f2e401538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:33.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.014+0000 7f2e45d35640 1 --2- 172.21.15.116:0/3914331718 >> v2:172.21.15.174:3300/0 conn(0x7f2e40150d20 0x7f2e400c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:33.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.014+0000 7f2e46536640 1 -- 172.21.15.116:0/3914331718 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e400c2f50 msgr2=0x7f2e401538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:19:33.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.014+0000 7f2e46536640 1 --2- 172.21.15.116:0/3914331718 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e400c2f50 0x7f2e401538e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:33.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.014+0000 7f2e46536640 1 -- 172.21.15.116:0/3914331718 >> v2:172.21.15.174:3300/0 conn(0x7f2e40150d20 msgr2=0x7f2e400c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:19:33.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.014+0000 7f2e46536640 1 --2- 172.21.15.116:0/3914331718 >> v2:172.21.15.174:3300/0 conn(0x7f2e40150d20 0x7f2e400c2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:33.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.014+0000 7f2e46536640 1 -- 172.21.15.116:0/3914331718 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e4006e660 con 0x7f2e400a94a0 2024-04-25T00:19:33.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.014+0000 7f2e46536640 1 --2- 172.21.15.116:0/3914331718 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e400a94a0 0x7f2e40153e20 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f2e40073550 tx=0x7f2e3001c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6207e77c61fc54a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:33.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.014+0000 7f2e44d33640 1 -- 172.21.15.116:0/3914331718 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e300a5740 con 0x7f2e400a94a0 2024-04-25T00:19:33.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.014+0000 7f2e44d33640 1 -- 172.21.15.116:0/3914331718 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e300a58e0 con 0x7f2e400a94a0 2024-04-25T00:19:33.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.014+0000 7f2e44d33640 1 -- 172.21.15.116:0/3914331718 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e300a5cb0 con 0x7f2e400a94a0 2024-04-25T00:19:33.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.014+0000 7f2e47f99640 1 -- 172.21.15.116:0/3914331718 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e400a94a0 msgr2=0x7f2e40153e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:33.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.014+0000 7f2e47f99640 1 --2- 172.21.15.116:0/3914331718 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e400a94a0 0x7f2e40153e20 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f2e40073550 tx=0x7f2e3001c9f0 comp rx=0 tx=0).stop 2024-04-25T00:19:33.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.014+0000 7f2e47f99640 1 -- 172.21.15.116:0/3914331718 shutdown_connections 2024-04-25T00:19:33.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.014+0000 7f2e47f99640 1 --2- 172.21.15.116:0/3914331718 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e400a94a0 0x7f2e40153e20 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:33.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.014+0000 7f2e47f99640 1 --2- 172.21.15.116:0/3914331718 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e400c2f50 0x7f2e401538e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:33.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.014+0000 7f2e47f99640 1 --2- 172.21.15.116:0/3914331718 >> v2:172.21.15.174:3300/0 conn(0x7f2e40150d20 0x7f2e400c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:33.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.014+0000 7f2e47f99640 1 -- 172.21.15.116:0/3914331718 >> 172.21.15.116:0/3914331718 conn(0x7f2e400c3b00 msgr2=0x7f2e400c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:19:33.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.014+0000 7f2e47f99640 1 -- 172.21.15.116:0/3914331718 shutdown_connections 2024-04-25T00:19:33.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.014+0000 7f2e47f99640 1 -- 172.21.15.116:0/3914331718 wait complete. 2024-04-25T00:19:33.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.018+0000 7f2e47f99640 1 Processor -- start 2024-04-25T00:19:33.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.018+0000 7f2e47f99640 1 -- start start 2024-04-25T00:19:33.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.018+0000 7f2e47f99640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e400a94a0 0x7f2e400c0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:33.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.018+0000 7f2e47f99640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e400c2f50 0x7f2e40072670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:33.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.018+0000 7f2e47f99640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e40150d20 0x7f2e4013d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:33.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.018+0000 7f2e47f99640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2e40157340 con 0x7f2e400c2f50 2024-04-25T00:19:33.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.018+0000 7f2e47f99640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2e40154570 con 0x7f2e400a94a0 2024-04-25T00:19:33.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.018+0000 7f2e47f99640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f2e401574c0 con 0x7f2e40150d20 2024-04-25T00:19:33.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.018+0000 7f2e45534640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e400c2f50 0x7f2e40072670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:33.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.018+0000 7f2e45534640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e400c2f50 0x7f2e40072670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:59188/0 (socket says 172.21.15.116:59188) 2024-04-25T00:19:33.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.018+0000 7f2e45534640 1 -- 172.21.15.116:0/4155824656 learned_addr learned my addr 172.21.15.116:0/4155824656 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:19:33.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.018+0000 7f2e45d35640 1 --2- 172.21.15.116:0/4155824656 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e400a94a0 0x7f2e400c0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:33.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.018+0000 7f2e46536640 1 --2- 172.21.15.116:0/4155824656 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e40150d20 0x7f2e4013d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:33.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.018+0000 7f2e45534640 1 -- 172.21.15.116:0/4155824656 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e40150d20 msgr2=0x7f2e4013d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:33.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.018+0000 7f2e45534640 1 --2- 172.21.15.116:0/4155824656 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e40150d20 0x7f2e4013d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:33.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.018+0000 7f2e45534640 1 -- 172.21.15.116:0/4155824656 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e400a94a0 msgr2=0x7f2e400c0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:33.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.018+0000 7f2e45534640 1 --2- 172.21.15.116:0/4155824656 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e400a94a0 0x7f2e400c0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:33.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.018+0000 7f2e45534640 1 -- 172.21.15.116:0/4155824656 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e4014c8a0 con 0x7f2e400c2f50 2024-04-25T00:19:33.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.018+0000 7f2e45534640 1 --2- 172.21.15.116:0/4155824656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e400c2f50 0x7f2e40072670 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f2e3400e8e0 tx=0x7f2e3400eda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:33.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.018+0000 7f2e3e7fc640 1 -- 172.21.15.116:0/4155824656 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e34017070 con 0x7f2e400c2f50 2024-04-25T00:19:33.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.018+0000 7f2e3e7fc640 1 -- 172.21.15.116:0/4155824656 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e3400ca00 con 0x7f2e400c2f50 2024-04-25T00:19:33.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.018+0000 7f2e47f99640 1 -- 172.21.15.116:0/4155824656 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e4014cb80 con 0x7f2e400c2f50 2024-04-25T00:19:33.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.018+0000 7f2e3e7fc640 1 -- 172.21.15.116:0/4155824656 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e3400cdd0 con 0x7f2e400c2f50 2024-04-25T00:19:33.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.018+0000 7f2e47f99640 1 -- 172.21.15.116:0/4155824656 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e400c0bd0 con 0x7f2e400c2f50 2024-04-25T00:19:33.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.018+0000 7f2e47f99640 1 -- 172.21.15.116:0/4155824656 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e40068350 con 0x7f2e400c2f50 2024-04-25T00:19:33.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.022+0000 7f2e3e7fc640 1 -- 172.21.15.116:0/4155824656 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2e3401a070 con 0x7f2e400c2f50 2024-04-25T00:19:33.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.022+0000 7f2e3e7fc640 1 --2- 172.21.15.116:0/4155824656 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2e2c0678c0 0x7f2e2c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:33.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.022+0000 7f2e3e7fc640 1 -- 172.21.15.116:0/4155824656 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 9160+0+0 (secure 0 0 0) 0x7f2e34086d50 con 0x7f2e400c2f50 2024-04-25T00:19:33.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.022+0000 7f2e45d35640 1 --2- 172.21.15.116:0/4155824656 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2e2c0678c0 0x7f2e2c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:33.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.022+0000 7f2e45d35640 1 --2- 172.21.15.116:0/4155824656 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2e2c0678c0 0x7f2e2c069d80 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f2e38009ac0 tx=0x7f2e38009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:33.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.026+0000 7f2e3e7fc640 1 -- 172.21.15.116:0/4155824656 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f2e3408a030 con 0x7f2e400c2f50 2024-04-25T00:19:33.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.222+0000 7f2e47f99640 1 -- 172.21.15.116:0/4155824656 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2e400aac30 con 0x7f2e400c2f50 2024-04-25T00:19:33.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.222+0000 7f2e3e7fc640 1 -- 172.21.15.116:0/4155824656 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 155 v155) ==== 95+0+5666 (secure 0 0 0) 0x7f2e34056500 con 0x7f2e400c2f50 2024-04-25T00:19:33.222 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:19:33.222 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":155,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":155,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:19:31.980845+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6586":{"gid":6586,"name":"d","rank":0,"incarnation":153,"state":"up:rejoin","state_seq":6,"addr":"172.21.15.174:6837/991183935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":991183935},{"type":"v1","addr":"172.21.15.174:6837","nonce":991183935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:19:08.844382+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7542},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7542":{"gid":7542,"name":"c","rank":0,"incarnation":148,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6835/879531976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":879531976},{"type":"v1","addr":"172.21.15.116:6835","nonce":879531976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7542,"qdb_cluster":[7542]},"id":2}]} 2024-04-25T00:19:33.222 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 155 2024-04-25T00:19:33.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.222+0000 7f2e47f99640 1 -- 172.21.15.116:0/4155824656 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2e2c0678c0 msgr2=0x7f2e2c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:33.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.222+0000 7f2e47f99640 1 --2- 172.21.15.116:0/4155824656 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2e2c0678c0 0x7f2e2c069d80 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f2e38009ac0 tx=0x7f2e38009190 comp rx=0 tx=0).stop 2024-04-25T00:19:33.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.222+0000 7f2e47f99640 1 -- 172.21.15.116:0/4155824656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e400c2f50 msgr2=0x7f2e40072670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:33.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.222+0000 7f2e47f99640 1 --2- 172.21.15.116:0/4155824656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e400c2f50 0x7f2e40072670 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f2e3400e8e0 tx=0x7f2e3400eda0 comp rx=0 tx=0).stop 2024-04-25T00:19:33.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.226+0000 7f2e47f99640 1 -- 172.21.15.116:0/4155824656 shutdown_connections 2024-04-25T00:19:33.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.226+0000 7f2e47f99640 1 --2- 172.21.15.116:0/4155824656 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2e2c0678c0 0x7f2e2c069d80 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:33.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.226+0000 7f2e47f99640 1 --2- 172.21.15.116:0/4155824656 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e40150d20 0x7f2e4013d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:33.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.226+0000 7f2e47f99640 1 --2- 172.21.15.116:0/4155824656 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e400c2f50 0x7f2e40072670 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:33.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.226+0000 7f2e47f99640 1 --2- 172.21.15.116:0/4155824656 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e400a94a0 0x7f2e400c0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:33.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.226+0000 7f2e47f99640 1 -- 172.21.15.116:0/4155824656 >> 172.21.15.116:0/4155824656 conn(0x7f2e400c3b00 msgr2=0x7f2e400a9f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:19:33.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.226+0000 7f2e47f99640 1 -- 172.21.15.116:0/4155824656 shutdown_connections 2024-04-25T00:19:33.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:33.226+0000 7f2e47f99640 1 -- 172.21.15.116:0/4155824656 wait complete. 2024-04-25T00:19:33.242 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 155, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:04.678737+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 155, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6586": { "addr": "172.21.15.174:6837/991183935", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6836", "nonce": 991183935, "type": "v2" }, { "addr": "172.21.15.174:6837", "nonce": 991183935, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6586, "incarnation": 153, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:rejoin", "state_seq": 6 } }, "last_failure": 0, "last_failure_osd_epoch": 57, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T00:19:31.980845+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6586 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:18.825887+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 151, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_7542": { "addr": "172.21.15.116:6835/879531976", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6834", "nonce": 879531976, "type": "v2" }, { "addr": "172.21.15.116:6835", "nonce": 879531976, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7542, "incarnation": 148, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 9 } }, "last_failure": 0, "last_failure_osd_epoch": 56, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-04-25T00:19:08.844382+0000", "qdb_cluster": [ 7542 ], "qdb_leader": 7542, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7542 } } } ], "standbys": [ { "addr": "172.21.15.174:6835/4112145782", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6834", "nonce": 4112145782, "type": "v2" }, { "addr": "172.21.15.174:6835", "nonce": 4112145782, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 112, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5582, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-25T00:19:35.243 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:19:35.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.398+0000 7fb3aa7fc640 1 Processor -- start 2024-04-25T00:19:35.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.398+0000 7fb3aa7fc640 1 -- start start 2024-04-25T00:19:35.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.398+0000 7fb3aa7fc640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fb3a4150d10 0x7fb3a40c5010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:35.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.398+0000 7fb3aa7fc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb3a40c5550 0x7fb3a4153a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:35.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.398+0000 7fb3aa7fc640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3a40c4950 con 0x7fb3a4072ca0 2024-04-25T00:19:35.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.398+0000 7fb3aa7fc640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fb3a41546d0 con 0x7fb3a40c5550 2024-04-25T00:19:35.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.398+0000 7fb3aa7fc640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fb3a41510f0 con 0x7fb3a4150d10 2024-04-25T00:19:35.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.398+0000 7fb39bfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb3a40c5550 0x7fb3a4153a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:35.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.398+0000 7fb39bfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb3a40c5550 0x7fb3a4153a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35458/0 (socket says 172.21.15.116:35458) 2024-04-25T00:19:35.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.398+0000 7fb39bfff640 1 -- 172.21.15.116:0/3067225647 learned_addr learned my addr 172.21.15.116:0/3067225647 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:19:35.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.398+0000 7fb3a3fff640 1 --2- 172.21.15.116:0/3067225647 >> v2:172.21.15.174:3300/0 conn(0x7fb3a4150d10 0x7fb3a40c5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:35.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.402+0000 7fb39bfff640 1 -- 172.21.15.116:0/3067225647 >> v2:172.21.15.174:3300/0 conn(0x7fb3a4150d10 msgr2=0x7fb3a40c5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:19:35.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.402+0000 7fb39bfff640 1 --2- 172.21.15.116:0/3067225647 >> v2:172.21.15.174:3300/0 conn(0x7fb3a4150d10 0x7fb3a40c5010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:35.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.402+0000 7fb39bfff640 1 -- 172.21.15.116:0/3067225647 >> v1:172.21.15.174:6789/0 conn(0x7fb3a4072ca0 legacy=0x7fb3a4153fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:35.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.402+0000 7fb39bfff640 1 -- 172.21.15.116:0/3067225647 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3a4157a20 con 0x7fb3a40c5550 2024-04-25T00:19:35.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.402+0000 7fb39bfff640 1 --2- 172.21.15.116:0/3067225647 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3a40c5550 0x7fb3a4153a80 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7fb3a40ad5e0 tx=0x7fb38c01c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfddb13d67ca8646 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:35.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.402+0000 7fb3a37fe640 1 -- 172.21.15.116:0/3067225647 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb38c0ab600 con 0x7fb3a40c5550 2024-04-25T00:19:35.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.402+0000 7fb3a37fe640 1 -- 172.21.15.116:0/3067225647 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb38c0b0070 con 0x7fb3a40c5550 2024-04-25T00:19:35.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.402+0000 7fb3a37fe640 1 -- 172.21.15.116:0/3067225647 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb38c0ab9c0 con 0x7fb3a40c5550 2024-04-25T00:19:35.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.402+0000 7fb3aa7fc640 1 -- 172.21.15.116:0/3067225647 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3a40c5550 msgr2=0x7fb3a4153a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:35.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.402+0000 7fb3aa7fc640 1 --2- 172.21.15.116:0/3067225647 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3a40c5550 0x7fb3a4153a80 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7fb3a40ad5e0 tx=0x7fb38c01c400 comp rx=0 tx=0).stop 2024-04-25T00:19:35.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.406+0000 7fb3aa7fc640 1 -- 172.21.15.116:0/3067225647 shutdown_connections 2024-04-25T00:19:35.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.406+0000 7fb3aa7fc640 1 --2- 172.21.15.116:0/3067225647 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3a40c5550 0x7fb3a4153a80 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:35.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.406+0000 7fb3aa7fc640 1 --2- 172.21.15.116:0/3067225647 >> v2:172.21.15.174:3300/0 conn(0x7fb3a4150d10 0x7fb3a40c5010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:35.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.406+0000 7fb3aa7fc640 1 -- 172.21.15.116:0/3067225647 >> 172.21.15.116:0/3067225647 conn(0x7fb3a406c720 msgr2=0x7fb3a406cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:19:35.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.406+0000 7fb3aa7fc640 1 -- 172.21.15.116:0/3067225647 shutdown_connections 2024-04-25T00:19:35.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.406+0000 7fb3aa7fc640 1 -- 172.21.15.116:0/3067225647 wait complete. 2024-04-25T00:19:35.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.406+0000 7fb3aa7fc640 1 Processor -- start 2024-04-25T00:19:35.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.406+0000 7fb3aa7fc640 1 -- start start 2024-04-25T00:19:35.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.406+0000 7fb3aa7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3a4072ca0 0x7fb3a4154b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:35.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.406+0000 7fb3aa7fc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb3a40c5550 0x7fb3a415c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:35.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.406+0000 7fb3aa7fc640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb3a4150d10 0x7fb3a4156580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:35.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.406+0000 7fb3aa7fc640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb3a40c2640 con 0x7fb3a4072ca0 2024-04-25T00:19:35.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.406+0000 7fb3aa7fc640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb3a41576a0 con 0x7fb3a40c5550 2024-04-25T00:19:35.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.406+0000 7fb3aa7fc640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fb3a40c27c0 con 0x7fb3a4150d10 2024-04-25T00:19:35.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.406+0000 7fb3a3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3a4072ca0 0x7fb3a4154b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:35.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.406+0000 7fb3a3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3a4072ca0 0x7fb3a4154b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35470/0 (socket says 172.21.15.116:35470) 2024-04-25T00:19:35.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.406+0000 7fb3a3fff640 1 -- 172.21.15.116:0/1958987026 learned_addr learned my addr 172.21.15.116:0/1958987026 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:19:35.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.406+0000 7fb39bfff640 1 --2- 172.21.15.116:0/1958987026 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb3a40c5550 0x7fb3a415c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:35.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.406+0000 7fb3a8d99640 1 --2- 172.21.15.116:0/1958987026 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb3a4150d10 0x7fb3a4156580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:35.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.406+0000 7fb3a3fff640 1 -- 172.21.15.116:0/1958987026 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb3a4150d10 msgr2=0x7fb3a4156580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:35.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.406+0000 7fb3a3fff640 1 --2- 172.21.15.116:0/1958987026 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb3a4150d10 0x7fb3a4156580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:35.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.406+0000 7fb3a3fff640 1 -- 172.21.15.116:0/1958987026 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb3a40c5550 msgr2=0x7fb3a415c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:35.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.406+0000 7fb3a3fff640 1 --2- 172.21.15.116:0/1958987026 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb3a40c5550 0x7fb3a415c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:35.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.406+0000 7fb3a3fff640 1 -- 172.21.15.116:0/1958987026 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3a41550b0 con 0x7fb3a4072ca0 2024-04-25T00:19:35.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.406+0000 7fb3a3fff640 1 --2- 172.21.15.116:0/1958987026 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3a4072ca0 0x7fb3a4154b70 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7fb39400d5f0 tx=0x7fb39400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:35.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.406+0000 7fb3a17fa640 1 -- 172.21.15.116:0/1958987026 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3940125f0 con 0x7fb3a4072ca0 2024-04-25T00:19:35.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.406+0000 7fb3a17fa640 1 -- 172.21.15.116:0/1958987026 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb394019070 con 0x7fb3a4072ca0 2024-04-25T00:19:35.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.406+0000 7fb3aa7fc640 1 -- 172.21.15.116:0/1958987026 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3a41eeb90 con 0x7fb3a4072ca0 2024-04-25T00:19:35.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.406+0000 7fb3a17fa640 1 -- 172.21.15.116:0/1958987026 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3940129b0 con 0x7fb3a4072ca0 2024-04-25T00:19:35.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.406+0000 7fb3aa7fc640 1 -- 172.21.15.116:0/1958987026 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3a415cba0 con 0x7fb3a4072ca0 2024-04-25T00:19:35.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.410+0000 7fb3aa7fc640 1 -- 172.21.15.116:0/1958987026 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3a4068a60 con 0x7fb3a4072ca0 2024-04-25T00:19:35.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.410+0000 7fb3a17fa640 1 -- 172.21.15.116:0/1958987026 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb394012b50 con 0x7fb3a4072ca0 2024-04-25T00:19:35.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.410+0000 7fb3a17fa640 1 --2- 172.21.15.116:0/1958987026 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb3840678f0 0x7fb384069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:35.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.410+0000 7fb3a17fa640 1 -- 172.21.15.116:0/1958987026 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 9160+0+0 (secure 0 0 0) 0x7fb394086dc0 con 0x7fb3a4072ca0 2024-04-25T00:19:35.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.410+0000 7fb39bfff640 1 --2- 172.21.15.116:0/1958987026 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb3840678f0 0x7fb384069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:35.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.410+0000 7fb39bfff640 1 --2- 172.21.15.116:0/1958987026 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb3840678f0 0x7fb384069db0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fb38c0af000 tx=0x7fb38c0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:35.414 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.414+0000 7fb3a17fa640 1 -- 172.21.15.116:0/1958987026 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fb3a41550b0 con 0x7fb3a4072ca0 2024-04-25T00:19:35.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.642+0000 7fb3aa7fc640 1 -- 172.21.15.116:0/1958987026 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb3a4072a80 con 0x7fb3a4072ca0 2024-04-25T00:19:35.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.646+0000 7fb3a17fa640 1 -- 172.21.15.116:0/1958987026 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 156 v156) ==== 95+0+5673 (secure 0 0 0) 0x7fb394056bc0 con 0x7fb3a4072ca0 2024-04-25T00:19:35.645 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:19:35.645 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":156,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":156,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:19:33.982945+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6586":{"gid":6586,"name":"d","rank":0,"incarnation":153,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/991183935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":991183935},{"type":"v1","addr":"172.21.15.174:6837","nonce":991183935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6586,"qdb_cluster":[6586]},"id":1},{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:19:08.844382+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7542},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7542":{"gid":7542,"name":"c","rank":0,"incarnation":148,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6835/879531976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":879531976},{"type":"v1","addr":"172.21.15.116:6835","nonce":879531976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7542,"qdb_cluster":[7542]},"id":2}]} 2024-04-25T00:19:35.645 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 156 2024-04-25T00:19:35.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.646+0000 7fb3aa7fc640 1 -- 172.21.15.116:0/1958987026 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb3840678f0 msgr2=0x7fb384069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:35.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.646+0000 7fb3aa7fc640 1 --2- 172.21.15.116:0/1958987026 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb3840678f0 0x7fb384069db0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fb38c0af000 tx=0x7fb38c0a7000 comp rx=0 tx=0).stop 2024-04-25T00:19:35.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.646+0000 7fb3aa7fc640 1 -- 172.21.15.116:0/1958987026 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3a4072ca0 msgr2=0x7fb3a4154b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:35.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.646+0000 7fb3aa7fc640 1 --2- 172.21.15.116:0/1958987026 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3a4072ca0 0x7fb3a4154b70 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7fb39400d5f0 tx=0x7fb39400dab0 comp rx=0 tx=0).stop 2024-04-25T00:19:35.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.646+0000 7fb3aa7fc640 1 -- 172.21.15.116:0/1958987026 shutdown_connections 2024-04-25T00:19:35.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.646+0000 7fb3aa7fc640 1 --2- 172.21.15.116:0/1958987026 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb3840678f0 0x7fb384069db0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:35.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.646+0000 7fb3aa7fc640 1 --2- 172.21.15.116:0/1958987026 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb3a4150d10 0x7fb3a4156580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:35.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.646+0000 7fb3aa7fc640 1 --2- 172.21.15.116:0/1958987026 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb3a40c5550 0x7fb3a415c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:35.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.646+0000 7fb3aa7fc640 1 --2- 172.21.15.116:0/1958987026 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb3a4072ca0 0x7fb3a4154b70 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:35.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.646+0000 7fb3aa7fc640 1 -- 172.21.15.116:0/1958987026 >> 172.21.15.116:0/1958987026 conn(0x7fb3a406c720 msgr2=0x7fb3a406cb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:19:35.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.646+0000 7fb3aa7fc640 1 -- 172.21.15.116:0/1958987026 shutdown_connections 2024-04-25T00:19:35.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:35.646+0000 7fb3aa7fc640 1 -- 172.21.15.116:0/1958987026 wait complete. 2024-04-25T00:19:35.668 INFO:tasks.mds_thrash.fs.[a]:mds.d has gained rank=0, replacing gid=7362 2024-04-25T00:19:35.669 INFO:tasks.mds_thrash.fs.[a]:waiting for 6 secs before reviving mds.a 2024-04-25T00:19:39.365 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:19:39.368 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:19:41.669 INFO:tasks.mds_thrash.fs.[a]:reviving mds.a 2024-04-25T00:19:41.670 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-25T00:19:41.670 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-25T00:19:41.673 INFO:tasks.ceph.mds.a:Started 2024-04-25T00:19:41.673 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:19:41.746 INFO:tasks.ceph.mds.a.smithi116.stdout:starting mds.a at 2024-04-25T00:19:41.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.814+0000 7f13fda83640 1 Processor -- start 2024-04-25T00:19:41.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.814+0000 7f13fda83640 1 -- start start 2024-04-25T00:19:41.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.814+0000 7f13fda83640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f13f80aeb20 0x7f13f80c9310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:41.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.814+0000 7f13fda83640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f13f8150f80 0x7f13f80c5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:41.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.814+0000 7f13fda83640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13f81535d0 con 0x7f13f80c9850 2024-04-25T00:19:41.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.814+0000 7f13fda83640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f13f8153850 con 0x7f13f80aeb20 2024-04-25T00:19:41.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.814+0000 7f13fda83640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f13f8153b30 con 0x7f13f8150f80 2024-04-25T00:19:41.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.814+0000 7f13effff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f13f80c9850 0x7f13f8150870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:53706/0 (socket says 172.21.15.116:53706) 2024-04-25T00:19:41.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.814+0000 7f13effff640 1 -- 172.21.15.116:0/1638336687 learned_addr learned my addr 172.21.15.116:0/1638336687 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:19:41.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.814+0000 7f13f7fff640 1 --2- 172.21.15.116:0/1638336687 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f13f8150f80 0x7f13f80c5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:41.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.814+0000 7f13f77fe640 1 --2- 172.21.15.116:0/1638336687 >> v2:172.21.15.174:3300/0 conn(0x7f13f80aeb20 0x7f13f80c9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:41.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.814+0000 7f13f6ffd640 1 -- 172.21.15.116:0/1638336687 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3000095079 0 0) 0x7f13e0001120 con 0x7f13f80c9850 2024-04-25T00:19:41.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.814+0000 7f13f6ffd640 1 -- 172.21.15.116:0/1638336687 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13e4003530 con 0x7f13f80c9850 2024-04-25T00:19:41.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.814+0000 7f13f7fff640 1 -- 172.21.15.116:0/1638336687 >> v2:172.21.15.174:3300/0 conn(0x7f13f80aeb20 msgr2=0x7f13f80c9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:19:41.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.814+0000 7f13f7fff640 1 --2- 172.21.15.116:0/1638336687 >> v2:172.21.15.174:3300/0 conn(0x7f13f80aeb20 0x7f13f80c9310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:41.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.814+0000 7f13f7fff640 1 -- 172.21.15.116:0/1638336687 >> v1:172.21.15.116:6789/0 conn(0x7f13f80c9850 legacy=0x7f13f8150870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:41.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.814+0000 7f13f6ffd640 1 -- 172.21.15.116:0/1638336687 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2262015283 0 0) 0x7f13e4003530 con 0x7f13f80c9850 2024-04-25T00:19:41.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.814+0000 7f13f7fff640 1 -- 172.21.15.116:0/1638336687 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13f8153e10 con 0x7f13f8150f80 2024-04-25T00:19:41.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.818+0000 7f13f7fff640 1 --2- 172.21.15.116:0/1638336687 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f13f8150f80 0x7f13f80c5270 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f13f0009d90 tx=0x7f13f001c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c4d2c784c10f36d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:41.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.818+0000 7f13f6ffd640 1 -- 172.21.15.116:0/1638336687 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13e0001850 con 0x7f13f8150f80 2024-04-25T00:19:41.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.818+0000 7f13f6ffd640 1 -- 172.21.15.116:0/1638336687 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13f00a6500 con 0x7f13f8150f80 2024-04-25T00:19:41.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.818+0000 7f13f6ffd640 1 -- 172.21.15.116:0/1638336687 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13f00a68b0 con 0x7f13f8150f80 2024-04-25T00:19:41.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.818+0000 7f13fda83640 1 -- 172.21.15.116:0/1638336687 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f13f8150f80 msgr2=0x7f13f80c5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:41.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.818+0000 7f13fda83640 1 --2- 172.21.15.116:0/1638336687 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f13f8150f80 0x7f13f80c5270 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f13f0009d90 tx=0x7f13f001c800 comp rx=0 tx=0).stop 2024-04-25T00:19:41.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.818+0000 7f13fda83640 1 -- 172.21.15.116:0/1638336687 shutdown_connections 2024-04-25T00:19:41.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.818+0000 7f13fda83640 1 --2- 172.21.15.116:0/1638336687 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f13f8150f80 0x7f13f80c5270 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:41.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.818+0000 7f13fda83640 1 --2- 172.21.15.116:0/1638336687 >> v2:172.21.15.174:3300/0 conn(0x7f13f80aeb20 0x7f13f80c9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:41.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.818+0000 7f13fda83640 1 -- 172.21.15.116:0/1638336687 >> 172.21.15.116:0/1638336687 conn(0x7f13f8074650 msgr2=0x7f13f8074a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:19:41.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.818+0000 7f13fda83640 1 -- 172.21.15.116:0/1638336687 shutdown_connections 2024-04-25T00:19:41.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.818+0000 7f13fda83640 1 -- 172.21.15.116:0/1638336687 wait complete. 2024-04-25T00:19:41.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.822+0000 7f13fda83640 1 Processor -- start 2024-04-25T00:19:41.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.822+0000 7f13fda83640 1 -- start start 2024-04-25T00:19:41.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.822+0000 7f13fda83640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f13f80aeb20 0x7f13f8142940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:41.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.822+0000 7f13fda83640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f13f80c9850 0x7f13f81405e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:41.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.822+0000 7f13fda83640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13f8150f80 0x7f13f8140b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:41.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.822+0000 7f13fda83640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f13f81600e0 con 0x7f13f8150f80 2024-04-25T00:19:41.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.822+0000 7f13fda83640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f13f815ff60 con 0x7f13f80c9850 2024-04-25T00:19:41.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.822+0000 7f13fda83640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f13f8160260 con 0x7f13f80aeb20 2024-04-25T00:19:41.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.822+0000 7f13f7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13f8150f80 0x7f13f8140b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:41.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.822+0000 7f13f7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13f8150f80 0x7f13f8140b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35482/0 (socket says 172.21.15.116:35482) 2024-04-25T00:19:41.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.822+0000 7f13f7fff640 1 -- 172.21.15.116:0/3722378720 learned_addr learned my addr 172.21.15.116:0/3722378720 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:19:41.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.822+0000 7f13f77fe640 1 --2- 172.21.15.116:0/3722378720 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f13f80aeb20 0x7f13f8142940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:41.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.822+0000 7f13effff640 1 --2- 172.21.15.116:0/3722378720 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f13f80c9850 0x7f13f81405e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:41.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.822+0000 7f13f7fff640 1 -- 172.21.15.116:0/3722378720 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f13f80aeb20 msgr2=0x7f13f8142940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:41.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.822+0000 7f13f7fff640 1 --2- 172.21.15.116:0/3722378720 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f13f80aeb20 0x7f13f8142940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:41.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.822+0000 7f13f7fff640 1 -- 172.21.15.116:0/3722378720 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f13f80c9850 msgr2=0x7f13f81405e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:41.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.822+0000 7f13f7fff640 1 --2- 172.21.15.116:0/3722378720 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f13f80c9850 0x7f13f81405e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:41.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.822+0000 7f13f7fff640 1 -- 172.21.15.116:0/3722378720 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13f81efe50 con 0x7f13f8150f80 2024-04-25T00:19:41.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.822+0000 7f13f7fff640 1 --2- 172.21.15.116:0/3722378720 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13f8150f80 0x7f13f8140b20 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f13f00b0040 tx=0x7f13f00195a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:41.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.822+0000 7f13f4ff9640 1 -- 172.21.15.116:0/3722378720 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13f00a6ac0 con 0x7f13f8150f80 2024-04-25T00:19:41.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.822+0000 7f13f4ff9640 1 -- 172.21.15.116:0/3722378720 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13f00bf070 con 0x7f13f8150f80 2024-04-25T00:19:41.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.822+0000 7f13fda83640 1 -- 172.21.15.116:0/3722378720 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13f80c8780 con 0x7f13f8150f80 2024-04-25T00:19:41.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.822+0000 7f13f4ff9640 1 -- 172.21.15.116:0/3722378720 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13f0004040 con 0x7f13f8150f80 2024-04-25T00:19:41.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.822+0000 7f13fda83640 1 -- 172.21.15.116:0/3722378720 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13f81efb70 con 0x7f13f8150f80 2024-04-25T00:19:41.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.826+0000 7f13fda83640 1 -- 172.21.15.116:0/3722378720 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13f8068a60 con 0x7f13f8150f80 2024-04-25T00:19:41.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.826+0000 7f13f4ff9640 1 -- 172.21.15.116:0/3722378720 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f13f00041e0 con 0x7f13f8150f80 2024-04-25T00:19:41.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.826+0000 7f13f4ff9640 1 --2- 172.21.15.116:0/3722378720 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f13d80678f0 0x7f13d8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:41.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.826+0000 7f13f4ff9640 1 -- 172.21.15.116:0/3722378720 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 9160+0+0 (secure 0 0 0) 0x7f13f01219e0 con 0x7f13f8150f80 2024-04-25T00:19:41.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.826+0000 7f13f77fe640 1 --2- 172.21.15.116:0/3722378720 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f13d80678f0 0x7f13d8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:41.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.826+0000 7f13f77fe640 1 --2- 172.21.15.116:0/3722378720 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f13d80678f0 0x7f13d8069db0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f13e8009b60 tx=0x7f13e8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:41.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:41.830+0000 7f13f4ff9640 1 -- 172.21.15.116:0/3722378720 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f13f0016480 con 0x7f13f8150f80 2024-04-25T00:19:42.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:42.066+0000 7f13fda83640 1 -- 172.21.15.116:0/3722378720 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f13f80705d0 con 0x7f13f8150f80 2024-04-25T00:19:42.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:42.070+0000 7f13f4ff9640 1 -- 172.21.15.116:0/3722378720 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 157 v157) ==== 95+0+6499 (secure 0 0 0) 0x7f13f00f1070 con 0x7f13f8150f80 2024-04-25T00:19:42.069 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:19:42.069 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":157,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7722,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":157}],"filesystems":[{"mdsmap":{"epoch":156,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:19:33.982945+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6586":{"gid":6586,"name":"d","rank":0,"incarnation":153,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/991183935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":991183935},{"type":"v1","addr":"172.21.15.174:6837","nonce":991183935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6586,"qdb_cluster":[6586]},"id":1},{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:19:08.844382+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7542},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7542":{"gid":7542,"name":"c","rank":0,"incarnation":148,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6835/879531976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":879531976},{"type":"v1","addr":"172.21.15.116:6835","nonce":879531976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7542,"qdb_cluster":[7542]},"id":2}]} 2024-04-25T00:19:42.069 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 157 2024-04-25T00:19:42.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:42.070+0000 7f13fda83640 1 -- 172.21.15.116:0/3722378720 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f13d80678f0 msgr2=0x7f13d8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:42.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:42.070+0000 7f13fda83640 1 --2- 172.21.15.116:0/3722378720 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f13d80678f0 0x7f13d8069db0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f13e8009b60 tx=0x7f13e8009190 comp rx=0 tx=0).stop 2024-04-25T00:19:42.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:42.070+0000 7f13fda83640 1 -- 172.21.15.116:0/3722378720 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13f8150f80 msgr2=0x7f13f8140b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:42.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:42.070+0000 7f13fda83640 1 --2- 172.21.15.116:0/3722378720 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13f8150f80 0x7f13f8140b20 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f13f00b0040 tx=0x7f13f00195a0 comp rx=0 tx=0).stop 2024-04-25T00:19:42.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:42.070+0000 7f13fda83640 1 -- 172.21.15.116:0/3722378720 shutdown_connections 2024-04-25T00:19:42.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:42.070+0000 7f13fda83640 1 --2- 172.21.15.116:0/3722378720 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f13d80678f0 0x7f13d8069db0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:42.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:42.070+0000 7f13fda83640 1 --2- 172.21.15.116:0/3722378720 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f13f8150f80 0x7f13f8140b20 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:42.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:42.070+0000 7f13fda83640 1 --2- 172.21.15.116:0/3722378720 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f13f80c9850 0x7f13f81405e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:42.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:42.070+0000 7f13fda83640 1 --2- 172.21.15.116:0/3722378720 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f13f80aeb20 0x7f13f8142940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:42.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:42.070+0000 7f13fda83640 1 -- 172.21.15.116:0/3722378720 >> 172.21.15.116:0/3722378720 conn(0x7f13f8074650 msgr2=0x7f13f8074a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:19:42.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:42.070+0000 7f13fda83640 1 -- 172.21.15.116:0/3722378720 shutdown_connections 2024-04-25T00:19:42.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:42.070+0000 7f13fda83640 1 -- 172.21.15.116:0/3722378720 wait complete. 2024-04-25T00:19:42.090 INFO:tasks.mds_thrash.fs.[a]:mds.a reported in up:standby state 2024-04-25T00:19:42.090 INFO:tasks.mds_thrash.fs.[a]:waiting for 54 secs before thrashing 2024-04-25T00:19:53.925 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:19:54.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.070+0000 7f558b3f2640 1 Processor -- start 2024-04-25T00:19:54.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.070+0000 7f558b3f2640 1 -- start start 2024-04-25T00:19:54.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.070+0000 7f558b3f2640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f5584148f20 0x7f55840c3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:54.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.070+0000 7f558b3f2640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5584152f70 0x7f558406e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:54.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.070+0000 7f558b3f2640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55840c2b40 con 0x7f55840c3300 2024-04-25T00:19:54.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.070+0000 7f558b3f2640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f55840c4790 con 0x7f5584152f70 2024-04-25T00:19:54.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.070+0000 7f558b3f2640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f55841533f0 con 0x7f5584148f20 2024-04-25T00:19:54.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.070+0000 7f558898d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5584152f70 0x7f558406e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:54.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.070+0000 7f558898d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5584152f70 0x7f558406e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53472/0 (socket says 172.21.15.116:53472) 2024-04-25T00:19:54.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.070+0000 7f558898d640 1 -- 172.21.15.116:0/3765604520 learned_addr learned my addr 172.21.15.116:0/3765604520 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:19:54.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.070+0000 7f558918e640 1 --2- 172.21.15.116:0/3765604520 >> v2:172.21.15.174:3300/0 conn(0x7f5584148f20 0x7f55840c3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:54.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.070+0000 7f558898d640 1 -- 172.21.15.116:0/3765604520 >> v2:172.21.15.174:3300/0 conn(0x7f5584148f20 msgr2=0x7f55840c3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:19:54.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.070+0000 7f558898d640 1 --2- 172.21.15.116:0/3765604520 >> v2:172.21.15.174:3300/0 conn(0x7f5584148f20 0x7f55840c3ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:54.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.070+0000 7f558898d640 1 -- 172.21.15.116:0/3765604520 >> v1:172.21.15.174:6789/0 conn(0x7f55840c3300 legacy=0x7f5584071d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:54.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.070+0000 7f558898d640 1 -- 172.21.15.116:0/3765604520 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55840c41d0 con 0x7f5584152f70 2024-04-25T00:19:54.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.074+0000 7f558898d640 1 --2- 172.21.15.116:0/3765604520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5584152f70 0x7f558406e480 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f55840c7ee0 tx=0x7f557401ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15dee26ce04b9298 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:54.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.074+0000 7f557bfff640 1 -- 172.21.15.116:0/3765604520 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55740b4070 con 0x7f5584152f70 2024-04-25T00:19:54.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.074+0000 7f557bfff640 1 -- 172.21.15.116:0/3765604520 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55740a7bc0 con 0x7f5584152f70 2024-04-25T00:19:54.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.074+0000 7f557bfff640 1 -- 172.21.15.116:0/3765604520 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55740af440 con 0x7f5584152f70 2024-04-25T00:19:54.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.074+0000 7f558b3f2640 1 -- 172.21.15.116:0/3765604520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5584152f70 msgr2=0x7f558406e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:54.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.074+0000 7f558b3f2640 1 --2- 172.21.15.116:0/3765604520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5584152f70 0x7f558406e480 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f55840c7ee0 tx=0x7f557401ebf0 comp rx=0 tx=0).stop 2024-04-25T00:19:54.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.074+0000 7f558b3f2640 1 -- 172.21.15.116:0/3765604520 shutdown_connections 2024-04-25T00:19:54.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.074+0000 7f558b3f2640 1 --2- 172.21.15.116:0/3765604520 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5584152f70 0x7f558406e480 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:54.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.074+0000 7f558b3f2640 1 --2- 172.21.15.116:0/3765604520 >> v2:172.21.15.174:3300/0 conn(0x7f5584148f20 0x7f55840c3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:54.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.074+0000 7f558b3f2640 1 -- 172.21.15.116:0/3765604520 >> 172.21.15.116:0/3765604520 conn(0x7f55840c7fa0 msgr2=0x7f55840c83b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:19:54.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.074+0000 7f558b3f2640 1 -- 172.21.15.116:0/3765604520 shutdown_connections 2024-04-25T00:19:54.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.074+0000 7f558b3f2640 1 -- 172.21.15.116:0/3765604520 wait complete. 2024-04-25T00:19:54.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.074+0000 7f558b3f2640 1 Processor -- start 2024-04-25T00:19:54.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.074+0000 7f558b3f2640 1 -- start start 2024-04-25T00:19:54.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.078+0000 7f558b3f2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55840c3300 0x7f55841519e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:54.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.078+0000 7f558b3f2640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5584148f20 0x7f5584136fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:54.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.078+0000 7f558b3f2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5584152f70 0x7f55841349d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:54.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.078+0000 7f558b3f2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5584072610 con 0x7f5584152f70 2024-04-25T00:19:54.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.078+0000 7f558b3f2640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5584072490 con 0x7f55840c3300 2024-04-25T00:19:54.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.078+0000 7f558b3f2640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f5584149300 con 0x7f5584148f20 2024-04-25T00:19:54.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.078+0000 7f558998f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5584152f70 0x7f55841349d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:54.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.078+0000 7f558998f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5584152f70 0x7f55841349d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53474/0 (socket says 172.21.15.116:53474) 2024-04-25T00:19:54.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.078+0000 7f558998f640 1 -- 172.21.15.116:0/605622395 learned_addr learned my addr 172.21.15.116:0/605622395 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:19:54.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.078+0000 7f558918e640 1 --2- 172.21.15.116:0/605622395 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55840c3300 0x7f55841519e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:54.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.078+0000 7f558898d640 1 --2- 172.21.15.116:0/605622395 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5584148f20 0x7f5584136fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:54.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.078+0000 7f558998f640 1 -- 172.21.15.116:0/605622395 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5584148f20 msgr2=0x7f5584136fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:54.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.078+0000 7f558998f640 1 --2- 172.21.15.116:0/605622395 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5584148f20 0x7f5584136fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:54.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.078+0000 7f558998f640 1 -- 172.21.15.116:0/605622395 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55840c3300 msgr2=0x7f55841519e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:54.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.078+0000 7f558998f640 1 --2- 172.21.15.116:0/605622395 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55840c3300 0x7f55841519e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:54.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.078+0000 7f558998f640 1 -- 172.21.15.116:0/605622395 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55840c7b30 con 0x7f5584152f70 2024-04-25T00:19:54.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.078+0000 7f558898d640 1 --2- 172.21.15.116:0/605622395 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5584148f20 0x7f5584136fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:19:54.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.078+0000 7f558918e640 1 --2- 172.21.15.116:0/605622395 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55840c3300 0x7f55841519e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:19:54.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.078+0000 7f558998f640 1 --2- 172.21.15.116:0/605622395 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5584152f70 0x7f55841349d0 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f557c00c760 tx=0x7f557c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:54.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.078+0000 7f5579ffb640 1 -- 172.21.15.116:0/605622395 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f557c003b80 con 0x7f5584152f70 2024-04-25T00:19:54.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.078+0000 7f5579ffb640 1 -- 172.21.15.116:0/605622395 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f557c003d20 con 0x7f5584152f70 2024-04-25T00:19:54.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.078+0000 7f5579ffb640 1 -- 172.21.15.116:0/605622395 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f557c00aa30 con 0x7f5584152f70 2024-04-25T00:19:54.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.078+0000 7f558b3f2640 1 -- 172.21.15.116:0/605622395 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5584152110 con 0x7f5584152f70 2024-04-25T00:19:54.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.078+0000 7f558b3f2640 1 -- 172.21.15.116:0/605622395 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55840c7050 con 0x7f5584152f70 2024-04-25T00:19:54.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.078+0000 7f558b3f2640 1 -- 172.21.15.116:0/605622395 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5584068a60 con 0x7f5584152f70 2024-04-25T00:19:54.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.078+0000 7f5579ffb640 1 -- 172.21.15.116:0/605622395 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f557c00abd0 con 0x7f5584152f70 2024-04-25T00:19:54.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.078+0000 7f5579ffb640 1 --2- 172.21.15.116:0/605622395 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f55680678f0 0x7f5568069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:54.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.078+0000 7f5579ffb640 1 -- 172.21.15.116:0/605622395 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 9160+0+0 (secure 0 0 0) 0x7f557c086890 con 0x7f5584152f70 2024-04-25T00:19:54.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.078+0000 7f558918e640 1 --2- 172.21.15.116:0/605622395 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f55680678f0 0x7f5568069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:54.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.082+0000 7f558918e640 1 --2- 172.21.15.116:0/605622395 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f55680678f0 0x7f5568069db0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f5580009bb0 tx=0x7f5580008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:54.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.086+0000 7f5579ffb640 1 -- 172.21.15.116:0/605622395 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f55840c7b30 con 0x7f5584152f70 2024-04-25T00:19:54.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.326+0000 7f558b3f2640 1 -- 172.21.15.116:0/605622395 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f558406e420 con 0x7f5584152f70 2024-04-25T00:19:54.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.326+0000 7f5579ffb640 1 -- 172.21.15.116:0/605622395 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 157 v157) ==== 95+0+6499 (secure 0 0 0) 0x7f557c056690 con 0x7f5584152f70 2024-04-25T00:19:54.326 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:19:54.326 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":157,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7722,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":157}],"filesystems":[{"mdsmap":{"epoch":156,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:19:33.982945+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6586":{"gid":6586,"name":"d","rank":0,"incarnation":153,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/991183935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":991183935},{"type":"v1","addr":"172.21.15.174:6837","nonce":991183935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6586,"qdb_cluster":[6586]},"id":1},{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:19:08.844382+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7542},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7542":{"gid":7542,"name":"c","rank":0,"incarnation":148,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6835/879531976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":879531976},{"type":"v1","addr":"172.21.15.116:6835","nonce":879531976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7542,"qdb_cluster":[7542]},"id":2}]} 2024-04-25T00:19:54.327 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 157 2024-04-25T00:19:54.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.330+0000 7f558b3f2640 1 -- 172.21.15.116:0/605622395 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f55680678f0 msgr2=0x7f5568069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:54.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.330+0000 7f558b3f2640 1 --2- 172.21.15.116:0/605622395 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f55680678f0 0x7f5568069db0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f5580009bb0 tx=0x7f5580008040 comp rx=0 tx=0).stop 2024-04-25T00:19:54.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.330+0000 7f558b3f2640 1 -- 172.21.15.116:0/605622395 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5584152f70 msgr2=0x7f55841349d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:54.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.330+0000 7f558b3f2640 1 --2- 172.21.15.116:0/605622395 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5584152f70 0x7f55841349d0 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f557c00c760 tx=0x7f557c00cc20 comp rx=0 tx=0).stop 2024-04-25T00:19:54.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.330+0000 7f558b3f2640 1 -- 172.21.15.116:0/605622395 shutdown_connections 2024-04-25T00:19:54.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.330+0000 7f558b3f2640 1 --2- 172.21.15.116:0/605622395 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f55680678f0 0x7f5568069db0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:54.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.330+0000 7f558b3f2640 1 --2- 172.21.15.116:0/605622395 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5584152f70 0x7f55841349d0 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:54.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.330+0000 7f558b3f2640 1 --2- 172.21.15.116:0/605622395 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5584148f20 0x7f5584136fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:54.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.330+0000 7f558b3f2640 1 --2- 172.21.15.116:0/605622395 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f55840c3300 0x7f55841519e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:54.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.330+0000 7f558b3f2640 1 -- 172.21.15.116:0/605622395 >> 172.21.15.116:0/605622395 conn(0x7f55840c7fa0 msgr2=0x7f5584071d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:19:54.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.330+0000 7f558b3f2640 1 -- 172.21.15.116:0/605622395 shutdown_connections 2024-04-25T00:19:54.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.330+0000 7f558b3f2640 1 -- 172.21.15.116:0/605622395 wait complete. 2024-04-25T00:19:54.348 INFO:tasks.mds_thrash.fs.[b]:kill mds.c (rank=0) 2024-04-25T00:19:54.348 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-25T00:19:54.348 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:19:54.358 INFO:tasks.ceph.mds.c:Stopped 2024-04-25T00:19:54.359 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:19:54.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.526+0000 7f57eb44f640 1 Processor -- start 2024-04-25T00:19:54.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.526+0000 7f57eb44f640 1 -- start start 2024-04-25T00:19:54.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.526+0000 7f57eb44f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f57e4155c00 0x7f57e4071520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:54.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.526+0000 7f57eb44f640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f57e4073540 0x7f57e414f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:54.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.526+0000 7f57eb44f640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57e41566f0 con 0x7f57e40730c0 2024-04-25T00:19:54.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.526+0000 7f57eb44f640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f57e4071bb0 con 0x7f57e4073540 2024-04-25T00:19:54.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.526+0000 7f57eb44f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f57e4072ee0 con 0x7f57e4155c00 2024-04-25T00:19:54.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.526+0000 7f57e99ec640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f57e4073540 0x7f57e414f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:54.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.526+0000 7f57e99ec640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f57e4073540 0x7f57e414f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:51100/0 (socket says 172.21.15.116:51100) 2024-04-25T00:19:54.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.526+0000 7f57e99ec640 1 -- 172.21.15.116:0/2522470583 learned_addr learned my addr 172.21.15.116:0/2522470583 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:19:54.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.526+0000 7f57e91eb640 1 --2- 172.21.15.116:0/2522470583 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f57e4155c00 0x7f57e4071520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:54.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.526+0000 7f57e99ec640 1 -- 172.21.15.116:0/2522470583 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f57e4155c00 msgr2=0x7f57e4071520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:19:54.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.526+0000 7f57e99ec640 1 --2- 172.21.15.116:0/2522470583 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f57e4155c00 0x7f57e4071520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:54.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.526+0000 7f57e99ec640 1 -- 172.21.15.116:0/2522470583 >> v1:172.21.15.174:6789/0 conn(0x7f57e40730c0 legacy=0x7f57e4155fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:54.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.526+0000 7f57e99ec640 1 -- 172.21.15.116:0/2522470583 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57e4072920 con 0x7f57e4073540 2024-04-25T00:19:54.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.530+0000 7f57e99ec640 1 --2- 172.21.15.116:0/2522470583 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f57e4073540 0x7f57e414f7a0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f57dc009e50 tx=0x7f57dc01c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa9c8e276e04590 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:54.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.530+0000 7f57dbfff640 1 -- 172.21.15.116:0/2522470583 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57dc0a55d0 con 0x7f57e4073540 2024-04-25T00:19:54.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.530+0000 7f57dbfff640 1 -- 172.21.15.116:0/2522470583 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57dc0a5770 con 0x7f57e4073540 2024-04-25T00:19:54.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.530+0000 7f57dbfff640 1 -- 172.21.15.116:0/2522470583 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57dc0a5b40 con 0x7f57e4073540 2024-04-25T00:19:54.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.530+0000 7f57eb44f640 1 -- 172.21.15.116:0/2522470583 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f57e4073540 msgr2=0x7f57e414f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:54.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.530+0000 7f57eb44f640 1 --2- 172.21.15.116:0/2522470583 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f57e4073540 0x7f57e414f7a0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f57dc009e50 tx=0x7f57dc01c910 comp rx=0 tx=0).stop 2024-04-25T00:19:54.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.530+0000 7f57eb44f640 1 -- 172.21.15.116:0/2522470583 shutdown_connections 2024-04-25T00:19:54.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.530+0000 7f57eb44f640 1 --2- 172.21.15.116:0/2522470583 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f57e4073540 0x7f57e414f7a0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:54.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.530+0000 7f57eb44f640 1 --2- 172.21.15.116:0/2522470583 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f57e4155c00 0x7f57e4071520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:54.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.530+0000 7f57eb44f640 1 -- 172.21.15.116:0/2522470583 >> 172.21.15.116:0/2522470583 conn(0x7f57e40c4200 msgr2=0x7f57e40c4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:19:54.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.530+0000 7f57eb44f640 1 -- 172.21.15.116:0/2522470583 shutdown_connections 2024-04-25T00:19:54.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.530+0000 7f57eb44f640 1 -- 172.21.15.116:0/2522470583 wait complete. 2024-04-25T00:19:54.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.534+0000 7f57eb44f640 1 Processor -- start 2024-04-25T00:19:54.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.534+0000 7f57eb44f640 1 -- start start 2024-04-25T00:19:54.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.534+0000 7f57eb44f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f57e40730c0 0x7f57e40ad0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:54.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.534+0000 7f57eb44f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f57e4073540 0x7f57e40ac730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:54.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.534+0000 7f57eb44f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57e4155c00 0x7f57e40ab4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:54.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.534+0000 7f57eb44f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f57e40c2100 con 0x7f57e4155c00 2024-04-25T00:19:54.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.534+0000 7f57eb44f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f57e40a7520 con 0x7f57e40730c0 2024-04-25T00:19:54.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.534+0000 7f57eb44f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f57e40c2280 con 0x7f57e4073540 2024-04-25T00:19:54.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.534+0000 7f57e99ec640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57e4155c00 0x7f57e40ab4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:54.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.534+0000 7f57e99ec640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57e4155c00 0x7f57e40ab4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:53486/0 (socket says 172.21.15.116:53486) 2024-04-25T00:19:54.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.534+0000 7f57e99ec640 1 -- 172.21.15.116:0/773249151 learned_addr learned my addr 172.21.15.116:0/773249151 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:19:54.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.534+0000 7f57e89ea640 1 --2- 172.21.15.116:0/773249151 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f57e4073540 0x7f57e40ac730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:54.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.534+0000 7f57e99ec640 1 -- 172.21.15.116:0/773249151 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f57e4073540 msgr2=0x7f57e40ac730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:54.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.534+0000 7f57e99ec640 1 --2- 172.21.15.116:0/773249151 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f57e4073540 0x7f57e40ac730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:54.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.534+0000 7f57e99ec640 1 -- 172.21.15.116:0/773249151 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f57e40730c0 msgr2=0x7f57e40ad0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:54.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.534+0000 7f57e99ec640 1 --2- 172.21.15.116:0/773249151 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f57e40730c0 0x7f57e40ad0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:54.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.534+0000 7f57e99ec640 1 -- 172.21.15.116:0/773249151 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57e40acc70 con 0x7f57e4155c00 2024-04-25T00:19:54.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.534+0000 7f57e99ec640 1 --2- 172.21.15.116:0/773249151 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57e4155c00 0x7f57e40ab4b0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f57dc01a540 tx=0x7f57dc0196b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:54.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.534+0000 7f57d9ffb640 1 -- 172.21.15.116:0/773249151 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57dc0bb030 con 0x7f57e4155c00 2024-04-25T00:19:54.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.534+0000 7f57d9ffb640 1 -- 172.21.15.116:0/773249151 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57dc0b6440 con 0x7f57e4155c00 2024-04-25T00:19:54.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.534+0000 7f57eb44f640 1 -- 172.21.15.116:0/773249151 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57e4153080 con 0x7f57e4155c00 2024-04-25T00:19:54.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.534+0000 7f57d9ffb640 1 -- 172.21.15.116:0/773249151 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57dc009510 con 0x7f57e4155c00 2024-04-25T00:19:54.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.534+0000 7f57eb44f640 1 -- 172.21.15.116:0/773249151 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57e4149cd0 con 0x7f57e4155c00 2024-04-25T00:19:54.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.538+0000 7f57eb44f640 1 -- 172.21.15.116:0/773249151 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57e4068a60 con 0x7f57e4155c00 2024-04-25T00:19:54.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.538+0000 7f57d9ffb640 1 -- 172.21.15.116:0/773249151 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f57dc0be070 con 0x7f57e4155c00 2024-04-25T00:19:54.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.538+0000 7f57d9ffb640 1 --2- 172.21.15.116:0/773249151 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f57c80678c0 0x7f57c8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:54.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.538+0000 7f57d9ffb640 1 -- 172.21.15.116:0/773249151 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 9160+0+0 (secure 0 0 0) 0x7f57dc121630 con 0x7f57e4155c00 2024-04-25T00:19:54.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.538+0000 7f57e91eb640 1 --2- 172.21.15.116:0/773249151 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f57c80678c0 0x7f57c8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:54.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.538+0000 7f57e91eb640 1 --2- 172.21.15.116:0/773249151 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f57c80678c0 0x7f57c8069d80 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f57e0006aa0 tx=0x7f57e0006a30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:54.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.542+0000 7f57d9ffb640 1 -- 172.21.15.116:0/773249151 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f57e40acc70 con 0x7f57e4155c00 2024-04-25T00:19:54.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.762+0000 7f57eb44f640 1 -- 172.21.15.116:0/773249151 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f57e4074980 con 0x7f57e4155c00 2024-04-25T00:19:54.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.766+0000 7f57d9ffb640 1 -- 172.21.15.116:0/773249151 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 157 v157) ==== 95+0+6499 (secure 0 0 0) 0x7f57dc016620 con 0x7f57e4155c00 2024-04-25T00:19:54.765 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:19:54.765 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":157,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7722,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":157}],"filesystems":[{"mdsmap":{"epoch":156,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:19:33.982945+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6586":{"gid":6586,"name":"d","rank":0,"incarnation":153,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/991183935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":991183935},{"type":"v1","addr":"172.21.15.174:6837","nonce":991183935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6586,"qdb_cluster":[6586]},"id":1},{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:19:08.844382+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7542},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7542":{"gid":7542,"name":"c","rank":0,"incarnation":148,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6835/879531976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":879531976},{"type":"v1","addr":"172.21.15.116:6835","nonce":879531976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7542,"qdb_cluster":[7542]},"id":2}]} 2024-04-25T00:19:54.765 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 157 2024-04-25T00:19:54.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.766+0000 7f57eb44f640 1 -- 172.21.15.116:0/773249151 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f57c80678c0 msgr2=0x7f57c8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:54.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.766+0000 7f57eb44f640 1 --2- 172.21.15.116:0/773249151 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f57c80678c0 0x7f57c8069d80 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f57e0006aa0 tx=0x7f57e0006a30 comp rx=0 tx=0).stop 2024-04-25T00:19:54.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.766+0000 7f57eb44f640 1 -- 172.21.15.116:0/773249151 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57e4155c00 msgr2=0x7f57e40ab4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:54.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.766+0000 7f57eb44f640 1 --2- 172.21.15.116:0/773249151 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57e4155c00 0x7f57e40ab4b0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f57dc01a540 tx=0x7f57dc0196b0 comp rx=0 tx=0).stop 2024-04-25T00:19:54.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.766+0000 7f57eb44f640 1 -- 172.21.15.116:0/773249151 shutdown_connections 2024-04-25T00:19:54.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.766+0000 7f57eb44f640 1 --2- 172.21.15.116:0/773249151 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f57c80678c0 0x7f57c8069d80 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:54.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.766+0000 7f57eb44f640 1 --2- 172.21.15.116:0/773249151 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57e4155c00 0x7f57e40ab4b0 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:54.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.766+0000 7f57eb44f640 1 --2- 172.21.15.116:0/773249151 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f57e4073540 0x7f57e40ac730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:54.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.766+0000 7f57eb44f640 1 --2- 172.21.15.116:0/773249151 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f57e40730c0 0x7f57e40ad0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:54.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.766+0000 7f57eb44f640 1 -- 172.21.15.116:0/773249151 >> 172.21.15.116:0/773249151 conn(0x7f57e40c4200 msgr2=0x7f57e4153a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:19:54.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.766+0000 7f57eb44f640 1 -- 172.21.15.116:0/773249151 shutdown_connections 2024-04-25T00:19:54.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:54.766+0000 7f57eb44f640 1 -- 172.21.15.116:0/773249151 wait complete. 2024-04-25T00:19:54.786 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:19:56.786 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:19:56.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.934+0000 7f7f0c8e9640 1 Processor -- start 2024-04-25T00:19:56.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.934+0000 7f7f0c8e9640 1 -- start start 2024-04-25T00:19:56.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.934+0000 7f7f0c8e9640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7f08152f60 0x7f7f080c6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:56.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.934+0000 7f7f0c8e9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7f080c3c30 0x7f7f0806fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:56.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.934+0000 7f7f0c8e9640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f0806e8f0 con 0x7f7f08158ea0 2024-04-25T00:19:56.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.934+0000 7f7f0c8e9640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f7f0806ebd0 con 0x7f7f080c3c30 2024-04-25T00:19:56.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.934+0000 7f7f0c8e9640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f7f0806f330 con 0x7f7f08152f60 2024-04-25T00:19:56.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.934+0000 7f7f05d9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7f080c3c30 0x7f7f0806fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:56.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.934+0000 7f7f06d9d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f7f08158ea0 0x7f7f0806c220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:56758/0 (socket says 172.21.15.116:56758) 2024-04-25T00:19:56.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.934+0000 7f7f06d9d640 1 -- 172.21.15.116:0/3572464219 learned_addr learned my addr 172.21.15.116:0/3572464219 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:19:56.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.934+0000 7f7f0659c640 1 --2- 172.21.15.116:0/3572464219 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7f08152f60 0x7f7f080c6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:56.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.934+0000 7f7f05d9b640 1 -- 172.21.15.116:0/3572464219 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7f08152f60 msgr2=0x7f7f080c6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:19:56.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.934+0000 7f7f05d9b640 1 --2- 172.21.15.116:0/3572464219 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7f08152f60 0x7f7f080c6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:56.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.934+0000 7f7f05d9b640 1 -- 172.21.15.116:0/3572464219 >> v1:172.21.15.116:6789/0 conn(0x7f7f08158ea0 legacy=0x7f7f0806c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:56.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.934+0000 7f7f05d9b640 1 -- 172.21.15.116:0/3572464219 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f0806f610 con 0x7f7f080c3c30 2024-04-25T00:19:56.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.934+0000 7f7f0559a640 1 -- 172.21.15.116:0/3572464219 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3567002830 0 0) 0x7f7f0806e8f0 con 0x7f7f08158ea0 2024-04-25T00:19:56.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.938+0000 7f7f05d9b640 1 --2- 172.21.15.116:0/3572464219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f080c3c30 0x7f7f0806fb30 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f7efc0097c0 tx=0x7f7efc01c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8283be4524a07a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:56.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.938+0000 7f7f0559a640 1 -- 172.21.15.116:0/3572464219 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7efc0b0070 con 0x7f7f080c3c30 2024-04-25T00:19:56.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.938+0000 7f7f0559a640 1 -- 172.21.15.116:0/3572464219 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7efc0ab600 con 0x7f7f080c3c30 2024-04-25T00:19:56.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.938+0000 7f7f0559a640 1 -- 172.21.15.116:0/3572464219 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7efc0ab9d0 con 0x7f7f080c3c30 2024-04-25T00:19:56.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.938+0000 7f7f0c8e9640 1 -- 172.21.15.116:0/3572464219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f080c3c30 msgr2=0x7f7f0806fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:56.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.938+0000 7f7f0c8e9640 1 --2- 172.21.15.116:0/3572464219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f080c3c30 0x7f7f0806fb30 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f7efc0097c0 tx=0x7f7efc01c410 comp rx=0 tx=0).stop 2024-04-25T00:19:56.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.938+0000 7f7f0c8e9640 1 -- 172.21.15.116:0/3572464219 shutdown_connections 2024-04-25T00:19:56.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.938+0000 7f7f0c8e9640 1 --2- 172.21.15.116:0/3572464219 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f080c3c30 0x7f7f0806fb30 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:56.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.938+0000 7f7f0c8e9640 1 --2- 172.21.15.116:0/3572464219 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7f08152f60 0x7f7f080c6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:56.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.938+0000 7f7f0c8e9640 1 -- 172.21.15.116:0/3572464219 >> 172.21.15.116:0/3572464219 conn(0x7f7f080c4270 msgr2=0x7f7f080c4680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:19:56.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.938+0000 7f7f0c8e9640 1 -- 172.21.15.116:0/3572464219 shutdown_connections 2024-04-25T00:19:56.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.938+0000 7f7f0c8e9640 1 -- 172.21.15.116:0/3572464219 wait complete. 2024-04-25T00:19:56.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.938+0000 7f7f0c8e9640 1 Processor -- start 2024-04-25T00:19:56.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.938+0000 7f7f0c8e9640 1 -- start start 2024-04-25T00:19:56.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.938+0000 7f7f0c8e9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f080c3c30 0x7f7f0814cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:56.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.938+0000 7f7f0c8e9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7f08152f60 0x7f7f081f0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:56.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.938+0000 7f7f0c8e9640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7f08158ea0 0x7f7f08157a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:56.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.938+0000 7f7f0c8e9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7f08162b70 con 0x7f7f080c3c30 2024-04-25T00:19:56.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.938+0000 7f7f0c8e9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7f081629f0 con 0x7f7f08152f60 2024-04-25T00:19:56.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.938+0000 7f7f0c8e9640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f7f08162cf0 con 0x7f7f08158ea0 2024-04-25T00:19:56.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.938+0000 7f7f0659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f080c3c30 0x7f7f0814cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:56.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.938+0000 7f7f0659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f080c3c30 0x7f7f0814cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57092/0 (socket says 172.21.15.116:57092) 2024-04-25T00:19:56.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.938+0000 7f7f0659c640 1 -- 172.21.15.116:0/4066567200 learned_addr learned my addr 172.21.15.116:0/4066567200 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:19:56.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.938+0000 7f7f06d9d640 1 --2- 172.21.15.116:0/4066567200 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7f08158ea0 0x7f7f08157a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:56.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.938+0000 7f7f0659c640 1 -- 172.21.15.116:0/4066567200 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7f08158ea0 msgr2=0x7f7f08157a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:56.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.938+0000 7f7f0659c640 1 --2- 172.21.15.116:0/4066567200 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7f08158ea0 0x7f7f08157a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:56.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.938+0000 7f7f05d9b640 1 --2- 172.21.15.116:0/4066567200 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7f08152f60 0x7f7f081f0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:56.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.938+0000 7f7f0659c640 1 -- 172.21.15.116:0/4066567200 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7f08152f60 msgr2=0x7f7f081f0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:56.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.938+0000 7f7f0659c640 1 --2- 172.21.15.116:0/4066567200 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7f08152f60 0x7f7f081f0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:56.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.938+0000 7f7f0659c640 1 -- 172.21.15.116:0/4066567200 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f081ee390 con 0x7f7f080c3c30 2024-04-25T00:19:56.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.938+0000 7f7f05d9b640 1 --2- 172.21.15.116:0/4066567200 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7f08152f60 0x7f7f081f0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:19:56.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.938+0000 7f7f0659c640 1 --2- 172.21.15.116:0/4066567200 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f080c3c30 0x7f7f0814cb40 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f7ef800d5f0 tx=0x7f7ef800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:56.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.942+0000 7f7ef6ffd640 1 -- 172.21.15.116:0/4066567200 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef80125f0 con 0x7f7f080c3c30 2024-04-25T00:19:56.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.942+0000 7f7ef6ffd640 1 -- 172.21.15.116:0/4066567200 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ef8019070 con 0x7f7f080c3c30 2024-04-25T00:19:56.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.942+0000 7f7f0c8e9640 1 -- 172.21.15.116:0/4066567200 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f08153840 con 0x7f7f080c3c30 2024-04-25T00:19:56.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.942+0000 7f7ef6ffd640 1 -- 172.21.15.116:0/4066567200 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef80129b0 con 0x7f7f080c3c30 2024-04-25T00:19:56.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.942+0000 7f7f0c8e9640 1 -- 172.21.15.116:0/4066567200 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f08153e00 con 0x7f7f080c3c30 2024-04-25T00:19:56.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.942+0000 7f7f0c8e9640 1 -- 172.21.15.116:0/4066567200 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f08068a60 con 0x7f7f080c3c30 2024-04-25T00:19:56.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.942+0000 7f7ef6ffd640 1 -- 172.21.15.116:0/4066567200 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7ef8012b50 con 0x7f7f080c3c30 2024-04-25T00:19:56.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.942+0000 7f7ef6ffd640 1 --2- 172.21.15.116:0/4066567200 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7ee80678c0 0x7f7ee8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:56.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.942+0000 7f7ef6ffd640 1 -- 172.21.15.116:0/4066567200 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 9160+0+0 (secure 0 0 0) 0x7f7ef8086e10 con 0x7f7f080c3c30 2024-04-25T00:19:56.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.942+0000 7f7f05d9b640 1 --2- 172.21.15.116:0/4066567200 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7ee80678c0 0x7f7ee8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:56.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.942+0000 7f7f05d9b640 1 --2- 172.21.15.116:0/4066567200 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7ee80678c0 0x7f7ee8069d80 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f7efc0ad000 tx=0x7f7efc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:56.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:56.946+0000 7f7ef6ffd640 1 -- 172.21.15.116:0/4066567200 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f7ef8056c10 con 0x7f7f080c3c30 2024-04-25T00:19:57.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:57.178+0000 7f7f0c8e9640 1 -- 172.21.15.116:0/4066567200 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7f0806fc40 con 0x7f7f080c3c30 2024-04-25T00:19:57.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:57.178+0000 7f7ef6ffd640 1 -- 172.21.15.116:0/4066567200 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 157 v157) ==== 95+0+6499 (secure 0 0 0) 0x7f7ef8056510 con 0x7f7f080c3c30 2024-04-25T00:19:57.178 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:19:57.178 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":157,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7722,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":157}],"filesystems":[{"mdsmap":{"epoch":156,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:19:33.982945+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6586":{"gid":6586,"name":"d","rank":0,"incarnation":153,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/991183935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":991183935},{"type":"v1","addr":"172.21.15.174:6837","nonce":991183935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6586,"qdb_cluster":[6586]},"id":1},{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:19:08.844382+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7542},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7542":{"gid":7542,"name":"c","rank":0,"incarnation":148,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6835/879531976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":879531976},{"type":"v1","addr":"172.21.15.116:6835","nonce":879531976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7542,"qdb_cluster":[7542]},"id":2}]} 2024-04-25T00:19:57.178 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 157 2024-04-25T00:19:57.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:57.178+0000 7f7f0c8e9640 1 -- 172.21.15.116:0/4066567200 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7ee80678c0 msgr2=0x7f7ee8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:57.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:57.178+0000 7f7f0c8e9640 1 --2- 172.21.15.116:0/4066567200 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7ee80678c0 0x7f7ee8069d80 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f7efc0ad000 tx=0x7f7efc008040 comp rx=0 tx=0).stop 2024-04-25T00:19:57.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:57.178+0000 7f7f0c8e9640 1 -- 172.21.15.116:0/4066567200 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f080c3c30 msgr2=0x7f7f0814cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:57.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:57.178+0000 7f7f0c8e9640 1 --2- 172.21.15.116:0/4066567200 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f080c3c30 0x7f7f0814cb40 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f7ef800d5f0 tx=0x7f7ef800dab0 comp rx=0 tx=0).stop 2024-04-25T00:19:57.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:57.178+0000 7f7f0c8e9640 1 -- 172.21.15.116:0/4066567200 shutdown_connections 2024-04-25T00:19:57.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:57.178+0000 7f7f0c8e9640 1 --2- 172.21.15.116:0/4066567200 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7ee80678c0 0x7f7ee8069d80 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:57.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:57.178+0000 7f7f0c8e9640 1 --2- 172.21.15.116:0/4066567200 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7f08158ea0 0x7f7f08157a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:57.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:57.178+0000 7f7f0c8e9640 1 --2- 172.21.15.116:0/4066567200 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7f08152f60 0x7f7f081f0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:57.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:57.178+0000 7f7f0c8e9640 1 --2- 172.21.15.116:0/4066567200 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7f080c3c30 0x7f7f0814cb40 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:57.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:57.178+0000 7f7f0c8e9640 1 -- 172.21.15.116:0/4066567200 >> 172.21.15.116:0/4066567200 conn(0x7f7f080c4270 msgr2=0x7f7f080c5320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:19:57.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:57.178+0000 7f7f0c8e9640 1 -- 172.21.15.116:0/4066567200 shutdown_connections 2024-04-25T00:19:57.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:57.178+0000 7f7f0c8e9640 1 -- 172.21.15.116:0/4066567200 wait complete. 2024-04-25T00:19:57.197 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:19:59.198 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:19:59.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.334+0000 7f11eaeac640 1 Processor -- start 2024-04-25T00:19:59.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.334+0000 7f11eaeac640 1 -- start start 2024-04-25T00:19:59.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.334+0000 7f11eaeac640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f11e4152d60 0x7f11e40ab270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:59.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.334+0000 7f11eaeac640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11e4155c50 con 0x7f11e40ab7b0 2024-04-25T00:19:59.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.334+0000 7f11eaeac640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11e4155f30 con 0x7f11e4153140 2024-04-25T00:19:59.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.334+0000 7f11eaeac640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f11e4156210 con 0x7f11e4152d60 2024-04-25T00:19:59.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.334+0000 7f11e8c48640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f11e4152d60 0x7f11e40ab270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:59.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.334+0000 7f11e3fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f11e40ab7b0 0x7f11e40c8450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:56764/0 (socket says 172.21.15.116:56764) 2024-04-25T00:19:59.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.334+0000 7f11e3fff640 1 -- 172.21.15.116:0/3876554933 learned_addr learned my addr 172.21.15.116:0/3876554933 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:19:59.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.334+0000 7f11e8c48640 1 -- 172.21.15.116:0/3876554933 >> v1:172.21.15.174:6789/0 conn(0x7f11e4153140 legacy=0x7f11e40c3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:19:59.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.334+0000 7f11e8c48640 1 -- 172.21.15.116:0/3876554933 >> v1:172.21.15.116:6789/0 conn(0x7f11e40ab7b0 legacy=0x7f11e40c8450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:59.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.334+0000 7f11e37fe640 1 -- 172.21.15.116:0/3876554933 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3751185295 0 0) 0x7f11e4155c50 con 0x7f11e40ab7b0 2024-04-25T00:19:59.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.334+0000 7f11e8c48640 1 -- 172.21.15.116:0/3876554933 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11e41564f0 con 0x7f11e4152d60 2024-04-25T00:19:59.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.338+0000 7f11e8c48640 1 --2- 172.21.15.116:0/3876554933 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11e4152d60 0x7f11e40ab270 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f11d80097c0 tx=0x7f11d801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b861e6af657e697 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:59.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.338+0000 7f11e37fe640 1 -- 172.21.15.116:0/3876554933 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d80b0070 con 0x7f11e4152d60 2024-04-25T00:19:59.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.338+0000 7f11e37fe640 1 -- 172.21.15.116:0/3876554933 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11d80ab600 con 0x7f11e4152d60 2024-04-25T00:19:59.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.338+0000 7f11e37fe640 1 -- 172.21.15.116:0/3876554933 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d80ab9d0 con 0x7f11e4152d60 2024-04-25T00:19:59.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.338+0000 7f11eaeac640 1 -- 172.21.15.116:0/3876554933 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11e4152d60 msgr2=0x7f11e40ab270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:59.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.338+0000 7f11eaeac640 1 --2- 172.21.15.116:0/3876554933 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11e4152d60 0x7f11e40ab270 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f11d80097c0 tx=0x7f11d801c460 comp rx=0 tx=0).stop 2024-04-25T00:19:59.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.338+0000 7f11eaeac640 1 -- 172.21.15.116:0/3876554933 shutdown_connections 2024-04-25T00:19:59.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.338+0000 7f11eaeac640 1 --2- 172.21.15.116:0/3876554933 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11e4152d60 0x7f11e40ab270 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:59.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.338+0000 7f11eaeac640 1 -- 172.21.15.116:0/3876554933 >> 172.21.15.116:0/3876554933 conn(0x7f11e406f0c0 msgr2=0x7f11e406f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:19:59.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.338+0000 7f11eaeac640 1 -- 172.21.15.116:0/3876554933 shutdown_connections 2024-04-25T00:19:59.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.338+0000 7f11eaeac640 1 -- 172.21.15.116:0/3876554933 wait complete. 2024-04-25T00:19:59.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.338+0000 7f11eaeac640 1 Processor -- start 2024-04-25T00:19:59.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.338+0000 7f11eaeac640 1 -- start start 2024-04-25T00:19:59.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.338+0000 7f11eaeac640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11e40ab7b0 0x7f11e4157920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:59.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.338+0000 7f11eaeac640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11e4152d60 0x7f11e406da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:59.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.338+0000 7f11eaeac640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11e4153140 0x7f11e414dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:59.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.338+0000 7f11eaeac640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f11e41639b0 con 0x7f11e4152d60 2024-04-25T00:19:59.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.338+0000 7f11eaeac640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f11e40c4a30 con 0x7f11e4153140 2024-04-25T00:19:59.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.338+0000 7f11eaeac640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f11e4163b30 con 0x7f11e40ab7b0 2024-04-25T00:19:59.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.338+0000 7f11e3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11e4152d60 0x7f11e406da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:59.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.338+0000 7f11e3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11e4152d60 0x7f11e406da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57110/0 (socket says 172.21.15.116:57110) 2024-04-25T00:19:59.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.338+0000 7f11e3fff640 1 -- 172.21.15.116:0/1909960108 learned_addr learned my addr 172.21.15.116:0/1909960108 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:19:59.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.338+0000 7f11e8c48640 1 --2- 172.21.15.116:0/1909960108 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11e40ab7b0 0x7f11e4157920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:59.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.338+0000 7f11e9449640 1 --2- 172.21.15.116:0/1909960108 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11e4153140 0x7f11e414dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:59.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.338+0000 7f11e3fff640 1 -- 172.21.15.116:0/1909960108 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11e40ab7b0 msgr2=0x7f11e4157920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:59.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.338+0000 7f11e3fff640 1 --2- 172.21.15.116:0/1909960108 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11e40ab7b0 0x7f11e4157920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:59.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.338+0000 7f11e3fff640 1 -- 172.21.15.116:0/1909960108 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11e4153140 msgr2=0x7f11e414dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:59.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.338+0000 7f11e3fff640 1 --2- 172.21.15.116:0/1909960108 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11e4153140 0x7f11e414dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:59.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.338+0000 7f11e3fff640 1 -- 172.21.15.116:0/1909960108 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11e414d130 con 0x7f11e4152d60 2024-04-25T00:19:59.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.342+0000 7f11e3fff640 1 --2- 172.21.15.116:0/1909960108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11e4152d60 0x7f11e406da60 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f11d000c480 tx=0x7f11d000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:59.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.342+0000 7f11e17fa640 1 -- 172.21.15.116:0/1909960108 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d0018070 con 0x7f11e4152d60 2024-04-25T00:19:59.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.342+0000 7f11e17fa640 1 -- 172.21.15.116:0/1909960108 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11d00074e0 con 0x7f11e4152d60 2024-04-25T00:19:59.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.342+0000 7f11eaeac640 1 -- 172.21.15.116:0/1909960108 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11e414ba50 con 0x7f11e4152d60 2024-04-25T00:19:59.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.342+0000 7f11e17fa640 1 -- 172.21.15.116:0/1909960108 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d00049f0 con 0x7f11e4152d60 2024-04-25T00:19:59.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.342+0000 7f11eaeac640 1 -- 172.21.15.116:0/1909960108 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11e40ae030 con 0x7f11e4152d60 2024-04-25T00:19:59.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.342+0000 7f11eaeac640 1 -- 172.21.15.116:0/1909960108 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11e4068a60 con 0x7f11e4152d60 2024-04-25T00:19:59.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.342+0000 7f11e17fa640 1 -- 172.21.15.116:0/1909960108 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f11d001b070 con 0x7f11e4152d60 2024-04-25T00:19:59.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.342+0000 7f11e17fa640 1 --2- 172.21.15.116:0/1909960108 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f11c80678c0 0x7f11c8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:19:59.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.342+0000 7f11e17fa640 1 -- 172.21.15.116:0/1909960108 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 9160+0+0 (secure 0 0 0) 0x7f11d0085e70 con 0x7f11e4152d60 2024-04-25T00:19:59.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.342+0000 7f11e8c48640 1 --2- 172.21.15.116:0/1909960108 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f11c80678c0 0x7f11c8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:19:59.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.342+0000 7f11e8c48640 1 --2- 172.21.15.116:0/1909960108 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f11c80678c0 0x7f11c8069d80 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f11d80ad000 tx=0x7f11d8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:19:59.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.346+0000 7f11e17fa640 1 -- 172.21.15.116:0/1909960108 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f11d0055c10 con 0x7f11e4152d60 2024-04-25T00:19:59.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.582+0000 7f11eaeac640 1 -- 172.21.15.116:0/1909960108 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f11e4160f70 con 0x7f11e4152d60 2024-04-25T00:19:59.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.582+0000 7f11e17fa640 1 -- 172.21.15.116:0/1909960108 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 157 v157) ==== 95+0+6499 (secure 0 0 0) 0x7f11d0055510 con 0x7f11e4152d60 2024-04-25T00:19:59.583 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:19:59.583 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":157,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7722,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":157}],"filesystems":[{"mdsmap":{"epoch":156,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:19:33.982945+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6586":{"gid":6586,"name":"d","rank":0,"incarnation":153,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/991183935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":991183935},{"type":"v1","addr":"172.21.15.174:6837","nonce":991183935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6586,"qdb_cluster":[6586]},"id":1},{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:19:08.844382+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7542},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7542":{"gid":7542,"name":"c","rank":0,"incarnation":148,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6835/879531976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":879531976},{"type":"v1","addr":"172.21.15.116:6835","nonce":879531976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7542,"qdb_cluster":[7542]},"id":2}]} 2024-04-25T00:19:59.583 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 157 2024-04-25T00:19:59.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.586+0000 7f11eaeac640 1 -- 172.21.15.116:0/1909960108 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f11c80678c0 msgr2=0x7f11c8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:59.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.586+0000 7f11eaeac640 1 --2- 172.21.15.116:0/1909960108 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f11c80678c0 0x7f11c8069d80 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f11d80ad000 tx=0x7f11d8008040 comp rx=0 tx=0).stop 2024-04-25T00:19:59.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.586+0000 7f11eaeac640 1 -- 172.21.15.116:0/1909960108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11e4152d60 msgr2=0x7f11e406da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:19:59.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.586+0000 7f11eaeac640 1 --2- 172.21.15.116:0/1909960108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11e4152d60 0x7f11e406da60 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f11d000c480 tx=0x7f11d000c940 comp rx=0 tx=0).stop 2024-04-25T00:19:59.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.586+0000 7f11eaeac640 1 -- 172.21.15.116:0/1909960108 shutdown_connections 2024-04-25T00:19:59.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.586+0000 7f11eaeac640 1 --2- 172.21.15.116:0/1909960108 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f11c80678c0 0x7f11c8069d80 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:59.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.586+0000 7f11eaeac640 1 --2- 172.21.15.116:0/1909960108 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11e4153140 0x7f11e414dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:59.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.586+0000 7f11eaeac640 1 --2- 172.21.15.116:0/1909960108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11e4152d60 0x7f11e406da60 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:59.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.586+0000 7f11eaeac640 1 --2- 172.21.15.116:0/1909960108 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11e40ab7b0 0x7f11e4157920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:19:59.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.586+0000 7f11eaeac640 1 -- 172.21.15.116:0/1909960108 >> 172.21.15.116:0/1909960108 conn(0x7f11e406f0c0 msgr2=0x7f11e4074480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:19:59.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.586+0000 7f11eaeac640 1 -- 172.21.15.116:0/1909960108 shutdown_connections 2024-04-25T00:19:59.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:19:59.586+0000 7f11eaeac640 1 -- 172.21.15.116:0/1909960108 wait complete. 2024-04-25T00:19:59.602 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:20:01.602 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:20:01.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.742+0000 7f11c4781640 1 Processor -- start 2024-04-25T00:20:01.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.742+0000 7f11c4781640 1 -- start start 2024-04-25T00:20:01.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.742+0000 7f11c4781640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f11bc152d80 0x7f11bc06bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:01.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.742+0000 7f11c4781640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11bc06c290 0x7f11bc157390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:01.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.742+0000 7f11c4781640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11bc0c9150 con 0x7f11bc06d5f0 2024-04-25T00:20:01.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.742+0000 7f11c4781640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f11bc0c9d60 con 0x7f11bc152d80 2024-04-25T00:20:01.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.742+0000 7f11c4781640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f11bc0c9430 con 0x7f11bc06c290 2024-04-25T00:20:01.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.742+0000 7f11c251d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f11bc152d80 0x7f11bc06bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:01.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.742+0000 7f11c1d1c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f11bc06d5f0 0x7f11bc156c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:56768/0 (socket says 172.21.15.116:56768) 2024-04-25T00:20:01.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.742+0000 7f11c1d1c640 1 -- 172.21.15.116:0/1558924002 learned_addr learned my addr 172.21.15.116:0/1558924002 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:20:01.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.742+0000 7f11c2d1e640 1 --2- 172.21.15.116:0/1558924002 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11bc06c290 0x7f11bc157390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:01.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.742+0000 7f11c251d640 1 -- 172.21.15.116:0/1558924002 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11bc06c290 msgr2=0x7f11bc157390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:20:01.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.742+0000 7f11c251d640 1 --2- 172.21.15.116:0/1558924002 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11bc06c290 0x7f11bc157390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:01.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.742+0000 7f11c251d640 1 -- 172.21.15.116:0/1558924002 >> v1:172.21.15.116:6789/0 conn(0x7f11bc06d5f0 legacy=0x7f11bc156c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:01.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.742+0000 7f11c251d640 1 -- 172.21.15.116:0/1558924002 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11bc0c9710 con 0x7f11bc152d80 2024-04-25T00:20:01.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.742+0000 7f11c151b640 1 -- 172.21.15.116:0/1558924002 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3130315922 0 0) 0x7f11bc0c9150 con 0x7f11bc06d5f0 2024-04-25T00:20:01.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.742+0000 7f11c2d1e640 1 --2- 172.21.15.116:0/1558924002 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11bc06c290 0x7f11bc157390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:20:01.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.746+0000 7f11c251d640 1 --2- 172.21.15.116:0/1558924002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11bc152d80 0x7f11bc06bd50 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f11b80097c0 tx=0x7f11b801c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52d66d072bc77497 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:01.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.746+0000 7f11c151b640 1 -- 172.21.15.116:0/1558924002 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11b80ab600 con 0x7f11bc152d80 2024-04-25T00:20:01.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.746+0000 7f11c151b640 1 -- 172.21.15.116:0/1558924002 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11b80b0070 con 0x7f11bc152d80 2024-04-25T00:20:01.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.746+0000 7f11c151b640 1 -- 172.21.15.116:0/1558924002 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11b80ab9c0 con 0x7f11bc152d80 2024-04-25T00:20:01.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.746+0000 7f11c4781640 1 -- 172.21.15.116:0/1558924002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11bc152d80 msgr2=0x7f11bc06bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:01.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.746+0000 7f11c4781640 1 --2- 172.21.15.116:0/1558924002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11bc152d80 0x7f11bc06bd50 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f11b80097c0 tx=0x7f11b801c3e0 comp rx=0 tx=0).stop 2024-04-25T00:20:01.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.746+0000 7f11c4781640 1 -- 172.21.15.116:0/1558924002 shutdown_connections 2024-04-25T00:20:01.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.746+0000 7f11c4781640 1 --2- 172.21.15.116:0/1558924002 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11bc06c290 0x7f11bc157390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:01.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.746+0000 7f11c4781640 1 --2- 172.21.15.116:0/1558924002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11bc152d80 0x7f11bc06bd50 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:01.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.746+0000 7f11c4781640 1 -- 172.21.15.116:0/1558924002 >> 172.21.15.116:0/1558924002 conn(0x7f11bc0acdd0 msgr2=0x7f11bc0ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:20:01.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.746+0000 7f11c4781640 1 -- 172.21.15.116:0/1558924002 shutdown_connections 2024-04-25T00:20:01.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.746+0000 7f11c4781640 1 -- 172.21.15.116:0/1558924002 wait complete. 2024-04-25T00:20:01.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.750+0000 7f11c4781640 1 Processor -- start 2024-04-25T00:20:01.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.750+0000 7f11c4781640 1 -- start start 2024-04-25T00:20:01.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.750+0000 7f11c4781640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11bc06c290 0x7f11bc151810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:01.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.750+0000 7f11c4781640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11bc06d5f0 0x7f11bc0abd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:01.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.750+0000 7f11c4781640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11bc152d80 0x7f11bc1f15e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:01.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.750+0000 7f11c4781640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f11bc0caf00 con 0x7f11bc152d80 2024-04-25T00:20:01.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.750+0000 7f11c4781640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f11bc0cad80 con 0x7f11bc06d5f0 2024-04-25T00:20:01.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.750+0000 7f11c4781640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f11bc1628d0 con 0x7f11bc06c290 2024-04-25T00:20:01.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.750+0000 7f11c2d1e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11bc152d80 0x7f11bc1f15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:01.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.750+0000 7f11c2d1e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11bc152d80 0x7f11bc1f15e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57128/0 (socket says 172.21.15.116:57128) 2024-04-25T00:20:01.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.750+0000 7f11c2d1e640 1 -- 172.21.15.116:0/542727974 learned_addr learned my addr 172.21.15.116:0/542727974 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:20:01.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.750+0000 7f11c251d640 1 --2- 172.21.15.116:0/542727974 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11bc06c290 0x7f11bc151810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:01.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.750+0000 7f11c1d1c640 1 --2- 172.21.15.116:0/542727974 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11bc06d5f0 0x7f11bc0abd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:01.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.750+0000 7f11c2d1e640 1 -- 172.21.15.116:0/542727974 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11bc06c290 msgr2=0x7f11bc151810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:01.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.750+0000 7f11c2d1e640 1 --2- 172.21.15.116:0/542727974 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11bc06c290 0x7f11bc151810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:01.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.750+0000 7f11c2d1e640 1 -- 172.21.15.116:0/542727974 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11bc06d5f0 msgr2=0x7f11bc0abd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:01.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.750+0000 7f11c2d1e640 1 --2- 172.21.15.116:0/542727974 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11bc06d5f0 0x7f11bc0abd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:01.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.750+0000 7f11c2d1e640 1 -- 172.21.15.116:0/542727974 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11bc0be6d0 con 0x7f11bc152d80 2024-04-25T00:20:01.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.750+0000 7f11c2d1e640 1 --2- 172.21.15.116:0/542727974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11bc152d80 0x7f11bc1f15e0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f11b400cd50 tx=0x7f11b400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:01.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.750+0000 7f11b2ffd640 1 -- 172.21.15.116:0/542727974 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11b400f680 con 0x7f11bc152d80 2024-04-25T00:20:01.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.750+0000 7f11b2ffd640 1 -- 172.21.15.116:0/542727974 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11b400f840 con 0x7f11bc152d80 2024-04-25T00:20:01.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.750+0000 7f11c4781640 1 -- 172.21.15.116:0/542727974 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11bc151010 con 0x7f11bc152d80 2024-04-25T00:20:01.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.750+0000 7f11b2ffd640 1 -- 172.21.15.116:0/542727974 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11b400fc30 con 0x7f11bc152d80 2024-04-25T00:20:01.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.750+0000 7f11c4781640 1 -- 172.21.15.116:0/542727974 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11bc151db0 con 0x7f11bc152d80 2024-04-25T00:20:01.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.750+0000 7f11c4781640 1 -- 172.21.15.116:0/542727974 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11bc068350 con 0x7f11bc152d80 2024-04-25T00:20:01.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.750+0000 7f11b2ffd640 1 -- 172.21.15.116:0/542727974 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f11b40213f0 con 0x7f11bc152d80 2024-04-25T00:20:01.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.754+0000 7f11b2ffd640 1 --2- 172.21.15.116:0/542727974 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f11a00678f0 0x7f11a0069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:01.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.754+0000 7f11b2ffd640 1 -- 172.21.15.116:0/542727974 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 9160+0+0 (secure 0 0 0) 0x7f11b40877c0 con 0x7f11bc152d80 2024-04-25T00:20:01.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.754+0000 7f11c251d640 1 --2- 172.21.15.116:0/542727974 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f11a00678f0 0x7f11a0069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:01.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.754+0000 7f11c251d640 1 --2- 172.21.15.116:0/542727974 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f11a00678f0 0x7f11a0069db0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f11b80af000 tx=0x7f11b80a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:01.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:01.758+0000 7f11b2ffd640 1 -- 172.21.15.116:0/542727974 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f11b4014030 con 0x7f11bc152d80 2024-04-25T00:20:02.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:02.002+0000 7f11c4781640 1 -- 172.21.15.116:0/542727974 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f11bc0cab60 con 0x7f11bc152d80 2024-04-25T00:20:02.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:02.002+0000 7f11b2ffd640 1 -- 172.21.15.116:0/542727974 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 157 v157) ==== 95+0+6499 (secure 0 0 0) 0x7f11b4057530 con 0x7f11bc152d80 2024-04-25T00:20:02.001 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:20:02.001 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":157,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7722,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":157}],"filesystems":[{"mdsmap":{"epoch":156,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:19:33.982945+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6586":{"gid":6586,"name":"d","rank":0,"incarnation":153,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/991183935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":991183935},{"type":"v1","addr":"172.21.15.174:6837","nonce":991183935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6586,"qdb_cluster":[6586]},"id":1},{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:19:08.844382+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7542},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7542":{"gid":7542,"name":"c","rank":0,"incarnation":148,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6835/879531976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":879531976},{"type":"v1","addr":"172.21.15.116:6835","nonce":879531976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7542,"qdb_cluster":[7542]},"id":2}]} 2024-04-25T00:20:02.001 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 157 2024-04-25T00:20:02.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:02.002+0000 7f11c4781640 1 -- 172.21.15.116:0/542727974 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f11a00678f0 msgr2=0x7f11a0069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:02.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:02.002+0000 7f11c4781640 1 --2- 172.21.15.116:0/542727974 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f11a00678f0 0x7f11a0069db0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f11b80af000 tx=0x7f11b80a7000 comp rx=0 tx=0).stop 2024-04-25T00:20:02.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:02.002+0000 7f11c4781640 1 -- 172.21.15.116:0/542727974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11bc152d80 msgr2=0x7f11bc1f15e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:02.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:02.002+0000 7f11c4781640 1 --2- 172.21.15.116:0/542727974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11bc152d80 0x7f11bc1f15e0 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f11b400cd50 tx=0x7f11b400a650 comp rx=0 tx=0).stop 2024-04-25T00:20:02.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:02.002+0000 7f11c4781640 1 -- 172.21.15.116:0/542727974 shutdown_connections 2024-04-25T00:20:02.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:02.002+0000 7f11c4781640 1 --2- 172.21.15.116:0/542727974 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f11a00678f0 0x7f11a0069db0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:02.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:02.002+0000 7f11c4781640 1 --2- 172.21.15.116:0/542727974 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11bc152d80 0x7f11bc1f15e0 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:02.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:02.002+0000 7f11c4781640 1 --2- 172.21.15.116:0/542727974 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11bc06d5f0 0x7f11bc0abd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:02.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:02.002+0000 7f11c4781640 1 --2- 172.21.15.116:0/542727974 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11bc06c290 0x7f11bc151810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:02.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:02.002+0000 7f11c4781640 1 -- 172.21.15.116:0/542727974 >> 172.21.15.116:0/542727974 conn(0x7f11bc0acdd0 msgr2=0x7f11bc14ed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:20:02.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:02.002+0000 7f11c4781640 1 -- 172.21.15.116:0/542727974 shutdown_connections 2024-04-25T00:20:02.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:02.002+0000 7f11c4781640 1 -- 172.21.15.116:0/542727974 wait complete. 2024-04-25T00:20:02.022 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:20:04.023 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:20:04.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.150+0000 7fe9967cd640 1 Processor -- start 2024-04-25T00:20:04.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.150+0000 7fe9967cd640 1 -- start start 2024-04-25T00:20:04.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.150+0000 7fe9967cd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe990071d90 0x7fe9901563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:04.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.150+0000 7fe9967cd640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe990071220 con 0x7fe9900a94a0 2024-04-25T00:20:04.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.150+0000 7fe9967cd640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe990152420 con 0x7fe99014c720 2024-04-25T00:20:04.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.150+0000 7fe9967cd640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fe990153460 con 0x7fe990071d90 2024-04-25T00:20:04.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.150+0000 7fe994d6a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe990071d90 0x7fe9901563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:04.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.150+0000 7fe98f7fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fe9900a94a0 0x7fe990071680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:56774/0 (socket says 172.21.15.116:56774) 2024-04-25T00:20:04.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.150+0000 7fe98f7fe640 1 -- 172.21.15.116:0/1622611195 learned_addr learned my addr 172.21.15.116:0/1622611195 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:20:04.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.154+0000 7fe994d6a640 1 -- 172.21.15.116:0/1622611195 >> v1:172.21.15.174:6789/0 conn(0x7fe99014c720 legacy=0x7fe9900c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:04.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.154+0000 7fe994d6a640 1 -- 172.21.15.116:0/1622611195 >> v1:172.21.15.116:6789/0 conn(0x7fe9900a94a0 legacy=0x7fe990071680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:04.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.154+0000 7fe994d6a640 1 -- 172.21.15.116:0/1622611195 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9900721c0 con 0x7fe990071d90 2024-04-25T00:20:04.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.154+0000 7fe98effd640 1 -- 172.21.15.116:0/1622611195 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1160734038 0 0) 0x7fe990071220 con 0x7fe9900a94a0 2024-04-25T00:20:04.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.154+0000 7fe994d6a640 1 --2- 172.21.15.116:0/1622611195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe990071d90 0x7fe9901563d0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7fe9880097c0 tx=0x7fe98801c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d514cac93bcb25ce server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:04.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.154+0000 7fe98effd640 1 -- 172.21.15.116:0/1622611195 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9880ab610 con 0x7fe990071d90 2024-04-25T00:20:04.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.154+0000 7fe98effd640 1 -- 172.21.15.116:0/1622611195 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9880b0070 con 0x7fe990071d90 2024-04-25T00:20:04.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.154+0000 7fe98effd640 1 -- 172.21.15.116:0/1622611195 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9880ab9d0 con 0x7fe990071d90 2024-04-25T00:20:04.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.154+0000 7fe9967cd640 1 -- 172.21.15.116:0/1622611195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe990071d90 msgr2=0x7fe9901563d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:04.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.154+0000 7fe9967cd640 1 --2- 172.21.15.116:0/1622611195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe990071d90 0x7fe9901563d0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7fe9880097c0 tx=0x7fe98801c410 comp rx=0 tx=0).stop 2024-04-25T00:20:04.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.154+0000 7fe9967cd640 1 -- 172.21.15.116:0/1622611195 shutdown_connections 2024-04-25T00:20:04.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.154+0000 7fe9967cd640 1 --2- 172.21.15.116:0/1622611195 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe990071d90 0x7fe9901563d0 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:04.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.154+0000 7fe9967cd640 1 -- 172.21.15.116:0/1622611195 >> 172.21.15.116:0/1622611195 conn(0x7fe9900c3b00 msgr2=0x7fe9900c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:20:04.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.154+0000 7fe9967cd640 1 -- 172.21.15.116:0/1622611195 shutdown_connections 2024-04-25T00:20:04.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.154+0000 7fe9967cd640 1 -- 172.21.15.116:0/1622611195 wait complete. 2024-04-25T00:20:04.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.158+0000 7fe9967cd640 1 Processor -- start 2024-04-25T00:20:04.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.158+0000 7fe9967cd640 1 -- start start 2024-04-25T00:20:04.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.158+0000 7fe9967cd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe990071d90 0x7fe99013a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:04.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.158+0000 7fe9967cd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9900a94a0 0x7fe9900ac7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:04.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.158+0000 7fe9967cd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe99014c720 0x7fe99013def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:04.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.158+0000 7fe9967cd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe990152f40 con 0x7fe99014c720 2024-04-25T00:20:04.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.158+0000 7fe9967cd640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe990152c60 con 0x7fe9900a94a0 2024-04-25T00:20:04.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.158+0000 7fe9967cd640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe9900c3120 con 0x7fe990071d90 2024-04-25T00:20:04.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.158+0000 7fe994d6a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe99014c720 0x7fe99013def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:04.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.158+0000 7fe994d6a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe99014c720 0x7fe99013def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57158/0 (socket says 172.21.15.116:57158) 2024-04-25T00:20:04.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.158+0000 7fe994d6a640 1 -- 172.21.15.116:0/2612990196 learned_addr learned my addr 172.21.15.116:0/2612990196 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:20:04.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.158+0000 7fe98f7fe640 1 --2- 172.21.15.116:0/2612990196 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9900a94a0 0x7fe9900ac7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:04.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.158+0000 7fe98ffff640 1 --2- 172.21.15.116:0/2612990196 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe990071d90 0x7fe99013a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:04.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.158+0000 7fe994d6a640 1 -- 172.21.15.116:0/2612990196 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe990071d90 msgr2=0x7fe99013a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:04.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.158+0000 7fe994d6a640 1 --2- 172.21.15.116:0/2612990196 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe990071d90 0x7fe99013a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:04.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.158+0000 7fe994d6a640 1 -- 172.21.15.116:0/2612990196 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9900a94a0 msgr2=0x7fe9900ac7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:04.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.158+0000 7fe994d6a640 1 --2- 172.21.15.116:0/2612990196 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9900a94a0 0x7fe9900ac7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:04.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.158+0000 7fe994d6a640 1 -- 172.21.15.116:0/2612990196 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe990140380 con 0x7fe99014c720 2024-04-25T00:20:04.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.158+0000 7fe98f7fe640 1 --2- 172.21.15.116:0/2612990196 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9900a94a0 0x7fe9900ac7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:20:04.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.158+0000 7fe98ffff640 1 --2- 172.21.15.116:0/2612990196 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe990071d90 0x7fe99013a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:20:04.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.158+0000 7fe994d6a640 1 --2- 172.21.15.116:0/2612990196 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe99014c720 0x7fe99013def0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7fe988019020 tx=0x7fe98801ac50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:04.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.158+0000 7fe98cff9640 1 -- 172.21.15.116:0/2612990196 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9880bf610 con 0x7fe99014c720 2024-04-25T00:20:04.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.158+0000 7fe98cff9640 1 -- 172.21.15.116:0/2612990196 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9880bf7b0 con 0x7fe99014c720 2024-04-25T00:20:04.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.158+0000 7fe9967cd640 1 -- 172.21.15.116:0/2612990196 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe990140660 con 0x7fe99014c720 2024-04-25T00:20:04.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.158+0000 7fe98cff9640 1 -- 172.21.15.116:0/2612990196 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9880bfba0 con 0x7fe99014c720 2024-04-25T00:20:04.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.158+0000 7fe9967cd640 1 -- 172.21.15.116:0/2612990196 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9901effe0 con 0x7fe99014c720 2024-04-25T00:20:04.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.158+0000 7fe9967cd640 1 -- 172.21.15.116:0/2612990196 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe990068350 con 0x7fe99014c720 2024-04-25T00:20:04.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.162+0000 7fe98cff9640 1 -- 172.21.15.116:0/2612990196 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe9880bd020 con 0x7fe99014c720 2024-04-25T00:20:04.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.162+0000 7fe98cff9640 1 --2- 172.21.15.116:0/2612990196 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe97c0678c0 0x7fe97c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:04.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.162+0000 7fe98cff9640 1 -- 172.21.15.116:0/2612990196 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 9160+0+0 (secure 0 0 0) 0x7fe9881213a0 con 0x7fe99014c720 2024-04-25T00:20:04.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.162+0000 7fe98ffff640 1 --2- 172.21.15.116:0/2612990196 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe97c0678c0 0x7fe97c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:04.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.162+0000 7fe98ffff640 1 --2- 172.21.15.116:0/2612990196 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe97c0678c0 0x7fe97c069d80 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fe980007fd0 tx=0x7fe980009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:04.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.166+0000 7fe98cff9640 1 -- 172.21.15.116:0/2612990196 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fe9880a8050 con 0x7fe99014c720 2024-04-25T00:20:04.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.418+0000 7fe9967cd640 1 -- 172.21.15.116:0/2612990196 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe990152200 con 0x7fe99014c720 2024-04-25T00:20:04.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.418+0000 7fe98cff9640 1 -- 172.21.15.116:0/2612990196 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 157 v157) ==== 95+0+6499 (secure 0 0 0) 0x7fe988016090 con 0x7fe99014c720 2024-04-25T00:20:04.420 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:20:04.420 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":157,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7722,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":157}],"filesystems":[{"mdsmap":{"epoch":156,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:19:33.982945+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6586":{"gid":6586,"name":"d","rank":0,"incarnation":153,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/991183935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":991183935},{"type":"v1","addr":"172.21.15.174:6837","nonce":991183935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6586,"qdb_cluster":[6586]},"id":1},{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:19:08.844382+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7542},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7542":{"gid":7542,"name":"c","rank":0,"incarnation":148,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6835/879531976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":879531976},{"type":"v1","addr":"172.21.15.116:6835","nonce":879531976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7542,"qdb_cluster":[7542]},"id":2}]} 2024-04-25T00:20:04.420 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 157 2024-04-25T00:20:04.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.422+0000 7fe9967cd640 1 -- 172.21.15.116:0/2612990196 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe97c0678c0 msgr2=0x7fe97c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:04.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.422+0000 7fe9967cd640 1 --2- 172.21.15.116:0/2612990196 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe97c0678c0 0x7fe97c069d80 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fe980007fd0 tx=0x7fe980009040 comp rx=0 tx=0).stop 2024-04-25T00:20:04.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.422+0000 7fe9967cd640 1 -- 172.21.15.116:0/2612990196 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe99014c720 msgr2=0x7fe99013def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:04.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.422+0000 7fe9967cd640 1 --2- 172.21.15.116:0/2612990196 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe99014c720 0x7fe99013def0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7fe988019020 tx=0x7fe98801ac50 comp rx=0 tx=0).stop 2024-04-25T00:20:04.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.422+0000 7fe9967cd640 1 -- 172.21.15.116:0/2612990196 shutdown_connections 2024-04-25T00:20:04.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.422+0000 7fe9967cd640 1 --2- 172.21.15.116:0/2612990196 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe97c0678c0 0x7fe97c069d80 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:04.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.422+0000 7fe9967cd640 1 --2- 172.21.15.116:0/2612990196 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe99014c720 0x7fe99013def0 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:04.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.422+0000 7fe9967cd640 1 --2- 172.21.15.116:0/2612990196 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe9900a94a0 0x7fe9900ac7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:04.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.422+0000 7fe9967cd640 1 --2- 172.21.15.116:0/2612990196 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe990071d90 0x7fe99013a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:04.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.422+0000 7fe9967cd640 1 -- 172.21.15.116:0/2612990196 >> 172.21.15.116:0/2612990196 conn(0x7fe9900c3b00 msgr2=0x7fe9900c3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:20:04.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.422+0000 7fe9967cd640 1 -- 172.21.15.116:0/2612990196 shutdown_connections 2024-04-25T00:20:04.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:04.422+0000 7fe9967cd640 1 -- 172.21.15.116:0/2612990196 wait complete. 2024-04-25T00:20:04.439 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:20:06.440 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:20:06.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.577+0000 7f2885c9a640 1 Processor -- start 2024-04-25T00:20:06.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.577+0000 7f2885c9a640 1 -- start start 2024-04-25T00:20:06.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.577+0000 7f2885c9a640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f28800c5a60 0x7f28800c5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:06.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.577+0000 7f2885c9a640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2880158eb0 con 0x7f28801558c0 2024-04-25T00:20:06.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.577+0000 7f2885c9a640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2880159190 con 0x7f2880152db0 2024-04-25T00:20:06.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.577+0000 7f2885c9a640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f2880159470 con 0x7f28800c5a60 2024-04-25T00:20:06.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.577+0000 7f287ffff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f28801558c0 0x7f2880155cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:52252/0 (socket says 172.21.15.116:52252) 2024-04-25T00:20:06.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.577+0000 7f287ffff640 1 -- 172.21.15.116:0/3435762750 learned_addr learned my addr 172.21.15.116:0/3435762750 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:20:06.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.577+0000 7f287effd640 1 --2- 172.21.15.116:0/3435762750 >> v2:172.21.15.174:3300/0 conn(0x7f28800c5a60 0x7f28800c5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:06.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.577+0000 7f287e7fc640 1 -- 172.21.15.116:0/3435762750 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2271674197 0 0) 0x7f2880158eb0 con 0x7f28801558c0 2024-04-25T00:20:06.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.577+0000 7f287e7fc640 1 -- 172.21.15.116:0/3435762750 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2868003530 con 0x7f28801558c0 2024-04-25T00:20:06.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.577+0000 7f287e7fc640 1 -- 172.21.15.116:0/3435762750 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1183183724 0 0) 0x7f2868003530 con 0x7f28801558c0 2024-04-25T00:20:06.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.577+0000 7f287e7fc640 1 -- 172.21.15.116:0/3435762750 >> v2:172.21.15.174:3300/0 conn(0x7f28800c5a60 msgr2=0x7f28800c5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:20:06.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.577+0000 7f287e7fc640 1 --2- 172.21.15.116:0/3435762750 >> v2:172.21.15.174:3300/0 conn(0x7f28800c5a60 0x7f28800c5ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:06.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.577+0000 7f287e7fc640 1 -- 172.21.15.116:0/3435762750 >> v1:172.21.15.174:6789/0 conn(0x7f2880152db0 legacy=0x7f28800c7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:06.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.577+0000 7f287e7fc640 1 -- 172.21.15.116:0/3435762750 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2880159750 con 0x7f28801558c0 2024-04-25T00:20:06.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.577+0000 7f287e7fc640 1 -- 172.21.15.116:0/3435762750 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f2878001a00 con 0x7f28801558c0 2024-04-25T00:20:06.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.577+0000 7f287effd640 1 --2- 172.21.15.116:0/3435762750 >> v2:172.21.15.174:3300/0 conn(0x7f28800c5a60 0x7f28800c5ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:20:06.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.577+0000 7f287e7fc640 1 -- 172.21.15.116:0/3435762750 >> v1:172.21.15.116:6789/0 conn(0x7f28801558c0 legacy=0x7f2880155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:06.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.577+0000 7f287e7fc640 1 --2- 172.21.15.116:0/3435762750 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28680039c0 0x7f2868003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:06.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.577+0000 7f287e7fc640 1 --2- 172.21.15.116:0/3435762750 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2868004810 0x7f2868004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:06.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.577+0000 7f287e7fc640 1 --2- 172.21.15.116:0/3435762750 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28680052f0 0x7f286800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:06.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.577+0000 7f287e7fc640 1 -- 172.21.15.116:0/3435762750 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2880159470 con 0x7f28680052f0 2024-04-25T00:20:06.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.577+0000 7f287e7fc640 1 -- 172.21.15.116:0/3435762750 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f286800c1c0 con 0x7f2868004810 2024-04-25T00:20:06.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.577+0000 7f287e7fc640 1 -- 172.21.15.116:0/3435762750 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f286800c3c0 con 0x7f28680039c0 2024-04-25T00:20:06.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.577+0000 7f287ffff640 1 --2- 172.21.15.116:0/3435762750 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28680052f0 0x7f286800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:06.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.577+0000 7f287effd640 1 --2- 172.21.15.116:0/3435762750 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2868004810 0x7f2868004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:06.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.577+0000 7f287f7fe640 1 --2- 172.21.15.116:0/3435762750 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28680039c0 0x7f2868003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:06.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.577+0000 7f287ffff640 1 -- 172.21.15.116:0/3435762750 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28680039c0 msgr2=0x7f2868003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:20:06.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.577+0000 7f287ffff640 1 --2- 172.21.15.116:0/3435762750 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28680039c0 0x7f2868003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:06.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.577+0000 7f287ffff640 1 -- 172.21.15.116:0/3435762750 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2868004810 msgr2=0x7f2868004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:20:06.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.577+0000 7f287ffff640 1 --2- 172.21.15.116:0/3435762750 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2868004810 0x7f2868004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:06.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.577+0000 7f287ffff640 1 -- 172.21.15.116:0/3435762750 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2868003530 con 0x7f28680052f0 2024-04-25T00:20:06.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.577+0000 7f287effd640 1 --2- 172.21.15.116:0/3435762750 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2868004810 0x7f2868004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:20:06.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.581+0000 7f287ffff640 1 -- 172.21.15.116:0/3435762750 reap_dead start 2024-04-25T00:20:06.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.581+0000 7f287ffff640 1 --2- 172.21.15.116:0/3435762750 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28680052f0 0x7f286800b9d0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f287800a810 tx=0x7f287801faa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4653cf493750256 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:06.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.581+0000 7f287e7fc640 1 -- 172.21.15.116:0/3435762750 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f287800a460 con 0x7f28680052f0 2024-04-25T00:20:06.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.581+0000 7f287e7fc640 1 -- 172.21.15.116:0/3435762750 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28780aa6f0 con 0x7f28680052f0 2024-04-25T00:20:06.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.581+0000 7f287e7fc640 1 -- 172.21.15.116:0/3435762750 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28780aaae0 con 0x7f28680052f0 2024-04-25T00:20:06.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.581+0000 7f2885c9a640 1 -- 172.21.15.116:0/3435762750 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28680052f0 msgr2=0x7f286800b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:06.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.581+0000 7f2885c9a640 1 --2- 172.21.15.116:0/3435762750 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28680052f0 0x7f286800b9d0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f287800a810 tx=0x7f287801faa0 comp rx=0 tx=0).stop 2024-04-25T00:20:06.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.581+0000 7f2885c9a640 1 -- 172.21.15.116:0/3435762750 shutdown_connections 2024-04-25T00:20:06.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.581+0000 7f2885c9a640 1 --2- 172.21.15.116:0/3435762750 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28680052f0 0x7f286800b9d0 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:06.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.581+0000 7f2885c9a640 1 -- 172.21.15.116:0/3435762750 >> 172.21.15.116:0/3435762750 conn(0x7f28800acc10 msgr2=0x7f28800ad020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:20:06.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.581+0000 7f2885c9a640 1 -- 172.21.15.116:0/3435762750 shutdown_connections 2024-04-25T00:20:06.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.581+0000 7f2885c9a640 1 -- 172.21.15.116:0/3435762750 wait complete. 2024-04-25T00:20:06.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.585+0000 7f2885c9a640 1 Processor -- start 2024-04-25T00:20:06.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.585+0000 7f2885c9a640 1 -- start start 2024-04-25T00:20:06.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.585+0000 7f2885c9a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28680052f0 0x7f2880151a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:06.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.585+0000 7f2885c9a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28801ef290 0x7f28800abd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:06.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.585+0000 7f2885c9a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28801467e0 0x7f2880146c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:06.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.585+0000 7f2885c9a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2880153190 con 0x7f28801467e0 2024-04-25T00:20:06.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.585+0000 7f2885c9a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f28800cadf0 con 0x7f28680052f0 2024-04-25T00:20:06.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.585+0000 7f2885c9a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f2880153310 con 0x7f28801ef290 2024-04-25T00:20:06.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.585+0000 7f287ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28801467e0 0x7f2880146c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:06.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.585+0000 7f287ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28801467e0 0x7f2880146c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48974/0 (socket says 172.21.15.116:48974) 2024-04-25T00:20:06.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.585+0000 7f287ffff640 1 -- 172.21.15.116:0/3601377564 learned_addr learned my addr 172.21.15.116:0/3601377564 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:20:06.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.585+0000 7f287effd640 1 --2- 172.21.15.116:0/3601377564 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28801ef290 0x7f28800abd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:06.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.585+0000 7f287f7fe640 1 --2- 172.21.15.116:0/3601377564 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28680052f0 0x7f2880151a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:06.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.585+0000 7f287ffff640 1 -- 172.21.15.116:0/3601377564 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28801ef290 msgr2=0x7f28800abd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:06.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.585+0000 7f287ffff640 1 --2- 172.21.15.116:0/3601377564 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28801ef290 0x7f28800abd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:06.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.585+0000 7f287ffff640 1 -- 172.21.15.116:0/3601377564 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28680052f0 msgr2=0x7f2880151a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:06.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.585+0000 7f287ffff640 1 --2- 172.21.15.116:0/3601377564 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28680052f0 0x7f2880151a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:06.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.585+0000 7f287ffff640 1 -- 172.21.15.116:0/3601377564 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2880151f90 con 0x7f28801467e0 2024-04-25T00:20:06.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.585+0000 7f287ffff640 1 --2- 172.21.15.116:0/3601377564 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28801467e0 0x7f2880146c70 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f2878020340 tx=0x7f2868005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:06.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.585+0000 7f2867fff640 1 -- 172.21.15.116:0/3601377564 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2878001fc0 con 0x7f28801467e0 2024-04-25T00:20:06.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.585+0000 7f2867fff640 1 -- 172.21.15.116:0/3601377564 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28780bb070 con 0x7f28801467e0 2024-04-25T00:20:06.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.585+0000 7f2885c9a640 1 -- 172.21.15.116:0/3601377564 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2880152260 con 0x7f28801467e0 2024-04-25T00:20:06.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.585+0000 7f2867fff640 1 -- 172.21.15.116:0/3601377564 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28780aaaa0 con 0x7f28801467e0 2024-04-25T00:20:06.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.585+0000 7f2885c9a640 1 -- 172.21.15.116:0/3601377564 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28801eea90 con 0x7f28801467e0 2024-04-25T00:20:06.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.585+0000 7f2885c9a640 1 -- 172.21.15.116:0/3601377564 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2880068a60 con 0x7f28801467e0 2024-04-25T00:20:06.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.585+0000 7f2867fff640 1 -- 172.21.15.116:0/3601377564 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f28780aac40 con 0x7f28801467e0 2024-04-25T00:20:06.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.585+0000 7f2867fff640 1 --2- 172.21.15.116:0/3601377564 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f286c0678f0 0x7f286c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:06.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.589+0000 7f2867fff640 1 -- 172.21.15.116:0/3601377564 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 9160+0+0 (secure 0 0 0) 0x7f28781241e0 con 0x7f28801467e0 2024-04-25T00:20:06.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.589+0000 7f287f7fe640 1 --2- 172.21.15.116:0/3601377564 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f286c0678f0 0x7f286c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:06.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.589+0000 7f287f7fe640 1 --2- 172.21.15.116:0/3601377564 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f286c0678f0 0x7f286c069db0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f2870000c10 tx=0x7f2870009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:06.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.593+0000 7f2867fff640 1 -- 172.21.15.116:0/3601377564 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f28780197b0 con 0x7f28801467e0 2024-04-25T00:20:06.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.817+0000 7f2885c9a640 1 -- 172.21.15.116:0/3601377564 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f28800c9ff0 con 0x7f28801467e0 2024-04-25T00:20:06.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.817+0000 7f2867fff640 1 -- 172.21.15.116:0/3601377564 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 157 v157) ==== 95+0+6499 (secure 0 0 0) 0x7f28780f8e20 con 0x7f28801467e0 2024-04-25T00:20:06.817 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:20:06.818 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":157,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7722,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":157}],"filesystems":[{"mdsmap":{"epoch":156,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:19:33.982945+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6586":{"gid":6586,"name":"d","rank":0,"incarnation":153,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/991183935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":991183935},{"type":"v1","addr":"172.21.15.174:6837","nonce":991183935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6586,"qdb_cluster":[6586]},"id":1},{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:19:08.844382+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7542},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7542":{"gid":7542,"name":"c","rank":0,"incarnation":148,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6835/879531976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":879531976},{"type":"v1","addr":"172.21.15.116:6835","nonce":879531976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7542,"qdb_cluster":[7542]},"id":2}]} 2024-04-25T00:20:06.818 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 157 2024-04-25T00:20:06.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.817+0000 7f2885c9a640 1 -- 172.21.15.116:0/3601377564 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f286c0678f0 msgr2=0x7f286c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:06.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.817+0000 7f2885c9a640 1 --2- 172.21.15.116:0/3601377564 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f286c0678f0 0x7f286c069db0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f2870000c10 tx=0x7f2870009040 comp rx=0 tx=0).stop 2024-04-25T00:20:06.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.817+0000 7f2885c9a640 1 -- 172.21.15.116:0/3601377564 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28801467e0 msgr2=0x7f2880146c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:06.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.817+0000 7f2885c9a640 1 --2- 172.21.15.116:0/3601377564 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28801467e0 0x7f2880146c70 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f2878020340 tx=0x7f2868005840 comp rx=0 tx=0).stop 2024-04-25T00:20:06.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.821+0000 7f2885c9a640 1 -- 172.21.15.116:0/3601377564 shutdown_connections 2024-04-25T00:20:06.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.821+0000 7f2885c9a640 1 --2- 172.21.15.116:0/3601377564 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f286c0678f0 0x7f286c069db0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:06.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.821+0000 7f2885c9a640 1 --2- 172.21.15.116:0/3601377564 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28801467e0 0x7f2880146c70 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:06.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.821+0000 7f2885c9a640 1 --2- 172.21.15.116:0/3601377564 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28801ef290 0x7f28800abd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:06.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.821+0000 7f2885c9a640 1 --2- 172.21.15.116:0/3601377564 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28680052f0 0x7f2880151a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:06.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.821+0000 7f2885c9a640 1 -- 172.21.15.116:0/3601377564 >> 172.21.15.116:0/3601377564 conn(0x7f28800acc10 msgr2=0x7f28800c6ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:20:06.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.821+0000 7f2885c9a640 1 -- 172.21.15.116:0/3601377564 shutdown_connections 2024-04-25T00:20:06.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:06.821+0000 7f2885c9a640 1 -- 172.21.15.116:0/3601377564 wait complete. 2024-04-25T00:20:06.840 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:20:08.841 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:20:08.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.981+0000 7fb61d606640 1 Processor -- start 2024-04-25T00:20:08.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.981+0000 7fb61d606640 1 -- start start 2024-04-25T00:20:08.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.985+0000 7fb61d606640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb618071520 0x7fb618155fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:08.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.985+0000 7fb61d606640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb61814f7a0 0x7fb61814fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:08.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.985+0000 7fb61d606640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6180737d0 con 0x7fb618155c00 2024-04-25T00:20:08.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.985+0000 7fb61d606640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fb618071bb0 con 0x7fb61814f7a0 2024-04-25T00:20:08.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.985+0000 7fb61d606640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fb618072ee0 con 0x7fb618071520 2024-04-25T00:20:08.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.985+0000 7fb616ffd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fb618155c00 0x7fb6180730c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:52260/0 (socket says 172.21.15.116:52260) 2024-04-25T00:20:08.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.985+0000 7fb6177fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb61814f7a0 0x7fb61814fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:08.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.985+0000 7fb616ffd640 1 -- 172.21.15.116:0/1467839486 learned_addr learned my addr 172.21.15.116:0/1467839486 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:20:08.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.985+0000 7fb6167fc640 1 --2- 172.21.15.116:0/1467839486 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb618071520 0x7fb618155fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:08.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.985+0000 7fb6177fe640 1 -- 172.21.15.116:0/1467839486 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb618071520 msgr2=0x7fb618155fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:20:08.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.985+0000 7fb6177fe640 1 --2- 172.21.15.116:0/1467839486 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb618071520 0x7fb618155fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:08.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.985+0000 7fb6177fe640 1 -- 172.21.15.116:0/1467839486 >> v1:172.21.15.116:6789/0 conn(0x7fb618155c00 legacy=0x7fb6180730c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:08.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.985+0000 7fb6177fe640 1 -- 172.21.15.116:0/1467839486 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb618072920 con 0x7fb61814f7a0 2024-04-25T00:20:08.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.985+0000 7fb615ffb640 1 -- 172.21.15.116:0/1467839486 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1178193885 0 0) 0x7fb6180737d0 con 0x7fb618155c00 2024-04-25T00:20:08.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.985+0000 7fb6177fe640 1 --2- 172.21.15.116:0/1467839486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb61814f7a0 0x7fb61814fbf0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7fb610009d90 tx=0x7fb61001c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69333a941fca5eef server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:08.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.985+0000 7fb615ffb640 1 -- 172.21.15.116:0/1467839486 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6100b2070 con 0x7fb61814f7a0 2024-04-25T00:20:08.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.985+0000 7fb615ffb640 1 -- 172.21.15.116:0/1467839486 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6100a5710 con 0x7fb61814f7a0 2024-04-25T00:20:08.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.985+0000 7fb615ffb640 1 -- 172.21.15.116:0/1467839486 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6100a5ae0 con 0x7fb61814f7a0 2024-04-25T00:20:08.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.989+0000 7fb61d606640 1 -- 172.21.15.116:0/1467839486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb61814f7a0 msgr2=0x7fb61814fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:08.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.989+0000 7fb61d606640 1 --2- 172.21.15.116:0/1467839486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb61814f7a0 0x7fb61814fbf0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7fb610009d90 tx=0x7fb61001c850 comp rx=0 tx=0).stop 2024-04-25T00:20:08.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.989+0000 7fb61d606640 1 -- 172.21.15.116:0/1467839486 shutdown_connections 2024-04-25T00:20:08.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.989+0000 7fb61d606640 1 --2- 172.21.15.116:0/1467839486 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb61814f7a0 0x7fb61814fbf0 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:08.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.989+0000 7fb61d606640 1 --2- 172.21.15.116:0/1467839486 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb618071520 0x7fb618155fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:08.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.989+0000 7fb61d606640 1 -- 172.21.15.116:0/1467839486 >> 172.21.15.116:0/1467839486 conn(0x7fb6180c4200 msgr2=0x7fb6180c4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:20:08.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.989+0000 7fb61d606640 1 -- 172.21.15.116:0/1467839486 shutdown_connections 2024-04-25T00:20:08.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.989+0000 7fb61d606640 1 -- 172.21.15.116:0/1467839486 wait complete. 2024-04-25T00:20:08.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.989+0000 7fb61d606640 1 Processor -- start 2024-04-25T00:20:08.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.989+0000 7fb61d606640 1 -- start start 2024-04-25T00:20:08.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.989+0000 7fb61d606640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb618071520 0x7fb618071f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:08.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.989+0000 7fb61d606640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb61814f7a0 0x7fb6181ee630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:08.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.989+0000 7fb61d606640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb618155c00 0x7fb6181eeb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:08.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.989+0000 7fb61d606640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb6180a7520 con 0x7fb618155c00 2024-04-25T00:20:08.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.989+0000 7fb61d606640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb61806c950 con 0x7fb618071520 2024-04-25T00:20:08.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.989+0000 7fb61d606640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fb6180c2100 con 0x7fb61814f7a0 2024-04-25T00:20:08.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.989+0000 7fb6177fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb618155c00 0x7fb6181eeb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:08.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.989+0000 7fb6177fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb618155c00 0x7fb6181eeb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48982/0 (socket says 172.21.15.116:48982) 2024-04-25T00:20:08.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.989+0000 7fb6177fe640 1 -- 172.21.15.116:0/2134634498 learned_addr learned my addr 172.21.15.116:0/2134634498 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:20:08.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.989+0000 7fb6167fc640 1 --2- 172.21.15.116:0/2134634498 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb61814f7a0 0x7fb6181ee630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:08.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.989+0000 7fb616ffd640 1 --2- 172.21.15.116:0/2134634498 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb618071520 0x7fb618071f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:08.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.989+0000 7fb6177fe640 1 -- 172.21.15.116:0/2134634498 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb61814f7a0 msgr2=0x7fb6181ee630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:08.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.989+0000 7fb6177fe640 1 --2- 172.21.15.116:0/2134634498 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb61814f7a0 0x7fb6181ee630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:08.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.989+0000 7fb6177fe640 1 -- 172.21.15.116:0/2134634498 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb618071520 msgr2=0x7fb618071f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:08.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.989+0000 7fb6177fe640 1 --2- 172.21.15.116:0/2134634498 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb618071520 0x7fb618071f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:08.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.989+0000 7fb6177fe640 1 -- 172.21.15.116:0/2134634498 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6181f1560 con 0x7fb618155c00 2024-04-25T00:20:08.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.989+0000 7fb6167fc640 1 --2- 172.21.15.116:0/2134634498 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb61814f7a0 0x7fb6181ee630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:20:08.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.993+0000 7fb6177fe640 1 --2- 172.21.15.116:0/2134634498 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb618155c00 0x7fb6181eeb70 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7fb6100b0040 tx=0x7fb6100195f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:08.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.993+0000 7fb5ff7fe640 1 -- 172.21.15.116:0/2134634498 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6100bd020 con 0x7fb618155c00 2024-04-25T00:20:08.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.993+0000 7fb61d606640 1 -- 172.21.15.116:0/2134634498 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb618149310 con 0x7fb618155c00 2024-04-25T00:20:08.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.993+0000 7fb5ff7fe640 1 -- 172.21.15.116:0/2134634498 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6100b8470 con 0x7fb618155c00 2024-04-25T00:20:08.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.993+0000 7fb5ff7fe640 1 -- 172.21.15.116:0/2134634498 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6100b2040 con 0x7fb618155c00 2024-04-25T00:20:08.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.993+0000 7fb61d606640 1 -- 172.21.15.116:0/2134634498 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6180aaa90 con 0x7fb618155c00 2024-04-25T00:20:08.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.993+0000 7fb5ff7fe640 1 -- 172.21.15.116:0/2134634498 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb6100c0020 con 0x7fb618155c00 2024-04-25T00:20:08.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.993+0000 7fb61d606640 1 -- 172.21.15.116:0/2134634498 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb618068a60 con 0x7fb618155c00 2024-04-25T00:20:08.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.993+0000 7fb5ff7fe640 1 --2- 172.21.15.116:0/2134634498 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb6040678c0 0x7fb604069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:08.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.993+0000 7fb5ff7fe640 1 -- 172.21.15.116:0/2134634498 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(57..57 src has 1..57) ==== 9160+0+0 (secure 0 0 0) 0x7fb610124170 con 0x7fb618155c00 2024-04-25T00:20:08.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.993+0000 7fb616ffd640 1 --2- 172.21.15.116:0/2134634498 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb6040678c0 0x7fb604069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:08.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.993+0000 7fb616ffd640 1 --2- 172.21.15.116:0/2134634498 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb6040678c0 0x7fb604069d80 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fb608000f70 tx=0x7fb608007c10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:08.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:08.997+0000 7fb5ff7fe640 1 -- 172.21.15.116:0/2134634498 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fb610016560 con 0x7fb618155c00 2024-04-25T00:20:09.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:09.229+0000 7fb61d606640 1 -- 172.21.15.116:0/2134634498 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb6180ad9f0 con 0x7fb618155c00 2024-04-25T00:20:09.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:09.233+0000 7fb5ff7fe640 1 -- 172.21.15.116:0/2134634498 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 157 v157) ==== 95+0+6499 (secure 0 0 0) 0x7fb6100f0c30 con 0x7fb618155c00 2024-04-25T00:20:09.233 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:20:09.233 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":157,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7722,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":157}],"filesystems":[{"mdsmap":{"epoch":156,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:19:33.982945+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6586":{"gid":6586,"name":"d","rank":0,"incarnation":153,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/991183935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":991183935},{"type":"v1","addr":"172.21.15.174:6837","nonce":991183935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6586,"qdb_cluster":[6586]},"id":1},{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:19:08.844382+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7542},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7542":{"gid":7542,"name":"c","rank":0,"incarnation":148,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6835/879531976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":879531976},{"type":"v1","addr":"172.21.15.116:6835","nonce":879531976}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7542,"qdb_cluster":[7542]},"id":2}]} 2024-04-25T00:20:09.233 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 157 2024-04-25T00:20:09.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:09.233+0000 7fb61d606640 1 -- 172.21.15.116:0/2134634498 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb6040678c0 msgr2=0x7fb604069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:09.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:09.233+0000 7fb61d606640 1 --2- 172.21.15.116:0/2134634498 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb6040678c0 0x7fb604069d80 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fb608000f70 tx=0x7fb608007c10 comp rx=0 tx=0).stop 2024-04-25T00:20:09.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:09.233+0000 7fb61d606640 1 -- 172.21.15.116:0/2134634498 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb618155c00 msgr2=0x7fb6181eeb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:09.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:09.233+0000 7fb61d606640 1 --2- 172.21.15.116:0/2134634498 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb618155c00 0x7fb6181eeb70 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7fb6100b0040 tx=0x7fb6100195f0 comp rx=0 tx=0).stop 2024-04-25T00:20:09.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:09.233+0000 7fb61d606640 1 -- 172.21.15.116:0/2134634498 shutdown_connections 2024-04-25T00:20:09.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:09.233+0000 7fb61d606640 1 --2- 172.21.15.116:0/2134634498 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb6040678c0 0x7fb604069d80 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:09.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:09.233+0000 7fb61d606640 1 --2- 172.21.15.116:0/2134634498 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb618155c00 0x7fb6181eeb70 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:09.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:09.233+0000 7fb61d606640 1 --2- 172.21.15.116:0/2134634498 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb61814f7a0 0x7fb6181ee630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:09.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:09.233+0000 7fb61d606640 1 --2- 172.21.15.116:0/2134634498 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb618071520 0x7fb618071f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:09.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:09.233+0000 7fb61d606640 1 -- 172.21.15.116:0/2134634498 >> 172.21.15.116:0/2134634498 conn(0x7fb6180c4200 msgr2=0x7fb6181537d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:20:09.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:09.233+0000 7fb61d606640 1 -- 172.21.15.116:0/2134634498 shutdown_connections 2024-04-25T00:20:09.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:09.233+0000 7fb61d606640 1 -- 172.21.15.116:0/2134634498 wait complete. 2024-04-25T00:20:09.257 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:20:09.487 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:20:09.489 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:20:11.259 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:20:11.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.377+0000 7ff2730c1640 1 Processor -- start 2024-04-25T00:20:11.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.377+0000 7ff2730c1640 1 -- start start 2024-04-25T00:20:11.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.377+0000 7ff2730c1640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff26c152f60 0x7ff26c0c6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:11.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.377+0000 7ff2730c1640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff26c0c3c30 0x7ff26c06fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:11.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.377+0000 7ff2730c1640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff26c06e8f0 con 0x7ff26c158ea0 2024-04-25T00:20:11.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.377+0000 7ff2730c1640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7ff26c06ebd0 con 0x7ff26c0c3c30 2024-04-25T00:20:11.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.377+0000 7ff2730c1640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff26c06f330 con 0x7ff26c152f60 2024-04-25T00:20:11.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.377+0000 7ff26bfff640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff26c0c3c30 0x7ff26c06fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:11.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.377+0000 7ff270e5d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff26c152f60 0x7ff26c0c6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:11.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.377+0000 7ff27165e640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7ff26c158ea0 0x7ff26c06c220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:52018/0 (socket says 172.21.15.116:52018) 2024-04-25T00:20:11.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.377+0000 7ff270e5d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff26c152f60 0x7ff26c0c6080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:49744/0 (socket says 172.21.15.116:49744) 2024-04-25T00:20:11.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.377+0000 7ff27165e640 1 -- 172.21.15.116:0/3408757866 learned_addr learned my addr 172.21.15.116:0/3408757866 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:20:11.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.377+0000 7ff26bfff640 1 -- 172.21.15.116:0/3408757866 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff26c152f60 msgr2=0x7ff26c0c6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:20:11.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.377+0000 7ff26bfff640 1 --2- 172.21.15.116:0/3408757866 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff26c152f60 0x7ff26c0c6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:11.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.377+0000 7ff26bfff640 1 -- 172.21.15.116:0/3408757866 >> v1:172.21.15.174:6789/0 conn(0x7ff26c158ea0 legacy=0x7ff26c06c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:11.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.377+0000 7ff26bfff640 1 -- 172.21.15.116:0/3408757866 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff26c06f610 con 0x7ff26c0c3c30 2024-04-25T00:20:11.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.377+0000 7ff270e5d640 1 --2- 172.21.15.116:0/3408757866 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff26c152f60 0x7ff26c0c6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:20:11.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.381+0000 7ff26bfff640 1 --2- 172.21.15.116:0/3408757866 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff26c0c3c30 0x7ff26c06fb30 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7ff2580097c0 tx=0x7ff25801c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc22652e7bbad91c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:11.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.381+0000 7ff26b7fe640 1 -- 172.21.15.116:0/3408757866 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2580b0070 con 0x7ff26c0c3c30 2024-04-25T00:20:11.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.381+0000 7ff26b7fe640 1 -- 172.21.15.116:0/3408757866 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2580ab600 con 0x7ff26c0c3c30 2024-04-25T00:20:11.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.381+0000 7ff26b7fe640 1 -- 172.21.15.116:0/3408757866 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2580ab9d0 con 0x7ff26c0c3c30 2024-04-25T00:20:11.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.381+0000 7ff2730c1640 1 -- 172.21.15.116:0/3408757866 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff26c0c3c30 msgr2=0x7ff26c06fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:11.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.381+0000 7ff2730c1640 1 --2- 172.21.15.116:0/3408757866 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff26c0c3c30 0x7ff26c06fb30 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7ff2580097c0 tx=0x7ff25801c410 comp rx=0 tx=0).stop 2024-04-25T00:20:11.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.381+0000 7ff2730c1640 1 -- 172.21.15.116:0/3408757866 shutdown_connections 2024-04-25T00:20:11.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.381+0000 7ff2730c1640 1 --2- 172.21.15.116:0/3408757866 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff26c0c3c30 0x7ff26c06fb30 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:11.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.381+0000 7ff2730c1640 1 --2- 172.21.15.116:0/3408757866 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff26c152f60 0x7ff26c0c6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:11.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.381+0000 7ff2730c1640 1 -- 172.21.15.116:0/3408757866 >> 172.21.15.116:0/3408757866 conn(0x7ff26c0c4270 msgr2=0x7ff26c0c4680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:20:11.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.381+0000 7ff2730c1640 1 -- 172.21.15.116:0/3408757866 shutdown_connections 2024-04-25T00:20:11.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.381+0000 7ff2730c1640 1 -- 172.21.15.116:0/3408757866 wait complete. 2024-04-25T00:20:11.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.381+0000 7ff2730c1640 1 Processor -- start 2024-04-25T00:20:11.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.381+0000 7ff2730c1640 1 -- start start 2024-04-25T00:20:11.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.385+0000 7ff2730c1640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff26c0c3c30 0x7ff26c158820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:11.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.385+0000 7ff2730c1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff26c152f60 0x7ff26c157440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:11.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.385+0000 7ff2730c1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff26c158ea0 0x7ff26c072b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:11.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.385+0000 7ff2730c1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff26c162b70 con 0x7ff26c158ea0 2024-04-25T00:20:11.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.385+0000 7ff2730c1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff26c1629f0 con 0x7ff26c152f60 2024-04-25T00:20:11.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.385+0000 7ff2730c1640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff26c162cf0 con 0x7ff26c0c3c30 2024-04-25T00:20:11.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.385+0000 7ff27165e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff26c158ea0 0x7ff26c072b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:11.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.385+0000 7ff27165e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff26c158ea0 0x7ff26c072b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48992/0 (socket says 172.21.15.116:48992) 2024-04-25T00:20:11.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.385+0000 7ff27165e640 1 -- 172.21.15.116:0/670284110 learned_addr learned my addr 172.21.15.116:0/670284110 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:20:11.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.385+0000 7ff26bfff640 1 --2- 172.21.15.116:0/670284110 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff26c152f60 0x7ff26c157440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:11.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.385+0000 7ff270e5d640 1 --2- 172.21.15.116:0/670284110 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff26c0c3c30 0x7ff26c158820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:11.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.385+0000 7ff27165e640 1 -- 172.21.15.116:0/670284110 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff26c0c3c30 msgr2=0x7ff26c158820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:11.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.385+0000 7ff27165e640 1 --2- 172.21.15.116:0/670284110 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff26c0c3c30 0x7ff26c158820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:11.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.385+0000 7ff27165e640 1 -- 172.21.15.116:0/670284110 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff26c152f60 msgr2=0x7ff26c157440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:11.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.385+0000 7ff27165e640 1 --2- 172.21.15.116:0/670284110 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff26c152f60 0x7ff26c157440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:11.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.385+0000 7ff27165e640 1 -- 172.21.15.116:0/670284110 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff26c0a88d0 con 0x7ff26c158ea0 2024-04-25T00:20:11.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.385+0000 7ff27165e640 1 --2- 172.21.15.116:0/670284110 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff26c158ea0 0x7ff26c072b40 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7ff26400cab0 tx=0x7ff26400cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:11.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.385+0000 7ff2697fa640 1 -- 172.21.15.116:0/670284110 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff264017070 con 0x7ff26c158ea0 2024-04-25T00:20:11.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.385+0000 7ff2697fa640 1 -- 172.21.15.116:0/670284110 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff26400ac30 con 0x7ff26c158ea0 2024-04-25T00:20:11.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.385+0000 7ff2730c1640 1 -- 172.21.15.116:0/670284110 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff26c158260 con 0x7ff26c158ea0 2024-04-25T00:20:11.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.385+0000 7ff2697fa640 1 -- 172.21.15.116:0/670284110 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff264012400 con 0x7ff26c158ea0 2024-04-25T00:20:11.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.385+0000 7ff2730c1640 1 -- 172.21.15.116:0/670284110 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff26c156c40 con 0x7ff26c158ea0 2024-04-25T00:20:11.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.385+0000 7ff2730c1640 1 -- 172.21.15.116:0/670284110 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff26c068a60 con 0x7ff26c158ea0 2024-04-25T00:20:11.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.385+0000 7ff2697fa640 1 -- 172.21.15.116:0/670284110 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff26401a070 con 0x7ff26c158ea0 2024-04-25T00:20:11.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.385+0000 7ff2697fa640 1 --2- 172.21.15.116:0/670284110 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff2500678c0 0x7ff250069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:11.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.385+0000 7ff2697fa640 1 -- 172.21.15.116:0/670284110 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 9246+0+0 (secure 0 0 0) 0x7ff264086660 con 0x7ff26c158ea0 2024-04-25T00:20:11.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.389+0000 7ff270e5d640 1 --2- 172.21.15.116:0/670284110 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff2500678c0 0x7ff250069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:11.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.389+0000 7ff270e5d640 1 --2- 172.21.15.116:0/670284110 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff2500678c0 0x7ff250069d80 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7ff260009a50 tx=0x7ff260008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:11.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.393+0000 7ff2697fa640 1 -- 172.21.15.116:0/670284110 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ff26c0a88d0 con 0x7ff26c158ea0 2024-04-25T00:20:11.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.625+0000 7ff2730c1640 1 -- 172.21.15.116:0/670284110 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff26c0a9300 con 0x7ff26c158ea0 2024-04-25T00:20:11.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.625+0000 7ff2697fa640 1 -- 172.21.15.116:0/670284110 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 158 v158) ==== 95+0+5666 (secure 0 0 0) 0x7ff264055d30 con 0x7ff26c158ea0 2024-04-25T00:20:11.630 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:20:11.630 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":158,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":156,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:19:33.982945+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6586":{"gid":6586,"name":"d","rank":0,"incarnation":153,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/991183935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":991183935},{"type":"v1","addr":"172.21.15.174:6837","nonce":991183935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6586,"qdb_cluster":[6586]},"id":1},{"mdsmap":{"epoch":158,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:20:10.643312+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7722":{"gid":7722,"name":"a","rank":0,"incarnation":158,"state":"up:replay","state_seq":1,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:20:11.631 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 158 2024-04-25T00:20:11.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.629+0000 7ff2730c1640 1 -- 172.21.15.116:0/670284110 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff2500678c0 msgr2=0x7ff250069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:11.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.629+0000 7ff2730c1640 1 --2- 172.21.15.116:0/670284110 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff2500678c0 0x7ff250069d80 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7ff260009a50 tx=0x7ff260008040 comp rx=0 tx=0).stop 2024-04-25T00:20:11.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.629+0000 7ff2730c1640 1 -- 172.21.15.116:0/670284110 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff26c158ea0 msgr2=0x7ff26c072b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:11.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.629+0000 7ff2730c1640 1 --2- 172.21.15.116:0/670284110 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff26c158ea0 0x7ff26c072b40 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7ff26400cab0 tx=0x7ff26400cf70 comp rx=0 tx=0).stop 2024-04-25T00:20:11.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.629+0000 7ff2730c1640 1 -- 172.21.15.116:0/670284110 shutdown_connections 2024-04-25T00:20:11.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.629+0000 7ff2730c1640 1 --2- 172.21.15.116:0/670284110 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff2500678c0 0x7ff250069d80 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:11.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.629+0000 7ff2730c1640 1 --2- 172.21.15.116:0/670284110 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff26c158ea0 0x7ff26c072b40 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:11.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.629+0000 7ff2730c1640 1 --2- 172.21.15.116:0/670284110 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff26c152f60 0x7ff26c157440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:11.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.629+0000 7ff2730c1640 1 --2- 172.21.15.116:0/670284110 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff26c0c3c30 0x7ff26c158820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:11.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.629+0000 7ff2730c1640 1 -- 172.21.15.116:0/670284110 >> 172.21.15.116:0/670284110 conn(0x7ff26c0c4270 msgr2=0x7ff26c15f920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:20:11.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.629+0000 7ff2730c1640 1 -- 172.21.15.116:0/670284110 shutdown_connections 2024-04-25T00:20:11.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.629+0000 7ff2730c1640 1 -- 172.21.15.116:0/670284110 wait complete. 2024-04-25T00:20:11.647 INFO:tasks.mds_thrash.fs.[b]:mds.c down, removed from mdsmap 2024-04-25T00:20:11.647 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-04-25T00:20:11.647 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:20:11.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.761+0000 7f8b9da0f640 1 Processor -- start 2024-04-25T00:20:11.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.761+0000 7f8b9da0f640 1 -- start start 2024-04-25T00:20:11.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.761+0000 7f8b9da0f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8b98152d80 0x7f8b9806bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:11.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.761+0000 7f8b9da0f640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f8b9806c290 0x7f8b98157390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:11.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.761+0000 7f8b9da0f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b980c9150 con 0x7f8b9806d5f0 2024-04-25T00:20:11.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.761+0000 7f8b9da0f640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f8b980c9d60 con 0x7f8b9806c290 2024-04-25T00:20:11.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.761+0000 7f8b9da0f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f8b980c9430 con 0x7f8b98152d80 2024-04-25T00:20:11.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.761+0000 7f8b967fc640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f8b9806d5f0 0x7f8b98156c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:52276/0 (socket says 172.21.15.116:52276) 2024-04-25T00:20:11.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.761+0000 7f8b967fc640 1 -- 172.21.15.116:0/1374142022 learned_addr learned my addr 172.21.15.116:0/1374142022 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:20:11.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.761+0000 7f8b96ffd640 1 --2- 172.21.15.116:0/1374142022 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8b98152d80 0x7f8b9806bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:11.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.761+0000 7f8b977fe640 1 --2- 172.21.15.116:0/1374142022 >> v2:172.21.15.174:3300/0 conn(0x7f8b9806c290 0x7f8b98157390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:11.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.761+0000 7f8b95ffb640 1 -- 172.21.15.116:0/1374142022 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2591191495 0 0) 0x7f8b980c9150 con 0x7f8b9806d5f0 2024-04-25T00:20:11.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.761+0000 7f8b95ffb640 1 -- 172.21.15.116:0/1374142022 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b80003530 con 0x7f8b9806d5f0 2024-04-25T00:20:11.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.761+0000 7f8b95ffb640 1 -- 172.21.15.116:0/1374142022 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2460493244 0 0) 0x7f8b80003530 con 0x7f8b9806d5f0 2024-04-25T00:20:11.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.761+0000 7f8b95ffb640 1 -- 172.21.15.116:0/1374142022 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8b98152d80 msgr2=0x7f8b9806bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:20:11.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.761+0000 7f8b95ffb640 1 --2- 172.21.15.116:0/1374142022 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8b98152d80 0x7f8b9806bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:11.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.761+0000 7f8b95ffb640 1 -- 172.21.15.116:0/1374142022 >> v2:172.21.15.174:3300/0 conn(0x7f8b9806c290 msgr2=0x7f8b98157390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:20:11.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.761+0000 7f8b95ffb640 1 --2- 172.21.15.116:0/1374142022 >> v2:172.21.15.174:3300/0 conn(0x7f8b9806c290 0x7f8b98157390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:11.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.761+0000 7f8b95ffb640 1 -- 172.21.15.116:0/1374142022 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b980c9710 con 0x7f8b9806d5f0 2024-04-25T00:20:11.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.761+0000 7f8b95ffb640 1 -- 172.21.15.116:0/1374142022 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f8b8c001680 con 0x7f8b9806d5f0 2024-04-25T00:20:11.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.761+0000 7f8b95ffb640 1 -- 172.21.15.116:0/1374142022 >> v1:172.21.15.116:6789/0 conn(0x7f8b9806d5f0 legacy=0x7f8b98156c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:11.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.761+0000 7f8b95ffb640 1 --2- 172.21.15.116:0/1374142022 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8b98152d80 0x7f8b800039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:11.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.761+0000 7f8b95ffb640 1 --2- 172.21.15.116:0/1374142022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b80003f40 0x7f8b80004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:11.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.761+0000 7f8b95ffb640 1 --2- 172.21.15.116:0/1374142022 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8b80004d50 0x7f8b800051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:11.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.761+0000 7f8b95ffb640 1 -- 172.21.15.116:0/1374142022 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8b980c9d60 con 0x7f8b80003f40 2024-04-25T00:20:11.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.761+0000 7f8b95ffb640 1 -- 172.21.15.116:0/1374142022 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8b980c9430 con 0x7f8b80004d50 2024-04-25T00:20:11.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.761+0000 7f8b95ffb640 1 -- 172.21.15.116:0/1374142022 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f8b8000ba20 con 0x7f8b98152d80 2024-04-25T00:20:11.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.761+0000 7f8b967fc640 1 --2- 172.21.15.116:0/1374142022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b80003f40 0x7f8b80004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:11.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.761+0000 7f8b977fe640 1 --2- 172.21.15.116:0/1374142022 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8b80004d50 0x7f8b800051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:11.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.761+0000 7f8b96ffd640 1 --2- 172.21.15.116:0/1374142022 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8b98152d80 0x7f8b800039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:11.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.761+0000 7f8b967fc640 1 -- 172.21.15.116:0/1374142022 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8b98152d80 msgr2=0x7f8b800039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:20:11.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.761+0000 7f8b967fc640 1 --2- 172.21.15.116:0/1374142022 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8b98152d80 0x7f8b800039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:11.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.761+0000 7f8b967fc640 1 -- 172.21.15.116:0/1374142022 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8b80004d50 msgr2=0x7f8b800051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:20:11.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.761+0000 7f8b967fc640 1 --2- 172.21.15.116:0/1374142022 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8b80004d50 0x7f8b800051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:11.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.761+0000 7f8b967fc640 1 -- 172.21.15.116:0/1374142022 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b80003530 con 0x7f8b80003f40 2024-04-25T00:20:11.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.765+0000 7f8b967fc640 1 --2- 172.21.15.116:0/1374142022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b80003f40 0x7f8b80004810 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c00c240 tx=0x7f8b8c01cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0795a64fc5946f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:11.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.765+0000 7f8b95ffb640 1 -- 172.21.15.116:0/1374142022 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b8c007d40 con 0x7f8b80003f40 2024-04-25T00:20:11.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.765+0000 7f8b95ffb640 1 -- 172.21.15.116:0/1374142022 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b8c0a9020 con 0x7f8b80003f40 2024-04-25T00:20:11.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.765+0000 7f8b95ffb640 1 -- 172.21.15.116:0/1374142022 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b8c0a5b50 con 0x7f8b80003f40 2024-04-25T00:20:11.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.765+0000 7f8b9da0f640 1 -- 172.21.15.116:0/1374142022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b80003f40 msgr2=0x7f8b80004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:11.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.765+0000 7f8b9da0f640 1 --2- 172.21.15.116:0/1374142022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b80003f40 0x7f8b80004810 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c00c240 tx=0x7f8b8c01cc30 comp rx=0 tx=0).stop 2024-04-25T00:20:11.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.765+0000 7f8b977fe640 1 -- 172.21.15.116:0/1374142022 reap_dead start 2024-04-25T00:20:11.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.765+0000 7f8b9da0f640 1 -- 172.21.15.116:0/1374142022 shutdown_connections 2024-04-25T00:20:11.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.765+0000 7f8b9da0f640 1 -- 172.21.15.116:0/1374142022 >> 172.21.15.116:0/1374142022 conn(0x7f8b980acdd0 msgr2=0x7f8b980ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:20:11.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.765+0000 7f8b9da0f640 1 -- 172.21.15.116:0/1374142022 shutdown_connections 2024-04-25T00:20:11.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.765+0000 7f8b9da0f640 1 -- 172.21.15.116:0/1374142022 wait complete. 2024-04-25T00:20:11.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.765+0000 7f8b9da0f640 1 Processor -- start 2024-04-25T00:20:11.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.765+0000 7f8b9da0f640 1 -- start start 2024-04-25T00:20:11.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.769+0000 7f8b9da0f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8b80003f40 0x7f8b9815f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:11.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.769+0000 7f8b9da0f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b980ae1a0 0x7f8b9814c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:11.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.769+0000 7f8b9da0f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8b9814c780 0x7f8b98150800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:11.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.769+0000 7f8b9da0f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8b980caf00 con 0x7f8b980ae1a0 2024-04-25T00:20:11.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.769+0000 7f8b9da0f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8b980cad80 con 0x7f8b80003f40 2024-04-25T00:20:11.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.769+0000 7f8b9da0f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f8b981608c0 con 0x7f8b9814c780 2024-04-25T00:20:11.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.769+0000 7f8b967fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b980ae1a0 0x7f8b9814c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:11.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.769+0000 7f8b967fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b980ae1a0 0x7f8b9814c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49022/0 (socket says 172.21.15.116:49022) 2024-04-25T00:20:11.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.769+0000 7f8b967fc640 1 -- 172.21.15.116:0/2165154164 learned_addr learned my addr 172.21.15.116:0/2165154164 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:20:11.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.769+0000 7f8b96ffd640 1 --2- 172.21.15.116:0/2165154164 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8b80003f40 0x7f8b9815f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:11.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.769+0000 7f8b977fe640 1 --2- 172.21.15.116:0/2165154164 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8b9814c780 0x7f8b98150800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:11.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.769+0000 7f8b967fc640 1 -- 172.21.15.116:0/2165154164 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8b9814c780 msgr2=0x7f8b98150800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:11.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.769+0000 7f8b967fc640 1 --2- 172.21.15.116:0/2165154164 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8b9814c780 0x7f8b98150800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:11.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.769+0000 7f8b967fc640 1 -- 172.21.15.116:0/2165154164 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8b80003f40 msgr2=0x7f8b9815f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:11.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.769+0000 7f8b967fc640 1 --2- 172.21.15.116:0/2165154164 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8b80003f40 0x7f8b9815f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:11.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.769+0000 7f8b967fc640 1 -- 172.21.15.116:0/2165154164 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b981f2c20 con 0x7f8b980ae1a0 2024-04-25T00:20:11.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.769+0000 7f8b967fc640 1 --2- 172.21.15.116:0/2165154164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b980ae1a0 0x7f8b9814c240 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c01ca60 tx=0x7f8b80005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:11.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.769+0000 7f8b7ffff640 1 -- 172.21.15.116:0/2165154164 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b8c0a5b30 con 0x7f8b980ae1a0 2024-04-25T00:20:11.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.769+0000 7f8b7ffff640 1 -- 172.21.15.116:0/2165154164 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b8c0b3030 con 0x7f8b980ae1a0 2024-04-25T00:20:11.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.769+0000 7f8b7ffff640 1 -- 172.21.15.116:0/2165154164 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b8c0047f0 con 0x7f8b980ae1a0 2024-04-25T00:20:11.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.769+0000 7f8b9da0f640 1 -- 172.21.15.116:0/2165154164 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b9815f2f0 con 0x7f8b980ae1a0 2024-04-25T00:20:11.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.769+0000 7f8b9da0f640 1 -- 172.21.15.116:0/2165154164 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b981502e0 con 0x7f8b980ae1a0 2024-04-25T00:20:11.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.769+0000 7f8b9da0f640 1 -- 172.21.15.116:0/2165154164 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b98068350 con 0x7f8b980ae1a0 2024-04-25T00:20:11.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.769+0000 7f8b7ffff640 1 -- 172.21.15.116:0/2165154164 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8b8c004990 con 0x7f8b980ae1a0 2024-04-25T00:20:11.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.773+0000 7f8b7ffff640 1 --2- 172.21.15.116:0/2165154164 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8b840678f0 0x7f8b84069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:11.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.773+0000 7f8b7ffff640 1 -- 172.21.15.116:0/2165154164 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 9246+0+0 (secure 0 0 0) 0x7f8b8c11fff0 con 0x7f8b980ae1a0 2024-04-25T00:20:11.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.773+0000 7f8b96ffd640 1 --2- 172.21.15.116:0/2165154164 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8b840678f0 0x7f8b84069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:11.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.773+0000 7f8b96ffd640 1 --2- 172.21.15.116:0/2165154164 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8b840678f0 0x7f8b84069db0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f8b88009950 tx=0x7f8b88008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:11.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.777+0000 7f8b7ffff640 1 -- 172.21.15.116:0/2165154164 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f8b8c016970 con 0x7f8b980ae1a0 2024-04-25T00:20:11.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.981+0000 7f8b9da0f640 1 -- 172.21.15.116:0/2165154164 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8b98154db0 con 0x7f8b980ae1a0 2024-04-25T00:20:11.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.981+0000 7f8b7ffff640 1 -- 172.21.15.116:0/2165154164 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) ==== 95+0+5669 (secure 0 0 0) 0x7f8b8c0efd60 con 0x7f8b980ae1a0 2024-04-25T00:20:11.984 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:20:11.985 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":159,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":156,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:19:33.982945+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6586":{"gid":6586,"name":"d","rank":0,"incarnation":153,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/991183935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":991183935},{"type":"v1","addr":"172.21.15.174:6837","nonce":991183935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6586,"qdb_cluster":[6586]},"id":1},{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:20:11.050824+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7722":{"gid":7722,"name":"a","rank":0,"incarnation":158,"state":"up:reconnect","state_seq":9,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:20:11.985 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 159 2024-04-25T00:20:11.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.985+0000 7f8b9da0f640 1 -- 172.21.15.116:0/2165154164 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8b840678f0 msgr2=0x7f8b84069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:11.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.985+0000 7f8b9da0f640 1 --2- 172.21.15.116:0/2165154164 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8b840678f0 0x7f8b84069db0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f8b88009950 tx=0x7f8b88008040 comp rx=0 tx=0).stop 2024-04-25T00:20:11.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.985+0000 7f8b9da0f640 1 -- 172.21.15.116:0/2165154164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b980ae1a0 msgr2=0x7f8b9814c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:11.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.985+0000 7f8b9da0f640 1 --2- 172.21.15.116:0/2165154164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b980ae1a0 0x7f8b9814c240 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c01ca60 tx=0x7f8b80005840 comp rx=0 tx=0).stop 2024-04-25T00:20:11.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.985+0000 7f8b9da0f640 1 -- 172.21.15.116:0/2165154164 shutdown_connections 2024-04-25T00:20:11.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.985+0000 7f8b9da0f640 1 --2- 172.21.15.116:0/2165154164 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8b840678f0 0x7f8b84069db0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:11.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.985+0000 7f8b9da0f640 1 --2- 172.21.15.116:0/2165154164 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8b9814c780 0x7f8b98150800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:11.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.985+0000 7f8b9da0f640 1 --2- 172.21.15.116:0/2165154164 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b980ae1a0 0x7f8b9814c240 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:11.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.985+0000 7f8b9da0f640 1 --2- 172.21.15.116:0/2165154164 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8b80003f40 0x7f8b9815f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:11.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.985+0000 7f8b9da0f640 1 -- 172.21.15.116:0/2165154164 >> 172.21.15.116:0/2165154164 conn(0x7f8b980acdd0 msgr2=0x7f8b9814e0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:20:11.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.985+0000 7f8b9da0f640 1 -- 172.21.15.116:0/2165154164 shutdown_connections 2024-04-25T00:20:11.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:11.985+0000 7f8b9da0f640 1 -- 172.21.15.116:0/2165154164 wait complete. 2024-04-25T00:20:12.003 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 159, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:04.678737+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 156, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6586": { "addr": "172.21.15.174:6837/991183935", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6836", "nonce": 991183935, "type": "v2" }, { "addr": "172.21.15.174:6837", "nonce": 991183935, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6586, "incarnation": 153, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 7 } }, "last_failure": 0, "last_failure_osd_epoch": 57, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T00:19:33.982945+0000", "qdb_cluster": [ 6586 ], "qdb_leader": 6586, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6586 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:18.825887+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 159, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_7722": { "addr": "172.21.15.116:6833/663910035", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6832", "nonce": 663910035, "type": "v2" }, { "addr": "172.21.15.116:6833", "nonce": 663910035, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7722, "incarnation": 158, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:reconnect", "state_seq": 9 } }, "last_failure": 0, "last_failure_osd_epoch": 58, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-04-25T00:20:11.050824+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7722 } } } ], "standbys": [ { "addr": "172.21.15.174:6835/4112145782", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6834", "nonce": 4112145782, "type": "v2" }, { "addr": "172.21.15.174:6835", "nonce": 4112145782, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 112, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5582, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-25T00:20:14.004 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:20:14.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.129+0000 7f2bc548e640 1 Processor -- start 2024-04-25T00:20:14.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.129+0000 7f2bc548e640 1 -- start start 2024-04-25T00:20:14.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.129+0000 7f2bc548e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2bc00aacd0 0x7f2bc015dec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:14.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.129+0000 7f2bc548e640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2bc00a9e60 con 0x7f2bc0155b20 2024-04-25T00:20:14.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.129+0000 7f2bc548e640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2bc00c3ef0 con 0x7f2bc014c700 2024-04-25T00:20:14.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.129+0000 7f2bc548e640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f2bc01527b0 con 0x7f2bc00aacd0 2024-04-25T00:20:14.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.129+0000 7f2bbe7fc640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f2bc0155b20 0x7f2bc0152c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:52288/0 (socket says 172.21.15.116:52288) 2024-04-25T00:20:14.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.129+0000 7f2bbe7fc640 1 -- 172.21.15.116:0/780554319 learned_addr learned my addr 172.21.15.116:0/780554319 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:20:14.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.129+0000 7f2bbf7fe640 1 --2- 172.21.15.116:0/780554319 >> v2:172.21.15.116:3300/0 conn(0x7f2bc00aacd0 0x7f2bc015dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:14.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.129+0000 7f2bbf7fe640 1 -- 172.21.15.116:0/780554319 >> v1:172.21.15.174:6789/0 conn(0x7f2bc014c700 legacy=0x7f2bc00c5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:20:14.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.133+0000 7f2bbf7fe640 1 -- 172.21.15.116:0/780554319 >> v1:172.21.15.116:6789/0 conn(0x7f2bc0155b20 legacy=0x7f2bc0152c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:14.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.133+0000 7f2bbf7fe640 1 -- 172.21.15.116:0/780554319 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bc00c41d0 con 0x7f2bc00aacd0 2024-04-25T00:20:14.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.133+0000 7f2bbdffb640 1 -- 172.21.15.116:0/780554319 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1269629836 0 0) 0x7f2bc00a9e60 con 0x7f2bc0155b20 2024-04-25T00:20:14.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.133+0000 7f2bbf7fe640 1 --2- 172.21.15.116:0/780554319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bc00aacd0 0x7f2bc015dec0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f2bb8009d90 tx=0x7f2bb801c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=912c83eabe61c363 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:14.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.133+0000 7f2bbdffb640 1 -- 172.21.15.116:0/780554319 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bb80b2070 con 0x7f2bc00aacd0 2024-04-25T00:20:14.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.133+0000 7f2bbdffb640 1 -- 172.21.15.116:0/780554319 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2bb80a5710 con 0x7f2bc00aacd0 2024-04-25T00:20:14.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.133+0000 7f2bbdffb640 1 -- 172.21.15.116:0/780554319 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bb80a5ae0 con 0x7f2bc00aacd0 2024-04-25T00:20:14.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.133+0000 7f2bc548e640 1 -- 172.21.15.116:0/780554319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bc00aacd0 msgr2=0x7f2bc015dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:14.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.133+0000 7f2bc548e640 1 --2- 172.21.15.116:0/780554319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bc00aacd0 0x7f2bc015dec0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f2bb8009d90 tx=0x7f2bb801c850 comp rx=0 tx=0).stop 2024-04-25T00:20:14.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.133+0000 7f2bc548e640 1 -- 172.21.15.116:0/780554319 shutdown_connections 2024-04-25T00:20:14.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.133+0000 7f2bc548e640 1 --2- 172.21.15.116:0/780554319 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bc00aacd0 0x7f2bc015dec0 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:14.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.133+0000 7f2bc548e640 1 -- 172.21.15.116:0/780554319 >> 172.21.15.116:0/780554319 conn(0x7f2bc006c720 msgr2=0x7f2bc006cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:20:14.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.133+0000 7f2bc548e640 1 -- 172.21.15.116:0/780554319 shutdown_connections 2024-04-25T00:20:14.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.133+0000 7f2bc548e640 1 -- 172.21.15.116:0/780554319 wait complete. 2024-04-25T00:20:14.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.137+0000 7f2bc548e640 1 Processor -- start 2024-04-25T00:20:14.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.137+0000 7f2bc548e640 1 -- start start 2024-04-25T00:20:14.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.137+0000 7f2bc548e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2bc00aacd0 0x7f2bc014ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:14.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.137+0000 7f2bc548e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2bc014c700 0x7f2bc0144510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:14.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.137+0000 7f2bc548e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bc0155b20 0x7f2bc0144a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:14.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.137+0000 7f2bc548e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc006b390 con 0x7f2bc0155b20 2024-04-25T00:20:14.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.137+0000 7f2bc548e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2bc006b210 con 0x7f2bc00aacd0 2024-04-25T00:20:14.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.137+0000 7f2bc548e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f2bc006b510 con 0x7f2bc014c700 2024-04-25T00:20:14.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.137+0000 7f2bbf7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bc0155b20 0x7f2bc0144a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:14.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.137+0000 7f2bbf7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bc0155b20 0x7f2bc0144a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49036/0 (socket says 172.21.15.116:49036) 2024-04-25T00:20:14.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.137+0000 7f2bbf7fe640 1 -- 172.21.15.116:0/306374036 learned_addr learned my addr 172.21.15.116:0/306374036 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:20:14.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.137+0000 7f2bbe7fc640 1 --2- 172.21.15.116:0/306374036 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2bc014c700 0x7f2bc0144510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:14.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.137+0000 7f2bbeffd640 1 --2- 172.21.15.116:0/306374036 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2bc00aacd0 0x7f2bc014ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:14.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.137+0000 7f2bbf7fe640 1 -- 172.21.15.116:0/306374036 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2bc014c700 msgr2=0x7f2bc0144510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:14.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.137+0000 7f2bbf7fe640 1 --2- 172.21.15.116:0/306374036 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2bc014c700 0x7f2bc0144510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:14.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.137+0000 7f2bbf7fe640 1 -- 172.21.15.116:0/306374036 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2bc00aacd0 msgr2=0x7f2bc014ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:14.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.137+0000 7f2bbf7fe640 1 --2- 172.21.15.116:0/306374036 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2bc00aacd0 0x7f2bc014ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:14.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.137+0000 7f2bbf7fe640 1 -- 172.21.15.116:0/306374036 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bc014e590 con 0x7f2bc0155b20 2024-04-25T00:20:14.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.137+0000 7f2bbeffd640 1 --2- 172.21.15.116:0/306374036 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2bc00aacd0 0x7f2bc014ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:20:14.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.137+0000 7f2bbf7fe640 1 --2- 172.21.15.116:0/306374036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bc0155b20 0x7f2bc0144a50 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f2bb80b0040 tx=0x7f2bb80195f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:14.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.137+0000 7f2ba77fe640 1 -- 172.21.15.116:0/306374036 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bb80bd020 con 0x7f2bc0155b20 2024-04-25T00:20:14.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.137+0000 7f2ba77fe640 1 -- 172.21.15.116:0/306374036 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2bb80b3040 con 0x7f2bc0155b20 2024-04-25T00:20:14.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.137+0000 7f2bc548e640 1 -- 172.21.15.116:0/306374036 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bc014ed20 con 0x7f2bc0155b20 2024-04-25T00:20:14.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.137+0000 7f2bc548e640 1 -- 172.21.15.116:0/306374036 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bc014f2e0 con 0x7f2bc0155b20 2024-04-25T00:20:14.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.137+0000 7f2ba77fe640 1 -- 172.21.15.116:0/306374036 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bb80b2050 con 0x7f2bc0155b20 2024-04-25T00:20:14.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.137+0000 7f2bc548e640 1 -- 172.21.15.116:0/306374036 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ba00052a0 con 0x7f2bc0155b20 2024-04-25T00:20:14.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.141+0000 7f2ba77fe640 1 -- 172.21.15.116:0/306374036 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2bb80ad650 con 0x7f2bc0155b20 2024-04-25T00:20:14.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.141+0000 7f2ba77fe640 1 --2- 172.21.15.116:0/306374036 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2bac0678c0 0x7f2bac069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:14.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.141+0000 7f2ba77fe640 1 -- 172.21.15.116:0/306374036 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 9246+0+0 (secure 0 0 0) 0x7f2bb8121d70 con 0x7f2bc0155b20 2024-04-25T00:20:14.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.141+0000 7f2bbeffd640 1 --2- 172.21.15.116:0/306374036 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2bac0678c0 0x7f2bac069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:14.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.141+0000 7f2bbeffd640 1 --2- 172.21.15.116:0/306374036 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2bac0678c0 0x7f2bac069d80 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f2bc0075a40 tx=0x7f2bb0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:14.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.145+0000 7f2ba77fe640 1 -- 172.21.15.116:0/306374036 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f2bb80c7060 con 0x7f2bc0155b20 2024-04-25T00:20:14.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.349+0000 7f2bc548e640 1 -- 172.21.15.116:0/306374036 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2ba0005590 con 0x7f2bc0155b20 2024-04-25T00:20:14.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.353+0000 7f2ba77fe640 1 -- 172.21.15.116:0/306374036 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 161 v161) ==== 95+0+5674 (secure 0 0 0) 0x7f2bb8016560 con 0x7f2bc0155b20 2024-04-25T00:20:14.353 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:20:14.353 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":161,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":156,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:19:33.982945+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6586":{"gid":6586,"name":"d","rank":0,"incarnation":153,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/991183935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":991183935},{"type":"v1","addr":"172.21.15.174:6837","nonce":991183935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6586,"qdb_cluster":[6586]},"id":1},{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:20:13.935378+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7722":{"gid":7722,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7722,"qdb_cluster":[7722]},"id":2}]} 2024-04-25T00:20:14.354 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 161 2024-04-25T00:20:14.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.353+0000 7f2bc548e640 1 -- 172.21.15.116:0/306374036 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2bac0678c0 msgr2=0x7f2bac069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:14.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.353+0000 7f2bc548e640 1 --2- 172.21.15.116:0/306374036 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2bac0678c0 0x7f2bac069d80 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f2bc0075a40 tx=0x7f2bb0009040 comp rx=0 tx=0).stop 2024-04-25T00:20:14.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.353+0000 7f2bc548e640 1 -- 172.21.15.116:0/306374036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bc0155b20 msgr2=0x7f2bc0144a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:14.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.353+0000 7f2bc548e640 1 --2- 172.21.15.116:0/306374036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bc0155b20 0x7f2bc0144a50 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f2bb80b0040 tx=0x7f2bb80195f0 comp rx=0 tx=0).stop 2024-04-25T00:20:14.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.353+0000 7f2bc548e640 1 -- 172.21.15.116:0/306374036 shutdown_connections 2024-04-25T00:20:14.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.353+0000 7f2bc548e640 1 --2- 172.21.15.116:0/306374036 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2bac0678c0 0x7f2bac069d80 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:14.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.353+0000 7f2bc548e640 1 --2- 172.21.15.116:0/306374036 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bc0155b20 0x7f2bc0144a50 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:14.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.353+0000 7f2bc548e640 1 --2- 172.21.15.116:0/306374036 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2bc014c700 0x7f2bc0144510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:14.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.353+0000 7f2bc548e640 1 --2- 172.21.15.116:0/306374036 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2bc00aacd0 0x7f2bc014ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:14.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.353+0000 7f2bc548e640 1 -- 172.21.15.116:0/306374036 >> 172.21.15.116:0/306374036 conn(0x7f2bc006c720 msgr2=0x7f2bc014bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:20:14.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.353+0000 7f2bc548e640 1 -- 172.21.15.116:0/306374036 shutdown_connections 2024-04-25T00:20:14.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:14.353+0000 7f2bc548e640 1 -- 172.21.15.116:0/306374036 wait complete. 2024-04-25T00:20:14.373 INFO:tasks.mds_thrash.fs.[b]:mds.a has gained rank=0, replacing gid=7542 2024-04-25T00:20:14.373 INFO:tasks.mds_thrash.fs.[b]:waiting for 4 secs before reviving mds.c 2024-04-25T00:20:16.840 INFO:tasks.ceph.mds.a.smithi116.stderr:2024-04-25T00:20:16.837+0000 7f9f11ce1640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T00:20:18.374 INFO:tasks.mds_thrash.fs.[b]:reviving mds.c 2024-04-25T00:20:18.375 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-25T00:20:18.375 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-25T00:20:18.377 INFO:tasks.ceph.mds.c:Started 2024-04-25T00:20:18.378 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:20:18.459 INFO:tasks.ceph.mds.c.smithi116.stdout:starting mds.c at 2024-04-25T00:20:18.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.533+0000 7f43662c8640 1 Processor -- start 2024-04-25T00:20:18.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.533+0000 7f43662c8640 1 -- start start 2024-04-25T00:20:18.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.533+0000 7f43662c8640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43600c5280 0x7f4360150840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:18.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.533+0000 7f43662c8640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43600c2c40 con 0x7f4360072c90 2024-04-25T00:20:18.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.533+0000 7f43662c8640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4360157660 con 0x7f4360072810 2024-04-25T00:20:18.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.533+0000 7f43662c8640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f4360157c20 con 0x7f43600c5280 2024-04-25T00:20:18.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.533+0000 7f435f7fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f4360072c90 0x7f4360150d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:41228/0 (socket says 172.21.15.116:41228) 2024-04-25T00:20:18.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.533+0000 7f435f7fe640 1 -- 172.21.15.116:0/190855621 learned_addr learned my addr 172.21.15.116:0/190855621 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:20:18.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.533+0000 7f435ffff640 1 --2- 172.21.15.116:0/190855621 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43600c5280 0x7f4360150840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:18.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.533+0000 7f435effd640 1 -- 172.21.15.116:0/190855621 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4002539393 0 0) 0x7f43600c2c40 con 0x7f4360072c90 2024-04-25T00:20:18.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.533+0000 7f435effd640 1 -- 172.21.15.116:0/190855621 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4348003530 con 0x7f4360072c90 2024-04-25T00:20:18.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.533+0000 7f435effd640 1 -- 172.21.15.116:0/190855621 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3017921381 0 0) 0x7f4348003530 con 0x7f4360072c90 2024-04-25T00:20:18.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.533+0000 7f435effd640 1 -- 172.21.15.116:0/190855621 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43600c5280 msgr2=0x7f4360150840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:20:18.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.533+0000 7f435effd640 1 --2- 172.21.15.116:0/190855621 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43600c5280 0x7f4360150840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:18.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.533+0000 7f435effd640 1 -- 172.21.15.116:0/190855621 >> v1:172.21.15.174:6789/0 conn(0x7f4360072810 legacy=0x7f4360153760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:18.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.533+0000 7f435effd640 1 -- 172.21.15.116:0/190855621 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4360157940 con 0x7f4360072c90 2024-04-25T00:20:18.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.533+0000 7f435effd640 1 -- 172.21.15.116:0/190855621 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f43540016a0 con 0x7f4360072c90 2024-04-25T00:20:18.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.533+0000 7f435effd640 1 -- 172.21.15.116:0/190855621 >> v1:172.21.15.116:6789/0 conn(0x7f4360072c90 legacy=0x7f4360150d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:18.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.533+0000 7f435effd640 1 --2- 172.21.15.116:0/190855621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43480039f0 0x7f4348003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:18.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.533+0000 7f435effd640 1 --2- 172.21.15.116:0/190855621 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43600c5280 0x7f4348004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:18.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.533+0000 7f435effd640 1 --2- 172.21.15.116:0/190855621 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4348004d50 0x7f43480051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:18.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.533+0000 7f435effd640 1 -- 172.21.15.116:0/190855621 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4360157c20 con 0x7f43480039f0 2024-04-25T00:20:18.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.533+0000 7f435effd640 1 -- 172.21.15.116:0/190855621 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f434800bb00 con 0x7f4348004d50 2024-04-25T00:20:18.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.533+0000 7f435effd640 1 -- 172.21.15.116:0/190855621 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f4348010110 con 0x7f43600c5280 2024-04-25T00:20:18.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.533+0000 7f435ffff640 1 --2- 172.21.15.116:0/190855621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43480039f0 0x7f4348003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:18.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.533+0000 7f435ffff640 1 -- 172.21.15.116:0/190855621 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43600c5280 msgr2=0x7f4348004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:20:18.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.533+0000 7f435ffff640 1 --2- 172.21.15.116:0/190855621 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43600c5280 0x7f4348004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:18.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.533+0000 7f435ffff640 1 -- 172.21.15.116:0/190855621 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4348004d50 msgr2=0x7f43480051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:20:18.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.533+0000 7f4364865640 1 --2- 172.21.15.116:0/190855621 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4348004d50 0x7f43480051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:18.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.533+0000 7f435ffff640 1 --2- 172.21.15.116:0/190855621 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4348004d50 0x7f43480051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:18.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.533+0000 7f435ffff640 1 -- 172.21.15.116:0/190855621 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4348003530 con 0x7f43480039f0 2024-04-25T00:20:18.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.537+0000 7f435ffff640 1 --2- 172.21.15.116:0/190855621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43480039f0 0x7f4348003e10 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f4350009ae0 tx=0x7f435001c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d8039b7449a05de server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:18.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.537+0000 7f435effd640 1 -- 172.21.15.116:0/190855621 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43500096b0 con 0x7f43480039f0 2024-04-25T00:20:18.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.537+0000 7f435effd640 1 -- 172.21.15.116:0/190855621 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43500a8430 con 0x7f43480039f0 2024-04-25T00:20:18.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.537+0000 7f435effd640 1 -- 172.21.15.116:0/190855621 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43500a8820 con 0x7f43480039f0 2024-04-25T00:20:18.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.537+0000 7f43662c8640 1 -- 172.21.15.116:0/190855621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43480039f0 msgr2=0x7f4348003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:18.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.537+0000 7f43662c8640 1 --2- 172.21.15.116:0/190855621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43480039f0 0x7f4348003e10 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f4350009ae0 tx=0x7f435001c9f0 comp rx=0 tx=0).stop 2024-04-25T00:20:18.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.537+0000 7f4364865640 1 -- 172.21.15.116:0/190855621 reap_dead start 2024-04-25T00:20:18.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.537+0000 7f43662c8640 1 -- 172.21.15.116:0/190855621 shutdown_connections 2024-04-25T00:20:18.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.537+0000 7f43662c8640 1 -- 172.21.15.116:0/190855621 >> 172.21.15.116:0/190855621 conn(0x7f436006c720 msgr2=0x7f436006cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:20:18.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.537+0000 7f43662c8640 1 -- 172.21.15.116:0/190855621 shutdown_connections 2024-04-25T00:20:18.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.537+0000 7f43662c8640 1 -- 172.21.15.116:0/190855621 wait complete. 2024-04-25T00:20:18.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.541+0000 7f43662c8640 1 Processor -- start 2024-04-25T00:20:18.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.541+0000 7f43662c8640 1 -- start start 2024-04-25T00:20:18.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.541+0000 7f43662c8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43480039f0 0x7f4360141680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:18.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.541+0000 7f43662c8640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43600c2320 0x7f43601514d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:18.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.541+0000 7f43662c8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4360151a10 0x7f43601edad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:18.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.541+0000 7f43662c8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4360154200 con 0x7f4360151a10 2024-04-25T00:20:18.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.541+0000 7f43662c8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4360154080 con 0x7f43480039f0 2024-04-25T00:20:18.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.541+0000 7f43662c8640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f4360154380 con 0x7f43600c2320 2024-04-25T00:20:18.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.541+0000 7f4364865640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4360151a10 0x7f43601edad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:18.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.541+0000 7f4364865640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4360151a10 0x7f43601edad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47308/0 (socket says 172.21.15.116:47308) 2024-04-25T00:20:18.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.541+0000 7f4364865640 1 -- 172.21.15.116:0/425180825 learned_addr learned my addr 172.21.15.116:0/425180825 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:20:18.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.541+0000 7f435ffff640 1 --2- 172.21.15.116:0/425180825 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43480039f0 0x7f4360141680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:18.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.541+0000 7f435f7fe640 1 --2- 172.21.15.116:0/425180825 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43600c2320 0x7f43601514d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:18.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.541+0000 7f4364865640 1 -- 172.21.15.116:0/425180825 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43600c2320 msgr2=0x7f43601514d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:18.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.541+0000 7f4364865640 1 --2- 172.21.15.116:0/425180825 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43600c2320 0x7f43601514d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:18.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.541+0000 7f4364865640 1 -- 172.21.15.116:0/425180825 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43480039f0 msgr2=0x7f4360141680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:18.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.541+0000 7f4364865640 1 --2- 172.21.15.116:0/425180825 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43480039f0 0x7f4360141680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:18.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.541+0000 7f4364865640 1 -- 172.21.15.116:0/425180825 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4360157240 con 0x7f4360151a10 2024-04-25T00:20:18.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.541+0000 7f435ffff640 1 --2- 172.21.15.116:0/425180825 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43480039f0 0x7f4360141680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:20:18.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.541+0000 7f4364865640 1 --2- 172.21.15.116:0/425180825 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4360151a10 0x7f43601edad0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f4360153660 tx=0x7f435800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:18.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.541+0000 7f435cff9640 1 -- 172.21.15.116:0/425180825 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4358018070 con 0x7f4360151a10 2024-04-25T00:20:18.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.541+0000 7f435cff9640 1 -- 172.21.15.116:0/425180825 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4358003b80 con 0x7f4360151a10 2024-04-25T00:20:18.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.541+0000 7f43662c8640 1 -- 172.21.15.116:0/425180825 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4360154e40 con 0x7f4360151a10 2024-04-25T00:20:18.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.541+0000 7f435cff9640 1 -- 172.21.15.116:0/425180825 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4358005070 con 0x7f4360151a10 2024-04-25T00:20:18.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.541+0000 7f43662c8640 1 -- 172.21.15.116:0/425180825 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4360154b60 con 0x7f4360151a10 2024-04-25T00:20:18.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.541+0000 7f43662c8640 1 -- 172.21.15.116:0/425180825 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4360068a60 con 0x7f4360151a10 2024-04-25T00:20:18.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.541+0000 7f435cff9640 1 -- 172.21.15.116:0/425180825 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f435801b070 con 0x7f4360151a10 2024-04-25T00:20:18.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.545+0000 7f435cff9640 1 --2- 172.21.15.116:0/425180825 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f434c0678c0 0x7f434c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:18.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.545+0000 7f435cff9640 1 -- 172.21.15.116:0/425180825 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 9246+0+0 (secure 0 0 0) 0x7f4358086180 con 0x7f4360151a10 2024-04-25T00:20:18.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.545+0000 7f435ffff640 1 --2- 172.21.15.116:0/425180825 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f434c0678c0 0x7f434c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:18.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.545+0000 7f435ffff640 1 --2- 172.21.15.116:0/425180825 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f434c0678c0 0x7f434c069d80 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f43500af000 tx=0x7f43500b2040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:18.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.549+0000 7f435cff9640 1 -- 172.21.15.116:0/425180825 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f4358055f30 con 0x7f4360151a10 2024-04-25T00:20:18.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.825+0000 7f43662c8640 1 -- 172.21.15.116:0/425180825 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f43600abcc0 con 0x7f4360151a10 2024-04-25T00:20:18.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.829+0000 7f435cff9640 1 -- 172.21.15.116:0/425180825 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 162 v162) ==== 95+0+6500 (secure 0 0 0) 0x7f4358055830 con 0x7f4360151a10 2024-04-25T00:20:18.832 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:20:18.832 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":162,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7809,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/980221971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":980221971},{"type":"v1","addr":"172.21.15.116:6835","nonce":980221971}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":162}],"filesystems":[{"mdsmap":{"epoch":156,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:19:33.982945+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6586":{"gid":6586,"name":"d","rank":0,"incarnation":153,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/991183935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":991183935},{"type":"v1","addr":"172.21.15.174:6837","nonce":991183935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6586,"qdb_cluster":[6586]},"id":1},{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:20:13.935378+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7722":{"gid":7722,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7722,"qdb_cluster":[7722]},"id":2}]} 2024-04-25T00:20:18.832 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 162 2024-04-25T00:20:18.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.833+0000 7f43662c8640 1 -- 172.21.15.116:0/425180825 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f434c0678c0 msgr2=0x7f434c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:18.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.833+0000 7f43662c8640 1 --2- 172.21.15.116:0/425180825 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f434c0678c0 0x7f434c069d80 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f43500af000 tx=0x7f43500b2040 comp rx=0 tx=0).stop 2024-04-25T00:20:18.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.833+0000 7f43662c8640 1 -- 172.21.15.116:0/425180825 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4360151a10 msgr2=0x7f43601edad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:18.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.833+0000 7f43662c8640 1 --2- 172.21.15.116:0/425180825 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4360151a10 0x7f43601edad0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f4360153660 tx=0x7f435800dc00 comp rx=0 tx=0).stop 2024-04-25T00:20:18.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.833+0000 7f43662c8640 1 -- 172.21.15.116:0/425180825 shutdown_connections 2024-04-25T00:20:18.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.833+0000 7f43662c8640 1 --2- 172.21.15.116:0/425180825 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f434c0678c0 0x7f434c069d80 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:18.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.833+0000 7f43662c8640 1 --2- 172.21.15.116:0/425180825 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4360151a10 0x7f43601edad0 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:18.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.833+0000 7f43662c8640 1 --2- 172.21.15.116:0/425180825 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43600c2320 0x7f43601514d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:18.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.833+0000 7f43662c8640 1 --2- 172.21.15.116:0/425180825 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43480039f0 0x7f4360141680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:18.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.833+0000 7f43662c8640 1 -- 172.21.15.116:0/425180825 >> 172.21.15.116:0/425180825 conn(0x7f436006c720 msgr2=0x7f43600ac960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:20:18.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.833+0000 7f43662c8640 1 -- 172.21.15.116:0/425180825 shutdown_connections 2024-04-25T00:20:18.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:18.833+0000 7f43662c8640 1 -- 172.21.15.116:0/425180825 wait complete. 2024-04-25T00:20:18.853 INFO:tasks.mds_thrash.fs.[b]:mds.c reported in up:standby state 2024-04-25T00:20:18.853 INFO:tasks.mds_thrash.fs.[b]:waiting for 119 secs before thrashing 2024-04-25T00:20:36.091 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:20:36.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.241+0000 7f8012530640 1 Processor -- start 2024-04-25T00:20:36.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.241+0000 7f8012530640 1 -- start start 2024-04-25T00:20:36.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.241+0000 7f8012530640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f800c148890 0x7f800c072590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:36.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.241+0000 7f8012530640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f800c070dd0 0x7f800c071270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:36.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.241+0000 7f8012530640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f800c06e890 0x7f800c0717b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:36.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.241+0000 7f8012530640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f800c152c10 con 0x7f800c070dd0 2024-04-25T00:20:36.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.241+0000 7f8012530640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f800c071cf0 con 0x7f800c148890 2024-04-25T00:20:36.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.241+0000 7f8012530640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f800c151e50 con 0x7f800c06e890 2024-04-25T00:20:36.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.241+0000 7f8003fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f800c070dd0 0x7f800c071270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:36.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.241+0000 7f8003fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f800c070dd0 0x7f800c071270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52358/0 (socket says 172.21.15.116:52358) 2024-04-25T00:20:36.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.241+0000 7f8003fff640 1 -- 172.21.15.116:0/3312248304 learned_addr learned my addr 172.21.15.116:0/3312248304 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:20:36.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.241+0000 7f8010acd640 1 --2- 172.21.15.116:0/3312248304 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f800c06e890 0x7f800c0717b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:36.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.241+0000 7f800bfff640 1 --2- 172.21.15.116:0/3312248304 >> v2:172.21.15.174:3300/0 conn(0x7f800c148890 0x7f800c072590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:36.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.241+0000 7f8003fff640 1 -- 172.21.15.116:0/3312248304 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f800c06e890 msgr2=0x7f800c0717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:20:36.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.241+0000 7f8003fff640 1 --2- 172.21.15.116:0/3312248304 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f800c06e890 0x7f800c0717b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:36.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.241+0000 7f8003fff640 1 -- 172.21.15.116:0/3312248304 >> v2:172.21.15.174:3300/0 conn(0x7f800c148890 msgr2=0x7f800c072590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:20:36.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.241+0000 7f8003fff640 1 --2- 172.21.15.116:0/3312248304 >> v2:172.21.15.174:3300/0 conn(0x7f800c148890 0x7f800c072590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:36.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.241+0000 7f8003fff640 1 -- 172.21.15.116:0/3312248304 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f800c152130 con 0x7f800c070dd0 2024-04-25T00:20:36.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.245+0000 7f8003fff640 1 --2- 172.21.15.116:0/3312248304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f800c070dd0 0x7f800c071270 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f800c06f3d0 tx=0x7f7ff401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e03e954ed88b47b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:36.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.245+0000 7f800b7fe640 1 -- 172.21.15.116:0/3312248304 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ff40b0070 con 0x7f800c070dd0 2024-04-25T00:20:36.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.245+0000 7f800b7fe640 1 -- 172.21.15.116:0/3312248304 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ff40ab600 con 0x7f800c070dd0 2024-04-25T00:20:36.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.245+0000 7f800b7fe640 1 -- 172.21.15.116:0/3312248304 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ff40ab9d0 con 0x7f800c070dd0 2024-04-25T00:20:36.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.245+0000 7f8012530640 1 -- 172.21.15.116:0/3312248304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f800c070dd0 msgr2=0x7f800c071270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:36.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.245+0000 7f8012530640 1 --2- 172.21.15.116:0/3312248304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f800c070dd0 0x7f800c071270 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f800c06f3d0 tx=0x7f7ff401c520 comp rx=0 tx=0).stop 2024-04-25T00:20:36.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.245+0000 7f8012530640 1 -- 172.21.15.116:0/3312248304 shutdown_connections 2024-04-25T00:20:36.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.245+0000 7f8012530640 1 --2- 172.21.15.116:0/3312248304 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f800c06e890 0x7f800c0717b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:36.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.245+0000 7f8012530640 1 --2- 172.21.15.116:0/3312248304 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f800c070dd0 0x7f800c071270 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:36.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.245+0000 7f8012530640 1 --2- 172.21.15.116:0/3312248304 >> v2:172.21.15.174:3300/0 conn(0x7f800c148890 0x7f800c072590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:36.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.245+0000 7f8012530640 1 -- 172.21.15.116:0/3312248304 >> 172.21.15.116:0/3312248304 conn(0x7f800c0c3b00 msgr2=0x7f800c0c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:20:36.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.245+0000 7f8012530640 1 -- 172.21.15.116:0/3312248304 shutdown_connections 2024-04-25T00:20:36.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.245+0000 7f8012530640 1 -- 172.21.15.116:0/3312248304 wait complete. 2024-04-25T00:20:36.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.245+0000 7f8012530640 1 Processor -- start 2024-04-25T00:20:36.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.245+0000 7f8012530640 1 -- start start 2024-04-25T00:20:36.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.249+0000 7f8012530640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f800c06e890 0x7f800c1435b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:36.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.249+0000 7f8012530640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f800c070dd0 0x7f800c0bc090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:36.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.249+0000 7f8012530640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f800c148890 0x7f800c143de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:36.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.249+0000 7f8012530640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f800c0ab710 con 0x7f800c070dd0 2024-04-25T00:20:36.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.249+0000 7f8012530640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f800c0ab590 con 0x7f800c06e890 2024-04-25T00:20:36.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.249+0000 7f8012530640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f800c0ab890 con 0x7f800c148890 2024-04-25T00:20:36.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.249+0000 7f8003fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f800c070dd0 0x7f800c0bc090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:36.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.249+0000 7f8003fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f800c070dd0 0x7f800c0bc090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52368/0 (socket says 172.21.15.116:52368) 2024-04-25T00:20:36.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.249+0000 7f8003fff640 1 -- 172.21.15.116:0/1209233722 learned_addr learned my addr 172.21.15.116:0/1209233722 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:20:36.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.249+0000 7f800bfff640 1 --2- 172.21.15.116:0/1209233722 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f800c06e890 0x7f800c1435b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:36.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.249+0000 7f8010acd640 1 --2- 172.21.15.116:0/1209233722 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f800c148890 0x7f800c143de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:36.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.249+0000 7f8003fff640 1 -- 172.21.15.116:0/1209233722 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f800c148890 msgr2=0x7f800c143de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:36.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.249+0000 7f8003fff640 1 --2- 172.21.15.116:0/1209233722 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f800c148890 0x7f800c143de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:36.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.249+0000 7f8003fff640 1 -- 172.21.15.116:0/1209233722 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f800c06e890 msgr2=0x7f800c1435b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:36.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.249+0000 7f8003fff640 1 --2- 172.21.15.116:0/1209233722 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f800c06e890 0x7f800c1435b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:36.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.249+0000 7f8003fff640 1 -- 172.21.15.116:0/1209233722 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f800c143070 con 0x7f800c070dd0 2024-04-25T00:20:36.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.249+0000 7f800bfff640 1 --2- 172.21.15.116:0/1209233722 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f800c06e890 0x7f800c1435b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:20:36.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.249+0000 7f8003fff640 1 --2- 172.21.15.116:0/1209233722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f800c070dd0 0x7f800c0bc090 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f7ff401abd0 tx=0x7f7ff40192c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:36.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.249+0000 7f80097fa640 1 -- 172.21.15.116:0/1209233722 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ff40b0040 con 0x7f800c070dd0 2024-04-25T00:20:36.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.249+0000 7f80097fa640 1 -- 172.21.15.116:0/1209233722 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ff40b4420 con 0x7f800c070dd0 2024-04-25T00:20:36.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.249+0000 7f8012530640 1 -- 172.21.15.116:0/1209233722 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f800c1ed310 con 0x7f800c070dd0 2024-04-25T00:20:36.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.249+0000 7f80097fa640 1 -- 172.21.15.116:0/1209233722 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ff40b47f0 con 0x7f800c070dd0 2024-04-25T00:20:36.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.249+0000 7f8012530640 1 -- 172.21.15.116:0/1209233722 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f800c144320 con 0x7f800c070dd0 2024-04-25T00:20:36.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.249+0000 7f8012530640 1 -- 172.21.15.116:0/1209233722 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ff00052a0 con 0x7f800c070dd0 2024-04-25T00:20:36.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.249+0000 7f80097fa640 1 -- 172.21.15.116:0/1209233722 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7ff40b4990 con 0x7f800c070dd0 2024-04-25T00:20:36.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.249+0000 7f80097fa640 1 --2- 172.21.15.116:0/1209233722 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7fec0678c0 0x7f7fec069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:36.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.249+0000 7f80097fa640 1 -- 172.21.15.116:0/1209233722 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 9246+0+0 (secure 0 0 0) 0x7f7ff41206a0 con 0x7f800c070dd0 2024-04-25T00:20:36.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.253+0000 7f800bfff640 1 --2- 172.21.15.116:0/1209233722 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7fec0678c0 0x7f7fec069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:36.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.253+0000 7f800bfff640 1 --2- 172.21.15.116:0/1209233722 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7fec0678c0 0x7f7fec069d80 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc00ba40 tx=0x7f7ffc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:36.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.257+0000 7f80097fa640 1 -- 172.21.15.116:0/1209233722 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f7ff4016110 con 0x7f800c070dd0 2024-04-25T00:20:36.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.513+0000 7f8012530640 1 -- 172.21.15.116:0/1209233722 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7ff0005590 con 0x7f800c070dd0 2024-04-25T00:20:36.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.513+0000 7f80097fa640 1 -- 172.21.15.116:0/1209233722 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 162 v162) ==== 95+0+6500 (secure 0 0 0) 0x7f7ff40f0450 con 0x7f800c070dd0 2024-04-25T00:20:36.514 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:20:36.514 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":162,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7809,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/980221971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":980221971},{"type":"v1","addr":"172.21.15.116:6835","nonce":980221971}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":162}],"filesystems":[{"mdsmap":{"epoch":156,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:19:33.982945+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6586":{"gid":6586,"name":"d","rank":0,"incarnation":153,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/991183935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":991183935},{"type":"v1","addr":"172.21.15.174:6837","nonce":991183935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6586,"qdb_cluster":[6586]},"id":1},{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:20:13.935378+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7722":{"gid":7722,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7722,"qdb_cluster":[7722]},"id":2}]} 2024-04-25T00:20:36.515 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 162 2024-04-25T00:20:36.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.517+0000 7f8012530640 1 -- 172.21.15.116:0/1209233722 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7fec0678c0 msgr2=0x7f7fec069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:36.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.517+0000 7f8012530640 1 --2- 172.21.15.116:0/1209233722 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7fec0678c0 0x7f7fec069d80 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc00ba40 tx=0x7f7ffc008040 comp rx=0 tx=0).stop 2024-04-25T00:20:36.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.517+0000 7f8012530640 1 -- 172.21.15.116:0/1209233722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f800c070dd0 msgr2=0x7f800c0bc090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:36.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.517+0000 7f8012530640 1 --2- 172.21.15.116:0/1209233722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f800c070dd0 0x7f800c0bc090 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f7ff401abd0 tx=0x7f7ff40192c0 comp rx=0 tx=0).stop 2024-04-25T00:20:36.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.517+0000 7f8012530640 1 -- 172.21.15.116:0/1209233722 shutdown_connections 2024-04-25T00:20:36.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.517+0000 7f8012530640 1 --2- 172.21.15.116:0/1209233722 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7fec0678c0 0x7f7fec069d80 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:36.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.517+0000 7f8012530640 1 --2- 172.21.15.116:0/1209233722 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f800c148890 0x7f800c143de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:36.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.517+0000 7f8012530640 1 --2- 172.21.15.116:0/1209233722 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f800c070dd0 0x7f800c0bc090 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:36.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.517+0000 7f8012530640 1 --2- 172.21.15.116:0/1209233722 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f800c06e890 0x7f800c1435b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:36.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.517+0000 7f8012530640 1 -- 172.21.15.116:0/1209233722 >> 172.21.15.116:0/1209233722 conn(0x7f800c0c3b00 msgr2=0x7f800c155440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:20:36.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.517+0000 7f8012530640 1 -- 172.21.15.116:0/1209233722 shutdown_connections 2024-04-25T00:20:36.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.517+0000 7f8012530640 1 -- 172.21.15.116:0/1209233722 wait complete. 2024-04-25T00:20:36.535 INFO:tasks.mds_thrash.fs.[a]:kill mds.d (rank=0) 2024-04-25T00:20:36.536 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-25T00:20:36.536 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:20:36.549 INFO:tasks.ceph.mds.d:Stopped 2024-04-25T00:20:36.549 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:20:36.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.669+0000 7f189a569640 1 Processor -- start 2024-04-25T00:20:36.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.669+0000 7f189a569640 1 -- start start 2024-04-25T00:20:36.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.669+0000 7f189a569640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f18941599a0 0x7f18941562d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:36.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.669+0000 7f189a569640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1894073200 con 0x7f18940c5010 2024-04-25T00:20:36.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.669+0000 7f189a569640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1894158ec0 con 0x7f1894152e60 2024-04-25T00:20:36.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.669+0000 7f189a569640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f1894153240 con 0x7f18941599a0 2024-04-25T00:20:36.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.669+0000 7f1898b06640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f18940c5010 0x7f1894160120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:37272/0 (socket says 172.21.15.116:37272) 2024-04-25T00:20:36.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.669+0000 7f1898b06640 1 -- 172.21.15.116:0/10012155 learned_addr learned my addr 172.21.15.116:0/10012155 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:20:36.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.669+0000 7f18937fe640 1 --2- 172.21.15.116:0/10012155 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f18941599a0 0x7f18941562d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:36.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.669+0000 7f1892ffd640 1 -- 172.21.15.116:0/10012155 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1323595978 0 0) 0x7f1894073200 con 0x7f18940c5010 2024-04-25T00:20:36.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.669+0000 7f1892ffd640 1 -- 172.21.15.116:0/10012155 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f187c003530 con 0x7f18940c5010 2024-04-25T00:20:36.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.669+0000 7f1892ffd640 1 -- 172.21.15.116:0/10012155 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1214323802 0 0) 0x7f1894158ec0 con 0x7f1894152e60 2024-04-25T00:20:36.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.669+0000 7f1892ffd640 1 -- 172.21.15.116:0/10012155 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1894073200 con 0x7f1894152e60 2024-04-25T00:20:36.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.669+0000 7f18937fe640 1 -- 172.21.15.116:0/10012155 >> v1:172.21.15.174:6789/0 conn(0x7f1894152e60 legacy=0x7f18940c36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:36.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.669+0000 7f18937fe640 1 -- 172.21.15.116:0/10012155 >> v1:172.21.15.116:6789/0 conn(0x7f18940c5010 legacy=0x7f1894160120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:36.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.669+0000 7f18937fe640 1 -- 172.21.15.116:0/10012155 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18941591a0 con 0x7f18941599a0 2024-04-25T00:20:36.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.673+0000 7f18937fe640 1 --2- 172.21.15.116:0/10012155 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f18941599a0 0x7f18941562d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f18880097f0 tx=0x7f188801c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4757965af1435715 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:36.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.673+0000 7f1892ffd640 1 -- 172.21.15.116:0/10012155 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18880b0070 con 0x7f18941599a0 2024-04-25T00:20:36.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.673+0000 7f1892ffd640 1 -- 172.21.15.116:0/10012155 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18880ab600 con 0x7f18941599a0 2024-04-25T00:20:36.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.673+0000 7f1892ffd640 1 -- 172.21.15.116:0/10012155 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18880ab9b0 con 0x7f18941599a0 2024-04-25T00:20:36.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.673+0000 7f189a569640 1 -- 172.21.15.116:0/10012155 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f18941599a0 msgr2=0x7f18941562d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:36.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.673+0000 7f189a569640 1 --2- 172.21.15.116:0/10012155 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f18941599a0 0x7f18941562d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f18880097f0 tx=0x7f188801c450 comp rx=0 tx=0).stop 2024-04-25T00:20:36.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.673+0000 7f189a569640 1 -- 172.21.15.116:0/10012155 shutdown_connections 2024-04-25T00:20:36.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.673+0000 7f189a569640 1 --2- 172.21.15.116:0/10012155 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f18941599a0 0x7f18941562d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:36.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.673+0000 7f189a569640 1 -- 172.21.15.116:0/10012155 >> 172.21.15.116:0/10012155 conn(0x7f189406c720 msgr2=0x7f189406cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:20:36.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.673+0000 7f189a569640 1 -- 172.21.15.116:0/10012155 shutdown_connections 2024-04-25T00:20:36.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.673+0000 7f189a569640 1 -- 172.21.15.116:0/10012155 wait complete. 2024-04-25T00:20:36.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.677+0000 7f189a569640 1 Processor -- start 2024-04-25T00:20:36.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.677+0000 7f189a569640 1 -- start start 2024-04-25T00:20:36.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.677+0000 7f189a569640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f18940c5010 0x7f18941548c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:36.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.677+0000 7f189a569640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1894152e60 0x7f18941eec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:36.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.677+0000 7f189a569640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18941599a0 0x7f18941ef1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:36.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.677+0000 7f189a569640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1894072e20 con 0x7f18941599a0 2024-04-25T00:20:36.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.677+0000 7f189a569640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f18940c5830 con 0x7f18940c5010 2024-04-25T00:20:36.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.677+0000 7f189a569640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f1894072fa0 con 0x7f1894152e60 2024-04-25T00:20:36.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.677+0000 7f1898b06640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18941599a0 0x7f18941ef1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:36.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.677+0000 7f1898b06640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18941599a0 0x7f18941ef1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52372/0 (socket says 172.21.15.116:52372) 2024-04-25T00:20:36.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.677+0000 7f1898b06640 1 -- 172.21.15.116:0/1726992851 learned_addr learned my addr 172.21.15.116:0/1726992851 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:20:36.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.677+0000 7f18937fe640 1 --2- 172.21.15.116:0/1726992851 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1894152e60 0x7f18941eec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:36.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.677+0000 7f1893fff640 1 --2- 172.21.15.116:0/1726992851 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f18940c5010 0x7f18941548c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:36.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.677+0000 7f1898b06640 1 -- 172.21.15.116:0/1726992851 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1894152e60 msgr2=0x7f18941eec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:36.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.677+0000 7f1898b06640 1 --2- 172.21.15.116:0/1726992851 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1894152e60 0x7f18941eec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:36.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.677+0000 7f1898b06640 1 -- 172.21.15.116:0/1726992851 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f18940c5010 msgr2=0x7f18941548c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:36.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.677+0000 7f1898b06640 1 --2- 172.21.15.116:0/1726992851 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f18940c5010 0x7f18941548c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:36.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.677+0000 7f1898b06640 1 -- 172.21.15.116:0/1726992851 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18941ee7b0 con 0x7f18941599a0 2024-04-25T00:20:36.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.677+0000 7f1893fff640 1 --2- 172.21.15.116:0/1726992851 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f18940c5010 0x7f18941548c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:20:36.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.677+0000 7f1898b06640 1 --2- 172.21.15.116:0/1726992851 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18941599a0 0x7f18941ef1d0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f188c00caf0 tx=0x7f188c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:36.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.677+0000 7f1890ff9640 1 -- 172.21.15.116:0/1726992851 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f188c017070 con 0x7f18941599a0 2024-04-25T00:20:36.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.677+0000 7f1890ff9640 1 -- 172.21.15.116:0/1726992851 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f188c00aca0 con 0x7f18941599a0 2024-04-25T00:20:36.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.677+0000 7f189a569640 1 -- 172.21.15.116:0/1726992851 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18940aced0 con 0x7f18941599a0 2024-04-25T00:20:36.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.677+0000 7f1890ff9640 1 -- 172.21.15.116:0/1726992851 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f188c012620 con 0x7f18941599a0 2024-04-25T00:20:36.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.677+0000 7f189a569640 1 -- 172.21.15.116:0/1726992851 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18940ad490 con 0x7f18941599a0 2024-04-25T00:20:36.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.681+0000 7f189a569640 1 -- 172.21.15.116:0/1726992851 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1894068a60 con 0x7f18941599a0 2024-04-25T00:20:36.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.681+0000 7f1890ff9640 1 -- 172.21.15.116:0/1726992851 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f188c01a070 con 0x7f18941599a0 2024-04-25T00:20:36.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.681+0000 7f1890ff9640 1 --2- 172.21.15.116:0/1726992851 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f18800678c0 0x7f1880069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:36.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.681+0000 7f1890ff9640 1 -- 172.21.15.116:0/1726992851 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 9246+0+0 (secure 0 0 0) 0x7f188c086700 con 0x7f18941599a0 2024-04-25T00:20:36.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.681+0000 7f1893fff640 1 --2- 172.21.15.116:0/1726992851 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f18800678c0 0x7f1880069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:36.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.681+0000 7f1893fff640 1 --2- 172.21.15.116:0/1726992851 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f18800678c0 0x7f1880069d80 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f18840011f0 tx=0x7f1884009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:36.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.685+0000 7f1890ff9640 1 -- 172.21.15.116:0/1726992851 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f188c055e60 con 0x7f18941599a0 2024-04-25T00:20:36.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.905+0000 7f189a569640 1 -- 172.21.15.116:0/1726992851 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f18940c24e0 con 0x7f18941599a0 2024-04-25T00:20:36.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.905+0000 7f1890ff9640 1 -- 172.21.15.116:0/1726992851 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 162 v162) ==== 95+0+6500 (secure 0 0 0) 0x7f188c05a160 con 0x7f18941599a0 2024-04-25T00:20:36.909 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:20:36.909 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":162,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7809,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/980221971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":980221971},{"type":"v1","addr":"172.21.15.116:6835","nonce":980221971}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":162}],"filesystems":[{"mdsmap":{"epoch":156,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:19:33.982945+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6586":{"gid":6586,"name":"d","rank":0,"incarnation":153,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/991183935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":991183935},{"type":"v1","addr":"172.21.15.174:6837","nonce":991183935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6586,"qdb_cluster":[6586]},"id":1},{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:20:13.935378+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7722":{"gid":7722,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7722,"qdb_cluster":[7722]},"id":2}]} 2024-04-25T00:20:36.909 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 162 2024-04-25T00:20:36.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.909+0000 7f189a569640 1 -- 172.21.15.116:0/1726992851 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f18800678c0 msgr2=0x7f1880069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:36.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.909+0000 7f189a569640 1 --2- 172.21.15.116:0/1726992851 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f18800678c0 0x7f1880069d80 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f18840011f0 tx=0x7f1884009040 comp rx=0 tx=0).stop 2024-04-25T00:20:36.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.909+0000 7f189a569640 1 -- 172.21.15.116:0/1726992851 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18941599a0 msgr2=0x7f18941ef1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:36.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.909+0000 7f189a569640 1 --2- 172.21.15.116:0/1726992851 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18941599a0 0x7f18941ef1d0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f188c00caf0 tx=0x7f188c00a060 comp rx=0 tx=0).stop 2024-04-25T00:20:36.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.909+0000 7f189a569640 1 -- 172.21.15.116:0/1726992851 shutdown_connections 2024-04-25T00:20:36.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.909+0000 7f189a569640 1 --2- 172.21.15.116:0/1726992851 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f18800678c0 0x7f1880069d80 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:36.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.909+0000 7f189a569640 1 --2- 172.21.15.116:0/1726992851 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18941599a0 0x7f18941ef1d0 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:36.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.909+0000 7f189a569640 1 --2- 172.21.15.116:0/1726992851 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1894152e60 0x7f18941eec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:36.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.909+0000 7f189a569640 1 --2- 172.21.15.116:0/1726992851 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f18940c5010 0x7f18941548c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:36.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.909+0000 7f189a569640 1 -- 172.21.15.116:0/1726992851 >> 172.21.15.116:0/1726992851 conn(0x7f189406c720 msgr2=0x7f189406cb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:20:36.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.909+0000 7f189a569640 1 -- 172.21.15.116:0/1726992851 shutdown_connections 2024-04-25T00:20:36.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:36.909+0000 7f189a569640 1 -- 172.21.15.116:0/1726992851 wait complete. 2024-04-25T00:20:36.933 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:20:38.934 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:20:39.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.065+0000 7f2932598640 1 Processor -- start 2024-04-25T00:20:39.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.065+0000 7f2932598640 1 -- start start 2024-04-25T00:20:39.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.065+0000 7f2932598640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f292c06d5f0 0x7f292c156c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:39.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.065+0000 7f2932598640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f292c155810 0x7f292c1571c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:39.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.065+0000 7f2932598640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f292c0c9150 con 0x7f292c152d80 2024-04-25T00:20:39.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.065+0000 7f2932598640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f292c0c9d60 con 0x7f292c155810 2024-04-25T00:20:39.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.065+0000 7f2932598640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f292c0c9430 con 0x7f292c06d5f0 2024-04-25T00:20:39.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.065+0000 7f2930b35640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f292c155810 0x7f292c1571c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:39.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.065+0000 7f2930b35640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f292c155810 0x7f292c1571c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52382/0 (socket says 172.21.15.116:52382) 2024-04-25T00:20:39.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.065+0000 7f2930b35640 1 -- 172.21.15.116:0/1325668202 learned_addr learned my addr 172.21.15.116:0/1325668202 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:20:39.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.065+0000 7f292b7fe640 1 --2- 172.21.15.116:0/1325668202 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f292c06d5f0 0x7f292c156c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:39.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.065+0000 7f2930b35640 1 -- 172.21.15.116:0/1325668202 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f292c06d5f0 msgr2=0x7f292c156c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:20:39.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.065+0000 7f2930b35640 1 --2- 172.21.15.116:0/1325668202 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f292c06d5f0 0x7f292c156c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:39.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.065+0000 7f2930b35640 1 -- 172.21.15.116:0/1325668202 >> v1:172.21.15.174:6789/0 conn(0x7f292c152d80 legacy=0x7f292c06bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:39.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.065+0000 7f2930b35640 1 -- 172.21.15.116:0/1325668202 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f292c0c9710 con 0x7f292c155810 2024-04-25T00:20:39.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.069+0000 7f2930b35640 1 --2- 172.21.15.116:0/1325668202 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f292c155810 0x7f292c1571c0 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f2924009e50 tx=0x7f292401c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56904add5c065beb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:39.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.069+0000 7f292affd640 1 -- 172.21.15.116:0/1325668202 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29240a56d0 con 0x7f292c155810 2024-04-25T00:20:39.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.069+0000 7f292affd640 1 -- 172.21.15.116:0/1325668202 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29240a5870 con 0x7f292c155810 2024-04-25T00:20:39.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.069+0000 7f292affd640 1 -- 172.21.15.116:0/1325668202 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29240a5c40 con 0x7f292c155810 2024-04-25T00:20:39.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.069+0000 7f2932598640 1 -- 172.21.15.116:0/1325668202 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f292c155810 msgr2=0x7f292c1571c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:39.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.069+0000 7f2932598640 1 --2- 172.21.15.116:0/1325668202 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f292c155810 0x7f292c1571c0 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f2924009e50 tx=0x7f292401c8e0 comp rx=0 tx=0).stop 2024-04-25T00:20:39.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.069+0000 7f2932598640 1 -- 172.21.15.116:0/1325668202 shutdown_connections 2024-04-25T00:20:39.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.069+0000 7f2932598640 1 --2- 172.21.15.116:0/1325668202 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f292c155810 0x7f292c1571c0 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:39.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.069+0000 7f2932598640 1 --2- 172.21.15.116:0/1325668202 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f292c06d5f0 0x7f292c156c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:39.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.069+0000 7f2932598640 1 -- 172.21.15.116:0/1325668202 >> 172.21.15.116:0/1325668202 conn(0x7f292c0acdd0 msgr2=0x7f292c0ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:20:39.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.069+0000 7f2932598640 1 -- 172.21.15.116:0/1325668202 shutdown_connections 2024-04-25T00:20:39.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.069+0000 7f2932598640 1 -- 172.21.15.116:0/1325668202 wait complete. 2024-04-25T00:20:39.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.069+0000 7f2932598640 1 Processor -- start 2024-04-25T00:20:39.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.069+0000 7f2932598640 1 -- start start 2024-04-25T00:20:39.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.069+0000 7f2932598640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f292c06d5f0 0x7f292c1f24f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:39.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.069+0000 7f2932598640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f292c152d80 0x7f292c1544c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:39.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.069+0000 7f2932598640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f292c155810 0x7f292c14bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:39.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.069+0000 7f2932598640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f292c0caf00 con 0x7f292c155810 2024-04-25T00:20:39.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.069+0000 7f2932598640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f292c0cad80 con 0x7f292c152d80 2024-04-25T00:20:39.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.069+0000 7f2932598640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f292c1628d0 con 0x7f292c06d5f0 2024-04-25T00:20:39.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.069+0000 7f2930b35640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f292c155810 0x7f292c14bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:39.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.069+0000 7f2930b35640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f292c155810 0x7f292c14bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52384/0 (socket says 172.21.15.116:52384) 2024-04-25T00:20:39.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.069+0000 7f2930b35640 1 -- 172.21.15.116:0/2973641552 learned_addr learned my addr 172.21.15.116:0/2973641552 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:20:39.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.069+0000 7f292b7fe640 1 --2- 172.21.15.116:0/2973641552 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f292c152d80 0x7f292c1544c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:39.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.069+0000 7f292bfff640 1 --2- 172.21.15.116:0/2973641552 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f292c06d5f0 0x7f292c1f24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:39.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.069+0000 7f2930b35640 1 -- 172.21.15.116:0/2973641552 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f292c06d5f0 msgr2=0x7f292c1f24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:39.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.069+0000 7f2930b35640 1 --2- 172.21.15.116:0/2973641552 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f292c06d5f0 0x7f292c1f24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:39.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.069+0000 7f2930b35640 1 -- 172.21.15.116:0/2973641552 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f292c152d80 msgr2=0x7f292c1544c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:39.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.069+0000 7f2930b35640 1 --2- 172.21.15.116:0/2973641552 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f292c152d80 0x7f292c1544c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:39.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.069+0000 7f2930b35640 1 -- 172.21.15.116:0/2973641552 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f292c1f2a30 con 0x7f292c155810 2024-04-25T00:20:39.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.069+0000 7f292bfff640 1 --2- 172.21.15.116:0/2973641552 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f292c06d5f0 0x7f292c1f24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:20:39.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.069+0000 7f2930b35640 1 --2- 172.21.15.116:0/2973641552 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f292c155810 0x7f292c14bf00 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f292401a510 tx=0x7f2924019680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:39.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.069+0000 7f2928ff9640 1 -- 172.21.15.116:0/2973641552 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29240bc030 con 0x7f292c155810 2024-04-25T00:20:39.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.069+0000 7f2928ff9640 1 -- 172.21.15.116:0/2973641552 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29240b8470 con 0x7f292c155810 2024-04-25T00:20:39.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.069+0000 7f2932598640 1 -- 172.21.15.116:0/2973641552 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f292c150dd0 con 0x7f292c155810 2024-04-25T00:20:39.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.069+0000 7f2928ff9640 1 -- 172.21.15.116:0/2973641552 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29240b8860 con 0x7f292c155810 2024-04-25T00:20:39.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.069+0000 7f2932598640 1 -- 172.21.15.116:0/2973641552 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f292c151e90 con 0x7f292c155810 2024-04-25T00:20:39.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.073+0000 7f2932598640 1 -- 172.21.15.116:0/2973641552 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f292c068350 con 0x7f292c155810 2024-04-25T00:20:39.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.073+0000 7f2928ff9640 1 -- 172.21.15.116:0/2973641552 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f29240bf070 con 0x7f292c155810 2024-04-25T00:20:39.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.073+0000 7f2928ff9640 1 --2- 172.21.15.116:0/2973641552 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f29180678c0 0x7f2918069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:39.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.073+0000 7f2928ff9640 1 -- 172.21.15.116:0/2973641552 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 9246+0+0 (secure 0 0 0) 0x7f29241221b0 con 0x7f292c155810 2024-04-25T00:20:39.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.073+0000 7f292bfff640 1 --2- 172.21.15.116:0/2973641552 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f29180678c0 0x7f2918069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:39.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.073+0000 7f292bfff640 1 --2- 172.21.15.116:0/2973641552 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f29180678c0 0x7f2918069d80 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f291c001650 tx=0x7f291c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:39.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.077+0000 7f2928ff9640 1 -- 172.21.15.116:0/2973641552 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f292c150dd0 con 0x7f292c155810 2024-04-25T00:20:39.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.277+0000 7f2932598640 1 -- 172.21.15.116:0/2973641552 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f292c0759d0 con 0x7f292c155810 2024-04-25T00:20:39.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.277+0000 7f2928ff9640 1 -- 172.21.15.116:0/2973641552 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 162 v162) ==== 95+0+6500 (secure 0 0 0) 0x7f2924016620 con 0x7f292c155810 2024-04-25T00:20:39.279 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:20:39.279 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":162,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7809,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/980221971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":980221971},{"type":"v1","addr":"172.21.15.116:6835","nonce":980221971}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":162}],"filesystems":[{"mdsmap":{"epoch":156,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:19:33.982945+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6586":{"gid":6586,"name":"d","rank":0,"incarnation":153,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/991183935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":991183935},{"type":"v1","addr":"172.21.15.174:6837","nonce":991183935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6586,"qdb_cluster":[6586]},"id":1},{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:20:13.935378+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7722":{"gid":7722,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7722,"qdb_cluster":[7722]},"id":2}]} 2024-04-25T00:20:39.279 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 162 2024-04-25T00:20:39.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.281+0000 7f2932598640 1 -- 172.21.15.116:0/2973641552 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f29180678c0 msgr2=0x7f2918069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:39.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.281+0000 7f2932598640 1 --2- 172.21.15.116:0/2973641552 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f29180678c0 0x7f2918069d80 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f291c001650 tx=0x7f291c009040 comp rx=0 tx=0).stop 2024-04-25T00:20:39.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.281+0000 7f2932598640 1 -- 172.21.15.116:0/2973641552 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f292c155810 msgr2=0x7f292c14bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:39.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.281+0000 7f2932598640 1 --2- 172.21.15.116:0/2973641552 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f292c155810 0x7f292c14bf00 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f292401a510 tx=0x7f2924019680 comp rx=0 tx=0).stop 2024-04-25T00:20:39.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.281+0000 7f2932598640 1 -- 172.21.15.116:0/2973641552 shutdown_connections 2024-04-25T00:20:39.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.281+0000 7f2932598640 1 --2- 172.21.15.116:0/2973641552 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f29180678c0 0x7f2918069d80 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:39.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.281+0000 7f2932598640 1 --2- 172.21.15.116:0/2973641552 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f292c155810 0x7f292c14bf00 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:39.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.281+0000 7f2932598640 1 --2- 172.21.15.116:0/2973641552 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f292c152d80 0x7f292c1544c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:39.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.281+0000 7f2932598640 1 --2- 172.21.15.116:0/2973641552 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f292c06d5f0 0x7f292c1f24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:39.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.281+0000 7f2932598640 1 -- 172.21.15.116:0/2973641552 >> 172.21.15.116:0/2973641552 conn(0x7f292c0acdd0 msgr2=0x7f292c155d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:20:39.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.281+0000 7f2932598640 1 -- 172.21.15.116:0/2973641552 shutdown_connections 2024-04-25T00:20:39.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:39.281+0000 7f2932598640 1 -- 172.21.15.116:0/2973641552 wait complete. 2024-04-25T00:20:39.298 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:20:39.801 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:20:39.804 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:20:41.299 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:20:41.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.429+0000 7fad692a0640 1 Processor -- start 2024-04-25T00:20:41.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.429+0000 7fad692a0640 1 -- start start 2024-04-25T00:20:41.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.433+0000 7fad692a0640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fad64150d20 0x7fad640c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:41.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.433+0000 7fad692a0640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad640c3910 con 0x7fad640a94a0 2024-04-25T00:20:41.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.433+0000 7fad692a0640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad640c16e0 con 0x7fad640c2f50 2024-04-25T00:20:41.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.433+0000 7fad692a0640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fad640c1400 con 0x7fad64150d20 2024-04-25T00:20:41.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.433+0000 7fad62ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fad64150d20 0x7fad640c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:41.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.433+0000 7fad62ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fad64150d20 0x7fad640c2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52398/0 (socket says 172.21.15.116:52398) 2024-04-25T00:20:41.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.433+0000 7fad62ffd640 1 -- 172.21.15.116:0/1540736734 learned_addr learned my addr 172.21.15.116:0/1540736734 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:20:41.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.433+0000 7fad637fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fad640a94a0 0x7fad64153ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:37274/0 (socket says 172.21.15.116:37274) 2024-04-25T00:20:41.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.433+0000 7fad62ffd640 1 -- 172.21.15.116:0/1540736734 >> v1:172.21.15.174:6789/0 conn(0x7fad640c2f50 legacy=0x7fad641538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:20:41.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.433+0000 7fad62ffd640 1 -- 172.21.15.116:0/1540736734 >> v1:172.21.15.116:6789/0 conn(0x7fad640a94a0 legacy=0x7fad64153ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:41.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.433+0000 7fad62ffd640 1 -- 172.21.15.116:0/1540736734 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad6406e660 con 0x7fad64150d20 2024-04-25T00:20:41.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.433+0000 7fad61ffb640 1 -- 172.21.15.116:0/1540736734 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1426964525 0 0) 0x7fad640c3910 con 0x7fad640a94a0 2024-04-25T00:20:41.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.433+0000 7fad62ffd640 1 --2- 172.21.15.116:0/1540736734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad64150d20 0x7fad640c2a10 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7fad54009870 tx=0x7fad5401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dafcf6823f477a57 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:41.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.433+0000 7fad61ffb640 1 -- 172.21.15.116:0/1540736734 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad540ab430 con 0x7fad64150d20 2024-04-25T00:20:41.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.433+0000 7fad61ffb640 1 -- 172.21.15.116:0/1540736734 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad540b0070 con 0x7fad64150d20 2024-04-25T00:20:41.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.433+0000 7fad61ffb640 1 -- 172.21.15.116:0/1540736734 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad540ab7f0 con 0x7fad64150d20 2024-04-25T00:20:41.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.437+0000 7fad692a0640 1 -- 172.21.15.116:0/1540736734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad64150d20 msgr2=0x7fad640c2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:41.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.437+0000 7fad692a0640 1 --2- 172.21.15.116:0/1540736734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad64150d20 0x7fad640c2a10 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7fad54009870 tx=0x7fad5401c520 comp rx=0 tx=0).stop 2024-04-25T00:20:41.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.437+0000 7fad692a0640 1 -- 172.21.15.116:0/1540736734 shutdown_connections 2024-04-25T00:20:41.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.437+0000 7fad692a0640 1 --2- 172.21.15.116:0/1540736734 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad64150d20 0x7fad640c2a10 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:41.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.437+0000 7fad692a0640 1 -- 172.21.15.116:0/1540736734 >> 172.21.15.116:0/1540736734 conn(0x7fad640c3b00 msgr2=0x7fad640c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:20:41.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.437+0000 7fad692a0640 1 -- 172.21.15.116:0/1540736734 shutdown_connections 2024-04-25T00:20:41.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.437+0000 7fad692a0640 1 -- 172.21.15.116:0/1540736734 wait complete. 2024-04-25T00:20:41.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.437+0000 7fad692a0640 1 Processor -- start 2024-04-25T00:20:41.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.437+0000 7fad692a0640 1 -- start start 2024-04-25T00:20:41.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.437+0000 7fad692a0640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fad640a94a0 0x7fad641440b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:41.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.437+0000 7fad692a0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fad640c2f50 0x7fad64153c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:41.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.437+0000 7fad692a0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad64150d20 0x7fad641f2570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:41.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.437+0000 7fad692a0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fad64161ad0 con 0x7fad64150d20 2024-04-25T00:20:41.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.437+0000 7fad692a0640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fad64161950 con 0x7fad640c2f50 2024-04-25T00:20:41.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.437+0000 7fad692a0640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fad64161c50 con 0x7fad640a94a0 2024-04-25T00:20:41.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.437+0000 7fad637fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad64150d20 0x7fad641f2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:41.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.437+0000 7fad637fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad64150d20 0x7fad641f2570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52410/0 (socket says 172.21.15.116:52410) 2024-04-25T00:20:41.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.437+0000 7fad637fe640 1 -- 172.21.15.116:0/2764012773 learned_addr learned my addr 172.21.15.116:0/2764012773 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:20:41.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.437+0000 7fad627fc640 1 --2- 172.21.15.116:0/2764012773 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fad640c2f50 0x7fad64153c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:41.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.437+0000 7fad62ffd640 1 --2- 172.21.15.116:0/2764012773 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fad640a94a0 0x7fad641440b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:41.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.437+0000 7fad637fe640 1 -- 172.21.15.116:0/2764012773 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fad640a94a0 msgr2=0x7fad641440b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:41.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.437+0000 7fad637fe640 1 --2- 172.21.15.116:0/2764012773 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fad640a94a0 0x7fad641440b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:41.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.437+0000 7fad637fe640 1 -- 172.21.15.116:0/2764012773 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fad640c2f50 msgr2=0x7fad64153c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:41.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.437+0000 7fad637fe640 1 --2- 172.21.15.116:0/2764012773 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fad640c2f50 0x7fad64153c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:41.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.437+0000 7fad637fe640 1 -- 172.21.15.116:0/2764012773 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad640728e0 con 0x7fad64150d20 2024-04-25T00:20:41.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.437+0000 7fad637fe640 1 --2- 172.21.15.116:0/2764012773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad64150d20 0x7fad641f2570 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7fad5c00c7d0 tx=0x7fad5c00cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:41.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.437+0000 7fad4b7fe640 1 -- 172.21.15.116:0/2764012773 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad5c018070 con 0x7fad64150d20 2024-04-25T00:20:41.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.437+0000 7fad4b7fe640 1 -- 172.21.15.116:0/2764012773 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad5c00a640 con 0x7fad64150d20 2024-04-25T00:20:41.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.437+0000 7fad692a0640 1 -- 172.21.15.116:0/2764012773 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad64072bc0 con 0x7fad64150d20 2024-04-25T00:20:41.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.437+0000 7fad4b7fe640 1 -- 172.21.15.116:0/2764012773 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad5c00aa70 con 0x7fad64150d20 2024-04-25T00:20:41.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.437+0000 7fad692a0640 1 -- 172.21.15.116:0/2764012773 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad641f2050 con 0x7fad64150d20 2024-04-25T00:20:41.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.441+0000 7fad692a0640 1 -- 172.21.15.116:0/2764012773 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad641f2250 con 0x7fad64150d20 2024-04-25T00:20:41.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.441+0000 7fad4b7fe640 1 -- 172.21.15.116:0/2764012773 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fad5c01b070 con 0x7fad64150d20 2024-04-25T00:20:41.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.441+0000 7fad4b7fe640 1 --2- 172.21.15.116:0/2764012773 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fad500678f0 0x7fad50069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:41.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.441+0000 7fad4b7fe640 1 -- 172.21.15.116:0/2764012773 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 9246+0+0 (secure 0 0 0) 0x7fad5c085f70 con 0x7fad64150d20 2024-04-25T00:20:41.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.441+0000 7fad62ffd640 1 --2- 172.21.15.116:0/2764012773 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fad500678f0 0x7fad50069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:41.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.441+0000 7fad62ffd640 1 --2- 172.21.15.116:0/2764012773 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fad500678f0 0x7fad50069db0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fad540af000 tx=0x7fad540a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:41.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.445+0000 7fad4b7fe640 1 -- 172.21.15.116:0/2764012773 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fad641f2250 con 0x7fad64150d20 2024-04-25T00:20:41.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.637+0000 7fad692a0640 1 -- 172.21.15.116:0/2764012773 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fad64068350 con 0x7fad64150d20 2024-04-25T00:20:41.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.641+0000 7fad4b7fe640 1 -- 172.21.15.116:0/2764012773 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 162 v162) ==== 95+0+6500 (secure 0 0 0) 0x7fad5c0556d0 con 0x7fad64150d20 2024-04-25T00:20:41.641 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:20:41.641 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":162,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7809,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/980221971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":980221971},{"type":"v1","addr":"172.21.15.116:6835","nonce":980221971}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":162}],"filesystems":[{"mdsmap":{"epoch":156,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:19:33.982945+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6586":{"gid":6586,"name":"d","rank":0,"incarnation":153,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/991183935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":991183935},{"type":"v1","addr":"172.21.15.174:6837","nonce":991183935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6586,"qdb_cluster":[6586]},"id":1},{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:20:13.935378+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7722":{"gid":7722,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7722,"qdb_cluster":[7722]},"id":2}]} 2024-04-25T00:20:41.641 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 162 2024-04-25T00:20:41.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.641+0000 7fad692a0640 1 -- 172.21.15.116:0/2764012773 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fad500678f0 msgr2=0x7fad50069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:41.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.641+0000 7fad692a0640 1 --2- 172.21.15.116:0/2764012773 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fad500678f0 0x7fad50069db0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fad540af000 tx=0x7fad540a7000 comp rx=0 tx=0).stop 2024-04-25T00:20:41.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.641+0000 7fad692a0640 1 -- 172.21.15.116:0/2764012773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad64150d20 msgr2=0x7fad641f2570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:41.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.641+0000 7fad692a0640 1 --2- 172.21.15.116:0/2764012773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad64150d20 0x7fad641f2570 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7fad5c00c7d0 tx=0x7fad5c00cc90 comp rx=0 tx=0).stop 2024-04-25T00:20:41.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.641+0000 7fad692a0640 1 -- 172.21.15.116:0/2764012773 shutdown_connections 2024-04-25T00:20:41.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.641+0000 7fad692a0640 1 --2- 172.21.15.116:0/2764012773 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fad500678f0 0x7fad50069db0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:41.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.641+0000 7fad692a0640 1 --2- 172.21.15.116:0/2764012773 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fad64150d20 0x7fad641f2570 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:41.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.641+0000 7fad692a0640 1 --2- 172.21.15.116:0/2764012773 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fad640c2f50 0x7fad64153c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:41.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.641+0000 7fad692a0640 1 --2- 172.21.15.116:0/2764012773 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fad640a94a0 0x7fad641440b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:41.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.641+0000 7fad692a0640 1 -- 172.21.15.116:0/2764012773 >> 172.21.15.116:0/2764012773 conn(0x7fad640c3b00 msgr2=0x7fad640aca90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:20:41.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.641+0000 7fad692a0640 1 -- 172.21.15.116:0/2764012773 shutdown_connections 2024-04-25T00:20:41.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:41.641+0000 7fad692a0640 1 -- 172.21.15.116:0/2764012773 wait complete. 2024-04-25T00:20:41.659 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:20:43.660 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:20:43.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.789+0000 7f4fc564e640 1 Processor -- start 2024-04-25T00:20:43.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.789+0000 7f4fc564e640 1 -- start start 2024-04-25T00:20:43.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.789+0000 7f4fc564e640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f4fc006d5f0 0x7f4fc0156c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:43.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.789+0000 7f4fc564e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4fc0155810 0x7f4fc01571c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:43.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.789+0000 7f4fc564e640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4fc00c9150 con 0x7f4fc0152d80 2024-04-25T00:20:43.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.789+0000 7f4fc564e640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f4fc00c9d60 con 0x7f4fc006d5f0 2024-04-25T00:20:43.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.789+0000 7f4fc564e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f4fc00c9430 con 0x7f4fc0155810 2024-04-25T00:20:43.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.789+0000 7f4fbf7fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4fc0155810 0x7f4fc01571c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:43.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.789+0000 7f4fb7fff640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f4fc006d5f0 0x7f4fc0156c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:43.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.789+0000 7f4fb7fff640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f4fc006d5f0 0x7f4fc0156c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:45158/0 (socket says 172.21.15.116:45158) 2024-04-25T00:20:43.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.789+0000 7f4fb7fff640 1 -- 172.21.15.116:0/4006213550 learned_addr learned my addr 172.21.15.116:0/4006213550 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:20:43.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.789+0000 7f4fbeffd640 1 --1- 172.21.15.116:0/4006213550 >> v1:172.21.15.174:6789/0 conn(0x7f4fc0152d80 0x7f4fc006bd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:59666/0 (socket says 172.21.15.116:59666) 2024-04-25T00:20:43.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.789+0000 7f4fb7fff640 1 -- 172.21.15.116:0/4006213550 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4fc0155810 msgr2=0x7f4fc01571c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:20:43.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.789+0000 7f4fb7fff640 1 --2- 172.21.15.116:0/4006213550 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4fc0155810 0x7f4fc01571c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:43.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.789+0000 7f4fb7fff640 1 -- 172.21.15.116:0/4006213550 >> v1:172.21.15.174:6789/0 conn(0x7f4fc0152d80 legacy=0x7f4fc006bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:43.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.789+0000 7f4fb7fff640 1 -- 172.21.15.116:0/4006213550 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fc00c9710 con 0x7f4fc006d5f0 2024-04-25T00:20:43.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.789+0000 7f4fbf7fe640 1 --2- 172.21.15.116:0/4006213550 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4fc0155810 0x7f4fc01571c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:20:43.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.793+0000 7f4fb7fff640 1 --2- 172.21.15.116:0/4006213550 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4fc006d5f0 0x7f4fc0156c80 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f4fa8009870 tx=0x7f4fa801c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9d687591f490a40 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:43.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.793+0000 7f4fbe7fc640 1 -- 172.21.15.116:0/4006213550 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa80b1070 con 0x7f4fc006d5f0 2024-04-25T00:20:43.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.793+0000 7f4fbe7fc640 1 -- 172.21.15.116:0/4006213550 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fa80ac630 con 0x7f4fc006d5f0 2024-04-25T00:20:43.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.793+0000 7f4fbe7fc640 1 -- 172.21.15.116:0/4006213550 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fa80a8050 con 0x7f4fc006d5f0 2024-04-25T00:20:43.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.793+0000 7f4fc564e640 1 -- 172.21.15.116:0/4006213550 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4fc006d5f0 msgr2=0x7f4fc0156c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:43.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.793+0000 7f4fc564e640 1 --2- 172.21.15.116:0/4006213550 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4fc006d5f0 0x7f4fc0156c80 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f4fa8009870 tx=0x7f4fa801c4a0 comp rx=0 tx=0).stop 2024-04-25T00:20:43.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.793+0000 7f4fc564e640 1 -- 172.21.15.116:0/4006213550 shutdown_connections 2024-04-25T00:20:43.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.793+0000 7f4fc564e640 1 --2- 172.21.15.116:0/4006213550 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4fc0155810 0x7f4fc01571c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:43.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.793+0000 7f4fc564e640 1 --2- 172.21.15.116:0/4006213550 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4fc006d5f0 0x7f4fc0156c80 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:43.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.793+0000 7f4fc564e640 1 -- 172.21.15.116:0/4006213550 >> 172.21.15.116:0/4006213550 conn(0x7f4fc00acdd0 msgr2=0x7f4fc00ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:20:43.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.793+0000 7f4fc564e640 1 -- 172.21.15.116:0/4006213550 shutdown_connections 2024-04-25T00:20:43.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.793+0000 7f4fc564e640 1 -- 172.21.15.116:0/4006213550 wait complete. 2024-04-25T00:20:43.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.797+0000 7f4fc564e640 1 Processor -- start 2024-04-25T00:20:43.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.797+0000 7f4fc564e640 1 -- start start 2024-04-25T00:20:43.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.797+0000 7f4fc564e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4fc006d5f0 0x7f4fc01f1db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:43.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.797+0000 7f4fc564e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4fc0152d80 0x7f4fc01eefe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:43.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.797+0000 7f4fc564e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fc0155810 0x7f4fc00abac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:43.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.797+0000 7f4fc564e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc00caf00 con 0x7f4fc0155810 2024-04-25T00:20:43.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.797+0000 7f4fc564e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc00cad80 con 0x7f4fc0152d80 2024-04-25T00:20:43.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.797+0000 7f4fc564e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f4fc01628d0 con 0x7f4fc006d5f0 2024-04-25T00:20:43.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.797+0000 7f4fbf7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fc0155810 0x7f4fc00abac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:43.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.797+0000 7f4fbf7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fc0155810 0x7f4fc00abac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52416/0 (socket says 172.21.15.116:52416) 2024-04-25T00:20:43.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.797+0000 7f4fbf7fe640 1 -- 172.21.15.116:0/4215600371 learned_addr learned my addr 172.21.15.116:0/4215600371 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:20:43.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.797+0000 7f4fbeffd640 1 --2- 172.21.15.116:0/4215600371 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4fc006d5f0 0x7f4fc01f1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:43.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.797+0000 7f4fb7fff640 1 --2- 172.21.15.116:0/4215600371 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4fc0152d80 0x7f4fc01eefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:43.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.797+0000 7f4fbf7fe640 1 -- 172.21.15.116:0/4215600371 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4fc006d5f0 msgr2=0x7f4fc01f1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:43.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.797+0000 7f4fbf7fe640 1 --2- 172.21.15.116:0/4215600371 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4fc006d5f0 0x7f4fc01f1db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:43.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.797+0000 7f4fbf7fe640 1 -- 172.21.15.116:0/4215600371 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4fc0152d80 msgr2=0x7f4fc01eefe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:43.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.797+0000 7f4fbf7fe640 1 --2- 172.21.15.116:0/4215600371 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4fc0152d80 0x7f4fc01eefe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:43.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.797+0000 7f4fbf7fe640 1 -- 172.21.15.116:0/4215600371 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fc01ee840 con 0x7f4fc0155810 2024-04-25T00:20:43.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.797+0000 7f4fbeffd640 1 --2- 172.21.15.116:0/4215600371 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4fc006d5f0 0x7f4fc01f1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:20:43.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.797+0000 7f4fbf7fe640 1 --2- 172.21.15.116:0/4215600371 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fc0155810 0x7f4fc00abac0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f4fb8009f00 tx=0x7f4fb800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:43.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.797+0000 7f4fb77fe640 1 -- 172.21.15.116:0/4215600371 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fb8018030 con 0x7f4fc0155810 2024-04-25T00:20:43.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.797+0000 7f4fb77fe640 1 -- 172.21.15.116:0/4215600371 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fb800ce50 con 0x7f4fc0155810 2024-04-25T00:20:43.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.797+0000 7f4fb77fe640 1 -- 172.21.15.116:0/4215600371 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fb8003bc0 con 0x7f4fc0155810 2024-04-25T00:20:43.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.797+0000 7f4fc564e640 1 -- 172.21.15.116:0/4215600371 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fc00ac720 con 0x7f4fc0155810 2024-04-25T00:20:43.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.797+0000 7f4fc564e640 1 -- 172.21.15.116:0/4215600371 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fc0151ec0 con 0x7f4fc0155810 2024-04-25T00:20:43.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.797+0000 7f4fc564e640 1 -- 172.21.15.116:0/4215600371 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fc0152090 con 0x7f4fc0155810 2024-04-25T00:20:43.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.801+0000 7f4fb77fe640 1 -- 172.21.15.116:0/4215600371 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4fb801b070 con 0x7f4fc0155810 2024-04-25T00:20:43.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.801+0000 7f4fb77fe640 1 --2- 172.21.15.116:0/4215600371 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4fa40678c0 0x7f4fa4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:43.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.801+0000 7f4fb77fe640 1 -- 172.21.15.116:0/4215600371 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 9246+0+0 (secure 0 0 0) 0x7f4fb80874b0 con 0x7f4fc0155810 2024-04-25T00:20:43.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.801+0000 7f4fbeffd640 1 --2- 172.21.15.116:0/4215600371 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4fa40678c0 0x7f4fa4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:43.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.801+0000 7f4fbeffd640 1 --2- 172.21.15.116:0/4215600371 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4fa40678c0 0x7f4fa4069d80 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f4fb0000f70 tx=0x7f4fb0007d70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:43.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:43.805+0000 7f4fb77fe640 1 -- 172.21.15.116:0/4215600371 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f4fc0152090 con 0x7f4fc0155810 2024-04-25T00:20:44.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:44.009+0000 7f4fc564e640 1 -- 172.21.15.116:0/4215600371 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4fc0151cb0 con 0x7f4fc0155810 2024-04-25T00:20:44.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:44.013+0000 7f4fb77fe640 1 -- 172.21.15.116:0/4215600371 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 162 v162) ==== 95+0+6500 (secure 0 0 0) 0x7f4fb8022070 con 0x7f4fc0155810 2024-04-25T00:20:44.013 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:20:44.013 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":162,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7809,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/980221971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":980221971},{"type":"v1","addr":"172.21.15.116:6835","nonce":980221971}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":162}],"filesystems":[{"mdsmap":{"epoch":156,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:19:33.982945+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6586":{"gid":6586,"name":"d","rank":0,"incarnation":153,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/991183935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":991183935},{"type":"v1","addr":"172.21.15.174:6837","nonce":991183935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6586,"qdb_cluster":[6586]},"id":1},{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:20:13.935378+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7722":{"gid":7722,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7722,"qdb_cluster":[7722]},"id":2}]} 2024-04-25T00:20:44.013 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 162 2024-04-25T00:20:44.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:44.013+0000 7f4fc564e640 1 -- 172.21.15.116:0/4215600371 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4fa40678c0 msgr2=0x7f4fa4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:44.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:44.013+0000 7f4fc564e640 1 --2- 172.21.15.116:0/4215600371 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4fa40678c0 0x7f4fa4069d80 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f4fb0000f70 tx=0x7f4fb0007d70 comp rx=0 tx=0).stop 2024-04-25T00:20:44.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:44.013+0000 7f4fc564e640 1 -- 172.21.15.116:0/4215600371 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fc0155810 msgr2=0x7f4fc00abac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:44.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:44.013+0000 7f4fc564e640 1 --2- 172.21.15.116:0/4215600371 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fc0155810 0x7f4fc00abac0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f4fb8009f00 tx=0x7f4fb800c490 comp rx=0 tx=0).stop 2024-04-25T00:20:44.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:44.013+0000 7f4fc564e640 1 -- 172.21.15.116:0/4215600371 shutdown_connections 2024-04-25T00:20:44.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:44.013+0000 7f4fc564e640 1 --2- 172.21.15.116:0/4215600371 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4fa40678c0 0x7f4fa4069d80 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:44.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:44.013+0000 7f4fc564e640 1 --2- 172.21.15.116:0/4215600371 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4fc0155810 0x7f4fc00abac0 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:44.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:44.013+0000 7f4fc564e640 1 --2- 172.21.15.116:0/4215600371 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4fc0152d80 0x7f4fc01eefe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:44.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:44.013+0000 7f4fc564e640 1 --2- 172.21.15.116:0/4215600371 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4fc006d5f0 0x7f4fc01f1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:44.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:44.013+0000 7f4fc564e640 1 -- 172.21.15.116:0/4215600371 >> 172.21.15.116:0/4215600371 conn(0x7f4fc00acdd0 msgr2=0x7f4fc0155d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:20:44.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:44.013+0000 7f4fc564e640 1 -- 172.21.15.116:0/4215600371 shutdown_connections 2024-04-25T00:20:44.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:44.013+0000 7f4fc564e640 1 -- 172.21.15.116:0/4215600371 wait complete. 2024-04-25T00:20:44.032 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:20:46.032 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:20:46.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.169+0000 7facb9d85640 1 Processor -- start 2024-04-25T00:20:46.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.169+0000 7facb9d85640 1 -- start start 2024-04-25T00:20:46.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.169+0000 7facb9d85640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7facb414c720 0x7facb41563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:46.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.169+0000 7facb9d85640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7facb40a94a0 0x7facb40c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:46.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.169+0000 7facb9d85640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7facb40c2f50 0x7facb4071680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:46.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.169+0000 7facb9d85640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7facb4153460 con 0x7facb414c720 2024-04-25T00:20:46.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.169+0000 7facb9d85640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7facb4071220 con 0x7facb40c2f50 2024-04-25T00:20:46.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.169+0000 7facb9d85640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7facb4152420 con 0x7facb40a94a0 2024-04-25T00:20:46.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.169+0000 7facb37fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7facb414c720 0x7facb41563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:46.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.169+0000 7facb37fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7facb414c720 0x7facb41563d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51864/0 (socket says 172.21.15.116:51864) 2024-04-25T00:20:46.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.169+0000 7facb37fe640 1 -- 172.21.15.116:0/1984378328 learned_addr learned my addr 172.21.15.116:0/1984378328 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:20:46.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.173+0000 7facb2ffd640 1 --2- 172.21.15.116:0/1984378328 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7facb40a94a0 0x7facb40c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:46.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.173+0000 7facb3fff640 1 --2- 172.21.15.116:0/1984378328 >> v2:172.21.15.174:3300/0 conn(0x7facb40c2f50 0x7facb4071680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:46.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.173+0000 7facb37fe640 1 -- 172.21.15.116:0/1984378328 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7facb40a94a0 msgr2=0x7facb40c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:20:46.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.173+0000 7facb37fe640 1 --2- 172.21.15.116:0/1984378328 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7facb40a94a0 0x7facb40c2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:46.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.173+0000 7facb37fe640 1 -- 172.21.15.116:0/1984378328 >> v2:172.21.15.174:3300/0 conn(0x7facb40c2f50 msgr2=0x7facb4071680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:20:46.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.173+0000 7facb37fe640 1 --2- 172.21.15.116:0/1984378328 >> v2:172.21.15.174:3300/0 conn(0x7facb40c2f50 0x7facb4071680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:46.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.173+0000 7facb37fe640 1 -- 172.21.15.116:0/1984378328 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facb414cb00 con 0x7facb414c720 2024-04-25T00:20:46.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.173+0000 7facb37fe640 1 --2- 172.21.15.116:0/1984378328 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facb414c720 0x7facb41563d0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7facb40c35f0 tx=0x7faca401eb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e114fc0a4d0db1e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:46.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.173+0000 7facb27fc640 1 -- 172.21.15.116:0/1984378328 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faca400b2d0 con 0x7facb414c720 2024-04-25T00:20:46.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.173+0000 7facb27fc640 1 -- 172.21.15.116:0/1984378328 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faca400b470 con 0x7facb414c720 2024-04-25T00:20:46.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.173+0000 7facb27fc640 1 -- 172.21.15.116:0/1984378328 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faca40a7ad0 con 0x7facb414c720 2024-04-25T00:20:46.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.173+0000 7facb9d85640 1 -- 172.21.15.116:0/1984378328 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facb414c720 msgr2=0x7facb41563d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:46.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.173+0000 7facb9d85640 1 --2- 172.21.15.116:0/1984378328 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facb414c720 0x7facb41563d0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7facb40c35f0 tx=0x7faca401eb30 comp rx=0 tx=0).stop 2024-04-25T00:20:46.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.173+0000 7facb9d85640 1 -- 172.21.15.116:0/1984378328 shutdown_connections 2024-04-25T00:20:46.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.173+0000 7facb9d85640 1 --2- 172.21.15.116:0/1984378328 >> v2:172.21.15.174:3300/0 conn(0x7facb40c2f50 0x7facb4071680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:46.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.173+0000 7facb9d85640 1 --2- 172.21.15.116:0/1984378328 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7facb40a94a0 0x7facb40c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:46.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.173+0000 7facb9d85640 1 --2- 172.21.15.116:0/1984378328 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facb414c720 0x7facb41563d0 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:46.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.173+0000 7facb9d85640 1 -- 172.21.15.116:0/1984378328 >> 172.21.15.116:0/1984378328 conn(0x7facb40c3b00 msgr2=0x7facb40c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:20:46.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.173+0000 7facb9d85640 1 -- 172.21.15.116:0/1984378328 shutdown_connections 2024-04-25T00:20:46.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.173+0000 7facb9d85640 1 -- 172.21.15.116:0/1984378328 wait complete. 2024-04-25T00:20:46.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.177+0000 7facb9d85640 1 Processor -- start 2024-04-25T00:20:46.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.177+0000 7facb9d85640 1 -- start start 2024-04-25T00:20:46.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.177+0000 7facb9d85640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facb40a94a0 0x7facb4148400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:46.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.177+0000 7facb9d85640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7facb40c2f50 0x7facb40c0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:46.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.177+0000 7facb9d85640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7facb414c720 0x7facb41f1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:46.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.177+0000 7facb9d85640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7facb4071fd0 con 0x7facb40a94a0 2024-04-25T00:20:46.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.177+0000 7facb9d85640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7facb4152f40 con 0x7facb40c2f50 2024-04-25T00:20:46.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.177+0000 7facb9d85640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7facb4072150 con 0x7facb414c720 2024-04-25T00:20:46.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.177+0000 7facb37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facb40a94a0 0x7facb4148400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:46.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.177+0000 7facb37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facb40a94a0 0x7facb4148400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51876/0 (socket says 172.21.15.116:51876) 2024-04-25T00:20:46.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.177+0000 7facb37fe640 1 -- 172.21.15.116:0/2806346266 learned_addr learned my addr 172.21.15.116:0/2806346266 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:20:46.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.177+0000 7facb3fff640 1 --2- 172.21.15.116:0/2806346266 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7facb414c720 0x7facb41f1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:46.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.177+0000 7facb2ffd640 1 --2- 172.21.15.116:0/2806346266 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7facb40c2f50 0x7facb40c0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:46.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.177+0000 7facb37fe640 1 -- 172.21.15.116:0/2806346266 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7facb414c720 msgr2=0x7facb41f1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:46.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.177+0000 7facb37fe640 1 --2- 172.21.15.116:0/2806346266 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7facb414c720 0x7facb41f1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:46.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.177+0000 7facb37fe640 1 -- 172.21.15.116:0/2806346266 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7facb40c2f50 msgr2=0x7facb40c0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:46.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.177+0000 7facb37fe640 1 --2- 172.21.15.116:0/2806346266 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7facb40c2f50 0x7facb40c0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:46.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.177+0000 7facb37fe640 1 -- 172.21.15.116:0/2806346266 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facb4157cf0 con 0x7facb40a94a0 2024-04-25T00:20:46.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.177+0000 7facb2ffd640 1 --2- 172.21.15.116:0/2806346266 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7facb40c2f50 0x7facb40c0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:20:46.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.177+0000 7facb37fe640 1 --2- 172.21.15.116:0/2806346266 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facb40a94a0 0x7facb4148400 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7faca401d1e0 tx=0x7faca401b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:46.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.177+0000 7fac9bfff640 1 -- 172.21.15.116:0/2806346266 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faca4009800 con 0x7facb40a94a0 2024-04-25T00:20:46.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.177+0000 7fac9bfff640 1 -- 172.21.15.116:0/2806346266 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faca40be070 con 0x7facb40a94a0 2024-04-25T00:20:46.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.177+0000 7facb9d85640 1 -- 172.21.15.116:0/2806346266 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facb415c930 con 0x7facb40a94a0 2024-04-25T00:20:46.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.177+0000 7fac9bfff640 1 -- 172.21.15.116:0/2806346266 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faca4009be0 con 0x7facb40a94a0 2024-04-25T00:20:46.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.177+0000 7facb9d85640 1 -- 172.21.15.116:0/2806346266 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facb41ee950 con 0x7facb40a94a0 2024-04-25T00:20:46.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.177+0000 7facb9d85640 1 -- 172.21.15.116:0/2806346266 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facb41ee460 con 0x7facb40a94a0 2024-04-25T00:20:46.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.177+0000 7fac9bfff640 1 -- 172.21.15.116:0/2806346266 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7faca40b6070 con 0x7facb40a94a0 2024-04-25T00:20:46.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.177+0000 7fac9bfff640 1 --2- 172.21.15.116:0/2806346266 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faca00678c0 0x7faca0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:46.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.181+0000 7fac9bfff640 1 -- 172.21.15.116:0/2806346266 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 9246+0+0 (secure 0 0 0) 0x7faca4121a70 con 0x7facb40a94a0 2024-04-25T00:20:46.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.181+0000 7facb2ffd640 1 --2- 172.21.15.116:0/2806346266 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faca00678c0 0x7faca0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:46.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.181+0000 7facb2ffd640 1 --2- 172.21.15.116:0/2806346266 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faca00678c0 0x7faca0069d80 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7facb40ab270 tx=0x7faca8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:46.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.185+0000 7fac9bfff640 1 -- 172.21.15.116:0/2806346266 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7faca40187e0 con 0x7facb40a94a0 2024-04-25T00:20:46.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.373+0000 7facb9d85640 1 -- 172.21.15.116:0/2806346266 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7facb4068350 con 0x7facb40a94a0 2024-04-25T00:20:46.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.373+0000 7fac9bfff640 1 -- 172.21.15.116:0/2806346266 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 162 v162) ==== 95+0+6500 (secure 0 0 0) 0x7faca40f18b0 con 0x7facb40a94a0 2024-04-25T00:20:46.376 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:20:46.376 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":162,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7809,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/980221971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":980221971},{"type":"v1","addr":"172.21.15.116:6835","nonce":980221971}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":162}],"filesystems":[{"mdsmap":{"epoch":156,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:19:33.982945+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6586":{"gid":6586,"name":"d","rank":0,"incarnation":153,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/991183935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":991183935},{"type":"v1","addr":"172.21.15.174:6837","nonce":991183935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6586,"qdb_cluster":[6586]},"id":1},{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:20:13.935378+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7722":{"gid":7722,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7722,"qdb_cluster":[7722]},"id":2}]} 2024-04-25T00:20:46.376 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 162 2024-04-25T00:20:46.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.377+0000 7facb9d85640 1 -- 172.21.15.116:0/2806346266 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faca00678c0 msgr2=0x7faca0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:46.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.377+0000 7facb9d85640 1 --2- 172.21.15.116:0/2806346266 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faca00678c0 0x7faca0069d80 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7facb40ab270 tx=0x7faca8008040 comp rx=0 tx=0).stop 2024-04-25T00:20:46.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.377+0000 7facb9d85640 1 -- 172.21.15.116:0/2806346266 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facb40a94a0 msgr2=0x7facb4148400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:46.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.377+0000 7facb9d85640 1 --2- 172.21.15.116:0/2806346266 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facb40a94a0 0x7facb4148400 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7faca401d1e0 tx=0x7faca401b8d0 comp rx=0 tx=0).stop 2024-04-25T00:20:46.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.377+0000 7facb9d85640 1 -- 172.21.15.116:0/2806346266 shutdown_connections 2024-04-25T00:20:46.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.377+0000 7facb9d85640 1 --2- 172.21.15.116:0/2806346266 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faca00678c0 0x7faca0069d80 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:46.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.377+0000 7facb9d85640 1 --2- 172.21.15.116:0/2806346266 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7facb414c720 0x7facb41f1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:46.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.377+0000 7facb9d85640 1 --2- 172.21.15.116:0/2806346266 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7facb40c2f50 0x7facb40c0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:46.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.377+0000 7facb9d85640 1 --2- 172.21.15.116:0/2806346266 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7facb40a94a0 0x7facb4148400 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:46.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.377+0000 7facb9d85640 1 -- 172.21.15.116:0/2806346266 >> 172.21.15.116:0/2806346266 conn(0x7facb40c3b00 msgr2=0x7facb414b910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:20:46.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.377+0000 7facb9d85640 1 -- 172.21.15.116:0/2806346266 shutdown_connections 2024-04-25T00:20:46.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:46.377+0000 7facb9d85640 1 -- 172.21.15.116:0/2806346266 wait complete. 2024-04-25T00:20:46.393 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:20:48.394 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:20:48.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.520+0000 7ff4e6c41640 1 Processor -- start 2024-04-25T00:20:48.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.520+0000 7ff4e6c41640 1 -- start start 2024-04-25T00:20:48.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.524+0000 7ff4e6c41640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff4e014c720 0x7ff4e01563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:48.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.524+0000 7ff4e6c41640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff4e0071680 0x7ff4e0071ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:48.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.524+0000 7ff4e6c41640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4e0071220 con 0x7ff4e00a94a0 2024-04-25T00:20:48.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.524+0000 7ff4e6c41640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7ff4e0153460 con 0x7ff4e0071680 2024-04-25T00:20:48.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.524+0000 7ff4e6c41640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff4e0152420 con 0x7ff4e014c720 2024-04-25T00:20:48.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.524+0000 7ff4e51de640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff4e0071680 0x7ff4e0071ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:48.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.524+0000 7ff4e51de640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff4e0071680 0x7ff4e0071ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51884/0 (socket says 172.21.15.116:51884) 2024-04-25T00:20:48.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.524+0000 7ff4e51de640 1 -- 172.21.15.116:0/1698063957 learned_addr learned my addr 172.21.15.116:0/1698063957 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:20:48.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.524+0000 7ff4e49dd640 1 --2- 172.21.15.116:0/1698063957 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff4e014c720 0x7ff4e01563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:48.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.524+0000 7ff4e51de640 1 -- 172.21.15.116:0/1698063957 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff4e014c720 msgr2=0x7ff4e01563d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:20:48.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.524+0000 7ff4e51de640 1 --2- 172.21.15.116:0/1698063957 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff4e014c720 0x7ff4e01563d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:48.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.524+0000 7ff4e51de640 1 -- 172.21.15.116:0/1698063957 >> v1:172.21.15.174:6789/0 conn(0x7ff4e00a94a0 legacy=0x7ff4e00c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:20:48.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.524+0000 7ff4e51de640 1 -- 172.21.15.116:0/1698063957 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4e014cb00 con 0x7ff4e0071680 2024-04-25T00:20:48.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.524+0000 7ff4e51de640 1 --2- 172.21.15.116:0/1698063957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4e0071680 0x7ff4e0071ad0 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7ff4d800d150 tx=0x7ff4d801eb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=389eafb811cab090 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:48.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.524+0000 7ff4df7fe640 1 -- 172.21.15.116:0/1698063957 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4d800b2d0 con 0x7ff4e0071680 2024-04-25T00:20:48.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.524+0000 7ff4df7fe640 1 -- 172.21.15.116:0/1698063957 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4d800b470 con 0x7ff4e0071680 2024-04-25T00:20:48.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.524+0000 7ff4df7fe640 1 -- 172.21.15.116:0/1698063957 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4d80a7ae0 con 0x7ff4e0071680 2024-04-25T00:20:48.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.528+0000 7ff4e6c41640 1 -- 172.21.15.116:0/1698063957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4e0071680 msgr2=0x7ff4e0071ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:48.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.528+0000 7ff4e6c41640 1 --2- 172.21.15.116:0/1698063957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4e0071680 0x7ff4e0071ad0 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7ff4d800d150 tx=0x7ff4d801eb30 comp rx=0 tx=0).stop 2024-04-25T00:20:48.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.528+0000 7ff4e6c41640 1 -- 172.21.15.116:0/1698063957 shutdown_connections 2024-04-25T00:20:48.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.528+0000 7ff4e6c41640 1 --2- 172.21.15.116:0/1698063957 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4e0071680 0x7ff4e0071ad0 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:48.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.528+0000 7ff4e6c41640 1 --2- 172.21.15.116:0/1698063957 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff4e014c720 0x7ff4e01563d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:48.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.528+0000 7ff4e6c41640 1 -- 172.21.15.116:0/1698063957 >> 172.21.15.116:0/1698063957 conn(0x7ff4e00c3b00 msgr2=0x7ff4e00c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:20:48.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.528+0000 7ff4e6c41640 1 -- 172.21.15.116:0/1698063957 shutdown_connections 2024-04-25T00:20:48.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.528+0000 7ff4e6c41640 1 -- 172.21.15.116:0/1698063957 wait complete. 2024-04-25T00:20:48.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.528+0000 7ff4e6c41640 1 Processor -- start 2024-04-25T00:20:48.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.528+0000 7ff4e6c41640 1 -- start start 2024-04-25T00:20:48.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.528+0000 7ff4e6c41640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4e0071680 0x7ff4e00c0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:48.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.528+0000 7ff4e6c41640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff4e00a94a0 0x7ff4e0150290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:48.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.528+0000 7ff4e6c41640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff4e014c720 0x7ff4e01507d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:48.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.528+0000 7ff4e6c41640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff4e00c3120 con 0x7ff4e0071680 2024-04-25T00:20:48.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.528+0000 7ff4e6c41640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff4e0152f40 con 0x7ff4e014c720 2024-04-25T00:20:48.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.528+0000 7ff4e6c41640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff4e00c32a0 con 0x7ff4e00a94a0 2024-04-25T00:20:48.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.528+0000 7ff4e49dd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4e0071680 0x7ff4e00c0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:48.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.528+0000 7ff4e49dd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4e0071680 0x7ff4e00c0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51888/0 (socket says 172.21.15.116:51888) 2024-04-25T00:20:48.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.528+0000 7ff4e49dd640 1 -- 172.21.15.116:0/2694087030 learned_addr learned my addr 172.21.15.116:0/2694087030 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:20:48.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.528+0000 7ff4dffff640 1 --2- 172.21.15.116:0/2694087030 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff4e00a94a0 0x7ff4e0150290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:48.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.528+0000 7ff4e51de640 1 --2- 172.21.15.116:0/2694087030 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff4e014c720 0x7ff4e01507d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:48.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.528+0000 7ff4e49dd640 1 -- 172.21.15.116:0/2694087030 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff4e00a94a0 msgr2=0x7ff4e0150290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:48.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.528+0000 7ff4e49dd640 1 --2- 172.21.15.116:0/2694087030 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff4e00a94a0 0x7ff4e0150290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:48.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.528+0000 7ff4e49dd640 1 -- 172.21.15.116:0/2694087030 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff4e014c720 msgr2=0x7ff4e01507d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:48.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.528+0000 7ff4e49dd640 1 --2- 172.21.15.116:0/2694087030 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff4e014c720 0x7ff4e01507d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:48.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.528+0000 7ff4e49dd640 1 -- 172.21.15.116:0/2694087030 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4e0150d10 con 0x7ff4e0071680 2024-04-25T00:20:48.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.528+0000 7ff4e49dd640 1 --2- 172.21.15.116:0/2694087030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4e0071680 0x7ff4e00c0690 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7ff4d000d5f0 tx=0x7ff4d000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:48.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.528+0000 7ff4dd7fa640 1 -- 172.21.15.116:0/2694087030 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4d00125f0 con 0x7ff4e0071680 2024-04-25T00:20:48.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.528+0000 7ff4dd7fa640 1 -- 172.21.15.116:0/2694087030 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4d0019070 con 0x7ff4e0071680 2024-04-25T00:20:48.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.528+0000 7ff4e6c41640 1 -- 172.21.15.116:0/2694087030 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4e014ba30 con 0x7ff4e0071680 2024-04-25T00:20:48.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.528+0000 7ff4dd7fa640 1 -- 172.21.15.116:0/2694087030 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4d00129b0 con 0x7ff4e0071680 2024-04-25T00:20:48.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.528+0000 7ff4e6c41640 1 -- 172.21.15.116:0/2694087030 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4e01edff0 con 0x7ff4e0071680 2024-04-25T00:20:48.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.528+0000 7ff4e6c41640 1 -- 172.21.15.116:0/2694087030 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4e0068350 con 0x7ff4e0071680 2024-04-25T00:20:48.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.532+0000 7ff4dd7fa640 1 -- 172.21.15.116:0/2694087030 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff4d0012b50 con 0x7ff4e0071680 2024-04-25T00:20:48.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.532+0000 7ff4dd7fa640 1 --2- 172.21.15.116:0/2694087030 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff4cc0678f0 0x7ff4cc069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:48.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.532+0000 7ff4dd7fa640 1 -- 172.21.15.116:0/2694087030 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(58..58 src has 1..58) ==== 9246+0+0 (secure 0 0 0) 0x7ff4d00878b0 con 0x7ff4e0071680 2024-04-25T00:20:48.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.532+0000 7ff4dffff640 1 --2- 172.21.15.116:0/2694087030 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff4cc0678f0 0x7ff4cc069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:48.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.532+0000 7ff4dffff640 1 --2- 172.21.15.116:0/2694087030 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff4cc0678f0 0x7ff4cc069db0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7ff4e015cd70 tx=0x7ff4d4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:48.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.536+0000 7ff4dd7fa640 1 -- 172.21.15.116:0/2694087030 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ff4d00576e0 con 0x7ff4e0071680 2024-04-25T00:20:48.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.732+0000 7ff4e6c41640 1 -- 172.21.15.116:0/2694087030 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff4e00c3640 con 0x7ff4e0071680 2024-04-25T00:20:48.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.732+0000 7ff4dd7fa640 1 -- 172.21.15.116:0/2694087030 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 162 v162) ==== 95+0+6500 (secure 0 0 0) 0x7ff4d0056fe0 con 0x7ff4e0071680 2024-04-25T00:20:48.734 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:20:48.734 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":162,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7809,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/980221971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":980221971},{"type":"v1","addr":"172.21.15.116:6835","nonce":980221971}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":162}],"filesystems":[{"mdsmap":{"epoch":156,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:19:33.982945+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6586":{"gid":6586,"name":"d","rank":0,"incarnation":153,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/991183935","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":991183935},{"type":"v1","addr":"172.21.15.174:6837","nonce":991183935}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6586,"qdb_cluster":[6586]},"id":1},{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:20:13.935378+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7722":{"gid":7722,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7722,"qdb_cluster":[7722]},"id":2}]} 2024-04-25T00:20:48.735 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 162 2024-04-25T00:20:48.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.732+0000 7ff4e6c41640 1 -- 172.21.15.116:0/2694087030 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff4cc0678f0 msgr2=0x7ff4cc069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:48.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.732+0000 7ff4e6c41640 1 --2- 172.21.15.116:0/2694087030 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff4cc0678f0 0x7ff4cc069db0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7ff4e015cd70 tx=0x7ff4d4009040 comp rx=0 tx=0).stop 2024-04-25T00:20:48.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.732+0000 7ff4e6c41640 1 -- 172.21.15.116:0/2694087030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4e0071680 msgr2=0x7ff4e00c0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:48.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.732+0000 7ff4e6c41640 1 --2- 172.21.15.116:0/2694087030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4e0071680 0x7ff4e00c0690 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7ff4d000d5f0 tx=0x7ff4d000dab0 comp rx=0 tx=0).stop 2024-04-25T00:20:48.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.732+0000 7ff4e6c41640 1 -- 172.21.15.116:0/2694087030 shutdown_connections 2024-04-25T00:20:48.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.732+0000 7ff4e6c41640 1 --2- 172.21.15.116:0/2694087030 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff4cc0678f0 0x7ff4cc069db0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:48.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.732+0000 7ff4e6c41640 1 --2- 172.21.15.116:0/2694087030 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff4e014c720 0x7ff4e01507d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:48.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.732+0000 7ff4e6c41640 1 --2- 172.21.15.116:0/2694087030 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff4e00a94a0 0x7ff4e0150290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:48.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.732+0000 7ff4e6c41640 1 --2- 172.21.15.116:0/2694087030 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4e0071680 0x7ff4e00c0690 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:48.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.732+0000 7ff4e6c41640 1 -- 172.21.15.116:0/2694087030 >> 172.21.15.116:0/2694087030 conn(0x7ff4e00c3b00 msgr2=0x7ff4e0072610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:20:48.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.732+0000 7ff4e6c41640 1 -- 172.21.15.116:0/2694087030 shutdown_connections 2024-04-25T00:20:48.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:48.732+0000 7ff4e6c41640 1 -- 172.21.15.116:0/2694087030 wait complete. 2024-04-25T00:20:48.753 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:20:50.753 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:20:50.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.864+0000 7fc7d7193640 1 Processor -- start 2024-04-25T00:20:50.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.864+0000 7fc7d7193640 1 -- start start 2024-04-25T00:20:50.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.868+0000 7fc7d7193640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc7d0152d60 0x7fc7d00ab250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:50.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.868+0000 7fc7d7193640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fc7d00ab790 0x7fc7d00c8450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:50.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.868+0000 7fc7d7193640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7d0156760 con 0x7fc7d0153140 2024-04-25T00:20:50.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.868+0000 7fc7d7193640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fc7d0155be0 con 0x7fc7d00ab790 2024-04-25T00:20:50.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.868+0000 7fc7d7193640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fc7d0155ec0 con 0x7fc7d0152d60 2024-04-25T00:20:50.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.868+0000 7fc7d4f2f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc7d0152d60 0x7fc7d00ab250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:50.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.868+0000 7fc7d4f2f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc7d0152d60 0x7fc7d00ab250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:43946/0 (socket says 172.21.15.116:43946) 2024-04-25T00:20:50.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.868+0000 7fc7d4f2f640 1 -- 172.21.15.116:0/1392525762 learned_addr learned my addr 172.21.15.116:0/1392525762 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:20:50.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.868+0000 7fc7cffff640 1 --2- 172.21.15.116:0/1392525762 >> v2:172.21.15.174:3300/0 conn(0x7fc7d00ab790 0x7fc7d00c8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:50.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.868+0000 7fc7d4f2f640 1 -- 172.21.15.116:0/1392525762 >> v2:172.21.15.174:3300/0 conn(0x7fc7d00ab790 msgr2=0x7fc7d00c8450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:20:50.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.868+0000 7fc7d4f2f640 1 --2- 172.21.15.116:0/1392525762 >> v2:172.21.15.174:3300/0 conn(0x7fc7d00ab790 0x7fc7d00c8450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:50.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.868+0000 7fc7cf7fe640 1 -- 172.21.15.116:0/1392525762 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1983916348 0 0) 0x7fc7d0156760 con 0x7fc7d0153140 2024-04-25T00:20:50.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.868+0000 7fc7d4f2f640 1 -- 172.21.15.116:0/1392525762 >> v1:172.21.15.174:6789/0 conn(0x7fc7d0153140 legacy=0x7fc7d00c3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:50.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.868+0000 7fc7d4f2f640 1 -- 172.21.15.116:0/1392525762 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7d01561a0 con 0x7fc7d0152d60 2024-04-25T00:20:50.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.868+0000 7fc7d4f2f640 1 --2- 172.21.15.116:0/1392525762 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc7d0152d60 0x7fc7d00ab250 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc7c0009870 tx=0x7fc7c001c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70148cf56a141aad server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:50.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.872+0000 7fc7cf7fe640 1 -- 172.21.15.116:0/1392525762 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c00b1070 con 0x7fc7d0152d60 2024-04-25T00:20:50.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.872+0000 7fc7cf7fe640 1 -- 172.21.15.116:0/1392525762 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7c00a8050 con 0x7fc7d0152d60 2024-04-25T00:20:50.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.872+0000 7fc7cf7fe640 1 -- 172.21.15.116:0/1392525762 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c00ac860 con 0x7fc7d0152d60 2024-04-25T00:20:50.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.872+0000 7fc7d7193640 1 -- 172.21.15.116:0/1392525762 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc7d0152d60 msgr2=0x7fc7d00ab250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:50.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.872+0000 7fc7d7193640 1 --2- 172.21.15.116:0/1392525762 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc7d0152d60 0x7fc7d00ab250 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc7c0009870 tx=0x7fc7c001c4d0 comp rx=0 tx=0).stop 2024-04-25T00:20:50.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.872+0000 7fc7d7193640 1 -- 172.21.15.116:0/1392525762 shutdown_connections 2024-04-25T00:20:50.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.872+0000 7fc7d7193640 1 --2- 172.21.15.116:0/1392525762 >> v2:172.21.15.174:3300/0 conn(0x7fc7d00ab790 0x7fc7d00c8450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:50.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.872+0000 7fc7d7193640 1 --2- 172.21.15.116:0/1392525762 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc7d0152d60 0x7fc7d00ab250 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:50.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.872+0000 7fc7d7193640 1 -- 172.21.15.116:0/1392525762 >> 172.21.15.116:0/1392525762 conn(0x7fc7d006f0c0 msgr2=0x7fc7d006f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:20:50.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.872+0000 7fc7d7193640 1 -- 172.21.15.116:0/1392525762 shutdown_connections 2024-04-25T00:20:50.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.872+0000 7fc7d7193640 1 -- 172.21.15.116:0/1392525762 wait complete. 2024-04-25T00:20:50.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.872+0000 7fc7d7193640 1 Processor -- start 2024-04-25T00:20:50.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.872+0000 7fc7d7193640 1 -- start start 2024-04-25T00:20:50.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.872+0000 7fc7d7193640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7d00ab790 0x7fc7d0145340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:50.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.872+0000 7fc7d7193640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc7d0152d60 0x7fc7d006cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:50.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.872+0000 7fc7d7193640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7d0153140 0x7fc7d0143140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:50.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.872+0000 7fc7d7193640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d01629a0 con 0x7fc7d0153140 2024-04-25T00:20:50.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.872+0000 7fc7d7193640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc7d00c49c0 con 0x7fc7d00ab790 2024-04-25T00:20:50.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.872+0000 7fc7d7193640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fc7d0162b20 con 0x7fc7d0152d60 2024-04-25T00:20:50.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.872+0000 7fc7d5730640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7d0153140 0x7fc7d0143140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:50.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.872+0000 7fc7d5730640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7d0153140 0x7fc7d0143140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51892/0 (socket says 172.21.15.116:51892) 2024-04-25T00:20:50.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.872+0000 7fc7d5730640 1 -- 172.21.15.116:0/607453431 learned_addr learned my addr 172.21.15.116:0/607453431 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:20:50.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.872+0000 7fc7d4f2f640 1 --2- 172.21.15.116:0/607453431 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7d00ab790 0x7fc7d0145340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:50.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.872+0000 7fc7cffff640 1 --2- 172.21.15.116:0/607453431 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc7d0152d60 0x7fc7d006cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:50.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.872+0000 7fc7d5730640 1 -- 172.21.15.116:0/607453431 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc7d0152d60 msgr2=0x7fc7d006cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:50.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.872+0000 7fc7d5730640 1 --2- 172.21.15.116:0/607453431 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc7d0152d60 0x7fc7d006cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:50.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.872+0000 7fc7d5730640 1 -- 172.21.15.116:0/607453431 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7d00ab790 msgr2=0x7fc7d0145340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:50.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.872+0000 7fc7d5730640 1 --2- 172.21.15.116:0/607453431 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7d00ab790 0x7fc7d0145340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:50.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.872+0000 7fc7d5730640 1 -- 172.21.15.116:0/607453431 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7d00ad810 con 0x7fc7d0153140 2024-04-25T00:20:50.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.872+0000 7fc7d5730640 1 --2- 172.21.15.116:0/607453431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7d0153140 0x7fc7d0143140 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7fc7c800c830 tx=0x7fc7c800ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:50.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.872+0000 7fc7cd7fa640 1 -- 172.21.15.116:0/607453431 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c8003db0 con 0x7fc7d0153140 2024-04-25T00:20:50.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.876+0000 7fc7cd7fa640 1 -- 172.21.15.116:0/607453431 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7c800ce20 con 0x7fc7d0153140 2024-04-25T00:20:50.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.876+0000 7fc7d7193640 1 -- 172.21.15.116:0/607453431 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7d00adad0 con 0x7fc7d0153140 2024-04-25T00:20:50.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.876+0000 7fc7cd7fa640 1 -- 172.21.15.116:0/607453431 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7c80052b0 con 0x7fc7d0153140 2024-04-25T00:20:50.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.876+0000 7fc7d7193640 1 -- 172.21.15.116:0/607453431 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7d00ae090 con 0x7fc7d0153140 2024-04-25T00:20:50.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.876+0000 7fc7b6ffd640 1 -- 172.21.15.116:0/607453431 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7a80052a0 con 0x7fc7d0153140 2024-04-25T00:20:50.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.876+0000 7fc7cd7fa640 1 -- 172.21.15.116:0/607453431 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc7c8005450 con 0x7fc7d0153140 2024-04-25T00:20:50.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.876+0000 7fc7cd7fa640 1 --2- 172.21.15.116:0/607453431 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc7bc0678c0 0x7fc7bc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:50.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.876+0000 7fc7cd7fa640 1 -- 172.21.15.116:0/607453431 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 9332+0+0 (secure 0 0 0) 0x7fc7c8086840 con 0x7fc7d0153140 2024-04-25T00:20:50.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.876+0000 7fc7d4f2f640 1 --2- 172.21.15.116:0/607453431 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc7bc0678c0 0x7fc7bc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:50.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.876+0000 7fc7d4f2f640 1 --2- 172.21.15.116:0/607453431 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc7bc0678c0 0x7fc7bc069d80 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fc7c00ae000 tx=0x7fc7c00a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:50.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:50.880+0000 7fc7cd7fa640 1 -- 172.21.15.116:0/607453431 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fc7c8055f70 con 0x7fc7d0153140 2024-04-25T00:20:51.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.076+0000 7fc7b6ffd640 1 -- 172.21.15.116:0/607453431 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc7a8005590 con 0x7fc7d0153140 2024-04-25T00:20:51.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.076+0000 7fc7cd7fa640 1 -- 172.21.15.116:0/607453431 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 163 v163) ==== 95+0+5667 (secure 0 0 0) 0x7fc7c805a270 con 0x7fc7d0153140 2024-04-25T00:20:51.080 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:20:51.080 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":163,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":163,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:20:50.654702+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7809},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7809":{"gid":7809,"name":"c","rank":0,"incarnation":163,"state":"up:replay","state_seq":1,"addr":"172.21.15.116:6835/980221971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":980221971},{"type":"v1","addr":"172.21.15.116:6835","nonce":980221971}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:20:13.935378+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7722":{"gid":7722,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7722,"qdb_cluster":[7722]},"id":2}]} 2024-04-25T00:20:51.080 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 163 2024-04-25T00:20:51.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.080+0000 7fc7b6ffd640 1 -- 172.21.15.116:0/607453431 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc7bc0678c0 msgr2=0x7fc7bc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:51.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.080+0000 7fc7b6ffd640 1 --2- 172.21.15.116:0/607453431 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc7bc0678c0 0x7fc7bc069d80 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fc7c00ae000 tx=0x7fc7c00a7000 comp rx=0 tx=0).stop 2024-04-25T00:20:51.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.080+0000 7fc7b6ffd640 1 -- 172.21.15.116:0/607453431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7d0153140 msgr2=0x7fc7d0143140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:51.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.080+0000 7fc7b6ffd640 1 --2- 172.21.15.116:0/607453431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7d0153140 0x7fc7d0143140 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7fc7c800c830 tx=0x7fc7c800ccf0 comp rx=0 tx=0).stop 2024-04-25T00:20:51.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.080+0000 7fc7b6ffd640 1 -- 172.21.15.116:0/607453431 shutdown_connections 2024-04-25T00:20:51.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.080+0000 7fc7b6ffd640 1 --2- 172.21.15.116:0/607453431 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc7bc0678c0 0x7fc7bc069d80 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:51.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.080+0000 7fc7b6ffd640 1 --2- 172.21.15.116:0/607453431 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc7d0153140 0x7fc7d0143140 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:51.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.080+0000 7fc7b6ffd640 1 --2- 172.21.15.116:0/607453431 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc7d0152d60 0x7fc7d006cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:51.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.080+0000 7fc7b6ffd640 1 --2- 172.21.15.116:0/607453431 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc7d00ab790 0x7fc7d0145340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:51.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.080+0000 7fc7b6ffd640 1 -- 172.21.15.116:0/607453431 >> 172.21.15.116:0/607453431 conn(0x7fc7d006f0c0 msgr2=0x7fc7d014f680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:20:51.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.080+0000 7fc7b6ffd640 1 -- 172.21.15.116:0/607453431 shutdown_connections 2024-04-25T00:20:51.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.080+0000 7fc7b6ffd640 1 -- 172.21.15.116:0/607453431 wait complete. 2024-04-25T00:20:51.097 INFO:tasks.mds_thrash.fs.[a]:mds.d down, removed from mdsmap 2024-04-25T00:20:51.098 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-04-25T00:20:51.098 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:20:51.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.216+0000 7fb049a23640 1 Processor -- start 2024-04-25T00:20:51.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.216+0000 7fb049a23640 1 -- start start 2024-04-25T00:20:51.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.216+0000 7fb049a23640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fb044150d20 0x7fb0440c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:51.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.216+0000 7fb049a23640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb0440c2f50 0x7fb0441538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:51.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.216+0000 7fb049a23640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0440c3910 con 0x7fb0440a94a0 2024-04-25T00:20:51.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.216+0000 7fb049a23640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fb044154530 con 0x7fb0440c2f50 2024-04-25T00:20:51.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.216+0000 7fb049a23640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fb0440c16e0 con 0x7fb044150d20 2024-04-25T00:20:51.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.216+0000 7fb03bfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb0440c2f50 0x7fb0441538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:51.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.216+0000 7fb03bfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb0440c2f50 0x7fb0441538e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51894/0 (socket says 172.21.15.116:51894) 2024-04-25T00:20:51.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.216+0000 7fb03bfff640 1 -- 172.21.15.116:0/2814349464 learned_addr learned my addr 172.21.15.116:0/2814349464 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:20:51.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.216+0000 7fb042ffd640 1 --2- 172.21.15.116:0/2814349464 >> v2:172.21.15.174:3300/0 conn(0x7fb044150d20 0x7fb0440c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:51.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.216+0000 7fb03bfff640 1 -- 172.21.15.116:0/2814349464 >> v2:172.21.15.174:3300/0 conn(0x7fb044150d20 msgr2=0x7fb0440c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:20:51.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.216+0000 7fb03bfff640 1 --2- 172.21.15.116:0/2814349464 >> v2:172.21.15.174:3300/0 conn(0x7fb044150d20 0x7fb0440c2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:51.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.216+0000 7fb03bfff640 1 -- 172.21.15.116:0/2814349464 >> v1:172.21.15.174:6789/0 conn(0x7fb0440a94a0 legacy=0x7fb044153e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:20:51.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.216+0000 7fb03bfff640 1 -- 172.21.15.116:0/2814349464 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0440c1400 con 0x7fb0440c2f50 2024-04-25T00:20:51.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.220+0000 7fb03bfff640 1 --2- 172.21.15.116:0/2814349464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0440c2f50 0x7fb0441538e0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7fb044073550 tx=0x7fb02c01c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f933673ad6b2d9cf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:51.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.220+0000 7fb0427fc640 1 -- 172.21.15.116:0/2814349464 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb02c0ab5f0 con 0x7fb0440c2f50 2024-04-25T00:20:51.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.220+0000 7fb0427fc640 1 -- 172.21.15.116:0/2814349464 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb02c0b0070 con 0x7fb0440c2f50 2024-04-25T00:20:51.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.220+0000 7fb0427fc640 1 -- 172.21.15.116:0/2814349464 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb02c0ab9b0 con 0x7fb0440c2f50 2024-04-25T00:20:51.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.220+0000 7fb049a23640 1 -- 172.21.15.116:0/2814349464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0440c2f50 msgr2=0x7fb0441538e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:51.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.220+0000 7fb049a23640 1 --2- 172.21.15.116:0/2814349464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0440c2f50 0x7fb0441538e0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7fb044073550 tx=0x7fb02c01c4e0 comp rx=0 tx=0).stop 2024-04-25T00:20:51.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.220+0000 7fb049a23640 1 -- 172.21.15.116:0/2814349464 shutdown_connections 2024-04-25T00:20:51.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.220+0000 7fb049a23640 1 --2- 172.21.15.116:0/2814349464 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0440c2f50 0x7fb0441538e0 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:51.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.220+0000 7fb049a23640 1 --2- 172.21.15.116:0/2814349464 >> v2:172.21.15.174:3300/0 conn(0x7fb044150d20 0x7fb0440c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:51.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.220+0000 7fb049a23640 1 -- 172.21.15.116:0/2814349464 >> 172.21.15.116:0/2814349464 conn(0x7fb0440c3b00 msgr2=0x7fb0440c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:20:51.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.220+0000 7fb049a23640 1 -- 172.21.15.116:0/2814349464 shutdown_connections 2024-04-25T00:20:51.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.220+0000 7fb049a23640 1 -- 172.21.15.116:0/2814349464 wait complete. 2024-04-25T00:20:51.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.220+0000 7fb049a23640 1 Processor -- start 2024-04-25T00:20:51.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.220+0000 7fb049a23640 1 -- start start 2024-04-25T00:20:51.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.220+0000 7fb049a23640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb0440a94a0 0x7fb04414f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:51.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.220+0000 7fb049a23640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0440c2f50 0x7fb04414b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:51.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.220+0000 7fb049a23640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb044150d20 0x7fb04414bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:51.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.220+0000 7fb049a23640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb0441608b0 con 0x7fb0440c2f50 2024-04-25T00:20:51.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.220+0000 7fb049a23640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb044160730 con 0x7fb044150d20 2024-04-25T00:20:51.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.220+0000 7fb049a23640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fb044160a30 con 0x7fb0440a94a0 2024-04-25T00:20:51.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.220+0000 7fb03bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0440c2f50 0x7fb04414b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:51.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.220+0000 7fb03bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0440c2f50 0x7fb04414b5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51904/0 (socket says 172.21.15.116:51904) 2024-04-25T00:20:51.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.220+0000 7fb03bfff640 1 -- 172.21.15.116:0/1009954024 learned_addr learned my addr 172.21.15.116:0/1009954024 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:20:51.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.220+0000 7fb042ffd640 1 --2- 172.21.15.116:0/1009954024 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb0440a94a0 0x7fb04414f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:51.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.220+0000 7fb0437fe640 1 --2- 172.21.15.116:0/1009954024 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb044150d20 0x7fb04414bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:51.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.220+0000 7fb03bfff640 1 -- 172.21.15.116:0/1009954024 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb0440a94a0 msgr2=0x7fb04414f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:51.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.220+0000 7fb03bfff640 1 --2- 172.21.15.116:0/1009954024 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb0440a94a0 0x7fb04414f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:51.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.220+0000 7fb03bfff640 1 -- 172.21.15.116:0/1009954024 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb044150d20 msgr2=0x7fb04414bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:51.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.220+0000 7fb03bfff640 1 --2- 172.21.15.116:0/1009954024 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb044150d20 0x7fb04414bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:51.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.220+0000 7fb03bfff640 1 -- 172.21.15.116:0/1009954024 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0441442f0 con 0x7fb0440c2f50 2024-04-25T00:20:51.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.220+0000 7fb03bfff640 1 --2- 172.21.15.116:0/1009954024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0440c2f50 0x7fb04414b5e0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7fb02c01c9c0 tx=0x7fb02c01ad20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:51.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.220+0000 7fb040ff9640 1 -- 172.21.15.116:0/1009954024 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb02c0be5f0 con 0x7fb0440c2f50 2024-04-25T00:20:51.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.220+0000 7fb040ff9640 1 -- 172.21.15.116:0/1009954024 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb02c0be790 con 0x7fb0440c2f50 2024-04-25T00:20:51.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.220+0000 7fb049a23640 1 -- 172.21.15.116:0/1009954024 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb044143a50 con 0x7fb0440c2f50 2024-04-25T00:20:51.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.220+0000 7fb040ff9640 1 -- 172.21.15.116:0/1009954024 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb02c0beb80 con 0x7fb0440c2f50 2024-04-25T00:20:51.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.224+0000 7fb049a23640 1 -- 172.21.15.116:0/1009954024 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb044144010 con 0x7fb0440c2f50 2024-04-25T00:20:51.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.224+0000 7fb049a23640 1 -- 172.21.15.116:0/1009954024 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb044068350 con 0x7fb0440c2f50 2024-04-25T00:20:51.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.224+0000 7fb040ff9640 1 -- 172.21.15.116:0/1009954024 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb02c0b43f0 con 0x7fb0440c2f50 2024-04-25T00:20:51.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.224+0000 7fb040ff9640 1 --2- 172.21.15.116:0/1009954024 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb0240678c0 0x7fb024069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:51.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.224+0000 7fb040ff9640 1 -- 172.21.15.116:0/1009954024 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 9332+0+0 (secure 0 0 0) 0x7fb02c1215a0 con 0x7fb0440c2f50 2024-04-25T00:20:51.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.224+0000 7fb042ffd640 1 --2- 172.21.15.116:0/1009954024 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb0240678c0 0x7fb024069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:51.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.224+0000 7fb042ffd640 1 --2- 172.21.15.116:0/1009954024 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb0240678c0 0x7fb024069d80 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fb04406dfe0 tx=0x7fb034008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:51.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.228+0000 7fb040ff9640 1 -- 172.21.15.116:0/1009954024 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fb02c0c4060 con 0x7fb0440c2f50 2024-04-25T00:20:51.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.408+0000 7fb049a23640 1 -- 172.21.15.116:0/1009954024 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb04406d8e0 con 0x7fb0440c2f50 2024-04-25T00:20:51.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.408+0000 7fb040ff9640 1 -- 172.21.15.116:0/1009954024 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 163 v163) ==== 95+0+5667 (secure 0 0 0) 0x7fb02c0160d0 con 0x7fb0440c2f50 2024-04-25T00:20:51.413 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:20:51.413 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":163,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":163,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:20:50.654702+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7809},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7809":{"gid":7809,"name":"c","rank":0,"incarnation":163,"state":"up:replay","state_seq":1,"addr":"172.21.15.116:6835/980221971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":980221971},{"type":"v1","addr":"172.21.15.116:6835","nonce":980221971}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:20:13.935378+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7722":{"gid":7722,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7722,"qdb_cluster":[7722]},"id":2}]} 2024-04-25T00:20:51.413 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 163 2024-04-25T00:20:51.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.412+0000 7fb049a23640 1 -- 172.21.15.116:0/1009954024 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb0240678c0 msgr2=0x7fb024069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:51.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.412+0000 7fb049a23640 1 --2- 172.21.15.116:0/1009954024 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb0240678c0 0x7fb024069d80 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fb04406dfe0 tx=0x7fb034008040 comp rx=0 tx=0).stop 2024-04-25T00:20:51.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.412+0000 7fb049a23640 1 -- 172.21.15.116:0/1009954024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0440c2f50 msgr2=0x7fb04414b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:51.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.412+0000 7fb049a23640 1 --2- 172.21.15.116:0/1009954024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0440c2f50 0x7fb04414b5e0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7fb02c01c9c0 tx=0x7fb02c01ad20 comp rx=0 tx=0).stop 2024-04-25T00:20:51.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.412+0000 7fb049a23640 1 -- 172.21.15.116:0/1009954024 shutdown_connections 2024-04-25T00:20:51.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.412+0000 7fb049a23640 1 --2- 172.21.15.116:0/1009954024 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb0240678c0 0x7fb024069d80 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:51.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.412+0000 7fb049a23640 1 --2- 172.21.15.116:0/1009954024 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb044150d20 0x7fb04414bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:51.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.412+0000 7fb049a23640 1 --2- 172.21.15.116:0/1009954024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0440c2f50 0x7fb04414b5e0 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:51.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.412+0000 7fb049a23640 1 --2- 172.21.15.116:0/1009954024 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb0440a94a0 0x7fb04414f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:51.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.412+0000 7fb049a23640 1 -- 172.21.15.116:0/1009954024 >> 172.21.15.116:0/1009954024 conn(0x7fb0440c3b00 msgr2=0x7fb0441538e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:20:51.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.412+0000 7fb049a23640 1 -- 172.21.15.116:0/1009954024 shutdown_connections 2024-04-25T00:20:51.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:51.412+0000 7fb049a23640 1 -- 172.21.15.116:0/1009954024 wait complete. 2024-04-25T00:20:51.431 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 163, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:04.678737+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 163, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_7809": { "addr": "172.21.15.116:6835/980221971", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6834", "nonce": 980221971, "type": "v2" }, { "addr": "172.21.15.116:6835", "nonce": 980221971, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7809, "incarnation": 163, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 59, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T00:20:50.654702+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7809 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:18.825887+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 161, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_7722": { "addr": "172.21.15.116:6833/663910035", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6832", "nonce": 663910035, "type": "v2" }, { "addr": "172.21.15.116:6833", "nonce": 663910035, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7722, "incarnation": 158, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 11 } }, "last_failure": 0, "last_failure_osd_epoch": 58, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-04-25T00:20:13.935378+0000", "qdb_cluster": [ 7722 ], "qdb_leader": 7722, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7722 } } } ], "standbys": [ { "addr": "172.21.15.174:6835/4112145782", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6834", "nonce": 4112145782, "type": "v2" }, { "addr": "172.21.15.174:6835", "nonce": 4112145782, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 112, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5582, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-25T00:20:53.433 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:20:53.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.560+0000 7fd071b4f640 1 Processor -- start 2024-04-25T00:20:53.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.564+0000 7fd071b4f640 1 -- start start 2024-04-25T00:20:53.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.564+0000 7fd071b4f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd06c0a94a0 0x7fd06c1563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:53.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.564+0000 7fd071b4f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd06c071680 0x7fd06c071ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:53.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.564+0000 7fd071b4f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd06c071220 con 0x7fd06c14c720 2024-04-25T00:20:53.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.564+0000 7fd071b4f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fd06c153460 con 0x7fd06c071680 2024-04-25T00:20:53.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.564+0000 7fd071b4f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fd06c152420 con 0x7fd06c0a94a0 2024-04-25T00:20:53.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.564+0000 7fd06b7fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fd06c14c720 0x7fd06c0c2a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:55150/0 (socket says 172.21.15.116:55150) 2024-04-25T00:20:53.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.564+0000 7fd06b7fe640 1 -- 172.21.15.116:0/3908081083 learned_addr learned my addr 172.21.15.116:0/3908081083 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:20:53.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.564+0000 7fd06bfff640 1 --2- 172.21.15.116:0/3908081083 >> v2:172.21.15.116:3300/0 conn(0x7fd06c071680 0x7fd06c071ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:53.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.564+0000 7fd063fff640 1 --2- 172.21.15.116:0/3908081083 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd06c0a94a0 0x7fd06c1563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:53.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.564+0000 7fd06affd640 1 -- 172.21.15.116:0/3908081083 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2253927468 0 0) 0x7fd06c071220 con 0x7fd06c14c720 2024-04-25T00:20:53.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.564+0000 7fd06affd640 1 -- 172.21.15.116:0/3908081083 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd058003530 con 0x7fd06c14c720 2024-04-25T00:20:53.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.564+0000 7fd06bfff640 1 -- 172.21.15.116:0/3908081083 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd06c0a94a0 msgr2=0x7fd06c1563d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:20:53.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.564+0000 7fd06bfff640 1 --2- 172.21.15.116:0/3908081083 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd06c0a94a0 0x7fd06c1563d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:53.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.564+0000 7fd06bfff640 1 -- 172.21.15.116:0/3908081083 >> v1:172.21.15.116:6789/0 conn(0x7fd06c14c720 legacy=0x7fd06c0c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:53.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.564+0000 7fd06affd640 1 -- 172.21.15.116:0/3908081083 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2687806370 0 0) 0x7fd058003530 con 0x7fd06c14c720 2024-04-25T00:20:53.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.564+0000 7fd06bfff640 1 -- 172.21.15.116:0/3908081083 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd06c14cb00 con 0x7fd06c071680 2024-04-25T00:20:53.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.568+0000 7fd06bfff640 1 --2- 172.21.15.116:0/3908081083 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd06c071680 0x7fd06c071ad0 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7fd064009d90 tx=0x7fd06401c7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f34a4f7cda2bbcd3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:53.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.568+0000 7fd06affd640 1 -- 172.21.15.116:0/3908081083 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0640b2070 con 0x7fd06c071680 2024-04-25T00:20:53.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.568+0000 7fd06affd640 1 -- 172.21.15.116:0/3908081083 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0640a55d0 con 0x7fd06c071680 2024-04-25T00:20:53.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.568+0000 7fd06affd640 1 -- 172.21.15.116:0/3908081083 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd054001680 con 0x7fd06c071680 2024-04-25T00:20:53.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.568+0000 7fd071b4f640 1 -- 172.21.15.116:0/3908081083 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd06c071680 msgr2=0x7fd06c071ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:53.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.568+0000 7fd071b4f640 1 --2- 172.21.15.116:0/3908081083 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd06c071680 0x7fd06c071ad0 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7fd064009d90 tx=0x7fd06401c7d0 comp rx=0 tx=0).stop 2024-04-25T00:20:53.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.568+0000 7fd071b4f640 1 -- 172.21.15.116:0/3908081083 shutdown_connections 2024-04-25T00:20:53.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.568+0000 7fd071b4f640 1 --2- 172.21.15.116:0/3908081083 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd06c071680 0x7fd06c071ad0 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:53.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.568+0000 7fd071b4f640 1 --2- 172.21.15.116:0/3908081083 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd06c0a94a0 0x7fd06c1563d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:53.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.568+0000 7fd071b4f640 1 -- 172.21.15.116:0/3908081083 >> 172.21.15.116:0/3908081083 conn(0x7fd06c0c3b00 msgr2=0x7fd06c0c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:20:53.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.568+0000 7fd071b4f640 1 -- 172.21.15.116:0/3908081083 shutdown_connections 2024-04-25T00:20:53.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.568+0000 7fd071b4f640 1 -- 172.21.15.116:0/3908081083 wait complete. 2024-04-25T00:20:53.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.568+0000 7fd071b4f640 1 Processor -- start 2024-04-25T00:20:53.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.568+0000 7fd071b4f640 1 -- start start 2024-04-25T00:20:53.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.568+0000 7fd071b4f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd06c071680 0x7fd06c1ec6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:53.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.568+0000 7fd071b4f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd06c0a94a0 0x7fd06c1efb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:53.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.568+0000 7fd071b4f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd06c14c720 0x7fd06c143ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:53.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.568+0000 7fd071b4f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd06c0c3120 con 0x7fd06c0a94a0 2024-04-25T00:20:53.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.568+0000 7fd071b4f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd06c152f40 con 0x7fd06c071680 2024-04-25T00:20:53.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.568+0000 7fd071b4f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fd06c0c32a0 con 0x7fd06c14c720 2024-04-25T00:20:53.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.568+0000 7fd063fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd06c0a94a0 0x7fd06c1efb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:53.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.568+0000 7fd063fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd06c0a94a0 0x7fd06c1efb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51910/0 (socket says 172.21.15.116:51910) 2024-04-25T00:20:53.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.568+0000 7fd063fff640 1 -- 172.21.15.116:0/3329565076 learned_addr learned my addr 172.21.15.116:0/3329565076 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:20:53.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.568+0000 7fd06b7fe640 1 --2- 172.21.15.116:0/3329565076 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd06c071680 0x7fd06c1ec6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:53.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.568+0000 7fd06bfff640 1 --2- 172.21.15.116:0/3329565076 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd06c14c720 0x7fd06c143ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:53.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.572+0000 7fd063fff640 1 -- 172.21.15.116:0/3329565076 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd06c14c720 msgr2=0x7fd06c143ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:53.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.572+0000 7fd063fff640 1 --2- 172.21.15.116:0/3329565076 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd06c14c720 0x7fd06c143ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:53.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.572+0000 7fd063fff640 1 -- 172.21.15.116:0/3329565076 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd06c071680 msgr2=0x7fd06c1ec6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:53.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.572+0000 7fd063fff640 1 --2- 172.21.15.116:0/3329565076 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd06c071680 0x7fd06c1ec6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:53.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.572+0000 7fd063fff640 1 -- 172.21.15.116:0/3329565076 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd06c1f0070 con 0x7fd06c0a94a0 2024-04-25T00:20:53.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.572+0000 7fd06b7fe640 1 --2- 172.21.15.116:0/3329565076 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd06c071680 0x7fd06c1ec6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:20:53.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.572+0000 7fd063fff640 1 --2- 172.21.15.116:0/3329565076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd06c0a94a0 0x7fd06c1efb30 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7fd05c00e960 tx=0x7fd05c00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:53.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.572+0000 7fd068ff9640 1 -- 172.21.15.116:0/3329565076 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd05c017070 con 0x7fd06c0a94a0 2024-04-25T00:20:53.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.572+0000 7fd068ff9640 1 -- 172.21.15.116:0/3329565076 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd05c00ca00 con 0x7fd06c0a94a0 2024-04-25T00:20:53.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.572+0000 7fd068ff9640 1 -- 172.21.15.116:0/3329565076 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd05c00cdb0 con 0x7fd06c0a94a0 2024-04-25T00:20:53.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.572+0000 7fd071b4f640 1 -- 172.21.15.116:0/3329565076 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd06c1ebc00 con 0x7fd06c0a94a0 2024-04-25T00:20:53.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.572+0000 7fd071b4f640 1 -- 172.21.15.116:0/3329565076 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd06c14aca0 con 0x7fd06c0a94a0 2024-04-25T00:20:53.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.572+0000 7fd071b4f640 1 -- 172.21.15.116:0/3329565076 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd06c068350 con 0x7fd06c0a94a0 2024-04-25T00:20:53.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.572+0000 7fd068ff9640 1 -- 172.21.15.116:0/3329565076 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd05c01a070 con 0x7fd06c0a94a0 2024-04-25T00:20:53.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.572+0000 7fd068ff9640 1 --2- 172.21.15.116:0/3329565076 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd04c0678c0 0x7fd04c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:53.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.572+0000 7fd068ff9640 1 -- 172.21.15.116:0/3329565076 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 9332+0+0 (secure 0 0 0) 0x7fd05c086ab0 con 0x7fd06c0a94a0 2024-04-25T00:20:53.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.572+0000 7fd06b7fe640 1 --2- 172.21.15.116:0/3329565076 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd04c0678c0 0x7fd04c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:53.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.576+0000 7fd06b7fe640 1 --2- 172.21.15.116:0/3329565076 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd04c0678c0 0x7fd04c069d80 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fd054001390 tx=0x7fd054009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:53.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.576+0000 7fd068ff9640 1 -- 172.21.15.116:0/3329565076 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fd05c056800 con 0x7fd06c0a94a0 2024-04-25T00:20:53.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.772+0000 7fd071b4f640 1 -- 172.21.15.116:0/3329565076 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd06c0c3640 con 0x7fd06c0a94a0 2024-04-25T00:20:53.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.772+0000 7fd068ff9640 1 -- 172.21.15.116:0/3329565076 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) ==== 95+0+5668 (secure 0 0 0) 0x7fd05c056100 con 0x7fd06c0a94a0 2024-04-25T00:20:53.774 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:20:53.775 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":165,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":165,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:20:51.861122+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7809},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7809":{"gid":7809,"name":"c","rank":0,"incarnation":163,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.116:6835/980221971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":980221971},{"type":"v1","addr":"172.21.15.116:6835","nonce":980221971}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:20:13.935378+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7722":{"gid":7722,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7722,"qdb_cluster":[7722]},"id":2}]} 2024-04-25T00:20:53.775 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 165 2024-04-25T00:20:53.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.772+0000 7fd071b4f640 1 -- 172.21.15.116:0/3329565076 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd04c0678c0 msgr2=0x7fd04c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:53.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.772+0000 7fd071b4f640 1 --2- 172.21.15.116:0/3329565076 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd04c0678c0 0x7fd04c069d80 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fd054001390 tx=0x7fd054009040 comp rx=0 tx=0).stop 2024-04-25T00:20:53.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.772+0000 7fd071b4f640 1 -- 172.21.15.116:0/3329565076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd06c0a94a0 msgr2=0x7fd06c1efb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:53.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.772+0000 7fd071b4f640 1 --2- 172.21.15.116:0/3329565076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd06c0a94a0 0x7fd06c1efb30 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7fd05c00e960 tx=0x7fd05c00ee20 comp rx=0 tx=0).stop 2024-04-25T00:20:53.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.772+0000 7fd071b4f640 1 -- 172.21.15.116:0/3329565076 shutdown_connections 2024-04-25T00:20:53.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.772+0000 7fd071b4f640 1 --2- 172.21.15.116:0/3329565076 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd04c0678c0 0x7fd04c069d80 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:53.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.772+0000 7fd071b4f640 1 --2- 172.21.15.116:0/3329565076 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd06c14c720 0x7fd06c143ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:53.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.772+0000 7fd071b4f640 1 --2- 172.21.15.116:0/3329565076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd06c0a94a0 0x7fd06c1efb30 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:53.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.772+0000 7fd071b4f640 1 --2- 172.21.15.116:0/3329565076 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd06c071680 0x7fd06c1ec6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:53.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.772+0000 7fd071b4f640 1 -- 172.21.15.116:0/3329565076 >> 172.21.15.116:0/3329565076 conn(0x7fd06c0c3b00 msgr2=0x7fd06c071c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:20:53.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.776+0000 7fd071b4f640 1 -- 172.21.15.116:0/3329565076 shutdown_connections 2024-04-25T00:20:53.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:53.776+0000 7fd071b4f640 1 -- 172.21.15.116:0/3329565076 wait complete. 2024-04-25T00:20:53.794 INFO:tasks.mds_thrash.fs.[a]:no change 2024-04-25T00:20:54.837 INFO:tasks.ceph.mds.c.smithi116.stderr:2024-04-25T00:20:54.836+0000 7f7132852640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T00:20:55.795 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:20:55.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:55.944+0000 7f82670f0640 1 Processor -- start 2024-04-25T00:20:55.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:55.944+0000 7f82670f0640 1 -- start start 2024-04-25T00:20:55.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:55.944+0000 7f82670f0640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f82600c2e20 0x7f8260155e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:55.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:55.944+0000 7f82670f0640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8260153140 con 0x7f8260152d60 2024-04-25T00:20:55.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:55.944+0000 7f82670f0640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f826006f790 con 0x7f8260158d10 2024-04-25T00:20:55.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:55.944+0000 7f82670f0640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f8260155720 con 0x7f82600c2e20 2024-04-25T00:20:55.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:55.944+0000 7f825ffff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f82600c2e20 0x7f8260155e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:55.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:55.944+0000 7f8264e8c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f8260152d60 0x7f826006c220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:36724/0 (socket says 172.21.15.116:36724) 2024-04-25T00:20:55.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:55.944+0000 7f8264e8c640 1 -- 172.21.15.116:0/2295763861 learned_addr learned my addr 172.21.15.116:0/2295763861 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:20:55.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:55.944+0000 7f825ffff640 1 -- 172.21.15.116:0/2295763861 >> v1:172.21.15.174:6789/0 conn(0x7f8260158d10 legacy=0x7f82601600a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:55.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:55.944+0000 7f825ffff640 1 -- 172.21.15.116:0/2295763861 >> v1:172.21.15.116:6789/0 conn(0x7f8260152d60 legacy=0x7f826006c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:55.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:55.948+0000 7f825ffff640 1 -- 172.21.15.116:0/2295763861 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f826006fa70 con 0x7f82600c2e20 2024-04-25T00:20:55.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:55.948+0000 7f825f7fe640 1 -- 172.21.15.116:0/2295763861 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 995889656 0 0) 0x7f8260153140 con 0x7f8260152d60 2024-04-25T00:20:55.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:55.948+0000 7f825ffff640 1 --2- 172.21.15.116:0/2295763861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82600c2e20 0x7f8260155e70 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f824c0097c0 tx=0x7f824c01c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b051e31e5ee7045e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:55.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:55.948+0000 7f825f7fe640 1 -- 172.21.15.116:0/2295763861 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f824c0ab610 con 0x7f82600c2e20 2024-04-25T00:20:55.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:55.948+0000 7f825f7fe640 1 -- 172.21.15.116:0/2295763861 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f824c0b0070 con 0x7f82600c2e20 2024-04-25T00:20:55.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:55.948+0000 7f825f7fe640 1 -- 172.21.15.116:0/2295763861 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f824c0ab9d0 con 0x7f82600c2e20 2024-04-25T00:20:55.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:55.948+0000 7f82670f0640 1 -- 172.21.15.116:0/2295763861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82600c2e20 msgr2=0x7f8260155e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:55.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:55.948+0000 7f82670f0640 1 --2- 172.21.15.116:0/2295763861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82600c2e20 0x7f8260155e70 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f824c0097c0 tx=0x7f824c01c410 comp rx=0 tx=0).stop 2024-04-25T00:20:55.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:55.948+0000 7f82670f0640 1 -- 172.21.15.116:0/2295763861 shutdown_connections 2024-04-25T00:20:55.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:55.948+0000 7f82670f0640 1 --2- 172.21.15.116:0/2295763861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82600c2e20 0x7f8260155e70 secure :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f824c0097c0 tx=0x7f824c01c410 comp rx=0 tx=0).stop 2024-04-25T00:20:55.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:55.948+0000 7f82670f0640 1 -- 172.21.15.116:0/2295763861 >> 172.21.15.116:0/2295763861 conn(0x7f82600c4170 msgr2=0x7f82600c4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:20:55.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:55.948+0000 7f82670f0640 1 -- 172.21.15.116:0/2295763861 shutdown_connections 2024-04-25T00:20:55.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:55.948+0000 7f82670f0640 1 -- 172.21.15.116:0/2295763861 wait complete. 2024-04-25T00:20:55.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:55.952+0000 7f82670f0640 1 Processor -- start 2024-04-25T00:20:55.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:55.952+0000 7f82670f0640 1 -- start start 2024-04-25T00:20:55.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:55.952+0000 7f82670f0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8260152d60 0x7f8260157b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:55.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:55.952+0000 7f82670f0640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8260158d10 0x7f8260143370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:55.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:55.952+0000 7f82670f0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82601447f0 0x7f82601438b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:55.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:55.952+0000 7f82670f0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f826006f4a0 con 0x7f8260152d60 2024-04-25T00:20:55.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:55.952+0000 7f82670f0640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f826006f320 con 0x7f82601447f0 2024-04-25T00:20:55.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:55.952+0000 7f82670f0640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f8260163fe0 con 0x7f8260158d10 2024-04-25T00:20:55.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:55.952+0000 7f8264e8c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8260152d60 0x7f8260157b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:55.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:55.952+0000 7f8264e8c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8260152d60 0x7f8260157b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52422/0 (socket says 172.21.15.116:52422) 2024-04-25T00:20:55.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:55.952+0000 7f8264e8c640 1 -- 172.21.15.116:0/3404580250 learned_addr learned my addr 172.21.15.116:0/3404580250 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:20:55.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:55.952+0000 7f826568d640 1 --2- 172.21.15.116:0/3404580250 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82601447f0 0x7f82601438b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:55.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:55.952+0000 7f825ffff640 1 --2- 172.21.15.116:0/3404580250 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8260158d10 0x7f8260143370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:55.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:55.952+0000 7f8264e8c640 1 -- 172.21.15.116:0/3404580250 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8260158d10 msgr2=0x7f8260143370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:55.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:55.952+0000 7f8264e8c640 1 --2- 172.21.15.116:0/3404580250 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8260158d10 0x7f8260143370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:55.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:55.952+0000 7f8264e8c640 1 -- 172.21.15.116:0/3404580250 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82601447f0 msgr2=0x7f82601438b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:55.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:55.952+0000 7f8264e8c640 1 --2- 172.21.15.116:0/3404580250 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82601447f0 0x7f82601438b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:55.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:55.952+0000 7f8264e8c640 1 -- 172.21.15.116:0/3404580250 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f826014bc00 con 0x7f8260152d60 2024-04-25T00:20:55.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:55.952+0000 7f8264e8c640 1 --2- 172.21.15.116:0/3404580250 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8260152d60 0x7f8260157b50 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f825400c4d0 tx=0x7f825400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:55.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:55.952+0000 7f825d7fa640 1 -- 172.21.15.116:0/3404580250 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8254018070 con 0x7f8260152d60 2024-04-25T00:20:55.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:55.952+0000 7f825d7fa640 1 -- 172.21.15.116:0/3404580250 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82540074e0 con 0x7f8260152d60 2024-04-25T00:20:55.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:55.952+0000 7f82670f0640 1 -- 172.21.15.116:0/3404580250 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f826014bee0 con 0x7f8260152d60 2024-04-25T00:20:55.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:55.952+0000 7f82670f0640 1 -- 172.21.15.116:0/3404580250 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82601465c0 con 0x7f8260152d60 2024-04-25T00:20:55.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:55.952+0000 7f825d7fa640 1 -- 172.21.15.116:0/3404580250 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8254004a30 con 0x7f8260152d60 2024-04-25T00:20:55.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:55.952+0000 7f82670f0640 1 -- 172.21.15.116:0/3404580250 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8260068a60 con 0x7f8260152d60 2024-04-25T00:20:55.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:55.956+0000 7f825d7fa640 1 -- 172.21.15.116:0/3404580250 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f825401b070 con 0x7f8260152d60 2024-04-25T00:20:55.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:55.956+0000 7f825d7fa640 1 --2- 172.21.15.116:0/3404580250 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f82440678c0 0x7f8244069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:55.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:55.956+0000 7f825d7fa640 1 -- 172.21.15.116:0/3404580250 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 9332+0+0 (secure 0 0 0) 0x7f8254085ec0 con 0x7f8260152d60 2024-04-25T00:20:55.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:55.956+0000 7f825ffff640 1 --2- 172.21.15.116:0/3404580250 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f82440678c0 0x7f8244069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:55.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:55.956+0000 7f825ffff640 1 --2- 172.21.15.116:0/3404580250 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f82440678c0 0x7f8244069d80 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f824c0af000 tx=0x7f824c0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:55.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:55.960+0000 7f825d7fa640 1 -- 172.21.15.116:0/3404580250 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f82540554a0 con 0x7f8260152d60 2024-04-25T00:20:56.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:56.216+0000 7f82670f0640 1 -- 172.21.15.116:0/3404580250 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f82600748f0 con 0x7f8260152d60 2024-04-25T00:20:56.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:56.220+0000 7f825d7fa640 1 -- 172.21.15.116:0/3404580250 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 166 v166) ==== 95+0+5675 (secure 0 0 0) 0x7f82540597a0 con 0x7f8260152d60 2024-04-25T00:20:56.221 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:20:56.221 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":166,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":166,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:20:53.870063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7809},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7809":{"gid":7809,"name":"c","rank":0,"incarnation":163,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6835/980221971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":980221971},{"type":"v1","addr":"172.21.15.116:6835","nonce":980221971}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7809,"qdb_cluster":[7809]},"id":1},{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:20:13.935378+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7722":{"gid":7722,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7722,"qdb_cluster":[7722]},"id":2}]} 2024-04-25T00:20:56.222 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 166 2024-04-25T00:20:56.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:56.220+0000 7f82670f0640 1 -- 172.21.15.116:0/3404580250 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f82440678c0 msgr2=0x7f8244069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:56.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:56.220+0000 7f82670f0640 1 --2- 172.21.15.116:0/3404580250 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f82440678c0 0x7f8244069d80 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f824c0af000 tx=0x7f824c0a7000 comp rx=0 tx=0).stop 2024-04-25T00:20:56.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:56.220+0000 7f82670f0640 1 -- 172.21.15.116:0/3404580250 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8260152d60 msgr2=0x7f8260157b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:56.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:56.220+0000 7f82670f0640 1 --2- 172.21.15.116:0/3404580250 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8260152d60 0x7f8260157b50 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f825400c4d0 tx=0x7f825400c990 comp rx=0 tx=0).stop 2024-04-25T00:20:56.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:56.220+0000 7f82670f0640 1 -- 172.21.15.116:0/3404580250 shutdown_connections 2024-04-25T00:20:56.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:56.220+0000 7f82670f0640 1 --2- 172.21.15.116:0/3404580250 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f82440678c0 0x7f8244069d80 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:56.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:56.220+0000 7f82670f0640 1 --2- 172.21.15.116:0/3404580250 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82601447f0 0x7f82601438b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:56.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:56.220+0000 7f82670f0640 1 --2- 172.21.15.116:0/3404580250 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8260158d10 0x7f8260143370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:56.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:56.220+0000 7f82670f0640 1 --2- 172.21.15.116:0/3404580250 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8260152d60 0x7f8260157b50 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:56.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:56.220+0000 7f82670f0640 1 -- 172.21.15.116:0/3404580250 >> 172.21.15.116:0/3404580250 conn(0x7f82600c4170 msgr2=0x7f826006c120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:20:56.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:56.220+0000 7f82670f0640 1 -- 172.21.15.116:0/3404580250 shutdown_connections 2024-04-25T00:20:56.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:56.220+0000 7f82670f0640 1 -- 172.21.15.116:0/3404580250 wait complete. 2024-04-25T00:20:56.245 INFO:tasks.mds_thrash.fs.[a]:mds.c has gained rank=0, replacing gid=6586 2024-04-25T00:20:56.246 INFO:tasks.mds_thrash.fs.[a]:waiting for 3 secs before reviving mds.d 2024-04-25T00:20:59.247 INFO:tasks.mds_thrash.fs.[a]:reviving mds.d 2024-04-25T00:20:59.247 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-25T00:20:59.247 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-25T00:20:59.250 INFO:tasks.ceph.mds.d:Started 2024-04-25T00:20:59.250 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:20:59.337 INFO:tasks.ceph.mds.d.smithi174.stdout:starting mds.d at 2024-04-25T00:20:59.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.372+0000 7ff6b41ab640 1 Processor -- start 2024-04-25T00:20:59.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.372+0000 7ff6b41ab640 1 -- start start 2024-04-25T00:20:59.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.372+0000 7ff6b41ab640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff6ac0c5200 0x7ff6ac154040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:59.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.372+0000 7ff6b41ab640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff6ac0c4760 0x7ff6ac14d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:59.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.372+0000 7ff6b41ab640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6ac0c2b40 con 0x7ff6ac14d770 2024-04-25T00:20:59.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.372+0000 7ff6b41ab640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7ff6ac072b80 con 0x7ff6ac0c4760 2024-04-25T00:20:59.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.372+0000 7ff6b41ab640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff6ac154580 con 0x7ff6ac0c5200 2024-04-25T00:20:59.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.372+0000 7ff6b1f47640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff6ac0c5200 0x7ff6ac154040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:59.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.372+0000 7ff6b2748640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7ff6ac14d770 0x7ff6ac156bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:36732/0 (socket says 172.21.15.116:36732) 2024-04-25T00:20:59.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.372+0000 7ff6b2748640 1 -- 172.21.15.116:0/2497700303 learned_addr learned my addr 172.21.15.116:0/2497700303 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:20:59.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.372+0000 7ff6b1746640 1 --2- 172.21.15.116:0/2497700303 >> v2:172.21.15.174:3300/0 conn(0x7ff6ac0c4760 0x7ff6ac14d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:59.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.372+0000 7ff6b1f47640 1 -- 172.21.15.116:0/2497700303 >> v2:172.21.15.174:3300/0 conn(0x7ff6ac0c4760 msgr2=0x7ff6ac14d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:20:59.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.372+0000 7ff6b1f47640 1 --2- 172.21.15.116:0/2497700303 >> v2:172.21.15.174:3300/0 conn(0x7ff6ac0c4760 0x7ff6ac14d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:59.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.372+0000 7ff6b1f47640 1 -- 172.21.15.116:0/2497700303 >> v1:172.21.15.116:6789/0 conn(0x7ff6ac14d770 legacy=0x7ff6ac156bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:59.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.372+0000 7ff6b1f47640 1 -- 172.21.15.116:0/2497700303 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6ac0c2db0 con 0x7ff6ac0c5200 2024-04-25T00:20:59.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.372+0000 7ff6b0f45640 1 -- 172.21.15.116:0/2497700303 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 81591856 0 0) 0x7ff6ac0c2b40 con 0x7ff6ac14d770 2024-04-25T00:20:59.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.376+0000 7ff6b1f47640 1 --2- 172.21.15.116:0/2497700303 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff6ac0c5200 0x7ff6ac154040 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff6a80097c0 tx=0x7ff6a801c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2cc9336fe7b0b34 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:59.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.376+0000 7ff6b0f45640 1 -- 172.21.15.116:0/2497700303 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6a80b0070 con 0x7ff6ac0c5200 2024-04-25T00:20:59.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.376+0000 7ff6b0f45640 1 -- 172.21.15.116:0/2497700303 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6a80ab600 con 0x7ff6ac0c5200 2024-04-25T00:20:59.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.376+0000 7ff6b0f45640 1 -- 172.21.15.116:0/2497700303 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6a80ab9d0 con 0x7ff6ac0c5200 2024-04-25T00:20:59.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.376+0000 7ff6b41ab640 1 -- 172.21.15.116:0/2497700303 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff6ac0c5200 msgr2=0x7ff6ac154040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:59.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.376+0000 7ff6b41ab640 1 --2- 172.21.15.116:0/2497700303 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff6ac0c5200 0x7ff6ac154040 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff6a80097c0 tx=0x7ff6a801c410 comp rx=0 tx=0).stop 2024-04-25T00:20:59.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.376+0000 7ff6b41ab640 1 -- 172.21.15.116:0/2497700303 shutdown_connections 2024-04-25T00:20:59.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.376+0000 7ff6b41ab640 1 --2- 172.21.15.116:0/2497700303 >> v2:172.21.15.174:3300/0 conn(0x7ff6ac0c4760 0x7ff6ac14d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:59.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.376+0000 7ff6b41ab640 1 --2- 172.21.15.116:0/2497700303 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff6ac0c5200 0x7ff6ac154040 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:59.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.376+0000 7ff6b41ab640 1 -- 172.21.15.116:0/2497700303 >> 172.21.15.116:0/2497700303 conn(0x7ff6ac06c720 msgr2=0x7ff6ac06cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:20:59.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.376+0000 7ff6b41ab640 1 -- 172.21.15.116:0/2497700303 shutdown_connections 2024-04-25T00:20:59.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.376+0000 7ff6b41ab640 1 -- 172.21.15.116:0/2497700303 wait complete. 2024-04-25T00:20:59.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.380+0000 7ff6b41ab640 1 Processor -- start 2024-04-25T00:20:59.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.380+0000 7ff6b41ab640 1 -- start start 2024-04-25T00:20:59.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.380+0000 7ff6b41ab640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff6ac0c4760 0x7ff6ac13c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:59.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.380+0000 7ff6b41ab640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6ac0c5200 0x7ff6ac0c65a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:59.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.380+0000 7ff6b41ab640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6ac14d770 0x7ff6ac1447f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:59.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.380+0000 7ff6b41ab640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff6ac0c55e0 con 0x7ff6ac14d770 2024-04-25T00:20:59.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.380+0000 7ff6b41ab640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff6ac0c2720 con 0x7ff6ac0c5200 2024-04-25T00:20:59.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.380+0000 7ff6b41ab640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff6ac0c5760 con 0x7ff6ac0c4760 2024-04-25T00:20:59.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.380+0000 7ff6b2748640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6ac14d770 0x7ff6ac1447f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:59.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.380+0000 7ff6b1f47640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff6ac0c4760 0x7ff6ac13c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:59.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.380+0000 7ff6b2748640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6ac14d770 0x7ff6ac1447f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52438/0 (socket says 172.21.15.116:52438) 2024-04-25T00:20:59.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.380+0000 7ff6b2748640 1 -- 172.21.15.116:0/1372014192 learned_addr learned my addr 172.21.15.116:0/1372014192 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:20:59.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.380+0000 7ff6b1746640 1 --2- 172.21.15.116:0/1372014192 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6ac0c5200 0x7ff6ac0c65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:59.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.380+0000 7ff6b2748640 1 -- 172.21.15.116:0/1372014192 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff6ac0c4760 msgr2=0x7ff6ac13c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:59.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.380+0000 7ff6b2748640 1 --2- 172.21.15.116:0/1372014192 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff6ac0c4760 0x7ff6ac13c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:59.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.380+0000 7ff6b2748640 1 -- 172.21.15.116:0/1372014192 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6ac0c5200 msgr2=0x7ff6ac0c65a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:59.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.380+0000 7ff6b2748640 1 --2- 172.21.15.116:0/1372014192 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6ac0c5200 0x7ff6ac0c65a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:59.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.380+0000 7ff6b2748640 1 -- 172.21.15.116:0/1372014192 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6ac1549b0 con 0x7ff6ac14d770 2024-04-25T00:20:59.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.380+0000 7ff6b1746640 1 --2- 172.21.15.116:0/1372014192 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6ac0c5200 0x7ff6ac0c65a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:20:59.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.380+0000 7ff6b1f47640 1 --2- 172.21.15.116:0/1372014192 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff6ac0c4760 0x7ff6ac13c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:20:59.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.380+0000 7ff6b2748640 1 --2- 172.21.15.116:0/1372014192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6ac14d770 0x7ff6ac1447f0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7ff6a400c700 tx=0x7ff6a400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:59.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.380+0000 7ff6a27fc640 1 -- 172.21.15.116:0/1372014192 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6a4018070 con 0x7ff6ac14d770 2024-04-25T00:20:59.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.380+0000 7ff6a27fc640 1 -- 172.21.15.116:0/1372014192 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6a400ce10 con 0x7ff6ac14d770 2024-04-25T00:20:59.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.380+0000 7ff6b41ab640 1 -- 172.21.15.116:0/1372014192 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6ac154c90 con 0x7ff6ac14d770 2024-04-25T00:20:59.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.380+0000 7ff6a27fc640 1 -- 172.21.15.116:0/1372014192 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6a400a860 con 0x7ff6ac14d770 2024-04-25T00:20:59.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.380+0000 7ff6b41ab640 1 -- 172.21.15.116:0/1372014192 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6ac155250 con 0x7ff6ac14d770 2024-04-25T00:20:59.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.384+0000 7ff6a27fc640 1 -- 172.21.15.116:0/1372014192 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff6a401b070 con 0x7ff6ac14d770 2024-04-25T00:20:59.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.384+0000 7ff6b41ab640 1 -- 172.21.15.116:0/1372014192 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6ac068a60 con 0x7ff6ac14d770 2024-04-25T00:20:59.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.384+0000 7ff6a27fc640 1 --2- 172.21.15.116:0/1372014192 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff6900678c0 0x7ff690069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:20:59.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.384+0000 7ff6a27fc640 1 -- 172.21.15.116:0/1372014192 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 9332+0+0 (secure 0 0 0) 0x7ff6a4085fa0 con 0x7ff6ac14d770 2024-04-25T00:20:59.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.384+0000 7ff6b1f47640 1 --2- 172.21.15.116:0/1372014192 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff6900678c0 0x7ff690069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:20:59.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.384+0000 7ff6b1f47640 1 --2- 172.21.15.116:0/1372014192 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff6900678c0 0x7ff690069d80 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7ff6a80ad000 tx=0x7ff6a80a4f90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:20:59.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.388+0000 7ff6a27fc640 1 -- 172.21.15.116:0/1372014192 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ff6a40556a0 con 0x7ff6ac14d770 2024-04-25T00:20:59.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.608+0000 7ff6b41ab640 1 -- 172.21.15.116:0/1372014192 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff6ac06fa30 con 0x7ff6ac14d770 2024-04-25T00:20:59.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.608+0000 7ff6a27fc640 1 -- 172.21.15.116:0/1372014192 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 166 v166) ==== 95+0+5675 (secure 0 0 0) 0x7ff6a40599a0 con 0x7ff6ac14d770 2024-04-25T00:20:59.611 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:20:59.611 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":166,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":166,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:20:53.870063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7809},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7809":{"gid":7809,"name":"c","rank":0,"incarnation":163,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6835/980221971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":980221971},{"type":"v1","addr":"172.21.15.116:6835","nonce":980221971}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7809,"qdb_cluster":[7809]},"id":1},{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:20:13.935378+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7722":{"gid":7722,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7722,"qdb_cluster":[7722]},"id":2}]} 2024-04-25T00:20:59.612 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 166 2024-04-25T00:20:59.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.612+0000 7ff6b41ab640 1 -- 172.21.15.116:0/1372014192 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff6900678c0 msgr2=0x7ff690069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:59.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.612+0000 7ff6b41ab640 1 --2- 172.21.15.116:0/1372014192 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff6900678c0 0x7ff690069d80 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7ff6a80ad000 tx=0x7ff6a80a4f90 comp rx=0 tx=0).stop 2024-04-25T00:20:59.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.612+0000 7ff6b41ab640 1 -- 172.21.15.116:0/1372014192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6ac14d770 msgr2=0x7ff6ac1447f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:20:59.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.612+0000 7ff6b41ab640 1 --2- 172.21.15.116:0/1372014192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6ac14d770 0x7ff6ac1447f0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7ff6a400c700 tx=0x7ff6a400cbc0 comp rx=0 tx=0).stop 2024-04-25T00:20:59.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.612+0000 7ff6b41ab640 1 -- 172.21.15.116:0/1372014192 shutdown_connections 2024-04-25T00:20:59.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.612+0000 7ff6b41ab640 1 --2- 172.21.15.116:0/1372014192 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff6900678c0 0x7ff690069d80 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:59.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.612+0000 7ff6b41ab640 1 --2- 172.21.15.116:0/1372014192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6ac14d770 0x7ff6ac1447f0 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:59.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.612+0000 7ff6b41ab640 1 --2- 172.21.15.116:0/1372014192 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6ac0c5200 0x7ff6ac0c65a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:59.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.612+0000 7ff6b41ab640 1 --2- 172.21.15.116:0/1372014192 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff6ac0c4760 0x7ff6ac13c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:20:59.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.612+0000 7ff6b41ab640 1 -- 172.21.15.116:0/1372014192 >> 172.21.15.116:0/1372014192 conn(0x7ff6ac06c720 msgr2=0x7ff6ac0aa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:20:59.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.612+0000 7ff6b41ab640 1 -- 172.21.15.116:0/1372014192 shutdown_connections 2024-04-25T00:20:59.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:20:59.612+0000 7ff6b41ab640 1 -- 172.21.15.116:0/1372014192 wait complete. 2024-04-25T00:20:59.632 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2024-04-25T00:21:01.633 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:21:01.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:01.780+0000 7fd3549ed640 1 Processor -- start 2024-04-25T00:21:01.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:01.780+0000 7fd3549ed640 1 -- start start 2024-04-25T00:21:01.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:01.784+0000 7fd3549ed640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd3500c3a20 0x7fd3501610f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:01.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:01.784+0000 7fd3549ed640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd350153220 con 0x7fd350152e40 2024-04-25T00:21:01.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:01.784+0000 7fd3549ed640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3500c7a90 con 0x7fd3500c3f70 2024-04-25T00:21:01.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:01.784+0000 7fd3549ed640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fd3500c4c90 con 0x7fd3500c3a20 2024-04-25T00:21:01.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:01.784+0000 7fd34ed9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd3500c3a20 0x7fd3501610f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:01.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:01.784+0000 7fd34e59c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fd350152e40 0x7fd3500710b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:36740/0 (socket says 172.21.15.116:36740) 2024-04-25T00:21:01.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:01.784+0000 7fd34e59c640 1 -- 172.21.15.116:0/4009669162 learned_addr learned my addr 172.21.15.116:0/4009669162 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:21:01.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:01.784+0000 7fd34ed9d640 1 -- 172.21.15.116:0/4009669162 >> v1:172.21.15.174:6789/0 conn(0x7fd3500c3f70 legacy=0x7fd3501557c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:01.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:01.784+0000 7fd34ed9d640 1 -- 172.21.15.116:0/4009669162 >> v1:172.21.15.116:6789/0 conn(0x7fd350152e40 legacy=0x7fd3500710b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:01.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:01.784+0000 7fd34ed9d640 1 -- 172.21.15.116:0/4009669162 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3500c7d70 con 0x7fd3500c3a20 2024-04-25T00:21:01.787 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:01.784+0000 7fd34d59a640 1 -- 172.21.15.116:0/4009669162 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2546576164 0 0) 0x7fd350153220 con 0x7fd350152e40 2024-04-25T00:21:01.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:01.788+0000 7fd34ed9d640 1 --2- 172.21.15.116:0/4009669162 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3500c3a20 0x7fd3501610f0 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7fd348009d90 tx=0x7fd34801c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=900b0ac2c9b544aa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:01.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:01.788+0000 7fd34d59a640 1 -- 172.21.15.116:0/4009669162 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3480b2070 con 0x7fd3500c3a20 2024-04-25T00:21:01.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:01.788+0000 7fd34d59a640 1 -- 172.21.15.116:0/4009669162 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3480a5990 con 0x7fd3500c3a20 2024-04-25T00:21:01.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:01.788+0000 7fd34d59a640 1 -- 172.21.15.116:0/4009669162 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3480a5d60 con 0x7fd3500c3a20 2024-04-25T00:21:01.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:01.788+0000 7fd3549ed640 1 -- 172.21.15.116:0/4009669162 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3500c3a20 msgr2=0x7fd3501610f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:01.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:01.788+0000 7fd3549ed640 1 --2- 172.21.15.116:0/4009669162 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3500c3a20 0x7fd3501610f0 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7fd348009d90 tx=0x7fd34801c8b0 comp rx=0 tx=0).stop 2024-04-25T00:21:01.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:01.788+0000 7fd3549ed640 1 -- 172.21.15.116:0/4009669162 shutdown_connections 2024-04-25T00:21:01.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:01.788+0000 7fd3549ed640 1 --2- 172.21.15.116:0/4009669162 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3500c3a20 0x7fd3501610f0 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:01.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:01.788+0000 7fd3549ed640 1 -- 172.21.15.116:0/4009669162 >> 172.21.15.116:0/4009669162 conn(0x7fd3500c9160 msgr2=0x7fd3500c9570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:21:01.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:01.788+0000 7fd3549ed640 1 -- 172.21.15.116:0/4009669162 shutdown_connections 2024-04-25T00:21:01.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:01.788+0000 7fd3549ed640 1 -- 172.21.15.116:0/4009669162 wait complete. 2024-04-25T00:21:01.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:01.788+0000 7fd3549ed640 1 Processor -- start 2024-04-25T00:21:01.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:01.788+0000 7fd3549ed640 1 -- start start 2024-04-25T00:21:01.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:01.788+0000 7fd3549ed640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3500c3a20 0x7fd3501483e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:01.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:01.788+0000 7fd3549ed640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3500c3f70 0x7fd350146110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:01.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:01.788+0000 7fd3549ed640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd350152e40 0x7fd350146650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:01.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:01.788+0000 7fd3549ed640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd3500c52a0 con 0x7fd3500c3f70 2024-04-25T00:21:01.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:01.788+0000 7fd3549ed640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd3500c5120 con 0x7fd350152e40 2024-04-25T00:21:01.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:01.788+0000 7fd3549ed640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fd350163e50 con 0x7fd3500c3a20 2024-04-25T00:21:01.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:01.788+0000 7fd34dd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3500c3f70 0x7fd350146110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:01.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:01.788+0000 7fd34dd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3500c3f70 0x7fd350146110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52444/0 (socket says 172.21.15.116:52444) 2024-04-25T00:21:01.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:01.788+0000 7fd34dd9b640 1 -- 172.21.15.116:0/524928029 learned_addr learned my addr 172.21.15.116:0/524928029 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:21:01.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:01.788+0000 7fd34e59c640 1 --2- 172.21.15.116:0/524928029 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3500c3a20 0x7fd3501483e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:01.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:01.788+0000 7fd34ed9d640 1 --2- 172.21.15.116:0/524928029 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd350152e40 0x7fd350146650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:01.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:01.792+0000 7fd34dd9b640 1 -- 172.21.15.116:0/524928029 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3500c3a20 msgr2=0x7fd3501483e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:01.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:01.792+0000 7fd34dd9b640 1 --2- 172.21.15.116:0/524928029 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3500c3a20 0x7fd3501483e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:01.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:01.792+0000 7fd34dd9b640 1 -- 172.21.15.116:0/524928029 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd350152e40 msgr2=0x7fd350146650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:01.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:01.792+0000 7fd34dd9b640 1 --2- 172.21.15.116:0/524928029 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd350152e40 0x7fd350146650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:01.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:01.792+0000 7fd34dd9b640 1 -- 172.21.15.116:0/524928029 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd350148de0 con 0x7fd3500c3f70 2024-04-25T00:21:01.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:01.792+0000 7fd34dd9b640 1 --2- 172.21.15.116:0/524928029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3500c3f70 0x7fd350146110 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7fd34400c3f0 tx=0x7fd34400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:01.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:01.792+0000 7fd33effd640 1 -- 172.21.15.116:0/524928029 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd344017070 con 0x7fd3500c3f70 2024-04-25T00:21:01.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:01.792+0000 7fd33effd640 1 -- 172.21.15.116:0/524928029 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3440052d0 con 0x7fd3500c3f70 2024-04-25T00:21:01.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:01.792+0000 7fd33effd640 1 -- 172.21.15.116:0/524928029 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd344003840 con 0x7fd3500c3f70 2024-04-25T00:21:01.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:01.792+0000 7fd3549ed640 1 -- 172.21.15.116:0/524928029 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3501efcf0 con 0x7fd3500c3f70 2024-04-25T00:21:01.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:01.792+0000 7fd3549ed640 1 -- 172.21.15.116:0/524928029 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3501f02b0 con 0x7fd3500c3f70 2024-04-25T00:21:01.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:01.792+0000 7fd3549ed640 1 -- 172.21.15.116:0/524928029 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd350068a60 con 0x7fd3500c3f70 2024-04-25T00:21:01.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:01.792+0000 7fd33effd640 1 -- 172.21.15.116:0/524928029 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd34401a070 con 0x7fd3500c3f70 2024-04-25T00:21:01.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:01.792+0000 7fd33effd640 1 --2- 172.21.15.116:0/524928029 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd3300678c0 0x7fd330069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:01.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:01.792+0000 7fd33effd640 1 -- 172.21.15.116:0/524928029 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 9332+0+0 (secure 0 0 0) 0x7fd344085da0 con 0x7fd3500c3f70 2024-04-25T00:21:01.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:01.792+0000 7fd34e59c640 1 --2- 172.21.15.116:0/524928029 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd3300678c0 0x7fd330069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:01.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:01.796+0000 7fd34e59c640 1 --2- 172.21.15.116:0/524928029 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd3300678c0 0x7fd330069d80 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fd340000f70 tx=0x7fd340009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:01.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:01.796+0000 7fd33effd640 1 -- 172.21.15.116:0/524928029 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fd350148de0 con 0x7fd3500c3f70 2024-04-25T00:21:02.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:02.040+0000 7fd3549ed640 1 -- 172.21.15.116:0/524928029 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd3500c7870 con 0x7fd3500c3f70 2024-04-25T00:21:02.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:02.040+0000 7fd33effd640 1 -- 172.21.15.116:0/524928029 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 167 v167) ==== 95+0+6501 (secure 0 0 0) 0x7fd344005480 con 0x7fd3500c3f70 2024-04-25T00:21:02.043 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:21:02.044 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":167,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6125,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/321466844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":321466844},{"type":"v1","addr":"172.21.15.174:6837","nonce":321466844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":167}],"filesystems":[{"mdsmap":{"epoch":166,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:20:53.870063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7809},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7809":{"gid":7809,"name":"c","rank":0,"incarnation":163,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6835/980221971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":980221971},{"type":"v1","addr":"172.21.15.116:6835","nonce":980221971}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7809,"qdb_cluster":[7809]},"id":1},{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:20:13.935378+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7722":{"gid":7722,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7722,"qdb_cluster":[7722]},"id":2}]} 2024-04-25T00:21:02.044 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 167 2024-04-25T00:21:02.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:02.044+0000 7fd3549ed640 1 -- 172.21.15.116:0/524928029 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd3300678c0 msgr2=0x7fd330069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:02.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:02.044+0000 7fd3549ed640 1 --2- 172.21.15.116:0/524928029 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd3300678c0 0x7fd330069d80 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fd340000f70 tx=0x7fd340009040 comp rx=0 tx=0).stop 2024-04-25T00:21:02.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:02.044+0000 7fd3549ed640 1 -- 172.21.15.116:0/524928029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3500c3f70 msgr2=0x7fd350146110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:02.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:02.044+0000 7fd3549ed640 1 --2- 172.21.15.116:0/524928029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3500c3f70 0x7fd350146110 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7fd34400c3f0 tx=0x7fd34400c8b0 comp rx=0 tx=0).stop 2024-04-25T00:21:02.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:02.044+0000 7fd3549ed640 1 -- 172.21.15.116:0/524928029 shutdown_connections 2024-04-25T00:21:02.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:02.044+0000 7fd3549ed640 1 --2- 172.21.15.116:0/524928029 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd3300678c0 0x7fd330069d80 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:02.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:02.044+0000 7fd3549ed640 1 --2- 172.21.15.116:0/524928029 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd350152e40 0x7fd350146650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:02.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:02.044+0000 7fd3549ed640 1 --2- 172.21.15.116:0/524928029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd3500c3f70 0x7fd350146110 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:02.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:02.044+0000 7fd3549ed640 1 --2- 172.21.15.116:0/524928029 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd3500c3a20 0x7fd3501483e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:02.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:02.044+0000 7fd3549ed640 1 -- 172.21.15.116:0/524928029 >> 172.21.15.116:0/524928029 conn(0x7fd3500c9160 msgr2=0x7fd3500c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:21:02.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:02.044+0000 7fd3549ed640 1 -- 172.21.15.116:0/524928029 shutdown_connections 2024-04-25T00:21:02.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:02.044+0000 7fd3549ed640 1 -- 172.21.15.116:0/524928029 wait complete. 2024-04-25T00:21:02.063 INFO:tasks.mds_thrash.fs.[a]:mds.d reported in up:standby state 2024-04-25T00:21:02.063 INFO:tasks.mds_thrash.fs.[a]:waiting for 21 secs before thrashing 2024-04-25T00:21:09.874 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:21:09.877 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:21:23.064 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:21:23.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.200+0000 7fc8e307c640 1 Processor -- start 2024-04-25T00:21:23.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.200+0000 7fc8e307c640 1 -- start start 2024-04-25T00:21:23.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.200+0000 7fc8e307c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc8dc0c2070 0x7fc8dc0c2c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:23.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.200+0000 7fc8e307c640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fc8dc0c2580 0x7fc8dc0c31a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:23.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.200+0000 7fc8e307c640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8dc0a9010 con 0x7fc8dc14f850 2024-04-25T00:21:23.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.200+0000 7fc8e307c640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fc8dc073890 con 0x7fc8dc0c2070 2024-04-25T00:21:23.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.200+0000 7fc8e307c640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fc8dc0c50c0 con 0x7fc8dc0c2580 2024-04-25T00:21:23.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.200+0000 7fc8e1879640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc8dc0c2070 0x7fc8dc0c2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:23.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.200+0000 7fc8e207a640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fc8dc14f850 0x7fc8dc071440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:58032/0 (socket says 172.21.15.116:58032) 2024-04-25T00:21:23.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.200+0000 7fc8e207a640 1 -- 172.21.15.116:0/2418914320 learned_addr learned my addr 172.21.15.116:0/2418914320 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:21:23.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.200+0000 7fc8e287b640 1 --2- 172.21.15.116:0/2418914320 >> v2:172.21.15.174:3300/0 conn(0x7fc8dc0c2580 0x7fc8dc0c31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:23.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.200+0000 7fc8e1879640 1 -- 172.21.15.116:0/2418914320 >> v2:172.21.15.174:3300/0 conn(0x7fc8dc0c2580 msgr2=0x7fc8dc0c31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:21:23.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.200+0000 7fc8e1879640 1 --2- 172.21.15.116:0/2418914320 >> v2:172.21.15.174:3300/0 conn(0x7fc8dc0c2580 0x7fc8dc0c31a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:23.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.200+0000 7fc8e1879640 1 -- 172.21.15.116:0/2418914320 >> v1:172.21.15.116:6789/0 conn(0x7fc8dc14f850 legacy=0x7fc8dc071440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:23.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.200+0000 7fc8e1879640 1 -- 172.21.15.116:0/2418914320 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8dc152ff0 con 0x7fc8dc0c2070 2024-04-25T00:21:23.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.200+0000 7fc8e1078640 1 -- 172.21.15.116:0/2418914320 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3348524868 0 0) 0x7fc8dc0a9010 con 0x7fc8dc14f850 2024-04-25T00:21:23.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.204+0000 7fc8e1879640 1 --2- 172.21.15.116:0/2418914320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8dc0c2070 0x7fc8dc0c2c60 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc0c1f20 tx=0x7fc8cc01c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9747ce5a795c5c61 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:23.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.204+0000 7fc8e1078640 1 -- 172.21.15.116:0/2418914320 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8cc0b0070 con 0x7fc8dc0c2070 2024-04-25T00:21:23.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.204+0000 7fc8e1078640 1 -- 172.21.15.116:0/2418914320 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8cc0ab600 con 0x7fc8dc0c2070 2024-04-25T00:21:23.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.204+0000 7fc8e1078640 1 -- 172.21.15.116:0/2418914320 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8cc0ab9d0 con 0x7fc8dc0c2070 2024-04-25T00:21:23.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.204+0000 7fc8e307c640 1 -- 172.21.15.116:0/2418914320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8dc0c2070 msgr2=0x7fc8dc0c2c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:23.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.204+0000 7fc8e307c640 1 --2- 172.21.15.116:0/2418914320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8dc0c2070 0x7fc8dc0c2c60 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc0c1f20 tx=0x7fc8cc01c3f0 comp rx=0 tx=0).stop 2024-04-25T00:21:23.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.208+0000 7fc8e307c640 1 -- 172.21.15.116:0/2418914320 shutdown_connections 2024-04-25T00:21:23.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.208+0000 7fc8e307c640 1 --2- 172.21.15.116:0/2418914320 >> v2:172.21.15.174:3300/0 conn(0x7fc8dc0c2580 0x7fc8dc0c31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:23.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.208+0000 7fc8e307c640 1 --2- 172.21.15.116:0/2418914320 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8dc0c2070 0x7fc8dc0c2c60 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:23.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.208+0000 7fc8e307c640 1 -- 172.21.15.116:0/2418914320 >> 172.21.15.116:0/2418914320 conn(0x7fc8dc0a9a20 msgr2=0x7fc8dc0c4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:21:23.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.208+0000 7fc8e307c640 1 -- 172.21.15.116:0/2418914320 shutdown_connections 2024-04-25T00:21:23.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.208+0000 7fc8e307c640 1 -- 172.21.15.116:0/2418914320 wait complete. 2024-04-25T00:21:23.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.208+0000 7fc8e307c640 1 Processor -- start 2024-04-25T00:21:23.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.208+0000 7fc8e307c640 1 -- start start 2024-04-25T00:21:23.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.208+0000 7fc8e307c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8dc0c2070 0x7fc8dc154680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:23.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.208+0000 7fc8e307c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc8dc0c2580 0x7fc8dc1f0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:23.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.208+0000 7fc8e307c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc8dc14f850 0x7fc8dc0ab5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:23.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.208+0000 7fc8e307c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc8dc14fdb0 con 0x7fc8dc0c2070 2024-04-25T00:21:23.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.208+0000 7fc8e307c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc8dc14fc30 con 0x7fc8dc14f850 2024-04-25T00:21:23.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.208+0000 7fc8e307c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fc8dc06e450 con 0x7fc8dc0c2580 2024-04-25T00:21:23.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.208+0000 7fc8e207a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8dc0c2070 0x7fc8dc154680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:23.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.208+0000 7fc8e207a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8dc0c2070 0x7fc8dc154680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46258/0 (socket says 172.21.15.116:46258) 2024-04-25T00:21:23.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.208+0000 7fc8e207a640 1 -- 172.21.15.116:0/1748311896 learned_addr learned my addr 172.21.15.116:0/1748311896 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:21:23.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.208+0000 7fc8e287b640 1 --2- 172.21.15.116:0/1748311896 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc8dc14f850 0x7fc8dc0ab5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:23.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.208+0000 7fc8e1879640 1 --2- 172.21.15.116:0/1748311896 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc8dc0c2580 0x7fc8dc1f0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:23.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.208+0000 7fc8e207a640 1 -- 172.21.15.116:0/1748311896 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc8dc0c2580 msgr2=0x7fc8dc1f0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:23.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.208+0000 7fc8e207a640 1 --2- 172.21.15.116:0/1748311896 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc8dc0c2580 0x7fc8dc1f0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:23.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.208+0000 7fc8e207a640 1 -- 172.21.15.116:0/1748311896 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc8dc14f850 msgr2=0x7fc8dc0ab5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:23.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.208+0000 7fc8e207a640 1 --2- 172.21.15.116:0/1748311896 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc8dc14f850 0x7fc8dc0ab5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:23.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.208+0000 7fc8e207a640 1 -- 172.21.15.116:0/1748311896 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8dc1f1800 con 0x7fc8dc0c2070 2024-04-25T00:21:23.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.208+0000 7fc8e207a640 1 --2- 172.21.15.116:0/1748311896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8dc0c2070 0x7fc8dc154680 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7fc8d800c480 tx=0x7fc8d800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:23.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.208+0000 7fc8d2ffd640 1 -- 172.21.15.116:0/1748311896 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8d8018070 con 0x7fc8dc0c2070 2024-04-25T00:21:23.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.212+0000 7fc8d2ffd640 1 -- 172.21.15.116:0/1748311896 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8d8007500 con 0x7fc8dc0c2070 2024-04-25T00:21:23.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.212+0000 7fc8e307c640 1 -- 172.21.15.116:0/1748311896 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8dc1f1ae0 con 0x7fc8dc0c2070 2024-04-25T00:21:23.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.212+0000 7fc8d2ffd640 1 -- 172.21.15.116:0/1748311896 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8d8004a30 con 0x7fc8dc0c2070 2024-04-25T00:21:23.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.212+0000 7fc8e307c640 1 -- 172.21.15.116:0/1748311896 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8dc1572e0 con 0x7fc8dc0c2070 2024-04-25T00:21:23.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.212+0000 7fc8d2ffd640 1 -- 172.21.15.116:0/1748311896 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc8d801b070 con 0x7fc8dc0c2070 2024-04-25T00:21:23.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.212+0000 7fc8e307c640 1 -- 172.21.15.116:0/1748311896 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8dc04f510 con 0x7fc8dc0c2070 2024-04-25T00:21:23.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.212+0000 7fc8d2ffd640 1 --2- 172.21.15.116:0/1748311896 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc8c00678c0 0x7fc8c0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:23.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.212+0000 7fc8d2ffd640 1 -- 172.21.15.116:0/1748311896 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 9332+0+0 (secure 0 0 0) 0x7fc8d8089170 con 0x7fc8dc0c2070 2024-04-25T00:21:23.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.212+0000 7fc8e1879640 1 --2- 172.21.15.116:0/1748311896 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc8c00678c0 0x7fc8c0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:23.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.212+0000 7fc8e1879640 1 --2- 172.21.15.116:0/1748311896 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc8c00678c0 0x7fc8c0069d80 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc0ad000 tx=0x7fc8cc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:23.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.220+0000 7fc8d2ffd640 1 -- 172.21.15.116:0/1748311896 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fc8d8055960 con 0x7fc8dc0c2070 2024-04-25T00:21:23.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.468+0000 7fc8e307c640 1 -- 172.21.15.116:0/1748311896 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc8dc0c4c30 con 0x7fc8dc0c2070 2024-04-25T00:21:23.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.468+0000 7fc8d2ffd640 1 -- 172.21.15.116:0/1748311896 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 167 v167) ==== 95+0+6501 (secure 0 0 0) 0x7fc8d8055260 con 0x7fc8dc0c2070 2024-04-25T00:21:23.471 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:21:23.471 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":167,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6125,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/321466844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":321466844},{"type":"v1","addr":"172.21.15.174:6837","nonce":321466844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":167}],"filesystems":[{"mdsmap":{"epoch":166,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:20:53.870063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7809},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7809":{"gid":7809,"name":"c","rank":0,"incarnation":163,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6835/980221971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":980221971},{"type":"v1","addr":"172.21.15.116:6835","nonce":980221971}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7809,"qdb_cluster":[7809]},"id":1},{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:20:13.935378+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7722":{"gid":7722,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7722,"qdb_cluster":[7722]},"id":2}]} 2024-04-25T00:21:23.471 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 167 2024-04-25T00:21:23.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.472+0000 7fc8e307c640 1 -- 172.21.15.116:0/1748311896 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc8c00678c0 msgr2=0x7fc8c0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:23.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.472+0000 7fc8e307c640 1 --2- 172.21.15.116:0/1748311896 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc8c00678c0 0x7fc8c0069d80 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc0ad000 tx=0x7fc8cc008040 comp rx=0 tx=0).stop 2024-04-25T00:21:23.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.472+0000 7fc8e307c640 1 -- 172.21.15.116:0/1748311896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8dc0c2070 msgr2=0x7fc8dc154680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:23.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.472+0000 7fc8e307c640 1 --2- 172.21.15.116:0/1748311896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8dc0c2070 0x7fc8dc154680 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7fc8d800c480 tx=0x7fc8d800c940 comp rx=0 tx=0).stop 2024-04-25T00:21:23.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.472+0000 7fc8e307c640 1 -- 172.21.15.116:0/1748311896 shutdown_connections 2024-04-25T00:21:23.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.472+0000 7fc8e307c640 1 --2- 172.21.15.116:0/1748311896 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc8c00678c0 0x7fc8c0069d80 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:23.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.472+0000 7fc8e307c640 1 --2- 172.21.15.116:0/1748311896 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc8dc14f850 0x7fc8dc0ab5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:23.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.472+0000 7fc8e307c640 1 --2- 172.21.15.116:0/1748311896 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc8dc0c2580 0x7fc8dc1f0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:23.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.472+0000 7fc8e307c640 1 --2- 172.21.15.116:0/1748311896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8dc0c2070 0x7fc8dc154680 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:23.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.472+0000 7fc8e307c640 1 -- 172.21.15.116:0/1748311896 >> 172.21.15.116:0/1748311896 conn(0x7fc8dc0a9a20 msgr2=0x7fc8dc06eac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:21:23.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.472+0000 7fc8e307c640 1 -- 172.21.15.116:0/1748311896 shutdown_connections 2024-04-25T00:21:23.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.472+0000 7fc8e307c640 1 -- 172.21.15.116:0/1748311896 wait complete. 2024-04-25T00:21:23.495 INFO:tasks.mds_thrash.fs.[a]:kill mds.c (rank=0) 2024-04-25T00:21:23.495 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-25T00:21:23.495 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:21:23.506 INFO:tasks.ceph.mds.c:Stopped 2024-04-25T00:21:23.506 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:21:23.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.656+0000 7f90aa280640 1 Processor -- start 2024-04-25T00:21:23.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.656+0000 7f90aa280640 1 -- start start 2024-04-25T00:21:23.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.656+0000 7f90aa280640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f90a40a94a0 0x7f90a41538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:23.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.656+0000 7f90aa280640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f90a40a99f0 0x7f90a4153e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:23.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.656+0000 7f90aa280640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90a40c3910 con 0x7f90a4150d20 2024-04-25T00:21:23.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.656+0000 7f90aa280640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f90a40c16e0 con 0x7f90a40a99f0 2024-04-25T00:21:23.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.656+0000 7f90aa280640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f90a40c1400 con 0x7f90a40a94a0 2024-04-25T00:21:23.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.660+0000 7f90a881d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f90a40a99f0 0x7f90a4153e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:23.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.660+0000 7f90a881d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f90a40a99f0 0x7f90a4153e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46264/0 (socket says 172.21.15.116:46264) 2024-04-25T00:21:23.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.660+0000 7f90a881d640 1 -- 172.21.15.116:0/2055922662 learned_addr learned my addr 172.21.15.116:0/2055922662 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:21:23.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.660+0000 7f90a37fe640 1 --2- 172.21.15.116:0/2055922662 >> v2:172.21.15.174:3300/0 conn(0x7f90a40a94a0 0x7f90a41538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:23.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.660+0000 7f90a881d640 1 -- 172.21.15.116:0/2055922662 >> v2:172.21.15.174:3300/0 conn(0x7f90a40a94a0 msgr2=0x7f90a41538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:21:23.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.660+0000 7f90a881d640 1 --2- 172.21.15.116:0/2055922662 >> v2:172.21.15.174:3300/0 conn(0x7f90a40a94a0 0x7f90a41538e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:23.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.660+0000 7f90a881d640 1 -- 172.21.15.116:0/2055922662 >> v1:172.21.15.174:6789/0 conn(0x7f90a4150d20 legacy=0x7f90a40c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:21:23.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.660+0000 7f90a881d640 1 -- 172.21.15.116:0/2055922662 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90a406e660 con 0x7f90a40a99f0 2024-04-25T00:21:23.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.660+0000 7f90a881d640 1 --2- 172.21.15.116:0/2055922662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90a40a99f0 0x7f90a4153e20 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f90a4070710 tx=0x7f909401c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=198d8efcd31727ff server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:23.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.660+0000 7f90a2ffd640 1 -- 172.21.15.116:0/2055922662 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90940b0070 con 0x7f90a40a99f0 2024-04-25T00:21:23.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.660+0000 7f90a2ffd640 1 -- 172.21.15.116:0/2055922662 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90940ab600 con 0x7f90a40a99f0 2024-04-25T00:21:23.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.660+0000 7f90a2ffd640 1 -- 172.21.15.116:0/2055922662 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90940ab9d0 con 0x7f90a40a99f0 2024-04-25T00:21:23.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.660+0000 7f90aa280640 1 -- 172.21.15.116:0/2055922662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90a40a99f0 msgr2=0x7f90a4153e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:23.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.660+0000 7f90aa280640 1 --2- 172.21.15.116:0/2055922662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90a40a99f0 0x7f90a4153e20 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f90a4070710 tx=0x7f909401c4e0 comp rx=0 tx=0).stop 2024-04-25T00:21:23.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.660+0000 7f90aa280640 1 -- 172.21.15.116:0/2055922662 shutdown_connections 2024-04-25T00:21:23.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.660+0000 7f90aa280640 1 --2- 172.21.15.116:0/2055922662 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90a40a99f0 0x7f90a4153e20 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:23.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.660+0000 7f90aa280640 1 --2- 172.21.15.116:0/2055922662 >> v2:172.21.15.174:3300/0 conn(0x7f90a40a94a0 0x7f90a41538e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:23.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.660+0000 7f90aa280640 1 -- 172.21.15.116:0/2055922662 >> 172.21.15.116:0/2055922662 conn(0x7f90a40c3b00 msgr2=0x7f90a40c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:21:23.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.664+0000 7f90aa280640 1 -- 172.21.15.116:0/2055922662 shutdown_connections 2024-04-25T00:21:23.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.664+0000 7f90aa280640 1 -- 172.21.15.116:0/2055922662 wait complete. 2024-04-25T00:21:23.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.664+0000 7f90aa280640 1 Processor -- start 2024-04-25T00:21:23.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.664+0000 7f90aa280640 1 -- start start 2024-04-25T00:21:23.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.664+0000 7f90aa280640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90a40a94a0 0x7f90a41407f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:23.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.664+0000 7f90aa280640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f90a40a99f0 0x7f90a40c0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:23.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.664+0000 7f90aa280640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f90a4150d20 0x7f90a4143ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:23.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.664+0000 7f90aa280640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f90a41544e0 con 0x7f90a40a94a0 2024-04-25T00:21:23.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.664+0000 7f90aa280640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f90a4154360 con 0x7f90a40a99f0 2024-04-25T00:21:23.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.664+0000 7f90aa280640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f90a4160730 con 0x7f90a4150d20 2024-04-25T00:21:23.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.664+0000 7f90a3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90a40a94a0 0x7f90a41407f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:23.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.664+0000 7f90a37fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f90a40a99f0 0x7f90a40c0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:23.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.664+0000 7f90a3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90a40a94a0 0x7f90a41407f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46268/0 (socket says 172.21.15.116:46268) 2024-04-25T00:21:23.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.664+0000 7f90a3fff640 1 -- 172.21.15.116:0/1921887202 learned_addr learned my addr 172.21.15.116:0/1921887202 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:21:23.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.664+0000 7f90a881d640 1 --2- 172.21.15.116:0/1921887202 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f90a4150d20 0x7f90a4143ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:23.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.664+0000 7f90a3fff640 1 -- 172.21.15.116:0/1921887202 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f90a4150d20 msgr2=0x7f90a4143ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:23.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.664+0000 7f90a3fff640 1 --2- 172.21.15.116:0/1921887202 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f90a4150d20 0x7f90a4143ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:23.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.664+0000 7f90a3fff640 1 -- 172.21.15.116:0/1921887202 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f90a40a99f0 msgr2=0x7f90a40c0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:23.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.664+0000 7f90a3fff640 1 --2- 172.21.15.116:0/1921887202 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f90a40a99f0 0x7f90a40c0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:23.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.664+0000 7f90a3fff640 1 -- 172.21.15.116:0/1921887202 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90a4142220 con 0x7f90a40a94a0 2024-04-25T00:21:23.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.664+0000 7f90a3fff640 1 --2- 172.21.15.116:0/1921887202 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90a40a94a0 0x7f90a41407f0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f909c00c7c0 tx=0x7f909c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:23.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.664+0000 7f90a0ff9640 1 -- 172.21.15.116:0/1921887202 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f909c016020 con 0x7f90a40a94a0 2024-04-25T00:21:23.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.664+0000 7f90a0ff9640 1 -- 172.21.15.116:0/1921887202 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f909c00aa50 con 0x7f90a40a94a0 2024-04-25T00:21:23.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.664+0000 7f90a0ff9640 1 -- 172.21.15.116:0/1921887202 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f909c00ae40 con 0x7f90a40a94a0 2024-04-25T00:21:23.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.664+0000 7f90aa280640 1 -- 172.21.15.116:0/1921887202 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90a4140510 con 0x7f90a40a94a0 2024-04-25T00:21:23.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.664+0000 7f90aa280640 1 -- 172.21.15.116:0/1921887202 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90a4072990 con 0x7f90a40a94a0 2024-04-25T00:21:23.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.668+0000 7f90aa280640 1 -- 172.21.15.116:0/1921887202 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90a4072d70 con 0x7f90a40a94a0 2024-04-25T00:21:23.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.668+0000 7f90a0ff9640 1 -- 172.21.15.116:0/1921887202 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f909c019020 con 0x7f90a40a94a0 2024-04-25T00:21:23.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.668+0000 7f90a0ff9640 1 --2- 172.21.15.116:0/1921887202 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f90840678c0 0x7f9084069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:23.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.668+0000 7f90a0ff9640 1 -- 172.21.15.116:0/1921887202 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 9332+0+0 (secure 0 0 0) 0x7f909c086a60 con 0x7f90a40a94a0 2024-04-25T00:21:23.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.668+0000 7f90a37fe640 1 --2- 172.21.15.116:0/1921887202 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f90840678c0 0x7f9084069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:23.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.668+0000 7f90a37fe640 1 --2- 172.21.15.116:0/1921887202 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f90840678c0 0x7f9084069d80 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f90a4144590 tx=0x7f908c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:23.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.672+0000 7f90a0ff9640 1 -- 172.21.15.116:0/1921887202 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f90a4072d70 con 0x7f90a40a94a0 2024-04-25T00:21:23.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.872+0000 7f90aa280640 1 -- 172.21.15.116:0/1921887202 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f90a4068350 con 0x7f90a40a94a0 2024-04-25T00:21:23.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.872+0000 7f90a0ff9640 1 -- 172.21.15.116:0/1921887202 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 167 v167) ==== 95+0+6501 (secure 0 0 0) 0x7f909c020030 con 0x7f90a40a94a0 2024-04-25T00:21:23.876 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:21:23.876 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":167,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6125,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/321466844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":321466844},{"type":"v1","addr":"172.21.15.174:6837","nonce":321466844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":167}],"filesystems":[{"mdsmap":{"epoch":166,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:20:53.870063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7809},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7809":{"gid":7809,"name":"c","rank":0,"incarnation":163,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6835/980221971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":980221971},{"type":"v1","addr":"172.21.15.116:6835","nonce":980221971}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7809,"qdb_cluster":[7809]},"id":1},{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:20:13.935378+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7722":{"gid":7722,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7722,"qdb_cluster":[7722]},"id":2}]} 2024-04-25T00:21:23.877 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 167 2024-04-25T00:21:23.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.876+0000 7f90aa280640 1 -- 172.21.15.116:0/1921887202 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f90840678c0 msgr2=0x7f9084069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:23.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.876+0000 7f90aa280640 1 --2- 172.21.15.116:0/1921887202 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f90840678c0 0x7f9084069d80 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f90a4144590 tx=0x7f908c008040 comp rx=0 tx=0).stop 2024-04-25T00:21:23.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.876+0000 7f90aa280640 1 -- 172.21.15.116:0/1921887202 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90a40a94a0 msgr2=0x7f90a41407f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:23.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.876+0000 7f90aa280640 1 --2- 172.21.15.116:0/1921887202 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90a40a94a0 0x7f90a41407f0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f909c00c7c0 tx=0x7f909c00cc80 comp rx=0 tx=0).stop 2024-04-25T00:21:23.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.876+0000 7f90aa280640 1 -- 172.21.15.116:0/1921887202 shutdown_connections 2024-04-25T00:21:23.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.876+0000 7f90aa280640 1 --2- 172.21.15.116:0/1921887202 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f90840678c0 0x7f9084069d80 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:23.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.876+0000 7f90aa280640 1 --2- 172.21.15.116:0/1921887202 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f90a4150d20 0x7f90a4143ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:23.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.876+0000 7f90aa280640 1 --2- 172.21.15.116:0/1921887202 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f90a40a99f0 0x7f90a40c0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:23.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.876+0000 7f90aa280640 1 --2- 172.21.15.116:0/1921887202 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f90a40a94a0 0x7f90a41407f0 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:23.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.876+0000 7f90aa280640 1 -- 172.21.15.116:0/1921887202 >> 172.21.15.116:0/1921887202 conn(0x7f90a40c3b00 msgr2=0x7f90a4146db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:21:23.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.876+0000 7f90aa280640 1 -- 172.21.15.116:0/1921887202 shutdown_connections 2024-04-25T00:21:23.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:23.876+0000 7f90aa280640 1 -- 172.21.15.116:0/1921887202 wait complete. 2024-04-25T00:21:23.895 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:21:25.896 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:21:26.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.024+0000 7fd7657b2640 1 Processor -- start 2024-04-25T00:21:26.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.024+0000 7fd7657b2640 1 -- start start 2024-04-25T00:21:26.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.028+0000 7fd7657b2640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fd760150d20 0x7fd7600c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:26.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.028+0000 7fd7657b2640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd7600c2f50 0x7fd7601538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:26.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.028+0000 7fd7657b2640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7600c3910 con 0x7fd7600a94a0 2024-04-25T00:21:26.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.028+0000 7fd7657b2640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fd760154530 con 0x7fd760150d20 2024-04-25T00:21:26.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.028+0000 7fd7657b2640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fd7600c16e0 con 0x7fd7600c2f50 2024-04-25T00:21:26.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.028+0000 7fd75effd640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fd760150d20 0x7fd7600c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:26.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.028+0000 7fd75e7fc640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd7600c2f50 0x7fd7601538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:26.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.028+0000 7fd75f7fe640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fd7600a94a0 0x7fd760153e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:36266/0 (socket says 172.21.15.116:36266) 2024-04-25T00:21:26.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.028+0000 7fd75f7fe640 1 -- 172.21.15.116:0/1159470182 learned_addr learned my addr 172.21.15.116:0/1159470182 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:21:26.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.028+0000 7fd75dffb640 1 -- 172.21.15.116:0/1159470182 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1962838997 0 0) 0x7fd7600c3910 con 0x7fd7600a94a0 2024-04-25T00:21:26.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.028+0000 7fd75effd640 1 -- 172.21.15.116:0/1159470182 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd7600c2f50 msgr2=0x7fd7601538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:21:26.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.028+0000 7fd75effd640 1 --2- 172.21.15.116:0/1159470182 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd7600c2f50 0x7fd7601538e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:26.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.028+0000 7fd75effd640 1 -- 172.21.15.116:0/1159470182 >> v1:172.21.15.174:6789/0 conn(0x7fd7600a94a0 legacy=0x7fd760153e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:26.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.028+0000 7fd75effd640 1 -- 172.21.15.116:0/1159470182 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7600c1400 con 0x7fd760150d20 2024-04-25T00:21:26.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.028+0000 7fd75e7fc640 1 --2- 172.21.15.116:0/1159470182 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd7600c2f50 0x7fd7601538e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:21:26.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.032+0000 7fd75effd640 1 --2- 172.21.15.116:0/1159470182 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd760150d20 0x7fd7600c2a10 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fd7540097c0 tx=0x7fd75401c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9efbab15bd4e24fb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:26.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.036+0000 7fd75dffb640 1 -- 172.21.15.116:0/1159470182 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7540b1070 con 0x7fd760150d20 2024-04-25T00:21:26.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.036+0000 7fd75dffb640 1 -- 172.21.15.116:0/1159470182 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7540ac410 con 0x7fd760150d20 2024-04-25T00:21:26.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.036+0000 7fd75dffb640 1 -- 172.21.15.116:0/1159470182 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7540ac7e0 con 0x7fd760150d20 2024-04-25T00:21:26.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.036+0000 7fd7657b2640 1 -- 172.21.15.116:0/1159470182 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd760150d20 msgr2=0x7fd7600c2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:26.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.036+0000 7fd7657b2640 1 --2- 172.21.15.116:0/1159470182 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd760150d20 0x7fd7600c2a10 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fd7540097c0 tx=0x7fd75401c4a0 comp rx=0 tx=0).stop 2024-04-25T00:21:26.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.036+0000 7fd7657b2640 1 -- 172.21.15.116:0/1159470182 shutdown_connections 2024-04-25T00:21:26.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.036+0000 7fd7657b2640 1 --2- 172.21.15.116:0/1159470182 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd7600c2f50 0x7fd7601538e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:26.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.036+0000 7fd7657b2640 1 --2- 172.21.15.116:0/1159470182 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd760150d20 0x7fd7600c2a10 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:26.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.036+0000 7fd7657b2640 1 -- 172.21.15.116:0/1159470182 >> 172.21.15.116:0/1159470182 conn(0x7fd7600c3b00 msgr2=0x7fd7600c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:21:26.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.036+0000 7fd7657b2640 1 -- 172.21.15.116:0/1159470182 shutdown_connections 2024-04-25T00:21:26.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.036+0000 7fd7657b2640 1 -- 172.21.15.116:0/1159470182 wait complete. 2024-04-25T00:21:26.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.036+0000 7fd7657b2640 1 Processor -- start 2024-04-25T00:21:26.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.036+0000 7fd7657b2640 1 -- start start 2024-04-25T00:21:26.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.036+0000 7fd7657b2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd7600a94a0 0x7fd760141620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:26.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.036+0000 7fd7657b2640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd7600c2f50 0x7fd760154b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:26.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.036+0000 7fd7657b2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd760150d20 0x7fd760143110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:26.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.036+0000 7fd7657b2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd7601608b0 con 0x7fd7600a94a0 2024-04-25T00:21:26.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.036+0000 7fd7657b2640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd760160730 con 0x7fd760150d20 2024-04-25T00:21:26.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.036+0000 7fd7657b2640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fd760160a30 con 0x7fd7600c2f50 2024-04-25T00:21:26.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.036+0000 7fd75effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd7600a94a0 0x7fd760141620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:26.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.036+0000 7fd75effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd7600a94a0 0x7fd760141620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48806/0 (socket says 172.21.15.116:48806) 2024-04-25T00:21:26.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.036+0000 7fd75effd640 1 -- 172.21.15.116:0/4106014258 learned_addr learned my addr 172.21.15.116:0/4106014258 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:21:26.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.036+0000 7fd75e7fc640 1 --2- 172.21.15.116:0/4106014258 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd7600c2f50 0x7fd760154b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:26.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.036+0000 7fd75f7fe640 1 --2- 172.21.15.116:0/4106014258 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd760150d20 0x7fd760143110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:26.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.036+0000 7fd75effd640 1 -- 172.21.15.116:0/4106014258 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd7600c2f50 msgr2=0x7fd760154b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:26.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.036+0000 7fd75effd640 1 --2- 172.21.15.116:0/4106014258 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd7600c2f50 0x7fd760154b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:26.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.036+0000 7fd75effd640 1 -- 172.21.15.116:0/4106014258 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd760150d20 msgr2=0x7fd760143110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:26.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.036+0000 7fd75effd640 1 --2- 172.21.15.116:0/4106014258 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd760150d20 0x7fd760143110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:26.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.036+0000 7fd75effd640 1 -- 172.21.15.116:0/4106014258 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd760142350 con 0x7fd7600a94a0 2024-04-25T00:21:26.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.040+0000 7fd75effd640 1 --2- 172.21.15.116:0/4106014258 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd7600a94a0 0x7fd760141620 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7fd75401ab50 tx=0x7fd754019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:26.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.040+0000 7fd7477fe640 1 -- 172.21.15.116:0/4106014258 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7540b9020 con 0x7fd7600a94a0 2024-04-25T00:21:26.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.040+0000 7fd7477fe640 1 -- 172.21.15.116:0/4106014258 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7540acd70 con 0x7fd7600a94a0 2024-04-25T00:21:26.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.040+0000 7fd7657b2640 1 -- 172.21.15.116:0/4106014258 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd760142630 con 0x7fd7600a94a0 2024-04-25T00:21:26.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.040+0000 7fd7477fe640 1 -- 172.21.15.116:0/4106014258 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7540b1040 con 0x7fd7600a94a0 2024-04-25T00:21:26.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.040+0000 7fd7657b2640 1 -- 172.21.15.116:0/4106014258 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd760141b60 con 0x7fd7600a94a0 2024-04-25T00:21:26.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.040+0000 7fd7657b2640 1 -- 172.21.15.116:0/4106014258 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd760068350 con 0x7fd7600a94a0 2024-04-25T00:21:26.043 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.040+0000 7fd7477fe640 1 -- 172.21.15.116:0/4106014258 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd7540bc020 con 0x7fd7600a94a0 2024-04-25T00:21:26.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.040+0000 7fd7477fe640 1 --2- 172.21.15.116:0/4106014258 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd74c0678c0 0x7fd74c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:26.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.040+0000 7fd7477fe640 1 -- 172.21.15.116:0/4106014258 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 9332+0+0 (secure 0 0 0) 0x7fd754120310 con 0x7fd7600a94a0 2024-04-25T00:21:26.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.040+0000 7fd75e7fc640 1 --2- 172.21.15.116:0/4106014258 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd74c0678c0 0x7fd74c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:26.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.040+0000 7fd75e7fc640 1 --2- 172.21.15.116:0/4106014258 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd74c0678c0 0x7fd74c069d80 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fd750009580 tx=0x7fd750009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:26.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.044+0000 7fd7477fe640 1 -- 172.21.15.116:0/4106014258 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fd7540c3030 con 0x7fd7600a94a0 2024-04-25T00:21:26.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.240+0000 7fd7657b2640 1 -- 172.21.15.116:0/4106014258 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd7600c34f0 con 0x7fd7600a94a0 2024-04-25T00:21:26.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.240+0000 7fd7477fe640 1 -- 172.21.15.116:0/4106014258 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 167 v167) ==== 95+0+6501 (secure 0 0 0) 0x7fd754016090 con 0x7fd7600a94a0 2024-04-25T00:21:26.245 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:21:26.245 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":167,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6125,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/321466844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":321466844},{"type":"v1","addr":"172.21.15.174:6837","nonce":321466844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":167}],"filesystems":[{"mdsmap":{"epoch":166,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:20:53.870063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7809},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7809":{"gid":7809,"name":"c","rank":0,"incarnation":163,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6835/980221971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":980221971},{"type":"v1","addr":"172.21.15.116:6835","nonce":980221971}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7809,"qdb_cluster":[7809]},"id":1},{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:20:13.935378+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7722":{"gid":7722,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7722,"qdb_cluster":[7722]},"id":2}]} 2024-04-25T00:21:26.245 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 167 2024-04-25T00:21:26.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.244+0000 7fd7657b2640 1 -- 172.21.15.116:0/4106014258 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd74c0678c0 msgr2=0x7fd74c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:26.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.244+0000 7fd7657b2640 1 --2- 172.21.15.116:0/4106014258 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd74c0678c0 0x7fd74c069d80 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fd750009580 tx=0x7fd750009190 comp rx=0 tx=0).stop 2024-04-25T00:21:26.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.244+0000 7fd7657b2640 1 -- 172.21.15.116:0/4106014258 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd7600a94a0 msgr2=0x7fd760141620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:26.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.244+0000 7fd7657b2640 1 --2- 172.21.15.116:0/4106014258 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd7600a94a0 0x7fd760141620 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7fd75401ab50 tx=0x7fd754019240 comp rx=0 tx=0).stop 2024-04-25T00:21:26.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.244+0000 7fd7657b2640 1 -- 172.21.15.116:0/4106014258 shutdown_connections 2024-04-25T00:21:26.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.244+0000 7fd7657b2640 1 --2- 172.21.15.116:0/4106014258 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd74c0678c0 0x7fd74c069d80 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:26.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.244+0000 7fd7657b2640 1 --2- 172.21.15.116:0/4106014258 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd760150d20 0x7fd760143110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:26.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.244+0000 7fd7657b2640 1 --2- 172.21.15.116:0/4106014258 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd7600c2f50 0x7fd760154b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:26.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.244+0000 7fd7657b2640 1 --2- 172.21.15.116:0/4106014258 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd7600a94a0 0x7fd760141620 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:26.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.244+0000 7fd7657b2640 1 -- 172.21.15.116:0/4106014258 >> 172.21.15.116:0/4106014258 conn(0x7fd7600c3b00 msgr2=0x7fd7601470e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:21:26.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.244+0000 7fd7657b2640 1 -- 172.21.15.116:0/4106014258 shutdown_connections 2024-04-25T00:21:26.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:26.244+0000 7fd7657b2640 1 -- 172.21.15.116:0/4106014258 wait complete. 2024-04-25T00:21:26.264 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:21:28.265 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:21:28.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.392+0000 7fe8f7d04640 1 Processor -- start 2024-04-25T00:21:28.394 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.392+0000 7fe8f7d04640 1 -- start start 2024-04-25T00:21:28.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.392+0000 7fe8f7d04640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe8f0152ca0 0x7fe8f006b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:28.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.392+0000 7fe8f7d04640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fe8f006b750 0x7fe8f006f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:28.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.392+0000 7fe8f7d04640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8f01558e0 con 0x7fe8f0158d30 2024-04-25T00:21:28.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.392+0000 7fe8f7d04640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fe8f0155bc0 con 0x7fe8f006b750 2024-04-25T00:21:28.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.392+0000 7fe8f7d04640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe8f0155ea0 con 0x7fe8f0152ca0 2024-04-25T00:21:28.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.392+0000 7fe8f529f640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fe8f0158d30 0x7fe8f006f270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:34308/0 (socket says 172.21.15.116:34308) 2024-04-25T00:21:28.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.392+0000 7fe8f529f640 1 -- 172.21.15.116:0/1320256062 learned_addr learned my addr 172.21.15.116:0/1320256062 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:21:28.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.392+0000 7fe8f62a1640 1 --2- 172.21.15.116:0/1320256062 >> v2:172.21.15.174:3300/0 conn(0x7fe8f006b750 0x7fe8f006f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:28.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.392+0000 7fe8f5aa0640 1 --2- 172.21.15.116:0/1320256062 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe8f0152ca0 0x7fe8f006b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:28.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.392+0000 7fe8f4a9e640 1 -- 172.21.15.116:0/1320256062 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3649232777 0 0) 0x7fe8f01558e0 con 0x7fe8f0158d30 2024-04-25T00:21:28.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.392+0000 7fe8f4a9e640 1 -- 172.21.15.116:0/1320256062 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe8dc003530 con 0x7fe8f0158d30 2024-04-25T00:21:28.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.392+0000 7fe8f4a9e640 1 -- 172.21.15.116:0/1320256062 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 24571976 0 0) 0x7fe8dc003530 con 0x7fe8f0158d30 2024-04-25T00:21:28.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.392+0000 7fe8f4a9e640 1 -- 172.21.15.116:0/1320256062 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe8f0152ca0 msgr2=0x7fe8f006b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:21:28.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.392+0000 7fe8f4a9e640 1 --2- 172.21.15.116:0/1320256062 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe8f0152ca0 0x7fe8f006b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:28.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.392+0000 7fe8f4a9e640 1 -- 172.21.15.116:0/1320256062 >> v2:172.21.15.174:3300/0 conn(0x7fe8f006b750 msgr2=0x7fe8f006f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:21:28.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.392+0000 7fe8f4a9e640 1 --2- 172.21.15.116:0/1320256062 >> v2:172.21.15.174:3300/0 conn(0x7fe8f006b750 0x7fe8f006f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:28.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.392+0000 7fe8f4a9e640 1 -- 172.21.15.116:0/1320256062 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8f0156180 con 0x7fe8f0158d30 2024-04-25T00:21:28.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.392+0000 7fe8f4a9e640 1 -- 172.21.15.116:0/1320256062 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7fe8e0001680 con 0x7fe8f0158d30 2024-04-25T00:21:28.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.392+0000 7fe8f4a9e640 1 -- 172.21.15.116:0/1320256062 >> v1:172.21.15.116:6789/0 conn(0x7fe8f0158d30 legacy=0x7fe8f006f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:28.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.392+0000 7fe8f4a9e640 1 --2- 172.21.15.116:0/1320256062 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe8dc0039c0 0x7fe8dc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:28.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.392+0000 7fe8f4a9e640 1 --2- 172.21.15.116:0/1320256062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8dc004810 0x7fe8dc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:28.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.392+0000 7fe8f4a9e640 1 --2- 172.21.15.116:0/1320256062 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe8f0152ca0 0x7fe8dc0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:28.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.392+0000 7fe8f4a9e640 1 -- 172.21.15.116:0/1320256062 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe8f0155bc0 con 0x7fe8dc004810 2024-04-25T00:21:28.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.392+0000 7fe8f4a9e640 1 -- 172.21.15.116:0/1320256062 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe8f0155ea0 con 0x7fe8dc0039c0 2024-04-25T00:21:28.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.392+0000 7fe8f4a9e640 1 -- 172.21.15.116:0/1320256062 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe8dc00b9d0 con 0x7fe8f0152ca0 2024-04-25T00:21:28.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.392+0000 7fe8f529f640 1 --2- 172.21.15.116:0/1320256062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8dc004810 0x7fe8dc004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:28.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.392+0000 7fe8f5aa0640 1 --2- 172.21.15.116:0/1320256062 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe8dc0039c0 0x7fe8dc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:28.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.392+0000 7fe8f62a1640 1 --2- 172.21.15.116:0/1320256062 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe8f0152ca0 0x7fe8dc0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:28.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.392+0000 7fe8f529f640 1 -- 172.21.15.116:0/1320256062 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe8f0152ca0 msgr2=0x7fe8dc0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:21:28.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.392+0000 7fe8f529f640 1 --2- 172.21.15.116:0/1320256062 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe8f0152ca0 0x7fe8dc0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:28.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.392+0000 7fe8f529f640 1 -- 172.21.15.116:0/1320256062 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe8dc0039c0 msgr2=0x7fe8dc003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:21:28.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.392+0000 7fe8f529f640 1 --2- 172.21.15.116:0/1320256062 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe8dc0039c0 0x7fe8dc003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:28.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.392+0000 7fe8f529f640 1 -- 172.21.15.116:0/1320256062 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8dc003530 con 0x7fe8dc004810 2024-04-25T00:21:28.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.392+0000 7fe8f62a1640 1 --2- 172.21.15.116:0/1320256062 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe8f0152ca0 0x7fe8dc0051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:21:28.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.396+0000 7fe8f529f640 1 --2- 172.21.15.116:0/1320256062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8dc004810 0x7fe8dc004c90 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7fe8e000c270 tx=0x7fe8e001cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5865b3859844965c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:28.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.396+0000 7fe8f4a9e640 1 -- 172.21.15.116:0/1320256062 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8e0007d40 con 0x7fe8dc004810 2024-04-25T00:21:28.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.396+0000 7fe8f4a9e640 1 -- 172.21.15.116:0/1320256062 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8e00a9020 con 0x7fe8dc004810 2024-04-25T00:21:28.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.396+0000 7fe8f4a9e640 1 -- 172.21.15.116:0/1320256062 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8e0005b00 con 0x7fe8dc004810 2024-04-25T00:21:28.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.396+0000 7fe8f7d04640 1 -- 172.21.15.116:0/1320256062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8dc004810 msgr2=0x7fe8dc004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:28.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.396+0000 7fe8f7d04640 1 --2- 172.21.15.116:0/1320256062 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8dc004810 0x7fe8dc004c90 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7fe8e000c270 tx=0x7fe8e001cca0 comp rx=0 tx=0).stop 2024-04-25T00:21:28.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.396+0000 7fe8f62a1640 1 -- 172.21.15.116:0/1320256062 reap_dead start 2024-04-25T00:21:28.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.396+0000 7fe8f7d04640 1 -- 172.21.15.116:0/1320256062 shutdown_connections 2024-04-25T00:21:28.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.396+0000 7fe8f7d04640 1 -- 172.21.15.116:0/1320256062 >> 172.21.15.116:0/1320256062 conn(0x7fe8f00c7fa0 msgr2=0x7fe8f00c83b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:21:28.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.396+0000 7fe8f7d04640 1 -- 172.21.15.116:0/1320256062 shutdown_connections 2024-04-25T00:21:28.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.396+0000 7fe8f7d04640 1 -- 172.21.15.116:0/1320256062 wait complete. 2024-04-25T00:21:28.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.400+0000 7fe8f7d04640 1 Processor -- start 2024-04-25T00:21:28.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.400+0000 7fe8f7d04640 1 -- start start 2024-04-25T00:21:28.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.400+0000 7fe8f7d04640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe8dc004810 0x7fe8f00c76a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:28.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.400+0000 7fe8f7d04640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8f015c890 0x7fe8f015cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:28.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.400+0000 7fe8f7d04640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe8f014c820 0x7fe8f014cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:28.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.400+0000 7fe8f7d04640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe8f0153200 con 0x7fe8f015c890 2024-04-25T00:21:28.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.400+0000 7fe8f7d04640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe8f0153080 con 0x7fe8dc004810 2024-04-25T00:21:28.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.400+0000 7fe8f7d04640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe8f01608d0 con 0x7fe8f014c820 2024-04-25T00:21:28.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.400+0000 7fe8f529f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8f015c890 0x7fe8f015cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:28.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.400+0000 7fe8f529f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8f015c890 0x7fe8f015cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48812/0 (socket says 172.21.15.116:48812) 2024-04-25T00:21:28.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.400+0000 7fe8f529f640 1 -- 172.21.15.116:0/3331611600 learned_addr learned my addr 172.21.15.116:0/3331611600 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:21:28.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.400+0000 7fe8f5aa0640 1 --2- 172.21.15.116:0/3331611600 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe8dc004810 0x7fe8f00c76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:28.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.400+0000 7fe8f62a1640 1 --2- 172.21.15.116:0/3331611600 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe8f014c820 0x7fe8f014cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:28.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.400+0000 7fe8f529f640 1 -- 172.21.15.116:0/3331611600 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe8f014c820 msgr2=0x7fe8f014cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:28.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.400+0000 7fe8f529f640 1 --2- 172.21.15.116:0/3331611600 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe8f014c820 0x7fe8f014cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:28.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.400+0000 7fe8f529f640 1 -- 172.21.15.116:0/3331611600 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe8dc004810 msgr2=0x7fe8f00c76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:28.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.400+0000 7fe8f529f640 1 --2- 172.21.15.116:0/3331611600 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe8dc004810 0x7fe8f00c76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:28.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.400+0000 7fe8f529f640 1 -- 172.21.15.116:0/3331611600 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8f014d1e0 con 0x7fe8f015c890 2024-04-25T00:21:28.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.400+0000 7fe8f529f640 1 --2- 172.21.15.116:0/3331611600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8f015c890 0x7fe8f015cd10 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7fe8e00ab040 tx=0x7fe8dc005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:28.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.400+0000 7fe8e67fc640 1 -- 172.21.15.116:0/3331611600 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8e00049f0 con 0x7fe8f015c890 2024-04-25T00:21:28.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.400+0000 7fe8e67fc640 1 -- 172.21.15.116:0/3331611600 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8e0004b90 con 0x7fe8f015c890 2024-04-25T00:21:28.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.400+0000 7fe8f7d04640 1 -- 172.21.15.116:0/3331611600 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8f015e610 con 0x7fe8f015c890 2024-04-25T00:21:28.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.400+0000 7fe8e67fc640 1 -- 172.21.15.116:0/3331611600 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8e00070d0 con 0x7fe8f015c890 2024-04-25T00:21:28.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.400+0000 7fe8f7d04640 1 -- 172.21.15.116:0/3331611600 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8f015ebd0 con 0x7fe8f015c890 2024-04-25T00:21:28.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.400+0000 7fe8f7d04640 1 -- 172.21.15.116:0/3331611600 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8f0068a60 con 0x7fe8f015c890 2024-04-25T00:21:28.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.400+0000 7fe8e67fc640 1 -- 172.21.15.116:0/3331611600 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe8e0007270 con 0x7fe8f015c890 2024-04-25T00:21:28.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.400+0000 7fe8e67fc640 1 --2- 172.21.15.116:0/3331611600 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe8d40678f0 0x7fe8d4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:28.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.400+0000 7fe8e67fc640 1 -- 172.21.15.116:0/3331611600 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 9332+0+0 (secure 0 0 0) 0x7fe8e0120080 con 0x7fe8f015c890 2024-04-25T00:21:28.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.404+0000 7fe8f5aa0640 1 --2- 172.21.15.116:0/3331611600 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe8d40678f0 0x7fe8d4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:28.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.404+0000 7fe8f5aa0640 1 --2- 172.21.15.116:0/3331611600 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe8d40678f0 0x7fe8d4069db0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec009980 tx=0x7fe8ec008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:28.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.408+0000 7fe8e67fc640 1 -- 172.21.15.116:0/3331611600 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fe8e00169b0 con 0x7fe8f015c890 2024-04-25T00:21:28.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.600+0000 7fe8f7d04640 1 -- 172.21.15.116:0/3331611600 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe8f0072de0 con 0x7fe8f015c890 2024-04-25T00:21:28.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.600+0000 7fe8e67fc640 1 -- 172.21.15.116:0/3331611600 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 167 v167) ==== 95+0+6501 (secure 0 0 0) 0x7fe8e00efd50 con 0x7fe8f015c890 2024-04-25T00:21:28.602 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:21:28.602 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":167,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6125,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/321466844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":321466844},{"type":"v1","addr":"172.21.15.174:6837","nonce":321466844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":167}],"filesystems":[{"mdsmap":{"epoch":166,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:20:53.870063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7809},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7809":{"gid":7809,"name":"c","rank":0,"incarnation":163,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6835/980221971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":980221971},{"type":"v1","addr":"172.21.15.116:6835","nonce":980221971}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7809,"qdb_cluster":[7809]},"id":1},{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:20:13.935378+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7722":{"gid":7722,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7722,"qdb_cluster":[7722]},"id":2}]} 2024-04-25T00:21:28.603 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 167 2024-04-25T00:21:28.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.600+0000 7fe8f7d04640 1 -- 172.21.15.116:0/3331611600 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe8d40678f0 msgr2=0x7fe8d4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:28.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.600+0000 7fe8f7d04640 1 --2- 172.21.15.116:0/3331611600 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe8d40678f0 0x7fe8d4069db0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec009980 tx=0x7fe8ec008040 comp rx=0 tx=0).stop 2024-04-25T00:21:28.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.600+0000 7fe8f7d04640 1 -- 172.21.15.116:0/3331611600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8f015c890 msgr2=0x7fe8f015cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:28.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.600+0000 7fe8f7d04640 1 --2- 172.21.15.116:0/3331611600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8f015c890 0x7fe8f015cd10 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7fe8e00ab040 tx=0x7fe8dc005840 comp rx=0 tx=0).stop 2024-04-25T00:21:28.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.600+0000 7fe8f7d04640 1 -- 172.21.15.116:0/3331611600 shutdown_connections 2024-04-25T00:21:28.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.600+0000 7fe8f7d04640 1 --2- 172.21.15.116:0/3331611600 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe8d40678f0 0x7fe8d4069db0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:28.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.600+0000 7fe8f7d04640 1 --2- 172.21.15.116:0/3331611600 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe8f014c820 0x7fe8f014cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:28.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.600+0000 7fe8f7d04640 1 --2- 172.21.15.116:0/3331611600 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe8f015c890 0x7fe8f015cd10 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:28.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.600+0000 7fe8f7d04640 1 --2- 172.21.15.116:0/3331611600 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe8dc004810 0x7fe8f00c76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:28.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.600+0000 7fe8f7d04640 1 -- 172.21.15.116:0/3331611600 >> 172.21.15.116:0/3331611600 conn(0x7fe8f00c7fa0 msgr2=0x7fe8f0157fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:21:28.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.600+0000 7fe8f7d04640 1 -- 172.21.15.116:0/3331611600 shutdown_connections 2024-04-25T00:21:28.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:28.600+0000 7fe8f7d04640 1 -- 172.21.15.116:0/3331611600 wait complete. 2024-04-25T00:21:28.620 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:21:30.621 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:21:30.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.752+0000 7f2db1a29640 1 Processor -- start 2024-04-25T00:21:30.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.752+0000 7f2db1a29640 1 -- start start 2024-04-25T00:21:30.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.752+0000 7f2db1a29640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f2dac0a94a0 0x7f2dac1538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:30.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.752+0000 7f2db1a29640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2dac0a99f0 0x7f2dac153e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:30.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.752+0000 7f2db1a29640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2dac0c3910 con 0x7f2dac150d20 2024-04-25T00:21:30.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.752+0000 7f2db1a29640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f2dac0c16e0 con 0x7f2dac0a99f0 2024-04-25T00:21:30.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.752+0000 7f2db1a29640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f2dac0c1400 con 0x7f2dac0a94a0 2024-04-25T00:21:30.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.752+0000 7f2daaffd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f2dac150d20 0x7f2dac0c2a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:34312/0 (socket says 172.21.15.116:34312) 2024-04-25T00:21:30.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.752+0000 7f2daaffd640 1 -- 172.21.15.116:0/2243792162 learned_addr learned my addr 172.21.15.116:0/2243792162 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:21:30.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.752+0000 7f2dab7fe640 1 --2- 172.21.15.116:0/2243792162 >> v2:172.21.15.116:3300/0 conn(0x7f2dac0a99f0 0x7f2dac153e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:30.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.752+0000 7f2daa7fc640 1 --2- 172.21.15.116:0/2243792162 >> v2:172.21.15.174:3300/0 conn(0x7f2dac0a94a0 0x7f2dac1538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:30.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.752+0000 7f2dab7fe640 1 -- 172.21.15.116:0/2243792162 >> v2:172.21.15.174:3300/0 conn(0x7f2dac0a94a0 msgr2=0x7f2dac1538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:21:30.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.752+0000 7f2dab7fe640 1 --2- 172.21.15.116:0/2243792162 >> v2:172.21.15.174:3300/0 conn(0x7f2dac0a94a0 0x7f2dac1538e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:30.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.752+0000 7f2dab7fe640 1 -- 172.21.15.116:0/2243792162 >> v1:172.21.15.116:6789/0 conn(0x7f2dac150d20 legacy=0x7f2dac0c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:30.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.752+0000 7f2dab7fe640 1 -- 172.21.15.116:0/2243792162 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dac06e660 con 0x7f2dac0a99f0 2024-04-25T00:21:30.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.752+0000 7f2da9ffb640 1 -- 172.21.15.116:0/2243792162 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1969059854 0 0) 0x7f2dac0c3910 con 0x7f2dac150d20 2024-04-25T00:21:30.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.752+0000 7f2dab7fe640 1 --2- 172.21.15.116:0/2243792162 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2dac0a99f0 0x7f2dac153e20 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f2dac070710 tx=0x7f2da401c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9562af782e53c71c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:30.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.756+0000 7f2da9ffb640 1 -- 172.21.15.116:0/2243792162 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da40b2070 con 0x7f2dac0a99f0 2024-04-25T00:21:30.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.756+0000 7f2da9ffb640 1 -- 172.21.15.116:0/2243792162 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2da40a57f0 con 0x7f2dac0a99f0 2024-04-25T00:21:30.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.756+0000 7f2da9ffb640 1 -- 172.21.15.116:0/2243792162 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da40a5bc0 con 0x7f2dac0a99f0 2024-04-25T00:21:30.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.756+0000 7f2db1a29640 1 -- 172.21.15.116:0/2243792162 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2dac0a99f0 msgr2=0x7f2dac153e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:30.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.756+0000 7f2db1a29640 1 --2- 172.21.15.116:0/2243792162 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2dac0a99f0 0x7f2dac153e20 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f2dac070710 tx=0x7f2da401c930 comp rx=0 tx=0).stop 2024-04-25T00:21:30.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.756+0000 7f2db1a29640 1 -- 172.21.15.116:0/2243792162 shutdown_connections 2024-04-25T00:21:30.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.756+0000 7f2db1a29640 1 --2- 172.21.15.116:0/2243792162 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2dac0a99f0 0x7f2dac153e20 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:30.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.756+0000 7f2db1a29640 1 --2- 172.21.15.116:0/2243792162 >> v2:172.21.15.174:3300/0 conn(0x7f2dac0a94a0 0x7f2dac1538e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:30.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.756+0000 7f2db1a29640 1 -- 172.21.15.116:0/2243792162 >> 172.21.15.116:0/2243792162 conn(0x7f2dac0c3b00 msgr2=0x7f2dac0c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:21:30.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.756+0000 7f2db1a29640 1 -- 172.21.15.116:0/2243792162 shutdown_connections 2024-04-25T00:21:30.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.756+0000 7f2db1a29640 1 -- 172.21.15.116:0/2243792162 wait complete. 2024-04-25T00:21:30.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.756+0000 7f2db1a29640 1 Processor -- start 2024-04-25T00:21:30.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.756+0000 7f2db1a29640 1 -- start start 2024-04-25T00:21:30.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.756+0000 7f2db1a29640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2dac0a94a0 0x7f2dac1efe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:30.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.756+0000 7f2db1a29640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2dac0a99f0 0x7f2dac143ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:30.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.756+0000 7f2db1a29640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2dac150d20 0x7f2dac141d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:30.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.756+0000 7f2db1a29640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2dac1544e0 con 0x7f2dac0a94a0 2024-04-25T00:21:30.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.756+0000 7f2db1a29640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2dac154360 con 0x7f2dac150d20 2024-04-25T00:21:30.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.756+0000 7f2db1a29640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f2dac160730 con 0x7f2dac0a99f0 2024-04-25T00:21:30.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.756+0000 7f2daaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2dac0a94a0 0x7f2dac1efe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:30.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.756+0000 7f2daaffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2dac0a94a0 0x7f2dac1efe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48832/0 (socket says 172.21.15.116:48832) 2024-04-25T00:21:30.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.756+0000 7f2daaffd640 1 -- 172.21.15.116:0/2089648173 learned_addr learned my addr 172.21.15.116:0/2089648173 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:21:30.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.756+0000 7f2daa7fc640 1 --2- 172.21.15.116:0/2089648173 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2dac0a99f0 0x7f2dac143ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:30.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.756+0000 7f2dab7fe640 1 --2- 172.21.15.116:0/2089648173 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2dac150d20 0x7f2dac141d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:30.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.756+0000 7f2daaffd640 1 -- 172.21.15.116:0/2089648173 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2dac0a99f0 msgr2=0x7f2dac143ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:30.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.756+0000 7f2daaffd640 1 --2- 172.21.15.116:0/2089648173 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2dac0a99f0 0x7f2dac143ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:30.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.756+0000 7f2daaffd640 1 -- 172.21.15.116:0/2089648173 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2dac150d20 msgr2=0x7f2dac141d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:30.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.756+0000 7f2daaffd640 1 --2- 172.21.15.116:0/2089648173 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2dac150d20 0x7f2dac141d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:30.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.756+0000 7f2daaffd640 1 -- 172.21.15.116:0/2089648173 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dac15d2e0 con 0x7f2dac0a94a0 2024-04-25T00:21:30.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.756+0000 7f2daa7fc640 1 --2- 172.21.15.116:0/2089648173 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2dac0a99f0 0x7f2dac143ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:21:30.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.756+0000 7f2daaffd640 1 --2- 172.21.15.116:0/2089648173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2dac0a94a0 0x7f2dac1efe50 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f2da00011f0 tx=0x7f2da000c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:30.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.756+0000 7f2d93fff640 1 -- 172.21.15.116:0/2089648173 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da0018070 con 0x7f2dac0a94a0 2024-04-25T00:21:30.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.756+0000 7f2d93fff640 1 -- 172.21.15.116:0/2089648173 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2da00074e0 con 0x7f2dac0a94a0 2024-04-25T00:21:30.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.756+0000 7f2db1a29640 1 -- 172.21.15.116:0/2089648173 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2dac15d5c0 con 0x7f2dac0a94a0 2024-04-25T00:21:30.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.756+0000 7f2d93fff640 1 -- 172.21.15.116:0/2089648173 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da0004a10 con 0x7f2dac0a94a0 2024-04-25T00:21:30.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.756+0000 7f2db1a29640 1 -- 172.21.15.116:0/2089648173 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2dac144010 con 0x7f2dac0a94a0 2024-04-25T00:21:30.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.760+0000 7f2db1a29640 1 -- 172.21.15.116:0/2089648173 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2dac068350 con 0x7f2dac0a94a0 2024-04-25T00:21:30.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.760+0000 7f2d93fff640 1 -- 172.21.15.116:0/2089648173 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2da001b070 con 0x7f2dac0a94a0 2024-04-25T00:21:30.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.760+0000 7f2d93fff640 1 --2- 172.21.15.116:0/2089648173 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2d980678c0 0x7f2d98069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:30.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.760+0000 7f2d93fff640 1 -- 172.21.15.116:0/2089648173 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 9332+0+0 (secure 0 0 0) 0x7f2da0086180 con 0x7f2dac0a94a0 2024-04-25T00:21:30.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.760+0000 7f2daa7fc640 1 --2- 172.21.15.116:0/2089648173 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2d980678c0 0x7f2d98069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:30.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.760+0000 7f2daa7fc640 1 --2- 172.21.15.116:0/2089648173 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2d980678c0 0x7f2d98069d80 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f2dac144590 tx=0x7f2d9c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:30.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.764+0000 7f2d93fff640 1 -- 172.21.15.116:0/2089648173 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f2da0055f50 con 0x7f2dac0a94a0 2024-04-25T00:21:30.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.960+0000 7f2db1a29640 1 -- 172.21.15.116:0/2089648173 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2dac0c34f0 con 0x7f2dac0a94a0 2024-04-25T00:21:30.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.960+0000 7f2d93fff640 1 -- 172.21.15.116:0/2089648173 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 167 v167) ==== 95+0+6501 (secure 0 0 0) 0x7f2da0055850 con 0x7f2dac0a94a0 2024-04-25T00:21:30.964 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:21:30.964 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":167,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6125,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/321466844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":321466844},{"type":"v1","addr":"172.21.15.174:6837","nonce":321466844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":167}],"filesystems":[{"mdsmap":{"epoch":166,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:20:53.870063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7809},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7809":{"gid":7809,"name":"c","rank":0,"incarnation":163,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6835/980221971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":980221971},{"type":"v1","addr":"172.21.15.116:6835","nonce":980221971}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7809,"qdb_cluster":[7809]},"id":1},{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:20:13.935378+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7722":{"gid":7722,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7722,"qdb_cluster":[7722]},"id":2}]} 2024-04-25T00:21:30.964 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 167 2024-04-25T00:21:30.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.964+0000 7f2db1a29640 1 -- 172.21.15.116:0/2089648173 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2d980678c0 msgr2=0x7f2d98069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:30.966 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.964+0000 7f2db1a29640 1 --2- 172.21.15.116:0/2089648173 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2d980678c0 0x7f2d98069d80 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f2dac144590 tx=0x7f2d9c008040 comp rx=0 tx=0).stop 2024-04-25T00:21:30.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.964+0000 7f2db1a29640 1 -- 172.21.15.116:0/2089648173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2dac0a94a0 msgr2=0x7f2dac1efe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:30.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.964+0000 7f2db1a29640 1 --2- 172.21.15.116:0/2089648173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2dac0a94a0 0x7f2dac1efe50 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f2da00011f0 tx=0x7f2da000c9b0 comp rx=0 tx=0).stop 2024-04-25T00:21:30.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.964+0000 7f2db1a29640 1 -- 172.21.15.116:0/2089648173 shutdown_connections 2024-04-25T00:21:30.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.964+0000 7f2db1a29640 1 --2- 172.21.15.116:0/2089648173 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2d980678c0 0x7f2d98069d80 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:30.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.964+0000 7f2db1a29640 1 --2- 172.21.15.116:0/2089648173 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2dac150d20 0x7f2dac141d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:30.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.964+0000 7f2db1a29640 1 --2- 172.21.15.116:0/2089648173 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2dac0a99f0 0x7f2dac143ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:30.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.964+0000 7f2db1a29640 1 --2- 172.21.15.116:0/2089648173 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2dac0a94a0 0x7f2dac1efe50 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:30.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.964+0000 7f2db1a29640 1 -- 172.21.15.116:0/2089648173 >> 172.21.15.116:0/2089648173 conn(0x7f2dac0c3b00 msgr2=0x7f2dac0aca10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:21:30.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.964+0000 7f2db1a29640 1 -- 172.21.15.116:0/2089648173 shutdown_connections 2024-04-25T00:21:30.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:30.964+0000 7f2db1a29640 1 -- 172.21.15.116:0/2089648173 wait complete. 2024-04-25T00:21:30.982 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:21:32.983 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:21:33.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.108+0000 7fce2a373640 1 Processor -- start 2024-04-25T00:21:33.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.112+0000 7fce2a373640 1 -- start start 2024-04-25T00:21:33.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.112+0000 7fce2a373640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fce24150d20 0x7fce240c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:33.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.112+0000 7fce2a373640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fce240c2f50 0x7fce241538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:33.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.112+0000 7fce2a373640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce240c3910 con 0x7fce240a94a0 2024-04-25T00:21:33.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.112+0000 7fce2a373640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fce24154530 con 0x7fce240c2f50 2024-04-25T00:21:33.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.112+0000 7fce2a373640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fce240c16e0 con 0x7fce24150d20 2024-04-25T00:21:33.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.112+0000 7fce237fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fce240c2f50 0x7fce241538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:33.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.112+0000 7fce28910640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fce240a94a0 0x7fce24153e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:34318/0 (socket says 172.21.15.116:34318) 2024-04-25T00:21:33.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.112+0000 7fce237fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fce240c2f50 0x7fce241538e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48838/0 (socket says 172.21.15.116:48838) 2024-04-25T00:21:33.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.112+0000 7fce28910640 1 -- 172.21.15.116:0/2282363350 learned_addr learned my addr 172.21.15.116:0/2282363350 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:21:33.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.112+0000 7fce23fff640 1 --2- 172.21.15.116:0/2282363350 >> v2:172.21.15.174:3300/0 conn(0x7fce24150d20 0x7fce240c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:33.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.112+0000 7fce237fe640 1 -- 172.21.15.116:0/2282363350 >> v2:172.21.15.174:3300/0 conn(0x7fce24150d20 msgr2=0x7fce240c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:21:33.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.112+0000 7fce237fe640 1 --2- 172.21.15.116:0/2282363350 >> v2:172.21.15.174:3300/0 conn(0x7fce24150d20 0x7fce240c2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:33.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.112+0000 7fce237fe640 1 -- 172.21.15.116:0/2282363350 >> v1:172.21.15.116:6789/0 conn(0x7fce240a94a0 legacy=0x7fce24153e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:33.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.112+0000 7fce237fe640 1 -- 172.21.15.116:0/2282363350 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce240c1400 con 0x7fce240c2f50 2024-04-25T00:21:33.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.112+0000 7fce22ffd640 1 -- 172.21.15.116:0/2282363350 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3286967493 0 0) 0x7fce240c3910 con 0x7fce240a94a0 2024-04-25T00:21:33.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.112+0000 7fce23fff640 1 --2- 172.21.15.116:0/2282363350 >> v2:172.21.15.174:3300/0 conn(0x7fce24150d20 0x7fce240c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:21:33.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.112+0000 7fce237fe640 1 --2- 172.21.15.116:0/2282363350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce240c2f50 0x7fce241538e0 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7fce24073550 tx=0x7fce1c01c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b00fcf187b8e285e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:33.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.112+0000 7fce22ffd640 1 -- 172.21.15.116:0/2282363350 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce1c0b2070 con 0x7fce240c2f50 2024-04-25T00:21:33.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.112+0000 7fce22ffd640 1 -- 172.21.15.116:0/2282363350 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce1c0a5900 con 0x7fce240c2f50 2024-04-25T00:21:33.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.112+0000 7fce22ffd640 1 -- 172.21.15.116:0/2282363350 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce1c0a5cd0 con 0x7fce240c2f50 2024-04-25T00:21:33.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.116+0000 7fce2a373640 1 -- 172.21.15.116:0/2282363350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce240c2f50 msgr2=0x7fce241538e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:33.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.116+0000 7fce2a373640 1 --2- 172.21.15.116:0/2282363350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce240c2f50 0x7fce241538e0 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7fce24073550 tx=0x7fce1c01c930 comp rx=0 tx=0).stop 2024-04-25T00:21:33.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.116+0000 7fce2a373640 1 -- 172.21.15.116:0/2282363350 shutdown_connections 2024-04-25T00:21:33.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.116+0000 7fce2a373640 1 --2- 172.21.15.116:0/2282363350 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce240c2f50 0x7fce241538e0 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:33.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.116+0000 7fce2a373640 1 --2- 172.21.15.116:0/2282363350 >> v2:172.21.15.174:3300/0 conn(0x7fce24150d20 0x7fce240c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:33.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.116+0000 7fce2a373640 1 -- 172.21.15.116:0/2282363350 >> 172.21.15.116:0/2282363350 conn(0x7fce240c3b00 msgr2=0x7fce240c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:21:33.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.116+0000 7fce2a373640 1 -- 172.21.15.116:0/2282363350 shutdown_connections 2024-04-25T00:21:33.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.116+0000 7fce2a373640 1 -- 172.21.15.116:0/2282363350 wait complete. 2024-04-25T00:21:33.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.116+0000 7fce2a373640 1 Processor -- start 2024-04-25T00:21:33.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.116+0000 7fce2a373640 1 -- start start 2024-04-25T00:21:33.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.116+0000 7fce2a373640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fce240a94a0 0x7fce240c4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:33.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.116+0000 7fce2a373640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fce240c2f50 0x7fce240726d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:33.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.116+0000 7fce2a373640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce24150d20 0x7fce240c0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:33.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.116+0000 7fce2a373640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fce241608b0 con 0x7fce24150d20 2024-04-25T00:21:33.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.116+0000 7fce2a373640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fce24160730 con 0x7fce240a94a0 2024-04-25T00:21:33.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.116+0000 7fce2a373640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fce24160a30 con 0x7fce240c2f50 2024-04-25T00:21:33.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.116+0000 7fce28910640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce24150d20 0x7fce240c0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:33.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.116+0000 7fce28910640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce24150d20 0x7fce240c0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48840/0 (socket says 172.21.15.116:48840) 2024-04-25T00:21:33.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.116+0000 7fce28910640 1 -- 172.21.15.116:0/2525589861 learned_addr learned my addr 172.21.15.116:0/2525589861 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:21:33.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.116+0000 7fce23fff640 1 --2- 172.21.15.116:0/2525589861 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fce240a94a0 0x7fce240c4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:33.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.116+0000 7fce237fe640 1 --2- 172.21.15.116:0/2525589861 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fce240c2f50 0x7fce240726d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:33.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.116+0000 7fce28910640 1 -- 172.21.15.116:0/2525589861 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fce240c2f50 msgr2=0x7fce240726d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:33.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.116+0000 7fce28910640 1 --2- 172.21.15.116:0/2525589861 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fce240c2f50 0x7fce240726d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:33.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.116+0000 7fce28910640 1 -- 172.21.15.116:0/2525589861 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fce240a94a0 msgr2=0x7fce240c4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:33.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.116+0000 7fce28910640 1 --2- 172.21.15.116:0/2525589861 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fce240a94a0 0x7fce240c4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:33.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.116+0000 7fce28910640 1 -- 172.21.15.116:0/2525589861 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce240717f0 con 0x7fce24150d20 2024-04-25T00:21:33.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.116+0000 7fce23fff640 1 --2- 172.21.15.116:0/2525589861 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fce240a94a0 0x7fce240c4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:21:33.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.116+0000 7fce28910640 1 --2- 172.21.15.116:0/2525589861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce24150d20 0x7fce240c0690 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7fce0c00c4b0 tx=0x7fce0c00c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:33.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.116+0000 7fce20ff9640 1 -- 172.21.15.116:0/2525589861 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce0c018070 con 0x7fce24150d20 2024-04-25T00:21:33.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.116+0000 7fce20ff9640 1 -- 172.21.15.116:0/2525589861 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce0c0074e0 con 0x7fce24150d20 2024-04-25T00:21:33.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.116+0000 7fce2a373640 1 -- 172.21.15.116:0/2525589861 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce24071a60 con 0x7fce24150d20 2024-04-25T00:21:33.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.116+0000 7fce20ff9640 1 -- 172.21.15.116:0/2525589861 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce0c004a30 con 0x7fce24150d20 2024-04-25T00:21:33.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.116+0000 7fce2a373640 1 -- 172.21.15.116:0/2525589861 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce240c0bd0 con 0x7fce24150d20 2024-04-25T00:21:33.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.120+0000 7fce2a373640 1 -- 172.21.15.116:0/2525589861 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce24068350 con 0x7fce24150d20 2024-04-25T00:21:33.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.120+0000 7fce20ff9640 1 -- 172.21.15.116:0/2525589861 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fce0c01b070 con 0x7fce24150d20 2024-04-25T00:21:33.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.120+0000 7fce20ff9640 1 --2- 172.21.15.116:0/2525589861 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fce040678f0 0x7fce04069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:33.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.120+0000 7fce20ff9640 1 -- 172.21.15.116:0/2525589861 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 9332+0+0 (secure 0 0 0) 0x7fce0c085d90 con 0x7fce24150d20 2024-04-25T00:21:33.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.120+0000 7fce23fff640 1 --2- 172.21.15.116:0/2525589861 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fce040678f0 0x7fce04069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:33.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.120+0000 7fce23fff640 1 --2- 172.21.15.116:0/2525589861 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fce040678f0 0x7fce04069db0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fce1400b990 tx=0x7fce14008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:33.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.124+0000 7fce20ff9640 1 -- 172.21.15.116:0/2525589861 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fce0c055a50 con 0x7fce24150d20 2024-04-25T00:21:33.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.320+0000 7fce2a373640 1 -- 172.21.15.116:0/2525589861 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fce24071c30 con 0x7fce24150d20 2024-04-25T00:21:33.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.320+0000 7fce20ff9640 1 -- 172.21.15.116:0/2525589861 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 167 v167) ==== 95+0+6501 (secure 0 0 0) 0x7fce0c055350 con 0x7fce24150d20 2024-04-25T00:21:33.323 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:21:33.323 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":167,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6125,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/321466844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":321466844},{"type":"v1","addr":"172.21.15.174:6837","nonce":321466844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":167}],"filesystems":[{"mdsmap":{"epoch":166,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:20:53.870063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7809},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7809":{"gid":7809,"name":"c","rank":0,"incarnation":163,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6835/980221971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":980221971},{"type":"v1","addr":"172.21.15.116:6835","nonce":980221971}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7809,"qdb_cluster":[7809]},"id":1},{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:20:13.935378+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7722":{"gid":7722,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7722,"qdb_cluster":[7722]},"id":2}]} 2024-04-25T00:21:33.323 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 167 2024-04-25T00:21:33.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.324+0000 7fce2a373640 1 -- 172.21.15.116:0/2525589861 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fce040678f0 msgr2=0x7fce04069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:33.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.324+0000 7fce2a373640 1 --2- 172.21.15.116:0/2525589861 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fce040678f0 0x7fce04069db0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fce1400b990 tx=0x7fce14008040 comp rx=0 tx=0).stop 2024-04-25T00:21:33.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.324+0000 7fce2a373640 1 -- 172.21.15.116:0/2525589861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce24150d20 msgr2=0x7fce240c0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:33.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.324+0000 7fce2a373640 1 --2- 172.21.15.116:0/2525589861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce24150d20 0x7fce240c0690 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7fce0c00c4b0 tx=0x7fce0c00c970 comp rx=0 tx=0).stop 2024-04-25T00:21:33.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.324+0000 7fce2a373640 1 -- 172.21.15.116:0/2525589861 shutdown_connections 2024-04-25T00:21:33.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.324+0000 7fce2a373640 1 --2- 172.21.15.116:0/2525589861 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fce040678f0 0x7fce04069db0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:33.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.324+0000 7fce2a373640 1 --2- 172.21.15.116:0/2525589861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fce24150d20 0x7fce240c0690 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:33.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.324+0000 7fce2a373640 1 --2- 172.21.15.116:0/2525589861 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fce240c2f50 0x7fce240726d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:33.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.324+0000 7fce2a373640 1 --2- 172.21.15.116:0/2525589861 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fce240a94a0 0x7fce240c4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:33.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.324+0000 7fce2a373640 1 -- 172.21.15.116:0/2525589861 >> 172.21.15.116:0/2525589861 conn(0x7fce240c3b00 msgr2=0x7fce240acd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:21:33.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.324+0000 7fce2a373640 1 -- 172.21.15.116:0/2525589861 shutdown_connections 2024-04-25T00:21:33.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:33.324+0000 7fce2a373640 1 -- 172.21.15.116:0/2525589861 wait complete. 2024-04-25T00:21:33.343 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:21:35.344 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:21:35.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.471+0000 7f457c89c640 1 Processor -- start 2024-04-25T00:21:35.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.471+0000 7f457c89c640 1 -- start start 2024-04-25T00:21:35.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.475+0000 7f457c89c640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f4578070dd0 0x7f4578071270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:35.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.475+0000 7f457c89c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f457806e890 0x7f45780717b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:35.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.475+0000 7f457c89c640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4578152c10 con 0x7f4578148890 2024-04-25T00:21:35.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.475+0000 7f457c89c640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f4578071cf0 con 0x7f457806e890 2024-04-25T00:21:35.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.475+0000 7f457c89c640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f4578151e50 con 0x7f4578070dd0 2024-04-25T00:21:35.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.475+0000 7f4576d9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f457806e890 0x7f45780717b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:35.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.475+0000 7f457659c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f4578148890 0x7f4578072590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:59342/0 (socket says 172.21.15.116:59342) 2024-04-25T00:21:35.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.475+0000 7f457659c640 1 -- 172.21.15.116:0/1918758220 learned_addr learned my addr 172.21.15.116:0/1918758220 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:21:35.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.475+0000 7f4575d9b640 1 --2- 172.21.15.116:0/1918758220 >> v2:172.21.15.174:3300/0 conn(0x7f4578070dd0 0x7f4578071270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:35.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.475+0000 7f4576d9d640 1 -- 172.21.15.116:0/1918758220 >> v2:172.21.15.174:3300/0 conn(0x7f4578070dd0 msgr2=0x7f4578071270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:21:35.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.475+0000 7f4576d9d640 1 --2- 172.21.15.116:0/1918758220 >> v2:172.21.15.174:3300/0 conn(0x7f4578070dd0 0x7f4578071270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:35.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.475+0000 7f4576d9d640 1 -- 172.21.15.116:0/1918758220 >> v1:172.21.15.116:6789/0 conn(0x7f4578148890 legacy=0x7f4578072590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:35.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.475+0000 7f4576d9d640 1 -- 172.21.15.116:0/1918758220 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4578152130 con 0x7f457806e890 2024-04-25T00:21:35.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.475+0000 7f457559a640 1 -- 172.21.15.116:0/1918758220 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 256514646 0 0) 0x7f4578152c10 con 0x7f4578148890 2024-04-25T00:21:35.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.475+0000 7f4576d9d640 1 --2- 172.21.15.116:0/1918758220 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f457806e890 0x7f45780717b0 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f45781562f0 tx=0x7f456801c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e54a89d6ec189a7b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:35.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.475+0000 7f457559a640 1 -- 172.21.15.116:0/1918758220 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45680b0070 con 0x7f457806e890 2024-04-25T00:21:35.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.475+0000 7f457559a640 1 -- 172.21.15.116:0/1918758220 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45680ab600 con 0x7f457806e890 2024-04-25T00:21:35.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.475+0000 7f457559a640 1 -- 172.21.15.116:0/1918758220 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45680ab9d0 con 0x7f457806e890 2024-04-25T00:21:35.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.479+0000 7f457c89c640 1 -- 172.21.15.116:0/1918758220 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f457806e890 msgr2=0x7f45780717b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:35.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.479+0000 7f457c89c640 1 --2- 172.21.15.116:0/1918758220 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f457806e890 0x7f45780717b0 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f45781562f0 tx=0x7f456801c3f0 comp rx=0 tx=0).stop 2024-04-25T00:21:35.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.479+0000 7f457c89c640 1 -- 172.21.15.116:0/1918758220 shutdown_connections 2024-04-25T00:21:35.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.479+0000 7f457c89c640 1 --2- 172.21.15.116:0/1918758220 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f457806e890 0x7f45780717b0 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:35.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.479+0000 7f457c89c640 1 --2- 172.21.15.116:0/1918758220 >> v2:172.21.15.174:3300/0 conn(0x7f4578070dd0 0x7f4578071270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:35.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.479+0000 7f457c89c640 1 -- 172.21.15.116:0/1918758220 >> 172.21.15.116:0/1918758220 conn(0x7f45780c3b00 msgr2=0x7f45780c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:21:35.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.479+0000 7f457c89c640 1 -- 172.21.15.116:0/1918758220 shutdown_connections 2024-04-25T00:21:35.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.479+0000 7f457c89c640 1 -- 172.21.15.116:0/1918758220 wait complete. 2024-04-25T00:21:35.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.479+0000 7f457c89c640 1 Processor -- start 2024-04-25T00:21:35.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.479+0000 7f457c89c640 1 -- start start 2024-04-25T00:21:35.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.479+0000 7f457c89c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f457806e890 0x7f457814faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:35.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.479+0000 7f457c89c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4578070dd0 0x7f4578160e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:35.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.479+0000 7f457c89c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4578148890 0x7f457815f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:35.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.479+0000 7f457c89c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4578072e20 con 0x7f4578070dd0 2024-04-25T00:21:35.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.479+0000 7f457c89c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4578072ca0 con 0x7f4578148890 2024-04-25T00:21:35.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.479+0000 7f457c89c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f45780ab590 con 0x7f457806e890 2024-04-25T00:21:35.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.479+0000 7f4575d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4578070dd0 0x7f4578160e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:35.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.479+0000 7f4575d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4578070dd0 0x7f4578160e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52516/0 (socket says 172.21.15.116:52516) 2024-04-25T00:21:35.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.479+0000 7f4575d9b640 1 -- 172.21.15.116:0/1606814255 learned_addr learned my addr 172.21.15.116:0/1606814255 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:21:35.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.479+0000 7f4576d9d640 1 --2- 172.21.15.116:0/1606814255 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4578148890 0x7f457815f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:35.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.479+0000 7f457659c640 1 --2- 172.21.15.116:0/1606814255 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f457806e890 0x7f457814faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:35.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.479+0000 7f4575d9b640 1 -- 172.21.15.116:0/1606814255 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f457806e890 msgr2=0x7f457814faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:35.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.479+0000 7f4575d9b640 1 --2- 172.21.15.116:0/1606814255 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f457806e890 0x7f457814faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:35.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.479+0000 7f4575d9b640 1 -- 172.21.15.116:0/1606814255 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4578148890 msgr2=0x7f457815f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:35.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.479+0000 7f4575d9b640 1 --2- 172.21.15.116:0/1606814255 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4578148890 0x7f457815f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:35.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.479+0000 7f4575d9b640 1 -- 172.21.15.116:0/1606814255 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4578143b50 con 0x7f4578070dd0 2024-04-25T00:21:35.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.479+0000 7f457659c640 1 --2- 172.21.15.116:0/1606814255 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f457806e890 0x7f457814faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:21:35.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.479+0000 7f4576d9d640 1 --2- 172.21.15.116:0/1606814255 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4578148890 0x7f457815f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:21:35.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.479+0000 7f4575d9b640 1 --2- 172.21.15.116:0/1606814255 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4578070dd0 0x7f4578160e40 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f456000d7d0 tx=0x7f456000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:35.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.479+0000 7f456effd640 1 -- 172.21.15.116:0/1606814255 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f456000b6c0 con 0x7f4578070dd0 2024-04-25T00:21:35.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.479+0000 7f456effd640 1 -- 172.21.15.116:0/1606814255 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4560019070 con 0x7f4578070dd0 2024-04-25T00:21:35.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.479+0000 7f457c89c640 1 -- 172.21.15.116:0/1606814255 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4578142410 con 0x7f4578070dd0 2024-04-25T00:21:35.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.479+0000 7f456effd640 1 -- 172.21.15.116:0/1606814255 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f456000baa0 con 0x7f4578070dd0 2024-04-25T00:21:35.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.479+0000 7f457c89c640 1 -- 172.21.15.116:0/1606814255 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45781440d0 con 0x7f4578070dd0 2024-04-25T00:21:35.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.479+0000 7f457c89c640 1 -- 172.21.15.116:0/1606814255 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4578068350 con 0x7f4578070dd0 2024-04-25T00:21:35.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.483+0000 7f456effd640 1 -- 172.21.15.116:0/1606814255 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f456000bc40 con 0x7f4578070dd0 2024-04-25T00:21:35.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.483+0000 7f456effd640 1 --2- 172.21.15.116:0/1606814255 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f45580678f0 0x7f4558069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:35.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.483+0000 7f456effd640 1 -- 172.21.15.116:0/1606814255 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 9332+0+0 (secure 0 0 0) 0x7f4560086f80 con 0x7f4578070dd0 2024-04-25T00:21:35.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.483+0000 7f457659c640 1 --2- 172.21.15.116:0/1606814255 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f45580678f0 0x7f4558069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:35.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.483+0000 7f457659c640 1 --2- 172.21.15.116:0/1606814255 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f45580678f0 0x7f4558069db0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f45700012f0 tx=0x7f4570009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:35.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.487+0000 7f456effd640 1 -- 172.21.15.116:0/1606814255 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f4560056c40 con 0x7f4578070dd0 2024-04-25T00:21:35.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.687+0000 7f457c89c640 1 -- 172.21.15.116:0/1606814255 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4578144310 con 0x7f4578070dd0 2024-04-25T00:21:35.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.687+0000 7f456effd640 1 -- 172.21.15.116:0/1606814255 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 167 v167) ==== 95+0+6501 (secure 0 0 0) 0x7f4560056540 con 0x7f4578070dd0 2024-04-25T00:21:35.692 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:21:35.692 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":167,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6125,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/321466844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":321466844},{"type":"v1","addr":"172.21.15.174:6837","nonce":321466844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":167}],"filesystems":[{"mdsmap":{"epoch":166,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:20:53.870063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7809},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7809":{"gid":7809,"name":"c","rank":0,"incarnation":163,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6835/980221971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":980221971},{"type":"v1","addr":"172.21.15.116:6835","nonce":980221971}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7809,"qdb_cluster":[7809]},"id":1},{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:20:13.935378+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7722":{"gid":7722,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7722,"qdb_cluster":[7722]},"id":2}]} 2024-04-25T00:21:35.692 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 167 2024-04-25T00:21:35.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.691+0000 7f457c89c640 1 -- 172.21.15.116:0/1606814255 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f45580678f0 msgr2=0x7f4558069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:35.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.691+0000 7f457c89c640 1 --2- 172.21.15.116:0/1606814255 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f45580678f0 0x7f4558069db0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f45700012f0 tx=0x7f4570009040 comp rx=0 tx=0).stop 2024-04-25T00:21:35.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.691+0000 7f457c89c640 1 -- 172.21.15.116:0/1606814255 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4578070dd0 msgr2=0x7f4578160e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:35.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.691+0000 7f457c89c640 1 --2- 172.21.15.116:0/1606814255 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4578070dd0 0x7f4578160e40 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f456000d7d0 tx=0x7f456000dc90 comp rx=0 tx=0).stop 2024-04-25T00:21:35.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.691+0000 7f457c89c640 1 -- 172.21.15.116:0/1606814255 shutdown_connections 2024-04-25T00:21:35.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.691+0000 7f457c89c640 1 --2- 172.21.15.116:0/1606814255 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f45580678f0 0x7f4558069db0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:35.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.691+0000 7f457c89c640 1 --2- 172.21.15.116:0/1606814255 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4578148890 0x7f457815f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:35.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.691+0000 7f457c89c640 1 --2- 172.21.15.116:0/1606814255 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4578070dd0 0x7f4578160e40 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:35.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.691+0000 7f457c89c640 1 --2- 172.21.15.116:0/1606814255 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f457806e890 0x7f457814faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:35.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.691+0000 7f457c89c640 1 -- 172.21.15.116:0/1606814255 >> 172.21.15.116:0/1606814255 conn(0x7f45780c3b00 msgr2=0x7f457806cbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:21:35.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.691+0000 7f457c89c640 1 -- 172.21.15.116:0/1606814255 shutdown_connections 2024-04-25T00:21:35.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:35.691+0000 7f457c89c640 1 -- 172.21.15.116:0/1606814255 wait complete. 2024-04-25T00:21:35.711 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:21:37.712 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:21:37.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.839+0000 7effce166640 1 Processor -- start 2024-04-25T00:21:37.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.839+0000 7effce166640 1 -- start start 2024-04-25T00:21:37.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.839+0000 7effce166640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7effc8150d20 0x7effc80c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:37.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.839+0000 7effce166640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7effc80c3910 con 0x7effc80a94a0 2024-04-25T00:21:37.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.839+0000 7effce166640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7effc80c16e0 con 0x7effc80c2f50 2024-04-25T00:21:37.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.839+0000 7effce166640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7effc80c1400 con 0x7effc8150d20 2024-04-25T00:21:37.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.839+0000 7effc7fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7effc80a94a0 0x7effc8153ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:59354/0 (socket says 172.21.15.116:59354) 2024-04-25T00:21:37.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.839+0000 7effc7fff640 1 -- 172.21.15.116:0/25869871 learned_addr learned my addr 172.21.15.116:0/25869871 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:21:37.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.839+0000 7effc77fe640 1 --2- 172.21.15.116:0/25869871 >> v2:172.21.15.174:3300/0 conn(0x7effc8150d20 0x7effc80c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:37.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.839+0000 7effc67fc640 1 -- 172.21.15.116:0/25869871 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 605393128 0 0) 0x7effc80c3910 con 0x7effc80a94a0 2024-04-25T00:21:37.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.839+0000 7effc67fc640 1 -- 172.21.15.116:0/25869871 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7effb0003530 con 0x7effc80a94a0 2024-04-25T00:21:37.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.839+0000 7effc67fc640 1 -- 172.21.15.116:0/25869871 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3939365695 0 0) 0x7effb0003530 con 0x7effc80a94a0 2024-04-25T00:21:37.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.839+0000 7effc67fc640 1 -- 172.21.15.116:0/25869871 >> v2:172.21.15.174:3300/0 conn(0x7effc8150d20 msgr2=0x7effc80c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:21:37.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.839+0000 7effc67fc640 1 --2- 172.21.15.116:0/25869871 >> v2:172.21.15.174:3300/0 conn(0x7effc8150d20 0x7effc80c2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:37.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.839+0000 7effc67fc640 1 -- 172.21.15.116:0/25869871 >> v1:172.21.15.174:6789/0 conn(0x7effc80c2f50 legacy=0x7effc81538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:37.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.839+0000 7effc67fc640 1 -- 172.21.15.116:0/25869871 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effc806e660 con 0x7effc80a94a0 2024-04-25T00:21:37.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.839+0000 7effc67fc640 1 -- 172.21.15.116:0/25869871 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7effc0001a00 con 0x7effc80a94a0 2024-04-25T00:21:37.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.839+0000 7effc67fc640 1 -- 172.21.15.116:0/25869871 >> v1:172.21.15.116:6789/0 conn(0x7effc80a94a0 legacy=0x7effc8153ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:37.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.839+0000 7effc67fc640 1 --2- 172.21.15.116:0/25869871 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7effb00039c0 0x7effb0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:37.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.839+0000 7effc67fc640 1 --2- 172.21.15.116:0/25869871 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7effb0004810 0x7effb0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:37.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.839+0000 7effc67fc640 1 --2- 172.21.15.116:0/25869871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effb00051d0 0x7effb000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:37.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.839+0000 7effc67fc640 1 -- 172.21.15.116:0/25869871 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7effc80c1400 con 0x7effb00051d0 2024-04-25T00:21:37.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.839+0000 7effc67fc640 1 -- 172.21.15.116:0/25869871 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7effb000c070 con 0x7effb0004810 2024-04-25T00:21:37.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.839+0000 7effc67fc640 1 -- 172.21.15.116:0/25869871 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7effb000c270 con 0x7effb00039c0 2024-04-25T00:21:37.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.839+0000 7effc7fff640 1 --2- 172.21.15.116:0/25869871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effb00051d0 0x7effb000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:37.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.839+0000 7effc6ffd640 1 --2- 172.21.15.116:0/25869871 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7effb0004810 0x7effb0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:37.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.839+0000 7effc77fe640 1 --2- 172.21.15.116:0/25869871 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7effb00039c0 0x7effb0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:37.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.839+0000 7effc7fff640 1 -- 172.21.15.116:0/25869871 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7effb00039c0 msgr2=0x7effb0003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:21:37.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.839+0000 7effc7fff640 1 --2- 172.21.15.116:0/25869871 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7effb00039c0 0x7effb0003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:37.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.839+0000 7effc7fff640 1 -- 172.21.15.116:0/25869871 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7effb0004810 msgr2=0x7effb0004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:21:37.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.839+0000 7effc7fff640 1 --2- 172.21.15.116:0/25869871 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7effb0004810 0x7effb0004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:37.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.839+0000 7effc7fff640 1 -- 172.21.15.116:0/25869871 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effb0003530 con 0x7effb00051d0 2024-04-25T00:21:37.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.839+0000 7effc6ffd640 1 --2- 172.21.15.116:0/25869871 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7effb0004810 0x7effb0004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:21:37.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.843+0000 7effc7fff640 1 -- 172.21.15.116:0/25869871 reap_dead start 2024-04-25T00:21:37.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.843+0000 7effc7fff640 1 --2- 172.21.15.116:0/25869871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effb00051d0 0x7effb000b9d0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7effc000a850 tx=0x7effc001d3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=484bef47c16611ec server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:37.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.843+0000 7effc67fc640 1 -- 172.21.15.116:0/25869871 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effc000a4a0 con 0x7effb00051d0 2024-04-25T00:21:37.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.843+0000 7effc67fc640 1 -- 172.21.15.116:0/25869871 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effc00ad650 con 0x7effb00051d0 2024-04-25T00:21:37.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.843+0000 7effc67fc640 1 -- 172.21.15.116:0/25869871 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effc00ada40 con 0x7effb00051d0 2024-04-25T00:21:37.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.843+0000 7effce166640 1 -- 172.21.15.116:0/25869871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effb00051d0 msgr2=0x7effb000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:37.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.843+0000 7effce166640 1 --2- 172.21.15.116:0/25869871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effb00051d0 0x7effb000b9d0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7effc000a850 tx=0x7effc001d3a0 comp rx=0 tx=0).stop 2024-04-25T00:21:37.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.843+0000 7effce166640 1 -- 172.21.15.116:0/25869871 shutdown_connections 2024-04-25T00:21:37.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.843+0000 7effce166640 1 --2- 172.21.15.116:0/25869871 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effb00051d0 0x7effb000b9d0 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:37.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.843+0000 7effce166640 1 -- 172.21.15.116:0/25869871 >> 172.21.15.116:0/25869871 conn(0x7effc80c3b00 msgr2=0x7effc80c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:21:37.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.843+0000 7effce166640 1 -- 172.21.15.116:0/25869871 shutdown_connections 2024-04-25T00:21:37.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.843+0000 7effce166640 1 -- 172.21.15.116:0/25869871 wait complete. 2024-04-25T00:21:37.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.843+0000 7effce166640 1 Processor -- start 2024-04-25T00:21:37.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.843+0000 7effce166640 1 -- start start 2024-04-25T00:21:37.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.847+0000 7effce166640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7effb00051d0 0x7effc8154980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:37.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.847+0000 7effce166640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7effc80c62c0 0x7effc81e9180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:37.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.847+0000 7effce166640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effc81e96c0 0x7effc81425d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:37.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.847+0000 7effce166640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7effc806ec20 con 0x7effc81e96c0 2024-04-25T00:21:37.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.847+0000 7effce166640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7effc80c1280 con 0x7effb00051d0 2024-04-25T00:21:37.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.847+0000 7effce166640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7effc80a99f0 con 0x7effc80c62c0 2024-04-25T00:21:37.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.847+0000 7effc7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effc81e96c0 0x7effc81425d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:37.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.847+0000 7effc7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effc81e96c0 0x7effc81425d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52526/0 (socket says 172.21.15.116:52526) 2024-04-25T00:21:37.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.847+0000 7effc7fff640 1 -- 172.21.15.116:0/3079646900 learned_addr learned my addr 172.21.15.116:0/3079646900 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:21:37.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.847+0000 7effc77fe640 1 --2- 172.21.15.116:0/3079646900 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7effb00051d0 0x7effc8154980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:37.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.847+0000 7effc6ffd640 1 --2- 172.21.15.116:0/3079646900 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7effc80c62c0 0x7effc81e9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:37.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.847+0000 7effc7fff640 1 -- 172.21.15.116:0/3079646900 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7effc80c62c0 msgr2=0x7effc81e9180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:37.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.847+0000 7effc7fff640 1 --2- 172.21.15.116:0/3079646900 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7effc80c62c0 0x7effc81e9180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:37.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.847+0000 7effc7fff640 1 -- 172.21.15.116:0/3079646900 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7effb00051d0 msgr2=0x7effc8154980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:37.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.847+0000 7effc7fff640 1 --2- 172.21.15.116:0/3079646900 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7effb00051d0 0x7effc8154980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:37.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.847+0000 7effc7fff640 1 -- 172.21.15.116:0/3079646900 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effc8153340 con 0x7effc81e96c0 2024-04-25T00:21:37.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.847+0000 7effc7fff640 1 --2- 172.21.15.116:0/3079646900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effc81e96c0 0x7effc81425d0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7effc001dc40 tx=0x7effb0005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:37.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.847+0000 7effaffff640 1 -- 172.21.15.116:0/3079646900 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effc0001940 con 0x7effc81e96c0 2024-04-25T00:21:37.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.847+0000 7effaffff640 1 -- 172.21.15.116:0/3079646900 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effc00b4070 con 0x7effc81e96c0 2024-04-25T00:21:37.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.847+0000 7effce166640 1 -- 172.21.15.116:0/3079646900 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effc8152e60 con 0x7effc81e96c0 2024-04-25T00:21:37.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.847+0000 7effaffff640 1 -- 172.21.15.116:0/3079646900 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effc000a480 con 0x7effc81e96c0 2024-04-25T00:21:37.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.847+0000 7effce166640 1 -- 172.21.15.116:0/3079646900 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effc81ed2a0 con 0x7effc81e96c0 2024-04-25T00:21:37.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.847+0000 7effce166640 1 -- 172.21.15.116:0/3079646900 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effc8153030 con 0x7effc81e96c0 2024-04-25T00:21:37.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.847+0000 7effaffff640 1 -- 172.21.15.116:0/3079646900 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7effc00b6070 con 0x7effc81e96c0 2024-04-25T00:21:37.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.847+0000 7effaffff640 1 --2- 172.21.15.116:0/3079646900 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7effb40678f0 0x7effb4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:37.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.847+0000 7effaffff640 1 -- 172.21.15.116:0/3079646900 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 9332+0+0 (secure 0 0 0) 0x7effc0120460 con 0x7effc81e96c0 2024-04-25T00:21:37.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.847+0000 7effc77fe640 1 --2- 172.21.15.116:0/3079646900 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7effb40678f0 0x7effb4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:37.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.851+0000 7effc77fe640 1 --2- 172.21.15.116:0/3079646900 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7effb40678f0 0x7effb4069db0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7effc8155fa0 tx=0x7effbc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:37.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:37.851+0000 7effaffff640 1 -- 172.21.15.116:0/3079646900 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7effc00be060 con 0x7effc81e96c0 2024-04-25T00:21:38.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:38.051+0000 7effce166640 1 -- 172.21.15.116:0/3079646900 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7effc8068350 con 0x7effc81e96c0 2024-04-25T00:21:38.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:38.051+0000 7effaffff640 1 -- 172.21.15.116:0/3079646900 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 167 v167) ==== 95+0+6501 (secure 0 0 0) 0x7effc00f0e20 con 0x7effc81e96c0 2024-04-25T00:21:38.055 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:21:38.055 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":167,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6125,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/321466844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":321466844},{"type":"v1","addr":"172.21.15.174:6837","nonce":321466844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":167}],"filesystems":[{"mdsmap":{"epoch":166,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:20:53.870063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7809},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7809":{"gid":7809,"name":"c","rank":0,"incarnation":163,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6835/980221971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":980221971},{"type":"v1","addr":"172.21.15.116:6835","nonce":980221971}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7809,"qdb_cluster":[7809]},"id":1},{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:20:13.935378+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7722":{"gid":7722,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7722,"qdb_cluster":[7722]},"id":2}]} 2024-04-25T00:21:38.055 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 167 2024-04-25T00:21:38.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:38.051+0000 7effce166640 1 -- 172.21.15.116:0/3079646900 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7effb40678f0 msgr2=0x7effb4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:38.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:38.051+0000 7effce166640 1 --2- 172.21.15.116:0/3079646900 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7effb40678f0 0x7effb4069db0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7effc8155fa0 tx=0x7effbc008040 comp rx=0 tx=0).stop 2024-04-25T00:21:38.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:38.051+0000 7effce166640 1 -- 172.21.15.116:0/3079646900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effc81e96c0 msgr2=0x7effc81425d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:38.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:38.051+0000 7effce166640 1 --2- 172.21.15.116:0/3079646900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effc81e96c0 0x7effc81425d0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7effc001dc40 tx=0x7effb0005840 comp rx=0 tx=0).stop 2024-04-25T00:21:38.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:38.051+0000 7effce166640 1 -- 172.21.15.116:0/3079646900 shutdown_connections 2024-04-25T00:21:38.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:38.051+0000 7effce166640 1 --2- 172.21.15.116:0/3079646900 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7effb40678f0 0x7effb4069db0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:38.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:38.051+0000 7effce166640 1 --2- 172.21.15.116:0/3079646900 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7effc81e96c0 0x7effc81425d0 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:38.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:38.051+0000 7effce166640 1 --2- 172.21.15.116:0/3079646900 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7effc80c62c0 0x7effc81e9180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:38.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:38.051+0000 7effce166640 1 --2- 172.21.15.116:0/3079646900 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7effb00051d0 0x7effc8154980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:38.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:38.051+0000 7effce166640 1 -- 172.21.15.116:0/3079646900 >> 172.21.15.116:0/3079646900 conn(0x7effc80c3b00 msgr2=0x7effc806d660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:21:38.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:38.055+0000 7effce166640 1 -- 172.21.15.116:0/3079646900 shutdown_connections 2024-04-25T00:21:38.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:38.055+0000 7effce166640 1 -- 172.21.15.116:0/3079646900 wait complete. 2024-04-25T00:21:38.073 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:21:39.940 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:21:39.943 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:21:40.074 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:21:40.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.199+0000 7fbe4c473640 1 Processor -- start 2024-04-25T00:21:40.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.199+0000 7fbe4c473640 1 -- start start 2024-04-25T00:21:40.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.199+0000 7fbe4c473640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbe44150d20 0x7fbe440c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:40.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.199+0000 7fbe4c473640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe440a94a0 0x7fbe44153ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:40.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.199+0000 7fbe4c473640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe440c3910 con 0x7fbe440c2f50 2024-04-25T00:21:40.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.199+0000 7fbe4c473640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fbe44154530 con 0x7fbe44150d20 2024-04-25T00:21:40.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.199+0000 7fbe4c473640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fbe440c16e0 con 0x7fbe440a94a0 2024-04-25T00:21:40.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.199+0000 7fbe4a20f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbe44150d20 0x7fbe440c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:40.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.199+0000 7fbe4a20f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbe44150d20 0x7fbe440c2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52536/0 (socket says 172.21.15.116:52536) 2024-04-25T00:21:40.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.199+0000 7fbe4a20f640 1 -- 172.21.15.116:0/1267069408 learned_addr learned my addr 172.21.15.116:0/1267069408 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:21:40.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.199+0000 7fbe4aa10640 1 --2- 172.21.15.116:0/1267069408 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe440a94a0 0x7fbe44153ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:40.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.199+0000 7fbe4a20f640 1 -- 172.21.15.116:0/1267069408 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe440a94a0 msgr2=0x7fbe44153ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:21:40.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.199+0000 7fbe4a20f640 1 --2- 172.21.15.116:0/1267069408 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe440a94a0 0x7fbe44153ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:40.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.199+0000 7fbe4a20f640 1 -- 172.21.15.116:0/1267069408 >> v1:172.21.15.174:6789/0 conn(0x7fbe440c2f50 legacy=0x7fbe441538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:40.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.199+0000 7fbe4a20f640 1 -- 172.21.15.116:0/1267069408 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe440c1400 con 0x7fbe44150d20 2024-04-25T00:21:40.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.203+0000 7fbe4a20f640 1 --2- 172.21.15.116:0/1267069408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe44150d20 0x7fbe440c2a10 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7fbe34009870 tx=0x7fbe3401c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a33389fe8c85d3a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:40.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.203+0000 7fbe4920d640 1 -- 172.21.15.116:0/1267069408 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe340a7070 con 0x7fbe44150d20 2024-04-25T00:21:40.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.203+0000 7fbe4920d640 1 -- 172.21.15.116:0/1267069408 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe340b0070 con 0x7fbe44150d20 2024-04-25T00:21:40.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.203+0000 7fbe4920d640 1 -- 172.21.15.116:0/1267069408 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe340ab890 con 0x7fbe44150d20 2024-04-25T00:21:40.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.203+0000 7fbe4c473640 1 -- 172.21.15.116:0/1267069408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe44150d20 msgr2=0x7fbe440c2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:40.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.203+0000 7fbe4c473640 1 --2- 172.21.15.116:0/1267069408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe44150d20 0x7fbe440c2a10 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7fbe34009870 tx=0x7fbe3401c560 comp rx=0 tx=0).stop 2024-04-25T00:21:40.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.203+0000 7fbe4c473640 1 -- 172.21.15.116:0/1267069408 shutdown_connections 2024-04-25T00:21:40.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.203+0000 7fbe4c473640 1 --2- 172.21.15.116:0/1267069408 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe440a94a0 0x7fbe44153ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:40.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.203+0000 7fbe4c473640 1 --2- 172.21.15.116:0/1267069408 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe44150d20 0x7fbe440c2a10 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:40.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.203+0000 7fbe4c473640 1 -- 172.21.15.116:0/1267069408 >> 172.21.15.116:0/1267069408 conn(0x7fbe440c3b00 msgr2=0x7fbe440c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:21:40.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.203+0000 7fbe4c473640 1 -- 172.21.15.116:0/1267069408 shutdown_connections 2024-04-25T00:21:40.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.203+0000 7fbe4c473640 1 -- 172.21.15.116:0/1267069408 wait complete. 2024-04-25T00:21:40.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.203+0000 7fbe4c473640 1 Processor -- start 2024-04-25T00:21:40.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.203+0000 7fbe4c473640 1 -- start start 2024-04-25T00:21:40.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.203+0000 7fbe4c473640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe440a94a0 0x7fbe440c5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:40.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.203+0000 7fbe4c473640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe440c2f50 0x7fbe4413b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:40.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.203+0000 7fbe4c473640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe44150d20 0x7fbe44151c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:40.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.203+0000 7fbe4c473640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbe441608b0 con 0x7fbe44150d20 2024-04-25T00:21:40.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.203+0000 7fbe4c473640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbe44160730 con 0x7fbe440a94a0 2024-04-25T00:21:40.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.203+0000 7fbe4c473640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fbe44160a30 con 0x7fbe440c2f50 2024-04-25T00:21:40.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.203+0000 7fbe4aa10640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe44150d20 0x7fbe44151c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:40.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.203+0000 7fbe4aa10640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe44150d20 0x7fbe44151c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52550/0 (socket says 172.21.15.116:52550) 2024-04-25T00:21:40.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.203+0000 7fbe4aa10640 1 -- 172.21.15.116:0/2069425778 learned_addr learned my addr 172.21.15.116:0/2069425778 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:21:40.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.203+0000 7fbe49a0e640 1 --2- 172.21.15.116:0/2069425778 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe440c2f50 0x7fbe4413b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:40.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.203+0000 7fbe4a20f640 1 --2- 172.21.15.116:0/2069425778 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe440a94a0 0x7fbe440c5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:40.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.203+0000 7fbe4aa10640 1 -- 172.21.15.116:0/2069425778 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe440c2f50 msgr2=0x7fbe4413b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:40.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.203+0000 7fbe4aa10640 1 --2- 172.21.15.116:0/2069425778 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe440c2f50 0x7fbe4413b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:40.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.203+0000 7fbe4aa10640 1 -- 172.21.15.116:0/2069425778 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe440a94a0 msgr2=0x7fbe440c5300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:40.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.203+0000 7fbe4aa10640 1 --2- 172.21.15.116:0/2069425778 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe440a94a0 0x7fbe440c5300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:40.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.203+0000 7fbe4aa10640 1 -- 172.21.15.116:0/2069425778 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe440ad380 con 0x7fbe44150d20 2024-04-25T00:21:40.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.203+0000 7fbe4aa10640 1 --2- 172.21.15.116:0/2069425778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe44150d20 0x7fbe44151c30 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c007e70 tx=0x7fbe3c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:40.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.207+0000 7fbe3affd640 1 -- 172.21.15.116:0/2069425778 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe3c014620 con 0x7fbe44150d20 2024-04-25T00:21:40.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.207+0000 7fbe3affd640 1 -- 172.21.15.116:0/2069425778 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe3c019070 con 0x7fbe44150d20 2024-04-25T00:21:40.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.207+0000 7fbe4c473640 1 -- 172.21.15.116:0/2069425778 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe441514a0 con 0x7fbe44150d20 2024-04-25T00:21:40.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.207+0000 7fbe3affd640 1 -- 172.21.15.116:0/2069425778 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe3c014a20 con 0x7fbe44150d20 2024-04-25T00:21:40.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.207+0000 7fbe4c473640 1 -- 172.21.15.116:0/2069425778 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe44144bb0 con 0x7fbe44150d20 2024-04-25T00:21:40.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.207+0000 7fbe4c473640 1 -- 172.21.15.116:0/2069425778 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe44068350 con 0x7fbe44150d20 2024-04-25T00:21:40.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.207+0000 7fbe3affd640 1 -- 172.21.15.116:0/2069425778 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbe3c014bc0 con 0x7fbe44150d20 2024-04-25T00:21:40.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.207+0000 7fbe3affd640 1 --2- 172.21.15.116:0/2069425778 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbe280678c0 0x7fbe28069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:40.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.207+0000 7fbe3affd640 1 -- 172.21.15.116:0/2069425778 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(59..59 src has 1..59) ==== 9332+0+0 (secure 0 0 0) 0x7fbe3c087350 con 0x7fbe44150d20 2024-04-25T00:21:40.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.207+0000 7fbe4a20f640 1 --2- 172.21.15.116:0/2069425778 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbe280678c0 0x7fbe28069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:40.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.207+0000 7fbe4a20f640 1 --2- 172.21.15.116:0/2069425778 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbe280678c0 0x7fbe28069d80 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fbe340af000 tx=0x7fbe340a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:40.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.211+0000 7fbe3affd640 1 -- 172.21.15.116:0/2069425778 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fbe3c011260 con 0x7fbe44150d20 2024-04-25T00:21:40.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.403+0000 7fbe4c473640 1 -- 172.21.15.116:0/2069425778 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbe440c34f0 con 0x7fbe44150d20 2024-04-25T00:21:40.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.407+0000 7fbe3affd640 1 -- 172.21.15.116:0/2069425778 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 167 v167) ==== 95+0+6501 (secure 0 0 0) 0x7fbe3c0570a0 con 0x7fbe44150d20 2024-04-25T00:21:40.410 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:21:40.410 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":167,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6125,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/321466844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":321466844},{"type":"v1","addr":"172.21.15.174:6837","nonce":321466844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":167}],"filesystems":[{"mdsmap":{"epoch":166,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:20:53.870063+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7809},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7809":{"gid":7809,"name":"c","rank":0,"incarnation":163,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6835/980221971","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":980221971},{"type":"v1","addr":"172.21.15.116:6835","nonce":980221971}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7809,"qdb_cluster":[7809]},"id":1},{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:20:13.935378+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7722":{"gid":7722,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7722,"qdb_cluster":[7722]},"id":2}]} 2024-04-25T00:21:40.410 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 167 2024-04-25T00:21:40.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.407+0000 7fbe4c473640 1 -- 172.21.15.116:0/2069425778 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbe280678c0 msgr2=0x7fbe28069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:40.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.407+0000 7fbe4c473640 1 --2- 172.21.15.116:0/2069425778 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbe280678c0 0x7fbe28069d80 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fbe340af000 tx=0x7fbe340a7000 comp rx=0 tx=0).stop 2024-04-25T00:21:40.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.407+0000 7fbe4c473640 1 -- 172.21.15.116:0/2069425778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe44150d20 msgr2=0x7fbe44151c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:40.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.407+0000 7fbe4c473640 1 --2- 172.21.15.116:0/2069425778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe44150d20 0x7fbe44151c30 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c007e70 tx=0x7fbe3c00e4b0 comp rx=0 tx=0).stop 2024-04-25T00:21:40.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.407+0000 7fbe4c473640 1 -- 172.21.15.116:0/2069425778 shutdown_connections 2024-04-25T00:21:40.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.407+0000 7fbe4c473640 1 --2- 172.21.15.116:0/2069425778 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbe280678c0 0x7fbe28069d80 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:40.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.407+0000 7fbe4c473640 1 --2- 172.21.15.116:0/2069425778 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe44150d20 0x7fbe44151c30 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:40.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.407+0000 7fbe4c473640 1 --2- 172.21.15.116:0/2069425778 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe440c2f50 0x7fbe4413b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:40.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.407+0000 7fbe4c473640 1 --2- 172.21.15.116:0/2069425778 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe440a94a0 0x7fbe440c5300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:40.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.407+0000 7fbe4c473640 1 -- 172.21.15.116:0/2069425778 >> 172.21.15.116:0/2069425778 conn(0x7fbe440c3b00 msgr2=0x7fbe4406cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:21:40.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.407+0000 7fbe4c473640 1 -- 172.21.15.116:0/2069425778 shutdown_connections 2024-04-25T00:21:40.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:40.407+0000 7fbe4c473640 1 -- 172.21.15.116:0/2069425778 wait complete. 2024-04-25T00:21:40.430 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:21:42.431 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:21:42.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.563+0000 7f662480b640 1 Processor -- start 2024-04-25T00:21:42.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.563+0000 7f662480b640 1 -- start start 2024-04-25T00:21:42.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.567+0000 7f662480b640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f6620150d20 0x7f66200c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:42.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.567+0000 7f662480b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f66200a94a0 0x7f6620153ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:42.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.567+0000 7f662480b640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66200c3910 con 0x7f66200c2f50 2024-04-25T00:21:42.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.567+0000 7f662480b640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f6620154530 con 0x7f66200a94a0 2024-04-25T00:21:42.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.567+0000 7f662480b640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f66200c16e0 con 0x7f6620150d20 2024-04-25T00:21:42.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.567+0000 7f661ed9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f66200a94a0 0x7f6620153ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:42.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.567+0000 7f661ed9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f66200a94a0 0x7f6620153ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52564/0 (socket says 172.21.15.116:52564) 2024-04-25T00:21:42.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.567+0000 7f661ed9d640 1 -- 172.21.15.116:0/2519733995 learned_addr learned my addr 172.21.15.116:0/2519733995 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:21:42.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.567+0000 7f661e59c640 1 --2- 172.21.15.116:0/2519733995 >> v2:172.21.15.174:3300/0 conn(0x7f6620150d20 0x7f66200c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:42.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.567+0000 7f661ed9d640 1 -- 172.21.15.116:0/2519733995 >> v2:172.21.15.174:3300/0 conn(0x7f6620150d20 msgr2=0x7f66200c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:21:42.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.567+0000 7f661ed9d640 1 --2- 172.21.15.116:0/2519733995 >> v2:172.21.15.174:3300/0 conn(0x7f6620150d20 0x7f66200c2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:42.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.567+0000 7f661ed9d640 1 -- 172.21.15.116:0/2519733995 >> v1:172.21.15.174:6789/0 conn(0x7f66200c2f50 legacy=0x7f66201538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:42.570 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.567+0000 7f661ed9d640 1 -- 172.21.15.116:0/2519733995 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66200c1400 con 0x7f66200a94a0 2024-04-25T00:21:42.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.567+0000 7f661ed9d640 1 --2- 172.21.15.116:0/2519733995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66200a94a0 0x7f6620153ff0 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f6620073550 tx=0x7f661801c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a6943fda0390e94 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:42.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.567+0000 7f661dd9b640 1 -- 172.21.15.116:0/2519733995 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66180a57d0 con 0x7f66200a94a0 2024-04-25T00:21:42.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.567+0000 7f661dd9b640 1 -- 172.21.15.116:0/2519733995 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66180a5970 con 0x7f66200a94a0 2024-04-25T00:21:42.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.567+0000 7f661dd9b640 1 -- 172.21.15.116:0/2519733995 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66180a5d40 con 0x7f66200a94a0 2024-04-25T00:21:42.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.567+0000 7f662480b640 1 -- 172.21.15.116:0/2519733995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66200a94a0 msgr2=0x7f6620153ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:42.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.567+0000 7f662480b640 1 --2- 172.21.15.116:0/2519733995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66200a94a0 0x7f6620153ff0 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f6620073550 tx=0x7f661801c9f0 comp rx=0 tx=0).stop 2024-04-25T00:21:42.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.567+0000 7f662480b640 1 -- 172.21.15.116:0/2519733995 shutdown_connections 2024-04-25T00:21:42.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.567+0000 7f662480b640 1 --2- 172.21.15.116:0/2519733995 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66200a94a0 0x7f6620153ff0 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:42.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.567+0000 7f662480b640 1 --2- 172.21.15.116:0/2519733995 >> v2:172.21.15.174:3300/0 conn(0x7f6620150d20 0x7f66200c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:42.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.567+0000 7f662480b640 1 -- 172.21.15.116:0/2519733995 >> 172.21.15.116:0/2519733995 conn(0x7f66200c3b00 msgr2=0x7f66200c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:21:42.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.567+0000 7f662480b640 1 -- 172.21.15.116:0/2519733995 shutdown_connections 2024-04-25T00:21:42.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.567+0000 7f662480b640 1 -- 172.21.15.116:0/2519733995 wait complete. 2024-04-25T00:21:42.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.571+0000 7f662480b640 1 Processor -- start 2024-04-25T00:21:42.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.571+0000 7f662480b640 1 -- start start 2024-04-25T00:21:42.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.571+0000 7f662480b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66200a94a0 0x7f6620142150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:42.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.571+0000 7f662480b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66200c2f50 0x7f6620140760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:42.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.571+0000 7f662480b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6620150d20 0x7f662014ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:42.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.571+0000 7f662480b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f66201608b0 con 0x7f66200a94a0 2024-04-25T00:21:42.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.571+0000 7f662480b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6620160730 con 0x7f66200c2f50 2024-04-25T00:21:42.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.571+0000 7f662480b640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6620160a30 con 0x7f6620150d20 2024-04-25T00:21:42.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.571+0000 7f661e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66200a94a0 0x7f6620142150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:42.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.571+0000 7f661e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66200a94a0 0x7f6620142150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52572/0 (socket says 172.21.15.116:52572) 2024-04-25T00:21:42.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.571+0000 7f661e59c640 1 -- 172.21.15.116:0/3921508506 learned_addr learned my addr 172.21.15.116:0/3921508506 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:21:42.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.571+0000 7f6617fff640 1 --2- 172.21.15.116:0/3921508506 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66200c2f50 0x7f6620140760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:42.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.571+0000 7f661ed9d640 1 --2- 172.21.15.116:0/3921508506 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6620150d20 0x7f662014ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:42.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.571+0000 7f661e59c640 1 -- 172.21.15.116:0/3921508506 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6620150d20 msgr2=0x7f662014ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:42.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.571+0000 7f661e59c640 1 --2- 172.21.15.116:0/3921508506 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6620150d20 0x7f662014ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:42.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.571+0000 7f661e59c640 1 -- 172.21.15.116:0/3921508506 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66200c2f50 msgr2=0x7f6620140760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:42.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.571+0000 7f661e59c640 1 --2- 172.21.15.116:0/3921508506 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66200c2f50 0x7f6620140760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:42.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.571+0000 7f661e59c640 1 -- 172.21.15.116:0/3921508506 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6620151630 con 0x7f66200a94a0 2024-04-25T00:21:42.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.571+0000 7f661ed9d640 1 --2- 172.21.15.116:0/3921508506 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6620150d20 0x7f662014ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:21:42.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.571+0000 7f661e59c640 1 --2- 172.21.15.116:0/3921508506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66200a94a0 0x7f6620142150 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f661000d5f0 tx=0x7f661000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:42.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.571+0000 7f6616ffd640 1 -- 172.21.15.116:0/3921508506 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66100125f0 con 0x7f66200a94a0 2024-04-25T00:21:42.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.571+0000 7f6616ffd640 1 -- 172.21.15.116:0/3921508506 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6610019070 con 0x7f66200a94a0 2024-04-25T00:21:42.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.571+0000 7f662480b640 1 -- 172.21.15.116:0/3921508506 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6620151910 con 0x7f66200a94a0 2024-04-25T00:21:42.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.571+0000 7f6616ffd640 1 -- 172.21.15.116:0/3921508506 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66100129b0 con 0x7f66200a94a0 2024-04-25T00:21:42.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.571+0000 7f662480b640 1 -- 172.21.15.116:0/3921508506 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6620151ed0 con 0x7f66200a94a0 2024-04-25T00:21:42.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.571+0000 7f662480b640 1 -- 172.21.15.116:0/3921508506 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6620068350 con 0x7f66200a94a0 2024-04-25T00:21:42.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.575+0000 7f6616ffd640 1 -- 172.21.15.116:0/3921508506 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6610012b50 con 0x7f66200a94a0 2024-04-25T00:21:42.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.575+0000 7f6616ffd640 1 --2- 172.21.15.116:0/3921508506 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f66000678c0 0x7f6600069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:42.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.575+0000 7f6616ffd640 1 -- 172.21.15.116:0/3921508506 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 9418+0+0 (secure 0 0 0) 0x7f6610086e80 con 0x7f66200a94a0 2024-04-25T00:21:42.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.575+0000 7f6617fff640 1 --2- 172.21.15.116:0/3921508506 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f66000678c0 0x7f6600069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:42.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.575+0000 7f6617fff640 1 --2- 172.21.15.116:0/3921508506 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f66000678c0 0x7f6600069d80 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f66200ac370 tx=0x7f6608009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:42.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.579+0000 7f6616ffd640 1 -- 172.21.15.116:0/3921508506 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f6610056b80 con 0x7f66200a94a0 2024-04-25T00:21:42.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.775+0000 7f662480b640 1 -- 172.21.15.116:0/3921508506 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f66200c34f0 con 0x7f66200a94a0 2024-04-25T00:21:42.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.775+0000 7f6616ffd640 1 -- 172.21.15.116:0/3921508506 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 169 v169) ==== 95+0+5671 (secure 0 0 0) 0x7f6610056480 con 0x7f66200a94a0 2024-04-25T00:21:42.780 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:21:42.780 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":169,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":169,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:21:41.238596+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6125},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6125":{"gid":6125,"name":"d","rank":0,"incarnation":168,"state":"up:reconnect","state_seq":12,"addr":"172.21.15.174:6837/321466844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":321466844},{"type":"v1","addr":"172.21.15.174:6837","nonce":321466844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:20:13.935378+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7722":{"gid":7722,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7722,"qdb_cluster":[7722]},"id":2}]} 2024-04-25T00:21:42.781 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 169 2024-04-25T00:21:42.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.779+0000 7f662480b640 1 -- 172.21.15.116:0/3921508506 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f66000678c0 msgr2=0x7f6600069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:42.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.779+0000 7f662480b640 1 --2- 172.21.15.116:0/3921508506 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f66000678c0 0x7f6600069d80 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f66200ac370 tx=0x7f6608009040 comp rx=0 tx=0).stop 2024-04-25T00:21:42.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.779+0000 7f662480b640 1 -- 172.21.15.116:0/3921508506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66200a94a0 msgr2=0x7f6620142150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:42.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.779+0000 7f662480b640 1 --2- 172.21.15.116:0/3921508506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66200a94a0 0x7f6620142150 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f661000d5f0 tx=0x7f661000dab0 comp rx=0 tx=0).stop 2024-04-25T00:21:42.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.779+0000 7f662480b640 1 -- 172.21.15.116:0/3921508506 shutdown_connections 2024-04-25T00:21:42.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.779+0000 7f662480b640 1 --2- 172.21.15.116:0/3921508506 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f66000678c0 0x7f6600069d80 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:42.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.779+0000 7f662480b640 1 --2- 172.21.15.116:0/3921508506 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6620150d20 0x7f662014ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:42.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.779+0000 7f662480b640 1 --2- 172.21.15.116:0/3921508506 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66200c2f50 0x7f6620140760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:42.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.779+0000 7f662480b640 1 --2- 172.21.15.116:0/3921508506 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66200a94a0 0x7f6620142150 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:42.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.779+0000 7f662480b640 1 -- 172.21.15.116:0/3921508506 >> 172.21.15.116:0/3921508506 conn(0x7f66200c3b00 msgr2=0x7f6620146cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:21:42.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.779+0000 7f662480b640 1 -- 172.21.15.116:0/3921508506 shutdown_connections 2024-04-25T00:21:42.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.779+0000 7f662480b640 1 -- 172.21.15.116:0/3921508506 wait complete. 2024-04-25T00:21:42.799 INFO:tasks.mds_thrash.fs.[a]:mds.c down, removed from mdsmap 2024-04-25T00:21:42.799 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-04-25T00:21:42.800 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:21:42.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.923+0000 7f578c0a7640 1 Processor -- start 2024-04-25T00:21:42.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.923+0000 7f578c0a7640 1 -- start start 2024-04-25T00:21:42.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.923+0000 7f578c0a7640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5784150d20 0x7f57840c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:42.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.923+0000 7f578c0a7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f57840a94a0 0x7f5784153ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:42.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.923+0000 7f578c0a7640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57840c3910 con 0x7f57840c2f50 2024-04-25T00:21:42.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.923+0000 7f578c0a7640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f5784154530 con 0x7f57840a94a0 2024-04-25T00:21:42.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.923+0000 7f578c0a7640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f57840c16e0 con 0x7f5784150d20 2024-04-25T00:21:42.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.923+0000 7f578a644640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f57840a94a0 0x7f5784153ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:42.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.923+0000 7f578a644640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f57840a94a0 0x7f5784153ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52574/0 (socket says 172.21.15.116:52574) 2024-04-25T00:21:42.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.923+0000 7f578a644640 1 -- 172.21.15.116:0/4279747786 learned_addr learned my addr 172.21.15.116:0/4279747786 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:21:42.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.923+0000 7f5789e43640 1 --2- 172.21.15.116:0/4279747786 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5784150d20 0x7f57840c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:42.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.923+0000 7f578a644640 1 -- 172.21.15.116:0/4279747786 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5784150d20 msgr2=0x7f57840c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:21:42.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.923+0000 7f578a644640 1 --2- 172.21.15.116:0/4279747786 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5784150d20 0x7f57840c2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:42.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.923+0000 7f578a644640 1 -- 172.21.15.116:0/4279747786 >> v1:172.21.15.116:6789/0 conn(0x7f57840c2f50 legacy=0x7f57841538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:42.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.923+0000 7f5788e41640 1 -- 172.21.15.116:0/4279747786 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3038034690 0 0) 0x7f57840c3910 con 0x7f57840c2f50 2024-04-25T00:21:42.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.923+0000 7f578a644640 1 -- 172.21.15.116:0/4279747786 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57840c1400 con 0x7f57840a94a0 2024-04-25T00:21:42.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.923+0000 7f578a644640 1 --2- 172.21.15.116:0/4279747786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57840a94a0 0x7f5784153ff0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f577c009e50 tx=0x7f577c01ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91995dd8915d4af9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:42.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.923+0000 7f5788e41640 1 -- 172.21.15.116:0/4279747786 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f577c0a5810 con 0x7f57840a94a0 2024-04-25T00:21:42.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.927+0000 7f5788e41640 1 -- 172.21.15.116:0/4279747786 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f577c0a59b0 con 0x7f57840a94a0 2024-04-25T00:21:42.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.927+0000 7f5788e41640 1 -- 172.21.15.116:0/4279747786 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f577c0a5d80 con 0x7f57840a94a0 2024-04-25T00:21:42.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.927+0000 7f578c0a7640 1 -- 172.21.15.116:0/4279747786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57840a94a0 msgr2=0x7f5784153ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:42.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.927+0000 7f578c0a7640 1 --2- 172.21.15.116:0/4279747786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57840a94a0 0x7f5784153ff0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f577c009e50 tx=0x7f577c01ca30 comp rx=0 tx=0).stop 2024-04-25T00:21:42.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.927+0000 7f578c0a7640 1 -- 172.21.15.116:0/4279747786 shutdown_connections 2024-04-25T00:21:42.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.927+0000 7f578c0a7640 1 --2- 172.21.15.116:0/4279747786 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57840a94a0 0x7f5784153ff0 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:42.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.927+0000 7f578c0a7640 1 --2- 172.21.15.116:0/4279747786 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5784150d20 0x7f57840c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:42.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.927+0000 7f578c0a7640 1 -- 172.21.15.116:0/4279747786 >> 172.21.15.116:0/4279747786 conn(0x7f57840c3b00 msgr2=0x7f57840c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:21:42.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.927+0000 7f578c0a7640 1 -- 172.21.15.116:0/4279747786 shutdown_connections 2024-04-25T00:21:42.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.927+0000 7f578c0a7640 1 -- 172.21.15.116:0/4279747786 wait complete. 2024-04-25T00:21:42.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.927+0000 7f578c0a7640 1 Processor -- start 2024-04-25T00:21:42.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.927+0000 7f578c0a7640 1 -- start start 2024-04-25T00:21:42.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.927+0000 7f578c0a7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57840a94a0 0x7f578414c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:42.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.927+0000 7f578c0a7640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f57840c2f50 0x7f57841ee740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:42.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.927+0000 7f578c0a7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5784150d20 0x7f57840c5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:42.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.927+0000 7f578c0a7640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f57841608b0 con 0x7f57840a94a0 2024-04-25T00:21:42.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.927+0000 7f578c0a7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5784160730 con 0x7f5784150d20 2024-04-25T00:21:42.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.927+0000 7f578c0a7640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f5784160a30 con 0x7f57840c2f50 2024-04-25T00:21:42.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.927+0000 7f5789e43640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57840a94a0 0x7f578414c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:42.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.927+0000 7f5789e43640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57840a94a0 0x7f578414c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52578/0 (socket says 172.21.15.116:52578) 2024-04-25T00:21:42.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.927+0000 7f5789e43640 1 -- 172.21.15.116:0/3574110649 learned_addr learned my addr 172.21.15.116:0/3574110649 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:21:42.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.927+0000 7f578a644640 1 --2- 172.21.15.116:0/3574110649 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5784150d20 0x7f57840c5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:42.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.927+0000 7f5789642640 1 --2- 172.21.15.116:0/3574110649 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f57840c2f50 0x7f57841ee740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:42.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.927+0000 7f5789e43640 1 -- 172.21.15.116:0/3574110649 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f57840c2f50 msgr2=0x7f57841ee740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:42.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.927+0000 7f5789e43640 1 --2- 172.21.15.116:0/3574110649 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f57840c2f50 0x7f57841ee740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:42.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.927+0000 7f5789e43640 1 -- 172.21.15.116:0/3574110649 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5784150d20 msgr2=0x7f57840c5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:42.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.927+0000 7f5789e43640 1 --2- 172.21.15.116:0/3574110649 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5784150d20 0x7f57840c5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:42.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.927+0000 7f5789e43640 1 -- 172.21.15.116:0/3574110649 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57840c61d0 con 0x7f57840a94a0 2024-04-25T00:21:42.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.927+0000 7f578a644640 1 --2- 172.21.15.116:0/3574110649 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5784150d20 0x7f57840c5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:21:42.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.927+0000 7f5789e43640 1 --2- 172.21.15.116:0/3574110649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57840a94a0 0x7f578414c330 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f578000d5f0 tx=0x7f578000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:42.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.927+0000 7f577a7fc640 1 -- 172.21.15.116:0/3574110649 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57800125f0 con 0x7f57840a94a0 2024-04-25T00:21:42.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.927+0000 7f577a7fc640 1 -- 172.21.15.116:0/3574110649 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5780019070 con 0x7f57840a94a0 2024-04-25T00:21:42.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.927+0000 7f578c0a7640 1 -- 172.21.15.116:0/3574110649 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57840c64b0 con 0x7f57840a94a0 2024-04-25T00:21:42.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.927+0000 7f577a7fc640 1 -- 172.21.15.116:0/3574110649 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57800129b0 con 0x7f57840a94a0 2024-04-25T00:21:42.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.927+0000 7f578c0a7640 1 -- 172.21.15.116:0/3574110649 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57840c4c90 con 0x7f57840a94a0 2024-04-25T00:21:42.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.931+0000 7f578c0a7640 1 -- 172.21.15.116:0/3574110649 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5784068350 con 0x7f57840a94a0 2024-04-25T00:21:42.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.931+0000 7f577a7fc640 1 -- 172.21.15.116:0/3574110649 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5780012b50 con 0x7f57840a94a0 2024-04-25T00:21:42.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.931+0000 7f577a7fc640 1 --2- 172.21.15.116:0/3574110649 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f57680678c0 0x7f5768069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:42.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.931+0000 7f577a7fc640 1 -- 172.21.15.116:0/3574110649 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 9418+0+0 (secure 0 0 0) 0x7f5780086ec0 con 0x7f57840a94a0 2024-04-25T00:21:42.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.931+0000 7f5789642640 1 --2- 172.21.15.116:0/3574110649 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f57680678c0 0x7f5768069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:42.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.931+0000 7f5789642640 1 --2- 172.21.15.116:0/3574110649 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f57680678c0 0x7f5768069d80 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f57841541a0 tx=0x7f5774009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:42.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:42.935+0000 7f577a7fc640 1 -- 172.21.15.116:0/3574110649 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f5780011340 con 0x7f57840a94a0 2024-04-25T00:21:43.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:43.131+0000 7f578c0a7640 1 -- 172.21.15.116:0/3574110649 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f57840c34f0 con 0x7f57840a94a0 2024-04-25T00:21:43.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:43.135+0000 7f577a7fc640 1 -- 172.21.15.116:0/3574110649 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 170 v170) ==== 95+0+5668 (secure 0 0 0) 0x7f5780056bc0 con 0x7f57840a94a0 2024-04-25T00:21:43.137 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:21:43.137 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":170,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:21:41.883692+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6125},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6125":{"gid":6125,"name":"d","rank":0,"incarnation":168,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.174:6837/321466844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":321466844},{"type":"v1","addr":"172.21.15.174:6837","nonce":321466844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:20:13.935378+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7722":{"gid":7722,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7722,"qdb_cluster":[7722]},"id":2}]} 2024-04-25T00:21:43.138 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 170 2024-04-25T00:21:43.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:43.135+0000 7f578c0a7640 1 -- 172.21.15.116:0/3574110649 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f57680678c0 msgr2=0x7f5768069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:43.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:43.135+0000 7f578c0a7640 1 --2- 172.21.15.116:0/3574110649 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f57680678c0 0x7f5768069d80 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f57841541a0 tx=0x7f5774009040 comp rx=0 tx=0).stop 2024-04-25T00:21:43.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:43.135+0000 7f578c0a7640 1 -- 172.21.15.116:0/3574110649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57840a94a0 msgr2=0x7f578414c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:43.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:43.135+0000 7f578c0a7640 1 --2- 172.21.15.116:0/3574110649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57840a94a0 0x7f578414c330 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f578000d5f0 tx=0x7f578000dab0 comp rx=0 tx=0).stop 2024-04-25T00:21:43.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:43.135+0000 7f578c0a7640 1 -- 172.21.15.116:0/3574110649 shutdown_connections 2024-04-25T00:21:43.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:43.135+0000 7f578c0a7640 1 --2- 172.21.15.116:0/3574110649 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f57680678c0 0x7f5768069d80 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:43.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:43.135+0000 7f578c0a7640 1 --2- 172.21.15.116:0/3574110649 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5784150d20 0x7f57840c5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:43.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:43.135+0000 7f578c0a7640 1 --2- 172.21.15.116:0/3574110649 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f57840c2f50 0x7f57841ee740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:43.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:43.135+0000 7f578c0a7640 1 --2- 172.21.15.116:0/3574110649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f57840a94a0 0x7f578414c330 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:43.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:43.135+0000 7f578c0a7640 1 -- 172.21.15.116:0/3574110649 >> 172.21.15.116:0/3574110649 conn(0x7f57840c3b00 msgr2=0x7f5784146bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:21:43.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:43.135+0000 7f578c0a7640 1 -- 172.21.15.116:0/3574110649 shutdown_connections 2024-04-25T00:21:43.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:43.135+0000 7f578c0a7640 1 -- 172.21.15.116:0/3574110649 wait complete. 2024-04-25T00:21:43.158 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 170, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:04.678737+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 170, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6125": { "addr": "172.21.15.174:6837/321466844", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6836", "nonce": 321466844, "type": "v2" }, { "addr": "172.21.15.174:6837", "nonce": 321466844, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6125, "incarnation": 168, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:rejoin", "state_seq": 13 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T00:21:41.883692+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6125 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:18.825887+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 161, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_7722": { "addr": "172.21.15.116:6833/663910035", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6832", "nonce": 663910035, "type": "v2" }, { "addr": "172.21.15.116:6833", "nonce": 663910035, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7722, "incarnation": 158, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 11 } }, "last_failure": 0, "last_failure_osd_epoch": 58, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-04-25T00:20:13.935378+0000", "qdb_cluster": [ 7722 ], "qdb_leader": 7722, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7722 } } } ], "standbys": [ { "addr": "172.21.15.174:6835/4112145782", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6834", "nonce": 4112145782, "type": "v2" }, { "addr": "172.21.15.174:6835", "nonce": 4112145782, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 112, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5582, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-25T00:21:44.901 INFO:tasks.ceph.mds.d.smithi174.stderr:2024-04-25T00:21:44.899+0000 7f643a132640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T00:21:45.159 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:21:45.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.287+0000 7f4c82c62640 1 Processor -- start 2024-04-25T00:21:45.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.287+0000 7f4c82c62640 1 -- start start 2024-04-25T00:21:45.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.291+0000 7f4c82c62640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4c7c0a94a0 0x7f4c7c1563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:45.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.291+0000 7f4c82c62640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f4c7c071680 0x7f4c7c071ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:45.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.291+0000 7f4c82c62640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c7c071220 con 0x7f4c7c14c720 2024-04-25T00:21:45.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.291+0000 7f4c82c62640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f4c7c153460 con 0x7f4c7c071680 2024-04-25T00:21:45.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.291+0000 7f4c82c62640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f4c7c152420 con 0x7f4c7c0a94a0 2024-04-25T00:21:45.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.291+0000 7f4c7bfff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4c7c0a94a0 0x7f4c7c1563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:45.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.291+0000 7f4c811ff640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f4c7c071680 0x7f4c7c071ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:45.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.291+0000 7f4c809fe640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f4c7c14c720 0x7f4c7c0c2a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:59478/0 (socket says 172.21.15.116:59478) 2024-04-25T00:21:45.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.291+0000 7f4c809fe640 1 -- 172.21.15.116:0/3783077820 learned_addr learned my addr 172.21.15.116:0/3783077820 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:21:45.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.291+0000 7f4c811ff640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f4c7c071680 0x7f4c7c071ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:59852/0 (socket says 172.21.15.116:59852) 2024-04-25T00:21:45.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.291+0000 7f4c811ff640 1 -- 172.21.15.116:0/3783077820 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4c7c0a94a0 msgr2=0x7f4c7c1563d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:21:45.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.291+0000 7f4c811ff640 1 --2- 172.21.15.116:0/3783077820 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4c7c0a94a0 0x7f4c7c1563d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:45.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.291+0000 7f4c811ff640 1 -- 172.21.15.116:0/3783077820 >> v1:172.21.15.174:6789/0 conn(0x7f4c7c14c720 legacy=0x7f4c7c0c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:45.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.291+0000 7f4c811ff640 1 -- 172.21.15.116:0/3783077820 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c7c14cb00 con 0x7f4c7c071680 2024-04-25T00:21:45.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.295+0000 7f4c811ff640 1 --2- 172.21.15.116:0/3783077820 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4c7c071680 0x7f4c7c071ad0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f4c74009d90 tx=0x7f4c7401c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e0419b266567799 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:45.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.295+0000 7f4c7b7fe640 1 -- 172.21.15.116:0/3783077820 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c740b2070 con 0x7f4c7c071680 2024-04-25T00:21:45.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.295+0000 7f4c7b7fe640 1 -- 172.21.15.116:0/3783077820 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c740a5970 con 0x7f4c7c071680 2024-04-25T00:21:45.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.295+0000 7f4c7b7fe640 1 -- 172.21.15.116:0/3783077820 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c740a5d40 con 0x7f4c7c071680 2024-04-25T00:21:45.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.295+0000 7f4c82c62640 1 -- 172.21.15.116:0/3783077820 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4c7c071680 msgr2=0x7f4c7c071ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:45.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.295+0000 7f4c82c62640 1 --2- 172.21.15.116:0/3783077820 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4c7c071680 0x7f4c7c071ad0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f4c74009d90 tx=0x7f4c7401c8b0 comp rx=0 tx=0).stop 2024-04-25T00:21:45.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.295+0000 7f4c82c62640 1 -- 172.21.15.116:0/3783077820 shutdown_connections 2024-04-25T00:21:45.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.295+0000 7f4c82c62640 1 --2- 172.21.15.116:0/3783077820 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4c7c071680 0x7f4c7c071ad0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:45.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.295+0000 7f4c82c62640 1 --2- 172.21.15.116:0/3783077820 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4c7c0a94a0 0x7f4c7c1563d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:45.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.295+0000 7f4c82c62640 1 -- 172.21.15.116:0/3783077820 >> 172.21.15.116:0/3783077820 conn(0x7f4c7c0c3b00 msgr2=0x7f4c7c0c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:21:45.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.295+0000 7f4c82c62640 1 -- 172.21.15.116:0/3783077820 shutdown_connections 2024-04-25T00:21:45.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.295+0000 7f4c82c62640 1 -- 172.21.15.116:0/3783077820 wait complete. 2024-04-25T00:21:45.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.295+0000 7f4c82c62640 1 Processor -- start 2024-04-25T00:21:45.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.295+0000 7f4c82c62640 1 -- start start 2024-04-25T00:21:45.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.295+0000 7f4c82c62640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c7c071680 0x7f4c7c14b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:45.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.295+0000 7f4c82c62640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4c7c0a94a0 0x7f4c7c1f19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:45.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.295+0000 7f4c82c62640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4c7c14c720 0x7f4c7c1f1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:45.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.295+0000 7f4c82c62640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4c7c0c3120 con 0x7f4c7c071680 2024-04-25T00:21:45.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.295+0000 7f4c82c62640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4c7c152f40 con 0x7f4c7c14c720 2024-04-25T00:21:45.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.295+0000 7f4c82c62640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f4c7c0c32a0 con 0x7f4c7c0a94a0 2024-04-25T00:21:45.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.295+0000 7f4c809fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c7c071680 0x7f4c7c14b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:45.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.299+0000 7f4c809fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c7c071680 0x7f4c7c14b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49096/0 (socket says 172.21.15.116:49096) 2024-04-25T00:21:45.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.299+0000 7f4c809fe640 1 -- 172.21.15.116:0/3727825643 learned_addr learned my addr 172.21.15.116:0/3727825643 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:21:45.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.299+0000 7f4c811ff640 1 --2- 172.21.15.116:0/3727825643 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4c7c14c720 0x7f4c7c1f1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:45.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.299+0000 7f4c7bfff640 1 --2- 172.21.15.116:0/3727825643 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4c7c0a94a0 0x7f4c7c1f19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:45.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.299+0000 7f4c809fe640 1 -- 172.21.15.116:0/3727825643 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4c7c0a94a0 msgr2=0x7f4c7c1f19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:45.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.299+0000 7f4c809fe640 1 --2- 172.21.15.116:0/3727825643 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4c7c0a94a0 0x7f4c7c1f19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:45.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.299+0000 7f4c809fe640 1 -- 172.21.15.116:0/3727825643 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4c7c14c720 msgr2=0x7f4c7c1f1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:45.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.299+0000 7f4c809fe640 1 --2- 172.21.15.116:0/3727825643 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4c7c14c720 0x7f4c7c1f1ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:45.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.299+0000 7f4c809fe640 1 -- 172.21.15.116:0/3727825643 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c7c13e250 con 0x7f4c7c071680 2024-04-25T00:21:45.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.299+0000 7f4c809fe640 1 --2- 172.21.15.116:0/3727825643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c7c071680 0x7f4c7c14b1c0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f4c7000c370 tx=0x7f4c7000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:45.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.299+0000 7f4c797fa640 1 -- 172.21.15.116:0/3727825643 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c70017070 con 0x7f4c7c071680 2024-04-25T00:21:45.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.299+0000 7f4c82c62640 1 -- 172.21.15.116:0/3727825643 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c7c13fea0 con 0x7f4c7c071680 2024-04-25T00:21:45.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.299+0000 7f4c82c62640 1 -- 172.21.15.116:0/3727825643 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c7c140460 con 0x7f4c7c071680 2024-04-25T00:21:45.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.299+0000 7f4c797fa640 1 -- 172.21.15.116:0/3727825643 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c70005cf0 con 0x7f4c7c071680 2024-04-25T00:21:45.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.299+0000 7f4c797fa640 1 -- 172.21.15.116:0/3727825643 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c700072d0 con 0x7f4c7c071680 2024-04-25T00:21:45.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.299+0000 7f4c82c62640 1 -- 172.21.15.116:0/3727825643 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c7c068350 con 0x7f4c7c071680 2024-04-25T00:21:45.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.299+0000 7f4c797fa640 1 -- 172.21.15.116:0/3727825643 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4c7001a070 con 0x7f4c7c071680 2024-04-25T00:21:45.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.299+0000 7f4c797fa640 1 --2- 172.21.15.116:0/3727825643 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4c600678f0 0x7f4c60069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:45.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.299+0000 7f4c797fa640 1 -- 172.21.15.116:0/3727825643 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 9418+0+0 (secure 0 0 0) 0x7f4c70085d70 con 0x7f4c7c071680 2024-04-25T00:21:45.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.299+0000 7f4c7bfff640 1 --2- 172.21.15.116:0/3727825643 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4c600678f0 0x7f4c60069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:45.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.303+0000 7f4c7bfff640 1 --2- 172.21.15.116:0/3727825643 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4c600678f0 0x7f4c60069db0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c150060 tx=0x7f4c68008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:45.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.303+0000 7f4c797fa640 1 -- 172.21.15.116:0/3727825643 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f4c7c13e250 con 0x7f4c7c071680 2024-04-25T00:21:45.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.535+0000 7f4c82c62640 1 -- 172.21.15.116:0/3727825643 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4c7c0c3640 con 0x7f4c7c071680 2024-04-25T00:21:45.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.539+0000 7f4c797fa640 1 -- 172.21.15.116:0/3727825643 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) ==== 95+0+5675 (secure 0 0 0) 0x7f4c700553a0 con 0x7f4c7c071680 2024-04-25T00:21:45.538 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:21:45.538 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":171,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":171,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:21:43.889982+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6125},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6125":{"gid":6125,"name":"d","rank":0,"incarnation":168,"state":"up:active","state_seq":14,"addr":"172.21.15.174:6837/321466844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":321466844},{"type":"v1","addr":"172.21.15.174:6837","nonce":321466844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6125,"qdb_cluster":[6125]},"id":1},{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:20:13.935378+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7722":{"gid":7722,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7722,"qdb_cluster":[7722]},"id":2}]} 2024-04-25T00:21:45.538 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 171 2024-04-25T00:21:45.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.539+0000 7f4c82c62640 1 -- 172.21.15.116:0/3727825643 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4c600678f0 msgr2=0x7f4c60069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:45.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.539+0000 7f4c82c62640 1 --2- 172.21.15.116:0/3727825643 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4c600678f0 0x7f4c60069db0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c150060 tx=0x7f4c68008040 comp rx=0 tx=0).stop 2024-04-25T00:21:45.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.539+0000 7f4c82c62640 1 -- 172.21.15.116:0/3727825643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c7c071680 msgr2=0x7f4c7c14b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:45.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.539+0000 7f4c82c62640 1 --2- 172.21.15.116:0/3727825643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c7c071680 0x7f4c7c14b1c0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f4c7000c370 tx=0x7f4c7000c830 comp rx=0 tx=0).stop 2024-04-25T00:21:45.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.539+0000 7f4c82c62640 1 -- 172.21.15.116:0/3727825643 shutdown_connections 2024-04-25T00:21:45.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.539+0000 7f4c82c62640 1 --2- 172.21.15.116:0/3727825643 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4c600678f0 0x7f4c60069db0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:45.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.539+0000 7f4c82c62640 1 --2- 172.21.15.116:0/3727825643 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4c7c14c720 0x7f4c7c1f1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:45.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.539+0000 7f4c82c62640 1 --2- 172.21.15.116:0/3727825643 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4c7c0a94a0 0x7f4c7c1f19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:45.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.539+0000 7f4c82c62640 1 --2- 172.21.15.116:0/3727825643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c7c071680 0x7f4c7c14b1c0 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:45.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.539+0000 7f4c82c62640 1 -- 172.21.15.116:0/3727825643 >> 172.21.15.116:0/3727825643 conn(0x7f4c7c0c3b00 msgr2=0x7f4c7c0c2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:21:45.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.539+0000 7f4c82c62640 1 -- 172.21.15.116:0/3727825643 shutdown_connections 2024-04-25T00:21:45.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:45.539+0000 7f4c82c62640 1 -- 172.21.15.116:0/3727825643 wait complete. 2024-04-25T00:21:45.558 INFO:tasks.mds_thrash.fs.[a]:mds.d has gained rank=0, replacing gid=7809 2024-04-25T00:21:45.558 INFO:tasks.mds_thrash.fs.[a]:waiting for 1 secs before reviving mds.c 2024-04-25T00:21:46.559 INFO:tasks.mds_thrash.fs.[a]:reviving mds.c 2024-04-25T00:21:46.560 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-25T00:21:46.560 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-25T00:21:46.563 INFO:tasks.ceph.mds.c:Started 2024-04-25T00:21:46.563 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:21:46.646 INFO:tasks.ceph.mds.c.smithi116.stdout:starting mds.c at 2024-04-25T00:21:46.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.711+0000 7fef8a9d0640 1 Processor -- start 2024-04-25T00:21:46.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.711+0000 7fef8a9d0640 1 -- start start 2024-04-25T00:21:46.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.711+0000 7fef8a9d0640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fef8406d5f0 0x7fef84156c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:46.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.711+0000 7fef8a9d0640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fef84155810 0x7fef841571c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:46.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.711+0000 7fef8a9d0640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef840c9150 con 0x7fef84152d80 2024-04-25T00:21:46.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.711+0000 7fef8a9d0640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fef840c9d60 con 0x7fef84155810 2024-04-25T00:21:46.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.711+0000 7fef8a9d0640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fef840c9430 con 0x7fef8406d5f0 2024-04-25T00:21:46.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.715+0000 7fef88f6d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fef84155810 0x7fef841571c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:46.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.715+0000 7fef83fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fef84152d80 0x7fef8406bd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:37864/0 (socket says 172.21.15.116:37864) 2024-04-25T00:21:46.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.715+0000 7fef83fff640 1 -- 172.21.15.116:0/2959338790 learned_addr learned my addr 172.21.15.116:0/2959338790 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:21:46.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.715+0000 7fef837fe640 1 --2- 172.21.15.116:0/2959338790 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fef8406d5f0 0x7fef84156c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:46.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.715+0000 7fef88f6d640 1 -- 172.21.15.116:0/2959338790 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fef8406d5f0 msgr2=0x7fef84156c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:21:46.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.715+0000 7fef88f6d640 1 --2- 172.21.15.116:0/2959338790 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fef8406d5f0 0x7fef84156c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:46.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.715+0000 7fef88f6d640 1 -- 172.21.15.116:0/2959338790 >> v1:172.21.15.116:6789/0 conn(0x7fef84152d80 legacy=0x7fef8406bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:46.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.715+0000 7fef88f6d640 1 -- 172.21.15.116:0/2959338790 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef840c9710 con 0x7fef84155810 2024-04-25T00:21:46.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.715+0000 7fef82ffd640 1 -- 172.21.15.116:0/2959338790 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3070921497 0 0) 0x7fef840c9150 con 0x7fef84152d80 2024-04-25T00:21:46.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.715+0000 7fef88f6d640 1 --2- 172.21.15.116:0/2959338790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef84155810 0x7fef841571c0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7fef7c009d90 tx=0x7fef7c01c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87d89fc11e6047ad server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:46.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.715+0000 7fef82ffd640 1 -- 172.21.15.116:0/2959338790 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef7c0b2070 con 0x7fef84155810 2024-04-25T00:21:46.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.715+0000 7fef82ffd640 1 -- 172.21.15.116:0/2959338790 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef7c0a5810 con 0x7fef84155810 2024-04-25T00:21:46.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.715+0000 7fef82ffd640 1 -- 172.21.15.116:0/2959338790 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef7c0a5be0 con 0x7fef84155810 2024-04-25T00:21:46.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.715+0000 7fef8a9d0640 1 -- 172.21.15.116:0/2959338790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef84155810 msgr2=0x7fef841571c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:46.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.715+0000 7fef8a9d0640 1 --2- 172.21.15.116:0/2959338790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef84155810 0x7fef841571c0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7fef7c009d90 tx=0x7fef7c01c820 comp rx=0 tx=0).stop 2024-04-25T00:21:46.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.715+0000 7fef8a9d0640 1 -- 172.21.15.116:0/2959338790 shutdown_connections 2024-04-25T00:21:46.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.715+0000 7fef8a9d0640 1 --2- 172.21.15.116:0/2959338790 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef84155810 0x7fef841571c0 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:46.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.715+0000 7fef8a9d0640 1 --2- 172.21.15.116:0/2959338790 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fef8406d5f0 0x7fef84156c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:46.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.715+0000 7fef8a9d0640 1 -- 172.21.15.116:0/2959338790 >> 172.21.15.116:0/2959338790 conn(0x7fef840acdd0 msgr2=0x7fef840ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:21:46.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.715+0000 7fef8a9d0640 1 -- 172.21.15.116:0/2959338790 shutdown_connections 2024-04-25T00:21:46.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.715+0000 7fef8a9d0640 1 -- 172.21.15.116:0/2959338790 wait complete. 2024-04-25T00:21:46.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.719+0000 7fef8a9d0640 1 Processor -- start 2024-04-25T00:21:46.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.719+0000 7fef8a9d0640 1 -- start start 2024-04-25T00:21:46.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.719+0000 7fef8a9d0640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fef8406d5f0 0x7fef840ac120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:46.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.719+0000 7fef8a9d0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef84152d80 0x7fef840c0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:46.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.719+0000 7fef8a9d0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef84155810 0x7fef840c0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:46.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.719+0000 7fef8a9d0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fef840caf00 con 0x7fef84155810 2024-04-25T00:21:46.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.719+0000 7fef8a9d0640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fef840cad80 con 0x7fef84152d80 2024-04-25T00:21:46.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.719+0000 7fef8a9d0640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fef841628d0 con 0x7fef8406d5f0 2024-04-25T00:21:46.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.719+0000 7fef88f6d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef84155810 0x7fef840c0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:46.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.719+0000 7fef88f6d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef84155810 0x7fef840c0bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:49140/0 (socket says 172.21.15.116:49140) 2024-04-25T00:21:46.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.719+0000 7fef88f6d640 1 -- 172.21.15.116:0/3511981475 learned_addr learned my addr 172.21.15.116:0/3511981475 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:21:46.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.719+0000 7fef83fff640 1 --2- 172.21.15.116:0/3511981475 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fef8406d5f0 0x7fef840ac120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:46.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.719+0000 7fef837fe640 1 --2- 172.21.15.116:0/3511981475 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef84152d80 0x7fef840c0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:46.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.719+0000 7fef88f6d640 1 -- 172.21.15.116:0/3511981475 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fef8406d5f0 msgr2=0x7fef840ac120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:46.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.719+0000 7fef88f6d640 1 --2- 172.21.15.116:0/3511981475 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fef8406d5f0 0x7fef840ac120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:46.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.719+0000 7fef88f6d640 1 -- 172.21.15.116:0/3511981475 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef84152d80 msgr2=0x7fef840c0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:46.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.719+0000 7fef88f6d640 1 --2- 172.21.15.116:0/3511981475 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef84152d80 0x7fef840c0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:46.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.719+0000 7fef88f6d640 1 -- 172.21.15.116:0/3511981475 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef841544f0 con 0x7fef84155810 2024-04-25T00:21:46.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.719+0000 7fef88f6d640 1 --2- 172.21.15.116:0/3511981475 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef84155810 0x7fef840c0bb0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7fef7c01aed0 tx=0x7fef7c0195c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:46.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.719+0000 7fef80ff9640 1 -- 172.21.15.116:0/3511981475 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef7c0bd020 con 0x7fef84155810 2024-04-25T00:21:46.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.719+0000 7fef80ff9640 1 -- 172.21.15.116:0/3511981475 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef7c0b8670 con 0x7fef84155810 2024-04-25T00:21:46.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.719+0000 7fef80ff9640 1 -- 172.21.15.116:0/3511981475 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef7c0b2040 con 0x7fef84155810 2024-04-25T00:21:46.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.719+0000 7fef8a9d0640 1 -- 172.21.15.116:0/3511981475 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef841547d0 con 0x7fef84155810 2024-04-25T00:21:46.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.719+0000 7fef8a9d0640 1 -- 172.21.15.116:0/3511981475 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef840ab6c0 con 0x7fef84155810 2024-04-25T00:21:46.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.719+0000 7fef8a9d0640 1 -- 172.21.15.116:0/3511981475 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef84068350 con 0x7fef84155810 2024-04-25T00:21:46.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.723+0000 7fef80ff9640 1 -- 172.21.15.116:0/3511981475 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fef7c0c0020 con 0x7fef84155810 2024-04-25T00:21:46.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.723+0000 7fef80ff9640 1 --2- 172.21.15.116:0/3511981475 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fef700678c0 0x7fef70069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:46.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.723+0000 7fef80ff9640 1 -- 172.21.15.116:0/3511981475 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 9418+0+0 (secure 0 0 0) 0x7fef7c121a70 con 0x7fef84155810 2024-04-25T00:21:46.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.723+0000 7fef83fff640 1 --2- 172.21.15.116:0/3511981475 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fef700678c0 0x7fef70069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:46.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.723+0000 7fef83fff640 1 --2- 172.21.15.116:0/3511981475 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fef700678c0 0x7fef70069d80 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fef74000f70 tx=0x7fef74007c10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:46.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.727+0000 7fef80ff9640 1 -- 172.21.15.116:0/3511981475 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fef7c0c7030 con 0x7fef84155810 2024-04-25T00:21:46.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.971+0000 7fef8a9d0640 1 -- 172.21.15.116:0/3511981475 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fef8406c160 con 0x7fef84155810 2024-04-25T00:21:46.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.971+0000 7fef80ff9640 1 -- 172.21.15.116:0/3511981475 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) ==== 95+0+6501 (secure 0 0 0) 0x7fef7c016560 con 0x7fef84155810 2024-04-25T00:21:46.971 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:21:46.971 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":172,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7989,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/221971311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":221971311},{"type":"v1","addr":"172.21.15.116:6835","nonce":221971311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":172}],"filesystems":[{"mdsmap":{"epoch":171,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:21:43.889982+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6125},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6125":{"gid":6125,"name":"d","rank":0,"incarnation":168,"state":"up:active","state_seq":14,"addr":"172.21.15.174:6837/321466844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":321466844},{"type":"v1","addr":"172.21.15.174:6837","nonce":321466844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6125,"qdb_cluster":[6125]},"id":1},{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:20:13.935378+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7722":{"gid":7722,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7722,"qdb_cluster":[7722]},"id":2}]} 2024-04-25T00:21:46.971 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 172 2024-04-25T00:21:46.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.975+0000 7fef8a9d0640 1 -- 172.21.15.116:0/3511981475 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fef700678c0 msgr2=0x7fef70069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:46.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.975+0000 7fef8a9d0640 1 --2- 172.21.15.116:0/3511981475 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fef700678c0 0x7fef70069d80 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fef74000f70 tx=0x7fef74007c10 comp rx=0 tx=0).stop 2024-04-25T00:21:46.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.975+0000 7fef8a9d0640 1 -- 172.21.15.116:0/3511981475 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef84155810 msgr2=0x7fef840c0bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:46.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.975+0000 7fef8a9d0640 1 --2- 172.21.15.116:0/3511981475 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef84155810 0x7fef840c0bb0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7fef7c01aed0 tx=0x7fef7c0195c0 comp rx=0 tx=0).stop 2024-04-25T00:21:46.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.975+0000 7fef8a9d0640 1 -- 172.21.15.116:0/3511981475 shutdown_connections 2024-04-25T00:21:46.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.975+0000 7fef8a9d0640 1 --2- 172.21.15.116:0/3511981475 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fef700678c0 0x7fef70069d80 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:46.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.975+0000 7fef8a9d0640 1 --2- 172.21.15.116:0/3511981475 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef84155810 0x7fef840c0bb0 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:46.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.975+0000 7fef8a9d0640 1 --2- 172.21.15.116:0/3511981475 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef84152d80 0x7fef840c0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:46.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.975+0000 7fef8a9d0640 1 --2- 172.21.15.116:0/3511981475 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fef8406d5f0 0x7fef840ac120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:46.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.975+0000 7fef8a9d0640 1 -- 172.21.15.116:0/3511981475 >> 172.21.15.116:0/3511981475 conn(0x7fef840acdd0 msgr2=0x7fef8414ed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:21:46.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.975+0000 7fef8a9d0640 1 -- 172.21.15.116:0/3511981475 shutdown_connections 2024-04-25T00:21:46.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:46.975+0000 7fef8a9d0640 1 -- 172.21.15.116:0/3511981475 wait complete. 2024-04-25T00:21:46.993 INFO:tasks.mds_thrash.fs.[a]:mds.c reported in up:standby state 2024-04-25T00:21:46.993 INFO:tasks.mds_thrash.fs.[a]:waiting for 11 secs before thrashing 2024-04-25T00:21:57.993 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:21:58.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.131+0000 7f53f2426640 1 Processor -- start 2024-04-25T00:21:58.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.131+0000 7f53f2426640 1 -- start start 2024-04-25T00:21:58.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.135+0000 7f53f2426640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f53ec150d10 0x7f53ec0c5010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:58.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.135+0000 7f53f2426640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53ec0c4950 con 0x7f53ec0c5550 2024-04-25T00:21:58.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.135+0000 7f53f2426640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53ec1510f0 con 0x7f53ec072ca0 2024-04-25T00:21:58.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.135+0000 7f53f2426640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f53ec157a20 con 0x7f53ec150d10 2024-04-25T00:21:58.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.135+0000 7f53ebfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f53ec150d10 0x7f53ec0c5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:58.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.135+0000 7f53ebfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f53ec150d10 0x7f53ec0c5010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46698/0 (socket says 172.21.15.116:46698) 2024-04-25T00:21:58.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.135+0000 7f53e3fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f53ec0c5550 0x7f53ec153a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:44530/0 (socket says 172.21.15.116:44530) 2024-04-25T00:21:58.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.135+0000 7f53ebfff640 1 -- 172.21.15.116:0/1359957094 learned_addr learned my addr 172.21.15.116:0/1359957094 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:21:58.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.135+0000 7f53ebfff640 1 -- 172.21.15.116:0/1359957094 >> v1:172.21.15.174:6789/0 conn(0x7f53ec072ca0 legacy=0x7f53ec15e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:21:58.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.135+0000 7f53eb7fe640 1 -- 172.21.15.116:0/1359957094 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3842727929 0 0) 0x7f53ec0c4950 con 0x7f53ec0c5550 2024-04-25T00:21:58.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.135+0000 7f53ebfff640 1 -- 172.21.15.116:0/1359957094 >> v1:172.21.15.116:6789/0 conn(0x7f53ec0c5550 legacy=0x7f53ec153a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:58.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.135+0000 7f53ebfff640 1 -- 172.21.15.116:0/1359957094 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53ec0c2c90 con 0x7f53ec150d10 2024-04-25T00:21:58.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.135+0000 7f53ebfff640 1 --2- 172.21.15.116:0/1359957094 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53ec150d10 0x7f53ec0c5010 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f53dc009870 tx=0x7f53dc01c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ddc159120a9cbf8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:58.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.135+0000 7f53eb7fe640 1 -- 172.21.15.116:0/1359957094 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53dc0ab660 con 0x7f53ec150d10 2024-04-25T00:21:58.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.135+0000 7f53eb7fe640 1 -- 172.21.15.116:0/1359957094 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53dc0b0070 con 0x7f53ec150d10 2024-04-25T00:21:58.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.135+0000 7f53eb7fe640 1 -- 172.21.15.116:0/1359957094 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53dc0aba20 con 0x7f53ec150d10 2024-04-25T00:21:58.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.139+0000 7f53f2426640 1 -- 172.21.15.116:0/1359957094 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53ec150d10 msgr2=0x7f53ec0c5010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:58.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.139+0000 7f53f2426640 1 --2- 172.21.15.116:0/1359957094 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53ec150d10 0x7f53ec0c5010 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f53dc009870 tx=0x7f53dc01c440 comp rx=0 tx=0).stop 2024-04-25T00:21:58.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.139+0000 7f53f2426640 1 -- 172.21.15.116:0/1359957094 shutdown_connections 2024-04-25T00:21:58.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.139+0000 7f53f2426640 1 --2- 172.21.15.116:0/1359957094 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53ec150d10 0x7f53ec0c5010 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:58.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.139+0000 7f53f2426640 1 -- 172.21.15.116:0/1359957094 >> 172.21.15.116:0/1359957094 conn(0x7f53ec06c720 msgr2=0x7f53ec06cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:21:58.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.139+0000 7f53f2426640 1 -- 172.21.15.116:0/1359957094 shutdown_connections 2024-04-25T00:21:58.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.139+0000 7f53f2426640 1 -- 172.21.15.116:0/1359957094 wait complete. 2024-04-25T00:21:58.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.139+0000 7f53f2426640 1 Processor -- start 2024-04-25T00:21:58.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.139+0000 7f53f2426640 1 -- start start 2024-04-25T00:21:58.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.139+0000 7f53f2426640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f53ec072ca0 0x7f53ec06f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:58.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.139+0000 7f53f2426640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f53ec0c5550 0x7f53ec140360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:58.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.139+0000 7f53f2426640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53ec150d10 0x7f53ec1408a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:58.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.139+0000 7f53f2426640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f53ec0c2640 con 0x7f53ec150d10 2024-04-25T00:21:58.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.139+0000 7f53f2426640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f53ec1576a0 con 0x7f53ec072ca0 2024-04-25T00:21:58.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.139+0000 7f53f2426640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f53ec0c27c0 con 0x7f53ec0c5550 2024-04-25T00:21:58.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.139+0000 7f53f09c3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53ec150d10 0x7f53ec1408a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:58.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.139+0000 7f53f09c3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53ec150d10 0x7f53ec1408a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46700/0 (socket says 172.21.15.116:46700) 2024-04-25T00:21:58.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.139+0000 7f53f09c3640 1 -- 172.21.15.116:0/1941169009 learned_addr learned my addr 172.21.15.116:0/1941169009 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:21:58.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.139+0000 7f53ebfff640 1 --2- 172.21.15.116:0/1941169009 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f53ec072ca0 0x7f53ec06f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:58.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.139+0000 7f53e3fff640 1 --2- 172.21.15.116:0/1941169009 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f53ec0c5550 0x7f53ec140360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:58.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.139+0000 7f53f09c3640 1 -- 172.21.15.116:0/1941169009 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f53ec0c5550 msgr2=0x7f53ec140360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:58.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.139+0000 7f53f09c3640 1 --2- 172.21.15.116:0/1941169009 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f53ec0c5550 0x7f53ec140360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:58.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.139+0000 7f53f09c3640 1 -- 172.21.15.116:0/1941169009 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f53ec072ca0 msgr2=0x7f53ec06f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:58.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.139+0000 7f53f09c3640 1 --2- 172.21.15.116:0/1941169009 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f53ec072ca0 0x7f53ec06f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:58.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.139+0000 7f53f09c3640 1 -- 172.21.15.116:0/1941169009 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53ec141200 con 0x7f53ec150d10 2024-04-25T00:21:58.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.139+0000 7f53f09c3640 1 --2- 172.21.15.116:0/1941169009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53ec150d10 0x7f53ec1408a0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f53e400c7a0 tx=0x7f53e400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:58.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.139+0000 7f53e97fa640 1 -- 172.21.15.116:0/1941169009 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53e4003b80 con 0x7f53ec150d10 2024-04-25T00:21:58.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.139+0000 7f53e97fa640 1 -- 172.21.15.116:0/1941169009 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53e4003d20 con 0x7f53ec150d10 2024-04-25T00:21:58.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.139+0000 7f53e97fa640 1 -- 172.21.15.116:0/1941169009 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53e400aa90 con 0x7f53ec150d10 2024-04-25T00:21:58.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.139+0000 7f53f2426640 1 -- 172.21.15.116:0/1941169009 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53ec147210 con 0x7f53ec150d10 2024-04-25T00:21:58.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.139+0000 7f53f2426640 1 -- 172.21.15.116:0/1941169009 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53ec06f800 con 0x7f53ec150d10 2024-04-25T00:21:58.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.143+0000 7f53f2426640 1 -- 172.21.15.116:0/1941169009 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53ec068a60 con 0x7f53ec150d10 2024-04-25T00:21:58.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.143+0000 7f53e97fa640 1 -- 172.21.15.116:0/1941169009 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f53e400ac30 con 0x7f53ec150d10 2024-04-25T00:21:58.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.143+0000 7f53e97fa640 1 --2- 172.21.15.116:0/1941169009 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f53cc0678c0 0x7f53cc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:58.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.143+0000 7f53e97fa640 1 -- 172.21.15.116:0/1941169009 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 9418+0+0 (secure 0 0 0) 0x7f53e4086a10 con 0x7f53ec150d10 2024-04-25T00:21:58.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.143+0000 7f53ebfff640 1 --2- 172.21.15.116:0/1941169009 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f53cc0678c0 0x7f53cc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:58.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.143+0000 7f53ebfff640 1 --2- 172.21.15.116:0/1941169009 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f53cc0678c0 0x7f53cc069d80 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f53dc0af000 tx=0x7f53dc0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:58.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.147+0000 7f53e97fa640 1 -- 172.21.15.116:0/1941169009 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f53e4056710 con 0x7f53ec150d10 2024-04-25T00:21:58.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.391+0000 7f53f2426640 1 -- 172.21.15.116:0/1941169009 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f53ec072a80 con 0x7f53ec150d10 2024-04-25T00:21:58.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.395+0000 7f53e97fa640 1 -- 172.21.15.116:0/1941169009 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) ==== 95+0+6501 (secure 0 0 0) 0x7f53e4056010 con 0x7f53ec150d10 2024-04-25T00:21:58.393 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:21:58.393 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":172,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7989,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/221971311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":221971311},{"type":"v1","addr":"172.21.15.116:6835","nonce":221971311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":172}],"filesystems":[{"mdsmap":{"epoch":171,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:21:43.889982+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6125},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6125":{"gid":6125,"name":"d","rank":0,"incarnation":168,"state":"up:active","state_seq":14,"addr":"172.21.15.174:6837/321466844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":321466844},{"type":"v1","addr":"172.21.15.174:6837","nonce":321466844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6125,"qdb_cluster":[6125]},"id":1},{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:20:13.935378+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7722":{"gid":7722,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7722,"qdb_cluster":[7722]},"id":2}]} 2024-04-25T00:21:58.394 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 172 2024-04-25T00:21:58.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.395+0000 7f53f2426640 1 -- 172.21.15.116:0/1941169009 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f53cc0678c0 msgr2=0x7f53cc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:58.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.395+0000 7f53f2426640 1 --2- 172.21.15.116:0/1941169009 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f53cc0678c0 0x7f53cc069d80 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f53dc0af000 tx=0x7f53dc0a7000 comp rx=0 tx=0).stop 2024-04-25T00:21:58.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.395+0000 7f53f2426640 1 -- 172.21.15.116:0/1941169009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53ec150d10 msgr2=0x7f53ec1408a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:58.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.395+0000 7f53f2426640 1 --2- 172.21.15.116:0/1941169009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53ec150d10 0x7f53ec1408a0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f53e400c7a0 tx=0x7f53e400cc60 comp rx=0 tx=0).stop 2024-04-25T00:21:58.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.395+0000 7f53f2426640 1 -- 172.21.15.116:0/1941169009 shutdown_connections 2024-04-25T00:21:58.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.395+0000 7f53f2426640 1 --2- 172.21.15.116:0/1941169009 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f53cc0678c0 0x7f53cc069d80 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:58.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.395+0000 7f53f2426640 1 --2- 172.21.15.116:0/1941169009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53ec150d10 0x7f53ec1408a0 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:58.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.395+0000 7f53f2426640 1 --2- 172.21.15.116:0/1941169009 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f53ec0c5550 0x7f53ec140360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:58.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.395+0000 7f53f2426640 1 --2- 172.21.15.116:0/1941169009 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f53ec072ca0 0x7f53ec06f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:58.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.395+0000 7f53f2426640 1 -- 172.21.15.116:0/1941169009 >> 172.21.15.116:0/1941169009 conn(0x7f53ec06c720 msgr2=0x7f53ec0a9c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:21:58.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.395+0000 7f53f2426640 1 -- 172.21.15.116:0/1941169009 shutdown_connections 2024-04-25T00:21:58.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.395+0000 7f53f2426640 1 -- 172.21.15.116:0/1941169009 wait complete. 2024-04-25T00:21:58.412 INFO:tasks.mds_thrash.fs.[a]:kill mds.d (rank=0) 2024-04-25T00:21:58.413 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-25T00:21:58.413 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:21:58.428 INFO:tasks.ceph.mds.d:Stopped 2024-04-25T00:21:58.428 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:21:58.503 INFO:tasks.workunit.client.0.smithi116.stdout:fs setup took 1097 secs 2024-04-25T00:21:58.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.543+0000 7fec8b32a640 1 Processor -- start 2024-04-25T00:21:58.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.543+0000 7fec8b32a640 1 -- start start 2024-04-25T00:21:58.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.543+0000 7fec8b32a640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fec840c4c20 0x7fec84154170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:58.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.543+0000 7fec8b32a640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec84070180 con 0x7fec84150d40 2024-04-25T00:21:58.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.543+0000 7fec8b32a640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec84070460 con 0x7fec84070770 2024-04-25T00:21:58.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.543+0000 7fec8b32a640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fec840c7530 con 0x7fec840c4c20 2024-04-25T00:21:58.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.543+0000 7fec890c6640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fec84150d40 0x7fec840c6e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:44532/0 (socket says 172.21.15.116:44532) 2024-04-25T00:21:58.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.543+0000 7fec890c6640 1 -- 172.21.15.116:0/3864942571 learned_addr learned my addr 172.21.15.116:0/3864942571 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:21:58.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.543+0000 7fec888c5640 1 --2- 172.21.15.116:0/3864942571 >> v2:172.21.15.174:3300/0 conn(0x7fec840c4c20 0x7fec84154170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:58.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.543+0000 7fec77fff640 1 -- 172.21.15.116:0/3864942571 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1565330540 0 0) 0x7fec84070180 con 0x7fec84150d40 2024-04-25T00:21:58.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.543+0000 7fec77fff640 1 -- 172.21.15.116:0/3864942571 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec70003530 con 0x7fec84150d40 2024-04-25T00:21:58.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.543+0000 7fec77fff640 1 -- 172.21.15.116:0/3864942571 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3732224766 0 0) 0x7fec84070460 con 0x7fec84070770 2024-04-25T00:21:58.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.543+0000 7fec77fff640 1 -- 172.21.15.116:0/3864942571 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec84070180 con 0x7fec84070770 2024-04-25T00:21:58.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.543+0000 7fec77fff640 1 -- 172.21.15.116:0/3864942571 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3942497174 0 0) 0x7fec70003530 con 0x7fec84150d40 2024-04-25T00:21:58.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.543+0000 7fec77fff640 1 -- 172.21.15.116:0/3864942571 >> v2:172.21.15.174:3300/0 conn(0x7fec840c4c20 msgr2=0x7fec84154170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:21:58.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.543+0000 7fec77fff640 1 --2- 172.21.15.116:0/3864942571 >> v2:172.21.15.174:3300/0 conn(0x7fec840c4c20 0x7fec84154170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:58.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.543+0000 7fec77fff640 1 -- 172.21.15.116:0/3864942571 >> v1:172.21.15.174:6789/0 conn(0x7fec84070770 legacy=0x7fec840ab200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:58.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.543+0000 7fec77fff640 1 -- 172.21.15.116:0/3864942571 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec84153970 con 0x7fec84150d40 2024-04-25T00:21:58.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.543+0000 7fec77fff640 1 -- 172.21.15.116:0/3864942571 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7fec80001680 con 0x7fec84150d40 2024-04-25T00:21:58.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.543+0000 7fec77fff640 1 -- 172.21.15.116:0/3864942571 >> v1:172.21.15.116:6789/0 conn(0x7fec84150d40 legacy=0x7fec840c6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:58.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.543+0000 7fec77fff640 1 --2- 172.21.15.116:0/3864942571 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec70003b50 0x7fec70003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:58.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.543+0000 7fec77fff640 1 --2- 172.21.15.116:0/3864942571 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fec70004810 0x7fec70004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:58.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.543+0000 7fec77fff640 1 --2- 172.21.15.116:0/3864942571 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fec700051d0 0x7fec7000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:58.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.543+0000 7fec77fff640 1 -- 172.21.15.116:0/3864942571 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fec840c7530 con 0x7fec70003b50 2024-04-25T00:21:58.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.543+0000 7fec77fff640 1 -- 172.21.15.116:0/3864942571 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fec7000c100 con 0x7fec70004810 2024-04-25T00:21:58.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.543+0000 7fec77fff640 1 -- 172.21.15.116:0/3864942571 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fec7000c300 con 0x7fec700051d0 2024-04-25T00:21:58.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.543+0000 7fec888c5640 1 --2- 172.21.15.116:0/3864942571 >> v2:172.21.15.174:3300/0 conn(0x7fec840c4c20 0x7fec84154170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:21:58.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.543+0000 7fec890c6640 1 --2- 172.21.15.116:0/3864942571 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec70003b50 0x7fec70003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:58.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.547+0000 7fec898c7640 1 --2- 172.21.15.116:0/3864942571 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fec700051d0 0x7fec7000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:58.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.547+0000 7fec888c5640 1 --2- 172.21.15.116:0/3864942571 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fec70004810 0x7fec70004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:58.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.547+0000 7fec890c6640 1 -- 172.21.15.116:0/3864942571 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fec700051d0 msgr2=0x7fec7000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:21:58.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.547+0000 7fec890c6640 1 --2- 172.21.15.116:0/3864942571 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fec700051d0 0x7fec7000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:58.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.547+0000 7fec890c6640 1 -- 172.21.15.116:0/3864942571 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fec70004810 msgr2=0x7fec70004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:21:58.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.547+0000 7fec890c6640 1 --2- 172.21.15.116:0/3864942571 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fec70004810 0x7fec70004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:58.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.547+0000 7fec890c6640 1 -- 172.21.15.116:0/3864942571 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec70003530 con 0x7fec70003b50 2024-04-25T00:21:58.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.547+0000 7fec898c7640 1 -- 172.21.15.116:0/3864942571 reap_dead start 2024-04-25T00:21:58.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.547+0000 7fec890c6640 1 --2- 172.21.15.116:0/3864942571 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec70003b50 0x7fec70003f50 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7fec8000c1c0 tx=0x7fec8001cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf9c094abbdee26a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:58.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.547+0000 7fec77fff640 1 -- 172.21.15.116:0/3864942571 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec80007cf0 con 0x7fec70003b50 2024-04-25T00:21:58.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.547+0000 7fec77fff640 1 -- 172.21.15.116:0/3864942571 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec800a9020 con 0x7fec70003b50 2024-04-25T00:21:58.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.547+0000 7fec77fff640 1 -- 172.21.15.116:0/3864942571 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec80005ae0 con 0x7fec70003b50 2024-04-25T00:21:58.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.547+0000 7fec8b32a640 1 -- 172.21.15.116:0/3864942571 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec70003b50 msgr2=0x7fec70003f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:58.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.547+0000 7fec8b32a640 1 --2- 172.21.15.116:0/3864942571 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec70003b50 0x7fec70003f50 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7fec8000c1c0 tx=0x7fec8001cc20 comp rx=0 tx=0).stop 2024-04-25T00:21:58.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.547+0000 7fec8b32a640 1 -- 172.21.15.116:0/3864942571 shutdown_connections 2024-04-25T00:21:58.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.547+0000 7fec8b32a640 1 --2- 172.21.15.116:0/3864942571 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec70003b50 0x7fec70003f50 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:58.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.547+0000 7fec8b32a640 1 -- 172.21.15.116:0/3864942571 >> 172.21.15.116:0/3864942571 conn(0x7fec8406f0c0 msgr2=0x7fec8406f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:21:58.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.547+0000 7fec8b32a640 1 -- 172.21.15.116:0/3864942571 shutdown_connections 2024-04-25T00:21:58.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.547+0000 7fec8b32a640 1 -- 172.21.15.116:0/3864942571 wait complete. 2024-04-25T00:21:58.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.551+0000 7fec8b32a640 1 Processor -- start 2024-04-25T00:21:58.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.551+0000 7fec8b32a640 1 -- start start 2024-04-25T00:21:58.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.551+0000 7fec8b32a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fec70003b50 0x7fec841e06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:58.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.551+0000 7fec8b32a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fec841577a0 0x7fec841ef810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:58.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.551+0000 7fec8b32a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec841efd50 0x7fec841e3210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:58.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.551+0000 7fec8b32a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fec840aba10 con 0x7fec841efd50 2024-04-25T00:21:58.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.551+0000 7fec8b32a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fec8406ff30 con 0x7fec841577a0 2024-04-25T00:21:58.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.551+0000 7fec8b32a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fec840c7230 con 0x7fec70003b50 2024-04-25T00:21:58.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.551+0000 7fec898c7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec841efd50 0x7fec841e3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:58.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.551+0000 7fec898c7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec841efd50 0x7fec841e3210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46726/0 (socket says 172.21.15.116:46726) 2024-04-25T00:21:58.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.551+0000 7fec898c7640 1 -- 172.21.15.116:0/2773851494 learned_addr learned my addr 172.21.15.116:0/2773851494 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:21:58.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.551+0000 7fec888c5640 1 --2- 172.21.15.116:0/2773851494 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fec841577a0 0x7fec841ef810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:58.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.551+0000 7fec890c6640 1 --2- 172.21.15.116:0/2773851494 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fec70003b50 0x7fec841e06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:58.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.551+0000 7fec898c7640 1 -- 172.21.15.116:0/2773851494 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fec70003b50 msgr2=0x7fec841e06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:58.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.551+0000 7fec898c7640 1 --2- 172.21.15.116:0/2773851494 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fec70003b50 0x7fec841e06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:58.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.551+0000 7fec898c7640 1 -- 172.21.15.116:0/2773851494 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fec841577a0 msgr2=0x7fec841ef810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:58.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.551+0000 7fec898c7640 1 --2- 172.21.15.116:0/2773851494 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fec841577a0 0x7fec841ef810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:58.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.551+0000 7fec898c7640 1 -- 172.21.15.116:0/2773851494 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec841564a0 con 0x7fec841efd50 2024-04-25T00:21:58.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.551+0000 7fec898c7640 1 --2- 172.21.15.116:0/2773851494 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec841efd50 0x7fec841e3210 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7fec7c00e700 tx=0x7fec7c00eb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:58.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.551+0000 7fec75ffb640 1 -- 172.21.15.116:0/2773851494 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec7c003b90 con 0x7fec841efd50 2024-04-25T00:21:58.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.551+0000 7fec75ffb640 1 -- 172.21.15.116:0/2773851494 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec7c018070 con 0x7fec841efd50 2024-04-25T00:21:58.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.551+0000 7fec8b32a640 1 -- 172.21.15.116:0/2773851494 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec84157240 con 0x7fec841efd50 2024-04-25T00:21:58.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.551+0000 7fec75ffb640 1 -- 172.21.15.116:0/2773851494 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec7c0050d0 con 0x7fec841efd50 2024-04-25T00:21:58.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.551+0000 7fec8b32a640 1 -- 172.21.15.116:0/2773851494 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec84156760 con 0x7fec841efd50 2024-04-25T00:21:58.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.551+0000 7fec8b32a640 1 -- 172.21.15.116:0/2773851494 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec84068a60 con 0x7fec841efd50 2024-04-25T00:21:58.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.551+0000 7fec75ffb640 1 -- 172.21.15.116:0/2773851494 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fec7c005270 con 0x7fec841efd50 2024-04-25T00:21:58.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.555+0000 7fec75ffb640 1 --2- 172.21.15.116:0/2773851494 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fec680678f0 0x7fec68069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:21:58.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.555+0000 7fec75ffb640 1 -- 172.21.15.116:0/2773851494 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 9418+0+0 (secure 0 0 0) 0x7fec7c086620 con 0x7fec841efd50 2024-04-25T00:21:58.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.555+0000 7fec890c6640 1 --2- 172.21.15.116:0/2773851494 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fec680678f0 0x7fec68069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:21:58.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.555+0000 7fec890c6640 1 --2- 172.21.15.116:0/2773851494 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fec680678f0 0x7fec68069db0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fec800aa000 tx=0x7fec800b0040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:21:58.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.559+0000 7fec75ffb640 1 -- 172.21.15.116:0/2773851494 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fec7c056290 con 0x7fec841efd50 2024-04-25T00:21:58.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.755+0000 7fec8b32a640 1 -- 172.21.15.116:0/2773851494 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fec840ae4b0 con 0x7fec841efd50 2024-04-25T00:21:58.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.755+0000 7fec75ffb640 1 -- 172.21.15.116:0/2773851494 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) ==== 95+0+6501 (secure 0 0 0) 0x7fec7c055b90 con 0x7fec841efd50 2024-04-25T00:21:58.756 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:21:58.756 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":172,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7989,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/221971311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":221971311},{"type":"v1","addr":"172.21.15.116:6835","nonce":221971311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":172}],"filesystems":[{"mdsmap":{"epoch":171,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:21:43.889982+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6125},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6125":{"gid":6125,"name":"d","rank":0,"incarnation":168,"state":"up:active","state_seq":14,"addr":"172.21.15.174:6837/321466844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":321466844},{"type":"v1","addr":"172.21.15.174:6837","nonce":321466844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6125,"qdb_cluster":[6125]},"id":1},{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:20:13.935378+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7722":{"gid":7722,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7722,"qdb_cluster":[7722]},"id":2}]} 2024-04-25T00:21:58.756 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 172 2024-04-25T00:21:58.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.759+0000 7fec8b32a640 1 -- 172.21.15.116:0/2773851494 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fec680678f0 msgr2=0x7fec68069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:58.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.759+0000 7fec8b32a640 1 --2- 172.21.15.116:0/2773851494 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fec680678f0 0x7fec68069db0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fec800aa000 tx=0x7fec800b0040 comp rx=0 tx=0).stop 2024-04-25T00:21:58.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.759+0000 7fec8b32a640 1 -- 172.21.15.116:0/2773851494 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec841efd50 msgr2=0x7fec841e3210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:21:58.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.759+0000 7fec8b32a640 1 --2- 172.21.15.116:0/2773851494 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec841efd50 0x7fec841e3210 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7fec7c00e700 tx=0x7fec7c00eb00 comp rx=0 tx=0).stop 2024-04-25T00:21:58.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.759+0000 7fec8b32a640 1 -- 172.21.15.116:0/2773851494 shutdown_connections 2024-04-25T00:21:58.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.759+0000 7fec8b32a640 1 --2- 172.21.15.116:0/2773851494 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fec680678f0 0x7fec68069db0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:58.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.759+0000 7fec8b32a640 1 --2- 172.21.15.116:0/2773851494 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fec841efd50 0x7fec841e3210 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:58.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.759+0000 7fec8b32a640 1 --2- 172.21.15.116:0/2773851494 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fec841577a0 0x7fec841ef810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:58.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.759+0000 7fec8b32a640 1 --2- 172.21.15.116:0/2773851494 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fec70003b50 0x7fec841e06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:21:58.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.759+0000 7fec8b32a640 1 -- 172.21.15.116:0/2773851494 >> 172.21.15.116:0/2773851494 conn(0x7fec8406f0c0 msgr2=0x7fec84070b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:21:58.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.759+0000 7fec8b32a640 1 -- 172.21.15.116:0/2773851494 shutdown_connections 2024-04-25T00:21:58.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:21:58.759+0000 7fec8b32a640 1 -- 172.21.15.116:0/2773851494 wait complete. 2024-04-25T00:21:58.774 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:21:59.415 INFO:tasks.workunit.client.0.smithi116.stdout:Syncing()...0 sec 2024-04-25T00:21:59.416 INFO:tasks.workunit.client.0.smithi116.stdout:Starting Actual Benchmark At: Thu Apr 25 00:21:59 2024 2024-04-25T00:21:59.416 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-04-25T00:22:00.775 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:22:00.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:00.907+0000 7f509d544640 1 Processor -- start 2024-04-25T00:22:00.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:00.907+0000 7f509d544640 1 -- start start 2024-04-25T00:22:00.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:00.907+0000 7f509d544640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5098150d20 0x7f50980c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:00.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:00.907+0000 7f509d544640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50980c3910 con 0x7f50980a94a0 2024-04-25T00:22:00.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:00.907+0000 7f509d544640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50980c16e0 con 0x7f50980c2f50 2024-04-25T00:22:00.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:00.907+0000 7f509d544640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f50980c1400 con 0x7f5098150d20 2024-04-25T00:22:00.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:00.907+0000 7f5096ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5098150d20 0x7f50980c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:00.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:00.907+0000 7f50977fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f50980a94a0 0x7f5098153ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:44546/0 (socket says 172.21.15.116:44546) 2024-04-25T00:22:00.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:00.907+0000 7f50977fe640 1 -- 172.21.15.116:0/1436974814 learned_addr learned my addr 172.21.15.116:0/1436974814 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:22:00.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:00.907+0000 7f5096ffd640 1 -- 172.21.15.116:0/1436974814 >> v1:172.21.15.174:6789/0 conn(0x7f50980c2f50 legacy=0x7f50981538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:00.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:00.907+0000 7f5096ffd640 1 -- 172.21.15.116:0/1436974814 >> v1:172.21.15.116:6789/0 conn(0x7f50980a94a0 legacy=0x7f5098153ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:00.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:00.907+0000 7f5096ffd640 1 -- 172.21.15.116:0/1436974814 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f509806e660 con 0x7f5098150d20 2024-04-25T00:22:00.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:00.907+0000 7f5095ffb640 1 -- 172.21.15.116:0/1436974814 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2216171322 0 0) 0x7f50980c3910 con 0x7f50980a94a0 2024-04-25T00:22:00.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:00.911+0000 7f5096ffd640 1 --2- 172.21.15.116:0/1436974814 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5098150d20 0x7f50980c2a10 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f50880097c0 tx=0x7f508801c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b48d78ca95de096 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:00.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:00.911+0000 7f5095ffb640 1 -- 172.21.15.116:0/1436974814 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50880b1070 con 0x7f5098150d20 2024-04-25T00:22:00.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:00.911+0000 7f5095ffb640 1 -- 172.21.15.116:0/1436974814 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50880ac410 con 0x7f5098150d20 2024-04-25T00:22:00.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:00.911+0000 7f5095ffb640 1 -- 172.21.15.116:0/1436974814 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50880ac7e0 con 0x7f5098150d20 2024-04-25T00:22:00.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:00.911+0000 7f509d544640 1 -- 172.21.15.116:0/1436974814 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5098150d20 msgr2=0x7f50980c2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:00.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:00.911+0000 7f509d544640 1 --2- 172.21.15.116:0/1436974814 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5098150d20 0x7f50980c2a10 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f50880097c0 tx=0x7f508801c4a0 comp rx=0 tx=0).stop 2024-04-25T00:22:00.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:00.911+0000 7f509d544640 1 -- 172.21.15.116:0/1436974814 shutdown_connections 2024-04-25T00:22:00.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:00.911+0000 7f509d544640 1 --2- 172.21.15.116:0/1436974814 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5098150d20 0x7f50980c2a10 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:00.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:00.911+0000 7f509d544640 1 -- 172.21.15.116:0/1436974814 >> 172.21.15.116:0/1436974814 conn(0x7f50980c3b00 msgr2=0x7f50980c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:00.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:00.911+0000 7f509d544640 1 -- 172.21.15.116:0/1436974814 shutdown_connections 2024-04-25T00:22:00.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:00.911+0000 7f509d544640 1 -- 172.21.15.116:0/1436974814 wait complete. 2024-04-25T00:22:00.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:00.911+0000 7f509d544640 1 Processor -- start 2024-04-25T00:22:00.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:00.911+0000 7f509d544640 1 -- start start 2024-04-25T00:22:00.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:00.915+0000 7f509d544640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50980a94a0 0x7f50981efe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:00.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:00.915+0000 7f509d544640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50980c2f50 0x7f5098141fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:00.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:00.915+0000 7f509d544640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5098150d20 0x7f5098142520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:00.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:00.915+0000 7f509d544640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5098161ad0 con 0x7f50980a94a0 2024-04-25T00:22:00.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:00.915+0000 7f509d544640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5098161950 con 0x7f50980c2f50 2024-04-25T00:22:00.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:00.915+0000 7f509d544640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f5098161c50 con 0x7f5098150d20 2024-04-25T00:22:00.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:00.915+0000 7f5096ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50980a94a0 0x7f50981efe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:00.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:00.915+0000 7f5096ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50980a94a0 0x7f50981efe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46744/0 (socket says 172.21.15.116:46744) 2024-04-25T00:22:00.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:00.915+0000 7f5096ffd640 1 -- 172.21.15.116:0/1637595279 learned_addr learned my addr 172.21.15.116:0/1637595279 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:22:00.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:00.915+0000 7f50967fc640 1 --2- 172.21.15.116:0/1637595279 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50980c2f50 0x7f5098141fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:00.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:00.915+0000 7f50977fe640 1 --2- 172.21.15.116:0/1637595279 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5098150d20 0x7f5098142520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:00.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:00.915+0000 7f5096ffd640 1 -- 172.21.15.116:0/1637595279 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5098150d20 msgr2=0x7f5098142520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:00.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:00.915+0000 7f5096ffd640 1 --2- 172.21.15.116:0/1637595279 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5098150d20 0x7f5098142520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:00.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:00.915+0000 7f5096ffd640 1 -- 172.21.15.116:0/1637595279 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50980c2f50 msgr2=0x7f5098141fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:00.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:00.915+0000 7f5096ffd640 1 --2- 172.21.15.116:0/1637595279 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50980c2f50 0x7f5098141fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:00.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:00.915+0000 7f5096ffd640 1 -- 172.21.15.116:0/1637595279 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50980aad10 con 0x7f50980a94a0 2024-04-25T00:22:00.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:00.915+0000 7f5096ffd640 1 --2- 172.21.15.116:0/1637595279 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50980a94a0 0x7f50981efe40 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f50880af040 tx=0x7f5088019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:00.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:00.915+0000 7f507f7fe640 1 -- 172.21.15.116:0/1637595279 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50880b1040 con 0x7f50980a94a0 2024-04-25T00:22:00.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:00.915+0000 7f507f7fe640 1 -- 172.21.15.116:0/1637595279 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50880bc070 con 0x7f50980a94a0 2024-04-25T00:22:00.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:00.915+0000 7f509d544640 1 -- 172.21.15.116:0/1637595279 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f509814e800 con 0x7f50980a94a0 2024-04-25T00:22:00.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:00.915+0000 7f507f7fe640 1 -- 172.21.15.116:0/1637595279 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50880b5630 con 0x7f50980a94a0 2024-04-25T00:22:00.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:00.915+0000 7f509d544640 1 -- 172.21.15.116:0/1637595279 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50981f0790 con 0x7f50980a94a0 2024-04-25T00:22:00.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:00.915+0000 7f509d544640 1 -- 172.21.15.116:0/1637595279 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5098068350 con 0x7f50980a94a0 2024-04-25T00:22:00.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:00.915+0000 7f507f7fe640 1 -- 172.21.15.116:0/1637595279 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f50880b57d0 con 0x7f50980a94a0 2024-04-25T00:22:00.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:00.915+0000 7f507f7fe640 1 --2- 172.21.15.116:0/1637595279 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f50840678c0 0x7f5084069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:00.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:00.915+0000 7f507f7fe640 1 -- 172.21.15.116:0/1637595279 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 9418+0+0 (secure 0 0 0) 0x7f5088120570 con 0x7f50980a94a0 2024-04-25T00:22:00.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:00.915+0000 7f50967fc640 1 --2- 172.21.15.116:0/1637595279 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f50840678c0 0x7f5084069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:00.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:00.919+0000 7f50967fc640 1 --2- 172.21.15.116:0/1637595279 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f50840678c0 0x7f5084069d80 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f509814f270 tx=0x7f508c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:00.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:00.919+0000 7f507f7fe640 1 -- 172.21.15.116:0/1637595279 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f5088016090 con 0x7f50980a94a0 2024-04-25T00:22:01.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:01.119+0000 7f509d544640 1 -- 172.21.15.116:0/1637595279 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f509814e100 con 0x7f50980a94a0 2024-04-25T00:22:01.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:01.119+0000 7f507f7fe640 1 -- 172.21.15.116:0/1637595279 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) ==== 95+0+6501 (secure 0 0 0) 0x7f50880f01e0 con 0x7f50980a94a0 2024-04-25T00:22:01.120 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:22:01.120 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":172,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7989,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/221971311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":221971311},{"type":"v1","addr":"172.21.15.116:6835","nonce":221971311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":172}],"filesystems":[{"mdsmap":{"epoch":171,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:21:43.889982+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6125},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6125":{"gid":6125,"name":"d","rank":0,"incarnation":168,"state":"up:active","state_seq":14,"addr":"172.21.15.174:6837/321466844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":321466844},{"type":"v1","addr":"172.21.15.174:6837","nonce":321466844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6125,"qdb_cluster":[6125]},"id":1},{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:20:13.935378+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7722":{"gid":7722,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7722,"qdb_cluster":[7722]},"id":2}]} 2024-04-25T00:22:01.120 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 172 2024-04-25T00:22:01.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:01.123+0000 7f509d544640 1 -- 172.21.15.116:0/1637595279 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f50840678c0 msgr2=0x7f5084069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:01.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:01.123+0000 7f509d544640 1 --2- 172.21.15.116:0/1637595279 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f50840678c0 0x7f5084069d80 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f509814f270 tx=0x7f508c009040 comp rx=0 tx=0).stop 2024-04-25T00:22:01.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:01.123+0000 7f509d544640 1 -- 172.21.15.116:0/1637595279 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50980a94a0 msgr2=0x7f50981efe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:01.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:01.123+0000 7f509d544640 1 --2- 172.21.15.116:0/1637595279 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50980a94a0 0x7f50981efe40 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f50880af040 tx=0x7f5088019240 comp rx=0 tx=0).stop 2024-04-25T00:22:01.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:01.123+0000 7f509d544640 1 -- 172.21.15.116:0/1637595279 shutdown_connections 2024-04-25T00:22:01.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:01.123+0000 7f509d544640 1 --2- 172.21.15.116:0/1637595279 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f50840678c0 0x7f5084069d80 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:01.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:01.123+0000 7f509d544640 1 --2- 172.21.15.116:0/1637595279 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5098150d20 0x7f5098142520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:01.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:01.123+0000 7f509d544640 1 --2- 172.21.15.116:0/1637595279 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50980c2f50 0x7f5098141fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:01.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:01.123+0000 7f509d544640 1 --2- 172.21.15.116:0/1637595279 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f50980a94a0 0x7f50981efe40 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:01.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:01.123+0000 7f509d544640 1 -- 172.21.15.116:0/1637595279 >> 172.21.15.116:0/1637595279 conn(0x7f50980c3b00 msgr2=0x7f50980ac330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:01.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:01.123+0000 7f509d544640 1 -- 172.21.15.116:0/1637595279 shutdown_connections 2024-04-25T00:22:01.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:01.123+0000 7f509d544640 1 -- 172.21.15.116:0/1637595279 wait complete. 2024-04-25T00:22:01.138 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:22:03.139 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:22:03.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.271+0000 7fc185048640 1 Processor -- start 2024-04-25T00:22:03.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.271+0000 7fc185048640 1 -- start start 2024-04-25T00:22:03.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.275+0000 7fc185048640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1800a94a0 0x7fc1801538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:03.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.275+0000 7fc185048640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc1800a99f0 0x7fc180153e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:03.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.275+0000 7fc185048640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1800c3910 con 0x7fc180150d20 2024-04-25T00:22:03.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.275+0000 7fc185048640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fc1800c16e0 con 0x7fc1800a99f0 2024-04-25T00:22:03.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.275+0000 7fc185048640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fc1800c1400 con 0x7fc1800a94a0 2024-04-25T00:22:03.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.275+0000 7fc17f59e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc1800a99f0 0x7fc180153e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:03.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.275+0000 7fc17ed9d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fc180150d20 0x7fc1800c2a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:44556/0 (socket says 172.21.15.116:44556) 2024-04-25T00:22:03.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.275+0000 7fc17ed9d640 1 -- 172.21.15.116:0/3657693017 learned_addr learned my addr 172.21.15.116:0/3657693017 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:22:03.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.275+0000 7fc17e59c640 1 --2- 172.21.15.116:0/3657693017 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1800a94a0 0x7fc1801538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:03.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.275+0000 7fc17f59e640 1 -- 172.21.15.116:0/3657693017 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1800a94a0 msgr2=0x7fc1801538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:03.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.275+0000 7fc17f59e640 1 --2- 172.21.15.116:0/3657693017 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1800a94a0 0x7fc1801538e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:03.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.275+0000 7fc17f59e640 1 -- 172.21.15.116:0/3657693017 >> v1:172.21.15.116:6789/0 conn(0x7fc180150d20 legacy=0x7fc1800c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:03.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.275+0000 7fc17dd9b640 1 -- 172.21.15.116:0/3657693017 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2300956363 0 0) 0x7fc1800c3910 con 0x7fc180150d20 2024-04-25T00:22:03.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.275+0000 7fc17f59e640 1 -- 172.21.15.116:0/3657693017 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc18006e660 con 0x7fc1800a99f0 2024-04-25T00:22:03.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.275+0000 7fc17f59e640 1 --2- 172.21.15.116:0/3657693017 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1800a99f0 0x7fc180153e20 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7fc178009d90 tx=0x7fc17801c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ab67fd79c2de016 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:03.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.275+0000 7fc17dd9b640 1 -- 172.21.15.116:0/3657693017 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1780b2070 con 0x7fc1800a99f0 2024-04-25T00:22:03.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.275+0000 7fc17dd9b640 1 -- 172.21.15.116:0/3657693017 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1780a5830 con 0x7fc1800a99f0 2024-04-25T00:22:03.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.275+0000 7fc17dd9b640 1 -- 172.21.15.116:0/3657693017 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1780a5c00 con 0x7fc1800a99f0 2024-04-25T00:22:03.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.275+0000 7fc185048640 1 -- 172.21.15.116:0/3657693017 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1800a99f0 msgr2=0x7fc180153e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:03.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.275+0000 7fc185048640 1 --2- 172.21.15.116:0/3657693017 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1800a99f0 0x7fc180153e20 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7fc178009d90 tx=0x7fc17801c970 comp rx=0 tx=0).stop 2024-04-25T00:22:03.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.275+0000 7fc185048640 1 -- 172.21.15.116:0/3657693017 shutdown_connections 2024-04-25T00:22:03.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.275+0000 7fc185048640 1 --2- 172.21.15.116:0/3657693017 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1800a99f0 0x7fc180153e20 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:03.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.275+0000 7fc185048640 1 --2- 172.21.15.116:0/3657693017 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1800a94a0 0x7fc1801538e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:03.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.275+0000 7fc185048640 1 -- 172.21.15.116:0/3657693017 >> 172.21.15.116:0/3657693017 conn(0x7fc1800c3b00 msgr2=0x7fc1800c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:03.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.279+0000 7fc185048640 1 -- 172.21.15.116:0/3657693017 shutdown_connections 2024-04-25T00:22:03.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.279+0000 7fc185048640 1 -- 172.21.15.116:0/3657693017 wait complete. 2024-04-25T00:22:03.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.279+0000 7fc185048640 1 Processor -- start 2024-04-25T00:22:03.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.279+0000 7fc185048640 1 -- start start 2024-04-25T00:22:03.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.279+0000 7fc185048640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1800a94a0 0x7fc1800c6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:03.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.279+0000 7fc185048640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1800a99f0 0x7fc1800c4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:03.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.279+0000 7fc185048640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc180150d20 0x7fc1800c0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:03.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.279+0000 7fc185048640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc1801544e0 con 0x7fc1800a99f0 2024-04-25T00:22:03.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.279+0000 7fc185048640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc180154360 con 0x7fc180150d20 2024-04-25T00:22:03.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.279+0000 7fc185048640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fc180160730 con 0x7fc1800a94a0 2024-04-25T00:22:03.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.279+0000 7fc17e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1800a99f0 0x7fc1800c4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:03.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.279+0000 7fc17e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1800a99f0 0x7fc1800c4f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46764/0 (socket says 172.21.15.116:46764) 2024-04-25T00:22:03.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.279+0000 7fc17e59c640 1 -- 172.21.15.116:0/3665184667 learned_addr learned my addr 172.21.15.116:0/3665184667 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:22:03.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.279+0000 7fc17f59e640 1 --2- 172.21.15.116:0/3665184667 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc180150d20 0x7fc1800c0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:03.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.279+0000 7fc17ed9d640 1 --2- 172.21.15.116:0/3665184667 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1800a94a0 0x7fc1800c6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:03.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.279+0000 7fc17e59c640 1 -- 172.21.15.116:0/3665184667 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1800a94a0 msgr2=0x7fc1800c6100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:03.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.279+0000 7fc17e59c640 1 --2- 172.21.15.116:0/3665184667 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1800a94a0 0x7fc1800c6100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:03.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.279+0000 7fc17e59c640 1 -- 172.21.15.116:0/3665184667 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc180150d20 msgr2=0x7fc1800c0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:03.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.279+0000 7fc17e59c640 1 --2- 172.21.15.116:0/3665184667 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc180150d20 0x7fc1800c0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:03.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.279+0000 7fc17e59c640 1 -- 172.21.15.116:0/3665184667 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc180152c00 con 0x7fc1800a99f0 2024-04-25T00:22:03.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.279+0000 7fc17e59c640 1 --2- 172.21.15.116:0/3665184667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1800a99f0 0x7fc1800c4f60 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7fc17400d5f0 tx=0x7fc17400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:03.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.279+0000 7fc1677fe640 1 -- 172.21.15.116:0/3665184667 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1740125f0 con 0x7fc1800a99f0 2024-04-25T00:22:03.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.279+0000 7fc1677fe640 1 -- 172.21.15.116:0/3665184667 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc174019070 con 0x7fc1800a99f0 2024-04-25T00:22:03.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.279+0000 7fc185048640 1 -- 172.21.15.116:0/3665184667 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc180152ee0 con 0x7fc1800a99f0 2024-04-25T00:22:03.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.279+0000 7fc1677fe640 1 -- 172.21.15.116:0/3665184667 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1740129b0 con 0x7fc1800a99f0 2024-04-25T00:22:03.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.279+0000 7fc185048640 1 -- 172.21.15.116:0/3665184667 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1800c0bd0 con 0x7fc1800a99f0 2024-04-25T00:22:03.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.279+0000 7fc185048640 1 -- 172.21.15.116:0/3665184667 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc180068350 con 0x7fc1800a99f0 2024-04-25T00:22:03.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.283+0000 7fc1677fe640 1 -- 172.21.15.116:0/3665184667 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc174012b50 con 0x7fc1800a99f0 2024-04-25T00:22:03.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.283+0000 7fc1677fe640 1 --2- 172.21.15.116:0/3665184667 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc16c0678f0 0x7fc16c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:03.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.283+0000 7fc1677fe640 1 -- 172.21.15.116:0/3665184667 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 9418+0+0 (secure 0 0 0) 0x7fc174086f20 con 0x7fc1800a99f0 2024-04-25T00:22:03.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.283+0000 7fc17ed9d640 1 --2- 172.21.15.116:0/3665184667 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc16c0678f0 0x7fc16c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:03.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.283+0000 7fc17ed9d640 1 --2- 172.21.15.116:0/3665184667 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc16c0678f0 0x7fc16c069db0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fc170000f70 tx=0x7fc170007b60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:03.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.287+0000 7fc1677fe640 1 -- 172.21.15.116:0/3665184667 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fc174056bc0 con 0x7fc1800a99f0 2024-04-25T00:22:03.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.483+0000 7fc185048640 1 -- 172.21.15.116:0/3665184667 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc1801530b0 con 0x7fc1800a99f0 2024-04-25T00:22:03.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.483+0000 7fc1677fe640 1 -- 172.21.15.116:0/3665184667 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) ==== 95+0+6501 (secure 0 0 0) 0x7fc1740564c0 con 0x7fc1800a99f0 2024-04-25T00:22:03.483 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:22:03.483 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":172,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7989,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/221971311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":221971311},{"type":"v1","addr":"172.21.15.116:6835","nonce":221971311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":172}],"filesystems":[{"mdsmap":{"epoch":171,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:21:43.889982+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6125},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6125":{"gid":6125,"name":"d","rank":0,"incarnation":168,"state":"up:active","state_seq":14,"addr":"172.21.15.174:6837/321466844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":321466844},{"type":"v1","addr":"172.21.15.174:6837","nonce":321466844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6125,"qdb_cluster":[6125]},"id":1},{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:20:13.935378+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7722":{"gid":7722,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7722,"qdb_cluster":[7722]},"id":2}]} 2024-04-25T00:22:03.484 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 172 2024-04-25T00:22:03.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.487+0000 7fc185048640 1 -- 172.21.15.116:0/3665184667 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc16c0678f0 msgr2=0x7fc16c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:03.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.487+0000 7fc185048640 1 --2- 172.21.15.116:0/3665184667 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc16c0678f0 0x7fc16c069db0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fc170000f70 tx=0x7fc170007b60 comp rx=0 tx=0).stop 2024-04-25T00:22:03.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.487+0000 7fc185048640 1 -- 172.21.15.116:0/3665184667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1800a99f0 msgr2=0x7fc1800c4f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:03.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.487+0000 7fc185048640 1 --2- 172.21.15.116:0/3665184667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1800a99f0 0x7fc1800c4f60 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7fc17400d5f0 tx=0x7fc17400dab0 comp rx=0 tx=0).stop 2024-04-25T00:22:03.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.487+0000 7fc185048640 1 -- 172.21.15.116:0/3665184667 shutdown_connections 2024-04-25T00:22:03.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.487+0000 7fc185048640 1 --2- 172.21.15.116:0/3665184667 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc16c0678f0 0x7fc16c069db0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:03.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.487+0000 7fc185048640 1 --2- 172.21.15.116:0/3665184667 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc180150d20 0x7fc1800c0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:03.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.487+0000 7fc185048640 1 --2- 172.21.15.116:0/3665184667 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc1800a99f0 0x7fc1800c4f60 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:03.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.487+0000 7fc185048640 1 --2- 172.21.15.116:0/3665184667 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc1800a94a0 0x7fc1800c6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:03.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.487+0000 7fc185048640 1 -- 172.21.15.116:0/3665184667 >> 172.21.15.116:0/3665184667 conn(0x7fc1800c3b00 msgr2=0x7fc1800c2a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:03.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.487+0000 7fc185048640 1 -- 172.21.15.116:0/3665184667 shutdown_connections 2024-04-25T00:22:03.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:03.487+0000 7fc185048640 1 -- 172.21.15.116:0/3665184667 wait complete. 2024-04-25T00:22:03.502 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:22:05.502 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:22:05.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.635+0000 7f02973b1640 1 Processor -- start 2024-04-25T00:22:05.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.635+0000 7f02973b1640 1 -- start start 2024-04-25T00:22:05.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.635+0000 7f02973b1640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f029006d5f0 0x7f0290156c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:05.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.635+0000 7f02973b1640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f0290155810 0x7f02901571c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:05.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.635+0000 7f02973b1640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02900c9150 con 0x7f0290152d80 2024-04-25T00:22:05.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.635+0000 7f02973b1640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f02900c9d60 con 0x7f029006d5f0 2024-04-25T00:22:05.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.635+0000 7f02973b1640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f02900c9430 con 0x7f0290155810 2024-04-25T00:22:05.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.635+0000 7f029494c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f029006d5f0 0x7f0290156c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:05.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.635+0000 7f029494c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f029006d5f0 0x7f0290156c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56930/0 (socket says 172.21.15.116:56930) 2024-04-25T00:22:05.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.635+0000 7f029514d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f0290152d80 0x7f029006bd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:49360/0 (socket says 172.21.15.116:49360) 2024-04-25T00:22:05.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.635+0000 7f029494c640 1 -- 172.21.15.116:0/427213705 learned_addr learned my addr 172.21.15.116:0/427213705 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:22:05.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.635+0000 7f029594e640 1 --2- 172.21.15.116:0/427213705 >> v2:172.21.15.174:3300/0 conn(0x7f0290155810 0x7f02901571c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:05.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.635+0000 7f029494c640 1 -- 172.21.15.116:0/427213705 >> v2:172.21.15.174:3300/0 conn(0x7f0290155810 msgr2=0x7f02901571c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:05.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.635+0000 7f029494c640 1 --2- 172.21.15.116:0/427213705 >> v2:172.21.15.174:3300/0 conn(0x7f0290155810 0x7f02901571c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:05.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.635+0000 7f029494c640 1 -- 172.21.15.116:0/427213705 >> v1:172.21.15.116:6789/0 conn(0x7f0290152d80 legacy=0x7f029006bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:05.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.635+0000 7f029494c640 1 -- 172.21.15.116:0/427213705 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02900c9710 con 0x7f029006d5f0 2024-04-25T00:22:05.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.635+0000 7f0287fff640 1 -- 172.21.15.116:0/427213705 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1602816278 0 0) 0x7f02900c9150 con 0x7f0290152d80 2024-04-25T00:22:05.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.639+0000 7f029494c640 1 --2- 172.21.15.116:0/427213705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f029006d5f0 0x7f0290156c80 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f0290155ca0 tx=0x7f028001c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3793917d941c0315 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:05.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.639+0000 7f0287fff640 1 -- 172.21.15.116:0/427213705 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02800b1070 con 0x7f029006d5f0 2024-04-25T00:22:05.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.639+0000 7f0287fff640 1 -- 172.21.15.116:0/427213705 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02800ac5f0 con 0x7f029006d5f0 2024-04-25T00:22:05.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.639+0000 7f0287fff640 1 -- 172.21.15.116:0/427213705 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02800ac9c0 con 0x7f029006d5f0 2024-04-25T00:22:05.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.639+0000 7f02973b1640 1 -- 172.21.15.116:0/427213705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f029006d5f0 msgr2=0x7f0290156c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:05.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.639+0000 7f02973b1640 1 --2- 172.21.15.116:0/427213705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f029006d5f0 0x7f0290156c80 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f0290155ca0 tx=0x7f028001c460 comp rx=0 tx=0).stop 2024-04-25T00:22:05.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.639+0000 7f02973b1640 1 -- 172.21.15.116:0/427213705 shutdown_connections 2024-04-25T00:22:05.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.639+0000 7f02973b1640 1 --2- 172.21.15.116:0/427213705 >> v2:172.21.15.174:3300/0 conn(0x7f0290155810 0x7f02901571c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:05.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.639+0000 7f02973b1640 1 --2- 172.21.15.116:0/427213705 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f029006d5f0 0x7f0290156c80 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:05.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.639+0000 7f02973b1640 1 -- 172.21.15.116:0/427213705 >> 172.21.15.116:0/427213705 conn(0x7f02900acdd0 msgr2=0x7f02900ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:05.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.639+0000 7f02973b1640 1 -- 172.21.15.116:0/427213705 shutdown_connections 2024-04-25T00:22:05.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.639+0000 7f02973b1640 1 -- 172.21.15.116:0/427213705 wait complete. 2024-04-25T00:22:05.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.639+0000 7f02973b1640 1 Processor -- start 2024-04-25T00:22:05.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.639+0000 7f02973b1640 1 -- start start 2024-04-25T00:22:05.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.639+0000 7f02973b1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f029006d5f0 0x7f02901f1d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:05.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.639+0000 7f02973b1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0290152d80 0x7f02901496b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:05.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.639+0000 7f02973b1640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0290155810 0x7f02900ab130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:05.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.639+0000 7f02973b1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f02900caf00 con 0x7f0290152d80 2024-04-25T00:22:05.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.639+0000 7f02973b1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f02900cad80 con 0x7f029006d5f0 2024-04-25T00:22:05.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.639+0000 7f02973b1640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f02901628d0 con 0x7f0290155810 2024-04-25T00:22:05.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.639+0000 7f029494c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0290152d80 0x7f02901496b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:05.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.639+0000 7f029494c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0290152d80 0x7f02901496b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56940/0 (socket says 172.21.15.116:56940) 2024-04-25T00:22:05.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.639+0000 7f029494c640 1 -- 172.21.15.116:0/1012524243 learned_addr learned my addr 172.21.15.116:0/1012524243 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:22:05.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.639+0000 7f029514d640 1 --2- 172.21.15.116:0/1012524243 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f029006d5f0 0x7f02901f1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:05.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.639+0000 7f029594e640 1 --2- 172.21.15.116:0/1012524243 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0290155810 0x7f02900ab130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:05.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.639+0000 7f029494c640 1 -- 172.21.15.116:0/1012524243 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0290155810 msgr2=0x7f02900ab130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:05.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.639+0000 7f029494c640 1 --2- 172.21.15.116:0/1012524243 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0290155810 0x7f02900ab130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:05.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.639+0000 7f029494c640 1 -- 172.21.15.116:0/1012524243 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f029006d5f0 msgr2=0x7f02901f1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:05.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.639+0000 7f029494c640 1 --2- 172.21.15.116:0/1012524243 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f029006d5f0 0x7f02901f1d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:05.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.639+0000 7f029494c640 1 -- 172.21.15.116:0/1012524243 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f029014a170 con 0x7f0290152d80 2024-04-25T00:22:05.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.639+0000 7f029594e640 1 --2- 172.21.15.116:0/1012524243 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0290155810 0x7f02900ab130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:22:05.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.639+0000 7f029494c640 1 --2- 172.21.15.116:0/1012524243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0290152d80 0x7f02901496b0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f028001ab10 tx=0x7f0280019200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:05.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.639+0000 7f0285ffb640 1 -- 172.21.15.116:0/1012524243 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02800bc070 con 0x7f0290152d80 2024-04-25T00:22:05.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.639+0000 7f0285ffb640 1 -- 172.21.15.116:0/1012524243 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02800ab5a0 con 0x7f0290152d80 2024-04-25T00:22:05.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.639+0000 7f02973b1640 1 -- 172.21.15.116:0/1012524243 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02901f2270 con 0x7f0290152d80 2024-04-25T00:22:05.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.639+0000 7f0285ffb640 1 -- 172.21.15.116:0/1012524243 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02800b1070 con 0x7f0290152d80 2024-04-25T00:22:05.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.639+0000 7f02973b1640 1 -- 172.21.15.116:0/1012524243 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02900ac540 con 0x7f0290152d80 2024-04-25T00:22:05.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.643+0000 7f02973b1640 1 -- 172.21.15.116:0/1012524243 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0290068350 con 0x7f0290152d80 2024-04-25T00:22:05.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.643+0000 7f0285ffb640 1 -- 172.21.15.116:0/1012524243 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f02800bf070 con 0x7f0290152d80 2024-04-25T00:22:05.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.643+0000 7f0285ffb640 1 --2- 172.21.15.116:0/1012524243 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f02740678c0 0x7f0274069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:05.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.643+0000 7f0285ffb640 1 -- 172.21.15.116:0/1012524243 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 9418+0+0 (secure 0 0 0) 0x7f02801204a0 con 0x7f0290152d80 2024-04-25T00:22:05.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.643+0000 7f029514d640 1 --2- 172.21.15.116:0/1012524243 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f02740678c0 0x7f0274069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:05.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.643+0000 7f029514d640 1 --2- 172.21.15.116:0/1012524243 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f02740678c0 0x7f0274069d80 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f028c000f70 tx=0x7f028c007b80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:05.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.647+0000 7f0285ffb640 1 -- 172.21.15.116:0/1012524243 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f02800a9070 con 0x7f0290152d80 2024-04-25T00:22:05.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.839+0000 7f02973b1640 1 -- 172.21.15.116:0/1012524243 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f02900cab60 con 0x7f0290152d80 2024-04-25T00:22:05.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.839+0000 7f0285ffb640 1 -- 172.21.15.116:0/1012524243 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) ==== 95+0+6501 (secure 0 0 0) 0x7f0280016110 con 0x7f0290152d80 2024-04-25T00:22:05.838 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:22:05.839 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":172,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7989,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/221971311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":221971311},{"type":"v1","addr":"172.21.15.116:6835","nonce":221971311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":172}],"filesystems":[{"mdsmap":{"epoch":171,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:21:43.889982+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6125},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6125":{"gid":6125,"name":"d","rank":0,"incarnation":168,"state":"up:active","state_seq":14,"addr":"172.21.15.174:6837/321466844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":321466844},{"type":"v1","addr":"172.21.15.174:6837","nonce":321466844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6125,"qdb_cluster":[6125]},"id":1},{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:20:13.935378+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7722":{"gid":7722,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7722,"qdb_cluster":[7722]},"id":2}]} 2024-04-25T00:22:05.839 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 172 2024-04-25T00:22:05.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.839+0000 7f02973b1640 1 -- 172.21.15.116:0/1012524243 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f02740678c0 msgr2=0x7f0274069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:05.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.839+0000 7f02973b1640 1 --2- 172.21.15.116:0/1012524243 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f02740678c0 0x7f0274069d80 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f028c000f70 tx=0x7f028c007b80 comp rx=0 tx=0).stop 2024-04-25T00:22:05.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.839+0000 7f02973b1640 1 -- 172.21.15.116:0/1012524243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0290152d80 msgr2=0x7f02901496b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:05.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.839+0000 7f02973b1640 1 --2- 172.21.15.116:0/1012524243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0290152d80 0x7f02901496b0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f028001ab10 tx=0x7f0280019200 comp rx=0 tx=0).stop 2024-04-25T00:22:05.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.839+0000 7f02973b1640 1 -- 172.21.15.116:0/1012524243 shutdown_connections 2024-04-25T00:22:05.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.839+0000 7f02973b1640 1 --2- 172.21.15.116:0/1012524243 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f02740678c0 0x7f0274069d80 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:05.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.839+0000 7f02973b1640 1 --2- 172.21.15.116:0/1012524243 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0290155810 0x7f02900ab130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:05.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.839+0000 7f02973b1640 1 --2- 172.21.15.116:0/1012524243 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0290152d80 0x7f02901496b0 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:05.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.839+0000 7f02973b1640 1 --2- 172.21.15.116:0/1012524243 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f029006d5f0 0x7f02901f1d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:05.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.839+0000 7f02973b1640 1 -- 172.21.15.116:0/1012524243 >> 172.21.15.116:0/1012524243 conn(0x7f02900acdd0 msgr2=0x7f029015f600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:05.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.839+0000 7f02973b1640 1 -- 172.21.15.116:0/1012524243 shutdown_connections 2024-04-25T00:22:05.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:05.839+0000 7f02973b1640 1 -- 172.21.15.116:0/1012524243 wait complete. 2024-04-25T00:22:05.856 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:22:07.857 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:22:07.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.987+0000 7f5037a6e640 1 Processor -- start 2024-04-25T00:22:07.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.987+0000 7f5037a6e640 1 -- start start 2024-04-25T00:22:08.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.991+0000 7f5037a6e640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f50300c7880 0x7f5030155d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:08.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.991+0000 7f5037a6e640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50300c8710 con 0x7f5030153150 2024-04-25T00:22:08.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.991+0000 7f5037a6e640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50300c89f0 con 0x7f5030152d70 2024-04-25T00:22:08.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.991+0000 7f5037a6e640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f5030159c10 con 0x7f50300c7880 2024-04-25T00:22:08.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.991+0000 7f503600b640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f5030153150 0x7f5030160010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:49362/0 (socket says 172.21.15.116:49362) 2024-04-25T00:22:08.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.991+0000 7f503600b640 1 -- 172.21.15.116:0/1041359740 learned_addr learned my addr 172.21.15.116:0/1041359740 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:22:08.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.991+0000 7f5034808640 1 -- 172.21.15.116:0/1041359740 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2186953076 0 0) 0x7f50300c8710 con 0x7f5030153150 2024-04-25T00:22:08.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.991+0000 7f5035009640 1 --2- 172.21.15.116:0/1041359740 >> v2:172.21.15.174:3300/0 conn(0x7f50300c7880 0x7f5030155d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:08.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.991+0000 7f5034808640 1 -- 172.21.15.116:0/1041359740 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f501c003530 con 0x7f5030153150 2024-04-25T00:22:08.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.991+0000 7f5034808640 1 -- 172.21.15.116:0/1041359740 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3922798074 0 0) 0x7f501c003530 con 0x7f5030153150 2024-04-25T00:22:08.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.991+0000 7f5034808640 1 -- 172.21.15.116:0/1041359740 >> v2:172.21.15.174:3300/0 conn(0x7f50300c7880 msgr2=0x7f5030155d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:08.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.991+0000 7f5034808640 1 --2- 172.21.15.116:0/1041359740 >> v2:172.21.15.174:3300/0 conn(0x7f50300c7880 0x7f5030155d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:08.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.991+0000 7f5034808640 1 -- 172.21.15.116:0/1041359740 >> v1:172.21.15.174:6789/0 conn(0x7f5030152d70 legacy=0x7f50300c5970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:08.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.991+0000 7f5034808640 1 -- 172.21.15.116:0/1041359740 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5030155840 con 0x7f5030153150 2024-04-25T00:22:08.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.991+0000 7f5034808640 1 -- 172.21.15.116:0/1041359740 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f5028001a00 con 0x7f5030153150 2024-04-25T00:22:08.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.991+0000 7f5034808640 1 -- 172.21.15.116:0/1041359740 >> v1:172.21.15.116:6789/0 conn(0x7f5030153150 legacy=0x7f5030160010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:08.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.991+0000 7f5034808640 1 --2- 172.21.15.116:0/1041359740 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f501c0039c0 0x7f501c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:08.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.991+0000 7f5034808640 1 --2- 172.21.15.116:0/1041359740 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f501c004810 0x7f501c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:08.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.991+0000 7f5034808640 1 --2- 172.21.15.116:0/1041359740 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f501c0051d0 0x7f501c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:08.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.991+0000 7f5034808640 1 -- 172.21.15.116:0/1041359740 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5030159c10 con 0x7f501c004810 2024-04-25T00:22:08.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.991+0000 7f5034808640 1 -- 172.21.15.116:0/1041359740 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f501c00c170 con 0x7f501c0039c0 2024-04-25T00:22:08.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.991+0000 7f5034808640 1 -- 172.21.15.116:0/1041359740 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f501c00c370 con 0x7f501c0051d0 2024-04-25T00:22:08.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.991+0000 7f5035009640 1 --2- 172.21.15.116:0/1041359740 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f501c004810 0x7f501c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:08.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.991+0000 7f503600b640 1 --2- 172.21.15.116:0/1041359740 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f501c0051d0 0x7f501c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:08.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.991+0000 7f5035009640 1 -- 172.21.15.116:0/1041359740 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f501c0051d0 msgr2=0x7f501c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:08.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.991+0000 7f5035009640 1 --2- 172.21.15.116:0/1041359740 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f501c0051d0 0x7f501c00b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:08.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.991+0000 7f5035009640 1 -- 172.21.15.116:0/1041359740 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f501c0039c0 msgr2=0x7f501c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:08.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.991+0000 7f5035009640 1 --2- 172.21.15.116:0/1041359740 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f501c0039c0 0x7f501c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:08.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.991+0000 7f5035009640 1 -- 172.21.15.116:0/1041359740 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f501c003530 con 0x7f501c004810 2024-04-25T00:22:08.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.991+0000 7f503600b640 1 -- 172.21.15.116:0/1041359740 reap_dead start 2024-04-25T00:22:08.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.991+0000 7f5035009640 1 --2- 172.21.15.116:0/1041359740 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f501c004810 0x7f501c004c90 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f5020009ab0 tx=0x7f502001c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ffe68326edd5f29 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:08.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.995+0000 7f5034808640 1 -- 172.21.15.116:0/1041359740 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50200b1070 con 0x7f501c004810 2024-04-25T00:22:08.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.995+0000 7f5034808640 1 -- 172.21.15.116:0/1041359740 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50200a4a80 con 0x7f501c004810 2024-04-25T00:22:08.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.995+0000 7f5034808640 1 -- 172.21.15.116:0/1041359740 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50200096b0 con 0x7f501c004810 2024-04-25T00:22:08.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.995+0000 7f5037a6e640 1 -- 172.21.15.116:0/1041359740 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f501c004810 msgr2=0x7f501c004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:08.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.995+0000 7f5037a6e640 1 --2- 172.21.15.116:0/1041359740 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f501c004810 0x7f501c004c90 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f5020009ab0 tx=0x7f502001c730 comp rx=0 tx=0).stop 2024-04-25T00:22:08.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.995+0000 7f5037a6e640 1 -- 172.21.15.116:0/1041359740 shutdown_connections 2024-04-25T00:22:08.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.995+0000 7f5037a6e640 1 --2- 172.21.15.116:0/1041359740 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f501c004810 0x7f501c004c90 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:08.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.995+0000 7f5037a6e640 1 -- 172.21.15.116:0/1041359740 >> 172.21.15.116:0/1041359740 conn(0x7f5030074a80 msgr2=0x7f50300c8f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:08.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.995+0000 7f5037a6e640 1 -- 172.21.15.116:0/1041359740 shutdown_connections 2024-04-25T00:22:08.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.995+0000 7f5037a6e640 1 -- 172.21.15.116:0/1041359740 wait complete. 2024-04-25T00:22:08.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.995+0000 7f5037a6e640 1 Processor -- start 2024-04-25T00:22:08.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.995+0000 7f5037a6e640 1 -- start start 2024-04-25T00:22:08.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.995+0000 7f5037a6e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f501c004810 0x7f50300ad040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:08.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.995+0000 7f5037a6e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f503006ff80 0x7f50301e0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:08.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.995+0000 7f5037a6e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50301f1f00 0x7f503014df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:08.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.995+0000 7f5037a6e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5030155d60 con 0x7f501c004810 2024-04-25T00:22:08.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.995+0000 7f5037a6e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f50300c9e30 con 0x7f50301f1f00 2024-04-25T00:22:08.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.995+0000 7f5037a6e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f5030155ee0 con 0x7f503006ff80 2024-04-25T00:22:08.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.995+0000 7f503580a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f501c004810 0x7f50300ad040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:08.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.995+0000 7f503580a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f501c004810 0x7f50300ad040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56956/0 (socket says 172.21.15.116:56956) 2024-04-25T00:22:08.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.995+0000 7f503580a640 1 -- 172.21.15.116:0/1185362628 learned_addr learned my addr 172.21.15.116:0/1185362628 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:22:08.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.995+0000 7f5035009640 1 --2- 172.21.15.116:0/1185362628 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f503006ff80 0x7f50301e0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:08.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.995+0000 7f503600b640 1 --2- 172.21.15.116:0/1185362628 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50301f1f00 0x7f503014df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:08.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.995+0000 7f503580a640 1 -- 172.21.15.116:0/1185362628 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f503006ff80 msgr2=0x7f50301e0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:08.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.995+0000 7f503580a640 1 --2- 172.21.15.116:0/1185362628 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f503006ff80 0x7f50301e0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:08.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.995+0000 7f503580a640 1 -- 172.21.15.116:0/1185362628 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50301f1f00 msgr2=0x7f503014df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:08.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.995+0000 7f503580a640 1 --2- 172.21.15.116:0/1185362628 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50301f1f00 0x7f503014df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:08.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.995+0000 7f503580a640 1 -- 172.21.15.116:0/1185362628 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50300acb60 con 0x7f501c004810 2024-04-25T00:22:08.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.995+0000 7f503600b640 1 --2- 172.21.15.116:0/1185362628 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50301f1f00 0x7f503014df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:22:08.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.995+0000 7f503580a640 1 --2- 172.21.15.116:0/1185362628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f501c004810 0x7f50300ad040 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f502c00c4d0 tx=0x7f502c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:08.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.995+0000 7f50267fc640 1 -- 172.21.15.116:0/1185362628 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f502c0052d0 con 0x7f501c004810 2024-04-25T00:22:08.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.995+0000 7f50267fc640 1 -- 172.21.15.116:0/1185362628 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f502c005470 con 0x7f501c004810 2024-04-25T00:22:08.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.995+0000 7f5037a6e640 1 -- 172.21.15.116:0/1185362628 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f503006f630 con 0x7f501c004810 2024-04-25T00:22:08.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.995+0000 7f50267fc640 1 -- 172.21.15.116:0/1185362628 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f502c012870 con 0x7f501c004810 2024-04-25T00:22:08.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.995+0000 7f5037a6e640 1 -- 172.21.15.116:0/1185362628 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50301f2900 con 0x7f501c004810 2024-04-25T00:22:08.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.999+0000 7f5037a6e640 1 -- 172.21.15.116:0/1185362628 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5030068350 con 0x7f501c004810 2024-04-25T00:22:08.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.999+0000 7f50267fc640 1 -- 172.21.15.116:0/1185362628 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f502c012a10 con 0x7f501c004810 2024-04-25T00:22:08.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.999+0000 7f50267fc640 1 --2- 172.21.15.116:0/1185362628 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f50140678c0 0x7f5014069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:08.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.999+0000 7f50267fc640 1 -- 172.21.15.116:0/1185362628 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 9418+0+0 (secure 0 0 0) 0x7f502c086770 con 0x7f501c004810 2024-04-25T00:22:08.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.999+0000 7f5035009640 1 --2- 172.21.15.116:0/1185362628 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f50140678c0 0x7f5014069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:08.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:07.999+0000 7f5035009640 1 --2- 172.21.15.116:0/1185362628 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f50140678c0 0x7f5014069d80 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f50200ae000 tx=0x7f5020008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:08.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:08.003+0000 7f50267fc640 1 -- 172.21.15.116:0/1185362628 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f502c056470 con 0x7f501c004810 2024-04-25T00:22:08.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:08.199+0000 7f5037a6e640 1 -- 172.21.15.116:0/1185362628 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f50300ae520 con 0x7f501c004810 2024-04-25T00:22:08.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:08.199+0000 7f50267fc640 1 -- 172.21.15.116:0/1185362628 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) ==== 95+0+6501 (secure 0 0 0) 0x7f502c055d70 con 0x7f501c004810 2024-04-25T00:22:08.199 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:22:08.200 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":172,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7989,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/221971311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":221971311},{"type":"v1","addr":"172.21.15.116:6835","nonce":221971311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":172}],"filesystems":[{"mdsmap":{"epoch":171,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:21:43.889982+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6125},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6125":{"gid":6125,"name":"d","rank":0,"incarnation":168,"state":"up:active","state_seq":14,"addr":"172.21.15.174:6837/321466844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":321466844},{"type":"v1","addr":"172.21.15.174:6837","nonce":321466844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6125,"qdb_cluster":[6125]},"id":1},{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:20:13.935378+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7722":{"gid":7722,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7722,"qdb_cluster":[7722]},"id":2}]} 2024-04-25T00:22:08.200 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 172 2024-04-25T00:22:08.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:08.203+0000 7f5037a6e640 1 -- 172.21.15.116:0/1185362628 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f50140678c0 msgr2=0x7f5014069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:08.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:08.203+0000 7f5037a6e640 1 --2- 172.21.15.116:0/1185362628 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f50140678c0 0x7f5014069d80 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f50200ae000 tx=0x7f5020008040 comp rx=0 tx=0).stop 2024-04-25T00:22:08.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:08.203+0000 7f5037a6e640 1 -- 172.21.15.116:0/1185362628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f501c004810 msgr2=0x7f50300ad040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:08.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:08.203+0000 7f5037a6e640 1 --2- 172.21.15.116:0/1185362628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f501c004810 0x7f50300ad040 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f502c00c4d0 tx=0x7f502c00c990 comp rx=0 tx=0).stop 2024-04-25T00:22:08.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:08.203+0000 7f5037a6e640 1 -- 172.21.15.116:0/1185362628 shutdown_connections 2024-04-25T00:22:08.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:08.203+0000 7f5037a6e640 1 --2- 172.21.15.116:0/1185362628 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f50140678c0 0x7f5014069d80 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:08.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:08.203+0000 7f5037a6e640 1 --2- 172.21.15.116:0/1185362628 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f50301f1f00 0x7f503014df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:08.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:08.203+0000 7f5037a6e640 1 --2- 172.21.15.116:0/1185362628 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f503006ff80 0x7f50301e0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:08.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:08.203+0000 7f5037a6e640 1 --2- 172.21.15.116:0/1185362628 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f501c004810 0x7f50300ad040 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:08.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:08.203+0000 7f5037a6e640 1 -- 172.21.15.116:0/1185362628 >> 172.21.15.116:0/1185362628 conn(0x7f5030074a80 msgr2=0x7f503006c460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:08.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:08.203+0000 7f5037a6e640 1 -- 172.21.15.116:0/1185362628 shutdown_connections 2024-04-25T00:22:08.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:08.203+0000 7f5037a6e640 1 -- 172.21.15.116:0/1185362628 wait complete. 2024-04-25T00:22:08.218 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:22:10.064 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:22:10.067 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:22:10.220 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:22:10.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.351+0000 7fde892e4640 1 Processor -- start 2024-04-25T00:22:10.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.351+0000 7fde892e4640 1 -- start start 2024-04-25T00:22:10.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.351+0000 7fde892e4640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fde84152d80 0x7fde8406bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:10.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.351+0000 7fde892e4640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fde8406c290 0x7fde84157390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:10.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.351+0000 7fde892e4640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde840c9150 con 0x7fde8406d5f0 2024-04-25T00:22:10.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.351+0000 7fde892e4640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fde840c9d60 con 0x7fde84152d80 2024-04-25T00:22:10.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.351+0000 7fde892e4640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fde840c9430 con 0x7fde8406c290 2024-04-25T00:22:10.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.351+0000 7fde82ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fde84152d80 0x7fde8406bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:10.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.351+0000 7fde827fc640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fde8406d5f0 0x7fde84156c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:49366/0 (socket says 172.21.15.116:49366) 2024-04-25T00:22:10.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.351+0000 7fde827fc640 1 -- 172.21.15.116:0/782474354 learned_addr learned my addr 172.21.15.116:0/782474354 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:22:10.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.351+0000 7fde837fe640 1 --2- 172.21.15.116:0/782474354 >> v2:172.21.15.174:3300/0 conn(0x7fde8406c290 0x7fde84157390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:10.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.351+0000 7fde81ffb640 1 -- 172.21.15.116:0/782474354 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 678585225 0 0) 0x7fde840c9150 con 0x7fde8406d5f0 2024-04-25T00:22:10.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.351+0000 7fde81ffb640 1 -- 172.21.15.116:0/782474354 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fde6c003530 con 0x7fde8406d5f0 2024-04-25T00:22:10.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.351+0000 7fde82ffd640 1 -- 172.21.15.116:0/782474354 >> v2:172.21.15.174:3300/0 conn(0x7fde8406c290 msgr2=0x7fde84157390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:10.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.351+0000 7fde82ffd640 1 --2- 172.21.15.116:0/782474354 >> v2:172.21.15.174:3300/0 conn(0x7fde8406c290 0x7fde84157390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:10.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.351+0000 7fde82ffd640 1 -- 172.21.15.116:0/782474354 >> v1:172.21.15.116:6789/0 conn(0x7fde8406d5f0 legacy=0x7fde84156c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:10.352 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.351+0000 7fde82ffd640 1 -- 172.21.15.116:0/782474354 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde840c9710 con 0x7fde84152d80 2024-04-25T00:22:10.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.355+0000 7fde82ffd640 1 --2- 172.21.15.116:0/782474354 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde84152d80 0x7fde8406bd50 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7fde740097c0 tx=0x7fde7401c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81e47951f62216d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:10.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.355+0000 7fde81ffb640 1 -- 172.21.15.116:0/782474354 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde740b1070 con 0x7fde84152d80 2024-04-25T00:22:10.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.355+0000 7fde81ffb640 1 -- 172.21.15.116:0/782474354 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde740ac620 con 0x7fde84152d80 2024-04-25T00:22:10.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.355+0000 7fde81ffb640 1 -- 172.21.15.116:0/782474354 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde740ac9d0 con 0x7fde84152d80 2024-04-25T00:22:10.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.355+0000 7fde892e4640 1 -- 172.21.15.116:0/782474354 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde84152d80 msgr2=0x7fde8406bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:10.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.355+0000 7fde892e4640 1 --2- 172.21.15.116:0/782474354 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde84152d80 0x7fde8406bd50 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7fde740097c0 tx=0x7fde7401c3a0 comp rx=0 tx=0).stop 2024-04-25T00:22:10.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.355+0000 7fde892e4640 1 -- 172.21.15.116:0/782474354 shutdown_connections 2024-04-25T00:22:10.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.355+0000 7fde892e4640 1 --2- 172.21.15.116:0/782474354 >> v2:172.21.15.174:3300/0 conn(0x7fde8406c290 0x7fde84157390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:10.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.355+0000 7fde892e4640 1 --2- 172.21.15.116:0/782474354 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde84152d80 0x7fde8406bd50 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:10.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.355+0000 7fde892e4640 1 -- 172.21.15.116:0/782474354 >> 172.21.15.116:0/782474354 conn(0x7fde840acdd0 msgr2=0x7fde840ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:10.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.355+0000 7fde892e4640 1 -- 172.21.15.116:0/782474354 shutdown_connections 2024-04-25T00:22:10.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.355+0000 7fde892e4640 1 -- 172.21.15.116:0/782474354 wait complete. 2024-04-25T00:22:10.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.355+0000 7fde892e4640 1 Processor -- start 2024-04-25T00:22:10.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.355+0000 7fde892e4640 1 -- start start 2024-04-25T00:22:10.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.355+0000 7fde892e4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde8406c290 0x7fde8414c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:10.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.355+0000 7fde892e4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fde8406d5f0 0x7fde84151f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:10.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.355+0000 7fde892e4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde84152d80 0x7fde840c6980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:10.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.355+0000 7fde892e4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fde840caf00 con 0x7fde84152d80 2024-04-25T00:22:10.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.355+0000 7fde892e4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fde840cad80 con 0x7fde8406c290 2024-04-25T00:22:10.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.355+0000 7fde892e4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fde841628d0 con 0x7fde8406d5f0 2024-04-25T00:22:10.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.355+0000 7fde837fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde84152d80 0x7fde840c6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:10.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.355+0000 7fde837fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde84152d80 0x7fde840c6980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56968/0 (socket says 172.21.15.116:56968) 2024-04-25T00:22:10.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.355+0000 7fde837fe640 1 -- 172.21.15.116:0/3833357053 learned_addr learned my addr 172.21.15.116:0/3833357053 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:22:10.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.355+0000 7fde827fc640 1 --2- 172.21.15.116:0/3833357053 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fde8406d5f0 0x7fde84151f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:10.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.355+0000 7fde82ffd640 1 --2- 172.21.15.116:0/3833357053 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde8406c290 0x7fde8414c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:10.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.355+0000 7fde837fe640 1 -- 172.21.15.116:0/3833357053 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fde8406d5f0 msgr2=0x7fde84151f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:10.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.355+0000 7fde837fe640 1 --2- 172.21.15.116:0/3833357053 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fde8406d5f0 0x7fde84151f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:10.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.355+0000 7fde837fe640 1 -- 172.21.15.116:0/3833357053 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde8406c290 msgr2=0x7fde8414c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:10.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.355+0000 7fde837fe640 1 --2- 172.21.15.116:0/3833357053 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde8406c290 0x7fde8414c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:10.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.355+0000 7fde837fe640 1 -- 172.21.15.116:0/3833357053 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde84151a60 con 0x7fde84152d80 2024-04-25T00:22:10.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.355+0000 7fde837fe640 1 --2- 172.21.15.116:0/3833357053 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde84152d80 0x7fde840c6980 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7fde7c009d30 tx=0x7fde7c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:10.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.355+0000 7fde6b7fe640 1 -- 172.21.15.116:0/3833357053 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde7c018020 con 0x7fde84152d80 2024-04-25T00:22:10.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.355+0000 7fde6b7fe640 1 -- 172.21.15.116:0/3833357053 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde7c014480 con 0x7fde84152d80 2024-04-25T00:22:10.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.355+0000 7fde892e4640 1 -- 172.21.15.116:0/3833357053 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde841f1640 con 0x7fde84152d80 2024-04-25T00:22:10.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.355+0000 7fde6b7fe640 1 -- 172.21.15.116:0/3833357053 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde7c014870 con 0x7fde84152d80 2024-04-25T00:22:10.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.355+0000 7fde892e4640 1 -- 172.21.15.116:0/3833357053 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde84152460 con 0x7fde84152d80 2024-04-25T00:22:10.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.359+0000 7fde892e4640 1 -- 172.21.15.116:0/3833357053 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde84068350 con 0x7fde84152d80 2024-04-25T00:22:10.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.359+0000 7fde6b7fe640 1 -- 172.21.15.116:0/3833357053 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fde7c01b020 con 0x7fde84152d80 2024-04-25T00:22:10.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.359+0000 7fde6b7fe640 1 --2- 172.21.15.116:0/3833357053 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fde700678c0 0x7fde70069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:10.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.359+0000 7fde6b7fe640 1 -- 172.21.15.116:0/3833357053 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 9418+0+0 (secure 0 0 0) 0x7fde7c087560 con 0x7fde84152d80 2024-04-25T00:22:10.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.359+0000 7fde82ffd640 1 --2- 172.21.15.116:0/3833357053 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fde700678c0 0x7fde70069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:10.360 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.359+0000 7fde82ffd640 1 --2- 172.21.15.116:0/3833357053 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fde700678c0 0x7fde70069d80 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fde740ae000 tx=0x7fde74008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:10.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.363+0000 7fde6b7fe640 1 -- 172.21.15.116:0/3833357053 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fde7c022070 con 0x7fde84152d80 2024-04-25T00:22:10.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.551+0000 7fde892e4640 1 -- 172.21.15.116:0/3833357053 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fde8414e430 con 0x7fde84152d80 2024-04-25T00:22:10.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.551+0000 7fde6b7fe640 1 -- 172.21.15.116:0/3833357053 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) ==== 95+0+6501 (secure 0 0 0) 0x7fde7c010040 con 0x7fde84152d80 2024-04-25T00:22:10.552 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:22:10.552 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":172,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7989,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/221971311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":221971311},{"type":"v1","addr":"172.21.15.116:6835","nonce":221971311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":172}],"filesystems":[{"mdsmap":{"epoch":171,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:21:43.889982+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6125},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6125":{"gid":6125,"name":"d","rank":0,"incarnation":168,"state":"up:active","state_seq":14,"addr":"172.21.15.174:6837/321466844","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":321466844},{"type":"v1","addr":"172.21.15.174:6837","nonce":321466844}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6125,"qdb_cluster":[6125]},"id":1},{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:20:13.935378+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7722":{"gid":7722,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7722,"qdb_cluster":[7722]},"id":2}]} 2024-04-25T00:22:10.552 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 172 2024-04-25T00:22:10.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.555+0000 7fde892e4640 1 -- 172.21.15.116:0/3833357053 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fde700678c0 msgr2=0x7fde70069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:10.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.555+0000 7fde892e4640 1 --2- 172.21.15.116:0/3833357053 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fde700678c0 0x7fde70069d80 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fde740ae000 tx=0x7fde74008040 comp rx=0 tx=0).stop 2024-04-25T00:22:10.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.555+0000 7fde892e4640 1 -- 172.21.15.116:0/3833357053 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde84152d80 msgr2=0x7fde840c6980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:10.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.555+0000 7fde892e4640 1 --2- 172.21.15.116:0/3833357053 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde84152d80 0x7fde840c6980 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7fde7c009d30 tx=0x7fde7c00c490 comp rx=0 tx=0).stop 2024-04-25T00:22:10.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.555+0000 7fde892e4640 1 -- 172.21.15.116:0/3833357053 shutdown_connections 2024-04-25T00:22:10.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.555+0000 7fde892e4640 1 --2- 172.21.15.116:0/3833357053 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fde700678c0 0x7fde70069d80 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:10.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.555+0000 7fde892e4640 1 --2- 172.21.15.116:0/3833357053 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fde84152d80 0x7fde840c6980 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:10.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.555+0000 7fde892e4640 1 --2- 172.21.15.116:0/3833357053 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fde8406d5f0 0x7fde84151f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:10.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.555+0000 7fde892e4640 1 --2- 172.21.15.116:0/3833357053 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fde8406c290 0x7fde8414c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:10.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.555+0000 7fde892e4640 1 -- 172.21.15.116:0/3833357053 >> 172.21.15.116:0/3833357053 conn(0x7fde840acdd0 msgr2=0x7fde8415f470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:10.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.555+0000 7fde892e4640 1 -- 172.21.15.116:0/3833357053 shutdown_connections 2024-04-25T00:22:10.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:10.555+0000 7fde892e4640 1 -- 172.21.15.116:0/3833357053 wait complete. 2024-04-25T00:22:10.569 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:22:12.570 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:22:12.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.703+0000 7ff2ec0ea640 1 Processor -- start 2024-04-25T00:22:12.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.703+0000 7ff2ec0ea640 1 -- start start 2024-04-25T00:22:12.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.703+0000 7ff2ec0ea640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff2e4150d20 0x7ff2e40c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:12.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.703+0000 7ff2ec0ea640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff2e40c2f50 0x7ff2e41538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:12.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.703+0000 7ff2ec0ea640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2e40c3910 con 0x7ff2e40a94a0 2024-04-25T00:22:12.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.703+0000 7ff2ec0ea640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7ff2e4154530 con 0x7ff2e4150d20 2024-04-25T00:22:12.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.703+0000 7ff2ec0ea640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7ff2e40c16e0 con 0x7ff2e40c2f50 2024-04-25T00:22:12.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.703+0000 7ff2e9e86640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff2e4150d20 0x7ff2e40c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:12.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.703+0000 7ff2ea687640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7ff2e40a94a0 0x7ff2e4153e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:49376/0 (socket says 172.21.15.116:49376) 2024-04-25T00:22:12.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.703+0000 7ff2ea687640 1 -- 172.21.15.116:0/2434506850 learned_addr learned my addr 172.21.15.116:0/2434506850 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:22:12.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.703+0000 7ff2e9685640 1 --2- 172.21.15.116:0/2434506850 >> v2:172.21.15.174:3300/0 conn(0x7ff2e40c2f50 0x7ff2e41538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:12.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.703+0000 7ff2e9e86640 1 -- 172.21.15.116:0/2434506850 >> v2:172.21.15.174:3300/0 conn(0x7ff2e40c2f50 msgr2=0x7ff2e41538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:12.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.703+0000 7ff2e9e86640 1 --2- 172.21.15.116:0/2434506850 >> v2:172.21.15.174:3300/0 conn(0x7ff2e40c2f50 0x7ff2e41538e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:12.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.703+0000 7ff2e9e86640 1 -- 172.21.15.116:0/2434506850 >> v1:172.21.15.116:6789/0 conn(0x7ff2e40a94a0 legacy=0x7ff2e4153e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:12.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.703+0000 7ff2e9e86640 1 -- 172.21.15.116:0/2434506850 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2e40c1400 con 0x7ff2e4150d20 2024-04-25T00:22:12.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.703+0000 7ff2e8e84640 1 -- 172.21.15.116:0/2434506850 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1827969166 0 0) 0x7ff2e40c3910 con 0x7ff2e40a94a0 2024-04-25T00:22:12.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.707+0000 7ff2e9e86640 1 --2- 172.21.15.116:0/2434506850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2e4150d20 0x7ff2e40c2a10 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7ff2e4070710 tx=0x7ff2e001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ade89e464e58fd4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:12.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.707+0000 7ff2e8e84640 1 -- 172.21.15.116:0/2434506850 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2e00b1070 con 0x7ff2e4150d20 2024-04-25T00:22:12.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.707+0000 7ff2e8e84640 1 -- 172.21.15.116:0/2434506850 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2e00ac5f0 con 0x7ff2e4150d20 2024-04-25T00:22:12.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.707+0000 7ff2e8e84640 1 -- 172.21.15.116:0/2434506850 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2e00ac9c0 con 0x7ff2e4150d20 2024-04-25T00:22:12.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.707+0000 7ff2ec0ea640 1 -- 172.21.15.116:0/2434506850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2e4150d20 msgr2=0x7ff2e40c2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:12.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.707+0000 7ff2ec0ea640 1 --2- 172.21.15.116:0/2434506850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2e4150d20 0x7ff2e40c2a10 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7ff2e4070710 tx=0x7ff2e001c480 comp rx=0 tx=0).stop 2024-04-25T00:22:12.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.707+0000 7ff2ec0ea640 1 -- 172.21.15.116:0/2434506850 shutdown_connections 2024-04-25T00:22:12.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.707+0000 7ff2ec0ea640 1 --2- 172.21.15.116:0/2434506850 >> v2:172.21.15.174:3300/0 conn(0x7ff2e40c2f50 0x7ff2e41538e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:12.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.707+0000 7ff2ec0ea640 1 --2- 172.21.15.116:0/2434506850 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2e4150d20 0x7ff2e40c2a10 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:12.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.707+0000 7ff2ec0ea640 1 -- 172.21.15.116:0/2434506850 >> 172.21.15.116:0/2434506850 conn(0x7ff2e40c3b00 msgr2=0x7ff2e40c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:12.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.707+0000 7ff2ec0ea640 1 -- 172.21.15.116:0/2434506850 shutdown_connections 2024-04-25T00:22:12.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.707+0000 7ff2ec0ea640 1 -- 172.21.15.116:0/2434506850 wait complete. 2024-04-25T00:22:12.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.707+0000 7ff2ec0ea640 1 Processor -- start 2024-04-25T00:22:12.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.707+0000 7ff2ec0ea640 1 -- start start 2024-04-25T00:22:12.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.707+0000 7ff2ec0ea640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff2e40a94a0 0x7ff2e4140a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:12.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.707+0000 7ff2ec0ea640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff2e40c2f50 0x7ff2e414f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:12.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.707+0000 7ff2ec0ea640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2e4150d20 0x7ff2e41ee950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:12.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.707+0000 7ff2ec0ea640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff2e41608b0 con 0x7ff2e4150d20 2024-04-25T00:22:12.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.707+0000 7ff2ec0ea640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff2e4160730 con 0x7ff2e40a94a0 2024-04-25T00:22:12.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.707+0000 7ff2ec0ea640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff2e4160a30 con 0x7ff2e40c2f50 2024-04-25T00:22:12.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.707+0000 7ff2ea687640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2e4150d20 0x7ff2e41ee950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:12.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.707+0000 7ff2ea687640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2e4150d20 0x7ff2e41ee950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56982/0 (socket says 172.21.15.116:56982) 2024-04-25T00:22:12.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.707+0000 7ff2ea687640 1 -- 172.21.15.116:0/1531159204 learned_addr learned my addr 172.21.15.116:0/1531159204 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:22:12.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.707+0000 7ff2e9e86640 1 --2- 172.21.15.116:0/1531159204 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff2e40a94a0 0x7ff2e4140a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:12.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.707+0000 7ff2e9685640 1 --2- 172.21.15.116:0/1531159204 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff2e40c2f50 0x7ff2e414f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:12.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.707+0000 7ff2ea687640 1 -- 172.21.15.116:0/1531159204 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff2e40c2f50 msgr2=0x7ff2e414f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:12.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.707+0000 7ff2ea687640 1 --2- 172.21.15.116:0/1531159204 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff2e40c2f50 0x7ff2e414f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:12.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.707+0000 7ff2ea687640 1 -- 172.21.15.116:0/1531159204 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff2e40a94a0 msgr2=0x7ff2e4140a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:12.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.707+0000 7ff2ea687640 1 --2- 172.21.15.116:0/1531159204 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff2e40a94a0 0x7ff2e4140a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:12.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.707+0000 7ff2ea687640 1 -- 172.21.15.116:0/1531159204 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2e40c5af0 con 0x7ff2e4150d20 2024-04-25T00:22:12.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.707+0000 7ff2e9e86640 1 --2- 172.21.15.116:0/1531159204 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff2e40a94a0 0x7ff2e4140a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:22:12.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.707+0000 7ff2ea687640 1 --2- 172.21.15.116:0/1531159204 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2e4150d20 0x7ff2e41ee950 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7ff2dc00c7d0 tx=0x7ff2dc00cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:12.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.707+0000 7ff2da7fc640 1 -- 172.21.15.116:0/1531159204 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2dc018070 con 0x7ff2e4150d20 2024-04-25T00:22:12.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.707+0000 7ff2da7fc640 1 -- 172.21.15.116:0/1531159204 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2dc00a640 con 0x7ff2e4150d20 2024-04-25T00:22:12.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.707+0000 7ff2ec0ea640 1 -- 172.21.15.116:0/1531159204 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2e41f2db0 con 0x7ff2e4150d20 2024-04-25T00:22:12.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.707+0000 7ff2da7fc640 1 -- 172.21.15.116:0/1531159204 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2dc00aa70 con 0x7ff2e4150d20 2024-04-25T00:22:12.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.711+0000 7ff2ec0ea640 1 -- 172.21.15.116:0/1531159204 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2e41f22d0 con 0x7ff2e4150d20 2024-04-25T00:22:12.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.711+0000 7ff2ec0ea640 1 -- 172.21.15.116:0/1531159204 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2e4068350 con 0x7ff2e4150d20 2024-04-25T00:22:12.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.711+0000 7ff2da7fc640 1 -- 172.21.15.116:0/1531159204 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff2dc01b070 con 0x7ff2e4150d20 2024-04-25T00:22:12.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.711+0000 7ff2da7fc640 1 --2- 172.21.15.116:0/1531159204 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff2c80678c0 0x7ff2c8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:12.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.711+0000 7ff2da7fc640 1 -- 172.21.15.116:0/1531159204 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 9504+0+0 (secure 0 0 0) 0x7ff2dc086110 con 0x7ff2e4150d20 2024-04-25T00:22:12.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.711+0000 7ff2e9e86640 1 --2- 172.21.15.116:0/1531159204 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff2c80678c0 0x7ff2c8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:12.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.711+0000 7ff2e9e86640 1 --2- 172.21.15.116:0/1531159204 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff2c80678c0 0x7ff2c8069d80 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7ff2e00ae000 tx=0x7ff2e00a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:12.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.715+0000 7ff2da7fc640 1 -- 172.21.15.116:0/1531159204 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ff2dc055dc0 con 0x7ff2e4150d20 2024-04-25T00:22:12.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.911+0000 7ff2ec0ea640 1 -- 172.21.15.116:0/1531159204 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff2e415d980 con 0x7ff2e4150d20 2024-04-25T00:22:12.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.911+0000 7ff2da7fc640 1 -- 172.21.15.116:0/1531159204 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 174 v174) ==== 95+0+5670 (secure 0 0 0) 0x7ff2dc0556c0 con 0x7ff2e4150d20 2024-04-25T00:22:12.911 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:22:12.911 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":174,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:22:11.026293+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7989},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7989":{"gid":7989,"name":"c","rank":0,"incarnation":173,"state":"up:reconnect","state_seq":8,"addr":"172.21.15.116:6835/221971311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":221971311},{"type":"v1","addr":"172.21.15.116:6835","nonce":221971311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:20:13.935378+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7722":{"gid":7722,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7722,"qdb_cluster":[7722]},"id":2}]} 2024-04-25T00:22:12.911 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 174 2024-04-25T00:22:12.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.911+0000 7ff2ec0ea640 1 -- 172.21.15.116:0/1531159204 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff2c80678c0 msgr2=0x7ff2c8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:12.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.911+0000 7ff2ec0ea640 1 --2- 172.21.15.116:0/1531159204 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff2c80678c0 0x7ff2c8069d80 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7ff2e00ae000 tx=0x7ff2e00a7000 comp rx=0 tx=0).stop 2024-04-25T00:22:12.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.911+0000 7ff2ec0ea640 1 -- 172.21.15.116:0/1531159204 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2e4150d20 msgr2=0x7ff2e41ee950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:12.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.911+0000 7ff2ec0ea640 1 --2- 172.21.15.116:0/1531159204 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2e4150d20 0x7ff2e41ee950 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7ff2dc00c7d0 tx=0x7ff2dc00cc90 comp rx=0 tx=0).stop 2024-04-25T00:22:12.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.911+0000 7ff2ec0ea640 1 -- 172.21.15.116:0/1531159204 shutdown_connections 2024-04-25T00:22:12.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.911+0000 7ff2ec0ea640 1 --2- 172.21.15.116:0/1531159204 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff2c80678c0 0x7ff2c8069d80 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:12.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.911+0000 7ff2ec0ea640 1 --2- 172.21.15.116:0/1531159204 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff2e4150d20 0x7ff2e41ee950 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:12.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.911+0000 7ff2ec0ea640 1 --2- 172.21.15.116:0/1531159204 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff2e40c2f50 0x7ff2e414f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:12.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.911+0000 7ff2ec0ea640 1 --2- 172.21.15.116:0/1531159204 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff2e40a94a0 0x7ff2e4140a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:12.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.911+0000 7ff2ec0ea640 1 -- 172.21.15.116:0/1531159204 >> 172.21.15.116:0/1531159204 conn(0x7ff2e40c3b00 msgr2=0x7ff2e40aa190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:12.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.911+0000 7ff2ec0ea640 1 -- 172.21.15.116:0/1531159204 shutdown_connections 2024-04-25T00:22:12.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:12.911+0000 7ff2ec0ea640 1 -- 172.21.15.116:0/1531159204 wait complete. 2024-04-25T00:22:12.934 INFO:tasks.mds_thrash.fs.[a]:mds.d down, removed from mdsmap 2024-04-25T00:22:12.934 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-04-25T00:22:12.935 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:22:13.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.055+0000 7f03d1e18640 1 Processor -- start 2024-04-25T00:22:13.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.055+0000 7f03d1e18640 1 -- start start 2024-04-25T00:22:13.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.055+0000 7f03d1e18640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f03cc14c720 0x7f03cc1563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:13.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.055+0000 7f03d1e18640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f03cc0a94a0 0x7f03cc0c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:13.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.055+0000 7f03d1e18640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f03cc0c2f50 0x7f03cc071680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:13.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.055+0000 7f03d1e18640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f03cc153460 con 0x7f03cc14c720 2024-04-25T00:22:13.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.055+0000 7f03d1e18640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f03cc071220 con 0x7f03cc0c2f50 2024-04-25T00:22:13.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.055+0000 7f03d1e18640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f03cc152420 con 0x7f03cc0a94a0 2024-04-25T00:22:13.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.059+0000 7f03cb7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f03cc14c720 0x7f03cc1563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:13.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.059+0000 7f03cb7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f03cc14c720 0x7f03cc1563d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56986/0 (socket says 172.21.15.116:56986) 2024-04-25T00:22:13.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.059+0000 7f03cb7fe640 1 -- 172.21.15.116:0/1425578329 learned_addr learned my addr 172.21.15.116:0/1425578329 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:22:13.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.059+0000 7f03caffd640 1 --2- 172.21.15.116:0/1425578329 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f03cc0a94a0 0x7f03cc0c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:13.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.059+0000 7f03cbfff640 1 --2- 172.21.15.116:0/1425578329 >> v2:172.21.15.174:3300/0 conn(0x7f03cc0c2f50 0x7f03cc071680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:13.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.059+0000 7f03cb7fe640 1 -- 172.21.15.116:0/1425578329 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f03cc0a94a0 msgr2=0x7f03cc0c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:13.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.059+0000 7f03cb7fe640 1 --2- 172.21.15.116:0/1425578329 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f03cc0a94a0 0x7f03cc0c2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:13.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.059+0000 7f03cb7fe640 1 -- 172.21.15.116:0/1425578329 >> v2:172.21.15.174:3300/0 conn(0x7f03cc0c2f50 msgr2=0x7f03cc071680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:13.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.059+0000 7f03cb7fe640 1 --2- 172.21.15.116:0/1425578329 >> v2:172.21.15.174:3300/0 conn(0x7f03cc0c2f50 0x7f03cc071680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:13.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.059+0000 7f03cb7fe640 1 -- 172.21.15.116:0/1425578329 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03cc14cb00 con 0x7f03cc14c720 2024-04-25T00:22:13.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.059+0000 7f03cb7fe640 1 --2- 172.21.15.116:0/1425578329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03cc14c720 0x7f03cc1563d0 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f03cc0c35f0 tx=0x7f03bc01c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bea29be6d0cb1682 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:13.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.059+0000 7f03ca7fc640 1 -- 172.21.15.116:0/1425578329 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03bc0ab620 con 0x7f03cc14c720 2024-04-25T00:22:13.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.059+0000 7f03ca7fc640 1 -- 172.21.15.116:0/1425578329 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03bc0b0070 con 0x7f03cc14c720 2024-04-25T00:22:13.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.059+0000 7f03ca7fc640 1 -- 172.21.15.116:0/1425578329 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03bc0ab9e0 con 0x7f03cc14c720 2024-04-25T00:22:13.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.063+0000 7f03d1e18640 1 -- 172.21.15.116:0/1425578329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03cc14c720 msgr2=0x7f03cc1563d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:13.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.063+0000 7f03d1e18640 1 --2- 172.21.15.116:0/1425578329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03cc14c720 0x7f03cc1563d0 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f03cc0c35f0 tx=0x7f03bc01c460 comp rx=0 tx=0).stop 2024-04-25T00:22:13.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.063+0000 7f03d1e18640 1 -- 172.21.15.116:0/1425578329 shutdown_connections 2024-04-25T00:22:13.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.063+0000 7f03d1e18640 1 --2- 172.21.15.116:0/1425578329 >> v2:172.21.15.174:3300/0 conn(0x7f03cc0c2f50 0x7f03cc071680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:13.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.063+0000 7f03d1e18640 1 --2- 172.21.15.116:0/1425578329 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f03cc0a94a0 0x7f03cc0c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:13.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.063+0000 7f03d1e18640 1 --2- 172.21.15.116:0/1425578329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03cc14c720 0x7f03cc1563d0 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:13.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.063+0000 7f03d1e18640 1 -- 172.21.15.116:0/1425578329 >> 172.21.15.116:0/1425578329 conn(0x7f03cc0c3b00 msgr2=0x7f03cc0c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:13.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.063+0000 7f03d1e18640 1 -- 172.21.15.116:0/1425578329 shutdown_connections 2024-04-25T00:22:13.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.063+0000 7f03d1e18640 1 -- 172.21.15.116:0/1425578329 wait complete. 2024-04-25T00:22:13.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.063+0000 7f03d1e18640 1 Processor -- start 2024-04-25T00:22:13.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.063+0000 7f03d1e18640 1 -- start start 2024-04-25T00:22:13.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.063+0000 7f03d1e18640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03cc0a94a0 0x7f03cc140070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:13.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.063+0000 7f03d1e18640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f03cc0c2f50 0x7f03cc0c5d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:13.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.063+0000 7f03d1e18640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f03cc14c720 0x7f03cc1ef7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:13.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.063+0000 7f03d1e18640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f03cc071fd0 con 0x7f03cc0a94a0 2024-04-25T00:22:13.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.063+0000 7f03d1e18640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f03cc152f40 con 0x7f03cc0c2f50 2024-04-25T00:22:13.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.063+0000 7f03d1e18640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f03cc072150 con 0x7f03cc14c720 2024-04-25T00:22:13.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.063+0000 7f03cb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03cc0a94a0 0x7f03cc140070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:13.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.063+0000 7f03cb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03cc0a94a0 0x7f03cc140070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57002/0 (socket says 172.21.15.116:57002) 2024-04-25T00:22:13.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.063+0000 7f03cb7fe640 1 -- 172.21.15.116:0/2567417369 learned_addr learned my addr 172.21.15.116:0/2567417369 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:22:13.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.063+0000 7f03caffd640 1 --2- 172.21.15.116:0/2567417369 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f03cc0c2f50 0x7f03cc0c5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:13.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.063+0000 7f03cbfff640 1 --2- 172.21.15.116:0/2567417369 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f03cc14c720 0x7f03cc1ef7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:13.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.063+0000 7f03cb7fe640 1 -- 172.21.15.116:0/2567417369 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f03cc14c720 msgr2=0x7f03cc1ef7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:13.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.063+0000 7f03cb7fe640 1 --2- 172.21.15.116:0/2567417369 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f03cc14c720 0x7f03cc1ef7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:13.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.063+0000 7f03cb7fe640 1 -- 172.21.15.116:0/2567417369 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f03cc0c2f50 msgr2=0x7f03cc0c5d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:13.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.063+0000 7f03cb7fe640 1 --2- 172.21.15.116:0/2567417369 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f03cc0c2f50 0x7f03cc0c5d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:13.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.063+0000 7f03cb7fe640 1 -- 172.21.15.116:0/2567417369 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03cc1f0420 con 0x7f03cc0a94a0 2024-04-25T00:22:13.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.063+0000 7f03cb7fe640 1 --2- 172.21.15.116:0/2567417369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03cc0a94a0 0x7f03cc140070 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f03bc01ab10 tx=0x7f03bc01aca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:13.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.063+0000 7f03b3fff640 1 -- 172.21.15.116:0/2567417369 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03bc0abda0 con 0x7f03cc0a94a0 2024-04-25T00:22:13.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.063+0000 7f03b3fff640 1 -- 172.21.15.116:0/2567417369 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03bc0be3f0 con 0x7f03cc0a94a0 2024-04-25T00:22:13.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.063+0000 7f03b3fff640 1 -- 172.21.15.116:0/2567417369 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03bc0be7e0 con 0x7f03cc0a94a0 2024-04-25T00:22:13.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.063+0000 7f03d1e18640 1 -- 172.21.15.116:0/2567417369 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03cc0c6940 con 0x7f03cc0a94a0 2024-04-25T00:22:13.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.063+0000 7f03d1e18640 1 -- 172.21.15.116:0/2567417369 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03cc0c5850 con 0x7f03cc0a94a0 2024-04-25T00:22:13.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.063+0000 7f03d1e18640 1 -- 172.21.15.116:0/2567417369 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03cc068350 con 0x7f03cc0a94a0 2024-04-25T00:22:13.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.067+0000 7f03b3fff640 1 -- 172.21.15.116:0/2567417369 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f03bc0c7440 con 0x7f03cc0a94a0 2024-04-25T00:22:13.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.067+0000 7f03b3fff640 1 --2- 172.21.15.116:0/2567417369 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f03b80678c0 0x7f03b8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:13.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.067+0000 7f03b3fff640 1 -- 172.21.15.116:0/2567417369 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 9504+0+0 (secure 0 0 0) 0x7f03bc1212b0 con 0x7f03cc0a94a0 2024-04-25T00:22:13.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.067+0000 7f03caffd640 1 --2- 172.21.15.116:0/2567417369 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f03b80678c0 0x7f03b8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:13.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.067+0000 7f03caffd640 1 --2- 172.21.15.116:0/2567417369 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f03b80678c0 0x7f03b8069d80 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f03cc0676c0 tx=0x7f03c0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:13.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.071+0000 7f03b3fff640 1 -- 172.21.15.116:0/2567417369 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f03bc016110 con 0x7f03cc0a94a0 2024-04-25T00:22:13.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.263+0000 7f03d1e18640 1 -- 172.21.15.116:0/2567417369 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f03cc149100 con 0x7f03cc0a94a0 2024-04-25T00:22:13.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.267+0000 7f03b3fff640 1 -- 172.21.15.116:0/2567417369 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 175 v175) ==== 95+0+5667 (secure 0 0 0) 0x7f03bc0f5e20 con 0x7f03cc0a94a0 2024-04-25T00:22:13.271 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:22:13.271 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":175,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:22:11.919886+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7989},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7989":{"gid":7989,"name":"c","rank":0,"incarnation":173,"state":"up:rejoin","state_seq":9,"addr":"172.21.15.116:6835/221971311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":221971311},{"type":"v1","addr":"172.21.15.116:6835","nonce":221971311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:20:13.935378+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7722":{"gid":7722,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7722,"qdb_cluster":[7722]},"id":2}]} 2024-04-25T00:22:13.273 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 175 2024-04-25T00:22:13.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.267+0000 7f03d1e18640 1 -- 172.21.15.116:0/2567417369 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f03b80678c0 msgr2=0x7f03b8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:13.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.267+0000 7f03d1e18640 1 --2- 172.21.15.116:0/2567417369 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f03b80678c0 0x7f03b8069d80 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f03cc0676c0 tx=0x7f03c0008040 comp rx=0 tx=0).stop 2024-04-25T00:22:13.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.267+0000 7f03d1e18640 1 -- 172.21.15.116:0/2567417369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03cc0a94a0 msgr2=0x7f03cc140070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:13.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.267+0000 7f03d1e18640 1 --2- 172.21.15.116:0/2567417369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03cc0a94a0 0x7f03cc140070 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f03bc01ab10 tx=0x7f03bc01aca0 comp rx=0 tx=0).stop 2024-04-25T00:22:13.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.267+0000 7f03d1e18640 1 -- 172.21.15.116:0/2567417369 shutdown_connections 2024-04-25T00:22:13.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.267+0000 7f03d1e18640 1 --2- 172.21.15.116:0/2567417369 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f03b80678c0 0x7f03b8069d80 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:13.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.267+0000 7f03d1e18640 1 --2- 172.21.15.116:0/2567417369 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f03cc14c720 0x7f03cc1ef7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:13.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.267+0000 7f03d1e18640 1 --2- 172.21.15.116:0/2567417369 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f03cc0c2f50 0x7f03cc0c5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:13.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.267+0000 7f03d1e18640 1 --2- 172.21.15.116:0/2567417369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03cc0a94a0 0x7f03cc140070 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:13.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.267+0000 7f03d1e18640 1 -- 172.21.15.116:0/2567417369 >> 172.21.15.116:0/2567417369 conn(0x7f03cc0c3b00 msgr2=0x7f03cc0725c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:13.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.267+0000 7f03d1e18640 1 -- 172.21.15.116:0/2567417369 shutdown_connections 2024-04-25T00:22:13.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:13.267+0000 7f03d1e18640 1 -- 172.21.15.116:0/2567417369 wait complete. 2024-04-25T00:22:13.286 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 175, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:04.678737+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 175, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_7989": { "addr": "172.21.15.116:6835/221971311", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6834", "nonce": 221971311, "type": "v2" }, { "addr": "172.21.15.116:6835", "nonce": 221971311, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7989, "incarnation": 173, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:rejoin", "state_seq": 9 } }, "last_failure": 0, "last_failure_osd_epoch": 61, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T00:22:11.919886+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7989 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:18.825887+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 161, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_7722": { "addr": "172.21.15.116:6833/663910035", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6832", "nonce": 663910035, "type": "v2" }, { "addr": "172.21.15.116:6833", "nonce": 663910035, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7722, "incarnation": 158, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 11 } }, "last_failure": 0, "last_failure_osd_epoch": 58, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-04-25T00:20:13.935378+0000", "qdb_cluster": [ 7722 ], "qdb_leader": 7722, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7722 } } } ], "standbys": [ { "addr": "172.21.15.174:6835/4112145782", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6834", "nonce": 4112145782, "type": "v2" }, { "addr": "172.21.15.174:6835", "nonce": 4112145782, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 112, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5582, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-25T00:22:15.287 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:22:15.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.463+0000 7f2b22075640 1 Processor -- start 2024-04-25T00:22:15.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.463+0000 7f2b22075640 1 -- start start 2024-04-25T00:22:15.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.463+0000 7f2b22075640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2b1c153f40 0x7f2b1c06e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:15.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.463+0000 7f2b22075640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b1c071a30 con 0x7f2b1c0c2ea0 2024-04-25T00:22:15.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.463+0000 7f2b22075640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b1c071ca0 con 0x7f2b1c0c2a10 2024-04-25T00:22:15.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.463+0000 7f2b22075640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f2b1c0725d0 con 0x7f2b1c153f40 2024-04-25T00:22:15.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.463+0000 7f2b1bfff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f2b1c0c2ea0 0x7f2b1c15a610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:45638/0 (socket says 172.21.15.116:45638) 2024-04-25T00:22:15.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.463+0000 7f2b1bfff640 1 -- 172.21.15.116:0/1548729290 learned_addr learned my addr 172.21.15.116:0/1548729290 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:22:15.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.463+0000 7f2b1b7fe640 1 --2- 172.21.15.116:0/1548729290 >> v2:172.21.15.116:3300/0 conn(0x7f2b1c153f40 0x7f2b1c06e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:15.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.463+0000 7f2b1affd640 1 -- 172.21.15.116:0/1548729290 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 115546226 0 0) 0x7f2b1c071a30 con 0x7f2b1c0c2ea0 2024-04-25T00:22:15.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.463+0000 7f2b1affd640 1 -- 172.21.15.116:0/1548729290 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b08003530 con 0x7f2b1c0c2ea0 2024-04-25T00:22:15.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.463+0000 7f2b1affd640 1 -- 172.21.15.116:0/1548729290 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4135093435 0 0) 0x7f2b08003530 con 0x7f2b1c0c2ea0 2024-04-25T00:22:15.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.463+0000 7f2b1affd640 1 -- 172.21.15.116:0/1548729290 >> v2:172.21.15.116:3300/0 conn(0x7f2b1c153f40 msgr2=0x7f2b1c06e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:15.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.463+0000 7f2b1affd640 1 --2- 172.21.15.116:0/1548729290 >> v2:172.21.15.116:3300/0 conn(0x7f2b1c153f40 0x7f2b1c06e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:15.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.463+0000 7f2b1affd640 1 -- 172.21.15.116:0/1548729290 >> v1:172.21.15.174:6789/0 conn(0x7f2b1c0c2a10 legacy=0x7f2b1c159f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:15.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.463+0000 7f2b1affd640 1 -- 172.21.15.116:0/1548729290 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b1c071f80 con 0x7f2b1c0c2ea0 2024-04-25T00:22:15.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.463+0000 7f2b1affd640 1 -- 172.21.15.116:0/1548729290 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f2b14001680 con 0x7f2b1c0c2ea0 2024-04-25T00:22:15.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.463+0000 7f2b1affd640 1 -- 172.21.15.116:0/1548729290 >> v1:172.21.15.116:6789/0 conn(0x7f2b1c0c2ea0 legacy=0x7f2b1c15a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:15.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.463+0000 7f2b1affd640 1 --2- 172.21.15.116:0/1548729290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b080039c0 0x7f2b08003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:15.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.463+0000 7f2b1affd640 1 --2- 172.21.15.116:0/1548729290 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b08004810 0x7f2b08004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:15.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.463+0000 7f2b1affd640 1 --2- 172.21.15.116:0/1548729290 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2b08005220 0x7f2b0800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:15.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.463+0000 7f2b1affd640 1 -- 172.21.15.116:0/1548729290 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2b1c0725d0 con 0x7f2b080039c0 2024-04-25T00:22:15.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.463+0000 7f2b1affd640 1 -- 172.21.15.116:0/1548729290 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2b0800c170 con 0x7f2b08004810 2024-04-25T00:22:15.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.463+0000 7f2b1affd640 1 -- 172.21.15.116:0/1548729290 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f2b0800c370 con 0x7f2b08005220 2024-04-25T00:22:15.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.467+0000 7f2b1b7fe640 1 --2- 172.21.15.116:0/1548729290 >> v2:172.21.15.116:3300/0 conn(0x7f2b1c153f40 0x7f2b1c06e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:22:15.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.467+0000 7f2b1bfff640 1 --2- 172.21.15.116:0/1548729290 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2b08005220 0x7f2b0800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:15.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.467+0000 7f2b13fff640 1 --2- 172.21.15.116:0/1548729290 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b08004810 0x7f2b08004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:15.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.467+0000 7f2b1b7fe640 1 --2- 172.21.15.116:0/1548729290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b080039c0 0x7f2b08003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:15.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.467+0000 7f2b1b7fe640 1 -- 172.21.15.116:0/1548729290 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2b08005220 msgr2=0x7f2b0800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:15.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.467+0000 7f2b1b7fe640 1 --2- 172.21.15.116:0/1548729290 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2b08005220 0x7f2b0800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:15.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.467+0000 7f2b1b7fe640 1 -- 172.21.15.116:0/1548729290 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b08004810 msgr2=0x7f2b08004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:15.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.467+0000 7f2b1b7fe640 1 --2- 172.21.15.116:0/1548729290 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b08004810 0x7f2b08004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:15.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.467+0000 7f2b1b7fe640 1 -- 172.21.15.116:0/1548729290 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b08003530 con 0x7f2b080039c0 2024-04-25T00:22:15.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.471+0000 7f2b1b7fe640 1 --2- 172.21.15.116:0/1548729290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b080039c0 0x7f2b08003da0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c00e350 tx=0x7f2b0c01ce80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a26608b527f5d717 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:15.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.471+0000 7f2b1bfff640 1 --2- 172.21.15.116:0/1548729290 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2b08005220 0x7f2b0800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:22:15.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.471+0000 7f2b1bfff640 1 -- 172.21.15.116:0/1548729290 reap_dead start 2024-04-25T00:22:15.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.471+0000 7f2b1affd640 1 -- 172.21.15.116:0/1548729290 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b0c009dc0 con 0x7f2b080039c0 2024-04-25T00:22:15.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.471+0000 7f2b1affd640 1 -- 172.21.15.116:0/1548729290 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b0c0a5c70 con 0x7f2b080039c0 2024-04-25T00:22:15.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.471+0000 7f2b22075640 1 -- 172.21.15.116:0/1548729290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b080039c0 msgr2=0x7f2b08003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:15.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.471+0000 7f2b22075640 1 --2- 172.21.15.116:0/1548729290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b080039c0 0x7f2b08003da0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c00e350 tx=0x7f2b0c01ce80 comp rx=0 tx=0).stop 2024-04-25T00:22:15.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.471+0000 7f2b22075640 1 -- 172.21.15.116:0/1548729290 shutdown_connections 2024-04-25T00:22:15.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.471+0000 7f2b22075640 1 --2- 172.21.15.116:0/1548729290 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b080039c0 0x7f2b08003da0 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:15.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.471+0000 7f2b22075640 1 -- 172.21.15.116:0/1548729290 >> 172.21.15.116:0/1548729290 conn(0x7f2b1c0c3b00 msgr2=0x7f2b1c0c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:15.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.471+0000 7f2b13fff640 1 --2- 172.21.15.116:0/1548729290 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b08004810 0x7f2b08004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:22:15.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.471+0000 7f2b22075640 1 -- 172.21.15.116:0/1548729290 shutdown_connections 2024-04-25T00:22:15.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.471+0000 7f2b22075640 1 -- 172.21.15.116:0/1548729290 wait complete. 2024-04-25T00:22:15.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.471+0000 7f2b22075640 1 Processor -- start 2024-04-25T00:22:15.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.471+0000 7f2b22075640 1 -- start start 2024-04-25T00:22:15.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.471+0000 7f2b22075640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b1c14ced0 0x7f2b1c08b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:15.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.471+0000 7f2b22075640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2b1c15de30 0x7f2b1c15b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:15.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.471+0000 7f2b22075640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b1c1502f0 0x7f2b1c0c5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:15.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.471+0000 7f2b22075640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2b1c0c38d0 con 0x7f2b1c1502f0 2024-04-25T00:22:15.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.475+0000 7f2b22075640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2b1c06eee0 con 0x7f2b1c14ced0 2024-04-25T00:22:15.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.475+0000 7f2b22075640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f2b1c06e790 con 0x7f2b1c15de30 2024-04-25T00:22:15.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.475+0000 7f2b1b7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b1c14ced0 0x7f2b1c08b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:15.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.475+0000 7f2b1b7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b1c14ced0 0x7f2b1c08b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:51856/0 (socket says 172.21.15.116:51856) 2024-04-25T00:22:15.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.475+0000 7f2b1b7fe640 1 -- 172.21.15.116:0/875592935 learned_addr learned my addr 172.21.15.116:0/875592935 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:22:15.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.475+0000 7f2b1bfff640 1 --2- 172.21.15.116:0/875592935 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b1c1502f0 0x7f2b1c0c5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:15.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.475+0000 7f2b1b7fe640 1 -- 172.21.15.116:0/875592935 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2b1c15de30 msgr2=0x7f2b1c15b080 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T00:22:15.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.475+0000 7f2b1b7fe640 1 --2- 172.21.15.116:0/875592935 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2b1c15de30 0x7f2b1c15b080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:15.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.475+0000 7f2b1b7fe640 1 -- 172.21.15.116:0/875592935 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b1c1502f0 msgr2=0x7f2b1c0c5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:15.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.475+0000 7f2b1b7fe640 1 --2- 172.21.15.116:0/875592935 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b1c1502f0 0x7f2b1c0c5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:15.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.475+0000 7f2b1b7fe640 1 -- 172.21.15.116:0/875592935 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b1c14d2b0 con 0x7f2b1c14ced0 2024-04-25T00:22:15.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.475+0000 7f2b1bfff640 1 --2- 172.21.15.116:0/875592935 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b1c1502f0 0x7f2b1c0c5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:22:15.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.475+0000 7f2b1b7fe640 1 --2- 172.21.15.116:0/875592935 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b1c14ced0 0x7f2b1c08b510 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c0af040 tx=0x7f2b0c009750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:15.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.475+0000 7f2b18ff9640 1 -- 172.21.15.116:0/875592935 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b0c0bb070 con 0x7f2b1c14ced0 2024-04-25T00:22:15.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.475+0000 7f2b18ff9640 1 -- 172.21.15.116:0/875592935 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b0c0b7660 con 0x7f2b1c14ced0 2024-04-25T00:22:15.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.475+0000 7f2b22075640 1 -- 172.21.15.116:0/875592935 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b1c14eea0 con 0x7f2b1c14ced0 2024-04-25T00:22:15.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.475+0000 7f2b18ff9640 1 -- 172.21.15.116:0/875592935 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b0c0b7a10 con 0x7f2b1c14ced0 2024-04-25T00:22:15.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.475+0000 7f2b22075640 1 -- 172.21.15.116:0/875592935 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b1c14faf0 con 0x7f2b1c14ced0 2024-04-25T00:22:15.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.475+0000 7f2b22075640 1 -- 172.21.15.116:0/875592935 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b1c068350 con 0x7f2b1c14ced0 2024-04-25T00:22:15.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.479+0000 7f2b18ff9640 1 -- 172.21.15.116:0/875592935 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2b0c0be070 con 0x7f2b1c14ced0 2024-04-25T00:22:15.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.479+0000 7f2b18ff9640 1 --2- 172.21.15.116:0/875592935 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2afc0ab9d0 0x7f2afc0ade90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:15.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.479+0000 7f2b18ff9640 1 -- 172.21.15.116:0/875592935 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 9504+0+0 (secure 0 0 0) 0x7f2b0c1218e0 con 0x7f2b1c14ced0 2024-04-25T00:22:15.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.483+0000 7f2b13fff640 1 --2- 172.21.15.116:0/875592935 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2afc0ab9d0 0x7f2afc0ade90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:15.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.483+0000 7f2b13fff640 1 --2- 172.21.15.116:0/875592935 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2afc0ab9d0 0x7f2afc0ade90 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f2b1c152d60 tx=0x7f2b04009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:15.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.483+0000 7f2b18ff9640 1 -- 172.21.15.116:0/875592935 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f2b0c016b90 con 0x7f2b1c14ced0 2024-04-25T00:22:15.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.855+0000 7f2b22075640 1 -- 172.21.15.116:0/875592935 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2b1c0c16e0 con 0x7f2b1c14ced0 2024-04-25T00:22:15.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.855+0000 7f2b18ff9640 1 -- 172.21.15.116:0/875592935 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) ==== 95+0+5675 (secure 0 0 0) 0x7f2b0c0f1670 con 0x7f2b1c14ced0 2024-04-25T00:22:15.855 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:22:15.855 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":176,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:22:13.925793+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7989},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7989":{"gid":7989,"name":"c","rank":0,"incarnation":173,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6835/221971311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":221971311},{"type":"v1","addr":"172.21.15.116:6835","nonce":221971311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7989,"qdb_cluster":[7989]},"id":1},{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:20:13.935378+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7722":{"gid":7722,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7722,"qdb_cluster":[7722]},"id":2}]} 2024-04-25T00:22:15.856 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 176 2024-04-25T00:22:15.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.859+0000 7f2b22075640 1 -- 172.21.15.116:0/875592935 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2afc0ab9d0 msgr2=0x7f2afc0ade90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:15.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.859+0000 7f2b22075640 1 --2- 172.21.15.116:0/875592935 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2afc0ab9d0 0x7f2afc0ade90 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f2b1c152d60 tx=0x7f2b04009040 comp rx=0 tx=0).stop 2024-04-25T00:22:15.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.859+0000 7f2b22075640 1 -- 172.21.15.116:0/875592935 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b1c14ced0 msgr2=0x7f2b1c08b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:15.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.859+0000 7f2b22075640 1 --2- 172.21.15.116:0/875592935 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b1c14ced0 0x7f2b1c08b510 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c0af040 tx=0x7f2b0c009750 comp rx=0 tx=0).stop 2024-04-25T00:22:15.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.859+0000 7f2b22075640 1 -- 172.21.15.116:0/875592935 shutdown_connections 2024-04-25T00:22:15.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.859+0000 7f2b22075640 1 --2- 172.21.15.116:0/875592935 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2afc0ab9d0 0x7f2afc0ade90 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:15.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.859+0000 7f2b22075640 1 --2- 172.21.15.116:0/875592935 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2b1c1502f0 0x7f2b1c0c5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:15.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.859+0000 7f2b22075640 1 --2- 172.21.15.116:0/875592935 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2b1c15de30 0x7f2b1c15b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:15.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.859+0000 7f2b22075640 1 --2- 172.21.15.116:0/875592935 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2b1c14ced0 0x7f2b1c08b510 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:15.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.859+0000 7f2b22075640 1 -- 172.21.15.116:0/875592935 >> 172.21.15.116:0/875592935 conn(0x7f2b1c0c3b00 msgr2=0x7f2b1c159ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:15.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.859+0000 7f2b22075640 1 -- 172.21.15.116:0/875592935 shutdown_connections 2024-04-25T00:22:15.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:15.859+0000 7f2b22075640 1 -- 172.21.15.116:0/875592935 wait complete. 2024-04-25T00:22:15.884 INFO:tasks.mds_thrash.fs.[a]:mds.c has gained rank=0, replacing gid=6125 2024-04-25T00:22:15.885 INFO:tasks.mds_thrash.fs.[a]:waiting for 8 secs before reviving mds.d 2024-04-25T00:22:17.854 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:22:17.922 INFO:tasks.ceph.mds.c.smithi116.stderr:2024-04-25T00:22:17.923+0000 7f9c9831f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T00:22:18.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.015+0000 7f3f67eed640 1 Processor -- start 2024-04-25T00:22:18.016 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.015+0000 7f3f67eed640 1 -- start start 2024-04-25T00:22:18.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.019+0000 7f3f67eed640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f6006f550 0x7f3f6006ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:18.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.019+0000 7f3f65c89640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f3f60072be0 0x7f3f601530b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:45654/0 (socket says 172.21.15.116:45654) 2024-04-25T00:22:18.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.019+0000 7f3f65c89640 1 -- 172.21.15.116:0/482359942 learned_addr learned my addr 172.21.15.116:0/482359942 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:22:18.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.019+0000 7f3f67eed640 1 --2- 172.21.15.116:0/482359942 >> v2:172.21.15.116:3300/0 conn(0x7f3f60152760 0x7f3f60072fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:18.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.019+0000 7f3f67eed640 1 -- 172.21.15.116:0/482359942 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f601537c0 con 0x7f3f60072be0 2024-04-25T00:22:18.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.019+0000 7f3f65488640 1 --2- 172.21.15.116:0/482359942 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f6006f550 0x7f3f6006ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:18.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.019+0000 7f3f67eed640 1 -- 172.21.15.116:0/482359942 --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f3f60152bf0 con 0x7f3f60152760 2024-04-25T00:22:18.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.019+0000 7f3f67eed640 1 -- 172.21.15.116:0/482359942 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f3f6006df50 con 0x7f3f6006f550 2024-04-25T00:22:18.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.019+0000 7f3f6648a640 1 --2- 172.21.15.116:0/482359942 >> v2:172.21.15.116:3300/0 conn(0x7f3f60152760 0x7f3f60072fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:18.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.019+0000 7f3f64c87640 1 -- 172.21.15.116:0/482359942 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2747917454 0 0) 0x7f3f601537c0 con 0x7f3f60072be0 2024-04-25T00:22:18.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.023+0000 7f3f64c87640 1 -- 172.21.15.116:0/482359942 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f4c003530 con 0x7f3f60072be0 2024-04-25T00:22:18.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.023+0000 7f3f65488640 1 -- 172.21.15.116:0/482359942 >> v2:172.21.15.116:3300/0 conn(0x7f3f60152760 msgr2=0x7f3f60072fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:18.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.023+0000 7f3f65488640 1 --2- 172.21.15.116:0/482359942 >> v2:172.21.15.116:3300/0 conn(0x7f3f60152760 0x7f3f60072fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:18.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.023+0000 7f3f65488640 1 -- 172.21.15.116:0/482359942 >> v1:172.21.15.116:6789/0 conn(0x7f3f60072be0 legacy=0x7f3f601530b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:18.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.023+0000 7f3f65488640 1 -- 172.21.15.116:0/482359942 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f6006fa30 con 0x7f3f6006f550 2024-04-25T00:22:18.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.023+0000 7f3f65488640 1 --2- 172.21.15.116:0/482359942 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f6006f550 0x7f3f6006ec80 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3f500097c0 tx=0x7f3f5001c380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1584ac331c783777 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:18.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.023+0000 7f3f64c87640 1 -- 172.21.15.116:0/482359942 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f500b0070 con 0x7f3f6006f550 2024-04-25T00:22:18.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.023+0000 7f3f64c87640 1 -- 172.21.15.116:0/482359942 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f500ab600 con 0x7f3f6006f550 2024-04-25T00:22:18.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.023+0000 7f3f64c87640 1 -- 172.21.15.116:0/482359942 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f500ab9b0 con 0x7f3f6006f550 2024-04-25T00:22:18.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.027+0000 7f3f67eed640 1 -- 172.21.15.116:0/482359942 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f6006f550 msgr2=0x7f3f6006ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:18.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.027+0000 7f3f67eed640 1 --2- 172.21.15.116:0/482359942 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f6006f550 0x7f3f6006ec80 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3f500097c0 tx=0x7f3f5001c380 comp rx=0 tx=0).stop 2024-04-25T00:22:18.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.027+0000 7f3f67eed640 1 -- 172.21.15.116:0/482359942 shutdown_connections 2024-04-25T00:22:18.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.027+0000 7f3f67eed640 1 --2- 172.21.15.116:0/482359942 >> v2:172.21.15.116:3300/0 conn(0x7f3f60152760 0x7f3f60072fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:18.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.027+0000 7f3f67eed640 1 --2- 172.21.15.116:0/482359942 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f6006f550 0x7f3f6006ec80 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:18.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.027+0000 7f3f67eed640 1 -- 172.21.15.116:0/482359942 >> 172.21.15.116:0/482359942 conn(0x7f3f600a94a0 msgr2=0x7f3f600a98b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:18.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.027+0000 7f3f67eed640 1 -- 172.21.15.116:0/482359942 shutdown_connections 2024-04-25T00:22:18.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.027+0000 7f3f67eed640 1 -- 172.21.15.116:0/482359942 wait complete. 2024-04-25T00:22:18.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.027+0000 7f3f67eed640 1 Processor -- start 2024-04-25T00:22:18.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.027+0000 7f3f67eed640 1 -- start start 2024-04-25T00:22:18.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.027+0000 7f3f67eed640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f6006f550 0x7f3f6014af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:18.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.027+0000 7f3f67eed640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f60072be0 0x7f3f601ed840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:18.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.027+0000 7f3f67eed640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f60152760 0x7f3f601440e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:18.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.027+0000 7f3f67eed640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3f6014bf60 con 0x7f3f60072be0 2024-04-25T00:22:18.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.027+0000 7f3f67eed640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3f60071410 con 0x7f3f6006f550 2024-04-25T00:22:18.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.027+0000 7f3f67eed640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f3f6014c0e0 con 0x7f3f60152760 2024-04-25T00:22:18.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.027+0000 7f3f65c89640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f6006f550 0x7f3f6014af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:18.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.027+0000 7f3f65c89640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f6006f550 0x7f3f6014af50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:51872/0 (socket says 172.21.15.116:51872) 2024-04-25T00:22:18.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.027+0000 7f3f65c89640 1 -- 172.21.15.116:0/3590806771 learned_addr learned my addr 172.21.15.116:0/3590806771 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:22:18.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.027+0000 7f3f6648a640 1 --2- 172.21.15.116:0/3590806771 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f60152760 0x7f3f601440e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:18.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.027+0000 7f3f65488640 1 --2- 172.21.15.116:0/3590806771 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f60072be0 0x7f3f601ed840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:18.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.031+0000 7f3f65c89640 1 -- 172.21.15.116:0/3590806771 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f60152760 msgr2=0x7f3f601440e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:18.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.031+0000 7f3f65c89640 1 --2- 172.21.15.116:0/3590806771 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f60152760 0x7f3f601440e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:18.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.031+0000 7f3f65c89640 1 -- 172.21.15.116:0/3590806771 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f60072be0 msgr2=0x7f3f601ed840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:18.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.031+0000 7f3f65c89640 1 --2- 172.21.15.116:0/3590806771 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f60072be0 0x7f3f601ed840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:18.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.031+0000 7f3f65c89640 1 -- 172.21.15.116:0/3590806771 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f6014a0a0 con 0x7f3f6006f550 2024-04-25T00:22:18.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.031+0000 7f3f65c89640 1 --2- 172.21.15.116:0/3590806771 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f6006f550 0x7f3f6014af50 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c00c480 tx=0x7f3f5c00c940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:18.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.031+0000 7f3f567fc640 1 -- 172.21.15.116:0/3590806771 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f5c017070 con 0x7f3f6006f550 2024-04-25T00:22:18.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.031+0000 7f3f567fc640 1 -- 172.21.15.116:0/3590806771 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f5c005cf0 con 0x7f3f6006f550 2024-04-25T00:22:18.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.031+0000 7f3f567fc640 1 -- 172.21.15.116:0/3590806771 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f5c007270 con 0x7f3f6006f550 2024-04-25T00:22:18.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.031+0000 7f3f67eed640 1 -- 172.21.15.116:0/3590806771 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f6014a380 con 0x7f3f6006f550 2024-04-25T00:22:18.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.031+0000 7f3f67eed640 1 -- 172.21.15.116:0/3590806771 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f60144a80 con 0x7f3f6006f550 2024-04-25T00:22:18.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.031+0000 7f3f567fc640 1 -- 172.21.15.116:0/3590806771 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3f5c01a070 con 0x7f3f6006f550 2024-04-25T00:22:18.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.031+0000 7f3f67eed640 1 -- 172.21.15.116:0/3590806771 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f480052a0 con 0x7f3f6006f550 2024-04-25T00:22:18.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.031+0000 7f3f567fc640 1 --2- 172.21.15.116:0/3590806771 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3f440678c0 0x7f3f44069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:18.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.031+0000 7f3f567fc640 1 -- 172.21.15.116:0/3590806771 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 9504+0+0 (secure 0 0 0) 0x7f3f5c085db0 con 0x7f3f6006f550 2024-04-25T00:22:18.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.031+0000 7f3f65488640 1 --2- 172.21.15.116:0/3590806771 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3f440678c0 0x7f3f44069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:18.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.035+0000 7f3f65488640 1 --2- 172.21.15.116:0/3590806771 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3f440678c0 0x7f3f44069d80 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f3f500ad000 tx=0x7f3f50008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:18.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.039+0000 7f3f567fc640 1 -- 172.21.15.116:0/3590806771 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f3f5c055410 con 0x7f3f6006f550 2024-04-25T00:22:18.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.439+0000 7f3f67eed640 1 -- 172.21.15.116:0/3590806771 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3f48005590 con 0x7f3f6006f550 2024-04-25T00:22:18.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.439+0000 7f3f567fc640 1 -- 172.21.15.116:0/3590806771 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) ==== 95+0+5675 (secure 0 0 0) 0x7f3f5c059710 con 0x7f3f6006f550 2024-04-25T00:22:18.439 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:22:18.439 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":176,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:22:13.925793+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7989},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7989":{"gid":7989,"name":"c","rank":0,"incarnation":173,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6835/221971311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":221971311},{"type":"v1","addr":"172.21.15.116:6835","nonce":221971311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7989,"qdb_cluster":[7989]},"id":1},{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:20:13.935378+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7722":{"gid":7722,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7722,"qdb_cluster":[7722]},"id":2}]} 2024-04-25T00:22:18.439 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 176 2024-04-25T00:22:18.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.443+0000 7f3f67eed640 1 -- 172.21.15.116:0/3590806771 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3f440678c0 msgr2=0x7f3f44069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:18.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.443+0000 7f3f67eed640 1 --2- 172.21.15.116:0/3590806771 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3f440678c0 0x7f3f44069d80 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f3f500ad000 tx=0x7f3f50008040 comp rx=0 tx=0).stop 2024-04-25T00:22:18.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.443+0000 7f3f67eed640 1 -- 172.21.15.116:0/3590806771 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f6006f550 msgr2=0x7f3f6014af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:18.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.443+0000 7f3f67eed640 1 --2- 172.21.15.116:0/3590806771 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f6006f550 0x7f3f6014af50 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c00c480 tx=0x7f3f5c00c940 comp rx=0 tx=0).stop 2024-04-25T00:22:18.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.443+0000 7f3f67eed640 1 -- 172.21.15.116:0/3590806771 shutdown_connections 2024-04-25T00:22:18.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.443+0000 7f3f67eed640 1 --2- 172.21.15.116:0/3590806771 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3f440678c0 0x7f3f44069d80 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:18.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.443+0000 7f3f67eed640 1 --2- 172.21.15.116:0/3590806771 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f60152760 0x7f3f601440e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:18.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.443+0000 7f3f67eed640 1 --2- 172.21.15.116:0/3590806771 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f60072be0 0x7f3f601ed840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:18.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.443+0000 7f3f67eed640 1 --2- 172.21.15.116:0/3590806771 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f6006f550 0x7f3f6014af50 secure :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c00c480 tx=0x7f3f5c00c940 comp rx=0 tx=0).stop 2024-04-25T00:22:18.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.443+0000 7f3f67eed640 1 -- 172.21.15.116:0/3590806771 >> 172.21.15.116:0/3590806771 conn(0x7f3f600a94a0 msgr2=0x7f3f600a9880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:18.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.443+0000 7f3f67eed640 1 -- 172.21.15.116:0/3590806771 shutdown_connections 2024-04-25T00:22:18.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.443+0000 7f3f67eed640 1 -- 172.21.15.116:0/3590806771 wait complete. 2024-04-25T00:22:18.465 INFO:tasks.mds_thrash.fs.[b]:kill mds.a (rank=0) 2024-04-25T00:22:18.466 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-25T00:22:18.466 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:22:18.481 INFO:tasks.ceph.mds.a:Stopped 2024-04-25T00:22:18.482 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:22:18.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.695+0000 7f83d70ff640 1 Processor -- start 2024-04-25T00:22:18.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.695+0000 7f83d70ff640 1 -- start start 2024-04-25T00:22:18.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.695+0000 7f83d70ff640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f83d0155b20 0x7f83d0152c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:18.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.695+0000 7f83d70ff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83d00aacd0 0x7f83d0153180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:18.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.695+0000 7f83cffff640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f83d0155b20 0x7f83d0152c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:18.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.695+0000 7f83cffff640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f83d0155b20 0x7f83d0152c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:51884/0 (socket says 172.21.15.116:51884) 2024-04-25T00:22:18.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.695+0000 7f83cffff640 1 -- 172.21.15.116:0/1030542690 learned_addr learned my addr 172.21.15.116:0/1030542690 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:22:18.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.695+0000 7f83d70ff640 1 -- 172.21.15.116:0/1030542690 --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83d00a9e60 con 0x7f83d014c700 2024-04-25T00:22:18.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.695+0000 7f83d70ff640 1 -- 172.21.15.116:0/1030542690 --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f83d01527b0 con 0x7f83d0155b20 2024-04-25T00:22:18.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.695+0000 7f83d70ff640 1 -- 172.21.15.116:0/1030542690 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f83d00c3ef0 con 0x7f83d00aacd0 2024-04-25T00:22:18.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.695+0000 7f83d569c640 1 --2- 172.21.15.116:0/1030542690 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83d00aacd0 0x7f83d0153180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:18.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.699+0000 7f83d569c640 1 -- 172.21.15.116:0/1030542690 >> v2:172.21.15.174:3300/0 conn(0x7f83d0155b20 msgr2=0x7f83d0152c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:18.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.699+0000 7f83d569c640 1 --2- 172.21.15.116:0/1030542690 >> v2:172.21.15.174:3300/0 conn(0x7f83d0155b20 0x7f83d0152c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:18.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.699+0000 7f83d569c640 1 -- 172.21.15.116:0/1030542690 >> v1:172.21.15.174:6789/0 conn(0x7f83d014c700 legacy=0x7f83d00c5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:18.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.699+0000 7f83d569c640 1 -- 172.21.15.116:0/1030542690 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83d00c41d0 con 0x7f83d00aacd0 2024-04-25T00:22:18.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.699+0000 7f83d569c640 1 --2- 172.21.15.116:0/1030542690 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83d00aacd0 0x7f83d0153180 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f83c8009d90 tx=0x7f83c801c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7dd36fc211509f09 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:18.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.699+0000 7f83cf7fe640 1 -- 172.21.15.116:0/1030542690 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c80b2070 con 0x7f83d00aacd0 2024-04-25T00:22:18.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.699+0000 7f83cf7fe640 1 -- 172.21.15.116:0/1030542690 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83c80a5710 con 0x7f83d00aacd0 2024-04-25T00:22:18.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.699+0000 7f83cf7fe640 1 -- 172.21.15.116:0/1030542690 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c80a5ae0 con 0x7f83d00aacd0 2024-04-25T00:22:18.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.703+0000 7f83d70ff640 1 -- 172.21.15.116:0/1030542690 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83d00aacd0 msgr2=0x7f83d0153180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:18.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.703+0000 7f83d70ff640 1 --2- 172.21.15.116:0/1030542690 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83d00aacd0 0x7f83d0153180 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f83c8009d90 tx=0x7f83c801c850 comp rx=0 tx=0).stop 2024-04-25T00:22:18.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.703+0000 7f83d70ff640 1 -- 172.21.15.116:0/1030542690 shutdown_connections 2024-04-25T00:22:18.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.703+0000 7f83d70ff640 1 --2- 172.21.15.116:0/1030542690 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83d00aacd0 0x7f83d0153180 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:18.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.703+0000 7f83d70ff640 1 --2- 172.21.15.116:0/1030542690 >> v2:172.21.15.174:3300/0 conn(0x7f83d0155b20 0x7f83d0152c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:18.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.703+0000 7f83d70ff640 1 -- 172.21.15.116:0/1030542690 >> 172.21.15.116:0/1030542690 conn(0x7f83d006c720 msgr2=0x7f83d006cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:18.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.703+0000 7f83d70ff640 1 -- 172.21.15.116:0/1030542690 shutdown_connections 2024-04-25T00:22:18.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.703+0000 7f83d70ff640 1 -- 172.21.15.116:0/1030542690 wait complete. 2024-04-25T00:22:18.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.703+0000 7f83d70ff640 1 Processor -- start 2024-04-25T00:22:18.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.703+0000 7f83d70ff640 1 -- start start 2024-04-25T00:22:18.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.703+0000 7f83d70ff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83d00aacd0 0x7f83d00ac980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:18.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.703+0000 7f83d70ff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83d014c700 0x7f83d01f8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:18.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.703+0000 7f83d70ff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f83d0155b20 0x7f83d014e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:18.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.703+0000 7f83d70ff640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f83d015fa60 con 0x7f83d00aacd0 2024-04-25T00:22:18.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.703+0000 7f83d70ff640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f83d015f8e0 con 0x7f83d0155b20 2024-04-25T00:22:18.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.703+0000 7f83d4e9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83d00aacd0 0x7f83d00ac980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:18.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.703+0000 7f83d4e9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83d00aacd0 0x7f83d00ac980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57664/0 (socket says 172.21.15.116:57664) 2024-04-25T00:22:18.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.703+0000 7f83d4e9b640 1 -- 172.21.15.116:0/3773414443 learned_addr learned my addr 172.21.15.116:0/3773414443 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:22:18.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.703+0000 7f83d70ff640 1 -- 172.21.15.116:0/3773414443 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f83d015fbe0 con 0x7f83d014c700 2024-04-25T00:22:18.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.703+0000 7f83d569c640 1 --2- 172.21.15.116:0/3773414443 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f83d0155b20 0x7f83d014e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:18.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.707+0000 7f83d569c640 1 -- 172.21.15.116:0/3773414443 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83d014c700 msgr2=0x7f83d01f8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:18.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.707+0000 7f83d569c640 1 --2- 172.21.15.116:0/3773414443 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83d014c700 0x7f83d01f8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:18.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.707+0000 7f83d569c640 1 -- 172.21.15.116:0/3773414443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83d00aacd0 msgr2=0x7f83d00ac980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:18.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.707+0000 7f83d569c640 1 --2- 172.21.15.116:0/3773414443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83d00aacd0 0x7f83d00ac980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:18.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.707+0000 7f83d569c640 1 -- 172.21.15.116:0/3773414443 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83d00c5fe0 con 0x7f83d0155b20 2024-04-25T00:22:18.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.707+0000 7f83d569c640 1 --2- 172.21.15.116:0/3773414443 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f83d0155b20 0x7f83d014e5b0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f83c80ac000 tx=0x7f83c80195f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:18.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.707+0000 7f83cd7fa640 1 -- 172.21.15.116:0/3773414443 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c80bd030 con 0x7f83d0155b20 2024-04-25T00:22:18.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.707+0000 7f83cd7fa640 1 -- 172.21.15.116:0/3773414443 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83c80a5510 con 0x7f83d0155b20 2024-04-25T00:22:18.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.707+0000 7f83cd7fa640 1 -- 172.21.15.116:0/3773414443 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c80b2040 con 0x7f83d0155b20 2024-04-25T00:22:18.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.707+0000 7f83d70ff640 1 -- 172.21.15.116:0/3773414443 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83d00c62c0 con 0x7f83d0155b20 2024-04-25T00:22:18.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.707+0000 7f83d70ff640 1 -- 172.21.15.116:0/3773414443 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83d00c6880 con 0x7f83d0155b20 2024-04-25T00:22:18.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.707+0000 7f83cd7fa640 1 -- 172.21.15.116:0/3773414443 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f83c80c0070 con 0x7f83d0155b20 2024-04-25T00:22:18.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.707+0000 7f83baffd640 1 -- 172.21.15.116:0/3773414443 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83a80052a0 con 0x7f83d0155b20 2024-04-25T00:22:18.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.711+0000 7f83cd7fa640 1 --2- 172.21.15.116:0/3773414443 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f83b4067940 0x7f83b4069e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:18.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.711+0000 7f83cd7fa640 1 -- 172.21.15.116:0/3773414443 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 9504+0+0 (secure 0 0 0) 0x7f83c81216c0 con 0x7f83d0155b20 2024-04-25T00:22:18.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.715+0000 7f83d4e9b640 1 --2- 172.21.15.116:0/3773414443 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f83b4067940 0x7f83b4069e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:18.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.715+0000 7f83d4e9b640 1 --2- 172.21.15.116:0/3773414443 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f83b4067940 0x7f83b4069e00 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f83bc007fd0 tx=0x7f83bc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:18.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:18.715+0000 7f83cd7fa640 1 -- 172.21.15.116:0/3773414443 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f83c8016560 con 0x7f83d0155b20 2024-04-25T00:22:19.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:19.123+0000 7f83baffd640 1 -- 172.21.15.116:0/3773414443 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f83a8005860 con 0x7f83d0155b20 2024-04-25T00:22:19.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:19.123+0000 7f83cd7fa640 1 -- 172.21.15.116:0/3773414443 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) ==== 95+0+5675 (secure 0 0 0) 0x7f83c80f1510 con 0x7f83d0155b20 2024-04-25T00:22:19.124 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:22:19.124 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":176,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:22:13.925793+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7989},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7989":{"gid":7989,"name":"c","rank":0,"incarnation":173,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6835/221971311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":221971311},{"type":"v1","addr":"172.21.15.116:6835","nonce":221971311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7989,"qdb_cluster":[7989]},"id":1},{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:20:13.935378+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7722":{"gid":7722,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7722,"qdb_cluster":[7722]},"id":2}]} 2024-04-25T00:22:19.124 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 176 2024-04-25T00:22:19.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:19.131+0000 7f83baffd640 1 -- 172.21.15.116:0/3773414443 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f83b4067940 msgr2=0x7f83b4069e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:19.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:19.131+0000 7f83baffd640 1 --2- 172.21.15.116:0/3773414443 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f83b4067940 0x7f83b4069e00 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f83bc007fd0 tx=0x7f83bc009040 comp rx=0 tx=0).stop 2024-04-25T00:22:19.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:19.131+0000 7f83baffd640 1 -- 172.21.15.116:0/3773414443 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f83d0155b20 msgr2=0x7f83d014e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:19.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:19.131+0000 7f83baffd640 1 --2- 172.21.15.116:0/3773414443 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f83d0155b20 0x7f83d014e5b0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f83c80ac000 tx=0x7f83c80195f0 comp rx=0 tx=0).stop 2024-04-25T00:22:19.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:19.131+0000 7f83baffd640 1 -- 172.21.15.116:0/3773414443 shutdown_connections 2024-04-25T00:22:19.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:19.131+0000 7f83baffd640 1 --2- 172.21.15.116:0/3773414443 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f83b4067940 0x7f83b4069e00 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:19.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:19.131+0000 7f83baffd640 1 --2- 172.21.15.116:0/3773414443 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f83d0155b20 0x7f83d014e5b0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:19.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:19.131+0000 7f83baffd640 1 --2- 172.21.15.116:0/3773414443 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83d014c700 0x7f83d01f8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:19.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:19.131+0000 7f83baffd640 1 --2- 172.21.15.116:0/3773414443 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83d00aacd0 0x7f83d00ac980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:19.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:19.131+0000 7f83baffd640 1 -- 172.21.15.116:0/3773414443 >> 172.21.15.116:0/3773414443 conn(0x7f83d006c720 msgr2=0x7f83d006cb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:19.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:19.135+0000 7f83baffd640 1 -- 172.21.15.116:0/3773414443 shutdown_connections 2024-04-25T00:22:19.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:19.135+0000 7f83baffd640 1 -- 172.21.15.116:0/3773414443 wait complete. 2024-04-25T00:22:19.163 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:22:21.165 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:22:21.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.363+0000 7f926e038640 1 Processor -- start 2024-04-25T00:22:21.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.363+0000 7f926e038640 1 -- start start 2024-04-25T00:22:21.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.363+0000 7f926e038640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f92600d5650 0x7f9260048370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:21.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.363+0000 7f926e038640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f92600db670 0x7f92600d8540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:21.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.363+0000 7f926e038640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92600d8080 con 0x7f92600488b0 2024-04-25T00:22:21.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.363+0000 7f926e038640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f926004bb10 con 0x7f92600db670 2024-04-25T00:22:21.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.363+0000 7f926e038640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f92600dbb70 con 0x7f92600d5650 2024-04-25T00:22:21.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.363+0000 7f926d837640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f92600488b0 0x7f92600d8a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:41820/0 (socket says 172.21.15.116:41820) 2024-04-25T00:22:21.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.363+0000 7f926d837640 1 -- 172.21.15.116:0/2535540175 learned_addr learned my addr 172.21.15.116:0/2535540175 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:22:21.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.363+0000 7f926d036640 1 --2- 172.21.15.116:0/2535540175 >> v2:172.21.15.174:3300/0 conn(0x7f92600d5650 0x7f9260048370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:21.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.363+0000 7f926c835640 1 --2- 172.21.15.116:0/2535540175 >> v2:172.21.15.116:3300/0 conn(0x7f92600db670 0x7f92600d8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:21.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.367+0000 7f925ffff640 1 -- 172.21.15.116:0/2535540175 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2880719644 0 0) 0x7f9268066f50 con 0x7f92600488b0 2024-04-25T00:22:21.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.367+0000 7f926c835640 1 -- 172.21.15.116:0/2535540175 >> v2:172.21.15.174:3300/0 conn(0x7f92600d5650 msgr2=0x7f9260048370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:21.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.367+0000 7f926c835640 1 --2- 172.21.15.116:0/2535540175 >> v2:172.21.15.174:3300/0 conn(0x7f92600d5650 0x7f9260048370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:21.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.367+0000 7f926c835640 1 -- 172.21.15.116:0/2535540175 >> v1:172.21.15.174:6789/0 conn(0x7f92600488b0 legacy=0x7f92600d8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:21.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.367+0000 7f926c835640 1 -- 172.21.15.116:0/2535540175 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92600dbe50 con 0x7f92600db670 2024-04-25T00:22:21.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.371+0000 7f926c835640 1 --2- 172.21.15.116:0/2535540175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92600db670 0x7f92600d8540 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f926000f740 tx=0x7f925801c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fa98b9a897424f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:21.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.371+0000 7f925ffff640 1 -- 172.21.15.116:0/2535540175 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92580b1070 con 0x7f92600db670 2024-04-25T00:22:21.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.371+0000 7f925ffff640 1 -- 172.21.15.116:0/2535540175 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92580ac620 con 0x7f92600db670 2024-04-25T00:22:21.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.371+0000 7f925ffff640 1 -- 172.21.15.116:0/2535540175 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92580ac9f0 con 0x7f92600db670 2024-04-25T00:22:21.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.371+0000 7f926e038640 1 -- 172.21.15.116:0/2535540175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92600db670 msgr2=0x7f92600d8540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:21.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.371+0000 7f926e038640 1 --2- 172.21.15.116:0/2535540175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92600db670 0x7f92600d8540 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f926000f740 tx=0x7f925801c3c0 comp rx=0 tx=0).stop 2024-04-25T00:22:21.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.371+0000 7f926e038640 1 -- 172.21.15.116:0/2535540175 shutdown_connections 2024-04-25T00:22:21.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.371+0000 7f926e038640 1 --2- 172.21.15.116:0/2535540175 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92600db670 0x7f92600d8540 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:21.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.371+0000 7f926e038640 1 --2- 172.21.15.116:0/2535540175 >> v2:172.21.15.174:3300/0 conn(0x7f92600d5650 0x7f9260048370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:21.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.371+0000 7f926e038640 1 -- 172.21.15.116:0/2535540175 >> 172.21.15.116:0/2535540175 conn(0x7f926002f5c0 msgr2=0x7f926002f9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:21.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.371+0000 7f926e038640 1 -- 172.21.15.116:0/2535540175 shutdown_connections 2024-04-25T00:22:21.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.371+0000 7f926e038640 1 -- 172.21.15.116:0/2535540175 wait complete. 2024-04-25T00:22:21.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.375+0000 7f926e038640 1 Processor -- start 2024-04-25T00:22:21.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.375+0000 7f926e038640 1 -- start start 2024-04-25T00:22:21.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.375+0000 7f926e038640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f92600488b0 0x7f92600c6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:21.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.375+0000 7f926e038640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92600d5650 0x7f92600c7400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:21.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.375+0000 7f926e038640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92600db670 0x7f926017aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:21.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.375+0000 7f926e038640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f926004ade0 con 0x7f92600db670 2024-04-25T00:22:21.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.375+0000 7f926e038640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f926004ac60 con 0x7f92600d5650 2024-04-25T00:22:21.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.375+0000 7f926e038640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f92600e5220 con 0x7f92600488b0 2024-04-25T00:22:21.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.375+0000 7f926d837640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92600db670 0x7f926017aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:21.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.375+0000 7f926d837640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92600db670 0x7f926017aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57694/0 (socket says 172.21.15.116:57694) 2024-04-25T00:22:21.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.375+0000 7f926d837640 1 -- 172.21.15.116:0/3345065255 learned_addr learned my addr 172.21.15.116:0/3345065255 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:22:21.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.375+0000 7f926d837640 1 -- 172.21.15.116:0/3345065255 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f92600488b0 msgr2=0x7f92600c6ec0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T00:22:21.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.375+0000 7f926d837640 1 --2- 172.21.15.116:0/3345065255 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f92600488b0 0x7f92600c6ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:21.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.375+0000 7f926d837640 1 -- 172.21.15.116:0/3345065255 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92600d5650 msgr2=0x7f92600c7400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T00:22:21.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.375+0000 7f926d837640 1 --2- 172.21.15.116:0/3345065255 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92600d5650 0x7f92600c7400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:21.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.375+0000 7f926d837640 1 -- 172.21.15.116:0/3345065255 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f926002ea70 con 0x7f92600db670 2024-04-25T00:22:21.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.375+0000 7f926d837640 1 --2- 172.21.15.116:0/3345065255 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92600db670 0x7f926017aac0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f9268069810 tx=0x7f9268068e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:21.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.375+0000 7f925dffb640 1 -- 172.21.15.116:0/3345065255 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92680705f0 con 0x7f92600db670 2024-04-25T00:22:21.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.375+0000 7f925dffb640 1 -- 172.21.15.116:0/3345065255 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92680707b0 con 0x7f92600db670 2024-04-25T00:22:21.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.375+0000 7f925dffb640 1 -- 172.21.15.116:0/3345065255 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92680709a0 con 0x7f92600db670 2024-04-25T00:22:21.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.379+0000 7f926e038640 1 -- 172.21.15.116:0/3345065255 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f926002e4b0 con 0x7f92600db670 2024-04-25T00:22:21.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.379+0000 7f926e038640 1 -- 172.21.15.116:0/3345065255 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92600cbb20 con 0x7f92600db670 2024-04-25T00:22:21.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.379+0000 7f926e038640 1 -- 172.21.15.116:0/3345065255 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92600057f0 con 0x7f92600db670 2024-04-25T00:22:21.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.379+0000 7f925dffb640 1 -- 172.21.15.116:0/3345065255 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f9268071040 con 0x7f92600db670 2024-04-25T00:22:21.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.379+0000 7f925dffb640 1 --2- 172.21.15.116:0/3345065255 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f924c0678c0 0x7f924c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:21.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.379+0000 7f925dffb640 1 -- 172.21.15.116:0/3345065255 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 9504+0+0 (secure 0 0 0) 0x7f9268102760 con 0x7f92600db670 2024-04-25T00:22:21.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.383+0000 7f926d036640 1 --2- 172.21.15.116:0/3345065255 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f924c0678c0 0x7f924c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:21.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.383+0000 7f926d036640 1 --2- 172.21.15.116:0/3345065255 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f924c0678c0 0x7f924c069d80 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f9264009e70 tx=0x7f9264009830 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:21.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.387+0000 7f925dffb640 1 -- 172.21.15.116:0/3345065255 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f926002ea70 con 0x7f92600db670 2024-04-25T00:22:21.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.791+0000 7f926e038640 1 -- 172.21.15.116:0/3345065255 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f92600db450 con 0x7f92600db670 2024-04-25T00:22:21.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.791+0000 7f925dffb640 1 -- 172.21.15.116:0/3345065255 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) ==== 95+0+5675 (secure 0 0 0) 0x7f92600d8080 con 0x7f92600db670 2024-04-25T00:22:21.790 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:22:21.790 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":176,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:22:13.925793+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7989},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7989":{"gid":7989,"name":"c","rank":0,"incarnation":173,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6835/221971311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":221971311},{"type":"v1","addr":"172.21.15.116:6835","nonce":221971311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7989,"qdb_cluster":[7989]},"id":1},{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:20:13.935378+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7722":{"gid":7722,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7722,"qdb_cluster":[7722]},"id":2}]} 2024-04-25T00:22:21.791 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 176 2024-04-25T00:22:21.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.795+0000 7f926e038640 1 -- 172.21.15.116:0/3345065255 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f924c0678c0 msgr2=0x7f924c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:21.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.795+0000 7f926e038640 1 --2- 172.21.15.116:0/3345065255 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f924c0678c0 0x7f924c069d80 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f9264009e70 tx=0x7f9264009830 comp rx=0 tx=0).stop 2024-04-25T00:22:21.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.795+0000 7f926e038640 1 -- 172.21.15.116:0/3345065255 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92600db670 msgr2=0x7f926017aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:21.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.795+0000 7f926e038640 1 --2- 172.21.15.116:0/3345065255 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92600db670 0x7f926017aac0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f9268069810 tx=0x7f9268068e30 comp rx=0 tx=0).stop 2024-04-25T00:22:21.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.795+0000 7f926e038640 1 -- 172.21.15.116:0/3345065255 shutdown_connections 2024-04-25T00:22:21.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.795+0000 7f926e038640 1 --2- 172.21.15.116:0/3345065255 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f924c0678c0 0x7f924c069d80 secure :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f9264009e70 tx=0x7f9264009830 comp rx=0 tx=0).stop 2024-04-25T00:22:21.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.795+0000 7f926e038640 1 --2- 172.21.15.116:0/3345065255 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92600db670 0x7f926017aac0 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:21.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.795+0000 7f926e038640 1 --2- 172.21.15.116:0/3345065255 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92600d5650 0x7f92600c7400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:21.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.795+0000 7f926e038640 1 --2- 172.21.15.116:0/3345065255 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f92600488b0 0x7f92600c6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:21.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.795+0000 7f926e038640 1 -- 172.21.15.116:0/3345065255 >> 172.21.15.116:0/3345065255 conn(0x7f926002f5c0 msgr2=0x7f92600e1960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:21.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.795+0000 7f926e038640 1 -- 172.21.15.116:0/3345065255 shutdown_connections 2024-04-25T00:22:21.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:21.795+0000 7f926e038640 1 -- 172.21.15.116:0/3345065255 wait complete. 2024-04-25T00:22:21.821 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:22:23.822 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:22:23.885 INFO:tasks.mds_thrash.fs.[a]:reviving mds.d 2024-04-25T00:22:23.885 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-25T00:22:23.886 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-25T00:22:23.888 INFO:tasks.ceph.mds.d:Started 2024-04-25T00:22:23.888 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:22:23.962 INFO:tasks.ceph.mds.d.smithi174.stdout:starting mds.d at 2024-04-25T00:22:23.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:23.991+0000 7ff1a6969640 1 Processor -- start 2024-04-25T00:22:23.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:23.991+0000 7ff1a6969640 1 -- start start 2024-04-25T00:22:23.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:23.991+0000 7ff1a6969640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff1a014bd10 0x7ff1a0071590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:23.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:23.991+0000 7ff1a6969640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff1a00c3330 0x7ff1a0152670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:23.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:23.991+0000 7ff1a6969640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff1a0155a10 0x7ff1a006f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:23.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:23.991+0000 7ff1a6969640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7ff1a014c0f0 con 0x7ff1a0155a10 2024-04-25T00:22:23.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:23.991+0000 7ff1a6969640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7ff1a0152bb0 con 0x7ff1a014bd10 2024-04-25T00:22:23.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:23.991+0000 7ff1a6969640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff1a00c3860 con 0x7ff1a00c3330 2024-04-25T00:22:23.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:23.995+0000 7ff19f7fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff1a00c3330 0x7ff1a0152670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:23.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:23.995+0000 7ff19f7fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff1a00c3330 0x7ff1a0152670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:34006/0 (socket says 172.21.15.116:34006) 2024-04-25T00:22:23.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:23.995+0000 7ff19f7fe640 1 -- 172.21.15.116:0/1469489154 learned_addr learned my addr 172.21.15.116:0/1469489154 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:22:23.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:23.995+0000 7ff19ffff640 1 --2- 172.21.15.116:0/1469489154 >> v2:172.21.15.174:3300/0 conn(0x7ff1a014bd10 0x7ff1a0071590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:23.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:23.995+0000 7ff1a4f06640 1 --2- 172.21.15.116:0/1469489154 >> v2:172.21.15.116:3300/0 conn(0x7ff1a0155a10 0x7ff1a006f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:23.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:23.995+0000 7ff19f7fe640 1 -- 172.21.15.116:0/1469489154 >> v2:172.21.15.174:3300/0 conn(0x7ff1a014bd10 msgr2=0x7ff1a0071590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:23.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:23.995+0000 7ff19f7fe640 1 --2- 172.21.15.116:0/1469489154 >> v2:172.21.15.174:3300/0 conn(0x7ff1a014bd10 0x7ff1a0071590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:23.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:23.995+0000 7ff19f7fe640 1 -- 172.21.15.116:0/1469489154 >> v2:172.21.15.116:3300/0 conn(0x7ff1a0155a10 msgr2=0x7ff1a006f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:23.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:23.995+0000 7ff19f7fe640 1 --2- 172.21.15.116:0/1469489154 >> v2:172.21.15.116:3300/0 conn(0x7ff1a0155a10 0x7ff1a006f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:23.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:23.995+0000 7ff19f7fe640 1 -- 172.21.15.116:0/1469489154 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1a00c3b40 con 0x7ff1a00c3330 2024-04-25T00:22:23.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:23.995+0000 7ff19f7fe640 1 --2- 172.21.15.116:0/1469489154 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff1a00c3330 0x7ff1a0152670 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff1a00c4160 tx=0x7ff19001e7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11a38f05cb77bd7b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:23.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:23.999+0000 7ff19effd640 1 -- 172.21.15.116:0/1469489154 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1900b4070 con 0x7ff1a00c3330 2024-04-25T00:22:23.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:23.999+0000 7ff19effd640 1 -- 172.21.15.116:0/1469489154 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1900a7750 con 0x7ff1a00c3330 2024-04-25T00:22:23.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:23.999+0000 7ff19effd640 1 -- 172.21.15.116:0/1469489154 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1900a7b00 con 0x7ff1a00c3330 2024-04-25T00:22:23.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:23.999+0000 7ff1a6969640 1 -- 172.21.15.116:0/1469489154 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff1a00c3330 msgr2=0x7ff1a0152670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:23.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:23.999+0000 7ff1a6969640 1 --2- 172.21.15.116:0/1469489154 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff1a00c3330 0x7ff1a0152670 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7ff1a00c4160 tx=0x7ff19001e7d0 comp rx=0 tx=0).stop 2024-04-25T00:22:24.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.003+0000 7ff1a6969640 1 -- 172.21.15.116:0/1469489154 shutdown_connections 2024-04-25T00:22:24.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.003+0000 7ff1a6969640 1 --2- 172.21.15.116:0/1469489154 >> v2:172.21.15.116:3300/0 conn(0x7ff1a0155a10 0x7ff1a006f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:24.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.003+0000 7ff1a6969640 1 --2- 172.21.15.116:0/1469489154 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff1a00c3330 0x7ff1a0152670 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:24.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.003+0000 7ff1a6969640 1 --2- 172.21.15.116:0/1469489154 >> v2:172.21.15.174:3300/0 conn(0x7ff1a014bd10 0x7ff1a0071590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:24.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.003+0000 7ff1a6969640 1 -- 172.21.15.116:0/1469489154 >> 172.21.15.116:0/1469489154 conn(0x7ff1a006b210 msgr2=0x7ff1a006b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:24.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.003+0000 7ff1a6969640 1 -- 172.21.15.116:0/1469489154 shutdown_connections 2024-04-25T00:22:24.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.003+0000 7ff1a6969640 1 -- 172.21.15.116:0/1469489154 wait complete. 2024-04-25T00:22:24.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.003+0000 7ff1a6969640 1 Processor -- start 2024-04-25T00:22:24.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.003+0000 7ff1a6969640 1 -- start start 2024-04-25T00:22:24.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.007+0000 7ff1a6969640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff1a00c3330 0x7ff1a014a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:24.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.007+0000 7ff1a6969640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1a014bd10 0x7ff1a00be9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:24.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.007+0000 7ff1a6969640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1a0155a10 0x7ff1a01f4140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:24.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.007+0000 7ff1a6969640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff1a00c2280 con 0x7ff1a014bd10 2024-04-25T00:22:24.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.007+0000 7ff1a6969640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff1a00c2100 con 0x7ff1a0155a10 2024-04-25T00:22:24.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.007+0000 7ff1a6969640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff1a00c2400 con 0x7ff1a00c3330 2024-04-25T00:22:24.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.007+0000 7ff1a4f06640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1a0155a10 0x7ff1a01f4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:24.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.007+0000 7ff1a4f06640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1a0155a10 0x7ff1a01f4140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:51910/0 (socket says 172.21.15.116:51910) 2024-04-25T00:22:24.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.007+0000 7ff1a4f06640 1 -- 172.21.15.116:0/1741252206 learned_addr learned my addr 172.21.15.116:0/1741252206 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:22:24.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.007+0000 7ff19f7fe640 1 --2- 172.21.15.116:0/1741252206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1a014bd10 0x7ff1a00be9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:24.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.007+0000 7ff1a4f06640 1 -- 172.21.15.116:0/1741252206 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff1a00c3330 msgr2=0x7ff1a014a8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T00:22:24.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.007+0000 7ff1a4f06640 1 --2- 172.21.15.116:0/1741252206 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff1a00c3330 0x7ff1a014a8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:24.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.007+0000 7ff1a4f06640 1 -- 172.21.15.116:0/1741252206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1a014bd10 msgr2=0x7ff1a00be9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:24.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.007+0000 7ff1a4f06640 1 --2- 172.21.15.116:0/1741252206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1a014bd10 0x7ff1a00be9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:24.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.007+0000 7ff1a4f06640 1 -- 172.21.15.116:0/1741252206 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1a00a9040 con 0x7ff1a0155a10 2024-04-25T00:22:24.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.007+0000 7ff19f7fe640 1 --2- 172.21.15.116:0/1741252206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1a014bd10 0x7ff1a00be9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:22:24.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.007+0000 7ff1a4f06640 1 --2- 172.21.15.116:0/1741252206 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1a0155a10 0x7ff1a01f4140 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7ff198007e60 tx=0x7ff19800e4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:24.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.007+0000 7ff19cff9640 1 -- 172.21.15.116:0/1741252206 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff198014420 con 0x7ff1a0155a10 2024-04-25T00:22:24.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.007+0000 7ff19cff9640 1 -- 172.21.15.116:0/1741252206 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff198019070 con 0x7ff1a0155a10 2024-04-25T00:22:24.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.007+0000 7ff19cff9640 1 -- 172.21.15.116:0/1741252206 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff198014820 con 0x7ff1a0155a10 2024-04-25T00:22:24.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.007+0000 7ff1a6969640 1 -- 172.21.15.116:0/1741252206 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1a00bd510 con 0x7ff1a0155a10 2024-04-25T00:22:24.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.007+0000 7ff1a6969640 1 -- 172.21.15.116:0/1741252206 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1a00bdad0 con 0x7ff1a0155a10 2024-04-25T00:22:24.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.011+0000 7ff1967fc640 1 -- 172.21.15.116:0/1741252206 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1780052a0 con 0x7ff1a0155a10 2024-04-25T00:22:24.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.015+0000 7ff19cff9640 1 -- 172.21.15.116:0/1741252206 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff1980149c0 con 0x7ff1a0155a10 2024-04-25T00:22:24.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.015+0000 7ff19cff9640 1 --2- 172.21.15.116:0/1741252206 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff1800679c0 0x7ff180069e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:24.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.015+0000 7ff19cff9640 1 -- 172.21.15.116:0/1741252206 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 9504+0+0 (secure 0 0 0) 0x7ff198087310 con 0x7ff1a0155a10 2024-04-25T00:22:24.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.015+0000 7ff19ffff640 1 --2- 172.21.15.116:0/1741252206 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff1800679c0 0x7ff180069e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:24.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.015+0000 7ff19ffff640 1 --2- 172.21.15.116:0/1741252206 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff1800679c0 0x7ff180069e80 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7ff1a00ad3a0 tx=0x7ff188009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:24.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.019+0000 7ff19cff9640 1 -- 172.21.15.116:0/1741252206 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ff198057000 con 0x7ff1a0155a10 2024-04-25T00:22:24.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.155+0000 7fd02c5ed640 1 Processor -- start 2024-04-25T00:22:24.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.159+0000 7fd02c5ed640 1 -- start start 2024-04-25T00:22:24.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.159+0000 7fd02c5ed640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd02408bb30 0x7fd02406cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:24.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.159+0000 7fd02c5ed640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0240c37f0 con 0x7fd02406d160 2024-04-25T00:22:24.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.159+0000 7fd02c5ed640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd024070de0 con 0x7fd024167260 2024-04-25T00:22:24.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.159+0000 7fd02c5ed640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fd02416a410 con 0x7fd02408bb30 2024-04-25T00:22:24.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.159+0000 7fd02a389640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd02408bb30 0x7fd02406cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:24.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.159+0000 7fd02a389640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd02408bb30 0x7fd02406cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:34016/0 (socket says 172.21.15.116:34016) 2024-04-25T00:22:24.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.159+0000 7fd02a389640 1 -- 172.21.15.116:0/2615816253 learned_addr learned my addr 172.21.15.116:0/2615816253 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:22:24.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.159+0000 7fd02a389640 1 -- 172.21.15.116:0/2615816253 >> v1:172.21.15.174:6789/0 conn(0x7fd024167260 legacy=0x7fd0240c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:24.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.159+0000 7fd02a389640 1 -- 172.21.15.116:0/2615816253 >> v1:172.21.15.116:6789/0 conn(0x7fd02406d160 legacy=0x7fd024169d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:24.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.159+0000 7fd02a389640 1 -- 172.21.15.116:0/2615816253 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0240710c0 con 0x7fd02408bb30 2024-04-25T00:22:24.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.163+0000 7fd02a389640 1 --2- 172.21.15.116:0/2615816253 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd02408bb30 0x7fd02406cb50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd0200098a0 tx=0x7fd020024c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11ada450f7474633 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:24.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.163+0000 7fd029387640 1 -- 172.21.15.116:0/2615816253 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0200ba070 con 0x7fd02408bb30 2024-04-25T00:22:24.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.163+0000 7fd029387640 1 -- 172.21.15.116:0/2615816253 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0200ad9f0 con 0x7fd02408bb30 2024-04-25T00:22:24.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.163+0000 7fd029387640 1 -- 172.21.15.116:0/2615816253 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0200addc0 con 0x7fd02408bb30 2024-04-25T00:22:24.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.163+0000 7fd02c5ed640 1 -- 172.21.15.116:0/2615816253 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd02408bb30 msgr2=0x7fd02406cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:24.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.163+0000 7fd02c5ed640 1 --2- 172.21.15.116:0/2615816253 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd02408bb30 0x7fd02406cb50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd0200098a0 tx=0x7fd020024c50 comp rx=0 tx=0).stop 2024-04-25T00:22:24.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.163+0000 7fd02c5ed640 1 -- 172.21.15.116:0/2615816253 shutdown_connections 2024-04-25T00:22:24.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.163+0000 7fd02c5ed640 1 --2- 172.21.15.116:0/2615816253 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd02408bb30 0x7fd02406cb50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:24.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.163+0000 7fd02c5ed640 1 -- 172.21.15.116:0/2615816253 >> 172.21.15.116:0/2615816253 conn(0x7fd0240c3b00 msgr2=0x7fd0240c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:24.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.163+0000 7fd02c5ed640 1 -- 172.21.15.116:0/2615816253 shutdown_connections 2024-04-25T00:22:24.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.163+0000 7fd02c5ed640 1 -- 172.21.15.116:0/2615816253 wait complete. 2024-04-25T00:22:24.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.163+0000 7fd02c5ed640 1 Processor -- start 2024-04-25T00:22:24.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.163+0000 7fd02c5ed640 1 -- start start 2024-04-25T00:22:24.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.167+0000 7fd02c5ed640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd02406d160 0x7fd0240c0080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:24.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.167+0000 7fd02c5ed640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd02408bb30 0x7fd0240ac0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:24.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.167+0000 7fd02c5ed640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd024167260 0x7fd02408a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:24.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.167+0000 7fd02c5ed640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd0240c32a0 con 0x7fd02406d160 2024-04-25T00:22:24.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.167+0000 7fd02c5ed640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd0240c3120 con 0x7fd024167260 2024-04-25T00:22:24.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.167+0000 7fd02c5ed640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fd024163bb0 con 0x7fd02408bb30 2024-04-25T00:22:24.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.167+0000 7fd02a389640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd02406d160 0x7fd0240c0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:24.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.167+0000 7fd02a389640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd02406d160 0x7fd0240c0080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57710/0 (socket says 172.21.15.116:57710) 2024-04-25T00:22:24.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.167+0000 7fd02a389640 1 -- 172.21.15.116:0/825178303 learned_addr learned my addr 172.21.15.116:0/825178303 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:22:24.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.167+0000 7fd02ab8a640 1 --2- 172.21.15.116:0/825178303 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd024167260 0x7fd02408a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:24.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.167+0000 7fd029b88640 1 --2- 172.21.15.116:0/825178303 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd02408bb30 0x7fd0240ac0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:24.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.167+0000 7fd02a389640 1 -- 172.21.15.116:0/825178303 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd02408bb30 msgr2=0x7fd0240ac0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:24.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.167+0000 7fd02a389640 1 --2- 172.21.15.116:0/825178303 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd02408bb30 0x7fd0240ac0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:24.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.167+0000 7fd02a389640 1 -- 172.21.15.116:0/825178303 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd024167260 msgr2=0x7fd02408a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:24.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.167+0000 7fd02a389640 1 --2- 172.21.15.116:0/825178303 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd024167260 0x7fd02408a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:24.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.167+0000 7fd02a389640 1 -- 172.21.15.116:0/825178303 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0241fde30 con 0x7fd02406d160 2024-04-25T00:22:24.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.167+0000 7fd02a389640 1 --2- 172.21.15.116:0/825178303 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd02406d160 0x7fd0240c0080 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7fd0200097c0 tx=0x7fd0200219f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:24.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.167+0000 7fd01affd640 1 -- 172.21.15.116:0/825178303 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0200b5d50 con 0x7fd02406d160 2024-04-25T00:22:24.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.167+0000 7fd02c5ed640 1 -- 172.21.15.116:0/825178303 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0241fe0d0 con 0x7fd02406d160 2024-04-25T00:22:24.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.167+0000 7fd01affd640 1 -- 172.21.15.116:0/825178303 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0200cb070 con 0x7fd02406d160 2024-04-25T00:22:24.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.167+0000 7fd01affd640 1 -- 172.21.15.116:0/825178303 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd020014270 con 0x7fd02406d160 2024-04-25T00:22:24.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.171+0000 7fd02c5ed640 1 -- 172.21.15.116:0/825178303 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0241fe690 con 0x7fd02406d160 2024-04-25T00:22:24.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.171+0000 7fd02c5ed640 1 -- 172.21.15.116:0/825178303 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd024068350 con 0x7fd02406d160 2024-04-25T00:22:24.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.171+0000 7fd01affd640 1 -- 172.21.15.116:0/825178303 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd020014410 con 0x7fd02406d160 2024-04-25T00:22:24.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.171+0000 7fd01affd640 1 --2- 172.21.15.116:0/825178303 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd008067990 0x7fd008069e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:24.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.171+0000 7fd01affd640 1 -- 172.21.15.116:0/825178303 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 9504+0+0 (secure 0 0 0) 0x7fd020132be0 con 0x7fd02406d160 2024-04-25T00:22:24.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.171+0000 7fd029b88640 1 --2- 172.21.15.116:0/825178303 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd008067990 0x7fd008069e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:24.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.175+0000 7fd029b88640 1 --2- 172.21.15.116:0/825178303 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd008067990 0x7fd008069e50 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fd02406ccd0 tx=0x7fd014009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:24.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.179+0000 7fd01affd640 1 -- 172.21.15.116:0/825178303 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fd02001e960 con 0x7fd02406d160 2024-04-25T00:22:24.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.551+0000 7ff1967fc640 1 -- 172.21.15.116:0/1741252206 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff178005590 con 0x7ff1a0155a10 2024-04-25T00:22:24.551 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:22:24.551 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":176,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:22:13.925793+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7989},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7989":{"gid":7989,"name":"c","rank":0,"incarnation":173,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6835/221971311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":221971311},{"type":"v1","addr":"172.21.15.116:6835","nonce":221971311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7989,"qdb_cluster":[7989]},"id":1},{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:20:13.935378+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7722":{"gid":7722,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7722,"qdb_cluster":[7722]},"id":2}]} 2024-04-25T00:22:24.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.551+0000 7ff19cff9640 1 -- 172.21.15.116:0/1741252206 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) ==== 95+0+5675 (secure 0 0 0) 0x7ff198056900 con 0x7ff1a0155a10 2024-04-25T00:22:24.552 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 176 2024-04-25T00:22:24.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.555+0000 7ff1967fc640 1 -- 172.21.15.116:0/1741252206 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff1800679c0 msgr2=0x7ff180069e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:24.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.555+0000 7ff1967fc640 1 --2- 172.21.15.116:0/1741252206 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff1800679c0 0x7ff180069e80 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7ff1a00ad3a0 tx=0x7ff188009190 comp rx=0 tx=0).stop 2024-04-25T00:22:24.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.559+0000 7ff1967fc640 1 -- 172.21.15.116:0/1741252206 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1a0155a10 msgr2=0x7ff1a01f4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:24.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.559+0000 7ff1967fc640 1 --2- 172.21.15.116:0/1741252206 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1a0155a10 0x7ff1a01f4140 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7ff198007e60 tx=0x7ff19800e4b0 comp rx=0 tx=0).stop 2024-04-25T00:22:24.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.559+0000 7ff1967fc640 1 -- 172.21.15.116:0/1741252206 shutdown_connections 2024-04-25T00:22:24.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.559+0000 7ff1967fc640 1 --2- 172.21.15.116:0/1741252206 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff1800679c0 0x7ff180069e80 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:24.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.559+0000 7ff1967fc640 1 --2- 172.21.15.116:0/1741252206 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1a0155a10 0x7ff1a01f4140 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:24.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.559+0000 7ff1967fc640 1 --2- 172.21.15.116:0/1741252206 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1a014bd10 0x7ff1a00be9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:24.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.559+0000 7ff1967fc640 1 --2- 172.21.15.116:0/1741252206 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff1a00c3330 0x7ff1a014a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:24.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.559+0000 7ff1967fc640 1 -- 172.21.15.116:0/1741252206 >> 172.21.15.116:0/1741252206 conn(0x7ff1a006b210 msgr2=0x7ff1a00a74b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:24.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.559+0000 7ff1967fc640 1 -- 172.21.15.116:0/1741252206 shutdown_connections 2024-04-25T00:22:24.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.559+0000 7ff1967fc640 1 -- 172.21.15.116:0/1741252206 wait complete. 2024-04-25T00:22:24.593 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:22:24.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.655+0000 7fd02c5ed640 1 -- 172.21.15.116:0/825178303 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd0240c2a50 con 0x7fd02406d160 2024-04-25T00:22:24.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.655+0000 7fd01affd640 1 -- 172.21.15.116:0/825178303 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) ==== 95+0+5675 (secure 0 0 0) 0x7fd0200ad9c0 con 0x7fd02406d160 2024-04-25T00:22:24.656 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:22:24.656 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":176,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:22:13.925793+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7989},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7989":{"gid":7989,"name":"c","rank":0,"incarnation":173,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6835/221971311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":221971311},{"type":"v1","addr":"172.21.15.116:6835","nonce":221971311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7989,"qdb_cluster":[7989]},"id":1},{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:20:13.935378+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7722":{"gid":7722,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7722,"qdb_cluster":[7722]},"id":2}]} 2024-04-25T00:22:24.656 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 176 2024-04-25T00:22:24.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.659+0000 7fd02c5ed640 1 -- 172.21.15.116:0/825178303 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd008067990 msgr2=0x7fd008069e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:24.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.659+0000 7fd02c5ed640 1 --2- 172.21.15.116:0/825178303 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd008067990 0x7fd008069e50 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fd02406ccd0 tx=0x7fd014009040 comp rx=0 tx=0).stop 2024-04-25T00:22:24.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.659+0000 7fd02c5ed640 1 -- 172.21.15.116:0/825178303 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd02406d160 msgr2=0x7fd0240c0080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:24.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.659+0000 7fd02c5ed640 1 --2- 172.21.15.116:0/825178303 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd02406d160 0x7fd0240c0080 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7fd0200097c0 tx=0x7fd0200219f0 comp rx=0 tx=0).stop 2024-04-25T00:22:24.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.659+0000 7fd02c5ed640 1 -- 172.21.15.116:0/825178303 shutdown_connections 2024-04-25T00:22:24.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.659+0000 7fd02c5ed640 1 --2- 172.21.15.116:0/825178303 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd008067990 0x7fd008069e50 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:24.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.659+0000 7fd02c5ed640 1 --2- 172.21.15.116:0/825178303 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd024167260 0x7fd02408a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:24.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.659+0000 7fd02c5ed640 1 --2- 172.21.15.116:0/825178303 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd02408bb30 0x7fd0240ac0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:24.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.659+0000 7fd02c5ed640 1 --2- 172.21.15.116:0/825178303 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd02406d160 0x7fd0240c0080 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:24.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.659+0000 7fd02c5ed640 1 -- 172.21.15.116:0/825178303 >> 172.21.15.116:0/825178303 conn(0x7fd0240c3b00 msgr2=0x7fd024169b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:24.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.659+0000 7fd02c5ed640 1 -- 172.21.15.116:0/825178303 shutdown_connections 2024-04-25T00:22:24.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:24.659+0000 7fd02c5ed640 1 -- 172.21.15.116:0/825178303 wait complete. 2024-04-25T00:22:24.693 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2024-04-25T00:22:26.594 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:22:26.694 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:22:26.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.766+0000 7fb223244640 1 Processor -- start 2024-04-25T00:22:26.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.766+0000 7fb223244640 1 -- start start 2024-04-25T00:22:26.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.770+0000 7fb223244640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb2140dc150 0x7fb2140478c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:26.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.770+0000 7fb223244640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fb2140e21b0 0x7fb2140e2640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:26.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.770+0000 7fb223244640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2140472e0 con 0x7fb21404bef0 2024-04-25T00:22:26.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.770+0000 7fb223244640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fb214049e60 con 0x7fb2140dc150 2024-04-25T00:22:26.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.770+0000 7fb223244640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fb214047550 con 0x7fb2140e21b0 2024-04-25T00:22:26.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.770+0000 7fb222a43640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fb2140e21b0 0x7fb2140e2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:26.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.770+0000 7fb222a43640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fb2140e21b0 0x7fb2140e2640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:52814/0 (socket says 172.21.15.116:52814) 2024-04-25T00:22:26.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.770+0000 7fb222a43640 1 -- 172.21.15.116:0/2028602876 learned_addr learned my addr 172.21.15.116:0/2028602876 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:22:26.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.770+0000 7fb222242640 1 --2- 172.21.15.116:0/2028602876 >> v2:172.21.15.116:3300/0 conn(0x7fb2140dc150 0x7fb2140478c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:26.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.770+0000 7fb222a43640 1 -- 172.21.15.116:0/2028602876 >> v2:172.21.15.116:3300/0 conn(0x7fb2140dc150 msgr2=0x7fb2140478c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:26.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.770+0000 7fb222a43640 1 --2- 172.21.15.116:0/2028602876 >> v2:172.21.15.116:3300/0 conn(0x7fb2140dc150 0x7fb2140478c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:26.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.770+0000 7fb222a43640 1 -- 172.21.15.116:0/2028602876 >> v1:172.21.15.174:6789/0 conn(0x7fb21404bef0 legacy=0x7fb21403f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:26.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.770+0000 7fb222a43640 1 -- 172.21.15.116:0/2028602876 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2140dc530 con 0x7fb2140e21b0 2024-04-25T00:22:26.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.778+0000 7fb222a43640 1 --2- 172.21.15.116:0/2028602876 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb2140e21b0 0x7fb2140e2640 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fb21c04fa30 tx=0x7fb21c097ff0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=44a310b7c8b2088b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:26.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.782+0000 7fb221240640 1 -- 172.21.15.116:0/2028602876 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb21c1206d0 con 0x7fb2140e21b0 2024-04-25T00:22:26.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.782+0000 7fb221240640 1 -- 172.21.15.116:0/2028602876 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb21c120870 con 0x7fb2140e21b0 2024-04-25T00:22:26.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.782+0000 7fb221240640 1 -- 172.21.15.116:0/2028602876 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb21c120ce0 con 0x7fb2140e21b0 2024-04-25T00:22:26.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.786+0000 7fb223244640 1 -- 172.21.15.116:0/2028602876 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb2140e21b0 msgr2=0x7fb2140e2640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:26.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.786+0000 7fb223244640 1 --2- 172.21.15.116:0/2028602876 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb2140e21b0 0x7fb2140e2640 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fb21c04fa30 tx=0x7fb21c097ff0 comp rx=0 tx=0).stop 2024-04-25T00:22:26.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.790+0000 7fb223244640 1 -- 172.21.15.116:0/2028602876 shutdown_connections 2024-04-25T00:22:26.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.790+0000 7fb223244640 1 --2- 172.21.15.116:0/2028602876 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb2140e21b0 0x7fb2140e2640 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:26.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.790+0000 7fb223244640 1 --2- 172.21.15.116:0/2028602876 >> v2:172.21.15.116:3300/0 conn(0x7fb2140dc150 0x7fb2140478c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:26.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.790+0000 7fb223244640 1 -- 172.21.15.116:0/2028602876 >> 172.21.15.116:0/2028602876 conn(0x7fb214049340 msgr2=0x7fb214049750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:26.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.798+0000 7fb223244640 1 -- 172.21.15.116:0/2028602876 shutdown_connections 2024-04-25T00:22:26.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.802+0000 7fb223244640 1 -- 172.21.15.116:0/2028602876 wait complete. 2024-04-25T00:22:26.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.802+0000 7fb223244640 1 Processor -- start 2024-04-25T00:22:26.804 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.802+0000 7fb223244640 1 -- start start 2024-04-25T00:22:26.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.806+0000 7fb223244640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb21404bef0 0x7fb21417b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:26.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.806+0000 7fb222242640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb21404bef0 0x7fb21417b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:26.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.806+0000 7fb222242640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb21404bef0 0x7fb21417b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40404/0 (socket says 172.21.15.116:40404) 2024-04-25T00:22:26.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.806+0000 7fb222242640 1 -- 172.21.15.116:0/985555563 learned_addr learned my addr 172.21.15.116:0/985555563 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:22:26.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.806+0000 7fb223244640 1 --2- 172.21.15.116:0/985555563 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb2140dc150 0x7fb2140d1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:26.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.806+0000 7fb223244640 1 --2- 172.21.15.116:0/985555563 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb2140e21b0 0x7fb2140e1450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:26.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.806+0000 7fb223244640 1 -- 172.21.15.116:0/985555563 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb2140e2d00 con 0x7fb21404bef0 2024-04-25T00:22:26.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.806+0000 7fb223244640 1 -- 172.21.15.116:0/985555563 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb2140e2b80 con 0x7fb2140dc150 2024-04-25T00:22:26.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.806+0000 7fb223244640 1 -- 172.21.15.116:0/985555563 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fb2140eb9d0 con 0x7fb2140e21b0 2024-04-25T00:22:26.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.806+0000 7fb221a41640 1 --2- 172.21.15.116:0/985555563 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb2140dc150 0x7fb2140d1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:26.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.806+0000 7fb222a43640 1 --2- 172.21.15.116:0/985555563 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb2140e21b0 0x7fb2140e1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:26.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.806+0000 7fb222242640 1 -- 172.21.15.116:0/985555563 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb2140e21b0 msgr2=0x7fb2140e1450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:26.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.806+0000 7fb222242640 1 --2- 172.21.15.116:0/985555563 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb2140e21b0 0x7fb2140e1450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:26.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.806+0000 7fb222242640 1 -- 172.21.15.116:0/985555563 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb2140dc150 msgr2=0x7fb2140d1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:26.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.806+0000 7fb222242640 1 --2- 172.21.15.116:0/985555563 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb2140dc150 0x7fb2140d1bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:26.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.806+0000 7fb222242640 1 -- 172.21.15.116:0/985555563 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb214040a40 con 0x7fb21404bef0 2024-04-25T00:22:26.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.806+0000 7fb222242640 1 --2- 172.21.15.116:0/985555563 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb21404bef0 0x7fb21417b480 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7fb20c00e820 tx=0x7fb20c00ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:26.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.806+0000 7fb212ffd640 1 -- 172.21.15.116:0/985555563 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb20c0059a0 con 0x7fb21404bef0 2024-04-25T00:22:26.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.806+0000 7fb212ffd640 1 -- 172.21.15.116:0/985555563 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb20c005b40 con 0x7fb21404bef0 2024-04-25T00:22:26.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.806+0000 7fb212ffd640 1 -- 172.21.15.116:0/985555563 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb20c00c8c0 con 0x7fb21404bef0 2024-04-25T00:22:26.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.806+0000 7fb223244640 1 -- 172.21.15.116:0/985555563 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb21402d830 con 0x7fb21404bef0 2024-04-25T00:22:26.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.810+0000 7fb212ffd640 1 -- 172.21.15.116:0/985555563 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb20c00ca60 con 0x7fb21404bef0 2024-04-25T00:22:26.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.814+0000 7fb212ffd640 1 --2- 172.21.15.116:0/985555563 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb2000678f0 0x7fb200069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:26.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.814+0000 7fb223244640 1 -- 172.21.15.116:0/985555563 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb21402db10 con 0x7fb21404bef0 2024-04-25T00:22:26.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.814+0000 7fb212ffd640 1 -- 172.21.15.116:0/985555563 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 9504+0+0 (secure 0 0 0) 0x7fb20c053910 con 0x7fb21404bef0 2024-04-25T00:22:26.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.814+0000 7fb223244640 1 -- 172.21.15.116:0/985555563 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2140057c0 con 0x7fb21404bef0 2024-04-25T00:22:26.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.814+0000 7fb221a41640 1 --2- 172.21.15.116:0/985555563 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb2000678f0 0x7fb200069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:26.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.814+0000 7fb221a41640 1 --2- 172.21.15.116:0/985555563 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb2000678f0 0x7fb200069db0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fb21417bb30 tx=0x7fb218009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:26.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.830+0000 7fb212ffd640 1 -- 172.21.15.116:0/985555563 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fb20c017020 con 0x7fb21404bef0 2024-04-25T00:22:26.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.910+0000 7f1b0359e640 1 Processor -- start 2024-04-25T00:22:26.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.910+0000 7f1b0359e640 1 -- start start 2024-04-25T00:22:26.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.914+0000 7f1b0359e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1b0415fd70 0x7f1b04070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:26.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.914+0000 7f1b0359e640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f1b04165ac0 0x7f1b04071530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:26.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.914+0000 7f1b0359e640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b041629a0 con 0x7f1b040c2ab0 2024-04-25T00:22:26.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.914+0000 7f1b0359e640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f1b04162c80 con 0x7f1b04165ac0 2024-04-25T00:22:26.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.914+0000 7f1b0359e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f1b04162f60 con 0x7f1b0415fd70 2024-04-25T00:22:26.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.914+0000 7f1b0259c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f1b040c2ab0 0x7f1b0415f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54112/0 (socket says 172.21.15.116:54112) 2024-04-25T00:22:26.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.914+0000 7f1b0259c640 1 -- 172.21.15.116:0/2129127281 learned_addr learned my addr 172.21.15.116:0/2129127281 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:22:26.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.914+0000 7f1b01d9b640 1 --2- 172.21.15.116:0/2129127281 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1b0415fd70 0x7f1b04070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:26.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.914+0000 7f1b0159a640 1 -- 172.21.15.116:0/2129127281 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3210205576 0 0) 0x7f1b041629a0 con 0x7f1b040c2ab0 2024-04-25T00:22:26.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.914+0000 7f1b02d9d640 1 --2- 172.21.15.116:0/2129127281 >> v2:172.21.15.174:3300/0 conn(0x7f1b04165ac0 0x7f1b04071530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:26.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.930+0000 7f1b0159a640 1 -- 172.21.15.116:0/2129127281 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1aec003510 con 0x7f1b040c2ab0 2024-04-25T00:22:26.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.930+0000 7f1b0159a640 1 -- 172.21.15.116:0/2129127281 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3427497172 0 0) 0x7f1aec003510 con 0x7f1b040c2ab0 2024-04-25T00:22:26.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.930+0000 7f1b0159a640 1 -- 172.21.15.116:0/2129127281 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1b0415fd70 msgr2=0x7f1b04070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:26.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.930+0000 7f1b0159a640 1 --2- 172.21.15.116:0/2129127281 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1b0415fd70 0x7f1b04070ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:26.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.930+0000 7f1b0159a640 1 -- 172.21.15.116:0/2129127281 >> v2:172.21.15.174:3300/0 conn(0x7f1b04165ac0 msgr2=0x7f1b04071530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:26.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.930+0000 7f1b0159a640 1 --2- 172.21.15.116:0/2129127281 >> v2:172.21.15.174:3300/0 conn(0x7f1b04165ac0 0x7f1b04071530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:26.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.930+0000 7f1b0159a640 1 -- 172.21.15.116:0/2129127281 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b04163240 con 0x7f1b040c2ab0 2024-04-25T00:22:26.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.930+0000 7f1b0159a640 1 -- 172.21.15.116:0/2129127281 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f1af4001680 con 0x7f1b040c2ab0 2024-04-25T00:22:26.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.930+0000 7f1b0159a640 1 -- 172.21.15.116:0/2129127281 >> v1:172.21.15.116:6789/0 conn(0x7f1b040c2ab0 legacy=0x7f1b0415f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:26.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.930+0000 7f1b0159a640 1 --2- 172.21.15.116:0/2129127281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1aec003b20 0x7f1aec003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:26.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.930+0000 7f1b0159a640 1 --2- 172.21.15.116:0/2129127281 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1aec004650 0x7f1aec004ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:26.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.930+0000 7f1b0159a640 1 --2- 172.21.15.116:0/2129127281 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1aec005190 0x7f1aec01c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:26.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.930+0000 7f1b0159a640 1 -- 172.21.15.116:0/2129127281 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1b04162c80 con 0x7f1aec003b20 2024-04-25T00:22:26.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.930+0000 7f1b0159a640 1 -- 172.21.15.116:0/2129127281 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1b04162f60 con 0x7f1aec005190 2024-04-25T00:22:26.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.930+0000 7f1b0159a640 1 -- 172.21.15.116:0/2129127281 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f1aec01d0c0 con 0x7f1aec004650 2024-04-25T00:22:26.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.930+0000 7f1b01d9b640 1 --2- 172.21.15.116:0/2129127281 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1aec004650 0x7f1aec004ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:26.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.930+0000 7f1b0259c640 1 --2- 172.21.15.116:0/2129127281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1aec003b20 0x7f1aec003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:26.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.930+0000 7f1b02d9d640 1 --2- 172.21.15.116:0/2129127281 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1aec005190 0x7f1aec01c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:26.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.930+0000 7f1b0259c640 1 -- 172.21.15.116:0/2129127281 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1aec004650 msgr2=0x7f1aec004ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:26.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.930+0000 7f1b0259c640 1 --2- 172.21.15.116:0/2129127281 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1aec004650 0x7f1aec004ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:26.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.930+0000 7f1b0259c640 1 -- 172.21.15.116:0/2129127281 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1aec005190 msgr2=0x7f1aec01c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:26.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.930+0000 7f1b0259c640 1 --2- 172.21.15.116:0/2129127281 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1aec005190 0x7f1aec01c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:26.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.930+0000 7f1b0259c640 1 -- 172.21.15.116:0/2129127281 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1aec003510 con 0x7f1aec003b20 2024-04-25T00:22:26.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.934+0000 7f1b0259c640 1 --2- 172.21.15.116:0/2129127281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1aec003b20 0x7f1aec003f40 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f1af400c160 tx=0x7f1af401cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=640f1d632869d67f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:26.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.934+0000 7f1b0159a640 1 -- 172.21.15.116:0/2129127281 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af4007c90 con 0x7f1aec003b20 2024-04-25T00:22:26.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.934+0000 7f1b0159a640 1 -- 172.21.15.116:0/2129127281 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1af40a9020 con 0x7f1aec003b20 2024-04-25T00:22:26.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.934+0000 7f1b0159a640 1 -- 172.21.15.116:0/2129127281 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af4005ae0 con 0x7f1aec003b20 2024-04-25T00:22:26.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.934+0000 7f1b0359e640 1 -- 172.21.15.116:0/2129127281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1aec003b20 msgr2=0x7f1aec003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:26.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.934+0000 7f1b0359e640 1 --2- 172.21.15.116:0/2129127281 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1aec003b20 0x7f1aec003f40 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f1af400c160 tx=0x7f1af401cc00 comp rx=0 tx=0).stop 2024-04-25T00:22:26.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.934+0000 7f1b02d9d640 1 -- 172.21.15.116:0/2129127281 reap_dead start 2024-04-25T00:22:26.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.934+0000 7f1b0359e640 1 -- 172.21.15.116:0/2129127281 shutdown_connections 2024-04-25T00:22:26.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.934+0000 7f1b0359e640 1 -- 172.21.15.116:0/2129127281 >> 172.21.15.116:0/2129127281 conn(0x7f1b040c3940 msgr2=0x7f1b040c3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:26.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.934+0000 7f1b0359e640 1 -- 172.21.15.116:0/2129127281 shutdown_connections 2024-04-25T00:22:26.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.934+0000 7f1b0359e640 1 -- 172.21.15.116:0/2129127281 wait complete. 2024-04-25T00:22:26.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.938+0000 7f1b0359e640 1 Processor -- start 2024-04-25T00:22:26.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.942+0000 7f1b0359e640 1 -- start start 2024-04-25T00:22:26.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.942+0000 7f1b0359e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1aec003b20 0x7f1b040c6600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:26.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.942+0000 7f1b0359e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b0415d470 0x7f1b041fbeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:26.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.942+0000 7f1b0359e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1b0415da50 0x7f1b04167020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:26.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.942+0000 7f1b0359e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1b040c2e90 con 0x7f1b0415d470 2024-04-25T00:22:26.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.942+0000 7f1b0359e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1b04162780 con 0x7f1b0415da50 2024-04-25T00:22:26.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.942+0000 7f1b0359e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f1b040c3010 con 0x7f1aec003b20 2024-04-25T00:22:26.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.942+0000 7f1b01d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b0415d470 0x7f1b041fbeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:26.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.942+0000 7f1b01d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b0415d470 0x7f1b041fbeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40412/0 (socket says 172.21.15.116:40412) 2024-04-25T00:22:26.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.942+0000 7f1b01d9b640 1 -- 172.21.15.116:0/1857579364 learned_addr learned my addr 172.21.15.116:0/1857579364 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:22:26.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.942+0000 7f1b02d9d640 1 --2- 172.21.15.116:0/1857579364 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1b0415da50 0x7f1b04167020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:26.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.942+0000 7f1b0259c640 1 --2- 172.21.15.116:0/1857579364 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1aec003b20 0x7f1b040c6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:26.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.942+0000 7f1b01d9b640 1 -- 172.21.15.116:0/1857579364 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1aec003b20 msgr2=0x7f1b040c6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:26.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.942+0000 7f1b01d9b640 1 --2- 172.21.15.116:0/1857579364 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1aec003b20 0x7f1b040c6600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:26.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.942+0000 7f1b01d9b640 1 -- 172.21.15.116:0/1857579364 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1b0415da50 msgr2=0x7f1b04167020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:26.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.942+0000 7f1b01d9b640 1 --2- 172.21.15.116:0/1857579364 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1b0415da50 0x7f1b04167020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:26.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.942+0000 7f1b01d9b640 1 -- 172.21.15.116:0/1857579364 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b041fc3f0 con 0x7f1b0415d470 2024-04-25T00:22:26.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.942+0000 7f1b01d9b640 1 --2- 172.21.15.116:0/1857579364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b0415d470 0x7f1b041fbeb0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f1af800f6e0 tx=0x7f1af800fba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:26.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.946+0000 7f1af2ffd640 1 -- 172.21.15.116:0/1857579364 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af8004230 con 0x7f1b0415d470 2024-04-25T00:22:26.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.946+0000 7f1af2ffd640 1 -- 172.21.15.116:0/1857579364 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1af80043d0 con 0x7f1b0415d470 2024-04-25T00:22:26.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.946+0000 7f1af2ffd640 1 -- 172.21.15.116:0/1857579364 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1af8003860 con 0x7f1b0415d470 2024-04-25T00:22:26.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.946+0000 7f1b0359e640 1 -- 172.21.15.116:0/1857579364 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b041ff4c0 con 0x7f1b0415d470 2024-04-25T00:22:26.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.946+0000 7f1b0359e640 1 -- 172.21.15.116:0/1857579364 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b041fb6b0 con 0x7f1b0415d470 2024-04-25T00:22:26.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.946+0000 7f1af2ffd640 1 -- 172.21.15.116:0/1857579364 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1af8013050 con 0x7f1b0415d470 2024-04-25T00:22:26.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.950+0000 7f1af2ffd640 1 --2- 172.21.15.116:0/1857579364 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1ae40679c0 0x7f1ae4069e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:26.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.950+0000 7f1af2ffd640 1 -- 172.21.15.116:0/1857579364 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 9504+0+0 (secure 0 0 0) 0x7f1af80860c0 con 0x7f1b0415d470 2024-04-25T00:22:26.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.950+0000 7f1b0259c640 1 --2- 172.21.15.116:0/1857579364 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1ae40679c0 0x7f1ae4069e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:26.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.950+0000 7f1b0259c640 1 --2- 172.21.15.116:0/1857579364 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1ae40679c0 0x7f1ae4069e80 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f1af40aa000 tx=0x7f1af4001980 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:26.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.950+0000 7f1b0359e640 1 -- 172.21.15.116:0/1857579364 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b041fb4a0 con 0x7f1b0415d470 2024-04-25T00:22:26.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:26.958+0000 7f1af2ffd640 1 -- 172.21.15.116:0/1857579364 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f1af801a020 con 0x7f1b0415d470 2024-04-25T00:22:27.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:27.266+0000 7fb223244640 1 -- 172.21.15.116:0/985555563 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb214008a90 con 0x7fb21404bef0 2024-04-25T00:22:27.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:27.270+0000 7fb212ffd640 1 -- 172.21.15.116:0/985555563 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 177 v177) ==== 95+0+6501 (secure 0 0 0) 0x7fb20c055870 con 0x7fb21404bef0 2024-04-25T00:22:27.269 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:22:27.270 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":177,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6230,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/492366509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":492366509},{"type":"v1","addr":"172.21.15.174:6837","nonce":492366509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":177}],"filesystems":[{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:22:13.925793+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7989},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7989":{"gid":7989,"name":"c","rank":0,"incarnation":173,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6835/221971311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":221971311},{"type":"v1","addr":"172.21.15.116:6835","nonce":221971311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7989,"qdb_cluster":[7989]},"id":1},{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:20:13.935378+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7722":{"gid":7722,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7722,"qdb_cluster":[7722]},"id":2}]} 2024-04-25T00:22:27.270 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 177 2024-04-25T00:22:27.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:27.270+0000 7fb223244640 1 -- 172.21.15.116:0/985555563 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb2000678f0 msgr2=0x7fb200069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:27.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:27.270+0000 7fb223244640 1 --2- 172.21.15.116:0/985555563 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb2000678f0 0x7fb200069db0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fb21417bb30 tx=0x7fb218009040 comp rx=0 tx=0).stop 2024-04-25T00:22:27.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:27.270+0000 7fb223244640 1 -- 172.21.15.116:0/985555563 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb21404bef0 msgr2=0x7fb21417b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:27.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:27.270+0000 7fb223244640 1 --2- 172.21.15.116:0/985555563 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb21404bef0 0x7fb21417b480 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7fb20c00e820 tx=0x7fb20c00ece0 comp rx=0 tx=0).stop 2024-04-25T00:22:27.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:27.278+0000 7fb223244640 1 -- 172.21.15.116:0/985555563 shutdown_connections 2024-04-25T00:22:27.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:27.278+0000 7fb223244640 1 --2- 172.21.15.116:0/985555563 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb2000678f0 0x7fb200069db0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:27.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:27.278+0000 7fb223244640 1 --2- 172.21.15.116:0/985555563 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb2140e21b0 0x7fb2140e1450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:27.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:27.278+0000 7fb223244640 1 --2- 172.21.15.116:0/985555563 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb2140dc150 0x7fb2140d1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:27.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:27.278+0000 7fb223244640 1 --2- 172.21.15.116:0/985555563 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb21404bef0 0x7fb21417b480 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:27.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:27.278+0000 7fb223244640 1 -- 172.21.15.116:0/985555563 >> 172.21.15.116:0/985555563 conn(0x7fb214049340 msgr2=0x7fb214049720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:27.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:27.278+0000 7fb223244640 1 -- 172.21.15.116:0/985555563 shutdown_connections 2024-04-25T00:22:27.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:27.278+0000 7fb223244640 1 -- 172.21.15.116:0/985555563 wait complete. 2024-04-25T00:22:27.310 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:22:27.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:27.442+0000 7f1b0359e640 1 -- 172.21.15.116:0/1857579364 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1b0404f510 con 0x7f1b0415d470 2024-04-25T00:22:27.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:27.442+0000 7f1af2ffd640 1 -- 172.21.15.116:0/1857579364 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 177 v177) ==== 95+0+6501 (secure 0 0 0) 0x7f1af8011040 con 0x7f1b0415d470 2024-04-25T00:22:27.445 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:22:27.446 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":177,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6230,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/492366509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":492366509},{"type":"v1","addr":"172.21.15.174:6837","nonce":492366509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":177}],"filesystems":[{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:22:13.925793+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7989},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7989":{"gid":7989,"name":"c","rank":0,"incarnation":173,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6835/221971311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":221971311},{"type":"v1","addr":"172.21.15.116:6835","nonce":221971311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7989,"qdb_cluster":[7989]},"id":1},{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:20:13.935378+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7722":{"gid":7722,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7722,"qdb_cluster":[7722]},"id":2}]} 2024-04-25T00:22:27.446 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 177 2024-04-25T00:22:27.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:27.450+0000 7f1b0359e640 1 -- 172.21.15.116:0/1857579364 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1ae40679c0 msgr2=0x7f1ae4069e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:27.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:27.450+0000 7f1b0359e640 1 --2- 172.21.15.116:0/1857579364 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1ae40679c0 0x7f1ae4069e80 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f1af40aa000 tx=0x7f1af4001980 comp rx=0 tx=0).stop 2024-04-25T00:22:27.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:27.450+0000 7f1b0359e640 1 -- 172.21.15.116:0/1857579364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b0415d470 msgr2=0x7f1b041fbeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:27.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:27.450+0000 7f1b0359e640 1 --2- 172.21.15.116:0/1857579364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b0415d470 0x7f1b041fbeb0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f1af800f6e0 tx=0x7f1af800fba0 comp rx=0 tx=0).stop 2024-04-25T00:22:27.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:27.454+0000 7f1b0359e640 1 -- 172.21.15.116:0/1857579364 shutdown_connections 2024-04-25T00:22:27.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:27.454+0000 7f1b0359e640 1 --2- 172.21.15.116:0/1857579364 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1ae40679c0 0x7f1ae4069e80 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:27.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:27.454+0000 7f1b0359e640 1 --2- 172.21.15.116:0/1857579364 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1b0415da50 0x7f1b04167020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:27.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:27.454+0000 7f1b0359e640 1 --2- 172.21.15.116:0/1857579364 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1b0415d470 0x7f1b041fbeb0 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:27.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:27.454+0000 7f1b0359e640 1 --2- 172.21.15.116:0/1857579364 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1aec003b20 0x7f1b040c6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:27.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:27.454+0000 7f1b0359e640 1 -- 172.21.15.116:0/1857579364 >> 172.21.15.116:0/1857579364 conn(0x7f1b040c3940 msgr2=0x7f1b0406c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:27.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:27.454+0000 7f1b0359e640 1 -- 172.21.15.116:0/1857579364 shutdown_connections 2024-04-25T00:22:27.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:27.454+0000 7f1b0359e640 1 -- 172.21.15.116:0/1857579364 wait complete. 2024-04-25T00:22:27.493 INFO:tasks.mds_thrash.fs.[a]:mds.d reported in up:standby state 2024-04-25T00:22:27.493 INFO:tasks.mds_thrash.fs.[a]:waiting for 7 secs before thrashing 2024-04-25T00:22:29.311 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:22:29.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.514+0000 7fd081967640 1 Processor -- start 2024-04-25T00:22:29.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.514+0000 7fd081967640 1 -- start start 2024-04-25T00:22:29.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.518+0000 7fd081967640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd07c13ed60 0x7fd07c1559c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:29.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.518+0000 7fd081967640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fd07c13e810 0x7fd07c155f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:29.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.518+0000 7fd081967640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd07c06d200 con 0x7fd07c13e310 2024-04-25T00:22:29.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.518+0000 7fd081967640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fd07c070d40 con 0x7fd07c13e810 2024-04-25T00:22:29.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.518+0000 7fd081967640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fd07c071020 con 0x7fd07c13ed60 2024-04-25T00:22:29.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.518+0000 7fd07b7fe640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fd07c13e810 0x7fd07c155f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:29.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.518+0000 7fd07b7fe640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fd07c13e810 0x7fd07c155f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:52866/0 (socket says 172.21.15.116:52866) 2024-04-25T00:22:29.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.518+0000 7fd07affd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fd07c13e310 0x7fd07c0c2a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54116/0 (socket says 172.21.15.116:54116) 2024-04-25T00:22:29.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.518+0000 7fd07b7fe640 1 -- 172.21.15.116:0/316813857 learned_addr learned my addr 172.21.15.116:0/316813857 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:22:29.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.518+0000 7fd07b7fe640 1 -- 172.21.15.116:0/316813857 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd07c13ed60 msgr2=0x7fd07c1559c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T00:22:29.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.518+0000 7fd07b7fe640 1 --2- 172.21.15.116:0/316813857 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd07c13ed60 0x7fd07c1559c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:29.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.518+0000 7fd07b7fe640 1 -- 172.21.15.116:0/316813857 >> v1:172.21.15.116:6789/0 conn(0x7fd07c13e310 legacy=0x7fd07c0c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:29.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.518+0000 7fd07b7fe640 1 -- 172.21.15.116:0/316813857 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd07c06cb50 con 0x7fd07c13e810 2024-04-25T00:22:29.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.522+0000 7fd07b7fe640 1 --2- 172.21.15.116:0/316813857 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd07c13e810 0x7fd07c155f00 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fd074009e50 tx=0x7fd07401c530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bbaef75549270cec server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:29.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.530+0000 7fd079ffb640 1 -- 172.21.15.116:0/316813857 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0740a49c0 con 0x7fd07c13e810 2024-04-25T00:22:29.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.530+0000 7fd079ffb640 1 -- 172.21.15.116:0/316813857 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0740a4b60 con 0x7fd07c13e810 2024-04-25T00:22:29.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.530+0000 7fd079ffb640 1 -- 172.21.15.116:0/316813857 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0740ac3f0 con 0x7fd07c13e810 2024-04-25T00:22:29.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.530+0000 7fd081967640 1 -- 172.21.15.116:0/316813857 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd07c13e810 msgr2=0x7fd07c155f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:29.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.530+0000 7fd081967640 1 --2- 172.21.15.116:0/316813857 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd07c13e810 0x7fd07c155f00 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fd074009e50 tx=0x7fd07401c530 comp rx=0 tx=0).stop 2024-04-25T00:22:29.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.534+0000 7fd081967640 1 -- 172.21.15.116:0/316813857 shutdown_connections 2024-04-25T00:22:29.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.534+0000 7fd081967640 1 --2- 172.21.15.116:0/316813857 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd07c13e810 0x7fd07c155f00 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:29.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.534+0000 7fd081967640 1 --2- 172.21.15.116:0/316813857 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd07c13ed60 0x7fd07c1559c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:29.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.534+0000 7fd081967640 1 -- 172.21.15.116:0/316813857 >> 172.21.15.116:0/316813857 conn(0x7fd07c0c3b00 msgr2=0x7fd07c0c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:29.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.534+0000 7fd081967640 1 -- 172.21.15.116:0/316813857 shutdown_connections 2024-04-25T00:22:29.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.534+0000 7fd081967640 1 -- 172.21.15.116:0/316813857 wait complete. 2024-04-25T00:22:29.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.534+0000 7fd081967640 1 Processor -- start 2024-04-25T00:22:29.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.538+0000 7fd081967640 1 -- start start 2024-04-25T00:22:29.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.538+0000 7fd081967640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd07c13e310 0x7fd07c13afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:29.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.538+0000 7fd081967640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd07c13e810 0x7fd07c13d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:29.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.538+0000 7fd081967640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd07c13ed60 0x7fd07c149360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:29.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.538+0000 7fd081967640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd07c1635a0 con 0x7fd07c13e810 2024-04-25T00:22:29.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.538+0000 7fd081967640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd07c163420 con 0x7fd07c13ed60 2024-04-25T00:22:29.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.538+0000 7fd081967640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fd07c163720 con 0x7fd07c13e310 2024-04-25T00:22:29.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.538+0000 7fd07a7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd07c13e810 0x7fd07c13d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:29.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.538+0000 7fd07a7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd07c13e810 0x7fd07c13d260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40418/0 (socket says 172.21.15.116:40418) 2024-04-25T00:22:29.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.538+0000 7fd07a7fc640 1 -- 172.21.15.116:0/2063096681 learned_addr learned my addr 172.21.15.116:0/2063096681 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:22:29.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.538+0000 7fd07b7fe640 1 --2- 172.21.15.116:0/2063096681 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd07c13ed60 0x7fd07c149360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:29.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.538+0000 7fd07b7fe640 1 -- 172.21.15.116:0/2063096681 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd07c13e310 msgr2=0x7fd07c13afb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T00:22:29.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.538+0000 7fd07b7fe640 1 --2- 172.21.15.116:0/2063096681 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd07c13e310 0x7fd07c13afb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:29.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.538+0000 7fd07b7fe640 1 -- 172.21.15.116:0/2063096681 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd07c13e810 msgr2=0x7fd07c13d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:29.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.538+0000 7fd07b7fe640 1 --2- 172.21.15.116:0/2063096681 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd07c13e810 0x7fd07c13d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:29.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.538+0000 7fd07b7fe640 1 -- 172.21.15.116:0/2063096681 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd07c08bd80 con 0x7fd07c13ed60 2024-04-25T00:22:29.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.538+0000 7fd07b7fe640 1 --2- 172.21.15.116:0/2063096681 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd07c13ed60 0x7fd07c149360 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fd0740aa000 tx=0x7fd0740195f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:29.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.542+0000 7fd063fff640 1 -- 172.21.15.116:0/2063096681 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0740bb030 con 0x7fd07c13ed60 2024-04-25T00:22:29.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.542+0000 7fd063fff640 1 -- 172.21.15.116:0/2063096681 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0740090d0 con 0x7fd07c13ed60 2024-04-25T00:22:29.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.546+0000 7fd063fff640 1 -- 172.21.15.116:0/2063096681 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0740b7470 con 0x7fd07c13ed60 2024-04-25T00:22:29.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.546+0000 7fd081967640 1 -- 172.21.15.116:0/2063096681 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd07c13b4f0 con 0x7fd07c13ed60 2024-04-25T00:22:29.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.546+0000 7fd081967640 1 -- 172.21.15.116:0/2063096681 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd07c1386f0 con 0x7fd07c13ed60 2024-04-25T00:22:29.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.550+0000 7fd081967640 1 -- 172.21.15.116:0/2063096681 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd07c068350 con 0x7fd07c13ed60 2024-04-25T00:22:29.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.550+0000 7fd063fff640 1 -- 172.21.15.116:0/2063096681 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd0740be070 con 0x7fd07c13ed60 2024-04-25T00:22:29.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.550+0000 7fd063fff640 1 --2- 172.21.15.116:0/2063096681 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd068070120 0x7fd0680725e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:29.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.550+0000 7fd063fff640 1 -- 172.21.15.116:0/2063096681 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 9504+0+0 (secure 0 0 0) 0x7fd074121160 con 0x7fd07c13ed60 2024-04-25T00:22:29.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.550+0000 7fd07affd640 1 --2- 172.21.15.116:0/2063096681 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd068070120 0x7fd0680725e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:29.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.550+0000 7fd07affd640 1 --2- 172.21.15.116:0/2063096681 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd068070120 0x7fd0680725e0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fd07c132ac0 tx=0x7fd06c0079b0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:29.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.558+0000 7fd063fff640 1 -- 172.21.15.116:0/2063096681 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fd0740c5070 con 0x7fd07c13ed60 2024-04-25T00:22:29.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.986+0000 7fd081967640 1 -- 172.21.15.116:0/2063096681 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd07c0c34f0 con 0x7fd07c13ed60 2024-04-25T00:22:29.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.986+0000 7fd063fff640 1 -- 172.21.15.116:0/2063096681 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 177 v177) ==== 95+0+6501 (secure 0 0 0) 0x7fd07c156650 con 0x7fd07c13ed60 2024-04-25T00:22:29.989 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:22:29.989 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":177,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6230,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/492366509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":492366509},{"type":"v1","addr":"172.21.15.174:6837","nonce":492366509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":177}],"filesystems":[{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:22:13.925793+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7989},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7989":{"gid":7989,"name":"c","rank":0,"incarnation":173,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6835/221971311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":221971311},{"type":"v1","addr":"172.21.15.116:6835","nonce":221971311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7989,"qdb_cluster":[7989]},"id":1},{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:20:13.935378+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7722":{"gid":7722,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7722,"qdb_cluster":[7722]},"id":2}]} 2024-04-25T00:22:29.990 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 177 2024-04-25T00:22:29.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.994+0000 7fd081967640 1 -- 172.21.15.116:0/2063096681 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd068070120 msgr2=0x7fd0680725e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:29.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.994+0000 7fd081967640 1 --2- 172.21.15.116:0/2063096681 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd068070120 0x7fd0680725e0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fd07c132ac0 tx=0x7fd06c0079b0 comp rx=0 tx=0).stop 2024-04-25T00:22:29.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.994+0000 7fd081967640 1 -- 172.21.15.116:0/2063096681 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd07c13ed60 msgr2=0x7fd07c149360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:29.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.994+0000 7fd081967640 1 --2- 172.21.15.116:0/2063096681 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd07c13ed60 0x7fd07c149360 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fd0740aa000 tx=0x7fd0740195f0 comp rx=0 tx=0).stop 2024-04-25T00:22:29.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.994+0000 7fd081967640 1 -- 172.21.15.116:0/2063096681 shutdown_connections 2024-04-25T00:22:29.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.994+0000 7fd081967640 1 --2- 172.21.15.116:0/2063096681 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd068070120 0x7fd0680725e0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:29.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.994+0000 7fd081967640 1 --2- 172.21.15.116:0/2063096681 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd07c13ed60 0x7fd07c149360 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:29.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.994+0000 7fd081967640 1 --2- 172.21.15.116:0/2063096681 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd07c13e810 0x7fd07c13d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:29.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.994+0000 7fd081967640 1 --2- 172.21.15.116:0/2063096681 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd07c13e310 0x7fd07c13afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:29.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.994+0000 7fd081967640 1 -- 172.21.15.116:0/2063096681 >> 172.21.15.116:0/2063096681 conn(0x7fd07c0c3b00 msgr2=0x7fd07c1559c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:29.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.994+0000 7fd081967640 1 -- 172.21.15.116:0/2063096681 shutdown_connections 2024-04-25T00:22:29.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:29.994+0000 7fd081967640 1 -- 172.21.15.116:0/2063096681 wait complete. 2024-04-25T00:22:30.021 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:22:32.022 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:22:32.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.190+0000 7f9cc8a94640 1 Processor -- start 2024-04-25T00:22:32.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.190+0000 7f9cc8a94640 1 -- start start 2024-04-25T00:22:32.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.190+0000 7f9cc8a94640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9cc40a94a0 0x7f9cc41538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:32.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.190+0000 7f9cc8a94640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9cc40c3910 con 0x7f9cc40a99f0 2024-04-25T00:22:32.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.190+0000 7f9cc8a94640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9cc4154530 con 0x7f9cc4150d20 2024-04-25T00:22:32.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.190+0000 7f9cc8a94640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f9cc40c16e0 con 0x7f9cc40a94a0 2024-04-25T00:22:32.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.190+0000 7f9cc2d9d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f9cc40a99f0 0x7f9cc4153e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54124/0 (socket says 172.21.15.116:54124) 2024-04-25T00:22:32.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.190+0000 7f9cc2d9d640 1 -- 172.21.15.116:0/3156930358 learned_addr learned my addr 172.21.15.116:0/3156930358 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:22:32.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.190+0000 7f9cc1d9b640 1 --2- 172.21.15.116:0/3156930358 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9cc40a94a0 0x7f9cc41538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:32.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.190+0000 7f9cc159a640 1 -- 172.21.15.116:0/3156930358 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 814350629 0 0) 0x7f9cc40c3910 con 0x7f9cc40a99f0 2024-04-25T00:22:32.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.190+0000 7f9cc159a640 1 -- 172.21.15.116:0/3156930358 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9cac003530 con 0x7f9cc40a99f0 2024-04-25T00:22:32.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.194+0000 7f9cc159a640 1 -- 172.21.15.116:0/3156930358 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3191872057 0 0) 0x7f9cac003530 con 0x7f9cc40a99f0 2024-04-25T00:22:32.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.194+0000 7f9cc159a640 1 -- 172.21.15.116:0/3156930358 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9cc40a94a0 msgr2=0x7f9cc41538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:32.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.194+0000 7f9cc159a640 1 --2- 172.21.15.116:0/3156930358 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9cc40a94a0 0x7f9cc41538e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:32.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.194+0000 7f9cc159a640 1 -- 172.21.15.116:0/3156930358 >> v1:172.21.15.174:6789/0 conn(0x7f9cc4150d20 legacy=0x7f9cc40c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:32.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.194+0000 7f9cc159a640 1 -- 172.21.15.116:0/3156930358 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cc40c1400 con 0x7f9cc40a99f0 2024-04-25T00:22:32.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.194+0000 7f9cc159a640 1 -- 172.21.15.116:0/3156930358 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f9cbc001a00 con 0x7f9cc40a99f0 2024-04-25T00:22:32.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.194+0000 7f9cc159a640 1 -- 172.21.15.116:0/3156930358 >> v1:172.21.15.116:6789/0 conn(0x7f9cc40a99f0 legacy=0x7f9cc4153e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:32.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.194+0000 7f9cc159a640 1 --2- 172.21.15.116:0/3156930358 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9cac0039f0 0x7f9cac003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:32.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.194+0000 7f9cc159a640 1 --2- 172.21.15.116:0/3156930358 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9cc40a94a0 0x7f9cac004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:32.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.194+0000 7f9cc159a640 1 --2- 172.21.15.116:0/3156930358 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9cac004d50 0x7f9cac0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:32.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.194+0000 7f9cc159a640 1 -- 172.21.15.116:0/3156930358 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9cc40c16e0 con 0x7f9cac0039f0 2024-04-25T00:22:32.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.194+0000 7f9cc159a640 1 -- 172.21.15.116:0/3156930358 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9cac00bab0 con 0x7f9cac004d50 2024-04-25T00:22:32.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.194+0000 7f9cc159a640 1 -- 172.21.15.116:0/3156930358 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f9cac00bcb0 con 0x7f9cc40a94a0 2024-04-25T00:22:32.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.194+0000 7f9cc259c640 1 --2- 172.21.15.116:0/3156930358 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9cac0039f0 0x7f9cac003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:32.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.194+0000 7f9cc1d9b640 1 --2- 172.21.15.116:0/3156930358 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9cc40a94a0 0x7f9cac004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:32.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.194+0000 7f9cc2d9d640 1 --2- 172.21.15.116:0/3156930358 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9cac004d50 0x7f9cac0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:32.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.194+0000 7f9cc259c640 1 -- 172.21.15.116:0/3156930358 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9cc40a94a0 msgr2=0x7f9cac004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:32.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.194+0000 7f9cc259c640 1 --2- 172.21.15.116:0/3156930358 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9cc40a94a0 0x7f9cac004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:32.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.194+0000 7f9cc259c640 1 -- 172.21.15.116:0/3156930358 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9cac004d50 msgr2=0x7f9cac0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:32.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.194+0000 7f9cc259c640 1 --2- 172.21.15.116:0/3156930358 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9cac004d50 0x7f9cac0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:32.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.194+0000 7f9cc259c640 1 -- 172.21.15.116:0/3156930358 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cac003530 con 0x7f9cac0039f0 2024-04-25T00:22:32.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.198+0000 7f9cc259c640 1 --2- 172.21.15.116:0/3156930358 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9cac0039f0 0x7f9cac003e10 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f9cb8001750 tx=0x7f9cb80207f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5830d1d1f97c7dc3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:32.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.198+0000 7f9cc159a640 1 -- 172.21.15.116:0/3156930358 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cb80013a0 con 0x7f9cac0039f0 2024-04-25T00:22:32.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.198+0000 7f9cc159a640 1 -- 172.21.15.116:0/3156930358 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cb80052f0 con 0x7f9cac0039f0 2024-04-25T00:22:32.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.198+0000 7f9cc159a640 1 -- 172.21.15.116:0/3156930358 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cb8003840 con 0x7f9cac0039f0 2024-04-25T00:22:32.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.198+0000 7f9cc8a94640 1 -- 172.21.15.116:0/3156930358 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9cac0039f0 msgr2=0x7f9cac003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:32.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.198+0000 7f9cc8a94640 1 --2- 172.21.15.116:0/3156930358 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9cac0039f0 0x7f9cac003e10 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f9cb8001750 tx=0x7f9cb80207f0 comp rx=0 tx=0).stop 2024-04-25T00:22:32.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.198+0000 7f9cc2d9d640 1 -- 172.21.15.116:0/3156930358 reap_dead start 2024-04-25T00:22:32.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.198+0000 7f9cc8a94640 1 -- 172.21.15.116:0/3156930358 shutdown_connections 2024-04-25T00:22:32.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.198+0000 7f9cc8a94640 1 -- 172.21.15.116:0/3156930358 >> 172.21.15.116:0/3156930358 conn(0x7f9cc40c3b00 msgr2=0x7f9cc40c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:32.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.198+0000 7f9cc8a94640 1 -- 172.21.15.116:0/3156930358 shutdown_connections 2024-04-25T00:22:32.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.198+0000 7f9cc8a94640 1 -- 172.21.15.116:0/3156930358 wait complete. 2024-04-25T00:22:32.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.198+0000 7f9cc8a94640 1 Processor -- start 2024-04-25T00:22:32.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.198+0000 7f9cc8a94640 1 -- start start 2024-04-25T00:22:32.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.202+0000 7f9cc8a94640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9cc414f5a0 0x7f9cc414f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:32.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.202+0000 7f9cc8a94640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9cc4143440 0x7f9cc4151f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:32.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.202+0000 7f9cc8a94640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9cc4143970 0x7f9cc4148c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:32.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.202+0000 7f9cc8a94640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9cc40c32a0 con 0x7f9cc4143440 2024-04-25T00:22:32.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.202+0000 7f9cc8a94640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9cc40c3120 con 0x7f9cc4143970 2024-04-25T00:22:32.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.202+0000 7f9cc8a94640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f9cc4071550 con 0x7f9cc414f5a0 2024-04-25T00:22:32.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.202+0000 7f9cc1d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9cc4143440 0x7f9cc4151f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:32.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.202+0000 7f9cc1d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9cc4143440 0x7f9cc4151f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40440/0 (socket says 172.21.15.116:40440) 2024-04-25T00:22:32.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.202+0000 7f9cc1d9b640 1 -- 172.21.15.116:0/2655594882 learned_addr learned my addr 172.21.15.116:0/2655594882 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:22:32.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.202+0000 7f9cc259c640 1 --2- 172.21.15.116:0/2655594882 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9cc414f5a0 0x7f9cc414f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:32.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.202+0000 7f9cc2d9d640 1 --2- 172.21.15.116:0/2655594882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9cc4143970 0x7f9cc4148c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:32.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.202+0000 7f9cc1d9b640 1 -- 172.21.15.116:0/2655594882 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9cc414f5a0 msgr2=0x7f9cc414f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:32.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.202+0000 7f9cc1d9b640 1 --2- 172.21.15.116:0/2655594882 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9cc414f5a0 0x7f9cc414f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:32.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.202+0000 7f9cc1d9b640 1 -- 172.21.15.116:0/2655594882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9cc4143970 msgr2=0x7f9cc4148c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:32.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.202+0000 7f9cc1d9b640 1 --2- 172.21.15.116:0/2655594882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9cc4143970 0x7f9cc4148c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:32.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.202+0000 7f9cc1d9b640 1 -- 172.21.15.116:0/2655594882 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cc41514a0 con 0x7f9cc4143440 2024-04-25T00:22:32.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.202+0000 7f9cc259c640 1 --2- 172.21.15.116:0/2655594882 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9cc414f5a0 0x7f9cc414f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:22:32.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.202+0000 7f9cc1d9b640 1 --2- 172.21.15.116:0/2655594882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9cc4143440 0x7f9cc4151f40 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f9cb4011610 tx=0x7f9cb4011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:32.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.202+0000 7f9cb2ffd640 1 -- 172.21.15.116:0/2655594882 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cb400f8f0 con 0x7f9cc4143440 2024-04-25T00:22:32.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.202+0000 7f9cb2ffd640 1 -- 172.21.15.116:0/2655594882 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cb400fa90 con 0x7f9cc4143440 2024-04-25T00:22:32.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.202+0000 7f9cc8a94640 1 -- 172.21.15.116:0/2655594882 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cc4153070 con 0x7f9cc4143440 2024-04-25T00:22:32.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.202+0000 7f9cb2ffd640 1 -- 172.21.15.116:0/2655594882 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cb400fe40 con 0x7f9cc4143440 2024-04-25T00:22:32.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.206+0000 7f9cc8a94640 1 -- 172.21.15.116:0/2655594882 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cc4153350 con 0x7f9cc4143440 2024-04-25T00:22:32.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.206+0000 7f9cb2ffd640 1 -- 172.21.15.116:0/2655594882 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f9cb4015460 con 0x7f9cc4143440 2024-04-25T00:22:32.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.206+0000 7f9cb2ffd640 1 --2- 172.21.15.116:0/2655594882 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9ca40678f0 0x7f9ca4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:32.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.206+0000 7f9cb2ffd640 1 -- 172.21.15.116:0/2655594882 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 9504+0+0 (secure 0 0 0) 0x7f9cb408a170 con 0x7f9cc4143440 2024-04-25T00:22:32.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.206+0000 7f9cc259c640 1 --2- 172.21.15.116:0/2655594882 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9ca40678f0 0x7f9ca4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:32.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.206+0000 7f9cc259c640 1 --2- 172.21.15.116:0/2655594882 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9ca40678f0 0x7f9ca4069db0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f9cac0057f0 tx=0x7f9cb80b6040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:32.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.206+0000 7f9cc8a94640 1 -- 172.21.15.116:0/2655594882 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9cc4068350 con 0x7f9cc4143440 2024-04-25T00:22:32.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.214+0000 7f9cb2ffd640 1 -- 172.21.15.116:0/2655594882 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f9cb4019030 con 0x7f9cc4143440 2024-04-25T00:22:32.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.706+0000 7f9cc8a94640 1 -- 172.21.15.116:0/2655594882 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9cc40c34f0 con 0x7f9cc4143440 2024-04-25T00:22:32.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.706+0000 7f9cb2ffd640 1 -- 172.21.15.116:0/2655594882 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 177 v177) ==== 95+0+6501 (secure 0 0 0) 0x7f9cb4056660 con 0x7f9cc4143440 2024-04-25T00:22:32.710 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:22:32.710 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":177,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6230,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/492366509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":492366509},{"type":"v1","addr":"172.21.15.174:6837","nonce":492366509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":177}],"filesystems":[{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:22:13.925793+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7989},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7989":{"gid":7989,"name":"c","rank":0,"incarnation":173,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6835/221971311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":221971311},{"type":"v1","addr":"172.21.15.116:6835","nonce":221971311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7989,"qdb_cluster":[7989]},"id":1},{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:20:13.935378+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7722":{"gid":7722,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7722,"qdb_cluster":[7722]},"id":2}]} 2024-04-25T00:22:32.710 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 177 2024-04-25T00:22:32.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.714+0000 7f9cc8a94640 1 -- 172.21.15.116:0/2655594882 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9ca40678f0 msgr2=0x7f9ca4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:32.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.714+0000 7f9cc8a94640 1 --2- 172.21.15.116:0/2655594882 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9ca40678f0 0x7f9ca4069db0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f9cac0057f0 tx=0x7f9cb80b6040 comp rx=0 tx=0).stop 2024-04-25T00:22:32.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.714+0000 7f9cc8a94640 1 -- 172.21.15.116:0/2655594882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9cc4143440 msgr2=0x7f9cc4151f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:32.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.714+0000 7f9cc8a94640 1 --2- 172.21.15.116:0/2655594882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9cc4143440 0x7f9cc4151f40 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f9cb4011610 tx=0x7f9cb4011ad0 comp rx=0 tx=0).stop 2024-04-25T00:22:32.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.714+0000 7f9cc8a94640 1 -- 172.21.15.116:0/2655594882 shutdown_connections 2024-04-25T00:22:32.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.714+0000 7f9cc8a94640 1 --2- 172.21.15.116:0/2655594882 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9ca40678f0 0x7f9ca4069db0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:32.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.714+0000 7f9cc8a94640 1 --2- 172.21.15.116:0/2655594882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9cc4143970 0x7f9cc4148c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:32.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.714+0000 7f9cc8a94640 1 --2- 172.21.15.116:0/2655594882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9cc4143440 0x7f9cc4151f40 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:32.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.714+0000 7f9cc8a94640 1 --2- 172.21.15.116:0/2655594882 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9cc414f5a0 0x7f9cc414f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:32.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.714+0000 7f9cc8a94640 1 -- 172.21.15.116:0/2655594882 >> 172.21.15.116:0/2655594882 conn(0x7f9cc40c3b00 msgr2=0x7f9cc4157ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:32.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.718+0000 7f9cc8a94640 1 -- 172.21.15.116:0/2655594882 shutdown_connections 2024-04-25T00:22:32.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:32.718+0000 7f9cc8a94640 1 -- 172.21.15.116:0/2655594882 wait complete. 2024-04-25T00:22:32.743 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:22:34.494 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:22:34.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.678+0000 7fc2b9032640 1 Processor -- start 2024-04-25T00:22:34.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.678+0000 7fc2b9032640 1 -- start start 2024-04-25T00:22:34.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.678+0000 7fc2b9032640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc2b41572d0 0x7fc2b40a84b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:34.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.678+0000 7fc2b9032640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc2b406e080 0x7fc2b4070f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:34.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.678+0000 7fc2b9032640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2b40c36f0 con 0x7fc2b40714d0 2024-04-25T00:22:34.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.678+0000 7fc2b9032640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fc2b415e140 con 0x7fc2b406e080 2024-04-25T00:22:34.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.678+0000 7fc2b9032640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fc2b40a89f0 con 0x7fc2b41572d0 2024-04-25T00:22:34.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.678+0000 7fc2b259c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc2b406e080 0x7fc2b4070f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:34.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.678+0000 7fc2b259c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc2b406e080 0x7fc2b4070f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40446/0 (socket says 172.21.15.116:40446) 2024-04-25T00:22:34.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.678+0000 7fc2b259c640 1 -- 172.21.15.116:0/1946386160 learned_addr learned my addr 172.21.15.116:0/1946386160 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:22:34.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.678+0000 7fc2b359e640 1 --1- 172.21.15.116:0/1946386160 >> v1:172.21.15.174:6789/0 conn(0x7fc2b40714d0 0x7fc2b40c5030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:45516/0 (socket says 172.21.15.116:45516) 2024-04-25T00:22:34.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.678+0000 7fc2b2d9d640 1 --2- 172.21.15.116:0/1946386160 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc2b41572d0 0x7fc2b40a84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:34.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.682+0000 7fc2b259c640 1 -- 172.21.15.116:0/1946386160 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc2b41572d0 msgr2=0x7fc2b40a84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:34.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.682+0000 7fc2b259c640 1 --2- 172.21.15.116:0/1946386160 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc2b41572d0 0x7fc2b40a84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:34.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.682+0000 7fc2b259c640 1 -- 172.21.15.116:0/1946386160 >> v1:172.21.15.174:6789/0 conn(0x7fc2b40714d0 legacy=0x7fc2b40c5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:34.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.682+0000 7fc2b259c640 1 -- 172.21.15.116:0/1946386160 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2b415d380 con 0x7fc2b406e080 2024-04-25T00:22:34.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.682+0000 7fc2b259c640 1 --2- 172.21.15.116:0/1946386160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2b406e080 0x7fc2b4070f90 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7fc2a80098a0 tx=0x7fc2a801c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a81ec32256e8821 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:34.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.682+0000 7fc2b1d9b640 1 -- 172.21.15.116:0/1946386160 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2a80b0070 con 0x7fc2b406e080 2024-04-25T00:22:34.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.682+0000 7fc2b1d9b640 1 -- 172.21.15.116:0/1946386160 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2a80ab600 con 0x7fc2b406e080 2024-04-25T00:22:34.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.682+0000 7fc2b1d9b640 1 -- 172.21.15.116:0/1946386160 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2a80ab9d0 con 0x7fc2b406e080 2024-04-25T00:22:34.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.686+0000 7fc2b9032640 1 -- 172.21.15.116:0/1946386160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2b406e080 msgr2=0x7fc2b4070f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:34.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.686+0000 7fc2b9032640 1 --2- 172.21.15.116:0/1946386160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2b406e080 0x7fc2b4070f90 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7fc2a80098a0 tx=0x7fc2a801c540 comp rx=0 tx=0).stop 2024-04-25T00:22:34.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.686+0000 7fc2b9032640 1 -- 172.21.15.116:0/1946386160 shutdown_connections 2024-04-25T00:22:34.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.686+0000 7fc2b9032640 1 --2- 172.21.15.116:0/1946386160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2b406e080 0x7fc2b4070f90 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:34.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.686+0000 7fc2b9032640 1 --2- 172.21.15.116:0/1946386160 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc2b41572d0 0x7fc2b40a84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:34.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.686+0000 7fc2b9032640 1 -- 172.21.15.116:0/1946386160 >> 172.21.15.116:0/1946386160 conn(0x7fc2b4074650 msgr2=0x7fc2b4074a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:34.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.690+0000 7fc2b9032640 1 -- 172.21.15.116:0/1946386160 shutdown_connections 2024-04-25T00:22:34.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.690+0000 7fc2b9032640 1 -- 172.21.15.116:0/1946386160 wait complete. 2024-04-25T00:22:34.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.690+0000 7fc2b9032640 1 Processor -- start 2024-04-25T00:22:34.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.690+0000 7fc2b9032640 1 -- start start 2024-04-25T00:22:34.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.694+0000 7fc2b9032640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc2b406e080 0x7fc2b4157ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:34.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.694+0000 7fc2b9032640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2b40714d0 0x7fc2b40c79f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:34.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.694+0000 7fc2b9032640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc2b41572d0 0x7fc2b40c7f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:34.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.694+0000 7fc2b9032640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b4166d50 con 0x7fc2b40714d0 2024-04-25T00:22:34.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.694+0000 7fc2b2d9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc2b406e080 0x7fc2b4157ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:34.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.694+0000 7fc2b359e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc2b41572d0 0x7fc2b40c7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:34.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.694+0000 7fc2b259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2b40714d0 0x7fc2b40c79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:34.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.694+0000 7fc2b359e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc2b41572d0 0x7fc2b40c7f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:55066/0 (socket says 172.21.15.116:55066) 2024-04-25T00:22:34.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.694+0000 7fc2b259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2b40714d0 0x7fc2b40c79f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40448/0 (socket says 172.21.15.116:40448) 2024-04-25T00:22:34.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.694+0000 7fc2b359e640 1 -- 172.21.15.116:0/67737767 learned_addr learned my addr 172.21.15.116:0/67737767 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:22:34.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.694+0000 7fc2b9032640 1 -- 172.21.15.116:0/67737767 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b40c5b50 con 0x7fc2b406e080 2024-04-25T00:22:34.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.694+0000 7fc2b9032640 1 -- 172.21.15.116:0/67737767 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fc2b4166ed0 con 0x7fc2b41572d0 2024-04-25T00:22:34.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.694+0000 7fc2b359e640 1 -- 172.21.15.116:0/67737767 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc2b406e080 msgr2=0x7fc2b4157ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:34.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.694+0000 7fc2b359e640 1 --2- 172.21.15.116:0/67737767 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc2b406e080 0x7fc2b4157ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:34.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.694+0000 7fc2b359e640 1 -- 172.21.15.116:0/67737767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2b40714d0 msgr2=0x7fc2b40c79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:34.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.694+0000 7fc2b359e640 1 --2- 172.21.15.116:0/67737767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2b40714d0 0x7fc2b40c79f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:34.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.694+0000 7fc2b359e640 1 -- 172.21.15.116:0/67737767 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2b40ae680 con 0x7fc2b41572d0 2024-04-25T00:22:34.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.694+0000 7fc2b259c640 1 --2- 172.21.15.116:0/67737767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2b40714d0 0x7fc2b40c79f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:22:34.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.694+0000 7fc2b359e640 1 --2- 172.21.15.116:0/67737767 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc2b41572d0 0x7fc2b40c7f30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac00c740 tx=0x7fc2ac00cc00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:34.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.694+0000 7fc29b7fe640 1 -- 172.21.15.116:0/67737767 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ac017070 con 0x7fc2b41572d0 2024-04-25T00:22:34.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.694+0000 7fc29b7fe640 1 -- 172.21.15.116:0/67737767 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2ac00ce50 con 0x7fc2b41572d0 2024-04-25T00:22:34.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.694+0000 7fc2b9032640 1 -- 172.21.15.116:0/67737767 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2b415a210 con 0x7fc2b41572d0 2024-04-25T00:22:34.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.694+0000 7fc29b7fe640 1 -- 172.21.15.116:0/67737767 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ac00a840 con 0x7fc2b41572d0 2024-04-25T00:22:34.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.694+0000 7fc2b9032640 1 -- 172.21.15.116:0/67737767 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2b40c8470 con 0x7fc2b41572d0 2024-04-25T00:22:34.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.694+0000 7fc29b7fe640 1 -- 172.21.15.116:0/67737767 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc2ac01a070 con 0x7fc2b41572d0 2024-04-25T00:22:34.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.698+0000 7fc29b7fe640 1 --2- 172.21.15.116:0/67737767 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc2a0067910 0x7fc2a0069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:34.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.698+0000 7fc29b7fe640 1 -- 172.21.15.116:0/67737767 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(61..61 src has 1..61) ==== 9504+0+0 (secure 0 0 0) 0x7fc2ac085f70 con 0x7fc2b41572d0 2024-04-25T00:22:34.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.698+0000 7fc2b2d9d640 1 --2- 172.21.15.116:0/67737767 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc2a0067910 0x7fc2a0069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:34.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.698+0000 7fc2b2d9d640 1 --2- 172.21.15.116:0/67737767 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc2a0067910 0x7fc2a0069dd0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fc2b40af430 tx=0x7fc2a4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:34.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.698+0000 7fc2b9032640 1 -- 172.21.15.116:0/67737767 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2b4068a60 con 0x7fc2b41572d0 2024-04-25T00:22:34.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.706+0000 7fc29b7fe640 1 -- 172.21.15.116:0/67737767 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fc2ac055650 con 0x7fc2b41572d0 2024-04-25T00:22:34.744 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:22:34.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.994+0000 7f83c86a4640 1 Processor -- start 2024-04-25T00:22:34.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.994+0000 7f83c86a4640 1 -- start start 2024-04-25T00:22:35.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.998+0000 7f83c86a4640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f83b8008750 0x7f83b8008b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:35.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.998+0000 7f83c86a4640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f83b800b240 0x7f83b8006f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:35.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.998+0000 7f83c6440640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f83b8008750 0x7f83b8008b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:35.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.998+0000 7f83c6440640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f83b8008750 0x7f83b8008b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37256/0 (socket says 172.21.15.116:37256) 2024-04-25T00:22:35.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.998+0000 7f83c6c41640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f83b80d8110 0x7f83b80d8540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:43876/0 (socket says 172.21.15.116:43876) 2024-04-25T00:22:35.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.998+0000 7f83c5c3f640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f83b800b240 0x7f83b8006f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:35.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.998+0000 7f83c5c3f640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f83b800b240 0x7f83b8006f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:39432/0 (socket says 172.21.15.116:39432) 2024-04-25T00:22:35.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.998+0000 7f83c5c3f640 1 -- 172.21.15.116:0/1229277646 learned_addr learned my addr 172.21.15.116:0/1229277646 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:22:35.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.998+0000 7f83c86a4640 1 -- 172.21.15.116:0/1229277646 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83b80db950 con 0x7f83b80d8110 2024-04-25T00:22:35.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.998+0000 7f83c86a4640 1 -- 172.21.15.116:0/1229277646 --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f83b80db700 con 0x7f83b8008750 2024-04-25T00:22:35.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.998+0000 7f83c86a4640 1 -- 172.21.15.116:0/1229277646 --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f83b80dc280 con 0x7f83b800b240 2024-04-25T00:22:35.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.998+0000 7f83c543e640 1 -- 172.21.15.116:0/1229277646 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3174988311 0 0) 0x7f83b80db950 con 0x7f83b80d8110 2024-04-25T00:22:35.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.998+0000 7f83c543e640 1 -- 172.21.15.116:0/1229277646 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f83ac003530 con 0x7f83b80d8110 2024-04-25T00:22:35.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.998+0000 7f83c6440640 1 -- 172.21.15.116:0/1229277646 >> v2:172.21.15.174:3300/0 conn(0x7f83b800b240 msgr2=0x7f83b8006f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:35.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.998+0000 7f83c6440640 1 --2- 172.21.15.116:0/1229277646 >> v2:172.21.15.174:3300/0 conn(0x7f83b800b240 0x7f83b8006f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:35.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.998+0000 7f83c6440640 1 -- 172.21.15.116:0/1229277646 >> v1:172.21.15.116:6789/0 conn(0x7f83b80d8110 legacy=0x7f83b80d8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:35.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:34.998+0000 7f83c6440640 1 -- 172.21.15.116:0/1229277646 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83b80dbc30 con 0x7f83b8008750 2024-04-25T00:22:35.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.002+0000 7f83c6440640 1 --2- 172.21.15.116:0/1229277646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83b8008750 0x7f83b8008b30 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f83b804bd10 tx=0x7f83bc01c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d3e94cc1a3d1f45 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:35.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.006+0000 7f83c543e640 1 -- 172.21.15.116:0/1229277646 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83bc0b0070 con 0x7f83b8008750 2024-04-25T00:22:35.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.006+0000 7f83c543e640 1 -- 172.21.15.116:0/1229277646 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83bc0ab600 con 0x7f83b8008750 2024-04-25T00:22:35.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.006+0000 7f83c543e640 1 -- 172.21.15.116:0/1229277646 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83bc0ab9d0 con 0x7f83b8008750 2024-04-25T00:22:35.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.006+0000 7f83c86a4640 1 -- 172.21.15.116:0/1229277646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83b8008750 msgr2=0x7f83b8008b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:35.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.006+0000 7f83c86a4640 1 --2- 172.21.15.116:0/1229277646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83b8008750 0x7f83b8008b30 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f83b804bd10 tx=0x7f83bc01c360 comp rx=0 tx=0).stop 2024-04-25T00:22:35.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.014+0000 7f83c86a4640 1 -- 172.21.15.116:0/1229277646 shutdown_connections 2024-04-25T00:22:35.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.014+0000 7f83c86a4640 1 --2- 172.21.15.116:0/1229277646 >> v2:172.21.15.174:3300/0 conn(0x7f83b800b240 0x7f83b8006f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:35.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.014+0000 7f83c86a4640 1 --2- 172.21.15.116:0/1229277646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83b8008750 0x7f83b8008b30 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:35.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.014+0000 7f83c86a4640 1 -- 172.21.15.116:0/1229277646 >> 172.21.15.116:0/1229277646 conn(0x7f83b8049370 msgr2=0x7f83b8049780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:35.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.014+0000 7f83c86a4640 1 -- 172.21.15.116:0/1229277646 shutdown_connections 2024-04-25T00:22:35.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.014+0000 7f83c86a4640 1 -- 172.21.15.116:0/1229277646 wait complete. 2024-04-25T00:22:35.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.018+0000 7f83c86a4640 1 Processor -- start 2024-04-25T00:22:35.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.018+0000 7f83c86a4640 1 -- start start 2024-04-25T00:22:35.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.022+0000 7f83c86a4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83b8008750 0x7f83b80cc050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:35.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.022+0000 7f83c86a4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83b800b240 0x7f83b80d6f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:35.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.022+0000 7f83c86a4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f83b80d8110 0x7f83b817b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:35.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.022+0000 7f83c86a4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f83b80e5430 con 0x7f83b8008750 2024-04-25T00:22:35.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.022+0000 7f83c86a4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f83b80e52b0 con 0x7f83b80d8110 2024-04-25T00:22:35.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.022+0000 7f83c86a4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f83b80e55b0 con 0x7f83b800b240 2024-04-25T00:22:35.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.022+0000 7f83c6440640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83b8008750 0x7f83b80cc050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:35.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.022+0000 7f83c6440640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83b8008750 0x7f83b80cc050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37264/0 (socket says 172.21.15.116:37264) 2024-04-25T00:22:35.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.022+0000 7f83c6440640 1 -- 172.21.15.116:0/2613334654 learned_addr learned my addr 172.21.15.116:0/2613334654 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:22:35.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.022+0000 7f83c5c3f640 1 --2- 172.21.15.116:0/2613334654 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83b800b240 0x7f83b80d6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:35.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.022+0000 7f83c6440640 1 -- 172.21.15.116:0/2613334654 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83b800b240 msgr2=0x7f83b80d6f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:35.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.022+0000 7f83c6440640 1 --2- 172.21.15.116:0/2613334654 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83b800b240 0x7f83b80d6f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:35.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.022+0000 7f83c6440640 1 -- 172.21.15.116:0/2613334654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f83b80d8110 msgr2=0x7f83b817b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:35.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.022+0000 7f83c6440640 1 --2- 172.21.15.116:0/2613334654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f83b80d8110 0x7f83b817b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:35.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.022+0000 7f83c6440640 1 -- 172.21.15.116:0/2613334654 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83b8000ec0 con 0x7f83b8008750 2024-04-25T00:22:35.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.022+0000 7f83c6440640 1 --2- 172.21.15.116:0/2613334654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83b8008750 0x7f83b80cc050 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f83bc018f70 tx=0x7f83bc019100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:35.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.022+0000 7f83b6ffd640 1 -- 172.21.15.116:0/2613334654 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83bc0b9070 con 0x7f83b8008750 2024-04-25T00:22:35.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.022+0000 7f83b6ffd640 1 -- 172.21.15.116:0/2613334654 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83bc0b5480 con 0x7f83b8008750 2024-04-25T00:22:35.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.022+0000 7f83b6ffd640 1 -- 172.21.15.116:0/2613334654 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83bc0b0040 con 0x7f83b8008750 2024-04-25T00:22:35.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.022+0000 7f83c86a4640 1 -- 172.21.15.116:0/2613334654 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83b80473c0 con 0x7f83b8008750 2024-04-25T00:22:35.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.022+0000 7f83c86a4640 1 -- 172.21.15.116:0/2613334654 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83b80daaa0 con 0x7f83b8008750 2024-04-25T00:22:35.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.026+0000 7f83b6ffd640 1 -- 172.21.15.116:0/2613334654 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f83bc0bc070 con 0x7f83b8008750 2024-04-25T00:22:35.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.026+0000 7f83b6ffd640 1 --2- 172.21.15.116:0/2613334654 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f83a40678c0 0x7f83a4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:35.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.026+0000 7f83b6ffd640 1 -- 172.21.15.116:0/2613334654 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 9504+0+0 (secure 0 0 0) 0x7f83bc120b30 con 0x7f83b8008750 2024-04-25T00:22:35.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.026+0000 7f83c5c3f640 1 --2- 172.21.15.116:0/2613334654 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f83a40678c0 0x7f83a4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:35.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.030+0000 7f83c86a4640 1 -- 172.21.15.116:0/2613334654 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83b8005c30 con 0x7f83b8008750 2024-04-25T00:22:35.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.038+0000 7f83c5c3f640 1 --2- 172.21.15.116:0/2613334654 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f83a40678c0 0x7f83a4069d80 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f83b80da750 tx=0x7f83b0009240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:35.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.054+0000 7f83b6ffd640 1 -- 172.21.15.116:0/2613334654 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f83bc0c3030 con 0x7f83b8008750 2024-04-25T00:22:35.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.302+0000 7fc2b9032640 1 -- 172.21.15.116:0/67737767 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc2b4070d70 con 0x7fc2b41572d0 2024-04-25T00:22:35.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.302+0000 7fc29b7fe640 1 -- 172.21.15.116:0/67737767 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 177 v177) ==== 95+0+6501 (secure 0 0 0) 0x7fc2ac059950 con 0x7fc2b41572d0 2024-04-25T00:22:35.302 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:22:35.303 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":177,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6230,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/492366509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":492366509},{"type":"v1","addr":"172.21.15.174:6837","nonce":492366509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":177}],"filesystems":[{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:22:13.925793+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7989},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7989":{"gid":7989,"name":"c","rank":0,"incarnation":173,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6835/221971311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":221971311},{"type":"v1","addr":"172.21.15.116:6835","nonce":221971311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7989,"qdb_cluster":[7989]},"id":1},{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:20:13.935378+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7722":{"gid":7722,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7722,"qdb_cluster":[7722]},"id":2}]} 2024-04-25T00:22:35.303 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 177 2024-04-25T00:22:35.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.306+0000 7fc2b9032640 1 -- 172.21.15.116:0/67737767 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc2a0067910 msgr2=0x7fc2a0069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:35.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.306+0000 7fc2b9032640 1 --2- 172.21.15.116:0/67737767 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc2a0067910 0x7fc2a0069dd0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fc2b40af430 tx=0x7fc2a4009190 comp rx=0 tx=0).stop 2024-04-25T00:22:35.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.306+0000 7fc2b9032640 1 -- 172.21.15.116:0/67737767 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc2b41572d0 msgr2=0x7fc2b40c7f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:35.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.306+0000 7fc2b9032640 1 --2- 172.21.15.116:0/67737767 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc2b41572d0 0x7fc2b40c7f30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac00c740 tx=0x7fc2ac00cc00 comp rx=0 tx=0).stop 2024-04-25T00:22:35.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.306+0000 7fc2b9032640 1 -- 172.21.15.116:0/67737767 shutdown_connections 2024-04-25T00:22:35.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.306+0000 7fc2b9032640 1 --2- 172.21.15.116:0/67737767 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc2a0067910 0x7fc2a0069dd0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:35.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.306+0000 7fc2b9032640 1 --2- 172.21.15.116:0/67737767 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc2b41572d0 0x7fc2b40c7f30 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:35.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.306+0000 7fc2b9032640 1 --2- 172.21.15.116:0/67737767 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc2b40714d0 0x7fc2b40c79f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:35.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.306+0000 7fc2b9032640 1 --2- 172.21.15.116:0/67737767 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc2b406e080 0x7fc2b4157ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:35.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.306+0000 7fc2b9032640 1 -- 172.21.15.116:0/67737767 >> 172.21.15.116:0/67737767 conn(0x7fc2b4074650 msgr2=0x7fc2b4163d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:35.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.306+0000 7fc2b9032640 1 -- 172.21.15.116:0/67737767 shutdown_connections 2024-04-25T00:22:35.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.306+0000 7fc2b9032640 1 -- 172.21.15.116:0/67737767 wait complete. 2024-04-25T00:22:35.344 INFO:tasks.mds_thrash.fs.[a]:kill mds.c (rank=0) 2024-04-25T00:22:35.345 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-25T00:22:35.345 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:22:35.361 INFO:tasks.ceph.mds.c:Stopped 2024-04-25T00:22:35.361 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:22:35.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.622+0000 7f845cc79640 1 Processor -- start 2024-04-25T00:22:35.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.622+0000 7f845cc79640 1 -- start start 2024-04-25T00:22:35.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.626+0000 7f845cc79640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f84580c4840 0x7f84580c7680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:35.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.626+0000 7f845cc79640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f845815a610 0x7f8458164510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:35.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.626+0000 7f845cc79640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f845815aaa0 con 0x7f84580c72a0 2024-04-25T00:22:35.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.626+0000 7f845cc79640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f84580c43b0 con 0x7f845815a610 2024-04-25T00:22:35.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.626+0000 7f845cc79640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f845806f7e0 con 0x7f84580c4840 2024-04-25T00:22:35.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.626+0000 7f845659c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f84580c72a0 0x7f8458072200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:43882/0 (socket says 172.21.15.116:43882) 2024-04-25T00:22:35.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.626+0000 7f845659c640 1 -- 172.21.15.116:0/2568177072 learned_addr learned my addr 172.21.15.116:0/2568177072 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:22:35.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.626+0000 7f8456d9d640 1 --2- 172.21.15.116:0/2568177072 >> v2:172.21.15.174:3300/0 conn(0x7f845815a610 0x7f8458164510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:35.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.626+0000 7f845559a640 1 -- 172.21.15.116:0/2568177072 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 252833446 0 0) 0x7f845815aaa0 con 0x7f84580c72a0 2024-04-25T00:22:35.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.626+0000 7f845559a640 1 -- 172.21.15.116:0/2568177072 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8444003530 con 0x7f84580c72a0 2024-04-25T00:22:35.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.626+0000 7f8455d9b640 1 --2- 172.21.15.116:0/2568177072 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f84580c4840 0x7f84580c7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:35.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.626+0000 7f845559a640 1 -- 172.21.15.116:0/2568177072 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3705638921 0 0) 0x7f8444003530 con 0x7f84580c72a0 2024-04-25T00:22:35.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.626+0000 7f845559a640 1 -- 172.21.15.116:0/2568177072 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f84580c4840 msgr2=0x7f84580c7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:35.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.626+0000 7f845559a640 1 --2- 172.21.15.116:0/2568177072 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f84580c4840 0x7f84580c7680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:35.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.626+0000 7f845559a640 1 -- 172.21.15.116:0/2568177072 >> v2:172.21.15.174:3300/0 conn(0x7f845815a610 msgr2=0x7f8458164510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:35.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.626+0000 7f845559a640 1 --2- 172.21.15.116:0/2568177072 >> v2:172.21.15.174:3300/0 conn(0x7f845815a610 0x7f8458164510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:35.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.626+0000 7f845559a640 1 -- 172.21.15.116:0/2568177072 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f845806fac0 con 0x7f84580c72a0 2024-04-25T00:22:35.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.626+0000 7f845559a640 1 -- 172.21.15.116:0/2568177072 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f8450001a00 con 0x7f84580c72a0 2024-04-25T00:22:35.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.626+0000 7f845559a640 1 -- 172.21.15.116:0/2568177072 >> v1:172.21.15.116:6789/0 conn(0x7f84580c72a0 legacy=0x7f8458072200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:35.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.626+0000 7f845559a640 1 --2- 172.21.15.116:0/2568177072 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f84440039c0 0x7f8444003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:35.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.626+0000 7f845559a640 1 --2- 172.21.15.116:0/2568177072 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8444004810 0x7f8444004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:35.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.626+0000 7f845559a640 1 --2- 172.21.15.116:0/2568177072 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84440051d0 0x7f844400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:35.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.626+0000 7f845559a640 1 -- 172.21.15.116:0/2568177072 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f84580c43b0 con 0x7f84440051d0 2024-04-25T00:22:35.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.626+0000 7f845559a640 1 -- 172.21.15.116:0/2568177072 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f845806f7e0 con 0x7f8444004810 2024-04-25T00:22:35.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.626+0000 7f845559a640 1 -- 172.21.15.116:0/2568177072 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f844400c180 con 0x7f84440039c0 2024-04-25T00:22:35.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.626+0000 7f8455d9b640 1 --2- 172.21.15.116:0/2568177072 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f84580c4840 0x7f84580c7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:22:35.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.626+0000 7f845659c640 1 --2- 172.21.15.116:0/2568177072 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f84440039c0 0x7f8444003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:35.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.626+0000 7f8455d9b640 1 --2- 172.21.15.116:0/2568177072 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8444004810 0x7f8444004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:35.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.626+0000 7f8456d9d640 1 --2- 172.21.15.116:0/2568177072 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84440051d0 0x7f844400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:35.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.626+0000 7f845659c640 1 -- 172.21.15.116:0/2568177072 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8444004810 msgr2=0x7f8444004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:35.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.626+0000 7f845659c640 1 --2- 172.21.15.116:0/2568177072 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8444004810 0x7f8444004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:35.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.626+0000 7f845659c640 1 -- 172.21.15.116:0/2568177072 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84440051d0 msgr2=0x7f844400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:35.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.626+0000 7f845659c640 1 --2- 172.21.15.116:0/2568177072 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84440051d0 0x7f844400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:35.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.626+0000 7f845659c640 1 -- 172.21.15.116:0/2568177072 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f845806fac0 con 0x7f84440039c0 2024-04-25T00:22:35.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.630+0000 7f8456d9d640 1 --2- 172.21.15.116:0/2568177072 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84440051d0 0x7f844400b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:22:35.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.630+0000 7f8455d9b640 1 --2- 172.21.15.116:0/2568177072 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8444004810 0x7f8444004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:22:35.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.630+0000 7f845659c640 1 --2- 172.21.15.116:0/2568177072 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f84440039c0 0x7f8444003de0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f845000a8d0 tx=0x7f845001fb80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e834e54230b59e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:35.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.630+0000 7f845559a640 1 -- 172.21.15.116:0/2568177072 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f845000a4a0 con 0x7f84440039c0 2024-04-25T00:22:35.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.630+0000 7f845559a640 1 -- 172.21.15.116:0/2568177072 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84500ad070 con 0x7f84440039c0 2024-04-25T00:22:35.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.630+0000 7f845559a640 1 -- 172.21.15.116:0/2568177072 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84500a9a00 con 0x7f84440039c0 2024-04-25T00:22:35.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.630+0000 7f845cc79640 1 -- 172.21.15.116:0/2568177072 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f84440039c0 msgr2=0x7f8444003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:35.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.630+0000 7f845cc79640 1 --2- 172.21.15.116:0/2568177072 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f84440039c0 0x7f8444003de0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f845000a8d0 tx=0x7f845001fb80 comp rx=0 tx=0).stop 2024-04-25T00:22:35.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.630+0000 7f8456d9d640 1 -- 172.21.15.116:0/2568177072 reap_dead start 2024-04-25T00:22:35.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.630+0000 7f845cc79640 1 -- 172.21.15.116:0/2568177072 shutdown_connections 2024-04-25T00:22:35.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.630+0000 7f845cc79640 1 -- 172.21.15.116:0/2568177072 >> 172.21.15.116:0/2568177072 conn(0x7f84580a94a0 msgr2=0x7f84580a98b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:35.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.634+0000 7f845cc79640 1 -- 172.21.15.116:0/2568177072 shutdown_connections 2024-04-25T00:22:35.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.634+0000 7f845cc79640 1 -- 172.21.15.116:0/2568177072 wait complete. 2024-04-25T00:22:35.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.634+0000 7f845cc79640 1 Processor -- start 2024-04-25T00:22:35.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.634+0000 7f845cc79640 1 -- start start 2024-04-25T00:22:35.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.634+0000 7f845cc79640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84440039c0 0x7f845814f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:35.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.634+0000 7f845cc79640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f84581551f0 0x7f8458153c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:35.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.634+0000 7f845cc79640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8458155d40 0x7f84581561c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:35.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.634+0000 7f845cc79640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8458072910 con 0x7f84440039c0 2024-04-25T00:22:35.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.634+0000 7f845cc79640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f84580c4040 con 0x7f8458155d40 2024-04-25T00:22:35.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.634+0000 7f845cc79640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f845806ecf0 con 0x7f84581551f0 2024-04-25T00:22:35.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.634+0000 7f8456d9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8458155d40 0x7f84581561c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:35.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.634+0000 7f8456d9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8458155d40 0x7f84581561c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:39462/0 (socket says 172.21.15.116:39462) 2024-04-25T00:22:35.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.634+0000 7f8456d9d640 1 -- 172.21.15.116:0/3502113118 learned_addr learned my addr 172.21.15.116:0/3502113118 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:22:35.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.634+0000 7f8456d9d640 1 -- 172.21.15.116:0/3502113118 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f84581551f0 msgr2=0x7f8458153c00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T00:22:35.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.634+0000 7f8456d9d640 1 --2- 172.21.15.116:0/3502113118 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f84581551f0 0x7f8458153c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:35.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.634+0000 7f8456d9d640 1 -- 172.21.15.116:0/3502113118 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84440039c0 msgr2=0x7f845814f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:35.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.634+0000 7f8456d9d640 1 --2- 172.21.15.116:0/3502113118 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84440039c0 0x7f845814f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:35.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.634+0000 7f8456d9d640 1 -- 172.21.15.116:0/3502113118 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f845815bbc0 con 0x7f8458155d40 2024-04-25T00:22:35.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.638+0000 7f8456d9d640 1 --2- 172.21.15.116:0/3502113118 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8458155d40 0x7f84581561c0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f84581576f0 tx=0x7f844000f670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:35.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.638+0000 7f844effd640 1 -- 172.21.15.116:0/3502113118 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8440014660 con 0x7f8458155d40 2024-04-25T00:22:35.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.638+0000 7f844effd640 1 -- 172.21.15.116:0/3502113118 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f844001a070 con 0x7f8458155d40 2024-04-25T00:22:35.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.638+0000 7f844effd640 1 -- 172.21.15.116:0/3502113118 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8440014a00 con 0x7f8458155d40 2024-04-25T00:22:35.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.638+0000 7f845cc79640 1 -- 172.21.15.116:0/3502113118 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84581f0530 con 0x7f8458155d40 2024-04-25T00:22:35.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.638+0000 7f845cc79640 1 -- 172.21.15.116:0/3502113118 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f845815c160 con 0x7f8458155d40 2024-04-25T00:22:35.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.642+0000 7f845cc79640 1 -- 172.21.15.116:0/3502113118 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f845815c360 con 0x7f8458155d40 2024-04-25T00:22:35.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.642+0000 7f844effd640 1 -- 172.21.15.116:0/3502113118 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8440012070 con 0x7f8458155d40 2024-04-25T00:22:35.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.642+0000 7f844effd640 1 --2- 172.21.15.116:0/3502113118 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8438067910 0x7f8438069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:35.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.642+0000 7f844effd640 1 -- 172.21.15.116:0/3502113118 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 9504+0+0 (secure 0 0 0) 0x7f8440086640 con 0x7f8458155d40 2024-04-25T00:22:35.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.642+0000 7f845659c640 1 --2- 172.21.15.116:0/3502113118 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8438067910 0x7f8438069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:35.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.642+0000 7f845659c640 1 --2- 172.21.15.116:0/3502113118 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8438067910 0x7f8438069dd0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f84500b1000 tx=0x7f8450001d00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:35.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.654+0000 7f83c86a4640 1 -- 172.21.15.116:0/2613334654 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f83b800b800 con 0x7f83b8008750 2024-04-25T00:22:35.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.658+0000 7f844effd640 1 -- 172.21.15.116:0/3502113118 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f845815c360 con 0x7f8458155d40 2024-04-25T00:22:35.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.658+0000 7f83b6ffd640 1 -- 172.21.15.116:0/2613334654 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 177 v177) ==== 95+0+6501 (secure 0 0 0) 0x7f83bc016070 con 0x7f83b8008750 2024-04-25T00:22:35.658 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:22:35.658 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":177,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6230,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/492366509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":492366509},{"type":"v1","addr":"172.21.15.174:6837","nonce":492366509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":177}],"filesystems":[{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:22:13.925793+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7989},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7989":{"gid":7989,"name":"c","rank":0,"incarnation":173,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6835/221971311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":221971311},{"type":"v1","addr":"172.21.15.116:6835","nonce":221971311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7989,"qdb_cluster":[7989]},"id":1},{"mdsmap":{"epoch":161,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:20:13.935378+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7722},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7722":{"gid":7722,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6833/663910035","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":663910035},{"type":"v1","addr":"172.21.15.116:6833","nonce":663910035}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7722,"qdb_cluster":[7722]},"id":2}]} 2024-04-25T00:22:35.658 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 177 2024-04-25T00:22:35.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.658+0000 7f83c86a4640 1 -- 172.21.15.116:0/2613334654 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f83a40678c0 msgr2=0x7f83a4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:35.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.658+0000 7f83c86a4640 1 --2- 172.21.15.116:0/2613334654 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f83a40678c0 0x7f83a4069d80 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f83b80da750 tx=0x7f83b0009240 comp rx=0 tx=0).stop 2024-04-25T00:22:35.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.658+0000 7f83c86a4640 1 -- 172.21.15.116:0/2613334654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83b8008750 msgr2=0x7f83b80cc050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:35.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.658+0000 7f83c86a4640 1 --2- 172.21.15.116:0/2613334654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83b8008750 0x7f83b80cc050 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f83bc018f70 tx=0x7f83bc019100 comp rx=0 tx=0).stop 2024-04-25T00:22:35.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.658+0000 7f83c86a4640 1 -- 172.21.15.116:0/2613334654 shutdown_connections 2024-04-25T00:22:35.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.658+0000 7f83c86a4640 1 --2- 172.21.15.116:0/2613334654 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f83a40678c0 0x7f83a4069d80 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:35.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.658+0000 7f83c86a4640 1 --2- 172.21.15.116:0/2613334654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f83b80d8110 0x7f83b817b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:35.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.658+0000 7f83c86a4640 1 --2- 172.21.15.116:0/2613334654 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f83b800b240 0x7f83b80d6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:35.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.658+0000 7f83c86a4640 1 --2- 172.21.15.116:0/2613334654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f83b8008750 0x7f83b80cc050 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:35.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.658+0000 7f83c86a4640 1 -- 172.21.15.116:0/2613334654 >> 172.21.15.116:0/2613334654 conn(0x7f83b8049370 msgr2=0x7f83b80c6ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:35.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.658+0000 7f83c86a4640 1 -- 172.21.15.116:0/2613334654 shutdown_connections 2024-04-25T00:22:35.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:35.658+0000 7f83c86a4640 1 -- 172.21.15.116:0/2613334654 wait complete. 2024-04-25T00:22:35.703 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:22:36.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:36.174+0000 7f845cc79640 1 -- 172.21.15.116:0/3502113118 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8458068a60 con 0x7f8458155d40 2024-04-25T00:22:36.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:36.178+0000 7f844effd640 1 -- 172.21.15.116:0/3502113118 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 178 v178) ==== 95+0+5667 (secure 0 0 0) 0x7f84400562e0 con 0x7f8458155d40 2024-04-25T00:22:36.180 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:22:36.180 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":178,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:22:13.925793+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7989},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7989":{"gid":7989,"name":"c","rank":0,"incarnation":173,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6835/221971311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":221971311},{"type":"v1","addr":"172.21.15.116:6835","nonce":221971311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7989,"qdb_cluster":[7989]},"id":1},{"mdsmap":{"epoch":178,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:22:35.677621+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6230":{"gid":6230,"name":"d","rank":0,"incarnation":178,"state":"up:replay","state_seq":1,"addr":"172.21.15.174:6837/492366509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":492366509},{"type":"v1","addr":"172.21.15.174:6837","nonce":492366509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:22:36.180 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 178 2024-04-25T00:22:36.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:36.182+0000 7f844cff9640 1 -- 172.21.15.116:0/3502113118 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8438067910 msgr2=0x7f8438069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:36.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:36.182+0000 7f844cff9640 1 --2- 172.21.15.116:0/3502113118 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8438067910 0x7f8438069dd0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f84500b1000 tx=0x7f8450001d00 comp rx=0 tx=0).stop 2024-04-25T00:22:36.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:36.182+0000 7f844cff9640 1 -- 172.21.15.116:0/3502113118 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8458155d40 msgr2=0x7f84581561c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:36.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:36.182+0000 7f844cff9640 1 --2- 172.21.15.116:0/3502113118 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8458155d40 0x7f84581561c0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f84581576f0 tx=0x7f844000f670 comp rx=0 tx=0).stop 2024-04-25T00:22:36.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:36.182+0000 7f844cff9640 1 -- 172.21.15.116:0/3502113118 shutdown_connections 2024-04-25T00:22:36.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:36.182+0000 7f844cff9640 1 --2- 172.21.15.116:0/3502113118 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8438067910 0x7f8438069dd0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:36.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:36.182+0000 7f844cff9640 1 --2- 172.21.15.116:0/3502113118 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8458155d40 0x7f84581561c0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:36.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:36.182+0000 7f844cff9640 1 --2- 172.21.15.116:0/3502113118 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f84581551f0 0x7f8458153c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:36.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:36.182+0000 7f844cff9640 1 --2- 172.21.15.116:0/3502113118 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84440039c0 0x7f845814f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:36.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:36.182+0000 7f844cff9640 1 -- 172.21.15.116:0/3502113118 >> 172.21.15.116:0/3502113118 conn(0x7f84580a94a0 msgr2=0x7f84580a98b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:36.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:36.182+0000 7f844cff9640 1 -- 172.21.15.116:0/3502113118 shutdown_connections 2024-04-25T00:22:36.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:36.182+0000 7f844cff9640 1 -- 172.21.15.116:0/3502113118 wait complete. 2024-04-25T00:22:36.221 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:22:37.704 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:22:37.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:37.878+0000 7f4b93070640 1 Processor -- start 2024-04-25T00:22:37.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:37.878+0000 7f4b93070640 1 -- start start 2024-04-25T00:22:37.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:37.878+0000 7f4b93070640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f4b8c14ebf0 0x7f4b8c0c2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:37.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:37.882+0000 7f4b93070640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4b8c06d110 0x7f4b8c15c050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:37.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:37.882+0000 7f4b93070640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b8c070e60 con 0x7f4b8c1516d0 2024-04-25T00:22:37.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:37.882+0000 7f4b93070640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f4b8c070c10 con 0x7f4b8c14ebf0 2024-04-25T00:22:37.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:37.882+0000 7f4b93070640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f4b8c071140 con 0x7f4b8c06d110 2024-04-25T00:22:37.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:37.882+0000 7f4b9286f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4b8c06d110 0x7f4b8c15c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:37.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:37.882+0000 7f4b9186d640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f4b8c1516d0 0x7f4b8c06ca00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:46580/0 (socket says 172.21.15.116:46580) 2024-04-25T00:22:37.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:37.882+0000 7f4b9186d640 1 -- 172.21.15.116:0/3160455400 learned_addr learned my addr 172.21.15.116:0/3160455400 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:22:37.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:37.882+0000 7f4b9206e640 1 --2- 172.21.15.116:0/3160455400 >> v2:172.21.15.174:3300/0 conn(0x7f4b8c14ebf0 0x7f4b8c0c2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:37.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:37.882+0000 7f4b9286f640 1 -- 172.21.15.116:0/3160455400 >> v2:172.21.15.174:3300/0 conn(0x7f4b8c14ebf0 msgr2=0x7f4b8c0c2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:37.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:37.882+0000 7f4b9286f640 1 --2- 172.21.15.116:0/3160455400 >> v2:172.21.15.174:3300/0 conn(0x7f4b8c14ebf0 0x7f4b8c0c2850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:37.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:37.882+0000 7f4b9286f640 1 -- 172.21.15.116:0/3160455400 >> v1:172.21.15.174:6789/0 conn(0x7f4b8c1516d0 legacy=0x7f4b8c06ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:37.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:37.882+0000 7f4b9286f640 1 -- 172.21.15.116:0/3160455400 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b8c071420 con 0x7f4b8c06d110 2024-04-25T00:22:37.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:37.890+0000 7f4b9286f640 1 --2- 172.21.15.116:0/3160455400 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4b8c06d110 0x7f4b8c15c050 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4b84009d90 tx=0x7f4b8401c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6af5c9b53145a347 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:37.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:37.894+0000 7f4b9106c640 1 -- 172.21.15.116:0/3160455400 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b840b2070 con 0x7f4b8c06d110 2024-04-25T00:22:37.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:37.894+0000 7f4b9106c640 1 -- 172.21.15.116:0/3160455400 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b840a5820 con 0x7f4b8c06d110 2024-04-25T00:22:37.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:37.894+0000 7f4b9106c640 1 -- 172.21.15.116:0/3160455400 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b840a5bf0 con 0x7f4b8c06d110 2024-04-25T00:22:37.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:37.894+0000 7f4b93070640 1 -- 172.21.15.116:0/3160455400 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4b8c06d110 msgr2=0x7f4b8c15c050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:37.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:37.894+0000 7f4b93070640 1 --2- 172.21.15.116:0/3160455400 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4b8c06d110 0x7f4b8c15c050 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4b84009d90 tx=0x7f4b8401c800 comp rx=0 tx=0).stop 2024-04-25T00:22:37.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:37.898+0000 7f4b93070640 1 -- 172.21.15.116:0/3160455400 shutdown_connections 2024-04-25T00:22:37.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:37.898+0000 7f4b93070640 1 --2- 172.21.15.116:0/3160455400 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4b8c06d110 0x7f4b8c15c050 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:37.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:37.898+0000 7f4b93070640 1 --2- 172.21.15.116:0/3160455400 >> v2:172.21.15.174:3300/0 conn(0x7f4b8c14ebf0 0x7f4b8c0c2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:37.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:37.898+0000 7f4b93070640 1 -- 172.21.15.116:0/3160455400 >> 172.21.15.116:0/3160455400 conn(0x7f4b8c0c3940 msgr2=0x7f4b8c0c3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:37.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:37.902+0000 7f4b93070640 1 -- 172.21.15.116:0/3160455400 shutdown_connections 2024-04-25T00:22:37.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:37.902+0000 7f4b93070640 1 -- 172.21.15.116:0/3160455400 wait complete. 2024-04-25T00:22:37.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:37.906+0000 7f4b93070640 1 Processor -- start 2024-04-25T00:22:37.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:37.906+0000 7f4b93070640 1 -- start start 2024-04-25T00:22:37.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:37.906+0000 7f4b93070640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4b8c06d110 0x7f4b8c0c51a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:37.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:37.906+0000 7f4b93070640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4b8c14ebf0 0x7f4b8c145060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:37.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:37.906+0000 7f4b93070640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b8c1516d0 0x7f4b8c1455a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:37.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:37.906+0000 7f4b93070640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4b8c14efd0 con 0x7f4b8c1516d0 2024-04-25T00:22:37.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:37.906+0000 7f4b93070640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4b8c072080 con 0x7f4b8c06d110 2024-04-25T00:22:37.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:37.906+0000 7f4b93070640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f4b8c14f150 con 0x7f4b8c14ebf0 2024-04-25T00:22:37.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:37.906+0000 7f4b9206e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4b8c06d110 0x7f4b8c0c51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:37.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:37.906+0000 7f4b9206e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4b8c06d110 0x7f4b8c0c51a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:39482/0 (socket says 172.21.15.116:39482) 2024-04-25T00:22:37.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:37.906+0000 7f4b9286f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b8c1516d0 0x7f4b8c1455a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:37.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:37.906+0000 7f4b9206e640 1 -- 172.21.15.116:0/3292557685 learned_addr learned my addr 172.21.15.116:0/3292557685 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:22:37.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:37.906+0000 7f4b9286f640 1 -- 172.21.15.116:0/3292557685 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4b8c14ebf0 msgr2=0x7f4b8c145060 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T00:22:37.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:37.906+0000 7f4b9286f640 1 --2- 172.21.15.116:0/3292557685 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4b8c14ebf0 0x7f4b8c145060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:37.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:37.906+0000 7f4b9286f640 1 -- 172.21.15.116:0/3292557685 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4b8c06d110 msgr2=0x7f4b8c0c51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:37.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:37.906+0000 7f4b9286f640 1 --2- 172.21.15.116:0/3292557685 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4b8c06d110 0x7f4b8c0c51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:37.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:37.906+0000 7f4b9286f640 1 -- 172.21.15.116:0/3292557685 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b8c145e60 con 0x7f4b8c1516d0 2024-04-25T00:22:37.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:37.906+0000 7f4b9206e640 1 --2- 172.21.15.116:0/3292557685 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4b8c06d110 0x7f4b8c0c51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:22:37.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:37.906+0000 7f4b9286f640 1 --2- 172.21.15.116:0/3292557685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b8c1516d0 0x7f4b8c1455a0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f4b840ac000 tx=0x7f4b840195a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:37.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:37.906+0000 7f4b82ffd640 1 -- 172.21.15.116:0/3292557685 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b840bd020 con 0x7f4b8c1516d0 2024-04-25T00:22:37.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:37.906+0000 7f4b82ffd640 1 -- 172.21.15.116:0/3292557685 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b840aec80 con 0x7f4b8c1516d0 2024-04-25T00:22:37.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:37.906+0000 7f4b82ffd640 1 -- 172.21.15.116:0/3292557685 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b840b2040 con 0x7f4b8c1516d0 2024-04-25T00:22:37.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:37.906+0000 7f4b93070640 1 -- 172.21.15.116:0/3292557685 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b8c15ebc0 con 0x7f4b8c1516d0 2024-04-25T00:22:37.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:37.910+0000 7f4b82ffd640 1 -- 172.21.15.116:0/3292557685 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4b840c0020 con 0x7f4b8c1516d0 2024-04-25T00:22:37.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:37.910+0000 7f4b93070640 1 -- 172.21.15.116:0/3292557685 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b8c0c59c0 con 0x7f4b8c1516d0 2024-04-25T00:22:37.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:37.910+0000 7f4b82ffd640 1 --2- 172.21.15.116:0/3292557685 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4b70067910 0x7f4b70069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:37.913 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:37.910+0000 7f4b93070640 1 -- 172.21.15.116:0/3292557685 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b8c15ed90 con 0x7f4b8c1516d0 2024-04-25T00:22:37.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:37.918+0000 7f4b9206e640 1 --2- 172.21.15.116:0/3292557685 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4b70067910 0x7f4b70069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:37.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:37.918+0000 7f4b9206e640 1 --2- 172.21.15.116:0/3292557685 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4b70067910 0x7f4b70069dd0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c00af80 tx=0x7f4b7c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:37.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:37.922+0000 7f4b82ffd640 1 -- 172.21.15.116:0/3292557685 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9590+0+0 (secure 0 0 0) 0x7f4b840eef80 con 0x7f4b8c1516d0 2024-04-25T00:22:37.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:37.930+0000 7f4b82ffd640 1 -- 172.21.15.116:0/3292557685 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f4b84016480 con 0x7f4b8c1516d0 2024-04-25T00:22:38.222 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:22:38.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.306+0000 7f4b93070640 1 -- 172.21.15.116:0/3292557685 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4b8c04f510 con 0x7f4b8c1516d0 2024-04-25T00:22:38.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.306+0000 7f4b82ffd640 1 -- 172.21.15.116:0/3292557685 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 180 v180) ==== 95+0+5667 (secure 0 0 0) 0x7f4b840f15e0 con 0x7f4b8c1516d0 2024-04-25T00:22:38.309 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:22:38.309 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":180,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:22:13.925793+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7989},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7989":{"gid":7989,"name":"c","rank":0,"incarnation":173,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6835/221971311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":221971311},{"type":"v1","addr":"172.21.15.116:6835","nonce":221971311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7989,"qdb_cluster":[7989]},"id":1},{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:22:36.913581+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6230":{"gid":6230,"name":"d","rank":0,"incarnation":178,"state":"up:rejoin","state_seq":6,"addr":"172.21.15.174:6837/492366509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":492366509},{"type":"v1","addr":"172.21.15.174:6837","nonce":492366509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:22:38.310 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 180 2024-04-25T00:22:38.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.310+0000 7f4b93070640 1 -- 172.21.15.116:0/3292557685 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4b70067910 msgr2=0x7f4b70069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:38.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.310+0000 7f4b93070640 1 --2- 172.21.15.116:0/3292557685 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4b70067910 0x7f4b70069dd0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c00af80 tx=0x7f4b7c008040 comp rx=0 tx=0).stop 2024-04-25T00:22:38.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.310+0000 7f4b93070640 1 -- 172.21.15.116:0/3292557685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b8c1516d0 msgr2=0x7f4b8c1455a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:38.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.310+0000 7f4b93070640 1 --2- 172.21.15.116:0/3292557685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b8c1516d0 0x7f4b8c1455a0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f4b840ac000 tx=0x7f4b840195a0 comp rx=0 tx=0).stop 2024-04-25T00:22:38.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.310+0000 7f4b93070640 1 -- 172.21.15.116:0/3292557685 shutdown_connections 2024-04-25T00:22:38.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.310+0000 7f4b93070640 1 --2- 172.21.15.116:0/3292557685 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4b70067910 0x7f4b70069dd0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:38.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.310+0000 7f4b93070640 1 --2- 172.21.15.116:0/3292557685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4b8c1516d0 0x7f4b8c1455a0 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:38.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.310+0000 7f4b93070640 1 --2- 172.21.15.116:0/3292557685 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4b8c14ebf0 0x7f4b8c145060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:38.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.310+0000 7f4b93070640 1 --2- 172.21.15.116:0/3292557685 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4b8c06d110 0x7f4b8c0c51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:38.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.310+0000 7f4b93070640 1 -- 172.21.15.116:0/3292557685 >> 172.21.15.116:0/3292557685 conn(0x7f4b8c0c3940 msgr2=0x7f4b8c14d340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:38.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.314+0000 7f4b93070640 1 -- 172.21.15.116:0/3292557685 shutdown_connections 2024-04-25T00:22:38.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.314+0000 7f4b93070640 1 -- 172.21.15.116:0/3292557685 wait complete. 2024-04-25T00:22:38.346 INFO:tasks.mds_thrash.fs.[b]:mds.a down, removed from mdsmap 2024-04-25T00:22:38.347 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-04-25T00:22:38.347 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:22:38.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.418+0000 7f4a34ae6640 1 Processor -- start 2024-04-25T00:22:38.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.418+0000 7f4a34ae6640 1 -- start start 2024-04-25T00:22:38.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.418+0000 7f4a34ae6640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4a3006dfc0 0x7f4a301559b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:38.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.418+0000 7f4a34ae6640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a300c40c0 con 0x7f4a3014c660 2024-04-25T00:22:38.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.418+0000 7f4a34ae6640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a300c3300 con 0x7f4a3014c280 2024-04-25T00:22:38.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.418+0000 7f4a34ae6640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f4a30156600 con 0x7f4a3006dfc0 2024-04-25T00:22:38.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.418+0000 7f4a2ffff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f4a3014c660 0x7f4a30155ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:43886/0 (socket says 172.21.15.116:43886) 2024-04-25T00:22:38.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.418+0000 7f4a2ffff640 1 -- 172.21.15.116:0/2261924342 learned_addr learned my addr 172.21.15.116:0/2261924342 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:22:38.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.422+0000 7f4a2e7fc640 1 -- 172.21.15.116:0/2261924342 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2202087151 0 0) 0x7f4a300c40c0 con 0x7f4a3014c660 2024-04-25T00:22:38.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.422+0000 7f4a2e7fc640 1 -- 172.21.15.116:0/2261924342 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a18003530 con 0x7f4a3014c660 2024-04-25T00:22:38.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.422+0000 7f4a2e7fc640 1 -- 172.21.15.116:0/2261924342 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3062962473 0 0) 0x7f4a18003530 con 0x7f4a3014c660 2024-04-25T00:22:38.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.422+0000 7f4a2e7fc640 1 -- 172.21.15.116:0/2261924342 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4a3006dfc0 msgr2=0x7f4a301559b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:38.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.422+0000 7f4a2e7fc640 1 --2- 172.21.15.116:0/2261924342 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4a3006dfc0 0x7f4a301559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:38.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.422+0000 7f4a2e7fc640 1 -- 172.21.15.116:0/2261924342 >> v1:172.21.15.174:6789/0 conn(0x7f4a3014c280 legacy=0x7f4a3006ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:38.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.422+0000 7f4a2e7fc640 1 -- 172.21.15.116:0/2261924342 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a300c35e0 con 0x7f4a3014c660 2024-04-25T00:22:38.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.422+0000 7f4a2e7fc640 1 -- 172.21.15.116:0/2261924342 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f4a280019e0 con 0x7f4a3014c660 2024-04-25T00:22:38.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.422+0000 7f4a2e7fc640 1 -- 172.21.15.116:0/2261924342 >> v1:172.21.15.116:6789/0 conn(0x7f4a3014c660 legacy=0x7f4a30155ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:38.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.422+0000 7f4a2e7fc640 1 --2- 172.21.15.116:0/2261924342 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4a180039f0 0x7f4a18003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:38.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.422+0000 7f4a2e7fc640 1 --2- 172.21.15.116:0/2261924342 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4a18004810 0x7f4a18004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:38.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.422+0000 7f4a2e7fc640 1 --2- 172.21.15.116:0/2261924342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a180052f0 0x7f4a1800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:38.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.422+0000 7f4a2e7fc640 1 -- 172.21.15.116:0/2261924342 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4a18014990 con 0x7f4a180052f0 2024-04-25T00:22:38.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.422+0000 7f4a2e7fc640 1 -- 172.21.15.116:0/2261924342 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4a18014b40 con 0x7f4a18004810 2024-04-25T00:22:38.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.422+0000 7f4a2e7fc640 1 -- 172.21.15.116:0/2261924342 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f4a18014d40 con 0x7f4a180039f0 2024-04-25T00:22:38.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.422+0000 7f4a2effd640 1 --2- 172.21.15.116:0/2261924342 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4a18004810 0x7f4a18004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:38.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.422+0000 7f4a2effd640 1 -- 172.21.15.116:0/2261924342 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4a180039f0 msgr2=0x7f4a18003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:38.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.422+0000 7f4a2effd640 1 --2- 172.21.15.116:0/2261924342 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4a180039f0 0x7f4a18003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:38.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.422+0000 7f4a2effd640 1 -- 172.21.15.116:0/2261924342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a180052f0 msgr2=0x7f4a1800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:38.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.422+0000 7f4a2ffff640 1 --2- 172.21.15.116:0/2261924342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a180052f0 0x7f4a1800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:38.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.422+0000 7f4a2effd640 1 --2- 172.21.15.116:0/2261924342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a180052f0 0x7f4a1800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:38.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.422+0000 7f4a2effd640 1 -- 172.21.15.116:0/2261924342 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a300c35e0 con 0x7f4a18004810 2024-04-25T00:22:38.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.422+0000 7f4a2ffff640 1 --2- 172.21.15.116:0/2261924342 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a180052f0 0x7f4a1800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:22:38.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.426+0000 7f4a2effd640 1 --2- 172.21.15.116:0/2261924342 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4a18004810 0x7f4a18004c90 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f4a300c26a0 tx=0x7f4a2401c240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=628ac963812dce96 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:38.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.426+0000 7f4a2e7fc640 1 -- 172.21.15.116:0/2261924342 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a240b1070 con 0x7f4a18004810 2024-04-25T00:22:38.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.426+0000 7f4a2e7fc640 1 -- 172.21.15.116:0/2261924342 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a240ac440 con 0x7f4a18004810 2024-04-25T00:22:38.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.430+0000 7f4a34ae6640 1 -- 172.21.15.116:0/2261924342 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4a18004810 msgr2=0x7f4a18004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:38.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.430+0000 7f4a34ae6640 1 --2- 172.21.15.116:0/2261924342 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4a18004810 0x7f4a18004c90 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f4a300c26a0 tx=0x7f4a2401c240 comp rx=0 tx=0).stop 2024-04-25T00:22:38.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.430+0000 7f4a2ffff640 1 -- 172.21.15.116:0/2261924342 reap_dead start 2024-04-25T00:22:38.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.430+0000 7f4a34ae6640 1 -- 172.21.15.116:0/2261924342 shutdown_connections 2024-04-25T00:22:38.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.430+0000 7f4a34ae6640 1 -- 172.21.15.116:0/2261924342 >> 172.21.15.116:0/2261924342 conn(0x7f4a300c4950 msgr2=0x7f4a300c4d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:38.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.430+0000 7f4a34ae6640 1 -- 172.21.15.116:0/2261924342 shutdown_connections 2024-04-25T00:22:38.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.430+0000 7f4a34ae6640 1 -- 172.21.15.116:0/2261924342 wait complete. 2024-04-25T00:22:38.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.430+0000 7f4a34ae6640 1 Processor -- start 2024-04-25T00:22:38.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.430+0000 7f4a34ae6640 1 -- start start 2024-04-25T00:22:38.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.434+0000 7f4a34ae6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a18004810 0x7f4a3020c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:38.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.434+0000 7f4a34ae6640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4a30209260 0x7f4a300acea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:38.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.434+0000 7f4a34ae6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4a302096e0 0x7f4a3020fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:38.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.434+0000 7f4a34ae6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4a30155d40 con 0x7f4a18004810 2024-04-25T00:22:38.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.434+0000 7f4a34ae6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4a30155bc0 con 0x7f4a302096e0 2024-04-25T00:22:38.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.434+0000 7f4a34ae6640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f4a3006ba20 con 0x7f4a30209260 2024-04-25T00:22:38.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.434+0000 7f4a2ffff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4a302096e0 0x7f4a3020fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:38.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.434+0000 7f4a2effd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4a30209260 0x7f4a300acea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:38.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.434+0000 7f4a2effd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4a30209260 0x7f4a300acea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:32818/0 (socket says 172.21.15.116:32818) 2024-04-25T00:22:38.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.434+0000 7f4a2effd640 1 -- 172.21.15.116:0/3425249898 learned_addr learned my addr 172.21.15.116:0/3425249898 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:22:38.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.434+0000 7f4a2f7fe640 1 --2- 172.21.15.116:0/3425249898 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a18004810 0x7f4a3020c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:38.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.434+0000 7f4a2effd640 1 -- 172.21.15.116:0/3425249898 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4a302096e0 msgr2=0x7f4a3020fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:38.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.434+0000 7f4a2effd640 1 --2- 172.21.15.116:0/3425249898 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4a302096e0 0x7f4a3020fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:38.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.434+0000 7f4a2effd640 1 -- 172.21.15.116:0/3425249898 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a18004810 msgr2=0x7f4a3020c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:38.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.434+0000 7f4a2effd640 1 --2- 172.21.15.116:0/3425249898 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a18004810 0x7f4a3020c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:38.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.434+0000 7f4a2effd640 1 -- 172.21.15.116:0/3425249898 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a3020ce80 con 0x7f4a30209260 2024-04-25T00:22:38.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.434+0000 7f4a2effd640 1 --2- 172.21.15.116:0/3425249898 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4a30209260 0x7f4a300acea0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4a2401c720 tx=0x7f4a18005840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:38.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.434+0000 7f4a2f7fe640 1 --2- 172.21.15.116:0/3425249898 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a18004810 0x7f4a3020c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:22:38.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.434+0000 7f4a17fff640 1 -- 172.21.15.116:0/3425249898 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a240b1040 con 0x7f4a30209260 2024-04-25T00:22:38.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.434+0000 7f4a17fff640 1 -- 172.21.15.116:0/3425249898 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a240b6520 con 0x7f4a30209260 2024-04-25T00:22:38.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.434+0000 7f4a17fff640 1 -- 172.21.15.116:0/3425249898 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a240b68d0 con 0x7f4a30209260 2024-04-25T00:22:38.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.434+0000 7f4a34ae6640 1 -- 172.21.15.116:0/3425249898 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a3014fe00 con 0x7f4a30209260 2024-04-25T00:22:38.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.434+0000 7f4a34ae6640 1 -- 172.21.15.116:0/3425249898 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a3014f840 con 0x7f4a30209260 2024-04-25T00:22:38.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.438+0000 7f4a17fff640 1 -- 172.21.15.116:0/3425249898 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4a240c0020 con 0x7f4a30209260 2024-04-25T00:22:38.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.438+0000 7f4a17fff640 1 --2- 172.21.15.116:0/3425249898 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4a1c0678c0 0x7f4a1c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:38.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.442+0000 7f4a17fff640 1 -- 172.21.15.116:0/3425249898 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(62..62 src has 1..62) ==== 9590+0+0 (secure 0 0 0) 0x7f4a241209b0 con 0x7f4a30209260 2024-04-25T00:22:38.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.442+0000 7f4a2f7fe640 1 --2- 172.21.15.116:0/3425249898 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4a1c0678c0 0x7f4a1c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:38.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.442+0000 7f4a2f7fe640 1 --2- 172.21.15.116:0/3425249898 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4a1c0678c0 0x7f4a1c069d80 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f4a20000c10 tx=0x7f4a20009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:38.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.446+0000 7f4a15ffb640 1 -- 172.21.15.116:0/3425249898 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a3004f510 con 0x7f4a30209260 2024-04-25T00:22:38.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.454+0000 7f4a17fff640 1 -- 172.21.15.116:0/3425249898 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f4a3020ce80 con 0x7f4a30209260 2024-04-25T00:22:38.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.538+0000 7f5c38eca640 1 Processor -- start 2024-04-25T00:22:38.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.538+0000 7f5c38eca640 1 -- start start 2024-04-25T00:22:38.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.538+0000 7f5c38eca640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5c2c0094c0 0x7f5c2c0df270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:38.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.542+0000 7f5c38eca640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c2c0e23a0 con 0x7f5c2c0dbe40 2024-04-25T00:22:38.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.542+0000 7f5c337fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f5c2c0dbe40 0x7f5c2c0e1c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:43898/0 (socket says 172.21.15.116:43898) 2024-04-25T00:22:38.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.542+0000 7f5c337fe640 1 -- 172.21.15.116:0/3271788160 learned_addr learned my addr 172.21.15.116:0/3271788160 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:22:38.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.542+0000 7f5c32ffd640 1 --2- 172.21.15.116:0/3271788160 >> v2:172.21.15.116:3300/0 conn(0x7f5c2c0094c0 0x7f5c2c0df270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:38.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.542+0000 7f5c38eca640 1 -- 172.21.15.116:0/3271788160 --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c2c00d0f0 con 0x7f5c2c00d8d0 2024-04-25T00:22:38.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.542+0000 7f5c38eca640 1 -- 172.21.15.116:0/3271788160 --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f5c2c00dd00 con 0x7f5c2c0094c0 2024-04-25T00:22:38.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.542+0000 7f5c327fc640 1 -- 172.21.15.116:0/3271788160 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 188539290 0 0) 0x7f5c2c0e23a0 con 0x7f5c2c0dbe40 2024-04-25T00:22:38.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.542+0000 7f5c327fc640 1 -- 172.21.15.116:0/3271788160 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c24003530 con 0x7f5c2c0dbe40 2024-04-25T00:22:38.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.542+0000 7f5c327fc640 1 -- 172.21.15.116:0/3271788160 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3768073443 0 0) 0x7f5c24003530 con 0x7f5c2c0dbe40 2024-04-25T00:22:38.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.542+0000 7f5c327fc640 1 -- 172.21.15.116:0/3271788160 >> v2:172.21.15.116:3300/0 conn(0x7f5c2c0094c0 msgr2=0x7f5c2c0df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:38.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.542+0000 7f5c327fc640 1 --2- 172.21.15.116:0/3271788160 >> v2:172.21.15.116:3300/0 conn(0x7f5c2c0094c0 0x7f5c2c0df270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:38.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.542+0000 7f5c327fc640 1 -- 172.21.15.116:0/3271788160 >> v1:172.21.15.174:6789/0 conn(0x7f5c2c00d8d0 legacy=0x7f5c2c040a00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T00:22:38.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.542+0000 7f5c327fc640 1 -- 172.21.15.116:0/3271788160 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c2c00d3d0 con 0x7f5c2c0dbe40 2024-04-25T00:22:38.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.542+0000 7f5c327fc640 1 -- 172.21.15.116:0/3271788160 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f5c28001800 con 0x7f5c2c0dbe40 2024-04-25T00:22:38.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.542+0000 7f5c327fc640 1 -- 172.21.15.116:0/3271788160 >> v1:172.21.15.116:6789/0 conn(0x7f5c2c0dbe40 legacy=0x7f5c2c0e1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:38.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.542+0000 7f5c327fc640 1 --2- 172.21.15.116:0/3271788160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c24003a10 0x7f5c24003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:38.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.542+0000 7f5c327fc640 1 --2- 172.21.15.116:0/3271788160 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5c24004810 0x7f5c24004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:38.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.542+0000 7f5c327fc640 1 --2- 172.21.15.116:0/3271788160 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5c240052f0 0x7f5c2400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:38.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.542+0000 7f5c327fc640 1 -- 172.21.15.116:0/3271788160 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5c2c00dd00 con 0x7f5c24003a10 2024-04-25T00:22:38.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.542+0000 7f5c327fc640 1 -- 172.21.15.116:0/3271788160 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5c2400c1f0 con 0x7f5c24004810 2024-04-25T00:22:38.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.542+0000 7f5c327fc640 1 -- 172.21.15.116:0/3271788160 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f5c2400c3f0 con 0x7f5c240052f0 2024-04-25T00:22:38.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.542+0000 7f5c337fe640 1 --2- 172.21.15.116:0/3271788160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c24003a10 0x7f5c24003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:38.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.542+0000 7f5c32ffd640 1 --2- 172.21.15.116:0/3271788160 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5c24004810 0x7f5c24004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:38.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.542+0000 7f5c337fe640 1 -- 172.21.15.116:0/3271788160 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5c240052f0 msgr2=0x7f5c2400b9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T00:22:38.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.542+0000 7f5c337fe640 1 --2- 172.21.15.116:0/3271788160 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5c240052f0 0x7f5c2400b9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:38.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.542+0000 7f5c337fe640 1 -- 172.21.15.116:0/3271788160 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5c24004810 msgr2=0x7f5c24004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:38.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.542+0000 7f5c337fe640 1 --2- 172.21.15.116:0/3271788160 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5c24004810 0x7f5c24004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:38.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.542+0000 7f5c337fe640 1 -- 172.21.15.116:0/3271788160 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c24003530 con 0x7f5c24003a10 2024-04-25T00:22:38.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.542+0000 7f5c32ffd640 1 --2- 172.21.15.116:0/3271788160 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5c24004810 0x7f5c24004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:22:38.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.542+0000 7f5c33fff640 1 -- 172.21.15.116:0/3271788160 reap_dead start 2024-04-25T00:22:38.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.542+0000 7f5c337fe640 1 --2- 172.21.15.116:0/3271788160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c24003a10 0x7f5c24003e10 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f5c2800c160 tx=0x7f5c2801cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c15370ad14ca6fcd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:38.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.546+0000 7f5c327fc640 1 -- 172.21.15.116:0/3271788160 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c28007c80 con 0x7f5c24003a10 2024-04-25T00:22:38.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.546+0000 7f5c327fc640 1 -- 172.21.15.116:0/3271788160 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c280a9070 con 0x7f5c24003a10 2024-04-25T00:22:38.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.546+0000 7f5c327fc640 1 -- 172.21.15.116:0/3271788160 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c280074f0 con 0x7f5c24003a10 2024-04-25T00:22:38.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.546+0000 7f5c38eca640 1 -- 172.21.15.116:0/3271788160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c24003a10 msgr2=0x7f5c24003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:38.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.546+0000 7f5c38eca640 1 --2- 172.21.15.116:0/3271788160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c24003a10 0x7f5c24003e10 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f5c2800c160 tx=0x7f5c2801cbc0 comp rx=0 tx=0).stop 2024-04-25T00:22:38.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.546+0000 7f5c38eca640 1 -- 172.21.15.116:0/3271788160 shutdown_connections 2024-04-25T00:22:38.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.546+0000 7f5c38eca640 1 --2- 172.21.15.116:0/3271788160 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c24003a10 0x7f5c24003e10 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:38.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.546+0000 7f5c38eca640 1 -- 172.21.15.116:0/3271788160 >> 172.21.15.116:0/3271788160 conn(0x7f5c2c02f5e0 msgr2=0x7f5c2c02f9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:38.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.546+0000 7f5c38eca640 1 -- 172.21.15.116:0/3271788160 shutdown_connections 2024-04-25T00:22:38.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.546+0000 7f5c38eca640 1 -- 172.21.15.116:0/3271788160 wait complete. 2024-04-25T00:22:38.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.550+0000 7f5c38eca640 1 Processor -- start 2024-04-25T00:22:38.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.550+0000 7f5c38eca640 1 -- start start 2024-04-25T00:22:38.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.550+0000 7f5c38eca640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5c24003a10 0x7f5c2c042190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:38.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.550+0000 7f5c38eca640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c2c0426d0 0x7f5c2c0daf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:38.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.550+0000 7f5c38eca640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5c2c046fe0 0x7f5c2c0db4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:38.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.550+0000 7f5c38eca640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5c2c0e20a0 con 0x7f5c2c0426d0 2024-04-25T00:22:38.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.550+0000 7f5c38eca640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5c2c0e1f20 con 0x7f5c2c046fe0 2024-04-25T00:22:38.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.550+0000 7f5c38eca640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f5c2c0e2220 con 0x7f5c24003a10 2024-04-25T00:22:38.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.550+0000 7f5c32ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c2c0426d0 0x7f5c2c0daf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:38.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.550+0000 7f5c33fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5c2c046fe0 0x7f5c2c0db4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:38.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.550+0000 7f5c32ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c2c0426d0 0x7f5c2c0daf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37320/0 (socket says 172.21.15.116:37320) 2024-04-25T00:22:38.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.550+0000 7f5c32ffd640 1 -- 172.21.15.116:0/3701040020 learned_addr learned my addr 172.21.15.116:0/3701040020 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:22:38.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.550+0000 7f5c337fe640 1 --2- 172.21.15.116:0/3701040020 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5c24003a10 0x7f5c2c042190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:38.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.550+0000 7f5c32ffd640 1 -- 172.21.15.116:0/3701040020 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5c24003a10 msgr2=0x7f5c2c042190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:38.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.550+0000 7f5c32ffd640 1 --2- 172.21.15.116:0/3701040020 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5c24003a10 0x7f5c2c042190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:38.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.550+0000 7f5c32ffd640 1 -- 172.21.15.116:0/3701040020 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5c2c046fe0 msgr2=0x7f5c2c0db4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:38.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.550+0000 7f5c32ffd640 1 --2- 172.21.15.116:0/3701040020 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5c2c046fe0 0x7f5c2c0db4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:38.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.550+0000 7f5c32ffd640 1 -- 172.21.15.116:0/3701040020 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c2c030870 con 0x7f5c2c0426d0 2024-04-25T00:22:38.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.550+0000 7f5c33fff640 1 --2- 172.21.15.116:0/3701040020 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5c2c046fe0 0x7f5c2c0db4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:22:38.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.550+0000 7f5c337fe640 1 --2- 172.21.15.116:0/3701040020 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5c24003a10 0x7f5c2c042190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:22:38.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.550+0000 7f5c32ffd640 1 --2- 172.21.15.116:0/3701040020 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c2c0426d0 0x7f5c2c0daf70 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f5c20010980 tx=0x7f5c20010e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:38.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.550+0000 7f5c1ffff640 1 -- 172.21.15.116:0/3701040020 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c2000f040 con 0x7f5c2c0426d0 2024-04-25T00:22:38.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.550+0000 7f5c1ffff640 1 -- 172.21.15.116:0/3701040020 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c2000ec20 con 0x7f5c2c0426d0 2024-04-25T00:22:38.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.550+0000 7f5c1ffff640 1 -- 172.21.15.116:0/3701040020 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c20004020 con 0x7f5c2c0426d0 2024-04-25T00:22:38.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.550+0000 7f5c38eca640 1 -- 172.21.15.116:0/3701040020 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c2c030b50 con 0x7f5c2c0426d0 2024-04-25T00:22:38.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.550+0000 7f5c38eca640 1 -- 172.21.15.116:0/3701040020 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c2c0476a0 con 0x7f5c2c0426d0 2024-04-25T00:22:38.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.554+0000 7f5c38eca640 1 -- 172.21.15.116:0/3701040020 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c2c0057f0 con 0x7f5c2c0426d0 2024-04-25T00:22:38.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.554+0000 7f5c1ffff640 1 -- 172.21.15.116:0/3701040020 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5c200041c0 con 0x7f5c2c0426d0 2024-04-25T00:22:38.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.554+0000 7f5c1ffff640 1 --2- 172.21.15.116:0/3701040020 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5c180678f0 0x7f5c18069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:38.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.554+0000 7f5c1ffff640 1 -- 172.21.15.116:0/3701040020 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9590+0+0 (secure 0 0 0) 0x7f5c20086c60 con 0x7f5c2c0426d0 2024-04-25T00:22:38.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.554+0000 7f5c337fe640 1 --2- 172.21.15.116:0/3701040020 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5c180678f0 0x7f5c18069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:38.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.554+0000 7f5c337fe640 1 --2- 172.21.15.116:0/3701040020 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5c180678f0 0x7f5c18069db0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f5c280ab000 tx=0x7f5c280a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:38.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.558+0000 7f5c1ffff640 1 -- 172.21.15.116:0/3701040020 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f5c20018030 con 0x7f5c2c0426d0 2024-04-25T00:22:38.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.882+0000 7f5c38eca640 1 -- 172.21.15.116:0/3701040020 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5c2c00ce30 con 0x7f5c2c0426d0 2024-04-25T00:22:38.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.882+0000 7f5c1ffff640 1 -- 172.21.15.116:0/3701040020 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 180 v180) ==== 95+0+5667 (secure 0 0 0) 0x7f5c200044b0 con 0x7f5c2c0426d0 2024-04-25T00:22:38.882 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:22:38.883 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":180,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:22:13.925793+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7989},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7989":{"gid":7989,"name":"c","rank":0,"incarnation":173,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6835/221971311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":221971311},{"type":"v1","addr":"172.21.15.116:6835","nonce":221971311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7989,"qdb_cluster":[7989]},"id":1},{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:22:36.913581+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6230":{"gid":6230,"name":"d","rank":0,"incarnation":178,"state":"up:rejoin","state_seq":6,"addr":"172.21.15.174:6837/492366509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":492366509},{"type":"v1","addr":"172.21.15.174:6837","nonce":492366509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:22:38.883 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 180 2024-04-25T00:22:38.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.882+0000 7f4a15ffb640 1 -- 172.21.15.116:0/3425249898 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4a301531b0 con 0x7f4a30209260 2024-04-25T00:22:38.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.886+0000 7f4a17fff640 1 -- 172.21.15.116:0/3425249898 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 180 v180) ==== 95+0+5667 (secure 0 0 0) 0x7f4a24015f50 con 0x7f4a30209260 2024-04-25T00:22:38.886 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:22:38.887 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":180,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:22:13.925793+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7989},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7989":{"gid":7989,"name":"c","rank":0,"incarnation":173,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6835/221971311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":221971311},{"type":"v1","addr":"172.21.15.116:6835","nonce":221971311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7989,"qdb_cluster":[7989]},"id":1},{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:22:36.913581+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6230":{"gid":6230,"name":"d","rank":0,"incarnation":178,"state":"up:rejoin","state_seq":6,"addr":"172.21.15.174:6837/492366509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":492366509},{"type":"v1","addr":"172.21.15.174:6837","nonce":492366509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:22:38.887 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 180 2024-04-25T00:22:38.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.886+0000 7f5c38eca640 1 -- 172.21.15.116:0/3701040020 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5c180678f0 msgr2=0x7f5c18069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:38.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.886+0000 7f5c38eca640 1 --2- 172.21.15.116:0/3701040020 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5c180678f0 0x7f5c18069db0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f5c280ab000 tx=0x7f5c280a7070 comp rx=0 tx=0).stop 2024-04-25T00:22:38.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.886+0000 7f5c38eca640 1 -- 172.21.15.116:0/3701040020 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c2c0426d0 msgr2=0x7f5c2c0daf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:38.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.886+0000 7f5c38eca640 1 --2- 172.21.15.116:0/3701040020 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c2c0426d0 0x7f5c2c0daf70 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f5c20010980 tx=0x7f5c20010e40 comp rx=0 tx=0).stop 2024-04-25T00:22:38.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.890+0000 7f5c38eca640 1 -- 172.21.15.116:0/3701040020 shutdown_connections 2024-04-25T00:22:38.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.890+0000 7f5c38eca640 1 --2- 172.21.15.116:0/3701040020 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5c180678f0 0x7f5c18069db0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:38.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.890+0000 7f5c38eca640 1 --2- 172.21.15.116:0/3701040020 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5c2c046fe0 0x7f5c2c0db4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:38.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.890+0000 7f5c38eca640 1 --2- 172.21.15.116:0/3701040020 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c2c0426d0 0x7f5c2c0daf70 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:38.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.890+0000 7f5c38eca640 1 --2- 172.21.15.116:0/3701040020 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5c24003a10 0x7f5c2c042190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:38.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.890+0000 7f5c38eca640 1 -- 172.21.15.116:0/3701040020 >> 172.21.15.116:0/3701040020 conn(0x7f5c2c02f5e0 msgr2=0x7f5c2c0087d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:38.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.890+0000 7f4a15ffb640 1 -- 172.21.15.116:0/3425249898 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4a1c0678c0 msgr2=0x7f4a1c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:38.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.890+0000 7f4a15ffb640 1 --2- 172.21.15.116:0/3425249898 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4a1c0678c0 0x7f4a1c069d80 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f4a20000c10 tx=0x7f4a20009040 comp rx=0 tx=0).stop 2024-04-25T00:22:38.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.890+0000 7f4a15ffb640 1 -- 172.21.15.116:0/3425249898 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4a30209260 msgr2=0x7f4a300acea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:38.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.890+0000 7f4a15ffb640 1 --2- 172.21.15.116:0/3425249898 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4a30209260 0x7f4a300acea0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4a2401c720 tx=0x7f4a18005840 comp rx=0 tx=0).stop 2024-04-25T00:22:38.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.890+0000 7f4a15ffb640 1 -- 172.21.15.116:0/3425249898 shutdown_connections 2024-04-25T00:22:38.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.890+0000 7f4a15ffb640 1 --2- 172.21.15.116:0/3425249898 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4a1c0678c0 0x7f4a1c069d80 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:38.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.890+0000 7f4a15ffb640 1 --2- 172.21.15.116:0/3425249898 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4a302096e0 0x7f4a3020fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:38.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.890+0000 7f4a15ffb640 1 --2- 172.21.15.116:0/3425249898 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4a30209260 0x7f4a300acea0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:38.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.890+0000 7f4a15ffb640 1 --2- 172.21.15.116:0/3425249898 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4a18004810 0x7f4a3020c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:38.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.890+0000 7f4a15ffb640 1 -- 172.21.15.116:0/3425249898 >> 172.21.15.116:0/3425249898 conn(0x7f4a300c4950 msgr2=0x7f4a30155200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:38.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.890+0000 7f4a15ffb640 1 -- 172.21.15.116:0/3425249898 shutdown_connections 2024-04-25T00:22:38.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.890+0000 7f4a15ffb640 1 -- 172.21.15.116:0/3425249898 wait complete. 2024-04-25T00:22:38.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.890+0000 7f5c38eca640 1 -- 172.21.15.116:0/3701040020 shutdown_connections 2024-04-25T00:22:38.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:38.890+0000 7f5c38eca640 1 -- 172.21.15.116:0/3701040020 wait complete. 2024-04-25T00:22:38.924 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:22:38.933 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 180, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:04.678737+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 176, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_7989": { "addr": "172.21.15.116:6835/221971311", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6834", "nonce": 221971311, "type": "v2" }, { "addr": "172.21.15.116:6835", "nonce": 221971311, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7989, "incarnation": 173, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 10 } }, "last_failure": 0, "last_failure_osd_epoch": 61, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T00:22:13.925793+0000", "qdb_cluster": [ 7989 ], "qdb_leader": 7989, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7989 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:18.825887+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 180, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6230": { "addr": "172.21.15.174:6837/492366509", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6836", "nonce": 492366509, "type": "v2" }, { "addr": "172.21.15.174:6837", "nonce": 492366509, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6230, "incarnation": 178, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:rejoin", "state_seq": 6 } }, "last_failure": 0, "last_failure_osd_epoch": 62, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-04-25T00:22:36.913581+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6230 } } } ], "standbys": [ { "addr": "172.21.15.174:6835/4112145782", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6834", "nonce": 4112145782, "type": "v2" }, { "addr": "172.21.15.174:6835", "nonce": 4112145782, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 112, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5582, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-25T00:22:39.994 INFO:tasks.ceph.mds.d.smithi174.stderr:2024-04-25T00:22:39.991+0000 7f36a7c04640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T00:22:40.120 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:22:40.123 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:22:40.924 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:22:40.934 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:22:41.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.054+0000 7f799b894640 1 Processor -- start 2024-04-25T00:22:41.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.054+0000 7f799b894640 1 -- start start 2024-04-25T00:22:41.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.054+0000 7f799b894640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7994153620 0x7f79940c3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:41.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.054+0000 7f799b894640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f799406f8c0 con 0x7f7994071050 2024-04-25T00:22:41.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.054+0000 7f799b894640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79940c5420 con 0x7f7994150d20 2024-04-25T00:22:41.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.054+0000 7f799b894640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f7994153a80 con 0x7f7994153620 2024-04-25T00:22:41.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.054+0000 7f7999e31640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f7994071050 0x7f799415df10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:43902/0 (socket says 172.21.15.116:43902) 2024-04-25T00:22:41.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.054+0000 7f7999e31640 1 -- 172.21.15.116:0/1517820465 learned_addr learned my addr 172.21.15.116:0/1517820465 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:22:41.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.054+0000 7f7998e2f640 1 --2- 172.21.15.116:0/1517820465 >> v2:172.21.15.116:3300/0 conn(0x7f7994153620 0x7f79940c3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:41.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.054+0000 7f798bfff640 1 -- 172.21.15.116:0/1517820465 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1774217148 0 0) 0x7f799406f8c0 con 0x7f7994071050 2024-04-25T00:22:41.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.054+0000 7f798bfff640 1 -- 172.21.15.116:0/1517820465 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7980003530 con 0x7f7994071050 2024-04-25T00:22:41.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.054+0000 7f7998e2f640 1 -- 172.21.15.116:0/1517820465 >> v1:172.21.15.174:6789/0 conn(0x7f7994150d20 legacy=0x7f79940c3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:41.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.054+0000 7f7998e2f640 1 -- 172.21.15.116:0/1517820465 >> v1:172.21.15.116:6789/0 conn(0x7f7994071050 legacy=0x7f799415df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:41.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.054+0000 7f7998e2f640 1 -- 172.21.15.116:0/1517820465 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79940c5700 con 0x7f7994153620 2024-04-25T00:22:41.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.058+0000 7f7998e2f640 1 --2- 172.21.15.116:0/1517820465 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7994153620 0x7f79940c3240 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f79840097c0 tx=0x7f798401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56942e4ada872216 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:41.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.058+0000 7f798bfff640 1 -- 172.21.15.116:0/1517820465 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79840b1070 con 0x7f7994153620 2024-04-25T00:22:41.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.058+0000 7f798bfff640 1 -- 172.21.15.116:0/1517820465 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79840ac410 con 0x7f7994153620 2024-04-25T00:22:41.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.058+0000 7f798bfff640 1 -- 172.21.15.116:0/1517820465 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79840ac7e0 con 0x7f7994153620 2024-04-25T00:22:41.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.058+0000 7f799b894640 1 -- 172.21.15.116:0/1517820465 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7994153620 msgr2=0x7f79940c3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:41.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.058+0000 7f799b894640 1 --2- 172.21.15.116:0/1517820465 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7994153620 0x7f79940c3240 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f79840097c0 tx=0x7f798401c4a0 comp rx=0 tx=0).stop 2024-04-25T00:22:41.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.058+0000 7f799b894640 1 -- 172.21.15.116:0/1517820465 shutdown_connections 2024-04-25T00:22:41.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.058+0000 7f799b894640 1 --2- 172.21.15.116:0/1517820465 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7994153620 0x7f79940c3240 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:41.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.058+0000 7f799b894640 1 -- 172.21.15.116:0/1517820465 >> 172.21.15.116:0/1517820465 conn(0x7f79940c9160 msgr2=0x7f79940c9570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:41.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.058+0000 7f799b894640 1 -- 172.21.15.116:0/1517820465 shutdown_connections 2024-04-25T00:22:41.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.058+0000 7f799b894640 1 -- 172.21.15.116:0/1517820465 wait complete. 2024-04-25T00:22:41.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.058+0000 7f799b894640 1 Processor -- start 2024-04-25T00:22:41.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.058+0000 7f799b894640 1 -- start start 2024-04-25T00:22:41.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.062+0000 7f799b894640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7994071050 0x7f7994148210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:41.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.062+0000 7f799b894640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7994150d20 0x7f799414c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:41.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.062+0000 7f799b894640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7994153620 0x7f79941ec180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:41.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.062+0000 7f799b894640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f799406f4c0 con 0x7f7994150d20 2024-04-25T00:22:41.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.062+0000 7f799b894640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f799406fba0 con 0x7f7994071050 2024-04-25T00:22:41.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.062+0000 7f799b894640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f799406f640 con 0x7f7994153620 2024-04-25T00:22:41.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.062+0000 7f7998e2f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7994150d20 0x7f799414c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:41.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.062+0000 7f7998e2f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7994150d20 0x7f799414c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37336/0 (socket says 172.21.15.116:37336) 2024-04-25T00:22:41.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.062+0000 7f7998e2f640 1 -- 172.21.15.116:0/2103094109 learned_addr learned my addr 172.21.15.116:0/2103094109 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:22:41.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.062+0000 7f7999e31640 1 --2- 172.21.15.116:0/2103094109 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7994153620 0x7f79941ec180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:41.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.062+0000 7f7999630640 1 --2- 172.21.15.116:0/2103094109 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7994071050 0x7f7994148210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:41.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.062+0000 7f7998e2f640 1 -- 172.21.15.116:0/2103094109 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7994153620 msgr2=0x7f79941ec180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:41.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.062+0000 7f7998e2f640 1 --2- 172.21.15.116:0/2103094109 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7994153620 0x7f79941ec180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:41.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.062+0000 7f7998e2f640 1 -- 172.21.15.116:0/2103094109 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7994071050 msgr2=0x7f7994148210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:41.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.062+0000 7f7998e2f640 1 --2- 172.21.15.116:0/2103094109 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7994071050 0x7f7994148210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:41.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.062+0000 7f7998e2f640 1 -- 172.21.15.116:0/2103094109 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7994148a60 con 0x7f7994150d20 2024-04-25T00:22:41.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.062+0000 7f7999630640 1 --2- 172.21.15.116:0/2103094109 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7994071050 0x7f7994148210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:22:41.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.062+0000 7f7998e2f640 1 --2- 172.21.15.116:0/2103094109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7994150d20 0x7f799414c460 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f79840af040 tx=0x7f7984019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:41.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.062+0000 7f7989ffb640 1 -- 172.21.15.116:0/2103094109 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79840b9020 con 0x7f7994150d20 2024-04-25T00:22:41.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.062+0000 7f7989ffb640 1 -- 172.21.15.116:0/2103094109 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79840acd90 con 0x7f7994150d20 2024-04-25T00:22:41.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.062+0000 7f799b894640 1 -- 172.21.15.116:0/2103094109 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f799414f510 con 0x7f7994150d20 2024-04-25T00:22:41.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.062+0000 7f7989ffb640 1 -- 172.21.15.116:0/2103094109 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79840b1040 con 0x7f7994150d20 2024-04-25T00:22:41.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.062+0000 7f799b894640 1 -- 172.21.15.116:0/2103094109 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7994148750 con 0x7f7994150d20 2024-04-25T00:22:41.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.062+0000 7f799b894640 1 -- 172.21.15.116:0/2103094109 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f799414fb20 con 0x7f7994150d20 2024-04-25T00:22:41.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.062+0000 7f7989ffb640 1 -- 172.21.15.116:0/2103094109 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f79840bc020 con 0x7f7994150d20 2024-04-25T00:22:41.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.062+0000 7f7989ffb640 1 --2- 172.21.15.116:0/2103094109 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f79780678c0 0x7f7978069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:41.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.066+0000 7f7989ffb640 1 -- 172.21.15.116:0/2103094109 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9590+0+0 (secure 0 0 0) 0x7f7984120370 con 0x7f7994150d20 2024-04-25T00:22:41.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.066+0000 7f7999630640 1 --2- 172.21.15.116:0/2103094109 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f79780678c0 0x7f7978069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:41.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.066+0000 7f7999630640 1 --2- 172.21.15.116:0/2103094109 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f79780678c0 0x7f7978069d80 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f7990000f70 tx=0x7f7990009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:41.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.070+0000 7f7989ffb640 1 -- 172.21.15.116:0/2103094109 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f79840c3030 con 0x7f7994150d20 2024-04-25T00:22:41.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.086+0000 7f1da25d7640 1 Processor -- start 2024-04-25T00:22:41.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.086+0000 7f1da25d7640 1 -- start start 2024-04-25T00:22:41.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.086+0000 7f1da25d7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1d9c0c1c30 0x7f1d9c0c3060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:41.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.086+0000 7f1da25d7640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d9c0c22a0 con 0x7f1d9c153170 2024-04-25T00:22:41.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.086+0000 7f1da25d7640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d9c0c2580 con 0x7f1d9c06c480 2024-04-25T00:22:41.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.086+0000 7f1da25d7640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f1d9c0c35a0 con 0x7f1d9c0c1c30 2024-04-25T00:22:41.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.086+0000 7f1d9b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1d9c0c1c30 0x7f1d9c0c3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:41.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.086+0000 7f1d9b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1d9c0c1c30 0x7f1d9c0c3060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37338/0 (socket says 172.21.15.116:37338) 2024-04-25T00:22:41.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.086+0000 7f1da0b74640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f1d9c153170 0x7f1d9c14fa80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:43914/0 (socket says 172.21.15.116:43914) 2024-04-25T00:22:41.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.086+0000 7f1d9b7fe640 1 -- 172.21.15.116:0/709143119 learned_addr learned my addr 172.21.15.116:0/709143119 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:22:41.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.090+0000 7f1d9b7fe640 1 -- 172.21.15.116:0/709143119 >> v1:172.21.15.174:6789/0 conn(0x7f1d9c06c480 legacy=0x7f1d9c071e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:41.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.090+0000 7f1d9b7fe640 1 -- 172.21.15.116:0/709143119 >> v1:172.21.15.116:6789/0 conn(0x7f1d9c153170 legacy=0x7f1d9c14fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:41.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.090+0000 7f1d9affd640 1 -- 172.21.15.116:0/709143119 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1812541054 0 0) 0x7f1d9c0c2580 con 0x7f1d9c06c480 2024-04-25T00:22:41.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.090+0000 7f1d9b7fe640 1 -- 172.21.15.116:0/709143119 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d9c0c2860 con 0x7f1d9c0c1c30 2024-04-25T00:22:41.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.090+0000 7f1d9b7fe640 1 --2- 172.21.15.116:0/709143119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d9c0c1c30 0x7f1d9c0c3060 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f1d8c009870 tx=0x7f1d8c01c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bff8032be75340d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:41.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.090+0000 7f1d9affd640 1 -- 172.21.15.116:0/709143119 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d8c0b1070 con 0x7f1d9c0c1c30 2024-04-25T00:22:41.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.090+0000 7f1d9affd640 1 -- 172.21.15.116:0/709143119 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d8c0ac5f0 con 0x7f1d9c0c1c30 2024-04-25T00:22:41.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.090+0000 7f1d9affd640 1 -- 172.21.15.116:0/709143119 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d8c0ac9c0 con 0x7f1d9c0c1c30 2024-04-25T00:22:41.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.090+0000 7f1da25d7640 1 -- 172.21.15.116:0/709143119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d9c0c1c30 msgr2=0x7f1d9c0c3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:41.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.090+0000 7f1da25d7640 1 --2- 172.21.15.116:0/709143119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d9c0c1c30 0x7f1d9c0c3060 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f1d8c009870 tx=0x7f1d8c01c3f0 comp rx=0 tx=0).stop 2024-04-25T00:22:41.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.090+0000 7f1da25d7640 1 -- 172.21.15.116:0/709143119 shutdown_connections 2024-04-25T00:22:41.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.090+0000 7f1da25d7640 1 --2- 172.21.15.116:0/709143119 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d9c0c1c30 0x7f1d9c0c3060 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:41.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.090+0000 7f1da25d7640 1 -- 172.21.15.116:0/709143119 >> 172.21.15.116:0/709143119 conn(0x7f1d9c0c4270 msgr2=0x7f1d9c0c4680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:41.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.090+0000 7f1da25d7640 1 -- 172.21.15.116:0/709143119 shutdown_connections 2024-04-25T00:22:41.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.090+0000 7f1da25d7640 1 -- 172.21.15.116:0/709143119 wait complete. 2024-04-25T00:22:41.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.094+0000 7f1da25d7640 1 Processor -- start 2024-04-25T00:22:41.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.094+0000 7f1da25d7640 1 -- start start 2024-04-25T00:22:41.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.094+0000 7f1da25d7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d9c06c480 0x7f1d9c1ec320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:41.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.094+0000 7f1da25d7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1d9c0c1c30 0x7f1d9c0c6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:41.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.094+0000 7f1da25d7640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1d9c153170 0x7f1d9c14ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:41.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.094+0000 7f1da25d7640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1d9c150190 con 0x7f1d9c06c480 2024-04-25T00:22:41.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.094+0000 7f1da25d7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1d9c153670 con 0x7f1d9c0c1c30 2024-04-25T00:22:41.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.094+0000 7f1da25d7640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f1d9c150310 con 0x7f1d9c153170 2024-04-25T00:22:41.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.094+0000 7f1d9bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d9c06c480 0x7f1d9c1ec320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:41.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.094+0000 7f1d9bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d9c06c480 0x7f1d9c1ec320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37346/0 (socket says 172.21.15.116:37346) 2024-04-25T00:22:41.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.094+0000 7f1d9bfff640 1 -- 172.21.15.116:0/1342496255 learned_addr learned my addr 172.21.15.116:0/1342496255 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:22:41.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.094+0000 7f1da0b74640 1 --2- 172.21.15.116:0/1342496255 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1d9c153170 0x7f1d9c14ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:41.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.094+0000 7f1d9b7fe640 1 --2- 172.21.15.116:0/1342496255 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1d9c0c1c30 0x7f1d9c0c6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:41.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.094+0000 7f1d9bfff640 1 -- 172.21.15.116:0/1342496255 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1d9c153170 msgr2=0x7f1d9c14ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:41.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.094+0000 7f1d9bfff640 1 --2- 172.21.15.116:0/1342496255 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1d9c153170 0x7f1d9c14ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:41.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.094+0000 7f1d9bfff640 1 -- 172.21.15.116:0/1342496255 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1d9c0c1c30 msgr2=0x7f1d9c0c6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:41.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.094+0000 7f1d9bfff640 1 --2- 172.21.15.116:0/1342496255 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1d9c0c1c30 0x7f1d9c0c6bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:41.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.094+0000 7f1d9bfff640 1 -- 172.21.15.116:0/1342496255 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d9c1eb880 con 0x7f1d9c06c480 2024-04-25T00:22:41.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.094+0000 7f1d9bfff640 1 --2- 172.21.15.116:0/1342496255 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d9c06c480 0x7f1d9c1ec320 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f1d9000c480 tx=0x7f1d9000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:41.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.094+0000 7f1d98ff9640 1 -- 172.21.15.116:0/1342496255 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d90005480 con 0x7f1d9c06c480 2024-04-25T00:22:41.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.094+0000 7f1d98ff9640 1 -- 172.21.15.116:0/1342496255 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d90003600 con 0x7f1d9c06c480 2024-04-25T00:22:41.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.094+0000 7f1d98ff9640 1 -- 172.21.15.116:0/1342496255 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d900039f0 con 0x7f1d9c06c480 2024-04-25T00:22:41.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.094+0000 7f1da25d7640 1 -- 172.21.15.116:0/1342496255 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d9c1e8e90 con 0x7f1d9c06c480 2024-04-25T00:22:41.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.094+0000 7f1da25d7640 1 -- 172.21.15.116:0/1342496255 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d9c1ebe00 con 0x7f1d9c06c480 2024-04-25T00:22:41.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.098+0000 7f1da25d7640 1 -- 172.21.15.116:0/1342496255 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d9c068a60 con 0x7f1d9c06c480 2024-04-25T00:22:41.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.098+0000 7f1d98ff9640 1 -- 172.21.15.116:0/1342496255 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1d90003b90 con 0x7f1d9c06c480 2024-04-25T00:22:41.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.098+0000 7f1d98ff9640 1 --2- 172.21.15.116:0/1342496255 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1d880678c0 0x7f1d88069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:41.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.098+0000 7f1d98ff9640 1 -- 172.21.15.116:0/1342496255 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9590+0+0 (secure 0 0 0) 0x7f1d90086740 con 0x7f1d9c06c480 2024-04-25T00:22:41.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.098+0000 7f1d9b7fe640 1 --2- 172.21.15.116:0/1342496255 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1d880678c0 0x7f1d88069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:41.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.098+0000 7f1d9b7fe640 1 --2- 172.21.15.116:0/1342496255 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1d880678c0 0x7f1d88069d80 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f1d8c0ae000 tx=0x7f1d8c0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:41.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.102+0000 7f1d98ff9640 1 -- 172.21.15.116:0/1342496255 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f1d90055d40 con 0x7f1d9c06c480 2024-04-25T00:22:41.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.270+0000 7f799b894640 1 -- 172.21.15.116:0/2103094109 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7994068a60 con 0x7f7994150d20 2024-04-25T00:22:41.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.270+0000 7f7989ffb640 1 -- 172.21.15.116:0/2103094109 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 181 v181) ==== 95+0+5674 (secure 0 0 0) 0x7f7984016090 con 0x7f7994150d20 2024-04-25T00:22:41.272 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:22:41.273 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":181,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:22:13.925793+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7989},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7989":{"gid":7989,"name":"c","rank":0,"incarnation":173,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6835/221971311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":221971311},{"type":"v1","addr":"172.21.15.116:6835","nonce":221971311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7989,"qdb_cluster":[7989]},"id":1},{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:22:38.936345+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6230":{"gid":6230,"name":"d","rank":0,"incarnation":178,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/492366509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":492366509},{"type":"v1","addr":"172.21.15.174:6837","nonce":492366509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6230,"qdb_cluster":[6230]},"id":2}]} 2024-04-25T00:22:41.273 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 181 2024-04-25T00:22:41.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.274+0000 7f799b894640 1 -- 172.21.15.116:0/2103094109 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f79780678c0 msgr2=0x7f7978069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:41.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.274+0000 7f799b894640 1 --2- 172.21.15.116:0/2103094109 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f79780678c0 0x7f7978069d80 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f7990000f70 tx=0x7f7990009040 comp rx=0 tx=0).stop 2024-04-25T00:22:41.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.274+0000 7f799b894640 1 -- 172.21.15.116:0/2103094109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7994150d20 msgr2=0x7f799414c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:41.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.274+0000 7f799b894640 1 --2- 172.21.15.116:0/2103094109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7994150d20 0x7f799414c460 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f79840af040 tx=0x7f7984019240 comp rx=0 tx=0).stop 2024-04-25T00:22:41.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.274+0000 7f799b894640 1 -- 172.21.15.116:0/2103094109 shutdown_connections 2024-04-25T00:22:41.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.274+0000 7f799b894640 1 --2- 172.21.15.116:0/2103094109 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f79780678c0 0x7f7978069d80 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:41.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.274+0000 7f799b894640 1 --2- 172.21.15.116:0/2103094109 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7994153620 0x7f79941ec180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:41.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.274+0000 7f799b894640 1 --2- 172.21.15.116:0/2103094109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7994150d20 0x7f799414c460 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:41.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.274+0000 7f799b894640 1 --2- 172.21.15.116:0/2103094109 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7994071050 0x7f7994148210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:41.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.274+0000 7f799b894640 1 -- 172.21.15.116:0/2103094109 >> 172.21.15.116:0/2103094109 conn(0x7f79940c9160 msgr2=0x7f79940671c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:41.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.274+0000 7f799b894640 1 -- 172.21.15.116:0/2103094109 shutdown_connections 2024-04-25T00:22:41.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.274+0000 7f799b894640 1 -- 172.21.15.116:0/2103094109 wait complete. 2024-04-25T00:22:41.291 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:22:41.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.298+0000 7f1da25d7640 1 -- 172.21.15.116:0/1342496255 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1d9c0a7100 con 0x7f1d9c06c480 2024-04-25T00:22:41.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.302+0000 7f1d98ff9640 1 -- 172.21.15.116:0/1342496255 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 181 v181) ==== 95+0+5674 (secure 0 0 0) 0x7f1d90085090 con 0x7f1d9c06c480 2024-04-25T00:22:41.303 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:22:41.303 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":181,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:22:13.925793+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7989},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7989":{"gid":7989,"name":"c","rank":0,"incarnation":173,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6835/221971311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":221971311},{"type":"v1","addr":"172.21.15.116:6835","nonce":221971311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7989,"qdb_cluster":[7989]},"id":1},{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:22:38.936345+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6230":{"gid":6230,"name":"d","rank":0,"incarnation":178,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/492366509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":492366509},{"type":"v1","addr":"172.21.15.174:6837","nonce":492366509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6230,"qdb_cluster":[6230]},"id":2}]} 2024-04-25T00:22:41.304 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 181 2024-04-25T00:22:41.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.302+0000 7f1da25d7640 1 -- 172.21.15.116:0/1342496255 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1d880678c0 msgr2=0x7f1d88069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:41.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.302+0000 7f1da25d7640 1 --2- 172.21.15.116:0/1342496255 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1d880678c0 0x7f1d88069d80 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f1d8c0ae000 tx=0x7f1d8c0a7000 comp rx=0 tx=0).stop 2024-04-25T00:22:41.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.302+0000 7f1da25d7640 1 -- 172.21.15.116:0/1342496255 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d9c06c480 msgr2=0x7f1d9c1ec320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:41.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.302+0000 7f1da25d7640 1 --2- 172.21.15.116:0/1342496255 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d9c06c480 0x7f1d9c1ec320 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f1d9000c480 tx=0x7f1d9000c940 comp rx=0 tx=0).stop 2024-04-25T00:22:41.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.302+0000 7f1da25d7640 1 -- 172.21.15.116:0/1342496255 shutdown_connections 2024-04-25T00:22:41.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.302+0000 7f1da25d7640 1 --2- 172.21.15.116:0/1342496255 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1d880678c0 0x7f1d88069d80 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:41.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.302+0000 7f1da25d7640 1 --2- 172.21.15.116:0/1342496255 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1d9c153170 0x7f1d9c14ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:41.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.302+0000 7f1da25d7640 1 --2- 172.21.15.116:0/1342496255 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1d9c0c1c30 0x7f1d9c0c6bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:41.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.302+0000 7f1da25d7640 1 --2- 172.21.15.116:0/1342496255 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d9c06c480 0x7f1d9c1ec320 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:41.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.302+0000 7f1da25d7640 1 -- 172.21.15.116:0/1342496255 >> 172.21.15.116:0/1342496255 conn(0x7f1d9c0c4270 msgr2=0x7f1d9c14d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:41.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.302+0000 7f1da25d7640 1 -- 172.21.15.116:0/1342496255 shutdown_connections 2024-04-25T00:22:41.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:41.302+0000 7f1da25d7640 1 -- 172.21.15.116:0/1342496255 wait complete. 2024-04-25T00:22:41.320 INFO:tasks.mds_thrash.fs.[b]:mds.d has gained rank=0, replacing gid=7722 2024-04-25T00:22:41.320 INFO:tasks.mds_thrash.fs.[b]:waiting for 7 secs before reviving mds.a 2024-04-25T00:22:43.292 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:22:43.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.418+0000 7f1e1e2b7640 1 Processor -- start 2024-04-25T00:22:43.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.418+0000 7f1e1e2b7640 1 -- start start 2024-04-25T00:22:43.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.422+0000 7f1e1e2b7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1e18148890 0x7f1e18072590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:43.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.422+0000 7f1e1e2b7640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f1e18070dd0 0x7f1e18071270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:43.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.422+0000 7f1e1e2b7640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1e1806e890 0x7f1e180717b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:43.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.422+0000 7f1e1e2b7640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f1e18152c10 con 0x7f1e18148890 2024-04-25T00:22:43.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.422+0000 7f1e1e2b7640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f1e18071cf0 con 0x7f1e18070dd0 2024-04-25T00:22:43.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.422+0000 7f1e1e2b7640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f1e18151e50 con 0x7f1e1806e890 2024-04-25T00:22:43.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.422+0000 7f1e17fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1e18148890 0x7f1e18072590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:43.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.422+0000 7f1e17fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1e18148890 0x7f1e18072590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37360/0 (socket says 172.21.15.116:37360) 2024-04-25T00:22:43.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.422+0000 7f1e17fff640 1 -- 172.21.15.116:0/1693992645 learned_addr learned my addr 172.21.15.116:0/1693992645 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:22:43.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.422+0000 7f1e177fe640 1 --2- 172.21.15.116:0/1693992645 >> v2:172.21.15.174:3300/0 conn(0x7f1e18070dd0 0x7f1e18071270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:43.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.422+0000 7f1e1c854640 1 --2- 172.21.15.116:0/1693992645 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1e1806e890 0x7f1e180717b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:43.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.422+0000 7f1e17fff640 1 -- 172.21.15.116:0/1693992645 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1e1806e890 msgr2=0x7f1e180717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:43.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.422+0000 7f1e17fff640 1 --2- 172.21.15.116:0/1693992645 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1e1806e890 0x7f1e180717b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:43.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.422+0000 7f1e17fff640 1 -- 172.21.15.116:0/1693992645 >> v2:172.21.15.174:3300/0 conn(0x7f1e18070dd0 msgr2=0x7f1e18071270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:43.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.422+0000 7f1e17fff640 1 --2- 172.21.15.116:0/1693992645 >> v2:172.21.15.174:3300/0 conn(0x7f1e18070dd0 0x7f1e18071270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:43.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.422+0000 7f1e17fff640 1 -- 172.21.15.116:0/1693992645 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e18152130 con 0x7f1e18148890 2024-04-25T00:22:43.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.422+0000 7f1e1c854640 1 --2- 172.21.15.116:0/1693992645 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1e1806e890 0x7f1e180717b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:22:43.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.422+0000 7f1e177fe640 1 --2- 172.21.15.116:0/1693992645 >> v2:172.21.15.174:3300/0 conn(0x7f1e18070dd0 0x7f1e18071270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:22:43.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.422+0000 7f1e17fff640 1 --2- 172.21.15.116:0/1693992645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e18148890 0x7f1e18072590 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f1e180711b0 tx=0x7f1e0801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b50c7be2b903bf85 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:43.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.422+0000 7f1e16ffd640 1 -- 172.21.15.116:0/1693992645 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e080b0070 con 0x7f1e18148890 2024-04-25T00:22:43.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.426+0000 7f1e16ffd640 1 -- 172.21.15.116:0/1693992645 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e080ab600 con 0x7f1e18148890 2024-04-25T00:22:43.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.426+0000 7f1e16ffd640 1 -- 172.21.15.116:0/1693992645 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e080ab9d0 con 0x7f1e18148890 2024-04-25T00:22:43.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.426+0000 7f1e1e2b7640 1 -- 172.21.15.116:0/1693992645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e18148890 msgr2=0x7f1e18072590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:43.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.426+0000 7f1e1e2b7640 1 --2- 172.21.15.116:0/1693992645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e18148890 0x7f1e18072590 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f1e180711b0 tx=0x7f1e0801c520 comp rx=0 tx=0).stop 2024-04-25T00:22:43.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.426+0000 7f1e1e2b7640 1 -- 172.21.15.116:0/1693992645 shutdown_connections 2024-04-25T00:22:43.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.426+0000 7f1e1e2b7640 1 --2- 172.21.15.116:0/1693992645 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1e1806e890 0x7f1e180717b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:43.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.426+0000 7f1e1e2b7640 1 --2- 172.21.15.116:0/1693992645 >> v2:172.21.15.174:3300/0 conn(0x7f1e18070dd0 0x7f1e18071270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:43.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.426+0000 7f1e1e2b7640 1 --2- 172.21.15.116:0/1693992645 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e18148890 0x7f1e18072590 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:43.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.426+0000 7f1e1e2b7640 1 -- 172.21.15.116:0/1693992645 >> 172.21.15.116:0/1693992645 conn(0x7f1e180c3b00 msgr2=0x7f1e180c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:43.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.426+0000 7f1e1e2b7640 1 -- 172.21.15.116:0/1693992645 shutdown_connections 2024-04-25T00:22:43.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.426+0000 7f1e1e2b7640 1 -- 172.21.15.116:0/1693992645 wait complete. 2024-04-25T00:22:43.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.426+0000 7f1e1e2b7640 1 Processor -- start 2024-04-25T00:22:43.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.426+0000 7f1e1e2b7640 1 -- start start 2024-04-25T00:22:43.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.426+0000 7f1e1e2b7640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1e1806e890 0x7f1e1813f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:43.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.426+0000 7f1e1e2b7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e18070dd0 0x7f1e1813c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:43.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.426+0000 7f1e1e2b7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1e18148890 0x7f1e18140ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:43.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.426+0000 7f1e1e2b7640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1e180abf20 con 0x7f1e18070dd0 2024-04-25T00:22:43.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.426+0000 7f1e1e2b7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1e180abda0 con 0x7f1e18148890 2024-04-25T00:22:43.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.426+0000 7f1e1e2b7640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f1e180ac0a0 con 0x7f1e1806e890 2024-04-25T00:22:43.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.426+0000 7f1e177fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e18070dd0 0x7f1e1813c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:43.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.426+0000 7f1e177fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e18070dd0 0x7f1e1813c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37362/0 (socket says 172.21.15.116:37362) 2024-04-25T00:22:43.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.426+0000 7f1e177fe640 1 -- 172.21.15.116:0/1643229412 learned_addr learned my addr 172.21.15.116:0/1643229412 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:22:43.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.426+0000 7f1e17fff640 1 --2- 172.21.15.116:0/1643229412 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1e1806e890 0x7f1e1813f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:43.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.426+0000 7f1e1c854640 1 --2- 172.21.15.116:0/1643229412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1e18148890 0x7f1e18140ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:43.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.426+0000 7f1e177fe640 1 -- 172.21.15.116:0/1643229412 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1e1806e890 msgr2=0x7f1e1813f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:43.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.426+0000 7f1e177fe640 1 --2- 172.21.15.116:0/1643229412 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1e1806e890 0x7f1e1813f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:43.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.426+0000 7f1e177fe640 1 -- 172.21.15.116:0/1643229412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1e18148890 msgr2=0x7f1e18140ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:43.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.426+0000 7f1e177fe640 1 --2- 172.21.15.116:0/1643229412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1e18148890 0x7f1e18140ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:43.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.426+0000 7f1e177fe640 1 -- 172.21.15.116:0/1643229412 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e1813c980 con 0x7f1e18070dd0 2024-04-25T00:22:43.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.426+0000 7f1e17fff640 1 --2- 172.21.15.116:0/1643229412 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1e1806e890 0x7f1e1813f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:22:43.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.426+0000 7f1e177fe640 1 --2- 172.21.15.116:0/1643229412 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e18070dd0 0x7f1e1813c440 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c00c7f0 tx=0x7f1e0c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:43.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.426+0000 7f1e14ff9640 1 -- 172.21.15.116:0/1643229412 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e0c015070 con 0x7f1e18070dd0 2024-04-25T00:22:43.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.426+0000 7f1e14ff9640 1 -- 172.21.15.116:0/1643229412 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e0c007a30 con 0x7f1e18070dd0 2024-04-25T00:22:43.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.426+0000 7f1e14ff9640 1 -- 172.21.15.116:0/1643229412 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e0c007e00 con 0x7f1e18070dd0 2024-04-25T00:22:43.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.426+0000 7f1e1e2b7640 1 -- 172.21.15.116:0/1643229412 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e1813cf00 con 0x7f1e18070dd0 2024-04-25T00:22:43.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.426+0000 7f1e1e2b7640 1 -- 172.21.15.116:0/1643229412 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e181eec00 con 0x7f1e18070dd0 2024-04-25T00:22:43.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.430+0000 7f1e1e2b7640 1 -- 172.21.15.116:0/1643229412 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e18068350 con 0x7f1e18070dd0 2024-04-25T00:22:43.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.430+0000 7f1e14ff9640 1 -- 172.21.15.116:0/1643229412 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1e0c018070 con 0x7f1e18070dd0 2024-04-25T00:22:43.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.430+0000 7f1e14ff9640 1 --2- 172.21.15.116:0/1643229412 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1e040678f0 0x7f1e04069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:43.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.430+0000 7f1e14ff9640 1 -- 172.21.15.116:0/1643229412 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9590+0+0 (secure 0 0 0) 0x7f1e0c086ac0 con 0x7f1e18070dd0 2024-04-25T00:22:43.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.430+0000 7f1e17fff640 1 --2- 172.21.15.116:0/1643229412 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1e040678f0 0x7f1e04069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:43.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.430+0000 7f1e17fff640 1 --2- 172.21.15.116:0/1643229412 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1e040678f0 0x7f1e04069db0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f1e080ad000 tx=0x7f1e08008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:43.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.434+0000 7f1e14ff9640 1 -- 172.21.15.116:0/1643229412 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f1e0c056710 con 0x7f1e18070dd0 2024-04-25T00:22:43.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.630+0000 7f1e1e2b7640 1 -- 172.21.15.116:0/1643229412 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1e1806e670 con 0x7f1e18070dd0 2024-04-25T00:22:43.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.634+0000 7f1e14ff9640 1 -- 172.21.15.116:0/1643229412 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 181 v181) ==== 95+0+5674 (secure 0 0 0) 0x7f1e0c056010 con 0x7f1e18070dd0 2024-04-25T00:22:43.633 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:22:43.633 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":181,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:22:13.925793+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7989},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7989":{"gid":7989,"name":"c","rank":0,"incarnation":173,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6835/221971311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":221971311},{"type":"v1","addr":"172.21.15.116:6835","nonce":221971311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7989,"qdb_cluster":[7989]},"id":1},{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:22:38.936345+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6230":{"gid":6230,"name":"d","rank":0,"incarnation":178,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/492366509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":492366509},{"type":"v1","addr":"172.21.15.174:6837","nonce":492366509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6230,"qdb_cluster":[6230]},"id":2}]} 2024-04-25T00:22:43.634 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 181 2024-04-25T00:22:43.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.634+0000 7f1e1e2b7640 1 -- 172.21.15.116:0/1643229412 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1e040678f0 msgr2=0x7f1e04069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:43.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.634+0000 7f1e1e2b7640 1 --2- 172.21.15.116:0/1643229412 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1e040678f0 0x7f1e04069db0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f1e080ad000 tx=0x7f1e08008040 comp rx=0 tx=0).stop 2024-04-25T00:22:43.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.634+0000 7f1e1e2b7640 1 -- 172.21.15.116:0/1643229412 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e18070dd0 msgr2=0x7f1e1813c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:43.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.634+0000 7f1e1e2b7640 1 --2- 172.21.15.116:0/1643229412 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e18070dd0 0x7f1e1813c440 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c00c7f0 tx=0x7f1e0c00ccb0 comp rx=0 tx=0).stop 2024-04-25T00:22:43.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.634+0000 7f1e1e2b7640 1 -- 172.21.15.116:0/1643229412 shutdown_connections 2024-04-25T00:22:43.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.634+0000 7f1e1e2b7640 1 --2- 172.21.15.116:0/1643229412 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1e040678f0 0x7f1e04069db0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:43.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.634+0000 7f1e1e2b7640 1 --2- 172.21.15.116:0/1643229412 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1e18148890 0x7f1e18140ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:43.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.634+0000 7f1e1e2b7640 1 --2- 172.21.15.116:0/1643229412 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1e18070dd0 0x7f1e1813c440 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:43.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.634+0000 7f1e1e2b7640 1 --2- 172.21.15.116:0/1643229412 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1e1806e890 0x7f1e1813f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:43.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.634+0000 7f1e1e2b7640 1 -- 172.21.15.116:0/1643229412 >> 172.21.15.116:0/1643229412 conn(0x7f1e180c3b00 msgr2=0x7f1e180c26c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:43.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.634+0000 7f1e1e2b7640 1 -- 172.21.15.116:0/1643229412 shutdown_connections 2024-04-25T00:22:43.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:43.634+0000 7f1e1e2b7640 1 -- 172.21.15.116:0/1643229412 wait complete. 2024-04-25T00:22:43.652 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:22:45.653 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:22:45.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.782+0000 7f9e75262640 1 Processor -- start 2024-04-25T00:22:45.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.782+0000 7f9e75262640 1 -- start start 2024-04-25T00:22:45.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.782+0000 7f9e75262640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9e70152d80 0x7f9e7006bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:45.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.782+0000 7f9e75262640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e700c9150 con 0x7f9e7006c290 2024-04-25T00:22:45.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.782+0000 7f9e75262640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e700c9430 con 0x7f9e7006d5f0 2024-04-25T00:22:45.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.782+0000 7f9e75262640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f9e700c9d60 con 0x7f9e70152d80 2024-04-25T00:22:45.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.782+0000 7f9e6effd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9e70152d80 0x7f9e7006bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:45.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.782+0000 7f9e6effd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f9e70152d80 0x7f9e7006bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39664/0 (socket says 172.21.15.116:39664) 2024-04-25T00:22:45.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.782+0000 7f9e6effd640 1 -- 172.21.15.116:0/2899600896 learned_addr learned my addr 172.21.15.116:0/2899600896 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:22:45.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.782+0000 7f9e6f7fe640 1 --1- 172.21.15.116:0/2899600896 >> v1:172.21.15.116:6789/0 conn(0x7f9e7006c290 0x7f9e70157390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:42862/0 (socket says 172.21.15.116:42862) 2024-04-25T00:22:45.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.782+0000 7f9e6effd640 1 -- 172.21.15.116:0/2899600896 >> v1:172.21.15.174:6789/0 conn(0x7f9e7006d5f0 legacy=0x7f9e70156c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:45.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.782+0000 7f9e6effd640 1 -- 172.21.15.116:0/2899600896 >> v1:172.21.15.116:6789/0 conn(0x7f9e7006c290 legacy=0x7f9e70157390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:45.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.782+0000 7f9e6effd640 1 -- 172.21.15.116:0/2899600896 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e700c9710 con 0x7f9e70152d80 2024-04-25T00:22:45.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.782+0000 7f9e6dffb640 1 -- 172.21.15.116:0/2899600896 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 299988954 0 0) 0x7f9e700c9150 con 0x7f9e7006c290 2024-04-25T00:22:45.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.786+0000 7f9e6effd640 1 --2- 172.21.15.116:0/2899600896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e70152d80 0x7f9e7006bd50 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f9e600098a0 tx=0x7f9e6001c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6bb3042fbd1fb2d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:45.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.786+0000 7f9e6dffb640 1 -- 172.21.15.116:0/2899600896 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e600b1070 con 0x7f9e70152d80 2024-04-25T00:22:45.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.786+0000 7f9e6dffb640 1 -- 172.21.15.116:0/2899600896 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e600ac640 con 0x7f9e70152d80 2024-04-25T00:22:45.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.786+0000 7f9e6dffb640 1 -- 172.21.15.116:0/2899600896 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e600aca10 con 0x7f9e70152d80 2024-04-25T00:22:45.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.786+0000 7f9e75262640 1 -- 172.21.15.116:0/2899600896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e70152d80 msgr2=0x7f9e7006bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:45.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.786+0000 7f9e75262640 1 --2- 172.21.15.116:0/2899600896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e70152d80 0x7f9e7006bd50 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f9e600098a0 tx=0x7f9e6001c3f0 comp rx=0 tx=0).stop 2024-04-25T00:22:45.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.786+0000 7f9e75262640 1 -- 172.21.15.116:0/2899600896 shutdown_connections 2024-04-25T00:22:45.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.786+0000 7f9e75262640 1 --2- 172.21.15.116:0/2899600896 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e70152d80 0x7f9e7006bd50 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:45.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.786+0000 7f9e75262640 1 -- 172.21.15.116:0/2899600896 >> 172.21.15.116:0/2899600896 conn(0x7f9e700acdd0 msgr2=0x7f9e700ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:45.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.786+0000 7f9e75262640 1 -- 172.21.15.116:0/2899600896 shutdown_connections 2024-04-25T00:22:45.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.786+0000 7f9e75262640 1 -- 172.21.15.116:0/2899600896 wait complete. 2024-04-25T00:22:45.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.786+0000 7f9e75262640 1 Processor -- start 2024-04-25T00:22:45.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.786+0000 7f9e75262640 1 -- start start 2024-04-25T00:22:45.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.786+0000 7f9e75262640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9e7006c290 0x7f9e700aaa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:45.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.786+0000 7f9e75262640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e7006d5f0 0x7f9e70151e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:45.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.786+0000 7f9e75262640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e70152d80 0x7f9e70154740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:45.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.786+0000 7f9e75262640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9e70163a60 con 0x7f9e70152d80 2024-04-25T00:22:45.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.786+0000 7f9e75262640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9e701638e0 con 0x7f9e7006d5f0 2024-04-25T00:22:45.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.786+0000 7f9e75262640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f9e70163be0 con 0x7f9e7006c290 2024-04-25T00:22:45.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.786+0000 7f9e6f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e70152d80 0x7f9e70154740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:45.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.786+0000 7f9e6f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e70152d80 0x7f9e70154740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39676/0 (socket says 172.21.15.116:39676) 2024-04-25T00:22:45.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.786+0000 7f9e6f7fe640 1 -- 172.21.15.116:0/3786372669 learned_addr learned my addr 172.21.15.116:0/3786372669 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:22:45.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.786+0000 7f9e6effd640 1 --2- 172.21.15.116:0/3786372669 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9e7006c290 0x7f9e700aaa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:45.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.786+0000 7f9e6e7fc640 1 --2- 172.21.15.116:0/3786372669 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e7006d5f0 0x7f9e70151e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:45.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.786+0000 7f9e6f7fe640 1 -- 172.21.15.116:0/3786372669 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9e7006c290 msgr2=0x7f9e700aaa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:45.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.786+0000 7f9e6f7fe640 1 --2- 172.21.15.116:0/3786372669 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9e7006c290 0x7f9e700aaa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:45.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.786+0000 7f9e6f7fe640 1 -- 172.21.15.116:0/3786372669 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e7006d5f0 msgr2=0x7f9e70151e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:45.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.786+0000 7f9e6f7fe640 1 --2- 172.21.15.116:0/3786372669 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e7006d5f0 0x7f9e70151e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:45.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.786+0000 7f9e6f7fe640 1 -- 172.21.15.116:0/3786372669 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e70153d90 con 0x7f9e70152d80 2024-04-25T00:22:45.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.786+0000 7f9e6f7fe640 1 --2- 172.21.15.116:0/3786372669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e70152d80 0x7f9e70154740 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f9e6800c700 tx=0x7f9e6800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:45.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.786+0000 7f9e577fe640 1 -- 172.21.15.116:0/3786372669 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e68018070 con 0x7f9e70152d80 2024-04-25T00:22:45.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.786+0000 7f9e577fe640 1 -- 172.21.15.116:0/3786372669 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e6800ce10 con 0x7f9e70152d80 2024-04-25T00:22:45.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.786+0000 7f9e75262640 1 -- 172.21.15.116:0/3786372669 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e701432f0 con 0x7f9e70152d80 2024-04-25T00:22:45.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.790+0000 7f9e577fe640 1 -- 172.21.15.116:0/3786372669 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e6800a880 con 0x7f9e70152d80 2024-04-25T00:22:45.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.790+0000 7f9e75262640 1 -- 172.21.15.116:0/3786372669 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e70151650 con 0x7f9e70152d80 2024-04-25T00:22:45.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.790+0000 7f9e75262640 1 -- 172.21.15.116:0/3786372669 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e70068350 con 0x7f9e70152d80 2024-04-25T00:22:45.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.790+0000 7f9e577fe640 1 -- 172.21.15.116:0/3786372669 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f9e6801b070 con 0x7f9e70152d80 2024-04-25T00:22:45.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.790+0000 7f9e577fe640 1 --2- 172.21.15.116:0/3786372669 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9e5c067910 0x7f9e5c069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:45.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.790+0000 7f9e577fe640 1 -- 172.21.15.116:0/3786372669 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9590+0+0 (secure 0 0 0) 0x7f9e68086160 con 0x7f9e70152d80 2024-04-25T00:22:45.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.790+0000 7f9e6effd640 1 --2- 172.21.15.116:0/3786372669 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9e5c067910 0x7f9e5c069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:45.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.790+0000 7f9e6effd640 1 --2- 172.21.15.116:0/3786372669 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9e5c067910 0x7f9e5c069dd0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f9e600ae000 tx=0x7f9e600a6070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:45.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.794+0000 7f9e577fe640 1 -- 172.21.15.116:0/3786372669 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f9e68055e30 con 0x7f9e70152d80 2024-04-25T00:22:45.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.990+0000 7f9e75262640 1 -- 172.21.15.116:0/3786372669 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9e700cb270 con 0x7f9e70152d80 2024-04-25T00:22:45.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.990+0000 7f9e577fe640 1 -- 172.21.15.116:0/3786372669 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 181 v181) ==== 95+0+5674 (secure 0 0 0) 0x7f9e68055730 con 0x7f9e70152d80 2024-04-25T00:22:45.991 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:22:45.991 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":181,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:22:13.925793+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7989},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7989":{"gid":7989,"name":"c","rank":0,"incarnation":173,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6835/221971311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":221971311},{"type":"v1","addr":"172.21.15.116:6835","nonce":221971311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7989,"qdb_cluster":[7989]},"id":1},{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:22:38.936345+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6230":{"gid":6230,"name":"d","rank":0,"incarnation":178,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/492366509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":492366509},{"type":"v1","addr":"172.21.15.174:6837","nonce":492366509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6230,"qdb_cluster":[6230]},"id":2}]} 2024-04-25T00:22:45.992 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 181 2024-04-25T00:22:45.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.990+0000 7f9e75262640 1 -- 172.21.15.116:0/3786372669 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9e5c067910 msgr2=0x7f9e5c069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:45.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.994+0000 7f9e75262640 1 --2- 172.21.15.116:0/3786372669 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9e5c067910 0x7f9e5c069dd0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f9e600ae000 tx=0x7f9e600a6070 comp rx=0 tx=0).stop 2024-04-25T00:22:45.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.994+0000 7f9e75262640 1 -- 172.21.15.116:0/3786372669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e70152d80 msgr2=0x7f9e70154740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:45.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.994+0000 7f9e75262640 1 --2- 172.21.15.116:0/3786372669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e70152d80 0x7f9e70154740 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f9e6800c700 tx=0x7f9e6800cbc0 comp rx=0 tx=0).stop 2024-04-25T00:22:45.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.994+0000 7f9e75262640 1 -- 172.21.15.116:0/3786372669 shutdown_connections 2024-04-25T00:22:45.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.994+0000 7f9e75262640 1 --2- 172.21.15.116:0/3786372669 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9e5c067910 0x7f9e5c069dd0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:45.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.994+0000 7f9e75262640 1 --2- 172.21.15.116:0/3786372669 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9e70152d80 0x7f9e70154740 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:45.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.994+0000 7f9e75262640 1 --2- 172.21.15.116:0/3786372669 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9e7006d5f0 0x7f9e70151e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:45.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.994+0000 7f9e75262640 1 --2- 172.21.15.116:0/3786372669 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9e7006c290 0x7f9e700aaa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:45.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.994+0000 7f9e75262640 1 -- 172.21.15.116:0/3786372669 >> 172.21.15.116:0/3786372669 conn(0x7f9e700acdd0 msgr2=0x7f9e7014da10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:45.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.994+0000 7f9e75262640 1 -- 172.21.15.116:0/3786372669 shutdown_connections 2024-04-25T00:22:45.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:45.994+0000 7f9e75262640 1 -- 172.21.15.116:0/3786372669 wait complete. 2024-04-25T00:22:46.011 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:22:48.013 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:22:48.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.146+0000 7f1709b7f640 1 Processor -- start 2024-04-25T00:22:48.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.146+0000 7f1709b7f640 1 -- start start 2024-04-25T00:22:48.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.146+0000 7f1709b7f640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f170414c720 0x7f17041563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:48.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.146+0000 7f1709b7f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1704071220 con 0x7f17040a94a0 2024-04-25T00:22:48.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.146+0000 7f1709b7f640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1704152420 con 0x7f1704071680 2024-04-25T00:22:48.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.146+0000 7f1709b7f640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f1704153460 con 0x7f170414c720 2024-04-25T00:22:48.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.146+0000 7f1702ffd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f17040a94a0 0x7f17040c2a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:42868/0 (socket says 172.21.15.116:42868) 2024-04-25T00:22:48.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.146+0000 7f1702ffd640 1 -- 172.21.15.116:0/452521978 learned_addr learned my addr 172.21.15.116:0/452521978 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:22:48.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.146+0000 7f17037fe640 1 --2- 172.21.15.116:0/452521978 >> v2:172.21.15.174:3300/0 conn(0x7f170414c720 0x7f17041563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:48.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.146+0000 7f17027fc640 1 -- 172.21.15.116:0/452521978 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 518421777 0 0) 0x7f1704071220 con 0x7f17040a94a0 2024-04-25T00:22:48.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.146+0000 7f17027fc640 1 -- 172.21.15.116:0/452521978 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f16ec003530 con 0x7f17040a94a0 2024-04-25T00:22:48.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.146+0000 7f17027fc640 1 -- 172.21.15.116:0/452521978 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 859081824 0 0) 0x7f16ec003530 con 0x7f17040a94a0 2024-04-25T00:22:48.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.146+0000 7f17027fc640 1 -- 172.21.15.116:0/452521978 >> v2:172.21.15.174:3300/0 conn(0x7f170414c720 msgr2=0x7f17041563d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:48.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.146+0000 7f17027fc640 1 --2- 172.21.15.116:0/452521978 >> v2:172.21.15.174:3300/0 conn(0x7f170414c720 0x7f17041563d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:48.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.146+0000 7f17027fc640 1 -- 172.21.15.116:0/452521978 >> v1:172.21.15.174:6789/0 conn(0x7f1704071680 legacy=0x7f1704071ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:48.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.146+0000 7f17027fc640 1 -- 172.21.15.116:0/452521978 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17040721c0 con 0x7f17040a94a0 2024-04-25T00:22:48.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.146+0000 7f17027fc640 1 -- 172.21.15.116:0/452521978 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f16f8001680 con 0x7f17040a94a0 2024-04-25T00:22:48.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.146+0000 7f17027fc640 1 -- 172.21.15.116:0/452521978 >> v1:172.21.15.116:6789/0 conn(0x7f17040a94a0 legacy=0x7f17040c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:48.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.146+0000 7f17027fc640 1 --2- 172.21.15.116:0/452521978 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f16ec0039c0 0x7f16ec003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:48.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.146+0000 7f17027fc640 1 --2- 172.21.15.116:0/452521978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16ec004810 0x7f16ec004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:48.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.146+0000 7f17027fc640 1 --2- 172.21.15.116:0/452521978 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f16ec0051d0 0x7f16ec00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:48.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.146+0000 7f17027fc640 1 -- 172.21.15.116:0/452521978 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1704153460 con 0x7f16ec004810 2024-04-25T00:22:48.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.146+0000 7f17027fc640 1 -- 172.21.15.116:0/452521978 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f16ec00c170 con 0x7f16ec0051d0 2024-04-25T00:22:48.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.146+0000 7f17027fc640 1 -- 172.21.15.116:0/452521978 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f16ec00c370 con 0x7f16ec0039c0 2024-04-25T00:22:48.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.146+0000 7f1702ffd640 1 --2- 172.21.15.116:0/452521978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16ec004810 0x7f16ec004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:48.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.146+0000 7f1702ffd640 1 -- 172.21.15.116:0/452521978 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f16ec0039c0 msgr2=0x7f16ec003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:48.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.146+0000 7f1702ffd640 1 --2- 172.21.15.116:0/452521978 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f16ec0039c0 0x7f16ec003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:48.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.146+0000 7f1702ffd640 1 -- 172.21.15.116:0/452521978 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f16ec0051d0 msgr2=0x7f16ec00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:48.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.146+0000 7f1702ffd640 1 --2- 172.21.15.116:0/452521978 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f16ec0051d0 0x7f16ec00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:48.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.146+0000 7f1702ffd640 1 -- 172.21.15.116:0/452521978 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16ec003530 con 0x7f16ec004810 2024-04-25T00:22:48.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.146+0000 7f1703fff640 1 -- 172.21.15.116:0/452521978 reap_dead start 2024-04-25T00:22:48.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.150+0000 7f1702ffd640 1 --2- 172.21.15.116:0/452521978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16ec004810 0x7f16ec004c90 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f16f800c210 tx=0x7f16f801cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=821eefbdf1a1964a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:48.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.150+0000 7f17027fc640 1 -- 172.21.15.116:0/452521978 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f8007d40 con 0x7f16ec004810 2024-04-25T00:22:48.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.150+0000 7f17027fc640 1 -- 172.21.15.116:0/452521978 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16f80a9070 con 0x7f16ec004810 2024-04-25T00:22:48.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.150+0000 7f17027fc640 1 -- 172.21.15.116:0/452521978 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f80a5e20 con 0x7f16ec004810 2024-04-25T00:22:48.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.150+0000 7f1709b7f640 1 -- 172.21.15.116:0/452521978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16ec004810 msgr2=0x7f16ec004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:48.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.150+0000 7f1709b7f640 1 --2- 172.21.15.116:0/452521978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16ec004810 0x7f16ec004c90 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f16f800c210 tx=0x7f16f801cca0 comp rx=0 tx=0).stop 2024-04-25T00:22:48.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.150+0000 7f1709b7f640 1 -- 172.21.15.116:0/452521978 shutdown_connections 2024-04-25T00:22:48.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.150+0000 7f1709b7f640 1 --2- 172.21.15.116:0/452521978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16ec004810 0x7f16ec004c90 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:48.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.150+0000 7f1709b7f640 1 -- 172.21.15.116:0/452521978 >> 172.21.15.116:0/452521978 conn(0x7f17040c3b00 msgr2=0x7f17040c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:48.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.150+0000 7f1709b7f640 1 -- 172.21.15.116:0/452521978 shutdown_connections 2024-04-25T00:22:48.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.150+0000 7f1709b7f640 1 -- 172.21.15.116:0/452521978 wait complete. 2024-04-25T00:22:48.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.150+0000 7f1709b7f640 1 Processor -- start 2024-04-25T00:22:48.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.150+0000 7f1709b7f640 1 -- start start 2024-04-25T00:22:48.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.150+0000 7f1709b7f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f16ec004810 0x7f1704140f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:48.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.150+0000 7f1709b7f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1704143cf0 0x7f170414a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:48.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.150+0000 7f1709b7f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f170414aaf0 0x7f17041f1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:48.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.150+0000 7f1709b7f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1704152980 con 0x7f170414aaf0 2024-04-25T00:22:48.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.150+0000 7f1709b7f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f17040c3220 con 0x7f16ec004810 2024-04-25T00:22:48.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.150+0000 7f1709b7f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f1704152c60 con 0x7f1704143cf0 2024-04-25T00:22:48.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.150+0000 7f1703fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f170414aaf0 0x7f17041f1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:48.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.150+0000 7f1703fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f170414aaf0 0x7f17041f1ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39684/0 (socket says 172.21.15.116:39684) 2024-04-25T00:22:48.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.150+0000 7f1703fff640 1 -- 172.21.15.116:0/2500002860 learned_addr learned my addr 172.21.15.116:0/2500002860 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:22:48.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.154+0000 7f17037fe640 1 --2- 172.21.15.116:0/2500002860 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f16ec004810 0x7f1704140f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:48.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.154+0000 7f1702ffd640 1 --2- 172.21.15.116:0/2500002860 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1704143cf0 0x7f170414a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:48.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.154+0000 7f1703fff640 1 -- 172.21.15.116:0/2500002860 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1704143cf0 msgr2=0x7f170414a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:48.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.154+0000 7f1703fff640 1 --2- 172.21.15.116:0/2500002860 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1704143cf0 0x7f170414a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:48.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.154+0000 7f1703fff640 1 -- 172.21.15.116:0/2500002860 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f16ec004810 msgr2=0x7f1704140f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:48.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.154+0000 7f1703fff640 1 --2- 172.21.15.116:0/2500002860 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f16ec004810 0x7f1704140f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:48.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.154+0000 7f1703fff640 1 -- 172.21.15.116:0/2500002860 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17040c6200 con 0x7f170414aaf0 2024-04-25T00:22:48.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.154+0000 7f1703fff640 1 --2- 172.21.15.116:0/2500002860 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f170414aaf0 0x7f17041f1ce0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f17040c35f0 tx=0x7f16fc00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:48.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.154+0000 7f16ebfff640 1 -- 172.21.15.116:0/2500002860 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16fc007070 con 0x7f170414aaf0 2024-04-25T00:22:48.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.154+0000 7f16ebfff640 1 -- 172.21.15.116:0/2500002860 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16fc007230 con 0x7f170414aaf0 2024-04-25T00:22:48.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.154+0000 7f16ebfff640 1 -- 172.21.15.116:0/2500002860 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16fc007620 con 0x7f170414aaf0 2024-04-25T00:22:48.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.154+0000 7f1709b7f640 1 -- 172.21.15.116:0/2500002860 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1704144500 con 0x7f170414aaf0 2024-04-25T00:22:48.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.154+0000 7f1709b7f640 1 -- 172.21.15.116:0/2500002860 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1704147510 con 0x7f170414aaf0 2024-04-25T00:22:48.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.154+0000 7f1709b7f640 1 -- 172.21.15.116:0/2500002860 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1704068350 con 0x7f170414aaf0 2024-04-25T00:22:48.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.154+0000 7f16ebfff640 1 -- 172.21.15.116:0/2500002860 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f16fc011070 con 0x7f170414aaf0 2024-04-25T00:22:48.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.154+0000 7f16ebfff640 1 --2- 172.21.15.116:0/2500002860 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f16f00678f0 0x7f16f0069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:48.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.154+0000 7f16ebfff640 1 -- 172.21.15.116:0/2500002860 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9590+0+0 (secure 0 0 0) 0x7f16fc0861c0 con 0x7f170414aaf0 2024-04-25T00:22:48.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.154+0000 7f17037fe640 1 --2- 172.21.15.116:0/2500002860 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f16f00678f0 0x7f16f0069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:48.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.158+0000 7f17037fe640 1 --2- 172.21.15.116:0/2500002860 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f16f00678f0 0x7f16f0069db0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f16f40047c0 tx=0x7f16f4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:48.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.158+0000 7f16ebfff640 1 -- 172.21.15.116:0/2500002860 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f16fc019060 con 0x7f170414aaf0 2024-04-25T00:22:48.321 INFO:tasks.mds_thrash.fs.[b]:reviving mds.a 2024-04-25T00:22:48.321 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-25T00:22:48.321 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-25T00:22:48.324 INFO:tasks.ceph.mds.a:Started 2024-04-25T00:22:48.324 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:22:48.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.358+0000 7f1709b7f640 1 -- 172.21.15.116:0/2500002860 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f17040c3640 con 0x7f170414aaf0 2024-04-25T00:22:48.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.358+0000 7f16ebfff640 1 -- 172.21.15.116:0/2500002860 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 181 v181) ==== 95+0+5674 (secure 0 0 0) 0x7f1704152420 con 0x7f170414aaf0 2024-04-25T00:22:48.358 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:22:48.359 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":181,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:22:13.925793+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7989},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7989":{"gid":7989,"name":"c","rank":0,"incarnation":173,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6835/221971311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":221971311},{"type":"v1","addr":"172.21.15.116:6835","nonce":221971311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7989,"qdb_cluster":[7989]},"id":1},{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:22:38.936345+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6230":{"gid":6230,"name":"d","rank":0,"incarnation":178,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/492366509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":492366509},{"type":"v1","addr":"172.21.15.174:6837","nonce":492366509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6230,"qdb_cluster":[6230]},"id":2}]} 2024-04-25T00:22:48.359 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 181 2024-04-25T00:22:48.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.358+0000 7f1709b7f640 1 -- 172.21.15.116:0/2500002860 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f16f00678f0 msgr2=0x7f16f0069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:48.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.358+0000 7f1709b7f640 1 --2- 172.21.15.116:0/2500002860 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f16f00678f0 0x7f16f0069db0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f16f40047c0 tx=0x7f16f4008040 comp rx=0 tx=0).stop 2024-04-25T00:22:48.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.358+0000 7f1709b7f640 1 -- 172.21.15.116:0/2500002860 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f170414aaf0 msgr2=0x7f17041f1ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:48.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.358+0000 7f1709b7f640 1 --2- 172.21.15.116:0/2500002860 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f170414aaf0 0x7f17041f1ce0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f17040c35f0 tx=0x7f16fc00c960 comp rx=0 tx=0).stop 2024-04-25T00:22:48.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.358+0000 7f1709b7f640 1 -- 172.21.15.116:0/2500002860 shutdown_connections 2024-04-25T00:22:48.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.358+0000 7f1709b7f640 1 --2- 172.21.15.116:0/2500002860 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f16f00678f0 0x7f16f0069db0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:48.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.358+0000 7f1709b7f640 1 --2- 172.21.15.116:0/2500002860 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f170414aaf0 0x7f17041f1ce0 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:48.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.358+0000 7f1709b7f640 1 --2- 172.21.15.116:0/2500002860 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1704143cf0 0x7f170414a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:48.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.358+0000 7f1709b7f640 1 --2- 172.21.15.116:0/2500002860 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f16ec004810 0x7f1704140f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:48.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.358+0000 7f1709b7f640 1 -- 172.21.15.116:0/2500002860 >> 172.21.15.116:0/2500002860 conn(0x7f17040c3b00 msgr2=0x7f1704145ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:48.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.358+0000 7f1709b7f640 1 -- 172.21.15.116:0/2500002860 shutdown_connections 2024-04-25T00:22:48.361 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.358+0000 7f1709b7f640 1 -- 172.21.15.116:0/2500002860 wait complete. 2024-04-25T00:22:48.379 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:22:48.394 INFO:tasks.ceph.mds.a.smithi116.stdout:starting mds.a at 2024-04-25T00:22:48.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.438+0000 7fa4d9d51640 1 Processor -- start 2024-04-25T00:22:48.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.438+0000 7fa4d9d51640 1 -- start start 2024-04-25T00:22:48.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.438+0000 7fa4d9d51640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fa4d406d5f0 0x7fa4d4156c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:48.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.438+0000 7fa4d9d51640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa4d4155810 0x7fa4d41571c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:48.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.438+0000 7fa4d9d51640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4d40c9150 con 0x7fa4d4152d80 2024-04-25T00:22:48.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.438+0000 7fa4d9d51640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fa4d40c9d60 con 0x7fa4d406d5f0 2024-04-25T00:22:48.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.438+0000 7fa4d9d51640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa4d40c9430 con 0x7fa4d4155810 2024-04-25T00:22:48.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.438+0000 7fa4caffd640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fa4d406d5f0 0x7fa4d4156c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:48.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.438+0000 7fa4d3fff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa4d4155810 0x7fa4d41571c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:48.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.438+0000 7fa4d37fe640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fa4d4152d80 0x7fa4d406bd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:57904/0 (socket says 172.21.15.116:57904) 2024-04-25T00:22:48.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.438+0000 7fa4d37fe640 1 -- 172.21.15.116:0/1091815748 learned_addr learned my addr 172.21.15.116:0/1091815748 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:22:48.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.438+0000 7fa4caffd640 1 -- 172.21.15.116:0/1091815748 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa4d4155810 msgr2=0x7fa4d41571c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:48.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.438+0000 7fa4caffd640 1 --2- 172.21.15.116:0/1091815748 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa4d4155810 0x7fa4d41571c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:48.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.438+0000 7fa4caffd640 1 -- 172.21.15.116:0/1091815748 >> v1:172.21.15.174:6789/0 conn(0x7fa4d4152d80 legacy=0x7fa4d406bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:48.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.438+0000 7fa4d2ffd640 1 -- 172.21.15.116:0/1091815748 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3114889846 0 0) 0x7fa4d40c9150 con 0x7fa4d4152d80 2024-04-25T00:22:48.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.438+0000 7fa4caffd640 1 -- 172.21.15.116:0/1091815748 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4d40c9710 con 0x7fa4d406d5f0 2024-04-25T00:22:48.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.442+0000 7fa4caffd640 1 --2- 172.21.15.116:0/1091815748 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa4d406d5f0 0x7fa4d4156c80 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc0097c0 tx=0x7fa4bc01c3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5aaae51428a11907 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:48.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.442+0000 7fa4d2ffd640 1 -- 172.21.15.116:0/1091815748 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4bc0b1070 con 0x7fa4d406d5f0 2024-04-25T00:22:48.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.442+0000 7fa4d2ffd640 1 -- 172.21.15.116:0/1091815748 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4bc0ac640 con 0x7fa4d406d5f0 2024-04-25T00:22:48.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.442+0000 7fa4d2ffd640 1 -- 172.21.15.116:0/1091815748 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4bc0aca10 con 0x7fa4d406d5f0 2024-04-25T00:22:48.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.442+0000 7fa4d9d51640 1 -- 172.21.15.116:0/1091815748 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa4d406d5f0 msgr2=0x7fa4d4156c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:48.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.442+0000 7fa4d9d51640 1 --2- 172.21.15.116:0/1091815748 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa4d406d5f0 0x7fa4d4156c80 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc0097c0 tx=0x7fa4bc01c3e0 comp rx=0 tx=0).stop 2024-04-25T00:22:48.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.442+0000 7fa4d9d51640 1 -- 172.21.15.116:0/1091815748 shutdown_connections 2024-04-25T00:22:48.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.442+0000 7fa4d9d51640 1 --2- 172.21.15.116:0/1091815748 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa4d4155810 0x7fa4d41571c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:48.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.442+0000 7fa4d9d51640 1 --2- 172.21.15.116:0/1091815748 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa4d406d5f0 0x7fa4d4156c80 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:48.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.442+0000 7fa4d9d51640 1 -- 172.21.15.116:0/1091815748 >> 172.21.15.116:0/1091815748 conn(0x7fa4d40acdd0 msgr2=0x7fa4d40ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:48.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.442+0000 7fa4d9d51640 1 -- 172.21.15.116:0/1091815748 shutdown_connections 2024-04-25T00:22:48.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.442+0000 7fa4d9d51640 1 -- 172.21.15.116:0/1091815748 wait complete. 2024-04-25T00:22:48.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.446+0000 7fa4d9d51640 1 Processor -- start 2024-04-25T00:22:48.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.446+0000 7fa4d9d51640 1 -- start start 2024-04-25T00:22:48.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.446+0000 7fa4d9d51640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa4d406d5f0 0x7fa4d4154640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:48.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.446+0000 7fa4d9d51640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa4d4152d80 0x7fa4d40ae680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:48.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.446+0000 7fa4d9d51640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4d4155810 0x7fa4d40aebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:48.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.446+0000 7fa4d9d51640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d40caf00 con 0x7fa4d4155810 2024-04-25T00:22:48.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.446+0000 7fa4d9d51640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa4d40cad80 con 0x7fa4d4152d80 2024-04-25T00:22:48.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.446+0000 7fa4d9d51640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa4d41628d0 con 0x7fa4d406d5f0 2024-04-25T00:22:48.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.446+0000 7fa4d3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4d4155810 0x7fa4d40aebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:48.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.446+0000 7fa4d3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4d4155810 0x7fa4d40aebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39694/0 (socket says 172.21.15.116:39694) 2024-04-25T00:22:48.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.446+0000 7fa4d3fff640 1 -- 172.21.15.116:0/2721129354 learned_addr learned my addr 172.21.15.116:0/2721129354 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:22:48.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.446+0000 7fa4d37fe640 1 --2- 172.21.15.116:0/2721129354 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa4d406d5f0 0x7fa4d4154640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:48.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.446+0000 7fa4caffd640 1 --2- 172.21.15.116:0/2721129354 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa4d4152d80 0x7fa4d40ae680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:48.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.446+0000 7fa4d3fff640 1 -- 172.21.15.116:0/2721129354 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa4d406d5f0 msgr2=0x7fa4d4154640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:48.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.446+0000 7fa4d3fff640 1 --2- 172.21.15.116:0/2721129354 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa4d406d5f0 0x7fa4d4154640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:48.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.446+0000 7fa4d3fff640 1 -- 172.21.15.116:0/2721129354 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa4d4152d80 msgr2=0x7fa4d40ae680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:48.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.446+0000 7fa4d3fff640 1 --2- 172.21.15.116:0/2721129354 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa4d4152d80 0x7fa4d40ae680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:48.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.446+0000 7fa4d3fff640 1 -- 172.21.15.116:0/2721129354 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4d40ab290 con 0x7fa4d4155810 2024-04-25T00:22:48.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.446+0000 7fa4d3fff640 1 --2- 172.21.15.116:0/2721129354 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4d4155810 0x7fa4d40aebc0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7fa4cc009dd0 tx=0x7fa4cc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:48.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.446+0000 7fa4d0ff9640 1 -- 172.21.15.116:0/2721129354 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4cc018020 con 0x7fa4d4155810 2024-04-25T00:22:48.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.446+0000 7fa4d0ff9640 1 -- 172.21.15.116:0/2721129354 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4cc014480 con 0x7fa4d4155810 2024-04-25T00:22:48.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.446+0000 7fa4d9d51640 1 -- 172.21.15.116:0/2721129354 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4d40ab570 con 0x7fa4d4155810 2024-04-25T00:22:48.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.446+0000 7fa4d0ff9640 1 -- 172.21.15.116:0/2721129354 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4cc014870 con 0x7fa4d4155810 2024-04-25T00:22:48.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.446+0000 7fa4d9d51640 1 -- 172.21.15.116:0/2721129354 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4d40abb30 con 0x7fa4d4155810 2024-04-25T00:22:48.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.446+0000 7fa4d9d51640 1 -- 172.21.15.116:0/2721129354 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4d4068350 con 0x7fa4d4155810 2024-04-25T00:22:48.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.446+0000 7fa4d0ff9640 1 -- 172.21.15.116:0/2721129354 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa4cc01b020 con 0x7fa4d4155810 2024-04-25T00:22:48.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.450+0000 7fa4d0ff9640 1 --2- 172.21.15.116:0/2721129354 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa4b40678f0 0x7fa4b4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:48.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.450+0000 7fa4d0ff9640 1 -- 172.21.15.116:0/2721129354 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 9590+0+0 (secure 0 0 0) 0x7fa4cc087720 con 0x7fa4d4155810 2024-04-25T00:22:48.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.450+0000 7fa4d37fe640 1 --2- 172.21.15.116:0/2721129354 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa4b40678f0 0x7fa4b4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:48.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.450+0000 7fa4d37fe640 1 --2- 172.21.15.116:0/2721129354 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa4b40678f0 0x7fa4b4069db0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fa4c4000f70 tx=0x7fa4c4007d10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:48.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.454+0000 7fa4d0ff9640 1 -- 172.21.15.116:0/2721129354 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fa4cc022070 con 0x7fa4d4155810 2024-04-25T00:22:48.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.642+0000 7fa4d9d51640 1 -- 172.21.15.116:0/2721129354 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa4d40cab60 con 0x7fa4d4155810 2024-04-25T00:22:48.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.642+0000 7fa4d0ff9640 1 -- 172.21.15.116:0/2721129354 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 181 v181) ==== 95+0+5674 (secure 0 0 0) 0x7fa4cc010040 con 0x7fa4d4155810 2024-04-25T00:22:48.645 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:22:48.645 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":181,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:22:13.925793+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7989},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7989":{"gid":7989,"name":"c","rank":0,"incarnation":173,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6835/221971311","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":221971311},{"type":"v1","addr":"172.21.15.116:6835","nonce":221971311}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7989,"qdb_cluster":[7989]},"id":1},{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:22:38.936345+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6230":{"gid":6230,"name":"d","rank":0,"incarnation":178,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/492366509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":492366509},{"type":"v1","addr":"172.21.15.174:6837","nonce":492366509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6230,"qdb_cluster":[6230]},"id":2}]} 2024-04-25T00:22:48.645 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 181 2024-04-25T00:22:48.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.646+0000 7fa4d9d51640 1 -- 172.21.15.116:0/2721129354 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa4b40678f0 msgr2=0x7fa4b4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:48.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.646+0000 7fa4d9d51640 1 --2- 172.21.15.116:0/2721129354 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa4b40678f0 0x7fa4b4069db0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fa4c4000f70 tx=0x7fa4c4007d10 comp rx=0 tx=0).stop 2024-04-25T00:22:48.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.646+0000 7fa4d9d51640 1 -- 172.21.15.116:0/2721129354 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4d4155810 msgr2=0x7fa4d40aebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:48.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.646+0000 7fa4d9d51640 1 --2- 172.21.15.116:0/2721129354 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4d4155810 0x7fa4d40aebc0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7fa4cc009dd0 tx=0x7fa4cc00c490 comp rx=0 tx=0).stop 2024-04-25T00:22:48.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.646+0000 7fa4d9d51640 1 -- 172.21.15.116:0/2721129354 shutdown_connections 2024-04-25T00:22:48.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.646+0000 7fa4d9d51640 1 --2- 172.21.15.116:0/2721129354 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa4b40678f0 0x7fa4b4069db0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:48.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.646+0000 7fa4d9d51640 1 --2- 172.21.15.116:0/2721129354 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa4d4155810 0x7fa4d40aebc0 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:48.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.646+0000 7fa4d9d51640 1 --2- 172.21.15.116:0/2721129354 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa4d4152d80 0x7fa4d40ae680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:48.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.646+0000 7fa4d9d51640 1 --2- 172.21.15.116:0/2721129354 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa4d406d5f0 0x7fa4d4154640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:48.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.646+0000 7fa4d9d51640 1 -- 172.21.15.116:0/2721129354 >> 172.21.15.116:0/2721129354 conn(0x7fa4d40acdd0 msgr2=0x7fa4d415f470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:48.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.646+0000 7fa4d9d51640 1 -- 172.21.15.116:0/2721129354 shutdown_connections 2024-04-25T00:22:48.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:48.646+0000 7fa4d9d51640 1 -- 172.21.15.116:0/2721129354 wait complete. 2024-04-25T00:22:48.663 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2024-04-25T00:22:50.380 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:22:50.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.514+0000 7f4985981640 1 Processor -- start 2024-04-25T00:22:50.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.514+0000 7f4985981640 1 -- start start 2024-04-25T00:22:50.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.514+0000 7f4985981640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f49800a94a0 0x7f49801563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:50.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.514+0000 7f4985981640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4980071220 con 0x7f4980071680 2024-04-25T00:22:50.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.514+0000 7f4985981640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49800721e0 con 0x7f498014c720 2024-04-25T00:22:50.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.514+0000 7f4985981640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f4980153460 con 0x7f49800a94a0 2024-04-25T00:22:50.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.514+0000 7f497f7fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f4980071680 0x7f4980071ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:42886/0 (socket says 172.21.15.116:42886) 2024-04-25T00:22:50.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.514+0000 7f497f7fe640 1 -- 172.21.15.116:0/3707799555 learned_addr learned my addr 172.21.15.116:0/3707799555 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:22:50.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.514+0000 7f497e7fc640 1 --2- 172.21.15.116:0/3707799555 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f49800a94a0 0x7f49801563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:50.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.514+0000 7f497dffb640 1 -- 172.21.15.116:0/3707799555 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1463075150 0 0) 0x7f4980071220 con 0x7f4980071680 2024-04-25T00:22:50.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.514+0000 7f497dffb640 1 -- 172.21.15.116:0/3707799555 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4968003530 con 0x7f4980071680 2024-04-25T00:22:50.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.514+0000 7f497dffb640 1 -- 172.21.15.116:0/3707799555 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1886778144 0 0) 0x7f4968003530 con 0x7f4980071680 2024-04-25T00:22:50.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.514+0000 7f497dffb640 1 -- 172.21.15.116:0/3707799555 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f49800a94a0 msgr2=0x7f49801563d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:50.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.514+0000 7f497dffb640 1 --2- 172.21.15.116:0/3707799555 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f49800a94a0 0x7f49801563d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:50.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.514+0000 7f497dffb640 1 -- 172.21.15.116:0/3707799555 >> v1:172.21.15.174:6789/0 conn(0x7f498014c720 legacy=0x7f49800c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:50.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.514+0000 7f497dffb640 1 -- 172.21.15.116:0/3707799555 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4980152420 con 0x7f4980071680 2024-04-25T00:22:50.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.514+0000 7f497dffb640 1 -- 172.21.15.116:0/3707799555 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f4978001a00 con 0x7f4980071680 2024-04-25T00:22:50.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.514+0000 7f497dffb640 1 -- 172.21.15.116:0/3707799555 >> v1:172.21.15.116:6789/0 conn(0x7f4980071680 legacy=0x7f4980071ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:50.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.514+0000 7f497dffb640 1 --2- 172.21.15.116:0/3707799555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49680039f0 0x7f4968003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:50.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.514+0000 7f497dffb640 1 --2- 172.21.15.116:0/3707799555 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f49800a94a0 0x7f4968004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:50.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.514+0000 7f497dffb640 1 --2- 172.21.15.116:0/3707799555 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4968004d50 0x7f49680051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:50.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.514+0000 7f497dffb640 1 -- 172.21.15.116:0/3707799555 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4980153460 con 0x7f49680039f0 2024-04-25T00:22:50.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.514+0000 7f497dffb640 1 -- 172.21.15.116:0/3707799555 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f496800bb30 con 0x7f4968004d50 2024-04-25T00:22:50.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.514+0000 7f497dffb640 1 -- 172.21.15.116:0/3707799555 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f496800bd30 con 0x7f49800a94a0 2024-04-25T00:22:50.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.514+0000 7f497effd640 1 --2- 172.21.15.116:0/3707799555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49680039f0 0x7f4968003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:50.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.514+0000 7f497f7fe640 1 --2- 172.21.15.116:0/3707799555 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4968004d50 0x7f49680051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:50.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.514+0000 7f497e7fc640 1 --2- 172.21.15.116:0/3707799555 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f49800a94a0 0x7f4968004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:50.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.514+0000 7f497effd640 1 -- 172.21.15.116:0/3707799555 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f49800a94a0 msgr2=0x7f4968004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:50.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.514+0000 7f497effd640 1 --2- 172.21.15.116:0/3707799555 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f49800a94a0 0x7f4968004810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:50.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.514+0000 7f497effd640 1 -- 172.21.15.116:0/3707799555 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4968004d50 msgr2=0x7f49680051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:50.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.514+0000 7f497effd640 1 --2- 172.21.15.116:0/3707799555 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4968004d50 0x7f49680051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:50.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.514+0000 7f497effd640 1 -- 172.21.15.116:0/3707799555 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4968003530 con 0x7f49680039f0 2024-04-25T00:22:50.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.514+0000 7f497f7fe640 1 --2- 172.21.15.116:0/3707799555 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4968004d50 0x7f49680051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:22:50.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.518+0000 7f497effd640 1 --2- 172.21.15.116:0/3707799555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49680039f0 0x7f4968003e10 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f4970001780 tx=0x7f497001c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4359781f4055e1a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:50.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.518+0000 7f497dffb640 1 -- 172.21.15.116:0/3707799555 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49700013a0 con 0x7f49680039f0 2024-04-25T00:22:50.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.518+0000 7f497dffb640 1 -- 172.21.15.116:0/3707799555 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49700052f0 con 0x7f49680039f0 2024-04-25T00:22:50.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.518+0000 7f497dffb640 1 -- 172.21.15.116:0/3707799555 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4970003840 con 0x7f49680039f0 2024-04-25T00:22:50.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.518+0000 7f4985981640 1 -- 172.21.15.116:0/3707799555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49680039f0 msgr2=0x7f4968003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:50.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.518+0000 7f4985981640 1 --2- 172.21.15.116:0/3707799555 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49680039f0 0x7f4968003e10 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f4970001780 tx=0x7f497001c360 comp rx=0 tx=0).stop 2024-04-25T00:22:50.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.518+0000 7f497f7fe640 1 -- 172.21.15.116:0/3707799555 reap_dead start 2024-04-25T00:22:50.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.518+0000 7f4985981640 1 -- 172.21.15.116:0/3707799555 shutdown_connections 2024-04-25T00:22:50.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.518+0000 7f4985981640 1 -- 172.21.15.116:0/3707799555 >> 172.21.15.116:0/3707799555 conn(0x7f49800c3b00 msgr2=0x7f49800c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:50.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.518+0000 7f4985981640 1 -- 172.21.15.116:0/3707799555 shutdown_connections 2024-04-25T00:22:50.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.518+0000 7f4985981640 1 -- 172.21.15.116:0/3707799555 wait complete. 2024-04-25T00:22:50.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.518+0000 7f4985981640 1 Processor -- start 2024-04-25T00:22:50.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.518+0000 7f4985981640 1 -- start start 2024-04-25T00:22:50.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.518+0000 7f4985981640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f49680039f0 0x7f4980142080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:50.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.518+0000 7f4985981640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f49801425c0 0x7f4980143f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:50.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.518+0000 7f4985981640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4980144490 0x7f498013e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:50.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.518+0000 7f4985981640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4980152c60 con 0x7f4980144490 2024-04-25T00:22:50.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.518+0000 7f4985981640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4980152980 con 0x7f49801425c0 2024-04-25T00:22:50.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.518+0000 7f4985981640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f4980152f40 con 0x7f49680039f0 2024-04-25T00:22:50.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.518+0000 7f497f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4980144490 0x7f498013e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:50.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.518+0000 7f497f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4980144490 0x7f498013e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39714/0 (socket says 172.21.15.116:39714) 2024-04-25T00:22:50.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.518+0000 7f497f7fe640 1 -- 172.21.15.116:0/4035476498 learned_addr learned my addr 172.21.15.116:0/4035476498 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:22:50.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.518+0000 7f497e7fc640 1 --2- 172.21.15.116:0/4035476498 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f49801425c0 0x7f4980143f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:50.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.518+0000 7f497effd640 1 --2- 172.21.15.116:0/4035476498 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f49680039f0 0x7f4980142080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:50.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.518+0000 7f497f7fe640 1 -- 172.21.15.116:0/4035476498 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f49680039f0 msgr2=0x7f4980142080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:50.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.518+0000 7f497f7fe640 1 --2- 172.21.15.116:0/4035476498 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f49680039f0 0x7f4980142080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:50.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.518+0000 7f497f7fe640 1 -- 172.21.15.116:0/4035476498 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f49801425c0 msgr2=0x7f4980143f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:50.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.518+0000 7f497f7fe640 1 --2- 172.21.15.116:0/4035476498 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f49801425c0 0x7f4980143f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:50.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.518+0000 7f497f7fe640 1 -- 172.21.15.116:0/4035476498 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f498013f8f0 con 0x7f4980144490 2024-04-25T00:22:50.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.522+0000 7f497f7fe640 1 --2- 172.21.15.116:0/4035476498 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4980144490 0x7f498013e360 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f4980070620 tx=0x7f497800b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:50.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.522+0000 7f4967fff640 1 -- 172.21.15.116:0/4035476498 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f497800de90 con 0x7f4980144490 2024-04-25T00:22:50.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.522+0000 7f4967fff640 1 -- 172.21.15.116:0/4035476498 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f497800bb80 con 0x7f4980144490 2024-04-25T00:22:50.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.522+0000 7f4967fff640 1 -- 172.21.15.116:0/4035476498 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4978007070 con 0x7f4980144490 2024-04-25T00:22:50.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.522+0000 7f4985981640 1 -- 172.21.15.116:0/4035476498 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49801414c0 con 0x7f4980144490 2024-04-25T00:22:50.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.522+0000 7f4985981640 1 -- 172.21.15.116:0/4035476498 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49801f0760 con 0x7f4980144490 2024-04-25T00:22:50.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.522+0000 7f4985981640 1 -- 172.21.15.116:0/4035476498 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49801f02b0 con 0x7f4980144490 2024-04-25T00:22:50.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.522+0000 7f4967fff640 1 -- 172.21.15.116:0/4035476498 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f497801b020 con 0x7f4980144490 2024-04-25T00:22:50.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.522+0000 7f4967fff640 1 --2- 172.21.15.116:0/4035476498 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f496c067a90 0x7f496c069f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:50.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.522+0000 7f4967fff640 1 -- 172.21.15.116:0/4035476498 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 9676+0+0 (secure 0 0 0) 0x7f4978086e40 con 0x7f4980144490 2024-04-25T00:22:50.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.522+0000 7f497effd640 1 --2- 172.21.15.116:0/4035476498 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f496c067a90 0x7f496c069f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:50.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.522+0000 7f497effd640 1 --2- 172.21.15.116:0/4035476498 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f496c067a90 0x7f496c069f50 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f4970001750 tx=0x7f49700011a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:50.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.526+0000 7f4967fff640 1 -- 172.21.15.116:0/4035476498 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f4980152420 con 0x7f4980144490 2024-04-25T00:22:50.665 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:22:50.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.726+0000 7f4985981640 1 -- 172.21.15.116:0/4035476498 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f498013fac0 con 0x7f4980144490 2024-04-25T00:22:50.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.726+0000 7f4967fff640 1 -- 172.21.15.116:0/4035476498 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 184 v184) ==== 95+0+5672 (secure 0 0 0) 0x7f49780569b0 con 0x7f4980144490 2024-04-25T00:22:50.729 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:22:50.729 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":184,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":184,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:22:49.315650+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8214},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":183,"state":"up:reconnect","state_seq":2,"addr":"172.21.15.116:6833/2998042514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2998042514},{"type":"v1","addr":"172.21.15.116:6833","nonce":2998042514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:22:38.936345+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6230":{"gid":6230,"name":"d","rank":0,"incarnation":178,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/492366509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":492366509},{"type":"v1","addr":"172.21.15.174:6837","nonce":492366509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6230,"qdb_cluster":[6230]},"id":2}]} 2024-04-25T00:22:50.729 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 184 2024-04-25T00:22:50.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.730+0000 7f4985981640 1 -- 172.21.15.116:0/4035476498 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f496c067a90 msgr2=0x7f496c069f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:50.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.730+0000 7f4985981640 1 --2- 172.21.15.116:0/4035476498 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f496c067a90 0x7f496c069f50 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f4970001750 tx=0x7f49700011a0 comp rx=0 tx=0).stop 2024-04-25T00:22:50.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.730+0000 7f4985981640 1 -- 172.21.15.116:0/4035476498 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4980144490 msgr2=0x7f498013e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:50.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.730+0000 7f4985981640 1 --2- 172.21.15.116:0/4035476498 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4980144490 0x7f498013e360 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f4980070620 tx=0x7f497800b4a0 comp rx=0 tx=0).stop 2024-04-25T00:22:50.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.730+0000 7f4985981640 1 -- 172.21.15.116:0/4035476498 shutdown_connections 2024-04-25T00:22:50.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.730+0000 7f4985981640 1 --2- 172.21.15.116:0/4035476498 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f496c067a90 0x7f496c069f50 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:50.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.730+0000 7f4985981640 1 --2- 172.21.15.116:0/4035476498 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4980144490 0x7f498013e360 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:50.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.730+0000 7f4985981640 1 --2- 172.21.15.116:0/4035476498 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f49801425c0 0x7f4980143f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:50.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.730+0000 7f4985981640 1 --2- 172.21.15.116:0/4035476498 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f49680039f0 0x7f4980142080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:50.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.730+0000 7f4985981640 1 -- 172.21.15.116:0/4035476498 >> 172.21.15.116:0/4035476498 conn(0x7f49800c3b00 msgr2=0x7f4980071a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:50.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.730+0000 7f4985981640 1 -- 172.21.15.116:0/4035476498 shutdown_connections 2024-04-25T00:22:50.732 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.730+0000 7f4985981640 1 -- 172.21.15.116:0/4035476498 wait complete. 2024-04-25T00:22:50.748 INFO:tasks.mds_thrash.fs.[a]:mds.c down, removed from mdsmap 2024-04-25T00:22:50.748 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-04-25T00:22:50.749 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:22:50.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.778+0000 7fe979518640 1 Processor -- start 2024-04-25T00:22:50.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.778+0000 7fe979518640 1 -- start start 2024-04-25T00:22:50.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.778+0000 7fe979518640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe9740c5200 0x7fe974154040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:50.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.778+0000 7fe979518640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fe974156bb0 0x7fe974157040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:50.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.778+0000 7fe979518640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9740c2b40 con 0x7fe9740c4760 2024-04-25T00:22:50.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.778+0000 7fe979518640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fe974072b80 con 0x7fe974156bb0 2024-04-25T00:22:50.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.778+0000 7fe979518640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe974154580 con 0x7fe9740c5200 2024-04-25T00:22:50.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.778+0000 7fe9727fc640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fe9740c4760 0x7fe97414d230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:42892/0 (socket says 172.21.15.116:42892) 2024-04-25T00:22:50.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.778+0000 7fe9727fc640 1 -- 172.21.15.116:0/4168903050 learned_addr learned my addr 172.21.15.116:0/4168903050 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:22:50.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.782+0000 7fe9737fe640 1 --2- 172.21.15.116:0/4168903050 >> v2:172.21.15.174:3300/0 conn(0x7fe974156bb0 0x7fe974157040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:50.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.782+0000 7fe972ffd640 1 --2- 172.21.15.116:0/4168903050 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe9740c5200 0x7fe974154040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:50.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.782+0000 7fe971ffb640 1 -- 172.21.15.116:0/4168903050 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1763085656 0 0) 0x7fe9740c2b40 con 0x7fe9740c4760 2024-04-25T00:22:50.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.782+0000 7fe971ffb640 1 -- 172.21.15.116:0/4168903050 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe95c003530 con 0x7fe9740c4760 2024-04-25T00:22:50.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.782+0000 7fe971ffb640 1 -- 172.21.15.116:0/4168903050 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 217161945 0 0) 0x7fe95c003530 con 0x7fe9740c4760 2024-04-25T00:22:50.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.782+0000 7fe971ffb640 1 -- 172.21.15.116:0/4168903050 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe9740c5200 msgr2=0x7fe974154040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:50.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.782+0000 7fe971ffb640 1 --2- 172.21.15.116:0/4168903050 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe9740c5200 0x7fe974154040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:50.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.782+0000 7fe971ffb640 1 -- 172.21.15.116:0/4168903050 >> v2:172.21.15.174:3300/0 conn(0x7fe974156bb0 msgr2=0x7fe974157040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:50.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.782+0000 7fe971ffb640 1 --2- 172.21.15.116:0/4168903050 >> v2:172.21.15.174:3300/0 conn(0x7fe974156bb0 0x7fe974157040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:50.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.782+0000 7fe971ffb640 1 -- 172.21.15.116:0/4168903050 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9740c2db0 con 0x7fe9740c4760 2024-04-25T00:22:50.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.782+0000 7fe971ffb640 1 -- 172.21.15.116:0/4168903050 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7fe964001680 con 0x7fe9740c4760 2024-04-25T00:22:50.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.782+0000 7fe9737fe640 1 --2- 172.21.15.116:0/4168903050 >> v2:172.21.15.174:3300/0 conn(0x7fe974156bb0 0x7fe974157040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:22:50.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.782+0000 7fe971ffb640 1 -- 172.21.15.116:0/4168903050 >> v1:172.21.15.116:6789/0 conn(0x7fe9740c4760 legacy=0x7fe97414d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:50.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.782+0000 7fe971ffb640 1 --2- 172.21.15.116:0/4168903050 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe9740c5200 0x7fe95c0039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:50.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.782+0000 7fe971ffb640 1 --2- 172.21.15.116:0/4168903050 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe95c003f40 0x7fe95c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:50.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.782+0000 7fe971ffb640 1 --2- 172.21.15.116:0/4168903050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe95c004d50 0x7fe95c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:50.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.782+0000 7fe971ffb640 1 -- 172.21.15.116:0/4168903050 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe974072b80 con 0x7fe95c004d50 2024-04-25T00:22:50.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.782+0000 7fe971ffb640 1 -- 172.21.15.116:0/4168903050 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe974154580 con 0x7fe95c003f40 2024-04-25T00:22:50.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.782+0000 7fe971ffb640 1 -- 172.21.15.116:0/4168903050 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe95c00bad0 con 0x7fe9740c5200 2024-04-25T00:22:50.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.782+0000 7fe9737fe640 1 --2- 172.21.15.116:0/4168903050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe95c004d50 0x7fe95c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:50.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.782+0000 7fe9727fc640 1 --2- 172.21.15.116:0/4168903050 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe95c003f40 0x7fe95c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:50.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.782+0000 7fe9737fe640 1 -- 172.21.15.116:0/4168903050 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe9740c5200 msgr2=0x7fe95c0039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:50.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.782+0000 7fe972ffd640 1 --2- 172.21.15.116:0/4168903050 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe9740c5200 0x7fe95c0039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:50.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.782+0000 7fe9737fe640 1 --2- 172.21.15.116:0/4168903050 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe9740c5200 0x7fe95c0039c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:50.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.782+0000 7fe9737fe640 1 -- 172.21.15.116:0/4168903050 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe95c003f40 msgr2=0x7fe95c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:50.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.782+0000 7fe9737fe640 1 --2- 172.21.15.116:0/4168903050 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe95c003f40 0x7fe95c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:50.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.782+0000 7fe9737fe640 1 -- 172.21.15.116:0/4168903050 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe95c003530 con 0x7fe95c004d50 2024-04-25T00:22:50.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.782+0000 7fe9737fe640 1 --2- 172.21.15.116:0/4168903050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe95c004d50 0x7fe95c0051d0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7fe96c007410 tx=0x7fe96c01cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=883aefe189cc12f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:50.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.782+0000 7fe971ffb640 1 -- 172.21.15.116:0/4168903050 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe96c0b2070 con 0x7fe95c004d50 2024-04-25T00:22:50.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.782+0000 7fe971ffb640 1 -- 172.21.15.116:0/4168903050 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe96c0a5d30 con 0x7fe95c004d50 2024-04-25T00:22:50.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.782+0000 7fe971ffb640 1 -- 172.21.15.116:0/4168903050 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe96c009c80 con 0x7fe95c004d50 2024-04-25T00:22:50.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.786+0000 7fe979518640 1 -- 172.21.15.116:0/4168903050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe95c004d50 msgr2=0x7fe95c0051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:50.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.786+0000 7fe979518640 1 --2- 172.21.15.116:0/4168903050 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe95c004d50 0x7fe95c0051d0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7fe96c007410 tx=0x7fe96c01cf40 comp rx=0 tx=0).stop 2024-04-25T00:22:50.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.786+0000 7fe9737fe640 1 -- 172.21.15.116:0/4168903050 reap_dead start 2024-04-25T00:22:50.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.786+0000 7fe979518640 1 -- 172.21.15.116:0/4168903050 shutdown_connections 2024-04-25T00:22:50.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.786+0000 7fe979518640 1 -- 172.21.15.116:0/4168903050 >> 172.21.15.116:0/4168903050 conn(0x7fe97406c720 msgr2=0x7fe97406cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:50.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.786+0000 7fe979518640 1 -- 172.21.15.116:0/4168903050 shutdown_connections 2024-04-25T00:22:50.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.786+0000 7fe979518640 1 -- 172.21.15.116:0/4168903050 wait complete. 2024-04-25T00:22:50.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.786+0000 7fe979518640 1 Processor -- start 2024-04-25T00:22:50.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.786+0000 7fe979518640 1 -- start start 2024-04-25T00:22:50.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.786+0000 7fe979518640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9740c1730 0x7fe97414c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:50.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.786+0000 7fe979518640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe974147890 0x7fe9741e8570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:50.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.786+0000 7fe979518640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe97406ea80 0x7fe9741eb160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:50.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.786+0000 7fe979518640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe9740c2720 con 0x7fe9740c1730 2024-04-25T00:22:50.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.786+0000 7fe979518640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe974072810 con 0x7fe974147890 2024-04-25T00:22:50.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.786+0000 7fe979518640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe97414d940 con 0x7fe97406ea80 2024-04-25T00:22:50.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.786+0000 7fe972ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9740c1730 0x7fe97414c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:50.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.786+0000 7fe972ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9740c1730 0x7fe97414c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39732/0 (socket says 172.21.15.116:39732) 2024-04-25T00:22:50.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.786+0000 7fe972ffd640 1 -- 172.21.15.116:0/489288808 learned_addr learned my addr 172.21.15.116:0/489288808 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:22:50.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.786+0000 7fe9727fc640 1 --2- 172.21.15.116:0/489288808 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe974147890 0x7fe9741e8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:50.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.786+0000 7fe9737fe640 1 --2- 172.21.15.116:0/489288808 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe97406ea80 0x7fe9741eb160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:50.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.786+0000 7fe972ffd640 1 -- 172.21.15.116:0/489288808 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe97406ea80 msgr2=0x7fe9741eb160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:50.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.786+0000 7fe972ffd640 1 --2- 172.21.15.116:0/489288808 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe97406ea80 0x7fe9741eb160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:50.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.786+0000 7fe972ffd640 1 -- 172.21.15.116:0/489288808 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe974147890 msgr2=0x7fe9741e8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:50.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.786+0000 7fe972ffd640 1 --2- 172.21.15.116:0/489288808 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe974147890 0x7fe9741e8570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:50.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.786+0000 7fe972ffd640 1 -- 172.21.15.116:0/489288808 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe97406fd30 con 0x7fe9740c1730 2024-04-25T00:22:50.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.786+0000 7fe9737fe640 1 --2- 172.21.15.116:0/489288808 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe97406ea80 0x7fe9741eb160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:22:50.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.786+0000 7fe972ffd640 1 --2- 172.21.15.116:0/489288808 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9740c1730 0x7fe97414c840 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7fe9680108c0 tx=0x7fe968010d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:50.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.786+0000 7fe95b7fe640 1 -- 172.21.15.116:0/489288808 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe96800f040 con 0x7fe9740c1730 2024-04-25T00:22:50.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.786+0000 7fe95b7fe640 1 -- 172.21.15.116:0/489288808 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe96800eb60 con 0x7fe9740c1730 2024-04-25T00:22:50.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.786+0000 7fe979518640 1 -- 172.21.15.116:0/489288808 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe97406fa50 con 0x7fe9740c1730 2024-04-25T00:22:50.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.786+0000 7fe95b7fe640 1 -- 172.21.15.116:0/489288808 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe96801c400 con 0x7fe9740c1730 2024-04-25T00:22:50.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.786+0000 7fe979518640 1 -- 172.21.15.116:0/489288808 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe97406e280 con 0x7fe9740c1730 2024-04-25T00:22:50.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.786+0000 7fe979518640 1 -- 172.21.15.116:0/489288808 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe974068a60 con 0x7fe9740c1730 2024-04-25T00:22:50.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.790+0000 7fe95b7fe640 1 -- 172.21.15.116:0/489288808 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe96801c5a0 con 0x7fe9740c1730 2024-04-25T00:22:50.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.790+0000 7fe95b7fe640 1 --2- 172.21.15.116:0/489288808 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe9600678c0 0x7fe960069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:50.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.790+0000 7fe95b7fe640 1 -- 172.21.15.116:0/489288808 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 9676+0+0 (secure 0 0 0) 0x7fe968087230 con 0x7fe9740c1730 2024-04-25T00:22:50.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.790+0000 7fe9727fc640 1 --2- 172.21.15.116:0/489288808 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe9600678c0 0x7fe960069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:50.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.790+0000 7fe9727fc640 1 --2- 172.21.15.116:0/489288808 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe9600678c0 0x7fe960069d80 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fe974147c70 tx=0x7fe964009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:50.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.794+0000 7fe95b7fe640 1 -- 172.21.15.116:0/489288808 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fe968018030 con 0x7fe9740c1730 2024-04-25T00:22:50.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.862+0000 7f687ac9a640 1 Processor -- start 2024-04-25T00:22:50.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.862+0000 7f687ac9a640 1 -- start start 2024-04-25T00:22:50.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.866+0000 7f687ac9a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6874150bf0 0x7f687406c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:50.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.866+0000 7f687ac9a640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f6874153570 0x7f687406eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:50.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.866+0000 7f687ac9a640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6874153aa0 con 0x7f687406c760 2024-04-25T00:22:50.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.866+0000 7f687ac9a640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f687406e630 con 0x7f6874153570 2024-04-25T00:22:50.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.866+0000 7f687ac9a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6874150fd0 con 0x7f6874150bf0 2024-04-25T00:22:50.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.866+0000 7f6879237640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f687406c760 0x7f687406f000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:42896/0 (socket says 172.21.15.116:42896) 2024-04-25T00:22:50.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.866+0000 7f6879237640 1 -- 172.21.15.116:0/3568040009 learned_addr learned my addr 172.21.15.116:0/3568040009 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:22:50.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.866+0000 7f6873fff640 1 --2- 172.21.15.116:0/3568040009 >> v2:172.21.15.174:3300/0 conn(0x7f6874153570 0x7f687406eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:50.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.866+0000 7f6878a36640 1 --2- 172.21.15.116:0/3568040009 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6874150bf0 0x7f687406c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:50.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.866+0000 7f68737fe640 1 -- 172.21.15.116:0/3568040009 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 442788847 0 0) 0x7f6874153aa0 con 0x7f687406c760 2024-04-25T00:22:50.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.866+0000 7f68737fe640 1 -- 172.21.15.116:0/3568040009 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6860003530 con 0x7f687406c760 2024-04-25T00:22:50.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.866+0000 7f68737fe640 1 -- 172.21.15.116:0/3568040009 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 108456985 0 0) 0x7f6860003530 con 0x7f687406c760 2024-04-25T00:22:50.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.866+0000 7f68737fe640 1 -- 172.21.15.116:0/3568040009 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6874150bf0 msgr2=0x7f687406c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:50.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.866+0000 7f68737fe640 1 --2- 172.21.15.116:0/3568040009 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6874150bf0 0x7f687406c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:50.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.866+0000 7f68737fe640 1 -- 172.21.15.116:0/3568040009 >> v2:172.21.15.174:3300/0 conn(0x7f6874153570 msgr2=0x7f687406eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:50.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.866+0000 7f68737fe640 1 --2- 172.21.15.116:0/3568040009 >> v2:172.21.15.174:3300/0 conn(0x7f6874153570 0x7f687406eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:50.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.866+0000 7f68737fe640 1 -- 172.21.15.116:0/3568040009 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6874153d80 con 0x7f687406c760 2024-04-25T00:22:50.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.866+0000 7f68737fe640 1 -- 172.21.15.116:0/3568040009 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f686c001680 con 0x7f687406c760 2024-04-25T00:22:50.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.866+0000 7f68737fe640 1 -- 172.21.15.116:0/3568040009 >> v1:172.21.15.116:6789/0 conn(0x7f687406c760 legacy=0x7f687406f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:50.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.866+0000 7f68737fe640 1 --2- 172.21.15.116:0/3568040009 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f68600039c0 0x7f6860003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:50.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.866+0000 7f68737fe640 1 --2- 172.21.15.116:0/3568040009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6860004810 0x7f6860004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:50.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.866+0000 7f68737fe640 1 --2- 172.21.15.116:0/3568040009 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6874150bf0 0x7f68600051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:50.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.866+0000 7f68737fe640 1 -- 172.21.15.116:0/3568040009 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f687406e630 con 0x7f6860004810 2024-04-25T00:22:50.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.866+0000 7f68737fe640 1 -- 172.21.15.116:0/3568040009 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6874150fd0 con 0x7f68600039c0 2024-04-25T00:22:50.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.866+0000 7f68737fe640 1 -- 172.21.15.116:0/3568040009 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f686000b9d0 con 0x7f6874150bf0 2024-04-25T00:22:50.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.866+0000 7f6873fff640 1 --2- 172.21.15.116:0/3568040009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6860004810 0x7f6860004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:50.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.866+0000 7f6873fff640 1 -- 172.21.15.116:0/3568040009 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6874150bf0 msgr2=0x7f68600051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:50.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.866+0000 7f6873fff640 1 --2- 172.21.15.116:0/3568040009 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6874150bf0 0x7f68600051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:50.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.866+0000 7f6873fff640 1 -- 172.21.15.116:0/3568040009 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f68600039c0 msgr2=0x7f6860003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:50.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.866+0000 7f6878a36640 1 --2- 172.21.15.116:0/3568040009 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f68600039c0 0x7f6860003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:50.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.866+0000 7f6873fff640 1 --2- 172.21.15.116:0/3568040009 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f68600039c0 0x7f6860003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:50.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.866+0000 7f6873fff640 1 -- 172.21.15.116:0/3568040009 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6860003530 con 0x7f6860004810 2024-04-25T00:22:50.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.866+0000 7f6873fff640 1 --2- 172.21.15.116:0/3568040009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6860004810 0x7f6860004c90 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f685c009ae0 tx=0x7f685c01ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c7fe5996f77b2d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:50.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.870+0000 7f68737fe640 1 -- 172.21.15.116:0/3568040009 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f685c0b2070 con 0x7f6860004810 2024-04-25T00:22:50.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.870+0000 7f68737fe640 1 -- 172.21.15.116:0/3568040009 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f685c0a5530 con 0x7f6860004810 2024-04-25T00:22:50.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.870+0000 7f68737fe640 1 -- 172.21.15.116:0/3568040009 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f685c0096b0 con 0x7f6860004810 2024-04-25T00:22:50.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.870+0000 7f687ac9a640 1 -- 172.21.15.116:0/3568040009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6860004810 msgr2=0x7f6860004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:50.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.870+0000 7f687ac9a640 1 --2- 172.21.15.116:0/3568040009 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6860004810 0x7f6860004c90 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f685c009ae0 tx=0x7f685c01ca40 comp rx=0 tx=0).stop 2024-04-25T00:22:50.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.870+0000 7f6879237640 1 -- 172.21.15.116:0/3568040009 reap_dead start 2024-04-25T00:22:50.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.870+0000 7f687ac9a640 1 -- 172.21.15.116:0/3568040009 shutdown_connections 2024-04-25T00:22:50.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.870+0000 7f687ac9a640 1 -- 172.21.15.116:0/3568040009 >> 172.21.15.116:0/3568040009 conn(0x7f68740c4270 msgr2=0x7f68740c4680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:50.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.870+0000 7f687ac9a640 1 -- 172.21.15.116:0/3568040009 shutdown_connections 2024-04-25T00:22:50.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.870+0000 7f687ac9a640 1 -- 172.21.15.116:0/3568040009 wait complete. 2024-04-25T00:22:50.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.870+0000 7f687ac9a640 1 Processor -- start 2024-04-25T00:22:50.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.870+0000 7f687ac9a640 1 -- start start 2024-04-25T00:22:50.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.870+0000 7f687ac9a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6860004810 0x7f68740c0e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:50.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.870+0000 7f687ac9a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f687413f810 0x7f687413fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:50.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.870+0000 7f687ac9a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68740ac630 0x7f68740acab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:50.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.870+0000 7f687ac9a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f68740a72b0 con 0x7f68740ac630 2024-04-25T00:22:50.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.870+0000 7f687ac9a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f68740a7130 con 0x7f6860004810 2024-04-25T00:22:50.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.870+0000 7f687ac9a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f687415e700 con 0x7f687413f810 2024-04-25T00:22:50.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.870+0000 7f6879237640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68740ac630 0x7f68740acab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:50.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.870+0000 7f6879237640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68740ac630 0x7f68740acab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39760/0 (socket says 172.21.15.116:39760) 2024-04-25T00:22:50.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.870+0000 7f6879237640 1 -- 172.21.15.116:0/3954914861 learned_addr learned my addr 172.21.15.116:0/3954914861 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:22:50.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.870+0000 7f6878a36640 1 --2- 172.21.15.116:0/3954914861 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6860004810 0x7f68740c0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:50.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.870+0000 7f6879237640 1 -- 172.21.15.116:0/3954914861 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f687413f810 msgr2=0x7f687413fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:50.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.870+0000 7f6879237640 1 --2- 172.21.15.116:0/3954914861 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f687413f810 0x7f687413fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:50.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.870+0000 7f6879237640 1 -- 172.21.15.116:0/3954914861 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6860004810 msgr2=0x7f68740c0e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:50.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.870+0000 7f6879237640 1 --2- 172.21.15.116:0/3954914861 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6860004810 0x7f68740c0e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:50.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.870+0000 7f6879237640 1 -- 172.21.15.116:0/3954914861 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68740ac150 con 0x7f68740ac630 2024-04-25T00:22:50.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.870+0000 7f6879237640 1 --2- 172.21.15.116:0/3954914861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68740ac630 0x7f68740acab0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f6874067fb0 tx=0x7f686c00cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:50.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.870+0000 7f68717fa640 1 -- 172.21.15.116:0/3954914861 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f686c007230 con 0x7f68740ac630 2024-04-25T00:22:50.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.870+0000 7f68717fa640 1 -- 172.21.15.116:0/3954914861 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f686c0073d0 con 0x7f68740ac630 2024-04-25T00:22:50.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.870+0000 7f68717fa640 1 -- 172.21.15.116:0/3954914861 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f686c00ce50 con 0x7f68740ac630 2024-04-25T00:22:50.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.870+0000 7f687ac9a640 1 -- 172.21.15.116:0/3954914861 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68740aa680 con 0x7f68740ac630 2024-04-25T00:22:50.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.870+0000 7f687ac9a640 1 -- 172.21.15.116:0/3954914861 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68740ab940 con 0x7f68740ac630 2024-04-25T00:22:50.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.874+0000 7f687ac9a640 1 -- 172.21.15.116:0/3954914861 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68741f2910 con 0x7f68740ac630 2024-04-25T00:22:50.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.874+0000 7f68717fa640 1 -- 172.21.15.116:0/3954914861 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f686c0047f0 con 0x7f68740ac630 2024-04-25T00:22:50.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.874+0000 7f68717fa640 1 --2- 172.21.15.116:0/3954914861 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f68540678c0 0x7f6854069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:50.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.874+0000 7f68717fa640 1 -- 172.21.15.116:0/3954914861 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 9676+0+0 (secure 0 0 0) 0x7f686c085eb0 con 0x7f68740ac630 2024-04-25T00:22:50.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.874+0000 7f6878a36640 1 --2- 172.21.15.116:0/3954914861 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f68540678c0 0x7f6854069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:50.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.874+0000 7f6878a36640 1 --2- 172.21.15.116:0/3954914861 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f68540678c0 0x7f6854069d80 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f6864009fd0 tx=0x7f686400d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:50.879 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.878+0000 7f68717fa640 1 -- 172.21.15.116:0/3954914861 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f686c01a030 con 0x7f68740ac630 2024-04-25T00:22:50.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.994+0000 7fe979518640 1 -- 172.21.15.116:0/489288808 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe9740a9760 con 0x7fe9740c1730 2024-04-25T00:22:50.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.994+0000 7fe95b7fe640 1 -- 172.21.15.116:0/489288808 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 185 v185) ==== 95+0+5669 (secure 0 0 0) 0x7fe968056e30 con 0x7fe9740c1730 2024-04-25T00:22:50.995 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:22:50.995 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":185,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:22:49.920943+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8214},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":183,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.116:6833/2998042514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2998042514},{"type":"v1","addr":"172.21.15.116:6833","nonce":2998042514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:22:38.936345+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6230":{"gid":6230,"name":"d","rank":0,"incarnation":178,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/492366509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":492366509},{"type":"v1","addr":"172.21.15.174:6837","nonce":492366509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6230,"qdb_cluster":[6230]},"id":2}]} 2024-04-25T00:22:50.995 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 185 2024-04-25T00:22:50.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.994+0000 7fe979518640 1 -- 172.21.15.116:0/489288808 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe9600678c0 msgr2=0x7fe960069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:50.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.994+0000 7fe979518640 1 --2- 172.21.15.116:0/489288808 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe9600678c0 0x7fe960069d80 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fe974147c70 tx=0x7fe964009040 comp rx=0 tx=0).stop 2024-04-25T00:22:50.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.994+0000 7fe979518640 1 -- 172.21.15.116:0/489288808 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9740c1730 msgr2=0x7fe97414c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:50.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.994+0000 7fe979518640 1 --2- 172.21.15.116:0/489288808 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9740c1730 0x7fe97414c840 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7fe9680108c0 tx=0x7fe968010d80 comp rx=0 tx=0).stop 2024-04-25T00:22:50.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.994+0000 7fe979518640 1 -- 172.21.15.116:0/489288808 shutdown_connections 2024-04-25T00:22:50.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.994+0000 7fe979518640 1 --2- 172.21.15.116:0/489288808 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe9600678c0 0x7fe960069d80 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:50.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.994+0000 7fe979518640 1 --2- 172.21.15.116:0/489288808 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe97406ea80 0x7fe9741eb160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:50.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.994+0000 7fe979518640 1 --2- 172.21.15.116:0/489288808 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe974147890 0x7fe9741e8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:50.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.994+0000 7fe979518640 1 --2- 172.21.15.116:0/489288808 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe9740c1730 0x7fe97414c840 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:50.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.994+0000 7fe979518640 1 -- 172.21.15.116:0/489288808 >> 172.21.15.116:0/489288808 conn(0x7fe97406c720 msgr2=0x7fe974145ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:50.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.994+0000 7fe979518640 1 -- 172.21.15.116:0/489288808 shutdown_connections 2024-04-25T00:22:50.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:50.994+0000 7fe979518640 1 -- 172.21.15.116:0/489288808 wait complete. 2024-04-25T00:22:51.014 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2024-04-25T00:22:51.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:51.078+0000 7f687ac9a640 1 -- 172.21.15.116:0/3954914861 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6874068a60 con 0x7f68740ac630 2024-04-25T00:22:51.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:51.078+0000 7f68717fa640 1 -- 172.21.15.116:0/3954914861 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 185 v185) ==== 95+0+5669 (secure 0 0 0) 0x7f686c055a20 con 0x7f68740ac630 2024-04-25T00:22:51.079 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:22:51.079 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":185,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:22:49.920943+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8214},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":183,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.116:6833/2998042514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2998042514},{"type":"v1","addr":"172.21.15.116:6833","nonce":2998042514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:22:38.936345+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6230":{"gid":6230,"name":"d","rank":0,"incarnation":178,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/492366509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":492366509},{"type":"v1","addr":"172.21.15.174:6837","nonce":492366509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6230,"qdb_cluster":[6230]},"id":2}]} 2024-04-25T00:22:51.080 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 185 2024-04-25T00:22:51.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:51.078+0000 7f687ac9a640 1 -- 172.21.15.116:0/3954914861 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f68540678c0 msgr2=0x7f6854069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:51.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:51.078+0000 7f687ac9a640 1 --2- 172.21.15.116:0/3954914861 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f68540678c0 0x7f6854069d80 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f6864009fd0 tx=0x7f686400d040 comp rx=0 tx=0).stop 2024-04-25T00:22:51.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:51.078+0000 7f687ac9a640 1 -- 172.21.15.116:0/3954914861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68740ac630 msgr2=0x7f68740acab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:51.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:51.078+0000 7f687ac9a640 1 --2- 172.21.15.116:0/3954914861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68740ac630 0x7f68740acab0 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f6874067fb0 tx=0x7f686c00cb10 comp rx=0 tx=0).stop 2024-04-25T00:22:51.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:51.078+0000 7f687ac9a640 1 -- 172.21.15.116:0/3954914861 shutdown_connections 2024-04-25T00:22:51.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:51.078+0000 7f687ac9a640 1 --2- 172.21.15.116:0/3954914861 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f68540678c0 0x7f6854069d80 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:51.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:51.078+0000 7f687ac9a640 1 --2- 172.21.15.116:0/3954914861 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68740ac630 0x7f68740acab0 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:51.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:51.078+0000 7f687ac9a640 1 --2- 172.21.15.116:0/3954914861 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f687413f810 0x7f687413fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:51.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:51.078+0000 7f687ac9a640 1 --2- 172.21.15.116:0/3954914861 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6860004810 0x7f68740c0e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:51.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:51.078+0000 7f687ac9a640 1 -- 172.21.15.116:0/3954914861 >> 172.21.15.116:0/3954914861 conn(0x7f68740c4270 msgr2=0x7f687406eac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:51.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:51.082+0000 7f687ac9a640 1 -- 172.21.15.116:0/3954914861 shutdown_connections 2024-04-25T00:22:51.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:51.082+0000 7f687ac9a640 1 -- 172.21.15.116:0/3954914861 wait complete. 2024-04-25T00:22:51.098 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 185, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:04.678737+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 185, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_8214": { "addr": "172.21.15.116:6833/2998042514", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6832", "nonce": 2998042514, "type": "v2" }, { "addr": "172.21.15.116:6833", "nonce": 2998042514, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8214, "incarnation": 183, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:rejoin", "state_seq": 3 } }, "last_failure": 0, "last_failure_osd_epoch": 63, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T00:22:49.920943+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8214 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:18.825887+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 181, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6230": { "addr": "172.21.15.174:6837/492366509", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6836", "nonce": 492366509, "type": "v2" }, { "addr": "172.21.15.174:6837", "nonce": 492366509, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6230, "incarnation": 178, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 7 } }, "last_failure": 0, "last_failure_osd_epoch": 62, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-04-25T00:22:38.936345+0000", "qdb_cluster": [ 6230 ], "qdb_leader": 6230, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6230 } } } ], "standbys": [ { "addr": "172.21.15.174:6835/4112145782", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6834", "nonce": 4112145782, "type": "v2" }, { "addr": "172.21.15.174:6835", "nonce": 4112145782, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 112, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5582, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-25T00:22:53.014 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:22:53.099 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:22:53.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.194+0000 7f6c2f59e640 1 Processor -- start 2024-04-25T00:22:53.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.194+0000 7f6c2f59e640 1 -- start start 2024-04-25T00:22:53.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.194+0000 7f6c2f59e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c2803ea10 0x7f6c280312c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:53.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.194+0000 7f6c2f59e640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f6c280de9e0 0x7f6c280dee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:53.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.194+0000 7f6c2f59e640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c28030fe0 con 0x7f6c2800b0e0 2024-04-25T00:22:53.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.194+0000 7f6c2f59e640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f6c28007220 con 0x7f6c280de9e0 2024-04-25T00:22:53.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.194+0000 7f6c2f59e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6c28006bd0 con 0x7f6c2803ea10 2024-04-25T00:22:53.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.194+0000 7f6c2ed9d640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f6c280de9e0 0x7f6c280dee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:53.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.194+0000 7f6c2dd9b640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f6c2800b0e0 0x7f6c2804ac50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:57930/0 (socket says 172.21.15.116:57930) 2024-04-25T00:22:53.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.194+0000 7f6c2ed9d640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f6c280de9e0 0x7f6c280dee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:34596/0 (socket says 172.21.15.116:34596) 2024-04-25T00:22:53.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.194+0000 7f6c2dd9b640 1 -- 172.21.15.116:0/866832950 learned_addr learned my addr 172.21.15.116:0/866832950 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:22:53.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.194+0000 7f6c2e59c640 1 --2- 172.21.15.116:0/866832950 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c2803ea10 0x7f6c280312c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:53.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.198+0000 7f6c2d59a640 1 -- 172.21.15.116:0/866832950 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1864697781 0 0) 0x7f6c28030fe0 con 0x7f6c2800b0e0 2024-04-25T00:22:53.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.198+0000 7f6c2d59a640 1 -- 172.21.15.116:0/866832950 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c18003530 con 0x7f6c2800b0e0 2024-04-25T00:22:53.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.198+0000 7f6c2e59c640 1 -- 172.21.15.116:0/866832950 >> v2:172.21.15.174:3300/0 conn(0x7f6c280de9e0 msgr2=0x7f6c280dee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:53.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.198+0000 7f6c2e59c640 1 --2- 172.21.15.116:0/866832950 >> v2:172.21.15.174:3300/0 conn(0x7f6c280de9e0 0x7f6c280dee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:53.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.198+0000 7f6c2e59c640 1 -- 172.21.15.116:0/866832950 >> v1:172.21.15.174:6789/0 conn(0x7f6c2800b0e0 legacy=0x7f6c2804ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:53.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.198+0000 7f6c2e59c640 1 -- 172.21.15.116:0/866832950 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c28006eb0 con 0x7f6c2803ea10 2024-04-25T00:22:53.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.198+0000 7f6c2ed9d640 1 --2- 172.21.15.116:0/866832950 >> v2:172.21.15.174:3300/0 conn(0x7f6c280de9e0 0x7f6c280dee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:22:53.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.198+0000 7f6c2e59c640 1 --2- 172.21.15.116:0/866832950 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c2803ea10 0x7f6c280312c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6c240097c0 tx=0x7f6c2401c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=307e3a269c3f37ab server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:53.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.198+0000 7f6c2d59a640 1 -- 172.21.15.116:0/866832950 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c240b0070 con 0x7f6c2803ea10 2024-04-25T00:22:53.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.202+0000 7f6c2d59a640 1 -- 172.21.15.116:0/866832950 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c240ab600 con 0x7f6c2803ea10 2024-04-25T00:22:53.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.202+0000 7f6c2d59a640 1 -- 172.21.15.116:0/866832950 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c240b0070 con 0x7f6c2803ea10 2024-04-25T00:22:53.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.202+0000 7f6c2f59e640 1 -- 172.21.15.116:0/866832950 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c2803ea10 msgr2=0x7f6c280312c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:53.202 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.202+0000 7f6c2f59e640 1 --2- 172.21.15.116:0/866832950 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c2803ea10 0x7f6c280312c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6c240097c0 tx=0x7f6c2401c410 comp rx=0 tx=0).stop 2024-04-25T00:22:53.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.202+0000 7f6c2f59e640 1 -- 172.21.15.116:0/866832950 shutdown_connections 2024-04-25T00:22:53.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.202+0000 7f6c2f59e640 1 --2- 172.21.15.116:0/866832950 >> v2:172.21.15.174:3300/0 conn(0x7f6c280de9e0 0x7f6c280dee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:53.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.202+0000 7f6c2f59e640 1 --2- 172.21.15.116:0/866832950 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c2803ea10 0x7f6c280312c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:53.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.202+0000 7f6c2f59e640 1 -- 172.21.15.116:0/866832950 >> 172.21.15.116:0/866832950 conn(0x7f6c28005ea0 msgr2=0x7f6c280062b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:53.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.202+0000 7f6c2f59e640 1 -- 172.21.15.116:0/866832950 shutdown_connections 2024-04-25T00:22:53.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.202+0000 7f6c2f59e640 1 -- 172.21.15.116:0/866832950 wait complete. 2024-04-25T00:22:53.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.202+0000 7f6c2f59e640 1 Processor -- start 2024-04-25T00:22:53.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.210+0000 7f6c2f59e640 1 -- start start 2024-04-25T00:22:53.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.210+0000 7f6c2f59e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c2800b0e0 0x7f6c28040fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:53.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.210+0000 7f6c2f59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c2803ea10 0x7f6c280e1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:53.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.210+0000 7f6c2f59e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c280de9e0 0x7f6c28044a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:53.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.210+0000 7f6c2f59e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6c2804b4e0 con 0x7f6c2803ea10 2024-04-25T00:22:53.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.210+0000 7f6c2f59e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6c2804b360 con 0x7f6c280de9e0 2024-04-25T00:22:53.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.210+0000 7f6c2f59e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6c2803edf0 con 0x7f6c2800b0e0 2024-04-25T00:22:53.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.210+0000 7f6c2ed9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c280de9e0 0x7f6c28044a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:53.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.210+0000 7f6c2ed9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c280de9e0 0x7f6c28044a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:34604/0 (socket says 172.21.15.116:34604) 2024-04-25T00:22:53.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.210+0000 7f6c2ed9d640 1 -- 172.21.15.116:0/4166954072 learned_addr learned my addr 172.21.15.116:0/4166954072 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:22:53.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.210+0000 7f6c2e59c640 1 --2- 172.21.15.116:0/4166954072 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c2800b0e0 0x7f6c28040fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:53.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.210+0000 7f6c2e59c640 1 -- 172.21.15.116:0/4166954072 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c280de9e0 msgr2=0x7f6c28044a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:53.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.210+0000 7f6c2e59c640 1 --2- 172.21.15.116:0/4166954072 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c280de9e0 0x7f6c28044a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:53.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.210+0000 7f6c2e59c640 1 -- 172.21.15.116:0/4166954072 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c2803ea10 msgr2=0x7f6c280e1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:53.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.210+0000 7f6c2e59c640 1 --2- 172.21.15.116:0/4166954072 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c2803ea10 0x7f6c280e1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:53.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.210+0000 7f6c2e59c640 1 -- 172.21.15.116:0/4166954072 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c280e1aa0 con 0x7f6c2800b0e0 2024-04-25T00:22:53.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.210+0000 7f6c2e59c640 1 --2- 172.21.15.116:0/4166954072 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c2800b0e0 0x7f6c28040fa0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6c280073a0 tx=0x7f6c240191b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:53.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.214+0000 7f6c2ed9d640 1 --2- 172.21.15.116:0/4166954072 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c280de9e0 0x7f6c28044a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:22:53.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.214+0000 7f6c1effd640 1 -- 172.21.15.116:0/4166954072 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c240b8020 con 0x7f6c2800b0e0 2024-04-25T00:22:53.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.214+0000 7f6c1effd640 1 -- 172.21.15.116:0/4166954072 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c240b4640 con 0x7f6c2800b0e0 2024-04-25T00:22:53.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.214+0000 7f6c2f59e640 1 -- 172.21.15.116:0/4166954072 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c280d3150 con 0x7f6c2800b0e0 2024-04-25T00:22:53.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.214+0000 7f6c2f59e640 1 -- 172.21.15.116:0/4166954072 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c2803f7e0 con 0x7f6c2800b0e0 2024-04-25T00:22:53.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.218+0000 7f6c1effd640 1 -- 172.21.15.116:0/4166954072 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c240b0070 con 0x7f6c2800b0e0 2024-04-25T00:22:53.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.218+0000 7f6c1effd640 1 -- 172.21.15.116:0/4166954072 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6c240bb020 con 0x7f6c2800b0e0 2024-04-25T00:22:53.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.218+0000 7f6c1effd640 1 --2- 172.21.15.116:0/4166954072 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6c100678f0 0x7f6c10069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:53.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.218+0000 7f6c1effd640 1 -- 172.21.15.116:0/4166954072 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(63..63 src has 1..63) ==== 9676+0+0 (secure 0 0 0) 0x7f6c24120950 con 0x7f6c2800b0e0 2024-04-25T00:22:53.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.218+0000 7f6c2dd9b640 1 --2- 172.21.15.116:0/4166954072 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6c100678f0 0x7f6c10069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:53.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.218+0000 7f6c2dd9b640 1 --2- 172.21.15.116:0/4166954072 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6c100678f0 0x7f6c10069db0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f6c20000f70 tx=0x7f6c20007900 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:53.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.222+0000 7f6c2f59e640 1 -- 172.21.15.116:0/4166954072 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c140052a0 con 0x7f6c2800b0e0 2024-04-25T00:22:53.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.234+0000 7f6c1effd640 1 -- 172.21.15.116:0/4166954072 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f6c240c2030 con 0x7f6c2800b0e0 2024-04-25T00:22:53.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.474+0000 7f4e81012640 1 Processor -- start 2024-04-25T00:22:53.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.474+0000 7f4e81012640 1 -- start start 2024-04-25T00:22:53.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.474+0000 7f4e81012640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f4e7c1572d0 0x7f4e7c0a84b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:53.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.474+0000 7f4e81012640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4e7c06e080 0x7f4e7c070f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:53.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.474+0000 7f4e81012640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e7c0c36f0 con 0x7f4e7c0714d0 2024-04-25T00:22:53.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.474+0000 7f4e81012640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f4e7c15e140 con 0x7f4e7c06e080 2024-04-25T00:22:53.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.474+0000 7f4e81012640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f4e7c0a89f0 con 0x7f4e7c1572d0 2024-04-25T00:22:53.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.474+0000 7f4e7ad9d640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f4e7c1572d0 0x7f4e7c0a84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:53.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.474+0000 7f4e7ad9d640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f4e7c1572d0 0x7f4e7c0a84b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:34620/0 (socket says 172.21.15.116:34620) 2024-04-25T00:22:53.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.474+0000 7f4e7ad9d640 1 -- 172.21.15.116:0/746076445 learned_addr learned my addr 172.21.15.116:0/746076445 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:22:53.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.474+0000 7f4e7a59c640 1 --2- 172.21.15.116:0/746076445 >> v2:172.21.15.116:3300/0 conn(0x7f4e7c06e080 0x7f4e7c070f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:53.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.478+0000 7f4e7ad9d640 1 -- 172.21.15.116:0/746076445 >> v2:172.21.15.116:3300/0 conn(0x7f4e7c06e080 msgr2=0x7f4e7c070f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:22:53.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.478+0000 7f4e79d9b640 1 -- 172.21.15.116:0/746076445 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 884347322 0 0) 0x7f4e7c0c36f0 con 0x7f4e7c0714d0 2024-04-25T00:22:53.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.478+0000 7f4e7ad9d640 1 --2- 172.21.15.116:0/746076445 >> v2:172.21.15.116:3300/0 conn(0x7f4e7c06e080 0x7f4e7c070f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:53.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.478+0000 7f4e7ad9d640 1 -- 172.21.15.116:0/746076445 >> v1:172.21.15.174:6789/0 conn(0x7f4e7c0714d0 legacy=0x7f4e7c0c5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:53.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.478+0000 7f4e7ad9d640 1 -- 172.21.15.116:0/746076445 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e7c15d380 con 0x7f4e7c1572d0 2024-04-25T00:22:53.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.478+0000 7f4e7ad9d640 1 --2- 172.21.15.116:0/746076445 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4e7c1572d0 0x7f4e7c0a84b0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c0c6ba0 tx=0x7f4e6c01c540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd0b2941fb5a15e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:53.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.478+0000 7f4e7a59c640 1 --2- 172.21.15.116:0/746076445 >> v2:172.21.15.116:3300/0 conn(0x7f4e7c06e080 0x7f4e7c070f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:22:53.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.478+0000 7f4e79d9b640 1 -- 172.21.15.116:0/746076445 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e6c0b0070 con 0x7f4e7c1572d0 2024-04-25T00:22:53.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.478+0000 7f4e79d9b640 1 -- 172.21.15.116:0/746076445 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e6c0ab600 con 0x7f4e7c1572d0 2024-04-25T00:22:53.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.478+0000 7f4e79d9b640 1 -- 172.21.15.116:0/746076445 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e6c0ab9d0 con 0x7f4e7c1572d0 2024-04-25T00:22:53.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.482+0000 7f4e81012640 1 -- 172.21.15.116:0/746076445 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4e7c1572d0 msgr2=0x7f4e7c0a84b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:53.481 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.482+0000 7f4e81012640 1 --2- 172.21.15.116:0/746076445 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4e7c1572d0 0x7f4e7c0a84b0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c0c6ba0 tx=0x7f4e6c01c540 comp rx=0 tx=0).stop 2024-04-25T00:22:53.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.482+0000 7f4e81012640 1 -- 172.21.15.116:0/746076445 shutdown_connections 2024-04-25T00:22:53.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.482+0000 7f4e81012640 1 --2- 172.21.15.116:0/746076445 >> v2:172.21.15.116:3300/0 conn(0x7f4e7c06e080 0x7f4e7c070f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:53.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.482+0000 7f4e81012640 1 --2- 172.21.15.116:0/746076445 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4e7c1572d0 0x7f4e7c0a84b0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:53.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.482+0000 7f4e81012640 1 -- 172.21.15.116:0/746076445 >> 172.21.15.116:0/746076445 conn(0x7f4e7c074650 msgr2=0x7f4e7c074a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:53.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.482+0000 7f4e81012640 1 -- 172.21.15.116:0/746076445 shutdown_connections 2024-04-25T00:22:53.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.482+0000 7f4e81012640 1 -- 172.21.15.116:0/746076445 wait complete. 2024-04-25T00:22:53.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.486+0000 7f4e81012640 1 Processor -- start 2024-04-25T00:22:53.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.486+0000 7f4e81012640 1 -- start start 2024-04-25T00:22:53.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.486+0000 7f4e81012640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e7c06e080 0x7f4e7c0adbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:53.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.486+0000 7f4e81012640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4e7c0714d0 0x7f4e7c162170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:53.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.486+0000 7f4e81012640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4e7c1572d0 0x7f4e7c1626b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:53.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.486+0000 7f4e81012640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4e7c166d50 con 0x7f4e7c06e080 2024-04-25T00:22:53.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.486+0000 7f4e81012640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4e7c0c5b50 con 0x7f4e7c0714d0 2024-04-25T00:22:53.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.486+0000 7f4e81012640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f4e7c166ed0 con 0x7f4e7c1572d0 2024-04-25T00:22:53.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.486+0000 7f4e7b59e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4e7c1572d0 0x7f4e7c1626b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:53.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.486+0000 7f4e7b59e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4e7c1572d0 0x7f4e7c1626b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:56722/0 (socket says 172.21.15.116:56722) 2024-04-25T00:22:53.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.486+0000 7f4e7b59e640 1 -- 172.21.15.116:0/2780328703 learned_addr learned my addr 172.21.15.116:0/2780328703 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:22:53.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.486+0000 7f4e7b59e640 1 -- 172.21.15.116:0/2780328703 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4e7c0714d0 msgr2=0x7f4e7c162170 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T00:22:53.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.486+0000 7f4e7b59e640 1 --2- 172.21.15.116:0/2780328703 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4e7c0714d0 0x7f4e7c162170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:53.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.486+0000 7f4e7b59e640 1 -- 172.21.15.116:0/2780328703 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e7c06e080 msgr2=0x7f4e7c0adbf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T00:22:53.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.486+0000 7f4e7b59e640 1 --2- 172.21.15.116:0/2780328703 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e7c06e080 0x7f4e7c0adbf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:53.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.486+0000 7f4e7b59e640 1 -- 172.21.15.116:0/2780328703 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e7c161900 con 0x7f4e7c1572d0 2024-04-25T00:22:53.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.486+0000 7f4e7b59e640 1 --2- 172.21.15.116:0/2780328703 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4e7c1572d0 0x7f4e7c1626b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4e7400c7b0 tx=0x7f4e7400cc70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:53.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.486+0000 7f4e637fe640 1 -- 172.21.15.116:0/2780328703 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e74003bb0 con 0x7f4e7c1572d0 2024-04-25T00:22:53.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.486+0000 7f4e637fe640 1 -- 172.21.15.116:0/2780328703 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e74003d50 con 0x7f4e7c1572d0 2024-04-25T00:22:53.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.486+0000 7f4e637fe640 1 -- 172.21.15.116:0/2780328703 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e740052b0 con 0x7f4e7c1572d0 2024-04-25T00:22:53.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.486+0000 7f4e81012640 1 -- 172.21.15.116:0/2780328703 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e7c162fc0 con 0x7f4e7c1572d0 2024-04-25T00:22:53.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.486+0000 7f4e81012640 1 -- 172.21.15.116:0/2780328703 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e7c1f2520 con 0x7f4e7c1572d0 2024-04-25T00:22:53.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.490+0000 7f4e81012640 1 -- 172.21.15.116:0/2780328703 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e7c068a60 con 0x7f4e7c1572d0 2024-04-25T00:22:53.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.490+0000 7f4e637fe640 1 -- 172.21.15.116:0/2780328703 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4e74005450 con 0x7f4e7c1572d0 2024-04-25T00:22:53.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.490+0000 7f4e637fe640 1 --2- 172.21.15.116:0/2780328703 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4e68067990 0x7f4e68069e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:22:53.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.490+0000 7f4e637fe640 1 -- 172.21.15.116:0/2780328703 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(63..63 src has 1..63) ==== 9676+0+0 (secure 0 0 0) 0x7f4e740979c0 con 0x7f4e7c1572d0 2024-04-25T00:22:53.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.490+0000 7f4e7ad9d640 1 --2- 172.21.15.116:0/2780328703 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4e68067990 0x7f4e68069e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:22:53.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.490+0000 7f4e7ad9d640 1 --2- 172.21.15.116:0/2780328703 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4e68067990 0x7f4e68069e50 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c0c9c60 tx=0x7f4e6c0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:22:53.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.498+0000 7f4e637fe640 1 -- 172.21.15.116:0/2780328703 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f4e740675e0 con 0x7f4e7c1572d0 2024-04-25T00:22:53.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.826+0000 7f6c2f59e640 1 -- 172.21.15.116:0/4166954072 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6c14005590 con 0x7f6c2800b0e0 2024-04-25T00:22:53.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.826+0000 7f6c1effd640 1 -- 172.21.15.116:0/4166954072 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 186 v186) ==== 95+0+5676 (secure 0 0 0) 0x7f6c24016090 con 0x7f6c2800b0e0 2024-04-25T00:22:53.829 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:22:53.829 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":186,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":186,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:22:51.924561+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8214},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":183,"state":"up:active","state_seq":4,"addr":"172.21.15.116:6833/2998042514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2998042514},{"type":"v1","addr":"172.21.15.116:6833","nonce":2998042514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":1},{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:22:38.936345+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6230":{"gid":6230,"name":"d","rank":0,"incarnation":178,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/492366509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":492366509},{"type":"v1","addr":"172.21.15.174:6837","nonce":492366509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6230,"qdb_cluster":[6230]},"id":2}]} 2024-04-25T00:22:53.830 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 186 2024-04-25T00:22:53.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.834+0000 7f6c2f59e640 1 -- 172.21.15.116:0/4166954072 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6c100678f0 msgr2=0x7f6c10069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:53.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.834+0000 7f6c2f59e640 1 --2- 172.21.15.116:0/4166954072 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6c100678f0 0x7f6c10069db0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f6c20000f70 tx=0x7f6c20007900 comp rx=0 tx=0).stop 2024-04-25T00:22:53.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.834+0000 7f6c2f59e640 1 -- 172.21.15.116:0/4166954072 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c2800b0e0 msgr2=0x7f6c28040fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:53.836 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.834+0000 7f6c2f59e640 1 --2- 172.21.15.116:0/4166954072 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c2800b0e0 0x7f6c28040fa0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6c280073a0 tx=0x7f6c240191b0 comp rx=0 tx=0).stop 2024-04-25T00:22:53.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.842+0000 7f6c2f59e640 1 -- 172.21.15.116:0/4166954072 shutdown_connections 2024-04-25T00:22:53.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.842+0000 7f6c2f59e640 1 --2- 172.21.15.116:0/4166954072 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6c100678f0 0x7f6c10069db0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:53.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.842+0000 7f6c2f59e640 1 --2- 172.21.15.116:0/4166954072 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6c280de9e0 0x7f6c28044a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:53.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.842+0000 7f6c2f59e640 1 --2- 172.21.15.116:0/4166954072 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6c2803ea10 0x7f6c280e1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:53.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.842+0000 7f6c2f59e640 1 --2- 172.21.15.116:0/4166954072 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6c2800b0e0 0x7f6c28040fa0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:53.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.842+0000 7f6c2f59e640 1 -- 172.21.15.116:0/4166954072 >> 172.21.15.116:0/4166954072 conn(0x7f6c28005ea0 msgr2=0x7f6c28006280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:53.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.842+0000 7f6c2f59e640 1 -- 172.21.15.116:0/4166954072 shutdown_connections 2024-04-25T00:22:53.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:53.842+0000 7f6c2f59e640 1 -- 172.21.15.116:0/4166954072 wait complete. 2024-04-25T00:22:53.888 INFO:tasks.mds_thrash.fs.[b]:mds.a reported in up:active state 2024-04-25T00:22:53.888 INFO:tasks.mds_thrash.fs.[b]:waiting for 67 secs before thrashing 2024-04-25T00:22:53.909 INFO:tasks.ceph.mds.a.smithi116.stderr:2024-04-25T00:22:53.906+0000 7f5166e2f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T00:22:54.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:54.030+0000 7f4e81012640 1 -- 172.21.15.116:0/2780328703 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4e7c070d70 con 0x7f4e7c1572d0 2024-04-25T00:22:54.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:54.034+0000 7f4e637fe640 1 -- 172.21.15.116:0/2780328703 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 186 v186) ==== 95+0+5676 (secure 0 0 0) 0x7f4e74066ee0 con 0x7f4e7c1572d0 2024-04-25T00:22:54.036 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:22:54.036 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":186,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":186,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:22:51.924561+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8214},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":183,"state":"up:active","state_seq":4,"addr":"172.21.15.116:6833/2998042514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2998042514},{"type":"v1","addr":"172.21.15.116:6833","nonce":2998042514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":1},{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:22:38.936345+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6230":{"gid":6230,"name":"d","rank":0,"incarnation":178,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/492366509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":492366509},{"type":"v1","addr":"172.21.15.174:6837","nonce":492366509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6230,"qdb_cluster":[6230]},"id":2}]} 2024-04-25T00:22:54.036 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 186 2024-04-25T00:22:54.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:54.038+0000 7f4e81012640 1 -- 172.21.15.116:0/2780328703 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4e68067990 msgr2=0x7f4e68069e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:54.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:54.038+0000 7f4e81012640 1 --2- 172.21.15.116:0/2780328703 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4e68067990 0x7f4e68069e50 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c0c9c60 tx=0x7f4e6c0a6000 comp rx=0 tx=0).stop 2024-04-25T00:22:54.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:54.038+0000 7f4e81012640 1 -- 172.21.15.116:0/2780328703 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4e7c1572d0 msgr2=0x7f4e7c1626b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:22:54.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:54.038+0000 7f4e81012640 1 --2- 172.21.15.116:0/2780328703 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4e7c1572d0 0x7f4e7c1626b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4e7400c7b0 tx=0x7f4e7400cc70 comp rx=0 tx=0).stop 2024-04-25T00:22:54.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:54.038+0000 7f4e81012640 1 -- 172.21.15.116:0/2780328703 shutdown_connections 2024-04-25T00:22:54.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:54.038+0000 7f4e81012640 1 --2- 172.21.15.116:0/2780328703 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4e68067990 0x7f4e68069e50 secure :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c0c9c60 tx=0x7f4e6c0a6000 comp rx=0 tx=0).stop 2024-04-25T00:22:54.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:54.038+0000 7f4e81012640 1 --2- 172.21.15.116:0/2780328703 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4e7c1572d0 0x7f4e7c1626b0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:54.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:54.038+0000 7f4e81012640 1 --2- 172.21.15.116:0/2780328703 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4e7c0714d0 0x7f4e7c162170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:54.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:54.038+0000 7f4e81012640 1 --2- 172.21.15.116:0/2780328703 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e7c06e080 0x7f4e7c0adbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:22:54.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:54.038+0000 7f4e81012640 1 -- 172.21.15.116:0/2780328703 >> 172.21.15.116:0/2780328703 conn(0x7f4e7c074650 msgr2=0x7f4e7c0a9290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:22:54.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:54.038+0000 7f4e81012640 1 -- 172.21.15.116:0/2780328703 shutdown_connections 2024-04-25T00:22:54.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:22:54.038+0000 7f4e81012640 1 -- 172.21.15.116:0/2780328703 wait complete. 2024-04-25T00:22:54.078 INFO:tasks.mds_thrash.fs.[a]:mds.a has gained rank=0, replacing gid=7989 2024-04-25T00:22:54.078 INFO:tasks.mds_thrash.fs.[a]:waiting for 6 secs before reviving mds.c 2024-04-25T00:23:00.079 INFO:tasks.mds_thrash.fs.[a]:reviving mds.c 2024-04-25T00:23:00.079 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-25T00:23:00.080 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-25T00:23:00.082 INFO:tasks.ceph.mds.c:Started 2024-04-25T00:23:00.082 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:23:00.261 INFO:tasks.ceph.mds.c.smithi116.stdout:starting mds.c at 2024-04-25T00:23:00.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.322+0000 7f171423d640 1 Processor -- start 2024-04-25T00:23:00.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.322+0000 7f171423d640 1 -- start start 2024-04-25T00:23:00.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.322+0000 7f171423d640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f170c0c7680 0x7f170c15a610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:00.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.322+0000 7f171423d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f170c06f7e0 con 0x7f170c0c4840 2024-04-25T00:23:00.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.322+0000 7f171423d640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f170c06fac0 con 0x7f170c0c72a0 2024-04-25T00:23:00.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.322+0000 7f171423d640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f170c0c7b10 con 0x7f170c0c7680 2024-04-25T00:23:00.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.322+0000 7f17117d8640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f170c0c4840 0x7f170c162500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:34240/0 (socket says 172.21.15.116:34240) 2024-04-25T00:23:00.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.322+0000 7f17117d8640 1 -- 172.21.15.116:0/1582748215 learned_addr learned my addr 172.21.15.116:0/1582748215 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:23:00.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.322+0000 7f17127da640 1 --2- 172.21.15.116:0/1582748215 >> v2:172.21.15.174:3300/0 conn(0x7f170c0c7680 0x7f170c15a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:00.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.326+0000 7f1710fd7640 1 -- 172.21.15.116:0/1582748215 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 588000299 0 0) 0x7f170c06fac0 con 0x7f170c0c72a0 2024-04-25T00:23:00.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.326+0000 7f1710fd7640 1 -- 172.21.15.116:0/1582748215 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f16f8003530 con 0x7f170c0c72a0 2024-04-25T00:23:00.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.326+0000 7f1710fd7640 1 -- 172.21.15.116:0/1582748215 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4126479580 0 0) 0x7f170c06f7e0 con 0x7f170c0c4840 2024-04-25T00:23:00.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.326+0000 7f1710fd7640 1 -- 172.21.15.116:0/1582748215 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f170c06fac0 con 0x7f170c0c4840 2024-04-25T00:23:00.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.326+0000 7f17127da640 1 -- 172.21.15.116:0/1582748215 >> v1:172.21.15.174:6789/0 conn(0x7f170c0c72a0 legacy=0x7f170c072200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:00.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.326+0000 7f17127da640 1 -- 172.21.15.116:0/1582748215 >> v1:172.21.15.116:6789/0 conn(0x7f170c0c4840 legacy=0x7f170c162500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:00.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.326+0000 7f17127da640 1 -- 172.21.15.116:0/1582748215 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f170c0c3d60 con 0x7f170c0c7680 2024-04-25T00:23:00.331 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.330+0000 7f17127da640 1 --2- 172.21.15.116:0/1582748215 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f170c0c7680 0x7f170c15a610 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f16fc0097c0 tx=0x7f16fc01c3d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9bcc56de1987632c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:00.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.330+0000 7f1710fd7640 1 -- 172.21.15.116:0/1582748215 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16fc0b0070 con 0x7f170c0c7680 2024-04-25T00:23:00.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.330+0000 7f1710fd7640 1 -- 172.21.15.116:0/1582748215 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16fc0ab600 con 0x7f170c0c7680 2024-04-25T00:23:00.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.330+0000 7f1710fd7640 1 -- 172.21.15.116:0/1582748215 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16fc0b0070 con 0x7f170c0c7680 2024-04-25T00:23:00.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.330+0000 7f171423d640 1 -- 172.21.15.116:0/1582748215 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f170c0c7680 msgr2=0x7f170c15a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:00.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.330+0000 7f171423d640 1 --2- 172.21.15.116:0/1582748215 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f170c0c7680 0x7f170c15a610 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f16fc0097c0 tx=0x7f16fc01c3d0 comp rx=0 tx=0).stop 2024-04-25T00:23:00.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.334+0000 7f171423d640 1 -- 172.21.15.116:0/1582748215 shutdown_connections 2024-04-25T00:23:00.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.334+0000 7f171423d640 1 --2- 172.21.15.116:0/1582748215 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f170c0c7680 0x7f170c15a610 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:00.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.334+0000 7f171423d640 1 -- 172.21.15.116:0/1582748215 >> 172.21.15.116:0/1582748215 conn(0x7f170c0a94a0 msgr2=0x7f170c0a98b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:00.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.334+0000 7f171423d640 1 -- 172.21.15.116:0/1582748215 shutdown_connections 2024-04-25T00:23:00.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.334+0000 7f171423d640 1 -- 172.21.15.116:0/1582748215 wait complete. 2024-04-25T00:23:00.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.334+0000 7f171423d640 1 Processor -- start 2024-04-25T00:23:00.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.334+0000 7f171423d640 1 -- start start 2024-04-25T00:23:00.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.334+0000 7f171423d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f170c0c4840 0x7f170c1fb650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:00.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.334+0000 7f171423d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f170c0c72a0 0x7f170c1f8210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:00.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.334+0000 7f171423d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f170c0c7680 0x7f170c14e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:00.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.334+0000 7f171423d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f170c06f400 con 0x7f170c0c4840 2024-04-25T00:23:00.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.334+0000 7f17117d8640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f170c0c72a0 0x7f170c1f8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:00.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.334+0000 7f17117d8640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f170c0c72a0 0x7f170c1f8210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:53636/0 (socket says 172.21.15.116:53636) 2024-04-25T00:23:00.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.334+0000 7f17117d8640 1 -- 172.21.15.116:0/3563644892 learned_addr learned my addr 172.21.15.116:0/3563644892 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:23:00.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.338+0000 7f17127da640 1 --2- 172.21.15.116:0/3563644892 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f170c0c7680 0x7f170c14e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:00.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.338+0000 7f171423d640 1 -- 172.21.15.116:0/3563644892 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f170c0c43b0 con 0x7f170c0c7680 2024-04-25T00:23:00.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.338+0000 7f1711fd9640 1 --2- 172.21.15.116:0/3563644892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f170c0c4840 0x7f170c1fb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:00.338 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.338+0000 7f171423d640 1 -- 172.21.15.116:0/3563644892 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f170c06f580 con 0x7f170c0c72a0 2024-04-25T00:23:00.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.338+0000 7f17117d8640 1 -- 172.21.15.116:0/3563644892 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f170c0c7680 msgr2=0x7f170c14e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:00.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.338+0000 7f17117d8640 1 --2- 172.21.15.116:0/3563644892 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f170c0c7680 0x7f170c14e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:00.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.338+0000 7f17117d8640 1 -- 172.21.15.116:0/3563644892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f170c0c4840 msgr2=0x7f170c1fb650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:00.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.338+0000 7f17117d8640 1 --2- 172.21.15.116:0/3563644892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f170c0c4840 0x7f170c1fb650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:00.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.338+0000 7f17117d8640 1 -- 172.21.15.116:0/3563644892 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f170c1453b0 con 0x7f170c0c72a0 2024-04-25T00:23:00.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.338+0000 7f17117d8640 1 --2- 172.21.15.116:0/3563644892 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f170c0c72a0 0x7f170c1f8210 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f170800c4d0 tx=0x7f170800c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:00.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.342+0000 7f17027fc640 1 -- 172.21.15.116:0/3563644892 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1708018070 con 0x7f170c0c72a0 2024-04-25T00:23:00.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.342+0000 7f17027fc640 1 -- 172.21.15.116:0/3563644892 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17080074e0 con 0x7f170c0c72a0 2024-04-25T00:23:00.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.342+0000 7f171423d640 1 -- 172.21.15.116:0/3563644892 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f170c145800 con 0x7f170c0c72a0 2024-04-25T00:23:00.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.342+0000 7f17027fc640 1 -- 172.21.15.116:0/3563644892 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1708004a30 con 0x7f170c0c72a0 2024-04-25T00:23:00.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.342+0000 7f171423d640 1 -- 172.21.15.116:0/3563644892 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f170c1471c0 con 0x7f170c0c72a0 2024-04-25T00:23:00.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.342+0000 7f171423d640 1 -- 172.21.15.116:0/3563644892 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f170c068a60 con 0x7f170c0c72a0 2024-04-25T00:23:00.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.342+0000 7f17027fc640 1 -- 172.21.15.116:0/3563644892 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f170801b070 con 0x7f170c0c72a0 2024-04-25T00:23:00.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.342+0000 7f17027fc640 1 --2- 172.21.15.116:0/3563644892 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f16f0067940 0x7f16f0069e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:00.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.342+0000 7f17027fc640 1 -- 172.21.15.116:0/3563644892 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(63..63 src has 1..63) ==== 9676+0+0 (secure 0 0 0) 0x7f1708085ef0 con 0x7f170c0c72a0 2024-04-25T00:23:00.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.342+0000 7f1711fd9640 1 --2- 172.21.15.116:0/3563644892 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f16f0067940 0x7f16f0069e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:00.344 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.342+0000 7f1711fd9640 1 --2- 172.21.15.116:0/3563644892 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f16f0067940 0x7f16f0069e00 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f1704001570 tx=0x7f1704009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:00.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.350+0000 7f17027fc640 1 -- 172.21.15.116:0/3563644892 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f1708055520 con 0x7f170c0c72a0 2024-04-25T00:23:00.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.830+0000 7f171423d640 1 -- 172.21.15.116:0/3563644892 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f170c06e940 con 0x7f170c0c72a0 2024-04-25T00:23:00.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.830+0000 7f17027fc640 1 -- 172.21.15.116:0/3563644892 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 186 v186) ==== 95+0+5676 (secure 0 0 0) 0x7f1708059820 con 0x7f170c0c72a0 2024-04-25T00:23:00.833 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:23:00.833 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":186,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":186,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:22:51.924561+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8214},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":183,"state":"up:active","state_seq":4,"addr":"172.21.15.116:6833/2998042514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2998042514},{"type":"v1","addr":"172.21.15.116:6833","nonce":2998042514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":1},{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:22:38.936345+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6230":{"gid":6230,"name":"d","rank":0,"incarnation":178,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/492366509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":492366509},{"type":"v1","addr":"172.21.15.174:6837","nonce":492366509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6230,"qdb_cluster":[6230]},"id":2}]} 2024-04-25T00:23:00.833 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 186 2024-04-25T00:23:00.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.834+0000 7f171423d640 1 -- 172.21.15.116:0/3563644892 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f16f0067940 msgr2=0x7f16f0069e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:00.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.834+0000 7f171423d640 1 --2- 172.21.15.116:0/3563644892 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f16f0067940 0x7f16f0069e00 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f1704001570 tx=0x7f1704009040 comp rx=0 tx=0).stop 2024-04-25T00:23:00.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.834+0000 7f171423d640 1 -- 172.21.15.116:0/3563644892 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f170c0c72a0 msgr2=0x7f170c1f8210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:00.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.834+0000 7f171423d640 1 --2- 172.21.15.116:0/3563644892 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f170c0c72a0 0x7f170c1f8210 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f170800c4d0 tx=0x7f170800c990 comp rx=0 tx=0).stop 2024-04-25T00:23:00.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.834+0000 7f171423d640 1 -- 172.21.15.116:0/3563644892 shutdown_connections 2024-04-25T00:23:00.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.834+0000 7f171423d640 1 --2- 172.21.15.116:0/3563644892 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f16f0067940 0x7f16f0069e00 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:00.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.834+0000 7f171423d640 1 --2- 172.21.15.116:0/3563644892 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f170c0c7680 0x7f170c14e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:00.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.834+0000 7f171423d640 1 --2- 172.21.15.116:0/3563644892 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f170c0c72a0 0x7f170c1f8210 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:00.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.834+0000 7f171423d640 1 --2- 172.21.15.116:0/3563644892 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f170c0c4840 0x7f170c1fb650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:00.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.834+0000 7f171423d640 1 -- 172.21.15.116:0/3563644892 >> 172.21.15.116:0/3563644892 conn(0x7f170c0a94a0 msgr2=0x7f170c14b030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:00.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.834+0000 7f171423d640 1 -- 172.21.15.116:0/3563644892 shutdown_connections 2024-04-25T00:23:00.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:00.834+0000 7f171423d640 1 -- 172.21.15.116:0/3563644892 wait complete. 2024-04-25T00:23:00.871 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2024-04-25T00:23:02.872 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:23:03.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.054+0000 7f619d3d2640 1 Processor -- start 2024-04-25T00:23:03.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.054+0000 7f619d3d2640 1 -- start start 2024-04-25T00:23:03.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.054+0000 7f619d3d2640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6198153f40 0x7f619806e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:03.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.054+0000 7f619d3d2640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f61980c2a10 0x7f61980c2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:03.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.054+0000 7f619d3d2640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6198071a30 con 0x7f6198159f00 2024-04-25T00:23:03.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.054+0000 7f619d3d2640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f61980725d0 con 0x7f6198153f40 2024-04-25T00:23:03.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.054+0000 7f619d3d2640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f6198071ca0 con 0x7f61980c2a10 2024-04-25T00:23:03.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.054+0000 7f61967fc640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f61980c2a10 0x7f61980c2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:03.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.054+0000 7f61967fc640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f61980c2a10 0x7f61980c2ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:33234/0 (socket says 172.21.15.116:33234) 2024-04-25T00:23:03.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.054+0000 7f61967fc640 1 -- 172.21.15.116:0/3835462461 learned_addr learned my addr 172.21.15.116:0/3835462461 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:23:03.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.054+0000 7f6196ffd640 1 --2- 172.21.15.116:0/3835462461 >> v2:172.21.15.116:3300/0 conn(0x7f6198153f40 0x7f619806e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:03.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.054+0000 7f61967fc640 1 -- 172.21.15.116:0/3835462461 >> v2:172.21.15.116:3300/0 conn(0x7f6198153f40 msgr2=0x7f619806e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:03.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.054+0000 7f6195ffb640 1 -- 172.21.15.116:0/3835462461 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1814666287 0 0) 0x7f6198071a30 con 0x7f6198159f00 2024-04-25T00:23:03.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.058+0000 7f61967fc640 1 --2- 172.21.15.116:0/3835462461 >> v2:172.21.15.116:3300/0 conn(0x7f6198153f40 0x7f619806e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:03.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.058+0000 7f61967fc640 1 -- 172.21.15.116:0/3835462461 >> v1:172.21.15.116:6789/0 conn(0x7f6198159f00 legacy=0x7f619815a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:03.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.058+0000 7f61967fc640 1 -- 172.21.15.116:0/3835462461 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6198071f80 con 0x7f61980c2a10 2024-04-25T00:23:03.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.058+0000 7f6196ffd640 1 --2- 172.21.15.116:0/3835462461 >> v2:172.21.15.116:3300/0 conn(0x7f6198153f40 0x7f619806e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:23:03.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.058+0000 7f61967fc640 1 --2- 172.21.15.116:0/3835462461 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61980c2a10 0x7f61980c2ea0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f61980733e0 tx=0x7f618c01c4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=113e08b93a6583d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:03.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.058+0000 7f6195ffb640 1 -- 172.21.15.116:0/3835462461 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f618c0b0070 con 0x7f61980c2a10 2024-04-25T00:23:03.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.058+0000 7f6195ffb640 1 -- 172.21.15.116:0/3835462461 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f618c0ab600 con 0x7f61980c2a10 2024-04-25T00:23:03.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.058+0000 7f6195ffb640 1 -- 172.21.15.116:0/3835462461 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f618c0ab9d0 con 0x7f61980c2a10 2024-04-25T00:23:03.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.062+0000 7f619d3d2640 1 -- 172.21.15.116:0/3835462461 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61980c2a10 msgr2=0x7f61980c2ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:03.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.062+0000 7f619d3d2640 1 --2- 172.21.15.116:0/3835462461 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61980c2a10 0x7f61980c2ea0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f61980733e0 tx=0x7f618c01c4b0 comp rx=0 tx=0).stop 2024-04-25T00:23:03.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.062+0000 7f619d3d2640 1 -- 172.21.15.116:0/3835462461 shutdown_connections 2024-04-25T00:23:03.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.062+0000 7f619d3d2640 1 --2- 172.21.15.116:0/3835462461 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61980c2a10 0x7f61980c2ea0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:03.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.062+0000 7f619d3d2640 1 --2- 172.21.15.116:0/3835462461 >> v2:172.21.15.116:3300/0 conn(0x7f6198153f40 0x7f619806e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:03.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.062+0000 7f619d3d2640 1 -- 172.21.15.116:0/3835462461 >> 172.21.15.116:0/3835462461 conn(0x7f61980c3b00 msgr2=0x7f61980c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:03.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.062+0000 7f619d3d2640 1 -- 172.21.15.116:0/3835462461 shutdown_connections 2024-04-25T00:23:03.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.062+0000 7f619d3d2640 1 -- 172.21.15.116:0/3835462461 wait complete. 2024-04-25T00:23:03.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.066+0000 7f619d3d2640 1 Processor -- start 2024-04-25T00:23:03.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.066+0000 7f619d3d2640 1 -- start start 2024-04-25T00:23:03.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.066+0000 7f619d3d2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61980c2a10 0x7f6198158db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:03.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.066+0000 7f619d3d2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6198153f40 0x7f61980c5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:03.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.066+0000 7f619d3d2640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6198159f00 0x7f61980c63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:03.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.066+0000 7f619d3d2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f61980c36c0 con 0x7f6198153f40 2024-04-25T00:23:03.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.066+0000 7f619d3d2640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f619806eee0 con 0x7f61980c2a10 2024-04-25T00:23:03.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.066+0000 7f619d3d2640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f61980c3840 con 0x7f6198159f00 2024-04-25T00:23:03.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.066+0000 7f61977fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6198159f00 0x7f61980c63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:03.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.066+0000 7f61977fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6198159f00 0x7f61980c63e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:53644/0 (socket says 172.21.15.116:53644) 2024-04-25T00:23:03.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.066+0000 7f61977fe640 1 -- 172.21.15.116:0/2790014024 learned_addr learned my addr 172.21.15.116:0/2790014024 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:23:03.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.066+0000 7f61967fc640 1 --2- 172.21.15.116:0/2790014024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6198153f40 0x7f61980c5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:03.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.066+0000 7f61977fe640 1 -- 172.21.15.116:0/2790014024 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61980c2a10 msgr2=0x7f6198158db0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T00:23:03.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.066+0000 7f61977fe640 1 --2- 172.21.15.116:0/2790014024 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61980c2a10 0x7f6198158db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:03.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.066+0000 7f61977fe640 1 -- 172.21.15.116:0/2790014024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6198153f40 msgr2=0x7f61980c5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:03.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.066+0000 7f61977fe640 1 --2- 172.21.15.116:0/2790014024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6198153f40 0x7f61980c5ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:03.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.066+0000 7f61977fe640 1 -- 172.21.15.116:0/2790014024 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61980c4d90 con 0x7f6198159f00 2024-04-25T00:23:03.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.066+0000 7f61967fc640 1 --2- 172.21.15.116:0/2790014024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6198153f40 0x7f61980c5ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:23:03.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.066+0000 7f61977fe640 1 --2- 172.21.15.116:0/2790014024 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6198159f00 0x7f61980c63e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f618800c440 tx=0x7f618800c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:03.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.070+0000 7f617f7fe640 1 -- 172.21.15.116:0/2790014024 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6188017070 con 0x7f6198159f00 2024-04-25T00:23:03.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.070+0000 7f617f7fe640 1 -- 172.21.15.116:0/2790014024 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6188003600 con 0x7f6198159f00 2024-04-25T00:23:03.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.070+0000 7f619d3d2640 1 -- 172.21.15.116:0/2790014024 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61980c5070 con 0x7f6198159f00 2024-04-25T00:23:03.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.070+0000 7f619d3d2640 1 -- 172.21.15.116:0/2790014024 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6198143270 con 0x7f6198159f00 2024-04-25T00:23:03.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.070+0000 7f617f7fe640 1 -- 172.21.15.116:0/2790014024 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61880039f0 con 0x7f6198159f00 2024-04-25T00:23:03.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.070+0000 7f617f7fe640 1 -- 172.21.15.116:0/2790014024 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f618801a070 con 0x7f6198159f00 2024-04-25T00:23:03.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.070+0000 7f617f7fe640 1 --2- 172.21.15.116:0/2790014024 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6184067990 0x7f6184069e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:03.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.070+0000 7f617f7fe640 1 -- 172.21.15.116:0/2790014024 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(63..63 src has 1..63) ==== 9676+0+0 (secure 0 0 0) 0x7f6188085fe0 con 0x7f6198159f00 2024-04-25T00:23:03.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.070+0000 7f6196ffd640 1 --2- 172.21.15.116:0/2790014024 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6184067990 0x7f6184069e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:03.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.074+0000 7f619d3d2640 1 -- 172.21.15.116:0/2790014024 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61780052a0 con 0x7f6198159f00 2024-04-25T00:23:03.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.074+0000 7f6196ffd640 1 --2- 172.21.15.116:0/2790014024 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6184067990 0x7f6184069e50 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f6190009540 tx=0x7f6190009430 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:03.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.078+0000 7f617f7fe640 1 -- 172.21.15.116:0/2790014024 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f6188055cc0 con 0x7f6198159f00 2024-04-25T00:23:03.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.526+0000 7f619d3d2640 1 -- 172.21.15.116:0/2790014024 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6178005590 con 0x7f6198159f00 2024-04-25T00:23:03.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.526+0000 7f617f7fe640 1 -- 172.21.15.116:0/2790014024 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 187 v187) ==== 95+0+6505 (secure 0 0 0) 0x7f61880555c0 con 0x7f6198159f00 2024-04-25T00:23:03.527 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:23:03.527 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":187,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":8244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1971675962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1971675962},{"type":"v1","addr":"172.21.15.116:6835","nonce":1971675962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":186,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:22:51.924561+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8214},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":183,"state":"up:active","state_seq":4,"addr":"172.21.15.116:6833/2998042514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2998042514},{"type":"v1","addr":"172.21.15.116:6833","nonce":2998042514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":1},{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:22:38.936345+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6230":{"gid":6230,"name":"d","rank":0,"incarnation":178,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/492366509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":492366509},{"type":"v1","addr":"172.21.15.174:6837","nonce":492366509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6230,"qdb_cluster":[6230]},"id":2}]} 2024-04-25T00:23:03.527 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 187 2024-04-25T00:23:03.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.530+0000 7f619d3d2640 1 -- 172.21.15.116:0/2790014024 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6184067990 msgr2=0x7f6184069e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:03.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.530+0000 7f619d3d2640 1 --2- 172.21.15.116:0/2790014024 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6184067990 0x7f6184069e50 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f6190009540 tx=0x7f6190009430 comp rx=0 tx=0).stop 2024-04-25T00:23:03.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.530+0000 7f619d3d2640 1 -- 172.21.15.116:0/2790014024 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6198159f00 msgr2=0x7f61980c63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:03.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.530+0000 7f619d3d2640 1 --2- 172.21.15.116:0/2790014024 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6198159f00 0x7f61980c63e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f618800c440 tx=0x7f618800c900 comp rx=0 tx=0).stop 2024-04-25T00:23:03.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.530+0000 7f619d3d2640 1 -- 172.21.15.116:0/2790014024 shutdown_connections 2024-04-25T00:23:03.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.530+0000 7f619d3d2640 1 --2- 172.21.15.116:0/2790014024 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6184067990 0x7f6184069e50 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:03.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.530+0000 7f619d3d2640 1 --2- 172.21.15.116:0/2790014024 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6198159f00 0x7f61980c63e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:03.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.530+0000 7f619d3d2640 1 --2- 172.21.15.116:0/2790014024 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6198153f40 0x7f61980c5ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:03.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.530+0000 7f619d3d2640 1 --2- 172.21.15.116:0/2790014024 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f61980c2a10 0x7f6198158db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:03.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.530+0000 7f619d3d2640 1 -- 172.21.15.116:0/2790014024 >> 172.21.15.116:0/2790014024 conn(0x7f61980c3b00 msgr2=0x7f6198152110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:03.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.530+0000 7f619d3d2640 1 -- 172.21.15.116:0/2790014024 shutdown_connections 2024-04-25T00:23:03.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:03.530+0000 7f619d3d2640 1 -- 172.21.15.116:0/2790014024 wait complete. 2024-04-25T00:23:03.556 INFO:tasks.mds_thrash.fs.[a]:mds.c reported in up:standby state 2024-04-25T00:23:03.556 INFO:tasks.mds_thrash.fs.[a]:waiting for 8 secs before thrashing 2024-04-25T00:23:10.249 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:23:10.251 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:23:11.557 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:23:11.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:11.790+0000 7fe60b59e640 1 Processor -- start 2024-04-25T00:23:11.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:11.790+0000 7fe60b59e640 1 -- start start 2024-04-25T00:23:11.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:11.790+0000 7fe60b59e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe60c1571c0 0x7fe60c0c2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:11.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:11.790+0000 7fe60b59e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe60c0c4880 0x7fe60c0c3230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:11.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:11.790+0000 7fe60b59e640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe60c06c8a0 con 0x7fe60c0c1a80 2024-04-25T00:23:11.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:11.790+0000 7fe60b59e640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fe60c0c2210 con 0x7fe60c0c4880 2024-04-25T00:23:11.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:11.790+0000 7fe60b59e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe60c0c24f0 con 0x7fe60c1571c0 2024-04-25T00:23:11.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:11.790+0000 7fe603fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe60c0c4880 0x7fe60c0c3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:11.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:11.790+0000 7fe603fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe60c0c4880 0x7fe60c0c3230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51830/0 (socket says 172.21.15.116:51830) 2024-04-25T00:23:11.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:11.790+0000 7fe603fff640 1 -- 172.21.15.116:0/704253712 learned_addr learned my addr 172.21.15.116:0/704253712 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:23:11.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:11.790+0000 7fe60ad9d640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fe60c0c1a80 0x7fe60c159cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:44034/0 (socket says 172.21.15.116:44034) 2024-04-25T00:23:11.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:11.790+0000 7fe603fff640 1 -- 172.21.15.116:0/704253712 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe60c1571c0 msgr2=0x7fe60c0c2cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T00:23:11.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:11.790+0000 7fe603fff640 1 --2- 172.21.15.116:0/704253712 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe60c1571c0 0x7fe60c0c2cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:11.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:11.790+0000 7fe603fff640 1 -- 172.21.15.116:0/704253712 >> v1:172.21.15.174:6789/0 conn(0x7fe60c0c1a80 legacy=0x7fe60c159cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:11.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:11.790+0000 7fe603fff640 1 -- 172.21.15.116:0/704253712 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe60c0c27d0 con 0x7fe60c0c4880 2024-04-25T00:23:11.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:11.790+0000 7fe609d9b640 1 -- 172.21.15.116:0/704253712 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2628204157 0 0) 0x7fe60c06c8a0 con 0x7fe60c0c1a80 2024-04-25T00:23:11.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:11.794+0000 7fe603fff640 1 --2- 172.21.15.116:0/704253712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe60c0c4880 0x7fe60c0c3230 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7fe5f4009870 tx=0x7fe5f401c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e301688a56ea90e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:11.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:11.798+0000 7fe609d9b640 1 -- 172.21.15.116:0/704253712 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f40b0070 con 0x7fe60c0c4880 2024-04-25T00:23:11.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:11.798+0000 7fe609d9b640 1 -- 172.21.15.116:0/704253712 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5f40ab600 con 0x7fe60c0c4880 2024-04-25T00:23:11.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:11.798+0000 7fe60b59e640 1 -- 172.21.15.116:0/704253712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe60c0c4880 msgr2=0x7fe60c0c3230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:11.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:11.798+0000 7fe60b59e640 1 --2- 172.21.15.116:0/704253712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe60c0c4880 0x7fe60c0c3230 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7fe5f4009870 tx=0x7fe5f401c460 comp rx=0 tx=0).stop 2024-04-25T00:23:11.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:11.798+0000 7fe60b59e640 1 -- 172.21.15.116:0/704253712 shutdown_connections 2024-04-25T00:23:11.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:11.798+0000 7fe60b59e640 1 --2- 172.21.15.116:0/704253712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe60c0c4880 0x7fe60c0c3230 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:11.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:11.798+0000 7fe60b59e640 1 --2- 172.21.15.116:0/704253712 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe60c1571c0 0x7fe60c0c2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:11.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:11.798+0000 7fe60b59e640 1 -- 172.21.15.116:0/704253712 >> 172.21.15.116:0/704253712 conn(0x7fe60c0a9a20 msgr2=0x7fe60c0c4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:11.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:11.798+0000 7fe60b59e640 1 -- 172.21.15.116:0/704253712 shutdown_connections 2024-04-25T00:23:11.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:11.798+0000 7fe60b59e640 1 -- 172.21.15.116:0/704253712 wait complete. 2024-04-25T00:23:11.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:11.802+0000 7fe60b59e640 1 Processor -- start 2024-04-25T00:23:11.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:11.802+0000 7fe60b59e640 1 -- start start 2024-04-25T00:23:11.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:11.802+0000 7fe60b59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe60c0c1a80 0x7fe60c15b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:11.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:11.802+0000 7fe60b59e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe60c0c4880 0x7fe60c1f30e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:11.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:11.802+0000 7fe60a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe60c0c1a80 0x7fe60c15b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:11.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:11.802+0000 7fe60a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe60c0c1a80 0x7fe60c15b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51832/0 (socket says 172.21.15.116:51832) 2024-04-25T00:23:11.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:11.802+0000 7fe60a59c640 1 -- 172.21.15.116:0/300516607 learned_addr learned my addr 172.21.15.116:0/300516607 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:23:11.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:11.802+0000 7fe60b59e640 1 --2- 172.21.15.116:0/300516607 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe60c1571c0 0x7fe60c156370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:11.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:11.802+0000 7fe60b59e640 1 -- 172.21.15.116:0/300516607 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe60c166d90 con 0x7fe60c0c1a80 2024-04-25T00:23:11.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:11.802+0000 7fe60b59e640 1 -- 172.21.15.116:0/300516607 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe60c0c5010 con 0x7fe60c0c4880 2024-04-25T00:23:11.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:11.802+0000 7fe60b59e640 1 -- 172.21.15.116:0/300516607 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe60c166f10 con 0x7fe60c1571c0 2024-04-25T00:23:11.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:11.802+0000 7fe603fff640 1 --2- 172.21.15.116:0/300516607 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe60c0c4880 0x7fe60c1f30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:11.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:11.802+0000 7fe60a59c640 1 -- 172.21.15.116:0/300516607 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe60c1571c0 msgr2=0x7fe60c156370 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T00:23:11.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:11.802+0000 7fe60a59c640 1 --2- 172.21.15.116:0/300516607 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe60c1571c0 0x7fe60c156370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:11.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:11.802+0000 7fe60a59c640 1 -- 172.21.15.116:0/300516607 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe60c0c4880 msgr2=0x7fe60c1f30e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:11.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:11.802+0000 7fe60a59c640 1 --2- 172.21.15.116:0/300516607 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe60c0c4880 0x7fe60c1f30e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:11.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:11.802+0000 7fe60a59c640 1 -- 172.21.15.116:0/300516607 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe60c15d650 con 0x7fe60c0c1a80 2024-04-25T00:23:11.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:11.802+0000 7fe603fff640 1 --2- 172.21.15.116:0/300516607 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe60c0c4880 0x7fe60c1f30e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:23:11.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:11.802+0000 7fe60a59c640 1 --2- 172.21.15.116:0/300516607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe60c0c1a80 0x7fe60c15b820 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc00b3e0 tx=0x7fe5fc00b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:11.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:11.806+0000 7fe602ffd640 1 -- 172.21.15.116:0/300516607 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5fc015020 con 0x7fe60c0c1a80 2024-04-25T00:23:11.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:11.806+0000 7fe602ffd640 1 -- 172.21.15.116:0/300516607 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5fc011850 con 0x7fe60c0c1a80 2024-04-25T00:23:11.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:11.806+0000 7fe602ffd640 1 -- 172.21.15.116:0/300516607 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5fc011c40 con 0x7fe60c0c1a80 2024-04-25T00:23:11.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:11.806+0000 7fe60b59e640 1 -- 172.21.15.116:0/300516607 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe60c15d930 con 0x7fe60c0c1a80 2024-04-25T00:23:11.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:11.806+0000 7fe602ffd640 1 -- 172.21.15.116:0/300516607 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe5fc018020 con 0x7fe60c0c1a80 2024-04-25T00:23:11.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:11.806+0000 7fe60b59e640 1 -- 172.21.15.116:0/300516607 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe60c1568b0 con 0x7fe60c0c1a80 2024-04-25T00:23:11.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:11.806+0000 7fe602ffd640 1 --2- 172.21.15.116:0/300516607 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe5ec067910 0x7fe5ec069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:11.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:11.806+0000 7fe602ffd640 1 -- 172.21.15.116:0/300516607 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 9676+0+0 (secure 0 0 0) 0x7fe5fc089170 con 0x7fe60c0c1a80 2024-04-25T00:23:11.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:11.810+0000 7fe603fff640 1 --2- 172.21.15.116:0/300516607 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe5ec067910 0x7fe5ec069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:11.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:11.810+0000 7fe603fff640 1 --2- 172.21.15.116:0/300516607 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe5ec067910 0x7fe5ec069dd0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fe5f40ad000 tx=0x7fe5f40a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:11.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:11.810+0000 7fe60b59e640 1 -- 172.21.15.116:0/300516607 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe60c04f510 con 0x7fe60c0c1a80 2024-04-25T00:23:11.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:11.818+0000 7fe602ffd640 1 -- 172.21.15.116:0/300516607 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fe5fc00d280 con 0x7fe60c0c1a80 2024-04-25T00:23:12.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.266+0000 7fe60b59e640 1 -- 172.21.15.116:0/300516607 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe60c164350 con 0x7fe60c0c1a80 2024-04-25T00:23:12.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.270+0000 7fe602ffd640 1 -- 172.21.15.116:0/300516607 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 187 v187) ==== 95+0+6505 (secure 0 0 0) 0x7fe5fc0559f0 con 0x7fe60c0c1a80 2024-04-25T00:23:12.270 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:23:12.270 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":187,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":8244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1971675962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1971675962},{"type":"v1","addr":"172.21.15.116:6835","nonce":1971675962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":186,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:22:51.924561+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8214},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":183,"state":"up:active","state_seq":4,"addr":"172.21.15.116:6833/2998042514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2998042514},{"type":"v1","addr":"172.21.15.116:6833","nonce":2998042514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":1},{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:22:38.936345+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6230":{"gid":6230,"name":"d","rank":0,"incarnation":178,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/492366509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":492366509},{"type":"v1","addr":"172.21.15.174:6837","nonce":492366509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6230,"qdb_cluster":[6230]},"id":2}]} 2024-04-25T00:23:12.270 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 187 2024-04-25T00:23:12.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.274+0000 7fe60b59e640 1 -- 172.21.15.116:0/300516607 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe5ec067910 msgr2=0x7fe5ec069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:12.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.274+0000 7fe60b59e640 1 --2- 172.21.15.116:0/300516607 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe5ec067910 0x7fe5ec069dd0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fe5f40ad000 tx=0x7fe5f40a6000 comp rx=0 tx=0).stop 2024-04-25T00:23:12.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.274+0000 7fe60b59e640 1 -- 172.21.15.116:0/300516607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe60c0c1a80 msgr2=0x7fe60c15b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:12.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.274+0000 7fe60b59e640 1 --2- 172.21.15.116:0/300516607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe60c0c1a80 0x7fe60c15b820 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc00b3e0 tx=0x7fe5fc00b8a0 comp rx=0 tx=0).stop 2024-04-25T00:23:12.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.274+0000 7fe60b59e640 1 -- 172.21.15.116:0/300516607 shutdown_connections 2024-04-25T00:23:12.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.274+0000 7fe60b59e640 1 --2- 172.21.15.116:0/300516607 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe5ec067910 0x7fe5ec069dd0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:12.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.274+0000 7fe60b59e640 1 --2- 172.21.15.116:0/300516607 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe60c1571c0 0x7fe60c156370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:12.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.274+0000 7fe60b59e640 1 --2- 172.21.15.116:0/300516607 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe60c0c4880 0x7fe60c1f30e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:12.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.274+0000 7fe60b59e640 1 --2- 172.21.15.116:0/300516607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe60c0c1a80 0x7fe60c15b820 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:12.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.274+0000 7fe60b59e640 1 -- 172.21.15.116:0/300516607 >> 172.21.15.116:0/300516607 conn(0x7fe60c0a9a20 msgr2=0x7fe60c06d0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:12.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.274+0000 7fe60b59e640 1 -- 172.21.15.116:0/300516607 shutdown_connections 2024-04-25T00:23:12.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.274+0000 7fe60b59e640 1 -- 172.21.15.116:0/300516607 wait complete. 2024-04-25T00:23:12.310 INFO:tasks.mds_thrash.fs.[a]:kill mds.a (rank=0) 2024-04-25T00:23:12.311 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-25T00:23:12.311 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:23:12.331 INFO:tasks.ceph.mds.a:Stopped 2024-04-25T00:23:12.332 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:23:12.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.610+0000 7f262e1de640 1 Processor -- start 2024-04-25T00:23:12.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.610+0000 7f262e1de640 1 -- start start 2024-04-25T00:23:12.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.610+0000 7f262e1de640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f262806c480 0x7f26280c2170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:12.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.610+0000 7f262e1de640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f262815a670 0x7f2628157060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:12.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.610+0000 7f262e1de640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f26281575a0 0x7f26280c62d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:12.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.610+0000 7f262e1de640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f262815ac60 con 0x7f262815a670 2024-04-25T00:23:12.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.610+0000 7f262e1de640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f26280c39a0 con 0x7f26281575a0 2024-04-25T00:23:12.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.610+0000 7f262e1de640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f26280c3bd0 con 0x7f262806c480 2024-04-25T00:23:12.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.610+0000 7f2627fff640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f26281575a0 0x7f26280c62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:12.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.610+0000 7f26277fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f262806c480 0x7f26280c2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:12.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.610+0000 7f2627fff640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f26281575a0 0x7f26280c62d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:33650/0 (socket says 172.21.15.116:33650) 2024-04-25T00:23:12.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.610+0000 7f2627fff640 1 -- 172.21.15.116:0/3365934915 learned_addr learned my addr 172.21.15.116:0/3365934915 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:23:12.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.610+0000 7f2626ffd640 1 --2- 172.21.15.116:0/3365934915 >> v2:172.21.15.116:3300/0 conn(0x7f262815a670 0x7f2628157060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:12.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.614+0000 7f26277fe640 1 -- 172.21.15.116:0/3365934915 >> v2:172.21.15.174:3300/0 conn(0x7f26281575a0 msgr2=0x7f26280c62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:12.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.614+0000 7f26277fe640 1 --2- 172.21.15.116:0/3365934915 >> v2:172.21.15.174:3300/0 conn(0x7f26281575a0 0x7f26280c62d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:12.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.614+0000 7f26277fe640 1 -- 172.21.15.116:0/3365934915 >> v2:172.21.15.116:3300/0 conn(0x7f262815a670 msgr2=0x7f2628157060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:12.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.614+0000 7f26277fe640 1 --2- 172.21.15.116:0/3365934915 >> v2:172.21.15.116:3300/0 conn(0x7f262815a670 0x7f2628157060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:12.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.614+0000 7f26277fe640 1 -- 172.21.15.116:0/3365934915 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26280c5dd0 con 0x7f262806c480 2024-04-25T00:23:12.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.614+0000 7f2627fff640 1 --2- 172.21.15.116:0/3365934915 >> v2:172.21.15.174:3300/0 conn(0x7f26281575a0 0x7f26280c62d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:23:12.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.614+0000 7f2626ffd640 1 --2- 172.21.15.116:0/3365934915 >> v2:172.21.15.116:3300/0 conn(0x7f262815a670 0x7f2628157060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:23:12.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.614+0000 7f26277fe640 1 --2- 172.21.15.116:0/3365934915 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f262806c480 0x7f26280c2170 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2628074a30 tx=0x7f261801c430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68be4f11a97a833e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:12.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.614+0000 7f26267fc640 1 -- 172.21.15.116:0/3365934915 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26180b0070 con 0x7f262806c480 2024-04-25T00:23:12.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.614+0000 7f26267fc640 1 -- 172.21.15.116:0/3365934915 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26180ab600 con 0x7f262806c480 2024-04-25T00:23:12.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.614+0000 7f26267fc640 1 -- 172.21.15.116:0/3365934915 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26180aba10 con 0x7f262806c480 2024-04-25T00:23:12.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.618+0000 7f262e1de640 1 -- 172.21.15.116:0/3365934915 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f262806c480 msgr2=0x7f26280c2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:12.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.618+0000 7f262e1de640 1 --2- 172.21.15.116:0/3365934915 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f262806c480 0x7f26280c2170 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2628074a30 tx=0x7f261801c430 comp rx=0 tx=0).stop 2024-04-25T00:23:12.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.618+0000 7f262e1de640 1 -- 172.21.15.116:0/3365934915 shutdown_connections 2024-04-25T00:23:12.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.618+0000 7f262e1de640 1 --2- 172.21.15.116:0/3365934915 >> v2:172.21.15.174:3300/0 conn(0x7f26281575a0 0x7f26280c62d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:12.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.618+0000 7f262e1de640 1 --2- 172.21.15.116:0/3365934915 >> v2:172.21.15.116:3300/0 conn(0x7f262815a670 0x7f2628157060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:12.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.618+0000 7f262e1de640 1 --2- 172.21.15.116:0/3365934915 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f262806c480 0x7f26280c2170 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:12.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.618+0000 7f262e1de640 1 -- 172.21.15.116:0/3365934915 >> 172.21.15.116:0/3365934915 conn(0x7f26280c4270 msgr2=0x7f26280c4680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:12.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.618+0000 7f262e1de640 1 -- 172.21.15.116:0/3365934915 shutdown_connections 2024-04-25T00:23:12.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.618+0000 7f262e1de640 1 -- 172.21.15.116:0/3365934915 wait complete. 2024-04-25T00:23:12.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.618+0000 7f262e1de640 1 Processor -- start 2024-04-25T00:23:12.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.618+0000 7f262e1de640 1 -- start start 2024-04-25T00:23:12.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.618+0000 7f262e1de640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f262806c480 0x7f2628072a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:12.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.618+0000 7f262e1de640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f26281575a0 0x7f26281f5b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:12.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.618+0000 7f262e1de640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f262815a670 0x7f26281f6060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:12.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.618+0000 7f26277fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f262806c480 0x7f2628072a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:12.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.622+0000 7f262e1de640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f262815d210 con 0x7f26281575a0 2024-04-25T00:23:12.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.622+0000 7f2626ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f26281575a0 0x7f26281f5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:12.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.622+0000 7f2626ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f26281575a0 0x7f26281f5b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51860/0 (socket says 172.21.15.116:51860) 2024-04-25T00:23:12.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.622+0000 7f2626ffd640 1 -- 172.21.15.116:0/1246630971 learned_addr learned my addr 172.21.15.116:0/1246630971 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:23:12.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.622+0000 7f2627fff640 1 --2- 172.21.15.116:0/1246630971 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f262815a670 0x7f26281f6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:12.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.622+0000 7f262e1de640 1 -- 172.21.15.116:0/1246630971 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f262806ca70 con 0x7f262806c480 2024-04-25T00:23:12.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.622+0000 7f262e1de640 1 -- 172.21.15.116:0/1246630971 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f262815d390 con 0x7f262815a670 2024-04-25T00:23:12.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.622+0000 7f26277fe640 1 -- 172.21.15.116:0/1246630971 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f262815a670 msgr2=0x7f26281f6060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:12.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.622+0000 7f26277fe640 1 --2- 172.21.15.116:0/1246630971 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f262815a670 0x7f26281f6060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:12.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.622+0000 7f26277fe640 1 -- 172.21.15.116:0/1246630971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f26281575a0 msgr2=0x7f26281f5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:12.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.622+0000 7f26277fe640 1 --2- 172.21.15.116:0/1246630971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f26281575a0 0x7f26281f5b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:12.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.622+0000 7f26277fe640 1 -- 172.21.15.116:0/1246630971 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2628150d80 con 0x7f262806c480 2024-04-25T00:23:12.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.622+0000 7f2627fff640 1 --2- 172.21.15.116:0/1246630971 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f262815a670 0x7f26281f6060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:23:12.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.630+0000 7f26277fe640 1 --2- 172.21.15.116:0/1246630971 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f262806c480 0x7f2628072a30 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f26280c3d50 tx=0x7f26180191d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:12.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.630+0000 7f260ffff640 1 -- 172.21.15.116:0/1246630971 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26180b0040 con 0x7f262806c480 2024-04-25T00:23:12.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.630+0000 7f260ffff640 1 -- 172.21.15.116:0/1246630971 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26180b9070 con 0x7f262806c480 2024-04-25T00:23:12.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.630+0000 7f260ffff640 1 -- 172.21.15.116:0/1246630971 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26180b4650 con 0x7f262806c480 2024-04-25T00:23:12.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.630+0000 7f262e1de640 1 -- 172.21.15.116:0/1246630971 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f262814e590 con 0x7f262806c480 2024-04-25T00:23:12.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.634+0000 7f262e1de640 1 -- 172.21.15.116:0/1246630971 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f262814f0d0 con 0x7f262806c480 2024-04-25T00:23:12.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.634+0000 7f262e1de640 1 -- 172.21.15.116:0/1246630971 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2628068a60 con 0x7f262806c480 2024-04-25T00:23:12.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.634+0000 7f260ffff640 1 -- 172.21.15.116:0/1246630971 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f26180da890 con 0x7f262806c480 2024-04-25T00:23:12.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.634+0000 7f260ffff640 1 --2- 172.21.15.116:0/1246630971 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f26140678c0 0x7f2614069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:12.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.638+0000 7f260ffff640 1 -- 172.21.15.116:0/1246630971 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 9676+0+0 (secure 0 0 0) 0x7f2618120910 con 0x7f262806c480 2024-04-25T00:23:12.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.638+0000 7f2626ffd640 1 --2- 172.21.15.116:0/1246630971 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f26140678c0 0x7f2614069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:12.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.638+0000 7f2626ffd640 1 --2- 172.21.15.116:0/1246630971 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f26140678c0 0x7f2614069d80 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f26280ada30 tx=0x7f261c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:12.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:12.646+0000 7f260ffff640 1 -- 172.21.15.116:0/1246630971 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f26180160b0 con 0x7f262806c480 2024-04-25T00:23:13.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:13.090+0000 7f262e1de640 1 -- 172.21.15.116:0/1246630971 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f26280a8f60 con 0x7f262806c480 2024-04-25T00:23:13.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:13.090+0000 7f260ffff640 1 -- 172.21.15.116:0/1246630971 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 187 v187) ==== 95+0+6505 (secure 0 0 0) 0x7f26180efe20 con 0x7f262806c480 2024-04-25T00:23:13.092 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:23:13.092 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":187,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":8244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1971675962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1971675962},{"type":"v1","addr":"172.21.15.116:6835","nonce":1971675962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":186,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:22:51.924561+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8214},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":183,"state":"up:active","state_seq":4,"addr":"172.21.15.116:6833/2998042514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2998042514},{"type":"v1","addr":"172.21.15.116:6833","nonce":2998042514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":1},{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:22:38.936345+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6230":{"gid":6230,"name":"d","rank":0,"incarnation":178,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/492366509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":492366509},{"type":"v1","addr":"172.21.15.174:6837","nonce":492366509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6230,"qdb_cluster":[6230]},"id":2}]} 2024-04-25T00:23:13.093 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 187 2024-04-25T00:23:13.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:13.110+0000 7f262e1de640 1 -- 172.21.15.116:0/1246630971 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f26140678c0 msgr2=0x7f2614069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:13.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:13.110+0000 7f262e1de640 1 --2- 172.21.15.116:0/1246630971 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f26140678c0 0x7f2614069d80 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f26280ada30 tx=0x7f261c009190 comp rx=0 tx=0).stop 2024-04-25T00:23:13.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:13.110+0000 7f262e1de640 1 -- 172.21.15.116:0/1246630971 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f262806c480 msgr2=0x7f2628072a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:13.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:13.110+0000 7f262e1de640 1 --2- 172.21.15.116:0/1246630971 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f262806c480 0x7f2628072a30 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f26280c3d50 tx=0x7f26180191d0 comp rx=0 tx=0).stop 2024-04-25T00:23:13.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:13.110+0000 7f262e1de640 1 -- 172.21.15.116:0/1246630971 shutdown_connections 2024-04-25T00:23:13.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:13.110+0000 7f262e1de640 1 --2- 172.21.15.116:0/1246630971 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f26140678c0 0x7f2614069d80 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:13.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:13.110+0000 7f262e1de640 1 --2- 172.21.15.116:0/1246630971 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f262815a670 0x7f26281f6060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:13.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:13.110+0000 7f262e1de640 1 --2- 172.21.15.116:0/1246630971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f26281575a0 0x7f26281f5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:13.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:13.110+0000 7f262e1de640 1 --2- 172.21.15.116:0/1246630971 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f262806c480 0x7f2628072a30 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:13.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:13.110+0000 7f262e1de640 1 -- 172.21.15.116:0/1246630971 >> 172.21.15.116:0/1246630971 conn(0x7f26280c4270 msgr2=0x7f262814a700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:13.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:13.110+0000 7f262e1de640 1 -- 172.21.15.116:0/1246630971 shutdown_connections 2024-04-25T00:23:13.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:13.110+0000 7f262e1de640 1 -- 172.21.15.116:0/1246630971 wait complete. 2024-04-25T00:23:13.143 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:23:15.144 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:23:15.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.310+0000 7f81e4a5e640 1 Processor -- start 2024-04-25T00:23:15.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.310+0000 7f81e4a5e640 1 -- start start 2024-04-25T00:23:15.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.310+0000 7f81e4a5e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f81e00c2270 0x7f81e01597e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:15.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.310+0000 7f81e4a5e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f81e0156220 0x7f81e00c4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:15.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.310+0000 7f81e4a5e640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81e00c3dc0 con 0x7f81e0152ea0 2024-04-25T00:23:15.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.310+0000 7f81e4a5e640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f81e0158d90 con 0x7f81e00c2270 2024-04-25T00:23:15.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.310+0000 7f81e4a5e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f81e0159350 con 0x7f81e0156220 2024-04-25T00:23:15.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.310+0000 7f81de59c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f81e0152ea0 0x7f81e00c36b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:33876/0 (socket says 172.21.15.116:33876) 2024-04-25T00:23:15.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.310+0000 7f81de59c640 1 -- 172.21.15.116:0/3732127930 learned_addr learned my addr 172.21.15.116:0/3732127930 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:23:15.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.310+0000 7f81ddd9b640 1 --2- 172.21.15.116:0/3732127930 >> v2:172.21.15.116:3300/0 conn(0x7f81e00c2270 0x7f81e01597e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:15.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.314+0000 7f81ddd9b640 1 -- 172.21.15.116:0/3732127930 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f81e0156220 msgr2=0x7f81e00c4fa0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T00:23:15.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.314+0000 7f81ddd9b640 1 --2- 172.21.15.116:0/3732127930 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f81e0156220 0x7f81e00c4fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:15.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.314+0000 7f81ddd9b640 1 -- 172.21.15.116:0/3732127930 >> v1:172.21.15.116:6789/0 conn(0x7f81e0152ea0 legacy=0x7f81e00c36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:15.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.314+0000 7f81ddd9b640 1 -- 172.21.15.116:0/3732127930 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81e0072f70 con 0x7f81e00c2270 2024-04-25T00:23:15.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.314+0000 7f81ddd9b640 1 --2- 172.21.15.116:0/3732127930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81e00c2270 0x7f81e01597e0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f81e0158f10 tx=0x7f81d001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47672864ed7eac09 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:15.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.318+0000 7f81dd59a640 1 -- 172.21.15.116:0/3732127930 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81d00b0070 con 0x7f81e00c2270 2024-04-25T00:23:15.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.318+0000 7f81dd59a640 1 -- 172.21.15.116:0/3732127930 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81d00ab600 con 0x7f81e00c2270 2024-04-25T00:23:15.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.318+0000 7f81e4a5e640 1 -- 172.21.15.116:0/3732127930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81e00c2270 msgr2=0x7f81e01597e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:15.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.318+0000 7f81e4a5e640 1 --2- 172.21.15.116:0/3732127930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81e00c2270 0x7f81e01597e0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f81e0158f10 tx=0x7f81d001c480 comp rx=0 tx=0).stop 2024-04-25T00:23:15.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.318+0000 7f81e4a5e640 1 -- 172.21.15.116:0/3732127930 shutdown_connections 2024-04-25T00:23:15.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.318+0000 7f81e4a5e640 1 --2- 172.21.15.116:0/3732127930 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f81e0156220 0x7f81e00c4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:15.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.318+0000 7f81e4a5e640 1 --2- 172.21.15.116:0/3732127930 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81e00c2270 0x7f81e01597e0 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:15.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.318+0000 7f81e4a5e640 1 -- 172.21.15.116:0/3732127930 >> 172.21.15.116:0/3732127930 conn(0x7f81e006c720 msgr2=0x7f81e006cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:15.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.318+0000 7f81e4a5e640 1 -- 172.21.15.116:0/3732127930 shutdown_connections 2024-04-25T00:23:15.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.318+0000 7f81e4a5e640 1 -- 172.21.15.116:0/3732127930 wait complete. 2024-04-25T00:23:15.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.322+0000 7f81e4a5e640 1 Processor -- start 2024-04-25T00:23:15.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.322+0000 7f81e4a5e640 1 -- start start 2024-04-25T00:23:15.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.322+0000 7f81e4a5e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81e00c2270 0x7f81e006f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:15.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.322+0000 7f81e4a5e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81e0152ea0 0x7f81e00a9e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:15.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.322+0000 7f81e4a5e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f81e0156220 0x7f81e0148a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:15.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.322+0000 7f81e4a5e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f81e0162ac0 con 0x7f81e0152ea0 2024-04-25T00:23:15.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.322+0000 7f81e4a5e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f81e0162940 con 0x7f81e00c2270 2024-04-25T00:23:15.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.322+0000 7f81e4a5e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f81e0162c40 con 0x7f81e0156220 2024-04-25T00:23:15.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.322+0000 7f81ddd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81e0152ea0 0x7f81e00a9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:15.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.322+0000 7f81ddd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81e0152ea0 0x7f81e00a9e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39562/0 (socket says 172.21.15.116:39562) 2024-04-25T00:23:15.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.322+0000 7f81ddd9b640 1 -- 172.21.15.116:0/997890980 learned_addr learned my addr 172.21.15.116:0/997890980 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:23:15.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.322+0000 7f81ddd9b640 1 -- 172.21.15.116:0/997890980 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f81e0156220 msgr2=0x7f81e0148a40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T00:23:15.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.322+0000 7f81ddd9b640 1 --2- 172.21.15.116:0/997890980 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f81e0156220 0x7f81e0148a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:15.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.322+0000 7f81ddd9b640 1 -- 172.21.15.116:0/997890980 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81e00c2270 msgr2=0x7f81e006f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:15.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.322+0000 7f81ddd9b640 1 --2- 172.21.15.116:0/997890980 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81e00c2270 0x7f81e006f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:15.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.322+0000 7f81ddd9b640 1 -- 172.21.15.116:0/997890980 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81e006fe90 con 0x7f81e0152ea0 2024-04-25T00:23:15.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.322+0000 7f81ddd9b640 1 --2- 172.21.15.116:0/997890980 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81e0152ea0 0x7f81e00a9e80 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f81d001ab30 tx=0x7f81d0019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:15.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.326+0000 7f81ceffd640 1 -- 172.21.15.116:0/997890980 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81d00b0040 con 0x7f81e0152ea0 2024-04-25T00:23:15.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.326+0000 7f81ceffd640 1 -- 172.21.15.116:0/997890980 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81d00bb070 con 0x7f81e0152ea0 2024-04-25T00:23:15.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.326+0000 7f81e4a5e640 1 -- 172.21.15.116:0/997890980 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81e006dfa0 con 0x7f81e0152ea0 2024-04-25T00:23:15.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.326+0000 7f81e4a5e640 1 -- 172.21.15.116:0/997890980 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81e006e560 con 0x7f81e0152ea0 2024-04-25T00:23:15.327 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.326+0000 7f81e4a5e640 1 -- 172.21.15.116:0/997890980 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81e006f020 con 0x7f81e0152ea0 2024-04-25T00:23:15.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.326+0000 7f81ceffd640 1 -- 172.21.15.116:0/997890980 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81d00aac40 con 0x7f81e0152ea0 2024-04-25T00:23:15.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.326+0000 7f81ceffd640 1 -- 172.21.15.116:0/997890980 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f81d00b3070 con 0x7f81e0152ea0 2024-04-25T00:23:15.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.330+0000 7f81ceffd640 1 --2- 172.21.15.116:0/997890980 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f81c00679c0 0x7f81c0069e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:15.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.330+0000 7f81de59c640 1 --2- 172.21.15.116:0/997890980 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f81c00679c0 0x7f81c0069e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:15.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.330+0000 7f81ceffd640 1 -- 172.21.15.116:0/997890980 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 9676+0+0 (secure 0 0 0) 0x7f81d0121940 con 0x7f81e0152ea0 2024-04-25T00:23:15.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.330+0000 7f81de59c640 1 --2- 172.21.15.116:0/997890980 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f81c00679c0 0x7f81c0069e80 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f81d8005c10 tx=0x7f81d8001630 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:15.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.334+0000 7f81ceffd640 1 -- 172.21.15.116:0/997890980 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f81e006f020 con 0x7f81e0152ea0 2024-04-25T00:23:15.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.690+0000 7f81e4a5e640 1 -- 172.21.15.116:0/997890980 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f81e0068a60 con 0x7f81e0152ea0 2024-04-25T00:23:15.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.690+0000 7f81ceffd640 1 -- 172.21.15.116:0/997890980 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 187 v187) ==== 95+0+6505 (secure 0 0 0) 0x7f81d0016070 con 0x7f81e0152ea0 2024-04-25T00:23:15.693 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:23:15.693 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":187,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":8244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1971675962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1971675962},{"type":"v1","addr":"172.21.15.116:6835","nonce":1971675962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":186,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:22:51.924561+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8214},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":183,"state":"up:active","state_seq":4,"addr":"172.21.15.116:6833/2998042514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2998042514},{"type":"v1","addr":"172.21.15.116:6833","nonce":2998042514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":1},{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:22:38.936345+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6230":{"gid":6230,"name":"d","rank":0,"incarnation":178,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/492366509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":492366509},{"type":"v1","addr":"172.21.15.174:6837","nonce":492366509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6230,"qdb_cluster":[6230]},"id":2}]} 2024-04-25T00:23:15.693 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 187 2024-04-25T00:23:15.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.694+0000 7f81e4a5e640 1 -- 172.21.15.116:0/997890980 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f81c00679c0 msgr2=0x7f81c0069e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:15.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.694+0000 7f81e4a5e640 1 --2- 172.21.15.116:0/997890980 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f81c00679c0 0x7f81c0069e80 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f81d8005c10 tx=0x7f81d8001630 comp rx=0 tx=0).stop 2024-04-25T00:23:15.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.694+0000 7f81e4a5e640 1 -- 172.21.15.116:0/997890980 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81e0152ea0 msgr2=0x7f81e00a9e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:15.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.694+0000 7f81e4a5e640 1 --2- 172.21.15.116:0/997890980 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81e0152ea0 0x7f81e00a9e80 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f81d001ab30 tx=0x7f81d0019220 comp rx=0 tx=0).stop 2024-04-25T00:23:15.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.694+0000 7f81e4a5e640 1 -- 172.21.15.116:0/997890980 shutdown_connections 2024-04-25T00:23:15.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.694+0000 7f81e4a5e640 1 --2- 172.21.15.116:0/997890980 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f81c00679c0 0x7f81c0069e80 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:15.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.694+0000 7f81e4a5e640 1 --2- 172.21.15.116:0/997890980 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f81e0156220 0x7f81e0148a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:15.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.694+0000 7f81e4a5e640 1 --2- 172.21.15.116:0/997890980 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f81e0152ea0 0x7f81e00a9e80 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:15.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.694+0000 7f81e4a5e640 1 --2- 172.21.15.116:0/997890980 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f81e00c2270 0x7f81e006f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:15.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.694+0000 7f81e4a5e640 1 -- 172.21.15.116:0/997890980 >> 172.21.15.116:0/997890980 conn(0x7f81e006c720 msgr2=0x7f81e006cb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:15.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.694+0000 7f81e4a5e640 1 -- 172.21.15.116:0/997890980 shutdown_connections 2024-04-25T00:23:15.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:15.694+0000 7f81e4a5e640 1 -- 172.21.15.116:0/997890980 wait complete. 2024-04-25T00:23:15.740 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:23:17.741 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:23:17.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.866+0000 7f01d1998640 1 Processor -- start 2024-04-25T00:23:17.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.866+0000 7f01d1998640 1 -- start start 2024-04-25T00:23:17.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.866+0000 7f01d1998640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f01cc1572e0 0x7f01cc1576e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:17.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.866+0000 7f01d1998640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01cc0702a0 con 0x7f01cc0c3460 2024-04-25T00:23:17.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.866+0000 7f01d1998640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01cc06b210 con 0x7f01cc14d7e0 2024-04-25T00:23:17.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.866+0000 7f01d1998640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f01cc0c2c30 con 0x7f01cc1572e0 2024-04-25T00:23:17.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.866+0000 7f01c3fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f01cc0c3460 0x7f01cc0a94a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:33878/0 (socket says 172.21.15.116:33878) 2024-04-25T00:23:17.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.866+0000 7f01c3fff640 1 -- 172.21.15.116:0/3905125366 learned_addr learned my addr 172.21.15.116:0/3905125366 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:23:17.867 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.866+0000 7f01cb7fe640 1 --2- 172.21.15.116:0/3905125366 >> v2:172.21.15.174:3300/0 conn(0x7f01cc1572e0 0x7f01cc1576e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:17.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.866+0000 7f01ca7fc640 1 -- 172.21.15.116:0/3905125366 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2535659813 0 0) 0x7f01cc0702a0 con 0x7f01cc0c3460 2024-04-25T00:23:17.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.866+0000 7f01ca7fc640 1 -- 172.21.15.116:0/3905125366 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01b8003530 con 0x7f01cc0c3460 2024-04-25T00:23:17.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.866+0000 7f01ca7fc640 1 -- 172.21.15.116:0/3905125366 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3116397463 0 0) 0x7f01cc06b210 con 0x7f01cc14d7e0 2024-04-25T00:23:17.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.866+0000 7f01ca7fc640 1 -- 172.21.15.116:0/3905125366 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01cc0702a0 con 0x7f01cc14d7e0 2024-04-25T00:23:17.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.866+0000 7f01ca7fc640 1 -- 172.21.15.116:0/3905125366 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3127612295 0 0) 0x7f01b8003530 con 0x7f01cc0c3460 2024-04-25T00:23:17.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.866+0000 7f01ca7fc640 1 -- 172.21.15.116:0/3905125366 >> v2:172.21.15.174:3300/0 conn(0x7f01cc1572e0 msgr2=0x7f01cc1576e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:17.870 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.866+0000 7f01ca7fc640 1 --2- 172.21.15.116:0/3905125366 >> v2:172.21.15.174:3300/0 conn(0x7f01cc1572e0 0x7f01cc1576e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:17.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.866+0000 7f01ca7fc640 1 -- 172.21.15.116:0/3905125366 >> v1:172.21.15.174:6789/0 conn(0x7f01cc14d7e0 legacy=0x7f01cc1540b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:17.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.866+0000 7f01ca7fc640 1 -- 172.21.15.116:0/3905125366 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01cc06b4b0 con 0x7f01cc0c3460 2024-04-25T00:23:17.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.866+0000 7f01ca7fc640 1 -- 172.21.15.116:0/3905125366 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f01bc001680 con 0x7f01cc0c3460 2024-04-25T00:23:17.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.866+0000 7f01ca7fc640 1 -- 172.21.15.116:0/3905125366 >> v1:172.21.15.116:6789/0 conn(0x7f01cc0c3460 legacy=0x7f01cc0a94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:17.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.866+0000 7f01ca7fc640 1 --2- 172.21.15.116:0/3905125366 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01b8003b50 0x7f01b8003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:17.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.866+0000 7f01ca7fc640 1 --2- 172.21.15.116:0/3905125366 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f01b8004810 0x7f01b8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:17.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.866+0000 7f01ca7fc640 1 --2- 172.21.15.116:0/3905125366 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f01b80051d0 0x7f01b800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:17.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.866+0000 7f01ca7fc640 1 -- 172.21.15.116:0/3905125366 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f01b800c0d0 con 0x7f01b8003b50 2024-04-25T00:23:17.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.866+0000 7f01ca7fc640 1 -- 172.21.15.116:0/3905125366 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f01b800c280 con 0x7f01b80051d0 2024-04-25T00:23:17.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.866+0000 7f01ca7fc640 1 -- 172.21.15.116:0/3905125366 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f01b800c480 con 0x7f01b8004810 2024-04-25T00:23:17.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.866+0000 7f01cb7fe640 1 --2- 172.21.15.116:0/3905125366 >> v2:172.21.15.174:3300/0 conn(0x7f01cc1572e0 0x7f01cc1576e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:23:17.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.866+0000 7f01caffd640 1 --2- 172.21.15.116:0/3905125366 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01b8003b50 0x7f01b8003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:17.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.866+0000 7f01cb7fe640 1 --2- 172.21.15.116:0/3905125366 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f01b80051d0 0x7f01b800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:17.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.866+0000 7f01c3fff640 1 --2- 172.21.15.116:0/3905125366 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f01b8004810 0x7f01b8004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:17.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.866+0000 7f01caffd640 1 -- 172.21.15.116:0/3905125366 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f01b8004810 msgr2=0x7f01b8004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:17.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.866+0000 7f01caffd640 1 --2- 172.21.15.116:0/3905125366 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f01b8004810 0x7f01b8004c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:17.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.866+0000 7f01caffd640 1 -- 172.21.15.116:0/3905125366 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f01b80051d0 msgr2=0x7f01b800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:17.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.866+0000 7f01caffd640 1 --2- 172.21.15.116:0/3905125366 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f01b80051d0 0x7f01b800b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:17.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.866+0000 7f01caffd640 1 -- 172.21.15.116:0/3905125366 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01b8003530 con 0x7f01b8003b50 2024-04-25T00:23:17.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.866+0000 7f01cb7fe640 1 -- 172.21.15.116:0/3905125366 reap_dead start 2024-04-25T00:23:17.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.870+0000 7f01caffd640 1 --2- 172.21.15.116:0/3905125366 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01b8003b50 0x7f01b8003f50 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f01cc06b3e0 tx=0x7f01b401c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=feff0bfc38df8430 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:17.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.870+0000 7f01ca7fc640 1 -- 172.21.15.116:0/3905125366 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01b4001530 con 0x7f01b8003b50 2024-04-25T00:23:17.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.870+0000 7f01ca7fc640 1 -- 172.21.15.116:0/3905125366 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01b40052d0 con 0x7f01b8003b50 2024-04-25T00:23:17.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.870+0000 7f01ca7fc640 1 -- 172.21.15.116:0/3905125366 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01b40054c0 con 0x7f01b8003b50 2024-04-25T00:23:17.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.870+0000 7f01d1998640 1 -- 172.21.15.116:0/3905125366 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01b8003b50 msgr2=0x7f01b8003f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:17.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.870+0000 7f01d1998640 1 --2- 172.21.15.116:0/3905125366 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01b8003b50 0x7f01b8003f50 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f01cc06b3e0 tx=0x7f01b401c500 comp rx=0 tx=0).stop 2024-04-25T00:23:17.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.870+0000 7f01d1998640 1 -- 172.21.15.116:0/3905125366 shutdown_connections 2024-04-25T00:23:17.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.870+0000 7f01d1998640 1 --2- 172.21.15.116:0/3905125366 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01b8003b50 0x7f01b8003f50 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:17.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.870+0000 7f01d1998640 1 -- 172.21.15.116:0/3905125366 >> 172.21.15.116:0/3905125366 conn(0x7f01cc0c7fa0 msgr2=0x7f01cc0c83b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:17.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.870+0000 7f01d1998640 1 -- 172.21.15.116:0/3905125366 shutdown_connections 2024-04-25T00:23:17.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.870+0000 7f01d1998640 1 -- 172.21.15.116:0/3905125366 wait complete. 2024-04-25T00:23:17.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.870+0000 7f01d1998640 1 Processor -- start 2024-04-25T00:23:17.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.870+0000 7f01d1998640 1 -- start start 2024-04-25T00:23:17.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.874+0000 7f01d1998640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f01b8003b50 0x7f01cc0c6e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:17.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.874+0000 7f01d1998640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f01cc155b20 0x7f01cc140860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:17.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.874+0000 7f01d1998640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01cc140da0 0x7f01cc1f0120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:17.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.874+0000 7f01d1998640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f01cc157ad0 con 0x7f01cc140da0 2024-04-25T00:23:17.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.874+0000 7f01d1998640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f01cc0a9cb0 con 0x7f01cc155b20 2024-04-25T00:23:17.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.874+0000 7f01d1998640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f01cc1544c0 con 0x7f01b8003b50 2024-04-25T00:23:17.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.874+0000 7f01cb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01cc140da0 0x7f01cc1f0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:17.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.874+0000 7f01cb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01cc140da0 0x7f01cc1f0120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39582/0 (socket says 172.21.15.116:39582) 2024-04-25T00:23:17.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.874+0000 7f01cb7fe640 1 -- 172.21.15.116:0/785991402 learned_addr learned my addr 172.21.15.116:0/785991402 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:23:17.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.874+0000 7f01c3fff640 1 --2- 172.21.15.116:0/785991402 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f01cc155b20 0x7f01cc140860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:17.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.874+0000 7f01caffd640 1 --2- 172.21.15.116:0/785991402 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f01b8003b50 0x7f01cc0c6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:17.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.874+0000 7f01cb7fe640 1 -- 172.21.15.116:0/785991402 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f01b8003b50 msgr2=0x7f01cc0c6e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:17.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.874+0000 7f01cb7fe640 1 --2- 172.21.15.116:0/785991402 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f01b8003b50 0x7f01cc0c6e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:17.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.874+0000 7f01cb7fe640 1 -- 172.21.15.116:0/785991402 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f01cc155b20 msgr2=0x7f01cc140860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:17.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.874+0000 7f01cb7fe640 1 --2- 172.21.15.116:0/785991402 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f01cc155b20 0x7f01cc140860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:17.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.874+0000 7f01cb7fe640 1 -- 172.21.15.116:0/785991402 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01cc141220 con 0x7f01cc140da0 2024-04-25T00:23:17.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.874+0000 7f01c3fff640 1 --2- 172.21.15.116:0/785991402 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f01cc155b20 0x7f01cc140860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:23:17.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.874+0000 7f01cb7fe640 1 --2- 172.21.15.116:0/785991402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01cc140da0 0x7f01cc1f0120 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f01cc06df70 tx=0x7f01c400f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:17.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.874+0000 7f01c8ff9640 1 -- 172.21.15.116:0/785991402 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01c40039a0 con 0x7f01cc140da0 2024-04-25T00:23:17.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.874+0000 7f01c8ff9640 1 -- 172.21.15.116:0/785991402 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01c4003b40 con 0x7f01cc140da0 2024-04-25T00:23:17.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.874+0000 7f01c8ff9640 1 -- 172.21.15.116:0/785991402 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01c4004020 con 0x7f01cc140da0 2024-04-25T00:23:17.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.874+0000 7f01d1998640 1 -- 172.21.15.116:0/785991402 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01cc1ed7f0 con 0x7f01cc140da0 2024-04-25T00:23:17.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.874+0000 7f01d1998640 1 -- 172.21.15.116:0/785991402 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01cc1ecf50 con 0x7f01cc140da0 2024-04-25T00:23:17.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.874+0000 7f01d1998640 1 -- 172.21.15.116:0/785991402 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01cc068a60 con 0x7f01cc140da0 2024-04-25T00:23:17.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.874+0000 7f01c8ff9640 1 -- 172.21.15.116:0/785991402 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f01c40041c0 con 0x7f01cc140da0 2024-04-25T00:23:17.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.874+0000 7f01c8ff9640 1 --2- 172.21.15.116:0/785991402 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f01ac0678f0 0x7f01ac069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:17.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.874+0000 7f01c8ff9640 1 -- 172.21.15.116:0/785991402 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 9676+0+0 (secure 0 0 0) 0x7f01c40879a0 con 0x7f01cc140da0 2024-04-25T00:23:17.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.874+0000 7f01caffd640 1 --2- 172.21.15.116:0/785991402 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f01ac0678f0 0x7f01ac069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:17.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.878+0000 7f01caffd640 1 --2- 172.21.15.116:0/785991402 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f01ac0678f0 0x7f01ac069db0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f01b40ac000 tx=0x7f01b4001370 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:17.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:17.882+0000 7f01c8ff9640 1 -- 172.21.15.116:0/785991402 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f01c4019030 con 0x7f01cc140da0 2024-04-25T00:23:18.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:18.062+0000 7f01d1998640 1 -- 172.21.15.116:0/785991402 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f01cc0c2680 con 0x7f01cc140da0 2024-04-25T00:23:18.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:18.066+0000 7f01c8ff9640 1 -- 172.21.15.116:0/785991402 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 187 v187) ==== 95+0+6505 (secure 0 0 0) 0x7f01c4010040 con 0x7f01cc140da0 2024-04-25T00:23:18.066 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:23:18.067 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":187,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":8244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1971675962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1971675962},{"type":"v1","addr":"172.21.15.116:6835","nonce":1971675962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":186,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:22:51.924561+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8214},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":183,"state":"up:active","state_seq":4,"addr":"172.21.15.116:6833/2998042514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2998042514},{"type":"v1","addr":"172.21.15.116:6833","nonce":2998042514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":1},{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:22:38.936345+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6230":{"gid":6230,"name":"d","rank":0,"incarnation":178,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/492366509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":492366509},{"type":"v1","addr":"172.21.15.174:6837","nonce":492366509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6230,"qdb_cluster":[6230]},"id":2}]} 2024-04-25T00:23:18.067 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 187 2024-04-25T00:23:18.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:18.066+0000 7f01d1998640 1 -- 172.21.15.116:0/785991402 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f01ac0678f0 msgr2=0x7f01ac069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:18.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:18.066+0000 7f01d1998640 1 --2- 172.21.15.116:0/785991402 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f01ac0678f0 0x7f01ac069db0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f01b40ac000 tx=0x7f01b4001370 comp rx=0 tx=0).stop 2024-04-25T00:23:18.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:18.066+0000 7f01d1998640 1 -- 172.21.15.116:0/785991402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01cc140da0 msgr2=0x7f01cc1f0120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:18.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:18.066+0000 7f01d1998640 1 --2- 172.21.15.116:0/785991402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01cc140da0 0x7f01cc1f0120 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f01cc06df70 tx=0x7f01c400f580 comp rx=0 tx=0).stop 2024-04-25T00:23:18.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:18.066+0000 7f01d1998640 1 -- 172.21.15.116:0/785991402 shutdown_connections 2024-04-25T00:23:18.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:18.066+0000 7f01d1998640 1 --2- 172.21.15.116:0/785991402 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f01ac0678f0 0x7f01ac069db0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:18.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:18.066+0000 7f01d1998640 1 --2- 172.21.15.116:0/785991402 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01cc140da0 0x7f01cc1f0120 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:18.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:18.066+0000 7f01d1998640 1 --2- 172.21.15.116:0/785991402 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f01cc155b20 0x7f01cc140860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:18.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:18.066+0000 7f01d1998640 1 --2- 172.21.15.116:0/785991402 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f01b8003b50 0x7f01cc0c6e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:18.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:18.066+0000 7f01d1998640 1 -- 172.21.15.116:0/785991402 >> 172.21.15.116:0/785991402 conn(0x7f01cc0c7fa0 msgr2=0x7f01cc14c7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:18.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:18.066+0000 7f01d1998640 1 -- 172.21.15.116:0/785991402 shutdown_connections 2024-04-25T00:23:18.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:18.066+0000 7f01d1998640 1 -- 172.21.15.116:0/785991402 wait complete. 2024-04-25T00:23:18.083 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:23:20.084 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:23:20.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.213+0000 7f0552b81640 1 Processor -- start 2024-04-25T00:23:20.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.213+0000 7f0552b81640 1 -- start start 2024-04-25T00:23:20.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.217+0000 7f0552b81640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f054c070dd0 0x7f054c071270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:20.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.217+0000 7f0552b81640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f054c06e890 0x7f054c0717b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:20.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.217+0000 7f0552b81640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f054c152c10 con 0x7f054c148890 2024-04-25T00:23:20.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.217+0000 7f0552b81640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f054c071cf0 con 0x7f054c070dd0 2024-04-25T00:23:20.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.217+0000 7f0552b81640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f054c151e50 con 0x7f054c06e890 2024-04-25T00:23:20.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.217+0000 7f054bfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f054c070dd0 0x7f054c071270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:20.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.217+0000 7f054bfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f054c070dd0 0x7f054c071270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39596/0 (socket says 172.21.15.116:39596) 2024-04-25T00:23:20.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.217+0000 7f054bfff640 1 -- 172.21.15.116:0/2761690416 learned_addr learned my addr 172.21.15.116:0/2761690416 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:23:20.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.217+0000 7f055111e640 1 --2- 172.21.15.116:0/2761690416 >> v2:172.21.15.174:3300/0 conn(0x7f054c06e890 0x7f054c0717b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:20.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.217+0000 7f054bfff640 1 -- 172.21.15.116:0/2761690416 >> v2:172.21.15.174:3300/0 conn(0x7f054c06e890 msgr2=0x7f054c0717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:20.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.217+0000 7f054bfff640 1 --2- 172.21.15.116:0/2761690416 >> v2:172.21.15.174:3300/0 conn(0x7f054c06e890 0x7f054c0717b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.217+0000 7f054bfff640 1 -- 172.21.15.116:0/2761690416 >> v1:172.21.15.116:6789/0 conn(0x7f054c148890 legacy=0x7f054c072590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:20.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.217+0000 7f054bfff640 1 -- 172.21.15.116:0/2761690416 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f054c152130 con 0x7f054c070dd0 2024-04-25T00:23:20.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.217+0000 7f054bfff640 1 --2- 172.21.15.116:0/2761690416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f054c070dd0 0x7f054c071270 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f054c0a94a0 tx=0x7f053c01eb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=481270eec52000f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:20.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.217+0000 7f054b7fe640 1 -- 172.21.15.116:0/2761690416 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f053c0b4070 con 0x7f054c070dd0 2024-04-25T00:23:20.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.217+0000 7f054b7fe640 1 -- 172.21.15.116:0/2761690416 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f053c0a79b0 con 0x7f054c070dd0 2024-04-25T00:23:20.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.217+0000 7f054b7fe640 1 -- 172.21.15.116:0/2761690416 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f053c0a7d80 con 0x7f054c070dd0 2024-04-25T00:23:20.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.217+0000 7f0552b81640 1 -- 172.21.15.116:0/2761690416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f054c070dd0 msgr2=0x7f054c071270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:20.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.217+0000 7f0552b81640 1 --2- 172.21.15.116:0/2761690416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f054c070dd0 0x7f054c071270 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f054c0a94a0 tx=0x7f053c01eb60 comp rx=0 tx=0).stop 2024-04-25T00:23:20.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.217+0000 7f0552b81640 1 -- 172.21.15.116:0/2761690416 shutdown_connections 2024-04-25T00:23:20.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.217+0000 7f0552b81640 1 --2- 172.21.15.116:0/2761690416 >> v2:172.21.15.174:3300/0 conn(0x7f054c06e890 0x7f054c0717b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.217+0000 7f0552b81640 1 --2- 172.21.15.116:0/2761690416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f054c070dd0 0x7f054c071270 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.217+0000 7f0552b81640 1 -- 172.21.15.116:0/2761690416 >> 172.21.15.116:0/2761690416 conn(0x7f054c0c3b00 msgr2=0x7f054c0c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:20.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.217+0000 7f0552b81640 1 -- 172.21.15.116:0/2761690416 shutdown_connections 2024-04-25T00:23:20.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.217+0000 7f0552b81640 1 -- 172.21.15.116:0/2761690416 wait complete. 2024-04-25T00:23:20.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.221+0000 7f0552b81640 1 Processor -- start 2024-04-25T00:23:20.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.221+0000 7f0552b81640 1 -- start start 2024-04-25T00:23:20.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.221+0000 7f0552b81640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f054c06e890 0x7f054c0c0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:20.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.221+0000 7f0552b81640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f054c070dd0 0x7f054c146be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:20.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.221+0000 7f0552b81640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f054c148890 0x7f054c147120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:20.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.221+0000 7f0552b81640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f054c072e20 con 0x7f054c148890 2024-04-25T00:23:20.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.221+0000 7f0552b81640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f054c072ca0 con 0x7f054c070dd0 2024-04-25T00:23:20.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.221+0000 7f0552b81640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f054c0ab590 con 0x7f054c06e890 2024-04-25T00:23:20.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.221+0000 7f055111e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f054c148890 0x7f054c147120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:20.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.221+0000 7f055111e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f054c148890 0x7f054c147120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39606/0 (socket says 172.21.15.116:39606) 2024-04-25T00:23:20.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.221+0000 7f055111e640 1 -- 172.21.15.116:0/3354786203 learned_addr learned my addr 172.21.15.116:0/3354786203 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:23:20.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.221+0000 7f054bfff640 1 --2- 172.21.15.116:0/3354786203 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f054c070dd0 0x7f054c146be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:20.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.221+0000 7f055091d640 1 --2- 172.21.15.116:0/3354786203 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f054c06e890 0x7f054c0c0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:20.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.221+0000 7f055111e640 1 -- 172.21.15.116:0/3354786203 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f054c06e890 msgr2=0x7f054c0c0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:20.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.221+0000 7f055111e640 1 --2- 172.21.15.116:0/3354786203 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f054c06e890 0x7f054c0c0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.221+0000 7f055111e640 1 -- 172.21.15.116:0/3354786203 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f054c070dd0 msgr2=0x7f054c146be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:20.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.221+0000 7f055111e640 1 --2- 172.21.15.116:0/3354786203 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f054c070dd0 0x7f054c146be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.221+0000 7f055111e640 1 -- 172.21.15.116:0/3354786203 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f054c1ee7a0 con 0x7f054c148890 2024-04-25T00:23:20.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.221+0000 7f055111e640 1 --2- 172.21.15.116:0/3354786203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f054c148890 0x7f054c147120 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f0544007e30 tx=0x7f054400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:20.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.221+0000 7f05497fa640 1 -- 172.21.15.116:0/3354786203 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0544014620 con 0x7f054c148890 2024-04-25T00:23:20.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.221+0000 7f05497fa640 1 -- 172.21.15.116:0/3354786203 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0544019070 con 0x7f054c148890 2024-04-25T00:23:20.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.221+0000 7f0552b81640 1 -- 172.21.15.116:0/3354786203 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f054c0c4d70 con 0x7f054c148890 2024-04-25T00:23:20.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.221+0000 7f05497fa640 1 -- 172.21.15.116:0/3354786203 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0544014a00 con 0x7f054c148890 2024-04-25T00:23:20.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.221+0000 7f0552b81640 1 -- 172.21.15.116:0/3354786203 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f054c1eed00 con 0x7f054c148890 2024-04-25T00:23:20.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.221+0000 7f0552b81640 1 -- 172.21.15.116:0/3354786203 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f054c068350 con 0x7f054c148890 2024-04-25T00:23:20.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.221+0000 7f05497fa640 1 -- 172.21.15.116:0/3354786203 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f0544014ba0 con 0x7f054c148890 2024-04-25T00:23:20.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.225+0000 7f05497fa640 1 --2- 172.21.15.116:0/3354786203 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f05380678f0 0x7f0538069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:20.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.225+0000 7f05497fa640 1 -- 172.21.15.116:0/3354786203 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 9676+0+0 (secure 0 0 0) 0x7f0544087450 con 0x7f054c148890 2024-04-25T00:23:20.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.225+0000 7f055091d640 1 --2- 172.21.15.116:0/3354786203 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f05380678f0 0x7f0538069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:20.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.225+0000 7f055091d640 1 --2- 172.21.15.116:0/3354786203 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f05380678f0 0x7f0538069db0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f054c142a80 tx=0x7f0540009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:20.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.229+0000 7f05497fa640 1 -- 172.21.15.116:0/3354786203 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f054c1ee7a0 con 0x7f054c148890 2024-04-25T00:23:20.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.421+0000 7f0552b81640 1 -- 172.21.15.116:0/3354786203 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f054c06e670 con 0x7f054c148890 2024-04-25T00:23:20.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.421+0000 7f05497fa640 1 -- 172.21.15.116:0/3354786203 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 187 v187) ==== 95+0+6505 (secure 0 0 0) 0x7f0544057050 con 0x7f054c148890 2024-04-25T00:23:20.423 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:23:20.423 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":187,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":8244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1971675962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1971675962},{"type":"v1","addr":"172.21.15.116:6835","nonce":1971675962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":186,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:22:51.924561+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8214},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":183,"state":"up:active","state_seq":4,"addr":"172.21.15.116:6833/2998042514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2998042514},{"type":"v1","addr":"172.21.15.116:6833","nonce":2998042514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":1},{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:22:38.936345+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6230":{"gid":6230,"name":"d","rank":0,"incarnation":178,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/492366509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":492366509},{"type":"v1","addr":"172.21.15.174:6837","nonce":492366509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6230,"qdb_cluster":[6230]},"id":2}]} 2024-04-25T00:23:20.424 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 187 2024-04-25T00:23:20.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.425+0000 7f0552b81640 1 -- 172.21.15.116:0/3354786203 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f05380678f0 msgr2=0x7f0538069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:20.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.425+0000 7f0552b81640 1 --2- 172.21.15.116:0/3354786203 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f05380678f0 0x7f0538069db0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f054c142a80 tx=0x7f0540009040 comp rx=0 tx=0).stop 2024-04-25T00:23:20.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.425+0000 7f0552b81640 1 -- 172.21.15.116:0/3354786203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f054c148890 msgr2=0x7f054c147120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:20.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.425+0000 7f0552b81640 1 --2- 172.21.15.116:0/3354786203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f054c148890 0x7f054c147120 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f0544007e30 tx=0x7f054400e4b0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.425+0000 7f0552b81640 1 -- 172.21.15.116:0/3354786203 shutdown_connections 2024-04-25T00:23:20.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.425+0000 7f0552b81640 1 --2- 172.21.15.116:0/3354786203 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f05380678f0 0x7f0538069db0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.425+0000 7f0552b81640 1 --2- 172.21.15.116:0/3354786203 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f054c148890 0x7f054c147120 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.425+0000 7f0552b81640 1 --2- 172.21.15.116:0/3354786203 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f054c070dd0 0x7f054c146be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.425+0000 7f0552b81640 1 --2- 172.21.15.116:0/3354786203 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f054c06e890 0x7f054c0c0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:20.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.425+0000 7f0552b81640 1 -- 172.21.15.116:0/3354786203 >> 172.21.15.116:0/3354786203 conn(0x7f054c0c3b00 msgr2=0x7f054c06d4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:20.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.425+0000 7f0552b81640 1 -- 172.21.15.116:0/3354786203 shutdown_connections 2024-04-25T00:23:20.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:20.425+0000 7f0552b81640 1 -- 172.21.15.116:0/3354786203 wait complete. 2024-04-25T00:23:20.442 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:23:22.443 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:23:22.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.569+0000 7f4880d63640 1 Processor -- start 2024-04-25T00:23:22.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.569+0000 7f4880d63640 1 -- start start 2024-04-25T00:23:22.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.573+0000 7f4880d63640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f487c0a94a0 0x7f487c1563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:22.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.573+0000 7f4880d63640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f487c071680 0x7f487c071ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:22.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.573+0000 7f4880d63640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f487c071220 con 0x7f487c14c720 2024-04-25T00:23:22.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.573+0000 7f4880d63640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f487c153460 con 0x7f487c0a94a0 2024-04-25T00:23:22.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.573+0000 7f4880d63640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f487c152420 con 0x7f487c071680 2024-04-25T00:23:22.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.573+0000 7f4871d9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f487c0a94a0 0x7f487c1563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:22.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.573+0000 7f487a59c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f487c14c720 0x7f487c0c2a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:33894/0 (socket says 172.21.15.116:33894) 2024-04-25T00:23:22.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.573+0000 7f4871d9b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f487c0a94a0 0x7f487c1563d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39618/0 (socket says 172.21.15.116:39618) 2024-04-25T00:23:22.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.573+0000 7f487a59c640 1 -- 172.21.15.116:0/517770383 learned_addr learned my addr 172.21.15.116:0/517770383 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:23:22.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.573+0000 7f487ad9d640 1 --2- 172.21.15.116:0/517770383 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f487c071680 0x7f487c071ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:22.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.573+0000 7f4871d9b640 1 -- 172.21.15.116:0/517770383 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f487c071680 msgr2=0x7f487c071ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:22.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.573+0000 7f4871d9b640 1 --2- 172.21.15.116:0/517770383 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f487c071680 0x7f487c071ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:22.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.573+0000 7f4871d9b640 1 -- 172.21.15.116:0/517770383 >> v1:172.21.15.116:6789/0 conn(0x7f487c14c720 legacy=0x7f487c0c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:22.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.573+0000 7f4871d9b640 1 -- 172.21.15.116:0/517770383 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f487c14cb00 con 0x7f487c0a94a0 2024-04-25T00:23:22.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.573+0000 7f487ad9d640 1 --2- 172.21.15.116:0/517770383 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f487c071680 0x7f487c071ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:23:22.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.573+0000 7f4871d9b640 1 --2- 172.21.15.116:0/517770383 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f487c0a94a0 0x7f487c1563d0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f48640097c0 tx=0x7f486401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b592f0caadb4431f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:22.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.573+0000 7f4879d9b640 1 -- 172.21.15.116:0/517770383 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48640b1070 con 0x7f487c0a94a0 2024-04-25T00:23:22.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.573+0000 7f4879d9b640 1 -- 172.21.15.116:0/517770383 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48640a8050 con 0x7f487c0a94a0 2024-04-25T00:23:22.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.573+0000 7f4879d9b640 1 -- 172.21.15.116:0/517770383 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48640ac860 con 0x7f487c0a94a0 2024-04-25T00:23:22.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.577+0000 7f4880d63640 1 -- 172.21.15.116:0/517770383 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f487c0a94a0 msgr2=0x7f487c1563d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:22.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.577+0000 7f4880d63640 1 --2- 172.21.15.116:0/517770383 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f487c0a94a0 0x7f487c1563d0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f48640097c0 tx=0x7f486401c3e0 comp rx=0 tx=0).stop 2024-04-25T00:23:22.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.577+0000 7f4880d63640 1 -- 172.21.15.116:0/517770383 shutdown_connections 2024-04-25T00:23:22.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.577+0000 7f4880d63640 1 --2- 172.21.15.116:0/517770383 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f487c071680 0x7f487c071ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:22.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.577+0000 7f4880d63640 1 --2- 172.21.15.116:0/517770383 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f487c0a94a0 0x7f487c1563d0 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:22.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.577+0000 7f4880d63640 1 -- 172.21.15.116:0/517770383 >> 172.21.15.116:0/517770383 conn(0x7f487c0c3b00 msgr2=0x7f487c0c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:22.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.577+0000 7f4880d63640 1 -- 172.21.15.116:0/517770383 shutdown_connections 2024-04-25T00:23:22.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.577+0000 7f4880d63640 1 -- 172.21.15.116:0/517770383 wait complete. 2024-04-25T00:23:22.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.577+0000 7f4880d63640 1 Processor -- start 2024-04-25T00:23:22.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.577+0000 7f4880d63640 1 -- start start 2024-04-25T00:23:22.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.577+0000 7f4880d63640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f487c071680 0x7f487c1f1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:22.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.577+0000 7f4880d63640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f487c0a94a0 0x7f487c0c0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:22.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.577+0000 7f4880d63640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f487c14c720 0x7f487c154d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:22.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.577+0000 7f4880d63640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f487c0c3120 con 0x7f487c0a94a0 2024-04-25T00:23:22.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.577+0000 7f4880d63640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f487c152f40 con 0x7f487c14c720 2024-04-25T00:23:22.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.577+0000 7f4880d63640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f487c0c32a0 con 0x7f487c071680 2024-04-25T00:23:22.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.577+0000 7f4871d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f487c0a94a0 0x7f487c0c0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:22.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.577+0000 7f4871d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f487c0a94a0 0x7f487c0c0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39634/0 (socket says 172.21.15.116:39634) 2024-04-25T00:23:22.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.577+0000 7f4871d9b640 1 -- 172.21.15.116:0/530195615 learned_addr learned my addr 172.21.15.116:0/530195615 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:23:22.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.577+0000 7f487a59c640 1 --2- 172.21.15.116:0/530195615 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f487c071680 0x7f487c1f1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:22.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.577+0000 7f487ad9d640 1 --2- 172.21.15.116:0/530195615 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f487c14c720 0x7f487c154d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:22.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.577+0000 7f4871d9b640 1 -- 172.21.15.116:0/530195615 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f487c071680 msgr2=0x7f487c1f1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:22.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.577+0000 7f4871d9b640 1 --2- 172.21.15.116:0/530195615 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f487c071680 0x7f487c1f1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:22.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.577+0000 7f4871d9b640 1 -- 172.21.15.116:0/530195615 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f487c14c720 msgr2=0x7f487c154d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:22.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.577+0000 7f4871d9b640 1 --2- 172.21.15.116:0/530195615 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f487c14c720 0x7f487c154d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:22.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.577+0000 7f4871d9b640 1 -- 172.21.15.116:0/530195615 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f487c1edf10 con 0x7f487c0a94a0 2024-04-25T00:23:22.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.577+0000 7f4871d9b640 1 --2- 172.21.15.116:0/530195615 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f487c0a94a0 0x7f487c0c0690 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f48640af040 tx=0x7f4864019180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:22.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.577+0000 7f48737fe640 1 -- 172.21.15.116:0/530195615 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48640b1040 con 0x7f487c0a94a0 2024-04-25T00:23:22.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.577+0000 7f48737fe640 1 -- 172.21.15.116:0/530195615 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48640bb070 con 0x7f487c0a94a0 2024-04-25T00:23:22.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.577+0000 7f4880d63640 1 -- 172.21.15.116:0/530195615 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f487c1ee1f0 con 0x7f487c0a94a0 2024-04-25T00:23:22.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.577+0000 7f48737fe640 1 -- 172.21.15.116:0/530195615 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48640b5650 con 0x7f487c0a94a0 2024-04-25T00:23:22.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.577+0000 7f4880d63640 1 -- 172.21.15.116:0/530195615 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f487c1ee7b0 con 0x7f487c0a94a0 2024-04-25T00:23:22.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.581+0000 7f4880d63640 1 -- 172.21.15.116:0/530195615 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f487c068350 con 0x7f487c0a94a0 2024-04-25T00:23:22.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.581+0000 7f48737fe640 1 -- 172.21.15.116:0/530195615 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f48640b57f0 con 0x7f487c0a94a0 2024-04-25T00:23:22.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.581+0000 7f48737fe640 1 --2- 172.21.15.116:0/530195615 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f485c0678c0 0x7f485c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:22.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.581+0000 7f48737fe640 1 -- 172.21.15.116:0/530195615 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 9676+0+0 (secure 0 0 0) 0x7f4864120c50 con 0x7f487c0a94a0 2024-04-25T00:23:22.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.581+0000 7f487a59c640 1 --2- 172.21.15.116:0/530195615 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f485c0678c0 0x7f485c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:22.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.581+0000 7f487a59c640 1 --2- 172.21.15.116:0/530195615 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f485c0678c0 0x7f485c069d80 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f486c000f70 tx=0x7f486c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:22.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.585+0000 7f48737fe640 1 -- 172.21.15.116:0/530195615 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f4864126020 con 0x7f487c0a94a0 2024-04-25T00:23:22.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.785+0000 7f4880d63640 1 -- 172.21.15.116:0/530195615 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f487c0c3640 con 0x7f487c0a94a0 2024-04-25T00:23:22.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.785+0000 7f48737fe640 1 -- 172.21.15.116:0/530195615 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 187 v187) ==== 95+0+6505 (secure 0 0 0) 0x7f4864016090 con 0x7f487c0a94a0 2024-04-25T00:23:22.786 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:23:22.787 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":187,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":8244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1971675962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1971675962},{"type":"v1","addr":"172.21.15.116:6835","nonce":1971675962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":186,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:22:51.924561+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8214},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":183,"state":"up:active","state_seq":4,"addr":"172.21.15.116:6833/2998042514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2998042514},{"type":"v1","addr":"172.21.15.116:6833","nonce":2998042514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":1},{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:22:38.936345+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6230":{"gid":6230,"name":"d","rank":0,"incarnation":178,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/492366509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":492366509},{"type":"v1","addr":"172.21.15.174:6837","nonce":492366509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6230,"qdb_cluster":[6230]},"id":2}]} 2024-04-25T00:23:22.787 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 187 2024-04-25T00:23:22.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.785+0000 7f4880d63640 1 -- 172.21.15.116:0/530195615 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f485c0678c0 msgr2=0x7f485c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:22.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.785+0000 7f4880d63640 1 --2- 172.21.15.116:0/530195615 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f485c0678c0 0x7f485c069d80 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f486c000f70 tx=0x7f486c009040 comp rx=0 tx=0).stop 2024-04-25T00:23:22.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.785+0000 7f4880d63640 1 -- 172.21.15.116:0/530195615 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f487c0a94a0 msgr2=0x7f487c0c0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:22.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.785+0000 7f4880d63640 1 --2- 172.21.15.116:0/530195615 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f487c0a94a0 0x7f487c0c0690 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f48640af040 tx=0x7f4864019180 comp rx=0 tx=0).stop 2024-04-25T00:23:22.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.785+0000 7f4880d63640 1 -- 172.21.15.116:0/530195615 shutdown_connections 2024-04-25T00:23:22.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.785+0000 7f4880d63640 1 --2- 172.21.15.116:0/530195615 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f485c0678c0 0x7f485c069d80 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:22.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.785+0000 7f4880d63640 1 --2- 172.21.15.116:0/530195615 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f487c14c720 0x7f487c154d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:22.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.785+0000 7f4880d63640 1 --2- 172.21.15.116:0/530195615 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f487c0a94a0 0x7f487c0c0690 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:22.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.785+0000 7f4880d63640 1 --2- 172.21.15.116:0/530195615 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f487c071680 0x7f487c1f1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:22.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.785+0000 7f4880d63640 1 -- 172.21.15.116:0/530195615 >> 172.21.15.116:0/530195615 conn(0x7f487c0c3b00 msgr2=0x7f487c072bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:22.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.785+0000 7f4880d63640 1 -- 172.21.15.116:0/530195615 shutdown_connections 2024-04-25T00:23:22.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:22.785+0000 7f4880d63640 1 -- 172.21.15.116:0/530195615 wait complete. 2024-04-25T00:23:22.807 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:23:24.809 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:23:24.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:24.937+0000 7f531b695640 1 Processor -- start 2024-04-25T00:23:24.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:24.937+0000 7f531b695640 1 -- start start 2024-04-25T00:23:24.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:24.941+0000 7f531b695640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f53140a94a0 0x7f53141563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:24.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:24.941+0000 7f531b695640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5314071680 0x7f5314071ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:24.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:24.941+0000 7f531b695640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5314071220 con 0x7f531414c720 2024-04-25T00:23:24.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:24.941+0000 7f531b695640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f5314153460 con 0x7f5314071680 2024-04-25T00:23:24.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:24.941+0000 7f531b695640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f5314152420 con 0x7f53140a94a0 2024-04-25T00:23:24.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:24.941+0000 7f5319431640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f531414c720 0x7f53140c2a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:46248/0 (socket says 172.21.15.116:46248) 2024-04-25T00:23:24.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:24.941+0000 7f5319431640 1 -- 172.21.15.116:0/1596411142 learned_addr learned my addr 172.21.15.116:0/1596411142 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:23:24.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:24.941+0000 7f5319c32640 1 --2- 172.21.15.116:0/1596411142 >> v2:172.21.15.116:3300/0 conn(0x7f5314071680 0x7f5314071ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:24.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:24.941+0000 7f5318c30640 1 --2- 172.21.15.116:0/1596411142 >> v2:172.21.15.174:3300/0 conn(0x7f53140a94a0 0x7f53141563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:24.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:24.941+0000 7f5319c32640 1 -- 172.21.15.116:0/1596411142 >> v2:172.21.15.174:3300/0 conn(0x7f53140a94a0 msgr2=0x7f53141563d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:24.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:24.941+0000 7f5319c32640 1 --2- 172.21.15.116:0/1596411142 >> v2:172.21.15.174:3300/0 conn(0x7f53140a94a0 0x7f53141563d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:24.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:24.941+0000 7f5319c32640 1 -- 172.21.15.116:0/1596411142 >> v1:172.21.15.116:6789/0 conn(0x7f531414c720 legacy=0x7f53140c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:24.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:24.941+0000 7f5319c32640 1 -- 172.21.15.116:0/1596411142 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f531414cb00 con 0x7f5314071680 2024-04-25T00:23:24.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:24.941+0000 7f5319c32640 1 --2- 172.21.15.116:0/1596411142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5314071680 0x7f5314071ab0 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f53140a9930 tx=0x7f530c01c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c37501b9dfb7c519 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:24.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:24.941+0000 7f530bfff640 1 -- 172.21.15.116:0/1596411142 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f530c0b2070 con 0x7f5314071680 2024-04-25T00:23:24.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:24.941+0000 7f530bfff640 1 -- 172.21.15.116:0/1596411142 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f530c0a5930 con 0x7f5314071680 2024-04-25T00:23:24.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:24.941+0000 7f530bfff640 1 -- 172.21.15.116:0/1596411142 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f530c0a5d00 con 0x7f5314071680 2024-04-25T00:23:24.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:24.945+0000 7f531b695640 1 -- 172.21.15.116:0/1596411142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5314071680 msgr2=0x7f5314071ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:24.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:24.945+0000 7f531b695640 1 --2- 172.21.15.116:0/1596411142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5314071680 0x7f5314071ab0 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f53140a9930 tx=0x7f530c01c870 comp rx=0 tx=0).stop 2024-04-25T00:23:24.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:24.945+0000 7f531b695640 1 -- 172.21.15.116:0/1596411142 shutdown_connections 2024-04-25T00:23:24.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:24.945+0000 7f531b695640 1 --2- 172.21.15.116:0/1596411142 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5314071680 0x7f5314071ab0 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:24.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:24.945+0000 7f531b695640 1 --2- 172.21.15.116:0/1596411142 >> v2:172.21.15.174:3300/0 conn(0x7f53140a94a0 0x7f53141563d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:24.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:24.945+0000 7f531b695640 1 -- 172.21.15.116:0/1596411142 >> 172.21.15.116:0/1596411142 conn(0x7f53140c3b00 msgr2=0x7f53140c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:24.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:24.945+0000 7f531b695640 1 -- 172.21.15.116:0/1596411142 shutdown_connections 2024-04-25T00:23:24.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:24.945+0000 7f531b695640 1 -- 172.21.15.116:0/1596411142 wait complete. 2024-04-25T00:23:24.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:24.945+0000 7f531b695640 1 Processor -- start 2024-04-25T00:23:24.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:24.945+0000 7f531b695640 1 -- start start 2024-04-25T00:23:24.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:24.945+0000 7f531b695640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5314071680 0x7f53140c5b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:24.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:24.945+0000 7f531b695640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f53140a94a0 0x7f531414a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:24.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:24.945+0000 7f531b695640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f531414c720 0x7f531414aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:24.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:24.945+0000 7f531b695640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f53140c3120 con 0x7f531414c720 2024-04-25T00:23:24.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:24.945+0000 7f531b695640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5314152f40 con 0x7f5314071680 2024-04-25T00:23:24.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:24.945+0000 7f531b695640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f53140c32a0 con 0x7f53140a94a0 2024-04-25T00:23:24.949 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:24.945+0000 7f5319c32640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f531414c720 0x7f531414aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:24.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:24.945+0000 7f5319c32640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f531414c720 0x7f531414aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57526/0 (socket says 172.21.15.116:57526) 2024-04-25T00:23:24.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:24.945+0000 7f5319c32640 1 -- 172.21.15.116:0/1504833636 learned_addr learned my addr 172.21.15.116:0/1504833636 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:23:24.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:24.945+0000 7f5318c30640 1 --2- 172.21.15.116:0/1504833636 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f53140a94a0 0x7f531414a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:24.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:24.945+0000 7f5319431640 1 --2- 172.21.15.116:0/1504833636 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5314071680 0x7f53140c5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:24.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:24.945+0000 7f5319c32640 1 -- 172.21.15.116:0/1504833636 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f53140a94a0 msgr2=0x7f531414a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:24.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:24.945+0000 7f5319c32640 1 --2- 172.21.15.116:0/1504833636 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f53140a94a0 0x7f531414a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:24.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:24.945+0000 7f5319c32640 1 -- 172.21.15.116:0/1504833636 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5314071680 msgr2=0x7f53140c5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:24.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:24.945+0000 7f5319c32640 1 --2- 172.21.15.116:0/1504833636 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5314071680 0x7f53140c5b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:24.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:24.945+0000 7f5319c32640 1 -- 172.21.15.116:0/1504833636 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5314151e50 con 0x7f531414c720 2024-04-25T00:23:24.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:24.945+0000 7f5319c32640 1 --2- 172.21.15.116:0/1504833636 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f531414c720 0x7f531414aaf0 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f530c019480 tx=0x7f530c019610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:24.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:24.945+0000 7f5309ffb640 1 -- 172.21.15.116:0/1504833636 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f530c0b2050 con 0x7f531414c720 2024-04-25T00:23:24.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:24.945+0000 7f5309ffb640 1 -- 172.21.15.116:0/1504833636 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f530c0be070 con 0x7f531414c720 2024-04-25T00:23:24.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:24.945+0000 7f531b695640 1 -- 172.21.15.116:0/1504833636 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53140ab840 con 0x7f531414c720 2024-04-25T00:23:24.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:24.945+0000 7f5309ffb640 1 -- 172.21.15.116:0/1504833636 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f530c0ada70 con 0x7f531414c720 2024-04-25T00:23:24.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:24.945+0000 7f531b695640 1 -- 172.21.15.116:0/1504833636 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53140abe00 con 0x7f531414c720 2024-04-25T00:23:24.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:24.949+0000 7f531b695640 1 -- 172.21.15.116:0/1504833636 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5314068350 con 0x7f531414c720 2024-04-25T00:23:24.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:24.949+0000 7f5309ffb640 1 -- 172.21.15.116:0/1504833636 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f530c0adc10 con 0x7f531414c720 2024-04-25T00:23:24.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:24.949+0000 7f5309ffb640 1 --2- 172.21.15.116:0/1504833636 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f52f80678c0 0x7f52f8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:24.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:24.949+0000 7f5309ffb640 1 -- 172.21.15.116:0/1504833636 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 9676+0+0 (secure 0 0 0) 0x7f530c122980 con 0x7f531414c720 2024-04-25T00:23:24.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:24.949+0000 7f5319431640 1 --2- 172.21.15.116:0/1504833636 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f52f80678c0 0x7f52f8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:24.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:24.949+0000 7f5319431640 1 --2- 172.21.15.116:0/1504833636 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f52f80678c0 0x7f52f8069d80 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f53140c1470 tx=0x7f5310009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:24.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:24.953+0000 7f5309ffb640 1 -- 172.21.15.116:0/1504833636 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f530c016520 con 0x7f531414c720 2024-04-25T00:23:25.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:25.153+0000 7f531b695640 1 -- 172.21.15.116:0/1504833636 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f53140c3640 con 0x7f531414c720 2024-04-25T00:23:25.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:25.153+0000 7f5309ffb640 1 -- 172.21.15.116:0/1504833636 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 187 v187) ==== 95+0+6505 (secure 0 0 0) 0x7f530c0f5980 con 0x7f531414c720 2024-04-25T00:23:25.155 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:23:25.155 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":187,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":8244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1971675962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1971675962},{"type":"v1","addr":"172.21.15.116:6835","nonce":1971675962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":186,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:22:51.924561+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8214},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":183,"state":"up:active","state_seq":4,"addr":"172.21.15.116:6833/2998042514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2998042514},{"type":"v1","addr":"172.21.15.116:6833","nonce":2998042514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":1},{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:22:38.936345+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6230":{"gid":6230,"name":"d","rank":0,"incarnation":178,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/492366509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":492366509},{"type":"v1","addr":"172.21.15.174:6837","nonce":492366509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6230,"qdb_cluster":[6230]},"id":2}]} 2024-04-25T00:23:25.155 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 187 2024-04-25T00:23:25.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:25.157+0000 7f531b695640 1 -- 172.21.15.116:0/1504833636 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f52f80678c0 msgr2=0x7f52f8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:25.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:25.157+0000 7f531b695640 1 --2- 172.21.15.116:0/1504833636 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f52f80678c0 0x7f52f8069d80 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f53140c1470 tx=0x7f5310009040 comp rx=0 tx=0).stop 2024-04-25T00:23:25.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:25.157+0000 7f531b695640 1 -- 172.21.15.116:0/1504833636 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f531414c720 msgr2=0x7f531414aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:25.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:25.157+0000 7f531b695640 1 --2- 172.21.15.116:0/1504833636 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f531414c720 0x7f531414aaf0 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f530c019480 tx=0x7f530c019610 comp rx=0 tx=0).stop 2024-04-25T00:23:25.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:25.157+0000 7f531b695640 1 -- 172.21.15.116:0/1504833636 shutdown_connections 2024-04-25T00:23:25.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:25.157+0000 7f531b695640 1 --2- 172.21.15.116:0/1504833636 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f52f80678c0 0x7f52f8069d80 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:25.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:25.157+0000 7f531b695640 1 --2- 172.21.15.116:0/1504833636 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f531414c720 0x7f531414aaf0 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:25.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:25.157+0000 7f531b695640 1 --2- 172.21.15.116:0/1504833636 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f53140a94a0 0x7f531414a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:25.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:25.157+0000 7f531b695640 1 --2- 172.21.15.116:0/1504833636 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5314071680 0x7f53140c5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:25.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:25.157+0000 7f531b695640 1 -- 172.21.15.116:0/1504833636 >> 172.21.15.116:0/1504833636 conn(0x7f53140c3b00 msgr2=0x7f5314072b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:25.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:25.157+0000 7f531b695640 1 -- 172.21.15.116:0/1504833636 shutdown_connections 2024-04-25T00:23:25.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:25.157+0000 7f531b695640 1 -- 172.21.15.116:0/1504833636 wait complete. 2024-04-25T00:23:25.175 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:23:27.176 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:23:27.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.305+0000 7f7ade59f640 1 Processor -- start 2024-04-25T00:23:27.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.305+0000 7f7ade59f640 1 -- start start 2024-04-25T00:23:27.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.305+0000 7f7ade59f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7ad8150b70 0x7f7ad80a94a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:27.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.305+0000 7f7ade59f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7ad806b210 0x7f7ad806b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:27.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.305+0000 7f7ade59f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ad80a99e0 con 0x7f7ad8072e80 2024-04-25T00:23:27.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.305+0000 7f7ade59f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f7ad81540a0 con 0x7f7ad806b210 2024-04-25T00:23:27.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.305+0000 7f7ade59f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f7ad8154380 con 0x7f7ad8150b70 2024-04-25T00:23:27.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.305+0000 7f7adcb3c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7ad806b210 0x7f7ad806b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:27.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.305+0000 7f7adcb3c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7ad806b210 0x7f7ad806b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57532/0 (socket says 172.21.15.116:57532) 2024-04-25T00:23:27.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.305+0000 7f7adcb3c640 1 -- 172.21.15.116:0/151707262 learned_addr learned my addr 172.21.15.116:0/151707262 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:23:27.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.305+0000 7f7ad7fff640 1 --2- 172.21.15.116:0/151707262 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7ad8150b70 0x7f7ad80a94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:27.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.305+0000 7f7adcb3c640 1 -- 172.21.15.116:0/151707262 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7ad8150b70 msgr2=0x7f7ad80a94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:27.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.305+0000 7f7adcb3c640 1 --2- 172.21.15.116:0/151707262 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7ad8150b70 0x7f7ad80a94a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:27.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.305+0000 7f7adcb3c640 1 -- 172.21.15.116:0/151707262 >> v1:172.21.15.116:6789/0 conn(0x7f7ad8072e80 legacy=0x7f7ad80c3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:27.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.305+0000 7f7adcb3c640 1 -- 172.21.15.116:0/151707262 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ad80c2c90 con 0x7f7ad806b210 2024-04-25T00:23:27.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.309+0000 7f7adcb3c640 1 --2- 172.21.15.116:0/151707262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ad806b210 0x7f7ad806b660 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f7ad0009e50 tx=0x7f7ad001c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=877a2f632f93b220 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:27.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.309+0000 7f7ad6ffd640 1 -- 172.21.15.116:0/151707262 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ad00a5770 con 0x7f7ad806b210 2024-04-25T00:23:27.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.309+0000 7f7ad6ffd640 1 -- 172.21.15.116:0/151707262 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ad00a5910 con 0x7f7ad806b210 2024-04-25T00:23:27.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.309+0000 7f7ad6ffd640 1 -- 172.21.15.116:0/151707262 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ad00a5ce0 con 0x7f7ad806b210 2024-04-25T00:23:27.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.309+0000 7f7ade59f640 1 -- 172.21.15.116:0/151707262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ad806b210 msgr2=0x7f7ad806b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:27.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.309+0000 7f7ade59f640 1 --2- 172.21.15.116:0/151707262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ad806b210 0x7f7ad806b660 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f7ad0009e50 tx=0x7f7ad001c9a0 comp rx=0 tx=0).stop 2024-04-25T00:23:27.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.309+0000 7f7ade59f640 1 -- 172.21.15.116:0/151707262 shutdown_connections 2024-04-25T00:23:27.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.309+0000 7f7ade59f640 1 --2- 172.21.15.116:0/151707262 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ad806b210 0x7f7ad806b660 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:27.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.309+0000 7f7ade59f640 1 --2- 172.21.15.116:0/151707262 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7ad8150b70 0x7f7ad80a94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:27.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.309+0000 7f7ade59f640 1 -- 172.21.15.116:0/151707262 >> 172.21.15.116:0/151707262 conn(0x7f7ad80c4c10 msgr2=0x7f7ad80c5020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:27.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.309+0000 7f7ade59f640 1 -- 172.21.15.116:0/151707262 shutdown_connections 2024-04-25T00:23:27.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.309+0000 7f7ade59f640 1 -- 172.21.15.116:0/151707262 wait complete. 2024-04-25T00:23:27.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.309+0000 7f7ade59f640 1 Processor -- start 2024-04-25T00:23:27.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.309+0000 7f7ade59f640 1 -- start start 2024-04-25T00:23:27.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.313+0000 7f7ade59f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ad806b210 0x7f7ad81ec760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:27.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.313+0000 7f7ade59f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7ad8072e80 0x7f7ad80a9e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:27.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.313+0000 7f7ade59f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7ad8150b70 0x7f7ad81515e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:27.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.313+0000 7f7ade59f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7ad8160880 con 0x7f7ad806b210 2024-04-25T00:23:27.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.313+0000 7f7ade59f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7ad8160700 con 0x7f7ad8072e80 2024-04-25T00:23:27.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.313+0000 7f7ade59f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f7ad8160a00 con 0x7f7ad8150b70 2024-04-25T00:23:27.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.313+0000 7f7ad7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ad806b210 0x7f7ad81ec760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:27.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.313+0000 7f7ad7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ad806b210 0x7f7ad81ec760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57534/0 (socket says 172.21.15.116:57534) 2024-04-25T00:23:27.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.313+0000 7f7ad7fff640 1 -- 172.21.15.116:0/3827650003 learned_addr learned my addr 172.21.15.116:0/3827650003 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:23:27.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.313+0000 7f7adcb3c640 1 --2- 172.21.15.116:0/3827650003 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7ad8150b70 0x7f7ad81515e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:27.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.313+0000 7f7ad77fe640 1 --2- 172.21.15.116:0/3827650003 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7ad8072e80 0x7f7ad80a9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:27.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.313+0000 7f7ad7fff640 1 -- 172.21.15.116:0/3827650003 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7ad8150b70 msgr2=0x7f7ad81515e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:27.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.313+0000 7f7ad7fff640 1 --2- 172.21.15.116:0/3827650003 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7ad8150b70 0x7f7ad81515e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:27.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.313+0000 7f7ad7fff640 1 -- 172.21.15.116:0/3827650003 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7ad8072e80 msgr2=0x7f7ad80a9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:27.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.313+0000 7f7ad7fff640 1 --2- 172.21.15.116:0/3827650003 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7ad8072e80 0x7f7ad80a9e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:27.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.313+0000 7f7ad7fff640 1 -- 172.21.15.116:0/3827650003 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ad8152a40 con 0x7f7ad806b210 2024-04-25T00:23:27.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.313+0000 7f7ad77fe640 1 --2- 172.21.15.116:0/3827650003 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7ad8072e80 0x7f7ad80a9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:23:27.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.313+0000 7f7ad7fff640 1 --2- 172.21.15.116:0/3827650003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ad806b210 0x7f7ad81ec760 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f7ac800d620 tx=0x7f7ac800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:27.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.313+0000 7f7ad4ff9640 1 -- 172.21.15.116:0/3827650003 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ac800b620 con 0x7f7ad806b210 2024-04-25T00:23:27.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.313+0000 7f7ad4ff9640 1 -- 172.21.15.116:0/3827650003 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ac8019070 con 0x7f7ad806b210 2024-04-25T00:23:27.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.313+0000 7f7ade59f640 1 -- 172.21.15.116:0/3827650003 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ad8152d20 con 0x7f7ad806b210 2024-04-25T00:23:27.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.313+0000 7f7ad4ff9640 1 -- 172.21.15.116:0/3827650003 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ac800b9e0 con 0x7f7ad806b210 2024-04-25T00:23:27.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.313+0000 7f7ade59f640 1 -- 172.21.15.116:0/3827650003 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ad81532e0 con 0x7f7ad806b210 2024-04-25T00:23:27.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.313+0000 7f7ade59f640 1 -- 172.21.15.116:0/3827650003 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ad8068350 con 0x7f7ad806b210 2024-04-25T00:23:27.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.313+0000 7f7ad4ff9640 1 -- 172.21.15.116:0/3827650003 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7ac800bb80 con 0x7f7ad806b210 2024-04-25T00:23:27.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.313+0000 7f7ad4ff9640 1 --2- 172.21.15.116:0/3827650003 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7ac40678c0 0x7f7ac4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:27.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.313+0000 7f7ad4ff9640 1 -- 172.21.15.116:0/3827650003 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 9676+0+0 (secure 0 0 0) 0x7f7ac8086ff0 con 0x7f7ad806b210 2024-04-25T00:23:27.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.313+0000 7f7ad77fe640 1 --2- 172.21.15.116:0/3827650003 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7ac40678c0 0x7f7ac4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:27.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.317+0000 7f7ad77fe640 1 --2- 172.21.15.116:0/3827650003 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7ac40678c0 0x7f7ac4069d80 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f7ad814dc20 tx=0x7f7acc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:27.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.317+0000 7f7ad4ff9640 1 -- 172.21.15.116:0/3827650003 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f7ac8056bf0 con 0x7f7ad806b210 2024-04-25T00:23:27.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.513+0000 7f7ade59f640 1 -- 172.21.15.116:0/3827650003 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7ad8151200 con 0x7f7ad806b210 2024-04-25T00:23:27.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.517+0000 7f7ad4ff9640 1 -- 172.21.15.116:0/3827650003 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 187 v187) ==== 95+0+6505 (secure 0 0 0) 0x7f7ac80564f0 con 0x7f7ad806b210 2024-04-25T00:23:27.518 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:23:27.518 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":187,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":8244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1971675962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1971675962},{"type":"v1","addr":"172.21.15.116:6835","nonce":1971675962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":186,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:22:51.924561+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8214},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":183,"state":"up:active","state_seq":4,"addr":"172.21.15.116:6833/2998042514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2998042514},{"type":"v1","addr":"172.21.15.116:6833","nonce":2998042514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":1},{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:22:38.936345+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6230":{"gid":6230,"name":"d","rank":0,"incarnation":178,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/492366509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":492366509},{"type":"v1","addr":"172.21.15.174:6837","nonce":492366509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6230,"qdb_cluster":[6230]},"id":2}]} 2024-04-25T00:23:27.518 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 187 2024-04-25T00:23:27.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.517+0000 7f7ade59f640 1 -- 172.21.15.116:0/3827650003 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7ac40678c0 msgr2=0x7f7ac4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:27.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.517+0000 7f7ade59f640 1 --2- 172.21.15.116:0/3827650003 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7ac40678c0 0x7f7ac4069d80 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f7ad814dc20 tx=0x7f7acc009040 comp rx=0 tx=0).stop 2024-04-25T00:23:27.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.517+0000 7f7ade59f640 1 -- 172.21.15.116:0/3827650003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ad806b210 msgr2=0x7f7ad81ec760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:27.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.517+0000 7f7ade59f640 1 --2- 172.21.15.116:0/3827650003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ad806b210 0x7f7ad81ec760 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f7ac800d620 tx=0x7f7ac800dae0 comp rx=0 tx=0).stop 2024-04-25T00:23:27.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.517+0000 7f7ade59f640 1 -- 172.21.15.116:0/3827650003 shutdown_connections 2024-04-25T00:23:27.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.517+0000 7f7ade59f640 1 --2- 172.21.15.116:0/3827650003 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7ac40678c0 0x7f7ac4069d80 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:27.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.517+0000 7f7ade59f640 1 --2- 172.21.15.116:0/3827650003 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7ad8150b70 0x7f7ad81515e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:27.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.517+0000 7f7ade59f640 1 --2- 172.21.15.116:0/3827650003 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7ad8072e80 0x7f7ad80a9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:27.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.517+0000 7f7ade59f640 1 --2- 172.21.15.116:0/3827650003 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ad806b210 0x7f7ad81ec760 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:27.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.517+0000 7f7ade59f640 1 -- 172.21.15.116:0/3827650003 >> 172.21.15.116:0/3827650003 conn(0x7f7ad80c4c10 msgr2=0x7f7ad814f8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:27.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.517+0000 7f7ade59f640 1 -- 172.21.15.116:0/3827650003 shutdown_connections 2024-04-25T00:23:27.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:27.517+0000 7f7ade59f640 1 -- 172.21.15.116:0/3827650003 wait complete. 2024-04-25T00:23:27.536 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:23:29.536 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:23:29.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.665+0000 7f2e4d274640 1 Processor -- start 2024-04-25T00:23:29.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.665+0000 7f2e4d274640 1 -- start start 2024-04-25T00:23:29.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.665+0000 7f2e4d274640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2e480a94a0 0x7f2e481538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:29.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.665+0000 7f2e4d274640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f2e480a99f0 0x7f2e48153e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:29.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.665+0000 7f2e4d274640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e480c3910 con 0x7f2e48150d20 2024-04-25T00:23:29.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.665+0000 7f2e4d274640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f2e480c16e0 con 0x7f2e480a94a0 2024-04-25T00:23:29.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.665+0000 7f2e4d274640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f2e480c1400 con 0x7f2e480a99f0 2024-04-25T00:23:29.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.665+0000 7f2e3ffff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2e480a94a0 0x7f2e481538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:29.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.665+0000 7f2e3ffff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2e480a94a0 0x7f2e481538e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57548/0 (socket says 172.21.15.116:57548) 2024-04-25T00:23:29.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.665+0000 7f2e3ffff640 1 -- 172.21.15.116:0/1847063235 learned_addr learned my addr 172.21.15.116:0/1847063235 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:23:29.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.665+0000 7f2e477fe640 1 --2- 172.21.15.116:0/1847063235 >> v2:172.21.15.174:3300/0 conn(0x7f2e480a99f0 0x7f2e48153e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:29.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.665+0000 7f2e3ffff640 1 -- 172.21.15.116:0/1847063235 >> v2:172.21.15.174:3300/0 conn(0x7f2e480a99f0 msgr2=0x7f2e48153e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:29.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.665+0000 7f2e3ffff640 1 --2- 172.21.15.116:0/1847063235 >> v2:172.21.15.174:3300/0 conn(0x7f2e480a99f0 0x7f2e48153e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:29.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.665+0000 7f2e3ffff640 1 -- 172.21.15.116:0/1847063235 >> v1:172.21.15.174:6789/0 conn(0x7f2e48150d20 legacy=0x7f2e480c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:29.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.665+0000 7f2e3ffff640 1 -- 172.21.15.116:0/1847063235 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e4806e660 con 0x7f2e480a94a0 2024-04-25T00:23:29.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.669+0000 7f2e3ffff640 1 --2- 172.21.15.116:0/1847063235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e480a94a0 0x7f2e481538e0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f2e48157b40 tx=0x7f2e3001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48c085328bd0a952 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:29.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.669+0000 7f2e467fc640 1 -- 172.21.15.116:0/1847063235 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e300ab430 con 0x7f2e480a94a0 2024-04-25T00:23:29.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.669+0000 7f2e467fc640 1 -- 172.21.15.116:0/1847063235 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e300b0070 con 0x7f2e480a94a0 2024-04-25T00:23:29.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.669+0000 7f2e467fc640 1 -- 172.21.15.116:0/1847063235 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e300ab7f0 con 0x7f2e480a94a0 2024-04-25T00:23:29.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.669+0000 7f2e4d274640 1 -- 172.21.15.116:0/1847063235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e480a94a0 msgr2=0x7f2e481538e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:29.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.669+0000 7f2e4d274640 1 --2- 172.21.15.116:0/1847063235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e480a94a0 0x7f2e481538e0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f2e48157b40 tx=0x7f2e3001c520 comp rx=0 tx=0).stop 2024-04-25T00:23:29.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.669+0000 7f2e4d274640 1 -- 172.21.15.116:0/1847063235 shutdown_connections 2024-04-25T00:23:29.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.669+0000 7f2e4d274640 1 --2- 172.21.15.116:0/1847063235 >> v2:172.21.15.174:3300/0 conn(0x7f2e480a99f0 0x7f2e48153e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:29.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.669+0000 7f2e4d274640 1 --2- 172.21.15.116:0/1847063235 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e480a94a0 0x7f2e481538e0 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:29.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.669+0000 7f2e4d274640 1 -- 172.21.15.116:0/1847063235 >> 172.21.15.116:0/1847063235 conn(0x7f2e480c3b00 msgr2=0x7f2e480c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:29.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.669+0000 7f2e4d274640 1 -- 172.21.15.116:0/1847063235 shutdown_connections 2024-04-25T00:23:29.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.669+0000 7f2e4d274640 1 -- 172.21.15.116:0/1847063235 wait complete. 2024-04-25T00:23:29.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.669+0000 7f2e4d274640 1 Processor -- start 2024-04-25T00:23:29.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.669+0000 7f2e4d274640 1 -- start start 2024-04-25T00:23:29.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.669+0000 7f2e4d274640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e480a94a0 0x7f2e48147020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:29.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.669+0000 7f2e4d274640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e480a99f0 0x7f2e480c5190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:29.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.669+0000 7f2e4d274640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e48150d20 0x7f2e480c56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:29.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.669+0000 7f2e4d274640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2e481544e0 con 0x7f2e480a94a0 2024-04-25T00:23:29.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.669+0000 7f2e4d274640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2e48154360 con 0x7f2e480a99f0 2024-04-25T00:23:29.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.669+0000 7f2e4d274640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f2e48160730 con 0x7f2e48150d20 2024-04-25T00:23:29.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.669+0000 7f2e46ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e480a94a0 0x7f2e48147020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:29.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.669+0000 7f2e46ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e480a94a0 0x7f2e48147020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57556/0 (socket says 172.21.15.116:57556) 2024-04-25T00:23:29.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.669+0000 7f2e46ffd640 1 -- 172.21.15.116:0/2091447620 learned_addr learned my addr 172.21.15.116:0/2091447620 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:23:29.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.669+0000 7f2e3ffff640 1 --2- 172.21.15.116:0/2091447620 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e480a99f0 0x7f2e480c5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:29.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.669+0000 7f2e46ffd640 1 -- 172.21.15.116:0/2091447620 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e48150d20 msgr2=0x7f2e480c56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:29.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.669+0000 7f2e46ffd640 1 --2- 172.21.15.116:0/2091447620 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e48150d20 0x7f2e480c56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:29.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.669+0000 7f2e46ffd640 1 -- 172.21.15.116:0/2091447620 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e480a99f0 msgr2=0x7f2e480c5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:29.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.669+0000 7f2e46ffd640 1 --2- 172.21.15.116:0/2091447620 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e480a99f0 0x7f2e480c5190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:29.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.669+0000 7f2e46ffd640 1 -- 172.21.15.116:0/2091447620 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e48145810 con 0x7f2e480a94a0 2024-04-25T00:23:29.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.673+0000 7f2e46ffd640 1 --2- 172.21.15.116:0/2091447620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e480a94a0 0x7f2e48147020 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f2e3800c450 tx=0x7f2e3800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:29.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.673+0000 7f2e3f7fe640 1 -- 172.21.15.116:0/2091447620 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e38016020 con 0x7f2e480a94a0 2024-04-25T00:23:29.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.673+0000 7f2e3f7fe640 1 -- 172.21.15.116:0/2091447620 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e380052d0 con 0x7f2e480a94a0 2024-04-25T00:23:29.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.673+0000 7f2e4d274640 1 -- 172.21.15.116:0/2091447620 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e480c69e0 con 0x7f2e480a94a0 2024-04-25T00:23:29.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.673+0000 7f2e3f7fe640 1 -- 172.21.15.116:0/2091447620 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e38007260 con 0x7f2e480a94a0 2024-04-25T00:23:29.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.673+0000 7f2e4d274640 1 -- 172.21.15.116:0/2091447620 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e4813b330 con 0x7f2e480a94a0 2024-04-25T00:23:29.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.673+0000 7f2e4d274640 1 -- 172.21.15.116:0/2091447620 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e48068350 con 0x7f2e480a94a0 2024-04-25T00:23:29.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.673+0000 7f2e3f7fe640 1 -- 172.21.15.116:0/2091447620 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2e38007400 con 0x7f2e480a94a0 2024-04-25T00:23:29.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.673+0000 7f2e3f7fe640 1 --2- 172.21.15.116:0/2091447620 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2e280678c0 0x7f2e28069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:29.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.673+0000 7f2e3f7fe640 1 -- 172.21.15.116:0/2091447620 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 9676+0+0 (secure 0 0 0) 0x7f2e380865a0 con 0x7f2e480a94a0 2024-04-25T00:23:29.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.673+0000 7f2e3ffff640 1 --2- 172.21.15.116:0/2091447620 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2e280678c0 0x7f2e28069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:29.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.673+0000 7f2e3ffff640 1 --2- 172.21.15.116:0/2091447620 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2e280678c0 0x7f2e28069d80 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f2e300af000 tx=0x7f2e300a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:29.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.677+0000 7f2e3f7fe640 1 -- 172.21.15.116:0/2091447620 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f2e38020020 con 0x7f2e480a94a0 2024-04-25T00:23:29.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.873+0000 7f2e4d274640 1 -- 172.21.15.116:0/2091447620 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2e480c34f0 con 0x7f2e480a94a0 2024-04-25T00:23:29.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.873+0000 7f2e3f7fe640 1 -- 172.21.15.116:0/2091447620 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 187 v187) ==== 95+0+6505 (secure 0 0 0) 0x7f2e38005480 con 0x7f2e480a94a0 2024-04-25T00:23:29.874 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:23:29.874 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":187,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":8244,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/1971675962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1971675962},{"type":"v1","addr":"172.21.15.116:6835","nonce":1971675962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":187}],"filesystems":[{"mdsmap":{"epoch":186,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:22:51.924561+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8214},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8214":{"gid":8214,"name":"a","rank":0,"incarnation":183,"state":"up:active","state_seq":4,"addr":"172.21.15.116:6833/2998042514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2998042514},{"type":"v1","addr":"172.21.15.116:6833","nonce":2998042514}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8214,"qdb_cluster":[8214]},"id":1},{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:22:38.936345+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6230":{"gid":6230,"name":"d","rank":0,"incarnation":178,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/492366509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":492366509},{"type":"v1","addr":"172.21.15.174:6837","nonce":492366509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6230,"qdb_cluster":[6230]},"id":2}]} 2024-04-25T00:23:29.875 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 187 2024-04-25T00:23:29.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.873+0000 7f2e4d274640 1 -- 172.21.15.116:0/2091447620 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2e280678c0 msgr2=0x7f2e28069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:29.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.873+0000 7f2e4d274640 1 --2- 172.21.15.116:0/2091447620 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2e280678c0 0x7f2e28069d80 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f2e300af000 tx=0x7f2e300a7000 comp rx=0 tx=0).stop 2024-04-25T00:23:29.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.873+0000 7f2e4d274640 1 -- 172.21.15.116:0/2091447620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e480a94a0 msgr2=0x7f2e48147020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:29.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.873+0000 7f2e4d274640 1 --2- 172.21.15.116:0/2091447620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e480a94a0 0x7f2e48147020 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f2e3800c450 tx=0x7f2e3800c910 comp rx=0 tx=0).stop 2024-04-25T00:23:29.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.873+0000 7f2e4d274640 1 -- 172.21.15.116:0/2091447620 shutdown_connections 2024-04-25T00:23:29.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.873+0000 7f2e4d274640 1 --2- 172.21.15.116:0/2091447620 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2e280678c0 0x7f2e28069d80 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:29.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.873+0000 7f2e4d274640 1 --2- 172.21.15.116:0/2091447620 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2e48150d20 0x7f2e480c56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:29.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.873+0000 7f2e4d274640 1 --2- 172.21.15.116:0/2091447620 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2e480a99f0 0x7f2e480c5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:29.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.873+0000 7f2e4d274640 1 --2- 172.21.15.116:0/2091447620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2e480a94a0 0x7f2e48147020 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:29.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.873+0000 7f2e4d274640 1 -- 172.21.15.116:0/2091447620 >> 172.21.15.116:0/2091447620 conn(0x7f2e480c3b00 msgr2=0x7f2e480ac0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:29.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.873+0000 7f2e4d274640 1 -- 172.21.15.116:0/2091447620 shutdown_connections 2024-04-25T00:23:29.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:29.873+0000 7f2e4d274640 1 -- 172.21.15.116:0/2091447620 wait complete. 2024-04-25T00:23:29.891 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:23:31.892 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:23:32.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.021+0000 7fd0b27a8640 1 Processor -- start 2024-04-25T00:23:32.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.021+0000 7fd0b27a8640 1 -- start start 2024-04-25T00:23:32.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.025+0000 7fd0b27a8640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd0ac152d80 0x7fd0ac06bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:32.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.025+0000 7fd0b27a8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd0ac06d5f0 0x7fd0ac156c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:32.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.025+0000 7fd0b27a8640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0ac0c9150 con 0x7fd0ac06c290 2024-04-25T00:23:32.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.025+0000 7fd0b27a8640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fd0ac0c9d60 con 0x7fd0ac06d5f0 2024-04-25T00:23:32.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.025+0000 7fd0b27a8640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fd0ac0c9430 con 0x7fd0ac152d80 2024-04-25T00:23:32.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.025+0000 7fd0ab7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd0ac06d5f0 0x7fd0ac156c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:32.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.025+0000 7fd0ab7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd0ac06d5f0 0x7fd0ac156c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57572/0 (socket says 172.21.15.116:57572) 2024-04-25T00:23:32.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.025+0000 7fd0ab7fe640 1 -- 172.21.15.116:0/2992248065 learned_addr learned my addr 172.21.15.116:0/2992248065 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:23:32.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.025+0000 7fd0abfff640 1 --2- 172.21.15.116:0/2992248065 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd0ac152d80 0x7fd0ac06bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:32.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.025+0000 7fd0ab7fe640 1 -- 172.21.15.116:0/2992248065 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd0ac152d80 msgr2=0x7fd0ac06bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:32.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.025+0000 7fd0ab7fe640 1 --2- 172.21.15.116:0/2992248065 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd0ac152d80 0x7fd0ac06bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:32.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.025+0000 7fd0ab7fe640 1 -- 172.21.15.116:0/2992248065 >> v1:172.21.15.116:6789/0 conn(0x7fd0ac06c290 legacy=0x7fd0ac1571c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:32.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.025+0000 7fd0ab7fe640 1 -- 172.21.15.116:0/2992248065 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0ac0c9710 con 0x7fd0ac06d5f0 2024-04-25T00:23:32.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.025+0000 7fd0ab7fe640 1 --2- 172.21.15.116:0/2992248065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0ac06d5f0 0x7fd0ac156c80 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7fd094009870 tx=0x7fd09401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc9c9ccb29854a94 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:32.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.025+0000 7fd0aaffd640 1 -- 172.21.15.116:0/2992248065 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0940aa620 con 0x7fd0ac06d5f0 2024-04-25T00:23:32.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.025+0000 7fd0aaffd640 1 -- 172.21.15.116:0/2992248065 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0940af070 con 0x7fd0ac06d5f0 2024-04-25T00:23:32.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.025+0000 7fd0aaffd640 1 -- 172.21.15.116:0/2992248065 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0940aa9e0 con 0x7fd0ac06d5f0 2024-04-25T00:23:32.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.025+0000 7fd0b27a8640 1 -- 172.21.15.116:0/2992248065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0ac06d5f0 msgr2=0x7fd0ac156c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:32.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.025+0000 7fd0b27a8640 1 --2- 172.21.15.116:0/2992248065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0ac06d5f0 0x7fd0ac156c80 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7fd094009870 tx=0x7fd09401c4a0 comp rx=0 tx=0).stop 2024-04-25T00:23:32.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.025+0000 7fd0b27a8640 1 -- 172.21.15.116:0/2992248065 shutdown_connections 2024-04-25T00:23:32.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.025+0000 7fd0b27a8640 1 --2- 172.21.15.116:0/2992248065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0ac06d5f0 0x7fd0ac156c80 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:32.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.025+0000 7fd0b27a8640 1 --2- 172.21.15.116:0/2992248065 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd0ac152d80 0x7fd0ac06bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:32.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.025+0000 7fd0b27a8640 1 -- 172.21.15.116:0/2992248065 >> 172.21.15.116:0/2992248065 conn(0x7fd0ac0acdd0 msgr2=0x7fd0ac0ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:32.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.029+0000 7fd0b27a8640 1 -- 172.21.15.116:0/2992248065 shutdown_connections 2024-04-25T00:23:32.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.029+0000 7fd0b27a8640 1 -- 172.21.15.116:0/2992248065 wait complete. 2024-04-25T00:23:32.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.029+0000 7fd0b27a8640 1 Processor -- start 2024-04-25T00:23:32.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.029+0000 7fd0b27a8640 1 -- start start 2024-04-25T00:23:32.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.029+0000 7fd0b27a8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0ac06c290 0x7fd0ac150900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:32.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.029+0000 7fd0b27a8640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd0ac06d5f0 0x7fd0ac147290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:32.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.029+0000 7fd0b27a8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0ac152d80 0x7fd0ac1efaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:32.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.029+0000 7fd0b27a8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd0ac0caf00 con 0x7fd0ac152d80 2024-04-25T00:23:32.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.029+0000 7fd0b27a8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd0ac0cad80 con 0x7fd0ac06c290 2024-04-25T00:23:32.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.029+0000 7fd0b27a8640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fd0ac1628d0 con 0x7fd0ac06d5f0 2024-04-25T00:23:32.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.029+0000 7fd0b0d45640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0ac152d80 0x7fd0ac1efaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:32.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.029+0000 7fd0b0d45640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0ac152d80 0x7fd0ac1efaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57576/0 (socket says 172.21.15.116:57576) 2024-04-25T00:23:32.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.029+0000 7fd0b0d45640 1 -- 172.21.15.116:0/4096775812 learned_addr learned my addr 172.21.15.116:0/4096775812 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:23:32.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.029+0000 7fd0abfff640 1 --2- 172.21.15.116:0/4096775812 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0ac06c290 0x7fd0ac150900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:32.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.029+0000 7fd0ab7fe640 1 --2- 172.21.15.116:0/4096775812 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd0ac06d5f0 0x7fd0ac147290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:32.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.029+0000 7fd0b0d45640 1 -- 172.21.15.116:0/4096775812 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd0ac06d5f0 msgr2=0x7fd0ac147290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:32.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.029+0000 7fd0b0d45640 1 --2- 172.21.15.116:0/4096775812 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd0ac06d5f0 0x7fd0ac147290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:32.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.029+0000 7fd0b0d45640 1 -- 172.21.15.116:0/4096775812 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0ac06c290 msgr2=0x7fd0ac150900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:32.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.029+0000 7fd0b0d45640 1 --2- 172.21.15.116:0/4096775812 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0ac06c290 0x7fd0ac150900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:32.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.029+0000 7fd0b0d45640 1 -- 172.21.15.116:0/4096775812 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0ac0ac660 con 0x7fd0ac152d80 2024-04-25T00:23:32.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.029+0000 7fd0abfff640 1 --2- 172.21.15.116:0/4096775812 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0ac06c290 0x7fd0ac150900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:23:32.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.029+0000 7fd0b0d45640 1 --2- 172.21.15.116:0/4096775812 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0ac152d80 0x7fd0ac1efaf0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7fd0a400c340 tx=0x7fd0a400c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:32.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.029+0000 7fd0a8ff9640 1 -- 172.21.15.116:0/4096775812 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a4005220 con 0x7fd0ac152d80 2024-04-25T00:23:32.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.029+0000 7fd0a8ff9640 1 -- 172.21.15.116:0/4096775812 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0a40053c0 con 0x7fd0ac152d80 2024-04-25T00:23:32.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.029+0000 7fd0b27a8640 1 -- 172.21.15.116:0/4096775812 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0ac0ab520 con 0x7fd0ac152d80 2024-04-25T00:23:32.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.029+0000 7fd0a8ff9640 1 -- 172.21.15.116:0/4096775812 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0a4007270 con 0x7fd0ac152d80 2024-04-25T00:23:32.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.029+0000 7fd0b27a8640 1 -- 172.21.15.116:0/4096775812 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0ac1477d0 con 0x7fd0ac152d80 2024-04-25T00:23:32.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.029+0000 7fd0b27a8640 1 -- 172.21.15.116:0/4096775812 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0ac068350 con 0x7fd0ac152d80 2024-04-25T00:23:32.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.033+0000 7fd0a8ff9640 1 -- 172.21.15.116:0/4096775812 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd0a4019020 con 0x7fd0ac152d80 2024-04-25T00:23:32.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.033+0000 7fd0a8ff9640 1 --2- 172.21.15.116:0/4096775812 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd08c0678c0 0x7fd08c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:32.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.033+0000 7fd0a8ff9640 1 -- 172.21.15.116:0/4096775812 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 9762+0+0 (secure 0 0 0) 0x7fd0a4085fd0 con 0x7fd0ac152d80 2024-04-25T00:23:32.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.033+0000 7fd0abfff640 1 --2- 172.21.15.116:0/4096775812 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd08c0678c0 0x7fd08c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:32.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.033+0000 7fd0abfff640 1 --2- 172.21.15.116:0/4096775812 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd08c0678c0 0x7fd08c069d80 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fd09c003fd0 tx=0x7fd09c003a00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:32.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.037+0000 7fd0a8ff9640 1 -- 172.21.15.116:0/4096775812 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fd0a4055ae0 con 0x7fd0ac152d80 2024-04-25T00:23:32.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.229+0000 7fd0b27a8640 1 -- 172.21.15.116:0/4096775812 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd0ac0cab60 con 0x7fd0ac152d80 2024-04-25T00:23:32.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.229+0000 7fd0a8ff9640 1 -- 172.21.15.116:0/4096775812 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 189 v189) ==== 95+0+5672 (secure 0 0 0) 0x7fd0a40553e0 con 0x7fd0ac152d80 2024-04-25T00:23:32.233 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:23:32.233 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":189,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":189,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:23:31.107103+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8244":{"gid":8244,"name":"c","rank":0,"incarnation":188,"state":"up:reconnect","state_seq":9,"addr":"172.21.15.116:6835/1971675962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1971675962},{"type":"v1","addr":"172.21.15.116:6835","nonce":1971675962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:22:38.936345+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6230":{"gid":6230,"name":"d","rank":0,"incarnation":178,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/492366509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":492366509},{"type":"v1","addr":"172.21.15.174:6837","nonce":492366509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6230,"qdb_cluster":[6230]},"id":2}]} 2024-04-25T00:23:32.233 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 189 2024-04-25T00:23:32.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.233+0000 7fd0b27a8640 1 -- 172.21.15.116:0/4096775812 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd08c0678c0 msgr2=0x7fd08c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:32.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.233+0000 7fd0b27a8640 1 --2- 172.21.15.116:0/4096775812 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd08c0678c0 0x7fd08c069d80 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fd09c003fd0 tx=0x7fd09c003a00 comp rx=0 tx=0).stop 2024-04-25T00:23:32.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.233+0000 7fd0b27a8640 1 -- 172.21.15.116:0/4096775812 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0ac152d80 msgr2=0x7fd0ac1efaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:32.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.233+0000 7fd0b27a8640 1 --2- 172.21.15.116:0/4096775812 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0ac152d80 0x7fd0ac1efaf0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7fd0a400c340 tx=0x7fd0a400c800 comp rx=0 tx=0).stop 2024-04-25T00:23:32.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.233+0000 7fd0b27a8640 1 -- 172.21.15.116:0/4096775812 shutdown_connections 2024-04-25T00:23:32.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.233+0000 7fd0b27a8640 1 --2- 172.21.15.116:0/4096775812 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd08c0678c0 0x7fd08c069d80 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:32.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.233+0000 7fd0b27a8640 1 --2- 172.21.15.116:0/4096775812 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0ac152d80 0x7fd0ac1efaf0 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:32.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.233+0000 7fd0b27a8640 1 --2- 172.21.15.116:0/4096775812 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd0ac06d5f0 0x7fd0ac147290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:32.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.233+0000 7fd0b27a8640 1 --2- 172.21.15.116:0/4096775812 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0ac06c290 0x7fd0ac150900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:32.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.233+0000 7fd0b27a8640 1 -- 172.21.15.116:0/4096775812 >> 172.21.15.116:0/4096775812 conn(0x7fd0ac0acdd0 msgr2=0x7fd0ac15f590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:32.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.233+0000 7fd0b27a8640 1 -- 172.21.15.116:0/4096775812 shutdown_connections 2024-04-25T00:23:32.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.233+0000 7fd0b27a8640 1 -- 172.21.15.116:0/4096775812 wait complete. 2024-04-25T00:23:32.252 INFO:tasks.mds_thrash.fs.[a]:mds.a down, removed from mdsmap 2024-04-25T00:23:32.252 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-04-25T00:23:32.253 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:23:32.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.373+0000 7f9386156640 1 Processor -- start 2024-04-25T00:23:32.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.373+0000 7f9386156640 1 -- start start 2024-04-25T00:23:32.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.373+0000 7f9386156640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f93800c0b30 0x7f9380073490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:32.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.373+0000 7f9386156640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9380158e70 0x7f93800c3780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:32.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.373+0000 7f9386156640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f938014fe40 con 0x7f938014fa60 2024-04-25T00:23:32.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.373+0000 7f9386156640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f9380155c10 con 0x7f93800c0b30 2024-04-25T00:23:32.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.373+0000 7f9386156640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f93801558c0 con 0x7f9380158e70 2024-04-25T00:23:32.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.373+0000 7f9377fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f93800c0b30 0x7f9380073490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:32.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.373+0000 7f9377fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f93800c0b30 0x7f9380073490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57592/0 (socket says 172.21.15.116:57592) 2024-04-25T00:23:32.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.373+0000 7f9377fff640 1 -- 172.21.15.116:0/146991636 learned_addr learned my addr 172.21.15.116:0/146991636 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:23:32.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.373+0000 7f937ffff640 1 --2- 172.21.15.116:0/146991636 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9380158e70 0x7f93800c3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:32.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.377+0000 7f9377fff640 1 -- 172.21.15.116:0/146991636 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9380158e70 msgr2=0x7f93800c3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:32.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.377+0000 7f9377fff640 1 --2- 172.21.15.116:0/146991636 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9380158e70 0x7f93800c3780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:32.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.377+0000 7f9377fff640 1 -- 172.21.15.116:0/146991636 >> v1:172.21.15.116:6789/0 conn(0x7f938014fa60 legacy=0x7f93801560a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:32.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.377+0000 7f9377fff640 1 -- 172.21.15.116:0/146991636 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93800c1040 con 0x7f93800c0b30 2024-04-25T00:23:32.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.377+0000 7f9377fff640 1 --2- 172.21.15.116:0/146991636 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93800c0b30 0x7f9380073490 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f9368009870 tx=0x7f936801c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7f007d33f66eba0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:32.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.377+0000 7f937effd640 1 -- 172.21.15.116:0/146991636 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93680ab610 con 0x7f93800c0b30 2024-04-25T00:23:32.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.377+0000 7f937effd640 1 -- 172.21.15.116:0/146991636 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93680b0070 con 0x7f93800c0b30 2024-04-25T00:23:32.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.377+0000 7f937effd640 1 -- 172.21.15.116:0/146991636 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93680ab9d0 con 0x7f93800c0b30 2024-04-25T00:23:32.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.377+0000 7f9386156640 1 -- 172.21.15.116:0/146991636 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93800c0b30 msgr2=0x7f9380073490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:32.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.377+0000 7f9386156640 1 --2- 172.21.15.116:0/146991636 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93800c0b30 0x7f9380073490 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f9368009870 tx=0x7f936801c410 comp rx=0 tx=0).stop 2024-04-25T00:23:32.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.377+0000 7f9386156640 1 -- 172.21.15.116:0/146991636 shutdown_connections 2024-04-25T00:23:32.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.377+0000 7f9386156640 1 --2- 172.21.15.116:0/146991636 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9380158e70 0x7f93800c3780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:32.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.377+0000 7f9386156640 1 --2- 172.21.15.116:0/146991636 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93800c0b30 0x7f9380073490 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:32.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.377+0000 7f9386156640 1 -- 172.21.15.116:0/146991636 >> 172.21.15.116:0/146991636 conn(0x7f93800c4200 msgr2=0x7f93800c4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:32.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.377+0000 7f9386156640 1 -- 172.21.15.116:0/146991636 shutdown_connections 2024-04-25T00:23:32.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.377+0000 7f9386156640 1 -- 172.21.15.116:0/146991636 wait complete. 2024-04-25T00:23:32.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.381+0000 7f9386156640 1 Processor -- start 2024-04-25T00:23:32.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.381+0000 7f9386156640 1 -- start start 2024-04-25T00:23:32.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.381+0000 7f9386156640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93800c0b30 0x7f93801ef050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:32.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.381+0000 7f9386156640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f938014fa60 0x7f93800acf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:32.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.381+0000 7f9386156640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9380158e70 0x7f93800ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:32.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.381+0000 7f9386156640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f938013ee10 con 0x7f93800c0b30 2024-04-25T00:23:32.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.381+0000 7f9386156640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f93800c1b90 con 0x7f938014fa60 2024-04-25T00:23:32.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.381+0000 7f9386156640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f938013ef90 con 0x7f9380158e70 2024-04-25T00:23:32.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.381+0000 7f937f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93800c0b30 0x7f93801ef050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:32.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.381+0000 7f937f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93800c0b30 0x7f93801ef050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57600/0 (socket says 172.21.15.116:57600) 2024-04-25T00:23:32.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.381+0000 7f937f7fe640 1 -- 172.21.15.116:0/2795267642 learned_addr learned my addr 172.21.15.116:0/2795267642 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:23:32.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.381+0000 7f937ffff640 1 --2- 172.21.15.116:0/2795267642 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9380158e70 0x7f93800ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:32.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.381+0000 7f9377fff640 1 --2- 172.21.15.116:0/2795267642 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f938014fa60 0x7f93800acf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:32.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.381+0000 7f937f7fe640 1 -- 172.21.15.116:0/2795267642 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9380158e70 msgr2=0x7f93800ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:32.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.381+0000 7f937f7fe640 1 --2- 172.21.15.116:0/2795267642 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9380158e70 0x7f93800ac0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:32.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.381+0000 7f937f7fe640 1 -- 172.21.15.116:0/2795267642 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f938014fa60 msgr2=0x7f93800acf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:32.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.381+0000 7f937f7fe640 1 --2- 172.21.15.116:0/2795267642 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f938014fa60 0x7f93800acf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:32.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.381+0000 7f937f7fe640 1 -- 172.21.15.116:0/2795267642 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f938014ed20 con 0x7f93800c0b30 2024-04-25T00:23:32.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.381+0000 7f9377fff640 1 --2- 172.21.15.116:0/2795267642 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f938014fa60 0x7f93800acf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:23:32.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.381+0000 7f937f7fe640 1 --2- 172.21.15.116:0/2795267642 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93800c0b30 0x7f93801ef050 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f937000c370 tx=0x7f937000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:32.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.381+0000 7f937cff9640 1 -- 172.21.15.116:0/2795267642 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93700052d0 con 0x7f93800c0b30 2024-04-25T00:23:32.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.381+0000 7f937cff9640 1 -- 172.21.15.116:0/2795267642 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9370005470 con 0x7f93800c0b30 2024-04-25T00:23:32.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.381+0000 7f9386156640 1 -- 172.21.15.116:0/2795267642 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f938014f000 con 0x7f93800c0b30 2024-04-25T00:23:32.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.381+0000 7f9386156640 1 -- 172.21.15.116:0/2795267642 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93800c5670 con 0x7f93800c0b30 2024-04-25T00:23:32.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.381+0000 7f937cff9640 1 -- 172.21.15.116:0/2795267642 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9370007260 con 0x7f93800c0b30 2024-04-25T00:23:32.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.381+0000 7f9386156640 1 -- 172.21.15.116:0/2795267642 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9380068a60 con 0x7f93800c0b30 2024-04-25T00:23:32.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.385+0000 7f937cff9640 1 -- 172.21.15.116:0/2795267642 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f9370007400 con 0x7f93800c0b30 2024-04-25T00:23:32.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.385+0000 7f937cff9640 1 --2- 172.21.15.116:0/2795267642 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f93600678c0 0x7f9360069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:32.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.385+0000 7f937cff9640 1 -- 172.21.15.116:0/2795267642 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 9762+0+0 (secure 0 0 0) 0x7f93700861b0 con 0x7f93800c0b30 2024-04-25T00:23:32.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.385+0000 7f9377fff640 1 --2- 172.21.15.116:0/2795267642 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f93600678c0 0x7f9360069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:32.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.385+0000 7f9377fff640 1 --2- 172.21.15.116:0/2795267642 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f93600678c0 0x7f9360069d80 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f93680af000 tx=0x7f9368008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:32.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.389+0000 7f937cff9640 1 -- 172.21.15.116:0/2795267642 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f938014fe40 con 0x7f93800c0b30 2024-04-25T00:23:32.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.581+0000 7f9386156640 1 -- 172.21.15.116:0/2795267642 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f938013e320 con 0x7f93800c0b30 2024-04-25T00:23:32.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.585+0000 7f937cff9640 1 -- 172.21.15.116:0/2795267642 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 189 v189) ==== 95+0+5672 (secure 0 0 0) 0x7f9370055c60 con 0x7f93800c0b30 2024-04-25T00:23:32.586 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:23:32.586 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":189,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":189,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:23:31.107103+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8244":{"gid":8244,"name":"c","rank":0,"incarnation":188,"state":"up:reconnect","state_seq":9,"addr":"172.21.15.116:6835/1971675962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1971675962},{"type":"v1","addr":"172.21.15.116:6835","nonce":1971675962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:22:38.936345+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6230":{"gid":6230,"name":"d","rank":0,"incarnation":178,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/492366509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":492366509},{"type":"v1","addr":"172.21.15.174:6837","nonce":492366509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6230,"qdb_cluster":[6230]},"id":2}]} 2024-04-25T00:23:32.586 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 189 2024-04-25T00:23:32.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.585+0000 7f9386156640 1 -- 172.21.15.116:0/2795267642 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f93600678c0 msgr2=0x7f9360069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:32.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.585+0000 7f9386156640 1 --2- 172.21.15.116:0/2795267642 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f93600678c0 0x7f9360069d80 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f93680af000 tx=0x7f9368008040 comp rx=0 tx=0).stop 2024-04-25T00:23:32.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.585+0000 7f9386156640 1 -- 172.21.15.116:0/2795267642 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93800c0b30 msgr2=0x7f93801ef050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:32.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.585+0000 7f9386156640 1 --2- 172.21.15.116:0/2795267642 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93800c0b30 0x7f93801ef050 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f937000c370 tx=0x7f937000c830 comp rx=0 tx=0).stop 2024-04-25T00:23:32.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.585+0000 7f9386156640 1 -- 172.21.15.116:0/2795267642 shutdown_connections 2024-04-25T00:23:32.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.585+0000 7f9386156640 1 --2- 172.21.15.116:0/2795267642 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f93600678c0 0x7f9360069d80 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:32.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.585+0000 7f9386156640 1 --2- 172.21.15.116:0/2795267642 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9380158e70 0x7f93800ac0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:32.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.585+0000 7f9386156640 1 --2- 172.21.15.116:0/2795267642 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f938014fa60 0x7f93800acf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:32.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.585+0000 7f9386156640 1 --2- 172.21.15.116:0/2795267642 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f93800c0b30 0x7f93801ef050 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:32.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.585+0000 7f9386156640 1 -- 172.21.15.116:0/2795267642 >> 172.21.15.116:0/2795267642 conn(0x7f93800c4200 msgr2=0x7f9380155110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:32.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.585+0000 7f9386156640 1 -- 172.21.15.116:0/2795267642 shutdown_connections 2024-04-25T00:23:32.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:32.585+0000 7f9386156640 1 -- 172.21.15.116:0/2795267642 wait complete. 2024-04-25T00:23:32.604 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 189, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:04.678737+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 189, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_8244": { "addr": "172.21.15.116:6835/1971675962", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6834", "nonce": 1971675962, "type": "v2" }, { "addr": "172.21.15.116:6835", "nonce": 1971675962, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8244, "incarnation": 188, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:reconnect", "state_seq": 9 } }, "last_failure": 0, "last_failure_osd_epoch": 64, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T00:23:31.107103+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8244 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:18.825887+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 181, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6230": { "addr": "172.21.15.174:6837/492366509", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6836", "nonce": 492366509, "type": "v2" }, { "addr": "172.21.15.174:6837", "nonce": 492366509, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6230, "incarnation": 178, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 7 } }, "last_failure": 0, "last_failure_osd_epoch": 62, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-04-25T00:22:38.936345+0000", "qdb_cluster": [ 6230 ], "qdb_leader": 6230, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6230 } } } ], "standbys": [ { "addr": "172.21.15.174:6835/4112145782", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6834", "nonce": 4112145782, "type": "v2" }, { "addr": "172.21.15.174:6835", "nonce": 4112145782, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 112, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5582, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-25T00:23:34.605 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:23:34.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.913+0000 7fe35d6e6640 1 Processor -- start 2024-04-25T00:23:34.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.913+0000 7fe35d6e6640 1 -- start start 2024-04-25T00:23:34.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.913+0000 7fe35d6e6640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fe3500d4670 0x7fe3500d4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:34.915 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.913+0000 7fe35d6e6640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3500084b0 con 0x7fe350047470 2024-04-25T00:23:34.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.913+0000 7fe35d6e6640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe350049b20 con 0x7fe3500068a0 2024-04-25T00:23:34.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.913+0000 7fe35d6e6640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fe350031af0 con 0x7fe3500d4670 2024-04-25T00:23:34.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.913+0000 7fe3577fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fe350047470 0x7fe350047850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:51964/0 (socket says 172.21.15.116:51964) 2024-04-25T00:23:34.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.913+0000 7fe3577fe640 1 -- 172.21.15.116:0/3758802694 learned_addr learned my addr 172.21.15.116:0/3758802694 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:23:34.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.913+0000 7fe3567fc640 1 -- 172.21.15.116:0/3758802694 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2331548030 0 0) 0x7fe3500084b0 con 0x7fe350047470 2024-04-25T00:23:34.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.913+0000 7fe3567fc640 1 -- 172.21.15.116:0/3758802694 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe33c003530 con 0x7fe350047470 2024-04-25T00:23:34.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.913+0000 7fe3567fc640 1 -- 172.21.15.116:0/3758802694 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 565227177 0 0) 0x7fe33c003530 con 0x7fe350047470 2024-04-25T00:23:34.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.913+0000 7fe3567fc640 1 -- 172.21.15.116:0/3758802694 >> v2:172.21.15.174:3300/0 conn(0x7fe3500d4670 msgr2=0x7fe3500d4a80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T00:23:34.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.913+0000 7fe3567fc640 1 --2- 172.21.15.116:0/3758802694 >> v2:172.21.15.174:3300/0 conn(0x7fe3500d4670 0x7fe3500d4a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:34.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.913+0000 7fe3567fc640 1 -- 172.21.15.116:0/3758802694 >> v1:172.21.15.174:6789/0 conn(0x7fe3500068a0 legacy=0x7fe350007da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:34.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.913+0000 7fe3567fc640 1 -- 172.21.15.116:0/3758802694 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe350046db0 con 0x7fe350047470 2024-04-25T00:23:34.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.913+0000 7fe3567fc640 1 -- 172.21.15.116:0/3758802694 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7fe34c001a00 con 0x7fe350047470 2024-04-25T00:23:34.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.913+0000 7fe3567fc640 1 -- 172.21.15.116:0/3758802694 >> v1:172.21.15.116:6789/0 conn(0x7fe350047470 legacy=0x7fe350047850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:34.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.913+0000 7fe3567fc640 1 --2- 172.21.15.116:0/3758802694 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe33c0039c0 0x7fe33c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:34.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.913+0000 7fe3567fc640 1 --2- 172.21.15.116:0/3758802694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe33c004810 0x7fe33c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:34.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.913+0000 7fe3567fc640 1 --2- 172.21.15.116:0/3758802694 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe33c005420 0x7fe33c00ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:34.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.913+0000 7fe3567fc640 1 -- 172.21.15.116:0/3758802694 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe350031af0 con 0x7fe33c004810 2024-04-25T00:23:34.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.913+0000 7fe3567fc640 1 -- 172.21.15.116:0/3758802694 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe33c00e540 con 0x7fe33c0039c0 2024-04-25T00:23:34.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.913+0000 7fe3567fc640 1 -- 172.21.15.116:0/3758802694 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe33c00e740 con 0x7fe33c005420 2024-04-25T00:23:34.918 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.913+0000 7fe356ffd640 1 --2- 172.21.15.116:0/3758802694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe33c004810 0x7fe33c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:34.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.913+0000 7fe3577fe640 1 --2- 172.21.15.116:0/3758802694 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe33c0039c0 0x7fe33c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:34.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.913+0000 7fe357fff640 1 --2- 172.21.15.116:0/3758802694 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe33c005420 0x7fe33c00ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:34.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.913+0000 7fe356ffd640 1 -- 172.21.15.116:0/3758802694 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe33c005420 msgr2=0x7fe33c00ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:34.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.913+0000 7fe356ffd640 1 --2- 172.21.15.116:0/3758802694 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe33c005420 0x7fe33c00ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:34.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.913+0000 7fe356ffd640 1 -- 172.21.15.116:0/3758802694 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe33c0039c0 msgr2=0x7fe33c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:23:34.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.913+0000 7fe356ffd640 1 --2- 172.21.15.116:0/3758802694 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe33c0039c0 0x7fe33c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:34.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.917+0000 7fe356ffd640 1 -- 172.21.15.116:0/3758802694 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe350046db0 con 0x7fe33c004810 2024-04-25T00:23:34.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.917+0000 7fe357fff640 1 -- 172.21.15.116:0/3758802694 reap_dead start 2024-04-25T00:23:34.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.917+0000 7fe356ffd640 1 --2- 172.21.15.116:0/3758802694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe33c004810 0x7fe33c004c90 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7fe344003dd0 tx=0x7fe34401c2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6073c0739cf26711 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:34.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.917+0000 7fe3567fc640 1 -- 172.21.15.116:0/3758802694 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3440b0070 con 0x7fe33c004810 2024-04-25T00:23:34.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.917+0000 7fe3567fc640 1 -- 172.21.15.116:0/3758802694 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3440ab600 con 0x7fe33c004810 2024-04-25T00:23:34.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.917+0000 7fe3567fc640 1 -- 172.21.15.116:0/3758802694 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3440b0070 con 0x7fe33c004810 2024-04-25T00:23:34.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.921+0000 7fe35d6e6640 1 -- 172.21.15.116:0/3758802694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe33c004810 msgr2=0x7fe33c004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:34.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.921+0000 7fe35d6e6640 1 --2- 172.21.15.116:0/3758802694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe33c004810 0x7fe33c004c90 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7fe344003dd0 tx=0x7fe34401c2e0 comp rx=0 tx=0).stop 2024-04-25T00:23:34.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.925+0000 7fe35d6e6640 1 -- 172.21.15.116:0/3758802694 shutdown_connections 2024-04-25T00:23:34.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.925+0000 7fe35d6e6640 1 --2- 172.21.15.116:0/3758802694 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe33c004810 0x7fe33c004c90 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:34.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.925+0000 7fe35d6e6640 1 -- 172.21.15.116:0/3758802694 >> 172.21.15.116:0/3758802694 conn(0x7fe350048ca0 msgr2=0x7fe3500490b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:34.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.925+0000 7fe35d6e6640 1 -- 172.21.15.116:0/3758802694 shutdown_connections 2024-04-25T00:23:34.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.925+0000 7fe35d6e6640 1 -- 172.21.15.116:0/3758802694 wait complete. 2024-04-25T00:23:34.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.925+0000 7fe35d6e6640 1 Processor -- start 2024-04-25T00:23:34.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.925+0000 7fe35d6e6640 1 -- start start 2024-04-25T00:23:34.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.925+0000 7fe35d6e6640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe33c004810 0x7fe350005e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:34.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.925+0000 7fe35d6e6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe3500439a0 0x7fe3500d9d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:34.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.925+0000 7fe35d6e6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe350043e20 0x7fe3500da240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:34.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.929+0000 7fe3577fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe33c004810 0x7fe350005e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:34.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.929+0000 7fe35d6e6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe350032040 con 0x7fe350043e20 2024-04-25T00:23:34.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.929+0000 7fe35d6e6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe350031ec0 con 0x7fe3500439a0 2024-04-25T00:23:34.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.929+0000 7fe35d6e6640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe350047850 con 0x7fe33c004810 2024-04-25T00:23:34.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.929+0000 7fe3577fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe33c004810 0x7fe350005e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:39520/0 (socket says 172.21.15.116:39520) 2024-04-25T00:23:34.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.929+0000 7fe3577fe640 1 -- 172.21.15.116:0/715861654 learned_addr learned my addr 172.21.15.116:0/715861654 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:23:34.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.929+0000 7fe356ffd640 1 --2- 172.21.15.116:0/715861654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe3500439a0 0x7fe3500d9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:34.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.929+0000 7fe3577fe640 1 -- 172.21.15.116:0/715861654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe3500439a0 msgr2=0x7fe3500d9d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:34.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.929+0000 7fe3577fe640 1 --2- 172.21.15.116:0/715861654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe3500439a0 0x7fe3500d9d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:34.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.929+0000 7fe3577fe640 1 -- 172.21.15.116:0/715861654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe350043e20 msgr2=0x7fe3500da240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:34.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.929+0000 7fe3577fe640 1 --2- 172.21.15.116:0/715861654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe350043e20 0x7fe3500da240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:34.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.929+0000 7fe3577fe640 1 -- 172.21.15.116:0/715861654 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3500d4130 con 0x7fe33c004810 2024-04-25T00:23:34.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.929+0000 7fe356ffd640 1 --2- 172.21.15.116:0/715861654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe3500439a0 0x7fe3500d9d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:23:34.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.929+0000 7fe3577fe640 1 --2- 172.21.15.116:0/715861654 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe33c004810 0x7fe350005e30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe34c007970 tx=0x7fe34c00a770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:34.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.929+0000 7fe354ff9640 1 -- 172.21.15.116:0/715861654 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe34c017020 con 0x7fe33c004810 2024-04-25T00:23:34.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.929+0000 7fe354ff9640 1 -- 172.21.15.116:0/715861654 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe34c0133f0 con 0x7fe33c004810 2024-04-25T00:23:34.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.929+0000 7fe35d6e6640 1 -- 172.21.15.116:0/715861654 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3500d72d0 con 0x7fe33c004810 2024-04-25T00:23:34.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.929+0000 7fe354ff9640 1 -- 172.21.15.116:0/715861654 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe34c0137c0 con 0x7fe33c004810 2024-04-25T00:23:34.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.929+0000 7fe35d6e6640 1 -- 172.21.15.116:0/715861654 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe35002d870 con 0x7fe33c004810 2024-04-25T00:23:34.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.929+0000 7fe354ff9640 1 -- 172.21.15.116:0/715861654 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe34c01a020 con 0x7fe33c004810 2024-04-25T00:23:34.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.929+0000 7fe35d6e6640 1 -- 172.21.15.116:0/715861654 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3500059c0 con 0x7fe33c004810 2024-04-25T00:23:34.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.929+0000 7fe354ff9640 1 --2- 172.21.15.116:0/715861654 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe3580e2be0 0x7fe3580e50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:34.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.929+0000 7fe354ff9640 1 -- 172.21.15.116:0/715861654 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(64..64 src has 1..64) ==== 9762+0+0 (secure 0 0 0) 0x7fe34c0879e0 con 0x7fe33c004810 2024-04-25T00:23:34.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.929+0000 7fe356ffd640 1 --2- 172.21.15.116:0/715861654 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe3580e2be0 0x7fe3580e50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:34.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.929+0000 7fe356ffd640 1 --2- 172.21.15.116:0/715861654 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe3580e2be0 0x7fe3580e50a0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fe3440ad000 tx=0x7fe344003770 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:34.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:34.941+0000 7fe354ff9640 1 -- 172.21.15.116:0/715861654 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fe34c021020 con 0x7fe33c004810 2024-04-25T00:23:35.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:35.329+0000 7fe35d6e6640 1 -- 172.21.15.116:0/715861654 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe350049890 con 0x7fe33c004810 2024-04-25T00:23:35.331 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:23:35.331 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":191,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":191,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:23:34.013510+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8244":{"gid":8244,"name":"c","rank":0,"incarnation":188,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6835/1971675962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1971675962},{"type":"v1","addr":"172.21.15.116:6835","nonce":1971675962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8244,"qdb_cluster":[8244]},"id":1},{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:22:38.936345+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6230":{"gid":6230,"name":"d","rank":0,"incarnation":178,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/492366509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":492366509},{"type":"v1","addr":"172.21.15.174:6837","nonce":492366509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6230,"qdb_cluster":[6230]},"id":2}]} 2024-04-25T00:23:35.331 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 191 2024-04-25T00:23:35.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:35.329+0000 7fe354ff9640 1 -- 172.21.15.116:0/715861654 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 191 v191) ==== 95+0+5677 (secure 0 0 0) 0x7fe34c0574a0 con 0x7fe33c004810 2024-04-25T00:23:35.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:35.329+0000 7fe35d6e6640 1 -- 172.21.15.116:0/715861654 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe3580e2be0 msgr2=0x7fe3580e50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:35.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:35.329+0000 7fe35d6e6640 1 --2- 172.21.15.116:0/715861654 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe3580e2be0 0x7fe3580e50a0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fe3440ad000 tx=0x7fe344003770 comp rx=0 tx=0).stop 2024-04-25T00:23:35.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:35.329+0000 7fe35d6e6640 1 -- 172.21.15.116:0/715861654 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe33c004810 msgr2=0x7fe350005e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:35.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:35.329+0000 7fe35d6e6640 1 --2- 172.21.15.116:0/715861654 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe33c004810 0x7fe350005e30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe34c007970 tx=0x7fe34c00a770 comp rx=0 tx=0).stop 2024-04-25T00:23:35.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:35.329+0000 7fe35d6e6640 1 -- 172.21.15.116:0/715861654 shutdown_connections 2024-04-25T00:23:35.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:35.329+0000 7fe35d6e6640 1 --2- 172.21.15.116:0/715861654 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe3580e2be0 0x7fe3580e50a0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:35.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:35.329+0000 7fe35d6e6640 1 --2- 172.21.15.116:0/715861654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe350043e20 0x7fe3500da240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:35.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:35.329+0000 7fe35d6e6640 1 --2- 172.21.15.116:0/715861654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe3500439a0 0x7fe3500d9d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:35.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:35.329+0000 7fe35d6e6640 1 --2- 172.21.15.116:0/715861654 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe33c004810 0x7fe350005e30 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:35.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:35.329+0000 7fe35d6e6640 1 -- 172.21.15.116:0/715861654 >> 172.21.15.116:0/715861654 conn(0x7fe350048ca0 msgr2=0x7fe350007830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:35.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:35.329+0000 7fe35d6e6640 1 -- 172.21.15.116:0/715861654 shutdown_connections 2024-04-25T00:23:35.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:35.329+0000 7fe35d6e6640 1 -- 172.21.15.116:0/715861654 wait complete. 2024-04-25T00:23:35.448 INFO:tasks.mds_thrash.fs.[a]:mds.c has gained rank=0, replacing gid=8214 2024-04-25T00:23:35.448 INFO:tasks.mds_thrash.fs.[a]:waiting for 7 secs before reviving mds.a 2024-04-25T00:23:35.989 INFO:tasks.ceph.mds.c.smithi116.stderr:2024-04-25T00:23:35.985+0000 7fe4c5ae8640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T00:23:40.303 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:23:40.305 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:23:42.449 INFO:tasks.mds_thrash.fs.[a]:reviving mds.a 2024-04-25T00:23:42.449 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-25T00:23:42.450 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-25T00:23:42.457 INFO:tasks.ceph.mds.a:Started 2024-04-25T00:23:42.458 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:23:42.595 INFO:tasks.ceph.mds.a.smithi116.stdout:starting mds.a at 2024-04-25T00:23:42.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.693+0000 7f7b17fff640 1 Processor -- start 2024-04-25T00:23:42.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.693+0000 7f7b17fff640 1 -- start start 2024-04-25T00:23:42.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.693+0000 7f7b17fff640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f7b10048350 0x7f7b100d82b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:42.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.693+0000 7f7b17fff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b100d2590 0x7f7b10048730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:42.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.693+0000 7f7b17fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7b1000be50 0x7f7b1000d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:42.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.693+0000 7f7b17fff640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f7b1000cf60 con 0x7f7b1000be50 2024-04-25T00:23:42.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.693+0000 7f7b17fff640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f7b1000c350 con 0x7f7b10048350 2024-04-25T00:23:42.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.693+0000 7f7b17fff640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f7b1000c630 con 0x7f7b100d2590 2024-04-25T00:23:42.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.693+0000 7f7b167fc640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b100d2590 0x7f7b10048730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:42.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.693+0000 7f7b167fc640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b100d2590 0x7f7b10048730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:39550/0 (socket says 172.21.15.116:39550) 2024-04-25T00:23:42.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.693+0000 7f7b167fc640 1 -- 172.21.15.116:0/2566119140 learned_addr learned my addr 172.21.15.116:0/2566119140 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:23:42.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.697+0000 7f7b167fc640 1 -- 172.21.15.116:0/2566119140 >> v2:172.21.15.174:3300/0 conn(0x7f7b10048350 msgr2=0x7f7b100d82b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T00:23:42.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.697+0000 7f7b167fc640 1 --2- 172.21.15.116:0/2566119140 >> v2:172.21.15.174:3300/0 conn(0x7f7b10048350 0x7f7b100d82b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:42.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.697+0000 7f7b167fc640 1 -- 172.21.15.116:0/2566119140 >> v2:172.21.15.116:3300/0 conn(0x7f7b1000be50 msgr2=0x7f7b1000d3f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T00:23:42.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.697+0000 7f7b167fc640 1 --2- 172.21.15.116:0/2566119140 >> v2:172.21.15.116:3300/0 conn(0x7f7b1000be50 0x7f7b1000d3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:42.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.697+0000 7f7b167fc640 1 -- 172.21.15.116:0/2566119140 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b1000c910 con 0x7f7b100d2590 2024-04-25T00:23:42.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.701+0000 7f7b167fc640 1 --2- 172.21.15.116:0/2566119140 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b100d2590 0x7f7b10048730 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7b1004ea40 tx=0x7f7b0c01c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4527ed5b6c29ff5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:42.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.701+0000 7f7b15ffb640 1 -- 172.21.15.116:0/2566119140 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b0c0b0070 con 0x7f7b100d2590 2024-04-25T00:23:42.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.701+0000 7f7b15ffb640 1 -- 172.21.15.116:0/2566119140 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b0c0ab600 con 0x7f7b100d2590 2024-04-25T00:23:42.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.701+0000 7f7b15ffb640 1 -- 172.21.15.116:0/2566119140 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b0c0ab9d0 con 0x7f7b100d2590 2024-04-25T00:23:42.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.701+0000 7f7b17fff640 1 -- 172.21.15.116:0/2566119140 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b100d2590 msgr2=0x7f7b10048730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:42.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.701+0000 7f7b17fff640 1 --2- 172.21.15.116:0/2566119140 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b100d2590 0x7f7b10048730 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7b1004ea40 tx=0x7f7b0c01c470 comp rx=0 tx=0).stop 2024-04-25T00:23:42.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.701+0000 7f7b17fff640 1 -- 172.21.15.116:0/2566119140 shutdown_connections 2024-04-25T00:23:42.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.701+0000 7f7b17fff640 1 --2- 172.21.15.116:0/2566119140 >> v2:172.21.15.116:3300/0 conn(0x7f7b1000be50 0x7f7b1000d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:42.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.701+0000 7f7b17fff640 1 --2- 172.21.15.116:0/2566119140 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b100d2590 0x7f7b10048730 secure :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7b1004ea40 tx=0x7f7b0c01c470 comp rx=0 tx=0).stop 2024-04-25T00:23:42.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.701+0000 7f7b17fff640 1 --2- 172.21.15.116:0/2566119140 >> v2:172.21.15.174:3300/0 conn(0x7f7b10048350 0x7f7b100d82b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:42.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.701+0000 7f7b17fff640 1 -- 172.21.15.116:0/2566119140 >> 172.21.15.116:0/2566119140 conn(0x7f7b1002f4c0 msgr2=0x7f7b1002f8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:42.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.701+0000 7f7b17fff640 1 -- 172.21.15.116:0/2566119140 shutdown_connections 2024-04-25T00:23:42.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.701+0000 7f7b17fff640 1 -- 172.21.15.116:0/2566119140 wait complete. 2024-04-25T00:23:42.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.705+0000 7f7b17fff640 1 Processor -- start 2024-04-25T00:23:42.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.705+0000 7f7b17fff640 1 -- start start 2024-04-25T00:23:42.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.705+0000 7f7b17fff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b1000be50 0x7f7b1017b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:42.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.705+0000 7f7b17fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b10048350 0x7f7b10008430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:42.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.705+0000 7f7b17fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b10008970 0x7f7b100ca740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:42.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.705+0000 7f7b17fff640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7b100dbc30 con 0x7f7b10048350 2024-04-25T00:23:42.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.705+0000 7f7b17fff640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7b100dbab0 con 0x7f7b10008970 2024-04-25T00:23:42.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.705+0000 7f7b17fff640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f7b100dbdb0 con 0x7f7b1000be50 2024-04-25T00:23:42.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.705+0000 7f7b167fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b10048350 0x7f7b10008430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:42.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.705+0000 7f7b167fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b10048350 0x7f7b10008430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40656/0 (socket says 172.21.15.116:40656) 2024-04-25T00:23:42.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.705+0000 7f7b167fc640 1 -- 172.21.15.116:0/2130447804 learned_addr learned my addr 172.21.15.116:0/2130447804 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:23:42.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.705+0000 7f7b16ffd640 1 --2- 172.21.15.116:0/2130447804 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b1000be50 0x7f7b1017b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:42.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.705+0000 7f7b167fc640 1 -- 172.21.15.116:0/2130447804 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b1000be50 msgr2=0x7f7b1017b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:42.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.705+0000 7f7b167fc640 1 --2- 172.21.15.116:0/2130447804 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b1000be50 0x7f7b1017b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:42.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.705+0000 7f7b167fc640 1 -- 172.21.15.116:0/2130447804 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b10008970 msgr2=0x7f7b100ca740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:42.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.705+0000 7f7b167fc640 1 --2- 172.21.15.116:0/2130447804 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b10008970 0x7f7b100ca740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:42.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.705+0000 7f7b167fc640 1 -- 172.21.15.116:0/2130447804 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b10007c70 con 0x7f7b10048350 2024-04-25T00:23:42.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.705+0000 7f7b167fc640 1 --2- 172.21.15.116:0/2130447804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b10048350 0x7f7b10008430 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c01a100 tx=0x7f7b0c019270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:42.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.705+0000 7f7aff7fe640 1 -- 172.21.15.116:0/2130447804 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b0c0b9070 con 0x7f7b10048350 2024-04-25T00:23:42.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.705+0000 7f7aff7fe640 1 -- 172.21.15.116:0/2130447804 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b0c0b1040 con 0x7f7b10048350 2024-04-25T00:23:42.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.705+0000 7f7aff7fe640 1 -- 172.21.15.116:0/2130447804 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b0c0b0050 con 0x7f7b10048350 2024-04-25T00:23:42.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.705+0000 7f7b17fff640 1 -- 172.21.15.116:0/2130447804 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b1017ba00 con 0x7f7b10048350 2024-04-25T00:23:42.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.705+0000 7f7b17fff640 1 -- 172.21.15.116:0/2130447804 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b1017bce0 con 0x7f7b10048350 2024-04-25T00:23:42.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.709+0000 7f7b17fff640 1 -- 172.21.15.116:0/2130447804 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7af80052a0 con 0x7f7b10048350 2024-04-25T00:23:42.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.709+0000 7f7aff7fe640 1 -- 172.21.15.116:0/2130447804 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7b0c004220 con 0x7f7b10048350 2024-04-25T00:23:42.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.709+0000 7f7aff7fe640 1 --2- 172.21.15.116:0/2130447804 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7b040678c0 0x7f7b04069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:23:42.711 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.709+0000 7f7aff7fe640 1 -- 172.21.15.116:0/2130447804 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 9762+0+0 (secure 0 0 0) 0x7f7b0c120940 con 0x7f7b10048350 2024-04-25T00:23:42.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.709+0000 7f7b16ffd640 1 --2- 172.21.15.116:0/2130447804 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7b040678c0 0x7f7b04069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:23:42.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.709+0000 7f7b16ffd640 1 --2- 172.21.15.116:0/2130447804 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7b040678c0 0x7f7b04069d80 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f7b10031920 tx=0x7f7b08009060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:23:42.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:42.717+0000 7f7aff7fe640 1 -- 172.21.15.116:0/2130447804 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f7b0c016150 con 0x7f7b10048350 2024-04-25T00:23:43.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:43.117+0000 7f7b17fff640 1 -- 172.21.15.116:0/2130447804 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7af8005590 con 0x7f7b10048350 2024-04-25T00:23:43.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:43.117+0000 7f7aff7fe640 1 -- 172.21.15.116:0/2130447804 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 192 v192) ==== 95+0+6506 (secure 0 0 0) 0x7f7b0c0f04c0 con 0x7f7b10048350 2024-04-25T00:23:43.120 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:23:43.120 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":192,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":8343,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/3633657268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3633657268},{"type":"v1","addr":"172.21.15.116:6833","nonce":3633657268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":191,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:23:34.013510+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8244":{"gid":8244,"name":"c","rank":0,"incarnation":188,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6835/1971675962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1971675962},{"type":"v1","addr":"172.21.15.116:6835","nonce":1971675962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8244,"qdb_cluster":[8244]},"id":1},{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:22:38.936345+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6230":{"gid":6230,"name":"d","rank":0,"incarnation":178,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/492366509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":492366509},{"type":"v1","addr":"172.21.15.174:6837","nonce":492366509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6230,"qdb_cluster":[6230]},"id":2}]} 2024-04-25T00:23:43.120 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 192 2024-04-25T00:23:43.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:43.125+0000 7f7afd7fa640 1 -- 172.21.15.116:0/2130447804 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7b040678c0 msgr2=0x7f7b04069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:43.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:43.125+0000 7f7afd7fa640 1 --2- 172.21.15.116:0/2130447804 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7b040678c0 0x7f7b04069d80 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f7b10031920 tx=0x7f7b08009060 comp rx=0 tx=0).stop 2024-04-25T00:23:43.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:43.125+0000 7f7afd7fa640 1 -- 172.21.15.116:0/2130447804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b10048350 msgr2=0x7f7b10008430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:23:43.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:43.125+0000 7f7afd7fa640 1 --2- 172.21.15.116:0/2130447804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b10048350 0x7f7b10008430 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c01a100 tx=0x7f7b0c019270 comp rx=0 tx=0).stop 2024-04-25T00:23:43.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:43.125+0000 7f7afd7fa640 1 -- 172.21.15.116:0/2130447804 shutdown_connections 2024-04-25T00:23:43.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:43.125+0000 7f7afd7fa640 1 --2- 172.21.15.116:0/2130447804 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7b040678c0 0x7f7b04069d80 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:43.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:43.125+0000 7f7afd7fa640 1 --2- 172.21.15.116:0/2130447804 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7b10008970 0x7f7b100ca740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:43.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:43.125+0000 7f7afd7fa640 1 --2- 172.21.15.116:0/2130447804 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7b10048350 0x7f7b10008430 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:43.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:43.125+0000 7f7afd7fa640 1 --2- 172.21.15.116:0/2130447804 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7b1000be50 0x7f7b1017b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:23:43.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:43.125+0000 7f7afd7fa640 1 -- 172.21.15.116:0/2130447804 >> 172.21.15.116:0/2130447804 conn(0x7f7b1002f4c0 msgr2=0x7f7b100cec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:23:43.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:43.129+0000 7f7afd7fa640 1 -- 172.21.15.116:0/2130447804 shutdown_connections 2024-04-25T00:23:43.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:23:43.129+0000 7f7afd7fa640 1 -- 172.21.15.116:0/2130447804 wait complete. 2024-04-25T00:23:43.156 INFO:tasks.mds_thrash.fs.[a]:mds.a reported in up:standby state 2024-04-25T00:23:43.157 INFO:tasks.mds_thrash.fs.[a]:waiting for 88 secs before thrashing 2024-04-25T00:24:00.890 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:24:01.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.189+0000 7fbe0e184640 1 Processor -- start 2024-04-25T00:24:01.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.189+0000 7fbe0e184640 1 -- start start 2024-04-25T00:24:01.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.189+0000 7fbe0e184640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe080c2a10 0x7fbe080c2df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:01.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.189+0000 7fbe0e184640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fbe080c1280 0x7fbe08154010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:01.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.189+0000 7fbe0e184640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe0806e660 con 0x7fbe080ac830 2024-04-25T00:24:01.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.189+0000 7fbe0e184640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fbe080c3940 con 0x7fbe080c1280 2024-04-25T00:24:01.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.189+0000 7fbe0e184640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fbe08154550 con 0x7fbe080c2a10 2024-04-25T00:24:01.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.189+0000 7fbe06ffd640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fbe080c1280 0x7fbe08154010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:01.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.189+0000 7fbe06ffd640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fbe080c1280 0x7fbe08154010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:57300/0 (socket says 172.21.15.116:57300) 2024-04-25T00:24:01.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.189+0000 7fbe06ffd640 1 -- 172.21.15.116:0/1153417106 learned_addr learned my addr 172.21.15.116:0/1153417106 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:24:01.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.189+0000 7fbe06ffd640 1 -- 172.21.15.116:0/1153417106 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe080c2a10 msgr2=0x7fbe080c2df0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T00:24:01.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.189+0000 7fbe06ffd640 1 --2- 172.21.15.116:0/1153417106 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe080c2a10 0x7fbe080c2df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:01.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.189+0000 7fbe06ffd640 1 -- 172.21.15.116:0/1153417106 >> v1:172.21.15.174:6789/0 conn(0x7fbe080ac830 legacy=0x7fbe08156b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T00:24:01.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.189+0000 7fbe06ffd640 1 -- 172.21.15.116:0/1153417106 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe0806e940 con 0x7fbe080c1280 2024-04-25T00:24:01.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.193+0000 7fbe06ffd640 1 --2- 172.21.15.116:0/1153417106 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe080c1280 0x7fbe08154010 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fbdf0009870 tx=0x7fbdf001c540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4fb887007898735 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:01.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.193+0000 7fbe067fc640 1 -- 172.21.15.116:0/1153417106 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdf00b0070 con 0x7fbe080c1280 2024-04-25T00:24:01.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.193+0000 7fbe067fc640 1 -- 172.21.15.116:0/1153417106 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbdf00ab600 con 0x7fbe080c1280 2024-04-25T00:24:01.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.193+0000 7fbe067fc640 1 -- 172.21.15.116:0/1153417106 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdf00ab9d0 con 0x7fbe080c1280 2024-04-25T00:24:01.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.193+0000 7fbe0e184640 1 -- 172.21.15.116:0/1153417106 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe080c1280 msgr2=0x7fbe08154010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:01.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.193+0000 7fbe0e184640 1 --2- 172.21.15.116:0/1153417106 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe080c1280 0x7fbe08154010 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fbdf0009870 tx=0x7fbdf001c540 comp rx=0 tx=0).stop 2024-04-25T00:24:01.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.197+0000 7fbe0e184640 1 -- 172.21.15.116:0/1153417106 shutdown_connections 2024-04-25T00:24:01.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.197+0000 7fbe0e184640 1 --2- 172.21.15.116:0/1153417106 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe080c1280 0x7fbe08154010 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:01.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.197+0000 7fbe0e184640 1 --2- 172.21.15.116:0/1153417106 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe080c2a10 0x7fbe080c2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:01.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.197+0000 7fbe0e184640 1 -- 172.21.15.116:0/1153417106 >> 172.21.15.116:0/1153417106 conn(0x7fbe080c3b00 msgr2=0x7fbe080c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:01.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.197+0000 7fbe0e184640 1 -- 172.21.15.116:0/1153417106 shutdown_connections 2024-04-25T00:24:01.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.197+0000 7fbe0e184640 1 -- 172.21.15.116:0/1153417106 wait complete. 2024-04-25T00:24:01.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.201+0000 7fbe0e184640 1 Processor -- start 2024-04-25T00:24:01.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.201+0000 7fbe0e184640 1 -- start start 2024-04-25T00:24:01.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.201+0000 7fbe0e184640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe080ac830 0x7fbe080c5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:01.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.201+0000 7fbe0e184640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe080c1280 0x7fbe08142b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:01.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.201+0000 7fbe0e184640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe080c2a10 0x7fbe081430d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:01.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.201+0000 7fbe0e184640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbe0806ccd0 con 0x7fbe080ac830 2024-04-25T00:24:01.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.201+0000 7fbe0e184640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbe0806cb50 con 0x7fbe080c1280 2024-04-25T00:24:01.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.201+0000 7fbe0e184640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fbe08150e40 con 0x7fbe080c2a10 2024-04-25T00:24:01.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.201+0000 7fbe077fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe080ac830 0x7fbe080c5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:01.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.201+0000 7fbe077fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe080ac830 0x7fbe080c5ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54278/0 (socket says 172.21.15.116:54278) 2024-04-25T00:24:01.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.201+0000 7fbe077fe640 1 -- 172.21.15.116:0/3635119131 learned_addr learned my addr 172.21.15.116:0/3635119131 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:24:01.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.201+0000 7fbe06ffd640 1 --2- 172.21.15.116:0/3635119131 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe080c1280 0x7fbe08142b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:01.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.201+0000 7fbe077fe640 1 -- 172.21.15.116:0/3635119131 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe080c2a10 msgr2=0x7fbe081430d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T00:24:01.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.201+0000 7fbe077fe640 1 --2- 172.21.15.116:0/3635119131 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe080c2a10 0x7fbe081430d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:01.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.201+0000 7fbe077fe640 1 -- 172.21.15.116:0/3635119131 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe080c1280 msgr2=0x7fbe08142b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:01.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.201+0000 7fbe077fe640 1 --2- 172.21.15.116:0/3635119131 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe080c1280 0x7fbe08142b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:01.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.201+0000 7fbe077fe640 1 -- 172.21.15.116:0/3635119131 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe08143610 con 0x7fbe080ac830 2024-04-25T00:24:01.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.201+0000 7fbe077fe640 1 --2- 172.21.15.116:0/3635119131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe080ac830 0x7fbe080c5ea0 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7fbe0000ba80 tx=0x7fbe0000bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:01.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.201+0000 7fbdfffff640 1 -- 172.21.15.116:0/3635119131 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe00019070 con 0x7fbe080ac830 2024-04-25T00:24:01.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.201+0000 7fbdfffff640 1 -- 172.21.15.116:0/3635119131 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe0000cc20 con 0x7fbe080ac830 2024-04-25T00:24:01.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.201+0000 7fbdfffff640 1 -- 172.21.15.116:0/3635119131 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe00014440 con 0x7fbe080ac830 2024-04-25T00:24:01.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.201+0000 7fbe0e184640 1 -- 172.21.15.116:0/3635119131 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe0814ae10 con 0x7fbe080ac830 2024-04-25T00:24:01.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.201+0000 7fbe0e184640 1 -- 172.21.15.116:0/3635119131 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe08071170 con 0x7fbe080ac830 2024-04-25T00:24:01.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.205+0000 7fbe0e184640 1 -- 172.21.15.116:0/3635119131 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe08068350 con 0x7fbe080ac830 2024-04-25T00:24:01.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.209+0000 7fbdfffff640 1 -- 172.21.15.116:0/3635119131 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbe0001c070 con 0x7fbe080ac830 2024-04-25T00:24:01.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.209+0000 7fbdfffff640 1 --2- 172.21.15.116:0/3635119131 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbde8067990 0x7fbde8069e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:01.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.209+0000 7fbdfffff640 1 -- 172.21.15.116:0/3635119131 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 9762+0+0 (secure 0 0 0) 0x7fbe00086970 con 0x7fbe080ac830 2024-04-25T00:24:01.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.213+0000 7fbe06ffd640 1 --2- 172.21.15.116:0/3635119131 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbde8067990 0x7fbde8069e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:01.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.213+0000 7fbe06ffd640 1 --2- 172.21.15.116:0/3635119131 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbde8067990 0x7fbde8069e50 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fbdf00ad000 tx=0x7fbdf00a4f90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:01.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.221+0000 7fbdfffff640 1 -- 172.21.15.116:0/3635119131 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fbe00055ee0 con 0x7fbe080ac830 2024-04-25T00:24:01.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.717+0000 7fbe0e184640 1 -- 172.21.15.116:0/3635119131 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbe0815ec60 con 0x7fbe080ac830 2024-04-25T00:24:01.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.721+0000 7fbdfffff640 1 -- 172.21.15.116:0/3635119131 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 192 v192) ==== 95+0+6506 (secure 0 0 0) 0x7fbe0005a1e0 con 0x7fbe080ac830 2024-04-25T00:24:01.722 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:24:01.722 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":192,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":8343,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/3633657268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3633657268},{"type":"v1","addr":"172.21.15.116:6833","nonce":3633657268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":191,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:23:34.013510+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8244":{"gid":8244,"name":"c","rank":0,"incarnation":188,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6835/1971675962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1971675962},{"type":"v1","addr":"172.21.15.116:6835","nonce":1971675962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8244,"qdb_cluster":[8244]},"id":1},{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:22:38.936345+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6230":{"gid":6230,"name":"d","rank":0,"incarnation":178,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/492366509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":492366509},{"type":"v1","addr":"172.21.15.174:6837","nonce":492366509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6230,"qdb_cluster":[6230]},"id":2}]} 2024-04-25T00:24:01.722 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 192 2024-04-25T00:24:01.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.721+0000 7fbe0e184640 1 -- 172.21.15.116:0/3635119131 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbde8067990 msgr2=0x7fbde8069e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:01.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.721+0000 7fbe0e184640 1 --2- 172.21.15.116:0/3635119131 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbde8067990 0x7fbde8069e50 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fbdf00ad000 tx=0x7fbdf00a4f90 comp rx=0 tx=0).stop 2024-04-25T00:24:01.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.721+0000 7fbe0e184640 1 -- 172.21.15.116:0/3635119131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe080ac830 msgr2=0x7fbe080c5ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:01.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.721+0000 7fbe0e184640 1 --2- 172.21.15.116:0/3635119131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe080ac830 0x7fbe080c5ea0 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7fbe0000ba80 tx=0x7fbe0000bf40 comp rx=0 tx=0).stop 2024-04-25T00:24:01.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.721+0000 7fbe0e184640 1 -- 172.21.15.116:0/3635119131 shutdown_connections 2024-04-25T00:24:01.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.721+0000 7fbe0e184640 1 --2- 172.21.15.116:0/3635119131 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbde8067990 0x7fbde8069e50 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:01.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.721+0000 7fbe0e184640 1 --2- 172.21.15.116:0/3635119131 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe080c2a10 0x7fbe081430d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:01.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.721+0000 7fbe0e184640 1 --2- 172.21.15.116:0/3635119131 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe080c1280 0x7fbe08142b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:01.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.721+0000 7fbe0e184640 1 --2- 172.21.15.116:0/3635119131 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe080ac830 0x7fbe080c5ea0 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:01.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.721+0000 7fbe0e184640 1 -- 172.21.15.116:0/3635119131 >> 172.21.15.116:0/3635119131 conn(0x7fbe080c3b00 msgr2=0x7fbe080a9d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:01.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.725+0000 7fbe0e184640 1 -- 172.21.15.116:0/3635119131 shutdown_connections 2024-04-25T00:24:01.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:01.725+0000 7fbe0e184640 1 -- 172.21.15.116:0/3635119131 wait complete. 2024-04-25T00:24:01.767 INFO:tasks.mds_thrash.fs.[b]:kill mds.d (rank=0) 2024-04-25T00:24:01.768 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-25T00:24:01.768 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:24:01.780 INFO:tasks.ceph.mds.d:Stopped 2024-04-25T00:24:01.780 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:24:02.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.001+0000 7f299f1ec640 1 Processor -- start 2024-04-25T00:24:02.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.001+0000 7f299f1ec640 1 -- start start 2024-04-25T00:24:02.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.001+0000 7f299f1ec640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f299806e100 0x7f29980c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:02.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.001+0000 7f299f1ec640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2998154df0 con 0x7f299814b830 2024-04-25T00:24:02.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.001+0000 7f299f1ec640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29981550d0 con 0x7f29980c2f50 2024-04-25T00:24:02.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.001+0000 7f299f1ec640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f299814bc10 con 0x7f299806e100 2024-04-25T00:24:02.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.001+0000 7f299cf88640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f299814b830 0x7f2998152150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:59548/0 (socket says 172.21.15.116:59548) 2024-04-25T00:24:02.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.001+0000 7f299cf88640 1 -- 172.21.15.116:0/1648665016 learned_addr learned my addr 172.21.15.116:0/1648665016 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:24:02.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.001+0000 7f299d789640 1 --1- 172.21.15.116:0/1648665016 >> v1:172.21.15.174:6789/0 conn(0x7f29980c2f50 0x7f2998070dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:57482/0 (socket says 172.21.15.116:57482) 2024-04-25T00:24:02.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.005+0000 7f29977fe640 1 -- 172.21.15.116:0/1648665016 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2798489655 0 0) 0x7f29981550d0 con 0x7f29980c2f50 2024-04-25T00:24:02.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.005+0000 7f29977fe640 1 -- 172.21.15.116:0/1648665016 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2988003530 con 0x7f29980c2f50 2024-04-25T00:24:02.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.013+0000 7f29977fe640 1 -- 172.21.15.116:0/1648665016 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2125890866 0 0) 0x7f2998154df0 con 0x7f299814b830 2024-04-25T00:24:02.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.013+0000 7f29977fe640 1 -- 172.21.15.116:0/1648665016 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29981550d0 con 0x7f299814b830 2024-04-25T00:24:02.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.013+0000 7f2997fff640 1 --2- 172.21.15.116:0/1648665016 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f299806e100 0x7f29980c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:02.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.013+0000 7f2997fff640 1 -- 172.21.15.116:0/1648665016 >> v1:172.21.15.174:6789/0 conn(0x7f29980c2f50 legacy=0x7f2998070dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:02.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.013+0000 7f2997fff640 1 -- 172.21.15.116:0/1648665016 >> v1:172.21.15.116:6789/0 conn(0x7f299814b830 legacy=0x7f2998152150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:02.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.013+0000 7f2997fff640 1 -- 172.21.15.116:0/1648665016 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29981553b0 con 0x7f299806e100 2024-04-25T00:24:02.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.017+0000 7f2997fff640 1 --2- 172.21.15.116:0/1648665016 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f299806e100 0x7f29980c2a10 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f298c0097f0 tx=0x7f298c01c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8ef4a24a81fccb5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:02.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.017+0000 7f29977fe640 1 -- 172.21.15.116:0/1648665016 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f298c0b0070 con 0x7f299806e100 2024-04-25T00:24:02.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.017+0000 7f29977fe640 1 -- 172.21.15.116:0/1648665016 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f298c0ab600 con 0x7f299806e100 2024-04-25T00:24:02.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.017+0000 7f29977fe640 1 -- 172.21.15.116:0/1648665016 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f298c0b0070 con 0x7f299806e100 2024-04-25T00:24:02.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.017+0000 7f299f1ec640 1 -- 172.21.15.116:0/1648665016 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f299806e100 msgr2=0x7f29980c2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:02.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.017+0000 7f299f1ec640 1 --2- 172.21.15.116:0/1648665016 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f299806e100 0x7f29980c2a10 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f298c0097f0 tx=0x7f298c01c3a0 comp rx=0 tx=0).stop 2024-04-25T00:24:02.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.025+0000 7f299f1ec640 1 -- 172.21.15.116:0/1648665016 shutdown_connections 2024-04-25T00:24:02.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.025+0000 7f299f1ec640 1 --2- 172.21.15.116:0/1648665016 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f299806e100 0x7f29980c2a10 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:02.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.025+0000 7f299f1ec640 1 -- 172.21.15.116:0/1648665016 >> 172.21.15.116:0/1648665016 conn(0x7f29980c3b00 msgr2=0x7f29980c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:02.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.025+0000 7f299f1ec640 1 -- 172.21.15.116:0/1648665016 shutdown_connections 2024-04-25T00:24:02.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.029+0000 7f299f1ec640 1 -- 172.21.15.116:0/1648665016 wait complete. 2024-04-25T00:24:02.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.029+0000 7f299f1ec640 1 Processor -- start 2024-04-25T00:24:02.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.029+0000 7f299f1ec640 1 -- start start 2024-04-25T00:24:02.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.033+0000 7f299f1ec640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f299806e100 0x7f2998150c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:02.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.033+0000 7f299f1ec640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f29980c2f50 0x7f29981ee480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:02.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.033+0000 7f299f1ec640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f299814b830 0x7f29981eb970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:02.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.033+0000 7f299f1ec640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f299815fc10 con 0x7f299814b830 2024-04-25T00:24:02.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.033+0000 7f299f1ec640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f299815fa90 con 0x7f29980c2f50 2024-04-25T00:24:02.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.033+0000 7f299f1ec640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f299815fd90 con 0x7f299806e100 2024-04-25T00:24:02.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.033+0000 7f299d789640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f299814b830 0x7f29981eb970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:02.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.033+0000 7f299cf88640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f299806e100 0x7f2998150c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:02.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.033+0000 7f299d789640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f299814b830 0x7f29981eb970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54290/0 (socket says 172.21.15.116:54290) 2024-04-25T00:24:02.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.033+0000 7f299d789640 1 -- 172.21.15.116:0/2780349375 learned_addr learned my addr 172.21.15.116:0/2780349375 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:24:02.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.033+0000 7f299cf88640 1 -- 172.21.15.116:0/2780349375 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f29980c2f50 msgr2=0x7f29981ee480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:02.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.033+0000 7f299cf88640 1 --2- 172.21.15.116:0/2780349375 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f29980c2f50 0x7f29981ee480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:02.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.033+0000 7f299cf88640 1 -- 172.21.15.116:0/2780349375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f299814b830 msgr2=0x7f29981eb970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:02.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.033+0000 7f299cf88640 1 --2- 172.21.15.116:0/2780349375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f299814b830 0x7f29981eb970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:02.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.033+0000 7f299cf88640 1 -- 172.21.15.116:0/2780349375 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2998068b60 con 0x7f299806e100 2024-04-25T00:24:02.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.033+0000 7f299cf88640 1 --2- 172.21.15.116:0/2780349375 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f299806e100 0x7f2998150c60 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2984001e60 tx=0x7f298400c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:02.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.033+0000 7f29957fa640 1 -- 172.21.15.116:0/2780349375 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29840074e0 con 0x7f299806e100 2024-04-25T00:24:02.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.033+0000 7f29957fa640 1 -- 172.21.15.116:0/2780349375 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29840047f0 con 0x7f299806e100 2024-04-25T00:24:02.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.033+0000 7f299f1ec640 1 -- 172.21.15.116:0/2780349375 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29981511a0 con 0x7f299806e100 2024-04-25T00:24:02.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.033+0000 7f29957fa640 1 -- 172.21.15.116:0/2780349375 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2984007680 con 0x7f299806e100 2024-04-25T00:24:02.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.033+0000 7f299f1ec640 1 -- 172.21.15.116:0/2780349375 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2998151b20 con 0x7f299806e100 2024-04-25T00:24:02.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.033+0000 7f29957fa640 1 -- 172.21.15.116:0/2780349375 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2984004c80 con 0x7f299806e100 2024-04-25T00:24:02.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.037+0000 7f29957fa640 1 --2- 172.21.15.116:0/2780349375 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f297c0678c0 0x7f297c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:02.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.037+0000 7f29957fa640 1 -- 172.21.15.116:0/2780349375 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(64..64 src has 1..64) ==== 9762+0+0 (secure 0 0 0) 0x7f298408c4f0 con 0x7f299806e100 2024-04-25T00:24:02.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.037+0000 7f2997fff640 1 --2- 172.21.15.116:0/2780349375 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f297c0678c0 0x7f297c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:02.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.037+0000 7f299f1ec640 1 -- 172.21.15.116:0/2780349375 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2998068350 con 0x7f299806e100 2024-04-25T00:24:02.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.037+0000 7f2997fff640 1 --2- 172.21.15.116:0/2780349375 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f297c0678c0 0x7f297c069d80 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f298c0ad000 tx=0x7f298c004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:02.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.045+0000 7f29957fa640 1 -- 172.21.15.116:0/2780349375 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f2984059010 con 0x7f299806e100 2024-04-25T00:24:02.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.645+0000 7f299f1ec640 1 -- 172.21.15.116:0/2780349375 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f299806d660 con 0x7f299806e100 2024-04-25T00:24:02.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.649+0000 7f29957fa640 1 -- 172.21.15.116:0/2780349375 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 192 v192) ==== 95+0+6506 (secure 0 0 0) 0x7f298405d310 con 0x7f299806e100 2024-04-25T00:24:02.651 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:24:02.651 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":192,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":8343,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/3633657268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3633657268},{"type":"v1","addr":"172.21.15.116:6833","nonce":3633657268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":191,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:23:34.013510+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8244":{"gid":8244,"name":"c","rank":0,"incarnation":188,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6835/1971675962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1971675962},{"type":"v1","addr":"172.21.15.116:6835","nonce":1971675962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8244,"qdb_cluster":[8244]},"id":1},{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:22:38.936345+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6230":{"gid":6230,"name":"d","rank":0,"incarnation":178,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/492366509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":492366509},{"type":"v1","addr":"172.21.15.174:6837","nonce":492366509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6230,"qdb_cluster":[6230]},"id":2}]} 2024-04-25T00:24:02.651 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 192 2024-04-25T00:24:02.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.657+0000 7f299f1ec640 1 -- 172.21.15.116:0/2780349375 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f297c0678c0 msgr2=0x7f297c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:02.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.657+0000 7f299f1ec640 1 --2- 172.21.15.116:0/2780349375 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f297c0678c0 0x7f297c069d80 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f298c0ad000 tx=0x7f298c004740 comp rx=0 tx=0).stop 2024-04-25T00:24:02.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.657+0000 7f299f1ec640 1 -- 172.21.15.116:0/2780349375 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f299806e100 msgr2=0x7f2998150c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:02.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.657+0000 7f299f1ec640 1 --2- 172.21.15.116:0/2780349375 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f299806e100 0x7f2998150c60 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2984001e60 tx=0x7f298400c8d0 comp rx=0 tx=0).stop 2024-04-25T00:24:02.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.657+0000 7f299f1ec640 1 -- 172.21.15.116:0/2780349375 shutdown_connections 2024-04-25T00:24:02.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.657+0000 7f299f1ec640 1 --2- 172.21.15.116:0/2780349375 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f297c0678c0 0x7f297c069d80 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:02.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.657+0000 7f299f1ec640 1 --2- 172.21.15.116:0/2780349375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f299814b830 0x7f29981eb970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:02.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.657+0000 7f299f1ec640 1 --2- 172.21.15.116:0/2780349375 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f29980c2f50 0x7f29981ee480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:02.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.657+0000 7f299f1ec640 1 --2- 172.21.15.116:0/2780349375 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f299806e100 0x7f2998150c60 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:02.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.657+0000 7f299f1ec640 1 -- 172.21.15.116:0/2780349375 >> 172.21.15.116:0/2780349375 conn(0x7f29980c3b00 msgr2=0x7f29981529a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:02.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.657+0000 7f299f1ec640 1 -- 172.21.15.116:0/2780349375 shutdown_connections 2024-04-25T00:24:02.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:02.657+0000 7f299f1ec640 1 -- 172.21.15.116:0/2780349375 wait complete. 2024-04-25T00:24:02.691 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:24:04.692 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:24:04.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:04.977+0000 7fbe425a2640 1 Processor -- start 2024-04-25T00:24:04.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:04.981+0000 7fbe425a2640 1 -- start start 2024-04-25T00:24:04.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:04.981+0000 7fbe425a2640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbe3c14b7a0 0x7fbe3c152210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:04.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:04.981+0000 7fbe425a2640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe3c06e270 0x7fbe3c0c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:04.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:04.981+0000 7fbe425a2640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe3c152750 con 0x7fbe3c0c2f50 2024-04-25T00:24:04.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:04.981+0000 7fbe425a2640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fbe3c06cd60 con 0x7fbe3c14b7a0 2024-04-25T00:24:04.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:04.981+0000 7fbe3bfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbe3c14b7a0 0x7fbe3c152210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:04.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:04.981+0000 7fbe33fff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe3c06e270 0x7fbe3c0c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:04.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:04.981+0000 7fbe3bfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbe3c14b7a0 0x7fbe3c152210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51386/0 (socket says 172.21.15.116:51386) 2024-04-25T00:24:04.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:04.981+0000 7fbe33fff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe3c06e270 0x7fbe3c0c2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:40766/0 (socket says 172.21.15.116:40766) 2024-04-25T00:24:04.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:04.981+0000 7fbe3bfff640 1 -- 172.21.15.116:0/957906274 learned_addr learned my addr 172.21.15.116:0/957906274 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:24:04.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:04.981+0000 7fbe425a2640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fbe3c06d040 con 0x7fbe3c06e270 2024-04-25T00:24:04.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:04.981+0000 7fbe3bfff640 1 -- 172.21.15.116:0/957906274 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe3c06e270 msgr2=0x7fbe3c0c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:04.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:04.981+0000 7fbe3bfff640 1 --2- 172.21.15.116:0/957906274 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe3c06e270 0x7fbe3c0c2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:04.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:04.981+0000 7fbe3bfff640 1 -- 172.21.15.116:0/957906274 >> v1:172.21.15.116:6789/0 conn(0x7fbe3c0c2f50 legacy=0x7fbe3c070e20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T00:24:04.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:04.981+0000 7fbe3bfff640 1 -- 172.21.15.116:0/957906274 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe3c06d320 con 0x7fbe3c14b7a0 2024-04-25T00:24:04.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:04.981+0000 7fbe33fff640 1 --2- 172.21.15.116:0/957906274 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe3c06e270 0x7fbe3c0c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:24:04.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:04.985+0000 7fbe3bfff640 1 --2- 172.21.15.116:0/957906274 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe3c14b7a0 0x7fbe3c152210 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7fbe24009ad0 tx=0x7fbe2401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f90d03afa6a9f27a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:04.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:04.985+0000 7fbe3b7fe640 1 -- 172.21.15.116:0/957906274 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe240b0070 con 0x7fbe3c14b7a0 2024-04-25T00:24:04.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:04.985+0000 7fbe3b7fe640 1 -- 172.21.15.116:0/957906274 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe240ab600 con 0x7fbe3c14b7a0 2024-04-25T00:24:04.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:04.985+0000 7fbe3b7fe640 1 -- 172.21.15.116:0/957906274 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe240ab9d0 con 0x7fbe3c14b7a0 2024-04-25T00:24:04.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:04.985+0000 7fbe425a2640 1 -- 172.21.15.116:0/957906274 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe3c14b7a0 msgr2=0x7fbe3c152210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:04.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:04.985+0000 7fbe425a2640 1 --2- 172.21.15.116:0/957906274 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe3c14b7a0 0x7fbe3c152210 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7fbe24009ad0 tx=0x7fbe2401c480 comp rx=0 tx=0).stop 2024-04-25T00:24:05.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:05.009+0000 7fbe425a2640 1 -- 172.21.15.116:0/957906274 shutdown_connections 2024-04-25T00:24:05.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:05.009+0000 7fbe425a2640 1 --2- 172.21.15.116:0/957906274 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe3c06e270 0x7fbe3c0c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:05.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:05.009+0000 7fbe425a2640 1 --2- 172.21.15.116:0/957906274 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe3c14b7a0 0x7fbe3c152210 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:05.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:05.009+0000 7fbe425a2640 1 -- 172.21.15.116:0/957906274 >> 172.21.15.116:0/957906274 conn(0x7fbe3c0c3b00 msgr2=0x7fbe3c0c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:05.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:05.009+0000 7fbe425a2640 1 -- 172.21.15.116:0/957906274 shutdown_connections 2024-04-25T00:24:05.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:05.009+0000 7fbe425a2640 1 -- 172.21.15.116:0/957906274 wait complete. 2024-04-25T00:24:05.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:05.009+0000 7fbe425a2640 1 Processor -- start 2024-04-25T00:24:05.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:05.009+0000 7fbe425a2640 1 -- start start 2024-04-25T00:24:05.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:05.021+0000 7fbe425a2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe3c06e270 0x7fbe3c0c5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:05.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:05.021+0000 7fbe3bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe3c06e270 0x7fbe3c0c5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:05.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:05.021+0000 7fbe3bfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe3c06e270 0x7fbe3c0c5ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51402/0 (socket says 172.21.15.116:51402) 2024-04-25T00:24:05.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:05.021+0000 7fbe3bfff640 1 -- 172.21.15.116:0/941295365 learned_addr learned my addr 172.21.15.116:0/941295365 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:24:05.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:05.021+0000 7fbe425a2640 1 --2- 172.21.15.116:0/941295365 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe3c0c2f50 0x7fbe3c143790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:05.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:05.021+0000 7fbe425a2640 1 --2- 172.21.15.116:0/941295365 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe3c14b7a0 0x7fbe3c143cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:05.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:05.021+0000 7fbe33fff640 1 --2- 172.21.15.116:0/941295365 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe3c0c2f50 0x7fbe3c143790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:05.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:05.021+0000 7fbe425a2640 1 -- 172.21.15.116:0/941295365 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbe3c15eab0 con 0x7fbe3c06e270 2024-04-25T00:24:05.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:05.021+0000 7fbe425a2640 1 -- 172.21.15.116:0/941295365 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbe3c15e930 con 0x7fbe3c0c2f50 2024-04-25T00:24:05.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:05.021+0000 7fbe425a2640 1 -- 172.21.15.116:0/941295365 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fbe3c15ec30 con 0x7fbe3c14b7a0 2024-04-25T00:24:05.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:05.025+0000 7fbe3bfff640 1 -- 172.21.15.116:0/941295365 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe3c14b7a0 msgr2=0x7fbe3c143cd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T00:24:05.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:05.025+0000 7fbe3bfff640 1 --2- 172.21.15.116:0/941295365 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe3c14b7a0 0x7fbe3c143cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:05.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:05.025+0000 7fbe3bfff640 1 -- 172.21.15.116:0/941295365 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe3c0c2f50 msgr2=0x7fbe3c143790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:05.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:05.025+0000 7fbe3bfff640 1 --2- 172.21.15.116:0/941295365 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe3c0c2f50 0x7fbe3c143790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:05.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:05.025+0000 7fbe3bfff640 1 -- 172.21.15.116:0/941295365 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe3c1532a0 con 0x7fbe3c06e270 2024-04-25T00:24:05.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:05.025+0000 7fbe3bfff640 1 --2- 172.21.15.116:0/941295365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe3c06e270 0x7fbe3c0c5ea0 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7fbe24009ad0 tx=0x7fbe24019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:05.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:05.025+0000 7fbe397fa640 1 -- 172.21.15.116:0/941295365 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe240b0040 con 0x7fbe3c06e270 2024-04-25T00:24:05.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:05.025+0000 7fbe397fa640 1 -- 172.21.15.116:0/941295365 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe240bc070 con 0x7fbe3c06e270 2024-04-25T00:24:05.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:05.025+0000 7fbe397fa640 1 -- 172.21.15.116:0/941295365 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe240b8df0 con 0x7fbe3c06e270 2024-04-25T00:24:05.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:05.025+0000 7fbe425a2640 1 -- 172.21.15.116:0/941295365 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe3c153560 con 0x7fbe3c06e270 2024-04-25T00:24:05.027 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:05.025+0000 7fbe425a2640 1 -- 172.21.15.116:0/941295365 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe3c152d80 con 0x7fbe3c06e270 2024-04-25T00:24:05.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:05.025+0000 7fbe327fc640 1 -- 172.21.15.116:0/941295365 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe140052a0 con 0x7fbe3c06e270 2024-04-25T00:24:05.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:05.029+0000 7fbe397fa640 1 -- 172.21.15.116:0/941295365 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbe24003620 con 0x7fbe3c06e270 2024-04-25T00:24:05.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:05.029+0000 7fbe397fa640 1 --2- 172.21.15.116:0/941295365 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbe1c0678c0 0x7fbe1c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:05.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:05.029+0000 7fbe397fa640 1 -- 172.21.15.116:0/941295365 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 9762+0+0 (secure 0 0 0) 0x7fbe24128f40 con 0x7fbe3c06e270 2024-04-25T00:24:05.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:05.029+0000 7fbe33fff640 1 --2- 172.21.15.116:0/941295365 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbe1c0678c0 0x7fbe1c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:05.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:05.033+0000 7fbe33fff640 1 --2- 172.21.15.116:0/941295365 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbe1c0678c0 0x7fbe1c069d80 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c0c1830 tx=0x7fbe2c00a2a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:05.040 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:05.037+0000 7fbe397fa640 1 -- 172.21.15.116:0/941295365 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fbe3c152750 con 0x7fbe3c06e270 2024-04-25T00:24:05.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:05.501+0000 7fbe327fc640 1 -- 172.21.15.116:0/941295365 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbe14005590 con 0x7fbe3c06e270 2024-04-25T00:24:05.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:05.505+0000 7fbe397fa640 1 -- 172.21.15.116:0/941295365 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 192 v192) ==== 95+0+6506 (secure 0 0 0) 0x7fbe240f8a80 con 0x7fbe3c06e270 2024-04-25T00:24:05.509 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:24:05.509 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":192,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":8343,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/3633657268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3633657268},{"type":"v1","addr":"172.21.15.116:6833","nonce":3633657268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":191,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:23:34.013510+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8244":{"gid":8244,"name":"c","rank":0,"incarnation":188,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6835/1971675962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1971675962},{"type":"v1","addr":"172.21.15.116:6835","nonce":1971675962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8244,"qdb_cluster":[8244]},"id":1},{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:22:38.936345+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6230":{"gid":6230,"name":"d","rank":0,"incarnation":178,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/492366509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":492366509},{"type":"v1","addr":"172.21.15.174:6837","nonce":492366509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6230,"qdb_cluster":[6230]},"id":2}]} 2024-04-25T00:24:05.509 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 192 2024-04-25T00:24:05.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:05.509+0000 7fbe327fc640 1 -- 172.21.15.116:0/941295365 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbe1c0678c0 msgr2=0x7fbe1c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:05.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:05.509+0000 7fbe327fc640 1 --2- 172.21.15.116:0/941295365 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbe1c0678c0 0x7fbe1c069d80 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fbe3c0c1830 tx=0x7fbe2c00a2a0 comp rx=0 tx=0).stop 2024-04-25T00:24:05.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:05.509+0000 7fbe327fc640 1 -- 172.21.15.116:0/941295365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe3c06e270 msgr2=0x7fbe3c0c5ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:05.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:05.509+0000 7fbe327fc640 1 --2- 172.21.15.116:0/941295365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe3c06e270 0x7fbe3c0c5ea0 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7fbe24009ad0 tx=0x7fbe24019220 comp rx=0 tx=0).stop 2024-04-25T00:24:05.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:05.509+0000 7fbe327fc640 1 -- 172.21.15.116:0/941295365 shutdown_connections 2024-04-25T00:24:05.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:05.509+0000 7fbe327fc640 1 --2- 172.21.15.116:0/941295365 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbe1c0678c0 0x7fbe1c069d80 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:05.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:05.509+0000 7fbe327fc640 1 --2- 172.21.15.116:0/941295365 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbe3c14b7a0 0x7fbe3c143cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:05.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:05.509+0000 7fbe327fc640 1 --2- 172.21.15.116:0/941295365 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbe3c0c2f50 0x7fbe3c143790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:05.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:05.509+0000 7fbe327fc640 1 --2- 172.21.15.116:0/941295365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbe3c06e270 0x7fbe3c0c5ea0 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:05.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:05.509+0000 7fbe327fc640 1 -- 172.21.15.116:0/941295365 >> 172.21.15.116:0/941295365 conn(0x7fbe3c0c3b00 msgr2=0x7fbe3c0c3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:05.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:05.509+0000 7fbe327fc640 1 -- 172.21.15.116:0/941295365 shutdown_connections 2024-04-25T00:24:05.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:05.509+0000 7fbe327fc640 1 -- 172.21.15.116:0/941295365 wait complete. 2024-04-25T00:24:05.547 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:24:07.548 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:24:07.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.833+0000 7ffa29a64640 1 Processor -- start 2024-04-25T00:24:07.835 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.833+0000 7ffa29a64640 1 -- start start 2024-04-25T00:24:07.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.833+0000 7ffa29a64640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffa24155810 0x7ffa24157390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:07.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.833+0000 7ffa29a64640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa240c9150 con 0x7ffa2406d5f0 2024-04-25T00:24:07.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.833+0000 7ffa29a64640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa240c9430 con 0x7ffa24152d80 2024-04-25T00:24:07.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.833+0000 7ffa29a64640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ffa240c9d60 con 0x7ffa24155810 2024-04-25T00:24:07.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.833+0000 7ffa22ffd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7ffa2406d5f0 0x7ffa24156c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:38454/0 (socket says 172.21.15.116:38454) 2024-04-25T00:24:07.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.833+0000 7ffa22ffd640 1 -- 172.21.15.116:0/890700827 learned_addr learned my addr 172.21.15.116:0/890700827 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:24:07.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.833+0000 7ffa23fff640 1 --2- 172.21.15.116:0/890700827 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffa24155810 0x7ffa24157390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:07.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.837+0000 7ffa227fc640 1 -- 172.21.15.116:0/890700827 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2945891729 0 0) 0x7ffa240c9150 con 0x7ffa2406d5f0 2024-04-25T00:24:07.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.837+0000 7ffa227fc640 1 -- 172.21.15.116:0/890700827 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffa0c003530 con 0x7ffa2406d5f0 2024-04-25T00:24:07.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.837+0000 7ffa227fc640 1 -- 172.21.15.116:0/890700827 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 439716887 0 0) 0x7ffa0c003530 con 0x7ffa2406d5f0 2024-04-25T00:24:07.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.837+0000 7ffa227fc640 1 -- 172.21.15.116:0/890700827 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffa24155810 msgr2=0x7ffa24157390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:07.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.837+0000 7ffa227fc640 1 --2- 172.21.15.116:0/890700827 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffa24155810 0x7ffa24157390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:07.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.837+0000 7ffa227fc640 1 -- 172.21.15.116:0/890700827 >> v1:172.21.15.174:6789/0 conn(0x7ffa24152d80 legacy=0x7ffa2406bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:07.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.837+0000 7ffa227fc640 1 -- 172.21.15.116:0/890700827 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa240c9710 con 0x7ffa2406d5f0 2024-04-25T00:24:07.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.837+0000 7ffa227fc640 1 -- 172.21.15.116:0/890700827 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7ffa18001680 con 0x7ffa2406d5f0 2024-04-25T00:24:07.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.837+0000 7ffa227fc640 1 -- 172.21.15.116:0/890700827 >> v1:172.21.15.116:6789/0 conn(0x7ffa2406d5f0 legacy=0x7ffa24156c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:07.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.837+0000 7ffa227fc640 1 --2- 172.21.15.116:0/890700827 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa0c0039f0 0x7ffa0c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:07.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.837+0000 7ffa227fc640 1 --2- 172.21.15.116:0/890700827 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffa0c004810 0x7ffa0c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:07.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.837+0000 7ffa227fc640 1 --2- 172.21.15.116:0/890700827 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa0c0051d0 0x7ffa0c01c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:07.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.837+0000 7ffa227fc640 1 -- 172.21.15.116:0/890700827 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ffa240c9d60 con 0x7ffa0c0039f0 2024-04-25T00:24:07.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.837+0000 7ffa227fc640 1 -- 172.21.15.116:0/890700827 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ffa0c01d0e0 con 0x7ffa0c0051d0 2024-04-25T00:24:07.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.837+0000 7ffa227fc640 1 -- 172.21.15.116:0/890700827 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ffa0c01d2e0 con 0x7ffa0c004810 2024-04-25T00:24:07.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.837+0000 7ffa23fff640 1 --2- 172.21.15.116:0/890700827 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa0c0051d0 0x7ffa0c01c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:07.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.837+0000 7ffa22ffd640 1 --2- 172.21.15.116:0/890700827 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffa0c004810 0x7ffa0c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:07.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.837+0000 7ffa237fe640 1 --2- 172.21.15.116:0/890700827 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa0c0039f0 0x7ffa0c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:07.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.837+0000 7ffa237fe640 1 -- 172.21.15.116:0/890700827 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffa0c004810 msgr2=0x7ffa0c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:07.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.837+0000 7ffa237fe640 1 --2- 172.21.15.116:0/890700827 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffa0c004810 0x7ffa0c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:07.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.837+0000 7ffa237fe640 1 -- 172.21.15.116:0/890700827 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa0c0051d0 msgr2=0x7ffa0c01c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:07.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.837+0000 7ffa237fe640 1 --2- 172.21.15.116:0/890700827 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa0c0051d0 0x7ffa0c01c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:07.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.837+0000 7ffa237fe640 1 -- 172.21.15.116:0/890700827 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa0c003530 con 0x7ffa0c0039f0 2024-04-25T00:24:07.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.845+0000 7ffa237fe640 1 --2- 172.21.15.116:0/890700827 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa0c0039f0 0x7ffa0c003e10 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7ffa140019c0 tx=0x7ffa1401c630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=631ef4877e15b396 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:07.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.845+0000 7ffa227fc640 1 -- 172.21.15.116:0/890700827 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa140b0070 con 0x7ffa0c0039f0 2024-04-25T00:24:07.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.845+0000 7ffa227fc640 1 -- 172.21.15.116:0/890700827 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa140a7070 con 0x7ffa0c0039f0 2024-04-25T00:24:07.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.845+0000 7ffa227fc640 1 -- 172.21.15.116:0/890700827 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa14001380 con 0x7ffa0c0039f0 2024-04-25T00:24:07.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.849+0000 7ffa29a64640 1 -- 172.21.15.116:0/890700827 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa0c0039f0 msgr2=0x7ffa0c003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:07.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.849+0000 7ffa29a64640 1 --2- 172.21.15.116:0/890700827 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa0c0039f0 0x7ffa0c003e10 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7ffa140019c0 tx=0x7ffa1401c630 comp rx=0 tx=0).stop 2024-04-25T00:24:07.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.849+0000 7ffa23fff640 1 -- 172.21.15.116:0/890700827 reap_dead start 2024-04-25T00:24:07.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.849+0000 7ffa29a64640 1 -- 172.21.15.116:0/890700827 shutdown_connections 2024-04-25T00:24:07.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.849+0000 7ffa29a64640 1 -- 172.21.15.116:0/890700827 >> 172.21.15.116:0/890700827 conn(0x7ffa240acdd0 msgr2=0x7ffa240ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:07.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.849+0000 7ffa29a64640 1 -- 172.21.15.116:0/890700827 shutdown_connections 2024-04-25T00:24:07.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.849+0000 7ffa29a64640 1 -- 172.21.15.116:0/890700827 wait complete. 2024-04-25T00:24:07.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.853+0000 7ffa29a64640 1 Processor -- start 2024-04-25T00:24:07.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.853+0000 7ffa29a64640 1 -- start start 2024-04-25T00:24:07.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.853+0000 7ffa29a64640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffa241ed890 0x7ffa2414c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:07.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.853+0000 7ffa29a64640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa241546c0 0x7ffa241edc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:07.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.853+0000 7ffa29a64640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa240b0cc0 0x7ffa240adcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:07.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.853+0000 7ffa29a64640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ffa240caf00 con 0x7ffa240b0cc0 2024-04-25T00:24:07.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.853+0000 7ffa29a64640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ffa240cad80 con 0x7ffa241546c0 2024-04-25T00:24:07.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.853+0000 7ffa29a64640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ffa241577a0 con 0x7ffa241ed890 2024-04-25T00:24:07.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.853+0000 7ffa23fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa240b0cc0 0x7ffa240adcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:07.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.853+0000 7ffa23fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa240b0cc0 0x7ffa240adcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51424/0 (socket says 172.21.15.116:51424) 2024-04-25T00:24:07.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.853+0000 7ffa23fff640 1 -- 172.21.15.116:0/2357725713 learned_addr learned my addr 172.21.15.116:0/2357725713 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:24:07.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.853+0000 7ffa237fe640 1 --2- 172.21.15.116:0/2357725713 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffa241ed890 0x7ffa2414c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:07.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.853+0000 7ffa23fff640 1 -- 172.21.15.116:0/2357725713 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffa241ed890 msgr2=0x7ffa2414c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:07.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.853+0000 7ffa23fff640 1 --2- 172.21.15.116:0/2357725713 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffa241ed890 0x7ffa2414c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:07.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.853+0000 7ffa23fff640 1 -- 172.21.15.116:0/2357725713 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa241546c0 msgr2=0x7ffa241edc70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T00:24:07.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.853+0000 7ffa23fff640 1 --2- 172.21.15.116:0/2357725713 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa241546c0 0x7ffa241edc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:07.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.853+0000 7ffa23fff640 1 -- 172.21.15.116:0/2357725713 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa241ee370 con 0x7ffa240b0cc0 2024-04-25T00:24:07.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.853+0000 7ffa23fff640 1 --2- 172.21.15.116:0/2357725713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa240b0cc0 0x7ffa240adcf0 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c009f20 tx=0x7ffa1c031590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:07.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.853+0000 7ffa0bfff640 1 -- 172.21.15.116:0/2357725713 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c003ba0 con 0x7ffa240b0cc0 2024-04-25T00:24:07.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.853+0000 7ffa0bfff640 1 -- 172.21.15.116:0/2357725713 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa1c03e070 con 0x7ffa240b0cc0 2024-04-25T00:24:07.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.853+0000 7ffa0bfff640 1 -- 172.21.15.116:0/2357725713 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c031e70 con 0x7ffa240b0cc0 2024-04-25T00:24:07.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.853+0000 7ffa29a64640 1 -- 172.21.15.116:0/2357725713 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa24155240 con 0x7ffa240b0cc0 2024-04-25T00:24:07.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.853+0000 7ffa29a64640 1 -- 172.21.15.116:0/2357725713 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa240ae230 con 0x7ffa240b0cc0 2024-04-25T00:24:07.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.857+0000 7ffa0bfff640 1 -- 172.21.15.116:0/2357725713 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ffa1c004030 con 0x7ffa240b0cc0 2024-04-25T00:24:07.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.857+0000 7ffa0bfff640 1 --2- 172.21.15.116:0/2357725713 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ffa100678f0 0x7ffa10069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:07.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.857+0000 7ffa0bfff640 1 -- 172.21.15.116:0/2357725713 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 9762+0+0 (secure 0 0 0) 0x7ffa1c0a9820 con 0x7ffa240b0cc0 2024-04-25T00:24:07.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.865+0000 7ffa29a64640 1 -- 172.21.15.116:0/2357725713 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa241ee540 con 0x7ffa240b0cc0 2024-04-25T00:24:07.869 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.869+0000 7ffa237fe640 1 --2- 172.21.15.116:0/2357725713 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ffa100678f0 0x7ffa10069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:07.874 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.873+0000 7ffa237fe640 1 --2- 172.21.15.116:0/2357725713 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ffa100678f0 0x7ffa10069db0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7ffa14001990 tx=0x7ffa14009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:07.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:07.873+0000 7ffa0bfff640 1 -- 172.21.15.116:0/2357725713 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ffa241ee540 con 0x7ffa240b0cc0 2024-04-25T00:24:08.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:08.293+0000 7ffa29a64640 1 -- 172.21.15.116:0/2357725713 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffa240ae550 con 0x7ffa240b0cc0 2024-04-25T00:24:08.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:08.297+0000 7ffa0bfff640 1 -- 172.21.15.116:0/2357725713 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 192 v192) ==== 95+0+6506 (secure 0 0 0) 0x7ffa1c0792d0 con 0x7ffa240b0cc0 2024-04-25T00:24:08.298 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:24:08.298 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":192,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":8343,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/3633657268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3633657268},{"type":"v1","addr":"172.21.15.116:6833","nonce":3633657268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":191,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:23:34.013510+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8244":{"gid":8244,"name":"c","rank":0,"incarnation":188,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6835/1971675962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1971675962},{"type":"v1","addr":"172.21.15.116:6835","nonce":1971675962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8244,"qdb_cluster":[8244]},"id":1},{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:22:38.936345+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6230":{"gid":6230,"name":"d","rank":0,"incarnation":178,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/492366509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":492366509},{"type":"v1","addr":"172.21.15.174:6837","nonce":492366509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6230,"qdb_cluster":[6230]},"id":2}]} 2024-04-25T00:24:08.299 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 192 2024-04-25T00:24:08.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:08.301+0000 7ffa29a64640 1 -- 172.21.15.116:0/2357725713 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ffa100678f0 msgr2=0x7ffa10069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:08.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:08.301+0000 7ffa29a64640 1 --2- 172.21.15.116:0/2357725713 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ffa100678f0 0x7ffa10069db0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7ffa14001990 tx=0x7ffa14009040 comp rx=0 tx=0).stop 2024-04-25T00:24:08.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:08.301+0000 7ffa29a64640 1 -- 172.21.15.116:0/2357725713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa240b0cc0 msgr2=0x7ffa240adcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:08.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:08.301+0000 7ffa29a64640 1 --2- 172.21.15.116:0/2357725713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa240b0cc0 0x7ffa240adcf0 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c009f20 tx=0x7ffa1c031590 comp rx=0 tx=0).stop 2024-04-25T00:24:08.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:08.301+0000 7ffa29a64640 1 -- 172.21.15.116:0/2357725713 shutdown_connections 2024-04-25T00:24:08.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:08.301+0000 7ffa29a64640 1 --2- 172.21.15.116:0/2357725713 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ffa100678f0 0x7ffa10069db0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:08.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:08.301+0000 7ffa29a64640 1 --2- 172.21.15.116:0/2357725713 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffa240b0cc0 0x7ffa240adcf0 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:08.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:08.301+0000 7ffa29a64640 1 --2- 172.21.15.116:0/2357725713 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffa241546c0 0x7ffa241edc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:08.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:08.301+0000 7ffa29a64640 1 --2- 172.21.15.116:0/2357725713 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffa241ed890 0x7ffa2414c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:08.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:08.301+0000 7ffa29a64640 1 -- 172.21.15.116:0/2357725713 >> 172.21.15.116:0/2357725713 conn(0x7ffa240acdd0 msgr2=0x7ffa24071c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:08.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:08.301+0000 7ffa29a64640 1 -- 172.21.15.116:0/2357725713 shutdown_connections 2024-04-25T00:24:08.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:08.301+0000 7ffa29a64640 1 -- 172.21.15.116:0/2357725713 wait complete. 2024-04-25T00:24:08.329 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:24:10.330 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:24:10.363 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:24:10.379 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:24:10.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.625+0000 7f7d6ffff640 1 Processor -- start 2024-04-25T00:24:10.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.625+0000 7f7d6ffff640 1 -- start start 2024-04-25T00:24:10.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.625+0000 7f7d6ffff640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f7d7014bf80 0x7f7d70159640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:10.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.625+0000 7f7d6ffff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7d70155fa0 0x7f7d700c8e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:10.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.625+0000 7f7d6ffff640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d700cbb70 con 0x7f7d700aca50 2024-04-25T00:24:10.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.625+0000 7f7d6ffff640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f7d700c61a0 con 0x7f7d70155fa0 2024-04-25T00:24:10.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.625+0000 7f7d6ffff640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f7d700c7740 con 0x7f7d7014bf80 2024-04-25T00:24:10.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.625+0000 7f7d6effd640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f7d7014bf80 0x7f7d70159640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:10.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.625+0000 7f7d6f7fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f7d700aca50 0x7f7d700afde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:38456/0 (socket says 172.21.15.116:38456) 2024-04-25T00:24:10.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.625+0000 7f7d6f7fe640 1 -- 172.21.15.116:0/316887266 learned_addr learned my addr 172.21.15.116:0/316887266 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:24:10.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.625+0000 7f7d6effd640 1 --2- 172.21.15.116:0/316887266 >> v2:172.21.15.174:3300/0 conn(0x7f7d7014bf80 0x7f7d70159640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:58476/0 (socket says 172.21.15.116:58476) 2024-04-25T00:24:10.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.625+0000 7f7d6e7fc640 1 --2- 172.21.15.116:0/316887266 >> v2:172.21.15.116:3300/0 conn(0x7f7d70155fa0 0x7f7d700c8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:10.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.629+0000 7f7d6e7fc640 1 -- 172.21.15.116:0/316887266 >> v2:172.21.15.174:3300/0 conn(0x7f7d7014bf80 msgr2=0x7f7d70159640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:10.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.629+0000 7f7d6e7fc640 1 --2- 172.21.15.116:0/316887266 >> v2:172.21.15.174:3300/0 conn(0x7f7d7014bf80 0x7f7d70159640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:10.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.629+0000 7f7d6e7fc640 1 -- 172.21.15.116:0/316887266 >> v1:172.21.15.116:6789/0 conn(0x7f7d700aca50 legacy=0x7f7d700afde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:10.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.629+0000 7f7d6e7fc640 1 -- 172.21.15.116:0/316887266 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d700c5be0 con 0x7f7d70155fa0 2024-04-25T00:24:10.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.629+0000 7f7d6dffb640 1 -- 172.21.15.116:0/316887266 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2122285935 0 0) 0x7f7d700cbb70 con 0x7f7d700aca50 2024-04-25T00:24:10.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.633+0000 7f7d6e7fc640 1 --2- 172.21.15.116:0/316887266 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d70155fa0 0x7f7d700c8e50 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f7d640097c0 tx=0x7f7d6401c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9f66f0e3c14062b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:10.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.633+0000 7f7d6dffb640 1 -- 172.21.15.116:0/316887266 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d640b0070 con 0x7f7d70155fa0 2024-04-25T00:24:10.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.633+0000 7f7d6dffb640 1 -- 172.21.15.116:0/316887266 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d640ab600 con 0x7f7d70155fa0 2024-04-25T00:24:10.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.633+0000 7f7d6dffb640 1 -- 172.21.15.116:0/316887266 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d640a4e60 con 0x7f7d70155fa0 2024-04-25T00:24:10.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.633+0000 7f7d6ffff640 1 -- 172.21.15.116:0/316887266 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d70155fa0 msgr2=0x7f7d700c8e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:10.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.633+0000 7f7d6ffff640 1 --2- 172.21.15.116:0/316887266 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d70155fa0 0x7f7d700c8e50 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f7d640097c0 tx=0x7f7d6401c3d0 comp rx=0 tx=0).stop 2024-04-25T00:24:10.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.633+0000 7f7d6ffff640 1 -- 172.21.15.116:0/316887266 shutdown_connections 2024-04-25T00:24:10.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.633+0000 7f7d6ffff640 1 --2- 172.21.15.116:0/316887266 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d70155fa0 0x7f7d700c8e50 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:10.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.633+0000 7f7d6ffff640 1 --2- 172.21.15.116:0/316887266 >> v2:172.21.15.174:3300/0 conn(0x7f7d7014bf80 0x7f7d70159640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:10.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.633+0000 7f7d6ffff640 1 -- 172.21.15.116:0/316887266 >> 172.21.15.116:0/316887266 conn(0x7f7d70074a80 msgr2=0x7f7d700caf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:10.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.637+0000 7f7d6ffff640 1 -- 172.21.15.116:0/316887266 shutdown_connections 2024-04-25T00:24:10.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.637+0000 7f7d6ffff640 1 -- 172.21.15.116:0/316887266 wait complete. 2024-04-25T00:24:10.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.637+0000 7f7d6ffff640 1 Processor -- start 2024-04-25T00:24:10.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.641+0000 7f7d6ffff640 1 -- start start 2024-04-25T00:24:10.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.641+0000 7f7d6ffff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d700aca50 0x7f7d7014b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:10.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.641+0000 7f7d6ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d7014bf80 0x7f7d701f04a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:10.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.641+0000 7f7d6ffff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d70155fa0 0x7f7d701f09e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:10.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.641+0000 7f7d6ffff640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7d700c58d0 con 0x7f7d7014bf80 2024-04-25T00:24:10.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.641+0000 7f7d6ffff640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7d700b0900 con 0x7f7d700aca50 2024-04-25T00:24:10.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.641+0000 7f7d6ffff640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f7d70156410 con 0x7f7d70155fa0 2024-04-25T00:24:10.643 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.641+0000 7f7d6f7fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d70155fa0 0x7f7d701f09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:10.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.641+0000 7f7d6f7fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d70155fa0 0x7f7d701f09e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:40822/0 (socket says 172.21.15.116:40822) 2024-04-25T00:24:10.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.641+0000 7f7d6f7fe640 1 -- 172.21.15.116:0/2082689518 learned_addr learned my addr 172.21.15.116:0/2082689518 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:24:10.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.641+0000 7f7d6effd640 1 --2- 172.21.15.116:0/2082689518 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d700aca50 0x7f7d7014b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:10.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.641+0000 7f7d6e7fc640 1 --2- 172.21.15.116:0/2082689518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d7014bf80 0x7f7d701f04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:10.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.641+0000 7f7d6f7fe640 1 -- 172.21.15.116:0/2082689518 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d700aca50 msgr2=0x7f7d7014b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:10.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.641+0000 7f7d6f7fe640 1 --2- 172.21.15.116:0/2082689518 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d700aca50 0x7f7d7014b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:10.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.641+0000 7f7d6f7fe640 1 -- 172.21.15.116:0/2082689518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d7014bf80 msgr2=0x7f7d701f04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:10.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.645+0000 7f7d6f7fe640 1 --2- 172.21.15.116:0/2082689518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d7014bf80 0x7f7d701f04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:10.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.645+0000 7f7d6f7fe640 1 -- 172.21.15.116:0/2082689518 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d700c4fa0 con 0x7f7d70155fa0 2024-04-25T00:24:10.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.645+0000 7f7d6e7fc640 1 --2- 172.21.15.116:0/2082689518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d7014bf80 0x7f7d701f04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:24:10.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.645+0000 7f7d6f7fe640 1 --2- 172.21.15.116:0/2082689518 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d70155fa0 0x7f7d701f09e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7d6800c7b0 tx=0x7f7d6800cc70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:10.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.645+0000 7f7d577fe640 1 -- 172.21.15.116:0/2082689518 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d68018070 con 0x7f7d70155fa0 2024-04-25T00:24:10.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.645+0000 7f7d577fe640 1 -- 172.21.15.116:0/2082689518 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d6800a640 con 0x7f7d70155fa0 2024-04-25T00:24:10.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.645+0000 7f7d577fe640 1 -- 172.21.15.116:0/2082689518 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d6800aa30 con 0x7f7d70155fa0 2024-04-25T00:24:10.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.645+0000 7f7d6ffff640 1 -- 172.21.15.116:0/2082689518 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d701f2560 con 0x7f7d70155fa0 2024-04-25T00:24:10.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.645+0000 7f7d6ffff640 1 -- 172.21.15.116:0/2082689518 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d701ef9c0 con 0x7f7d70155fa0 2024-04-25T00:24:10.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.645+0000 7f7d6ffff640 1 -- 172.21.15.116:0/2082689518 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d7004f510 con 0x7f7d70155fa0 2024-04-25T00:24:10.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.645+0000 7f7d577fe640 1 -- 172.21.15.116:0/2082689518 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7d6800abd0 con 0x7f7d70155fa0 2024-04-25T00:24:10.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.649+0000 7f7d577fe640 1 --2- 172.21.15.116:0/2082689518 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7d5c0678c0 0x7f7d5c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:10.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.649+0000 7f7d577fe640 1 -- 172.21.15.116:0/2082689518 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(64..64 src has 1..64) ==== 9762+0+0 (secure 0 0 0) 0x7f7d68086060 con 0x7f7d70155fa0 2024-04-25T00:24:10.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.653+0000 7f7d6effd640 1 --2- 172.21.15.116:0/2082689518 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7d5c0678c0 0x7f7d5c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:10.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.653+0000 7f7d577fe640 1 -- 172.21.15.116:0/2082689518 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f7d680555b0 con 0x7f7d70155fa0 2024-04-25T00:24:10.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:10.657+0000 7f7d6effd640 1 --2- 172.21.15.116:0/2082689518 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7d5c0678c0 0x7f7d5c069d80 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f7d700ac210 tx=0x7f7d60009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:11.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:11.205+0000 7f7d6ffff640 1 -- 172.21.15.116:0/2082689518 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7d700cb950 con 0x7f7d70155fa0 2024-04-25T00:24:11.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:11.205+0000 7f7d577fe640 1 -- 172.21.15.116:0/2082689518 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 192 v192) ==== 95+0+6506 (secure 0 0 0) 0x7f7d680598b0 con 0x7f7d70155fa0 2024-04-25T00:24:11.207 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:24:11.208 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":192,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":8343,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/3633657268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3633657268},{"type":"v1","addr":"172.21.15.116:6833","nonce":3633657268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":191,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:23:34.013510+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8244":{"gid":8244,"name":"c","rank":0,"incarnation":188,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6835/1971675962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1971675962},{"type":"v1","addr":"172.21.15.116:6835","nonce":1971675962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8244,"qdb_cluster":[8244]},"id":1},{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:22:38.936345+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6230":{"gid":6230,"name":"d","rank":0,"incarnation":178,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/492366509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":492366509},{"type":"v1","addr":"172.21.15.174:6837","nonce":492366509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6230,"qdb_cluster":[6230]},"id":2}]} 2024-04-25T00:24:11.208 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 192 2024-04-25T00:24:11.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:11.213+0000 7f7d557fa640 1 -- 172.21.15.116:0/2082689518 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7d5c0678c0 msgr2=0x7f7d5c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:11.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:11.213+0000 7f7d557fa640 1 --2- 172.21.15.116:0/2082689518 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7d5c0678c0 0x7f7d5c069d80 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f7d700ac210 tx=0x7f7d60009190 comp rx=0 tx=0).stop 2024-04-25T00:24:11.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:11.213+0000 7f7d557fa640 1 -- 172.21.15.116:0/2082689518 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d70155fa0 msgr2=0x7f7d701f09e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:11.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:11.213+0000 7f7d557fa640 1 --2- 172.21.15.116:0/2082689518 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d70155fa0 0x7f7d701f09e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7d6800c7b0 tx=0x7f7d6800cc70 comp rx=0 tx=0).stop 2024-04-25T00:24:11.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:11.213+0000 7f7d557fa640 1 -- 172.21.15.116:0/2082689518 shutdown_connections 2024-04-25T00:24:11.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:11.213+0000 7f7d557fa640 1 --2- 172.21.15.116:0/2082689518 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7d5c0678c0 0x7f7d5c069d80 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:11.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:11.213+0000 7f7d557fa640 1 --2- 172.21.15.116:0/2082689518 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d70155fa0 0x7f7d701f09e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:11.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:11.213+0000 7f7d557fa640 1 --2- 172.21.15.116:0/2082689518 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d7014bf80 0x7f7d701f04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:11.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:11.213+0000 7f7d557fa640 1 --2- 172.21.15.116:0/2082689518 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d700aca50 0x7f7d7014b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:11.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:11.213+0000 7f7d557fa640 1 -- 172.21.15.116:0/2082689518 >> 172.21.15.116:0/2082689518 conn(0x7f7d70074a80 msgr2=0x7f7d7006ca50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:11.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:11.213+0000 7f7d557fa640 1 -- 172.21.15.116:0/2082689518 shutdown_connections 2024-04-25T00:24:11.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:11.213+0000 7f7d557fa640 1 -- 172.21.15.116:0/2082689518 wait complete. 2024-04-25T00:24:11.264 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:24:13.264 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:24:13.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.489+0000 7f2d0c341640 1 Processor -- start 2024-04-25T00:24:13.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.489+0000 7f2d0c341640 1 -- start start 2024-04-25T00:24:13.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.489+0000 7f2d0c341640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f2d04152ce0 0x7f2d0406bc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:13.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.489+0000 7f2d0c341640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2d04159970 0x7f2d04155720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:13.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.489+0000 7f2d0c341640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2d04155c60 0x7f2d040c3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:13.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.489+0000 7f2d0c341640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f2d041594e0 con 0x7f2d04159970 2024-04-25T00:24:13.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.489+0000 7f2d0c341640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f2d0406c160 con 0x7f2d04152ce0 2024-04-25T00:24:13.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.489+0000 7f2d0c341640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f2d04073260 con 0x7f2d04155c60 2024-04-25T00:24:13.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.489+0000 7f2d098dc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2d04159970 0x7f2d04155720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:13.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.489+0000 7f2d098dc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2d04159970 0x7f2d04155720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51446/0 (socket says 172.21.15.116:51446) 2024-04-25T00:24:13.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.489+0000 7f2d098dc640 1 -- 172.21.15.116:0/2221382831 learned_addr learned my addr 172.21.15.116:0/2221382831 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:24:13.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.489+0000 7f2d0a8de640 1 --2- 172.21.15.116:0/2221382831 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2d04155c60 0x7f2d040c3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:13.492 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.489+0000 7f2d0a0dd640 1 --2- 172.21.15.116:0/2221382831 >> v2:172.21.15.174:3300/0 conn(0x7f2d04152ce0 0x7f2d0406bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:13.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.489+0000 7f2d098dc640 1 -- 172.21.15.116:0/2221382831 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2d04155c60 msgr2=0x7f2d040c3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:13.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.489+0000 7f2d098dc640 1 --2- 172.21.15.116:0/2221382831 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2d04155c60 0x7f2d040c3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:13.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.489+0000 7f2d098dc640 1 -- 172.21.15.116:0/2221382831 >> v2:172.21.15.174:3300/0 conn(0x7f2d04152ce0 msgr2=0x7f2d0406bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:13.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.489+0000 7f2d098dc640 1 --2- 172.21.15.116:0/2221382831 >> v2:172.21.15.174:3300/0 conn(0x7f2d04152ce0 0x7f2d0406bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:13.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.489+0000 7f2d098dc640 1 -- 172.21.15.116:0/2221382831 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d04158e90 con 0x7f2d04159970 2024-04-25T00:24:13.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.497+0000 7f2d098dc640 1 --2- 172.21.15.116:0/2221382831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d04159970 0x7f2d04155720 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f2d04071800 tx=0x7f2cf401c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f3681c10a2b4e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:13.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.497+0000 7f2d090db640 1 -- 172.21.15.116:0/2221382831 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cf40b0070 con 0x7f2d04159970 2024-04-25T00:24:13.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.497+0000 7f2d090db640 1 -- 172.21.15.116:0/2221382831 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2cf40ab600 con 0x7f2d04159970 2024-04-25T00:24:13.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.497+0000 7f2d090db640 1 -- 172.21.15.116:0/2221382831 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cf40ab9d0 con 0x7f2d04159970 2024-04-25T00:24:13.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.505+0000 7f2d0c341640 1 -- 172.21.15.116:0/2221382831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d04159970 msgr2=0x7f2d04155720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:13.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.505+0000 7f2d0c341640 1 --2- 172.21.15.116:0/2221382831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d04159970 0x7f2d04155720 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f2d04071800 tx=0x7f2cf401c460 comp rx=0 tx=0).stop 2024-04-25T00:24:13.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.509+0000 7f2d0c341640 1 -- 172.21.15.116:0/2221382831 shutdown_connections 2024-04-25T00:24:13.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.509+0000 7f2d0c341640 1 --2- 172.21.15.116:0/2221382831 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2d04155c60 0x7f2d040c3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:13.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.509+0000 7f2d0c341640 1 --2- 172.21.15.116:0/2221382831 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d04159970 0x7f2d04155720 secure :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f2d04071800 tx=0x7f2cf401c460 comp rx=0 tx=0).stop 2024-04-25T00:24:13.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.509+0000 7f2d0c341640 1 --2- 172.21.15.116:0/2221382831 >> v2:172.21.15.174:3300/0 conn(0x7f2d04152ce0 0x7f2d0406bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:13.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.509+0000 7f2d0c341640 1 -- 172.21.15.116:0/2221382831 >> 172.21.15.116:0/2221382831 conn(0x7f2d040c4c10 msgr2=0x7f2d040c5020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:13.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.517+0000 7f2d0c341640 1 -- 172.21.15.116:0/2221382831 shutdown_connections 2024-04-25T00:24:13.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.517+0000 7f2d0c341640 1 -- 172.21.15.116:0/2221382831 wait complete. 2024-04-25T00:24:13.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.517+0000 7f2d0c341640 1 Processor -- start 2024-04-25T00:24:13.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.517+0000 7f2d0c341640 1 -- start start 2024-04-25T00:24:13.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.521+0000 7f2d0c341640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2d04152ce0 0x7f2d040abd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:13.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.521+0000 7f2d0c341640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d04155c60 0x7f2d041f1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:13.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.521+0000 7f2d0a0dd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2d04152ce0 0x7f2d040abd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:13.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.521+0000 7f2d0a0dd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2d04152ce0 0x7f2d040abd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:40850/0 (socket says 172.21.15.116:40850) 2024-04-25T00:24:13.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.521+0000 7f2d0a0dd640 1 -- 172.21.15.116:0/435162032 learned_addr learned my addr 172.21.15.116:0/435162032 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:24:13.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.521+0000 7f2d0c341640 1 --2- 172.21.15.116:0/435162032 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d040ab930 0x7f2d041f2390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:13.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.521+0000 7f2d0c341640 1 -- 172.21.15.116:0/435162032 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2d04073000 con 0x7f2d04155c60 2024-04-25T00:24:13.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.521+0000 7f2d0c341640 1 -- 172.21.15.116:0/435162032 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2d04072e80 con 0x7f2d040ab930 2024-04-25T00:24:13.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.521+0000 7f2d0c341640 1 -- 172.21.15.116:0/435162032 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f2d040c1d50 con 0x7f2d04152ce0 2024-04-25T00:24:13.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.521+0000 7f2d0a8de640 1 --2- 172.21.15.116:0/435162032 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d040ab930 0x7f2d041f2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:13.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.521+0000 7f2d098dc640 1 --2- 172.21.15.116:0/435162032 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d04155c60 0x7f2d041f1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:13.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.521+0000 7f2d0a0dd640 1 -- 172.21.15.116:0/435162032 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d040ab930 msgr2=0x7f2d041f2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:13.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.521+0000 7f2d0a0dd640 1 --2- 172.21.15.116:0/435162032 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d040ab930 0x7f2d041f2390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:13.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.521+0000 7f2d0a0dd640 1 -- 172.21.15.116:0/435162032 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d04155c60 msgr2=0x7f2d041f1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:13.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.521+0000 7f2d0a0dd640 1 --2- 172.21.15.116:0/435162032 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d04155c60 0x7f2d041f1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:13.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.521+0000 7f2d0a0dd640 1 -- 172.21.15.116:0/435162032 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d041ef100 con 0x7f2d04152ce0 2024-04-25T00:24:13.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.521+0000 7f2d098dc640 1 --2- 172.21.15.116:0/435162032 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d04155c60 0x7f2d041f1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:24:13.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.525+0000 7f2d0a0dd640 1 --2- 172.21.15.116:0/435162032 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2d04152ce0 0x7f2d040abd80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2d0000e930 tx=0x7f2d0000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:13.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.525+0000 7f2cfaffd640 1 -- 172.21.15.116:0/435162032 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d00017070 con 0x7f2d04152ce0 2024-04-25T00:24:13.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.525+0000 7f2cfaffd640 1 -- 172.21.15.116:0/435162032 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d0000c9a0 con 0x7f2d04152ce0 2024-04-25T00:24:13.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.525+0000 7f2cfaffd640 1 -- 172.21.15.116:0/435162032 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d0000cd50 con 0x7f2d04152ce0 2024-04-25T00:24:13.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.525+0000 7f2d0c341640 1 -- 172.21.15.116:0/435162032 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d041f2970 con 0x7f2d04152ce0 2024-04-25T00:24:13.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.525+0000 7f2d0c341640 1 -- 172.21.15.116:0/435162032 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d0415e8c0 con 0x7f2d04152ce0 2024-04-25T00:24:13.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.529+0000 7f2d0c341640 1 -- 172.21.15.116:0/435162032 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d041ef5e0 con 0x7f2d04152ce0 2024-04-25T00:24:13.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.529+0000 7f2cfaffd640 1 -- 172.21.15.116:0/435162032 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2d0001a070 con 0x7f2d04152ce0 2024-04-25T00:24:13.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.529+0000 7f2cfaffd640 1 --2- 172.21.15.116:0/435162032 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2ce80678f0 0x7f2ce8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:13.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.529+0000 7f2cfaffd640 1 -- 172.21.15.116:0/435162032 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(64..64 src has 1..64) ==== 9762+0+0 (secure 0 0 0) 0x7f2d00086be0 con 0x7f2d04152ce0 2024-04-25T00:24:13.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.533+0000 7f2d098dc640 1 --2- 172.21.15.116:0/435162032 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2ce80678f0 0x7f2ce8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:13.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.533+0000 7f2d098dc640 1 --2- 172.21.15.116:0/435162032 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2ce80678f0 0x7f2ce8069db0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f2cf40ad000 tx=0x7f2cf40a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:13.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:13.537+0000 7f2cfaffd640 1 -- 172.21.15.116:0/435162032 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f2d000560a0 con 0x7f2d04152ce0 2024-04-25T00:24:14.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:14.189+0000 7f2d0c341640 1 -- 172.21.15.116:0/435162032 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2d04068a60 con 0x7f2d04152ce0 2024-04-25T00:24:14.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:14.189+0000 7f2cfaffd640 1 -- 172.21.15.116:0/435162032 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 192 v192) ==== 95+0+6506 (secure 0 0 0) 0x7f2d0005a3a0 con 0x7f2d04152ce0 2024-04-25T00:24:14.194 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:24:14.195 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":192,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":8343,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/3633657268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3633657268},{"type":"v1","addr":"172.21.15.116:6833","nonce":3633657268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":191,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:23:34.013510+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8244":{"gid":8244,"name":"c","rank":0,"incarnation":188,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6835/1971675962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1971675962},{"type":"v1","addr":"172.21.15.116:6835","nonce":1971675962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8244,"qdb_cluster":[8244]},"id":1},{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:22:38.936345+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6230},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6230":{"gid":6230,"name":"d","rank":0,"incarnation":178,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/492366509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":492366509},{"type":"v1","addr":"172.21.15.174:6837","nonce":492366509}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6230,"qdb_cluster":[6230]},"id":2}]} 2024-04-25T00:24:14.195 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 192 2024-04-25T00:24:14.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:14.201+0000 7f2d0c341640 1 -- 172.21.15.116:0/435162032 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2ce80678f0 msgr2=0x7f2ce8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:14.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:14.201+0000 7f2d0c341640 1 --2- 172.21.15.116:0/435162032 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2ce80678f0 0x7f2ce8069db0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f2cf40ad000 tx=0x7f2cf40a6000 comp rx=0 tx=0).stop 2024-04-25T00:24:14.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:14.201+0000 7f2d0c341640 1 -- 172.21.15.116:0/435162032 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2d04152ce0 msgr2=0x7f2d040abd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:14.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:14.201+0000 7f2d0c341640 1 --2- 172.21.15.116:0/435162032 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2d04152ce0 0x7f2d040abd80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2d0000e930 tx=0x7f2d0000edf0 comp rx=0 tx=0).stop 2024-04-25T00:24:14.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:14.201+0000 7f2d0c341640 1 -- 172.21.15.116:0/435162032 shutdown_connections 2024-04-25T00:24:14.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:14.201+0000 7f2d0c341640 1 --2- 172.21.15.116:0/435162032 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2ce80678f0 0x7f2ce8069db0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:14.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:14.201+0000 7f2d0c341640 1 --2- 172.21.15.116:0/435162032 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2d040ab930 0x7f2d041f2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:14.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:14.201+0000 7f2d0c341640 1 --2- 172.21.15.116:0/435162032 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2d04155c60 0x7f2d041f1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:14.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:14.201+0000 7f2d0c341640 1 --2- 172.21.15.116:0/435162032 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2d04152ce0 0x7f2d040abd80 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:14.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:14.201+0000 7f2d0c341640 1 -- 172.21.15.116:0/435162032 >> 172.21.15.116:0/435162032 conn(0x7f2d040c4c10 msgr2=0x7f2d04072700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:14.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:14.201+0000 7f2d0c341640 1 -- 172.21.15.116:0/435162032 shutdown_connections 2024-04-25T00:24:14.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:14.201+0000 7f2d0c341640 1 -- 172.21.15.116:0/435162032 wait complete. 2024-04-25T00:24:14.257 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:24:16.258 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:24:16.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:16.549+0000 7fdc3b59e640 1 Processor -- start 2024-04-25T00:24:16.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:16.549+0000 7fdc3b59e640 1 -- start start 2024-04-25T00:24:16.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:16.549+0000 7fdc3b59e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdc3400d6e0 0x7fdc34049770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:16.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:16.549+0000 7fdc3b59e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fdc340df2e0 0x7fdc34040d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:16.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:16.549+0000 7fdc3b59e640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc3400c560 con 0x7fdc340409a0 2024-04-25T00:24:16.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:16.549+0000 7fdc3b59e640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fdc34049dc0 con 0x7fdc340df2e0 2024-04-25T00:24:16.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:16.549+0000 7fdc3b59e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fdc3404a0a0 con 0x7fdc3400d6e0 2024-04-25T00:24:16.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:16.549+0000 7fdc3ad9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fdc340df2e0 0x7fdc34040d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:16.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:16.549+0000 7fdc3ad9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fdc340df2e0 0x7fdc34040d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56002/0 (socket says 172.21.15.116:56002) 2024-04-25T00:24:16.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:16.549+0000 7fdc3ad9d640 1 -- 172.21.15.116:0/2497686171 learned_addr learned my addr 172.21.15.116:0/2497686171 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:24:16.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:16.549+0000 7fdc39d9b640 1 --2- 172.21.15.116:0/2497686171 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdc3400d6e0 0x7fdc34049770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:16.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:16.549+0000 7fdc3ad9d640 1 -- 172.21.15.116:0/2497686171 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdc3400d6e0 msgr2=0x7fdc34049770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:16.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:16.549+0000 7fdc3ad9d640 1 --2- 172.21.15.116:0/2497686171 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdc3400d6e0 0x7fdc34049770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:16.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:16.549+0000 7fdc3ad9d640 1 -- 172.21.15.116:0/2497686171 >> v1:172.21.15.116:6789/0 conn(0x7fdc340409a0 legacy=0x7fdc3400be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:16.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:16.549+0000 7fdc3ad9d640 1 -- 172.21.15.116:0/2497686171 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc340deae0 con 0x7fdc340df2e0 2024-04-25T00:24:16.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:16.553+0000 7fdc3ad9d640 1 --2- 172.21.15.116:0/2497686171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc340df2e0 0x7fdc34040d80 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c04fa30 tx=0x7fdc3c09a8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0d8e5af7c3eaa5b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:16.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:16.553+0000 7fdc3959a640 1 -- 172.21.15.116:0/2497686171 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc3c06e410 con 0x7fdc340df2e0 2024-04-25T00:24:16.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:16.553+0000 7fdc3959a640 1 -- 172.21.15.116:0/2497686171 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc3c06e5b0 con 0x7fdc340df2e0 2024-04-25T00:24:16.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:16.553+0000 7fdc3959a640 1 -- 172.21.15.116:0/2497686171 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc3c06ea40 con 0x7fdc340df2e0 2024-04-25T00:24:16.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:16.553+0000 7fdc3b59e640 1 -- 172.21.15.116:0/2497686171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc340df2e0 msgr2=0x7fdc34040d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:16.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:16.553+0000 7fdc3b59e640 1 --2- 172.21.15.116:0/2497686171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc340df2e0 0x7fdc34040d80 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c04fa30 tx=0x7fdc3c09a8e0 comp rx=0 tx=0).stop 2024-04-25T00:24:16.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:16.553+0000 7fdc3b59e640 1 -- 172.21.15.116:0/2497686171 shutdown_connections 2024-04-25T00:24:16.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:16.553+0000 7fdc3b59e640 1 --2- 172.21.15.116:0/2497686171 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc340df2e0 0x7fdc34040d80 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:16.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:16.553+0000 7fdc3b59e640 1 --2- 172.21.15.116:0/2497686171 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdc3400d6e0 0x7fdc34049770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:16.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:16.553+0000 7fdc3b59e640 1 -- 172.21.15.116:0/2497686171 >> 172.21.15.116:0/2497686171 conn(0x7fdc3402f4c0 msgr2=0x7fdc3402f8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:16.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:16.557+0000 7fdc3b59e640 1 -- 172.21.15.116:0/2497686171 shutdown_connections 2024-04-25T00:24:16.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:16.557+0000 7fdc3b59e640 1 -- 172.21.15.116:0/2497686171 wait complete. 2024-04-25T00:24:16.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:16.557+0000 7fdc3b59e640 1 Processor -- start 2024-04-25T00:24:16.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:16.557+0000 7fdc3b59e640 1 -- start start 2024-04-25T00:24:16.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:16.557+0000 7fdc3b59e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdc3400d6e0 0x7fdc3416b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:16.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:16.557+0000 7fdc3b59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc340409a0 0x7fdc340465e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:16.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:16.557+0000 7fdc3b59e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdc340df2e0 0x7fdc34046b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:16.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:16.557+0000 7fdc3b59e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fdc340302d0 con 0x7fdc340409a0 2024-04-25T00:24:16.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:16.557+0000 7fdc3b59e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fdc340de7b0 con 0x7fdc3400d6e0 2024-04-25T00:24:16.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:16.557+0000 7fdc3b59e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fdc340dc250 con 0x7fdc340df2e0 2024-04-25T00:24:16.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:16.561+0000 7fdc39d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc340409a0 0x7fdc340465e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:16.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:16.561+0000 7fdc39d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc340409a0 0x7fdc340465e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56006/0 (socket says 172.21.15.116:56006) 2024-04-25T00:24:16.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:16.561+0000 7fdc39d9b640 1 -- 172.21.15.116:0/863415883 learned_addr learned my addr 172.21.15.116:0/863415883 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:24:16.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:16.561+0000 7fdc3ad9d640 1 --2- 172.21.15.116:0/863415883 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdc340df2e0 0x7fdc34046b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:16.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:16.561+0000 7fdc3ad9d640 1 -- 172.21.15.116:0/863415883 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdc3400d6e0 msgr2=0x7fdc3416b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:16.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:16.561+0000 7fdc3ad9d640 1 --2- 172.21.15.116:0/863415883 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdc3400d6e0 0x7fdc3416b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:16.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:16.561+0000 7fdc3ad9d640 1 -- 172.21.15.116:0/863415883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc340409a0 msgr2=0x7fdc340465e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:16.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:16.561+0000 7fdc3ad9d640 1 --2- 172.21.15.116:0/863415883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc340409a0 0x7fdc340465e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:16.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:16.561+0000 7fdc3ad9d640 1 -- 172.21.15.116:0/863415883 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc340da530 con 0x7fdc340df2e0 2024-04-25T00:24:16.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:16.561+0000 7fdc3ad9d640 1 --2- 172.21.15.116:0/863415883 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdc340df2e0 0x7fdc34046b20 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c06a140 tx=0x7fdc3c097720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:16.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:16.561+0000 7fdc2affd640 1 -- 172.21.15.116:0/863415883 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc3c13a020 con 0x7fdc340df2e0 2024-04-25T00:24:16.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:16.561+0000 7fdc2affd640 1 -- 172.21.15.116:0/863415883 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc3c070190 con 0x7fdc340df2e0 2024-04-25T00:24:16.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:16.561+0000 7fdc3b59e640 1 -- 172.21.15.116:0/863415883 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc340db500 con 0x7fdc340df2e0 2024-04-25T00:24:16.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:16.561+0000 7fdc2affd640 1 -- 172.21.15.116:0/863415883 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc3c06e410 con 0x7fdc340df2e0 2024-04-25T00:24:16.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:16.561+0000 7fdc3b59e640 1 -- 172.21.15.116:0/863415883 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc3416bde0 con 0x7fdc340df2e0 2024-04-25T00:24:16.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:16.561+0000 7fdc28ff9640 1 -- 172.21.15.116:0/863415883 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc200052a0 con 0x7fdc340df2e0 2024-04-25T00:24:16.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:16.561+0000 7fdc2affd640 1 -- 172.21.15.116:0/863415883 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fdc3c13d070 con 0x7fdc340df2e0 2024-04-25T00:24:16.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:16.561+0000 7fdc2affd640 1 --2- 172.21.15.116:0/863415883 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdc1c0678c0 0x7fdc1c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:16.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:16.561+0000 7fdc2affd640 1 -- 172.21.15.116:0/863415883 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(65..65 src has 1..65) ==== 9848+0+0 (secure 0 0 0) 0x7fdc3c19fa80 con 0x7fdc340df2e0 2024-04-25T00:24:16.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:16.565+0000 7fdc3a59c640 1 --2- 172.21.15.116:0/863415883 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdc1c0678c0 0x7fdc1c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:16.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:16.565+0000 7fdc3a59c640 1 --2- 172.21.15.116:0/863415883 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdc1c0678c0 0x7fdc1c069d80 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c000f70 tx=0x7fdc2c0079c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:16.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:16.573+0000 7fdc2affd640 1 -- 172.21.15.116:0/863415883 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fdc3c094510 con 0x7fdc340df2e0 2024-04-25T00:24:17.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.129+0000 7fdc28ff9640 1 -- 172.21.15.116:0/863415883 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdc20005590 con 0x7fdc340df2e0 2024-04-25T00:24:17.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.133+0000 7fdc2affd640 1 -- 172.21.15.116:0/863415883 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 193 v193) ==== 95+0+5673 (secure 0 0 0) 0x7fdc3c16f160 con 0x7fdc340df2e0 2024-04-25T00:24:17.138 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:24:17.138 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":193,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":191,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:23:34.013510+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8244":{"gid":8244,"name":"c","rank":0,"incarnation":188,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6835/1971675962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1971675962},{"type":"v1","addr":"172.21.15.116:6835","nonce":1971675962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8244,"qdb_cluster":[8244]},"id":1},{"mdsmap":{"epoch":193,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:24:15.702139+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8343":{"gid":8343,"name":"a","rank":0,"incarnation":193,"state":"up:replay","state_seq":1,"addr":"172.21.15.116:6833/3633657268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3633657268},{"type":"v1","addr":"172.21.15.116:6833","nonce":3633657268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:24:17.138 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 193 2024-04-25T00:24:17.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.137+0000 7fdc28ff9640 1 -- 172.21.15.116:0/863415883 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdc1c0678c0 msgr2=0x7fdc1c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:17.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.137+0000 7fdc28ff9640 1 --2- 172.21.15.116:0/863415883 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdc1c0678c0 0x7fdc1c069d80 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c000f70 tx=0x7fdc2c0079c0 comp rx=0 tx=0).stop 2024-04-25T00:24:17.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.137+0000 7fdc28ff9640 1 -- 172.21.15.116:0/863415883 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdc340df2e0 msgr2=0x7fdc34046b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:17.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.137+0000 7fdc28ff9640 1 --2- 172.21.15.116:0/863415883 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdc340df2e0 0x7fdc34046b20 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c06a140 tx=0x7fdc3c097720 comp rx=0 tx=0).stop 2024-04-25T00:24:17.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.141+0000 7fdc28ff9640 1 -- 172.21.15.116:0/863415883 shutdown_connections 2024-04-25T00:24:17.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.141+0000 7fdc28ff9640 1 --2- 172.21.15.116:0/863415883 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fdc1c0678c0 0x7fdc1c069d80 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:17.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.141+0000 7fdc28ff9640 1 --2- 172.21.15.116:0/863415883 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fdc340df2e0 0x7fdc34046b20 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:17.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.141+0000 7fdc28ff9640 1 --2- 172.21.15.116:0/863415883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fdc340409a0 0x7fdc340465e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:17.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.141+0000 7fdc28ff9640 1 --2- 172.21.15.116:0/863415883 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fdc3400d6e0 0x7fdc3416b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:17.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.141+0000 7fdc28ff9640 1 -- 172.21.15.116:0/863415883 >> 172.21.15.116:0/863415883 conn(0x7fdc3402f4c0 msgr2=0x7fdc340dda30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:17.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.141+0000 7fdc28ff9640 1 -- 172.21.15.116:0/863415883 shutdown_connections 2024-04-25T00:24:17.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.141+0000 7fdc28ff9640 1 -- 172.21.15.116:0/863415883 wait complete. 2024-04-25T00:24:17.216 INFO:tasks.mds_thrash.fs.[b]:mds.d down, removed from mdsmap 2024-04-25T00:24:17.216 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-04-25T00:24:17.217 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:24:17.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.545+0000 7ff469ff4640 1 Processor -- start 2024-04-25T00:24:17.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.545+0000 7ff469ff4640 1 -- start start 2024-04-25T00:24:17.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.545+0000 7ff469ff4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff464150c60 0x7ff4640ad5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:17.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.545+0000 7ff469ff4640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff4640adb20 0x7ff46415e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:17.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.545+0000 7ff469ff4640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff46406f9f0 con 0x7ff4640a94a0 2024-04-25T00:24:17.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.545+0000 7ff469ff4640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7ff46406ff90 con 0x7ff4640adb20 2024-04-25T00:24:17.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.545+0000 7ff469ff4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff46406fcb0 con 0x7ff464150c60 2024-04-25T00:24:17.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.545+0000 7ff4637fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff464150c60 0x7ff4640ad5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:17.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.545+0000 7ff4637fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff464150c60 0x7ff4640ad5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:43344/0 (socket says 172.21.15.116:43344) 2024-04-25T00:24:17.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.549+0000 7ff4637fe640 1 -- 172.21.15.116:0/3619053879 learned_addr learned my addr 172.21.15.116:0/3619053879 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:24:17.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.549+0000 7ff463fff640 1 --2- 172.21.15.116:0/3619053879 >> v2:172.21.15.174:3300/0 conn(0x7ff4640adb20 0x7ff46415e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:17.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.549+0000 7ff4627fc640 1 -- 172.21.15.116:0/3619053879 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3620064065 0 0) 0x7ff46406f9f0 con 0x7ff4640a94a0 2024-04-25T00:24:17.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.549+0000 7ff4627fc640 1 -- 172.21.15.116:0/3619053879 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff44c003530 con 0x7ff4640a94a0 2024-04-25T00:24:17.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.549+0000 7ff4637fe640 1 -- 172.21.15.116:0/3619053879 >> v2:172.21.15.174:3300/0 conn(0x7ff4640adb20 msgr2=0x7ff46415e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:17.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.549+0000 7ff4637fe640 1 --2- 172.21.15.116:0/3619053879 >> v2:172.21.15.174:3300/0 conn(0x7ff4640adb20 0x7ff46415e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:17.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.549+0000 7ff4637fe640 1 -- 172.21.15.116:0/3619053879 >> v1:172.21.15.174:6789/0 conn(0x7ff4640a94a0 legacy=0x7ff464153750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:17.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.549+0000 7ff4637fe640 1 -- 172.21.15.116:0/3619053879 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff464072dc0 con 0x7ff464150c60 2024-04-25T00:24:17.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.549+0000 7ff4637fe640 1 --2- 172.21.15.116:0/3619053879 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff464150c60 0x7ff4640ad5e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff454009870 tx=0x7ff45401c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43d56ed9536c61e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:17.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.549+0000 7ff4627fc640 1 -- 172.21.15.116:0/3619053879 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4540b0070 con 0x7ff464150c60 2024-04-25T00:24:17.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.549+0000 7ff4627fc640 1 -- 172.21.15.116:0/3619053879 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4540ab600 con 0x7ff464150c60 2024-04-25T00:24:17.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.549+0000 7ff4627fc640 1 -- 172.21.15.116:0/3619053879 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4540ab9b0 con 0x7ff464150c60 2024-04-25T00:24:17.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.553+0000 7ff469ff4640 1 -- 172.21.15.116:0/3619053879 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff464150c60 msgr2=0x7ff4640ad5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:17.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.553+0000 7ff469ff4640 1 --2- 172.21.15.116:0/3619053879 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff464150c60 0x7ff4640ad5e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff454009870 tx=0x7ff45401c4d0 comp rx=0 tx=0).stop 2024-04-25T00:24:17.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.585+0000 7ff469ff4640 1 -- 172.21.15.116:0/3619053879 shutdown_connections 2024-04-25T00:24:17.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.585+0000 7ff469ff4640 1 --2- 172.21.15.116:0/3619053879 >> v2:172.21.15.174:3300/0 conn(0x7ff4640adb20 0x7ff46415e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:17.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.585+0000 7ff469ff4640 1 --2- 172.21.15.116:0/3619053879 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff464150c60 0x7ff4640ad5e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:17.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.585+0000 7ff469ff4640 1 -- 172.21.15.116:0/3619053879 >> 172.21.15.116:0/3619053879 conn(0x7ff4640c8140 msgr2=0x7ff4640c8550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:17.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.585+0000 7ff469ff4640 1 -- 172.21.15.116:0/3619053879 shutdown_connections 2024-04-25T00:24:17.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.585+0000 7ff469ff4640 1 -- 172.21.15.116:0/3619053879 wait complete. 2024-04-25T00:24:17.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.589+0000 7ff469ff4640 1 Processor -- start 2024-04-25T00:24:17.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.589+0000 7ff469ff4640 1 -- start start 2024-04-25T00:24:17.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.589+0000 7ff469ff4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4640a94a0 0x7ff4640ab920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:17.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.589+0000 7ff469ff4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff4640adb20 0x7ff464140800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:17.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.589+0000 7ff469ff4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff464150c60 0x7ff464140d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:17.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.589+0000 7ff469ff4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff4640c8e50 con 0x7ff4640a94a0 2024-04-25T00:24:17.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.589+0000 7ff469ff4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff4640c8cd0 con 0x7ff464150c60 2024-04-25T00:24:17.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.589+0000 7ff469ff4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff464151040 con 0x7ff4640adb20 2024-04-25T00:24:17.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.589+0000 7ff4637fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4640a94a0 0x7ff4640ab920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:17.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.589+0000 7ff4637fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4640a94a0 0x7ff4640ab920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56016/0 (socket says 172.21.15.116:56016) 2024-04-25T00:24:17.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.589+0000 7ff4637fe640 1 -- 172.21.15.116:0/415097080 learned_addr learned my addr 172.21.15.116:0/415097080 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:24:17.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.589+0000 7ff462ffd640 1 --2- 172.21.15.116:0/415097080 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff4640adb20 0x7ff464140800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:17.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.589+0000 7ff463fff640 1 --2- 172.21.15.116:0/415097080 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff464150c60 0x7ff464140d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:17.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.589+0000 7ff4637fe640 1 -- 172.21.15.116:0/415097080 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff4640adb20 msgr2=0x7ff464140800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:17.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.589+0000 7ff4637fe640 1 --2- 172.21.15.116:0/415097080 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff4640adb20 0x7ff464140800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:17.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.589+0000 7ff4637fe640 1 -- 172.21.15.116:0/415097080 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff464150c60 msgr2=0x7ff464140d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:17.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.589+0000 7ff4637fe640 1 --2- 172.21.15.116:0/415097080 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff464150c60 0x7ff464140d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:17.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.589+0000 7ff4637fe640 1 -- 172.21.15.116:0/415097080 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4640acab0 con 0x7ff4640a94a0 2024-04-25T00:24:17.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.593+0000 7ff4637fe640 1 --2- 172.21.15.116:0/415097080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4640a94a0 0x7ff4640ab920 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7ff4540ae040 tx=0x7ff454019270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:17.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.593+0000 7ff44bfff640 1 -- 172.21.15.116:0/415097080 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4540b9020 con 0x7ff4640a94a0 2024-04-25T00:24:17.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.593+0000 7ff44bfff640 1 -- 172.21.15.116:0/415097080 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4540b1040 con 0x7ff4640a94a0 2024-04-25T00:24:17.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.593+0000 7ff469ff4640 1 -- 172.21.15.116:0/415097080 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4640ad1a0 con 0x7ff4640a94a0 2024-04-25T00:24:17.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.593+0000 7ff44bfff640 1 -- 172.21.15.116:0/415097080 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4540b0070 con 0x7ff4640a94a0 2024-04-25T00:24:17.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.593+0000 7ff44bfff640 1 -- 172.21.15.116:0/415097080 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff4540bc020 con 0x7ff4640a94a0 2024-04-25T00:24:17.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.593+0000 7ff469ff4640 1 -- 172.21.15.116:0/415097080 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4640c23c0 con 0x7ff4640a94a0 2024-04-25T00:24:17.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.597+0000 7ff44bfff640 1 --2- 172.21.15.116:0/415097080 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff450067940 0x7ff450069e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:17.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.597+0000 7ff462ffd640 1 --2- 172.21.15.116:0/415097080 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff450067940 0x7ff450069e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:17.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.597+0000 7ff449ffb640 1 -- 172.21.15.116:0/415097080 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff46404f580 con 0x7ff4640a94a0 2024-04-25T00:24:17.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.601+0000 7ff44bfff640 1 -- 172.21.15.116:0/415097080 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 9848+0+0 (secure 0 0 0) 0x7ff454123170 con 0x7ff4640a94a0 2024-04-25T00:24:17.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.601+0000 7ff462ffd640 1 --2- 172.21.15.116:0/415097080 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff450067940 0x7ff450069e00 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7ff4641f6db0 tx=0x7ff458009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:17.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:17.609+0000 7ff44bfff640 1 -- 172.21.15.116:0/415097080 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ff454016150 con 0x7ff4640a94a0 2024-04-25T00:24:18.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:18.085+0000 7ff449ffb640 1 -- 172.21.15.116:0/415097080 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff4640c9010 con 0x7ff4640a94a0 2024-04-25T00:24:18.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:18.089+0000 7ff44bfff640 1 -- 172.21.15.116:0/415097080 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 194 v194) ==== 95+0+5677 (secure 0 0 0) 0x7ff4540efe60 con 0x7ff4640a94a0 2024-04-25T00:24:18.091 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:24:18.093 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":194,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":191,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:23:34.013510+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8244":{"gid":8244,"name":"c","rank":0,"incarnation":188,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6835/1971675962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1971675962},{"type":"v1","addr":"172.21.15.116:6835","nonce":1971675962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8244,"qdb_cluster":[8244]},"id":1},{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:24:17.367912+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8343":{"gid":8343,"name":"a","rank":0,"incarnation":193,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.116:6833/3633657268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3633657268},{"type":"v1","addr":"172.21.15.116:6833","nonce":3633657268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:24:18.094 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 194 2024-04-25T00:24:18.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:18.093+0000 7ff449ffb640 1 -- 172.21.15.116:0/415097080 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff450067940 msgr2=0x7ff450069e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:18.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:18.093+0000 7ff449ffb640 1 --2- 172.21.15.116:0/415097080 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff450067940 0x7ff450069e00 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7ff4641f6db0 tx=0x7ff458009040 comp rx=0 tx=0).stop 2024-04-25T00:24:18.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:18.093+0000 7ff449ffb640 1 -- 172.21.15.116:0/415097080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4640a94a0 msgr2=0x7ff4640ab920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:18.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:18.093+0000 7ff449ffb640 1 --2- 172.21.15.116:0/415097080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4640a94a0 0x7ff4640ab920 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7ff4540ae040 tx=0x7ff454019270 comp rx=0 tx=0).stop 2024-04-25T00:24:18.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:18.093+0000 7ff449ffb640 1 -- 172.21.15.116:0/415097080 shutdown_connections 2024-04-25T00:24:18.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:18.093+0000 7ff449ffb640 1 --2- 172.21.15.116:0/415097080 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff450067940 0x7ff450069e00 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:18.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:18.093+0000 7ff449ffb640 1 --2- 172.21.15.116:0/415097080 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff464150c60 0x7ff464140d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:18.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:18.093+0000 7ff449ffb640 1 --2- 172.21.15.116:0/415097080 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff4640adb20 0x7ff464140800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:18.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:18.093+0000 7ff449ffb640 1 --2- 172.21.15.116:0/415097080 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff4640a94a0 0x7ff4640ab920 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:18.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:18.093+0000 7ff449ffb640 1 -- 172.21.15.116:0/415097080 >> 172.21.15.116:0/415097080 conn(0x7ff4640c8140 msgr2=0x7ff4640c7620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:18.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:18.093+0000 7ff449ffb640 1 -- 172.21.15.116:0/415097080 shutdown_connections 2024-04-25T00:24:18.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:18.093+0000 7ff449ffb640 1 -- 172.21.15.116:0/415097080 wait complete. 2024-04-25T00:24:18.149 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 194, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:04.678737+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 191, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_8244": { "addr": "172.21.15.116:6835/1971675962", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6834", "nonce": 1971675962, "type": "v2" }, { "addr": "172.21.15.116:6835", "nonce": 1971675962, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8244, "incarnation": 188, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 11 } }, "last_failure": 0, "last_failure_osd_epoch": 64, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T00:23:34.013510+0000", "qdb_cluster": [ 8244 ], "qdb_leader": 8244, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8244 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:18.825887+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 194, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_8343": { "addr": "172.21.15.116:6833/3633657268", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6832", "nonce": 3633657268, "type": "v2" }, { "addr": "172.21.15.116:6833", "nonce": 3633657268, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8343, "incarnation": 193, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:reconnect", "state_seq": 10 } }, "last_failure": 0, "last_failure_osd_epoch": 65, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-04-25T00:24:17.367912+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8343 } } } ], "standbys": [ { "addr": "172.21.15.174:6835/4112145782", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6834", "nonce": 4112145782, "type": "v2" }, { "addr": "172.21.15.174:6835", "nonce": 4112145782, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 112, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5582, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-25T00:24:20.151 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:24:20.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.380+0000 7f0e0176e640 1 Processor -- start 2024-04-25T00:24:20.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.380+0000 7f0e0176e640 1 -- start start 2024-04-25T00:24:20.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.384+0000 7f0e0176e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0dfc0b0830 0x7f0dfc159960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:20.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.384+0000 7f0e0176e640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f0dfc06cf80 0x7f0dfc0c7a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:20.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.384+0000 7f0e0176e640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0dfc06d3c0 con 0x7f0dfc0c8f80 2024-04-25T00:24:20.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.384+0000 7f0e0176e640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f0dfc06c250 con 0x7f0dfc06cf80 2024-04-25T00:24:20.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.384+0000 7f0dfb7fe640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f0dfc06cf80 0x7f0dfc0c7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:20.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.384+0000 7f0e0176e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f0dfc153280 con 0x7f0dfc0b0830 2024-04-25T00:24:20.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.384+0000 7f0dfb7fe640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f0dfc06cf80 0x7f0dfc0c7a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:60974/0 (socket says 172.21.15.116:60974) 2024-04-25T00:24:20.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.384+0000 7f0dfb7fe640 1 -- 172.21.15.116:0/1735501297 learned_addr learned my addr 172.21.15.116:0/1735501297 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:24:20.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.384+0000 7f0e00f6d640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f0dfc0c8f80 0x7f0dfc0c5a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:60000/0 (socket says 172.21.15.116:60000) 2024-04-25T00:24:20.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.384+0000 7f0dfb7fe640 1 -- 172.21.15.116:0/1735501297 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0dfc0b0830 msgr2=0x7f0dfc159960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:24:20.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.384+0000 7f0dfb7fe640 1 --2- 172.21.15.116:0/1735501297 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0dfc0b0830 0x7f0dfc159960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:20.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.384+0000 7f0dfb7fe640 1 -- 172.21.15.116:0/1735501297 >> v1:172.21.15.174:6789/0 conn(0x7f0dfc0c8f80 legacy=0x7f0dfc0c5a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:20.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.384+0000 7f0dfb7fe640 1 -- 172.21.15.116:0/1735501297 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dfc0c82c0 con 0x7f0dfc06cf80 2024-04-25T00:24:20.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.388+0000 7f0dfb7fe640 1 --2- 172.21.15.116:0/1735501297 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0dfc06cf80 0x7f0dfc0c7a90 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f0df0009870 tx=0x7f0df001c460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=149856e1e87f1a1b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:20.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.388+0000 7f0dfaffd640 1 -- 172.21.15.116:0/1735501297 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0df00b0070 con 0x7f0dfc06cf80 2024-04-25T00:24:20.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.388+0000 7f0dfaffd640 1 -- 172.21.15.116:0/1735501297 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0df00ab600 con 0x7f0dfc06cf80 2024-04-25T00:24:20.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.388+0000 7f0dfaffd640 1 -- 172.21.15.116:0/1735501297 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0df00ab9d0 con 0x7f0dfc06cf80 2024-04-25T00:24:20.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.388+0000 7f0e0176e640 1 -- 172.21.15.116:0/1735501297 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0dfc06cf80 msgr2=0x7f0dfc0c7a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:20.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.388+0000 7f0e0176e640 1 --2- 172.21.15.116:0/1735501297 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0dfc06cf80 0x7f0dfc0c7a90 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f0df0009870 tx=0x7f0df001c460 comp rx=0 tx=0).stop 2024-04-25T00:24:20.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.388+0000 7f0e0176e640 1 -- 172.21.15.116:0/1735501297 shutdown_connections 2024-04-25T00:24:20.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.388+0000 7f0e0176e640 1 --2- 172.21.15.116:0/1735501297 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0dfc06cf80 0x7f0dfc0c7a90 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:20.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.388+0000 7f0e0176e640 1 --2- 172.21.15.116:0/1735501297 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0dfc0b0830 0x7f0dfc159960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:20.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.388+0000 7f0e0176e640 1 -- 172.21.15.116:0/1735501297 >> 172.21.15.116:0/1735501297 conn(0x7f0dfc0acd60 msgr2=0x7f0dfc0ad170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:20.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.388+0000 7f0e0176e640 1 -- 172.21.15.116:0/1735501297 shutdown_connections 2024-04-25T00:24:20.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.392+0000 7f0e0176e640 1 -- 172.21.15.116:0/1735501297 wait complete. 2024-04-25T00:24:20.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.392+0000 7f0e0176e640 1 Processor -- start 2024-04-25T00:24:20.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.392+0000 7f0e0176e640 1 -- start start 2024-04-25T00:24:20.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.392+0000 7f0e0176e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0dfc06cf80 0x7f0dfc1f21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:20.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.392+0000 7f0e0176e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0dfc0b0830 0x7f0dfc0af400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:20.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.392+0000 7f0e0176e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0dfc0c8f80 0x7f0dfc154e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:20.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.396+0000 7f0dfb7fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0dfc0b0830 0x7f0dfc0af400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:20.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.396+0000 7f0e00f6d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0dfc0c8f80 0x7f0dfc154e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:20.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.396+0000 7f0e00f6d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0dfc0c8f80 0x7f0dfc154e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56022/0 (socket says 172.21.15.116:56022) 2024-04-25T00:24:20.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.396+0000 7f0dfb7fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0dfc0b0830 0x7f0dfc0af400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:43382/0 (socket says 172.21.15.116:43382) 2024-04-25T00:24:20.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.396+0000 7f0e00f6d640 1 -- 172.21.15.116:0/3140463202 learned_addr learned my addr 172.21.15.116:0/3140463202 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:24:20.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.396+0000 7f0e0176e640 1 -- 172.21.15.116:0/3140463202 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0dfc0c6110 con 0x7f0dfc0c8f80 2024-04-25T00:24:20.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.396+0000 7f0e0176e640 1 -- 172.21.15.116:0/3140463202 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0dfc0c7fd0 con 0x7f0dfc06cf80 2024-04-25T00:24:20.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.396+0000 7f0e0176e640 1 -- 172.21.15.116:0/3140463202 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f0dfc0c6290 con 0x7f0dfc0b0830 2024-04-25T00:24:20.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.396+0000 7f0e00f6d640 1 -- 172.21.15.116:0/3140463202 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0dfc0b0830 msgr2=0x7f0dfc0af400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:20.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.396+0000 7f0e00f6d640 1 --2- 172.21.15.116:0/3140463202 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0dfc0b0830 0x7f0dfc0af400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:20.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.396+0000 7f0e00f6d640 1 -- 172.21.15.116:0/3140463202 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0dfc06cf80 msgr2=0x7f0dfc1f21e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T00:24:20.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.396+0000 7f0e00f6d640 1 --2- 172.21.15.116:0/3140463202 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0dfc06cf80 0x7f0dfc1f21e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:20.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.396+0000 7f0e00f6d640 1 -- 172.21.15.116:0/3140463202 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dfc14d190 con 0x7f0dfc0c8f80 2024-04-25T00:24:20.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.396+0000 7f0e00f6d640 1 --2- 172.21.15.116:0/3140463202 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0dfc0c8f80 0x7f0dfc154e50 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f0df4002120 tx=0x7f0df4014420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:20.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.400+0000 7f0df8ff9640 1 -- 172.21.15.116:0/3140463202 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0df4014e00 con 0x7f0dfc0c8f80 2024-04-25T00:24:20.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.400+0000 7f0df8ff9640 1 -- 172.21.15.116:0/3140463202 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0df4004b70 con 0x7f0dfc0c8f80 2024-04-25T00:24:20.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.400+0000 7f0e0176e640 1 -- 172.21.15.116:0/3140463202 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dfc14d470 con 0x7f0dfc0c8f80 2024-04-25T00:24:20.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.400+0000 7f0df8ff9640 1 -- 172.21.15.116:0/3140463202 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0df4010850 con 0x7f0dfc0c8f80 2024-04-25T00:24:20.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.400+0000 7f0e0176e640 1 -- 172.21.15.116:0/3140463202 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dfc0af940 con 0x7f0dfc0c8f80 2024-04-25T00:24:20.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.404+0000 7f0e0176e640 1 -- 172.21.15.116:0/3140463202 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0dfc04f510 con 0x7f0dfc0c8f80 2024-04-25T00:24:20.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.404+0000 7f0df8ff9640 1 -- 172.21.15.116:0/3140463202 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f0df40109f0 con 0x7f0dfc0c8f80 2024-04-25T00:24:20.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.404+0000 7f0df8ff9640 1 --2- 172.21.15.116:0/3140463202 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0de8067910 0x7f0de8069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:20.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.404+0000 7f0df8ff9640 1 -- 172.21.15.116:0/3140463202 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 9848+0+0 (secure 0 0 0) 0x7f0df408efb0 con 0x7f0dfc0c8f80 2024-04-25T00:24:20.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.408+0000 7f0dfbfff640 1 --2- 172.21.15.116:0/3140463202 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0de8067910 0x7f0de8069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:20.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.408+0000 7f0dfbfff640 1 --2- 172.21.15.116:0/3140463202 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0de8067910 0x7f0de8069dd0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f0dec009670 tx=0x7f0dec006b90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:20.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.412+0000 7f0df8ff9640 1 -- 172.21.15.116:0/3140463202 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f0df405ea90 con 0x7f0dfc0c8f80 2024-04-25T00:24:21.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:20.996+0000 7f0e0176e640 1 -- 172.21.15.116:0/3140463202 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0dfc06d850 con 0x7f0dfc0c8f80 2024-04-25T00:24:21.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:21.000+0000 7f0df8ff9640 1 -- 172.21.15.116:0/3140463202 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 196 v196) ==== 95+0+5681 (secure 0 0 0) 0x7f0df405e390 con 0x7f0dfc0c8f80 2024-04-25T00:24:21.008 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:24:21.008 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":196,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":191,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:23:34.013510+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8244":{"gid":8244,"name":"c","rank":0,"incarnation":188,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6835/1971675962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1971675962},{"type":"v1","addr":"172.21.15.116:6835","nonce":1971675962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8244,"qdb_cluster":[8244]},"id":1},{"mdsmap":{"epoch":196,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:24:19.977863+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8343":{"gid":8343,"name":"a","rank":0,"incarnation":193,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6833/3633657268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3633657268},{"type":"v1","addr":"172.21.15.116:6833","nonce":3633657268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8343,"qdb_cluster":[8343]},"id":2}]} 2024-04-25T00:24:21.009 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 196 2024-04-25T00:24:21.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:21.004+0000 7f0de27fc640 1 -- 172.21.15.116:0/3140463202 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0de8067910 msgr2=0x7f0de8069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:21.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:21.004+0000 7f0de27fc640 1 --2- 172.21.15.116:0/3140463202 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0de8067910 0x7f0de8069dd0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f0dec009670 tx=0x7f0dec006b90 comp rx=0 tx=0).stop 2024-04-25T00:24:21.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:21.004+0000 7f0de27fc640 1 -- 172.21.15.116:0/3140463202 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0dfc0c8f80 msgr2=0x7f0dfc154e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:21.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:21.004+0000 7f0de27fc640 1 --2- 172.21.15.116:0/3140463202 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0dfc0c8f80 0x7f0dfc154e50 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f0df4002120 tx=0x7f0df4014420 comp rx=0 tx=0).stop 2024-04-25T00:24:21.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:21.004+0000 7f0de27fc640 1 -- 172.21.15.116:0/3140463202 shutdown_connections 2024-04-25T00:24:21.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:21.004+0000 7f0de27fc640 1 --2- 172.21.15.116:0/3140463202 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0de8067910 0x7f0de8069dd0 secure :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f0dec009670 tx=0x7f0dec006b90 comp rx=0 tx=0).stop 2024-04-25T00:24:21.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:21.004+0000 7f0de27fc640 1 --2- 172.21.15.116:0/3140463202 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0dfc0c8f80 0x7f0dfc154e50 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:21.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:21.004+0000 7f0de27fc640 1 --2- 172.21.15.116:0/3140463202 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0dfc0b0830 0x7f0dfc0af400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:21.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:21.004+0000 7f0de27fc640 1 --2- 172.21.15.116:0/3140463202 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0dfc06cf80 0x7f0dfc1f21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:21.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:21.004+0000 7f0de27fc640 1 -- 172.21.15.116:0/3140463202 >> 172.21.15.116:0/3140463202 conn(0x7f0dfc0acd60 msgr2=0x7f0dfc0aab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:21.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:21.004+0000 7f0de27fc640 1 -- 172.21.15.116:0/3140463202 shutdown_connections 2024-04-25T00:24:21.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:21.004+0000 7f0de27fc640 1 -- 172.21.15.116:0/3140463202 wait complete. 2024-04-25T00:24:21.033 INFO:tasks.mds_thrash.fs.[b]:mds.a has gained rank=0, replacing gid=6230 2024-04-25T00:24:21.033 INFO:tasks.mds_thrash.fs.[b]:waiting for 6 secs before reviving mds.d 2024-04-25T00:24:22.952 INFO:tasks.ceph.mds.a.smithi116.stderr:2024-04-25T00:24:22.948+0000 7f2c31930640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T00:24:27.034 INFO:tasks.mds_thrash.fs.[b]:reviving mds.d 2024-04-25T00:24:27.034 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-25T00:24:27.035 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-25T00:24:27.037 INFO:tasks.ceph.mds.d:Started 2024-04-25T00:24:27.037 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:24:27.113 INFO:tasks.ceph.mds.d.smithi174.stdout:starting mds.d at 2024-04-25T00:24:27.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.344+0000 7f6eb07c6640 1 Processor -- start 2024-04-25T00:24:27.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.344+0000 7f6eb07c6640 1 -- start start 2024-04-25T00:24:27.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.348+0000 7f6eb07c6640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f6ea80c36c0 0x7f6ea80c2df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:27.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.348+0000 7f6eb07c6640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ea8070fd0 con 0x7f6ea80c2a10 2024-04-25T00:24:27.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.348+0000 7f6eb07c6640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ea806e830 con 0x7f6ea81631d0 2024-04-25T00:24:27.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.348+0000 7f6eb07c6640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f6ea80712b0 con 0x7f6ea80c36c0 2024-04-25T00:24:27.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.348+0000 7f6eaed63640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f6ea80c36c0 0x7f6ea80c2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:27.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.348+0000 7f6eaed63640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f6ea80c36c0 0x7f6ea80c2df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:47676/0 (socket says 172.21.15.116:47676) 2024-04-25T00:24:27.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.348+0000 7f6eaed63640 1 -- 172.21.15.116:0/1541771675 learned_addr learned my addr 172.21.15.116:0/1541771675 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:24:27.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.348+0000 7f6eaed63640 1 -- 172.21.15.116:0/1541771675 >> v1:172.21.15.174:6789/0 conn(0x7f6ea81631d0 legacy=0x7f6ea816ad40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T00:24:27.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.348+0000 7f6eaed63640 1 -- 172.21.15.116:0/1541771675 >> v1:172.21.15.116:6789/0 conn(0x7f6ea80c2a10 legacy=0x7f6ea8162ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:27.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.348+0000 7f6eaed63640 1 -- 172.21.15.116:0/1541771675 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ea806eb10 con 0x7f6ea80c36c0 2024-04-25T00:24:27.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.352+0000 7f6eaed63640 1 --2- 172.21.15.116:0/1541771675 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6ea80c36c0 0x7f6ea80c2df0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0006140 tx=0x7f6ea0025120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d2474655c19bbf0b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:27.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.352+0000 7f6ead560640 1 -- 172.21.15.116:0/1541771675 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea00bb070 con 0x7f6ea80c36c0 2024-04-25T00:24:27.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.352+0000 7f6ead560640 1 -- 172.21.15.116:0/1541771675 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ea00b6410 con 0x7f6ea80c36c0 2024-04-25T00:24:27.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.352+0000 7f6ead560640 1 -- 172.21.15.116:0/1541771675 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea00b67e0 con 0x7f6ea80c36c0 2024-04-25T00:24:27.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.352+0000 7f6eb07c6640 1 -- 172.21.15.116:0/1541771675 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6ea80c36c0 msgr2=0x7f6ea80c2df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:27.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.352+0000 7f6eb07c6640 1 --2- 172.21.15.116:0/1541771675 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6ea80c36c0 0x7f6ea80c2df0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0006140 tx=0x7f6ea0025120 comp rx=0 tx=0).stop 2024-04-25T00:24:27.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.352+0000 7f6eb07c6640 1 -- 172.21.15.116:0/1541771675 shutdown_connections 2024-04-25T00:24:27.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.352+0000 7f6eb07c6640 1 --2- 172.21.15.116:0/1541771675 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6ea80c36c0 0x7f6ea80c2df0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.352+0000 7f6eb07c6640 1 -- 172.21.15.116:0/1541771675 >> 172.21.15.116:0/1541771675 conn(0x7f6ea80c3b00 msgr2=0x7f6ea80c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:27.356 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.352+0000 7f6eb07c6640 1 -- 172.21.15.116:0/1541771675 shutdown_connections 2024-04-25T00:24:27.357 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.352+0000 7f6eb07c6640 1 -- 172.21.15.116:0/1541771675 wait complete. 2024-04-25T00:24:27.358 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.356+0000 7f6eb07c6640 1 Processor -- start 2024-04-25T00:24:27.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.356+0000 7f6eb07c6640 1 -- start start 2024-04-25T00:24:27.359 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.356+0000 7f6eb07c6640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6ea80c2a10 0x7f6ea81531f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:27.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.356+0000 7f6eb07c6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ea80c36c0 0x7f6ea80ac020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:27.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.356+0000 7f6eb07c6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6ea81631d0 0x7f6ea80ac560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:27.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.356+0000 7f6eb07c6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea80718b0 con 0x7f6ea80c36c0 2024-04-25T00:24:27.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.356+0000 7f6eb07c6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6ea8071730 con 0x7f6ea81631d0 2024-04-25T00:24:27.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.356+0000 7f6eb07c6640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6ea806cb50 con 0x7f6ea80c2a10 2024-04-25T00:24:27.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.356+0000 7f6eadd61640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ea80c36c0 0x7f6ea80ac020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:27.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.356+0000 7f6eadd61640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ea80c36c0 0x7f6ea80ac020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46798/0 (socket says 172.21.15.116:46798) 2024-04-25T00:24:27.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.356+0000 7f6eadd61640 1 -- 172.21.15.116:0/1467800433 learned_addr learned my addr 172.21.15.116:0/1467800433 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:24:27.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.356+0000 7f6eaed63640 1 --2- 172.21.15.116:0/1467800433 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6ea81631d0 0x7f6ea80ac560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:27.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.356+0000 7f6eae562640 1 --2- 172.21.15.116:0/1467800433 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6ea80c2a10 0x7f6ea81531f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:27.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.356+0000 7f6eaed63640 1 -- 172.21.15.116:0/1467800433 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6ea80c2a10 msgr2=0x7f6ea81531f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:27.362 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.356+0000 7f6eaed63640 1 --2- 172.21.15.116:0/1467800433 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6ea80c2a10 0x7f6ea81531f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.356+0000 7f6eaed63640 1 -- 172.21.15.116:0/1467800433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ea80c36c0 msgr2=0x7f6ea80ac020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:27.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.356+0000 7f6eaed63640 1 --2- 172.21.15.116:0/1467800433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ea80c36c0 0x7f6ea80ac020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.356+0000 7f6eaed63640 1 -- 172.21.15.116:0/1467800433 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ea80acaa0 con 0x7f6ea81631d0 2024-04-25T00:24:27.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.356+0000 7f6eadd61640 1 --2- 172.21.15.116:0/1467800433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ea80c36c0 0x7f6ea80ac020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:24:27.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.360+0000 7f6eaed63640 1 --2- 172.21.15.116:0/1467800433 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6ea81631d0 0x7f6ea80ac560 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f6ea80c49b0 tx=0x7f6ea0021ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:27.363 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.360+0000 7f6e9effd640 1 -- 172.21.15.116:0/1467800433 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea00bb040 con 0x7f6ea81631d0 2024-04-25T00:24:27.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.360+0000 7f6e9effd640 1 -- 172.21.15.116:0/1467800433 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ea00c6070 con 0x7f6ea81631d0 2024-04-25T00:24:27.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.360+0000 7f6e9effd640 1 -- 172.21.15.116:0/1467800433 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea00c1850 con 0x7f6ea81631d0 2024-04-25T00:24:27.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.360+0000 7f6eb07c6640 1 -- 172.21.15.116:0/1467800433 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ea815b4e0 con 0x7f6ea81631d0 2024-04-25T00:24:27.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.360+0000 7f6eb07c6640 1 -- 172.21.15.116:0/1467800433 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ea8154f90 con 0x7f6ea81631d0 2024-04-25T00:24:27.366 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.360+0000 7f6eb07c6640 1 -- 172.21.15.116:0/1467800433 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ea8161350 con 0x7f6ea81631d0 2024-04-25T00:24:27.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.364+0000 7f6e9effd640 1 -- 172.21.15.116:0/1467800433 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6ea00be070 con 0x7f6ea81631d0 2024-04-25T00:24:27.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.364+0000 7f6e9effd640 1 --2- 172.21.15.116:0/1467800433 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6e8c0678f0 0x7f6e8c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:24:27.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.364+0000 7f6eae562640 1 --2- 172.21.15.116:0/1467800433 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6e8c0678f0 0x7f6e8c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:24:27.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.368+0000 7f6eae562640 1 --2- 172.21.15.116:0/1467800433 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6e8c0678f0 0x7f6e8c069db0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f6ea806e020 tx=0x7f6ea4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:24:27.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.368+0000 7f6e9effd640 1 -- 172.21.15.116:0/1467800433 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 9848+0+0 (secure 0 0 0) 0x7f6ea00f7ad0 con 0x7f6ea81631d0 2024-04-25T00:24:27.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.368+0000 7f6e9effd640 1 -- 172.21.15.116:0/1467800433 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f6ea8161350 con 0x7f6ea81631d0 2024-04-25T00:24:27.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.956+0000 7f6eb07c6640 1 -- 172.21.15.116:0/1467800433 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6ea8068350 con 0x7f6ea81631d0 2024-04-25T00:24:27.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.956+0000 7f6e9effd640 1 -- 172.21.15.116:0/1467800433 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 197 v197) ==== 95+0+6510 (secure 0 0 0) 0x7f6ea8161350 con 0x7f6ea81631d0 2024-04-25T00:24:27.961 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:24:27.962 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":197,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6413,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/4181466530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":4181466530},{"type":"v1","addr":"172.21.15.174:6837","nonce":4181466530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":197}],"filesystems":[{"mdsmap":{"epoch":191,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:23:34.013510+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8244":{"gid":8244,"name":"c","rank":0,"incarnation":188,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6835/1971675962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1971675962},{"type":"v1","addr":"172.21.15.116:6835","nonce":1971675962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8244,"qdb_cluster":[8244]},"id":1},{"mdsmap":{"epoch":196,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:24:19.977863+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8343":{"gid":8343,"name":"a","rank":0,"incarnation":193,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6833/3633657268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3633657268},{"type":"v1","addr":"172.21.15.116:6833","nonce":3633657268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8343,"qdb_cluster":[8343]},"id":2}]} 2024-04-25T00:24:27.962 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 197 2024-04-25T00:24:27.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.972+0000 7f6eb07c6640 1 -- 172.21.15.116:0/1467800433 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6e8c0678f0 msgr2=0x7f6e8c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:27.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.972+0000 7f6eb07c6640 1 --2- 172.21.15.116:0/1467800433 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6e8c0678f0 0x7f6e8c069db0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f6ea806e020 tx=0x7f6ea4009040 comp rx=0 tx=0).stop 2024-04-25T00:24:27.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.972+0000 7f6eb07c6640 1 -- 172.21.15.116:0/1467800433 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6ea81631d0 msgr2=0x7f6ea80ac560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:24:27.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.972+0000 7f6eb07c6640 1 --2- 172.21.15.116:0/1467800433 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6ea81631d0 0x7f6ea80ac560 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f6ea80c49b0 tx=0x7f6ea0021ec0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.972+0000 7f6eb07c6640 1 -- 172.21.15.116:0/1467800433 shutdown_connections 2024-04-25T00:24:27.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.972+0000 7f6eb07c6640 1 --2- 172.21.15.116:0/1467800433 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6e8c0678f0 0x7f6e8c069db0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.972+0000 7f6eb07c6640 1 --2- 172.21.15.116:0/1467800433 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6ea81631d0 0x7f6ea80ac560 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.972+0000 7f6eb07c6640 1 --2- 172.21.15.116:0/1467800433 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ea80c36c0 0x7f6ea80ac020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.972+0000 7f6eb07c6640 1 --2- 172.21.15.116:0/1467800433 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6ea80c2a10 0x7f6ea81531f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:24:27.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.972+0000 7f6eb07c6640 1 -- 172.21.15.116:0/1467800433 >> 172.21.15.116:0/1467800433 conn(0x7f6ea80c3b00 msgr2=0x7f6ea80c3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:24:27.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.972+0000 7f6eb07c6640 1 -- 172.21.15.116:0/1467800433 shutdown_connections 2024-04-25T00:24:27.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:24:27.972+0000 7f6eb07c6640 1 -- 172.21.15.116:0/1467800433 wait complete. 2024-04-25T00:24:28.002 INFO:tasks.mds_thrash.fs.[b]:mds.d reported in up:standby state 2024-04-25T00:24:28.002 INFO:tasks.mds_thrash.fs.[b]:waiting for 55 secs before thrashing 2024-04-25T00:24:40.475 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:24:40.478 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:24:50.224 INFO:tasks.workunit.client.1.smithi174.stdout:fs setup took 1264 secs 2024-04-25T00:24:54.351 INFO:tasks.workunit.client.1.smithi174.stdout:Syncing()...4 sec 2024-04-25T00:24:54.351 INFO:tasks.workunit.client.1.smithi174.stdout:Starting Actual Benchmark At: Thu Apr 25 00:24:54 2024 2024-04-25T00:24:54.351 INFO:tasks.workunit.client.1.smithi174.stdout: 2024-04-25T00:25:10.749 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:25:10.850 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:25:11.158 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:25:11.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:11.876+0000 7f5a06296640 1 Processor -- start 2024-04-25T00:25:11.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:11.876+0000 7f5a06296640 1 -- start start 2024-04-25T00:25:11.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:11.876+0000 7f5a06296640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f5a00152200 0x7f5a0006dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:11.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:11.876+0000 7f5a06296640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5a00071c80 0x7f5a000720f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:11.880 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:11.876+0000 7f5a06296640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a00151740 con 0x7f5a0013aa00 2024-04-25T00:25:11.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:11.876+0000 7f5a06296640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f5a000c2630 con 0x7f5a00152200 2024-04-25T00:25:11.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:11.876+0000 7f5a06296640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f5a000c2910 con 0x7f5a00071c80 2024-04-25T00:25:11.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:11.876+0000 7f5a05a95640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5a00071c80 0x7f5a000720f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:11.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:11.876+0000 7f5a05a95640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5a00071c80 0x7f5a000720f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:40866/0 (socket says 172.21.15.116:40866) 2024-04-25T00:25:11.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:11.876+0000 7f5a04a93640 1 --2- 172.21.15.116:0/2256144049 >> v2:172.21.15.174:3300/0 conn(0x7f5a00152200 0x7f5a0006dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:11.881 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:11.876+0000 7f5a05a95640 1 -- 172.21.15.116:0/2256144049 learned_addr learned my addr 172.21.15.116:0/2256144049 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:25:11.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:11.880+0000 7f5a04a93640 1 -- 172.21.15.116:0/2256144049 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5a00071c80 msgr2=0x7f5a000720f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:11.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:11.880+0000 7f5a04a93640 1 --2- 172.21.15.116:0/2256144049 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5a00071c80 0x7f5a000720f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:11.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:11.880+0000 7f5a04a93640 1 -- 172.21.15.116:0/2256144049 >> v1:172.21.15.174:6789/0 conn(0x7f5a0013aa00 legacy=0x7f5a001556f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:11.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:11.880+0000 7f5a04a93640 1 -- 172.21.15.116:0/2256144049 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a00151a00 con 0x7f5a00152200 2024-04-25T00:25:11.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:11.888+0000 7f5a04a93640 1 --2- 172.21.15.116:0/2256144049 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5a00152200 0x7f5a0006dc00 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f59f800d1e0 tx=0x7f59f801e850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54cc70eb20e24282 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:11.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:11.916+0000 7f59f7fff640 1 -- 172.21.15.116:0/2256144049 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f80ac070 con 0x7f5a00152200 2024-04-25T00:25:11.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:11.916+0000 7f59f7fff640 1 -- 172.21.15.116:0/2256144049 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59f80b0400 con 0x7f5a00152200 2024-04-25T00:25:11.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:11.916+0000 7f59f7fff640 1 -- 172.21.15.116:0/2256144049 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f80b07f0 con 0x7f5a00152200 2024-04-25T00:25:11.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:11.916+0000 7f5a06296640 1 -- 172.21.15.116:0/2256144049 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5a00152200 msgr2=0x7f5a0006dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:11.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:11.916+0000 7f5a06296640 1 --2- 172.21.15.116:0/2256144049 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5a00152200 0x7f5a0006dc00 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f59f800d1e0 tx=0x7f59f801e850 comp rx=0 tx=0).stop 2024-04-25T00:25:11.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:11.916+0000 7f5a06296640 1 -- 172.21.15.116:0/2256144049 shutdown_connections 2024-04-25T00:25:11.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:11.916+0000 7f5a06296640 1 --2- 172.21.15.116:0/2256144049 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5a00071c80 0x7f5a000720f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:11.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:11.916+0000 7f5a06296640 1 --2- 172.21.15.116:0/2256144049 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5a00152200 0x7f5a0006dc00 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:11.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:11.916+0000 7f5a06296640 1 -- 172.21.15.116:0/2256144049 >> 172.21.15.116:0/2256144049 conn(0x7f5a000c6db0 msgr2=0x7f5a000c71c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:11.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:11.916+0000 7f5a06296640 1 -- 172.21.15.116:0/2256144049 shutdown_connections 2024-04-25T00:25:11.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:11.916+0000 7f5a06296640 1 -- 172.21.15.116:0/2256144049 wait complete. 2024-04-25T00:25:11.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:11.920+0000 7f5a06296640 1 Processor -- start 2024-04-25T00:25:11.923 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:11.920+0000 7f5a06296640 1 -- start start 2024-04-25T00:25:11.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:11.920+0000 7f5a06296640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5a00071c80 0x7f5a00130c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:11.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:11.920+0000 7f5a06296640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a0013aa00 0x7f5a0012a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:11.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:11.920+0000 7f5a06296640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5a00152200 0x7f5a0012adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:11.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:11.920+0000 7f5a06296640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5a00163690 con 0x7f5a0013aa00 2024-04-25T00:25:11.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:11.920+0000 7f5a06296640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5a00163510 con 0x7f5a00152200 2024-04-25T00:25:11.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:11.920+0000 7f5a06296640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f5a00163810 con 0x7f5a00071c80 2024-04-25T00:25:11.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:11.920+0000 7f5a05294640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5a00071c80 0x7f5a00130c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:11.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:11.920+0000 7f5a04a93640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a0013aa00 0x7f5a0012a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:11.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:11.920+0000 7f5a04a93640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a0013aa00 0x7f5a0012a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42128/0 (socket says 172.21.15.116:42128) 2024-04-25T00:25:11.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:11.920+0000 7f5a04a93640 1 -- 172.21.15.116:0/3857387155 learned_addr learned my addr 172.21.15.116:0/3857387155 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:25:11.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:11.924+0000 7f5a05294640 1 -- 172.21.15.116:0/3857387155 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5a00152200 msgr2=0x7f5a0012adf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T00:25:11.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:11.924+0000 7f5a05294640 1 --2- 172.21.15.116:0/3857387155 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5a00152200 0x7f5a0012adf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:11.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:11.924+0000 7f5a05294640 1 -- 172.21.15.116:0/3857387155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a0013aa00 msgr2=0x7f5a0012a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:11.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:11.924+0000 7f5a05294640 1 --2- 172.21.15.116:0/3857387155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a0013aa00 0x7f5a0012a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:11.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:11.924+0000 7f5a05294640 1 -- 172.21.15.116:0/3857387155 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a0014e850 con 0x7f5a00071c80 2024-04-25T00:25:11.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:11.924+0000 7f5a05294640 1 --2- 172.21.15.116:0/3857387155 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5a00071c80 0x7f5a00130c10 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f59fc00c550 tx=0x7f59fc00ca10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:11.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:11.924+0000 7f59f5ffb640 1 -- 172.21.15.116:0/3857387155 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59fc028070 con 0x7f5a00071c80 2024-04-25T00:25:11.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:11.924+0000 7f5a06296640 1 -- 172.21.15.116:0/3857387155 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a0006f240 con 0x7f5a00071c80 2024-04-25T00:25:11.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:11.924+0000 7f5a06296640 1 -- 172.21.15.116:0/3857387155 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a00153ea0 con 0x7f5a00071c80 2024-04-25T00:25:11.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:11.924+0000 7f5a06296640 1 -- 172.21.15.116:0/3857387155 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a0004f510 con 0x7f5a00071c80 2024-04-25T00:25:11.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:11.924+0000 7f59f5ffb640 1 -- 172.21.15.116:0/3857387155 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59fc005280 con 0x7f5a00071c80 2024-04-25T00:25:11.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:11.924+0000 7f59f5ffb640 1 -- 172.21.15.116:0/3857387155 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59fc003600 con 0x7f5a00071c80 2024-04-25T00:25:11.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:11.928+0000 7f59f5ffb640 1 -- 172.21.15.116:0/3857387155 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f59fc02b070 con 0x7f5a00071c80 2024-04-25T00:25:11.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:11.932+0000 7f59f5ffb640 1 --2- 172.21.15.116:0/3857387155 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f59e40678c0 0x7f59e4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:11.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:11.932+0000 7f59f5ffb640 1 -- 172.21.15.116:0/3857387155 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(65..65 src has 1..65) ==== 9848+0+0 (secure 0 0 0) 0x7f59fc096ef0 con 0x7f5a00071c80 2024-04-25T00:25:11.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:11.932+0000 7f5a04a93640 1 --2- 172.21.15.116:0/3857387155 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f59e40678c0 0x7f59e4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:11.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:11.940+0000 7f5a04a93640 1 --2- 172.21.15.116:0/3857387155 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f59e40678c0 0x7f59e4069d80 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f59f801f0f0 tx=0x7f59f800a040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:11.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:11.952+0000 7f59f5ffb640 1 -- 172.21.15.116:0/3857387155 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f59fc0c35c0 con 0x7f5a00071c80 2024-04-25T00:25:12.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:12.696+0000 7f5a06296640 1 -- 172.21.15.116:0/3857387155 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5a0014fa30 con 0x7f5a00071c80 2024-04-25T00:25:12.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:12.700+0000 7f59f5ffb640 1 -- 172.21.15.116:0/3857387155 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 197 v197) ==== 95+0+6510 (secure 0 0 0) 0x7f59fc066470 con 0x7f5a00071c80 2024-04-25T00:25:12.705 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:25:12.705 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":197,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6413,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/4181466530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":4181466530},{"type":"v1","addr":"172.21.15.174:6837","nonce":4181466530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":197}],"filesystems":[{"mdsmap":{"epoch":191,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:23:34.013510+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8244":{"gid":8244,"name":"c","rank":0,"incarnation":188,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6835/1971675962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1971675962},{"type":"v1","addr":"172.21.15.116:6835","nonce":1971675962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8244,"qdb_cluster":[8244]},"id":1},{"mdsmap":{"epoch":196,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:24:19.977863+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8343":{"gid":8343,"name":"a","rank":0,"incarnation":193,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6833/3633657268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3633657268},{"type":"v1","addr":"172.21.15.116:6833","nonce":3633657268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8343,"qdb_cluster":[8343]},"id":2}]} 2024-04-25T00:25:12.705 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 197 2024-04-25T00:25:12.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:12.712+0000 7f5a06296640 1 -- 172.21.15.116:0/3857387155 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f59e40678c0 msgr2=0x7f59e4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:12.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:12.712+0000 7f5a06296640 1 --2- 172.21.15.116:0/3857387155 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f59e40678c0 0x7f59e4069d80 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f59f801f0f0 tx=0x7f59f800a040 comp rx=0 tx=0).stop 2024-04-25T00:25:12.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:12.712+0000 7f5a06296640 1 -- 172.21.15.116:0/3857387155 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5a00071c80 msgr2=0x7f5a00130c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:12.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:12.712+0000 7f5a06296640 1 --2- 172.21.15.116:0/3857387155 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5a00071c80 0x7f5a00130c10 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f59fc00c550 tx=0x7f59fc00ca10 comp rx=0 tx=0).stop 2024-04-25T00:25:12.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:12.716+0000 7f5a06296640 1 -- 172.21.15.116:0/3857387155 shutdown_connections 2024-04-25T00:25:12.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:12.732+0000 7f5a06296640 1 --2- 172.21.15.116:0/3857387155 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f59e40678c0 0x7f59e4069d80 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:12.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:12.732+0000 7f5a06296640 1 --2- 172.21.15.116:0/3857387155 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5a00152200 0x7f5a0012adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:12.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:12.732+0000 7f5a06296640 1 --2- 172.21.15.116:0/3857387155 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a0013aa00 0x7f5a0012a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:12.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:12.732+0000 7f5a06296640 1 --2- 172.21.15.116:0/3857387155 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5a00071c80 0x7f5a00130c10 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:12.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:12.736+0000 7f5a06296640 1 -- 172.21.15.116:0/3857387155 >> 172.21.15.116:0/3857387155 conn(0x7f5a000c6db0 msgr2=0x7f5a0012eb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:12.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:12.736+0000 7f5a06296640 1 -- 172.21.15.116:0/3857387155 shutdown_connections 2024-04-25T00:25:12.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:12.736+0000 7f5a06296640 1 -- 172.21.15.116:0/3857387155 wait complete. 2024-04-25T00:25:12.852 INFO:tasks.mds_thrash.fs.[a]:kill mds.c (rank=0) 2024-04-25T00:25:12.852 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-25T00:25:12.852 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:25:12.869 INFO:tasks.ceph.mds.c:Stopped 2024-04-25T00:25:12.870 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:25:13.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:13.384+0000 7f7d802a8640 1 Processor -- start 2024-04-25T00:25:13.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:13.384+0000 7f7d802a8640 1 -- start start 2024-04-25T00:25:13.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:13.384+0000 7f7d802a8640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d7814b830 0x7f7d78152150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:13.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:13.384+0000 7f7d802a8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7d78070dd0 0x7f7d78071230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:13.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:13.384+0000 7f7d802a8640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d78152690 con 0x7f7d7806e100 2024-04-25T00:25:13.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:13.384+0000 7f7d802a8640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f7d7814bc10 con 0x7f7d78070dd0 2024-04-25T00:25:13.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:13.384+0000 7f7d802a8640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f7d78154df0 con 0x7f7d7814b830 2024-04-25T00:25:13.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:13.384+0000 7f7d7e044640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d7814b830 0x7f7d78152150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:13.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:13.384+0000 7f7d7e044640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d7814b830 0x7f7d78152150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:40886/0 (socket says 172.21.15.116:40886) 2024-04-25T00:25:13.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:13.384+0000 7f7d7e044640 1 -- 172.21.15.116:0/2618292855 learned_addr learned my addr 172.21.15.116:0/2618292855 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:25:13.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:13.384+0000 7f7d7e044640 1 -- 172.21.15.116:0/2618292855 >> v2:172.21.15.116:3300/0 conn(0x7f7d78070dd0 msgr2=0x7f7d78071230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T00:25:13.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:13.384+0000 7f7d7e044640 1 --2- 172.21.15.116:0/2618292855 >> v2:172.21.15.116:3300/0 conn(0x7f7d78070dd0 0x7f7d78071230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:13.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:13.384+0000 7f7d7e044640 1 -- 172.21.15.116:0/2618292855 >> v1:172.21.15.116:6789/0 conn(0x7f7d7806e100 legacy=0x7f7d780c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:13.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:13.384+0000 7f7d7e044640 1 -- 172.21.15.116:0/2618292855 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d781550d0 con 0x7f7d7814b830 2024-04-25T00:25:13.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:13.404+0000 7f7d7e044640 1 --2- 172.21.15.116:0/2618292855 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d7814b830 0x7f7d78152150 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7d68009870 tx=0x7f7d6801e460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cba4910693cba0c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:13.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:13.404+0000 7f7d7d042640 1 -- 172.21.15.116:0/2618292855 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d680b2070 con 0x7f7d7814b830 2024-04-25T00:25:13.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:13.404+0000 7f7d7d042640 1 -- 172.21.15.116:0/2618292855 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d680ad600 con 0x7f7d7814b830 2024-04-25T00:25:13.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:13.404+0000 7f7d802a8640 1 -- 172.21.15.116:0/2618292855 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d7814b830 msgr2=0x7f7d78152150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:13.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:13.404+0000 7f7d802a8640 1 --2- 172.21.15.116:0/2618292855 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d7814b830 0x7f7d78152150 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7d68009870 tx=0x7f7d6801e460 comp rx=0 tx=0).stop 2024-04-25T00:25:13.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:13.404+0000 7f7d802a8640 1 -- 172.21.15.116:0/2618292855 shutdown_connections 2024-04-25T00:25:13.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:13.404+0000 7f7d802a8640 1 --2- 172.21.15.116:0/2618292855 >> v2:172.21.15.116:3300/0 conn(0x7f7d78070dd0 0x7f7d78071230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:13.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:13.404+0000 7f7d802a8640 1 --2- 172.21.15.116:0/2618292855 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d7814b830 0x7f7d78152150 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:13.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:13.404+0000 7f7d802a8640 1 -- 172.21.15.116:0/2618292855 >> 172.21.15.116:0/2618292855 conn(0x7f7d780c3b00 msgr2=0x7f7d780c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:13.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:13.404+0000 7f7d802a8640 1 -- 172.21.15.116:0/2618292855 shutdown_connections 2024-04-25T00:25:13.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:13.404+0000 7f7d802a8640 1 -- 172.21.15.116:0/2618292855 wait complete. 2024-04-25T00:25:13.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:13.408+0000 7f7d802a8640 1 Processor -- start 2024-04-25T00:25:13.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:13.408+0000 7f7d802a8640 1 -- start start 2024-04-25T00:25:13.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:13.408+0000 7f7d802a8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d7806e100 0x7f7d7806d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:13.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:13.408+0000 7f7d802a8640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d78070dd0 0x7f7d781f3220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:13.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:13.408+0000 7f7d802a8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d781f6b70 0x7f7d781f3760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:13.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:13.408+0000 7f7d802a8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7d7815e9f0 con 0x7f7d781f6b70 2024-04-25T00:25:13.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:13.408+0000 7f7d802a8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7d7815e870 con 0x7f7d7806e100 2024-04-25T00:25:13.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:13.408+0000 7f7d802a8640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f7d7815eb70 con 0x7f7d78070dd0 2024-04-25T00:25:13.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:13.408+0000 7f7d7d843640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d78070dd0 0x7f7d781f3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:13.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:13.408+0000 7f7d7d843640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d78070dd0 0x7f7d781f3220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:40898/0 (socket says 172.21.15.116:40898) 2024-04-25T00:25:13.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:13.408+0000 7f7d7d843640 1 -- 172.21.15.116:0/2129380093 learned_addr learned my addr 172.21.15.116:0/2129380093 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:25:13.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:13.408+0000 7f7d7e845640 1 --2- 172.21.15.116:0/2129380093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d781f6b70 0x7f7d781f3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:13.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:13.408+0000 7f7d7d843640 1 -- 172.21.15.116:0/2129380093 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d7806e100 msgr2=0x7f7d7806d100 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T00:25:13.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:13.408+0000 7f7d7d843640 1 --2- 172.21.15.116:0/2129380093 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d7806e100 0x7f7d7806d100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:13.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:13.408+0000 7f7d7d843640 1 -- 172.21.15.116:0/2129380093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d781f6b70 msgr2=0x7f7d781f3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:13.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:13.408+0000 7f7d7d843640 1 --2- 172.21.15.116:0/2129380093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d781f6b70 0x7f7d781f3760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:13.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:13.408+0000 7f7d7d843640 1 -- 172.21.15.116:0/2129380093 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d78152fe0 con 0x7f7d78070dd0 2024-04-25T00:25:13.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:13.408+0000 7f7d7e845640 1 --2- 172.21.15.116:0/2129380093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d781f6b70 0x7f7d781f3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:25:13.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:13.412+0000 7f7d7d843640 1 --2- 172.21.15.116:0/2129380093 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d78070dd0 0x7f7d781f3220 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7d7400c4b0 tx=0x7f7d7400c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:13.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:13.412+0000 7f7d6effd640 1 -- 172.21.15.116:0/2129380093 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d74012830 con 0x7f7d78070dd0 2024-04-25T00:25:13.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:13.416+0000 7f7d802a8640 1 -- 172.21.15.116:0/2129380093 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d781532c0 con 0x7f7d78070dd0 2024-04-25T00:25:13.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:13.416+0000 7f7d802a8640 1 -- 172.21.15.116:0/2129380093 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d7813ebe0 con 0x7f7d78070dd0 2024-04-25T00:25:13.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:13.416+0000 7f7d6effd640 1 -- 172.21.15.116:0/2129380093 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d74017070 con 0x7f7d78070dd0 2024-04-25T00:25:13.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:13.416+0000 7f7d6effd640 1 -- 172.21.15.116:0/2129380093 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d74012c30 con 0x7f7d78070dd0 2024-04-25T00:25:13.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:13.424+0000 7f7d6effd640 1 -- 172.21.15.116:0/2129380093 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7d7401c460 con 0x7f7d78070dd0 2024-04-25T00:25:13.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:13.428+0000 7f7d802a8640 1 -- 172.21.15.116:0/2129380093 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d600052a0 con 0x7f7d78070dd0 2024-04-25T00:25:13.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:13.428+0000 7f7d6effd640 1 --2- 172.21.15.116:0/2129380093 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7d5c0679c0 0x7f7d5c069e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:13.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:13.428+0000 7f7d6effd640 1 -- 172.21.15.116:0/2129380093 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(65..65 src has 1..65) ==== 9848+0+0 (secure 0 0 0) 0x7f7d74086900 con 0x7f7d78070dd0 2024-04-25T00:25:13.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:13.428+0000 7f7d7e044640 1 --2- 172.21.15.116:0/2129380093 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7d5c0679c0 0x7f7d5c069e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:13.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:13.428+0000 7f7d7e044640 1 --2- 172.21.15.116:0/2129380093 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7d5c0679c0 0x7f7d5c069e80 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f7d680af000 tx=0x7f7d680a8000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:13.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:13.444+0000 7f7d6effd640 1 -- 172.21.15.116:0/2129380093 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f7d74056470 con 0x7f7d78070dd0 2024-04-25T00:25:14.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:14.440+0000 7f7d802a8640 1 -- 172.21.15.116:0/2129380093 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7d60005590 con 0x7f7d78070dd0 2024-04-25T00:25:14.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:14.444+0000 7f7d6effd640 1 -- 172.21.15.116:0/2129380093 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 197 v197) ==== 95+0+6510 (secure 0 0 0) 0x7f7d74055d70 con 0x7f7d78070dd0 2024-04-25T00:25:14.447 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:25:14.447 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":197,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6413,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/4181466530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":4181466530},{"type":"v1","addr":"172.21.15.174:6837","nonce":4181466530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":197}],"filesystems":[{"mdsmap":{"epoch":191,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:23:34.013510+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8244":{"gid":8244,"name":"c","rank":0,"incarnation":188,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6835/1971675962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1971675962},{"type":"v1","addr":"172.21.15.116:6835","nonce":1971675962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8244,"qdb_cluster":[8244]},"id":1},{"mdsmap":{"epoch":196,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:24:19.977863+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8343":{"gid":8343,"name":"a","rank":0,"incarnation":193,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6833/3633657268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3633657268},{"type":"v1","addr":"172.21.15.116:6833","nonce":3633657268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8343,"qdb_cluster":[8343]},"id":2}]} 2024-04-25T00:25:14.447 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 197 2024-04-25T00:25:14.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:14.448+0000 7f7d802a8640 1 -- 172.21.15.116:0/2129380093 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7d5c0679c0 msgr2=0x7f7d5c069e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:14.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:14.448+0000 7f7d802a8640 1 --2- 172.21.15.116:0/2129380093 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7d5c0679c0 0x7f7d5c069e80 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f7d680af000 tx=0x7f7d680a8000 comp rx=0 tx=0).stop 2024-04-25T00:25:14.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:14.448+0000 7f7d802a8640 1 -- 172.21.15.116:0/2129380093 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d78070dd0 msgr2=0x7f7d781f3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:14.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:14.448+0000 7f7d802a8640 1 --2- 172.21.15.116:0/2129380093 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d78070dd0 0x7f7d781f3220 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7d7400c4b0 tx=0x7f7d7400c970 comp rx=0 tx=0).stop 2024-04-25T00:25:14.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:14.448+0000 7f7d802a8640 1 -- 172.21.15.116:0/2129380093 shutdown_connections 2024-04-25T00:25:14.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:14.448+0000 7f7d802a8640 1 --2- 172.21.15.116:0/2129380093 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7d5c0679c0 0x7f7d5c069e80 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:14.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:14.448+0000 7f7d802a8640 1 --2- 172.21.15.116:0/2129380093 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d781f6b70 0x7f7d781f3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:14.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:14.448+0000 7f7d802a8640 1 --2- 172.21.15.116:0/2129380093 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d78070dd0 0x7f7d781f3220 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:14.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:14.448+0000 7f7d802a8640 1 --2- 172.21.15.116:0/2129380093 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d7806e100 0x7f7d7806d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:14.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:14.448+0000 7f7d802a8640 1 -- 172.21.15.116:0/2129380093 >> 172.21.15.116:0/2129380093 conn(0x7f7d780c3b00 msgr2=0x7f7d780c3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:14.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:14.448+0000 7f7d802a8640 1 -- 172.21.15.116:0/2129380093 shutdown_connections 2024-04-25T00:25:14.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:14.448+0000 7f7d802a8640 1 -- 172.21.15.116:0/2129380093 wait complete. 2024-04-25T00:25:14.507 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:25:16.508 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:25:16.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:16.820+0000 7f5fa0fc3640 1 Processor -- start 2024-04-25T00:25:16.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:16.820+0000 7f5fa0fc3640 1 -- start start 2024-04-25T00:25:16.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:16.820+0000 7f5fa0fc3640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5f9c13a360 0x7f5f9c0c29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:16.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:16.820+0000 7f5fa0fc3640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f5f9c0c2ee0 0x7f5f9c071180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:16.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:16.820+0000 7f5fa0fc3640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f9c073320 con 0x7f5f9c13acb0 2024-04-25T00:25:16.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:16.820+0000 7f5fa0fc3640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f5f9c070cf0 con 0x7f5f9c13a360 2024-04-25T00:25:16.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:16.820+0000 7f5fa0fc3640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f5f9c073040 con 0x7f5f9c0c2ee0 2024-04-25T00:25:16.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:16.820+0000 7f5f9a59c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5f9c13a360 0x7f5f9c0c29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:16.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:16.820+0000 7f5f9a59c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5f9c13a360 0x7f5f9c0c29a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52918/0 (socket says 172.21.15.116:52918) 2024-04-25T00:25:16.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:16.820+0000 7f5f9a59c640 1 -- 172.21.15.116:0/2109422170 learned_addr learned my addr 172.21.15.116:0/2109422170 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:25:16.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:16.824+0000 7f5f9a59c640 1 -- 172.21.15.116:0/2109422170 >> v2:172.21.15.174:3300/0 conn(0x7f5f9c0c2ee0 msgr2=0x7f5f9c071180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:16.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:16.824+0000 7f5f9a59c640 1 --2- 172.21.15.116:0/2109422170 >> v2:172.21.15.174:3300/0 conn(0x7f5f9c0c2ee0 0x7f5f9c071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:16.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:16.824+0000 7f5f9a59c640 1 -- 172.21.15.116:0/2109422170 >> v1:172.21.15.174:6789/0 conn(0x7f5f9c13acb0 legacy=0x7f5f9c0716c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:16.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:16.824+0000 7f5f9a59c640 1 -- 172.21.15.116:0/2109422170 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f9c1557e0 con 0x7f5f9c13a360 2024-04-25T00:25:16.837 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:16.836+0000 7f5f9a59c640 1 --2- 172.21.15.116:0/2109422170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f9c13a360 0x7f5f9c0c29a0 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c0c34c0 tx=0x7f5f8c01e8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc9a4adbf8eb47f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:16.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:16.836+0000 7f5f9959a640 1 -- 172.21.15.116:0/2109422170 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f8c0ad070 con 0x7f5f9c13a360 2024-04-25T00:25:16.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:16.836+0000 7f5f9959a640 1 -- 172.21.15.116:0/2109422170 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f8c0a77a0 con 0x7f5f9c13a360 2024-04-25T00:25:16.838 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:16.836+0000 7f5f9959a640 1 -- 172.21.15.116:0/2109422170 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f8c0a7b70 con 0x7f5f9c13a360 2024-04-25T00:25:16.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:16.836+0000 7f5fa0fc3640 1 -- 172.21.15.116:0/2109422170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f9c13a360 msgr2=0x7f5f9c0c29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:16.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:16.836+0000 7f5fa0fc3640 1 --2- 172.21.15.116:0/2109422170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f9c13a360 0x7f5f9c0c29a0 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c0c34c0 tx=0x7f5f8c01e8d0 comp rx=0 tx=0).stop 2024-04-25T00:25:16.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:16.836+0000 7f5fa0fc3640 1 -- 172.21.15.116:0/2109422170 shutdown_connections 2024-04-25T00:25:16.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:16.836+0000 7f5fa0fc3640 1 --2- 172.21.15.116:0/2109422170 >> v2:172.21.15.174:3300/0 conn(0x7f5f9c0c2ee0 0x7f5f9c071180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:16.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:16.836+0000 7f5fa0fc3640 1 --2- 172.21.15.116:0/2109422170 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f9c13a360 0x7f5f9c0c29a0 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:16.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:16.836+0000 7f5fa0fc3640 1 -- 172.21.15.116:0/2109422170 >> 172.21.15.116:0/2109422170 conn(0x7f5f9c0c3a90 msgr2=0x7f5f9c0c3ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:16.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:16.880+0000 7f5fa0fc3640 1 -- 172.21.15.116:0/2109422170 shutdown_connections 2024-04-25T00:25:16.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:16.880+0000 7f5fa0fc3640 1 -- 172.21.15.116:0/2109422170 wait complete. 2024-04-25T00:25:16.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:16.880+0000 7f5fa0fc3640 1 Processor -- start 2024-04-25T00:25:16.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:16.880+0000 7f5fa0fc3640 1 -- start start 2024-04-25T00:25:16.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:16.884+0000 7f5fa0fc3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f9c0c2ee0 0x7f5f9c1f3240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:16.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:16.884+0000 7f5fa0fc3640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5f9c13a360 0x7f5f9c133d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:16.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:16.884+0000 7f5fa0fc3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5f9c13acb0 0x7f5f9c1f7050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:16.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:16.884+0000 7f5fa0fc3640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5f9c15ea60 con 0x7f5f9c0c2ee0 2024-04-25T00:25:16.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:16.884+0000 7f5fa0fc3640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5f9c15e8e0 con 0x7f5f9c13acb0 2024-04-25T00:25:16.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:16.884+0000 7f5fa0fc3640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f5f9c15ebe0 con 0x7f5f9c13a360 2024-04-25T00:25:16.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:16.884+0000 7f5f9a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f9c0c2ee0 0x7f5f9c1f3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:16.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:16.884+0000 7f5f9a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f9c0c2ee0 0x7f5f9c1f3240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52926/0 (socket says 172.21.15.116:52926) 2024-04-25T00:25:16.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:16.884+0000 7f5f9a59c640 1 -- 172.21.15.116:0/963321377 learned_addr learned my addr 172.21.15.116:0/963321377 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:25:16.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:16.884+0000 7f5f9ad9d640 1 --2- 172.21.15.116:0/963321377 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5f9c13acb0 0x7f5f9c1f7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:16.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:16.884+0000 7f5f99d9b640 1 --2- 172.21.15.116:0/963321377 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5f9c13a360 0x7f5f9c133d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:16.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:16.884+0000 7f5f9a59c640 1 -- 172.21.15.116:0/963321377 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5f9c13a360 msgr2=0x7f5f9c133d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:16.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:16.884+0000 7f5f9a59c640 1 --2- 172.21.15.116:0/963321377 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5f9c13a360 0x7f5f9c133d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:16.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:16.884+0000 7f5f9a59c640 1 -- 172.21.15.116:0/963321377 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5f9c13acb0 msgr2=0x7f5f9c1f7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:16.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:16.884+0000 7f5f9a59c640 1 --2- 172.21.15.116:0/963321377 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5f9c13acb0 0x7f5f9c1f7050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:16.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:16.884+0000 7f5f9a59c640 1 -- 172.21.15.116:0/963321377 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f9c0c6140 con 0x7f5f9c0c2ee0 2024-04-25T00:25:16.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:16.884+0000 7f5f9a59c640 1 --2- 172.21.15.116:0/963321377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f9c0c2ee0 0x7f5f9c1f3240 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f5f8c01c500 tx=0x7f5f8c01d110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:16.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:16.884+0000 7f5f8affd640 1 -- 172.21.15.116:0/963321377 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f8c0c0070 con 0x7f5f9c0c2ee0 2024-04-25T00:25:16.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:16.884+0000 7f5f8affd640 1 -- 172.21.15.116:0/963321377 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f8c008cb0 con 0x7f5f9c0c2ee0 2024-04-25T00:25:16.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:16.888+0000 7f5f8affd640 1 -- 172.21.15.116:0/963321377 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f8c0ad040 con 0x7f5f9c0c2ee0 2024-04-25T00:25:16.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:16.888+0000 7f5fa0fc3640 1 -- 172.21.15.116:0/963321377 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f9c151de0 con 0x7f5f9c0c2ee0 2024-04-25T00:25:16.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:16.888+0000 7f5fa0fc3640 1 -- 172.21.15.116:0/963321377 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f9c1f6850 con 0x7f5f9c0c2ee0 2024-04-25T00:25:16.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:16.892+0000 7f5f8affd640 1 -- 172.21.15.116:0/963321377 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5f8c00b3a0 con 0x7f5f9c0c2ee0 2024-04-25T00:25:16.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:16.892+0000 7f5fa0fc3640 1 -- 172.21.15.116:0/963321377 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f9c04f580 con 0x7f5f9c0c2ee0 2024-04-25T00:25:16.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:16.892+0000 7f5f8affd640 1 --2- 172.21.15.116:0/963321377 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5f7c0678c0 0x7f5f7c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:16.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:16.892+0000 7f5f8affd640 1 -- 172.21.15.116:0/963321377 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 9848+0+0 (secure 0 0 0) 0x7f5f8c120670 con 0x7f5f9c0c2ee0 2024-04-25T00:25:16.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:16.896+0000 7f5f99d9b640 1 --2- 172.21.15.116:0/963321377 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5f7c0678c0 0x7f5f7c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:16.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:16.900+0000 7f5f8affd640 1 -- 172.21.15.116:0/963321377 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f5f8c0185e0 con 0x7f5f9c0c2ee0 2024-04-25T00:25:16.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:16.904+0000 7f5f99d9b640 1 --2- 172.21.15.116:0/963321377 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5f7c0678c0 0x7f5f7c069d80 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c15b770 tx=0x7f5f90006f90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:17.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:17.660+0000 7f5fa0fc3640 1 -- 172.21.15.116:0/963321377 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5f9c0ad320 con 0x7f5f9c0c2ee0 2024-04-25T00:25:17.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:17.660+0000 7f5f8affd640 1 -- 172.21.15.116:0/963321377 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 197 v197) ==== 95+0+6510 (secure 0 0 0) 0x7f5f8c0efb10 con 0x7f5f9c0c2ee0 2024-04-25T00:25:17.665 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:25:17.665 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":197,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6413,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/4181466530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":4181466530},{"type":"v1","addr":"172.21.15.174:6837","nonce":4181466530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":197}],"filesystems":[{"mdsmap":{"epoch":191,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:23:34.013510+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8244":{"gid":8244,"name":"c","rank":0,"incarnation":188,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6835/1971675962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1971675962},{"type":"v1","addr":"172.21.15.116:6835","nonce":1971675962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8244,"qdb_cluster":[8244]},"id":1},{"mdsmap":{"epoch":196,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:24:19.977863+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8343":{"gid":8343,"name":"a","rank":0,"incarnation":193,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6833/3633657268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3633657268},{"type":"v1","addr":"172.21.15.116:6833","nonce":3633657268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8343,"qdb_cluster":[8343]},"id":2}]} 2024-04-25T00:25:17.666 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 197 2024-04-25T00:25:17.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:17.676+0000 7f5f88ff9640 1 -- 172.21.15.116:0/963321377 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5f7c0678c0 msgr2=0x7f5f7c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:17.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:17.676+0000 7f5f88ff9640 1 --2- 172.21.15.116:0/963321377 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5f7c0678c0 0x7f5f7c069d80 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c15b770 tx=0x7f5f90006f90 comp rx=0 tx=0).stop 2024-04-25T00:25:17.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:17.676+0000 7f5f88ff9640 1 -- 172.21.15.116:0/963321377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f9c0c2ee0 msgr2=0x7f5f9c1f3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:17.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:17.676+0000 7f5f88ff9640 1 --2- 172.21.15.116:0/963321377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f9c0c2ee0 0x7f5f9c1f3240 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7f5f8c01c500 tx=0x7f5f8c01d110 comp rx=0 tx=0).stop 2024-04-25T00:25:17.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:17.676+0000 7f5f88ff9640 1 -- 172.21.15.116:0/963321377 shutdown_connections 2024-04-25T00:25:17.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:17.676+0000 7f5f88ff9640 1 --2- 172.21.15.116:0/963321377 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5f7c0678c0 0x7f5f7c069d80 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:17.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:17.676+0000 7f5f88ff9640 1 --2- 172.21.15.116:0/963321377 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5f9c13acb0 0x7f5f9c1f7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:17.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:17.676+0000 7f5f88ff9640 1 --2- 172.21.15.116:0/963321377 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5f9c13a360 0x7f5f9c133d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:17.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:17.676+0000 7f5f88ff9640 1 --2- 172.21.15.116:0/963321377 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5f9c0c2ee0 0x7f5f9c1f3240 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:17.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:17.676+0000 7f5f88ff9640 1 -- 172.21.15.116:0/963321377 >> 172.21.15.116:0/963321377 conn(0x7f5f9c0c3a90 msgr2=0x7f5f9c138ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:17.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:17.680+0000 7f5f88ff9640 1 -- 172.21.15.116:0/963321377 shutdown_connections 2024-04-25T00:25:17.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:17.680+0000 7f5f88ff9640 1 -- 172.21.15.116:0/963321377 wait complete. 2024-04-25T00:25:17.731 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:25:19.732 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:25:19.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.968+0000 7fe1aad59640 1 Processor -- start 2024-04-25T00:25:19.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.972+0000 7fe1aad59640 1 -- start start 2024-04-25T00:25:19.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.972+0000 7fe1aad59640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fe1a414f990 0x7fe1a406cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:19.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.972+0000 7fe1aad59640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe1a4156260 0x7fe1a40c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:19.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.972+0000 7fe1aad59640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1a41566b0 con 0x7fe1a406d160 2024-04-25T00:25:19.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.972+0000 7fe1aad59640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fe1a40c0b40 con 0x7fe1a4156260 2024-04-25T00:25:19.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.972+0000 7fe1aad59640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fe1a4155dd0 con 0x7fe1a414f990 2024-04-25T00:25:19.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.972+0000 7fe1a3fff640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fe1a406d160 0x7fe1a40c0430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:35206/0 (socket says 172.21.15.116:35206) 2024-04-25T00:25:19.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.972+0000 7fe1a3fff640 1 -- 172.21.15.116:0/2513775524 learned_addr learned my addr 172.21.15.116:0/2513775524 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:25:19.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.972+0000 7fe1a37fe640 1 -- 172.21.15.116:0/2513775524 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1048165178 0 0) 0x7fe1a41566b0 con 0x7fe1a406d160 2024-04-25T00:25:19.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.972+0000 7fe1a37fe640 1 -- 172.21.15.116:0/2513775524 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe194003530 con 0x7fe1a406d160 2024-04-25T00:25:19.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.972+0000 7fe1a92f6640 1 --2- 172.21.15.116:0/2513775524 >> v2:172.21.15.116:3300/0 conn(0x7fe1a4156260 0x7fe1a40c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:19.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.972+0000 7fe1a8af5640 1 --2- 172.21.15.116:0/2513775524 >> v2:172.21.15.174:3300/0 conn(0x7fe1a414f990 0x7fe1a406cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:19.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.972+0000 7fe1a92f6640 1 -- 172.21.15.116:0/2513775524 >> v2:172.21.15.174:3300/0 conn(0x7fe1a414f990 msgr2=0x7fe1a406cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:19.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.972+0000 7fe1a92f6640 1 --2- 172.21.15.116:0/2513775524 >> v2:172.21.15.174:3300/0 conn(0x7fe1a414f990 0x7fe1a406cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:19.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.972+0000 7fe1a92f6640 1 -- 172.21.15.116:0/2513775524 >> v1:172.21.15.174:6789/0 conn(0x7fe1a406d160 legacy=0x7fe1a40c0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:19.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.972+0000 7fe1a37fe640 1 -- 172.21.15.116:0/2513775524 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2773827946 0 0) 0x7fe194003530 con 0x7fe1a406d160 2024-04-25T00:25:19.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.972+0000 7fe1a92f6640 1 -- 172.21.15.116:0/2513775524 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1a4070de0 con 0x7fe1a4156260 2024-04-25T00:25:19.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.972+0000 7fe1a8af5640 1 --2- 172.21.15.116:0/2513775524 >> v2:172.21.15.174:3300/0 conn(0x7fe1a414f990 0x7fe1a406cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:25:19.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.976+0000 7fe1a92f6640 1 --2- 172.21.15.116:0/2513775524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1a4156260 0x7fe1a40c2a10 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7fe1a40c7f60 tx=0x7fe19c01c700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=917f785bc9723f62 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:19.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.976+0000 7fe1a37fe640 1 -- 172.21.15.116:0/2513775524 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe19c0b2070 con 0x7fe1a4156260 2024-04-25T00:25:19.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.976+0000 7fe1a37fe640 1 -- 172.21.15.116:0/2513775524 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe19c0a5540 con 0x7fe1a4156260 2024-04-25T00:25:19.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.976+0000 7fe1a37fe640 1 -- 172.21.15.116:0/2513775524 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe19c0b2070 con 0x7fe1a4156260 2024-04-25T00:25:19.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.980+0000 7fe1aad59640 1 -- 172.21.15.116:0/2513775524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1a4156260 msgr2=0x7fe1a40c2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:19.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.980+0000 7fe1aad59640 1 --2- 172.21.15.116:0/2513775524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1a4156260 0x7fe1a40c2a10 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7fe1a40c7f60 tx=0x7fe19c01c700 comp rx=0 tx=0).stop 2024-04-25T00:25:19.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.980+0000 7fe1aad59640 1 -- 172.21.15.116:0/2513775524 shutdown_connections 2024-04-25T00:25:19.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.980+0000 7fe1aad59640 1 --2- 172.21.15.116:0/2513775524 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1a4156260 0x7fe1a40c2a10 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:19.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.980+0000 7fe1aad59640 1 --2- 172.21.15.116:0/2513775524 >> v2:172.21.15.174:3300/0 conn(0x7fe1a414f990 0x7fe1a406cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:19.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.980+0000 7fe1aad59640 1 -- 172.21.15.116:0/2513775524 >> 172.21.15.116:0/2513775524 conn(0x7fe1a40c3b00 msgr2=0x7fe1a40c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:19.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.980+0000 7fe1aad59640 1 -- 172.21.15.116:0/2513775524 shutdown_connections 2024-04-25T00:25:19.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.980+0000 7fe1aad59640 1 -- 172.21.15.116:0/2513775524 wait complete. 2024-04-25T00:25:19.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.984+0000 7fe1aad59640 1 Processor -- start 2024-04-25T00:25:19.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.984+0000 7fe1aad59640 1 -- start start 2024-04-25T00:25:19.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.984+0000 7fe1aad59640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1a406d160 0x7fe1a414e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:19.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.984+0000 7fe1aad59640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe1a414f990 0x7fe1a4152df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:19.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.984+0000 7fe1aad59640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe1a4156260 0x7fe1a40abb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:19.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.984+0000 7fe1aad59640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe1a41628b0 con 0x7fe1a406d160 2024-04-25T00:25:19.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.984+0000 7fe1aad59640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe1a4155710 con 0x7fe1a4156260 2024-04-25T00:25:19.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.984+0000 7fe1aad59640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe1a4162a30 con 0x7fe1a414f990 2024-04-25T00:25:19.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.984+0000 7fe1a8af5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1a406d160 0x7fe1a414e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:19.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.984+0000 7fe1a8af5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1a406d160 0x7fe1a414e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52930/0 (socket says 172.21.15.116:52930) 2024-04-25T00:25:19.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.984+0000 7fe1a8af5640 1 -- 172.21.15.116:0/867430180 learned_addr learned my addr 172.21.15.116:0/867430180 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:25:19.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.984+0000 7fe1a92f6640 1 --2- 172.21.15.116:0/867430180 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe1a4156260 0x7fe1a40abb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:19.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.984+0000 7fe1a92f6640 1 -- 172.21.15.116:0/867430180 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe1a414f990 msgr2=0x7fe1a4152df0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T00:25:19.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.984+0000 7fe1a92f6640 1 --2- 172.21.15.116:0/867430180 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe1a414f990 0x7fe1a4152df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:19.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.984+0000 7fe1a92f6640 1 -- 172.21.15.116:0/867430180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1a406d160 msgr2=0x7fe1a414e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:19.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.984+0000 7fe1a92f6640 1 --2- 172.21.15.116:0/867430180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1a406d160 0x7fe1a414e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:19.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.984+0000 7fe1a92f6640 1 -- 172.21.15.116:0/867430180 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1a40c6640 con 0x7fe1a4156260 2024-04-25T00:25:19.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.984+0000 7fe1a8af5640 1 --2- 172.21.15.116:0/867430180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1a406d160 0x7fe1a414e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:25:19.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.984+0000 7fe1a92f6640 1 --2- 172.21.15.116:0/867430180 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe1a4156260 0x7fe1a40abb40 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fe1a40c34c0 tx=0x7fe19c0194a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:19.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.984+0000 7fe1a17fa640 1 -- 172.21.15.116:0/867430180 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe19c0bd020 con 0x7fe1a4156260 2024-04-25T00:25:19.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.984+0000 7fe1a17fa640 1 -- 172.21.15.116:0/867430180 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe19c0b9660 con 0x7fe1a4156260 2024-04-25T00:25:19.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.984+0000 7fe1a17fa640 1 -- 172.21.15.116:0/867430180 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe19c0b2070 con 0x7fe1a4156260 2024-04-25T00:25:19.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.984+0000 7fe1aad59640 1 -- 172.21.15.116:0/867430180 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1a40c0570 con 0x7fe1a4156260 2024-04-25T00:25:19.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.984+0000 7fe1aad59640 1 -- 172.21.15.116:0/867430180 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1a40aa2c0 con 0x7fe1a4156260 2024-04-25T00:25:19.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.988+0000 7fe1a17fa640 1 -- 172.21.15.116:0/867430180 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe19c0c0020 con 0x7fe1a4156260 2024-04-25T00:25:19.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.988+0000 7fe1aad59640 1 -- 172.21.15.116:0/867430180 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1a4068350 con 0x7fe1a4156260 2024-04-25T00:25:19.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.988+0000 7fe1a17fa640 1 --2- 172.21.15.116:0/867430180 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe188067910 0x7fe188069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:19.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.988+0000 7fe1a8af5640 1 --2- 172.21.15.116:0/867430180 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe188067910 0x7fe188069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:19.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.988+0000 7fe1a17fa640 1 -- 172.21.15.116:0/867430180 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 9848+0+0 (secure 0 0 0) 0x7fe19c0eead0 con 0x7fe1a4156260 2024-04-25T00:25:19.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:19.992+0000 7fe1a8af5640 1 --2- 172.21.15.116:0/867430180 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe188067910 0x7fe188069dd0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fe198006fd0 tx=0x7fe198008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:20.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:20.000+0000 7fe1a17fa640 1 -- 172.21.15.116:0/867430180 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fe19c0c7030 con 0x7fe1a4156260 2024-04-25T00:25:20.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:20.564+0000 7fe1aad59640 1 -- 172.21.15.116:0/867430180 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe1a40acd80 con 0x7fe1a4156260 2024-04-25T00:25:20.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:20.568+0000 7fe1a17fa640 1 -- 172.21.15.116:0/867430180 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 197 v197) ==== 95+0+6510 (secure 0 0 0) 0x7fe19c016440 con 0x7fe1a4156260 2024-04-25T00:25:20.577 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:25:20.578 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":197,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6413,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/4181466530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":4181466530},{"type":"v1","addr":"172.21.15.174:6837","nonce":4181466530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":197}],"filesystems":[{"mdsmap":{"epoch":191,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:23:34.013510+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8244":{"gid":8244,"name":"c","rank":0,"incarnation":188,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6835/1971675962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1971675962},{"type":"v1","addr":"172.21.15.116:6835","nonce":1971675962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8244,"qdb_cluster":[8244]},"id":1},{"mdsmap":{"epoch":196,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:24:19.977863+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8343":{"gid":8343,"name":"a","rank":0,"incarnation":193,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6833/3633657268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3633657268},{"type":"v1","addr":"172.21.15.116:6833","nonce":3633657268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8343,"qdb_cluster":[8343]},"id":2}]} 2024-04-25T00:25:20.578 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 197 2024-04-25T00:25:20.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:20.576+0000 7fe18effd640 1 -- 172.21.15.116:0/867430180 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe188067910 msgr2=0x7fe188069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:20.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:20.580+0000 7fe18effd640 1 --2- 172.21.15.116:0/867430180 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe188067910 0x7fe188069dd0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fe198006fd0 tx=0x7fe198008040 comp rx=0 tx=0).stop 2024-04-25T00:25:20.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:20.580+0000 7fe18effd640 1 -- 172.21.15.116:0/867430180 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe1a4156260 msgr2=0x7fe1a40abb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:20.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:20.580+0000 7fe18effd640 1 --2- 172.21.15.116:0/867430180 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe1a4156260 0x7fe1a40abb40 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fe1a40c34c0 tx=0x7fe19c0194a0 comp rx=0 tx=0).stop 2024-04-25T00:25:20.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:20.580+0000 7fe18effd640 1 -- 172.21.15.116:0/867430180 shutdown_connections 2024-04-25T00:25:20.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:20.580+0000 7fe18effd640 1 --2- 172.21.15.116:0/867430180 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe188067910 0x7fe188069dd0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:20.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:20.580+0000 7fe18effd640 1 --2- 172.21.15.116:0/867430180 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe1a4156260 0x7fe1a40abb40 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:20.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:20.580+0000 7fe18effd640 1 --2- 172.21.15.116:0/867430180 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe1a414f990 0x7fe1a4152df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:20.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:20.580+0000 7fe18effd640 1 --2- 172.21.15.116:0/867430180 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe1a406d160 0x7fe1a414e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:20.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:20.580+0000 7fe18effd640 1 -- 172.21.15.116:0/867430180 >> 172.21.15.116:0/867430180 conn(0x7fe1a40c3b00 msgr2=0x7fe1a40c3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:20.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:20.588+0000 7fe18effd640 1 -- 172.21.15.116:0/867430180 shutdown_connections 2024-04-25T00:25:20.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:20.588+0000 7fe18effd640 1 -- 172.21.15.116:0/867430180 wait complete. 2024-04-25T00:25:20.688 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:25:22.689 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:25:22.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:22.815+0000 7fcfe64da640 1 Processor -- start 2024-04-25T00:25:22.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:22.819+0000 7fcfe64da640 1 -- start start 2024-04-25T00:25:22.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:22.819+0000 7fcfe64da640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcfe0152d80 0x7fcfe006bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:22.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:22.819+0000 7fcfe64da640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfe00c9150 con 0x7fcfe006d5f0 2024-04-25T00:25:22.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:22.819+0000 7fcfe64da640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfe00c9430 con 0x7fcfe006c290 2024-04-25T00:25:22.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:22.819+0000 7fcfe64da640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fcfe00c9d60 con 0x7fcfe0152d80 2024-04-25T00:25:22.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:22.819+0000 7fcfd77fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fcfe006d5f0 0x7fcfe0156c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:45056/0 (socket says 172.21.15.116:45056) 2024-04-25T00:25:22.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:22.819+0000 7fcfd77fe640 1 -- 172.21.15.116:0/2925253501 learned_addr learned my addr 172.21.15.116:0/2925253501 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:25:22.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:22.819+0000 7fcfdffff640 1 --2- 172.21.15.116:0/2925253501 >> v2:172.21.15.116:3300/0 conn(0x7fcfe0152d80 0x7fcfe006bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:22.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:22.819+0000 7fcfdffff640 1 -- 172.21.15.116:0/2925253501 >> v1:172.21.15.174:6789/0 conn(0x7fcfe006c290 legacy=0x7fcfe0157390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:22.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:22.819+0000 7fcfdffff640 1 -- 172.21.15.116:0/2925253501 >> v1:172.21.15.116:6789/0 conn(0x7fcfe006d5f0 legacy=0x7fcfe0156c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:22.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:22.819+0000 7fcfdffff640 1 -- 172.21.15.116:0/2925253501 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfe00c9710 con 0x7fcfe0152d80 2024-04-25T00:25:22.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:22.823+0000 7fcfdffff640 1 --2- 172.21.15.116:0/2925253501 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfe0152d80 0x7fcfe006bd50 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fcfd00097f0 tx=0x7fcfd001bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12c0c43c902403e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:22.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:22.823+0000 7fcfdf7fe640 1 -- 172.21.15.116:0/2925253501 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd00a5660 con 0x7fcfe0152d80 2024-04-25T00:25:22.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:22.823+0000 7fcfdf7fe640 1 -- 172.21.15.116:0/2925253501 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfd00a5800 con 0x7fcfe0152d80 2024-04-25T00:25:22.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:22.823+0000 7fcfdf7fe640 1 -- 172.21.15.116:0/2925253501 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd00a5bd0 con 0x7fcfe0152d80 2024-04-25T00:25:22.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:22.823+0000 7fcfe64da640 1 -- 172.21.15.116:0/2925253501 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfe0152d80 msgr2=0x7fcfe006bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:22.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:22.823+0000 7fcfe64da640 1 --2- 172.21.15.116:0/2925253501 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfe0152d80 0x7fcfe006bd50 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fcfd00097f0 tx=0x7fcfd001bf50 comp rx=0 tx=0).stop 2024-04-25T00:25:22.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:22.823+0000 7fcfe64da640 1 -- 172.21.15.116:0/2925253501 shutdown_connections 2024-04-25T00:25:22.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:22.823+0000 7fcfe64da640 1 --2- 172.21.15.116:0/2925253501 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfe0152d80 0x7fcfe006bd50 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:22.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:22.823+0000 7fcfe64da640 1 -- 172.21.15.116:0/2925253501 >> 172.21.15.116:0/2925253501 conn(0x7fcfe00acdd0 msgr2=0x7fcfe00ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:22.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:22.823+0000 7fcfe64da640 1 -- 172.21.15.116:0/2925253501 shutdown_connections 2024-04-25T00:25:22.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:22.823+0000 7fcfe64da640 1 -- 172.21.15.116:0/2925253501 wait complete. 2024-04-25T00:25:22.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:22.823+0000 7fcfe64da640 1 Processor -- start 2024-04-25T00:25:22.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:22.823+0000 7fcfe64da640 1 -- start start 2024-04-25T00:25:22.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:22.827+0000 7fcfe64da640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfe006c290 0x7fcfe0146840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:22.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:22.827+0000 7fcfe64da640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfe006d5f0 0x7fcfe0154e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:22.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:22.827+0000 7fcfdffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfe006c290 0x7fcfe0146840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:22.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:22.827+0000 7fcfdffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfe006c290 0x7fcfe0146840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52942/0 (socket says 172.21.15.116:52942) 2024-04-25T00:25:22.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:22.827+0000 7fcfe64da640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcfe0152d80 0x7fcfe00aaa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:22.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:22.827+0000 7fcfdffff640 1 -- 172.21.15.116:0/3993770866 learned_addr learned my addr 172.21.15.116:0/3993770866 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:25:22.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:22.827+0000 7fcfe64da640 1 -- 172.21.15.116:0/3993770866 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe01648a0 con 0x7fcfe006c290 2024-04-25T00:25:22.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:22.827+0000 7fcfe64da640 1 -- 172.21.15.116:0/3993770866 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe0164720 con 0x7fcfe006d5f0 2024-04-25T00:25:22.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:22.827+0000 7fcfe64da640 1 -- 172.21.15.116:0/3993770866 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fcfe0164a20 con 0x7fcfe0152d80 2024-04-25T00:25:22.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:22.827+0000 7fcfdffff640 1 -- 172.21.15.116:0/3993770866 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcfe0152d80 msgr2=0x7fcfe00aaa70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T00:25:22.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:22.827+0000 7fcfdffff640 1 --2- 172.21.15.116:0/3993770866 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcfe0152d80 0x7fcfe00aaa70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:22.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:22.827+0000 7fcfdffff640 1 -- 172.21.15.116:0/3993770866 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfe006d5f0 msgr2=0x7fcfe0154e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:22.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:22.827+0000 7fcfdffff640 1 --2- 172.21.15.116:0/3993770866 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfe006d5f0 0x7fcfe0154e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:22.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:22.827+0000 7fcfdffff640 1 -- 172.21.15.116:0/3993770866 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfe0153e20 con 0x7fcfe006c290 2024-04-25T00:25:22.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:22.827+0000 7fcfdffff640 1 --2- 172.21.15.116:0/3993770866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfe006c290 0x7fcfe0146840 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7fcfd0018e80 tx=0x7fcfd001a790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:22.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:22.827+0000 7fcfdd7fa640 1 -- 172.21.15.116:0/3993770866 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd00bd070 con 0x7fcfe006c290 2024-04-25T00:25:22.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:22.827+0000 7fcfdd7fa640 1 -- 172.21.15.116:0/3993770866 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfd00a6850 con 0x7fcfe006c290 2024-04-25T00:25:22.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:22.827+0000 7fcfe64da640 1 -- 172.21.15.116:0/3993770866 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfe0153b40 con 0x7fcfe006c290 2024-04-25T00:25:22.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:22.827+0000 7fcfdd7fa640 1 -- 172.21.15.116:0/3993770866 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd00a6c00 con 0x7fcfe006c290 2024-04-25T00:25:22.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:22.827+0000 7fcfe64da640 1 -- 172.21.15.116:0/3993770866 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfe0153840 con 0x7fcfe006c290 2024-04-25T00:25:22.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:22.827+0000 7fcfdd7fa640 1 -- 172.21.15.116:0/3993770866 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fcfd00af400 con 0x7fcfe006c290 2024-04-25T00:25:22.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:22.827+0000 7fcfe64da640 1 -- 172.21.15.116:0/3993770866 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfe0068350 con 0x7fcfe006c290 2024-04-25T00:25:22.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:22.827+0000 7fcfdd7fa640 1 --2- 172.21.15.116:0/3993770866 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcfc00678c0 0x7fcfc0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:22.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:22.831+0000 7fcfdd7fa640 1 -- 172.21.15.116:0/3993770866 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 9848+0+0 (secure 0 0 0) 0x7fcfd0120790 con 0x7fcfe006c290 2024-04-25T00:25:22.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:22.831+0000 7fcfd77fe640 1 --2- 172.21.15.116:0/3993770866 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcfc00678c0 0x7fcfc0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:22.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:22.831+0000 7fcfd77fe640 1 --2- 172.21.15.116:0/3993770866 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcfc00678c0 0x7fcfc0069d80 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fcfe00686a0 tx=0x7fcfc80079b0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:22.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:22.835+0000 7fcfdd7fa640 1 -- 172.21.15.116:0/3993770866 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fcfd00ab030 con 0x7fcfe006c290 2024-04-25T00:25:23.004 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:25:23.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.115+0000 7fcfe64da640 1 -- 172.21.15.116:0/3993770866 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcfe006c9a0 con 0x7fcfe006c290 2024-04-25T00:25:23.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.115+0000 7fcfdd7fa640 1 -- 172.21.15.116:0/3993770866 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 197 v197) ==== 95+0+6510 (secure 0 0 0) 0x7fcfd0015ff0 con 0x7fcfe006c290 2024-04-25T00:25:23.118 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:25:23.118 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":197,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6413,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/4181466530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":4181466530},{"type":"v1","addr":"172.21.15.174:6837","nonce":4181466530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":197}],"filesystems":[{"mdsmap":{"epoch":191,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:23:34.013510+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8244":{"gid":8244,"name":"c","rank":0,"incarnation":188,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6835/1971675962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1971675962},{"type":"v1","addr":"172.21.15.116:6835","nonce":1971675962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8244,"qdb_cluster":[8244]},"id":1},{"mdsmap":{"epoch":196,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:24:19.977863+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8343":{"gid":8343,"name":"a","rank":0,"incarnation":193,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6833/3633657268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3633657268},{"type":"v1","addr":"172.21.15.116:6833","nonce":3633657268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8343,"qdb_cluster":[8343]},"id":2}]} 2024-04-25T00:25:23.119 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 197 2024-04-25T00:25:23.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.115+0000 7fcfe64da640 1 -- 172.21.15.116:0/3993770866 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcfc00678c0 msgr2=0x7fcfc0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:23.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.115+0000 7fcfe64da640 1 --2- 172.21.15.116:0/3993770866 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcfc00678c0 0x7fcfc0069d80 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fcfe00686a0 tx=0x7fcfc80079b0 comp rx=0 tx=0).stop 2024-04-25T00:25:23.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.115+0000 7fcfe64da640 1 -- 172.21.15.116:0/3993770866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfe006c290 msgr2=0x7fcfe0146840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:23.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.115+0000 7fcfe64da640 1 --2- 172.21.15.116:0/3993770866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfe006c290 0x7fcfe0146840 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7fcfd0018e80 tx=0x7fcfd001a790 comp rx=0 tx=0).stop 2024-04-25T00:25:23.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.115+0000 7fcfe64da640 1 -- 172.21.15.116:0/3993770866 shutdown_connections 2024-04-25T00:25:23.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.115+0000 7fcfe64da640 1 --2- 172.21.15.116:0/3993770866 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcfc00678c0 0x7fcfc0069d80 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:23.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.115+0000 7fcfe64da640 1 --2- 172.21.15.116:0/3993770866 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcfe0152d80 0x7fcfe00aaa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:23.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.115+0000 7fcfe64da640 1 --2- 172.21.15.116:0/3993770866 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfe006d5f0 0x7fcfe0154e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:23.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.115+0000 7fcfe64da640 1 --2- 172.21.15.116:0/3993770866 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfe006c290 0x7fcfe0146840 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:23.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.115+0000 7fcfe64da640 1 -- 172.21.15.116:0/3993770866 >> 172.21.15.116:0/3993770866 conn(0x7fcfe00acdd0 msgr2=0x7fcfe014dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:23.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.115+0000 7fcfe64da640 1 -- 172.21.15.116:0/3993770866 shutdown_connections 2024-04-25T00:25:23.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.115+0000 7fcfe64da640 1 -- 172.21.15.116:0/3993770866 wait complete. 2024-04-25T00:25:23.145 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:25:23.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.143+0000 7fbb1406d640 1 Processor -- start 2024-04-25T00:25:23.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.143+0000 7fbb1406d640 1 -- start start 2024-04-25T00:25:23.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.143+0000 7fbb1406d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbb0c156200 0x7fbb0c15de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:23.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.143+0000 7fbb1406d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb0c156650 con 0x7fbb0c0aa170 2024-04-25T00:25:23.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.143+0000 7fbb1406d640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb0c1526a0 con 0x7fbb0c0c4300 2024-04-25T00:25:23.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.143+0000 7fbb1406d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fbb0c153460 con 0x7fbb0c156200 2024-04-25T00:25:23.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.143+0000 7fbb1260a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbb0c156200 0x7fbb0c15de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:23.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.143+0000 7fbb11608640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fbb0c0aa170 0x7fbb0c155af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:45066/0 (socket says 172.21.15.116:45066) 2024-04-25T00:25:23.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.143+0000 7fbb11608640 1 -- 172.21.15.116:0/4030008671 learned_addr learned my addr 172.21.15.116:0/4030008671 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:25:23.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.143+0000 7fbb1260a640 1 --2- 172.21.15.116:0/4030008671 >> v2:172.21.15.116:3300/0 conn(0x7fbb0c156200 0x7fbb0c15de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52944/0 (socket says 172.21.15.116:52944) 2024-04-25T00:25:23.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.147+0000 7fbb1260a640 1 -- 172.21.15.116:0/4030008671 >> v1:172.21.15.174:6789/0 conn(0x7fbb0c0c4300 legacy=0x7fbb0c0c5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:23.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.147+0000 7fbb1260a640 1 -- 172.21.15.116:0/4030008671 >> v1:172.21.15.116:6789/0 conn(0x7fbb0c0aa170 legacy=0x7fbb0c155af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:23.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.147+0000 7fbb1260a640 1 -- 172.21.15.116:0/4030008671 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb0c152980 con 0x7fbb0c156200 2024-04-25T00:25:23.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.147+0000 7fbb10e07640 1 -- 172.21.15.116:0/4030008671 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 792408612 0 0) 0x7fbb0c156650 con 0x7fbb0c0aa170 2024-04-25T00:25:23.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.147+0000 7fbb1260a640 1 --2- 172.21.15.116:0/4030008671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb0c156200 0x7fbb0c15de90 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7fbb04009d90 tx=0x7fbb0401c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31c1a441f907f76a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:23.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.147+0000 7fbb10e07640 1 -- 172.21.15.116:0/4030008671 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb040b2070 con 0x7fbb0c156200 2024-04-25T00:25:23.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.147+0000 7fbb10e07640 1 -- 172.21.15.116:0/4030008671 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb040a5830 con 0x7fbb0c156200 2024-04-25T00:25:23.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.147+0000 7fbb10e07640 1 -- 172.21.15.116:0/4030008671 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb040a5c00 con 0x7fbb0c156200 2024-04-25T00:25:23.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.151+0000 7fbb1406d640 1 -- 172.21.15.116:0/4030008671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb0c156200 msgr2=0x7fbb0c15de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:23.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.151+0000 7fbb1406d640 1 --2- 172.21.15.116:0/4030008671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb0c156200 0x7fbb0c15de90 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7fbb04009d90 tx=0x7fbb0401c970 comp rx=0 tx=0).stop 2024-04-25T00:25:23.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.151+0000 7fbb1406d640 1 -- 172.21.15.116:0/4030008671 shutdown_connections 2024-04-25T00:25:23.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.151+0000 7fbb1406d640 1 --2- 172.21.15.116:0/4030008671 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb0c156200 0x7fbb0c15de90 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:23.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.151+0000 7fbb1406d640 1 -- 172.21.15.116:0/4030008671 >> 172.21.15.116:0/4030008671 conn(0x7fbb0c06c720 msgr2=0x7fbb0c06cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:23.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.151+0000 7fbb1406d640 1 -- 172.21.15.116:0/4030008671 shutdown_connections 2024-04-25T00:25:23.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.151+0000 7fbb1406d640 1 -- 172.21.15.116:0/4030008671 wait complete. 2024-04-25T00:25:23.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.151+0000 7fbb1406d640 1 Processor -- start 2024-04-25T00:25:23.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.151+0000 7fbb1406d640 1 -- start start 2024-04-25T00:25:23.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.151+0000 7fbb1406d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb0c0aa170 0x7fbb0c14bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:23.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.151+0000 7fbb1406d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb0c0c4300 0x7fbb0c06b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:23.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.151+0000 7fbb1406d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb0c156200 0x7fbb0c0ab3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:23.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.151+0000 7fbb1406d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbb0c0aacd0 con 0x7fbb0c0c4300 2024-04-25T00:25:23.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.151+0000 7fbb1406d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbb0c06ffd0 con 0x7fbb0c156200 2024-04-25T00:25:23.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.151+0000 7fbb1406d640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fbb0c0aae50 con 0x7fbb0c0aa170 2024-04-25T00:25:23.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.155+0000 7fbb11608640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb0c0c4300 0x7fbb0c06b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:23.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.155+0000 7fbb11608640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb0c0c4300 0x7fbb0c06b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52960/0 (socket says 172.21.15.116:52960) 2024-04-25T00:25:23.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.155+0000 7fbb11608640 1 -- 172.21.15.116:0/499222939 learned_addr learned my addr 172.21.15.116:0/499222939 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:25:23.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.155+0000 7fbb1260a640 1 --2- 172.21.15.116:0/499222939 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb0c156200 0x7fbb0c0ab3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:23.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.155+0000 7fbb11e09640 1 --2- 172.21.15.116:0/499222939 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb0c0aa170 0x7fbb0c14bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:23.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.155+0000 7fbb1260a640 1 -- 172.21.15.116:0/499222939 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb0c0aa170 msgr2=0x7fbb0c14bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:23.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.155+0000 7fbb1260a640 1 --2- 172.21.15.116:0/499222939 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb0c0aa170 0x7fbb0c14bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:23.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.155+0000 7fbb1260a640 1 -- 172.21.15.116:0/499222939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb0c0c4300 msgr2=0x7fbb0c06b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:23.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.155+0000 7fbb1260a640 1 --2- 172.21.15.116:0/499222939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb0c0c4300 0x7fbb0c06b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:23.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.155+0000 7fbb1260a640 1 -- 172.21.15.116:0/499222939 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb0c0ad220 con 0x7fbb0c156200 2024-04-25T00:25:23.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.155+0000 7fbb1260a640 1 --2- 172.21.15.116:0/499222939 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb0c156200 0x7fbb0c0ab3f0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fbb040ac000 tx=0x7fbb04019710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:23.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.155+0000 7fbb027fc640 1 -- 172.21.15.116:0/499222939 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb040bb020 con 0x7fbb0c156200 2024-04-25T00:25:23.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.155+0000 7fbb027fc640 1 -- 172.21.15.116:0/499222939 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb040ad640 con 0x7fbb0c156200 2024-04-25T00:25:23.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.155+0000 7fbb1406d640 1 -- 172.21.15.116:0/499222939 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb0c1f2060 con 0x7fbb0c156200 2024-04-25T00:25:23.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.155+0000 7fbb027fc640 1 -- 172.21.15.116:0/499222939 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb040b2040 con 0x7fbb0c156200 2024-04-25T00:25:23.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.155+0000 7fbb1406d640 1 -- 172.21.15.116:0/499222939 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb0c1ee080 con 0x7fbb0c156200 2024-04-25T00:25:23.160 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.155+0000 7fbb1406d640 1 -- 172.21.15.116:0/499222939 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb0c068a60 con 0x7fbb0c156200 2024-04-25T00:25:23.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.155+0000 7fbb027fc640 1 -- 172.21.15.116:0/499222939 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbb040be020 con 0x7fbb0c156200 2024-04-25T00:25:23.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.159+0000 7fbb027fc640 1 --2- 172.21.15.116:0/499222939 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbaf0067990 0x7fbaf0069e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:23.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.159+0000 7fbb027fc640 1 -- 172.21.15.116:0/499222939 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 9848+0+0 (secure 0 0 0) 0x7fbb04121320 con 0x7fbb0c156200 2024-04-25T00:25:23.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.159+0000 7fbb11e09640 1 --2- 172.21.15.116:0/499222939 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbaf0067990 0x7fbaf0069e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:23.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.159+0000 7fbb11e09640 1 --2- 172.21.15.116:0/499222939 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbaf0067990 0x7fbaf0069e50 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fbb08007fd0 tx=0x7fbb08009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:23.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.163+0000 7fbb027fc640 1 -- 172.21.15.116:0/499222939 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fbb040c5070 con 0x7fbb0c156200 2024-04-25T00:25:23.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.431+0000 7fbb1406d640 1 -- 172.21.15.116:0/499222939 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbb0c0a9f50 con 0x7fbb0c156200 2024-04-25T00:25:23.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.431+0000 7fbb027fc640 1 -- 172.21.15.116:0/499222939 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 197 v197) ==== 95+0+6510 (secure 0 0 0) 0x7fbb0c0ad220 con 0x7fbb0c156200 2024-04-25T00:25:23.435 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:25:23.435 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":197,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6413,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/4181466530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":4181466530},{"type":"v1","addr":"172.21.15.174:6837","nonce":4181466530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":197}],"filesystems":[{"mdsmap":{"epoch":191,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:23:34.013510+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8244":{"gid":8244,"name":"c","rank":0,"incarnation":188,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6835/1971675962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1971675962},{"type":"v1","addr":"172.21.15.116:6835","nonce":1971675962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8244,"qdb_cluster":[8244]},"id":1},{"mdsmap":{"epoch":196,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:24:19.977863+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8343":{"gid":8343,"name":"a","rank":0,"incarnation":193,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6833/3633657268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3633657268},{"type":"v1","addr":"172.21.15.116:6833","nonce":3633657268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8343,"qdb_cluster":[8343]},"id":2}]} 2024-04-25T00:25:23.435 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 197 2024-04-25T00:25:23.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.431+0000 7fbb1406d640 1 -- 172.21.15.116:0/499222939 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbaf0067990 msgr2=0x7fbaf0069e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:23.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.431+0000 7fbb1406d640 1 --2- 172.21.15.116:0/499222939 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbaf0067990 0x7fbaf0069e50 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fbb08007fd0 tx=0x7fbb08009040 comp rx=0 tx=0).stop 2024-04-25T00:25:23.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.431+0000 7fbb1406d640 1 -- 172.21.15.116:0/499222939 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb0c156200 msgr2=0x7fbb0c0ab3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:23.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.431+0000 7fbb1406d640 1 --2- 172.21.15.116:0/499222939 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb0c156200 0x7fbb0c0ab3f0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fbb040ac000 tx=0x7fbb04019710 comp rx=0 tx=0).stop 2024-04-25T00:25:23.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.431+0000 7fbb1406d640 1 -- 172.21.15.116:0/499222939 shutdown_connections 2024-04-25T00:25:23.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.431+0000 7fbb1406d640 1 --2- 172.21.15.116:0/499222939 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbaf0067990 0x7fbaf0069e50 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:23.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.431+0000 7fbb1406d640 1 --2- 172.21.15.116:0/499222939 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb0c156200 0x7fbb0c0ab3f0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:23.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.431+0000 7fbb1406d640 1 --2- 172.21.15.116:0/499222939 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb0c0c4300 0x7fbb0c06b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:23.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.431+0000 7fbb1406d640 1 --2- 172.21.15.116:0/499222939 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb0c0aa170 0x7fbb0c14bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:23.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.431+0000 7fbb1406d640 1 -- 172.21.15.116:0/499222939 >> 172.21.15.116:0/499222939 conn(0x7fbb0c06c720 msgr2=0x7fbb0c0c5100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:23.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.435+0000 7fbb1406d640 1 -- 172.21.15.116:0/499222939 shutdown_connections 2024-04-25T00:25:23.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.435+0000 7fbb1406d640 1 -- 172.21.15.116:0/499222939 wait complete. 2024-04-25T00:25:23.459 INFO:tasks.mds_thrash.fs.[b]:kill mds.a (rank=0) 2024-04-25T00:25:23.459 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-25T00:25:23.459 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:25:23.515 INFO:tasks.ceph.mds.a:Stopped 2024-04-25T00:25:23.516 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:25:23.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.687+0000 7fe2a5bbd640 1 Processor -- start 2024-04-25T00:25:23.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.687+0000 7fe2a5bbd640 1 -- start start 2024-04-25T00:25:23.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.687+0000 7fe2a5bbd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe2a0152f60 0x7fe2a00c6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:23.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.687+0000 7fe2a5bbd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe2a00c3c30 0x7fe2a006fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:23.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.687+0000 7fe2a5bbd640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2a006e8f0 con 0x7fe2a0158ea0 2024-04-25T00:25:23.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.687+0000 7fe2a5bbd640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fe2a006ebd0 con 0x7fe2a00c3c30 2024-04-25T00:25:23.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.687+0000 7fe2a5bbd640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe2a006f330 con 0x7fe2a0152f60 2024-04-25T00:25:23.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.687+0000 7fe297fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe2a00c3c30 0x7fe2a006fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:23.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.687+0000 7fe297fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe2a00c3c30 0x7fe2a006fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52966/0 (socket says 172.21.15.116:52966) 2024-04-25T00:25:23.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.687+0000 7fe297fff640 1 -- 172.21.15.116:0/2494864049 learned_addr learned my addr 172.21.15.116:0/2494864049 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:25:23.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.687+0000 7fe29f7fe640 1 --2- 172.21.15.116:0/2494864049 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe2a0152f60 0x7fe2a00c6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:23.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.687+0000 7fe297fff640 1 -- 172.21.15.116:0/2494864049 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe2a0152f60 msgr2=0x7fe2a00c6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:23.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.687+0000 7fe297fff640 1 --2- 172.21.15.116:0/2494864049 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe2a0152f60 0x7fe2a00c6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:23.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.687+0000 7fe297fff640 1 -- 172.21.15.116:0/2494864049 >> v1:172.21.15.174:6789/0 conn(0x7fe2a0158ea0 legacy=0x7fe2a006c220 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T00:25:23.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.687+0000 7fe297fff640 1 -- 172.21.15.116:0/2494864049 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2a006f610 con 0x7fe2a00c3c30 2024-04-25T00:25:23.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.691+0000 7fe297fff640 1 --2- 172.21.15.116:0/2494864049 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2a00c3c30 0x7fe2a006fb30 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7fe288009870 tx=0x7fe28801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61c23f57a891c250 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:23.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.691+0000 7fe29effd640 1 -- 172.21.15.116:0/2494864049 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2880b0070 con 0x7fe2a00c3c30 2024-04-25T00:25:23.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.691+0000 7fe29effd640 1 -- 172.21.15.116:0/2494864049 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2880ab600 con 0x7fe2a00c3c30 2024-04-25T00:25:23.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.691+0000 7fe29effd640 1 -- 172.21.15.116:0/2494864049 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2880ab9d0 con 0x7fe2a00c3c30 2024-04-25T00:25:23.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.691+0000 7fe2a5bbd640 1 -- 172.21.15.116:0/2494864049 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2a00c3c30 msgr2=0x7fe2a006fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:23.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.691+0000 7fe2a5bbd640 1 --2- 172.21.15.116:0/2494864049 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2a00c3c30 0x7fe2a006fb30 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7fe288009870 tx=0x7fe28801c490 comp rx=0 tx=0).stop 2024-04-25T00:25:23.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.691+0000 7fe2a5bbd640 1 -- 172.21.15.116:0/2494864049 shutdown_connections 2024-04-25T00:25:23.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.691+0000 7fe2a5bbd640 1 --2- 172.21.15.116:0/2494864049 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2a00c3c30 0x7fe2a006fb30 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:23.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.691+0000 7fe2a5bbd640 1 --2- 172.21.15.116:0/2494864049 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe2a0152f60 0x7fe2a00c6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:23.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.691+0000 7fe2a5bbd640 1 -- 172.21.15.116:0/2494864049 >> 172.21.15.116:0/2494864049 conn(0x7fe2a00c4270 msgr2=0x7fe2a00c4680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:23.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.691+0000 7fe2a5bbd640 1 -- 172.21.15.116:0/2494864049 shutdown_connections 2024-04-25T00:25:23.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.691+0000 7fe2a5bbd640 1 -- 172.21.15.116:0/2494864049 wait complete. 2024-04-25T00:25:23.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.691+0000 7fe2a5bbd640 1 Processor -- start 2024-04-25T00:25:23.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.691+0000 7fe2a5bbd640 1 -- start start 2024-04-25T00:25:23.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.695+0000 7fe2a5bbd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe2a0152f60 0x7fe2a01f23f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:23.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.695+0000 7fe2a5bbd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe2a0158ea0 0x7fe2a00721c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:23.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.695+0000 7fe2a5bbd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2a00acf70 0x7fe2a0149a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:23.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.695+0000 7fe2a5bbd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe2a0162b70 con 0x7fe2a00acf70 2024-04-25T00:25:23.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.695+0000 7fe2a5bbd640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe2a01629f0 con 0x7fe2a0158ea0 2024-04-25T00:25:23.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.695+0000 7fe2a5bbd640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe2a0162cf0 con 0x7fe2a0152f60 2024-04-25T00:25:23.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.695+0000 7fe29ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2a00acf70 0x7fe2a0149a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:23.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.695+0000 7fe29ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2a00acf70 0x7fe2a0149a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:52968/0 (socket says 172.21.15.116:52968) 2024-04-25T00:25:23.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.695+0000 7fe29ffff640 1 -- 172.21.15.116:0/94098941 learned_addr learned my addr 172.21.15.116:0/94098941 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:25:23.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.695+0000 7fe29ffff640 1 -- 172.21.15.116:0/94098941 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe2a0152f60 msgr2=0x7fe2a01f23f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:23.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.695+0000 7fe29ffff640 1 --2- 172.21.15.116:0/94098941 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe2a0152f60 0x7fe2a01f23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:23.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.695+0000 7fe29ffff640 1 -- 172.21.15.116:0/94098941 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe2a0158ea0 msgr2=0x7fe2a00721c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:23.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.695+0000 7fe29ffff640 1 --2- 172.21.15.116:0/94098941 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe2a0158ea0 0x7fe2a00721c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:23.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.695+0000 7fe29ffff640 1 -- 172.21.15.116:0/94098941 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2a0151700 con 0x7fe2a00acf70 2024-04-25T00:25:23.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.695+0000 7fe29ffff640 1 --2- 172.21.15.116:0/94098941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2a00acf70 0x7fe2a0149a40 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7fe29800b9c0 tx=0x7fe29800be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:23.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.695+0000 7fe29cff9640 1 -- 172.21.15.116:0/94098941 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe298019070 con 0x7fe2a00acf70 2024-04-25T00:25:23.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.695+0000 7fe29cff9640 1 -- 172.21.15.116:0/94098941 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe29800cb40 con 0x7fe2a00acf70 2024-04-25T00:25:23.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.695+0000 7fe2a5bbd640 1 -- 172.21.15.116:0/94098941 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2a01519e0 con 0x7fe2a00acf70 2024-04-25T00:25:23.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.695+0000 7fe29cff9640 1 -- 172.21.15.116:0/94098941 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe298014440 con 0x7fe2a00acf70 2024-04-25T00:25:23.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.695+0000 7fe2a5bbd640 1 -- 172.21.15.116:0/94098941 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2a0151fa0 con 0x7fe2a00acf70 2024-04-25T00:25:23.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.695+0000 7fe2a5bbd640 1 -- 172.21.15.116:0/94098941 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2a0068a60 con 0x7fe2a00acf70 2024-04-25T00:25:23.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.695+0000 7fe29cff9640 1 -- 172.21.15.116:0/94098941 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe29801c070 con 0x7fe2a00acf70 2024-04-25T00:25:23.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.695+0000 7fe29cff9640 1 --2- 172.21.15.116:0/94098941 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe2800678f0 0x7fe280069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:23.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.695+0000 7fe29cff9640 1 -- 172.21.15.116:0/94098941 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 9848+0+0 (secure 0 0 0) 0x7fe2980869c0 con 0x7fe2a00acf70 2024-04-25T00:25:23.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.695+0000 7fe29f7fe640 1 --2- 172.21.15.116:0/94098941 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe2800678f0 0x7fe280069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:23.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.699+0000 7fe29f7fe640 1 --2- 172.21.15.116:0/94098941 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe2800678f0 0x7fe280069db0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fe290006aa0 tx=0x7fe290006a30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:23.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.703+0000 7fe29cff9640 1 -- 172.21.15.116:0/94098941 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fe298088e20 con 0x7fe2a00acf70 2024-04-25T00:25:23.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.975+0000 7fe2a5bbd640 1 -- 172.21.15.116:0/94098941 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe2a00c54a0 con 0x7fe2a00acf70 2024-04-25T00:25:23.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.975+0000 7fe29cff9640 1 -- 172.21.15.116:0/94098941 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 197 v197) ==== 95+0+6510 (secure 0 0 0) 0x7fe298056510 con 0x7fe2a00acf70 2024-04-25T00:25:23.980 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:25:23.980 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":197,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6413,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/4181466530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":4181466530},{"type":"v1","addr":"172.21.15.174:6837","nonce":4181466530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":197}],"filesystems":[{"mdsmap":{"epoch":191,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:23:34.013510+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8244":{"gid":8244,"name":"c","rank":0,"incarnation":188,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6835/1971675962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1971675962},{"type":"v1","addr":"172.21.15.116:6835","nonce":1971675962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8244,"qdb_cluster":[8244]},"id":1},{"mdsmap":{"epoch":196,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:24:19.977863+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8343":{"gid":8343,"name":"a","rank":0,"incarnation":193,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6833/3633657268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3633657268},{"type":"v1","addr":"172.21.15.116:6833","nonce":3633657268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8343,"qdb_cluster":[8343]},"id":2}]} 2024-04-25T00:25:23.980 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 197 2024-04-25T00:25:23.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.979+0000 7fe2a5bbd640 1 -- 172.21.15.116:0/94098941 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe2800678f0 msgr2=0x7fe280069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:23.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.979+0000 7fe2a5bbd640 1 --2- 172.21.15.116:0/94098941 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe2800678f0 0x7fe280069db0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fe290006aa0 tx=0x7fe290006a30 comp rx=0 tx=0).stop 2024-04-25T00:25:23.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.979+0000 7fe2a5bbd640 1 -- 172.21.15.116:0/94098941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2a00acf70 msgr2=0x7fe2a0149a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:23.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.979+0000 7fe2a5bbd640 1 --2- 172.21.15.116:0/94098941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2a00acf70 0x7fe2a0149a40 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7fe29800b9c0 tx=0x7fe29800be80 comp rx=0 tx=0).stop 2024-04-25T00:25:23.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.979+0000 7fe2a5bbd640 1 -- 172.21.15.116:0/94098941 shutdown_connections 2024-04-25T00:25:23.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.979+0000 7fe2a5bbd640 1 --2- 172.21.15.116:0/94098941 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe2800678f0 0x7fe280069db0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:23.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.979+0000 7fe2a5bbd640 1 --2- 172.21.15.116:0/94098941 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe2a00acf70 0x7fe2a0149a40 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:23.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.979+0000 7fe2a5bbd640 1 --2- 172.21.15.116:0/94098941 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe2a0158ea0 0x7fe2a00721c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:23.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.979+0000 7fe2a5bbd640 1 --2- 172.21.15.116:0/94098941 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe2a0152f60 0x7fe2a01f23f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:23.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.979+0000 7fe2a5bbd640 1 -- 172.21.15.116:0/94098941 >> 172.21.15.116:0/94098941 conn(0x7fe2a00c4270 msgr2=0x7fe2a00a7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:23.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.979+0000 7fe2a5bbd640 1 -- 172.21.15.116:0/94098941 shutdown_connections 2024-04-25T00:25:23.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:23.979+0000 7fe2a5bbd640 1 -- 172.21.15.116:0/94098941 wait complete. 2024-04-25T00:25:24.004 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:25:25.147 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:25:25.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.287+0000 7fb7bf9bc640 1 Processor -- start 2024-04-25T00:25:25.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.287+0000 7fb7bf9bc640 1 -- start start 2024-04-25T00:25:25.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.291+0000 7fb7bf9bc640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb7b80c3240 0x7fb7b815ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:25.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.291+0000 7fb7bf9bc640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7b80c36b0 con 0x7fb7b8153620 2024-04-25T00:25:25.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.291+0000 7fb7bf9bc640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7b806f8c0 con 0x7fb7b8150d20 2024-04-25T00:25:25.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.291+0000 7fb7bf9bc640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fb7b8153a80 con 0x7fb7b80c3240 2024-04-25T00:25:25.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.291+0000 7fb7bcf57640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fb7b8153620 0x7fb7b8071050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54070/0 (socket says 172.21.15.116:54070) 2024-04-25T00:25:25.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.291+0000 7fb7bcf57640 1 -- 172.21.15.116:0/2149523593 learned_addr learned my addr 172.21.15.116:0/2149523593 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:25:25.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.291+0000 7fb7bdf59640 1 --2- 172.21.15.116:0/2149523593 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb7b80c3240 0x7fb7b815ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:25.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.291+0000 7fb7affff640 1 -- 172.21.15.116:0/2149523593 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2170349621 0 0) 0x7fb7a8001120 con 0x7fb7b8153620 2024-04-25T00:25:25.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.291+0000 7fb7affff640 1 -- 172.21.15.116:0/2149523593 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb7a4003530 con 0x7fb7b8153620 2024-04-25T00:25:25.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.291+0000 7fb7bdf59640 1 -- 172.21.15.116:0/2149523593 >> v1:172.21.15.174:6789/0 conn(0x7fb7b8150d20 legacy=0x7fb7b80c3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:25.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.291+0000 7fb7affff640 1 -- 172.21.15.116:0/2149523593 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 356699894 0 0) 0x7fb7a4003530 con 0x7fb7b8153620 2024-04-25T00:25:25.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.291+0000 7fb7bdf59640 1 -- 172.21.15.116:0/2149523593 >> v1:172.21.15.116:6789/0 conn(0x7fb7b8153620 legacy=0x7fb7b8071050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:25.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.291+0000 7fb7bdf59640 1 -- 172.21.15.116:0/2149523593 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7b80c5420 con 0x7fb7b80c3240 2024-04-25T00:25:25.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.295+0000 7fb7bdf59640 1 --2- 172.21.15.116:0/2149523593 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb7b80c3240 0x7fb7b815ef20 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb7b0009dc0 tx=0x7fb7b001c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f40b23d33e91621a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:25.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.295+0000 7fb7affff640 1 -- 172.21.15.116:0/2149523593 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b00b2070 con 0x7fb7b80c3240 2024-04-25T00:25:25.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.295+0000 7fb7affff640 1 -- 172.21.15.116:0/2149523593 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7b00a5600 con 0x7fb7b80c3240 2024-04-25T00:25:25.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.295+0000 7fb7affff640 1 -- 172.21.15.116:0/2149523593 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a8001850 con 0x7fb7b80c3240 2024-04-25T00:25:25.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.295+0000 7fb7bf9bc640 1 -- 172.21.15.116:0/2149523593 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb7b80c3240 msgr2=0x7fb7b815ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:25.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.295+0000 7fb7bf9bc640 1 --2- 172.21.15.116:0/2149523593 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb7b80c3240 0x7fb7b815ef20 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb7b0009dc0 tx=0x7fb7b001c8b0 comp rx=0 tx=0).stop 2024-04-25T00:25:25.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.295+0000 7fb7bf9bc640 1 -- 172.21.15.116:0/2149523593 shutdown_connections 2024-04-25T00:25:25.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.295+0000 7fb7bf9bc640 1 --2- 172.21.15.116:0/2149523593 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb7b80c3240 0x7fb7b815ef20 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:25.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.295+0000 7fb7bf9bc640 1 -- 172.21.15.116:0/2149523593 >> 172.21.15.116:0/2149523593 conn(0x7fb7b80c9160 msgr2=0x7fb7b80c9570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:25.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.299+0000 7fb7bf9bc640 1 -- 172.21.15.116:0/2149523593 shutdown_connections 2024-04-25T00:25:25.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.299+0000 7fb7bf9bc640 1 -- 172.21.15.116:0/2149523593 wait complete. 2024-04-25T00:25:25.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.299+0000 7fb7bf9bc640 1 Processor -- start 2024-04-25T00:25:25.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.299+0000 7fb7bf9bc640 1 -- start start 2024-04-25T00:25:25.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.299+0000 7fb7bf9bc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb7b80c3240 0x7fb7b814fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:25.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.299+0000 7fb7bf9bc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7b8150d20 0x7fb7b814afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:25.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.299+0000 7fb7bf9bc640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb7b8153620 0x7fb7b814b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:25.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.299+0000 7fb7bf9bc640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b806f4c0 con 0x7fb7b8150d20 2024-04-25T00:25:25.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.299+0000 7fb7bf9bc640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b806fba0 con 0x7fb7b80c3240 2024-04-25T00:25:25.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.299+0000 7fb7bf9bc640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fb7b806f640 con 0x7fb7b8153620 2024-04-25T00:25:25.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.299+0000 7fb7bcf57640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7b8150d20 0x7fb7b814afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:25.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.299+0000 7fb7bcf57640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7b8150d20 0x7fb7b814afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44272/0 (socket says 172.21.15.116:44272) 2024-04-25T00:25:25.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.299+0000 7fb7bcf57640 1 -- 172.21.15.116:0/3169934897 learned_addr learned my addr 172.21.15.116:0/3169934897 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:25:25.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.299+0000 7fb7bd758640 1 --2- 172.21.15.116:0/3169934897 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb7b80c3240 0x7fb7b814fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:25.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.299+0000 7fb7bdf59640 1 --2- 172.21.15.116:0/3169934897 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb7b8153620 0x7fb7b814b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:25.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.299+0000 7fb7bcf57640 1 -- 172.21.15.116:0/3169934897 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb7b8153620 msgr2=0x7fb7b814b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:25.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.299+0000 7fb7bcf57640 1 --2- 172.21.15.116:0/3169934897 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb7b8153620 0x7fb7b814b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:25.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.299+0000 7fb7bcf57640 1 -- 172.21.15.116:0/3169934897 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb7b80c3240 msgr2=0x7fb7b814fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:25.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.299+0000 7fb7bcf57640 1 --2- 172.21.15.116:0/3169934897 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb7b80c3240 0x7fb7b814fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:25.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.299+0000 7fb7bcf57640 1 -- 172.21.15.116:0/3169934897 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7b80ac5c0 con 0x7fb7b8150d20 2024-04-25T00:25:25.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.299+0000 7fb7bcf57640 1 --2- 172.21.15.116:0/3169934897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7b8150d20 0x7fb7b814afd0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7fb7a800c780 tx=0x7fb7a800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:25.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.303+0000 7fb7adffb640 1 -- 172.21.15.116:0/3169934897 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a8005ae0 con 0x7fb7b8150d20 2024-04-25T00:25:25.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.303+0000 7fb7adffb640 1 -- 172.21.15.116:0/3169934897 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7a8005ca0 con 0x7fb7b8150d20 2024-04-25T00:25:25.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.303+0000 7fb7bf9bc640 1 -- 172.21.15.116:0/3169934897 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7b80aec40 con 0x7fb7b8150d20 2024-04-25T00:25:25.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.303+0000 7fb7adffb640 1 -- 172.21.15.116:0/3169934897 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a80072b0 con 0x7fb7b8150d20 2024-04-25T00:25:25.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.303+0000 7fb7bf9bc640 1 -- 172.21.15.116:0/3169934897 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7b814d2e0 con 0x7fb7b8150d20 2024-04-25T00:25:25.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.303+0000 7fb7adffb640 1 -- 172.21.15.116:0/3169934897 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb7a8007450 con 0x7fb7b8150d20 2024-04-25T00:25:25.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.303+0000 7fb7adffb640 1 --2- 172.21.15.116:0/3169934897 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb79c067940 0x7fb79c069e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:25.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.303+0000 7fb7adffb640 1 -- 172.21.15.116:0/3169934897 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 9848+0+0 (secure 0 0 0) 0x7fb7a8086830 con 0x7fb7b8150d20 2024-04-25T00:25:25.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.303+0000 7fb7bd758640 1 --2- 172.21.15.116:0/3169934897 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb79c067940 0x7fb79c069e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:25.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.303+0000 7fb7bf9bc640 1 -- 172.21.15.116:0/3169934897 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7b8068a60 con 0x7fb7b8150d20 2024-04-25T00:25:25.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.303+0000 7fb7bd758640 1 --2- 172.21.15.116:0/3169934897 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb79c067940 0x7fb79c069e00 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fb7b80af2b0 tx=0x7fb7b4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:25.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.311+0000 7fb7adffb640 1 -- 172.21.15.116:0/3169934897 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fb7b80c36b0 con 0x7fb7b8150d20 2024-04-25T00:25:25.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.611+0000 7fb7bf9bc640 1 -- 172.21.15.116:0/3169934897 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb7b80aaac0 con 0x7fb7b8150d20 2024-04-25T00:25:25.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.611+0000 7fb7adffb640 1 -- 172.21.15.116:0/3169934897 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 197 v197) ==== 95+0+6510 (secure 0 0 0) 0x7fb7a8055e70 con 0x7fb7b8150d20 2024-04-25T00:25:25.615 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:25:25.615 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":197,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6413,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/4181466530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":4181466530},{"type":"v1","addr":"172.21.15.174:6837","nonce":4181466530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":197}],"filesystems":[{"mdsmap":{"epoch":191,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:23:34.013510+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8244},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8244":{"gid":8244,"name":"c","rank":0,"incarnation":188,"state":"up:active","state_seq":11,"addr":"172.21.15.116:6835/1971675962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":1971675962},{"type":"v1","addr":"172.21.15.116:6835","nonce":1971675962}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8244,"qdb_cluster":[8244]},"id":1},{"mdsmap":{"epoch":196,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:24:19.977863+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8343":{"gid":8343,"name":"a","rank":0,"incarnation":193,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6833/3633657268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3633657268},{"type":"v1","addr":"172.21.15.116:6833","nonce":3633657268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8343,"qdb_cluster":[8343]},"id":2}]} 2024-04-25T00:25:25.615 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 197 2024-04-25T00:25:25.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.611+0000 7fb7bf9bc640 1 -- 172.21.15.116:0/3169934897 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb79c067940 msgr2=0x7fb79c069e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:25.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.611+0000 7fb7bf9bc640 1 --2- 172.21.15.116:0/3169934897 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb79c067940 0x7fb79c069e00 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fb7b80af2b0 tx=0x7fb7b4009040 comp rx=0 tx=0).stop 2024-04-25T00:25:25.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.611+0000 7fb7bf9bc640 1 -- 172.21.15.116:0/3169934897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7b8150d20 msgr2=0x7fb7b814afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:25.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.611+0000 7fb7bf9bc640 1 --2- 172.21.15.116:0/3169934897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7b8150d20 0x7fb7b814afd0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7fb7a800c780 tx=0x7fb7a800cc40 comp rx=0 tx=0).stop 2024-04-25T00:25:25.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.611+0000 7fb7bf9bc640 1 -- 172.21.15.116:0/3169934897 shutdown_connections 2024-04-25T00:25:25.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.611+0000 7fb7bf9bc640 1 --2- 172.21.15.116:0/3169934897 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb79c067940 0x7fb79c069e00 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:25.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.611+0000 7fb7bf9bc640 1 --2- 172.21.15.116:0/3169934897 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb7b8153620 0x7fb7b814b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:25.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.611+0000 7fb7bf9bc640 1 --2- 172.21.15.116:0/3169934897 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb7b8150d20 0x7fb7b814afd0 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:25.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.611+0000 7fb7bf9bc640 1 --2- 172.21.15.116:0/3169934897 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb7b80c3240 0x7fb7b814fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:25.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.611+0000 7fb7bf9bc640 1 -- 172.21.15.116:0/3169934897 >> 172.21.15.116:0/3169934897 conn(0x7fb7b80c9160 msgr2=0x7fb7b814db20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:25.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.611+0000 7fb7bf9bc640 1 -- 172.21.15.116:0/3169934897 shutdown_connections 2024-04-25T00:25:25.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:25.615+0000 7fb7bf9bc640 1 -- 172.21.15.116:0/3169934897 wait complete. 2024-04-25T00:25:25.639 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:25:26.006 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:25:26.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.199+0000 7f369ede2640 1 Processor -- start 2024-04-25T00:25:26.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.199+0000 7f369ede2640 1 -- start start 2024-04-25T00:25:26.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.203+0000 7f369ede2640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f36900d7bd0 0x7f36900daf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:26.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.203+0000 7f369ede2640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f36900de4d0 0x7f369000c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:26.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.203+0000 7f369dde0640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f36900d7bd0 0x7f36900daf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:26.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.203+0000 7f369ede2640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3690009010 con 0x7f3690008710 2024-04-25T00:25:26.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.203+0000 7f369ede2640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f36900dda80 con 0x7f36900de4d0 2024-04-25T00:25:26.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.203+0000 7f369ede2640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f36900de040 con 0x7f36900d7bd0 2024-04-25T00:25:26.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.203+0000 7f369d5df640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f3690008710 0x7f3690048090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:50434/0 (socket says 172.21.15.116:50434) 2024-04-25T00:25:26.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.203+0000 7f369d5df640 1 -- 172.21.15.116:0/1795693189 learned_addr learned my addr 172.21.15.116:0/1795693189 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:25:26.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.203+0000 7f369e5e1640 1 --2- 172.21.15.116:0/1795693189 >> v2:172.21.15.116:3300/0 conn(0x7f36900de4d0 0x7f369000c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:26.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.203+0000 7f369dde0640 1 -- 172.21.15.116:0/1795693189 >> v2:172.21.15.116:3300/0 conn(0x7f36900de4d0 msgr2=0x7f369000c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:26.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.203+0000 7f369dde0640 1 --2- 172.21.15.116:0/1795693189 >> v2:172.21.15.116:3300/0 conn(0x7f36900de4d0 0x7f369000c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:26.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.203+0000 7f369dde0640 1 -- 172.21.15.116:0/1795693189 >> v1:172.21.15.174:6789/0 conn(0x7f3690008710 legacy=0x7f3690048090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:26.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.203+0000 7f369dde0640 1 -- 172.21.15.116:0/1795693189 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36900092f0 con 0x7f36900d7bd0 2024-04-25T00:25:26.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.207+0000 7f369dde0640 1 --2- 172.21.15.116:0/1795693189 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36900d7bd0 0x7f36900daf90 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f369000d680 tx=0x7f369401c460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=27afc4796a59a3a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:26.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.207+0000 7f369cdde640 1 -- 172.21.15.116:0/1795693189 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36940b0070 con 0x7f36900d7bd0 2024-04-25T00:25:26.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.207+0000 7f369cdde640 1 -- 172.21.15.116:0/1795693189 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36940ab600 con 0x7f36900d7bd0 2024-04-25T00:25:26.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.207+0000 7f369cdde640 1 -- 172.21.15.116:0/1795693189 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36940ab9d0 con 0x7f36900d7bd0 2024-04-25T00:25:26.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.207+0000 7f369ede2640 1 -- 172.21.15.116:0/1795693189 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36900d7bd0 msgr2=0x7f36900daf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:26.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.207+0000 7f369ede2640 1 --2- 172.21.15.116:0/1795693189 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36900d7bd0 0x7f36900daf90 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f369000d680 tx=0x7f369401c460 comp rx=0 tx=0).stop 2024-04-25T00:25:26.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.207+0000 7f369ede2640 1 -- 172.21.15.116:0/1795693189 shutdown_connections 2024-04-25T00:25:26.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.207+0000 7f369ede2640 1 --2- 172.21.15.116:0/1795693189 >> v2:172.21.15.116:3300/0 conn(0x7f36900de4d0 0x7f369000c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:26.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.207+0000 7f369ede2640 1 --2- 172.21.15.116:0/1795693189 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f36900d7bd0 0x7f36900daf90 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:26.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.207+0000 7f369ede2640 1 -- 172.21.15.116:0/1795693189 >> 172.21.15.116:0/1795693189 conn(0x7f3690007bf0 msgr2=0x7f3690008000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:26.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.207+0000 7f369ede2640 1 -- 172.21.15.116:0/1795693189 shutdown_connections 2024-04-25T00:25:26.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.207+0000 7f369ede2640 1 -- 172.21.15.116:0/1795693189 wait complete. 2024-04-25T00:25:26.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.211+0000 7f369ede2640 1 Processor -- start 2024-04-25T00:25:26.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.211+0000 7f369ede2640 1 -- start start 2024-04-25T00:25:26.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.211+0000 7f369ede2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3690008710 0x7f3690031200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:26.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.211+0000 7f369ede2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36900d7bd0 0x7f369002f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:26.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.211+0000 7f369ede2640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f36900de4d0 0x7f3690045dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:26.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.211+0000 7f369ede2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f36900e7840 con 0x7f36900d7bd0 2024-04-25T00:25:26.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.211+0000 7f369ede2640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f36900d7fb0 con 0x7f3690008710 2024-04-25T00:25:26.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.211+0000 7f369ede2640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f36900e79c0 con 0x7f36900de4d0 2024-04-25T00:25:26.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.211+0000 7f369e5e1640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f36900de4d0 0x7f3690045dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:26.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.211+0000 7f369d5df640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36900d7bd0 0x7f369002f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:26.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.211+0000 7f369e5e1640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f36900de4d0 0x7f3690045dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:45102/0 (socket says 172.21.15.116:45102) 2024-04-25T00:25:26.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.211+0000 7f369d5df640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36900d7bd0 0x7f369002f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44288/0 (socket says 172.21.15.116:44288) 2024-04-25T00:25:26.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.211+0000 7f369e5e1640 1 -- 172.21.15.116:0/2371122099 learned_addr learned my addr 172.21.15.116:0/2371122099 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:25:26.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.211+0000 7f369e5e1640 1 -- 172.21.15.116:0/2371122099 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3690008710 msgr2=0x7f3690031200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:26.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.211+0000 7f369e5e1640 1 --2- 172.21.15.116:0/2371122099 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3690008710 0x7f3690031200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:26.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.211+0000 7f369e5e1640 1 -- 172.21.15.116:0/2371122099 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36900d7bd0 msgr2=0x7f369002f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:26.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.211+0000 7f369e5e1640 1 --2- 172.21.15.116:0/2371122099 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36900d7bd0 0x7f369002f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:26.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.211+0000 7f369e5e1640 1 -- 172.21.15.116:0/2371122099 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3690173f70 con 0x7f36900de4d0 2024-04-25T00:25:26.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.211+0000 7f369d5df640 1 --2- 172.21.15.116:0/2371122099 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36900d7bd0 0x7f369002f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:25:26.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.211+0000 7f369e5e1640 1 --2- 172.21.15.116:0/2371122099 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f36900de4d0 0x7f3690045dc0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f369806a140 tx=0x7f369804f740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:26.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.211+0000 7f368e7fc640 1 -- 172.21.15.116:0/2371122099 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3698073390 con 0x7f36900de4d0 2024-04-25T00:25:26.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.211+0000 7f368e7fc640 1 -- 172.21.15.116:0/2371122099 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3698073530 con 0x7f36900de4d0 2024-04-25T00:25:26.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.211+0000 7f369ede2640 1 -- 172.21.15.116:0/2371122099 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f369002fbc0 con 0x7f36900de4d0 2024-04-25T00:25:26.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.211+0000 7f368e7fc640 1 -- 172.21.15.116:0/2371122099 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f369806e410 con 0x7f36900de4d0 2024-04-25T00:25:26.218 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.211+0000 7f369ede2640 1 -- 172.21.15.116:0/2371122099 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3690173c90 con 0x7f36900de4d0 2024-04-25T00:25:26.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.215+0000 7f369ede2640 1 -- 172.21.15.116:0/2371122099 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36900db4e0 con 0x7f36900de4d0 2024-04-25T00:25:26.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.215+0000 7f368e7fc640 1 -- 172.21.15.116:0/2371122099 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f369806e5b0 con 0x7f36900de4d0 2024-04-25T00:25:26.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.215+0000 7f368e7fc640 1 --2- 172.21.15.116:0/2371122099 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f367c0678c0 0x7f367c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:26.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.215+0000 7f368e7fc640 1 -- 172.21.15.116:0/2371122099 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(66..66 src has 1..66) ==== 9934+0+0 (secure 0 0 0) 0x7f36981019a0 con 0x7f36900de4d0 2024-04-25T00:25:26.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.215+0000 7f369dde0640 1 --2- 172.21.15.116:0/2371122099 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f367c0678c0 0x7f367c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:26.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.215+0000 7f369dde0640 1 --2- 172.21.15.116:0/2371122099 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f367c0678c0 0x7f367c069d80 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f36940ad000 tx=0x7f3694004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:26.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.223+0000 7f368e7fc640 1 -- 172.21.15.116:0/2371122099 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f36900db4e0 con 0x7f36900de4d0 2024-04-25T00:25:26.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.495+0000 7f369ede2640 1 -- 172.21.15.116:0/2371122099 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3690173a80 con 0x7f36900de4d0 2024-04-25T00:25:26.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.495+0000 7f368e7fc640 1 -- 172.21.15.116:0/2371122099 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 198 v198) ==== 95+0+5673 (secure 0 0 0) 0x7f36980d13b0 con 0x7f36900de4d0 2024-04-25T00:25:26.499 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:25:26.499 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":198,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":198,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:25:25.722272+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6413},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6413":{"gid":6413,"name":"d","rank":0,"incarnation":198,"state":"up:replay","state_seq":1,"addr":"172.21.15.174:6837/4181466530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":4181466530},{"type":"v1","addr":"172.21.15.174:6837","nonce":4181466530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":196,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:24:19.977863+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8343":{"gid":8343,"name":"a","rank":0,"incarnation":193,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6833/3633657268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3633657268},{"type":"v1","addr":"172.21.15.116:6833","nonce":3633657268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8343,"qdb_cluster":[8343]},"id":2}]} 2024-04-25T00:25:26.499 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 198 2024-04-25T00:25:26.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.499+0000 7f369ede2640 1 -- 172.21.15.116:0/2371122099 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f367c0678c0 msgr2=0x7f367c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:26.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.499+0000 7f369ede2640 1 --2- 172.21.15.116:0/2371122099 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f367c0678c0 0x7f367c069d80 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f36940ad000 tx=0x7f3694004740 comp rx=0 tx=0).stop 2024-04-25T00:25:26.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.499+0000 7f369ede2640 1 -- 172.21.15.116:0/2371122099 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f36900de4d0 msgr2=0x7f3690045dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:26.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.499+0000 7f369ede2640 1 --2- 172.21.15.116:0/2371122099 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f36900de4d0 0x7f3690045dc0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f369806a140 tx=0x7f369804f740 comp rx=0 tx=0).stop 2024-04-25T00:25:26.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.499+0000 7f369ede2640 1 -- 172.21.15.116:0/2371122099 shutdown_connections 2024-04-25T00:25:26.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.499+0000 7f369ede2640 1 --2- 172.21.15.116:0/2371122099 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f367c0678c0 0x7f367c069d80 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:26.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.499+0000 7f369ede2640 1 --2- 172.21.15.116:0/2371122099 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f36900de4d0 0x7f3690045dc0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:26.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.499+0000 7f369ede2640 1 --2- 172.21.15.116:0/2371122099 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f36900d7bd0 0x7f369002f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:26.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.499+0000 7f369ede2640 1 --2- 172.21.15.116:0/2371122099 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3690008710 0x7f3690031200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:26.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.499+0000 7f369ede2640 1 -- 172.21.15.116:0/2371122099 >> 172.21.15.116:0/2371122099 conn(0x7f3690007bf0 msgr2=0x7f369002d840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:26.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.503+0000 7f369ede2640 1 -- 172.21.15.116:0/2371122099 shutdown_connections 2024-04-25T00:25:26.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:26.503+0000 7f369ede2640 1 -- 172.21.15.116:0/2371122099 wait complete. 2024-04-25T00:25:26.528 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:25:27.640 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:25:27.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.767+0000 7f211806e640 1 Processor -- start 2024-04-25T00:25:27.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.767+0000 7f211806e640 1 -- start start 2024-04-25T00:25:27.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.767+0000 7f211806e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f211014c720 0x7f21101563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:27.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.767+0000 7f211806e640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2110071220 con 0x7f2110071680 2024-04-25T00:25:27.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.767+0000 7f211806e640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2110152420 con 0x7f21100a94a0 2024-04-25T00:25:27.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.767+0000 7f211806e640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f2110153460 con 0x7f211014c720 2024-04-25T00:25:27.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.767+0000 7f2115e0a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f211014c720 0x7f21101563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:27.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.767+0000 7f211660b640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f2110071680 0x7f2110071ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54080/0 (socket says 172.21.15.116:54080) 2024-04-25T00:25:27.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.767+0000 7f2115e0a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f211014c720 0x7f21101563d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44294/0 (socket says 172.21.15.116:44294) 2024-04-25T00:25:27.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.767+0000 7f211660b640 1 -- 172.21.15.116:0/1797345476 learned_addr learned my addr 172.21.15.116:0/1797345476 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:25:27.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.767+0000 7f2115e0a640 1 -- 172.21.15.116:0/1797345476 >> v1:172.21.15.174:6789/0 conn(0x7f21100a94a0 legacy=0x7f21100c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:27.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.767+0000 7f2114e08640 1 -- 172.21.15.116:0/1797345476 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1970846860 0 0) 0x7f2110071220 con 0x7f2110071680 2024-04-25T00:25:27.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.767+0000 7f2115e0a640 1 -- 172.21.15.116:0/1797345476 >> v1:172.21.15.116:6789/0 conn(0x7f2110071680 legacy=0x7f2110071ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:27.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.767+0000 7f2115e0a640 1 -- 172.21.15.116:0/1797345476 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21100721c0 con 0x7f211014c720 2024-04-25T00:25:27.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.771+0000 7f2115e0a640 1 --2- 172.21.15.116:0/1797345476 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f211014c720 0x7f21101563d0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f210c0097c0 tx=0x7f210c01c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f16085862a7fdd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:27.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.771+0000 7f2114e08640 1 -- 172.21.15.116:0/1797345476 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f210c0ab610 con 0x7f211014c720 2024-04-25T00:25:27.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.771+0000 7f2114e08640 1 -- 172.21.15.116:0/1797345476 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f210c0b0070 con 0x7f211014c720 2024-04-25T00:25:27.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.771+0000 7f2114e08640 1 -- 172.21.15.116:0/1797345476 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f210c0ab9d0 con 0x7f211014c720 2024-04-25T00:25:27.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.771+0000 7f211806e640 1 -- 172.21.15.116:0/1797345476 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f211014c720 msgr2=0x7f21101563d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:27.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.771+0000 7f211806e640 1 --2- 172.21.15.116:0/1797345476 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f211014c720 0x7f21101563d0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f210c0097c0 tx=0x7f210c01c410 comp rx=0 tx=0).stop 2024-04-25T00:25:27.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.771+0000 7f211806e640 1 -- 172.21.15.116:0/1797345476 shutdown_connections 2024-04-25T00:25:27.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.771+0000 7f211806e640 1 --2- 172.21.15.116:0/1797345476 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f211014c720 0x7f21101563d0 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:27.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.771+0000 7f211806e640 1 -- 172.21.15.116:0/1797345476 >> 172.21.15.116:0/1797345476 conn(0x7f21100c3b00 msgr2=0x7f21100c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:27.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.771+0000 7f211806e640 1 -- 172.21.15.116:0/1797345476 shutdown_connections 2024-04-25T00:25:27.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.771+0000 7f211806e640 1 -- 172.21.15.116:0/1797345476 wait complete. 2024-04-25T00:25:27.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.771+0000 7f211806e640 1 Processor -- start 2024-04-25T00:25:27.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.771+0000 7f211806e640 1 -- start start 2024-04-25T00:25:27.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.771+0000 7f211806e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2110071680 0x7f21100c66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:27.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.771+0000 7f211806e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f21100a94a0 0x7f21100c0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:27.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.771+0000 7f211806e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f211014c720 0x7f211006d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:27.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.771+0000 7f211806e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2110152f40 con 0x7f2110071680 2024-04-25T00:25:27.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.771+0000 7f211806e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2110152c60 con 0x7f211014c720 2024-04-25T00:25:27.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.771+0000 7f211806e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f21100c3120 con 0x7f21100a94a0 2024-04-25T00:25:27.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.771+0000 7f2115e0a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2110071680 0x7f21100c66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:27.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.771+0000 7f2115e0a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2110071680 0x7f21100c66a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44300/0 (socket says 172.21.15.116:44300) 2024-04-25T00:25:27.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.771+0000 7f2115e0a640 1 -- 172.21.15.116:0/2099282782 learned_addr learned my addr 172.21.15.116:0/2099282782 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:25:27.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.771+0000 7f2115e0a640 1 -- 172.21.15.116:0/2099282782 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f21100a94a0 msgr2=0x7f21100c0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:27.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.771+0000 7f2115e0a640 1 --2- 172.21.15.116:0/2099282782 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f21100a94a0 0x7f21100c0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:27.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.771+0000 7f211660b640 1 --2- 172.21.15.116:0/2099282782 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f211014c720 0x7f211006d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:27.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.771+0000 7f2115e0a640 1 -- 172.21.15.116:0/2099282782 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f211014c720 msgr2=0x7f211006d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:27.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.771+0000 7f2115e0a640 1 --2- 172.21.15.116:0/2099282782 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f211014c720 0x7f211006d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:27.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.771+0000 7f2115e0a640 1 -- 172.21.15.116:0/2099282782 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21101f1b60 con 0x7f2110071680 2024-04-25T00:25:27.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.771+0000 7f2115e0a640 1 --2- 172.21.15.116:0/2099282782 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2110071680 0x7f21100c66a0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f210c0ad040 tx=0x7f210c01ac50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:27.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.771+0000 7f21067fc640 1 -- 172.21.15.116:0/2099282782 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f210c0bf610 con 0x7f2110071680 2024-04-25T00:25:27.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.771+0000 7f21067fc640 1 -- 172.21.15.116:0/2099282782 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f210c0bf7d0 con 0x7f2110071680 2024-04-25T00:25:27.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.771+0000 7f211806e640 1 -- 172.21.15.116:0/2099282782 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21101f1e40 con 0x7f2110071680 2024-04-25T00:25:27.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.771+0000 7f21067fc640 1 -- 172.21.15.116:0/2099282782 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f210c0bfbc0 con 0x7f2110071680 2024-04-25T00:25:27.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.771+0000 7f211806e640 1 -- 172.21.15.116:0/2099282782 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f211006cd50 con 0x7f2110071680 2024-04-25T00:25:27.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.775+0000 7f211806e640 1 -- 172.21.15.116:0/2099282782 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2110068350 con 0x7f2110071680 2024-04-25T00:25:27.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.775+0000 7f21067fc640 1 -- 172.21.15.116:0/2099282782 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f210c0bd020 con 0x7f2110071680 2024-04-25T00:25:27.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.775+0000 7f21067fc640 1 --2- 172.21.15.116:0/2099282782 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f20f40678c0 0x7f20f4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:27.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.775+0000 7f21067fc640 1 -- 172.21.15.116:0/2099282782 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 9934+0+0 (secure 0 0 0) 0x7f210c121530 con 0x7f2110071680 2024-04-25T00:25:27.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.775+0000 7f2115609640 1 --2- 172.21.15.116:0/2099282782 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f20f40678c0 0x7f20f4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:27.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.775+0000 7f2115609640 1 --2- 172.21.15.116:0/2099282782 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f20f40678c0 0x7f20f4069d80 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f2100000f70 tx=0x7f21000079c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:27.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.779+0000 7f21067fc640 1 -- 172.21.15.116:0/2099282782 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f210c0a8050 con 0x7f2110071680 2024-04-25T00:25:27.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.975+0000 7f211806e640 1 -- 172.21.15.116:0/2099282782 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f21100c3640 con 0x7f2110071680 2024-04-25T00:25:27.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.975+0000 7f21067fc640 1 -- 172.21.15.116:0/2099282782 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 199 v199) ==== 95+0+5677 (secure 0 0 0) 0x7f210c016090 con 0x7f2110071680 2024-04-25T00:25:27.979 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:25:27.979 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":199,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:25:26.648427+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6413},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6413":{"gid":6413,"name":"d","rank":0,"incarnation":198,"state":"up:reconnect","state_seq":16,"addr":"172.21.15.174:6837/4181466530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":4181466530},{"type":"v1","addr":"172.21.15.174:6837","nonce":4181466530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":196,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:24:19.977863+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8343":{"gid":8343,"name":"a","rank":0,"incarnation":193,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6833/3633657268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3633657268},{"type":"v1","addr":"172.21.15.116:6833","nonce":3633657268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8343,"qdb_cluster":[8343]},"id":2}]} 2024-04-25T00:25:27.979 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 199 2024-04-25T00:25:27.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.975+0000 7f211806e640 1 -- 172.21.15.116:0/2099282782 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f20f40678c0 msgr2=0x7f20f4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:27.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.975+0000 7f211806e640 1 --2- 172.21.15.116:0/2099282782 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f20f40678c0 0x7f20f4069d80 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f2100000f70 tx=0x7f21000079c0 comp rx=0 tx=0).stop 2024-04-25T00:25:27.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.975+0000 7f211806e640 1 -- 172.21.15.116:0/2099282782 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2110071680 msgr2=0x7f21100c66a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:27.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.975+0000 7f211806e640 1 --2- 172.21.15.116:0/2099282782 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2110071680 0x7f21100c66a0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7f210c0ad040 tx=0x7f210c01ac50 comp rx=0 tx=0).stop 2024-04-25T00:25:27.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.975+0000 7f211806e640 1 -- 172.21.15.116:0/2099282782 shutdown_connections 2024-04-25T00:25:27.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.975+0000 7f211806e640 1 --2- 172.21.15.116:0/2099282782 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f20f40678c0 0x7f20f4069d80 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:27.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.975+0000 7f211806e640 1 --2- 172.21.15.116:0/2099282782 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f211014c720 0x7f211006d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:27.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.975+0000 7f211806e640 1 --2- 172.21.15.116:0/2099282782 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f21100a94a0 0x7f21100c0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:27.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.975+0000 7f211806e640 1 --2- 172.21.15.116:0/2099282782 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2110071680 0x7f21100c66a0 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:27.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.975+0000 7f211806e640 1 -- 172.21.15.116:0/2099282782 >> 172.21.15.116:0/2099282782 conn(0x7f21100c3b00 msgr2=0x7f2110150d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:27.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.979+0000 7f211806e640 1 -- 172.21.15.116:0/2099282782 shutdown_connections 2024-04-25T00:25:27.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:27.979+0000 7f211806e640 1 -- 172.21.15.116:0/2099282782 wait complete. 2024-04-25T00:25:27.998 INFO:tasks.mds_thrash.fs.[a]:mds.c down, removed from mdsmap 2024-04-25T00:25:27.999 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-04-25T00:25:27.999 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:25:28.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.119+0000 7fe0f0b33640 1 Processor -- start 2024-04-25T00:25:28.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.119+0000 7fe0f0b33640 1 -- start start 2024-04-25T00:25:28.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.119+0000 7fe0f0b33640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe0ec152ce0 0x7fe0ec06bc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:28.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.119+0000 7fe0f0b33640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe0ec0c3ea0 0x7fe0ec0c4330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:28.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.119+0000 7fe0f0b33640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0ec073260 con 0x7fe0ec159970 2024-04-25T00:25:28.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.119+0000 7fe0f0b33640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fe0ec1594e0 con 0x7fe0ec152ce0 2024-04-25T00:25:28.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.119+0000 7fe0f0b33640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe0ec06c160 con 0x7fe0ec0c3ea0 2024-04-25T00:25:28.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.119+0000 7fe0ea59c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe0ec152ce0 0x7fe0ec06bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:28.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.119+0000 7fe0e9d9b640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fe0ec159970 0x7fe0ec155720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54094/0 (socket says 172.21.15.116:54094) 2024-04-25T00:25:28.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.119+0000 7fe0ea59c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe0ec152ce0 0x7fe0ec06bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44316/0 (socket says 172.21.15.116:44316) 2024-04-25T00:25:28.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.119+0000 7fe0e9d9b640 1 -- 172.21.15.116:0/3533233099 learned_addr learned my addr 172.21.15.116:0/3533233099 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:25:28.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.119+0000 7fe0ead9d640 1 --2- 172.21.15.116:0/3533233099 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe0ec0c3ea0 0x7fe0ec0c4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:28.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.119+0000 7fe0ea59c640 1 -- 172.21.15.116:0/3533233099 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe0ec0c3ea0 msgr2=0x7fe0ec0c4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:28.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.119+0000 7fe0ea59c640 1 --2- 172.21.15.116:0/3533233099 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe0ec0c3ea0 0x7fe0ec0c4330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:28.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.119+0000 7fe0ea59c640 1 -- 172.21.15.116:0/3533233099 >> v1:172.21.15.116:6789/0 conn(0x7fe0ec159970 legacy=0x7fe0ec155720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:28.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.119+0000 7fe0ea59c640 1 -- 172.21.15.116:0/3533233099 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0ec158e90 con 0x7fe0ec152ce0 2024-04-25T00:25:28.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.119+0000 7fe0e959a640 1 -- 172.21.15.116:0/3533233099 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3869668376 0 0) 0x7fe0ec073260 con 0x7fe0ec159970 2024-04-25T00:25:28.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.119+0000 7fe0ead9d640 1 --2- 172.21.15.116:0/3533233099 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe0ec0c3ea0 0x7fe0ec0c4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:25:28.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.119+0000 7fe0ea59c640 1 --2- 172.21.15.116:0/3533233099 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0ec152ce0 0x7fe0ec06bc20 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc0097c0 tx=0x7fe0dc01c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bb766263c4a4d3f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:28.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.123+0000 7fe0e959a640 1 -- 172.21.15.116:0/3533233099 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0dc0b0070 con 0x7fe0ec152ce0 2024-04-25T00:25:28.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.123+0000 7fe0e959a640 1 -- 172.21.15.116:0/3533233099 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0dc0ab600 con 0x7fe0ec152ce0 2024-04-25T00:25:28.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.123+0000 7fe0e959a640 1 -- 172.21.15.116:0/3533233099 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0dc0ab9d0 con 0x7fe0ec152ce0 2024-04-25T00:25:28.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.123+0000 7fe0f0b33640 1 -- 172.21.15.116:0/3533233099 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0ec152ce0 msgr2=0x7fe0ec06bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:28.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.123+0000 7fe0f0b33640 1 --2- 172.21.15.116:0/3533233099 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0ec152ce0 0x7fe0ec06bc20 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc0097c0 tx=0x7fe0dc01c3e0 comp rx=0 tx=0).stop 2024-04-25T00:25:28.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.123+0000 7fe0f0b33640 1 -- 172.21.15.116:0/3533233099 shutdown_connections 2024-04-25T00:25:28.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.123+0000 7fe0f0b33640 1 --2- 172.21.15.116:0/3533233099 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe0ec0c3ea0 0x7fe0ec0c4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:28.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.123+0000 7fe0f0b33640 1 --2- 172.21.15.116:0/3533233099 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0ec152ce0 0x7fe0ec06bc20 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:28.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.123+0000 7fe0f0b33640 1 -- 172.21.15.116:0/3533233099 >> 172.21.15.116:0/3533233099 conn(0x7fe0ec0c4c10 msgr2=0x7fe0ec0c5020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:28.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.123+0000 7fe0f0b33640 1 -- 172.21.15.116:0/3533233099 shutdown_connections 2024-04-25T00:25:28.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.123+0000 7fe0f0b33640 1 -- 172.21.15.116:0/3533233099 wait complete. 2024-04-25T00:25:28.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.123+0000 7fe0f0b33640 1 Processor -- start 2024-04-25T00:25:28.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.123+0000 7fe0f0b33640 1 -- start start 2024-04-25T00:25:28.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.123+0000 7fe0f0b33640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe0ec0c3ea0 0x7fe0ec14dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:28.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.123+0000 7fe0f0b33640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe0ec152ce0 0x7fe0ec1f1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:28.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.123+0000 7fe0f0b33640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0ec159970 0x7fe0ec1f1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:28.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.123+0000 7fe0f0b33640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe0ec073000 con 0x7fe0ec159970 2024-04-25T00:25:28.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.123+0000 7fe0f0b33640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe0ec072e80 con 0x7fe0ec152ce0 2024-04-25T00:25:28.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.123+0000 7fe0f0b33640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe0ec155e30 con 0x7fe0ec0c3ea0 2024-04-25T00:25:28.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.123+0000 7fe0ead9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0ec159970 0x7fe0ec1f1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:28.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.123+0000 7fe0ead9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0ec159970 0x7fe0ec1f1c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44328/0 (socket says 172.21.15.116:44328) 2024-04-25T00:25:28.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.123+0000 7fe0ead9d640 1 -- 172.21.15.116:0/1773759012 learned_addr learned my addr 172.21.15.116:0/1773759012 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:25:28.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.123+0000 7fe0e9d9b640 1 --2- 172.21.15.116:0/1773759012 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe0ec152ce0 0x7fe0ec1f1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:28.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.123+0000 7fe0ead9d640 1 -- 172.21.15.116:0/1773759012 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe0ec0c3ea0 msgr2=0x7fe0ec14dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:28.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.123+0000 7fe0ead9d640 1 --2- 172.21.15.116:0/1773759012 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe0ec0c3ea0 0x7fe0ec14dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:28.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.123+0000 7fe0ead9d640 1 -- 172.21.15.116:0/1773759012 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe0ec152ce0 msgr2=0x7fe0ec1f1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:28.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.123+0000 7fe0ead9d640 1 --2- 172.21.15.116:0/1773759012 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe0ec152ce0 0x7fe0ec1f1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:28.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.123+0000 7fe0ead9d640 1 -- 172.21.15.116:0/1773759012 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0ec06e150 con 0x7fe0ec159970 2024-04-25T00:25:28.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.123+0000 7fe0e9d9b640 1 --2- 172.21.15.116:0/1773759012 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe0ec152ce0 0x7fe0ec1f1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:25:28.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.123+0000 7fe0ead9d640 1 --2- 172.21.15.116:0/1773759012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0ec159970 0x7fe0ec1f1c70 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7fe0e400cd50 tx=0x7fe0e400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:28.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.123+0000 7fe0e2ffd640 1 -- 172.21.15.116:0/1773759012 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0e400f680 con 0x7fe0ec159970 2024-04-25T00:25:28.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.123+0000 7fe0e2ffd640 1 -- 172.21.15.116:0/1773759012 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0e400f840 con 0x7fe0ec159970 2024-04-25T00:25:28.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.123+0000 7fe0f0b33640 1 -- 172.21.15.116:0/1773759012 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0ec153630 con 0x7fe0ec159970 2024-04-25T00:25:28.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.123+0000 7fe0e2ffd640 1 -- 172.21.15.116:0/1773759012 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0e400fc30 con 0x7fe0ec159970 2024-04-25T00:25:28.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.123+0000 7fe0f0b33640 1 -- 172.21.15.116:0/1773759012 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0ec1f1210 con 0x7fe0ec159970 2024-04-25T00:25:28.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.127+0000 7fe0f0b33640 1 -- 172.21.15.116:0/1773759012 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0ec068a60 con 0x7fe0ec159970 2024-04-25T00:25:28.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.127+0000 7fe0e2ffd640 1 -- 172.21.15.116:0/1773759012 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe0e40213f0 con 0x7fe0ec159970 2024-04-25T00:25:28.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.127+0000 7fe0e2ffd640 1 --2- 172.21.15.116:0/1773759012 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe0cc0678c0 0x7fe0cc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:28.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.127+0000 7fe0e2ffd640 1 -- 172.21.15.116:0/1773759012 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 9934+0+0 (secure 0 0 0) 0x7fe0e4087d80 con 0x7fe0ec159970 2024-04-25T00:25:28.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.127+0000 7fe0ea59c640 1 --2- 172.21.15.116:0/1773759012 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe0cc0678c0 0x7fe0cc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:28.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.127+0000 7fe0ea59c640 1 --2- 172.21.15.116:0/1773759012 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe0cc0678c0 0x7fe0cc069d80 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc0ad000 tx=0x7fe0dc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:28.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.131+0000 7fe0e2ffd640 1 -- 172.21.15.116:0/1773759012 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fe0e4014030 con 0x7fe0ec159970 2024-04-25T00:25:28.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.327+0000 7fe0f0b33640 1 -- 172.21.15.116:0/1773759012 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe0ec1f13e0 con 0x7fe0ec159970 2024-04-25T00:25:28.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.331+0000 7fe0e2ffd640 1 -- 172.21.15.116:0/1773759012 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) ==== 95+0+5674 (secure 0 0 0) 0x7fe0e4057900 con 0x7fe0ec159970 2024-04-25T00:25:28.334 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:25:28.334 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":200,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":200,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:25:27.024395+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6413},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6413":{"gid":6413,"name":"d","rank":0,"incarnation":198,"state":"up:rejoin","state_seq":17,"addr":"172.21.15.174:6837/4181466530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":4181466530},{"type":"v1","addr":"172.21.15.174:6837","nonce":4181466530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":196,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:24:19.977863+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8343":{"gid":8343,"name":"a","rank":0,"incarnation":193,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6833/3633657268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3633657268},{"type":"v1","addr":"172.21.15.116:6833","nonce":3633657268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8343,"qdb_cluster":[8343]},"id":2}]} 2024-04-25T00:25:28.334 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 200 2024-04-25T00:25:28.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.331+0000 7fe0f0b33640 1 -- 172.21.15.116:0/1773759012 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe0cc0678c0 msgr2=0x7fe0cc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:28.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.331+0000 7fe0f0b33640 1 --2- 172.21.15.116:0/1773759012 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe0cc0678c0 0x7fe0cc069d80 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc0ad000 tx=0x7fe0dc008040 comp rx=0 tx=0).stop 2024-04-25T00:25:28.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.331+0000 7fe0f0b33640 1 -- 172.21.15.116:0/1773759012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0ec159970 msgr2=0x7fe0ec1f1c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:28.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.331+0000 7fe0f0b33640 1 --2- 172.21.15.116:0/1773759012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0ec159970 0x7fe0ec1f1c70 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7fe0e400cd50 tx=0x7fe0e400a650 comp rx=0 tx=0).stop 2024-04-25T00:25:28.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.331+0000 7fe0f0b33640 1 -- 172.21.15.116:0/1773759012 shutdown_connections 2024-04-25T00:25:28.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.331+0000 7fe0f0b33640 1 --2- 172.21.15.116:0/1773759012 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe0cc0678c0 0x7fe0cc069d80 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:28.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.331+0000 7fe0f0b33640 1 --2- 172.21.15.116:0/1773759012 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe0ec159970 0x7fe0ec1f1c70 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:28.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.331+0000 7fe0f0b33640 1 --2- 172.21.15.116:0/1773759012 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe0ec152ce0 0x7fe0ec1f1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:28.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.331+0000 7fe0f0b33640 1 --2- 172.21.15.116:0/1773759012 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe0ec0c3ea0 0x7fe0ec14dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:28.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.331+0000 7fe0f0b33640 1 -- 172.21.15.116:0/1773759012 >> 172.21.15.116:0/1773759012 conn(0x7fe0ec0c4c10 msgr2=0x7fe0ec0c5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:28.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.331+0000 7fe0f0b33640 1 -- 172.21.15.116:0/1773759012 shutdown_connections 2024-04-25T00:25:28.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.331+0000 7fe0f0b33640 1 -- 172.21.15.116:0/1773759012 wait complete. 2024-04-25T00:25:28.354 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 200, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:04.678737+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 200, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6413": { "addr": "172.21.15.174:6837/4181466530", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6836", "nonce": 4181466530, "type": "v2" }, { "addr": "172.21.15.174:6837", "nonce": 4181466530, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6413, "incarnation": 198, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:rejoin", "state_seq": 17 } }, "last_failure": 0, "last_failure_osd_epoch": 66, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T00:25:27.024395+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6413 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:18.825887+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 196, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_8343": { "addr": "172.21.15.116:6833/3633657268", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6832", "nonce": 3633657268, "type": "v2" }, { "addr": "172.21.15.116:6833", "nonce": 3633657268, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8343, "incarnation": 193, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 12 } }, "last_failure": 0, "last_failure_osd_epoch": 65, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-04-25T00:24:19.977863+0000", "qdb_cluster": [ 8343 ], "qdb_leader": 8343, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8343 } } } ], "standbys": [ { "addr": "172.21.15.174:6835/4112145782", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6834", "nonce": 4112145782, "type": "v2" }, { "addr": "172.21.15.174:6835", "nonce": 4112145782, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 112, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5582, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-25T00:25:28.530 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:25:28.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.655+0000 7ffbdd17e640 1 Processor -- start 2024-04-25T00:25:28.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.655+0000 7ffbdd17e640 1 -- start start 2024-04-25T00:25:28.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.659+0000 7ffbdd17e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffbd80a94a0 0x7ffbd81538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:28.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.659+0000 7ffbdd17e640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ffbd80a99f0 0x7ffbd8153e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:28.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.659+0000 7ffbdd17e640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffbd80c3910 con 0x7ffbd8150d20 2024-04-25T00:25:28.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.659+0000 7ffbdd17e640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7ffbd80c16e0 con 0x7ffbd80a99f0 2024-04-25T00:25:28.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.659+0000 7ffbdd17e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ffbd80c1400 con 0x7ffbd80a94a0 2024-04-25T00:25:28.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.659+0000 7ffbd759e640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ffbd80a99f0 0x7ffbd8153e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:28.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.659+0000 7ffbce59c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffbd80a94a0 0x7ffbd81538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:28.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.659+0000 7ffbd759e640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ffbd80a99f0 0x7ffbd8153e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:40638/0 (socket says 172.21.15.116:40638) 2024-04-25T00:25:28.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.659+0000 7ffbd759e640 1 -- 172.21.15.116:0/2836003831 learned_addr learned my addr 172.21.15.116:0/2836003831 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:25:28.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.659+0000 7ffbd759e640 1 -- 172.21.15.116:0/2836003831 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffbd80a94a0 msgr2=0x7ffbd81538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:28.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.659+0000 7ffbd759e640 1 --2- 172.21.15.116:0/2836003831 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffbd80a94a0 0x7ffbd81538e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:28.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.659+0000 7ffbd759e640 1 -- 172.21.15.116:0/2836003831 >> v1:172.21.15.174:6789/0 conn(0x7ffbd8150d20 legacy=0x7ffbd80c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:28.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.659+0000 7ffbd659c640 1 -- 172.21.15.116:0/2836003831 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4293077833 0 0) 0x7ffbd80c3910 con 0x7ffbd8150d20 2024-04-25T00:25:28.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.659+0000 7ffbd759e640 1 -- 172.21.15.116:0/2836003831 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbd806e660 con 0x7ffbd80a99f0 2024-04-25T00:25:28.664 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.659+0000 7ffbce59c640 1 --2- 172.21.15.116:0/2836003831 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffbd80a94a0 0x7ffbd81538e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:25:28.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.663+0000 7ffbd759e640 1 --2- 172.21.15.116:0/2836003831 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbd80a99f0 0x7ffbd8153e20 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7ffbd0009e50 tx=0x7ffbd001ca30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c12fefbbdf8b9559 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:28.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.663+0000 7ffbd659c640 1 -- 172.21.15.116:0/2836003831 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbd00a5700 con 0x7ffbd80a99f0 2024-04-25T00:25:28.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.663+0000 7ffbd659c640 1 -- 172.21.15.116:0/2836003831 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffbd00a58a0 con 0x7ffbd80a99f0 2024-04-25T00:25:28.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.663+0000 7ffbd659c640 1 -- 172.21.15.116:0/2836003831 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbd00a5c70 con 0x7ffbd80a99f0 2024-04-25T00:25:28.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.663+0000 7ffbdd17e640 1 -- 172.21.15.116:0/2836003831 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbd80a99f0 msgr2=0x7ffbd8153e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:28.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.663+0000 7ffbdd17e640 1 --2- 172.21.15.116:0/2836003831 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbd80a99f0 0x7ffbd8153e20 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7ffbd0009e50 tx=0x7ffbd001ca30 comp rx=0 tx=0).stop 2024-04-25T00:25:28.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.663+0000 7ffbdd17e640 1 -- 172.21.15.116:0/2836003831 shutdown_connections 2024-04-25T00:25:28.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.663+0000 7ffbdd17e640 1 --2- 172.21.15.116:0/2836003831 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbd80a99f0 0x7ffbd8153e20 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:28.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.663+0000 7ffbdd17e640 1 --2- 172.21.15.116:0/2836003831 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffbd80a94a0 0x7ffbd81538e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:28.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.663+0000 7ffbdd17e640 1 -- 172.21.15.116:0/2836003831 >> 172.21.15.116:0/2836003831 conn(0x7ffbd80c3b00 msgr2=0x7ffbd80c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:28.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.663+0000 7ffbdd17e640 1 -- 172.21.15.116:0/2836003831 shutdown_connections 2024-04-25T00:25:28.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.663+0000 7ffbdd17e640 1 -- 172.21.15.116:0/2836003831 wait complete. 2024-04-25T00:25:28.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.663+0000 7ffbdd17e640 1 Processor -- start 2024-04-25T00:25:28.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.663+0000 7ffbdd17e640 1 -- start start 2024-04-25T00:25:28.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.667+0000 7ffbdd17e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbd80a94a0 0x7ffbd815c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:28.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.667+0000 7ffbdd17e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbd80a99f0 0x7ffbd813d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:28.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.667+0000 7ffbdd17e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffbd8150d20 0x7ffbd813d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:28.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.667+0000 7ffbdd17e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ffbd81544e0 con 0x7ffbd80a99f0 2024-04-25T00:25:28.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.667+0000 7ffbdd17e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ffbd8154360 con 0x7ffbd80a94a0 2024-04-25T00:25:28.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.667+0000 7ffbdd17e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ffbd8160730 con 0x7ffbd8150d20 2024-04-25T00:25:28.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.667+0000 7ffbce59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbd80a99f0 0x7ffbd813d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:28.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.667+0000 7ffbce59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbd80a99f0 0x7ffbd813d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44334/0 (socket says 172.21.15.116:44334) 2024-04-25T00:25:28.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.667+0000 7ffbce59c640 1 -- 172.21.15.116:0/2008910034 learned_addr learned my addr 172.21.15.116:0/2008910034 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:25:28.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.667+0000 7ffbd6d9d640 1 --2- 172.21.15.116:0/2008910034 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbd80a94a0 0x7ffbd815c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:28.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.667+0000 7ffbd759e640 1 --2- 172.21.15.116:0/2008910034 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffbd8150d20 0x7ffbd813d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:28.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.667+0000 7ffbce59c640 1 -- 172.21.15.116:0/2008910034 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffbd8150d20 msgr2=0x7ffbd813d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:28.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.667+0000 7ffbce59c640 1 --2- 172.21.15.116:0/2008910034 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffbd8150d20 0x7ffbd813d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:28.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.667+0000 7ffbce59c640 1 -- 172.21.15.116:0/2008910034 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbd80a94a0 msgr2=0x7ffbd815c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:28.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.667+0000 7ffbce59c640 1 --2- 172.21.15.116:0/2008910034 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbd80a94a0 0x7ffbd815c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:28.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.667+0000 7ffbce59c640 1 -- 172.21.15.116:0/2008910034 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffbd80c5c90 con 0x7ffbd80a99f0 2024-04-25T00:25:28.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.667+0000 7ffbce59c640 1 --2- 172.21.15.116:0/2008910034 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbd80a99f0 0x7ffbd813d0a0 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7ffbc000e9b0 tx=0x7ffbc000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:28.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.667+0000 7ffbcffff640 1 -- 172.21.15.116:0/2008910034 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc0017070 con 0x7ffbd80a99f0 2024-04-25T00:25:28.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.667+0000 7ffbcffff640 1 -- 172.21.15.116:0/2008910034 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffbc000ca00 con 0x7ffbd80a99f0 2024-04-25T00:25:28.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.667+0000 7ffbdd17e640 1 -- 172.21.15.116:0/2008910034 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffbd80c5f70 con 0x7ffbd80a99f0 2024-04-25T00:25:28.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.667+0000 7ffbcffff640 1 -- 172.21.15.116:0/2008910034 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffbc000cdd0 con 0x7ffbd80a99f0 2024-04-25T00:25:28.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.667+0000 7ffbdd17e640 1 -- 172.21.15.116:0/2008910034 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffbd80c6530 con 0x7ffbd80a99f0 2024-04-25T00:25:28.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.667+0000 7ffbdd17e640 1 -- 172.21.15.116:0/2008910034 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffbd8068350 con 0x7ffbd80a99f0 2024-04-25T00:25:28.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.667+0000 7ffbcffff640 1 -- 172.21.15.116:0/2008910034 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ffbc001a070 con 0x7ffbd80a99f0 2024-04-25T00:25:28.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.667+0000 7ffbcffff640 1 --2- 172.21.15.116:0/2008910034 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ffbb80678c0 0x7ffbb8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:28.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.667+0000 7ffbcffff640 1 -- 172.21.15.116:0/2008910034 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 9934+0+0 (secure 0 0 0) 0x7ffbc0087030 con 0x7ffbd80a99f0 2024-04-25T00:25:28.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.667+0000 7ffbd6d9d640 1 --2- 172.21.15.116:0/2008910034 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ffbb80678c0 0x7ffbb8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:28.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.671+0000 7ffbd6d9d640 1 --2- 172.21.15.116:0/2008910034 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ffbb80678c0 0x7ffbb8069d80 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7ffbc8000f70 tx=0x7ffbc8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:28.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.671+0000 7ffbcffff640 1 -- 172.21.15.116:0/2008910034 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ffbc0056aa0 con 0x7ffbd80a99f0 2024-04-25T00:25:28.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.867+0000 7ffbdd17e640 1 -- 172.21.15.116:0/2008910034 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffbd813e140 con 0x7ffbd80a99f0 2024-04-25T00:25:28.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.867+0000 7ffbcffff640 1 -- 172.21.15.116:0/2008910034 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) ==== 95+0+5674 (secure 0 0 0) 0x7ffbc00563a0 con 0x7ffbd80a99f0 2024-04-25T00:25:28.873 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:25:28.873 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":200,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":200,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:25:27.024395+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6413},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6413":{"gid":6413,"name":"d","rank":0,"incarnation":198,"state":"up:rejoin","state_seq":17,"addr":"172.21.15.174:6837/4181466530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":4181466530},{"type":"v1","addr":"172.21.15.174:6837","nonce":4181466530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":196,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:24:19.977863+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8343":{"gid":8343,"name":"a","rank":0,"incarnation":193,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6833/3633657268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3633657268},{"type":"v1","addr":"172.21.15.116:6833","nonce":3633657268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8343,"qdb_cluster":[8343]},"id":2}]} 2024-04-25T00:25:28.874 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 200 2024-04-25T00:25:28.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.871+0000 7ffbdd17e640 1 -- 172.21.15.116:0/2008910034 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ffbb80678c0 msgr2=0x7ffbb8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:28.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.871+0000 7ffbdd17e640 1 --2- 172.21.15.116:0/2008910034 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ffbb80678c0 0x7ffbb8069d80 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7ffbc8000f70 tx=0x7ffbc8009040 comp rx=0 tx=0).stop 2024-04-25T00:25:28.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.871+0000 7ffbdd17e640 1 -- 172.21.15.116:0/2008910034 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbd80a99f0 msgr2=0x7ffbd813d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:28.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.871+0000 7ffbdd17e640 1 --2- 172.21.15.116:0/2008910034 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbd80a99f0 0x7ffbd813d0a0 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7ffbc000e9b0 tx=0x7ffbc000ee70 comp rx=0 tx=0).stop 2024-04-25T00:25:28.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.871+0000 7ffbdd17e640 1 -- 172.21.15.116:0/2008910034 shutdown_connections 2024-04-25T00:25:28.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.871+0000 7ffbdd17e640 1 --2- 172.21.15.116:0/2008910034 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ffbb80678c0 0x7ffbb8069d80 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:28.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.871+0000 7ffbdd17e640 1 --2- 172.21.15.116:0/2008910034 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffbd8150d20 0x7ffbd813d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:28.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.871+0000 7ffbdd17e640 1 --2- 172.21.15.116:0/2008910034 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffbd80a99f0 0x7ffbd813d0a0 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:28.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.871+0000 7ffbdd17e640 1 --2- 172.21.15.116:0/2008910034 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffbd80a94a0 0x7ffbd815c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:28.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.871+0000 7ffbdd17e640 1 -- 172.21.15.116:0/2008910034 >> 172.21.15.116:0/2008910034 conn(0x7ffbd80c3b00 msgr2=0x7ffbd80accc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:28.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.871+0000 7ffbdd17e640 1 -- 172.21.15.116:0/2008910034 shutdown_connections 2024-04-25T00:25:28.876 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:28.871+0000 7ffbdd17e640 1 -- 172.21.15.116:0/2008910034 wait complete. 2024-04-25T00:25:28.892 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:25:30.356 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:25:30.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:30.591+0000 7f20828ef640 1 Processor -- start 2024-04-25T00:25:30.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:30.591+0000 7f20828ef640 1 -- start start 2024-04-25T00:25:30.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:30.591+0000 7f20828ef640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f207c1571b0 0x7f207c0c2b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:30.591+0000 7f20828ef640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f207c15da60 0x7f207c0716f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:30.591+0000 7f20828ef640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f207c157590 con 0x7f207c0c2190 2024-04-25T00:25:30.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:30.591+0000 7f20828ef640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f207c0c3180 con 0x7f207c1571b0 2024-04-25T00:25:30.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:30.591+0000 7f20828ef640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f207c0c3460 con 0x7f207c15da60 2024-04-25T00:25:30.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:30.591+0000 7f20818ed640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f207c1571b0 0x7f207c0c2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:30.591+0000 7f20818ed640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f207c1571b0 0x7f207c0c2b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:40658/0 (socket says 172.21.15.116:40658) 2024-04-25T00:25:30.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:30.591+0000 7f20818ed640 1 -- 172.21.15.116:0/2461752588 learned_addr learned my addr 172.21.15.116:0/2461752588 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:25:30.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:30.591+0000 7f20820ee640 1 --2- 172.21.15.116:0/2461752588 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f207c15da60 0x7f207c0716f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:30.591+0000 7f20818ed640 1 -- 172.21.15.116:0/2461752588 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f207c15da60 msgr2=0x7f207c0716f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:30.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:30.591+0000 7f20818ed640 1 --2- 172.21.15.116:0/2461752588 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f207c15da60 0x7f207c0716f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:30.591+0000 7f20818ed640 1 -- 172.21.15.116:0/2461752588 >> v1:172.21.15.174:6789/0 conn(0x7f207c0c2190 legacy=0x7f207c06dc00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T00:25:30.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:30.591+0000 7f20818ed640 1 -- 172.21.15.116:0/2461752588 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f207c15d260 con 0x7f207c1571b0 2024-04-25T00:25:30.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:30.595+0000 7f20818ed640 1 --2- 172.21.15.116:0/2461752588 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f207c1571b0 0x7f207c0c2b10 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f2074009e50 tx=0x7f207401c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=587333f284e37cec server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:30.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:30.595+0000 7f20808eb640 1 -- 172.21.15.116:0/2461752588 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20740a5640 con 0x7f207c1571b0 2024-04-25T00:25:30.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:30.595+0000 7f20808eb640 1 -- 172.21.15.116:0/2461752588 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20740a57e0 con 0x7f207c1571b0 2024-04-25T00:25:30.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:30.595+0000 7f20808eb640 1 -- 172.21.15.116:0/2461752588 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20740a5bf0 con 0x7f207c1571b0 2024-04-25T00:25:30.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:30.599+0000 7f20828ef640 1 -- 172.21.15.116:0/2461752588 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f207c1571b0 msgr2=0x7f207c0c2b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:30.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:30.599+0000 7f20828ef640 1 --2- 172.21.15.116:0/2461752588 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f207c1571b0 0x7f207c0c2b10 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f2074009e50 tx=0x7f207401c8f0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:30.603+0000 7f20828ef640 1 -- 172.21.15.116:0/2461752588 shutdown_connections 2024-04-25T00:25:30.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:30.603+0000 7f20828ef640 1 --2- 172.21.15.116:0/2461752588 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f207c15da60 0x7f207c0716f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:30.603+0000 7f20828ef640 1 --2- 172.21.15.116:0/2461752588 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f207c1571b0 0x7f207c0c2b10 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:30.603+0000 7f20828ef640 1 -- 172.21.15.116:0/2461752588 >> 172.21.15.116:0/2461752588 conn(0x7f207c0c6db0 msgr2=0x7f207c0c71c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:30.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:30.603+0000 7f20828ef640 1 -- 172.21.15.116:0/2461752588 shutdown_connections 2024-04-25T00:25:30.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:30.603+0000 7f20828ef640 1 -- 172.21.15.116:0/2461752588 wait complete. 2024-04-25T00:25:30.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:30.607+0000 7f20828ef640 1 Processor -- start 2024-04-25T00:25:30.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:30.607+0000 7f20828ef640 1 -- start start 2024-04-25T00:25:30.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:30.607+0000 7f20828ef640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f207c0c2190 0x7f207c0c53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:30.607+0000 7f20828ef640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f207c1571b0 0x7f207c0aad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:30.607+0000 7f20828ef640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f207c15da60 0x7f207c0ab240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:30.607+0000 7f20828ef640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f207c164de0 con 0x7f207c0c2190 2024-04-25T00:25:30.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:30.607+0000 7f20828ef640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f207c164c60 con 0x7f207c15da60 2024-04-25T00:25:30.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:30.607+0000 7f20828ef640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f207c164f60 con 0x7f207c1571b0 2024-04-25T00:25:30.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:30.607+0000 7f20810ec640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f207c1571b0 0x7f207c0aad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:30.607+0000 7f20810ec640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f207c1571b0 0x7f207c0aad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:45172/0 (socket says 172.21.15.116:45172) 2024-04-25T00:25:30.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:30.607+0000 7f20810ec640 1 -- 172.21.15.116:0/184934977 learned_addr learned my addr 172.21.15.116:0/184934977 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:25:30.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:30.607+0000 7f20810ec640 1 -- 172.21.15.116:0/184934977 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f207c15da60 msgr2=0x7f207c0ab240 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T00:25:30.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:30.607+0000 7f20810ec640 1 --2- 172.21.15.116:0/184934977 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f207c15da60 0x7f207c0ab240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:30.607+0000 7f20810ec640 1 -- 172.21.15.116:0/184934977 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f207c0c2190 msgr2=0x7f207c0c53f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T00:25:30.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:30.607+0000 7f20810ec640 1 --2- 172.21.15.116:0/184934977 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f207c0c2190 0x7f207c0c53f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:30.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:30.607+0000 7f20810ec640 1 -- 172.21.15.116:0/184934977 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f207c08bb70 con 0x7f207c1571b0 2024-04-25T00:25:30.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:30.607+0000 7f20810ec640 1 --2- 172.21.15.116:0/184934977 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f207c1571b0 0x7f207c0aad00 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f206c00b4a0 tx=0x7f206c00b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:30.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:30.607+0000 7f20727fc640 1 -- 172.21.15.116:0/184934977 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f206c014070 con 0x7f207c1571b0 2024-04-25T00:25:30.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:30.611+0000 7f20828ef640 1 -- 172.21.15.116:0/184934977 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f207c08b8d0 con 0x7f207c1571b0 2024-04-25T00:25:30.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:30.611+0000 7f20828ef640 1 -- 172.21.15.116:0/184934977 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f207c0aa500 con 0x7f207c1571b0 2024-04-25T00:25:30.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:30.611+0000 7f20727fc640 1 -- 172.21.15.116:0/184934977 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f206c010610 con 0x7f207c1571b0 2024-04-25T00:25:30.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:30.611+0000 7f20727fc640 1 -- 172.21.15.116:0/184934977 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f206c0109c0 con 0x7f207c1571b0 2024-04-25T00:25:30.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:30.611+0000 7f20727fc640 1 -- 172.21.15.116:0/184934977 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f206c017070 con 0x7f207c1571b0 2024-04-25T00:25:30.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:30.615+0000 7f20727fc640 1 --2- 172.21.15.116:0/184934977 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2060067990 0x7f2060069e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:30.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:30.615+0000 7f20727fc640 1 -- 172.21.15.116:0/184934977 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(66..66 src has 1..66) ==== 9934+0+0 (secure 0 0 0) 0x7f206c086670 con 0x7f207c1571b0 2024-04-25T00:25:30.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:30.615+0000 7f20818ed640 1 --2- 172.21.15.116:0/184934977 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2060067990 0x7f2060069e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:30.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:30.615+0000 7f20828ef640 1 -- 172.21.15.116:0/184934977 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20640052a0 con 0x7f207c1571b0 2024-04-25T00:25:30.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:30.615+0000 7f20818ed640 1 --2- 172.21.15.116:0/184934977 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2060067990 0x7f2060069e50 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f20740ae000 tx=0x7f20740a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:30.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:30.619+0000 7f20727fc640 1 -- 172.21.15.116:0/184934977 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f206c056190 con 0x7f207c1571b0 2024-04-25T00:25:30.893 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:25:31.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.279+0000 7f20828ef640 1 -- 172.21.15.116:0/184934977 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2064005590 con 0x7f207c1571b0 2024-04-25T00:25:31.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.279+0000 7f20727fc640 1 -- 172.21.15.116:0/184934977 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 201 v201) ==== 95+0+5681 (secure 0 0 0) 0x7f206c055a90 con 0x7f207c1571b0 2024-04-25T00:25:31.283 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:25:31.283 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":201,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":201,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:25:29.028925+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6413},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6413":{"gid":6413,"name":"d","rank":0,"incarnation":198,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/4181466530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":4181466530},{"type":"v1","addr":"172.21.15.174:6837","nonce":4181466530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6413,"qdb_cluster":[6413]},"id":1},{"mdsmap":{"epoch":196,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:24:19.977863+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8343":{"gid":8343,"name":"a","rank":0,"incarnation":193,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6833/3633657268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3633657268},{"type":"v1","addr":"172.21.15.116:6833","nonce":3633657268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8343,"qdb_cluster":[8343]},"id":2}]} 2024-04-25T00:25:31.283 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 201 2024-04-25T00:25:31.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.287+0000 7f20828ef640 1 -- 172.21.15.116:0/184934977 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2060067990 msgr2=0x7f2060069e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:31.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.287+0000 7f20828ef640 1 --2- 172.21.15.116:0/184934977 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2060067990 0x7f2060069e50 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f20740ae000 tx=0x7f20740a7070 comp rx=0 tx=0).stop 2024-04-25T00:25:31.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.287+0000 7f20828ef640 1 -- 172.21.15.116:0/184934977 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f207c1571b0 msgr2=0x7f207c0aad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:31.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.287+0000 7f20828ef640 1 --2- 172.21.15.116:0/184934977 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f207c1571b0 0x7f207c0aad00 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f206c00b4a0 tx=0x7f206c00b960 comp rx=0 tx=0).stop 2024-04-25T00:25:31.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.287+0000 7f20828ef640 1 -- 172.21.15.116:0/184934977 shutdown_connections 2024-04-25T00:25:31.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.287+0000 7f20828ef640 1 --2- 172.21.15.116:0/184934977 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2060067990 0x7f2060069e50 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:31.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.287+0000 7f20828ef640 1 --2- 172.21.15.116:0/184934977 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f207c15da60 0x7f207c0ab240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:31.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.287+0000 7f20828ef640 1 --2- 172.21.15.116:0/184934977 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f207c1571b0 0x7f207c0aad00 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:31.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.287+0000 7f20828ef640 1 --2- 172.21.15.116:0/184934977 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f207c0c2190 0x7f207c0c53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:31.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.287+0000 7f20828ef640 1 -- 172.21.15.116:0/184934977 >> 172.21.15.116:0/184934977 conn(0x7f207c0c6db0 msgr2=0x7f207c153aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:31.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.287+0000 7f20828ef640 1 -- 172.21.15.116:0/184934977 shutdown_connections 2024-04-25T00:25:31.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.287+0000 7f20828ef640 1 -- 172.21.15.116:0/184934977 wait complete. 2024-04-25T00:25:31.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.291+0000 7f5338ca1640 1 Processor -- start 2024-04-25T00:25:31.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.291+0000 7f5338ca1640 1 -- start start 2024-04-25T00:25:31.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.291+0000 7f5338ca1640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5334157300 0x7f5334074480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:31.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.291+0000 7f5338ca1640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f53340aa010 0x7f5334159bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:31.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.291+0000 7f5338ca1640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53340c30f0 con 0x7f53340749c0 2024-04-25T00:25:31.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.291+0000 7f5338ca1640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f53340c2ba0 con 0x7f5334157300 2024-04-25T00:25:31.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.291+0000 7f5338ca1640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f533406ca90 con 0x7f53340aa010 2024-04-25T00:25:31.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.291+0000 7f533259c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5334157300 0x7f5334074480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:31.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.291+0000 7f533259c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5334157300 0x7f5334074480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44336/0 (socket says 172.21.15.116:44336) 2024-04-25T00:25:31.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.291+0000 7f533259c640 1 -- 172.21.15.116:0/1639602798 learned_addr learned my addr 172.21.15.116:0/1639602798 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:25:31.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.291+0000 7f533259c640 1 -- 172.21.15.116:0/1639602798 >> v2:172.21.15.174:3300/0 conn(0x7f53340aa010 msgr2=0x7f5334159bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:31.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.291+0000 7f533259c640 1 --2- 172.21.15.116:0/1639602798 >> v2:172.21.15.174:3300/0 conn(0x7f53340aa010 0x7f5334159bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:31.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.291+0000 7f533259c640 1 -- 172.21.15.116:0/1639602798 >> v1:172.21.15.174:6789/0 conn(0x7f53340749c0 legacy=0x7f533415a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:31.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.291+0000 7f533259c640 1 -- 172.21.15.116:0/1639602798 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53340c2dd0 con 0x7f5334157300 2024-04-25T00:25:31.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.295+0000 7f533259c640 1 --2- 172.21.15.116:0/1639602798 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5334157300 0x7f5334074480 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f533406cce0 tx=0x7f532801c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68471885061ad8a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:31.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.295+0000 7f533159a640 1 -- 172.21.15.116:0/1639602798 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53280b0070 con 0x7f5334157300 2024-04-25T00:25:31.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.295+0000 7f533159a640 1 -- 172.21.15.116:0/1639602798 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53280ab600 con 0x7f5334157300 2024-04-25T00:25:31.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.295+0000 7f533159a640 1 -- 172.21.15.116:0/1639602798 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53280ab9d0 con 0x7f5334157300 2024-04-25T00:25:31.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.295+0000 7f5338ca1640 1 -- 172.21.15.116:0/1639602798 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5334157300 msgr2=0x7f5334074480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:31.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.295+0000 7f5338ca1640 1 --2- 172.21.15.116:0/1639602798 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5334157300 0x7f5334074480 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f533406cce0 tx=0x7f532801c450 comp rx=0 tx=0).stop 2024-04-25T00:25:31.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.299+0000 7f5338ca1640 1 -- 172.21.15.116:0/1639602798 shutdown_connections 2024-04-25T00:25:31.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.299+0000 7f5338ca1640 1 --2- 172.21.15.116:0/1639602798 >> v2:172.21.15.174:3300/0 conn(0x7f53340aa010 0x7f5334159bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:31.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.299+0000 7f5338ca1640 1 --2- 172.21.15.116:0/1639602798 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5334157300 0x7f5334074480 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:31.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.299+0000 7f5338ca1640 1 -- 172.21.15.116:0/1639602798 >> 172.21.15.116:0/1639602798 conn(0x7f53340c4c10 msgr2=0x7f53340c5020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:31.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.299+0000 7f5338ca1640 1 -- 172.21.15.116:0/1639602798 shutdown_connections 2024-04-25T00:25:31.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.299+0000 7f5338ca1640 1 -- 172.21.15.116:0/1639602798 wait complete. 2024-04-25T00:25:31.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.299+0000 7f5338ca1640 1 Processor -- start 2024-04-25T00:25:31.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.307+0000 7f5338ca1640 1 -- start start 2024-04-25T00:25:31.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.311+0000 7f5338ca1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f53340749c0 0x7f53340aca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:31.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.311+0000 7f5338ca1640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f53340aa010 0x7f5334153f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:31.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.311+0000 7f5338ca1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53341509d0 0x7f533414d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:31.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.311+0000 7f5338ca1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5334166d50 con 0x7f53341509d0 2024-04-25T00:25:31.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.311+0000 7f5338ca1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f53340aaef0 con 0x7f53340749c0 2024-04-25T00:25:31.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.311+0000 7f5338ca1640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f5334166ed0 con 0x7f53340aa010 2024-04-25T00:25:31.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.311+0000 7f5332d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53341509d0 0x7f533414d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:31.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.311+0000 7f5332d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53341509d0 0x7f533414d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44338/0 (socket says 172.21.15.116:44338) 2024-04-25T00:25:31.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.311+0000 7f5332d9d640 1 -- 172.21.15.116:0/307110428 learned_addr learned my addr 172.21.15.116:0/307110428 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:25:31.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.311+0000 7f533259c640 1 --2- 172.21.15.116:0/307110428 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f53340749c0 0x7f53340aca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:31.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.311+0000 7f5331d9b640 1 --2- 172.21.15.116:0/307110428 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f53340aa010 0x7f5334153f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:31.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.311+0000 7f5332d9d640 1 -- 172.21.15.116:0/307110428 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f53340aa010 msgr2=0x7f5334153f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:31.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.311+0000 7f5332d9d640 1 --2- 172.21.15.116:0/307110428 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f53340aa010 0x7f5334153f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:31.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.311+0000 7f5332d9d640 1 -- 172.21.15.116:0/307110428 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f53340749c0 msgr2=0x7f53340aca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:31.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.311+0000 7f5332d9d640 1 --2- 172.21.15.116:0/307110428 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f53340749c0 0x7f53340aca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:31.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.311+0000 7f5332d9d640 1 -- 172.21.15.116:0/307110428 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53340ad480 con 0x7f53341509d0 2024-04-25T00:25:31.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.311+0000 7f5332d9d640 1 --2- 172.21.15.116:0/307110428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53341509d0 0x7f533414d220 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f532400c440 tx=0x7f532400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:31.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.311+0000 7f5322ffd640 1 -- 172.21.15.116:0/307110428 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5324017070 con 0x7f53341509d0 2024-04-25T00:25:31.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.311+0000 7f5338ca1640 1 -- 172.21.15.116:0/307110428 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5334153020 con 0x7f53341509d0 2024-04-25T00:25:31.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.311+0000 7f5338ca1640 1 -- 172.21.15.116:0/307110428 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53340acf50 con 0x7f53341509d0 2024-04-25T00:25:31.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.311+0000 7f5322ffd640 1 -- 172.21.15.116:0/307110428 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5324003600 con 0x7f53341509d0 2024-04-25T00:25:31.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.311+0000 7f5322ffd640 1 -- 172.21.15.116:0/307110428 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53240039f0 con 0x7f53341509d0 2024-04-25T00:25:31.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.315+0000 7f5338ca1640 1 -- 172.21.15.116:0/307110428 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53180052a0 con 0x7f53341509d0 2024-04-25T00:25:31.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.343+0000 7f5322ffd640 1 -- 172.21.15.116:0/307110428 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f532401a070 con 0x7f53341509d0 2024-04-25T00:25:31.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.343+0000 7f5322ffd640 1 --2- 172.21.15.116:0/307110428 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5314067990 0x7f5314069e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:31.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.343+0000 7f5322ffd640 1 -- 172.21.15.116:0/307110428 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 9934+0+0 (secure 0 0 0) 0x7f5324086010 con 0x7f53341509d0 2024-04-25T00:25:31.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.343+0000 7f5322ffd640 1 -- 172.21.15.116:0/307110428 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f53240554e0 con 0x7f53341509d0 2024-04-25T00:25:31.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.343+0000 7f533259c640 1 --2- 172.21.15.116:0/307110428 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5314067990 0x7f5314069e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:31.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:31.343+0000 7f533259c640 1 --2- 172.21.15.116:0/307110428 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5314067990 0x7f5314069e50 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f53280ad000 tx=0x7f53280a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:31.379 INFO:tasks.mds_thrash.fs.[a]:mds.d has gained rank=0, replacing gid=8244 2024-04-25T00:25:31.379 INFO:tasks.mds_thrash.fs.[a]:waiting for 1 secs before reviving mds.c 2024-04-25T00:25:32.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.019+0000 7f5338ca1640 1 -- 172.21.15.116:0/307110428 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5318005590 con 0x7f53341509d0 2024-04-25T00:25:32.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.019+0000 7f5322ffd640 1 -- 172.21.15.116:0/307110428 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 201 v201) ==== 95+0+5681 (secure 0 0 0) 0x7f53240597e0 con 0x7f53341509d0 2024-04-25T00:25:32.029 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:25:32.029 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":201,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":201,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:25:29.028925+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6413},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6413":{"gid":6413,"name":"d","rank":0,"incarnation":198,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/4181466530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":4181466530},{"type":"v1","addr":"172.21.15.174:6837","nonce":4181466530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6413,"qdb_cluster":[6413]},"id":1},{"mdsmap":{"epoch":196,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:24:19.977863+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8343":{"gid":8343,"name":"a","rank":0,"incarnation":193,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6833/3633657268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3633657268},{"type":"v1","addr":"172.21.15.116:6833","nonce":3633657268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8343,"qdb_cluster":[8343]},"id":2}]} 2024-04-25T00:25:32.030 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 201 2024-04-25T00:25:32.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.027+0000 7f5338ca1640 1 -- 172.21.15.116:0/307110428 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5314067990 msgr2=0x7f5314069e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:32.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.027+0000 7f5338ca1640 1 --2- 172.21.15.116:0/307110428 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5314067990 0x7f5314069e50 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f53280ad000 tx=0x7f53280a6000 comp rx=0 tx=0).stop 2024-04-25T00:25:32.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.027+0000 7f5338ca1640 1 -- 172.21.15.116:0/307110428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53341509d0 msgr2=0x7f533414d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:32.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.027+0000 7f5338ca1640 1 --2- 172.21.15.116:0/307110428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53341509d0 0x7f533414d220 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f532400c440 tx=0x7f532400c900 comp rx=0 tx=0).stop 2024-04-25T00:25:32.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.027+0000 7f5338ca1640 1 -- 172.21.15.116:0/307110428 shutdown_connections 2024-04-25T00:25:32.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.027+0000 7f5338ca1640 1 --2- 172.21.15.116:0/307110428 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5314067990 0x7f5314069e50 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.027+0000 7f5338ca1640 1 --2- 172.21.15.116:0/307110428 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f53341509d0 0x7f533414d220 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.027+0000 7f5338ca1640 1 --2- 172.21.15.116:0/307110428 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f53340aa010 0x7f5334153f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.027+0000 7f5338ca1640 1 --2- 172.21.15.116:0/307110428 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f53340749c0 0x7f53340aca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.027+0000 7f5338ca1640 1 -- 172.21.15.116:0/307110428 >> 172.21.15.116:0/307110428 conn(0x7f53340c4c10 msgr2=0x7f533406f350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:32.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.027+0000 7f5338ca1640 1 -- 172.21.15.116:0/307110428 shutdown_connections 2024-04-25T00:25:32.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.027+0000 7f5338ca1640 1 -- 172.21.15.116:0/307110428 wait complete. 2024-04-25T00:25:32.063 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:25:32.379 INFO:tasks.mds_thrash.fs.[a]:reviving mds.c 2024-04-25T00:25:32.379 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-25T00:25:32.380 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-25T00:25:32.383 INFO:tasks.ceph.mds.c:Started 2024-04-25T00:25:32.383 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:25:32.545 INFO:tasks.ceph.mds.c.smithi116.stdout:starting mds.c at 2024-04-25T00:25:32.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.755+0000 7f11ec959640 1 Processor -- start 2024-04-25T00:25:32.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.755+0000 7f11ec959640 1 -- start start 2024-04-25T00:25:32.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.755+0000 7f11ec959640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f11e80c4460 0x7f11e814ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:32.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.755+0000 7f11ec959640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11e81516d0 0x7f11e80c2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:32.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.755+0000 7f11ec959640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11e8070e60 con 0x7f11e8072dc0 2024-04-25T00:25:32.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.755+0000 7f11ec959640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f11e8070c10 con 0x7f11e80c4460 2024-04-25T00:25:32.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.755+0000 7f11ec959640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f11e814f130 con 0x7f11e81516d0 2024-04-25T00:25:32.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.755+0000 7f11e759e640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f11e8072dc0 0x7f11e806ca00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54122/0 (socket says 172.21.15.116:54122) 2024-04-25T00:25:32.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.755+0000 7f11e759e640 1 -- 172.21.15.116:0/78814840 learned_addr learned my addr 172.21.15.116:0/78814840 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:25:32.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.755+0000 7f11dffff640 1 --2- 172.21.15.116:0/78814840 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11e81516d0 0x7f11e80c2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:32.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.755+0000 7f11e6d9d640 1 --2- 172.21.15.116:0/78814840 >> v2:172.21.15.174:3300/0 conn(0x7f11e80c4460 0x7f11e814ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:32.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.755+0000 7f11e659c640 1 -- 172.21.15.116:0/78814840 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1147364236 0 0) 0x7f11e8070e60 con 0x7f11e8072dc0 2024-04-25T00:25:32.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.755+0000 7f11e659c640 1 -- 172.21.15.116:0/78814840 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f11d4007940 con 0x7f11e8072dc0 2024-04-25T00:25:32.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.755+0000 7f11e6d9d640 1 -- 172.21.15.116:0/78814840 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11e81516d0 msgr2=0x7f11e80c2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:32.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.755+0000 7f11e6d9d640 1 --2- 172.21.15.116:0/78814840 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11e81516d0 0x7f11e80c2850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.755+0000 7f11e6d9d640 1 -- 172.21.15.116:0/78814840 >> v1:172.21.15.116:6789/0 conn(0x7f11e8072dc0 legacy=0x7f11e806ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:32.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.755+0000 7f11e6d9d640 1 -- 172.21.15.116:0/78814840 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11e8071140 con 0x7f11e80c4460 2024-04-25T00:25:32.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.755+0000 7f11dffff640 1 --2- 172.21.15.116:0/78814840 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11e81516d0 0x7f11e80c2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:25:32.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.767+0000 7f11e6d9d640 1 --2- 172.21.15.116:0/78814840 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11e80c4460 0x7f11e814ebf0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f11d80097c0 tx=0x7f11d8024cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d2edd083aa74f95c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:32.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.767+0000 7f11e659c640 1 -- 172.21.15.116:0/78814840 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d80ba070 con 0x7f11e80c4460 2024-04-25T00:25:32.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.767+0000 7f11e659c640 1 -- 172.21.15.116:0/78814840 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11d80ada80 con 0x7f11e80c4460 2024-04-25T00:25:32.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.767+0000 7f11e659c640 1 -- 172.21.15.116:0/78814840 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d80ade50 con 0x7f11e80c4460 2024-04-25T00:25:32.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.779+0000 7f11ec959640 1 -- 172.21.15.116:0/78814840 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11e80c4460 msgr2=0x7f11e814ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:32.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.779+0000 7f11ec959640 1 --2- 172.21.15.116:0/78814840 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11e80c4460 0x7f11e814ebf0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f11d80097c0 tx=0x7f11d8024cb0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.787+0000 7f11ec959640 1 -- 172.21.15.116:0/78814840 shutdown_connections 2024-04-25T00:25:32.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.787+0000 7f11ec959640 1 --2- 172.21.15.116:0/78814840 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11e81516d0 0x7f11e80c2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.787+0000 7f11ec959640 1 --2- 172.21.15.116:0/78814840 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11e80c4460 0x7f11e814ebf0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.787+0000 7f11ec959640 1 -- 172.21.15.116:0/78814840 >> 172.21.15.116:0/78814840 conn(0x7f11e80c3940 msgr2=0x7f11e80c3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:32.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.787+0000 7f11ec959640 1 -- 172.21.15.116:0/78814840 shutdown_connections 2024-04-25T00:25:32.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.787+0000 7f11ec959640 1 -- 172.21.15.116:0/78814840 wait complete. 2024-04-25T00:25:32.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.787+0000 7f11ec959640 1 Processor -- start 2024-04-25T00:25:32.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.787+0000 7f11ec959640 1 -- start start 2024-04-25T00:25:32.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.787+0000 7f11ec959640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11e8072dc0 0x7f11e8149a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:32.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.787+0000 7f11ec959640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11e80c4460 0x7f11e815bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:32.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.787+0000 7f11ec959640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11e81516d0 0x7f11e814b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:32.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.787+0000 7f11ec959640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f11e8071da0 con 0x7f11e80c4460 2024-04-25T00:25:32.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.787+0000 7f11ec959640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f11e8071c20 con 0x7f11e8072dc0 2024-04-25T00:25:32.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.787+0000 7f11ec959640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f11e80c35c0 con 0x7f11e81516d0 2024-04-25T00:25:32.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.787+0000 7f11dffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11e80c4460 0x7f11e815bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:32.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.787+0000 7f11e759e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11e81516d0 0x7f11e814b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:32.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.787+0000 7f11e6d9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11e8072dc0 0x7f11e8149a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:32.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.787+0000 7f11dffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11e80c4460 0x7f11e815bde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44364/0 (socket says 172.21.15.116:44364) 2024-04-25T00:25:32.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.787+0000 7f11dffff640 1 -- 172.21.15.116:0/1954308798 learned_addr learned my addr 172.21.15.116:0/1954308798 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:25:32.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.791+0000 7f11e759e640 1 -- 172.21.15.116:0/1954308798 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11e8072dc0 msgr2=0x7f11e8149a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:32.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.791+0000 7f11e759e640 1 --2- 172.21.15.116:0/1954308798 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11e8072dc0 0x7f11e8149a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.791+0000 7f11e759e640 1 -- 172.21.15.116:0/1954308798 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11e80c4460 msgr2=0x7f11e815bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:32.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.791+0000 7f11e759e640 1 --2- 172.21.15.116:0/1954308798 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11e80c4460 0x7f11e815bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:32.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.791+0000 7f11e759e640 1 -- 172.21.15.116:0/1954308798 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11e8152d70 con 0x7f11e81516d0 2024-04-25T00:25:32.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.791+0000 7f11e6d9d640 1 --2- 172.21.15.116:0/1954308798 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11e8072dc0 0x7f11e8149a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:25:32.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.791+0000 7f11dffff640 1 --2- 172.21.15.116:0/1954308798 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11e80c4460 0x7f11e815bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:25:32.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.791+0000 7f11e759e640 1 --2- 172.21.15.116:0/1954308798 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11e81516d0 0x7f11e814b2f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f11e000caa0 tx=0x7f11e000cf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:32.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.795+0000 7f11df7fe640 1 -- 172.21.15.116:0/1954308798 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e0017070 con 0x7f11e81516d0 2024-04-25T00:25:32.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.795+0000 7f11df7fe640 1 -- 172.21.15.116:0/1954308798 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11e000aba0 con 0x7f11e81516d0 2024-04-25T00:25:32.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.795+0000 7f11df7fe640 1 -- 172.21.15.116:0/1954308798 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e0012400 con 0x7f11e81516d0 2024-04-25T00:25:32.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.795+0000 7f11ec959640 1 -- 172.21.15.116:0/1954308798 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11e81e7fc0 con 0x7f11e81516d0 2024-04-25T00:25:32.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.795+0000 7f11ec959640 1 -- 172.21.15.116:0/1954308798 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f11e8153010 con 0x7f11e81516d0 2024-04-25T00:25:32.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.795+0000 7f11ec959640 1 -- 172.21.15.116:0/1954308798 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11e804f510 con 0x7f11e81516d0 2024-04-25T00:25:32.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.795+0000 7f11df7fe640 1 -- 172.21.15.116:0/1954308798 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f11e001a070 con 0x7f11e81516d0 2024-04-25T00:25:32.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.795+0000 7f11df7fe640 1 --2- 172.21.15.116:0/1954308798 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f11c80678c0 0x7f11c8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:32.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.795+0000 7f11e6d9d640 1 --2- 172.21.15.116:0/1954308798 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f11c80678c0 0x7f11c8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:32.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.799+0000 7f11e6d9d640 1 --2- 172.21.15.116:0/1954308798 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f11c80678c0 0x7f11c8069d80 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f11d80b7000 tx=0x7f11d80af070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:32.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.799+0000 7f11df7fe640 1 -- 172.21.15.116:0/1954308798 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(66..66 src has 1..66) ==== 9934+0+0 (secure 0 0 0) 0x7f11e0053ee0 con 0x7f11e81516d0 2024-04-25T00:25:32.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:32.807+0000 7f11df7fe640 1 -- 172.21.15.116:0/1954308798 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f11e0055e40 con 0x7f11e81516d0 2024-04-25T00:25:32.950 INFO:tasks.ceph.mds.d.smithi174.stderr:2024-04-25T00:25:32.948+0000 7fae58f00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T00:25:33.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:33.443+0000 7f11ec959640 1 -- 172.21.15.116:0/1954308798 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f11e80728a0 con 0x7f11e81516d0 2024-04-25T00:25:33.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:33.443+0000 7f11df7fe640 1 -- 172.21.15.116:0/1954308798 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 202 v202) ==== 95+0+6510 (secure 0 0 0) 0x7f11e005a140 con 0x7f11e81516d0 2024-04-25T00:25:33.448 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:25:33.448 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":202,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":8496,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/3693049068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3693049068},{"type":"v1","addr":"172.21.15.116:6833","nonce":3693049068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":202}],"filesystems":[{"mdsmap":{"epoch":201,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:25:29.028925+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6413},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6413":{"gid":6413,"name":"d","rank":0,"incarnation":198,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/4181466530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":4181466530},{"type":"v1","addr":"172.21.15.174:6837","nonce":4181466530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6413,"qdb_cluster":[6413]},"id":1},{"mdsmap":{"epoch":196,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:24:19.977863+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8343":{"gid":8343,"name":"a","rank":0,"incarnation":193,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6833/3633657268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3633657268},{"type":"v1","addr":"172.21.15.116:6833","nonce":3633657268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8343,"qdb_cluster":[8343]},"id":2}]} 2024-04-25T00:25:33.448 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 202 2024-04-25T00:25:33.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:33.447+0000 7f11ec959640 1 -- 172.21.15.116:0/1954308798 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f11c80678c0 msgr2=0x7f11c8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:33.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:33.447+0000 7f11ec959640 1 --2- 172.21.15.116:0/1954308798 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f11c80678c0 0x7f11c8069d80 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f11d80b7000 tx=0x7f11d80af070 comp rx=0 tx=0).stop 2024-04-25T00:25:33.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:33.447+0000 7f11ec959640 1 -- 172.21.15.116:0/1954308798 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11e81516d0 msgr2=0x7f11e814b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:33.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:33.447+0000 7f11ec959640 1 --2- 172.21.15.116:0/1954308798 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11e81516d0 0x7f11e814b2f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f11e000caa0 tx=0x7f11e000cf60 comp rx=0 tx=0).stop 2024-04-25T00:25:33.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:33.447+0000 7f11ec959640 1 -- 172.21.15.116:0/1954308798 shutdown_connections 2024-04-25T00:25:33.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:33.447+0000 7f11ec959640 1 --2- 172.21.15.116:0/1954308798 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f11c80678c0 0x7f11c8069d80 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:33.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:33.447+0000 7f11ec959640 1 --2- 172.21.15.116:0/1954308798 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11e81516d0 0x7f11e814b2f0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:33.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:33.447+0000 7f11ec959640 1 --2- 172.21.15.116:0/1954308798 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11e80c4460 0x7f11e815bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:33.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:33.447+0000 7f11ec959640 1 --2- 172.21.15.116:0/1954308798 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11e8072dc0 0x7f11e8149a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:33.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:33.447+0000 7f11ec959640 1 -- 172.21.15.116:0/1954308798 >> 172.21.15.116:0/1954308798 conn(0x7f11e80c3940 msgr2=0x7f11e80c3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:33.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:33.447+0000 7f11ec959640 1 -- 172.21.15.116:0/1954308798 shutdown_connections 2024-04-25T00:25:33.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:33.447+0000 7f11ec959640 1 -- 172.21.15.116:0/1954308798 wait complete. 2024-04-25T00:25:33.492 INFO:tasks.mds_thrash.fs.[a]:mds.c reported in up:standby state 2024-04-25T00:25:33.492 INFO:tasks.mds_thrash.fs.[a]:waiting for 24 secs before thrashing 2024-04-25T00:25:34.064 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:25:34.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.299+0000 7f3f2e30e640 1 Processor -- start 2024-04-25T00:25:34.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.299+0000 7f3f2e30e640 1 -- start start 2024-04-25T00:25:34.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.299+0000 7f3f2e30e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3f200462f0 0x7f3f200070d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:34.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.299+0000 7f3f2e30e640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f3f200496e0 0x7f3f20049b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:34.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.299+0000 7f3f2e30e640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f20046820 con 0x7f3f200d3ab0 2024-04-25T00:25:34.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.299+0000 7f3f2e30e640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f3f20046fb0 con 0x7f3f200462f0 2024-04-25T00:25:34.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.299+0000 7f3f2e30e640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f3f20007720 con 0x7f3f200496e0 2024-04-25T00:25:34.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.299+0000 7f3f2d30c640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f3f200d3ab0 0x7f3f200069c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:50490/0 (socket says 172.21.15.116:50490) 2024-04-25T00:25:34.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.299+0000 7f3f2db0d640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f3f200496e0 0x7f3f20049b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:34.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.299+0000 7f3f2cb0b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3f200462f0 0x7f3f200070d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:34.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.299+0000 7f3f2db0d640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f3f200496e0 0x7f3f20049b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:40728/0 (socket says 172.21.15.116:40728) 2024-04-25T00:25:34.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.299+0000 7f3f2db0d640 1 -- 172.21.15.116:0/1643891883 learned_addr learned my addr 172.21.15.116:0/1643891883 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:25:34.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.299+0000 7f3f2db0d640 1 -- 172.21.15.116:0/1643891883 >> v2:172.21.15.116:3300/0 conn(0x7f3f200462f0 msgr2=0x7f3f200070d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:34.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.299+0000 7f3f2db0d640 1 --2- 172.21.15.116:0/1643891883 >> v2:172.21.15.116:3300/0 conn(0x7f3f200462f0 0x7f3f200070d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:34.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.299+0000 7f3f1ffff640 1 -- 172.21.15.116:0/1643891883 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2020681951 0 0) 0x7f3f20046820 con 0x7f3f200d3ab0 2024-04-25T00:25:34.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.299+0000 7f3f2db0d640 1 -- 172.21.15.116:0/1643891883 >> v1:172.21.15.174:6789/0 conn(0x7f3f200d3ab0 legacy=0x7f3f200069c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:34.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.299+0000 7f3f2db0d640 1 -- 172.21.15.116:0/1643891883 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f200d7040 con 0x7f3f200496e0 2024-04-25T00:25:34.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.299+0000 7f3f2cb0b640 1 --2- 172.21.15.116:0/1643891883 >> v2:172.21.15.116:3300/0 conn(0x7f3f200462f0 0x7f3f200070d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:25:34.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.303+0000 7f3f2db0d640 1 --2- 172.21.15.116:0/1643891883 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f200496e0 0x7f3f20049b10 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f3f20046780 tx=0x7f3f2401c4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=867740d5e4a853af server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:34.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.303+0000 7f3f1ffff640 1 -- 172.21.15.116:0/1643891883 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f240b0070 con 0x7f3f200496e0 2024-04-25T00:25:34.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.303+0000 7f3f1ffff640 1 -- 172.21.15.116:0/1643891883 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f240ab600 con 0x7f3f200496e0 2024-04-25T00:25:34.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.303+0000 7f3f1ffff640 1 -- 172.21.15.116:0/1643891883 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f240b0070 con 0x7f3f200496e0 2024-04-25T00:25:34.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.303+0000 7f3f2e30e640 1 -- 172.21.15.116:0/1643891883 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f200496e0 msgr2=0x7f3f20049b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:34.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.303+0000 7f3f2e30e640 1 --2- 172.21.15.116:0/1643891883 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f200496e0 0x7f3f20049b10 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f3f20046780 tx=0x7f3f2401c4b0 comp rx=0 tx=0).stop 2024-04-25T00:25:34.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.307+0000 7f3f2e30e640 1 -- 172.21.15.116:0/1643891883 shutdown_connections 2024-04-25T00:25:34.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.307+0000 7f3f2e30e640 1 --2- 172.21.15.116:0/1643891883 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f200496e0 0x7f3f20049b10 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:34.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.307+0000 7f3f2e30e640 1 --2- 172.21.15.116:0/1643891883 >> v2:172.21.15.116:3300/0 conn(0x7f3f200462f0 0x7f3f200070d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:34.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.307+0000 7f3f2e30e640 1 -- 172.21.15.116:0/1643891883 >> 172.21.15.116:0/1643891883 conn(0x7f3f20005ea0 msgr2=0x7f3f200062b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:34.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.307+0000 7f3f2e30e640 1 -- 172.21.15.116:0/1643891883 shutdown_connections 2024-04-25T00:25:34.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.307+0000 7f3f2e30e640 1 -- 172.21.15.116:0/1643891883 wait complete. 2024-04-25T00:25:34.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.307+0000 7f3f2e30e640 1 Processor -- start 2024-04-25T00:25:34.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.311+0000 7f3f2e30e640 1 -- start start 2024-04-25T00:25:34.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.311+0000 7f3f2e30e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f200462f0 0x7f3f20030a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:34.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.311+0000 7f3f2e30e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f200496e0 0x7f3f200d8420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:34.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.311+0000 7f3f2e30e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f200d3ab0 0x7f3f2017acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:34.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.311+0000 7f3f2e30e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3f200e35e0 con 0x7f3f200462f0 2024-04-25T00:25:34.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.311+0000 7f3f2e30e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3f200d3e90 con 0x7f3f200d3ab0 2024-04-25T00:25:34.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.311+0000 7f3f2e30e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f3f200e3760 con 0x7f3f200496e0 2024-04-25T00:25:34.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.311+0000 7f3f2d30c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f200462f0 0x7f3f20030a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:34.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.311+0000 7f3f2d30c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f200462f0 0x7f3f20030a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44382/0 (socket says 172.21.15.116:44382) 2024-04-25T00:25:34.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.311+0000 7f3f2d30c640 1 -- 172.21.15.116:0/182848953 learned_addr learned my addr 172.21.15.116:0/182848953 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:25:34.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.311+0000 7f3f2d30c640 1 -- 172.21.15.116:0/182848953 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f200496e0 msgr2=0x7f3f200d8420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:34.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.311+0000 7f3f2d30c640 1 --2- 172.21.15.116:0/182848953 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f200496e0 0x7f3f200d8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:34.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.311+0000 7f3f2d30c640 1 -- 172.21.15.116:0/182848953 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f200d3ab0 msgr2=0x7f3f2017acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:34.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.311+0000 7f3f2d30c640 1 --2- 172.21.15.116:0/182848953 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f200d3ab0 0x7f3f2017acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:34.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.311+0000 7f3f2d30c640 1 -- 172.21.15.116:0/182848953 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f200d3120 con 0x7f3f200462f0 2024-04-25T00:25:34.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.311+0000 7f3f2d30c640 1 --2- 172.21.15.116:0/182848953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f200462f0 0x7f3f20030a90 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f3f28069930 tx=0x7f3f28073780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:34.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.315+0000 7f3f1dffb640 1 -- 172.21.15.116:0/182848953 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f280728f0 con 0x7f3f200462f0 2024-04-25T00:25:34.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.315+0000 7f3f1dffb640 1 -- 172.21.15.116:0/182848953 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f2808f030 con 0x7f3f200462f0 2024-04-25T00:25:34.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.315+0000 7f3f1dffb640 1 -- 172.21.15.116:0/182848953 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f28072d70 con 0x7f3f200462f0 2024-04-25T00:25:34.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.319+0000 7f3f2e30e640 1 -- 172.21.15.116:0/182848953 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f2017b780 con 0x7f3f200462f0 2024-04-25T00:25:34.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.319+0000 7f3f2e30e640 1 -- 172.21.15.116:0/182848953 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f2002f910 con 0x7f3f200462f0 2024-04-25T00:25:34.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.319+0000 7f3f1dffb640 1 -- 172.21.15.116:0/182848953 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3f28071040 con 0x7f3f200462f0 2024-04-25T00:25:34.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.319+0000 7f3f2e30e640 1 -- 172.21.15.116:0/182848953 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f200053a0 con 0x7f3f200462f0 2024-04-25T00:25:34.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.323+0000 7f3f1dffb640 1 --2- 172.21.15.116:0/182848953 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3f0c067990 0x7f3f0c069e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:34.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.323+0000 7f3f2cb0b640 1 --2- 172.21.15.116:0/182848953 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3f0c067990 0x7f3f0c069e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:34.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.323+0000 7f3f1dffb640 1 -- 172.21.15.116:0/182848953 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 9934+0+0 (secure 0 0 0) 0x7f3f281029c0 con 0x7f3f200462f0 2024-04-25T00:25:34.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.327+0000 7f3f2cb0b640 1 --2- 172.21.15.116:0/182848953 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3f0c067990 0x7f3f0c069e50 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f3f1800afd0 tx=0x7f3f1800c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:34.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.327+0000 7f3f1dffb640 1 -- 172.21.15.116:0/182848953 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f3f200d3120 con 0x7f3f200462f0 2024-04-25T00:25:35.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.995+0000 7f3f2e30e640 1 -- 172.21.15.116:0/182848953 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3f20048020 con 0x7f3f200462f0 2024-04-25T00:25:35.003 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:25:35.003 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":202,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":8496,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/3693049068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3693049068},{"type":"v1","addr":"172.21.15.116:6833","nonce":3693049068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":202}],"filesystems":[{"mdsmap":{"epoch":201,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:25:29.028925+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6413},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6413":{"gid":6413,"name":"d","rank":0,"incarnation":198,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/4181466530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":4181466530},{"type":"v1","addr":"172.21.15.174:6837","nonce":4181466530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6413,"qdb_cluster":[6413]},"id":1},{"mdsmap":{"epoch":196,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:24:19.977863+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8343":{"gid":8343,"name":"a","rank":0,"incarnation":193,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6833/3633657268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3633657268},{"type":"v1","addr":"172.21.15.116:6833","nonce":3633657268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8343,"qdb_cluster":[8343]},"id":2}]} 2024-04-25T00:25:35.004 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 202 2024-04-25T00:25:35.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:34.999+0000 7f3f1dffb640 1 -- 172.21.15.116:0/182848953 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 202 v202) ==== 95+0+6510 (secure 0 0 0) 0x7f3f280d2480 con 0x7f3f200462f0 2024-04-25T00:25:35.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:35.003+0000 7f3f0b7fe640 1 -- 172.21.15.116:0/182848953 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3f0c067990 msgr2=0x7f3f0c069e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:35.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:35.003+0000 7f3f0b7fe640 1 --2- 172.21.15.116:0/182848953 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3f0c067990 0x7f3f0c069e50 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f3f1800afd0 tx=0x7f3f1800c040 comp rx=0 tx=0).stop 2024-04-25T00:25:35.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:35.003+0000 7f3f0b7fe640 1 -- 172.21.15.116:0/182848953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f200462f0 msgr2=0x7f3f20030a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:35.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:35.003+0000 7f3f0b7fe640 1 --2- 172.21.15.116:0/182848953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f200462f0 0x7f3f20030a90 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f3f28069930 tx=0x7f3f28073780 comp rx=0 tx=0).stop 2024-04-25T00:25:35.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:35.003+0000 7f3f0b7fe640 1 -- 172.21.15.116:0/182848953 shutdown_connections 2024-04-25T00:25:35.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:35.003+0000 7f3f0b7fe640 1 --2- 172.21.15.116:0/182848953 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3f0c067990 0x7f3f0c069e50 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:35.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:35.003+0000 7f3f0b7fe640 1 --2- 172.21.15.116:0/182848953 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3f200d3ab0 0x7f3f2017acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:35.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:35.003+0000 7f3f0b7fe640 1 --2- 172.21.15.116:0/182848953 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3f200496e0 0x7f3f200d8420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:35.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:35.003+0000 7f3f0b7fe640 1 --2- 172.21.15.116:0/182848953 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3f200462f0 0x7f3f20030a90 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:35.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:35.003+0000 7f3f0b7fe640 1 -- 172.21.15.116:0/182848953 >> 172.21.15.116:0/182848953 conn(0x7f3f20005ea0 msgr2=0x7f3f200e0210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:35.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:35.003+0000 7f3f0b7fe640 1 -- 172.21.15.116:0/182848953 shutdown_connections 2024-04-25T00:25:35.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:35.003+0000 7f3f0b7fe640 1 -- 172.21.15.116:0/182848953 wait complete. 2024-04-25T00:25:35.049 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:25:37.050 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:25:37.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:37.423+0000 7f925498f640 1 Processor -- start 2024-04-25T00:25:37.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:37.423+0000 7f925498f640 1 -- start start 2024-04-25T00:25:37.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:37.423+0000 7f925498f640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f9250152db0 0x7f92500c7af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:37.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:37.423+0000 7f925498f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f92501558c0 0x7f9250155d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:37.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:37.423+0000 7f925498f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9250158eb0 con 0x7f92500c8030 2024-04-25T00:25:37.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:37.423+0000 7f925498f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f9250159190 con 0x7f92501558c0 2024-04-25T00:25:37.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:37.423+0000 7f925498f640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f9250159470 con 0x7f9250152db0 2024-04-25T00:25:37.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:37.423+0000 7f924ed9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f92501558c0 0x7f9250155d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:37.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:37.423+0000 7f924ed9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f92501558c0 0x7f9250155d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46680/0 (socket says 172.21.15.116:46680) 2024-04-25T00:25:37.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:37.423+0000 7f924ed9d640 1 -- 172.21.15.116:0/3542156769 learned_addr learned my addr 172.21.15.116:0/3542156769 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:25:37.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:37.423+0000 7f924e59c640 1 --2- 172.21.15.116:0/3542156769 >> v2:172.21.15.174:3300/0 conn(0x7f9250152db0 0x7f92500c7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:37.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:37.423+0000 7f924ed9d640 1 -- 172.21.15.116:0/3542156769 >> v2:172.21.15.174:3300/0 conn(0x7f9250152db0 msgr2=0x7f92500c7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:37.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:37.423+0000 7f924ed9d640 1 --2- 172.21.15.116:0/3542156769 >> v2:172.21.15.174:3300/0 conn(0x7f9250152db0 0x7f92500c7af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:37.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:37.423+0000 7f924ed9d640 1 -- 172.21.15.116:0/3542156769 >> v1:172.21.15.116:6789/0 conn(0x7f92500c8030 legacy=0x7f92500c5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:37.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:37.423+0000 7f924ed9d640 1 -- 172.21.15.116:0/3542156769 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9250159750 con 0x7f92501558c0 2024-04-25T00:25:37.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:37.431+0000 7f924ed9d640 1 --2- 172.21.15.116:0/3542156769 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92501558c0 0x7f9250155d10 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f92500c8c60 tx=0x7f9248025100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41f885bfbe48f21f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:37.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:37.431+0000 7f924d59a640 1 -- 172.21.15.116:0/3542156769 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92480b9070 con 0x7f92501558c0 2024-04-25T00:25:37.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:37.431+0000 7f924d59a640 1 -- 172.21.15.116:0/3542156769 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92480b4400 con 0x7f92501558c0 2024-04-25T00:25:37.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:37.431+0000 7f924d59a640 1 -- 172.21.15.116:0/3542156769 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92480b47d0 con 0x7f92501558c0 2024-04-25T00:25:37.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:37.431+0000 7f925498f640 1 -- 172.21.15.116:0/3542156769 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92501558c0 msgr2=0x7f9250155d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:37.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:37.431+0000 7f925498f640 1 --2- 172.21.15.116:0/3542156769 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92501558c0 0x7f9250155d10 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f92500c8c60 tx=0x7f9248025100 comp rx=0 tx=0).stop 2024-04-25T00:25:37.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:37.439+0000 7f925498f640 1 -- 172.21.15.116:0/3542156769 shutdown_connections 2024-04-25T00:25:37.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:37.439+0000 7f925498f640 1 --2- 172.21.15.116:0/3542156769 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92501558c0 0x7f9250155d10 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:37.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:37.439+0000 7f925498f640 1 --2- 172.21.15.116:0/3542156769 >> v2:172.21.15.174:3300/0 conn(0x7f9250152db0 0x7f92500c7af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:37.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:37.439+0000 7f925498f640 1 -- 172.21.15.116:0/3542156769 >> 172.21.15.116:0/3542156769 conn(0x7f92500acc10 msgr2=0x7f92500ad020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:37.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:37.439+0000 7f925498f640 1 -- 172.21.15.116:0/3542156769 shutdown_connections 2024-04-25T00:25:37.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:37.443+0000 7f925498f640 1 -- 172.21.15.116:0/3542156769 wait complete. 2024-04-25T00:25:37.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:37.443+0000 7f925498f640 1 Processor -- start 2024-04-25T00:25:37.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:37.443+0000 7f925498f640 1 -- start start 2024-04-25T00:25:37.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:37.443+0000 7f925498f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92500c8030 0x7f9250148330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:37.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:37.443+0000 7f925498f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9250152db0 0x7f9250148870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:37.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:37.443+0000 7f925498f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92501558c0 0x7f9250154370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:37.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:37.443+0000 7f925498f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9250153310 con 0x7f92501558c0 2024-04-25T00:25:37.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:37.443+0000 7f925498f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9250153190 con 0x7f92500c8030 2024-04-25T00:25:37.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:37.443+0000 7f925498f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f9250070d50 con 0x7f9250152db0 2024-04-25T00:25:37.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:37.447+0000 7f924ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92501558c0 0x7f9250154370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:37.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:37.447+0000 7f924ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92501558c0 0x7f9250154370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46682/0 (socket says 172.21.15.116:46682) 2024-04-25T00:25:37.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:37.447+0000 7f924ed9d640 1 -- 172.21.15.116:0/845903129 learned_addr learned my addr 172.21.15.116:0/845903129 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:25:37.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:37.447+0000 7f924ed9d640 1 -- 172.21.15.116:0/845903129 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9250152db0 msgr2=0x7f9250148870 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T00:25:37.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:37.447+0000 7f924ed9d640 1 --2- 172.21.15.116:0/845903129 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9250152db0 0x7f9250148870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:37.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:37.447+0000 7f924ed9d640 1 -- 172.21.15.116:0/845903129 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92500c8030 msgr2=0x7f9250148330 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T00:25:37.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:37.447+0000 7f924ed9d640 1 --2- 172.21.15.116:0/845903129 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92500c8030 0x7f9250148330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:37.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:37.447+0000 7f924ed9d640 1 -- 172.21.15.116:0/845903129 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9250155300 con 0x7f92501558c0 2024-04-25T00:25:37.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:37.447+0000 7f924ed9d640 1 --2- 172.21.15.116:0/845903129 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92501558c0 0x7f9250154370 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f9248021d10 tx=0x7f9248021ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:37.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:37.447+0000 7f923effd640 1 -- 172.21.15.116:0/845903129 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92480b9020 con 0x7f92501558c0 2024-04-25T00:25:37.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:37.447+0000 7f925498f640 1 -- 172.21.15.116:0/845903129 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92501555e0 con 0x7f92501558c0 2024-04-25T00:25:37.447 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:37.447+0000 7f925498f640 1 -- 172.21.15.116:0/845903129 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92500ac770 con 0x7f92501558c0 2024-04-25T00:25:37.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:37.447+0000 7f923effd640 1 -- 172.21.15.116:0/845903129 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92480b4bc0 con 0x7f92501558c0 2024-04-25T00:25:37.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:37.447+0000 7f925498f640 1 -- 172.21.15.116:0/845903129 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92500ac970 con 0x7f92501558c0 2024-04-25T00:25:37.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:37.455+0000 7f923effd640 1 -- 172.21.15.116:0/845903129 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9248014410 con 0x7f92501558c0 2024-04-25T00:25:37.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:37.455+0000 7f923effd640 1 -- 172.21.15.116:0/845903129 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f92480bc020 con 0x7f92501558c0 2024-04-25T00:25:37.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:37.455+0000 7f923effd640 1 --2- 172.21.15.116:0/845903129 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f92300679c0 0x7f9230069e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:37.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:37.455+0000 7f923effd640 1 -- 172.21.15.116:0/845903129 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 9934+0+0 (secure 0 0 0) 0x7f9248121110 con 0x7f92501558c0 2024-04-25T00:25:37.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:37.455+0000 7f923effd640 1 -- 172.21.15.116:0/845903129 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f92500ac970 con 0x7f92501558c0 2024-04-25T00:25:37.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:37.455+0000 7f924e59c640 1 --2- 172.21.15.116:0/845903129 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f92300679c0 0x7f9230069e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:37.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:37.459+0000 7f924e59c640 1 --2- 172.21.15.116:0/845903129 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f92300679c0 0x7f9230069e80 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f92400099f0 tx=0x7f9240009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:38.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:38.139+0000 7f925498f640 1 -- 172.21.15.116:0/845903129 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9250068a60 con 0x7f92501558c0 2024-04-25T00:25:38.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:38.139+0000 7f923effd640 1 -- 172.21.15.116:0/845903129 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 202 v202) ==== 95+0+6510 (secure 0 0 0) 0x7f92480c3030 con 0x7f92501558c0 2024-04-25T00:25:38.146 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:25:38.146 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":202,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":8496,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/3693049068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3693049068},{"type":"v1","addr":"172.21.15.116:6833","nonce":3693049068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":202}],"filesystems":[{"mdsmap":{"epoch":201,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:25:29.028925+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6413},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6413":{"gid":6413,"name":"d","rank":0,"incarnation":198,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/4181466530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":4181466530},{"type":"v1","addr":"172.21.15.174:6837","nonce":4181466530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6413,"qdb_cluster":[6413]},"id":1},{"mdsmap":{"epoch":196,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:24:19.977863+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8343},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8343":{"gid":8343,"name":"a","rank":0,"incarnation":193,"state":"up:active","state_seq":12,"addr":"172.21.15.116:6833/3633657268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3633657268},{"type":"v1","addr":"172.21.15.116:6833","nonce":3633657268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8343,"qdb_cluster":[8343]},"id":2}]} 2024-04-25T00:25:38.146 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 202 2024-04-25T00:25:38.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:38.151+0000 7f925498f640 1 -- 172.21.15.116:0/845903129 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f92300679c0 msgr2=0x7f9230069e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:38.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:38.151+0000 7f925498f640 1 --2- 172.21.15.116:0/845903129 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f92300679c0 0x7f9230069e80 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f92400099f0 tx=0x7f9240009190 comp rx=0 tx=0).stop 2024-04-25T00:25:38.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:38.151+0000 7f925498f640 1 -- 172.21.15.116:0/845903129 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92501558c0 msgr2=0x7f9250154370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:38.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:38.151+0000 7f925498f640 1 --2- 172.21.15.116:0/845903129 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92501558c0 0x7f9250154370 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f9248021d10 tx=0x7f9248021ea0 comp rx=0 tx=0).stop 2024-04-25T00:25:38.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:38.155+0000 7f925498f640 1 -- 172.21.15.116:0/845903129 shutdown_connections 2024-04-25T00:25:38.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:38.155+0000 7f925498f640 1 --2- 172.21.15.116:0/845903129 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f92300679c0 0x7f9230069e80 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:38.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:38.155+0000 7f925498f640 1 --2- 172.21.15.116:0/845903129 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f92501558c0 0x7f9250154370 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:38.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:38.155+0000 7f925498f640 1 --2- 172.21.15.116:0/845903129 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9250152db0 0x7f9250148870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:38.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:38.155+0000 7f925498f640 1 --2- 172.21.15.116:0/845903129 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f92500c8030 0x7f9250148330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:38.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:38.155+0000 7f925498f640 1 -- 172.21.15.116:0/845903129 >> 172.21.15.116:0/845903129 conn(0x7f92500acc10 msgr2=0x7f925014d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:38.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:38.155+0000 7f925498f640 1 -- 172.21.15.116:0/845903129 shutdown_connections 2024-04-25T00:25:38.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:38.159+0000 7f925498f640 1 -- 172.21.15.116:0/845903129 wait complete. 2024-04-25T00:25:38.222 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:25:40.223 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:25:40.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:40.531+0000 7faa1b4c8640 1 Processor -- start 2024-04-25T00:25:40.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:40.531+0000 7faa1b4c8640 1 -- start start 2024-04-25T00:25:40.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:40.535+0000 7faa1b4c8640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7faa1414c700 0x7faa140c5090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:40.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:40.535+0000 7faa1b4c8640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa140a9e60 con 0x7faa14155b20 2024-04-25T00:25:40.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:40.535+0000 7faa1b4c8640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa140c3ef0 con 0x7faa140c55d0 2024-04-25T00:25:40.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:40.535+0000 7faa1b4c8640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7faa141527b0 con 0x7faa1414c700 2024-04-25T00:25:40.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:40.535+0000 7faa19a65640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7faa140c55d0 0x7faa1415ceb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:40604/0 (socket says 172.21.15.116:40604) 2024-04-25T00:25:40.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:40.535+0000 7faa19a65640 1 -- 172.21.15.116:0/3481606096 learned_addr learned my addr 172.21.15.116:0/3481606096 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:25:40.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:40.535+0000 7faa19264640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7faa1414c700 0x7faa140c5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:40.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:40.535+0000 7faa0bfff640 1 -- 172.21.15.116:0/3481606096 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1951030527 0 0) 0x7faa140c3ef0 con 0x7faa140c55d0 2024-04-25T00:25:40.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:40.535+0000 7faa0bfff640 1 -- 172.21.15.116:0/3481606096 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa00003530 con 0x7faa140c55d0 2024-04-25T00:25:40.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:40.535+0000 7faa19264640 1 -- 172.21.15.116:0/3481606096 >> v1:172.21.15.174:6789/0 conn(0x7faa140c55d0 legacy=0x7faa1415ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:40.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:40.535+0000 7faa19264640 1 -- 172.21.15.116:0/3481606096 >> v1:172.21.15.116:6789/0 conn(0x7faa14155b20 legacy=0x7faa14152c40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T00:25:40.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:40.535+0000 7faa19264640 1 -- 172.21.15.116:0/3481606096 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa140c41d0 con 0x7faa1414c700 2024-04-25T00:25:40.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:40.547+0000 7faa19264640 1 --2- 172.21.15.116:0/3481606096 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa1414c700 0x7faa140c5090 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7faa040097c0 tx=0x7faa0401c340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47643d786e6fbe29 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:40.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:40.551+0000 7faa0bfff640 1 -- 172.21.15.116:0/3481606096 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa040b0070 con 0x7faa1414c700 2024-04-25T00:25:40.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:40.551+0000 7faa0bfff640 1 -- 172.21.15.116:0/3481606096 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa040ab600 con 0x7faa1414c700 2024-04-25T00:25:40.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:40.551+0000 7faa0bfff640 1 -- 172.21.15.116:0/3481606096 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa040ab9d0 con 0x7faa1414c700 2024-04-25T00:25:40.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:40.551+0000 7faa1b4c8640 1 -- 172.21.15.116:0/3481606096 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa1414c700 msgr2=0x7faa140c5090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:40.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:40.551+0000 7faa1b4c8640 1 --2- 172.21.15.116:0/3481606096 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa1414c700 0x7faa140c5090 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7faa040097c0 tx=0x7faa0401c340 comp rx=0 tx=0).stop 2024-04-25T00:25:40.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:40.551+0000 7faa1b4c8640 1 -- 172.21.15.116:0/3481606096 shutdown_connections 2024-04-25T00:25:40.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:40.551+0000 7faa1b4c8640 1 --2- 172.21.15.116:0/3481606096 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa1414c700 0x7faa140c5090 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:40.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:40.551+0000 7faa1b4c8640 1 -- 172.21.15.116:0/3481606096 >> 172.21.15.116:0/3481606096 conn(0x7faa1406c720 msgr2=0x7faa1406cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:40.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:40.551+0000 7faa1b4c8640 1 -- 172.21.15.116:0/3481606096 shutdown_connections 2024-04-25T00:25:40.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:40.551+0000 7faa1b4c8640 1 -- 172.21.15.116:0/3481606096 wait complete. 2024-04-25T00:25:40.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:40.555+0000 7faa1b4c8640 1 Processor -- start 2024-04-25T00:25:40.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:40.555+0000 7faa1b4c8640 1 -- start start 2024-04-25T00:25:40.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:40.555+0000 7faa1b4c8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa140c55d0 0x7faa140ab800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:40.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:40.555+0000 7faa1b4c8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa1414c700 0x7faa14143820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:40.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:40.555+0000 7faa1b4c8640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa14155b20 0x7faa14155230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:40.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:40.555+0000 7faa1b4c8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7faa1406be30 con 0x7faa1414c700 2024-04-25T00:25:40.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:40.555+0000 7faa19264640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa140c55d0 0x7faa140ab800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:40.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:40.555+0000 7faa19264640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa140c55d0 0x7faa140ab800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:60580/0 (socket says 172.21.15.116:60580) 2024-04-25T00:25:40.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:40.555+0000 7faa19264640 1 -- 172.21.15.116:0/3939744793 learned_addr learned my addr 172.21.15.116:0/3939744793 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:25:40.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:40.555+0000 7faa1b4c8640 1 -- 172.21.15.116:0/3939744793 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7faa1406bcb0 con 0x7faa140c55d0 2024-04-25T00:25:40.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:40.555+0000 7faa1b4c8640 1 -- 172.21.15.116:0/3939744793 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7faa1406bfb0 con 0x7faa14155b20 2024-04-25T00:25:40.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:40.555+0000 7faa19a65640 1 --2- 172.21.15.116:0/3939744793 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa14155b20 0x7faa14155230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:40.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:40.555+0000 7faa18a63640 1 --2- 172.21.15.116:0/3939744793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa1414c700 0x7faa14143820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:40.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:40.555+0000 7faa19264640 1 -- 172.21.15.116:0/3939744793 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa14155b20 msgr2=0x7faa14155230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:40.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:40.555+0000 7faa19264640 1 --2- 172.21.15.116:0/3939744793 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa14155b20 0x7faa14155230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:40.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:40.555+0000 7faa19264640 1 -- 172.21.15.116:0/3939744793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa1414c700 msgr2=0x7faa14143820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:40.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:40.555+0000 7faa19264640 1 --2- 172.21.15.116:0/3939744793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa1414c700 0x7faa14143820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:40.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:40.555+0000 7faa19264640 1 -- 172.21.15.116:0/3939744793 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa14153600 con 0x7faa140c55d0 2024-04-25T00:25:40.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:40.555+0000 7faa18a63640 1 --2- 172.21.15.116:0/3939744793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa1414c700 0x7faa14143820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:25:40.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:40.559+0000 7faa19264640 1 --2- 172.21.15.116:0/3939744793 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa140c55d0 0x7faa140ab800 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7faa040ae040 tx=0x7faa0401ab80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:40.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:40.559+0000 7faa09ffb640 1 -- 172.21.15.116:0/3939744793 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa040b0050 con 0x7faa140c55d0 2024-04-25T00:25:40.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:40.559+0000 7faa09ffb640 1 -- 172.21.15.116:0/3939744793 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa040bc070 con 0x7faa140c55d0 2024-04-25T00:25:40.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:40.559+0000 7faa09ffb640 1 -- 172.21.15.116:0/3939744793 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa040b5870 con 0x7faa140c55d0 2024-04-25T00:25:40.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:40.559+0000 7faa1b4c8640 1 -- 172.21.15.116:0/3939744793 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa141f39f0 con 0x7faa140c55d0 2024-04-25T00:25:40.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:40.559+0000 7faa1b4c8640 1 -- 172.21.15.116:0/3939744793 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa141f3cd0 con 0x7faa140c55d0 2024-04-25T00:25:40.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:40.563+0000 7faa1b4c8640 1 -- 172.21.15.116:0/3939744793 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa14068a60 con 0x7faa140c55d0 2024-04-25T00:25:40.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:40.563+0000 7faa09ffb640 1 -- 172.21.15.116:0/3939744793 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7faa040b5a10 con 0x7faa140c55d0 2024-04-25T00:25:40.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:40.563+0000 7faa09ffb640 1 --2- 172.21.15.116:0/3939744793 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa9f80678c0 0x7fa9f8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:40.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:40.563+0000 7faa09ffb640 1 -- 172.21.15.116:0/3939744793 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 9934+0+0 (secure 0 0 0) 0x7faa041213f0 con 0x7faa140c55d0 2024-04-25T00:25:40.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:40.563+0000 7faa18a63640 1 --2- 172.21.15.116:0/3939744793 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa9f80678c0 0x7fa9f8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:40.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:40.563+0000 7faa18a63640 1 --2- 172.21.15.116:0/3939744793 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa9f80678c0 0x7fa9f8069d80 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7faa140c2560 tx=0x7faa100095c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:40.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:40.571+0000 7faa09ffb640 1 -- 172.21.15.116:0/3939744793 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7faa04016050 con 0x7faa140c55d0 2024-04-25T00:25:41.156 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:25:41.158 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:25:41.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.471+0000 7faa1b4c8640 1 -- 172.21.15.116:0/3939744793 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faa140ad630 con 0x7faa140c55d0 2024-04-25T00:25:41.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.471+0000 7faa09ffb640 1 -- 172.21.15.116:0/3939744793 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 203 v203) ==== 95+0+5673 (secure 0 0 0) 0x7faa040f0e00 con 0x7faa140c55d0 2024-04-25T00:25:41.471 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:25:41.471 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":203,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":201,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:25:29.028925+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6413},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6413":{"gid":6413,"name":"d","rank":0,"incarnation":198,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/4181466530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":4181466530},{"type":"v1","addr":"172.21.15.174:6837","nonce":4181466530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6413,"qdb_cluster":[6413]},"id":1},{"mdsmap":{"epoch":203,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:25:40.724980+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8496":{"gid":8496,"name":"c","rank":0,"incarnation":203,"state":"up:replay","state_seq":1,"addr":"172.21.15.116:6833/3693049068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3693049068},{"type":"v1","addr":"172.21.15.116:6833","nonce":3693049068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:25:41.472 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 203 2024-04-25T00:25:41.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.475+0000 7faa1b4c8640 1 -- 172.21.15.116:0/3939744793 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa9f80678c0 msgr2=0x7fa9f8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:41.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.475+0000 7faa1b4c8640 1 --2- 172.21.15.116:0/3939744793 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa9f80678c0 0x7fa9f8069d80 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7faa140c2560 tx=0x7faa100095c0 comp rx=0 tx=0).stop 2024-04-25T00:25:41.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.475+0000 7faa1b4c8640 1 -- 172.21.15.116:0/3939744793 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa140c55d0 msgr2=0x7faa140ab800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:41.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.475+0000 7faa1b4c8640 1 --2- 172.21.15.116:0/3939744793 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa140c55d0 0x7faa140ab800 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7faa040ae040 tx=0x7faa0401ab80 comp rx=0 tx=0).stop 2024-04-25T00:25:41.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.475+0000 7faa1b4c8640 1 -- 172.21.15.116:0/3939744793 shutdown_connections 2024-04-25T00:25:41.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.475+0000 7faa1b4c8640 1 --2- 172.21.15.116:0/3939744793 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa9f80678c0 0x7fa9f8069d80 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:41.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.475+0000 7faa1b4c8640 1 --2- 172.21.15.116:0/3939744793 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faa14155b20 0x7faa14155230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:41.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.475+0000 7faa1b4c8640 1 --2- 172.21.15.116:0/3939744793 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faa1414c700 0x7faa14143820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:41.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.475+0000 7faa1b4c8640 1 --2- 172.21.15.116:0/3939744793 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faa140c55d0 0x7faa140ab800 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:41.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.479+0000 7faa1b4c8640 1 -- 172.21.15.116:0/3939744793 >> 172.21.15.116:0/3939744793 conn(0x7faa1406c720 msgr2=0x7faa141605e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:41.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.483+0000 7faa1b4c8640 1 -- 172.21.15.116:0/3939744793 shutdown_connections 2024-04-25T00:25:41.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.483+0000 7faa1b4c8640 1 -- 172.21.15.116:0/3939744793 wait complete. 2024-04-25T00:25:41.536 INFO:tasks.mds_thrash.fs.[b]:mds.a down, removed from mdsmap 2024-04-25T00:25:41.536 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-04-25T00:25:41.536 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:25:41.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.883+0000 7f6b8f316640 1 Processor -- start 2024-04-25T00:25:41.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.883+0000 7f6b8f316640 1 -- start start 2024-04-25T00:25:41.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.887+0000 7f6b8f316640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b8000b0f0 0x7f6b800e2740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:41.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.887+0000 7f6b8f316640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6b800ded50 0x7f6b800df200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:41.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.887+0000 7f6b8f316640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b800de8a0 con 0x7f6b800dbd10 2024-04-25T00:25:41.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.887+0000 7f6b8f316640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f6b800e2c80 con 0x7f6b800ded50 2024-04-25T00:25:41.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.887+0000 7f6b8f316640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6b800df740 con 0x7f6b8000b0f0 2024-04-25T00:25:41.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.887+0000 7f6b8d0b2640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f6b800dbd10 0x7f6b80006ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:40608/0 (socket says 172.21.15.116:40608) 2024-04-25T00:25:41.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.887+0000 7f6b8d0b2640 1 -- 172.21.15.116:0/365963938 learned_addr learned my addr 172.21.15.116:0/365963938 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:25:41.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.887+0000 7f6b8d8b3640 1 --2- 172.21.15.116:0/365963938 >> v2:172.21.15.116:3300/0 conn(0x7f6b800ded50 0x7f6b800df200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:41.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.887+0000 7f6b8d8b3640 1 -- 172.21.15.116:0/365963938 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b8000b0f0 msgr2=0x7f6b800e2740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T00:25:41.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.887+0000 7f6b8d8b3640 1 --2- 172.21.15.116:0/365963938 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b8000b0f0 0x7f6b800e2740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:41.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.887+0000 7f6b8d8b3640 1 -- 172.21.15.116:0/365963938 >> v1:172.21.15.174:6789/0 conn(0x7f6b800dbd10 legacy=0x7f6b80006ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:41.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.887+0000 7f6b8d8b3640 1 -- 172.21.15.116:0/365963938 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b80047650 con 0x7f6b800ded50 2024-04-25T00:25:41.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.907+0000 7f6b8d8b3640 1 --2- 172.21.15.116:0/365963938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b800ded50 0x7f6b800df200 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f6b8802d710 tx=0x7f6b88097f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33c91ea062cfb450 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:41.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.907+0000 7f6b7ffff640 1 -- 172.21.15.116:0/365963938 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b88120860 con 0x7f6b800ded50 2024-04-25T00:25:41.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.907+0000 7f6b7ffff640 1 -- 172.21.15.116:0/365963938 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b88120a00 con 0x7f6b800ded50 2024-04-25T00:25:41.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.907+0000 7f6b7ffff640 1 -- 172.21.15.116:0/365963938 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b88128440 con 0x7f6b800ded50 2024-04-25T00:25:41.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.923+0000 7f6b8f316640 1 -- 172.21.15.116:0/365963938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b800ded50 msgr2=0x7f6b800df200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:41.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.923+0000 7f6b8f316640 1 --2- 172.21.15.116:0/365963938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b800ded50 0x7f6b800df200 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f6b8802d710 tx=0x7f6b88097f50 comp rx=0 tx=0).stop 2024-04-25T00:25:41.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.927+0000 7f6b8f316640 1 -- 172.21.15.116:0/365963938 shutdown_connections 2024-04-25T00:25:41.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.927+0000 7f6b8f316640 1 --2- 172.21.15.116:0/365963938 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b800ded50 0x7f6b800df200 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:41.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.927+0000 7f6b8f316640 1 --2- 172.21.15.116:0/365963938 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b8000b0f0 0x7f6b800e2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:41.928 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.927+0000 7f6b8f316640 1 -- 172.21.15.116:0/365963938 >> 172.21.15.116:0/365963938 conn(0x7f6b80048290 msgr2=0x7f6b800486a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:41.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.943+0000 7f6b8f316640 1 -- 172.21.15.116:0/365963938 shutdown_connections 2024-04-25T00:25:41.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.943+0000 7f6b8f316640 1 -- 172.21.15.116:0/365963938 wait complete. 2024-04-25T00:25:41.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.951+0000 7f6b8f316640 1 Processor -- start 2024-04-25T00:25:41.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.951+0000 7f6b8f316640 1 -- start start 2024-04-25T00:25:41.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.951+0000 7f6b8f316640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b8000b0f0 0x7f6b80180f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:41.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.951+0000 7f6b8f316640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b800dbd10 0x7f6b801800a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:41.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.951+0000 7f6b8f316640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b800ded50 0x7f6b801805e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:41.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.951+0000 7f6b8f316640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6b800dfc80 con 0x7f6b8000b0f0 2024-04-25T00:25:41.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.951+0000 7f6b8f316640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6b80047ef0 con 0x7f6b800dbd10 2024-04-25T00:25:41.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.951+0000 7f6b8f316640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6b800dc0f0 con 0x7f6b800ded50 2024-04-25T00:25:41.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.951+0000 7f6b8d8b3640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b800ded50 0x7f6b801805e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:41.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.951+0000 7f6b8d8b3640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b800ded50 0x7f6b801805e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:51598/0 (socket says 172.21.15.116:51598) 2024-04-25T00:25:41.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.951+0000 7f6b8d8b3640 1 -- 172.21.15.116:0/1087113258 learned_addr learned my addr 172.21.15.116:0/1087113258 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:25:41.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.951+0000 7f6b8d8b3640 1 -- 172.21.15.116:0/1087113258 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b800dbd10 msgr2=0x7f6b801800a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T00:25:41.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.951+0000 7f6b8d8b3640 1 --2- 172.21.15.116:0/1087113258 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b800dbd10 0x7f6b801800a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:41.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.951+0000 7f6b8d8b3640 1 -- 172.21.15.116:0/1087113258 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b8000b0f0 msgr2=0x7f6b80180f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:41.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.951+0000 7f6b8d8b3640 1 --2- 172.21.15.116:0/1087113258 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b8000b0f0 0x7f6b80180f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:41.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.951+0000 7f6b8d8b3640 1 -- 172.21.15.116:0/1087113258 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b800dd1a0 con 0x7f6b800ded50 2024-04-25T00:25:41.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.955+0000 7f6b8d8b3640 1 --2- 172.21.15.116:0/1087113258 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b800ded50 0x7f6b801805e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6b88050eb0 tx=0x7f6b88094d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:41.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.955+0000 7f6b7dffb640 1 -- 172.21.15.116:0/1087113258 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b88128750 con 0x7f6b800ded50 2024-04-25T00:25:41.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.955+0000 7f6b7dffb640 1 -- 172.21.15.116:0/1087113258 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b881288f0 con 0x7f6b800ded50 2024-04-25T00:25:41.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.955+0000 7f6b7dffb640 1 -- 172.21.15.116:0/1087113258 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b88128ca0 con 0x7f6b800ded50 2024-04-25T00:25:41.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.955+0000 7f6b8f316640 1 -- 172.21.15.116:0/1087113258 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b80180b20 con 0x7f6b800ded50 2024-04-25T00:25:41.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.955+0000 7f6b8f316640 1 -- 172.21.15.116:0/1087113258 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b8002e850 con 0x7f6b800ded50 2024-04-25T00:25:41.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.955+0000 7f6b8f316640 1 -- 172.21.15.116:0/1087113258 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b800053d0 con 0x7f6b800ded50 2024-04-25T00:25:41.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.955+0000 7f6b7dffb640 1 -- 172.21.15.116:0/1087113258 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6b8806e410 con 0x7f6b800ded50 2024-04-25T00:25:41.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.955+0000 7f6b7dffb640 1 --2- 172.21.15.116:0/1087113258 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6b74067a00 0x7f6b74069ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:41.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.955+0000 7f6b7dffb640 1 -- 172.21.15.116:0/1087113258 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(67..67 src has 1..67) ==== 10020+0+0 (secure 0 0 0) 0x7f6b8819b6a0 con 0x7f6b800ded50 2024-04-25T00:25:41.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.955+0000 7f6b8d0b2640 1 --2- 172.21.15.116:0/1087113258 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6b74067a00 0x7f6b74069ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:41.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.959+0000 7f6b8d0b2640 1 --2- 172.21.15.116:0/1087113258 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6b74067a00 0x7f6b74069ec0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f6b78001600 tx=0x7f6b78007a50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:41.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:41.963+0000 7f6b7dffb640 1 -- 172.21.15.116:0/1087113258 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f6b800dd1a0 con 0x7f6b800ded50 2024-04-25T00:25:42.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:42.763+0000 7f6b8f316640 1 -- 172.21.15.116:0/1087113258 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6b8003efc0 con 0x7f6b800ded50 2024-04-25T00:25:42.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:42.763+0000 7f6b7dffb640 1 -- 172.21.15.116:0/1087113258 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 203 v203) ==== 95+0+5673 (secure 0 0 0) 0x7f6b88091dd0 con 0x7f6b800ded50 2024-04-25T00:25:42.770 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:25:42.770 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":203,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":201,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:25:29.028925+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6413},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6413":{"gid":6413,"name":"d","rank":0,"incarnation":198,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/4181466530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":4181466530},{"type":"v1","addr":"172.21.15.174:6837","nonce":4181466530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6413,"qdb_cluster":[6413]},"id":1},{"mdsmap":{"epoch":203,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:25:40.724980+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8496":{"gid":8496,"name":"c","rank":0,"incarnation":203,"state":"up:replay","state_seq":1,"addr":"172.21.15.116:6833/3693049068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3693049068},{"type":"v1","addr":"172.21.15.116:6833","nonce":3693049068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:25:42.770 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 203 2024-04-25T00:25:42.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:42.775+0000 7f6b8f316640 1 -- 172.21.15.116:0/1087113258 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6b74067a00 msgr2=0x7f6b74069ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:42.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:42.775+0000 7f6b8f316640 1 --2- 172.21.15.116:0/1087113258 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6b74067a00 0x7f6b74069ec0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f6b78001600 tx=0x7f6b78007a50 comp rx=0 tx=0).stop 2024-04-25T00:25:42.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:42.775+0000 7f6b8f316640 1 -- 172.21.15.116:0/1087113258 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b800ded50 msgr2=0x7f6b801805e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:42.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:42.775+0000 7f6b8f316640 1 --2- 172.21.15.116:0/1087113258 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b800ded50 0x7f6b801805e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6b88050eb0 tx=0x7f6b88094d90 comp rx=0 tx=0).stop 2024-04-25T00:25:42.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:42.775+0000 7f6b8f316640 1 -- 172.21.15.116:0/1087113258 shutdown_connections 2024-04-25T00:25:42.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:42.775+0000 7f6b8f316640 1 --2- 172.21.15.116:0/1087113258 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6b74067a00 0x7f6b74069ec0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:42.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:42.775+0000 7f6b8f316640 1 --2- 172.21.15.116:0/1087113258 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b800ded50 0x7f6b801805e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:42.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:42.775+0000 7f6b8f316640 1 --2- 172.21.15.116:0/1087113258 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b800dbd10 0x7f6b801800a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:42.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:42.775+0000 7f6b8f316640 1 --2- 172.21.15.116:0/1087113258 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b8000b0f0 0x7f6b80180f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:42.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:42.775+0000 7f6b8f316640 1 -- 172.21.15.116:0/1087113258 >> 172.21.15.116:0/1087113258 conn(0x7f6b80048290 msgr2=0x7f6b800e8660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:42.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:42.775+0000 7f6b8f316640 1 -- 172.21.15.116:0/1087113258 shutdown_connections 2024-04-25T00:25:42.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:42.775+0000 7f6b8f316640 1 -- 172.21.15.116:0/1087113258 wait complete. 2024-04-25T00:25:42.885 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 203, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:04.678737+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 201, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6413": { "addr": "172.21.15.174:6837/4181466530", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6836", "nonce": 4181466530, "type": "v2" }, { "addr": "172.21.15.174:6837", "nonce": 4181466530, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6413, "incarnation": 198, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 18 } }, "last_failure": 0, "last_failure_osd_epoch": 66, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T00:25:29.028925+0000", "qdb_cluster": [ 6413 ], "qdb_leader": 6413, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6413 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:18.825887+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 203, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_8496": { "addr": "172.21.15.116:6833/3693049068", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6832", "nonce": 3693049068, "type": "v2" }, { "addr": "172.21.15.116:6833", "nonce": 3693049068, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8496, "incarnation": 203, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 67, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-04-25T00:25:40.724980+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8496 } } } ], "standbys": [ { "addr": "172.21.15.174:6835/4112145782", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6834", "nonce": 4112145782, "type": "v2" }, { "addr": "172.21.15.174:6835", "nonce": 4112145782, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 112, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5582, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-25T00:25:44.886 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:25:45.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.255+0000 7f6ba3671640 1 Processor -- start 2024-04-25T00:25:45.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.255+0000 7f6ba3671640 1 -- start start 2024-04-25T00:25:45.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.255+0000 7f6ba3671640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b9c14c700 0x7f6b9c0c5090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:45.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.255+0000 7f6ba3671640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6b9c155b20 0x7f6b9c152c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:45.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.255+0000 7f6ba0c0c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6b9c155b20 0x7f6b9c152c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:45.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.255+0000 7f6ba0c0c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6b9c155b20 0x7f6b9c152c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60404/0 (socket says 172.21.15.116:60404) 2024-04-25T00:25:45.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.255+0000 7f6ba140d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b9c14c700 0x7f6b9c0c5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:45.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.255+0000 7f6ba0c0c640 1 -- 172.21.15.116:0/1016031562 learned_addr learned my addr 172.21.15.116:0/1016031562 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:25:45.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.255+0000 7f6ba140d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b9c14c700 0x7f6b9c0c5090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:34852/0 (socket says 172.21.15.116:34852) 2024-04-25T00:25:45.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.255+0000 7f6ba3671640 1 -- 172.21.15.116:0/1016031562 --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b9c0a9e60 con 0x7f6b9c0c55d0 2024-04-25T00:25:45.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.259+0000 7f6ba3671640 1 -- 172.21.15.116:0/1016031562 --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f6b9c1527b0 con 0x7f6b9c155b20 2024-04-25T00:25:45.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.259+0000 7f6ba3671640 1 -- 172.21.15.116:0/1016031562 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6b9c0c3ef0 con 0x7f6b9c14c700 2024-04-25T00:25:45.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.259+0000 7f6ba0c0c640 1 -- 172.21.15.116:0/1016031562 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b9c14c700 msgr2=0x7f6b9c0c5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:45.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.259+0000 7f6ba0c0c640 1 --2- 172.21.15.116:0/1016031562 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b9c14c700 0x7f6b9c0c5090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:45.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.259+0000 7f6ba0c0c640 1 -- 172.21.15.116:0/1016031562 >> v1:172.21.15.174:6789/0 conn(0x7f6b9c0c55d0 legacy=0x7f6b9c071df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:45.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.259+0000 7f6ba0c0c640 1 -- 172.21.15.116:0/1016031562 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b9c0c41d0 con 0x7f6b9c155b20 2024-04-25T00:25:45.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.267+0000 7f6ba0c0c640 1 --2- 172.21.15.116:0/1016031562 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b9c155b20 0x7f6b9c152c40 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c152930 tx=0x7f6b8c01e670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b66c56822b9f7e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:45.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.267+0000 7f6ba140d640 1 --2- 172.21.15.116:0/1016031562 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b9c14c700 0x7f6b9c0c5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:25:45.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.267+0000 7f6b93fff640 1 -- 172.21.15.116:0/1016031562 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b8c0b2070 con 0x7f6b9c155b20 2024-04-25T00:25:45.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.267+0000 7f6b93fff640 1 -- 172.21.15.116:0/1016031562 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b8c004230 con 0x7f6b9c155b20 2024-04-25T00:25:45.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.267+0000 7f6b93fff640 1 -- 172.21.15.116:0/1016031562 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b8c003620 con 0x7f6b9c155b20 2024-04-25T00:25:45.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.271+0000 7f6ba3671640 1 -- 172.21.15.116:0/1016031562 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b9c155b20 msgr2=0x7f6b9c152c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:45.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.271+0000 7f6ba3671640 1 --2- 172.21.15.116:0/1016031562 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b9c155b20 0x7f6b9c152c40 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c152930 tx=0x7f6b8c01e670 comp rx=0 tx=0).stop 2024-04-25T00:25:45.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.271+0000 7f6ba3671640 1 -- 172.21.15.116:0/1016031562 shutdown_connections 2024-04-25T00:25:45.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.271+0000 7f6ba3671640 1 --2- 172.21.15.116:0/1016031562 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b9c155b20 0x7f6b9c152c40 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:45.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.271+0000 7f6ba3671640 1 --2- 172.21.15.116:0/1016031562 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b9c14c700 0x7f6b9c0c5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:45.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.271+0000 7f6ba3671640 1 -- 172.21.15.116:0/1016031562 >> 172.21.15.116:0/1016031562 conn(0x7f6b9c06c720 msgr2=0x7f6b9c06cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:45.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.271+0000 7f6ba3671640 1 -- 172.21.15.116:0/1016031562 shutdown_connections 2024-04-25T00:25:45.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.271+0000 7f6ba3671640 1 -- 172.21.15.116:0/1016031562 wait complete. 2024-04-25T00:25:45.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.275+0000 7f6ba3671640 1 Processor -- start 2024-04-25T00:25:45.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.275+0000 7f6ba3671640 1 -- start start 2024-04-25T00:25:45.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.275+0000 7f6ba3671640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b9c0c55d0 0x7f6b9c148140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:45.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.275+0000 7f6ba3671640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b9c14c700 0x7f6b9c14b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:45.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.275+0000 7f6ba3671640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b9c155b20 0x7f6b9c14ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:45.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.275+0000 7f6ba3671640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6b9c153510 con 0x7f6b9c0c55d0 2024-04-25T00:25:45.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.275+0000 7f6ba3671640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6b9c153390 con 0x7f6b9c155b20 2024-04-25T00:25:45.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.275+0000 7f6ba3671640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6b9c06b210 con 0x7f6b9c14c700 2024-04-25T00:25:45.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.275+0000 7f6ba140d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b9c0c55d0 0x7f6b9c148140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:45.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.275+0000 7f6ba140d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b9c0c55d0 0x7f6b9c148140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:60406/0 (socket says 172.21.15.116:60406) 2024-04-25T00:25:45.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.275+0000 7f6ba140d640 1 -- 172.21.15.116:0/3328323757 learned_addr learned my addr 172.21.15.116:0/3328323757 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:25:45.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.275+0000 7f6ba0c0c640 1 --2- 172.21.15.116:0/3328323757 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b9c14c700 0x7f6b9c14b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:45.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.275+0000 7f6ba0c0c640 1 -- 172.21.15.116:0/3328323757 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b9c155b20 msgr2=0x7f6b9c14ba90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T00:25:45.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.275+0000 7f6ba0c0c640 1 --2- 172.21.15.116:0/3328323757 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b9c155b20 0x7f6b9c14ba90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:45.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.275+0000 7f6ba0c0c640 1 -- 172.21.15.116:0/3328323757 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b9c0c55d0 msgr2=0x7f6b9c148140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:45.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.275+0000 7f6ba0c0c640 1 --2- 172.21.15.116:0/3328323757 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b9c0c55d0 0x7f6b9c148140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:45.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.275+0000 7f6ba0c0c640 1 -- 172.21.15.116:0/3328323757 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b9c06e3e0 con 0x7f6b9c14c700 2024-04-25T00:25:45.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.279+0000 7f6ba0c0c640 1 --2- 172.21.15.116:0/3328323757 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b9c14c700 0x7f6b9c14b550 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c004000 tx=0x7f6b8c01b410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:45.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.279+0000 7f6b91ffb640 1 -- 172.21.15.116:0/3328323757 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b8c0b2070 con 0x7f6b9c14c700 2024-04-25T00:25:45.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.279+0000 7f6b91ffb640 1 -- 172.21.15.116:0/3328323757 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b8c0bc070 con 0x7f6b9c14c700 2024-04-25T00:25:45.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.279+0000 7f6b91ffb640 1 -- 172.21.15.116:0/3328323757 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b8c0ad870 con 0x7f6b9c14c700 2024-04-25T00:25:45.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.279+0000 7f6ba3671640 1 -- 172.21.15.116:0/3328323757 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b9c152ce0 con 0x7f6b9c14c700 2024-04-25T00:25:45.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.279+0000 7f6ba3671640 1 -- 172.21.15.116:0/3328323757 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b9c1edef0 con 0x7f6b9c14c700 2024-04-25T00:25:45.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.283+0000 7f6ba3671640 1 -- 172.21.15.116:0/3328323757 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b9c068a60 con 0x7f6b9c14c700 2024-04-25T00:25:45.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.283+0000 7f6b91ffb640 1 -- 172.21.15.116:0/3328323757 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6b8c0ada10 con 0x7f6b9c14c700 2024-04-25T00:25:45.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.283+0000 7f6b91ffb640 1 --2- 172.21.15.116:0/3328323757 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6b800678c0 0x7f6b80069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:45.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.283+0000 7f6b91ffb640 1 -- 172.21.15.116:0/3328323757 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(67..67 src has 1..67) ==== 10020+0+0 (secure 0 0 0) 0x7f6b8c125550 con 0x7f6b9c14c700 2024-04-25T00:25:45.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.283+0000 7f6ba140d640 1 --2- 172.21.15.116:0/3328323757 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6b800678c0 0x7f6b80069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:45.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.283+0000 7f6ba140d640 1 --2- 172.21.15.116:0/3328323757 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6b800678c0 0x7f6b80069d80 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c151700 tx=0x7f6b98008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:45.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:45.291+0000 7f6b91ffb640 1 -- 172.21.15.116:0/3328323757 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f6b8c018380 con 0x7f6b9c14c700 2024-04-25T00:25:46.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:46.587+0000 7f6ba3671640 1 -- 172.21.15.116:0/3328323757 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6b9c0ad630 con 0x7f6b9c14c700 2024-04-25T00:25:46.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:46.591+0000 7f6b91ffb640 1 -- 172.21.15.116:0/3328323757 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 206 v206) ==== 95+0+5680 (secure 0 0 0) 0x7f6b8c0f4f80 con 0x7f6b9c14c700 2024-04-25T00:25:46.594 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:25:46.594 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":206,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":201,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:25:29.028925+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6413},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6413":{"gid":6413,"name":"d","rank":0,"incarnation":198,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/4181466530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":4181466530},{"type":"v1","addr":"172.21.15.174:6837","nonce":4181466530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6413,"qdb_cluster":[6413]},"id":1},{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:25:45.002412+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8496":{"gid":8496,"name":"c","rank":0,"incarnation":203,"state":"up:active","state_seq":6,"addr":"172.21.15.116:6833/3693049068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3693049068},{"type":"v1","addr":"172.21.15.116:6833","nonce":3693049068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8496,"qdb_cluster":[8496]},"id":2}]} 2024-04-25T00:25:46.595 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 206 2024-04-25T00:25:46.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:46.595+0000 7f6ba3671640 1 -- 172.21.15.116:0/3328323757 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6b800678c0 msgr2=0x7f6b80069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:46.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:46.599+0000 7f6ba3671640 1 --2- 172.21.15.116:0/3328323757 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6b800678c0 0x7f6b80069d80 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c151700 tx=0x7f6b98008040 comp rx=0 tx=0).stop 2024-04-25T00:25:46.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:46.599+0000 7f6ba3671640 1 -- 172.21.15.116:0/3328323757 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b9c14c700 msgr2=0x7f6b9c14b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:46.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:46.603+0000 7f6ba3671640 1 --2- 172.21.15.116:0/3328323757 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b9c14c700 0x7f6b9c14b550 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c004000 tx=0x7f6b8c01b410 comp rx=0 tx=0).stop 2024-04-25T00:25:46.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:46.603+0000 7f6ba3671640 1 -- 172.21.15.116:0/3328323757 shutdown_connections 2024-04-25T00:25:46.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:46.603+0000 7f6ba3671640 1 --2- 172.21.15.116:0/3328323757 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6b800678c0 0x7f6b80069d80 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:46.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:46.603+0000 7f6ba3671640 1 --2- 172.21.15.116:0/3328323757 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b9c155b20 0x7f6b9c14ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:46.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:46.615+0000 7f6ba3671640 1 --2- 172.21.15.116:0/3328323757 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b9c14c700 0x7f6b9c14b550 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:46.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:46.615+0000 7f6ba3671640 1 --2- 172.21.15.116:0/3328323757 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b9c0c55d0 0x7f6b9c148140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:46.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:46.615+0000 7f6ba3671640 1 -- 172.21.15.116:0/3328323757 >> 172.21.15.116:0/3328323757 conn(0x7f6b9c06c720 msgr2=0x7f6b9c071df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:46.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:46.619+0000 7f6ba3671640 1 -- 172.21.15.116:0/3328323757 shutdown_connections 2024-04-25T00:25:46.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:46.619+0000 7f6ba3671640 1 -- 172.21.15.116:0/3328323757 wait complete. 2024-04-25T00:25:46.679 INFO:tasks.mds_thrash.fs.[b]:mds.c has gained rank=0, replacing gid=8343 2024-04-25T00:25:46.679 INFO:tasks.mds_thrash.fs.[b]:waiting for 4 secs before reviving mds.a 2024-04-25T00:25:48.373 INFO:tasks.ceph.mds.c.smithi116.stderr:2024-04-25T00:25:48.375+0000 7f356e5dc640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T00:25:50.680 INFO:tasks.mds_thrash.fs.[b]:reviving mds.a 2024-04-25T00:25:50.680 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-25T00:25:50.680 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-25T00:25:50.684 INFO:tasks.ceph.mds.a:Started 2024-04-25T00:25:50.684 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:25:50.960 INFO:tasks.ceph.mds.a.smithi116.stdout:starting mds.a at 2024-04-25T00:25:51.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.103+0000 7fbc449e7640 1 Processor -- start 2024-04-25T00:25:51.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.103+0000 7fbc449e7640 1 -- start start 2024-04-25T00:25:51.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.103+0000 7fbc449e7640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbc400725e0 0x7fbc4015a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:51.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.103+0000 7fbc449e7640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fbc4015af70 0x7fbc4015b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:51.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.103+0000 7fbc449e7640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc40071d70 con 0x7fbc400c2c00 2024-04-25T00:25:51.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.103+0000 7fbc449e7640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fbc40070cf0 con 0x7fbc4015af70 2024-04-25T00:25:51.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.103+0000 7fbc449e7640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fbc40070fb0 con 0x7fbc400725e0 2024-04-25T00:25:51.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.103+0000 7fbc3e59c640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fbc400c2c00 0x7fbc40153700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:51498/0 (socket says 172.21.15.116:51498) 2024-04-25T00:25:51.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.103+0000 7fbc3e59c640 1 -- 172.21.15.116:0/2366917105 learned_addr learned my addr 172.21.15.116:0/2366917105 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:25:51.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.107+0000 7fbc3dd9b640 1 --2- 172.21.15.116:0/2366917105 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbc400725e0 0x7fbc4015a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:51.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.107+0000 7fbc3dd9b640 1 -- 172.21.15.116:0/2366917105 >> v2:172.21.15.174:3300/0 conn(0x7fbc4015af70 msgr2=0x7fbc4015b3c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T00:25:51.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.107+0000 7fbc3dd9b640 1 --2- 172.21.15.116:0/2366917105 >> v2:172.21.15.174:3300/0 conn(0x7fbc4015af70 0x7fbc4015b3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:51.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.107+0000 7fbc3dd9b640 1 -- 172.21.15.116:0/2366917105 >> v1:172.21.15.174:6789/0 conn(0x7fbc400c2c00 legacy=0x7fbc40153700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:51.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.107+0000 7fbc3dd9b640 1 -- 172.21.15.116:0/2366917105 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc40071290 con 0x7fbc400725e0 2024-04-25T00:25:51.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.119+0000 7fbc3dd9b640 1 --2- 172.21.15.116:0/2366917105 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbc400725e0 0x7fbc4015a4d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fbc38009d90 tx=0x7fbc3801c790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3d01d7af94242c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:51.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.119+0000 7fbc3d59a640 1 -- 172.21.15.116:0/2366917105 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc380a66b0 con 0x7fbc400725e0 2024-04-25T00:25:51.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.119+0000 7fbc3d59a640 1 -- 172.21.15.116:0/2366917105 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc380a6850 con 0x7fbc400725e0 2024-04-25T00:25:51.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.119+0000 7fbc449e7640 1 -- 172.21.15.116:0/2366917105 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbc400725e0 msgr2=0x7fbc4015a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:51.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.119+0000 7fbc449e7640 1 --2- 172.21.15.116:0/2366917105 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbc400725e0 0x7fbc4015a4d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fbc38009d90 tx=0x7fbc3801c790 comp rx=0 tx=0).stop 2024-04-25T00:25:51.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.123+0000 7fbc449e7640 1 -- 172.21.15.116:0/2366917105 shutdown_connections 2024-04-25T00:25:51.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.123+0000 7fbc449e7640 1 --2- 172.21.15.116:0/2366917105 >> v2:172.21.15.174:3300/0 conn(0x7fbc4015af70 0x7fbc4015b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:51.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.123+0000 7fbc449e7640 1 --2- 172.21.15.116:0/2366917105 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbc400725e0 0x7fbc4015a4d0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:51.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.123+0000 7fbc449e7640 1 -- 172.21.15.116:0/2366917105 >> 172.21.15.116:0/2366917105 conn(0x7fbc400c3a90 msgr2=0x7fbc400c3ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:51.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.143+0000 7fbc449e7640 1 -- 172.21.15.116:0/2366917105 shutdown_connections 2024-04-25T00:25:51.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.143+0000 7fbc449e7640 1 -- 172.21.15.116:0/2366917105 wait complete. 2024-04-25T00:25:51.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.147+0000 7fbc449e7640 1 Processor -- start 2024-04-25T00:25:51.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.147+0000 7fbc449e7640 1 -- start start 2024-04-25T00:25:51.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.147+0000 7fbc449e7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc400725e0 0x7fbc4014e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:51.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.147+0000 7fbc449e7640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbc400c2c00 0x7fbc40154810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:51.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.147+0000 7fbc449e7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc4015af70 0x7fbc4006dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:51.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.147+0000 7fbc449e7640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbc4015ba80 con 0x7fbc4015af70 2024-04-25T00:25:51.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.147+0000 7fbc449e7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbc4015b900 con 0x7fbc400725e0 2024-04-25T00:25:51.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.147+0000 7fbc449e7640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fbc400c2fe0 con 0x7fbc400c2c00 2024-04-25T00:25:51.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.147+0000 7fbc3dd9b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbc400c2c00 0x7fbc40154810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:51.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.147+0000 7fbc3dd9b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbc400c2c00 0x7fbc40154810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:34896/0 (socket says 172.21.15.116:34896) 2024-04-25T00:25:51.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.147+0000 7fbc3dd9b640 1 -- 172.21.15.116:0/330681956 learned_addr learned my addr 172.21.15.116:0/330681956 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:25:51.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.147+0000 7fbc3e59c640 1 --2- 172.21.15.116:0/330681956 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc400725e0 0x7fbc4014e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:51.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.147+0000 7fbc3dd9b640 1 -- 172.21.15.116:0/330681956 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc400725e0 msgr2=0x7fbc4014e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:51.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.147+0000 7fbc3dd9b640 1 --2- 172.21.15.116:0/330681956 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc400725e0 0x7fbc4014e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:51.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.147+0000 7fbc3dd9b640 1 -- 172.21.15.116:0/330681956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc4015af70 msgr2=0x7fbc4006dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:51.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.147+0000 7fbc3ed9d640 1 --2- 172.21.15.116:0/330681956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc4015af70 0x7fbc4006dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:51.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.147+0000 7fbc3dd9b640 1 --2- 172.21.15.116:0/330681956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc4015af70 0x7fbc4006dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:51.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.147+0000 7fbc3dd9b640 1 -- 172.21.15.116:0/330681956 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc400a9e70 con 0x7fbc400c2c00 2024-04-25T00:25:51.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.151+0000 7fbc3ed9d640 1 --2- 172.21.15.116:0/330681956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc4015af70 0x7fbc4006dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:25:51.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.151+0000 7fbc3dd9b640 1 --2- 172.21.15.116:0/330681956 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbc400c2c00 0x7fbc40154810 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fbc380b0040 tx=0x7fbc38019530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:51.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.151+0000 7fbc2effd640 1 -- 172.21.15.116:0/330681956 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc380be020 con 0x7fbc400c2c00 2024-04-25T00:25:51.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.151+0000 7fbc2effd640 1 -- 172.21.15.116:0/330681956 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc380ae9f0 con 0x7fbc400c2c00 2024-04-25T00:25:51.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.151+0000 7fbc2effd640 1 -- 172.21.15.116:0/330681956 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc380aedc0 con 0x7fbc400c2c00 2024-04-25T00:25:51.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.151+0000 7fbc449e7640 1 -- 172.21.15.116:0/330681956 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc400aa150 con 0x7fbc400c2c00 2024-04-25T00:25:51.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.151+0000 7fbc449e7640 1 -- 172.21.15.116:0/330681956 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc400aa710 con 0x7fbc400c2c00 2024-04-25T00:25:51.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.151+0000 7fbc449e7640 1 -- 172.21.15.116:0/330681956 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc400aa910 con 0x7fbc400c2c00 2024-04-25T00:25:51.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.159+0000 7fbc2effd640 1 -- 172.21.15.116:0/330681956 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbc380c1020 con 0x7fbc400c2c00 2024-04-25T00:25:51.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.159+0000 7fbc2effd640 1 --2- 172.21.15.116:0/330681956 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbc28067ae0 0x7fbc28069fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:51.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.163+0000 7fbc2effd640 1 -- 172.21.15.116:0/330681956 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(67..67 src has 1..67) ==== 10020+0+0 (secure 0 0 0) 0x7fbc38121fe0 con 0x7fbc400c2c00 2024-04-25T00:25:51.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.163+0000 7fbc3e59c640 1 --2- 172.21.15.116:0/330681956 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbc28067ae0 0x7fbc28069fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:51.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.167+0000 7fbc3e59c640 1 --2- 172.21.15.116:0/330681956 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbc28067ae0 0x7fbc28069fa0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fbc34000c10 tx=0x7fbc34009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:51.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.167+0000 7fbc2effd640 1 -- 172.21.15.116:0/330681956 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fbc400aa910 con 0x7fbc400c2c00 2024-04-25T00:25:51.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.799+0000 7fbc449e7640 1 -- 172.21.15.116:0/330681956 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbc4004f580 con 0x7fbc400c2c00 2024-04-25T00:25:51.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.799+0000 7fbc2effd640 1 -- 172.21.15.116:0/330681956 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 206 v206) ==== 95+0+5680 (secure 0 0 0) 0x7fbc380c8030 con 0x7fbc400c2c00 2024-04-25T00:25:51.801 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:25:51.801 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":206,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":201,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:25:29.028925+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6413},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6413":{"gid":6413,"name":"d","rank":0,"incarnation":198,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/4181466530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":4181466530},{"type":"v1","addr":"172.21.15.174:6837","nonce":4181466530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6413,"qdb_cluster":[6413]},"id":1},{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:25:45.002412+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8496":{"gid":8496,"name":"c","rank":0,"incarnation":203,"state":"up:active","state_seq":6,"addr":"172.21.15.116:6833/3693049068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3693049068},{"type":"v1","addr":"172.21.15.116:6833","nonce":3693049068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8496,"qdb_cluster":[8496]},"id":2}]} 2024-04-25T00:25:51.801 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 206 2024-04-25T00:25:51.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.807+0000 7fbc449e7640 1 -- 172.21.15.116:0/330681956 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbc28067ae0 msgr2=0x7fbc28069fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:51.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.807+0000 7fbc449e7640 1 --2- 172.21.15.116:0/330681956 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbc28067ae0 0x7fbc28069fa0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fbc34000c10 tx=0x7fbc34009040 comp rx=0 tx=0).stop 2024-04-25T00:25:51.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.807+0000 7fbc449e7640 1 -- 172.21.15.116:0/330681956 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbc400c2c00 msgr2=0x7fbc40154810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:51.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.807+0000 7fbc449e7640 1 --2- 172.21.15.116:0/330681956 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbc400c2c00 0x7fbc40154810 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fbc380b0040 tx=0x7fbc38019530 comp rx=0 tx=0).stop 2024-04-25T00:25:51.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.811+0000 7fbc449e7640 1 -- 172.21.15.116:0/330681956 shutdown_connections 2024-04-25T00:25:51.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.811+0000 7fbc449e7640 1 --2- 172.21.15.116:0/330681956 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbc28067ae0 0x7fbc28069fa0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:51.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.811+0000 7fbc449e7640 1 --2- 172.21.15.116:0/330681956 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbc4015af70 0x7fbc4006dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:51.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.811+0000 7fbc449e7640 1 --2- 172.21.15.116:0/330681956 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbc400c2c00 0x7fbc40154810 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:51.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.811+0000 7fbc449e7640 1 --2- 172.21.15.116:0/330681956 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbc400725e0 0x7fbc4014e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:51.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.811+0000 7fbc449e7640 1 -- 172.21.15.116:0/330681956 >> 172.21.15.116:0/330681956 conn(0x7fbc400c3a90 msgr2=0x7fbc40072be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:51.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.811+0000 7fbc449e7640 1 -- 172.21.15.116:0/330681956 shutdown_connections 2024-04-25T00:25:51.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:51.811+0000 7fbc449e7640 1 -- 172.21.15.116:0/330681956 wait complete. 2024-04-25T00:25:51.887 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2024-04-25T00:25:53.888 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:25:54.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:54.395+0000 7f4c295cf640 1 Processor -- start 2024-04-25T00:25:54.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:54.395+0000 7f4c295cf640 1 -- start start 2024-04-25T00:25:54.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:54.395+0000 7f4c295cf640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4c2408ba60 0x7f4c2415f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:54.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:54.395+0000 7f4c295cf640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c240c2bb0 con 0x7f4c2415fba0 2024-04-25T00:25:54.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:54.395+0000 7f4c295cf640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c240c2e90 con 0x7f4c24070e20 2024-04-25T00:25:54.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:54.395+0000 7f4c295cf640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f4c240c3730 con 0x7f4c2408ba60 2024-04-25T00:25:54.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:54.395+0000 7f4c23fff640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f4c24070e20 0x7f4c240c2190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:51506/0 (socket says 172.21.15.116:51506) 2024-04-25T00:25:54.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:54.395+0000 7f4c23fff640 1 -- 172.21.15.116:0/1518784453 learned_addr learned my addr 172.21.15.116:0/1518784453 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:25:54.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:54.395+0000 7f4c237fe640 1 --2- 172.21.15.116:0/1518784453 >> v2:172.21.15.116:3300/0 conn(0x7f4c2408ba60 0x7f4c2415f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:54.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:54.395+0000 7f4c22ffd640 1 -- 172.21.15.116:0/1518784453 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3603120532 0 0) 0x7f4c240c2e90 con 0x7f4c24070e20 2024-04-25T00:25:54.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:54.395+0000 7f4c22ffd640 1 -- 172.21.15.116:0/1518784453 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c0c003430 con 0x7f4c24070e20 2024-04-25T00:25:54.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:54.395+0000 7f4c237fe640 1 -- 172.21.15.116:0/1518784453 >> v1:172.21.15.174:6789/0 conn(0x7f4c24070e20 legacy=0x7f4c240c2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:54.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:54.395+0000 7f4c237fe640 1 -- 172.21.15.116:0/1518784453 >> v1:172.21.15.116:6789/0 conn(0x7f4c2415fba0 legacy=0x7f4c241628d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:54.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:54.399+0000 7f4c237fe640 1 -- 172.21.15.116:0/1518784453 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c240c3170 con 0x7f4c2408ba60 2024-04-25T00:25:54.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:54.403+0000 7f4c237fe640 1 --2- 172.21.15.116:0/1518784453 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c2408ba60 0x7f4c2415f660 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f4c140096c0 tx=0x7f4c1401c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56d0977a55d0d88f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:54.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:54.403+0000 7f4c22ffd640 1 -- 172.21.15.116:0/1518784453 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c140b0070 con 0x7f4c2408ba60 2024-04-25T00:25:54.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:54.403+0000 7f4c22ffd640 1 -- 172.21.15.116:0/1518784453 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c140ab400 con 0x7f4c2408ba60 2024-04-25T00:25:54.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:54.407+0000 7f4c295cf640 1 -- 172.21.15.116:0/1518784453 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c2408ba60 msgr2=0x7f4c2415f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:54.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:54.407+0000 7f4c295cf640 1 --2- 172.21.15.116:0/1518784453 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c2408ba60 0x7f4c2415f660 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f4c140096c0 tx=0x7f4c1401c330 comp rx=0 tx=0).stop 2024-04-25T00:25:54.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:54.419+0000 7f4c295cf640 1 -- 172.21.15.116:0/1518784453 shutdown_connections 2024-04-25T00:25:54.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:54.419+0000 7f4c295cf640 1 --2- 172.21.15.116:0/1518784453 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c2408ba60 0x7f4c2415f660 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:54.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:54.419+0000 7f4c295cf640 1 -- 172.21.15.116:0/1518784453 >> 172.21.15.116:0/1518784453 conn(0x7f4c240c6db0 msgr2=0x7f4c240c71c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:54.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:54.435+0000 7f4c295cf640 1 -- 172.21.15.116:0/1518784453 shutdown_connections 2024-04-25T00:25:54.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:54.435+0000 7f4c295cf640 1 -- 172.21.15.116:0/1518784453 wait complete. 2024-04-25T00:25:54.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:54.435+0000 7f4c295cf640 1 Processor -- start 2024-04-25T00:25:54.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:54.435+0000 7f4c295cf640 1 -- start start 2024-04-25T00:25:54.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:54.443+0000 7f4c295cf640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4c24070e20 0x7f4c240c5f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:54.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:54.443+0000 7f4c295cf640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4c2408ba60 0x7f4c2408a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:54.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:54.443+0000 7f4c295cf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c2415fba0 0x7f4c2408a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:54.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:54.443+0000 7f4c295cf640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4c2416e1c0 con 0x7f4c2415fba0 2024-04-25T00:25:54.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:54.443+0000 7f4c295cf640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4c2416e040 con 0x7f4c24070e20 2024-04-25T00:25:54.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:54.443+0000 7f4c295cf640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f4c2416e340 con 0x7f4c2408ba60 2024-04-25T00:25:54.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:54.443+0000 7f4c28dce640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c2415fba0 0x7f4c2408a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:54.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:54.443+0000 7f4c237fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4c2408ba60 0x7f4c2408a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:54.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:54.443+0000 7f4c237fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4c2408ba60 0x7f4c2408a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:34906/0 (socket says 172.21.15.116:34906) 2024-04-25T00:25:54.442 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:54.443+0000 7f4c237fe640 1 -- 172.21.15.116:0/3526627199 learned_addr learned my addr 172.21.15.116:0/3526627199 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:25:54.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:54.443+0000 7f4c28dce640 1 -- 172.21.15.116:0/3526627199 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4c2408ba60 msgr2=0x7f4c2408a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:54.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:54.443+0000 7f4c28dce640 1 --2- 172.21.15.116:0/3526627199 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4c2408ba60 0x7f4c2408a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:54.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:54.443+0000 7f4c28dce640 1 -- 172.21.15.116:0/3526627199 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4c24070e20 msgr2=0x7f4c240c5f90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T00:25:54.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:54.443+0000 7f4c28dce640 1 --2- 172.21.15.116:0/3526627199 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4c24070e20 0x7f4c240c5f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:54.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:54.443+0000 7f4c28dce640 1 -- 172.21.15.116:0/3526627199 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c24163720 con 0x7f4c2415fba0 2024-04-25T00:25:54.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:54.443+0000 7f4c28dce640 1 --2- 172.21.15.116:0/3526627199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c2415fba0 0x7f4c2408a5f0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f4c1800c3c0 tx=0x7f4c1800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:54.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:54.443+0000 7f4c20ff9640 1 -- 172.21.15.116:0/3526627199 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c180052d0 con 0x7f4c2415fba0 2024-04-25T00:25:54.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:54.443+0000 7f4c20ff9640 1 -- 172.21.15.116:0/3526627199 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c18005470 con 0x7f4c2415fba0 2024-04-25T00:25:54.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:54.443+0000 7f4c295cf640 1 -- 172.21.15.116:0/3526627199 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c2420cdb0 con 0x7f4c2415fba0 2024-04-25T00:25:54.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:54.443+0000 7f4c295cf640 1 -- 172.21.15.116:0/3526627199 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c240ac190 con 0x7f4c2415fba0 2024-04-25T00:25:54.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:54.447+0000 7f4c20ff9640 1 -- 172.21.15.116:0/3526627199 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c18007260 con 0x7f4c2415fba0 2024-04-25T00:25:54.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:54.447+0000 7f4c20ff9640 1 -- 172.21.15.116:0/3526627199 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4c180074c0 con 0x7f4c2415fba0 2024-04-25T00:25:54.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:54.447+0000 7f4c20ff9640 1 --2- 172.21.15.116:0/3526627199 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4c10067940 0x7f4c10069e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:54.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:54.447+0000 7f4c23fff640 1 --2- 172.21.15.116:0/3526627199 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4c10067940 0x7f4c10069e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:54.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:54.447+0000 7f4c20ff9640 1 -- 172.21.15.116:0/3526627199 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 10020+0+0 (secure 0 0 0) 0x7f4c180534e0 con 0x7f4c2415fba0 2024-04-25T00:25:54.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:54.447+0000 7f4c295cf640 1 -- 172.21.15.116:0/3526627199 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c040052a0 con 0x7f4c2415fba0 2024-04-25T00:25:54.465 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:54.467+0000 7f4c20ff9640 1 -- 172.21.15.116:0/3526627199 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f4c18055b40 con 0x7f4c2415fba0 2024-04-25T00:25:54.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:54.471+0000 7f4c23fff640 1 --2- 172.21.15.116:0/3526627199 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4c10067940 0x7f4c10069e00 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c0012f0 tx=0x7f4c1c001430 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:55.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:55.371+0000 7f4c295cf640 1 -- 172.21.15.116:0/3526627199 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4c04005860 con 0x7f4c2415fba0 2024-04-25T00:25:55.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:55.375+0000 7f4c20ff9640 1 -- 172.21.15.116:0/3526627199 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 207 v207) ==== 95+0+6509 (secure 0 0 0) 0x7f4c18055440 con 0x7f4c2415fba0 2024-04-25T00:25:55.376 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:25:55.376 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":207,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6485,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/2371126056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2371126056},{"type":"v1","addr":"172.21.15.116:6835","nonce":2371126056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":207}],"filesystems":[{"mdsmap":{"epoch":201,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:25:29.028925+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6413},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6413":{"gid":6413,"name":"d","rank":0,"incarnation":198,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/4181466530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":4181466530},{"type":"v1","addr":"172.21.15.174:6837","nonce":4181466530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6413,"qdb_cluster":[6413]},"id":1},{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:25:45.002412+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8496":{"gid":8496,"name":"c","rank":0,"incarnation":203,"state":"up:active","state_seq":6,"addr":"172.21.15.116:6833/3693049068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3693049068},{"type":"v1","addr":"172.21.15.116:6833","nonce":3693049068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8496,"qdb_cluster":[8496]},"id":2}]} 2024-04-25T00:25:55.376 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 207 2024-04-25T00:25:55.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:55.395+0000 7f4c295cf640 1 -- 172.21.15.116:0/3526627199 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4c10067940 msgr2=0x7f4c10069e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:55.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:55.395+0000 7f4c295cf640 1 --2- 172.21.15.116:0/3526627199 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4c10067940 0x7f4c10069e00 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c0012f0 tx=0x7f4c1c001430 comp rx=0 tx=0).stop 2024-04-25T00:25:55.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:55.395+0000 7f4c295cf640 1 -- 172.21.15.116:0/3526627199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c2415fba0 msgr2=0x7f4c2408a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:55.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:55.395+0000 7f4c295cf640 1 --2- 172.21.15.116:0/3526627199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c2415fba0 0x7f4c2408a5f0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f4c1800c3c0 tx=0x7f4c1800c880 comp rx=0 tx=0).stop 2024-04-25T00:25:55.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:55.395+0000 7f4c295cf640 1 -- 172.21.15.116:0/3526627199 shutdown_connections 2024-04-25T00:25:55.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:55.395+0000 7f4c295cf640 1 --2- 172.21.15.116:0/3526627199 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4c10067940 0x7f4c10069e00 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:55.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:55.395+0000 7f4c295cf640 1 --2- 172.21.15.116:0/3526627199 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4c2415fba0 0x7f4c2408a5f0 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:55.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:55.395+0000 7f4c295cf640 1 --2- 172.21.15.116:0/3526627199 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4c2408ba60 0x7f4c2408a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:55.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:55.395+0000 7f4c295cf640 1 --2- 172.21.15.116:0/3526627199 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4c24070e20 0x7f4c240c5f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:55.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:55.395+0000 7f4c295cf640 1 -- 172.21.15.116:0/3526627199 >> 172.21.15.116:0/3526627199 conn(0x7f4c240c6db0 msgr2=0x7f4c2416cfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:55.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:55.395+0000 7f4c295cf640 1 -- 172.21.15.116:0/3526627199 shutdown_connections 2024-04-25T00:25:55.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:55.395+0000 7f4c295cf640 1 -- 172.21.15.116:0/3526627199 wait complete. 2024-04-25T00:25:55.449 INFO:tasks.mds_thrash.fs.[b]:mds.a reported in up:standby state 2024-04-25T00:25:55.449 INFO:tasks.mds_thrash.fs.[b]:waiting for 35 secs before thrashing 2024-04-25T00:25:57.492 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:25:57.839 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:57.839+0000 7f8f3fef5640 1 Processor -- start 2024-04-25T00:25:57.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:57.839+0000 7f8f3fef5640 1 -- start start 2024-04-25T00:25:57.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:57.839+0000 7f8f3fef5640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f8f3813a360 0x7f8f380c29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:57.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:57.839+0000 7f8f3fef5640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8f380c2ee0 0x7f8f38071180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:57.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:57.839+0000 7f8f3fef5640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f38073320 con 0x7f8f3813acb0 2024-04-25T00:25:57.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:57.839+0000 7f8f3fef5640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f8f38070cf0 con 0x7f8f380c2ee0 2024-04-25T00:25:57.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:57.839+0000 7f8f3fef5640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f8f38073040 con 0x7f8f3813a360 2024-04-25T00:25:57.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:57.839+0000 7f8f3e492640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f8f3813acb0 0x7f8f380716c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:53232/0 (socket says 172.21.15.116:53232) 2024-04-25T00:25:57.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:57.839+0000 7f8f3d490640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8f380c2ee0 0x7f8f38071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:57.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:57.839+0000 7f8f3d490640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8f380c2ee0 0x7f8f38071180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45538/0 (socket says 172.21.15.116:45538) 2024-04-25T00:25:57.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:57.839+0000 7f8f3d490640 1 -- 172.21.15.116:0/2507089526 learned_addr learned my addr 172.21.15.116:0/2507089526 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:25:57.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:57.839+0000 7f8f3dc91640 1 --2- 172.21.15.116:0/2507089526 >> v2:172.21.15.174:3300/0 conn(0x7f8f3813a360 0x7f8f380c29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:57.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:57.843+0000 7f8f3dc91640 1 -- 172.21.15.116:0/2507089526 >> v2:172.21.15.116:3300/0 conn(0x7f8f380c2ee0 msgr2=0x7f8f38071180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:57.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:57.843+0000 7f8f3dc91640 1 --2- 172.21.15.116:0/2507089526 >> v2:172.21.15.116:3300/0 conn(0x7f8f380c2ee0 0x7f8f38071180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:57.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:57.843+0000 7f8f3dc91640 1 -- 172.21.15.116:0/2507089526 >> v1:172.21.15.116:6789/0 conn(0x7f8f3813acb0 legacy=0x7f8f380716c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:57.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:57.843+0000 7f8f3dc91640 1 -- 172.21.15.116:0/2507089526 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f381557e0 con 0x7f8f3813a360 2024-04-25T00:25:57.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:57.847+0000 7f8f3dc91640 1 --2- 172.21.15.116:0/2507089526 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8f3813a360 0x7f8f380c29a0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f8f380c7f80 tx=0x7f8f3001d930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6cd4f2b46dc23d8e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:57.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:57.847+0000 7f8f3cc8f640 1 -- 172.21.15.116:0/2507089526 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f3000a0d0 con 0x7f8f3813a360 2024-04-25T00:25:57.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:57.847+0000 7f8f3cc8f640 1 -- 172.21.15.116:0/2507089526 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f3000a270 con 0x7f8f3813a360 2024-04-25T00:25:57.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:57.847+0000 7f8f3cc8f640 1 -- 172.21.15.116:0/2507089526 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f3000a620 con 0x7f8f3813a360 2024-04-25T00:25:57.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:57.847+0000 7f8f3fef5640 1 -- 172.21.15.116:0/2507089526 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8f3813a360 msgr2=0x7f8f380c29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:57.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:57.847+0000 7f8f3fef5640 1 --2- 172.21.15.116:0/2507089526 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8f3813a360 0x7f8f380c29a0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f8f380c7f80 tx=0x7f8f3001d930 comp rx=0 tx=0).stop 2024-04-25T00:25:57.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:57.847+0000 7f8f3fef5640 1 -- 172.21.15.116:0/2507089526 shutdown_connections 2024-04-25T00:25:57.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:57.847+0000 7f8f3fef5640 1 --2- 172.21.15.116:0/2507089526 >> v2:172.21.15.116:3300/0 conn(0x7f8f380c2ee0 0x7f8f38071180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:57.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:57.847+0000 7f8f3fef5640 1 --2- 172.21.15.116:0/2507089526 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8f3813a360 0x7f8f380c29a0 secure :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f8f380c7f80 tx=0x7f8f3001d930 comp rx=0 tx=0).stop 2024-04-25T00:25:57.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:57.847+0000 7f8f3fef5640 1 -- 172.21.15.116:0/2507089526 >> 172.21.15.116:0/2507089526 conn(0x7f8f380c3a90 msgr2=0x7f8f380c3ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:57.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:57.851+0000 7f8f3fef5640 1 -- 172.21.15.116:0/2507089526 shutdown_connections 2024-04-25T00:25:57.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:57.855+0000 7f8f3fef5640 1 -- 172.21.15.116:0/2507089526 wait complete. 2024-04-25T00:25:57.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:57.855+0000 7f8f3fef5640 1 Processor -- start 2024-04-25T00:25:57.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:57.855+0000 7f8f3fef5640 1 -- start start 2024-04-25T00:25:57.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:57.859+0000 7f8f3fef5640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8f380c2ee0 0x7f8f38153de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:57.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:57.859+0000 7f8f3fef5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f3813acb0 0x7f8f38132580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:57.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:57.859+0000 7f8f3fef5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8f3814ee60 0x7f8f38132ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:57.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:57.859+0000 7f8f3fef5640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8f3815f080 con 0x7f8f3813acb0 2024-04-25T00:25:57.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:57.859+0000 7f8f3fef5640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8f3815ef00 con 0x7f8f3814ee60 2024-04-25T00:25:57.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:57.859+0000 7f8f3fef5640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f8f3815f200 con 0x7f8f380c2ee0 2024-04-25T00:25:57.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:57.859+0000 7f8f3d490640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f3813acb0 0x7f8f38132580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:57.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:57.859+0000 7f8f3d490640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f3813acb0 0x7f8f38132580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45550/0 (socket says 172.21.15.116:45550) 2024-04-25T00:25:57.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:57.859+0000 7f8f3d490640 1 -- 172.21.15.116:0/1617838146 learned_addr learned my addr 172.21.15.116:0/1617838146 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:25:57.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:57.859+0000 7f8f3d490640 1 -- 172.21.15.116:0/1617838146 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8f380c2ee0 msgr2=0x7f8f38153de0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T00:25:57.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:57.859+0000 7f8f3d490640 1 --2- 172.21.15.116:0/1617838146 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8f380c2ee0 0x7f8f38153de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:57.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:57.859+0000 7f8f3d490640 1 -- 172.21.15.116:0/1617838146 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8f3814ee60 msgr2=0x7f8f38132ac0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T00:25:57.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:57.859+0000 7f8f3d490640 1 --2- 172.21.15.116:0/1617838146 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8f3814ee60 0x7f8f38132ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:57.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:57.859+0000 7f8f3d490640 1 -- 172.21.15.116:0/1617838146 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f38155520 con 0x7f8f3813acb0 2024-04-25T00:25:57.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:57.859+0000 7f8f3d490640 1 --2- 172.21.15.116:0/1617838146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f3813acb0 0x7f8f38132580 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f8f2800e7f0 tx=0x7f8f2800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:57.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:57.859+0000 7f8f2e7fc640 1 -- 172.21.15.116:0/1617838146 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f280135a0 con 0x7f8f3813acb0 2024-04-25T00:25:57.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:57.859+0000 7f8f2e7fc640 1 -- 172.21.15.116:0/1617838146 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f2800d740 con 0x7f8f3813acb0 2024-04-25T00:25:57.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:57.863+0000 7f8f3fef5640 1 -- 172.21.15.116:0/1617838146 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f380aab50 con 0x7f8f3813acb0 2024-04-25T00:25:57.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:57.863+0000 7f8f3fef5640 1 -- 172.21.15.116:0/1617838146 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f38150800 con 0x7f8f3813acb0 2024-04-25T00:25:57.862 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:57.863+0000 7f8f2e7fc640 1 -- 172.21.15.116:0/1617838146 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f2800db30 con 0x7f8f3813acb0 2024-04-25T00:25:57.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:57.867+0000 7f8f3fef5640 1 -- 172.21.15.116:0/1617838146 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f3804f580 con 0x7f8f3813acb0 2024-04-25T00:25:57.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:57.867+0000 7f8f2e7fc640 1 -- 172.21.15.116:0/1617838146 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8f2800dcd0 con 0x7f8f3813acb0 2024-04-25T00:25:57.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:57.867+0000 7f8f2e7fc640 1 --2- 172.21.15.116:0/1617838146 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8f24067ad0 0x7f8f24069f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:57.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:57.867+0000 7f8f2e7fc640 1 -- 172.21.15.116:0/1617838146 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 10020+0+0 (secure 0 0 0) 0x7f8f2808f7f0 con 0x7f8f3813acb0 2024-04-25T00:25:57.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:57.867+0000 7f8f3dc91640 1 --2- 172.21.15.116:0/1617838146 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8f24067ad0 0x7f8f24069f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:57.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:57.871+0000 7f8f3dc91640 1 --2- 172.21.15.116:0/1617838146 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8f24067ad0 0x7f8f24069f90 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f8f300ae000 tx=0x7f8f300a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:57.883 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:57.883+0000 7f8f2e7fc640 1 -- 172.21.15.116:0/1617838146 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f8f2805f2b0 con 0x7f8f3813acb0 2024-04-25T00:25:58.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:58.375+0000 7f8f3fef5640 1 -- 172.21.15.116:0/1617838146 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8f380c13c0 con 0x7f8f3813acb0 2024-04-25T00:25:58.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:58.375+0000 7f8f2e7fc640 1 -- 172.21.15.116:0/1617838146 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 207 v207) ==== 95+0+6509 (secure 0 0 0) 0x7f8f2805ebb0 con 0x7f8f3813acb0 2024-04-25T00:25:58.376 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:25:58.376 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":207,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6485,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/2371126056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2371126056},{"type":"v1","addr":"172.21.15.116:6835","nonce":2371126056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":207}],"filesystems":[{"mdsmap":{"epoch":201,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:25:29.028925+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6413},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6413":{"gid":6413,"name":"d","rank":0,"incarnation":198,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/4181466530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":4181466530},{"type":"v1","addr":"172.21.15.174:6837","nonce":4181466530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6413,"qdb_cluster":[6413]},"id":1},{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:25:45.002412+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8496":{"gid":8496,"name":"c","rank":0,"incarnation":203,"state":"up:active","state_seq":6,"addr":"172.21.15.116:6833/3693049068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3693049068},{"type":"v1","addr":"172.21.15.116:6833","nonce":3693049068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8496,"qdb_cluster":[8496]},"id":2}]} 2024-04-25T00:25:58.377 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 207 2024-04-25T00:25:58.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:58.395+0000 7f8f3fef5640 1 -- 172.21.15.116:0/1617838146 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8f24067ad0 msgr2=0x7f8f24069f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:58.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:58.395+0000 7f8f3fef5640 1 --2- 172.21.15.116:0/1617838146 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8f24067ad0 0x7f8f24069f90 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f8f300ae000 tx=0x7f8f300a7070 comp rx=0 tx=0).stop 2024-04-25T00:25:58.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:58.395+0000 7f8f3fef5640 1 -- 172.21.15.116:0/1617838146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f3813acb0 msgr2=0x7f8f38132580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:58.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:58.395+0000 7f8f3fef5640 1 --2- 172.21.15.116:0/1617838146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f3813acb0 0x7f8f38132580 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f8f2800e7f0 tx=0x7f8f2800ecb0 comp rx=0 tx=0).stop 2024-04-25T00:25:58.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:58.395+0000 7f8f3fef5640 1 -- 172.21.15.116:0/1617838146 shutdown_connections 2024-04-25T00:25:58.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:58.395+0000 7f8f3fef5640 1 --2- 172.21.15.116:0/1617838146 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8f24067ad0 0x7f8f24069f90 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:58.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:58.395+0000 7f8f3fef5640 1 --2- 172.21.15.116:0/1617838146 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8f3814ee60 0x7f8f38132ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:58.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:58.395+0000 7f8f3fef5640 1 --2- 172.21.15.116:0/1617838146 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8f3813acb0 0x7f8f38132580 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:58.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:58.399+0000 7f8f3fef5640 1 --2- 172.21.15.116:0/1617838146 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8f380c2ee0 0x7f8f38153de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:58.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:58.399+0000 7f8f3fef5640 1 -- 172.21.15.116:0/1617838146 >> 172.21.15.116:0/1617838146 conn(0x7f8f380c3a90 msgr2=0x7f8f38138490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:58.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:58.415+0000 7f8f3fef5640 1 -- 172.21.15.116:0/1617838146 shutdown_connections 2024-04-25T00:25:58.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:58.415+0000 7f8f3fef5640 1 -- 172.21.15.116:0/1617838146 wait complete. 2024-04-25T00:25:58.482 INFO:tasks.mds_thrash.fs.[a]:kill mds.d (rank=0) 2024-04-25T00:25:58.483 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-25T00:25:58.483 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:25:58.507 INFO:tasks.ceph.mds.d:Stopped 2024-04-25T00:25:58.507 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:25:59.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:59.075+0000 7f384759e640 1 Processor -- start 2024-04-25T00:25:59.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:59.075+0000 7f384759e640 1 -- start start 2024-04-25T00:25:59.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:59.075+0000 7f384759e640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f38480c2190 0x7f38480c2b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:59.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:59.075+0000 7f384759e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f384815da60 0x7f38480716f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:59.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:59.075+0000 7f384759e640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3848157590 con 0x7f38481571b0 2024-04-25T00:25:59.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:59.075+0000 7f384759e640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f38480c3180 con 0x7f38480c2190 2024-04-25T00:25:59.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:59.075+0000 7f384759e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f38480c3460 con 0x7f384815da60 2024-04-25T00:25:59.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:59.079+0000 7f3846d9d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f384815da60 0x7f38480716f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:59.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:59.079+0000 7f3846d9d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f384815da60 0x7f38480716f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:47852/0 (socket says 172.21.15.116:47852) 2024-04-25T00:25:59.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:59.079+0000 7f3846d9d640 1 -- 172.21.15.116:0/2728278137 learned_addr learned my addr 172.21.15.116:0/2728278137 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:25:59.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:59.079+0000 7f3845d9b640 1 --2- 172.21.15.116:0/2728278137 >> v2:172.21.15.174:3300/0 conn(0x7f38480c2190 0x7f38480c2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:59.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:59.079+0000 7f3845d9b640 1 -- 172.21.15.116:0/2728278137 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f384815da60 msgr2=0x7f38480716f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:25:59.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:59.079+0000 7f3845d9b640 1 --2- 172.21.15.116:0/2728278137 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f384815da60 0x7f38480716f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:59.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:59.079+0000 7f3845d9b640 1 -- 172.21.15.116:0/2728278137 >> v1:172.21.15.174:6789/0 conn(0x7f38481571b0 legacy=0x7f384806dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:59.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:59.079+0000 7f3845d9b640 1 -- 172.21.15.116:0/2728278137 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f384815d260 con 0x7f38480c2190 2024-04-25T00:25:59.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:59.083+0000 7f3845d9b640 1 --2- 172.21.15.116:0/2728278137 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f38480c2190 0x7f38480c2b10 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f38380097c0 tx=0x7f383801c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb4948f44387538a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:59.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:59.087+0000 7f384559a640 1 -- 172.21.15.116:0/2728278137 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38380b0070 con 0x7f38480c2190 2024-04-25T00:25:59.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:59.087+0000 7f384559a640 1 -- 172.21.15.116:0/2728278137 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38380ab600 con 0x7f38480c2190 2024-04-25T00:25:59.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:59.087+0000 7f384759e640 1 -- 172.21.15.116:0/2728278137 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f38480c2190 msgr2=0x7f38480c2b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:59.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:59.087+0000 7f384759e640 1 --2- 172.21.15.116:0/2728278137 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f38480c2190 0x7f38480c2b10 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f38380097c0 tx=0x7f383801c3a0 comp rx=0 tx=0).stop 2024-04-25T00:25:59.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:59.087+0000 7f384759e640 1 -- 172.21.15.116:0/2728278137 shutdown_connections 2024-04-25T00:25:59.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:59.087+0000 7f384759e640 1 --2- 172.21.15.116:0/2728278137 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f384815da60 0x7f38480716f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:59.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:59.087+0000 7f384759e640 1 --2- 172.21.15.116:0/2728278137 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f38480c2190 0x7f38480c2b10 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:59.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:59.087+0000 7f384759e640 1 -- 172.21.15.116:0/2728278137 >> 172.21.15.116:0/2728278137 conn(0x7f38480c6db0 msgr2=0x7f38480c71c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:25:59.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:59.099+0000 7f384759e640 1 -- 172.21.15.116:0/2728278137 shutdown_connections 2024-04-25T00:25:59.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:59.103+0000 7f384759e640 1 -- 172.21.15.116:0/2728278137 wait complete. 2024-04-25T00:25:59.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:59.103+0000 7f384759e640 1 Processor -- start 2024-04-25T00:25:59.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:59.103+0000 7f384759e640 1 -- start start 2024-04-25T00:25:59.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:59.107+0000 7f384759e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38480c2190 0x7f38480aa370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:59.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:59.107+0000 7f384759e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f38481571b0 0x7f384815b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:59.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:59.107+0000 7f384759e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f384815da60 0x7f38480abfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:59.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:59.107+0000 7f384759e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3848164de0 con 0x7f38480c2190 2024-04-25T00:25:59.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:59.107+0000 7f384759e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3848164c60 con 0x7f38481571b0 2024-04-25T00:25:59.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:59.107+0000 7f384759e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f3848164f60 con 0x7f384815da60 2024-04-25T00:25:59.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:59.107+0000 7f3845d9b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f38481571b0 0x7f384815b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:59.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:59.107+0000 7f3846d9d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f384815da60 0x7f38480abfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:59.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:59.107+0000 7f3845d9b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f38481571b0 0x7f384815b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:57662/0 (socket says 172.21.15.116:57662) 2024-04-25T00:25:59.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:59.107+0000 7f3845d9b640 1 -- 172.21.15.116:0/3079258476 learned_addr learned my addr 172.21.15.116:0/3079258476 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:25:59.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:59.107+0000 7f384659c640 1 --2- 172.21.15.116:0/3079258476 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38480c2190 0x7f38480aa370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:59.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:59.107+0000 7f3845d9b640 1 -- 172.21.15.116:0/3079258476 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f384815da60 msgr2=0x7f38480abfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:59.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:59.107+0000 7f3845d9b640 1 --2- 172.21.15.116:0/3079258476 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f384815da60 0x7f38480abfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:59.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:59.107+0000 7f3845d9b640 1 -- 172.21.15.116:0/3079258476 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38480c2190 msgr2=0x7f38480aa370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:25:59.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:59.107+0000 7f3845d9b640 1 --2- 172.21.15.116:0/3079258476 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38480c2190 0x7f38480aa370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:25:59.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:59.107+0000 7f3845d9b640 1 -- 172.21.15.116:0/3079258476 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38480c5d40 con 0x7f38481571b0 2024-04-25T00:25:59.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:59.107+0000 7f384659c640 1 --2- 172.21.15.116:0/3079258476 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38480c2190 0x7f38480aa370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:25:59.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:59.107+0000 7f3845d9b640 1 --2- 172.21.15.116:0/3079258476 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f38481571b0 0x7f384815b020 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f38380ae040 tx=0x7f3838019140 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:59.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:59.107+0000 7f3836ffd640 1 -- 172.21.15.116:0/3079258476 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38380b8020 con 0x7f38481571b0 2024-04-25T00:25:59.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:59.107+0000 7f384759e640 1 -- 172.21.15.116:0/3079258476 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38480c6020 con 0x7f38481571b0 2024-04-25T00:25:59.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:59.107+0000 7f384759e640 1 -- 172.21.15.116:0/3079258476 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38480a9e50 con 0x7f38481571b0 2024-04-25T00:25:59.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:59.111+0000 7f3836ffd640 1 -- 172.21.15.116:0/3079258476 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38380aaa90 con 0x7f38481571b0 2024-04-25T00:25:59.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:59.111+0000 7f3836ffd640 1 -- 172.21.15.116:0/3079258476 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38380b0040 con 0x7f38481571b0 2024-04-25T00:25:59.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:59.111+0000 7f384759e640 1 -- 172.21.15.116:0/3079258476 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f382c0052a0 con 0x7f38481571b0 2024-04-25T00:25:59.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:59.111+0000 7f3836ffd640 1 -- 172.21.15.116:0/3079258476 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f38380bb020 con 0x7f38481571b0 2024-04-25T00:25:59.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:59.111+0000 7f3836ffd640 1 --2- 172.21.15.116:0/3079258476 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3828067910 0x7f3828069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:25:59.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:59.111+0000 7f3836ffd640 1 -- 172.21.15.116:0/3079258476 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 10020+0+0 (secure 0 0 0) 0x7f3838120ba0 con 0x7f38481571b0 2024-04-25T00:25:59.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:59.115+0000 7f384659c640 1 --2- 172.21.15.116:0/3079258476 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3828067910 0x7f3828069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:25:59.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:59.115+0000 7f384659c640 1 --2- 172.21.15.116:0/3079258476 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3828067910 0x7f3828069dd0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f383c000f70 tx=0x7f383c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:25:59.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:25:59.127+0000 7f3836ffd640 1 -- 172.21.15.116:0/3079258476 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f38380c2030 con 0x7f38481571b0 2024-04-25T00:26:00.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:00.275+0000 7f384759e640 1 -- 172.21.15.116:0/3079258476 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f382c005860 con 0x7f38481571b0 2024-04-25T00:26:00.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:00.279+0000 7f3836ffd640 1 -- 172.21.15.116:0/3079258476 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 207 v207) ==== 95+0+6509 (secure 0 0 0) 0x7f38380160b0 con 0x7f38481571b0 2024-04-25T00:26:00.280 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:26:00.280 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":207,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6485,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/2371126056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2371126056},{"type":"v1","addr":"172.21.15.116:6835","nonce":2371126056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":207}],"filesystems":[{"mdsmap":{"epoch":201,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:25:29.028925+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6413},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6413":{"gid":6413,"name":"d","rank":0,"incarnation":198,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/4181466530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":4181466530},{"type":"v1","addr":"172.21.15.174:6837","nonce":4181466530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6413,"qdb_cluster":[6413]},"id":1},{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:25:45.002412+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8496":{"gid":8496,"name":"c","rank":0,"incarnation":203,"state":"up:active","state_seq":6,"addr":"172.21.15.116:6833/3693049068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3693049068},{"type":"v1","addr":"172.21.15.116:6833","nonce":3693049068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8496,"qdb_cluster":[8496]},"id":2}]} 2024-04-25T00:26:00.280 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 207 2024-04-25T00:26:00.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:00.291+0000 7f384759e640 1 -- 172.21.15.116:0/3079258476 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3828067910 msgr2=0x7f3828069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:00.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:00.291+0000 7f384759e640 1 --2- 172.21.15.116:0/3079258476 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3828067910 0x7f3828069dd0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f383c000f70 tx=0x7f383c009040 comp rx=0 tx=0).stop 2024-04-25T00:26:00.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:00.291+0000 7f384759e640 1 -- 172.21.15.116:0/3079258476 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f38481571b0 msgr2=0x7f384815b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:00.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:00.291+0000 7f384759e640 1 --2- 172.21.15.116:0/3079258476 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f38481571b0 0x7f384815b020 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f38380ae040 tx=0x7f3838019140 comp rx=0 tx=0).stop 2024-04-25T00:26:00.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:00.295+0000 7f384759e640 1 -- 172.21.15.116:0/3079258476 shutdown_connections 2024-04-25T00:26:00.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:00.295+0000 7f384759e640 1 --2- 172.21.15.116:0/3079258476 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3828067910 0x7f3828069dd0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:00.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:00.295+0000 7f384759e640 1 --2- 172.21.15.116:0/3079258476 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f384815da60 0x7f38480abfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:00.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:00.295+0000 7f384759e640 1 --2- 172.21.15.116:0/3079258476 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f38481571b0 0x7f384815b020 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:00.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:00.295+0000 7f384759e640 1 --2- 172.21.15.116:0/3079258476 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f38480c2190 0x7f38480aa370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:00.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:00.295+0000 7f384759e640 1 -- 172.21.15.116:0/3079258476 >> 172.21.15.116:0/3079258476 conn(0x7f38480c6db0 msgr2=0x7f3848155dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:00.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:00.295+0000 7f384759e640 1 -- 172.21.15.116:0/3079258476 shutdown_connections 2024-04-25T00:26:00.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:00.295+0000 7f384759e640 1 -- 172.21.15.116:0/3079258476 wait complete. 2024-04-25T00:26:00.336 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:26:02.338 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:26:02.673 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:02.671+0000 7fcfc759e640 1 Processor -- start 2024-04-25T00:26:02.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:02.671+0000 7fcfc759e640 1 -- start start 2024-04-25T00:26:02.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:02.675+0000 7fcfc759e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcfc815f660 0x7fcfc815faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:02.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:02.675+0000 7fcfc759e640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fcfc8070ff0 0x7fcfc8071440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:02.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:02.675+0000 7fcfc759e640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfc815ffe0 con 0x7fcfc8165ac0 2024-04-25T00:26:02.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:02.675+0000 7fcfc759e640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fcfc8162c80 con 0x7fcfc8070ff0 2024-04-25T00:26:02.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:02.675+0000 7fcfc759e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fcfc8162f60 con 0x7fcfc815f660 2024-04-25T00:26:02.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:02.675+0000 7fcfc5d9b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcfc815f660 0x7fcfc815faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:02.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:02.675+0000 7fcfc5d9b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcfc815f660 0x7fcfc815faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:47864/0 (socket says 172.21.15.116:47864) 2024-04-25T00:26:02.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:02.675+0000 7fcfc5d9b640 1 -- 172.21.15.116:0/1086118512 learned_addr learned my addr 172.21.15.116:0/1086118512 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:26:02.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:02.675+0000 7fcfc6d9d640 1 --2- 172.21.15.116:0/1086118512 >> v2:172.21.15.174:3300/0 conn(0x7fcfc8070ff0 0x7fcfc8071440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:02.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:02.675+0000 7fcfc5d9b640 1 -- 172.21.15.116:0/1086118512 >> v2:172.21.15.174:3300/0 conn(0x7fcfc8070ff0 msgr2=0x7fcfc8071440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:02.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:02.675+0000 7fcfc5d9b640 1 --2- 172.21.15.116:0/1086118512 >> v2:172.21.15.174:3300/0 conn(0x7fcfc8070ff0 0x7fcfc8071440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:02.675+0000 7fcfc5d9b640 1 -- 172.21.15.116:0/1086118512 >> v1:172.21.15.116:6789/0 conn(0x7fcfc8165ac0 legacy=0x7fcfc80c2850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T00:26:02.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:02.675+0000 7fcfc5d9b640 1 -- 172.21.15.116:0/1086118512 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfc8163240 con 0x7fcfc815f660 2024-04-25T00:26:02.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:02.683+0000 7fcfc5d9b640 1 --2- 172.21.15.116:0/1086118512 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcfc815f660 0x7fcfc815faa0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fcfc000bf60 tx=0x7fcfc0020e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64ed467a97aa48a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:02.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:02.683+0000 7fcfc559a640 1 -- 172.21.15.116:0/1086118512 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc000b2d0 con 0x7fcfc815f660 2024-04-25T00:26:02.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:02.683+0000 7fcfc559a640 1 -- 172.21.15.116:0/1086118512 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfc000b470 con 0x7fcfc815f660 2024-04-25T00:26:02.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:02.683+0000 7fcfc759e640 1 -- 172.21.15.116:0/1086118512 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcfc815f660 msgr2=0x7fcfc815faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:02.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:02.683+0000 7fcfc759e640 1 --2- 172.21.15.116:0/1086118512 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcfc815f660 0x7fcfc815faa0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fcfc000bf60 tx=0x7fcfc0020e00 comp rx=0 tx=0).stop 2024-04-25T00:26:02.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:02.683+0000 7fcfc759e640 1 -- 172.21.15.116:0/1086118512 shutdown_connections 2024-04-25T00:26:02.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:02.683+0000 7fcfc759e640 1 --2- 172.21.15.116:0/1086118512 >> v2:172.21.15.174:3300/0 conn(0x7fcfc8070ff0 0x7fcfc8071440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:02.683+0000 7fcfc759e640 1 --2- 172.21.15.116:0/1086118512 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcfc815f660 0x7fcfc815faa0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:02.683+0000 7fcfc759e640 1 -- 172.21.15.116:0/1086118512 >> 172.21.15.116:0/1086118512 conn(0x7fcfc80c3940 msgr2=0x7fcfc80c3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:02.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:02.687+0000 7fcfc759e640 1 -- 172.21.15.116:0/1086118512 shutdown_connections 2024-04-25T00:26:02.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:02.687+0000 7fcfc759e640 1 -- 172.21.15.116:0/1086118512 wait complete. 2024-04-25T00:26:02.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:02.687+0000 7fcfc759e640 1 Processor -- start 2024-04-25T00:26:02.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:02.687+0000 7fcfc759e640 1 -- start start 2024-04-25T00:26:02.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:02.687+0000 7fcfc759e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcfc8070ff0 0x7fcfc80ab310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:02.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:02.687+0000 7fcfc759e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfc815f660 0x7fcfc80a9700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:02.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:02.687+0000 7fcfc759e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfc8165ac0 0x7fcfc8158770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:02.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:02.687+0000 7fcfc759e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc81626f0 con 0x7fcfc815f660 2024-04-25T00:26:02.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:02.687+0000 7fcfc759e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcfc8162570 con 0x7fcfc8165ac0 2024-04-25T00:26:02.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:02.687+0000 7fcfc759e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fcfc81675d0 con 0x7fcfc8070ff0 2024-04-25T00:26:02.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:02.687+0000 7fcfc5d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfc815f660 0x7fcfc80a9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:02.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:02.687+0000 7fcfc5d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfc815f660 0x7fcfc80a9700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:45570/0 (socket says 172.21.15.116:45570) 2024-04-25T00:26:02.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:02.687+0000 7fcfc6d9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfc8165ac0 0x7fcfc8158770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:02.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:02.687+0000 7fcfc5d9b640 1 -- 172.21.15.116:0/3630405766 learned_addr learned my addr 172.21.15.116:0/3630405766 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:26:02.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:02.687+0000 7fcfc659c640 1 --2- 172.21.15.116:0/3630405766 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcfc8070ff0 0x7fcfc80ab310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:02.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:02.691+0000 7fcfc6d9d640 1 -- 172.21.15.116:0/3630405766 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcfc8070ff0 msgr2=0x7fcfc80ab310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:02.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:02.691+0000 7fcfc6d9d640 1 --2- 172.21.15.116:0/3630405766 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcfc8070ff0 0x7fcfc80ab310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:02.691+0000 7fcfc6d9d640 1 -- 172.21.15.116:0/3630405766 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfc815f660 msgr2=0x7fcfc80a9700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:02.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:02.691+0000 7fcfc6d9d640 1 --2- 172.21.15.116:0/3630405766 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfc815f660 0x7fcfc80a9700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:02.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:02.691+0000 7fcfc6d9d640 1 -- 172.21.15.116:0/3630405766 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfc808aea0 con 0x7fcfc8165ac0 2024-04-25T00:26:02.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:02.691+0000 7fcfc659c640 1 --2- 172.21.15.116:0/3630405766 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcfc8070ff0 0x7fcfc80ab310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:26:02.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:02.691+0000 7fcfc6d9d640 1 --2- 172.21.15.116:0/3630405766 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfc8165ac0 0x7fcfc8158770 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fcfb800d580 tx=0x7fcfb800da40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:02.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:02.691+0000 7fcfb6ffd640 1 -- 172.21.15.116:0/3630405766 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb8018020 con 0x7fcfc8165ac0 2024-04-25T00:26:02.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:02.691+0000 7fcfc759e640 1 -- 172.21.15.116:0/3630405766 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfc8203b30 con 0x7fcfc8165ac0 2024-04-25T00:26:02.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:02.691+0000 7fcfc759e640 1 -- 172.21.15.116:0/3630405766 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfc82040f0 con 0x7fcfc8165ac0 2024-04-25T00:26:02.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:02.691+0000 7fcfb6ffd640 1 -- 172.21.15.116:0/3630405766 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfb8014620 con 0x7fcfc8165ac0 2024-04-25T00:26:02.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:02.691+0000 7fcfb6ffd640 1 -- 172.21.15.116:0/3630405766 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfb80149f0 con 0x7fcfc8165ac0 2024-04-25T00:26:02.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:02.691+0000 7fcfb6ffd640 1 -- 172.21.15.116:0/3630405766 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fcfb801b020 con 0x7fcfc8165ac0 2024-04-25T00:26:02.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:02.691+0000 7fcfb6ffd640 1 --2- 172.21.15.116:0/3630405766 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcfa8067990 0x7fcfa8069e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:02.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:02.691+0000 7fcfb6ffd640 1 -- 172.21.15.116:0/3630405766 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 10020+0+0 (secure 0 0 0) 0x7fcfb80870f0 con 0x7fcfc8165ac0 2024-04-25T00:26:02.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:02.691+0000 7fcfc659c640 1 --2- 172.21.15.116:0/3630405766 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcfa8067990 0x7fcfa8069e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:02.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:02.691+0000 7fcfc759e640 1 -- 172.21.15.116:0/3630405766 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfac0052a0 con 0x7fcfc8165ac0 2024-04-25T00:26:02.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:02.699+0000 7fcfc659c640 1 --2- 172.21.15.116:0/3630405766 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcfa8067990 0x7fcfa8069e50 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc009cd0 tx=0x7fcfbc009060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:02.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:02.699+0000 7fcfb6ffd640 1 -- 172.21.15.116:0/3630405766 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fcfb8022020 con 0x7fcfc8165ac0 2024-04-25T00:26:03.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:03.383+0000 7fcfc759e640 1 -- 172.21.15.116:0/3630405766 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcfac005590 con 0x7fcfc8165ac0 2024-04-25T00:26:03.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:03.387+0000 7fcfb6ffd640 1 -- 172.21.15.116:0/3630405766 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 207 v207) ==== 95+0+6509 (secure 0 0 0) 0x7fcfb80564e0 con 0x7fcfc8165ac0 2024-04-25T00:26:03.387 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:26:03.388 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":207,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6485,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/2371126056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2371126056},{"type":"v1","addr":"172.21.15.116:6835","nonce":2371126056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":207}],"filesystems":[{"mdsmap":{"epoch":201,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:25:29.028925+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6413},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6413":{"gid":6413,"name":"d","rank":0,"incarnation":198,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/4181466530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":4181466530},{"type":"v1","addr":"172.21.15.174:6837","nonce":4181466530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6413,"qdb_cluster":[6413]},"id":1},{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:25:45.002412+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8496":{"gid":8496,"name":"c","rank":0,"incarnation":203,"state":"up:active","state_seq":6,"addr":"172.21.15.116:6833/3693049068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3693049068},{"type":"v1","addr":"172.21.15.116:6833","nonce":3693049068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8496,"qdb_cluster":[8496]},"id":2}]} 2024-04-25T00:26:03.388 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 207 2024-04-25T00:26:03.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:03.391+0000 7fcfc759e640 1 -- 172.21.15.116:0/3630405766 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcfa8067990 msgr2=0x7fcfa8069e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:03.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:03.391+0000 7fcfc759e640 1 --2- 172.21.15.116:0/3630405766 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcfa8067990 0x7fcfa8069e50 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fcfbc009cd0 tx=0x7fcfbc009060 comp rx=0 tx=0).stop 2024-04-25T00:26:03.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:03.391+0000 7fcfc759e640 1 -- 172.21.15.116:0/3630405766 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfc8165ac0 msgr2=0x7fcfc8158770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:03.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:03.391+0000 7fcfc759e640 1 --2- 172.21.15.116:0/3630405766 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfc8165ac0 0x7fcfc8158770 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fcfb800d580 tx=0x7fcfb800da40 comp rx=0 tx=0).stop 2024-04-25T00:26:03.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:03.391+0000 7fcfc759e640 1 -- 172.21.15.116:0/3630405766 shutdown_connections 2024-04-25T00:26:03.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:03.391+0000 7fcfc759e640 1 --2- 172.21.15.116:0/3630405766 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcfa8067990 0x7fcfa8069e50 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:03.391+0000 7fcfc759e640 1 --2- 172.21.15.116:0/3630405766 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcfc8165ac0 0x7fcfc8158770 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:03.391+0000 7fcfc759e640 1 --2- 172.21.15.116:0/3630405766 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcfc815f660 0x7fcfc80a9700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:03.391+0000 7fcfc759e640 1 --2- 172.21.15.116:0/3630405766 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcfc8070ff0 0x7fcfc80ab310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:03.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:03.391+0000 7fcfc759e640 1 -- 172.21.15.116:0/3630405766 >> 172.21.15.116:0/3630405766 conn(0x7fcfc80c3940 msgr2=0x7fcfc815dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:03.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:03.395+0000 7fcfc759e640 1 -- 172.21.15.116:0/3630405766 shutdown_connections 2024-04-25T00:26:03.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:03.395+0000 7fcfc759e640 1 -- 172.21.15.116:0/3630405766 wait complete. 2024-04-25T00:26:03.467 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:26:05.469 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:26:05.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.623+0000 7f073184c640 1 Processor -- start 2024-04-25T00:26:05.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.623+0000 7f073184c640 1 -- start start 2024-04-25T00:26:05.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.623+0000 7f073184c640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f072c072c90 0x7f072c150f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:05.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.623+0000 7f073184c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f072c072810 0x7f072c153760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:05.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.623+0000 7f073184c640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f072c0c2c40 con 0x7f072c0c5280 2024-04-25T00:26:05.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.623+0000 7f073184c640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f072c157c20 con 0x7f072c072810 2024-04-25T00:26:05.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.623+0000 7f073184c640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f072c0c2f70 con 0x7f072c072c90 2024-04-25T00:26:05.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.623+0000 7f072b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f072c072810 0x7f072c153760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:05.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.623+0000 7f072b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f072c072810 0x7f072c153760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37272/0 (socket says 172.21.15.116:37272) 2024-04-25T00:26:05.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.623+0000 7f072b7fe640 1 -- 172.21.15.116:0/3084775699 learned_addr learned my addr 172.21.15.116:0/3084775699 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:26:05.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.623+0000 7f072a7fc640 1 --2- 172.21.15.116:0/3084775699 >> v2:172.21.15.174:3300/0 conn(0x7f072c072c90 0x7f072c150f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:05.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.623+0000 7f072b7fe640 1 -- 172.21.15.116:0/3084775699 >> v2:172.21.15.174:3300/0 conn(0x7f072c072c90 msgr2=0x7f072c150f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:05.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.623+0000 7f072b7fe640 1 --2- 172.21.15.116:0/3084775699 >> v2:172.21.15.174:3300/0 conn(0x7f072c072c90 0x7f072c150f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:05.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.623+0000 7f072b7fe640 1 -- 172.21.15.116:0/3084775699 >> v1:172.21.15.174:6789/0 conn(0x7f072c0c5280 legacy=0x7f072c150840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:05.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.623+0000 7f072b7fe640 1 -- 172.21.15.116:0/3084775699 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f072c157660 con 0x7f072c072810 2024-04-25T00:26:05.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.623+0000 7f072a7fc640 1 --2- 172.21.15.116:0/3084775699 >> v2:172.21.15.174:3300/0 conn(0x7f072c072c90 0x7f072c150f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:26:05.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.627+0000 7f072b7fe640 1 --2- 172.21.15.116:0/3084775699 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f072c072810 0x7f072c153760 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f072c0c2970 tx=0x7f072401c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=682e5f9b5855a6e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:05.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.627+0000 7f0729ffb640 1 -- 172.21.15.116:0/3084775699 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07240ab5f0 con 0x7f072c072810 2024-04-25T00:26:05.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.627+0000 7f0729ffb640 1 -- 172.21.15.116:0/3084775699 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07240b0070 con 0x7f072c072810 2024-04-25T00:26:05.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.627+0000 7f0729ffb640 1 -- 172.21.15.116:0/3084775699 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07240ab9b0 con 0x7f072c072810 2024-04-25T00:26:05.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.627+0000 7f073184c640 1 -- 172.21.15.116:0/3084775699 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f072c072810 msgr2=0x7f072c153760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:05.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.627+0000 7f073184c640 1 --2- 172.21.15.116:0/3084775699 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f072c072810 0x7f072c153760 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f072c0c2970 tx=0x7f072401c3d0 comp rx=0 tx=0).stop 2024-04-25T00:26:05.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.627+0000 7f073184c640 1 -- 172.21.15.116:0/3084775699 shutdown_connections 2024-04-25T00:26:05.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.627+0000 7f073184c640 1 --2- 172.21.15.116:0/3084775699 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f072c072810 0x7f072c153760 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:05.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.627+0000 7f073184c640 1 --2- 172.21.15.116:0/3084775699 >> v2:172.21.15.174:3300/0 conn(0x7f072c072c90 0x7f072c150f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:05.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.627+0000 7f073184c640 1 -- 172.21.15.116:0/3084775699 >> 172.21.15.116:0/3084775699 conn(0x7f072c06c720 msgr2=0x7f072c06cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:05.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.627+0000 7f073184c640 1 -- 172.21.15.116:0/3084775699 shutdown_connections 2024-04-25T00:26:05.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.627+0000 7f073184c640 1 -- 172.21.15.116:0/3084775699 wait complete. 2024-04-25T00:26:05.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.631+0000 7f073184c640 1 Processor -- start 2024-04-25T00:26:05.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.631+0000 7f073184c640 1 -- start start 2024-04-25T00:26:05.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.631+0000 7f073184c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f072c072810 0x7f072c147210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:05.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.631+0000 7f073184c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f072c072c90 0x7f072c142bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:05.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.631+0000 7f073184c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f072c0c5280 0x7f072c1eddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:05.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.631+0000 7f073184c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f072c160380 con 0x7f072c072c90 2024-04-25T00:26:05.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.631+0000 7f073184c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f072c160200 con 0x7f072c0c5280 2024-04-25T00:26:05.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.631+0000 7f073184c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f072c160500 con 0x7f072c072810 2024-04-25T00:26:05.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.631+0000 7f072a7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f072c072c90 0x7f072c142bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:05.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.631+0000 7f072a7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f072c072c90 0x7f072c142bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37286/0 (socket says 172.21.15.116:37286) 2024-04-25T00:26:05.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.631+0000 7f072a7fc640 1 -- 172.21.15.116:0/994397885 learned_addr learned my addr 172.21.15.116:0/994397885 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:26:05.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.631+0000 7f072affd640 1 --2- 172.21.15.116:0/994397885 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f072c072810 0x7f072c147210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:05.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.631+0000 7f072b7fe640 1 --2- 172.21.15.116:0/994397885 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f072c0c5280 0x7f072c1eddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:05.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.631+0000 7f072a7fc640 1 -- 172.21.15.116:0/994397885 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f072c072810 msgr2=0x7f072c147210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:05.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.631+0000 7f072a7fc640 1 --2- 172.21.15.116:0/994397885 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f072c072810 0x7f072c147210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:05.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.631+0000 7f072a7fc640 1 -- 172.21.15.116:0/994397885 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f072c0c5280 msgr2=0x7f072c1eddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:05.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.631+0000 7f072a7fc640 1 --2- 172.21.15.116:0/994397885 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f072c0c5280 0x7f072c1eddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:05.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.631+0000 7f072a7fc640 1 -- 172.21.15.116:0/994397885 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f072c0ad1f0 con 0x7f072c072c90 2024-04-25T00:26:05.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.631+0000 7f072a7fc640 1 --2- 172.21.15.116:0/994397885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f072c072c90 0x7f072c142bc0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f072000ebe0 tx=0x7f072000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:05.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.631+0000 7f0713fff640 1 -- 172.21.15.116:0/994397885 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0720017070 con 0x7f072c072c90 2024-04-25T00:26:05.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.631+0000 7f0713fff640 1 -- 172.21.15.116:0/994397885 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f072000cc30 con 0x7f072c072c90 2024-04-25T00:26:05.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.631+0000 7f073184c640 1 -- 172.21.15.116:0/994397885 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f072c06fa80 con 0x7f072c072c90 2024-04-25T00:26:05.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.631+0000 7f0713fff640 1 -- 172.21.15.116:0/994397885 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0720013470 con 0x7f072c072c90 2024-04-25T00:26:05.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.631+0000 7f073184c640 1 -- 172.21.15.116:0/994397885 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f072c143100 con 0x7f072c072c90 2024-04-25T00:26:05.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.631+0000 7f073184c640 1 -- 172.21.15.116:0/994397885 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f072c068a60 con 0x7f072c072c90 2024-04-25T00:26:05.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.635+0000 7f0713fff640 1 -- 172.21.15.116:0/994397885 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f072001a070 con 0x7f072c072c90 2024-04-25T00:26:05.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.635+0000 7f0713fff640 1 --2- 172.21.15.116:0/994397885 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f07180678c0 0x7f0718069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:05.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.635+0000 7f0713fff640 1 -- 172.21.15.116:0/994397885 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 10020+0+0 (secure 0 0 0) 0x7f0720086f70 con 0x7f072c072c90 2024-04-25T00:26:05.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.635+0000 7f072affd640 1 --2- 172.21.15.116:0/994397885 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f07180678c0 0x7f0718069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:05.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.635+0000 7f072affd640 1 --2- 172.21.15.116:0/994397885 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f07180678c0 0x7f0718069d80 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f071c0012f0 tx=0x7f071c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:05.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.639+0000 7f0713fff640 1 -- 172.21.15.116:0/994397885 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f07200563c0 con 0x7f072c072c90 2024-04-25T00:26:05.916 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.915+0000 7f073184c640 1 -- 172.21.15.116:0/994397885 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f072c0aced0 con 0x7f072c072c90 2024-04-25T00:26:05.917 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.915+0000 7f0713fff640 1 -- 172.21.15.116:0/994397885 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 207 v207) ==== 95+0+6509 (secure 0 0 0) 0x7f072005a6c0 con 0x7f072c072c90 2024-04-25T00:26:05.917 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:26:05.917 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":207,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6485,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/2371126056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2371126056},{"type":"v1","addr":"172.21.15.116:6835","nonce":2371126056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":207}],"filesystems":[{"mdsmap":{"epoch":201,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:25:29.028925+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6413},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6413":{"gid":6413,"name":"d","rank":0,"incarnation":198,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/4181466530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":4181466530},{"type":"v1","addr":"172.21.15.174:6837","nonce":4181466530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6413,"qdb_cluster":[6413]},"id":1},{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:25:45.002412+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8496":{"gid":8496,"name":"c","rank":0,"incarnation":203,"state":"up:active","state_seq":6,"addr":"172.21.15.116:6833/3693049068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3693049068},{"type":"v1","addr":"172.21.15.116:6833","nonce":3693049068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8496,"qdb_cluster":[8496]},"id":2}]} 2024-04-25T00:26:05.917 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 207 2024-04-25T00:26:05.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.919+0000 7f073184c640 1 -- 172.21.15.116:0/994397885 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f07180678c0 msgr2=0x7f0718069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:05.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.919+0000 7f073184c640 1 --2- 172.21.15.116:0/994397885 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f07180678c0 0x7f0718069d80 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f071c0012f0 tx=0x7f071c009040 comp rx=0 tx=0).stop 2024-04-25T00:26:05.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.919+0000 7f073184c640 1 -- 172.21.15.116:0/994397885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f072c072c90 msgr2=0x7f072c142bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:05.919 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.919+0000 7f073184c640 1 --2- 172.21.15.116:0/994397885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f072c072c90 0x7f072c142bc0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f072000ebe0 tx=0x7f072000c520 comp rx=0 tx=0).stop 2024-04-25T00:26:05.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.919+0000 7f073184c640 1 -- 172.21.15.116:0/994397885 shutdown_connections 2024-04-25T00:26:05.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.919+0000 7f073184c640 1 --2- 172.21.15.116:0/994397885 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f07180678c0 0x7f0718069d80 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:05.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.919+0000 7f073184c640 1 --2- 172.21.15.116:0/994397885 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f072c0c5280 0x7f072c1eddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:05.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.919+0000 7f073184c640 1 --2- 172.21.15.116:0/994397885 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f072c072c90 0x7f072c142bc0 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:05.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.919+0000 7f073184c640 1 --2- 172.21.15.116:0/994397885 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f072c072810 0x7f072c147210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:05.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.919+0000 7f073184c640 1 -- 172.21.15.116:0/994397885 >> 172.21.15.116:0/994397885 conn(0x7f072c06c720 msgr2=0x7f072c156f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:05.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.919+0000 7f073184c640 1 -- 172.21.15.116:0/994397885 shutdown_connections 2024-04-25T00:26:05.921 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:05.919+0000 7f073184c640 1 -- 172.21.15.116:0/994397885 wait complete. 2024-04-25T00:26:05.944 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:26:07.946 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:26:08.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.095+0000 7f0301aaf640 1 Processor -- start 2024-04-25T00:26:08.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.095+0000 7f0301aaf640 1 -- start start 2024-04-25T00:26:08.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.095+0000 7f0301aaf640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f02fc06fb80 0x7f02fc1611d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:08.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.095+0000 7f0301aaf640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02fc13e7a0 con 0x7f02fc06be40 2024-04-25T00:26:08.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.095+0000 7f0301aaf640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02fc13ea80 con 0x7f02fc13e210 2024-04-25T00:26:08.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.095+0000 7f0301aaf640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f02fc13e5f0 con 0x7f02fc06fb80 2024-04-25T00:26:08.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.095+0000 7f02f3fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f02fc06be40 0x7f02fc155f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:41148/0 (socket says 172.21.15.116:41148) 2024-04-25T00:26:08.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.095+0000 7f02f3fff640 1 -- 172.21.15.116:0/1895458607 learned_addr learned my addr 172.21.15.116:0/1895458607 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:26:08.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.095+0000 7f02fbfff640 1 --2- 172.21.15.116:0/1895458607 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f02fc06fb80 0x7f02fc1611d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:08.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.095+0000 7f02faffd640 1 -- 172.21.15.116:0/1895458607 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2890545551 0 0) 0x7f02fc13e7a0 con 0x7f02fc06be40 2024-04-25T00:26:08.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.095+0000 7f02faffd640 1 -- 172.21.15.116:0/1895458607 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f02e8003530 con 0x7f02fc06be40 2024-04-25T00:26:08.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.095+0000 7f02faffd640 1 -- 172.21.15.116:0/1895458607 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2889208681 0 0) 0x7f02fc13ea80 con 0x7f02fc13e210 2024-04-25T00:26:08.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.095+0000 7f02faffd640 1 -- 172.21.15.116:0/1895458607 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f02fc13e7a0 con 0x7f02fc13e210 2024-04-25T00:26:08.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.095+0000 7f02faffd640 1 -- 172.21.15.116:0/1895458607 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 346893096 0 0) 0x7f02e8003530 con 0x7f02fc06be40 2024-04-25T00:26:08.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.095+0000 7f02faffd640 1 -- 172.21.15.116:0/1895458607 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f02fc06fb80 msgr2=0x7f02fc1611d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:08.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.095+0000 7f02faffd640 1 --2- 172.21.15.116:0/1895458607 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f02fc06fb80 0x7f02fc1611d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:08.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.095+0000 7f02faffd640 1 -- 172.21.15.116:0/1895458607 >> v1:172.21.15.174:6789/0 conn(0x7f02fc13e210 legacy=0x7f02fc06b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:08.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.095+0000 7f02faffd640 1 -- 172.21.15.116:0/1895458607 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02fc13ed60 con 0x7f02fc06be40 2024-04-25T00:26:08.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.095+0000 7f02faffd640 1 -- 172.21.15.116:0/1895458607 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f02e4001680 con 0x7f02fc06be40 2024-04-25T00:26:08.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.095+0000 7f02faffd640 1 -- 172.21.15.116:0/1895458607 >> v1:172.21.15.116:6789/0 conn(0x7f02fc06be40 legacy=0x7f02fc155f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:08.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.095+0000 7f02faffd640 1 --2- 172.21.15.116:0/1895458607 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f02e8003b80 0x7f02e8003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:08.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.095+0000 7f02faffd640 1 --2- 172.21.15.116:0/1895458607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02e8004810 0x7f02e8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:08.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.095+0000 7f02faffd640 1 --2- 172.21.15.116:0/1895458607 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f02fc06fb80 0x7f02e800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:08.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.095+0000 7f02faffd640 1 -- 172.21.15.116:0/1895458607 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f02e80055e0 con 0x7f02e8004810 2024-04-25T00:26:08.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.095+0000 7f02faffd640 1 -- 172.21.15.116:0/1895458607 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f02e800bf10 con 0x7f02e8003b80 2024-04-25T00:26:08.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.095+0000 7f02faffd640 1 -- 172.21.15.116:0/1895458607 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f02e800c090 con 0x7f02fc06fb80 2024-04-25T00:26:08.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.095+0000 7f02f3fff640 1 --2- 172.21.15.116:0/1895458607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02e8004810 0x7f02e8004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:08.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.095+0000 7f02fb7fe640 1 --2- 172.21.15.116:0/1895458607 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f02e8003b80 0x7f02e8003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:08.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.095+0000 7f02f3fff640 1 -- 172.21.15.116:0/1895458607 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f02fc06fb80 msgr2=0x7f02e800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:08.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.095+0000 7f02f3fff640 1 --2- 172.21.15.116:0/1895458607 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f02fc06fb80 0x7f02e800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:08.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.095+0000 7f02f3fff640 1 -- 172.21.15.116:0/1895458607 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f02e8003b80 msgr2=0x7f02e8003f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:08.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.095+0000 7f02f3fff640 1 --2- 172.21.15.116:0/1895458607 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f02e8003b80 0x7f02e8003f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:08.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.095+0000 7f02f3fff640 1 -- 172.21.15.116:0/1895458607 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02e8003530 con 0x7f02e8004810 2024-04-25T00:26:08.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.099+0000 7f02f3fff640 1 --2- 172.21.15.116:0/1895458607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02e8004810 0x7f02e8004c90 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f02e400c190 tx=0x7f02e401cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ce7ce54f999900f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:08.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.099+0000 7f02faffd640 1 -- 172.21.15.116:0/1895458607 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e4007cb0 con 0x7f02e8004810 2024-04-25T00:26:08.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.099+0000 7f02faffd640 1 -- 172.21.15.116:0/1895458607 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02e40a9020 con 0x7f02e8004810 2024-04-25T00:26:08.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.099+0000 7f02faffd640 1 -- 172.21.15.116:0/1895458607 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e4005ae0 con 0x7f02e8004810 2024-04-25T00:26:08.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.099+0000 7f0301aaf640 1 -- 172.21.15.116:0/1895458607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02e8004810 msgr2=0x7f02e8004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:08.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.099+0000 7f0301aaf640 1 --2- 172.21.15.116:0/1895458607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02e8004810 0x7f02e8004c90 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f02e400c190 tx=0x7f02e401cbe0 comp rx=0 tx=0).stop 2024-04-25T00:26:08.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.099+0000 7f02fbfff640 1 -- 172.21.15.116:0/1895458607 reap_dead start 2024-04-25T00:26:08.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.099+0000 7f0301aaf640 1 -- 172.21.15.116:0/1895458607 shutdown_connections 2024-04-25T00:26:08.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.099+0000 7f0301aaf640 1 -- 172.21.15.116:0/1895458607 >> 172.21.15.116:0/1895458607 conn(0x7f02fc0c4aa0 msgr2=0x7f02fc0c4eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:08.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.099+0000 7f0301aaf640 1 -- 172.21.15.116:0/1895458607 shutdown_connections 2024-04-25T00:26:08.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.099+0000 7f0301aaf640 1 -- 172.21.15.116:0/1895458607 wait complete. 2024-04-25T00:26:08.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.103+0000 7f0301aaf640 1 Processor -- start 2024-04-25T00:26:08.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.103+0000 7f0301aaf640 1 -- start start 2024-04-25T00:26:08.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.103+0000 7f0301aaf640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f02e8004810 0x7f02fc13d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:08.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.103+0000 7f0301aaf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02fc14e3c0 0x7f02fc1f22c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:08.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.103+0000 7f0301aaf640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f02fc135480 0x7f02fc13bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:08.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.103+0000 7f0301aaf640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f02fc13f040 con 0x7f02fc14e3c0 2024-04-25T00:26:08.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.103+0000 7f0301aaf640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f02fc1619e0 con 0x7f02e8004810 2024-04-25T00:26:08.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.103+0000 7f0301aaf640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f02fc161b60 con 0x7f02fc135480 2024-04-25T00:26:08.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.103+0000 7f02fbfff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f02fc135480 0x7f02fc13bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:08.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.103+0000 7f02fbfff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f02fc135480 0x7f02fc13bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:47228/0 (socket says 172.21.15.116:47228) 2024-04-25T00:26:08.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.103+0000 7f02fbfff640 1 -- 172.21.15.116:0/296192381 learned_addr learned my addr 172.21.15.116:0/296192381 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:26:08.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.103+0000 7f02f3fff640 1 --2- 172.21.15.116:0/296192381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02fc14e3c0 0x7f02fc1f22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:08.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.103+0000 7f02fb7fe640 1 --2- 172.21.15.116:0/296192381 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f02e8004810 0x7f02fc13d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:08.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.103+0000 7f02fbfff640 1 -- 172.21.15.116:0/296192381 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f02e8004810 msgr2=0x7f02fc13d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:08.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.103+0000 7f02fbfff640 1 --2- 172.21.15.116:0/296192381 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f02e8004810 0x7f02fc13d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:08.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.103+0000 7f02fbfff640 1 -- 172.21.15.116:0/296192381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02fc14e3c0 msgr2=0x7f02fc1f22c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:08.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.103+0000 7f02fbfff640 1 --2- 172.21.15.116:0/296192381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02fc14e3c0 0x7f02fc1f22c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:08.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.103+0000 7f02fbfff640 1 -- 172.21.15.116:0/296192381 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02fc135a20 con 0x7f02fc135480 2024-04-25T00:26:08.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.103+0000 7f02fbfff640 1 --2- 172.21.15.116:0/296192381 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f02fc135480 0x7f02fc13bda0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f02fc04fbb0 tx=0x7f02f400a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:08.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.103+0000 7f02f8ff9640 1 -- 172.21.15.116:0/296192381 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f400cea0 con 0x7f02fc135480 2024-04-25T00:26:08.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.107+0000 7f02f8ff9640 1 -- 172.21.15.116:0/296192381 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02f40039a0 con 0x7f02fc135480 2024-04-25T00:26:08.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.107+0000 7f0301aaf640 1 -- 172.21.15.116:0/296192381 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02fc14d8e0 con 0x7f02fc135480 2024-04-25T00:26:08.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.107+0000 7f02f8ff9640 1 -- 172.21.15.116:0/296192381 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f4003d70 con 0x7f02fc135480 2024-04-25T00:26:08.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.107+0000 7f0301aaf640 1 -- 172.21.15.116:0/296192381 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f02fc14dea0 con 0x7f02fc135480 2024-04-25T00:26:08.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.107+0000 7f0301aaf640 1 -- 172.21.15.116:0/296192381 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02fc068a60 con 0x7f02fc135480 2024-04-25T00:26:08.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.107+0000 7f02f8ff9640 1 -- 172.21.15.116:0/296192381 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f02f4036660 con 0x7f02fc135480 2024-04-25T00:26:08.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.107+0000 7f02f8ff9640 1 --2- 172.21.15.116:0/296192381 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f02dc0678f0 0x7f02dc069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:08.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.107+0000 7f02f8ff9640 1 -- 172.21.15.116:0/296192381 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(67..67 src has 1..67) ==== 10020+0+0 (secure 0 0 0) 0x7f02f4087240 con 0x7f02fc135480 2024-04-25T00:26:08.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.107+0000 7f02fb7fe640 1 --2- 172.21.15.116:0/296192381 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f02dc0678f0 0x7f02dc069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:08.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.107+0000 7f02fb7fe640 1 --2- 172.21.15.116:0/296192381 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f02dc0678f0 0x7f02dc069db0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f02ec007fd0 tx=0x7f02ec009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:08.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.119+0000 7f02f8ff9640 1 -- 172.21.15.116:0/296192381 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f02f4056d60 con 0x7f02fc135480 2024-04-25T00:26:08.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.403+0000 7f0301aaf640 1 -- 172.21.15.116:0/296192381 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f02fc13b820 con 0x7f02fc135480 2024-04-25T00:26:08.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.407+0000 7f02f8ff9640 1 -- 172.21.15.116:0/296192381 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 207 v207) ==== 95+0+6509 (secure 0 0 0) 0x7f02f4056660 con 0x7f02fc135480 2024-04-25T00:26:08.408 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:26:08.408 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":207,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6485,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/2371126056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2371126056},{"type":"v1","addr":"172.21.15.116:6835","nonce":2371126056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":207}],"filesystems":[{"mdsmap":{"epoch":201,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:25:29.028925+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6413},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6413":{"gid":6413,"name":"d","rank":0,"incarnation":198,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/4181466530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":4181466530},{"type":"v1","addr":"172.21.15.174:6837","nonce":4181466530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6413,"qdb_cluster":[6413]},"id":1},{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:25:45.002412+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8496":{"gid":8496,"name":"c","rank":0,"incarnation":203,"state":"up:active","state_seq":6,"addr":"172.21.15.116:6833/3693049068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3693049068},{"type":"v1","addr":"172.21.15.116:6833","nonce":3693049068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8496,"qdb_cluster":[8496]},"id":2}]} 2024-04-25T00:26:08.408 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 207 2024-04-25T00:26:08.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.411+0000 7f0301aaf640 1 -- 172.21.15.116:0/296192381 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f02dc0678f0 msgr2=0x7f02dc069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:08.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.411+0000 7f0301aaf640 1 --2- 172.21.15.116:0/296192381 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f02dc0678f0 0x7f02dc069db0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f02ec007fd0 tx=0x7f02ec009040 comp rx=0 tx=0).stop 2024-04-25T00:26:08.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.411+0000 7f0301aaf640 1 -- 172.21.15.116:0/296192381 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f02fc135480 msgr2=0x7f02fc13bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:08.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.411+0000 7f0301aaf640 1 --2- 172.21.15.116:0/296192381 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f02fc135480 0x7f02fc13bda0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f02fc04fbb0 tx=0x7f02f400a660 comp rx=0 tx=0).stop 2024-04-25T00:26:08.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.411+0000 7f0301aaf640 1 -- 172.21.15.116:0/296192381 shutdown_connections 2024-04-25T00:26:08.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.411+0000 7f0301aaf640 1 --2- 172.21.15.116:0/296192381 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f02dc0678f0 0x7f02dc069db0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:08.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.411+0000 7f0301aaf640 1 --2- 172.21.15.116:0/296192381 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f02fc135480 0x7f02fc13bda0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:08.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.411+0000 7f0301aaf640 1 --2- 172.21.15.116:0/296192381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f02fc14e3c0 0x7f02fc1f22c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:08.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.411+0000 7f0301aaf640 1 --2- 172.21.15.116:0/296192381 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f02e8004810 0x7f02fc13d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:08.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.411+0000 7f0301aaf640 1 -- 172.21.15.116:0/296192381 >> 172.21.15.116:0/296192381 conn(0x7f02fc0c4aa0 msgr2=0x7f02fc06f170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:08.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.411+0000 7f0301aaf640 1 -- 172.21.15.116:0/296192381 shutdown_connections 2024-04-25T00:26:08.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:08.411+0000 7f0301aaf640 1 -- 172.21.15.116:0/296192381 wait complete. 2024-04-25T00:26:08.437 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:26:10.439 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:26:10.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.591+0000 7f6f3d1d2640 1 Processor -- start 2024-04-25T00:26:10.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.591+0000 7f6f3d1d2640 1 -- start start 2024-04-25T00:26:10.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.591+0000 7f6f3d1d2640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f6f3813e310 0x7f6f380c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:10.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.591+0000 7f6f3d1d2640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f3806d200 con 0x7f6f380c2f50 2024-04-25T00:26:10.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.591+0000 7f6f3d1d2640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f3806cb50 con 0x7f6f3813ed60 2024-04-25T00:26:10.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.591+0000 7f6f3d1d2640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f6f38070d40 con 0x7f6f3813e310 2024-04-25T00:26:10.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.591+0000 7f6f3659c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f6f380c2f50 0x7f6f381559c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:41162/0 (socket says 172.21.15.116:41162) 2024-04-25T00:26:10.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.591+0000 7f6f3659c640 1 -- 172.21.15.116:0/2729217434 learned_addr learned my addr 172.21.15.116:0/2729217434 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:26:10.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.591+0000 7f6f36d9d640 1 --2- 172.21.15.116:0/2729217434 >> v2:172.21.15.174:3300/0 conn(0x7f6f3813e310 0x7f6f380c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:10.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.595+0000 7f6f35d9b640 1 -- 172.21.15.116:0/2729217434 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 507995938 0 0) 0x7f6f3806d200 con 0x7f6f380c2f50 2024-04-25T00:26:10.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.595+0000 7f6f35d9b640 1 -- 172.21.15.116:0/2729217434 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f20003530 con 0x7f6f380c2f50 2024-04-25T00:26:10.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.595+0000 7f6f35d9b640 1 -- 172.21.15.116:0/2729217434 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1121375316 0 0) 0x7f6f3806cb50 con 0x7f6f3813ed60 2024-04-25T00:26:10.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.595+0000 7f6f35d9b640 1 -- 172.21.15.116:0/2729217434 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6f3806d200 con 0x7f6f3813ed60 2024-04-25T00:26:10.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.595+0000 7f6f36d9d640 1 -- 172.21.15.116:0/2729217434 >> v1:172.21.15.174:6789/0 conn(0x7f6f3813ed60 legacy=0x7f6f381560d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:10.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.595+0000 7f6f36d9d640 1 -- 172.21.15.116:0/2729217434 >> v1:172.21.15.116:6789/0 conn(0x7f6f380c2f50 legacy=0x7f6f381559c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:10.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.595+0000 7f6f36d9d640 1 -- 172.21.15.116:0/2729217434 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f3806d470 con 0x7f6f3813e310 2024-04-25T00:26:10.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.595+0000 7f6f36d9d640 1 --2- 172.21.15.116:0/2729217434 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6f3813e310 0x7f6f380c2a10 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f6f280097c0 tx=0x7f6f2801c380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65e4f5b99deea4e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:10.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.595+0000 7f6f35d9b640 1 -- 172.21.15.116:0/2729217434 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f280b0070 con 0x7f6f3813e310 2024-04-25T00:26:10.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.595+0000 7f6f35d9b640 1 -- 172.21.15.116:0/2729217434 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f280ab600 con 0x7f6f3813e310 2024-04-25T00:26:10.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.595+0000 7f6f35d9b640 1 -- 172.21.15.116:0/2729217434 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f280ab9b0 con 0x7f6f3813e310 2024-04-25T00:26:10.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.599+0000 7f6f3d1d2640 1 -- 172.21.15.116:0/2729217434 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6f3813e310 msgr2=0x7f6f380c2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:10.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.599+0000 7f6f3d1d2640 1 --2- 172.21.15.116:0/2729217434 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6f3813e310 0x7f6f380c2a10 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f6f280097c0 tx=0x7f6f2801c380 comp rx=0 tx=0).stop 2024-04-25T00:26:10.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.599+0000 7f6f3d1d2640 1 -- 172.21.15.116:0/2729217434 shutdown_connections 2024-04-25T00:26:10.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.599+0000 7f6f3d1d2640 1 --2- 172.21.15.116:0/2729217434 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6f3813e310 0x7f6f380c2a10 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:10.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.599+0000 7f6f3d1d2640 1 -- 172.21.15.116:0/2729217434 >> 172.21.15.116:0/2729217434 conn(0x7f6f380c3b00 msgr2=0x7f6f380c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:10.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.599+0000 7f6f3d1d2640 1 -- 172.21.15.116:0/2729217434 shutdown_connections 2024-04-25T00:26:10.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.599+0000 7f6f3d1d2640 1 -- 172.21.15.116:0/2729217434 wait complete. 2024-04-25T00:26:10.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.603+0000 7f6f3d1d2640 1 Processor -- start 2024-04-25T00:26:10.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.603+0000 7f6f3d1d2640 1 -- start start 2024-04-25T00:26:10.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.603+0000 7f6f3d1d2640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6f380c2f50 0x7f6f38150a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:10.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.603+0000 7f6f3d1d2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6f3813e310 0x7f6f381304b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:10.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.603+0000 7f6f3d1d2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f3813ed60 0x7f6f381309f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:10.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.603+0000 7f6f3d1d2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6f38163c10 con 0x7f6f3813ed60 2024-04-25T00:26:10.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.603+0000 7f6f3d1d2640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6f38071020 con 0x7f6f3813e310 2024-04-25T00:26:10.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.603+0000 7f6f3d1d2640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6f38163d90 con 0x7f6f380c2f50 2024-04-25T00:26:10.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.603+0000 7f6f3759e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f3813ed60 0x7f6f381309f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:10.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.603+0000 7f6f3759e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f3813ed60 0x7f6f381309f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37314/0 (socket says 172.21.15.116:37314) 2024-04-25T00:26:10.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.603+0000 7f6f3759e640 1 -- 172.21.15.116:0/2772521365 learned_addr learned my addr 172.21.15.116:0/2772521365 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:26:10.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.603+0000 7f6f36d9d640 1 --2- 172.21.15.116:0/2772521365 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6f380c2f50 0x7f6f38150a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:10.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.603+0000 7f6f3759e640 1 -- 172.21.15.116:0/2772521365 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6f380c2f50 msgr2=0x7f6f38150a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:10.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.603+0000 7f6f3759e640 1 --2- 172.21.15.116:0/2772521365 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6f380c2f50 0x7f6f38150a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:10.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.603+0000 7f6f3759e640 1 -- 172.21.15.116:0/2772521365 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6f3813e310 msgr2=0x7f6f381304b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:10.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.603+0000 7f6f3759e640 1 --2- 172.21.15.116:0/2772521365 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6f3813e310 0x7f6f381304b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:10.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.603+0000 7f6f3759e640 1 -- 172.21.15.116:0/2772521365 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f380730a0 con 0x7f6f3813ed60 2024-04-25T00:26:10.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.603+0000 7f6f3759e640 1 --2- 172.21.15.116:0/2772521365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f3813ed60 0x7f6f381309f0 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f6f2c00c410 tx=0x7f6f2c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:10.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.603+0000 7f6f1f7fe640 1 -- 172.21.15.116:0/2772521365 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f2c017070 con 0x7f6f3813ed60 2024-04-25T00:26:10.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.603+0000 7f6f1f7fe640 1 -- 172.21.15.116:0/2772521365 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f2c0052d0 con 0x7f6f3813ed60 2024-04-25T00:26:10.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.603+0000 7f6f3d1d2640 1 -- 172.21.15.116:0/2772521365 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f38151f70 con 0x7f6f3813ed60 2024-04-25T00:26:10.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.603+0000 7f6f1f7fe640 1 -- 172.21.15.116:0/2772521365 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f2c003800 con 0x7f6f3813ed60 2024-04-25T00:26:10.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.603+0000 7f6f3d1d2640 1 -- 172.21.15.116:0/2772521365 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f38138960 con 0x7f6f3813ed60 2024-04-25T00:26:10.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.607+0000 7f6f3d1d2640 1 -- 172.21.15.116:0/2772521365 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f380c66a0 con 0x7f6f3813ed60 2024-04-25T00:26:10.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.607+0000 7f6f1f7fe640 1 -- 172.21.15.116:0/2772521365 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6f2c01a070 con 0x7f6f3813ed60 2024-04-25T00:26:10.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.607+0000 7f6f1f7fe640 1 --2- 172.21.15.116:0/2772521365 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6f240678f0 0x7f6f24069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:10.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.607+0000 7f6f1f7fe640 1 -- 172.21.15.116:0/2772521365 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 10020+0+0 (secure 0 0 0) 0x7f6f2c086000 con 0x7f6f3813ed60 2024-04-25T00:26:10.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.607+0000 7f6f36d9d640 1 --2- 172.21.15.116:0/2772521365 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6f240678f0 0x7f6f24069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:10.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.607+0000 7f6f36d9d640 1 --2- 172.21.15.116:0/2772521365 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6f240678f0 0x7f6f24069db0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f6f280ad000 tx=0x7f6f280a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:10.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.615+0000 7f6f1f7fe640 1 -- 172.21.15.116:0/2772521365 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f6f380c66a0 con 0x7f6f3813ed60 2024-04-25T00:26:10.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.895+0000 7f6f3d1d2640 1 -- 172.21.15.116:0/2772521365 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6f38068350 con 0x7f6f3813ed60 2024-04-25T00:26:10.898 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:26:10.899 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":207,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6485,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/2371126056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2371126056},{"type":"v1","addr":"172.21.15.116:6835","nonce":2371126056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":207}],"filesystems":[{"mdsmap":{"epoch":201,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:25:29.028925+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6413},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6413":{"gid":6413,"name":"d","rank":0,"incarnation":198,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/4181466530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":4181466530},{"type":"v1","addr":"172.21.15.174:6837","nonce":4181466530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6413,"qdb_cluster":[6413]},"id":1},{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:25:45.002412+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8496":{"gid":8496,"name":"c","rank":0,"incarnation":203,"state":"up:active","state_seq":6,"addr":"172.21.15.116:6833/3693049068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3693049068},{"type":"v1","addr":"172.21.15.116:6833","nonce":3693049068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8496,"qdb_cluster":[8496]},"id":2}]} 2024-04-25T00:26:10.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.899+0000 7f6f1f7fe640 1 -- 172.21.15.116:0/2772521365 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 207 v207) ==== 95+0+6509 (secure 0 0 0) 0x7f6f2c005480 con 0x7f6f3813ed60 2024-04-25T00:26:10.899 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 207 2024-04-25T00:26:10.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.903+0000 7f6f3d1d2640 1 -- 172.21.15.116:0/2772521365 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6f240678f0 msgr2=0x7f6f24069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:10.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.903+0000 7f6f3d1d2640 1 --2- 172.21.15.116:0/2772521365 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6f240678f0 0x7f6f24069db0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f6f280ad000 tx=0x7f6f280a6000 comp rx=0 tx=0).stop 2024-04-25T00:26:10.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.903+0000 7f6f3d1d2640 1 -- 172.21.15.116:0/2772521365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f3813ed60 msgr2=0x7f6f381309f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:10.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.903+0000 7f6f3d1d2640 1 --2- 172.21.15.116:0/2772521365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f3813ed60 0x7f6f381309f0 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f6f2c00c410 tx=0x7f6f2c00c8d0 comp rx=0 tx=0).stop 2024-04-25T00:26:10.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.903+0000 7f6f3d1d2640 1 -- 172.21.15.116:0/2772521365 shutdown_connections 2024-04-25T00:26:10.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.903+0000 7f6f3d1d2640 1 --2- 172.21.15.116:0/2772521365 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6f240678f0 0x7f6f24069db0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:10.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.903+0000 7f6f3d1d2640 1 --2- 172.21.15.116:0/2772521365 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6f3813ed60 0x7f6f381309f0 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:10.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.903+0000 7f6f3d1d2640 1 --2- 172.21.15.116:0/2772521365 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6f3813e310 0x7f6f381304b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:10.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.903+0000 7f6f3d1d2640 1 --2- 172.21.15.116:0/2772521365 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6f380c2f50 0x7f6f38150a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:10.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.903+0000 7f6f3d1d2640 1 -- 172.21.15.116:0/2772521365 >> 172.21.15.116:0/2772521365 conn(0x7f6f380c3b00 msgr2=0x7f6f380c0430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:10.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.903+0000 7f6f3d1d2640 1 -- 172.21.15.116:0/2772521365 shutdown_connections 2024-04-25T00:26:10.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:10.903+0000 7f6f3d1d2640 1 -- 172.21.15.116:0/2772521365 wait complete. 2024-04-25T00:26:10.923 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:26:11.314 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:26:11.316 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:26:12.924 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:26:13.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.063+0000 7f1ed6dd5640 1 Processor -- start 2024-04-25T00:26:13.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.063+0000 7f1ed6dd5640 1 -- start start 2024-04-25T00:26:13.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.063+0000 7f1ed6dd5640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f1ed0152d80 0x7f1ed006bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:13.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.063+0000 7f1ed6dd5640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ed00c9150 con 0x7f1ed006c290 2024-04-25T00:26:13.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.063+0000 7f1ed6dd5640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ed00c9430 con 0x7f1ed006d5f0 2024-04-25T00:26:13.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.063+0000 7f1ed6dd5640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f1ed00c9d60 con 0x7f1ed0152d80 2024-04-25T00:26:13.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.063+0000 7f1ed4b71640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f1ed0152d80 0x7f1ed006bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:13.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.063+0000 7f1ed4b71640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f1ed0152d80 0x7f1ed006bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:47336/0 (socket says 172.21.15.116:47336) 2024-04-25T00:26:13.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.063+0000 7f1ed5372640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f1ed006c290 0x7f1ed0157390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:41172/0 (socket says 172.21.15.116:41172) 2024-04-25T00:26:13.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.063+0000 7f1ed4b71640 1 -- 172.21.15.116:0/2558484261 learned_addr learned my addr 172.21.15.116:0/2558484261 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:26:13.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.063+0000 7f1ed4b71640 1 -- 172.21.15.116:0/2558484261 >> v1:172.21.15.174:6789/0 conn(0x7f1ed006d5f0 legacy=0x7f1ed0156c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:13.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.063+0000 7f1ecf7fe640 1 -- 172.21.15.116:0/2558484261 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3677904355 0 0) 0x7f1ed00c9150 con 0x7f1ed006c290 2024-04-25T00:26:13.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.063+0000 7f1ed4b71640 1 -- 172.21.15.116:0/2558484261 >> v1:172.21.15.116:6789/0 conn(0x7f1ed006c290 legacy=0x7f1ed0157390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:13.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.063+0000 7f1ed4b71640 1 -- 172.21.15.116:0/2558484261 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ed00c9710 con 0x7f1ed0152d80 2024-04-25T00:26:13.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.071+0000 7f1ed4b71640 1 --2- 172.21.15.116:0/2558484261 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1ed0152d80 0x7f1ed006bd50 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f1ec80098a0 tx=0x7f1ec801c3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd3aeb541977b758 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:13.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.075+0000 7f1ecf7fe640 1 -- 172.21.15.116:0/2558484261 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec80b1070 con 0x7f1ed0152d80 2024-04-25T00:26:13.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.075+0000 7f1ecf7fe640 1 -- 172.21.15.116:0/2558484261 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ec80ac640 con 0x7f1ed0152d80 2024-04-25T00:26:13.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.075+0000 7f1ecf7fe640 1 -- 172.21.15.116:0/2558484261 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec80aca10 con 0x7f1ed0152d80 2024-04-25T00:26:13.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.075+0000 7f1ed6dd5640 1 -- 172.21.15.116:0/2558484261 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1ed0152d80 msgr2=0x7f1ed006bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:13.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.075+0000 7f1ed6dd5640 1 --2- 172.21.15.116:0/2558484261 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1ed0152d80 0x7f1ed006bd50 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f1ec80098a0 tx=0x7f1ec801c3f0 comp rx=0 tx=0).stop 2024-04-25T00:26:13.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.075+0000 7f1ed6dd5640 1 -- 172.21.15.116:0/2558484261 shutdown_connections 2024-04-25T00:26:13.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.075+0000 7f1ed6dd5640 1 --2- 172.21.15.116:0/2558484261 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1ed0152d80 0x7f1ed006bd50 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:13.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.075+0000 7f1ed6dd5640 1 -- 172.21.15.116:0/2558484261 >> 172.21.15.116:0/2558484261 conn(0x7f1ed00acdd0 msgr2=0x7f1ed00ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:13.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.075+0000 7f1ed6dd5640 1 -- 172.21.15.116:0/2558484261 shutdown_connections 2024-04-25T00:26:13.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.075+0000 7f1ed6dd5640 1 -- 172.21.15.116:0/2558484261 wait complete. 2024-04-25T00:26:13.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.075+0000 7f1ed6dd5640 1 Processor -- start 2024-04-25T00:26:13.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.075+0000 7f1ed6dd5640 1 -- start start 2024-04-25T00:26:13.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.075+0000 7f1ed6dd5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ed006c290 0x7f1ed00abd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:13.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.075+0000 7f1ed6dd5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1ed006d5f0 0x7f1ed014c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:13.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.075+0000 7f1ed6dd5640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1ed0152d80 0x7f1ed014ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:13.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.075+0000 7f1ed6dd5640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1ed0163a60 con 0x7f1ed006c290 2024-04-25T00:26:13.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.075+0000 7f1ed6dd5640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1ed01638e0 con 0x7f1ed006d5f0 2024-04-25T00:26:13.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.075+0000 7f1ed6dd5640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f1ed0163be0 con 0x7f1ed0152d80 2024-04-25T00:26:13.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.075+0000 7f1ed4b71640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ed006c290 0x7f1ed00abd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:13.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.075+0000 7f1ed4b71640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ed006c290 0x7f1ed00abd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37322/0 (socket says 172.21.15.116:37322) 2024-04-25T00:26:13.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.075+0000 7f1ed4b71640 1 -- 172.21.15.116:0/1734679876 learned_addr learned my addr 172.21.15.116:0/1734679876 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:26:13.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.075+0000 7f1ecffff640 1 --2- 172.21.15.116:0/1734679876 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1ed006d5f0 0x7f1ed014c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:13.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.079+0000 7f1ed4b71640 1 -- 172.21.15.116:0/1734679876 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1ed0152d80 msgr2=0x7f1ed014ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:13.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.079+0000 7f1ed4b71640 1 --2- 172.21.15.116:0/1734679876 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1ed0152d80 0x7f1ed014ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:13.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.079+0000 7f1ed4b71640 1 -- 172.21.15.116:0/1734679876 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1ed006d5f0 msgr2=0x7f1ed014c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:13.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.079+0000 7f1ed4b71640 1 --2- 172.21.15.116:0/1734679876 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1ed006d5f0 0x7f1ed014c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:13.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.079+0000 7f1ed4b71640 1 -- 172.21.15.116:0/1734679876 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ed01f2240 con 0x7f1ed006c290 2024-04-25T00:26:13.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.079+0000 7f1ed4b71640 1 --2- 172.21.15.116:0/1734679876 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ed006c290 0x7f1ed00abd20 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f1ec801a020 tx=0x7f1ec8019190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:13.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.079+0000 7f1ecd7fa640 1 -- 172.21.15.116:0/1734679876 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec80b1050 con 0x7f1ed006c290 2024-04-25T00:26:13.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.079+0000 7f1ecd7fa640 1 -- 172.21.15.116:0/1734679876 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ec80bb070 con 0x7f1ed006c290 2024-04-25T00:26:13.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.079+0000 7f1ecd7fa640 1 -- 172.21.15.116:0/1734679876 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ec80b59f0 con 0x7f1ed006c290 2024-04-25T00:26:13.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.079+0000 7f1ed6dd5640 1 -- 172.21.15.116:0/1734679876 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ed01f2520 con 0x7f1ed006c290 2024-04-25T00:26:13.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.079+0000 7f1ed6dd5640 1 -- 172.21.15.116:0/1734679876 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ed014cf80 con 0x7f1ed006c290 2024-04-25T00:26:13.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.079+0000 7f1ed6dd5640 1 -- 172.21.15.116:0/1734679876 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1eb40052a0 con 0x7f1ed006c290 2024-04-25T00:26:13.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.079+0000 7f1ecd7fa640 1 -- 172.21.15.116:0/1734679876 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1ec80b5b90 con 0x7f1ed006c290 2024-04-25T00:26:13.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.083+0000 7f1ecd7fa640 1 --2- 172.21.15.116:0/1734679876 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1eb0067940 0x7f1eb0069e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:13.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.083+0000 7f1ecd7fa640 1 -- 172.21.15.116:0/1734679876 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 10020+0+0 (secure 0 0 0) 0x7f1ec8120f90 con 0x7f1ed006c290 2024-04-25T00:26:13.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.083+0000 7f1ecffff640 1 --2- 172.21.15.116:0/1734679876 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1eb0067940 0x7f1eb0069e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:13.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.083+0000 7f1ecffff640 1 --2- 172.21.15.116:0/1734679876 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1eb0067940 0x7f1eb0069e00 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f1ed01ef230 tx=0x7f1eb8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:13.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.087+0000 7f1ecd7fa640 1 -- 172.21.15.116:0/1734679876 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f1ec8016130 con 0x7f1ed006c290 2024-04-25T00:26:13.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.375+0000 7f1ed6dd5640 1 -- 172.21.15.116:0/1734679876 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1eb4005590 con 0x7f1ed006c290 2024-04-25T00:26:13.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.375+0000 7f1ecd7fa640 1 -- 172.21.15.116:0/1734679876 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 207 v207) ==== 95+0+6509 (secure 0 0 0) 0x7f1ec80f09c0 con 0x7f1ed006c290 2024-04-25T00:26:13.376 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:26:13.377 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":207,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6485,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/2371126056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2371126056},{"type":"v1","addr":"172.21.15.116:6835","nonce":2371126056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":207}],"filesystems":[{"mdsmap":{"epoch":201,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:25:29.028925+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6413},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6413":{"gid":6413,"name":"d","rank":0,"incarnation":198,"state":"up:active","state_seq":18,"addr":"172.21.15.174:6837/4181466530","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":4181466530},{"type":"v1","addr":"172.21.15.174:6837","nonce":4181466530}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6413,"qdb_cluster":[6413]},"id":1},{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:25:45.002412+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8496":{"gid":8496,"name":"c","rank":0,"incarnation":203,"state":"up:active","state_seq":6,"addr":"172.21.15.116:6833/3693049068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3693049068},{"type":"v1","addr":"172.21.15.116:6833","nonce":3693049068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8496,"qdb_cluster":[8496]},"id":2}]} 2024-04-25T00:26:13.377 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 207 2024-04-25T00:26:13.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.379+0000 7f1ed6dd5640 1 -- 172.21.15.116:0/1734679876 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1eb0067940 msgr2=0x7f1eb0069e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:13.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.379+0000 7f1ed6dd5640 1 --2- 172.21.15.116:0/1734679876 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1eb0067940 0x7f1eb0069e00 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f1ed01ef230 tx=0x7f1eb8009040 comp rx=0 tx=0).stop 2024-04-25T00:26:13.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.379+0000 7f1ed6dd5640 1 -- 172.21.15.116:0/1734679876 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ed006c290 msgr2=0x7f1ed00abd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:13.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.379+0000 7f1ed6dd5640 1 --2- 172.21.15.116:0/1734679876 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ed006c290 0x7f1ed00abd20 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f1ec801a020 tx=0x7f1ec8019190 comp rx=0 tx=0).stop 2024-04-25T00:26:13.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.379+0000 7f1ed6dd5640 1 -- 172.21.15.116:0/1734679876 shutdown_connections 2024-04-25T00:26:13.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.379+0000 7f1ed6dd5640 1 --2- 172.21.15.116:0/1734679876 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1eb0067940 0x7f1eb0069e00 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:13.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.379+0000 7f1ed6dd5640 1 --2- 172.21.15.116:0/1734679876 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1ed0152d80 0x7f1ed014ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:13.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.379+0000 7f1ed6dd5640 1 --2- 172.21.15.116:0/1734679876 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1ed006d5f0 0x7f1ed014c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:13.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.379+0000 7f1ed6dd5640 1 --2- 172.21.15.116:0/1734679876 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1ed006c290 0x7f1ed00abd20 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:13.380 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.379+0000 7f1ed6dd5640 1 -- 172.21.15.116:0/1734679876 >> 172.21.15.116:0/1734679876 conn(0x7f1ed00acdd0 msgr2=0x7f1ed014fb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:13.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.379+0000 7f1ed6dd5640 1 -- 172.21.15.116:0/1734679876 shutdown_connections 2024-04-25T00:26:13.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:13.383+0000 7f1ed6dd5640 1 -- 172.21.15.116:0/1734679876 wait complete. 2024-04-25T00:26:13.403 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:26:15.404 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:26:15.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.559+0000 7fe13f882640 1 Processor -- start 2024-04-25T00:26:15.559 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.559+0000 7fe13f882640 1 -- start start 2024-04-25T00:26:15.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.559+0000 7fe13f882640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe138152d80 0x7fe13806bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:15.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.559+0000 7fe13f882640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fe13806c290 0x7fe138157390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:15.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.559+0000 7fe13f882640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1380c9150 con 0x7fe13806d5f0 2024-04-25T00:26:15.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.559+0000 7fe13f882640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fe1380c9d60 con 0x7fe13806c290 2024-04-25T00:26:15.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.559+0000 7fe13f882640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe1380c9430 con 0x7fe138152d80 2024-04-25T00:26:15.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.559+0000 7fe13ce1d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fe13806d5f0 0x7fe138156c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:56954/0 (socket says 172.21.15.116:56954) 2024-04-25T00:26:15.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.559+0000 7fe13ce1d640 1 -- 172.21.15.116:0/3262869484 learned_addr learned my addr 172.21.15.116:0/3262869484 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:26:15.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.559+0000 7fe13d61e640 1 --2- 172.21.15.116:0/3262869484 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe138152d80 0x7fe13806bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:15.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.559+0000 7fe13de1f640 1 --2- 172.21.15.116:0/3262869484 >> v2:172.21.15.174:3300/0 conn(0x7fe13806c290 0x7fe138157390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:15.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.559+0000 7fe12ffff640 1 -- 172.21.15.116:0/3262869484 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2437226828 0 0) 0x7fe1380c9150 con 0x7fe13806d5f0 2024-04-25T00:26:15.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.559+0000 7fe12ffff640 1 -- 172.21.15.116:0/3262869484 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe124003530 con 0x7fe13806d5f0 2024-04-25T00:26:15.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.559+0000 7fe13de1f640 1 -- 172.21.15.116:0/3262869484 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe138152d80 msgr2=0x7fe13806bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:15.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.559+0000 7fe13de1f640 1 --2- 172.21.15.116:0/3262869484 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe138152d80 0x7fe13806bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:15.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.559+0000 7fe13de1f640 1 -- 172.21.15.116:0/3262869484 >> v1:172.21.15.116:6789/0 conn(0x7fe13806d5f0 legacy=0x7fe138156c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:15.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.559+0000 7fe12ffff640 1 -- 172.21.15.116:0/3262869484 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2135276259 0 0) 0x7fe124003530 con 0x7fe13806d5f0 2024-04-25T00:26:15.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.559+0000 7fe13de1f640 1 -- 172.21.15.116:0/3262869484 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1380c9710 con 0x7fe13806c290 2024-04-25T00:26:15.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.559+0000 7fe13d61e640 1 --2- 172.21.15.116:0/3262869484 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe138152d80 0x7fe13806bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:26:15.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.563+0000 7fe13de1f640 1 --2- 172.21.15.116:0/3262869484 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe13806c290 0x7fe138157390 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fe130009d90 tx=0x7fe13001c7d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9a74dc2272a81e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:15.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.563+0000 7fe12ffff640 1 -- 172.21.15.116:0/3262869484 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1300b2070 con 0x7fe13806c290 2024-04-25T00:26:15.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.563+0000 7fe12ffff640 1 -- 172.21.15.116:0/3262869484 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1300a55f0 con 0x7fe13806c290 2024-04-25T00:26:15.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.563+0000 7fe12ffff640 1 -- 172.21.15.116:0/3262869484 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe128001680 con 0x7fe13806c290 2024-04-25T00:26:15.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.563+0000 7fe13f882640 1 -- 172.21.15.116:0/3262869484 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe13806c290 msgr2=0x7fe138157390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:15.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.563+0000 7fe13f882640 1 --2- 172.21.15.116:0/3262869484 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe13806c290 0x7fe138157390 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fe130009d90 tx=0x7fe13001c7d0 comp rx=0 tx=0).stop 2024-04-25T00:26:15.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.563+0000 7fe13f882640 1 -- 172.21.15.116:0/3262869484 shutdown_connections 2024-04-25T00:26:15.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.563+0000 7fe13f882640 1 --2- 172.21.15.116:0/3262869484 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe13806c290 0x7fe138157390 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:15.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.563+0000 7fe13f882640 1 --2- 172.21.15.116:0/3262869484 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe138152d80 0x7fe13806bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:15.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.563+0000 7fe13f882640 1 -- 172.21.15.116:0/3262869484 >> 172.21.15.116:0/3262869484 conn(0x7fe1380acdd0 msgr2=0x7fe1380ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:15.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.563+0000 7fe13f882640 1 -- 172.21.15.116:0/3262869484 shutdown_connections 2024-04-25T00:26:15.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.567+0000 7fe13f882640 1 -- 172.21.15.116:0/3262869484 wait complete. 2024-04-25T00:26:15.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.567+0000 7fe13f882640 1 Processor -- start 2024-04-25T00:26:15.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.571+0000 7fe13f882640 1 -- start start 2024-04-25T00:26:15.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.571+0000 7fe13f882640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe13806c290 0x7fe138147190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:15.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.571+0000 7fe13f882640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe13806d5f0 0x7fe1381eea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:15.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.571+0000 7fe13f882640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe138152d80 0x7fe138148490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:15.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.571+0000 7fe13f882640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe1380caf00 con 0x7fe13806d5f0 2024-04-25T00:26:15.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.571+0000 7fe13f882640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe1380cad80 con 0x7fe138152d80 2024-04-25T00:26:15.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.571+0000 7fe13f882640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe1381628d0 con 0x7fe13806c290 2024-04-25T00:26:15.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.571+0000 7fe13de1f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe138152d80 0x7fe138148490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:15.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.571+0000 7fe13de1f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe138152d80 0x7fe138148490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:59826/0 (socket says 172.21.15.116:59826) 2024-04-25T00:26:15.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.571+0000 7fe13de1f640 1 -- 172.21.15.116:0/3159691148 learned_addr learned my addr 172.21.15.116:0/3159691148 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:26:15.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.571+0000 7fe13ce1d640 1 --2- 172.21.15.116:0/3159691148 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe13806d5f0 0x7fe1381eea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:15.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.571+0000 7fe13d61e640 1 --2- 172.21.15.116:0/3159691148 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe13806c290 0x7fe138147190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:15.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.571+0000 7fe13de1f640 1 -- 172.21.15.116:0/3159691148 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe13806c290 msgr2=0x7fe138147190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:15.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.571+0000 7fe13de1f640 1 --2- 172.21.15.116:0/3159691148 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe13806c290 0x7fe138147190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:15.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.571+0000 7fe13de1f640 1 -- 172.21.15.116:0/3159691148 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe13806d5f0 msgr2=0x7fe1381eea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:15.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.571+0000 7fe13de1f640 1 --2- 172.21.15.116:0/3159691148 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe13806d5f0 0x7fe1381eea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:15.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.571+0000 7fe13de1f640 1 -- 172.21.15.116:0/3159691148 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe138149ba0 con 0x7fe138152d80 2024-04-25T00:26:15.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.571+0000 7fe13ce1d640 1 --2- 172.21.15.116:0/3159691148 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe13806d5f0 0x7fe1381eea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:26:15.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.575+0000 7fe13de1f640 1 --2- 172.21.15.116:0/3159691148 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe138152d80 0x7fe138148490 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fe13001d070 tx=0x7fe130019570 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:15.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.575+0000 7fe12dffb640 1 -- 172.21.15.116:0/3159691148 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1300b2050 con 0x7fe138152d80 2024-04-25T00:26:15.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.575+0000 7fe12dffb640 1 -- 172.21.15.116:0/3159691148 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1300be070 con 0x7fe138152d80 2024-04-25T00:26:15.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.575+0000 7fe13f882640 1 -- 172.21.15.116:0/3159691148 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe138149e80 con 0x7fe138152d80 2024-04-25T00:26:15.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.575+0000 7fe12dffb640 1 -- 172.21.15.116:0/3159691148 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe130009450 con 0x7fe138152d80 2024-04-25T00:26:15.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.575+0000 7fe13f882640 1 -- 172.21.15.116:0/3159691148 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe138155090 con 0x7fe138152d80 2024-04-25T00:26:15.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.575+0000 7fe13f882640 1 -- 172.21.15.116:0/3159691148 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe138068350 con 0x7fe138152d80 2024-04-25T00:26:15.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.575+0000 7fe12dffb640 1 -- 172.21.15.116:0/3159691148 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe1300b9850 con 0x7fe138152d80 2024-04-25T00:26:15.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.575+0000 7fe12dffb640 1 --2- 172.21.15.116:0/3159691148 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe11c0678c0 0x7fe11c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:15.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.575+0000 7fe12dffb640 1 -- 172.21.15.116:0/3159691148 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 10020+0+0 (secure 0 0 0) 0x7fe130122720 con 0x7fe138152d80 2024-04-25T00:26:15.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.575+0000 7fe13d61e640 1 --2- 172.21.15.116:0/3159691148 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe11c0678c0 0x7fe11c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:15.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.579+0000 7fe13d61e640 1 --2- 172.21.15.116:0/3159691148 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe11c0678c0 0x7fe11c069d80 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fe134009580 tx=0x7fe134008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:15.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.583+0000 7fe12dffb640 1 -- 172.21.15.116:0/3159691148 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fe130016480 con 0x7fe138152d80 2024-04-25T00:26:15.944 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.943+0000 7fe13f882640 1 -- 172.21.15.116:0/3159691148 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe138159120 con 0x7fe138152d80 2024-04-25T00:26:15.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.943+0000 7fe12dffb640 1 -- 172.21.15.116:0/3159691148 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) ==== 95+0+5672 (secure 0 0 0) 0x7fe1300f21e0 con 0x7fe138152d80 2024-04-25T00:26:15.945 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:26:15.945 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":208,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:26:15.734944+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6485},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6485":{"gid":6485,"name":"a","rank":0,"incarnation":208,"state":"up:replay","state_seq":1,"addr":"172.21.15.116:6835/2371126056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2371126056},{"type":"v1","addr":"172.21.15.116:6835","nonce":2371126056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:25:45.002412+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8496":{"gid":8496,"name":"c","rank":0,"incarnation":203,"state":"up:active","state_seq":6,"addr":"172.21.15.116:6833/3693049068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3693049068},{"type":"v1","addr":"172.21.15.116:6833","nonce":3693049068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8496,"qdb_cluster":[8496]},"id":2}]} 2024-04-25T00:26:15.945 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 208 2024-04-25T00:26:15.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.951+0000 7fe13f882640 1 -- 172.21.15.116:0/3159691148 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe11c0678c0 msgr2=0x7fe11c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:15.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.951+0000 7fe13f882640 1 --2- 172.21.15.116:0/3159691148 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe11c0678c0 0x7fe11c069d80 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fe134009580 tx=0x7fe134008040 comp rx=0 tx=0).stop 2024-04-25T00:26:15.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.951+0000 7fe13f882640 1 -- 172.21.15.116:0/3159691148 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe138152d80 msgr2=0x7fe138148490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:15.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.951+0000 7fe13f882640 1 --2- 172.21.15.116:0/3159691148 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe138152d80 0x7fe138148490 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fe13001d070 tx=0x7fe130019570 comp rx=0 tx=0).stop 2024-04-25T00:26:15.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.951+0000 7fe13f882640 1 -- 172.21.15.116:0/3159691148 shutdown_connections 2024-04-25T00:26:15.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.951+0000 7fe13f882640 1 --2- 172.21.15.116:0/3159691148 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe11c0678c0 0x7fe11c069d80 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:15.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.951+0000 7fe13f882640 1 --2- 172.21.15.116:0/3159691148 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe138152d80 0x7fe138148490 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:15.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.951+0000 7fe13f882640 1 --2- 172.21.15.116:0/3159691148 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe13806d5f0 0x7fe1381eea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:15.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.951+0000 7fe13f882640 1 --2- 172.21.15.116:0/3159691148 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe13806c290 0x7fe138147190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:15.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.951+0000 7fe13f882640 1 -- 172.21.15.116:0/3159691148 >> 172.21.15.116:0/3159691148 conn(0x7fe1380acdd0 msgr2=0x7fe13814d650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:15.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.951+0000 7fe13f882640 1 -- 172.21.15.116:0/3159691148 shutdown_connections 2024-04-25T00:26:15.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:15.951+0000 7fe13f882640 1 -- 172.21.15.116:0/3159691148 wait complete. 2024-04-25T00:26:15.979 INFO:tasks.mds_thrash.fs.[a]:mds.d down, removed from mdsmap 2024-04-25T00:26:15.979 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-04-25T00:26:15.980 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:26:16.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.171+0000 7fafa3fa5640 1 Processor -- start 2024-04-25T00:26:16.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.171+0000 7fafa3fa5640 1 -- start start 2024-04-25T00:26:16.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.171+0000 7fafa3fa5640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7faf9c0c2b10 0x7faf9c1541f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:16.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.171+0000 7fafa3fa5640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faf9c06bc20 0x7faf9c0c3260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:16.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.171+0000 7fafa3fa5640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf9c0734b0 con 0x7faf9c150d10 2024-04-25T00:26:16.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.171+0000 7fafa3fa5640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7faf9c153750 con 0x7faf9c0c2b10 2024-04-25T00:26:16.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.171+0000 7fafa3fa5640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7faf9c0a9bb0 con 0x7faf9c06bc20 2024-04-25T00:26:16.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.171+0000 7fafa1d41640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7faf9c150d10 0x7faf9c0a94a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:56966/0 (socket says 172.21.15.116:56966) 2024-04-25T00:26:16.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.171+0000 7fafa1d41640 1 -- 172.21.15.116:0/454064151 learned_addr learned my addr 172.21.15.116:0/454064151 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:26:16.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.171+0000 7fafa1540640 1 --2- 172.21.15.116:0/454064151 >> v2:172.21.15.174:3300/0 conn(0x7faf9c0c2b10 0x7faf9c1541f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:16.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.171+0000 7fafa2542640 1 --2- 172.21.15.116:0/454064151 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faf9c06bc20 0x7faf9c0c3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:16.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.171+0000 7fafa0d3f640 1 -- 172.21.15.116:0/454064151 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2429617487 0 0) 0x7faf9c0734b0 con 0x7faf9c150d10 2024-04-25T00:26:16.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.171+0000 7fafa0d3f640 1 -- 172.21.15.116:0/454064151 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faf88003530 con 0x7faf9c150d10 2024-04-25T00:26:16.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.175+0000 7fafa1540640 1 -- 172.21.15.116:0/454064151 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faf9c06bc20 msgr2=0x7faf9c0c3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:16.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.175+0000 7fafa1540640 1 --2- 172.21.15.116:0/454064151 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faf9c06bc20 0x7faf9c0c3260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:16.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.175+0000 7fafa1540640 1 -- 172.21.15.116:0/454064151 >> v1:172.21.15.116:6789/0 conn(0x7faf9c150d10 legacy=0x7faf9c0a94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:16.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.175+0000 7fafa1540640 1 -- 172.21.15.116:0/454064151 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf9c0731d0 con 0x7faf9c0c2b10 2024-04-25T00:26:16.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.175+0000 7fafa1540640 1 --2- 172.21.15.116:0/454064151 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf9c0c2b10 0x7faf9c1541f0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7faf8c0097c0 tx=0x7faf8c01c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61212d9c8deff09d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:16.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.175+0000 7fafa0d3f640 1 -- 172.21.15.116:0/454064151 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf8c0b0070 con 0x7faf9c0c2b10 2024-04-25T00:26:16.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.175+0000 7fafa0d3f640 1 -- 172.21.15.116:0/454064151 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf8c0ab600 con 0x7faf9c0c2b10 2024-04-25T00:26:16.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.175+0000 7fafa0d3f640 1 -- 172.21.15.116:0/454064151 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf8c0ab9d0 con 0x7faf9c0c2b10 2024-04-25T00:26:16.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.179+0000 7fafa3fa5640 1 -- 172.21.15.116:0/454064151 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf9c0c2b10 msgr2=0x7faf9c1541f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:16.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.179+0000 7fafa3fa5640 1 --2- 172.21.15.116:0/454064151 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf9c0c2b10 0x7faf9c1541f0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7faf8c0097c0 tx=0x7faf8c01c410 comp rx=0 tx=0).stop 2024-04-25T00:26:16.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.179+0000 7fafa3fa5640 1 -- 172.21.15.116:0/454064151 shutdown_connections 2024-04-25T00:26:16.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.179+0000 7fafa3fa5640 1 --2- 172.21.15.116:0/454064151 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faf9c06bc20 0x7faf9c0c3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:16.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.179+0000 7fafa3fa5640 1 --2- 172.21.15.116:0/454064151 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf9c0c2b10 0x7faf9c1541f0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:16.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.179+0000 7fafa3fa5640 1 -- 172.21.15.116:0/454064151 >> 172.21.15.116:0/454064151 conn(0x7faf9c0c4c10 msgr2=0x7faf9c0c5020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:16.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.179+0000 7fafa3fa5640 1 -- 172.21.15.116:0/454064151 shutdown_connections 2024-04-25T00:26:16.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.179+0000 7fafa3fa5640 1 -- 172.21.15.116:0/454064151 wait complete. 2024-04-25T00:26:16.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.179+0000 7fafa3fa5640 1 Processor -- start 2024-04-25T00:26:16.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.179+0000 7fafa3fa5640 1 -- start start 2024-04-25T00:26:16.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.183+0000 7fafa3fa5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf9c06bc20 0x7faf9c1eee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:16.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.183+0000 7fafa3fa5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf9c0c2b10 0x7faf9c1f2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:16.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.183+0000 7fafa3fa5640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faf9c150d10 0x7faf9c06dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:16.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.183+0000 7fafa3fa5640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7faf9c151270 con 0x7faf9c0c2b10 2024-04-25T00:26:16.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.183+0000 7fafa3fa5640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7faf9c1510f0 con 0x7faf9c06bc20 2024-04-25T00:26:16.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.183+0000 7fafa3fa5640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7faf9c160750 con 0x7faf9c150d10 2024-04-25T00:26:16.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.183+0000 7fafa1d41640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf9c06bc20 0x7faf9c1eee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:16.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.183+0000 7fafa1d41640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf9c06bc20 0x7faf9c1eee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:59842/0 (socket says 172.21.15.116:59842) 2024-04-25T00:26:16.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.183+0000 7fafa1d41640 1 -- 172.21.15.116:0/768721620 learned_addr learned my addr 172.21.15.116:0/768721620 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:26:16.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.183+0000 7fafa2542640 1 --2- 172.21.15.116:0/768721620 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faf9c150d10 0x7faf9c06dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:16.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.183+0000 7fafa1d41640 1 -- 172.21.15.116:0/768721620 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faf9c150d10 msgr2=0x7faf9c06dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:16.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.183+0000 7fafa1d41640 1 --2- 172.21.15.116:0/768721620 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faf9c150d10 0x7faf9c06dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:16.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.183+0000 7fafa1d41640 1 -- 172.21.15.116:0/768721620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf9c0c2b10 msgr2=0x7faf9c1f2300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T00:26:16.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.183+0000 7fafa1d41640 1 --2- 172.21.15.116:0/768721620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf9c0c2b10 0x7faf9c1f2300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:16.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.183+0000 7fafa1d41640 1 -- 172.21.15.116:0/768721620 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf9c0ad5f0 con 0x7faf9c06bc20 2024-04-25T00:26:16.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.183+0000 7fafa1d41640 1 --2- 172.21.15.116:0/768721620 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf9c06bc20 0x7faf9c1eee90 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7faf9800c450 tx=0x7faf9800c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:16.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.183+0000 7faf927fc640 1 -- 172.21.15.116:0/768721620 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf98017070 con 0x7faf9c06bc20 2024-04-25T00:26:16.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.183+0000 7faf927fc640 1 -- 172.21.15.116:0/768721620 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf98005db0 con 0x7faf9c06bc20 2024-04-25T00:26:16.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.183+0000 7faf927fc640 1 -- 172.21.15.116:0/768721620 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf98007290 con 0x7faf9c06bc20 2024-04-25T00:26:16.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.183+0000 7fafa3fa5640 1 -- 172.21.15.116:0/768721620 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf9c0ad8d0 con 0x7faf9c06bc20 2024-04-25T00:26:16.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.183+0000 7fafa3fa5640 1 -- 172.21.15.116:0/768721620 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf9c1ee380 con 0x7faf9c06bc20 2024-04-25T00:26:16.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.183+0000 7fafa3fa5640 1 -- 172.21.15.116:0/768721620 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf9c068a60 con 0x7faf9c06bc20 2024-04-25T00:26:16.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.187+0000 7faf927fc640 1 -- 172.21.15.116:0/768721620 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7faf9801a070 con 0x7faf9c06bc20 2024-04-25T00:26:16.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.187+0000 7faf927fc640 1 --2- 172.21.15.116:0/768721620 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faf80067910 0x7faf80069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:16.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.187+0000 7faf927fc640 1 -- 172.21.15.116:0/768721620 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 10106+0+0 (secure 0 0 0) 0x7faf980861a0 con 0x7faf9c06bc20 2024-04-25T00:26:16.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.187+0000 7fafa1540640 1 --2- 172.21.15.116:0/768721620 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faf80067910 0x7faf80069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:16.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.191+0000 7fafa1540640 1 --2- 172.21.15.116:0/768721620 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faf80067910 0x7faf80069dd0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7faf8c0ad000 tx=0x7faf8c0a4f90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:16.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.195+0000 7faf927fc640 1 -- 172.21.15.116:0/768721620 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7faf98055620 con 0x7faf9c06bc20 2024-04-25T00:26:16.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.483+0000 7fafa3fa5640 1 -- 172.21.15.116:0/768721620 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faf9c0c2540 con 0x7faf9c06bc20 2024-04-25T00:26:16.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.483+0000 7faf927fc640 1 -- 172.21.15.116:0/768721620 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) ==== 95+0+5672 (secure 0 0 0) 0x7faf98059920 con 0x7faf9c06bc20 2024-04-25T00:26:16.485 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:26:16.485 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":208,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:26:15.734944+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6485},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6485":{"gid":6485,"name":"a","rank":0,"incarnation":208,"state":"up:replay","state_seq":1,"addr":"172.21.15.116:6835/2371126056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2371126056},{"type":"v1","addr":"172.21.15.116:6835","nonce":2371126056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:25:45.002412+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8496":{"gid":8496,"name":"c","rank":0,"incarnation":203,"state":"up:active","state_seq":6,"addr":"172.21.15.116:6833/3693049068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3693049068},{"type":"v1","addr":"172.21.15.116:6833","nonce":3693049068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8496,"qdb_cluster":[8496]},"id":2}]} 2024-04-25T00:26:16.485 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 208 2024-04-25T00:26:16.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.495+0000 7fafa3fa5640 1 -- 172.21.15.116:0/768721620 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faf80067910 msgr2=0x7faf80069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:16.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.495+0000 7fafa3fa5640 1 --2- 172.21.15.116:0/768721620 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faf80067910 0x7faf80069dd0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7faf8c0ad000 tx=0x7faf8c0a4f90 comp rx=0 tx=0).stop 2024-04-25T00:26:16.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.495+0000 7fafa3fa5640 1 -- 172.21.15.116:0/768721620 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf9c06bc20 msgr2=0x7faf9c1eee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:16.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.495+0000 7fafa3fa5640 1 --2- 172.21.15.116:0/768721620 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf9c06bc20 0x7faf9c1eee90 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7faf9800c450 tx=0x7faf9800c910 comp rx=0 tx=0).stop 2024-04-25T00:26:16.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.495+0000 7fafa3fa5640 1 -- 172.21.15.116:0/768721620 shutdown_connections 2024-04-25T00:26:16.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.495+0000 7fafa3fa5640 1 --2- 172.21.15.116:0/768721620 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faf80067910 0x7faf80069dd0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:16.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.495+0000 7fafa3fa5640 1 --2- 172.21.15.116:0/768721620 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faf9c150d10 0x7faf9c06dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:16.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.495+0000 7fafa3fa5640 1 --2- 172.21.15.116:0/768721620 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf9c0c2b10 0x7faf9c1f2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:16.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.495+0000 7fafa3fa5640 1 --2- 172.21.15.116:0/768721620 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf9c06bc20 0x7faf9c1eee90 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:16.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.495+0000 7fafa3fa5640 1 -- 172.21.15.116:0/768721620 >> 172.21.15.116:0/768721620 conn(0x7faf9c0c4c10 msgr2=0x7faf9c0aa2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:16.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.495+0000 7fafa3fa5640 1 -- 172.21.15.116:0/768721620 shutdown_connections 2024-04-25T00:26:16.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:16.495+0000 7fafa3fa5640 1 -- 172.21.15.116:0/768721620 wait complete. 2024-04-25T00:26:16.515 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 208, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:04.678737+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 208, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6485": { "addr": "172.21.15.116:6835/2371126056", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6834", "nonce": 2371126056, "type": "v2" }, { "addr": "172.21.15.116:6835", "nonce": 2371126056, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6485, "incarnation": 208, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 68, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T00:26:15.734944+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6485 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:18.825887+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 206, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_8496": { "addr": "172.21.15.116:6833/3693049068", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6832", "nonce": 3693049068, "type": "v2" }, { "addr": "172.21.15.116:6833", "nonce": 3693049068, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8496, "incarnation": 203, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 6 } }, "last_failure": 0, "last_failure_osd_epoch": 67, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-04-25T00:25:45.002412+0000", "qdb_cluster": [ 8496 ], "qdb_leader": 8496, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8496 } } } ], "standbys": [ { "addr": "172.21.15.174:6835/4112145782", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6834", "nonce": 4112145782, "type": "v2" }, { "addr": "172.21.15.174:6835", "nonce": 4112145782, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 112, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5582, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-25T00:26:18.517 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:26:18.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.663+0000 7ff1f5003640 1 Processor -- start 2024-04-25T00:26:18.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.667+0000 7ff1f5003640 1 -- start start 2024-04-25T00:26:18.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.667+0000 7ff1f5003640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff1f0152d80 0x7ff1f006bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:18.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.667+0000 7ff1f5003640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff1f006d5f0 0x7ff1f0156c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:18.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.667+0000 7ff1f5003640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1f00c9150 con 0x7ff1f006c290 2024-04-25T00:26:18.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.667+0000 7ff1f5003640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7ff1f00c9d60 con 0x7ff1f0152d80 2024-04-25T00:26:18.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.667+0000 7ff1f5003640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff1f00c9430 con 0x7ff1f006d5f0 2024-04-25T00:26:18.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.667+0000 7ff1eed9d640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff1f0152d80 0x7ff1f006bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:18.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.667+0000 7ff1eed9d640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff1f0152d80 0x7ff1f006bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:59844/0 (socket says 172.21.15.116:59844) 2024-04-25T00:26:18.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.667+0000 7ff1eed9d640 1 -- 172.21.15.116:0/2366490543 learned_addr learned my addr 172.21.15.116:0/2366490543 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:26:18.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.675+0000 7ff1eed9d640 1 -- 172.21.15.116:0/2366490543 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff1f006d5f0 msgr2=0x7ff1f0156c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:18.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.675+0000 7ff1eed9d640 1 --2- 172.21.15.116:0/2366490543 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff1f006d5f0 0x7ff1f0156c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:18.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.675+0000 7ff1eed9d640 1 -- 172.21.15.116:0/2366490543 >> v1:172.21.15.174:6789/0 conn(0x7ff1f006c290 legacy=0x7ff1f01571c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:18.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.675+0000 7ff1eed9d640 1 -- 172.21.15.116:0/2366490543 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1f00c9710 con 0x7ff1f0152d80 2024-04-25T00:26:18.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.679+0000 7ff1eed9d640 1 --2- 172.21.15.116:0/2366490543 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1f0152d80 0x7ff1f006bd50 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7ff1e0009870 tx=0x7ff1e001c460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e121a6fb4c4bdf0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:18.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.679+0000 7ff1edd9b640 1 -- 172.21.15.116:0/2366490543 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e00ab640 con 0x7ff1f0152d80 2024-04-25T00:26:18.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.679+0000 7ff1edd9b640 1 -- 172.21.15.116:0/2366490543 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1e00b0070 con 0x7ff1f0152d80 2024-04-25T00:26:18.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.679+0000 7ff1edd9b640 1 -- 172.21.15.116:0/2366490543 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e00aba00 con 0x7ff1f0152d80 2024-04-25T00:26:18.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.679+0000 7ff1f5003640 1 -- 172.21.15.116:0/2366490543 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1f0152d80 msgr2=0x7ff1f006bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:18.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.679+0000 7ff1f5003640 1 --2- 172.21.15.116:0/2366490543 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1f0152d80 0x7ff1f006bd50 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7ff1e0009870 tx=0x7ff1e001c460 comp rx=0 tx=0).stop 2024-04-25T00:26:18.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.679+0000 7ff1f5003640 1 -- 172.21.15.116:0/2366490543 shutdown_connections 2024-04-25T00:26:18.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.679+0000 7ff1f5003640 1 --2- 172.21.15.116:0/2366490543 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff1f006d5f0 0x7ff1f0156c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:18.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.679+0000 7ff1f5003640 1 --2- 172.21.15.116:0/2366490543 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1f0152d80 0x7ff1f006bd50 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:18.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.679+0000 7ff1f5003640 1 -- 172.21.15.116:0/2366490543 >> 172.21.15.116:0/2366490543 conn(0x7ff1f00acdd0 msgr2=0x7ff1f00ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:18.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.679+0000 7ff1f5003640 1 -- 172.21.15.116:0/2366490543 shutdown_connections 2024-04-25T00:26:18.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.679+0000 7ff1f5003640 1 -- 172.21.15.116:0/2366490543 wait complete. 2024-04-25T00:26:18.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.679+0000 7ff1f5003640 1 Processor -- start 2024-04-25T00:26:18.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.679+0000 7ff1f5003640 1 -- start start 2024-04-25T00:26:18.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.683+0000 7ff1f5003640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1f006c290 0x7ff1f0150b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:18.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.683+0000 7ff1f5003640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1f006d5f0 0x7ff1f0151060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:18.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.683+0000 7ff1f5003640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff1f0152d80 0x7ff1f01f21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:18.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.683+0000 7ff1f5003640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f00caf00 con 0x7ff1f006d5f0 2024-04-25T00:26:18.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.683+0000 7ff1f5003640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f00cad80 con 0x7ff1f006c290 2024-04-25T00:26:18.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.683+0000 7ff1f5003640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff1f01628d0 con 0x7ff1f0152d80 2024-04-25T00:26:18.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.683+0000 7ff1ee59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1f006d5f0 0x7ff1f0151060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:18.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.683+0000 7ff1eed9d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1f006c290 0x7ff1f0150b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:18.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.683+0000 7ff1ee59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1f006d5f0 0x7ff1f0151060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48954/0 (socket says 172.21.15.116:48954) 2024-04-25T00:26:18.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.683+0000 7ff1ee59c640 1 -- 172.21.15.116:0/4039275654 learned_addr learned my addr 172.21.15.116:0/4039275654 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:26:18.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.683+0000 7ff1ef59e640 1 --2- 172.21.15.116:0/4039275654 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff1f0152d80 0x7ff1f01f21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:18.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.683+0000 7ff1ee59c640 1 -- 172.21.15.116:0/4039275654 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff1f0152d80 msgr2=0x7ff1f01f21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:18.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.683+0000 7ff1ee59c640 1 --2- 172.21.15.116:0/4039275654 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff1f0152d80 0x7ff1f01f21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:18.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.683+0000 7ff1ee59c640 1 -- 172.21.15.116:0/4039275654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1f006c290 msgr2=0x7ff1f0150b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:18.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.683+0000 7ff1ee59c640 1 --2- 172.21.15.116:0/4039275654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1f006c290 0x7ff1f0150b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:18.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.683+0000 7ff1ee59c640 1 -- 172.21.15.116:0/4039275654 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1f0152070 con 0x7ff1f006d5f0 2024-04-25T00:26:18.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.683+0000 7ff1ee59c640 1 --2- 172.21.15.116:0/4039275654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1f006d5f0 0x7ff1f0151060 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7ff1e400b420 tx=0x7ff1e400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:18.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.683+0000 7ff1d77fe640 1 -- 172.21.15.116:0/4039275654 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e4011830 con 0x7ff1f006d5f0 2024-04-25T00:26:18.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.683+0000 7ff1d77fe640 1 -- 172.21.15.116:0/4039275654 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1e4016070 con 0x7ff1f006d5f0 2024-04-25T00:26:18.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.683+0000 7ff1f5003640 1 -- 172.21.15.116:0/4039275654 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1f0152350 con 0x7ff1f006d5f0 2024-04-25T00:26:18.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.683+0000 7ff1d77fe640 1 -- 172.21.15.116:0/4039275654 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1e4011c10 con 0x7ff1f006d5f0 2024-04-25T00:26:18.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.683+0000 7ff1f5003640 1 -- 172.21.15.116:0/4039275654 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1f00ae5b0 con 0x7ff1f006d5f0 2024-04-25T00:26:18.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.683+0000 7ff1f5003640 1 -- 172.21.15.116:0/4039275654 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1f0152550 con 0x7ff1f006d5f0 2024-04-25T00:26:18.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.683+0000 7ff1d77fe640 1 -- 172.21.15.116:0/4039275654 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff1e401b460 con 0x7ff1f006d5f0 2024-04-25T00:26:18.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.683+0000 7ff1d77fe640 1 --2- 172.21.15.116:0/4039275654 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff1dc0678c0 0x7ff1dc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:18.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.683+0000 7ff1d77fe640 1 -- 172.21.15.116:0/4039275654 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 10106+0+0 (secure 0 0 0) 0x7ff1e4086960 con 0x7ff1f006d5f0 2024-04-25T00:26:18.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.683+0000 7ff1eed9d640 1 --2- 172.21.15.116:0/4039275654 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff1dc0678c0 0x7ff1dc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:18.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.687+0000 7ff1eed9d640 1 --2- 172.21.15.116:0/4039275654 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff1dc0678c0 0x7ff1dc069d80 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7ff1e00af000 tx=0x7ff1e00a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:18.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.691+0000 7ff1d77fe640 1 -- 172.21.15.116:0/4039275654 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ff1f0152550 con 0x7ff1f006d5f0 2024-04-25T00:26:18.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.979+0000 7ff1f5003640 1 -- 172.21.15.116:0/4039275654 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff1f0068350 con 0x7ff1f006d5f0 2024-04-25T00:26:18.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.979+0000 7ff1d77fe640 1 -- 172.21.15.116:0/4039275654 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 210 v210) ==== 95+0+5672 (secure 0 0 0) 0x7ff1e4055d00 con 0x7ff1f006d5f0 2024-04-25T00:26:18.982 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:26:18.982 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":210,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":210,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:26:17.015451+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6485},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6485":{"gid":6485,"name":"a","rank":0,"incarnation":208,"state":"up:rejoin","state_seq":9,"addr":"172.21.15.116:6835/2371126056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2371126056},{"type":"v1","addr":"172.21.15.116:6835","nonce":2371126056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:25:45.002412+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8496":{"gid":8496,"name":"c","rank":0,"incarnation":203,"state":"up:active","state_seq":6,"addr":"172.21.15.116:6833/3693049068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3693049068},{"type":"v1","addr":"172.21.15.116:6833","nonce":3693049068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8496,"qdb_cluster":[8496]},"id":2}]} 2024-04-25T00:26:18.982 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 210 2024-04-25T00:26:18.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.983+0000 7ff1f5003640 1 -- 172.21.15.116:0/4039275654 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff1dc0678c0 msgr2=0x7ff1dc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:18.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.983+0000 7ff1f5003640 1 --2- 172.21.15.116:0/4039275654 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff1dc0678c0 0x7ff1dc069d80 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7ff1e00af000 tx=0x7ff1e00a7000 comp rx=0 tx=0).stop 2024-04-25T00:26:18.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.983+0000 7ff1f5003640 1 -- 172.21.15.116:0/4039275654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1f006d5f0 msgr2=0x7ff1f0151060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:18.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.983+0000 7ff1f5003640 1 --2- 172.21.15.116:0/4039275654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1f006d5f0 0x7ff1f0151060 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7ff1e400b420 tx=0x7ff1e400b8e0 comp rx=0 tx=0).stop 2024-04-25T00:26:18.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.983+0000 7ff1f5003640 1 -- 172.21.15.116:0/4039275654 shutdown_connections 2024-04-25T00:26:18.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.983+0000 7ff1f5003640 1 --2- 172.21.15.116:0/4039275654 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff1dc0678c0 0x7ff1dc069d80 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:18.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.983+0000 7ff1f5003640 1 --2- 172.21.15.116:0/4039275654 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff1f0152d80 0x7ff1f01f21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:18.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.983+0000 7ff1f5003640 1 --2- 172.21.15.116:0/4039275654 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff1f006d5f0 0x7ff1f0151060 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:18.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.983+0000 7ff1f5003640 1 --2- 172.21.15.116:0/4039275654 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff1f006c290 0x7ff1f0150b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:18.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.983+0000 7ff1f5003640 1 -- 172.21.15.116:0/4039275654 >> 172.21.15.116:0/4039275654 conn(0x7ff1f00acdd0 msgr2=0x7ff1f014dae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:18.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.983+0000 7ff1f5003640 1 -- 172.21.15.116:0/4039275654 shutdown_connections 2024-04-25T00:26:18.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:18.983+0000 7ff1f5003640 1 -- 172.21.15.116:0/4039275654 wait complete. 2024-04-25T00:26:19.007 INFO:tasks.mds_thrash.fs.[a]:no change 2024-04-25T00:26:21.009 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:26:21.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:21.515+0000 7fd92ddab640 1 Processor -- start 2024-04-25T00:26:21.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:21.515+0000 7fd92ddab640 1 -- start start 2024-04-25T00:26:21.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:21.515+0000 7fd92ddab640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fd9280c2c00 0x7fd92815a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:21.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:21.519+0000 7fd92ddab640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd92815af70 0x7fd92815b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:21.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:21.519+0000 7fd92ddab640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd928071d70 con 0x7fd9280725e0 2024-04-25T00:26:21.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:21.519+0000 7fd92ddab640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fd928070cf0 con 0x7fd9280c2c00 2024-04-25T00:26:21.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:21.519+0000 7fd92ddab640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fd928070fb0 con 0x7fd92815af70 2024-04-25T00:26:21.560 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:21.519+0000 7fd927fff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd92815af70 0x7fd92815b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:21.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:21.519+0000 7fd927fff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd92815af70 0x7fd92815b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:35400/0 (socket says 172.21.15.116:35400) 2024-04-25T00:26:21.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:21.519+0000 7fd927fff640 1 -- 172.21.15.116:0/1481792354 learned_addr learned my addr 172.21.15.116:0/1481792354 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:26:21.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:21.519+0000 7fd927fff640 1 -- 172.21.15.116:0/1481792354 >> v2:172.21.15.174:3300/0 conn(0x7fd9280c2c00 msgr2=0x7fd92815a4d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T00:26:21.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:21.519+0000 7fd927fff640 1 --2- 172.21.15.116:0/1481792354 >> v2:172.21.15.174:3300/0 conn(0x7fd9280c2c00 0x7fd92815a4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:21.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:21.519+0000 7fd927fff640 1 -- 172.21.15.116:0/1481792354 >> v1:172.21.15.116:6789/0 conn(0x7fd9280725e0 legacy=0x7fd928153700 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T00:26:21.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:21.519+0000 7fd927fff640 1 -- 172.21.15.116:0/1481792354 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd928071290 con 0x7fd92815af70 2024-04-25T00:26:21.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:21.527+0000 7fd927fff640 1 --2- 172.21.15.116:0/1481792354 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd92815af70 0x7fd92815b3a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fd9280c7fc0 tx=0x7fd91c01c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10c4a423f8d63142 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:21.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:21.527+0000 7fd9267fc640 1 -- 172.21.15.116:0/1481792354 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd91c0b2070 con 0x7fd92815af70 2024-04-25T00:26:21.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:21.527+0000 7fd9267fc640 1 -- 172.21.15.116:0/1481792354 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd91c0a9070 con 0x7fd92815af70 2024-04-25T00:26:21.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:21.527+0000 7fd92ddab640 1 -- 172.21.15.116:0/1481792354 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd92815af70 msgr2=0x7fd92815b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:21.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:21.527+0000 7fd92ddab640 1 --2- 172.21.15.116:0/1481792354 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd92815af70 0x7fd92815b3a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fd9280c7fc0 tx=0x7fd91c01c490 comp rx=0 tx=0).stop 2024-04-25T00:26:21.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:21.567+0000 7fd92ddab640 1 -- 172.21.15.116:0/1481792354 shutdown_connections 2024-04-25T00:26:21.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:21.567+0000 7fd92ddab640 1 --2- 172.21.15.116:0/1481792354 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd92815af70 0x7fd92815b3a0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:21.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:21.567+0000 7fd92ddab640 1 --2- 172.21.15.116:0/1481792354 >> v2:172.21.15.174:3300/0 conn(0x7fd9280c2c00 0x7fd92815a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:21.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:21.567+0000 7fd92ddab640 1 -- 172.21.15.116:0/1481792354 >> 172.21.15.116:0/1481792354 conn(0x7fd9280c3a90 msgr2=0x7fd9280c3ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:21.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:21.567+0000 7fd92ddab640 1 -- 172.21.15.116:0/1481792354 shutdown_connections 2024-04-25T00:26:21.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:21.567+0000 7fd92ddab640 1 -- 172.21.15.116:0/1481792354 wait complete. 2024-04-25T00:26:21.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:21.567+0000 7fd92ddab640 1 Processor -- start 2024-04-25T00:26:21.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:21.567+0000 7fd92ddab640 1 -- start start 2024-04-25T00:26:21.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:21.571+0000 7fd92ddab640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9280725e0 0x7fd928140830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:21.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:21.571+0000 7fd92ddab640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd9280c2c00 0x7fd928155df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:21.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:21.571+0000 7fd92ddab640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd92815af70 0x7fd9280aba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:21.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:21.571+0000 7fd92ddab640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd92815ba60 con 0x7fd9280725e0 2024-04-25T00:26:21.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:21.571+0000 7fd92ddab640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd92815b8e0 con 0x7fd92815af70 2024-04-25T00:26:21.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:21.571+0000 7fd92ddab640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fd9280c2fe0 con 0x7fd9280c2c00 2024-04-25T00:26:21.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:21.571+0000 7fd9277fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9280725e0 0x7fd928140830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:21.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:21.571+0000 7fd9277fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9280725e0 0x7fd928140830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48964/0 (socket says 172.21.15.116:48964) 2024-04-25T00:26:21.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:21.571+0000 7fd9277fe640 1 -- 172.21.15.116:0/2678851739 learned_addr learned my addr 172.21.15.116:0/2678851739 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:26:21.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:21.571+0000 7fd926ffd640 1 --2- 172.21.15.116:0/2678851739 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd9280c2c00 0x7fd928155df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:21.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:21.571+0000 7fd9277fe640 1 -- 172.21.15.116:0/2678851739 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd9280c2c00 msgr2=0x7fd928155df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:21.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:21.571+0000 7fd9277fe640 1 --2- 172.21.15.116:0/2678851739 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd9280c2c00 0x7fd928155df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:21.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:21.571+0000 7fd9277fe640 1 -- 172.21.15.116:0/2678851739 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd92815af70 msgr2=0x7fd9280aba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:21.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:21.571+0000 7fd9277fe640 1 --2- 172.21.15.116:0/2678851739 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd92815af70 0x7fd9280aba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:21.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:21.571+0000 7fd9277fe640 1 -- 172.21.15.116:0/2678851739 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd928156bf0 con 0x7fd9280725e0 2024-04-25T00:26:21.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:21.571+0000 7fd9277fe640 1 --2- 172.21.15.116:0/2678851739 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9280725e0 0x7fd928140830 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7fd92000ba80 tx=0x7fd92000bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:21.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:21.595+0000 7fd917fff640 1 -- 172.21.15.116:0/2678851739 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd920019070 con 0x7fd9280725e0 2024-04-25T00:26:21.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:21.595+0000 7fd917fff640 1 -- 172.21.15.116:0/2678851739 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd92000cc20 con 0x7fd9280725e0 2024-04-25T00:26:21.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:21.595+0000 7fd917fff640 1 -- 172.21.15.116:0/2678851739 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd920014440 con 0x7fd9280725e0 2024-04-25T00:26:21.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:21.595+0000 7fd92ddab640 1 -- 172.21.15.116:0/2678851739 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd928156e90 con 0x7fd9280725e0 2024-04-25T00:26:21.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:21.595+0000 7fd92ddab640 1 -- 172.21.15.116:0/2678851739 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9280ac5d0 con 0x7fd9280725e0 2024-04-25T00:26:21.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:21.595+0000 7fd92ddab640 1 -- 172.21.15.116:0/2678851739 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd92804f580 con 0x7fd9280725e0 2024-04-25T00:26:21.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:21.595+0000 7fd917fff640 1 -- 172.21.15.116:0/2678851739 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd920003a40 con 0x7fd9280725e0 2024-04-25T00:26:21.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:21.595+0000 7fd917fff640 1 --2- 172.21.15.116:0/2678851739 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd90c0679c0 0x7fd90c069e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:21.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:21.595+0000 7fd926ffd640 1 --2- 172.21.15.116:0/2678851739 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd90c0679c0 0x7fd90c069e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:21.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:21.595+0000 7fd917fff640 1 -- 172.21.15.116:0/2678851739 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 10106+0+0 (secure 0 0 0) 0x7fd920086bf0 con 0x7fd9280725e0 2024-04-25T00:26:21.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:21.599+0000 7fd926ffd640 1 --2- 172.21.15.116:0/2678851739 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd90c0679c0 0x7fd90c069e80 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fd92815d940 tx=0x7fd918008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:21.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:21.615+0000 7fd917fff640 1 -- 172.21.15.116:0/2678851739 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fd920056660 con 0x7fd9280725e0 2024-04-25T00:26:21.999 INFO:tasks.ceph.mds.a.smithi116.stderr:2024-04-25T00:26:21.995+0000 7f8389b7d640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T00:26:23.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:23.147+0000 7fd92ddab640 1 -- 172.21.15.116:0/2678851739 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd9280acdd0 con 0x7fd9280725e0 2024-04-25T00:26:23.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:23.147+0000 7fd917fff640 1 -- 172.21.15.116:0/2678851739 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 211 v211) ==== 95+0+5680 (secure 0 0 0) 0x7fd920055f60 con 0x7fd9280725e0 2024-04-25T00:26:23.151 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:26:23.151 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":211,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":211,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:26:19.034064+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6485},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6485":{"gid":6485,"name":"a","rank":0,"incarnation":208,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6835/2371126056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2371126056},{"type":"v1","addr":"172.21.15.116:6835","nonce":2371126056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6485,"qdb_cluster":[6485]},"id":1},{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:25:45.002412+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8496":{"gid":8496,"name":"c","rank":0,"incarnation":203,"state":"up:active","state_seq":6,"addr":"172.21.15.116:6833/3693049068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3693049068},{"type":"v1","addr":"172.21.15.116:6833","nonce":3693049068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8496,"qdb_cluster":[8496]},"id":2}]} 2024-04-25T00:26:23.152 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 211 2024-04-25T00:26:23.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:23.151+0000 7fd915ffb640 1 -- 172.21.15.116:0/2678851739 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd90c0679c0 msgr2=0x7fd90c069e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:23.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:23.151+0000 7fd915ffb640 1 --2- 172.21.15.116:0/2678851739 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd90c0679c0 0x7fd90c069e80 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fd92815d940 tx=0x7fd918008040 comp rx=0 tx=0).stop 2024-04-25T00:26:23.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:23.151+0000 7fd915ffb640 1 -- 172.21.15.116:0/2678851739 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9280725e0 msgr2=0x7fd928140830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:23.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:23.151+0000 7fd915ffb640 1 --2- 172.21.15.116:0/2678851739 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9280725e0 0x7fd928140830 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7fd92000ba80 tx=0x7fd92000bf40 comp rx=0 tx=0).stop 2024-04-25T00:26:23.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:23.155+0000 7fd915ffb640 1 -- 172.21.15.116:0/2678851739 shutdown_connections 2024-04-25T00:26:23.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:23.155+0000 7fd915ffb640 1 --2- 172.21.15.116:0/2678851739 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd90c0679c0 0x7fd90c069e80 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:23.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:23.155+0000 7fd915ffb640 1 --2- 172.21.15.116:0/2678851739 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd92815af70 0x7fd9280aba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:23.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:23.155+0000 7fd915ffb640 1 --2- 172.21.15.116:0/2678851739 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd9280c2c00 0x7fd928155df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:23.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:23.155+0000 7fd915ffb640 1 --2- 172.21.15.116:0/2678851739 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd9280725e0 0x7fd928140830 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:23.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:23.155+0000 7fd915ffb640 1 -- 172.21.15.116:0/2678851739 >> 172.21.15.116:0/2678851739 conn(0x7fd9280c3a90 msgr2=0x7fd928163af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:23.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:23.155+0000 7fd915ffb640 1 -- 172.21.15.116:0/2678851739 shutdown_connections 2024-04-25T00:26:23.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:23.155+0000 7fd915ffb640 1 -- 172.21.15.116:0/2678851739 wait complete. 2024-04-25T00:26:23.223 INFO:tasks.mds_thrash.fs.[a]:mds.a has gained rank=0, replacing gid=6413 2024-04-25T00:26:23.223 INFO:tasks.mds_thrash.fs.[a]:waiting for 9 secs before reviving mds.d 2024-04-25T00:26:30.451 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:26:31.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.130+0000 7f11e7492640 1 Processor -- start 2024-04-25T00:26:31.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.130+0000 7f11e7492640 1 -- start start 2024-04-25T00:26:31.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.130+0000 7f11e7492640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f11e006ebc0 0x7f11e006f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:31.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.130+0000 7f11e7492640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11e0155a50 0x7f11e0155ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:31.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.130+0000 7f11e7492640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11e00711b0 con 0x7f11e014c280 2024-04-25T00:26:31.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.130+0000 7f11e7492640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f11e0071490 con 0x7f11e006ebc0 2024-04-25T00:26:31.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.130+0000 7f11e7492640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f11e0071770 con 0x7f11e0155a50 2024-04-25T00:26:31.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.130+0000 7f11e5a2f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11e0155a50 0x7f11e0155ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:31.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.130+0000 7f11e5a2f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11e0155a50 0x7f11e0155ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:59574/0 (socket says 172.21.15.116:59574) 2024-04-25T00:26:31.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.130+0000 7f11e5a2f640 1 -- 172.21.15.116:0/2779207087 learned_addr learned my addr 172.21.15.116:0/2779207087 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:26:31.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.130+0000 7f11e4a2d640 1 --2- 172.21.15.116:0/2779207087 >> v2:172.21.15.116:3300/0 conn(0x7f11e006ebc0 0x7f11e006f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:31.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.134+0000 7f11e5a2f640 1 -- 172.21.15.116:0/2779207087 >> v2:172.21.15.116:3300/0 conn(0x7f11e006ebc0 msgr2=0x7f11e006f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:31.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.134+0000 7f11e5a2f640 1 --2- 172.21.15.116:0/2779207087 >> v2:172.21.15.116:3300/0 conn(0x7f11e006ebc0 0x7f11e006f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:31.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.134+0000 7f11e5a2f640 1 -- 172.21.15.116:0/2779207087 >> v1:172.21.15.116:6789/0 conn(0x7f11e014c280 legacy=0x7f11e00c29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:31.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.134+0000 7f11e5a2f640 1 -- 172.21.15.116:0/2779207087 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11e0071a50 con 0x7f11e0155a50 2024-04-25T00:26:31.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.138+0000 7f11e5a2f640 1 --2- 172.21.15.116:0/2779207087 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11e0155a50 0x7f11e0155ec0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f11d8014480 tx=0x7f11d8024ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da5e3fcf59e910f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:31.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.234+0000 7f11d7fff640 1 -- 172.21.15.116:0/2779207087 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d80ba070 con 0x7f11e0155a50 2024-04-25T00:26:31.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.234+0000 7f11d7fff640 1 -- 172.21.15.116:0/2779207087 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11d80adcf0 con 0x7f11e0155a50 2024-04-25T00:26:31.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.234+0000 7f11d7fff640 1 -- 172.21.15.116:0/2779207087 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d80b5660 con 0x7f11e0155a50 2024-04-25T00:26:31.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.238+0000 7f11e7492640 1 -- 172.21.15.116:0/2779207087 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11e0155a50 msgr2=0x7f11e0155ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:31.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.238+0000 7f11e7492640 1 --2- 172.21.15.116:0/2779207087 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11e0155a50 0x7f11e0155ec0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f11d8014480 tx=0x7f11d8024ee0 comp rx=0 tx=0).stop 2024-04-25T00:26:31.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.242+0000 7f11e7492640 1 -- 172.21.15.116:0/2779207087 shutdown_connections 2024-04-25T00:26:31.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.242+0000 7f11e7492640 1 --2- 172.21.15.116:0/2779207087 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11e0155a50 0x7f11e0155ec0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:31.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.242+0000 7f11e7492640 1 --2- 172.21.15.116:0/2779207087 >> v2:172.21.15.116:3300/0 conn(0x7f11e006ebc0 0x7f11e006f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:31.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.242+0000 7f11e7492640 1 -- 172.21.15.116:0/2779207087 >> 172.21.15.116:0/2779207087 conn(0x7f11e00c3a90 msgr2=0x7f11e00c3ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:31.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.254+0000 7f11e7492640 1 -- 172.21.15.116:0/2779207087 shutdown_connections 2024-04-25T00:26:31.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.254+0000 7f11e7492640 1 -- 172.21.15.116:0/2779207087 wait complete. 2024-04-25T00:26:31.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.254+0000 7f11e7492640 1 Processor -- start 2024-04-25T00:26:31.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.254+0000 7f11e7492640 1 -- start start 2024-04-25T00:26:31.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.254+0000 7f11e7492640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11e006ebc0 0x7f11e0151360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:31.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.254+0000 7f11e7492640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11e014c280 0x7f11e01eab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:31.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.254+0000 7f11e7492640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11e0155a50 0x7f11e00a9670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:31.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.254+0000 7f11e7492640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f11e00beba0 con 0x7f11e0155a50 2024-04-25T00:26:31.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.254+0000 7f11e7492640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f11e00bea20 con 0x7f11e006ebc0 2024-04-25T00:26:31.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.254+0000 7f11e7492640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f11e00bed20 con 0x7f11e014c280 2024-04-25T00:26:31.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.258+0000 7f11e522e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11e006ebc0 0x7f11e0151360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:31.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.258+0000 7f11e522e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11e006ebc0 0x7f11e0151360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:38784/0 (socket says 172.21.15.116:38784) 2024-04-25T00:26:31.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.258+0000 7f11e522e640 1 -- 172.21.15.116:0/327574526 learned_addr learned my addr 172.21.15.116:0/327574526 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:26:31.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.258+0000 7f11e522e640 1 -- 172.21.15.116:0/327574526 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11e014c280 msgr2=0x7f11e01eab10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T00:26:31.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.258+0000 7f11e522e640 1 --2- 172.21.15.116:0/327574526 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11e014c280 0x7f11e01eab10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:31.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.258+0000 7f11e522e640 1 -- 172.21.15.116:0/327574526 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11e0155a50 msgr2=0x7f11e00a9670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:31.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.258+0000 7f11e522e640 1 --2- 172.21.15.116:0/327574526 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11e0155a50 0x7f11e00a9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:31.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.258+0000 7f11e522e640 1 -- 172.21.15.116:0/327574526 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11e0153580 con 0x7f11e006ebc0 2024-04-25T00:26:31.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.258+0000 7f11e522e640 1 --2- 172.21.15.116:0/327574526 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11e006ebc0 0x7f11e0151360 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f11dc00c800 tx=0x7f11dc00ccc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:31.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.258+0000 7f11d5ffb640 1 -- 172.21.15.116:0/327574526 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11dc017070 con 0x7f11e006ebc0 2024-04-25T00:26:31.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.258+0000 7f11e7492640 1 -- 172.21.15.116:0/327574526 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11e0153860 con 0x7f11e006ebc0 2024-04-25T00:26:31.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.258+0000 7f11e7492640 1 -- 172.21.15.116:0/327574526 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11e0153e20 con 0x7f11e006ebc0 2024-04-25T00:26:31.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.258+0000 7f11d5ffb640 1 -- 172.21.15.116:0/327574526 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11dc005290 con 0x7f11e006ebc0 2024-04-25T00:26:31.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.258+0000 7f11d5ffb640 1 -- 172.21.15.116:0/327574526 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11dc003600 con 0x7f11e006ebc0 2024-04-25T00:26:31.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.258+0000 7f11e7492640 1 -- 172.21.15.116:0/327574526 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11c00052a0 con 0x7f11e006ebc0 2024-04-25T00:26:31.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.262+0000 7f11d5ffb640 1 -- 172.21.15.116:0/327574526 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f11dc01a070 con 0x7f11e006ebc0 2024-04-25T00:26:31.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.262+0000 7f11d5ffb640 1 --2- 172.21.15.116:0/327574526 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f11cc067ad0 0x7f11cc069f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:31.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.262+0000 7f11d5ffb640 1 -- 172.21.15.116:0/327574526 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 10106+0+0 (secure 0 0 0) 0x7f11dc086720 con 0x7f11e006ebc0 2024-04-25T00:26:31.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.262+0000 7f11e4a2d640 1 --2- 172.21.15.116:0/327574526 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f11cc067ad0 0x7f11cc069f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:31.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.266+0000 7f11e4a2d640 1 --2- 172.21.15.116:0/327574526 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f11cc067ad0 0x7f11cc069f90 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f11e00aa2c0 tx=0x7f11d0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:31.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.278+0000 7f11d5ffb640 1 -- 172.21.15.116:0/327574526 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f11dc021020 con 0x7f11e006ebc0 2024-04-25T00:26:31.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.794+0000 7f11e7492640 1 -- 172.21.15.116:0/327574526 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f11c0005590 con 0x7f11e006ebc0 2024-04-25T00:26:31.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.798+0000 7f11d5ffb640 1 -- 172.21.15.116:0/327574526 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 211 v211) ==== 95+0+5680 (secure 0 0 0) 0x7f11dc056100 con 0x7f11e006ebc0 2024-04-25T00:26:31.801 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:26:31.802 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":211,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":211,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:26:19.034064+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6485},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6485":{"gid":6485,"name":"a","rank":0,"incarnation":208,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6835/2371126056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2371126056},{"type":"v1","addr":"172.21.15.116:6835","nonce":2371126056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6485,"qdb_cluster":[6485]},"id":1},{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:25:45.002412+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8496":{"gid":8496,"name":"c","rank":0,"incarnation":203,"state":"up:active","state_seq":6,"addr":"172.21.15.116:6833/3693049068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3693049068},{"type":"v1","addr":"172.21.15.116:6833","nonce":3693049068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8496,"qdb_cluster":[8496]},"id":2}]} 2024-04-25T00:26:31.802 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 211 2024-04-25T00:26:31.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.814+0000 7f11cb7fe640 1 -- 172.21.15.116:0/327574526 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f11cc067ad0 msgr2=0x7f11cc069f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:31.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.814+0000 7f11cb7fe640 1 --2- 172.21.15.116:0/327574526 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f11cc067ad0 0x7f11cc069f90 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f11e00aa2c0 tx=0x7f11d0008040 comp rx=0 tx=0).stop 2024-04-25T00:26:31.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.814+0000 7f11cb7fe640 1 -- 172.21.15.116:0/327574526 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11e006ebc0 msgr2=0x7f11e0151360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:31.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.814+0000 7f11cb7fe640 1 --2- 172.21.15.116:0/327574526 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11e006ebc0 0x7f11e0151360 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f11dc00c800 tx=0x7f11dc00ccc0 comp rx=0 tx=0).stop 2024-04-25T00:26:31.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.818+0000 7f11cb7fe640 1 -- 172.21.15.116:0/327574526 shutdown_connections 2024-04-25T00:26:31.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.818+0000 7f11cb7fe640 1 --2- 172.21.15.116:0/327574526 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f11cc067ad0 0x7f11cc069f90 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:31.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.818+0000 7f11cb7fe640 1 --2- 172.21.15.116:0/327574526 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11e0155a50 0x7f11e00a9670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:31.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.818+0000 7f11cb7fe640 1 --2- 172.21.15.116:0/327574526 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11e014c280 0x7f11e01eab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:31.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.818+0000 7f11cb7fe640 1 --2- 172.21.15.116:0/327574526 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11e006ebc0 0x7f11e0151360 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:31.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.818+0000 7f11cb7fe640 1 -- 172.21.15.116:0/327574526 >> 172.21.15.116:0/327574526 conn(0x7f11e00c3a90 msgr2=0x7f11e0072d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:31.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.818+0000 7f11cb7fe640 1 -- 172.21.15.116:0/327574526 shutdown_connections 2024-04-25T00:26:31.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:31.818+0000 7f11cb7fe640 1 -- 172.21.15.116:0/327574526 wait complete. 2024-04-25T00:26:31.867 INFO:tasks.mds_thrash.fs.[b]:kill mds.c (rank=0) 2024-04-25T00:26:31.867 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-25T00:26:31.867 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:26:31.893 INFO:tasks.ceph.mds.c:Stopped 2024-04-25T00:26:31.893 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:26:32.224 INFO:tasks.mds_thrash.fs.[a]:reviving mds.d 2024-04-25T00:26:32.224 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-25T00:26:32.224 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-25T00:26:32.230 INFO:tasks.ceph.mds.d:Started 2024-04-25T00:26:32.230 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:26:32.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.282+0000 7f69bf989640 1 Processor -- start 2024-04-25T00:26:32.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.282+0000 7f69bf989640 1 -- start start 2024-04-25T00:26:32.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.286+0000 7f69bf989640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f69b80c2c00 0x7f69b815a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:32.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.286+0000 7f69bf989640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f69b80725e0 0x7f69b8153700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:32.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.286+0000 7f69bf989640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f69b8153c40 0x7f69b815af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:32.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.286+0000 7f69bf989640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f69b8071d70 con 0x7f69b80c2c00 2024-04-25T00:26:32.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.286+0000 7f69bf989640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f69b8070cf0 con 0x7f69b8153c40 2024-04-25T00:26:32.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.286+0000 7f69bf989640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f69b8070fb0 con 0x7f69b80725e0 2024-04-25T00:26:32.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.294+0000 7f69bdf26640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f69b8153c40 0x7f69b815af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:32.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.294+0000 7f69bdf26640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f69b8153c40 0x7f69b815af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:38792/0 (socket says 172.21.15.116:38792) 2024-04-25T00:26:32.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.294+0000 7f69bdf26640 1 -- 172.21.15.116:0/3542945289 learned_addr learned my addr 172.21.15.116:0/3542945289 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:26:32.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.294+0000 7f69bdf26640 1 -- 172.21.15.116:0/3542945289 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f69b80725e0 msgr2=0x7f69b8153700 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T00:26:32.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.294+0000 7f69bdf26640 1 --2- 172.21.15.116:0/3542945289 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f69b80725e0 0x7f69b8153700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:32.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.294+0000 7f69bdf26640 1 -- 172.21.15.116:0/3542945289 >> v2:172.21.15.116:3300/0 conn(0x7f69b80c2c00 msgr2=0x7f69b815a4d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T00:26:32.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.294+0000 7f69bdf26640 1 --2- 172.21.15.116:0/3542945289 >> v2:172.21.15.116:3300/0 conn(0x7f69b80c2c00 0x7f69b815a4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:32.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.294+0000 7f69bdf26640 1 -- 172.21.15.116:0/3542945289 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69b8071290 con 0x7f69b8153c40 2024-04-25T00:26:32.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.302+0000 7f69bdf26640 1 --2- 172.21.15.116:0/3542945289 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69b8153c40 0x7f69b815af70 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f69b8073910 tx=0x7f69b001c9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a656f16792ac763b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:32.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.306+0000 7f69affff640 1 -- 172.21.15.116:0/3542945289 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69b00a5730 con 0x7f69b8153c40 2024-04-25T00:26:32.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.306+0000 7f69affff640 1 -- 172.21.15.116:0/3542945289 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69b00a58d0 con 0x7f69b8153c40 2024-04-25T00:26:32.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.306+0000 7f69bf989640 1 -- 172.21.15.116:0/3542945289 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69b8153c40 msgr2=0x7f69b815af70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:32.307 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.306+0000 7f69bf989640 1 --2- 172.21.15.116:0/3542945289 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69b8153c40 0x7f69b815af70 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f69b8073910 tx=0x7f69b001c9a0 comp rx=0 tx=0).stop 2024-04-25T00:26:32.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.314+0000 7f69bf989640 1 -- 172.21.15.116:0/3542945289 shutdown_connections 2024-04-25T00:26:32.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.314+0000 7f69bf989640 1 --2- 172.21.15.116:0/3542945289 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69b8153c40 0x7f69b815af70 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:32.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.314+0000 7f69bf989640 1 --2- 172.21.15.116:0/3542945289 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f69b80725e0 0x7f69b8153700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:32.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.314+0000 7f69bf989640 1 --2- 172.21.15.116:0/3542945289 >> v2:172.21.15.116:3300/0 conn(0x7f69b80c2c00 0x7f69b815a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:32.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.314+0000 7f69bf989640 1 -- 172.21.15.116:0/3542945289 >> 172.21.15.116:0/3542945289 conn(0x7f69b80c3a90 msgr2=0x7f69b80c3ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:32.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.322+0000 7f69bf989640 1 -- 172.21.15.116:0/3542945289 shutdown_connections 2024-04-25T00:26:32.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.322+0000 7f69bf989640 1 -- 172.21.15.116:0/3542945289 wait complete. 2024-04-25T00:26:32.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.326+0000 7f69bf989640 1 Processor -- start 2024-04-25T00:26:32.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.326+0000 7f69bf989640 1 -- start start 2024-04-25T00:26:32.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.326+0000 7f69bf989640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69b80725e0 0x7f69b8142c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:32.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.326+0000 7f69bf989640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69b80c2c00 0x7f69b80c59c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:32.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.326+0000 7f69bf989640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f69b8153c40 0x7f69b80c5f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:32.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.326+0000 7f69bf989640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f69b80c2fe0 con 0x7f69b80725e0 2024-04-25T00:26:32.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.326+0000 7f69bf989640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f69b80c3160 con 0x7f69b80c2c00 2024-04-25T00:26:32.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.326+0000 7f69bf989640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f69b815d880 con 0x7f69b8153c40 2024-04-25T00:26:32.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.326+0000 7f69bcf24640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69b80c2c00 0x7f69b80c59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:32.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.326+0000 7f69bd725640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69b80725e0 0x7f69b8142c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:32.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.326+0000 7f69bd725640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69b80725e0 0x7f69b8142c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38026/0 (socket says 172.21.15.116:38026) 2024-04-25T00:26:32.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.326+0000 7f69bd725640 1 -- 172.21.15.116:0/2893528378 learned_addr learned my addr 172.21.15.116:0/2893528378 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:26:32.328 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.326+0000 7f69bd725640 1 -- 172.21.15.116:0/2893528378 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f69b8153c40 msgr2=0x7f69b80c5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:32.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.326+0000 7f69bd725640 1 --2- 172.21.15.116:0/2893528378 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f69b8153c40 0x7f69b80c5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:32.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.326+0000 7f69bd725640 1 -- 172.21.15.116:0/2893528378 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69b80c2c00 msgr2=0x7f69b80c59c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:32.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.326+0000 7f69bd725640 1 --2- 172.21.15.116:0/2893528378 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69b80c2c00 0x7f69b80c59c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:32.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.326+0000 7f69bd725640 1 -- 172.21.15.116:0/2893528378 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69b8143140 con 0x7f69b80725e0 2024-04-25T00:26:32.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.326+0000 7f69bd725640 1 --2- 172.21.15.116:0/2893528378 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69b80725e0 0x7f69b8142c00 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f69b400b4a0 tx=0x7f69b400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:32.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.334+0000 7f69adffb640 1 -- 172.21.15.116:0/2893528378 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69b4014070 con 0x7f69b80725e0 2024-04-25T00:26:32.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.334+0000 7f69bf989640 1 -- 172.21.15.116:0/2893528378 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69b80c4c00 con 0x7f69b80725e0 2024-04-25T00:26:32.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.334+0000 7f69bf989640 1 -- 172.21.15.116:0/2893528378 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69b8152610 con 0x7f69b80725e0 2024-04-25T00:26:32.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.334+0000 7f69adffb640 1 -- 172.21.15.116:0/2893528378 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69b4010610 con 0x7f69b80725e0 2024-04-25T00:26:32.339 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.334+0000 7f69adffb640 1 -- 172.21.15.116:0/2893528378 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69b4010a40 con 0x7f69b80725e0 2024-04-25T00:26:32.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.342+0000 7f69adffb640 1 -- 172.21.15.116:0/2893528378 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f69b4017070 con 0x7f69b80725e0 2024-04-25T00:26:32.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.342+0000 7f69adffb640 1 --2- 172.21.15.116:0/2893528378 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f699c067940 0x7f699c069e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:32.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.342+0000 7f69adffb640 1 -- 172.21.15.116:0/2893528378 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 10106+0+0 (secure 0 0 0) 0x7f69b4086770 con 0x7f69b80725e0 2024-04-25T00:26:32.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.342+0000 7f69bf989640 1 -- 172.21.15.116:0/2893528378 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69b81577a0 con 0x7f69b80725e0 2024-04-25T00:26:32.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.342+0000 7f69bcf24640 1 --2- 172.21.15.116:0/2893528378 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f699c067940 0x7f699c069e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:32.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.346+0000 7f69bcf24640 1 --2- 172.21.15.116:0/2893528378 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f699c067940 0x7f699c069e00 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f69b806eb90 tx=0x7f69a8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:32.355 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.354+0000 7f69adffb640 1 -- 172.21.15.116:0/2893528378 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f69b81577a0 con 0x7f69b80725e0 2024-04-25T00:26:32.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.550+0000 7f34bfcaf640 1 Processor -- start 2024-04-25T00:26:32.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.550+0000 7f34bfcaf640 1 -- start start 2024-04-25T00:26:32.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.554+0000 7f34bfcaf640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f34b813a360 0x7f34b80c29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:32.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.554+0000 7f34bfcaf640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f34b80c2ee0 0x7f34b8071180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:32.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.554+0000 7f34bfcaf640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34b8073320 con 0x7f34b813acb0 2024-04-25T00:26:32.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.554+0000 7f34bfcaf640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f34b8070cf0 con 0x7f34b80c2ee0 2024-04-25T00:26:32.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.554+0000 7f34bfcaf640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f34b8073040 con 0x7f34b813a360 2024-04-25T00:26:32.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.554+0000 7f34bd24a640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f34b80c2ee0 0x7f34b8071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:32.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.554+0000 7f34bd24a640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f34b80c2ee0 0x7f34b8071180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:38820/0 (socket says 172.21.15.116:38820) 2024-04-25T00:26:32.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.554+0000 7f34bd24a640 1 -- 172.21.15.116:0/1734235693 learned_addr learned my addr 172.21.15.116:0/1734235693 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:26:32.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.558+0000 7f34bda4b640 1 --2- 172.21.15.116:0/1734235693 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f34b813a360 0x7f34b80c29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:32.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.558+0000 7f34bda4b640 1 -- 172.21.15.116:0/1734235693 >> v2:172.21.15.174:3300/0 conn(0x7f34b80c2ee0 msgr2=0x7f34b8071180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:32.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.558+0000 7f34bda4b640 1 --2- 172.21.15.116:0/1734235693 >> v2:172.21.15.174:3300/0 conn(0x7f34b80c2ee0 0x7f34b8071180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:32.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.558+0000 7f34bda4b640 1 -- 172.21.15.116:0/1734235693 >> v1:172.21.15.174:6789/0 conn(0x7f34b813acb0 legacy=0x7f34b80716c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:32.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.558+0000 7f34bda4b640 1 -- 172.21.15.116:0/1734235693 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34b81557e0 con 0x7f34b813a360 2024-04-25T00:26:32.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.562+0000 7f34bda4b640 1 --2- 172.21.15.116:0/1734235693 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f34b813a360 0x7f34b80c29a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f34a80097c0 tx=0x7f34a801c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2dc7cca479b1ad1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:32.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.562+0000 7f34bca49640 1 -- 172.21.15.116:0/1734235693 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34a80b0070 con 0x7f34b813a360 2024-04-25T00:26:32.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.562+0000 7f34bca49640 1 -- 172.21.15.116:0/1734235693 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34a80ab600 con 0x7f34b813a360 2024-04-25T00:26:32.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.562+0000 7f34bfcaf640 1 -- 172.21.15.116:0/1734235693 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f34b813a360 msgr2=0x7f34b80c29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:32.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.562+0000 7f34bfcaf640 1 --2- 172.21.15.116:0/1734235693 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f34b813a360 0x7f34b80c29a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f34a80097c0 tx=0x7f34a801c360 comp rx=0 tx=0).stop 2024-04-25T00:26:32.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.566+0000 7f34bfcaf640 1 -- 172.21.15.116:0/1734235693 shutdown_connections 2024-04-25T00:26:32.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.566+0000 7f34bfcaf640 1 --2- 172.21.15.116:0/1734235693 >> v2:172.21.15.174:3300/0 conn(0x7f34b80c2ee0 0x7f34b8071180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:32.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.566+0000 7f34bfcaf640 1 --2- 172.21.15.116:0/1734235693 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f34b813a360 0x7f34b80c29a0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:32.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.566+0000 7f34bfcaf640 1 -- 172.21.15.116:0/1734235693 >> 172.21.15.116:0/1734235693 conn(0x7f34b80c3a90 msgr2=0x7f34b80c3ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:32.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.570+0000 7f34bfcaf640 1 -- 172.21.15.116:0/1734235693 shutdown_connections 2024-04-25T00:26:32.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.574+0000 7f34bfcaf640 1 -- 172.21.15.116:0/1734235693 wait complete. 2024-04-25T00:26:32.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.574+0000 7f34bfcaf640 1 Processor -- start 2024-04-25T00:26:32.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.574+0000 7f34bfcaf640 1 -- start start 2024-04-25T00:26:32.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.574+0000 7f34bfcaf640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f34b80c2ee0 0x7f34b81f6e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:32.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.574+0000 7f34bfcaf640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f34b813a360 0x7f34b81548a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:32.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.574+0000 7f34bfcaf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34b813acb0 0x7f34b815a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:32.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.574+0000 7f34bfcaf640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f34b815ea60 con 0x7f34b813acb0 2024-04-25T00:26:32.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.574+0000 7f34bfcaf640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f34b815e8e0 con 0x7f34b813a360 2024-04-25T00:26:32.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.574+0000 7f34bfcaf640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f34b815ebe0 con 0x7f34b80c2ee0 2024-04-25T00:26:32.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.574+0000 7f34bd24a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f34b813a360 0x7f34b81548a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:32.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.574+0000 7f34be24c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34b813acb0 0x7f34b815a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:32.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.574+0000 7f34bd24a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f34b813a360 0x7f34b81548a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:38824/0 (socket says 172.21.15.116:38824) 2024-04-25T00:26:32.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.574+0000 7f34bd24a640 1 -- 172.21.15.116:0/2276721559 learned_addr learned my addr 172.21.15.116:0/2276721559 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:26:32.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.574+0000 7f34bda4b640 1 --2- 172.21.15.116:0/2276721559 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f34b80c2ee0 0x7f34b81f6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:32.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.574+0000 7f34be24c640 1 -- 172.21.15.116:0/2276721559 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f34b80c2ee0 msgr2=0x7f34b81f6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:32.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.574+0000 7f34be24c640 1 --2- 172.21.15.116:0/2276721559 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f34b80c2ee0 0x7f34b81f6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:32.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.574+0000 7f34be24c640 1 -- 172.21.15.116:0/2276721559 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f34b813a360 msgr2=0x7f34b81548a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:32.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.574+0000 7f34be24c640 1 --2- 172.21.15.116:0/2276721559 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f34b813a360 0x7f34b81548a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:32.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.574+0000 7f34be24c640 1 -- 172.21.15.116:0/2276721559 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34b80abf30 con 0x7f34b813acb0 2024-04-25T00:26:32.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.578+0000 7f34bda4b640 1 --2- 172.21.15.116:0/2276721559 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f34b80c2ee0 0x7f34b81f6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:26:32.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.578+0000 7f34bd24a640 1 --2- 172.21.15.116:0/2276721559 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f34b813a360 0x7f34b81548a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:26:32.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.578+0000 7f34be24c640 1 --2- 172.21.15.116:0/2276721559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34b813acb0 0x7f34b815a5a0 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f34b400c440 tx=0x7f34b400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:32.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.578+0000 7f34ae7fc640 1 -- 172.21.15.116:0/2276721559 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b4017070 con 0x7f34b813acb0 2024-04-25T00:26:32.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.578+0000 7f34bfcaf640 1 -- 172.21.15.116:0/2276721559 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34b80ac1d0 con 0x7f34b813acb0 2024-04-25T00:26:32.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.578+0000 7f34bfcaf640 1 -- 172.21.15.116:0/2276721559 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34b8156230 con 0x7f34b813acb0 2024-04-25T00:26:32.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.578+0000 7f34ae7fc640 1 -- 172.21.15.116:0/2276721559 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34b4003600 con 0x7f34b813acb0 2024-04-25T00:26:32.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.578+0000 7f34ae7fc640 1 -- 172.21.15.116:0/2276721559 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b40039f0 con 0x7f34b813acb0 2024-04-25T00:26:32.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.578+0000 7f34bfcaf640 1 -- 172.21.15.116:0/2276721559 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34b8156430 con 0x7f34b813acb0 2024-04-25T00:26:32.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.590+0000 7f34ae7fc640 1 -- 172.21.15.116:0/2276721559 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f34b401a070 con 0x7f34b813acb0 2024-04-25T00:26:32.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.594+0000 7f34ae7fc640 1 --2- 172.21.15.116:0/2276721559 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f349c067990 0x7f349c069e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:32.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.594+0000 7f34ae7fc640 1 -- 172.21.15.116:0/2276721559 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 10106+0+0 (secure 0 0 0) 0x7f34b408d350 con 0x7f34b813acb0 2024-04-25T00:26:32.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.594+0000 7f34bda4b640 1 --2- 172.21.15.116:0/2276721559 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f349c067990 0x7f349c069e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:32.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.598+0000 7f34bda4b640 1 --2- 172.21.15.116:0/2276721559 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f349c067990 0x7f349c069e50 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f34a80ad000 tx=0x7f34a80a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:32.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:32.594+0000 7f34ae7fc640 1 -- 172.21.15.116:0/2276721559 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f34b8156430 con 0x7f34b813acb0 2024-04-25T00:26:32.871 INFO:tasks.ceph.mds.d.smithi174.stdout:starting mds.d at 2024-04-25T00:26:33.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:33.282+0000 7f69bf989640 1 -- 172.21.15.116:0/2893528378 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f69b804f580 con 0x7f69b80725e0 2024-04-25T00:26:33.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:33.282+0000 7f69adffb640 1 -- 172.21.15.116:0/2893528378 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 211 v211) ==== 95+0+5680 (secure 0 0 0) 0x7f69b4056240 con 0x7f69b80725e0 2024-04-25T00:26:33.288 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:26:33.288 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":211,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":211,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:26:19.034064+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6485},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6485":{"gid":6485,"name":"a","rank":0,"incarnation":208,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6835/2371126056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2371126056},{"type":"v1","addr":"172.21.15.116:6835","nonce":2371126056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6485,"qdb_cluster":[6485]},"id":1},{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:25:45.002412+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8496":{"gid":8496,"name":"c","rank":0,"incarnation":203,"state":"up:active","state_seq":6,"addr":"172.21.15.116:6833/3693049068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3693049068},{"type":"v1","addr":"172.21.15.116:6833","nonce":3693049068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8496,"qdb_cluster":[8496]},"id":2}]} 2024-04-25T00:26:33.288 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 211 2024-04-25T00:26:33.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:33.322+0000 7f69bf989640 1 -- 172.21.15.116:0/2893528378 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f699c067940 msgr2=0x7f699c069e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:33.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:33.326+0000 7f69bf989640 1 --2- 172.21.15.116:0/2893528378 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f699c067940 0x7f699c069e00 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f69b806eb90 tx=0x7f69a8008040 comp rx=0 tx=0).stop 2024-04-25T00:26:33.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:33.326+0000 7f69bf989640 1 -- 172.21.15.116:0/2893528378 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69b80725e0 msgr2=0x7f69b8142c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:33.326 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:33.326+0000 7f69bf989640 1 --2- 172.21.15.116:0/2893528378 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69b80725e0 0x7f69b8142c00 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f69b400b4a0 tx=0x7f69b400b960 comp rx=0 tx=0).stop 2024-04-25T00:26:33.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:33.334+0000 7f69bf989640 1 -- 172.21.15.116:0/2893528378 shutdown_connections 2024-04-25T00:26:33.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:33.334+0000 7f69bf989640 1 --2- 172.21.15.116:0/2893528378 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f699c067940 0x7f699c069e00 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:33.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:33.334+0000 7f69bf989640 1 --2- 172.21.15.116:0/2893528378 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f69b8153c40 0x7f69b80c5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:33.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:33.334+0000 7f69bf989640 1 --2- 172.21.15.116:0/2893528378 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f69b80c2c00 0x7f69b80c59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:33.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:33.334+0000 7f69bf989640 1 --2- 172.21.15.116:0/2893528378 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f69b80725e0 0x7f69b8142c00 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:33.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:33.334+0000 7f69bf989640 1 -- 172.21.15.116:0/2893528378 >> 172.21.15.116:0/2893528378 conn(0x7f69b80c3a90 msgr2=0x7f69b80c1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:33.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:33.338+0000 7f69bf989640 1 -- 172.21.15.116:0/2893528378 shutdown_connections 2024-04-25T00:26:33.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:33.338+0000 7f69bf989640 1 -- 172.21.15.116:0/2893528378 wait complete. 2024-04-25T00:26:33.546 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:26:33.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:33.766+0000 7f34bfcaf640 1 -- 172.21.15.116:0/2276721559 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f34b80ac6d0 con 0x7f34b813acb0 2024-04-25T00:26:33.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:33.766+0000 7f34ae7fc640 1 -- 172.21.15.116:0/2276721559 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 211 v211) ==== 95+0+5680 (secure 0 0 0) 0x7f34b4055210 con 0x7f34b813acb0 2024-04-25T00:26:33.786 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:26:33.786 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":211,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":211,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:26:19.034064+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6485},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6485":{"gid":6485,"name":"a","rank":0,"incarnation":208,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6835/2371126056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2371126056},{"type":"v1","addr":"172.21.15.116:6835","nonce":2371126056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6485,"qdb_cluster":[6485]},"id":1},{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:25:45.002412+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8496":{"gid":8496,"name":"c","rank":0,"incarnation":203,"state":"up:active","state_seq":6,"addr":"172.21.15.116:6833/3693049068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3693049068},{"type":"v1","addr":"172.21.15.116:6833","nonce":3693049068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8496,"qdb_cluster":[8496]},"id":2}]} 2024-04-25T00:26:33.787 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 211 2024-04-25T00:26:33.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:33.798+0000 7f34bfcaf640 1 -- 172.21.15.116:0/2276721559 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f349c067990 msgr2=0x7f349c069e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:33.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:33.798+0000 7f34bfcaf640 1 --2- 172.21.15.116:0/2276721559 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f349c067990 0x7f349c069e50 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f34a80ad000 tx=0x7f34a80a6000 comp rx=0 tx=0).stop 2024-04-25T00:26:33.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:33.798+0000 7f34bfcaf640 1 -- 172.21.15.116:0/2276721559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34b813acb0 msgr2=0x7f34b815a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:33.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:33.802+0000 7f34bfcaf640 1 --2- 172.21.15.116:0/2276721559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34b813acb0 0x7f34b815a5a0 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f34b400c440 tx=0x7f34b400c900 comp rx=0 tx=0).stop 2024-04-25T00:26:33.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:33.802+0000 7f34bfcaf640 1 -- 172.21.15.116:0/2276721559 shutdown_connections 2024-04-25T00:26:33.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:33.810+0000 7f34bfcaf640 1 --2- 172.21.15.116:0/2276721559 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f349c067990 0x7f349c069e50 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:33.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:33.810+0000 7f34bfcaf640 1 --2- 172.21.15.116:0/2276721559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f34b813acb0 0x7f34b815a5a0 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:33.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:33.810+0000 7f34bfcaf640 1 --2- 172.21.15.116:0/2276721559 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f34b813a360 0x7f34b81548a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:33.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:33.810+0000 7f34bfcaf640 1 --2- 172.21.15.116:0/2276721559 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f34b80c2ee0 0x7f34b81f6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:33.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:33.810+0000 7f34bfcaf640 1 -- 172.21.15.116:0/2276721559 >> 172.21.15.116:0/2276721559 conn(0x7f34b80c3a90 msgr2=0x7f34b8139150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:33.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:33.810+0000 7f34bfcaf640 1 -- 172.21.15.116:0/2276721559 shutdown_connections 2024-04-25T00:26:33.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:33.810+0000 7f34bfcaf640 1 -- 172.21.15.116:0/2276721559 wait complete. 2024-04-25T00:26:33.877 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2024-04-25T00:26:35.547 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:26:35.878 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:26:36.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.150+0000 7f8a33065640 1 Processor -- start 2024-04-25T00:26:36.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.158+0000 7f8a33065640 1 -- start start 2024-04-25T00:26:36.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.162+0000 7f8a33065640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f8a240dc160 0x7f8a24049e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:36.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.162+0000 7f8a33065640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a24008e00 0x7f8a2403f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:36.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.162+0000 7f8a33065640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8a2403fde0 0x7f8a240dea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:36.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.162+0000 7f8a33065640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f8a2402c210 con 0x7f8a2403fde0 2024-04-25T00:26:36.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.162+0000 7f8a33065640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f8a240e1f50 con 0x7f8a240dc160 2024-04-25T00:26:36.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.162+0000 7f8a33065640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f8a240e21f0 con 0x7f8a24008e00 2024-04-25T00:26:36.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.162+0000 7f8a32864640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8a2403fde0 0x7f8a240dea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:36.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.162+0000 7f8a32864640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8a2403fde0 0x7f8a240dea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41040/0 (socket says 172.21.15.116:41040) 2024-04-25T00:26:36.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.162+0000 7f8a32864640 1 -- 172.21.15.116:0/586104303 learned_addr learned my addr 172.21.15.116:0/586104303 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:26:36.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.162+0000 7f8a32864640 1 -- 172.21.15.116:0/586104303 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a24008e00 msgr2=0x7f8a2403f8a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T00:26:36.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.162+0000 7f8a32864640 1 --2- 172.21.15.116:0/586104303 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a24008e00 0x7f8a2403f8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:36.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.162+0000 7f8a32864640 1 -- 172.21.15.116:0/586104303 >> v2:172.21.15.174:3300/0 conn(0x7f8a240dc160 msgr2=0x7f8a24049e70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T00:26:36.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.162+0000 7f8a32864640 1 --2- 172.21.15.116:0/586104303 >> v2:172.21.15.174:3300/0 conn(0x7f8a240dc160 0x7f8a24049e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:36.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.162+0000 7f8a32864640 1 -- 172.21.15.116:0/586104303 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a240e24d0 con 0x7f8a2403fde0 2024-04-25T00:26:36.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.166+0000 7f8a32864640 1 --2- 172.21.15.116:0/586104303 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a2403fde0 0x7f8a240dea20 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f8a2400caa0 tx=0x7f8a1c01c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bded00292d9a1881 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:36.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.166+0000 7f8a31061640 1 -- 172.21.15.116:0/586104303 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c0a9610 con 0x7f8a2403fde0 2024-04-25T00:26:36.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.166+0000 7f8a31061640 1 -- 172.21.15.116:0/586104303 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a1c0a97b0 con 0x7f8a2403fde0 2024-04-25T00:26:36.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.170+0000 7f8a33065640 1 -- 172.21.15.116:0/586104303 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a2403fde0 msgr2=0x7f8a240dea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:36.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.170+0000 7f8a33065640 1 --2- 172.21.15.116:0/586104303 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a2403fde0 0x7f8a240dea20 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f8a2400caa0 tx=0x7f8a1c01c6c0 comp rx=0 tx=0).stop 2024-04-25T00:26:36.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.174+0000 7f8a33065640 1 -- 172.21.15.116:0/586104303 shutdown_connections 2024-04-25T00:26:36.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.174+0000 7f8a33065640 1 --2- 172.21.15.116:0/586104303 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a2403fde0 0x7f8a240dea20 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:36.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.174+0000 7f8a33065640 1 --2- 172.21.15.116:0/586104303 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a24008e00 0x7f8a2403f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:36.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.174+0000 7f8a33065640 1 --2- 172.21.15.116:0/586104303 >> v2:172.21.15.174:3300/0 conn(0x7f8a240dc160 0x7f8a24049e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:36.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.174+0000 7f8a33065640 1 -- 172.21.15.116:0/586104303 >> 172.21.15.116:0/586104303 conn(0x7f8a24049350 msgr2=0x7f8a24049760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:36.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.174+0000 7f8a33065640 1 -- 172.21.15.116:0/586104303 shutdown_connections 2024-04-25T00:26:36.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.174+0000 7f8a33065640 1 -- 172.21.15.116:0/586104303 wait complete. 2024-04-25T00:26:36.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.174+0000 7f8a33065640 1 Processor -- start 2024-04-25T00:26:36.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.186+0000 7f8a33065640 1 -- start start 2024-04-25T00:26:36.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.186+0000 7f8a33065640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a24008e00 0x7f8a240c8520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:36.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.186+0000 7f8a33065640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a240dc160 0x7f8a240c5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:36.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.186+0000 7f8a33065640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a240dc790 0x7f8a240c6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:36.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.186+0000 7f8a33065640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8a2404aba0 con 0x7f8a24008e00 2024-04-25T00:26:36.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.186+0000 7f8a33065640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8a2404aa20 con 0x7f8a240dc790 2024-04-25T00:26:36.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.186+0000 7f8a33065640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f8a2404ad20 con 0x7f8a240dc160 2024-04-25T00:26:36.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.186+0000 7f8a31862640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a240dc160 0x7f8a240c5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:36.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.186+0000 7f8a31862640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a240dc160 0x7f8a240c5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:36322/0 (socket says 172.21.15.116:36322) 2024-04-25T00:26:36.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.186+0000 7f8a31862640 1 -- 172.21.15.116:0/423053716 learned_addr learned my addr 172.21.15.116:0/423053716 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:26:36.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.186+0000 7f8a31862640 1 -- 172.21.15.116:0/423053716 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a240dc790 msgr2=0x7f8a240c6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:36.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.186+0000 7f8a31862640 1 --2- 172.21.15.116:0/423053716 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a240dc790 0x7f8a240c6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:36.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.186+0000 7f8a31862640 1 -- 172.21.15.116:0/423053716 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a24008e00 msgr2=0x7f8a240c8520 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T00:26:36.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.186+0000 7f8a31862640 1 --2- 172.21.15.116:0/423053716 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a24008e00 0x7f8a240c8520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:36.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.186+0000 7f8a31862640 1 -- 172.21.15.116:0/423053716 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a240de820 con 0x7f8a240dc160 2024-04-25T00:26:36.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.190+0000 7f8a31862640 1 --2- 172.21.15.116:0/423053716 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a240dc160 0x7f8a240c5ee0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f8a2800b4a0 tx=0x7f8a2800b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:36.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.214+0000 7f8a22ffd640 1 -- 172.21.15.116:0/423053716 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a28011830 con 0x7f8a240dc160 2024-04-25T00:26:36.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.214+0000 7f8a22ffd640 1 -- 172.21.15.116:0/423053716 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a28016070 con 0x7f8a240dc160 2024-04-25T00:26:36.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.214+0000 7f8a33065640 1 -- 172.21.15.116:0/423053716 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a240de540 con 0x7f8a240dc160 2024-04-25T00:26:36.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.214+0000 7f8a22ffd640 1 -- 172.21.15.116:0/423053716 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a28011c50 con 0x7f8a240dc160 2024-04-25T00:26:36.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.214+0000 7f8a33065640 1 -- 172.21.15.116:0/423053716 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a240e0030 con 0x7f8a240dc160 2024-04-25T00:26:36.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.222+0000 7f8a22ffd640 1 -- 172.21.15.116:0/423053716 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8a2801b460 con 0x7f8a240dc160 2024-04-25T00:26:36.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.226+0000 7f8a33065640 1 -- 172.21.15.116:0/423053716 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a240057f0 con 0x7f8a240dc160 2024-04-25T00:26:36.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.226+0000 7f8a22ffd640 1 --2- 172.21.15.116:0/423053716 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8a10067940 0x7f8a10069e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:36.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.226+0000 7f8a32063640 1 --2- 172.21.15.116:0/423053716 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8a10067940 0x7f8a10069e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:36.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.226+0000 7f8a22ffd640 1 -- 172.21.15.116:0/423053716 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(68..68 src has 1..68) ==== 10106+0+0 (secure 0 0 0) 0x7f8a28086a70 con 0x7f8a240dc160 2024-04-25T00:26:36.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.226+0000 7f8a32063640 1 --2- 172.21.15.116:0/423053716 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8a10067940 0x7f8a10069e00 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f8a2c064d10 tx=0x7f8a2c06b0e0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:36.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.234+0000 7f8a22ffd640 1 -- 172.21.15.116:0/423053716 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f8a240de820 con 0x7f8a240dc160 2024-04-25T00:26:36.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.970+0000 7f84fbfff640 1 Processor -- start 2024-04-25T00:26:36.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.970+0000 7f84fbfff640 1 -- start start 2024-04-25T00:26:36.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.978+0000 7f84fbfff640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f84fc14c350 0x7f84fc071b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:36.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.978+0000 7f84fbfff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f84fc152de0 0x7f84fc155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:36.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.978+0000 7f84fbfff640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84fc1532e0 con 0x7f84fc0720b0 2024-04-25T00:26:36.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.978+0000 7f84fbfff640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f84fc0c3f90 con 0x7f84fc14c350 2024-04-25T00:26:36.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.978+0000 7f84fbfff640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f84fc0c4270 con 0x7f84fc152de0 2024-04-25T00:26:36.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.978+0000 7f84faffd640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f84fc14c350 0x7f84fc071b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:36.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.978+0000 7f84faffd640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f84fc14c350 0x7f84fc071b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:46910/0 (socket says 172.21.15.116:46910) 2024-04-25T00:26:36.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.978+0000 7f84faffd640 1 -- 172.21.15.116:0/1226165750 learned_addr learned my addr 172.21.15.116:0/1226165750 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:26:36.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.978+0000 7f84faffd640 1 -- 172.21.15.116:0/1226165750 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f84fc152de0 msgr2=0x7f84fc155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:36.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.978+0000 7f84faffd640 1 --2- 172.21.15.116:0/1226165750 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f84fc152de0 0x7f84fc155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:36.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.978+0000 7f84faffd640 1 -- 172.21.15.116:0/1226165750 >> v1:172.21.15.174:6789/0 conn(0x7f84fc0720b0 legacy=0x7f84fc156230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T00:26:36.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.978+0000 7f84faffd640 1 -- 172.21.15.116:0/1226165750 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84fc0c4550 con 0x7f84fc14c350 2024-04-25T00:26:36.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.982+0000 7f84faffd640 1 --2- 172.21.15.116:0/1226165750 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f84fc14c350 0x7f84fc071b70 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f84fc073180 tx=0x7f84f401eab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a99d7fce9ff0585d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:36.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.990+0000 7f84f9ffb640 1 -- 172.21.15.116:0/1226165750 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f400b2d0 con 0x7f84fc14c350 2024-04-25T00:26:36.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.990+0000 7f84f9ffb640 1 -- 172.21.15.116:0/1226165750 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84f400b470 con 0x7f84fc14c350 2024-04-25T00:26:36.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.990+0000 7f84f9ffb640 1 -- 172.21.15.116:0/1226165750 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f40a7970 con 0x7f84fc14c350 2024-04-25T00:26:36.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.990+0000 7f84fbfff640 1 -- 172.21.15.116:0/1226165750 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f84fc14c350 msgr2=0x7f84fc071b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:36.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:36.990+0000 7f84fbfff640 1 --2- 172.21.15.116:0/1226165750 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f84fc14c350 0x7f84fc071b70 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f84fc073180 tx=0x7f84f401eab0 comp rx=0 tx=0).stop 2024-04-25T00:26:37.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:37.010+0000 7f84fbfff640 1 -- 172.21.15.116:0/1226165750 shutdown_connections 2024-04-25T00:26:37.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:37.010+0000 7f84fbfff640 1 --2- 172.21.15.116:0/1226165750 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f84fc152de0 0x7f84fc155cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:37.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:37.010+0000 7f84fbfff640 1 --2- 172.21.15.116:0/1226165750 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f84fc14c350 0x7f84fc071b70 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:37.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:37.010+0000 7f84fbfff640 1 -- 172.21.15.116:0/1226165750 >> 172.21.15.116:0/1226165750 conn(0x7f84fc0c7e50 msgr2=0x7f84fc0c8260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:37.028 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:37.026+0000 7f84fbfff640 1 -- 172.21.15.116:0/1226165750 shutdown_connections 2024-04-25T00:26:37.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:37.026+0000 7f84fbfff640 1 -- 172.21.15.116:0/1226165750 wait complete. 2024-04-25T00:26:37.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:37.030+0000 7f84fbfff640 1 Processor -- start 2024-04-25T00:26:37.030 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:37.030+0000 7f84fbfff640 1 -- start start 2024-04-25T00:26:37.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:37.030+0000 7f84fbfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84fc0720b0 0x7f84fc153940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:37.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:37.030+0000 7f84fbfff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f84fc14c350 0x7f84fc153e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:37.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:37.030+0000 7f84fbfff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f84fc152de0 0x7f84fc13f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:37.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:37.030+0000 7f84fbfff640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f84fc06bdf0 con 0x7f84fc0720b0 2024-04-25T00:26:37.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:37.030+0000 7f84fbfff640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f84fc06bc70 con 0x7f84fc14c350 2024-04-25T00:26:37.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:37.030+0000 7f84fbfff640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f84fc06bf70 con 0x7f84fc152de0 2024-04-25T00:26:37.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:37.030+0000 7f84fb7fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f84fc152de0 0x7f84fc13f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:37.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:37.030+0000 7f84faffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84fc0720b0 0x7f84fc153940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:37.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:37.030+0000 7f84faffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84fc0720b0 0x7f84fc153940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41056/0 (socket says 172.21.15.116:41056) 2024-04-25T00:26:37.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:37.030+0000 7f84faffd640 1 -- 172.21.15.116:0/2227394404 learned_addr learned my addr 172.21.15.116:0/2227394404 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:26:37.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:37.030+0000 7f84fa7fc640 1 --2- 172.21.15.116:0/2227394404 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f84fc14c350 0x7f84fc153e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:37.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:37.030+0000 7f84faffd640 1 -- 172.21.15.116:0/2227394404 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f84fc152de0 msgr2=0x7f84fc13f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:37.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:37.030+0000 7f84faffd640 1 --2- 172.21.15.116:0/2227394404 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f84fc152de0 0x7f84fc13f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:37.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:37.030+0000 7f84faffd640 1 -- 172.21.15.116:0/2227394404 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f84fc14c350 msgr2=0x7f84fc153e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:37.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:37.030+0000 7f84faffd640 1 --2- 172.21.15.116:0/2227394404 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f84fc14c350 0x7f84fc153e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:37.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:37.030+0000 7f84faffd640 1 -- 172.21.15.116:0/2227394404 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84fc0c7410 con 0x7f84fc0720b0 2024-04-25T00:26:37.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:37.034+0000 7f84faffd640 1 --2- 172.21.15.116:0/2227394404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84fc0720b0 0x7f84fc153940 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f84f401d160 tx=0x7f84f401b850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:37.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:37.034+0000 7f84e3fff640 1 -- 172.21.15.116:0/2227394404 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f4009800 con 0x7f84fc0720b0 2024-04-25T00:26:37.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:37.034+0000 7f84e3fff640 1 -- 172.21.15.116:0/2227394404 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84f40be070 con 0x7f84fc0720b0 2024-04-25T00:26:37.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:37.034+0000 7f84fbfff640 1 -- 172.21.15.116:0/2227394404 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84fc1ead30 con 0x7f84fc0720b0 2024-04-25T00:26:37.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:37.034+0000 7f84fbfff640 1 -- 172.21.15.116:0/2227394404 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84fc0c76f0 con 0x7f84fc0720b0 2024-04-25T00:26:37.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:37.034+0000 7f84e3fff640 1 -- 172.21.15.116:0/2227394404 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f4009be0 con 0x7f84fc0720b0 2024-04-25T00:26:37.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:37.038+0000 7f84e3fff640 1 -- 172.21.15.116:0/2227394404 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f84f40b6070 con 0x7f84fc0720b0 2024-04-25T00:26:37.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:37.038+0000 7f84e3fff640 1 --2- 172.21.15.116:0/2227394404 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f84e8067990 0x7f84e8069e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:37.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:37.038+0000 7f84e3fff640 1 -- 172.21.15.116:0/2227394404 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 10106+0+0 (secure 0 0 0) 0x7f84f4125060 con 0x7f84fc0720b0 2024-04-25T00:26:37.041 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:37.042+0000 7f84fa7fc640 1 --2- 172.21.15.116:0/2227394404 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f84e8067990 0x7f84e8069e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:37.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:37.042+0000 7f84fa7fc640 1 --2- 172.21.15.116:0/2227394404 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f84e8067990 0x7f84e8069e50 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f84fc1523a0 tx=0x7f84f0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:37.048 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:37.046+0000 7f84fbfff640 1 -- 172.21.15.116:0/2227394404 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84dc0052a0 con 0x7f84fc0720b0 2024-04-25T00:26:37.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:37.062+0000 7f84e3fff640 1 -- 172.21.15.116:0/2227394404 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f84f40187c0 con 0x7f84fc0720b0 2024-04-25T00:26:37.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:37.218+0000 7f8a33065640 1 -- 172.21.15.116:0/423053716 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8a24046ad0 con 0x7f8a240dc160 2024-04-25T00:26:37.220 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:26:37.220 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":212,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7384,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/349772087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":349772087},{"type":"v1","addr":"172.21.15.174:6837","nonce":349772087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":212}],"filesystems":[{"mdsmap":{"epoch":211,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:26:19.034064+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6485},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6485":{"gid":6485,"name":"a","rank":0,"incarnation":208,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6835/2371126056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2371126056},{"type":"v1","addr":"172.21.15.116:6835","nonce":2371126056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6485,"qdb_cluster":[6485]},"id":1},{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:25:45.002412+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8496":{"gid":8496,"name":"c","rank":0,"incarnation":203,"state":"up:active","state_seq":6,"addr":"172.21.15.116:6833/3693049068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3693049068},{"type":"v1","addr":"172.21.15.116:6833","nonce":3693049068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8496,"qdb_cluster":[8496]},"id":2}]} 2024-04-25T00:26:37.220 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 212 2024-04-25T00:26:37.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:37.218+0000 7f8a22ffd640 1 -- 172.21.15.116:0/423053716 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 212 v212) ==== 95+0+6506 (secure 0 0 0) 0x7f8a280564e0 con 0x7f8a240dc160 2024-04-25T00:26:37.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:37.222+0000 7f8a20ef9640 1 -- 172.21.15.116:0/423053716 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8a10067940 msgr2=0x7f8a10069e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:37.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:37.222+0000 7f8a20ef9640 1 --2- 172.21.15.116:0/423053716 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8a10067940 0x7f8a10069e00 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f8a2c064d10 tx=0x7f8a2c06b0e0 comp rx=0 tx=0).stop 2024-04-25T00:26:37.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:37.222+0000 7f8a20ef9640 1 -- 172.21.15.116:0/423053716 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a240dc160 msgr2=0x7f8a240c5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:37.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:37.222+0000 7f8a20ef9640 1 --2- 172.21.15.116:0/423053716 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a240dc160 0x7f8a240c5ee0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f8a2800b4a0 tx=0x7f8a2800b960 comp rx=0 tx=0).stop 2024-04-25T00:26:37.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:37.222+0000 7f8a20ef9640 1 -- 172.21.15.116:0/423053716 shutdown_connections 2024-04-25T00:26:37.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:37.222+0000 7f8a20ef9640 1 --2- 172.21.15.116:0/423053716 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8a10067940 0x7f8a10069e00 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:37.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:37.222+0000 7f8a20ef9640 1 --2- 172.21.15.116:0/423053716 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8a240dc790 0x7f8a240c6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:37.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:37.222+0000 7f8a20ef9640 1 --2- 172.21.15.116:0/423053716 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8a240dc160 0x7f8a240c5ee0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:37.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:37.222+0000 7f8a20ef9640 1 --2- 172.21.15.116:0/423053716 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8a24008e00 0x7f8a240c8520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:37.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:37.222+0000 7f8a20ef9640 1 -- 172.21.15.116:0/423053716 >> 172.21.15.116:0/423053716 conn(0x7f8a24049350 msgr2=0x7f8a240473e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:37.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:37.222+0000 7f8a20ef9640 1 -- 172.21.15.116:0/423053716 shutdown_connections 2024-04-25T00:26:37.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:37.222+0000 7f8a20ef9640 1 -- 172.21.15.116:0/423053716 wait complete. 2024-04-25T00:26:37.301 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:26:38.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:38.054+0000 7f84fbfff640 1 -- 172.21.15.116:0/2227394404 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f84dc005860 con 0x7f84fc0720b0 2024-04-25T00:26:38.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:38.058+0000 7f84e3fff640 1 -- 172.21.15.116:0/2227394404 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 212 v212) ==== 95+0+6506 (secure 0 0 0) 0x7f84f40f0ef0 con 0x7f84fc0720b0 2024-04-25T00:26:38.060 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:26:38.060 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":212,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7384,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/349772087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":349772087},{"type":"v1","addr":"172.21.15.174:6837","nonce":349772087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":212}],"filesystems":[{"mdsmap":{"epoch":211,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:26:19.034064+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6485},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6485":{"gid":6485,"name":"a","rank":0,"incarnation":208,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6835/2371126056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2371126056},{"type":"v1","addr":"172.21.15.116:6835","nonce":2371126056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6485,"qdb_cluster":[6485]},"id":1},{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:25:45.002412+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8496":{"gid":8496,"name":"c","rank":0,"incarnation":203,"state":"up:active","state_seq":6,"addr":"172.21.15.116:6833/3693049068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3693049068},{"type":"v1","addr":"172.21.15.116:6833","nonce":3693049068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8496,"qdb_cluster":[8496]},"id":2}]} 2024-04-25T00:26:38.060 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 212 2024-04-25T00:26:38.072 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:38.070+0000 7f84fbfff640 1 -- 172.21.15.116:0/2227394404 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f84e8067990 msgr2=0x7f84e8069e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:38.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:38.070+0000 7f84fbfff640 1 --2- 172.21.15.116:0/2227394404 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f84e8067990 0x7f84e8069e50 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f84fc1523a0 tx=0x7f84f0008040 comp rx=0 tx=0).stop 2024-04-25T00:26:38.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:38.082+0000 7f84fbfff640 1 -- 172.21.15.116:0/2227394404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84fc0720b0 msgr2=0x7f84fc153940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:38.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:38.086+0000 7f84fbfff640 1 --2- 172.21.15.116:0/2227394404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84fc0720b0 0x7f84fc153940 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f84f401d160 tx=0x7f84f401b850 comp rx=0 tx=0).stop 2024-04-25T00:26:38.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:38.086+0000 7f84fbfff640 1 -- 172.21.15.116:0/2227394404 shutdown_connections 2024-04-25T00:26:38.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:38.086+0000 7f84fbfff640 1 --2- 172.21.15.116:0/2227394404 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f84e8067990 0x7f84e8069e50 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:38.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:38.086+0000 7f84fbfff640 1 --2- 172.21.15.116:0/2227394404 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f84fc152de0 0x7f84fc13f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:38.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:38.086+0000 7f84fbfff640 1 --2- 172.21.15.116:0/2227394404 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f84fc14c350 0x7f84fc153e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:38.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:38.086+0000 7f84fbfff640 1 --2- 172.21.15.116:0/2227394404 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f84fc0720b0 0x7f84fc153940 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:38.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:38.086+0000 7f84fbfff640 1 -- 172.21.15.116:0/2227394404 >> 172.21.15.116:0/2227394404 conn(0x7f84fc0c7e50 msgr2=0x7f84fc14a600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:38.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:38.086+0000 7f84fbfff640 1 -- 172.21.15.116:0/2227394404 shutdown_connections 2024-04-25T00:26:38.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:38.086+0000 7f84fbfff640 1 -- 172.21.15.116:0/2227394404 wait complete. 2024-04-25T00:26:38.143 INFO:tasks.mds_thrash.fs.[a]:mds.d reported in up:standby state 2024-04-25T00:26:38.143 INFO:tasks.mds_thrash.fs.[a]:waiting for 42 secs before thrashing 2024-04-25T00:26:39.302 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:26:39.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:39.558+0000 7f0167ad6640 1 Processor -- start 2024-04-25T00:26:39.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:39.558+0000 7f0167ad6640 1 -- start start 2024-04-25T00:26:39.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:39.562+0000 7f0167ad6640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0158033080 0x7f01580db7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:39.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:39.562+0000 7f0167ad6640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f015804d250 0x7f01580dbcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:39.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:39.562+0000 7f0167ad6640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01580dc6f0 con 0x7f0158048b70 2024-04-25T00:26:39.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:39.562+0000 7f0167ad6640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f015804be10 con 0x7f015804d250 2024-04-25T00:26:39.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:39.562+0000 7f0167ad6640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f01580dc410 con 0x7f0158033080 2024-04-25T00:26:39.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:39.562+0000 7f0165872640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0158033080 0x7f01580db7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:39.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:39.562+0000 7f0166874640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f015804d250 0x7f01580dbcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:39.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:39.562+0000 7f0165872640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0158033080 0x7f01580db7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:36338/0 (socket says 172.21.15.116:36338) 2024-04-25T00:26:39.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:39.562+0000 7f0165872640 1 -- 172.21.15.116:0/54057872 learned_addr learned my addr 172.21.15.116:0/54057872 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:26:39.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:39.562+0000 7f0166874640 1 -- 172.21.15.116:0/54057872 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0158033080 msgr2=0x7f01580db7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:39.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:39.562+0000 7f0166874640 1 --2- 172.21.15.116:0/54057872 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0158033080 0x7f01580db7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:39.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:39.562+0000 7f0166874640 1 -- 172.21.15.116:0/54057872 >> v1:172.21.15.116:6789/0 conn(0x7f0158048b70 legacy=0x7f015804cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:39.564 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:39.562+0000 7f0166874640 1 -- 172.21.15.116:0/54057872 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0158048650 con 0x7f015804d250 2024-04-25T00:26:39.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:39.566+0000 7f0166874640 1 --2- 172.21.15.116:0/54057872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f015804d250 0x7f01580dbcf0 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f016002d710 tx=0x7f0160097f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c885f82084e52b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:39.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:39.566+0000 7f0164870640 1 -- 172.21.15.116:0/54057872 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0160120670 con 0x7f015804d250 2024-04-25T00:26:39.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:39.566+0000 7f0164870640 1 -- 172.21.15.116:0/54057872 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0160120810 con 0x7f015804d250 2024-04-25T00:26:39.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:39.566+0000 7f0164870640 1 -- 172.21.15.116:0/54057872 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0160120c40 con 0x7f015804d250 2024-04-25T00:26:39.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:39.566+0000 7f0167ad6640 1 -- 172.21.15.116:0/54057872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f015804d250 msgr2=0x7f01580dbcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:39.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:39.566+0000 7f0167ad6640 1 --2- 172.21.15.116:0/54057872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f015804d250 0x7f01580dbcf0 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f016002d710 tx=0x7f0160097f50 comp rx=0 tx=0).stop 2024-04-25T00:26:39.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:39.566+0000 7f0167ad6640 1 -- 172.21.15.116:0/54057872 shutdown_connections 2024-04-25T00:26:39.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:39.566+0000 7f0167ad6640 1 --2- 172.21.15.116:0/54057872 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f015804d250 0x7f01580dbcf0 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:39.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:39.566+0000 7f0167ad6640 1 --2- 172.21.15.116:0/54057872 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0158033080 0x7f01580db7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:39.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:39.566+0000 7f0167ad6640 1 -- 172.21.15.116:0/54057872 >> 172.21.15.116:0/54057872 conn(0x7f015802f690 msgr2=0x7f015802faa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:39.568 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:39.566+0000 7f0167ad6640 1 -- 172.21.15.116:0/54057872 shutdown_connections 2024-04-25T00:26:39.569 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:39.566+0000 7f0167ad6640 1 -- 172.21.15.116:0/54057872 wait complete. 2024-04-25T00:26:39.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:39.570+0000 7f0167ad6640 1 Processor -- start 2024-04-25T00:26:39.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:39.570+0000 7f0167ad6640 1 -- start start 2024-04-25T00:26:39.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:39.570+0000 7f0167ad6640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0158033080 0x7f015802e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:39.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:39.570+0000 7f0167ad6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0158048b70 0x7f01580d0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:39.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:39.570+0000 7f0167ad6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f015804d250 0x7f01580d06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:39.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:39.570+0000 7f0166874640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f015804d250 0x7f01580d06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:39.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:39.570+0000 7f0166874640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f015804d250 0x7f01580d06d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41064/0 (socket says 172.21.15.116:41064) 2024-04-25T00:26:39.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:39.570+0000 7f0166874640 1 -- 172.21.15.116:0/2220782728 learned_addr learned my addr 172.21.15.116:0/2220782728 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:26:39.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:39.570+0000 7f0167ad6640 1 -- 172.21.15.116:0/2220782728 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0158009db0 con 0x7f015804d250 2024-04-25T00:26:39.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:39.570+0000 7f0167ad6640 1 -- 172.21.15.116:0/2220782728 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0158033b40 con 0x7f0158048b70 2024-04-25T00:26:39.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:39.570+0000 7f0167ad6640 1 -- 172.21.15.116:0/2220782728 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f01580e3350 con 0x7f0158033080 2024-04-25T00:26:39.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:39.570+0000 7f0166874640 1 -- 172.21.15.116:0/2220782728 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0158033080 msgr2=0x7f015802e940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T00:26:39.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:39.570+0000 7f0166874640 1 --2- 172.21.15.116:0/2220782728 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0158033080 0x7f015802e940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:39.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:39.570+0000 7f0166874640 1 -- 172.21.15.116:0/2220782728 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0158048b70 msgr2=0x7f01580d0190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T00:26:39.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:39.570+0000 7f0166874640 1 --2- 172.21.15.116:0/2220782728 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0158048b70 0x7f01580d0190 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:39.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:39.570+0000 7f0166874640 1 -- 172.21.15.116:0/2220782728 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0158044c80 con 0x7f015804d250 2024-04-25T00:26:39.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:39.570+0000 7f0166874640 1 --2- 172.21.15.116:0/2220782728 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f015804d250 0x7f01580d06d0 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f01600757a0 tx=0x7f0160094d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:39.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:39.574+0000 7f01567fc640 1 -- 172.21.15.116:0/2220782728 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0160128440 con 0x7f015804d250 2024-04-25T00:26:39.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:39.574+0000 7f0167ad6640 1 -- 172.21.15.116:0/2220782728 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0158044f20 con 0x7f015804d250 2024-04-25T00:26:39.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:39.574+0000 7f0167ad6640 1 -- 172.21.15.116:0/2220782728 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01580454e0 con 0x7f015804d250 2024-04-25T00:26:39.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:39.574+0000 7f0167ad6640 1 -- 172.21.15.116:0/2220782728 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01580059c0 con 0x7f015804d250 2024-04-25T00:26:39.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:39.586+0000 7f01567fc640 1 -- 172.21.15.116:0/2220782728 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01601285e0 con 0x7f015804d250 2024-04-25T00:26:39.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:39.586+0000 7f01567fc640 1 -- 172.21.15.116:0/2220782728 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0160128990 con 0x7f015804d250 2024-04-25T00:26:39.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:39.586+0000 7f01567fc640 1 -- 172.21.15.116:0/2220782728 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f0160128bb0 con 0x7f015804d250 2024-04-25T00:26:39.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:39.586+0000 7f01567fc640 1 --2- 172.21.15.116:0/2220782728 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f01440679c0 0x7f0144069e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:39.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:39.590+0000 7f01567fc640 1 -- 172.21.15.116:0/2220782728 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 10106+0+0 (secure 0 0 0) 0x7f016019b970 con 0x7f015804d250 2024-04-25T00:26:39.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:39.590+0000 7f01567fc640 1 -- 172.21.15.116:0/2220782728 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f0160091d70 con 0x7f015804d250 2024-04-25T00:26:39.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:39.590+0000 7f0165872640 1 --2- 172.21.15.116:0/2220782728 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f01440679c0 0x7f0144069e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:39.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:39.590+0000 7f0165872640 1 --2- 172.21.15.116:0/2220782728 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f01440679c0 0x7f0144069e80 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f015c0099e0 tx=0x7f015c009240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:40.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:40.142+0000 7f0167ad6640 1 -- 172.21.15.116:0/2220782728 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f01580449a0 con 0x7f015804d250 2024-04-25T00:26:40.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:40.146+0000 7f01567fc640 1 -- 172.21.15.116:0/2220782728 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 212 v212) ==== 95+0+6506 (secure 0 0 0) 0x7f016016b2c0 con 0x7f015804d250 2024-04-25T00:26:40.150 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:26:40.150 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":212,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7384,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/349772087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":349772087},{"type":"v1","addr":"172.21.15.174:6837","nonce":349772087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":212}],"filesystems":[{"mdsmap":{"epoch":211,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:26:19.034064+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6485},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6485":{"gid":6485,"name":"a","rank":0,"incarnation":208,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6835/2371126056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2371126056},{"type":"v1","addr":"172.21.15.116:6835","nonce":2371126056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6485,"qdb_cluster":[6485]},"id":1},{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:25:45.002412+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8496":{"gid":8496,"name":"c","rank":0,"incarnation":203,"state":"up:active","state_seq":6,"addr":"172.21.15.116:6833/3693049068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3693049068},{"type":"v1","addr":"172.21.15.116:6833","nonce":3693049068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8496,"qdb_cluster":[8496]},"id":2}]} 2024-04-25T00:26:40.150 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 212 2024-04-25T00:26:40.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:40.154+0000 7f0167ad6640 1 -- 172.21.15.116:0/2220782728 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f01440679c0 msgr2=0x7f0144069e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:40.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:40.154+0000 7f0167ad6640 1 --2- 172.21.15.116:0/2220782728 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f01440679c0 0x7f0144069e80 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f015c0099e0 tx=0x7f015c009240 comp rx=0 tx=0).stop 2024-04-25T00:26:40.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:40.154+0000 7f0167ad6640 1 -- 172.21.15.116:0/2220782728 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f015804d250 msgr2=0x7f01580d06d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:40.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:40.154+0000 7f0167ad6640 1 --2- 172.21.15.116:0/2220782728 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f015804d250 0x7f01580d06d0 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f01600757a0 tx=0x7f0160094d90 comp rx=0 tx=0).stop 2024-04-25T00:26:40.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:40.158+0000 7f0167ad6640 1 -- 172.21.15.116:0/2220782728 shutdown_connections 2024-04-25T00:26:40.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:40.158+0000 7f0167ad6640 1 --2- 172.21.15.116:0/2220782728 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f01440679c0 0x7f0144069e80 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:40.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:40.158+0000 7f0167ad6640 1 --2- 172.21.15.116:0/2220782728 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f015804d250 0x7f01580d06d0 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:40.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:40.158+0000 7f0167ad6640 1 --2- 172.21.15.116:0/2220782728 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0158048b70 0x7f01580d0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:40.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:40.158+0000 7f0167ad6640 1 --2- 172.21.15.116:0/2220782728 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0158033080 0x7f015802e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:40.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:40.158+0000 7f0167ad6640 1 -- 172.21.15.116:0/2220782728 >> 172.21.15.116:0/2220782728 conn(0x7f015802f690 msgr2=0x7f015802fa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:40.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:40.158+0000 7f0167ad6640 1 -- 172.21.15.116:0/2220782728 shutdown_connections 2024-04-25T00:26:40.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:40.158+0000 7f0167ad6640 1 -- 172.21.15.116:0/2220782728 wait complete. 2024-04-25T00:26:40.243 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:26:41.426 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:26:41.430 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:26:42.244 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:26:42.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.534+0000 7f43b4f4f640 1 Processor -- start 2024-04-25T00:26:42.535 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.534+0000 7f43b4f4f640 1 -- start start 2024-04-25T00:26:42.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.534+0000 7f43b4f4f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f43b014f700 0x7f43b00c8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:42.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.534+0000 7f43b4f4f640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f43b00c5880 0x7f43b01561c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:42.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.534+0000 7f43af7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f43b014f700 0x7f43b00c8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:42.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.534+0000 7f43af7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f43b014f700 0x7f43b00c8e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41066/0 (socket says 172.21.15.116:41066) 2024-04-25T00:26:42.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.534+0000 7f43aeffd640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f43b00c5880 0x7f43b01561c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:42.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.534+0000 7f43aeffd640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f43b00c5880 0x7f43b01561c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:46928/0 (socket says 172.21.15.116:46928) 2024-04-25T00:26:42.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.534+0000 7f43af7fe640 1 -- 172.21.15.116:0/4046926124 learned_addr learned my addr 172.21.15.116:0/4046926124 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:26:42.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.534+0000 7f43b4f4f640 1 --2- 172.21.15.116:0/4046926124 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43b01596d0 0x7f43b00cb2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:42.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.538+0000 7f43b4f4f640 1 -- 172.21.15.116:0/4046926124 --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f43b0159240 con 0x7f43b014f700 2024-04-25T00:26:42.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.538+0000 7f43b4f4f640 1 -- 172.21.15.116:0/4046926124 --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f43b00c9340 con 0x7f43b00c5880 2024-04-25T00:26:42.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.538+0000 7f43affff640 1 --2- 172.21.15.116:0/4046926124 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43b01596d0 0x7f43b00cb2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:42.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.538+0000 7f43b4f4f640 1 -- 172.21.15.116:0/4046926124 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f43b0156700 con 0x7f43b01596d0 2024-04-25T00:26:42.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.538+0000 7f43af7fe640 1 -- 172.21.15.116:0/4046926124 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43b01596d0 msgr2=0x7f43b00cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:42.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.538+0000 7f43af7fe640 1 --2- 172.21.15.116:0/4046926124 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43b01596d0 0x7f43b00cb2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:42.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.538+0000 7f43af7fe640 1 -- 172.21.15.116:0/4046926124 >> v2:172.21.15.174:3300/0 conn(0x7f43b00c5880 msgr2=0x7f43b01561c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:42.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.538+0000 7f43af7fe640 1 --2- 172.21.15.116:0/4046926124 >> v2:172.21.15.174:3300/0 conn(0x7f43b00c5880 0x7f43b01561c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:42.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.538+0000 7f43af7fe640 1 -- 172.21.15.116:0/4046926124 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43b006d950 con 0x7f43b014f700 2024-04-25T00:26:42.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.538+0000 7f43af7fe640 1 --2- 172.21.15.116:0/4046926124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43b014f700 0x7f43b00c8e00 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f43b006d5c0 tx=0x7f43a001c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89a7b2aa9ae93d87 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:42.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.538+0000 7f43ae7fc640 1 -- 172.21.15.116:0/4046926124 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a00b0070 con 0x7f43b014f700 2024-04-25T00:26:42.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.538+0000 7f43ae7fc640 1 -- 172.21.15.116:0/4046926124 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43a00ab600 con 0x7f43b014f700 2024-04-25T00:26:42.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.538+0000 7f43ae7fc640 1 -- 172.21.15.116:0/4046926124 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a00ab9d0 con 0x7f43b014f700 2024-04-25T00:26:42.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.542+0000 7f43b4f4f640 1 -- 172.21.15.116:0/4046926124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43b014f700 msgr2=0x7f43b00c8e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:42.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.542+0000 7f43b4f4f640 1 --2- 172.21.15.116:0/4046926124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43b014f700 0x7f43b00c8e00 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f43b006d5c0 tx=0x7f43a001c490 comp rx=0 tx=0).stop 2024-04-25T00:26:42.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.542+0000 7f43b4f4f640 1 -- 172.21.15.116:0/4046926124 shutdown_connections 2024-04-25T00:26:42.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.542+0000 7f43b4f4f640 1 --2- 172.21.15.116:0/4046926124 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43b01596d0 0x7f43b00cb2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:42.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.542+0000 7f43b4f4f640 1 --2- 172.21.15.116:0/4046926124 >> v2:172.21.15.174:3300/0 conn(0x7f43b00c5880 0x7f43b01561c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:42.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.542+0000 7f43b4f4f640 1 --2- 172.21.15.116:0/4046926124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43b014f700 0x7f43b00c8e00 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:42.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.542+0000 7f43b4f4f640 1 -- 172.21.15.116:0/4046926124 >> 172.21.15.116:0/4046926124 conn(0x7f43b0074650 msgr2=0x7f43b0074a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:42.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.542+0000 7f43b4f4f640 1 -- 172.21.15.116:0/4046926124 shutdown_connections 2024-04-25T00:26:42.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.542+0000 7f43b4f4f640 1 -- 172.21.15.116:0/4046926124 wait complete. 2024-04-25T00:26:42.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.546+0000 7f43b4f4f640 1 Processor -- start 2024-04-25T00:26:42.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.546+0000 7f43b4f4f640 1 -- start start 2024-04-25T00:26:42.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.546+0000 7f43b4f4f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43b00c5880 0x7f43b014c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:42.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.546+0000 7f43b4f4f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43b014f700 0x7f43b00c03b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:42.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.546+0000 7f43b4f4f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43b01596d0 0x7f43b00c44b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:42.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.546+0000 7f43b4f4f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f43b014fc60 con 0x7f43b01596d0 2024-04-25T00:26:42.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.546+0000 7f43b4f4f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f43b014fae0 con 0x7f43b00c5880 2024-04-25T00:26:42.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.546+0000 7f43b4f4f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f43b0159ab0 con 0x7f43b014f700 2024-04-25T00:26:42.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.550+0000 7f43af7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43b00c5880 0x7f43b014c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:42.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.550+0000 7f43af7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43b00c5880 0x7f43b014c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:46936/0 (socket says 172.21.15.116:46936) 2024-04-25T00:26:42.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.550+0000 7f43af7fe640 1 -- 172.21.15.116:0/2772154565 learned_addr learned my addr 172.21.15.116:0/2772154565 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:26:42.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.550+0000 7f43aeffd640 1 --2- 172.21.15.116:0/2772154565 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43b014f700 0x7f43b00c03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:42.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.550+0000 7f43affff640 1 --2- 172.21.15.116:0/2772154565 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43b01596d0 0x7f43b00c44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:42.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.550+0000 7f43af7fe640 1 -- 172.21.15.116:0/2772154565 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43b014f700 msgr2=0x7f43b00c03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:42.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.550+0000 7f43af7fe640 1 --2- 172.21.15.116:0/2772154565 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43b014f700 0x7f43b00c03b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:42.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.550+0000 7f43af7fe640 1 -- 172.21.15.116:0/2772154565 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43b01596d0 msgr2=0x7f43b00c44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:42.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.550+0000 7f43af7fe640 1 --2- 172.21.15.116:0/2772154565 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43b01596d0 0x7f43b00c44b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:42.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.550+0000 7f43af7fe640 1 -- 172.21.15.116:0/2772154565 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43b00c1130 con 0x7f43b00c5880 2024-04-25T00:26:42.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.550+0000 7f43af7fe640 1 --2- 172.21.15.116:0/2772154565 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43b00c5880 0x7f43b014c1d0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f43a001cd30 tx=0x7f43a0019230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:42.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.550+0000 7f4397fff640 1 -- 172.21.15.116:0/2772154565 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a00bb070 con 0x7f43b00c5880 2024-04-25T00:26:42.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.550+0000 7f4397fff640 1 -- 172.21.15.116:0/2772154565 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43a00b6440 con 0x7f43b00c5880 2024-04-25T00:26:42.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.550+0000 7f4397fff640 1 -- 172.21.15.116:0/2772154565 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a00b0040 con 0x7f43b00c5880 2024-04-25T00:26:42.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.550+0000 7f43b4f4f640 1 -- 172.21.15.116:0/2772154565 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43b00c1410 con 0x7f43b00c5880 2024-04-25T00:26:42.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.550+0000 7f43b4f4f640 1 -- 172.21.15.116:0/2772154565 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43b00c08f0 con 0x7f43b00c5880 2024-04-25T00:26:42.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.550+0000 7f4397fff640 1 -- 172.21.15.116:0/2772154565 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f43a00be070 con 0x7f43b00c5880 2024-04-25T00:26:42.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.550+0000 7f4397fff640 1 --2- 172.21.15.116:0/2772154565 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f439c0678c0 0x7f439c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:42.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.550+0000 7f4397fff640 1 -- 172.21.15.116:0/2772154565 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 10106+0+0 (secure 0 0 0) 0x7f43a01207f0 con 0x7f43b00c5880 2024-04-25T00:26:42.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.554+0000 7f43aeffd640 1 --2- 172.21.15.116:0/2772154565 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f439c0678c0 0x7f439c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:42.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.554+0000 7f43b4f4f640 1 -- 172.21.15.116:0/2772154565 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43b0073710 con 0x7f43b00c5880 2024-04-25T00:26:42.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.558+0000 7f43aeffd640 1 --2- 172.21.15.116:0/2772154565 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f439c0678c0 0x7f439c069d80 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f43b00c5350 tx=0x7f43a4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:42.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:42.562+0000 7f4397fff640 1 -- 172.21.15.116:0/2772154565 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f43a0016110 con 0x7f43b00c5880 2024-04-25T00:26:43.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:43.178+0000 7f43b4f4f640 1 -- 172.21.15.116:0/2772154565 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f43b004f510 con 0x7f43b00c5880 2024-04-25T00:26:43.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:43.182+0000 7f4397fff640 1 -- 172.21.15.116:0/2772154565 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 212 v212) ==== 95+0+6506 (secure 0 0 0) 0x7f43a00f01e0 con 0x7f43b00c5880 2024-04-25T00:26:43.183 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:26:43.183 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":212,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":7384,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/349772087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":349772087},{"type":"v1","addr":"172.21.15.174:6837","nonce":349772087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":212}],"filesystems":[{"mdsmap":{"epoch":211,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:26:19.034064+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6485},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6485":{"gid":6485,"name":"a","rank":0,"incarnation":208,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6835/2371126056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2371126056},{"type":"v1","addr":"172.21.15.116:6835","nonce":2371126056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6485,"qdb_cluster":[6485]},"id":1},{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:25:45.002412+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8496},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8496":{"gid":8496,"name":"c","rank":0,"incarnation":203,"state":"up:active","state_seq":6,"addr":"172.21.15.116:6833/3693049068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3693049068},{"type":"v1","addr":"172.21.15.116:6833","nonce":3693049068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8496,"qdb_cluster":[8496]},"id":2}]} 2024-04-25T00:26:43.183 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 212 2024-04-25T00:26:43.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:43.186+0000 7f43b4f4f640 1 -- 172.21.15.116:0/2772154565 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f439c0678c0 msgr2=0x7f439c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:43.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:43.186+0000 7f43b4f4f640 1 --2- 172.21.15.116:0/2772154565 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f439c0678c0 0x7f439c069d80 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f43b00c5350 tx=0x7f43a4008040 comp rx=0 tx=0).stop 2024-04-25T00:26:43.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:43.186+0000 7f43b4f4f640 1 -- 172.21.15.116:0/2772154565 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43b00c5880 msgr2=0x7f43b014c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:43.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:43.186+0000 7f43b4f4f640 1 --2- 172.21.15.116:0/2772154565 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43b00c5880 0x7f43b014c1d0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f43a001cd30 tx=0x7f43a0019230 comp rx=0 tx=0).stop 2024-04-25T00:26:43.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:43.186+0000 7f43b4f4f640 1 -- 172.21.15.116:0/2772154565 shutdown_connections 2024-04-25T00:26:43.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:43.186+0000 7f43b4f4f640 1 --2- 172.21.15.116:0/2772154565 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f439c0678c0 0x7f439c069d80 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:43.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:43.186+0000 7f43b4f4f640 1 --2- 172.21.15.116:0/2772154565 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f43b01596d0 0x7f43b00c44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:43.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:43.186+0000 7f43b4f4f640 1 --2- 172.21.15.116:0/2772154565 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43b014f700 0x7f43b00c03b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:43.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:43.186+0000 7f43b4f4f640 1 --2- 172.21.15.116:0/2772154565 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43b00c5880 0x7f43b014c1d0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:43.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:43.186+0000 7f43b4f4f640 1 -- 172.21.15.116:0/2772154565 >> 172.21.15.116:0/2772154565 conn(0x7f43b0074650 msgr2=0x7f43b014e680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:43.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:43.186+0000 7f43b4f4f640 1 -- 172.21.15.116:0/2772154565 shutdown_connections 2024-04-25T00:26:43.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:43.186+0000 7f43b4f4f640 1 -- 172.21.15.116:0/2772154565 wait complete. 2024-04-25T00:26:43.235 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:26:45.236 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:26:45.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.550+0000 7f7758f0d640 1 Processor -- start 2024-04-25T00:26:45.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.550+0000 7f7758f0d640 1 -- start start 2024-04-25T00:26:45.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.554+0000 7f7758f0d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f774c0f8950 0x7f774c046160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:45.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.554+0000 7f7758f0d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f774c0466a0 0x7f774c102930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:45.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.554+0000 7f7758f0d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f774c0071d0 con 0x7f774c0f55e0 2024-04-25T00:26:45.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.554+0000 7f7758f0d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f774c047c60 con 0x7f774c0466a0 2024-04-25T00:26:45.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.554+0000 7f7758f0d640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f774c0069c0 con 0x7f774c0f8950 2024-04-25T00:26:45.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.554+0000 7f7753fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f774c0466a0 0x7f774c102930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:45.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.554+0000 7f7753fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f774c0466a0 0x7f774c102930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:56994/0 (socket says 172.21.15.116:56994) 2024-04-25T00:26:45.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.554+0000 7f7753fff640 1 -- 172.21.15.116:0/859660137 learned_addr learned my addr 172.21.15.116:0/859660137 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:26:45.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.562+0000 7f7753fff640 1 -- 172.21.15.116:0/859660137 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f774c0f8950 msgr2=0x7f774c046160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:45.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.554+0000 7f7752ffd640 1 --2- 172.21.15.116:0/859660137 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f774c0f8950 0x7f774c046160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:45.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.562+0000 7f7753fff640 1 --2- 172.21.15.116:0/859660137 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f774c0f8950 0x7f774c046160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:45.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.562+0000 7f7753fff640 1 -- 172.21.15.116:0/859660137 >> v1:172.21.15.116:6789/0 conn(0x7f774c0f55e0 legacy=0x7f774c0f94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:45.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.562+0000 7f7753fff640 1 -- 172.21.15.116:0/859660137 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f774c0f59c0 con 0x7f774c0466a0 2024-04-25T00:26:45.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.562+0000 7f7753fff640 1 --2- 172.21.15.116:0/859660137 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f774c0466a0 0x7f774c102930 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f775404fa30 tx=0x7f7754087040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=416422c3a4bd60af server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:45.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.562+0000 7f7752ffd640 1 --2- 172.21.15.116:0/859660137 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f774c0f8950 0x7f774c046160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:26:45.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.566+0000 7f77527fc640 1 -- 172.21.15.116:0/859660137 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7754071390 con 0x7f774c0466a0 2024-04-25T00:26:45.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.566+0000 7f77527fc640 1 -- 172.21.15.116:0/859660137 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f775410f850 con 0x7f774c0466a0 2024-04-25T00:26:45.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.566+0000 7f77527fc640 1 -- 172.21.15.116:0/859660137 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f775410fcc0 con 0x7f774c0466a0 2024-04-25T00:26:45.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.566+0000 7f7758f0d640 1 -- 172.21.15.116:0/859660137 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f774c0466a0 msgr2=0x7f774c102930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:45.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.566+0000 7f7758f0d640 1 --2- 172.21.15.116:0/859660137 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f774c0466a0 0x7f774c102930 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f775404fa30 tx=0x7f7754087040 comp rx=0 tx=0).stop 2024-04-25T00:26:45.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.566+0000 7f7758f0d640 1 -- 172.21.15.116:0/859660137 shutdown_connections 2024-04-25T00:26:45.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.566+0000 7f7758f0d640 1 --2- 172.21.15.116:0/859660137 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f774c0466a0 0x7f774c102930 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:45.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.566+0000 7f7758f0d640 1 --2- 172.21.15.116:0/859660137 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f774c0f8950 0x7f774c046160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:45.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.566+0000 7f7758f0d640 1 -- 172.21.15.116:0/859660137 >> 172.21.15.116:0/859660137 conn(0x7f774c005ea0 msgr2=0x7f774c0062b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:45.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.566+0000 7f7758f0d640 1 -- 172.21.15.116:0/859660137 shutdown_connections 2024-04-25T00:26:45.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.566+0000 7f7758f0d640 1 -- 172.21.15.116:0/859660137 wait complete. 2024-04-25T00:26:45.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.566+0000 7f7758f0d640 1 Processor -- start 2024-04-25T00:26:45.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.566+0000 7f7758f0d640 1 -- start start 2024-04-25T00:26:45.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.566+0000 7f7758f0d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f774c0466a0 0x7f774c0e6a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:45.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.566+0000 7f7758f0d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f774c0f55e0 0x7f774c0eb380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:45.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.566+0000 7f7758f0d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f774c0f8950 0x7f774c1b2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:45.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.566+0000 7f7758f0d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f774c0073d0 con 0x7f774c0f8950 2024-04-25T00:26:45.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.566+0000 7f7758f0d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f774c0f8430 con 0x7f774c0f55e0 2024-04-25T00:26:45.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.566+0000 7f7758f0d640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f774c0e6250 con 0x7f774c0466a0 2024-04-25T00:26:45.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.566+0000 7f7752ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f774c0f55e0 0x7f774c0eb380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:45.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.566+0000 7f77537fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f774c0466a0 0x7f774c0e6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:45.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.566+0000 7f7753fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f774c0f8950 0x7f774c1b2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:45.578 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.566+0000 7f7752ffd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f774c0f55e0 0x7f774c0eb380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:40116/0 (socket says 172.21.15.116:40116) 2024-04-25T00:26:45.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.566+0000 7f7752ffd640 1 -- 172.21.15.116:0/1040354606 learned_addr learned my addr 172.21.15.116:0/1040354606 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:26:45.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.566+0000 7f7753fff640 1 -- 172.21.15.116:0/1040354606 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f774c0466a0 msgr2=0x7f774c0e6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:45.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.566+0000 7f7753fff640 1 --2- 172.21.15.116:0/1040354606 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f774c0466a0 0x7f774c0e6a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:45.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.566+0000 7f7753fff640 1 -- 172.21.15.116:0/1040354606 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f774c0f55e0 msgr2=0x7f774c0eb380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:45.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.566+0000 7f7753fff640 1 --2- 172.21.15.116:0/1040354606 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f774c0f55e0 0x7f774c0eb380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:45.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.566+0000 7f7753fff640 1 -- 172.21.15.116:0/1040354606 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f774c0e85f0 con 0x7f774c0f8950 2024-04-25T00:26:45.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.570+0000 7f7753fff640 1 --2- 172.21.15.116:0/1040354606 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f774c0f8950 0x7f774c1b2e20 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f775406a140 tx=0x7f7754076be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:45.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.570+0000 7f773bfff640 1 -- 172.21.15.116:0/1040354606 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f775406f720 con 0x7f774c0f8950 2024-04-25T00:26:45.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.570+0000 7f7758f0d640 1 -- 172.21.15.116:0/1040354606 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f774c0e88d0 con 0x7f774c0f8950 2024-04-25T00:26:45.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.570+0000 7f7758f0d640 1 -- 172.21.15.116:0/1040354606 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f774c1b2900 con 0x7f774c0f8950 2024-04-25T00:26:45.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.574+0000 7f773bfff640 1 -- 172.21.15.116:0/1040354606 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f775406f8c0 con 0x7f774c0f8950 2024-04-25T00:26:45.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.574+0000 7f7758f0d640 1 -- 172.21.15.116:0/1040354606 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f774c0053a0 con 0x7f774c0f8950 2024-04-25T00:26:45.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.574+0000 7f773bfff640 1 -- 172.21.15.116:0/1040354606 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f775406fc70 con 0x7f774c0f8950 2024-04-25T00:26:45.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.574+0000 7f773bfff640 1 -- 172.21.15.116:0/1040354606 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7754117440 con 0x7f774c0f8950 2024-04-25T00:26:45.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.574+0000 7f773bfff640 1 --2- 172.21.15.116:0/1040354606 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7740067910 0x7f7740069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:45.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.574+0000 7f773bfff640 1 -- 172.21.15.116:0/1040354606 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 10106+0+0 (secure 0 0 0) 0x7f775418acd0 con 0x7f774c0f8950 2024-04-25T00:26:45.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.578+0000 7f77537fe640 1 --2- 172.21.15.116:0/1040354606 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7740067910 0x7f7740069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:45.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.578+0000 7f77537fe640 1 --2- 172.21.15.116:0/1040354606 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7740067910 0x7f7740069dd0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f7748007fd0 tx=0x7f7748009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:45.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:45.586+0000 7f773bfff640 1 -- 172.21.15.116:0/1040354606 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f7754131080 con 0x7f774c0f8950 2024-04-25T00:26:46.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.294+0000 7f7758f0d640 1 -- 172.21.15.116:0/1040354606 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f774c0f99e0 con 0x7f774c0f8950 2024-04-25T00:26:46.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.294+0000 7f773bfff640 1 -- 172.21.15.116:0/1040354606 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) ==== 95+0+5670 (secure 0 0 0) 0x7f774c0071d0 con 0x7f774c0f8950 2024-04-25T00:26:46.304 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:26:46.304 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":213,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":211,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:26:19.034064+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6485},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6485":{"gid":6485,"name":"a","rank":0,"incarnation":208,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6835/2371126056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2371126056},{"type":"v1","addr":"172.21.15.116:6835","nonce":2371126056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6485,"qdb_cluster":[6485]},"id":1},{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:26:45.740121+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7384},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7384":{"gid":7384,"name":"d","rank":0,"incarnation":213,"state":"up:replay","state_seq":1,"addr":"172.21.15.174:6837/349772087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":349772087},{"type":"v1","addr":"172.21.15.174:6837","nonce":349772087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:26:46.305 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 213 2024-04-25T00:26:46.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.302+0000 7f7758f0d640 1 -- 172.21.15.116:0/1040354606 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7740067910 msgr2=0x7f7740069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:46.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.302+0000 7f7758f0d640 1 --2- 172.21.15.116:0/1040354606 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7740067910 0x7f7740069dd0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f7748007fd0 tx=0x7f7748009040 comp rx=0 tx=0).stop 2024-04-25T00:26:46.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.302+0000 7f7758f0d640 1 -- 172.21.15.116:0/1040354606 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f774c0f8950 msgr2=0x7f774c1b2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:46.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.302+0000 7f7758f0d640 1 --2- 172.21.15.116:0/1040354606 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f774c0f8950 0x7f774c1b2e20 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f775406a140 tx=0x7f7754076be0 comp rx=0 tx=0).stop 2024-04-25T00:26:46.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.302+0000 7f7758f0d640 1 -- 172.21.15.116:0/1040354606 shutdown_connections 2024-04-25T00:26:46.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.302+0000 7f7758f0d640 1 --2- 172.21.15.116:0/1040354606 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7740067910 0x7f7740069dd0 secure :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f7748007fd0 tx=0x7f7748009040 comp rx=0 tx=0).stop 2024-04-25T00:26:46.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.302+0000 7f7758f0d640 1 --2- 172.21.15.116:0/1040354606 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f774c0f8950 0x7f774c1b2e20 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:46.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.302+0000 7f7758f0d640 1 --2- 172.21.15.116:0/1040354606 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f774c0f55e0 0x7f774c0eb380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:46.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.302+0000 7f7758f0d640 1 --2- 172.21.15.116:0/1040354606 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f774c0466a0 0x7f774c0e6a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:46.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.302+0000 7f7758f0d640 1 -- 172.21.15.116:0/1040354606 >> 172.21.15.116:0/1040354606 conn(0x7f774c005ea0 msgr2=0x7f774c0f8d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:46.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.306+0000 7f7758f0d640 1 -- 172.21.15.116:0/1040354606 shutdown_connections 2024-04-25T00:26:46.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.306+0000 7f7758f0d640 1 -- 172.21.15.116:0/1040354606 wait complete. 2024-04-25T00:26:46.347 INFO:tasks.mds_thrash.fs.[b]:mds.c down, removed from mdsmap 2024-04-25T00:26:46.347 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-04-25T00:26:46.347 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:26:46.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.758+0000 7f7ae85c9640 1 Processor -- start 2024-04-25T00:26:46.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.758+0000 7f7ae85c9640 1 -- start start 2024-04-25T00:26:46.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.758+0000 7f7ae85c9640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f7ae0153620 0x7f7ae00c3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:46.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.762+0000 7f7ae6b66640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f7ae0071050 0x7f7ae015df10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:56606/0 (socket says 172.21.15.116:56606) 2024-04-25T00:26:46.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.762+0000 7f7ae6b66640 1 -- 172.21.15.116:0/1992157639 learned_addr learned my addr 172.21.15.116:0/1992157639 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:26:46.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.762+0000 7f7ae85c9640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ae006f8c0 con 0x7f7ae0071050 2024-04-25T00:26:46.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.762+0000 7f7ae85c9640 1 -- 172.21.15.116:0/1992157639 --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ae00c5420 con 0x7f7ae0150d20 2024-04-25T00:26:46.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.762+0000 7f7ae5b64640 1 --2- 172.21.15.116:0/1992157639 >> v2:172.21.15.174:3300/0 conn(0x7f7ae0153620 0x7f7ae00c3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:46.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.762+0000 7f7ae85c9640 1 -- 172.21.15.116:0/1992157639 --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f7ae0153a80 con 0x7f7ae0153620 2024-04-25T00:26:46.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.762+0000 7f7ae5363640 1 -- 172.21.15.116:0/1992157639 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3300616438 0 0) 0x7f7ae006f8c0 con 0x7f7ae0071050 2024-04-25T00:26:46.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.762+0000 7f7ae5363640 1 -- 172.21.15.116:0/1992157639 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7acc003530 con 0x7f7ae0071050 2024-04-25T00:26:46.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.762+0000 7f7ae5363640 1 -- 172.21.15.116:0/1992157639 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3405937947 0 0) 0x7f7ae00c5420 con 0x7f7ae0150d20 2024-04-25T00:26:46.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.762+0000 7f7ae5363640 1 -- 172.21.15.116:0/1992157639 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7ae006f8c0 con 0x7f7ae0150d20 2024-04-25T00:26:46.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.762+0000 7f7ae5b64640 1 -- 172.21.15.116:0/1992157639 >> v1:172.21.15.174:6789/0 conn(0x7f7ae0150d20 legacy=0x7f7ae00c3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:46.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.762+0000 7f7ae5b64640 1 -- 172.21.15.116:0/1992157639 >> v1:172.21.15.116:6789/0 conn(0x7f7ae0071050 legacy=0x7f7ae015df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:46.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.762+0000 7f7ae5b64640 1 -- 172.21.15.116:0/1992157639 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ae00c5700 con 0x7f7ae0153620 2024-04-25T00:26:46.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.766+0000 7f7ae5b64640 1 --2- 172.21.15.116:0/1992157639 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7ae0153620 0x7f7ae00c3240 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f7ad00097c0 tx=0x7f7ad001c460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cbe6897a6900ce8d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:46.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.770+0000 7f7ae5363640 1 -- 172.21.15.116:0/1992157639 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ad00b0070 con 0x7f7ae0153620 2024-04-25T00:26:46.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.770+0000 7f7ae5363640 1 -- 172.21.15.116:0/1992157639 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ad00ab600 con 0x7f7ae0153620 2024-04-25T00:26:46.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.770+0000 7f7ae5363640 1 -- 172.21.15.116:0/1992157639 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ad00ab9d0 con 0x7f7ae0153620 2024-04-25T00:26:46.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.770+0000 7f7ae85c9640 1 -- 172.21.15.116:0/1992157639 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7ae0153620 msgr2=0x7f7ae00c3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:46.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.770+0000 7f7ae85c9640 1 --2- 172.21.15.116:0/1992157639 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7ae0153620 0x7f7ae00c3240 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f7ad00097c0 tx=0x7f7ad001c460 comp rx=0 tx=0).stop 2024-04-25T00:26:46.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.770+0000 7f7ae85c9640 1 -- 172.21.15.116:0/1992157639 shutdown_connections 2024-04-25T00:26:46.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.770+0000 7f7ae85c9640 1 --2- 172.21.15.116:0/1992157639 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7ae0153620 0x7f7ae00c3240 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:46.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.770+0000 7f7ae85c9640 1 -- 172.21.15.116:0/1992157639 >> 172.21.15.116:0/1992157639 conn(0x7f7ae00c9160 msgr2=0x7f7ae00c9570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:46.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.770+0000 7f7ae85c9640 1 -- 172.21.15.116:0/1992157639 shutdown_connections 2024-04-25T00:26:46.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.770+0000 7f7ae85c9640 1 -- 172.21.15.116:0/1992157639 wait complete. 2024-04-25T00:26:46.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.774+0000 7f7ae85c9640 1 Processor -- start 2024-04-25T00:26:46.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.774+0000 7f7ae85c9640 1 -- start start 2024-04-25T00:26:46.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.774+0000 7f7ae85c9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7ae0071050 0x7f7ae01443d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:46.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.774+0000 7f7ae85c9640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7ae0150d20 0x7f7ae00add60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:46.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.774+0000 7f7ae85c9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ae0153620 0x7f7ae00ae2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:46.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.774+0000 7f7ae85c9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7ae006f4c0 con 0x7f7ae0153620 2024-04-25T00:26:46.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.774+0000 7f7ae85c9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7ae006fba0 con 0x7f7ae0071050 2024-04-25T00:26:46.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.774+0000 7f7ae85c9640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f7ae006f640 con 0x7f7ae0150d20 2024-04-25T00:26:46.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.774+0000 7f7ae6b66640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ae0153620 0x7f7ae00ae2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:46.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.774+0000 7f7ae5b64640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7ae0150d20 0x7f7ae00add60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:46.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.774+0000 7f7ae6b66640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ae0153620 0x7f7ae00ae2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57022/0 (socket says 172.21.15.116:57022) 2024-04-25T00:26:46.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.774+0000 7f7ae6b66640 1 -- 172.21.15.116:0/3034557018 learned_addr learned my addr 172.21.15.116:0/3034557018 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:26:46.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.774+0000 7f7ae6b66640 1 -- 172.21.15.116:0/3034557018 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7ae0150d20 msgr2=0x7f7ae00add60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:46.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.774+0000 7f7ae6b66640 1 --2- 172.21.15.116:0/3034557018 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7ae0150d20 0x7f7ae00add60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:46.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.774+0000 7f7ae6b66640 1 -- 172.21.15.116:0/3034557018 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7ae0071050 msgr2=0x7f7ae01443d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T00:26:46.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.774+0000 7f7ae6b66640 1 --2- 172.21.15.116:0/3034557018 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7ae0071050 0x7f7ae01443d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:46.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.774+0000 7f7ae6b66640 1 -- 172.21.15.116:0/3034557018 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ae00ae7e0 con 0x7f7ae0153620 2024-04-25T00:26:46.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.774+0000 7f7ae5b64640 1 --2- 172.21.15.116:0/3034557018 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7ae0150d20 0x7f7ae00add60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:26:46.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.774+0000 7f7ae6b66640 1 --2- 172.21.15.116:0/3034557018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ae0153620 0x7f7ae00ae2a0 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f7ad800cb40 tx=0x7f7ad800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:46.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.774+0000 7f7ad6ffd640 1 -- 172.21.15.116:0/3034557018 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ad8017070 con 0x7f7ae0153620 2024-04-25T00:26:46.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.774+0000 7f7ad6ffd640 1 -- 172.21.15.116:0/3034557018 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ad800aca0 con 0x7f7ae0153620 2024-04-25T00:26:46.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.774+0000 7f7ad6ffd640 1 -- 172.21.15.116:0/3034557018 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ad8012620 con 0x7f7ae0153620 2024-04-25T00:26:46.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.774+0000 7f7ae85c9640 1 -- 172.21.15.116:0/3034557018 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ae00a9ac0 con 0x7f7ae0153620 2024-04-25T00:26:46.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.774+0000 7f7ae85c9640 1 -- 172.21.15.116:0/3034557018 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ae0140dd0 con 0x7f7ae0153620 2024-04-25T00:26:46.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.778+0000 7f7ad6ffd640 1 -- 172.21.15.116:0/3034557018 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7ad801a070 con 0x7f7ae0153620 2024-04-25T00:26:46.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.778+0000 7f7ad6ffd640 1 --2- 172.21.15.116:0/3034557018 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7acc06c350 0x7f7acc06e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:46.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.778+0000 7f7ad6ffd640 1 -- 172.21.15.116:0/3034557018 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 10192+0+0 (secure 0 0 0) 0x7f7ad8086bb0 con 0x7f7ae0153620 2024-04-25T00:26:46.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.778+0000 7f7ae6365640 1 --2- 172.21.15.116:0/3034557018 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7acc06c350 0x7f7acc06e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:46.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.778+0000 7f7ae6365640 1 --2- 172.21.15.116:0/3034557018 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7acc06c350 0x7f7acc06e810 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f7adc007fd0 tx=0x7f7adc007900 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:46.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.782+0000 7f7ae85c9640 1 -- 172.21.15.116:0/3034557018 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ac80052a0 con 0x7f7ae0153620 2024-04-25T00:26:46.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:46.786+0000 7f7ad6ffd640 1 -- 172.21.15.116:0/3034557018 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f7ad8055fa0 con 0x7f7ae0153620 2024-04-25T00:26:47.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:47.462+0000 7f7ae85c9640 1 -- 172.21.15.116:0/3034557018 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7ac8005590 con 0x7f7ae0153620 2024-04-25T00:26:47.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:47.462+0000 7f7ad6ffd640 1 -- 172.21.15.116:0/3034557018 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 214 v214) ==== 95+0+5673 (secure 0 0 0) 0x7f7ad805a2a0 con 0x7f7ae0153620 2024-04-25T00:26:47.463 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:26:47.463 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":214,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":211,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:26:19.034064+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6485},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6485":{"gid":6485,"name":"a","rank":0,"incarnation":208,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6835/2371126056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2371126056},{"type":"v1","addr":"172.21.15.116:6835","nonce":2371126056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6485,"qdb_cluster":[6485]},"id":1},{"mdsmap":{"epoch":214,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:26:46.498071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7384},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7384":{"gid":7384,"name":"d","rank":0,"incarnation":213,"state":"up:reconnect","state_seq":5,"addr":"172.21.15.174:6837/349772087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":349772087},{"type":"v1","addr":"172.21.15.174:6837","nonce":349772087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:26:47.463 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 214 2024-04-25T00:26:47.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:47.470+0000 7f7ad4ff9640 1 -- 172.21.15.116:0/3034557018 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7acc06c350 msgr2=0x7f7acc06e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:47.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:47.470+0000 7f7ad4ff9640 1 --2- 172.21.15.116:0/3034557018 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7acc06c350 0x7f7acc06e810 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f7adc007fd0 tx=0x7f7adc007900 comp rx=0 tx=0).stop 2024-04-25T00:26:47.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:47.470+0000 7f7ad4ff9640 1 -- 172.21.15.116:0/3034557018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ae0153620 msgr2=0x7f7ae00ae2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:47.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:47.470+0000 7f7ad4ff9640 1 --2- 172.21.15.116:0/3034557018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ae0153620 0x7f7ae00ae2a0 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f7ad800cb40 tx=0x7f7ad800a060 comp rx=0 tx=0).stop 2024-04-25T00:26:47.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:47.470+0000 7f7ad4ff9640 1 -- 172.21.15.116:0/3034557018 shutdown_connections 2024-04-25T00:26:47.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:47.470+0000 7f7ad4ff9640 1 --2- 172.21.15.116:0/3034557018 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7acc06c350 0x7f7acc06e810 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:47.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:47.470+0000 7f7ad4ff9640 1 --2- 172.21.15.116:0/3034557018 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7ae0153620 0x7f7ae00ae2a0 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:47.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:47.470+0000 7f7ad4ff9640 1 --2- 172.21.15.116:0/3034557018 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7ae0150d20 0x7f7ae00add60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:47.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:47.470+0000 7f7ad4ff9640 1 --2- 172.21.15.116:0/3034557018 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7ae0071050 0x7f7ae01443d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:47.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:47.470+0000 7f7ad4ff9640 1 -- 172.21.15.116:0/3034557018 >> 172.21.15.116:0/3034557018 conn(0x7f7ae00c9160 msgr2=0x7f7ae0152ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:47.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:47.470+0000 7f7ad4ff9640 1 -- 172.21.15.116:0/3034557018 shutdown_connections 2024-04-25T00:26:47.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:47.470+0000 7f7ad4ff9640 1 -- 172.21.15.116:0/3034557018 wait complete. 2024-04-25T00:26:47.523 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 214, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:04.678737+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 211, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6485": { "addr": "172.21.15.116:6835/2371126056", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6834", "nonce": 2371126056, "type": "v2" }, { "addr": "172.21.15.116:6835", "nonce": 2371126056, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6485, "incarnation": 208, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 10 } }, "last_failure": 0, "last_failure_osd_epoch": 68, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T00:26:19.034064+0000", "qdb_cluster": [ 6485 ], "qdb_leader": 6485, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6485 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:18.825887+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 214, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_7384": { "addr": "172.21.15.174:6837/349772087", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6836", "nonce": 349772087, "type": "v2" }, { "addr": "172.21.15.174:6837", "nonce": 349772087, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 7384, "incarnation": 213, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:reconnect", "state_seq": 5 } }, "last_failure": 0, "last_failure_osd_epoch": 69, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-04-25T00:26:46.498071+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7384 } } } ], "standbys": [ { "addr": "172.21.15.174:6835/4112145782", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6834", "nonce": 4112145782, "type": "v2" }, { "addr": "172.21.15.174:6835", "nonce": 4112145782, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 112, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5582, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-25T00:26:49.525 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:26:49.912 INFO:tasks.ceph.mds.d.smithi174.stderr:2024-04-25T00:26:49.907+0000 7f1d64c07640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T00:26:50.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:50.102+0000 7f1702cac640 1 Processor -- start 2024-04-25T00:26:50.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:50.102+0000 7f1702cac640 1 -- start start 2024-04-25T00:26:50.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:50.106+0000 7f1702cac640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f16fc157240 0x7f16fc159bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:50.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:50.106+0000 7f1702cac640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f16fc071130 0x7f16fc071580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:50.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:50.106+0000 7f1702cac640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16fc157620 con 0x7f16fc15d130 2024-04-25T00:26:50.110 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:50.106+0000 7f1702cac640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f16fc070ca0 con 0x7f16fc157240 2024-04-25T00:26:50.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:50.106+0000 7f1702cac640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f16fc15a1a0 con 0x7f16fc071130 2024-04-25T00:26:50.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:50.110+0000 7f1701caa640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f16fc157240 0x7f16fc159bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:50.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:50.110+0000 7f1701caa640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f16fc157240 0x7f16fc159bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:57028/0 (socket says 172.21.15.116:57028) 2024-04-25T00:26:50.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:50.110+0000 7f1701caa640 1 -- 172.21.15.116:0/2742926229 learned_addr learned my addr 172.21.15.116:0/2742926229 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:26:50.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:50.110+0000 7f1701caa640 1 -- 172.21.15.116:0/2742926229 >> v2:172.21.15.174:3300/0 conn(0x7f16fc071130 msgr2=0x7f16fc071580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:26:50.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:50.110+0000 7f1701caa640 1 --2- 172.21.15.116:0/2742926229 >> v2:172.21.15.174:3300/0 conn(0x7f16fc071130 0x7f16fc071580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:50.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:50.110+0000 7f1701caa640 1 -- 172.21.15.116:0/2742926229 >> v1:172.21.15.174:6789/0 conn(0x7f16fc15d130 legacy=0x7f16fc0c2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:50.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:50.110+0000 7f1701caa640 1 -- 172.21.15.116:0/2742926229 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16fc15a480 con 0x7f16fc157240 2024-04-25T00:26:50.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:50.130+0000 7f1701caa640 1 --2- 172.21.15.116:0/2742926229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16fc157240 0x7f16fc159bc0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f16fc0c3410 tx=0x7f16f801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1db4c93099369b4a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:50.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:50.134+0000 7f1700ca8640 1 -- 172.21.15.116:0/2742926229 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f80b5070 con 0x7f16fc157240 2024-04-25T00:26:50.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:50.134+0000 7f1700ca8640 1 -- 172.21.15.116:0/2742926229 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16f80a8a30 con 0x7f16fc157240 2024-04-25T00:26:50.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:50.134+0000 7f1700ca8640 1 -- 172.21.15.116:0/2742926229 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f80a8e00 con 0x7f16fc157240 2024-04-25T00:26:50.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:50.138+0000 7f1702cac640 1 -- 172.21.15.116:0/2742926229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16fc157240 msgr2=0x7f16fc159bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:50.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:50.138+0000 7f1702cac640 1 --2- 172.21.15.116:0/2742926229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16fc157240 0x7f16fc159bc0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f16fc0c3410 tx=0x7f16f801ed30 comp rx=0 tx=0).stop 2024-04-25T00:26:50.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:50.158+0000 7f1702cac640 1 -- 172.21.15.116:0/2742926229 shutdown_connections 2024-04-25T00:26:50.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:50.158+0000 7f1702cac640 1 --2- 172.21.15.116:0/2742926229 >> v2:172.21.15.174:3300/0 conn(0x7f16fc071130 0x7f16fc071580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:50.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:50.158+0000 7f1702cac640 1 --2- 172.21.15.116:0/2742926229 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16fc157240 0x7f16fc159bc0 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:50.158 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:50.158+0000 7f1702cac640 1 -- 172.21.15.116:0/2742926229 >> 172.21.15.116:0/2742926229 conn(0x7f16fc0c3940 msgr2=0x7f16fc0c3d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:50.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:50.174+0000 7f1702cac640 1 -- 172.21.15.116:0/2742926229 shutdown_connections 2024-04-25T00:26:50.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:50.174+0000 7f1702cac640 1 -- 172.21.15.116:0/2742926229 wait complete. 2024-04-25T00:26:50.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:50.174+0000 7f1702cac640 1 Processor -- start 2024-04-25T00:26:50.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:50.178+0000 7f1702cac640 1 -- start start 2024-04-25T00:26:50.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:50.178+0000 7f1702cac640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16fc071130 0x7f16fc0c64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:50.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:50.178+0000 7f1702cac640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f16fc157240 0x7f16fc15c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:50.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:50.178+0000 7f1702cac640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f16fc15d130 0x7f16fc1528a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:50.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:50.178+0000 7f1702cac640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f16fc0c1470 con 0x7f16fc071130 2024-04-25T00:26:50.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:50.178+0000 7f1702cac640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f16fc15aa40 con 0x7f16fc15d130 2024-04-25T00:26:50.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:50.178+0000 7f1702cac640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f16fc0c2f60 con 0x7f16fc157240 2024-04-25T00:26:50.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:50.178+0000 7f17014a9640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f16fc157240 0x7f16fc15c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:50.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:50.178+0000 7f17014a9640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f16fc157240 0x7f16fc15c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:39490/0 (socket says 172.21.15.116:39490) 2024-04-25T00:26:50.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:50.178+0000 7f17014a9640 1 -- 172.21.15.116:0/1474618317 learned_addr learned my addr 172.21.15.116:0/1474618317 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:26:50.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:50.178+0000 7f1701caa640 1 --2- 172.21.15.116:0/1474618317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16fc071130 0x7f16fc0c64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:50.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:50.178+0000 7f17014a9640 1 -- 172.21.15.116:0/1474618317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f16fc15d130 msgr2=0x7f16fc1528a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T00:26:50.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:50.178+0000 7f17014a9640 1 --2- 172.21.15.116:0/1474618317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f16fc15d130 0x7f16fc1528a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:50.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:50.178+0000 7f17014a9640 1 -- 172.21.15.116:0/1474618317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16fc071130 msgr2=0x7f16fc0c64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:50.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:50.178+0000 7f17014a9640 1 --2- 172.21.15.116:0/1474618317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16fc071130 0x7f16fc0c64b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:50.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:50.178+0000 7f17014a9640 1 -- 172.21.15.116:0/1474618317 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16fc165060 con 0x7f16fc157240 2024-04-25T00:26:50.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:50.178+0000 7f17014a9640 1 --2- 172.21.15.116:0/1474618317 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f16fc157240 0x7f16fc15c8a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f16f400c7d0 tx=0x7f16f400cc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:50.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:50.214+0000 7f16f27fc640 1 -- 172.21.15.116:0/1474618317 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f4003b80 con 0x7f16fc157240 2024-04-25T00:26:50.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:50.214+0000 7f16f27fc640 1 -- 172.21.15.116:0/1474618317 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16f4003d20 con 0x7f16fc157240 2024-04-25T00:26:50.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:50.214+0000 7f16f27fc640 1 -- 172.21.15.116:0/1474618317 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16f400aa90 con 0x7f16fc157240 2024-04-25T00:26:50.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:50.214+0000 7f1702cac640 1 -- 172.21.15.116:0/1474618317 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16fc0c4bd0 con 0x7f16fc157240 2024-04-25T00:26:50.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:50.214+0000 7f1702cac640 1 -- 172.21.15.116:0/1474618317 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f16fc152de0 con 0x7f16fc157240 2024-04-25T00:26:50.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:50.218+0000 7f16f27fc640 1 -- 172.21.15.116:0/1474618317 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f16f400ac30 con 0x7f16fc157240 2024-04-25T00:26:50.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:50.218+0000 7f1702cac640 1 -- 172.21.15.116:0/1474618317 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16fc04f510 con 0x7f16fc157240 2024-04-25T00:26:50.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:50.226+0000 7f16f27fc640 1 --2- 172.21.15.116:0/1474618317 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f16e00679c0 0x7f16e0069e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:50.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:50.226+0000 7f16f27fc640 1 -- 172.21.15.116:0/1474618317 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(69..69 src has 1..69) ==== 10192+0+0 (secure 0 0 0) 0x7f16f4086be0 con 0x7f16fc157240 2024-04-25T00:26:50.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:50.226+0000 7f1701caa640 1 --2- 172.21.15.116:0/1474618317 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f16e00679c0 0x7f16e0069e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:50.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:50.226+0000 7f1701caa640 1 --2- 172.21.15.116:0/1474618317 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f16e00679c0 0x7f16e0069e80 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f16f80b2000 tx=0x7f16f80aa070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:50.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:50.230+0000 7f16f27fc640 1 -- 172.21.15.116:0/1474618317 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f16f4056620 con 0x7f16fc157240 2024-04-25T00:26:51.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:51.482+0000 7f1702cac640 1 -- 172.21.15.116:0/1474618317 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f16fc0c0eb0 con 0x7f16fc157240 2024-04-25T00:26:51.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:51.482+0000 7f16f27fc640 1 -- 172.21.15.116:0/1474618317 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 216 v216) ==== 95+0+5677 (secure 0 0 0) 0x7f16f4055f20 con 0x7f16fc157240 2024-04-25T00:26:51.487 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:26:51.487 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":216,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":211,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:26:19.034064+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6485},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6485":{"gid":6485,"name":"a","rank":0,"incarnation":208,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6835/2371126056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2371126056},{"type":"v1","addr":"172.21.15.116:6835","nonce":2371126056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6485,"qdb_cluster":[6485]},"id":1},{"mdsmap":{"epoch":216,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:26:49.039944+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7384},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7384":{"gid":7384,"name":"d","rank":0,"incarnation":213,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/349772087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":349772087},{"type":"v1","addr":"172.21.15.174:6837","nonce":349772087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7384,"qdb_cluster":[7384]},"id":2}]} 2024-04-25T00:26:51.488 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 216 2024-04-25T00:26:51.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:51.494+0000 7f1702cac640 1 -- 172.21.15.116:0/1474618317 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f16e00679c0 msgr2=0x7f16e0069e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:51.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:51.494+0000 7f1702cac640 1 --2- 172.21.15.116:0/1474618317 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f16e00679c0 0x7f16e0069e80 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f16f80b2000 tx=0x7f16f80aa070 comp rx=0 tx=0).stop 2024-04-25T00:26:51.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:51.494+0000 7f1702cac640 1 -- 172.21.15.116:0/1474618317 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f16fc157240 msgr2=0x7f16fc15c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:51.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:51.494+0000 7f1702cac640 1 --2- 172.21.15.116:0/1474618317 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f16fc157240 0x7f16fc15c8a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f16f400c7d0 tx=0x7f16f400cc90 comp rx=0 tx=0).stop 2024-04-25T00:26:51.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:51.498+0000 7f1702cac640 1 -- 172.21.15.116:0/1474618317 shutdown_connections 2024-04-25T00:26:51.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:51.498+0000 7f1702cac640 1 --2- 172.21.15.116:0/1474618317 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f16e00679c0 0x7f16e0069e80 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:51.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:51.498+0000 7f1702cac640 1 --2- 172.21.15.116:0/1474618317 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f16fc15d130 0x7f16fc1528a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:51.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:51.498+0000 7f1702cac640 1 --2- 172.21.15.116:0/1474618317 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f16fc157240 0x7f16fc15c8a0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:51.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:51.498+0000 7f1702cac640 1 --2- 172.21.15.116:0/1474618317 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f16fc071130 0x7f16fc0c64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:51.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:51.498+0000 7f1702cac640 1 -- 172.21.15.116:0/1474618317 >> 172.21.15.116:0/1474618317 conn(0x7f16fc0c3940 msgr2=0x7f16fc06d900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:51.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:51.498+0000 7f1702cac640 1 -- 172.21.15.116:0/1474618317 shutdown_connections 2024-04-25T00:26:51.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:51.498+0000 7f1702cac640 1 -- 172.21.15.116:0/1474618317 wait complete. 2024-04-25T00:26:51.593 INFO:tasks.mds_thrash.fs.[b]:mds.d has gained rank=0, replacing gid=8496 2024-04-25T00:26:51.593 INFO:tasks.mds_thrash.fs.[b]:waiting for 4 secs before reviving mds.c 2024-04-25T00:26:55.594 INFO:tasks.mds_thrash.fs.[b]:reviving mds.c 2024-04-25T00:26:55.594 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-25T00:26:55.594 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-25T00:26:55.596 INFO:tasks.ceph.mds.c:Started 2024-04-25T00:26:55.596 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:26:56.007 INFO:tasks.ceph.mds.c.smithi116.stdout:starting mds.c at 2024-04-25T00:26:56.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:56.126+0000 7fa40494b640 1 Processor -- start 2024-04-25T00:26:56.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:56.126+0000 7fa40494b640 1 -- start start 2024-04-25T00:26:56.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:56.126+0000 7fa40494b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa40013acb0 0x7fa400071180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:56.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:56.126+0000 7fa40494b640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fa40006cb70 0x7fa4000716c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:56.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:56.126+0000 7fa40494b640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa400073320 con 0x7fa40013a360 2024-04-25T00:26:56.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:56.126+0000 7fa40494b640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fa400070cf0 con 0x7fa40006cb70 2024-04-25T00:26:56.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:56.126+0000 7fa40494b640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa400073040 con 0x7fa40013acb0 2024-04-25T00:26:56.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:56.130+0000 7fa3fe59c640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fa40013a360 0x7fa4000c29a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:34148/0 (socket says 172.21.15.116:34148) 2024-04-25T00:26:56.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:56.130+0000 7fa3fe59c640 1 -- 172.21.15.116:0/634727117 learned_addr learned my addr 172.21.15.116:0/634727117 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:26:56.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:56.130+0000 7fa3fdd9b640 1 --2- 172.21.15.116:0/634727117 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa40013acb0 0x7fa400071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:56.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:56.130+0000 7fa3fdd9b640 1 -- 172.21.15.116:0/634727117 >> v2:172.21.15.174:3300/0 conn(0x7fa40006cb70 msgr2=0x7fa4000716c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T00:26:56.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:56.130+0000 7fa3fdd9b640 1 --2- 172.21.15.116:0/634727117 >> v2:172.21.15.174:3300/0 conn(0x7fa40006cb70 0x7fa4000716c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:56.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:56.130+0000 7fa3fdd9b640 1 -- 172.21.15.116:0/634727117 >> v1:172.21.15.174:6789/0 conn(0x7fa40013a360 legacy=0x7fa4000c29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:56.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:56.130+0000 7fa3fdd9b640 1 -- 172.21.15.116:0/634727117 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4001557e0 con 0x7fa40013acb0 2024-04-25T00:26:56.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:56.134+0000 7fa3fdd9b640 1 --2- 172.21.15.116:0/634727117 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa40013acb0 0x7fa400071180 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa3f0008820 tx=0x7fa3f00206c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef12ea0a493ecde7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:56.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:56.134+0000 7fa3fd59a640 1 -- 172.21.15.116:0/634727117 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f00ae070 con 0x7fa40013acb0 2024-04-25T00:26:56.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:56.134+0000 7fa3fd59a640 1 -- 172.21.15.116:0/634727117 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3f00b7ac0 con 0x7fa40013acb0 2024-04-25T00:26:56.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:56.134+0000 7fa3fd59a640 1 -- 172.21.15.116:0/634727117 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f00a8a40 con 0x7fa40013acb0 2024-04-25T00:26:56.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:56.138+0000 7fa40494b640 1 -- 172.21.15.116:0/634727117 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa40013acb0 msgr2=0x7fa400071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:56.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:56.138+0000 7fa40494b640 1 --2- 172.21.15.116:0/634727117 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa40013acb0 0x7fa400071180 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa3f0008820 tx=0x7fa3f00206c0 comp rx=0 tx=0).stop 2024-04-25T00:26:56.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:56.146+0000 7fa40494b640 1 -- 172.21.15.116:0/634727117 shutdown_connections 2024-04-25T00:26:56.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:56.146+0000 7fa40494b640 1 --2- 172.21.15.116:0/634727117 >> v2:172.21.15.174:3300/0 conn(0x7fa40006cb70 0x7fa4000716c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:56.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:56.146+0000 7fa40494b640 1 --2- 172.21.15.116:0/634727117 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa40013acb0 0x7fa400071180 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:56.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:56.146+0000 7fa40494b640 1 -- 172.21.15.116:0/634727117 >> 172.21.15.116:0/634727117 conn(0x7fa4000c3a90 msgr2=0x7fa4000c3ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:56.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:56.146+0000 7fa40494b640 1 -- 172.21.15.116:0/634727117 shutdown_connections 2024-04-25T00:26:56.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:56.146+0000 7fa40494b640 1 -- 172.21.15.116:0/634727117 wait complete. 2024-04-25T00:26:56.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:56.150+0000 7fa40494b640 1 Processor -- start 2024-04-25T00:26:56.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:56.150+0000 7fa40494b640 1 -- start start 2024-04-25T00:26:56.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:56.150+0000 7fa40494b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa40006cb70 0x7fa40012d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:56.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:56.150+0000 7fa40494b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa40013a360 0x7fa40006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:56.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:56.150+0000 7fa40494b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa40013acb0 0x7fa400152d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:56.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:56.150+0000 7fa40494b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa40015f280 con 0x7fa40013acb0 2024-04-25T00:26:56.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:56.150+0000 7fa40494b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa40015f100 con 0x7fa40013a360 2024-04-25T00:26:56.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:56.150+0000 7fa40494b640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa40015f400 con 0x7fa40006cb70 2024-04-25T00:26:56.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:56.150+0000 7fa3fdd9b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa40013a360 0x7fa40006d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:56.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:56.150+0000 7fa3fdd9b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa40013a360 0x7fa40006d5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:55402/0 (socket says 172.21.15.116:55402) 2024-04-25T00:26:56.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:56.150+0000 7fa3fdd9b640 1 -- 172.21.15.116:0/505730063 learned_addr learned my addr 172.21.15.116:0/505730063 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:26:56.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:56.150+0000 7fa3fdd9b640 1 -- 172.21.15.116:0/505730063 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa40006cb70 msgr2=0x7fa40012d300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T00:26:56.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:56.150+0000 7fa3fdd9b640 1 --2- 172.21.15.116:0/505730063 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa40006cb70 0x7fa40012d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:56.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:56.150+0000 7fa3fdd9b640 1 -- 172.21.15.116:0/505730063 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa40013acb0 msgr2=0x7fa400152d40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T00:26:56.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:56.150+0000 7fa3fdd9b640 1 --2- 172.21.15.116:0/505730063 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa40013acb0 0x7fa400152d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:56.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:56.150+0000 7fa3fdd9b640 1 -- 172.21.15.116:0/505730063 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa40012e6a0 con 0x7fa40013a360 2024-04-25T00:26:56.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:56.154+0000 7fa3fdd9b640 1 --2- 172.21.15.116:0/505730063 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa40013a360 0x7fa40006d5c0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fa3f0008820 tx=0x7fa3f001d780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:56.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:56.158+0000 7fa3eeffd640 1 -- 172.21.15.116:0/505730063 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f00ae040 con 0x7fa40013a360 2024-04-25T00:26:56.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:56.158+0000 7fa40494b640 1 -- 172.21.15.116:0/505730063 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa40012df70 con 0x7fa40013a360 2024-04-25T00:26:56.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:56.158+0000 7fa40494b640 1 -- 172.21.15.116:0/505730063 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa40012c820 con 0x7fa40013a360 2024-04-25T00:26:56.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:56.158+0000 7fa3eeffd640 1 -- 172.21.15.116:0/505730063 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3f00b7450 con 0x7fa40013a360 2024-04-25T00:26:56.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:56.158+0000 7fa3eeffd640 1 -- 172.21.15.116:0/505730063 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f00b7840 con 0x7fa40013a360 2024-04-25T00:26:56.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:56.162+0000 7fa3eeffd640 1 -- 172.21.15.116:0/505730063 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa3f00b2460 con 0x7fa40013a360 2024-04-25T00:26:56.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:56.162+0000 7fa3eeffd640 1 --2- 172.21.15.116:0/505730063 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa3e00679c0 0x7fa3e0069e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:26:56.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:56.162+0000 7fa3fe59c640 1 --2- 172.21.15.116:0/505730063 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa3e00679c0 0x7fa3e0069e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:26:56.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:56.162+0000 7fa3eeffd640 1 -- 172.21.15.116:0/505730063 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 10192+0+0 (secure 0 0 0) 0x7fa3f0120570 con 0x7fa40013a360 2024-04-25T00:26:56.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:56.162+0000 7fa3fe59c640 1 --2- 172.21.15.116:0/505730063 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa3e00679c0 0x7fa3e0069e80 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fa3f8001e30 tx=0x7fa3f8007d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:26:56.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:56.170+0000 7fa40494b640 1 -- 172.21.15.116:0/505730063 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3e40052a0 con 0x7fa40013a360 2024-04-25T00:26:56.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:56.190+0000 7fa3eeffd640 1 -- 172.21.15.116:0/505730063 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fa3f00c5070 con 0x7fa40013a360 2024-04-25T00:26:57.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:57.190+0000 7fa40494b640 1 -- 172.21.15.116:0/505730063 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa3e4005590 con 0x7fa40013a360 2024-04-25T00:26:57.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:57.190+0000 7fa3eeffd640 1 -- 172.21.15.116:0/505730063 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 217 v217) ==== 95+0+6506 (secure 0 0 0) 0x7fa3f001a6f0 con 0x7fa40013a360 2024-04-25T00:26:57.192 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:26:57.192 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":217,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":8652,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/3891103804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3891103804},{"type":"v1","addr":"172.21.15.116:6833","nonce":3891103804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":217}],"filesystems":[{"mdsmap":{"epoch":211,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:26:19.034064+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6485},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6485":{"gid":6485,"name":"a","rank":0,"incarnation":208,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6835/2371126056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2371126056},{"type":"v1","addr":"172.21.15.116:6835","nonce":2371126056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6485,"qdb_cluster":[6485]},"id":1},{"mdsmap":{"epoch":216,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:26:49.039944+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7384},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7384":{"gid":7384,"name":"d","rank":0,"incarnation":213,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/349772087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":349772087},{"type":"v1","addr":"172.21.15.174:6837","nonce":349772087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7384,"qdb_cluster":[7384]},"id":2}]} 2024-04-25T00:26:57.192 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 217 2024-04-25T00:26:57.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:57.198+0000 7fa3ecff9640 1 -- 172.21.15.116:0/505730063 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa3e00679c0 msgr2=0x7fa3e0069e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:57.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:57.198+0000 7fa3ecff9640 1 --2- 172.21.15.116:0/505730063 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa3e00679c0 0x7fa3e0069e80 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fa3f8001e30 tx=0x7fa3f8007d30 comp rx=0 tx=0).stop 2024-04-25T00:26:57.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:57.198+0000 7fa3ecff9640 1 -- 172.21.15.116:0/505730063 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa40013a360 msgr2=0x7fa40006d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:26:57.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:57.198+0000 7fa3ecff9640 1 --2- 172.21.15.116:0/505730063 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa40013a360 0x7fa40006d5c0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fa3f0008820 tx=0x7fa3f001d780 comp rx=0 tx=0).stop 2024-04-25T00:26:57.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:57.198+0000 7fa3ecff9640 1 -- 172.21.15.116:0/505730063 shutdown_connections 2024-04-25T00:26:57.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:57.198+0000 7fa3ecff9640 1 --2- 172.21.15.116:0/505730063 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa3e00679c0 0x7fa3e0069e80 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:57.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:57.198+0000 7fa3ecff9640 1 --2- 172.21.15.116:0/505730063 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa40013acb0 0x7fa400152d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:57.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:57.198+0000 7fa3ecff9640 1 --2- 172.21.15.116:0/505730063 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa40013a360 0x7fa40006d5c0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:57.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:57.198+0000 7fa3ecff9640 1 --2- 172.21.15.116:0/505730063 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa40006cb70 0x7fa40012d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:26:57.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:57.198+0000 7fa3ecff9640 1 -- 172.21.15.116:0/505730063 >> 172.21.15.116:0/505730063 conn(0x7fa4000c3a90 msgr2=0x7fa400138c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:26:57.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:57.202+0000 7fa3ecff9640 1 -- 172.21.15.116:0/505730063 shutdown_connections 2024-04-25T00:26:57.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:26:57.202+0000 7fa3ecff9640 1 -- 172.21.15.116:0/505730063 wait complete. 2024-04-25T00:26:57.365 INFO:tasks.mds_thrash.fs.[b]:mds.c reported in up:standby state 2024-04-25T00:26:57.365 INFO:tasks.mds_thrash.fs.[b]:waiting for 5 secs before thrashing 2024-04-25T00:27:02.366 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:27:03.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.098+0000 7f4bb5769640 1 Processor -- start 2024-04-25T00:27:03.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.098+0000 7f4bb5769640 1 -- start start 2024-04-25T00:27:03.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.102+0000 7f4bb5769640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4bb013acb0 0x7f4bb00c29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:03.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.102+0000 7f4bb5769640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f4bb00c2ee0 0x7f4bb0071180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:03.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.102+0000 7f4bb5769640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bb0073320 con 0x7f4bb00c45b0 2024-04-25T00:27:03.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.102+0000 7f4bb5769640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f4bb0070cf0 con 0x7f4bb00c2ee0 2024-04-25T00:27:03.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.102+0000 7f4bb5769640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f4bb0073040 con 0x7f4bb013acb0 2024-04-25T00:27:03.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.102+0000 7f4baf7fe640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f4bb00c2ee0 0x7f4bb0071180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:03.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.102+0000 7f4baf7fe640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f4bb00c2ee0 0x7f4bb0071180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:55408/0 (socket says 172.21.15.116:55408) 2024-04-25T00:27:03.105 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.102+0000 7f4baf7fe640 1 -- 172.21.15.116:0/2186388207 learned_addr learned my addr 172.21.15.116:0/2186388207 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:27:03.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.106+0000 7f4baf7fe640 1 -- 172.21.15.116:0/2186388207 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4bb013acb0 msgr2=0x7f4bb00c29a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T00:27:03.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.106+0000 7f4baf7fe640 1 --2- 172.21.15.116:0/2186388207 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4bb013acb0 0x7f4bb00c29a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:03.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.106+0000 7f4baf7fe640 1 -- 172.21.15.116:0/2186388207 >> v1:172.21.15.116:6789/0 conn(0x7f4bb00c45b0 legacy=0x7f4bb013a2d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T00:27:03.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.106+0000 7f4baf7fe640 1 -- 172.21.15.116:0/2186388207 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bb01557e0 con 0x7f4bb00c2ee0 2024-04-25T00:27:03.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.110+0000 7f4baf7fe640 1 --2- 172.21.15.116:0/2186388207 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4bb00c2ee0 0x7f4bb0071180 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f4ba8009e50 tx=0x7f4ba801c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e4b2c7e697fc78da server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:03.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.114+0000 7f4badffb640 1 -- 172.21.15.116:0/2186388207 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba80ac020 con 0x7f4bb00c2ee0 2024-04-25T00:27:03.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.114+0000 7f4badffb640 1 -- 172.21.15.116:0/2186388207 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ba80a6480 con 0x7f4bb00c2ee0 2024-04-25T00:27:03.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.118+0000 7f4bb5769640 1 -- 172.21.15.116:0/2186388207 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4bb00c2ee0 msgr2=0x7f4bb0071180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:03.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.118+0000 7f4bb5769640 1 --2- 172.21.15.116:0/2186388207 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4bb00c2ee0 0x7f4bb0071180 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f4ba8009e50 tx=0x7f4ba801c8f0 comp rx=0 tx=0).stop 2024-04-25T00:27:03.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.118+0000 7f4bb5769640 1 -- 172.21.15.116:0/2186388207 shutdown_connections 2024-04-25T00:27:03.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.118+0000 7f4bb5769640 1 --2- 172.21.15.116:0/2186388207 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4bb00c2ee0 0x7f4bb0071180 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:03.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.118+0000 7f4bb5769640 1 --2- 172.21.15.116:0/2186388207 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4bb013acb0 0x7f4bb00c29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:03.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.118+0000 7f4bb5769640 1 -- 172.21.15.116:0/2186388207 >> 172.21.15.116:0/2186388207 conn(0x7f4bb00c3a90 msgr2=0x7f4bb00c3ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:27:03.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.126+0000 7f4bb5769640 1 -- 172.21.15.116:0/2186388207 shutdown_connections 2024-04-25T00:27:03.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.126+0000 7f4bb5769640 1 -- 172.21.15.116:0/2186388207 wait complete. 2024-04-25T00:27:03.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.126+0000 7f4bb5769640 1 Processor -- start 2024-04-25T00:27:03.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.126+0000 7f4bb5769640 1 -- start start 2024-04-25T00:27:03.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.130+0000 7f4bb5769640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4bb00c2ee0 0x7f4bb01358c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:03.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.130+0000 7f4bb5769640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4bb00c45b0 0x7f4bb0152dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:03.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.130+0000 7f4bb5769640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4bb013acb0 0x7f4bb0153300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:03.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.130+0000 7f4bb5769640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4bb01568a0 con 0x7f4bb00c45b0 2024-04-25T00:27:03.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.130+0000 7f4bb5769640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4bb015b1c0 con 0x7f4bb013acb0 2024-04-25T00:27:03.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.130+0000 7f4bb5769640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f4bb0156a20 con 0x7f4bb00c2ee0 2024-04-25T00:27:03.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.130+0000 7f4baf7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4bb013acb0 0x7f4bb0153300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:03.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.130+0000 7f4baf7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4bb013acb0 0x7f4bb0153300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:55416/0 (socket says 172.21.15.116:55416) 2024-04-25T00:27:03.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.130+0000 7f4baf7fe640 1 -- 172.21.15.116:0/177050064 learned_addr learned my addr 172.21.15.116:0/177050064 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:27:03.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.130+0000 7f4baf7fe640 1 -- 172.21.15.116:0/177050064 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4bb00c2ee0 msgr2=0x7f4bb01358c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T00:27:03.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.130+0000 7f4baf7fe640 1 --2- 172.21.15.116:0/177050064 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4bb00c2ee0 0x7f4bb01358c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:03.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.130+0000 7f4baf7fe640 1 -- 172.21.15.116:0/177050064 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4bb00c45b0 msgr2=0x7f4bb0152dc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T00:27:03.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.130+0000 7f4baf7fe640 1 --2- 172.21.15.116:0/177050064 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4bb00c45b0 0x7f4bb0152dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:03.132 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.130+0000 7f4baf7fe640 1 -- 172.21.15.116:0/177050064 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bb00c58a0 con 0x7f4bb013acb0 2024-04-25T00:27:03.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.138+0000 7f4baf7fe640 1 --2- 172.21.15.116:0/177050064 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4bb013acb0 0x7f4bb0153300 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f4ba801afa0 tx=0x7f4ba8019690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:03.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.138+0000 7f4b9f7fe640 1 -- 172.21.15.116:0/177050064 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba80ac040 con 0x7f4bb013acb0 2024-04-25T00:27:03.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.138+0000 7f4bb5769640 1 -- 172.21.15.116:0/177050064 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bb0138c50 con 0x7f4bb013acb0 2024-04-25T00:27:03.140 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.138+0000 7f4bb5769640 1 -- 172.21.15.116:0/177050064 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bb0139210 con 0x7f4bb013acb0 2024-04-25T00:27:03.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.138+0000 7f4b9f7fe640 1 -- 172.21.15.116:0/177050064 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ba80b6070 con 0x7f4bb013acb0 2024-04-25T00:27:03.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.142+0000 7f4b9f7fe640 1 -- 172.21.15.116:0/177050064 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba80a7af0 con 0x7f4bb013acb0 2024-04-25T00:27:03.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.142+0000 7f4b9f7fe640 1 -- 172.21.15.116:0/177050064 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4ba80af020 con 0x7f4bb013acb0 2024-04-25T00:27:03.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.150+0000 7f4bb5769640 1 -- 172.21.15.116:0/177050064 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bb004f580 con 0x7f4bb013acb0 2024-04-25T00:27:03.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.150+0000 7f4b9f7fe640 1 --2- 172.21.15.116:0/177050064 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4b940678c0 0x7f4b94069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:03.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.150+0000 7f4b9f7fe640 1 -- 172.21.15.116:0/177050064 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 10192+0+0 (secure 0 0 0) 0x7f4ba81216d0 con 0x7f4bb013acb0 2024-04-25T00:27:03.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.154+0000 7f4baeffd640 1 --2- 172.21.15.116:0/177050064 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4b940678c0 0x7f4b94069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:03.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.154+0000 7f4baeffd640 1 --2- 172.21.15.116:0/177050064 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4b940678c0 0x7f4b94069d80 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f4bb01528d0 tx=0x7f4ba0006be0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:03.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.162+0000 7f4b9f7fe640 1 -- 172.21.15.116:0/177050064 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f4ba8016600 con 0x7f4bb013acb0 2024-04-25T00:27:03.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.926+0000 7f4bb5769640 1 -- 172.21.15.116:0/177050064 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4bb0071080 con 0x7f4bb013acb0 2024-04-25T00:27:03.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.930+0000 7f4b9f7fe640 1 -- 172.21.15.116:0/177050064 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 217 v217) ==== 95+0+6506 (secure 0 0 0) 0x7f4ba80f1090 con 0x7f4bb013acb0 2024-04-25T00:27:03.931 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:27:03.931 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":217,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":8652,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/3891103804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3891103804},{"type":"v1","addr":"172.21.15.116:6833","nonce":3891103804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":217}],"filesystems":[{"mdsmap":{"epoch":211,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:26:19.034064+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6485},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6485":{"gid":6485,"name":"a","rank":0,"incarnation":208,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6835/2371126056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2371126056},{"type":"v1","addr":"172.21.15.116:6835","nonce":2371126056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6485,"qdb_cluster":[6485]},"id":1},{"mdsmap":{"epoch":216,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:26:49.039944+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7384},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7384":{"gid":7384,"name":"d","rank":0,"incarnation":213,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/349772087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":349772087},{"type":"v1","addr":"172.21.15.174:6837","nonce":349772087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7384,"qdb_cluster":[7384]},"id":2}]} 2024-04-25T00:27:03.931 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 217 2024-04-25T00:27:03.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.934+0000 7f4b9d7fa640 1 -- 172.21.15.116:0/177050064 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4b940678c0 msgr2=0x7f4b94069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:03.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.934+0000 7f4b9d7fa640 1 --2- 172.21.15.116:0/177050064 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4b940678c0 0x7f4b94069d80 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f4bb01528d0 tx=0x7f4ba0006be0 comp rx=0 tx=0).stop 2024-04-25T00:27:03.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.934+0000 7f4b9d7fa640 1 -- 172.21.15.116:0/177050064 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4bb013acb0 msgr2=0x7f4bb0153300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:03.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.934+0000 7f4b9d7fa640 1 --2- 172.21.15.116:0/177050064 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4bb013acb0 0x7f4bb0153300 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f4ba801afa0 tx=0x7f4ba8019690 comp rx=0 tx=0).stop 2024-04-25T00:27:03.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.934+0000 7f4b9d7fa640 1 -- 172.21.15.116:0/177050064 shutdown_connections 2024-04-25T00:27:03.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.934+0000 7f4b9d7fa640 1 --2- 172.21.15.116:0/177050064 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4b940678c0 0x7f4b94069d80 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:03.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.934+0000 7f4b9d7fa640 1 --2- 172.21.15.116:0/177050064 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4bb013acb0 0x7f4bb0153300 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:03.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.934+0000 7f4b9d7fa640 1 --2- 172.21.15.116:0/177050064 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4bb00c45b0 0x7f4bb0152dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:03.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.934+0000 7f4b9d7fa640 1 --2- 172.21.15.116:0/177050064 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4bb00c2ee0 0x7f4bb01358c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:03.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.934+0000 7f4b9d7fa640 1 -- 172.21.15.116:0/177050064 >> 172.21.15.116:0/177050064 conn(0x7f4bb00c3a90 msgr2=0x7f4bb014c640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:27:03.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.934+0000 7f4b9d7fa640 1 -- 172.21.15.116:0/177050064 shutdown_connections 2024-04-25T00:27:03.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:03.934+0000 7f4b9d7fa640 1 -- 172.21.15.116:0/177050064 wait complete. 2024-04-25T00:27:03.968 INFO:tasks.mds_thrash.fs.[b]:kill mds.d (rank=0) 2024-04-25T00:27:03.969 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-25T00:27:03.969 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:27:04.025 INFO:tasks.ceph.mds.d:Stopped 2024-04-25T00:27:04.026 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:27:04.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.194+0000 7fc156402640 1 Processor -- start 2024-04-25T00:27:04.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.194+0000 7fc156402640 1 -- start start 2024-04-25T00:27:04.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.194+0000 7fc156402640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fc15006d5f0 0x7fc150156c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:04.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.194+0000 7fc156402640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc150155810 0x7fc1501571c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:04.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.194+0000 7fc156402640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1500c9150 con 0x7fc150152d80 2024-04-25T00:27:04.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.194+0000 7fc156402640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fc1500c9d60 con 0x7fc15006d5f0 2024-04-25T00:27:04.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.194+0000 7fc156402640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fc1500c9430 con 0x7fc150155810 2024-04-25T00:27:04.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.194+0000 7fc14ffff640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fc150152d80 0x7fc15006bd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:34158/0 (socket says 172.21.15.116:34158) 2024-04-25T00:27:04.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.194+0000 7fc14ffff640 1 -- 172.21.15.116:0/2710940803 learned_addr learned my addr 172.21.15.116:0/2710940803 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:27:04.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.194+0000 7fc15499f640 1 --2- 172.21.15.116:0/2710940803 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc150155810 0x7fc1501571c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:04.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.194+0000 7fc147fff640 1 --2- 172.21.15.116:0/2710940803 >> v2:172.21.15.174:3300/0 conn(0x7fc15006d5f0 0x7fc150156c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:04.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.194+0000 7fc14f7fe640 1 -- 172.21.15.116:0/2710940803 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3784548417 0 0) 0x7fc1500c9150 con 0x7fc150152d80 2024-04-25T00:27:04.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.194+0000 7fc14f7fe640 1 -- 172.21.15.116:0/2710940803 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc13c003530 con 0x7fc150152d80 2024-04-25T00:27:04.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.194+0000 7fc15499f640 1 -- 172.21.15.116:0/2710940803 >> v2:172.21.15.174:3300/0 conn(0x7fc15006d5f0 msgr2=0x7fc150156c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:27:04.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.194+0000 7fc15499f640 1 --2- 172.21.15.116:0/2710940803 >> v2:172.21.15.174:3300/0 conn(0x7fc15006d5f0 0x7fc150156c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:04.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.194+0000 7fc15499f640 1 -- 172.21.15.116:0/2710940803 >> v1:172.21.15.174:6789/0 conn(0x7fc150152d80 legacy=0x7fc15006bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:04.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.194+0000 7fc15499f640 1 -- 172.21.15.116:0/2710940803 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1500c9710 con 0x7fc150155810 2024-04-25T00:27:04.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.198+0000 7fc15499f640 1 --2- 172.21.15.116:0/2710940803 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc150155810 0x7fc1501571c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc148009d90 tx=0x7fc14801c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc2af6050056223 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:04.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.198+0000 7fc14f7fe640 1 -- 172.21.15.116:0/2710940803 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1480b2070 con 0x7fc150155810 2024-04-25T00:27:04.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.198+0000 7fc14f7fe640 1 -- 172.21.15.116:0/2710940803 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1480a58a0 con 0x7fc150155810 2024-04-25T00:27:04.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.198+0000 7fc14f7fe640 1 -- 172.21.15.116:0/2710940803 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1480a5c70 con 0x7fc150155810 2024-04-25T00:27:04.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.198+0000 7fc156402640 1 -- 172.21.15.116:0/2710940803 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc150155810 msgr2=0x7fc1501571c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:04.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.198+0000 7fc156402640 1 --2- 172.21.15.116:0/2710940803 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc150155810 0x7fc1501571c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc148009d90 tx=0x7fc14801c8b0 comp rx=0 tx=0).stop 2024-04-25T00:27:04.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.198+0000 7fc156402640 1 -- 172.21.15.116:0/2710940803 shutdown_connections 2024-04-25T00:27:04.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.198+0000 7fc156402640 1 --2- 172.21.15.116:0/2710940803 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc150155810 0x7fc1501571c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:04.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.198+0000 7fc156402640 1 --2- 172.21.15.116:0/2710940803 >> v2:172.21.15.174:3300/0 conn(0x7fc15006d5f0 0x7fc150156c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:04.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.198+0000 7fc156402640 1 -- 172.21.15.116:0/2710940803 >> 172.21.15.116:0/2710940803 conn(0x7fc1500acdd0 msgr2=0x7fc1500ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:27:04.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.198+0000 7fc156402640 1 -- 172.21.15.116:0/2710940803 shutdown_connections 2024-04-25T00:27:04.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.198+0000 7fc156402640 1 -- 172.21.15.116:0/2710940803 wait complete. 2024-04-25T00:27:04.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.198+0000 7fc156402640 1 Processor -- start 2024-04-25T00:27:04.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.198+0000 7fc156402640 1 -- start start 2024-04-25T00:27:04.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.202+0000 7fc156402640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc15006d5f0 0x7fc1501546b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:04.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.202+0000 7fc156402640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc150152d80 0x7fc1501f59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:04.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.202+0000 7fc156402640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc150155810 0x7fc1501f5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:04.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.202+0000 7fc156402640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc1500caf00 con 0x7fc150152d80 2024-04-25T00:27:04.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.202+0000 7fc156402640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc1500cad80 con 0x7fc15006d5f0 2024-04-25T00:27:04.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.202+0000 7fc156402640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fc1501628d0 con 0x7fc150155810 2024-04-25T00:27:04.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.202+0000 7fc147fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc150152d80 0x7fc1501f59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:04.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.202+0000 7fc147fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc150152d80 0x7fc1501f59b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39962/0 (socket says 172.21.15.116:39962) 2024-04-25T00:27:04.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.202+0000 7fc147fff640 1 -- 172.21.15.116:0/3512555539 learned_addr learned my addr 172.21.15.116:0/3512555539 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:27:04.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.202+0000 7fc15499f640 1 --2- 172.21.15.116:0/3512555539 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc150155810 0x7fc1501f5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:04.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.202+0000 7fc14ffff640 1 --2- 172.21.15.116:0/3512555539 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc15006d5f0 0x7fc1501546b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:04.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.202+0000 7fc147fff640 1 -- 172.21.15.116:0/3512555539 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc150155810 msgr2=0x7fc1501f5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:04.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.202+0000 7fc147fff640 1 --2- 172.21.15.116:0/3512555539 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc150155810 0x7fc1501f5ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:04.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.202+0000 7fc147fff640 1 -- 172.21.15.116:0/3512555539 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc15006d5f0 msgr2=0x7fc1501546b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:04.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.202+0000 7fc147fff640 1 --2- 172.21.15.116:0/3512555539 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc15006d5f0 0x7fc1501546b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:04.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.202+0000 7fc147fff640 1 -- 172.21.15.116:0/3512555539 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1500adf10 con 0x7fc150152d80 2024-04-25T00:27:04.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.202+0000 7fc147fff640 1 --2- 172.21.15.116:0/3512555539 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc150152d80 0x7fc1501f59b0 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7fc13800e9e0 tx=0x7fc13800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:04.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.202+0000 7fc14d7fa640 1 -- 172.21.15.116:0/3512555539 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc138017070 con 0x7fc150152d80 2024-04-25T00:27:04.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.202+0000 7fc14d7fa640 1 -- 172.21.15.116:0/3512555539 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc13800c9d0 con 0x7fc150152d80 2024-04-25T00:27:04.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.202+0000 7fc156402640 1 -- 172.21.15.116:0/3512555539 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1500ae1f0 con 0x7fc150152d80 2024-04-25T00:27:04.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.202+0000 7fc14d7fa640 1 -- 172.21.15.116:0/3512555539 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc13800cda0 con 0x7fc150152d80 2024-04-25T00:27:04.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.202+0000 7fc156402640 1 -- 172.21.15.116:0/3512555539 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1500ae7b0 con 0x7fc150152d80 2024-04-25T00:27:04.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.206+0000 7fc156402640 1 -- 172.21.15.116:0/3512555539 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc150068350 con 0x7fc150152d80 2024-04-25T00:27:04.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.206+0000 7fc14d7fa640 1 -- 172.21.15.116:0/3512555539 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc13801a070 con 0x7fc150152d80 2024-04-25T00:27:04.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.206+0000 7fc14d7fa640 1 --2- 172.21.15.116:0/3512555539 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc130067910 0x7fc130069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:04.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.206+0000 7fc14d7fa640 1 -- 172.21.15.116:0/3512555539 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 10192+0+0 (secure 0 0 0) 0x7fc138087160 con 0x7fc150152d80 2024-04-25T00:27:04.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.206+0000 7fc14ffff640 1 --2- 172.21.15.116:0/3512555539 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc130067910 0x7fc130069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:04.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.206+0000 7fc14ffff640 1 --2- 172.21.15.116:0/3512555539 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc130067910 0x7fc130069dd0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fc1400011f0 tx=0x7fc140007d40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:04.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.210+0000 7fc14d7fa640 1 -- 172.21.15.116:0/3512555539 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fc138056500 con 0x7fc150152d80 2024-04-25T00:27:04.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.474+0000 7fc156402640 1 -- 172.21.15.116:0/3512555539 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc1500adc90 con 0x7fc150152d80 2024-04-25T00:27:04.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.474+0000 7fc14d7fa640 1 -- 172.21.15.116:0/3512555539 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 217 v217) ==== 95+0+6506 (secure 0 0 0) 0x7fc1500adc90 con 0x7fc150152d80 2024-04-25T00:27:04.476 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:27:04.476 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":217,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":8652,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/3891103804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3891103804},{"type":"v1","addr":"172.21.15.116:6833","nonce":3891103804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":217}],"filesystems":[{"mdsmap":{"epoch":211,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:26:19.034064+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6485},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6485":{"gid":6485,"name":"a","rank":0,"incarnation":208,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6835/2371126056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2371126056},{"type":"v1","addr":"172.21.15.116:6835","nonce":2371126056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6485,"qdb_cluster":[6485]},"id":1},{"mdsmap":{"epoch":216,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:26:49.039944+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7384},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7384":{"gid":7384,"name":"d","rank":0,"incarnation":213,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/349772087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":349772087},{"type":"v1","addr":"172.21.15.174:6837","nonce":349772087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7384,"qdb_cluster":[7384]},"id":2}]} 2024-04-25T00:27:04.476 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 217 2024-04-25T00:27:04.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.478+0000 7fc156402640 1 -- 172.21.15.116:0/3512555539 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc130067910 msgr2=0x7fc130069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:04.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.478+0000 7fc156402640 1 --2- 172.21.15.116:0/3512555539 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc130067910 0x7fc130069dd0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fc1400011f0 tx=0x7fc140007d40 comp rx=0 tx=0).stop 2024-04-25T00:27:04.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.478+0000 7fc156402640 1 -- 172.21.15.116:0/3512555539 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc150152d80 msgr2=0x7fc1501f59b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:04.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.478+0000 7fc156402640 1 --2- 172.21.15.116:0/3512555539 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc150152d80 0x7fc1501f59b0 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7fc13800e9e0 tx=0x7fc13800eea0 comp rx=0 tx=0).stop 2024-04-25T00:27:04.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.478+0000 7fc156402640 1 -- 172.21.15.116:0/3512555539 shutdown_connections 2024-04-25T00:27:04.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.478+0000 7fc156402640 1 --2- 172.21.15.116:0/3512555539 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc130067910 0x7fc130069dd0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:04.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.478+0000 7fc156402640 1 --2- 172.21.15.116:0/3512555539 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc150155810 0x7fc1501f5ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:04.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.478+0000 7fc156402640 1 --2- 172.21.15.116:0/3512555539 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc150152d80 0x7fc1501f59b0 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:04.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.478+0000 7fc156402640 1 --2- 172.21.15.116:0/3512555539 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc15006d5f0 0x7fc1501546b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:04.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.478+0000 7fc156402640 1 -- 172.21.15.116:0/3512555539 >> 172.21.15.116:0/3512555539 conn(0x7fc1500acdd0 msgr2=0x7fc15014e0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:27:04.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.478+0000 7fc156402640 1 -- 172.21.15.116:0/3512555539 shutdown_connections 2024-04-25T00:27:04.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:04.478+0000 7fc156402640 1 -- 172.21.15.116:0/3512555539 wait complete. 2024-04-25T00:27:04.500 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:27:06.502 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:27:06.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.646+0000 7f28d39eb640 1 Processor -- start 2024-04-25T00:27:06.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.646+0000 7f28d39eb640 1 -- start start 2024-04-25T00:27:06.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.646+0000 7f28d39eb640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28cc1599a0 0x7f28cc1562d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:06.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.646+0000 7f28d39eb640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f28cc0c5010 0x7f28cc0c54a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:06.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.646+0000 7f28d39eb640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28cc073200 con 0x7f28cc152e60 2024-04-25T00:27:06.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.646+0000 7f28d39eb640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f28cc153240 con 0x7f28cc0c5010 2024-04-25T00:27:06.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.646+0000 7f28d39eb640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f28cc0734e0 con 0x7f28cc1599a0 2024-04-25T00:27:06.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.646+0000 7f28d0f86640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28cc1599a0 0x7f28cc1562d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:06.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.646+0000 7f28d0f86640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28cc1599a0 0x7f28cc1562d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:58918/0 (socket says 172.21.15.116:58918) 2024-04-25T00:27:06.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.646+0000 7f28d0f86640 1 -- 172.21.15.116:0/3060719432 learned_addr learned my addr 172.21.15.116:0/3060719432 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:27:06.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.646+0000 7f28d1f88640 1 --2- 172.21.15.116:0/3060719432 >> v2:172.21.15.174:3300/0 conn(0x7f28cc0c5010 0x7f28cc0c54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:06.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.650+0000 7f28d0f86640 1 -- 172.21.15.116:0/3060719432 >> v2:172.21.15.174:3300/0 conn(0x7f28cc0c5010 msgr2=0x7f28cc0c54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:27:06.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.650+0000 7f28d0f86640 1 --2- 172.21.15.116:0/3060719432 >> v2:172.21.15.174:3300/0 conn(0x7f28cc0c5010 0x7f28cc0c54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:06.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.650+0000 7f28d0f86640 1 -- 172.21.15.116:0/3060719432 >> v1:172.21.15.174:6789/0 conn(0x7f28cc152e60 legacy=0x7f28cc0c36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:27:06.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.650+0000 7f28d0f86640 1 -- 172.21.15.116:0/3060719432 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28cc158ec0 con 0x7f28cc1599a0 2024-04-25T00:27:06.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.650+0000 7f28d0f86640 1 --2- 172.21.15.116:0/3060719432 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28cc1599a0 0x7f28cc1562d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f28bc009870 tx=0x7f28bc01c4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73a66c6f1799e3f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:06.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.654+0000 7f28c3fff640 1 -- 172.21.15.116:0/3060719432 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28bc0b2070 con 0x7f28cc1599a0 2024-04-25T00:27:06.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.654+0000 7f28c3fff640 1 -- 172.21.15.116:0/3060719432 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28bc0a9070 con 0x7f28cc1599a0 2024-04-25T00:27:06.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.654+0000 7f28c3fff640 1 -- 172.21.15.116:0/3060719432 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28bc0ad8c0 con 0x7f28cc1599a0 2024-04-25T00:27:06.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.654+0000 7f28d39eb640 1 -- 172.21.15.116:0/3060719432 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28cc1599a0 msgr2=0x7f28cc1562d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:06.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.654+0000 7f28d39eb640 1 --2- 172.21.15.116:0/3060719432 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28cc1599a0 0x7f28cc1562d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f28bc009870 tx=0x7f28bc01c4b0 comp rx=0 tx=0).stop 2024-04-25T00:27:06.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.658+0000 7f28d39eb640 1 -- 172.21.15.116:0/3060719432 shutdown_connections 2024-04-25T00:27:06.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.658+0000 7f28d39eb640 1 --2- 172.21.15.116:0/3060719432 >> v2:172.21.15.174:3300/0 conn(0x7f28cc0c5010 0x7f28cc0c54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:06.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.658+0000 7f28d39eb640 1 --2- 172.21.15.116:0/3060719432 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28cc1599a0 0x7f28cc1562d0 secure :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f28bc009870 tx=0x7f28bc01c4b0 comp rx=0 tx=0).stop 2024-04-25T00:27:06.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.658+0000 7f28d39eb640 1 -- 172.21.15.116:0/3060719432 >> 172.21.15.116:0/3060719432 conn(0x7f28cc06c720 msgr2=0x7f28cc06cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:27:06.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.658+0000 7f28d39eb640 1 -- 172.21.15.116:0/3060719432 shutdown_connections 2024-04-25T00:27:06.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.658+0000 7f28d39eb640 1 -- 172.21.15.116:0/3060719432 wait complete. 2024-04-25T00:27:06.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.662+0000 7f28d39eb640 1 Processor -- start 2024-04-25T00:27:06.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.662+0000 7f28d39eb640 1 -- start start 2024-04-25T00:27:06.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.662+0000 7f28d39eb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28cc0c5010 0x7f28cc1587e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:06.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.662+0000 7f28d39eb640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28cc152e60 0x7f28cc147e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:06.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.662+0000 7f28d39eb640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28cc1ed900 0x7f28cc150d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:06.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.662+0000 7f28d39eb640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f28cc072e20 con 0x7f28cc0c5010 2024-04-25T00:27:06.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.662+0000 7f28d39eb640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f28cc0c3df0 con 0x7f28cc1ed900 2024-04-25T00:27:06.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.662+0000 7f28d39eb640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f28cc072fa0 con 0x7f28cc152e60 2024-04-25T00:27:06.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.662+0000 7f28d1787640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28cc0c5010 0x7f28cc1587e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:06.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.662+0000 7f28d0f86640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28cc152e60 0x7f28cc147e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:06.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.662+0000 7f28d0f86640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28cc152e60 0x7f28cc147e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:58922/0 (socket says 172.21.15.116:58922) 2024-04-25T00:27:06.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.662+0000 7f28d0f86640 1 -- 172.21.15.116:0/495809448 learned_addr learned my addr 172.21.15.116:0/495809448 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:27:06.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.662+0000 7f28d1f88640 1 --2- 172.21.15.116:0/495809448 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28cc1ed900 0x7f28cc150d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:06.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.666+0000 7f28d1f88640 1 -- 172.21.15.116:0/495809448 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28cc152e60 msgr2=0x7f28cc147e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:06.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.666+0000 7f28d1f88640 1 --2- 172.21.15.116:0/495809448 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28cc152e60 0x7f28cc147e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:06.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.666+0000 7f28d1f88640 1 -- 172.21.15.116:0/495809448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28cc0c5010 msgr2=0x7f28cc1587e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:06.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.666+0000 7f28d1f88640 1 --2- 172.21.15.116:0/495809448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28cc0c5010 0x7f28cc1587e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:06.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.666+0000 7f28d1f88640 1 -- 172.21.15.116:0/495809448 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28cc145340 con 0x7f28cc1ed900 2024-04-25T00:27:06.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.666+0000 7f28d1787640 1 --2- 172.21.15.116:0/495809448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28cc0c5010 0x7f28cc1587e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:27:06.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.670+0000 7f28d1f88640 1 --2- 172.21.15.116:0/495809448 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28cc1ed900 0x7f28cc150d20 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f28c400d100 tx=0x7f28c400d5c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:06.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.674+0000 7f28c1ffb640 1 -- 172.21.15.116:0/495809448 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c401b580 con 0x7f28cc1ed900 2024-04-25T00:27:06.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.678+0000 7f28d39eb640 1 -- 172.21.15.116:0/495809448 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28cc153b40 con 0x7f28cc1ed900 2024-04-25T00:27:06.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.678+0000 7f28d39eb640 1 -- 172.21.15.116:0/495809448 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28cc153860 con 0x7f28cc1ed900 2024-04-25T00:27:06.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.678+0000 7f28c1ffb640 1 -- 172.21.15.116:0/495809448 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28c401b720 con 0x7f28cc1ed900 2024-04-25T00:27:06.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.678+0000 7f28c1ffb640 1 -- 172.21.15.116:0/495809448 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c401baf0 con 0x7f28cc1ed900 2024-04-25T00:27:06.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.678+0000 7f28d39eb640 1 -- 172.21.15.116:0/495809448 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28ac0052a0 con 0x7f28cc1ed900 2024-04-25T00:27:06.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.678+0000 7f28c1ffb640 1 -- 172.21.15.116:0/495809448 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f28c401bc90 con 0x7f28cc1ed900 2024-04-25T00:27:06.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.682+0000 7f28c1ffb640 1 --2- 172.21.15.116:0/495809448 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f28b8067ad0 0x7f28b8069f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:06.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.682+0000 7f28c1ffb640 1 -- 172.21.15.116:0/495809448 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 10192+0+0 (secure 0 0 0) 0x7f28c40876a0 con 0x7f28cc1ed900 2024-04-25T00:27:06.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.682+0000 7f28d1787640 1 --2- 172.21.15.116:0/495809448 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f28b8067ad0 0x7f28b8069f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:06.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.682+0000 7f28d1787640 1 --2- 172.21.15.116:0/495809448 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f28b8067ad0 0x7f28b8069f90 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f28cc0ad6c0 tx=0x7f28c8007e40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:06.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.690+0000 7f28c1ffb640 1 -- 172.21.15.116:0/495809448 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f28c4010070 con 0x7f28cc1ed900 2024-04-25T00:27:06.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.978+0000 7f28d39eb640 1 -- 172.21.15.116:0/495809448 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f28ac005590 con 0x7f28cc1ed900 2024-04-25T00:27:06.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.978+0000 7f28c1ffb640 1 -- 172.21.15.116:0/495809448 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 217 v217) ==== 95+0+6506 (secure 0 0 0) 0x7f28c4056a90 con 0x7f28cc1ed900 2024-04-25T00:27:06.982 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:27:06.982 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":217,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":8652,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/3891103804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3891103804},{"type":"v1","addr":"172.21.15.116:6833","nonce":3891103804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":217}],"filesystems":[{"mdsmap":{"epoch":211,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:26:19.034064+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6485},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6485":{"gid":6485,"name":"a","rank":0,"incarnation":208,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6835/2371126056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2371126056},{"type":"v1","addr":"172.21.15.116:6835","nonce":2371126056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6485,"qdb_cluster":[6485]},"id":1},{"mdsmap":{"epoch":216,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:26:49.039944+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7384},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7384":{"gid":7384,"name":"d","rank":0,"incarnation":213,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/349772087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":349772087},{"type":"v1","addr":"172.21.15.174:6837","nonce":349772087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7384,"qdb_cluster":[7384]},"id":2}]} 2024-04-25T00:27:06.982 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 217 2024-04-25T00:27:06.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.982+0000 7f28d39eb640 1 -- 172.21.15.116:0/495809448 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f28b8067ad0 msgr2=0x7f28b8069f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:06.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.982+0000 7f28d39eb640 1 --2- 172.21.15.116:0/495809448 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f28b8067ad0 0x7f28b8069f90 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f28cc0ad6c0 tx=0x7f28c8007e40 comp rx=0 tx=0).stop 2024-04-25T00:27:06.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.986+0000 7f28d39eb640 1 -- 172.21.15.116:0/495809448 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28cc1ed900 msgr2=0x7f28cc150d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:06.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.986+0000 7f28d39eb640 1 --2- 172.21.15.116:0/495809448 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28cc1ed900 0x7f28cc150d20 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f28c400d100 tx=0x7f28c400d5c0 comp rx=0 tx=0).stop 2024-04-25T00:27:06.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.986+0000 7f28d39eb640 1 -- 172.21.15.116:0/495809448 shutdown_connections 2024-04-25T00:27:06.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.986+0000 7f28d39eb640 1 --2- 172.21.15.116:0/495809448 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f28b8067ad0 0x7f28b8069f90 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:06.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.986+0000 7f28d39eb640 1 --2- 172.21.15.116:0/495809448 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28cc1ed900 0x7f28cc150d20 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:06.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.986+0000 7f28d39eb640 1 --2- 172.21.15.116:0/495809448 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28cc152e60 0x7f28cc147e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:06.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.986+0000 7f28d39eb640 1 --2- 172.21.15.116:0/495809448 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28cc0c5010 0x7f28cc1587e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:06.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.986+0000 7f28d39eb640 1 -- 172.21.15.116:0/495809448 >> 172.21.15.116:0/495809448 conn(0x7f28cc06c720 msgr2=0x7f28cc06cb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:27:06.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.986+0000 7f28d39eb640 1 -- 172.21.15.116:0/495809448 shutdown_connections 2024-04-25T00:27:06.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:06.986+0000 7f28d39eb640 1 -- 172.21.15.116:0/495809448 wait complete. 2024-04-25T00:27:07.007 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:27:08.808 INFO:tasks.workunit.client.0.smithi116.stdout:Syncing()...4 sec 2024-04-25T00:27:08.808 INFO:tasks.workunit.client.0.smithi116.stdout:FFSB benchmark finished at: Thu Apr 25 00:27:08 2024 2024-04-25T00:27:08.809 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-04-25T00:27:08.809 INFO:tasks.workunit.client.0.smithi116.stdout:Results: 2024-04-25T00:27:08.809 INFO:tasks.workunit.client.0.smithi116.stdout:Benchmark took 309.39 sec 2024-04-25T00:27:08.809 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-04-25T00:27:08.809 INFO:tasks.workunit.client.0.smithi116.stdout:Total Results 2024-04-25T00:27:08.809 INFO:tasks.workunit.client.0.smithi116.stdout:=============== 2024-04-25T00:27:08.809 INFO:tasks.workunit.client.0.smithi116.stdout: Op Name Transactions Trans/sec % Trans % Op Weight Throughput 2024-04-25T00:27:08.809 INFO:tasks.workunit.client.0.smithi116.stdout: ======= ============ ========= ======= =========== ========== 2024-04-25T00:27:08.809 INFO:tasks.workunit.client.0.smithi116.stdout: write : 1422080 4596.35 100.000% 100.000% 18MB/sec 2024-04-25T00:27:08.809 INFO:tasks.workunit.client.0.smithi116.stdout:- 2024-04-25T00:27:08.809 INFO:tasks.workunit.client.0.smithi116.stdout:4596.35 Transactions per Second 2024-04-25T00:27:08.809 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-04-25T00:27:08.809 INFO:tasks.workunit.client.0.smithi116.stdout:Throughput Results 2024-04-25T00:27:08.809 INFO:tasks.workunit.client.0.smithi116.stdout:=================== 2024-04-25T00:27:08.810 INFO:tasks.workunit.client.0.smithi116.stdout:Write Throughput: 18MB/sec 2024-04-25T00:27:08.816 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-04-25T00:27:08.816 INFO:tasks.workunit.client.0.smithi116.stdout:System Call Latency statistics in millisecs 2024-04-25T00:27:08.816 INFO:tasks.workunit.client.0.smithi116.stdout:===== 2024-04-25T00:27:08.816 INFO:tasks.workunit.client.0.smithi116.stdout: Min Avg Max Total Calls 2024-04-25T00:27:08.816 INFO:tasks.workunit.client.0.smithi116.stdout: ======== ======== ======== ============ 2024-04-25T00:27:08.816 INFO:tasks.workunit.client.0.smithi116.stdout:[ open] 0.294000 2215.346366 21563.943359 1111 2024-04-25T00:27:08.816 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[0] 5000.000000 - 10000.000000 : 14 2024-04-25T00:27:08.816 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-04-25T00:27:08.816 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-04-25T00:27:08.817 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-04-25T00:27:08.817 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-04-25T00:27:08.817 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[5] 100.000000 - 200.000000 : 3 2024-04-25T00:27:08.817 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[6] 50.000000 - 100.000000 : 25 2024-04-25T00:27:08.817 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[7] 20.000000 - 50.000000 : 265 2024-04-25T00:27:08.817 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[8] 10.000000 - 20.000000 : 404 2024-04-25T00:27:08.817 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[9] 5.000000 - 10.000000 : 151 2024-04-25T00:27:08.817 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[10] 2.000000 - 5.000000 : 48 2024-04-25T00:27:08.817 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[11] 1.000000 - 2.000000 : 34 2024-04-25T00:27:08.817 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[12] 0.500000 - 1.000000 : 16 2024-04-25T00:27:08.817 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[13] 0.200000 - 0.500000 : 5 2024-04-25T00:27:08.817 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[14] 0.100000 - 0.200000 : 0 2024-04-25T00:27:08.817 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[15] 0.050000 - 0.100000 : 0 2024-04-25T00:27:08.817 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[16] 0.020000 - 0.050000 : 0 2024-04-25T00:27:08.817 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[17] 0.010000 - 0.020000 : 0 2024-04-25T00:27:08.818 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[18] 0.000000 - 0.010000 : 0 2024-04-25T00:27:08.818 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-04-25T00:27:08.818 INFO:tasks.workunit.client.0.smithi116.stdout:[ write] 0.083000 5.074585 508.902008 1422080 2024-04-25T00:27:08.818 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-04-25T00:27:08.818 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-04-25T00:27:08.818 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-04-25T00:27:08.818 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[3] 500.000000 - 1000.000000 : 3 2024-04-25T00:27:08.818 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[4] 200.000000 - 500.000000 : 29 2024-04-25T00:27:08.818 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[5] 100.000000 - 200.000000 : 204 2024-04-25T00:27:08.818 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[6] 50.000000 - 100.000000 : 4985 2024-04-25T00:27:08.818 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[7] 20.000000 - 50.000000 : 59546 2024-04-25T00:27:08.818 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[8] 10.000000 - 20.000000 : 133431 2024-04-25T00:27:08.818 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[9] 5.000000 - 10.000000 : 252626 2024-04-25T00:27:08.819 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[10] 2.000000 - 5.000000 : 348994 2024-04-25T00:27:08.819 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[11] 1.000000 - 2.000000 : 150817 2024-04-25T00:27:08.819 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[12] 0.500000 - 1.000000 : 69127 2024-04-25T00:27:08.819 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[13] 0.200000 - 0.500000 : 202376 2024-04-25T00:27:08.819 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[14] 0.100000 - 0.200000 : 192582 2024-04-25T00:27:08.819 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[15] 0.050000 - 0.100000 : 7360 2024-04-25T00:27:08.819 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[16] 0.020000 - 0.050000 : 0 2024-04-25T00:27:08.819 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[17] 0.010000 - 0.020000 : 0 2024-04-25T00:27:08.819 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[18] 0.000000 - 0.010000 : 0 2024-04-25T00:27:08.819 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-04-25T00:27:08.819 INFO:tasks.workunit.client.0.smithi116.stdout:[ lseek] 0.000000 0.000975 7.420000 1422080 2024-04-25T00:27:08.820 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-04-25T00:27:08.820 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-04-25T00:27:08.820 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-04-25T00:27:08.820 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-04-25T00:27:08.820 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-04-25T00:27:08.820 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2024-04-25T00:27:08.820 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[6] 50.000000 - 100.000000 : 0 2024-04-25T00:27:08.820 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[7] 20.000000 - 50.000000 : 0 2024-04-25T00:27:08.820 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[8] 10.000000 - 20.000000 : 0 2024-04-25T00:27:08.820 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[9] 5.000000 - 10.000000 : 1 2024-04-25T00:27:08.820 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[10] 2.000000 - 5.000000 : 2 2024-04-25T00:27:08.820 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[11] 1.000000 - 2.000000 : 7 2024-04-25T00:27:08.820 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[12] 0.500000 - 1.000000 : 9 2024-04-25T00:27:08.820 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[13] 0.200000 - 0.500000 : 23 2024-04-25T00:27:08.821 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[14] 0.100000 - 0.200000 : 32 2024-04-25T00:27:08.821 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[15] 0.050000 - 0.100000 : 58 2024-04-25T00:27:08.821 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[16] 0.020000 - 0.050000 : 314 2024-04-25T00:27:08.821 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[17] 0.010000 - 0.020000 : 753 2024-04-25T00:27:08.821 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[18] 0.000000 - 0.010000 : 1420881 2024-04-25T00:27:08.821 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-04-25T00:27:08.821 INFO:tasks.workunit.client.0.smithi116.stdout:[ close] 0.018000 2.234867 61.318001 1111 2024-04-25T00:27:08.821 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-04-25T00:27:08.821 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-04-25T00:27:08.821 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-04-25T00:27:08.821 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-04-25T00:27:08.821 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-04-25T00:27:08.821 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2024-04-25T00:27:08.821 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[6] 50.000000 - 100.000000 : 2 2024-04-25T00:27:08.821 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[7] 20.000000 - 50.000000 : 24 2024-04-25T00:27:08.822 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[8] 10.000000 - 20.000000 : 34 2024-04-25T00:27:08.822 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[9] 5.000000 - 10.000000 : 71 2024-04-25T00:27:08.822 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[10] 2.000000 - 5.000000 : 144 2024-04-25T00:27:08.822 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[11] 1.000000 - 2.000000 : 111 2024-04-25T00:27:08.822 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[12] 0.500000 - 1.000000 : 81 2024-04-25T00:27:08.822 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[13] 0.200000 - 0.500000 : 67 2024-04-25T00:27:08.822 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[14] 0.100000 - 0.200000 : 109 2024-04-25T00:27:08.822 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[15] 0.050000 - 0.100000 : 127 2024-04-25T00:27:08.822 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[16] 0.020000 - 0.050000 : 340 2024-04-25T00:27:08.822 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[17] 0.010000 - 0.020000 : 1 2024-04-25T00:27:08.822 INFO:tasks.workunit.client.0.smithi116.stdout: msec_range[18] 0.000000 - 0.010000 : 0 2024-04-25T00:27:08.822 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-04-25T00:27:08.822 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-04-25T00:27:08.822 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-04-25T00:27:08.823 INFO:tasks.workunit.client.0.smithi116.stdout: 2024-04-25T00:27:08.823 INFO:tasks.workunit.client.0.smithi116.stdout:0.5% User Time 2024-04-25T00:27:08.823 INFO:tasks.workunit.client.0.smithi116.stdout:45.5% System Time 2024-04-25T00:27:08.823 INFO:tasks.workunit.client.0.smithi116.stdout:46.0% CPU Utilization 2024-04-25T00:27:08.823 INFO:tasks.workunit.client.0.smithi116.stderr:+ cd .. 2024-04-25T00:27:08.826 INFO:tasks.workunit.client.0.smithi116.stderr:+ rm -r tmp ffsb 2024-04-25T00:27:09.008 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:27:09.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.158+0000 7f6fe5576640 1 Processor -- start 2024-04-25T00:27:09.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.158+0000 7f6fe5576640 1 -- start start 2024-04-25T00:27:09.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.162+0000 7f6fe5576640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6fe00c19d0 0x7f6fe00c0c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:09.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.162+0000 7f6fe5576640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6fe00c56e0 0x7f6fe00a6d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:09.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.162+0000 7f6fe5576640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fe00c1150 con 0x7f6fe00c3cd0 2024-04-25T00:27:09.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.162+0000 7f6fe5576640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f6fe00c21e0 con 0x7f6fe00c56e0 2024-04-25T00:27:09.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.162+0000 7f6fe5576640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6fe00c5dc0 con 0x7f6fe00c19d0 2024-04-25T00:27:09.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.162+0000 7f6fde7fc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6fe00c56e0 0x7f6fe00a6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:09.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.162+0000 7f6fde7fc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6fe00c56e0 0x7f6fe00a6d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35042/0 (socket says 172.21.15.116:35042) 2024-04-25T00:27:09.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.162+0000 7f6fde7fc640 1 -- 172.21.15.116:0/3489455701 learned_addr learned my addr 172.21.15.116:0/3489455701 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:27:09.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.162+0000 7f6fdeffd640 1 --2- 172.21.15.116:0/3489455701 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6fe00c19d0 0x7f6fe00c0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:09.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.162+0000 7f6fde7fc640 1 -- 172.21.15.116:0/3489455701 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6fe00c19d0 msgr2=0x7f6fe00c0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:27:09.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.162+0000 7f6fde7fc640 1 --2- 172.21.15.116:0/3489455701 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6fe00c19d0 0x7f6fe00c0c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:09.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.162+0000 7f6fde7fc640 1 -- 172.21.15.116:0/3489455701 >> v1:172.21.15.174:6789/0 conn(0x7f6fe00c3cd0 legacy=0x7f6fe006c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:27:09.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.162+0000 7f6fde7fc640 1 -- 172.21.15.116:0/3489455701 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fe0074770 con 0x7f6fe00c56e0 2024-04-25T00:27:09.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.166+0000 7f6fde7fc640 1 --2- 172.21.15.116:0/3489455701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fe00c56e0 0x7f6fe00a6d10 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f6fd4009870 tx=0x7f6fd401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c989c1b9c01bf8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:09.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.166+0000 7f6fddffb640 1 -- 172.21.15.116:0/3489455701 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fd40b0070 con 0x7f6fe00c56e0 2024-04-25T00:27:09.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.166+0000 7f6fddffb640 1 -- 172.21.15.116:0/3489455701 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fd40ab600 con 0x7f6fe00c56e0 2024-04-25T00:27:09.165 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.166+0000 7f6fddffb640 1 -- 172.21.15.116:0/3489455701 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fd40ab9d0 con 0x7f6fe00c56e0 2024-04-25T00:27:09.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.166+0000 7f6fe5576640 1 -- 172.21.15.116:0/3489455701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fe00c56e0 msgr2=0x7f6fe00a6d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:09.166 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.166+0000 7f6fe5576640 1 --2- 172.21.15.116:0/3489455701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fe00c56e0 0x7f6fe00a6d10 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f6fd4009870 tx=0x7f6fd401c4d0 comp rx=0 tx=0).stop 2024-04-25T00:27:09.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.166+0000 7f6fe5576640 1 -- 172.21.15.116:0/3489455701 shutdown_connections 2024-04-25T00:27:09.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.166+0000 7f6fe5576640 1 --2- 172.21.15.116:0/3489455701 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fe00c56e0 0x7f6fe00a6d10 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:09.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.166+0000 7f6fe5576640 1 --2- 172.21.15.116:0/3489455701 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6fe00c19d0 0x7f6fe00c0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:09.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.166+0000 7f6fe5576640 1 -- 172.21.15.116:0/3489455701 >> 172.21.15.116:0/3489455701 conn(0x7f6fe00c42e0 msgr2=0x7f6fe00c46f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:27:09.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.166+0000 7f6fe5576640 1 -- 172.21.15.116:0/3489455701 shutdown_connections 2024-04-25T00:27:09.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.166+0000 7f6fe5576640 1 -- 172.21.15.116:0/3489455701 wait complete. 2024-04-25T00:27:09.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.166+0000 7f6fe5576640 1 Processor -- start 2024-04-25T00:27:09.167 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.166+0000 7f6fe5576640 1 -- start start 2024-04-25T00:27:09.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.166+0000 7f6fe5576640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6fe00c19d0 0x7f6fe0155e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:09.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.166+0000 7f6fe5576640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fe00c3cd0 0x7f6fe0156380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:09.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.166+0000 7f6fe5576640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6fe00c56e0 0x7f6fe01f1860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:09.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.166+0000 7f6fe5576640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6fe006c930 con 0x7f6fe00c3cd0 2024-04-25T00:27:09.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.166+0000 7f6fe5576640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6fe00a7250 con 0x7f6fe00c19d0 2024-04-25T00:27:09.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.166+0000 7f6fe5576640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6fe006cab0 con 0x7f6fe00c56e0 2024-04-25T00:27:09.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.166+0000 7f6fde7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fe00c3cd0 0x7f6fe0156380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:09.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.166+0000 7f6fde7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fe00c3cd0 0x7f6fe0156380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35058/0 (socket says 172.21.15.116:35058) 2024-04-25T00:27:09.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.166+0000 7f6fde7fc640 1 -- 172.21.15.116:0/4091645039 learned_addr learned my addr 172.21.15.116:0/4091645039 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:27:09.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.166+0000 7f6fdeffd640 1 --2- 172.21.15.116:0/4091645039 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6fe00c19d0 0x7f6fe0155e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:09.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.166+0000 7f6fdf7fe640 1 --2- 172.21.15.116:0/4091645039 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6fe00c56e0 0x7f6fe01f1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:09.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.166+0000 7f6fde7fc640 1 -- 172.21.15.116:0/4091645039 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6fe00c56e0 msgr2=0x7f6fe01f1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:09.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.170+0000 7f6fde7fc640 1 --2- 172.21.15.116:0/4091645039 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6fe00c56e0 0x7f6fe01f1860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:09.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.170+0000 7f6fde7fc640 1 -- 172.21.15.116:0/4091645039 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6fe00c19d0 msgr2=0x7f6fe0155e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:09.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.170+0000 7f6fde7fc640 1 --2- 172.21.15.116:0/4091645039 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6fe00c19d0 0x7f6fe0155e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:09.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.170+0000 7f6fde7fc640 1 -- 172.21.15.116:0/4091645039 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fe01536e0 con 0x7f6fe00c3cd0 2024-04-25T00:27:09.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.170+0000 7f6fde7fc640 1 --2- 172.21.15.116:0/4091645039 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fe00c3cd0 0x7f6fe0156380 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f6fd401ab80 tx=0x7f6fd4019270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:09.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.170+0000 7f6fc77fe640 1 -- 172.21.15.116:0/4091645039 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fd40b0040 con 0x7f6fe00c3cd0 2024-04-25T00:27:09.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.170+0000 7f6fc77fe640 1 -- 172.21.15.116:0/4091645039 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fd40b9070 con 0x7f6fe00c3cd0 2024-04-25T00:27:09.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.170+0000 7f6fe5576640 1 -- 172.21.15.116:0/4091645039 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fe01ef0c0 con 0x7f6fe00c3cd0 2024-04-25T00:27:09.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.170+0000 7f6fc77fe640 1 -- 172.21.15.116:0/4091645039 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fd40b4880 con 0x7f6fe00c3cd0 2024-04-25T00:27:09.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.170+0000 7f6fe5576640 1 -- 172.21.15.116:0/4091645039 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fe00acb40 con 0x7f6fe00c3cd0 2024-04-25T00:27:09.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.170+0000 7f6fe5576640 1 -- 172.21.15.116:0/4091645039 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fe00acd80 con 0x7f6fe00c3cd0 2024-04-25T00:27:09.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.170+0000 7f6fc77fe640 1 -- 172.21.15.116:0/4091645039 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6fd40b4a20 con 0x7f6fe00c3cd0 2024-04-25T00:27:09.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.170+0000 7f6fc77fe640 1 --2- 172.21.15.116:0/4091645039 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6fcc0678c0 0x7f6fcc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:09.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.170+0000 7f6fc77fe640 1 -- 172.21.15.116:0/4091645039 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 10192+0+0 (secure 0 0 0) 0x7f6fd41215e0 con 0x7f6fe00c3cd0 2024-04-25T00:27:09.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.170+0000 7f6fdeffd640 1 --2- 172.21.15.116:0/4091645039 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6fcc0678c0 0x7f6fcc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:09.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.174+0000 7f6fdeffd640 1 --2- 172.21.15.116:0/4091645039 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6fcc0678c0 0x7f6fcc069d80 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f6fd0003ab0 tx=0x7f6fd0009830 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:09.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.178+0000 7f6fc77fe640 1 -- 172.21.15.116:0/4091645039 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f6fd4016150 con 0x7f6fe00c3cd0 2024-04-25T00:27:09.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.430+0000 7f6fe5576640 1 -- 172.21.15.116:0/4091645039 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6fe00ac650 con 0x7f6fe00c3cd0 2024-04-25T00:27:09.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.430+0000 7f6fc77fe640 1 -- 172.21.15.116:0/4091645039 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 217 v217) ==== 95+0+6506 (secure 0 0 0) 0x7f6fd40f1180 con 0x7f6fe00c3cd0 2024-04-25T00:27:09.432 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:27:09.432 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":217,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":8652,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/3891103804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3891103804},{"type":"v1","addr":"172.21.15.116:6833","nonce":3891103804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":217}],"filesystems":[{"mdsmap":{"epoch":211,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:26:19.034064+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6485},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6485":{"gid":6485,"name":"a","rank":0,"incarnation":208,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6835/2371126056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2371126056},{"type":"v1","addr":"172.21.15.116:6835","nonce":2371126056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6485,"qdb_cluster":[6485]},"id":1},{"mdsmap":{"epoch":216,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:26:49.039944+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7384},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7384":{"gid":7384,"name":"d","rank":0,"incarnation":213,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/349772087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":349772087},{"type":"v1","addr":"172.21.15.174:6837","nonce":349772087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7384,"qdb_cluster":[7384]},"id":2}]} 2024-04-25T00:27:09.433 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 217 2024-04-25T00:27:09.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.434+0000 7f6fe5576640 1 -- 172.21.15.116:0/4091645039 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6fcc0678c0 msgr2=0x7f6fcc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:09.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.434+0000 7f6fe5576640 1 --2- 172.21.15.116:0/4091645039 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6fcc0678c0 0x7f6fcc069d80 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f6fd0003ab0 tx=0x7f6fd0009830 comp rx=0 tx=0).stop 2024-04-25T00:27:09.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.434+0000 7f6fe5576640 1 -- 172.21.15.116:0/4091645039 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fe00c3cd0 msgr2=0x7f6fe0156380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:09.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.434+0000 7f6fe5576640 1 --2- 172.21.15.116:0/4091645039 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fe00c3cd0 0x7f6fe0156380 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f6fd401ab80 tx=0x7f6fd4019270 comp rx=0 tx=0).stop 2024-04-25T00:27:09.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.434+0000 7f6fe5576640 1 -- 172.21.15.116:0/4091645039 shutdown_connections 2024-04-25T00:27:09.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.434+0000 7f6fe5576640 1 --2- 172.21.15.116:0/4091645039 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6fcc0678c0 0x7f6fcc069d80 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:09.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.434+0000 7f6fe5576640 1 --2- 172.21.15.116:0/4091645039 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6fe00c56e0 0x7f6fe01f1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:09.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.434+0000 7f6fe5576640 1 --2- 172.21.15.116:0/4091645039 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6fe00c3cd0 0x7f6fe0156380 unknown :-1 s=CLOSED pgs=1601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:09.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.434+0000 7f6fe5576640 1 --2- 172.21.15.116:0/4091645039 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6fe00c19d0 0x7f6fe0155e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:09.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.434+0000 7f6fe5576640 1 -- 172.21.15.116:0/4091645039 >> 172.21.15.116:0/4091645039 conn(0x7f6fe00c42e0 msgr2=0x7f6fe00c6f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:27:09.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.434+0000 7f6fe5576640 1 -- 172.21.15.116:0/4091645039 shutdown_connections 2024-04-25T00:27:09.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:09.434+0000 7f6fe5576640 1 -- 172.21.15.116:0/4091645039 wait complete. 2024-04-25T00:27:09.471 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:27:11.473 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:27:11.533 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:27:11.536 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:27:11.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.622+0000 7f787429d640 1 Processor -- start 2024-04-25T00:27:11.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.622+0000 7f787429d640 1 -- start start 2024-04-25T00:27:11.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.626+0000 7f787429d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f786c0c9040 0x7f786c0c9420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:11.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.626+0000 7f787429d640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f786c0c6e20 0x7f786c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:11.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.626+0000 7f787429d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f786c159b60 con 0x7f786c0c3310 2024-04-25T00:27:11.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.626+0000 7f787429d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f786c0c7860 con 0x7f786c0c9040 2024-04-25T00:27:11.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.626+0000 7f787429d640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f786c0c3920 con 0x7f786c0c6e20 2024-04-25T00:27:11.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.626+0000 7f7872039640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f786c0c9040 0x7f786c0c9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:11.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.626+0000 7f7872039640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f786c0c9040 0x7f786c0c9420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35068/0 (socket says 172.21.15.116:35068) 2024-04-25T00:27:11.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.626+0000 7f7872039640 1 -- 172.21.15.116:0/1769873372 learned_addr learned my addr 172.21.15.116:0/1769873372 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:27:11.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.626+0000 7f787283a640 1 --1- 172.21.15.116:0/1769873372 >> v1:172.21.15.116:6789/0 conn(0x7f786c0c3310 0x7f786c14bcd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:43466/0 (socket says 172.21.15.116:43466) 2024-04-25T00:27:11.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.626+0000 7f7871838640 1 --2- 172.21.15.116:0/1769873372 >> v2:172.21.15.174:3300/0 conn(0x7f786c0c6e20 0x7f786c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:11.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.626+0000 7f7872039640 1 -- 172.21.15.116:0/1769873372 >> v2:172.21.15.174:3300/0 conn(0x7f786c0c6e20 msgr2=0x7f786c06d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:27:11.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.626+0000 7f7872039640 1 --2- 172.21.15.116:0/1769873372 >> v2:172.21.15.174:3300/0 conn(0x7f786c0c6e20 0x7f786c06d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:11.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.626+0000 7f7872039640 1 -- 172.21.15.116:0/1769873372 >> v1:172.21.15.116:6789/0 conn(0x7f786c0c3310 legacy=0x7f786c14bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:11.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.626+0000 7f7872039640 1 -- 172.21.15.116:0/1769873372 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f786c0c3b70 con 0x7f786c0c9040 2024-04-25T00:27:11.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.626+0000 7f7871037640 1 -- 172.21.15.116:0/1769873372 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4023533489 0 0) 0x7f786c159b60 con 0x7f786c0c3310 2024-04-25T00:27:11.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.630+0000 7f7872039640 1 --2- 172.21.15.116:0/1769873372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f786c0c9040 0x7f786c0c9420 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f786c14fe10 tx=0x7f786801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9306706c4b666738 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:11.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.630+0000 7f7871037640 1 -- 172.21.15.116:0/1769873372 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78680b0070 con 0x7f786c0c9040 2024-04-25T00:27:11.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.630+0000 7f7871037640 1 -- 172.21.15.116:0/1769873372 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78680ab600 con 0x7f786c0c9040 2024-04-25T00:27:11.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.630+0000 7f7871037640 1 -- 172.21.15.116:0/1769873372 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78680ab9d0 con 0x7f786c0c9040 2024-04-25T00:27:11.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.630+0000 7f787429d640 1 -- 172.21.15.116:0/1769873372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f786c0c9040 msgr2=0x7f786c0c9420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:11.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.630+0000 7f787429d640 1 --2- 172.21.15.116:0/1769873372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f786c0c9040 0x7f786c0c9420 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f786c14fe10 tx=0x7f786801c490 comp rx=0 tx=0).stop 2024-04-25T00:27:11.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.630+0000 7f787429d640 1 -- 172.21.15.116:0/1769873372 shutdown_connections 2024-04-25T00:27:11.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.630+0000 7f787429d640 1 --2- 172.21.15.116:0/1769873372 >> v2:172.21.15.174:3300/0 conn(0x7f786c0c6e20 0x7f786c06d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:11.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.630+0000 7f787429d640 1 --2- 172.21.15.116:0/1769873372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f786c0c9040 0x7f786c0c9420 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:11.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.630+0000 7f787429d640 1 -- 172.21.15.116:0/1769873372 >> 172.21.15.116:0/1769873372 conn(0x7f786c074480 msgr2=0x7f786c074890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:27:11.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.630+0000 7f787429d640 1 -- 172.21.15.116:0/1769873372 shutdown_connections 2024-04-25T00:27:11.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.630+0000 7f787429d640 1 -- 172.21.15.116:0/1769873372 wait complete. 2024-04-25T00:27:11.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.630+0000 7f787429d640 1 Processor -- start 2024-04-25T00:27:11.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.630+0000 7f787429d640 1 -- start start 2024-04-25T00:27:11.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.630+0000 7f787429d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f786c0c3310 0x7f786c1f2960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:11.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.630+0000 7f787429d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f786c0c6e20 0x7f786c158640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:11.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.634+0000 7f787429d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f786c0c9040 0x7f786c1ee870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:11.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.634+0000 7f787429d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f786c1565a0 con 0x7f786c0c6e20 2024-04-25T00:27:11.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.634+0000 7f787429d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f786c156420 con 0x7f786c0c3310 2024-04-25T00:27:11.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.634+0000 7f787429d640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f786c156720 con 0x7f786c0c9040 2024-04-25T00:27:11.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.634+0000 7f7871838640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f786c0c6e20 0x7f786c158640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:11.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.634+0000 7f7871838640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f786c0c6e20 0x7f786c158640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35082/0 (socket says 172.21.15.116:35082) 2024-04-25T00:27:11.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.634+0000 7f7871838640 1 -- 172.21.15.116:0/2491918698 learned_addr learned my addr 172.21.15.116:0/2491918698 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:27:11.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.634+0000 7f7872039640 1 --2- 172.21.15.116:0/2491918698 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f786c0c3310 0x7f786c1f2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:11.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.634+0000 7f787283a640 1 --2- 172.21.15.116:0/2491918698 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f786c0c9040 0x7f786c1ee870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:11.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.634+0000 7f7871838640 1 -- 172.21.15.116:0/2491918698 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f786c0c9040 msgr2=0x7f786c1ee870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:11.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.634+0000 7f7871838640 1 --2- 172.21.15.116:0/2491918698 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f786c0c9040 0x7f786c1ee870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:11.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.634+0000 7f7871838640 1 -- 172.21.15.116:0/2491918698 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f786c0c3310 msgr2=0x7f786c1f2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:11.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.634+0000 7f7871838640 1 --2- 172.21.15.116:0/2491918698 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f786c0c3310 0x7f786c1f2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:11.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.634+0000 7f7871838640 1 -- 172.21.15.116:0/2491918698 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f786c146100 con 0x7f786c0c6e20 2024-04-25T00:27:11.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.634+0000 7f7871838640 1 --2- 172.21.15.116:0/2491918698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f786c0c6e20 0x7f786c158640 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f785c00e930 tx=0x7f785c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:11.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.634+0000 7f7862ffd640 1 -- 172.21.15.116:0/2491918698 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f785c017070 con 0x7f786c0c6e20 2024-04-25T00:27:11.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.634+0000 7f7862ffd640 1 -- 172.21.15.116:0/2491918698 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f785c00ca00 con 0x7f786c0c6e20 2024-04-25T00:27:11.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.634+0000 7f787429d640 1 -- 172.21.15.116:0/2491918698 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f786c145e20 con 0x7f786c0c6e20 2024-04-25T00:27:11.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.634+0000 7f7862ffd640 1 -- 172.21.15.116:0/2491918698 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f785c00cdb0 con 0x7f786c0c6e20 2024-04-25T00:27:11.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.634+0000 7f787429d640 1 -- 172.21.15.116:0/2491918698 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f786c1f2160 con 0x7f786c0c6e20 2024-04-25T00:27:11.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.634+0000 7f787429d640 1 -- 172.21.15.116:0/2491918698 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f786c068a60 con 0x7f786c0c6e20 2024-04-25T00:27:11.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.634+0000 7f7862ffd640 1 -- 172.21.15.116:0/2491918698 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f785c01a070 con 0x7f786c0c6e20 2024-04-25T00:27:11.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.634+0000 7f7862ffd640 1 --2- 172.21.15.116:0/2491918698 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f78500678c0 0x7f7850069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:11.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.634+0000 7f7862ffd640 1 -- 172.21.15.116:0/2491918698 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 10192+0+0 (secure 0 0 0) 0x7f785c086df0 con 0x7f786c0c6e20 2024-04-25T00:27:11.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.634+0000 7f7872039640 1 --2- 172.21.15.116:0/2491918698 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f78500678c0 0x7f7850069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:11.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.638+0000 7f7872039640 1 --2- 172.21.15.116:0/2491918698 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f78500678c0 0x7f7850069d80 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f78680ad000 tx=0x7f78680a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:11.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.642+0000 7f7862ffd640 1 -- 172.21.15.116:0/2491918698 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f785c0561d0 con 0x7f786c0c6e20 2024-04-25T00:27:11.747 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T00:27:11.747 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T00:27:11.764 INFO:tasks.workunit:Stopping ['suites/ffsb.sh'] on client.0... 2024-04-25T00:27:11.765 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-25T00:27:11.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.894+0000 7f787429d640 1 -- 172.21.15.116:0/2491918698 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f786c0c30f0 con 0x7f786c0c6e20 2024-04-25T00:27:11.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.894+0000 7f7862ffd640 1 -- 172.21.15.116:0/2491918698 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 217 v217) ==== 95+0+6506 (secure 0 0 0) 0x7f785c05a4d0 con 0x7f786c0c6e20 2024-04-25T00:27:11.895 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:27:11.895 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":217,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":8652,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/3891103804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3891103804},{"type":"v1","addr":"172.21.15.116:6833","nonce":3891103804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":217}],"filesystems":[{"mdsmap":{"epoch":211,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:26:19.034064+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6485},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6485":{"gid":6485,"name":"a","rank":0,"incarnation":208,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6835/2371126056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2371126056},{"type":"v1","addr":"172.21.15.116:6835","nonce":2371126056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6485,"qdb_cluster":[6485]},"id":1},{"mdsmap":{"epoch":216,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:26:49.039944+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7384},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7384":{"gid":7384,"name":"d","rank":0,"incarnation":213,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/349772087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":349772087},{"type":"v1","addr":"172.21.15.174:6837","nonce":349772087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7384,"qdb_cluster":[7384]},"id":2}]} 2024-04-25T00:27:11.896 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 217 2024-04-25T00:27:11.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.894+0000 7f787429d640 1 -- 172.21.15.116:0/2491918698 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f78500678c0 msgr2=0x7f7850069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:11.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.894+0000 7f787429d640 1 --2- 172.21.15.116:0/2491918698 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f78500678c0 0x7f7850069d80 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f78680ad000 tx=0x7f78680a6000 comp rx=0 tx=0).stop 2024-04-25T00:27:11.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.894+0000 7f787429d640 1 -- 172.21.15.116:0/2491918698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f786c0c6e20 msgr2=0x7f786c158640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:11.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.894+0000 7f787429d640 1 --2- 172.21.15.116:0/2491918698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f786c0c6e20 0x7f786c158640 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f785c00e930 tx=0x7f785c00edf0 comp rx=0 tx=0).stop 2024-04-25T00:27:11.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.894+0000 7f787429d640 1 -- 172.21.15.116:0/2491918698 shutdown_connections 2024-04-25T00:27:11.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.894+0000 7f787429d640 1 --2- 172.21.15.116:0/2491918698 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f78500678c0 0x7f7850069d80 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:11.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.894+0000 7f787429d640 1 --2- 172.21.15.116:0/2491918698 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f786c0c9040 0x7f786c1ee870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:11.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.894+0000 7f787429d640 1 --2- 172.21.15.116:0/2491918698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f786c0c6e20 0x7f786c158640 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:11.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.894+0000 7f787429d640 1 --2- 172.21.15.116:0/2491918698 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f786c0c3310 0x7f786c1f2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:11.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.894+0000 7f787429d640 1 -- 172.21.15.116:0/2491918698 >> 172.21.15.116:0/2491918698 conn(0x7f786c074480 msgr2=0x7f786c160680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:27:11.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.894+0000 7f787429d640 1 -- 172.21.15.116:0/2491918698 shutdown_connections 2024-04-25T00:27:11.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:11.894+0000 7f787429d640 1 -- 172.21.15.116:0/2491918698 wait complete. 2024-04-25T00:27:11.935 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:27:12.277 DEBUG:teuthology.parallel:result is None 2024-04-25T00:27:13.936 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:27:14.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.214+0000 7f8ae9e8f640 1 Processor -- start 2024-04-25T00:27:14.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.214+0000 7f8ae9e8f640 1 -- start start 2024-04-25T00:27:14.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.214+0000 7f8ae9e8f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8ae40c38a0 0x7f8ae406b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:14.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.214+0000 7f8ae9e8f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ae40c28e0 con 0x7f8ae40ab390 2024-04-25T00:27:14.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.214+0000 7f8ae9e8f640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ae40c2bc0 con 0x7f8ae40c34c0 2024-04-25T00:27:14.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.214+0000 7f8ae9e8f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f8ae40c3d20 con 0x7f8ae40c38a0 2024-04-25T00:27:14.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.214+0000 7f8ae968e640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f8ae40ab390 0x7f8ae415c3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:43470/0 (socket says 172.21.15.116:43470) 2024-04-25T00:27:14.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.214+0000 7f8ae968e640 1 -- 172.21.15.116:0/3493770452 learned_addr learned my addr 172.21.15.116:0/3493770452 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:27:14.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.218+0000 7f8ae37fe640 1 -- 172.21.15.116:0/3493770452 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2886348856 0 0) 0x7f8ae40c28e0 con 0x7f8ae40ab390 2024-04-25T00:27:14.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.218+0000 7f8ae37fe640 1 -- 172.21.15.116:0/3493770452 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8acc003530 con 0x7f8ae40ab390 2024-04-25T00:27:14.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.218+0000 7f8ae3fff640 1 --2- 172.21.15.116:0/3493770452 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8ae40c38a0 0x7f8ae406b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:14.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.218+0000 7f8ae37fe640 1 -- 172.21.15.116:0/3493770452 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2465997838 0 0) 0x7f8acc003530 con 0x7f8ae40ab390 2024-04-25T00:27:14.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.218+0000 7f8ae37fe640 1 -- 172.21.15.116:0/3493770452 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8ae40c38a0 msgr2=0x7f8ae406b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:27:14.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.218+0000 7f8ae37fe640 1 --2- 172.21.15.116:0/3493770452 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8ae40c38a0 0x7f8ae406b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:14.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.218+0000 7f8ae37fe640 1 -- 172.21.15.116:0/3493770452 >> v1:172.21.15.174:6789/0 conn(0x7f8ae40c34c0 legacy=0x7f8ae4155a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:14.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.218+0000 7f8ae37fe640 1 -- 172.21.15.116:0/3493770452 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ae40aa8b0 con 0x7f8ae40ab390 2024-04-25T00:27:14.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.218+0000 7f8ae37fe640 1 -- 172.21.15.116:0/3493770452 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f8adc001a00 con 0x7f8ae40ab390 2024-04-25T00:27:14.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.218+0000 7f8ae37fe640 1 -- 172.21.15.116:0/3493770452 >> v1:172.21.15.116:6789/0 conn(0x7f8ae40ab390 legacy=0x7f8ae415c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:14.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.218+0000 7f8ae37fe640 1 --2- 172.21.15.116:0/3493770452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8acc0039f0 0x7f8acc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:14.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.218+0000 7f8ae37fe640 1 --2- 172.21.15.116:0/3493770452 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8acc0043a0 0x7f8acc004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:14.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.218+0000 7f8ae37fe640 1 --2- 172.21.15.116:0/3493770452 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8ae40c38a0 0x7f8acc005170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:14.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.218+0000 7f8ae37fe640 1 -- 172.21.15.116:0/3493770452 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8ae40c3d20 con 0x7f8acc0039f0 2024-04-25T00:27:14.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.218+0000 7f8ae37fe640 1 -- 172.21.15.116:0/3493770452 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8acc00cb90 con 0x7f8acc0043a0 2024-04-25T00:27:14.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.218+0000 7f8ae37fe640 1 -- 172.21.15.116:0/3493770452 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f8acc00cd90 con 0x7f8ae40c38a0 2024-04-25T00:27:14.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.218+0000 7f8ae8e8d640 1 --2- 172.21.15.116:0/3493770452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8acc0039f0 0x7f8acc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:14.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.218+0000 7f8ae968e640 1 --2- 172.21.15.116:0/3493770452 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8ae40c38a0 0x7f8acc005170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:14.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.218+0000 7f8ae8e8d640 1 -- 172.21.15.116:0/3493770452 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8ae40c38a0 msgr2=0x7f8acc005170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:27:14.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.218+0000 7f8ae3fff640 1 --2- 172.21.15.116:0/3493770452 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8acc0043a0 0x7f8acc004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:14.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.218+0000 7f8ae8e8d640 1 --2- 172.21.15.116:0/3493770452 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8ae40c38a0 0x7f8acc005170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:14.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.218+0000 7f8ae8e8d640 1 -- 172.21.15.116:0/3493770452 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8acc0043a0 msgr2=0x7f8acc004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:27:14.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.218+0000 7f8ae8e8d640 1 --2- 172.21.15.116:0/3493770452 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8acc0043a0 0x7f8acc004870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:14.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.218+0000 7f8ae8e8d640 1 -- 172.21.15.116:0/3493770452 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ae40aa8b0 con 0x7f8acc0039f0 2024-04-25T00:27:14.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.218+0000 7f8ae968e640 1 --2- 172.21.15.116:0/3493770452 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8ae40c38a0 0x7f8acc005170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:27:14.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.218+0000 7f8ae3fff640 1 --2- 172.21.15.116:0/3493770452 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8acc0043a0 0x7f8acc004870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:27:14.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.222+0000 7f8ae8e8d640 1 --2- 172.21.15.116:0/3493770452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8acc0039f0 0x7f8acc003dd0 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f8ad8001860 tx=0x7f8ad801c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2aa44807645957b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:14.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.222+0000 7f8ae37fe640 1 -- 172.21.15.116:0/3493770452 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ad80b2070 con 0x7f8acc0039f0 2024-04-25T00:27:14.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.222+0000 7f8ae37fe640 1 -- 172.21.15.116:0/3493770452 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ad80ad640 con 0x7f8acc0039f0 2024-04-25T00:27:14.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.222+0000 7f8ae37fe640 1 -- 172.21.15.116:0/3493770452 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ad8001460 con 0x7f8acc0039f0 2024-04-25T00:27:14.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.226+0000 7f8ae9e8f640 1 -- 172.21.15.116:0/3493770452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8acc0039f0 msgr2=0x7f8acc003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:14.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.226+0000 7f8ae9e8f640 1 --2- 172.21.15.116:0/3493770452 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8acc0039f0 0x7f8acc003dd0 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f8ad8001860 tx=0x7f8ad801c4f0 comp rx=0 tx=0).stop 2024-04-25T00:27:14.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.226+0000 7f8ae968e640 1 -- 172.21.15.116:0/3493770452 reap_dead start 2024-04-25T00:27:14.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.230+0000 7f8ae9e8f640 1 -- 172.21.15.116:0/3493770452 shutdown_connections 2024-04-25T00:27:14.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.230+0000 7f8ae9e8f640 1 -- 172.21.15.116:0/3493770452 >> 172.21.15.116:0/3493770452 conn(0x7f8ae40c7af0 msgr2=0x7f8ae40c3ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:27:14.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.230+0000 7f8ae9e8f640 1 -- 172.21.15.116:0/3493770452 shutdown_connections 2024-04-25T00:27:14.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.234+0000 7f8ae9e8f640 1 -- 172.21.15.116:0/3493770452 wait complete. 2024-04-25T00:27:14.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.234+0000 7f8ae9e8f640 1 Processor -- start 2024-04-25T00:27:14.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.234+0000 7f8ae9e8f640 1 -- start start 2024-04-25T00:27:14.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.234+0000 7f8ae9e8f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8acc0039f0 0x7f8ae4148bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:14.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.234+0000 7f8ae9e8f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ae414a340 0x7f8ae41f21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:14.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.234+0000 7f8ae9e8f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8ae41523d0 0x7f8ae41f2710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:14.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.234+0000 7f8ae9e8f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8ae40c2680 con 0x7f8ae414a340 2024-04-25T00:27:14.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.234+0000 7f8ae9e8f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8ae40c2500 con 0x7f8acc0039f0 2024-04-25T00:27:14.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.234+0000 7f8ae9e8f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f8ae406b620 con 0x7f8ae41523d0 2024-04-25T00:27:14.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.234+0000 7f8ae3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ae414a340 0x7f8ae41f21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:14.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.234+0000 7f8ae3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ae414a340 0x7f8ae41f21d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:35108/0 (socket says 172.21.15.116:35108) 2024-04-25T00:27:14.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.234+0000 7f8ae3fff640 1 -- 172.21.15.116:0/2155786161 learned_addr learned my addr 172.21.15.116:0/2155786161 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:27:14.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.234+0000 7f8ae8e8d640 1 --2- 172.21.15.116:0/2155786161 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8acc0039f0 0x7f8ae4148bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:14.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.234+0000 7f8ae3fff640 1 -- 172.21.15.116:0/2155786161 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8ae41523d0 msgr2=0x7f8ae41f2710 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T00:27:14.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.234+0000 7f8ae3fff640 1 --2- 172.21.15.116:0/2155786161 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8ae41523d0 0x7f8ae41f2710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:14.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.234+0000 7f8ae3fff640 1 -- 172.21.15.116:0/2155786161 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8acc0039f0 msgr2=0x7f8ae4148bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:14.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.234+0000 7f8ae3fff640 1 --2- 172.21.15.116:0/2155786161 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8acc0039f0 0x7f8ae4148bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:14.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.234+0000 7f8ae3fff640 1 -- 172.21.15.116:0/2155786161 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ae41521c0 con 0x7f8ae414a340 2024-04-25T00:27:14.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.250+0000 7f8ae3fff640 1 --2- 172.21.15.116:0/2155786161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ae414a340 0x7f8ae41f21d0 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f8ad4010c30 tx=0x7f8ad400e540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:14.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.250+0000 7f8ae17fa640 1 -- 172.21.15.116:0/2155786161 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ad400f040 con 0x7f8ae414a340 2024-04-25T00:27:14.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.250+0000 7f8ae17fa640 1 -- 172.21.15.116:0/2155786161 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ad400ec50 con 0x7f8ae414a340 2024-04-25T00:27:14.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.250+0000 7f8ae17fa640 1 -- 172.21.15.116:0/2155786161 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ad4004020 con 0x7f8ae414a340 2024-04-25T00:27:14.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.250+0000 7f8ae9e8f640 1 -- 172.21.15.116:0/2155786161 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ae4151f20 con 0x7f8ae414a340 2024-04-25T00:27:14.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.250+0000 7f8ae9e8f640 1 -- 172.21.15.116:0/2155786161 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ae4149e60 con 0x7f8ae414a340 2024-04-25T00:27:14.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.254+0000 7f8ae9e8f640 1 -- 172.21.15.116:0/2155786161 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ae4075970 con 0x7f8ae414a340 2024-04-25T00:27:14.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.254+0000 7f8ae17fa640 1 -- 172.21.15.116:0/2155786161 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8ad40041c0 con 0x7f8ae414a340 2024-04-25T00:27:14.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.254+0000 7f8ae17fa640 1 --2- 172.21.15.116:0/2155786161 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8acc069070 0x7f8acc06b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:14.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.254+0000 7f8ae17fa640 1 -- 172.21.15.116:0/2155786161 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 10192+0+0 (secure 0 0 0) 0x7f8ad4087010 con 0x7f8ae414a340 2024-04-25T00:27:14.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.258+0000 7f8ae8e8d640 1 --2- 172.21.15.116:0/2155786161 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8acc069070 0x7f8acc06b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:14.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.258+0000 7f8ae8e8d640 1 --2- 172.21.15.116:0/2155786161 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8acc069070 0x7f8acc06b530 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f8ad80af000 tx=0x7f8ad80a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:14.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.262+0000 7f8ae17fa640 1 -- 172.21.15.116:0/2155786161 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f8ad4018030 con 0x7f8ae414a340 2024-04-25T00:27:14.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.786+0000 7f8ae9e8f640 1 -- 172.21.15.116:0/2155786161 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8ae40c22e0 con 0x7f8ae414a340 2024-04-25T00:27:14.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.786+0000 7f8ae17fa640 1 -- 172.21.15.116:0/2155786161 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 217 v217) ==== 95+0+6506 (secure 0 0 0) 0x7f8ad40044b0 con 0x7f8ae414a340 2024-04-25T00:27:14.789 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:27:14.789 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":217,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":8652,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/3891103804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3891103804},{"type":"v1","addr":"172.21.15.116:6833","nonce":3891103804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":217}],"filesystems":[{"mdsmap":{"epoch":211,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:26:19.034064+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6485},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6485":{"gid":6485,"name":"a","rank":0,"incarnation":208,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6835/2371126056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2371126056},{"type":"v1","addr":"172.21.15.116:6835","nonce":2371126056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6485,"qdb_cluster":[6485]},"id":1},{"mdsmap":{"epoch":216,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:26:49.039944+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":7384},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7384":{"gid":7384,"name":"d","rank":0,"incarnation":213,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/349772087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":349772087},{"type":"v1","addr":"172.21.15.174:6837","nonce":349772087}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":7384,"qdb_cluster":[7384]},"id":2}]} 2024-04-25T00:27:14.789 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 217 2024-04-25T00:27:14.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.794+0000 7f8ae9e8f640 1 -- 172.21.15.116:0/2155786161 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8acc069070 msgr2=0x7f8acc06b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:14.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.794+0000 7f8ae9e8f640 1 --2- 172.21.15.116:0/2155786161 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8acc069070 0x7f8acc06b530 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f8ad80af000 tx=0x7f8ad80a7070 comp rx=0 tx=0).stop 2024-04-25T00:27:14.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.794+0000 7f8ae9e8f640 1 -- 172.21.15.116:0/2155786161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ae414a340 msgr2=0x7f8ae41f21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:14.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.794+0000 7f8ae9e8f640 1 --2- 172.21.15.116:0/2155786161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ae414a340 0x7f8ae41f21d0 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f8ad4010c30 tx=0x7f8ad400e540 comp rx=0 tx=0).stop 2024-04-25T00:27:14.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.794+0000 7f8ae9e8f640 1 -- 172.21.15.116:0/2155786161 shutdown_connections 2024-04-25T00:27:14.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.794+0000 7f8ae9e8f640 1 --2- 172.21.15.116:0/2155786161 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8acc069070 0x7f8acc06b530 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:14.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.794+0000 7f8ae9e8f640 1 --2- 172.21.15.116:0/2155786161 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8ae41523d0 0x7f8ae41f2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:14.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.794+0000 7f8ae9e8f640 1 --2- 172.21.15.116:0/2155786161 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8ae414a340 0x7f8ae41f21d0 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:14.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.794+0000 7f8ae9e8f640 1 --2- 172.21.15.116:0/2155786161 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8acc0039f0 0x7f8ae4148bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:14.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.794+0000 7f8ae9e8f640 1 -- 172.21.15.116:0/2155786161 >> 172.21.15.116:0/2155786161 conn(0x7f8ae40c7af0 msgr2=0x7f8ae414cc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:27:14.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.794+0000 7f8ae9e8f640 1 -- 172.21.15.116:0/2155786161 shutdown_connections 2024-04-25T00:27:14.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:14.794+0000 7f8ae9e8f640 1 -- 172.21.15.116:0/2155786161 wait complete. 2024-04-25T00:27:14.824 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:27:16.825 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:27:16.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:16.954+0000 7fe579760640 1 Processor -- start 2024-04-25T00:27:16.953 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:16.954+0000 7fe579760640 1 -- start start 2024-04-25T00:27:16.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:16.954+0000 7fe579760640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe574150d20 0x7fe5740c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:16.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:16.954+0000 7fe579760640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fe5740a94a0 0x7fe574153ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:16.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:16.954+0000 7fe579760640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5740c3910 con 0x7fe5740c2f50 2024-04-25T00:27:16.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:16.954+0000 7fe579760640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fe574154530 con 0x7fe5740a94a0 2024-04-25T00:27:16.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:16.954+0000 7fe579760640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe5740c16e0 con 0x7fe574150d20 2024-04-25T00:27:16.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:16.954+0000 7fe572ffd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe574150d20 0x7fe5740c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:16.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:16.954+0000 7fe572ffd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe574150d20 0x7fe5740c2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:44662/0 (socket says 172.21.15.116:44662) 2024-04-25T00:27:16.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:16.954+0000 7fe572ffd640 1 -- 172.21.15.116:0/1035223938 learned_addr learned my addr 172.21.15.116:0/1035223938 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:27:16.955 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:16.954+0000 7fe5737fe640 1 --2- 172.21.15.116:0/1035223938 >> v2:172.21.15.174:3300/0 conn(0x7fe5740a94a0 0x7fe574153ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:16.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:16.954+0000 7fe572ffd640 1 -- 172.21.15.116:0/1035223938 >> v2:172.21.15.174:3300/0 conn(0x7fe5740a94a0 msgr2=0x7fe574153ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:27:16.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:16.954+0000 7fe572ffd640 1 --2- 172.21.15.116:0/1035223938 >> v2:172.21.15.174:3300/0 conn(0x7fe5740a94a0 0x7fe574153ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:16.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:16.954+0000 7fe572ffd640 1 -- 172.21.15.116:0/1035223938 >> v1:172.21.15.174:6789/0 conn(0x7fe5740c2f50 legacy=0x7fe5741538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:16.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:16.954+0000 7fe572ffd640 1 -- 172.21.15.116:0/1035223938 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5740c1400 con 0x7fe574150d20 2024-04-25T00:27:16.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:16.958+0000 7fe572ffd640 1 --2- 172.21.15.116:0/1035223938 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe574150d20 0x7fe5740c2a10 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe564009870 tx=0x7fe56401c560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ad42b25b36b0349 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:16.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:16.958+0000 7fe571ffb640 1 -- 172.21.15.116:0/1035223938 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5640a7070 con 0x7fe574150d20 2024-04-25T00:27:16.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:16.958+0000 7fe571ffb640 1 -- 172.21.15.116:0/1035223938 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5640b0070 con 0x7fe574150d20 2024-04-25T00:27:16.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:16.958+0000 7fe571ffb640 1 -- 172.21.15.116:0/1035223938 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5640ab890 con 0x7fe574150d20 2024-04-25T00:27:16.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:16.958+0000 7fe579760640 1 -- 172.21.15.116:0/1035223938 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe574150d20 msgr2=0x7fe5740c2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:16.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:16.958+0000 7fe579760640 1 --2- 172.21.15.116:0/1035223938 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe574150d20 0x7fe5740c2a10 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe564009870 tx=0x7fe56401c560 comp rx=0 tx=0).stop 2024-04-25T00:27:16.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:16.958+0000 7fe579760640 1 -- 172.21.15.116:0/1035223938 shutdown_connections 2024-04-25T00:27:16.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:16.958+0000 7fe579760640 1 --2- 172.21.15.116:0/1035223938 >> v2:172.21.15.174:3300/0 conn(0x7fe5740a94a0 0x7fe574153ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:16.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:16.958+0000 7fe579760640 1 --2- 172.21.15.116:0/1035223938 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe574150d20 0x7fe5740c2a10 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:16.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:16.958+0000 7fe579760640 1 -- 172.21.15.116:0/1035223938 >> 172.21.15.116:0/1035223938 conn(0x7fe5740c3b00 msgr2=0x7fe5740c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:27:16.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:16.958+0000 7fe579760640 1 -- 172.21.15.116:0/1035223938 shutdown_connections 2024-04-25T00:27:16.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:16.958+0000 7fe579760640 1 -- 172.21.15.116:0/1035223938 wait complete. 2024-04-25T00:27:16.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:16.958+0000 7fe579760640 1 Processor -- start 2024-04-25T00:27:16.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:16.958+0000 7fe579760640 1 -- start start 2024-04-25T00:27:16.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:16.958+0000 7fe579760640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe5740a94a0 0x7fe574147e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:16.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:16.958+0000 7fe579760640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5740c2f50 0x7fe574152670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:16.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:16.958+0000 7fe579760640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe574150d20 0x7fe574152bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:16.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:16.958+0000 7fe579760640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe5741608b0 con 0x7fe574150d20 2024-04-25T00:27:16.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:16.958+0000 7fe579760640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe574160730 con 0x7fe5740c2f50 2024-04-25T00:27:16.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:16.958+0000 7fe579760640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe574160a30 con 0x7fe5740a94a0 2024-04-25T00:27:16.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:16.958+0000 7fe5737fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe574150d20 0x7fe574152bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:16.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:16.958+0000 7fe5737fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe574150d20 0x7fe574152bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46478/0 (socket says 172.21.15.116:46478) 2024-04-25T00:27:16.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:16.958+0000 7fe5737fe640 1 -- 172.21.15.116:0/1418301027 learned_addr learned my addr 172.21.15.116:0/1418301027 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:27:16.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:16.958+0000 7fe5727fc640 1 --2- 172.21.15.116:0/1418301027 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5740c2f50 0x7fe574152670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:16.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:16.958+0000 7fe572ffd640 1 --2- 172.21.15.116:0/1418301027 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe5740a94a0 0x7fe574147e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:16.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:16.962+0000 7fe5737fe640 1 -- 172.21.15.116:0/1418301027 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe5740a94a0 msgr2=0x7fe574147e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:16.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:16.962+0000 7fe5737fe640 1 --2- 172.21.15.116:0/1418301027 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe5740a94a0 0x7fe574147e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:16.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:16.962+0000 7fe5737fe640 1 -- 172.21.15.116:0/1418301027 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5740c2f50 msgr2=0x7fe574152670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:16.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:16.962+0000 7fe5737fe640 1 --2- 172.21.15.116:0/1418301027 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5740c2f50 0x7fe574152670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:16.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:16.962+0000 7fe5737fe640 1 -- 172.21.15.116:0/1418301027 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe574068560 con 0x7fe574150d20 2024-04-25T00:27:16.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:16.962+0000 7fe5737fe640 1 --2- 172.21.15.116:0/1418301027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe574150d20 0x7fe574152bb0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7fe56c009dd0 tx=0x7fe56c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:16.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:16.962+0000 7fe56b7fe640 1 -- 172.21.15.116:0/1418301027 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe56c018020 con 0x7fe574150d20 2024-04-25T00:27:16.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:16.962+0000 7fe56b7fe640 1 -- 172.21.15.116:0/1418301027 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe56c014480 con 0x7fe574150d20 2024-04-25T00:27:16.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:16.962+0000 7fe56b7fe640 1 -- 172.21.15.116:0/1418301027 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe56c014870 con 0x7fe574150d20 2024-04-25T00:27:16.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:16.962+0000 7fe579760640 1 -- 172.21.15.116:0/1418301027 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe574149dc0 con 0x7fe574150d20 2024-04-25T00:27:16.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:16.962+0000 7fe579760640 1 -- 172.21.15.116:0/1418301027 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe57414a380 con 0x7fe574150d20 2024-04-25T00:27:16.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:16.962+0000 7fe579760640 1 -- 172.21.15.116:0/1418301027 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5580052a0 con 0x7fe574150d20 2024-04-25T00:27:16.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:16.962+0000 7fe56b7fe640 1 -- 172.21.15.116:0/1418301027 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe56c01b020 con 0x7fe574150d20 2024-04-25T00:27:16.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:16.962+0000 7fe56b7fe640 1 --2- 172.21.15.116:0/1418301027 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe5540678c0 0x7fe554069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:16.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:16.962+0000 7fe56b7fe640 1 -- 172.21.15.116:0/1418301027 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 10278+0+0 (secure 0 0 0) 0x7fe56c0877d0 con 0x7fe574150d20 2024-04-25T00:27:16.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:16.962+0000 7fe572ffd640 1 --2- 172.21.15.116:0/1418301027 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe5540678c0 0x7fe554069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:16.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:16.962+0000 7fe572ffd640 1 --2- 172.21.15.116:0/1418301027 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe5540678c0 0x7fe554069d80 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fe5640af000 tx=0x7fe5640a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:16.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:16.966+0000 7fe56b7fe640 1 -- 172.21.15.116:0/1418301027 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fe56c022070 con 0x7fe574150d20 2024-04-25T00:27:17.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.150+0000 7fe579760640 1 -- 172.21.15.116:0/1418301027 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe558005860 con 0x7fe574150d20 2024-04-25T00:27:17.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.150+0000 7fe56b7fe640 1 -- 172.21.15.116:0/1418301027 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 219 v219) ==== 95+0+5676 (secure 0 0 0) 0x7fe56c010040 con 0x7fe574150d20 2024-04-25T00:27:17.151 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:27:17.151 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":219,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":211,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:26:19.034064+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6485},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6485":{"gid":6485,"name":"a","rank":0,"incarnation":208,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6835/2371126056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2371126056},{"type":"v1","addr":"172.21.15.116:6835","nonce":2371126056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6485,"qdb_cluster":[6485]},"id":1},{"mdsmap":{"epoch":219,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:27:16.202113+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8652":{"gid":8652,"name":"c","rank":0,"incarnation":218,"state":"up:reconnect","state_seq":7,"addr":"172.21.15.116:6833/3891103804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3891103804},{"type":"v1","addr":"172.21.15.116:6833","nonce":3891103804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:27:17.152 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 219 2024-04-25T00:27:17.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.154+0000 7fe579760640 1 -- 172.21.15.116:0/1418301027 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe5540678c0 msgr2=0x7fe554069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:17.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.154+0000 7fe579760640 1 --2- 172.21.15.116:0/1418301027 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe5540678c0 0x7fe554069d80 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fe5640af000 tx=0x7fe5640a7000 comp rx=0 tx=0).stop 2024-04-25T00:27:17.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.154+0000 7fe579760640 1 -- 172.21.15.116:0/1418301027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe574150d20 msgr2=0x7fe574152bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:17.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.154+0000 7fe579760640 1 --2- 172.21.15.116:0/1418301027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe574150d20 0x7fe574152bb0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7fe56c009dd0 tx=0x7fe56c00c490 comp rx=0 tx=0).stop 2024-04-25T00:27:17.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.154+0000 7fe579760640 1 -- 172.21.15.116:0/1418301027 shutdown_connections 2024-04-25T00:27:17.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.154+0000 7fe579760640 1 --2- 172.21.15.116:0/1418301027 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe5540678c0 0x7fe554069d80 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:17.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.154+0000 7fe579760640 1 --2- 172.21.15.116:0/1418301027 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe574150d20 0x7fe574152bb0 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:17.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.154+0000 7fe579760640 1 --2- 172.21.15.116:0/1418301027 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5740c2f50 0x7fe574152670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:17.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.154+0000 7fe579760640 1 --2- 172.21.15.116:0/1418301027 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe5740a94a0 0x7fe574147e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:17.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.154+0000 7fe579760640 1 -- 172.21.15.116:0/1418301027 >> 172.21.15.116:0/1418301027 conn(0x7fe5740c3b00 msgr2=0x7fe57414ed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:27:17.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.154+0000 7fe579760640 1 -- 172.21.15.116:0/1418301027 shutdown_connections 2024-04-25T00:27:17.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.154+0000 7fe579760640 1 -- 172.21.15.116:0/1418301027 wait complete. 2024-04-25T00:27:17.169 INFO:tasks.mds_thrash.fs.[b]:mds.d down, removed from mdsmap 2024-04-25T00:27:17.169 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-04-25T00:27:17.169 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:27:17.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.282+0000 7f94793c2640 1 Processor -- start 2024-04-25T00:27:17.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.282+0000 7f94793c2640 1 -- start start 2024-04-25T00:27:17.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.282+0000 7f94793c2640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f94740aa170 0x7f9474155af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:17.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.282+0000 7f94793c2640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9474156030 0x7f947415ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:17.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.282+0000 7f94793c2640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94741526a0 con 0x7f94740c4300 2024-04-25T00:27:17.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.282+0000 7f94793c2640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f9474153460 con 0x7f94740aa170 2024-04-25T00:27:17.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.282+0000 7f94793c2640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f947406ffd0 con 0x7f9474156030 2024-04-25T00:27:17.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.282+0000 7f94737fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9474156030 0x7f947415ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:17.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.282+0000 7f94727fc640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f94740aa170 0x7f9474155af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:17.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.282+0000 7f94737fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9474156030 0x7f947415ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:44690/0 (socket says 172.21.15.116:44690) 2024-04-25T00:27:17.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.282+0000 7f9472ffd640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f94740c4300 0x7f94740c5100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:33372/0 (socket says 172.21.15.116:33372) 2024-04-25T00:27:17.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.282+0000 7f94737fe640 1 -- 172.21.15.116:0/3046380871 learned_addr learned my addr 172.21.15.116:0/3046380871 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:27:17.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.282+0000 7f94727fc640 1 -- 172.21.15.116:0/3046380871 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9474156030 msgr2=0x7f947415ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:27:17.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.282+0000 7f94727fc640 1 --2- 172.21.15.116:0/3046380871 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9474156030 0x7f947415ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:17.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.282+0000 7f94727fc640 1 -- 172.21.15.116:0/3046380871 >> v1:172.21.15.174:6789/0 conn(0x7f94740c4300 legacy=0x7f94740c5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:17.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.282+0000 7f94727fc640 1 -- 172.21.15.116:0/3046380871 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9474152980 con 0x7f94740aa170 2024-04-25T00:27:17.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.282+0000 7f94737fe640 1 --2- 172.21.15.116:0/3046380871 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9474156030 0x7f947415ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:27:17.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.286+0000 7f94727fc640 1 --2- 172.21.15.116:0/3046380871 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f94740aa170 0x7f9474155af0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f946800ba60 tx=0x7f946801eb70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=455e33574181a9e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:17.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.286+0000 7f9471ffb640 1 -- 172.21.15.116:0/3046380871 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94680b4070 con 0x7f94740aa170 2024-04-25T00:27:17.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.286+0000 7f9471ffb640 1 -- 172.21.15.116:0/3046380871 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94680a7960 con 0x7f94740aa170 2024-04-25T00:27:17.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.286+0000 7f9471ffb640 1 -- 172.21.15.116:0/3046380871 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94680a7d30 con 0x7f94740aa170 2024-04-25T00:27:17.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.286+0000 7f94793c2640 1 -- 172.21.15.116:0/3046380871 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f94740aa170 msgr2=0x7f9474155af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:17.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.286+0000 7f94793c2640 1 --2- 172.21.15.116:0/3046380871 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f94740aa170 0x7f9474155af0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f946800ba60 tx=0x7f946801eb70 comp rx=0 tx=0).stop 2024-04-25T00:27:17.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.286+0000 7f94793c2640 1 -- 172.21.15.116:0/3046380871 shutdown_connections 2024-04-25T00:27:17.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.286+0000 7f94793c2640 1 --2- 172.21.15.116:0/3046380871 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9474156030 0x7f947415ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:17.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.286+0000 7f94793c2640 1 --2- 172.21.15.116:0/3046380871 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f94740aa170 0x7f9474155af0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:17.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.286+0000 7f94793c2640 1 -- 172.21.15.116:0/3046380871 >> 172.21.15.116:0/3046380871 conn(0x7f947406c720 msgr2=0x7f947406cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:27:17.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.286+0000 7f94793c2640 1 -- 172.21.15.116:0/3046380871 shutdown_connections 2024-04-25T00:27:17.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.286+0000 7f94793c2640 1 -- 172.21.15.116:0/3046380871 wait complete. 2024-04-25T00:27:17.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.286+0000 7f94793c2640 1 Processor -- start 2024-04-25T00:27:17.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.286+0000 7f94793c2640 1 -- start start 2024-04-25T00:27:17.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.290+0000 7f94793c2640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f94740aa170 0x7f94741447e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:17.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.290+0000 7f94793c2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94740c4300 0x7f94740ac400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:17.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.290+0000 7f94793c2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9474156030 0x7f947414b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:17.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.290+0000 7f94793c2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f94740aae50 con 0x7f94740c4300 2024-04-25T00:27:17.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.290+0000 7f94793c2640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f94740aacd0 con 0x7f9474156030 2024-04-25T00:27:17.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.290+0000 7f94793c2640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f947406fb80 con 0x7f94740aa170 2024-04-25T00:27:17.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.290+0000 7f94727fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94740c4300 0x7f94740ac400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:17.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.290+0000 7f94727fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94740c4300 0x7f94740ac400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46480/0 (socket says 172.21.15.116:46480) 2024-04-25T00:27:17.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.290+0000 7f94727fc640 1 -- 172.21.15.116:0/2253849193 learned_addr learned my addr 172.21.15.116:0/2253849193 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:27:17.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.290+0000 7f9472ffd640 1 --2- 172.21.15.116:0/2253849193 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f94740aa170 0x7f94741447e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:17.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.290+0000 7f94737fe640 1 --2- 172.21.15.116:0/2253849193 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9474156030 0x7f947414b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:17.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.290+0000 7f94727fc640 1 -- 172.21.15.116:0/2253849193 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f94740aa170 msgr2=0x7f94741447e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:17.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.290+0000 7f94727fc640 1 --2- 172.21.15.116:0/2253849193 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f94740aa170 0x7f94741447e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:17.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.290+0000 7f94727fc640 1 -- 172.21.15.116:0/2253849193 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9474156030 msgr2=0x7f947414b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:17.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.290+0000 7f94727fc640 1 --2- 172.21.15.116:0/2253849193 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9474156030 0x7f947414b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:17.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.290+0000 7f94727fc640 1 -- 172.21.15.116:0/2253849193 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94741458e0 con 0x7f94740c4300 2024-04-25T00:27:17.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.290+0000 7f94727fc640 1 --2- 172.21.15.116:0/2253849193 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94740c4300 0x7f94740ac400 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f946801c7a0 tx=0x7f946801b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:17.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.290+0000 7f945b7fe640 1 -- 172.21.15.116:0/2253849193 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94680b4070 con 0x7f94740c4300 2024-04-25T00:27:17.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.290+0000 7f945b7fe640 1 -- 172.21.15.116:0/2253849193 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94680be070 con 0x7f94740c4300 2024-04-25T00:27:17.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.290+0000 7f94793c2640 1 -- 172.21.15.116:0/2253849193 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9474149730 con 0x7f94740c4300 2024-04-25T00:27:17.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.290+0000 7f945b7fe640 1 -- 172.21.15.116:0/2253849193 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94680afa70 con 0x7f94740c4300 2024-04-25T00:27:17.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.290+0000 7f94793c2640 1 -- 172.21.15.116:0/2253849193 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9474149cf0 con 0x7f94740c4300 2024-04-25T00:27:17.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.290+0000 7f94793c2640 1 -- 172.21.15.116:0/2253849193 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9474068a60 con 0x7f94740c4300 2024-04-25T00:27:17.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.290+0000 7f945b7fe640 1 -- 172.21.15.116:0/2253849193 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f94680afc10 con 0x7f94740c4300 2024-04-25T00:27:17.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.290+0000 7f945b7fe640 1 --2- 172.21.15.116:0/2253849193 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f94600678c0 0x7f9460069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:17.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.290+0000 7f945b7fe640 1 -- 172.21.15.116:0/2253849193 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 10278+0+0 (secure 0 0 0) 0x7f9468123a20 con 0x7f94740c4300 2024-04-25T00:27:17.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.290+0000 7f9472ffd640 1 --2- 172.21.15.116:0/2253849193 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f94600678c0 0x7f9460069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:17.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.290+0000 7f9472ffd640 1 --2- 172.21.15.116:0/2253849193 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f94600678c0 0x7f9460069d80 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f946c007fd0 tx=0x7f946c005b80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:17.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.294+0000 7f945b7fe640 1 -- 172.21.15.116:0/2253849193 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f9468018760 con 0x7f94740c4300 2024-04-25T00:27:17.474 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.474+0000 7f94793c2640 1 -- 172.21.15.116:0/2253849193 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f94740a9f50 con 0x7f94740c4300 2024-04-25T00:27:17.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.474+0000 7f945b7fe640 1 -- 172.21.15.116:0/2253849193 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 219 v219) ==== 95+0+5676 (secure 0 0 0) 0x7f94680f3340 con 0x7f94740c4300 2024-04-25T00:27:17.476 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:27:17.476 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":219,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":211,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:26:19.034064+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6485},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6485":{"gid":6485,"name":"a","rank":0,"incarnation":208,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6835/2371126056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2371126056},{"type":"v1","addr":"172.21.15.116:6835","nonce":2371126056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6485,"qdb_cluster":[6485]},"id":1},{"mdsmap":{"epoch":219,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:27:16.202113+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8652":{"gid":8652,"name":"c","rank":0,"incarnation":218,"state":"up:reconnect","state_seq":7,"addr":"172.21.15.116:6833/3891103804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3891103804},{"type":"v1","addr":"172.21.15.116:6833","nonce":3891103804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:27:17.476 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 219 2024-04-25T00:27:17.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.478+0000 7f94793c2640 1 -- 172.21.15.116:0/2253849193 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f94600678c0 msgr2=0x7f9460069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:17.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.478+0000 7f94793c2640 1 --2- 172.21.15.116:0/2253849193 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f94600678c0 0x7f9460069d80 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f946c007fd0 tx=0x7f946c005b80 comp rx=0 tx=0).stop 2024-04-25T00:27:17.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.478+0000 7f94793c2640 1 -- 172.21.15.116:0/2253849193 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94740c4300 msgr2=0x7f94740ac400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:17.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.478+0000 7f94793c2640 1 --2- 172.21.15.116:0/2253849193 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94740c4300 0x7f94740ac400 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f946801c7a0 tx=0x7f946801b910 comp rx=0 tx=0).stop 2024-04-25T00:27:17.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.478+0000 7f94793c2640 1 -- 172.21.15.116:0/2253849193 shutdown_connections 2024-04-25T00:27:17.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.478+0000 7f94793c2640 1 --2- 172.21.15.116:0/2253849193 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f94600678c0 0x7f9460069d80 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:17.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.478+0000 7f94793c2640 1 --2- 172.21.15.116:0/2253849193 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9474156030 0x7f947414b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:17.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.478+0000 7f94793c2640 1 --2- 172.21.15.116:0/2253849193 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f94740c4300 0x7f94740ac400 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:17.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.478+0000 7f94793c2640 1 --2- 172.21.15.116:0/2253849193 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f94740aa170 0x7f94741447e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:17.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.478+0000 7f94793c2640 1 -- 172.21.15.116:0/2253849193 >> 172.21.15.116:0/2253849193 conn(0x7f947406c720 msgr2=0x7f94740c5d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:27:17.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.478+0000 7f94793c2640 1 -- 172.21.15.116:0/2253849193 shutdown_connections 2024-04-25T00:27:17.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:17.478+0000 7f94793c2640 1 -- 172.21.15.116:0/2253849193 wait complete. 2024-04-25T00:27:17.494 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 219, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:04.678737+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 211, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6485": { "addr": "172.21.15.116:6835/2371126056", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6834", "nonce": 2371126056, "type": "v2" }, { "addr": "172.21.15.116:6835", "nonce": 2371126056, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6485, "incarnation": 208, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 10 } }, "last_failure": 0, "last_failure_osd_epoch": 68, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T00:26:19.034064+0000", "qdb_cluster": [ 6485 ], "qdb_leader": 6485, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6485 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:18.825887+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 219, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_8652": { "addr": "172.21.15.116:6833/3891103804", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6832", "nonce": 3891103804, "type": "v2" }, { "addr": "172.21.15.116:6833", "nonce": 3891103804, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8652, "incarnation": 218, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:reconnect", "state_seq": 7 } }, "last_failure": 0, "last_failure_osd_epoch": 70, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-04-25T00:27:16.202113+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8652 } } } ], "standbys": [ { "addr": "172.21.15.174:6835/4112145782", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6834", "nonce": 4112145782, "type": "v2" }, { "addr": "172.21.15.174:6835", "nonce": 4112145782, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 112, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5582, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-25T00:27:19.495 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:27:19.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.634+0000 7efde759e640 1 Processor -- start 2024-04-25T00:27:19.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.634+0000 7efde759e640 1 -- start start 2024-04-25T00:27:19.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.638+0000 7efde759e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7efde80c3040 0x7efde806e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:19.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.638+0000 7efde759e640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7efde80c24b0 0x7efde8152350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:19.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.638+0000 7efde759e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efde8073490 0x7efde8155210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:19.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.638+0000 7efde759e640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7efde80c37d0 con 0x7efde80c3040 2024-04-25T00:27:19.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.638+0000 7efde659c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7efde80c3040 0x7efde806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:19.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.638+0000 7efde659c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7efde80c3040 0x7efde806e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46492/0 (socket says 172.21.15.116:46492) 2024-04-25T00:27:19.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.638+0000 7efde659c640 1 -- 172.21.15.116:0/627517369 learned_addr learned my addr 172.21.15.116:0/627517369 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:27:19.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.638+0000 7efde759e640 1 -- 172.21.15.116:0/627517369 --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7efde80c3a20 con 0x7efde80c24b0 2024-04-25T00:27:19.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.638+0000 7efde759e640 1 -- 172.21.15.116:0/627517369 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7efde80c3d00 con 0x7efde8073490 2024-04-25T00:27:19.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.638+0000 7efde5d9b640 1 --2- 172.21.15.116:0/627517369 >> v2:172.21.15.174:3300/0 conn(0x7efde80c24b0 0x7efde8152350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:19.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.638+0000 7efde659c640 1 -- 172.21.15.116:0/627517369 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efde8073490 msgr2=0x7efde8155210 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T00:27:19.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.638+0000 7efde659c640 1 --2- 172.21.15.116:0/627517369 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efde8073490 0x7efde8155210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:19.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.638+0000 7efde659c640 1 -- 172.21.15.116:0/627517369 >> v2:172.21.15.174:3300/0 conn(0x7efde80c24b0 msgr2=0x7efde8152350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:27:19.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.638+0000 7efde659c640 1 --2- 172.21.15.116:0/627517369 >> v2:172.21.15.174:3300/0 conn(0x7efde80c24b0 0x7efde8152350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:19.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.638+0000 7efde659c640 1 -- 172.21.15.116:0/627517369 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efde8151e30 con 0x7efde80c3040 2024-04-25T00:27:19.640 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.638+0000 7efde659c640 1 --2- 172.21.15.116:0/627517369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efde80c3040 0x7efde806e350 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7efde80aa4e0 tx=0x7efdd801c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fea03cdf998ec87 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:19.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.638+0000 7efde559a640 1 -- 172.21.15.116:0/627517369 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd80b2070 con 0x7efde80c3040 2024-04-25T00:27:19.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.642+0000 7efde559a640 1 -- 172.21.15.116:0/627517369 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efdd80ad680 con 0x7efde80c3040 2024-04-25T00:27:19.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.642+0000 7efde559a640 1 -- 172.21.15.116:0/627517369 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd80ada50 con 0x7efde80c3040 2024-04-25T00:27:19.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.642+0000 7efde759e640 1 -- 172.21.15.116:0/627517369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efde80c3040 msgr2=0x7efde806e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:19.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.642+0000 7efde759e640 1 --2- 172.21.15.116:0/627517369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efde80c3040 0x7efde806e350 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7efde80aa4e0 tx=0x7efdd801c4b0 comp rx=0 tx=0).stop 2024-04-25T00:27:19.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.642+0000 7efde759e640 1 -- 172.21.15.116:0/627517369 shutdown_connections 2024-04-25T00:27:19.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.642+0000 7efde759e640 1 --2- 172.21.15.116:0/627517369 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efde8073490 0x7efde8155210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:19.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.642+0000 7efde759e640 1 --2- 172.21.15.116:0/627517369 >> v2:172.21.15.174:3300/0 conn(0x7efde80c24b0 0x7efde8152350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:19.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.642+0000 7efde759e640 1 --2- 172.21.15.116:0/627517369 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efde80c3040 0x7efde806e350 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:19.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.642+0000 7efde759e640 1 -- 172.21.15.116:0/627517369 >> 172.21.15.116:0/627517369 conn(0x7efde80a9a20 msgr2=0x7efde80c4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:27:19.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.642+0000 7efde759e640 1 -- 172.21.15.116:0/627517369 shutdown_connections 2024-04-25T00:27:19.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.642+0000 7efde759e640 1 -- 172.21.15.116:0/627517369 wait complete. 2024-04-25T00:27:19.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.646+0000 7efde759e640 1 Processor -- start 2024-04-25T00:27:19.646 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.646+0000 7efde759e640 1 -- start start 2024-04-25T00:27:19.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.646+0000 7efde759e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efde8073490 0x7efde80c09d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:19.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.646+0000 7efde759e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efde80c24b0 0x7efde80c0f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:19.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.646+0000 7efde759e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efde80c3040 0x7efde814e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:19.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.646+0000 7efde759e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7efde8152890 con 0x7efde8073490 2024-04-25T00:27:19.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.646+0000 7efde759e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7efde80c2280 con 0x7efde80c24b0 2024-04-25T00:27:19.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.646+0000 7efde759e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7efde8152a10 con 0x7efde80c3040 2024-04-25T00:27:19.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.646+0000 7efde659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efde8073490 0x7efde80c09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:19.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.646+0000 7efde659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efde8073490 0x7efde80c09d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46504/0 (socket says 172.21.15.116:46504) 2024-04-25T00:27:19.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.646+0000 7efde659c640 1 -- 172.21.15.116:0/3546658324 learned_addr learned my addr 172.21.15.116:0/3546658324 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:27:19.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.646+0000 7efde6d9d640 1 --2- 172.21.15.116:0/3546658324 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efde80c3040 0x7efde814e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:19.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.646+0000 7efde5d9b640 1 --2- 172.21.15.116:0/3546658324 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efde80c24b0 0x7efde80c0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:19.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.646+0000 7efde659c640 1 -- 172.21.15.116:0/3546658324 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efde80c3040 msgr2=0x7efde814e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:19.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.646+0000 7efde659c640 1 --2- 172.21.15.116:0/3546658324 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efde80c3040 0x7efde814e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:19.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.646+0000 7efde659c640 1 -- 172.21.15.116:0/3546658324 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efde80c24b0 msgr2=0x7efde80c0f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:19.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.646+0000 7efde659c640 1 --2- 172.21.15.116:0/3546658324 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efde80c24b0 0x7efde80c0f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:19.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.646+0000 7efde659c640 1 -- 172.21.15.116:0/3546658324 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efde81f0940 con 0x7efde8073490 2024-04-25T00:27:19.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.646+0000 7efde659c640 1 --2- 172.21.15.116:0/3546658324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efde8073490 0x7efde80c09d0 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7efdd801cd50 tx=0x7efdd801acf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:19.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.646+0000 7efdd6ffd640 1 -- 172.21.15.116:0/3546658324 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd80bc020 con 0x7efde8073490 2024-04-25T00:27:19.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.646+0000 7efdd6ffd640 1 -- 172.21.15.116:0/3546658324 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efdd80adbf0 con 0x7efde8073490 2024-04-25T00:27:19.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.646+0000 7efdd6ffd640 1 -- 172.21.15.116:0/3546658324 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd80b2040 con 0x7efde8073490 2024-04-25T00:27:19.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.646+0000 7efde759e640 1 -- 172.21.15.116:0/3546658324 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efde80a9080 con 0x7efde8073490 2024-04-25T00:27:19.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.646+0000 7efde759e640 1 -- 172.21.15.116:0/3546658324 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efde814f200 con 0x7efde8073490 2024-04-25T00:27:19.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.650+0000 7efde759e640 1 -- 172.21.15.116:0/3546658324 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efde804f510 con 0x7efde8073490 2024-04-25T00:27:19.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.650+0000 7efdd6ffd640 1 -- 172.21.15.116:0/3546658324 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7efdd80bf020 con 0x7efde8073490 2024-04-25T00:27:19.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.650+0000 7efdd6ffd640 1 --2- 172.21.15.116:0/3546658324 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efdc80678c0 0x7efdc8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:19.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.650+0000 7efdd6ffd640 1 -- 172.21.15.116:0/3546658324 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 10278+0+0 (secure 0 0 0) 0x7efdd8120d50 con 0x7efde8073490 2024-04-25T00:27:19.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.650+0000 7efde5d9b640 1 --2- 172.21.15.116:0/3546658324 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efdc80678c0 0x7efdc8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:19.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.650+0000 7efde5d9b640 1 --2- 172.21.15.116:0/3546658324 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efdc80678c0 0x7efdc8069d80 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7efde8156790 tx=0x7efddc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:19.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.658+0000 7efdd6ffd640 1 -- 172.21.15.116:0/3546658324 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7efdd8124030 con 0x7efde8073490 2024-04-25T00:27:19.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.958+0000 7efde759e640 1 -- 172.21.15.116:0/3546658324 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efde80c5010 con 0x7efde8073490 2024-04-25T00:27:19.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.958+0000 7efdd6ffd640 1 -- 172.21.15.116:0/3546658324 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 221 v221) ==== 95+0+5680 (secure 0 0 0) 0x7efdd80c6030 con 0x7efde8073490 2024-04-25T00:27:19.961 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:27:19.961 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":221,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":211,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:26:19.034064+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6485},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6485":{"gid":6485,"name":"a","rank":0,"incarnation":208,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6835/2371126056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2371126056},{"type":"v1","addr":"172.21.15.116:6835","nonce":2371126056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6485,"qdb_cluster":[6485]},"id":1},{"mdsmap":{"epoch":221,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:27:19.052286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8652":{"gid":8652,"name":"c","rank":0,"incarnation":218,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6833/3891103804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3891103804},{"type":"v1","addr":"172.21.15.116:6833","nonce":3891103804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8652,"qdb_cluster":[8652]},"id":2}]} 2024-04-25T00:27:19.961 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 221 2024-04-25T00:27:19.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.962+0000 7efde759e640 1 -- 172.21.15.116:0/3546658324 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efdc80678c0 msgr2=0x7efdc8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:19.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.962+0000 7efde759e640 1 --2- 172.21.15.116:0/3546658324 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efdc80678c0 0x7efdc8069d80 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7efde8156790 tx=0x7efddc008040 comp rx=0 tx=0).stop 2024-04-25T00:27:19.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.962+0000 7efde759e640 1 -- 172.21.15.116:0/3546658324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efde8073490 msgr2=0x7efde80c09d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:19.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.962+0000 7efde759e640 1 --2- 172.21.15.116:0/3546658324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efde8073490 0x7efde80c09d0 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7efdd801cd50 tx=0x7efdd801acf0 comp rx=0 tx=0).stop 2024-04-25T00:27:19.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.962+0000 7efde759e640 1 -- 172.21.15.116:0/3546658324 shutdown_connections 2024-04-25T00:27:19.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.962+0000 7efde759e640 1 --2- 172.21.15.116:0/3546658324 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efdc80678c0 0x7efdc8069d80 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:19.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.962+0000 7efde759e640 1 --2- 172.21.15.116:0/3546658324 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efde80c3040 0x7efde814e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:19.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.962+0000 7efde759e640 1 --2- 172.21.15.116:0/3546658324 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efde80c24b0 0x7efde80c0f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:19.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.962+0000 7efde759e640 1 --2- 172.21.15.116:0/3546658324 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efde8073490 0x7efde80c09d0 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:19.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.962+0000 7efde759e640 1 -- 172.21.15.116:0/3546658324 >> 172.21.15.116:0/3546658324 conn(0x7efde80a9a20 msgr2=0x7efde80c1770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:27:19.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.962+0000 7efde759e640 1 -- 172.21.15.116:0/3546658324 shutdown_connections 2024-04-25T00:27:19.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:19.962+0000 7efde759e640 1 -- 172.21.15.116:0/3546658324 wait complete. 2024-04-25T00:27:19.988 INFO:tasks.mds_thrash.fs.[b]:mds.c has gained rank=0, replacing gid=7384 2024-04-25T00:27:19.988 INFO:tasks.mds_thrash.fs.[b]:waiting for 3 secs before reviving mds.d 2024-04-25T00:27:20.145 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:27:20.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.290+0000 7ff6a71a0640 1 Processor -- start 2024-04-25T00:27:20.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.290+0000 7ff6a71a0640 1 -- start start 2024-04-25T00:27:20.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.290+0000 7ff6a71a0640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff6a0150c90 0x7ff6a00c72a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:20.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.290+0000 7ff6a71a0640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff6a00c77e0 0x7ff6a00c3c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:20.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.290+0000 7ff6a71a0640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6a006ea20 con 0x7ff6a006f880 2024-04-25T00:27:20.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.290+0000 7ff6a71a0640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7ff6a00c2480 con 0x7ff6a0150c90 2024-04-25T00:27:20.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.290+0000 7ff6a71a0640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff6a00aa550 con 0x7ff6a00c77e0 2024-04-25T00:27:20.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.290+0000 7ff69ffff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff6a00c77e0 0x7ff6a00c3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:20.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.290+0000 7ff6a4f3c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff6a0150c90 0x7ff6a00c72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:20.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.290+0000 7ff69ffff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff6a00c77e0 0x7ff6a00c3c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:44730/0 (socket says 172.21.15.116:44730) 2024-04-25T00:27:20.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.290+0000 7ff69ffff640 1 -- 172.21.15.116:0/3688301132 learned_addr learned my addr 172.21.15.116:0/3688301132 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:27:20.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.294+0000 7ff6a4f3c640 1 -- 172.21.15.116:0/3688301132 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff6a00c77e0 msgr2=0x7ff6a00c3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:27:20.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.294+0000 7ff6a4f3c640 1 --2- 172.21.15.116:0/3688301132 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff6a00c77e0 0x7ff6a00c3c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:20.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.294+0000 7ff6a4f3c640 1 -- 172.21.15.116:0/3688301132 >> v1:172.21.15.174:6789/0 conn(0x7ff6a006f880 legacy=0x7ff6a00c4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:27:20.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.294+0000 7ff6a4f3c640 1 -- 172.21.15.116:0/3688301132 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6a006ece0 con 0x7ff6a0150c90 2024-04-25T00:27:20.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.294+0000 7ff6a4f3c640 1 --2- 172.21.15.116:0/3688301132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6a0150c90 0x7ff6a00c72a0 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7ff6940097c0 tx=0x7ff69401c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55f2fd352fd0f0ba server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:20.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.294+0000 7ff69f7fe640 1 -- 172.21.15.116:0/3688301132 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6940b0070 con 0x7ff6a0150c90 2024-04-25T00:27:20.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.294+0000 7ff69f7fe640 1 -- 172.21.15.116:0/3688301132 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6940ab600 con 0x7ff6a0150c90 2024-04-25T00:27:20.296 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.294+0000 7ff69f7fe640 1 -- 172.21.15.116:0/3688301132 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6940ab9d0 con 0x7ff6a0150c90 2024-04-25T00:27:20.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.298+0000 7ff6a71a0640 1 -- 172.21.15.116:0/3688301132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6a0150c90 msgr2=0x7ff6a00c72a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:20.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.298+0000 7ff6a71a0640 1 --2- 172.21.15.116:0/3688301132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6a0150c90 0x7ff6a00c72a0 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7ff6940097c0 tx=0x7ff69401c340 comp rx=0 tx=0).stop 2024-04-25T00:27:20.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.298+0000 7ff6a71a0640 1 -- 172.21.15.116:0/3688301132 shutdown_connections 2024-04-25T00:27:20.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.298+0000 7ff6a71a0640 1 --2- 172.21.15.116:0/3688301132 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff6a00c77e0 0x7ff6a00c3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:20.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.298+0000 7ff6a71a0640 1 --2- 172.21.15.116:0/3688301132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6a0150c90 0x7ff6a00c72a0 secure :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7ff6940097c0 tx=0x7ff69401c340 comp rx=0 tx=0).stop 2024-04-25T00:27:20.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.298+0000 7ff6a71a0640 1 -- 172.21.15.116:0/3688301132 >> 172.21.15.116:0/3688301132 conn(0x7ff6a00a94a0 msgr2=0x7ff6a00a98b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:27:20.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.298+0000 7ff6a71a0640 1 -- 172.21.15.116:0/3688301132 shutdown_connections 2024-04-25T00:27:20.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.298+0000 7ff6a71a0640 1 -- 172.21.15.116:0/3688301132 wait complete. 2024-04-25T00:27:20.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.298+0000 7ff6a71a0640 1 Processor -- start 2024-04-25T00:27:20.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.302+0000 7ff6a71a0640 1 -- start start 2024-04-25T00:27:20.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.302+0000 7ff6a71a0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6a006f880 0x7ff6a0142cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:20.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.302+0000 7ff6a71a0640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff6a00c77e0 0x7ff6a00c6c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:20.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.302+0000 7ff6a71a0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6a00c5320 0x7ff6a014b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:20.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.302+0000 7ff6a71a0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a00c1d10 con 0x7ff6a00c5320 2024-04-25T00:27:20.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.302+0000 7ff6a71a0640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a00c48a0 con 0x7ff6a006f880 2024-04-25T00:27:20.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.302+0000 7ff6a71a0640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff6a01608f0 con 0x7ff6a00c77e0 2024-04-25T00:27:20.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.302+0000 7ff6a573d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6a00c5320 0x7ff6a014b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:20.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.302+0000 7ff6a573d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6a00c5320 0x7ff6a014b670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46534/0 (socket says 172.21.15.116:46534) 2024-04-25T00:27:20.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.302+0000 7ff6a573d640 1 -- 172.21.15.116:0/1853462952 learned_addr learned my addr 172.21.15.116:0/1853462952 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:27:20.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.302+0000 7ff6a4f3c640 1 --2- 172.21.15.116:0/1853462952 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6a006f880 0x7ff6a0142cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:20.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.302+0000 7ff69ffff640 1 --2- 172.21.15.116:0/1853462952 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff6a00c77e0 0x7ff6a00c6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:20.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.302+0000 7ff6a573d640 1 -- 172.21.15.116:0/1853462952 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff6a00c77e0 msgr2=0x7ff6a00c6c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:20.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.302+0000 7ff6a573d640 1 --2- 172.21.15.116:0/1853462952 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff6a00c77e0 0x7ff6a00c6c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:20.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.302+0000 7ff6a573d640 1 -- 172.21.15.116:0/1853462952 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6a006f880 msgr2=0x7ff6a0142cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:20.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.302+0000 7ff6a573d640 1 --2- 172.21.15.116:0/1853462952 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6a006f880 0x7ff6a0142cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:20.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.302+0000 7ff6a573d640 1 -- 172.21.15.116:0/1853462952 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6a014c360 con 0x7ff6a00c5320 2024-04-25T00:27:20.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.302+0000 7ff6a4f3c640 1 --2- 172.21.15.116:0/1853462952 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6a006f880 0x7ff6a0142cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:27:20.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.302+0000 7ff6a573d640 1 --2- 172.21.15.116:0/1853462952 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6a00c5320 0x7ff6a014b670 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7ff69800c7b0 tx=0x7ff69800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:20.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.302+0000 7ff69d7fa640 1 -- 172.21.15.116:0/1853462952 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff698017070 con 0x7ff6a00c5320 2024-04-25T00:27:20.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.302+0000 7ff69d7fa640 1 -- 172.21.15.116:0/1853462952 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff698003db0 con 0x7ff6a00c5320 2024-04-25T00:27:20.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.302+0000 7ff69d7fa640 1 -- 172.21.15.116:0/1853462952 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6980052b0 con 0x7ff6a00c5320 2024-04-25T00:27:20.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.302+0000 7ff6a71a0640 1 -- 172.21.15.116:0/1853462952 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6a014c640 con 0x7ff6a00c5320 2024-04-25T00:27:20.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.302+0000 7ff6a71a0640 1 -- 172.21.15.116:0/1853462952 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6a014b150 con 0x7ff6a00c5320 2024-04-25T00:27:20.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.306+0000 7ff68affd640 1 -- 172.21.15.116:0/1853462952 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6780052a0 con 0x7ff6a00c5320 2024-04-25T00:27:20.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.306+0000 7ff69d7fa640 1 -- 172.21.15.116:0/1853462952 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff698005450 con 0x7ff6a00c5320 2024-04-25T00:27:20.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.306+0000 7ff69d7fa640 1 --2- 172.21.15.116:0/1853462952 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff6840678f0 0x7ff684069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:20.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.306+0000 7ff69d7fa640 1 -- 172.21.15.116:0/1853462952 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 10278+0+0 (secure 0 0 0) 0x7ff6980862b0 con 0x7ff6a00c5320 2024-04-25T00:27:20.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.310+0000 7ff6a4f3c640 1 --2- 172.21.15.116:0/1853462952 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff6840678f0 0x7ff684069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:20.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.310+0000 7ff6a4f3c640 1 --2- 172.21.15.116:0/1853462952 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff6840678f0 0x7ff684069db0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7ff6940ad000 tx=0x7ff694008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:20.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.310+0000 7ff69d7fa640 1 -- 172.21.15.116:0/1853462952 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ff698055600 con 0x7ff6a00c5320 2024-04-25T00:27:20.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.586+0000 7ff68affd640 1 -- 172.21.15.116:0/1853462952 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff678005590 con 0x7ff6a00c5320 2024-04-25T00:27:20.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.586+0000 7ff69d7fa640 1 -- 172.21.15.116:0/1853462952 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 221 v221) ==== 95+0+5680 (secure 0 0 0) 0x7ff698059900 con 0x7ff6a00c5320 2024-04-25T00:27:20.590 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:27:20.590 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":221,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":211,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:26:19.034064+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6485},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6485":{"gid":6485,"name":"a","rank":0,"incarnation":208,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6835/2371126056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2371126056},{"type":"v1","addr":"172.21.15.116:6835","nonce":2371126056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6485,"qdb_cluster":[6485]},"id":1},{"mdsmap":{"epoch":221,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:27:19.052286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8652":{"gid":8652,"name":"c","rank":0,"incarnation":218,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6833/3891103804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3891103804},{"type":"v1","addr":"172.21.15.116:6833","nonce":3891103804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8652,"qdb_cluster":[8652]},"id":2}]} 2024-04-25T00:27:20.590 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 221 2024-04-25T00:27:20.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.590+0000 7ff68affd640 1 -- 172.21.15.116:0/1853462952 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff6840678f0 msgr2=0x7ff684069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:20.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.590+0000 7ff68affd640 1 --2- 172.21.15.116:0/1853462952 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff6840678f0 0x7ff684069db0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7ff6940ad000 tx=0x7ff694008040 comp rx=0 tx=0).stop 2024-04-25T00:27:20.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.590+0000 7ff68affd640 1 -- 172.21.15.116:0/1853462952 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6a00c5320 msgr2=0x7ff6a014b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:20.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.590+0000 7ff68affd640 1 --2- 172.21.15.116:0/1853462952 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6a00c5320 0x7ff6a014b670 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7ff69800c7b0 tx=0x7ff69800cc70 comp rx=0 tx=0).stop 2024-04-25T00:27:20.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.590+0000 7ff68affd640 1 -- 172.21.15.116:0/1853462952 shutdown_connections 2024-04-25T00:27:20.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.590+0000 7ff68affd640 1 --2- 172.21.15.116:0/1853462952 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff6840678f0 0x7ff684069db0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:20.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.590+0000 7ff68affd640 1 --2- 172.21.15.116:0/1853462952 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff6a00c5320 0x7ff6a014b670 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:20.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.590+0000 7ff68affd640 1 --2- 172.21.15.116:0/1853462952 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff6a00c77e0 0x7ff6a00c6c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:20.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.590+0000 7ff68affd640 1 --2- 172.21.15.116:0/1853462952 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff6a006f880 0x7ff6a0142cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:20.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.590+0000 7ff68affd640 1 -- 172.21.15.116:0/1853462952 >> 172.21.15.116:0/1853462952 conn(0x7ff6a00a94a0 msgr2=0x7ff6a00c3a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:27:20.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.590+0000 7ff68affd640 1 -- 172.21.15.116:0/1853462952 shutdown_connections 2024-04-25T00:27:20.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.590+0000 7ff68affd640 1 -- 172.21.15.116:0/1853462952 wait complete. 2024-04-25T00:27:20.613 INFO:tasks.mds_thrash.fs.[a]:kill mds.a (rank=0) 2024-04-25T00:27:20.614 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-25T00:27:20.614 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:27:20.629 INFO:tasks.ceph.mds.a:Stopped 2024-04-25T00:27:20.629 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:27:20.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.814+0000 7f521ddc1640 1 Processor -- start 2024-04-25T00:27:20.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.814+0000 7f521ddc1640 1 -- start start 2024-04-25T00:27:20.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.814+0000 7f521ddc1640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f52180aeb20 0x7f52180c9310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:20.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.814+0000 7f521ddc1640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5218150f80 0x7f52180c5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:20.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.814+0000 7f521ddc1640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52181535d0 con 0x7f52180c9850 2024-04-25T00:27:20.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.814+0000 7f521ddc1640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f5218153850 con 0x7f5218150f80 2024-04-25T00:27:20.815 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.814+0000 7f521ddc1640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f5218153b30 con 0x7f52180aeb20 2024-04-25T00:27:20.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.814+0000 7f5217fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5218150f80 0x7f52180c5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:20.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.814+0000 7f5217fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5218150f80 0x7f52180c5270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46538/0 (socket says 172.21.15.116:46538) 2024-04-25T00:27:20.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.814+0000 7f5217fff640 1 -- 172.21.15.116:0/2249450856 learned_addr learned my addr 172.21.15.116:0/2249450856 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:27:20.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.814+0000 7f52177fe640 1 --2- 172.21.15.116:0/2249450856 >> v2:172.21.15.174:3300/0 conn(0x7f52180aeb20 0x7f52180c9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:20.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.814+0000 7f5217fff640 1 -- 172.21.15.116:0/2249450856 >> v2:172.21.15.174:3300/0 conn(0x7f52180aeb20 msgr2=0x7f52180c9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:27:20.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.814+0000 7f5217fff640 1 --2- 172.21.15.116:0/2249450856 >> v2:172.21.15.174:3300/0 conn(0x7f52180aeb20 0x7f52180c9310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:20.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.814+0000 7f5217fff640 1 -- 172.21.15.116:0/2249450856 >> v1:172.21.15.116:6789/0 conn(0x7f52180c9850 legacy=0x7f5218150870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:20.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.814+0000 7f5216ffd640 1 -- 172.21.15.116:0/2249450856 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 500313283 0 0) 0x7f5200001120 con 0x7f52180c9850 2024-04-25T00:27:20.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.814+0000 7f5217fff640 1 -- 172.21.15.116:0/2249450856 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5218153e10 con 0x7f5218150f80 2024-04-25T00:27:20.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.818+0000 7f5217fff640 1 --2- 172.21.15.116:0/2249450856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5218150f80 0x7f52180c5270 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f52180705f0 tx=0x7f521001c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a50e904c2a1a990 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:20.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.818+0000 7f5216ffd640 1 -- 172.21.15.116:0/2249450856 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52100a5680 con 0x7f5218150f80 2024-04-25T00:27:20.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.818+0000 7f5216ffd640 1 -- 172.21.15.116:0/2249450856 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52100a5820 con 0x7f5218150f80 2024-04-25T00:27:20.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.818+0000 7f5216ffd640 1 -- 172.21.15.116:0/2249450856 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52100a5bf0 con 0x7f5218150f80 2024-04-25T00:27:20.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.822+0000 7f521ddc1640 1 -- 172.21.15.116:0/2249450856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5218150f80 msgr2=0x7f52180c5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:20.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.822+0000 7f521ddc1640 1 --2- 172.21.15.116:0/2249450856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5218150f80 0x7f52180c5270 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f52180705f0 tx=0x7f521001c8d0 comp rx=0 tx=0).stop 2024-04-25T00:27:20.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.822+0000 7f521ddc1640 1 -- 172.21.15.116:0/2249450856 shutdown_connections 2024-04-25T00:27:20.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.822+0000 7f521ddc1640 1 --2- 172.21.15.116:0/2249450856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5218150f80 0x7f52180c5270 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:20.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.822+0000 7f521ddc1640 1 --2- 172.21.15.116:0/2249450856 >> v2:172.21.15.174:3300/0 conn(0x7f52180aeb20 0x7f52180c9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:20.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.822+0000 7f521ddc1640 1 -- 172.21.15.116:0/2249450856 >> 172.21.15.116:0/2249450856 conn(0x7f5218074650 msgr2=0x7f5218074a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:27:20.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.822+0000 7f521ddc1640 1 -- 172.21.15.116:0/2249450856 shutdown_connections 2024-04-25T00:27:20.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.822+0000 7f521ddc1640 1 -- 172.21.15.116:0/2249450856 wait complete. 2024-04-25T00:27:20.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.822+0000 7f521ddc1640 1 Processor -- start 2024-04-25T00:27:20.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.822+0000 7f521ddc1640 1 -- start start 2024-04-25T00:27:20.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.822+0000 7f521ddc1640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f52180aeb20 0x7f52180ae2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:20.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.822+0000 7f521ddc1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52180c9850 0x7f52180ad610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:20.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.822+0000 7f521ddc1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5218150f80 0x7f5218144d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:20.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.822+0000 7f521ddc1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f521815ff60 con 0x7f5218150f80 2024-04-25T00:27:20.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.822+0000 7f521ddc1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f52180af150 con 0x7f52180c9850 2024-04-25T00:27:20.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.822+0000 7f521ddc1640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f52181600e0 con 0x7f52180aeb20 2024-04-25T00:27:20.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.822+0000 7f5217fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5218150f80 0x7f5218144d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:20.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.822+0000 7f5217fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5218150f80 0x7f5218144d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46544/0 (socket says 172.21.15.116:46544) 2024-04-25T00:27:20.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.822+0000 7f5217fff640 1 -- 172.21.15.116:0/2534083978 learned_addr learned my addr 172.21.15.116:0/2534083978 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:27:20.826 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.822+0000 7f52177fe640 1 --2- 172.21.15.116:0/2534083978 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f52180aeb20 0x7f52180ae2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:20.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.826+0000 7f52177fe640 1 -- 172.21.15.116:0/2534083978 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52180c9850 msgr2=0x7f52180ad610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:20.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.826+0000 7f520ffff640 1 --2- 172.21.15.116:0/2534083978 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52180c9850 0x7f52180ad610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:20.829 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.826+0000 7f52177fe640 1 --2- 172.21.15.116:0/2534083978 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52180c9850 0x7f52180ad610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:20.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.826+0000 7f52177fe640 1 -- 172.21.15.116:0/2534083978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5218150f80 msgr2=0x7f5218144d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:20.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.826+0000 7f52177fe640 1 --2- 172.21.15.116:0/2534083978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5218150f80 0x7f5218144d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:20.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.826+0000 7f52177fe640 1 -- 172.21.15.116:0/2534083978 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52181eba50 con 0x7f52180aeb20 2024-04-25T00:27:20.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.826+0000 7f52177fe640 1 --2- 172.21.15.116:0/2534083978 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f52180aeb20 0x7f52180ae2d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f520800e880 tx=0x7f520800ed40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:20.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.826+0000 7f5214ff9640 1 -- 172.21.15.116:0/2534083978 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5208017070 con 0x7f52180aeb20 2024-04-25T00:27:20.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.826+0000 7f5214ff9640 1 -- 172.21.15.116:0/2534083978 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f520800c940 con 0x7f52180aeb20 2024-04-25T00:27:20.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.826+0000 7f521ddc1640 1 -- 172.21.15.116:0/2534083978 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52181ebd30 con 0x7f52180aeb20 2024-04-25T00:27:20.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.826+0000 7f5214ff9640 1 -- 172.21.15.116:0/2534083978 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f520800cd30 con 0x7f52180aeb20 2024-04-25T00:27:20.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.830+0000 7f521ddc1640 1 -- 172.21.15.116:0/2534083978 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f52181ec2f0 con 0x7f52180aeb20 2024-04-25T00:27:20.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.830+0000 7f521ddc1640 1 -- 172.21.15.116:0/2534083978 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5218068a60 con 0x7f52180aeb20 2024-04-25T00:27:20.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.830+0000 7f5214ff9640 1 -- 172.21.15.116:0/2534083978 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f520801a070 con 0x7f52180aeb20 2024-04-25T00:27:20.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.830+0000 7f5214ff9640 1 --2- 172.21.15.116:0/2534083978 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f51f80678c0 0x7f51f8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:20.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.830+0000 7f5214ff9640 1 -- 172.21.15.116:0/2534083978 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(70..70 src has 1..70) ==== 10278+0+0 (secure 0 0 0) 0x7f52080874e0 con 0x7f52180aeb20 2024-04-25T00:27:20.832 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.830+0000 7f520ffff640 1 --2- 172.21.15.116:0/2534083978 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f51f80678c0 0x7f51f8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:20.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.834+0000 7f520ffff640 1 --2- 172.21.15.116:0/2534083978 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f51f80678c0 0x7f51f8069d80 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f52180ae810 tx=0x7f5200009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:20.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:20.846+0000 7f5214ff9640 1 -- 172.21.15.116:0/2534083978 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f52080568b0 con 0x7f52180aeb20 2024-04-25T00:27:21.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:21.118+0000 7f521ddc1640 1 -- 172.21.15.116:0/2534083978 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f521806e530 con 0x7f52180aeb20 2024-04-25T00:27:21.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:21.118+0000 7f5214ff9640 1 -- 172.21.15.116:0/2534083978 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 221 v221) ==== 95+0+5680 (secure 0 0 0) 0x7f520805d0d0 con 0x7f52180aeb20 2024-04-25T00:27:21.118 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:27:21.118 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":221,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":211,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:26:19.034064+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6485},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6485":{"gid":6485,"name":"a","rank":0,"incarnation":208,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6835/2371126056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2371126056},{"type":"v1","addr":"172.21.15.116:6835","nonce":2371126056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6485,"qdb_cluster":[6485]},"id":1},{"mdsmap":{"epoch":221,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:27:19.052286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8652":{"gid":8652,"name":"c","rank":0,"incarnation":218,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6833/3891103804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3891103804},{"type":"v1","addr":"172.21.15.116:6833","nonce":3891103804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8652,"qdb_cluster":[8652]},"id":2}]} 2024-04-25T00:27:21.118 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 221 2024-04-25T00:27:21.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:21.122+0000 7f521ddc1640 1 -- 172.21.15.116:0/2534083978 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f51f80678c0 msgr2=0x7f51f8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:21.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:21.122+0000 7f521ddc1640 1 --2- 172.21.15.116:0/2534083978 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f51f80678c0 0x7f51f8069d80 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f52180ae810 tx=0x7f5200009040 comp rx=0 tx=0).stop 2024-04-25T00:27:21.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:21.122+0000 7f521ddc1640 1 -- 172.21.15.116:0/2534083978 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f52180aeb20 msgr2=0x7f52180ae2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:21.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:21.122+0000 7f521ddc1640 1 --2- 172.21.15.116:0/2534083978 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f52180aeb20 0x7f52180ae2d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f520800e880 tx=0x7f520800ed40 comp rx=0 tx=0).stop 2024-04-25T00:27:21.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:21.122+0000 7f521ddc1640 1 -- 172.21.15.116:0/2534083978 shutdown_connections 2024-04-25T00:27:21.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:21.122+0000 7f521ddc1640 1 --2- 172.21.15.116:0/2534083978 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f51f80678c0 0x7f51f8069d80 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:21.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:21.122+0000 7f521ddc1640 1 --2- 172.21.15.116:0/2534083978 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5218150f80 0x7f5218144d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:21.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:21.122+0000 7f521ddc1640 1 --2- 172.21.15.116:0/2534083978 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f52180c9850 0x7f52180ad610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:21.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:21.122+0000 7f521ddc1640 1 --2- 172.21.15.116:0/2534083978 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f52180aeb20 0x7f52180ae2d0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:21.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:21.122+0000 7f521ddc1640 1 -- 172.21.15.116:0/2534083978 >> 172.21.15.116:0/2534083978 conn(0x7f5218074650 msgr2=0x7f5218152bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:27:21.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:21.122+0000 7f521ddc1640 1 -- 172.21.15.116:0/2534083978 shutdown_connections 2024-04-25T00:27:21.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:21.122+0000 7f521ddc1640 1 -- 172.21.15.116:0/2534083978 wait complete. 2024-04-25T00:27:21.147 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:27:22.989 INFO:tasks.mds_thrash.fs.[b]:reviving mds.d 2024-04-25T00:27:22.990 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-25T00:27:22.990 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-25T00:27:22.992 INFO:tasks.ceph.mds.d:Started 2024-04-25T00:27:22.993 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:27:23.044 INFO:tasks.ceph.mds.c.smithi116.stderr:2024-04-25T00:27:23.042+0000 7f09d008e640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T00:27:23.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.130+0000 7f1d57c13640 1 Processor -- start 2024-04-25T00:27:23.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.130+0000 7f1d57c13640 1 -- start start 2024-04-25T00:27:23.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.130+0000 7f1d57c13640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1d5006d5f0 0x7f1d50156c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:23.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.130+0000 7f1d57c13640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d500c9150 con 0x7f1d50152d80 2024-04-25T00:27:23.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.130+0000 7f1d57c13640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d500c9430 con 0x7f1d50155810 2024-04-25T00:27:23.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.130+0000 7f1d57c13640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f1d500c9d60 con 0x7f1d5006d5f0 2024-04-25T00:27:23.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.130+0000 7f1d559af640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f1d50152d80 0x7f1d5006bd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:37194/0 (socket says 172.21.15.116:37194) 2024-04-25T00:27:23.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.130+0000 7f1d559af640 1 -- 172.21.15.116:0/524729163 learned_addr learned my addr 172.21.15.116:0/524729163 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:27:23.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.130+0000 7f1d551ae640 1 --2- 172.21.15.116:0/524729163 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1d5006d5f0 0x7f1d50156c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:23.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.130+0000 7f1d549ad640 1 -- 172.21.15.116:0/524729163 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3931080206 0 0) 0x7f1d500c9150 con 0x7f1d50152d80 2024-04-25T00:27:23.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.130+0000 7f1d549ad640 1 -- 172.21.15.116:0/524729163 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d3c003530 con 0x7f1d50152d80 2024-04-25T00:27:23.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.130+0000 7f1d549ad640 1 -- 172.21.15.116:0/524729163 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 319406528 0 0) 0x7f1d3c003530 con 0x7f1d50152d80 2024-04-25T00:27:23.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.130+0000 7f1d549ad640 1 -- 172.21.15.116:0/524729163 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1d5006d5f0 msgr2=0x7f1d50156c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:27:23.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.130+0000 7f1d549ad640 1 --2- 172.21.15.116:0/524729163 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1d5006d5f0 0x7f1d50156c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:23.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.130+0000 7f1d549ad640 1 -- 172.21.15.116:0/524729163 >> v1:172.21.15.174:6789/0 conn(0x7f1d50155810 legacy=0x7f1d501571c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:23.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.130+0000 7f1d549ad640 1 -- 172.21.15.116:0/524729163 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d500c9710 con 0x7f1d50152d80 2024-04-25T00:27:23.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.130+0000 7f1d549ad640 1 -- 172.21.15.116:0/524729163 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f1d4c001680 con 0x7f1d50152d80 2024-04-25T00:27:23.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.130+0000 7f1d549ad640 1 -- 172.21.15.116:0/524729163 >> v1:172.21.15.116:6789/0 conn(0x7f1d50152d80 legacy=0x7f1d5006bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:23.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.130+0000 7f1d549ad640 1 --2- 172.21.15.116:0/524729163 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1d3c0039f0 0x7f1d3c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:23.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.130+0000 7f1d549ad640 1 --2- 172.21.15.116:0/524729163 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1d5006d5f0 0x7f1d3c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:23.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.130+0000 7f1d549ad640 1 --2- 172.21.15.116:0/524729163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d3c004d50 0x7f1d3c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:23.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.130+0000 7f1d549ad640 1 -- 172.21.15.116:0/524729163 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1d500c9d60 con 0x7f1d3c004d50 2024-04-25T00:27:23.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.130+0000 7f1d549ad640 1 -- 172.21.15.116:0/524729163 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1d3c01cba0 con 0x7f1d3c0039f0 2024-04-25T00:27:23.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.130+0000 7f1d549ad640 1 -- 172.21.15.116:0/524729163 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f1d3c01cda0 con 0x7f1d5006d5f0 2024-04-25T00:27:23.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.130+0000 7f1d559af640 1 --2- 172.21.15.116:0/524729163 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1d3c0039f0 0x7f1d3c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:23.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.130+0000 7f1d561b0640 1 --2- 172.21.15.116:0/524729163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d3c004d50 0x7f1d3c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:23.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.130+0000 7f1d551ae640 1 --2- 172.21.15.116:0/524729163 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1d5006d5f0 0x7f1d3c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:23.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.134+0000 7f1d561b0640 1 -- 172.21.15.116:0/524729163 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1d5006d5f0 msgr2=0x7f1d3c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:27:23.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.134+0000 7f1d561b0640 1 --2- 172.21.15.116:0/524729163 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1d5006d5f0 0x7f1d3c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:23.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.134+0000 7f1d561b0640 1 -- 172.21.15.116:0/524729163 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1d3c0039f0 msgr2=0x7f1d3c003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:27:23.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.134+0000 7f1d561b0640 1 --2- 172.21.15.116:0/524729163 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1d3c0039f0 0x7f1d3c003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:23.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.134+0000 7f1d561b0640 1 -- 172.21.15.116:0/524729163 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d3c003530 con 0x7f1d3c004d50 2024-04-25T00:27:23.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.134+0000 7f1d561b0640 1 --2- 172.21.15.116:0/524729163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d3c004d50 0x7f1d3c0051d0 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f1d48001e70 tx=0x7f1d4801c750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39a4c886fc3fca73 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:23.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.134+0000 7f1d549ad640 1 -- 172.21.15.116:0/524729163 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d48001a90 con 0x7f1d3c004d50 2024-04-25T00:27:23.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.134+0000 7f1d549ad640 1 -- 172.21.15.116:0/524729163 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d48003b80 con 0x7f1d3c004d50 2024-04-25T00:27:23.136 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.134+0000 7f1d549ad640 1 -- 172.21.15.116:0/524729163 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d48005080 con 0x7f1d3c004d50 2024-04-25T00:27:23.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.134+0000 7f1d57c13640 1 -- 172.21.15.116:0/524729163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d3c004d50 msgr2=0x7f1d3c0051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:23.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.134+0000 7f1d57c13640 1 --2- 172.21.15.116:0/524729163 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d3c004d50 0x7f1d3c0051d0 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f1d48001e70 tx=0x7f1d4801c750 comp rx=0 tx=0).stop 2024-04-25T00:27:23.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.138+0000 7f1d561b0640 1 -- 172.21.15.116:0/524729163 reap_dead start 2024-04-25T00:27:23.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.138+0000 7f1d57c13640 1 -- 172.21.15.116:0/524729163 shutdown_connections 2024-04-25T00:27:23.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.138+0000 7f1d57c13640 1 -- 172.21.15.116:0/524729163 >> 172.21.15.116:0/524729163 conn(0x7f1d500acdd0 msgr2=0x7f1d500ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:27:23.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.138+0000 7f1d57c13640 1 -- 172.21.15.116:0/524729163 shutdown_connections 2024-04-25T00:27:23.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.138+0000 7f1d57c13640 1 -- 172.21.15.116:0/524729163 wait complete. 2024-04-25T00:27:23.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.138+0000 7f1d57c13640 1 Processor -- start 2024-04-25T00:27:23.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.138+0000 7f1d57c13640 1 -- start start 2024-04-25T00:27:23.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.142+0000 7f1d57c13640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1d500c66c0 0x7f1d50153680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:23.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.142+0000 7f1d57c13640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1d50151750 0x7f1d500c5930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:23.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.142+0000 7f1d57c13640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d50151bd0 0x7f1d5014b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:23.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.142+0000 7f1d57c13640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1d50153160 con 0x7f1d50151bd0 2024-04-25T00:27:23.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.142+0000 7f1d57c13640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1d500cad80 con 0x7f1d500c66c0 2024-04-25T00:27:23.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.142+0000 7f1d57c13640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f1d501eab20 con 0x7f1d50151750 2024-04-25T00:27:23.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.142+0000 7f1d561b0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d50151bd0 0x7f1d5014b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:23.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.142+0000 7f1d559af640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1d500c66c0 0x7f1d50153680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:23.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.142+0000 7f1d551ae640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1d50151750 0x7f1d500c5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:23.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.142+0000 7f1d561b0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d50151bd0 0x7f1d5014b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46572/0 (socket says 172.21.15.116:46572) 2024-04-25T00:27:23.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.142+0000 7f1d561b0640 1 -- 172.21.15.116:0/1112696724 learned_addr learned my addr 172.21.15.116:0/1112696724 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:27:23.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.142+0000 7f1d559af640 1 -- 172.21.15.116:0/1112696724 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1d50151750 msgr2=0x7f1d500c5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:23.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.142+0000 7f1d559af640 1 --2- 172.21.15.116:0/1112696724 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1d50151750 0x7f1d500c5930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:23.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.142+0000 7f1d559af640 1 -- 172.21.15.116:0/1112696724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d50151bd0 msgr2=0x7f1d5014b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:23.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.142+0000 7f1d559af640 1 --2- 172.21.15.116:0/1112696724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d50151bd0 0x7f1d5014b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:23.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.142+0000 7f1d559af640 1 -- 172.21.15.116:0/1112696724 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d501eae00 con 0x7f1d500c66c0 2024-04-25T00:27:23.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.142+0000 7f1d559af640 1 --2- 172.21.15.116:0/1112696724 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1d500c66c0 0x7f1d50153680 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f1d4c000c10 tx=0x7f1d4c007430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:23.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.142+0000 7f1d467fc640 1 -- 172.21.15.116:0/1112696724 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d4c0319c0 con 0x7f1d500c66c0 2024-04-25T00:27:23.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.142+0000 7f1d467fc640 1 -- 172.21.15.116:0/1112696724 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d4c03a070 con 0x7f1d500c66c0 2024-04-25T00:27:23.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.142+0000 7f1d57c13640 1 -- 172.21.15.116:0/1112696724 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d50150c70 con 0x7f1d500c66c0 2024-04-25T00:27:23.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.142+0000 7f1d467fc640 1 -- 172.21.15.116:0/1112696724 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d4c031dc0 con 0x7f1d500c66c0 2024-04-25T00:27:23.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.142+0000 7f1d57c13640 1 -- 172.21.15.116:0/1112696724 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d50151230 con 0x7f1d500c66c0 2024-04-25T00:27:23.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.146+0000 7f1d57c13640 1 -- 172.21.15.116:0/1112696724 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d50068350 con 0x7f1d500c66c0 2024-04-25T00:27:23.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.146+0000 7f1d467fc640 1 -- 172.21.15.116:0/1112696724 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1d4c034070 con 0x7f1d500c66c0 2024-04-25T00:27:23.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.146+0000 7f1d467fc640 1 --2- 172.21.15.116:0/1112696724 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1d340678c0 0x7f1d34069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:23.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.146+0000 7f1d467fc640 1 -- 172.21.15.116:0/1112696724 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 10278+0+0 (secure 0 0 0) 0x7f1d4c0a87d0 con 0x7f1d500c66c0 2024-04-25T00:27:23.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.146+0000 7f1d551ae640 1 --2- 172.21.15.116:0/1112696724 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1d340678c0 0x7f1d34069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:23.148 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:27:23.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.150+0000 7f1d551ae640 1 --2- 172.21.15.116:0/1112696724 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1d340678c0 0x7f1d34069d80 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f1d40011fd0 tx=0x7f1d40015040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:23.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.154+0000 7f1d467fc640 1 -- 172.21.15.116:0/1112696724 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f1d4c078170 con 0x7f1d500c66c0 2024-04-25T00:27:23.236 INFO:tasks.ceph.mds.d.smithi174.stdout:starting mds.d at 2024-04-25T00:27:23.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.298+0000 7fd6474e8640 1 Processor -- start 2024-04-25T00:27:23.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.298+0000 7fd6474e8640 1 -- start start 2024-04-25T00:27:23.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.298+0000 7fd6474e8640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd6400c2a10 0x7fd6400c2df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:23.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.298+0000 7fd6474e8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd6400c36c0 0x7fd640162ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:23.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.298+0000 7fd6474e8640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fd640163000 0x7fd640165a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:23.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.298+0000 7fd6474e8640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fd6400712b0 con 0x7fd6400c36c0 2024-04-25T00:27:23.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.298+0000 7fd6474e8640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fd640070fd0 con 0x7fd640163000 2024-04-25T00:27:23.302 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.298+0000 7fd6474e8640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fd64006e830 con 0x7fd6400c2a10 2024-04-25T00:27:23.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.298+0000 7fd644a83640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd6400c36c0 0x7fd640162ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:23.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.298+0000 7fd644a83640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd6400c36c0 0x7fd640162ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46586/0 (socket says 172.21.15.116:46586) 2024-04-25T00:27:23.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.298+0000 7fd644a83640 1 -- 172.21.15.116:0/269447082 learned_addr learned my addr 172.21.15.116:0/269447082 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:27:23.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.298+0000 7fd645284640 1 --2- 172.21.15.116:0/269447082 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd6400c2a10 0x7fd6400c2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:23.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.298+0000 7fd645a85640 1 --2- 172.21.15.116:0/269447082 >> v2:172.21.15.174:3300/0 conn(0x7fd640163000 0x7fd640165a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:23.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.302+0000 7fd644a83640 1 -- 172.21.15.116:0/269447082 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd6400c2a10 msgr2=0x7fd6400c2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:27:23.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.302+0000 7fd644a83640 1 --2- 172.21.15.116:0/269447082 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd6400c2a10 0x7fd6400c2df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:23.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.302+0000 7fd644a83640 1 -- 172.21.15.116:0/269447082 >> v2:172.21.15.174:3300/0 conn(0x7fd640163000 msgr2=0x7fd640165a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:27:23.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.302+0000 7fd644a83640 1 --2- 172.21.15.116:0/269447082 >> v2:172.21.15.174:3300/0 conn(0x7fd640163000 0x7fd640165a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:23.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.302+0000 7fd644a83640 1 -- 172.21.15.116:0/269447082 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd64006eb10 con 0x7fd6400c36c0 2024-04-25T00:27:23.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.302+0000 7fd645a85640 1 --2- 172.21.15.116:0/269447082 >> v2:172.21.15.174:3300/0 conn(0x7fd640163000 0x7fd640165a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:27:23.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.306+0000 7fd644a83640 1 --2- 172.21.15.116:0/269447082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6400c36c0 0x7fd640162ac0 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7fd64006c1c0 tx=0x7fd63c01c690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb379e81943324cf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:23.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.306+0000 7fd637fff640 1 -- 172.21.15.116:0/269447082 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd63c0ac070 con 0x7fd6400c36c0 2024-04-25T00:27:23.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.306+0000 7fd637fff640 1 -- 172.21.15.116:0/269447082 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd63c0a6610 con 0x7fd6400c36c0 2024-04-25T00:27:23.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.306+0000 7fd637fff640 1 -- 172.21.15.116:0/269447082 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd63c0a69e0 con 0x7fd6400c36c0 2024-04-25T00:27:23.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.306+0000 7fd6474e8640 1 -- 172.21.15.116:0/269447082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6400c36c0 msgr2=0x7fd640162ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:23.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.306+0000 7fd6474e8640 1 --2- 172.21.15.116:0/269447082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6400c36c0 0x7fd640162ac0 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7fd64006c1c0 tx=0x7fd63c01c690 comp rx=0 tx=0).stop 2024-04-25T00:27:23.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.310+0000 7fd6474e8640 1 -- 172.21.15.116:0/269447082 shutdown_connections 2024-04-25T00:27:23.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.310+0000 7fd6474e8640 1 --2- 172.21.15.116:0/269447082 >> v2:172.21.15.174:3300/0 conn(0x7fd640163000 0x7fd640165a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:23.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.310+0000 7fd6474e8640 1 --2- 172.21.15.116:0/269447082 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd6400c36c0 0x7fd640162ac0 unknown :-1 s=CLOSED pgs=1618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:23.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.310+0000 7fd6474e8640 1 --2- 172.21.15.116:0/269447082 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd6400c2a10 0x7fd6400c2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:23.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.310+0000 7fd6474e8640 1 -- 172.21.15.116:0/269447082 >> 172.21.15.116:0/269447082 conn(0x7fd6400c3b00 msgr2=0x7fd6400c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:27:23.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.314+0000 7fd6474e8640 1 -- 172.21.15.116:0/269447082 shutdown_connections 2024-04-25T00:27:23.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.314+0000 7fd6474e8640 1 -- 172.21.15.116:0/269447082 wait complete. 2024-04-25T00:27:23.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.314+0000 7fd6474e8640 1 Processor -- start 2024-04-25T00:27:23.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.314+0000 7fd6474e8640 1 -- start start 2024-04-25T00:27:23.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.318+0000 7fd6474e8640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd6400c2a10 0x7fd640160ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:23.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.318+0000 7fd6474e8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd640163000 0x7fd640160ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:23.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.318+0000 7fd6474e8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd6400a9d40 0x7fd6400ac720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:23.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.318+0000 7fd6474e8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd6400718b0 con 0x7fd640163000 2024-04-25T00:27:23.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.318+0000 7fd6474e8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd640071730 con 0x7fd6400a9d40 2024-04-25T00:27:23.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.318+0000 7fd6474e8640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fd64006cb50 con 0x7fd6400c2a10 2024-04-25T00:27:23.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.318+0000 7fd645284640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd6400c2a10 0x7fd640160ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:23.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.318+0000 7fd645284640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd6400c2a10 0x7fd640160ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:44794/0 (socket says 172.21.15.116:44794) 2024-04-25T00:27:23.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.318+0000 7fd645284640 1 -- 172.21.15.116:0/2248778104 learned_addr learned my addr 172.21.15.116:0/2248778104 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:27:23.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.318+0000 7fd645284640 1 -- 172.21.15.116:0/2248778104 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd6400a9d40 msgr2=0x7fd6400ac720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:23.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.318+0000 7fd645284640 1 --2- 172.21.15.116:0/2248778104 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd6400a9d40 0x7fd6400ac720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:23.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.318+0000 7fd645284640 1 -- 172.21.15.116:0/2248778104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd640163000 msgr2=0x7fd640160ff0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T00:27:23.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.318+0000 7fd645284640 1 --2- 172.21.15.116:0/2248778104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd640163000 0x7fd640160ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:23.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.318+0000 7fd645284640 1 -- 172.21.15.116:0/2248778104 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd64006df50 con 0x7fd6400c2a10 2024-04-25T00:27:23.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.322+0000 7fd645284640 1 --2- 172.21.15.116:0/2248778104 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd6400c2a10 0x7fd640160ab0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd63000cdd0 tx=0x7fd63000a720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:23.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.322+0000 7fd635ffb640 1 -- 172.21.15.116:0/2248778104 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd63001d070 con 0x7fd6400c2a10 2024-04-25T00:27:23.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.322+0000 7fd6474e8640 1 -- 172.21.15.116:0/2248778104 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6400ab480 con 0x7fd6400c2a10 2024-04-25T00:27:23.323 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.322+0000 7fd6474e8640 1 -- 172.21.15.116:0/2248778104 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd64015ec00 con 0x7fd6400c2a10 2024-04-25T00:27:23.324 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.322+0000 7fd6474e8640 1 -- 172.21.15.116:0/2248778104 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd640068350 con 0x7fd6400c2a10 2024-04-25T00:27:23.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.334+0000 7fd635ffb640 1 -- 172.21.15.116:0/2248778104 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd63000e240 con 0x7fd6400c2a10 2024-04-25T00:27:23.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.334+0000 7fd635ffb640 1 -- 172.21.15.116:0/2248778104 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd630018440 con 0x7fd6400c2a10 2024-04-25T00:27:23.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.334+0000 7fd635ffb640 1 -- 172.21.15.116:0/2248778104 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd630020070 con 0x7fd6400c2a10 2024-04-25T00:27:23.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.334+0000 7fd635ffb640 1 --2- 172.21.15.116:0/2248778104 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd6240678c0 0x7fd624069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:23.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.334+0000 7fd635ffb640 1 -- 172.21.15.116:0/2248778104 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(70..70 src has 1..70) ==== 10278+0+0 (secure 0 0 0) 0x7fd630087bb0 con 0x7fd6400c2a10 2024-04-25T00:27:23.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.334+0000 7fd644a83640 1 --2- 172.21.15.116:0/2248778104 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd6240678c0 0x7fd624069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:23.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.338+0000 7fd644a83640 1 --2- 172.21.15.116:0/2248778104 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd6240678c0 0x7fd624069d80 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fd63c0ae000 tx=0x7fd63c0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:23.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.342+0000 7fd635ffb640 1 -- 172.21.15.116:0/2248778104 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fd630027070 con 0x7fd6400c2a10 2024-04-25T00:27:23.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.470+0000 7f1d57c13640 1 -- 172.21.15.116:0/1112696724 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1d50152540 con 0x7f1d500c66c0 2024-04-25T00:27:23.472 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.470+0000 7f1d467fc640 1 -- 172.21.15.116:0/1112696724 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 222 v222) ==== 95+0+6509 (secure 0 0 0) 0x7f1d4c077a70 con 0x7f1d500c66c0 2024-04-25T00:27:23.474 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:27:23.474 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":222,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6641,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/2915274073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2915274073},{"type":"v1","addr":"172.21.15.174:6837","nonce":2915274073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":222}],"filesystems":[{"mdsmap":{"epoch":211,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:26:19.034064+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6485},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6485":{"gid":6485,"name":"a","rank":0,"incarnation":208,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6835/2371126056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2371126056},{"type":"v1","addr":"172.21.15.116:6835","nonce":2371126056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6485,"qdb_cluster":[6485]},"id":1},{"mdsmap":{"epoch":221,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:27:19.052286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8652":{"gid":8652,"name":"c","rank":0,"incarnation":218,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6833/3891103804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3891103804},{"type":"v1","addr":"172.21.15.116:6833","nonce":3891103804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8652,"qdb_cluster":[8652]},"id":2}]} 2024-04-25T00:27:23.474 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 222 2024-04-25T00:27:23.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.474+0000 7f1d57c13640 1 -- 172.21.15.116:0/1112696724 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1d340678c0 msgr2=0x7f1d34069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:23.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.474+0000 7f1d57c13640 1 --2- 172.21.15.116:0/1112696724 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1d340678c0 0x7f1d34069d80 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f1d40011fd0 tx=0x7f1d40015040 comp rx=0 tx=0).stop 2024-04-25T00:27:23.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.478+0000 7f1d57c13640 1 -- 172.21.15.116:0/1112696724 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1d500c66c0 msgr2=0x7f1d50153680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:23.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.478+0000 7f1d57c13640 1 --2- 172.21.15.116:0/1112696724 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1d500c66c0 0x7f1d50153680 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f1d4c000c10 tx=0x7f1d4c007430 comp rx=0 tx=0).stop 2024-04-25T00:27:23.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.478+0000 7f1d57c13640 1 -- 172.21.15.116:0/1112696724 shutdown_connections 2024-04-25T00:27:23.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.478+0000 7f1d57c13640 1 --2- 172.21.15.116:0/1112696724 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1d340678c0 0x7f1d34069d80 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:23.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.478+0000 7f1d57c13640 1 --2- 172.21.15.116:0/1112696724 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1d50151bd0 0x7f1d5014b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:23.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.478+0000 7f1d57c13640 1 --2- 172.21.15.116:0/1112696724 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1d50151750 0x7f1d500c5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:23.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.478+0000 7f1d57c13640 1 --2- 172.21.15.116:0/1112696724 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1d500c66c0 0x7f1d50153680 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:23.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.478+0000 7f1d57c13640 1 -- 172.21.15.116:0/1112696724 >> 172.21.15.116:0/1112696724 conn(0x7f1d500acdd0 msgr2=0x7f1d500ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:27:23.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.478+0000 7f1d57c13640 1 -- 172.21.15.116:0/1112696724 shutdown_connections 2024-04-25T00:27:23.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.478+0000 7f1d57c13640 1 -- 172.21.15.116:0/1112696724 wait complete. 2024-04-25T00:27:23.503 INFO:tasks.mds_thrash.fs.[b]:mds.d reported in up:standby state 2024-04-25T00:27:23.503 INFO:tasks.mds_thrash.fs.[b]:waiting for 73 secs before thrashing 2024-04-25T00:27:23.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.642+0000 7fd6474e8640 1 -- 172.21.15.116:0/2248778104 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd64006dd30 con 0x7fd6400c2a10 2024-04-25T00:27:23.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.642+0000 7fd635ffb640 1 -- 172.21.15.116:0/2248778104 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 222 v222) ==== 95+0+6509 (secure 0 0 0) 0x7fd630057570 con 0x7fd6400c2a10 2024-04-25T00:27:23.645 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:27:23.645 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":222,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6641,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/2915274073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2915274073},{"type":"v1","addr":"172.21.15.174:6837","nonce":2915274073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":222}],"filesystems":[{"mdsmap":{"epoch":211,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:26:19.034064+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6485},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6485":{"gid":6485,"name":"a","rank":0,"incarnation":208,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6835/2371126056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2371126056},{"type":"v1","addr":"172.21.15.116:6835","nonce":2371126056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6485,"qdb_cluster":[6485]},"id":1},{"mdsmap":{"epoch":221,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:27:19.052286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8652":{"gid":8652,"name":"c","rank":0,"incarnation":218,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6833/3891103804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3891103804},{"type":"v1","addr":"172.21.15.116:6833","nonce":3891103804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8652,"qdb_cluster":[8652]},"id":2}]} 2024-04-25T00:27:23.645 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 222 2024-04-25T00:27:23.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.646+0000 7fd6474e8640 1 -- 172.21.15.116:0/2248778104 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd6240678c0 msgr2=0x7fd624069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:23.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.646+0000 7fd6474e8640 1 --2- 172.21.15.116:0/2248778104 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd6240678c0 0x7fd624069d80 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fd63c0ae000 tx=0x7fd63c0a7000 comp rx=0 tx=0).stop 2024-04-25T00:27:23.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.646+0000 7fd6474e8640 1 -- 172.21.15.116:0/2248778104 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd6400c2a10 msgr2=0x7fd640160ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:23.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.646+0000 7fd6474e8640 1 --2- 172.21.15.116:0/2248778104 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd6400c2a10 0x7fd640160ab0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd63000cdd0 tx=0x7fd63000a720 comp rx=0 tx=0).stop 2024-04-25T00:27:23.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.646+0000 7fd6474e8640 1 -- 172.21.15.116:0/2248778104 shutdown_connections 2024-04-25T00:27:23.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.646+0000 7fd6474e8640 1 --2- 172.21.15.116:0/2248778104 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd6240678c0 0x7fd624069d80 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:23.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.646+0000 7fd6474e8640 1 --2- 172.21.15.116:0/2248778104 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd6400a9d40 0x7fd6400ac720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:23.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.646+0000 7fd6474e8640 1 --2- 172.21.15.116:0/2248778104 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd640163000 0x7fd640160ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:23.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.646+0000 7fd6474e8640 1 --2- 172.21.15.116:0/2248778104 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd6400c2a10 0x7fd640160ab0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:23.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.646+0000 7fd6474e8640 1 -- 172.21.15.116:0/2248778104 >> 172.21.15.116:0/2248778104 conn(0x7fd6400c3b00 msgr2=0x7fd640072be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:27:23.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.646+0000 7fd6474e8640 1 -- 172.21.15.116:0/2248778104 shutdown_connections 2024-04-25T00:27:23.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:23.646+0000 7fd6474e8640 1 -- 172.21.15.116:0/2248778104 wait complete. 2024-04-25T00:27:23.668 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:27:25.669 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:27:25.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:25.806+0000 7f27bee0c640 1 Processor -- start 2024-04-25T00:27:25.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:25.806+0000 7f27bee0c640 1 -- start start 2024-04-25T00:27:25.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:25.806+0000 7f27bee0c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f27b8152f50 0x7f27b80aa100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:25.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:25.806+0000 7f27bee0c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f27b80aa640 0x7f27b815cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:25.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:25.806+0000 7f27bee0c640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27b8073260 con 0x7f27b814fc20 2024-04-25T00:27:25.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:25.806+0000 7f27bee0c640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f27b80a9620 con 0x7f27b8152f50 2024-04-25T00:27:25.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:25.806+0000 7f27bee0c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f27b80a9900 con 0x7f27b80aa640 2024-04-25T00:27:25.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:25.806+0000 7f27b7fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f27b8152f50 0x7f27b80aa100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:25.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:25.806+0000 7f27bcba8640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f27b814fc20 0x7f27b806f6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:37528/0 (socket says 172.21.15.116:37528) 2024-04-25T00:27:25.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:25.806+0000 7f27b7fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f27b8152f50 0x7f27b80aa100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44928/0 (socket says 172.21.15.116:44928) 2024-04-25T00:27:25.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:25.806+0000 7f27b7fff640 1 -- 172.21.15.116:0/3173430532 learned_addr learned my addr 172.21.15.116:0/3173430532 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:27:25.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:25.806+0000 7f27b7fff640 1 -- 172.21.15.116:0/3173430532 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f27b80aa640 msgr2=0x7f27b815cdc0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T00:27:25.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:25.806+0000 7f27b7fff640 1 --2- 172.21.15.116:0/3173430532 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f27b80aa640 0x7f27b815cdc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:25.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:25.806+0000 7f27b7fff640 1 -- 172.21.15.116:0/3173430532 >> v1:172.21.15.174:6789/0 conn(0x7f27b814fc20 legacy=0x7f27b806f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:27:25.808 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:25.806+0000 7f27b7fff640 1 -- 172.21.15.116:0/3173430532 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27b80a9be0 con 0x7f27b8152f50 2024-04-25T00:27:25.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:25.810+0000 7f27b7fff640 1 --2- 172.21.15.116:0/3173430532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27b8152f50 0x7f27b80aa100 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f27a4009870 tx=0x7f27a401c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33b327bb1f1ee562 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:25.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:25.810+0000 7f27b77fe640 1 -- 172.21.15.116:0/3173430532 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27a40b0070 con 0x7f27b8152f50 2024-04-25T00:27:25.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:25.810+0000 7f27b77fe640 1 -- 172.21.15.116:0/3173430532 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27a40ab600 con 0x7f27b8152f50 2024-04-25T00:27:25.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:25.810+0000 7f27b77fe640 1 -- 172.21.15.116:0/3173430532 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27a40ab9d0 con 0x7f27b8152f50 2024-04-25T00:27:25.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:25.810+0000 7f27bee0c640 1 -- 172.21.15.116:0/3173430532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27b8152f50 msgr2=0x7f27b80aa100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:25.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:25.810+0000 7f27bee0c640 1 --2- 172.21.15.116:0/3173430532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27b8152f50 0x7f27b80aa100 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f27a4009870 tx=0x7f27a401c420 comp rx=0 tx=0).stop 2024-04-25T00:27:25.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:25.810+0000 7f27bee0c640 1 -- 172.21.15.116:0/3173430532 shutdown_connections 2024-04-25T00:27:25.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:25.810+0000 7f27bee0c640 1 --2- 172.21.15.116:0/3173430532 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f27b80aa640 0x7f27b815cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:25.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:25.810+0000 7f27bee0c640 1 --2- 172.21.15.116:0/3173430532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27b8152f50 0x7f27b80aa100 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:25.811 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:25.810+0000 7f27bee0c640 1 -- 172.21.15.116:0/3173430532 >> 172.21.15.116:0/3173430532 conn(0x7f27b80c4c10 msgr2=0x7f27b80c5020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:27:25.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:25.810+0000 7f27bee0c640 1 -- 172.21.15.116:0/3173430532 shutdown_connections 2024-04-25T00:27:25.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:25.810+0000 7f27bee0c640 1 -- 172.21.15.116:0/3173430532 wait complete. 2024-04-25T00:27:25.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:25.814+0000 7f27bee0c640 1 Processor -- start 2024-04-25T00:27:25.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:25.814+0000 7f27bee0c640 1 -- start start 2024-04-25T00:27:25.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:25.814+0000 7f27bee0c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f27b80aa640 0x7f27b8151fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:25.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:25.814+0000 7f27bee0c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f27b814fc20 0x7f27b8144050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:25.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:25.814+0000 7f27bee0c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27b8152f50 0x7f27b8140960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:25.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:25.814+0000 7f27bee0c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f27b806ff50 con 0x7f27b8152f50 2024-04-25T00:27:25.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:25.814+0000 7f27bee0c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f27b806fdd0 con 0x7f27b80aa640 2024-04-25T00:27:25.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:25.814+0000 7f27bee0c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f27b80aaab0 con 0x7f27b814fc20 2024-04-25T00:27:25.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:25.814+0000 7f27bd3a9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27b8152f50 0x7f27b8140960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:25.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:25.814+0000 7f27bd3a9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27b8152f50 0x7f27b8140960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44940/0 (socket says 172.21.15.116:44940) 2024-04-25T00:27:25.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:25.814+0000 7f27bd3a9640 1 -- 172.21.15.116:0/705662564 learned_addr learned my addr 172.21.15.116:0/705662564 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:27:25.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:25.814+0000 7f27bcba8640 1 --2- 172.21.15.116:0/705662564 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f27b80aa640 0x7f27b8151fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:25.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:25.814+0000 7f27b7fff640 1 --2- 172.21.15.116:0/705662564 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f27b814fc20 0x7f27b8144050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:25.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:25.814+0000 7f27bd3a9640 1 -- 172.21.15.116:0/705662564 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f27b814fc20 msgr2=0x7f27b8144050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:25.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:25.814+0000 7f27bd3a9640 1 --2- 172.21.15.116:0/705662564 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f27b814fc20 0x7f27b8144050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:25.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:25.814+0000 7f27bd3a9640 1 -- 172.21.15.116:0/705662564 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f27b80aa640 msgr2=0x7f27b8151fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:25.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:25.814+0000 7f27bd3a9640 1 --2- 172.21.15.116:0/705662564 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f27b80aa640 0x7f27b8151fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:25.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:25.814+0000 7f27bd3a9640 1 -- 172.21.15.116:0/705662564 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27b80c1960 con 0x7f27b8152f50 2024-04-25T00:27:25.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:25.814+0000 7f27bd3a9640 1 --2- 172.21.15.116:0/705662564 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27b8152f50 0x7f27b8140960 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f27b000b990 tx=0x7f27b000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:25.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:25.814+0000 7f27b57fa640 1 -- 172.21.15.116:0/705662564 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27b0019070 con 0x7f27b8152f50 2024-04-25T00:27:25.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:25.814+0000 7f27b57fa640 1 -- 172.21.15.116:0/705662564 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27b000cae0 con 0x7f27b8152f50 2024-04-25T00:27:25.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:25.814+0000 7f27bee0c640 1 -- 172.21.15.116:0/705662564 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27b81f2370 con 0x7f27b8152f50 2024-04-25T00:27:25.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:25.814+0000 7f27b57fa640 1 -- 172.21.15.116:0/705662564 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27b0014440 con 0x7f27b8152f50 2024-04-25T00:27:25.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:25.814+0000 7f27bee0c640 1 -- 172.21.15.116:0/705662564 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27b8150450 con 0x7f27b8152f50 2024-04-25T00:27:25.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:25.814+0000 7f27bee0c640 1 -- 172.21.15.116:0/705662564 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27b8150650 con 0x7f27b8152f50 2024-04-25T00:27:25.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:25.818+0000 7f27b57fa640 1 -- 172.21.15.116:0/705662564 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f27b001c070 con 0x7f27b8152f50 2024-04-25T00:27:25.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:25.818+0000 7f27b57fa640 1 --2- 172.21.15.116:0/705662564 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f279c0678f0 0x7f279c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:25.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:25.818+0000 7f27b57fa640 1 -- 172.21.15.116:0/705662564 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 10278+0+0 (secure 0 0 0) 0x7f27b0086a40 con 0x7f27b8152f50 2024-04-25T00:27:25.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:25.818+0000 7f27bcba8640 1 --2- 172.21.15.116:0/705662564 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f279c0678f0 0x7f279c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:25.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:25.822+0000 7f27bcba8640 1 --2- 172.21.15.116:0/705662564 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f279c0678f0 0x7f279c069db0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f27ac007240 tx=0x7f27ac007e70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:25.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:25.826+0000 7f27b57fa640 1 -- 172.21.15.116:0/705662564 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f27b8150650 con 0x7f27b8152f50 2024-04-25T00:27:26.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:26.090+0000 7f27bee0c640 1 -- 172.21.15.116:0/705662564 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f27b80aa2a0 con 0x7f27b8152f50 2024-04-25T00:27:26.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:26.090+0000 7f27b57fa640 1 -- 172.21.15.116:0/705662564 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 222 v222) ==== 95+0+6509 (secure 0 0 0) 0x7f27b0055d90 con 0x7f27b8152f50 2024-04-25T00:27:26.093 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:27:26.093 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":222,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6641,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/2915274073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2915274073},{"type":"v1","addr":"172.21.15.174:6837","nonce":2915274073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":222}],"filesystems":[{"mdsmap":{"epoch":211,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:26:19.034064+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6485},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6485":{"gid":6485,"name":"a","rank":0,"incarnation":208,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6835/2371126056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2371126056},{"type":"v1","addr":"172.21.15.116:6835","nonce":2371126056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6485,"qdb_cluster":[6485]},"id":1},{"mdsmap":{"epoch":221,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:27:19.052286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8652":{"gid":8652,"name":"c","rank":0,"incarnation":218,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6833/3891103804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3891103804},{"type":"v1","addr":"172.21.15.116:6833","nonce":3891103804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8652,"qdb_cluster":[8652]},"id":2}]} 2024-04-25T00:27:26.094 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 222 2024-04-25T00:27:26.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:26.094+0000 7f27bee0c640 1 -- 172.21.15.116:0/705662564 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f279c0678f0 msgr2=0x7f279c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:26.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:26.094+0000 7f27bee0c640 1 --2- 172.21.15.116:0/705662564 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f279c0678f0 0x7f279c069db0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f27ac007240 tx=0x7f27ac007e70 comp rx=0 tx=0).stop 2024-04-25T00:27:26.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:26.094+0000 7f27bee0c640 1 -- 172.21.15.116:0/705662564 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27b8152f50 msgr2=0x7f27b8140960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:26.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:26.094+0000 7f27bee0c640 1 --2- 172.21.15.116:0/705662564 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27b8152f50 0x7f27b8140960 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f27b000b990 tx=0x7f27b000be50 comp rx=0 tx=0).stop 2024-04-25T00:27:26.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:26.094+0000 7f27bee0c640 1 -- 172.21.15.116:0/705662564 shutdown_connections 2024-04-25T00:27:26.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:26.094+0000 7f27bee0c640 1 --2- 172.21.15.116:0/705662564 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f279c0678f0 0x7f279c069db0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:26.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:26.094+0000 7f27bee0c640 1 --2- 172.21.15.116:0/705662564 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27b8152f50 0x7f27b8140960 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:26.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:26.094+0000 7f27bee0c640 1 --2- 172.21.15.116:0/705662564 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f27b814fc20 0x7f27b8144050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:26.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:26.094+0000 7f27bee0c640 1 --2- 172.21.15.116:0/705662564 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f27b80aa640 0x7f27b8151fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:26.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:26.094+0000 7f27bee0c640 1 -- 172.21.15.116:0/705662564 >> 172.21.15.116:0/705662564 conn(0x7f27b80c4c10 msgr2=0x7f27b8147bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:27:26.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:26.094+0000 7f27bee0c640 1 -- 172.21.15.116:0/705662564 shutdown_connections 2024-04-25T00:27:26.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:26.094+0000 7f27bee0c640 1 -- 172.21.15.116:0/705662564 wait complete. 2024-04-25T00:27:26.115 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:27:28.117 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:27:28.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.254+0000 7fbf972ad640 1 Processor -- start 2024-04-25T00:27:28.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.254+0000 7fbf972ad640 1 -- start start 2024-04-25T00:27:28.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.254+0000 7fbf972ad640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fbf90072d70 0x7fbf9014fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:28.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.254+0000 7fbf972ad640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbf90153970 0x7fbf90153dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:28.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.254+0000 7fbf972ad640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf90158de0 con 0x7fbf90154310 2024-04-25T00:27:28.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.254+0000 7fbf972ad640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fbf900c4770 con 0x7fbf90072d70 2024-04-25T00:27:28.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.254+0000 7fbf972ad640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fbf900c3570 con 0x7fbf90153970 2024-04-25T00:27:28.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.254+0000 7fbf9584a640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fbf90154310 0x7fbf90155a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:56982/0 (socket says 172.21.15.116:56982) 2024-04-25T00:27:28.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.254+0000 7fbf9584a640 1 -- 172.21.15.116:0/3235533538 learned_addr learned my addr 172.21.15.116:0/3235533538 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:27:28.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.254+0000 7fbf95049640 1 --2- 172.21.15.116:0/3235533538 >> v2:172.21.15.174:3300/0 conn(0x7fbf90072d70 0x7fbf9014fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:28.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.254+0000 7fbf87fff640 1 -- 172.21.15.116:0/3235533538 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2530710674 0 0) 0x7fbf90158de0 con 0x7fbf90154310 2024-04-25T00:27:28.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.254+0000 7fbf94848640 1 --2- 172.21.15.116:0/3235533538 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbf90153970 0x7fbf90153dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:28.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.254+0000 7fbf87fff640 1 -- 172.21.15.116:0/3235533538 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf7c014540 con 0x7fbf90154310 2024-04-25T00:27:28.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.254+0000 7fbf95049640 1 -- 172.21.15.116:0/3235533538 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbf90153970 msgr2=0x7fbf90153dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:27:28.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.254+0000 7fbf95049640 1 --2- 172.21.15.116:0/3235533538 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbf90153970 0x7fbf90153dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:28.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.254+0000 7fbf95049640 1 -- 172.21.15.116:0/3235533538 >> v1:172.21.15.116:6789/0 conn(0x7fbf90154310 legacy=0x7fbf90155a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:28.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.254+0000 7fbf87fff640 1 -- 172.21.15.116:0/3235533538 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3933133605 0 0) 0x7fbf7c014540 con 0x7fbf90154310 2024-04-25T00:27:28.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.254+0000 7fbf95049640 1 -- 172.21.15.116:0/3235533538 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf900c3850 con 0x7fbf90072d70 2024-04-25T00:27:28.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.254+0000 7fbf94848640 1 --2- 172.21.15.116:0/3235533538 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbf90153970 0x7fbf90153dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:27:28.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.258+0000 7fbf95049640 1 --2- 172.21.15.116:0/3235533538 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf90072d70 0x7fbf9014fa90 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c0097c0 tx=0x7fbf8c01c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad490adbf7ae7688 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:28.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.258+0000 7fbf87fff640 1 -- 172.21.15.116:0/3235533538 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf8c0b0070 con 0x7fbf90072d70 2024-04-25T00:27:28.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.258+0000 7fbf87fff640 1 -- 172.21.15.116:0/3235533538 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf8c0ab400 con 0x7fbf90072d70 2024-04-25T00:27:28.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.258+0000 7fbf87fff640 1 -- 172.21.15.116:0/3235533538 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf88001a20 con 0x7fbf90072d70 2024-04-25T00:27:28.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.262+0000 7fbf972ad640 1 -- 172.21.15.116:0/3235533538 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf90072d70 msgr2=0x7fbf9014fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:28.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.262+0000 7fbf972ad640 1 --2- 172.21.15.116:0/3235533538 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf90072d70 0x7fbf9014fa90 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c0097c0 tx=0x7fbf8c01c330 comp rx=0 tx=0).stop 2024-04-25T00:27:28.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.262+0000 7fbf972ad640 1 -- 172.21.15.116:0/3235533538 shutdown_connections 2024-04-25T00:27:28.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.262+0000 7fbf972ad640 1 --2- 172.21.15.116:0/3235533538 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbf90153970 0x7fbf90153dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:28.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.262+0000 7fbf972ad640 1 --2- 172.21.15.116:0/3235533538 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf90072d70 0x7fbf9014fa90 secure :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c0097c0 tx=0x7fbf8c01c330 comp rx=0 tx=0).stop 2024-04-25T00:27:28.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.262+0000 7fbf972ad640 1 -- 172.21.15.116:0/3235533538 >> 172.21.15.116:0/3235533538 conn(0x7fbf900c4aa0 msgr2=0x7fbf900c4eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:27:28.262 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.262+0000 7fbf972ad640 1 -- 172.21.15.116:0/3235533538 shutdown_connections 2024-04-25T00:27:28.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.262+0000 7fbf972ad640 1 -- 172.21.15.116:0/3235533538 wait complete. 2024-04-25T00:27:28.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.262+0000 7fbf972ad640 1 Processor -- start 2024-04-25T00:27:28.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.262+0000 7fbf972ad640 1 -- start start 2024-04-25T00:27:28.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.266+0000 7fbf972ad640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf90153970 0x7fbf90158080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:28.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.266+0000 7fbf972ad640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbf90154310 0x7fbf900c63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:28.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.266+0000 7fbf972ad640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf901eb400 0x7fbf90145360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:28.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.266+0000 7fbf972ad640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbf9006b5e0 con 0x7fbf901eb400 2024-04-25T00:27:28.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.266+0000 7fbf972ad640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbf9006b460 con 0x7fbf90153970 2024-04-25T00:27:28.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.266+0000 7fbf972ad640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fbf9006b760 con 0x7fbf90154310 2024-04-25T00:27:28.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.266+0000 7fbf9584a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf901eb400 0x7fbf90145360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:28.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.266+0000 7fbf9584a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf901eb400 0x7fbf90145360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44946/0 (socket says 172.21.15.116:44946) 2024-04-25T00:27:28.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.266+0000 7fbf95049640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf90153970 0x7fbf90158080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:28.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.266+0000 7fbf9584a640 1 -- 172.21.15.116:0/2928395964 learned_addr learned my addr 172.21.15.116:0/2928395964 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:27:28.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.266+0000 7fbf9584a640 1 -- 172.21.15.116:0/2928395964 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbf90154310 msgr2=0x7fbf900c63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:28.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.266+0000 7fbf9584a640 1 --2- 172.21.15.116:0/2928395964 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbf90154310 0x7fbf900c63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:28.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.266+0000 7fbf9584a640 1 -- 172.21.15.116:0/2928395964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf90153970 msgr2=0x7fbf90158080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:28.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.266+0000 7fbf9584a640 1 --2- 172.21.15.116:0/2928395964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf90153970 0x7fbf90158080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:28.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.266+0000 7fbf9584a640 1 -- 172.21.15.116:0/2928395964 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf9014be00 con 0x7fbf901eb400 2024-04-25T00:27:28.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.266+0000 7fbf9584a640 1 --2- 172.21.15.116:0/2928395964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf901eb400 0x7fbf90145360 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7fbf88007fd0 tx=0x7fbf8800ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:28.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.266+0000 7fbf85ffb640 1 -- 172.21.15.116:0/2928395964 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf8803a070 con 0x7fbf901eb400 2024-04-25T00:27:28.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.266+0000 7fbf85ffb640 1 -- 172.21.15.116:0/2928395964 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf88004d80 con 0x7fbf901eb400 2024-04-25T00:27:28.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.266+0000 7fbf972ad640 1 -- 172.21.15.116:0/2928395964 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf900c5c30 con 0x7fbf901eb400 2024-04-25T00:27:28.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.266+0000 7fbf85ffb640 1 -- 172.21.15.116:0/2928395964 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf8800a880 con 0x7fbf901eb400 2024-04-25T00:27:28.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.266+0000 7fbf972ad640 1 -- 172.21.15.116:0/2928395964 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf900c5ed0 con 0x7fbf901eb400 2024-04-25T00:27:28.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.266+0000 7fbf972ad640 1 -- 172.21.15.116:0/2928395964 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf700052a0 con 0x7fbf901eb400 2024-04-25T00:27:28.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.270+0000 7fbf85ffb640 1 -- 172.21.15.116:0/2928395964 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbf8803d070 con 0x7fbf901eb400 2024-04-25T00:27:28.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.270+0000 7fbf85ffb640 1 --2- 172.21.15.116:0/2928395964 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbf740678f0 0x7fbf74069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:28.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.270+0000 7fbf85ffb640 1 -- 172.21.15.116:0/2928395964 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 10278+0+0 (secure 0 0 0) 0x7fbf880a8290 con 0x7fbf901eb400 2024-04-25T00:27:28.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.270+0000 7fbf95049640 1 --2- 172.21.15.116:0/2928395964 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbf740678f0 0x7fbf74069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:28.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.270+0000 7fbf95049640 1 --2- 172.21.15.116:0/2928395964 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbf740678f0 0x7fbf74069db0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c0ad000 tx=0x7fbf8c004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:28.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.274+0000 7fbf85ffb640 1 -- 172.21.15.116:0/2928395964 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fbf880775c0 con 0x7fbf901eb400 2024-04-25T00:27:28.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.550+0000 7fbf972ad640 1 -- 172.21.15.116:0/2928395964 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbf70005590 con 0x7fbf901eb400 2024-04-25T00:27:28.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.550+0000 7fbf85ffb640 1 -- 172.21.15.116:0/2928395964 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 222 v222) ==== 95+0+6509 (secure 0 0 0) 0x7fbf8807b8c0 con 0x7fbf901eb400 2024-04-25T00:27:28.552 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:27:28.553 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":222,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6641,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/2915274073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2915274073},{"type":"v1","addr":"172.21.15.174:6837","nonce":2915274073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":222}],"filesystems":[{"mdsmap":{"epoch":211,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:26:19.034064+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6485},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6485":{"gid":6485,"name":"a","rank":0,"incarnation":208,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6835/2371126056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2371126056},{"type":"v1","addr":"172.21.15.116:6835","nonce":2371126056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6485,"qdb_cluster":[6485]},"id":1},{"mdsmap":{"epoch":221,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:27:19.052286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8652":{"gid":8652,"name":"c","rank":0,"incarnation":218,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6833/3891103804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3891103804},{"type":"v1","addr":"172.21.15.116:6833","nonce":3891103804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8652,"qdb_cluster":[8652]},"id":2}]} 2024-04-25T00:27:28.553 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 222 2024-04-25T00:27:28.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.554+0000 7fbf972ad640 1 -- 172.21.15.116:0/2928395964 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbf740678f0 msgr2=0x7fbf74069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:28.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.554+0000 7fbf972ad640 1 --2- 172.21.15.116:0/2928395964 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbf740678f0 0x7fbf74069db0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c0ad000 tx=0x7fbf8c004740 comp rx=0 tx=0).stop 2024-04-25T00:27:28.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.554+0000 7fbf972ad640 1 -- 172.21.15.116:0/2928395964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf901eb400 msgr2=0x7fbf90145360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:28.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.554+0000 7fbf972ad640 1 --2- 172.21.15.116:0/2928395964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf901eb400 0x7fbf90145360 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7fbf88007fd0 tx=0x7fbf8800ccd0 comp rx=0 tx=0).stop 2024-04-25T00:27:28.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.554+0000 7fbf972ad640 1 -- 172.21.15.116:0/2928395964 shutdown_connections 2024-04-25T00:27:28.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.554+0000 7fbf972ad640 1 --2- 172.21.15.116:0/2928395964 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbf740678f0 0x7fbf74069db0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:28.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.554+0000 7fbf972ad640 1 --2- 172.21.15.116:0/2928395964 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf901eb400 0x7fbf90145360 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:28.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.554+0000 7fbf972ad640 1 --2- 172.21.15.116:0/2928395964 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbf90154310 0x7fbf900c63f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:28.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.554+0000 7fbf972ad640 1 --2- 172.21.15.116:0/2928395964 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf90153970 0x7fbf90158080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:28.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.554+0000 7fbf972ad640 1 -- 172.21.15.116:0/2928395964 >> 172.21.15.116:0/2928395964 conn(0x7fbf900c4aa0 msgr2=0x7fbf9014ca90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:27:28.555 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.554+0000 7fbf972ad640 1 -- 172.21.15.116:0/2928395964 shutdown_connections 2024-04-25T00:27:28.556 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:28.554+0000 7fbf972ad640 1 -- 172.21.15.116:0/2928395964 wait complete. 2024-04-25T00:27:28.575 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:27:30.575 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:27:30.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:30.726+0000 7fb58083c640 1 Processor -- start 2024-04-25T00:27:30.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:30.726+0000 7fb58083c640 1 -- start start 2024-04-25T00:27:30.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:30.726+0000 7fb58083c640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fb57c152e40 0x7fb57c0710b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:30.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:30.726+0000 7fb58083c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb57c0c3f70 0x7fb57c1600e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:30.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:30.726+0000 7fb58083c640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb57c153220 con 0x7fb57c0715f0 2024-04-25T00:27:30.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:30.726+0000 7fb58083c640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fb57c0c4c90 con 0x7fb57c152e40 2024-04-25T00:27:30.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:30.726+0000 7fb58083c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fb57c0c7a90 con 0x7fb57c0c3f70 2024-04-25T00:27:30.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:30.726+0000 7fb57a59c640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fb57c152e40 0x7fb57c0710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:30.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:30.726+0000 7fb57a59c640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fb57c152e40 0x7fb57c0710b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:43248/0 (socket says 172.21.15.116:43248) 2024-04-25T00:27:30.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:30.726+0000 7fb57a59c640 1 -- 172.21.15.116:0/1790725740 learned_addr learned my addr 172.21.15.116:0/1790725740 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:27:30.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:30.726+0000 7fb57ad9d640 1 --2- 172.21.15.116:0/1790725740 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb57c0c3f70 0x7fb57c1600e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:30.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:30.726+0000 7fb579d9b640 1 -- 172.21.15.116:0/1790725740 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3960880236 0 0) 0x7fb57c153220 con 0x7fb57c0715f0 2024-04-25T00:27:30.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:30.726+0000 7fb57ad9d640 1 -- 172.21.15.116:0/1790725740 >> v2:172.21.15.174:3300/0 conn(0x7fb57c152e40 msgr2=0x7fb57c0710b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:27:30.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:30.726+0000 7fb57ad9d640 1 --2- 172.21.15.116:0/1790725740 >> v2:172.21.15.174:3300/0 conn(0x7fb57c152e40 0x7fb57c0710b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:30.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:30.726+0000 7fb57ad9d640 1 -- 172.21.15.116:0/1790725740 >> v1:172.21.15.174:6789/0 conn(0x7fb57c0715f0 legacy=0x7fb57c1557c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:30.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:30.726+0000 7fb57ad9d640 1 -- 172.21.15.116:0/1790725740 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb57c0c7d70 con 0x7fb57c0c3f70 2024-04-25T00:27:30.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:30.726+0000 7fb57a59c640 1 --2- 172.21.15.116:0/1790725740 >> v2:172.21.15.174:3300/0 conn(0x7fb57c152e40 0x7fb57c0710b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:27:30.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:30.730+0000 7fb57ad9d640 1 --2- 172.21.15.116:0/1790725740 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb57c0c3f70 0x7fb57c1600e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb574009d90 tx=0x7fb57401c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb0fc779c5a63666 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:30.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:30.734+0000 7fb579d9b640 1 -- 172.21.15.116:0/1790725740 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5740b2070 con 0x7fb57c0c3f70 2024-04-25T00:27:30.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:30.734+0000 7fb579d9b640 1 -- 172.21.15.116:0/1790725740 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5740a5970 con 0x7fb57c0c3f70 2024-04-25T00:27:30.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:30.734+0000 7fb579d9b640 1 -- 172.21.15.116:0/1790725740 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5740a5d40 con 0x7fb57c0c3f70 2024-04-25T00:27:30.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:30.734+0000 7fb58083c640 1 -- 172.21.15.116:0/1790725740 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb57c0c3f70 msgr2=0x7fb57c1600e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:30.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:30.734+0000 7fb58083c640 1 --2- 172.21.15.116:0/1790725740 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb57c0c3f70 0x7fb57c1600e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb574009d90 tx=0x7fb57401c8b0 comp rx=0 tx=0).stop 2024-04-25T00:27:30.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:30.734+0000 7fb58083c640 1 -- 172.21.15.116:0/1790725740 shutdown_connections 2024-04-25T00:27:30.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:30.734+0000 7fb58083c640 1 --2- 172.21.15.116:0/1790725740 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb57c0c3f70 0x7fb57c1600e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:30.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:30.734+0000 7fb58083c640 1 --2- 172.21.15.116:0/1790725740 >> v2:172.21.15.174:3300/0 conn(0x7fb57c152e40 0x7fb57c0710b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:30.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:30.734+0000 7fb58083c640 1 -- 172.21.15.116:0/1790725740 >> 172.21.15.116:0/1790725740 conn(0x7fb57c0c9160 msgr2=0x7fb57c0c9570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:27:30.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:30.734+0000 7fb58083c640 1 -- 172.21.15.116:0/1790725740 shutdown_connections 2024-04-25T00:27:30.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:30.734+0000 7fb58083c640 1 -- 172.21.15.116:0/1790725740 wait complete. 2024-04-25T00:27:30.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:30.738+0000 7fb58083c640 1 Processor -- start 2024-04-25T00:27:30.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:30.738+0000 7fb58083c640 1 -- start start 2024-04-25T00:27:30.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:30.738+0000 7fb58083c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb57c0715f0 0x7fb57c1eec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:30.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:30.738+0000 7fb58083c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb57c0c3f70 0x7fb57c149100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:30.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:30.738+0000 7fb58083c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb57c152e40 0x7fb57c1f1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:30.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:30.738+0000 7fb58083c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb57c162fc0 con 0x7fb57c0c3f70 2024-04-25T00:27:30.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:30.738+0000 7fb58083c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb57c162e40 con 0x7fb57c0715f0 2024-04-25T00:27:30.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:30.738+0000 7fb58083c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fb57c163140 con 0x7fb57c152e40 2024-04-25T00:27:30.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:30.738+0000 7fb573fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb57c0c3f70 0x7fb57c149100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:30.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:30.738+0000 7fb573fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb57c0c3f70 0x7fb57c149100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44958/0 (socket says 172.21.15.116:44958) 2024-04-25T00:27:30.739 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:30.738+0000 7fb573fff640 1 -- 172.21.15.116:0/2581291179 learned_addr learned my addr 172.21.15.116:0/2581291179 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:27:30.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:30.738+0000 7fb573fff640 1 -- 172.21.15.116:0/2581291179 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb57c152e40 msgr2=0x7fb57c1f1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:30.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:30.738+0000 7fb573fff640 1 --2- 172.21.15.116:0/2581291179 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb57c152e40 0x7fb57c1f1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:30.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:30.738+0000 7fb573fff640 1 -- 172.21.15.116:0/2581291179 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb57c0715f0 msgr2=0x7fb57c1eec10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T00:27:30.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:30.738+0000 7fb573fff640 1 --2- 172.21.15.116:0/2581291179 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb57c0715f0 0x7fb57c1eec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:30.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:30.738+0000 7fb573fff640 1 -- 172.21.15.116:0/2581291179 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb57c159ab0 con 0x7fb57c0c3f70 2024-04-25T00:27:30.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:30.738+0000 7fb573fff640 1 --2- 172.21.15.116:0/2581291179 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb57c0c3f70 0x7fb57c149100 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7fb56400c450 tx=0x7fb56400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:30.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:30.738+0000 7fb572ffd640 1 -- 172.21.15.116:0/2581291179 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb564017070 con 0x7fb57c0c3f70 2024-04-25T00:27:30.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:30.738+0000 7fb572ffd640 1 -- 172.21.15.116:0/2581291179 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb564005480 con 0x7fb57c0c3f70 2024-04-25T00:27:30.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:30.738+0000 7fb572ffd640 1 -- 172.21.15.116:0/2581291179 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb564003840 con 0x7fb57c0c3f70 2024-04-25T00:27:30.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:30.738+0000 7fb58083c640 1 -- 172.21.15.116:0/2581291179 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb57c156da0 con 0x7fb57c0c3f70 2024-04-25T00:27:30.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:30.742+0000 7fb58083c640 1 -- 172.21.15.116:0/2581291179 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb57c14a0a0 con 0x7fb57c0c3f70 2024-04-25T00:27:30.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:30.742+0000 7fb58083c640 1 -- 172.21.15.116:0/2581291179 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb57c1572c0 con 0x7fb57c0c3f70 2024-04-25T00:27:30.742 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:30.742+0000 7fb572ffd640 1 -- 172.21.15.116:0/2581291179 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb56401a070 con 0x7fb57c0c3f70 2024-04-25T00:27:30.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:30.742+0000 7fb572ffd640 1 --2- 172.21.15.116:0/2581291179 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb55c067910 0x7fb55c069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:30.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:30.742+0000 7fb572ffd640 1 -- 172.21.15.116:0/2581291179 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 10278+0+0 (secure 0 0 0) 0x7fb564086140 con 0x7fb57c0c3f70 2024-04-25T00:27:30.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:30.742+0000 7fb57a59c640 1 --2- 172.21.15.116:0/2581291179 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb55c067910 0x7fb55c069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:30.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:30.742+0000 7fb57a59c640 1 --2- 172.21.15.116:0/2581291179 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb55c067910 0x7fb55c069dd0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fb56c006f50 tx=0x7fb56c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:30.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:30.750+0000 7fb572ffd640 1 -- 172.21.15.116:0/2581291179 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fb564055b60 con 0x7fb57c0c3f70 2024-04-25T00:27:31.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:31.054+0000 7fb58083c640 1 -- 172.21.15.116:0/2581291179 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb57c0c7870 con 0x7fb57c0c3f70 2024-04-25T00:27:31.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:31.058+0000 7fb572ffd640 1 -- 172.21.15.116:0/2581291179 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 222 v222) ==== 95+0+6509 (secure 0 0 0) 0x7fb564055460 con 0x7fb57c0c3f70 2024-04-25T00:27:31.058 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:27:31.058 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":222,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6641,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/2915274073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2915274073},{"type":"v1","addr":"172.21.15.174:6837","nonce":2915274073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":222}],"filesystems":[{"mdsmap":{"epoch":211,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:26:19.034064+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6485},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6485":{"gid":6485,"name":"a","rank":0,"incarnation":208,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6835/2371126056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2371126056},{"type":"v1","addr":"172.21.15.116:6835","nonce":2371126056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6485,"qdb_cluster":[6485]},"id":1},{"mdsmap":{"epoch":221,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:27:19.052286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8652":{"gid":8652,"name":"c","rank":0,"incarnation":218,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6833/3891103804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3891103804},{"type":"v1","addr":"172.21.15.116:6833","nonce":3891103804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8652,"qdb_cluster":[8652]},"id":2}]} 2024-04-25T00:27:31.058 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 222 2024-04-25T00:27:31.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:31.062+0000 7fb58083c640 1 -- 172.21.15.116:0/2581291179 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb55c067910 msgr2=0x7fb55c069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:31.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:31.062+0000 7fb58083c640 1 --2- 172.21.15.116:0/2581291179 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb55c067910 0x7fb55c069dd0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fb56c006f50 tx=0x7fb56c008040 comp rx=0 tx=0).stop 2024-04-25T00:27:31.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:31.062+0000 7fb58083c640 1 -- 172.21.15.116:0/2581291179 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb57c0c3f70 msgr2=0x7fb57c149100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:31.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:31.062+0000 7fb58083c640 1 --2- 172.21.15.116:0/2581291179 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb57c0c3f70 0x7fb57c149100 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7fb56400c450 tx=0x7fb56400c910 comp rx=0 tx=0).stop 2024-04-25T00:27:31.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:31.062+0000 7fb58083c640 1 -- 172.21.15.116:0/2581291179 shutdown_connections 2024-04-25T00:27:31.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:31.062+0000 7fb58083c640 1 --2- 172.21.15.116:0/2581291179 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb55c067910 0x7fb55c069dd0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:31.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:31.062+0000 7fb58083c640 1 --2- 172.21.15.116:0/2581291179 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb57c152e40 0x7fb57c1f1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:31.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:31.062+0000 7fb58083c640 1 --2- 172.21.15.116:0/2581291179 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb57c0c3f70 0x7fb57c149100 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:31.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:31.062+0000 7fb58083c640 1 --2- 172.21.15.116:0/2581291179 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb57c0715f0 0x7fb57c1eec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:31.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:31.062+0000 7fb58083c640 1 -- 172.21.15.116:0/2581291179 >> 172.21.15.116:0/2581291179 conn(0x7fb57c0c9160 msgr2=0x7fb57c06da10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:27:31.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:31.062+0000 7fb58083c640 1 -- 172.21.15.116:0/2581291179 shutdown_connections 2024-04-25T00:27:31.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:31.062+0000 7fb58083c640 1 -- 172.21.15.116:0/2581291179 wait complete. 2024-04-25T00:27:31.081 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:27:33.082 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:27:33.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.234+0000 7fc8db3dc640 1 Processor -- start 2024-04-25T00:27:33.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.234+0000 7fc8db3dc640 1 -- start start 2024-04-25T00:27:33.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.234+0000 7fc8db3dc640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc8d40c5a60 0x7fc8d40c5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:33.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.234+0000 7fc8db3dc640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fc8d41558c0 0x7fc8d4155d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:33.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.234+0000 7fc8db3dc640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8d4158eb0 con 0x7fc8d4152db0 2024-04-25T00:27:33.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.234+0000 7fc8db3dc640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fc8d4159190 con 0x7fc8d41558c0 2024-04-25T00:27:33.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.234+0000 7fc8d9178640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fc8d4152db0 0x7fc8d40c7af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:56988/0 (socket says 172.21.15.116:56988) 2024-04-25T00:27:33.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.234+0000 7fc8db3dc640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fc8d4159470 con 0x7fc8d40c5a60 2024-04-25T00:27:33.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.234+0000 7fc8d9178640 1 -- 172.21.15.116:0/3767196051 learned_addr learned my addr 172.21.15.116:0/3767196051 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:27:33.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.234+0000 7fc8d9979640 1 --2- 172.21.15.116:0/3767196051 >> v2:172.21.15.174:3300/0 conn(0x7fc8d41558c0 0x7fc8d4155d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:33.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.234+0000 7fc8d8977640 1 --2- 172.21.15.116:0/3767196051 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc8d40c5a60 0x7fc8d40c5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:33.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.234+0000 7fc8cbfff640 1 -- 172.21.15.116:0/3767196051 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2027861013 0 0) 0x7fc8d4158eb0 con 0x7fc8d4152db0 2024-04-25T00:27:33.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.234+0000 7fc8d9979640 1 -- 172.21.15.116:0/3767196051 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc8d40c5a60 msgr2=0x7fc8d40c5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:27:33.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.234+0000 7fc8d9979640 1 --2- 172.21.15.116:0/3767196051 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc8d40c5a60 0x7fc8d40c5ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:33.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.234+0000 7fc8d9979640 1 -- 172.21.15.116:0/3767196051 >> v1:172.21.15.116:6789/0 conn(0x7fc8d4152db0 legacy=0x7fc8d40c7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:33.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.234+0000 7fc8d9979640 1 -- 172.21.15.116:0/3767196051 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8d4159750 con 0x7fc8d41558c0 2024-04-25T00:27:33.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.238+0000 7fc8d9979640 1 --2- 172.21.15.116:0/3767196051 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc8d41558c0 0x7fc8d4155d10 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc009d90 tx=0x7fc8cc01d330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7bc79a1b6b9065a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:33.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.238+0000 7fc8cbfff640 1 -- 172.21.15.116:0/3767196051 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8cc0b2070 con 0x7fc8d41558c0 2024-04-25T00:27:33.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.238+0000 7fc8cbfff640 1 -- 172.21.15.116:0/3767196051 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8cc0a59b0 con 0x7fc8d41558c0 2024-04-25T00:27:33.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.238+0000 7fc8cbfff640 1 -- 172.21.15.116:0/3767196051 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8cc0a5d80 con 0x7fc8d41558c0 2024-04-25T00:27:33.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.238+0000 7fc8db3dc640 1 -- 172.21.15.116:0/3767196051 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc8d41558c0 msgr2=0x7fc8d4155d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:33.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.238+0000 7fc8db3dc640 1 --2- 172.21.15.116:0/3767196051 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc8d41558c0 0x7fc8d4155d10 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc009d90 tx=0x7fc8cc01d330 comp rx=0 tx=0).stop 2024-04-25T00:27:33.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.242+0000 7fc8db3dc640 1 -- 172.21.15.116:0/3767196051 shutdown_connections 2024-04-25T00:27:33.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.242+0000 7fc8db3dc640 1 --2- 172.21.15.116:0/3767196051 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc8d41558c0 0x7fc8d4155d10 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:33.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.242+0000 7fc8db3dc640 1 --2- 172.21.15.116:0/3767196051 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc8d40c5a60 0x7fc8d40c5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:33.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.242+0000 7fc8db3dc640 1 -- 172.21.15.116:0/3767196051 >> 172.21.15.116:0/3767196051 conn(0x7fc8d40acc10 msgr2=0x7fc8d40ad020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:27:33.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.242+0000 7fc8db3dc640 1 -- 172.21.15.116:0/3767196051 shutdown_connections 2024-04-25T00:27:33.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.242+0000 7fc8db3dc640 1 -- 172.21.15.116:0/3767196051 wait complete. 2024-04-25T00:27:33.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.242+0000 7fc8db3dc640 1 Processor -- start 2024-04-25T00:27:33.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.242+0000 7fc8db3dc640 1 -- start start 2024-04-25T00:27:33.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.246+0000 7fc8db3dc640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc8d40c5a60 0x7fc8d41ee030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:33.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.246+0000 7fc8db3dc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8d4152db0 0x7fc8d4144060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:33.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.246+0000 7fc8db3dc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc8d41558c0 0x7fc8d414b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:33.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.246+0000 7fc8db3dc640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc8d4153310 con 0x7fc8d4152db0 2024-04-25T00:27:33.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.246+0000 7fc8db3dc640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc8d4153190 con 0x7fc8d41558c0 2024-04-25T00:27:33.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.246+0000 7fc8db3dc640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fc8d4162a60 con 0x7fc8d40c5a60 2024-04-25T00:27:33.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.246+0000 7fc8d8977640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8d4152db0 0x7fc8d4144060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:33.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.246+0000 7fc8d8977640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8d4152db0 0x7fc8d4144060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:44970/0 (socket says 172.21.15.116:44970) 2024-04-25T00:27:33.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.246+0000 7fc8d8977640 1 -- 172.21.15.116:0/4005680931 learned_addr learned my addr 172.21.15.116:0/4005680931 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:27:33.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.246+0000 7fc8d9178640 1 --2- 172.21.15.116:0/4005680931 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc8d40c5a60 0x7fc8d41ee030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:33.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.246+0000 7fc8d9979640 1 --2- 172.21.15.116:0/4005680931 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc8d41558c0 0x7fc8d414b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:33.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.246+0000 7fc8d8977640 1 -- 172.21.15.116:0/4005680931 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc8d40c5a60 msgr2=0x7fc8d41ee030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:33.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.246+0000 7fc8d8977640 1 --2- 172.21.15.116:0/4005680931 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc8d40c5a60 0x7fc8d41ee030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:33.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.246+0000 7fc8d8977640 1 -- 172.21.15.116:0/4005680931 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc8d41558c0 msgr2=0x7fc8d414b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:33.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.246+0000 7fc8d8977640 1 --2- 172.21.15.116:0/4005680931 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc8d41558c0 0x7fc8d414b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:33.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.246+0000 7fc8d8977640 1 -- 172.21.15.116:0/4005680931 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8d41ee660 con 0x7fc8d4152db0 2024-04-25T00:27:33.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.246+0000 7fc8d8977640 1 --2- 172.21.15.116:0/4005680931 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8d4152db0 0x7fc8d4144060 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7fc8d000e930 tx=0x7fc8d000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:33.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.246+0000 7fc8c9ffb640 1 -- 172.21.15.116:0/4005680931 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8d0017070 con 0x7fc8d4152db0 2024-04-25T00:27:33.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.246+0000 7fc8c9ffb640 1 -- 172.21.15.116:0/4005680931 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8d000ca00 con 0x7fc8d4152db0 2024-04-25T00:27:33.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.246+0000 7fc8c9ffb640 1 -- 172.21.15.116:0/4005680931 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8d000cdb0 con 0x7fc8d4152db0 2024-04-25T00:27:33.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.246+0000 7fc8db3dc640 1 -- 172.21.15.116:0/4005680931 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8d41ee940 con 0x7fc8d4152db0 2024-04-25T00:27:33.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.246+0000 7fc8db3dc640 1 -- 172.21.15.116:0/4005680931 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8d41eef00 con 0x7fc8d4152db0 2024-04-25T00:27:33.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.246+0000 7fc8db3dc640 1 -- 172.21.15.116:0/4005680931 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8d4068a60 con 0x7fc8d4152db0 2024-04-25T00:27:33.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.246+0000 7fc8c9ffb640 1 -- 172.21.15.116:0/4005680931 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc8d001a070 con 0x7fc8d4152db0 2024-04-25T00:27:33.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.250+0000 7fc8c9ffb640 1 --2- 172.21.15.116:0/4005680931 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc8b8067910 0x7fc8b8069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:33.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.250+0000 7fc8c9ffb640 1 -- 172.21.15.116:0/4005680931 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 10278+0+0 (secure 0 0 0) 0x7fc8d0087220 con 0x7fc8d4152db0 2024-04-25T00:27:33.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.250+0000 7fc8d9178640 1 --2- 172.21.15.116:0/4005680931 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc8b8067910 0x7fc8b8069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:33.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.250+0000 7fc8d9178640 1 --2- 172.21.15.116:0/4005680931 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc8b8067910 0x7fc8b8069dd0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fc8c4007fd0 tx=0x7fc8c4007ba0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:33.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.254+0000 7fc8c9ffb640 1 -- 172.21.15.116:0/4005680931 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fc8d0056ca0 con 0x7fc8d4152db0 2024-04-25T00:27:33.557 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.554+0000 7fc8db3dc640 1 -- 172.21.15.116:0/4005680931 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc8d40cabd0 con 0x7fc8d4152db0 2024-04-25T00:27:33.558 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.558+0000 7fc8c9ffb640 1 -- 172.21.15.116:0/4005680931 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 222 v222) ==== 95+0+6509 (secure 0 0 0) 0x7fc8d00565a0 con 0x7fc8d4152db0 2024-04-25T00:27:33.559 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:27:33.559 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":222,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6641,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/2915274073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2915274073},{"type":"v1","addr":"172.21.15.174:6837","nonce":2915274073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":222}],"filesystems":[{"mdsmap":{"epoch":211,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:26:19.034064+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6485},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6485":{"gid":6485,"name":"a","rank":0,"incarnation":208,"state":"up:active","state_seq":10,"addr":"172.21.15.116:6835/2371126056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":2371126056},{"type":"v1","addr":"172.21.15.116:6835","nonce":2371126056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6485,"qdb_cluster":[6485]},"id":1},{"mdsmap":{"epoch":221,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:27:19.052286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8652":{"gid":8652,"name":"c","rank":0,"incarnation":218,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6833/3891103804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3891103804},{"type":"v1","addr":"172.21.15.116:6833","nonce":3891103804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8652,"qdb_cluster":[8652]},"id":2}]} 2024-04-25T00:27:33.559 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 222 2024-04-25T00:27:33.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.562+0000 7fc8db3dc640 1 -- 172.21.15.116:0/4005680931 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc8b8067910 msgr2=0x7fc8b8069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:33.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.562+0000 7fc8db3dc640 1 --2- 172.21.15.116:0/4005680931 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc8b8067910 0x7fc8b8069dd0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fc8c4007fd0 tx=0x7fc8c4007ba0 comp rx=0 tx=0).stop 2024-04-25T00:27:33.561 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.562+0000 7fc8db3dc640 1 -- 172.21.15.116:0/4005680931 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8d4152db0 msgr2=0x7fc8d4144060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:33.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.562+0000 7fc8db3dc640 1 --2- 172.21.15.116:0/4005680931 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8d4152db0 0x7fc8d4144060 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7fc8d000e930 tx=0x7fc8d000edf0 comp rx=0 tx=0).stop 2024-04-25T00:27:33.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.562+0000 7fc8db3dc640 1 -- 172.21.15.116:0/4005680931 shutdown_connections 2024-04-25T00:27:33.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.562+0000 7fc8db3dc640 1 --2- 172.21.15.116:0/4005680931 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc8b8067910 0x7fc8b8069dd0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:33.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.562+0000 7fc8db3dc640 1 --2- 172.21.15.116:0/4005680931 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc8d41558c0 0x7fc8d414b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:33.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.562+0000 7fc8db3dc640 1 --2- 172.21.15.116:0/4005680931 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc8d4152db0 0x7fc8d4144060 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:33.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.562+0000 7fc8db3dc640 1 --2- 172.21.15.116:0/4005680931 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc8d40c5a60 0x7fc8d41ee030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:33.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.562+0000 7fc8db3dc640 1 -- 172.21.15.116:0/4005680931 >> 172.21.15.116:0/4005680931 conn(0x7fc8d40acc10 msgr2=0x7fc8d40acff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:27:33.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.562+0000 7fc8db3dc640 1 -- 172.21.15.116:0/4005680931 shutdown_connections 2024-04-25T00:27:33.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:33.562+0000 7fc8db3dc640 1 -- 172.21.15.116:0/4005680931 wait complete. 2024-04-25T00:27:33.586 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:27:35.587 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:27:35.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.742+0000 7fa8bac9e640 1 Processor -- start 2024-04-25T00:27:35.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.742+0000 7fa8bac9e640 1 -- start start 2024-04-25T00:27:35.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.746+0000 7fa8bac9e640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fa8b40a94a0 0x7fa8b41563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:35.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.746+0000 7fa8bac9e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa8b4071680 0x7fa8b4071ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:35.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.746+0000 7fa8bac9e640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8b4071220 con 0x7fa8b414c720 2024-04-25T00:27:35.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.746+0000 7fa8bac9e640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fa8b4153460 con 0x7fa8b40a94a0 2024-04-25T00:27:35.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.746+0000 7fa8bac9e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa8b4152420 con 0x7fa8b4071680 2024-04-25T00:27:35.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.746+0000 7fa8b8a3a640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fa8b414c720 0x7fa8b40c2a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:56364/0 (socket says 172.21.15.116:56364) 2024-04-25T00:27:35.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.746+0000 7fa8b8a3a640 1 -- 172.21.15.116:0/482914095 learned_addr learned my addr 172.21.15.116:0/482914095 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:27:35.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.746+0000 7fa8b3fff640 1 --2- 172.21.15.116:0/482914095 >> v2:172.21.15.174:3300/0 conn(0x7fa8b40a94a0 0x7fa8b41563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:35.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.746+0000 7fa8b37fe640 1 -- 172.21.15.116:0/482914095 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4146565153 0 0) 0x7fa8b4071220 con 0x7fa8b414c720 2024-04-25T00:27:35.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.746+0000 7fa8b37fe640 1 -- 172.21.15.116:0/482914095 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa8a4003530 con 0x7fa8b414c720 2024-04-25T00:27:35.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.746+0000 7fa8b3fff640 1 -- 172.21.15.116:0/482914095 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa8b4071680 msgr2=0x7fa8b4071ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:27:35.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.746+0000 7fa8b3fff640 1 --2- 172.21.15.116:0/482914095 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa8b4071680 0x7fa8b4071ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:35.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.746+0000 7fa8b3fff640 1 -- 172.21.15.116:0/482914095 >> v1:172.21.15.116:6789/0 conn(0x7fa8b414c720 legacy=0x7fa8b40c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:35.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.746+0000 7fa8b3fff640 1 -- 172.21.15.116:0/482914095 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8b414cb00 con 0x7fa8b40a94a0 2024-04-25T00:27:35.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.754+0000 7fa8b3fff640 1 --2- 172.21.15.116:0/482914095 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8b40a94a0 0x7fa8b41563d0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fa8a00097c0 tx=0x7fa8a001c3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=37427a2c7dbcd258 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:35.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.754+0000 7fa8b37fe640 1 -- 172.21.15.116:0/482914095 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8a00b0070 con 0x7fa8b40a94a0 2024-04-25T00:27:35.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.754+0000 7fa8b37fe640 1 -- 172.21.15.116:0/482914095 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8a00ab600 con 0x7fa8b40a94a0 2024-04-25T00:27:35.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.754+0000 7fa8b37fe640 1 -- 172.21.15.116:0/482914095 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8a00ab9b0 con 0x7fa8b40a94a0 2024-04-25T00:27:35.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.754+0000 7fa8bac9e640 1 -- 172.21.15.116:0/482914095 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8b40a94a0 msgr2=0x7fa8b41563d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:35.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.754+0000 7fa8bac9e640 1 --2- 172.21.15.116:0/482914095 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8b40a94a0 0x7fa8b41563d0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fa8a00097c0 tx=0x7fa8a001c3e0 comp rx=0 tx=0).stop 2024-04-25T00:27:35.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.754+0000 7fa8bac9e640 1 -- 172.21.15.116:0/482914095 shutdown_connections 2024-04-25T00:27:35.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.754+0000 7fa8bac9e640 1 --2- 172.21.15.116:0/482914095 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa8b4071680 0x7fa8b4071ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:35.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.754+0000 7fa8bac9e640 1 --2- 172.21.15.116:0/482914095 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8b40a94a0 0x7fa8b41563d0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:35.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.754+0000 7fa8bac9e640 1 -- 172.21.15.116:0/482914095 >> 172.21.15.116:0/482914095 conn(0x7fa8b40c3b00 msgr2=0x7fa8b40c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:27:35.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.754+0000 7fa8bac9e640 1 -- 172.21.15.116:0/482914095 shutdown_connections 2024-04-25T00:27:35.755 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.754+0000 7fa8bac9e640 1 -- 172.21.15.116:0/482914095 wait complete. 2024-04-25T00:27:35.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.758+0000 7fa8bac9e640 1 Processor -- start 2024-04-25T00:27:35.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.758+0000 7fa8bac9e640 1 -- start start 2024-04-25T00:27:35.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.758+0000 7fa8bac9e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8b4071680 0x7fa8b413d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:35.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.758+0000 7fa8bac9e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa8b40a94a0 0x7fa8b414b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:35.760 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.758+0000 7fa8bac9e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8b414c720 0x7fa8b4151540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:35.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.758+0000 7fa8bac9e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa8b40c3120 con 0x7fa8b4071680 2024-04-25T00:27:35.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.758+0000 7fa8bac9e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa8b4152f40 con 0x7fa8b414c720 2024-04-25T00:27:35.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.758+0000 7fa8bac9e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa8b40c32a0 con 0x7fa8b40a94a0 2024-04-25T00:27:35.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.758+0000 7fa8b923b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8b414c720 0x7fa8b4151540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:35.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.758+0000 7fa8b923b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8b414c720 0x7fa8b4151540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:34040/0 (socket says 172.21.15.116:34040) 2024-04-25T00:27:35.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.758+0000 7fa8b923b640 1 -- 172.21.15.116:0/3889878774 learned_addr learned my addr 172.21.15.116:0/3889878774 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:27:35.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.762+0000 7fa8b3fff640 1 --2- 172.21.15.116:0/3889878774 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa8b40a94a0 0x7fa8b414b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:35.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.762+0000 7fa8b8a3a640 1 --2- 172.21.15.116:0/3889878774 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8b4071680 0x7fa8b413d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:35.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.762+0000 7fa8b923b640 1 -- 172.21.15.116:0/3889878774 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa8b40a94a0 msgr2=0x7fa8b414b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:35.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.762+0000 7fa8b923b640 1 --2- 172.21.15.116:0/3889878774 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa8b40a94a0 0x7fa8b414b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:35.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.762+0000 7fa8b923b640 1 -- 172.21.15.116:0/3889878774 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8b4071680 msgr2=0x7fa8b413d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:35.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.762+0000 7fa8b923b640 1 --2- 172.21.15.116:0/3889878774 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8b4071680 0x7fa8b413d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:35.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.762+0000 7fa8b923b640 1 -- 172.21.15.116:0/3889878774 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8b40ad0f0 con 0x7fa8b414c720 2024-04-25T00:27:35.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.762+0000 7fa8b923b640 1 --2- 172.21.15.116:0/3889878774 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8b414c720 0x7fa8b4151540 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac00b970 tx=0x7fa8ac00be30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:35.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.762+0000 7fa8b17fa640 1 -- 172.21.15.116:0/3889878774 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8ac019070 con 0x7fa8b414c720 2024-04-25T00:27:35.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.762+0000 7fa8b17fa640 1 -- 172.21.15.116:0/3889878774 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8ac00cb60 con 0x7fa8b414c720 2024-04-25T00:27:35.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.762+0000 7fa8b17fa640 1 -- 172.21.15.116:0/3889878774 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8ac014440 con 0x7fa8b414c720 2024-04-25T00:27:35.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.762+0000 7fa8bac9e640 1 -- 172.21.15.116:0/3889878774 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8b414bb70 con 0x7fa8b414c720 2024-04-25T00:27:35.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.762+0000 7fa8bac9e640 1 -- 172.21.15.116:0/3889878774 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8b40acaa0 con 0x7fa8b414c720 2024-04-25T00:27:35.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.766+0000 7fa8bac9e640 1 -- 172.21.15.116:0/3889878774 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8940052a0 con 0x7fa8b414c720 2024-04-25T00:27:35.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.778+0000 7fa8b17fa640 1 -- 172.21.15.116:0/3889878774 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa8ac01c070 con 0x7fa8b414c720 2024-04-25T00:27:35.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.778+0000 7fa8b17fa640 1 --2- 172.21.15.116:0/3889878774 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa898067990 0x7fa898069e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:35.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.778+0000 7fa8b17fa640 1 -- 172.21.15.116:0/3889878774 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 10364+0+0 (secure 0 0 0) 0x7fa8ac086ff0 con 0x7fa8b414c720 2024-04-25T00:27:35.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.782+0000 7fa8b8a3a640 1 --2- 172.21.15.116:0/3889878774 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa898067990 0x7fa898069e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:35.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.782+0000 7fa8b8a3a640 1 --2- 172.21.15.116:0/3889878774 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa898067990 0x7fa898069e50 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fa8a80011f0 tx=0x7fa8a8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:35.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:35.782+0000 7fa8b17fa640 1 -- 172.21.15.116:0/3889878774 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fa8ac023020 con 0x7fa8b414c720 2024-04-25T00:27:36.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.062+0000 7fa8bac9e640 1 -- 172.21.15.116:0/3889878774 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa894005590 con 0x7fa8b414c720 2024-04-25T00:27:36.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.062+0000 7fa8b17fa640 1 -- 172.21.15.116:0/3889878774 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 223 v223) ==== 95+0+5672 (secure 0 0 0) 0x7fa8ac056280 con 0x7fa8b414c720 2024-04-25T00:27:36.063 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:27:36.063 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":223,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":223,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:27:35.755060+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6641},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6641":{"gid":6641,"name":"d","rank":0,"incarnation":223,"state":"up:replay","state_seq":1,"addr":"172.21.15.174:6837/2915274073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2915274073},{"type":"v1","addr":"172.21.15.174:6837","nonce":2915274073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":221,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:27:19.052286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8652":{"gid":8652,"name":"c","rank":0,"incarnation":218,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6833/3891103804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3891103804},{"type":"v1","addr":"172.21.15.116:6833","nonce":3891103804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8652,"qdb_cluster":[8652]},"id":2}]} 2024-04-25T00:27:36.063 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 223 2024-04-25T00:27:36.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.066+0000 7fa8bac9e640 1 -- 172.21.15.116:0/3889878774 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa898067990 msgr2=0x7fa898069e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:36.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.066+0000 7fa8bac9e640 1 --2- 172.21.15.116:0/3889878774 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa898067990 0x7fa898069e50 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fa8a80011f0 tx=0x7fa8a8009040 comp rx=0 tx=0).stop 2024-04-25T00:27:36.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.066+0000 7fa8bac9e640 1 -- 172.21.15.116:0/3889878774 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8b414c720 msgr2=0x7fa8b4151540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:36.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.066+0000 7fa8bac9e640 1 --2- 172.21.15.116:0/3889878774 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8b414c720 0x7fa8b4151540 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac00b970 tx=0x7fa8ac00be30 comp rx=0 tx=0).stop 2024-04-25T00:27:36.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.066+0000 7fa8bac9e640 1 -- 172.21.15.116:0/3889878774 shutdown_connections 2024-04-25T00:27:36.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.066+0000 7fa8bac9e640 1 --2- 172.21.15.116:0/3889878774 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa898067990 0x7fa898069e50 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:36.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.066+0000 7fa8bac9e640 1 --2- 172.21.15.116:0/3889878774 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8b414c720 0x7fa8b4151540 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:36.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.066+0000 7fa8bac9e640 1 --2- 172.21.15.116:0/3889878774 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa8b40a94a0 0x7fa8b414b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:36.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.066+0000 7fa8bac9e640 1 --2- 172.21.15.116:0/3889878774 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8b4071680 0x7fa8b413d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:36.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.066+0000 7fa8bac9e640 1 -- 172.21.15.116:0/3889878774 >> 172.21.15.116:0/3889878774 conn(0x7fa8b40c3b00 msgr2=0x7fa8b4072d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:27:36.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.066+0000 7fa8bac9e640 1 -- 172.21.15.116:0/3889878774 shutdown_connections 2024-04-25T00:27:36.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.066+0000 7fa8bac9e640 1 -- 172.21.15.116:0/3889878774 wait complete. 2024-04-25T00:27:36.085 INFO:tasks.mds_thrash.fs.[a]:mds.a down, removed from mdsmap 2024-04-25T00:27:36.086 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-04-25T00:27:36.086 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:27:36.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.218+0000 7f866e6fd640 1 Processor -- start 2024-04-25T00:27:36.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.218+0000 7f866e6fd640 1 -- start start 2024-04-25T00:27:36.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.222+0000 7f866e6fd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8668070dd0 0x7f8668071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:36.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.222+0000 7f866e6fd640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8668154df0 con 0x7f866806e100 2024-04-25T00:27:36.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.222+0000 7f866e6fd640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86681550d0 con 0x7f866814b830 2024-04-25T00:27:36.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.222+0000 7f866e6fd640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f866814bc10 con 0x7f8668070dd0 2024-04-25T00:27:36.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.222+0000 7f86677fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f866806e100 0x7f86680c2a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:56368/0 (socket says 172.21.15.116:56368) 2024-04-25T00:27:36.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.222+0000 7f86677fe640 1 -- 172.21.15.116:0/1533664134 learned_addr learned my addr 172.21.15.116:0/1533664134 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:27:36.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.222+0000 7f866cc9a640 1 --2- 172.21.15.116:0/1533664134 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8668070dd0 0x7f8668071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:36.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.222+0000 7f8666ffd640 1 -- 172.21.15.116:0/1533664134 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2332565293 0 0) 0x7f8668154df0 con 0x7f866806e100 2024-04-25T00:27:36.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.222+0000 7f8666ffd640 1 -- 172.21.15.116:0/1533664134 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8650003530 con 0x7f866806e100 2024-04-25T00:27:36.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.222+0000 7f8666ffd640 1 -- 172.21.15.116:0/1533664134 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 923252532 0 0) 0x7f86681550d0 con 0x7f866814b830 2024-04-25T00:27:36.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.222+0000 7f8666ffd640 1 -- 172.21.15.116:0/1533664134 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8668154df0 con 0x7f866814b830 2024-04-25T00:27:36.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.222+0000 7f8666ffd640 1 -- 172.21.15.116:0/1533664134 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3833795012 0 0) 0x7f8650003530 con 0x7f866806e100 2024-04-25T00:27:36.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.222+0000 7f8666ffd640 1 -- 172.21.15.116:0/1533664134 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8668070dd0 msgr2=0x7f8668071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:27:36.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.222+0000 7f8666ffd640 1 --2- 172.21.15.116:0/1533664134 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8668070dd0 0x7f8668071260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:36.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.222+0000 7f8666ffd640 1 -- 172.21.15.116:0/1533664134 >> v1:172.21.15.174:6789/0 conn(0x7f866814b830 legacy=0x7f8668152150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:36.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.222+0000 7f8666ffd640 1 -- 172.21.15.116:0/1533664134 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86681553b0 con 0x7f866806e100 2024-04-25T00:27:36.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.222+0000 7f8666ffd640 1 -- 172.21.15.116:0/1533664134 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f865c001680 con 0x7f866806e100 2024-04-25T00:27:36.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.222+0000 7f8666ffd640 1 -- 172.21.15.116:0/1533664134 >> v1:172.21.15.116:6789/0 conn(0x7f866806e100 legacy=0x7f86680c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:36.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.222+0000 7f8666ffd640 1 --2- 172.21.15.116:0/1533664134 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8650003b80 0x7f8650003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:36.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.222+0000 7f8666ffd640 1 --2- 172.21.15.116:0/1533664134 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8668070dd0 0x7f8650004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:36.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.222+0000 7f8666ffd640 1 --2- 172.21.15.116:0/1533664134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8650004d50 0x7f86500051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:36.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.222+0000 7f8666ffd640 1 -- 172.21.15.116:0/1533664134 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f866814bc10 con 0x7f8650004d50 2024-04-25T00:27:36.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.222+0000 7f8666ffd640 1 -- 172.21.15.116:0/1533664134 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f865000de20 con 0x7f8650003b80 2024-04-25T00:27:36.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.222+0000 7f8666ffd640 1 -- 172.21.15.116:0/1533664134 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f865000e020 con 0x7f8668070dd0 2024-04-25T00:27:36.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.222+0000 7f866cc9a640 1 --2- 172.21.15.116:0/1533664134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8650004d50 0x7f86500051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:36.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.222+0000 7f8667fff640 1 --2- 172.21.15.116:0/1533664134 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8650003b80 0x7f8650003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:36.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.222+0000 7f86677fe640 1 --2- 172.21.15.116:0/1533664134 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8668070dd0 0x7f8650004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:36.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.222+0000 7f866cc9a640 1 -- 172.21.15.116:0/1533664134 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8668070dd0 msgr2=0x7f8650004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:27:36.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.222+0000 7f866cc9a640 1 --2- 172.21.15.116:0/1533664134 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8668070dd0 0x7f8650004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:36.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.222+0000 7f866cc9a640 1 -- 172.21.15.116:0/1533664134 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8650003b80 msgr2=0x7f8650003f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:27:36.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.222+0000 7f866cc9a640 1 --2- 172.21.15.116:0/1533664134 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8650003b80 0x7f8650003f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:36.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.222+0000 7f866cc9a640 1 -- 172.21.15.116:0/1533664134 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8650003530 con 0x7f8650004d50 2024-04-25T00:27:36.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.222+0000 7f8667fff640 1 --2- 172.21.15.116:0/1533664134 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8650003b80 0x7f8650003f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:27:36.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.226+0000 7f866cc9a640 1 --2- 172.21.15.116:0/1533664134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8650004d50 0x7f86500051d0 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f8658009ae0 tx=0x7f865801ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1248c76087203f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:36.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.226+0000 7f8666ffd640 1 -- 172.21.15.116:0/1533664134 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86580096b0 con 0x7f8650004d50 2024-04-25T00:27:36.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.226+0000 7f8666ffd640 1 -- 172.21.15.116:0/1533664134 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86580a6450 con 0x7f8650004d50 2024-04-25T00:27:36.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.226+0000 7f8666ffd640 1 -- 172.21.15.116:0/1533664134 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86580a6840 con 0x7f8650004d50 2024-04-25T00:27:36.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.226+0000 7f866e6fd640 1 -- 172.21.15.116:0/1533664134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8650004d50 msgr2=0x7f86500051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:36.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.226+0000 7f866e6fd640 1 --2- 172.21.15.116:0/1533664134 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8650004d50 0x7f86500051d0 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f8658009ae0 tx=0x7f865801ca70 comp rx=0 tx=0).stop 2024-04-25T00:27:36.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.226+0000 7f866cc9a640 1 -- 172.21.15.116:0/1533664134 reap_dead start 2024-04-25T00:27:36.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.226+0000 7f866e6fd640 1 -- 172.21.15.116:0/1533664134 shutdown_connections 2024-04-25T00:27:36.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.226+0000 7f866e6fd640 1 -- 172.21.15.116:0/1533664134 >> 172.21.15.116:0/1533664134 conn(0x7f86680c3b00 msgr2=0x7f86680c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:27:36.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.226+0000 7f866e6fd640 1 -- 172.21.15.116:0/1533664134 shutdown_connections 2024-04-25T00:27:36.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.226+0000 7f866e6fd640 1 -- 172.21.15.116:0/1533664134 wait complete. 2024-04-25T00:27:36.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.230+0000 7f866e6fd640 1 Processor -- start 2024-04-25T00:27:36.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.230+0000 7f866e6fd640 1 -- start start 2024-04-25T00:27:36.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.230+0000 7f866e6fd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f866814ec50 0x7f866814f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:36.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.230+0000 7f866e6fd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f866814bd90 0x7f86680aa980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:36.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.230+0000 7f866e6fd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f866814c210 0x7f86681470c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:36.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.230+0000 7f866e6fd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f866815d3f0 con 0x7f866814c210 2024-04-25T00:27:36.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.230+0000 7f866e6fd640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f866815d270 con 0x7f866814ec50 2024-04-25T00:27:36.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.230+0000 7f866e6fd640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f866815d570 con 0x7f866814bd90 2024-04-25T00:27:36.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.230+0000 7f8667fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f866814ec50 0x7f866814f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:36.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.230+0000 7f8667fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f866814ec50 0x7f866814f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:34064/0 (socket says 172.21.15.116:34064) 2024-04-25T00:27:36.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.230+0000 7f8667fff640 1 -- 172.21.15.116:0/1285968989 learned_addr learned my addr 172.21.15.116:0/1285968989 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:27:36.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.230+0000 7f86677fe640 1 --2- 172.21.15.116:0/1285968989 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f866814bd90 0x7f86680aa980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:36.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.230+0000 7f866cc9a640 1 --2- 172.21.15.116:0/1285968989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f866814c210 0x7f86681470c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:36.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.230+0000 7f866cc9a640 1 -- 172.21.15.116:0/1285968989 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f866814bd90 msgr2=0x7f86680aa980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:36.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.230+0000 7f866cc9a640 1 --2- 172.21.15.116:0/1285968989 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f866814bd90 0x7f86680aa980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:36.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.230+0000 7f866cc9a640 1 -- 172.21.15.116:0/1285968989 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f866814ec50 msgr2=0x7f866814f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:36.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.230+0000 7f866cc9a640 1 --2- 172.21.15.116:0/1285968989 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f866814ec50 0x7f866814f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:36.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.230+0000 7f866cc9a640 1 -- 172.21.15.116:0/1285968989 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86680ac760 con 0x7f866814c210 2024-04-25T00:27:36.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.230+0000 7f8667fff640 1 --2- 172.21.15.116:0/1285968989 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f866814ec50 0x7f866814f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:27:36.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.230+0000 7f866cc9a640 1 --2- 172.21.15.116:0/1285968989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f866814c210 0x7f86681470c0 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f8658008000 tx=0x7f8658019810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:36.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.230+0000 7f8664ff9640 1 -- 172.21.15.116:0/1285968989 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8660001c20 con 0x7f866814c210 2024-04-25T00:27:36.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.230+0000 7f8664ff9640 1 -- 172.21.15.116:0/1285968989 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86580a6b40 con 0x7f866814c210 2024-04-25T00:27:36.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.230+0000 7f8664ff9640 1 -- 172.21.15.116:0/1285968989 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86580096b0 con 0x7f866814c210 2024-04-25T00:27:36.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.230+0000 7f866e6fd640 1 -- 172.21.15.116:0/1285968989 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86680aca40 con 0x7f866814c210 2024-04-25T00:27:36.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.230+0000 7f866e6fd640 1 -- 172.21.15.116:0/1285968989 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86680ad000 con 0x7f866814c210 2024-04-25T00:27:36.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.230+0000 7f866e6fd640 1 -- 172.21.15.116:0/1285968989 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8668068350 con 0x7f866814c210 2024-04-25T00:27:36.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.230+0000 7f8664ff9640 1 -- 172.21.15.116:0/1285968989 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8658004020 con 0x7f866814c210 2024-04-25T00:27:36.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.234+0000 7f8664ff9640 1 --2- 172.21.15.116:0/1285968989 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f86540678c0 0x7f8654069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:36.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.234+0000 7f8664ff9640 1 -- 172.21.15.116:0/1285968989 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 10364+0+0 (secure 0 0 0) 0x7f86581206c0 con 0x7f866814c210 2024-04-25T00:27:36.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.234+0000 7f8667fff640 1 --2- 172.21.15.116:0/1285968989 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f86540678c0 0x7f8654069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:36.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.234+0000 7f8667fff640 1 --2- 172.21.15.116:0/1285968989 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f86540678c0 0x7f8654069d80 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f8660001470 tx=0x7f8660009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:36.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.238+0000 7f8664ff9640 1 -- 172.21.15.116:0/1285968989 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f86580ac020 con 0x7f866814c210 2024-04-25T00:27:36.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.506+0000 7f866e6fd640 1 -- 172.21.15.116:0/1285968989 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8668151920 con 0x7f866814c210 2024-04-25T00:27:36.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.510+0000 7f8664ff9640 1 -- 172.21.15.116:0/1285968989 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 223 v223) ==== 95+0+5672 (secure 0 0 0) 0x7f8658016780 con 0x7f866814c210 2024-04-25T00:27:36.509 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:27:36.509 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":223,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":223,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:27:35.755060+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6641},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6641":{"gid":6641,"name":"d","rank":0,"incarnation":223,"state":"up:replay","state_seq":1,"addr":"172.21.15.174:6837/2915274073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2915274073},{"type":"v1","addr":"172.21.15.174:6837","nonce":2915274073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":221,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:27:19.052286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8652":{"gid":8652,"name":"c","rank":0,"incarnation":218,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6833/3891103804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3891103804},{"type":"v1","addr":"172.21.15.116:6833","nonce":3891103804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8652,"qdb_cluster":[8652]},"id":2}]} 2024-04-25T00:27:36.510 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 223 2024-04-25T00:27:36.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.514+0000 7f866e6fd640 1 -- 172.21.15.116:0/1285968989 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f86540678c0 msgr2=0x7f8654069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:36.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.514+0000 7f866e6fd640 1 --2- 172.21.15.116:0/1285968989 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f86540678c0 0x7f8654069d80 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f8660001470 tx=0x7f8660009040 comp rx=0 tx=0).stop 2024-04-25T00:27:36.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.514+0000 7f866e6fd640 1 -- 172.21.15.116:0/1285968989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f866814c210 msgr2=0x7f86681470c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:36.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.514+0000 7f866e6fd640 1 --2- 172.21.15.116:0/1285968989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f866814c210 0x7f86681470c0 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f8658008000 tx=0x7f8658019810 comp rx=0 tx=0).stop 2024-04-25T00:27:36.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.514+0000 7f866e6fd640 1 -- 172.21.15.116:0/1285968989 shutdown_connections 2024-04-25T00:27:36.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.514+0000 7f866e6fd640 1 --2- 172.21.15.116:0/1285968989 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f86540678c0 0x7f8654069d80 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:36.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.514+0000 7f866e6fd640 1 --2- 172.21.15.116:0/1285968989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f866814c210 0x7f86681470c0 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:36.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.514+0000 7f866e6fd640 1 --2- 172.21.15.116:0/1285968989 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f866814bd90 0x7f86680aa980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:36.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.514+0000 7f866e6fd640 1 --2- 172.21.15.116:0/1285968989 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f866814ec50 0x7f866814f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:36.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.514+0000 7f866e6fd640 1 -- 172.21.15.116:0/1285968989 >> 172.21.15.116:0/1285968989 conn(0x7f86680c3b00 msgr2=0x7f8668149c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:27:36.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.514+0000 7f866e6fd640 1 -- 172.21.15.116:0/1285968989 shutdown_connections 2024-04-25T00:27:36.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:36.514+0000 7f866e6fd640 1 -- 172.21.15.116:0/1285968989 wait complete. 2024-04-25T00:27:36.533 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 223, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:04.678737+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 223, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6641": { "addr": "172.21.15.174:6837/2915274073", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6836", "nonce": 2915274073, "type": "v2" }, { "addr": "172.21.15.174:6837", "nonce": 2915274073, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6641, "incarnation": 223, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 71, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T00:27:35.755060+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6641 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:18.825887+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 221, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_8652": { "addr": "172.21.15.116:6833/3891103804", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6832", "nonce": 3891103804, "type": "v2" }, { "addr": "172.21.15.116:6833", "nonce": 3891103804, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8652, "incarnation": 218, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 9 } }, "last_failure": 0, "last_failure_osd_epoch": 70, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-04-25T00:27:19.052286+0000", "qdb_cluster": [ 8652 ], "qdb_leader": 8652, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8652 } } } ], "standbys": [ { "addr": "172.21.15.174:6835/4112145782", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6834", "nonce": 4112145782, "type": "v2" }, { "addr": "172.21.15.174:6835", "nonce": 4112145782, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 112, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5582, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-25T00:27:38.534 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:27:38.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.694+0000 7f591b784640 1 Processor -- start 2024-04-25T00:27:38.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.694+0000 7f591b784640 1 -- start start 2024-04-25T00:27:38.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.694+0000 7f591b784640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f591414f300 0x7f591406c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:38.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.694+0000 7f591b784640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f591406c720 0x7f5914155870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:38.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.694+0000 7f5919520640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f591414f300 0x7f591406c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:38.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.694+0000 7f5919520640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f591414f300 0x7f591406c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:55676/0 (socket says 172.21.15.116:55676) 2024-04-25T00:27:38.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.694+0000 7f5918d1f640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f59140c2d10 0x7f5914155160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:57402/0 (socket says 172.21.15.116:57402) 2024-04-25T00:27:38.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.694+0000 7f5919d21640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f591406c720 0x7f5914155870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:38.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.694+0000 7f5919d21640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f591406c720 0x7f5914155870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47854/0 (socket says 172.21.15.116:47854) 2024-04-25T00:27:38.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.694+0000 7f5919520640 1 -- 172.21.15.116:0/2888855589 learned_addr learned my addr 172.21.15.116:0/2888855589 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:27:38.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.694+0000 7f591b784640 1 -- 172.21.15.116:0/2888855589 --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5914155e50 con 0x7f59140c2d10 2024-04-25T00:27:38.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.694+0000 7f591b784640 1 -- 172.21.15.116:0/2888855589 --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f59140c3780 con 0x7f591406c720 2024-04-25T00:27:38.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.694+0000 7f591b784640 1 -- 172.21.15.116:0/2888855589 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f59140c3a60 con 0x7f591414f300 2024-04-25T00:27:38.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.694+0000 7f5913fff640 1 -- 172.21.15.116:0/2888855589 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1152669001 0 0) 0x7f5914155e50 con 0x7f59140c2d10 2024-04-25T00:27:38.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.694+0000 7f5913fff640 1 -- 172.21.15.116:0/2888855589 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f58fc003530 con 0x7f59140c2d10 2024-04-25T00:27:38.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.694+0000 7f5919520640 1 -- 172.21.15.116:0/2888855589 >> v2:172.21.15.116:3300/0 conn(0x7f591406c720 msgr2=0x7f5914155870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:27:38.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.694+0000 7f5919520640 1 --2- 172.21.15.116:0/2888855589 >> v2:172.21.15.116:3300/0 conn(0x7f591406c720 0x7f5914155870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:38.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.694+0000 7f5919520640 1 -- 172.21.15.116:0/2888855589 >> v1:172.21.15.174:6789/0 conn(0x7f59140c2d10 legacy=0x7f5914155160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:38.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.694+0000 7f5919520640 1 -- 172.21.15.116:0/2888855589 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f591406bcc0 con 0x7f591414f300 2024-04-25T00:27:38.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.698+0000 7f5919520640 1 --2- 172.21.15.116:0/2888855589 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f591414f300 0x7f591406c1e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f590c009870 tx=0x7f590c01c460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5934e58bdafacb4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:38.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.698+0000 7f5913fff640 1 -- 172.21.15.116:0/2888855589 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f590c0b0070 con 0x7f591414f300 2024-04-25T00:27:38.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.698+0000 7f5913fff640 1 -- 172.21.15.116:0/2888855589 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f590c0ab600 con 0x7f591414f300 2024-04-25T00:27:38.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.698+0000 7f5913fff640 1 -- 172.21.15.116:0/2888855589 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f590c0ab9d0 con 0x7f591414f300 2024-04-25T00:27:38.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.702+0000 7f591b784640 1 -- 172.21.15.116:0/2888855589 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f591414f300 msgr2=0x7f591406c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:38.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.702+0000 7f591b784640 1 --2- 172.21.15.116:0/2888855589 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f591414f300 0x7f591406c1e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f590c009870 tx=0x7f590c01c460 comp rx=0 tx=0).stop 2024-04-25T00:27:38.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.702+0000 7f591b784640 1 -- 172.21.15.116:0/2888855589 shutdown_connections 2024-04-25T00:27:38.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.702+0000 7f591b784640 1 --2- 172.21.15.116:0/2888855589 >> v2:172.21.15.116:3300/0 conn(0x7f591406c720 0x7f5914155870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:38.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.702+0000 7f591b784640 1 --2- 172.21.15.116:0/2888855589 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f591414f300 0x7f591406c1e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:38.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.702+0000 7f591b784640 1 -- 172.21.15.116:0/2888855589 >> 172.21.15.116:0/2888855589 conn(0x7f59140c4200 msgr2=0x7f59140c4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:27:38.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.702+0000 7f591b784640 1 -- 172.21.15.116:0/2888855589 shutdown_connections 2024-04-25T00:27:38.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.702+0000 7f591b784640 1 -- 172.21.15.116:0/2888855589 wait complete. 2024-04-25T00:27:38.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.702+0000 7f591b784640 1 Processor -- start 2024-04-25T00:27:38.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.702+0000 7f591b784640 1 -- start start 2024-04-25T00:27:38.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.702+0000 7f591b784640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f591406c720 0x7f59140a82c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:38.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.702+0000 7f591b784640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59140c2d10 0x7f59140c5d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:38.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.702+0000 7f591b784640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f591414f300 0x7f59140bfc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:38.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.702+0000 7f591b784640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f59140716a0 con 0x7f591414f300 2024-04-25T00:27:38.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.702+0000 7f591b784640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5914071520 con 0x7f59140c2d10 2024-04-25T00:27:38.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.702+0000 7f591b784640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f591415e500 con 0x7f591406c720 2024-04-25T00:27:38.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.702+0000 7f5919d21640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f591414f300 0x7f59140bfc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:38.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.702+0000 7f5919d21640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f591414f300 0x7f59140bfc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47864/0 (socket says 172.21.15.116:47864) 2024-04-25T00:27:38.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.702+0000 7f5919d21640 1 -- 172.21.15.116:0/2316804581 learned_addr learned my addr 172.21.15.116:0/2316804581 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:27:38.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.702+0000 7f5918d1f640 1 --2- 172.21.15.116:0/2316804581 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59140c2d10 0x7f59140c5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:38.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.702+0000 7f5919d21640 1 -- 172.21.15.116:0/2316804581 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f591406c720 msgr2=0x7f59140a82c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:38.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.702+0000 7f5919d21640 1 --2- 172.21.15.116:0/2316804581 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f591406c720 0x7f59140a82c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:38.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.702+0000 7f5919d21640 1 -- 172.21.15.116:0/2316804581 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59140c2d10 msgr2=0x7f59140c5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:38.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.702+0000 7f5919d21640 1 --2- 172.21.15.116:0/2316804581 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59140c2d10 0x7f59140c5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:38.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.702+0000 7f5919d21640 1 -- 172.21.15.116:0/2316804581 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59140bdaa0 con 0x7f591414f300 2024-04-25T00:27:38.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.702+0000 7f5919d21640 1 --2- 172.21.15.116:0/2316804581 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f591414f300 0x7f59140bfc20 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f5904009dd0 tx=0x7f590400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:38.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.702+0000 7f5911ffb640 1 -- 172.21.15.116:0/2316804581 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5904018020 con 0x7f591414f300 2024-04-25T00:27:38.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.702+0000 7f5911ffb640 1 -- 172.21.15.116:0/2316804581 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5904014480 con 0x7f591414f300 2024-04-25T00:27:38.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.706+0000 7f591b784640 1 -- 172.21.15.116:0/2316804581 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59140bdd80 con 0x7f591414f300 2024-04-25T00:27:38.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.706+0000 7f5911ffb640 1 -- 172.21.15.116:0/2316804581 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5904014870 con 0x7f591414f300 2024-04-25T00:27:38.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.706+0000 7f591b784640 1 -- 172.21.15.116:0/2316804581 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59140bf420 con 0x7f591414f300 2024-04-25T00:27:38.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.706+0000 7f5911ffb640 1 -- 172.21.15.116:0/2316804581 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f590401b020 con 0x7f591414f300 2024-04-25T00:27:38.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.706+0000 7f5911ffb640 1 --2- 172.21.15.116:0/2316804581 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f59000678c0 0x7f5900069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:38.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.706+0000 7f5911ffb640 1 -- 172.21.15.116:0/2316804581 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 10364+0+0 (secure 0 0 0) 0x7f5904087830 con 0x7f591414f300 2024-04-25T00:27:38.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.706+0000 7f591b784640 1 -- 172.21.15.116:0/2316804581 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5914068a60 con 0x7f591414f300 2024-04-25T00:27:38.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.706+0000 7f5919520640 1 --2- 172.21.15.116:0/2316804581 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f59000678c0 0x7f5900069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:38.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.706+0000 7f5919520640 1 --2- 172.21.15.116:0/2316804581 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f59000678c0 0x7f5900069d80 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f590c0ad000 tx=0x7f590c004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:38.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:38.718+0000 7f5911ffb640 1 -- 172.21.15.116:0/2316804581 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f5904022070 con 0x7f591414f300 2024-04-25T00:27:39.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:39.018+0000 7f591b784640 1 -- 172.21.15.116:0/2316804581 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f59140c31b0 con 0x7f591414f300 2024-04-25T00:27:39.021 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:39.022+0000 7f5911ffb640 1 -- 172.21.15.116:0/2316804581 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 224 v224) ==== 95+0+5675 (secure 0 0 0) 0x7f5904010040 con 0x7f591414f300 2024-04-25T00:27:39.021 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:27:39.021 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":224,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:27:37.591734+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6641},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6641":{"gid":6641,"name":"d","rank":0,"incarnation":223,"state":"up:reconnect","state_seq":5,"addr":"172.21.15.174:6837/2915274073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2915274073},{"type":"v1","addr":"172.21.15.174:6837","nonce":2915274073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":221,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:27:19.052286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8652":{"gid":8652,"name":"c","rank":0,"incarnation":218,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6833/3891103804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3891103804},{"type":"v1","addr":"172.21.15.116:6833","nonce":3891103804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8652,"qdb_cluster":[8652]},"id":2}]} 2024-04-25T00:27:39.021 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 224 2024-04-25T00:27:39.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:39.022+0000 7f591b784640 1 -- 172.21.15.116:0/2316804581 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f59000678c0 msgr2=0x7f5900069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:39.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:39.022+0000 7f591b784640 1 --2- 172.21.15.116:0/2316804581 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f59000678c0 0x7f5900069d80 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f590c0ad000 tx=0x7f590c004740 comp rx=0 tx=0).stop 2024-04-25T00:27:39.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:39.022+0000 7f591b784640 1 -- 172.21.15.116:0/2316804581 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f591414f300 msgr2=0x7f59140bfc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:39.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:39.022+0000 7f591b784640 1 --2- 172.21.15.116:0/2316804581 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f591414f300 0x7f59140bfc20 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f5904009dd0 tx=0x7f590400c490 comp rx=0 tx=0).stop 2024-04-25T00:27:39.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:39.022+0000 7f591b784640 1 -- 172.21.15.116:0/2316804581 shutdown_connections 2024-04-25T00:27:39.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:39.022+0000 7f591b784640 1 --2- 172.21.15.116:0/2316804581 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f59000678c0 0x7f5900069d80 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:39.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:39.022+0000 7f591b784640 1 --2- 172.21.15.116:0/2316804581 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f591414f300 0x7f59140bfc20 unknown :-1 s=CLOSED pgs=1630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:39.024 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:39.022+0000 7f591b784640 1 --2- 172.21.15.116:0/2316804581 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59140c2d10 0x7f59140c5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:39.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:39.022+0000 7f591b784640 1 --2- 172.21.15.116:0/2316804581 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f591406c720 0x7f59140a82c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:39.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:39.022+0000 7f591b784640 1 -- 172.21.15.116:0/2316804581 >> 172.21.15.116:0/2316804581 conn(0x7f59140c4200 msgr2=0x7f591415f250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:27:39.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:39.026+0000 7f591b784640 1 -- 172.21.15.116:0/2316804581 shutdown_connections 2024-04-25T00:27:39.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:39.026+0000 7f591b784640 1 -- 172.21.15.116:0/2316804581 wait complete. 2024-04-25T00:27:39.049 INFO:tasks.mds_thrash.fs.[a]:no change 2024-04-25T00:27:41.050 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:27:41.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.189+0000 7fd29f501640 1 Processor -- start 2024-04-25T00:27:41.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.189+0000 7fd29f501640 1 -- start start 2024-04-25T00:27:41.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.189+0000 7fd29f501640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd298072c90 0x7fd298150f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:41.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.189+0000 7fd29f501640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fd298072810 0x7fd298153760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:41.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.189+0000 7fd29f501640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2980c2c40 con 0x7fd2980c5280 2024-04-25T00:27:41.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.189+0000 7fd29f501640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fd298157c20 con 0x7fd298072c90 2024-04-25T00:27:41.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.189+0000 7fd29f501640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fd2980c2f70 con 0x7fd298072810 2024-04-25T00:27:41.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.193+0000 7fd29ca9c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd298072c90 0x7fd298150f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:41.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.193+0000 7fd29d29d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fd2980c5280 0x7fd298150840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:56382/0 (socket says 172.21.15.116:56382) 2024-04-25T00:27:41.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.193+0000 7fd29d29d640 1 -- 172.21.15.116:0/3552909651 learned_addr learned my addr 172.21.15.116:0/3552909651 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:27:41.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.193+0000 7fd28ffff640 1 -- 172.21.15.116:0/3552909651 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3403251528 0 0) 0x7fd2980c2c40 con 0x7fd2980c5280 2024-04-25T00:27:41.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.193+0000 7fd28ffff640 1 -- 172.21.15.116:0/3552909651 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd284003530 con 0x7fd2980c5280 2024-04-25T00:27:41.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.193+0000 7fd28ffff640 1 -- 172.21.15.116:0/3552909651 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3465044071 0 0) 0x7fd284003530 con 0x7fd2980c5280 2024-04-25T00:27:41.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.193+0000 7fd29ca9c640 1 -- 172.21.15.116:0/3552909651 >> v2:172.21.15.174:3300/0 conn(0x7fd298072810 msgr2=0x7fd298153760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:27:41.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.193+0000 7fd29ca9c640 1 --2- 172.21.15.116:0/3552909651 >> v2:172.21.15.174:3300/0 conn(0x7fd298072810 0x7fd298153760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:41.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.193+0000 7fd29ca9c640 1 -- 172.21.15.116:0/3552909651 >> v1:172.21.15.116:6789/0 conn(0x7fd2980c5280 legacy=0x7fd298150840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:41.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.193+0000 7fd29ca9c640 1 -- 172.21.15.116:0/3552909651 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd298157660 con 0x7fd298072c90 2024-04-25T00:27:41.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.193+0000 7fd29ca9c640 1 --2- 172.21.15.116:0/3552909651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd298072c90 0x7fd298150f50 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7fd2980c36e0 tx=0x7fd28801c250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a86c586820918003 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:41.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.193+0000 7fd28ffff640 1 -- 172.21.15.116:0/3552909651 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2880b0070 con 0x7fd298072c90 2024-04-25T00:27:41.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.193+0000 7fd28ffff640 1 -- 172.21.15.116:0/3552909651 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2880ab400 con 0x7fd298072c90 2024-04-25T00:27:41.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.193+0000 7fd28ffff640 1 -- 172.21.15.116:0/3552909651 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2940016a0 con 0x7fd298072c90 2024-04-25T00:27:41.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.197+0000 7fd29f501640 1 -- 172.21.15.116:0/3552909651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd298072c90 msgr2=0x7fd298150f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:41.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.197+0000 7fd29f501640 1 --2- 172.21.15.116:0/3552909651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd298072c90 0x7fd298150f50 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7fd2980c36e0 tx=0x7fd28801c250 comp rx=0 tx=0).stop 2024-04-25T00:27:41.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.197+0000 7fd29f501640 1 -- 172.21.15.116:0/3552909651 shutdown_connections 2024-04-25T00:27:41.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.197+0000 7fd29f501640 1 --2- 172.21.15.116:0/3552909651 >> v2:172.21.15.174:3300/0 conn(0x7fd298072810 0x7fd298153760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:41.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.197+0000 7fd29f501640 1 --2- 172.21.15.116:0/3552909651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd298072c90 0x7fd298150f50 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:41.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.197+0000 7fd29f501640 1 -- 172.21.15.116:0/3552909651 >> 172.21.15.116:0/3552909651 conn(0x7fd29806c720 msgr2=0x7fd29806cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:27:41.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.197+0000 7fd29f501640 1 -- 172.21.15.116:0/3552909651 shutdown_connections 2024-04-25T00:27:41.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.197+0000 7fd29f501640 1 -- 172.21.15.116:0/3552909651 wait complete. 2024-04-25T00:27:41.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.197+0000 7fd29f501640 1 Processor -- start 2024-04-25T00:27:41.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.197+0000 7fd29f501640 1 -- start start 2024-04-25T00:27:41.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.197+0000 7fd29f501640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd298072810 0x7fd298142b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:41.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.197+0000 7fd29f501640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd298072c90 0x7fd2980c7360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:41.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.197+0000 7fd29f501640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2980c5280 0x7fd2980c78a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:41.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.197+0000 7fd29f501640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd298160380 con 0x7fd2980c5280 2024-04-25T00:27:41.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.197+0000 7fd29f501640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd298160200 con 0x7fd298072c90 2024-04-25T00:27:41.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.197+0000 7fd29f501640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fd298160500 con 0x7fd298072810 2024-04-25T00:27:41.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.197+0000 7fd29da9e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2980c5280 0x7fd2980c78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:41.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.197+0000 7fd29da9e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2980c5280 0x7fd2980c78a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47880/0 (socket says 172.21.15.116:47880) 2024-04-25T00:27:41.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.197+0000 7fd29da9e640 1 -- 172.21.15.116:0/3331341508 learned_addr learned my addr 172.21.15.116:0/3331341508 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:27:41.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.197+0000 7fd29ca9c640 1 --2- 172.21.15.116:0/3331341508 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd298072c90 0x7fd2980c7360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:41.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.197+0000 7fd29d29d640 1 --2- 172.21.15.116:0/3331341508 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd298072810 0x7fd298142b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:41.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.201+0000 7fd29d29d640 1 -- 172.21.15.116:0/3331341508 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd298072c90 msgr2=0x7fd2980c7360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:41.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.201+0000 7fd29d29d640 1 --2- 172.21.15.116:0/3331341508 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd298072c90 0x7fd2980c7360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:41.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.201+0000 7fd29d29d640 1 -- 172.21.15.116:0/3331341508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2980c5280 msgr2=0x7fd2980c78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:41.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.201+0000 7fd29d29d640 1 --2- 172.21.15.116:0/3331341508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2980c5280 0x7fd2980c78a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:41.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.201+0000 7fd29d29d640 1 -- 172.21.15.116:0/3331341508 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2981551b0 con 0x7fd298072810 2024-04-25T00:27:41.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.201+0000 7fd29da9e640 1 --2- 172.21.15.116:0/3331341508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2980c5280 0x7fd2980c78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:27:41.204 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.201+0000 7fd29d29d640 1 --2- 172.21.15.116:0/3331341508 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd298072810 0x7fd298142b50 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd2980c2e10 tx=0x7fd29400cbe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:41.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.205+0000 7fd28dffb640 1 -- 172.21.15.116:0/3331341508 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd294018070 con 0x7fd298072810 2024-04-25T00:27:41.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.205+0000 7fd28dffb640 1 -- 172.21.15.116:0/3331341508 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2940072f0 con 0x7fd298072810 2024-04-25T00:27:41.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.205+0000 7fd28dffb640 1 -- 172.21.15.116:0/3331341508 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2940047f0 con 0x7fd298072810 2024-04-25T00:27:41.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.209+0000 7fd29f501640 1 -- 172.21.15.116:0/3331341508 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2981554f0 con 0x7fd298072810 2024-04-25T00:27:41.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.209+0000 7fd29f501640 1 -- 172.21.15.116:0/3331341508 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd298154bf0 con 0x7fd298072810 2024-04-25T00:27:41.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.209+0000 7fd27b7fe640 1 -- 172.21.15.116:0/3331341508 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd298068a60 con 0x7fd298072810 2024-04-25T00:27:41.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.209+0000 7fd28dffb640 1 -- 172.21.15.116:0/3331341508 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd29401b070 con 0x7fd298072810 2024-04-25T00:27:41.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.209+0000 7fd28dffb640 1 --2- 172.21.15.116:0/3331341508 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd27c067910 0x7fd27c069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:41.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.209+0000 7fd28dffb640 1 -- 172.21.15.116:0/3331341508 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(71..71 src has 1..71) ==== 10364+0+0 (secure 0 0 0) 0x7fd2940862b0 con 0x7fd298072810 2024-04-25T00:27:41.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.209+0000 7fd29ca9c640 1 --2- 172.21.15.116:0/3331341508 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd27c067910 0x7fd27c069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:41.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.213+0000 7fd29ca9c640 1 --2- 172.21.15.116:0/3331341508 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd27c067910 0x7fd27c069dd0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fd2880ad000 tx=0x7fd2880a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:41.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.225+0000 7fd28dffb640 1 -- 172.21.15.116:0/3331341508 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fd294055630 con 0x7fd298072810 2024-04-25T00:27:41.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.497+0000 7fd27b7fe640 1 -- 172.21.15.116:0/3331341508 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd2981445b0 con 0x7fd298072810 2024-04-25T00:27:41.499 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.497+0000 7fd28dffb640 1 -- 172.21.15.116:0/3331341508 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 226 v226) ==== 95+0+5679 (secure 0 0 0) 0x7fd294059930 con 0x7fd298072810 2024-04-25T00:27:41.500 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:27:41.500 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":226,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:27:40.040636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6641},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6641":{"gid":6641,"name":"d","rank":0,"incarnation":223,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/2915274073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2915274073},{"type":"v1","addr":"172.21.15.174:6837","nonce":2915274073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6641,"qdb_cluster":[6641]},"id":1},{"mdsmap":{"epoch":221,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:27:19.052286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8652":{"gid":8652,"name":"c","rank":0,"incarnation":218,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6833/3891103804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3891103804},{"type":"v1","addr":"172.21.15.116:6833","nonce":3891103804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8652,"qdb_cluster":[8652]},"id":2}]} 2024-04-25T00:27:41.500 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 226 2024-04-25T00:27:41.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.501+0000 7fd27b7fe640 1 -- 172.21.15.116:0/3331341508 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd27c067910 msgr2=0x7fd27c069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:41.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.501+0000 7fd27b7fe640 1 --2- 172.21.15.116:0/3331341508 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd27c067910 0x7fd27c069dd0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fd2880ad000 tx=0x7fd2880a6000 comp rx=0 tx=0).stop 2024-04-25T00:27:41.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.501+0000 7fd27b7fe640 1 -- 172.21.15.116:0/3331341508 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd298072810 msgr2=0x7fd298142b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:41.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.501+0000 7fd27b7fe640 1 --2- 172.21.15.116:0/3331341508 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd298072810 0x7fd298142b50 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd2980c2e10 tx=0x7fd29400cbe0 comp rx=0 tx=0).stop 2024-04-25T00:27:41.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.501+0000 7fd27b7fe640 1 -- 172.21.15.116:0/3331341508 shutdown_connections 2024-04-25T00:27:41.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.501+0000 7fd27b7fe640 1 --2- 172.21.15.116:0/3331341508 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd27c067910 0x7fd27c069dd0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:41.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.501+0000 7fd27b7fe640 1 --2- 172.21.15.116:0/3331341508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd2980c5280 0x7fd2980c78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:41.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.501+0000 7fd27b7fe640 1 --2- 172.21.15.116:0/3331341508 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd298072c90 0x7fd2980c7360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:41.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.501+0000 7fd27b7fe640 1 --2- 172.21.15.116:0/3331341508 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd298072810 0x7fd298142b50 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:41.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.501+0000 7fd27b7fe640 1 -- 172.21.15.116:0/3331341508 >> 172.21.15.116:0/3331341508 conn(0x7fd29806c720 msgr2=0x7fd2980aa630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:27:41.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.501+0000 7fd27b7fe640 1 -- 172.21.15.116:0/3331341508 shutdown_connections 2024-04-25T00:27:41.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:41.501+0000 7fd27b7fe640 1 -- 172.21.15.116:0/3331341508 wait complete. 2024-04-25T00:27:41.532 INFO:tasks.mds_thrash.fs.[a]:mds.d has gained rank=0, replacing gid=6485 2024-04-25T00:27:41.533 INFO:tasks.mds_thrash.fs.[a]:waiting for 3 secs before reviving mds.a 2024-04-25T00:27:41.656 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:27:41.659 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:27:44.534 INFO:tasks.mds_thrash.fs.[a]:reviving mds.a 2024-04-25T00:27:44.534 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-25T00:27:44.534 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-25T00:27:44.539 INFO:tasks.ceph.mds.a:Started 2024-04-25T00:27:44.539 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:27:44.680 INFO:tasks.ceph.mds.a.smithi116.stdout:starting mds.a at 2024-04-25T00:27:44.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.681+0000 7f7989be9640 1 Processor -- start 2024-04-25T00:27:44.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.685+0000 7f7989be9640 1 -- start start 2024-04-25T00:27:44.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.689+0000 7f7989be9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f7984150bf0 0x7f798406c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:44.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.689+0000 7f7989be9640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f79840c4e90 0x7f79841538f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:44.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.689+0000 7f7989be9640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f798406fde0 con 0x7f798406c760 2024-04-25T00:27:44.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.689+0000 7f7989be9640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f79840c58a0 con 0x7f7984150bf0 2024-04-25T00:27:44.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.689+0000 7f7989be9640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f798406f820 con 0x7f79840c4e90 2024-04-25T00:27:44.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.689+0000 7f7982ffd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f79840c4e90 0x7f79841538f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:44.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.689+0000 7f7982ffd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f79840c4e90 0x7f79841538f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:55710/0 (socket says 172.21.15.116:55710) 2024-04-25T00:27:44.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.689+0000 7f7982ffd640 1 -- 172.21.15.116:0/415147619 learned_addr learned my addr 172.21.15.116:0/415147619 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:27:44.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.689+0000 7f79837fe640 1 --2- 172.21.15.116:0/415147619 >> v2:172.21.15.116:3300/0 conn(0x7f7984150bf0 0x7f798406c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:44.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.689+0000 7f79827fc640 1 -- 172.21.15.116:0/415147619 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1487205613 0 0) 0x7f798406fde0 con 0x7f798406c760 2024-04-25T00:27:44.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.689+0000 7f79827fc640 1 -- 172.21.15.116:0/415147619 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f796c003530 con 0x7f798406c760 2024-04-25T00:27:44.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.689+0000 7f79837fe640 1 -- 172.21.15.116:0/415147619 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f79840c4e90 msgr2=0x7f79841538f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:27:44.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.689+0000 7f79837fe640 1 --2- 172.21.15.116:0/415147619 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f79840c4e90 0x7f79841538f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:44.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.689+0000 7f79837fe640 1 -- 172.21.15.116:0/415147619 >> v1:172.21.15.174:6789/0 conn(0x7f798406c760 legacy=0x7f7984153e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:44.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.689+0000 7f79837fe640 1 -- 172.21.15.116:0/415147619 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f798406fb00 con 0x7f7984150bf0 2024-04-25T00:27:44.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.693+0000 7f79837fe640 1 --2- 172.21.15.116:0/415147619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7984150bf0 0x7f798406c220 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f7974009a20 tx=0x7f797401c700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff3953169c1aa195 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:44.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.693+0000 7f79827fc640 1 -- 172.21.15.116:0/415147619 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79740a8610 con 0x7f7984150bf0 2024-04-25T00:27:44.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.693+0000 7f79827fc640 1 -- 172.21.15.116:0/415147619 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79740a87b0 con 0x7f7984150bf0 2024-04-25T00:27:44.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.693+0000 7f79827fc640 1 -- 172.21.15.116:0/415147619 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79740a8b80 con 0x7f7984150bf0 2024-04-25T00:27:44.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.693+0000 7f7989be9640 1 -- 172.21.15.116:0/415147619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7984150bf0 msgr2=0x7f798406c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:44.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.693+0000 7f7989be9640 1 --2- 172.21.15.116:0/415147619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7984150bf0 0x7f798406c220 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f7974009a20 tx=0x7f797401c700 comp rx=0 tx=0).stop 2024-04-25T00:27:44.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.693+0000 7f7989be9640 1 -- 172.21.15.116:0/415147619 shutdown_connections 2024-04-25T00:27:44.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.693+0000 7f7989be9640 1 --2- 172.21.15.116:0/415147619 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f79840c4e90 0x7f79841538f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:44.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.693+0000 7f7989be9640 1 --2- 172.21.15.116:0/415147619 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7984150bf0 0x7f798406c220 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:44.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.693+0000 7f7989be9640 1 -- 172.21.15.116:0/415147619 >> 172.21.15.116:0/415147619 conn(0x7f79840a9b20 msgr2=0x7f79840c4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:27:44.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.693+0000 7f7989be9640 1 -- 172.21.15.116:0/415147619 shutdown_connections 2024-04-25T00:27:44.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.693+0000 7f7989be9640 1 -- 172.21.15.116:0/415147619 wait complete. 2024-04-25T00:27:44.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.697+0000 7f7989be9640 1 Processor -- start 2024-04-25T00:27:44.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.697+0000 7f7989be9640 1 -- start start 2024-04-25T00:27:44.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.697+0000 7f7989be9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f798406c760 0x7f7984071990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:44.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.697+0000 7f7989be9640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f79840c4e90 0x7f79840729d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:44.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.697+0000 7f7989be9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7984150bf0 0x7f798415c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:44.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.697+0000 7f7989be9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7984160870 con 0x7f798406c760 2024-04-25T00:27:44.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.697+0000 7f7989be9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7984150fd0 con 0x7f7984150bf0 2024-04-25T00:27:44.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.697+0000 7f7989be9640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f79841609f0 con 0x7f79840c4e90 2024-04-25T00:27:44.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.697+0000 7f7983fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7984150bf0 0x7f798415c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:44.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.697+0000 7f7983fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7984150bf0 0x7f798415c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:34108/0 (socket says 172.21.15.116:34108) 2024-04-25T00:27:44.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.697+0000 7f7983fff640 1 -- 172.21.15.116:0/3509264138 learned_addr learned my addr 172.21.15.116:0/3509264138 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:27:44.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.697+0000 7f7983fff640 1 -- 172.21.15.116:0/3509264138 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f79840c4e90 msgr2=0x7f79840729d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:44.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.697+0000 7f7983fff640 1 --2- 172.21.15.116:0/3509264138 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f79840c4e90 0x7f79840729d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:44.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.697+0000 7f7983fff640 1 -- 172.21.15.116:0/3509264138 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f798406c760 msgr2=0x7f7984071990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:44.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.697+0000 7f7983fff640 1 --2- 172.21.15.116:0/3509264138 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f798406c760 0x7f7984071990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:44.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.697+0000 7f7983fff640 1 -- 172.21.15.116:0/3509264138 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7984153350 con 0x7f7984150bf0 2024-04-25T00:27:44.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.697+0000 7f7983fff640 1 --2- 172.21.15.116:0/3509264138 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7984150bf0 0x7f798415c690 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f797c00c790 tx=0x7f797c00cc50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:44.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.697+0000 7f796bfff640 1 -- 172.21.15.116:0/3509264138 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f797c003b80 con 0x7f7984150bf0 2024-04-25T00:27:44.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.697+0000 7f796bfff640 1 -- 172.21.15.116:0/3509264138 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f797c003d20 con 0x7f7984150bf0 2024-04-25T00:27:44.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.697+0000 7f7989be9640 1 -- 172.21.15.116:0/3509264138 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f798414ba50 con 0x7f7984150bf0 2024-04-25T00:27:44.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.697+0000 7f796bfff640 1 -- 172.21.15.116:0/3509264138 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f797c00aa50 con 0x7f7984150bf0 2024-04-25T00:27:44.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.697+0000 7f7989be9640 1 -- 172.21.15.116:0/3509264138 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f798414f980 con 0x7f7984150bf0 2024-04-25T00:27:44.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.697+0000 7f7989be9640 1 -- 172.21.15.116:0/3509264138 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7984068a60 con 0x7f7984150bf0 2024-04-25T00:27:44.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.701+0000 7f796bfff640 1 -- 172.21.15.116:0/3509264138 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f797c00abf0 con 0x7f7984150bf0 2024-04-25T00:27:44.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.701+0000 7f796bfff640 1 --2- 172.21.15.116:0/3509264138 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f79700678c0 0x7f7970069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:44.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.701+0000 7f796bfff640 1 -- 172.21.15.116:0/3509264138 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 10364+0+0 (secure 0 0 0) 0x7f797c087840 con 0x7f7984150bf0 2024-04-25T00:27:44.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.701+0000 7f79837fe640 1 --2- 172.21.15.116:0/3509264138 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f79700678c0 0x7f7970069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:44.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.701+0000 7f79837fe640 1 --2- 172.21.15.116:0/3509264138 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f79700678c0 0x7f7970069d80 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f79740af000 tx=0x7f7974004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:44.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.709+0000 7f796bfff640 1 -- 172.21.15.116:0/3509264138 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f797c0571b0 con 0x7f7984150bf0 2024-04-25T00:27:44.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.989+0000 7f7989be9640 1 -- 172.21.15.116:0/3509264138 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7984153520 con 0x7f7984150bf0 2024-04-25T00:27:44.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.989+0000 7f796bfff640 1 -- 172.21.15.116:0/3509264138 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 226 v226) ==== 95+0+5679 (secure 0 0 0) 0x7f797c056ab0 con 0x7f7984150bf0 2024-04-25T00:27:44.990 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:27:44.990 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":226,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:27:40.040636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6641},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6641":{"gid":6641,"name":"d","rank":0,"incarnation":223,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/2915274073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2915274073},{"type":"v1","addr":"172.21.15.174:6837","nonce":2915274073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6641,"qdb_cluster":[6641]},"id":1},{"mdsmap":{"epoch":221,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:27:19.052286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8652":{"gid":8652,"name":"c","rank":0,"incarnation":218,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6833/3891103804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3891103804},{"type":"v1","addr":"172.21.15.116:6833","nonce":3891103804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8652,"qdb_cluster":[8652]},"id":2}]} 2024-04-25T00:27:44.990 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 226 2024-04-25T00:27:44.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.989+0000 7f7989be9640 1 -- 172.21.15.116:0/3509264138 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f79700678c0 msgr2=0x7f7970069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:44.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.989+0000 7f7989be9640 1 --2- 172.21.15.116:0/3509264138 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f79700678c0 0x7f7970069d80 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f79740af000 tx=0x7f7974004740 comp rx=0 tx=0).stop 2024-04-25T00:27:44.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.989+0000 7f7989be9640 1 -- 172.21.15.116:0/3509264138 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7984150bf0 msgr2=0x7f798415c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:44.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.989+0000 7f7989be9640 1 --2- 172.21.15.116:0/3509264138 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7984150bf0 0x7f798415c690 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f797c00c790 tx=0x7f797c00cc50 comp rx=0 tx=0).stop 2024-04-25T00:27:44.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.989+0000 7f7989be9640 1 -- 172.21.15.116:0/3509264138 shutdown_connections 2024-04-25T00:27:44.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.989+0000 7f7989be9640 1 --2- 172.21.15.116:0/3509264138 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f79700678c0 0x7f7970069d80 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:44.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.989+0000 7f7989be9640 1 --2- 172.21.15.116:0/3509264138 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7984150bf0 0x7f798415c690 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:44.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.989+0000 7f7989be9640 1 --2- 172.21.15.116:0/3509264138 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f79840c4e90 0x7f79840729d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:44.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.989+0000 7f7989be9640 1 --2- 172.21.15.116:0/3509264138 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f798406c760 0x7f7984071990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:44.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.989+0000 7f7989be9640 1 -- 172.21.15.116:0/3509264138 >> 172.21.15.116:0/3509264138 conn(0x7f79840a9b20 msgr2=0x7f79840a8690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:27:44.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.993+0000 7f7989be9640 1 -- 172.21.15.116:0/3509264138 shutdown_connections 2024-04-25T00:27:44.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:44.993+0000 7f7989be9640 1 -- 172.21.15.116:0/3509264138 wait complete. 2024-04-25T00:27:45.014 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2024-04-25T00:27:47.015 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:27:47.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.169+0000 7ff10a040640 1 Processor -- start 2024-04-25T00:27:47.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.169+0000 7ff10a040640 1 -- start start 2024-04-25T00:27:47.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.169+0000 7ff10a040640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff10406d5f0 0x7ff104156c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:47.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.169+0000 7ff10a040640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff104155810 0x7ff1041571c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:47.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.169+0000 7ff10a040640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1040c9150 con 0x7ff104152d80 2024-04-25T00:27:47.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.169+0000 7ff10a040640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7ff1040c9d60 con 0x7ff104155810 2024-04-25T00:27:47.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.169+0000 7ff10a040640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7ff1040c9430 con 0x7ff10406d5f0 2024-04-25T00:27:47.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.169+0000 7ff103fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff104155810 0x7ff1041571c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:47.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.169+0000 7ff1037fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7ff104152d80 0x7ff10406bd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:35040/0 (socket says 172.21.15.116:35040) 2024-04-25T00:27:47.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.169+0000 7ff1037fe640 1 -- 172.21.15.116:0/2988766643 learned_addr learned my addr 172.21.15.116:0/2988766643 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:27:47.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.169+0000 7ff103fff640 1 -- 172.21.15.116:0/2988766643 >> v2:172.21.15.174:3300/0 conn(0x7ff10406d5f0 msgr2=0x7ff104156c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:27:47.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.169+0000 7ff103fff640 1 --2- 172.21.15.116:0/2988766643 >> v2:172.21.15.174:3300/0 conn(0x7ff10406d5f0 0x7ff104156c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:47.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.169+0000 7ff103fff640 1 -- 172.21.15.116:0/2988766643 >> v1:172.21.15.116:6789/0 conn(0x7ff104152d80 legacy=0x7ff10406bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:47.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.169+0000 7ff103fff640 1 -- 172.21.15.116:0/2988766643 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1040c9710 con 0x7ff104155810 2024-04-25T00:27:47.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.169+0000 7ff1027fc640 1 -- 172.21.15.116:0/2988766643 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4114560929 0 0) 0x7ff1040c9150 con 0x7ff104152d80 2024-04-25T00:27:47.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.173+0000 7ff103fff640 1 --2- 172.21.15.116:0/2988766643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff104155810 0x7ff1041571c0 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7ff10408ca90 tx=0x7ff0fc01c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5924f28c2ccd6427 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:47.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.173+0000 7ff1027fc640 1 -- 172.21.15.116:0/2988766643 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0fc0b2070 con 0x7ff104155810 2024-04-25T00:27:47.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.173+0000 7ff1027fc640 1 -- 172.21.15.116:0/2988766643 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0fc0a5880 con 0x7ff104155810 2024-04-25T00:27:47.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.173+0000 7ff1027fc640 1 -- 172.21.15.116:0/2988766643 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0fc0a5c50 con 0x7ff104155810 2024-04-25T00:27:47.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.177+0000 7ff10a040640 1 -- 172.21.15.116:0/2988766643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff104155810 msgr2=0x7ff1041571c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:47.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.177+0000 7ff10a040640 1 --2- 172.21.15.116:0/2988766643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff104155810 0x7ff1041571c0 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7ff10408ca90 tx=0x7ff0fc01c890 comp rx=0 tx=0).stop 2024-04-25T00:27:47.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.177+0000 7ff10a040640 1 -- 172.21.15.116:0/2988766643 shutdown_connections 2024-04-25T00:27:47.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.177+0000 7ff10a040640 1 --2- 172.21.15.116:0/2988766643 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff104155810 0x7ff1041571c0 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:47.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.177+0000 7ff10a040640 1 --2- 172.21.15.116:0/2988766643 >> v2:172.21.15.174:3300/0 conn(0x7ff10406d5f0 0x7ff104156c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:47.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.177+0000 7ff10a040640 1 -- 172.21.15.116:0/2988766643 >> 172.21.15.116:0/2988766643 conn(0x7ff1040acdd0 msgr2=0x7ff1040ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:27:47.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.177+0000 7ff10a040640 1 -- 172.21.15.116:0/2988766643 shutdown_connections 2024-04-25T00:27:47.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.177+0000 7ff10a040640 1 -- 172.21.15.116:0/2988766643 wait complete. 2024-04-25T00:27:47.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.181+0000 7ff10a040640 1 Processor -- start 2024-04-25T00:27:47.182 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.181+0000 7ff10a040640 1 -- start start 2024-04-25T00:27:47.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.181+0000 7ff10a040640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff10406d5f0 0x7ff104151860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:47.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.181+0000 7ff10a040640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff104152d80 0x7ff1040affc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:47.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.181+0000 7ff10a040640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff104155810 0x7ff1040b0500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:47.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.181+0000 7ff10a040640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff1040caf00 con 0x7ff104152d80 2024-04-25T00:27:47.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.181+0000 7ff10a040640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff1040cad80 con 0x7ff104155810 2024-04-25T00:27:47.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.181+0000 7ff10a040640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff1041628d0 con 0x7ff10406d5f0 2024-04-25T00:27:47.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.181+0000 7ff102ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff104152d80 0x7ff1040affc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:47.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.181+0000 7ff103fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff104155810 0x7ff1040b0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:47.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.181+0000 7ff102ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff104152d80 0x7ff1040affc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33896/0 (socket says 172.21.15.116:33896) 2024-04-25T00:27:47.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.181+0000 7ff102ffd640 1 -- 172.21.15.116:0/4162090263 learned_addr learned my addr 172.21.15.116:0/4162090263 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:27:47.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.181+0000 7ff102ffd640 1 -- 172.21.15.116:0/4162090263 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff10406d5f0 msgr2=0x7ff104151860 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T00:27:47.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.181+0000 7ff102ffd640 1 --2- 172.21.15.116:0/4162090263 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff10406d5f0 0x7ff104151860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:47.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.181+0000 7ff102ffd640 1 -- 172.21.15.116:0/4162090263 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff104155810 msgr2=0x7ff1040b0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:47.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.181+0000 7ff102ffd640 1 --2- 172.21.15.116:0/4162090263 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff104155810 0x7ff1040b0500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:47.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.181+0000 7ff102ffd640 1 -- 172.21.15.116:0/4162090263 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1040c1760 con 0x7ff104152d80 2024-04-25T00:27:47.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.181+0000 7ff102ffd640 1 --2- 172.21.15.116:0/4162090263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff104152d80 0x7ff1040affc0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7ff0f400b410 tx=0x7ff0f400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:47.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.181+0000 7ff0ebfff640 1 -- 172.21.15.116:0/4162090263 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f4015020 con 0x7ff104152d80 2024-04-25T00:27:47.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.181+0000 7ff0ebfff640 1 -- 172.21.15.116:0/4162090263 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0f4011830 con 0x7ff104152d80 2024-04-25T00:27:47.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.181+0000 7ff0ebfff640 1 -- 172.21.15.116:0/4162090263 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0f4011c20 con 0x7ff104152d80 2024-04-25T00:27:47.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.181+0000 7ff10a040640 1 -- 172.21.15.116:0/4162090263 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1041f1b60 con 0x7ff104152d80 2024-04-25T00:27:47.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.181+0000 7ff10a040640 1 -- 172.21.15.116:0/4162090263 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1041f2120 con 0x7ff104152d80 2024-04-25T00:27:47.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.185+0000 7ff0ebfff640 1 -- 172.21.15.116:0/4162090263 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff0f4018020 con 0x7ff104152d80 2024-04-25T00:27:47.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.185+0000 7ff0ebfff640 1 --2- 172.21.15.116:0/4162090263 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff0f00678c0 0x7ff0f0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:27:47.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.185+0000 7ff0ebfff640 1 -- 172.21.15.116:0/4162090263 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 10364+0+0 (secure 0 0 0) 0x7ff0f4086df0 con 0x7ff104152d80 2024-04-25T00:27:47.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.185+0000 7ff10a040640 1 -- 172.21.15.116:0/4162090263 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0e40052a0 con 0x7ff104152d80 2024-04-25T00:27:47.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.189+0000 7ff1037fe640 1 --2- 172.21.15.116:0/4162090263 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff0f00678c0 0x7ff0f0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:27:47.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.189+0000 7ff1037fe640 1 --2- 172.21.15.116:0/4162090263 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff0f00678c0 0x7ff0f0069d80 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8007fd0 tx=0x7ff0f80078c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:27:47.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.193+0000 7ff0ebfff640 1 -- 172.21.15.116:0/4162090263 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ff0f401f020 con 0x7ff104152d80 2024-04-25T00:27:47.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.485+0000 7ff10a040640 1 -- 172.21.15.116:0/4162090263 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff0e4005590 con 0x7ff104152d80 2024-04-25T00:27:47.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.485+0000 7ff0ebfff640 1 -- 172.21.15.116:0/4162090263 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 227 v227) ==== 95+0+6508 (secure 0 0 0) 0x7ff0f4056650 con 0x7ff104152d80 2024-04-25T00:27:47.488 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:27:47.488 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":227,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":8772,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/3499878473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3499878473},{"type":"v1","addr":"172.21.15.116:6835","nonce":3499878473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":227}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:27:40.040636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6641},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6641":{"gid":6641,"name":"d","rank":0,"incarnation":223,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/2915274073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2915274073},{"type":"v1","addr":"172.21.15.174:6837","nonce":2915274073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6641,"qdb_cluster":[6641]},"id":1},{"mdsmap":{"epoch":221,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:27:19.052286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8652":{"gid":8652,"name":"c","rank":0,"incarnation":218,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6833/3891103804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3891103804},{"type":"v1","addr":"172.21.15.116:6833","nonce":3891103804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8652,"qdb_cluster":[8652]},"id":2}]} 2024-04-25T00:27:47.488 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 227 2024-04-25T00:27:47.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.489+0000 7ff10a040640 1 -- 172.21.15.116:0/4162090263 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff0f00678c0 msgr2=0x7ff0f0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:47.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.489+0000 7ff10a040640 1 --2- 172.21.15.116:0/4162090263 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff0f00678c0 0x7ff0f0069d80 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8007fd0 tx=0x7ff0f80078c0 comp rx=0 tx=0).stop 2024-04-25T00:27:47.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.489+0000 7ff10a040640 1 -- 172.21.15.116:0/4162090263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff104152d80 msgr2=0x7ff1040affc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:27:47.490 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.489+0000 7ff10a040640 1 --2- 172.21.15.116:0/4162090263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff104152d80 0x7ff1040affc0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7ff0f400b410 tx=0x7ff0f400b8d0 comp rx=0 tx=0).stop 2024-04-25T00:27:47.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.489+0000 7ff10a040640 1 -- 172.21.15.116:0/4162090263 shutdown_connections 2024-04-25T00:27:47.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.489+0000 7ff10a040640 1 --2- 172.21.15.116:0/4162090263 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff0f00678c0 0x7ff0f0069d80 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:47.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.489+0000 7ff10a040640 1 --2- 172.21.15.116:0/4162090263 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff104155810 0x7ff1040b0500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:47.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.489+0000 7ff10a040640 1 --2- 172.21.15.116:0/4162090263 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff104152d80 0x7ff1040affc0 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:47.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.489+0000 7ff10a040640 1 --2- 172.21.15.116:0/4162090263 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff10406d5f0 0x7ff104151860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:27:47.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.489+0000 7ff10a040640 1 -- 172.21.15.116:0/4162090263 >> 172.21.15.116:0/4162090263 conn(0x7ff1040acdd0 msgr2=0x7ff10414d250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:27:47.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.489+0000 7ff10a040640 1 -- 172.21.15.116:0/4162090263 shutdown_connections 2024-04-25T00:27:47.491 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:27:47.489+0000 7ff10a040640 1 -- 172.21.15.116:0/4162090263 wait complete. 2024-04-25T00:27:47.515 INFO:tasks.mds_thrash.fs.[a]:mds.a reported in up:standby state 2024-04-25T00:27:47.515 INFO:tasks.mds_thrash.fs.[a]:waiting for 109 secs before thrashing 2024-04-25T00:28:11.781 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:28:11.784 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:28:36.504 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:28:36.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.681+0000 7efe66e44640 1 Processor -- start 2024-04-25T00:28:36.683 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.681+0000 7efe66e44640 1 -- start start 2024-04-25T00:28:36.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.681+0000 7efe66e44640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7efe600aeb20 0x7efe600c9310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:36.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.681+0000 7efe64be0640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7efe600aeb20 0x7efe600c9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:28:36.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.681+0000 7efe5ffff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7efe600c9850 0x7efe60150870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57196/0 (socket says 172.21.15.116:57196) 2024-04-25T00:28:36.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.681+0000 7efe64be0640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7efe600aeb20 0x7efe600c9310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47938/0 (socket says 172.21.15.116:47938) 2024-04-25T00:28:36.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.681+0000 7efe5ffff640 1 -- 172.21.15.116:0/2845029215 learned_addr learned my addr 172.21.15.116:0/2845029215 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:28:36.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.681+0000 7efe66e44640 1 -- 172.21.15.116:0/2845029215 --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe601535d0 con 0x7efe600c9850 2024-04-25T00:28:36.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.681+0000 7efe66e44640 1 -- 172.21.15.116:0/2845029215 --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe60153850 con 0x7efe60150f80 2024-04-25T00:28:36.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.681+0000 7efe66e44640 1 -- 172.21.15.116:0/2845029215 --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7efe60153b30 con 0x7efe600aeb20 2024-04-25T00:28:36.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.681+0000 7efe5f7fe640 1 -- 172.21.15.116:0/2845029215 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3503013898 0 0) 0x7efe4c001120 con 0x7efe600c9850 2024-04-25T00:28:36.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.681+0000 7efe5f7fe640 1 -- 172.21.15.116:0/2845029215 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe50003530 con 0x7efe600c9850 2024-04-25T00:28:36.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.681+0000 7efe64be0640 1 -- 172.21.15.116:0/2845029215 >> v1:172.21.15.174:6789/0 conn(0x7efe60150f80 legacy=0x7efe600c5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:36.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.681+0000 7efe5f7fe640 1 -- 172.21.15.116:0/2845029215 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2420618449 0 0) 0x7efe60153850 con 0x7efe60150f80 2024-04-25T00:28:36.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.681+0000 7efe64be0640 1 -- 172.21.15.116:0/2845029215 >> v1:172.21.15.116:6789/0 conn(0x7efe600c9850 legacy=0x7efe60150870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:36.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.681+0000 7efe64be0640 1 -- 172.21.15.116:0/2845029215 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe60153e10 con 0x7efe600aeb20 2024-04-25T00:28:36.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.685+0000 7efe64be0640 1 --2- 172.21.15.116:0/2845029215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe600aeb20 0x7efe600c9310 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7efe540097c0 tx=0x7efe5401c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e271da43379d1ac7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:28:36.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.685+0000 7efe5f7fe640 1 -- 172.21.15.116:0/2845029215 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe540b0070 con 0x7efe600aeb20 2024-04-25T00:28:36.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.685+0000 7efe5f7fe640 1 -- 172.21.15.116:0/2845029215 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe540ab400 con 0x7efe600aeb20 2024-04-25T00:28:36.688 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.689+0000 7efe5f7fe640 1 -- 172.21.15.116:0/2845029215 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe540ab7d0 con 0x7efe600aeb20 2024-04-25T00:28:36.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.689+0000 7efe66e44640 1 -- 172.21.15.116:0/2845029215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe600aeb20 msgr2=0x7efe600c9310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:36.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.689+0000 7efe66e44640 1 --2- 172.21.15.116:0/2845029215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe600aeb20 0x7efe600c9310 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7efe540097c0 tx=0x7efe5401c380 comp rx=0 tx=0).stop 2024-04-25T00:28:36.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.693+0000 7efe66e44640 1 -- 172.21.15.116:0/2845029215 shutdown_connections 2024-04-25T00:28:36.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.693+0000 7efe66e44640 1 --2- 172.21.15.116:0/2845029215 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe600aeb20 0x7efe600c9310 unknown :-1 s=CLOSED pgs=1640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:36.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.693+0000 7efe66e44640 1 -- 172.21.15.116:0/2845029215 >> 172.21.15.116:0/2845029215 conn(0x7efe60074650 msgr2=0x7efe60074a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:28:36.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.693+0000 7efe66e44640 1 -- 172.21.15.116:0/2845029215 shutdown_connections 2024-04-25T00:28:36.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.693+0000 7efe66e44640 1 -- 172.21.15.116:0/2845029215 wait complete. 2024-04-25T00:28:36.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.693+0000 7efe66e44640 1 Processor -- start 2024-04-25T00:28:36.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.693+0000 7efe66e44640 1 -- start start 2024-04-25T00:28:36.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.697+0000 7efe66e44640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efe600aeb20 0x7efe600acb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:36.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.697+0000 7efe66e44640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efe600c9850 0x7efe601f60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:36.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.697+0000 7efe66e44640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe60150f80 0x7efe60151e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:36.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.697+0000 7efe66e44640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7efe600af080 con 0x7efe60150f80 2024-04-25T00:28:36.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.697+0000 7efe66e44640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7efe600aef00 con 0x7efe600c9850 2024-04-25T00:28:36.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.697+0000 7efe66e44640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7efe6015f770 con 0x7efe600aeb20 2024-04-25T00:28:36.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.697+0000 7efe64be0640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efe600aeb20 0x7efe600acb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:28:36.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.697+0000 7efe64be0640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efe600aeb20 0x7efe600acb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:46294/0 (socket says 172.21.15.116:46294) 2024-04-25T00:28:36.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.697+0000 7efe64be0640 1 -- 172.21.15.116:0/1361849775 learned_addr learned my addr 172.21.15.116:0/1361849775 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:28:36.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.697+0000 7efe64be0640 1 -- 172.21.15.116:0/1361849775 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efe600c9850 msgr2=0x7efe601f60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:36.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.697+0000 7efe64be0640 1 --2- 172.21.15.116:0/1361849775 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efe600c9850 0x7efe601f60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:36.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.697+0000 7efe64be0640 1 -- 172.21.15.116:0/1361849775 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe60150f80 msgr2=0x7efe60151e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:36.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.697+0000 7efe64be0640 1 --2- 172.21.15.116:0/1361849775 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe60150f80 0x7efe60151e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:36.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.697+0000 7efe64be0640 1 -- 172.21.15.116:0/1361849775 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe601f6630 con 0x7efe600aeb20 2024-04-25T00:28:36.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.697+0000 7efe64be0640 1 --2- 172.21.15.116:0/1361849775 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efe600aeb20 0x7efe600acb70 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7efe540ae040 tx=0x7efe5401abc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:28:36.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.697+0000 7efe5d7fa640 1 -- 172.21.15.116:0/1361849775 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe540bb020 con 0x7efe600aeb20 2024-04-25T00:28:36.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.697+0000 7efe5d7fa640 1 -- 172.21.15.116:0/1361849775 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe540b1040 con 0x7efe600aeb20 2024-04-25T00:28:36.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.697+0000 7efe66e44640 1 -- 172.21.15.116:0/1361849775 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe600ac0b0 con 0x7efe600aeb20 2024-04-25T00:28:36.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.697+0000 7efe5d7fa640 1 -- 172.21.15.116:0/1361849775 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe540b0050 con 0x7efe600aeb20 2024-04-25T00:28:36.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.697+0000 7efe66e44640 1 -- 172.21.15.116:0/1361849775 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7efe600ae1e0 con 0x7efe600aeb20 2024-04-25T00:28:36.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.697+0000 7efe66e44640 1 -- 172.21.15.116:0/1361849775 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe60068a60 con 0x7efe600aeb20 2024-04-25T00:28:36.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.701+0000 7efe5d7fa640 1 -- 172.21.15.116:0/1361849775 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7efe540be070 con 0x7efe600aeb20 2024-04-25T00:28:36.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.701+0000 7efe5d7fa640 1 --2- 172.21.15.116:0/1361849775 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efe440678c0 0x7efe44069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:36.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.701+0000 7efe5d7fa640 1 -- 172.21.15.116:0/1361849775 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(71..71 src has 1..71) ==== 10364+0+0 (secure 0 0 0) 0x7efe541206d0 con 0x7efe600aeb20 2024-04-25T00:28:36.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.705+0000 7efe5ffff640 1 --2- 172.21.15.116:0/1361849775 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efe440678c0 0x7efe44069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:28:36.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.705+0000 7efe5ffff640 1 --2- 172.21.15.116:0/1361849775 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efe440678c0 0x7efe44069d80 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7efe4c000f70 tx=0x7efe4c001120 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:28:36.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.709+0000 7efe5d7fa640 1 -- 172.21.15.116:0/1361849775 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7efe54016090 con 0x7efe600aeb20 2024-04-25T00:28:36.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.989+0000 7efe66e44640 1 -- 172.21.15.116:0/1361849775 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efe600c53f0 con 0x7efe600aeb20 2024-04-25T00:28:36.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.989+0000 7efe5d7fa640 1 -- 172.21.15.116:0/1361849775 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 227 v227) ==== 95+0+6508 (secure 0 0 0) 0x7efe540b3300 con 0x7efe600aeb20 2024-04-25T00:28:36.991 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:28:36.991 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":227,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":8772,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/3499878473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3499878473},{"type":"v1","addr":"172.21.15.116:6835","nonce":3499878473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":227}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:27:40.040636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6641},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6641":{"gid":6641,"name":"d","rank":0,"incarnation":223,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/2915274073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2915274073},{"type":"v1","addr":"172.21.15.174:6837","nonce":2915274073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6641,"qdb_cluster":[6641]},"id":1},{"mdsmap":{"epoch":221,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:27:19.052286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8652":{"gid":8652,"name":"c","rank":0,"incarnation":218,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6833/3891103804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3891103804},{"type":"v1","addr":"172.21.15.116:6833","nonce":3891103804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8652,"qdb_cluster":[8652]},"id":2}]} 2024-04-25T00:28:36.991 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 227 2024-04-25T00:28:36.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.993+0000 7efe66e44640 1 -- 172.21.15.116:0/1361849775 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efe440678c0 msgr2=0x7efe44069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:36.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.993+0000 7efe66e44640 1 --2- 172.21.15.116:0/1361849775 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efe440678c0 0x7efe44069d80 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7efe4c000f70 tx=0x7efe4c001120 comp rx=0 tx=0).stop 2024-04-25T00:28:36.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.993+0000 7efe66e44640 1 -- 172.21.15.116:0/1361849775 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efe600aeb20 msgr2=0x7efe600acb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:36.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.993+0000 7efe66e44640 1 --2- 172.21.15.116:0/1361849775 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efe600aeb20 0x7efe600acb70 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7efe540ae040 tx=0x7efe5401abc0 comp rx=0 tx=0).stop 2024-04-25T00:28:36.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.993+0000 7efe66e44640 1 -- 172.21.15.116:0/1361849775 shutdown_connections 2024-04-25T00:28:36.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.993+0000 7efe66e44640 1 --2- 172.21.15.116:0/1361849775 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efe440678c0 0x7efe44069d80 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:36.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.993+0000 7efe66e44640 1 --2- 172.21.15.116:0/1361849775 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efe60150f80 0x7efe60151e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:36.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.993+0000 7efe66e44640 1 --2- 172.21.15.116:0/1361849775 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efe600c9850 0x7efe601f60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:36.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.993+0000 7efe66e44640 1 --2- 172.21.15.116:0/1361849775 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efe600aeb20 0x7efe600acb70 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:36.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.993+0000 7efe66e44640 1 -- 172.21.15.116:0/1361849775 >> 172.21.15.116:0/1361849775 conn(0x7efe60074650 msgr2=0x7efe600aa5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:28:36.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.993+0000 7efe66e44640 1 -- 172.21.15.116:0/1361849775 shutdown_connections 2024-04-25T00:28:36.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:36.993+0000 7efe66e44640 1 -- 172.21.15.116:0/1361849775 wait complete. 2024-04-25T00:28:37.022 INFO:tasks.mds_thrash.fs.[b]:kill mds.c (rank=0) 2024-04-25T00:28:37.022 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-25T00:28:37.022 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:28:37.037 INFO:tasks.ceph.mds.c:Stopped 2024-04-25T00:28:37.037 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:28:37.219 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.213+0000 7f6b81612640 1 Processor -- start 2024-04-25T00:28:37.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.213+0000 7f6b81612640 1 -- start start 2024-04-25T00:28:37.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.217+0000 7f6b81612640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b7c153df0 0x7f6b7c06e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:37.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.217+0000 7f6b81612640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f6b7c0c36c0 0x7f6b7c0c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:37.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.217+0000 7f6b81612640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b7c070f90 con 0x7f6b7c0c2f50 2024-04-25T00:28:37.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.217+0000 7f6b81612640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f6b7c070d40 con 0x7f6b7c0c36c0 2024-04-25T00:28:37.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.217+0000 7f6b81612640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6b7c071270 con 0x7f6b7c153df0 2024-04-25T00:28:37.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.217+0000 7f6b7a7fc640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f6b7c0c36c0 0x7f6b7c0c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:28:37.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.217+0000 7f6b7a7fc640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f6b7c0c36c0 0x7f6b7c0c2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:40144/0 (socket says 172.21.15.116:40144) 2024-04-25T00:28:37.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.217+0000 7f6b7a7fc640 1 -- 172.21.15.116:0/2386708959 learned_addr learned my addr 172.21.15.116:0/2386708959 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:28:37.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.217+0000 7f6b7affd640 1 --2- 172.21.15.116:0/2386708959 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b7c153df0 0x7f6b7c06e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:28:37.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.217+0000 7f6b7a7fc640 1 -- 172.21.15.116:0/2386708959 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b7c153df0 msgr2=0x7f6b7c06e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:28:37.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.217+0000 7f6b7a7fc640 1 --2- 172.21.15.116:0/2386708959 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b7c153df0 0x7f6b7c06e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:37.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.217+0000 7f6b7a7fc640 1 -- 172.21.15.116:0/2386708959 >> v1:172.21.15.174:6789/0 conn(0x7f6b7c0c2f50 legacy=0x7f6b7c159cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:37.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.217+0000 7f6b7a7fc640 1 -- 172.21.15.116:0/2386708959 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b7c071550 con 0x7f6b7c0c36c0 2024-04-25T00:28:37.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.221+0000 7f6b7a7fc640 1 --2- 172.21.15.116:0/2386708959 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b7c0c36c0 0x7f6b7c0c2a10 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f6b70009870 tx=0x7f6b7001c4e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5cbe9adf585a88d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:28:37.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.221+0000 7f6b79ffb640 1 -- 172.21.15.116:0/2386708959 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b700ab660 con 0x7f6b7c0c36c0 2024-04-25T00:28:37.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.221+0000 7f6b79ffb640 1 -- 172.21.15.116:0/2386708959 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b700b0070 con 0x7f6b7c0c36c0 2024-04-25T00:28:37.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.221+0000 7f6b79ffb640 1 -- 172.21.15.116:0/2386708959 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b700aba20 con 0x7f6b7c0c36c0 2024-04-25T00:28:37.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.221+0000 7f6b81612640 1 -- 172.21.15.116:0/2386708959 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b7c0c36c0 msgr2=0x7f6b7c0c2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:37.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.221+0000 7f6b81612640 1 --2- 172.21.15.116:0/2386708959 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b7c0c36c0 0x7f6b7c0c2a10 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f6b70009870 tx=0x7f6b7001c4e0 comp rx=0 tx=0).stop 2024-04-25T00:28:37.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.221+0000 7f6b81612640 1 -- 172.21.15.116:0/2386708959 shutdown_connections 2024-04-25T00:28:37.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.221+0000 7f6b81612640 1 --2- 172.21.15.116:0/2386708959 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b7c0c36c0 0x7f6b7c0c2a10 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:37.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.221+0000 7f6b81612640 1 --2- 172.21.15.116:0/2386708959 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b7c153df0 0x7f6b7c06e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:37.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.221+0000 7f6b81612640 1 -- 172.21.15.116:0/2386708959 >> 172.21.15.116:0/2386708959 conn(0x7f6b7c0c3b00 msgr2=0x7f6b7c0c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:28:37.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.221+0000 7f6b81612640 1 -- 172.21.15.116:0/2386708959 shutdown_connections 2024-04-25T00:28:37.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.221+0000 7f6b81612640 1 -- 172.21.15.116:0/2386708959 wait complete. 2024-04-25T00:28:37.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.221+0000 7f6b81612640 1 Processor -- start 2024-04-25T00:28:37.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.221+0000 7f6b81612640 1 -- start start 2024-04-25T00:28:37.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.225+0000 7f6b81612640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b7c0c2f50 0x7f6b7c1486a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:37.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.225+0000 7f6b81612640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b7c0c36c0 0x7f6b7c158f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:37.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.225+0000 7f6b81612640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b7c153df0 0x7f6b7c159460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:37.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.225+0000 7f6b81612640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6b7c166d90 con 0x7f6b7c0c36c0 2024-04-25T00:28:37.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.225+0000 7f6b81612640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6b7c15aa20 con 0x7f6b7c153df0 2024-04-25T00:28:37.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.225+0000 7f6b81612640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6b7c166f10 con 0x7f6b7c0c2f50 2024-04-25T00:28:37.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.225+0000 7f6b7b7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b7c153df0 0x7f6b7c159460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:28:37.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.225+0000 7f6b7b7fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b7c153df0 0x7f6b7c159460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:40154/0 (socket says 172.21.15.116:40154) 2024-04-25T00:28:37.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.225+0000 7f6b7b7fe640 1 -- 172.21.15.116:0/1421054559 learned_addr learned my addr 172.21.15.116:0/1421054559 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:28:37.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.225+0000 7f6b7a7fc640 1 --2- 172.21.15.116:0/1421054559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b7c0c36c0 0x7f6b7c158f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:28:37.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.225+0000 7f6b7a7fc640 1 -- 172.21.15.116:0/1421054559 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b7c0c2f50 msgr2=0x7f6b7c1486a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:37.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.225+0000 7f6b7a7fc640 1 --2- 172.21.15.116:0/1421054559 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b7c0c2f50 0x7f6b7c1486a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:37.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.225+0000 7f6b7a7fc640 1 -- 172.21.15.116:0/1421054559 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b7c153df0 msgr2=0x7f6b7c159460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:37.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.225+0000 7f6b7a7fc640 1 --2- 172.21.15.116:0/1421054559 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b7c153df0 0x7f6b7c159460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:37.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.225+0000 7f6b7a7fc640 1 -- 172.21.15.116:0/1421054559 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b7c1599a0 con 0x7f6b7c0c36c0 2024-04-25T00:28:37.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.225+0000 7f6b7a7fc640 1 --2- 172.21.15.116:0/1421054559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b7c0c36c0 0x7f6b7c158f20 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c068110 tx=0x7f6b7001ad20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:28:37.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.225+0000 7f6b637fe640 1 -- 172.21.15.116:0/1421054559 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b700b4800 con 0x7f6b7c0c36c0 2024-04-25T00:28:37.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.225+0000 7f6b637fe640 1 -- 172.21.15.116:0/1421054559 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b700b49a0 con 0x7f6b7c0c36c0 2024-04-25T00:28:37.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.225+0000 7f6b81612640 1 -- 172.21.15.116:0/1421054559 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b7c158490 con 0x7f6b7c0c36c0 2024-04-25T00:28:37.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.225+0000 7f6b637fe640 1 -- 172.21.15.116:0/1421054559 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b700b4d90 con 0x7f6b7c0c36c0 2024-04-25T00:28:37.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.225+0000 7f6b81612640 1 -- 172.21.15.116:0/1421054559 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b7c157c00 con 0x7f6b7c0c36c0 2024-04-25T00:28:37.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.225+0000 7f6b81612640 1 -- 172.21.15.116:0/1421054559 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b7c1586b0 con 0x7f6b7c0c36c0 2024-04-25T00:28:37.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.229+0000 7f6b637fe640 1 -- 172.21.15.116:0/1421054559 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6b700b3380 con 0x7f6b7c0c36c0 2024-04-25T00:28:37.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.229+0000 7f6b637fe640 1 --2- 172.21.15.116:0/1421054559 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6b680678c0 0x7f6b68069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:37.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.229+0000 7f6b637fe640 1 -- 172.21.15.116:0/1421054559 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 10364+0+0 (secure 0 0 0) 0x7f6b70121680 con 0x7f6b7c0c36c0 2024-04-25T00:28:37.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.229+0000 7f6b7affd640 1 --2- 172.21.15.116:0/1421054559 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6b680678c0 0x7f6b68069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:28:37.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.229+0000 7f6b7affd640 1 --2- 172.21.15.116:0/1421054559 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6b680678c0 0x7f6b68069d80 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c14fa10 tx=0x7f6b6c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:28:37.236 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.233+0000 7f6b637fe640 1 -- 172.21.15.116:0/1421054559 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f6b7c1586b0 con 0x7f6b7c0c36c0 2024-04-25T00:28:37.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.517+0000 7f6b81612640 1 -- 172.21.15.116:0/1421054559 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6b7c068350 con 0x7f6b7c0c36c0 2024-04-25T00:28:37.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.517+0000 7f6b637fe640 1 -- 172.21.15.116:0/1421054559 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 227 v227) ==== 95+0+6508 (secure 0 0 0) 0x7f6b70016190 con 0x7f6b7c0c36c0 2024-04-25T00:28:37.520 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:28:37.520 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":227,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":8772,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/3499878473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3499878473},{"type":"v1","addr":"172.21.15.116:6835","nonce":3499878473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":227}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:27:40.040636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6641},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6641":{"gid":6641,"name":"d","rank":0,"incarnation":223,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/2915274073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2915274073},{"type":"v1","addr":"172.21.15.174:6837","nonce":2915274073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6641,"qdb_cluster":[6641]},"id":1},{"mdsmap":{"epoch":221,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:27:19.052286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8652":{"gid":8652,"name":"c","rank":0,"incarnation":218,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6833/3891103804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3891103804},{"type":"v1","addr":"172.21.15.116:6833","nonce":3891103804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8652,"qdb_cluster":[8652]},"id":2}]} 2024-04-25T00:28:37.521 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 227 2024-04-25T00:28:37.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.521+0000 7f6b81612640 1 -- 172.21.15.116:0/1421054559 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6b680678c0 msgr2=0x7f6b68069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:37.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.521+0000 7f6b81612640 1 --2- 172.21.15.116:0/1421054559 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6b680678c0 0x7f6b68069d80 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c14fa10 tx=0x7f6b6c008040 comp rx=0 tx=0).stop 2024-04-25T00:28:37.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.521+0000 7f6b81612640 1 -- 172.21.15.116:0/1421054559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b7c0c36c0 msgr2=0x7f6b7c158f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:37.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.521+0000 7f6b81612640 1 --2- 172.21.15.116:0/1421054559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b7c0c36c0 0x7f6b7c158f20 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c068110 tx=0x7f6b7001ad20 comp rx=0 tx=0).stop 2024-04-25T00:28:37.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.521+0000 7f6b81612640 1 -- 172.21.15.116:0/1421054559 shutdown_connections 2024-04-25T00:28:37.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.521+0000 7f6b81612640 1 --2- 172.21.15.116:0/1421054559 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6b680678c0 0x7f6b68069d80 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:37.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.521+0000 7f6b81612640 1 --2- 172.21.15.116:0/1421054559 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6b7c153df0 0x7f6b7c159460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:37.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.521+0000 7f6b81612640 1 --2- 172.21.15.116:0/1421054559 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6b7c0c36c0 0x7f6b7c158f20 secure :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f6b7c068110 tx=0x7f6b7001ad20 comp rx=0 tx=0).stop 2024-04-25T00:28:37.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.521+0000 7f6b81612640 1 --2- 172.21.15.116:0/1421054559 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6b7c0c2f50 0x7f6b7c1486a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:37.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.521+0000 7f6b81612640 1 -- 172.21.15.116:0/1421054559 >> 172.21.15.116:0/1421054559 conn(0x7f6b7c0c3b00 msgr2=0x7f6b7c14e230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:28:37.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.521+0000 7f6b81612640 1 -- 172.21.15.116:0/1421054559 shutdown_connections 2024-04-25T00:28:37.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:37.521+0000 7f6b81612640 1 -- 172.21.15.116:0/1421054559 wait complete. 2024-04-25T00:28:37.551 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:28:39.552 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:28:39.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:39.697+0000 7fef52fbd640 1 Processor -- start 2024-04-25T00:28:39.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:39.697+0000 7fef52fbd640 1 -- start start 2024-04-25T00:28:39.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:39.697+0000 7fef52fbd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fef4c150d20 0x7fef4c153620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:39.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:39.697+0000 7fef52fbd640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fef4c0c3240 0x7fef4c0c3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:39.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:39.697+0000 7fef52fbd640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef4c0c36a0 con 0x7fef4c0c4380 2024-04-25T00:28:39.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:39.697+0000 7fef52fbd640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fef4c06fba0 con 0x7fef4c0c3240 2024-04-25T00:28:39.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:39.697+0000 7fef52fbd640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fef4c06f8c0 con 0x7fef4c150d20 2024-04-25T00:28:39.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:39.697+0000 7fef5155a640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fef4c0c4380 0x7fef4c071050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57198/0 (socket says 172.21.15.116:57198) 2024-04-25T00:28:39.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:39.697+0000 7fef5155a640 1 -- 172.21.15.116:0/273674476 learned_addr learned my addr 172.21.15.116:0/273674476 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:28:39.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:39.697+0000 7fef4bfff640 1 --2- 172.21.15.116:0/273674476 >> v2:172.21.15.174:3300/0 conn(0x7fef4c0c3240 0x7fef4c0c3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:28:39.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:39.697+0000 7fef4bfff640 1 -- 172.21.15.116:0/273674476 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fef4c150d20 msgr2=0x7fef4c153620 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T00:28:39.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:39.697+0000 7fef4bfff640 1 --2- 172.21.15.116:0/273674476 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fef4c150d20 0x7fef4c153620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:39.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:39.697+0000 7fef4bfff640 1 -- 172.21.15.116:0/273674476 >> v1:172.21.15.116:6789/0 conn(0x7fef4c0c4380 legacy=0x7fef4c071050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:28:39.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:39.697+0000 7fef4bfff640 1 -- 172.21.15.116:0/273674476 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef4c0c5420 con 0x7fef4c0c3240 2024-04-25T00:28:39.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:39.701+0000 7fef4bfff640 1 --2- 172.21.15.116:0/273674476 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef4c0c3240 0x7fef4c0c3e40 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fef3c0097c0 tx=0x7fef3c01c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e315d4d3d0463ff server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:28:39.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:39.705+0000 7fef4b7fe640 1 -- 172.21.15.116:0/273674476 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef3c0b0070 con 0x7fef4c0c3240 2024-04-25T00:28:39.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:39.705+0000 7fef4b7fe640 1 -- 172.21.15.116:0/273674476 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef3c004230 con 0x7fef4c0c3240 2024-04-25T00:28:39.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:39.705+0000 7fef4b7fe640 1 -- 172.21.15.116:0/273674476 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef3c003620 con 0x7fef4c0c3240 2024-04-25T00:28:39.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:39.705+0000 7fef52fbd640 1 -- 172.21.15.116:0/273674476 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef4c0c3240 msgr2=0x7fef4c0c3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:39.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:39.705+0000 7fef52fbd640 1 --2- 172.21.15.116:0/273674476 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef4c0c3240 0x7fef4c0c3e40 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fef3c0097c0 tx=0x7fef3c01c480 comp rx=0 tx=0).stop 2024-04-25T00:28:39.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:39.705+0000 7fef52fbd640 1 -- 172.21.15.116:0/273674476 shutdown_connections 2024-04-25T00:28:39.706 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:39.705+0000 7fef52fbd640 1 --2- 172.21.15.116:0/273674476 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef4c0c3240 0x7fef4c0c3e40 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:39.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:39.705+0000 7fef52fbd640 1 --2- 172.21.15.116:0/273674476 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fef4c150d20 0x7fef4c153620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:39.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:39.705+0000 7fef52fbd640 1 -- 172.21.15.116:0/273674476 >> 172.21.15.116:0/273674476 conn(0x7fef4c0c9160 msgr2=0x7fef4c0c9570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:28:39.709 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:39.705+0000 7fef52fbd640 1 -- 172.21.15.116:0/273674476 shutdown_connections 2024-04-25T00:28:39.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:39.705+0000 7fef52fbd640 1 -- 172.21.15.116:0/273674476 wait complete. 2024-04-25T00:28:39.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:39.709+0000 7fef52fbd640 1 Processor -- start 2024-04-25T00:28:39.710 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:39.709+0000 7fef52fbd640 1 -- start start 2024-04-25T00:28:39.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:39.709+0000 7fef52fbd640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef4c0c3240 0x7fef4c149940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:39.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:39.709+0000 7fef52fbd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fef4c0c4380 0x7fef4c14afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:39.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:39.709+0000 7fef52fbd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef4c150d20 0x7fef4c14b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:39.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:39.709+0000 7fef52fbd640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fef4c0718e0 con 0x7fef4c150d20 2024-04-25T00:28:39.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:39.709+0000 7fef52fbd640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fef4c071760 con 0x7fef4c0c3240 2024-04-25T00:28:39.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:39.709+0000 7fef52fbd640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fef4c151100 con 0x7fef4c0c4380 2024-04-25T00:28:39.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:39.709+0000 7fef50d59640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef4c0c3240 0x7fef4c149940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:28:39.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:39.709+0000 7fef50d59640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef4c0c3240 0x7fef4c149940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:40174/0 (socket says 172.21.15.116:40174) 2024-04-25T00:28:39.712 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:39.709+0000 7fef50d59640 1 -- 172.21.15.116:0/125592727 learned_addr learned my addr 172.21.15.116:0/125592727 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:28:39.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:39.709+0000 7fef4bfff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fef4c0c4380 0x7fef4c14afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:28:39.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:39.709+0000 7fef50d59640 1 -- 172.21.15.116:0/125592727 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fef4c0c4380 msgr2=0x7fef4c14afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:39.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:39.709+0000 7fef50d59640 1 --2- 172.21.15.116:0/125592727 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fef4c0c4380 0x7fef4c14afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:39.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:39.709+0000 7fef50d59640 1 -- 172.21.15.116:0/125592727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef4c150d20 msgr2=0x7fef4c14b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:39.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:39.709+0000 7fef50d59640 1 --2- 172.21.15.116:0/125592727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef4c150d20 0x7fef4c14b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:39.713 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:39.709+0000 7fef50d59640 1 -- 172.21.15.116:0/125592727 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef4c0ae8f0 con 0x7fef4c0c3240 2024-04-25T00:28:39.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:39.709+0000 7fef50d59640 1 --2- 172.21.15.116:0/125592727 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef4c0c3240 0x7fef4c149940 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fef4400ba80 tx=0x7fef4400bf40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:28:39.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:39.709+0000 7fef497fa640 1 -- 172.21.15.116:0/125592727 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef44019070 con 0x7fef4c0c3240 2024-04-25T00:28:39.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:39.713+0000 7fef52fbd640 1 -- 172.21.15.116:0/125592727 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef4c0aebd0 con 0x7fef4c0c3240 2024-04-25T00:28:39.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:39.713+0000 7fef52fbd640 1 -- 172.21.15.116:0/125592727 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef4c149e80 con 0x7fef4c0c3240 2024-04-25T00:28:39.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:39.713+0000 7fef497fa640 1 -- 172.21.15.116:0/125592727 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef4400cc30 con 0x7fef4c0c3240 2024-04-25T00:28:39.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:39.713+0000 7fef497fa640 1 -- 172.21.15.116:0/125592727 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef44014440 con 0x7fef4c0c3240 2024-04-25T00:28:39.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:39.717+0000 7fef52fbd640 1 -- 172.21.15.116:0/125592727 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef2c0052a0 con 0x7fef4c0c3240 2024-04-25T00:28:39.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:39.725+0000 7fef497fa640 1 -- 172.21.15.116:0/125592727 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fef4401c070 con 0x7fef4c0c3240 2024-04-25T00:28:39.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:39.725+0000 7fef497fa640 1 --2- 172.21.15.116:0/125592727 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fef38067990 0x7fef38069e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:39.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:39.725+0000 7fef497fa640 1 -- 172.21.15.116:0/125592727 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 10364+0+0 (secure 0 0 0) 0x7fef44087020 con 0x7fef4c0c3240 2024-04-25T00:28:39.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:39.725+0000 7fef4bfff640 1 --2- 172.21.15.116:0/125592727 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fef38067990 0x7fef38069e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:28:39.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:39.725+0000 7fef4bfff640 1 --2- 172.21.15.116:0/125592727 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fef38067990 0x7fef38069e50 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fef3c0ad000 tx=0x7fef3c0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:28:39.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:39.729+0000 7fef497fa640 1 -- 172.21.15.116:0/125592727 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fef44023020 con 0x7fef4c0c3240 2024-04-25T00:28:40.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:40.013+0000 7fef52fbd640 1 -- 172.21.15.116:0/125592727 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fef2c005590 con 0x7fef4c0c3240 2024-04-25T00:28:40.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:40.013+0000 7fef497fa640 1 -- 172.21.15.116:0/125592727 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 227 v227) ==== 95+0+6508 (secure 0 0 0) 0x7fef44056340 con 0x7fef4c0c3240 2024-04-25T00:28:40.015 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:28:40.015 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":227,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":8772,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/3499878473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3499878473},{"type":"v1","addr":"172.21.15.116:6835","nonce":3499878473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":227}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:27:40.040636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6641},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6641":{"gid":6641,"name":"d","rank":0,"incarnation":223,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/2915274073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2915274073},{"type":"v1","addr":"172.21.15.174:6837","nonce":2915274073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6641,"qdb_cluster":[6641]},"id":1},{"mdsmap":{"epoch":221,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:27:19.052286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8652":{"gid":8652,"name":"c","rank":0,"incarnation":218,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6833/3891103804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3891103804},{"type":"v1","addr":"172.21.15.116:6833","nonce":3891103804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8652,"qdb_cluster":[8652]},"id":2}]} 2024-04-25T00:28:40.015 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 227 2024-04-25T00:28:40.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:40.017+0000 7fef52fbd640 1 -- 172.21.15.116:0/125592727 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fef38067990 msgr2=0x7fef38069e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:40.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:40.017+0000 7fef52fbd640 1 --2- 172.21.15.116:0/125592727 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fef38067990 0x7fef38069e50 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fef3c0ad000 tx=0x7fef3c0a6000 comp rx=0 tx=0).stop 2024-04-25T00:28:40.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:40.017+0000 7fef52fbd640 1 -- 172.21.15.116:0/125592727 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef4c0c3240 msgr2=0x7fef4c149940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:40.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:40.017+0000 7fef52fbd640 1 --2- 172.21.15.116:0/125592727 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef4c0c3240 0x7fef4c149940 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fef4400ba80 tx=0x7fef4400bf40 comp rx=0 tx=0).stop 2024-04-25T00:28:40.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:40.017+0000 7fef52fbd640 1 -- 172.21.15.116:0/125592727 shutdown_connections 2024-04-25T00:28:40.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:40.017+0000 7fef52fbd640 1 --2- 172.21.15.116:0/125592727 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fef38067990 0x7fef38069e50 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:40.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:40.017+0000 7fef52fbd640 1 --2- 172.21.15.116:0/125592727 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fef4c150d20 0x7fef4c14b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:40.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:40.017+0000 7fef52fbd640 1 --2- 172.21.15.116:0/125592727 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fef4c0c4380 0x7fef4c14afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:40.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:40.017+0000 7fef52fbd640 1 --2- 172.21.15.116:0/125592727 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fef4c0c3240 0x7fef4c149940 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:40.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:40.017+0000 7fef52fbd640 1 -- 172.21.15.116:0/125592727 >> 172.21.15.116:0/125592727 conn(0x7fef4c0c9160 msgr2=0x7fef4c14a580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:28:40.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:40.017+0000 7fef52fbd640 1 -- 172.21.15.116:0/125592727 shutdown_connections 2024-04-25T00:28:40.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:40.017+0000 7fef52fbd640 1 -- 172.21.15.116:0/125592727 wait complete. 2024-04-25T00:28:40.039 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:28:42.040 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:28:42.185 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:28:42.187 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:28:42.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.189+0000 7f00fdf7a640 1 Processor -- start 2024-04-25T00:28:42.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.189+0000 7f00fdf7a640 1 -- start start 2024-04-25T00:28:42.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.189+0000 7f00fdf7a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f00f80c9a20 0x7f00f8150870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:42.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.189+0000 7f00fdf7a640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f00f8150db0 0x7f00f80c5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:42.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.189+0000 7f00fdf7a640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00f81535d0 con 0x7f00f80aeb20 2024-04-25T00:28:42.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.189+0000 7f00fdf7a640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f00f8153850 con 0x7f00f80c9a20 2024-04-25T00:28:42.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.189+0000 7f00fdf7a640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f00f8153b30 con 0x7f00f8150db0 2024-04-25T00:28:42.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.189+0000 7f00f6ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f00f80c9a20 0x7f00f8150870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:28:42.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.189+0000 7f00f6ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f00f80c9a20 0x7f00f8150870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47992/0 (socket says 172.21.15.116:47992) 2024-04-25T00:28:42.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.189+0000 7f00f6ffd640 1 -- 172.21.15.116:0/3026109735 learned_addr learned my addr 172.21.15.116:0/3026109735 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:28:42.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.189+0000 7f00f7fff640 1 --2- 172.21.15.116:0/3026109735 >> v2:172.21.15.174:3300/0 conn(0x7f00f8150db0 0x7f00f80c5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:28:42.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.189+0000 7f00f6ffd640 1 -- 172.21.15.116:0/3026109735 >> v2:172.21.15.174:3300/0 conn(0x7f00f8150db0 msgr2=0x7f00f80c5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:28:42.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.189+0000 7f00f6ffd640 1 --2- 172.21.15.116:0/3026109735 >> v2:172.21.15.174:3300/0 conn(0x7f00f8150db0 0x7f00f80c5270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:42.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.189+0000 7f00f6ffd640 1 -- 172.21.15.116:0/3026109735 >> v1:172.21.15.116:6789/0 conn(0x7f00f80aeb20 legacy=0x7f00f80c9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:42.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.189+0000 7f00f6ffd640 1 -- 172.21.15.116:0/3026109735 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00f8153e10 con 0x7f00f80c9a20 2024-04-25T00:28:42.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.193+0000 7f00f6ffd640 1 --2- 172.21.15.116:0/3026109735 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00f80c9a20 0x7f00f8150870 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f00f80af340 tx=0x7f00ec01c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93acc66e997e4a76 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:28:42.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.193+0000 7f00f67fc640 1 -- 172.21.15.116:0/3026109735 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00ec0ab660 con 0x7f00f80c9a20 2024-04-25T00:28:42.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.193+0000 7f00f67fc640 1 -- 172.21.15.116:0/3026109735 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00ec0b0070 con 0x7f00f80c9a20 2024-04-25T00:28:42.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.193+0000 7f00f67fc640 1 -- 172.21.15.116:0/3026109735 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00ec0aba20 con 0x7f00f80c9a20 2024-04-25T00:28:42.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.193+0000 7f00fdf7a640 1 -- 172.21.15.116:0/3026109735 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00f80c9a20 msgr2=0x7f00f8150870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:42.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.193+0000 7f00fdf7a640 1 --2- 172.21.15.116:0/3026109735 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00f80c9a20 0x7f00f8150870 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f00f80af340 tx=0x7f00ec01c460 comp rx=0 tx=0).stop 2024-04-25T00:28:42.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.193+0000 7f00fdf7a640 1 -- 172.21.15.116:0/3026109735 shutdown_connections 2024-04-25T00:28:42.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.193+0000 7f00fdf7a640 1 --2- 172.21.15.116:0/3026109735 >> v2:172.21.15.174:3300/0 conn(0x7f00f8150db0 0x7f00f80c5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:42.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.193+0000 7f00fdf7a640 1 --2- 172.21.15.116:0/3026109735 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00f80c9a20 0x7f00f8150870 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:42.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.193+0000 7f00fdf7a640 1 -- 172.21.15.116:0/3026109735 >> 172.21.15.116:0/3026109735 conn(0x7f00f8074650 msgr2=0x7f00f8074a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:28:42.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.193+0000 7f00fdf7a640 1 -- 172.21.15.116:0/3026109735 shutdown_connections 2024-04-25T00:28:42.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.193+0000 7f00fdf7a640 1 -- 172.21.15.116:0/3026109735 wait complete. 2024-04-25T00:28:42.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.193+0000 7f00fdf7a640 1 Processor -- start 2024-04-25T00:28:42.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.193+0000 7f00fdf7a640 1 -- start start 2024-04-25T00:28:42.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.193+0000 7f00fdf7a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00f80aeb20 0x7f00f81ec950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:42.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.193+0000 7f00fdf7a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f00f80c9a20 0x7f00f814ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:42.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.193+0000 7f00fdf7a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f00f8150db0 0x7f00f80acc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:42.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.193+0000 7f00fdf7a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f00f80af080 con 0x7f00f80aeb20 2024-04-25T00:28:42.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.193+0000 7f00fdf7a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f00f80aef00 con 0x7f00f80c9a20 2024-04-25T00:28:42.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.193+0000 7f00fdf7a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f00f815ff60 con 0x7f00f8150db0 2024-04-25T00:28:42.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.193+0000 7f00f77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00f80aeb20 0x7f00f81ec950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:28:42.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.193+0000 7f00f77fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00f80aeb20 0x7f00f81ec950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:47998/0 (socket says 172.21.15.116:47998) 2024-04-25T00:28:42.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.197+0000 7f00f77fe640 1 -- 172.21.15.116:0/2779593562 learned_addr learned my addr 172.21.15.116:0/2779593562 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:28:42.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.197+0000 7f00f7fff640 1 --2- 172.21.15.116:0/2779593562 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f00f8150db0 0x7f00f80acc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:28:42.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.197+0000 7f00f6ffd640 1 --2- 172.21.15.116:0/2779593562 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f00f80c9a20 0x7f00f814ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:28:42.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.197+0000 7f00f77fe640 1 -- 172.21.15.116:0/2779593562 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f00f8150db0 msgr2=0x7f00f80acc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:42.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.197+0000 7f00f77fe640 1 --2- 172.21.15.116:0/2779593562 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f00f8150db0 0x7f00f80acc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:42.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.197+0000 7f00f77fe640 1 -- 172.21.15.116:0/2779593562 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f00f80c9a20 msgr2=0x7f00f814ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:42.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.197+0000 7f00f77fe640 1 --2- 172.21.15.116:0/2779593562 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f00f80c9a20 0x7f00f814ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:42.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.197+0000 7f00f77fe640 1 -- 172.21.15.116:0/2779593562 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00f80addc0 con 0x7f00f80aeb20 2024-04-25T00:28:42.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.197+0000 7f00f7fff640 1 --2- 172.21.15.116:0/2779593562 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f00f8150db0 0x7f00f80acc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:28:42.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.197+0000 7f00f6ffd640 1 --2- 172.21.15.116:0/2779593562 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f00f80c9a20 0x7f00f814ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:28:42.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.197+0000 7f00f77fe640 1 --2- 172.21.15.116:0/2779593562 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00f80aeb20 0x7f00f81ec950 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f00e800c450 tx=0x7f00e800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:28:42.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.197+0000 7f00dffff640 1 -- 172.21.15.116:0/2779593562 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00e8016020 con 0x7f00f80aeb20 2024-04-25T00:28:42.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.197+0000 7f00dffff640 1 -- 172.21.15.116:0/2779593562 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00e80052d0 con 0x7f00f80aeb20 2024-04-25T00:28:42.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.197+0000 7f00fdf7a640 1 -- 172.21.15.116:0/2779593562 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00f80ac1e0 con 0x7f00f80aeb20 2024-04-25T00:28:42.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.197+0000 7f00dffff640 1 -- 172.21.15.116:0/2779593562 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00e8007260 con 0x7f00f80aeb20 2024-04-25T00:28:42.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.197+0000 7f00fdf7a640 1 -- 172.21.15.116:0/2779593562 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00f80adae0 con 0x7f00f80aeb20 2024-04-25T00:28:42.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.197+0000 7f00fdf7a640 1 -- 172.21.15.116:0/2779593562 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00f8068a60 con 0x7f00f80aeb20 2024-04-25T00:28:42.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.197+0000 7f00dffff640 1 -- 172.21.15.116:0/2779593562 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f00e8007400 con 0x7f00f80aeb20 2024-04-25T00:28:42.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.197+0000 7f00dffff640 1 --2- 172.21.15.116:0/2779593562 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f00e40678c0 0x7f00e4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:42.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.197+0000 7f00dffff640 1 -- 172.21.15.116:0/2779593562 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 10364+0+0 (secure 0 0 0) 0x7f00e8086850 con 0x7f00f80aeb20 2024-04-25T00:28:42.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.197+0000 7f00f6ffd640 1 --2- 172.21.15.116:0/2779593562 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f00e40678c0 0x7f00e4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:28:42.201 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.197+0000 7f00f6ffd640 1 --2- 172.21.15.116:0/2779593562 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f00e40678c0 0x7f00e4069d80 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f00ec0af000 tx=0x7f00ec0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:28:42.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.201+0000 7f00dffff640 1 -- 172.21.15.116:0/2779593562 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f00e8020020 con 0x7f00f80aeb20 2024-04-25T00:28:42.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.461+0000 7f00fdf7a640 1 -- 172.21.15.116:0/2779593562 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f00f80c5170 con 0x7f00f80aeb20 2024-04-25T00:28:42.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.461+0000 7f00dffff640 1 -- 172.21.15.116:0/2779593562 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 227 v227) ==== 95+0+6508 (secure 0 0 0) 0x7f00f81535d0 con 0x7f00f80aeb20 2024-04-25T00:28:42.464 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:28:42.464 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":227,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":8772,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/3499878473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3499878473},{"type":"v1","addr":"172.21.15.116:6835","nonce":3499878473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":227}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:27:40.040636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6641},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6641":{"gid":6641,"name":"d","rank":0,"incarnation":223,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/2915274073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2915274073},{"type":"v1","addr":"172.21.15.174:6837","nonce":2915274073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6641,"qdb_cluster":[6641]},"id":1},{"mdsmap":{"epoch":221,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:27:19.052286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8652":{"gid":8652,"name":"c","rank":0,"incarnation":218,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6833/3891103804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3891103804},{"type":"v1","addr":"172.21.15.116:6833","nonce":3891103804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8652,"qdb_cluster":[8652]},"id":2}]} 2024-04-25T00:28:42.465 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 227 2024-04-25T00:28:42.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.465+0000 7f00fdf7a640 1 -- 172.21.15.116:0/2779593562 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f00e40678c0 msgr2=0x7f00e4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:42.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.465+0000 7f00fdf7a640 1 --2- 172.21.15.116:0/2779593562 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f00e40678c0 0x7f00e4069d80 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f00ec0af000 tx=0x7f00ec0a7000 comp rx=0 tx=0).stop 2024-04-25T00:28:42.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.465+0000 7f00fdf7a640 1 -- 172.21.15.116:0/2779593562 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00f80aeb20 msgr2=0x7f00f81ec950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:42.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.465+0000 7f00fdf7a640 1 --2- 172.21.15.116:0/2779593562 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00f80aeb20 0x7f00f81ec950 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f00e800c450 tx=0x7f00e800c910 comp rx=0 tx=0).stop 2024-04-25T00:28:42.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.465+0000 7f00fdf7a640 1 -- 172.21.15.116:0/2779593562 shutdown_connections 2024-04-25T00:28:42.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.465+0000 7f00fdf7a640 1 --2- 172.21.15.116:0/2779593562 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f00e40678c0 0x7f00e4069d80 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:42.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.465+0000 7f00fdf7a640 1 --2- 172.21.15.116:0/2779593562 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f00f8150db0 0x7f00f80acc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:42.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.465+0000 7f00fdf7a640 1 --2- 172.21.15.116:0/2779593562 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f00f80c9a20 0x7f00f814ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:42.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.465+0000 7f00fdf7a640 1 --2- 172.21.15.116:0/2779593562 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f00f80aeb20 0x7f00f81ec950 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:42.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.465+0000 7f00fdf7a640 1 -- 172.21.15.116:0/2779593562 >> 172.21.15.116:0/2779593562 conn(0x7f00f8074650 msgr2=0x7f00f81476d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:28:42.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.465+0000 7f00fdf7a640 1 -- 172.21.15.116:0/2779593562 shutdown_connections 2024-04-25T00:28:42.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:42.465+0000 7f00fdf7a640 1 -- 172.21.15.116:0/2779593562 wait complete. 2024-04-25T00:28:42.482 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:28:44.483 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:28:44.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.613+0000 7f015c684640 1 Processor -- start 2024-04-25T00:28:44.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.613+0000 7f015c684640 1 -- start start 2024-04-25T00:28:44.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.613+0000 7f015c684640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f01540a94a0 0x7f01541538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:44.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.613+0000 7f015c684640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f01540a99f0 0x7f0154153e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:44.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.613+0000 7f015c684640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01540c3910 con 0x7f0154150d20 2024-04-25T00:28:44.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.613+0000 7f015c684640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f01540c16e0 con 0x7f01540a99f0 2024-04-25T00:28:44.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.613+0000 7f015c684640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f01540c1400 con 0x7f01540a94a0 2024-04-25T00:28:44.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.613+0000 7f015ac21640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f01540a99f0 0x7f0154153e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:28:44.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.613+0000 7f015ac21640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f01540a99f0 0x7f0154153e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48004/0 (socket says 172.21.15.116:48004) 2024-04-25T00:28:44.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.613+0000 7f015ac21640 1 -- 172.21.15.116:0/43175599 learned_addr learned my addr 172.21.15.116:0/43175599 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:28:44.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.613+0000 7f0159c1f640 1 --2- 172.21.15.116:0/43175599 >> v2:172.21.15.174:3300/0 conn(0x7f01540a94a0 0x7f01541538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:28:44.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.613+0000 7f015ac21640 1 -- 172.21.15.116:0/43175599 >> v2:172.21.15.174:3300/0 conn(0x7f01540a94a0 msgr2=0x7f01541538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:28:44.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.613+0000 7f015ac21640 1 --2- 172.21.15.116:0/43175599 >> v2:172.21.15.174:3300/0 conn(0x7f01540a94a0 0x7f01541538e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:44.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.613+0000 7f015ac21640 1 -- 172.21.15.116:0/43175599 >> v1:172.21.15.116:6789/0 conn(0x7f0154150d20 legacy=0x7f01540c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:44.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.613+0000 7f015ac21640 1 -- 172.21.15.116:0/43175599 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f015406e660 con 0x7f01540a99f0 2024-04-25T00:28:44.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.617+0000 7f015ac21640 1 --2- 172.21.15.116:0/43175599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01540a99f0 0x7f0154153e20 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f0154070710 tx=0x7f014c01c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb1c84824e10b7c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:28:44.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.617+0000 7f015941e640 1 -- 172.21.15.116:0/43175599 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f014c0a56c0 con 0x7f01540a99f0 2024-04-25T00:28:44.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.617+0000 7f015941e640 1 -- 172.21.15.116:0/43175599 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f014c0a5860 con 0x7f01540a99f0 2024-04-25T00:28:44.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.617+0000 7f015941e640 1 -- 172.21.15.116:0/43175599 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f014c0a5c30 con 0x7f01540a99f0 2024-04-25T00:28:44.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.617+0000 7f015c684640 1 -- 172.21.15.116:0/43175599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01540a99f0 msgr2=0x7f0154153e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:44.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.617+0000 7f015c684640 1 --2- 172.21.15.116:0/43175599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01540a99f0 0x7f0154153e20 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f0154070710 tx=0x7f014c01c9f0 comp rx=0 tx=0).stop 2024-04-25T00:28:44.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.617+0000 7f015c684640 1 -- 172.21.15.116:0/43175599 shutdown_connections 2024-04-25T00:28:44.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.617+0000 7f015c684640 1 --2- 172.21.15.116:0/43175599 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01540a99f0 0x7f0154153e20 unknown :-1 s=CLOSED pgs=1647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:44.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.617+0000 7f015c684640 1 --2- 172.21.15.116:0/43175599 >> v2:172.21.15.174:3300/0 conn(0x7f01540a94a0 0x7f01541538e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:44.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.617+0000 7f015c684640 1 -- 172.21.15.116:0/43175599 >> 172.21.15.116:0/43175599 conn(0x7f01540c3b00 msgr2=0x7f01540c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:28:44.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.617+0000 7f015c684640 1 -- 172.21.15.116:0/43175599 shutdown_connections 2024-04-25T00:28:44.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.617+0000 7f015c684640 1 -- 172.21.15.116:0/43175599 wait complete. 2024-04-25T00:28:44.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.621+0000 7f015c684640 1 Processor -- start 2024-04-25T00:28:44.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.621+0000 7f015c684640 1 -- start start 2024-04-25T00:28:44.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.621+0000 7f015c684640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f01540a94a0 0x7f015414bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:44.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.621+0000 7f015c684640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f01540a99f0 0x7f01541408d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:44.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.621+0000 7f015c684640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0154150d20 0x7f0154140e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:44.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.621+0000 7f015c684640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f01541544e0 con 0x7f0154150d20 2024-04-25T00:28:44.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.621+0000 7f015c684640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0154154360 con 0x7f01540a94a0 2024-04-25T00:28:44.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.621+0000 7f015c684640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f0154160730 con 0x7f01540a99f0 2024-04-25T00:28:44.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.621+0000 7f015ac21640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0154150d20 0x7f0154140e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:28:44.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.621+0000 7f015ac21640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0154150d20 0x7f0154140e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:48012/0 (socket says 172.21.15.116:48012) 2024-04-25T00:28:44.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.621+0000 7f015ac21640 1 -- 172.21.15.116:0/1225116595 learned_addr learned my addr 172.21.15.116:0/1225116595 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:28:44.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.621+0000 7f015a420640 1 --2- 172.21.15.116:0/1225116595 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f01540a94a0 0x7f015414bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:28:44.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.621+0000 7f0159c1f640 1 --2- 172.21.15.116:0/1225116595 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f01540a99f0 0x7f01541408d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:28:44.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.621+0000 7f015ac21640 1 -- 172.21.15.116:0/1225116595 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f01540a99f0 msgr2=0x7f01541408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:44.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.621+0000 7f015ac21640 1 --2- 172.21.15.116:0/1225116595 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f01540a99f0 0x7f01541408d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:44.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.621+0000 7f015ac21640 1 -- 172.21.15.116:0/1225116595 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f01540a94a0 msgr2=0x7f015414bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:44.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.621+0000 7f015ac21640 1 --2- 172.21.15.116:0/1225116595 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f01540a94a0 0x7f015414bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:44.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.621+0000 7f015ac21640 1 -- 172.21.15.116:0/1225116595 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0154141a20 con 0x7f0154150d20 2024-04-25T00:28:44.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.621+0000 7f015ac21640 1 --2- 172.21.15.116:0/1225116595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0154150d20 0x7f0154140e10 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f014c0af040 tx=0x7f014c019790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:28:44.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.621+0000 7f014affd640 1 -- 172.21.15.116:0/1225116595 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f014c0bc020 con 0x7f0154150d20 2024-04-25T00:28:44.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.621+0000 7f014affd640 1 -- 172.21.15.116:0/1225116595 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f014c0b8690 con 0x7f0154150d20 2024-04-25T00:28:44.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.621+0000 7f014affd640 1 -- 172.21.15.116:0/1225116595 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f014c0b8a80 con 0x7f0154150d20 2024-04-25T00:28:44.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.621+0000 7f015c684640 1 -- 172.21.15.116:0/1225116595 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0154141740 con 0x7f0154150d20 2024-04-25T00:28:44.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.621+0000 7f015c684640 1 -- 172.21.15.116:0/1225116595 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f015414c320 con 0x7f0154150d20 2024-04-25T00:28:44.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.621+0000 7f015c684640 1 -- 172.21.15.116:0/1225116595 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0154068350 con 0x7f0154150d20 2024-04-25T00:28:44.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.621+0000 7f014affd640 1 -- 172.21.15.116:0/1225116595 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f014c0bf020 con 0x7f0154150d20 2024-04-25T00:28:44.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.621+0000 7f014affd640 1 --2- 172.21.15.116:0/1225116595 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f01380678c0 0x7f0138069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:44.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.625+0000 7f014affd640 1 -- 172.21.15.116:0/1225116595 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 10364+0+0 (secure 0 0 0) 0x7f014c121eb0 con 0x7f0154150d20 2024-04-25T00:28:44.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.625+0000 7f015a420640 1 --2- 172.21.15.116:0/1225116595 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f01380678c0 0x7f0138069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:28:44.626 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.625+0000 7f015a420640 1 --2- 172.21.15.116:0/1225116595 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f01380678c0 0x7f0138069d80 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f0150001650 tx=0x7f0150009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:28:44.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.629+0000 7f014affd640 1 -- 172.21.15.116:0/1225116595 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f014c0c6030 con 0x7f0154150d20 2024-04-25T00:28:44.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.821+0000 7f015c684640 1 -- 172.21.15.116:0/1225116595 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f01540c34f0 con 0x7f0154150d20 2024-04-25T00:28:44.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.821+0000 7f014affd640 1 -- 172.21.15.116:0/1225116595 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 227 v227) ==== 95+0+6508 (secure 0 0 0) 0x7f014c0165e0 con 0x7f0154150d20 2024-04-25T00:28:44.824 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:28:44.824 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":227,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":8772,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/3499878473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3499878473},{"type":"v1","addr":"172.21.15.116:6835","nonce":3499878473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":227}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:27:40.040636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6641},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6641":{"gid":6641,"name":"d","rank":0,"incarnation":223,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/2915274073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2915274073},{"type":"v1","addr":"172.21.15.174:6837","nonce":2915274073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6641,"qdb_cluster":[6641]},"id":1},{"mdsmap":{"epoch":221,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:27:19.052286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8652":{"gid":8652,"name":"c","rank":0,"incarnation":218,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6833/3891103804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3891103804},{"type":"v1","addr":"172.21.15.116:6833","nonce":3891103804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8652,"qdb_cluster":[8652]},"id":2}]} 2024-04-25T00:28:44.824 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 227 2024-04-25T00:28:44.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.825+0000 7f015c684640 1 -- 172.21.15.116:0/1225116595 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f01380678c0 msgr2=0x7f0138069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:44.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.825+0000 7f015c684640 1 --2- 172.21.15.116:0/1225116595 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f01380678c0 0x7f0138069d80 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f0150001650 tx=0x7f0150009040 comp rx=0 tx=0).stop 2024-04-25T00:28:44.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.825+0000 7f015c684640 1 -- 172.21.15.116:0/1225116595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0154150d20 msgr2=0x7f0154140e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:44.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.825+0000 7f015c684640 1 --2- 172.21.15.116:0/1225116595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0154150d20 0x7f0154140e10 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f014c0af040 tx=0x7f014c019790 comp rx=0 tx=0).stop 2024-04-25T00:28:44.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.825+0000 7f015c684640 1 -- 172.21.15.116:0/1225116595 shutdown_connections 2024-04-25T00:28:44.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.825+0000 7f015c684640 1 --2- 172.21.15.116:0/1225116595 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f01380678c0 0x7f0138069d80 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:44.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.825+0000 7f015c684640 1 --2- 172.21.15.116:0/1225116595 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0154150d20 0x7f0154140e10 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:44.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.825+0000 7f015c684640 1 --2- 172.21.15.116:0/1225116595 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f01540a99f0 0x7f01541408d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:44.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.825+0000 7f015c684640 1 --2- 172.21.15.116:0/1225116595 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f01540a94a0 0x7f015414bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:44.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.825+0000 7f015c684640 1 -- 172.21.15.116:0/1225116595 >> 172.21.15.116:0/1225116595 conn(0x7f01540c3b00 msgr2=0x7f015406cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:28:44.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.825+0000 7f015c684640 1 -- 172.21.15.116:0/1225116595 shutdown_connections 2024-04-25T00:28:44.828 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:44.825+0000 7f015c684640 1 -- 172.21.15.116:0/1225116595 wait complete. 2024-04-25T00:28:44.844 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:28:46.845 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:28:46.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.977+0000 7f19ba02d640 1 Processor -- start 2024-04-25T00:28:46.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.977+0000 7f19ba02d640 1 -- start start 2024-04-25T00:28:46.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.977+0000 7f19ba02d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f19b406d5f0 0x7f19b4156c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:46.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.977+0000 7f19ba02d640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f19b4155810 0x7f19b41571c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:46.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.977+0000 7f19ba02d640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19b40c9150 con 0x7f19b4152d80 2024-04-25T00:28:46.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.977+0000 7f19ba02d640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f19b40c9d60 con 0x7f19b4155810 2024-04-25T00:28:46.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.977+0000 7f19ba02d640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f19b40c9430 con 0x7f19b406d5f0 2024-04-25T00:28:46.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.977+0000 7f19b37fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f19b4152d80 0x7f19b406bd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:49208/0 (socket says 172.21.15.116:49208) 2024-04-25T00:28:46.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.977+0000 7f19b37fe640 1 -- 172.21.15.116:0/1201395683 learned_addr learned my addr 172.21.15.116:0/1201395683 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:28:46.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.977+0000 7f19b3fff640 1 --2- 172.21.15.116:0/1201395683 >> v2:172.21.15.174:3300/0 conn(0x7f19b4155810 0x7f19b41571c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:28:46.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.977+0000 7f19b2ffd640 1 --2- 172.21.15.116:0/1201395683 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f19b406d5f0 0x7f19b4156c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:28:46.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.977+0000 7f19b27fc640 1 -- 172.21.15.116:0/1201395683 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2745337118 0 0) 0x7f19b40c9150 con 0x7f19b4152d80 2024-04-25T00:28:46.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.977+0000 7f19b27fc640 1 -- 172.21.15.116:0/1201395683 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f199c003530 con 0x7f19b4152d80 2024-04-25T00:28:46.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.977+0000 7f19b27fc640 1 -- 172.21.15.116:0/1201395683 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1204559708 0 0) 0x7f199c003530 con 0x7f19b4152d80 2024-04-25T00:28:46.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.977+0000 7f19b27fc640 1 -- 172.21.15.116:0/1201395683 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f19b406d5f0 msgr2=0x7f19b4156c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:28:46.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.977+0000 7f19b27fc640 1 --2- 172.21.15.116:0/1201395683 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f19b406d5f0 0x7f19b4156c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:46.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.977+0000 7f19b27fc640 1 -- 172.21.15.116:0/1201395683 >> v2:172.21.15.174:3300/0 conn(0x7f19b4155810 msgr2=0x7f19b41571c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:28:46.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.977+0000 7f19b27fc640 1 --2- 172.21.15.116:0/1201395683 >> v2:172.21.15.174:3300/0 conn(0x7f19b4155810 0x7f19b41571c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:46.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.977+0000 7f19b27fc640 1 -- 172.21.15.116:0/1201395683 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19b40c9710 con 0x7f19b4152d80 2024-04-25T00:28:46.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.977+0000 7f19b27fc640 1 -- 172.21.15.116:0/1201395683 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f19a4001680 con 0x7f19b4152d80 2024-04-25T00:28:46.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.977+0000 7f19b27fc640 1 -- 172.21.15.116:0/1201395683 >> v1:172.21.15.116:6789/0 conn(0x7f19b4152d80 legacy=0x7f19b406bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:46.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.977+0000 7f19b27fc640 1 --2- 172.21.15.116:0/1201395683 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f19b406d5f0 0x7f199c0039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:46.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.977+0000 7f19b27fc640 1 --2- 172.21.15.116:0/1201395683 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f199c003f40 0x7f199c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:46.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.977+0000 7f19b27fc640 1 --2- 172.21.15.116:0/1201395683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f199c004d50 0x7f199c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:46.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.977+0000 7f19b27fc640 1 -- 172.21.15.116:0/1201395683 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f19b40c9d60 con 0x7f199c004d50 2024-04-25T00:28:46.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.977+0000 7f19b27fc640 1 -- 172.21.15.116:0/1201395683 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f19b40c9430 con 0x7f199c003f40 2024-04-25T00:28:46.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.977+0000 7f19b27fc640 1 -- 172.21.15.116:0/1201395683 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f199c00bab0 con 0x7f19b406d5f0 2024-04-25T00:28:46.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.977+0000 7f19b3fff640 1 --2- 172.21.15.116:0/1201395683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f199c004d50 0x7f199c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:28:46.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.977+0000 7f19b37fe640 1 --2- 172.21.15.116:0/1201395683 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f19b406d5f0 0x7f199c0039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:28:46.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.977+0000 7f19b3fff640 1 -- 172.21.15.116:0/1201395683 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f19b406d5f0 msgr2=0x7f199c0039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:28:46.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.977+0000 7f19b3fff640 1 --2- 172.21.15.116:0/1201395683 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f19b406d5f0 0x7f199c0039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:46.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.977+0000 7f19b3fff640 1 -- 172.21.15.116:0/1201395683 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f199c003f40 msgr2=0x7f199c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:28:46.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.977+0000 7f19b3fff640 1 --2- 172.21.15.116:0/1201395683 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f199c003f40 0x7f199c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:46.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.977+0000 7f19b3fff640 1 -- 172.21.15.116:0/1201395683 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f199c003530 con 0x7f199c004d50 2024-04-25T00:28:46.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.981+0000 7f19b3fff640 1 --2- 172.21.15.116:0/1201395683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f199c004d50 0x7f199c0051d0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f19ac009fc0 tx=0x7f19ac01cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a7fd9f4957f8832 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:28:46.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.981+0000 7f19b27fc640 1 -- 172.21.15.116:0/1201395683 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19ac0b2070 con 0x7f199c004d50 2024-04-25T00:28:46.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.981+0000 7f19b27fc640 1 -- 172.21.15.116:0/1201395683 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19ac0a5bb0 con 0x7f199c004d50 2024-04-25T00:28:46.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.981+0000 7f19b27fc640 1 -- 172.21.15.116:0/1201395683 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19ac009c10 con 0x7f199c004d50 2024-04-25T00:28:46.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.981+0000 7f19ba02d640 1 -- 172.21.15.116:0/1201395683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f199c004d50 msgr2=0x7f199c0051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:46.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.981+0000 7f19ba02d640 1 --2- 172.21.15.116:0/1201395683 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f199c004d50 0x7f199c0051d0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f19ac009fc0 tx=0x7f19ac01cc90 comp rx=0 tx=0).stop 2024-04-25T00:28:46.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.981+0000 7f19b3fff640 1 -- 172.21.15.116:0/1201395683 reap_dead start 2024-04-25T00:28:46.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.981+0000 7f19ba02d640 1 -- 172.21.15.116:0/1201395683 shutdown_connections 2024-04-25T00:28:46.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.981+0000 7f19ba02d640 1 -- 172.21.15.116:0/1201395683 >> 172.21.15.116:0/1201395683 conn(0x7f19b40acdd0 msgr2=0x7f19b40ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:28:46.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.981+0000 7f19ba02d640 1 -- 172.21.15.116:0/1201395683 shutdown_connections 2024-04-25T00:28:46.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.981+0000 7f19ba02d640 1 -- 172.21.15.116:0/1201395683 wait complete. 2024-04-25T00:28:46.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.981+0000 7f19ba02d640 1 Processor -- start 2024-04-25T00:28:46.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.981+0000 7f19ba02d640 1 -- start start 2024-04-25T00:28:46.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.981+0000 7f19ba02d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f19b40abe50 0x7f19b414d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:46.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.981+0000 7f19ba02d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19b41f1c00 0x7f19b4151f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:46.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.981+0000 7f19ba02d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f19b414c360 0x7f19b414c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:46.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.981+0000 7f19ba02d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f19b40caf00 con 0x7f19b41f1c00 2024-04-25T00:28:46.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.981+0000 7f19ba02d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f19b40cad80 con 0x7f19b40abe50 2024-04-25T00:28:46.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.981+0000 7f19ba02d640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f19b41608c0 con 0x7f19b414c360 2024-04-25T00:28:46.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.981+0000 7f19b2ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19b41f1c00 0x7f19b4151f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:28:46.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.981+0000 7f19b2ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19b41f1c00 0x7f19b4151f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34856/0 (socket says 172.21.15.116:34856) 2024-04-25T00:28:46.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.981+0000 7f19b2ffd640 1 -- 172.21.15.116:0/1199289076 learned_addr learned my addr 172.21.15.116:0/1199289076 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:28:46.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.985+0000 7f19b3fff640 1 --2- 172.21.15.116:0/1199289076 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f19b414c360 0x7f19b414c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:28:46.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.985+0000 7f19b2ffd640 1 -- 172.21.15.116:0/1199289076 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f19b414c360 msgr2=0x7f19b414c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:46.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.985+0000 7f19b2ffd640 1 --2- 172.21.15.116:0/1199289076 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f19b414c360 0x7f19b414c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:46.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.985+0000 7f19b2ffd640 1 -- 172.21.15.116:0/1199289076 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f19b40abe50 msgr2=0x7f19b414d660 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T00:28:46.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.985+0000 7f19b2ffd640 1 --2- 172.21.15.116:0/1199289076 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f19b40abe50 0x7f19b414d660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:46.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.985+0000 7f19b2ffd640 1 -- 172.21.15.116:0/1199289076 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19b414d0c0 con 0x7f19b41f1c00 2024-04-25T00:28:46.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.985+0000 7f19b2ffd640 1 --2- 172.21.15.116:0/1199289076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19b41f1c00 0x7f19b4151f90 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f19a800e7c0 tx=0x7f19a800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:28:46.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.985+0000 7f199bfff640 1 -- 172.21.15.116:0/1199289076 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19a8017070 con 0x7f19b41f1c00 2024-04-25T00:28:46.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.985+0000 7f199bfff640 1 -- 172.21.15.116:0/1199289076 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19a800ee70 con 0x7f19b41f1c00 2024-04-25T00:28:46.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.985+0000 7f199bfff640 1 -- 172.21.15.116:0/1199289076 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19a800cae0 con 0x7f19b41f1c00 2024-04-25T00:28:46.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.985+0000 7f19ba02d640 1 -- 172.21.15.116:0/1199289076 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19b414d380 con 0x7f19b41f1c00 2024-04-25T00:28:46.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.985+0000 7f19ba02d640 1 -- 172.21.15.116:0/1199289076 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19b4151550 con 0x7f19b41f1c00 2024-04-25T00:28:46.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.985+0000 7f19ba02d640 1 -- 172.21.15.116:0/1199289076 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19b4068350 con 0x7f19b41f1c00 2024-04-25T00:28:46.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.985+0000 7f199bfff640 1 -- 172.21.15.116:0/1199289076 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f19a801a070 con 0x7f19b41f1c00 2024-04-25T00:28:46.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.985+0000 7f199bfff640 1 --2- 172.21.15.116:0/1199289076 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f19a00678c0 0x7f19a0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:46.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.985+0000 7f199bfff640 1 -- 172.21.15.116:0/1199289076 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 10364+0+0 (secure 0 0 0) 0x7f19a8086cd0 con 0x7f19b41f1c00 2024-04-25T00:28:46.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.985+0000 7f19b37fe640 1 --2- 172.21.15.116:0/1199289076 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f19a00678c0 0x7f19a0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:28:46.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.985+0000 7f19b37fe640 1 --2- 172.21.15.116:0/1199289076 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f19a00678c0 0x7f19a0069d80 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f19a4005a60 tx=0x7f19a40047f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:28:46.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:46.989+0000 7f199bfff640 1 -- 172.21.15.116:0/1199289076 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f19a80565c0 con 0x7f19b41f1c00 2024-04-25T00:28:47.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:47.177+0000 7f19ba02d640 1 -- 172.21.15.116:0/1199289076 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f19b4151bb0 con 0x7f19b41f1c00 2024-04-25T00:28:47.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:47.177+0000 7f199bfff640 1 -- 172.21.15.116:0/1199289076 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 227 v227) ==== 95+0+6508 (secure 0 0 0) 0x7f19a8055ec0 con 0x7f19b41f1c00 2024-04-25T00:28:47.181 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:28:47.181 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":227,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":8772,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/3499878473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3499878473},{"type":"v1","addr":"172.21.15.116:6835","nonce":3499878473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":227}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:27:40.040636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6641},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6641":{"gid":6641,"name":"d","rank":0,"incarnation":223,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/2915274073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2915274073},{"type":"v1","addr":"172.21.15.174:6837","nonce":2915274073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6641,"qdb_cluster":[6641]},"id":1},{"mdsmap":{"epoch":221,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:27:19.052286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8652":{"gid":8652,"name":"c","rank":0,"incarnation":218,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6833/3891103804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3891103804},{"type":"v1","addr":"172.21.15.116:6833","nonce":3891103804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8652,"qdb_cluster":[8652]},"id":2}]} 2024-04-25T00:28:47.181 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 227 2024-04-25T00:28:47.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:47.181+0000 7f19ba02d640 1 -- 172.21.15.116:0/1199289076 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f19a00678c0 msgr2=0x7f19a0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:47.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:47.181+0000 7f19ba02d640 1 --2- 172.21.15.116:0/1199289076 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f19a00678c0 0x7f19a0069d80 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f19a4005a60 tx=0x7f19a40047f0 comp rx=0 tx=0).stop 2024-04-25T00:28:47.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:47.181+0000 7f19ba02d640 1 -- 172.21.15.116:0/1199289076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19b41f1c00 msgr2=0x7f19b4151f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:47.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:47.181+0000 7f19ba02d640 1 --2- 172.21.15.116:0/1199289076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19b41f1c00 0x7f19b4151f90 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f19a800e7c0 tx=0x7f19a800ec80 comp rx=0 tx=0).stop 2024-04-25T00:28:47.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:47.181+0000 7f19ba02d640 1 -- 172.21.15.116:0/1199289076 shutdown_connections 2024-04-25T00:28:47.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:47.181+0000 7f19ba02d640 1 --2- 172.21.15.116:0/1199289076 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f19a00678c0 0x7f19a0069d80 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:47.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:47.181+0000 7f19ba02d640 1 --2- 172.21.15.116:0/1199289076 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f19b414c360 0x7f19b414c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:47.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:47.181+0000 7f19ba02d640 1 --2- 172.21.15.116:0/1199289076 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f19b41f1c00 0x7f19b4151f90 unknown :-1 s=CLOSED pgs=1651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:47.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:47.181+0000 7f19ba02d640 1 --2- 172.21.15.116:0/1199289076 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f19b40abe50 0x7f19b414d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:47.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:47.181+0000 7f19ba02d640 1 -- 172.21.15.116:0/1199289076 >> 172.21.15.116:0/1199289076 conn(0x7f19b40acdd0 msgr2=0x7f19b4157b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:28:47.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:47.181+0000 7f19ba02d640 1 -- 172.21.15.116:0/1199289076 shutdown_connections 2024-04-25T00:28:47.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:47.181+0000 7f19ba02d640 1 -- 172.21.15.116:0/1199289076 wait complete. 2024-04-25T00:28:47.198 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:28:49.199 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:28:49.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.329+0000 7f5a4942a640 1 Processor -- start 2024-04-25T00:28:49.332 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.329+0000 7f5a4942a640 1 -- start start 2024-04-25T00:28:49.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.333+0000 7f5a4942a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5a44150d20 0x7f5a440c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:49.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.333+0000 7f5a4942a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5a440a94a0 0x7f5a44153ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:49.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.333+0000 7f5a4942a640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a440c3910 con 0x7f5a440c2f50 2024-04-25T00:28:49.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.333+0000 7f5a4942a640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f5a44154530 con 0x7f5a440a94a0 2024-04-25T00:28:49.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.333+0000 7f5a4942a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f5a440c16e0 con 0x7f5a44150d20 2024-04-25T00:28:49.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.333+0000 7f5a437fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5a440a94a0 0x7f5a44153ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:28:49.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.333+0000 7f5a437fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5a440a94a0 0x7f5a44153ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34864/0 (socket says 172.21.15.116:34864) 2024-04-25T00:28:49.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.333+0000 7f5a437fe640 1 -- 172.21.15.116:0/1639954196 learned_addr learned my addr 172.21.15.116:0/1639954196 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:28:49.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.333+0000 7f5a42ffd640 1 --2- 172.21.15.116:0/1639954196 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5a44150d20 0x7f5a440c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:28:49.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.333+0000 7f5a437fe640 1 -- 172.21.15.116:0/1639954196 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5a44150d20 msgr2=0x7f5a440c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:28:49.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.333+0000 7f5a437fe640 1 --2- 172.21.15.116:0/1639954196 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5a44150d20 0x7f5a440c2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:49.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.333+0000 7f5a437fe640 1 -- 172.21.15.116:0/1639954196 >> v1:172.21.15.174:6789/0 conn(0x7f5a440c2f50 legacy=0x7f5a441538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:28:49.335 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.333+0000 7f5a437fe640 1 -- 172.21.15.116:0/1639954196 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a440c1400 con 0x7f5a440a94a0 2024-04-25T00:28:49.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.333+0000 7f5a437fe640 1 --2- 172.21.15.116:0/1639954196 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a440a94a0 0x7f5a44153ff0 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c009e50 tx=0x7f5a3c01ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1862f361af59e9b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:28:49.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.333+0000 7f5a41ffb640 1 -- 172.21.15.116:0/1639954196 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a3c0a5810 con 0x7f5a440a94a0 2024-04-25T00:28:49.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.333+0000 7f5a41ffb640 1 -- 172.21.15.116:0/1639954196 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a3c0a59b0 con 0x7f5a440a94a0 2024-04-25T00:28:49.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.333+0000 7f5a41ffb640 1 -- 172.21.15.116:0/1639954196 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a3c0a5d80 con 0x7f5a440a94a0 2024-04-25T00:28:49.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.333+0000 7f5a4942a640 1 -- 172.21.15.116:0/1639954196 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a440a94a0 msgr2=0x7f5a44153ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:49.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.333+0000 7f5a4942a640 1 --2- 172.21.15.116:0/1639954196 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a440a94a0 0x7f5a44153ff0 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c009e50 tx=0x7f5a3c01ca30 comp rx=0 tx=0).stop 2024-04-25T00:28:49.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.333+0000 7f5a4942a640 1 -- 172.21.15.116:0/1639954196 shutdown_connections 2024-04-25T00:28:49.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.333+0000 7f5a4942a640 1 --2- 172.21.15.116:0/1639954196 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a440a94a0 0x7f5a44153ff0 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:49.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.333+0000 7f5a4942a640 1 --2- 172.21.15.116:0/1639954196 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5a44150d20 0x7f5a440c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:49.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.333+0000 7f5a4942a640 1 -- 172.21.15.116:0/1639954196 >> 172.21.15.116:0/1639954196 conn(0x7f5a440c3b00 msgr2=0x7f5a440c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:28:49.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.333+0000 7f5a4942a640 1 -- 172.21.15.116:0/1639954196 shutdown_connections 2024-04-25T00:28:49.337 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.333+0000 7f5a4942a640 1 -- 172.21.15.116:0/1639954196 wait complete. 2024-04-25T00:28:49.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.337+0000 7f5a4942a640 1 Processor -- start 2024-04-25T00:28:49.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.337+0000 7f5a4942a640 1 -- start start 2024-04-25T00:28:49.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.337+0000 7f5a4942a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5a440a94a0 0x7f5a441f26f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:49.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.337+0000 7f5a4942a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a440c2f50 0x7f5a44070d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:49.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.337+0000 7f5a4942a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5a44150d20 0x7f5a44071660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:49.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.337+0000 7f5a4942a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5a441608b0 con 0x7f5a440c2f50 2024-04-25T00:28:49.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.337+0000 7f5a4942a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5a44160730 con 0x7f5a44150d20 2024-04-25T00:28:49.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.337+0000 7f5a4942a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f5a44160a30 con 0x7f5a440a94a0 2024-04-25T00:28:49.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.337+0000 7f5a427fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a440c2f50 0x7f5a44070d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:28:49.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.337+0000 7f5a427fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a440c2f50 0x7f5a44070d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34876/0 (socket says 172.21.15.116:34876) 2024-04-25T00:28:49.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.337+0000 7f5a427fc640 1 -- 172.21.15.116:0/2405277770 learned_addr learned my addr 172.21.15.116:0/2405277770 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:28:49.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.337+0000 7f5a42ffd640 1 --2- 172.21.15.116:0/2405277770 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5a440a94a0 0x7f5a441f26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:28:49.340 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.337+0000 7f5a437fe640 1 --2- 172.21.15.116:0/2405277770 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5a44150d20 0x7f5a44071660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:28:49.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.337+0000 7f5a427fc640 1 -- 172.21.15.116:0/2405277770 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5a440a94a0 msgr2=0x7f5a441f26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:49.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.337+0000 7f5a427fc640 1 --2- 172.21.15.116:0/2405277770 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5a440a94a0 0x7f5a441f26f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:49.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.337+0000 7f5a427fc640 1 -- 172.21.15.116:0/2405277770 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5a44150d20 msgr2=0x7f5a44071660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:49.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.337+0000 7f5a427fc640 1 --2- 172.21.15.116:0/2405277770 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5a44150d20 0x7f5a44071660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:49.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.337+0000 7f5a427fc640 1 -- 172.21.15.116:0/2405277770 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a441f1b60 con 0x7f5a440c2f50 2024-04-25T00:28:49.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.337+0000 7f5a437fe640 1 --2- 172.21.15.116:0/2405277770 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5a44150d20 0x7f5a44071660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:28:49.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.337+0000 7f5a427fc640 1 --2- 172.21.15.116:0/2405277770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a440c2f50 0x7f5a44070d40 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f5a3800c420 tx=0x7f5a3800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:28:49.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.337+0000 7f5a2b7fe640 1 -- 172.21.15.116:0/2405277770 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a380052d0 con 0x7f5a440c2f50 2024-04-25T00:28:49.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.337+0000 7f5a2b7fe640 1 -- 172.21.15.116:0/2405277770 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a38005470 con 0x7f5a440c2f50 2024-04-25T00:28:49.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.337+0000 7f5a4942a640 1 -- 172.21.15.116:0/2405277770 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a4414d5d0 con 0x7f5a440c2f50 2024-04-25T00:28:49.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.337+0000 7f5a2b7fe640 1 -- 172.21.15.116:0/2405277770 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a38007260 con 0x7f5a440c2f50 2024-04-25T00:28:49.341 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.337+0000 7f5a4942a640 1 -- 172.21.15.116:0/2405277770 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a4414db90 con 0x7f5a440c2f50 2024-04-25T00:28:49.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.337+0000 7f5a4942a640 1 -- 172.21.15.116:0/2405277770 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a44068350 con 0x7f5a440c2f50 2024-04-25T00:28:49.342 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.341+0000 7f5a2b7fe640 1 -- 172.21.15.116:0/2405277770 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5a38007400 con 0x7f5a440c2f50 2024-04-25T00:28:49.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.341+0000 7f5a2b7fe640 1 --2- 172.21.15.116:0/2405277770 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5a300678c0 0x7f5a30069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:49.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.341+0000 7f5a2b7fe640 1 -- 172.21.15.116:0/2405277770 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 10364+0+0 (secure 0 0 0) 0x7f5a38086450 con 0x7f5a440c2f50 2024-04-25T00:28:49.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.341+0000 7f5a42ffd640 1 --2- 172.21.15.116:0/2405277770 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5a300678c0 0x7f5a30069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:28:49.343 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.341+0000 7f5a42ffd640 1 --2- 172.21.15.116:0/2405277770 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5a300678c0 0x7f5a30069d80 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f5a3400a8d0 tx=0x7f5a34008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:28:49.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.345+0000 7f5a2b7fe640 1 -- 172.21.15.116:0/2405277770 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f5a38055d10 con 0x7f5a440c2f50 2024-04-25T00:28:49.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.541+0000 7f5a4942a640 1 -- 172.21.15.116:0/2405277770 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5a4415dcf0 con 0x7f5a440c2f50 2024-04-25T00:28:49.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.541+0000 7f5a2b7fe640 1 -- 172.21.15.116:0/2405277770 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 227 v227) ==== 95+0+6508 (secure 0 0 0) 0x7f5a38055610 con 0x7f5a440c2f50 2024-04-25T00:28:49.544 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:28:49.544 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":227,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":8772,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/3499878473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3499878473},{"type":"v1","addr":"172.21.15.116:6835","nonce":3499878473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":227}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:27:40.040636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6641},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6641":{"gid":6641,"name":"d","rank":0,"incarnation":223,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/2915274073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2915274073},{"type":"v1","addr":"172.21.15.174:6837","nonce":2915274073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6641,"qdb_cluster":[6641]},"id":1},{"mdsmap":{"epoch":221,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:27:19.052286+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8652},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8652":{"gid":8652,"name":"c","rank":0,"incarnation":218,"state":"up:active","state_seq":9,"addr":"172.21.15.116:6833/3891103804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":3891103804},{"type":"v1","addr":"172.21.15.116:6833","nonce":3891103804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8652,"qdb_cluster":[8652]},"id":2}]} 2024-04-25T00:28:49.544 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 227 2024-04-25T00:28:49.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.545+0000 7f5a4942a640 1 -- 172.21.15.116:0/2405277770 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5a300678c0 msgr2=0x7f5a30069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:49.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.545+0000 7f5a4942a640 1 --2- 172.21.15.116:0/2405277770 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5a300678c0 0x7f5a30069d80 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f5a3400a8d0 tx=0x7f5a34008040 comp rx=0 tx=0).stop 2024-04-25T00:28:49.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.545+0000 7f5a4942a640 1 -- 172.21.15.116:0/2405277770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a440c2f50 msgr2=0x7f5a44070d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:49.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.545+0000 7f5a4942a640 1 --2- 172.21.15.116:0/2405277770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a440c2f50 0x7f5a44070d40 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f5a3800c420 tx=0x7f5a3800c8e0 comp rx=0 tx=0).stop 2024-04-25T00:28:49.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.545+0000 7f5a4942a640 1 -- 172.21.15.116:0/2405277770 shutdown_connections 2024-04-25T00:28:49.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.545+0000 7f5a4942a640 1 --2- 172.21.15.116:0/2405277770 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5a300678c0 0x7f5a30069d80 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:49.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.545+0000 7f5a4942a640 1 --2- 172.21.15.116:0/2405277770 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5a44150d20 0x7f5a44071660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:49.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.545+0000 7f5a4942a640 1 --2- 172.21.15.116:0/2405277770 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a440c2f50 0x7f5a44070d40 unknown :-1 s=CLOSED pgs=1653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:49.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.545+0000 7f5a4942a640 1 --2- 172.21.15.116:0/2405277770 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5a440a94a0 0x7f5a441f26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:49.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.545+0000 7f5a4942a640 1 -- 172.21.15.116:0/2405277770 >> 172.21.15.116:0/2405277770 conn(0x7f5a440c3b00 msgr2=0x7f5a4406cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:28:49.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.545+0000 7f5a4942a640 1 -- 172.21.15.116:0/2405277770 shutdown_connections 2024-04-25T00:28:49.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:49.545+0000 7f5a4942a640 1 -- 172.21.15.116:0/2405277770 wait complete. 2024-04-25T00:28:49.564 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:28:51.565 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:28:51.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.689+0000 7f25a92b7640 1 Processor -- start 2024-04-25T00:28:51.689 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.689+0000 7f25a92b7640 1 -- start start 2024-04-25T00:28:51.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.689+0000 7f25a92b7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f25a4071d90 0x7f25a41563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:51.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.689+0000 7f25a92b7640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25a4071220 con 0x7f25a40a94a0 2024-04-25T00:28:51.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.689+0000 7f25a92b7640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25a4152420 con 0x7f25a414c720 2024-04-25T00:28:51.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.689+0000 7f25a92b7640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f25a4153460 con 0x7f25a4071d90 2024-04-25T00:28:51.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.689+0000 7f25a37fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f25a4071d90 0x7f25a41563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:28:51.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.689+0000 7f25a27fc640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f25a40a94a0 0x7f25a4071680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:49222/0 (socket says 172.21.15.116:49222) 2024-04-25T00:28:51.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.689+0000 7f25a37fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f25a4071d90 0x7f25a41563d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34892/0 (socket says 172.21.15.116:34892) 2024-04-25T00:28:51.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.689+0000 7f25a27fc640 1 -- 172.21.15.116:0/2430872110 learned_addr learned my addr 172.21.15.116:0/2430872110 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:28:51.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.689+0000 7f25a37fe640 1 -- 172.21.15.116:0/2430872110 >> v1:172.21.15.174:6789/0 conn(0x7f25a414c720 legacy=0x7f25a40c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:28:51.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.689+0000 7f25a37fe640 1 -- 172.21.15.116:0/2430872110 >> v1:172.21.15.116:6789/0 conn(0x7f25a40a94a0 legacy=0x7f25a4071680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:51.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.689+0000 7f25a37fe640 1 -- 172.21.15.116:0/2430872110 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25a40721c0 con 0x7f25a4071d90 2024-04-25T00:28:51.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.689+0000 7f25a37fe640 1 --2- 172.21.15.116:0/2430872110 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25a4071d90 0x7f25a41563d0 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f259c0097c0 tx=0x7f259c01c1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a0220923b49d52f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:28:51.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.689+0000 7f25a1ffb640 1 -- 172.21.15.116:0/2430872110 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f259c0b0070 con 0x7f25a4071d90 2024-04-25T00:28:51.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.693+0000 7f25a1ffb640 1 -- 172.21.15.116:0/2430872110 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f259c0ab600 con 0x7f25a4071d90 2024-04-25T00:28:51.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.693+0000 7f25a1ffb640 1 -- 172.21.15.116:0/2430872110 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f259c0ab9d0 con 0x7f25a4071d90 2024-04-25T00:28:51.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.693+0000 7f25a92b7640 1 -- 172.21.15.116:0/2430872110 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25a4071d90 msgr2=0x7f25a41563d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:51.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.693+0000 7f25a92b7640 1 --2- 172.21.15.116:0/2430872110 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25a4071d90 0x7f25a41563d0 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f259c0097c0 tx=0x7f259c01c1f0 comp rx=0 tx=0).stop 2024-04-25T00:28:51.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.693+0000 7f25a92b7640 1 -- 172.21.15.116:0/2430872110 shutdown_connections 2024-04-25T00:28:51.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.693+0000 7f25a92b7640 1 --2- 172.21.15.116:0/2430872110 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25a4071d90 0x7f25a41563d0 unknown :-1 s=CLOSED pgs=1655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:51.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.693+0000 7f25a92b7640 1 -- 172.21.15.116:0/2430872110 >> 172.21.15.116:0/2430872110 conn(0x7f25a40c3b00 msgr2=0x7f25a40c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:28:51.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.693+0000 7f25a92b7640 1 -- 172.21.15.116:0/2430872110 shutdown_connections 2024-04-25T00:28:51.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.693+0000 7f25a92b7640 1 -- 172.21.15.116:0/2430872110 wait complete. 2024-04-25T00:28:51.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.693+0000 7f25a92b7640 1 Processor -- start 2024-04-25T00:28:51.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.693+0000 7f25a92b7640 1 -- start start 2024-04-25T00:28:51.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.693+0000 7f25a92b7640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25a4071d90 0x7f25a41414f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:51.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.693+0000 7f25a92b7640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f25a40a94a0 0x7f25a406ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:51.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.693+0000 7f25a92b7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25a414c720 0x7f25a40ab500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:51.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.693+0000 7f25a92b7640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f25a4152f40 con 0x7f25a414c720 2024-04-25T00:28:51.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.693+0000 7f25a92b7640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f25a4152c60 con 0x7f25a4071d90 2024-04-25T00:28:51.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.693+0000 7f25a92b7640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f25a40c3120 con 0x7f25a40a94a0 2024-04-25T00:28:51.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.693+0000 7f25a37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25a414c720 0x7f25a40ab500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:28:51.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.693+0000 7f25a37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25a414c720 0x7f25a40ab500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34906/0 (socket says 172.21.15.116:34906) 2024-04-25T00:28:51.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.693+0000 7f25a37fe640 1 -- 172.21.15.116:0/768451945 learned_addr learned my addr 172.21.15.116:0/768451945 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:28:51.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.693+0000 7f25a2ffd640 1 --2- 172.21.15.116:0/768451945 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25a4071d90 0x7f25a41414f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:28:51.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.693+0000 7f25a27fc640 1 --2- 172.21.15.116:0/768451945 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f25a40a94a0 0x7f25a406ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:28:51.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.693+0000 7f25a37fe640 1 -- 172.21.15.116:0/768451945 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f25a40a94a0 msgr2=0x7f25a406ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:51.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.693+0000 7f25a37fe640 1 --2- 172.21.15.116:0/768451945 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f25a40a94a0 0x7f25a406ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:51.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.693+0000 7f25a37fe640 1 -- 172.21.15.116:0/768451945 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25a4071d90 msgr2=0x7f25a41414f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:51.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.693+0000 7f25a37fe640 1 --2- 172.21.15.116:0/768451945 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25a4071d90 0x7f25a41414f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:51.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.693+0000 7f25a37fe640 1 -- 172.21.15.116:0/768451945 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25a4141a30 con 0x7f25a414c720 2024-04-25T00:28:51.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.693+0000 7f25a27fc640 1 --2- 172.21.15.116:0/768451945 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f25a40a94a0 0x7f25a406ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:28:51.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.693+0000 7f25a37fe640 1 --2- 172.21.15.116:0/768451945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25a414c720 0x7f25a40ab500 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f259c019e20 tx=0x7f259c01aa30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:28:51.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.693+0000 7f259b7fe640 1 -- 172.21.15.116:0/768451945 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f259c0b0050 con 0x7f25a414c720 2024-04-25T00:28:51.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.693+0000 7f259b7fe640 1 -- 172.21.15.116:0/768451945 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f259c0bc070 con 0x7f25a414c720 2024-04-25T00:28:51.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.693+0000 7f25a92b7640 1 -- 172.21.15.116:0/768451945 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25a4141e60 con 0x7f25a414c720 2024-04-25T00:28:51.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.693+0000 7f259b7fe640 1 -- 172.21.15.116:0/768451945 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f259c006e20 con 0x7f25a414c720 2024-04-25T00:28:51.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.693+0000 7f25a92b7640 1 -- 172.21.15.116:0/768451945 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25a41f0110 con 0x7f25a414c720 2024-04-25T00:28:51.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.697+0000 7f25a92b7640 1 -- 172.21.15.116:0/768451945 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25a4068350 con 0x7f25a414c720 2024-04-25T00:28:51.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.697+0000 7f259b7fe640 1 -- 172.21.15.116:0/768451945 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f259c0b5a30 con 0x7f25a414c720 2024-04-25T00:28:51.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.697+0000 7f259b7fe640 1 --2- 172.21.15.116:0/768451945 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f25840678c0 0x7f2584069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:51.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.697+0000 7f259b7fe640 1 -- 172.21.15.116:0/768451945 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 10450+0+0 (secure 0 0 0) 0x7f259c1222a0 con 0x7f25a414c720 2024-04-25T00:28:51.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.697+0000 7f25a2ffd640 1 --2- 172.21.15.116:0/768451945 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f25840678c0 0x7f2584069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:28:51.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.697+0000 7f25a2ffd640 1 --2- 172.21.15.116:0/768451945 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f25840678c0 0x7f2584069d80 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f259400b810 tx=0x7f2594008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:28:51.702 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.701+0000 7f259b7fe640 1 -- 172.21.15.116:0/768451945 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f25a4141a30 con 0x7f25a414c720 2024-04-25T00:28:51.884 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.881+0000 7f25a92b7640 1 -- 172.21.15.116:0/768451945 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f25a40c3640 con 0x7f25a414c720 2024-04-25T00:28:51.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.885+0000 7f259b7fe640 1 -- 172.21.15.116:0/768451945 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 229 v229) ==== 95+0+5676 (secure 0 0 0) 0x7f25a4152420 con 0x7f25a414c720 2024-04-25T00:28:51.885 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:28:51.885 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":229,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:27:40.040636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6641},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6641":{"gid":6641,"name":"d","rank":0,"incarnation":223,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/2915274073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2915274073},{"type":"v1","addr":"172.21.15.174:6837","nonce":2915274073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6641,"qdb_cluster":[6641]},"id":1},{"mdsmap":{"epoch":229,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:28:51.269732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8772":{"gid":8772,"name":"a","rank":0,"incarnation":228,"state":"up:reconnect","state_seq":18,"addr":"172.21.15.116:6835/3499878473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3499878473},{"type":"v1","addr":"172.21.15.116:6835","nonce":3499878473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:28:51.886 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 229 2024-04-25T00:28:51.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.885+0000 7f25a92b7640 1 -- 172.21.15.116:0/768451945 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f25840678c0 msgr2=0x7f2584069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:51.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.885+0000 7f25a92b7640 1 --2- 172.21.15.116:0/768451945 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f25840678c0 0x7f2584069d80 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f259400b810 tx=0x7f2594008040 comp rx=0 tx=0).stop 2024-04-25T00:28:51.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.885+0000 7f25a92b7640 1 -- 172.21.15.116:0/768451945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25a414c720 msgr2=0x7f25a40ab500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:51.887 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.885+0000 7f25a92b7640 1 --2- 172.21.15.116:0/768451945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25a414c720 0x7f25a40ab500 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f259c019e20 tx=0x7f259c01aa30 comp rx=0 tx=0).stop 2024-04-25T00:28:51.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.885+0000 7f25a92b7640 1 -- 172.21.15.116:0/768451945 shutdown_connections 2024-04-25T00:28:51.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.885+0000 7f25a92b7640 1 --2- 172.21.15.116:0/768451945 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f25840678c0 0x7f2584069d80 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:51.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.885+0000 7f25a92b7640 1 --2- 172.21.15.116:0/768451945 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f25a414c720 0x7f25a40ab500 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:51.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.885+0000 7f25a92b7640 1 --2- 172.21.15.116:0/768451945 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f25a40a94a0 0x7f25a406ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:51.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.885+0000 7f25a92b7640 1 --2- 172.21.15.116:0/768451945 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f25a4071d90 0x7f25a41414f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:51.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.885+0000 7f25a92b7640 1 -- 172.21.15.116:0/768451945 >> 172.21.15.116:0/768451945 conn(0x7f25a40c3b00 msgr2=0x7f25a40c20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:28:51.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.885+0000 7f25a92b7640 1 -- 172.21.15.116:0/768451945 shutdown_connections 2024-04-25T00:28:51.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:51.885+0000 7f25a92b7640 1 -- 172.21.15.116:0/768451945 wait complete. 2024-04-25T00:28:51.903 INFO:tasks.mds_thrash.fs.[b]:mds.c down, removed from mdsmap 2024-04-25T00:28:51.903 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-04-25T00:28:51.904 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:28:52.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.009+0000 7f8631221640 1 Processor -- start 2024-04-25T00:28:52.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.009+0000 7f8631221640 1 -- start start 2024-04-25T00:28:52.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.009+0000 7f8631221640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f862c14c720 0x7f862c1563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:52.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.009+0000 7f8631221640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f862c0a94a0 0x7f862c0c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:52.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.009+0000 7f8631221640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f862c071220 con 0x7f862c0c2f50 2024-04-25T00:28:52.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.009+0000 7f8631221640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f862c153460 con 0x7f862c14c720 2024-04-25T00:28:52.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.009+0000 7f8631221640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f862c152420 con 0x7f862c0a94a0 2024-04-25T00:28:52.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.009+0000 7f862b59e640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f862c0c2f50 0x7f862c071680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:49232/0 (socket says 172.21.15.116:49232) 2024-04-25T00:28:52.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.009+0000 7f862b59e640 1 -- 172.21.15.116:0/2760918015 learned_addr learned my addr 172.21.15.116:0/2760918015 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:28:52.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.009+0000 7f862ad9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f862c14c720 0x7f862c1563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:28:52.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.009+0000 7f862a59c640 1 --2- 172.21.15.116:0/2760918015 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f862c0a94a0 0x7f862c0c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:28:52.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.009+0000 7f862ad9d640 1 -- 172.21.15.116:0/2760918015 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f862c0a94a0 msgr2=0x7f862c0c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:28:52.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.009+0000 7f862ad9d640 1 --2- 172.21.15.116:0/2760918015 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f862c0a94a0 0x7f862c0c2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:52.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.009+0000 7f862ad9d640 1 -- 172.21.15.116:0/2760918015 >> v1:172.21.15.116:6789/0 conn(0x7f862c0c2f50 legacy=0x7f862c071680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:52.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.009+0000 7f862ad9d640 1 -- 172.21.15.116:0/2760918015 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f862c14cb00 con 0x7f862c14c720 2024-04-25T00:28:52.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.009+0000 7f862a59c640 1 --2- 172.21.15.116:0/2760918015 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f862c0a94a0 0x7f862c0c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:28:52.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.013+0000 7f862ad9d640 1 --2- 172.21.15.116:0/2760918015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f862c14c720 0x7f862c1563d0 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f86200097c0 tx=0x7f862001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10d1a553a433d78a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:28:52.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.013+0000 7f8629d9b640 1 -- 172.21.15.116:0/2760918015 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86200ab600 con 0x7f862c14c720 2024-04-25T00:28:52.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.013+0000 7f8629d9b640 1 -- 172.21.15.116:0/2760918015 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86200b0070 con 0x7f862c14c720 2024-04-25T00:28:52.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.013+0000 7f8629d9b640 1 -- 172.21.15.116:0/2760918015 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86200ab9c0 con 0x7f862c14c720 2024-04-25T00:28:52.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.013+0000 7f8631221640 1 -- 172.21.15.116:0/2760918015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f862c14c720 msgr2=0x7f862c1563d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:52.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.013+0000 7f8631221640 1 --2- 172.21.15.116:0/2760918015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f862c14c720 0x7f862c1563d0 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f86200097c0 tx=0x7f862001c3e0 comp rx=0 tx=0).stop 2024-04-25T00:28:52.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.013+0000 7f8631221640 1 -- 172.21.15.116:0/2760918015 shutdown_connections 2024-04-25T00:28:52.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.013+0000 7f8631221640 1 --2- 172.21.15.116:0/2760918015 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f862c0a94a0 0x7f862c0c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:52.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.013+0000 7f8631221640 1 --2- 172.21.15.116:0/2760918015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f862c14c720 0x7f862c1563d0 unknown :-1 s=CLOSED pgs=1658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:52.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.013+0000 7f8631221640 1 -- 172.21.15.116:0/2760918015 >> 172.21.15.116:0/2760918015 conn(0x7f862c0c3b00 msgr2=0x7f862c0c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:28:52.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.013+0000 7f8631221640 1 -- 172.21.15.116:0/2760918015 shutdown_connections 2024-04-25T00:28:52.015 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.013+0000 7f8631221640 1 -- 172.21.15.116:0/2760918015 wait complete. 2024-04-25T00:28:52.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.013+0000 7f8631221640 1 Processor -- start 2024-04-25T00:28:52.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.013+0000 7f8631221640 1 -- start start 2024-04-25T00:28:52.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.013+0000 7f8631221640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f862c0a94a0 0x7f862c1ec620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:52.017 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.013+0000 7f8631221640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f862c0c2f50 0x7f862c1454e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:52.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.013+0000 7f8631221640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f862c14c720 0x7f862c0acb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:52.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.013+0000 7f8631221640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f862c152f40 con 0x7f862c14c720 2024-04-25T00:28:52.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.013+0000 7f8631221640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f862c152c60 con 0x7f862c0c2f50 2024-04-25T00:28:52.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.013+0000 7f8631221640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f862c15f5c0 con 0x7f862c0a94a0 2024-04-25T00:28:52.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.013+0000 7f862b59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f862c14c720 0x7f862c0acb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:28:52.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.013+0000 7f862b59e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f862c14c720 0x7f862c0acb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:34924/0 (socket says 172.21.15.116:34924) 2024-04-25T00:28:52.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.013+0000 7f862b59e640 1 -- 172.21.15.116:0/3650017987 learned_addr learned my addr 172.21.15.116:0/3650017987 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:28:52.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.013+0000 7f862ad9d640 1 --2- 172.21.15.116:0/3650017987 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f862c0a94a0 0x7f862c1ec620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:28:52.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.013+0000 7f862b59e640 1 -- 172.21.15.116:0/3650017987 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f862c0a94a0 msgr2=0x7f862c1ec620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:52.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.013+0000 7f862a59c640 1 --2- 172.21.15.116:0/3650017987 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f862c0c2f50 0x7f862c1454e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:28:52.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.013+0000 7f862b59e640 1 --2- 172.21.15.116:0/3650017987 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f862c0a94a0 0x7f862c1ec620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:52.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.013+0000 7f862b59e640 1 -- 172.21.15.116:0/3650017987 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f862c0c2f50 msgr2=0x7f862c1454e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:52.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.013+0000 7f862b59e640 1 --2- 172.21.15.116:0/3650017987 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f862c0c2f50 0x7f862c1454e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:52.018 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.013+0000 7f862b59e640 1 -- 172.21.15.116:0/3650017987 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f862c140fe0 con 0x7f862c14c720 2024-04-25T00:28:52.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.013+0000 7f862ad9d640 1 --2- 172.21.15.116:0/3650017987 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f862c0a94a0 0x7f862c1ec620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:28:52.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.017+0000 7f862b59e640 1 --2- 172.21.15.116:0/3650017987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f862c14c720 0x7f862c0acb20 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f862400cbc0 tx=0x7f862400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:28:52.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.017+0000 7f86137fe640 1 -- 172.21.15.116:0/3650017987 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8624017070 con 0x7f862c14c720 2024-04-25T00:28:52.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.017+0000 7f86137fe640 1 -- 172.21.15.116:0/3650017987 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f862400ad40 con 0x7f862c14c720 2024-04-25T00:28:52.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.017+0000 7f8631221640 1 -- 172.21.15.116:0/3650017987 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f862c140a20 con 0x7f862c14c720 2024-04-25T00:28:52.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.017+0000 7f86137fe640 1 -- 172.21.15.116:0/3650017987 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8624012600 con 0x7f862c14c720 2024-04-25T00:28:52.019 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.017+0000 7f8631221640 1 -- 172.21.15.116:0/3650017987 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f862c154c30 con 0x7f862c14c720 2024-04-25T00:28:52.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.017+0000 7f8631221640 1 -- 172.21.15.116:0/3650017987 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f862c068350 con 0x7f862c14c720 2024-04-25T00:28:52.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.017+0000 7f86137fe640 1 -- 172.21.15.116:0/3650017987 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f862401a070 con 0x7f862c14c720 2024-04-25T00:28:52.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.017+0000 7f86137fe640 1 --2- 172.21.15.116:0/3650017987 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f86180678c0 0x7f8618069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:52.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.017+0000 7f86137fe640 1 -- 172.21.15.116:0/3650017987 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 10450+0+0 (secure 0 0 0) 0x7f8624086b30 con 0x7f862c14c720 2024-04-25T00:28:52.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.017+0000 7f862ad9d640 1 --2- 172.21.15.116:0/3650017987 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f86180678c0 0x7f8618069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:28:52.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.017+0000 7f862ad9d640 1 --2- 172.21.15.116:0/3650017987 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f86180678c0 0x7f8618069d80 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f86200af000 tx=0x7f8620008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:28:52.023 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.021+0000 7f86137fe640 1 -- 172.21.15.116:0/3650017987 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f8624055dd0 con 0x7f862c14c720 2024-04-25T00:28:52.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.205+0000 7f8631221640 1 -- 172.21.15.116:0/3650017987 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f862c0c3640 con 0x7f862c14c720 2024-04-25T00:28:52.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.205+0000 7f86137fe640 1 -- 172.21.15.116:0/3650017987 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 229 v229) ==== 95+0+5676 (secure 0 0 0) 0x7f862405a0d0 con 0x7f862c14c720 2024-04-25T00:28:52.206 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:28:52.206 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":229,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:27:40.040636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6641},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6641":{"gid":6641,"name":"d","rank":0,"incarnation":223,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/2915274073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2915274073},{"type":"v1","addr":"172.21.15.174:6837","nonce":2915274073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6641,"qdb_cluster":[6641]},"id":1},{"mdsmap":{"epoch":229,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:28:51.269732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8772":{"gid":8772,"name":"a","rank":0,"incarnation":228,"state":"up:reconnect","state_seq":18,"addr":"172.21.15.116:6835/3499878473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3499878473},{"type":"v1","addr":"172.21.15.116:6835","nonce":3499878473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:28:52.206 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 229 2024-04-25T00:28:52.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.205+0000 7f8631221640 1 -- 172.21.15.116:0/3650017987 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f86180678c0 msgr2=0x7f8618069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:52.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.205+0000 7f8631221640 1 --2- 172.21.15.116:0/3650017987 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f86180678c0 0x7f8618069d80 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f86200af000 tx=0x7f8620008040 comp rx=0 tx=0).stop 2024-04-25T00:28:52.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.205+0000 7f8631221640 1 -- 172.21.15.116:0/3650017987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f862c14c720 msgr2=0x7f862c0acb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:52.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.205+0000 7f8631221640 1 --2- 172.21.15.116:0/3650017987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f862c14c720 0x7f862c0acb20 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f862400cbc0 tx=0x7f862400a430 comp rx=0 tx=0).stop 2024-04-25T00:28:52.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.205+0000 7f8631221640 1 -- 172.21.15.116:0/3650017987 shutdown_connections 2024-04-25T00:28:52.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.205+0000 7f8631221640 1 --2- 172.21.15.116:0/3650017987 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f86180678c0 0x7f8618069d80 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:52.208 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.205+0000 7f8631221640 1 --2- 172.21.15.116:0/3650017987 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f862c14c720 0x7f862c0acb20 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:52.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.205+0000 7f8631221640 1 --2- 172.21.15.116:0/3650017987 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f862c0c2f50 0x7f862c1454e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:52.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.205+0000 7f8631221640 1 --2- 172.21.15.116:0/3650017987 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f862c0a94a0 0x7f862c1ec620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:52.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.205+0000 7f8631221640 1 -- 172.21.15.116:0/3650017987 >> 172.21.15.116:0/3650017987 conn(0x7f862c0c3b00 msgr2=0x7f862c149690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:28:52.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.205+0000 7f8631221640 1 -- 172.21.15.116:0/3650017987 shutdown_connections 2024-04-25T00:28:52.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:52.205+0000 7f8631221640 1 -- 172.21.15.116:0/3650017987 wait complete. 2024-04-25T00:28:52.224 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 229, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:04.678737+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 226, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_6641": { "addr": "172.21.15.174:6837/2915274073", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6836", "nonce": 2915274073, "type": "v2" }, { "addr": "172.21.15.174:6837", "nonce": 2915274073, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6641, "incarnation": 223, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 7 } }, "last_failure": 0, "last_failure_osd_epoch": 71, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T00:27:40.040636+0000", "qdb_cluster": [ 6641 ], "qdb_leader": 6641, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6641 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:18.825887+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 229, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_8772": { "addr": "172.21.15.116:6835/3499878473", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6834", "nonce": 3499878473, "type": "v2" }, { "addr": "172.21.15.116:6835", "nonce": 3499878473, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8772, "incarnation": 228, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:reconnect", "state_seq": 18 } }, "last_failure": 0, "last_failure_osd_epoch": 72, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-04-25T00:28:51.269732+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8772 } } } ], "standbys": [ { "addr": "172.21.15.174:6835/4112145782", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6834", "nonce": 4112145782, "type": "v2" }, { "addr": "172.21.15.174:6835", "nonce": 4112145782, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 112, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5582, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-25T00:28:54.225 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:28:54.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.400+0000 7f82fe081640 1 Processor -- start 2024-04-25T00:28:54.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.400+0000 7f82fe081640 1 -- start start 2024-04-25T00:28:54.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.400+0000 7f82fe081640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f82f8150ba0 0x7f82f80c5010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:54.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.400+0000 7f82fe081640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f82f8072820 0x7f82f8153770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:54.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.400+0000 7f82fe081640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82f80c2cb0 con 0x7f82f8072c60 2024-04-25T00:28:54.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.400+0000 7f82fe081640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f82f8157c30 con 0x7f82f8150ba0 2024-04-25T00:28:54.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.400+0000 7f82fe081640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f82f8157670 con 0x7f82f8072820 2024-04-25T00:28:54.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.400+0000 7f82f7fff640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f82f8072c60 0x7f82f8153cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:45004/0 (socket says 172.21.15.116:45004) 2024-04-25T00:28:54.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.400+0000 7f82f7fff640 1 -- 172.21.15.116:0/2587112091 learned_addr learned my addr 172.21.15.116:0/2587112091 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:28:54.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.400+0000 7f82f77fe640 1 --2- 172.21.15.116:0/2587112091 >> v2:172.21.15.116:3300/0 conn(0x7f82f8150ba0 0x7f82f80c5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:28:54.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.400+0000 7f82f67fc640 1 -- 172.21.15.116:0/2587112091 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3495075264 0 0) 0x7f82f80c2cb0 con 0x7f82f8072c60 2024-04-25T00:28:54.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.400+0000 7f82f67fc640 1 -- 172.21.15.116:0/2587112091 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f82e0003530 con 0x7f82f8072c60 2024-04-25T00:28:54.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.400+0000 7f82f77fe640 1 -- 172.21.15.116:0/2587112091 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f82f8072820 msgr2=0x7f82f8153770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:28:54.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.400+0000 7f82f77fe640 1 --2- 172.21.15.116:0/2587112091 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f82f8072820 0x7f82f8153770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:54.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.400+0000 7f82f77fe640 1 -- 172.21.15.116:0/2587112091 >> v1:172.21.15.174:6789/0 conn(0x7f82f8072c60 legacy=0x7f82f8153cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:54.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.400+0000 7f82f77fe640 1 -- 172.21.15.116:0/2587112091 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82f8157950 con 0x7f82f8150ba0 2024-04-25T00:28:54.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.404+0000 7f82f77fe640 1 --2- 172.21.15.116:0/2587112091 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82f8150ba0 0x7f82f80c5010 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f82e80097c0 tx=0x7f82e801c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=565a1d0512e7519e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:28:54.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.404+0000 7f82f67fc640 1 -- 172.21.15.116:0/2587112091 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e80b0070 con 0x7f82f8150ba0 2024-04-25T00:28:54.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.404+0000 7f82f67fc640 1 -- 172.21.15.116:0/2587112091 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82e80ab600 con 0x7f82f8150ba0 2024-04-25T00:28:54.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.404+0000 7f82f67fc640 1 -- 172.21.15.116:0/2587112091 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e80ab9d0 con 0x7f82f8150ba0 2024-04-25T00:28:54.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.408+0000 7f82fe081640 1 -- 172.21.15.116:0/2587112091 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82f8150ba0 msgr2=0x7f82f80c5010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:54.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.408+0000 7f82fe081640 1 --2- 172.21.15.116:0/2587112091 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82f8150ba0 0x7f82f80c5010 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f82e80097c0 tx=0x7f82e801c410 comp rx=0 tx=0).stop 2024-04-25T00:28:54.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.408+0000 7f82fe081640 1 -- 172.21.15.116:0/2587112091 shutdown_connections 2024-04-25T00:28:54.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.408+0000 7f82fe081640 1 --2- 172.21.15.116:0/2587112091 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f82f8072820 0x7f82f8153770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:54.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.408+0000 7f82fe081640 1 --2- 172.21.15.116:0/2587112091 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82f8150ba0 0x7f82f80c5010 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:54.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.408+0000 7f82fe081640 1 -- 172.21.15.116:0/2587112091 >> 172.21.15.116:0/2587112091 conn(0x7f82f806c720 msgr2=0x7f82f806cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:28:54.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.408+0000 7f82fe081640 1 -- 172.21.15.116:0/2587112091 shutdown_connections 2024-04-25T00:28:54.412 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.408+0000 7f82fe081640 1 -- 172.21.15.116:0/2587112091 wait complete. 2024-04-25T00:28:54.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.408+0000 7f82fe081640 1 Processor -- start 2024-04-25T00:28:54.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.408+0000 7f82fe081640 1 -- start start 2024-04-25T00:28:54.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.412+0000 7f82fe081640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82f8072820 0x7f82f814ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:54.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.412+0000 7f82fe081640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82f8072c60 0x7f82f806f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:54.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.412+0000 7f82fe081640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f82f81554e0 0x7f82f80c6d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:54.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.412+0000 7f82fe081640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f82f80ab440 con 0x7f82f8072c60 2024-04-25T00:28:54.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.412+0000 7f82fe081640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f82f80ab2c0 con 0x7f82f8072820 2024-04-25T00:28:54.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.412+0000 7f82fe081640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f82f80ab5c0 con 0x7f82f81554e0 2024-04-25T00:28:54.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.412+0000 7f82f77fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82f8072820 0x7f82f814ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:28:54.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.412+0000 7f82f77fe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82f8072820 0x7f82f814ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:54380/0 (socket says 172.21.15.116:54380) 2024-04-25T00:28:54.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.412+0000 7f82f77fe640 1 -- 172.21.15.116:0/1638664409 learned_addr learned my addr 172.21.15.116:0/1638664409 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:28:54.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.412+0000 7f82f77fe640 1 -- 172.21.15.116:0/1638664409 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f82f81554e0 msgr2=0x7f82f80c6d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:54.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.412+0000 7f82f77fe640 1 --2- 172.21.15.116:0/1638664409 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f82f81554e0 0x7f82f80c6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:54.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.412+0000 7f82f77fe640 1 -- 172.21.15.116:0/1638664409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82f8072c60 msgr2=0x7f82f806f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:54.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.412+0000 7f82f77fe640 1 --2- 172.21.15.116:0/1638664409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82f8072c60 0x7f82f806f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:54.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.412+0000 7f82f77fe640 1 -- 172.21.15.116:0/1638664409 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82f8156620 con 0x7f82f8072820 2024-04-25T00:28:54.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.412+0000 7f82f77fe640 1 --2- 172.21.15.116:0/1638664409 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82f8072820 0x7f82f814ae50 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f82e801a040 tx=0x7f82e80191b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:28:54.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.412+0000 7f82dffff640 1 -- 172.21.15.116:0/1638664409 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e80b9070 con 0x7f82f8072820 2024-04-25T00:28:54.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.412+0000 7f82dffff640 1 -- 172.21.15.116:0/1638664409 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82e80b5480 con 0x7f82f8072820 2024-04-25T00:28:54.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.412+0000 7f82dffff640 1 -- 172.21.15.116:0/1638664409 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e80b0040 con 0x7f82f8072820 2024-04-25T00:28:54.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.416+0000 7f82fe081640 1 -- 172.21.15.116:0/1638664409 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82f8156340 con 0x7f82f8072820 2024-04-25T00:28:54.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.416+0000 7f82fe081640 1 -- 172.21.15.116:0/1638664409 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82f8153500 con 0x7f82f8072820 2024-04-25T00:28:54.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.420+0000 7f82ddffb640 1 -- 172.21.15.116:0/1638664409 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82d00052a0 con 0x7f82f8072820 2024-04-25T00:28:54.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.420+0000 7f82dffff640 1 -- 172.21.15.116:0/1638664409 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f82e80bc070 con 0x7f82f8072820 2024-04-25T00:28:54.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.420+0000 7f82dffff640 1 --2- 172.21.15.116:0/1638664409 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f82e40678f0 0x7f82e4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:54.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.420+0000 7f82f6ffd640 1 --2- 172.21.15.116:0/1638664409 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f82e40678f0 0x7f82e4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:28:54.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.420+0000 7f82f6ffd640 1 --2- 172.21.15.116:0/1638664409 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f82e40678f0 0x7f82e4069db0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f82f8150480 tx=0x7f82ec006ca0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:28:54.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.424+0000 7f82dffff640 1 -- 172.21.15.116:0/1638664409 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 10450+0+0 (secure 0 0 0) 0x7f82e80ee9f0 con 0x7f82f8072820 2024-04-25T00:28:54.431 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.428+0000 7f82dffff640 1 -- 172.21.15.116:0/1638664409 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f82e80c3030 con 0x7f82f8072820 2024-04-25T00:28:54.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.752+0000 7f82ddffb640 1 -- 172.21.15.116:0/1638664409 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f82d0005590 con 0x7f82f8072820 2024-04-25T00:28:54.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.752+0000 7f82dffff640 1 -- 172.21.15.116:0/1638664409 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 231 v231) ==== 95+0+5680 (secure 0 0 0) 0x7f82e8016090 con 0x7f82f8072820 2024-04-25T00:28:54.755 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:28:54.755 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":231,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:27:40.040636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6641},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6641":{"gid":6641,"name":"d","rank":0,"incarnation":223,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/2915274073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2915274073},{"type":"v1","addr":"172.21.15.174:6837","nonce":2915274073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6641,"qdb_cluster":[6641]},"id":1},{"mdsmap":{"epoch":231,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:28:53.808539+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8772":{"gid":8772,"name":"a","rank":0,"incarnation":228,"state":"up:active","state_seq":20,"addr":"172.21.15.116:6835/3499878473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3499878473},{"type":"v1","addr":"172.21.15.116:6835","nonce":3499878473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8772,"qdb_cluster":[8772]},"id":2}]} 2024-04-25T00:28:54.755 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 231 2024-04-25T00:28:54.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.756+0000 7f82ddffb640 1 -- 172.21.15.116:0/1638664409 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f82e40678f0 msgr2=0x7f82e4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:54.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.756+0000 7f82ddffb640 1 --2- 172.21.15.116:0/1638664409 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f82e40678f0 0x7f82e4069db0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f82f8150480 tx=0x7f82ec006ca0 comp rx=0 tx=0).stop 2024-04-25T00:28:54.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.756+0000 7f82ddffb640 1 -- 172.21.15.116:0/1638664409 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82f8072820 msgr2=0x7f82f814ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:54.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.756+0000 7f82ddffb640 1 --2- 172.21.15.116:0/1638664409 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82f8072820 0x7f82f814ae50 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f82e801a040 tx=0x7f82e80191b0 comp rx=0 tx=0).stop 2024-04-25T00:28:54.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.756+0000 7f82ddffb640 1 -- 172.21.15.116:0/1638664409 shutdown_connections 2024-04-25T00:28:54.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.756+0000 7f82ddffb640 1 --2- 172.21.15.116:0/1638664409 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f82e40678f0 0x7f82e4069db0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:54.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.756+0000 7f82ddffb640 1 --2- 172.21.15.116:0/1638664409 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f82f81554e0 0x7f82f80c6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:54.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.756+0000 7f82ddffb640 1 --2- 172.21.15.116:0/1638664409 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f82f8072c60 0x7f82f806f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:54.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.756+0000 7f82ddffb640 1 --2- 172.21.15.116:0/1638664409 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f82f8072820 0x7f82f814ae50 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:54.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.756+0000 7f82ddffb640 1 -- 172.21.15.116:0/1638664409 >> 172.21.15.116:0/1638664409 conn(0x7f82f806c720 msgr2=0x7f82f814e3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:28:54.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.756+0000 7f82ddffb640 1 -- 172.21.15.116:0/1638664409 shutdown_connections 2024-04-25T00:28:54.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:54.756+0000 7f82ddffb640 1 -- 172.21.15.116:0/1638664409 wait complete. 2024-04-25T00:28:54.786 INFO:tasks.mds_thrash.fs.[b]:mds.a has gained rank=0, replacing gid=8652 2024-04-25T00:28:54.786 INFO:tasks.mds_thrash.fs.[b]:waiting for 3 secs before reviving mds.c 2024-04-25T00:28:56.085 INFO:tasks.ceph.mds.a.smithi116.stderr:2024-04-25T00:28:56.084+0000 7f3c48131640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T00:28:57.786 INFO:tasks.mds_thrash.fs.[b]:reviving mds.c 2024-04-25T00:28:57.787 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-25T00:28:57.787 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-25T00:28:57.789 INFO:tasks.ceph.mds.c:Started 2024-04-25T00:28:57.789 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:28:57.885 INFO:tasks.ceph.mds.c.smithi116.stdout:starting mds.c at 2024-04-25T00:28:57.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.948+0000 7f520e6a6640 1 Processor -- start 2024-04-25T00:28:57.950 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.948+0000 7f520e6a6640 1 -- start start 2024-04-25T00:28:57.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.948+0000 7f520e6a6640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f52080c2a10 0x7f52080c2e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:57.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.948+0000 7f520e6a6640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5208071a30 con 0x7f5208153f40 2024-04-25T00:28:57.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.948+0000 7f520e6a6640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5208071ca0 con 0x7f5208159f00 2024-04-25T00:28:57.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.948+0000 7f520e6a6640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f52080725d0 con 0x7f52080c2a10 2024-04-25T00:28:57.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.948+0000 7f5207fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f5208153f40 0x7f520806e790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:48182/0 (socket says 172.21.15.116:48182) 2024-04-25T00:28:57.952 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.948+0000 7f5207fff640 1 -- 172.21.15.116:0/3063960605 learned_addr learned my addr 172.21.15.116:0/3063960605 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:28:57.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.952+0000 7f5206ffd640 1 -- 172.21.15.116:0/3063960605 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2096004910 0 0) 0x7f5208071a30 con 0x7f5208153f40 2024-04-25T00:28:57.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.952+0000 7f5206ffd640 1 -- 172.21.15.116:0/3063960605 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f51f0003530 con 0x7f5208153f40 2024-04-25T00:28:57.954 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.952+0000 7f52077fe640 1 --2- 172.21.15.116:0/3063960605 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f52080c2a10 0x7f52080c2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:28:57.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.952+0000 7f5206ffd640 1 -- 172.21.15.116:0/3063960605 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1132258425 0 0) 0x7f5208071ca0 con 0x7f5208159f00 2024-04-25T00:28:57.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.952+0000 7f5206ffd640 1 -- 172.21.15.116:0/3063960605 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5208071a30 con 0x7f5208159f00 2024-04-25T00:28:57.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.952+0000 7f5206ffd640 1 -- 172.21.15.116:0/3063960605 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2734792859 0 0) 0x7f51f0003530 con 0x7f5208153f40 2024-04-25T00:28:57.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.952+0000 7f5206ffd640 1 -- 172.21.15.116:0/3063960605 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f52080c2a10 msgr2=0x7f52080c2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:28:57.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.952+0000 7f5206ffd640 1 --2- 172.21.15.116:0/3063960605 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f52080c2a10 0x7f52080c2e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:57.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.952+0000 7f5206ffd640 1 -- 172.21.15.116:0/3063960605 >> v1:172.21.15.174:6789/0 conn(0x7f5208159f00 legacy=0x7f520815a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:57.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.952+0000 7f5206ffd640 1 -- 172.21.15.116:0/3063960605 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5208071f80 con 0x7f5208153f40 2024-04-25T00:28:57.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.952+0000 7f5206ffd640 1 -- 172.21.15.116:0/3063960605 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f51fc001680 con 0x7f5208153f40 2024-04-25T00:28:57.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.952+0000 7f5206ffd640 1 -- 172.21.15.116:0/3063960605 >> v1:172.21.15.116:6789/0 conn(0x7f5208153f40 legacy=0x7f520806e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:57.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.952+0000 7f5206ffd640 1 --2- 172.21.15.116:0/3063960605 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f51f0003b80 0x7f51f0003f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:57.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.952+0000 7f5206ffd640 1 --2- 172.21.15.116:0/3063960605 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f51f0004810 0x7f51f0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:57.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.952+0000 7f5206ffd640 1 --2- 172.21.15.116:0/3063960605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51f00051d0 0x7f51f000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:57.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.952+0000 7f5206ffd640 1 -- 172.21.15.116:0/3063960605 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f52080725d0 con 0x7f51f00051d0 2024-04-25T00:28:57.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.952+0000 7f5206ffd640 1 -- 172.21.15.116:0/3063960605 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f51f000c020 con 0x7f51f0004810 2024-04-25T00:28:57.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.952+0000 7f5206ffd640 1 -- 172.21.15.116:0/3063960605 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f51f000c220 con 0x7f51f0003b80 2024-04-25T00:28:57.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.952+0000 7f520cc43640 1 --2- 172.21.15.116:0/3063960605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51f00051d0 0x7f51f000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:28:57.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.952+0000 7f5207fff640 1 --2- 172.21.15.116:0/3063960605 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f51f0003b80 0x7f51f0003f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:28:57.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.952+0000 7f520cc43640 1 -- 172.21.15.116:0/3063960605 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f51f0003b80 msgr2=0x7f51f0003f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:28:57.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.952+0000 7f520cc43640 1 --2- 172.21.15.116:0/3063960605 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f51f0003b80 0x7f51f0003f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:57.957 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.952+0000 7f520cc43640 1 -- 172.21.15.116:0/3063960605 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f51f0004810 msgr2=0x7f51f0004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:28:57.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.952+0000 7f520cc43640 1 --2- 172.21.15.116:0/3063960605 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f51f0004810 0x7f51f0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:57.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.952+0000 7f520cc43640 1 -- 172.21.15.116:0/3063960605 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51f0003530 con 0x7f51f00051d0 2024-04-25T00:28:57.961 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.960+0000 7f520cc43640 1 --2- 172.21.15.116:0/3063960605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51f00051d0 0x7f51f000b9d0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f5200002050 tx=0x7f520001f0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32c2cf585c6af073 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:28:57.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.960+0000 7f5206ffd640 1 -- 172.21.15.116:0/3063960605 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52000b5070 con 0x7f51f00051d0 2024-04-25T00:28:57.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.960+0000 7f5206ffd640 1 -- 172.21.15.116:0/3063960605 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52000a8cf0 con 0x7f51f00051d0 2024-04-25T00:28:57.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.960+0000 7f5206ffd640 1 -- 172.21.15.116:0/3063960605 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5200001c50 con 0x7f51f00051d0 2024-04-25T00:28:57.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.960+0000 7f520e6a6640 1 -- 172.21.15.116:0/3063960605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51f00051d0 msgr2=0x7f51f000b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:57.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.960+0000 7f520e6a6640 1 --2- 172.21.15.116:0/3063960605 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f51f00051d0 0x7f51f000b9d0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f5200002050 tx=0x7f520001f0d0 comp rx=0 tx=0).stop 2024-04-25T00:28:57.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.960+0000 7f520cc43640 1 -- 172.21.15.116:0/3063960605 reap_dead start 2024-04-25T00:28:57.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.960+0000 7f520e6a6640 1 -- 172.21.15.116:0/3063960605 shutdown_connections 2024-04-25T00:28:57.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.960+0000 7f520e6a6640 1 -- 172.21.15.116:0/3063960605 >> 172.21.15.116:0/3063960605 conn(0x7f52080c3b00 msgr2=0x7f52080c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:28:57.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.960+0000 7f520e6a6640 1 -- 172.21.15.116:0/3063960605 shutdown_connections 2024-04-25T00:28:57.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.960+0000 7f520e6a6640 1 -- 172.21.15.116:0/3063960605 wait complete. 2024-04-25T00:28:57.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.964+0000 7f520e6a6640 1 Processor -- start 2024-04-25T00:28:57.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.964+0000 7f520e6a6640 1 -- start start 2024-04-25T00:28:57.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.964+0000 7f520e6a6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f520806cdb0 0x7f52081458b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:57.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.964+0000 7f520e6a6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52081597c0 0x7f5208145df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:57.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.964+0000 7f520e6a6640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f52081f3190 0x7f52081f3640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:57.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.964+0000 7f520e6a6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f520815aaa0 con 0x7f52081597c0 2024-04-25T00:28:57.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.964+0000 7f520e6a6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f520806eea0 con 0x7f520806cdb0 2024-04-25T00:28:57.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.964+0000 7f520e6a6640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f520815dee0 con 0x7f52081f3190 2024-04-25T00:28:57.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.964+0000 7f52077fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52081597c0 0x7f5208145df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:28:57.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.964+0000 7f52077fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52081597c0 0x7f5208145df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:54956/0 (socket says 172.21.15.116:54956) 2024-04-25T00:28:57.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.964+0000 7f52077fe640 1 -- 172.21.15.116:0/3484728795 learned_addr learned my addr 172.21.15.116:0/3484728795 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:28:57.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.964+0000 7f5207fff640 1 --2- 172.21.15.116:0/3484728795 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f520806cdb0 0x7f52081458b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:28:57.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.964+0000 7f520cc43640 1 --2- 172.21.15.116:0/3484728795 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f52081f3190 0x7f52081f3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:28:57.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.964+0000 7f52077fe640 1 -- 172.21.15.116:0/3484728795 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f52081f3190 msgr2=0x7f52081f3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:57.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.964+0000 7f52077fe640 1 --2- 172.21.15.116:0/3484728795 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f52081f3190 0x7f52081f3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:57.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.964+0000 7f52077fe640 1 -- 172.21.15.116:0/3484728795 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f520806cdb0 msgr2=0x7f52081458b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:57.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.964+0000 7f52077fe640 1 --2- 172.21.15.116:0/3484728795 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f520806cdb0 0x7f52081458b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:57.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.964+0000 7f52077fe640 1 -- 172.21.15.116:0/3484728795 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f520815d120 con 0x7f52081597c0 2024-04-25T00:28:57.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.964+0000 7f52077fe640 1 --2- 172.21.15.116:0/3484728795 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52081597c0 0x7f5208145df0 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f51f800e7a0 tx=0x7f51f800ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:28:57.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.964+0000 7f5204ff9640 1 -- 172.21.15.116:0/3484728795 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f8017070 con 0x7f52081597c0 2024-04-25T00:28:57.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.964+0000 7f5204ff9640 1 -- 172.21.15.116:0/3484728795 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51f800ee50 con 0x7f52081597c0 2024-04-25T00:28:57.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.964+0000 7f520e6a6640 1 -- 172.21.15.116:0/3484728795 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f520815d400 con 0x7f52081597c0 2024-04-25T00:28:57.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.964+0000 7f5204ff9640 1 -- 172.21.15.116:0/3484728795 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f800cae0 con 0x7f52081597c0 2024-04-25T00:28:57.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.964+0000 7f520e6a6640 1 -- 172.21.15.116:0/3484728795 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f520815d9c0 con 0x7f52081597c0 2024-04-25T00:28:57.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.964+0000 7f520e6a6640 1 -- 172.21.15.116:0/3484728795 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5208068350 con 0x7f52081597c0 2024-04-25T00:28:57.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.968+0000 7f5204ff9640 1 -- 172.21.15.116:0/3484728795 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f51f801a070 con 0x7f52081597c0 2024-04-25T00:28:57.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.968+0000 7f5204ff9640 1 --2- 172.21.15.116:0/3484728795 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f51f40678c0 0x7f51f4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:28:57.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.968+0000 7f5207fff640 1 --2- 172.21.15.116:0/3484728795 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f51f40678c0 0x7f51f4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:28:57.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.968+0000 7f5204ff9640 1 -- 172.21.15.116:0/3484728795 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 10450+0+0 (secure 0 0 0) 0x7f51f8054800 con 0x7f52081597c0 2024-04-25T00:28:57.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.968+0000 7f5207fff640 1 --2- 172.21.15.116:0/3484728795 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f51f40678c0 0x7f51f4069d80 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f51fc005aa0 tx=0x7f51fc007040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:28:57.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:57.976+0000 7f5204ff9640 1 -- 172.21.15.116:0/3484728795 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f51f8056760 con 0x7f52081597c0 2024-04-25T00:28:58.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:58.264+0000 7f520e6a6640 1 -- 172.21.15.116:0/3484728795 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f52080c16e0 con 0x7f52081597c0 2024-04-25T00:28:58.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:58.264+0000 7f5204ff9640 1 -- 172.21.15.116:0/3484728795 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 232 v232) ==== 95+0+6509 (secure 0 0 0) 0x7f51f805aa60 con 0x7f52081597c0 2024-04-25T00:28:58.268 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:28:58.268 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":232,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":8847,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/2662028892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2662028892},{"type":"v1","addr":"172.21.15.116:6833","nonce":2662028892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":232}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:27:40.040636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6641},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6641":{"gid":6641,"name":"d","rank":0,"incarnation":223,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/2915274073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2915274073},{"type":"v1","addr":"172.21.15.174:6837","nonce":2915274073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6641,"qdb_cluster":[6641]},"id":1},{"mdsmap":{"epoch":231,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:28:53.808539+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8772":{"gid":8772,"name":"a","rank":0,"incarnation":228,"state":"up:active","state_seq":20,"addr":"172.21.15.116:6835/3499878473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3499878473},{"type":"v1","addr":"172.21.15.116:6835","nonce":3499878473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8772,"qdb_cluster":[8772]},"id":2}]} 2024-04-25T00:28:58.268 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 232 2024-04-25T00:28:58.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:58.268+0000 7f520e6a6640 1 -- 172.21.15.116:0/3484728795 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f51f40678c0 msgr2=0x7f51f4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:58.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:58.268+0000 7f520e6a6640 1 --2- 172.21.15.116:0/3484728795 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f51f40678c0 0x7f51f4069d80 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f51fc005aa0 tx=0x7f51fc007040 comp rx=0 tx=0).stop 2024-04-25T00:28:58.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:58.268+0000 7f520e6a6640 1 -- 172.21.15.116:0/3484728795 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52081597c0 msgr2=0x7f5208145df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:28:58.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:58.268+0000 7f520e6a6640 1 --2- 172.21.15.116:0/3484728795 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52081597c0 0x7f5208145df0 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f51f800e7a0 tx=0x7f51f800ec60 comp rx=0 tx=0).stop 2024-04-25T00:28:58.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:58.268+0000 7f520e6a6640 1 -- 172.21.15.116:0/3484728795 shutdown_connections 2024-04-25T00:28:58.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:58.268+0000 7f520e6a6640 1 --2- 172.21.15.116:0/3484728795 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f51f40678c0 0x7f51f4069d80 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:58.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:58.268+0000 7f520e6a6640 1 --2- 172.21.15.116:0/3484728795 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f52081f3190 0x7f52081f3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:58.271 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:58.268+0000 7f520e6a6640 1 --2- 172.21.15.116:0/3484728795 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f52081597c0 0x7f5208145df0 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:58.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:58.268+0000 7f520e6a6640 1 --2- 172.21.15.116:0/3484728795 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f520806cdb0 0x7f52081458b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:28:58.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:58.268+0000 7f520e6a6640 1 -- 172.21.15.116:0/3484728795 >> 172.21.15.116:0/3484728795 conn(0x7f52080c3b00 msgr2=0x7f5208153570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:28:58.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:58.268+0000 7f520e6a6640 1 -- 172.21.15.116:0/3484728795 shutdown_connections 2024-04-25T00:28:58.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:28:58.268+0000 7f520e6a6640 1 -- 172.21.15.116:0/3484728795 wait complete. 2024-04-25T00:28:58.294 INFO:tasks.mds_thrash.fs.[b]:mds.c reported in up:standby state 2024-04-25T00:28:58.294 INFO:tasks.mds_thrash.fs.[b]:waiting for 60 secs before thrashing 2024-04-25T00:29:12.241 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:29:12.244 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:29:36.516 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:29:36.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.652+0000 7f21aaa05640 1 Processor -- start 2024-04-25T00:29:36.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.652+0000 7f21aaa05640 1 -- start start 2024-04-25T00:29:36.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.656+0000 7f21aaa05640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f21a4152d80 0x7f21a406bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:36.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.656+0000 7f21aaa05640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f21a406d5f0 0x7f21a4156c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:36.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.656+0000 7f21aaa05640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f21a406c290 0x7f21a41571c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:36.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.656+0000 7f21aaa05640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f21a40c9d60 con 0x7f21a406c290 2024-04-25T00:29:36.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.656+0000 7f21aaa05640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f21a40c9150 con 0x7f21a406d5f0 2024-04-25T00:29:36.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.656+0000 7f21aaa05640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f21a40c9430 con 0x7f21a4152d80 2024-04-25T00:29:36.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.656+0000 7f21a8fa2640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f21a406c290 0x7f21a41571c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:36.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.656+0000 7f21a37fe640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f21a406d5f0 0x7f21a4156c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:36.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.656+0000 7f21a8fa2640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f21a406c290 0x7f21a41571c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37478/0 (socket says 172.21.15.116:37478) 2024-04-25T00:29:36.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.656+0000 7f21a8fa2640 1 -- 172.21.15.116:0/3058564841 learned_addr learned my addr 172.21.15.116:0/3058564841 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:29:36.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.656+0000 7f21a37fe640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f21a406d5f0 0x7f21a4156c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:58778/0 (socket says 172.21.15.116:58778) 2024-04-25T00:29:36.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.656+0000 7f21a8fa2640 1 -- 172.21.15.116:0/3058564841 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f21a4152d80 msgr2=0x7f21a406bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:29:36.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.656+0000 7f21a8fa2640 1 --2- 172.21.15.116:0/3058564841 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f21a4152d80 0x7f21a406bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:36.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.656+0000 7f21a8fa2640 1 -- 172.21.15.116:0/3058564841 >> v2:172.21.15.174:3300/0 conn(0x7f21a406d5f0 msgr2=0x7f21a4156c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:29:36.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.656+0000 7f21a8fa2640 1 --2- 172.21.15.116:0/3058564841 >> v2:172.21.15.174:3300/0 conn(0x7f21a406d5f0 0x7f21a4156c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:36.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.656+0000 7f21a8fa2640 1 -- 172.21.15.116:0/3058564841 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21a40c9710 con 0x7f21a406c290 2024-04-25T00:29:36.660 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.656+0000 7f21a37fe640 1 --2- 172.21.15.116:0/3058564841 >> v2:172.21.15.174:3300/0 conn(0x7f21a406d5f0 0x7f21a4156c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:29:36.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.668+0000 7f21a8fa2640 1 --2- 172.21.15.116:0/3058564841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21a406c290 0x7f21a41571c0 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f21a40c8c30 tx=0x7f219c01c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67cdb5b4457419c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:29:36.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.668+0000 7f21a2ffd640 1 -- 172.21.15.116:0/3058564841 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f219c0a56d0 con 0x7f21a406c290 2024-04-25T00:29:36.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.668+0000 7f21a2ffd640 1 -- 172.21.15.116:0/3058564841 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f219c0a5870 con 0x7f21a406c290 2024-04-25T00:29:36.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.668+0000 7f21a2ffd640 1 -- 172.21.15.116:0/3058564841 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f219c0a5c40 con 0x7f21a406c290 2024-04-25T00:29:36.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.668+0000 7f21aaa05640 1 -- 172.21.15.116:0/3058564841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21a406c290 msgr2=0x7f21a41571c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:36.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.668+0000 7f21aaa05640 1 --2- 172.21.15.116:0/3058564841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21a406c290 0x7f21a41571c0 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f21a40c8c30 tx=0x7f219c01c930 comp rx=0 tx=0).stop 2024-04-25T00:29:36.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.668+0000 7f21aaa05640 1 -- 172.21.15.116:0/3058564841 shutdown_connections 2024-04-25T00:29:36.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.668+0000 7f21aaa05640 1 --2- 172.21.15.116:0/3058564841 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21a406c290 0x7f21a41571c0 unknown :-1 s=CLOSED pgs=1667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:36.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.668+0000 7f21aaa05640 1 --2- 172.21.15.116:0/3058564841 >> v2:172.21.15.174:3300/0 conn(0x7f21a406d5f0 0x7f21a4156c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:36.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.668+0000 7f21aaa05640 1 --2- 172.21.15.116:0/3058564841 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f21a4152d80 0x7f21a406bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:36.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.668+0000 7f21aaa05640 1 -- 172.21.15.116:0/3058564841 >> 172.21.15.116:0/3058564841 conn(0x7f21a40acdd0 msgr2=0x7f21a40ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:29:36.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.672+0000 7f21aaa05640 1 -- 172.21.15.116:0/3058564841 shutdown_connections 2024-04-25T00:29:36.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.672+0000 7f21aaa05640 1 -- 172.21.15.116:0/3058564841 wait complete. 2024-04-25T00:29:36.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.672+0000 7f21aaa05640 1 Processor -- start 2024-04-25T00:29:36.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.672+0000 7f21aaa05640 1 -- start start 2024-04-25T00:29:36.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.672+0000 7f21aaa05640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f21a406c290 0x7f21a41f69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:36.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.672+0000 7f21aaa05640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21a406d5f0 0x7f21a40c6d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:36.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.672+0000 7f21aaa05640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f21a4152d80 0x7f21a41f32e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:36.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.672+0000 7f21aaa05640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f21a4159710 con 0x7f21a406d5f0 2024-04-25T00:29:36.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.672+0000 7f21aaa05640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f21a4159590 con 0x7f21a406c290 2024-04-25T00:29:36.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.672+0000 7f21aaa05640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f21a4159890 con 0x7f21a4152d80 2024-04-25T00:29:36.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.672+0000 7f21a37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21a406d5f0 0x7f21a40c6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:36.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.672+0000 7f21a37fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21a406d5f0 0x7f21a40c6d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37490/0 (socket says 172.21.15.116:37490) 2024-04-25T00:29:36.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.672+0000 7f21a37fe640 1 -- 172.21.15.116:0/2251680466 learned_addr learned my addr 172.21.15.116:0/2251680466 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:29:36.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.672+0000 7f21a3fff640 1 --2- 172.21.15.116:0/2251680466 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f21a406c290 0x7f21a41f69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:36.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.672+0000 7f21a8fa2640 1 --2- 172.21.15.116:0/2251680466 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f21a4152d80 0x7f21a41f32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:36.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.672+0000 7f21a37fe640 1 -- 172.21.15.116:0/2251680466 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f21a4152d80 msgr2=0x7f21a41f32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:36.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.672+0000 7f21a37fe640 1 --2- 172.21.15.116:0/2251680466 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f21a4152d80 0x7f21a41f32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:36.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.672+0000 7f21a37fe640 1 -- 172.21.15.116:0/2251680466 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f21a406c290 msgr2=0x7f21a41f69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:36.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.672+0000 7f21a37fe640 1 --2- 172.21.15.116:0/2251680466 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f21a406c290 0x7f21a41f69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:36.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.672+0000 7f21a37fe640 1 -- 172.21.15.116:0/2251680466 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21a41f6ee0 con 0x7f21a406d5f0 2024-04-25T00:29:36.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.676+0000 7f21a37fe640 1 --2- 172.21.15.116:0/2251680466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21a406d5f0 0x7f21a40c6d10 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f219400e9d0 tx=0x7f219400ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:29:36.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.676+0000 7f21a0ff9640 1 -- 172.21.15.116:0/2251680466 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2194017070 con 0x7f21a406d5f0 2024-04-25T00:29:36.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.676+0000 7f21a0ff9640 1 -- 172.21.15.116:0/2251680466 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f219400ca60 con 0x7f21a406d5f0 2024-04-25T00:29:36.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.676+0000 7f21aaa05640 1 -- 172.21.15.116:0/2251680466 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21a4151ee0 con 0x7f21a406d5f0 2024-04-25T00:29:36.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.676+0000 7f21aaa05640 1 -- 172.21.15.116:0/2251680466 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21a41524a0 con 0x7f21a406d5f0 2024-04-25T00:29:36.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.676+0000 7f21a0ff9640 1 -- 172.21.15.116:0/2251680466 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f219400ce50 con 0x7f21a406d5f0 2024-04-25T00:29:36.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.676+0000 7f218a7fc640 1 -- 172.21.15.116:0/2251680466 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f217c0052a0 con 0x7f21a406d5f0 2024-04-25T00:29:36.679 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.676+0000 7f21a0ff9640 1 -- 172.21.15.116:0/2251680466 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f219401a070 con 0x7f21a406d5f0 2024-04-25T00:29:36.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.676+0000 7f21a0ff9640 1 --2- 172.21.15.116:0/2251680466 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2190067a60 0x7f2190069f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:36.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.676+0000 7f21a0ff9640 1 -- 172.21.15.116:0/2251680466 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 10450+0+0 (secure 0 0 0) 0x7f219408a060 con 0x7f21a406d5f0 2024-04-25T00:29:36.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.676+0000 7f21a3fff640 1 --2- 172.21.15.116:0/2251680466 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2190067a60 0x7f2190069f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:36.680 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.676+0000 7f21a3fff640 1 --2- 172.21.15.116:0/2251680466 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2190067a60 0x7f2190069f20 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f21980098f0 tx=0x7f2198006f90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:29:36.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.684+0000 7f21a0ff9640 1 -- 172.21.15.116:0/2251680466 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f2194055950 con 0x7f21a406d5f0 2024-04-25T00:29:36.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.964+0000 7f218a7fc640 1 -- 172.21.15.116:0/2251680466 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f217c005590 con 0x7f21a406d5f0 2024-04-25T00:29:36.972 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.968+0000 7f21a0ff9640 1 -- 172.21.15.116:0/2251680466 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 232 v232) ==== 95+0+6509 (secure 0 0 0) 0x7f2194059c50 con 0x7f21a406d5f0 2024-04-25T00:29:36.974 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:29:36.974 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":232,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":8847,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/2662028892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2662028892},{"type":"v1","addr":"172.21.15.116:6833","nonce":2662028892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":232}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:27:40.040636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6641},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6641":{"gid":6641,"name":"d","rank":0,"incarnation":223,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/2915274073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2915274073},{"type":"v1","addr":"172.21.15.174:6837","nonce":2915274073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6641,"qdb_cluster":[6641]},"id":1},{"mdsmap":{"epoch":231,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:28:53.808539+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8772":{"gid":8772,"name":"a","rank":0,"incarnation":228,"state":"up:active","state_seq":20,"addr":"172.21.15.116:6835/3499878473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3499878473},{"type":"v1","addr":"172.21.15.116:6835","nonce":3499878473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8772,"qdb_cluster":[8772]},"id":2}]} 2024-04-25T00:29:36.974 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 232 2024-04-25T00:29:36.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.972+0000 7f218a7fc640 1 -- 172.21.15.116:0/2251680466 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2190067a60 msgr2=0x7f2190069f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:36.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.972+0000 7f218a7fc640 1 --2- 172.21.15.116:0/2251680466 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2190067a60 0x7f2190069f20 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f21980098f0 tx=0x7f2198006f90 comp rx=0 tx=0).stop 2024-04-25T00:29:36.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.972+0000 7f218a7fc640 1 -- 172.21.15.116:0/2251680466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21a406d5f0 msgr2=0x7f21a40c6d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:36.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.972+0000 7f218a7fc640 1 --2- 172.21.15.116:0/2251680466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21a406d5f0 0x7f21a40c6d10 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f219400e9d0 tx=0x7f219400ee90 comp rx=0 tx=0).stop 2024-04-25T00:29:36.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.976+0000 7f218a7fc640 1 -- 172.21.15.116:0/2251680466 shutdown_connections 2024-04-25T00:29:36.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.976+0000 7f218a7fc640 1 --2- 172.21.15.116:0/2251680466 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2190067a60 0x7f2190069f20 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:36.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.976+0000 7f218a7fc640 1 --2- 172.21.15.116:0/2251680466 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f21a4152d80 0x7f21a41f32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:36.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.976+0000 7f218a7fc640 1 --2- 172.21.15.116:0/2251680466 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21a406d5f0 0x7f21a40c6d10 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:36.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.976+0000 7f218a7fc640 1 --2- 172.21.15.116:0/2251680466 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f21a406c290 0x7f21a41f69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:36.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.976+0000 7f218a7fc640 1 -- 172.21.15.116:0/2251680466 >> 172.21.15.116:0/2251680466 conn(0x7f21a40acdd0 msgr2=0x7f21a415f6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:29:36.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.976+0000 7f218a7fc640 1 -- 172.21.15.116:0/2251680466 shutdown_connections 2024-04-25T00:29:36.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:36.976+0000 7f218a7fc640 1 -- 172.21.15.116:0/2251680466 wait complete. 2024-04-25T00:29:36.998 INFO:tasks.mds_thrash.fs.[a]:kill mds.d (rank=0) 2024-04-25T00:29:36.998 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-25T00:29:36.998 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:29:37.013 INFO:tasks.ceph.mds.d:Stopped 2024-04-25T00:29:37.013 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:29:37.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.160+0000 7ff937604640 1 Processor -- start 2024-04-25T00:29:37.161 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.160+0000 7ff937604640 1 -- start start 2024-04-25T00:29:37.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.160+0000 7ff937604640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff930150d10 0x7ff9301541f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:37.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.160+0000 7ff937604640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff9300c2b10 0x7ff9300a94a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:37.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.160+0000 7ff937604640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9300734b0 con 0x7ff9300a99e0 2024-04-25T00:29:37.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.160+0000 7ff937604640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7ff930153750 con 0x7ff9300c2b10 2024-04-25T00:29:37.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.160+0000 7ff937604640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff9300731d0 con 0x7ff930150d10 2024-04-25T00:29:37.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.160+0000 7ff934b9f640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ff9300c2b10 0x7ff9300a94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:37.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.160+0000 7ff935ba1640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7ff9300a99e0 0x7ff9300c3260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:42020/0 (socket says 172.21.15.116:42020) 2024-04-25T00:29:37.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.160+0000 7ff935ba1640 1 -- 172.21.15.116:0/3520060478 learned_addr learned my addr 172.21.15.116:0/3520060478 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:29:37.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.160+0000 7ff927fff640 1 -- 172.21.15.116:0/3520060478 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 623570947 0 0) 0x7ff9300734b0 con 0x7ff9300a99e0 2024-04-25T00:29:37.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.160+0000 7ff927fff640 1 -- 172.21.15.116:0/3520060478 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff91c003530 con 0x7ff9300a99e0 2024-04-25T00:29:37.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.160+0000 7ff934b9f640 1 -- 172.21.15.116:0/3520060478 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff930150d10 msgr2=0x7ff9301541f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:29:37.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.160+0000 7ff934b9f640 1 --2- 172.21.15.116:0/3520060478 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff930150d10 0x7ff9301541f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:37.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.160+0000 7ff934b9f640 1 -- 172.21.15.116:0/3520060478 >> v1:172.21.15.174:6789/0 conn(0x7ff9300a99e0 legacy=0x7ff9300c3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:37.164 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.160+0000 7ff934b9f640 1 -- 172.21.15.116:0/3520060478 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9301539f0 con 0x7ff9300c2b10 2024-04-25T00:29:37.168 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.164+0000 7ff934b9f640 1 --2- 172.21.15.116:0/3520060478 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff9300c2b10 0x7ff9300a94a0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7ff9200097c0 tx=0x7ff92001c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4549f4c7dfa425d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:29:37.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.164+0000 7ff927fff640 1 -- 172.21.15.116:0/3520060478 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9200b0070 con 0x7ff9300c2b10 2024-04-25T00:29:37.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.164+0000 7ff927fff640 1 -- 172.21.15.116:0/3520060478 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9200ab600 con 0x7ff9300c2b10 2024-04-25T00:29:37.169 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.164+0000 7ff927fff640 1 -- 172.21.15.116:0/3520060478 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9200ab9d0 con 0x7ff9300c2b10 2024-04-25T00:29:37.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.168+0000 7ff937604640 1 -- 172.21.15.116:0/3520060478 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff9300c2b10 msgr2=0x7ff9300a94a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:37.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.168+0000 7ff937604640 1 --2- 172.21.15.116:0/3520060478 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff9300c2b10 0x7ff9300a94a0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7ff9200097c0 tx=0x7ff92001c410 comp rx=0 tx=0).stop 2024-04-25T00:29:37.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.168+0000 7ff937604640 1 -- 172.21.15.116:0/3520060478 shutdown_connections 2024-04-25T00:29:37.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.168+0000 7ff937604640 1 --2- 172.21.15.116:0/3520060478 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff9300c2b10 0x7ff9300a94a0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:37.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.168+0000 7ff937604640 1 --2- 172.21.15.116:0/3520060478 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff930150d10 0x7ff9301541f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:37.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.168+0000 7ff937604640 1 -- 172.21.15.116:0/3520060478 >> 172.21.15.116:0/3520060478 conn(0x7ff9300c4c10 msgr2=0x7ff9300c5020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:29:37.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.168+0000 7ff937604640 1 -- 172.21.15.116:0/3520060478 shutdown_connections 2024-04-25T00:29:37.170 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.168+0000 7ff937604640 1 -- 172.21.15.116:0/3520060478 wait complete. 2024-04-25T00:29:37.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.168+0000 7ff937604640 1 Processor -- start 2024-04-25T00:29:37.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.168+0000 7ff937604640 1 -- start start 2024-04-25T00:29:37.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.168+0000 7ff937604640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff9300c2b10 0x7ff9301409e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:37.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.168+0000 7ff937604640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff930150d10 0x7ff9300c17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:37.171 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.168+0000 7ff937604640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff9301416c0 0x7ff93006dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:37.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.168+0000 7ff937604640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff930151270 con 0x7ff930150d10 2024-04-25T00:29:37.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.168+0000 7ff937604640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff9301510f0 con 0x7ff9301416c0 2024-04-25T00:29:37.172 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.168+0000 7ff937604640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff930160750 con 0x7ff9300c2b10 2024-04-25T00:29:37.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.168+0000 7ff934b9f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff930150d10 0x7ff9300c17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:37.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.168+0000 7ff935ba1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff9301416c0 0x7ff93006dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:37.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.168+0000 7ff9353a0640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff9300c2b10 0x7ff9301409e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:37.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.168+0000 7ff934b9f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff930150d10 0x7ff9300c17a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37504/0 (socket says 172.21.15.116:37504) 2024-04-25T00:29:37.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.168+0000 7ff934b9f640 1 -- 172.21.15.116:0/1854446781 learned_addr learned my addr 172.21.15.116:0/1854446781 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:29:37.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.168+0000 7ff934b9f640 1 -- 172.21.15.116:0/1854446781 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff9300c2b10 msgr2=0x7ff9301409e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:37.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.168+0000 7ff934b9f640 1 --2- 172.21.15.116:0/1854446781 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff9300c2b10 0x7ff9301409e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:37.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.168+0000 7ff934b9f640 1 -- 172.21.15.116:0/1854446781 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff9301416c0 msgr2=0x7ff93006dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:37.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.168+0000 7ff934b9f640 1 --2- 172.21.15.116:0/1854446781 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff9301416c0 0x7ff93006dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:37.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.168+0000 7ff934b9f640 1 -- 172.21.15.116:0/1854446781 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff93014f7d0 con 0x7ff930150d10 2024-04-25T00:29:37.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.168+0000 7ff934b9f640 1 --2- 172.21.15.116:0/1854446781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff930150d10 0x7ff9300c17a0 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7ff92001a040 tx=0x7ff9200191b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:29:37.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.168+0000 7ff925ffb640 1 -- 172.21.15.116:0/1854446781 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9200b8070 con 0x7ff930150d10 2024-04-25T00:29:37.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.168+0000 7ff925ffb640 1 -- 172.21.15.116:0/1854446781 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9200b4410 con 0x7ff930150d10 2024-04-25T00:29:37.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.168+0000 7ff925ffb640 1 -- 172.21.15.116:0/1854446781 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9200b0040 con 0x7ff930150d10 2024-04-25T00:29:37.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.168+0000 7ff937604640 1 -- 172.21.15.116:0/1854446781 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff930140f20 con 0x7ff930150d10 2024-04-25T00:29:37.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.172+0000 7ff937604640 1 -- 172.21.15.116:0/1854446781 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9301ee880 con 0x7ff930150d10 2024-04-25T00:29:37.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.172+0000 7ff925ffb640 1 -- 172.21.15.116:0/1854446781 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff9200bb070 con 0x7ff930150d10 2024-04-25T00:29:37.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.172+0000 7ff925ffb640 1 --2- 172.21.15.116:0/1854446781 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff9140678c0 0x7ff914069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:37.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.172+0000 7ff925ffb640 1 -- 172.21.15.116:0/1854446781 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 10450+0+0 (secure 0 0 0) 0x7ff920120ae0 con 0x7ff930150d10 2024-04-25T00:29:37.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.172+0000 7ff937604640 1 -- 172.21.15.116:0/1854446781 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff930068a60 con 0x7ff930150d10 2024-04-25T00:29:37.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.176+0000 7ff9353a0640 1 --2- 172.21.15.116:0/1854446781 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff9140678c0 0x7ff914069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:37.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.176+0000 7ff9353a0640 1 --2- 172.21.15.116:0/1854446781 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff9140678c0 0x7ff914069d80 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7ff92c009990 tx=0x7ff92c009110 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:29:37.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.184+0000 7ff925ffb640 1 -- 172.21.15.116:0/1854446781 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ff930140f20 con 0x7ff930150d10 2024-04-25T00:29:37.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.472+0000 7ff937604640 1 -- 172.21.15.116:0/1854446781 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff93014e9c0 con 0x7ff930150d10 2024-04-25T00:29:37.477 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.472+0000 7ff925ffb640 1 -- 172.21.15.116:0/1854446781 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 232 v232) ==== 95+0+6509 (secure 0 0 0) 0x7ff920016090 con 0x7ff930150d10 2024-04-25T00:29:37.477 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:29:37.477 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":232,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":8847,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/2662028892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2662028892},{"type":"v1","addr":"172.21.15.116:6833","nonce":2662028892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":232}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:27:40.040636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6641},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6641":{"gid":6641,"name":"d","rank":0,"incarnation":223,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/2915274073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2915274073},{"type":"v1","addr":"172.21.15.174:6837","nonce":2915274073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6641,"qdb_cluster":[6641]},"id":1},{"mdsmap":{"epoch":231,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:28:53.808539+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8772":{"gid":8772,"name":"a","rank":0,"incarnation":228,"state":"up:active","state_seq":20,"addr":"172.21.15.116:6835/3499878473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3499878473},{"type":"v1","addr":"172.21.15.116:6835","nonce":3499878473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8772,"qdb_cluster":[8772]},"id":2}]} 2024-04-25T00:29:37.478 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 232 2024-04-25T00:29:37.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.476+0000 7ff937604640 1 -- 172.21.15.116:0/1854446781 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff9140678c0 msgr2=0x7ff914069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:37.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.476+0000 7ff937604640 1 --2- 172.21.15.116:0/1854446781 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff9140678c0 0x7ff914069d80 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7ff92c009990 tx=0x7ff92c009110 comp rx=0 tx=0).stop 2024-04-25T00:29:37.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.476+0000 7ff937604640 1 -- 172.21.15.116:0/1854446781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff930150d10 msgr2=0x7ff9300c17a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:37.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.476+0000 7ff937604640 1 --2- 172.21.15.116:0/1854446781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff930150d10 0x7ff9300c17a0 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7ff92001a040 tx=0x7ff9200191b0 comp rx=0 tx=0).stop 2024-04-25T00:29:37.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.476+0000 7ff937604640 1 -- 172.21.15.116:0/1854446781 shutdown_connections 2024-04-25T00:29:37.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.476+0000 7ff937604640 1 --2- 172.21.15.116:0/1854446781 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff9140678c0 0x7ff914069d80 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:37.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.476+0000 7ff937604640 1 --2- 172.21.15.116:0/1854446781 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff9301416c0 0x7ff93006dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:37.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.476+0000 7ff937604640 1 --2- 172.21.15.116:0/1854446781 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff930150d10 0x7ff9300c17a0 unknown :-1 s=CLOSED pgs=1669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:37.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.476+0000 7ff937604640 1 --2- 172.21.15.116:0/1854446781 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff9300c2b10 0x7ff9301409e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:37.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.476+0000 7ff937604640 1 -- 172.21.15.116:0/1854446781 >> 172.21.15.116:0/1854446781 conn(0x7ff9300c4c10 msgr2=0x7ff9300c4ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:29:37.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.476+0000 7ff937604640 1 -- 172.21.15.116:0/1854446781 shutdown_connections 2024-04-25T00:29:37.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:37.476+0000 7ff937604640 1 -- 172.21.15.116:0/1854446781 wait complete. 2024-04-25T00:29:37.498 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:29:39.499 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:29:39.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.652+0000 7f5c950b8640 1 Processor -- start 2024-04-25T00:29:39.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.652+0000 7f5c950b8640 1 -- start start 2024-04-25T00:29:39.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.652+0000 7f5c950b8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5c90158e70 0x7f5c90073490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:39.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.652+0000 7f5c950b8640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c9014fe40 con 0x7f5c9014fa60 2024-04-25T00:29:39.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.652+0000 7f5c950b8640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c901558c0 con 0x7f5c900c0b30 2024-04-25T00:29:39.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.652+0000 7f5c950b8640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f5c90155c10 con 0x7f5c90158e70 2024-04-25T00:29:39.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.652+0000 7f5c8f59e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5c90158e70 0x7f5c90073490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:39.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.652+0000 7f5c8f59e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5c90158e70 0x7f5c90073490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37516/0 (socket says 172.21.15.116:37516) 2024-04-25T00:29:39.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.652+0000 7f5c8f59e640 1 -- 172.21.15.116:0/1769636807 learned_addr learned my addr 172.21.15.116:0/1769636807 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:29:39.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.652+0000 7f5c8f59e640 1 -- 172.21.15.116:0/1769636807 >> v1:172.21.15.174:6789/0 conn(0x7f5c900c0b30 legacy=0x7f5c900c3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:39.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.652+0000 7f5c8f59e640 1 -- 172.21.15.116:0/1769636807 >> v1:172.21.15.116:6789/0 conn(0x7f5c9014fa60 legacy=0x7f5c901560a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:39.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.652+0000 7f5c8f59e640 1 -- 172.21.15.116:0/1769636807 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c900c1040 con 0x7f5c90158e70 2024-04-25T00:29:39.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.656+0000 7f5c8f59e640 1 --2- 172.21.15.116:0/1769636807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c90158e70 0x7f5c90073490 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f5c88009e50 tx=0x7f5c8801c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=912c4b99217f85ae server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:29:39.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.656+0000 7f5c8e59c640 1 -- 172.21.15.116:0/1769636807 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c880a57a0 con 0x7f5c90158e70 2024-04-25T00:29:39.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.656+0000 7f5c8e59c640 1 -- 172.21.15.116:0/1769636807 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c880a5940 con 0x7f5c90158e70 2024-04-25T00:29:39.658 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.656+0000 7f5c8e59c640 1 -- 172.21.15.116:0/1769636807 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c880a5d10 con 0x7f5c90158e70 2024-04-25T00:29:39.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.660+0000 7f5c950b8640 1 -- 172.21.15.116:0/1769636807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c90158e70 msgr2=0x7f5c90073490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:39.662 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.660+0000 7f5c950b8640 1 --2- 172.21.15.116:0/1769636807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c90158e70 0x7f5c90073490 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f5c88009e50 tx=0x7f5c8801c8e0 comp rx=0 tx=0).stop 2024-04-25T00:29:39.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.660+0000 7f5c950b8640 1 -- 172.21.15.116:0/1769636807 shutdown_connections 2024-04-25T00:29:39.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.660+0000 7f5c950b8640 1 --2- 172.21.15.116:0/1769636807 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c90158e70 0x7f5c90073490 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:39.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.660+0000 7f5c950b8640 1 -- 172.21.15.116:0/1769636807 >> 172.21.15.116:0/1769636807 conn(0x7f5c900c4200 msgr2=0x7f5c900c4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:29:39.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.660+0000 7f5c950b8640 1 -- 172.21.15.116:0/1769636807 shutdown_connections 2024-04-25T00:29:39.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.660+0000 7f5c950b8640 1 -- 172.21.15.116:0/1769636807 wait complete. 2024-04-25T00:29:39.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.660+0000 7f5c950b8640 1 Processor -- start 2024-04-25T00:29:39.663 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.660+0000 7f5c950b8640 1 -- start start 2024-04-25T00:29:39.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.660+0000 7f5c950b8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c900c0b30 0x7f5c90152450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:39.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.660+0000 7f5c950b8640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5c9014fa60 0x7f5c901fe710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:39.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.660+0000 7f5c950b8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5c90158e70 0x7f5c900bc720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:39.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.660+0000 7f5c950b8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5c900c3e90 con 0x7f5c900c0b30 2024-04-25T00:29:39.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.660+0000 7f5c950b8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5c900c1b90 con 0x7f5c90158e70 2024-04-25T00:29:39.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.660+0000 7f5c950b8640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f5c900c4010 con 0x7f5c9014fa60 2024-04-25T00:29:39.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.660+0000 7f5c8ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c900c0b30 0x7f5c90152450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:39.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.660+0000 7f5c8f59e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5c90158e70 0x7f5c900bc720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:39.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.660+0000 7f5c8ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c900c0b30 0x7f5c90152450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37530/0 (socket says 172.21.15.116:37530) 2024-04-25T00:29:39.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.660+0000 7f5c8ed9d640 1 -- 172.21.15.116:0/403282191 learned_addr learned my addr 172.21.15.116:0/403282191 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:29:39.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.660+0000 7f5c87fff640 1 --2- 172.21.15.116:0/403282191 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5c9014fa60 0x7f5c901fe710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:39.665 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.664+0000 7f5c8ed9d640 1 -- 172.21.15.116:0/403282191 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5c9014fa60 msgr2=0x7f5c901fe710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:39.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.664+0000 7f5c8ed9d640 1 --2- 172.21.15.116:0/403282191 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5c9014fa60 0x7f5c901fe710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:39.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.664+0000 7f5c8ed9d640 1 -- 172.21.15.116:0/403282191 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5c90158e70 msgr2=0x7f5c900bc720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:39.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.664+0000 7f5c8ed9d640 1 --2- 172.21.15.116:0/403282191 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5c90158e70 0x7f5c900bc720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:39.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.664+0000 7f5c8ed9d640 1 -- 172.21.15.116:0/403282191 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c9013de00 con 0x7f5c900c0b30 2024-04-25T00:29:39.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.664+0000 7f5c8ed9d640 1 --2- 172.21.15.116:0/403282191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c900c0b30 0x7f5c90152450 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f5c7800c3f0 tx=0x7f5c7800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:29:39.666 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.664+0000 7f5c877fe640 1 -- 172.21.15.116:0/403282191 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c78017070 con 0x7f5c900c0b30 2024-04-25T00:29:39.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.664+0000 7f5c877fe640 1 -- 172.21.15.116:0/403282191 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c780052d0 con 0x7f5c900c0b30 2024-04-25T00:29:39.667 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.664+0000 7f5c877fe640 1 -- 172.21.15.116:0/403282191 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c78003840 con 0x7f5c900c0b30 2024-04-25T00:29:39.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.664+0000 7f5c950b8640 1 -- 172.21.15.116:0/403282191 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c90153050 con 0x7f5c900c0b30 2024-04-25T00:29:39.668 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.664+0000 7f5c950b8640 1 -- 172.21.15.116:0/403282191 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c90152990 con 0x7f5c900c0b30 2024-04-25T00:29:39.669 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.668+0000 7f5c950b8640 1 -- 172.21.15.116:0/403282191 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c90068a60 con 0x7f5c900c0b30 2024-04-25T00:29:39.670 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.668+0000 7f5c877fe640 1 -- 172.21.15.116:0/403282191 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5c7801a070 con 0x7f5c900c0b30 2024-04-25T00:29:39.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.668+0000 7f5c877fe640 1 --2- 172.21.15.116:0/403282191 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5c700678c0 0x7f5c70069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:39.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.668+0000 7f5c877fe640 1 -- 172.21.15.116:0/403282191 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 10450+0+0 (secure 0 0 0) 0x7f5c780861e0 con 0x7f5c900c0b30 2024-04-25T00:29:39.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.668+0000 7f5c87fff640 1 --2- 172.21.15.116:0/403282191 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5c700678c0 0x7f5c70069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:39.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.668+0000 7f5c87fff640 1 --2- 172.21.15.116:0/403282191 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5c700678c0 0x7f5c70069d80 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f5c901580f0 tx=0x7f5c80009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:29:39.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.676+0000 7f5c877fe640 1 -- 172.21.15.116:0/403282191 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f5c9014fe40 con 0x7f5c900c0b30 2024-04-25T00:29:39.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.964+0000 7f5c950b8640 1 -- 172.21.15.116:0/403282191 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5c9006c790 con 0x7f5c900c0b30 2024-04-25T00:29:39.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.964+0000 7f5c877fe640 1 -- 172.21.15.116:0/403282191 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 232 v232) ==== 95+0+6509 (secure 0 0 0) 0x7f5c78005480 con 0x7f5c900c0b30 2024-04-25T00:29:39.968 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:29:39.968 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":232,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":8847,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/2662028892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2662028892},{"type":"v1","addr":"172.21.15.116:6833","nonce":2662028892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":232}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:27:40.040636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6641},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6641":{"gid":6641,"name":"d","rank":0,"incarnation":223,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/2915274073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2915274073},{"type":"v1","addr":"172.21.15.174:6837","nonce":2915274073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6641,"qdb_cluster":[6641]},"id":1},{"mdsmap":{"epoch":231,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:28:53.808539+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8772":{"gid":8772,"name":"a","rank":0,"incarnation":228,"state":"up:active","state_seq":20,"addr":"172.21.15.116:6835/3499878473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3499878473},{"type":"v1","addr":"172.21.15.116:6835","nonce":3499878473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8772,"qdb_cluster":[8772]},"id":2}]} 2024-04-25T00:29:39.968 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 232 2024-04-25T00:29:39.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.968+0000 7f5c950b8640 1 -- 172.21.15.116:0/403282191 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5c700678c0 msgr2=0x7f5c70069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:39.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.968+0000 7f5c950b8640 1 --2- 172.21.15.116:0/403282191 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5c700678c0 0x7f5c70069d80 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f5c901580f0 tx=0x7f5c80009040 comp rx=0 tx=0).stop 2024-04-25T00:29:39.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.968+0000 7f5c950b8640 1 -- 172.21.15.116:0/403282191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c900c0b30 msgr2=0x7f5c90152450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:39.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.968+0000 7f5c950b8640 1 --2- 172.21.15.116:0/403282191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c900c0b30 0x7f5c90152450 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f5c7800c3f0 tx=0x7f5c7800c8b0 comp rx=0 tx=0).stop 2024-04-25T00:29:39.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.968+0000 7f5c950b8640 1 -- 172.21.15.116:0/403282191 shutdown_connections 2024-04-25T00:29:39.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.968+0000 7f5c950b8640 1 --2- 172.21.15.116:0/403282191 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5c700678c0 0x7f5c70069d80 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:39.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.968+0000 7f5c950b8640 1 --2- 172.21.15.116:0/403282191 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5c90158e70 0x7f5c900bc720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:39.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.968+0000 7f5c950b8640 1 --2- 172.21.15.116:0/403282191 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5c9014fa60 0x7f5c901fe710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:39.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.968+0000 7f5c950b8640 1 --2- 172.21.15.116:0/403282191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5c900c0b30 0x7f5c90152450 secure :-1 s=CLOSED pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f5c7800c3f0 tx=0x7f5c7800c8b0 comp rx=0 tx=0).stop 2024-04-25T00:29:39.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.968+0000 7f5c950b8640 1 -- 172.21.15.116:0/403282191 >> 172.21.15.116:0/403282191 conn(0x7f5c900c4200 msgr2=0x7f5c9006e510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:29:39.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.968+0000 7f5c950b8640 1 -- 172.21.15.116:0/403282191 shutdown_connections 2024-04-25T00:29:39.971 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:39.968+0000 7f5c950b8640 1 -- 172.21.15.116:0/403282191 wait complete. 2024-04-25T00:29:39.991 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:29:41.993 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:29:42.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.124+0000 7f3a96e97640 1 Processor -- start 2024-04-25T00:29:42.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.124+0000 7f3a96e97640 1 -- start start 2024-04-25T00:29:42.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.124+0000 7f3a96e97640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3a900c9040 0x7f3a900c9470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:42.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.124+0000 7f3a96e97640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a90157630 con 0x7f3a900ad180 2024-04-25T00:29:42.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.124+0000 7f3a96e97640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a90157910 con 0x7f3a900adf40 2024-04-25T00:29:42.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.124+0000 7f3a96e97640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f3a900ad600 con 0x7f3a900c9040 2024-04-25T00:29:42.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.124+0000 7f3a95434640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3a900c9040 0x7f3a900c9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:42.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.124+0000 7f3a8ffff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f3a900ad180 0x7f3a900c83b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:59474/0 (socket says 172.21.15.116:59474) 2024-04-25T00:29:42.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.124+0000 7f3a8ffff640 1 -- 172.21.15.116:0/555229071 learned_addr learned my addr 172.21.15.116:0/555229071 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:29:42.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.124+0000 7f3a95434640 1 -- 172.21.15.116:0/555229071 >> v1:172.21.15.174:6789/0 conn(0x7f3a900adf40 legacy=0x7f3a90156c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:42.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.124+0000 7f3a95434640 1 -- 172.21.15.116:0/555229071 >> v1:172.21.15.116:6789/0 conn(0x7f3a900ad180 legacy=0x7f3a900c83b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:42.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.124+0000 7f3a95434640 1 -- 172.21.15.116:0/555229071 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a900c7bb0 con 0x7f3a900c9040 2024-04-25T00:29:42.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.128+0000 7f3a95434640 1 --2- 172.21.15.116:0/555229071 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a900c9040 0x7f3a900c9470 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f3a88009d90 tx=0x7f3a8801c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29e8df53c798f6ec server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:29:42.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.128+0000 7f3a8f7fe640 1 -- 172.21.15.116:0/555229071 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a880b2070 con 0x7f3a900c9040 2024-04-25T00:29:42.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.128+0000 7f3a8f7fe640 1 -- 172.21.15.116:0/555229071 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a880a5830 con 0x7f3a900c9040 2024-04-25T00:29:42.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.128+0000 7f3a8f7fe640 1 -- 172.21.15.116:0/555229071 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a880a5c00 con 0x7f3a900c9040 2024-04-25T00:29:42.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.128+0000 7f3a96e97640 1 -- 172.21.15.116:0/555229071 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a900c9040 msgr2=0x7f3a900c9470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:42.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.128+0000 7f3a96e97640 1 --2- 172.21.15.116:0/555229071 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a900c9040 0x7f3a900c9470 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f3a88009d90 tx=0x7f3a8801c970 comp rx=0 tx=0).stop 2024-04-25T00:29:42.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.128+0000 7f3a96e97640 1 -- 172.21.15.116:0/555229071 shutdown_connections 2024-04-25T00:29:42.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.128+0000 7f3a96e97640 1 --2- 172.21.15.116:0/555229071 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a900c9040 0x7f3a900c9470 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:42.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.128+0000 7f3a96e97640 1 -- 172.21.15.116:0/555229071 >> 172.21.15.116:0/555229071 conn(0x7f3a90074480 msgr2=0x7f3a90074890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:29:42.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.128+0000 7f3a96e97640 1 -- 172.21.15.116:0/555229071 shutdown_connections 2024-04-25T00:29:42.131 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.128+0000 7f3a96e97640 1 -- 172.21.15.116:0/555229071 wait complete. 2024-04-25T00:29:42.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.128+0000 7f3a96e97640 1 Processor -- start 2024-04-25T00:29:42.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.128+0000 7f3a96e97640 1 -- start start 2024-04-25T00:29:42.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.128+0000 7f3a96e97640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3a900ad180 0x7f3a9014db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:42.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.128+0000 7f3a96e97640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a900adf40 0x7f3a901fec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:42.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.128+0000 7f3a96e97640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a900c9040 0x7f3a900c3a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:42.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.128+0000 7f3a96e97640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3a900ae320 con 0x7f3a900adf40 2024-04-25T00:29:42.133 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.128+0000 7f3a96e97640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3a900c7f20 con 0x7f3a900c9040 2024-04-25T00:29:42.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.128+0000 7f3a96e97640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f3a900ae4a0 con 0x7f3a900ad180 2024-04-25T00:29:42.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.128+0000 7f3a8ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a900adf40 0x7f3a901fec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:42.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.128+0000 7f3a8ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a900adf40 0x7f3a901fec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37554/0 (socket says 172.21.15.116:37554) 2024-04-25T00:29:42.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.128+0000 7f3a8ffff640 1 -- 172.21.15.116:0/1198777992 learned_addr learned my addr 172.21.15.116:0/1198777992 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:29:42.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.128+0000 7f3a95434640 1 --2- 172.21.15.116:0/1198777992 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a900c9040 0x7f3a900c3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:42.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.128+0000 7f3a94c33640 1 --2- 172.21.15.116:0/1198777992 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3a900ad180 0x7f3a9014db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:42.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.128+0000 7f3a8ffff640 1 -- 172.21.15.116:0/1198777992 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3a900ad180 msgr2=0x7f3a9014db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:42.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.128+0000 7f3a8ffff640 1 --2- 172.21.15.116:0/1198777992 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3a900ad180 0x7f3a9014db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:42.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.128+0000 7f3a8ffff640 1 -- 172.21.15.116:0/1198777992 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a900c9040 msgr2=0x7f3a900c3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:42.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.128+0000 7f3a8ffff640 1 --2- 172.21.15.116:0/1198777992 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a900c9040 0x7f3a900c3a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:42.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.128+0000 7f3a8ffff640 1 -- 172.21.15.116:0/1198777992 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a90070c60 con 0x7f3a900adf40 2024-04-25T00:29:42.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.128+0000 7f3a8ffff640 1 --2- 172.21.15.116:0/1198777992 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a900adf40 0x7f3a901fec10 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c02e500 tx=0x7f3a7c02e9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:29:42.134 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.132+0000 7f3a8d7fa640 1 -- 172.21.15.116:0/1198777992 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a7c038020 con 0x7f3a900adf40 2024-04-25T00:29:42.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.132+0000 7f3a8d7fa640 1 -- 172.21.15.116:0/1198777992 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a7c034830 con 0x7f3a900adf40 2024-04-25T00:29:42.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.132+0000 7f3a96e97640 1 -- 172.21.15.116:0/1198777992 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a90070f40 con 0x7f3a900adf40 2024-04-25T00:29:42.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.132+0000 7f3a8d7fa640 1 -- 172.21.15.116:0/1198777992 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a7c034c60 con 0x7f3a900adf40 2024-04-25T00:29:42.135 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.132+0000 7f3a96e97640 1 -- 172.21.15.116:0/1198777992 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a9006f540 con 0x7f3a900adf40 2024-04-25T00:29:42.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.136+0000 7f3a8d7fa640 1 -- 172.21.15.116:0/1198777992 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3a7c03b020 con 0x7f3a900adf40 2024-04-25T00:29:42.137 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.136+0000 7f3a96e97640 1 -- 172.21.15.116:0/1198777992 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a90068a60 con 0x7f3a900adf40 2024-04-25T00:29:42.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.136+0000 7f3a8d7fa640 1 --2- 172.21.15.116:0/1198777992 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3a740678c0 0x7f3a74069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:42.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.136+0000 7f3a8d7fa640 1 -- 172.21.15.116:0/1198777992 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 10450+0+0 (secure 0 0 0) 0x7f3a7c0af350 con 0x7f3a900adf40 2024-04-25T00:29:42.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.136+0000 7f3a94c33640 1 --2- 172.21.15.116:0/1198777992 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3a740678c0 0x7f3a74069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:42.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.136+0000 7f3a94c33640 1 --2- 172.21.15.116:0/1198777992 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3a740678c0 0x7f3a74069d80 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f3a84001810 tx=0x7f3a84009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:29:42.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.140+0000 7f3a8d7fa640 1 -- 172.21.15.116:0/1198777992 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f3a7c0772f0 con 0x7f3a900adf40 2024-04-25T00:29:42.288 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:29:42.291 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:29:42.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.416+0000 7f3a96e97640 1 -- 172.21.15.116:0/1198777992 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3a90071110 con 0x7f3a900adf40 2024-04-25T00:29:42.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.416+0000 7f3a8d7fa640 1 -- 172.21.15.116:0/1198777992 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 232 v232) ==== 95+0+6509 (secure 0 0 0) 0x7f3a7c07b5f0 con 0x7f3a900adf40 2024-04-25T00:29:42.419 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:29:42.419 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":232,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":8847,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/2662028892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2662028892},{"type":"v1","addr":"172.21.15.116:6833","nonce":2662028892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":232}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:27:40.040636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6641},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6641":{"gid":6641,"name":"d","rank":0,"incarnation":223,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/2915274073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2915274073},{"type":"v1","addr":"172.21.15.174:6837","nonce":2915274073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6641,"qdb_cluster":[6641]},"id":1},{"mdsmap":{"epoch":231,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:28:53.808539+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8772":{"gid":8772,"name":"a","rank":0,"incarnation":228,"state":"up:active","state_seq":20,"addr":"172.21.15.116:6835/3499878473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3499878473},{"type":"v1","addr":"172.21.15.116:6835","nonce":3499878473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8772,"qdb_cluster":[8772]},"id":2}]} 2024-04-25T00:29:42.421 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 232 2024-04-25T00:29:42.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.416+0000 7f3a96e97640 1 -- 172.21.15.116:0/1198777992 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3a740678c0 msgr2=0x7f3a74069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:42.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.416+0000 7f3a96e97640 1 --2- 172.21.15.116:0/1198777992 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3a740678c0 0x7f3a74069d80 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f3a84001810 tx=0x7f3a84009040 comp rx=0 tx=0).stop 2024-04-25T00:29:42.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.416+0000 7f3a96e97640 1 -- 172.21.15.116:0/1198777992 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a900adf40 msgr2=0x7f3a901fec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:42.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.416+0000 7f3a96e97640 1 --2- 172.21.15.116:0/1198777992 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a900adf40 0x7f3a901fec10 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c02e500 tx=0x7f3a7c02e9c0 comp rx=0 tx=0).stop 2024-04-25T00:29:42.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.416+0000 7f3a96e97640 1 -- 172.21.15.116:0/1198777992 shutdown_connections 2024-04-25T00:29:42.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.416+0000 7f3a96e97640 1 --2- 172.21.15.116:0/1198777992 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3a740678c0 0x7f3a74069d80 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:42.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.416+0000 7f3a96e97640 1 --2- 172.21.15.116:0/1198777992 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a900c9040 0x7f3a900c3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:42.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.416+0000 7f3a96e97640 1 --2- 172.21.15.116:0/1198777992 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a900adf40 0x7f3a901fec10 secure :-1 s=CLOSED pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c02e500 tx=0x7f3a7c02e9c0 comp rx=0 tx=0).stop 2024-04-25T00:29:42.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.416+0000 7f3a96e97640 1 --2- 172.21.15.116:0/1198777992 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3a900ad180 0x7f3a9014db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:42.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.416+0000 7f3a96e97640 1 -- 172.21.15.116:0/1198777992 >> 172.21.15.116:0/1198777992 conn(0x7f3a90074480 msgr2=0x7f3a900c83b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:29:42.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.420+0000 7f3a96e97640 1 -- 172.21.15.116:0/1198777992 shutdown_connections 2024-04-25T00:29:42.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:42.420+0000 7f3a96e97640 1 -- 172.21.15.116:0/1198777992 wait complete. 2024-04-25T00:29:42.442 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:29:44.443 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:29:44.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.579+0000 7f0163444640 1 Processor -- start 2024-04-25T00:29:44.584 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.579+0000 7f0163444640 1 -- start start 2024-04-25T00:29:44.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.579+0000 7f0163444640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f015c152db0 0x7f015c0c7af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:44.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.579+0000 7f0163444640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f015c0c8030 0x7f015c0c5a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:44.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.579+0000 7f0163444640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f015c0c5fa0 0x7f015c1558c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:44.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.579+0000 7f0163444640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f015c158eb0 con 0x7f015c0c5fa0 2024-04-25T00:29:44.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.579+0000 7f0163444640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f015c159190 con 0x7f015c0c8030 2024-04-25T00:29:44.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.579+0000 7f0163444640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f015c159470 con 0x7f015c152db0 2024-04-25T00:29:44.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.579+0000 7f01611e0640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f015c152db0 0x7f015c0c7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:44.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.583+0000 7f01611e0640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f015c152db0 0x7f015c0c7af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:43870/0 (socket says 172.21.15.116:43870) 2024-04-25T00:29:44.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.583+0000 7f01611e0640 1 -- 172.21.15.116:0/2601254285 learned_addr learned my addr 172.21.15.116:0/2601254285 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:29:44.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.583+0000 7f01609df640 1 --2- 172.21.15.116:0/2601254285 >> v2:172.21.15.174:3300/0 conn(0x7f015c0c8030 0x7f015c0c5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:44.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.583+0000 7f01619e1640 1 --2- 172.21.15.116:0/2601254285 >> v2:172.21.15.116:3300/0 conn(0x7f015c0c5fa0 0x7f015c1558c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:44.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.583+0000 7f01619e1640 1 -- 172.21.15.116:0/2601254285 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f015c152db0 msgr2=0x7f015c0c7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:29:44.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.583+0000 7f01619e1640 1 --2- 172.21.15.116:0/2601254285 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f015c152db0 0x7f015c0c7af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:44.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.583+0000 7f01619e1640 1 -- 172.21.15.116:0/2601254285 >> v2:172.21.15.174:3300/0 conn(0x7f015c0c8030 msgr2=0x7f015c0c5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:29:44.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.583+0000 7f01619e1640 1 --2- 172.21.15.116:0/2601254285 >> v2:172.21.15.174:3300/0 conn(0x7f015c0c8030 0x7f015c0c5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:44.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.583+0000 7f01619e1640 1 -- 172.21.15.116:0/2601254285 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f015c159750 con 0x7f015c0c5fa0 2024-04-25T00:29:44.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.583+0000 7f01619e1640 1 --2- 172.21.15.116:0/2601254285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f015c0c5fa0 0x7f015c1558c0 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f015c0c8570 tx=0x7f015401c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc73021ff3585f74 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:29:44.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.583+0000 7f0153fff640 1 -- 172.21.15.116:0/2601254285 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01540b2070 con 0x7f015c0c5fa0 2024-04-25T00:29:44.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.583+0000 7f0153fff640 1 -- 172.21.15.116:0/2601254285 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01540a5910 con 0x7f015c0c5fa0 2024-04-25T00:29:44.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.583+0000 7f0153fff640 1 -- 172.21.15.116:0/2601254285 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01540a5ce0 con 0x7f015c0c5fa0 2024-04-25T00:29:44.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.587+0000 7f0163444640 1 -- 172.21.15.116:0/2601254285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f015c0c5fa0 msgr2=0x7f015c1558c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:44.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.587+0000 7f0163444640 1 --2- 172.21.15.116:0/2601254285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f015c0c5fa0 0x7f015c1558c0 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f015c0c8570 tx=0x7f015401c870 comp rx=0 tx=0).stop 2024-04-25T00:29:44.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.587+0000 7f0163444640 1 -- 172.21.15.116:0/2601254285 shutdown_connections 2024-04-25T00:29:44.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.587+0000 7f0163444640 1 --2- 172.21.15.116:0/2601254285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f015c0c5fa0 0x7f015c1558c0 unknown :-1 s=CLOSED pgs=1676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:44.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.587+0000 7f0163444640 1 --2- 172.21.15.116:0/2601254285 >> v2:172.21.15.174:3300/0 conn(0x7f015c0c8030 0x7f015c0c5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:44.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.587+0000 7f0163444640 1 --2- 172.21.15.116:0/2601254285 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f015c152db0 0x7f015c0c7af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:44.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.587+0000 7f0163444640 1 -- 172.21.15.116:0/2601254285 >> 172.21.15.116:0/2601254285 conn(0x7f015c0acc10 msgr2=0x7f015c0ad020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:29:44.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.587+0000 7f0163444640 1 -- 172.21.15.116:0/2601254285 shutdown_connections 2024-04-25T00:29:44.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.587+0000 7f0163444640 1 -- 172.21.15.116:0/2601254285 wait complete. 2024-04-25T00:29:44.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.587+0000 7f0163444640 1 Processor -- start 2024-04-25T00:29:44.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.587+0000 7f0163444640 1 -- start start 2024-04-25T00:29:44.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.587+0000 7f0163444640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f015c0c5fa0 0x7f015c1551d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:44.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.587+0000 7f0163444640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f015c0c8030 0x7f015c158090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:44.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.587+0000 7f0163444640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f015c152db0 0x7f015c1ef650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:44.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.587+0000 7f0163444640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f015c0c9160 con 0x7f015c0c8030 2024-04-25T00:29:44.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.587+0000 7f0163444640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f015c0c8fe0 con 0x7f015c152db0 2024-04-25T00:29:44.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.587+0000 7f0163444640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f015c0c92e0 con 0x7f015c0c5fa0 2024-04-25T00:29:44.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.587+0000 7f01609df640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f015c0c8030 0x7f015c158090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:44.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.587+0000 7f01609df640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f015c0c8030 0x7f015c158090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37572/0 (socket says 172.21.15.116:37572) 2024-04-25T00:29:44.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.587+0000 7f01609df640 1 -- 172.21.15.116:0/1929181474 learned_addr learned my addr 172.21.15.116:0/1929181474 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:29:44.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.587+0000 7f01611e0640 1 --2- 172.21.15.116:0/1929181474 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f015c0c5fa0 0x7f015c1551d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:44.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.587+0000 7f01609df640 1 -- 172.21.15.116:0/1929181474 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f015c0c5fa0 msgr2=0x7f015c1551d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:44.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.587+0000 7f01609df640 1 --2- 172.21.15.116:0/1929181474 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f015c0c5fa0 0x7f015c1551d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:44.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.587+0000 7f01609df640 1 -- 172.21.15.116:0/1929181474 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f015c152db0 msgr2=0x7f015c1ef650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:44.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.587+0000 7f01609df640 1 --2- 172.21.15.116:0/1929181474 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f015c152db0 0x7f015c1ef650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:44.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.587+0000 7f01609df640 1 -- 172.21.15.116:0/1929181474 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f015c1eed00 con 0x7f015c0c8030 2024-04-25T00:29:44.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.591+0000 7f01609df640 1 --2- 172.21.15.116:0/1929181474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f015c0c8030 0x7f015c158090 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7f014c00e8b0 tx=0x7f014c00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:29:44.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.591+0000 7f0151ffb640 1 -- 172.21.15.116:0/1929181474 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f014c017070 con 0x7f015c0c8030 2024-04-25T00:29:44.594 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.591+0000 7f0151ffb640 1 -- 172.21.15.116:0/1929181474 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f014c00c940 con 0x7f015c0c8030 2024-04-25T00:29:44.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.591+0000 7f0151ffb640 1 -- 172.21.15.116:0/1929181474 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f014c00cd30 con 0x7f015c0c8030 2024-04-25T00:29:44.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.591+0000 7f0163444640 1 -- 172.21.15.116:0/1929181474 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f015c1eefa0 con 0x7f015c0c8030 2024-04-25T00:29:44.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.591+0000 7f0163444640 1 -- 172.21.15.116:0/1929181474 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f015c1517b0 con 0x7f015c0c8030 2024-04-25T00:29:44.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.591+0000 7f0163444640 1 -- 172.21.15.116:0/1929181474 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f015c1ef170 con 0x7f015c0c8030 2024-04-25T00:29:44.595 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.591+0000 7f0151ffb640 1 -- 172.21.15.116:0/1929181474 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f014c01a070 con 0x7f015c0c8030 2024-04-25T00:29:44.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.591+0000 7f0151ffb640 1 --2- 172.21.15.116:0/1929181474 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f01400678c0 0x7f0140069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:44.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.591+0000 7f0151ffb640 1 -- 172.21.15.116:0/1929181474 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 10450+0+0 (secure 0 0 0) 0x7f014c087150 con 0x7f015c0c8030 2024-04-25T00:29:44.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.591+0000 7f01611e0640 1 --2- 172.21.15.116:0/1929181474 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f01400678c0 0x7f0140069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:44.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.591+0000 7f01611e0640 1 --2- 172.21.15.116:0/1929181474 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f01400678c0 0x7f0140069d80 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f015c0b10c0 tx=0x7f0158009240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:29:44.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.599+0000 7f0151ffb640 1 -- 172.21.15.116:0/1929181474 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f014c0563f0 con 0x7f015c0c8030 2024-04-25T00:29:44.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.899+0000 7f0163444640 1 -- 172.21.15.116:0/1929181474 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f015c068a60 con 0x7f015c0c8030 2024-04-25T00:29:44.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.899+0000 7f0151ffb640 1 -- 172.21.15.116:0/1929181474 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 232 v232) ==== 95+0+6509 (secure 0 0 0) 0x7f014c05a6f0 con 0x7f015c0c8030 2024-04-25T00:29:44.903 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:29:44.903 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":232,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":8847,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/2662028892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2662028892},{"type":"v1","addr":"172.21.15.116:6833","nonce":2662028892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":232}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:27:40.040636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6641},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6641":{"gid":6641,"name":"d","rank":0,"incarnation":223,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/2915274073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2915274073},{"type":"v1","addr":"172.21.15.174:6837","nonce":2915274073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6641,"qdb_cluster":[6641]},"id":1},{"mdsmap":{"epoch":231,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:28:53.808539+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8772":{"gid":8772,"name":"a","rank":0,"incarnation":228,"state":"up:active","state_seq":20,"addr":"172.21.15.116:6835/3499878473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3499878473},{"type":"v1","addr":"172.21.15.116:6835","nonce":3499878473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8772,"qdb_cluster":[8772]},"id":2}]} 2024-04-25T00:29:44.903 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 232 2024-04-25T00:29:44.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.899+0000 7f0163444640 1 -- 172.21.15.116:0/1929181474 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f01400678c0 msgr2=0x7f0140069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:44.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.899+0000 7f0163444640 1 --2- 172.21.15.116:0/1929181474 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f01400678c0 0x7f0140069d80 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f015c0b10c0 tx=0x7f0158009240 comp rx=0 tx=0).stop 2024-04-25T00:29:44.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.899+0000 7f0163444640 1 -- 172.21.15.116:0/1929181474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f015c0c8030 msgr2=0x7f015c158090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:44.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.899+0000 7f0163444640 1 --2- 172.21.15.116:0/1929181474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f015c0c8030 0x7f015c158090 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7f014c00e8b0 tx=0x7f014c00ed70 comp rx=0 tx=0).stop 2024-04-25T00:29:44.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.899+0000 7f0163444640 1 -- 172.21.15.116:0/1929181474 shutdown_connections 2024-04-25T00:29:44.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.899+0000 7f0163444640 1 --2- 172.21.15.116:0/1929181474 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f01400678c0 0x7f0140069d80 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:44.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.899+0000 7f0163444640 1 --2- 172.21.15.116:0/1929181474 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f015c152db0 0x7f015c1ef650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:44.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.899+0000 7f0163444640 1 --2- 172.21.15.116:0/1929181474 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f015c0c8030 0x7f015c158090 unknown :-1 s=CLOSED pgs=1677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:44.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.903+0000 7f0163444640 1 --2- 172.21.15.116:0/1929181474 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f015c0c5fa0 0x7f015c1551d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:44.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.903+0000 7f0163444640 1 -- 172.21.15.116:0/1929181474 >> 172.21.15.116:0/1929181474 conn(0x7f015c0acc10 msgr2=0x7f015c15f600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:29:44.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.903+0000 7f0163444640 1 -- 172.21.15.116:0/1929181474 shutdown_connections 2024-04-25T00:29:44.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:44.903+0000 7f0163444640 1 -- 172.21.15.116:0/1929181474 wait complete. 2024-04-25T00:29:44.928 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:29:46.928 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:29:47.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.071+0000 7efd9e26d640 1 Processor -- start 2024-04-25T00:29:47.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.071+0000 7efd9e26d640 1 -- start start 2024-04-25T00:29:47.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.071+0000 7efd9e26d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7efd981599a0 0x7efd981562d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:47.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.071+0000 7efd9e26d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efd980c5010 0x7efd980c54a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:47.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.071+0000 7efd9e26d640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd98073200 con 0x7efd98152e60 2024-04-25T00:29:47.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.071+0000 7efd9e26d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7efd98153240 con 0x7efd981599a0 2024-04-25T00:29:47.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.071+0000 7efd9e26d640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7efd980734e0 con 0x7efd980c5010 2024-04-25T00:29:47.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.071+0000 7efd8f7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7efd981599a0 0x7efd981562d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:47.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.071+0000 7efd8f7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7efd981599a0 0x7efd981562d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42264/0 (socket says 172.21.15.116:42264) 2024-04-25T00:29:47.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.071+0000 7efd8f7fe640 1 -- 172.21.15.116:0/3833013527 learned_addr learned my addr 172.21.15.116:0/3833013527 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:29:47.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.071+0000 7efd9c80a640 1 --2- 172.21.15.116:0/3833013527 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efd980c5010 0x7efd980c54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:47.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.071+0000 7efd8f7fe640 1 -- 172.21.15.116:0/3833013527 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efd980c5010 msgr2=0x7efd980c54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:29:47.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.071+0000 7efd8f7fe640 1 --2- 172.21.15.116:0/3833013527 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efd980c5010 0x7efd980c54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:47.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.071+0000 7efd8f7fe640 1 -- 172.21.15.116:0/3833013527 >> v1:172.21.15.174:6789/0 conn(0x7efd98152e60 legacy=0x7efd980c36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:47.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.071+0000 7efd977fe640 1 -- 172.21.15.116:0/3833013527 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 258311245 0 0) 0x7efd98073200 con 0x7efd98152e60 2024-04-25T00:29:47.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.071+0000 7efd8f7fe640 1 -- 172.21.15.116:0/3833013527 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd98158ec0 con 0x7efd981599a0 2024-04-25T00:29:47.081 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.079+0000 7efd8f7fe640 1 --2- 172.21.15.116:0/3833013527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd981599a0 0x7efd981562d0 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7efd80009870 tx=0x7efd8001c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a141de00d1a7203e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:29:47.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.079+0000 7efd977fe640 1 -- 172.21.15.116:0/3833013527 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd800b1070 con 0x7efd981599a0 2024-04-25T00:29:47.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.079+0000 7efd977fe640 1 -- 172.21.15.116:0/3833013527 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd800ac620 con 0x7efd981599a0 2024-04-25T00:29:47.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.079+0000 7efd977fe640 1 -- 172.21.15.116:0/3833013527 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd800ac9f0 con 0x7efd981599a0 2024-04-25T00:29:47.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.083+0000 7efd9e26d640 1 -- 172.21.15.116:0/3833013527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd981599a0 msgr2=0x7efd981562d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:47.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.083+0000 7efd9e26d640 1 --2- 172.21.15.116:0/3833013527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd981599a0 0x7efd981562d0 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7efd80009870 tx=0x7efd8001c4d0 comp rx=0 tx=0).stop 2024-04-25T00:29:47.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.083+0000 7efd9e26d640 1 -- 172.21.15.116:0/3833013527 shutdown_connections 2024-04-25T00:29:47.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.083+0000 7efd9e26d640 1 --2- 172.21.15.116:0/3833013527 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efd980c5010 0x7efd980c54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:47.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.083+0000 7efd9e26d640 1 --2- 172.21.15.116:0/3833013527 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd981599a0 0x7efd981562d0 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:47.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.083+0000 7efd9e26d640 1 -- 172.21.15.116:0/3833013527 >> 172.21.15.116:0/3833013527 conn(0x7efd9806c720 msgr2=0x7efd9806cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:29:47.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.083+0000 7efd9e26d640 1 -- 172.21.15.116:0/3833013527 shutdown_connections 2024-04-25T00:29:47.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.083+0000 7efd9e26d640 1 -- 172.21.15.116:0/3833013527 wait complete. 2024-04-25T00:29:47.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.083+0000 7efd9e26d640 1 Processor -- start 2024-04-25T00:29:47.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.083+0000 7efd9e26d640 1 -- start start 2024-04-25T00:29:47.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.087+0000 7efd9e26d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd980c5010 0x7efd981ee700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:47.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.087+0000 7efd9e26d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efd98152e60 0x7efd98150d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:47.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.087+0000 7efd9e26d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efd981599a0 0x7efd980ab7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:47.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.087+0000 7efd9e26d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7efd98072e20 con 0x7efd980c5010 2024-04-25T00:29:47.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.087+0000 7efd9e26d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7efd980c3df0 con 0x7efd98152e60 2024-04-25T00:29:47.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.087+0000 7efd9e26d640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7efd98072fa0 con 0x7efd981599a0 2024-04-25T00:29:47.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.087+0000 7efd97fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd980c5010 0x7efd981ee700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:47.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.087+0000 7efd97fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd980c5010 0x7efd981ee700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42280/0 (socket says 172.21.15.116:42280) 2024-04-25T00:29:47.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.087+0000 7efd97fff640 1 -- 172.21.15.116:0/482690184 learned_addr learned my addr 172.21.15.116:0/482690184 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:29:47.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.087+0000 7efd9c80a640 1 --2- 172.21.15.116:0/482690184 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efd981599a0 0x7efd980ab7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:47.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.087+0000 7efd8f7fe640 1 --2- 172.21.15.116:0/482690184 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efd98152e60 0x7efd98150d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:47.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.087+0000 7efd97fff640 1 -- 172.21.15.116:0/482690184 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efd981599a0 msgr2=0x7efd980ab7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:47.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.087+0000 7efd97fff640 1 --2- 172.21.15.116:0/482690184 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efd981599a0 0x7efd980ab7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:47.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.087+0000 7efd97fff640 1 -- 172.21.15.116:0/482690184 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efd98152e60 msgr2=0x7efd98150d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:47.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.087+0000 7efd97fff640 1 --2- 172.21.15.116:0/482690184 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efd98152e60 0x7efd98150d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:47.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.087+0000 7efd97fff640 1 -- 172.21.15.116:0/482690184 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd980ac700 con 0x7efd980c5010 2024-04-25T00:29:47.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.087+0000 7efd9c80a640 1 --2- 172.21.15.116:0/482690184 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efd981599a0 0x7efd980ab7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:29:47.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.087+0000 7efd97fff640 1 --2- 172.21.15.116:0/482690184 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd980c5010 0x7efd981ee700 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7efd8800c440 tx=0x7efd8800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:29:47.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.087+0000 7efd957fa640 1 -- 172.21.15.116:0/482690184 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd88017070 con 0x7efd980c5010 2024-04-25T00:29:47.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.087+0000 7efd957fa640 1 -- 172.21.15.116:0/482690184 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd88003600 con 0x7efd980c5010 2024-04-25T00:29:47.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.087+0000 7efd9e26d640 1 -- 172.21.15.116:0/482690184 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd980ac9e0 con 0x7efd980c5010 2024-04-25T00:29:47.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.087+0000 7efd957fa640 1 -- 172.21.15.116:0/482690184 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd880039f0 con 0x7efd980c5010 2024-04-25T00:29:47.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.087+0000 7efd9e26d640 1 -- 172.21.15.116:0/482690184 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd980ab2b0 con 0x7efd980c5010 2024-04-25T00:29:47.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.087+0000 7efd9e26d640 1 -- 172.21.15.116:0/482690184 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd98068a60 con 0x7efd980c5010 2024-04-25T00:29:47.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.087+0000 7efd957fa640 1 -- 172.21.15.116:0/482690184 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7efd8801a070 con 0x7efd980c5010 2024-04-25T00:29:47.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.087+0000 7efd957fa640 1 --2- 172.21.15.116:0/482690184 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efd780678c0 0x7efd78069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:47.093 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.091+0000 7efd957fa640 1 -- 172.21.15.116:0/482690184 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 10450+0+0 (secure 0 0 0) 0x7efd880861d0 con 0x7efd980c5010 2024-04-25T00:29:47.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.091+0000 7efd8f7fe640 1 --2- 172.21.15.116:0/482690184 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efd780678c0 0x7efd78069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:47.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.091+0000 7efd8f7fe640 1 --2- 172.21.15.116:0/482690184 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efd780678c0 0x7efd78069d80 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7efd800ae000 tx=0x7efd800a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:29:47.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.095+0000 7efd957fa640 1 -- 172.21.15.116:0/482690184 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7efd88055470 con 0x7efd980c5010 2024-04-25T00:29:47.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.383+0000 7efd9e26d640 1 -- 172.21.15.116:0/482690184 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efd980c24e0 con 0x7efd980c5010 2024-04-25T00:29:47.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.383+0000 7efd957fa640 1 -- 172.21.15.116:0/482690184 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 232 v232) ==== 95+0+6509 (secure 0 0 0) 0x7efd88059770 con 0x7efd980c5010 2024-04-25T00:29:47.387 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:29:47.387 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":232,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":8847,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/2662028892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2662028892},{"type":"v1","addr":"172.21.15.116:6833","nonce":2662028892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":232}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:27:40.040636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6641},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6641":{"gid":6641,"name":"d","rank":0,"incarnation":223,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/2915274073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2915274073},{"type":"v1","addr":"172.21.15.174:6837","nonce":2915274073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6641,"qdb_cluster":[6641]},"id":1},{"mdsmap":{"epoch":231,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:28:53.808539+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8772":{"gid":8772,"name":"a","rank":0,"incarnation":228,"state":"up:active","state_seq":20,"addr":"172.21.15.116:6835/3499878473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3499878473},{"type":"v1","addr":"172.21.15.116:6835","nonce":3499878473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8772,"qdb_cluster":[8772]},"id":2}]} 2024-04-25T00:29:47.387 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 232 2024-04-25T00:29:47.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.383+0000 7efd9e26d640 1 -- 172.21.15.116:0/482690184 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efd780678c0 msgr2=0x7efd78069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:47.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.383+0000 7efd9e26d640 1 --2- 172.21.15.116:0/482690184 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efd780678c0 0x7efd78069d80 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7efd800ae000 tx=0x7efd800a7000 comp rx=0 tx=0).stop 2024-04-25T00:29:47.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.383+0000 7efd9e26d640 1 -- 172.21.15.116:0/482690184 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd980c5010 msgr2=0x7efd981ee700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:47.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.383+0000 7efd9e26d640 1 --2- 172.21.15.116:0/482690184 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd980c5010 0x7efd981ee700 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7efd8800c440 tx=0x7efd8800c900 comp rx=0 tx=0).stop 2024-04-25T00:29:47.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.383+0000 7efd9e26d640 1 -- 172.21.15.116:0/482690184 shutdown_connections 2024-04-25T00:29:47.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.383+0000 7efd9e26d640 1 --2- 172.21.15.116:0/482690184 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efd780678c0 0x7efd78069d80 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:47.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.387+0000 7efd9e26d640 1 --2- 172.21.15.116:0/482690184 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efd981599a0 0x7efd980ab7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:47.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.387+0000 7efd9e26d640 1 --2- 172.21.15.116:0/482690184 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efd98152e60 0x7efd98150d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:47.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.387+0000 7efd9e26d640 1 --2- 172.21.15.116:0/482690184 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd980c5010 0x7efd981ee700 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:47.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.387+0000 7efd9e26d640 1 -- 172.21.15.116:0/482690184 >> 172.21.15.116:0/482690184 conn(0x7efd9806c720 msgr2=0x7efd98154df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:29:47.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.387+0000 7efd9e26d640 1 -- 172.21.15.116:0/482690184 shutdown_connections 2024-04-25T00:29:47.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:47.387+0000 7efd9e26d640 1 -- 172.21.15.116:0/482690184 wait complete. 2024-04-25T00:29:47.410 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:29:49.411 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:29:49.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.567+0000 7f8744958640 1 Processor -- start 2024-04-25T00:29:49.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.567+0000 7f8744958640 1 -- start start 2024-04-25T00:29:49.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.567+0000 7f8744958640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f87400c2170 0x7f874015a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:49.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.567+0000 7f8744958640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f87400c62d0 0x7f87400c6720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:49.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.567+0000 7f8744958640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87400c3bd0 con 0x7f874006c480 2024-04-25T00:29:49.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.567+0000 7f8744958640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f874015ac60 con 0x7f87400c62d0 2024-04-25T00:29:49.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.567+0000 7f873ed9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f87400c62d0 0x7f87400c6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:49.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.567+0000 7f873ed9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f87400c62d0 0x7f87400c6720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42296/0 (socket says 172.21.15.116:42296) 2024-04-25T00:29:49.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.567+0000 7f873ed9d640 1 -- 172.21.15.116:0/536296228 learned_addr learned my addr 172.21.15.116:0/536296228 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:29:49.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.567+0000 7f8744958640 1 -- 172.21.15.116:0/536296228 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f87400c39a0 con 0x7f87400c2170 2024-04-25T00:29:49.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.567+0000 7f873dd9b640 1 --2- 172.21.15.116:0/536296228 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f87400c2170 0x7f874015a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:49.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.567+0000 7f873ed9d640 1 -- 172.21.15.116:0/536296228 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f87400c2170 msgr2=0x7f874015a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:29:49.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.567+0000 7f873ed9d640 1 --2- 172.21.15.116:0/536296228 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f87400c2170 0x7f874015a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:49.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.567+0000 7f873ed9d640 1 -- 172.21.15.116:0/536296228 >> v1:172.21.15.174:6789/0 conn(0x7f874006c480 legacy=0x7f8740157060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:29:49.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.567+0000 7f873ed9d640 1 -- 172.21.15.116:0/536296228 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87400c5dd0 con 0x7f87400c62d0 2024-04-25T00:29:49.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.571+0000 7f873ed9d640 1 --2- 172.21.15.116:0/536296228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87400c62d0 0x7f87400c6720 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f87400c3da0 tx=0x7f873801c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48a5786cac75a5fa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:29:49.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.571+0000 7f873d59a640 1 -- 172.21.15.116:0/536296228 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87380a56d0 con 0x7f87400c62d0 2024-04-25T00:29:49.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.571+0000 7f873d59a640 1 -- 172.21.15.116:0/536296228 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87380a5870 con 0x7f87400c62d0 2024-04-25T00:29:49.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.571+0000 7f873d59a640 1 -- 172.21.15.116:0/536296228 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87380a5c40 con 0x7f87400c62d0 2024-04-25T00:29:49.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.571+0000 7f8744958640 1 -- 172.21.15.116:0/536296228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87400c62d0 msgr2=0x7f87400c6720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:49.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.571+0000 7f8744958640 1 --2- 172.21.15.116:0/536296228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87400c62d0 0x7f87400c6720 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f87400c3da0 tx=0x7f873801c960 comp rx=0 tx=0).stop 2024-04-25T00:29:49.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.571+0000 7f8744958640 1 -- 172.21.15.116:0/536296228 shutdown_connections 2024-04-25T00:29:49.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.571+0000 7f8744958640 1 --2- 172.21.15.116:0/536296228 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f87400c62d0 0x7f87400c6720 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:49.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.571+0000 7f8744958640 1 --2- 172.21.15.116:0/536296228 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f87400c2170 0x7f874015a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:49.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.571+0000 7f8744958640 1 -- 172.21.15.116:0/536296228 >> 172.21.15.116:0/536296228 conn(0x7f87400c4270 msgr2=0x7f87400c4680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:29:49.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.571+0000 7f8744958640 1 -- 172.21.15.116:0/536296228 shutdown_connections 2024-04-25T00:29:49.576 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.571+0000 7f8744958640 1 -- 172.21.15.116:0/536296228 wait complete. 2024-04-25T00:29:49.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.571+0000 7f8744958640 1 Processor -- start 2024-04-25T00:29:49.577 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.571+0000 7f8744958640 1 -- start start 2024-04-25T00:29:49.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.575+0000 7f8744958640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f874006c480 0x7f87401450d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:49.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.575+0000 7f8744958640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f87400c2170 0x7f8740157a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:49.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.575+0000 7f8744958640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f87400c62d0 0x7f87401fe6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:49.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.575+0000 7f8744958640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f874006c9e0 con 0x7f874006c480 2024-04-25T00:29:49.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.575+0000 7f8744958640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f874006c860 con 0x7f87400c62d0 2024-04-25T00:29:49.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.575+0000 7f8744958640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f8740164db0 con 0x7f87400c2170 2024-04-25T00:29:49.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.575+0000 7f873e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f874006c480 0x7f87401450d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:49.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.575+0000 7f873e59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f874006c480 0x7f87401450d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42298/0 (socket says 172.21.15.116:42298) 2024-04-25T00:29:49.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.575+0000 7f873e59c640 1 -- 172.21.15.116:0/3374655653 learned_addr learned my addr 172.21.15.116:0/3374655653 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:29:49.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.575+0000 7f873dd9b640 1 --2- 172.21.15.116:0/3374655653 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f87400c2170 0x7f8740157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:49.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.575+0000 7f873ed9d640 1 --2- 172.21.15.116:0/3374655653 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f87400c62d0 0x7f87401fe6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:49.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.575+0000 7f873e59c640 1 -- 172.21.15.116:0/3374655653 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f87400c2170 msgr2=0x7f8740157a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:49.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.575+0000 7f873e59c640 1 --2- 172.21.15.116:0/3374655653 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f87400c2170 0x7f8740157a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:49.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.575+0000 7f873e59c640 1 -- 172.21.15.116:0/3374655653 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f87400c62d0 msgr2=0x7f87401fe6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:49.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.575+0000 7f873e59c640 1 --2- 172.21.15.116:0/3374655653 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f87400c62d0 0x7f87401fe6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:49.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.575+0000 7f873e59c640 1 -- 172.21.15.116:0/3374655653 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8740146e20 con 0x7f874006c480 2024-04-25T00:29:49.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.575+0000 7f873e59c640 1 --2- 172.21.15.116:0/3374655653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f874006c480 0x7f87401450d0 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f873000c4c0 tx=0x7f8730001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:29:49.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.575+0000 7f872effd640 1 -- 172.21.15.116:0/3374655653 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8730017070 con 0x7f874006c480 2024-04-25T00:29:49.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.575+0000 7f872effd640 1 -- 172.21.15.116:0/3374655653 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8730003600 con 0x7f874006c480 2024-04-25T00:29:49.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.575+0000 7f8744958640 1 -- 172.21.15.116:0/3374655653 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8740147100 con 0x7f874006c480 2024-04-25T00:29:49.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.575+0000 7f872effd640 1 -- 172.21.15.116:0/3374655653 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87300039f0 con 0x7f874006c480 2024-04-25T00:29:49.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.575+0000 7f8744958640 1 -- 172.21.15.116:0/3374655653 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87401476c0 con 0x7f874006c480 2024-04-25T00:29:49.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.575+0000 7f8744958640 1 -- 172.21.15.116:0/3374655653 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8740068a60 con 0x7f874006c480 2024-04-25T00:29:49.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.575+0000 7f872effd640 1 -- 172.21.15.116:0/3374655653 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f873001a070 con 0x7f874006c480 2024-04-25T00:29:49.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.579+0000 7f872effd640 1 --2- 172.21.15.116:0/3374655653 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f87200678c0 0x7f8720069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:49.581 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.579+0000 7f872effd640 1 -- 172.21.15.116:0/3374655653 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 10450+0+0 (secure 0 0 0) 0x7f87300861b0 con 0x7f874006c480 2024-04-25T00:29:49.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.583+0000 7f872effd640 1 -- 172.21.15.116:0/3374655653 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f8730055450 con 0x7f874006c480 2024-04-25T00:29:49.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.587+0000 7f873dd9b640 1 --2- 172.21.15.116:0/3374655653 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f87200678c0 0x7f8720069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:49.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.587+0000 7f873dd9b640 1 --2- 172.21.15.116:0/3374655653 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f87200678c0 0x7f8720069d80 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f8740159d10 tx=0x7f8734009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:29:49.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.871+0000 7f8744958640 1 -- 172.21.15.116:0/3374655653 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8740073630 con 0x7f874006c480 2024-04-25T00:29:49.875 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.871+0000 7f872effd640 1 -- 172.21.15.116:0/3374655653 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 232 v232) ==== 95+0+6509 (secure 0 0 0) 0x7f8730059750 con 0x7f874006c480 2024-04-25T00:29:49.875 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:29:49.875 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":232,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":8847,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6833/2662028892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2662028892},{"type":"v1","addr":"172.21.15.116:6833","nonce":2662028892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":232}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:27:40.040636+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6641},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6641":{"gid":6641,"name":"d","rank":0,"incarnation":223,"state":"up:active","state_seq":7,"addr":"172.21.15.174:6837/2915274073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":2915274073},{"type":"v1","addr":"172.21.15.174:6837","nonce":2915274073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6641,"qdb_cluster":[6641]},"id":1},{"mdsmap":{"epoch":231,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:28:53.808539+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8772":{"gid":8772,"name":"a","rank":0,"incarnation":228,"state":"up:active","state_seq":20,"addr":"172.21.15.116:6835/3499878473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3499878473},{"type":"v1","addr":"172.21.15.116:6835","nonce":3499878473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8772,"qdb_cluster":[8772]},"id":2}]} 2024-04-25T00:29:49.876 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 232 2024-04-25T00:29:49.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.875+0000 7f8744958640 1 -- 172.21.15.116:0/3374655653 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f87200678c0 msgr2=0x7f8720069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:49.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.875+0000 7f8744958640 1 --2- 172.21.15.116:0/3374655653 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f87200678c0 0x7f8720069d80 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f8740159d10 tx=0x7f8734009190 comp rx=0 tx=0).stop 2024-04-25T00:29:49.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.875+0000 7f8744958640 1 -- 172.21.15.116:0/3374655653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f874006c480 msgr2=0x7f87401450d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:49.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.875+0000 7f8744958640 1 --2- 172.21.15.116:0/3374655653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f874006c480 0x7f87401450d0 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f873000c4c0 tx=0x7f8730001e80 comp rx=0 tx=0).stop 2024-04-25T00:29:49.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.875+0000 7f8744958640 1 -- 172.21.15.116:0/3374655653 shutdown_connections 2024-04-25T00:29:49.877 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.875+0000 7f8744958640 1 --2- 172.21.15.116:0/3374655653 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f87200678c0 0x7f8720069d80 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:49.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.875+0000 7f8744958640 1 --2- 172.21.15.116:0/3374655653 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f87400c62d0 0x7f87401fe6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:49.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.875+0000 7f8744958640 1 --2- 172.21.15.116:0/3374655653 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f87400c2170 0x7f8740157a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:49.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.875+0000 7f8744958640 1 --2- 172.21.15.116:0/3374655653 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f874006c480 0x7f87401450d0 unknown :-1 s=CLOSED pgs=1681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:49.878 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.875+0000 7f8744958640 1 -- 172.21.15.116:0/3374655653 >> 172.21.15.116:0/3374655653 conn(0x7f87400c4270 msgr2=0x7f87400a76a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:29:49.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.879+0000 7f8744958640 1 -- 172.21.15.116:0/3374655653 shutdown_connections 2024-04-25T00:29:49.882 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:49.879+0000 7f8744958640 1 -- 172.21.15.116:0/3374655653 wait complete. 2024-04-25T00:29:49.901 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2024-04-25T00:29:51.903 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:29:52.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.039+0000 7efd710ea640 1 Processor -- start 2024-04-25T00:29:52.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.039+0000 7efd710ea640 1 -- start start 2024-04-25T00:29:52.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.039+0000 7efd710ea640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7efd6c152e60 0x7efd6c1599a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:52.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.039+0000 7efd710ea640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efd6c1562d0 0x7efd6c0c36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:52.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.039+0000 7efd710ea640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7efd6c0c3c20 0x7efd6c0c5010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:52.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.039+0000 7efd710ea640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7efd6c153240 con 0x7efd6c152e60 2024-04-25T00:29:52.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.039+0000 7efd710ea640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7efd6c0734e0 con 0x7efd6c0c3c20 2024-04-25T00:29:52.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.039+0000 7efd710ea640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7efd6c156730 con 0x7efd6c1562d0 2024-04-25T00:29:52.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.039+0000 7efd6ad9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7efd6c152e60 0x7efd6c1599a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:52.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.043+0000 7efd6a59c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efd6c1562d0 0x7efd6c0c36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:52.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.043+0000 7efd6ad9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7efd6c152e60 0x7efd6c1599a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42314/0 (socket says 172.21.15.116:42314) 2024-04-25T00:29:52.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.043+0000 7efd6ad9d640 1 -- 172.21.15.116:0/308045979 learned_addr learned my addr 172.21.15.116:0/308045979 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:29:52.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.043+0000 7efd6b59e640 1 --2- 172.21.15.116:0/308045979 >> v2:172.21.15.174:3300/0 conn(0x7efd6c0c3c20 0x7efd6c0c5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:52.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.043+0000 7efd6a59c640 1 -- 172.21.15.116:0/308045979 >> v2:172.21.15.174:3300/0 conn(0x7efd6c0c3c20 msgr2=0x7efd6c0c5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:29:52.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.043+0000 7efd6a59c640 1 --2- 172.21.15.116:0/308045979 >> v2:172.21.15.174:3300/0 conn(0x7efd6c0c3c20 0x7efd6c0c5010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:52.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.043+0000 7efd6a59c640 1 -- 172.21.15.116:0/308045979 >> v2:172.21.15.116:3300/0 conn(0x7efd6c152e60 msgr2=0x7efd6c1599a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:29:52.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.043+0000 7efd6a59c640 1 --2- 172.21.15.116:0/308045979 >> v2:172.21.15.116:3300/0 conn(0x7efd6c152e60 0x7efd6c1599a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:52.046 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.043+0000 7efd6a59c640 1 -- 172.21.15.116:0/308045979 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd6c073200 con 0x7efd6c1562d0 2024-04-25T00:29:52.049 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.047+0000 7efd6a59c640 1 --2- 172.21.15.116:0/308045979 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efd6c1562d0 0x7efd6c0c36e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7efd6c072ac0 tx=0x7efd6001c410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb56d65816cd1308 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:29:52.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.047+0000 7efd69d9b640 1 -- 172.21.15.116:0/308045979 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd600b0070 con 0x7efd6c1562d0 2024-04-25T00:29:52.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.047+0000 7efd69d9b640 1 -- 172.21.15.116:0/308045979 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd600ab600 con 0x7efd6c1562d0 2024-04-25T00:29:52.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.047+0000 7efd69d9b640 1 -- 172.21.15.116:0/308045979 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd600ab9d0 con 0x7efd6c1562d0 2024-04-25T00:29:52.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.047+0000 7efd710ea640 1 -- 172.21.15.116:0/308045979 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efd6c1562d0 msgr2=0x7efd6c0c36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:52.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.047+0000 7efd710ea640 1 --2- 172.21.15.116:0/308045979 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efd6c1562d0 0x7efd6c0c36e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7efd6c072ac0 tx=0x7efd6001c410 comp rx=0 tx=0).stop 2024-04-25T00:29:52.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.051+0000 7efd710ea640 1 -- 172.21.15.116:0/308045979 shutdown_connections 2024-04-25T00:29:52.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.051+0000 7efd710ea640 1 --2- 172.21.15.116:0/308045979 >> v2:172.21.15.174:3300/0 conn(0x7efd6c0c3c20 0x7efd6c0c5010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:52.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.051+0000 7efd710ea640 1 --2- 172.21.15.116:0/308045979 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efd6c1562d0 0x7efd6c0c36e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:52.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.051+0000 7efd710ea640 1 --2- 172.21.15.116:0/308045979 >> v2:172.21.15.116:3300/0 conn(0x7efd6c152e60 0x7efd6c1599a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:52.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.051+0000 7efd710ea640 1 -- 172.21.15.116:0/308045979 >> 172.21.15.116:0/308045979 conn(0x7efd6c06c720 msgr2=0x7efd6c06cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:29:52.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.051+0000 7efd710ea640 1 -- 172.21.15.116:0/308045979 shutdown_connections 2024-04-25T00:29:52.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.051+0000 7efd710ea640 1 -- 172.21.15.116:0/308045979 wait complete. 2024-04-25T00:29:52.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.055+0000 7efd710ea640 1 Processor -- start 2024-04-25T00:29:52.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.055+0000 7efd710ea640 1 -- start start 2024-04-25T00:29:52.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.055+0000 7efd710ea640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efd6c0c3c20 0x7efd6c14a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:52.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.055+0000 7efd710ea640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd6c152e60 0x7efd6c0abd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:52.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.055+0000 7efd710ea640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efd6c1562d0 0x7efd6c14b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:52.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.055+0000 7efd710ea640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7efd6c072e20 con 0x7efd6c152e60 2024-04-25T00:29:52.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.055+0000 7efd710ea640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7efd6c072080 con 0x7efd6c0c3c20 2024-04-25T00:29:52.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.055+0000 7efd710ea640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7efd6c072fa0 con 0x7efd6c1562d0 2024-04-25T00:29:52.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.055+0000 7efd6a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd6c152e60 0x7efd6c0abd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:52.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.055+0000 7efd6a59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd6c152e60 0x7efd6c0abd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:42330/0 (socket says 172.21.15.116:42330) 2024-04-25T00:29:52.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.055+0000 7efd6a59c640 1 -- 172.21.15.116:0/1459317168 learned_addr learned my addr 172.21.15.116:0/1459317168 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:29:52.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.055+0000 7efd6b59e640 1 --2- 172.21.15.116:0/1459317168 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efd6c1562d0 0x7efd6c14b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:52.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.055+0000 7efd6ad9d640 1 --2- 172.21.15.116:0/1459317168 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efd6c0c3c20 0x7efd6c14a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:52.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.055+0000 7efd6a59c640 1 -- 172.21.15.116:0/1459317168 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efd6c1562d0 msgr2=0x7efd6c14b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:52.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.055+0000 7efd6a59c640 1 --2- 172.21.15.116:0/1459317168 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efd6c1562d0 0x7efd6c14b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:52.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.055+0000 7efd6a59c640 1 -- 172.21.15.116:0/1459317168 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efd6c0c3c20 msgr2=0x7efd6c14a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:52.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.055+0000 7efd6a59c640 1 --2- 172.21.15.116:0/1459317168 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efd6c0c3c20 0x7efd6c14a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:52.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.055+0000 7efd6a59c640 1 -- 172.21.15.116:0/1459317168 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd6c1f25a0 con 0x7efd6c152e60 2024-04-25T00:29:52.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.055+0000 7efd6a59c640 1 --2- 172.21.15.116:0/1459317168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd6c152e60 0x7efd6c0abd50 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7efd6001ccb0 tx=0x7efd6001ac50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:29:52.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.055+0000 7efd537fe640 1 -- 172.21.15.116:0/1459317168 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd600b0050 con 0x7efd6c152e60 2024-04-25T00:29:52.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.055+0000 7efd537fe640 1 -- 172.21.15.116:0/1459317168 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd600bc070 con 0x7efd6c152e60 2024-04-25T00:29:52.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.055+0000 7efd537fe640 1 -- 172.21.15.116:0/1459317168 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd600b5830 con 0x7efd6c152e60 2024-04-25T00:29:52.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.055+0000 7efd710ea640 1 -- 172.21.15.116:0/1459317168 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd6c1f2880 con 0x7efd6c152e60 2024-04-25T00:29:52.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.055+0000 7efd710ea640 1 -- 172.21.15.116:0/1459317168 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd6c157eb0 con 0x7efd6c152e60 2024-04-25T00:29:52.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.059+0000 7efd710ea640 1 -- 172.21.15.116:0/1459317168 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd6c068a60 con 0x7efd6c152e60 2024-04-25T00:29:52.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.059+0000 7efd537fe640 1 -- 172.21.15.116:0/1459317168 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7efd600b59d0 con 0x7efd6c152e60 2024-04-25T00:29:52.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.059+0000 7efd537fe640 1 --2- 172.21.15.116:0/1459317168 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efd580678f0 0x7efd58069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:52.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.059+0000 7efd537fe640 1 -- 172.21.15.116:0/1459317168 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 10536+0+0 (secure 0 0 0) 0x7efd60121610 con 0x7efd6c152e60 2024-04-25T00:29:52.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.059+0000 7efd6ad9d640 1 --2- 172.21.15.116:0/1459317168 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efd580678f0 0x7efd58069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:52.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.059+0000 7efd6ad9d640 1 --2- 172.21.15.116:0/1459317168 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efd580678f0 0x7efd58069db0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7efd6c0c45c0 tx=0x7efd5c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:29:52.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.063+0000 7efd537fe640 1 -- 172.21.15.116:0/1459317168 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7efd60016090 con 0x7efd6c152e60 2024-04-25T00:29:52.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.327+0000 7efd710ea640 1 -- 172.21.15.116:0/1459317168 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efd6c15ff00 con 0x7efd6c152e60 2024-04-25T00:29:52.333 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.331+0000 7efd537fe640 1 -- 172.21.15.116:0/1459317168 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 234 v234) ==== 95+0+5677 (secure 0 0 0) 0x7efd600f0eb0 con 0x7efd6c152e60 2024-04-25T00:29:52.334 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:29:52.334 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":234,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":234,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:29:51.644624+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8847},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8847":{"gid":8847,"name":"c","rank":0,"incarnation":233,"state":"up:reconnect","state_seq":15,"addr":"172.21.15.116:6833/2662028892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2662028892},{"type":"v1","addr":"172.21.15.116:6833","nonce":2662028892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":231,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:28:53.808539+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8772":{"gid":8772,"name":"a","rank":0,"incarnation":228,"state":"up:active","state_seq":20,"addr":"172.21.15.116:6835/3499878473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3499878473},{"type":"v1","addr":"172.21.15.116:6835","nonce":3499878473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8772,"qdb_cluster":[8772]},"id":2}]} 2024-04-25T00:29:52.334 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 234 2024-04-25T00:29:52.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.331+0000 7efd710ea640 1 -- 172.21.15.116:0/1459317168 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efd580678f0 msgr2=0x7efd58069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:52.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.331+0000 7efd710ea640 1 --2- 172.21.15.116:0/1459317168 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efd580678f0 0x7efd58069db0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7efd6c0c45c0 tx=0x7efd5c008040 comp rx=0 tx=0).stop 2024-04-25T00:29:52.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.331+0000 7efd710ea640 1 -- 172.21.15.116:0/1459317168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd6c152e60 msgr2=0x7efd6c0abd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:52.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.331+0000 7efd710ea640 1 --2- 172.21.15.116:0/1459317168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd6c152e60 0x7efd6c0abd50 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7efd6001ccb0 tx=0x7efd6001ac50 comp rx=0 tx=0).stop 2024-04-25T00:29:52.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.331+0000 7efd710ea640 1 -- 172.21.15.116:0/1459317168 shutdown_connections 2024-04-25T00:29:52.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.331+0000 7efd710ea640 1 --2- 172.21.15.116:0/1459317168 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efd580678f0 0x7efd58069db0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:52.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.331+0000 7efd710ea640 1 --2- 172.21.15.116:0/1459317168 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efd6c1562d0 0x7efd6c14b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:52.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.331+0000 7efd710ea640 1 --2- 172.21.15.116:0/1459317168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efd6c152e60 0x7efd6c0abd50 unknown :-1 s=CLOSED pgs=1682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:52.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.331+0000 7efd710ea640 1 --2- 172.21.15.116:0/1459317168 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efd6c0c3c20 0x7efd6c14a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:52.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.331+0000 7efd710ea640 1 -- 172.21.15.116:0/1459317168 >> 172.21.15.116:0/1459317168 conn(0x7efd6c06c720 msgr2=0x7efd6c1539b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:29:52.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.331+0000 7efd710ea640 1 -- 172.21.15.116:0/1459317168 shutdown_connections 2024-04-25T00:29:52.336 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.331+0000 7efd710ea640 1 -- 172.21.15.116:0/1459317168 wait complete. 2024-04-25T00:29:52.354 INFO:tasks.mds_thrash.fs.[a]:mds.d down, removed from mdsmap 2024-04-25T00:29:52.354 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-04-25T00:29:52.355 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:29:52.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.499+0000 7fd992329640 1 Processor -- start 2024-04-25T00:29:52.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.499+0000 7fd992329640 1 -- start start 2024-04-25T00:29:52.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.499+0000 7fd992329640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd98c06bc20 0x7fd98c1541f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:52.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.499+0000 7fd992329640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd98c0734b0 con 0x7fd98c150d10 2024-04-25T00:29:52.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.499+0000 7fd992329640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd98c153750 con 0x7fd98c0c2b10 2024-04-25T00:29:52.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.499+0000 7fd992329640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fd98c0a9bb0 con 0x7fd98c06bc20 2024-04-25T00:29:52.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.499+0000 7fd98bfff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fd98c150d10 0x7fd98c0a94a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:43200/0 (socket says 172.21.15.116:43200) 2024-04-25T00:29:52.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.499+0000 7fd98bfff640 1 -- 172.21.15.116:0/3783006570 learned_addr learned my addr 172.21.15.116:0/3783006570 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:29:52.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.499+0000 7fd9908c6640 1 --2- 172.21.15.116:0/3783006570 >> v2:172.21.15.116:3300/0 conn(0x7fd98c06bc20 0x7fd98c1541f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:52.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.499+0000 7fd98affd640 1 -- 172.21.15.116:0/3783006570 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1895420568 0 0) 0x7fd98c0734b0 con 0x7fd98c150d10 2024-04-25T00:29:52.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.503+0000 7fd98affd640 1 -- 172.21.15.116:0/3783006570 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd974003530 con 0x7fd98c150d10 2024-04-25T00:29:52.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.503+0000 7fd9908c6640 1 -- 172.21.15.116:0/3783006570 >> v1:172.21.15.174:6789/0 conn(0x7fd98c0c2b10 legacy=0x7fd98c0c3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:29:52.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.503+0000 7fd9908c6640 1 -- 172.21.15.116:0/3783006570 >> v1:172.21.15.116:6789/0 conn(0x7fd98c150d10 legacy=0x7fd98c0a94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:52.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.503+0000 7fd9908c6640 1 -- 172.21.15.116:0/3783006570 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd98c0731d0 con 0x7fd98c06bc20 2024-04-25T00:29:52.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.503+0000 7fd9908c6640 1 --2- 172.21.15.116:0/3783006570 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd98c06bc20 0x7fd98c1541f0 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7fd984009d90 tx=0x7fd98401c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbf8704bf74b2e90 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:29:52.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.503+0000 7fd98affd640 1 -- 172.21.15.116:0/3783006570 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9840b2070 con 0x7fd98c06bc20 2024-04-25T00:29:52.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.503+0000 7fd98affd640 1 -- 172.21.15.116:0/3783006570 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9840a57c0 con 0x7fd98c06bc20 2024-04-25T00:29:52.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.503+0000 7fd98affd640 1 -- 172.21.15.116:0/3783006570 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9840a5b90 con 0x7fd98c06bc20 2024-04-25T00:29:52.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.503+0000 7fd992329640 1 -- 172.21.15.116:0/3783006570 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd98c06bc20 msgr2=0x7fd98c1541f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:52.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.503+0000 7fd992329640 1 --2- 172.21.15.116:0/3783006570 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd98c06bc20 0x7fd98c1541f0 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7fd984009d90 tx=0x7fd98401c810 comp rx=0 tx=0).stop 2024-04-25T00:29:52.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.503+0000 7fd992329640 1 -- 172.21.15.116:0/3783006570 shutdown_connections 2024-04-25T00:29:52.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.503+0000 7fd992329640 1 --2- 172.21.15.116:0/3783006570 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd98c06bc20 0x7fd98c1541f0 unknown :-1 s=CLOSED pgs=1684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:52.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.503+0000 7fd992329640 1 -- 172.21.15.116:0/3783006570 >> 172.21.15.116:0/3783006570 conn(0x7fd98c0c4c10 msgr2=0x7fd98c0c5020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:29:52.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.503+0000 7fd992329640 1 -- 172.21.15.116:0/3783006570 shutdown_connections 2024-04-25T00:29:52.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.503+0000 7fd992329640 1 -- 172.21.15.116:0/3783006570 wait complete. 2024-04-25T00:29:52.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.507+0000 7fd992329640 1 Processor -- start 2024-04-25T00:29:52.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.507+0000 7fd992329640 1 -- start start 2024-04-25T00:29:52.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.507+0000 7fd992329640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd98c06bc20 0x7fd98c145ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:52.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.507+0000 7fd992329640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd98c0c2b10 0x7fd98c0c6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:52.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.507+0000 7fd992329640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd98c150d10 0x7fd98c1ee550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:52.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.507+0000 7fd992329640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd98c0c3af0 con 0x7fd98c150d10 2024-04-25T00:29:52.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.507+0000 7fd992329640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd98c0c3970 con 0x7fd98c06bc20 2024-04-25T00:29:52.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.507+0000 7fd992329640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fd98c1510f0 con 0x7fd98c0c2b10 2024-04-25T00:29:52.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.507+0000 7fd9908c6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd98c150d10 0x7fd98c1ee550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:52.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.507+0000 7fd98b7fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd98c0c2b10 0x7fd98c0c6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:52.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.507+0000 7fd98b7fe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd98c0c2b10 0x7fd98c0c6fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:45990/0 (socket says 172.21.15.116:45990) 2024-04-25T00:29:52.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.507+0000 7fd98b7fe640 1 -- 172.21.15.116:0/1984192329 learned_addr learned my addr 172.21.15.116:0/1984192329 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:29:52.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.507+0000 7fd9908c6640 1 -- 172.21.15.116:0/1984192329 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd98c0c2b10 msgr2=0x7fd98c0c6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:52.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.507+0000 7fd9908c6640 1 --2- 172.21.15.116:0/1984192329 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd98c0c2b10 0x7fd98c0c6fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:52.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.507+0000 7fd9908c6640 1 -- 172.21.15.116:0/1984192329 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd98c06bc20 msgr2=0x7fd98c145ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:52.513 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.507+0000 7fd9908c6640 1 --2- 172.21.15.116:0/1984192329 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd98c06bc20 0x7fd98c145ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:52.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.507+0000 7fd9908c6640 1 -- 172.21.15.116:0/1984192329 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd98c146010 con 0x7fd98c150d10 2024-04-25T00:29:52.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.507+0000 7fd9908c6640 1 --2- 172.21.15.116:0/1984192329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd98c150d10 0x7fd98c1ee550 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7fd9840ac000 tx=0x7fd9840195b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:29:52.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.507+0000 7fd988ff9640 1 -- 172.21.15.116:0/1984192329 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9840b2050 con 0x7fd98c150d10 2024-04-25T00:29:52.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.507+0000 7fd988ff9640 1 -- 172.21.15.116:0/1984192329 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9840b3040 con 0x7fd98c150d10 2024-04-25T00:29:52.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.511+0000 7fd992329640 1 -- 172.21.15.116:0/1984192329 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd98c1ed010 con 0x7fd98c150d10 2024-04-25T00:29:52.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.511+0000 7fd988ff9640 1 -- 172.21.15.116:0/1984192329 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9840ad650 con 0x7fd98c150d10 2024-04-25T00:29:52.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.511+0000 7fd992329640 1 -- 172.21.15.116:0/1984192329 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd98c143b30 con 0x7fd98c150d10 2024-04-25T00:29:52.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.511+0000 7fd9727fc640 1 -- 172.21.15.116:0/1984192329 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd98c068a60 con 0x7fd98c150d10 2024-04-25T00:29:52.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.511+0000 7fd988ff9640 1 -- 172.21.15.116:0/1984192329 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd9840c0020 con 0x7fd98c150d10 2024-04-25T00:29:52.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.511+0000 7fd988ff9640 1 --2- 172.21.15.116:0/1984192329 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd9780678c0 0x7fd978069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:52.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.511+0000 7fd988ff9640 1 -- 172.21.15.116:0/1984192329 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 10536+0+0 (secure 0 0 0) 0x7fd984121500 con 0x7fd98c150d10 2024-04-25T00:29:52.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.511+0000 7fd98bfff640 1 --2- 172.21.15.116:0/1984192329 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd9780678c0 0x7fd978069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:52.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.515+0000 7fd98bfff640 1 --2- 172.21.15.116:0/1984192329 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd9780678c0 0x7fd978069d80 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fd97c007fd0 tx=0x7fd97c007900 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:29:52.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.519+0000 7fd988ff9640 1 -- 172.21.15.116:0/1984192329 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fd9840c7020 con 0x7fd98c150d10 2024-04-25T00:29:52.806 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.803+0000 7fd9727fc640 1 -- 172.21.15.116:0/1984192329 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd98c0c2540 con 0x7fd98c150d10 2024-04-25T00:29:52.807 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.803+0000 7fd988ff9640 1 -- 172.21.15.116:0/1984192329 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 234 v234) ==== 95+0+5677 (secure 0 0 0) 0x7fd98c1ed010 con 0x7fd98c150d10 2024-04-25T00:29:52.807 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:29:52.807 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":234,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":234,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:29:51.644624+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8847},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8847":{"gid":8847,"name":"c","rank":0,"incarnation":233,"state":"up:reconnect","state_seq":15,"addr":"172.21.15.116:6833/2662028892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2662028892},{"type":"v1","addr":"172.21.15.116:6833","nonce":2662028892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":231,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:28:53.808539+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8772":{"gid":8772,"name":"a","rank":0,"incarnation":228,"state":"up:active","state_seq":20,"addr":"172.21.15.116:6835/3499878473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3499878473},{"type":"v1","addr":"172.21.15.116:6835","nonce":3499878473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8772,"qdb_cluster":[8772]},"id":2}]} 2024-04-25T00:29:52.807 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 234 2024-04-25T00:29:52.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.803+0000 7fd9727fc640 1 -- 172.21.15.116:0/1984192329 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd9780678c0 msgr2=0x7fd978069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:52.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.803+0000 7fd9727fc640 1 --2- 172.21.15.116:0/1984192329 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd9780678c0 0x7fd978069d80 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fd97c007fd0 tx=0x7fd97c007900 comp rx=0 tx=0).stop 2024-04-25T00:29:52.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.803+0000 7fd9727fc640 1 -- 172.21.15.116:0/1984192329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd98c150d10 msgr2=0x7fd98c1ee550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:52.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.803+0000 7fd9727fc640 1 --2- 172.21.15.116:0/1984192329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd98c150d10 0x7fd98c1ee550 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7fd9840ac000 tx=0x7fd9840195b0 comp rx=0 tx=0).stop 2024-04-25T00:29:52.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.803+0000 7fd9727fc640 1 -- 172.21.15.116:0/1984192329 shutdown_connections 2024-04-25T00:29:52.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.803+0000 7fd9727fc640 1 --2- 172.21.15.116:0/1984192329 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd9780678c0 0x7fd978069d80 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:52.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.803+0000 7fd9727fc640 1 --2- 172.21.15.116:0/1984192329 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd98c150d10 0x7fd98c1ee550 unknown :-1 s=CLOSED pgs=1685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:52.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.803+0000 7fd9727fc640 1 --2- 172.21.15.116:0/1984192329 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd98c0c2b10 0x7fd98c0c6fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:52.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.803+0000 7fd9727fc640 1 --2- 172.21.15.116:0/1984192329 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd98c06bc20 0x7fd98c145ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:52.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.803+0000 7fd9727fc640 1 -- 172.21.15.116:0/1984192329 >> 172.21.15.116:0/1984192329 conn(0x7fd98c0c4c10 msgr2=0x7fd98c0c3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:29:52.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.807+0000 7fd9727fc640 1 -- 172.21.15.116:0/1984192329 shutdown_connections 2024-04-25T00:29:52.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:52.807+0000 7fd9727fc640 1 -- 172.21.15.116:0/1984192329 wait complete. 2024-04-25T00:29:52.832 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 234, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:04.678737+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 234, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_8847": { "addr": "172.21.15.116:6833/2662028892", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6832", "nonce": 2662028892, "type": "v2" }, { "addr": "172.21.15.116:6833", "nonce": 2662028892, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8847, "incarnation": 233, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:reconnect", "state_seq": 15 } }, "last_failure": 0, "last_failure_osd_epoch": 73, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T00:29:51.644624+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8847 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:18.825887+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 231, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_8772": { "addr": "172.21.15.116:6835/3499878473", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6834", "nonce": 3499878473, "type": "v2" }, { "addr": "172.21.15.116:6835", "nonce": 3499878473, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8772, "incarnation": 228, "join_fscid": -1, "name": "a", "rank": 0, "state": "up:active", "state_seq": 20 } }, "last_failure": 0, "last_failure_osd_epoch": 72, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-04-25T00:28:53.808539+0000", "qdb_cluster": [ 8772 ], "qdb_leader": 8772, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8772 } } } ], "standbys": [ { "addr": "172.21.15.174:6835/4112145782", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6834", "nonce": 4112145782, "type": "v2" }, { "addr": "172.21.15.174:6835", "nonce": 4112145782, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 112, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5582, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-25T00:29:54.833 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:29:54.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:54.971+0000 7f034a047640 1 Processor -- start 2024-04-25T00:29:54.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:54.971+0000 7f034a047640 1 -- start start 2024-04-25T00:29:54.974 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:54.971+0000 7f034a047640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0344072080 0x7f0344159950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:54.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:54.971+0000 7f034a047640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f03441562a0 0x7f03440c5020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:54.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:54.971+0000 7f034a047640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03440730b0 con 0x7f0344152d80 2024-04-25T00:29:54.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:54.971+0000 7f034a047640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f0344072e60 con 0x7f0344072080 2024-04-25T00:29:54.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:54.971+0000 7f034a047640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f0344073390 con 0x7f03441562a0 2024-04-25T00:29:54.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:54.971+0000 7f0342ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0344072080 0x7f0344159950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:54.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:54.971+0000 7f0343fff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f03441562a0 0x7f03440c5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:54.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:54.971+0000 7f0343fff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f03441562a0 0x7f03440c5020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:55280/0 (socket says 172.21.15.116:55280) 2024-04-25T00:29:54.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:54.971+0000 7f0343fff640 1 -- 172.21.15.116:0/4211068372 learned_addr learned my addr 172.21.15.116:0/4211068372 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:29:54.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:54.971+0000 7f0342ffd640 1 -- 172.21.15.116:0/4211068372 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f03441562a0 msgr2=0x7f03440c5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:29:54.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:54.971+0000 7f0342ffd640 1 --2- 172.21.15.116:0/4211068372 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f03441562a0 0x7f03440c5020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:54.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:54.971+0000 7f0342ffd640 1 -- 172.21.15.116:0/4211068372 >> v1:172.21.15.174:6789/0 conn(0x7f0344152d80 legacy=0x7f03440c36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:29:54.976 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:54.971+0000 7f0342ffd640 1 -- 172.21.15.116:0/4211068372 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0344158e70 con 0x7f0344072080 2024-04-25T00:29:54.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:54.975+0000 7f0342ffd640 1 --2- 172.21.15.116:0/4211068372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0344072080 0x7f0344159950 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f03380097c0 tx=0x7f033801c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0b16bc3dd723baf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:29:54.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:54.975+0000 7f03427fc640 1 -- 172.21.15.116:0/4211068372 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03380b2070 con 0x7f0344072080 2024-04-25T00:29:54.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:54.975+0000 7f03427fc640 1 -- 172.21.15.116:0/4211068372 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03380a9070 con 0x7f0344072080 2024-04-25T00:29:54.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:54.975+0000 7f03427fc640 1 -- 172.21.15.116:0/4211068372 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03380ad880 con 0x7f0344072080 2024-04-25T00:29:54.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:54.975+0000 7f034a047640 1 -- 172.21.15.116:0/4211068372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0344072080 msgr2=0x7f0344159950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:54.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:54.975+0000 7f034a047640 1 --2- 172.21.15.116:0/4211068372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0344072080 0x7f0344159950 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f03380097c0 tx=0x7f033801c3a0 comp rx=0 tx=0).stop 2024-04-25T00:29:54.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:54.975+0000 7f034a047640 1 -- 172.21.15.116:0/4211068372 shutdown_connections 2024-04-25T00:29:54.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:54.975+0000 7f034a047640 1 --2- 172.21.15.116:0/4211068372 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f03441562a0 0x7f03440c5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:54.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:54.975+0000 7f034a047640 1 --2- 172.21.15.116:0/4211068372 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0344072080 0x7f0344159950 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:54.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:54.975+0000 7f034a047640 1 -- 172.21.15.116:0/4211068372 >> 172.21.15.116:0/4211068372 conn(0x7f034406c720 msgr2=0x7f034406cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:29:54.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:54.975+0000 7f034a047640 1 -- 172.21.15.116:0/4211068372 shutdown_connections 2024-04-25T00:29:54.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:54.975+0000 7f034a047640 1 -- 172.21.15.116:0/4211068372 wait complete. 2024-04-25T00:29:54.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:54.975+0000 7f034a047640 1 Processor -- start 2024-04-25T00:29:54.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:54.975+0000 7f034a047640 1 -- start start 2024-04-25T00:29:54.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:54.979+0000 7f034a047640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0344072080 0x7f034414fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:54.982 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:54.979+0000 7f034a047640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0344152d80 0x7f0344149040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:54.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:54.979+0000 7f034a047640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03441562a0 0x7f03441478d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:54.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:54.979+0000 7f034a047640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0344153160 con 0x7f03441562a0 2024-04-25T00:29:54.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:54.979+0000 7f034a047640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f03441566b0 con 0x7f0344072080 2024-04-25T00:29:54.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:54.979+0000 7f034a047640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f03441532e0 con 0x7f0344152d80 2024-04-25T00:29:54.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:54.979+0000 7f0343fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03441562a0 0x7f03441478d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:54.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:54.979+0000 7f0343fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03441562a0 0x7f03441478d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41456/0 (socket says 172.21.15.116:41456) 2024-04-25T00:29:54.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:54.979+0000 7f0343fff640 1 -- 172.21.15.116:0/1500480109 learned_addr learned my addr 172.21.15.116:0/1500480109 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:29:54.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:54.979+0000 7f03437fe640 1 --2- 172.21.15.116:0/1500480109 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0344072080 0x7f034414fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:54.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:54.979+0000 7f0342ffd640 1 --2- 172.21.15.116:0/1500480109 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0344152d80 0x7f0344149040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:54.983 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:54.979+0000 7f03437fe640 1 -- 172.21.15.116:0/1500480109 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0344152d80 msgr2=0x7f0344149040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:54.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:54.979+0000 7f03437fe640 1 --2- 172.21.15.116:0/1500480109 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0344152d80 0x7f0344149040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:54.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:54.979+0000 7f03437fe640 1 -- 172.21.15.116:0/1500480109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03441562a0 msgr2=0x7f03441478d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:54.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:54.979+0000 7f03437fe640 1 --2- 172.21.15.116:0/1500480109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03441562a0 0x7f03441478d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:54.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:54.979+0000 7f03437fe640 1 -- 172.21.15.116:0/1500480109 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03440acb60 con 0x7f0344072080 2024-04-25T00:29:54.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:54.979+0000 7f03437fe640 1 --2- 172.21.15.116:0/1500480109 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0344072080 0x7f034414fd10 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f033400c390 tx=0x7f033400c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:29:54.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:54.979+0000 7f032bfff640 1 -- 172.21.15.116:0/1500480109 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0334017070 con 0x7f0344072080 2024-04-25T00:29:54.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:54.979+0000 7f032bfff640 1 -- 172.21.15.116:0/1500480109 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0334003600 con 0x7f0344072080 2024-04-25T00:29:54.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:54.979+0000 7f032bfff640 1 -- 172.21.15.116:0/1500480109 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03340039d0 con 0x7f0344072080 2024-04-25T00:29:54.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:54.979+0000 7f034a047640 1 -- 172.21.15.116:0/1500480109 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0344150400 con 0x7f0344072080 2024-04-25T00:29:54.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:54.979+0000 7f034a047640 1 -- 172.21.15.116:0/1500480109 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0344149580 con 0x7f0344072080 2024-04-25T00:29:54.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:54.979+0000 7f034a047640 1 -- 172.21.15.116:0/1500480109 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03441eef90 con 0x7f0344072080 2024-04-25T00:29:54.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:54.983+0000 7f032bfff640 1 -- 172.21.15.116:0/1500480109 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f033401a070 con 0x7f0344072080 2024-04-25T00:29:54.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:54.983+0000 7f032bfff640 1 --2- 172.21.15.116:0/1500480109 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f03300678c0 0x7f0330069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:54.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:54.983+0000 7f032bfff640 1 -- 172.21.15.116:0/1500480109 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 10536+0+0 (secure 0 0 0) 0x7f03340863c0 con 0x7f0344072080 2024-04-25T00:29:54.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:54.983+0000 7f0342ffd640 1 --2- 172.21.15.116:0/1500480109 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f03300678c0 0x7f0330069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:54.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:54.983+0000 7f0342ffd640 1 --2- 172.21.15.116:0/1500480109 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f03300678c0 0x7f0330069d80 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f03380af000 tx=0x7f03380a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:29:54.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:54.987+0000 7f032bfff640 1 -- 172.21.15.116:0/1500480109 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f03441eef90 con 0x7f0344072080 2024-04-25T00:29:55.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:55.267+0000 7f034a047640 1 -- 172.21.15.116:0/1500480109 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0344068a60 con 0x7f0344072080 2024-04-25T00:29:55.270 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:55.267+0000 7f032bfff640 1 -- 172.21.15.116:0/1500480109 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 236 v236) ==== 95+0+5681 (secure 0 0 0) 0x7f0334055ca0 con 0x7f0344072080 2024-04-25T00:29:55.270 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:29:55.270 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":236,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":236,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:29:54.168353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8847},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8847":{"gid":8847,"name":"c","rank":0,"incarnation":233,"state":"up:active","state_seq":17,"addr":"172.21.15.116:6833/2662028892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2662028892},{"type":"v1","addr":"172.21.15.116:6833","nonce":2662028892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8847,"qdb_cluster":[8847]},"id":1},{"mdsmap":{"epoch":231,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:28:53.808539+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8772":{"gid":8772,"name":"a","rank":0,"incarnation":228,"state":"up:active","state_seq":20,"addr":"172.21.15.116:6835/3499878473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3499878473},{"type":"v1","addr":"172.21.15.116:6835","nonce":3499878473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8772,"qdb_cluster":[8772]},"id":2}]} 2024-04-25T00:29:55.271 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 236 2024-04-25T00:29:55.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:55.271+0000 7f034a047640 1 -- 172.21.15.116:0/1500480109 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f03300678c0 msgr2=0x7f0330069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:55.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:55.271+0000 7f034a047640 1 --2- 172.21.15.116:0/1500480109 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f03300678c0 0x7f0330069d80 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f03380af000 tx=0x7f03380a7000 comp rx=0 tx=0).stop 2024-04-25T00:29:55.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:55.271+0000 7f034a047640 1 -- 172.21.15.116:0/1500480109 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0344072080 msgr2=0x7f034414fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:55.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:55.271+0000 7f034a047640 1 --2- 172.21.15.116:0/1500480109 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0344072080 0x7f034414fd10 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f033400c390 tx=0x7f033400c850 comp rx=0 tx=0).stop 2024-04-25T00:29:55.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:55.271+0000 7f034a047640 1 -- 172.21.15.116:0/1500480109 shutdown_connections 2024-04-25T00:29:55.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:55.271+0000 7f034a047640 1 --2- 172.21.15.116:0/1500480109 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f03300678c0 0x7f0330069d80 secure :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f03380af000 tx=0x7f03380a7000 comp rx=0 tx=0).stop 2024-04-25T00:29:55.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:55.271+0000 7f034a047640 1 --2- 172.21.15.116:0/1500480109 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f03441562a0 0x7f03441478d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:55.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:55.271+0000 7f034a047640 1 --2- 172.21.15.116:0/1500480109 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0344152d80 0x7f0344149040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:55.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:55.271+0000 7f034a047640 1 --2- 172.21.15.116:0/1500480109 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0344072080 0x7f034414fd10 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:55.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:55.271+0000 7f034a047640 1 -- 172.21.15.116:0/1500480109 >> 172.21.15.116:0/1500480109 conn(0x7f034406c720 msgr2=0x7f03440c1a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:29:55.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:55.271+0000 7f034a047640 1 -- 172.21.15.116:0/1500480109 shutdown_connections 2024-04-25T00:29:55.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:55.271+0000 7f034a047640 1 -- 172.21.15.116:0/1500480109 wait complete. 2024-04-25T00:29:55.295 INFO:tasks.mds_thrash.fs.[a]:mds.c has gained rank=0, replacing gid=6641 2024-04-25T00:29:55.295 INFO:tasks.mds_thrash.fs.[a]:waiting for 1 secs before reviving mds.d 2024-04-25T00:29:56.296 INFO:tasks.mds_thrash.fs.[a]:reviving mds.d 2024-04-25T00:29:56.296 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-25T00:29:56.296 DEBUG:teuthology.orchestra.run.smithi174:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-25T00:29:56.299 INFO:tasks.ceph.mds.d:Started 2024-04-25T00:29:56.299 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:29:56.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.443+0000 7f2bd477c640 1 Processor -- start 2024-04-25T00:29:56.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.443+0000 7f2bd477c640 1 -- start start 2024-04-25T00:29:56.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.443+0000 7f2bd477c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2bcc15bea0 0x7f2bcc15e290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:56.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.443+0000 7f2bd477c640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2bcc0a6cc0 con 0x7f2bcc071590 2024-04-25T00:29:56.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.443+0000 7f2bd477c640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2bcc0c6bb0 con 0x7f2bcc14fe10 2024-04-25T00:29:56.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.443+0000 7f2bd477c640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f2bcc074790 con 0x7f2bcc15bea0 2024-04-25T00:29:56.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.443+0000 7f2bd2d19640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2bcc15bea0 0x7f2bcc15e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:56.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.443+0000 7f2bd2d19640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f2bcc15bea0 0x7f2bcc15e290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41462/0 (socket says 172.21.15.116:41462) 2024-04-25T00:29:56.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.443+0000 7f2bd2518640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f2bcc14fe10 0x7f2bcc06c220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:40908/0 (socket says 172.21.15.116:40908) 2024-04-25T00:29:56.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.443+0000 7f2bd2d19640 1 -- 172.21.15.116:0/1538629469 learned_addr learned my addr 172.21.15.116:0/1538629469 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:29:56.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.447+0000 7f2bd1516640 1 -- 172.21.15.116:0/1538629469 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 412015163 0 0) 0x7f2bcc0c6bb0 con 0x7f2bcc14fe10 2024-04-25T00:29:56.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.447+0000 7f2bd1516640 1 -- 172.21.15.116:0/1538629469 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2bb8003530 con 0x7f2bcc14fe10 2024-04-25T00:29:56.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.447+0000 7f2bd2d19640 1 -- 172.21.15.116:0/1538629469 >> v1:172.21.15.174:6789/0 conn(0x7f2bcc14fe10 legacy=0x7f2bcc06c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:56.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.447+0000 7f2bd2d19640 1 -- 172.21.15.116:0/1538629469 >> v1:172.21.15.116:6789/0 conn(0x7f2bcc071590 legacy=0x7f2bcc156180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:56.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.447+0000 7f2bd2d19640 1 -- 172.21.15.116:0/1538629469 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bcc0a7150 con 0x7f2bcc15bea0 2024-04-25T00:29:56.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.447+0000 7f2bd2d19640 1 --2- 172.21.15.116:0/1538629469 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bcc15bea0 0x7f2bcc15e290 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f2bc400d150 tx=0x7f2bc401eb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed69da5a4b593353 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:29:56.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.447+0000 7f2bd1516640 1 -- 172.21.15.116:0/1538629469 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bc400b2d0 con 0x7f2bcc15bea0 2024-04-25T00:29:56.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.447+0000 7f2bd1516640 1 -- 172.21.15.116:0/1538629469 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2bc400b470 con 0x7f2bcc15bea0 2024-04-25T00:29:56.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.447+0000 7f2bd1516640 1 -- 172.21.15.116:0/1538629469 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bc40a7900 con 0x7f2bcc15bea0 2024-04-25T00:29:56.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.451+0000 7f2bd477c640 1 -- 172.21.15.116:0/1538629469 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bcc15bea0 msgr2=0x7f2bcc15e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:56.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.451+0000 7f2bd477c640 1 --2- 172.21.15.116:0/1538629469 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bcc15bea0 0x7f2bcc15e290 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f2bc400d150 tx=0x7f2bc401eb30 comp rx=0 tx=0).stop 2024-04-25T00:29:56.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.451+0000 7f2bd477c640 1 -- 172.21.15.116:0/1538629469 shutdown_connections 2024-04-25T00:29:56.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.451+0000 7f2bd477c640 1 --2- 172.21.15.116:0/1538629469 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bcc15bea0 0x7f2bcc15e290 unknown :-1 s=CLOSED pgs=1688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:56.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.451+0000 7f2bd477c640 1 -- 172.21.15.116:0/1538629469 >> 172.21.15.116:0/1538629469 conn(0x7f2bcc0c42e0 msgr2=0x7f2bcc0c46f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:29:56.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.451+0000 7f2bd477c640 1 -- 172.21.15.116:0/1538629469 shutdown_connections 2024-04-25T00:29:56.450 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.451+0000 7f2bd477c640 1 -- 172.21.15.116:0/1538629469 wait complete. 2024-04-25T00:29:56.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.451+0000 7f2bd477c640 1 Processor -- start 2024-04-25T00:29:56.451 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.451+0000 7f2bd477c640 1 -- start start 2024-04-25T00:29:56.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.451+0000 7f2bd477c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2bcc071590 0x7f2bcc1ef6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:56.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.451+0000 7f2bd477c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2bcc14fe10 0x7f2bcc0c0e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:56.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.451+0000 7f2bd477c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bcc15bea0 0x7f2bcc0aa680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:56.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.451+0000 7f2bd477c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2bcc06c930 con 0x7f2bcc15bea0 2024-04-25T00:29:56.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.451+0000 7f2bd477c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2bcc155cf0 con 0x7f2bcc14fe10 2024-04-25T00:29:56.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.451+0000 7f2bd477c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f2bcc06cab0 con 0x7f2bcc071590 2024-04-25T00:29:56.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.451+0000 7f2bd1d17640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2bcc14fe10 0x7f2bcc0c0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:56.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.451+0000 7f2bd1d17640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2bcc14fe10 0x7f2bcc0c0e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:51898/0 (socket says 172.21.15.116:51898) 2024-04-25T00:29:56.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.451+0000 7f2bd1d17640 1 -- 172.21.15.116:0/276780489 learned_addr learned my addr 172.21.15.116:0/276780489 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:29:56.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.451+0000 7f2bd2518640 1 --2- 172.21.15.116:0/276780489 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2bcc071590 0x7f2bcc1ef6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:56.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.451+0000 7f2bd2d19640 1 --2- 172.21.15.116:0/276780489 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bcc15bea0 0x7f2bcc0aa680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:56.453 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.451+0000 7f2bd2518640 1 -- 172.21.15.116:0/276780489 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2bcc14fe10 msgr2=0x7f2bcc0c0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:56.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.451+0000 7f2bd2518640 1 --2- 172.21.15.116:0/276780489 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2bcc14fe10 0x7f2bcc0c0e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:56.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.451+0000 7f2bd2518640 1 -- 172.21.15.116:0/276780489 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bcc15bea0 msgr2=0x7f2bcc0aa680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:56.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.451+0000 7f2bd2518640 1 --2- 172.21.15.116:0/276780489 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bcc15bea0 0x7f2bcc0aa680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:56.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.451+0000 7f2bd2518640 1 -- 172.21.15.116:0/276780489 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bcc0ac670 con 0x7f2bcc071590 2024-04-25T00:29:56.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.451+0000 7f2bd1d17640 1 --2- 172.21.15.116:0/276780489 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2bcc14fe10 0x7f2bcc0c0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:29:56.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.451+0000 7f2bd2518640 1 --2- 172.21.15.116:0/276780489 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2bcc071590 0x7f2bcc1ef6b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc00c390 tx=0x7f2bbc00c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:29:56.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.455+0000 7f2bc2ffd640 1 -- 172.21.15.116:0/276780489 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc0074e0 con 0x7f2bcc071590 2024-04-25T00:29:56.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.455+0000 7f2bc2ffd640 1 -- 172.21.15.116:0/276780489 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2bbc0076a0 con 0x7f2bcc071590 2024-04-25T00:29:56.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.455+0000 7f2bd477c640 1 -- 172.21.15.116:0/276780489 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bcc0ac390 con 0x7f2bcc071590 2024-04-25T00:29:56.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.455+0000 7f2bc2ffd640 1 -- 172.21.15.116:0/276780489 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc004a30 con 0x7f2bcc071590 2024-04-25T00:29:56.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.455+0000 7f2bd477c640 1 -- 172.21.15.116:0/276780489 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bcc1eeeb0 con 0x7f2bcc071590 2024-04-25T00:29:56.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.455+0000 7f2bd477c640 1 -- 172.21.15.116:0/276780489 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2bcc068a60 con 0x7f2bcc071590 2024-04-25T00:29:56.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.455+0000 7f2bc2ffd640 1 -- 172.21.15.116:0/276780489 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2bbc01a020 con 0x7f2bcc071590 2024-04-25T00:29:56.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.455+0000 7f2bc2ffd640 1 --2- 172.21.15.116:0/276780489 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2bb00678f0 0x7f2bb0069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:56.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.455+0000 7f2bc2ffd640 1 -- 172.21.15.116:0/276780489 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(73..73 src has 1..73) ==== 10536+0+0 (secure 0 0 0) 0x7f2bbc086590 con 0x7f2bcc071590 2024-04-25T00:29:56.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.455+0000 7f2bd1d17640 1 --2- 172.21.15.116:0/276780489 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2bb00678f0 0x7f2bb0069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:56.456 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.455+0000 7f2bd1d17640 1 --2- 172.21.15.116:0/276780489 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2bb00678f0 0x7f2bb0069db0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc1ec900 tx=0x7f2bc8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:29:56.464 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.463+0000 7f2bc2ffd640 1 -- 172.21.15.116:0/276780489 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f2bbc0557e0 con 0x7f2bcc071590 2024-04-25T00:29:56.499 INFO:tasks.ceph.mds.d.smithi174.stdout:starting mds.d at 2024-04-25T00:29:56.737 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.735+0000 7f2bd477c640 1 -- 172.21.15.116:0/276780489 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2bcc073300 con 0x7f2bcc071590 2024-04-25T00:29:56.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.739+0000 7f2bc2ffd640 1 -- 172.21.15.116:0/276780489 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 236 v236) ==== 95+0+5681 (secure 0 0 0) 0x7f2bbc059ae0 con 0x7f2bcc071590 2024-04-25T00:29:56.741 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:29:56.741 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":236,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":236,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:29:54.168353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8847},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8847":{"gid":8847,"name":"c","rank":0,"incarnation":233,"state":"up:active","state_seq":17,"addr":"172.21.15.116:6833/2662028892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2662028892},{"type":"v1","addr":"172.21.15.116:6833","nonce":2662028892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8847,"qdb_cluster":[8847]},"id":1},{"mdsmap":{"epoch":231,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:28:53.808539+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8772":{"gid":8772,"name":"a","rank":0,"incarnation":228,"state":"up:active","state_seq":20,"addr":"172.21.15.116:6835/3499878473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3499878473},{"type":"v1","addr":"172.21.15.116:6835","nonce":3499878473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8772,"qdb_cluster":[8772]},"id":2}]} 2024-04-25T00:29:56.742 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 236 2024-04-25T00:29:56.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.743+0000 7f2bd477c640 1 -- 172.21.15.116:0/276780489 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2bb00678f0 msgr2=0x7f2bb0069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:56.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.743+0000 7f2bd477c640 1 --2- 172.21.15.116:0/276780489 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2bb00678f0 0x7f2bb0069db0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc1ec900 tx=0x7f2bc8009040 comp rx=0 tx=0).stop 2024-04-25T00:29:56.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.743+0000 7f2bd477c640 1 -- 172.21.15.116:0/276780489 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2bcc071590 msgr2=0x7f2bcc1ef6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:56.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.743+0000 7f2bd477c640 1 --2- 172.21.15.116:0/276780489 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2bcc071590 0x7f2bcc1ef6b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2bbc00c390 tx=0x7f2bbc00c850 comp rx=0 tx=0).stop 2024-04-25T00:29:56.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.747+0000 7f2bd477c640 1 -- 172.21.15.116:0/276780489 shutdown_connections 2024-04-25T00:29:56.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.747+0000 7f2bd477c640 1 --2- 172.21.15.116:0/276780489 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2bb00678f0 0x7f2bb0069db0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:56.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.747+0000 7f2bd477c640 1 --2- 172.21.15.116:0/276780489 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2bcc15bea0 0x7f2bcc0aa680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:56.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.747+0000 7f2bd477c640 1 --2- 172.21.15.116:0/276780489 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2bcc14fe10 0x7f2bcc0c0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:56.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.747+0000 7f2bd477c640 1 --2- 172.21.15.116:0/276780489 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2bcc071590 0x7f2bcc1ef6b0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:56.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.747+0000 7f2bd477c640 1 -- 172.21.15.116:0/276780489 >> 172.21.15.116:0/276780489 conn(0x7f2bcc0c42e0 msgr2=0x7f2bcc0c46c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:29:56.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.747+0000 7f2bd477c640 1 -- 172.21.15.116:0/276780489 shutdown_connections 2024-04-25T00:29:56.746 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:56.747+0000 7f2bd477c640 1 -- 172.21.15.116:0/276780489 wait complete. 2024-04-25T00:29:56.770 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2024-04-25T00:29:58.295 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:29:58.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.435+0000 7fa3d9d79640 1 Processor -- start 2024-04-25T00:29:58.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.435+0000 7fa3d9d79640 1 -- start start 2024-04-25T00:29:58.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.435+0000 7fa3d9d79640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa3d40a94a0 0x7fa3d41563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:58.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.435+0000 7fa3d9d79640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3d4071220 con 0x7fa3d414c720 2024-04-25T00:29:58.436 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.435+0000 7fa3d9d79640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3d4152420 con 0x7fa3d4071680 2024-04-25T00:29:58.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.435+0000 7fa3d9d79640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fa3d4153460 con 0x7fa3d40a94a0 2024-04-25T00:29:58.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.435+0000 7fa3d2ffd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa3d40a94a0 0x7fa3d41563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:58.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.435+0000 7fa3d37fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fa3d414c720 0x7fa3d40c2a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:42696/0 (socket says 172.21.15.116:42696) 2024-04-25T00:29:58.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.435+0000 7fa3d37fe640 1 -- 172.21.15.116:0/1433506381 learned_addr learned my addr 172.21.15.116:0/1433506381 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:29:58.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.435+0000 7fa3d2ffd640 1 -- 172.21.15.116:0/1433506381 >> v1:172.21.15.174:6789/0 conn(0x7fa3d4071680 legacy=0x7fa3d4071ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:29:58.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.439+0000 7fa3d2ffd640 1 -- 172.21.15.116:0/1433506381 >> v1:172.21.15.116:6789/0 conn(0x7fa3d414c720 legacy=0x7fa3d40c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:58.437 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.439+0000 7fa3d27fc640 1 -- 172.21.15.116:0/1433506381 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1100611744 0 0) 0x7fa3d4071220 con 0x7fa3d414c720 2024-04-25T00:29:58.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.439+0000 7fa3d2ffd640 1 -- 172.21.15.116:0/1433506381 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3d40721c0 con 0x7fa3d40a94a0 2024-04-25T00:29:58.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.439+0000 7fa3d2ffd640 1 --2- 172.21.15.116:0/1433506381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3d40a94a0 0x7fa3d41563d0 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7fa3c80097c0 tx=0x7fa3c801c410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c02f808ddca150d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:29:58.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.439+0000 7fa3d27fc640 1 -- 172.21.15.116:0/1433506381 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c80ab610 con 0x7fa3d40a94a0 2024-04-25T00:29:58.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.439+0000 7fa3d27fc640 1 -- 172.21.15.116:0/1433506381 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3c80b0070 con 0x7fa3d40a94a0 2024-04-25T00:29:58.439 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.439+0000 7fa3d27fc640 1 -- 172.21.15.116:0/1433506381 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c80ab9d0 con 0x7fa3d40a94a0 2024-04-25T00:29:58.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.439+0000 7fa3d9d79640 1 -- 172.21.15.116:0/1433506381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3d40a94a0 msgr2=0x7fa3d41563d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:58.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.439+0000 7fa3d9d79640 1 --2- 172.21.15.116:0/1433506381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3d40a94a0 0x7fa3d41563d0 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7fa3c80097c0 tx=0x7fa3c801c410 comp rx=0 tx=0).stop 2024-04-25T00:29:58.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.439+0000 7fa3d9d79640 1 -- 172.21.15.116:0/1433506381 shutdown_connections 2024-04-25T00:29:58.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.439+0000 7fa3d9d79640 1 --2- 172.21.15.116:0/1433506381 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3d40a94a0 0x7fa3d41563d0 unknown :-1 s=CLOSED pgs=1691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:58.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.439+0000 7fa3d9d79640 1 -- 172.21.15.116:0/1433506381 >> 172.21.15.116:0/1433506381 conn(0x7fa3d40c3b00 msgr2=0x7fa3d40c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:29:58.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.439+0000 7fa3d9d79640 1 -- 172.21.15.116:0/1433506381 shutdown_connections 2024-04-25T00:29:58.440 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.439+0000 7fa3d9d79640 1 -- 172.21.15.116:0/1433506381 wait complete. 2024-04-25T00:29:58.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.443+0000 7fa3d9d79640 1 Processor -- start 2024-04-25T00:29:58.441 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.443+0000 7fa3d9d79640 1 -- start start 2024-04-25T00:29:58.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.443+0000 7fa3d9d79640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3d4071680 0x7fa3d40c5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:58.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.443+0000 7fa3d9d79640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3d40a94a0 0x7fa3d4148030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:58.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.443+0000 7fa3d9d79640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3d414c720 0x7fa3d4146980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:58.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.443+0000 7fa3d9d79640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa3d4152f40 con 0x7fa3d414c720 2024-04-25T00:29:58.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.443+0000 7fa3d9d79640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa3d4152c60 con 0x7fa3d40a94a0 2024-04-25T00:29:58.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.443+0000 7fa3d9d79640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa3d40c3120 con 0x7fa3d4071680 2024-04-25T00:29:58.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.443+0000 7fa3d3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3d414c720 0x7fa3d4146980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:58.443 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.443+0000 7fa3d3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3d414c720 0x7fa3d4146980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41490/0 (socket says 172.21.15.116:41490) 2024-04-25T00:29:58.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.443+0000 7fa3d3fff640 1 -- 172.21.15.116:0/2375362698 learned_addr learned my addr 172.21.15.116:0/2375362698 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:29:58.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.443+0000 7fa3d2ffd640 1 --2- 172.21.15.116:0/2375362698 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3d40a94a0 0x7fa3d4148030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:58.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.443+0000 7fa3d37fe640 1 --2- 172.21.15.116:0/2375362698 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3d4071680 0x7fa3d40c5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:58.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.443+0000 7fa3d3fff640 1 -- 172.21.15.116:0/2375362698 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3d4071680 msgr2=0x7fa3d40c5230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:58.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.443+0000 7fa3d3fff640 1 --2- 172.21.15.116:0/2375362698 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3d4071680 0x7fa3d40c5230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:58.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.443+0000 7fa3d3fff640 1 -- 172.21.15.116:0/2375362698 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3d40a94a0 msgr2=0x7fa3d4148030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:58.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.443+0000 7fa3d3fff640 1 --2- 172.21.15.116:0/2375362698 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3d40a94a0 0x7fa3d4148030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:58.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.443+0000 7fa3d3fff640 1 -- 172.21.15.116:0/2375362698 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3d40c5c00 con 0x7fa3d414c720 2024-04-25T00:29:58.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.443+0000 7fa3d37fe640 1 --2- 172.21.15.116:0/2375362698 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3d4071680 0x7fa3d40c5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:29:58.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.443+0000 7fa3d3fff640 1 --2- 172.21.15.116:0/2375362698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3d414c720 0x7fa3d4146980 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc00c770 tx=0x7fa3cc00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:29:58.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.443+0000 7fa3bbfff640 1 -- 172.21.15.116:0/2375362698 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3cc003b80 con 0x7fa3d414c720 2024-04-25T00:29:58.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.443+0000 7fa3bbfff640 1 -- 172.21.15.116:0/2375362698 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3cc003d20 con 0x7fa3d414c720 2024-04-25T00:29:58.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.443+0000 7fa3d9d79640 1 -- 172.21.15.116:0/2375362698 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3d40c5ee0 con 0x7fa3d414c720 2024-04-25T00:29:58.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.443+0000 7fa3bbfff640 1 -- 172.21.15.116:0/2375362698 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3cc00aa90 con 0x7fa3d414c720 2024-04-25T00:29:58.444 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.443+0000 7fa3d9d79640 1 -- 172.21.15.116:0/2375362698 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3d40aae70 con 0x7fa3d414c720 2024-04-25T00:29:58.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.443+0000 7fa3d9d79640 1 -- 172.21.15.116:0/2375362698 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3d4068350 con 0x7fa3d414c720 2024-04-25T00:29:58.445 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.443+0000 7fa3bbfff640 1 -- 172.21.15.116:0/2375362698 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa3cc00ac30 con 0x7fa3d414c720 2024-04-25T00:29:58.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.443+0000 7fa3bbfff640 1 --2- 172.21.15.116:0/2375362698 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa3c00678c0 0x7fa3c0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:58.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.443+0000 7fa3bbfff640 1 -- 172.21.15.116:0/2375362698 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 10536+0+0 (secure 0 0 0) 0x7fa3cc086eb0 con 0x7fa3d414c720 2024-04-25T00:29:58.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.447+0000 7fa3d37fe640 1 --2- 172.21.15.116:0/2375362698 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa3c00678c0 0x7fa3c0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:58.446 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.447+0000 7fa3d37fe640 1 --2- 172.21.15.116:0/2375362698 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa3c00678c0 0x7fa3c0069d80 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fa3c4000f70 tx=0x7fa3c4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:29:58.449 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.451+0000 7fa3bbfff640 1 -- 172.21.15.116:0/2375362698 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fa3d4152420 con 0x7fa3d414c720 2024-04-25T00:29:58.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.651+0000 7fa3d9d79640 1 -- 172.21.15.116:0/2375362698 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa3d40c3640 con 0x7fa3d414c720 2024-04-25T00:29:58.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.651+0000 7fa3bbfff640 1 -- 172.21.15.116:0/2375362698 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 237 v237) ==== 95+0+6510 (secure 0 0 0) 0x7fa3cc056750 con 0x7fa3d414c720 2024-04-25T00:29:58.652 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:29:58.652 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":237,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6764,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3873739116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3873739116},{"type":"v1","addr":"172.21.15.174:6837","nonce":3873739116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":237}],"filesystems":[{"mdsmap":{"epoch":236,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:29:54.168353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8847},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8847":{"gid":8847,"name":"c","rank":0,"incarnation":233,"state":"up:active","state_seq":17,"addr":"172.21.15.116:6833/2662028892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2662028892},{"type":"v1","addr":"172.21.15.116:6833","nonce":2662028892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8847,"qdb_cluster":[8847]},"id":1},{"mdsmap":{"epoch":231,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:28:53.808539+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8772":{"gid":8772,"name":"a","rank":0,"incarnation":228,"state":"up:active","state_seq":20,"addr":"172.21.15.116:6835/3499878473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3499878473},{"type":"v1","addr":"172.21.15.116:6835","nonce":3499878473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8772,"qdb_cluster":[8772]},"id":2}]} 2024-04-25T00:29:58.652 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 237 2024-04-25T00:29:58.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.655+0000 7fa3d9d79640 1 -- 172.21.15.116:0/2375362698 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa3c00678c0 msgr2=0x7fa3c0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:58.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.655+0000 7fa3d9d79640 1 --2- 172.21.15.116:0/2375362698 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa3c00678c0 0x7fa3c0069d80 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fa3c4000f70 tx=0x7fa3c4009040 comp rx=0 tx=0).stop 2024-04-25T00:29:58.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.655+0000 7fa3d9d79640 1 -- 172.21.15.116:0/2375362698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3d414c720 msgr2=0x7fa3d4146980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:58.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.655+0000 7fa3d9d79640 1 --2- 172.21.15.116:0/2375362698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3d414c720 0x7fa3d4146980 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7fa3cc00c770 tx=0x7fa3cc00cc30 comp rx=0 tx=0).stop 2024-04-25T00:29:58.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.655+0000 7fa3d9d79640 1 -- 172.21.15.116:0/2375362698 shutdown_connections 2024-04-25T00:29:58.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.655+0000 7fa3d9d79640 1 --2- 172.21.15.116:0/2375362698 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa3c00678c0 0x7fa3c0069d80 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:58.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.655+0000 7fa3d9d79640 1 --2- 172.21.15.116:0/2375362698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa3d414c720 0x7fa3d4146980 unknown :-1 s=CLOSED pgs=1692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:58.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.655+0000 7fa3d9d79640 1 --2- 172.21.15.116:0/2375362698 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa3d40a94a0 0x7fa3d4148030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:58.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.655+0000 7fa3d9d79640 1 --2- 172.21.15.116:0/2375362698 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa3d4071680 0x7fa3d40c5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:58.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.655+0000 7fa3d9d79640 1 -- 172.21.15.116:0/2375362698 >> 172.21.15.116:0/2375362698 conn(0x7fa3d40c3b00 msgr2=0x7fa3d4145e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:29:58.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.655+0000 7fa3d9d79640 1 -- 172.21.15.116:0/2375362698 shutdown_connections 2024-04-25T00:29:58.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.655+0000 7fa3d9d79640 1 -- 172.21.15.116:0/2375362698 wait complete. 2024-04-25T00:29:58.671 INFO:tasks.mds_thrash.fs.[b]:kill mds.a (rank=0) 2024-04-25T00:29:58.672 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-25T00:29:58.672 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:29:58.682 INFO:tasks.ceph.mds.a:Stopped 2024-04-25T00:29:58.683 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:29:58.771 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:29:58.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.843+0000 7fcbfafca640 1 Processor -- start 2024-04-25T00:29:58.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.843+0000 7fcbfafca640 1 -- start start 2024-04-25T00:29:58.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.847+0000 7fcbfafca640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fcbf40c6080 0x7fcbf406fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:58.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.847+0000 7fcbfafca640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbf40c6490 con 0x7fcbf40c3c30 2024-04-25T00:29:58.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.847+0000 7fcbfafca640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbf406e8f0 con 0x7fcbf4152f60 2024-04-25T00:29:58.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.847+0000 7fcbfafca640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fcbf41595b0 con 0x7fcbf40c6080 2024-04-25T00:29:58.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.847+0000 7fcbf3fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fcbf40c3c30 0x7fcbf406c220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:42698/0 (socket says 172.21.15.116:42698) 2024-04-25T00:29:58.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.847+0000 7fcbf3fff640 1 -- 172.21.15.116:0/3939122960 learned_addr learned my addr 172.21.15.116:0/3939122960 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:29:58.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.847+0000 7fcbf9567640 1 --2- 172.21.15.116:0/3939122960 >> v2:172.21.15.174:3300/0 conn(0x7fcbf40c6080 0x7fcbf406fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:58.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.847+0000 7fcbf37fe640 1 -- 172.21.15.116:0/3939122960 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3688650855 0 0) 0x7fcbe0001120 con 0x7fcbf40c3c30 2024-04-25T00:29:58.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.847+0000 7fcbf37fe640 1 -- 172.21.15.116:0/3939122960 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcbe4003530 con 0x7fcbf40c3c30 2024-04-25T00:29:58.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.847+0000 7fcbf37fe640 1 -- 172.21.15.116:0/3939122960 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 234746058 0 0) 0x7fcbe4003530 con 0x7fcbf40c3c30 2024-04-25T00:29:58.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.847+0000 7fcbf37fe640 1 -- 172.21.15.116:0/3939122960 >> v2:172.21.15.174:3300/0 conn(0x7fcbf40c6080 msgr2=0x7fcbf406fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:29:58.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.847+0000 7fcbf37fe640 1 --2- 172.21.15.116:0/3939122960 >> v2:172.21.15.174:3300/0 conn(0x7fcbf40c6080 0x7fcbf406fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:58.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.847+0000 7fcbf37fe640 1 -- 172.21.15.116:0/3939122960 >> v1:172.21.15.174:6789/0 conn(0x7fcbf4152f60 legacy=0x7fcbf4158ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:58.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.847+0000 7fcbf37fe640 1 -- 172.21.15.116:0/3939122960 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbf406ebd0 con 0x7fcbf40c3c30 2024-04-25T00:29:58.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.847+0000 7fcbf37fe640 1 -- 172.21.15.116:0/3939122960 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7fcbe0001870 con 0x7fcbf40c3c30 2024-04-25T00:29:58.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.847+0000 7fcbf37fe640 1 -- 172.21.15.116:0/3939122960 >> v1:172.21.15.116:6789/0 conn(0x7fcbf40c3c30 legacy=0x7fcbf406c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:58.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.847+0000 7fcbf37fe640 1 --2- 172.21.15.116:0/3939122960 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcbe40039c0 0x7fcbe4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:58.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.847+0000 7fcbf37fe640 1 --2- 172.21.15.116:0/3939122960 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcbe4004810 0x7fcbe4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:58.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.847+0000 7fcbf37fe640 1 --2- 172.21.15.116:0/3939122960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbe40051d0 0x7fcbe400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:58.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.847+0000 7fcbf37fe640 1 -- 172.21.15.116:0/3939122960 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcbf41595b0 con 0x7fcbe40051d0 2024-04-25T00:29:58.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.847+0000 7fcbf37fe640 1 -- 172.21.15.116:0/3939122960 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcbe400c170 con 0x7fcbe40039c0 2024-04-25T00:29:58.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.847+0000 7fcbf37fe640 1 -- 172.21.15.116:0/3939122960 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fcbe400c370 con 0x7fcbe4004810 2024-04-25T00:29:58.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.847+0000 7fcbf9567640 1 --2- 172.21.15.116:0/3939122960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbe40051d0 0x7fcbe400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:58.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.847+0000 7fcbf3fff640 1 --2- 172.21.15.116:0/3939122960 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcbe4004810 0x7fcbe4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:58.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.847+0000 7fcbf9567640 1 -- 172.21.15.116:0/3939122960 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcbe4004810 msgr2=0x7fcbe4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:29:58.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.847+0000 7fcbf9567640 1 --2- 172.21.15.116:0/3939122960 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcbe4004810 0x7fcbe4004c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:58.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.847+0000 7fcbf9567640 1 -- 172.21.15.116:0/3939122960 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcbe40039c0 msgr2=0x7fcbe4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:29:58.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.847+0000 7fcbf9567640 1 --2- 172.21.15.116:0/3939122960 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcbe40039c0 0x7fcbe4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:58.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.847+0000 7fcbf9567640 1 -- 172.21.15.116:0/3939122960 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbe4003530 con 0x7fcbe40051d0 2024-04-25T00:29:58.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.847+0000 7fcbf9567640 1 -- 172.21.15.116:0/3939122960 reap_dead start 2024-04-25T00:29:58.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.847+0000 7fcbf9567640 1 --2- 172.21.15.116:0/3939122960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbe40051d0 0x7fcbe400b9d0 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7fcbec009fc0 tx=0x7fcbec01ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff0976e31096a001 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:29:58.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.847+0000 7fcbf37fe640 1 -- 172.21.15.116:0/3939122960 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbec009c10 con 0x7fcbe40051d0 2024-04-25T00:29:58.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.847+0000 7fcbf37fe640 1 -- 172.21.15.116:0/3939122960 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbec0a5d10 con 0x7fcbe40051d0 2024-04-25T00:29:58.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.847+0000 7fcbf37fe640 1 -- 172.21.15.116:0/3939122960 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbec003be0 con 0x7fcbe40051d0 2024-04-25T00:29:58.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.851+0000 7fcbfafca640 1 -- 172.21.15.116:0/3939122960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbe40051d0 msgr2=0x7fcbe400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:58.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.851+0000 7fcbfafca640 1 --2- 172.21.15.116:0/3939122960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbe40051d0 0x7fcbe400b9d0 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7fcbec009fc0 tx=0x7fcbec01ccc0 comp rx=0 tx=0).stop 2024-04-25T00:29:58.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.851+0000 7fcbfafca640 1 -- 172.21.15.116:0/3939122960 shutdown_connections 2024-04-25T00:29:58.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.851+0000 7fcbfafca640 1 --2- 172.21.15.116:0/3939122960 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbe40051d0 0x7fcbe400b9d0 unknown :-1 s=CLOSED pgs=1694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:58.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.851+0000 7fcbfafca640 1 -- 172.21.15.116:0/3939122960 >> 172.21.15.116:0/3939122960 conn(0x7fcbf40c4270 msgr2=0x7fcbf40c4680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:29:58.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.851+0000 7fcbfafca640 1 -- 172.21.15.116:0/3939122960 shutdown_connections 2024-04-25T00:29:58.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.851+0000 7fcbfafca640 1 -- 172.21.15.116:0/3939122960 wait complete. 2024-04-25T00:29:58.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.851+0000 7fcbfafca640 1 Processor -- start 2024-04-25T00:29:58.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.851+0000 7fcbfafca640 1 -- start start 2024-04-25T00:29:58.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.851+0000 7fcbfafca640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcbe40051d0 0x7fcbf41589b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:58.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.851+0000 7fcbfafca640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcbf40aa5e0 0x7fcbf4072a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:58.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.851+0000 7fcbfafca640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbf41eb2f0 0x7fcbf41e23b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:58.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.851+0000 7fcbfafca640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcbf4153340 con 0x7fcbf41eb2f0 2024-04-25T00:29:58.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.851+0000 7fcbfafca640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcbf406f610 con 0x7fcbe40051d0 2024-04-25T00:29:58.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.851+0000 7fcbfafca640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fcbf4074a70 con 0x7fcbf40aa5e0 2024-04-25T00:29:58.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.851+0000 7fcbf9567640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbf41eb2f0 0x7fcbf41e23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:58.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.851+0000 7fcbf9567640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbf41eb2f0 0x7fcbf41e23b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41500/0 (socket says 172.21.15.116:41500) 2024-04-25T00:29:58.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.851+0000 7fcbf9567640 1 -- 172.21.15.116:0/96624836 learned_addr learned my addr 172.21.15.116:0/96624836 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:29:58.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.851+0000 7fcbf8d66640 1 --2- 172.21.15.116:0/96624836 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcbe40051d0 0x7fcbf41589b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:58.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.851+0000 7fcbf3fff640 1 --2- 172.21.15.116:0/96624836 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcbf40aa5e0 0x7fcbf4072a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:58.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.851+0000 7fcbf9567640 1 -- 172.21.15.116:0/96624836 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcbf40aa5e0 msgr2=0x7fcbf4072a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:58.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.851+0000 7fcbf9567640 1 --2- 172.21.15.116:0/96624836 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcbf40aa5e0 0x7fcbf4072a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:58.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.851+0000 7fcbf9567640 1 -- 172.21.15.116:0/96624836 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcbe40051d0 msgr2=0x7fcbf41589b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:58.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.851+0000 7fcbf9567640 1 --2- 172.21.15.116:0/96624836 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcbe40051d0 0x7fcbf41589b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:58.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.851+0000 7fcbf9567640 1 -- 172.21.15.116:0/96624836 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbf4073840 con 0x7fcbf41eb2f0 2024-04-25T00:29:58.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.851+0000 7fcbf3fff640 1 --2- 172.21.15.116:0/96624836 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcbf40aa5e0 0x7fcbf4072a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:29:58.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.851+0000 7fcbf9567640 1 --2- 172.21.15.116:0/96624836 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbf41eb2f0 0x7fcbf41e23b0 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7fcbec01d1a0 tx=0x7fcbec009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:29:58.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.851+0000 7fcbf17fa640 1 -- 172.21.15.116:0/96624836 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbec009c10 con 0x7fcbf41eb2f0 2024-04-25T00:29:58.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.851+0000 7fcbf17fa640 1 -- 172.21.15.116:0/96624836 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbec0bd070 con 0x7fcbf41eb2f0 2024-04-25T00:29:58.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.851+0000 7fcbfafca640 1 -- 172.21.15.116:0/96624836 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbf41ef760 con 0x7fcbf41eb2f0 2024-04-25T00:29:58.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.851+0000 7fcbf17fa640 1 -- 172.21.15.116:0/96624836 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbec003db0 con 0x7fcbf41eb2f0 2024-04-25T00:29:58.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.851+0000 7fcbfafca640 1 -- 172.21.15.116:0/96624836 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbf41ef480 con 0x7fcbf41eb2f0 2024-04-25T00:29:58.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.855+0000 7fcbfafca640 1 -- 172.21.15.116:0/96624836 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbf4068a60 con 0x7fcbf41eb2f0 2024-04-25T00:29:58.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.855+0000 7fcbf17fa640 1 -- 172.21.15.116:0/96624836 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fcbec0ae8d0 con 0x7fcbf41eb2f0 2024-04-25T00:29:58.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.855+0000 7fcbf17fa640 1 --2- 172.21.15.116:0/96624836 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcbd80678c0 0x7fcbd8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:58.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.855+0000 7fcbf17fa640 1 -- 172.21.15.116:0/96624836 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 10536+0+0 (secure 0 0 0) 0x7fcbec122880 con 0x7fcbf41eb2f0 2024-04-25T00:29:58.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.855+0000 7fcbf8d66640 1 --2- 172.21.15.116:0/96624836 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcbd80678c0 0x7fcbd8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:58.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.855+0000 7fcbf8d66640 1 --2- 172.21.15.116:0/96624836 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcbd80678c0 0x7fcbd8069d80 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fcbe80016b0 tx=0x7fcbe8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:29:58.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.859+0000 7fcbf17fa640 1 -- 172.21.15.116:0/96624836 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fcbf4073840 con 0x7fcbf41eb2f0 2024-04-25T00:29:58.885 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.887+0000 7f058fee1640 1 Processor -- start 2024-04-25T00:29:58.886 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.887+0000 7f058fee1640 1 -- start start 2024-04-25T00:29:58.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.887+0000 7f058fee1640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f058806d5f0 0x7f0588156c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:58.888 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.887+0000 7f058fee1640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05880c9150 con 0x7f0588152d80 2024-04-25T00:29:58.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.887+0000 7f058fee1640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05880c9430 con 0x7f0588155810 2024-04-25T00:29:58.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.887+0000 7f058fee1640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f05880c9d60 con 0x7f058806d5f0 2024-04-25T00:29:58.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.887+0000 7f058dc7d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f0588152d80 0x7f058806bd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:42700/0 (socket says 172.21.15.116:42700) 2024-04-25T00:29:58.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.887+0000 7f058dc7d640 1 -- 172.21.15.116:0/177171984 learned_addr learned my addr 172.21.15.116:0/177171984 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:29:58.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.887+0000 7f058d47c640 1 --2- 172.21.15.116:0/177171984 >> v2:172.21.15.174:3300/0 conn(0x7f058806d5f0 0x7f0588156c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:58.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.887+0000 7f058cc7b640 1 -- 172.21.15.116:0/177171984 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3469559531 0 0) 0x7f05880c9150 con 0x7f0588152d80 2024-04-25T00:29:58.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.887+0000 7f058cc7b640 1 -- 172.21.15.116:0/177171984 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0574003530 con 0x7f0588152d80 2024-04-25T00:29:58.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.887+0000 7f058cc7b640 1 -- 172.21.15.116:0/177171984 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1372737659 0 0) 0x7f0574003530 con 0x7f0588152d80 2024-04-25T00:29:58.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.887+0000 7f058cc7b640 1 -- 172.21.15.116:0/177171984 >> v2:172.21.15.174:3300/0 conn(0x7f058806d5f0 msgr2=0x7f0588156c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:29:58.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.887+0000 7f058cc7b640 1 --2- 172.21.15.116:0/177171984 >> v2:172.21.15.174:3300/0 conn(0x7f058806d5f0 0x7f0588156c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:58.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.887+0000 7f058cc7b640 1 -- 172.21.15.116:0/177171984 >> v1:172.21.15.174:6789/0 conn(0x7f0588155810 legacy=0x7f05881571c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:58.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.887+0000 7f058cc7b640 1 -- 172.21.15.116:0/177171984 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05880c9710 con 0x7f0588152d80 2024-04-25T00:29:58.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.887+0000 7f058cc7b640 1 -- 172.21.15.116:0/177171984 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f0578001680 con 0x7f0588152d80 2024-04-25T00:29:58.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.887+0000 7f058cc7b640 1 -- 172.21.15.116:0/177171984 >> v1:172.21.15.116:6789/0 conn(0x7f0588152d80 legacy=0x7f058806bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:58.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.887+0000 7f058cc7b640 1 --2- 172.21.15.116:0/177171984 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f05740039c0 0x7f0574003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:58.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.887+0000 7f058cc7b640 1 --2- 172.21.15.116:0/177171984 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0574004810 0x7f0574004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:58.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.887+0000 7f058cc7b640 1 --2- 172.21.15.116:0/177171984 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05740051d0 0x7f057400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:58.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.887+0000 7f058cc7b640 1 -- 172.21.15.116:0/177171984 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f05880c9d60 con 0x7f05740051d0 2024-04-25T00:29:58.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.887+0000 7f058cc7b640 1 -- 172.21.15.116:0/177171984 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f057400c170 con 0x7f05740039c0 2024-04-25T00:29:58.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.887+0000 7f058cc7b640 1 -- 172.21.15.116:0/177171984 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f057400c370 con 0x7f0574004810 2024-04-25T00:29:58.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.887+0000 7f058dc7d640 1 --2- 172.21.15.116:0/177171984 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f05740039c0 0x7f0574003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:58.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.887+0000 7f058e47e640 1 --2- 172.21.15.116:0/177171984 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05740051d0 0x7f057400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:58.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.887+0000 7f058d47c640 1 --2- 172.21.15.116:0/177171984 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0574004810 0x7f0574004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:58.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.887+0000 7f058e47e640 1 -- 172.21.15.116:0/177171984 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0574004810 msgr2=0x7f0574004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:29:58.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.887+0000 7f058e47e640 1 --2- 172.21.15.116:0/177171984 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0574004810 0x7f0574004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:58.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.887+0000 7f058e47e640 1 -- 172.21.15.116:0/177171984 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f05740039c0 msgr2=0x7f0574003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:29:58.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.887+0000 7f058e47e640 1 --2- 172.21.15.116:0/177171984 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f05740039c0 0x7f0574003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:58.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.887+0000 7f058e47e640 1 -- 172.21.15.116:0/177171984 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0574003530 con 0x7f05740051d0 2024-04-25T00:29:58.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.891+0000 7f058e47e640 1 -- 172.21.15.116:0/177171984 reap_dead start 2024-04-25T00:29:58.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.891+0000 7f058e47e640 1 --2- 172.21.15.116:0/177171984 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05740051d0 0x7f057400b9d0 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7f0580001780 tx=0x7f058001c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a189aa2e23fe19a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:29:58.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.891+0000 7f058cc7b640 1 -- 172.21.15.116:0/177171984 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05800013a0 con 0x7f05740051d0 2024-04-25T00:29:58.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.891+0000 7f058cc7b640 1 -- 172.21.15.116:0/177171984 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05800052f0 con 0x7f05740051d0 2024-04-25T00:29:58.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.891+0000 7f058cc7b640 1 -- 172.21.15.116:0/177171984 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0580003840 con 0x7f05740051d0 2024-04-25T00:29:58.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.891+0000 7f058fee1640 1 -- 172.21.15.116:0/177171984 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05740051d0 msgr2=0x7f057400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:58.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.891+0000 7f058fee1640 1 --2- 172.21.15.116:0/177171984 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05740051d0 0x7f057400b9d0 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7f0580001780 tx=0x7f058001c420 comp rx=0 tx=0).stop 2024-04-25T00:29:58.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.891+0000 7f058fee1640 1 -- 172.21.15.116:0/177171984 shutdown_connections 2024-04-25T00:29:58.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.891+0000 7f058fee1640 1 --2- 172.21.15.116:0/177171984 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05740051d0 0x7f057400b9d0 unknown :-1 s=CLOSED pgs=1697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:58.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.891+0000 7f058fee1640 1 -- 172.21.15.116:0/177171984 >> 172.21.15.116:0/177171984 conn(0x7f05880acdd0 msgr2=0x7f05880ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:29:58.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.891+0000 7f058fee1640 1 -- 172.21.15.116:0/177171984 shutdown_connections 2024-04-25T00:29:58.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.891+0000 7f058fee1640 1 -- 172.21.15.116:0/177171984 wait complete. 2024-04-25T00:29:58.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.895+0000 7f058fee1640 1 Processor -- start 2024-04-25T00:29:58.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.895+0000 7f058fee1640 1 -- start start 2024-04-25T00:29:58.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.895+0000 7f058fee1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f05740051d0 0x7f05881ec510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:58.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.895+0000 7f058fee1640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f05881ebaa0 0x7f0588157040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:58.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.895+0000 7f058fee1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05880c61d0 0x7f05881e8050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:58.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.895+0000 7f058fee1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0588153160 con 0x7f05880c61d0 2024-04-25T00:29:58.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.895+0000 7f058fee1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f05880c8fb0 con 0x7f05740051d0 2024-04-25T00:29:58.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.895+0000 7f058fee1640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f05880c5d40 con 0x7f05881ebaa0 2024-04-25T00:29:58.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.895+0000 7f058e47e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05880c61d0 0x7f05881e8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:58.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.895+0000 7f058e47e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05880c61d0 0x7f05881e8050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41508/0 (socket says 172.21.15.116:41508) 2024-04-25T00:29:58.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.895+0000 7f058e47e640 1 -- 172.21.15.116:0/714284106 learned_addr learned my addr 172.21.15.116:0/714284106 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:29:58.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.895+0000 7f058dc7d640 1 --2- 172.21.15.116:0/714284106 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f05740051d0 0x7f05881ec510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:58.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.895+0000 7f058d47c640 1 --2- 172.21.15.116:0/714284106 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f05881ebaa0 0x7f0588157040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:58.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.895+0000 7f058e47e640 1 -- 172.21.15.116:0/714284106 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f05881ebaa0 msgr2=0x7f0588157040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:58.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.895+0000 7f058e47e640 1 --2- 172.21.15.116:0/714284106 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f05881ebaa0 0x7f0588157040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:58.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.895+0000 7f058e47e640 1 -- 172.21.15.116:0/714284106 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f05740051d0 msgr2=0x7f05881ec510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:58.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.895+0000 7f058e47e640 1 --2- 172.21.15.116:0/714284106 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f05740051d0 0x7f05881ec510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:58.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.895+0000 7f058e47e640 1 -- 172.21.15.116:0/714284106 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05880c7b90 con 0x7f05880c61d0 2024-04-25T00:29:58.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.895+0000 7f058e47e640 1 --2- 172.21.15.116:0/714284106 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05880c61d0 0x7f05881e8050 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f058001ccc0 tx=0x7f0574005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:29:58.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.895+0000 7f057e7fc640 1 -- 172.21.15.116:0/714284106 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05800013a0 con 0x7f05880c61d0 2024-04-25T00:29:58.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.895+0000 7f057e7fc640 1 -- 172.21.15.116:0/714284106 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05800054a0 con 0x7f05880c61d0 2024-04-25T00:29:58.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.895+0000 7f058fee1640 1 -- 172.21.15.116:0/714284106 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05881ebf20 con 0x7f05880c61d0 2024-04-25T00:29:58.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.895+0000 7f057e7fc640 1 -- 172.21.15.116:0/714284106 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0580007a40 con 0x7f05880c61d0 2024-04-25T00:29:58.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.895+0000 7f058fee1640 1 -- 172.21.15.116:0/714284106 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05880c78b0 con 0x7f05880c61d0 2024-04-25T00:29:58.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.895+0000 7f058fee1640 1 -- 172.21.15.116:0/714284106 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0588068350 con 0x7f05880c61d0 2024-04-25T00:29:58.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.895+0000 7f057e7fc640 1 -- 172.21.15.116:0/714284106 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f0580007030 con 0x7f05880c61d0 2024-04-25T00:29:58.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.899+0000 7f057e7fc640 1 --2- 172.21.15.116:0/714284106 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f056c0678c0 0x7f056c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:29:58.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.899+0000 7f057e7fc640 1 -- 172.21.15.116:0/714284106 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 10536+0+0 (secure 0 0 0) 0x7f05801207a0 con 0x7f05880c61d0 2024-04-25T00:29:58.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.899+0000 7f058dc7d640 1 --2- 172.21.15.116:0/714284106 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f056c0678c0 0x7f056c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:29:58.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.899+0000 7f058dc7d640 1 --2- 172.21.15.116:0/714284106 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f056c0678c0 0x7f056c069d80 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f05780050c0 tx=0x7f0578007900 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:29:58.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:58.903+0000 7f057e7fc640 1 -- 172.21.15.116:0/714284106 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f05800c4080 con 0x7f05880c61d0 2024-04-25T00:29:59.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:59.059+0000 7fcbfafca640 1 -- 172.21.15.116:0/96624836 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcbf4158710 con 0x7fcbf41eb2f0 2024-04-25T00:29:59.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:59.059+0000 7fcbf17fa640 1 -- 172.21.15.116:0/96624836 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 237 v237) ==== 95+0+6510 (secure 0 0 0) 0x7fcbec0169d0 con 0x7fcbf41eb2f0 2024-04-25T00:29:59.060 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:29:59.060 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":237,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6764,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3873739116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3873739116},{"type":"v1","addr":"172.21.15.174:6837","nonce":3873739116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":237}],"filesystems":[{"mdsmap":{"epoch":236,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:29:54.168353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8847},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8847":{"gid":8847,"name":"c","rank":0,"incarnation":233,"state":"up:active","state_seq":17,"addr":"172.21.15.116:6833/2662028892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2662028892},{"type":"v1","addr":"172.21.15.116:6833","nonce":2662028892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8847,"qdb_cluster":[8847]},"id":1},{"mdsmap":{"epoch":231,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:28:53.808539+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8772":{"gid":8772,"name":"a","rank":0,"incarnation":228,"state":"up:active","state_seq":20,"addr":"172.21.15.116:6835/3499878473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3499878473},{"type":"v1","addr":"172.21.15.116:6835","nonce":3499878473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8772,"qdb_cluster":[8772]},"id":2}]} 2024-04-25T00:29:59.060 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 237 2024-04-25T00:29:59.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:59.063+0000 7fcbfafca640 1 -- 172.21.15.116:0/96624836 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcbd80678c0 msgr2=0x7fcbd8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:59.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:59.063+0000 7fcbfafca640 1 --2- 172.21.15.116:0/96624836 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcbd80678c0 0x7fcbd8069d80 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fcbe80016b0 tx=0x7fcbe8009040 comp rx=0 tx=0).stop 2024-04-25T00:29:59.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:59.063+0000 7fcbfafca640 1 -- 172.21.15.116:0/96624836 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbf41eb2f0 msgr2=0x7fcbf41e23b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:59.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:59.063+0000 7fcbfafca640 1 --2- 172.21.15.116:0/96624836 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbf41eb2f0 0x7fcbf41e23b0 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7fcbec01d1a0 tx=0x7fcbec009750 comp rx=0 tx=0).stop 2024-04-25T00:29:59.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:59.063+0000 7fcbfafca640 1 -- 172.21.15.116:0/96624836 shutdown_connections 2024-04-25T00:29:59.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:59.063+0000 7fcbfafca640 1 --2- 172.21.15.116:0/96624836 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcbd80678c0 0x7fcbd8069d80 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:59.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:59.063+0000 7fcbfafca640 1 --2- 172.21.15.116:0/96624836 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcbf41eb2f0 0x7fcbf41e23b0 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:59.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:59.063+0000 7fcbfafca640 1 --2- 172.21.15.116:0/96624836 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcbf40aa5e0 0x7fcbf4072a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:59.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:59.063+0000 7fcbfafca640 1 --2- 172.21.15.116:0/96624836 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcbe40051d0 0x7fcbf41589b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:59.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:59.063+0000 7fcbfafca640 1 -- 172.21.15.116:0/96624836 >> 172.21.15.116:0/96624836 conn(0x7fcbf40c4270 msgr2=0x7fcbf406b950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:29:59.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:59.063+0000 7fcbfafca640 1 -- 172.21.15.116:0/96624836 shutdown_connections 2024-04-25T00:29:59.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:59.063+0000 7fcbfafca640 1 -- 172.21.15.116:0/96624836 wait complete. 2024-04-25T00:29:59.079 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:29:59.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:59.095+0000 7f058fee1640 1 -- 172.21.15.116:0/714284106 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f05880cab60 con 0x7f05880c61d0 2024-04-25T00:29:59.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:59.099+0000 7f057e7fc640 1 -- 172.21.15.116:0/714284106 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 237 v237) ==== 95+0+6510 (secure 0 0 0) 0x7f0580016130 con 0x7f05880c61d0 2024-04-25T00:29:59.097 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:29:59.097 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":237,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6764,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3873739116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3873739116},{"type":"v1","addr":"172.21.15.174:6837","nonce":3873739116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":237}],"filesystems":[{"mdsmap":{"epoch":236,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:29:54.168353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8847},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8847":{"gid":8847,"name":"c","rank":0,"incarnation":233,"state":"up:active","state_seq":17,"addr":"172.21.15.116:6833/2662028892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2662028892},{"type":"v1","addr":"172.21.15.116:6833","nonce":2662028892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8847,"qdb_cluster":[8847]},"id":1},{"mdsmap":{"epoch":231,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:28:53.808539+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8772":{"gid":8772,"name":"a","rank":0,"incarnation":228,"state":"up:active","state_seq":20,"addr":"172.21.15.116:6835/3499878473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3499878473},{"type":"v1","addr":"172.21.15.116:6835","nonce":3499878473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8772,"qdb_cluster":[8772]},"id":2}]} 2024-04-25T00:29:59.098 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 237 2024-04-25T00:29:59.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:59.099+0000 7f058fee1640 1 -- 172.21.15.116:0/714284106 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f056c0678c0 msgr2=0x7f056c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:59.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:59.099+0000 7f058fee1640 1 --2- 172.21.15.116:0/714284106 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f056c0678c0 0x7f056c069d80 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f05780050c0 tx=0x7f0578007900 comp rx=0 tx=0).stop 2024-04-25T00:29:59.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:59.099+0000 7f058fee1640 1 -- 172.21.15.116:0/714284106 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05880c61d0 msgr2=0x7f05881e8050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:29:59.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:59.099+0000 7f058fee1640 1 --2- 172.21.15.116:0/714284106 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05880c61d0 0x7f05881e8050 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f058001ccc0 tx=0x7f0574005840 comp rx=0 tx=0).stop 2024-04-25T00:29:59.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:59.099+0000 7f058fee1640 1 -- 172.21.15.116:0/714284106 shutdown_connections 2024-04-25T00:29:59.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:59.099+0000 7f058fee1640 1 --2- 172.21.15.116:0/714284106 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f056c0678c0 0x7f056c069d80 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:59.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:59.099+0000 7f058fee1640 1 --2- 172.21.15.116:0/714284106 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05880c61d0 0x7f05881e8050 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:59.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:59.099+0000 7f058fee1640 1 --2- 172.21.15.116:0/714284106 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f05881ebaa0 0x7f0588157040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:59.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:59.099+0000 7f058fee1640 1 --2- 172.21.15.116:0/714284106 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f05740051d0 0x7f05881ec510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:29:59.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:59.099+0000 7f058fee1640 1 -- 172.21.15.116:0/714284106 >> 172.21.15.116:0/714284106 conn(0x7f05880acdd0 msgr2=0x7f05880caf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:29:59.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:59.099+0000 7f058fee1640 1 -- 172.21.15.116:0/714284106 shutdown_connections 2024-04-25T00:29:59.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:29:59.099+0000 7f058fee1640 1 -- 172.21.15.116:0/714284106 wait complete. 2024-04-25T00:29:59.116 INFO:tasks.mds_thrash.fs.[a]:mds.d reported in up:standby state 2024-04-25T00:29:59.116 INFO:tasks.mds_thrash.fs.[a]:waiting for 81 secs before thrashing 2024-04-25T00:30:00.752 INFO:tasks.workunit.client.1.smithi174.stdout:Syncing()...2 sec 2024-04-25T00:30:00.752 INFO:tasks.workunit.client.1.smithi174.stdout:FFSB benchmark finished at: Thu Apr 25 00:30:00 2024 2024-04-25T00:30:00.752 INFO:tasks.workunit.client.1.smithi174.stdout: 2024-04-25T00:30:00.752 INFO:tasks.workunit.client.1.smithi174.stdout:Results: 2024-04-25T00:30:00.752 INFO:tasks.workunit.client.1.smithi174.stdout:Benchmark took 306.40 sec 2024-04-25T00:30:00.752 INFO:tasks.workunit.client.1.smithi174.stdout: 2024-04-25T00:30:00.752 INFO:tasks.workunit.client.1.smithi174.stdout:Total Results 2024-04-25T00:30:00.753 INFO:tasks.workunit.client.1.smithi174.stdout:=============== 2024-04-25T00:30:00.753 INFO:tasks.workunit.client.1.smithi174.stdout: Op Name Transactions Trans/sec % Trans % Op Weight Throughput 2024-04-25T00:30:00.753 INFO:tasks.workunit.client.1.smithi174.stdout: ======= ============ ========= ======= =========== ========== 2024-04-25T00:30:00.753 INFO:tasks.workunit.client.1.smithi174.stdout: write : 1555200 5075.71 100.000% 100.000% 19.8MB/sec 2024-04-25T00:30:00.753 INFO:tasks.workunit.client.1.smithi174.stdout:- 2024-04-25T00:30:00.753 INFO:tasks.workunit.client.1.smithi174.stdout:5075.71 Transactions per Second 2024-04-25T00:30:00.753 INFO:tasks.workunit.client.1.smithi174.stdout: 2024-04-25T00:30:00.753 INFO:tasks.workunit.client.1.smithi174.stdout:Throughput Results 2024-04-25T00:30:00.753 INFO:tasks.workunit.client.1.smithi174.stdout:=================== 2024-04-25T00:30:00.753 INFO:tasks.workunit.client.1.smithi174.stdout:Write Throughput: 19.8MB/sec 2024-04-25T00:30:00.760 INFO:tasks.workunit.client.1.smithi174.stdout: 2024-04-25T00:30:00.760 INFO:tasks.workunit.client.1.smithi174.stdout:System Call Latency statistics in millisecs 2024-04-25T00:30:00.760 INFO:tasks.workunit.client.1.smithi174.stdout:===== 2024-04-25T00:30:00.760 INFO:tasks.workunit.client.1.smithi174.stdout: Min Avg Max Total Calls 2024-04-25T00:30:00.760 INFO:tasks.workunit.client.1.smithi174.stdout: ======== ======== ======== ============ 2024-04-25T00:30:00.761 INFO:tasks.workunit.client.1.smithi174.stdout:[ open] 0.346000 1482.700909 21538.996094 1215 2024-04-25T00:30:00.761 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-04-25T00:30:00.761 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-04-25T00:30:00.761 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[2] 1000.000000 - 2000.000000 : 1 2024-04-25T00:30:00.761 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-04-25T00:30:00.761 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[4] 200.000000 - 500.000000 : 2 2024-04-25T00:30:00.761 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[5] 100.000000 - 200.000000 : 15 2024-04-25T00:30:00.761 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[6] 50.000000 - 100.000000 : 24 2024-04-25T00:30:00.761 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[7] 20.000000 - 50.000000 : 341 2024-04-25T00:30:00.761 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[8] 10.000000 - 20.000000 : 437 2024-04-25T00:30:00.761 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[9] 5.000000 - 10.000000 : 160 2024-04-25T00:30:00.761 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[10] 2.000000 - 5.000000 : 68 2024-04-25T00:30:00.761 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[11] 1.000000 - 2.000000 : 31 2024-04-25T00:30:00.761 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[12] 0.500000 - 1.000000 : 10 2024-04-25T00:30:00.762 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[13] 0.200000 - 0.500000 : 7 2024-04-25T00:30:00.762 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[14] 0.100000 - 0.200000 : 0 2024-04-25T00:30:00.762 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[15] 0.050000 - 0.100000 : 0 2024-04-25T00:30:00.762 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[16] 0.020000 - 0.050000 : 0 2024-04-25T00:30:00.762 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[17] 0.010000 - 0.020000 : 0 2024-04-25T00:30:00.762 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[18] 0.000000 - 0.010000 : 0 2024-04-25T00:30:00.762 INFO:tasks.workunit.client.1.smithi174.stdout: 2024-04-25T00:30:00.762 INFO:tasks.workunit.client.1.smithi174.stdout:[ write] 0.084000 5.060673 17597.976562 1555200 2024-04-25T00:30:00.762 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-04-25T00:30:00.762 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[1] 2000.000000 - 5000.000000 : 2 2024-04-25T00:30:00.762 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[2] 1000.000000 - 2000.000000 : 37 2024-04-25T00:30:00.762 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[3] 500.000000 - 1000.000000 : 9 2024-04-25T00:30:00.762 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[4] 200.000000 - 500.000000 : 21 2024-04-25T00:30:00.763 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[5] 100.000000 - 200.000000 : 70 2024-04-25T00:30:00.763 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[6] 50.000000 - 100.000000 : 4177 2024-04-25T00:30:00.763 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[7] 20.000000 - 50.000000 : 64625 2024-04-25T00:30:00.763 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[8] 10.000000 - 20.000000 : 142782 2024-04-25T00:30:00.763 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[9] 5.000000 - 10.000000 : 271173 2024-04-25T00:30:00.763 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[10] 2.000000 - 5.000000 : 370590 2024-04-25T00:30:00.763 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[11] 1.000000 - 2.000000 : 177122 2024-04-25T00:30:00.763 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[12] 0.500000 - 1.000000 : 74467 2024-04-25T00:30:00.763 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[13] 0.200000 - 0.500000 : 194127 2024-04-25T00:30:00.763 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[14] 0.100000 - 0.200000 : 245433 2024-04-25T00:30:00.763 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[15] 0.050000 - 0.100000 : 10556 2024-04-25T00:30:00.763 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[16] 0.020000 - 0.050000 : 0 2024-04-25T00:30:00.763 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[17] 0.010000 - 0.020000 : 0 2024-04-25T00:30:00.763 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[18] 0.000000 - 0.010000 : 0 2024-04-25T00:30:00.763 INFO:tasks.workunit.client.1.smithi174.stdout: 2024-04-25T00:30:00.764 INFO:tasks.workunit.client.1.smithi174.stdout:[ lseek] 0.000000 0.000968 6.727000 1555200 2024-04-25T00:30:00.764 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-04-25T00:30:00.764 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-04-25T00:30:00.764 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-04-25T00:30:00.764 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-04-25T00:30:00.764 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-04-25T00:30:00.764 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2024-04-25T00:30:00.764 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[6] 50.000000 - 100.000000 : 0 2024-04-25T00:30:00.764 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[7] 20.000000 - 50.000000 : 0 2024-04-25T00:30:00.764 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[8] 10.000000 - 20.000000 : 0 2024-04-25T00:30:00.764 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[9] 5.000000 - 10.000000 : 1 2024-04-25T00:30:00.764 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[10] 2.000000 - 5.000000 : 3 2024-04-25T00:30:00.764 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[11] 1.000000 - 2.000000 : 0 2024-04-25T00:30:00.764 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[12] 0.500000 - 1.000000 : 5 2024-04-25T00:30:00.765 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[13] 0.200000 - 0.500000 : 20 2024-04-25T00:30:00.765 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[14] 0.100000 - 0.200000 : 29 2024-04-25T00:30:00.765 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[15] 0.050000 - 0.100000 : 41 2024-04-25T00:30:00.765 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[16] 0.020000 - 0.050000 : 295 2024-04-25T00:30:00.765 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[17] 0.010000 - 0.020000 : 892 2024-04-25T00:30:00.765 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[18] 0.000000 - 0.010000 : 1553914 2024-04-25T00:30:00.765 INFO:tasks.workunit.client.1.smithi174.stdout: 2024-04-25T00:30:00.765 INFO:tasks.workunit.client.1.smithi174.stdout:[ close] 0.019000 2.461963 70.737000 1215 2024-04-25T00:30:00.765 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-04-25T00:30:00.765 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-04-25T00:30:00.765 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-04-25T00:30:00.765 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-04-25T00:30:00.765 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-04-25T00:30:00.765 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2024-04-25T00:30:00.765 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[6] 50.000000 - 100.000000 : 3 2024-04-25T00:30:00.766 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[7] 20.000000 - 50.000000 : 37 2024-04-25T00:30:00.766 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[8] 10.000000 - 20.000000 : 34 2024-04-25T00:30:00.766 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[9] 5.000000 - 10.000000 : 75 2024-04-25T00:30:00.766 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[10] 2.000000 - 5.000000 : 118 2024-04-25T00:30:00.766 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[11] 1.000000 - 2.000000 : 134 2024-04-25T00:30:00.766 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[12] 0.500000 - 1.000000 : 106 2024-04-25T00:30:00.766 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[13] 0.200000 - 0.500000 : 70 2024-04-25T00:30:00.766 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[14] 0.100000 - 0.200000 : 139 2024-04-25T00:30:00.766 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[15] 0.050000 - 0.100000 : 115 2024-04-25T00:30:00.766 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[16] 0.020000 - 0.050000 : 382 2024-04-25T00:30:00.766 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[17] 0.010000 - 0.020000 : 2 2024-04-25T00:30:00.766 INFO:tasks.workunit.client.1.smithi174.stdout: msec_range[18] 0.000000 - 0.010000 : 0 2024-04-25T00:30:00.766 INFO:tasks.workunit.client.1.smithi174.stdout: 2024-04-25T00:30:00.766 INFO:tasks.workunit.client.1.smithi174.stdout: 2024-04-25T00:30:00.767 INFO:tasks.workunit.client.1.smithi174.stdout: 2024-04-25T00:30:00.767 INFO:tasks.workunit.client.1.smithi174.stdout: 2024-04-25T00:30:00.767 INFO:tasks.workunit.client.1.smithi174.stdout:0.5% User Time 2024-04-25T00:30:00.767 INFO:tasks.workunit.client.1.smithi174.stdout:53.0% System Time 2024-04-25T00:30:00.767 INFO:tasks.workunit.client.1.smithi174.stdout:53.5% CPU Utilization 2024-04-25T00:30:00.767 INFO:tasks.workunit.client.1.smithi174.stderr:+ cd .. 2024-04-25T00:30:01.079 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:30:01.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.211+0000 7faf23f2c640 1 Processor -- start 2024-04-25T00:30:01.210 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.211+0000 7faf23f2c640 1 -- start start 2024-04-25T00:30:01.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.211+0000 7faf23f2c640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7faf1c0a99f0 0x7faf1c153ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:01.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.211+0000 7faf23f2c640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf1c0c3910 con 0x7faf1c0a94a0 2024-04-25T00:30:01.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.211+0000 7faf23f2c640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf1c154530 con 0x7faf1c150d20 2024-04-25T00:30:01.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.211+0000 7faf23f2c640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7faf1c0c16e0 con 0x7faf1c0a99f0 2024-04-25T00:30:01.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.211+0000 7faf214c7640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7faf1c0a94a0 0x7faf1c1538e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:42702/0 (socket says 172.21.15.116:42702) 2024-04-25T00:30:01.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.211+0000 7faf214c7640 1 -- 172.21.15.116:0/4274490292 learned_addr learned my addr 172.21.15.116:0/4274490292 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:30:01.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.211+0000 7faf224c9640 1 --2- 172.21.15.116:0/4274490292 >> v2:172.21.15.174:3300/0 conn(0x7faf1c0a99f0 0x7faf1c153ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:01.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.211+0000 7faf20cc6640 1 -- 172.21.15.116:0/4274490292 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1685108938 0 0) 0x7faf1c0c3910 con 0x7faf1c0a94a0 2024-04-25T00:30:01.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.211+0000 7faf20cc6640 1 -- 172.21.15.116:0/4274490292 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faf08003530 con 0x7faf1c0a94a0 2024-04-25T00:30:01.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.211+0000 7faf20cc6640 1 -- 172.21.15.116:0/4274490292 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1163870731 0 0) 0x7faf08003530 con 0x7faf1c0a94a0 2024-04-25T00:30:01.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.211+0000 7faf20cc6640 1 -- 172.21.15.116:0/4274490292 >> v2:172.21.15.174:3300/0 conn(0x7faf1c0a99f0 msgr2=0x7faf1c153ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:01.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.211+0000 7faf20cc6640 1 --2- 172.21.15.116:0/4274490292 >> v2:172.21.15.174:3300/0 conn(0x7faf1c0a99f0 0x7faf1c153ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:01.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.211+0000 7faf20cc6640 1 -- 172.21.15.116:0/4274490292 >> v1:172.21.15.174:6789/0 conn(0x7faf1c150d20 legacy=0x7faf1c0c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:01.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.211+0000 7faf20cc6640 1 -- 172.21.15.116:0/4274490292 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf1c0c1400 con 0x7faf1c0a94a0 2024-04-25T00:30:01.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.211+0000 7faf20cc6640 1 -- 172.21.15.116:0/4274490292 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7faf0c001680 con 0x7faf1c0a94a0 2024-04-25T00:30:01.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.211+0000 7faf20cc6640 1 -- 172.21.15.116:0/4274490292 >> v1:172.21.15.116:6789/0 conn(0x7faf1c0a94a0 legacy=0x7faf1c1538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:01.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.211+0000 7faf20cc6640 1 --2- 172.21.15.116:0/4274490292 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faf080039c0 0x7faf08003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:01.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.211+0000 7faf20cc6640 1 --2- 172.21.15.116:0/4274490292 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf08004810 0x7faf08004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:01.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.211+0000 7faf20cc6640 1 --2- 172.21.15.116:0/4274490292 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf080051d0 0x7faf0800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:01.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.211+0000 7faf20cc6640 1 -- 172.21.15.116:0/4274490292 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7faf1c0c16e0 con 0x7faf08004810 2024-04-25T00:30:01.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.211+0000 7faf20cc6640 1 -- 172.21.15.116:0/4274490292 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7faf0800c070 con 0x7faf080051d0 2024-04-25T00:30:01.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.211+0000 7faf20cc6640 1 -- 172.21.15.116:0/4274490292 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7faf0800c270 con 0x7faf080039c0 2024-04-25T00:30:01.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.211+0000 7faf214c7640 1 --2- 172.21.15.116:0/4274490292 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf08004810 0x7faf08004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:01.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.211+0000 7faf21cc8640 1 --2- 172.21.15.116:0/4274490292 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faf080039c0 0x7faf08003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:01.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.211+0000 7faf214c7640 1 -- 172.21.15.116:0/4274490292 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faf080039c0 msgr2=0x7faf08003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:01.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.211+0000 7faf224c9640 1 --2- 172.21.15.116:0/4274490292 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf080051d0 0x7faf0800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:01.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.211+0000 7faf214c7640 1 --2- 172.21.15.116:0/4274490292 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faf080039c0 0x7faf08003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:01.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.211+0000 7faf214c7640 1 -- 172.21.15.116:0/4274490292 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf080051d0 msgr2=0x7faf0800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:01.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.211+0000 7faf214c7640 1 --2- 172.21.15.116:0/4274490292 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf080051d0 0x7faf0800b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:01.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.211+0000 7faf214c7640 1 -- 172.21.15.116:0/4274490292 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf08003530 con 0x7faf08004810 2024-04-25T00:30:01.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.211+0000 7faf224c9640 1 -- 172.21.15.116:0/4274490292 reap_dead start 2024-04-25T00:30:01.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.211+0000 7faf21cc8640 1 --2- 172.21.15.116:0/4274490292 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faf080039c0 0x7faf08003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:30:01.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.215+0000 7faf214c7640 1 --2- 172.21.15.116:0/4274490292 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf08004810 0x7faf08004c90 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7faf0c00c210 tx=0x7faf0c01f370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caf1323c167016c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:01.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.215+0000 7faf20cc6640 1 -- 172.21.15.116:0/4274490292 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf0c007d40 con 0x7faf08004810 2024-04-25T00:30:01.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.215+0000 7faf20cc6640 1 -- 172.21.15.116:0/4274490292 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf0c005b00 con 0x7faf08004810 2024-04-25T00:30:01.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.215+0000 7faf20cc6640 1 -- 172.21.15.116:0/4274490292 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf0c007070 con 0x7faf08004810 2024-04-25T00:30:01.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.215+0000 7faf23f2c640 1 -- 172.21.15.116:0/4274490292 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf08004810 msgr2=0x7faf08004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:01.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.215+0000 7faf23f2c640 1 --2- 172.21.15.116:0/4274490292 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf08004810 0x7faf08004c90 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7faf0c00c210 tx=0x7faf0c01f370 comp rx=0 tx=0).stop 2024-04-25T00:30:01.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.215+0000 7faf23f2c640 1 -- 172.21.15.116:0/4274490292 shutdown_connections 2024-04-25T00:30:01.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.215+0000 7faf23f2c640 1 --2- 172.21.15.116:0/4274490292 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf08004810 0x7faf08004c90 unknown :-1 s=CLOSED pgs=1700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:01.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.215+0000 7faf23f2c640 1 -- 172.21.15.116:0/4274490292 >> 172.21.15.116:0/4274490292 conn(0x7faf1c0c3b00 msgr2=0x7faf1c0c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:01.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.215+0000 7faf23f2c640 1 -- 172.21.15.116:0/4274490292 shutdown_connections 2024-04-25T00:30:01.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.215+0000 7faf23f2c640 1 -- 172.21.15.116:0/4274490292 wait complete. 2024-04-25T00:30:01.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.215+0000 7faf23f2c640 1 Processor -- start 2024-04-25T00:30:01.217 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.215+0000 7faf23f2c640 1 -- start start 2024-04-25T00:30:01.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.215+0000 7faf23f2c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf08004810 0x7faf1c0c0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:01.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.215+0000 7faf23f2c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf1c072aa0 0x7faf1c1ef100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:01.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.215+0000 7faf23f2c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faf1c1ef640 0x7faf1c156620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:01.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.215+0000 7faf23f2c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7faf1c06ec20 con 0x7faf08004810 2024-04-25T00:30:01.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.215+0000 7faf23f2c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7faf1c06e940 con 0x7faf1c072aa0 2024-04-25T00:30:01.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.215+0000 7faf23f2c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7faf1c0c2a10 con 0x7faf1c1ef640 2024-04-25T00:30:01.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.215+0000 7faf21cc8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf08004810 0x7faf1c0c0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:01.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.215+0000 7faf21cc8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf08004810 0x7faf1c0c0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41540/0 (socket says 172.21.15.116:41540) 2024-04-25T00:30:01.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.215+0000 7faf21cc8640 1 -- 172.21.15.116:0/2708748006 learned_addr learned my addr 172.21.15.116:0/2708748006 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:01.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.219+0000 7faf214c7640 1 --2- 172.21.15.116:0/2708748006 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf1c072aa0 0x7faf1c1ef100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:01.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.219+0000 7faf224c9640 1 --2- 172.21.15.116:0/2708748006 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faf1c1ef640 0x7faf1c156620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:01.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.219+0000 7faf21cc8640 1 -- 172.21.15.116:0/2708748006 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faf1c1ef640 msgr2=0x7faf1c156620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:01.220 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.219+0000 7faf21cc8640 1 --2- 172.21.15.116:0/2708748006 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faf1c1ef640 0x7faf1c156620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:01.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.219+0000 7faf21cc8640 1 -- 172.21.15.116:0/2708748006 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf1c072aa0 msgr2=0x7faf1c1ef100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:01.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.219+0000 7faf21cc8640 1 --2- 172.21.15.116:0/2708748006 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf1c072aa0 0x7faf1c1ef100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:01.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.219+0000 7faf21cc8640 1 -- 172.21.15.116:0/2708748006 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf1c071f10 con 0x7faf08004810 2024-04-25T00:30:01.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.219+0000 7faf224c9640 1 --2- 172.21.15.116:0/2708748006 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faf1c1ef640 0x7faf1c156620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:30:01.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.219+0000 7faf21cc8640 1 --2- 172.21.15.116:0/2708748006 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf08004810 0x7faf1c0c0690 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7faf1800d4d0 tx=0x7faf1800d990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:01.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.219+0000 7faf127fc640 1 -- 172.21.15.116:0/2708748006 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf18019070 con 0x7faf08004810 2024-04-25T00:30:01.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.219+0000 7faf127fc640 1 -- 172.21.15.116:0/2708748006 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf180054a0 con 0x7faf08004810 2024-04-25T00:30:01.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.219+0000 7faf23f2c640 1 -- 172.21.15.116:0/2708748006 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf1c157310 con 0x7faf08004810 2024-04-25T00:30:01.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.219+0000 7faf127fc640 1 -- 172.21.15.116:0/2708748006 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf18007270 con 0x7faf08004810 2024-04-25T00:30:01.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.219+0000 7faf23f2c640 1 -- 172.21.15.116:0/2708748006 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf1c0c5070 con 0x7faf08004810 2024-04-25T00:30:01.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.219+0000 7faf23f2c640 1 -- 172.21.15.116:0/2708748006 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf1c068350 con 0x7faf08004810 2024-04-25T00:30:01.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.219+0000 7faf127fc640 1 -- 172.21.15.116:0/2708748006 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7faf1801c070 con 0x7faf08004810 2024-04-25T00:30:01.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.219+0000 7faf127fc640 1 --2- 172.21.15.116:0/2708748006 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faf000678c0 0x7faf00069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:01.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.219+0000 7faf127fc640 1 -- 172.21.15.116:0/2708748006 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 10536+0+0 (secure 0 0 0) 0x7faf18086630 con 0x7faf08004810 2024-04-25T00:30:01.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.219+0000 7faf214c7640 1 --2- 172.21.15.116:0/2708748006 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faf000678c0 0x7faf00069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:01.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.219+0000 7faf214c7640 1 --2- 172.21.15.116:0/2708748006 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faf000678c0 0x7faf00069d80 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7faf0c0af000 tx=0x7faf0c0aa000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:01.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.223+0000 7faf127fc640 1 -- 172.21.15.116:0/2708748006 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7faf1c071f10 con 0x7faf08004810 2024-04-25T00:30:01.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.419+0000 7faf23f2c640 1 -- 172.21.15.116:0/2708748006 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faf1c157920 con 0x7faf08004810 2024-04-25T00:30:01.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.419+0000 7faf127fc640 1 -- 172.21.15.116:0/2708748006 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 237 v237) ==== 95+0+6510 (secure 0 0 0) 0x7faf18055f90 con 0x7faf08004810 2024-04-25T00:30:01.421 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:30:01.421 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":237,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6764,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3873739116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3873739116},{"type":"v1","addr":"172.21.15.174:6837","nonce":3873739116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":237}],"filesystems":[{"mdsmap":{"epoch":236,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:29:54.168353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8847},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8847":{"gid":8847,"name":"c","rank":0,"incarnation":233,"state":"up:active","state_seq":17,"addr":"172.21.15.116:6833/2662028892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2662028892},{"type":"v1","addr":"172.21.15.116:6833","nonce":2662028892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8847,"qdb_cluster":[8847]},"id":1},{"mdsmap":{"epoch":231,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:28:53.808539+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8772":{"gid":8772,"name":"a","rank":0,"incarnation":228,"state":"up:active","state_seq":20,"addr":"172.21.15.116:6835/3499878473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3499878473},{"type":"v1","addr":"172.21.15.116:6835","nonce":3499878473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8772,"qdb_cluster":[8772]},"id":2}]} 2024-04-25T00:30:01.421 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 237 2024-04-25T00:30:01.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.423+0000 7faf23f2c640 1 -- 172.21.15.116:0/2708748006 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faf000678c0 msgr2=0x7faf00069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:01.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.423+0000 7faf23f2c640 1 --2- 172.21.15.116:0/2708748006 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faf000678c0 0x7faf00069d80 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7faf0c0af000 tx=0x7faf0c0aa000 comp rx=0 tx=0).stop 2024-04-25T00:30:01.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.423+0000 7faf23f2c640 1 -- 172.21.15.116:0/2708748006 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf08004810 msgr2=0x7faf1c0c0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:01.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.423+0000 7faf23f2c640 1 --2- 172.21.15.116:0/2708748006 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf08004810 0x7faf1c0c0690 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7faf1800d4d0 tx=0x7faf1800d990 comp rx=0 tx=0).stop 2024-04-25T00:30:01.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.423+0000 7faf23f2c640 1 -- 172.21.15.116:0/2708748006 shutdown_connections 2024-04-25T00:30:01.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.423+0000 7faf23f2c640 1 --2- 172.21.15.116:0/2708748006 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faf000678c0 0x7faf00069d80 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:01.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.423+0000 7faf23f2c640 1 --2- 172.21.15.116:0/2708748006 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faf1c1ef640 0x7faf1c156620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:01.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.423+0000 7faf23f2c640 1 --2- 172.21.15.116:0/2708748006 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf1c072aa0 0x7faf1c1ef100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:01.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.423+0000 7faf23f2c640 1 --2- 172.21.15.116:0/2708748006 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf08004810 0x7faf1c0c0690 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:01.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.423+0000 7faf23f2c640 1 -- 172.21.15.116:0/2708748006 >> 172.21.15.116:0/2708748006 conn(0x7faf1c0c3b00 msgr2=0x7faf1c06cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:01.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.423+0000 7faf23f2c640 1 -- 172.21.15.116:0/2708748006 shutdown_connections 2024-04-25T00:30:01.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:01.423+0000 7faf23f2c640 1 -- 172.21.15.116:0/2708748006 wait complete. 2024-04-25T00:30:01.440 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:30:03.441 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:30:03.573 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.575+0000 7f7d74e44640 1 Processor -- start 2024-04-25T00:30:03.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.575+0000 7f7d74e44640 1 -- start start 2024-04-25T00:30:03.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.575+0000 7f7d74e44640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d7014c720 0x7f7d701563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:03.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.575+0000 7f7d74e44640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f7d700a94a0 0x7f7d700c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:03.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.575+0000 7f7d74e44640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d70071220 con 0x7f7d700c2f50 2024-04-25T00:30:03.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.575+0000 7f7d74e44640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f7d70153460 con 0x7f7d700a94a0 2024-04-25T00:30:03.585 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.575+0000 7f7d74e44640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f7d70152420 con 0x7f7d7014c720 2024-04-25T00:30:03.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.575+0000 7f7d6ed9d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f7d700c2f50 0x7f7d70071680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:42712/0 (socket says 172.21.15.116:42712) 2024-04-25T00:30:03.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.575+0000 7f7d6ed9d640 1 -- 172.21.15.116:0/423277523 learned_addr learned my addr 172.21.15.116:0/423277523 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:30:03.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.575+0000 7f7d6e59c640 1 --2- 172.21.15.116:0/423277523 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d7014c720 0x7f7d701563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:03.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.575+0000 7f7d65d9b640 1 --2- 172.21.15.116:0/423277523 >> v2:172.21.15.174:3300/0 conn(0x7f7d700a94a0 0x7f7d700c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:03.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.575+0000 7f7d6dd9b640 1 -- 172.21.15.116:0/423277523 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1940301684 0 0) 0x7f7d70071220 con 0x7f7d700c2f50 2024-04-25T00:30:03.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.575+0000 7f7d6dd9b640 1 -- 172.21.15.116:0/423277523 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d5c005740 con 0x7f7d700c2f50 2024-04-25T00:30:03.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.575+0000 7f7d6dd9b640 1 -- 172.21.15.116:0/423277523 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1974896876 0 0) 0x7f7d5c005740 con 0x7f7d700c2f50 2024-04-25T00:30:03.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.575+0000 7f7d6dd9b640 1 -- 172.21.15.116:0/423277523 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d7014c720 msgr2=0x7f7d701563d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:03.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.575+0000 7f7d6dd9b640 1 --2- 172.21.15.116:0/423277523 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d7014c720 0x7f7d701563d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:03.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.575+0000 7f7d6dd9b640 1 -- 172.21.15.116:0/423277523 >> v2:172.21.15.174:3300/0 conn(0x7f7d700a94a0 msgr2=0x7f7d700c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:03.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.575+0000 7f7d6dd9b640 1 --2- 172.21.15.116:0/423277523 >> v2:172.21.15.174:3300/0 conn(0x7f7d700a94a0 0x7f7d700c2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:03.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.575+0000 7f7d6dd9b640 1 -- 172.21.15.116:0/423277523 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d7014cb00 con 0x7f7d700c2f50 2024-04-25T00:30:03.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.575+0000 7f7d6dd9b640 1 -- 172.21.15.116:0/423277523 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f7d68001a00 con 0x7f7d700c2f50 2024-04-25T00:30:03.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.575+0000 7f7d6dd9b640 1 -- 172.21.15.116:0/423277523 >> v1:172.21.15.116:6789/0 conn(0x7f7d700c2f50 legacy=0x7f7d70071680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:03.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.575+0000 7f7d6dd9b640 1 --2- 172.21.15.116:0/423277523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d5c005bd0 0x7f7d5c005ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:03.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.575+0000 7f7d6dd9b640 1 --2- 172.21.15.116:0/423277523 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d5c006a20 0x7f7d5c006ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:03.587 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.575+0000 7f7d6dd9b640 1 --2- 172.21.15.116:0/423277523 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d7014c720 0x7f7d5c0073e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:03.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.575+0000 7f7d6dd9b640 1 -- 172.21.15.116:0/423277523 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7d70153460 con 0x7f7d5c005bd0 2024-04-25T00:30:03.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.575+0000 7f7d6dd9b640 1 -- 172.21.15.116:0/423277523 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7d70152420 con 0x7f7d5c006a20 2024-04-25T00:30:03.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.575+0000 7f7d6dd9b640 1 -- 172.21.15.116:0/423277523 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f7d5c00dbe0 con 0x7f7d7014c720 2024-04-25T00:30:03.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.575+0000 7f7d6e59c640 1 --2- 172.21.15.116:0/423277523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d5c005bd0 0x7f7d5c005ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:03.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.575+0000 7f7d6ed9d640 1 --2- 172.21.15.116:0/423277523 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d7014c720 0x7f7d5c0073e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:03.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.575+0000 7f7d6e59c640 1 -- 172.21.15.116:0/423277523 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d7014c720 msgr2=0x7f7d5c0073e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:03.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.575+0000 7f7d6e59c640 1 --2- 172.21.15.116:0/423277523 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d7014c720 0x7f7d5c0073e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:03.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.575+0000 7f7d6e59c640 1 -- 172.21.15.116:0/423277523 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d5c006a20 msgr2=0x7f7d5c006ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:03.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.575+0000 7f7d65d9b640 1 --2- 172.21.15.116:0/423277523 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d5c006a20 0x7f7d5c006ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:03.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.575+0000 7f7d6e59c640 1 --2- 172.21.15.116:0/423277523 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d5c006a20 0x7f7d5c006ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:03.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.575+0000 7f7d6e59c640 1 -- 172.21.15.116:0/423277523 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d5c005740 con 0x7f7d5c005bd0 2024-04-25T00:30:03.588 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.579+0000 7f7d6e59c640 1 --2- 172.21.15.116:0/423277523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d5c005bd0 0x7f7d5c005ff0 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f7d60009af0 tx=0x7f7d6001ca00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8c8786cff9667d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:03.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.579+0000 7f7d6dd9b640 1 -- 172.21.15.116:0/423277523 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d60009690 con 0x7f7d5c005bd0 2024-04-25T00:30:03.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.579+0000 7f7d6dd9b640 1 -- 172.21.15.116:0/423277523 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d600a6420 con 0x7f7d5c005bd0 2024-04-25T00:30:03.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.579+0000 7f7d6dd9b640 1 -- 172.21.15.116:0/423277523 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d600a67f0 con 0x7f7d5c005bd0 2024-04-25T00:30:03.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.579+0000 7f7d74e44640 1 -- 172.21.15.116:0/423277523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d5c005bd0 msgr2=0x7f7d5c005ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:03.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.579+0000 7f7d74e44640 1 --2- 172.21.15.116:0/423277523 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d5c005bd0 0x7f7d5c005ff0 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f7d60009af0 tx=0x7f7d6001ca00 comp rx=0 tx=0).stop 2024-04-25T00:30:03.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.579+0000 7f7d6ed9d640 1 -- 172.21.15.116:0/423277523 reap_dead start 2024-04-25T00:30:03.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.579+0000 7f7d74e44640 1 -- 172.21.15.116:0/423277523 shutdown_connections 2024-04-25T00:30:03.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.579+0000 7f7d74e44640 1 -- 172.21.15.116:0/423277523 >> 172.21.15.116:0/423277523 conn(0x7f7d700c3b00 msgr2=0x7f7d700c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:03.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.579+0000 7f7d74e44640 1 -- 172.21.15.116:0/423277523 shutdown_connections 2024-04-25T00:30:03.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.579+0000 7f7d74e44640 1 -- 172.21.15.116:0/423277523 wait complete. 2024-04-25T00:30:03.589 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.579+0000 7f7d74e44640 1 Processor -- start 2024-04-25T00:30:03.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.579+0000 7f7d74e44640 1 -- start start 2024-04-25T00:30:03.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.579+0000 7f7d74e44640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d5c005bd0 0x7f7d700c4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:03.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.579+0000 7f7d74e44640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d701eb5c0 0x7f7d700c5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:03.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.579+0000 7f7d74e44640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d701edd10 0x7f7d701ee190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:03.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.579+0000 7f7d74e44640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7d70152c60 con 0x7f7d701eb5c0 2024-04-25T00:30:03.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.579+0000 7f7d74e44640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7d70152980 con 0x7f7d5c005bd0 2024-04-25T00:30:03.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.579+0000 7f7d74e44640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f7d70152f40 con 0x7f7d701edd10 2024-04-25T00:30:03.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.579+0000 7f7d65d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d701eb5c0 0x7f7d700c5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:03.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.579+0000 7f7d65d9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d701eb5c0 0x7f7d700c5fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:41560/0 (socket says 172.21.15.116:41560) 2024-04-25T00:30:03.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.579+0000 7f7d65d9b640 1 -- 172.21.15.116:0/1836298949 learned_addr learned my addr 172.21.15.116:0/1836298949 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:03.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.579+0000 7f7d6ed9d640 1 --2- 172.21.15.116:0/1836298949 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d701edd10 0x7f7d701ee190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:03.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.579+0000 7f7d6e59c640 1 --2- 172.21.15.116:0/1836298949 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d5c005bd0 0x7f7d700c4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:03.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.579+0000 7f7d65d9b640 1 -- 172.21.15.116:0/1836298949 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d701edd10 msgr2=0x7f7d701ee190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:03.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.579+0000 7f7d65d9b640 1 --2- 172.21.15.116:0/1836298949 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d701edd10 0x7f7d701ee190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:03.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.579+0000 7f7d65d9b640 1 -- 172.21.15.116:0/1836298949 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d5c005bd0 msgr2=0x7f7d700c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:03.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.579+0000 7f7d65d9b640 1 --2- 172.21.15.116:0/1836298949 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d5c005bd0 0x7f7d700c4e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:03.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.579+0000 7f7d65d9b640 1 -- 172.21.15.116:0/1836298949 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d700c6ad0 con 0x7f7d701eb5c0 2024-04-25T00:30:03.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.579+0000 7f7d65d9b640 1 --2- 172.21.15.116:0/1836298949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d701eb5c0 0x7f7d700c5fb0 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f7d58010900 tx=0x7f7d58010dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:03.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.579+0000 7f7d677fe640 1 -- 172.21.15.116:0/1836298949 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d5800f040 con 0x7f7d701eb5c0 2024-04-25T00:30:03.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.579+0000 7f7d677fe640 1 -- 172.21.15.116:0/1836298949 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d5800ec20 con 0x7f7d701eb5c0 2024-04-25T00:30:03.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.579+0000 7f7d74e44640 1 -- 172.21.15.116:0/1836298949 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d700c6d90 con 0x7f7d701eb5c0 2024-04-25T00:30:03.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.579+0000 7f7d677fe640 1 -- 172.21.15.116:0/1836298949 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d5801c400 con 0x7f7d701eb5c0 2024-04-25T00:30:03.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.583+0000 7f7d74e44640 1 -- 172.21.15.116:0/1836298949 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d701ed7f0 con 0x7f7d701eb5c0 2024-04-25T00:30:03.591 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.583+0000 7f7d74e44640 1 -- 172.21.15.116:0/1836298949 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d7013c6b0 con 0x7f7d701eb5c0 2024-04-25T00:30:03.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.583+0000 7f7d677fe640 1 -- 172.21.15.116:0/1836298949 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7d5801c5a0 con 0x7f7d701eb5c0 2024-04-25T00:30:03.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.583+0000 7f7d677fe640 1 --2- 172.21.15.116:0/1836298949 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7d500678c0 0x7f7d50069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:03.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.583+0000 7f7d677fe640 1 -- 172.21.15.116:0/1836298949 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 10536+0+0 (secure 0 0 0) 0x7f7d58087560 con 0x7f7d701eb5c0 2024-04-25T00:30:03.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.583+0000 7f7d6e59c640 1 --2- 172.21.15.116:0/1836298949 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7d500678c0 0x7f7d50069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:03.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.583+0000 7f7d6e59c640 1 --2- 172.21.15.116:0/1836298949 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7d500678c0 0x7f7d50069d80 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f7d600ad000 tx=0x7f7d600b2040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:03.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.587+0000 7f7d677fe640 1 -- 172.21.15.116:0/1836298949 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f7d7013c6b0 con 0x7f7d701eb5c0 2024-04-25T00:30:03.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.783+0000 7f7d74e44640 1 -- 172.21.15.116:0/1836298949 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7d70068350 con 0x7f7d701eb5c0 2024-04-25T00:30:03.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.787+0000 7f7d677fe640 1 -- 172.21.15.116:0/1836298949 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 237 v237) ==== 95+0+6510 (secure 0 0 0) 0x7f7d58018030 con 0x7f7d701eb5c0 2024-04-25T00:30:03.786 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:30:03.786 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":237,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6764,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3873739116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3873739116},{"type":"v1","addr":"172.21.15.174:6837","nonce":3873739116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":237}],"filesystems":[{"mdsmap":{"epoch":236,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:29:54.168353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8847},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8847":{"gid":8847,"name":"c","rank":0,"incarnation":233,"state":"up:active","state_seq":17,"addr":"172.21.15.116:6833/2662028892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2662028892},{"type":"v1","addr":"172.21.15.116:6833","nonce":2662028892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8847,"qdb_cluster":[8847]},"id":1},{"mdsmap":{"epoch":231,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:28:53.808539+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8772":{"gid":8772,"name":"a","rank":0,"incarnation":228,"state":"up:active","state_seq":20,"addr":"172.21.15.116:6835/3499878473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3499878473},{"type":"v1","addr":"172.21.15.116:6835","nonce":3499878473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8772,"qdb_cluster":[8772]},"id":2}]} 2024-04-25T00:30:03.787 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 237 2024-04-25T00:30:03.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.787+0000 7f7d74e44640 1 -- 172.21.15.116:0/1836298949 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7d500678c0 msgr2=0x7f7d50069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:03.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.787+0000 7f7d74e44640 1 --2- 172.21.15.116:0/1836298949 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7d500678c0 0x7f7d50069d80 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f7d600ad000 tx=0x7f7d600b2040 comp rx=0 tx=0).stop 2024-04-25T00:30:03.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.787+0000 7f7d74e44640 1 -- 172.21.15.116:0/1836298949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d701eb5c0 msgr2=0x7f7d700c5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:03.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.787+0000 7f7d74e44640 1 --2- 172.21.15.116:0/1836298949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d701eb5c0 0x7f7d700c5fb0 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f7d58010900 tx=0x7f7d58010dc0 comp rx=0 tx=0).stop 2024-04-25T00:30:03.789 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.787+0000 7f7d74e44640 1 -- 172.21.15.116:0/1836298949 shutdown_connections 2024-04-25T00:30:03.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.787+0000 7f7d74e44640 1 --2- 172.21.15.116:0/1836298949 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7d500678c0 0x7f7d50069d80 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:03.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.787+0000 7f7d74e44640 1 --2- 172.21.15.116:0/1836298949 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7d701edd10 0x7f7d701ee190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:03.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.787+0000 7f7d74e44640 1 --2- 172.21.15.116:0/1836298949 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7d701eb5c0 0x7f7d700c5fb0 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:03.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.787+0000 7f7d74e44640 1 --2- 172.21.15.116:0/1836298949 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7d5c005bd0 0x7f7d700c4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:03.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.787+0000 7f7d74e44640 1 -- 172.21.15.116:0/1836298949 >> 172.21.15.116:0/1836298949 conn(0x7f7d700c3b00 msgr2=0x7f7d70150820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:03.790 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.787+0000 7f7d74e44640 1 -- 172.21.15.116:0/1836298949 shutdown_connections 2024-04-25T00:30:03.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:03.787+0000 7f7d74e44640 1 -- 172.21.15.116:0/1836298949 wait complete. 2024-04-25T00:30:03.804 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:30:05.805 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:30:05.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.935+0000 7fed8ec70640 1 Processor -- start 2024-04-25T00:30:05.937 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.935+0000 7fed8ec70640 1 -- start start 2024-04-25T00:30:05.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.939+0000 7fed8ec70640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fed88153f40 0x7fed8806e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:05.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.939+0000 7fed8ec70640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fed880c2ea0 0x7fed8815a610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:05.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.939+0000 7fed8ec70640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed88071a30 con 0x7fed880c2a10 2024-04-25T00:30:05.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.939+0000 7fed8ec70640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fed880725d0 con 0x7fed88153f40 2024-04-25T00:30:05.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.939+0000 7fed8ec70640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fed88071ca0 con 0x7fed880c2ea0 2024-04-25T00:30:05.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.939+0000 7fed8ca0c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fed88153f40 0x7fed8806e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:05.938 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.939+0000 7fed8ca0c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fed88153f40 0x7fed8806e790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36954/0 (socket says 172.21.15.116:36954) 2024-04-25T00:30:05.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.939+0000 7fed8ca0c640 1 -- 172.21.15.116:0/2783808285 learned_addr learned my addr 172.21.15.116:0/2783808285 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:05.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.939+0000 7fed87fff640 1 --1- 172.21.15.116:0/2783808285 >> v1:172.21.15.116:6789/0 conn(0x7fed880c2a10 0x7fed88159f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:53238/0 (socket says 172.21.15.116:53238) 2024-04-25T00:30:05.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.939+0000 7fed8d20d640 1 --2- 172.21.15.116:0/2783808285 >> v2:172.21.15.174:3300/0 conn(0x7fed880c2ea0 0x7fed8815a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:05.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.939+0000 7fed8ca0c640 1 -- 172.21.15.116:0/2783808285 >> v2:172.21.15.174:3300/0 conn(0x7fed880c2ea0 msgr2=0x7fed8815a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:05.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.939+0000 7fed8ca0c640 1 --2- 172.21.15.116:0/2783808285 >> v2:172.21.15.174:3300/0 conn(0x7fed880c2ea0 0x7fed8815a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:05.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.939+0000 7fed8ca0c640 1 -- 172.21.15.116:0/2783808285 >> v1:172.21.15.116:6789/0 conn(0x7fed880c2a10 legacy=0x7fed88159f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:05.939 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.939+0000 7fed8ca0c640 1 -- 172.21.15.116:0/2783808285 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed88071f80 con 0x7fed88153f40 2024-04-25T00:30:05.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.939+0000 7fed8ca0c640 1 --2- 172.21.15.116:0/2783808285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed88153f40 0x7fed8806e790 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7fed8815e320 tx=0x7fed7001c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=321115c037124027 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:05.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.939+0000 7fed877fe640 1 -- 172.21.15.116:0/2783808285 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed700b0070 con 0x7fed88153f40 2024-04-25T00:30:05.940 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.939+0000 7fed877fe640 1 -- 172.21.15.116:0/2783808285 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed700ab600 con 0x7fed88153f40 2024-04-25T00:30:05.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.939+0000 7fed877fe640 1 -- 172.21.15.116:0/2783808285 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed700ab9d0 con 0x7fed88153f40 2024-04-25T00:30:05.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.943+0000 7fed8ec70640 1 -- 172.21.15.116:0/2783808285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed88153f40 msgr2=0x7fed8806e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:05.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.943+0000 7fed8ec70640 1 --2- 172.21.15.116:0/2783808285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed88153f40 0x7fed8806e790 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7fed8815e320 tx=0x7fed7001c490 comp rx=0 tx=0).stop 2024-04-25T00:30:05.941 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.943+0000 7fed8ec70640 1 -- 172.21.15.116:0/2783808285 shutdown_connections 2024-04-25T00:30:05.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.943+0000 7fed8ec70640 1 --2- 172.21.15.116:0/2783808285 >> v2:172.21.15.174:3300/0 conn(0x7fed880c2ea0 0x7fed8815a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:05.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.943+0000 7fed8ec70640 1 --2- 172.21.15.116:0/2783808285 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed88153f40 0x7fed8806e790 unknown :-1 s=CLOSED pgs=1706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:05.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.943+0000 7fed8ec70640 1 -- 172.21.15.116:0/2783808285 >> 172.21.15.116:0/2783808285 conn(0x7fed880c3b00 msgr2=0x7fed880c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:05.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.943+0000 7fed8ec70640 1 -- 172.21.15.116:0/2783808285 shutdown_connections 2024-04-25T00:30:05.942 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.943+0000 7fed8ec70640 1 -- 172.21.15.116:0/2783808285 wait complete. 2024-04-25T00:30:05.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.943+0000 7fed8ec70640 1 Processor -- start 2024-04-25T00:30:05.943 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.943+0000 7fed8ec70640 1 -- start start 2024-04-25T00:30:05.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.943+0000 7fed8ec70640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed880c2a10 0x7fed880c6630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:05.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.943+0000 7fed8ec70640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fed880c2ea0 0x7fed88146ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:05.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.943+0000 7fed8ec70640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fed88153f40 0x7fed88146ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:05.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.943+0000 7fed8ec70640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fed880c3840 con 0x7fed880c2a10 2024-04-25T00:30:05.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.943+0000 7fed8ec70640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fed880c36c0 con 0x7fed880c2ea0 2024-04-25T00:30:05.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.943+0000 7fed8ec70640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fed88166e30 con 0x7fed88153f40 2024-04-25T00:30:05.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.943+0000 7fed8ca0c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed880c2a10 0x7fed880c6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:05.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.943+0000 7fed8ca0c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed880c2a10 0x7fed880c6630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36964/0 (socket says 172.21.15.116:36964) 2024-04-25T00:30:05.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.943+0000 7fed8ca0c640 1 -- 172.21.15.116:0/3769237484 learned_addr learned my addr 172.21.15.116:0/3769237484 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:05.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.943+0000 7fed87fff640 1 --2- 172.21.15.116:0/3769237484 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fed880c2ea0 0x7fed88146ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:05.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.943+0000 7fed8ca0c640 1 -- 172.21.15.116:0/3769237484 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fed88153f40 msgr2=0x7fed88146ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:05.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.943+0000 7fed8ca0c640 1 --2- 172.21.15.116:0/3769237484 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fed88153f40 0x7fed88146ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:05.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.943+0000 7fed8ca0c640 1 -- 172.21.15.116:0/3769237484 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fed880c2ea0 msgr2=0x7fed88146ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:05.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.943+0000 7fed8ca0c640 1 --2- 172.21.15.116:0/3769237484 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fed880c2ea0 0x7fed88146ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:05.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.943+0000 7fed8ca0c640 1 -- 172.21.15.116:0/3769237484 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed88158e90 con 0x7fed880c2a10 2024-04-25T00:30:05.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.943+0000 7fed87fff640 1 --2- 172.21.15.116:0/3769237484 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fed880c2ea0 0x7fed88146ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:30:05.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.943+0000 7fed8ca0c640 1 --2- 172.21.15.116:0/3769237484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed880c2a10 0x7fed880c6630 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7fed700ae040 tx=0x7fed70019230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:05.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.943+0000 7fed857fa640 1 -- 172.21.15.116:0/3769237484 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed700b0040 con 0x7fed880c2a10 2024-04-25T00:30:05.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.943+0000 7fed857fa640 1 -- 172.21.15.116:0/3769237484 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed700b4420 con 0x7fed880c2a10 2024-04-25T00:30:05.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.943+0000 7fed857fa640 1 -- 172.21.15.116:0/3769237484 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed700b4810 con 0x7fed880c2a10 2024-04-25T00:30:05.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.943+0000 7fed8ec70640 1 -- 172.21.15.116:0/3769237484 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed88157eb0 con 0x7fed880c2a10 2024-04-25T00:30:05.946 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.943+0000 7fed8ec70640 1 -- 172.21.15.116:0/3769237484 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed88159150 con 0x7fed880c2a10 2024-04-25T00:30:05.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.947+0000 7fed8ec70640 1 -- 172.21.15.116:0/3769237484 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed88068350 con 0x7fed880c2a10 2024-04-25T00:30:05.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.947+0000 7fed857fa640 1 -- 172.21.15.116:0/3769237484 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fed700b49b0 con 0x7fed880c2a10 2024-04-25T00:30:05.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.947+0000 7fed857fa640 1 --2- 172.21.15.116:0/3769237484 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fed680678c0 0x7fed68069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:05.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.947+0000 7fed857fa640 1 -- 172.21.15.116:0/3769237484 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 10536+0+0 (secure 0 0 0) 0x7fed70120bd0 con 0x7fed880c2a10 2024-04-25T00:30:05.947 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.947+0000 7fed87fff640 1 --2- 172.21.15.116:0/3769237484 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fed680678c0 0x7fed68069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:05.948 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.947+0000 7fed87fff640 1 --2- 172.21.15.116:0/3769237484 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fed680678c0 0x7fed68069d80 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fed78007fd0 tx=0x7fed78009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:05.951 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:05.951+0000 7fed857fa640 1 -- 172.21.15.116:0/3769237484 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fed70016110 con 0x7fed880c2a10 2024-04-25T00:30:06.138 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:06.139+0000 7fed8ec70640 1 -- 172.21.15.116:0/3769237484 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fed880c16e0 con 0x7fed880c2a10 2024-04-25T00:30:06.139 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:06.139+0000 7fed857fa640 1 -- 172.21.15.116:0/3769237484 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 237 v237) ==== 95+0+6510 (secure 0 0 0) 0x7fed700f0410 con 0x7fed880c2a10 2024-04-25T00:30:06.139 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:30:06.139 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":237,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6764,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3873739116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3873739116},{"type":"v1","addr":"172.21.15.174:6837","nonce":3873739116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":237}],"filesystems":[{"mdsmap":{"epoch":236,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:29:54.168353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8847},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8847":{"gid":8847,"name":"c","rank":0,"incarnation":233,"state":"up:active","state_seq":17,"addr":"172.21.15.116:6833/2662028892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2662028892},{"type":"v1","addr":"172.21.15.116:6833","nonce":2662028892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8847,"qdb_cluster":[8847]},"id":1},{"mdsmap":{"epoch":231,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:28:53.808539+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8772":{"gid":8772,"name":"a","rank":0,"incarnation":228,"state":"up:active","state_seq":20,"addr":"172.21.15.116:6835/3499878473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3499878473},{"type":"v1","addr":"172.21.15.116:6835","nonce":3499878473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8772,"qdb_cluster":[8772]},"id":2}]} 2024-04-25T00:30:06.140 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 237 2024-04-25T00:30:06.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:06.139+0000 7fed8ec70640 1 -- 172.21.15.116:0/3769237484 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fed680678c0 msgr2=0x7fed68069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:06.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:06.139+0000 7fed8ec70640 1 --2- 172.21.15.116:0/3769237484 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fed680678c0 0x7fed68069d80 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fed78007fd0 tx=0x7fed78009040 comp rx=0 tx=0).stop 2024-04-25T00:30:06.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:06.139+0000 7fed8ec70640 1 -- 172.21.15.116:0/3769237484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed880c2a10 msgr2=0x7fed880c6630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:06.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:06.139+0000 7fed8ec70640 1 --2- 172.21.15.116:0/3769237484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed880c2a10 0x7fed880c6630 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7fed700ae040 tx=0x7fed70019230 comp rx=0 tx=0).stop 2024-04-25T00:30:06.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:06.139+0000 7fed8ec70640 1 -- 172.21.15.116:0/3769237484 shutdown_connections 2024-04-25T00:30:06.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:06.139+0000 7fed8ec70640 1 --2- 172.21.15.116:0/3769237484 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fed680678c0 0x7fed68069d80 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:06.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:06.139+0000 7fed8ec70640 1 --2- 172.21.15.116:0/3769237484 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fed88153f40 0x7fed88146ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:06.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:06.139+0000 7fed8ec70640 1 --2- 172.21.15.116:0/3769237484 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fed880c2ea0 0x7fed88146ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:06.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:06.139+0000 7fed8ec70640 1 --2- 172.21.15.116:0/3769237484 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fed880c2a10 0x7fed880c6630 unknown :-1 s=CLOSED pgs=1707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:06.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:06.143+0000 7fed8ec70640 1 -- 172.21.15.116:0/3769237484 >> 172.21.15.116:0/3769237484 conn(0x7fed880c3b00 msgr2=0x7fed88152ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:06.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:06.143+0000 7fed8ec70640 1 -- 172.21.15.116:0/3769237484 shutdown_connections 2024-04-25T00:30:06.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:06.143+0000 7fed8ec70640 1 -- 172.21.15.116:0/3769237484 wait complete. 2024-04-25T00:30:06.157 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:30:08.159 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:30:08.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.291+0000 7f5a8e035640 1 Processor -- start 2024-04-25T00:30:08.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.291+0000 7f5a8e035640 1 -- start start 2024-04-25T00:30:08.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.291+0000 7f5a8e035640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5a88150d20 0x7f5a880c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:08.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.291+0000 7f5a8e035640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5a880c2f50 0x7f5a881538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:08.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.291+0000 7f5a8e035640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a880c3910 con 0x7f5a880a94a0 2024-04-25T00:30:08.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.291+0000 7f5a8e035640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f5a88154530 con 0x7f5a88150d20 2024-04-25T00:30:08.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.291+0000 7f5a8e035640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f5a880c16e0 con 0x7f5a880c2f50 2024-04-25T00:30:08.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.291+0000 7f5a877fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5a88150d20 0x7f5a880c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:08.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.291+0000 7f5a877fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5a88150d20 0x7f5a880c2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36980/0 (socket says 172.21.15.116:36980) 2024-04-25T00:30:08.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.291+0000 7f5a877fe640 1 -- 172.21.15.116:0/3833466115 learned_addr learned my addr 172.21.15.116:0/3833466115 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:08.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.291+0000 7f5a7effd640 1 --2- 172.21.15.116:0/3833466115 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5a880c2f50 0x7f5a881538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:08.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.291+0000 7f5a877fe640 1 -- 172.21.15.116:0/3833466115 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5a880c2f50 msgr2=0x7f5a881538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:08.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.291+0000 7f5a877fe640 1 --2- 172.21.15.116:0/3833466115 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5a880c2f50 0x7f5a881538e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:08.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.291+0000 7f5a877fe640 1 -- 172.21.15.116:0/3833466115 >> v1:172.21.15.174:6789/0 conn(0x7f5a880a94a0 legacy=0x7f5a88153e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:08.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.291+0000 7f5a877fe640 1 -- 172.21.15.116:0/3833466115 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a880c1400 con 0x7f5a88150d20 2024-04-25T00:30:08.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.295+0000 7f5a877fe640 1 --2- 172.21.15.116:0/3833466115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a88150d20 0x7f5a880c2a10 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f5a70009870 tx=0x7f5a7001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d8336bbccf0b854 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:08.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.295+0000 7f5a86ffd640 1 -- 172.21.15.116:0/3833466115 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a700ab5f0 con 0x7f5a88150d20 2024-04-25T00:30:08.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.295+0000 7f5a86ffd640 1 -- 172.21.15.116:0/3833466115 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a700b0070 con 0x7f5a88150d20 2024-04-25T00:30:08.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.295+0000 7f5a86ffd640 1 -- 172.21.15.116:0/3833466115 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a700ab9b0 con 0x7f5a88150d20 2024-04-25T00:30:08.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.295+0000 7f5a8e035640 1 -- 172.21.15.116:0/3833466115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a88150d20 msgr2=0x7f5a880c2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:08.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.295+0000 7f5a8e035640 1 --2- 172.21.15.116:0/3833466115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a88150d20 0x7f5a880c2a10 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f5a70009870 tx=0x7f5a7001c520 comp rx=0 tx=0).stop 2024-04-25T00:30:08.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.295+0000 7f5a8e035640 1 -- 172.21.15.116:0/3833466115 shutdown_connections 2024-04-25T00:30:08.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.295+0000 7f5a8e035640 1 --2- 172.21.15.116:0/3833466115 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5a880c2f50 0x7f5a881538e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:08.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.295+0000 7f5a8e035640 1 --2- 172.21.15.116:0/3833466115 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a88150d20 0x7f5a880c2a10 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:08.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.295+0000 7f5a8e035640 1 -- 172.21.15.116:0/3833466115 >> 172.21.15.116:0/3833466115 conn(0x7f5a880c3b00 msgr2=0x7f5a880c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:08.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.295+0000 7f5a8e035640 1 -- 172.21.15.116:0/3833466115 shutdown_connections 2024-04-25T00:30:08.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.295+0000 7f5a8e035640 1 -- 172.21.15.116:0/3833466115 wait complete. 2024-04-25T00:30:08.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.295+0000 7f5a8e035640 1 Processor -- start 2024-04-25T00:30:08.295 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.295+0000 7f5a8e035640 1 -- start start 2024-04-25T00:30:08.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.295+0000 7f5a8e035640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a880a94a0 0x7f5a8815c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:08.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.295+0000 7f5a8e035640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5a880c2f50 0x7f5a881407b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:08.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.295+0000 7f5a8e035640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5a88150d20 0x7f5a88140cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:08.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.295+0000 7f5a8e035640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5a881608b0 con 0x7f5a880a94a0 2024-04-25T00:30:08.297 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.295+0000 7f5a8e035640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5a88160730 con 0x7f5a880c2f50 2024-04-25T00:30:08.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.295+0000 7f5a8e035640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f5a88160a30 con 0x7f5a88150d20 2024-04-25T00:30:08.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.295+0000 7f5a877fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a880a94a0 0x7f5a8815c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:08.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.295+0000 7f5a877fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a880a94a0 0x7f5a8815c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36984/0 (socket says 172.21.15.116:36984) 2024-04-25T00:30:08.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.295+0000 7f5a877fe640 1 -- 172.21.15.116:0/1940197197 learned_addr learned my addr 172.21.15.116:0/1940197197 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:08.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.295+0000 7f5a87fff640 1 --2- 172.21.15.116:0/1940197197 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5a88150d20 0x7f5a88140cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:08.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.295+0000 7f5a7effd640 1 --2- 172.21.15.116:0/1940197197 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5a880c2f50 0x7f5a881407b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:08.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.295+0000 7f5a877fe640 1 -- 172.21.15.116:0/1940197197 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5a88150d20 msgr2=0x7f5a88140cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:08.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.295+0000 7f5a877fe640 1 --2- 172.21.15.116:0/1940197197 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5a88150d20 0x7f5a88140cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:08.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.295+0000 7f5a877fe640 1 -- 172.21.15.116:0/1940197197 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5a880c2f50 msgr2=0x7f5a881407b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:08.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.295+0000 7f5a877fe640 1 --2- 172.21.15.116:0/1940197197 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5a880c2f50 0x7f5a881407b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:08.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.295+0000 7f5a877fe640 1 -- 172.21.15.116:0/1940197197 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a88141230 con 0x7f5a880a94a0 2024-04-25T00:30:08.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.295+0000 7f5a7effd640 1 --2- 172.21.15.116:0/1940197197 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5a880c2f50 0x7f5a881407b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:30:08.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.295+0000 7f5a877fe640 1 --2- 172.21.15.116:0/1940197197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a880a94a0 0x7f5a8815c550 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f5a7001cdc0 tx=0x7f5a7001ad60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:08.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.299+0000 7f5a84ff9640 1 -- 172.21.15.116:0/1940197197 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a700be5f0 con 0x7f5a880a94a0 2024-04-25T00:30:08.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.299+0000 7f5a84ff9640 1 -- 172.21.15.116:0/1940197197 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a700be7b0 con 0x7f5a880a94a0 2024-04-25T00:30:08.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.299+0000 7f5a84ff9640 1 -- 172.21.15.116:0/1940197197 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a700beba0 con 0x7f5a880a94a0 2024-04-25T00:30:08.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.299+0000 7f5a8e035640 1 -- 172.21.15.116:0/1940197197 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a881423e0 con 0x7f5a880a94a0 2024-04-25T00:30:08.299 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.299+0000 7f5a8e035640 1 -- 172.21.15.116:0/1940197197 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a881f21a0 con 0x7f5a880a94a0 2024-04-25T00:30:08.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.299+0000 7f5a8e035640 1 -- 172.21.15.116:0/1940197197 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a88068350 con 0x7f5a880a94a0 2024-04-25T00:30:08.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.299+0000 7f5a84ff9640 1 -- 172.21.15.116:0/1940197197 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5a700b43f0 con 0x7f5a880a94a0 2024-04-25T00:30:08.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.299+0000 7f5a84ff9640 1 --2- 172.21.15.116:0/1940197197 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5a6c0678f0 0x7f5a6c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:08.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.299+0000 7f5a84ff9640 1 -- 172.21.15.116:0/1940197197 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 10536+0+0 (secure 0 0 0) 0x7f5a70121050 con 0x7f5a880a94a0 2024-04-25T00:30:08.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.299+0000 7f5a7effd640 1 --2- 172.21.15.116:0/1940197197 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5a6c0678f0 0x7f5a6c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:08.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.299+0000 7f5a7effd640 1 --2- 172.21.15.116:0/1940197197 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5a6c0678f0 0x7f5a6c069db0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f5a88070fa0 tx=0x7f5a78008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:08.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.303+0000 7f5a84ff9640 1 -- 172.21.15.116:0/1940197197 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f5a700c4060 con 0x7f5a880a94a0 2024-04-25T00:30:08.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.495+0000 7f5a8e035640 1 -- 172.21.15.116:0/1940197197 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5a8806d770 con 0x7f5a880a94a0 2024-04-25T00:30:08.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.495+0000 7f5a84ff9640 1 -- 172.21.15.116:0/1940197197 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 237 v237) ==== 95+0+6510 (secure 0 0 0) 0x7f5a70016110 con 0x7f5a880a94a0 2024-04-25T00:30:08.496 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:30:08.496 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":237,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6764,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3873739116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3873739116},{"type":"v1","addr":"172.21.15.174:6837","nonce":3873739116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":237}],"filesystems":[{"mdsmap":{"epoch":236,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:29:54.168353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8847},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8847":{"gid":8847,"name":"c","rank":0,"incarnation":233,"state":"up:active","state_seq":17,"addr":"172.21.15.116:6833/2662028892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2662028892},{"type":"v1","addr":"172.21.15.116:6833","nonce":2662028892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8847,"qdb_cluster":[8847]},"id":1},{"mdsmap":{"epoch":231,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:28:53.808539+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8772":{"gid":8772,"name":"a","rank":0,"incarnation":228,"state":"up:active","state_seq":20,"addr":"172.21.15.116:6835/3499878473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3499878473},{"type":"v1","addr":"172.21.15.116:6835","nonce":3499878473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8772,"qdb_cluster":[8772]},"id":2}]} 2024-04-25T00:30:08.496 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 237 2024-04-25T00:30:08.497 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.499+0000 7f5a8e035640 1 -- 172.21.15.116:0/1940197197 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5a6c0678f0 msgr2=0x7f5a6c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:08.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.499+0000 7f5a8e035640 1 --2- 172.21.15.116:0/1940197197 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5a6c0678f0 0x7f5a6c069db0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f5a88070fa0 tx=0x7f5a78008040 comp rx=0 tx=0).stop 2024-04-25T00:30:08.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.499+0000 7f5a8e035640 1 -- 172.21.15.116:0/1940197197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a880a94a0 msgr2=0x7f5a8815c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:08.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.499+0000 7f5a8e035640 1 --2- 172.21.15.116:0/1940197197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a880a94a0 0x7f5a8815c550 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f5a7001cdc0 tx=0x7f5a7001ad60 comp rx=0 tx=0).stop 2024-04-25T00:30:08.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.499+0000 7f5a8e035640 1 -- 172.21.15.116:0/1940197197 shutdown_connections 2024-04-25T00:30:08.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.499+0000 7f5a8e035640 1 --2- 172.21.15.116:0/1940197197 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5a6c0678f0 0x7f5a6c069db0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:08.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.499+0000 7f5a8e035640 1 --2- 172.21.15.116:0/1940197197 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5a88150d20 0x7f5a88140cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:08.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.499+0000 7f5a8e035640 1 --2- 172.21.15.116:0/1940197197 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5a880c2f50 0x7f5a881407b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:08.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.499+0000 7f5a8e035640 1 --2- 172.21.15.116:0/1940197197 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5a880a94a0 0x7f5a8815c550 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:08.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.499+0000 7f5a8e035640 1 -- 172.21.15.116:0/1940197197 >> 172.21.15.116:0/1940197197 conn(0x7f5a880c3b00 msgr2=0x7f5a8814ff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:08.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.499+0000 7f5a8e035640 1 -- 172.21.15.116:0/1940197197 shutdown_connections 2024-04-25T00:30:08.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:08.499+0000 7f5a8e035640 1 -- 172.21.15.116:0/1940197197 wait complete. 2024-04-25T00:30:08.513 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:30:10.514 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:30:10.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.647+0000 7fbec0c8f640 1 Processor -- start 2024-04-25T00:30:10.645 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.647+0000 7fbec0c8f640 1 -- start start 2024-04-25T00:30:10.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.647+0000 7fbec0c8f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbebc13e310 0x7fbebc0c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:10.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.647+0000 7fbec0c8f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbebc13ed60 0x7fbebc1560d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:10.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.647+0000 7fbec0c8f640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbebc06d200 con 0x7fbebc0c2f50 2024-04-25T00:30:10.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.647+0000 7fbec0c8f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fbebc070d40 con 0x7fbebc13ed60 2024-04-25T00:30:10.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.647+0000 7fbec0c8f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fbebc071020 con 0x7fbebc13e310 2024-04-25T00:30:10.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.647+0000 7fbebad9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbebc13ed60 0x7fbebc1560d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:10.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.647+0000 7fbebad9d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbebc13ed60 0x7fbebc1560d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36996/0 (socket says 172.21.15.116:36996) 2024-04-25T00:30:10.647 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.647+0000 7fbebad9d640 1 -- 172.21.15.116:0/2225801582 learned_addr learned my addr 172.21.15.116:0/2225801582 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:10.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.647+0000 7fbeba59c640 1 --2- 172.21.15.116:0/2225801582 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbebc13e310 0x7fbebc0c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:10.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.647+0000 7fbebad9d640 1 -- 172.21.15.116:0/2225801582 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbebc13e310 msgr2=0x7fbebc0c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:10.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.647+0000 7fbebad9d640 1 --2- 172.21.15.116:0/2225801582 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbebc13e310 0x7fbebc0c2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:10.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.647+0000 7fbebad9d640 1 -- 172.21.15.116:0/2225801582 >> v1:172.21.15.174:6789/0 conn(0x7fbebc0c2f50 legacy=0x7fbebc1559c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:10.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.647+0000 7fbebad9d640 1 -- 172.21.15.116:0/2225801582 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbebc156610 con 0x7fbebc13ed60 2024-04-25T00:30:10.648 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.647+0000 7fbeba59c640 1 --2- 172.21.15.116:0/2225801582 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbebc13e310 0x7fbebc0c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:30:10.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.647+0000 7fbebad9d640 1 --2- 172.21.15.116:0/2225801582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbebc13ed60 0x7fbebc1560d0 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7fbeb4009e50 tx=0x7fbeb401c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9e7052f7472a0b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:10.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.651+0000 7fbeb9d9b640 1 -- 172.21.15.116:0/2225801582 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeb40a5660 con 0x7fbebc13ed60 2024-04-25T00:30:10.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.651+0000 7fbeb9d9b640 1 -- 172.21.15.116:0/2225801582 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbeb40a5800 con 0x7fbebc13ed60 2024-04-25T00:30:10.649 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.651+0000 7fbeb9d9b640 1 -- 172.21.15.116:0/2225801582 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeb40a5bd0 con 0x7fbebc13ed60 2024-04-25T00:30:10.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.651+0000 7fbec0c8f640 1 -- 172.21.15.116:0/2225801582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbebc13ed60 msgr2=0x7fbebc1560d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:10.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.651+0000 7fbec0c8f640 1 --2- 172.21.15.116:0/2225801582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbebc13ed60 0x7fbebc1560d0 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7fbeb4009e50 tx=0x7fbeb401c8d0 comp rx=0 tx=0).stop 2024-04-25T00:30:10.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.651+0000 7fbec0c8f640 1 -- 172.21.15.116:0/2225801582 shutdown_connections 2024-04-25T00:30:10.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.651+0000 7fbec0c8f640 1 --2- 172.21.15.116:0/2225801582 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbebc13ed60 0x7fbebc1560d0 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:10.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.651+0000 7fbec0c8f640 1 --2- 172.21.15.116:0/2225801582 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbebc13e310 0x7fbebc0c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:10.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.651+0000 7fbec0c8f640 1 -- 172.21.15.116:0/2225801582 >> 172.21.15.116:0/2225801582 conn(0x7fbebc0c3b00 msgr2=0x7fbebc0c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:10.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.651+0000 7fbec0c8f640 1 -- 172.21.15.116:0/2225801582 shutdown_connections 2024-04-25T00:30:10.650 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.651+0000 7fbec0c8f640 1 -- 172.21.15.116:0/2225801582 wait complete. 2024-04-25T00:30:10.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.651+0000 7fbec0c8f640 1 Processor -- start 2024-04-25T00:30:10.651 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.651+0000 7fbec0c8f640 1 -- start start 2024-04-25T00:30:10.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.651+0000 7fbec0c8f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbebc0c2f50 0x7fbebc133320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:10.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.651+0000 7fbec0c8f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbebc13e310 0x7fbebc1f2a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:10.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.651+0000 7fbec0c8f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbebc13ed60 0x7fbebc0c63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:10.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.651+0000 7fbec0c8f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbebc162b70 con 0x7fbebc13ed60 2024-04-25T00:30:10.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.651+0000 7fbec0c8f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbebc1629f0 con 0x7fbebc0c2f50 2024-04-25T00:30:10.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.651+0000 7fbec0c8f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fbebc162cf0 con 0x7fbebc13e310 2024-04-25T00:30:10.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.651+0000 7fbebad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbebc13ed60 0x7fbebc0c63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:10.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.651+0000 7fbebad9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbebc13ed60 0x7fbebc0c63d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36998/0 (socket says 172.21.15.116:36998) 2024-04-25T00:30:10.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.651+0000 7fbebad9d640 1 -- 172.21.15.116:0/1998985504 learned_addr learned my addr 172.21.15.116:0/1998985504 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:10.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.651+0000 7fbeba59c640 1 --2- 172.21.15.116:0/1998985504 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbebc0c2f50 0x7fbebc133320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:10.654 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.651+0000 7fbeb3fff640 1 --2- 172.21.15.116:0/1998985504 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbebc13e310 0x7fbebc1f2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:10.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.651+0000 7fbebad9d640 1 -- 172.21.15.116:0/1998985504 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbebc13e310 msgr2=0x7fbebc1f2a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:10.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.651+0000 7fbebad9d640 1 --2- 172.21.15.116:0/1998985504 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbebc13e310 0x7fbebc1f2a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:10.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.651+0000 7fbebad9d640 1 -- 172.21.15.116:0/1998985504 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbebc0c2f50 msgr2=0x7fbebc133320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:10.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.651+0000 7fbebad9d640 1 --2- 172.21.15.116:0/1998985504 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbebc0c2f50 0x7fbebc133320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:10.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.651+0000 7fbebad9d640 1 -- 172.21.15.116:0/1998985504 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbebc13b600 con 0x7fbebc13ed60 2024-04-25T00:30:10.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.651+0000 7fbeb3fff640 1 --2- 172.21.15.116:0/1998985504 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbebc13e310 0x7fbebc1f2a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:30:10.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.651+0000 7fbebad9d640 1 --2- 172.21.15.116:0/1998985504 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbebc13ed60 0x7fbebc0c63d0 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7fbeb401af80 tx=0x7fbeb4019670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:10.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.651+0000 7fbeb2ffd640 1 -- 172.21.15.116:0/1998985504 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeb40bc020 con 0x7fbebc13ed60 2024-04-25T00:30:10.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.651+0000 7fbeb2ffd640 1 -- 172.21.15.116:0/1998985504 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbeb40b8670 con 0x7fbebc13ed60 2024-04-25T00:30:10.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.651+0000 7fbec0c8f640 1 -- 172.21.15.116:0/1998985504 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbebc1386f0 con 0x7fbebc13ed60 2024-04-25T00:30:10.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.651+0000 7fbeb2ffd640 1 -- 172.21.15.116:0/1998985504 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeb40b8a60 con 0x7fbebc13ed60 2024-04-25T00:30:10.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.651+0000 7fbec0c8f640 1 -- 172.21.15.116:0/1998985504 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbebc138990 con 0x7fbebc13ed60 2024-04-25T00:30:10.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.655+0000 7fbec0c8f640 1 -- 172.21.15.116:0/1998985504 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbebc068350 con 0x7fbebc13ed60 2024-04-25T00:30:10.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.655+0000 7fbeb2ffd640 1 -- 172.21.15.116:0/1998985504 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbeb40bf020 con 0x7fbebc13ed60 2024-04-25T00:30:10.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.655+0000 7fbeb2ffd640 1 --2- 172.21.15.116:0/1998985504 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbe9c0678c0 0x7fbe9c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:10.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.655+0000 7fbeb2ffd640 1 -- 172.21.15.116:0/1998985504 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 10536+0+0 (secure 0 0 0) 0x7fbeb4121f00 con 0x7fbebc13ed60 2024-04-25T00:30:10.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.655+0000 7fbeba59c640 1 --2- 172.21.15.116:0/1998985504 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbe9c0678c0 0x7fbe9c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:10.657 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.655+0000 7fbeba59c640 1 --2- 172.21.15.116:0/1998985504 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbe9c0678c0 0x7fbe9c069d80 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fbeac006fd0 tx=0x7fbeac008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:10.659 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.659+0000 7fbeb2ffd640 1 -- 172.21.15.116:0/1998985504 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fbeb40c6030 con 0x7fbebc13ed60 2024-04-25T00:30:10.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.847+0000 7fbec0c8f640 1 -- 172.21.15.116:0/1998985504 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbebc15ffb0 con 0x7fbebc13ed60 2024-04-25T00:30:10.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.851+0000 7fbeb2ffd640 1 -- 172.21.15.116:0/1998985504 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 237 v237) ==== 95+0+6510 (secure 0 0 0) 0x7fbeb40165e0 con 0x7fbebc13ed60 2024-04-25T00:30:10.850 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:30:10.850 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":237,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6764,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3873739116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3873739116},{"type":"v1","addr":"172.21.15.174:6837","nonce":3873739116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":237}],"filesystems":[{"mdsmap":{"epoch":236,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:29:54.168353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8847},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8847":{"gid":8847,"name":"c","rank":0,"incarnation":233,"state":"up:active","state_seq":17,"addr":"172.21.15.116:6833/2662028892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2662028892},{"type":"v1","addr":"172.21.15.116:6833","nonce":2662028892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8847,"qdb_cluster":[8847]},"id":1},{"mdsmap":{"epoch":231,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:28:53.808539+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8772":{"gid":8772,"name":"a","rank":0,"incarnation":228,"state":"up:active","state_seq":20,"addr":"172.21.15.116:6835/3499878473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3499878473},{"type":"v1","addr":"172.21.15.116:6835","nonce":3499878473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8772,"qdb_cluster":[8772]},"id":2}]} 2024-04-25T00:30:10.851 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 237 2024-04-25T00:30:10.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.851+0000 7fbec0c8f640 1 -- 172.21.15.116:0/1998985504 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbe9c0678c0 msgr2=0x7fbe9c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:10.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.851+0000 7fbec0c8f640 1 --2- 172.21.15.116:0/1998985504 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbe9c0678c0 0x7fbe9c069d80 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fbeac006fd0 tx=0x7fbeac008040 comp rx=0 tx=0).stop 2024-04-25T00:30:10.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.851+0000 7fbec0c8f640 1 -- 172.21.15.116:0/1998985504 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbebc13ed60 msgr2=0x7fbebc0c63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:10.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.851+0000 7fbec0c8f640 1 --2- 172.21.15.116:0/1998985504 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbebc13ed60 0x7fbebc0c63d0 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7fbeb401af80 tx=0x7fbeb4019670 comp rx=0 tx=0).stop 2024-04-25T00:30:10.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.851+0000 7fbec0c8f640 1 -- 172.21.15.116:0/1998985504 shutdown_connections 2024-04-25T00:30:10.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.851+0000 7fbec0c8f640 1 --2- 172.21.15.116:0/1998985504 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbe9c0678c0 0x7fbe9c069d80 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:10.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.851+0000 7fbec0c8f640 1 --2- 172.21.15.116:0/1998985504 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbebc13ed60 0x7fbebc0c63d0 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:10.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.851+0000 7fbec0c8f640 1 --2- 172.21.15.116:0/1998985504 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbebc13e310 0x7fbebc1f2a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:10.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.851+0000 7fbec0c8f640 1 --2- 172.21.15.116:0/1998985504 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbebc0c2f50 0x7fbebc133320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:10.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.851+0000 7fbec0c8f640 1 -- 172.21.15.116:0/1998985504 >> 172.21.15.116:0/1998985504 conn(0x7fbebc0c3b00 msgr2=0x7fbebc06e360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:10.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.851+0000 7fbec0c8f640 1 -- 172.21.15.116:0/1998985504 shutdown_connections 2024-04-25T00:30:10.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:10.851+0000 7fbec0c8f640 1 -- 172.21.15.116:0/1998985504 wait complete. 2024-04-25T00:30:10.868 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:30:12.353 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:30:12.356 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:30:12.869 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:30:13.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:12.999+0000 7fbb7167d640 1 Processor -- start 2024-04-25T00:30:13.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:12.999+0000 7fbb7167d640 1 -- start start 2024-04-25T00:30:13.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.003+0000 7fbb7167d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbb6c150d20 0x7fbb6c0c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:13.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.003+0000 7fbb7167d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb6c0c2f50 0x7fbb6c1538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:13.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.003+0000 7fbb7167d640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fbb6c0a94a0 0x7fbb6c153e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:13.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.003+0000 7fbb7167d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fbb6c0c3910 con 0x7fbb6c150d20 2024-04-25T00:30:13.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.003+0000 7fbb7167d640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fbb6c0c16e0 con 0x7fbb6c0a94a0 2024-04-25T00:30:13.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.003+0000 7fbb7167d640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fbb6c0c1400 con 0x7fbb6c0c2f50 2024-04-25T00:30:13.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.003+0000 7fbb6affd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbb6c150d20 0x7fbb6c0c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:13.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.003+0000 7fbb6affd640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbb6c150d20 0x7fbb6c0c2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37010/0 (socket says 172.21.15.116:37010) 2024-04-25T00:30:13.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.003+0000 7fbb6affd640 1 -- 172.21.15.116:0/2503545847 learned_addr learned my addr 172.21.15.116:0/2503545847 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:13.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.003+0000 7fbb6affd640 1 -- 172.21.15.116:0/2503545847 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb6c0c2f50 msgr2=0x7fbb6c1538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:13.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.003+0000 7fbb6affd640 1 --2- 172.21.15.116:0/2503545847 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb6c0c2f50 0x7fbb6c1538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:13.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.003+0000 7fbb6b7fe640 1 --2- 172.21.15.116:0/2503545847 >> v2:172.21.15.174:3300/0 conn(0x7fbb6c0a94a0 0x7fbb6c153e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:13.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.003+0000 7fbb6affd640 1 -- 172.21.15.116:0/2503545847 >> v2:172.21.15.174:3300/0 conn(0x7fbb6c0a94a0 msgr2=0x7fbb6c153e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:13.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.003+0000 7fbb6affd640 1 --2- 172.21.15.116:0/2503545847 >> v2:172.21.15.174:3300/0 conn(0x7fbb6c0a94a0 0x7fbb6c153e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:13.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.003+0000 7fbb6affd640 1 -- 172.21.15.116:0/2503545847 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb6c06e660 con 0x7fbb6c150d20 2024-04-25T00:30:13.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.003+0000 7fbb6affd640 1 --2- 172.21.15.116:0/2503545847 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb6c150d20 0x7fbb6c0c2a10 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c0732b0 tx=0x7fbb5c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9aade910348b3a7a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:13.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.003+0000 7fbb6a7fc640 1 -- 172.21.15.116:0/2503545847 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb5c0b1070 con 0x7fbb6c150d20 2024-04-25T00:30:13.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.003+0000 7fbb6a7fc640 1 -- 172.21.15.116:0/2503545847 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb5c0ac410 con 0x7fbb6c150d20 2024-04-25T00:30:13.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.003+0000 7fbb6a7fc640 1 -- 172.21.15.116:0/2503545847 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb5c0ac7e0 con 0x7fbb6c150d20 2024-04-25T00:30:13.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.003+0000 7fbb7167d640 1 -- 172.21.15.116:0/2503545847 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb6c150d20 msgr2=0x7fbb6c0c2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:13.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.003+0000 7fbb7167d640 1 --2- 172.21.15.116:0/2503545847 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb6c150d20 0x7fbb6c0c2a10 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c0732b0 tx=0x7fbb5c01c520 comp rx=0 tx=0).stop 2024-04-25T00:30:13.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.003+0000 7fbb7167d640 1 -- 172.21.15.116:0/2503545847 shutdown_connections 2024-04-25T00:30:13.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.003+0000 7fbb7167d640 1 --2- 172.21.15.116:0/2503545847 >> v2:172.21.15.174:3300/0 conn(0x7fbb6c0a94a0 0x7fbb6c153e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:13.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.003+0000 7fbb7167d640 1 --2- 172.21.15.116:0/2503545847 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb6c0c2f50 0x7fbb6c1538e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:13.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.003+0000 7fbb7167d640 1 --2- 172.21.15.116:0/2503545847 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb6c150d20 0x7fbb6c0c2a10 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:13.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.003+0000 7fbb7167d640 1 -- 172.21.15.116:0/2503545847 >> 172.21.15.116:0/2503545847 conn(0x7fbb6c0c3b00 msgr2=0x7fbb6c0c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:13.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.003+0000 7fbb7167d640 1 -- 172.21.15.116:0/2503545847 shutdown_connections 2024-04-25T00:30:13.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.003+0000 7fbb7167d640 1 -- 172.21.15.116:0/2503545847 wait complete. 2024-04-25T00:30:13.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.007+0000 7fbb7167d640 1 Processor -- start 2024-04-25T00:30:13.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.007+0000 7fbb7167d640 1 -- start start 2024-04-25T00:30:13.008 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.007+0000 7fbb7167d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb6c0a94a0 0x7fbb6c1ecbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:13.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.007+0000 7fbb7167d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb6c0c2f50 0x7fbb6c0c54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:13.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.007+0000 7fbb7167d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb6c150d20 0x7fbb6c155f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:13.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.007+0000 7fbb7167d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbb6c157340 con 0x7fbb6c150d20 2024-04-25T00:30:13.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.007+0000 7fbb7167d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbb6c154570 con 0x7fbb6c0c2f50 2024-04-25T00:30:13.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.007+0000 7fbb7167d640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fbb6c1574c0 con 0x7fbb6c0a94a0 2024-04-25T00:30:13.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.007+0000 7fbb6b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb6c150d20 0x7fbb6c155f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:13.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.007+0000 7fbb6b7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb6c150d20 0x7fbb6c155f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:37022/0 (socket says 172.21.15.116:37022) 2024-04-25T00:30:13.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.007+0000 7fbb6b7fe640 1 -- 172.21.15.116:0/135796362 learned_addr learned my addr 172.21.15.116:0/135796362 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:13.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.007+0000 7fbb6affd640 1 --2- 172.21.15.116:0/135796362 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb6c0a94a0 0x7fbb6c1ecbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:13.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.007+0000 7fbb627fc640 1 --2- 172.21.15.116:0/135796362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb6c0c2f50 0x7fbb6c0c54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:13.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.007+0000 7fbb6b7fe640 1 -- 172.21.15.116:0/135796362 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb6c0a94a0 msgr2=0x7fbb6c1ecbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:13.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.007+0000 7fbb6b7fe640 1 --2- 172.21.15.116:0/135796362 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb6c0a94a0 0x7fbb6c1ecbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:13.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.007+0000 7fbb6b7fe640 1 -- 172.21.15.116:0/135796362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb6c0c2f50 msgr2=0x7fbb6c0c54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:13.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.007+0000 7fbb6b7fe640 1 --2- 172.21.15.116:0/135796362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb6c0c2f50 0x7fbb6c0c54b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:13.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.007+0000 7fbb6b7fe640 1 -- 172.21.15.116:0/135796362 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb6c154cb0 con 0x7fbb6c150d20 2024-04-25T00:30:13.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.007+0000 7fbb6affd640 1 --2- 172.21.15.116:0/135796362 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb6c0a94a0 0x7fbb6c1ecbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:30:13.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.007+0000 7fbb627fc640 1 --2- 172.21.15.116:0/135796362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb6c0c2f50 0x7fbb6c0c54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:30:13.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.007+0000 7fbb6b7fe640 1 --2- 172.21.15.116:0/135796362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb6c150d20 0x7fbb6c155f90 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7fbb6400cda0 tx=0x7fbb6400a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:13.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.007+0000 7fbb63fff640 1 -- 172.21.15.116:0/135796362 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb64003ba0 con 0x7fbb6c150d20 2024-04-25T00:30:13.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.007+0000 7fbb63fff640 1 -- 172.21.15.116:0/135796362 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb64003d40 con 0x7fbb6c150d20 2024-04-25T00:30:13.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.007+0000 7fbb7167d640 1 -- 172.21.15.116:0/135796362 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb6c154f90 con 0x7fbb6c150d20 2024-04-25T00:30:13.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.007+0000 7fbb63fff640 1 -- 172.21.15.116:0/135796362 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb64004240 con 0x7fbb6c150d20 2024-04-25T00:30:13.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.007+0000 7fbb7167d640 1 -- 172.21.15.116:0/135796362 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb6c155790 con 0x7fbb6c150d20 2024-04-25T00:30:13.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.007+0000 7fbb7167d640 1 -- 172.21.15.116:0/135796362 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb6c068350 con 0x7fbb6c150d20 2024-04-25T00:30:13.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.011+0000 7fbb63fff640 1 -- 172.21.15.116:0/135796362 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbb640043e0 con 0x7fbb6c150d20 2024-04-25T00:30:13.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.011+0000 7fbb63fff640 1 --2- 172.21.15.116:0/135796362 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbb4c0678c0 0x7fbb4c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:13.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.011+0000 7fbb63fff640 1 -- 172.21.15.116:0/135796362 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 10536+0+0 (secure 0 0 0) 0x7fbb64087aa0 con 0x7fbb6c150d20 2024-04-25T00:30:13.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.011+0000 7fbb6affd640 1 --2- 172.21.15.116:0/135796362 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbb4c0678c0 0x7fbb4c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:13.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.011+0000 7fbb6affd640 1 --2- 172.21.15.116:0/135796362 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbb4c0678c0 0x7fbb4c069d80 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c0ae000 tx=0x7fbb5c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:13.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.015+0000 7fbb63fff640 1 -- 172.21.15.116:0/135796362 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fbb64057340 con 0x7fbb6c150d20 2024-04-25T00:30:13.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.211+0000 7fbb7167d640 1 -- 172.21.15.116:0/135796362 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbb6c0c34f0 con 0x7fbb6c150d20 2024-04-25T00:30:13.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.211+0000 7fbb63fff640 1 -- 172.21.15.116:0/135796362 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 237 v237) ==== 95+0+6510 (secure 0 0 0) 0x7fbb64056c40 con 0x7fbb6c150d20 2024-04-25T00:30:13.213 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:30:13.213 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":237,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6764,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3873739116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3873739116},{"type":"v1","addr":"172.21.15.174:6837","nonce":3873739116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":237}],"filesystems":[{"mdsmap":{"epoch":236,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:29:54.168353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8847},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8847":{"gid":8847,"name":"c","rank":0,"incarnation":233,"state":"up:active","state_seq":17,"addr":"172.21.15.116:6833/2662028892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2662028892},{"type":"v1","addr":"172.21.15.116:6833","nonce":2662028892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8847,"qdb_cluster":[8847]},"id":1},{"mdsmap":{"epoch":231,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:28:53.808539+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8772":{"gid":8772,"name":"a","rank":0,"incarnation":228,"state":"up:active","state_seq":20,"addr":"172.21.15.116:6835/3499878473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3499878473},{"type":"v1","addr":"172.21.15.116:6835","nonce":3499878473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8772,"qdb_cluster":[8772]},"id":2}]} 2024-04-25T00:30:13.213 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 237 2024-04-25T00:30:13.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.215+0000 7fbb7167d640 1 -- 172.21.15.116:0/135796362 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbb4c0678c0 msgr2=0x7fbb4c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:13.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.215+0000 7fbb7167d640 1 --2- 172.21.15.116:0/135796362 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbb4c0678c0 0x7fbb4c069d80 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c0ae000 tx=0x7fbb5c008040 comp rx=0 tx=0).stop 2024-04-25T00:30:13.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.215+0000 7fbb7167d640 1 -- 172.21.15.116:0/135796362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb6c150d20 msgr2=0x7fbb6c155f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:13.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.215+0000 7fbb7167d640 1 --2- 172.21.15.116:0/135796362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb6c150d20 0x7fbb6c155f90 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7fbb6400cda0 tx=0x7fbb6400a660 comp rx=0 tx=0).stop 2024-04-25T00:30:13.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.215+0000 7fbb7167d640 1 -- 172.21.15.116:0/135796362 shutdown_connections 2024-04-25T00:30:13.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.215+0000 7fbb7167d640 1 --2- 172.21.15.116:0/135796362 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbb4c0678c0 0x7fbb4c069d80 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:13.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.215+0000 7fbb7167d640 1 --2- 172.21.15.116:0/135796362 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbb6c150d20 0x7fbb6c155f90 unknown :-1 s=CLOSED pgs=1713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:13.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.215+0000 7fbb7167d640 1 --2- 172.21.15.116:0/135796362 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbb6c0c2f50 0x7fbb6c0c54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:13.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.215+0000 7fbb7167d640 1 --2- 172.21.15.116:0/135796362 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbb6c0a94a0 0x7fbb6c1ecbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:13.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.215+0000 7fbb7167d640 1 -- 172.21.15.116:0/135796362 >> 172.21.15.116:0/135796362 conn(0x7fbb6c0c3b00 msgr2=0x7fbb6c0a9f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:13.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.215+0000 7fbb7167d640 1 -- 172.21.15.116:0/135796362 shutdown_connections 2024-04-25T00:30:13.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:13.215+0000 7fbb7167d640 1 -- 172.21.15.116:0/135796362 wait complete. 2024-04-25T00:30:13.232 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:30:15.233 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:30:15.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.367+0000 7f664f761640 1 Processor -- start 2024-04-25T00:30:15.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.367+0000 7f664f761640 1 -- start start 2024-04-25T00:30:15.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.367+0000 7f664f761640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6648150d20 0x7f66480c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:15.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.367+0000 7f664f761640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f66480c2f50 0x7f66481538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:15.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.367+0000 7f664f761640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f66480a94a0 0x7f6648153e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:15.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.367+0000 7f664f761640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f66480c3910 con 0x7f66480c2f50 2024-04-25T00:30:15.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.367+0000 7f664f761640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f66480c16e0 con 0x7f66480a94a0 2024-04-25T00:30:15.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.367+0000 7f664f761640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f66480c1400 con 0x7f6648150d20 2024-04-25T00:30:15.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.367+0000 7f664ccfc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f66480c2f50 0x7f66481538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:15.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.367+0000 7f664ccfc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f66480c2f50 0x7f66481538e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46320/0 (socket says 172.21.15.116:46320) 2024-04-25T00:30:15.369 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.367+0000 7f664ccfc640 1 -- 172.21.15.116:0/3243338674 learned_addr learned my addr 172.21.15.116:0/3243338674 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:15.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.367+0000 7f664d4fd640 1 --2- 172.21.15.116:0/3243338674 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6648150d20 0x7f66480c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:15.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.367+0000 7f664dcfe640 1 --2- 172.21.15.116:0/3243338674 >> v2:172.21.15.174:3300/0 conn(0x7f66480a94a0 0x7f6648153e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:15.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.367+0000 7f664ccfc640 1 -- 172.21.15.116:0/3243338674 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6648150d20 msgr2=0x7f66480c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:15.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.367+0000 7f664ccfc640 1 --2- 172.21.15.116:0/3243338674 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6648150d20 0x7f66480c2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:15.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.367+0000 7f664ccfc640 1 -- 172.21.15.116:0/3243338674 >> v2:172.21.15.174:3300/0 conn(0x7f66480a94a0 msgr2=0x7f6648153e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:15.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.367+0000 7f664ccfc640 1 --2- 172.21.15.116:0/3243338674 >> v2:172.21.15.174:3300/0 conn(0x7f66480a94a0 0x7f6648153e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:15.370 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.367+0000 7f664ccfc640 1 -- 172.21.15.116:0/3243338674 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f664806e660 con 0x7f66480c2f50 2024-04-25T00:30:15.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.371+0000 7f664ccfc640 1 --2- 172.21.15.116:0/3243338674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66480c2f50 0x7f66481538e0 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f66480732b0 tx=0x7f663c01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b867f287ff2a7da8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:15.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.371+0000 7f6647fff640 1 -- 172.21.15.116:0/3243338674 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f663c0b1070 con 0x7f66480c2f50 2024-04-25T00:30:15.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.371+0000 7f6647fff640 1 -- 172.21.15.116:0/3243338674 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f663c0ac410 con 0x7f66480c2f50 2024-04-25T00:30:15.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.371+0000 7f6647fff640 1 -- 172.21.15.116:0/3243338674 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f663c0ac7e0 con 0x7f66480c2f50 2024-04-25T00:30:15.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.371+0000 7f664f761640 1 -- 172.21.15.116:0/3243338674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66480c2f50 msgr2=0x7f66481538e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:15.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.371+0000 7f664f761640 1 --2- 172.21.15.116:0/3243338674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66480c2f50 0x7f66481538e0 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f66480732b0 tx=0x7f663c01c520 comp rx=0 tx=0).stop 2024-04-25T00:30:15.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.371+0000 7f664f761640 1 -- 172.21.15.116:0/3243338674 shutdown_connections 2024-04-25T00:30:15.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.371+0000 7f664f761640 1 --2- 172.21.15.116:0/3243338674 >> v2:172.21.15.174:3300/0 conn(0x7f66480a94a0 0x7f6648153e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:15.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.371+0000 7f664f761640 1 --2- 172.21.15.116:0/3243338674 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66480c2f50 0x7f66481538e0 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:15.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.371+0000 7f664f761640 1 --2- 172.21.15.116:0/3243338674 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6648150d20 0x7f66480c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:15.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.371+0000 7f664f761640 1 -- 172.21.15.116:0/3243338674 >> 172.21.15.116:0/3243338674 conn(0x7f66480c3b00 msgr2=0x7f66480c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:15.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.371+0000 7f664f761640 1 -- 172.21.15.116:0/3243338674 shutdown_connections 2024-04-25T00:30:15.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.371+0000 7f664f761640 1 -- 172.21.15.116:0/3243338674 wait complete. 2024-04-25T00:30:15.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.371+0000 7f664f761640 1 Processor -- start 2024-04-25T00:30:15.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.371+0000 7f664f761640 1 -- start start 2024-04-25T00:30:15.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.375+0000 7f664f761640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f66480a94a0 0x7f66481efad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:15.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.375+0000 7f664f761640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66480c2f50 0x7f6648154f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:15.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.375+0000 7f664f761640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6648150d20 0x7f664814ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:15.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.375+0000 7f664f761640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6648157340 con 0x7f66480c2f50 2024-04-25T00:30:15.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.375+0000 7f664f761640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6648154570 con 0x7f6648150d20 2024-04-25T00:30:15.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.375+0000 7f664f761640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f66481574c0 con 0x7f66480a94a0 2024-04-25T00:30:15.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.375+0000 7f664ccfc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66480c2f50 0x7f6648154f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:15.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.375+0000 7f664ccfc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66480c2f50 0x7f6648154f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46326/0 (socket says 172.21.15.116:46326) 2024-04-25T00:30:15.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.375+0000 7f664ccfc640 1 -- 172.21.15.116:0/2527389246 learned_addr learned my addr 172.21.15.116:0/2527389246 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:15.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.375+0000 7f664d4fd640 1 --2- 172.21.15.116:0/2527389246 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f66480a94a0 0x7f66481efad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:15.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.375+0000 7f664dcfe640 1 --2- 172.21.15.116:0/2527389246 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6648150d20 0x7f664814ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:15.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.375+0000 7f664ccfc640 1 -- 172.21.15.116:0/2527389246 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f66480a94a0 msgr2=0x7f66481efad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:15.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.375+0000 7f664ccfc640 1 --2- 172.21.15.116:0/2527389246 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f66480a94a0 0x7f66481efad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:15.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.375+0000 7f664ccfc640 1 -- 172.21.15.116:0/2527389246 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6648150d20 msgr2=0x7f664814ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:15.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.375+0000 7f664ccfc640 1 --2- 172.21.15.116:0/2527389246 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6648150d20 0x7f664814ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:15.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.375+0000 7f664ccfc640 1 -- 172.21.15.116:0/2527389246 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66481f0010 con 0x7f66480c2f50 2024-04-25T00:30:15.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.375+0000 7f664ccfc640 1 --2- 172.21.15.116:0/2527389246 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66480c2f50 0x7f6648154f20 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f663c0af040 tx=0x7f663c0192c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:15.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.375+0000 7f6645ffb640 1 -- 172.21.15.116:0/2527389246 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f663c0bc070 con 0x7f66480c2f50 2024-04-25T00:30:15.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.375+0000 7f6645ffb640 1 -- 172.21.15.116:0/2527389246 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f663c0ace40 con 0x7f66480c2f50 2024-04-25T00:30:15.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.375+0000 7f664f761640 1 -- 172.21.15.116:0/2527389246 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f664814b220 con 0x7f66480c2f50 2024-04-25T00:30:15.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.375+0000 7f6645ffb640 1 -- 172.21.15.116:0/2527389246 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f663c0b1070 con 0x7f66480c2f50 2024-04-25T00:30:15.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.375+0000 7f664f761640 1 -- 172.21.15.116:0/2527389246 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f664814b7e0 con 0x7f66480c2f50 2024-04-25T00:30:15.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.375+0000 7f664f761640 1 -- 172.21.15.116:0/2527389246 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6648068350 con 0x7f66480c2f50 2024-04-25T00:30:15.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.375+0000 7f6645ffb640 1 -- 172.21.15.116:0/2527389246 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f663c0bf070 con 0x7f66480c2f50 2024-04-25T00:30:15.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.375+0000 7f6645ffb640 1 --2- 172.21.15.116:0/2527389246 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f66340678f0 0x7f6634069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:15.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.375+0000 7f6645ffb640 1 -- 172.21.15.116:0/2527389246 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 10536+0+0 (secure 0 0 0) 0x7f663c1206e0 con 0x7f66480c2f50 2024-04-25T00:30:15.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.375+0000 7f664d4fd640 1 --2- 172.21.15.116:0/2527389246 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f66340678f0 0x7f6634069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:15.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.379+0000 7f664d4fd640 1 --2- 172.21.15.116:0/2527389246 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f66340678f0 0x7f6634069db0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f6640009e90 tx=0x7f664000b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:15.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.379+0000 7f6645ffb640 1 -- 172.21.15.116:0/2527389246 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f663c016110 con 0x7f66480c2f50 2024-04-25T00:30:15.571 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.571+0000 7f664f761640 1 -- 172.21.15.116:0/2527389246 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f66480c34f0 con 0x7f66480c2f50 2024-04-25T00:30:15.572 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.571+0000 7f6645ffb640 1 -- 172.21.15.116:0/2527389246 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 237 v237) ==== 95+0+6510 (secure 0 0 0) 0x7f663c0efef0 con 0x7f66480c2f50 2024-04-25T00:30:15.573 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:30:15.573 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":237,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":6764,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6837/3873739116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3873739116},{"type":"v1","addr":"172.21.15.174:6837","nonce":3873739116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":237}],"filesystems":[{"mdsmap":{"epoch":236,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:29:54.168353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8847},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8847":{"gid":8847,"name":"c","rank":0,"incarnation":233,"state":"up:active","state_seq":17,"addr":"172.21.15.116:6833/2662028892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2662028892},{"type":"v1","addr":"172.21.15.116:6833","nonce":2662028892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8847,"qdb_cluster":[8847]},"id":1},{"mdsmap":{"epoch":231,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:28:53.808539+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8772},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8772":{"gid":8772,"name":"a","rank":0,"incarnation":228,"state":"up:active","state_seq":20,"addr":"172.21.15.116:6835/3499878473","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":3499878473},{"type":"v1","addr":"172.21.15.116:6835","nonce":3499878473}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8772,"qdb_cluster":[8772]},"id":2}]} 2024-04-25T00:30:15.573 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 237 2024-04-25T00:30:15.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.575+0000 7f664f761640 1 -- 172.21.15.116:0/2527389246 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f66340678f0 msgr2=0x7f6634069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:15.574 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.575+0000 7f664f761640 1 --2- 172.21.15.116:0/2527389246 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f66340678f0 0x7f6634069db0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f6640009e90 tx=0x7f664000b040 comp rx=0 tx=0).stop 2024-04-25T00:30:15.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.575+0000 7f664f761640 1 -- 172.21.15.116:0/2527389246 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66480c2f50 msgr2=0x7f6648154f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:15.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.575+0000 7f664f761640 1 --2- 172.21.15.116:0/2527389246 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66480c2f50 0x7f6648154f20 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f663c0af040 tx=0x7f663c0192c0 comp rx=0 tx=0).stop 2024-04-25T00:30:15.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.575+0000 7f664f761640 1 -- 172.21.15.116:0/2527389246 shutdown_connections 2024-04-25T00:30:15.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.575+0000 7f664f761640 1 --2- 172.21.15.116:0/2527389246 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f66340678f0 0x7f6634069db0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:15.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.575+0000 7f664f761640 1 --2- 172.21.15.116:0/2527389246 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6648150d20 0x7f664814ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:15.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.575+0000 7f664f761640 1 --2- 172.21.15.116:0/2527389246 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f66480c2f50 0x7f6648154f20 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:15.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.575+0000 7f664f761640 1 --2- 172.21.15.116:0/2527389246 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f66480a94a0 0x7f66481efad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:15.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.575+0000 7f664f761640 1 -- 172.21.15.116:0/2527389246 >> 172.21.15.116:0/2527389246 conn(0x7f66480c3b00 msgr2=0x7f66480a9f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:15.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.575+0000 7f664f761640 1 -- 172.21.15.116:0/2527389246 shutdown_connections 2024-04-25T00:30:15.575 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:15.575+0000 7f664f761640 1 -- 172.21.15.116:0/2527389246 wait complete. 2024-04-25T00:30:15.590 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2024-04-25T00:30:17.592 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:30:17.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.719+0000 7fb10474d640 1 Processor -- start 2024-04-25T00:30:17.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.719+0000 7fb10474d640 1 -- start start 2024-04-25T00:30:17.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.723+0000 7fb10474d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb0fc150d20 0x7fb0fc0c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:17.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.723+0000 7fb10474d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb0fc0a94a0 0x7fb0fc153ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:17.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.723+0000 7fb10474d640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0fc0c3910 con 0x7fb0fc0c2f50 2024-04-25T00:30:17.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.723+0000 7fb10474d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fb0fc154530 con 0x7fb0fc150d20 2024-04-25T00:30:17.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.723+0000 7fb10474d640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fb0fc0c16e0 con 0x7fb0fc0a94a0 2024-04-25T00:30:17.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.723+0000 7fb1024e9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb0fc150d20 0x7fb0fc0c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:17.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.723+0000 7fb1024e9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb0fc150d20 0x7fb0fc0c2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46336/0 (socket says 172.21.15.116:46336) 2024-04-25T00:30:17.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.723+0000 7fb1024e9640 1 -- 172.21.15.116:0/3295148551 learned_addr learned my addr 172.21.15.116:0/3295148551 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:17.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.723+0000 7fb102cea640 1 --2- 172.21.15.116:0/3295148551 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb0fc0a94a0 0x7fb0fc153ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:17.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.723+0000 7fb1024e9640 1 -- 172.21.15.116:0/3295148551 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb0fc0a94a0 msgr2=0x7fb0fc153ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:17.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.723+0000 7fb1024e9640 1 --2- 172.21.15.116:0/3295148551 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb0fc0a94a0 0x7fb0fc153ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:17.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.723+0000 7fb1024e9640 1 -- 172.21.15.116:0/3295148551 >> v1:172.21.15.174:6789/0 conn(0x7fb0fc0c2f50 legacy=0x7fb0fc1538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:17.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.723+0000 7fb1024e9640 1 -- 172.21.15.116:0/3295148551 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0fc0c1400 con 0x7fb0fc150d20 2024-04-25T00:30:17.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.723+0000 7fb1024e9640 1 --2- 172.21.15.116:0/3295148551 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0fc150d20 0x7fb0fc0c2a10 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7fb0f8009870 tx=0x7fb0f801c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3042fc9cbaa1cf8a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:17.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.723+0000 7fb1014e7640 1 -- 172.21.15.116:0/3295148551 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f80ab5f0 con 0x7fb0fc150d20 2024-04-25T00:30:17.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.723+0000 7fb1014e7640 1 -- 172.21.15.116:0/3295148551 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0f80b0070 con 0x7fb0fc150d20 2024-04-25T00:30:17.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.723+0000 7fb1014e7640 1 -- 172.21.15.116:0/3295148551 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f80ab9b0 con 0x7fb0fc150d20 2024-04-25T00:30:17.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.727+0000 7fb10474d640 1 -- 172.21.15.116:0/3295148551 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0fc150d20 msgr2=0x7fb0fc0c2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:17.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.727+0000 7fb10474d640 1 --2- 172.21.15.116:0/3295148551 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0fc150d20 0x7fb0fc0c2a10 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7fb0f8009870 tx=0x7fb0f801c520 comp rx=0 tx=0).stop 2024-04-25T00:30:17.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.727+0000 7fb10474d640 1 -- 172.21.15.116:0/3295148551 shutdown_connections 2024-04-25T00:30:17.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.727+0000 7fb10474d640 1 --2- 172.21.15.116:0/3295148551 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb0fc0a94a0 0x7fb0fc153ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:17.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.727+0000 7fb10474d640 1 --2- 172.21.15.116:0/3295148551 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0fc150d20 0x7fb0fc0c2a10 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:17.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.727+0000 7fb10474d640 1 -- 172.21.15.116:0/3295148551 >> 172.21.15.116:0/3295148551 conn(0x7fb0fc0c3b00 msgr2=0x7fb0fc0c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:17.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.727+0000 7fb10474d640 1 -- 172.21.15.116:0/3295148551 shutdown_connections 2024-04-25T00:30:17.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.727+0000 7fb10474d640 1 -- 172.21.15.116:0/3295148551 wait complete. 2024-04-25T00:30:17.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.727+0000 7fb10474d640 1 Processor -- start 2024-04-25T00:30:17.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.727+0000 7fb10474d640 1 -- start start 2024-04-25T00:30:17.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.727+0000 7fb10474d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb0fc0a94a0 0x7fb0fc144bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:17.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.727+0000 7fb10474d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0fc0c2f50 0x7fb0fc1f1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:17.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.727+0000 7fb10474d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb0fc150d20 0x7fb0fc15c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:17.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.727+0000 7fb10474d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb0fc1608b0 con 0x7fb0fc0c2f50 2024-04-25T00:30:17.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.727+0000 7fb10474d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb0fc160730 con 0x7fb0fc0a94a0 2024-04-25T00:30:17.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.727+0000 7fb10474d640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fb0fc160a30 con 0x7fb0fc150d20 2024-04-25T00:30:17.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.727+0000 7fb101ce8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0fc0c2f50 0x7fb0fc1f1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:17.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.727+0000 7fb101ce8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0fc0c2f50 0x7fb0fc1f1ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46340/0 (socket says 172.21.15.116:46340) 2024-04-25T00:30:17.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.727+0000 7fb101ce8640 1 -- 172.21.15.116:0/3261045810 learned_addr learned my addr 172.21.15.116:0/3261045810 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:17.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.727+0000 7fb1024e9640 1 --2- 172.21.15.116:0/3261045810 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb0fc0a94a0 0x7fb0fc144bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:17.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.727+0000 7fb102cea640 1 --2- 172.21.15.116:0/3261045810 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb0fc150d20 0x7fb0fc15c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:17.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.727+0000 7fb101ce8640 1 -- 172.21.15.116:0/3261045810 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb0fc150d20 msgr2=0x7fb0fc15c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:17.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.727+0000 7fb101ce8640 1 --2- 172.21.15.116:0/3261045810 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb0fc150d20 0x7fb0fc15c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:17.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.727+0000 7fb101ce8640 1 -- 172.21.15.116:0/3261045810 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb0fc0a94a0 msgr2=0x7fb0fc144bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:17.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.727+0000 7fb101ce8640 1 --2- 172.21.15.116:0/3261045810 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb0fc0a94a0 0x7fb0fc144bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:17.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.727+0000 7fb101ce8640 1 -- 172.21.15.116:0/3261045810 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0fc14ee60 con 0x7fb0fc0c2f50 2024-04-25T00:30:17.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.727+0000 7fb101ce8640 1 --2- 172.21.15.116:0/3261045810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0fc0c2f50 0x7fb0fc1f1ad0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec00c450 tx=0x7fb0ec00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:17.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.727+0000 7fb0f2ffd640 1 -- 172.21.15.116:0/3261045810 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0ec012830 con 0x7fb0fc0c2f50 2024-04-25T00:30:17.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.727+0000 7fb0f2ffd640 1 -- 172.21.15.116:0/3261045810 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0ec017070 con 0x7fb0fc0c2f50 2024-04-25T00:30:17.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.727+0000 7fb10474d640 1 -- 172.21.15.116:0/3261045810 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0fc14afd0 con 0x7fb0fc0c2f50 2024-04-25T00:30:17.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.727+0000 7fb0f2ffd640 1 -- 172.21.15.116:0/3261045810 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0ec012c30 con 0x7fb0fc0c2f50 2024-04-25T00:30:17.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.727+0000 7fb10474d640 1 -- 172.21.15.116:0/3261045810 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0fc14b590 con 0x7fb0fc0c2f50 2024-04-25T00:30:17.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.731+0000 7fb10474d640 1 -- 172.21.15.116:0/3261045810 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0fc068350 con 0x7fb0fc0c2f50 2024-04-25T00:30:17.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.731+0000 7fb0f2ffd640 1 -- 172.21.15.116:0/3261045810 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb0ec01c460 con 0x7fb0fc0c2f50 2024-04-25T00:30:17.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.731+0000 7fb0f2ffd640 1 --2- 172.21.15.116:0/3261045810 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb0e00678c0 0x7fb0e0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:17.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.731+0000 7fb0f2ffd640 1 -- 172.21.15.116:0/3261045810 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7fb0ec086e20 con 0x7fb0fc0c2f50 2024-04-25T00:30:17.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.731+0000 7fb1024e9640 1 --2- 172.21.15.116:0/3261045810 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb0e00678c0 0x7fb0e0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:17.731 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.731+0000 7fb1024e9640 1 --2- 172.21.15.116:0/3261045810 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb0e00678c0 0x7fb0e0069d80 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fb0f80af000 tx=0x7fb0f80a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:17.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.735+0000 7fb0f2ffd640 1 -- 172.21.15.116:0/3261045810 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fb0ec056580 con 0x7fb0fc0c2f50 2024-04-25T00:30:17.929 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.927+0000 7fb10474d640 1 -- 172.21.15.116:0/3261045810 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb0fc1475e0 con 0x7fb0fc0c2f50 2024-04-25T00:30:17.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.931+0000 7fb0f2ffd640 1 -- 172.21.15.116:0/3261045810 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 239 v239) ==== 95+0+5676 (secure 0 0 0) 0x7fb0ec055e80 con 0x7fb0fc0c2f50 2024-04-25T00:30:17.930 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:30:17.930 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":239,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":236,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:29:54.168353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8847},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8847":{"gid":8847,"name":"c","rank":0,"incarnation":233,"state":"up:active","state_seq":17,"addr":"172.21.15.116:6833/2662028892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2662028892},{"type":"v1","addr":"172.21.15.116:6833","nonce":2662028892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8847,"qdb_cluster":[8847]},"id":1},{"mdsmap":{"epoch":239,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:30:16.321617+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6764},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6764":{"gid":6764,"name":"d","rank":0,"incarnation":238,"state":"up:reconnect","state_seq":6,"addr":"172.21.15.174:6837/3873739116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3873739116},{"type":"v1","addr":"172.21.15.174:6837","nonce":3873739116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:30:17.930 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 239 2024-04-25T00:30:17.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.931+0000 7fb10474d640 1 -- 172.21.15.116:0/3261045810 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb0e00678c0 msgr2=0x7fb0e0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:17.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.931+0000 7fb10474d640 1 --2- 172.21.15.116:0/3261045810 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb0e00678c0 0x7fb0e0069d80 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fb0f80af000 tx=0x7fb0f80a7000 comp rx=0 tx=0).stop 2024-04-25T00:30:17.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.931+0000 7fb10474d640 1 -- 172.21.15.116:0/3261045810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0fc0c2f50 msgr2=0x7fb0fc1f1ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:17.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.931+0000 7fb10474d640 1 --2- 172.21.15.116:0/3261045810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0fc0c2f50 0x7fb0fc1f1ad0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec00c450 tx=0x7fb0ec00c910 comp rx=0 tx=0).stop 2024-04-25T00:30:17.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.931+0000 7fb10474d640 1 -- 172.21.15.116:0/3261045810 shutdown_connections 2024-04-25T00:30:17.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.931+0000 7fb10474d640 1 --2- 172.21.15.116:0/3261045810 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb0e00678c0 0x7fb0e0069d80 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:17.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.931+0000 7fb10474d640 1 --2- 172.21.15.116:0/3261045810 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb0fc150d20 0x7fb0fc15c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:17.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.931+0000 7fb10474d640 1 --2- 172.21.15.116:0/3261045810 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb0fc0c2f50 0x7fb0fc1f1ad0 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:17.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.931+0000 7fb10474d640 1 --2- 172.21.15.116:0/3261045810 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb0fc0a94a0 0x7fb0fc144bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:17.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.931+0000 7fb10474d640 1 -- 172.21.15.116:0/3261045810 >> 172.21.15.116:0/3261045810 conn(0x7fb0fc0c3b00 msgr2=0x7fb0fc15d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:17.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.931+0000 7fb10474d640 1 -- 172.21.15.116:0/3261045810 shutdown_connections 2024-04-25T00:30:17.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:17.931+0000 7fb10474d640 1 -- 172.21.15.116:0/3261045810 wait complete. 2024-04-25T00:30:17.948 INFO:tasks.mds_thrash.fs.[b]:mds.a down, removed from mdsmap 2024-04-25T00:30:17.948 INFO:tasks.mds_thrash.fs.[b]:waiting for mds cluster to stabilize... 2024-04-25T00:30:17.948 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:30:18.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.059+0000 7fbf27ed0640 1 Processor -- start 2024-04-25T00:30:18.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.059+0000 7fbf27ed0640 1 -- start start 2024-04-25T00:30:18.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.063+0000 7fbf27ed0640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbf2006d5f0 0x7fbf20156c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:18.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.063+0000 7fbf27ed0640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbf20155810 0x7fbf201571c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:18.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.063+0000 7fbf27ed0640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf200c9150 con 0x7fbf20152d80 2024-04-25T00:30:18.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.063+0000 7fbf27ed0640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fbf200c9d60 con 0x7fbf2006d5f0 2024-04-25T00:30:18.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.063+0000 7fbf27ed0640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fbf200c9430 con 0x7fbf20155810 2024-04-25T00:30:18.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.063+0000 7fbf2546b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbf2006d5f0 0x7fbf20156c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:18.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.063+0000 7fbf2546b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fbf2006d5f0 0x7fbf20156c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46342/0 (socket says 172.21.15.116:46342) 2024-04-25T00:30:18.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.063+0000 7fbf2546b640 1 -- 172.21.15.116:0/2646345882 learned_addr learned my addr 172.21.15.116:0/2646345882 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:18.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.063+0000 7fbf2646d640 1 --2- 172.21.15.116:0/2646345882 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbf20155810 0x7fbf201571c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:18.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.063+0000 7fbf2546b640 1 -- 172.21.15.116:0/2646345882 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbf20155810 msgr2=0x7fbf201571c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:18.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.063+0000 7fbf2546b640 1 --2- 172.21.15.116:0/2646345882 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbf20155810 0x7fbf201571c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:18.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.063+0000 7fbf2546b640 1 -- 172.21.15.116:0/2646345882 >> v1:172.21.15.174:6789/0 conn(0x7fbf20152d80 legacy=0x7fbf2006bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:18.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.063+0000 7fbf2546b640 1 -- 172.21.15.116:0/2646345882 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf200c9710 con 0x7fbf2006d5f0 2024-04-25T00:30:18.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.063+0000 7fbf2546b640 1 --2- 172.21.15.116:0/2646345882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf2006d5f0 0x7fbf20156c80 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c009870 tx=0x7fbf1c01c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41513c5ff3619bb3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:18.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.063+0000 7fbf24c6a640 1 -- 172.21.15.116:0/2646345882 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf1c0b1070 con 0x7fbf2006d5f0 2024-04-25T00:30:18.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.063+0000 7fbf24c6a640 1 -- 172.21.15.116:0/2646345882 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf1c0ac630 con 0x7fbf2006d5f0 2024-04-25T00:30:18.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.063+0000 7fbf24c6a640 1 -- 172.21.15.116:0/2646345882 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf1c0a8050 con 0x7fbf2006d5f0 2024-04-25T00:30:18.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.063+0000 7fbf27ed0640 1 -- 172.21.15.116:0/2646345882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf2006d5f0 msgr2=0x7fbf20156c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:18.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.063+0000 7fbf27ed0640 1 --2- 172.21.15.116:0/2646345882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf2006d5f0 0x7fbf20156c80 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c009870 tx=0x7fbf1c01c4a0 comp rx=0 tx=0).stop 2024-04-25T00:30:18.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.063+0000 7fbf27ed0640 1 -- 172.21.15.116:0/2646345882 shutdown_connections 2024-04-25T00:30:18.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.063+0000 7fbf27ed0640 1 --2- 172.21.15.116:0/2646345882 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbf20155810 0x7fbf201571c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:18.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.063+0000 7fbf27ed0640 1 --2- 172.21.15.116:0/2646345882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf2006d5f0 0x7fbf20156c80 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:18.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.063+0000 7fbf27ed0640 1 -- 172.21.15.116:0/2646345882 >> 172.21.15.116:0/2646345882 conn(0x7fbf200acdd0 msgr2=0x7fbf200ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:18.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.063+0000 7fbf27ed0640 1 -- 172.21.15.116:0/2646345882 shutdown_connections 2024-04-25T00:30:18.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.063+0000 7fbf27ed0640 1 -- 172.21.15.116:0/2646345882 wait complete. 2024-04-25T00:30:18.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.067+0000 7fbf27ed0640 1 Processor -- start 2024-04-25T00:30:18.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.067+0000 7fbf27ed0640 1 -- start start 2024-04-25T00:30:18.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.067+0000 7fbf27ed0640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf2006d5f0 0x7fbf20151f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:18.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.067+0000 7fbf27ed0640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbf20152d80 0x7fbf200adcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:18.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.067+0000 7fbf27ed0640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf20155810 0x7fbf2014be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:18.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.067+0000 7fbf27ed0640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fbf200caf00 con 0x7fbf2006d5f0 2024-04-25T00:30:18.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.067+0000 7fbf27ed0640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fbf200cad80 con 0x7fbf20155810 2024-04-25T00:30:18.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.067+0000 7fbf27ed0640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fbf201628d0 con 0x7fbf20152d80 2024-04-25T00:30:18.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.067+0000 7fbf25c6c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf2006d5f0 0x7fbf20151f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:18.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.067+0000 7fbf25c6c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf2006d5f0 0x7fbf20151f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46344/0 (socket says 172.21.15.116:46344) 2024-04-25T00:30:18.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.067+0000 7fbf25c6c640 1 -- 172.21.15.116:0/3862682688 learned_addr learned my addr 172.21.15.116:0/3862682688 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:18.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.067+0000 7fbf2646d640 1 --2- 172.21.15.116:0/3862682688 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf20155810 0x7fbf2014be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:18.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.067+0000 7fbf2546b640 1 --2- 172.21.15.116:0/3862682688 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbf20152d80 0x7fbf200adcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:18.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.067+0000 7fbf25c6c640 1 -- 172.21.15.116:0/3862682688 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbf20152d80 msgr2=0x7fbf200adcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:18.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.067+0000 7fbf25c6c640 1 --2- 172.21.15.116:0/3862682688 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbf20152d80 0x7fbf200adcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:18.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.067+0000 7fbf25c6c640 1 -- 172.21.15.116:0/3862682688 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf20155810 msgr2=0x7fbf2014be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:18.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.067+0000 7fbf25c6c640 1 --2- 172.21.15.116:0/3862682688 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf20155810 0x7fbf2014be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:18.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.067+0000 7fbf25c6c640 1 -- 172.21.15.116:0/3862682688 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf200b05a0 con 0x7fbf2006d5f0 2024-04-25T00:30:18.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.067+0000 7fbf2546b640 1 --2- 172.21.15.116:0/3862682688 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbf20152d80 0x7fbf200adcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:30:18.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.067+0000 7fbf25c6c640 1 --2- 172.21.15.116:0/3862682688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf2006d5f0 0x7fbf20151f70 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7fbf1800c730 tx=0x7fbf1800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:18.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.067+0000 7fbf167fc640 1 -- 172.21.15.116:0/3862682688 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf18016020 con 0x7fbf2006d5f0 2024-04-25T00:30:18.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.067+0000 7fbf167fc640 1 -- 172.21.15.116:0/3862682688 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf18006d60 con 0x7fbf2006d5f0 2024-04-25T00:30:18.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.067+0000 7fbf27ed0640 1 -- 172.21.15.116:0/3862682688 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf201ef9e0 con 0x7fbf2006d5f0 2024-04-25T00:30:18.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.067+0000 7fbf167fc640 1 -- 172.21.15.116:0/3862682688 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf18003bc0 con 0x7fbf2006d5f0 2024-04-25T00:30:18.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.067+0000 7fbf27ed0640 1 -- 172.21.15.116:0/3862682688 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf201ef700 con 0x7fbf2006d5f0 2024-04-25T00:30:18.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.067+0000 7fbf27ed0640 1 -- 172.21.15.116:0/3862682688 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf20068350 con 0x7fbf2006d5f0 2024-04-25T00:30:18.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.067+0000 7fbf167fc640 1 -- 172.21.15.116:0/3862682688 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fbf1800a850 con 0x7fbf2006d5f0 2024-04-25T00:30:18.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.071+0000 7fbf167fc640 1 --2- 172.21.15.116:0/3862682688 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbf0c0678c0 0x7fbf0c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:18.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.071+0000 7fbf167fc640 1 -- 172.21.15.116:0/3862682688 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7fbf18086c50 con 0x7fbf2006d5f0 2024-04-25T00:30:18.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.071+0000 7fbf2546b640 1 --2- 172.21.15.116:0/3862682688 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbf0c0678c0 0x7fbf0c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:18.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.071+0000 7fbf2546b640 1 --2- 172.21.15.116:0/3862682688 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbf0c0678c0 0x7fbf0c069d80 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c0ae000 tx=0x7fbf1c0a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:18.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.075+0000 7fbf167fc640 1 -- 172.21.15.116:0/3862682688 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fbf1808c020 con 0x7fbf2006d5f0 2024-04-25T00:30:18.268 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.267+0000 7fbf27ed0640 1 -- 172.21.15.116:0/3862682688 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbf200759d0 con 0x7fbf2006d5f0 2024-04-25T00:30:18.269 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.271+0000 7fbf167fc640 1 -- 172.21.15.116:0/3862682688 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 240 v240) ==== 95+0+5673 (secure 0 0 0) 0x7fbf18020020 con 0x7fbf2006d5f0 2024-04-25T00:30:18.269 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:30:18.270 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":240,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":236,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:29:54.168353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8847},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8847":{"gid":8847,"name":"c","rank":0,"incarnation":233,"state":"up:active","state_seq":17,"addr":"172.21.15.116:6833/2662028892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2662028892},{"type":"v1","addr":"172.21.15.116:6833","nonce":2662028892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8847,"qdb_cluster":[8847]},"id":1},{"mdsmap":{"epoch":240,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:30:17.111250+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6764},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6764":{"gid":6764,"name":"d","rank":0,"incarnation":238,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.174:6837/3873739116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3873739116},{"type":"v1","addr":"172.21.15.174:6837","nonce":3873739116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T00:30:18.270 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 240 2024-04-25T00:30:18.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.271+0000 7fbf27ed0640 1 -- 172.21.15.116:0/3862682688 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbf0c0678c0 msgr2=0x7fbf0c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:18.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.271+0000 7fbf27ed0640 1 --2- 172.21.15.116:0/3862682688 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbf0c0678c0 0x7fbf0c069d80 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c0ae000 tx=0x7fbf1c0a7000 comp rx=0 tx=0).stop 2024-04-25T00:30:18.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.271+0000 7fbf27ed0640 1 -- 172.21.15.116:0/3862682688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf2006d5f0 msgr2=0x7fbf20151f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:18.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.271+0000 7fbf27ed0640 1 --2- 172.21.15.116:0/3862682688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf2006d5f0 0x7fbf20151f70 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7fbf1800c730 tx=0x7fbf1800cbf0 comp rx=0 tx=0).stop 2024-04-25T00:30:18.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.271+0000 7fbf27ed0640 1 -- 172.21.15.116:0/3862682688 shutdown_connections 2024-04-25T00:30:18.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.271+0000 7fbf27ed0640 1 --2- 172.21.15.116:0/3862682688 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fbf0c0678c0 0x7fbf0c069d80 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:18.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.271+0000 7fbf27ed0640 1 --2- 172.21.15.116:0/3862682688 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fbf20155810 0x7fbf2014be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:18.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.271+0000 7fbf27ed0640 1 --2- 172.21.15.116:0/3862682688 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fbf20152d80 0x7fbf200adcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:18.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.271+0000 7fbf27ed0640 1 --2- 172.21.15.116:0/3862682688 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fbf2006d5f0 0x7fbf20151f70 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:18.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.271+0000 7fbf27ed0640 1 -- 172.21.15.116:0/3862682688 >> 172.21.15.116:0/3862682688 conn(0x7fbf200acdd0 msgr2=0x7fbf20159380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:18.272 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.271+0000 7fbf27ed0640 1 -- 172.21.15.116:0/3862682688 shutdown_connections 2024-04-25T00:30:18.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:18.271+0000 7fbf27ed0640 1 -- 172.21.15.116:0/3862682688 wait complete. 2024-04-25T00:30:18.288 INFO:tasks.mds_thrash.fs.[b]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 240, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:04.678737+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 236, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": { "gid_8847": { "addr": "172.21.15.116:6833/2662028892", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6832", "nonce": 2662028892, "type": "v2" }, { "addr": "172.21.15.116:6833", "nonce": 2662028892, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 8847, "incarnation": 233, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:active", "state_seq": 17 } }, "last_failure": 0, "last_failure_osd_epoch": 73, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T00:29:54.168353+0000", "qdb_cluster": [ 8847 ], "qdb_leader": 8847, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8847 } } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:18.825887+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 240, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6764": { "addr": "172.21.15.174:6837/3873739116", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6836", "nonce": 3873739116, "type": "v2" }, { "addr": "172.21.15.174:6837", "nonce": 3873739116, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6764, "incarnation": 238, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:rejoin", "state_seq": 7 } }, "last_failure": 0, "last_failure_osd_epoch": 74, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-04-25T00:30:17.111250+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6764 } } } ], "standbys": [ { "addr": "172.21.15.174:6835/4112145782", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6834", "nonce": 4112145782, "type": "v2" }, { "addr": "172.21.15.174:6835", "nonce": 4112145782, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 112, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5582, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-25T00:30:19.194 INFO:tasks.workunit.client.1.smithi174.stderr:+ rm -r tmp ffsb 2024-04-25T00:30:20.289 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:30:20.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.423+0000 7f65f5b58640 1 Processor -- start 2024-04-25T00:30:20.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.423+0000 7f65f5b58640 1 -- start start 2024-04-25T00:30:20.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.423+0000 7f65f5b58640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f65f00a94a0 0x7f65f01538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:20.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.423+0000 7f65f5b58640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65f00c3910 con 0x7f65f0150d20 2024-04-25T00:30:20.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.423+0000 7f65f5b58640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65f0154530 con 0x7f65f00a99f0 2024-04-25T00:30:20.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.423+0000 7f65f5b58640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f65f00c16e0 con 0x7f65f00a94a0 2024-04-25T00:30:20.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.423+0000 7f65ef7fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f65f0150d20 0x7f65f00c2a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54538/0 (socket says 172.21.15.116:54538) 2024-04-25T00:30:20.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.423+0000 7f65ef7fe640 1 -- 172.21.15.116:0/652211273 learned_addr learned my addr 172.21.15.116:0/652211273 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:30:20.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.423+0000 7f65e7fff640 1 --2- 172.21.15.116:0/652211273 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f65f00a94a0 0x7f65f01538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:20.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.423+0000 7f65eeffd640 1 -- 172.21.15.116:0/652211273 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4273347527 0 0) 0x7f65f00c3910 con 0x7f65f0150d20 2024-04-25T00:30:20.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.423+0000 7f65eeffd640 1 -- 172.21.15.116:0/652211273 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65dc003530 con 0x7f65f0150d20 2024-04-25T00:30:20.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.423+0000 7f65eeffd640 1 -- 172.21.15.116:0/652211273 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2624301760 0 0) 0x7f65dc003530 con 0x7f65f0150d20 2024-04-25T00:30:20.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.423+0000 7f65eeffd640 1 -- 172.21.15.116:0/652211273 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f65f00a94a0 msgr2=0x7f65f01538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:20.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.423+0000 7f65eeffd640 1 --2- 172.21.15.116:0/652211273 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f65f00a94a0 0x7f65f01538e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:20.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.423+0000 7f65eeffd640 1 -- 172.21.15.116:0/652211273 >> v1:172.21.15.174:6789/0 conn(0x7f65f00a99f0 legacy=0x7f65f0153e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:20.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.423+0000 7f65eeffd640 1 -- 172.21.15.116:0/652211273 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65f00c1400 con 0x7f65f0150d20 2024-04-25T00:30:20.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.423+0000 7f65eeffd640 1 -- 172.21.15.116:0/652211273 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f65e0001680 con 0x7f65f0150d20 2024-04-25T00:30:20.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.423+0000 7f65eeffd640 1 -- 172.21.15.116:0/652211273 >> v1:172.21.15.116:6789/0 conn(0x7f65f0150d20 legacy=0x7f65f00c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:20.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.423+0000 7f65eeffd640 1 --2- 172.21.15.116:0/652211273 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65dc0039f0 0x7f65dc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:20.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.423+0000 7f65eeffd640 1 --2- 172.21.15.116:0/652211273 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f65f00a94a0 0x7f65dc004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:20.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.423+0000 7f65eeffd640 1 --2- 172.21.15.116:0/652211273 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65dc004d50 0x7f65dc0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:20.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.423+0000 7f65eeffd640 1 -- 172.21.15.116:0/652211273 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f65f00c16e0 con 0x7f65dc004d50 2024-04-25T00:30:20.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.423+0000 7f65eeffd640 1 -- 172.21.15.116:0/652211273 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f65dc00b9d0 con 0x7f65dc0039f0 2024-04-25T00:30:20.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.423+0000 7f65eeffd640 1 -- 172.21.15.116:0/652211273 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f65dc00bbd0 con 0x7f65f00a94a0 2024-04-25T00:30:20.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.423+0000 7f65effff640 1 --2- 172.21.15.116:0/652211273 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65dc004d50 0x7f65dc0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:20.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.423+0000 7f65ef7fe640 1 --2- 172.21.15.116:0/652211273 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65dc0039f0 0x7f65dc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:20.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.423+0000 7f65e7fff640 1 --2- 172.21.15.116:0/652211273 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f65f00a94a0 0x7f65dc004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:20.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.427+0000 7f65effff640 1 -- 172.21.15.116:0/652211273 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f65f00a94a0 msgr2=0x7f65dc004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:20.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.427+0000 7f65effff640 1 --2- 172.21.15.116:0/652211273 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f65f00a94a0 0x7f65dc004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:20.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.427+0000 7f65effff640 1 -- 172.21.15.116:0/652211273 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65dc0039f0 msgr2=0x7f65dc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:20.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.427+0000 7f65effff640 1 --2- 172.21.15.116:0/652211273 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65dc0039f0 0x7f65dc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:20.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.427+0000 7f65effff640 1 -- 172.21.15.116:0/652211273 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65dc003530 con 0x7f65dc004d50 2024-04-25T00:30:20.428 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.427+0000 7f65ef7fe640 1 --2- 172.21.15.116:0/652211273 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65dc0039f0 0x7f65dc003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:30:20.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.427+0000 7f65effff640 1 --2- 172.21.15.116:0/652211273 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65dc004d50 0x7f65dc0051d0 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f65e8002010 tx=0x7f65e801c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ab180c78c36b3cb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:20.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.427+0000 7f65eeffd640 1 -- 172.21.15.116:0/652211273 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65e8001c10 con 0x7f65dc004d50 2024-04-25T00:30:20.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.427+0000 7f65eeffd640 1 -- 172.21.15.116:0/652211273 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65e80a66f0 con 0x7f65dc004d50 2024-04-25T00:30:20.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.427+0000 7f65eeffd640 1 -- 172.21.15.116:0/652211273 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65e80a6ae0 con 0x7f65dc004d50 2024-04-25T00:30:20.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.427+0000 7f65f5b58640 1 -- 172.21.15.116:0/652211273 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65dc004d50 msgr2=0x7f65dc0051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:20.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.427+0000 7f65f5b58640 1 --2- 172.21.15.116:0/652211273 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65dc004d50 0x7f65dc0051d0 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f65e8002010 tx=0x7f65e801c8e0 comp rx=0 tx=0).stop 2024-04-25T00:30:20.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.427+0000 7f65effff640 1 -- 172.21.15.116:0/652211273 reap_dead start 2024-04-25T00:30:20.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.427+0000 7f65f5b58640 1 -- 172.21.15.116:0/652211273 shutdown_connections 2024-04-25T00:30:20.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.427+0000 7f65f5b58640 1 -- 172.21.15.116:0/652211273 >> 172.21.15.116:0/652211273 conn(0x7f65f00c3b00 msgr2=0x7f65f00c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:20.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.427+0000 7f65f5b58640 1 -- 172.21.15.116:0/652211273 shutdown_connections 2024-04-25T00:30:20.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.427+0000 7f65f5b58640 1 -- 172.21.15.116:0/652211273 wait complete. 2024-04-25T00:30:20.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.431+0000 7f65f5b58640 1 Processor -- start 2024-04-25T00:30:20.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.431+0000 7f65f5b58640 1 -- start start 2024-04-25T00:30:20.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.431+0000 7f65f5b58640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65dc004d50 0x7f65f00c0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:20.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.431+0000 7f65f5b58640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65f00c66f0 0x7f65f01ee940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:20.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.431+0000 7f65f5b58640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f65f01eee80 0x7f65f013f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:20.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.431+0000 7f65f5b58640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f65f006ec20 con 0x7f65dc004d50 2024-04-25T00:30:20.432 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.431+0000 7f65f5b58640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f65f006e940 con 0x7f65f00c66f0 2024-04-25T00:30:20.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.431+0000 7f65f5b58640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f65f00c3120 con 0x7f65f01eee80 2024-04-25T00:30:20.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.431+0000 7f65ef7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65dc004d50 0x7f65f00c0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:20.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.431+0000 7f65ef7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65dc004d50 0x7f65f00c0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46368/0 (socket says 172.21.15.116:46368) 2024-04-25T00:30:20.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.431+0000 7f65ef7fe640 1 -- 172.21.15.116:0/1209435168 learned_addr learned my addr 172.21.15.116:0/1209435168 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:20.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.431+0000 7f65e7fff640 1 --2- 172.21.15.116:0/1209435168 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65f00c66f0 0x7f65f01ee940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:20.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.431+0000 7f65effff640 1 --2- 172.21.15.116:0/1209435168 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f65f01eee80 0x7f65f013f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:20.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.431+0000 7f65ef7fe640 1 -- 172.21.15.116:0/1209435168 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f65f01eee80 msgr2=0x7f65f013f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:20.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.431+0000 7f65ef7fe640 1 --2- 172.21.15.116:0/1209435168 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f65f01eee80 0x7f65f013f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:20.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.431+0000 7f65ef7fe640 1 -- 172.21.15.116:0/1209435168 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65f00c66f0 msgr2=0x7f65f01ee940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:20.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.431+0000 7f65ef7fe640 1 --2- 172.21.15.116:0/1209435168 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65f00c66f0 0x7f65f01ee940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:20.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.431+0000 7f65ef7fe640 1 -- 172.21.15.116:0/1209435168 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65f0071430 con 0x7f65dc004d50 2024-04-25T00:30:20.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.431+0000 7f65ef7fe640 1 --2- 172.21.15.116:0/1209435168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65dc004d50 0x7f65f00c0690 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f65e0011980 tx=0x7f65e0011e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:20.433 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.431+0000 7f65ecff9640 1 -- 172.21.15.116:0/1209435168 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65e0010040 con 0x7f65dc004d50 2024-04-25T00:30:20.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.431+0000 7f65ecff9640 1 -- 172.21.15.116:0/1209435168 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65e001a070 con 0x7f65dc004d50 2024-04-25T00:30:20.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.431+0000 7f65f5b58640 1 -- 172.21.15.116:0/1209435168 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65f0071710 con 0x7f65dc004d50 2024-04-25T00:30:20.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.431+0000 7f65ecff9640 1 -- 172.21.15.116:0/1209435168 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65e000fe20 con 0x7f65dc004d50 2024-04-25T00:30:20.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.431+0000 7f65f5b58640 1 -- 172.21.15.116:0/1209435168 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65f00c0bd0 con 0x7f65dc004d50 2024-04-25T00:30:20.434 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.431+0000 7f65f5b58640 1 -- 172.21.15.116:0/1209435168 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65f01ee240 con 0x7f65dc004d50 2024-04-25T00:30:20.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.431+0000 7f65ecff9640 1 -- 172.21.15.116:0/1209435168 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f65e001f460 con 0x7f65dc004d50 2024-04-25T00:30:20.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.435+0000 7f65ecff9640 1 --2- 172.21.15.116:0/1209435168 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f65d00678f0 0x7f65d0069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:20.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.435+0000 7f65ecff9640 1 -- 172.21.15.116:0/1209435168 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7f65e008a230 con 0x7f65dc004d50 2024-04-25T00:30:20.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.435+0000 7f65e7fff640 1 --2- 172.21.15.116:0/1209435168 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f65d00678f0 0x7f65d0069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:20.435 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.435+0000 7f65e7fff640 1 --2- 172.21.15.116:0/1209435168 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f65d00678f0 0x7f65d0069db0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f65f00c66a0 tx=0x7f65d8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:20.438 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.439+0000 7f65ecff9640 1 -- 172.21.15.116:0/1209435168 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f65f01ee240 con 0x7f65dc004d50 2024-04-25T00:30:20.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.635+0000 7f65f5b58640 1 -- 172.21.15.116:0/1209435168 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f65f0068350 con 0x7f65dc004d50 2024-04-25T00:30:20.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.635+0000 7f65ecff9640 1 -- 172.21.15.116:0/1209435168 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 241 v241) ==== 95+0+5680 (secure 0 0 0) 0x7f65e00599f0 con 0x7f65dc004d50 2024-04-25T00:30:20.636 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:30:20.636 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":241,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":236,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:29:54.168353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8847},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8847":{"gid":8847,"name":"c","rank":0,"incarnation":233,"state":"up:active","state_seq":17,"addr":"172.21.15.116:6833/2662028892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2662028892},{"type":"v1","addr":"172.21.15.116:6833","nonce":2662028892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8847,"qdb_cluster":[8847]},"id":1},{"mdsmap":{"epoch":241,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:30:19.111393+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6764},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6764":{"gid":6764,"name":"d","rank":0,"incarnation":238,"state":"up:active","state_seq":8,"addr":"172.21.15.174:6837/3873739116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3873739116},{"type":"v1","addr":"172.21.15.174:6837","nonce":3873739116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6764,"qdb_cluster":[6764]},"id":2}]} 2024-04-25T00:30:20.636 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 241 2024-04-25T00:30:20.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.639+0000 7f65f5b58640 1 -- 172.21.15.116:0/1209435168 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f65d00678f0 msgr2=0x7f65d0069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:20.638 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.639+0000 7f65f5b58640 1 --2- 172.21.15.116:0/1209435168 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f65d00678f0 0x7f65d0069db0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f65f00c66a0 tx=0x7f65d8008040 comp rx=0 tx=0).stop 2024-04-25T00:30:20.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.639+0000 7f65f5b58640 1 -- 172.21.15.116:0/1209435168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65dc004d50 msgr2=0x7f65f00c0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:20.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.639+0000 7f65f5b58640 1 --2- 172.21.15.116:0/1209435168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65dc004d50 0x7f65f00c0690 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f65e0011980 tx=0x7f65e0011e40 comp rx=0 tx=0).stop 2024-04-25T00:30:20.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.639+0000 7f65f5b58640 1 -- 172.21.15.116:0/1209435168 shutdown_connections 2024-04-25T00:30:20.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.639+0000 7f65f5b58640 1 --2- 172.21.15.116:0/1209435168 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f65d00678f0 0x7f65d0069db0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:20.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.639+0000 7f65f5b58640 1 --2- 172.21.15.116:0/1209435168 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f65f01eee80 0x7f65f013f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:20.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.639+0000 7f65f5b58640 1 --2- 172.21.15.116:0/1209435168 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f65f00c66f0 0x7f65f01ee940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:20.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.639+0000 7f65f5b58640 1 --2- 172.21.15.116:0/1209435168 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f65dc004d50 0x7f65f00c0690 unknown :-1 s=CLOSED pgs=1722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:20.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.639+0000 7f65f5b58640 1 -- 172.21.15.116:0/1209435168 >> 172.21.15.116:0/1209435168 conn(0x7f65f00c3b00 msgr2=0x7f65f006d880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:20.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.639+0000 7f65f5b58640 1 -- 172.21.15.116:0/1209435168 shutdown_connections 2024-04-25T00:30:20.639 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.639+0000 7f65f5b58640 1 -- 172.21.15.116:0/1209435168 wait complete. 2024-04-25T00:30:20.656 INFO:tasks.mds_thrash.fs.[b]:mds.d has gained rank=0, replacing gid=8772 2024-04-25T00:30:20.656 INFO:tasks.mds_thrash.fs.[b]:waiting for 0 secs before reviving mds.a 2024-04-25T00:30:20.656 INFO:tasks.mds_thrash.fs.[b]:reviving mds.a 2024-04-25T00:30:20.656 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-25T00:30:20.656 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-25T00:30:20.658 INFO:tasks.ceph.mds.a:Started 2024-04-25T00:30:20.659 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:30:20.727 INFO:tasks.ceph.mds.a.smithi116.stdout:starting mds.a at 2024-04-25T00:30:20.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.775+0000 7ffac6e93640 1 Processor -- start 2024-04-25T00:30:20.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.775+0000 7ffac6e93640 1 -- start start 2024-04-25T00:30:20.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.775+0000 7ffac6e93640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffac00aeb20 0x7ffac00c9310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:20.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.775+0000 7ffac6e93640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7ffac00c9850 0x7ffac0150870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:20.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.775+0000 7ffac6e93640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffac01535d0 con 0x7ffac0150db0 2024-04-25T00:30:20.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.775+0000 7ffac6e93640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7ffac00c9cb0 con 0x7ffac00c9850 2024-04-25T00:30:20.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.775+0000 7ffac6e93640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ffac0151220 con 0x7ffac00aeb20 2024-04-25T00:30:20.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.775+0000 7ffac5430640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7ffac0150db0 0x7ffac00c5270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54552/0 (socket says 172.21.15.116:54552) 2024-04-25T00:30:20.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.775+0000 7ffac5430640 1 -- 172.21.15.116:0/1620912785 learned_addr learned my addr 172.21.15.116:0/1620912785 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:30:20.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.775+0000 7ffabffff640 1 --2- 172.21.15.116:0/1620912785 >> v2:172.21.15.174:3300/0 conn(0x7ffac00c9850 0x7ffac0150870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:20.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.775+0000 7ffac4c2f640 1 --2- 172.21.15.116:0/1620912785 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffac00aeb20 0x7ffac00c9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:20.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.775+0000 7ffabf7fe640 1 -- 172.21.15.116:0/1620912785 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3039106541 0 0) 0x7ffab8001120 con 0x7ffac0150db0 2024-04-25T00:30:20.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.775+0000 7ffabf7fe640 1 -- 172.21.15.116:0/1620912785 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffaa8003530 con 0x7ffac0150db0 2024-04-25T00:30:20.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.775+0000 7ffabf7fe640 1 -- 172.21.15.116:0/1620912785 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2430682828 0 0) 0x7ffaa8003530 con 0x7ffac0150db0 2024-04-25T00:30:20.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.775+0000 7ffabf7fe640 1 -- 172.21.15.116:0/1620912785 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffac00aeb20 msgr2=0x7ffac00c9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:20.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.775+0000 7ffabf7fe640 1 --2- 172.21.15.116:0/1620912785 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffac00aeb20 0x7ffac00c9310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:20.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.775+0000 7ffabf7fe640 1 -- 172.21.15.116:0/1620912785 >> v2:172.21.15.174:3300/0 conn(0x7ffac00c9850 msgr2=0x7ffac0150870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:20.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.775+0000 7ffabf7fe640 1 --2- 172.21.15.116:0/1620912785 >> v2:172.21.15.174:3300/0 conn(0x7ffac00c9850 0x7ffac0150870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:20.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.775+0000 7ffabf7fe640 1 -- 172.21.15.116:0/1620912785 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffac0153850 con 0x7ffac0150db0 2024-04-25T00:30:20.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.775+0000 7ffabf7fe640 1 -- 172.21.15.116:0/1620912785 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7ffab8001850 con 0x7ffac0150db0 2024-04-25T00:30:20.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.775+0000 7ffabf7fe640 1 -- 172.21.15.116:0/1620912785 >> v1:172.21.15.116:6789/0 conn(0x7ffac0150db0 legacy=0x7ffac00c5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:20.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.775+0000 7ffabf7fe640 1 --2- 172.21.15.116:0/1620912785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffaa80039c0 0x7ffaa8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:20.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.775+0000 7ffabf7fe640 1 --2- 172.21.15.116:0/1620912785 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffac00aeb20 0x7ffaa8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:20.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.775+0000 7ffabf7fe640 1 --2- 172.21.15.116:0/1620912785 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffaa8004d50 0x7ffaa80051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:20.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.775+0000 7ffabf7fe640 1 -- 172.21.15.116:0/1620912785 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ffac0151220 con 0x7ffaa80039c0 2024-04-25T00:30:20.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.775+0000 7ffabf7fe640 1 -- 172.21.15.116:0/1620912785 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ffaa800b9d0 con 0x7ffaa8004d50 2024-04-25T00:30:20.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.775+0000 7ffabf7fe640 1 -- 172.21.15.116:0/1620912785 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ffaa800bb50 con 0x7ffac00aeb20 2024-04-25T00:30:20.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.775+0000 7ffac4c2f640 1 --2- 172.21.15.116:0/1620912785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffaa80039c0 0x7ffaa8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:20.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.775+0000 7ffac5430640 1 --2- 172.21.15.116:0/1620912785 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffaa8004d50 0x7ffaa80051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:20.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.775+0000 7ffac4c2f640 1 -- 172.21.15.116:0/1620912785 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffac00aeb20 msgr2=0x7ffaa8004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:20.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.775+0000 7ffac4c2f640 1 --2- 172.21.15.116:0/1620912785 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffac00aeb20 0x7ffaa8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:20.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.775+0000 7ffac4c2f640 1 -- 172.21.15.116:0/1620912785 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffaa8004d50 msgr2=0x7ffaa80051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:20.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.775+0000 7ffac4c2f640 1 --2- 172.21.15.116:0/1620912785 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffaa8004d50 0x7ffaa80051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:20.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.775+0000 7ffac4c2f640 1 -- 172.21.15.116:0/1620912785 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffaa8003530 con 0x7ffaa80039c0 2024-04-25T00:30:20.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.775+0000 7ffac5430640 1 --2- 172.21.15.116:0/1620912785 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffaa8004d50 0x7ffaa80051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:30:20.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.779+0000 7ffac4c2f640 1 --2- 172.21.15.116:0/1620912785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffaa80039c0 0x7ffaa8003de0 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7ffab4009ac0 tx=0x7ffab401ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f73be7469eadbd9d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:20.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.779+0000 7ffabf7fe640 1 -- 172.21.15.116:0/1620912785 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab40b2070 con 0x7ffaa80039c0 2024-04-25T00:30:20.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.779+0000 7ffabf7fe640 1 -- 172.21.15.116:0/1620912785 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffab40a5540 con 0x7ffaa80039c0 2024-04-25T00:30:20.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.779+0000 7ffabf7fe640 1 -- 172.21.15.116:0/1620912785 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab4009690 con 0x7ffaa80039c0 2024-04-25T00:30:20.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.779+0000 7ffac6e93640 1 -- 172.21.15.116:0/1620912785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffaa80039c0 msgr2=0x7ffaa8003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:20.782 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.779+0000 7ffac6e93640 1 --2- 172.21.15.116:0/1620912785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffaa80039c0 0x7ffaa8003de0 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7ffab4009ac0 tx=0x7ffab401ca60 comp rx=0 tx=0).stop 2024-04-25T00:30:20.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.779+0000 7ffac5430640 1 -- 172.21.15.116:0/1620912785 reap_dead start 2024-04-25T00:30:20.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.779+0000 7ffac6e93640 1 -- 172.21.15.116:0/1620912785 shutdown_connections 2024-04-25T00:30:20.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.779+0000 7ffac6e93640 1 -- 172.21.15.116:0/1620912785 >> 172.21.15.116:0/1620912785 conn(0x7ffac0074650 msgr2=0x7ffac0074a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:20.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.779+0000 7ffac6e93640 1 -- 172.21.15.116:0/1620912785 shutdown_connections 2024-04-25T00:30:20.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.779+0000 7ffac6e93640 1 -- 172.21.15.116:0/1620912785 wait complete. 2024-04-25T00:30:20.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.779+0000 7ffac6e93640 1 Processor -- start 2024-04-25T00:30:20.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.779+0000 7ffac6e93640 1 -- start start 2024-04-25T00:30:20.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.779+0000 7ffac6e93640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffaa80039c0 0x7ffac015afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:20.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.779+0000 7ffac6e93640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffac01575c0 0x7ffac015b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:20.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.779+0000 7ffac6e93640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffac014b040 0x7ffac014b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:20.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.779+0000 7ffac6e93640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ffac00c5d90 con 0x7ffac01575c0 2024-04-25T00:30:20.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.779+0000 7ffac6e93640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ffac01543d0 con 0x7ffaa80039c0 2024-04-25T00:30:20.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.779+0000 7ffac6e93640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ffac00c5f10 con 0x7ffac014b040 2024-04-25T00:30:20.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.779+0000 7ffabffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffac01575c0 0x7ffac015b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:20.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.779+0000 7ffabffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffac01575c0 0x7ffac015b520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46410/0 (socket says 172.21.15.116:46410) 2024-04-25T00:30:20.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.779+0000 7ffabffff640 1 -- 172.21.15.116:0/3015857968 learned_addr learned my addr 172.21.15.116:0/3015857968 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:20.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.779+0000 7ffac4c2f640 1 --2- 172.21.15.116:0/3015857968 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffaa80039c0 0x7ffac015afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:20.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.779+0000 7ffac5430640 1 --2- 172.21.15.116:0/3015857968 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffac014b040 0x7ffac014b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:20.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.783+0000 7ffabffff640 1 -- 172.21.15.116:0/3015857968 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffac014b040 msgr2=0x7ffac014b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:20.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.783+0000 7ffabffff640 1 --2- 172.21.15.116:0/3015857968 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffac014b040 0x7ffac014b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:20.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.783+0000 7ffabffff640 1 -- 172.21.15.116:0/3015857968 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffaa80039c0 msgr2=0x7ffac015afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:20.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.783+0000 7ffabffff640 1 --2- 172.21.15.116:0/3015857968 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffaa80039c0 0x7ffac015afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:20.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.783+0000 7ffabffff640 1 -- 172.21.15.116:0/3015857968 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffac00a8d00 con 0x7ffac01575c0 2024-04-25T00:30:20.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.783+0000 7ffac4c2f640 1 --2- 172.21.15.116:0/3015857968 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffaa80039c0 0x7ffac015afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:30:20.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.783+0000 7ffabffff640 1 --2- 172.21.15.116:0/3015857968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffac01575c0 0x7ffac015b520 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7ffab0009db0 tx=0x7ffab000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:20.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.783+0000 7ffabd7fa640 1 -- 172.21.15.116:0/3015857968 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab0018020 con 0x7ffac01575c0 2024-04-25T00:30:20.784 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.783+0000 7ffabd7fa640 1 -- 172.21.15.116:0/3015857968 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffab0014480 con 0x7ffac01575c0 2024-04-25T00:30:20.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.783+0000 7ffac6e93640 1 -- 172.21.15.116:0/3015857968 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffac00a8fe0 con 0x7ffac01575c0 2024-04-25T00:30:20.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.783+0000 7ffabd7fa640 1 -- 172.21.15.116:0/3015857968 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab0014870 con 0x7ffac01575c0 2024-04-25T00:30:20.785 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.783+0000 7ffac6e93640 1 -- 172.21.15.116:0/3015857968 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffac0155800 con 0x7ffac01575c0 2024-04-25T00:30:20.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.783+0000 7ffac6e93640 1 -- 172.21.15.116:0/3015857968 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffac0068a60 con 0x7ffac01575c0 2024-04-25T00:30:20.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.783+0000 7ffabd7fa640 1 -- 172.21.15.116:0/3015857968 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ffab001b020 con 0x7ffac01575c0 2024-04-25T00:30:20.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.783+0000 7ffabd7fa640 1 --2- 172.21.15.116:0/3015857968 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ffaac0678c0 0x7ffaac069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:20.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.783+0000 7ffabd7fa640 1 -- 172.21.15.116:0/3015857968 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7ffab0087a50 con 0x7ffac01575c0 2024-04-25T00:30:20.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.783+0000 7ffac4c2f640 1 --2- 172.21.15.116:0/3015857968 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ffaac0678c0 0x7ffaac069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:20.786 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.783+0000 7ffac4c2f640 1 --2- 172.21.15.116:0/3015857968 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ffaac0678c0 0x7ffaac069d80 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7ffab40af000 tx=0x7ffab40046d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:20.788 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.787+0000 7ffabd7fa640 1 -- 172.21.15.116:0/3015857968 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ffab0022070 con 0x7ffac01575c0 2024-04-25T00:30:20.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.975+0000 7ffac6e93640 1 -- 172.21.15.116:0/3015857968 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffac00c6290 con 0x7ffac01575c0 2024-04-25T00:30:20.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.975+0000 7ffabd7fa640 1 -- 172.21.15.116:0/3015857968 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 241 v241) ==== 95+0+5680 (secure 0 0 0) 0x7ffab0010040 con 0x7ffac01575c0 2024-04-25T00:30:20.975 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:30:20.975 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":241,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112}],"filesystems":[{"mdsmap":{"epoch":236,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:29:54.168353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8847},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8847":{"gid":8847,"name":"c","rank":0,"incarnation":233,"state":"up:active","state_seq":17,"addr":"172.21.15.116:6833/2662028892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2662028892},{"type":"v1","addr":"172.21.15.116:6833","nonce":2662028892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8847,"qdb_cluster":[8847]},"id":1},{"mdsmap":{"epoch":241,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:30:19.111393+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6764},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6764":{"gid":6764,"name":"d","rank":0,"incarnation":238,"state":"up:active","state_seq":8,"addr":"172.21.15.174:6837/3873739116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3873739116},{"type":"v1","addr":"172.21.15.174:6837","nonce":3873739116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6764,"qdb_cluster":[6764]},"id":2}]} 2024-04-25T00:30:20.975 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 241 2024-04-25T00:30:20.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.975+0000 7ffac6e93640 1 -- 172.21.15.116:0/3015857968 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ffaac0678c0 msgr2=0x7ffaac069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:20.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.975+0000 7ffac6e93640 1 --2- 172.21.15.116:0/3015857968 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ffaac0678c0 0x7ffaac069d80 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7ffab40af000 tx=0x7ffab40046d0 comp rx=0 tx=0).stop 2024-04-25T00:30:20.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.975+0000 7ffac6e93640 1 -- 172.21.15.116:0/3015857968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffac01575c0 msgr2=0x7ffac015b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:20.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.975+0000 7ffac6e93640 1 --2- 172.21.15.116:0/3015857968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffac01575c0 0x7ffac015b520 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7ffab0009db0 tx=0x7ffab000c490 comp rx=0 tx=0).stop 2024-04-25T00:30:20.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.975+0000 7ffac6e93640 1 -- 172.21.15.116:0/3015857968 shutdown_connections 2024-04-25T00:30:20.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.975+0000 7ffac6e93640 1 --2- 172.21.15.116:0/3015857968 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ffaac0678c0 0x7ffaac069d80 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:20.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.975+0000 7ffac6e93640 1 --2- 172.21.15.116:0/3015857968 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ffac014b040 0x7ffac014b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:20.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.975+0000 7ffac6e93640 1 --2- 172.21.15.116:0/3015857968 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ffac01575c0 0x7ffac015b520 unknown :-1 s=CLOSED pgs=1728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:20.977 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.975+0000 7ffac6e93640 1 --2- 172.21.15.116:0/3015857968 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ffaa80039c0 0x7ffac015afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:20.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.975+0000 7ffac6e93640 1 -- 172.21.15.116:0/3015857968 >> 172.21.15.116:0/3015857968 conn(0x7ffac0074650 msgr2=0x7ffac014f3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:20.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.979+0000 7ffac6e93640 1 -- 172.21.15.116:0/3015857968 shutdown_connections 2024-04-25T00:30:20.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:20.979+0000 7ffac6e93640 1 -- 172.21.15.116:0/3015857968 wait complete. 2024-04-25T00:30:20.992 INFO:tasks.mds_thrash.fs.[b]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2024-04-25T00:30:22.218 INFO:tasks.ceph.mds.d.smithi174.stderr:2024-04-25T00:30:22.216+0000 7f236179d640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T00:30:22.453 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T00:30:22.454 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1/tmp 2024-04-25T00:30:22.479 INFO:tasks.workunit:Stopping ['suites/ffsb.sh'] on client.1... 2024-04-25T00:30:22.479 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.1 /home/ubuntu/cephtest/clone.client.1 2024-04-25T00:30:22.803 DEBUG:teuthology.parallel:result is None 2024-04-25T00:30:22.804 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-04-25T00:30:22.825 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-04-25T00:30:22.826 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1 2024-04-25T00:30:22.878 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.1/client.1 2024-04-25T00:30:22.879 DEBUG:teuthology.run_tasks:Unwinding manager mds_thrash 2024-04-25T00:30:22.889 INFO:tasks.mds_thrash:joining mds_thrasher 2024-04-25T00:30:22.993 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:30:23.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.263+0000 7faeb690a640 1 Processor -- start 2024-04-25T00:30:23.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.263+0000 7faeb690a640 1 -- start start 2024-04-25T00:30:23.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.263+0000 7faeb690a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faeb00c78d0 0x7faeb0152200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:23.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.263+0000 7faeb690a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7faeb006dc00 0x7faeb0071c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:23.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.263+0000 7faeb690a640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faeb0151740 con 0x7faeb013a970 2024-04-25T00:30:23.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.263+0000 7faeb690a640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7faeb0152740 con 0x7faeb006dc00 2024-04-25T00:30:23.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.263+0000 7faeb690a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7faeb00c2630 con 0x7faeb00c78d0 2024-04-25T00:30:23.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.267+0000 7faeb6109640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7faeb006dc00 0x7faeb0071c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:23.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.267+0000 7faeb5908640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faeb00c78d0 0x7faeb0152200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:23.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.267+0000 7faeb5107640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7faeb013a970 0x7faeb01556f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:39498/0 (socket says 172.21.15.116:39498) 2024-04-25T00:30:23.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.267+0000 7faeb5908640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faeb00c78d0 0x7faeb0152200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:35952/0 (socket says 172.21.15.116:35952) 2024-04-25T00:30:23.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.267+0000 7faeb5107640 1 -- 172.21.15.116:0/618396928 learned_addr learned my addr 172.21.15.116:0/618396928 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:30:23.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.267+0000 7faeb5908640 1 -- 172.21.15.116:0/618396928 >> v2:172.21.15.116:3300/0 conn(0x7faeb006dc00 msgr2=0x7faeb0071c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:23.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.267+0000 7faeb5908640 1 --2- 172.21.15.116:0/618396928 >> v2:172.21.15.116:3300/0 conn(0x7faeb006dc00 0x7faeb0071c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:23.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.267+0000 7faeb5908640 1 -- 172.21.15.116:0/618396928 >> v1:172.21.15.174:6789/0 conn(0x7faeb013a970 legacy=0x7faeb01556f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:23.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.267+0000 7faeb4906640 1 -- 172.21.15.116:0/618396928 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1386698228 0 0) 0x7faeb0151740 con 0x7faeb013a970 2024-04-25T00:30:23.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.267+0000 7faeb5908640 1 -- 172.21.15.116:0/618396928 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faeb00c2910 con 0x7faeb00c78d0 2024-04-25T00:30:23.273 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.271+0000 7faeb5908640 1 --2- 172.21.15.116:0/618396928 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faeb00c78d0 0x7faeb0152200 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7faeac0097c0 tx=0x7faeac01c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40efeee18c99bae4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:23.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.275+0000 7faeb4906640 1 -- 172.21.15.116:0/618396928 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeac0b0070 con 0x7faeb00c78d0 2024-04-25T00:30:23.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.279+0000 7faeb4906640 1 -- 172.21.15.116:0/618396928 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faeac0ab600 con 0x7faeb00c78d0 2024-04-25T00:30:23.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.279+0000 7faeb4906640 1 -- 172.21.15.116:0/618396928 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeac0b0070 con 0x7faeb00c78d0 2024-04-25T00:30:23.280 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.279+0000 7faeb690a640 1 -- 172.21.15.116:0/618396928 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faeb00c78d0 msgr2=0x7faeb0152200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:23.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.279+0000 7faeb690a640 1 --2- 172.21.15.116:0/618396928 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faeb00c78d0 0x7faeb0152200 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7faeac0097c0 tx=0x7faeac01c4a0 comp rx=0 tx=0).stop 2024-04-25T00:30:23.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.283+0000 7faeb690a640 1 -- 172.21.15.116:0/618396928 shutdown_connections 2024-04-25T00:30:23.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.283+0000 7faeb690a640 1 --2- 172.21.15.116:0/618396928 >> v2:172.21.15.116:3300/0 conn(0x7faeb006dc00 0x7faeb0071c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:23.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.283+0000 7faeb690a640 1 --2- 172.21.15.116:0/618396928 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faeb00c78d0 0x7faeb0152200 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:23.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.283+0000 7faeb690a640 1 -- 172.21.15.116:0/618396928 >> 172.21.15.116:0/618396928 conn(0x7faeb00c6db0 msgr2=0x7faeb00c71c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:23.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.283+0000 7faeb690a640 1 -- 172.21.15.116:0/618396928 shutdown_connections 2024-04-25T00:30:23.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.283+0000 7faeb690a640 1 -- 172.21.15.116:0/618396928 wait complete. 2024-04-25T00:30:23.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.283+0000 7faeb690a640 1 Processor -- start 2024-04-25T00:30:23.285 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.283+0000 7faeb690a640 1 -- start start 2024-04-25T00:30:23.286 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.287+0000 7faeb690a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faeb006dc00 0x7faeb01ee970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:23.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.287+0000 7faeb690a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faeb00c78d0 0x7faeb014f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:23.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.287+0000 7faeb690a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faeb013a970 0x7faeb0130580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:23.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.287+0000 7faeb690a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7faeb013af10 con 0x7faeb00c78d0 2024-04-25T00:30:23.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.287+0000 7faeb690a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7faeb013ad90 con 0x7faeb013a970 2024-04-25T00:30:23.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.287+0000 7faeb690a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7faeb006e010 con 0x7faeb006dc00 2024-04-25T00:30:23.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.287+0000 7faeb5908640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faeb006dc00 0x7faeb01ee970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:23.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.287+0000 7faeb6109640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faeb013a970 0x7faeb0130580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:23.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.287+0000 7faeb5908640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faeb006dc00 0x7faeb01ee970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:35958/0 (socket says 172.21.15.116:35958) 2024-04-25T00:30:23.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.287+0000 7faeb6109640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faeb013a970 0x7faeb0130580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:57270/0 (socket says 172.21.15.116:57270) 2024-04-25T00:30:23.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.287+0000 7faeb5908640 1 -- 172.21.15.116:0/580364859 learned_addr learned my addr 172.21.15.116:0/580364859 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:23.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.287+0000 7faeb6109640 1 -- 172.21.15.116:0/580364859 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faeb006dc00 msgr2=0x7faeb01ee970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:23.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.287+0000 7faeb6109640 1 --2- 172.21.15.116:0/580364859 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faeb006dc00 0x7faeb01ee970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:23.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.287+0000 7faeb6109640 1 -- 172.21.15.116:0/580364859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faeb00c78d0 msgr2=0x7faeb014f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:23.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.287+0000 7faeb6109640 1 --2- 172.21.15.116:0/580364859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faeb00c78d0 0x7faeb014f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:23.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.287+0000 7faeb6109640 1 -- 172.21.15.116:0/580364859 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faeb0131200 con 0x7faeb013a970 2024-04-25T00:30:23.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.287+0000 7faeb5908640 1 --2- 172.21.15.116:0/580364859 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faeb006dc00 0x7faeb01ee970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:30:23.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.287+0000 7faeb6109640 1 --2- 172.21.15.116:0/580364859 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faeb013a970 0x7faeb0130580 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7faea000f780 tx=0x7faea000fc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:23.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.291+0000 7faea67fc640 1 -- 172.21.15.116:0/580364859 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faea000db50 con 0x7faeb013a970 2024-04-25T00:30:23.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.291+0000 7faeb690a640 1 -- 172.21.15.116:0/580364859 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faeb00c6a60 con 0x7faeb013a970 2024-04-25T00:30:23.289 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.291+0000 7faeb690a640 1 -- 172.21.15.116:0/580364859 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faeb01eddc0 con 0x7faeb013a970 2024-04-25T00:30:23.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.291+0000 7faea67fc640 1 -- 172.21.15.116:0/580364859 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faea000fe90 con 0x7faeb013a970 2024-04-25T00:30:23.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.291+0000 7faeb690a640 1 -- 172.21.15.116:0/580364859 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faeb004f510 con 0x7faeb013a970 2024-04-25T00:30:23.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.291+0000 7faea67fc640 1 -- 172.21.15.116:0/580364859 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faea001a480 con 0x7faeb013a970 2024-04-25T00:30:23.292 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.291+0000 7faea67fc640 1 -- 172.21.15.116:0/580364859 <== mon.1 v2:172.21.15.174:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7faea00389d0 con 0x7faeb013a970 2024-04-25T00:30:23.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.295+0000 7faea67fc640 1 --2- 172.21.15.116:0/580364859 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fae94067910 0x7fae94069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:23.293 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.295+0000 7faea67fc640 1 -- 172.21.15.116:0/580364859 <== mon.1 v2:172.21.15.174:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7faea0087540 con 0x7faeb013a970 2024-04-25T00:30:23.294 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.295+0000 7faeb5908640 1 --2- 172.21.15.116:0/580364859 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fae94067910 0x7fae94069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:23.298 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.299+0000 7faeb5908640 1 --2- 172.21.15.116:0/580364859 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fae94067910 0x7fae94069dd0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7faeac01a0d0 tx=0x7faeac0a6000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:23.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.303+0000 7faea67fc640 1 -- 172.21.15.116:0/580364859 <== mon.1 v2:172.21.15.174:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7faea0056ed0 con 0x7faeb013a970 2024-04-25T00:30:23.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.679+0000 7faeb690a640 1 -- 172.21.15.116:0/580364859 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faeb01eeeb0 con 0x7faeb013a970 2024-04-25T00:30:23.681 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.679+0000 7faea67fc640 1 -- 172.21.15.116:0/580364859 <== mon.1 v2:172.21.15.174:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 242 v242) ==== 95+0+6506 (secure 0 0 0) 0x7faea00567d0 con 0x7faeb013a970 2024-04-25T00:30:23.682 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:30:23.682 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":242,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":9027,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/426062493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":426062493},{"type":"v1","addr":"172.21.15.116:6835","nonce":426062493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":242}],"filesystems":[{"mdsmap":{"epoch":236,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:29:54.168353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8847},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8847":{"gid":8847,"name":"c","rank":0,"incarnation":233,"state":"up:active","state_seq":17,"addr":"172.21.15.116:6833/2662028892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2662028892},{"type":"v1","addr":"172.21.15.116:6833","nonce":2662028892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8847,"qdb_cluster":[8847]},"id":1},{"mdsmap":{"epoch":241,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:30:19.111393+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6764},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6764":{"gid":6764,"name":"d","rank":0,"incarnation":238,"state":"up:active","state_seq":8,"addr":"172.21.15.174:6837/3873739116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3873739116},{"type":"v1","addr":"172.21.15.174:6837","nonce":3873739116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6764,"qdb_cluster":[6764]},"id":2}]} 2024-04-25T00:30:23.682 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 242 2024-04-25T00:30:23.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.691+0000 7faeb690a640 1 -- 172.21.15.116:0/580364859 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fae94067910 msgr2=0x7fae94069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:23.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.691+0000 7faeb690a640 1 --2- 172.21.15.116:0/580364859 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fae94067910 0x7fae94069dd0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7faeac01a0d0 tx=0x7faeac0a6000 comp rx=0 tx=0).stop 2024-04-25T00:30:23.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.691+0000 7faeb690a640 1 -- 172.21.15.116:0/580364859 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faeb013a970 msgr2=0x7faeb0130580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:23.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.691+0000 7faeb690a640 1 --2- 172.21.15.116:0/580364859 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faeb013a970 0x7faeb0130580 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7faea000f780 tx=0x7faea000fc40 comp rx=0 tx=0).stop 2024-04-25T00:30:23.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.691+0000 7faeb690a640 1 -- 172.21.15.116:0/580364859 shutdown_connections 2024-04-25T00:30:23.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.691+0000 7faeb690a640 1 --2- 172.21.15.116:0/580364859 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fae94067910 0x7fae94069dd0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:23.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.691+0000 7faeb690a640 1 --2- 172.21.15.116:0/580364859 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faeb013a970 0x7faeb0130580 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:23.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.691+0000 7faeb690a640 1 --2- 172.21.15.116:0/580364859 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faeb00c78d0 0x7faeb014f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:23.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.691+0000 7faeb690a640 1 --2- 172.21.15.116:0/580364859 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faeb006dc00 0x7faeb01ee970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:23.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.691+0000 7faeb690a640 1 -- 172.21.15.116:0/580364859 >> 172.21.15.116:0/580364859 conn(0x7faeb00c6db0 msgr2=0x7faeb00c3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:23.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.691+0000 7faeb690a640 1 -- 172.21.15.116:0/580364859 shutdown_connections 2024-04-25T00:30:23.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:23.691+0000 7faeb690a640 1 -- 172.21.15.116:0/580364859 wait complete. 2024-04-25T00:30:23.722 INFO:tasks.mds_thrash.fs.[b]:mds.a reported in up:standby state 2024-04-25T00:30:23.723 INFO:tasks.mds_thrash.fs.[b]:stat['max_mds'] = 0 2024-04-25T00:30:23.723 INFO:tasks.mds_thrash.fs.[b]:stat['deactivate'] = 0 2024-04-25T00:30:23.723 INFO:tasks.mds_thrash.fs.[b]:stat['kill'] = 23 2024-04-25T00:30:23.723 INFO:tasks.mds_thrash:done joining 2024-04-25T00:30:23.723 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-04-25T00:30:23.732 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-04-25T00:30:23.732 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-04-25T00:30:23.733 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T00:30:23.754 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-04-25T00:30:23.755 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T00:30:23.810 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T00:30:24.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.035+0000 7f0a37b7c640 1 Processor -- start 2024-04-25T00:30:24.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.035+0000 7f0a37b7c640 1 -- start start 2024-04-25T00:30:24.035 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.035+0000 7f0a37b7c640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f0a30150d20 0x7f0a30153620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:24.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.035+0000 7f0a37b7c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0a300c3240 0x7f0a300c3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:24.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.035+0000 7f0a37b7c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0a300c4380 0x7f0a30071050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:24.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.035+0000 7f0a37b7c640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f0a3006fba0 con 0x7f0a300c4380 2024-04-25T00:30:24.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.035+0000 7f0a37b7c640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f0a300c36a0 con 0x7f0a30150d20 2024-04-25T00:30:24.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.035+0000 7f0a37b7c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f0a3006f8c0 con 0x7f0a300c3240 2024-04-25T00:30:24.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.035+0000 7f0a36119640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0a300c4380 0x7f0a30071050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:24.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.035+0000 7f0a36119640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0a300c4380 0x7f0a30071050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46444/0 (socket says 172.21.15.116:46444) 2024-04-25T00:30:24.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.035+0000 7f0a36119640 1 -- 172.21.15.116:0/621847867 learned_addr learned my addr 172.21.15.116:0/621847867 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:24.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.035+0000 7f0a35117640 1 --2- 172.21.15.116:0/621847867 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0a300c3240 0x7f0a300c3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:24.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.035+0000 7f0a35918640 1 --2- 172.21.15.116:0/621847867 >> v2:172.21.15.174:3300/0 conn(0x7f0a30150d20 0x7f0a30153620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:24.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.035+0000 7f0a36119640 1 -- 172.21.15.116:0/621847867 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0a300c3240 msgr2=0x7f0a300c3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:24.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.035+0000 7f0a36119640 1 --2- 172.21.15.116:0/621847867 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0a300c3240 0x7f0a300c3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:24.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.035+0000 7f0a36119640 1 -- 172.21.15.116:0/621847867 >> v2:172.21.15.174:3300/0 conn(0x7f0a30150d20 msgr2=0x7f0a30153620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:24.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.035+0000 7f0a36119640 1 --2- 172.21.15.116:0/621847867 >> v2:172.21.15.174:3300/0 conn(0x7f0a30150d20 0x7f0a30153620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:24.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.035+0000 7f0a36119640 1 -- 172.21.15.116:0/621847867 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a300c5420 con 0x7f0a300c4380 2024-04-25T00:30:24.047 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.047+0000 7f0a36119640 1 --2- 172.21.15.116:0/621847867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a300c4380 0x7f0a30071050 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f0a300c6b50 tx=0x7f0a2801c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d41b107529420ae0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:24.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.051+0000 7f0a34916640 1 -- 172.21.15.116:0/621847867 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a280a5780 con 0x7f0a300c4380 2024-04-25T00:30:24.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.051+0000 7f0a34916640 1 -- 172.21.15.116:0/621847867 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a280a5920 con 0x7f0a300c4380 2024-04-25T00:30:24.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.051+0000 7f0a34916640 1 -- 172.21.15.116:0/621847867 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a280a5cf0 con 0x7f0a300c4380 2024-04-25T00:30:24.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.051+0000 7f0a37b7c640 1 -- 172.21.15.116:0/621847867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a300c4380 msgr2=0x7f0a30071050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:24.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.051+0000 7f0a37b7c640 1 --2- 172.21.15.116:0/621847867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a300c4380 0x7f0a30071050 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f0a300c6b50 tx=0x7f0a2801c9f0 comp rx=0 tx=0).stop 2024-04-25T00:30:24.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.051+0000 7f0a37b7c640 1 -- 172.21.15.116:0/621847867 shutdown_connections 2024-04-25T00:30:24.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.051+0000 7f0a37b7c640 1 --2- 172.21.15.116:0/621847867 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a300c4380 0x7f0a30071050 unknown :-1 s=CLOSED pgs=1729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:24.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.051+0000 7f0a37b7c640 1 --2- 172.21.15.116:0/621847867 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0a300c3240 0x7f0a300c3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:24.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.051+0000 7f0a37b7c640 1 --2- 172.21.15.116:0/621847867 >> v2:172.21.15.174:3300/0 conn(0x7f0a30150d20 0x7f0a30153620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:24.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.051+0000 7f0a37b7c640 1 -- 172.21.15.116:0/621847867 >> 172.21.15.116:0/621847867 conn(0x7f0a300c9160 msgr2=0x7f0a300c9570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:24.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.051+0000 7f0a37b7c640 1 -- 172.21.15.116:0/621847867 shutdown_connections 2024-04-25T00:30:24.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.051+0000 7f0a37b7c640 1 -- 172.21.15.116:0/621847867 wait complete. 2024-04-25T00:30:24.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.051+0000 7f0a37b7c640 1 Processor -- start 2024-04-25T00:30:24.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.051+0000 7f0a37b7c640 1 -- start start 2024-04-25T00:30:24.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.055+0000 7f0a37b7c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0a300c3240 0x7f0a30141630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:24.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.055+0000 7f0a37b7c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a300c4380 0x7f0a3014a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:24.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.055+0000 7f0a37b7c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0a30150d20 0x7f0a3013dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:24.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.055+0000 7f0a37b7c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0a30151280 con 0x7f0a300c4380 2024-04-25T00:30:24.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.055+0000 7f0a37b7c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0a30151100 con 0x7f0a30150d20 2024-04-25T00:30:24.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.055+0000 7f0a37b7c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f0a300c76a0 con 0x7f0a300c3240 2024-04-25T00:30:24.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.055+0000 7f0a36119640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0a30150d20 0x7f0a3013dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:24.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.055+0000 7f0a36119640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0a30150d20 0x7f0a3013dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:57284/0 (socket says 172.21.15.116:57284) 2024-04-25T00:30:24.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.055+0000 7f0a36119640 1 -- 172.21.15.116:0/1265234292 learned_addr learned my addr 172.21.15.116:0/1265234292 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:24.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.055+0000 7f0a35117640 1 --2- 172.21.15.116:0/1265234292 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a300c4380 0x7f0a3014a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:24.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.055+0000 7f0a35117640 1 -- 172.21.15.116:0/1265234292 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0a300c3240 msgr2=0x7f0a30141630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:24.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.055+0000 7f0a35117640 1 --2- 172.21.15.116:0/1265234292 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0a300c3240 0x7f0a30141630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:24.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.055+0000 7f0a35117640 1 -- 172.21.15.116:0/1265234292 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0a30150d20 msgr2=0x7f0a3013dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:24.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.055+0000 7f0a35117640 1 --2- 172.21.15.116:0/1265234292 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0a30150d20 0x7f0a3013dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:24.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.055+0000 7f0a35117640 1 -- 172.21.15.116:0/1265234292 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a3006de70 con 0x7f0a300c4380 2024-04-25T00:30:24.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.055+0000 7f0a35117640 1 --2- 172.21.15.116:0/1265234292 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a300c4380 0x7f0a3014a150 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7f0a2000e8a0 tx=0x7f0a2000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:24.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.055+0000 7f0a267fc640 1 -- 172.21.15.116:0/1265234292 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a20017070 con 0x7f0a300c4380 2024-04-25T00:30:24.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.055+0000 7f0a267fc640 1 -- 172.21.15.116:0/1265234292 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a2000c940 con 0x7f0a300c4380 2024-04-25T00:30:24.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.055+0000 7f0a267fc640 1 -- 172.21.15.116:0/1265234292 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a2000ccf0 con 0x7f0a300c4380 2024-04-25T00:30:24.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.055+0000 7f0a37b7c640 1 -- 172.21.15.116:0/1265234292 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a3006e150 con 0x7f0a300c4380 2024-04-25T00:30:24.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.055+0000 7f0a37b7c640 1 -- 172.21.15.116:0/1265234292 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a3008b280 con 0x7f0a300c4380 2024-04-25T00:30:24.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.059+0000 7f0a267fc640 1 -- 172.21.15.116:0/1265234292 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f0a2001a070 con 0x7f0a300c4380 2024-04-25T00:30:24.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.059+0000 7f0a267fc640 1 --2- 172.21.15.116:0/1265234292 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0a14067940 0x7f0a14069e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:24.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.059+0000 7f0a267fc640 1 -- 172.21.15.116:0/1265234292 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7f0a200871c0 con 0x7f0a300c4380 2024-04-25T00:30:24.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.059+0000 7f0a35918640 1 --2- 172.21.15.116:0/1265234292 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0a14067940 0x7f0a14069e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:24.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.067+0000 7f0a37b7c640 1 -- 172.21.15.116:0/1265234292 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a180052a0 con 0x7f0a300c4380 2024-04-25T00:30:24.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.075+0000 7f0a267fc640 1 -- 172.21.15.116:0/1265234292 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f0a20056a90 con 0x7f0a300c4380 2024-04-25T00:30:24.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.075+0000 7f0a35918640 1 --2- 172.21.15.116:0/1265234292 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0a14067940 0x7f0a14069e00 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c0099c0 tx=0x7f0a2c009480 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:24.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.403+0000 7f0a37b7c640 1 -- 172.21.15.116:0/1265234292 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0a18005860 con 0x7f0a300c4380 2024-04-25T00:30:24.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.407+0000 7f0a267fc640 1 -- 172.21.15.116:0/1265234292 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v74) ==== 74+0+26655 (secure 0 0 0) 0x7f0a20056390 con 0x7f0a300c4380 2024-04-25T00:30:24.406 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:30:24.407 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":74,"fsid":"6d6a1aa7-0478-4c83-8057-05917e81f046","created":"2024-04-25T00:00:40.682493+0000","modified":"2024-04-25T00:30:15.781714+0000","last_up_change":"2024-04-25T00:00:51.822174+0000","last_in_change":"2024-04-25T00:00:42.980473+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T00:00:53.324376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-04-25T00:01:03.197326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-04-25T00:01:03.728618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-04-25T00:01:17.125210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-04-25T00:01:17.821873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8b6ddcb2-1ad9-4167-961c-bd6936bb9060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6809","nonce":3751765682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6811","nonce":3751765682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6815","nonce":3751765682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6813","nonce":3751765682}]},"public_addr":"172.21.15.116:6809/3751765682","cluster_addr":"172.21.15.116:6811/3751765682","heartbeat_back_addr":"172.21.15.116:6815/3751765682","heartbeat_front_addr":"172.21.15.116:6813/3751765682","state":["exists","up"]},{"osd":1,"uuid":"a6431768-994f-41da-8853-886f5687fb91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6801","nonce":377879124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6803","nonce":377879124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6807","nonce":377879124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6805","nonce":377879124}]},"public_addr":"172.21.15.116:6801/377879124","cluster_addr":"172.21.15.116:6803/377879124","heartbeat_back_addr":"172.21.15.116:6807/377879124","heartbeat_front_addr":"172.21.15.116:6805/377879124","state":["exists","up"]},{"osd":2,"uuid":"4f69bd40-5a23-45ff-8693-e1de75e1aa9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6817","nonce":1099282267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6819","nonce":1099282267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6823","nonce":1099282267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6821","nonce":1099282267}]},"public_addr":"172.21.15.116:6817/1099282267","cluster_addr":"172.21.15.116:6819/1099282267","heartbeat_back_addr":"172.21.15.116:6823/1099282267","heartbeat_front_addr":"172.21.15.116:6821/1099282267","state":["exists","up"]},{"osd":3,"uuid":"6a0d5344-8b6f-40d3-918d-0f0a4be688e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6825","nonce":1179942661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6827","nonce":1179942661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6831","nonce":1179942661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6829","nonce":1179942661}]},"public_addr":"172.21.15.116:6825/1179942661","cluster_addr":"172.21.15.116:6827/1179942661","heartbeat_back_addr":"172.21.15.116:6831/1179942661","heartbeat_front_addr":"172.21.15.116:6829/1179942661","state":["exists","up"]},{"osd":4,"uuid":"132e92e4-b825-441e-965a-ca00533fa86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6801","nonce":3132444330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6803","nonce":3132444330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6807","nonce":3132444330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6805","nonce":3132444330}]},"public_addr":"172.21.15.174:6801/3132444330","cluster_addr":"172.21.15.174:6803/3132444330","heartbeat_back_addr":"172.21.15.174:6807/3132444330","heartbeat_front_addr":"172.21.15.174:6805/3132444330","state":["exists","up"]},{"osd":5,"uuid":"a2eeb5b6-1b31-43cc-8306-a0adf3ee15bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6809","nonce":881817164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6811","nonce":881817164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6815","nonce":881817164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6813","nonce":881817164}]},"public_addr":"172.21.15.174:6809/881817164","cluster_addr":"172.21.15.174:6811/881817164","heartbeat_back_addr":"172.21.15.174:6815/881817164","heartbeat_front_addr":"172.21.15.174:6813/881817164","state":["exists","up"]},{"osd":6,"uuid":"588e873a-c841-4ba7-90f2-8956c87a9983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6825","nonce":117574251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6827","nonce":117574251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6831","nonce":117574251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6829","nonce":117574251}]},"public_addr":"172.21.15.174:6825/117574251","cluster_addr":"172.21.15.174:6827/117574251","heartbeat_back_addr":"172.21.15.174:6831/117574251","heartbeat_front_addr":"172.21.15.174:6829/117574251","state":["exists","up"]},{"osd":7,"uuid":"3b90f9b0-8a0e-4883-b088-ca1d4012870c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6817","nonce":2543899196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6819","nonce":2543899196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6823","nonce":2543899196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6821","nonce":2543899196}]},"public_addr":"172.21.15.174:6817/2543899196","cluster_addr":"172.21.15.174:6819/2543899196","heartbeat_back_addr":"172.21.15.174:6823/2543899196","heartbeat_front_addr":"172.21.15.174:6821/2543899196","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.293769+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.310034+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.404865+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.549658+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.192174+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.297266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.493649+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.452309+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2c","mappings":[{"from":0,"to":7}]},{"pgid":"5.3","mappings":[{"from":4,"to":5}]},{"pgid":"5.5","mappings":[{"from":4,"to":5}]},{"pgid":"5.14","mappings":[{"from":4,"to":5}]},{"pgid":"5.19","mappings":[{"from":4,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:6835/3499878473":"2024-04-26T00:30:15.781624+0000","172.21.15.116:6834/3499878473":"2024-04-26T00:30:15.781624+0000","172.21.15.174:6836/2915274073":"2024-04-26T00:29:50.776462+0000","172.21.15.116:6833/3891103804":"2024-04-26T00:28:50.767503+0000","172.21.15.116:6834/2371126056":"2024-04-26T00:27:35.754759+0000","172.21.15.174:6837/349772087":"2024-04-26T00:27:15.748274+0000","172.21.15.174:6836/349772087":"2024-04-26T00:27:15.748274+0000","172.21.15.116:6833/3693049068":"2024-04-26T00:26:45.739776+0000","172.21.15.116:6832/3693049068":"2024-04-26T00:26:45.739776+0000","172.21.15.174:6837/4181466530":"2024-04-26T00:26:15.734586+0000","172.21.15.174:6836/4181466530":"2024-04-26T00:26:15.734586+0000","172.21.15.174:6836/492366509":"2024-04-26T00:24:15.701887+0000","172.21.15.116:6832/2998042514":"2024-04-26T00:23:30.691827+0000","172.21.15.116:6833/663910035":"2024-04-26T00:22:35.677323+0000","172.21.15.116:6832/663910035":"2024-04-26T00:22:35.677323+0000","172.21.15.174:6837/321466844":"2024-04-26T00:22:10.671673+0000","172.21.15.116:6835/980221971":"2024-04-26T00:21:40.665115+0000","172.21.15.174:6837/991183935":"2024-04-26T00:20:50.654151+0000","172.21.15.174:6836/991183935":"2024-04-26T00:20:50.654151+0000","172.21.15.174:6836/321466844":"2024-04-26T00:22:10.671673+0000","172.21.15.116:6834/1358527848":"2024-04-26T00:11:00.511032+0000","172.21.15.116:6832/3841609112":"2024-04-26T00:07:00.448498+0000","172.21.15.116:6835/1670061091":"2024-04-26T00:09:30.490485+0000","172.21.15.116:6834/1971675962":"2024-04-26T00:25:25.721965+0000","172.21.15.116:6835/1233829678":"2024-04-26T00:07:30.454393+0000","172.21.15.116:6835/2371126056":"2024-04-26T00:27:35.754759+0000","172.21.15.174:6834/2883220846":"2024-04-26T00:11:50.521053+0000","172.21.15.116:6834/1233829678":"2024-04-26T00:07:30.454393+0000","172.21.15.174:6837/3180024481":"2024-04-26T00:15:25.567768+0000","172.21.15.116:6835/879531976":"2024-04-26T00:20:10.642563+0000","172.21.15.116:6833/174781672":"2024-04-26T00:02:40.399073+0000","172.21.15.174:6835/4215220222":"2024-04-26T00:15:40.572498+0000","172.21.15.116:6834/221971311":"2024-04-26T00:22:48.898075+0000","172.21.15.174:6836/1281466397":"2024-04-26T00:18:20.613347+0000","172.21.15.116:6832/3891103804":"2024-04-26T00:28:50.767503+0000","172.21.15.116:6835/1358527848":"2024-04-26T00:11:00.511032+0000","172.21.15.116:6835/221971311":"2024-04-26T00:22:48.898075+0000","172.21.15.116:6832/3006671671":"2024-04-26T00:09:30.490406+0000","172.21.15.116:6834/980221971":"2024-04-26T00:21:40.665115+0000","172.21.15.174:6836/1762356055":"2024-04-26T00:03:36.597345+0000","172.21.15.116:6833/3006671671":"2024-04-26T00:09:30.490406+0000","172.21.15.116:6834/1459177921":"2024-04-26T00:03:30.406778+0000","172.21.15.116:6834/462190523":"2024-04-26T00:14:55.560782+0000","172.21.15.116:6834/879531976":"2024-04-26T00:20:10.642563+0000","172.21.15.116:6832/1459177921":"2024-04-26T00:03:30.406778+0000","172.21.15.174:6837/1159002662":"2024-04-26T00:10:20.502218+0000","172.21.15.174:6837/1762356055":"2024-04-26T00:03:36.597345+0000","172.21.15.174:6834/4215220222":"2024-04-26T00:15:40.572498+0000","172.21.15.116:6835/1971675962":"2024-04-26T00:25:25.721965+0000","172.21.15.174:6836/3081407040":"2024-04-26T00:17:25.598516+0000","172.21.15.116:6834/1599708526":"2024-04-26T00:12:20.526742+0000","172.21.15.174:6836/2276529656":"2024-04-26T00:19:05.629048+0000","172.21.15.174:6836/1159002662":"2024-04-26T00:10:20.502218+0000","172.21.15.116:6833/2998042514":"2024-04-26T00:23:30.691827+0000","172.21.15.174:6837/3081407040":"2024-04-26T00:17:25.598516+0000","172.21.15.174:6835/2883220846":"2024-04-26T00:11:50.521053+0000","172.21.15.174:6837/492366509":"2024-04-26T00:24:15.701887+0000","172.21.15.116:6835/2004876235":"2024-04-26T00:04:45.421611+0000","172.21.15.116:6835/174781672":"2024-04-26T00:02:40.399073+0000","172.21.15.116:6833/3633657268":"2024-04-26T00:25:40.724634+0000","172.21.15.116:6832/365773583":"2024-04-26T00:14:58.857970+0000","172.21.15.174:6836/1889314958":"2024-04-26T00:06:30.441408+0000","172.21.15.116:6833/3705654249":"2024-04-26T00:12:45.533219+0000","172.21.15.116:6832/3705654249":"2024-04-26T00:12:45.533219+0000","172.21.15.116:6834/1670061091":"2024-04-26T00:09:30.490485+0000","172.21.15.116:6835/1599708526":"2024-04-26T00:12:20.526742+0000","172.21.15.116:6832/3633657268":"2024-04-26T00:25:40.724634+0000","172.21.15.174:6836/3583791822":"2024-04-26T00:08:00.460587+0000","172.21.15.174:6837/3583791822":"2024-04-26T00:08:00.460587+0000","172.21.15.174:6836/3398690896":"2024-04-26T00:13:30.543729+0000","172.21.15.174:6837/3398690896":"2024-04-26T00:13:30.543729+0000","172.21.15.116:6835/462190523":"2024-04-26T00:14:55.560782+0000","172.21.15.116:6834/3336339376":"2024-04-26T00:18:35.616887+0000","172.21.15.174:6836/3180024481":"2024-04-26T00:15:25.567768+0000","172.21.15.116:6833/365773583":"2024-04-26T00:14:58.857970+0000","172.21.15.116:6835/2012470875":"2024-04-26T00:16:35.584511+0000","172.21.15.116:6833/2004876235":"2024-04-26T00:04:45.421611+0000","172.21.15.174:6837/1281466397":"2024-04-26T00:18:20.613347+0000","172.21.15.116:6832/2340581953":"2024-04-26T00:16:45.587651+0000","172.21.15.174:6837/2915274073":"2024-04-26T00:29:50.776462+0000","172.21.15.116:6834/1649769369":"2024-04-26T00:04:35.418892+0000","172.21.15.116:6832/1649769369":"2024-04-26T00:04:35.418892+0000","172.21.15.116:6833/2340581953":"2024-04-26T00:16:45.587651+0000","172.21.15.174:6837/1889314958":"2024-04-26T00:06:30.441408+0000","172.21.15.116:6832/3645553116":"2024-04-26T00:17:35.601559+0000","172.21.15.116:6833/3645553116":"2024-04-26T00:17:35.601559+0000","172.21.15.116:6833/3841609112":"2024-04-26T00:07:00.448498+0000","172.21.15.116:6834/2012470875":"2024-04-26T00:16:35.584511+0000","172.21.15.116:6835/3336339376":"2024-04-26T00:18:35.616887+0000","172.21.15.174:6837/2276529656":"2024-04-26T00:19:05.629048+0000","172.21.15.116:6832/1655959433":"2024-04-26T00:19:30.633957+0000","172.21.15.116:6833/1655959433":"2024-04-26T00:19:30.633957+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:30:24.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.411+0000 7f0a37b7c640 1 -- 172.21.15.116:0/1265234292 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0a14067940 msgr2=0x7f0a14069e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:24.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.411+0000 7f0a37b7c640 1 --2- 172.21.15.116:0/1265234292 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0a14067940 0x7f0a14069e00 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c0099c0 tx=0x7f0a2c009480 comp rx=0 tx=0).stop 2024-04-25T00:30:24.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.411+0000 7f0a37b7c640 1 -- 172.21.15.116:0/1265234292 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a300c4380 msgr2=0x7f0a3014a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:24.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.411+0000 7f0a37b7c640 1 --2- 172.21.15.116:0/1265234292 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a300c4380 0x7f0a3014a150 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7f0a2000e8a0 tx=0x7f0a2000ed60 comp rx=0 tx=0).stop 2024-04-25T00:30:24.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.411+0000 7f0a37b7c640 1 -- 172.21.15.116:0/1265234292 shutdown_connections 2024-04-25T00:30:24.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.411+0000 7f0a37b7c640 1 --2- 172.21.15.116:0/1265234292 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0a14067940 0x7f0a14069e00 secure :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f0a2c0099c0 tx=0x7f0a2c009480 comp rx=0 tx=0).stop 2024-04-25T00:30:24.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.411+0000 7f0a37b7c640 1 --2- 172.21.15.116:0/1265234292 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0a30150d20 0x7f0a3013dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:24.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.411+0000 7f0a37b7c640 1 --2- 172.21.15.116:0/1265234292 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a300c4380 0x7f0a3014a150 unknown :-1 s=CLOSED pgs=1730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:24.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.411+0000 7f0a37b7c640 1 --2- 172.21.15.116:0/1265234292 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0a300c3240 0x7f0a30141630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:24.410 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.411+0000 7f0a37b7c640 1 -- 172.21.15.116:0/1265234292 >> 172.21.15.116:0/1265234292 conn(0x7f0a300c9160 msgr2=0x7f0a300c9540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:24.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.411+0000 7f0a37b7c640 1 -- 172.21.15.116:0/1265234292 shutdown_connections 2024-04-25T00:30:24.411 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.411+0000 7f0a37b7c640 1 -- 172.21.15.116:0/1265234292 wait complete. 2024-04-25T00:30:24.444 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T00:00:53.324376+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'a_metadata', 'create_time': '2024-04-25T00:01:03.197326+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'a'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'a_data', 'create_time': '2024-04-25T00:01:03.728618+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'a'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'b_metadata', 'create_time': '2024-04-25T00:01:17.125210+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'b'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'b_data', 'create_time': '2024-04-25T00:01:17.821873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'b'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T00:30:24.444 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T00:30:24.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.611+0000 7f01d7fff640 1 Processor -- start 2024-04-25T00:30:24.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.611+0000 7f01d7fff640 1 -- start start 2024-04-25T00:30:24.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.611+0000 7f01d7fff640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f01d00469f0 0x7f01d00d7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:24.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.611+0000 7f01d7fff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f01d00da7b0 0x7f01d0007af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:24.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.611+0000 7f01d7fff640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01d000aa00 con 0x7f01d00318e0 2024-04-25T00:30:24.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.611+0000 7f01d7fff640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f01d00d4270 con 0x7f01d00469f0 2024-04-25T00:30:24.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.611+0000 7f01d7fff640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f01d000c340 con 0x7f01d00da7b0 2024-04-25T00:30:24.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.615+0000 7f01d6ffd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f01d00318e0 0x7f01d000b470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:54562/0 (socket says 172.21.15.116:54562) 2024-04-25T00:30:24.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.615+0000 7f01d6ffd640 1 -- 172.21.15.116:0/3805690736 learned_addr learned my addr 172.21.15.116:0/3805690736 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:30:24.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.615+0000 7f01d67fc640 1 --2- 172.21.15.116:0/3805690736 >> v2:172.21.15.174:3300/0 conn(0x7f01d00469f0 0x7f01d00d7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:24.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.615+0000 7f01d77fe640 1 --2- 172.21.15.116:0/3805690736 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f01d00da7b0 0x7f01d0007af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:24.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.615+0000 7f01d67fc640 1 -- 172.21.15.116:0/3805690736 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f01d00da7b0 msgr2=0x7f01d0007af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:24.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.615+0000 7f01d67fc640 1 --2- 172.21.15.116:0/3805690736 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f01d00da7b0 0x7f01d0007af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:24.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.615+0000 7f01d5ffb640 1 -- 172.21.15.116:0/3805690736 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2999776054 0 0) 0x7f01d000aa00 con 0x7f01d00318e0 2024-04-25T00:30:24.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.615+0000 7f01d67fc640 1 -- 172.21.15.116:0/3805690736 >> v1:172.21.15.116:6789/0 conn(0x7f01d00318e0 legacy=0x7f01d000b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:24.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.615+0000 7f01d67fc640 1 -- 172.21.15.116:0/3805690736 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01d00dac60 con 0x7f01d00469f0 2024-04-25T00:30:24.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.615+0000 7f01d67fc640 1 --2- 172.21.15.116:0/3805690736 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f01d00469f0 0x7f01d00d7ca0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f01cc00ba50 tx=0x7f01cc01eab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1794ebbb05ad2ee5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:24.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.619+0000 7f01d5ffb640 1 -- 172.21.15.116:0/3805690736 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01cc0ad070 con 0x7f01d00469f0 2024-04-25T00:30:24.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.619+0000 7f01d5ffb640 1 -- 172.21.15.116:0/3805690736 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01cc0a7930 con 0x7f01d00469f0 2024-04-25T00:30:24.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.619+0000 7f01d5ffb640 1 -- 172.21.15.116:0/3805690736 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01cc0a7d00 con 0x7f01d00469f0 2024-04-25T00:30:24.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.619+0000 7f01d7fff640 1 -- 172.21.15.116:0/3805690736 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f01d00469f0 msgr2=0x7f01d00d7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:24.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.619+0000 7f01d7fff640 1 --2- 172.21.15.116:0/3805690736 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f01d00469f0 0x7f01d00d7ca0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f01cc00ba50 tx=0x7f01cc01eab0 comp rx=0 tx=0).stop 2024-04-25T00:30:24.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.619+0000 7f01d7fff640 1 -- 172.21.15.116:0/3805690736 shutdown_connections 2024-04-25T00:30:24.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.619+0000 7f01d7fff640 1 --2- 172.21.15.116:0/3805690736 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f01d00da7b0 0x7f01d0007af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:24.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.619+0000 7f01d7fff640 1 --2- 172.21.15.116:0/3805690736 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f01d00469f0 0x7f01d00d7ca0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:24.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.619+0000 7f01d7fff640 1 -- 172.21.15.116:0/3805690736 >> 172.21.15.116:0/3805690736 conn(0x7f01d002dba0 msgr2=0x7f01d002dfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:24.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.627+0000 7f01d7fff640 1 -- 172.21.15.116:0/3805690736 shutdown_connections 2024-04-25T00:30:24.625 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.627+0000 7f01d7fff640 1 -- 172.21.15.116:0/3805690736 wait complete. 2024-04-25T00:30:24.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.631+0000 7f01d7fff640 1 Processor -- start 2024-04-25T00:30:24.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.631+0000 7f01d7fff640 1 -- start start 2024-04-25T00:30:24.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.631+0000 7f01d7fff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f01d00318e0 0x7f01d0045720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:24.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.631+0000 7f01d7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01d00469f0 0x7f01d00d6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:24.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.631+0000 7f01d7fff640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f01d00da7b0 0x7f01d0178960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:24.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.631+0000 7f01d7fff640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f01d0047310 con 0x7f01d00469f0 2024-04-25T00:30:24.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.631+0000 7f01d7fff640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f01d0047190 con 0x7f01d00da7b0 2024-04-25T00:30:24.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.631+0000 7f01d7fff640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f01d00e3140 con 0x7f01d00318e0 2024-04-25T00:30:24.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.631+0000 7f01d67fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01d00469f0 0x7f01d00d6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:24.632 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.631+0000 7f01d67fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01d00469f0 0x7f01d00d6cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:46464/0 (socket says 172.21.15.116:46464) 2024-04-25T00:30:24.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.631+0000 7f01d67fc640 1 -- 172.21.15.116:0/989336883 learned_addr learned my addr 172.21.15.116:0/989336883 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:24.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.631+0000 7f01d77fe640 1 --2- 172.21.15.116:0/989336883 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f01d00da7b0 0x7f01d0178960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:24.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.631+0000 7f01d67fc640 1 -- 172.21.15.116:0/989336883 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f01d00318e0 msgr2=0x7f01d0045720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T00:30:24.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.631+0000 7f01d67fc640 1 --2- 172.21.15.116:0/989336883 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f01d00318e0 0x7f01d0045720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:24.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.631+0000 7f01d67fc640 1 -- 172.21.15.116:0/989336883 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f01d00da7b0 msgr2=0x7f01d0178960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:24.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.631+0000 7f01d67fc640 1 --2- 172.21.15.116:0/989336883 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f01d00da7b0 0x7f01d0178960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:24.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.631+0000 7f01d67fc640 1 -- 172.21.15.116:0/989336883 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01d00d04d0 con 0x7f01d00469f0 2024-04-25T00:30:24.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.631+0000 7f01d67fc640 1 --2- 172.21.15.116:0/989336883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01d00469f0 0x7f01d00d6cb0 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f01cc01c6e0 tx=0x7f01cc01b850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:24.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.635+0000 7f01bf7fe640 1 -- 172.21.15.116:0/989336883 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01cc0c0070 con 0x7f01d00469f0 2024-04-25T00:30:24.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.635+0000 7f01bf7fe640 1 -- 172.21.15.116:0/989336883 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01cc0b18c0 con 0x7f01d00469f0 2024-04-25T00:30:24.636 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.635+0000 7f01bf7fe640 1 -- 172.21.15.116:0/989336883 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01cc0ad040 con 0x7f01d00469f0 2024-04-25T00:30:24.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.635+0000 7f01d7fff640 1 -- 172.21.15.116:0/989336883 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01d00d0230 con 0x7f01d00469f0 2024-04-25T00:30:24.637 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.639+0000 7f01d7fff640 1 -- 172.21.15.116:0/989336883 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01d0043eb0 con 0x7f01d00469f0 2024-04-25T00:30:24.641 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.639+0000 7f01bf7fe640 1 -- 172.21.15.116:0/989336883 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f01cc0c3070 con 0x7f01d00469f0 2024-04-25T00:30:24.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.643+0000 7f01bf7fe640 1 --2- 172.21.15.116:0/989336883 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f01c4067910 0x7f01c4069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:24.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.643+0000 7f01bf7fe640 1 -- 172.21.15.116:0/989336883 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7f01cc120cd0 con 0x7f01d00469f0 2024-04-25T00:30:24.642 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.643+0000 7f01d6ffd640 1 --2- 172.21.15.116:0/989336883 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f01c4067910 0x7f01c4069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:24.644 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.643+0000 7f01d7fff640 1 -- 172.21.15.116:0/989336883 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01d00057f0 con 0x7f01d00469f0 2024-04-25T00:30:24.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.651+0000 7f01d6ffd640 1 --2- 172.21.15.116:0/989336883 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f01c4067910 0x7f01c4069dd0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f01c8000c10 tx=0x7f01c80078a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:24.661 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.663+0000 7f01bf7fe640 1 -- 172.21.15.116:0/989336883 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f01cc018730 con 0x7f01d00469f0 2024-04-25T00:30:24.994 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.995+0000 7f01d7fff640 1 -- 172.21.15.116:0/989336883 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f01d000bfd0 con 0x7f01d00469f0 2024-04-25T00:30:24.996 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-04-25T00:30:24.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.995+0000 7f01bf7fe640 1 -- 172.21.15.116:0/989336883 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v74) ==== 93+0+10 (secure 0 0 0) 0x7f01cc0a7900 con 0x7f01d00469f0 2024-04-25T00:30:25.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.999+0000 7f01d7fff640 1 -- 172.21.15.116:0/989336883 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f01c4067910 msgr2=0x7f01c4069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:25.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.999+0000 7f01d7fff640 1 --2- 172.21.15.116:0/989336883 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f01c4067910 0x7f01c4069dd0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f01c8000c10 tx=0x7f01c80078a0 comp rx=0 tx=0).stop 2024-04-25T00:30:25.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.999+0000 7f01d7fff640 1 -- 172.21.15.116:0/989336883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01d00469f0 msgr2=0x7f01d00d6cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:25.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.999+0000 7f01d7fff640 1 --2- 172.21.15.116:0/989336883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01d00469f0 0x7f01d00d6cb0 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f01cc01c6e0 tx=0x7f01cc01b850 comp rx=0 tx=0).stop 2024-04-25T00:30:25.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.999+0000 7f01d7fff640 1 -- 172.21.15.116:0/989336883 shutdown_connections 2024-04-25T00:30:25.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.999+0000 7f01d7fff640 1 --2- 172.21.15.116:0/989336883 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f01c4067910 0x7f01c4069dd0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:25.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.999+0000 7f01d7fff640 1 --2- 172.21.15.116:0/989336883 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f01d00da7b0 0x7f01d0178960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:25.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.999+0000 7f01d7fff640 1 --2- 172.21.15.116:0/989336883 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01d00469f0 0x7f01d00d6cb0 unknown :-1 s=CLOSED pgs=1732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:25.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.999+0000 7f01d7fff640 1 --2- 172.21.15.116:0/989336883 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f01d00318e0 0x7f01d0045720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:25.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.999+0000 7f01d7fff640 1 -- 172.21.15.116:0/989336883 >> 172.21.15.116:0/989336883 conn(0x7f01d002dba0 msgr2=0x7f01d004aa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:25.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.999+0000 7f01d7fff640 1 -- 172.21.15.116:0/989336883 shutdown_connections 2024-04-25T00:30:25.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:24.999+0000 7f01d7fff640 1 -- 172.21.15.116:0/989336883 wait complete. 2024-04-25T00:30:25.035 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_metadata pg_num 2024-04-25T00:30:25.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.223+0000 7f7c97390640 1 Processor -- start 2024-04-25T00:30:25.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.223+0000 7f7c97390640 1 -- start start 2024-04-25T00:30:25.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.227+0000 7f7c97390640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f7c88048350 0x7f7c880d82b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:25.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.227+0000 7f7c97390640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c8800c350 con 0x7f7c880d2590 2024-04-25T00:30:25.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.227+0000 7f7c97390640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c8800c630 con 0x7f7c8800be50 2024-04-25T00:30:25.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.227+0000 7f7c97390640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f7c8800cf60 con 0x7f7c88048350 2024-04-25T00:30:25.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.227+0000 7f7c9638e640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f7c88048350 0x7f7c880d82b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:25.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.227+0000 7f7c96b8f640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f7c8800be50 0x7f7c880e2c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:52522/0 (socket says 172.21.15.116:52522) 2024-04-25T00:30:25.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.227+0000 7f7c96b8f640 1 -- 172.21.15.116:0/1813061619 learned_addr learned my addr 172.21.15.116:0/1813061619 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:30:25.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.227+0000 7f7c9538c640 1 -- 172.21.15.116:0/1813061619 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2099526919 0 0) 0x7f7c8800c630 con 0x7f7c8800be50 2024-04-25T00:30:25.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.227+0000 7f7c9538c640 1 -- 172.21.15.116:0/1813061619 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c7c003530 con 0x7f7c8800be50 2024-04-25T00:30:25.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.227+0000 7f7c9638e640 1 -- 172.21.15.116:0/1813061619 >> v1:172.21.15.174:6789/0 conn(0x7f7c8800be50 legacy=0x7f7c880e2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:25.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.227+0000 7f7c9638e640 1 -- 172.21.15.116:0/1813061619 >> v1:172.21.15.116:6789/0 conn(0x7f7c880d2590 legacy=0x7f7c8804baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:25.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.227+0000 7f7c9638e640 1 -- 172.21.15.116:0/1813061619 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c8800c910 con 0x7f7c88048350 2024-04-25T00:30:25.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.231+0000 7f7c9638e640 1 --2- 172.21.15.116:0/1813061619 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7c88048350 0x7f7c880d82b0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f7c800097c0 tx=0x7f7c8001c3d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=983fa16c120154ee server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:25.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.231+0000 7f7c9538c640 1 -- 172.21.15.116:0/1813061619 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c800b0070 con 0x7f7c88048350 2024-04-25T00:30:25.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.231+0000 7f7c9538c640 1 -- 172.21.15.116:0/1813061619 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c800ab600 con 0x7f7c88048350 2024-04-25T00:30:25.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.231+0000 7f7c9538c640 1 -- 172.21.15.116:0/1813061619 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c800a4e60 con 0x7f7c88048350 2024-04-25T00:30:25.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.231+0000 7f7c97390640 1 -- 172.21.15.116:0/1813061619 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7c88048350 msgr2=0x7f7c880d82b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:25.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.231+0000 7f7c97390640 1 --2- 172.21.15.116:0/1813061619 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7c88048350 0x7f7c880d82b0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f7c800097c0 tx=0x7f7c8001c3d0 comp rx=0 tx=0).stop 2024-04-25T00:30:25.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.231+0000 7f7c97390640 1 -- 172.21.15.116:0/1813061619 shutdown_connections 2024-04-25T00:30:25.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.231+0000 7f7c97390640 1 --2- 172.21.15.116:0/1813061619 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7c88048350 0x7f7c880d82b0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:25.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.231+0000 7f7c97390640 1 -- 172.21.15.116:0/1813061619 >> 172.21.15.116:0/1813061619 conn(0x7f7c8802f4c0 msgr2=0x7f7c8802f8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:25.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.231+0000 7f7c97390640 1 -- 172.21.15.116:0/1813061619 shutdown_connections 2024-04-25T00:30:25.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.231+0000 7f7c97390640 1 -- 172.21.15.116:0/1813061619 wait complete. 2024-04-25T00:30:25.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.235+0000 7f7c97390640 1 Processor -- start 2024-04-25T00:30:25.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.235+0000 7f7c97390640 1 -- start start 2024-04-25T00:30:25.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.239+0000 7f7c97390640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c8800be50 0x7f7c880c7e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:25.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.239+0000 7f7c97390640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7c88048350 0x7f7c880c5940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:25.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.239+0000 7f7c97390640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7c880d2590 0x7f7c880c5e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:25.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.239+0000 7f7c97390640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7c880e6b70 con 0x7f7c8800be50 2024-04-25T00:30:25.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.239+0000 7f7c97390640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7c88030340 con 0x7f7c88048350 2024-04-25T00:30:25.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.239+0000 7f7c97390640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f7c880e6cf0 con 0x7f7c880d2590 2024-04-25T00:30:25.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.239+0000 7f7c96b8f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7c880d2590 0x7f7c880c5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:25.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.239+0000 7f7c96b8f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7c880d2590 0x7f7c880c5e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:50624/0 (socket says 172.21.15.116:50624) 2024-04-25T00:30:25.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.239+0000 7f7c96b8f640 1 -- 172.21.15.116:0/1062373741 learned_addr learned my addr 172.21.15.116:0/1062373741 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:25.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.239+0000 7f7c95b8d640 1 --2- 172.21.15.116:0/1062373741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7c88048350 0x7f7c880c5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:25.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.239+0000 7f7c96b8f640 1 -- 172.21.15.116:0/1062373741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7c88048350 msgr2=0x7f7c880c5940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:25.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.239+0000 7f7c96b8f640 1 --2- 172.21.15.116:0/1062373741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7c88048350 0x7f7c880c5940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:25.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.239+0000 7f7c96b8f640 1 -- 172.21.15.116:0/1062373741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c8800be50 msgr2=0x7f7c880c7e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T00:30:25.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.239+0000 7f7c96b8f640 1 --2- 172.21.15.116:0/1062373741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c8800be50 0x7f7c880c7e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:25.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.239+0000 7f7c96b8f640 1 -- 172.21.15.116:0/1062373741 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c8802f230 con 0x7f7c880d2590 2024-04-25T00:30:25.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.239+0000 7f7c96b8f640 1 --2- 172.21.15.116:0/1062373741 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7c880d2590 0x7f7c880c5e80 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7c9006a140 tx=0x7f7c90073780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:25.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.239+0000 7f7c86ffd640 1 -- 172.21.15.116:0/1062373741 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c900728f0 con 0x7f7c880d2590 2024-04-25T00:30:25.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.239+0000 7f7c86ffd640 1 -- 172.21.15.116:0/1062373741 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c9007e030 con 0x7f7c880d2590 2024-04-25T00:30:25.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.239+0000 7f7c86ffd640 1 -- 172.21.15.116:0/1062373741 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c90072ad0 con 0x7f7c880d2590 2024-04-25T00:30:25.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.239+0000 7f7c97390640 1 -- 172.21.15.116:0/1062373741 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c88175a00 con 0x7f7c880d2590 2024-04-25T00:30:25.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.239+0000 7f7c97390640 1 -- 172.21.15.116:0/1062373741 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c880cda20 con 0x7f7c880d2590 2024-04-25T00:30:25.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.243+0000 7f7c97390640 1 -- 172.21.15.116:0/1062373741 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c880057f0 con 0x7f7c880d2590 2024-04-25T00:30:25.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.247+0000 7f7c86ffd640 1 -- 172.21.15.116:0/1062373741 <== mon.2 v2:172.21.15.174:3301/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7c90071040 con 0x7f7c880d2590 2024-04-25T00:30:25.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.247+0000 7f7c86ffd640 1 --2- 172.21.15.116:0/1062373741 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7c74070130 0x7f7c740725f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:25.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.247+0000 7f7c86ffd640 1 -- 172.21.15.116:0/1062373741 <== mon.2 v2:172.21.15.174:3301/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7f7c901138d0 con 0x7f7c880d2590 2024-04-25T00:30:25.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.247+0000 7f7c9638e640 1 --2- 172.21.15.116:0/1062373741 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7c74070130 0x7f7c740725f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:25.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.247+0000 7f7c9638e640 1 --2- 172.21.15.116:0/1062373741 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7c74070130 0x7f7c740725f0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f7c800ad000 tx=0x7f7c80008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:25.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.255+0000 7f7c86ffd640 1 -- 172.21.15.116:0/1062373741 <== mon.2 v2:172.21.15.174:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f7c900b2020 con 0x7f7c880d2590 2024-04-25T00:30:25.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.543+0000 7f7c97390640 1 -- 172.21.15.116:0/1062373741 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"} v 0) -- 0x7f7c88030120 con 0x7f7c880d2590 2024-04-25T00:30:25.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.543+0000 7f7c86ffd640 1 -- 172.21.15.116:0/1062373741 <== mon.2 v2:172.21.15.174:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"}]=0 v74) ==== 99+0+11 (secure 0 0 0) 0x7f7c900e2ac0 con 0x7f7c880d2590 2024-04-25T00:30:25.543 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-04-25T00:30:25.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.547+0000 7f7c97390640 1 -- 172.21.15.116:0/1062373741 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7c74070130 msgr2=0x7f7c740725f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:25.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.547+0000 7f7c97390640 1 --2- 172.21.15.116:0/1062373741 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7c74070130 0x7f7c740725f0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f7c800ad000 tx=0x7f7c80008040 comp rx=0 tx=0).stop 2024-04-25T00:30:25.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.547+0000 7f7c97390640 1 -- 172.21.15.116:0/1062373741 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7c880d2590 msgr2=0x7f7c880c5e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:25.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.547+0000 7f7c97390640 1 --2- 172.21.15.116:0/1062373741 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7c880d2590 0x7f7c880c5e80 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7c9006a140 tx=0x7f7c90073780 comp rx=0 tx=0).stop 2024-04-25T00:30:25.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.547+0000 7f7c97390640 1 -- 172.21.15.116:0/1062373741 shutdown_connections 2024-04-25T00:30:25.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.547+0000 7f7c97390640 1 --2- 172.21.15.116:0/1062373741 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7c74070130 0x7f7c740725f0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:25.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.547+0000 7f7c97390640 1 --2- 172.21.15.116:0/1062373741 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7c880d2590 0x7f7c880c5e80 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:25.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.547+0000 7f7c97390640 1 --2- 172.21.15.116:0/1062373741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7c88048350 0x7f7c880c5940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:25.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.547+0000 7f7c97390640 1 --2- 172.21.15.116:0/1062373741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7c8800be50 0x7f7c880c7e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:25.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.547+0000 7f7c97390640 1 -- 172.21.15.116:0/1062373741 >> 172.21.15.116:0/1062373741 conn(0x7f7c8802f4c0 msgr2=0x7f7c880d0a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:25.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.547+0000 7f7c97390640 1 -- 172.21.15.116:0/1062373741 shutdown_connections 2024-04-25T00:30:25.548 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.547+0000 7f7c97390640 1 -- 172.21.15.116:0/1062373741 wait complete. 2024-04-25T00:30:25.574 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_data pg_num 2024-04-25T00:30:25.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.839+0000 7fd0fca80640 1 Processor -- start 2024-04-25T00:30:25.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.839+0000 7fd0fca80640 1 -- start start 2024-04-25T00:30:25.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.843+0000 7fd0fca80640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fd0f80c3160 0x7fd0f8070170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:25.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.843+0000 7fd0fca80640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fd0f80c3ae0 0x7fd0f815df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:25.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.843+0000 7fd0fca80640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0f80c42c0 con 0x7fd0f806fd90 2024-04-25T00:30:25.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.843+0000 7fd0fca80640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fd0f8157740 con 0x7fd0f80c3ae0 2024-04-25T00:30:25.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.843+0000 7fd0fca80640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fd0f8157a20 con 0x7fd0f80c3160 2024-04-25T00:30:25.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.843+0000 7fd0f659c640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7fd0f806fd90 0x7fd0f8153b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:52528/0 (socket says 172.21.15.116:52528) 2024-04-25T00:30:25.845 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.843+0000 7fd0f659c640 1 -- 172.21.15.116:0/3451621607 learned_addr learned my addr 172.21.15.116:0/3451621607 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:30:25.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.847+0000 7fd0f559a640 1 -- 172.21.15.116:0/3451621607 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3420645278 0 0) 0x7fd0f80c42c0 con 0x7fd0f806fd90 2024-04-25T00:30:25.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.847+0000 7fd0f559a640 1 -- 172.21.15.116:0/3451621607 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd0e4003530 con 0x7fd0f806fd90 2024-04-25T00:30:25.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.847+0000 7fd0f6d9d640 1 --2- 172.21.15.116:0/3451621607 >> v2:172.21.15.116:3300/0 conn(0x7fd0f80c3ae0 0x7fd0f815df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:25.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.847+0000 7fd0f5d9b640 1 --2- 172.21.15.116:0/3451621607 >> v2:172.21.15.174:3300/0 conn(0x7fd0f80c3160 0x7fd0f8070170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:25.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.847+0000 7fd0f6d9d640 1 -- 172.21.15.116:0/3451621607 >> v2:172.21.15.174:3300/0 conn(0x7fd0f80c3160 msgr2=0x7fd0f8070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:25.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.847+0000 7fd0f559a640 1 -- 172.21.15.116:0/3451621607 <== mon.1 v1:172.21.15.174:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3915792694 0 0) 0x7fd0e4003530 con 0x7fd0f806fd90 2024-04-25T00:30:25.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.847+0000 7fd0f6d9d640 1 --2- 172.21.15.116:0/3451621607 >> v2:172.21.15.174:3300/0 conn(0x7fd0f80c3160 0x7fd0f8070170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:25.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.847+0000 7fd0f6d9d640 1 -- 172.21.15.116:0/3451621607 >> v1:172.21.15.174:6789/0 conn(0x7fd0f806fd90 legacy=0x7fd0f8153b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:25.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.847+0000 7fd0f6d9d640 1 -- 172.21.15.116:0/3451621607 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0f80a94a0 con 0x7fd0f80c3ae0 2024-04-25T00:30:25.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.847+0000 7fd0f5d9b640 1 --2- 172.21.15.116:0/3451621607 >> v2:172.21.15.174:3300/0 conn(0x7fd0f80c3160 0x7fd0f8070170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:30:25.848 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.847+0000 7fd0f6d9d640 1 --2- 172.21.15.116:0/3451621607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0f80c3ae0 0x7fd0f815df40 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7fd0f80c3860 tx=0x7fd0e801c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa6fcaaa790890cd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:25.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.847+0000 7fd0f559a640 1 -- 172.21.15.116:0/3451621607 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e80b0070 con 0x7fd0f80c3ae0 2024-04-25T00:30:25.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.847+0000 7fd0f559a640 1 -- 172.21.15.116:0/3451621607 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0e80ab400 con 0x7fd0f80c3ae0 2024-04-25T00:30:25.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.847+0000 7fd0f559a640 1 -- 172.21.15.116:0/3451621607 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e0001680 con 0x7fd0f80c3ae0 2024-04-25T00:30:25.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.851+0000 7fd0fca80640 1 -- 172.21.15.116:0/3451621607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0f80c3ae0 msgr2=0x7fd0f815df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:25.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.851+0000 7fd0fca80640 1 --2- 172.21.15.116:0/3451621607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0f80c3ae0 0x7fd0f815df40 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7fd0f80c3860 tx=0x7fd0e801c330 comp rx=0 tx=0).stop 2024-04-25T00:30:25.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.851+0000 7fd0fca80640 1 -- 172.21.15.116:0/3451621607 shutdown_connections 2024-04-25T00:30:25.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.851+0000 7fd0fca80640 1 --2- 172.21.15.116:0/3451621607 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0f80c3ae0 0x7fd0f815df40 unknown :-1 s=CLOSED pgs=1733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:25.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.851+0000 7fd0fca80640 1 --2- 172.21.15.116:0/3451621607 >> v2:172.21.15.174:3300/0 conn(0x7fd0f80c3160 0x7fd0f8070170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:25.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.851+0000 7fd0fca80640 1 -- 172.21.15.116:0/3451621607 >> 172.21.15.116:0/3451621607 conn(0x7fd0f80c49c0 msgr2=0x7fd0f80c4dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:25.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.855+0000 7fd0fca80640 1 -- 172.21.15.116:0/3451621607 shutdown_connections 2024-04-25T00:30:25.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.855+0000 7fd0fca80640 1 -- 172.21.15.116:0/3451621607 wait complete. 2024-04-25T00:30:25.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.855+0000 7fd0fca80640 1 Processor -- start 2024-04-25T00:30:25.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.855+0000 7fd0fca80640 1 -- start start 2024-04-25T00:30:25.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.859+0000 7fd0fca80640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0f806fd90 0x7fd0f81f04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:25.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.859+0000 7fd0f659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0f806fd90 0x7fd0f81f04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:25.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.859+0000 7fd0f659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0f806fd90 0x7fd0f81f04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38856/0 (socket says 172.21.15.116:38856) 2024-04-25T00:30:25.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.859+0000 7fd0f659c640 1 -- 172.21.15.116:0/398910961 learned_addr learned my addr 172.21.15.116:0/398910961 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:25.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.859+0000 7fd0fca80640 1 --2- 172.21.15.116:0/398910961 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd0f80c3160 0x7fd0f80c7300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:25.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.859+0000 7fd0fca80640 1 --2- 172.21.15.116:0/398910961 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0f80c3ae0 0x7fd0f81ecba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:25.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.859+0000 7fd0fca80640 1 -- 172.21.15.116:0/398910961 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f8161030 con 0x7fd0f806fd90 2024-04-25T00:30:25.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.859+0000 7fd0f5d9b640 1 --2- 172.21.15.116:0/398910961 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd0f80c3160 0x7fd0f80c7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:25.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.859+0000 7fd0fca80640 1 -- 172.21.15.116:0/398910961 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f8160eb0 con 0x7fd0f80c3ae0 2024-04-25T00:30:25.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.859+0000 7fd0f6d9d640 1 --2- 172.21.15.116:0/398910961 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0f80c3ae0 0x7fd0f81ecba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:25.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.859+0000 7fd0fca80640 1 -- 172.21.15.116:0/398910961 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fd0f81611b0 con 0x7fd0f80c3160 2024-04-25T00:30:25.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.859+0000 7fd0f659c640 1 -- 172.21.15.116:0/398910961 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd0f80c3160 msgr2=0x7fd0f80c7300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:25.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.859+0000 7fd0f659c640 1 --2- 172.21.15.116:0/398910961 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd0f80c3160 0x7fd0f80c7300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:25.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.859+0000 7fd0f659c640 1 -- 172.21.15.116:0/398910961 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0f80c3ae0 msgr2=0x7fd0f81ecba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:25.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.859+0000 7fd0f659c640 1 --2- 172.21.15.116:0/398910961 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0f80c3ae0 0x7fd0f81ecba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:25.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.859+0000 7fd0f659c640 1 -- 172.21.15.116:0/398910961 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0f814a9f0 con 0x7fd0f806fd90 2024-04-25T00:30:25.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.859+0000 7fd0f5d9b640 1 --2- 172.21.15.116:0/398910961 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd0f80c3160 0x7fd0f80c7300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:30:25.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.859+0000 7fd0f659c640 1 --2- 172.21.15.116:0/398910961 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0f806fd90 0x7fd0f81f04f0 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7fd0e0007f00 tx=0x7fd0e0010e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:25.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.859+0000 7fd0eeffd640 1 -- 172.21.15.116:0/398910961 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e001f070 con 0x7fd0f806fd90 2024-04-25T00:30:25.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.859+0000 7fd0eeffd640 1 -- 172.21.15.116:0/398910961 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0e0005ae0 con 0x7fd0f806fd90 2024-04-25T00:30:25.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.859+0000 7fd0eeffd640 1 -- 172.21.15.116:0/398910961 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e0007070 con 0x7fd0f806fd90 2024-04-25T00:30:25.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.859+0000 7fd0f6d9d640 1 --2- 172.21.15.116:0/398910961 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0f80c3ae0 0x7fd0f81ecba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:30:25.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.859+0000 7fd0fca80640 1 -- 172.21.15.116:0/398910961 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0f814acd0 con 0x7fd0f806fd90 2024-04-25T00:30:25.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.863+0000 7fd0eeffd640 1 -- 172.21.15.116:0/398910961 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fd0e0007210 con 0x7fd0f806fd90 2024-04-25T00:30:25.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.863+0000 7fd0eeffd640 1 --2- 172.21.15.116:0/398910961 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd0d80678f0 0x7fd0d8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:25.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.863+0000 7fd0fca80640 1 -- 172.21.15.116:0/398910961 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0f81ec680 con 0x7fd0f806fd90 2024-04-25T00:30:25.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.863+0000 7fd0f5d9b640 1 --2- 172.21.15.116:0/398910961 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd0d80678f0 0x7fd0d8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:25.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.863+0000 7fd0eeffd640 1 -- 172.21.15.116:0/398910961 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7fd0e005b5f0 con 0x7fd0f806fd90 2024-04-25T00:30:25.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.867+0000 7fd0f5d9b640 1 --2- 172.21.15.116:0/398910961 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd0d80678f0 0x7fd0d8069db0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fd0f80aa370 tx=0x7fd0f0007810 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:25.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.867+0000 7fd0fca80640 1 -- 172.21.15.116:0/398910961 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0f804f510 con 0x7fd0f806fd90 2024-04-25T00:30:25.872 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:25.871+0000 7fd0eeffd640 1 -- 172.21.15.116:0/398910961 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fd0e005d550 con 0x7fd0f806fd90 2024-04-25T00:30:26.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.191+0000 7fd0fca80640 1 -- 172.21.15.116:0/398910961 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"} v 0) -- 0x7fd0f80c5850 con 0x7fd0f806fd90 2024-04-25T00:30:26.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.191+0000 7fd0eeffd640 1 -- 172.21.15.116:0/398910961 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"}]=0 v74) ==== 95+0+11 (secure 0 0 0) 0x7fd0e0061850 con 0x7fd0f806fd90 2024-04-25T00:30:26.192 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-04-25T00:30:26.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.199+0000 7fd0fca80640 1 -- 172.21.15.116:0/398910961 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd0d80678f0 msgr2=0x7fd0d8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:26.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.199+0000 7fd0fca80640 1 --2- 172.21.15.116:0/398910961 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd0d80678f0 0x7fd0d8069db0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fd0f80aa370 tx=0x7fd0f0007810 comp rx=0 tx=0).stop 2024-04-25T00:30:26.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.199+0000 7fd0fca80640 1 -- 172.21.15.116:0/398910961 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0f806fd90 msgr2=0x7fd0f81f04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:26.198 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.199+0000 7fd0fca80640 1 --2- 172.21.15.116:0/398910961 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0f806fd90 0x7fd0f81f04f0 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7fd0e0007f00 tx=0x7fd0e0010e90 comp rx=0 tx=0).stop 2024-04-25T00:30:26.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.199+0000 7fd0fca80640 1 -- 172.21.15.116:0/398910961 shutdown_connections 2024-04-25T00:30:26.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.199+0000 7fd0fca80640 1 --2- 172.21.15.116:0/398910961 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fd0d80678f0 0x7fd0d8069db0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:26.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.199+0000 7fd0fca80640 1 --2- 172.21.15.116:0/398910961 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fd0f80c3ae0 0x7fd0f81ecba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:26.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.199+0000 7fd0fca80640 1 --2- 172.21.15.116:0/398910961 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fd0f80c3160 0x7fd0f80c7300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:26.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.199+0000 7fd0fca80640 1 --2- 172.21.15.116:0/398910961 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fd0f806fd90 0x7fd0f81f04f0 unknown :-1 s=CLOSED pgs=1734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:26.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.199+0000 7fd0fca80640 1 -- 172.21.15.116:0/398910961 >> 172.21.15.116:0/398910961 conn(0x7fd0f80c49c0 msgr2=0x7fd0f80acac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:26.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.199+0000 7fd0fca80640 1 -- 172.21.15.116:0/398910961 shutdown_connections 2024-04-25T00:30:26.200 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.199+0000 7fd0fca80640 1 -- 172.21.15.116:0/398910961 wait complete. 2024-04-25T00:30:26.229 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_metadata pg_num 2024-04-25T00:30:26.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.467+0000 7f3ffc00d640 1 Processor -- start 2024-04-25T00:30:26.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.467+0000 7f3ffc00d640 1 -- start start 2024-04-25T00:30:26.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.467+0000 7f3ffc00d640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f3ff4152da0 0x7f3ff40adc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:26.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.467+0000 7f3ffc00d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3ff406df30 0x7f3ff40c3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:26.469 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.467+0000 7f3ffc00d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3ff40c44e0 0x7f3ff4158f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:26.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.471+0000 7f3ffc00d640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f3ff406dd30 con 0x7f3ff406df30 2024-04-25T00:30:26.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.471+0000 7f3ffc00d640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f3ff4153180 con 0x7f3ff4152da0 2024-04-25T00:30:26.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.471+0000 7f3ffc00d640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f3ff406e5b0 con 0x7f3ff40c44e0 2024-04-25T00:30:26.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.471+0000 7f3ff9da9640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f3ff4152da0 0x7f3ff40adc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:26.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.471+0000 7f3ff9da9640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f3ff4152da0 0x7f3ff40adc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:60786/0 (socket says 172.21.15.116:60786) 2024-04-25T00:30:26.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.471+0000 7f3ff9da9640 1 -- 172.21.15.116:0/4124066825 learned_addr learned my addr 172.21.15.116:0/4124066825 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:26.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.471+0000 7f3ffa5aa640 1 --2- 172.21.15.116:0/4124066825 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3ff40c44e0 0x7f3ff4158f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:26.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.471+0000 7f3ff9da9640 1 -- 172.21.15.116:0/4124066825 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3ff40c44e0 msgr2=0x7f3ff4158f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:26.470 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.471+0000 7f3ff9da9640 1 --2- 172.21.15.116:0/4124066825 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3ff40c44e0 0x7f3ff4158f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:26.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.471+0000 7f3ff9da9640 1 -- 172.21.15.116:0/4124066825 >> v2:172.21.15.116:3300/0 conn(0x7f3ff406df30 msgr2=0x7f3ff40c3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:26.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.471+0000 7f3ff9da9640 1 --2- 172.21.15.116:0/4124066825 >> v2:172.21.15.116:3300/0 conn(0x7f3ff406df30 0x7f3ff40c3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:26.471 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.471+0000 7f3ff9da9640 1 -- 172.21.15.116:0/4124066825 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ff406e890 con 0x7f3ff4152da0 2024-04-25T00:30:26.473 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.475+0000 7f3ff9da9640 1 --2- 172.21.15.116:0/4124066825 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ff4152da0 0x7f3ff40adc80 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f3ff4070de0 tx=0x7f3fec01c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9bc8f23a5c0c5fc0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:26.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.479+0000 7f3ff8da7640 1 -- 172.21.15.116:0/4124066825 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fec0a5640 con 0x7f3ff4152da0 2024-04-25T00:30:26.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.479+0000 7f3ff8da7640 1 -- 172.21.15.116:0/4124066825 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3fec0a57e0 con 0x7f3ff4152da0 2024-04-25T00:30:26.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.479+0000 7f3ffc00d640 1 -- 172.21.15.116:0/4124066825 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ff4152da0 msgr2=0x7f3ff40adc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:26.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.479+0000 7f3ffc00d640 1 --2- 172.21.15.116:0/4124066825 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ff4152da0 0x7f3ff40adc80 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f3ff4070de0 tx=0x7f3fec01c8d0 comp rx=0 tx=0).stop 2024-04-25T00:30:26.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.479+0000 7f3ffc00d640 1 -- 172.21.15.116:0/4124066825 shutdown_connections 2024-04-25T00:30:26.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.479+0000 7f3ffc00d640 1 --2- 172.21.15.116:0/4124066825 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3ff40c44e0 0x7f3ff4158f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:26.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.479+0000 7f3ffc00d640 1 --2- 172.21.15.116:0/4124066825 >> v2:172.21.15.116:3300/0 conn(0x7f3ff406df30 0x7f3ff40c3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:26.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.479+0000 7f3ffc00d640 1 --2- 172.21.15.116:0/4124066825 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ff4152da0 0x7f3ff40adc80 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:26.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.479+0000 7f3ffc00d640 1 -- 172.21.15.116:0/4124066825 >> 172.21.15.116:0/4124066825 conn(0x7f3ff40c49c0 msgr2=0x7f3ff40c4dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:26.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.479+0000 7f3ffc00d640 1 -- 172.21.15.116:0/4124066825 shutdown_connections 2024-04-25T00:30:26.480 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.479+0000 7f3ffc00d640 1 -- 172.21.15.116:0/4124066825 wait complete. 2024-04-25T00:30:26.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.483+0000 7f3ffc00d640 1 Processor -- start 2024-04-25T00:30:26.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.483+0000 7f3ffc00d640 1 -- start start 2024-04-25T00:30:26.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.483+0000 7f3ffc00d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ff406df30 0x7f3ff41ee4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:26.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.483+0000 7f3ffc00d640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ff40c44e0 0x7f3ff414cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:26.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.483+0000 7f3ffc00d640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3ff4152da0 0x7f3ff4149c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:26.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.483+0000 7f3ffc00d640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3ff40c2a40 con 0x7f3ff406df30 2024-04-25T00:30:26.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.483+0000 7f3ff9da9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ff406df30 0x7f3ff41ee4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:26.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.483+0000 7f3ffc00d640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3ff40c28c0 con 0x7f3ff40c44e0 2024-04-25T00:30:26.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.483+0000 7f3ff9da9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ff406df30 0x7f3ff41ee4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38874/0 (socket says 172.21.15.116:38874) 2024-04-25T00:30:26.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.483+0000 7f3ff9da9640 1 -- 172.21.15.116:0/1554548135 learned_addr learned my addr 172.21.15.116:0/1554548135 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:26.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.483+0000 7f3ffc00d640 1 -- 172.21.15.116:0/1554548135 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f3ff40c2bc0 con 0x7f3ff4152da0 2024-04-25T00:30:26.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.483+0000 7f3ffa5aa640 1 --2- 172.21.15.116:0/1554548135 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3ff4152da0 0x7f3ff4149c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:26.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.483+0000 7f3ff9da9640 1 -- 172.21.15.116:0/1554548135 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3ff4152da0 msgr2=0x7f3ff4149c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:26.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.483+0000 7f3ff9da9640 1 --2- 172.21.15.116:0/1554548135 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3ff4152da0 0x7f3ff4149c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:26.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.483+0000 7f3ff9da9640 1 -- 172.21.15.116:0/1554548135 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ff40c44e0 msgr2=0x7f3ff414cf20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T00:30:26.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.483+0000 7f3ff9da9640 1 --2- 172.21.15.116:0/1554548135 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ff40c44e0 0x7f3ff414cf20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:26.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.483+0000 7f3ff9da9640 1 -- 172.21.15.116:0/1554548135 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ff414a1a0 con 0x7f3ff406df30 2024-04-25T00:30:26.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.483+0000 7f3ff9da9640 1 --2- 172.21.15.116:0/1554548135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ff406df30 0x7f3ff41ee4a0 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f3fec0af040 tx=0x7f3fec019670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:26.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.487+0000 7f3fea7fc640 1 -- 172.21.15.116:0/1554548135 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fec0a5640 con 0x7f3ff406df30 2024-04-25T00:30:26.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.487+0000 7f3fea7fc640 1 -- 172.21.15.116:0/1554548135 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3fec0bc070 con 0x7f3ff406df30 2024-04-25T00:30:26.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.487+0000 7f3fea7fc640 1 -- 172.21.15.116:0/1554548135 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fec0b68e0 con 0x7f3ff406df30 2024-04-25T00:30:26.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.487+0000 7f3ffc00d640 1 -- 172.21.15.116:0/1554548135 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ff414d520 con 0x7f3ff406df30 2024-04-25T00:30:26.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.487+0000 7f3ffc00d640 1 -- 172.21.15.116:0/1554548135 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ff414dae0 con 0x7f3ff406df30 2024-04-25T00:30:26.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.487+0000 7f3ffc00d640 1 -- 172.21.15.116:0/1554548135 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ff404f510 con 0x7f3ff406df30 2024-04-25T00:30:26.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.495+0000 7f3fea7fc640 1 -- 172.21.15.116:0/1554548135 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3fec0a57e0 con 0x7f3ff406df30 2024-04-25T00:30:26.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.495+0000 7f3fea7fc640 1 --2- 172.21.15.116:0/1554548135 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3fd80678c0 0x7f3fd8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:26.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.495+0000 7f3fea7fc640 1 -- 172.21.15.116:0/1554548135 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7f3fec122270 con 0x7f3ff406df30 2024-04-25T00:30:26.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.495+0000 7f3ff95a8640 1 --2- 172.21.15.116:0/1554548135 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3fd80678c0 0x7f3fd8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:26.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.495+0000 7f3ff95a8640 1 --2- 172.21.15.116:0/1554548135 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3fd80678c0 0x7f3fd8069d80 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f3ff4153940 tx=0x7f3ff0006c30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:26.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.503+0000 7f3fea7fc640 1 -- 172.21.15.116:0/1554548135 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f3fec0165e0 con 0x7f3ff406df30 2024-04-25T00:30:26.830 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.831+0000 7f3ffc00d640 1 -- 172.21.15.116:0/1554548135 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"} v 0) -- 0x7f3ff41573a0 con 0x7f3ff406df30 2024-04-25T00:30:26.831 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.831+0000 7f3fea7fc640 1 -- 172.21.15.116:0/1554548135 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"}]=0 v74) ==== 99+0+11 (secure 0 0 0) 0x7f3fec0f1a90 con 0x7f3ff406df30 2024-04-25T00:30:26.831 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-04-25T00:30:26.833 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.831+0000 7f3ffc00d640 1 -- 172.21.15.116:0/1554548135 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3fd80678c0 msgr2=0x7f3fd8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:26.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.831+0000 7f3ffc00d640 1 --2- 172.21.15.116:0/1554548135 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3fd80678c0 0x7f3fd8069d80 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f3ff4153940 tx=0x7f3ff0006c30 comp rx=0 tx=0).stop 2024-04-25T00:30:26.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.831+0000 7f3ffc00d640 1 -- 172.21.15.116:0/1554548135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ff406df30 msgr2=0x7f3ff41ee4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:26.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.835+0000 7f3ffc00d640 1 --2- 172.21.15.116:0/1554548135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ff406df30 0x7f3ff41ee4a0 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f3fec0af040 tx=0x7f3fec019670 comp rx=0 tx=0).stop 2024-04-25T00:30:26.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.835+0000 7f3ffc00d640 1 -- 172.21.15.116:0/1554548135 shutdown_connections 2024-04-25T00:30:26.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.835+0000 7f3ffc00d640 1 --2- 172.21.15.116:0/1554548135 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3fd80678c0 0x7f3fd8069d80 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:26.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.835+0000 7f3ffc00d640 1 --2- 172.21.15.116:0/1554548135 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3ff4152da0 0x7f3ff4149c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:26.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.835+0000 7f3ffc00d640 1 --2- 172.21.15.116:0/1554548135 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3ff40c44e0 0x7f3ff414cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:26.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.835+0000 7f3ffc00d640 1 --2- 172.21.15.116:0/1554548135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3ff406df30 0x7f3ff41ee4a0 unknown :-1 s=CLOSED pgs=1735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:26.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.835+0000 7f3ffc00d640 1 -- 172.21.15.116:0/1554548135 >> 172.21.15.116:0/1554548135 conn(0x7f3ff40c49c0 msgr2=0x7f3ff4154ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:26.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.835+0000 7f3ffc00d640 1 -- 172.21.15.116:0/1554548135 shutdown_connections 2024-04-25T00:30:26.834 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.835+0000 7f3ffc00d640 1 -- 172.21.15.116:0/1554548135 wait complete. 2024-04-25T00:30:26.848 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_data pg_num 2024-04-25T00:30:26.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.959+0000 7fb284694640 1 Processor -- start 2024-04-25T00:30:26.960 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.959+0000 7fb284694640 1 -- start start 2024-04-25T00:30:26.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.959+0000 7fb284694640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb27c148f20 0x7fb27c0c2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:26.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.963+0000 7fb284694640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fb27c06d510 0x7fb27c072500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:26.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.963+0000 7fb284694640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb27c06d000 0x7fb27c072a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:26.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.963+0000 7fb284694640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fb27c153350 con 0x7fb27c06d000 2024-04-25T00:30:26.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.963+0000 7fb284694640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fb27c149300 con 0x7fb27c06d510 2024-04-25T00:30:26.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.963+0000 7fb284694640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fb27c0a95d0 con 0x7fb27c148f20 2024-04-25T00:30:26.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.963+0000 7fb282c31640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb27c06d000 0x7fb27c072a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:26.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.963+0000 7fb282c31640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb27c06d000 0x7fb27c072a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38888/0 (socket says 172.21.15.116:38888) 2024-04-25T00:30:26.962 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.963+0000 7fb282c31640 1 -- 172.21.15.116:0/688455108 learned_addr learned my addr 172.21.15.116:0/688455108 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:26.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.963+0000 7fb282430640 1 --2- 172.21.15.116:0/688455108 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb27c148f20 0x7fb27c0c2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:26.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.963+0000 7fb281c2f640 1 --2- 172.21.15.116:0/688455108 >> v2:172.21.15.174:3300/0 conn(0x7fb27c06d510 0x7fb27c072500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:26.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.963+0000 7fb282c31640 1 -- 172.21.15.116:0/688455108 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb27c148f20 msgr2=0x7fb27c0c2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:26.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.963+0000 7fb282c31640 1 --2- 172.21.15.116:0/688455108 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb27c148f20 0x7fb27c0c2930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:26.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.963+0000 7fb282c31640 1 -- 172.21.15.116:0/688455108 >> v2:172.21.15.174:3300/0 conn(0x7fb27c06d510 msgr2=0x7fb27c072500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:26.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.963+0000 7fb282c31640 1 --2- 172.21.15.116:0/688455108 >> v2:172.21.15.174:3300/0 conn(0x7fb27c06d510 0x7fb27c072500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:26.963 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.963+0000 7fb282c31640 1 -- 172.21.15.116:0/688455108 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb27c153070 con 0x7fb27c06d000 2024-04-25T00:30:26.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.963+0000 7fb282c31640 1 --2- 172.21.15.116:0/688455108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb27c06d000 0x7fb27c072a40 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7fb27c068110 tx=0x7fb27401ead0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80e708237bf557e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:26.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.963+0000 7fb28142e640 1 -- 172.21.15.116:0/688455108 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb27400b2d0 con 0x7fb27c06d000 2024-04-25T00:30:26.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.963+0000 7fb28142e640 1 -- 172.21.15.116:0/688455108 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb27400b470 con 0x7fb27c06d000 2024-04-25T00:30:26.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.963+0000 7fb28142e640 1 -- 172.21.15.116:0/688455108 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2740a7980 con 0x7fb27c06d000 2024-04-25T00:30:26.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.963+0000 7fb284694640 1 -- 172.21.15.116:0/688455108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb27c06d000 msgr2=0x7fb27c072a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:26.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.963+0000 7fb284694640 1 --2- 172.21.15.116:0/688455108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb27c06d000 0x7fb27c072a40 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7fb27c068110 tx=0x7fb27401ead0 comp rx=0 tx=0).stop 2024-04-25T00:30:26.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.963+0000 7fb284694640 1 -- 172.21.15.116:0/688455108 shutdown_connections 2024-04-25T00:30:26.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.963+0000 7fb284694640 1 --2- 172.21.15.116:0/688455108 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb27c06d000 0x7fb27c072a40 unknown :-1 s=CLOSED pgs=1736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:26.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.963+0000 7fb284694640 1 --2- 172.21.15.116:0/688455108 >> v2:172.21.15.174:3300/0 conn(0x7fb27c06d510 0x7fb27c072500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:26.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.963+0000 7fb284694640 1 --2- 172.21.15.116:0/688455108 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb27c148f20 0x7fb27c0c2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:26.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.963+0000 7fb284694640 1 -- 172.21.15.116:0/688455108 >> 172.21.15.116:0/688455108 conn(0x7fb27c0c3a20 msgr2=0x7fb27c0c3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:26.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.963+0000 7fb284694640 1 -- 172.21.15.116:0/688455108 shutdown_connections 2024-04-25T00:30:26.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.963+0000 7fb284694640 1 -- 172.21.15.116:0/688455108 wait complete. 2024-04-25T00:30:26.967 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.967+0000 7fb284694640 1 Processor -- start 2024-04-25T00:30:26.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.967+0000 7fb284694640 1 -- start start 2024-04-25T00:30:26.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.967+0000 7fb284694640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb27c06d000 0x7fb27c14ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:26.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.967+0000 7fb284694640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb27c06d510 0x7fb27c143760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:26.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.967+0000 7fb284694640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb27c148f20 0x7fb27c145540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:26.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.967+0000 7fb284694640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb27c1563d0 con 0x7fb27c06d510 2024-04-25T00:30:26.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.967+0000 7fb284694640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb27c156250 con 0x7fb27c06d000 2024-04-25T00:30:26.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.967+0000 7fb284694640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fb27c156550 con 0x7fb27c148f20 2024-04-25T00:30:26.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.967+0000 7fb281c2f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb27c06d510 0x7fb27c143760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:26.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.967+0000 7fb281c2f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb27c06d510 0x7fb27c143760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38892/0 (socket says 172.21.15.116:38892) 2024-04-25T00:30:26.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.967+0000 7fb281c2f640 1 -- 172.21.15.116:0/1382516103 learned_addr learned my addr 172.21.15.116:0/1382516103 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:26.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.967+0000 7fb282430640 1 --2- 172.21.15.116:0/1382516103 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb27c06d000 0x7fb27c14ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:26.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.967+0000 7fb281c2f640 1 -- 172.21.15.116:0/1382516103 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb27c148f20 msgr2=0x7fb27c145540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:26.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.967+0000 7fb281c2f640 1 --2- 172.21.15.116:0/1382516103 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb27c148f20 0x7fb27c145540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:26.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.967+0000 7fb281c2f640 1 -- 172.21.15.116:0/1382516103 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb27c06d000 msgr2=0x7fb27c14ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:26.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.967+0000 7fb281c2f640 1 --2- 172.21.15.116:0/1382516103 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb27c06d000 0x7fb27c14ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:26.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.967+0000 7fb281c2f640 1 -- 172.21.15.116:0/1382516103 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb27c153620 con 0x7fb27c06d510 2024-04-25T00:30:26.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.967+0000 7fb282430640 1 --2- 172.21.15.116:0/1382516103 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb27c06d000 0x7fb27c14ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:30:26.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.967+0000 7fb281c2f640 1 --2- 172.21.15.116:0/1382516103 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb27c06d510 0x7fb27c143760 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7fb26c00d800 tx=0x7fb26c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:26.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.967+0000 7fb272ffd640 1 -- 172.21.15.116:0/1382516103 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb26c018020 con 0x7fb27c06d510 2024-04-25T00:30:26.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.967+0000 7fb272ffd640 1 -- 172.21.15.116:0/1382516103 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb26c0096a0 con 0x7fb27c06d510 2024-04-25T00:30:26.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.967+0000 7fb272ffd640 1 -- 172.21.15.116:0/1382516103 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb26c009a50 con 0x7fb27c06d510 2024-04-25T00:30:26.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.967+0000 7fb284694640 1 -- 172.21.15.116:0/1382516103 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb27c153900 con 0x7fb27c06d510 2024-04-25T00:30:26.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.967+0000 7fb284694640 1 -- 172.21.15.116:0/1382516103 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb27c143ca0 con 0x7fb27c06d510 2024-04-25T00:30:26.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.967+0000 7fb284694640 1 -- 172.21.15.116:0/1382516103 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb27c153ad0 con 0x7fb27c06d510 2024-04-25T00:30:26.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.967+0000 7fb272ffd640 1 -- 172.21.15.116:0/1382516103 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb26c01b020 con 0x7fb27c06d510 2024-04-25T00:30:26.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.967+0000 7fb272ffd640 1 --2- 172.21.15.116:0/1382516103 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb2600678c0 0x7fb260069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:26.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.967+0000 7fb272ffd640 1 -- 172.21.15.116:0/1382516103 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7fb26c0875f0 con 0x7fb27c06d510 2024-04-25T00:30:26.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.971+0000 7fb282430640 1 --2- 172.21.15.116:0/1382516103 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb2600678c0 0x7fb260069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:26.970 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.971+0000 7fb282430640 1 --2- 172.21.15.116:0/1382516103 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb2600678c0 0x7fb260069d80 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fb2780097a0 tx=0x7fb278008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:26.973 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:26.975+0000 7fb272ffd640 1 -- 172.21.15.116:0/1382516103 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fb27c153ad0 con 0x7fb27c06d510 2024-04-25T00:30:27.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.119+0000 7fb284694640 1 -- 172.21.15.116:0/1382516103 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"} v 0) -- 0x7fb27c04f510 con 0x7fb27c06d510 2024-04-25T00:30:27.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.123+0000 7fb272ffd640 1 -- 172.21.15.116:0/1382516103 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"}]=0 v74) ==== 95+0+11 (secure 0 0 0) 0x7fb26c022020 con 0x7fb27c06d510 2024-04-25T00:30:27.122 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-04-25T00:30:27.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.123+0000 7fb284694640 1 -- 172.21.15.116:0/1382516103 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb2600678c0 msgr2=0x7fb260069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:27.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.123+0000 7fb284694640 1 --2- 172.21.15.116:0/1382516103 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb2600678c0 0x7fb260069d80 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fb2780097a0 tx=0x7fb278008040 comp rx=0 tx=0).stop 2024-04-25T00:30:27.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.123+0000 7fb284694640 1 -- 172.21.15.116:0/1382516103 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb27c06d510 msgr2=0x7fb27c143760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:27.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.123+0000 7fb284694640 1 --2- 172.21.15.116:0/1382516103 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb27c06d510 0x7fb27c143760 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7fb26c00d800 tx=0x7fb26c00dcc0 comp rx=0 tx=0).stop 2024-04-25T00:30:27.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.123+0000 7fb284694640 1 -- 172.21.15.116:0/1382516103 shutdown_connections 2024-04-25T00:30:27.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.123+0000 7fb284694640 1 --2- 172.21.15.116:0/1382516103 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb2600678c0 0x7fb260069d80 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:27.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.123+0000 7fb284694640 1 --2- 172.21.15.116:0/1382516103 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb27c148f20 0x7fb27c145540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:27.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.123+0000 7fb284694640 1 --2- 172.21.15.116:0/1382516103 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb27c06d510 0x7fb27c143760 unknown :-1 s=CLOSED pgs=1737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:27.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.123+0000 7fb284694640 1 --2- 172.21.15.116:0/1382516103 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb27c06d000 0x7fb27c14ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:27.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.123+0000 7fb284694640 1 -- 172.21.15.116:0/1382516103 >> 172.21.15.116:0/1382516103 conn(0x7fb27c0c3a20 msgr2=0x7fb27c06e640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:27.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.123+0000 7fb284694640 1 -- 172.21.15.116:0/1382516103 shutdown_connections 2024-04-25T00:30:27.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.123+0000 7fb284694640 1 -- 172.21.15.116:0/1382516103 wait complete. 2024-04-25T00:30:27.139 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T00:30:27.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.247+0000 7f10bfd14640 1 Processor -- start 2024-04-25T00:30:27.247 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.247+0000 7f10bfd14640 1 -- start start 2024-04-25T00:30:27.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.247+0000 7f10bfd14640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f10b80a99f0 0x7f10b8153ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:27.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.247+0000 7f10bfd14640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10b80c3910 con 0x7f10b8150d20 2024-04-25T00:30:27.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.247+0000 7f10bfd14640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10b8154530 con 0x7f10b80a94a0 2024-04-25T00:30:27.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.247+0000 7f10bfd14640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f10b80c16e0 con 0x7f10b80a99f0 2024-04-25T00:30:27.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.247+0000 7f10bdab0640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f10b8150d20 0x7f10b80c2a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:46536/0 (socket says 172.21.15.116:46536) 2024-04-25T00:30:27.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.247+0000 7f10bdab0640 1 -- 172.21.15.116:0/1414734218 learned_addr learned my addr 172.21.15.116:0/1414734218 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:30:27.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.247+0000 7f10bcaae640 1 -- 172.21.15.116:0/1414734218 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1449268834 0 0) 0x7f10b80c3910 con 0x7f10b8150d20 2024-04-25T00:30:27.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.247+0000 7f10bcaae640 1 -- 172.21.15.116:0/1414734218 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f10a4003530 con 0x7f10b8150d20 2024-04-25T00:30:27.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.247+0000 7f10be2b1640 1 --2- 172.21.15.116:0/1414734218 >> v2:172.21.15.174:3300/0 conn(0x7f10b80a99f0 0x7f10b8153ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:27.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.247+0000 7f10bcaae640 1 -- 172.21.15.116:0/1414734218 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2039890123 0 0) 0x7f10a4003530 con 0x7f10b8150d20 2024-04-25T00:30:27.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.247+0000 7f10bcaae640 1 -- 172.21.15.116:0/1414734218 >> v2:172.21.15.174:3300/0 conn(0x7f10b80a99f0 msgr2=0x7f10b8153ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:27.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.247+0000 7f10bcaae640 1 --2- 172.21.15.116:0/1414734218 >> v2:172.21.15.174:3300/0 conn(0x7f10b80a99f0 0x7f10b8153ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:27.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.247+0000 7f10bcaae640 1 -- 172.21.15.116:0/1414734218 >> v1:172.21.15.174:6789/0 conn(0x7f10b80a94a0 legacy=0x7f10b81538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:27.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.247+0000 7f10bcaae640 1 -- 172.21.15.116:0/1414734218 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10b80c1400 con 0x7f10b8150d20 2024-04-25T00:30:27.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.247+0000 7f10bcaae640 1 -- 172.21.15.116:0/1414734218 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f10b4001680 con 0x7f10b8150d20 2024-04-25T00:30:27.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.247+0000 7f10bcaae640 1 -- 172.21.15.116:0/1414734218 >> v1:172.21.15.116:6789/0 conn(0x7f10b8150d20 legacy=0x7f10b80c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:27.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.247+0000 7f10bcaae640 1 --2- 172.21.15.116:0/1414734218 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f10a40039c0 0x7f10a4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:27.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.247+0000 7f10bcaae640 1 --2- 172.21.15.116:0/1414734218 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f10a4004810 0x7f10a4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:27.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.247+0000 7f10bcaae640 1 --2- 172.21.15.116:0/1414734218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10a40051d0 0x7f10a400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:27.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.247+0000 7f10bcaae640 1 -- 172.21.15.116:0/1414734218 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f10b80c16e0 con 0x7f10a40051d0 2024-04-25T00:30:27.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.247+0000 7f10bcaae640 1 -- 172.21.15.116:0/1414734218 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f10a400c070 con 0x7f10a4004810 2024-04-25T00:30:27.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.247+0000 7f10bcaae640 1 -- 172.21.15.116:0/1414734218 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f10a400c270 con 0x7f10a40039c0 2024-04-25T00:30:27.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.247+0000 7f10be2b1640 1 --2- 172.21.15.116:0/1414734218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10a40051d0 0x7f10a400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:27.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.247+0000 7f10bd2af640 1 --2- 172.21.15.116:0/1414734218 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f10a4004810 0x7f10a4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:27.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.247+0000 7f10bdab0640 1 --2- 172.21.15.116:0/1414734218 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f10a40039c0 0x7f10a4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:27.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.251+0000 7f10be2b1640 1 -- 172.21.15.116:0/1414734218 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f10a40039c0 msgr2=0x7f10a4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:27.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.251+0000 7f10be2b1640 1 --2- 172.21.15.116:0/1414734218 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f10a40039c0 0x7f10a4003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:27.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.251+0000 7f10be2b1640 1 -- 172.21.15.116:0/1414734218 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f10a4004810 msgr2=0x7f10a4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:27.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.251+0000 7f10be2b1640 1 --2- 172.21.15.116:0/1414734218 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f10a4004810 0x7f10a4004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:27.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.251+0000 7f10be2b1640 1 -- 172.21.15.116:0/1414734218 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10a4003530 con 0x7f10a40051d0 2024-04-25T00:30:27.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.251+0000 7f10bd2af640 1 --2- 172.21.15.116:0/1414734218 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f10a4004810 0x7f10a4004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:30:27.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.251+0000 7f10be2b1640 1 -- 172.21.15.116:0/1414734218 reap_dead start 2024-04-25T00:30:27.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.251+0000 7f10be2b1640 1 --2- 172.21.15.116:0/1414734218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10a40051d0 0x7f10a400b9d0 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f10b0009fc0 tx=0x7f10b001ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30f3d7f6ee0087e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:27.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.251+0000 7f10bcaae640 1 -- 172.21.15.116:0/1414734218 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10b0009c10 con 0x7f10a40051d0 2024-04-25T00:30:27.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.251+0000 7f10bcaae640 1 -- 172.21.15.116:0/1414734218 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10b00a5d10 con 0x7f10a40051d0 2024-04-25T00:30:27.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.251+0000 7f10bcaae640 1 -- 172.21.15.116:0/1414734218 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10b0003be0 con 0x7f10a40051d0 2024-04-25T00:30:27.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.251+0000 7f10bfd14640 1 -- 172.21.15.116:0/1414734218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10a40051d0 msgr2=0x7f10a400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:27.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.251+0000 7f10bfd14640 1 --2- 172.21.15.116:0/1414734218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10a40051d0 0x7f10a400b9d0 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f10b0009fc0 tx=0x7f10b001ccc0 comp rx=0 tx=0).stop 2024-04-25T00:30:27.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.251+0000 7f10bfd14640 1 -- 172.21.15.116:0/1414734218 shutdown_connections 2024-04-25T00:30:27.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.251+0000 7f10bfd14640 1 --2- 172.21.15.116:0/1414734218 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10a40051d0 0x7f10a400b9d0 unknown :-1 s=CLOSED pgs=1739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:27.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.251+0000 7f10bfd14640 1 -- 172.21.15.116:0/1414734218 >> 172.21.15.116:0/1414734218 conn(0x7f10b80c3b00 msgr2=0x7f10b80c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:27.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.251+0000 7f10bfd14640 1 -- 172.21.15.116:0/1414734218 shutdown_connections 2024-04-25T00:30:27.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.251+0000 7f10bfd14640 1 -- 172.21.15.116:0/1414734218 wait complete. 2024-04-25T00:30:27.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.255+0000 7f10bfd14640 1 Processor -- start 2024-04-25T00:30:27.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.255+0000 7f10bfd14640 1 -- start start 2024-04-25T00:30:27.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.255+0000 7f10bfd14640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f10a40051d0 0x7f10b8152b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:27.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.255+0000 7f10bfd14640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f10b814afb0 0x7f10b8154b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:27.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.255+0000 7f10bfd14640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10b8142880 0x7f10b8142d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:27.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.255+0000 7f10bfd14640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f10b806e940 con 0x7f10b8142880 2024-04-25T00:30:27.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.255+0000 7f10bfd14640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f10b80c1280 con 0x7f10b814afb0 2024-04-25T00:30:27.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.255+0000 7f10bfd14640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f10b806ec20 con 0x7f10a40051d0 2024-04-25T00:30:27.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.255+0000 7f10be2b1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10b8142880 0x7f10b8142d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:27.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.255+0000 7f10be2b1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10b8142880 0x7f10b8142d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38908/0 (socket says 172.21.15.116:38908) 2024-04-25T00:30:27.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.255+0000 7f10be2b1640 1 -- 172.21.15.116:0/1743285301 learned_addr learned my addr 172.21.15.116:0/1743285301 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:27.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.255+0000 7f10bd2af640 1 --2- 172.21.15.116:0/1743285301 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f10b814afb0 0x7f10b8154b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:27.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.255+0000 7f10bdab0640 1 --2- 172.21.15.116:0/1743285301 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f10a40051d0 0x7f10b8152b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:27.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.255+0000 7f10be2b1640 1 -- 172.21.15.116:0/1743285301 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f10a40051d0 msgr2=0x7f10b8152b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:27.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.255+0000 7f10be2b1640 1 --2- 172.21.15.116:0/1743285301 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f10a40051d0 0x7f10b8152b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:27.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.255+0000 7f10be2b1640 1 -- 172.21.15.116:0/1743285301 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f10b814afb0 msgr2=0x7f10b8154b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:27.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.255+0000 7f10be2b1640 1 --2- 172.21.15.116:0/1743285301 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f10b814afb0 0x7f10b8154b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:27.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.255+0000 7f10be2b1640 1 -- 172.21.15.116:0/1743285301 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10b80aa9c0 con 0x7f10b8142880 2024-04-25T00:30:27.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.255+0000 7f10be2b1640 1 --2- 172.21.15.116:0/1743285301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10b8142880 0x7f10b8142d10 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f10b001d1a0 tx=0x7f10b0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:27.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.255+0000 7f10ae7fc640 1 -- 172.21.15.116:0/1743285301 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10b0009c10 con 0x7f10b8142880 2024-04-25T00:30:27.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.255+0000 7f10ae7fc640 1 -- 172.21.15.116:0/1743285301 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10b00bd070 con 0x7f10b8142880 2024-04-25T00:30:27.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.255+0000 7f10bfd14640 1 -- 172.21.15.116:0/1743285301 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10b814ca20 con 0x7f10b8142880 2024-04-25T00:30:27.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.255+0000 7f10ae7fc640 1 -- 172.21.15.116:0/1743285301 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10b00039a0 con 0x7f10b8142880 2024-04-25T00:30:27.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.255+0000 7f10bfd14640 1 -- 172.21.15.116:0/1743285301 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10b814cfe0 con 0x7f10b8142880 2024-04-25T00:30:27.258 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.255+0000 7f10bfd14640 1 -- 172.21.15.116:0/1743285301 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10b8068350 con 0x7f10b8142880 2024-04-25T00:30:27.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.255+0000 7f10ae7fc640 1 -- 172.21.15.116:0/1743285301 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f10b0003b40 con 0x7f10b8142880 2024-04-25T00:30:27.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.259+0000 7f10ae7fc640 1 --2- 172.21.15.116:0/1743285301 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f109c0678c0 0x7f109c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:27.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.259+0000 7f10ae7fc640 1 -- 172.21.15.116:0/1743285301 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7f10b0122a80 con 0x7f10b8142880 2024-04-25T00:30:27.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.259+0000 7f10bdab0640 1 --2- 172.21.15.116:0/1743285301 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f109c0678c0 0x7f109c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:27.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.259+0000 7f10bdab0640 1 --2- 172.21.15.116:0/1743285301 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f109c0678c0 0x7f109c069d80 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f10b4000c10 tx=0x7f10b4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:27.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.263+0000 7f10ae7fc640 1 -- 172.21.15.116:0/1743285301 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f10b00169d0 con 0x7f10b8142880 2024-04-25T00:30:27.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.419+0000 7f10bfd14640 1 -- 172.21.15.116:0/1743285301 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f10b80c34f0 con 0x7f10b8142880 2024-04-25T00:30:27.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.423+0000 7f10ae7fc640 1 -- 172.21.15.116:0/1743285301 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 92 entries v74) ==== 81+0+5682 (secure 0 0 0) 0x7f10b00f22d0 con 0x7f10b8142880 2024-04-25T00:30:27.422 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/3499878473 2024-04-26T00:30:15.781624+0000 2024-04-25T00:30:27.422 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/3499878473 2024-04-26T00:30:15.781624+0000 2024-04-25T00:30:27.422 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/2915274073 2024-04-26T00:29:50.776462+0000 2024-04-25T00:30:27.422 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6833/3891103804 2024-04-26T00:28:50.767503+0000 2024-04-25T00:30:27.422 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/2371126056 2024-04-26T00:27:35.754759+0000 2024-04-25T00:30:27.422 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/349772087 2024-04-26T00:27:15.748274+0000 2024-04-25T00:30:27.422 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/349772087 2024-04-26T00:27:15.748274+0000 2024-04-25T00:30:27.422 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6833/3693049068 2024-04-26T00:26:45.739776+0000 2024-04-25T00:30:27.422 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6832/3693049068 2024-04-26T00:26:45.739776+0000 2024-04-25T00:30:27.422 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/4181466530 2024-04-26T00:26:15.734586+0000 2024-04-25T00:30:27.422 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/4181466530 2024-04-26T00:26:15.734586+0000 2024-04-25T00:30:27.423 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/492366509 2024-04-26T00:24:15.701887+0000 2024-04-25T00:30:27.423 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6832/2998042514 2024-04-26T00:23:30.691827+0000 2024-04-25T00:30:27.423 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6833/663910035 2024-04-26T00:22:35.677323+0000 2024-04-25T00:30:27.423 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6832/663910035 2024-04-26T00:22:35.677323+0000 2024-04-25T00:30:27.423 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/321466844 2024-04-26T00:22:10.671673+0000 2024-04-25T00:30:27.423 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/980221971 2024-04-26T00:21:40.665115+0000 2024-04-25T00:30:27.423 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/991183935 2024-04-26T00:20:50.654151+0000 2024-04-25T00:30:27.423 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/991183935 2024-04-26T00:20:50.654151+0000 2024-04-25T00:30:27.423 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/321466844 2024-04-26T00:22:10.671673+0000 2024-04-25T00:30:27.423 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/1358527848 2024-04-26T00:11:00.511032+0000 2024-04-25T00:30:27.423 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6832/3841609112 2024-04-26T00:07:00.448498+0000 2024-04-25T00:30:27.423 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/1670061091 2024-04-26T00:09:30.490485+0000 2024-04-25T00:30:27.423 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/1971675962 2024-04-26T00:25:25.721965+0000 2024-04-25T00:30:27.423 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/1233829678 2024-04-26T00:07:30.454393+0000 2024-04-25T00:30:27.423 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/2371126056 2024-04-26T00:27:35.754759+0000 2024-04-25T00:30:27.424 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6834/2883220846 2024-04-26T00:11:50.521053+0000 2024-04-25T00:30:27.424 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/1233829678 2024-04-26T00:07:30.454393+0000 2024-04-25T00:30:27.424 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/3180024481 2024-04-26T00:15:25.567768+0000 2024-04-25T00:30:27.424 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/879531976 2024-04-26T00:20:10.642563+0000 2024-04-25T00:30:27.424 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6833/174781672 2024-04-26T00:02:40.399073+0000 2024-04-25T00:30:27.424 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6835/4215220222 2024-04-26T00:15:40.572498+0000 2024-04-25T00:30:27.424 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/221971311 2024-04-26T00:22:48.898075+0000 2024-04-25T00:30:27.424 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/1281466397 2024-04-26T00:18:20.613347+0000 2024-04-25T00:30:27.424 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6832/3891103804 2024-04-26T00:28:50.767503+0000 2024-04-25T00:30:27.424 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/1358527848 2024-04-26T00:11:00.511032+0000 2024-04-25T00:30:27.424 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/221971311 2024-04-26T00:22:48.898075+0000 2024-04-25T00:30:27.424 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6832/3006671671 2024-04-26T00:09:30.490406+0000 2024-04-25T00:30:27.424 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/980221971 2024-04-26T00:21:40.665115+0000 2024-04-25T00:30:27.424 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/1762356055 2024-04-26T00:03:36.597345+0000 2024-04-25T00:30:27.425 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6833/3006671671 2024-04-26T00:09:30.490406+0000 2024-04-25T00:30:27.425 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/1459177921 2024-04-26T00:03:30.406778+0000 2024-04-25T00:30:27.425 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/462190523 2024-04-26T00:14:55.560782+0000 2024-04-25T00:30:27.425 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/879531976 2024-04-26T00:20:10.642563+0000 2024-04-25T00:30:27.425 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6832/1459177921 2024-04-26T00:03:30.406778+0000 2024-04-25T00:30:27.425 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/1159002662 2024-04-26T00:10:20.502218+0000 2024-04-25T00:30:27.425 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/1762356055 2024-04-26T00:03:36.597345+0000 2024-04-25T00:30:27.425 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6834/4215220222 2024-04-26T00:15:40.572498+0000 2024-04-25T00:30:27.425 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/1971675962 2024-04-26T00:25:25.721965+0000 2024-04-25T00:30:27.425 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/3081407040 2024-04-26T00:17:25.598516+0000 2024-04-25T00:30:27.425 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/1599708526 2024-04-26T00:12:20.526742+0000 2024-04-25T00:30:27.425 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/2276529656 2024-04-26T00:19:05.629048+0000 2024-04-25T00:30:27.425 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/1159002662 2024-04-26T00:10:20.502218+0000 2024-04-25T00:30:27.425 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6833/2998042514 2024-04-26T00:23:30.691827+0000 2024-04-25T00:30:27.425 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/3081407040 2024-04-26T00:17:25.598516+0000 2024-04-25T00:30:27.426 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6835/2883220846 2024-04-26T00:11:50.521053+0000 2024-04-25T00:30:27.426 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/492366509 2024-04-26T00:24:15.701887+0000 2024-04-25T00:30:27.426 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/2004876235 2024-04-26T00:04:45.421611+0000 2024-04-25T00:30:27.426 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/174781672 2024-04-26T00:02:40.399073+0000 2024-04-25T00:30:27.426 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6833/3633657268 2024-04-26T00:25:40.724634+0000 2024-04-25T00:30:27.426 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6832/365773583 2024-04-26T00:14:58.857970+0000 2024-04-25T00:30:27.426 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/1889314958 2024-04-26T00:06:30.441408+0000 2024-04-25T00:30:27.426 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6833/3705654249 2024-04-26T00:12:45.533219+0000 2024-04-25T00:30:27.426 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6832/3705654249 2024-04-26T00:12:45.533219+0000 2024-04-25T00:30:27.426 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/1670061091 2024-04-26T00:09:30.490485+0000 2024-04-25T00:30:27.426 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/1599708526 2024-04-26T00:12:20.526742+0000 2024-04-25T00:30:27.426 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6832/3633657268 2024-04-26T00:25:40.724634+0000 2024-04-25T00:30:27.426 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/3583791822 2024-04-26T00:08:00.460587+0000 2024-04-25T00:30:27.426 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/3583791822 2024-04-26T00:08:00.460587+0000 2024-04-25T00:30:27.426 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/3398690896 2024-04-26T00:13:30.543729+0000 2024-04-25T00:30:27.427 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/3398690896 2024-04-26T00:13:30.543729+0000 2024-04-25T00:30:27.427 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/462190523 2024-04-26T00:14:55.560782+0000 2024-04-25T00:30:27.427 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/3336339376 2024-04-26T00:18:35.616887+0000 2024-04-25T00:30:27.427 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/3180024481 2024-04-26T00:15:25.567768+0000 2024-04-25T00:30:27.427 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6833/365773583 2024-04-26T00:14:58.857970+0000 2024-04-25T00:30:27.427 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/2012470875 2024-04-26T00:16:35.584511+0000 2024-04-25T00:30:27.427 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6833/2004876235 2024-04-26T00:04:45.421611+0000 2024-04-25T00:30:27.427 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/1281466397 2024-04-26T00:18:20.613347+0000 2024-04-25T00:30:27.427 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6832/2340581953 2024-04-26T00:16:45.587651+0000 2024-04-25T00:30:27.427 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/2915274073 2024-04-26T00:29:50.776462+0000 2024-04-25T00:30:27.427 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/1649769369 2024-04-26T00:04:35.418892+0000 2024-04-25T00:30:27.427 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6832/1649769369 2024-04-26T00:04:35.418892+0000 2024-04-25T00:30:27.427 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6833/2340581953 2024-04-26T00:16:45.587651+0000 2024-04-25T00:30:27.427 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/1889314958 2024-04-26T00:06:30.441408+0000 2024-04-25T00:30:27.427 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6832/3645553116 2024-04-26T00:17:35.601559+0000 2024-04-25T00:30:27.428 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6833/3645553116 2024-04-26T00:17:35.601559+0000 2024-04-25T00:30:27.428 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6833/3841609112 2024-04-26T00:07:00.448498+0000 2024-04-25T00:30:27.428 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/2012470875 2024-04-26T00:16:35.584511+0000 2024-04-25T00:30:27.428 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/3336339376 2024-04-26T00:18:35.616887+0000 2024-04-25T00:30:27.428 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/2276529656 2024-04-26T00:19:05.629048+0000 2024-04-25T00:30:27.428 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6832/1655959433 2024-04-26T00:19:30.633957+0000 2024-04-25T00:30:27.428 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6833/1655959433 2024-04-26T00:19:30.633957+0000 2024-04-25T00:30:27.429 INFO:teuthology.orchestra.run.smithi116.stderr:listed 92 entries 2024-04-25T00:30:27.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.423+0000 7f10bfd14640 1 -- 172.21.15.116:0/1743285301 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f109c0678c0 msgr2=0x7f109c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:27.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.423+0000 7f10bfd14640 1 --2- 172.21.15.116:0/1743285301 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f109c0678c0 0x7f109c069d80 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f10b4000c10 tx=0x7f10b4009040 comp rx=0 tx=0).stop 2024-04-25T00:30:27.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.423+0000 7f10bfd14640 1 -- 172.21.15.116:0/1743285301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10b8142880 msgr2=0x7f10b8142d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:27.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.423+0000 7f10bfd14640 1 --2- 172.21.15.116:0/1743285301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10b8142880 0x7f10b8142d10 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f10b001d1a0 tx=0x7f10b0009750 comp rx=0 tx=0).stop 2024-04-25T00:30:27.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.423+0000 7f10bfd14640 1 -- 172.21.15.116:0/1743285301 shutdown_connections 2024-04-25T00:30:27.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.423+0000 7f10bfd14640 1 --2- 172.21.15.116:0/1743285301 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f109c0678c0 0x7f109c069d80 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:27.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.423+0000 7f10bfd14640 1 --2- 172.21.15.116:0/1743285301 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10b8142880 0x7f10b8142d10 unknown :-1 s=CLOSED pgs=1740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:27.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.423+0000 7f10bfd14640 1 --2- 172.21.15.116:0/1743285301 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f10b814afb0 0x7f10b8154b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:27.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.423+0000 7f10bfd14640 1 --2- 172.21.15.116:0/1743285301 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f10a40051d0 0x7f10b8152b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:27.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.423+0000 7f10bfd14640 1 -- 172.21.15.116:0/1743285301 >> 172.21.15.116:0/1743285301 conn(0x7f10b80c3b00 msgr2=0x7f10b8147090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:27.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.423+0000 7f10bfd14640 1 -- 172.21.15.116:0/1743285301 shutdown_connections 2024-04-25T00:30:27.430 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.423+0000 7f10bfd14640 1 -- 172.21.15.116:0/1743285301 wait complete. 2024-04-25T00:30:27.439 DEBUG:teuthology.orchestra.run.smithi116:> set -ex 2024-04-25T00:30:27.440 DEBUG:teuthology.orchestra.run.smithi116:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T00:30:27.445 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T00:30:27.596 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.595+0000 7f4be27a3640 1 Processor -- start 2024-04-25T00:30:27.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.595+0000 7f4be27a3640 1 -- start start 2024-04-25T00:30:27.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.599+0000 7f4be27a3640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4bdc0a94a0 0x7f4bdc1538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:27.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.599+0000 7f4be27a3640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bdc0c3910 con 0x7f4bdc150d20 2024-04-25T00:30:27.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.599+0000 7f4be27a3640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bdc154530 con 0x7f4bdc0a99f0 2024-04-25T00:30:27.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.599+0000 7f4be27a3640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f4bdc0c16e0 con 0x7f4bdc0a94a0 2024-04-25T00:30:27.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.599+0000 7f4bdb7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4bdc0a94a0 0x7f4bdc1538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:27.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.599+0000 7f4bdb7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4bdc0a94a0 0x7f4bdc1538e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38918/0 (socket says 172.21.15.116:38918) 2024-04-25T00:30:27.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.599+0000 7f4bdb7fe640 1 -- 172.21.15.116:0/3578400135 learned_addr learned my addr 172.21.15.116:0/3578400135 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:27.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.599+0000 7f4bdbfff640 1 --1- 172.21.15.116:0/3578400135 >> v1:172.21.15.116:6789/0 conn(0x7f4bdc150d20 0x7f4bdc0c2a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:46538/0 (socket says 172.21.15.116:46538) 2024-04-25T00:30:27.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.599+0000 7f4bdb7fe640 1 -- 172.21.15.116:0/3578400135 >> v1:172.21.15.174:6789/0 conn(0x7f4bdc0a99f0 legacy=0x7f4bdc153e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:27.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.599+0000 7f4bdb7fe640 1 -- 172.21.15.116:0/3578400135 >> v1:172.21.15.116:6789/0 conn(0x7f4bdc150d20 legacy=0x7f4bdc0c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:27.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.599+0000 7f4bdaffd640 1 -- 172.21.15.116:0/3578400135 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2995920386 0 0) 0x7f4bdc0c3910 con 0x7f4bdc150d20 2024-04-25T00:30:27.599 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.599+0000 7f4bdb7fe640 1 -- 172.21.15.116:0/3578400135 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bdc0c1400 con 0x7f4bdc0a94a0 2024-04-25T00:30:27.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.599+0000 7f4bdb7fe640 1 --2- 172.21.15.116:0/3578400135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4bdc0a94a0 0x7f4bdc1538e0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc009870 tx=0x7f4bcc01c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5eae42a28ab301d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:27.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.599+0000 7f4bdaffd640 1 -- 172.21.15.116:0/3578400135 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bcc0b0070 con 0x7f4bdc0a94a0 2024-04-25T00:30:27.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.599+0000 7f4bdaffd640 1 -- 172.21.15.116:0/3578400135 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bcc0ab600 con 0x7f4bdc0a94a0 2024-04-25T00:30:27.600 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.599+0000 7f4bdaffd640 1 -- 172.21.15.116:0/3578400135 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bcc0ab9d0 con 0x7f4bdc0a94a0 2024-04-25T00:30:27.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.599+0000 7f4be27a3640 1 -- 172.21.15.116:0/3578400135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4bdc0a94a0 msgr2=0x7f4bdc1538e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:27.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.599+0000 7f4be27a3640 1 --2- 172.21.15.116:0/3578400135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4bdc0a94a0 0x7f4bdc1538e0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc009870 tx=0x7f4bcc01c520 comp rx=0 tx=0).stop 2024-04-25T00:30:27.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.599+0000 7f4be27a3640 1 -- 172.21.15.116:0/3578400135 shutdown_connections 2024-04-25T00:30:27.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.599+0000 7f4be27a3640 1 --2- 172.21.15.116:0/3578400135 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4bdc0a94a0 0x7f4bdc1538e0 unknown :-1 s=CLOSED pgs=1742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:27.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.599+0000 7f4be27a3640 1 -- 172.21.15.116:0/3578400135 >> 172.21.15.116:0/3578400135 conn(0x7f4bdc0c3b00 msgr2=0x7f4bdc0c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:27.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.599+0000 7f4be27a3640 1 -- 172.21.15.116:0/3578400135 shutdown_connections 2024-04-25T00:30:27.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.599+0000 7f4be27a3640 1 -- 172.21.15.116:0/3578400135 wait complete. 2024-04-25T00:30:27.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.603+0000 7f4be27a3640 1 Processor -- start 2024-04-25T00:30:27.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.603+0000 7f4be27a3640 1 -- start start 2024-04-25T00:30:27.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.603+0000 7f4be27a3640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4bdc0a94a0 0x7f4bdc0c0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:27.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.603+0000 7f4be27a3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4bdc0a99f0 0x7f4bdc14d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:27.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.603+0000 7f4be27a3640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4bdc150d20 0x7f4bdc14c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:27.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.603+0000 7f4be27a3640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4bdc0c32a0 con 0x7f4bdc0a99f0 2024-04-25T00:30:27.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.603+0000 7f4be27a3640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4bdc0c3120 con 0x7f4bdc150d20 2024-04-25T00:30:27.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.603+0000 7f4be27a3640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f4bdc161740 con 0x7f4bdc0a94a0 2024-04-25T00:30:27.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.603+0000 7f4bdb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4bdc0a99f0 0x7f4bdc14d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:27.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.603+0000 7f4bdb7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4bdc0a99f0 0x7f4bdc14d760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38934/0 (socket says 172.21.15.116:38934) 2024-04-25T00:30:27.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.603+0000 7f4bdb7fe640 1 -- 172.21.15.116:0/1465892603 learned_addr learned my addr 172.21.15.116:0/1465892603 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:27.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.603+0000 7f4be0d40640 1 --2- 172.21.15.116:0/1465892603 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4bdc150d20 0x7f4bdc14c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:27.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.603+0000 7f4bdb7fe640 1 -- 172.21.15.116:0/1465892603 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4bdc0a94a0 msgr2=0x7f4bdc0c0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:27.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.603+0000 7f4bdbfff640 1 --2- 172.21.15.116:0/1465892603 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4bdc0a94a0 0x7f4bdc0c0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:27.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.603+0000 7f4bdb7fe640 1 --2- 172.21.15.116:0/1465892603 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4bdc0a94a0 0x7f4bdc0c0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:27.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.603+0000 7f4bdb7fe640 1 -- 172.21.15.116:0/1465892603 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4bdc150d20 msgr2=0x7f4bdc14c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:27.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.603+0000 7f4bdb7fe640 1 --2- 172.21.15.116:0/1465892603 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4bdc150d20 0x7f4bdc14c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:27.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.603+0000 7f4bdb7fe640 1 -- 172.21.15.116:0/1465892603 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bdc1f1430 con 0x7f4bdc0a99f0 2024-04-25T00:30:27.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.603+0000 7f4be0d40640 1 --2- 172.21.15.116:0/1465892603 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4bdc150d20 0x7f4bdc14c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:30:27.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.603+0000 7f4bdb7fe640 1 --2- 172.21.15.116:0/1465892603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4bdc0a99f0 0x7f4bdc14d760 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc0ae040 tx=0x7f4bcc01ad60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:27.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.603+0000 7f4bd8ff9640 1 -- 172.21.15.116:0/1465892603 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bcc0ba020 con 0x7f4bdc0a99f0 2024-04-25T00:30:27.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.603+0000 7f4bd8ff9640 1 -- 172.21.15.116:0/1465892603 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bcc0b6620 con 0x7f4bdc0a99f0 2024-04-25T00:30:27.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.603+0000 7f4be27a3640 1 -- 172.21.15.116:0/1465892603 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bdc1f1710 con 0x7f4bdc0a99f0 2024-04-25T00:30:27.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.603+0000 7f4bd8ff9640 1 -- 172.21.15.116:0/1465892603 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bcc0b0040 con 0x7f4bdc0a99f0 2024-04-25T00:30:27.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.603+0000 7f4be27a3640 1 -- 172.21.15.116:0/1465892603 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bdc1f1cd0 con 0x7f4bdc0a99f0 2024-04-25T00:30:27.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.603+0000 7f4be27a3640 1 -- 172.21.15.116:0/1465892603 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bdc1f2000 con 0x7f4bdc0a99f0 2024-04-25T00:30:27.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.603+0000 7f4bd8ff9640 1 -- 172.21.15.116:0/1465892603 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4bcc0bd020 con 0x7f4bdc0a99f0 2024-04-25T00:30:27.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.603+0000 7f4bd8ff9640 1 --2- 172.21.15.116:0/1465892603 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4bc80678c0 0x7f4bc8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:27.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.607+0000 7f4bd8ff9640 1 -- 172.21.15.116:0/1465892603 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7f4bcc121300 con 0x7f4bdc0a99f0 2024-04-25T00:30:27.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.607+0000 7f4bdbfff640 1 --2- 172.21.15.116:0/1465892603 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4bc80678c0 0x7f4bc8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:27.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.607+0000 7f4bdbfff640 1 --2- 172.21.15.116:0/1465892603 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4bc80678c0 0x7f4bc8069d80 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f4bd0000f70 tx=0x7f4bd0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:27.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.611+0000 7f4bd8ff9640 1 -- 172.21.15.116:0/1465892603 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f4bcc0c4030 con 0x7f4bdc0a99f0 2024-04-25T00:30:27.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.759+0000 7f4be27a3640 1 -- 172.21.15.116:0/1465892603 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4bdc068350 con 0x7f4bdc0a99f0 2024-04-25T00:30:27.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.763+0000 7f4bd8ff9640 1 -- 172.21.15.116:0/1465892603 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v74) ==== 74+0+26655 (secure 0 0 0) 0x7f4bcc016110 con 0x7f4bdc0a99f0 2024-04-25T00:30:27.762 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:30:27.762 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":74,"fsid":"6d6a1aa7-0478-4c83-8057-05917e81f046","created":"2024-04-25T00:00:40.682493+0000","modified":"2024-04-25T00:30:15.781714+0000","last_up_change":"2024-04-25T00:00:51.822174+0000","last_in_change":"2024-04-25T00:00:42.980473+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T00:00:53.324376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-04-25T00:01:03.197326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-04-25T00:01:03.728618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-04-25T00:01:17.125210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-04-25T00:01:17.821873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8b6ddcb2-1ad9-4167-961c-bd6936bb9060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6809","nonce":3751765682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6811","nonce":3751765682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6815","nonce":3751765682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6813","nonce":3751765682}]},"public_addr":"172.21.15.116:6809/3751765682","cluster_addr":"172.21.15.116:6811/3751765682","heartbeat_back_addr":"172.21.15.116:6815/3751765682","heartbeat_front_addr":"172.21.15.116:6813/3751765682","state":["exists","up"]},{"osd":1,"uuid":"a6431768-994f-41da-8853-886f5687fb91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6801","nonce":377879124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6803","nonce":377879124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6807","nonce":377879124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6805","nonce":377879124}]},"public_addr":"172.21.15.116:6801/377879124","cluster_addr":"172.21.15.116:6803/377879124","heartbeat_back_addr":"172.21.15.116:6807/377879124","heartbeat_front_addr":"172.21.15.116:6805/377879124","state":["exists","up"]},{"osd":2,"uuid":"4f69bd40-5a23-45ff-8693-e1de75e1aa9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6817","nonce":1099282267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6819","nonce":1099282267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6823","nonce":1099282267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6821","nonce":1099282267}]},"public_addr":"172.21.15.116:6817/1099282267","cluster_addr":"172.21.15.116:6819/1099282267","heartbeat_back_addr":"172.21.15.116:6823/1099282267","heartbeat_front_addr":"172.21.15.116:6821/1099282267","state":["exists","up"]},{"osd":3,"uuid":"6a0d5344-8b6f-40d3-918d-0f0a4be688e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6825","nonce":1179942661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6827","nonce":1179942661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6831","nonce":1179942661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6829","nonce":1179942661}]},"public_addr":"172.21.15.116:6825/1179942661","cluster_addr":"172.21.15.116:6827/1179942661","heartbeat_back_addr":"172.21.15.116:6831/1179942661","heartbeat_front_addr":"172.21.15.116:6829/1179942661","state":["exists","up"]},{"osd":4,"uuid":"132e92e4-b825-441e-965a-ca00533fa86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6801","nonce":3132444330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6803","nonce":3132444330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6807","nonce":3132444330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6805","nonce":3132444330}]},"public_addr":"172.21.15.174:6801/3132444330","cluster_addr":"172.21.15.174:6803/3132444330","heartbeat_back_addr":"172.21.15.174:6807/3132444330","heartbeat_front_addr":"172.21.15.174:6805/3132444330","state":["exists","up"]},{"osd":5,"uuid":"a2eeb5b6-1b31-43cc-8306-a0adf3ee15bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6809","nonce":881817164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6811","nonce":881817164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6815","nonce":881817164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6813","nonce":881817164}]},"public_addr":"172.21.15.174:6809/881817164","cluster_addr":"172.21.15.174:6811/881817164","heartbeat_back_addr":"172.21.15.174:6815/881817164","heartbeat_front_addr":"172.21.15.174:6813/881817164","state":["exists","up"]},{"osd":6,"uuid":"588e873a-c841-4ba7-90f2-8956c87a9983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6825","nonce":117574251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6827","nonce":117574251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6831","nonce":117574251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6829","nonce":117574251}]},"public_addr":"172.21.15.174:6825/117574251","cluster_addr":"172.21.15.174:6827/117574251","heartbeat_back_addr":"172.21.15.174:6831/117574251","heartbeat_front_addr":"172.21.15.174:6829/117574251","state":["exists","up"]},{"osd":7,"uuid":"3b90f9b0-8a0e-4883-b088-ca1d4012870c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6817","nonce":2543899196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6819","nonce":2543899196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6823","nonce":2543899196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6821","nonce":2543899196}]},"public_addr":"172.21.15.174:6817/2543899196","cluster_addr":"172.21.15.174:6819/2543899196","heartbeat_back_addr":"172.21.15.174:6823/2543899196","heartbeat_front_addr":"172.21.15.174:6821/2543899196","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.293769+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.310034+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.404865+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.549658+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.192174+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.297266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.493649+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.452309+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2c","mappings":[{"from":0,"to":7}]},{"pgid":"5.3","mappings":[{"from":4,"to":5}]},{"pgid":"5.5","mappings":[{"from":4,"to":5}]},{"pgid":"5.14","mappings":[{"from":4,"to":5}]},{"pgid":"5.19","mappings":[{"from":4,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:6835/3499878473":"2024-04-26T00:30:15.781624+0000","172.21.15.116:6834/3499878473":"2024-04-26T00:30:15.781624+0000","172.21.15.174:6836/2915274073":"2024-04-26T00:29:50.776462+0000","172.21.15.116:6833/3891103804":"2024-04-26T00:28:50.767503+0000","172.21.15.116:6834/2371126056":"2024-04-26T00:27:35.754759+0000","172.21.15.174:6837/349772087":"2024-04-26T00:27:15.748274+0000","172.21.15.174:6836/349772087":"2024-04-26T00:27:15.748274+0000","172.21.15.116:6833/3693049068":"2024-04-26T00:26:45.739776+0000","172.21.15.116:6832/3693049068":"2024-04-26T00:26:45.739776+0000","172.21.15.174:6837/4181466530":"2024-04-26T00:26:15.734586+0000","172.21.15.174:6836/4181466530":"2024-04-26T00:26:15.734586+0000","172.21.15.174:6836/492366509":"2024-04-26T00:24:15.701887+0000","172.21.15.116:6832/2998042514":"2024-04-26T00:23:30.691827+0000","172.21.15.116:6833/663910035":"2024-04-26T00:22:35.677323+0000","172.21.15.116:6832/663910035":"2024-04-26T00:22:35.677323+0000","172.21.15.174:6837/321466844":"2024-04-26T00:22:10.671673+0000","172.21.15.116:6835/980221971":"2024-04-26T00:21:40.665115+0000","172.21.15.174:6837/991183935":"2024-04-26T00:20:50.654151+0000","172.21.15.174:6836/991183935":"2024-04-26T00:20:50.654151+0000","172.21.15.174:6836/321466844":"2024-04-26T00:22:10.671673+0000","172.21.15.116:6834/1358527848":"2024-04-26T00:11:00.511032+0000","172.21.15.116:6832/3841609112":"2024-04-26T00:07:00.448498+0000","172.21.15.116:6835/1670061091":"2024-04-26T00:09:30.490485+0000","172.21.15.116:6834/1971675962":"2024-04-26T00:25:25.721965+0000","172.21.15.116:6835/1233829678":"2024-04-26T00:07:30.454393+0000","172.21.15.116:6835/2371126056":"2024-04-26T00:27:35.754759+0000","172.21.15.174:6834/2883220846":"2024-04-26T00:11:50.521053+0000","172.21.15.116:6834/1233829678":"2024-04-26T00:07:30.454393+0000","172.21.15.174:6837/3180024481":"2024-04-26T00:15:25.567768+0000","172.21.15.116:6835/879531976":"2024-04-26T00:20:10.642563+0000","172.21.15.116:6833/174781672":"2024-04-26T00:02:40.399073+0000","172.21.15.174:6835/4215220222":"2024-04-26T00:15:40.572498+0000","172.21.15.116:6834/221971311":"2024-04-26T00:22:48.898075+0000","172.21.15.174:6836/1281466397":"2024-04-26T00:18:20.613347+0000","172.21.15.116:6832/3891103804":"2024-04-26T00:28:50.767503+0000","172.21.15.116:6835/1358527848":"2024-04-26T00:11:00.511032+0000","172.21.15.116:6835/221971311":"2024-04-26T00:22:48.898075+0000","172.21.15.116:6832/3006671671":"2024-04-26T00:09:30.490406+0000","172.21.15.116:6834/980221971":"2024-04-26T00:21:40.665115+0000","172.21.15.174:6836/1762356055":"2024-04-26T00:03:36.597345+0000","172.21.15.116:6833/3006671671":"2024-04-26T00:09:30.490406+0000","172.21.15.116:6834/1459177921":"2024-04-26T00:03:30.406778+0000","172.21.15.116:6834/462190523":"2024-04-26T00:14:55.560782+0000","172.21.15.116:6834/879531976":"2024-04-26T00:20:10.642563+0000","172.21.15.116:6832/1459177921":"2024-04-26T00:03:30.406778+0000","172.21.15.174:6837/1159002662":"2024-04-26T00:10:20.502218+0000","172.21.15.174:6837/1762356055":"2024-04-26T00:03:36.597345+0000","172.21.15.174:6834/4215220222":"2024-04-26T00:15:40.572498+0000","172.21.15.116:6835/1971675962":"2024-04-26T00:25:25.721965+0000","172.21.15.174:6836/3081407040":"2024-04-26T00:17:25.598516+0000","172.21.15.116:6834/1599708526":"2024-04-26T00:12:20.526742+0000","172.21.15.174:6836/2276529656":"2024-04-26T00:19:05.629048+0000","172.21.15.174:6836/1159002662":"2024-04-26T00:10:20.502218+0000","172.21.15.116:6833/2998042514":"2024-04-26T00:23:30.691827+0000","172.21.15.174:6837/3081407040":"2024-04-26T00:17:25.598516+0000","172.21.15.174:6835/2883220846":"2024-04-26T00:11:50.521053+0000","172.21.15.174:6837/492366509":"2024-04-26T00:24:15.701887+0000","172.21.15.116:6835/2004876235":"2024-04-26T00:04:45.421611+0000","172.21.15.116:6835/174781672":"2024-04-26T00:02:40.399073+0000","172.21.15.116:6833/3633657268":"2024-04-26T00:25:40.724634+0000","172.21.15.116:6832/365773583":"2024-04-26T00:14:58.857970+0000","172.21.15.174:6836/1889314958":"2024-04-26T00:06:30.441408+0000","172.21.15.116:6833/3705654249":"2024-04-26T00:12:45.533219+0000","172.21.15.116:6832/3705654249":"2024-04-26T00:12:45.533219+0000","172.21.15.116:6834/1670061091":"2024-04-26T00:09:30.490485+0000","172.21.15.116:6835/1599708526":"2024-04-26T00:12:20.526742+0000","172.21.15.116:6832/3633657268":"2024-04-26T00:25:40.724634+0000","172.21.15.174:6836/3583791822":"2024-04-26T00:08:00.460587+0000","172.21.15.174:6837/3583791822":"2024-04-26T00:08:00.460587+0000","172.21.15.174:6836/3398690896":"2024-04-26T00:13:30.543729+0000","172.21.15.174:6837/3398690896":"2024-04-26T00:13:30.543729+0000","172.21.15.116:6835/462190523":"2024-04-26T00:14:55.560782+0000","172.21.15.116:6834/3336339376":"2024-04-26T00:18:35.616887+0000","172.21.15.174:6836/3180024481":"2024-04-26T00:15:25.567768+0000","172.21.15.116:6833/365773583":"2024-04-26T00:14:58.857970+0000","172.21.15.116:6835/2012470875":"2024-04-26T00:16:35.584511+0000","172.21.15.116:6833/2004876235":"2024-04-26T00:04:45.421611+0000","172.21.15.174:6837/1281466397":"2024-04-26T00:18:20.613347+0000","172.21.15.116:6832/2340581953":"2024-04-26T00:16:45.587651+0000","172.21.15.174:6837/2915274073":"2024-04-26T00:29:50.776462+0000","172.21.15.116:6834/1649769369":"2024-04-26T00:04:35.418892+0000","172.21.15.116:6832/1649769369":"2024-04-26T00:04:35.418892+0000","172.21.15.116:6833/2340581953":"2024-04-26T00:16:45.587651+0000","172.21.15.174:6837/1889314958":"2024-04-26T00:06:30.441408+0000","172.21.15.116:6832/3645553116":"2024-04-26T00:17:35.601559+0000","172.21.15.116:6833/3645553116":"2024-04-26T00:17:35.601559+0000","172.21.15.116:6833/3841609112":"2024-04-26T00:07:00.448498+0000","172.21.15.116:6834/2012470875":"2024-04-26T00:16:35.584511+0000","172.21.15.116:6835/3336339376":"2024-04-26T00:18:35.616887+0000","172.21.15.174:6837/2276529656":"2024-04-26T00:19:05.629048+0000","172.21.15.116:6832/1655959433":"2024-04-26T00:19:30.633957+0000","172.21.15.116:6833/1655959433":"2024-04-26T00:19:30.633957+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:30:27.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.763+0000 7f4be27a3640 1 -- 172.21.15.116:0/1465892603 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4bc80678c0 msgr2=0x7f4bc8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:27.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.763+0000 7f4be27a3640 1 --2- 172.21.15.116:0/1465892603 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4bc80678c0 0x7f4bc8069d80 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f4bd0000f70 tx=0x7f4bd0009040 comp rx=0 tx=0).stop 2024-04-25T00:30:27.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.763+0000 7f4be27a3640 1 -- 172.21.15.116:0/1465892603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4bdc0a99f0 msgr2=0x7f4bdc14d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:27.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.763+0000 7f4be27a3640 1 --2- 172.21.15.116:0/1465892603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4bdc0a99f0 0x7f4bdc14d760 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc0ae040 tx=0x7f4bcc01ad60 comp rx=0 tx=0).stop 2024-04-25T00:30:27.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.763+0000 7f4be27a3640 1 -- 172.21.15.116:0/1465892603 shutdown_connections 2024-04-25T00:30:27.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.763+0000 7f4be27a3640 1 --2- 172.21.15.116:0/1465892603 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4bc80678c0 0x7f4bc8069d80 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:27.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.763+0000 7f4be27a3640 1 --2- 172.21.15.116:0/1465892603 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4bdc150d20 0x7f4bdc14c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:27.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.763+0000 7f4be27a3640 1 --2- 172.21.15.116:0/1465892603 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4bdc0a99f0 0x7f4bdc14d760 unknown :-1 s=CLOSED pgs=1743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:27.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.763+0000 7f4be27a3640 1 --2- 172.21.15.116:0/1465892603 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4bdc0a94a0 0x7f4bdc0c0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:27.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.763+0000 7f4be27a3640 1 -- 172.21.15.116:0/1465892603 >> 172.21.15.116:0/1465892603 conn(0x7f4bdc0c3b00 msgr2=0x7f4bdc0ac990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:27.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.763+0000 7f4be27a3640 1 -- 172.21.15.116:0/1465892603 shutdown_connections 2024-04-25T00:30:27.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.763+0000 7f4be27a3640 1 -- 172.21.15.116:0/1465892603 wait complete. 2024-04-25T00:30:27.780 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T00:00:53.324376+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'a_metadata', 'create_time': '2024-04-25T00:01:03.197326+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'a'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'a_data', 'create_time': '2024-04-25T00:01:03.728618+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'a'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'b_metadata', 'create_time': '2024-04-25T00:01:17.125210+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'b'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'b_data', 'create_time': '2024-04-25T00:01:17.821873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'b'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T00:30:27.780 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T00:30:27.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.887+0000 7f21b225f640 1 Processor -- start 2024-04-25T00:30:27.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.887+0000 7f21b225f640 1 -- start start 2024-04-25T00:30:27.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.891+0000 7f21b225f640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f21ac0c3ca0 0x7f21ac15a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:27.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.891+0000 7f21b225f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f21ac15b040 0x7f21ac15b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:27.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.891+0000 7f21b225f640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21ac0c3820 con 0x7f21ac153cc0 2024-04-25T00:30:27.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.891+0000 7f21b225f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f21ac0c2a60 con 0x7f21ac15b040 2024-04-25T00:30:27.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.891+0000 7f21b225f640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f21ac0c2d40 con 0x7f21ac0c3ca0 2024-04-25T00:30:27.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.891+0000 7f21abfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f21ac15b040 0x7f21ac15b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:27.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.891+0000 7f21abfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f21ac15b040 0x7f21ac15b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38946/0 (socket says 172.21.15.116:38946) 2024-04-25T00:30:27.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.891+0000 7f21abfff640 1 -- 172.21.15.116:0/1993442642 learned_addr learned my addr 172.21.15.116:0/1993442642 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:27.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.891+0000 7f21aaffd640 1 --2- 172.21.15.116:0/1993442642 >> v2:172.21.15.174:3300/0 conn(0x7f21ac0c3ca0 0x7f21ac15a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:27.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.891+0000 7f21abfff640 1 -- 172.21.15.116:0/1993442642 >> v2:172.21.15.174:3300/0 conn(0x7f21ac0c3ca0 msgr2=0x7f21ac15a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:27.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.891+0000 7f21abfff640 1 --2- 172.21.15.116:0/1993442642 >> v2:172.21.15.174:3300/0 conn(0x7f21ac0c3ca0 0x7f21ac15a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:27.891 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.891+0000 7f21abfff640 1 -- 172.21.15.116:0/1993442642 >> v1:172.21.15.174:6789/0 conn(0x7f21ac153cc0 legacy=0x7f21ac06b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:27.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.891+0000 7f21abfff640 1 -- 172.21.15.116:0/1993442642 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21ac0c3020 con 0x7f21ac15b040 2024-04-25T00:30:27.892 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.891+0000 7f21aaffd640 1 --2- 172.21.15.116:0/1993442642 >> v2:172.21.15.174:3300/0 conn(0x7f21ac0c3ca0 0x7f21ac15a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:30:27.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.891+0000 7f21abfff640 1 --2- 172.21.15.116:0/1993442642 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21ac15b040 0x7f21ac15b4f0 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f219c009870 tx=0x7f219c01c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d495c352964117e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:27.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.895+0000 7f21aa7fc640 1 -- 172.21.15.116:0/1993442642 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f219c0b0070 con 0x7f21ac15b040 2024-04-25T00:30:27.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.895+0000 7f21aa7fc640 1 -- 172.21.15.116:0/1993442642 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f219c0ab600 con 0x7f21ac15b040 2024-04-25T00:30:27.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.895+0000 7f21aa7fc640 1 -- 172.21.15.116:0/1993442642 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f219c0ab9d0 con 0x7f21ac15b040 2024-04-25T00:30:27.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.895+0000 7f21b225f640 1 -- 172.21.15.116:0/1993442642 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21ac15b040 msgr2=0x7f21ac15b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:27.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.895+0000 7f21b225f640 1 --2- 172.21.15.116:0/1993442642 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21ac15b040 0x7f21ac15b4f0 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f219c009870 tx=0x7f219c01c450 comp rx=0 tx=0).stop 2024-04-25T00:30:27.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.895+0000 7f21b225f640 1 -- 172.21.15.116:0/1993442642 shutdown_connections 2024-04-25T00:30:27.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.895+0000 7f21b225f640 1 --2- 172.21.15.116:0/1993442642 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21ac15b040 0x7f21ac15b4f0 unknown :-1 s=CLOSED pgs=1744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:27.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.895+0000 7f21b225f640 1 --2- 172.21.15.116:0/1993442642 >> v2:172.21.15.174:3300/0 conn(0x7f21ac0c3ca0 0x7f21ac15a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:27.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.895+0000 7f21b225f640 1 -- 172.21.15.116:0/1993442642 >> 172.21.15.116:0/1993442642 conn(0x7f21ac0c49c0 msgr2=0x7f21ac0c4dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:27.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.895+0000 7f21b225f640 1 -- 172.21.15.116:0/1993442642 shutdown_connections 2024-04-25T00:30:27.895 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.895+0000 7f21b225f640 1 -- 172.21.15.116:0/1993442642 wait complete. 2024-04-25T00:30:27.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.895+0000 7f21b225f640 1 Processor -- start 2024-04-25T00:30:27.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.895+0000 7f21b225f640 1 -- start start 2024-04-25T00:30:27.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.895+0000 7f21b225f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21ac0c3ca0 0x7f21ac1f5c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:27.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.895+0000 7f21b225f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f21ac153cc0 0x7f21ac1429e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:27.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.895+0000 7f21b225f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f21ac15b040 0x7f21ac142f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:27.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.895+0000 7f21b225f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f21ac06baa0 con 0x7f21ac0c3ca0 2024-04-25T00:30:27.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.895+0000 7f21b225f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f21ac06b920 con 0x7f21ac153cc0 2024-04-25T00:30:27.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.895+0000 7f21b225f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f21ac166cf0 con 0x7f21ac15b040 2024-04-25T00:30:27.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.895+0000 7f21ab7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21ac0c3ca0 0x7f21ac1f5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:27.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.895+0000 7f21ab7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21ac0c3ca0 0x7f21ac1f5c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38962/0 (socket says 172.21.15.116:38962) 2024-04-25T00:30:27.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.895+0000 7f21ab7fe640 1 -- 172.21.15.116:0/2688856427 learned_addr learned my addr 172.21.15.116:0/2688856427 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:27.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.895+0000 7f21abfff640 1 --2- 172.21.15.116:0/2688856427 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f21ac15b040 0x7f21ac142f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:27.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.895+0000 7f21aaffd640 1 --2- 172.21.15.116:0/2688856427 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f21ac153cc0 0x7f21ac1429e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:27.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.895+0000 7f21ab7fe640 1 -- 172.21.15.116:0/2688856427 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f21ac15b040 msgr2=0x7f21ac142f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:27.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.895+0000 7f21ab7fe640 1 --2- 172.21.15.116:0/2688856427 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f21ac15b040 0x7f21ac142f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:27.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.895+0000 7f21ab7fe640 1 -- 172.21.15.116:0/2688856427 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f21ac153cc0 msgr2=0x7f21ac1429e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:27.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.895+0000 7f21ab7fe640 1 --2- 172.21.15.116:0/2688856427 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f21ac153cc0 0x7f21ac1429e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:27.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.895+0000 7f21ab7fe640 1 -- 172.21.15.116:0/2688856427 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21ac1f64e0 con 0x7f21ac0c3ca0 2024-04-25T00:30:27.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.895+0000 7f21aaffd640 1 --2- 172.21.15.116:0/2688856427 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f21ac153cc0 0x7f21ac1429e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:30:27.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.895+0000 7f21ab7fe640 1 --2- 172.21.15.116:0/2688856427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21ac0c3ca0 0x7f21ac1f5c00 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7f21a400ba50 tx=0x7f21a400bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:27.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.895+0000 7f2193fff640 1 -- 172.21.15.116:0/2688856427 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a4019070 con 0x7f21ac0c3ca0 2024-04-25T00:30:27.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.895+0000 7f2193fff640 1 -- 172.21.15.116:0/2688856427 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21a400cc20 con 0x7f21ac0c3ca0 2024-04-25T00:30:27.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.895+0000 7f2193fff640 1 -- 172.21.15.116:0/2688856427 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21a4014440 con 0x7f21ac0c3ca0 2024-04-25T00:30:27.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.895+0000 7f21b225f640 1 -- 172.21.15.116:0/2688856427 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21ac1594a0 con 0x7f21ac0c3ca0 2024-04-25T00:30:27.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.895+0000 7f21b225f640 1 -- 172.21.15.116:0/2688856427 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21ac159a60 con 0x7f21ac0c3ca0 2024-04-25T00:30:27.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.899+0000 7f21b225f640 1 -- 172.21.15.116:0/2688856427 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21ac04f510 con 0x7f21ac0c3ca0 2024-04-25T00:30:27.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.899+0000 7f2193fff640 1 -- 172.21.15.116:0/2688856427 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f21a401c070 con 0x7f21ac0c3ca0 2024-04-25T00:30:27.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.899+0000 7f2193fff640 1 --2- 172.21.15.116:0/2688856427 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f21980678c0 0x7f2198069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:27.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.899+0000 7f2193fff640 1 -- 172.21.15.116:0/2688856427 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7f21a4086c50 con 0x7f21ac0c3ca0 2024-04-25T00:30:27.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.899+0000 7f21aaffd640 1 --2- 172.21.15.116:0/2688856427 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f21980678c0 0x7f2198069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:27.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.899+0000 7f21aaffd640 1 --2- 172.21.15.116:0/2688856427 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f21980678c0 0x7f2198069d80 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f21ac06e900 tx=0x7f21a0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:27.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:27.903+0000 7f2193fff640 1 -- 172.21.15.116:0/2688856427 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f21a408c020 con 0x7f21ac0c3ca0 2024-04-25T00:30:28.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.051+0000 7f21b225f640 1 -- 172.21.15.116:0/2688856427 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f21ac06e930 con 0x7f21ac0c3ca0 2024-04-25T00:30:28.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.051+0000 7f2193fff640 1 -- 172.21.15.116:0/2688856427 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v74) ==== 93+0+10 (secure 0 0 0) 0x7f21a4055e50 con 0x7f21ac0c3ca0 2024-04-25T00:30:28.050 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-04-25T00:30:28.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.051+0000 7f21b225f640 1 -- 172.21.15.116:0/2688856427 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f21980678c0 msgr2=0x7f2198069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:28.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.051+0000 7f21b225f640 1 --2- 172.21.15.116:0/2688856427 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f21980678c0 0x7f2198069d80 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f21ac06e900 tx=0x7f21a0008040 comp rx=0 tx=0).stop 2024-04-25T00:30:28.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.051+0000 7f21b225f640 1 -- 172.21.15.116:0/2688856427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21ac0c3ca0 msgr2=0x7f21ac1f5c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:28.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.051+0000 7f21b225f640 1 --2- 172.21.15.116:0/2688856427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21ac0c3ca0 0x7f21ac1f5c00 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7f21a400ba50 tx=0x7f21a400bf10 comp rx=0 tx=0).stop 2024-04-25T00:30:28.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.051+0000 7f21b225f640 1 -- 172.21.15.116:0/2688856427 shutdown_connections 2024-04-25T00:30:28.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.051+0000 7f21b225f640 1 --2- 172.21.15.116:0/2688856427 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f21980678c0 0x7f2198069d80 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:28.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.051+0000 7f21b225f640 1 --2- 172.21.15.116:0/2688856427 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f21ac15b040 0x7f21ac142f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:28.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.051+0000 7f21b225f640 1 --2- 172.21.15.116:0/2688856427 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f21ac153cc0 0x7f21ac1429e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:28.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.051+0000 7f21b225f640 1 --2- 172.21.15.116:0/2688856427 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f21ac0c3ca0 0x7f21ac1f5c00 unknown :-1 s=CLOSED pgs=1745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:28.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.051+0000 7f21b225f640 1 -- 172.21.15.116:0/2688856427 >> 172.21.15.116:0/2688856427 conn(0x7f21ac0c49c0 msgr2=0x7f21ac06b210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:28.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.051+0000 7f21b225f640 1 -- 172.21.15.116:0/2688856427 shutdown_connections 2024-04-25T00:30:28.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.051+0000 7f21b225f640 1 -- 172.21.15.116:0/2688856427 wait complete. 2024-04-25T00:30:28.068 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_metadata pg_num 2024-04-25T00:30:28.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.187+0000 7f5cf7263640 1 Processor -- start 2024-04-25T00:30:28.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.187+0000 7f5cf7263640 1 -- start start 2024-04-25T00:30:28.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.187+0000 7f5cf7263640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5cf0148f20 0x7f5cf00c2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:28.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.187+0000 7f5cf7263640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f5cf006d510 0x7f5cf0072500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:28.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.187+0000 7f5cf7263640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5cf0149300 con 0x7f5cf006d000 2024-04-25T00:30:28.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.187+0000 7f5cf7263640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f5cf0153350 con 0x7f5cf006d510 2024-04-25T00:30:28.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.187+0000 7f5cf7263640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f5cf00a95d0 con 0x7f5cf0148f20 2024-04-25T00:30:28.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.187+0000 7f5cf4fff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5cf0148f20 0x7f5cf00c2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:28.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.187+0000 7f5ceffff640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f5cf006d510 0x7f5cf0072500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:28.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.187+0000 7f5cf4fff640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5cf0148f20 0x7f5cf00c2930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:50712/0 (socket says 172.21.15.116:50712) 2024-04-25T00:30:28.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.187+0000 7f5cf4fff640 1 -- 172.21.15.116:0/203232935 learned_addr learned my addr 172.21.15.116:0/203232935 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:28.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.187+0000 7f5ceffff640 1 -- 172.21.15.116:0/203232935 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5cf0148f20 msgr2=0x7f5cf00c2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:28.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.187+0000 7f5ceffff640 1 --2- 172.21.15.116:0/203232935 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5cf0148f20 0x7f5cf00c2930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:28.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.187+0000 7f5ceffff640 1 -- 172.21.15.116:0/203232935 >> v1:172.21.15.174:6789/0 conn(0x7f5cf006d000 legacy=0x7f5cf0155a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:28.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.187+0000 7f5ceffff640 1 -- 172.21.15.116:0/203232935 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cf0153070 con 0x7f5cf006d510 2024-04-25T00:30:28.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.191+0000 7f5ceffff640 1 --2- 172.21.15.116:0/203232935 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5cf006d510 0x7f5cf0072500 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f5ce40097c0 tx=0x7f5ce401c380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9256645c8287f67 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:28.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.191+0000 7f5cef7fe640 1 -- 172.21.15.116:0/203232935 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce40b0070 con 0x7f5cf006d510 2024-04-25T00:30:28.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.191+0000 7f5cef7fe640 1 -- 172.21.15.116:0/203232935 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ce40ab600 con 0x7f5cf006d510 2024-04-25T00:30:28.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.191+0000 7f5cef7fe640 1 -- 172.21.15.116:0/203232935 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce40ab9d0 con 0x7f5cf006d510 2024-04-25T00:30:28.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.191+0000 7f5cf7263640 1 -- 172.21.15.116:0/203232935 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5cf006d510 msgr2=0x7f5cf0072500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:28.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.191+0000 7f5cf7263640 1 --2- 172.21.15.116:0/203232935 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5cf006d510 0x7f5cf0072500 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f5ce40097c0 tx=0x7f5ce401c380 comp rx=0 tx=0).stop 2024-04-25T00:30:28.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.191+0000 7f5cf7263640 1 -- 172.21.15.116:0/203232935 shutdown_connections 2024-04-25T00:30:28.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.191+0000 7f5cf7263640 1 --2- 172.21.15.116:0/203232935 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5cf006d510 0x7f5cf0072500 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:28.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.191+0000 7f5cf7263640 1 --2- 172.21.15.116:0/203232935 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5cf0148f20 0x7f5cf00c2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:28.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.191+0000 7f5cf7263640 1 -- 172.21.15.116:0/203232935 >> 172.21.15.116:0/203232935 conn(0x7f5cf00c3a20 msgr2=0x7f5cf00c3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:28.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.191+0000 7f5cf7263640 1 -- 172.21.15.116:0/203232935 shutdown_connections 2024-04-25T00:30:28.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.191+0000 7f5cf7263640 1 -- 172.21.15.116:0/203232935 wait complete. 2024-04-25T00:30:28.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.191+0000 7f5cf7263640 1 Processor -- start 2024-04-25T00:30:28.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.191+0000 7f5cf7263640 1 -- start start 2024-04-25T00:30:28.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.191+0000 7f5cf7263640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5cf006d000 0x7f5cf01514e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:28.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.191+0000 7f5cf7263640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5cf006d510 0x7f5cf01f0470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:28.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.191+0000 7f5cf7263640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cf0148f20 0x7f5cf01f09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:28.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.191+0000 7f5cf7263640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf00c3850 con 0x7f5cf0148f20 2024-04-25T00:30:28.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.191+0000 7f5cf7263640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf00c36d0 con 0x7f5cf006d000 2024-04-25T00:30:28.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.191+0000 7f5cf7263640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f5cf0070da0 con 0x7f5cf006d510 2024-04-25T00:30:28.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.191+0000 7f5cf5800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cf0148f20 0x7f5cf01f09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:28.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.191+0000 7f5cf5800640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cf0148f20 0x7f5cf01f09b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38970/0 (socket says 172.21.15.116:38970) 2024-04-25T00:30:28.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.191+0000 7f5cf5800640 1 -- 172.21.15.116:0/288865618 learned_addr learned my addr 172.21.15.116:0/288865618 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:28.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.195+0000 7f5ceffff640 1 --2- 172.21.15.116:0/288865618 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5cf006d510 0x7f5cf01f0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:28.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.195+0000 7f5cf4fff640 1 --2- 172.21.15.116:0/288865618 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5cf006d000 0x7f5cf01514e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:28.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.195+0000 7f5cf5800640 1 -- 172.21.15.116:0/288865618 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5cf006d510 msgr2=0x7f5cf01f0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:28.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.195+0000 7f5cf5800640 1 --2- 172.21.15.116:0/288865618 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5cf006d510 0x7f5cf01f0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:28.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.195+0000 7f5cf5800640 1 -- 172.21.15.116:0/288865618 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5cf006d000 msgr2=0x7f5cf01514e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:28.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.195+0000 7f5cf5800640 1 --2- 172.21.15.116:0/288865618 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5cf006d000 0x7f5cf01514e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:28.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.195+0000 7f5cf5800640 1 -- 172.21.15.116:0/288865618 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cf00ab6e0 con 0x7f5cf0148f20 2024-04-25T00:30:28.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.195+0000 7f5cf5800640 1 --2- 172.21.15.116:0/288865618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cf0148f20 0x7f5cf01f09b0 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7f5ce800c7a0 tx=0x7f5ce800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:28.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.195+0000 7f5ced7fa640 1 -- 172.21.15.116:0/288865618 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce8003b80 con 0x7f5cf0148f20 2024-04-25T00:30:28.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.195+0000 7f5ced7fa640 1 -- 172.21.15.116:0/288865618 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ce8003d20 con 0x7f5cf0148f20 2024-04-25T00:30:28.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.195+0000 7f5cf7263640 1 -- 172.21.15.116:0/288865618 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cf01f2bf0 con 0x7f5cf0148f20 2024-04-25T00:30:28.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.195+0000 7f5ced7fa640 1 -- 172.21.15.116:0/288865618 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce800aa90 con 0x7f5cf0148f20 2024-04-25T00:30:28.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.195+0000 7f5cf7263640 1 -- 172.21.15.116:0/288865618 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cf0151f30 con 0x7f5cf0148f20 2024-04-25T00:30:28.196 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.195+0000 7f5cf7263640 1 -- 172.21.15.116:0/288865618 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5cf004f510 con 0x7f5cf0148f20 2024-04-25T00:30:28.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.195+0000 7f5ced7fa640 1 -- 172.21.15.116:0/288865618 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5ce800ac30 con 0x7f5cf0148f20 2024-04-25T00:30:28.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.195+0000 7f5ced7fa640 1 --2- 172.21.15.116:0/288865618 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5cdc0678f0 0x7f5cdc069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:28.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.195+0000 7f5ced7fa640 1 -- 172.21.15.116:0/288865618 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7f5ce8086ee0 con 0x7f5cf0148f20 2024-04-25T00:30:28.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.195+0000 7f5cf4fff640 1 --2- 172.21.15.116:0/288865618 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5cdc0678f0 0x7f5cdc069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:28.197 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.195+0000 7f5cf4fff640 1 --2- 172.21.15.116:0/288865618 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5cdc0678f0 0x7f5cdc069db0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f5cf015dd30 tx=0x7f5ce0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:28.199 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.199+0000 7f5ced7fa640 1 -- 172.21.15.116:0/288865618 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f5cf0149300 con 0x7f5cf0148f20 2024-04-25T00:30:28.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.347+0000 7f5cf7263640 1 -- 172.21.15.116:0/288865618 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"} v 0) -- 0x7f5cf006e2c0 con 0x7f5cf0148f20 2024-04-25T00:30:28.347 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.347+0000 7f5ced7fa640 1 -- 172.21.15.116:0/288865618 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"}]=0 v74) ==== 99+0+11 (secure 0 0 0) 0x7f5ce8056730 con 0x7f5cf0148f20 2024-04-25T00:30:28.347 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-04-25T00:30:28.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.347+0000 7f5cf7263640 1 -- 172.21.15.116:0/288865618 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5cdc0678f0 msgr2=0x7f5cdc069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:28.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.347+0000 7f5cf7263640 1 --2- 172.21.15.116:0/288865618 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5cdc0678f0 0x7f5cdc069db0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f5cf015dd30 tx=0x7f5ce0008040 comp rx=0 tx=0).stop 2024-04-25T00:30:28.349 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.347+0000 7f5cf7263640 1 -- 172.21.15.116:0/288865618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cf0148f20 msgr2=0x7f5cf01f09b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:28.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.347+0000 7f5cf7263640 1 --2- 172.21.15.116:0/288865618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cf0148f20 0x7f5cf01f09b0 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7f5ce800c7a0 tx=0x7f5ce800cc60 comp rx=0 tx=0).stop 2024-04-25T00:30:28.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.347+0000 7f5cf7263640 1 -- 172.21.15.116:0/288865618 shutdown_connections 2024-04-25T00:30:28.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.347+0000 7f5cf7263640 1 --2- 172.21.15.116:0/288865618 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5cdc0678f0 0x7f5cdc069db0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:28.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.347+0000 7f5cf7263640 1 --2- 172.21.15.116:0/288865618 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5cf0148f20 0x7f5cf01f09b0 unknown :-1 s=CLOSED pgs=1746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:28.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.347+0000 7f5cf7263640 1 --2- 172.21.15.116:0/288865618 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5cf006d510 0x7f5cf01f0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:28.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.347+0000 7f5cf7263640 1 --2- 172.21.15.116:0/288865618 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5cf006d000 0x7f5cf01514e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:28.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.347+0000 7f5cf7263640 1 -- 172.21.15.116:0/288865618 >> 172.21.15.116:0/288865618 conn(0x7f5cf00c3a20 msgr2=0x7f5cf0155960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:28.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.351+0000 7f5cf7263640 1 -- 172.21.15.116:0/288865618 shutdown_connections 2024-04-25T00:30:28.350 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.351+0000 7f5cf7263640 1 -- 172.21.15.116:0/288865618 wait complete. 2024-04-25T00:30:28.365 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_data pg_num 2024-04-25T00:30:28.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.483+0000 7f0bdb8d2640 1 Processor -- start 2024-04-25T00:30:28.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.483+0000 7f0bdb8d2640 1 -- start start 2024-04-25T00:30:28.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.483+0000 7f0bdb8d2640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f0bd4150d60 0x7f0bd40c2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:28.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.483+0000 7f0bdb8d2640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0bd40a9cb0 con 0x7f0bd40a9630 2024-04-25T00:30:28.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.483+0000 7f0bdb8d2640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0bd40a9f90 con 0x7f0bd40c2e70 2024-04-25T00:30:28.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.483+0000 7f0bdb8d2640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f0bd40aa270 con 0x7f0bd4150d60 2024-04-25T00:30:28.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.483+0000 7f0bd8e6d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f0bd40a9630 0x7f0bd4156e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:46550/0 (socket says 172.21.15.116:46550) 2024-04-25T00:30:28.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.483+0000 7f0bd8e6d640 1 -- 172.21.15.116:0/2670694481 learned_addr learned my addr 172.21.15.116:0/2670694481 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:30:28.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.483+0000 7f0bcbfff640 1 -- 172.21.15.116:0/2670694481 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 296426019 0 0) 0x7f0bd40a9cb0 con 0x7f0bd40a9630 2024-04-25T00:30:28.486 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.483+0000 7f0bcbfff640 1 -- 172.21.15.116:0/2670694481 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0bc0003530 con 0x7f0bd40a9630 2024-04-25T00:30:28.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.483+0000 7f0bd966e640 1 --2- 172.21.15.116:0/2670694481 >> v2:172.21.15.174:3300/0 conn(0x7f0bd4150d60 0x7f0bd40c2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:28.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.483+0000 7f0bcbfff640 1 -- 172.21.15.116:0/2670694481 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4215463223 0 0) 0x7f0bc0003530 con 0x7f0bd40a9630 2024-04-25T00:30:28.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.483+0000 7f0bcbfff640 1 -- 172.21.15.116:0/2670694481 >> v2:172.21.15.174:3300/0 conn(0x7f0bd4150d60 msgr2=0x7f0bd40c2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:28.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.483+0000 7f0bcbfff640 1 --2- 172.21.15.116:0/2670694481 >> v2:172.21.15.174:3300/0 conn(0x7f0bd4150d60 0x7f0bd40c2930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:28.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.483+0000 7f0bcbfff640 1 -- 172.21.15.116:0/2670694481 >> v1:172.21.15.174:6789/0 conn(0x7f0bd40c2e70 legacy=0x7f0bd4157510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:28.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.483+0000 7f0bcbfff640 1 -- 172.21.15.116:0/2670694481 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bd40aa550 con 0x7f0bd40a9630 2024-04-25T00:30:28.487 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.483+0000 7f0bcbfff640 1 -- 172.21.15.116:0/2670694481 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f0bd0001680 con 0x7f0bd40a9630 2024-04-25T00:30:28.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.483+0000 7f0bcbfff640 1 -- 172.21.15.116:0/2670694481 >> v1:172.21.15.116:6789/0 conn(0x7f0bd40a9630 legacy=0x7f0bd4156e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:28.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.483+0000 7f0bcbfff640 1 --2- 172.21.15.116:0/2670694481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bc00039c0 0x7f0bc0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:28.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.483+0000 7f0bcbfff640 1 --2- 172.21.15.116:0/2670694481 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0bc0004810 0x7f0bc0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:28.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.483+0000 7f0bcbfff640 1 --2- 172.21.15.116:0/2670694481 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0bc00053d0 0x7f0bc000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:28.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.483+0000 7f0bcbfff640 1 -- 172.21.15.116:0/2670694481 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0bd40aa270 con 0x7f0bc00039c0 2024-04-25T00:30:28.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.483+0000 7f0bcbfff640 1 -- 172.21.15.116:0/2670694481 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0bc000c260 con 0x7f0bc00053d0 2024-04-25T00:30:28.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.483+0000 7f0bcbfff640 1 -- 172.21.15.116:0/2670694481 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f0bc000c460 con 0x7f0bc0004810 2024-04-25T00:30:28.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.483+0000 7f0bd966e640 1 --2- 172.21.15.116:0/2670694481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bc00039c0 0x7f0bc0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:28.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.483+0000 7f0bd966e640 1 -- 172.21.15.116:0/2670694481 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0bc0004810 msgr2=0x7f0bc0004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:28.488 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.483+0000 7f0bd966e640 1 --2- 172.21.15.116:0/2670694481 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0bc0004810 0x7f0bc0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:28.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.483+0000 7f0bd9e6f640 1 --2- 172.21.15.116:0/2670694481 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0bc00053d0 0x7f0bc000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:28.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.483+0000 7f0bd966e640 1 -- 172.21.15.116:0/2670694481 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0bc00053d0 msgr2=0x7f0bc000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:28.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.483+0000 7f0bd966e640 1 --2- 172.21.15.116:0/2670694481 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0bc00053d0 0x7f0bc000b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:28.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.483+0000 7f0bd966e640 1 -- 172.21.15.116:0/2670694481 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bc0003530 con 0x7f0bc00039c0 2024-04-25T00:30:28.489 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.483+0000 7f0bd9e6f640 1 -- 172.21.15.116:0/2670694481 reap_dead start 2024-04-25T00:30:28.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.487+0000 7f0bd966e640 1 --2- 172.21.15.116:0/2670694481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bc00039c0 0x7f0bc0003de0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f0bc400ba30 tx=0x7f0bc401c670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea14ecd45f40f52c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:28.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.487+0000 7f0bcbfff640 1 -- 172.21.15.116:0/2670694481 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bc400b600 con 0x7f0bc00039c0 2024-04-25T00:30:28.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.487+0000 7f0bcbfff640 1 -- 172.21.15.116:0/2670694481 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bc40b1070 con 0x7f0bc00039c0 2024-04-25T00:30:28.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.487+0000 7f0bcbfff640 1 -- 172.21.15.116:0/2670694481 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bc40ac850 con 0x7f0bc00039c0 2024-04-25T00:30:28.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.487+0000 7f0bdb8d2640 1 -- 172.21.15.116:0/2670694481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bc00039c0 msgr2=0x7f0bc0003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:28.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.487+0000 7f0bdb8d2640 1 --2- 172.21.15.116:0/2670694481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bc00039c0 0x7f0bc0003de0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f0bc400ba30 tx=0x7f0bc401c670 comp rx=0 tx=0).stop 2024-04-25T00:30:28.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.487+0000 7f0bdb8d2640 1 -- 172.21.15.116:0/2670694481 shutdown_connections 2024-04-25T00:30:28.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.487+0000 7f0bdb8d2640 1 --2- 172.21.15.116:0/2670694481 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bc00039c0 0x7f0bc0003de0 unknown :-1 s=CLOSED pgs=1748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:28.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.487+0000 7f0bdb8d2640 1 -- 172.21.15.116:0/2670694481 >> 172.21.15.116:0/2670694481 conn(0x7f0bd40c3a20 msgr2=0x7f0bd40c3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:28.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.487+0000 7f0bdb8d2640 1 -- 172.21.15.116:0/2670694481 shutdown_connections 2024-04-25T00:30:28.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.487+0000 7f0bdb8d2640 1 -- 172.21.15.116:0/2670694481 wait complete. 2024-04-25T00:30:28.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.487+0000 7f0bdb8d2640 1 Processor -- start 2024-04-25T00:30:28.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.487+0000 7f0bdb8d2640 1 -- start start 2024-04-25T00:30:28.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.487+0000 7f0bdb8d2640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0bc00039c0 0x7f0bd4140ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:28.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.487+0000 7f0bdb8d2640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bd414f840 0x7f0bd40c4b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:28.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.487+0000 7f0bdb8d2640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0bd40c5050 0x7f0bd413faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:28.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.487+0000 7f0bdb8d2640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0bd40ab7a0 con 0x7f0bd414f840 2024-04-25T00:30:28.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.487+0000 7f0bdb8d2640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0bd40721e0 con 0x7f0bc00039c0 2024-04-25T00:30:28.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.487+0000 7f0bdb8d2640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f0bd406f010 con 0x7f0bd40c5050 2024-04-25T00:30:28.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.487+0000 7f0bd8e6d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bd414f840 0x7f0bd40c4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:28.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.487+0000 7f0bd8e6d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bd414f840 0x7f0bd40c4b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:38996/0 (socket says 172.21.15.116:38996) 2024-04-25T00:30:28.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.487+0000 7f0bd8e6d640 1 -- 172.21.15.116:0/1097932782 learned_addr learned my addr 172.21.15.116:0/1097932782 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:28.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.487+0000 7f0bd9e6f640 1 --2- 172.21.15.116:0/1097932782 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0bd40c5050 0x7f0bd413faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:28.494 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.487+0000 7f0bd966e640 1 --2- 172.21.15.116:0/1097932782 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0bc00039c0 0x7f0bd4140ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:28.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.487+0000 7f0bd8e6d640 1 -- 172.21.15.116:0/1097932782 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0bd40c5050 msgr2=0x7f0bd413faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:28.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.487+0000 7f0bd8e6d640 1 --2- 172.21.15.116:0/1097932782 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0bd40c5050 0x7f0bd413faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:28.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.487+0000 7f0bd8e6d640 1 -- 172.21.15.116:0/1097932782 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0bc00039c0 msgr2=0x7f0bd4140ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:28.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.487+0000 7f0bd8e6d640 1 --2- 172.21.15.116:0/1097932782 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0bc00039c0 0x7f0bd4140ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:28.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.487+0000 7f0bd8e6d640 1 -- 172.21.15.116:0/1097932782 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bd41eb3e0 con 0x7f0bd414f840 2024-04-25T00:30:28.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.491+0000 7f0bd8e6d640 1 --2- 172.21.15.116:0/1097932782 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bd414f840 0x7f0bd40c4b10 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f0bd000c690 tx=0x7f0bd000cb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:28.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.491+0000 7f0bc9ffb640 1 -- 172.21.15.116:0/1097932782 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bd0018070 con 0x7f0bd414f840 2024-04-25T00:30:28.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.491+0000 7f0bc9ffb640 1 -- 172.21.15.116:0/1097932782 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bd00072d0 con 0x7f0bd414f840 2024-04-25T00:30:28.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.491+0000 7f0bdb8d2640 1 -- 172.21.15.116:0/1097932782 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bd41ea620 con 0x7f0bd414f840 2024-04-25T00:30:28.495 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.491+0000 7f0bc9ffb640 1 -- 172.21.15.116:0/1097932782 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bd00047f0 con 0x7f0bd414f840 2024-04-25T00:30:28.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.491+0000 7f0bdb8d2640 1 -- 172.21.15.116:0/1097932782 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bd41eabe0 con 0x7f0bd414f840 2024-04-25T00:30:28.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.491+0000 7f0bdb8d2640 1 -- 172.21.15.116:0/1097932782 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0bd404f510 con 0x7f0bd414f840 2024-04-25T00:30:28.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.491+0000 7f0bc9ffb640 1 -- 172.21.15.116:0/1097932782 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f0bd001b070 con 0x7f0bd414f840 2024-04-25T00:30:28.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.491+0000 7f0bc9ffb640 1 --2- 172.21.15.116:0/1097932782 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0bb80678f0 0x7f0bb8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:28.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.491+0000 7f0bc9ffb640 1 -- 172.21.15.116:0/1097932782 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7f0bd0086430 con 0x7f0bd414f840 2024-04-25T00:30:28.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.491+0000 7f0bd966e640 1 --2- 172.21.15.116:0/1097932782 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0bb80678f0 0x7f0bb8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:28.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.491+0000 7f0bd966e640 1 --2- 172.21.15.116:0/1097932782 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0bb80678f0 0x7f0bb8069db0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f0bc40b0000 tx=0x7f0bc40046d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:28.496 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.495+0000 7f0bc9ffb640 1 -- 172.21.15.116:0/1097932782 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f0bd0055c80 con 0x7f0bd414f840 2024-04-25T00:30:28.652 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.651+0000 7f0bdb8d2640 1 -- 172.21.15.116:0/1097932782 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"} v 0) -- 0x7f0bd4071db0 con 0x7f0bd414f840 2024-04-25T00:30:28.653 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.651+0000 7f0bc9ffb640 1 -- 172.21.15.116:0/1097932782 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"}]=0 v74) ==== 95+0+11 (secure 0 0 0) 0x7f0bd0055580 con 0x7f0bd414f840 2024-04-25T00:30:28.653 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-04-25T00:30:28.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.655+0000 7f0bdb8d2640 1 -- 172.21.15.116:0/1097932782 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0bb80678f0 msgr2=0x7f0bb8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:28.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.655+0000 7f0bdb8d2640 1 --2- 172.21.15.116:0/1097932782 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0bb80678f0 0x7f0bb8069db0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f0bc40b0000 tx=0x7f0bc40046d0 comp rx=0 tx=0).stop 2024-04-25T00:30:28.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.655+0000 7f0bdb8d2640 1 -- 172.21.15.116:0/1097932782 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bd414f840 msgr2=0x7f0bd40c4b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:28.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.655+0000 7f0bdb8d2640 1 --2- 172.21.15.116:0/1097932782 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bd414f840 0x7f0bd40c4b10 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f0bd000c690 tx=0x7f0bd000cb50 comp rx=0 tx=0).stop 2024-04-25T00:30:28.655 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.655+0000 7f0bdb8d2640 1 -- 172.21.15.116:0/1097932782 shutdown_connections 2024-04-25T00:30:28.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.655+0000 7f0bdb8d2640 1 --2- 172.21.15.116:0/1097932782 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0bb80678f0 0x7f0bb8069db0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:28.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.655+0000 7f0bdb8d2640 1 --2- 172.21.15.116:0/1097932782 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0bd40c5050 0x7f0bd413faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:28.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.655+0000 7f0bdb8d2640 1 --2- 172.21.15.116:0/1097932782 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0bd414f840 0x7f0bd40c4b10 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:28.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.655+0000 7f0bdb8d2640 1 --2- 172.21.15.116:0/1097932782 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0bc00039c0 0x7f0bd4140ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:28.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.655+0000 7f0bdb8d2640 1 -- 172.21.15.116:0/1097932782 >> 172.21.15.116:0/1097932782 conn(0x7f0bd40c3a20 msgr2=0x7f0bd40abf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:28.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.655+0000 7f0bdb8d2640 1 -- 172.21.15.116:0/1097932782 shutdown_connections 2024-04-25T00:30:28.656 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.655+0000 7f0bdb8d2640 1 -- 172.21.15.116:0/1097932782 wait complete. 2024-04-25T00:30:28.671 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_metadata pg_num 2024-04-25T00:30:28.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.791+0000 7f682910e640 1 Processor -- start 2024-04-25T00:30:28.792 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.791+0000 7f682910e640 1 -- start start 2024-04-25T00:30:28.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.795+0000 7f682910e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f68240c9260 0x7f68240c9710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:28.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.795+0000 7f682910e640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68240caa70 con 0x7f68240ca1b0 2024-04-25T00:30:28.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.795+0000 7f682910e640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68240cad50 con 0x7f6824153130 2024-04-25T00:30:28.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.795+0000 7f682910e640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f68240ca820 con 0x7f68240c9260 2024-04-25T00:30:28.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.795+0000 7f682359e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f68240c9260 0x7f68240c9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:28.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.795+0000 7f682359e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f68240c9260 0x7f68240c9710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39000/0 (socket says 172.21.15.116:39000) 2024-04-25T00:30:28.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.795+0000 7f682359e640 1 -- 172.21.15.116:0/428642022 learned_addr learned my addr 172.21.15.116:0/428642022 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:28.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.795+0000 7f682359e640 1 -- 172.21.15.116:0/428642022 >> v1:172.21.15.174:6789/0 conn(0x7f6824153130 legacy=0x7f68241562e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:28.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.795+0000 7f682359e640 1 -- 172.21.15.116:0/428642022 >> v1:172.21.15.116:6789/0 conn(0x7f68240ca1b0 legacy=0x7f682406bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:28.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.795+0000 7f682359e640 1 -- 172.21.15.116:0/428642022 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6824155ae0 con 0x7f68240c9260 2024-04-25T00:30:28.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.795+0000 7f682359e640 1 --2- 172.21.15.116:0/428642022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68240c9260 0x7f68240c9710 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f681c009e50 tx=0x7f681c01c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcabedad84554b5f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:28.795 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.795+0000 7f682259c640 1 -- 172.21.15.116:0/428642022 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f681c0a5590 con 0x7f68240c9260 2024-04-25T00:30:28.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.795+0000 7f682259c640 1 -- 172.21.15.116:0/428642022 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f681c0a5730 con 0x7f68240c9260 2024-04-25T00:30:28.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.795+0000 7f682259c640 1 -- 172.21.15.116:0/428642022 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f681c0a5b00 con 0x7f68240c9260 2024-04-25T00:30:28.796 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.795+0000 7f682910e640 1 -- 172.21.15.116:0/428642022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68240c9260 msgr2=0x7f68240c9710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:28.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.795+0000 7f682910e640 1 --2- 172.21.15.116:0/428642022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68240c9260 0x7f68240c9710 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f681c009e50 tx=0x7f681c01c8d0 comp rx=0 tx=0).stop 2024-04-25T00:30:28.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.795+0000 7f682910e640 1 -- 172.21.15.116:0/428642022 shutdown_connections 2024-04-25T00:30:28.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.795+0000 7f682910e640 1 --2- 172.21.15.116:0/428642022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68240c9260 0x7f68240c9710 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:28.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.795+0000 7f682910e640 1 -- 172.21.15.116:0/428642022 >> 172.21.15.116:0/428642022 conn(0x7f68240cb310 msgr2=0x7f68240cb720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:28.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.795+0000 7f682910e640 1 -- 172.21.15.116:0/428642022 shutdown_connections 2024-04-25T00:30:28.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.795+0000 7f682910e640 1 -- 172.21.15.116:0/428642022 wait complete. 2024-04-25T00:30:28.797 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.799+0000 7f682910e640 1 Processor -- start 2024-04-25T00:30:28.798 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.799+0000 7f682910e640 1 -- start start 2024-04-25T00:30:28.799 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.799+0000 7f682910e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68240c9260 0x7f68241f24a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:28.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.799+0000 7f682910e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f68240ca1b0 0x7f68241ee570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:28.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.799+0000 7f682910e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6824153130 0x7f6824147da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:28.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.799+0000 7f682910e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6824163d30 con 0x7f68240c9260 2024-04-25T00:30:28.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.799+0000 7f682910e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6824163bb0 con 0x7f68240ca1b0 2024-04-25T00:30:28.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.799+0000 7f682910e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6824163eb0 con 0x7f6824153130 2024-04-25T00:30:28.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.799+0000 7f6822d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68240c9260 0x7f68241f24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:28.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.799+0000 7f6822d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68240c9260 0x7f68241f24a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39006/0 (socket says 172.21.15.116:39006) 2024-04-25T00:30:28.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.799+0000 7f6822d9d640 1 -- 172.21.15.116:0/3352578489 learned_addr learned my addr 172.21.15.116:0/3352578489 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:28.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.799+0000 7f682359e640 1 --2- 172.21.15.116:0/3352578489 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6824153130 0x7f6824147da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:28.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.799+0000 7f6822d9d640 1 -- 172.21.15.116:0/3352578489 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6824153130 msgr2=0x7f6824147da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:28.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.799+0000 7f6822d9d640 1 --2- 172.21.15.116:0/3352578489 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6824153130 0x7f6824147da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:28.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.799+0000 7f6822d9d640 1 -- 172.21.15.116:0/3352578489 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f68240ca1b0 msgr2=0x7f68241ee570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:28.800 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.799+0000 7f681a59c640 1 --2- 172.21.15.116:0/3352578489 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f68240ca1b0 0x7f68241ee570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:28.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.799+0000 7f6822d9d640 1 --2- 172.21.15.116:0/3352578489 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f68240ca1b0 0x7f68241ee570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:28.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.799+0000 7f6822d9d640 1 -- 172.21.15.116:0/3352578489 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68240abb90 con 0x7f68240c9260 2024-04-25T00:30:28.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.799+0000 7f6822d9d640 1 --2- 172.21.15.116:0/3352578489 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68240c9260 0x7f68241f24a0 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7f680c00b410 tx=0x7f680c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:28.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.799+0000 7f681bfff640 1 -- 172.21.15.116:0/3352578489 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f680c015030 con 0x7f68240c9260 2024-04-25T00:30:28.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.799+0000 7f681bfff640 1 -- 172.21.15.116:0/3352578489 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f680c004020 con 0x7f68240c9260 2024-04-25T00:30:28.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.799+0000 7f681bfff640 1 -- 172.21.15.116:0/3352578489 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f680c004450 con 0x7f68240c9260 2024-04-25T00:30:28.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.799+0000 7f682910e640 1 -- 172.21.15.116:0/3352578489 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68241f29e0 con 0x7f68240c9260 2024-04-25T00:30:28.801 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.799+0000 7f682910e640 1 -- 172.21.15.116:0/3352578489 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68240ab5d0 con 0x7f68240c9260 2024-04-25T00:30:28.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.799+0000 7f682910e640 1 -- 172.21.15.116:0/3352578489 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f682404f510 con 0x7f68240c9260 2024-04-25T00:30:28.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.799+0000 7f681bfff640 1 -- 172.21.15.116:0/3352578489 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f680c018070 con 0x7f68240c9260 2024-04-25T00:30:28.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.803+0000 7f681bfff640 1 --2- 172.21.15.116:0/3352578489 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f68040678f0 0x7f6804069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:28.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.803+0000 7f681bfff640 1 -- 172.21.15.116:0/3352578489 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7f680c086bf0 con 0x7f68240c9260 2024-04-25T00:30:28.802 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.803+0000 7f681a59c640 1 --2- 172.21.15.116:0/3352578489 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f68040678f0 0x7f6804069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:28.803 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.803+0000 7f681a59c640 1 --2- 172.21.15.116:0/3352578489 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f68040678f0 0x7f6804069db0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f6814007fd0 tx=0x7f6814009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:28.805 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.807+0000 7f681bfff640 1 -- 172.21.15.116:0/3352578489 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f680c056320 con 0x7f68240c9260 2024-04-25T00:30:28.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.955+0000 7f682910e640 1 -- 172.21.15.116:0/3352578489 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"} v 0) -- 0x7f6824153740 con 0x7f68240c9260 2024-04-25T00:30:28.956 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.955+0000 7f681bfff640 1 -- 172.21.15.116:0/3352578489 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"}]=0 v74) ==== 99+0+11 (secure 0 0 0) 0x7f680c055c20 con 0x7f68240c9260 2024-04-25T00:30:28.956 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-04-25T00:30:28.958 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.959+0000 7f682910e640 1 -- 172.21.15.116:0/3352578489 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f68040678f0 msgr2=0x7f6804069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:28.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.959+0000 7f682910e640 1 --2- 172.21.15.116:0/3352578489 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f68040678f0 0x7f6804069db0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f6814007fd0 tx=0x7f6814009040 comp rx=0 tx=0).stop 2024-04-25T00:30:28.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.959+0000 7f682910e640 1 -- 172.21.15.116:0/3352578489 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68240c9260 msgr2=0x7f68241f24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:28.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.959+0000 7f682910e640 1 --2- 172.21.15.116:0/3352578489 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68240c9260 0x7f68241f24a0 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7f680c00b410 tx=0x7f680c00b8d0 comp rx=0 tx=0).stop 2024-04-25T00:30:28.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.959+0000 7f682910e640 1 -- 172.21.15.116:0/3352578489 shutdown_connections 2024-04-25T00:30:28.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.959+0000 7f682910e640 1 --2- 172.21.15.116:0/3352578489 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f68040678f0 0x7f6804069db0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:28.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.959+0000 7f682910e640 1 --2- 172.21.15.116:0/3352578489 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6824153130 0x7f6824147da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:28.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.959+0000 7f682910e640 1 --2- 172.21.15.116:0/3352578489 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f68240ca1b0 0x7f68241ee570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:28.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.959+0000 7f682910e640 1 --2- 172.21.15.116:0/3352578489 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f68240c9260 0x7f68241f24a0 unknown :-1 s=CLOSED pgs=1752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:28.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.959+0000 7f682910e640 1 -- 172.21.15.116:0/3352578489 >> 172.21.15.116:0/3352578489 conn(0x7f68240cb310 msgr2=0x7f68240cb6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:28.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.959+0000 7f682910e640 1 -- 172.21.15.116:0/3352578489 shutdown_connections 2024-04-25T00:30:28.959 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:28.959+0000 7f682910e640 1 -- 172.21.15.116:0/3352578489 wait complete. 2024-04-25T00:30:28.974 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_data pg_num 2024-04-25T00:30:29.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.087+0000 7fcda031b640 1 Processor -- start 2024-04-25T00:30:29.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.087+0000 7fcda031b640 1 -- start start 2024-04-25T00:30:29.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.087+0000 7fcda031b640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fcd980a9630 0x7fcd98156e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:29.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.087+0000 7fcda031b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcd98157340 0x7fcd98157790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:29.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.087+0000 7fcda031b640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd980a9cb0 con 0x7fcd98150d60 2024-04-25T00:30:29.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.087+0000 7fcda031b640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fcd980a9f90 con 0x7fcd98157340 2024-04-25T00:30:29.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.087+0000 7fcda031b640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fcd980aa270 con 0x7fcd980a9630 2024-04-25T00:30:29.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.087+0000 7fcd9e8b8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcd98157340 0x7fcd98157790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:29.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.087+0000 7fcd9e8b8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcd98157340 0x7fcd98157790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39016/0 (socket says 172.21.15.116:39016) 2024-04-25T00:30:29.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.087+0000 7fcd9e8b8640 1 -- 172.21.15.116:0/2963030307 learned_addr learned my addr 172.21.15.116:0/2963030307 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:29.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.087+0000 7fcd9d8b6640 1 --2- 172.21.15.116:0/2963030307 >> v2:172.21.15.174:3300/0 conn(0x7fcd980a9630 0x7fcd98156e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:29.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.087+0000 7fcd9e8b8640 1 -- 172.21.15.116:0/2963030307 >> v2:172.21.15.174:3300/0 conn(0x7fcd980a9630 msgr2=0x7fcd98156e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:29.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.087+0000 7fcd9e8b8640 1 --2- 172.21.15.116:0/2963030307 >> v2:172.21.15.174:3300/0 conn(0x7fcd980a9630 0x7fcd98156e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:29.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.087+0000 7fcd9e8b8640 1 -- 172.21.15.116:0/2963030307 >> v1:172.21.15.116:6789/0 conn(0x7fcd98150d60 legacy=0x7fcd980c2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:29.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.087+0000 7fcd9e8b8640 1 -- 172.21.15.116:0/2963030307 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd980aa550 con 0x7fcd98157340 2024-04-25T00:30:29.090 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.091+0000 7fcd9e8b8640 1 --2- 172.21.15.116:0/2963030307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd98157340 0x7fcd98157790 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7fcd980c1350 tx=0x7fcd9001c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=116e4b8a7a8a0911 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:29.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.091+0000 7fcd9d0b5640 1 -- 172.21.15.116:0/2963030307 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd900a5690 con 0x7fcd98157340 2024-04-25T00:30:29.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.091+0000 7fcd9d0b5640 1 -- 172.21.15.116:0/2963030307 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd900a5830 con 0x7fcd98157340 2024-04-25T00:30:29.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.091+0000 7fcd9d0b5640 1 -- 172.21.15.116:0/2963030307 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd900a5c00 con 0x7fcd98157340 2024-04-25T00:30:29.091 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.091+0000 7fcda031b640 1 -- 172.21.15.116:0/2963030307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd98157340 msgr2=0x7fcd98157790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:29.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.091+0000 7fcda031b640 1 --2- 172.21.15.116:0/2963030307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd98157340 0x7fcd98157790 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7fcd980c1350 tx=0x7fcd9001c8a0 comp rx=0 tx=0).stop 2024-04-25T00:30:29.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.091+0000 7fcda031b640 1 -- 172.21.15.116:0/2963030307 shutdown_connections 2024-04-25T00:30:29.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.091+0000 7fcda031b640 1 --2- 172.21.15.116:0/2963030307 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd98157340 0x7fcd98157790 unknown :-1 s=CLOSED pgs=1754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:29.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.091+0000 7fcda031b640 1 --2- 172.21.15.116:0/2963030307 >> v2:172.21.15.174:3300/0 conn(0x7fcd980a9630 0x7fcd98156e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:29.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.091+0000 7fcda031b640 1 -- 172.21.15.116:0/2963030307 >> 172.21.15.116:0/2963030307 conn(0x7fcd980c3a20 msgr2=0x7fcd980c3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:29.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.091+0000 7fcda031b640 1 -- 172.21.15.116:0/2963030307 shutdown_connections 2024-04-25T00:30:29.092 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.091+0000 7fcda031b640 1 -- 172.21.15.116:0/2963030307 wait complete. 2024-04-25T00:30:29.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.091+0000 7fcda031b640 1 Processor -- start 2024-04-25T00:30:29.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.091+0000 7fcda031b640 1 -- start start 2024-04-25T00:30:29.094 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.091+0000 7fcda031b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcd980a9630 0x7fcd98142000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:29.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.091+0000 7fcda031b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcd98150d60 0x7fcd981ebe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:29.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.091+0000 7fcda031b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd98157340 0x7fcd981ec380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:29.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.091+0000 7fcda031b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcd98072150 con 0x7fcd98157340 2024-04-25T00:30:29.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.091+0000 7fcda031b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcd98071fd0 con 0x7fcd980a9630 2024-04-25T00:30:29.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.091+0000 7fcda031b640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fcd98160770 con 0x7fcd98150d60 2024-04-25T00:30:29.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.095+0000 7fcd9e8b8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd98157340 0x7fcd981ec380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:29.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.095+0000 7fcd9e8b8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd98157340 0x7fcd981ec380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39032/0 (socket says 172.21.15.116:39032) 2024-04-25T00:30:29.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.095+0000 7fcd9e8b8640 1 -- 172.21.15.116:0/1084792698 learned_addr learned my addr 172.21.15.116:0/1084792698 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:29.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.095+0000 7fcd9e8b8640 1 -- 172.21.15.116:0/1084792698 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcd98150d60 msgr2=0x7fcd981ebe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:29.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.095+0000 7fcd9e8b8640 1 --2- 172.21.15.116:0/1084792698 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcd98150d60 0x7fcd981ebe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:29.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.095+0000 7fcd9e0b7640 1 --2- 172.21.15.116:0/1084792698 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcd980a9630 0x7fcd98142000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:29.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.095+0000 7fcd9e8b8640 1 -- 172.21.15.116:0/1084792698 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcd980a9630 msgr2=0x7fcd98142000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:29.095 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.095+0000 7fcd9e8b8640 1 --2- 172.21.15.116:0/1084792698 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcd980a9630 0x7fcd98142000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:29.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.095+0000 7fcd9e8b8640 1 -- 172.21.15.116:0/1084792698 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd980c55b0 con 0x7fcd98157340 2024-04-25T00:30:29.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.095+0000 7fcd9e8b8640 1 --2- 172.21.15.116:0/1084792698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd98157340 0x7fcd981ec380 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7fcd900af040 tx=0x7fcd90019640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:29.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.095+0000 7fcd8effd640 1 -- 172.21.15.116:0/1084792698 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd900bc020 con 0x7fcd98157340 2024-04-25T00:30:29.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.095+0000 7fcd8effd640 1 -- 172.21.15.116:0/1084792698 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd900b8670 con 0x7fcd98157340 2024-04-25T00:30:29.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.095+0000 7fcda031b640 1 -- 172.21.15.116:0/1084792698 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd9813f780 con 0x7fcd98157340 2024-04-25T00:30:29.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.095+0000 7fcd8effd640 1 -- 172.21.15.116:0/1084792698 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd900b8a40 con 0x7fcd98157340 2024-04-25T00:30:29.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.095+0000 7fcda031b640 1 -- 172.21.15.116:0/1084792698 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd9813fa20 con 0x7fcd98157340 2024-04-25T00:30:29.096 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.095+0000 7fcda031b640 1 -- 172.21.15.116:0/1084792698 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd9804f510 con 0x7fcd98157340 2024-04-25T00:30:29.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.095+0000 7fcd8effd640 1 -- 172.21.15.116:0/1084792698 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fcd900bf020 con 0x7fcd98157340 2024-04-25T00:30:29.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.095+0000 7fcd8effd640 1 --2- 172.21.15.116:0/1084792698 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcd7c0678c0 0x7fcd7c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:29.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.095+0000 7fcd8effd640 1 -- 172.21.15.116:0/1084792698 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7fcd90121780 con 0x7fcd98157340 2024-04-25T00:30:29.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.095+0000 7fcd9e0b7640 1 --2- 172.21.15.116:0/1084792698 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcd7c0678c0 0x7fcd7c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:29.097 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.095+0000 7fcd9e0b7640 1 --2- 172.21.15.116:0/1084792698 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcd7c0678c0 0x7fcd7c069d80 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fcd94007fd0 tx=0x7fcd94009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:29.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.099+0000 7fcd8effd640 1 -- 172.21.15.116:0/1084792698 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fcd900c6030 con 0x7fcd98157340 2024-04-25T00:30:29.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.251+0000 7fcda031b640 1 -- 172.21.15.116:0/1084792698 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"} v 0) -- 0x7fcd98071db0 con 0x7fcd98157340 2024-04-25T00:30:29.251 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.251+0000 7fcd8effd640 1 -- 172.21.15.116:0/1084792698 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"}]=0 v74) ==== 95+0+11 (secure 0 0 0) 0x7fcd980c55b0 con 0x7fcd98157340 2024-04-25T00:30:29.252 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-04-25T00:30:29.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.255+0000 7fcda031b640 1 -- 172.21.15.116:0/1084792698 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcd7c0678c0 msgr2=0x7fcd7c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:29.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.255+0000 7fcda031b640 1 --2- 172.21.15.116:0/1084792698 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcd7c0678c0 0x7fcd7c069d80 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fcd94007fd0 tx=0x7fcd94009040 comp rx=0 tx=0).stop 2024-04-25T00:30:29.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.255+0000 7fcda031b640 1 -- 172.21.15.116:0/1084792698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd98157340 msgr2=0x7fcd981ec380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:29.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.255+0000 7fcda031b640 1 --2- 172.21.15.116:0/1084792698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd98157340 0x7fcd981ec380 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7fcd900af040 tx=0x7fcd90019640 comp rx=0 tx=0).stop 2024-04-25T00:30:29.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.255+0000 7fcda031b640 1 -- 172.21.15.116:0/1084792698 shutdown_connections 2024-04-25T00:30:29.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.255+0000 7fcda031b640 1 --2- 172.21.15.116:0/1084792698 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcd7c0678c0 0x7fcd7c069d80 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:29.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.255+0000 7fcda031b640 1 --2- 172.21.15.116:0/1084792698 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd98157340 0x7fcd981ec380 unknown :-1 s=CLOSED pgs=1755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:29.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.255+0000 7fcda031b640 1 --2- 172.21.15.116:0/1084792698 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcd98150d60 0x7fcd981ebe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:29.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.255+0000 7fcda031b640 1 --2- 172.21.15.116:0/1084792698 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcd980a9630 0x7fcd98142000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:29.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.255+0000 7fcda031b640 1 -- 172.21.15.116:0/1084792698 >> 172.21.15.116:0/1084792698 conn(0x7fcd980c3a20 msgr2=0x7fcd9815d760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:29.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.255+0000 7fcda031b640 1 -- 172.21.15.116:0/1084792698 shutdown_connections 2024-04-25T00:30:29.255 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.255+0000 7fcda031b640 1 -- 172.21.15.116:0/1084792698 wait complete. 2024-04-25T00:30:29.269 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T00:30:29.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.379+0000 7f9075ed4640 1 Processor -- start 2024-04-25T00:30:29.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.379+0000 7f9075ed4640 1 -- start start 2024-04-25T00:30:29.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.379+0000 7f9075ed4640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f9070150d20 0x7f90700c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:29.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.379+0000 7f9075ed4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f90700a94a0 0x7f9070153ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:29.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.379+0000 7f9075ed4640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90700c3910 con 0x7f90700c2f50 2024-04-25T00:30:29.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.379+0000 7f9075ed4640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f9070154530 con 0x7f9070150d20 2024-04-25T00:30:29.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.379+0000 7f9075ed4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f90700c16e0 con 0x7f90700a94a0 2024-04-25T00:30:29.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.379+0000 7f906effd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f90700c2f50 0x7f90701538e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:46568/0 (socket says 172.21.15.116:46568) 2024-04-25T00:30:29.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.379+0000 7f906effd640 1 -- 172.21.15.116:0/2869531430 learned_addr learned my addr 172.21.15.116:0/2869531430 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:30:29.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.379+0000 7f906ffff640 1 --2- 172.21.15.116:0/2869531430 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f90700a94a0 0x7f9070153ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:29.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.379+0000 7f906f7fe640 1 --2- 172.21.15.116:0/2869531430 >> v2:172.21.15.174:3300/0 conn(0x7f9070150d20 0x7f90700c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:29.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.379+0000 7f906e7fc640 1 -- 172.21.15.116:0/2869531430 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1079111285 0 0) 0x7f90700c3910 con 0x7f90700c2f50 2024-04-25T00:30:29.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.379+0000 7f906e7fc640 1 -- 172.21.15.116:0/2869531430 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9058003530 con 0x7f90700c2f50 2024-04-25T00:30:29.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.379+0000 7f906e7fc640 1 -- 172.21.15.116:0/2869531430 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1268893426 0 0) 0x7f9058003530 con 0x7f90700c2f50 2024-04-25T00:30:29.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.379+0000 7f906e7fc640 1 -- 172.21.15.116:0/2869531430 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f90700a94a0 msgr2=0x7f9070153ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:29.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.379+0000 7f906e7fc640 1 --2- 172.21.15.116:0/2869531430 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f90700a94a0 0x7f9070153ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:29.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.379+0000 7f906e7fc640 1 -- 172.21.15.116:0/2869531430 >> v2:172.21.15.174:3300/0 conn(0x7f9070150d20 msgr2=0x7f90700c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:29.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.379+0000 7f906e7fc640 1 --2- 172.21.15.116:0/2869531430 >> v2:172.21.15.174:3300/0 conn(0x7f9070150d20 0x7f90700c2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:29.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.379+0000 7f906e7fc640 1 -- 172.21.15.116:0/2869531430 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90700c1400 con 0x7f90700c2f50 2024-04-25T00:30:29.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.379+0000 7f906e7fc640 1 -- 172.21.15.116:0/2869531430 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f9064001680 con 0x7f90700c2f50 2024-04-25T00:30:29.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.379+0000 7f906e7fc640 1 -- 172.21.15.116:0/2869531430 >> v1:172.21.15.116:6789/0 conn(0x7f90700c2f50 legacy=0x7f90701538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:29.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.379+0000 7f906e7fc640 1 --2- 172.21.15.116:0/2869531430 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f90700a94a0 0x7f90580039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:29.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.379+0000 7f906e7fc640 1 --2- 172.21.15.116:0/2869531430 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9058003f40 0x7f9058004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:29.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.379+0000 7f906e7fc640 1 --2- 172.21.15.116:0/2869531430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9058004d50 0x7f90580051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:29.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.379+0000 7f906e7fc640 1 -- 172.21.15.116:0/2869531430 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9070154530 con 0x7f9058004d50 2024-04-25T00:30:29.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.379+0000 7f906e7fc640 1 -- 172.21.15.116:0/2869531430 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f90700c16e0 con 0x7f9058003f40 2024-04-25T00:30:29.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.379+0000 7f906e7fc640 1 -- 172.21.15.116:0/2869531430 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f905800b9d0 con 0x7f90700a94a0 2024-04-25T00:30:29.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.379+0000 7f906ffff640 1 --2- 172.21.15.116:0/2869531430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9058004d50 0x7f90580051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:29.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.379+0000 7f906f7fe640 1 --2- 172.21.15.116:0/2869531430 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f90700a94a0 0x7f90580039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:29.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.379+0000 7f906effd640 1 --2- 172.21.15.116:0/2869531430 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9058003f40 0x7f9058004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:29.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.379+0000 7f906ffff640 1 -- 172.21.15.116:0/2869531430 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f90700a94a0 msgr2=0x7f90580039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:29.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.379+0000 7f906ffff640 1 --2- 172.21.15.116:0/2869531430 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f90700a94a0 0x7f90580039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:29.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.379+0000 7f906ffff640 1 -- 172.21.15.116:0/2869531430 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9058003f40 msgr2=0x7f9058004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:29.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.379+0000 7f906ffff640 1 --2- 172.21.15.116:0/2869531430 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9058003f40 0x7f9058004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:29.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.379+0000 7f906ffff640 1 -- 172.21.15.116:0/2869531430 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9058003530 con 0x7f9058004d50 2024-04-25T00:30:29.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.383+0000 7f906ffff640 1 --2- 172.21.15.116:0/2869531430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9058004d50 0x7f90580051d0 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7f9068009fa0 tx=0x7f906801cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b4885737f4318b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:29.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.383+0000 7f906e7fc640 1 -- 172.21.15.116:0/2869531430 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90680b2070 con 0x7f9058004d50 2024-04-25T00:30:29.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.383+0000 7f906e7fc640 1 -- 172.21.15.116:0/2869531430 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90680a5d10 con 0x7f9058004d50 2024-04-25T00:30:29.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.383+0000 7f906e7fc640 1 -- 172.21.15.116:0/2869531430 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9068009bf0 con 0x7f9058004d50 2024-04-25T00:30:29.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.383+0000 7f9075ed4640 1 -- 172.21.15.116:0/2869531430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9058004d50 msgr2=0x7f90580051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:29.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.383+0000 7f9075ed4640 1 --2- 172.21.15.116:0/2869531430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9058004d50 0x7f90580051d0 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7f9068009fa0 tx=0x7f906801cbe0 comp rx=0 tx=0).stop 2024-04-25T00:30:29.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.383+0000 7f906ffff640 1 -- 172.21.15.116:0/2869531430 reap_dead start 2024-04-25T00:30:29.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.383+0000 7f9075ed4640 1 -- 172.21.15.116:0/2869531430 shutdown_connections 2024-04-25T00:30:29.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.383+0000 7f9075ed4640 1 -- 172.21.15.116:0/2869531430 >> 172.21.15.116:0/2869531430 conn(0x7f90700c3b00 msgr2=0x7f90700c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:29.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.383+0000 7f9075ed4640 1 -- 172.21.15.116:0/2869531430 shutdown_connections 2024-04-25T00:30:29.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.383+0000 7f9075ed4640 1 -- 172.21.15.116:0/2869531430 wait complete. 2024-04-25T00:30:29.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.383+0000 7f9075ed4640 1 Processor -- start 2024-04-25T00:30:29.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.383+0000 7f9075ed4640 1 -- start start 2024-04-25T00:30:29.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.383+0000 7f9075ed4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9070148480 0x7f90701ec7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:29.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.383+0000 7f9075ed4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9070153280 0x7f90701ecd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:29.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.383+0000 7f9075ed4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f90701ed270 0x7f9070151400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:29.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.383+0000 7f9075ed4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f907006ec20 con 0x7f9070148480 2024-04-25T00:30:29.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.383+0000 7f9075ed4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f907006e940 con 0x7f9070153280 2024-04-25T00:30:29.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.383+0000 7f9075ed4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f90700a99f0 con 0x7f90701ed270 2024-04-25T00:30:29.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.383+0000 7f906f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9070148480 0x7f90701ec7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:29.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.383+0000 7f906f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9070148480 0x7f90701ec7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:39062/0 (socket says 172.21.15.116:39062) 2024-04-25T00:30:29.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.383+0000 7f906f7fe640 1 -- 172.21.15.116:0/646004033 learned_addr learned my addr 172.21.15.116:0/646004033 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:29.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.383+0000 7f906effd640 1 --2- 172.21.15.116:0/646004033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9070153280 0x7f90701ecd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:29.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.383+0000 7f906ffff640 1 --2- 172.21.15.116:0/646004033 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f90701ed270 0x7f9070151400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:29.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.387+0000 7f906f7fe640 1 -- 172.21.15.116:0/646004033 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f90701ed270 msgr2=0x7f9070151400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:29.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.387+0000 7f906f7fe640 1 --2- 172.21.15.116:0/646004033 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f90701ed270 0x7f9070151400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:29.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.387+0000 7f906f7fe640 1 -- 172.21.15.116:0/646004033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9070153280 msgr2=0x7f90701ecd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:29.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.387+0000 7f906f7fe640 1 --2- 172.21.15.116:0/646004033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9070153280 0x7f90701ecd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:29.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.387+0000 7f906f7fe640 1 -- 172.21.15.116:0/646004033 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f907014d070 con 0x7f9070148480 2024-04-25T00:30:29.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.387+0000 7f906ffff640 1 --2- 172.21.15.116:0/646004033 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f90701ed270 0x7f9070151400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:30:29.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.387+0000 7f906f7fe640 1 --2- 172.21.15.116:0/646004033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9070148480 0x7f90701ec7f0 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f9060011660 tx=0x7f9060011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:29.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.387+0000 7f9057fff640 1 -- 172.21.15.116:0/646004033 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f906000f9c0 con 0x7f9070148480 2024-04-25T00:30:29.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.387+0000 7f9057fff640 1 -- 172.21.15.116:0/646004033 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f906000fb60 con 0x7f9070148480 2024-04-25T00:30:29.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.387+0000 7f9075ed4640 1 -- 172.21.15.116:0/646004033 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f907014d310 con 0x7f9070148480 2024-04-25T00:30:29.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.387+0000 7f9057fff640 1 -- 172.21.15.116:0/646004033 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9060015460 con 0x7f9070148480 2024-04-25T00:30:29.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.387+0000 7f9075ed4640 1 -- 172.21.15.116:0/646004033 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f907014d8d0 con 0x7f9070148480 2024-04-25T00:30:29.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.387+0000 7f9075ed4640 1 -- 172.21.15.116:0/646004033 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9070068350 con 0x7f9070148480 2024-04-25T00:30:29.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.387+0000 7f9057fff640 1 -- 172.21.15.116:0/646004033 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f9060015600 con 0x7f9070148480 2024-04-25T00:30:29.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.387+0000 7f9057fff640 1 --2- 172.21.15.116:0/646004033 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f905c0678c0 0x7f905c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:29.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.387+0000 7f9057fff640 1 -- 172.21.15.116:0/646004033 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7f9060087900 con 0x7f9070148480 2024-04-25T00:30:29.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.387+0000 7f906effd640 1 --2- 172.21.15.116:0/646004033 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f905c0678c0 0x7f905c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:29.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.387+0000 7f906effd640 1 --2- 172.21.15.116:0/646004033 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f905c0678c0 0x7f905c069d80 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f9064001060 tx=0x7f9064009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:29.393 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.391+0000 7f9057fff640 1 -- 172.21.15.116:0/646004033 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f9060019030 con 0x7f9070148480 2024-04-25T00:30:29.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.551+0000 7f9075ed4640 1 -- 172.21.15.116:0/646004033 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f90700ac8b0 con 0x7f9070148480 2024-04-25T00:30:29.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.551+0000 7f9057fff640 1 -- 172.21.15.116:0/646004033 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 92 entries v74) ==== 81+0+5682 (secure 0 0 0) 0x7f90600571f0 con 0x7f9070148480 2024-04-25T00:30:29.551 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/3499878473 2024-04-26T00:30:15.781624+0000 2024-04-25T00:30:29.551 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/3499878473 2024-04-26T00:30:15.781624+0000 2024-04-25T00:30:29.551 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/2915274073 2024-04-26T00:29:50.776462+0000 2024-04-25T00:30:29.551 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6833/3891103804 2024-04-26T00:28:50.767503+0000 2024-04-25T00:30:29.551 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/2371126056 2024-04-26T00:27:35.754759+0000 2024-04-25T00:30:29.551 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/349772087 2024-04-26T00:27:15.748274+0000 2024-04-25T00:30:29.551 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/349772087 2024-04-26T00:27:15.748274+0000 2024-04-25T00:30:29.551 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6833/3693049068 2024-04-26T00:26:45.739776+0000 2024-04-25T00:30:29.552 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6832/3693049068 2024-04-26T00:26:45.739776+0000 2024-04-25T00:30:29.552 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/4181466530 2024-04-26T00:26:15.734586+0000 2024-04-25T00:30:29.552 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/4181466530 2024-04-26T00:26:15.734586+0000 2024-04-25T00:30:29.552 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/492366509 2024-04-26T00:24:15.701887+0000 2024-04-25T00:30:29.552 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6832/2998042514 2024-04-26T00:23:30.691827+0000 2024-04-25T00:30:29.552 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6833/663910035 2024-04-26T00:22:35.677323+0000 2024-04-25T00:30:29.552 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6832/663910035 2024-04-26T00:22:35.677323+0000 2024-04-25T00:30:29.552 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/321466844 2024-04-26T00:22:10.671673+0000 2024-04-25T00:30:29.553 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/980221971 2024-04-26T00:21:40.665115+0000 2024-04-25T00:30:29.553 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/991183935 2024-04-26T00:20:50.654151+0000 2024-04-25T00:30:29.553 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/991183935 2024-04-26T00:20:50.654151+0000 2024-04-25T00:30:29.553 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/321466844 2024-04-26T00:22:10.671673+0000 2024-04-25T00:30:29.553 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/1358527848 2024-04-26T00:11:00.511032+0000 2024-04-25T00:30:29.553 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6832/3841609112 2024-04-26T00:07:00.448498+0000 2024-04-25T00:30:29.553 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/1670061091 2024-04-26T00:09:30.490485+0000 2024-04-25T00:30:29.553 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/1971675962 2024-04-26T00:25:25.721965+0000 2024-04-25T00:30:29.553 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/1233829678 2024-04-26T00:07:30.454393+0000 2024-04-25T00:30:29.553 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/2371126056 2024-04-26T00:27:35.754759+0000 2024-04-25T00:30:29.554 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6834/2883220846 2024-04-26T00:11:50.521053+0000 2024-04-25T00:30:29.554 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/1233829678 2024-04-26T00:07:30.454393+0000 2024-04-25T00:30:29.554 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/3180024481 2024-04-26T00:15:25.567768+0000 2024-04-25T00:30:29.554 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/879531976 2024-04-26T00:20:10.642563+0000 2024-04-25T00:30:29.554 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6833/174781672 2024-04-26T00:02:40.399073+0000 2024-04-25T00:30:29.554 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6835/4215220222 2024-04-26T00:15:40.572498+0000 2024-04-25T00:30:29.555 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/221971311 2024-04-26T00:22:48.898075+0000 2024-04-25T00:30:29.555 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/1281466397 2024-04-26T00:18:20.613347+0000 2024-04-25T00:30:29.555 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6832/3891103804 2024-04-26T00:28:50.767503+0000 2024-04-25T00:30:29.555 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/1358527848 2024-04-26T00:11:00.511032+0000 2024-04-25T00:30:29.555 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/221971311 2024-04-26T00:22:48.898075+0000 2024-04-25T00:30:29.555 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6832/3006671671 2024-04-26T00:09:30.490406+0000 2024-04-25T00:30:29.555 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/980221971 2024-04-26T00:21:40.665115+0000 2024-04-25T00:30:29.555 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/1762356055 2024-04-26T00:03:36.597345+0000 2024-04-25T00:30:29.555 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6833/3006671671 2024-04-26T00:09:30.490406+0000 2024-04-25T00:30:29.556 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/1459177921 2024-04-26T00:03:30.406778+0000 2024-04-25T00:30:29.556 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/462190523 2024-04-26T00:14:55.560782+0000 2024-04-25T00:30:29.556 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/879531976 2024-04-26T00:20:10.642563+0000 2024-04-25T00:30:29.556 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6832/1459177921 2024-04-26T00:03:30.406778+0000 2024-04-25T00:30:29.556 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/1159002662 2024-04-26T00:10:20.502218+0000 2024-04-25T00:30:29.556 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/1762356055 2024-04-26T00:03:36.597345+0000 2024-04-25T00:30:29.557 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6834/4215220222 2024-04-26T00:15:40.572498+0000 2024-04-25T00:30:29.557 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/1971675962 2024-04-26T00:25:25.721965+0000 2024-04-25T00:30:29.557 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/3081407040 2024-04-26T00:17:25.598516+0000 2024-04-25T00:30:29.557 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/1599708526 2024-04-26T00:12:20.526742+0000 2024-04-25T00:30:29.557 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/2276529656 2024-04-26T00:19:05.629048+0000 2024-04-25T00:30:29.557 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/1159002662 2024-04-26T00:10:20.502218+0000 2024-04-25T00:30:29.557 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6833/2998042514 2024-04-26T00:23:30.691827+0000 2024-04-25T00:30:29.557 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/3081407040 2024-04-26T00:17:25.598516+0000 2024-04-25T00:30:29.557 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6835/2883220846 2024-04-26T00:11:50.521053+0000 2024-04-25T00:30:29.557 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/492366509 2024-04-26T00:24:15.701887+0000 2024-04-25T00:30:29.557 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/2004876235 2024-04-26T00:04:45.421611+0000 2024-04-25T00:30:29.558 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/174781672 2024-04-26T00:02:40.399073+0000 2024-04-25T00:30:29.558 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6833/3633657268 2024-04-26T00:25:40.724634+0000 2024-04-25T00:30:29.558 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6832/365773583 2024-04-26T00:14:58.857970+0000 2024-04-25T00:30:29.558 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/1889314958 2024-04-26T00:06:30.441408+0000 2024-04-25T00:30:29.558 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6833/3705654249 2024-04-26T00:12:45.533219+0000 2024-04-25T00:30:29.558 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6832/3705654249 2024-04-26T00:12:45.533219+0000 2024-04-25T00:30:29.558 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/1670061091 2024-04-26T00:09:30.490485+0000 2024-04-25T00:30:29.558 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/1599708526 2024-04-26T00:12:20.526742+0000 2024-04-25T00:30:29.558 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6832/3633657268 2024-04-26T00:25:40.724634+0000 2024-04-25T00:30:29.558 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/3583791822 2024-04-26T00:08:00.460587+0000 2024-04-25T00:30:29.559 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/3583791822 2024-04-26T00:08:00.460587+0000 2024-04-25T00:30:29.559 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/3398690896 2024-04-26T00:13:30.543729+0000 2024-04-25T00:30:29.559 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/3398690896 2024-04-26T00:13:30.543729+0000 2024-04-25T00:30:29.559 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/462190523 2024-04-26T00:14:55.560782+0000 2024-04-25T00:30:29.559 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/3336339376 2024-04-26T00:18:35.616887+0000 2024-04-25T00:30:29.559 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/3180024481 2024-04-26T00:15:25.567768+0000 2024-04-25T00:30:29.559 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6833/365773583 2024-04-26T00:14:58.857970+0000 2024-04-25T00:30:29.559 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/2012470875 2024-04-26T00:16:35.584511+0000 2024-04-25T00:30:29.559 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6833/2004876235 2024-04-26T00:04:45.421611+0000 2024-04-25T00:30:29.559 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/1281466397 2024-04-26T00:18:20.613347+0000 2024-04-25T00:30:29.559 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6832/2340581953 2024-04-26T00:16:45.587651+0000 2024-04-25T00:30:29.560 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/2915274073 2024-04-26T00:29:50.776462+0000 2024-04-25T00:30:29.560 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/1649769369 2024-04-26T00:04:35.418892+0000 2024-04-25T00:30:29.560 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6832/1649769369 2024-04-26T00:04:35.418892+0000 2024-04-25T00:30:29.560 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6833/2340581953 2024-04-26T00:16:45.587651+0000 2024-04-25T00:30:29.560 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/1889314958 2024-04-26T00:06:30.441408+0000 2024-04-25T00:30:29.560 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6832/3645553116 2024-04-26T00:17:35.601559+0000 2024-04-25T00:30:29.560 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6833/3645553116 2024-04-26T00:17:35.601559+0000 2024-04-25T00:30:29.560 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6833/3841609112 2024-04-26T00:07:00.448498+0000 2024-04-25T00:30:29.560 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/2012470875 2024-04-26T00:16:35.584511+0000 2024-04-25T00:30:29.560 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/3336339376 2024-04-26T00:18:35.616887+0000 2024-04-25T00:30:29.561 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/2276529656 2024-04-26T00:19:05.629048+0000 2024-04-25T00:30:29.561 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6832/1655959433 2024-04-26T00:19:30.633957+0000 2024-04-25T00:30:29.561 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6833/1655959433 2024-04-26T00:19:30.633957+0000 2024-04-25T00:30:29.562 INFO:teuthology.orchestra.run.smithi116.stderr:listed 92 entries 2024-04-25T00:30:29.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.551+0000 7f9075ed4640 1 -- 172.21.15.116:0/646004033 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f905c0678c0 msgr2=0x7f905c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:29.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.551+0000 7f9075ed4640 1 --2- 172.21.15.116:0/646004033 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f905c0678c0 0x7f905c069d80 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f9064001060 tx=0x7f9064009040 comp rx=0 tx=0).stop 2024-04-25T00:30:29.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.551+0000 7f9075ed4640 1 -- 172.21.15.116:0/646004033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9070148480 msgr2=0x7f90701ec7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:29.562 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.551+0000 7f9075ed4640 1 --2- 172.21.15.116:0/646004033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9070148480 0x7f90701ec7f0 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f9060011660 tx=0x7f9060011b20 comp rx=0 tx=0).stop 2024-04-25T00:30:29.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.551+0000 7f9075ed4640 1 -- 172.21.15.116:0/646004033 shutdown_connections 2024-04-25T00:30:29.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.551+0000 7f9075ed4640 1 --2- 172.21.15.116:0/646004033 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f905c0678c0 0x7f905c069d80 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:29.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.551+0000 7f9075ed4640 1 --2- 172.21.15.116:0/646004033 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f90701ed270 0x7f9070151400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:29.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.551+0000 7f9075ed4640 1 --2- 172.21.15.116:0/646004033 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9070153280 0x7f90701ecd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:29.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.551+0000 7f9075ed4640 1 --2- 172.21.15.116:0/646004033 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9070148480 0x7f90701ec7f0 unknown :-1 s=CLOSED pgs=1758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:29.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.551+0000 7f9075ed4640 1 -- 172.21.15.116:0/646004033 >> 172.21.15.116:0/646004033 conn(0x7f90700c3b00 msgr2=0x7f90701537e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:29.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.551+0000 7f9075ed4640 1 -- 172.21.15.116:0/646004033 shutdown_connections 2024-04-25T00:30:29.563 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:29.551+0000 7f9075ed4640 1 -- 172.21.15.116:0/646004033 wait complete. 2024-04-25T00:30:29.568 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi116.front.sepia.ceph.com... 2024-04-25T00:30:29.568 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:30:29.569 DEBUG:teuthology.orchestra.run.smithi116:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-25T00:30:29.590 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:30:29.591 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi116.stderr:ceph-fuse[36045]: fuse finished with error 0 and tester_r 0 2024-04-25T00:30:35.692 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi116.front.sepia.ceph.com 2024-04-25T00:30:35.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:30:35.692 DEBUG:teuthology.orchestra.run.smithi116:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-25T00:30:35.710 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-04-25T00:30:35.710 DEBUG:teuthology.orchestra.run.smithi174:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T00:30:35.801 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-04-25T00:30:35.802 DEBUG:teuthology.orchestra.run.smithi174:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T00:30:35.852 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T00:30:35.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:35.983+0000 7f4496970640 1 Processor -- start 2024-04-25T00:30:35.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:35.983+0000 7f4496970640 1 -- start start 2024-04-25T00:30:35.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:35.983+0000 7f4496970640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4490150d20 0x7f44900c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:35.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:35.983+0000 7f4496970640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f44900a94a0 0x7f4490153ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:35.989 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:35.983+0000 7f4496970640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44900c3910 con 0x7f44900c2f50 2024-04-25T00:30:35.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:35.983+0000 7f4496970640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f4490154530 con 0x7f44900a94a0 2024-04-25T00:30:35.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:35.983+0000 7f4496970640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f44900c16e0 con 0x7f4490150d20 2024-04-25T00:30:35.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:35.983+0000 7f4494f0d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f44900a94a0 0x7f4490153ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:35.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:35.983+0000 7f4494f0d640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f44900a94a0 0x7f4490153ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36202/0 (socket says 172.21.15.116:36202) 2024-04-25T00:30:35.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:35.983+0000 7f4494f0d640 1 -- 172.21.15.116:0/3896006532 learned_addr learned my addr 172.21.15.116:0/3896006532 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:35.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:35.983+0000 7f448ffff640 1 --2- 172.21.15.116:0/3896006532 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4490150d20 0x7f44900c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:35.990 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:35.983+0000 7f4494f0d640 1 -- 172.21.15.116:0/3896006532 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4490150d20 msgr2=0x7f44900c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:35.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:35.983+0000 7f4494f0d640 1 --2- 172.21.15.116:0/3896006532 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4490150d20 0x7f44900c2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:35.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:35.983+0000 7f4494f0d640 1 -- 172.21.15.116:0/3896006532 >> v1:172.21.15.174:6789/0 conn(0x7f44900c2f50 legacy=0x7f44901538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:35.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:35.987+0000 7f4494f0d640 1 -- 172.21.15.116:0/3896006532 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44900c1400 con 0x7f44900a94a0 2024-04-25T00:30:35.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:35.987+0000 7f4494f0d640 1 --2- 172.21.15.116:0/3896006532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44900a94a0 0x7f4490153ff0 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f4488009e50 tx=0x7f448801c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6400448d0b50e2d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:35.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:35.987+0000 7f448effd640 1 -- 172.21.15.116:0/3896006532 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44880a5780 con 0x7f44900a94a0 2024-04-25T00:30:35.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:35.987+0000 7f448effd640 1 -- 172.21.15.116:0/3896006532 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44880a5920 con 0x7f44900a94a0 2024-04-25T00:30:35.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:35.987+0000 7f448effd640 1 -- 172.21.15.116:0/3896006532 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44880a5cf0 con 0x7f44900a94a0 2024-04-25T00:30:35.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:35.987+0000 7f4496970640 1 -- 172.21.15.116:0/3896006532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44900a94a0 msgr2=0x7f4490153ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:35.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:35.987+0000 7f4496970640 1 --2- 172.21.15.116:0/3896006532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44900a94a0 0x7f4490153ff0 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f4488009e50 tx=0x7f448801c9f0 comp rx=0 tx=0).stop 2024-04-25T00:30:35.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:35.987+0000 7f4496970640 1 -- 172.21.15.116:0/3896006532 shutdown_connections 2024-04-25T00:30:35.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:35.987+0000 7f4496970640 1 --2- 172.21.15.116:0/3896006532 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f44900a94a0 0x7f4490153ff0 unknown :-1 s=CLOSED pgs=1759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:35.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:35.987+0000 7f4496970640 1 --2- 172.21.15.116:0/3896006532 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4490150d20 0x7f44900c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:35.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:35.987+0000 7f4496970640 1 -- 172.21.15.116:0/3896006532 >> 172.21.15.116:0/3896006532 conn(0x7f44900c3b00 msgr2=0x7f44900c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:35.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:35.987+0000 7f4496970640 1 -- 172.21.15.116:0/3896006532 shutdown_connections 2024-04-25T00:30:35.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:35.987+0000 7f4496970640 1 -- 172.21.15.116:0/3896006532 wait complete. 2024-04-25T00:30:35.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:35.991+0000 7f4496970640 1 Processor -- start 2024-04-25T00:30:35.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:35.991+0000 7f4496970640 1 -- start start 2024-04-25T00:30:35.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:35.991+0000 7f4496970640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f44900a94a0 0x7f4490140510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:35.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:35.991+0000 7f4496970640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f44900c2f50 0x7f449014ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:35.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:35.991+0000 7f4496970640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4490150d20 0x7f449014d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:35.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:35.991+0000 7f4496970640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f44901608b0 con 0x7f4490150d20 2024-04-25T00:30:36.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:35.991+0000 7f4496970640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4490160730 con 0x7f44900c2f50 2024-04-25T00:30:36.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:35.991+0000 7f4496970640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f4490160a30 con 0x7f44900a94a0 2024-04-25T00:30:36.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:35.991+0000 7f4494f0d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4490150d20 0x7f449014d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:36.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:35.991+0000 7f4494f0d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4490150d20 0x7f449014d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36210/0 (socket says 172.21.15.116:36210) 2024-04-25T00:30:36.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:35.991+0000 7f4494f0d640 1 -- 172.21.15.116:0/2151558508 learned_addr learned my addr 172.21.15.116:0/2151558508 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:36.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:35.991+0000 7f448ffff640 1 --2- 172.21.15.116:0/2151558508 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f44900a94a0 0x7f4490140510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:36.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:35.991+0000 7f448f7fe640 1 --2- 172.21.15.116:0/2151558508 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f44900c2f50 0x7f449014ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:36.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:35.991+0000 7f4494f0d640 1 -- 172.21.15.116:0/2151558508 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f44900a94a0 msgr2=0x7f4490140510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:36.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:35.991+0000 7f4494f0d640 1 --2- 172.21.15.116:0/2151558508 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f44900a94a0 0x7f4490140510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:36.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:35.991+0000 7f4494f0d640 1 -- 172.21.15.116:0/2151558508 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f44900c2f50 msgr2=0x7f449014ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:36.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:35.991+0000 7f4494f0d640 1 --2- 172.21.15.116:0/2151558508 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f44900c2f50 0x7f449014ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:36.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:35.991+0000 7f4494f0d640 1 -- 172.21.15.116:0/2151558508 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44901417f0 con 0x7f4490150d20 2024-04-25T00:30:36.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:35.991+0000 7f4494f0d640 1 --2- 172.21.15.116:0/2151558508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4490150d20 0x7f449014d3d0 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7f44880af040 tx=0x7f4488019790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:36.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:35.991+0000 7f448cff9640 1 -- 172.21.15.116:0/2151558508 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44880bc020 con 0x7f4490150d20 2024-04-25T00:30:36.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:35.991+0000 7f448cff9640 1 -- 172.21.15.116:0/2151558508 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44880b8670 con 0x7f4490150d20 2024-04-25T00:30:36.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:35.991+0000 7f4496970640 1 -- 172.21.15.116:0/2151558508 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f449014afd0 con 0x7f4490150d20 2024-04-25T00:30:36.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:35.991+0000 7f448cff9640 1 -- 172.21.15.116:0/2151558508 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44880b8a20 con 0x7f4490150d20 2024-04-25T00:30:36.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:35.991+0000 7f4496970640 1 -- 172.21.15.116:0/2151558508 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4490146c00 con 0x7f4490150d20 2024-04-25T00:30:36.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:35.991+0000 7f4496970640 1 -- 172.21.15.116:0/2151558508 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4490068350 con 0x7f4490150d20 2024-04-25T00:30:36.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:35.991+0000 7f448cff9640 1 -- 172.21.15.116:0/2151558508 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f44880bf020 con 0x7f4490150d20 2024-04-25T00:30:36.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:35.991+0000 7f448cff9640 1 --2- 172.21.15.116:0/2151558508 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f447c0678c0 0x7f447c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:36.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:35.995+0000 7f448cff9640 1 -- 172.21.15.116:0/2151558508 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7f4488121e00 con 0x7f4490150d20 2024-04-25T00:30:36.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:35.995+0000 7f448ffff640 1 --2- 172.21.15.116:0/2151558508 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f447c0678c0 0x7f447c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:36.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:35.995+0000 7f448ffff640 1 --2- 172.21.15.116:0/2151558508 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f447c0678c0 0x7f447c069d80 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f4480006aa0 tx=0x7f4480008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:36.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:35.999+0000 7f448cff9640 1 -- 172.21.15.116:0/2151558508 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f44880c6030 con 0x7f4490150d20 2024-04-25T00:30:36.157 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.155+0000 7f4496970640 1 -- 172.21.15.116:0/2151558508 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f44900c34f0 con 0x7f4490150d20 2024-04-25T00:30:36.159 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.159+0000 7f448cff9640 1 -- 172.21.15.116:0/2151558508 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v74) ==== 74+0+26655 (secure 0 0 0) 0x7f44880165e0 con 0x7f4490150d20 2024-04-25T00:30:36.160 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:30:36.160 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":74,"fsid":"6d6a1aa7-0478-4c83-8057-05917e81f046","created":"2024-04-25T00:00:40.682493+0000","modified":"2024-04-25T00:30:15.781714+0000","last_up_change":"2024-04-25T00:00:51.822174+0000","last_in_change":"2024-04-25T00:00:42.980473+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T00:00:53.324376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-04-25T00:01:03.197326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-04-25T00:01:03.728618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-04-25T00:01:17.125210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-04-25T00:01:17.821873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8b6ddcb2-1ad9-4167-961c-bd6936bb9060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6809","nonce":3751765682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6811","nonce":3751765682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6815","nonce":3751765682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6813","nonce":3751765682}]},"public_addr":"172.21.15.116:6809/3751765682","cluster_addr":"172.21.15.116:6811/3751765682","heartbeat_back_addr":"172.21.15.116:6815/3751765682","heartbeat_front_addr":"172.21.15.116:6813/3751765682","state":["exists","up"]},{"osd":1,"uuid":"a6431768-994f-41da-8853-886f5687fb91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6801","nonce":377879124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6803","nonce":377879124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6807","nonce":377879124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6805","nonce":377879124}]},"public_addr":"172.21.15.116:6801/377879124","cluster_addr":"172.21.15.116:6803/377879124","heartbeat_back_addr":"172.21.15.116:6807/377879124","heartbeat_front_addr":"172.21.15.116:6805/377879124","state":["exists","up"]},{"osd":2,"uuid":"4f69bd40-5a23-45ff-8693-e1de75e1aa9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6817","nonce":1099282267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6819","nonce":1099282267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6823","nonce":1099282267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6821","nonce":1099282267}]},"public_addr":"172.21.15.116:6817/1099282267","cluster_addr":"172.21.15.116:6819/1099282267","heartbeat_back_addr":"172.21.15.116:6823/1099282267","heartbeat_front_addr":"172.21.15.116:6821/1099282267","state":["exists","up"]},{"osd":3,"uuid":"6a0d5344-8b6f-40d3-918d-0f0a4be688e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6825","nonce":1179942661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6827","nonce":1179942661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6831","nonce":1179942661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6829","nonce":1179942661}]},"public_addr":"172.21.15.116:6825/1179942661","cluster_addr":"172.21.15.116:6827/1179942661","heartbeat_back_addr":"172.21.15.116:6831/1179942661","heartbeat_front_addr":"172.21.15.116:6829/1179942661","state":["exists","up"]},{"osd":4,"uuid":"132e92e4-b825-441e-965a-ca00533fa86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6801","nonce":3132444330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6803","nonce":3132444330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6807","nonce":3132444330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6805","nonce":3132444330}]},"public_addr":"172.21.15.174:6801/3132444330","cluster_addr":"172.21.15.174:6803/3132444330","heartbeat_back_addr":"172.21.15.174:6807/3132444330","heartbeat_front_addr":"172.21.15.174:6805/3132444330","state":["exists","up"]},{"osd":5,"uuid":"a2eeb5b6-1b31-43cc-8306-a0adf3ee15bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6809","nonce":881817164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6811","nonce":881817164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6815","nonce":881817164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6813","nonce":881817164}]},"public_addr":"172.21.15.174:6809/881817164","cluster_addr":"172.21.15.174:6811/881817164","heartbeat_back_addr":"172.21.15.174:6815/881817164","heartbeat_front_addr":"172.21.15.174:6813/881817164","state":["exists","up"]},{"osd":6,"uuid":"588e873a-c841-4ba7-90f2-8956c87a9983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6825","nonce":117574251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6827","nonce":117574251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6831","nonce":117574251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6829","nonce":117574251}]},"public_addr":"172.21.15.174:6825/117574251","cluster_addr":"172.21.15.174:6827/117574251","heartbeat_back_addr":"172.21.15.174:6831/117574251","heartbeat_front_addr":"172.21.15.174:6829/117574251","state":["exists","up"]},{"osd":7,"uuid":"3b90f9b0-8a0e-4883-b088-ca1d4012870c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6817","nonce":2543899196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6819","nonce":2543899196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6823","nonce":2543899196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6821","nonce":2543899196}]},"public_addr":"172.21.15.174:6817/2543899196","cluster_addr":"172.21.15.174:6819/2543899196","heartbeat_back_addr":"172.21.15.174:6823/2543899196","heartbeat_front_addr":"172.21.15.174:6821/2543899196","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.293769+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.310034+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.404865+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.549658+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.192174+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.297266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.493649+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.452309+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2c","mappings":[{"from":0,"to":7}]},{"pgid":"5.3","mappings":[{"from":4,"to":5}]},{"pgid":"5.5","mappings":[{"from":4,"to":5}]},{"pgid":"5.14","mappings":[{"from":4,"to":5}]},{"pgid":"5.19","mappings":[{"from":4,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:6835/3499878473":"2024-04-26T00:30:15.781624+0000","172.21.15.116:6834/3499878473":"2024-04-26T00:30:15.781624+0000","172.21.15.174:6836/2915274073":"2024-04-26T00:29:50.776462+0000","172.21.15.116:6833/3891103804":"2024-04-26T00:28:50.767503+0000","172.21.15.116:6834/2371126056":"2024-04-26T00:27:35.754759+0000","172.21.15.174:6837/349772087":"2024-04-26T00:27:15.748274+0000","172.21.15.174:6836/349772087":"2024-04-26T00:27:15.748274+0000","172.21.15.116:6833/3693049068":"2024-04-26T00:26:45.739776+0000","172.21.15.116:6832/3693049068":"2024-04-26T00:26:45.739776+0000","172.21.15.174:6837/4181466530":"2024-04-26T00:26:15.734586+0000","172.21.15.174:6836/4181466530":"2024-04-26T00:26:15.734586+0000","172.21.15.174:6836/492366509":"2024-04-26T00:24:15.701887+0000","172.21.15.116:6832/2998042514":"2024-04-26T00:23:30.691827+0000","172.21.15.116:6833/663910035":"2024-04-26T00:22:35.677323+0000","172.21.15.116:6832/663910035":"2024-04-26T00:22:35.677323+0000","172.21.15.174:6837/321466844":"2024-04-26T00:22:10.671673+0000","172.21.15.116:6835/980221971":"2024-04-26T00:21:40.665115+0000","172.21.15.174:6837/991183935":"2024-04-26T00:20:50.654151+0000","172.21.15.174:6836/991183935":"2024-04-26T00:20:50.654151+0000","172.21.15.174:6836/321466844":"2024-04-26T00:22:10.671673+0000","172.21.15.116:6834/1358527848":"2024-04-26T00:11:00.511032+0000","172.21.15.116:6832/3841609112":"2024-04-26T00:07:00.448498+0000","172.21.15.116:6835/1670061091":"2024-04-26T00:09:30.490485+0000","172.21.15.116:6834/1971675962":"2024-04-26T00:25:25.721965+0000","172.21.15.116:6835/1233829678":"2024-04-26T00:07:30.454393+0000","172.21.15.116:6835/2371126056":"2024-04-26T00:27:35.754759+0000","172.21.15.174:6834/2883220846":"2024-04-26T00:11:50.521053+0000","172.21.15.116:6834/1233829678":"2024-04-26T00:07:30.454393+0000","172.21.15.174:6837/3180024481":"2024-04-26T00:15:25.567768+0000","172.21.15.116:6835/879531976":"2024-04-26T00:20:10.642563+0000","172.21.15.116:6833/174781672":"2024-04-26T00:02:40.399073+0000","172.21.15.174:6835/4215220222":"2024-04-26T00:15:40.572498+0000","172.21.15.116:6834/221971311":"2024-04-26T00:22:48.898075+0000","172.21.15.174:6836/1281466397":"2024-04-26T00:18:20.613347+0000","172.21.15.116:6832/3891103804":"2024-04-26T00:28:50.767503+0000","172.21.15.116:6835/1358527848":"2024-04-26T00:11:00.511032+0000","172.21.15.116:6835/221971311":"2024-04-26T00:22:48.898075+0000","172.21.15.116:6832/3006671671":"2024-04-26T00:09:30.490406+0000","172.21.15.116:6834/980221971":"2024-04-26T00:21:40.665115+0000","172.21.15.174:6836/1762356055":"2024-04-26T00:03:36.597345+0000","172.21.15.116:6833/3006671671":"2024-04-26T00:09:30.490406+0000","172.21.15.116:6834/1459177921":"2024-04-26T00:03:30.406778+0000","172.21.15.116:6834/462190523":"2024-04-26T00:14:55.560782+0000","172.21.15.116:6834/879531976":"2024-04-26T00:20:10.642563+0000","172.21.15.116:6832/1459177921":"2024-04-26T00:03:30.406778+0000","172.21.15.174:6837/1159002662":"2024-04-26T00:10:20.502218+0000","172.21.15.174:6837/1762356055":"2024-04-26T00:03:36.597345+0000","172.21.15.174:6834/4215220222":"2024-04-26T00:15:40.572498+0000","172.21.15.116:6835/1971675962":"2024-04-26T00:25:25.721965+0000","172.21.15.174:6836/3081407040":"2024-04-26T00:17:25.598516+0000","172.21.15.116:6834/1599708526":"2024-04-26T00:12:20.526742+0000","172.21.15.174:6836/2276529656":"2024-04-26T00:19:05.629048+0000","172.21.15.174:6836/1159002662":"2024-04-26T00:10:20.502218+0000","172.21.15.116:6833/2998042514":"2024-04-26T00:23:30.691827+0000","172.21.15.174:6837/3081407040":"2024-04-26T00:17:25.598516+0000","172.21.15.174:6835/2883220846":"2024-04-26T00:11:50.521053+0000","172.21.15.174:6837/492366509":"2024-04-26T00:24:15.701887+0000","172.21.15.116:6835/2004876235":"2024-04-26T00:04:45.421611+0000","172.21.15.116:6835/174781672":"2024-04-26T00:02:40.399073+0000","172.21.15.116:6833/3633657268":"2024-04-26T00:25:40.724634+0000","172.21.15.116:6832/365773583":"2024-04-26T00:14:58.857970+0000","172.21.15.174:6836/1889314958":"2024-04-26T00:06:30.441408+0000","172.21.15.116:6833/3705654249":"2024-04-26T00:12:45.533219+0000","172.21.15.116:6832/3705654249":"2024-04-26T00:12:45.533219+0000","172.21.15.116:6834/1670061091":"2024-04-26T00:09:30.490485+0000","172.21.15.116:6835/1599708526":"2024-04-26T00:12:20.526742+0000","172.21.15.116:6832/3633657268":"2024-04-26T00:25:40.724634+0000","172.21.15.174:6836/3583791822":"2024-04-26T00:08:00.460587+0000","172.21.15.174:6837/3583791822":"2024-04-26T00:08:00.460587+0000","172.21.15.174:6836/3398690896":"2024-04-26T00:13:30.543729+0000","172.21.15.174:6837/3398690896":"2024-04-26T00:13:30.543729+0000","172.21.15.116:6835/462190523":"2024-04-26T00:14:55.560782+0000","172.21.15.116:6834/3336339376":"2024-04-26T00:18:35.616887+0000","172.21.15.174:6836/3180024481":"2024-04-26T00:15:25.567768+0000","172.21.15.116:6833/365773583":"2024-04-26T00:14:58.857970+0000","172.21.15.116:6835/2012470875":"2024-04-26T00:16:35.584511+0000","172.21.15.116:6833/2004876235":"2024-04-26T00:04:45.421611+0000","172.21.15.174:6837/1281466397":"2024-04-26T00:18:20.613347+0000","172.21.15.116:6832/2340581953":"2024-04-26T00:16:45.587651+0000","172.21.15.174:6837/2915274073":"2024-04-26T00:29:50.776462+0000","172.21.15.116:6834/1649769369":"2024-04-26T00:04:35.418892+0000","172.21.15.116:6832/1649769369":"2024-04-26T00:04:35.418892+0000","172.21.15.116:6833/2340581953":"2024-04-26T00:16:45.587651+0000","172.21.15.174:6837/1889314958":"2024-04-26T00:06:30.441408+0000","172.21.15.116:6832/3645553116":"2024-04-26T00:17:35.601559+0000","172.21.15.116:6833/3645553116":"2024-04-26T00:17:35.601559+0000","172.21.15.116:6833/3841609112":"2024-04-26T00:07:00.448498+0000","172.21.15.116:6834/2012470875":"2024-04-26T00:16:35.584511+0000","172.21.15.116:6835/3336339376":"2024-04-26T00:18:35.616887+0000","172.21.15.174:6837/2276529656":"2024-04-26T00:19:05.629048+0000","172.21.15.116:6832/1655959433":"2024-04-26T00:19:30.633957+0000","172.21.15.116:6833/1655959433":"2024-04-26T00:19:30.633957+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:30:36.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.163+0000 7f4496970640 1 -- 172.21.15.116:0/2151558508 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f447c0678c0 msgr2=0x7f447c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:36.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.163+0000 7f4496970640 1 --2- 172.21.15.116:0/2151558508 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f447c0678c0 0x7f447c069d80 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f4480006aa0 tx=0x7f4480008040 comp rx=0 tx=0).stop 2024-04-25T00:30:36.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.163+0000 7f4496970640 1 -- 172.21.15.116:0/2151558508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4490150d20 msgr2=0x7f449014d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:36.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.163+0000 7f4496970640 1 --2- 172.21.15.116:0/2151558508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4490150d20 0x7f449014d3d0 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7f44880af040 tx=0x7f4488019790 comp rx=0 tx=0).stop 2024-04-25T00:30:36.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.163+0000 7f4496970640 1 -- 172.21.15.116:0/2151558508 shutdown_connections 2024-04-25T00:30:36.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.163+0000 7f4496970640 1 --2- 172.21.15.116:0/2151558508 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f447c0678c0 0x7f447c069d80 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:36.162 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.163+0000 7f4496970640 1 --2- 172.21.15.116:0/2151558508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4490150d20 0x7f449014d3d0 unknown :-1 s=CLOSED pgs=1760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:36.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.163+0000 7f4496970640 1 --2- 172.21.15.116:0/2151558508 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f44900c2f50 0x7f449014ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:36.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.163+0000 7f4496970640 1 --2- 172.21.15.116:0/2151558508 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f44900a94a0 0x7f4490140510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:36.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.163+0000 7f4496970640 1 -- 172.21.15.116:0/2151558508 >> 172.21.15.116:0/2151558508 conn(0x7f44900c3b00 msgr2=0x7f4490144150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:36.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.163+0000 7f4496970640 1 -- 172.21.15.116:0/2151558508 shutdown_connections 2024-04-25T00:30:36.163 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.163+0000 7f4496970640 1 -- 172.21.15.116:0/2151558508 wait complete. 2024-04-25T00:30:36.178 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T00:00:53.324376+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'a_metadata', 'create_time': '2024-04-25T00:01:03.197326+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'a'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'a_data', 'create_time': '2024-04-25T00:01:03.728618+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'a'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'b_metadata', 'create_time': '2024-04-25T00:01:17.125210+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'b'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'b_data', 'create_time': '2024-04-25T00:01:17.821873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'b'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T00:30:36.179 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T00:30:36.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.299+0000 7faf4efbe640 1 Processor -- start 2024-04-25T00:30:36.301 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.299+0000 7faf4efbe640 1 -- start start 2024-04-25T00:30:36.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.303+0000 7faf4efbe640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7faf48150d60 0x7faf480c2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:36.303 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.303+0000 7faf4efbe640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf480a9cb0 con 0x7faf480c2e70 2024-04-25T00:30:36.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.303+0000 7faf4efbe640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf480a9f90 con 0x7faf480a9630 2024-04-25T00:30:36.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.303+0000 7faf4efbe640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7faf480aa270 con 0x7faf48150d60 2024-04-25T00:30:36.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.303+0000 7faf4d55b640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7faf480c2e70 0x7faf48157510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57340/0 (socket says 172.21.15.116:57340) 2024-04-25T00:30:36.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.303+0000 7faf4d55b640 1 -- 172.21.15.116:0/3283459989 learned_addr learned my addr 172.21.15.116:0/3283459989 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:30:36.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.303+0000 7faf477fe640 1 -- 172.21.15.116:0/3283459989 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1721882930 0 0) 0x7faf480a9cb0 con 0x7faf480c2e70 2024-04-25T00:30:36.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.303+0000 7faf477fe640 1 -- 172.21.15.116:0/3283459989 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faf30003530 con 0x7faf480c2e70 2024-04-25T00:30:36.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.303+0000 7faf4cd5a640 1 --2- 172.21.15.116:0/3283459989 >> v2:172.21.15.174:3300/0 conn(0x7faf48150d60 0x7faf480c2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:36.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.303+0000 7faf477fe640 1 -- 172.21.15.116:0/3283459989 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3118313891 0 0) 0x7faf30003530 con 0x7faf480c2e70 2024-04-25T00:30:36.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.303+0000 7faf477fe640 1 -- 172.21.15.116:0/3283459989 >> v2:172.21.15.174:3300/0 conn(0x7faf48150d60 msgr2=0x7faf480c2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:36.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.303+0000 7faf477fe640 1 --2- 172.21.15.116:0/3283459989 >> v2:172.21.15.174:3300/0 conn(0x7faf48150d60 0x7faf480c2930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:36.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.303+0000 7faf477fe640 1 -- 172.21.15.116:0/3283459989 >> v1:172.21.15.174:6789/0 conn(0x7faf480a9630 legacy=0x7faf48156e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:36.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.303+0000 7faf477fe640 1 -- 172.21.15.116:0/3283459989 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf480aa550 con 0x7faf480c2e70 2024-04-25T00:30:36.304 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.303+0000 7faf477fe640 1 -- 172.21.15.116:0/3283459989 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7faf40001a00 con 0x7faf480c2e70 2024-04-25T00:30:36.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.303+0000 7faf477fe640 1 -- 172.21.15.116:0/3283459989 >> v1:172.21.15.116:6789/0 conn(0x7faf480c2e70 legacy=0x7faf48157510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:36.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.303+0000 7faf477fe640 1 --2- 172.21.15.116:0/3283459989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf300039c0 0x7faf30003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:36.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.303+0000 7faf477fe640 1 --2- 172.21.15.116:0/3283459989 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faf30004810 0x7faf30004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:36.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.303+0000 7faf477fe640 1 --2- 172.21.15.116:0/3283459989 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf300053d0 0x7faf3000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:36.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.303+0000 7faf477fe640 1 -- 172.21.15.116:0/3283459989 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7faf480aa270 con 0x7faf300039c0 2024-04-25T00:30:36.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.303+0000 7faf477fe640 1 -- 172.21.15.116:0/3283459989 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7faf3000c260 con 0x7faf300053d0 2024-04-25T00:30:36.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.303+0000 7faf477fe640 1 -- 172.21.15.116:0/3283459989 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7faf3000c460 con 0x7faf30004810 2024-04-25T00:30:36.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.303+0000 7faf4cd5a640 1 --2- 172.21.15.116:0/3283459989 >> v2:172.21.15.174:3300/0 conn(0x7faf48150d60 0x7faf480c2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:30:36.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.303+0000 7faf4cd5a640 1 --2- 172.21.15.116:0/3283459989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf300039c0 0x7faf30003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:36.305 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.303+0000 7faf47fff640 1 --2- 172.21.15.116:0/3283459989 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faf30004810 0x7faf30004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:36.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.303+0000 7faf4d55b640 1 --2- 172.21.15.116:0/3283459989 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf300053d0 0x7faf3000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:36.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.303+0000 7faf4cd5a640 1 -- 172.21.15.116:0/3283459989 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faf30004810 msgr2=0x7faf30004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:36.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.303+0000 7faf4cd5a640 1 --2- 172.21.15.116:0/3283459989 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faf30004810 0x7faf30004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:36.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.303+0000 7faf4cd5a640 1 -- 172.21.15.116:0/3283459989 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf300053d0 msgr2=0x7faf3000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:36.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.303+0000 7faf4cd5a640 1 --2- 172.21.15.116:0/3283459989 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf300053d0 0x7faf3000b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:36.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.303+0000 7faf4cd5a640 1 -- 172.21.15.116:0/3283459989 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf30003530 con 0x7faf300039c0 2024-04-25T00:30:36.306 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.303+0000 7faf4d55b640 1 -- 172.21.15.116:0/3283459989 reap_dead start 2024-04-25T00:30:36.308 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.303+0000 7faf4cd5a640 1 --2- 172.21.15.116:0/3283459989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf300039c0 0x7faf30003de0 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7faf380098b0 tx=0x7faf3801c580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=323f693ff930e357 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:36.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.307+0000 7faf477fe640 1 -- 172.21.15.116:0/3283459989 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf3800f040 con 0x7faf300039c0 2024-04-25T00:30:36.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.307+0000 7faf477fe640 1 -- 172.21.15.116:0/3283459989 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf38004020 con 0x7faf300039c0 2024-04-25T00:30:36.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.307+0000 7faf477fe640 1 -- 172.21.15.116:0/3283459989 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf380043f0 con 0x7faf300039c0 2024-04-25T00:30:36.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.307+0000 7faf4efbe640 1 -- 172.21.15.116:0/3283459989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf300039c0 msgr2=0x7faf30003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:36.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.307+0000 7faf4efbe640 1 --2- 172.21.15.116:0/3283459989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf300039c0 0x7faf30003de0 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7faf380098b0 tx=0x7faf3801c580 comp rx=0 tx=0).stop 2024-04-25T00:30:36.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.307+0000 7faf4efbe640 1 -- 172.21.15.116:0/3283459989 shutdown_connections 2024-04-25T00:30:36.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.307+0000 7faf4efbe640 1 --2- 172.21.15.116:0/3283459989 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf300039c0 0x7faf30003de0 unknown :-1 s=CLOSED pgs=1762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:36.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.307+0000 7faf4efbe640 1 -- 172.21.15.116:0/3283459989 >> 172.21.15.116:0/3283459989 conn(0x7faf480c3a20 msgr2=0x7faf480c3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:36.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.307+0000 7faf4efbe640 1 -- 172.21.15.116:0/3283459989 shutdown_connections 2024-04-25T00:30:36.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.307+0000 7faf4efbe640 1 -- 172.21.15.116:0/3283459989 wait complete. 2024-04-25T00:30:36.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.307+0000 7faf4efbe640 1 Processor -- start 2024-04-25T00:30:36.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.307+0000 7faf4efbe640 1 -- start start 2024-04-25T00:30:36.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.307+0000 7faf4efbe640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf300039c0 0x7faf480c05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:36.309 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.307+0000 7faf4efbe640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faf4806e140 0x7faf481f2600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:36.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.307+0000 7faf4efbe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf481f2b40 0x7faf481549c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:36.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.307+0000 7faf4efbe640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7faf480ab7a0 con 0x7faf481f2b40 2024-04-25T00:30:36.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.307+0000 7faf4efbe640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7faf480721e0 con 0x7faf300039c0 2024-04-25T00:30:36.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.307+0000 7faf4efbe640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7faf48157610 con 0x7faf4806e140 2024-04-25T00:30:36.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.307+0000 7faf4d55b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf481f2b40 0x7faf481549c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:36.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.307+0000 7faf4d55b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf481f2b40 0x7faf481549c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36224/0 (socket says 172.21.15.116:36224) 2024-04-25T00:30:36.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.307+0000 7faf4d55b640 1 -- 172.21.15.116:0/1317713153 learned_addr learned my addr 172.21.15.116:0/1317713153 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:36.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.307+0000 7faf47fff640 1 --2- 172.21.15.116:0/1317713153 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faf4806e140 0x7faf481f2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:36.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.307+0000 7faf4cd5a640 1 --2- 172.21.15.116:0/1317713153 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf300039c0 0x7faf480c05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:36.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.307+0000 7faf4d55b640 1 -- 172.21.15.116:0/1317713153 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faf4806e140 msgr2=0x7faf481f2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:36.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.307+0000 7faf4d55b640 1 --2- 172.21.15.116:0/1317713153 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faf4806e140 0x7faf481f2600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:36.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.307+0000 7faf4d55b640 1 -- 172.21.15.116:0/1317713153 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf300039c0 msgr2=0x7faf480c05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:36.310 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.307+0000 7faf4d55b640 1 --2- 172.21.15.116:0/1317713153 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf300039c0 0x7faf480c05b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:36.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.307+0000 7faf4d55b640 1 -- 172.21.15.116:0/1317713153 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf4806d350 con 0x7faf481f2b40 2024-04-25T00:30:36.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.307+0000 7faf4d55b640 1 --2- 172.21.15.116:0/1317713153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf481f2b40 0x7faf481549c0 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7faf48068110 tx=0x7faf40007bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:36.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.307+0000 7faf457fa640 1 -- 172.21.15.116:0/1317713153 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf4000ea00 con 0x7faf481f2b40 2024-04-25T00:30:36.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.307+0000 7faf457fa640 1 -- 172.21.15.116:0/1317713153 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf40018070 con 0x7faf481f2b40 2024-04-25T00:30:36.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.307+0000 7faf4efbe640 1 -- 172.21.15.116:0/1317713153 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf4806d5c0 con 0x7faf481f2b40 2024-04-25T00:30:36.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.307+0000 7faf457fa640 1 -- 172.21.15.116:0/1317713153 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf4000edc0 con 0x7faf481f2b40 2024-04-25T00:30:36.311 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.307+0000 7faf4efbe640 1 -- 172.21.15.116:0/1317713153 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf4806db80 con 0x7faf481f2b40 2024-04-25T00:30:36.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.311+0000 7faf4efbe640 1 -- 172.21.15.116:0/1317713153 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf4804f510 con 0x7faf481f2b40 2024-04-25T00:30:36.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.311+0000 7faf457fa640 1 -- 172.21.15.116:0/1317713153 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7faf4001d460 con 0x7faf481f2b40 2024-04-25T00:30:36.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.311+0000 7faf457fa640 1 --2- 172.21.15.116:0/1317713153 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faf340678c0 0x7faf34069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:36.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.311+0000 7faf457fa640 1 -- 172.21.15.116:0/1317713153 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7faf400871e0 con 0x7faf481f2b40 2024-04-25T00:30:36.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.311+0000 7faf4cd5a640 1 --2- 172.21.15.116:0/1317713153 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faf340678c0 0x7faf34069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:36.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.311+0000 7faf4cd5a640 1 --2- 172.21.15.116:0/1317713153 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faf340678c0 0x7faf34069d80 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7faf380ad000 tx=0x7faf380046d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:36.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.315+0000 7faf457fa640 1 -- 172.21.15.116:0/1317713153 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7faf400569a0 con 0x7faf481f2b40 2024-04-25T00:30:36.475 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.475+0000 7faf4efbe640 1 -- 172.21.15.116:0/1317713153 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7faf4814dc40 con 0x7faf481f2b40 2024-04-25T00:30:36.476 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.475+0000 7faf457fa640 1 -- 172.21.15.116:0/1317713153 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v74) ==== 93+0+10 (secure 0 0 0) 0x7faf400562a0 con 0x7faf481f2b40 2024-04-25T00:30:36.476 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-04-25T00:30:36.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.479+0000 7faf4efbe640 1 -- 172.21.15.116:0/1317713153 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faf340678c0 msgr2=0x7faf34069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:36.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.479+0000 7faf4efbe640 1 --2- 172.21.15.116:0/1317713153 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faf340678c0 0x7faf34069d80 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7faf380ad000 tx=0x7faf380046d0 comp rx=0 tx=0).stop 2024-04-25T00:30:36.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.479+0000 7faf4efbe640 1 -- 172.21.15.116:0/1317713153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf481f2b40 msgr2=0x7faf481549c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:36.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.479+0000 7faf4efbe640 1 --2- 172.21.15.116:0/1317713153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf481f2b40 0x7faf481549c0 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7faf48068110 tx=0x7faf40007bc0 comp rx=0 tx=0).stop 2024-04-25T00:30:36.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.479+0000 7faf4efbe640 1 -- 172.21.15.116:0/1317713153 shutdown_connections 2024-04-25T00:30:36.478 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.479+0000 7faf4efbe640 1 --2- 172.21.15.116:0/1317713153 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faf340678c0 0x7faf34069d80 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:36.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.479+0000 7faf4efbe640 1 --2- 172.21.15.116:0/1317713153 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf481f2b40 0x7faf481549c0 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:36.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.479+0000 7faf4efbe640 1 --2- 172.21.15.116:0/1317713153 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faf4806e140 0x7faf481f2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:36.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.479+0000 7faf4efbe640 1 --2- 172.21.15.116:0/1317713153 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf300039c0 0x7faf480c05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:36.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.479+0000 7faf4efbe640 1 -- 172.21.15.116:0/1317713153 >> 172.21.15.116:0/1317713153 conn(0x7faf480c3a20 msgr2=0x7faf480acb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:36.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.479+0000 7faf4efbe640 1 -- 172.21.15.116:0/1317713153 shutdown_connections 2024-04-25T00:30:36.479 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.479+0000 7faf4efbe640 1 -- 172.21.15.116:0/1317713153 wait complete. 2024-04-25T00:30:36.495 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_metadata pg_num 2024-04-25T00:30:36.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.619+0000 7f7dbc618640 1 Processor -- start 2024-04-25T00:30:36.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.619+0000 7f7dbc618640 1 -- start start 2024-04-25T00:30:36.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.619+0000 7f7dbc618640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f7db40c9710 0x7f7db40c5e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:36.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.619+0000 7f7dbc618640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7db40c8630 con 0x7f7db40c7ad0 2024-04-25T00:30:36.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.619+0000 7f7dbc618640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7db40c8070 con 0x7f7db40c5a70 2024-04-25T00:30:36.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.619+0000 7f7dbc618640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f7db40c9b80 con 0x7f7db40c9710 2024-04-25T00:30:36.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.619+0000 7f7db9bb3640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f7db40c7ad0 0x7f7db415e2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57342/0 (socket says 172.21.15.116:57342) 2024-04-25T00:30:36.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.619+0000 7f7db9bb3640 1 -- 172.21.15.116:0/244306430 learned_addr learned my addr 172.21.15.116:0/244306430 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:30:36.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.619+0000 7f7db93b2640 1 -- 172.21.15.116:0/244306430 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1887098486 0 0) 0x7f7db40c8630 con 0x7f7db40c7ad0 2024-04-25T00:30:36.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.619+0000 7f7dbabb5640 1 --2- 172.21.15.116:0/244306430 >> v2:172.21.15.174:3300/0 conn(0x7f7db40c9710 0x7f7db40c5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:36.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.619+0000 7f7db93b2640 1 -- 172.21.15.116:0/244306430 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7da0003530 con 0x7f7db40c7ad0 2024-04-25T00:30:36.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.619+0000 7f7db93b2640 1 -- 172.21.15.116:0/244306430 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 738893996 0 0) 0x7f7da0003530 con 0x7f7db40c7ad0 2024-04-25T00:30:36.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.619+0000 7f7db93b2640 1 -- 172.21.15.116:0/244306430 >> v2:172.21.15.174:3300/0 conn(0x7f7db40c9710 msgr2=0x7f7db40c5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:36.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.619+0000 7f7db93b2640 1 --2- 172.21.15.116:0/244306430 >> v2:172.21.15.174:3300/0 conn(0x7f7db40c9710 0x7f7db40c5e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:36.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.619+0000 7f7db93b2640 1 -- 172.21.15.116:0/244306430 >> v1:172.21.15.174:6789/0 conn(0x7f7db40c5a70 legacy=0x7f7db406d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:36.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.619+0000 7f7db93b2640 1 -- 172.21.15.116:0/244306430 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7db40c8350 con 0x7f7db40c7ad0 2024-04-25T00:30:36.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.619+0000 7f7db93b2640 1 -- 172.21.15.116:0/244306430 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f7da4001680 con 0x7f7db40c7ad0 2024-04-25T00:30:36.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.619+0000 7f7db93b2640 1 -- 172.21.15.116:0/244306430 >> v1:172.21.15.116:6789/0 conn(0x7f7db40c7ad0 legacy=0x7f7db415e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:36.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.619+0000 7f7db93b2640 1 --2- 172.21.15.116:0/244306430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7da00039c0 0x7f7da0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:36.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.619+0000 7f7db93b2640 1 --2- 172.21.15.116:0/244306430 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7da0004810 0x7f7da0004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:36.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.619+0000 7f7db93b2640 1 --2- 172.21.15.116:0/244306430 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7da00051d0 0x7f7da000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:36.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.619+0000 7f7db93b2640 1 -- 172.21.15.116:0/244306430 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7db40c9b80 con 0x7f7da00039c0 2024-04-25T00:30:36.621 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.619+0000 7f7db93b2640 1 -- 172.21.15.116:0/244306430 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7da000c170 con 0x7f7da0004810 2024-04-25T00:30:36.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.619+0000 7f7db93b2640 1 -- 172.21.15.116:0/244306430 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f7da000c370 con 0x7f7da00051d0 2024-04-25T00:30:36.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.619+0000 7f7dba3b4640 1 --2- 172.21.15.116:0/244306430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7da00039c0 0x7f7da0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:36.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.619+0000 7f7dbabb5640 1 --2- 172.21.15.116:0/244306430 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7da00051d0 0x7f7da000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:36.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.619+0000 7f7dba3b4640 1 -- 172.21.15.116:0/244306430 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7da00051d0 msgr2=0x7f7da000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:36.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.619+0000 7f7dba3b4640 1 --2- 172.21.15.116:0/244306430 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7da00051d0 0x7f7da000b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:36.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.619+0000 7f7dba3b4640 1 -- 172.21.15.116:0/244306430 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7da0004810 msgr2=0x7f7da0004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:36.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.619+0000 7f7dba3b4640 1 --2- 172.21.15.116:0/244306430 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7da0004810 0x7f7da0004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:36.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.619+0000 7f7dbabb5640 1 -- 172.21.15.116:0/244306430 reap_dead start 2024-04-25T00:30:36.622 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.619+0000 7f7dba3b4640 1 -- 172.21.15.116:0/244306430 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7da0003530 con 0x7f7da00039c0 2024-04-25T00:30:36.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.623+0000 7f7dba3b4640 1 --2- 172.21.15.116:0/244306430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7da00039c0 0x7f7da0003de0 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f7db0001780 tx=0x7f7db001c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7d2d8ac67d6a74b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:36.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.623+0000 7f7db93b2640 1 -- 172.21.15.116:0/244306430 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7db00013a0 con 0x7f7da00039c0 2024-04-25T00:30:36.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.623+0000 7f7db93b2640 1 -- 172.21.15.116:0/244306430 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7db00052f0 con 0x7f7da00039c0 2024-04-25T00:30:36.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.623+0000 7f7db93b2640 1 -- 172.21.15.116:0/244306430 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7db0003840 con 0x7f7da00039c0 2024-04-25T00:30:36.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.623+0000 7f7dbc618640 1 -- 172.21.15.116:0/244306430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7da00039c0 msgr2=0x7f7da0003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:36.623 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.623+0000 7f7dbc618640 1 --2- 172.21.15.116:0/244306430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7da00039c0 0x7f7da0003de0 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f7db0001780 tx=0x7f7db001c400 comp rx=0 tx=0).stop 2024-04-25T00:30:36.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.623+0000 7f7dbc618640 1 -- 172.21.15.116:0/244306430 shutdown_connections 2024-04-25T00:30:36.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.623+0000 7f7dbc618640 1 --2- 172.21.15.116:0/244306430 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7da00039c0 0x7f7da0003de0 unknown :-1 s=CLOSED pgs=1765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:36.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.623+0000 7f7dbc618640 1 -- 172.21.15.116:0/244306430 >> 172.21.15.116:0/244306430 conn(0x7f7db40cb0f0 msgr2=0x7f7db40cb500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:36.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.623+0000 7f7dbc618640 1 -- 172.21.15.116:0/244306430 shutdown_connections 2024-04-25T00:30:36.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.623+0000 7f7dbc618640 1 -- 172.21.15.116:0/244306430 wait complete. 2024-04-25T00:30:36.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.623+0000 7f7dbc618640 1 Processor -- start 2024-04-25T00:30:36.624 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.623+0000 7f7dbc618640 1 -- start start 2024-04-25T00:30:36.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.627+0000 7f7dbc618640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7da00039c0 0x7f7db41ef4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:36.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.627+0000 7f7dbc618640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7db40c1940 0x7f7db40c21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:36.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.627+0000 7f7dbc618640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7db40bf5f0 0x7f7db40bfda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:36.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.627+0000 7f7dbc618640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7db4153230 con 0x7f7db40c1940 2024-04-25T00:30:36.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.627+0000 7f7dbc618640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7db406cfc0 con 0x7f7da00039c0 2024-04-25T00:30:36.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.627+0000 7f7dbc618640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f7db41533b0 con 0x7f7db40bf5f0 2024-04-25T00:30:36.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.627+0000 7f7db9bb3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7db40c1940 0x7f7db40c21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:36.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.627+0000 7f7db9bb3640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7db40c1940 0x7f7db40c21c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36242/0 (socket says 172.21.15.116:36242) 2024-04-25T00:30:36.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.627+0000 7f7db9bb3640 1 -- 172.21.15.116:0/522391916 learned_addr learned my addr 172.21.15.116:0/522391916 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:36.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.627+0000 7f7dbabb5640 1 --2- 172.21.15.116:0/522391916 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7db40bf5f0 0x7f7db40bfda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:36.627 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.627+0000 7f7dba3b4640 1 --2- 172.21.15.116:0/522391916 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7da00039c0 0x7f7db41ef4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:36.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.627+0000 7f7db9bb3640 1 -- 172.21.15.116:0/522391916 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7db40bf5f0 msgr2=0x7f7db40bfda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:36.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.627+0000 7f7db9bb3640 1 --2- 172.21.15.116:0/522391916 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7db40bf5f0 0x7f7db40bfda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:36.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.627+0000 7f7db9bb3640 1 -- 172.21.15.116:0/522391916 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7da00039c0 msgr2=0x7f7db41ef4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:36.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.627+0000 7f7db9bb3640 1 --2- 172.21.15.116:0/522391916 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7da00039c0 0x7f7db41ef4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:36.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.627+0000 7f7db9bb3640 1 -- 172.21.15.116:0/522391916 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7db41efa30 con 0x7f7db40c1940 2024-04-25T00:30:36.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.627+0000 7f7db9bb3640 1 --2- 172.21.15.116:0/522391916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7db40c1940 0x7f7db40c21c0 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f7da400c730 tx=0x7f7da400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:36.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.627+0000 7f7daaffd640 1 -- 172.21.15.116:0/522391916 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7da4018070 con 0x7f7db40c1940 2024-04-25T00:30:36.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.627+0000 7f7daaffd640 1 -- 172.21.15.116:0/522391916 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7da40072d0 con 0x7f7db40c1940 2024-04-25T00:30:36.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.627+0000 7f7daaffd640 1 -- 172.21.15.116:0/522391916 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7da40047f0 con 0x7f7db40c1940 2024-04-25T00:30:36.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.627+0000 7f7dbc618640 1 -- 172.21.15.116:0/522391916 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7db40c2700 con 0x7f7db40c1940 2024-04-25T00:30:36.628 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.627+0000 7f7dbc618640 1 -- 172.21.15.116:0/522391916 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7db414ec60 con 0x7f7db40c1940 2024-04-25T00:30:36.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.627+0000 7f7dbc618640 1 -- 172.21.15.116:0/522391916 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7db404f510 con 0x7f7db40c1940 2024-04-25T00:30:36.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.627+0000 7f7daaffd640 1 -- 172.21.15.116:0/522391916 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7da4004ba0 con 0x7f7db40c1940 2024-04-25T00:30:36.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.627+0000 7f7daaffd640 1 --2- 172.21.15.116:0/522391916 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7d980678c0 0x7f7d98069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:36.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.631+0000 7f7daaffd640 1 -- 172.21.15.116:0/522391916 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7f7da40865d0 con 0x7f7db40c1940 2024-04-25T00:30:36.629 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.631+0000 7f7dba3b4640 1 --2- 172.21.15.116:0/522391916 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7d980678c0 0x7f7d98069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:36.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.631+0000 7f7dba3b4640 1 --2- 172.21.15.116:0/522391916 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7d980678c0 0x7f7d98069d80 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f7db0001750 tx=0x7f7db00011a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:36.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.635+0000 7f7daaffd640 1 -- 172.21.15.116:0/522391916 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f7da4055d90 con 0x7f7db40c1940 2024-04-25T00:30:36.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.791+0000 7f7dbc618640 1 -- 172.21.15.116:0/522391916 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"} v 0) -- 0x7f7db414ee60 con 0x7f7db40c1940 2024-04-25T00:30:36.791 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.791+0000 7f7daaffd640 1 -- 172.21.15.116:0/522391916 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"}]=0 v74) ==== 99+0+11 (secure 0 0 0) 0x7f7da4055690 con 0x7f7db40c1940 2024-04-25T00:30:36.791 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-04-25T00:30:36.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.795+0000 7f7dbc618640 1 -- 172.21.15.116:0/522391916 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7d980678c0 msgr2=0x7f7d98069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:36.793 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.795+0000 7f7dbc618640 1 --2- 172.21.15.116:0/522391916 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7d980678c0 0x7f7d98069d80 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f7db0001750 tx=0x7f7db00011a0 comp rx=0 tx=0).stop 2024-04-25T00:30:36.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.795+0000 7f7dbc618640 1 -- 172.21.15.116:0/522391916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7db40c1940 msgr2=0x7f7db40c21c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:36.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.795+0000 7f7dbc618640 1 --2- 172.21.15.116:0/522391916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7db40c1940 0x7f7db40c21c0 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f7da400c730 tx=0x7f7da400cbf0 comp rx=0 tx=0).stop 2024-04-25T00:30:36.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.795+0000 7f7dbc618640 1 -- 172.21.15.116:0/522391916 shutdown_connections 2024-04-25T00:30:36.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.795+0000 7f7dbc618640 1 --2- 172.21.15.116:0/522391916 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7d980678c0 0x7f7d98069d80 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:36.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.795+0000 7f7dbc618640 1 --2- 172.21.15.116:0/522391916 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7db40bf5f0 0x7f7db40bfda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:36.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.795+0000 7f7dbc618640 1 --2- 172.21.15.116:0/522391916 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7db40c1940 0x7f7db40c21c0 unknown :-1 s=CLOSED pgs=1766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:36.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.795+0000 7f7dbc618640 1 --2- 172.21.15.116:0/522391916 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7da00039c0 0x7f7db41ef4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:36.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.795+0000 7f7dbc618640 1 -- 172.21.15.116:0/522391916 >> 172.21.15.116:0/522391916 conn(0x7f7db40cb0f0 msgr2=0x7f7db4150660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:36.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.795+0000 7f7dbc618640 1 -- 172.21.15.116:0/522391916 shutdown_connections 2024-04-25T00:30:36.794 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.795+0000 7f7dbc618640 1 -- 172.21.15.116:0/522391916 wait complete. 2024-04-25T00:30:36.810 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_data pg_num 2024-04-25T00:30:36.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.923+0000 7f667e63c640 1 Processor -- start 2024-04-25T00:30:36.922 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.923+0000 7f667e63c640 1 -- start start 2024-04-25T00:30:36.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.923+0000 7f667e63c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6678157a20 0x7f66780c5930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:36.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.923+0000 7f667e63c640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6678157eb0 con 0x7f66780c8740 2024-04-25T00:30:36.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.923+0000 7f667e63c640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6678153cb0 con 0x7f6678151260 2024-04-25T00:30:36.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.923+0000 7f667e63c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f66781548c0 con 0x7f6678157a20 2024-04-25T00:30:36.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.923+0000 7f66777fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f66780c8740 0x7f66780c7750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57344/0 (socket says 172.21.15.116:57344) 2024-04-25T00:30:36.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.923+0000 7f66777fe640 1 -- 172.21.15.116:0/1497639453 learned_addr learned my addr 172.21.15.116:0/1497639453 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:30:36.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.923+0000 7f667cbd9640 1 --2- 172.21.15.116:0/1497639453 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6678157a20 0x7f66780c5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:36.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.923+0000 7f6676ffd640 1 -- 172.21.15.116:0/1497639453 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3518866682 0 0) 0x7f6678157eb0 con 0x7f66780c8740 2024-04-25T00:30:36.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.923+0000 7f6676ffd640 1 -- 172.21.15.116:0/1497639453 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6660003530 con 0x7f66780c8740 2024-04-25T00:30:36.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.923+0000 7f6676ffd640 1 -- 172.21.15.116:0/1497639453 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1805749961 0 0) 0x7f6660003530 con 0x7f66780c8740 2024-04-25T00:30:36.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.923+0000 7f6676ffd640 1 -- 172.21.15.116:0/1497639453 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6678157a20 msgr2=0x7f66780c5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:36.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.923+0000 7f6676ffd640 1 --2- 172.21.15.116:0/1497639453 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6678157a20 0x7f66780c5930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:36.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.923+0000 7f6676ffd640 1 -- 172.21.15.116:0/1497639453 >> v1:172.21.15.174:6789/0 conn(0x7f6678151260 legacy=0x7f66780c7040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:36.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.923+0000 7f6676ffd640 1 -- 172.21.15.116:0/1497639453 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6678153f90 con 0x7f66780c8740 2024-04-25T00:30:36.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.923+0000 7f6676ffd640 1 -- 172.21.15.116:0/1497639453 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f666c001680 con 0x7f66780c8740 2024-04-25T00:30:36.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.923+0000 7f6676ffd640 1 -- 172.21.15.116:0/1497639453 >> v1:172.21.15.116:6789/0 conn(0x7f66780c8740 legacy=0x7f66780c7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:36.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.923+0000 7f6676ffd640 1 --2- 172.21.15.116:0/1497639453 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66600039f0 0x7f6660003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:36.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.923+0000 7f6676ffd640 1 --2- 172.21.15.116:0/1497639453 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6660004810 0x7f6660004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:36.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.923+0000 7f6676ffd640 1 --2- 172.21.15.116:0/1497639453 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6678157a20 0x7f66600052a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:36.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.923+0000 7f6676ffd640 1 -- 172.21.15.116:0/1497639453 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f66781548c0 con 0x7f6660004810 2024-04-25T00:30:36.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.923+0000 7f6676ffd640 1 -- 172.21.15.116:0/1497639453 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f666000bbc0 con 0x7f66600039f0 2024-04-25T00:30:36.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.923+0000 7f6676ffd640 1 -- 172.21.15.116:0/1497639453 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f666000bdc0 con 0x7f6678157a20 2024-04-25T00:30:36.926 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.923+0000 7f66777fe640 1 --2- 172.21.15.116:0/1497639453 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6660004810 0x7f6660004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:36.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.923+0000 7f6677fff640 1 --2- 172.21.15.116:0/1497639453 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66600039f0 0x7f6660003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:36.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.923+0000 7f667cbd9640 1 --2- 172.21.15.116:0/1497639453 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6678157a20 0x7f66600052a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:36.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.923+0000 7f66777fe640 1 -- 172.21.15.116:0/1497639453 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6678157a20 msgr2=0x7f66600052a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:36.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.923+0000 7f66777fe640 1 --2- 172.21.15.116:0/1497639453 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6678157a20 0x7f66600052a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:36.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.923+0000 7f66777fe640 1 -- 172.21.15.116:0/1497639453 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66600039f0 msgr2=0x7f6660003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:36.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.923+0000 7f66777fe640 1 --2- 172.21.15.116:0/1497639453 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66600039f0 0x7f6660003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:36.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.923+0000 7f66777fe640 1 -- 172.21.15.116:0/1497639453 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6660003530 con 0x7f6660004810 2024-04-25T00:30:36.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.927+0000 7f66777fe640 1 --2- 172.21.15.116:0/1497639453 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6660004810 0x7f6660004c90 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f666c00c210 tx=0x7f666c01cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98c2433c10918b94 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:36.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.927+0000 7f6676ffd640 1 -- 172.21.15.116:0/1497639453 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f666c007d40 con 0x7f6660004810 2024-04-25T00:30:36.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.927+0000 7f6676ffd640 1 -- 172.21.15.116:0/1497639453 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f666c0a9070 con 0x7f6660004810 2024-04-25T00:30:36.930 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.927+0000 7f6676ffd640 1 -- 172.21.15.116:0/1497639453 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f666c005d40 con 0x7f6660004810 2024-04-25T00:30:36.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.927+0000 7f667e63c640 1 -- 172.21.15.116:0/1497639453 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6660004810 msgr2=0x7f6660004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:36.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.927+0000 7f667e63c640 1 --2- 172.21.15.116:0/1497639453 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6660004810 0x7f6660004c90 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f666c00c210 tx=0x7f666c01cca0 comp rx=0 tx=0).stop 2024-04-25T00:30:36.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.927+0000 7f667cbd9640 1 -- 172.21.15.116:0/1497639453 reap_dead start 2024-04-25T00:30:36.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.927+0000 7f667e63c640 1 -- 172.21.15.116:0/1497639453 shutdown_connections 2024-04-25T00:30:36.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.927+0000 7f667e63c640 1 -- 172.21.15.116:0/1497639453 >> 172.21.15.116:0/1497639453 conn(0x7f66780c8f90 msgr2=0x7f66780c93a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:36.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.927+0000 7f667e63c640 1 -- 172.21.15.116:0/1497639453 shutdown_connections 2024-04-25T00:30:36.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.927+0000 7f667e63c640 1 -- 172.21.15.116:0/1497639453 wait complete. 2024-04-25T00:30:36.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.927+0000 7f667e63c640 1 Processor -- start 2024-04-25T00:30:36.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.927+0000 7f667e63c640 1 -- start start 2024-04-25T00:30:36.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.927+0000 7f667e63c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6660004810 0x7f66781f1a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:36.931 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.927+0000 7f667e63c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f66781f1120 0x7f66780aa260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:36.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.927+0000 7f667e63c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66780aa7a0 0x7f667815c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:36.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.927+0000 7f667e63c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f667815f4d0 con 0x7f6660004810 2024-04-25T00:30:36.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.927+0000 7f667e63c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f667815f350 con 0x7f66780aa7a0 2024-04-25T00:30:36.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.927+0000 7f667e63c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f667815f650 con 0x7f66781f1120 2024-04-25T00:30:36.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.927+0000 7f6677fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6660004810 0x7f66781f1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:36.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.927+0000 7f6677fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6660004810 0x7f66781f1a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36248/0 (socket says 172.21.15.116:36248) 2024-04-25T00:30:36.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.927+0000 7f6677fff640 1 -- 172.21.15.116:0/1798697542 learned_addr learned my addr 172.21.15.116:0/1798697542 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:36.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.927+0000 7f66777fe640 1 --2- 172.21.15.116:0/1798697542 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f66781f1120 0x7f66780aa260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:36.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.927+0000 7f667cbd9640 1 --2- 172.21.15.116:0/1798697542 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66780aa7a0 0x7f667815c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:36.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.927+0000 7f6677fff640 1 -- 172.21.15.116:0/1798697542 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f66781f1120 msgr2=0x7f66780aa260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:36.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.927+0000 7f6677fff640 1 --2- 172.21.15.116:0/1798697542 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f66781f1120 0x7f66780aa260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:36.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.927+0000 7f6677fff640 1 -- 172.21.15.116:0/1798697542 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66780aa7a0 msgr2=0x7f667815c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:36.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.927+0000 7f6677fff640 1 --2- 172.21.15.116:0/1798697542 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66780aa7a0 0x7f667815c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:36.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.927+0000 7f6677fff640 1 -- 172.21.15.116:0/1798697542 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66780c8110 con 0x7f6660004810 2024-04-25T00:30:36.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.931+0000 7f6677fff640 1 --2- 172.21.15.116:0/1798697542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6660004810 0x7f66781f1a90 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f666800e4d0 tx=0x7f666800e990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:36.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.931+0000 7f6674ff9640 1 -- 172.21.15.116:0/1798697542 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66680052e0 con 0x7f6660004810 2024-04-25T00:30:36.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.931+0000 7f6674ff9640 1 -- 172.21.15.116:0/1798697542 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6668018070 con 0x7f6660004810 2024-04-25T00:30:36.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.931+0000 7f667e63c640 1 -- 172.21.15.116:0/1798697542 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66780aafc0 con 0x7f6660004810 2024-04-25T00:30:36.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.931+0000 7f6674ff9640 1 -- 172.21.15.116:0/1798697542 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6668003820 con 0x7f6660004810 2024-04-25T00:30:36.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.931+0000 7f667e63c640 1 -- 172.21.15.116:0/1798697542 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66780ab580 con 0x7f6660004810 2024-04-25T00:30:36.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.931+0000 7f667e63c640 1 -- 172.21.15.116:0/1798697542 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f667804f510 con 0x7f6660004810 2024-04-25T00:30:36.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.931+0000 7f6674ff9640 1 -- 172.21.15.116:0/1798697542 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f66680039c0 con 0x7f6660004810 2024-04-25T00:30:36.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.931+0000 7f6674ff9640 1 --2- 172.21.15.116:0/1798697542 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f66640678c0 0x7f6664069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:36.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.931+0000 7f6674ff9640 1 -- 172.21.15.116:0/1798697542 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7f6668086be0 con 0x7f6660004810 2024-04-25T00:30:36.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.931+0000 7f66777fe640 1 --2- 172.21.15.116:0/1798697542 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f66640678c0 0x7f6664069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:36.934 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.931+0000 7f66777fe640 1 --2- 172.21.15.116:0/1798697542 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f66640678c0 0x7f6664069d80 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f666c0ab000 tx=0x7f666c0b1040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:36.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:36.935+0000 7f6674ff9640 1 -- 172.21.15.116:0/1798697542 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f6668056430 con 0x7f6660004810 2024-04-25T00:30:37.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.083+0000 7f667e63c640 1 -- 172.21.15.116:0/1798697542 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"} v 0) -- 0x7f667814bb60 con 0x7f6660004810 2024-04-25T00:30:37.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.083+0000 7f6674ff9640 1 -- 172.21.15.116:0/1798697542 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"}]=0 v74) ==== 95+0+11 (secure 0 0 0) 0x7f6668005480 con 0x7f6660004810 2024-04-25T00:30:37.084 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-04-25T00:30:37.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.087+0000 7f667e63c640 1 -- 172.21.15.116:0/1798697542 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f66640678c0 msgr2=0x7f6664069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:37.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.087+0000 7f667e63c640 1 --2- 172.21.15.116:0/1798697542 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f66640678c0 0x7f6664069d80 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f666c0ab000 tx=0x7f666c0b1040 comp rx=0 tx=0).stop 2024-04-25T00:30:37.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.087+0000 7f667e63c640 1 -- 172.21.15.116:0/1798697542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6660004810 msgr2=0x7f66781f1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:37.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.087+0000 7f667e63c640 1 --2- 172.21.15.116:0/1798697542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6660004810 0x7f66781f1a90 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f666800e4d0 tx=0x7f666800e990 comp rx=0 tx=0).stop 2024-04-25T00:30:37.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.087+0000 7f667e63c640 1 -- 172.21.15.116:0/1798697542 shutdown_connections 2024-04-25T00:30:37.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.087+0000 7f667e63c640 1 --2- 172.21.15.116:0/1798697542 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f66640678c0 0x7f6664069d80 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:37.087 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.087+0000 7f667e63c640 1 --2- 172.21.15.116:0/1798697542 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f66780aa7a0 0x7f667815c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:37.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.087+0000 7f667e63c640 1 --2- 172.21.15.116:0/1798697542 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f66781f1120 0x7f66780aa260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:37.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.087+0000 7f667e63c640 1 --2- 172.21.15.116:0/1798697542 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6660004810 0x7f66781f1a90 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:37.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.087+0000 7f667e63c640 1 -- 172.21.15.116:0/1798697542 >> 172.21.15.116:0/1798697542 conn(0x7f66780c8f90 msgr2=0x7f66780bf910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:37.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.087+0000 7f667e63c640 1 -- 172.21.15.116:0/1798697542 shutdown_connections 2024-04-25T00:30:37.088 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.087+0000 7f667e63c640 1 -- 172.21.15.116:0/1798697542 wait complete. 2024-04-25T00:30:37.103 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_metadata pg_num 2024-04-25T00:30:37.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.223+0000 7fb58798a640 1 Processor -- start 2024-04-25T00:30:37.222 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.223+0000 7fb58798a640 1 -- start start 2024-04-25T00:30:37.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.223+0000 7fb58798a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb580153280 0x7fb580155d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:37.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.223+0000 7fb58798a640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5801567b0 con 0x7fb5800c4490 2024-04-25T00:30:37.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.223+0000 7fb58798a640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb580156a90 con 0x7fb580156270 2024-04-25T00:30:37.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.223+0000 7fb58798a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fb5800c4000 con 0x7fb580153280 2024-04-25T00:30:37.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.223+0000 7fb585f27640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fb5800c4490 0x7fb5800c4920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57358/0 (socket says 172.21.15.116:57358) 2024-04-25T00:30:37.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.223+0000 7fb585f27640 1 -- 172.21.15.116:0/796326059 learned_addr learned my addr 172.21.15.116:0/796326059 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:30:37.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.223+0000 7fb585726640 1 --2- 172.21.15.116:0/796326059 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb580153280 0x7fb580155d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:37.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.223+0000 7fb577fff640 1 -- 172.21.15.116:0/796326059 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1233500070 0 0) 0x7fb5801567b0 con 0x7fb5800c4490 2024-04-25T00:30:37.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.223+0000 7fb577fff640 1 -- 172.21.15.116:0/796326059 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb56c003530 con 0x7fb5800c4490 2024-04-25T00:30:37.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.223+0000 7fb577fff640 1 -- 172.21.15.116:0/796326059 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1222035077 0 0) 0x7fb56c003530 con 0x7fb5800c4490 2024-04-25T00:30:37.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.223+0000 7fb577fff640 1 -- 172.21.15.116:0/796326059 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb580153280 msgr2=0x7fb580155d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:37.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.223+0000 7fb577fff640 1 --2- 172.21.15.116:0/796326059 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb580153280 0x7fb580155d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:37.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.223+0000 7fb577fff640 1 -- 172.21.15.116:0/796326059 >> v1:172.21.15.174:6789/0 conn(0x7fb580156270 legacy=0x7fb5800c5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:37.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.223+0000 7fb577fff640 1 -- 172.21.15.116:0/796326059 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5800c39b0 con 0x7fb5800c4490 2024-04-25T00:30:37.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.223+0000 7fb577fff640 1 -- 172.21.15.116:0/796326059 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7fb578001a00 con 0x7fb5800c4490 2024-04-25T00:30:37.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.223+0000 7fb577fff640 1 -- 172.21.15.116:0/796326059 >> v1:172.21.15.116:6789/0 conn(0x7fb5800c4490 legacy=0x7fb5800c4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:37.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.223+0000 7fb577fff640 1 --2- 172.21.15.116:0/796326059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb56c0039f0 0x7fb56c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:37.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.223+0000 7fb577fff640 1 --2- 172.21.15.116:0/796326059 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb56c004810 0x7fb56c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:37.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.223+0000 7fb577fff640 1 --2- 172.21.15.116:0/796326059 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb580153280 0x7fb56c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:37.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.223+0000 7fb577fff640 1 -- 172.21.15.116:0/796326059 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb5800c4000 con 0x7fb56c0039f0 2024-04-25T00:30:37.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.223+0000 7fb577fff640 1 -- 172.21.15.116:0/796326059 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb56c00ba80 con 0x7fb56c004810 2024-04-25T00:30:37.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.223+0000 7fb577fff640 1 -- 172.21.15.116:0/796326059 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fb56c00bc80 con 0x7fb580153280 2024-04-25T00:30:37.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.223+0000 7fb585726640 1 --2- 172.21.15.116:0/796326059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb56c0039f0 0x7fb56c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:37.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.223+0000 7fb584f25640 1 --2- 172.21.15.116:0/796326059 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb56c004810 0x7fb56c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:37.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.223+0000 7fb585f27640 1 --2- 172.21.15.116:0/796326059 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb580153280 0x7fb56c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:37.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.223+0000 7fb585726640 1 -- 172.21.15.116:0/796326059 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb580153280 msgr2=0x7fb56c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:37.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.223+0000 7fb585726640 1 --2- 172.21.15.116:0/796326059 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb580153280 0x7fb56c0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:37.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.223+0000 7fb585726640 1 -- 172.21.15.116:0/796326059 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb56c004810 msgr2=0x7fb56c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:37.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.223+0000 7fb585726640 1 --2- 172.21.15.116:0/796326059 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb56c004810 0x7fb56c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:37.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.223+0000 7fb585726640 1 -- 172.21.15.116:0/796326059 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb56c003530 con 0x7fb56c0039f0 2024-04-25T00:30:37.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.223+0000 7fb584f25640 1 --2- 172.21.15.116:0/796326059 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb56c004810 0x7fb56c004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:30:37.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.223+0000 7fb585f27640 1 --2- 172.21.15.116:0/796326059 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb580153280 0x7fb56c0051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:30:37.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.227+0000 7fb585726640 1 --2- 172.21.15.116:0/796326059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb56c0039f0 0x7fb56c003e10 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7fb57c009ab0 tx=0x7fb57c01ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5451b1feadb40c27 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:37.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.227+0000 7fb577fff640 1 -- 172.21.15.116:0/796326059 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb57c0096b0 con 0x7fb56c0039f0 2024-04-25T00:30:37.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.227+0000 7fb577fff640 1 -- 172.21.15.116:0/796326059 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb57c0a6410 con 0x7fb56c0039f0 2024-04-25T00:30:37.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.227+0000 7fb577fff640 1 -- 172.21.15.116:0/796326059 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb57c0a6800 con 0x7fb56c0039f0 2024-04-25T00:30:37.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.227+0000 7fb58798a640 1 -- 172.21.15.116:0/796326059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb56c0039f0 msgr2=0x7fb56c003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:37.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.227+0000 7fb58798a640 1 --2- 172.21.15.116:0/796326059 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb56c0039f0 0x7fb56c003e10 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7fb57c009ab0 tx=0x7fb57c01ca30 comp rx=0 tx=0).stop 2024-04-25T00:30:37.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.227+0000 7fb585f27640 1 -- 172.21.15.116:0/796326059 reap_dead start 2024-04-25T00:30:37.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.227+0000 7fb58798a640 1 -- 172.21.15.116:0/796326059 shutdown_connections 2024-04-25T00:30:37.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.227+0000 7fb58798a640 1 -- 172.21.15.116:0/796326059 >> 172.21.15.116:0/796326059 conn(0x7fb5800c8f90 msgr2=0x7fb5800c93a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:37.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.227+0000 7fb58798a640 1 -- 172.21.15.116:0/796326059 shutdown_connections 2024-04-25T00:30:37.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.227+0000 7fb58798a640 1 -- 172.21.15.116:0/796326059 wait complete. 2024-04-25T00:30:37.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.227+0000 7fb58798a640 1 Processor -- start 2024-04-25T00:30:37.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.227+0000 7fb58798a640 1 -- start start 2024-04-25T00:30:37.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.227+0000 7fb58798a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb56c0039f0 0x7fb5800701b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:37.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.227+0000 7fb58798a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb580158740 0x7fb580158bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:37.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.227+0000 7fb58798a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb580159100 0x7fb58014c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:37.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.227+0000 7fb58798a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb58006d930 con 0x7fb580159100 2024-04-25T00:30:37.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.227+0000 7fb58798a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb58006d7b0 con 0x7fb580158740 2024-04-25T00:30:37.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.227+0000 7fb58798a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fb580153660 con 0x7fb56c0039f0 2024-04-25T00:30:37.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.227+0000 7fb585f27640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb580159100 0x7fb58014c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:37.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.227+0000 7fb585f27640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb580159100 0x7fb58014c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36268/0 (socket says 172.21.15.116:36268) 2024-04-25T00:30:37.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.227+0000 7fb585f27640 1 -- 172.21.15.116:0/3923329832 learned_addr learned my addr 172.21.15.116:0/3923329832 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:37.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.227+0000 7fb584f25640 1 --2- 172.21.15.116:0/3923329832 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb580158740 0x7fb580158bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:37.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.227+0000 7fb585726640 1 --2- 172.21.15.116:0/3923329832 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb56c0039f0 0x7fb5800701b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:37.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.227+0000 7fb585f27640 1 -- 172.21.15.116:0/3923329832 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb56c0039f0 msgr2=0x7fb5800701b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:37.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.227+0000 7fb585f27640 1 --2- 172.21.15.116:0/3923329832 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb56c0039f0 0x7fb5800701b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:37.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.227+0000 7fb585f27640 1 -- 172.21.15.116:0/3923329832 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb580158740 msgr2=0x7fb580158bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:37.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.227+0000 7fb585f27640 1 --2- 172.21.15.116:0/3923329832 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb580158740 0x7fb580158bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:37.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.227+0000 7fb585f27640 1 -- 172.21.15.116:0/3923329832 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb580156f10 con 0x7fb580159100 2024-04-25T00:30:37.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.231+0000 7fb585f27640 1 --2- 172.21.15.116:0/3923329832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb580159100 0x7fb58014c620 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7fb5800af120 tx=0x7fb57800b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:37.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.231+0000 7fb575ffb640 1 -- 172.21.15.116:0/3923329832 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb578004d70 con 0x7fb580159100 2024-04-25T00:30:37.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.231+0000 7fb575ffb640 1 -- 172.21.15.116:0/3923329832 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb57800bb20 con 0x7fb580159100 2024-04-25T00:30:37.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.231+0000 7fb58798a640 1 -- 172.21.15.116:0/3923329832 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5801f26a0 con 0x7fb580159100 2024-04-25T00:30:37.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.231+0000 7fb575ffb640 1 -- 172.21.15.116:0/3923329832 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb578007110 con 0x7fb580159100 2024-04-25T00:30:37.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.231+0000 7fb58798a640 1 -- 172.21.15.116:0/3923329832 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5800a8db0 con 0x7fb580159100 2024-04-25T00:30:37.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.231+0000 7fb58798a640 1 -- 172.21.15.116:0/3923329832 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb58004f510 con 0x7fb580159100 2024-04-25T00:30:37.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.231+0000 7fb575ffb640 1 -- 172.21.15.116:0/3923329832 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb57801b020 con 0x7fb580159100 2024-04-25T00:30:37.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.231+0000 7fb575ffb640 1 --2- 172.21.15.116:0/3923329832 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb5640678f0 0x7fb564069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:37.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.231+0000 7fb575ffb640 1 -- 172.21.15.116:0/3923329832 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7fb5780870a0 con 0x7fb580159100 2024-04-25T00:30:37.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.231+0000 7fb585726640 1 --2- 172.21.15.116:0/3923329832 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb5640678f0 0x7fb564069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:37.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.231+0000 7fb585726640 1 --2- 172.21.15.116:0/3923329832 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb5640678f0 0x7fb564069db0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fb57c0ad000 tx=0x7fb57c0b2040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:37.235 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.235+0000 7fb575ffb640 1 -- 172.21.15.116:0/3923329832 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fb5780568f0 con 0x7fb580159100 2024-04-25T00:30:37.395 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.395+0000 7fb58798a640 1 -- 172.21.15.116:0/3923329832 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"} v 0) -- 0x7fb5800715a0 con 0x7fb580159100 2024-04-25T00:30:37.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.395+0000 7fb575ffb640 1 -- 172.21.15.116:0/3923329832 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"}]=0 v74) ==== 99+0+11 (secure 0 0 0) 0x7fb5780561f0 con 0x7fb580159100 2024-04-25T00:30:37.396 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-04-25T00:30:37.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.399+0000 7fb58798a640 1 -- 172.21.15.116:0/3923329832 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb5640678f0 msgr2=0x7fb564069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:37.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.399+0000 7fb58798a640 1 --2- 172.21.15.116:0/3923329832 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb5640678f0 0x7fb564069db0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fb57c0ad000 tx=0x7fb57c0b2040 comp rx=0 tx=0).stop 2024-04-25T00:30:37.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.399+0000 7fb58798a640 1 -- 172.21.15.116:0/3923329832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb580159100 msgr2=0x7fb58014c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:37.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.399+0000 7fb58798a640 1 --2- 172.21.15.116:0/3923329832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb580159100 0x7fb58014c620 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7fb5800af120 tx=0x7fb57800b4a0 comp rx=0 tx=0).stop 2024-04-25T00:30:37.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.399+0000 7fb58798a640 1 -- 172.21.15.116:0/3923329832 shutdown_connections 2024-04-25T00:30:37.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.399+0000 7fb58798a640 1 --2- 172.21.15.116:0/3923329832 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb5640678f0 0x7fb564069db0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:37.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.399+0000 7fb58798a640 1 --2- 172.21.15.116:0/3923329832 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb580159100 0x7fb58014c620 unknown :-1 s=CLOSED pgs=1772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:37.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.399+0000 7fb58798a640 1 --2- 172.21.15.116:0/3923329832 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb580158740 0x7fb580158bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:37.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.399+0000 7fb58798a640 1 --2- 172.21.15.116:0/3923329832 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb56c0039f0 0x7fb5800701b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:37.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.399+0000 7fb58798a640 1 -- 172.21.15.116:0/3923329832 >> 172.21.15.116:0/3923329832 conn(0x7fb5800c8f90 msgr2=0x7fb58006dab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:37.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.399+0000 7fb58798a640 1 -- 172.21.15.116:0/3923329832 shutdown_connections 2024-04-25T00:30:37.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.399+0000 7fb58798a640 1 -- 172.21.15.116:0/3923329832 wait complete. 2024-04-25T00:30:37.416 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_data pg_num 2024-04-25T00:30:37.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.539+0000 7fb31c7a6640 1 Processor -- start 2024-04-25T00:30:37.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.539+0000 7fb31c7a6640 1 -- start start 2024-04-25T00:30:37.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.543+0000 7fb31c7a6640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb314148f20 0x7fb3140c2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:37.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.543+0000 7fb31c7a6640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fb31406d000 0x7fb31415cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:37.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.543+0000 7fb31c7a6640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb314149300 con 0x7fb31406d510 2024-04-25T00:30:37.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.543+0000 7fb31c7a6640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fb314153350 con 0x7fb314148f20 2024-04-25T00:30:37.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.543+0000 7fb31c7a6640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fb3140a95d0 con 0x7fb31406d000 2024-04-25T00:30:37.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.543+0000 7fb31a542640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb314148f20 0x7fb3140c2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:37.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.543+0000 7fb31a542640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb314148f20 0x7fb3140c2930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36270/0 (socket says 172.21.15.116:36270) 2024-04-25T00:30:37.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.543+0000 7fb31a542640 1 -- 172.21.15.116:0/213031351 learned_addr learned my addr 172.21.15.116:0/213031351 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:37.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.543+0000 7fb31ad43640 1 --2- 172.21.15.116:0/213031351 >> v2:172.21.15.174:3300/0 conn(0x7fb31406d000 0x7fb31415cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:37.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.543+0000 7fb31a542640 1 -- 172.21.15.116:0/213031351 >> v2:172.21.15.174:3300/0 conn(0x7fb31406d000 msgr2=0x7fb31415cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:37.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.543+0000 7fb31a542640 1 --2- 172.21.15.116:0/213031351 >> v2:172.21.15.174:3300/0 conn(0x7fb31406d000 0x7fb31415cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:37.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.543+0000 7fb31a542640 1 -- 172.21.15.116:0/213031351 >> v1:172.21.15.174:6789/0 conn(0x7fb31406d510 legacy=0x7fb314072500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:37.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.543+0000 7fb31a542640 1 -- 172.21.15.116:0/213031351 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb314153070 con 0x7fb314148f20 2024-04-25T00:30:37.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.543+0000 7fb31a542640 1 --2- 172.21.15.116:0/213031351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb314148f20 0x7fb3140c2930 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7fb31406c190 tx=0x7fb30401c1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de2708ffb0e70e5d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:37.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.543+0000 7fb319540640 1 -- 172.21.15.116:0/213031351 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3040b0070 con 0x7fb314148f20 2024-04-25T00:30:37.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.543+0000 7fb319540640 1 -- 172.21.15.116:0/213031351 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3040ab600 con 0x7fb314148f20 2024-04-25T00:30:37.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.543+0000 7fb319540640 1 -- 172.21.15.116:0/213031351 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3040ab9d0 con 0x7fb314148f20 2024-04-25T00:30:37.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.547+0000 7fb31c7a6640 1 -- 172.21.15.116:0/213031351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb314148f20 msgr2=0x7fb3140c2930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:37.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.547+0000 7fb31c7a6640 1 --2- 172.21.15.116:0/213031351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb314148f20 0x7fb3140c2930 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7fb31406c190 tx=0x7fb30401c1e0 comp rx=0 tx=0).stop 2024-04-25T00:30:37.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.547+0000 7fb31c7a6640 1 -- 172.21.15.116:0/213031351 shutdown_connections 2024-04-25T00:30:37.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.547+0000 7fb31c7a6640 1 --2- 172.21.15.116:0/213031351 >> v2:172.21.15.174:3300/0 conn(0x7fb31406d000 0x7fb31415cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:37.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.547+0000 7fb31c7a6640 1 --2- 172.21.15.116:0/213031351 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb314148f20 0x7fb3140c2930 unknown :-1 s=CLOSED pgs=1773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:37.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.547+0000 7fb31c7a6640 1 -- 172.21.15.116:0/213031351 >> 172.21.15.116:0/213031351 conn(0x7fb3140c3a20 msgr2=0x7fb3140c3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:37.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.547+0000 7fb31c7a6640 1 -- 172.21.15.116:0/213031351 shutdown_connections 2024-04-25T00:30:37.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.547+0000 7fb31c7a6640 1 -- 172.21.15.116:0/213031351 wait complete. 2024-04-25T00:30:37.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.547+0000 7fb31c7a6640 1 Processor -- start 2024-04-25T00:30:37.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.547+0000 7fb31c7a6640 1 -- start start 2024-04-25T00:30:37.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.547+0000 7fb31c7a6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb31406d000 0x7fb3141ee6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:37.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.547+0000 7fb31c7a6640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb31406d510 0x7fb314149740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:37.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.547+0000 7fb31c7a6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb314148f20 0x7fb3141f21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:37.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.547+0000 7fb31c7a6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb31415fcb0 con 0x7fb31406d000 2024-04-25T00:30:37.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.547+0000 7fb31c7a6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb31415fb30 con 0x7fb314148f20 2024-04-25T00:30:37.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.547+0000 7fb31c7a6640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fb31415fe30 con 0x7fb31406d510 2024-04-25T00:30:37.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.547+0000 7fb31a542640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb31406d000 0x7fb3141ee6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:37.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.547+0000 7fb31a542640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb31406d000 0x7fb3141ee6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36280/0 (socket says 172.21.15.116:36280) 2024-04-25T00:30:37.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.547+0000 7fb31a542640 1 -- 172.21.15.116:0/115845990 learned_addr learned my addr 172.21.15.116:0/115845990 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:37.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.547+0000 7fb31a542640 1 -- 172.21.15.116:0/115845990 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb31406d510 msgr2=0x7fb314149740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:37.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.547+0000 7fb31a542640 1 --2- 172.21.15.116:0/115845990 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb31406d510 0x7fb314149740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:37.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.547+0000 7fb31a542640 1 -- 172.21.15.116:0/115845990 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb314148f20 msgr2=0x7fb3141f21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:37.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.547+0000 7fb31a542640 1 --2- 172.21.15.116:0/115845990 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb314148f20 0x7fb3141f21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:37.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.547+0000 7fb31a542640 1 -- 172.21.15.116:0/115845990 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3140abff0 con 0x7fb31406d000 2024-04-25T00:30:37.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.547+0000 7fb31a542640 1 --2- 172.21.15.116:0/115845990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb31406d000 0x7fb3141ee6b0 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7fb3040ae040 tx=0x7fb304018f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:37.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.547+0000 7fb30affd640 1 -- 172.21.15.116:0/115845990 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3040b8020 con 0x7fb31406d000 2024-04-25T00:30:37.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.547+0000 7fb30affd640 1 -- 172.21.15.116:0/115845990 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3040b4410 con 0x7fb31406d000 2024-04-25T00:30:37.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.547+0000 7fb31c7a6640 1 -- 172.21.15.116:0/115845990 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3141eed00 con 0x7fb31406d000 2024-04-25T00:30:37.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.547+0000 7fb30affd640 1 -- 172.21.15.116:0/115845990 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3040b0040 con 0x7fb31406d000 2024-04-25T00:30:37.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.547+0000 7fb31c7a6640 1 -- 172.21.15.116:0/115845990 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3141ef2c0 con 0x7fb31406d000 2024-04-25T00:30:37.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.551+0000 7fb31c7a6640 1 -- 172.21.15.116:0/115845990 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb31404f510 con 0x7fb31406d000 2024-04-25T00:30:37.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.551+0000 7fb30affd640 1 -- 172.21.15.116:0/115845990 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb3040bb020 con 0x7fb31406d000 2024-04-25T00:30:37.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.551+0000 7fb30affd640 1 --2- 172.21.15.116:0/115845990 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb2f80678f0 0x7fb2f8069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:37.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.551+0000 7fb30affd640 1 -- 172.21.15.116:0/115845990 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7fb3041209f0 con 0x7fb31406d000 2024-04-25T00:30:37.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.551+0000 7fb319d41640 1 --2- 172.21.15.116:0/115845990 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb2f80678f0 0x7fb2f8069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:37.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.551+0000 7fb319d41640 1 --2- 172.21.15.116:0/115845990 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb2f80678f0 0x7fb2f8069db0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fb3140c2cf0 tx=0x7fb3100079c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:37.554 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.555+0000 7fb30affd640 1 -- 172.21.15.116:0/115845990 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fb3040c2030 con 0x7fb31406d000 2024-04-25T00:30:37.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.715+0000 7fb31c7a6640 1 -- 172.21.15.116:0/115845990 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"} v 0) -- 0x7fb3140c34b0 con 0x7fb31406d000 2024-04-25T00:30:37.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.715+0000 7fb30affd640 1 -- 172.21.15.116:0/115845990 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"}]=0 v74) ==== 95+0+11 (secure 0 0 0) 0x7fb314149300 con 0x7fb31406d000 2024-04-25T00:30:37.714 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-04-25T00:30:37.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.715+0000 7fb31c7a6640 1 -- 172.21.15.116:0/115845990 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb2f80678f0 msgr2=0x7fb2f8069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:37.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.715+0000 7fb31c7a6640 1 --2- 172.21.15.116:0/115845990 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb2f80678f0 0x7fb2f8069db0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fb3140c2cf0 tx=0x7fb3100079c0 comp rx=0 tx=0).stop 2024-04-25T00:30:37.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.715+0000 7fb31c7a6640 1 -- 172.21.15.116:0/115845990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb31406d000 msgr2=0x7fb3141ee6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:37.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.715+0000 7fb31c7a6640 1 --2- 172.21.15.116:0/115845990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb31406d000 0x7fb3141ee6b0 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7fb3040ae040 tx=0x7fb304018f80 comp rx=0 tx=0).stop 2024-04-25T00:30:37.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.715+0000 7fb31c7a6640 1 -- 172.21.15.116:0/115845990 shutdown_connections 2024-04-25T00:30:37.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.715+0000 7fb31c7a6640 1 --2- 172.21.15.116:0/115845990 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb2f80678f0 0x7fb2f8069db0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:37.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.715+0000 7fb31c7a6640 1 --2- 172.21.15.116:0/115845990 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb314148f20 0x7fb3141f21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:37.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.715+0000 7fb31c7a6640 1 --2- 172.21.15.116:0/115845990 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb31406d510 0x7fb314149740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:37.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.715+0000 7fb31c7a6640 1 --2- 172.21.15.116:0/115845990 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb31406d000 0x7fb3141ee6b0 unknown :-1 s=CLOSED pgs=1774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:37.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.715+0000 7fb31c7a6640 1 -- 172.21.15.116:0/115845990 >> 172.21.15.116:0/115845990 conn(0x7fb3140c3a20 msgr2=0x7fb3140c0350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:37.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.715+0000 7fb31c7a6640 1 -- 172.21.15.116:0/115845990 shutdown_connections 2024-04-25T00:30:37.717 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.715+0000 7fb31c7a6640 1 -- 172.21.15.116:0/115845990 wait complete. 2024-04-25T00:30:37.734 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T00:30:37.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.855+0000 7fe5fe469640 1 Processor -- start 2024-04-25T00:30:37.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.855+0000 7fe5fe469640 1 -- start start 2024-04-25T00:30:37.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.855+0000 7fe5fe469640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fe5f80a94a0 0x7fe5f81563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:37.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.855+0000 7fe5fe469640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe5f8071680 0x7fe5f8071ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:37.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.855+0000 7fe5fe469640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5f8071220 con 0x7fe5f814c720 2024-04-25T00:30:37.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.855+0000 7fe5fe469640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fe5f8153460 con 0x7fe5f8071680 2024-04-25T00:30:37.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.855+0000 7fe5fe469640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fe5f8152420 con 0x7fe5f80a94a0 2024-04-25T00:30:37.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.855+0000 7fe5fca06640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe5f8071680 0x7fe5f8071ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:37.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.855+0000 7fe5fca06640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe5f8071680 0x7fe5f8071ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36296/0 (socket says 172.21.15.116:36296) 2024-04-25T00:30:37.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.855+0000 7fe5fca06640 1 -- 172.21.15.116:0/54389191 learned_addr learned my addr 172.21.15.116:0/54389191 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:37.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.855+0000 7fe5f77fe640 1 --2- 172.21.15.116:0/54389191 >> v2:172.21.15.174:3300/0 conn(0x7fe5f80a94a0 0x7fe5f81563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:37.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.855+0000 7fe5fca06640 1 -- 172.21.15.116:0/54389191 >> v2:172.21.15.174:3300/0 conn(0x7fe5f80a94a0 msgr2=0x7fe5f81563d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:37.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.855+0000 7fe5fca06640 1 --2- 172.21.15.116:0/54389191 >> v2:172.21.15.174:3300/0 conn(0x7fe5f80a94a0 0x7fe5f81563d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:37.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.855+0000 7fe5fca06640 1 -- 172.21.15.116:0/54389191 >> v1:172.21.15.174:6789/0 conn(0x7fe5f814c720 legacy=0x7fe5f80c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:37.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.855+0000 7fe5fca06640 1 -- 172.21.15.116:0/54389191 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5f814cb00 con 0x7fe5f8071680 2024-04-25T00:30:37.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.859+0000 7fe5fca06640 1 --2- 172.21.15.116:0/54389191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5f8071680 0x7fe5f8071ab0 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7fe5f80a9930 tx=0x7fe5f001eb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3b623dda7037c0e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:37.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.859+0000 7fe5f6ffd640 1 -- 172.21.15.116:0/54389191 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f000b2d0 con 0x7fe5f8071680 2024-04-25T00:30:37.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.859+0000 7fe5f6ffd640 1 -- 172.21.15.116:0/54389191 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5f000b470 con 0x7fe5f8071680 2024-04-25T00:30:37.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.859+0000 7fe5f6ffd640 1 -- 172.21.15.116:0/54389191 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f00a7b20 con 0x7fe5f8071680 2024-04-25T00:30:37.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.859+0000 7fe5fe469640 1 -- 172.21.15.116:0/54389191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5f8071680 msgr2=0x7fe5f8071ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:37.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.859+0000 7fe5fe469640 1 --2- 172.21.15.116:0/54389191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5f8071680 0x7fe5f8071ab0 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7fe5f80a9930 tx=0x7fe5f001eb30 comp rx=0 tx=0).stop 2024-04-25T00:30:37.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.859+0000 7fe5fe469640 1 -- 172.21.15.116:0/54389191 shutdown_connections 2024-04-25T00:30:37.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.859+0000 7fe5fe469640 1 --2- 172.21.15.116:0/54389191 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5f8071680 0x7fe5f8071ab0 unknown :-1 s=CLOSED pgs=1775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:37.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.859+0000 7fe5fe469640 1 --2- 172.21.15.116:0/54389191 >> v2:172.21.15.174:3300/0 conn(0x7fe5f80a94a0 0x7fe5f81563d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:37.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.859+0000 7fe5fe469640 1 -- 172.21.15.116:0/54389191 >> 172.21.15.116:0/54389191 conn(0x7fe5f80c3b00 msgr2=0x7fe5f80c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:37.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.859+0000 7fe5fe469640 1 -- 172.21.15.116:0/54389191 shutdown_connections 2024-04-25T00:30:37.861 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.859+0000 7fe5fe469640 1 -- 172.21.15.116:0/54389191 wait complete. 2024-04-25T00:30:37.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.859+0000 7fe5fe469640 1 Processor -- start 2024-04-25T00:30:37.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.859+0000 7fe5fe469640 1 -- start start 2024-04-25T00:30:37.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.863+0000 7fe5fe469640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5f8071680 0x7fe5f814a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:37.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.863+0000 7fe5fe469640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe5f80a94a0 0x7fe5f80ab350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:37.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.863+0000 7fe5fe469640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5f814c720 0x7fe5f80ab890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:37.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.863+0000 7fe5fe469640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe5f80c3120 con 0x7fe5f814c720 2024-04-25T00:30:37.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.863+0000 7fe5fe469640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe5f8152f40 con 0x7fe5f8071680 2024-04-25T00:30:37.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.863+0000 7fe5fe469640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe5f80c32a0 con 0x7fe5f80a94a0 2024-04-25T00:30:37.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.863+0000 7fe5fca06640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5f814c720 0x7fe5f80ab890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:37.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.863+0000 7fe5fca06640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5f814c720 0x7fe5f80ab890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36298/0 (socket says 172.21.15.116:36298) 2024-04-25T00:30:37.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.863+0000 7fe5fca06640 1 -- 172.21.15.116:0/2661264132 learned_addr learned my addr 172.21.15.116:0/2661264132 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:37.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.863+0000 7fe5f7fff640 1 --2- 172.21.15.116:0/2661264132 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5f8071680 0x7fe5f814a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:37.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.863+0000 7fe5fca06640 1 -- 172.21.15.116:0/2661264132 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe5f80a94a0 msgr2=0x7fe5f80ab350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:37.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.863+0000 7fe5fca06640 1 --2- 172.21.15.116:0/2661264132 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe5f80a94a0 0x7fe5f80ab350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:37.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.863+0000 7fe5fca06640 1 -- 172.21.15.116:0/2661264132 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5f8071680 msgr2=0x7fe5f814a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:37.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.863+0000 7fe5fca06640 1 --2- 172.21.15.116:0/2661264132 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5f8071680 0x7fe5f814a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:37.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.863+0000 7fe5fca06640 1 -- 172.21.15.116:0/2661264132 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5f813e430 con 0x7fe5f814c720 2024-04-25T00:30:37.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.863+0000 7fe5fca06640 1 --2- 172.21.15.116:0/2661264132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5f814c720 0x7fe5f80ab890 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7fe5f001b740 tx=0x7fe5f001b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:37.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.863+0000 7fe5f4ff9640 1 -- 172.21.15.116:0/2661264132 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f00be020 con 0x7fe5f814c720 2024-04-25T00:30:37.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.863+0000 7fe5f4ff9640 1 -- 172.21.15.116:0/2661264132 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5f0009820 con 0x7fe5f814c720 2024-04-25T00:30:37.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.863+0000 7fe5fe469640 1 -- 172.21.15.116:0/2661264132 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5f8155520 con 0x7fe5f814c720 2024-04-25T00:30:37.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.863+0000 7fe5f4ff9640 1 -- 172.21.15.116:0/2661264132 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f0009c10 con 0x7fe5f814c720 2024-04-25T00:30:37.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.863+0000 7fe5fe469640 1 -- 172.21.15.116:0/2661264132 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5f81551e0 con 0x7fe5f814c720 2024-04-25T00:30:37.864 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.863+0000 7fe5fe469640 1 -- 172.21.15.116:0/2661264132 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5f81558e0 con 0x7fe5f814c720 2024-04-25T00:30:37.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.863+0000 7fe5f4ff9640 1 -- 172.21.15.116:0/2661264132 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe5f00c1020 con 0x7fe5f814c720 2024-04-25T00:30:37.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.863+0000 7fe5f4ff9640 1 --2- 172.21.15.116:0/2661264132 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe5e40678c0 0x7fe5e4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:37.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.863+0000 7fe5f4ff9640 1 -- 172.21.15.116:0/2661264132 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7fe5f01241e0 con 0x7fe5f814c720 2024-04-25T00:30:37.865 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.863+0000 7fe5f7fff640 1 --2- 172.21.15.116:0/2661264132 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe5e40678c0 0x7fe5e4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:37.866 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.867+0000 7fe5f7fff640 1 --2- 172.21.15.116:0/2661264132 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe5e40678c0 0x7fe5e4069d80 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fe5e8007fd0 tx=0x7fe5e8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:37.868 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:37.867+0000 7fe5f4ff9640 1 -- 172.21.15.116:0/2661264132 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fe5f81558e0 con 0x7fe5f814c720 2024-04-25T00:30:38.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.035+0000 7fe5fe469640 1 -- 172.21.15.116:0/2661264132 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fe5f8154fd0 con 0x7fe5f814c720 2024-04-25T00:30:38.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.035+0000 7fe5f4ff9640 1 -- 172.21.15.116:0/2661264132 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 92 entries v74) ==== 81+0+5682 (secure 0 0 0) 0x7fe5f00c8030 con 0x7fe5f814c720 2024-04-25T00:30:38.037 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/3499878473 2024-04-26T00:30:15.781624+0000 2024-04-25T00:30:38.037 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/3499878473 2024-04-26T00:30:15.781624+0000 2024-04-25T00:30:38.037 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/2915274073 2024-04-26T00:29:50.776462+0000 2024-04-25T00:30:38.037 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6833/3891103804 2024-04-26T00:28:50.767503+0000 2024-04-25T00:30:38.037 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/2371126056 2024-04-26T00:27:35.754759+0000 2024-04-25T00:30:38.037 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/349772087 2024-04-26T00:27:15.748274+0000 2024-04-25T00:30:38.037 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/349772087 2024-04-26T00:27:15.748274+0000 2024-04-25T00:30:38.037 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6833/3693049068 2024-04-26T00:26:45.739776+0000 2024-04-25T00:30:38.037 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6832/3693049068 2024-04-26T00:26:45.739776+0000 2024-04-25T00:30:38.037 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/4181466530 2024-04-26T00:26:15.734586+0000 2024-04-25T00:30:38.037 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/4181466530 2024-04-26T00:26:15.734586+0000 2024-04-25T00:30:38.037 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/492366509 2024-04-26T00:24:15.701887+0000 2024-04-25T00:30:38.037 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6832/2998042514 2024-04-26T00:23:30.691827+0000 2024-04-25T00:30:38.038 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6833/663910035 2024-04-26T00:22:35.677323+0000 2024-04-25T00:30:38.038 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6832/663910035 2024-04-26T00:22:35.677323+0000 2024-04-25T00:30:38.038 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/321466844 2024-04-26T00:22:10.671673+0000 2024-04-25T00:30:38.038 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/980221971 2024-04-26T00:21:40.665115+0000 2024-04-25T00:30:38.038 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/991183935 2024-04-26T00:20:50.654151+0000 2024-04-25T00:30:38.038 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/991183935 2024-04-26T00:20:50.654151+0000 2024-04-25T00:30:38.038 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/321466844 2024-04-26T00:22:10.671673+0000 2024-04-25T00:30:38.038 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/1358527848 2024-04-26T00:11:00.511032+0000 2024-04-25T00:30:38.038 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6832/3841609112 2024-04-26T00:07:00.448498+0000 2024-04-25T00:30:38.038 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/1670061091 2024-04-26T00:09:30.490485+0000 2024-04-25T00:30:38.038 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/1971675962 2024-04-26T00:25:25.721965+0000 2024-04-25T00:30:38.038 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/1233829678 2024-04-26T00:07:30.454393+0000 2024-04-25T00:30:38.038 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/2371126056 2024-04-26T00:27:35.754759+0000 2024-04-25T00:30:38.039 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6834/2883220846 2024-04-26T00:11:50.521053+0000 2024-04-25T00:30:38.039 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/1233829678 2024-04-26T00:07:30.454393+0000 2024-04-25T00:30:38.039 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/3180024481 2024-04-26T00:15:25.567768+0000 2024-04-25T00:30:38.039 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/879531976 2024-04-26T00:20:10.642563+0000 2024-04-25T00:30:38.039 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6833/174781672 2024-04-26T00:02:40.399073+0000 2024-04-25T00:30:38.039 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6835/4215220222 2024-04-26T00:15:40.572498+0000 2024-04-25T00:30:38.039 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/221971311 2024-04-26T00:22:48.898075+0000 2024-04-25T00:30:38.039 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/1281466397 2024-04-26T00:18:20.613347+0000 2024-04-25T00:30:38.039 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6832/3891103804 2024-04-26T00:28:50.767503+0000 2024-04-25T00:30:38.039 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/1358527848 2024-04-26T00:11:00.511032+0000 2024-04-25T00:30:38.039 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/221971311 2024-04-26T00:22:48.898075+0000 2024-04-25T00:30:38.039 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6832/3006671671 2024-04-26T00:09:30.490406+0000 2024-04-25T00:30:38.039 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/980221971 2024-04-26T00:21:40.665115+0000 2024-04-25T00:30:38.039 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/1762356055 2024-04-26T00:03:36.597345+0000 2024-04-25T00:30:38.039 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6833/3006671671 2024-04-26T00:09:30.490406+0000 2024-04-25T00:30:38.040 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/1459177921 2024-04-26T00:03:30.406778+0000 2024-04-25T00:30:38.040 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/462190523 2024-04-26T00:14:55.560782+0000 2024-04-25T00:30:38.040 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/879531976 2024-04-26T00:20:10.642563+0000 2024-04-25T00:30:38.040 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6832/1459177921 2024-04-26T00:03:30.406778+0000 2024-04-25T00:30:38.040 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/1159002662 2024-04-26T00:10:20.502218+0000 2024-04-25T00:30:38.040 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/1762356055 2024-04-26T00:03:36.597345+0000 2024-04-25T00:30:38.040 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6834/4215220222 2024-04-26T00:15:40.572498+0000 2024-04-25T00:30:38.040 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/1971675962 2024-04-26T00:25:25.721965+0000 2024-04-25T00:30:38.040 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/3081407040 2024-04-26T00:17:25.598516+0000 2024-04-25T00:30:38.040 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/1599708526 2024-04-26T00:12:20.526742+0000 2024-04-25T00:30:38.040 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/2276529656 2024-04-26T00:19:05.629048+0000 2024-04-25T00:30:38.041 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/1159002662 2024-04-26T00:10:20.502218+0000 2024-04-25T00:30:38.041 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6833/2998042514 2024-04-26T00:23:30.691827+0000 2024-04-25T00:30:38.041 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/3081407040 2024-04-26T00:17:25.598516+0000 2024-04-25T00:30:38.041 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6835/2883220846 2024-04-26T00:11:50.521053+0000 2024-04-25T00:30:38.041 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/492366509 2024-04-26T00:24:15.701887+0000 2024-04-25T00:30:38.041 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/2004876235 2024-04-26T00:04:45.421611+0000 2024-04-25T00:30:38.041 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/174781672 2024-04-26T00:02:40.399073+0000 2024-04-25T00:30:38.041 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6833/3633657268 2024-04-26T00:25:40.724634+0000 2024-04-25T00:30:38.041 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6832/365773583 2024-04-26T00:14:58.857970+0000 2024-04-25T00:30:38.041 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/1889314958 2024-04-26T00:06:30.441408+0000 2024-04-25T00:30:38.041 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6833/3705654249 2024-04-26T00:12:45.533219+0000 2024-04-25T00:30:38.041 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6832/3705654249 2024-04-26T00:12:45.533219+0000 2024-04-25T00:30:38.041 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/1670061091 2024-04-26T00:09:30.490485+0000 2024-04-25T00:30:38.041 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/1599708526 2024-04-26T00:12:20.526742+0000 2024-04-25T00:30:38.042 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6832/3633657268 2024-04-26T00:25:40.724634+0000 2024-04-25T00:30:38.042 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/3583791822 2024-04-26T00:08:00.460587+0000 2024-04-25T00:30:38.042 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/3583791822 2024-04-26T00:08:00.460587+0000 2024-04-25T00:30:38.042 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/3398690896 2024-04-26T00:13:30.543729+0000 2024-04-25T00:30:38.042 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/3398690896 2024-04-26T00:13:30.543729+0000 2024-04-25T00:30:38.042 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/462190523 2024-04-26T00:14:55.560782+0000 2024-04-25T00:30:38.042 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/3336339376 2024-04-26T00:18:35.616887+0000 2024-04-25T00:30:38.042 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/3180024481 2024-04-26T00:15:25.567768+0000 2024-04-25T00:30:38.042 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6833/365773583 2024-04-26T00:14:58.857970+0000 2024-04-25T00:30:38.042 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/2012470875 2024-04-26T00:16:35.584511+0000 2024-04-25T00:30:38.042 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6833/2004876235 2024-04-26T00:04:45.421611+0000 2024-04-25T00:30:38.042 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/1281466397 2024-04-26T00:18:20.613347+0000 2024-04-25T00:30:38.042 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6832/2340581953 2024-04-26T00:16:45.587651+0000 2024-04-25T00:30:38.042 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/2915274073 2024-04-26T00:29:50.776462+0000 2024-04-25T00:30:38.043 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/1649769369 2024-04-26T00:04:35.418892+0000 2024-04-25T00:30:38.043 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6832/1649769369 2024-04-26T00:04:35.418892+0000 2024-04-25T00:30:38.043 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6833/2340581953 2024-04-26T00:16:45.587651+0000 2024-04-25T00:30:38.043 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/1889314958 2024-04-26T00:06:30.441408+0000 2024-04-25T00:30:38.043 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6832/3645553116 2024-04-26T00:17:35.601559+0000 2024-04-25T00:30:38.043 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6833/3645553116 2024-04-26T00:17:35.601559+0000 2024-04-25T00:30:38.043 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6833/3841609112 2024-04-26T00:07:00.448498+0000 2024-04-25T00:30:38.043 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/2012470875 2024-04-26T00:16:35.584511+0000 2024-04-25T00:30:38.043 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/3336339376 2024-04-26T00:18:35.616887+0000 2024-04-25T00:30:38.043 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/2276529656 2024-04-26T00:19:05.629048+0000 2024-04-25T00:30:38.043 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6832/1655959433 2024-04-26T00:19:30.633957+0000 2024-04-25T00:30:38.043 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6833/1655959433 2024-04-26T00:19:30.633957+0000 2024-04-25T00:30:38.044 INFO:teuthology.orchestra.run.smithi116.stderr:listed 92 entries 2024-04-25T00:30:38.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.039+0000 7fe5fe469640 1 -- 172.21.15.116:0/2661264132 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe5e40678c0 msgr2=0x7fe5e4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:38.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.039+0000 7fe5fe469640 1 --2- 172.21.15.116:0/2661264132 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe5e40678c0 0x7fe5e4069d80 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fe5e8007fd0 tx=0x7fe5e8009040 comp rx=0 tx=0).stop 2024-04-25T00:30:38.044 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.039+0000 7fe5fe469640 1 -- 172.21.15.116:0/2661264132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5f814c720 msgr2=0x7fe5f80ab890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:38.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.039+0000 7fe5fe469640 1 --2- 172.21.15.116:0/2661264132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5f814c720 0x7fe5f80ab890 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7fe5f001b740 tx=0x7fe5f001b8d0 comp rx=0 tx=0).stop 2024-04-25T00:30:38.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.039+0000 7fe5fe469640 1 -- 172.21.15.116:0/2661264132 shutdown_connections 2024-04-25T00:30:38.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.039+0000 7fe5fe469640 1 --2- 172.21.15.116:0/2661264132 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe5e40678c0 0x7fe5e4069d80 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:38.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.039+0000 7fe5fe469640 1 --2- 172.21.15.116:0/2661264132 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5f814c720 0x7fe5f80ab890 unknown :-1 s=CLOSED pgs=1776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:38.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.039+0000 7fe5fe469640 1 --2- 172.21.15.116:0/2661264132 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe5f80a94a0 0x7fe5f80ab350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:38.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.039+0000 7fe5fe469640 1 --2- 172.21.15.116:0/2661264132 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5f8071680 0x7fe5f814a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:38.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.039+0000 7fe5fe469640 1 -- 172.21.15.116:0/2661264132 >> 172.21.15.116:0/2661264132 conn(0x7fe5f80c3b00 msgr2=0x7fe5f80c2720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:38.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.039+0000 7fe5fe469640 1 -- 172.21.15.116:0/2661264132 shutdown_connections 2024-04-25T00:30:38.045 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.039+0000 7fe5fe469640 1 -- 172.21.15.116:0/2661264132 wait complete. 2024-04-25T00:30:38.054 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-04-25T00:30:38.054 DEBUG:teuthology.orchestra.run.smithi174:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T00:30:38.062 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T00:30:38.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.175+0000 7fa922881640 1 Processor -- start 2024-04-25T00:30:38.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.175+0000 7fa922881640 1 -- start start 2024-04-25T00:30:38.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.175+0000 7fa922881640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fa91c0a94a0 0x7fa91c1538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:38.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.175+0000 7fa922881640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa91c0a99f0 0x7fa91c153e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:38.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.175+0000 7fa922881640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa91c0c3910 con 0x7fa91c150d20 2024-04-25T00:30:38.176 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.175+0000 7fa922881640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fa91c0c16e0 con 0x7fa91c0a94a0 2024-04-25T00:30:38.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.175+0000 7fa922881640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa91c0c1400 con 0x7fa91c0a99f0 2024-04-25T00:30:38.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.175+0000 7fa91bfff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fa91c150d20 0x7fa91c0c2a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57370/0 (socket says 172.21.15.116:57370) 2024-04-25T00:30:38.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.175+0000 7fa91bfff640 1 -- 172.21.15.116:0/1088782470 learned_addr learned my addr 172.21.15.116:0/1088782470 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:30:38.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.175+0000 7fa91b7fe640 1 --2- 172.21.15.116:0/1088782470 >> v2:172.21.15.174:3300/0 conn(0x7fa91c0a94a0 0x7fa91c1538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:38.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.175+0000 7fa920e1e640 1 --2- 172.21.15.116:0/1088782470 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa91c0a99f0 0x7fa91c153e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:38.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.175+0000 7fa91affd640 1 -- 172.21.15.116:0/1088782470 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 245187918 0 0) 0x7fa91c0c3910 con 0x7fa91c150d20 2024-04-25T00:30:38.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.175+0000 7fa91affd640 1 -- 172.21.15.116:0/1088782470 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa904003530 con 0x7fa91c150d20 2024-04-25T00:30:38.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.175+0000 7fa91affd640 1 -- 172.21.15.116:0/1088782470 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 713247677 0 0) 0x7fa904003530 con 0x7fa91c150d20 2024-04-25T00:30:38.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.175+0000 7fa91affd640 1 -- 172.21.15.116:0/1088782470 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa91c0a99f0 msgr2=0x7fa91c153e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:38.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.175+0000 7fa91affd640 1 --2- 172.21.15.116:0/1088782470 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa91c0a99f0 0x7fa91c153e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:38.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.175+0000 7fa91affd640 1 -- 172.21.15.116:0/1088782470 >> v2:172.21.15.174:3300/0 conn(0x7fa91c0a94a0 msgr2=0x7fa91c1538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:38.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.175+0000 7fa91affd640 1 --2- 172.21.15.116:0/1088782470 >> v2:172.21.15.174:3300/0 conn(0x7fa91c0a94a0 0x7fa91c1538e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:38.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.175+0000 7fa91affd640 1 -- 172.21.15.116:0/1088782470 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa91c06e660 con 0x7fa91c150d20 2024-04-25T00:30:38.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.175+0000 7fa91affd640 1 -- 172.21.15.116:0/1088782470 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7fa90c001680 con 0x7fa91c150d20 2024-04-25T00:30:38.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.175+0000 7fa91affd640 1 -- 172.21.15.116:0/1088782470 >> v1:172.21.15.116:6789/0 conn(0x7fa91c150d20 legacy=0x7fa91c0c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:38.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.175+0000 7fa91affd640 1 --2- 172.21.15.116:0/1088782470 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9040039c0 0x7fa904003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:38.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.175+0000 7fa91affd640 1 --2- 172.21.15.116:0/1088782470 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa91c0a99f0 0x7fa904004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:38.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.175+0000 7fa91affd640 1 --2- 172.21.15.116:0/1088782470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa904004d50 0x7fa9040051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:38.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.175+0000 7fa91affd640 1 -- 172.21.15.116:0/1088782470 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa91c0c16e0 con 0x7fa904004d50 2024-04-25T00:30:38.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.175+0000 7fa91affd640 1 -- 172.21.15.116:0/1088782470 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa91c0c1400 con 0x7fa9040039c0 2024-04-25T00:30:38.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.175+0000 7fa91affd640 1 -- 172.21.15.116:0/1088782470 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa90400b9d0 con 0x7fa91c0a99f0 2024-04-25T00:30:38.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.175+0000 7fa920e1e640 1 --2- 172.21.15.116:0/1088782470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa904004d50 0x7fa9040051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:38.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.175+0000 7fa91b7fe640 1 --2- 172.21.15.116:0/1088782470 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa91c0a99f0 0x7fa904004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:38.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.175+0000 7fa91bfff640 1 --2- 172.21.15.116:0/1088782470 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9040039c0 0x7fa904003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:38.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.175+0000 7fa920e1e640 1 -- 172.21.15.116:0/1088782470 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa91c0a99f0 msgr2=0x7fa904004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:38.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.175+0000 7fa920e1e640 1 --2- 172.21.15.116:0/1088782470 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa91c0a99f0 0x7fa904004810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:38.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.175+0000 7fa920e1e640 1 -- 172.21.15.116:0/1088782470 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9040039c0 msgr2=0x7fa904003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:38.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.175+0000 7fa920e1e640 1 --2- 172.21.15.116:0/1088782470 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9040039c0 0x7fa904003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:38.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.175+0000 7fa920e1e640 1 -- 172.21.15.116:0/1088782470 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa904003530 con 0x7fa904004d50 2024-04-25T00:30:38.179 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.175+0000 7fa91bfff640 1 --2- 172.21.15.116:0/1088782470 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa9040039c0 0x7fa904003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:30:38.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.179+0000 7fa920e1e640 1 --2- 172.21.15.116:0/1088782470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa904004d50 0x7fa9040051d0 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7fa914009fa0 tx=0x7fa91401cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42d4b44429bc32ce server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:38.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.179+0000 7fa91affd640 1 -- 172.21.15.116:0/1088782470 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9140b2070 con 0x7fa904004d50 2024-04-25T00:30:38.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.179+0000 7fa91affd640 1 -- 172.21.15.116:0/1088782470 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9140a5d10 con 0x7fa904004d50 2024-04-25T00:30:38.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.179+0000 7fa91affd640 1 -- 172.21.15.116:0/1088782470 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa914009bf0 con 0x7fa904004d50 2024-04-25T00:30:38.180 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.179+0000 7fa922881640 1 -- 172.21.15.116:0/1088782470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa904004d50 msgr2=0x7fa9040051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:38.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.179+0000 7fa922881640 1 --2- 172.21.15.116:0/1088782470 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa904004d50 0x7fa9040051d0 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7fa914009fa0 tx=0x7fa91401cc40 comp rx=0 tx=0).stop 2024-04-25T00:30:38.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.179+0000 7fa920e1e640 1 -- 172.21.15.116:0/1088782470 reap_dead start 2024-04-25T00:30:38.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.179+0000 7fa922881640 1 -- 172.21.15.116:0/1088782470 shutdown_connections 2024-04-25T00:30:38.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.179+0000 7fa922881640 1 -- 172.21.15.116:0/1088782470 >> 172.21.15.116:0/1088782470 conn(0x7fa91c0c3b00 msgr2=0x7fa91c0c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:38.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.179+0000 7fa922881640 1 -- 172.21.15.116:0/1088782470 shutdown_connections 2024-04-25T00:30:38.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.179+0000 7fa922881640 1 -- 172.21.15.116:0/1088782470 wait complete. 2024-04-25T00:30:38.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.179+0000 7fa922881640 1 Processor -- start 2024-04-25T00:30:38.181 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.179+0000 7fa922881640 1 -- start start 2024-04-25T00:30:38.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.179+0000 7fa922881640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa91c13b330 0x7fa91c13fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:38.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.179+0000 7fa922881640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa91c1ef160 0x7fa91c072730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:38.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.179+0000 7fa922881640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa91c13b710 0x7fa91c0c0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:38.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.179+0000 7fa922881640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa91c0c3120 con 0x7fa91c13b710 2024-04-25T00:30:38.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.179+0000 7fa922881640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa91c06ec20 con 0x7fa91c13b330 2024-04-25T00:30:38.183 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.179+0000 7fa922881640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa91c0c32a0 con 0x7fa91c1ef160 2024-04-25T00:30:38.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.179+0000 7fa920e1e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa91c13b710 0x7fa91c0c0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:38.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.179+0000 7fa920e1e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa91c13b710 0x7fa91c0c0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36310/0 (socket says 172.21.15.116:36310) 2024-04-25T00:30:38.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.179+0000 7fa920e1e640 1 -- 172.21.15.116:0/3148596704 learned_addr learned my addr 172.21.15.116:0/3148596704 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:38.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.183+0000 7fa91bfff640 1 --2- 172.21.15.116:0/3148596704 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa91c13b330 0x7fa91c13fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:38.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.183+0000 7fa91b7fe640 1 --2- 172.21.15.116:0/3148596704 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa91c1ef160 0x7fa91c072730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:38.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.183+0000 7fa920e1e640 1 -- 172.21.15.116:0/3148596704 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa91c1ef160 msgr2=0x7fa91c072730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:38.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.183+0000 7fa920e1e640 1 --2- 172.21.15.116:0/3148596704 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa91c1ef160 0x7fa91c072730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:38.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.183+0000 7fa920e1e640 1 -- 172.21.15.116:0/3148596704 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa91c13b330 msgr2=0x7fa91c13fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:38.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.183+0000 7fa920e1e640 1 --2- 172.21.15.116:0/3148596704 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa91c13b330 0x7fa91c13fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:38.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.183+0000 7fa920e1e640 1 -- 172.21.15.116:0/3148596704 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa91c13bfb0 con 0x7fa91c13b710 2024-04-25T00:30:38.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.183+0000 7fa920e1e640 1 --2- 172.21.15.116:0/3148596704 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa91c13b710 0x7fa91c0c0430 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7fa9140b0040 tx=0x7fa914009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:38.184 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.183+0000 7fa918ff9640 1 -- 172.21.15.116:0/3148596704 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9140b2040 con 0x7fa91c13b710 2024-04-25T00:30:38.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.183+0000 7fa918ff9640 1 -- 172.21.15.116:0/3148596704 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9140a5b10 con 0x7fa91c13b710 2024-04-25T00:30:38.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.183+0000 7fa922881640 1 -- 172.21.15.116:0/3148596704 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa91c1ee680 con 0x7fa91c13b710 2024-04-25T00:30:38.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.183+0000 7fa918ff9640 1 -- 172.21.15.116:0/3148596704 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa914009bf0 con 0x7fa91c13b710 2024-04-25T00:30:38.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.183+0000 7fa922881640 1 -- 172.21.15.116:0/3148596704 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa91c13bc40 con 0x7fa91c13b710 2024-04-25T00:30:38.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.183+0000 7fa922881640 1 -- 172.21.15.116:0/3148596704 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa91c068350 con 0x7fa91c13b710 2024-04-25T00:30:38.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.183+0000 7fa918ff9640 1 -- 172.21.15.116:0/3148596704 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa9140a5d10 con 0x7fa91c13b710 2024-04-25T00:30:38.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.183+0000 7fa918ff9640 1 --2- 172.21.15.116:0/3148596704 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa9080678e0 0x7fa908069da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:38.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.183+0000 7fa918ff9640 1 -- 172.21.15.116:0/3148596704 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7fa914122130 con 0x7fa91c13b710 2024-04-25T00:30:38.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.183+0000 7fa91bfff640 1 --2- 172.21.15.116:0/3148596704 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa9080678e0 0x7fa908069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:38.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.187+0000 7fa91bfff640 1 --2- 172.21.15.116:0/3148596704 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa9080678e0 0x7fa908069da0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fa90c000c10 tx=0x7fa90c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:38.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.187+0000 7fa918ff9640 1 -- 172.21.15.116:0/3148596704 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fa9140c7060 con 0x7fa91c13b710 2024-04-25T00:30:38.348 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.347+0000 7fa922881640 1 -- 172.21.15.116:0/3148596704 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa91c0c34f0 con 0x7fa91c13b710 2024-04-25T00:30:38.351 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.351+0000 7fa918ff9640 1 -- 172.21.15.116:0/3148596704 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v74) ==== 74+0+26655 (secure 0 0 0) 0x7fa914016950 con 0x7fa91c13b710 2024-04-25T00:30:38.351 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:30:38.352 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":74,"fsid":"6d6a1aa7-0478-4c83-8057-05917e81f046","created":"2024-04-25T00:00:40.682493+0000","modified":"2024-04-25T00:30:15.781714+0000","last_up_change":"2024-04-25T00:00:51.822174+0000","last_in_change":"2024-04-25T00:00:42.980473+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T00:00:53.324376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-04-25T00:01:03.197326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-04-25T00:01:03.728618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-04-25T00:01:17.125210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-04-25T00:01:17.821873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8b6ddcb2-1ad9-4167-961c-bd6936bb9060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6809","nonce":3751765682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6811","nonce":3751765682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6815","nonce":3751765682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6813","nonce":3751765682}]},"public_addr":"172.21.15.116:6809/3751765682","cluster_addr":"172.21.15.116:6811/3751765682","heartbeat_back_addr":"172.21.15.116:6815/3751765682","heartbeat_front_addr":"172.21.15.116:6813/3751765682","state":["exists","up"]},{"osd":1,"uuid":"a6431768-994f-41da-8853-886f5687fb91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6801","nonce":377879124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6803","nonce":377879124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6807","nonce":377879124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6805","nonce":377879124}]},"public_addr":"172.21.15.116:6801/377879124","cluster_addr":"172.21.15.116:6803/377879124","heartbeat_back_addr":"172.21.15.116:6807/377879124","heartbeat_front_addr":"172.21.15.116:6805/377879124","state":["exists","up"]},{"osd":2,"uuid":"4f69bd40-5a23-45ff-8693-e1de75e1aa9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6817","nonce":1099282267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6819","nonce":1099282267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6823","nonce":1099282267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6821","nonce":1099282267}]},"public_addr":"172.21.15.116:6817/1099282267","cluster_addr":"172.21.15.116:6819/1099282267","heartbeat_back_addr":"172.21.15.116:6823/1099282267","heartbeat_front_addr":"172.21.15.116:6821/1099282267","state":["exists","up"]},{"osd":3,"uuid":"6a0d5344-8b6f-40d3-918d-0f0a4be688e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6825","nonce":1179942661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6827","nonce":1179942661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6831","nonce":1179942661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6829","nonce":1179942661}]},"public_addr":"172.21.15.116:6825/1179942661","cluster_addr":"172.21.15.116:6827/1179942661","heartbeat_back_addr":"172.21.15.116:6831/1179942661","heartbeat_front_addr":"172.21.15.116:6829/1179942661","state":["exists","up"]},{"osd":4,"uuid":"132e92e4-b825-441e-965a-ca00533fa86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6801","nonce":3132444330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6803","nonce":3132444330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6807","nonce":3132444330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6805","nonce":3132444330}]},"public_addr":"172.21.15.174:6801/3132444330","cluster_addr":"172.21.15.174:6803/3132444330","heartbeat_back_addr":"172.21.15.174:6807/3132444330","heartbeat_front_addr":"172.21.15.174:6805/3132444330","state":["exists","up"]},{"osd":5,"uuid":"a2eeb5b6-1b31-43cc-8306-a0adf3ee15bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6809","nonce":881817164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6811","nonce":881817164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6815","nonce":881817164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6813","nonce":881817164}]},"public_addr":"172.21.15.174:6809/881817164","cluster_addr":"172.21.15.174:6811/881817164","heartbeat_back_addr":"172.21.15.174:6815/881817164","heartbeat_front_addr":"172.21.15.174:6813/881817164","state":["exists","up"]},{"osd":6,"uuid":"588e873a-c841-4ba7-90f2-8956c87a9983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6825","nonce":117574251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6827","nonce":117574251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6831","nonce":117574251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6829","nonce":117574251}]},"public_addr":"172.21.15.174:6825/117574251","cluster_addr":"172.21.15.174:6827/117574251","heartbeat_back_addr":"172.21.15.174:6831/117574251","heartbeat_front_addr":"172.21.15.174:6829/117574251","state":["exists","up"]},{"osd":7,"uuid":"3b90f9b0-8a0e-4883-b088-ca1d4012870c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6817","nonce":2543899196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6819","nonce":2543899196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6823","nonce":2543899196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6821","nonce":2543899196}]},"public_addr":"172.21.15.174:6817/2543899196","cluster_addr":"172.21.15.174:6819/2543899196","heartbeat_back_addr":"172.21.15.174:6823/2543899196","heartbeat_front_addr":"172.21.15.174:6821/2543899196","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.293769+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.310034+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.404865+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.549658+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.192174+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.297266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.493649+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.452309+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2c","mappings":[{"from":0,"to":7}]},{"pgid":"5.3","mappings":[{"from":4,"to":5}]},{"pgid":"5.5","mappings":[{"from":4,"to":5}]},{"pgid":"5.14","mappings":[{"from":4,"to":5}]},{"pgid":"5.19","mappings":[{"from":4,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:6835/3499878473":"2024-04-26T00:30:15.781624+0000","172.21.15.116:6834/3499878473":"2024-04-26T00:30:15.781624+0000","172.21.15.174:6836/2915274073":"2024-04-26T00:29:50.776462+0000","172.21.15.116:6833/3891103804":"2024-04-26T00:28:50.767503+0000","172.21.15.116:6834/2371126056":"2024-04-26T00:27:35.754759+0000","172.21.15.174:6837/349772087":"2024-04-26T00:27:15.748274+0000","172.21.15.174:6836/349772087":"2024-04-26T00:27:15.748274+0000","172.21.15.116:6833/3693049068":"2024-04-26T00:26:45.739776+0000","172.21.15.116:6832/3693049068":"2024-04-26T00:26:45.739776+0000","172.21.15.174:6837/4181466530":"2024-04-26T00:26:15.734586+0000","172.21.15.174:6836/4181466530":"2024-04-26T00:26:15.734586+0000","172.21.15.174:6836/492366509":"2024-04-26T00:24:15.701887+0000","172.21.15.116:6832/2998042514":"2024-04-26T00:23:30.691827+0000","172.21.15.116:6833/663910035":"2024-04-26T00:22:35.677323+0000","172.21.15.116:6832/663910035":"2024-04-26T00:22:35.677323+0000","172.21.15.174:6837/321466844":"2024-04-26T00:22:10.671673+0000","172.21.15.116:6835/980221971":"2024-04-26T00:21:40.665115+0000","172.21.15.174:6837/991183935":"2024-04-26T00:20:50.654151+0000","172.21.15.174:6836/991183935":"2024-04-26T00:20:50.654151+0000","172.21.15.174:6836/321466844":"2024-04-26T00:22:10.671673+0000","172.21.15.116:6834/1358527848":"2024-04-26T00:11:00.511032+0000","172.21.15.116:6832/3841609112":"2024-04-26T00:07:00.448498+0000","172.21.15.116:6835/1670061091":"2024-04-26T00:09:30.490485+0000","172.21.15.116:6834/1971675962":"2024-04-26T00:25:25.721965+0000","172.21.15.116:6835/1233829678":"2024-04-26T00:07:30.454393+0000","172.21.15.116:6835/2371126056":"2024-04-26T00:27:35.754759+0000","172.21.15.174:6834/2883220846":"2024-04-26T00:11:50.521053+0000","172.21.15.116:6834/1233829678":"2024-04-26T00:07:30.454393+0000","172.21.15.174:6837/3180024481":"2024-04-26T00:15:25.567768+0000","172.21.15.116:6835/879531976":"2024-04-26T00:20:10.642563+0000","172.21.15.116:6833/174781672":"2024-04-26T00:02:40.399073+0000","172.21.15.174:6835/4215220222":"2024-04-26T00:15:40.572498+0000","172.21.15.116:6834/221971311":"2024-04-26T00:22:48.898075+0000","172.21.15.174:6836/1281466397":"2024-04-26T00:18:20.613347+0000","172.21.15.116:6832/3891103804":"2024-04-26T00:28:50.767503+0000","172.21.15.116:6835/1358527848":"2024-04-26T00:11:00.511032+0000","172.21.15.116:6835/221971311":"2024-04-26T00:22:48.898075+0000","172.21.15.116:6832/3006671671":"2024-04-26T00:09:30.490406+0000","172.21.15.116:6834/980221971":"2024-04-26T00:21:40.665115+0000","172.21.15.174:6836/1762356055":"2024-04-26T00:03:36.597345+0000","172.21.15.116:6833/3006671671":"2024-04-26T00:09:30.490406+0000","172.21.15.116:6834/1459177921":"2024-04-26T00:03:30.406778+0000","172.21.15.116:6834/462190523":"2024-04-26T00:14:55.560782+0000","172.21.15.116:6834/879531976":"2024-04-26T00:20:10.642563+0000","172.21.15.116:6832/1459177921":"2024-04-26T00:03:30.406778+0000","172.21.15.174:6837/1159002662":"2024-04-26T00:10:20.502218+0000","172.21.15.174:6837/1762356055":"2024-04-26T00:03:36.597345+0000","172.21.15.174:6834/4215220222":"2024-04-26T00:15:40.572498+0000","172.21.15.116:6835/1971675962":"2024-04-26T00:25:25.721965+0000","172.21.15.174:6836/3081407040":"2024-04-26T00:17:25.598516+0000","172.21.15.116:6834/1599708526":"2024-04-26T00:12:20.526742+0000","172.21.15.174:6836/2276529656":"2024-04-26T00:19:05.629048+0000","172.21.15.174:6836/1159002662":"2024-04-26T00:10:20.502218+0000","172.21.15.116:6833/2998042514":"2024-04-26T00:23:30.691827+0000","172.21.15.174:6837/3081407040":"2024-04-26T00:17:25.598516+0000","172.21.15.174:6835/2883220846":"2024-04-26T00:11:50.521053+0000","172.21.15.174:6837/492366509":"2024-04-26T00:24:15.701887+0000","172.21.15.116:6835/2004876235":"2024-04-26T00:04:45.421611+0000","172.21.15.116:6835/174781672":"2024-04-26T00:02:40.399073+0000","172.21.15.116:6833/3633657268":"2024-04-26T00:25:40.724634+0000","172.21.15.116:6832/365773583":"2024-04-26T00:14:58.857970+0000","172.21.15.174:6836/1889314958":"2024-04-26T00:06:30.441408+0000","172.21.15.116:6833/3705654249":"2024-04-26T00:12:45.533219+0000","172.21.15.116:6832/3705654249":"2024-04-26T00:12:45.533219+0000","172.21.15.116:6834/1670061091":"2024-04-26T00:09:30.490485+0000","172.21.15.116:6835/1599708526":"2024-04-26T00:12:20.526742+0000","172.21.15.116:6832/3633657268":"2024-04-26T00:25:40.724634+0000","172.21.15.174:6836/3583791822":"2024-04-26T00:08:00.460587+0000","172.21.15.174:6837/3583791822":"2024-04-26T00:08:00.460587+0000","172.21.15.174:6836/3398690896":"2024-04-26T00:13:30.543729+0000","172.21.15.174:6837/3398690896":"2024-04-26T00:13:30.543729+0000","172.21.15.116:6835/462190523":"2024-04-26T00:14:55.560782+0000","172.21.15.116:6834/3336339376":"2024-04-26T00:18:35.616887+0000","172.21.15.174:6836/3180024481":"2024-04-26T00:15:25.567768+0000","172.21.15.116:6833/365773583":"2024-04-26T00:14:58.857970+0000","172.21.15.116:6835/2012470875":"2024-04-26T00:16:35.584511+0000","172.21.15.116:6833/2004876235":"2024-04-26T00:04:45.421611+0000","172.21.15.174:6837/1281466397":"2024-04-26T00:18:20.613347+0000","172.21.15.116:6832/2340581953":"2024-04-26T00:16:45.587651+0000","172.21.15.174:6837/2915274073":"2024-04-26T00:29:50.776462+0000","172.21.15.116:6834/1649769369":"2024-04-26T00:04:35.418892+0000","172.21.15.116:6832/1649769369":"2024-04-26T00:04:35.418892+0000","172.21.15.116:6833/2340581953":"2024-04-26T00:16:45.587651+0000","172.21.15.174:6837/1889314958":"2024-04-26T00:06:30.441408+0000","172.21.15.116:6832/3645553116":"2024-04-26T00:17:35.601559+0000","172.21.15.116:6833/3645553116":"2024-04-26T00:17:35.601559+0000","172.21.15.116:6833/3841609112":"2024-04-26T00:07:00.448498+0000","172.21.15.116:6834/2012470875":"2024-04-26T00:16:35.584511+0000","172.21.15.116:6835/3336339376":"2024-04-26T00:18:35.616887+0000","172.21.15.174:6837/2276529656":"2024-04-26T00:19:05.629048+0000","172.21.15.116:6832/1655959433":"2024-04-26T00:19:30.633957+0000","172.21.15.116:6833/1655959433":"2024-04-26T00:19:30.633957+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:30:38.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.355+0000 7fa922881640 1 -- 172.21.15.116:0/3148596704 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa9080678e0 msgr2=0x7fa908069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:38.353 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.355+0000 7fa922881640 1 --2- 172.21.15.116:0/3148596704 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa9080678e0 0x7fa908069da0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fa90c000c10 tx=0x7fa90c009040 comp rx=0 tx=0).stop 2024-04-25T00:30:38.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.355+0000 7fa922881640 1 -- 172.21.15.116:0/3148596704 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa91c13b710 msgr2=0x7fa91c0c0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:38.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.355+0000 7fa922881640 1 --2- 172.21.15.116:0/3148596704 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa91c13b710 0x7fa91c0c0430 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7fa9140b0040 tx=0x7fa914009750 comp rx=0 tx=0).stop 2024-04-25T00:30:38.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.355+0000 7fa922881640 1 -- 172.21.15.116:0/3148596704 shutdown_connections 2024-04-25T00:30:38.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.355+0000 7fa922881640 1 --2- 172.21.15.116:0/3148596704 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa9080678e0 0x7fa908069da0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:38.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.355+0000 7fa922881640 1 --2- 172.21.15.116:0/3148596704 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa91c13b710 0x7fa91c0c0430 unknown :-1 s=CLOSED pgs=1779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:38.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.355+0000 7fa922881640 1 --2- 172.21.15.116:0/3148596704 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa91c1ef160 0x7fa91c072730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:38.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.355+0000 7fa922881640 1 --2- 172.21.15.116:0/3148596704 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa91c13b330 0x7fa91c13fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:38.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.355+0000 7fa922881640 1 -- 172.21.15.116:0/3148596704 >> 172.21.15.116:0/3148596704 conn(0x7fa91c0c3b00 msgr2=0x7fa91c06cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:38.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.355+0000 7fa922881640 1 -- 172.21.15.116:0/3148596704 shutdown_connections 2024-04-25T00:30:38.354 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.355+0000 7fa922881640 1 -- 172.21.15.116:0/3148596704 wait complete. 2024-04-25T00:30:38.371 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T00:00:53.324376+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'a_metadata', 'create_time': '2024-04-25T00:01:03.197326+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'a'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'a_data', 'create_time': '2024-04-25T00:01:03.728618+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'a'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'b_metadata', 'create_time': '2024-04-25T00:01:17.125210+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'b'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'b_data', 'create_time': '2024-04-25T00:01:17.821873+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '27', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'b'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T00:30:38.371 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T00:30:38.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.499+0000 7f0b849bb640 1 Processor -- start 2024-04-25T00:30:38.498 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.499+0000 7f0b849bb640 1 -- start start 2024-04-25T00:30:38.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.499+0000 7f0b849bb640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f0b80152da0 0x7f0b8006da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:38.500 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.499+0000 7f0b849bb640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b8006dfc0 0x7f0b8006b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:38.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.499+0000 7f0b849bb640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b801566c0 con 0x7f0b800adcf0 2024-04-25T00:30:38.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.499+0000 7f0b849bb640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f0b800c31f0 con 0x7f0b80152da0 2024-04-25T00:30:38.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.499+0000 7f0b849bb640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f0b800c28c0 con 0x7f0b8006dfc0 2024-04-25T00:30:38.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.499+0000 7f0b7ed9d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f0b800adcf0 0x7f0b80158e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57380/0 (socket says 172.21.15.116:57380) 2024-04-25T00:30:38.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.499+0000 7f0b7ed9d640 1 -- 172.21.15.116:0/2856598946 learned_addr learned my addr 172.21.15.116:0/2856598946 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:30:38.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.499+0000 7f0b7dd9b640 1 --2- 172.21.15.116:0/2856598946 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b8006dfc0 0x7f0b8006b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:38.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.499+0000 7f0b7e59c640 1 --2- 172.21.15.116:0/2856598946 >> v2:172.21.15.174:3300/0 conn(0x7f0b80152da0 0x7f0b8006da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:38.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.499+0000 7f0b7d59a640 1 -- 172.21.15.116:0/2856598946 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3708478848 0 0) 0x7f0b801566c0 con 0x7f0b800adcf0 2024-04-25T00:30:38.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.499+0000 7f0b7d59a640 1 -- 172.21.15.116:0/2856598946 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b68003530 con 0x7f0b800adcf0 2024-04-25T00:30:38.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.499+0000 7f0b7d59a640 1 -- 172.21.15.116:0/2856598946 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1119377767 0 0) 0x7f0b68003530 con 0x7f0b800adcf0 2024-04-25T00:30:38.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.499+0000 7f0b7d59a640 1 -- 172.21.15.116:0/2856598946 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b8006dfc0 msgr2=0x7f0b8006b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:38.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.499+0000 7f0b7d59a640 1 --2- 172.21.15.116:0/2856598946 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b8006dfc0 0x7f0b8006b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:38.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.499+0000 7f0b7d59a640 1 -- 172.21.15.116:0/2856598946 >> v2:172.21.15.174:3300/0 conn(0x7f0b80152da0 msgr2=0x7f0b8006da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:38.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.499+0000 7f0b7d59a640 1 --2- 172.21.15.116:0/2856598946 >> v2:172.21.15.174:3300/0 conn(0x7f0b80152da0 0x7f0b8006da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:38.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.499+0000 7f0b7d59a640 1 -- 172.21.15.116:0/2856598946 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b800c2ba0 con 0x7f0b800adcf0 2024-04-25T00:30:38.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.499+0000 7f0b7d59a640 1 -- 172.21.15.116:0/2856598946 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f0b78001a00 con 0x7f0b800adcf0 2024-04-25T00:30:38.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.499+0000 7f0b7e59c640 1 --2- 172.21.15.116:0/2856598946 >> v2:172.21.15.174:3300/0 conn(0x7f0b80152da0 0x7f0b8006da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:30:38.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.499+0000 7f0b7d59a640 1 -- 172.21.15.116:0/2856598946 >> v1:172.21.15.116:6789/0 conn(0x7f0b800adcf0 legacy=0x7f0b80158e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:38.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.499+0000 7f0b7d59a640 1 --2- 172.21.15.116:0/2856598946 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b8006dfc0 0x7f0b680039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:38.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.499+0000 7f0b7d59a640 1 --2- 172.21.15.116:0/2856598946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b68003f40 0x7f0b68004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:38.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.499+0000 7f0b7d59a640 1 --2- 172.21.15.116:0/2856598946 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b68004d50 0x7f0b680051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:38.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.499+0000 7f0b7d59a640 1 -- 172.21.15.116:0/2856598946 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0b800c31f0 con 0x7f0b68003f40 2024-04-25T00:30:38.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.499+0000 7f0b7d59a640 1 -- 172.21.15.116:0/2856598946 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0b800c28c0 con 0x7f0b68004d50 2024-04-25T00:30:38.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.499+0000 7f0b7d59a640 1 -- 172.21.15.116:0/2856598946 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f0b6800ba20 con 0x7f0b8006dfc0 2024-04-25T00:30:38.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.499+0000 7f0b7dd9b640 1 --2- 172.21.15.116:0/2856598946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b68003f40 0x7f0b68004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:38.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.499+0000 7f0b7ed9d640 1 --2- 172.21.15.116:0/2856598946 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b68004d50 0x7f0b680051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:38.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.499+0000 7f0b7dd9b640 1 -- 172.21.15.116:0/2856598946 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b8006dfc0 msgr2=0x7f0b680039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:38.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.499+0000 7f0b7e59c640 1 --2- 172.21.15.116:0/2856598946 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b8006dfc0 0x7f0b680039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:38.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.499+0000 7f0b7dd9b640 1 --2- 172.21.15.116:0/2856598946 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b8006dfc0 0x7f0b680039c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:38.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.499+0000 7f0b7dd9b640 1 -- 172.21.15.116:0/2856598946 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b68004d50 msgr2=0x7f0b680051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:38.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.499+0000 7f0b7dd9b640 1 --2- 172.21.15.116:0/2856598946 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b68004d50 0x7f0b680051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:38.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.499+0000 7f0b7dd9b640 1 -- 172.21.15.116:0/2856598946 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b68003530 con 0x7f0b68003f40 2024-04-25T00:30:38.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.499+0000 7f0b7ed9d640 1 --2- 172.21.15.116:0/2856598946 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b68004d50 0x7f0b680051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:30:38.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.503+0000 7f0b7dd9b640 1 --2- 172.21.15.116:0/2856598946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b68003f40 0x7f0b68004810 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f0b74009ab0 tx=0x7f0b7401ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a01da4c773fa289 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:38.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.503+0000 7f0b7d59a640 1 -- 172.21.15.116:0/2856598946 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b740b2070 con 0x7f0b68003f40 2024-04-25T00:30:38.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.503+0000 7f0b7d59a640 1 -- 172.21.15.116:0/2856598946 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b740a5640 con 0x7f0b68003f40 2024-04-25T00:30:38.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.503+0000 7f0b7d59a640 1 -- 172.21.15.116:0/2856598946 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b740096b0 con 0x7f0b68003f40 2024-04-25T00:30:38.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.503+0000 7f0b849bb640 1 -- 172.21.15.116:0/2856598946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b68003f40 msgr2=0x7f0b68004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:38.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.503+0000 7f0b849bb640 1 --2- 172.21.15.116:0/2856598946 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b68003f40 0x7f0b68004810 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f0b74009ab0 tx=0x7f0b7401ca50 comp rx=0 tx=0).stop 2024-04-25T00:30:38.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.503+0000 7f0b7ed9d640 1 -- 172.21.15.116:0/2856598946 reap_dead start 2024-04-25T00:30:38.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.503+0000 7f0b849bb640 1 -- 172.21.15.116:0/2856598946 shutdown_connections 2024-04-25T00:30:38.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.503+0000 7f0b849bb640 1 -- 172.21.15.116:0/2856598946 >> 172.21.15.116:0/2856598946 conn(0x7f0b800c49c0 msgr2=0x7f0b800c4dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:38.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.503+0000 7f0b849bb640 1 -- 172.21.15.116:0/2856598946 shutdown_connections 2024-04-25T00:30:38.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.503+0000 7f0b849bb640 1 -- 172.21.15.116:0/2856598946 wait complete. 2024-04-25T00:30:38.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.503+0000 7f0b849bb640 1 Processor -- start 2024-04-25T00:30:38.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.503+0000 7f0b849bb640 1 -- start start 2024-04-25T00:30:38.507 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.503+0000 7f0b849bb640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b68003f40 0x7f0b801eb5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:38.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.503+0000 7f0b849bb640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b800bd2f0 0x7f0b80159180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:38.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.503+0000 7f0b849bb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b80157e90 0x7f0b801eeb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:38.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.503+0000 7f0b849bb640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0b80153300 con 0x7f0b80157e90 2024-04-25T00:30:38.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.503+0000 7f0b849bb640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0b80153180 con 0x7f0b800bd2f0 2024-04-25T00:30:38.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.503+0000 7f0b849bb640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f0b801562e0 con 0x7f0b68003f40 2024-04-25T00:30:38.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.507+0000 7f0b7ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b80157e90 0x7f0b801eeb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:38.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.507+0000 7f0b7ed9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b80157e90 0x7f0b801eeb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36316/0 (socket says 172.21.15.116:36316) 2024-04-25T00:30:38.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.507+0000 7f0b7ed9d640 1 -- 172.21.15.116:0/1132286513 learned_addr learned my addr 172.21.15.116:0/1132286513 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:38.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.507+0000 7f0b7e59c640 1 --2- 172.21.15.116:0/1132286513 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b68003f40 0x7f0b801eb5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:38.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.507+0000 7f0b7dd9b640 1 --2- 172.21.15.116:0/1132286513 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b800bd2f0 0x7f0b80159180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:38.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.507+0000 7f0b7ed9d640 1 -- 172.21.15.116:0/1132286513 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b68003f40 msgr2=0x7f0b801eb5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:38.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.507+0000 7f0b7ed9d640 1 --2- 172.21.15.116:0/1132286513 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b68003f40 0x7f0b801eb5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:38.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.507+0000 7f0b7ed9d640 1 -- 172.21.15.116:0/1132286513 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b800bd2f0 msgr2=0x7f0b80159180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:38.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.507+0000 7f0b7ed9d640 1 --2- 172.21.15.116:0/1132286513 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b800bd2f0 0x7f0b80159180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:38.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.507+0000 7f0b7ed9d640 1 -- 172.21.15.116:0/1132286513 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b80156df0 con 0x7f0b80157e90 2024-04-25T00:30:38.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.507+0000 7f0b7ed9d640 1 --2- 172.21.15.116:0/1132286513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b80157e90 0x7f0b801eeb30 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f0b78000f90 tx=0x7f0b78010f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:38.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.507+0000 7f0b6effd640 1 -- 172.21.15.116:0/1132286513 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b7801a070 con 0x7f0b80157e90 2024-04-25T00:30:38.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.507+0000 7f0b6effd640 1 -- 172.21.15.116:0/1132286513 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b78011040 con 0x7f0b80157e90 2024-04-25T00:30:38.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.507+0000 7f0b849bb640 1 -- 172.21.15.116:0/1132286513 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b801570d0 con 0x7f0b80157e90 2024-04-25T00:30:38.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.507+0000 7f0b6effd640 1 -- 172.21.15.116:0/1132286513 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b7800ede0 con 0x7f0b80157e90 2024-04-25T00:30:38.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.507+0000 7f0b849bb640 1 -- 172.21.15.116:0/1132286513 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b80157690 con 0x7f0b80157e90 2024-04-25T00:30:38.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.507+0000 7f0b849bb640 1 -- 172.21.15.116:0/1132286513 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b8004f510 con 0x7f0b80157e90 2024-04-25T00:30:38.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.507+0000 7f0b6effd640 1 -- 172.21.15.116:0/1132286513 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f0b7801d070 con 0x7f0b80157e90 2024-04-25T00:30:38.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.507+0000 7f0b6effd640 1 --2- 172.21.15.116:0/1132286513 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0b600678f0 0x7f0b60069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:38.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.507+0000 7f0b6effd640 1 -- 172.21.15.116:0/1132286513 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7f0b780895c0 con 0x7f0b80157e90 2024-04-25T00:30:38.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.507+0000 7f0b7e59c640 1 --2- 172.21.15.116:0/1132286513 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0b600678f0 0x7f0b60069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:38.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.507+0000 7f0b7e59c640 1 --2- 172.21.15.116:0/1132286513 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0b600678f0 0x7f0b60069db0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f0b70009a10 tx=0x7f0b70008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:38.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.511+0000 7f0b6effd640 1 -- 172.21.15.116:0/1132286513 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f0b78058e10 con 0x7f0b80157e90 2024-04-25T00:30:38.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.671+0000 7f0b849bb640 1 -- 172.21.15.116:0/1132286513 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f0b800adad0 con 0x7f0b80157e90 2024-04-25T00:30:38.672 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.671+0000 7f0b6effd640 1 -- 172.21.15.116:0/1132286513 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v74) ==== 93+0+10 (secure 0 0 0) 0x7f0b78058710 con 0x7f0b80157e90 2024-04-25T00:30:38.672 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 1 2024-04-25T00:30:38.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.675+0000 7f0b849bb640 1 -- 172.21.15.116:0/1132286513 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0b600678f0 msgr2=0x7f0b60069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:38.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.675+0000 7f0b849bb640 1 --2- 172.21.15.116:0/1132286513 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0b600678f0 0x7f0b60069db0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f0b70009a10 tx=0x7f0b70008040 comp rx=0 tx=0).stop 2024-04-25T00:30:38.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.675+0000 7f0b849bb640 1 -- 172.21.15.116:0/1132286513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b80157e90 msgr2=0x7f0b801eeb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:38.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.675+0000 7f0b849bb640 1 --2- 172.21.15.116:0/1132286513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b80157e90 0x7f0b801eeb30 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f0b78000f90 tx=0x7f0b78010f20 comp rx=0 tx=0).stop 2024-04-25T00:30:38.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.675+0000 7f0b849bb640 1 -- 172.21.15.116:0/1132286513 shutdown_connections 2024-04-25T00:30:38.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.675+0000 7f0b849bb640 1 --2- 172.21.15.116:0/1132286513 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0b600678f0 0x7f0b60069db0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:38.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.675+0000 7f0b849bb640 1 --2- 172.21.15.116:0/1132286513 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0b80157e90 0x7f0b801eeb30 unknown :-1 s=CLOSED pgs=1782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:38.675 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.675+0000 7f0b849bb640 1 --2- 172.21.15.116:0/1132286513 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0b800bd2f0 0x7f0b80159180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:38.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.675+0000 7f0b849bb640 1 --2- 172.21.15.116:0/1132286513 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0b68003f40 0x7f0b801eb5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:38.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.675+0000 7f0b849bb640 1 -- 172.21.15.116:0/1132286513 >> 172.21.15.116:0/1132286513 conn(0x7f0b800c49c0 msgr2=0x7f0b801550d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:38.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.675+0000 7f0b849bb640 1 -- 172.21.15.116:0/1132286513 shutdown_connections 2024-04-25T00:30:38.676 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.675+0000 7f0b849bb640 1 -- 172.21.15.116:0/1132286513 wait complete. 2024-04-25T00:30:38.692 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_metadata pg_num 2024-04-25T00:30:38.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.811+0000 7f94aa821640 1 Processor -- start 2024-04-25T00:30:38.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.815+0000 7f94aa821640 1 -- start start 2024-04-25T00:30:38.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.815+0000 7f94aa821640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f94a40c9260 0x7f94a40c9710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:38.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.815+0000 7f94aa821640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94a40caa70 con 0x7f94a4153130 2024-04-25T00:30:38.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.815+0000 7f94aa821640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94a40cad50 con 0x7f94a40ca1b0 2024-04-25T00:30:38.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.815+0000 7f94aa821640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f94a40ca820 con 0x7f94a40c9260 2024-04-25T00:30:38.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.815+0000 7f94a3fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f94a4153130 0x7f94a41562e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57386/0 (socket says 172.21.15.116:57386) 2024-04-25T00:30:38.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.815+0000 7f94a3fff640 1 -- 172.21.15.116:0/1128866022 learned_addr learned my addr 172.21.15.116:0/1128866022 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:30:38.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.815+0000 7f94a2ffd640 1 -- 172.21.15.116:0/1128866022 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3355659529 0 0) 0x7f94a40caa70 con 0x7f94a4153130 2024-04-25T00:30:38.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.815+0000 7f94a8dbe640 1 --2- 172.21.15.116:0/1128866022 >> v2:172.21.15.174:3300/0 conn(0x7f94a40c9260 0x7f94a40c9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:38.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.815+0000 7f94a2ffd640 1 -- 172.21.15.116:0/1128866022 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9490003530 con 0x7f94a4153130 2024-04-25T00:30:38.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.815+0000 7f94a2ffd640 1 -- 172.21.15.116:0/1128866022 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1213632676 0 0) 0x7f9490003530 con 0x7f94a4153130 2024-04-25T00:30:38.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.815+0000 7f94a2ffd640 1 -- 172.21.15.116:0/1128866022 >> v2:172.21.15.174:3300/0 conn(0x7f94a40c9260 msgr2=0x7f94a40c9710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:38.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.815+0000 7f94a2ffd640 1 --2- 172.21.15.116:0/1128866022 >> v2:172.21.15.174:3300/0 conn(0x7f94a40c9260 0x7f94a40c9710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:38.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.815+0000 7f94a2ffd640 1 -- 172.21.15.116:0/1128866022 >> v1:172.21.15.174:6789/0 conn(0x7f94a40ca1b0 legacy=0x7f94a406bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:38.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.815+0000 7f94a2ffd640 1 -- 172.21.15.116:0/1128866022 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94a4155ae0 con 0x7f94a4153130 2024-04-25T00:30:38.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.815+0000 7f94a2ffd640 1 -- 172.21.15.116:0/1128866022 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f949c001680 con 0x7f94a4153130 2024-04-25T00:30:38.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.815+0000 7f94a2ffd640 1 -- 172.21.15.116:0/1128866022 >> v1:172.21.15.116:6789/0 conn(0x7f94a4153130 legacy=0x7f94a41562e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:38.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.815+0000 7f94a2ffd640 1 --2- 172.21.15.116:0/1128866022 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f94900039c0 0x7f9490003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:38.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.815+0000 7f94a2ffd640 1 --2- 172.21.15.116:0/1128866022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9490004810 0x7f9490004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:38.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.815+0000 7f94a2ffd640 1 --2- 172.21.15.116:0/1128866022 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f94900052f0 0x7f949000b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:38.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.815+0000 7f94a2ffd640 1 -- 172.21.15.116:0/1128866022 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f94a40ca820 con 0x7f9490004810 2024-04-25T00:30:38.817 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.815+0000 7f94a2ffd640 1 -- 172.21.15.116:0/1128866022 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f949000c1c0 con 0x7f94900052f0 2024-04-25T00:30:38.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.815+0000 7f94a2ffd640 1 -- 172.21.15.116:0/1128866022 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f949000c3c0 con 0x7f94900039c0 2024-04-25T00:30:38.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.815+0000 7f94a37fe640 1 --2- 172.21.15.116:0/1128866022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9490004810 0x7f9490004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:38.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.815+0000 7f94a3fff640 1 --2- 172.21.15.116:0/1128866022 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f94900039c0 0x7f9490003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:38.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.815+0000 7f94a8dbe640 1 --2- 172.21.15.116:0/1128866022 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f94900052f0 0x7f949000b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:38.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.815+0000 7f94a37fe640 1 -- 172.21.15.116:0/1128866022 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f94900039c0 msgr2=0x7f9490003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:38.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.815+0000 7f94a37fe640 1 --2- 172.21.15.116:0/1128866022 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f94900039c0 0x7f9490003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:38.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.815+0000 7f94a37fe640 1 -- 172.21.15.116:0/1128866022 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f94900052f0 msgr2=0x7f949000b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:38.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.815+0000 7f94a37fe640 1 --2- 172.21.15.116:0/1128866022 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f94900052f0 0x7f949000b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:38.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.815+0000 7f94a8dbe640 1 -- 172.21.15.116:0/1128866022 reap_dead start 2024-04-25T00:30:38.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.815+0000 7f94a37fe640 1 -- 172.21.15.116:0/1128866022 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9490003530 con 0x7f9490004810 2024-04-25T00:30:38.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.815+0000 7f94a3fff640 1 --2- 172.21.15.116:0/1128866022 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f94900039c0 0x7f9490003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:30:38.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.819+0000 7f94a37fe640 1 --2- 172.21.15.116:0/1128866022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9490004810 0x7f9490004c90 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f94a404fec0 tx=0x7f948c01c700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5671dbd64069aec6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:38.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.819+0000 7f94a2ffd640 1 -- 172.21.15.116:0/1128866022 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f948c0b0070 con 0x7f9490004810 2024-04-25T00:30:38.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.819+0000 7f94a2ffd640 1 -- 172.21.15.116:0/1128866022 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f948c003b80 con 0x7f9490004810 2024-04-25T00:30:38.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.819+0000 7f94a2ffd640 1 -- 172.21.15.116:0/1128866022 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f948c001a90 con 0x7f9490004810 2024-04-25T00:30:38.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.819+0000 7f94aa821640 1 -- 172.21.15.116:0/1128866022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9490004810 msgr2=0x7f9490004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:38.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.819+0000 7f94aa821640 1 --2- 172.21.15.116:0/1128866022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9490004810 0x7f9490004c90 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f94a404fec0 tx=0x7f948c01c700 comp rx=0 tx=0).stop 2024-04-25T00:30:38.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.819+0000 7f94aa821640 1 -- 172.21.15.116:0/1128866022 shutdown_connections 2024-04-25T00:30:38.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.819+0000 7f94aa821640 1 --2- 172.21.15.116:0/1128866022 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9490004810 0x7f9490004c90 unknown :-1 s=CLOSED pgs=1784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:38.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.819+0000 7f94aa821640 1 -- 172.21.15.116:0/1128866022 >> 172.21.15.116:0/1128866022 conn(0x7f94a40cb310 msgr2=0x7f94a40cb720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:38.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.819+0000 7f94aa821640 1 -- 172.21.15.116:0/1128866022 shutdown_connections 2024-04-25T00:30:38.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.819+0000 7f94aa821640 1 -- 172.21.15.116:0/1128866022 wait complete. 2024-04-25T00:30:38.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.819+0000 7f94aa821640 1 Processor -- start 2024-04-25T00:30:38.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.819+0000 7f94aa821640 1 -- start start 2024-04-25T00:30:38.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.819+0000 7f94aa821640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9490004810 0x7f94a41f2630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:38.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.819+0000 7f94aa821640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f94a414bf90 0x7f94a40c1440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:38.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.819+0000 7f94aa821640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f94a41f2b70 0x7f94a40c1980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:38.822 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.819+0000 7f94aa821640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f94a4158e40 con 0x7f9490004810 2024-04-25T00:30:38.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.819+0000 7f94aa821640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f94a4155e50 con 0x7f94a41f2b70 2024-04-25T00:30:38.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.819+0000 7f94aa821640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f94a4158fc0 con 0x7f94a414bf90 2024-04-25T00:30:38.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.819+0000 7f94a3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9490004810 0x7f94a41f2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:38.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.819+0000 7f94a3fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9490004810 0x7f94a41f2630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36336/0 (socket says 172.21.15.116:36336) 2024-04-25T00:30:38.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.819+0000 7f94a3fff640 1 -- 172.21.15.116:0/3773396839 learned_addr learned my addr 172.21.15.116:0/3773396839 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:38.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.819+0000 7f94a8dbe640 1 --2- 172.21.15.116:0/3773396839 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f94a41f2b70 0x7f94a40c1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:38.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.819+0000 7f94a37fe640 1 --2- 172.21.15.116:0/3773396839 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f94a414bf90 0x7f94a40c1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:38.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.819+0000 7f94a3fff640 1 -- 172.21.15.116:0/3773396839 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f94a414bf90 msgr2=0x7f94a40c1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:38.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.819+0000 7f94a3fff640 1 --2- 172.21.15.116:0/3773396839 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f94a414bf90 0x7f94a40c1440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:38.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.819+0000 7f94a3fff640 1 -- 172.21.15.116:0/3773396839 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f94a41f2b70 msgr2=0x7f94a40c1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:38.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.819+0000 7f94a3fff640 1 --2- 172.21.15.116:0/3773396839 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f94a41f2b70 0x7f94a40c1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:38.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.819+0000 7f94a3fff640 1 -- 172.21.15.116:0/3773396839 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94a40ac6d0 con 0x7f9490004810 2024-04-25T00:30:38.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.819+0000 7f94a37fe640 1 --2- 172.21.15.116:0/3773396839 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f94a414bf90 0x7f94a40c1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:30:38.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.819+0000 7f94a3fff640 1 --2- 172.21.15.116:0/3773396839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9490004810 0x7f94a41f2630 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f949c00d7a0 tx=0x7f949c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:38.823 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.819+0000 7f94a0ff9640 1 -- 172.21.15.116:0/3773396839 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f949c019070 con 0x7f9490004810 2024-04-25T00:30:38.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.819+0000 7f94a0ff9640 1 -- 172.21.15.116:0/3773396839 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f949c00de50 con 0x7f9490004810 2024-04-25T00:30:38.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.819+0000 7f94a0ff9640 1 -- 172.21.15.116:0/3773396839 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f949c0047f0 con 0x7f9490004810 2024-04-25T00:30:38.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.819+0000 7f94aa821640 1 -- 172.21.15.116:0/3773396839 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94a40ac9b0 con 0x7f9490004810 2024-04-25T00:30:38.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.819+0000 7f94aa821640 1 -- 172.21.15.116:0/3773396839 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94a406c690 con 0x7f9490004810 2024-04-25T00:30:38.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.823+0000 7f94aa821640 1 -- 172.21.15.116:0/3773396839 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94a41f32e0 con 0x7f9490004810 2024-04-25T00:30:38.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.823+0000 7f94a0ff9640 1 -- 172.21.15.116:0/3773396839 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f949c004990 con 0x7f9490004810 2024-04-25T00:30:38.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.823+0000 7f94a0ff9640 1 --2- 172.21.15.116:0/3773396839 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f94840678f0 0x7f9484069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:38.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.823+0000 7f94a0ff9640 1 -- 172.21.15.116:0/3773396839 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7f949c086ae0 con 0x7f9490004810 2024-04-25T00:30:38.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.823+0000 7f94a37fe640 1 --2- 172.21.15.116:0/3773396839 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f94840678f0 0x7f9484069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:38.825 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.823+0000 7f94a37fe640 1 --2- 172.21.15.116:0/3773396839 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f94840678f0 0x7f9484069db0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f948c0ad000 tx=0x7f948c001520 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:38.827 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.827+0000 7f94a0ff9640 1 -- 172.21.15.116:0/3773396839 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f94a41f32e0 con 0x7f9490004810 2024-04-25T00:30:38.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.975+0000 7f94aa821640 1 -- 172.21.15.116:0/3773396839 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"} v 0) -- 0x7f94a404f510 con 0x7f9490004810 2024-04-25T00:30:38.975 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.975+0000 7f94a0ff9640 1 -- 172.21.15.116:0/3773396839 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_metadata", "var": "pg_num"}]=0 v74) ==== 99+0+11 (secure 0 0 0) 0x7f949c055b40 con 0x7f9490004810 2024-04-25T00:30:38.976 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-04-25T00:30:38.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.979+0000 7f94aa821640 1 -- 172.21.15.116:0/3773396839 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f94840678f0 msgr2=0x7f9484069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:38.978 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.979+0000 7f94aa821640 1 --2- 172.21.15.116:0/3773396839 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f94840678f0 0x7f9484069db0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f948c0ad000 tx=0x7f948c001520 comp rx=0 tx=0).stop 2024-04-25T00:30:38.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.979+0000 7f94aa821640 1 -- 172.21.15.116:0/3773396839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9490004810 msgr2=0x7f94a41f2630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:38.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.979+0000 7f94aa821640 1 --2- 172.21.15.116:0/3773396839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9490004810 0x7f94a41f2630 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f949c00d7a0 tx=0x7f949c00dc60 comp rx=0 tx=0).stop 2024-04-25T00:30:38.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.979+0000 7f94aa821640 1 -- 172.21.15.116:0/3773396839 shutdown_connections 2024-04-25T00:30:38.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.979+0000 7f94aa821640 1 --2- 172.21.15.116:0/3773396839 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f94840678f0 0x7f9484069db0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:38.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.979+0000 7f94aa821640 1 --2- 172.21.15.116:0/3773396839 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f94a41f2b70 0x7f94a40c1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:38.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.979+0000 7f94aa821640 1 --2- 172.21.15.116:0/3773396839 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f94a414bf90 0x7f94a40c1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:38.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.979+0000 7f94aa821640 1 --2- 172.21.15.116:0/3773396839 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9490004810 0x7f94a41f2630 unknown :-1 s=CLOSED pgs=1785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:38.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.979+0000 7f94aa821640 1 -- 172.21.15.116:0/3773396839 >> 172.21.15.116:0/3773396839 conn(0x7f94a40cb310 msgr2=0x7f94a41495b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:38.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.979+0000 7f94aa821640 1 -- 172.21.15.116:0/3773396839 shutdown_connections 2024-04-25T00:30:38.979 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:38.979+0000 7f94aa821640 1 -- 172.21.15.116:0/3773396839 wait complete. 2024-04-25T00:30:38.994 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get a_data pg_num 2024-04-25T00:30:39.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.119+0000 7f28af17f640 1 Processor -- start 2024-04-25T00:30:39.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.119+0000 7f28af17f640 1 -- start start 2024-04-25T00:30:39.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.119+0000 7f28af17f640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f28a8150d60 0x7f28a80c2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:39.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.119+0000 7f28af17f640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f28a80a9630 0x7f28a8156e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:39.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.119+0000 7f28af17f640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28a80a9cb0 con 0x7f28a80c2e70 2024-04-25T00:30:39.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.119+0000 7f28af17f640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f28a80a9f90 con 0x7f28a8150d60 2024-04-25T00:30:39.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.119+0000 7f28af17f640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f28a80aa270 con 0x7f28a80a9630 2024-04-25T00:30:39.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.119+0000 7f28acf1b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f28a8150d60 0x7f28a80c2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:39.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.119+0000 7f28acf1b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f28a8150d60 0x7f28a80c2930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36350/0 (socket says 172.21.15.116:36350) 2024-04-25T00:30:39.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.119+0000 7f28acf1b640 1 -- 172.21.15.116:0/1043334843 learned_addr learned my addr 172.21.15.116:0/1043334843 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:39.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.119+0000 7f28a7fff640 1 --2- 172.21.15.116:0/1043334843 >> v2:172.21.15.174:3300/0 conn(0x7f28a80a9630 0x7f28a8156e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:39.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.119+0000 7f28acf1b640 1 -- 172.21.15.116:0/1043334843 >> v2:172.21.15.174:3300/0 conn(0x7f28a80a9630 msgr2=0x7f28a8156e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:39.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.119+0000 7f28acf1b640 1 --2- 172.21.15.116:0/1043334843 >> v2:172.21.15.174:3300/0 conn(0x7f28a80a9630 0x7f28a8156e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:39.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.119+0000 7f28acf1b640 1 -- 172.21.15.116:0/1043334843 >> v1:172.21.15.174:6789/0 conn(0x7f28a80c2e70 legacy=0x7f28a8157340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:39.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.119+0000 7f28acf1b640 1 -- 172.21.15.116:0/1043334843 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28a80aa550 con 0x7f28a8150d60 2024-04-25T00:30:39.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.123+0000 7f28acf1b640 1 --2- 172.21.15.116:0/1043334843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28a8150d60 0x7f28a80c2930 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f28a80c1350 tx=0x7f289c01c390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b54f420e77b94adc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:39.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.123+0000 7f28a77fe640 1 -- 172.21.15.116:0/1043334843 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f289c0b1070 con 0x7f28a8150d60 2024-04-25T00:30:39.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.123+0000 7f28a77fe640 1 -- 172.21.15.116:0/1043334843 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f289c0ac410 con 0x7f28a8150d60 2024-04-25T00:30:39.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.123+0000 7f28a77fe640 1 -- 172.21.15.116:0/1043334843 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f289c0ac7e0 con 0x7f28a8150d60 2024-04-25T00:30:39.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.123+0000 7f28af17f640 1 -- 172.21.15.116:0/1043334843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28a8150d60 msgr2=0x7f28a80c2930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:39.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.123+0000 7f28af17f640 1 --2- 172.21.15.116:0/1043334843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28a8150d60 0x7f28a80c2930 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f28a80c1350 tx=0x7f289c01c390 comp rx=0 tx=0).stop 2024-04-25T00:30:39.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.123+0000 7f28af17f640 1 -- 172.21.15.116:0/1043334843 shutdown_connections 2024-04-25T00:30:39.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.123+0000 7f28af17f640 1 --2- 172.21.15.116:0/1043334843 >> v2:172.21.15.174:3300/0 conn(0x7f28a80a9630 0x7f28a8156e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:39.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.123+0000 7f28af17f640 1 --2- 172.21.15.116:0/1043334843 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28a8150d60 0x7f28a80c2930 unknown :-1 s=CLOSED pgs=1786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:39.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.123+0000 7f28af17f640 1 -- 172.21.15.116:0/1043334843 >> 172.21.15.116:0/1043334843 conn(0x7f28a80c3a20 msgr2=0x7f28a80c3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:39.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.123+0000 7f28af17f640 1 -- 172.21.15.116:0/1043334843 shutdown_connections 2024-04-25T00:30:39.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.123+0000 7f28af17f640 1 -- 172.21.15.116:0/1043334843 wait complete. 2024-04-25T00:30:39.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.123+0000 7f28af17f640 1 Processor -- start 2024-04-25T00:30:39.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.123+0000 7f28af17f640 1 -- start start 2024-04-25T00:30:39.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.123+0000 7f28af17f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28a80a9630 0x7f28a814ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:39.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.123+0000 7f28af17f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28a80c2e70 0x7f28a814a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:39.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.123+0000 7f28af17f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28a8150d60 0x7f28a814a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:39.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.123+0000 7f28af17f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f28a8071fd0 con 0x7f28a80c2e70 2024-04-25T00:30:39.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.123+0000 7f28af17f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f28a80ab7a0 con 0x7f28a80a9630 2024-04-25T00:30:39.125 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.123+0000 7f28af17f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f28a8072150 con 0x7f28a8150d60 2024-04-25T00:30:39.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.123+0000 7f28a7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28a80c2e70 0x7f28a814a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:39.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.123+0000 7f28a7fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28a80c2e70 0x7f28a814a020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36358/0 (socket says 172.21.15.116:36358) 2024-04-25T00:30:39.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.123+0000 7f28a7fff640 1 -- 172.21.15.116:0/3106794856 learned_addr learned my addr 172.21.15.116:0/3106794856 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:39.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.123+0000 7f28acf1b640 1 --2- 172.21.15.116:0/3106794856 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28a80a9630 0x7f28a814ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:39.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.123+0000 7f28ad71c640 1 --2- 172.21.15.116:0/3106794856 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28a8150d60 0x7f28a814a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:39.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.123+0000 7f28a7fff640 1 -- 172.21.15.116:0/3106794856 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28a8150d60 msgr2=0x7f28a814a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:39.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.123+0000 7f28a7fff640 1 --2- 172.21.15.116:0/3106794856 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28a8150d60 0x7f28a814a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:39.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.123+0000 7f28a7fff640 1 -- 172.21.15.116:0/3106794856 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28a80a9630 msgr2=0x7f28a814ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:39.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.123+0000 7f28a7fff640 1 --2- 172.21.15.116:0/3106794856 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28a80a9630 0x7f28a814ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:39.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.123+0000 7f28a7fff640 1 -- 172.21.15.116:0/3106794856 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28a814c5c0 con 0x7f28a80c2e70 2024-04-25T00:30:39.126 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.123+0000 7f28acf1b640 1 --2- 172.21.15.116:0/3106794856 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28a80a9630 0x7f28a814ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:30:39.127 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.123+0000 7f28a7fff640 1 --2- 172.21.15.116:0/3106794856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28a80c2e70 0x7f28a814a020 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f289400c7f0 tx=0x7f289400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:39.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.123+0000 7f28a57fa640 1 -- 172.21.15.116:0/3106794856 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2894015070 con 0x7f28a80c2e70 2024-04-25T00:30:39.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.123+0000 7f28a57fa640 1 -- 172.21.15.116:0/3106794856 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2894007820 con 0x7f28a80c2e70 2024-04-25T00:30:39.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.123+0000 7f28a57fa640 1 -- 172.21.15.116:0/3106794856 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2894007c10 con 0x7f28a80c2e70 2024-04-25T00:30:39.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.123+0000 7f28af17f640 1 -- 172.21.15.116:0/3106794856 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28a8148180 con 0x7f28a80c2e70 2024-04-25T00:30:39.128 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.123+0000 7f28af17f640 1 -- 172.21.15.116:0/3106794856 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28a80c6c50 con 0x7f28a80c2e70 2024-04-25T00:30:39.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.127+0000 7f28af17f640 1 -- 172.21.15.116:0/3106794856 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28a804f510 con 0x7f28a80c2e70 2024-04-25T00:30:39.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.127+0000 7f28a57fa640 1 -- 172.21.15.116:0/3106794856 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2894018070 con 0x7f28a80c2e70 2024-04-25T00:30:39.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.127+0000 7f28a57fa640 1 --2- 172.21.15.116:0/3106794856 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f288c0678c0 0x7f288c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:39.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.127+0000 7f28a57fa640 1 -- 172.21.15.116:0/3106794856 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7f2894087030 con 0x7f28a80c2e70 2024-04-25T00:30:39.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.127+0000 7f28acf1b640 1 --2- 172.21.15.116:0/3106794856 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f288c0678c0 0x7f288c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:39.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.127+0000 7f28acf1b640 1 --2- 172.21.15.116:0/3106794856 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f288c0678c0 0x7f288c069d80 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f289c0ae000 tx=0x7f289c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:39.130 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.131+0000 7f28a57fa640 1 -- 172.21.15.116:0/3106794856 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f2894056880 con 0x7f28a80c2e70 2024-04-25T00:30:39.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.287+0000 7f28af17f640 1 -- 172.21.15.116:0/3106794856 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"} v 0) -- 0x7f28a8071db0 con 0x7f28a80c2e70 2024-04-25T00:30:39.287 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.287+0000 7f28a57fa640 1 -- 172.21.15.116:0/3106794856 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "a_data", "var": "pg_num"}]=0 v74) ==== 95+0+11 (secure 0 0 0) 0x7f2894056180 con 0x7f28a80c2e70 2024-04-25T00:30:39.288 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-04-25T00:30:39.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.291+0000 7f28af17f640 1 -- 172.21.15.116:0/3106794856 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f288c0678c0 msgr2=0x7f288c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:39.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.291+0000 7f28af17f640 1 --2- 172.21.15.116:0/3106794856 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f288c0678c0 0x7f288c069d80 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f289c0ae000 tx=0x7f289c008040 comp rx=0 tx=0).stop 2024-04-25T00:30:39.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.291+0000 7f28af17f640 1 -- 172.21.15.116:0/3106794856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28a80c2e70 msgr2=0x7f28a814a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:39.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.291+0000 7f28af17f640 1 --2- 172.21.15.116:0/3106794856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28a80c2e70 0x7f28a814a020 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f289400c7f0 tx=0x7f289400ccb0 comp rx=0 tx=0).stop 2024-04-25T00:30:39.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.291+0000 7f28af17f640 1 -- 172.21.15.116:0/3106794856 shutdown_connections 2024-04-25T00:30:39.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.291+0000 7f28af17f640 1 --2- 172.21.15.116:0/3106794856 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f288c0678c0 0x7f288c069d80 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:39.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.291+0000 7f28af17f640 1 --2- 172.21.15.116:0/3106794856 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f28a8150d60 0x7f28a814a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:39.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.291+0000 7f28af17f640 1 --2- 172.21.15.116:0/3106794856 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f28a80c2e70 0x7f28a814a020 unknown :-1 s=CLOSED pgs=1787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:39.290 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.291+0000 7f28af17f640 1 --2- 172.21.15.116:0/3106794856 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f28a80a9630 0x7f28a814ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:39.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.291+0000 7f28af17f640 1 -- 172.21.15.116:0/3106794856 >> 172.21.15.116:0/3106794856 conn(0x7f28a80c3a20 msgr2=0x7f28a8072360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:39.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.291+0000 7f28af17f640 1 -- 172.21.15.116:0/3106794856 shutdown_connections 2024-04-25T00:30:39.291 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.291+0000 7f28af17f640 1 -- 172.21.15.116:0/3106794856 wait complete. 2024-04-25T00:30:39.306 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_metadata pg_num 2024-04-25T00:30:39.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.415+0000 7fc91833e640 1 Processor -- start 2024-04-25T00:30:39.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.415+0000 7fc91833e640 1 -- start start 2024-04-25T00:30:39.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.419+0000 7fc91833e640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fc91006ea20 0x7fc91006dc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:39.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.419+0000 7fc91833e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc9100c3500 0x7fc91006e180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:39.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.419+0000 7fc91833e640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc91006d1a0 con 0x7fc9100aa550 2024-04-25T00:30:39.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.419+0000 7fc91833e640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fc910152c30 con 0x7fc9100c3500 2024-04-25T00:30:39.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.419+0000 7fc91833e640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fc910152670 con 0x7fc91006ea20 2024-04-25T00:30:39.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.419+0000 7fc9158d9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc9100c3500 0x7fc91006e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:39.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.419+0000 7fc9158d9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc9100c3500 0x7fc91006e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36370/0 (socket says 172.21.15.116:36370) 2024-04-25T00:30:39.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.419+0000 7fc9158d9640 1 -- 172.21.15.116:0/4177800676 learned_addr learned my addr 172.21.15.116:0/4177800676 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:39.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.419+0000 7fc9160da640 1 --2- 172.21.15.116:0/4177800676 >> v2:172.21.15.174:3300/0 conn(0x7fc91006ea20 0x7fc91006dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:39.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.419+0000 7fc9158d9640 1 -- 172.21.15.116:0/4177800676 >> v2:172.21.15.174:3300/0 conn(0x7fc91006ea20 msgr2=0x7fc91006dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:39.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.419+0000 7fc9158d9640 1 --2- 172.21.15.116:0/4177800676 >> v2:172.21.15.174:3300/0 conn(0x7fc91006ea20 0x7fc91006dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:39.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.419+0000 7fc9158d9640 1 -- 172.21.15.116:0/4177800676 >> v1:172.21.15.174:6789/0 conn(0x7fc9100aa550 legacy=0x7fc91014edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:39.419 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.419+0000 7fc9158d9640 1 -- 172.21.15.116:0/4177800676 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc910152950 con 0x7fc9100c3500 2024-04-25T00:30:39.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.419+0000 7fc9158d9640 1 --2- 172.21.15.116:0/4177800676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9100c3500 0x7fc91006e180 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7fc910073410 tx=0x7fc90001c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad41798e16238bf4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:39.420 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.419+0000 7fc9150d8640 1 -- 172.21.15.116:0/4177800676 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9000b0070 con 0x7fc9100c3500 2024-04-25T00:30:39.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.419+0000 7fc9150d8640 1 -- 172.21.15.116:0/4177800676 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9000ab600 con 0x7fc9100c3500 2024-04-25T00:30:39.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.419+0000 7fc9150d8640 1 -- 172.21.15.116:0/4177800676 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9000ab9d0 con 0x7fc9100c3500 2024-04-25T00:30:39.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.423+0000 7fc91833e640 1 -- 172.21.15.116:0/4177800676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9100c3500 msgr2=0x7fc91006e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:39.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.423+0000 7fc91833e640 1 --2- 172.21.15.116:0/4177800676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9100c3500 0x7fc91006e180 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7fc910073410 tx=0x7fc90001c450 comp rx=0 tx=0).stop 2024-04-25T00:30:39.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.423+0000 7fc91833e640 1 -- 172.21.15.116:0/4177800676 shutdown_connections 2024-04-25T00:30:39.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.423+0000 7fc91833e640 1 --2- 172.21.15.116:0/4177800676 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9100c3500 0x7fc91006e180 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:39.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.423+0000 7fc91833e640 1 --2- 172.21.15.116:0/4177800676 >> v2:172.21.15.174:3300/0 conn(0x7fc91006ea20 0x7fc91006dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:39.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.423+0000 7fc91833e640 1 -- 172.21.15.116:0/4177800676 >> 172.21.15.116:0/4177800676 conn(0x7fc9100c3a20 msgr2=0x7fc9100c3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:39.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.423+0000 7fc91833e640 1 -- 172.21.15.116:0/4177800676 shutdown_connections 2024-04-25T00:30:39.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.423+0000 7fc91833e640 1 -- 172.21.15.116:0/4177800676 wait complete. 2024-04-25T00:30:39.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.423+0000 7fc91833e640 1 Processor -- start 2024-04-25T00:30:39.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.423+0000 7fc91833e640 1 -- start start 2024-04-25T00:30:39.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.423+0000 7fc91833e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc91006ea20 0x7fc910139230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:39.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.423+0000 7fc91833e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc9100aa550 0x7fc910139770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:39.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.423+0000 7fc91833e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9100c3500 0x7fc9101efea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:39.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.423+0000 7fc91833e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc91006f050 con 0x7fc9100c3500 2024-04-25T00:30:39.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.423+0000 7fc91833e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc9100c2930 con 0x7fc9100aa550 2024-04-25T00:30:39.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.423+0000 7fc91833e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fc91014e7e0 con 0x7fc91006ea20 2024-04-25T00:30:39.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.423+0000 7fc9168db640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9100c3500 0x7fc9101efea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:39.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.423+0000 7fc9168db640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9100c3500 0x7fc9101efea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36382/0 (socket says 172.21.15.116:36382) 2024-04-25T00:30:39.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.423+0000 7fc9168db640 1 -- 172.21.15.116:0/1741104685 learned_addr learned my addr 172.21.15.116:0/1741104685 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:39.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.423+0000 7fc9160da640 1 --2- 172.21.15.116:0/1741104685 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc91006ea20 0x7fc910139230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:39.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.423+0000 7fc9158d9640 1 --2- 172.21.15.116:0/1741104685 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc9100aa550 0x7fc910139770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:39.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.423+0000 7fc9168db640 1 -- 172.21.15.116:0/1741104685 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc91006ea20 msgr2=0x7fc910139230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:39.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.423+0000 7fc9168db640 1 --2- 172.21.15.116:0/1741104685 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc91006ea20 0x7fc910139230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:39.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.423+0000 7fc9168db640 1 -- 172.21.15.116:0/1741104685 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc9100aa550 msgr2=0x7fc910139770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:39.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.423+0000 7fc9168db640 1 --2- 172.21.15.116:0/1741104685 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc9100aa550 0x7fc910139770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:39.425 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.423+0000 7fc9168db640 1 -- 172.21.15.116:0/1741104685 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc910151400 con 0x7fc9100c3500 2024-04-25T00:30:39.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.423+0000 7fc9168db640 1 --2- 172.21.15.116:0/1741104685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9100c3500 0x7fc9101efea0 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7fc90800c780 tx=0x7fc90800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:39.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.423+0000 7fc906ffd640 1 -- 172.21.15.116:0/1741104685 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc908017070 con 0x7fc9100c3500 2024-04-25T00:30:39.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.423+0000 7fc906ffd640 1 -- 172.21.15.116:0/1741104685 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc908003db0 con 0x7fc9100c3500 2024-04-25T00:30:39.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.423+0000 7fc91833e640 1 -- 172.21.15.116:0/1741104685 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc910151980 con 0x7fc9100c3500 2024-04-25T00:30:39.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.423+0000 7fc906ffd640 1 -- 172.21.15.116:0/1741104685 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9080052b0 con 0x7fc9100c3500 2024-04-25T00:30:39.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.423+0000 7fc91833e640 1 -- 172.21.15.116:0/1741104685 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9101ef980 con 0x7fc9100c3500 2024-04-25T00:30:39.426 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.423+0000 7fc91833e640 1 -- 172.21.15.116:0/1741104685 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc91004f510 con 0x7fc9100c3500 2024-04-25T00:30:39.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.427+0000 7fc906ffd640 1 -- 172.21.15.116:0/1741104685 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc90801a070 con 0x7fc9100c3500 2024-04-25T00:30:39.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.427+0000 7fc906ffd640 1 --2- 172.21.15.116:0/1741104685 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc8f40678c0 0x7fc8f4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:39.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.427+0000 7fc906ffd640 1 -- 172.21.15.116:0/1741104685 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7fc908086670 con 0x7fc9100c3500 2024-04-25T00:30:39.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.427+0000 7fc9160da640 1 --2- 172.21.15.116:0/1741104685 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc8f40678c0 0x7fc8f4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:39.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.427+0000 7fc9160da640 1 --2- 172.21.15.116:0/1741104685 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc8f40678c0 0x7fc8f4069d80 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fc91013b5d0 tx=0x7fc90c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:39.429 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.431+0000 7fc906ffd640 1 -- 172.21.15.116:0/1741104685 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fc908055e30 con 0x7fc9100c3500 2024-04-25T00:30:39.579 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.579+0000 7fc91833e640 1 -- 172.21.15.116:0/1741104685 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"} v 0) -- 0x7fc91013d350 con 0x7fc9100c3500 2024-04-25T00:30:39.580 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.579+0000 7fc906ffd640 1 -- 172.21.15.116:0/1741104685 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_metadata", "var": "pg_num"}]=0 v74) ==== 99+0+11 (secure 0 0 0) 0x7fc908055730 con 0x7fc9100c3500 2024-04-25T00:30:39.580 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 32 2024-04-25T00:30:39.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.583+0000 7fc91833e640 1 -- 172.21.15.116:0/1741104685 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc8f40678c0 msgr2=0x7fc8f4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:39.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.583+0000 7fc91833e640 1 --2- 172.21.15.116:0/1741104685 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc8f40678c0 0x7fc8f4069d80 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fc91013b5d0 tx=0x7fc90c008040 comp rx=0 tx=0).stop 2024-04-25T00:30:39.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.583+0000 7fc91833e640 1 -- 172.21.15.116:0/1741104685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9100c3500 msgr2=0x7fc9101efea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:39.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.583+0000 7fc91833e640 1 --2- 172.21.15.116:0/1741104685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9100c3500 0x7fc9101efea0 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7fc90800c780 tx=0x7fc90800cc40 comp rx=0 tx=0).stop 2024-04-25T00:30:39.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.583+0000 7fc91833e640 1 -- 172.21.15.116:0/1741104685 shutdown_connections 2024-04-25T00:30:39.582 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.583+0000 7fc91833e640 1 --2- 172.21.15.116:0/1741104685 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc8f40678c0 0x7fc8f4069d80 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:39.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.583+0000 7fc91833e640 1 --2- 172.21.15.116:0/1741104685 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc9100c3500 0x7fc9101efea0 unknown :-1 s=CLOSED pgs=1789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:39.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.583+0000 7fc91833e640 1 --2- 172.21.15.116:0/1741104685 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc9100aa550 0x7fc910139770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:39.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.583+0000 7fc91833e640 1 --2- 172.21.15.116:0/1741104685 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc91006ea20 0x7fc910139230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:39.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.583+0000 7fc91833e640 1 -- 172.21.15.116:0/1741104685 >> 172.21.15.116:0/1741104685 conn(0x7fc9100c3a20 msgr2=0x7fc91006db40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:39.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.583+0000 7fc91833e640 1 -- 172.21.15.116:0/1741104685 shutdown_connections 2024-04-25T00:30:39.583 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.583+0000 7fc91833e640 1 -- 172.21.15.116:0/1741104685 wait complete. 2024-04-25T00:30:39.598 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get b_data pg_num 2024-04-25T00:30:39.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.714+0000 7f05703aa640 1 Processor -- start 2024-04-25T00:30:39.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.714+0000 7f05703aa640 1 -- start start 2024-04-25T00:30:39.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.718+0000 7f05703aa640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f056806ea20 0x7f056806dc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:39.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.718+0000 7f05703aa640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f05680c3500 0x7f056806e180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:39.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.718+0000 7f05703aa640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f056806d1a0 con 0x7f05680aa550 2024-04-25T00:30:39.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.718+0000 7f05703aa640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f0568152c30 con 0x7f056806ea20 2024-04-25T00:30:39.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.718+0000 7f05703aa640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f0568152670 con 0x7f05680c3500 2024-04-25T00:30:39.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.718+0000 7f056e146640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f056806ea20 0x7f056806dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:39.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.718+0000 7f056e947640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f05680aa550 0x7f056814edb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57396/0 (socket says 172.21.15.116:57396) 2024-04-25T00:30:39.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.718+0000 7f056e947640 1 -- 172.21.15.116:0/413127114 learned_addr learned my addr 172.21.15.116:0/413127114 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:30:39.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.718+0000 7f056d945640 1 --2- 172.21.15.116:0/413127114 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f05680c3500 0x7f056806e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:39.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.718+0000 7f056e146640 1 -- 172.21.15.116:0/413127114 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f05680c3500 msgr2=0x7f056806e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:39.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.718+0000 7f056e146640 1 --2- 172.21.15.116:0/413127114 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f05680c3500 0x7f056806e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:39.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.718+0000 7f056e146640 1 -- 172.21.15.116:0/413127114 >> v1:172.21.15.116:6789/0 conn(0x7f05680aa550 legacy=0x7f056814edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:39.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.718+0000 7f056e146640 1 -- 172.21.15.116:0/413127114 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0568152950 con 0x7f056806ea20 2024-04-25T00:30:39.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.718+0000 7f056d144640 1 -- 172.21.15.116:0/413127114 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1734955287 0 0) 0x7f056806d1a0 con 0x7f05680aa550 2024-04-25T00:30:39.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.718+0000 7f056e146640 1 --2- 172.21.15.116:0/413127114 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f056806ea20 0x7f056806dc40 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f056400ba50 tx=0x7f056401e8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5b183563b88b69a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:39.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.718+0000 7f056d144640 1 -- 172.21.15.116:0/413127114 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05640ad070 con 0x7f056806ea20 2024-04-25T00:30:39.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.718+0000 7f056d144640 1 -- 172.21.15.116:0/413127114 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05640a7880 con 0x7f056806ea20 2024-04-25T00:30:39.720 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.718+0000 7f056d144640 1 -- 172.21.15.116:0/413127114 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05640a7c50 con 0x7f056806ea20 2024-04-25T00:30:39.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.718+0000 7f05703aa640 1 -- 172.21.15.116:0/413127114 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f056806ea20 msgr2=0x7f056806dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:39.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.718+0000 7f05703aa640 1 --2- 172.21.15.116:0/413127114 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f056806ea20 0x7f056806dc40 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f056400ba50 tx=0x7f056401e8c0 comp rx=0 tx=0).stop 2024-04-25T00:30:39.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.718+0000 7f05703aa640 1 -- 172.21.15.116:0/413127114 shutdown_connections 2024-04-25T00:30:39.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.718+0000 7f05703aa640 1 --2- 172.21.15.116:0/413127114 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f05680c3500 0x7f056806e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:39.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.718+0000 7f05703aa640 1 --2- 172.21.15.116:0/413127114 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f056806ea20 0x7f056806dc40 unknown :-1 s=CLOSED pgs=1791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:39.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.718+0000 7f05703aa640 1 -- 172.21.15.116:0/413127114 >> 172.21.15.116:0/413127114 conn(0x7f05680c3a20 msgr2=0x7f05680c3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:39.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.718+0000 7f05703aa640 1 -- 172.21.15.116:0/413127114 shutdown_connections 2024-04-25T00:30:39.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.718+0000 7f05703aa640 1 -- 172.21.15.116:0/413127114 wait complete. 2024-04-25T00:30:39.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.722+0000 7f05703aa640 1 Processor -- start 2024-04-25T00:30:39.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.722+0000 7f05703aa640 1 -- start start 2024-04-25T00:30:39.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.722+0000 7f05703aa640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f056806ea20 0x7f05681ee4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:39.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.722+0000 7f05703aa640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f05680aa550 0x7f0568153220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:39.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.722+0000 7f05703aa640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05680c3500 0x7f05681ea510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:39.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.722+0000 7f05703aa640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f056806f010 con 0x7f05680c3500 2024-04-25T00:30:39.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.722+0000 7f05703aa640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f056815e580 con 0x7f05680aa550 2024-04-25T00:30:39.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.722+0000 7f05703aa640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f05680c6570 con 0x7f056806ea20 2024-04-25T00:30:39.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.722+0000 7f056e947640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05680c3500 0x7f05681ea510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:39.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.722+0000 7f056e947640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05680c3500 0x7f05681ea510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36402/0 (socket says 172.21.15.116:36402) 2024-04-25T00:30:39.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.722+0000 7f056e947640 1 -- 172.21.15.116:0/1488367371 learned_addr learned my addr 172.21.15.116:0/1488367371 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:39.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.722+0000 7f056e146640 1 --2- 172.21.15.116:0/1488367371 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f056806ea20 0x7f05681ee4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:39.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.722+0000 7f056e947640 1 -- 172.21.15.116:0/1488367371 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f056806ea20 msgr2=0x7f05681ee4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:39.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.722+0000 7f056e947640 1 --2- 172.21.15.116:0/1488367371 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f056806ea20 0x7f05681ee4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:39.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.722+0000 7f056e947640 1 -- 172.21.15.116:0/1488367371 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f05680aa550 msgr2=0x7f0568153220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:39.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.722+0000 7f056d945640 1 --2- 172.21.15.116:0/1488367371 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f05680aa550 0x7f0568153220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:39.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.722+0000 7f056e947640 1 --2- 172.21.15.116:0/1488367371 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f05680aa550 0x7f0568153220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:39.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.722+0000 7f056e947640 1 -- 172.21.15.116:0/1488367371 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05680c67c0 con 0x7f05680c3500 2024-04-25T00:30:39.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.722+0000 7f056e947640 1 --2- 172.21.15.116:0/1488367371 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05680c3500 0x7f05681ea510 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f056000c7b0 tx=0x7f056000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:39.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.722+0000 7f055effd640 1 -- 172.21.15.116:0/1488367371 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0560018070 con 0x7f05680c3500 2024-04-25T00:30:39.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.722+0000 7f055effd640 1 -- 172.21.15.116:0/1488367371 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f056000a640 con 0x7f05680c3500 2024-04-25T00:30:39.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.722+0000 7f055effd640 1 -- 172.21.15.116:0/1488367371 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f056000aa70 con 0x7f05680c3500 2024-04-25T00:30:39.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.722+0000 7f05703aa640 1 -- 172.21.15.116:0/1488367371 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05680c6aa0 con 0x7f05680c3500 2024-04-25T00:30:39.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.722+0000 7f05703aa640 1 -- 172.21.15.116:0/1488367371 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0568153760 con 0x7f05680c3500 2024-04-25T00:30:39.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.722+0000 7f05703aa640 1 -- 172.21.15.116:0/1488367371 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f056804f510 con 0x7f05680c3500 2024-04-25T00:30:39.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.726+0000 7f055effd640 1 -- 172.21.15.116:0/1488367371 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f056001b070 con 0x7f05680c3500 2024-04-25T00:30:39.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.726+0000 7f055effd640 1 --2- 172.21.15.116:0/1488367371 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f054c0678c0 0x7f054c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:39.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.726+0000 7f055effd640 1 -- 172.21.15.116:0/1488367371 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7f05600865a0 con 0x7f05680c3500 2024-04-25T00:30:39.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.726+0000 7f056e146640 1 --2- 172.21.15.116:0/1488367371 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f054c0678c0 0x7f054c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:39.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.726+0000 7f056e146640 1 --2- 172.21.15.116:0/1488367371 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f054c0678c0 0x7f054c069d80 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f05640b2000 tx=0x7f05640a9070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:39.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.730+0000 7f055effd640 1 -- 172.21.15.116:0/1488367371 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f0560055df0 con 0x7f05680c3500 2024-04-25T00:30:39.889 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.890+0000 7f05703aa640 1 -- 172.21.15.116:0/1488367371 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"} v 0) -- 0x7f05680c13a0 con 0x7f05680c3500 2024-04-25T00:30:39.890 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.890+0000 7f055effd640 1 -- 172.21.15.116:0/1488367371 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "b_data", "var": "pg_num"}]=0 v74) ==== 95+0+11 (secure 0 0 0) 0x7f05600556f0 con 0x7f05680c3500 2024-04-25T00:30:39.890 INFO:teuthology.orchestra.run.smithi116.stdout:pg_num: 64 2024-04-25T00:30:39.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.890+0000 7f05703aa640 1 -- 172.21.15.116:0/1488367371 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f054c0678c0 msgr2=0x7f054c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:39.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.890+0000 7f05703aa640 1 --2- 172.21.15.116:0/1488367371 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f054c0678c0 0x7f054c069d80 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f05640b2000 tx=0x7f05640a9070 comp rx=0 tx=0).stop 2024-04-25T00:30:39.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.890+0000 7f05703aa640 1 -- 172.21.15.116:0/1488367371 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05680c3500 msgr2=0x7f05681ea510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:39.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.890+0000 7f05703aa640 1 --2- 172.21.15.116:0/1488367371 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05680c3500 0x7f05681ea510 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f056000c7b0 tx=0x7f056000cc70 comp rx=0 tx=0).stop 2024-04-25T00:30:39.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.890+0000 7f05703aa640 1 -- 172.21.15.116:0/1488367371 shutdown_connections 2024-04-25T00:30:39.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.890+0000 7f05703aa640 1 --2- 172.21.15.116:0/1488367371 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f054c0678c0 0x7f054c069d80 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:39.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.890+0000 7f05703aa640 1 --2- 172.21.15.116:0/1488367371 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f05680c3500 0x7f05681ea510 unknown :-1 s=CLOSED pgs=1792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:39.893 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.890+0000 7f05703aa640 1 --2- 172.21.15.116:0/1488367371 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f05680aa550 0x7f0568153220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:39.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.890+0000 7f05703aa640 1 --2- 172.21.15.116:0/1488367371 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f056806ea20 0x7f05681ee4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:39.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.890+0000 7f05703aa640 1 -- 172.21.15.116:0/1488367371 >> 172.21.15.116:0/1488367371 conn(0x7f05680c3a20 msgr2=0x7f05680c3e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:39.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.890+0000 7f05703aa640 1 -- 172.21.15.116:0/1488367371 shutdown_connections 2024-04-25T00:30:39.894 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:39.890+0000 7f05703aa640 1 -- 172.21.15.116:0/1488367371 wait complete. 2024-04-25T00:30:39.910 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T00:30:40.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.026+0000 7ff9a6539640 1 Processor -- start 2024-04-25T00:30:40.029 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.026+0000 7ff9a6539640 1 -- start start 2024-04-25T00:30:40.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.030+0000 7ff9a6539640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff9a00c5270 0x7ff9a0159770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:40.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.030+0000 7ff9a6539640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff9a00c3b60 0x7ff9a01527d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:40.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.030+0000 7ff9a6539640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9a0158cd0 con 0x7ff9a00c36e0 2024-04-25T00:30:40.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.030+0000 7ff9a6539640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7ff9a01592e0 con 0x7ff9a00c3b60 2024-04-25T00:30:40.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.030+0000 7ff9a6539640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff9a0156280 con 0x7ff9a00c5270 2024-04-25T00:30:40.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.030+0000 7ff9a4ad6640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff9a00c3b60 0x7ff9a01527d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:40.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.030+0000 7ff99f7fe640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7ff9a00c36e0 0x7ff9a01557c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57408/0 (socket says 172.21.15.116:57408) 2024-04-25T00:30:40.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.030+0000 7ff99f7fe640 1 -- 172.21.15.116:0/4225152663 learned_addr learned my addr 172.21.15.116:0/4225152663 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:30:40.031 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.030+0000 7ff99ffff640 1 --2- 172.21.15.116:0/4225152663 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff9a00c5270 0x7ff9a0159770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:40.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.030+0000 7ff99effd640 1 -- 172.21.15.116:0/4225152663 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2877952723 0 0) 0x7ff9a0158cd0 con 0x7ff9a00c36e0 2024-04-25T00:30:40.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.030+0000 7ff99effd640 1 -- 172.21.15.116:0/4225152663 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff988003530 con 0x7ff9a00c36e0 2024-04-25T00:30:40.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.030+0000 7ff9a4ad6640 1 -- 172.21.15.116:0/4225152663 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff9a00c5270 msgr2=0x7ff9a0159770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:40.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.030+0000 7ff9a4ad6640 1 --2- 172.21.15.116:0/4225152663 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff9a00c5270 0x7ff9a0159770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:40.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.030+0000 7ff9a4ad6640 1 -- 172.21.15.116:0/4225152663 >> v1:172.21.15.116:6789/0 conn(0x7ff9a00c36e0 legacy=0x7ff9a01557c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:40.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.030+0000 7ff9a4ad6640 1 -- 172.21.15.116:0/4225152663 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9a0156560 con 0x7ff9a00c3b60 2024-04-25T00:30:40.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.030+0000 7ff9a4ad6640 1 --2- 172.21.15.116:0/4225152663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff9a00c3b60 0x7ff9a01527d0 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7ff9900097f0 tx=0x7ff99001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34f359485508a212 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:40.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.030+0000 7ff99effd640 1 -- 172.21.15.116:0/4225152663 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9900b0070 con 0x7ff9a00c3b60 2024-04-25T00:30:40.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.030+0000 7ff99effd640 1 -- 172.21.15.116:0/4225152663 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9900ab600 con 0x7ff9a00c3b60 2024-04-25T00:30:40.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.030+0000 7ff99effd640 1 -- 172.21.15.116:0/4225152663 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9900ab9d0 con 0x7ff9a00c3b60 2024-04-25T00:30:40.033 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.030+0000 7ff9a6539640 1 -- 172.21.15.116:0/4225152663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff9a00c3b60 msgr2=0x7ff9a01527d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:40.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.030+0000 7ff9a6539640 1 --2- 172.21.15.116:0/4225152663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff9a00c3b60 0x7ff9a01527d0 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7ff9900097f0 tx=0x7ff99001c4a0 comp rx=0 tx=0).stop 2024-04-25T00:30:40.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.030+0000 7ff9a6539640 1 -- 172.21.15.116:0/4225152663 shutdown_connections 2024-04-25T00:30:40.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.030+0000 7ff9a6539640 1 --2- 172.21.15.116:0/4225152663 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff9a00c3b60 0x7ff9a01527d0 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:40.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.030+0000 7ff9a6539640 1 --2- 172.21.15.116:0/4225152663 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff9a00c5270 0x7ff9a0159770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:40.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.030+0000 7ff9a6539640 1 -- 172.21.15.116:0/4225152663 >> 172.21.15.116:0/4225152663 conn(0x7ff9a006c720 msgr2=0x7ff9a006cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:40.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.034+0000 7ff9a6539640 1 -- 172.21.15.116:0/4225152663 shutdown_connections 2024-04-25T00:30:40.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.034+0000 7ff9a6539640 1 -- 172.21.15.116:0/4225152663 wait complete. 2024-04-25T00:30:40.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.034+0000 7ff9a6539640 1 Processor -- start 2024-04-25T00:30:40.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.034+0000 7ff9a6539640 1 -- start start 2024-04-25T00:30:40.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.034+0000 7ff9a6539640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff9a00c36e0 0x7ff9a00acad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:40.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.034+0000 7ff9a6539640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff9a00c3b60 0x7ff9a014be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:40.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.034+0000 7ff9a6539640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff9a00c5270 0x7ff9a01f16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:40.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.034+0000 7ff9a6539640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff9a00c57d0 con 0x7ff9a00c36e0 2024-04-25T00:30:40.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.034+0000 7ff9a6539640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff9a00c5650 con 0x7ff9a00c3b60 2024-04-25T00:30:40.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.034+0000 7ff9a6539640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff9a0162880 con 0x7ff9a00c5270 2024-04-25T00:30:40.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.034+0000 7ff99ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff9a00c36e0 0x7ff9a00acad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:40.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.034+0000 7ff99ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff9a00c36e0 0x7ff9a00acad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36420/0 (socket says 172.21.15.116:36420) 2024-04-25T00:30:40.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.034+0000 7ff99ffff640 1 -- 172.21.15.116:0/2061756948 learned_addr learned my addr 172.21.15.116:0/2061756948 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:40.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.034+0000 7ff99f7fe640 1 --2- 172.21.15.116:0/2061756948 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff9a00c3b60 0x7ff9a014be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:40.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.034+0000 7ff9a4ad6640 1 --2- 172.21.15.116:0/2061756948 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff9a00c5270 0x7ff9a01f16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:40.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.034+0000 7ff99ffff640 1 -- 172.21.15.116:0/2061756948 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff9a00c5270 msgr2=0x7ff9a01f16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:40.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.034+0000 7ff99ffff640 1 --2- 172.21.15.116:0/2061756948 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff9a00c5270 0x7ff9a01f16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:40.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.034+0000 7ff99ffff640 1 -- 172.21.15.116:0/2061756948 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff9a00c3b60 msgr2=0x7ff9a014be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:40.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.034+0000 7ff99ffff640 1 --2- 172.21.15.116:0/2061756948 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff9a00c3b60 0x7ff9a014be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:40.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.034+0000 7ff99ffff640 1 -- 172.21.15.116:0/2061756948 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9a014fec0 con 0x7ff9a00c36e0 2024-04-25T00:30:40.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.034+0000 7ff99ffff640 1 --2- 172.21.15.116:0/2061756948 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff9a00c36e0 0x7ff9a00acad0 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7ff998007ef0 tx=0x7ff99800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:40.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.034+0000 7ff99cff9640 1 -- 172.21.15.116:0/2061756948 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff998014620 con 0x7ff9a00c36e0 2024-04-25T00:30:40.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.034+0000 7ff99cff9640 1 -- 172.21.15.116:0/2061756948 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff998019070 con 0x7ff9a00c36e0 2024-04-25T00:30:40.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.034+0000 7ff9a6539640 1 -- 172.21.15.116:0/2061756948 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9a00ad0b0 con 0x7ff9a00c36e0 2024-04-25T00:30:40.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.034+0000 7ff99cff9640 1 -- 172.21.15.116:0/2061756948 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff998014a00 con 0x7ff9a00c36e0 2024-04-25T00:30:40.038 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.034+0000 7ff9a6539640 1 -- 172.21.15.116:0/2061756948 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9a014fbe0 con 0x7ff9a00c36e0 2024-04-25T00:30:40.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.034+0000 7ff9a6539640 1 -- 172.21.15.116:0/2061756948 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9a0068a60 con 0x7ff9a00c36e0 2024-04-25T00:30:40.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.038+0000 7ff99cff9640 1 -- 172.21.15.116:0/2061756948 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff998014ba0 con 0x7ff9a00c36e0 2024-04-25T00:30:40.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.038+0000 7ff99cff9640 1 --2- 172.21.15.116:0/2061756948 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff98c0678c0 0x7ff98c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:40.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.038+0000 7ff99cff9640 1 -- 172.21.15.116:0/2061756948 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7ff9980878f0 con 0x7ff9a00c36e0 2024-04-25T00:30:40.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.038+0000 7ff99f7fe640 1 --2- 172.21.15.116:0/2061756948 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff98c0678c0 0x7ff98c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:40.039 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.038+0000 7ff99f7fe640 1 --2- 172.21.15.116:0/2061756948 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff98c0678c0 0x7ff98c069d80 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7ff9a01ebf50 tx=0x7ff994009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:40.042 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.042+0000 7ff99cff9640 1 -- 172.21.15.116:0/2061756948 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ff9a014fec0 con 0x7ff9a00c36e0 2024-04-25T00:30:40.215 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.214+0000 7ff9a6539640 1 -- 172.21.15.116:0/2061756948 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7ff9a0158af0 con 0x7ff9a00c36e0 2024-04-25T00:30:40.216 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.214+0000 7ff99cff9640 1 -- 172.21.15.116:0/2061756948 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 92 entries v74) ==== 81+0+5682 (secure 0 0 0) 0x7ff998057140 con 0x7ff9a00c36e0 2024-04-25T00:30:40.216 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/3499878473 2024-04-26T00:30:15.781624+0000 2024-04-25T00:30:40.216 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/3499878473 2024-04-26T00:30:15.781624+0000 2024-04-25T00:30:40.216 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/2915274073 2024-04-26T00:29:50.776462+0000 2024-04-25T00:30:40.216 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6833/3891103804 2024-04-26T00:28:50.767503+0000 2024-04-25T00:30:40.216 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/2371126056 2024-04-26T00:27:35.754759+0000 2024-04-25T00:30:40.216 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/349772087 2024-04-26T00:27:15.748274+0000 2024-04-25T00:30:40.216 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/349772087 2024-04-26T00:27:15.748274+0000 2024-04-25T00:30:40.216 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6833/3693049068 2024-04-26T00:26:45.739776+0000 2024-04-25T00:30:40.216 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6832/3693049068 2024-04-26T00:26:45.739776+0000 2024-04-25T00:30:40.217 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/4181466530 2024-04-26T00:26:15.734586+0000 2024-04-25T00:30:40.217 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/4181466530 2024-04-26T00:26:15.734586+0000 2024-04-25T00:30:40.217 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/492366509 2024-04-26T00:24:15.701887+0000 2024-04-25T00:30:40.217 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6832/2998042514 2024-04-26T00:23:30.691827+0000 2024-04-25T00:30:40.217 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6833/663910035 2024-04-26T00:22:35.677323+0000 2024-04-25T00:30:40.217 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6832/663910035 2024-04-26T00:22:35.677323+0000 2024-04-25T00:30:40.217 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/321466844 2024-04-26T00:22:10.671673+0000 2024-04-25T00:30:40.217 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/980221971 2024-04-26T00:21:40.665115+0000 2024-04-25T00:30:40.217 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/991183935 2024-04-26T00:20:50.654151+0000 2024-04-25T00:30:40.217 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/991183935 2024-04-26T00:20:50.654151+0000 2024-04-25T00:30:40.217 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/321466844 2024-04-26T00:22:10.671673+0000 2024-04-25T00:30:40.217 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/1358527848 2024-04-26T00:11:00.511032+0000 2024-04-25T00:30:40.217 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6832/3841609112 2024-04-26T00:07:00.448498+0000 2024-04-25T00:30:40.218 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/1670061091 2024-04-26T00:09:30.490485+0000 2024-04-25T00:30:40.218 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/1971675962 2024-04-26T00:25:25.721965+0000 2024-04-25T00:30:40.218 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/1233829678 2024-04-26T00:07:30.454393+0000 2024-04-25T00:30:40.218 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/2371126056 2024-04-26T00:27:35.754759+0000 2024-04-25T00:30:40.218 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6834/2883220846 2024-04-26T00:11:50.521053+0000 2024-04-25T00:30:40.218 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/1233829678 2024-04-26T00:07:30.454393+0000 2024-04-25T00:30:40.218 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/3180024481 2024-04-26T00:15:25.567768+0000 2024-04-25T00:30:40.218 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/879531976 2024-04-26T00:20:10.642563+0000 2024-04-25T00:30:40.218 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6833/174781672 2024-04-26T00:02:40.399073+0000 2024-04-25T00:30:40.218 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6835/4215220222 2024-04-26T00:15:40.572498+0000 2024-04-25T00:30:40.218 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/221971311 2024-04-26T00:22:48.898075+0000 2024-04-25T00:30:40.218 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/1281466397 2024-04-26T00:18:20.613347+0000 2024-04-25T00:30:40.218 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6832/3891103804 2024-04-26T00:28:50.767503+0000 2024-04-25T00:30:40.218 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/1358527848 2024-04-26T00:11:00.511032+0000 2024-04-25T00:30:40.219 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/221971311 2024-04-26T00:22:48.898075+0000 2024-04-25T00:30:40.219 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6832/3006671671 2024-04-26T00:09:30.490406+0000 2024-04-25T00:30:40.219 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/980221971 2024-04-26T00:21:40.665115+0000 2024-04-25T00:30:40.219 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/1762356055 2024-04-26T00:03:36.597345+0000 2024-04-25T00:30:40.219 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6833/3006671671 2024-04-26T00:09:30.490406+0000 2024-04-25T00:30:40.219 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/1459177921 2024-04-26T00:03:30.406778+0000 2024-04-25T00:30:40.219 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/462190523 2024-04-26T00:14:55.560782+0000 2024-04-25T00:30:40.219 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/879531976 2024-04-26T00:20:10.642563+0000 2024-04-25T00:30:40.219 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6832/1459177921 2024-04-26T00:03:30.406778+0000 2024-04-25T00:30:40.219 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/1159002662 2024-04-26T00:10:20.502218+0000 2024-04-25T00:30:40.219 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/1762356055 2024-04-26T00:03:36.597345+0000 2024-04-25T00:30:40.219 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6834/4215220222 2024-04-26T00:15:40.572498+0000 2024-04-25T00:30:40.219 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/1971675962 2024-04-26T00:25:25.721965+0000 2024-04-25T00:30:40.219 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/3081407040 2024-04-26T00:17:25.598516+0000 2024-04-25T00:30:40.219 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/1599708526 2024-04-26T00:12:20.526742+0000 2024-04-25T00:30:40.220 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/2276529656 2024-04-26T00:19:05.629048+0000 2024-04-25T00:30:40.220 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/1159002662 2024-04-26T00:10:20.502218+0000 2024-04-25T00:30:40.220 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6833/2998042514 2024-04-26T00:23:30.691827+0000 2024-04-25T00:30:40.220 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/3081407040 2024-04-26T00:17:25.598516+0000 2024-04-25T00:30:40.220 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6835/2883220846 2024-04-26T00:11:50.521053+0000 2024-04-25T00:30:40.220 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/492366509 2024-04-26T00:24:15.701887+0000 2024-04-25T00:30:40.220 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/2004876235 2024-04-26T00:04:45.421611+0000 2024-04-25T00:30:40.220 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/174781672 2024-04-26T00:02:40.399073+0000 2024-04-25T00:30:40.220 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6833/3633657268 2024-04-26T00:25:40.724634+0000 2024-04-25T00:30:40.220 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6832/365773583 2024-04-26T00:14:58.857970+0000 2024-04-25T00:30:40.220 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/1889314958 2024-04-26T00:06:30.441408+0000 2024-04-25T00:30:40.220 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6833/3705654249 2024-04-26T00:12:45.533219+0000 2024-04-25T00:30:40.220 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6832/3705654249 2024-04-26T00:12:45.533219+0000 2024-04-25T00:30:40.220 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/1670061091 2024-04-26T00:09:30.490485+0000 2024-04-25T00:30:40.220 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/1599708526 2024-04-26T00:12:20.526742+0000 2024-04-25T00:30:40.221 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6832/3633657268 2024-04-26T00:25:40.724634+0000 2024-04-25T00:30:40.221 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/3583791822 2024-04-26T00:08:00.460587+0000 2024-04-25T00:30:40.221 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/3583791822 2024-04-26T00:08:00.460587+0000 2024-04-25T00:30:40.221 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/3398690896 2024-04-26T00:13:30.543729+0000 2024-04-25T00:30:40.221 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/3398690896 2024-04-26T00:13:30.543729+0000 2024-04-25T00:30:40.221 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/462190523 2024-04-26T00:14:55.560782+0000 2024-04-25T00:30:40.221 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/3336339376 2024-04-26T00:18:35.616887+0000 2024-04-25T00:30:40.221 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6836/3180024481 2024-04-26T00:15:25.567768+0000 2024-04-25T00:30:40.221 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6833/365773583 2024-04-26T00:14:58.857970+0000 2024-04-25T00:30:40.221 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/2012470875 2024-04-26T00:16:35.584511+0000 2024-04-25T00:30:40.221 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6833/2004876235 2024-04-26T00:04:45.421611+0000 2024-04-25T00:30:40.221 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/1281466397 2024-04-26T00:18:20.613347+0000 2024-04-25T00:30:40.221 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6832/2340581953 2024-04-26T00:16:45.587651+0000 2024-04-25T00:30:40.221 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/2915274073 2024-04-26T00:29:50.776462+0000 2024-04-25T00:30:40.222 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/1649769369 2024-04-26T00:04:35.418892+0000 2024-04-25T00:30:40.222 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6832/1649769369 2024-04-26T00:04:35.418892+0000 2024-04-25T00:30:40.222 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6833/2340581953 2024-04-26T00:16:45.587651+0000 2024-04-25T00:30:40.222 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/1889314958 2024-04-26T00:06:30.441408+0000 2024-04-25T00:30:40.222 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6832/3645553116 2024-04-26T00:17:35.601559+0000 2024-04-25T00:30:40.222 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6833/3645553116 2024-04-26T00:17:35.601559+0000 2024-04-25T00:30:40.222 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6833/3841609112 2024-04-26T00:07:00.448498+0000 2024-04-25T00:30:40.222 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6834/2012470875 2024-04-26T00:16:35.584511+0000 2024-04-25T00:30:40.222 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6835/3336339376 2024-04-26T00:18:35.616887+0000 2024-04-25T00:30:40.222 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.174:6837/2276529656 2024-04-26T00:19:05.629048+0000 2024-04-25T00:30:40.222 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6832/1655959433 2024-04-26T00:19:30.633957+0000 2024-04-25T00:30:40.222 INFO:teuthology.orchestra.run.smithi116.stdout:172.21.15.116:6833/1655959433 2024-04-26T00:19:30.633957+0000 2024-04-25T00:30:40.223 INFO:teuthology.orchestra.run.smithi116.stderr:listed 92 entries 2024-04-25T00:30:40.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.218+0000 7ff9a6539640 1 -- 172.21.15.116:0/2061756948 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff98c0678c0 msgr2=0x7ff98c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:40.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.218+0000 7ff9a6539640 1 --2- 172.21.15.116:0/2061756948 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff98c0678c0 0x7ff98c069d80 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7ff9a01ebf50 tx=0x7ff994009040 comp rx=0 tx=0).stop 2024-04-25T00:30:40.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.218+0000 7ff9a6539640 1 -- 172.21.15.116:0/2061756948 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff9a00c36e0 msgr2=0x7ff9a00acad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:40.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.218+0000 7ff9a6539640 1 --2- 172.21.15.116:0/2061756948 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff9a00c36e0 0x7ff9a00acad0 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7ff998007ef0 tx=0x7ff99800e4b0 comp rx=0 tx=0).stop 2024-04-25T00:30:40.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.218+0000 7ff9a6539640 1 -- 172.21.15.116:0/2061756948 shutdown_connections 2024-04-25T00:30:40.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.218+0000 7ff9a6539640 1 --2- 172.21.15.116:0/2061756948 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff98c0678c0 0x7ff98c069d80 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:40.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.218+0000 7ff9a6539640 1 --2- 172.21.15.116:0/2061756948 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff9a00c5270 0x7ff9a01f16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:40.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.218+0000 7ff9a6539640 1 --2- 172.21.15.116:0/2061756948 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff9a00c3b60 0x7ff9a014be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:40.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.218+0000 7ff9a6539640 1 --2- 172.21.15.116:0/2061756948 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff9a00c36e0 0x7ff9a00acad0 unknown :-1 s=CLOSED pgs=1795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:40.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.218+0000 7ff9a6539640 1 -- 172.21.15.116:0/2061756948 >> 172.21.15.116:0/2061756948 conn(0x7ff9a006c720 msgr2=0x7ff9a006cb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:40.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.218+0000 7ff9a6539640 1 -- 172.21.15.116:0/2061756948 shutdown_connections 2024-04-25T00:30:40.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:40.218+0000 7ff9a6539640 1 -- 172.21.15.116:0/2061756948 wait complete. 2024-04-25T00:30:40.235 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi174.front.sepia.ceph.com... 2024-04-25T00:30:40.236 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:30:40.236 DEBUG:teuthology.orchestra.run.smithi174:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-04-25T00:30:40.327 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:30:40.333 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi174.stderr:ceph-fuse[33357]: fuse finished with error 0 and tester_r 0 2024-04-25T00:30:40.415 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi174.front.sepia.ceph.com 2024-04-25T00:30:40.416 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:30:40.416 DEBUG:teuthology.orchestra.run.smithi174:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-25T00:30:40.479 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:30:40.479 DEBUG:teuthology.orchestra.run.smithi116:> ip netns list 2024-04-25T00:30:40.545 INFO:teuthology.orchestra.run.smithi116.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-25T00:30:40.546 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:30:40.546 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T00:30:40.561 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:30:40.561 DEBUG:teuthology.orchestra.run.smithi116:> sudo ip link delete ceph-brx 2024-04-25T00:30:40.731 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:30:40.731 DEBUG:teuthology.orchestra.run.smithi174:> ip netns list 2024-04-25T00:30:40.778 INFO:teuthology.orchestra.run.smithi174.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-25T00:30:40.778 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:30:40.778 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T00:30:40.794 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T00:30:40.794 DEBUG:teuthology.orchestra.run.smithi174:> sudo ip link delete ceph-brx 2024-04-25T00:30:40.967 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-25T00:30:40.977 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-25T00:30:40.978 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T00:30:41.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.106+0000 7f11f2f7f640 1 Processor -- start 2024-04-25T00:30:41.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.106+0000 7f11f2f7f640 1 -- start start 2024-04-25T00:30:41.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.110+0000 7f11f2f7f640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f11ec150d20 0x7f11ec0c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:41.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.110+0000 7f11f2f7f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11ec0c3910 con 0x7f11ec0a94a0 2024-04-25T00:30:41.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.110+0000 7f11f2f7f640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11ec0c16e0 con 0x7f11ec0c2f50 2024-04-25T00:30:41.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.110+0000 7f11f2f7f640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f11ec0c1400 con 0x7f11ec150d20 2024-04-25T00:30:41.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.110+0000 7f11f151c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f11ec0a94a0 0x7f11ec153ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57414/0 (socket says 172.21.15.116:57414) 2024-04-25T00:30:41.111 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.110+0000 7f11f151c640 1 -- 172.21.15.116:0/3027173554 learned_addr learned my addr 172.21.15.116:0/3027173554 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:30:41.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.110+0000 7f11eb7fe640 1 -- 172.21.15.116:0/3027173554 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3256117261 0 0) 0x7f11ec0c3910 con 0x7f11ec0a94a0 2024-04-25T00:30:41.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.110+0000 7f11eb7fe640 1 -- 172.21.15.116:0/3027173554 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f11d4003530 con 0x7f11ec0a94a0 2024-04-25T00:30:41.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.110+0000 7f11f0d1b640 1 --2- 172.21.15.116:0/3027173554 >> v2:172.21.15.174:3300/0 conn(0x7f11ec150d20 0x7f11ec0c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:41.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.110+0000 7f11eb7fe640 1 -- 172.21.15.116:0/3027173554 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2959626880 0 0) 0x7f11d4003530 con 0x7f11ec0a94a0 2024-04-25T00:30:41.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.110+0000 7f11eb7fe640 1 -- 172.21.15.116:0/3027173554 >> v2:172.21.15.174:3300/0 conn(0x7f11ec150d20 msgr2=0x7f11ec0c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:41.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.110+0000 7f11eb7fe640 1 --2- 172.21.15.116:0/3027173554 >> v2:172.21.15.174:3300/0 conn(0x7f11ec150d20 0x7f11ec0c2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:41.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.110+0000 7f11eb7fe640 1 -- 172.21.15.116:0/3027173554 >> v1:172.21.15.174:6789/0 conn(0x7f11ec0c2f50 legacy=0x7f11ec1538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:41.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.110+0000 7f11eb7fe640 1 -- 172.21.15.116:0/3027173554 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11ec06e660 con 0x7f11ec0a94a0 2024-04-25T00:30:41.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.110+0000 7f11eb7fe640 1 -- 172.21.15.116:0/3027173554 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f11e4001a00 con 0x7f11ec0a94a0 2024-04-25T00:30:41.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.110+0000 7f11eb7fe640 1 -- 172.21.15.116:0/3027173554 >> v1:172.21.15.116:6789/0 conn(0x7f11ec0a94a0 legacy=0x7f11ec153ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:41.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.110+0000 7f11eb7fe640 1 --2- 172.21.15.116:0/3027173554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11d40039c0 0x7f11d4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:41.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.110+0000 7f11eb7fe640 1 --2- 172.21.15.116:0/3027173554 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11d4004810 0x7f11d4004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:41.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.110+0000 7f11eb7fe640 1 --2- 172.21.15.116:0/3027173554 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11d40051d0 0x7f11d400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:41.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.110+0000 7f11eb7fe640 1 -- 172.21.15.116:0/3027173554 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec0c1400 con 0x7f11d40039c0 2024-04-25T00:30:41.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.110+0000 7f11eb7fe640 1 -- 172.21.15.116:0/3027173554 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f11d400c100 con 0x7f11d40051d0 2024-04-25T00:30:41.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.110+0000 7f11eb7fe640 1 -- 172.21.15.116:0/3027173554 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f11d400c300 con 0x7f11d4004810 2024-04-25T00:30:41.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.110+0000 7f11f0d1b640 1 --2- 172.21.15.116:0/3027173554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11d40039c0 0x7f11d4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:41.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.110+0000 7f11f151c640 1 --2- 172.21.15.116:0/3027173554 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11d40051d0 0x7f11d400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:41.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.110+0000 7f11ebfff640 1 --2- 172.21.15.116:0/3027173554 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11d4004810 0x7f11d4004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:41.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.110+0000 7f11f0d1b640 1 -- 172.21.15.116:0/3027173554 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11d4004810 msgr2=0x7f11d4004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:41.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.110+0000 7f11f0d1b640 1 --2- 172.21.15.116:0/3027173554 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11d4004810 0x7f11d4004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:41.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.110+0000 7f11f0d1b640 1 -- 172.21.15.116:0/3027173554 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11d40051d0 msgr2=0x7f11d400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:41.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.110+0000 7f11f0d1b640 1 --2- 172.21.15.116:0/3027173554 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11d40051d0 0x7f11d400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:41.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.110+0000 7f11f0d1b640 1 -- 172.21.15.116:0/3027173554 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11d4003530 con 0x7f11d40039c0 2024-04-25T00:30:41.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.110+0000 7f11f151c640 1 -- 172.21.15.116:0/3027173554 reap_dead start 2024-04-25T00:30:41.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.110+0000 7f11ebfff640 1 --2- 172.21.15.116:0/3027173554 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11d4004810 0x7f11d4004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:30:41.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.110+0000 7f11f0d1b640 1 --2- 172.21.15.116:0/3027173554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11d40039c0 0x7f11d4003dc0 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f11dc00ba00 tx=0x7f11dc01c660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86536f99863c2433 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:41.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.114+0000 7f11eb7fe640 1 -- 172.21.15.116:0/3027173554 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11dc00b600 con 0x7f11d40039c0 2024-04-25T00:30:41.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.114+0000 7f11eb7fe640 1 -- 172.21.15.116:0/3027173554 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11dc0b1070 con 0x7f11d40039c0 2024-04-25T00:30:41.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.114+0000 7f11eb7fe640 1 -- 172.21.15.116:0/3027173554 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11dc0ac840 con 0x7f11d40039c0 2024-04-25T00:30:41.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.114+0000 7f11f2f7f640 1 -- 172.21.15.116:0/3027173554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11d40039c0 msgr2=0x7f11d4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:41.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.114+0000 7f11f2f7f640 1 --2- 172.21.15.116:0/3027173554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11d40039c0 0x7f11d4003dc0 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f11dc00ba00 tx=0x7f11dc01c660 comp rx=0 tx=0).stop 2024-04-25T00:30:41.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.114+0000 7f11f2f7f640 1 -- 172.21.15.116:0/3027173554 shutdown_connections 2024-04-25T00:30:41.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.114+0000 7f11f2f7f640 1 --2- 172.21.15.116:0/3027173554 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11d40039c0 0x7f11d4003dc0 unknown :-1 s=CLOSED pgs=1797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:41.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.114+0000 7f11f2f7f640 1 -- 172.21.15.116:0/3027173554 >> 172.21.15.116:0/3027173554 conn(0x7f11ec0c3b00 msgr2=0x7f11ec0c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:41.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.114+0000 7f11f2f7f640 1 -- 172.21.15.116:0/3027173554 shutdown_connections 2024-04-25T00:30:41.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.114+0000 7f11f2f7f640 1 -- 172.21.15.116:0/3027173554 wait complete. 2024-04-25T00:30:41.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.114+0000 7f11f2f7f640 1 Processor -- start 2024-04-25T00:30:41.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.114+0000 7f11f2f7f640 1 -- start start 2024-04-25T00:30:41.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.114+0000 7f11f2f7f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11d40039c0 0x7f11ec13c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:41.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.114+0000 7f11f2f7f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11ec14f560 0x7f11ec155e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:41.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.114+0000 7f11f2f7f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11ec14fef0 0x7f11ec13e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:41.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.114+0000 7f11f2f7f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec154480 con 0x7f11ec14fef0 2024-04-25T00:30:41.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.114+0000 7f11f2f7f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f11ec154300 con 0x7f11d40039c0 2024-04-25T00:30:41.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.114+0000 7f11f2f7f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f11ec156b30 con 0x7f11ec14f560 2024-04-25T00:30:41.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.118+0000 7f11f151c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11ec14fef0 0x7f11ec13e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:41.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.118+0000 7f11f151c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11ec14fef0 0x7f11ec13e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36444/0 (socket says 172.21.15.116:36444) 2024-04-25T00:30:41.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.118+0000 7f11f151c640 1 -- 172.21.15.116:0/1118342192 learned_addr learned my addr 172.21.15.116:0/1118342192 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:41.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.118+0000 7f11f0d1b640 1 --2- 172.21.15.116:0/1118342192 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11d40039c0 0x7f11ec13c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:41.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.118+0000 7f11ebfff640 1 --2- 172.21.15.116:0/1118342192 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11ec14f560 0x7f11ec155e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:41.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.118+0000 7f11f151c640 1 -- 172.21.15.116:0/1118342192 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11ec14f560 msgr2=0x7f11ec155e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:41.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.118+0000 7f11f151c640 1 --2- 172.21.15.116:0/1118342192 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11ec14f560 0x7f11ec155e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:41.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.118+0000 7f11f151c640 1 -- 172.21.15.116:0/1118342192 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11d40039c0 msgr2=0x7f11ec13c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:41.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.118+0000 7f11f151c640 1 --2- 172.21.15.116:0/1118342192 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11d40039c0 0x7f11ec13c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:41.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.118+0000 7f11f151c640 1 -- 172.21.15.116:0/1118342192 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11ec13dbf0 con 0x7f11ec14fef0 2024-04-25T00:30:41.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.118+0000 7f11f151c640 1 --2- 172.21.15.116:0/1118342192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11ec14fef0 0x7f11ec13e690 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f11ec070890 tx=0x7f11e400e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:41.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.118+0000 7f11e97fa640 1 -- 172.21.15.116:0/1118342192 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e400ec10 con 0x7f11ec14fef0 2024-04-25T00:30:41.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.118+0000 7f11e97fa640 1 -- 172.21.15.116:0/1118342192 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11e400edb0 con 0x7f11ec14fef0 2024-04-25T00:30:41.119 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.118+0000 7f11f2f7f640 1 -- 172.21.15.116:0/1118342192 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11ec156350 con 0x7f11ec14fef0 2024-04-25T00:30:41.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.118+0000 7f11e97fa640 1 -- 172.21.15.116:0/1118342192 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e4014670 con 0x7f11ec14fef0 2024-04-25T00:30:41.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.118+0000 7f11f2f7f640 1 -- 172.21.15.116:0/1118342192 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11ec13de90 con 0x7f11ec14fef0 2024-04-25T00:30:41.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.118+0000 7f11f2f7f640 1 -- 172.21.15.116:0/1118342192 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11ec068350 con 0x7f11ec14fef0 2024-04-25T00:30:41.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.118+0000 7f11e97fa640 1 -- 172.21.15.116:0/1118342192 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f11e4014810 con 0x7f11ec14fef0 2024-04-25T00:30:41.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.118+0000 7f11e97fa640 1 --2- 172.21.15.116:0/1118342192 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f11d80678c0 0x7f11d8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:41.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.118+0000 7f11e97fa640 1 -- 172.21.15.116:0/1118342192 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7f11e4086f30 con 0x7f11ec14fef0 2024-04-25T00:30:41.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.118+0000 7f11f0d1b640 1 --2- 172.21.15.116:0/1118342192 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f11d80678c0 0x7f11d8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:41.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.118+0000 7f11f0d1b640 1 --2- 172.21.15.116:0/1118342192 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f11d80678c0 0x7f11d8069d80 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f11dc0b0000 tx=0x7f11dc0046d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:41.124 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.122+0000 7f11e97fa640 1 -- 172.21.15.116:0/1118342192 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f11e4056660 con 0x7f11ec14fef0 2024-04-25T00:30:41.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.282+0000 7f11f2f7f640 1 -- 172.21.15.116:0/1118342192 --> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f11ec0c34f0 con 0x7f11d80678c0 2024-04-25T00:30:41.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.318+0000 7f11e97fa640 1 -- 172.21.15.116:0/1118342192 <== mgr.4105 v2:172.21.15.174:6832/2004448211 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+493789 (secure 0 0 0) 0x7f11ec0c34f0 con 0x7f11d80678c0 2024-04-25T00:30:41.324 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:30:41.328 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-04-25T00:30:41.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.326+0000 7f11f2f7f640 1 -- 172.21.15.116:0/1118342192 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f11d80678c0 msgr2=0x7f11d8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:41.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.326+0000 7f11f2f7f640 1 --2- 172.21.15.116:0/1118342192 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f11d80678c0 0x7f11d8069d80 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f11dc0b0000 tx=0x7f11dc0046d0 comp rx=0 tx=0).stop 2024-04-25T00:30:41.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.326+0000 7f11f2f7f640 1 -- 172.21.15.116:0/1118342192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11ec14fef0 msgr2=0x7f11ec13e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:41.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.326+0000 7f11f2f7f640 1 --2- 172.21.15.116:0/1118342192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11ec14fef0 0x7f11ec13e690 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f11ec070890 tx=0x7f11e400e530 comp rx=0 tx=0).stop 2024-04-25T00:30:41.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.326+0000 7f11f2f7f640 1 -- 172.21.15.116:0/1118342192 shutdown_connections 2024-04-25T00:30:41.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.326+0000 7f11f2f7f640 1 --2- 172.21.15.116:0/1118342192 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f11d80678c0 0x7f11d8069d80 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:41.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.326+0000 7f11f2f7f640 1 --2- 172.21.15.116:0/1118342192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f11ec14fef0 0x7f11ec13e690 unknown :-1 s=CLOSED pgs=1798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:41.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.326+0000 7f11f2f7f640 1 --2- 172.21.15.116:0/1118342192 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f11ec14f560 0x7f11ec155e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:41.329 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.326+0000 7f11f2f7f640 1 --2- 172.21.15.116:0/1118342192 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f11d40039c0 0x7f11ec13c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:41.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.326+0000 7f11f2f7f640 1 -- 172.21.15.116:0/1118342192 >> 172.21.15.116:0/1118342192 conn(0x7f11ec0c3b00 msgr2=0x7f11ec06d6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:41.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.326+0000 7f11f2f7f640 1 -- 172.21.15.116:0/1118342192 shutdown_connections 2024-04-25T00:30:41.330 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.326+0000 7f11f2f7f640 1 -- 172.21.15.116:0/1118342192 wait complete. 2024-04-25T00:30:41.345 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":962,"stamp":"2024-04-25T00:30:40.026992+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":39263178,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":3556,"num_read_kb":487326,"num_write":2966736,"num_write_kb":38127636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":551297,"ondisk_log_size":551297,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":7166992,"kb_used_data":84432,"kb_used_omap":12,"kb_used_meta":7082483,"kb_avail":747807728,"statfs":{"total":773094113280,"available":765755113472,"internally_reserved":0,"allocated":86458368,"data_stored":82754316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":7252463153},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-5424637476,"num_objects":-1401,"num_object_clones":0,"num_object_copies":-2802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1401,"num_whiteouts":0,"num_read":3,"num_read_kb":19,"num_write":1414,"num_write_kb":133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005794"},"pg_stats":[{"pgid":"5.18","version":"74'4491","reported_seq":4612,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.766890+0000","last_change":"2024-04-25T00:01:19.508728+0000","last_active":"2024-04-25T00:30:26.766890+0000","last_peered":"2024-04-25T00:30:26.766890+0000","last_clean":"2024-04-25T00:30:26.766890+0000","last_became_active":"2024-04-25T00:01:19.508390+0000","last_became_peered":"2024-04-25T00:01:19.508390+0000","last_unstale":"2024-04-25T00:30:26.766890+0000","last_undegraded":"2024-04-25T00:30:26.766890+0000","last_fullsized":"2024-04-25T00:30:26.766890+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4491,"log_dups_size":0,"ondisk_log_size":4491,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:45.591905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":12,"num_write":23917,"num_write_kb":321110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1f","version":"70'1087","reported_seq":1273,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.794182+0000","last_change":"2024-04-25T00:01:04.356032+0000","last_active":"2024-04-25T00:30:15.794182+0000","last_peered":"2024-04-25T00:30:15.794182+0000","last_clean":"2024-04-25T00:30:15.794182+0000","last_became_active":"2024-04-25T00:01:04.355293+0000","last_became_peered":"2024-04-25T00:01:04.355293+0000","last_unstale":"2024-04-25T00:30:15.794182+0000","last_undegraded":"2024-04-25T00:30:15.794182+0000","last_fullsized":"2024-04-25T00:30:15.794182+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1087,"log_dups_size":0,"ondisk_log_size":1087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:56:16.678132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":16397,"num_write":1061,"num_write_kb":4715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"70'3760","reported_seq":3888,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.801356+0000","last_change":"2024-04-25T00:01:05.367207+0000","last_active":"2024-04-25T00:30:15.801356+0000","last_peered":"2024-04-25T00:30:15.801356+0000","last_clean":"2024-04-25T00:30:15.801356+0000","last_became_active":"2024-04-25T00:01:05.366471+0000","last_became_peered":"2024-04-25T00:01:05.366471+0000","last_unstale":"2024-04-25T00:30:15.801356+0000","last_undegraded":"2024-04-25T00:30:15.801356+0000","last_fullsized":"2024-04-25T00:30:15.801356+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3760,"log_dups_size":0,"ondisk_log_size":3760,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:30:23.557381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21072,"num_write_kb":264901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"74'3","reported_seq":123,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.743279+0000","last_change":"2024-04-25T00:01:18.502364+0000","last_active":"2024-04-25T00:30:26.743279+0000","last_peered":"2024-04-25T00:30:26.743279+0000","last_clean":"2024-04-25T00:30:26.743279+0000","last_became_active":"2024-04-25T00:01:18.501633+0000","last_became_peered":"2024-04-25T00:01:18.501633+0000","last_unstale":"2024-04-25T00:30:26.743279+0000","last_undegraded":"2024-04-25T00:30:26.743279+0000","last_fullsized":"2024-04-25T00:30:26.743279+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:55.074018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"74'5031","reported_seq":5145,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.765069+0000","last_change":"2024-04-25T00:01:46.662019+0000","last_active":"2024-04-25T00:30:26.765069+0000","last_peered":"2024-04-25T00:30:26.765069+0000","last_clean":"2024-04-25T00:30:26.765069+0000","last_became_active":"2024-04-25T00:01:46.661167+0000","last_became_peered":"2024-04-25T00:01:46.661167+0000","last_unstale":"2024-04-25T00:30:26.765069+0000","last_undegraded":"2024-04-25T00:30:26.765069+0000","last_fullsized":"2024-04-25T00:30:26.765069+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":5031,"log_dups_size":0,"ondisk_log_size":5031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:13:34.153117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":22,"num_write":27250,"num_write_kb":338892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"73'28","reported_seq":158,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.802312+0000","last_change":"2024-04-25T00:01:04.361809+0000","last_active":"2024-04-25T00:30:15.802312+0000","last_peered":"2024-04-25T00:30:15.802312+0000","last_clean":"2024-04-25T00:30:15.802312+0000","last_became_active":"2024-04-25T00:01:04.361059+0000","last_became_peered":"2024-04-25T00:01:04.361059+0000","last_unstale":"2024-04-25T00:30:15.802312+0000","last_undegraded":"2024-04-25T00:30:15.802312+0000","last_fullsized":"2024-04-25T00:30:15.802312+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"70'5376","reported_seq":5504,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.799500+0000","last_change":"2024-04-25T00:01:05.370818+0000","last_active":"2024-04-25T00:30:15.799500+0000","last_peered":"2024-04-25T00:30:15.799500+0000","last_clean":"2024-04-25T00:30:15.799500+0000","last_became_active":"2024-04-25T00:01:05.369991+0000","last_became_peered":"2024-04-25T00:01:05.369991+0000","last_unstale":"2024-04-25T00:30:15.799500+0000","last_undegraded":"2024-04-25T00:30:15.799500+0000","last_fullsized":"2024-04-25T00:30:15.799500+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":5376,"log_dups_size":0,"ondisk_log_size":5376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:43.615895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29984,"num_write_kb":386541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"74'2","reported_seq":122,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:23.313841+0000","last_change":"2024-04-25T00:01:18.497682+0000","last_active":"2024-04-25T00:30:23.313841+0000","last_peered":"2024-04-25T00:30:23.313841+0000","last_clean":"2024-04-25T00:30:23.313841+0000","last_became_active":"2024-04-25T00:01:18.497397+0000","last_became_peered":"2024-04-25T00:01:18.497397+0000","last_unstale":"2024-04-25T00:30:23.313841+0000","last_undegraded":"2024-04-25T00:30:23.313841+0000","last_fullsized":"2024-04-25T00:30:23.313841+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:22.983972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"74'4080","reported_seq":4199,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.684096+0000","last_change":"2024-04-25T00:01:19.510591+0000","last_active":"2024-04-25T00:30:26.684096+0000","last_peered":"2024-04-25T00:30:26.684096+0000","last_clean":"2024-04-25T00:30:26.684096+0000","last_became_active":"2024-04-25T00:01:19.510252+0000","last_became_peered":"2024-04-25T00:01:19.510252+0000","last_unstale":"2024-04-25T00:30:26.684096+0000","last_undegraded":"2024-04-25T00:30:26.684096+0000","last_fullsized":"2024-04-25T00:30:26.684096+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4080,"log_dups_size":0,"ondisk_log_size":4080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:38.005233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":21789,"num_write_kb":296332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"74'53","reported_seq":203,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.793824+0000","last_change":"2024-04-25T00:01:04.362030+0000","last_active":"2024-04-25T00:30:15.793824+0000","last_peered":"2024-04-25T00:30:15.793824+0000","last_clean":"2024-04-25T00:30:15.793824+0000","last_became_active":"2024-04-25T00:01:04.361708+0000","last_became_peered":"2024-04-25T00:01:04.361708+0000","last_unstale":"2024-04-25T00:30:15.793824+0000","last_undegraded":"2024-04-25T00:30:15.793824+0000","last_fullsized":"2024-04-25T00:30:15.793824+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:38:33.392719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":69,"num_read_kb":69,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"69'3669","reported_seq":3794,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.942949+0000","last_change":"2024-04-25T00:01:05.387195+0000","last_active":"2024-04-25T00:30:15.942949+0000","last_peered":"2024-04-25T00:30:15.942949+0000","last_clean":"2024-04-25T00:30:15.942949+0000","last_became_active":"2024-04-25T00:01:05.386853+0000","last_became_peered":"2024-04-25T00:01:05.386853+0000","last_unstale":"2024-04-25T00:30:15.942949+0000","last_undegraded":"2024-04-25T00:30:15.942949+0000","last_fullsized":"2024-04-25T00:30:15.942949+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3669,"log_dups_size":0,"ondisk_log_size":3669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:06:19.307378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19904,"num_write_kb":264274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"74'723","reported_seq":869,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.740062+0000","last_change":"2024-04-25T00:01:18.487835+0000","last_active":"2024-04-25T00:30:26.740062+0000","last_peered":"2024-04-25T00:30:26.740062+0000","last_clean":"2024-04-25T00:30:26.740062+0000","last_became_active":"2024-04-25T00:01:18.487571+0000","last_became_peered":"2024-04-25T00:01:18.487571+0000","last_unstale":"2024-04-25T00:30:26.740062+0000","last_undegraded":"2024-04-25T00:30:26.740062+0000","last_fullsized":"2024-04-25T00:30:26.740062+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":723,"log_dups_size":0,"ondisk_log_size":723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:44.582148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":27,"num_read_kb":82727,"num_write":711,"num_write_kb":4310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"74'3586","reported_seq":3704,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.700653+0000","last_change":"2024-04-25T00:01:19.519834+0000","last_active":"2024-04-25T00:30:26.700653+0000","last_peered":"2024-04-25T00:30:26.700653+0000","last_clean":"2024-04-25T00:30:26.700653+0000","last_became_active":"2024-04-25T00:01:19.519470+0000","last_became_peered":"2024-04-25T00:01:19.519470+0000","last_unstale":"2024-04-25T00:30:26.700653+0000","last_undegraded":"2024-04-25T00:30:26.700653+0000","last_fullsized":"2024-04-25T00:30:26.700653+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3586,"log_dups_size":0,"ondisk_log_size":3586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:47.859626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":5,"num_write":18586,"num_write_kb":254663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"70'6","reported_seq":179,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.793266+0000","last_change":"2024-04-25T00:01:04.362832+0000","last_active":"2024-04-25T00:30:15.793266+0000","last_peered":"2024-04-25T00:30:15.793266+0000","last_clean":"2024-04-25T00:30:15.793266+0000","last_became_active":"2024-04-25T00:01:04.362059+0000","last_became_peered":"2024-04-25T00:01:04.362059+0000","last_unstale":"2024-04-25T00:30:15.793266+0000","last_undegraded":"2024-04-25T00:30:15.793266+0000","last_fullsized":"2024-04-25T00:30:15.793266+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:05.441048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"70'4973","reported_seq":5098,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.813402+0000","last_change":"2024-04-25T00:01:05.381120+0000","last_active":"2024-04-25T00:30:15.813402+0000","last_peered":"2024-04-25T00:30:15.813402+0000","last_clean":"2024-04-25T00:30:15.813402+0000","last_became_active":"2024-04-25T00:01:05.378320+0000","last_became_peered":"2024-04-25T00:01:05.378320+0000","last_unstale":"2024-04-25T00:30:15.813402+0000","last_undegraded":"2024-04-25T00:30:15.813402+0000","last_fullsized":"2024-04-25T00:30:15.813402+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4973,"log_dups_size":0,"ondisk_log_size":4973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:34:06.175918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27225,"num_write_kb":350946,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"74'355","reported_seq":476,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:31.515142+0000","last_change":"2024-04-25T00:01:18.490300+0000","last_active":"2024-04-25T00:30:31.515142+0000","last_peered":"2024-04-25T00:30:31.515142+0000","last_clean":"2024-04-25T00:30:31.515142+0000","last_became_active":"2024-04-25T00:01:18.490050+0000","last_became_peered":"2024-04-25T00:01:18.490050+0000","last_unstale":"2024-04-25T00:30:31.515142+0000","last_undegraded":"2024-04-25T00:30:31.515142+0000","last_fullsized":"2024-04-25T00:30:31.515142+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":355,"log_dups_size":0,"ondisk_log_size":355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:13:05.971606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2691528,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":907,"num_write":326,"num_write_kb":2825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"74'4320","reported_seq":4442,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.772385+0000","last_change":"2024-04-25T00:01:19.500026+0000","last_active":"2024-04-25T00:30:26.772385+0000","last_peered":"2024-04-25T00:30:26.772385+0000","last_clean":"2024-04-25T00:30:26.772385+0000","last_became_active":"2024-04-25T00:01:19.499244+0000","last_became_peered":"2024-04-25T00:01:19.499244+0000","last_unstale":"2024-04-25T00:30:26.772385+0000","last_undegraded":"2024-04-25T00:30:26.772385+0000","last_fullsized":"2024-04-25T00:30:26.772385+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4320,"log_dups_size":0,"ondisk_log_size":4320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:40:36.350832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":15,"num_write":23810,"num_write_kb":308534,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"70'4158","reported_seq":4285,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.809005+0000","last_change":"2024-04-25T00:01:05.376984+0000","last_active":"2024-04-25T00:30:15.809005+0000","last_peered":"2024-04-25T00:30:15.809005+0000","last_clean":"2024-04-25T00:30:15.809005+0000","last_became_active":"2024-04-25T00:01:05.376259+0000","last_became_peered":"2024-04-25T00:01:05.376259+0000","last_unstale":"2024-04-25T00:30:15.809005+0000","last_undegraded":"2024-04-25T00:30:15.809005+0000","last_fullsized":"2024-04-25T00:30:15.809005+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4158,"log_dups_size":0,"ondisk_log_size":4158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:13:28.543302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23006,"num_write_kb":301346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"74'1526","reported_seq":1710,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:23.032874+0000","last_change":"2024-04-25T00:01:04.359262+0000","last_active":"2024-04-25T00:30:23.032874+0000","last_peered":"2024-04-25T00:30:23.032874+0000","last_clean":"2024-04-25T00:30:23.032874+0000","last_became_active":"2024-04-25T00:01:04.357971+0000","last_became_peered":"2024-04-25T00:01:04.357971+0000","last_unstale":"2024-04-25T00:30:23.032874+0000","last_undegraded":"2024-04-25T00:30:23.032874+0000","last_fullsized":"2024-04-25T00:30:23.032874+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1526,"log_dups_size":0,"ondisk_log_size":1526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:05.441048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":78,"num_read_kb":52062,"num_write":1499,"num_write_kb":4525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"74'6","reported_seq":150,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.768405+0000","last_change":"2024-04-25T00:01:18.475624+0000","last_active":"2024-04-25T00:30:26.768405+0000","last_peered":"2024-04-25T00:30:26.768405+0000","last_clean":"2024-04-25T00:30:26.768405+0000","last_became_active":"2024-04-25T00:01:18.475149+0000","last_became_peered":"2024-04-25T00:01:18.475149+0000","last_unstale":"2024-04-25T00:30:26.768405+0000","last_undegraded":"2024-04-25T00:30:26.768405+0000","last_fullsized":"2024-04-25T00:30:26.768405+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:40:35.341075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"74'4556","reported_seq":4678,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.774727+0000","last_change":"2024-04-25T00:01:19.531113+0000","last_active":"2024-04-25T00:30:26.774727+0000","last_peered":"2024-04-25T00:30:26.774727+0000","last_clean":"2024-04-25T00:30:26.774727+0000","last_became_active":"2024-04-25T00:01:19.520150+0000","last_became_peered":"2024-04-25T00:01:19.520150+0000","last_unstale":"2024-04-25T00:30:26.774727+0000","last_undegraded":"2024-04-25T00:30:26.774727+0000","last_fullsized":"2024-04-25T00:30:26.774727+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4556,"log_dups_size":0,"ondisk_log_size":4556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:38.152376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":19,"num_write":25634,"num_write_kb":315935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"70'3060","reported_seq":3187,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.803143+0000","last_change":"2024-04-25T00:01:05.362611+0000","last_active":"2024-04-25T00:30:15.803143+0000","last_peered":"2024-04-25T00:30:15.803143+0000","last_clean":"2024-04-25T00:30:15.803143+0000","last_became_active":"2024-04-25T00:01:05.361817+0000","last_became_peered":"2024-04-25T00:01:05.361817+0000","last_unstale":"2024-04-25T00:30:15.803143+0000","last_undegraded":"2024-04-25T00:30:15.803143+0000","last_fullsized":"2024-04-25T00:30:15.803143+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3060,"log_dups_size":0,"ondisk_log_size":3060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:50.489125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16241,"num_write_kb":228970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"70'3","reported_seq":132,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.810550+0000","last_change":"2024-04-25T00:01:04.371935+0000","last_active":"2024-04-25T00:30:15.810550+0000","last_peered":"2024-04-25T00:30:15.810550+0000","last_clean":"2024-04-25T00:30:15.810550+0000","last_became_active":"2024-04-25T00:01:04.371420+0000","last_became_peered":"2024-04-25T00:01:04.371420+0000","last_unstale":"2024-04-25T00:30:15.810550+0000","last_undegraded":"2024-04-25T00:30:15.810550+0000","last_fullsized":"2024-04-25T00:30:15.810550+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"74'5","reported_seq":126,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.747597+0000","last_change":"2024-04-25T00:01:18.488623+0000","last_active":"2024-04-25T00:30:26.747597+0000","last_peered":"2024-04-25T00:30:26.747597+0000","last_clean":"2024-04-25T00:30:26.747597+0000","last_became_active":"2024-04-25T00:01:18.488291+0000","last_became_peered":"2024-04-25T00:01:18.488291+0000","last_unstale":"2024-04-25T00:30:26.747597+0000","last_undegraded":"2024-04-25T00:30:26.747597+0000","last_fullsized":"2024-04-25T00:30:26.747597+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:30:21.436681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"74'3591","reported_seq":3711,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.734674+0000","last_change":"2024-04-25T00:01:19.516731+0000","last_active":"2024-04-25T00:30:26.734674+0000","last_peered":"2024-04-25T00:30:26.734674+0000","last_clean":"2024-04-25T00:30:26.734674+0000","last_became_active":"2024-04-25T00:01:19.516405+0000","last_became_peered":"2024-04-25T00:01:19.516405+0000","last_unstale":"2024-04-25T00:30:26.734674+0000","last_undegraded":"2024-04-25T00:30:26.734674+0000","last_fullsized":"2024-04-25T00:30:26.734674+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3591,"log_dups_size":0,"ondisk_log_size":3591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:44:37.936092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":19584,"num_write_kb":250628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"69'3777","reported_seq":3904,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.790888+0000","last_change":"2024-04-25T00:01:05.355455+0000","last_active":"2024-04-25T00:30:15.790888+0000","last_peered":"2024-04-25T00:30:15.790888+0000","last_clean":"2024-04-25T00:30:15.790888+0000","last_became_active":"2024-04-25T00:01:05.354699+0000","last_became_peered":"2024-04-25T00:01:05.354699+0000","last_unstale":"2024-04-25T00:30:15.790888+0000","last_undegraded":"2024-04-25T00:30:15.790888+0000","last_fullsized":"2024-04-25T00:30:15.790888+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3777,"log_dups_size":0,"ondisk_log_size":3777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:43:58.069271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20993,"num_write_kb":256390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"69'2","reported_seq":131,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.791647+0000","last_change":"2024-04-25T00:01:04.362268+0000","last_active":"2024-04-25T00:30:15.791647+0000","last_peered":"2024-04-25T00:30:15.791647+0000","last_clean":"2024-04-25T00:30:15.791647+0000","last_became_active":"2024-04-25T00:01:04.361622+0000","last_became_peered":"2024-04-25T00:01:04.361622+0000","last_unstale":"2024-04-25T00:30:15.791647+0000","last_undegraded":"2024-04-25T00:30:15.791647+0000","last_fullsized":"2024-04-25T00:30:15.791647+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:58:46.021343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"74'4","reported_seq":147,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.744595+0000","last_change":"2024-04-25T00:01:18.494047+0000","last_active":"2024-04-25T00:30:26.744595+0000","last_peered":"2024-04-25T00:30:26.744595+0000","last_clean":"2024-04-25T00:30:26.744595+0000","last_became_active":"2024-04-25T00:01:18.493614+0000","last_became_peered":"2024-04-25T00:01:18.493614+0000","last_unstale":"2024-04-25T00:30:26.744595+0000","last_undegraded":"2024-04-25T00:30:26.744595+0000","last_fullsized":"2024-04-25T00:30:26.744595+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:37:08.562016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"74'4895","reported_seq":5013,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.674895+0000","last_change":"2024-04-25T00:01:19.505397+0000","last_active":"2024-04-25T00:30:26.674895+0000","last_peered":"2024-04-25T00:30:26.674895+0000","last_clean":"2024-04-25T00:30:26.674895+0000","last_became_active":"2024-04-25T00:01:19.505126+0000","last_became_peered":"2024-04-25T00:01:19.505126+0000","last_unstale":"2024-04-25T00:30:26.674895+0000","last_undegraded":"2024-04-25T00:30:26.674895+0000","last_fullsized":"2024-04-25T00:30:26.674895+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4895,"log_dups_size":0,"ondisk_log_size":4895,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:08:49.640267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26799,"num_write_kb":349350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"69'3934","reported_seq":4061,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.805653+0000","last_change":"2024-04-25T00:01:05.354953+0000","last_active":"2024-04-25T00:30:15.805653+0000","last_peered":"2024-04-25T00:30:15.805653+0000","last_clean":"2024-04-25T00:30:15.805653+0000","last_became_active":"2024-04-25T00:01:05.354118+0000","last_became_peered":"2024-04-25T00:01:05.354118+0000","last_unstale":"2024-04-25T00:30:15.805653+0000","last_undegraded":"2024-04-25T00:30:15.805653+0000","last_fullsized":"2024-04-25T00:30:15.805653+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3934,"log_dups_size":0,"ondisk_log_size":3934,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:09.858617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21059,"num_write_kb":273031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"70'6","reported_seq":135,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.789548+0000","last_change":"2024-04-25T00:01:04.365515+0000","last_active":"2024-04-25T00:30:15.789548+0000","last_peered":"2024-04-25T00:30:15.789548+0000","last_clean":"2024-04-25T00:30:15.789548+0000","last_became_active":"2024-04-25T00:01:04.364942+0000","last_became_peered":"2024-04-25T00:01:04.364942+0000","last_unstale":"2024-04-25T00:30:15.789548+0000","last_undegraded":"2024-04-25T00:30:15.789548+0000","last_fullsized":"2024-04-25T00:30:15.789548+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"74'7","reported_seq":174,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.741808+0000","last_change":"2024-04-25T00:01:18.476207+0000","last_active":"2024-04-25T00:30:26.741808+0000","last_peered":"2024-04-25T00:30:26.741808+0000","last_clean":"2024-04-25T00:30:26.741808+0000","last_became_active":"2024-04-25T00:01:18.475862+0000","last_became_peered":"2024-04-25T00:01:18.475862+0000","last_unstale":"2024-04-25T00:30:26.741808+0000","last_undegraded":"2024-04-25T00:30:26.741808+0000","last_fullsized":"2024-04-25T00:30:26.741808+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:11:40.170249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"74'3090","reported_seq":3210,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.776689+0000","last_change":"2024-04-25T00:01:19.529607+0000","last_active":"2024-04-25T00:30:26.776689+0000","last_peered":"2024-04-25T00:30:26.776689+0000","last_clean":"2024-04-25T00:30:26.776689+0000","last_became_active":"2024-04-25T00:01:19.528509+0000","last_became_peered":"2024-04-25T00:01:19.528509+0000","last_unstale":"2024-04-25T00:30:26.776689+0000","last_undegraded":"2024-04-25T00:30:26.776689+0000","last_fullsized":"2024-04-25T00:30:26.776689+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3090,"log_dups_size":0,"ondisk_log_size":3090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:45:01.068091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17043,"num_write_kb":228060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"70'4336","reported_seq":4463,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.793683+0000","last_change":"2024-04-25T00:01:05.358106+0000","last_active":"2024-04-25T00:30:15.793683+0000","last_peered":"2024-04-25T00:30:15.793683+0000","last_clean":"2024-04-25T00:30:15.793683+0000","last_became_active":"2024-04-25T00:01:05.357135+0000","last_became_peered":"2024-04-25T00:01:05.357135+0000","last_unstale":"2024-04-25T00:30:15.793683+0000","last_undegraded":"2024-04-25T00:30:15.793683+0000","last_fullsized":"2024-04-25T00:30:15.793683+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4336,"log_dups_size":0,"ondisk_log_size":4336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:41:42.228234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23030,"num_write_kb":325887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"71'9","reported_seq":184,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.814467+0000","last_change":"2024-04-25T00:01:04.369917+0000","last_active":"2024-04-25T00:30:15.814467+0000","last_peered":"2024-04-25T00:30:15.814467+0000","last_clean":"2024-04-25T00:30:15.814467+0000","last_became_active":"2024-04-25T00:01:04.369309+0000","last_became_peered":"2024-04-25T00:01:04.369309+0000","last_unstale":"2024-04-25T00:30:15.814467+0000","last_undegraded":"2024-04-25T00:30:15.814467+0000","last_fullsized":"2024-04-25T00:30:15.814467+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:55:08.063965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":8,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"74'51","reported_seq":241,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:28.848602+0000","last_change":"2024-04-25T00:01:18.477189+0000","last_active":"2024-04-25T00:30:28.848602+0000","last_peered":"2024-04-25T00:30:28.848602+0000","last_clean":"2024-04-25T00:30:28.848602+0000","last_became_active":"2024-04-25T00:01:18.476909+0000","last_became_peered":"2024-04-25T00:01:18.476909+0000","last_unstale":"2024-04-25T00:30:28.848602+0000","last_undegraded":"2024-04-25T00:30:28.848602+0000","last_fullsized":"2024-04-25T00:30:28.848602+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:36.995477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84646,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":68,"num_read_kb":112,"num_write":46,"num_write_kb":112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"74'3821","reported_seq":3942,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.704068+0000","last_change":"2024-04-25T00:01:19.517500+0000","last_active":"2024-04-25T00:30:26.704068+0000","last_peered":"2024-04-25T00:30:26.704068+0000","last_clean":"2024-04-25T00:30:26.704068+0000","last_became_active":"2024-04-25T00:01:19.517182+0000","last_became_peered":"2024-04-25T00:01:19.517182+0000","last_unstale":"2024-04-25T00:30:26.704068+0000","last_undegraded":"2024-04-25T00:30:26.704068+0000","last_fullsized":"2024-04-25T00:30:26.704068+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3821,"log_dups_size":0,"ondisk_log_size":3821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:58:49.738233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20929,"num_write_kb":256020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"70'3227","reported_seq":3354,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.793834+0000","last_change":"2024-04-25T00:01:05.364829+0000","last_active":"2024-04-25T00:30:15.793834+0000","last_peered":"2024-04-25T00:30:15.793834+0000","last_clean":"2024-04-25T00:30:15.793834+0000","last_became_active":"2024-04-25T00:01:05.364125+0000","last_became_peered":"2024-04-25T00:01:05.364125+0000","last_unstale":"2024-04-25T00:30:15.793834+0000","last_undegraded":"2024-04-25T00:30:15.793834+0000","last_fullsized":"2024-04-25T00:30:15.793834+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3227,"log_dups_size":0,"ondisk_log_size":3227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:15:45.531760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17808,"num_write_kb":231231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"74'458","reported_seq":610,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:25.796746+0000","last_change":"2024-04-25T00:01:04.361603+0000","last_active":"2024-04-25T00:30:25.796746+0000","last_peered":"2024-04-25T00:30:25.796746+0000","last_clean":"2024-04-25T00:30:25.796746+0000","last_became_active":"2024-04-25T00:01:04.361046+0000","last_became_peered":"2024-04-25T00:01:04.361046+0000","last_unstale":"2024-04-25T00:30:25.796746+0000","last_undegraded":"2024-04-25T00:30:25.796746+0000","last_fullsized":"2024-04-25T00:30:25.796746+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":458,"log_dups_size":0,"ondisk_log_size":458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:35:18.036843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":46,"num_read_kb":63,"num_write":1069,"num_write_kb":360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"74'12","reported_seq":134,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.743234+0000","last_change":"2024-04-25T00:01:18.488352+0000","last_active":"2024-04-25T00:30:26.743234+0000","last_peered":"2024-04-25T00:30:26.743234+0000","last_clean":"2024-04-25T00:30:26.743234+0000","last_became_active":"2024-04-25T00:01:18.487321+0000","last_became_peered":"2024-04-25T00:01:18.487321+0000","last_unstale":"2024-04-25T00:30:26.743234+0000","last_undegraded":"2024-04-25T00:30:26.743234+0000","last_fullsized":"2024-04-25T00:30:26.743234+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:37:22.329655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"74'4221","reported_seq":4341,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.775265+0000","last_change":"2024-04-25T00:01:19.519653+0000","last_active":"2024-04-25T00:30:26.775265+0000","last_peered":"2024-04-25T00:30:26.775265+0000","last_clean":"2024-04-25T00:30:26.775265+0000","last_became_active":"2024-04-25T00:01:19.519052+0000","last_became_peered":"2024-04-25T00:01:19.519052+0000","last_unstale":"2024-04-25T00:30:26.775265+0000","last_undegraded":"2024-04-25T00:30:26.775265+0000","last_fullsized":"2024-04-25T00:30:26.775265+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4221,"log_dups_size":0,"ondisk_log_size":4221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:40:36.350832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22937,"num_write_kb":280510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"70'5174","reported_seq":5302,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.813035+0000","last_change":"2024-04-25T00:01:05.374011+0000","last_active":"2024-04-25T00:30:15.813035+0000","last_peered":"2024-04-25T00:30:15.813035+0000","last_clean":"2024-04-25T00:30:15.813035+0000","last_became_active":"2024-04-25T00:01:05.373276+0000","last_became_peered":"2024-04-25T00:01:05.373276+0000","last_unstale":"2024-04-25T00:30:15.813035+0000","last_undegraded":"2024-04-25T00:30:15.813035+0000","last_fullsized":"2024-04-25T00:30:15.813035+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":5174,"log_dups_size":0,"ondisk_log_size":5174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:40:53.349428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28022,"num_write_kb":362384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"69'2","reported_seq":132,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.814680+0000","last_change":"2024-04-25T00:01:04.365391+0000","last_active":"2024-04-25T00:30:15.814680+0000","last_peered":"2024-04-25T00:30:15.814680+0000","last_clean":"2024-04-25T00:30:15.814680+0000","last_became_active":"2024-04-25T00:01:04.364989+0000","last_became_peered":"2024-04-25T00:01:04.364989+0000","last_unstale":"2024-04-25T00:30:15.814680+0000","last_undegraded":"2024-04-25T00:30:15.814680+0000","last_fullsized":"2024-04-25T00:30:15.814680+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:06:18.298057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"74'3","reported_seq":125,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.737749+0000","last_change":"2024-04-25T00:01:18.491597+0000","last_active":"2024-04-25T00:30:26.737749+0000","last_peered":"2024-04-25T00:30:26.737749+0000","last_clean":"2024-04-25T00:30:26.737749+0000","last_became_active":"2024-04-25T00:01:18.490859+0000","last_became_peered":"2024-04-25T00:01:18.490859+0000","last_unstale":"2024-04-25T00:30:26.737749+0000","last_undegraded":"2024-04-25T00:30:26.737749+0000","last_fullsized":"2024-04-25T00:30:26.737749+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:36.995477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"74'5316","reported_seq":5429,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.548329+0000","last_change":"2024-04-25T00:01:46.667934+0000","last_active":"2024-04-25T00:30:26.548329+0000","last_peered":"2024-04-25T00:30:26.548329+0000","last_clean":"2024-04-25T00:30:26.548329+0000","last_became_active":"2024-04-25T00:01:46.667075+0000","last_became_peered":"2024-04-25T00:01:46.667075+0000","last_unstale":"2024-04-25T00:30:26.548329+0000","last_undegraded":"2024-04-25T00:30:26.548329+0000","last_fullsized":"2024-04-25T00:30:26.548329+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":5316,"log_dups_size":0,"ondisk_log_size":5316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:28:18.234546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29390,"num_write_kb":363802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"70'3961","reported_seq":4089,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.804843+0000","last_change":"2024-04-25T00:01:05.378340+0000","last_active":"2024-04-25T00:30:15.804843+0000","last_peered":"2024-04-25T00:30:15.804843+0000","last_clean":"2024-04-25T00:30:15.804843+0000","last_became_active":"2024-04-25T00:01:05.377593+0000","last_became_peered":"2024-04-25T00:01:05.377593+0000","last_unstale":"2024-04-25T00:30:15.804843+0000","last_undegraded":"2024-04-25T00:30:15.804843+0000","last_fullsized":"2024-04-25T00:30:15.804843+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3961,"log_dups_size":0,"ondisk_log_size":3961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:48.202251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21237,"num_write_kb":269574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"70'2","reported_seq":132,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.799354+0000","last_change":"2024-04-25T00:01:04.366577+0000","last_active":"2024-04-25T00:30:15.799354+0000","last_peered":"2024-04-25T00:30:15.799354+0000","last_clean":"2024-04-25T00:30:15.799354+0000","last_became_active":"2024-04-25T00:01:04.365881+0000","last_became_peered":"2024-04-25T00:01:04.365881+0000","last_unstale":"2024-04-25T00:30:15.799354+0000","last_undegraded":"2024-04-25T00:30:15.799354+0000","last_fullsized":"2024-04-25T00:30:15.799354+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:49:22.673421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'4","reported_seq":126,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.132180+0000","last_change":"2024-04-25T00:01:18.504341+0000","last_active":"2024-04-25T00:30:26.132180+0000","last_peered":"2024-04-25T00:30:26.132180+0000","last_clean":"2024-04-25T00:30:26.132180+0000","last_became_active":"2024-04-25T00:01:18.491498+0000","last_became_peered":"2024-04-25T00:01:18.491498+0000","last_unstale":"2024-04-25T00:30:26.132180+0000","last_undegraded":"2024-04-25T00:30:26.132180+0000","last_fullsized":"2024-04-25T00:30:26.132180+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:56.741250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"74'4156","reported_seq":4283,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.774804+0000","last_change":"2024-04-25T00:01:46.667022+0000","last_active":"2024-04-25T00:30:26.774804+0000","last_peered":"2024-04-25T00:30:26.774804+0000","last_clean":"2024-04-25T00:30:26.774804+0000","last_became_active":"2024-04-25T00:01:46.666224+0000","last_became_peered":"2024-04-25T00:01:46.666224+0000","last_unstale":"2024-04-25T00:30:26.774804+0000","last_undegraded":"2024-04-25T00:30:26.774804+0000","last_fullsized":"2024-04-25T00:30:26.774804+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4156,"log_dups_size":0,"ondisk_log_size":4156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:25:20.185355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":7,"num_write":23008,"num_write_kb":280758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"69'4491","reported_seq":4619,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.793150+0000","last_change":"2024-04-25T00:01:05.368769+0000","last_active":"2024-04-25T00:30:15.793150+0000","last_peered":"2024-04-25T00:30:15.793150+0000","last_clean":"2024-04-25T00:30:15.793150+0000","last_became_active":"2024-04-25T00:01:05.366817+0000","last_became_peered":"2024-04-25T00:01:05.366817+0000","last_unstale":"2024-04-25T00:30:15.793150+0000","last_undegraded":"2024-04-25T00:30:15.793150+0000","last_fullsized":"2024-04-25T00:30:15.793150+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4491,"log_dups_size":0,"ondisk_log_size":4491,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:11:27.044893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24298,"num_write_kb":330898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"69'5","reported_seq":135,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.810345+0000","last_change":"2024-04-25T00:01:04.361900+0000","last_active":"2024-04-25T00:30:15.810345+0000","last_peered":"2024-04-25T00:30:15.810345+0000","last_clean":"2024-04-25T00:30:15.810345+0000","last_became_active":"2024-04-25T00:01:04.361572+0000","last_became_peered":"2024-04-25T00:01:04.361572+0000","last_unstale":"2024-04-25T00:30:15.810345+0000","last_undegraded":"2024-04-25T00:30:15.810345+0000","last_fullsized":"2024-04-25T00:30:15.810345+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:29.218739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"74'2","reported_seq":124,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.744419+0000","last_change":"2024-04-25T00:01:18.504139+0000","last_active":"2024-04-25T00:30:26.744419+0000","last_peered":"2024-04-25T00:30:26.744419+0000","last_clean":"2024-04-25T00:30:26.744419+0000","last_became_active":"2024-04-25T00:01:18.503379+0000","last_became_peered":"2024-04-25T00:01:18.503379+0000","last_unstale":"2024-04-25T00:30:26.744419+0000","last_undegraded":"2024-04-25T00:30:26.744419+0000","last_fullsized":"2024-04-25T00:30:26.744419+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:02:56.363258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"74'5402","reported_seq":5525,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.707463+0000","last_change":"2024-04-25T00:01:19.531756+0000","last_active":"2024-04-25T00:30:26.707463+0000","last_peered":"2024-04-25T00:30:26.707463+0000","last_clean":"2024-04-25T00:30:26.707463+0000","last_became_active":"2024-04-25T00:01:19.531145+0000","last_became_peered":"2024-04-25T00:01:19.531145+0000","last_unstale":"2024-04-25T00:30:26.707463+0000","last_undegraded":"2024-04-25T00:30:26.707463+0000","last_fullsized":"2024-04-25T00:30:26.707463+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":5402,"log_dups_size":0,"ondisk_log_size":5402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:13:42.678415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":13,"num_write":29532,"num_write_kb":356063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"70'3335","reported_seq":3463,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.794572+0000","last_change":"2024-04-25T00:01:05.354557+0000","last_active":"2024-04-25T00:30:15.794572+0000","last_peered":"2024-04-25T00:30:15.794572+0000","last_clean":"2024-04-25T00:30:15.794572+0000","last_became_active":"2024-04-25T00:01:05.353556+0000","last_became_peered":"2024-04-25T00:01:05.353556+0000","last_unstale":"2024-04-25T00:30:15.794572+0000","last_undegraded":"2024-04-25T00:30:15.794572+0000","last_fullsized":"2024-04-25T00:30:15.794572+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3335,"log_dups_size":0,"ondisk_log_size":3335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:24.964055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18377,"num_write_kb":241696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"74'53","reported_seq":252,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:25.820987+0000","last_change":"2024-04-25T00:01:04.372916+0000","last_active":"2024-04-25T00:30:25.820987+0000","last_peered":"2024-04-25T00:30:25.820987+0000","last_clean":"2024-04-25T00:30:25.820987+0000","last_became_active":"2024-04-25T00:01:04.372185+0000","last_became_peered":"2024-04-25T00:01:04.372185+0000","last_unstale":"2024-04-25T00:30:25.820987+0000","last_undegraded":"2024-04-25T00:30:25.820987+0000","last_fullsized":"2024-04-25T00:30:25.820987+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88686,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":69,"num_read_kb":117,"num_write":46,"num_write_kb":117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"74'9","reported_seq":176,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.750624+0000","last_change":"2024-04-25T00:01:18.504614+0000","last_active":"2024-04-25T00:30:26.750624+0000","last_peered":"2024-04-25T00:30:26.750624+0000","last_clean":"2024-04-25T00:30:26.750624+0000","last_became_active":"2024-04-25T00:01:18.501448+0000","last_became_peered":"2024-04-25T00:01:18.501448+0000","last_unstale":"2024-04-25T00:30:26.750624+0000","last_undegraded":"2024-04-25T00:30:26.750624+0000","last_fullsized":"2024-04-25T00:30:26.750624+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:22.983972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":8,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"74'3337","reported_seq":3456,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.763497+0000","last_change":"2024-04-25T00:01:19.517510+0000","last_active":"2024-04-25T00:30:26.763497+0000","last_peered":"2024-04-25T00:30:26.763497+0000","last_clean":"2024-04-25T00:30:26.763497+0000","last_became_active":"2024-04-25T00:01:19.517243+0000","last_became_peered":"2024-04-25T00:01:19.517243+0000","last_unstale":"2024-04-25T00:30:26.763497+0000","last_undegraded":"2024-04-25T00:30:26.763497+0000","last_fullsized":"2024-04-25T00:30:26.763497+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3337,"log_dups_size":0,"ondisk_log_size":3337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:42:38.149502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18533,"num_write_kb":242287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"69'3723","reported_seq":3852,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.809022+0000","last_change":"2024-04-25T00:01:05.370968+0000","last_active":"2024-04-25T00:30:15.809022+0000","last_peered":"2024-04-25T00:30:15.809022+0000","last_clean":"2024-04-25T00:30:15.809022+0000","last_became_active":"2024-04-25T00:01:05.369987+0000","last_became_peered":"2024-04-25T00:01:05.369987+0000","last_unstale":"2024-04-25T00:30:15.809022+0000","last_undegraded":"2024-04-25T00:30:15.809022+0000","last_fullsized":"2024-04-25T00:30:15.809022+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3723,"log_dups_size":0,"ondisk_log_size":3723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20208,"num_write_kb":269376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"73'13","reported_seq":143,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.804977+0000","last_change":"2024-04-25T00:01:04.370226+0000","last_active":"2024-04-25T00:30:15.804977+0000","last_peered":"2024-04-25T00:30:15.804977+0000","last_clean":"2024-04-25T00:30:15.804977+0000","last_became_active":"2024-04-25T00:01:04.369904+0000","last_became_peered":"2024-04-25T00:01:04.369904+0000","last_unstale":"2024-04-25T00:30:15.804977+0000","last_undegraded":"2024-04-25T00:30:15.804977+0000","last_fullsized":"2024-04-25T00:30:15.804977+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:47:25.595275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"74'487","reported_seq":631,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.794202+0000","last_change":"2024-04-25T00:01:18.497156+0000","last_active":"2024-04-25T00:30:26.794202+0000","last_peered":"2024-04-25T00:30:26.794202+0000","last_clean":"2024-04-25T00:30:26.794202+0000","last_became_active":"2024-04-25T00:01:18.495573+0000","last_became_peered":"2024-04-25T00:01:18.495573+0000","last_unstale":"2024-04-25T00:30:26.794202+0000","last_undegraded":"2024-04-25T00:30:26.794202+0000","last_fullsized":"2024-04-25T00:30:26.794202+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":487,"log_dups_size":0,"ondisk_log_size":487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:29.119299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":46,"num_read_kb":69,"num_write":1147,"num_write_kb":382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"74'5079","reported_seq":5200,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.738941+0000","last_change":"2024-04-25T00:01:19.519411+0000","last_active":"2024-04-25T00:30:26.738941+0000","last_peered":"2024-04-25T00:30:26.738941+0000","last_clean":"2024-04-25T00:30:26.738941+0000","last_became_active":"2024-04-25T00:01:19.519020+0000","last_became_peered":"2024-04-25T00:01:19.519020+0000","last_unstale":"2024-04-25T00:30:26.738941+0000","last_undegraded":"2024-04-25T00:30:26.738941+0000","last_fullsized":"2024-04-25T00:30:26.738941+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":5079,"log_dups_size":0,"ondisk_log_size":5079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:50.351599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":14,"num_write":27774,"num_write_kb":353080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"70'3676","reported_seq":3804,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.793841+0000","last_change":"2024-04-25T00:01:05.364939+0000","last_active":"2024-04-25T00:30:15.793841+0000","last_peered":"2024-04-25T00:30:15.793841+0000","last_clean":"2024-04-25T00:30:15.793841+0000","last_became_active":"2024-04-25T00:01:05.364272+0000","last_became_peered":"2024-04-25T00:01:05.364272+0000","last_unstale":"2024-04-25T00:30:15.793841+0000","last_undegraded":"2024-04-25T00:30:15.793841+0000","last_fullsized":"2024-04-25T00:30:15.793841+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3676,"log_dups_size":0,"ondisk_log_size":3676,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:24:59.679908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19925,"num_write_kb":256146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"69'3","reported_seq":133,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.809642+0000","last_change":"2024-04-25T00:01:04.357028+0000","last_active":"2024-04-25T00:30:15.809642+0000","last_peered":"2024-04-25T00:30:15.809642+0000","last_clean":"2024-04-25T00:30:15.809642+0000","last_became_active":"2024-04-25T00:01:04.356644+0000","last_became_peered":"2024-04-25T00:01:04.356644+0000","last_unstale":"2024-04-25T00:30:15.809642+0000","last_undegraded":"2024-04-25T00:30:15.809642+0000","last_fullsized":"2024-04-25T00:30:15.809642+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:21.729893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":286,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.942363+0000","last_change":"2024-04-25T00:00:55.084492+0000","last_active":"2024-04-25T00:30:15.942363+0000","last_peered":"2024-04-25T00:30:15.942363+0000","last_clean":"2024-04-25T00:30:15.942363+0000","last_became_active":"2024-04-25T00:00:55.083493+0000","last_became_peered":"2024-04-25T00:00:55.083493+0000","last_unstale":"2024-04-25T00:30:15.942363+0000","last_undegraded":"2024-04-25T00:30:15.942363+0000","last_fullsized":"2024-04-25T00:30:15.942363+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:00:54.060578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:00:54.060578+0000","last_clean_scrub_stamp":"2024-04-25T00:00:54.060578+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:14:58.472477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"74'2","reported_seq":124,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.137996+0000","last_change":"2024-04-25T00:01:18.476298+0000","last_active":"2024-04-25T00:30:26.137996+0000","last_peered":"2024-04-25T00:30:26.137996+0000","last_clean":"2024-04-25T00:30:26.137996+0000","last_became_active":"2024-04-25T00:01:18.475898+0000","last_became_peered":"2024-04-25T00:01:18.475898+0000","last_unstale":"2024-04-25T00:30:26.137996+0000","last_undegraded":"2024-04-25T00:30:26.137996+0000","last_fullsized":"2024-04-25T00:30:26.137996+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:49.341842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"74'3757","reported_seq":3877,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.774597+0000","last_change":"2024-04-25T00:01:19.518102+0000","last_active":"2024-04-25T00:30:26.774597+0000","last_peered":"2024-04-25T00:30:26.774597+0000","last_clean":"2024-04-25T00:30:26.774597+0000","last_became_active":"2024-04-25T00:01:19.517376+0000","last_became_peered":"2024-04-25T00:01:19.517376+0000","last_unstale":"2024-04-25T00:30:26.774597+0000","last_undegraded":"2024-04-25T00:30:26.774597+0000","last_fullsized":"2024-04-25T00:30:26.774597+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3757,"log_dups_size":0,"ondisk_log_size":3757,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:55:46.853565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":2,"num_write":20920,"num_write_kb":243770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"70'4367","reported_seq":4495,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.813789+0000","last_change":"2024-04-25T00:01:05.376609+0000","last_active":"2024-04-25T00:30:15.813789+0000","last_peered":"2024-04-25T00:30:15.813789+0000","last_clean":"2024-04-25T00:30:15.813789+0000","last_became_active":"2024-04-25T00:01:05.375881+0000","last_became_peered":"2024-04-25T00:01:05.375881+0000","last_unstale":"2024-04-25T00:30:15.813789+0000","last_undegraded":"2024-04-25T00:30:15.813789+0000","last_fullsized":"2024-04-25T00:30:15.813789+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4367,"log_dups_size":0,"ondisk_log_size":4367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:46:05.592233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":2,"num_write":23695,"num_write_kb":308143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"70'6","reported_seq":135,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.813175+0000","last_change":"2024-04-25T00:01:04.368954+0000","last_active":"2024-04-25T00:30:15.813175+0000","last_peered":"2024-04-25T00:30:15.813175+0000","last_clean":"2024-04-25T00:30:15.813175+0000","last_became_active":"2024-04-25T00:01:04.367903+0000","last_became_peered":"2024-04-25T00:01:04.367903+0000","last_unstale":"2024-04-25T00:30:15.813175+0000","last_undegraded":"2024-04-25T00:30:15.813175+0000","last_fullsized":"2024-04-25T00:30:15.813175+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"74'3","reported_seq":124,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.737742+0000","last_change":"2024-04-25T00:01:18.476525+0000","last_active":"2024-04-25T00:30:26.737742+0000","last_peered":"2024-04-25T00:30:26.737742+0000","last_clean":"2024-04-25T00:30:26.737742+0000","last_became_active":"2024-04-25T00:01:18.476247+0000","last_became_peered":"2024-04-25T00:01:18.476247+0000","last_unstale":"2024-04-25T00:30:26.737742+0000","last_undegraded":"2024-04-25T00:30:26.737742+0000","last_fullsized":"2024-04-25T00:30:26.737742+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:30:36.682736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"74'4936","reported_seq":5056,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.737754+0000","last_change":"2024-04-25T00:01:19.514615+0000","last_active":"2024-04-25T00:30:26.737754+0000","last_peered":"2024-04-25T00:30:26.737754+0000","last_clean":"2024-04-25T00:30:26.737754+0000","last_became_active":"2024-04-25T00:01:19.513640+0000","last_became_peered":"2024-04-25T00:01:19.513640+0000","last_unstale":"2024-04-25T00:30:26.737754+0000","last_undegraded":"2024-04-25T00:30:26.737754+0000","last_fullsized":"2024-04-25T00:30:26.737754+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4936,"log_dups_size":0,"ondisk_log_size":4936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:08.620602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":27403,"num_write_kb":306834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"70'4466","reported_seq":4593,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.793110+0000","last_change":"2024-04-25T00:01:05.385670+0000","last_active":"2024-04-25T00:30:15.793110+0000","last_peered":"2024-04-25T00:30:15.793110+0000","last_clean":"2024-04-25T00:30:15.793110+0000","last_became_active":"2024-04-25T00:01:05.384822+0000","last_became_peered":"2024-04-25T00:01:05.384822+0000","last_unstale":"2024-04-25T00:30:15.793110+0000","last_undegraded":"2024-04-25T00:30:15.793110+0000","last_fullsized":"2024-04-25T00:30:15.793110+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4466,"log_dups_size":0,"ondisk_log_size":4466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:24.017263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23831,"num_write_kb":316890,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"70'6","reported_seq":158,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.798998+0000","last_change":"2024-04-25T00:01:04.356627+0000","last_active":"2024-04-25T00:30:15.798998+0000","last_peered":"2024-04-25T00:30:15.798998+0000","last_clean":"2024-04-25T00:30:15.798998+0000","last_became_active":"2024-04-25T00:01:04.354419+0000","last_became_peered":"2024-04-25T00:01:04.354419+0000","last_unstale":"2024-04-25T00:30:15.798998+0000","last_undegraded":"2024-04-25T00:30:15.798998+0000","last_fullsized":"2024-04-25T00:30:15.798998+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:26.243635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'1453","reported_seq":1626,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.754238+0000","last_change":"2024-04-25T00:01:18.490511+0000","last_active":"2024-04-25T00:30:26.754238+0000","last_peered":"2024-04-25T00:30:26.754238+0000","last_clean":"2024-04-25T00:30:26.754238+0000","last_became_active":"2024-04-25T00:01:18.489629+0000","last_became_peered":"2024-04-25T00:01:18.489629+0000","last_unstale":"2024-04-25T00:30:26.754238+0000","last_undegraded":"2024-04-25T00:30:26.754238+0000","last_fullsized":"2024-04-25T00:30:26.754238+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":1453,"log_dups_size":0,"ondisk_log_size":1453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:07.610846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":75,"num_read_kb":41971,"num_write":1427,"num_write_kb":4503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'3796","reported_seq":3915,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.602587+0000","last_change":"2024-04-25T00:01:19.508931+0000","last_active":"2024-04-25T00:30:26.602587+0000","last_peered":"2024-04-25T00:30:26.602587+0000","last_clean":"2024-04-25T00:30:26.602587+0000","last_became_active":"2024-04-25T00:01:19.507867+0000","last_became_peered":"2024-04-25T00:01:19.507867+0000","last_unstale":"2024-04-25T00:30:26.602587+0000","last_undegraded":"2024-04-25T00:30:26.602587+0000","last_fullsized":"2024-04-25T00:30:26.602587+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3796,"log_dups_size":0,"ondisk_log_size":3796,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:38.152376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20867,"num_write_kb":276131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"70'5324","reported_seq":5452,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.792960+0000","last_change":"2024-04-25T00:01:05.378840+0000","last_active":"2024-04-25T00:30:15.792960+0000","last_peered":"2024-04-25T00:30:15.792960+0000","last_clean":"2024-04-25T00:30:15.792960+0000","last_became_active":"2024-04-25T00:01:05.378112+0000","last_became_peered":"2024-04-25T00:01:05.378112+0000","last_unstale":"2024-04-25T00:30:15.792960+0000","last_undegraded":"2024-04-25T00:30:15.792960+0000","last_fullsized":"2024-04-25T00:30:15.792960+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":5324,"log_dups_size":0,"ondisk_log_size":5324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:43.615895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":28190,"num_write_kb":387378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"69'7","reported_seq":182,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.795644+0000","last_change":"2024-04-25T00:01:04.370794+0000","last_active":"2024-04-25T00:30:15.795644+0000","last_peered":"2024-04-25T00:30:15.795644+0000","last_clean":"2024-04-25T00:30:15.795644+0000","last_became_active":"2024-04-25T00:01:04.370099+0000","last_became_peered":"2024-04-25T00:01:04.370099+0000","last_unstale":"2024-04-25T00:30:15.795644+0000","last_undegraded":"2024-04-25T00:30:15.795644+0000","last_fullsized":"2024-04-25T00:30:15.795644+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:42:03.181036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"74'5","reported_seq":126,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.138150+0000","last_change":"2024-04-25T00:01:18.475256+0000","last_active":"2024-04-25T00:30:26.138150+0000","last_peered":"2024-04-25T00:30:26.138150+0000","last_clean":"2024-04-25T00:30:26.138150+0000","last_became_active":"2024-04-25T00:01:18.474952+0000","last_became_peered":"2024-04-25T00:01:18.474952+0000","last_unstale":"2024-04-25T00:30:26.138150+0000","last_undegraded":"2024-04-25T00:30:26.138150+0000","last_fullsized":"2024-04-25T00:30:26.138150+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:46.849870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"74'4048","reported_seq":4167,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.238601+0000","last_change":"2024-04-25T00:01:19.516833+0000","last_active":"2024-04-25T00:30:26.238601+0000","last_peered":"2024-04-25T00:30:26.238601+0000","last_clean":"2024-04-25T00:30:26.238601+0000","last_became_active":"2024-04-25T00:01:19.515926+0000","last_became_peered":"2024-04-25T00:01:19.515926+0000","last_unstale":"2024-04-25T00:30:26.238601+0000","last_undegraded":"2024-04-25T00:30:26.238601+0000","last_fullsized":"2024-04-25T00:30:26.238601+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4048,"log_dups_size":0,"ondisk_log_size":4048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:23:48.855381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23468,"num_write_kb":270462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"70'3273","reported_seq":3401,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.808120+0000","last_change":"2024-04-25T00:01:05.372591+0000","last_active":"2024-04-25T00:30:15.808120+0000","last_peered":"2024-04-25T00:30:15.808120+0000","last_clean":"2024-04-25T00:30:15.808120+0000","last_became_active":"2024-04-25T00:01:05.371204+0000","last_became_peered":"2024-04-25T00:01:05.371204+0000","last_unstale":"2024-04-25T00:30:15.808120+0000","last_undegraded":"2024-04-25T00:30:15.808120+0000","last_fullsized":"2024-04-25T00:30:15.808120+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3273,"log_dups_size":0,"ondisk_log_size":3273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:15.879673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":17647,"num_write_kb":246988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"70'4","reported_seq":156,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.793404+0000","last_change":"2024-04-25T00:01:04.372376+0000","last_active":"2024-04-25T00:30:15.793404+0000","last_peered":"2024-04-25T00:30:15.793404+0000","last_clean":"2024-04-25T00:30:15.793404+0000","last_became_active":"2024-04-25T00:01:04.372040+0000","last_became_peered":"2024-04-25T00:01:04.372040+0000","last_unstale":"2024-04-25T00:30:15.793404+0000","last_undegraded":"2024-04-25T00:30:15.793404+0000","last_fullsized":"2024-04-25T00:30:15.793404+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:21.729893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"74'2","reported_seq":123,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.736656+0000","last_change":"2024-04-25T00:01:18.487548+0000","last_active":"2024-04-25T00:30:26.736656+0000","last_peered":"2024-04-25T00:30:26.736656+0000","last_clean":"2024-04-25T00:30:26.736656+0000","last_became_active":"2024-04-25T00:01:18.487227+0000","last_became_peered":"2024-04-25T00:01:18.487227+0000","last_unstale":"2024-04-25T00:30:26.736656+0000","last_undegraded":"2024-04-25T00:30:26.736656+0000","last_fullsized":"2024-04-25T00:30:26.736656+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:55.074018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"74'4306","reported_seq":4427,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.642530+0000","last_change":"2024-04-25T00:01:19.528574+0000","last_active":"2024-04-25T00:30:26.642530+0000","last_peered":"2024-04-25T00:30:26.642530+0000","last_clean":"2024-04-25T00:30:26.642530+0000","last_became_active":"2024-04-25T00:01:19.525531+0000","last_became_peered":"2024-04-25T00:01:19.525531+0000","last_unstale":"2024-04-25T00:30:26.642530+0000","last_undegraded":"2024-04-25T00:30:26.642530+0000","last_fullsized":"2024-04-25T00:30:26.642530+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4306,"log_dups_size":0,"ondisk_log_size":4306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:47:08.132895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":31,"num_write":22571,"num_write_kb":279217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"70'3794","reported_seq":3920,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.804529+0000","last_change":"2024-04-25T00:01:05.364103+0000","last_active":"2024-04-25T00:30:15.804529+0000","last_peered":"2024-04-25T00:30:15.804529+0000","last_clean":"2024-04-25T00:30:15.804529+0000","last_became_active":"2024-04-25T00:01:05.363320+0000","last_became_peered":"2024-04-25T00:01:05.363320+0000","last_unstale":"2024-04-25T00:30:15.804529+0000","last_undegraded":"2024-04-25T00:30:15.804529+0000","last_fullsized":"2024-04-25T00:30:15.804529+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3794,"log_dups_size":0,"ondisk_log_size":3794,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:31.456792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20831,"num_write_kb":276143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"74'15","reported_seq":257,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.791248+0000","last_change":"2024-04-25T00:01:04.358365+0000","last_active":"2024-04-25T00:30:15.791248+0000","last_peered":"2024-04-25T00:30:15.791248+0000","last_clean":"2024-04-25T00:30:15.791248+0000","last_became_active":"2024-04-25T00:01:04.357571+0000","last_became_peered":"2024-04-25T00:01:04.357571+0000","last_unstale":"2024-04-25T00:30:15.791248+0000","last_undegraded":"2024-04-25T00:30:15.791248+0000","last_fullsized":"2024-04-25T00:30:15.791248+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:50:23.023114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":342,"num_read_kb":342,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"74'3","reported_seq":123,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.747647+0000","last_change":"2024-04-25T00:01:18.497679+0000","last_active":"2024-04-25T00:30:26.747647+0000","last_peered":"2024-04-25T00:30:26.747647+0000","last_clean":"2024-04-25T00:30:26.747647+0000","last_became_active":"2024-04-25T00:01:18.496489+0000","last_became_peered":"2024-04-25T00:01:18.496489+0000","last_unstale":"2024-04-25T00:30:26.747647+0000","last_undegraded":"2024-04-25T00:30:26.747647+0000","last_fullsized":"2024-04-25T00:30:26.747647+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:23:47.845624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"74'4849","reported_seq":4970,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.756589+0000","last_change":"2024-04-25T00:01:19.498082+0000","last_active":"2024-04-25T00:30:26.756589+0000","last_peered":"2024-04-25T00:30:26.756589+0000","last_clean":"2024-04-25T00:30:26.756589+0000","last_became_active":"2024-04-25T00:01:19.496805+0000","last_became_peered":"2024-04-25T00:01:19.496805+0000","last_unstale":"2024-04-25T00:30:26.756589+0000","last_undegraded":"2024-04-25T00:30:26.756589+0000","last_fullsized":"2024-04-25T00:30:26.756589+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4849,"log_dups_size":0,"ondisk_log_size":4849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:57:36.256250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":16,"num_write":26442,"num_write_kb":335585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"70'2974","reported_seq":3100,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.941797+0000","last_change":"2024-04-25T00:01:05.373927+0000","last_active":"2024-04-25T00:30:15.941797+0000","last_peered":"2024-04-25T00:30:15.941797+0000","last_clean":"2024-04-25T00:30:15.941797+0000","last_became_active":"2024-04-25T00:01:05.373077+0000","last_became_peered":"2024-04-25T00:01:05.373077+0000","last_unstale":"2024-04-25T00:30:15.941797+0000","last_undegraded":"2024-04-25T00:30:15.941797+0000","last_fullsized":"2024-04-25T00:30:15.941797+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":2974,"log_dups_size":0,"ondisk_log_size":2974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:50.489125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15895,"num_write_kb":231645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"70'5","reported_seq":133,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.791512+0000","last_change":"2024-04-25T00:01:04.357265+0000","last_active":"2024-04-25T00:30:15.791512+0000","last_peered":"2024-04-25T00:30:15.791512+0000","last_clean":"2024-04-25T00:30:15.791512+0000","last_became_active":"2024-04-25T00:01:04.357004+0000","last_became_peered":"2024-04-25T00:01:04.357004+0000","last_unstale":"2024-04-25T00:30:15.791512+0000","last_undegraded":"2024-04-25T00:30:15.791512+0000","last_fullsized":"2024-04-25T00:30:15.791512+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:47:25.595275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"74'5","reported_seq":125,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.707490+0000","last_change":"2024-04-25T00:01:18.487276+0000","last_active":"2024-04-25T00:30:26.707490+0000","last_peered":"2024-04-25T00:30:26.707490+0000","last_clean":"2024-04-25T00:30:26.707490+0000","last_became_active":"2024-04-25T00:01:18.486279+0000","last_became_peered":"2024-04-25T00:01:18.486279+0000","last_unstale":"2024-04-25T00:30:26.707490+0000","last_undegraded":"2024-04-25T00:30:26.707490+0000","last_fullsized":"2024-04-25T00:30:26.707490+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:17:37.141843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"74'4040","reported_seq":4158,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.630478+0000","last_change":"2024-04-25T00:01:19.516827+0000","last_active":"2024-04-25T00:30:26.630478+0000","last_peered":"2024-04-25T00:30:26.630478+0000","last_clean":"2024-04-25T00:30:26.630478+0000","last_became_active":"2024-04-25T00:01:19.515765+0000","last_became_peered":"2024-04-25T00:01:19.515765+0000","last_unstale":"2024-04-25T00:30:26.630478+0000","last_undegraded":"2024-04-25T00:30:26.630478+0000","last_fullsized":"2024-04-25T00:30:26.630478+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4040,"log_dups_size":0,"ondisk_log_size":4040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:13:34.153117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23288,"num_write_kb":269674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"70'4514","reported_seq":4640,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.806193+0000","last_change":"2024-04-25T00:01:05.379501+0000","last_active":"2024-04-25T00:30:15.806193+0000","last_peered":"2024-04-25T00:30:15.806193+0000","last_clean":"2024-04-25T00:30:15.806193+0000","last_became_active":"2024-04-25T00:01:05.378644+0000","last_became_peered":"2024-04-25T00:01:05.378644+0000","last_unstale":"2024-04-25T00:30:15.806193+0000","last_undegraded":"2024-04-25T00:30:15.806193+0000","last_fullsized":"2024-04-25T00:30:15.806193+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4514,"log_dups_size":0,"ondisk_log_size":4514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:08:35.505154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25321,"num_write_kb":326961,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"70'3","reported_seq":131,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.809820+0000","last_change":"2024-04-25T00:01:04.360987+0000","last_active":"2024-04-25T00:30:15.809820+0000","last_peered":"2024-04-25T00:30:15.809820+0000","last_clean":"2024-04-25T00:30:15.809820+0000","last_became_active":"2024-04-25T00:01:04.360664+0000","last_became_peered":"2024-04-25T00:01:04.360664+0000","last_unstale":"2024-04-25T00:30:15.809820+0000","last_undegraded":"2024-04-25T00:30:15.809820+0000","last_fullsized":"2024-04-25T00:30:15.809820+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:55.283076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"74'334","reported_seq":571,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:19.197817+0000","last_change":"2024-04-25T00:01:18.505577+0000","last_active":"2024-04-25T00:30:19.197817+0000","last_peered":"2024-04-25T00:30:19.197817+0000","last_clean":"2024-04-25T00:30:19.197817+0000","last_became_active":"2024-04-25T00:01:18.504963+0000","last_became_peered":"2024-04-25T00:01:18.504963+0000","last_unstale":"2024-04-25T00:30:19.197817+0000","last_undegraded":"2024-04-25T00:30:19.197817+0000","last_fullsized":"2024-04-25T00:30:19.197817+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":334,"log_dups_size":0,"ondisk_log_size":334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:37.142619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":209,"num_read_kb":90302,"num_write":330,"num_write_kb":4274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"74'4674","reported_seq":4789,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.733748+0000","last_change":"2024-04-25T00:01:46.658442+0000","last_active":"2024-04-25T00:30:26.733748+0000","last_peered":"2024-04-25T00:30:26.733748+0000","last_clean":"2024-04-25T00:30:26.733748+0000","last_became_active":"2024-04-25T00:01:46.657588+0000","last_became_peered":"2024-04-25T00:01:46.657588+0000","last_unstale":"2024-04-25T00:30:26.733748+0000","last_undegraded":"2024-04-25T00:30:26.733748+0000","last_fullsized":"2024-04-25T00:30:26.733748+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4674,"log_dups_size":0,"ondisk_log_size":4674,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:58:49.738233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":16,"num_write":25396,"num_write_kb":335454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"70'3777","reported_seq":3903,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.810481+0000","last_change":"2024-04-25T00:01:05.379102+0000","last_active":"2024-04-25T00:30:15.810481+0000","last_peered":"2024-04-25T00:30:15.810481+0000","last_clean":"2024-04-25T00:30:15.810481+0000","last_became_active":"2024-04-25T00:01:05.377625+0000","last_became_peered":"2024-04-25T00:01:05.377625+0000","last_unstale":"2024-04-25T00:30:15.810481+0000","last_undegraded":"2024-04-25T00:30:15.810481+0000","last_fullsized":"2024-04-25T00:30:15.810481+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3777,"log_dups_size":0,"ondisk_log_size":3777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:15.879673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20730,"num_write_kb":283926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"73'9","reported_seq":183,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.803482+0000","last_change":"2024-04-25T00:01:04.370640+0000","last_active":"2024-04-25T00:30:15.803482+0000","last_peered":"2024-04-25T00:30:15.803482+0000","last_clean":"2024-04-25T00:30:15.803482+0000","last_became_active":"2024-04-25T00:01:04.370031+0000","last_became_peered":"2024-04-25T00:01:04.370031+0000","last_unstale":"2024-04-25T00:30:15.803482+0000","last_undegraded":"2024-04-25T00:30:15.803482+0000","last_fullsized":"2024-04-25T00:30:15.803482+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:26.243635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":184,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"74'31","reported_seq":220,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.781231+0000","last_change":"2024-04-25T00:01:18.473113+0000","last_active":"2024-04-25T00:30:26.781231+0000","last_peered":"2024-04-25T00:30:26.781231+0000","last_clean":"2024-04-25T00:30:26.781231+0000","last_became_active":"2024-04-25T00:01:18.472931+0000","last_became_peered":"2024-04-25T00:01:18.472931+0000","last_unstale":"2024-04-25T00:30:26.781231+0000","last_undegraded":"2024-04-25T00:30:26.781231+0000","last_fullsized":"2024-04-25T00:30:26.781231+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:07.498985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":161,"num_read_kb":161,"num_write":32,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"74'5042","reported_seq":5161,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.730299+0000","last_change":"2024-04-25T00:01:19.533664+0000","last_active":"2024-04-25T00:30:26.730299+0000","last_peered":"2024-04-25T00:30:26.730299+0000","last_clean":"2024-04-25T00:30:26.730299+0000","last_became_active":"2024-04-25T00:01:19.533048+0000","last_became_peered":"2024-04-25T00:01:19.533048+0000","last_unstale":"2024-04-25T00:30:26.730299+0000","last_undegraded":"2024-04-25T00:30:26.730299+0000","last_fullsized":"2024-04-25T00:30:26.730299+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":5042,"log_dups_size":0,"ondisk_log_size":5042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:45:43.141569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":28058,"num_write_kb":350341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"69'3549","reported_seq":3675,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.790799+0000","last_change":"2024-04-25T00:01:05.385827+0000","last_active":"2024-04-25T00:30:15.790799+0000","last_peered":"2024-04-25T00:30:15.790799+0000","last_clean":"2024-04-25T00:30:15.790799+0000","last_became_active":"2024-04-25T00:01:05.385258+0000","last_became_peered":"2024-04-25T00:01:05.385258+0000","last_unstale":"2024-04-25T00:30:15.790799+0000","last_undegraded":"2024-04-25T00:30:15.790799+0000","last_fullsized":"2024-04-25T00:30:15.790799+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3549,"log_dups_size":0,"ondisk_log_size":3549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:23.870121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19806,"num_write_kb":251506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"74'341","reported_seq":585,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:22.899825+0000","last_change":"2024-04-25T00:01:04.365185+0000","last_active":"2024-04-25T00:30:22.899825+0000","last_peered":"2024-04-25T00:30:22.899825+0000","last_clean":"2024-04-25T00:30:22.899825+0000","last_became_active":"2024-04-25T00:01:04.364930+0000","last_became_peered":"2024-04-25T00:01:04.364930+0000","last_unstale":"2024-04-25T00:30:22.899825+0000","last_undegraded":"2024-04-25T00:30:22.899825+0000","last_fullsized":"2024-04-25T00:30:22.899825+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":341,"log_dups_size":0,"ondisk_log_size":341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:45:03.618340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":208,"num_read_kb":94392,"num_write":336,"num_write_kb":4266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"74'3","reported_seq":123,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.747231+0000","last_change":"2024-04-25T00:01:18.496220+0000","last_active":"2024-04-25T00:30:26.747231+0000","last_peered":"2024-04-25T00:30:26.747231+0000","last_clean":"2024-04-25T00:30:26.747231+0000","last_became_active":"2024-04-25T00:01:18.492850+0000","last_became_peered":"2024-04-25T00:01:18.492850+0000","last_unstale":"2024-04-25T00:30:26.747231+0000","last_undegraded":"2024-04-25T00:30:26.747231+0000","last_fullsized":"2024-04-25T00:30:26.747231+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:50:16.922314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"74'4133","reported_seq":4252,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.680551+0000","last_change":"2024-04-25T00:01:19.515725+0000","last_active":"2024-04-25T00:30:26.680551+0000","last_peered":"2024-04-25T00:30:26.680551+0000","last_clean":"2024-04-25T00:30:26.680551+0000","last_became_active":"2024-04-25T00:01:19.509285+0000","last_became_peered":"2024-04-25T00:01:19.509285+0000","last_unstale":"2024-04-25T00:30:26.680551+0000","last_undegraded":"2024-04-25T00:30:26.680551+0000","last_fullsized":"2024-04-25T00:30:26.680551+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4133,"log_dups_size":0,"ondisk_log_size":4133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:25:20.185355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":4,"num_write":22847,"num_write_kb":280058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"70'5270","reported_seq":5396,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.809540+0000","last_change":"2024-04-25T00:01:05.371061+0000","last_active":"2024-04-25T00:30:15.809540+0000","last_peered":"2024-04-25T00:30:15.809540+0000","last_clean":"2024-04-25T00:30:15.809540+0000","last_became_active":"2024-04-25T00:01:05.369950+0000","last_became_peered":"2024-04-25T00:01:05.369950+0000","last_unstale":"2024-04-25T00:30:15.809540+0000","last_undegraded":"2024-04-25T00:30:15.809540+0000","last_fullsized":"2024-04-25T00:30:15.809540+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":5270,"log_dups_size":0,"ondisk_log_size":5270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:02:43.237902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28224,"num_write_kb":363198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"74'31","reported_seq":205,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:25.818373+0000","last_change":"2024-04-25T00:01:04.366503+0000","last_active":"2024-04-25T00:30:25.818373+0000","last_peered":"2024-04-25T00:30:25.818373+0000","last_clean":"2024-04-25T00:30:25.818373+0000","last_became_active":"2024-04-25T00:01:04.365945+0000","last_became_peered":"2024-04-25T00:01:04.365945+0000","last_unstale":"2024-04-25T00:30:25.818373+0000","last_undegraded":"2024-04-25T00:30:25.818373+0000","last_fullsized":"2024-04-25T00:30:25.818373+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":92,"num_write":32,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"74'8","reported_seq":174,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.742489+0000","last_change":"2024-04-25T00:01:18.476456+0000","last_active":"2024-04-25T00:30:26.742489+0000","last_peered":"2024-04-25T00:30:26.742489+0000","last_clean":"2024-04-25T00:30:26.742489+0000","last_became_active":"2024-04-25T00:01:18.475876+0000","last_became_peered":"2024-04-25T00:01:18.475876+0000","last_unstale":"2024-04-25T00:30:26.742489+0000","last_undegraded":"2024-04-25T00:30:26.742489+0000","last_fullsized":"2024-04-25T00:30:26.742489+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:56.741250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":184,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"74'4233","reported_seq":4354,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.727907+0000","last_change":"2024-04-25T00:01:19.515376+0000","last_active":"2024-04-25T00:30:26.727907+0000","last_peered":"2024-04-25T00:30:26.727907+0000","last_clean":"2024-04-25T00:30:26.727907+0000","last_became_active":"2024-04-25T00:01:19.512425+0000","last_became_peered":"2024-04-25T00:01:19.512425+0000","last_unstale":"2024-04-25T00:30:26.727907+0000","last_undegraded":"2024-04-25T00:30:26.727907+0000","last_fullsized":"2024-04-25T00:30:26.727907+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4233,"log_dups_size":0,"ondisk_log_size":4233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:43:16.957273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":24291,"num_write_kb":290168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"70'4413","reported_seq":4539,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.800988+0000","last_change":"2024-04-25T00:01:05.361723+0000","last_active":"2024-04-25T00:30:15.800988+0000","last_peered":"2024-04-25T00:30:15.800988+0000","last_clean":"2024-04-25T00:30:15.800988+0000","last_became_active":"2024-04-25T00:01:05.360956+0000","last_became_peered":"2024-04-25T00:01:05.360956+0000","last_unstale":"2024-04-25T00:30:15.800988+0000","last_undegraded":"2024-04-25T00:30:15.800988+0000","last_fullsized":"2024-04-25T00:30:15.800988+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4413,"log_dups_size":0,"ondisk_log_size":4413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:23.870121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22903,"num_write_kb":321461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"70'3","reported_seq":131,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.790859+0000","last_change":"2024-04-25T00:01:04.362129+0000","last_active":"2024-04-25T00:30:15.790859+0000","last_peered":"2024-04-25T00:30:15.790859+0000","last_clean":"2024-04-25T00:30:15.790859+0000","last_became_active":"2024-04-25T00:01:04.361827+0000","last_became_peered":"2024-04-25T00:01:04.361827+0000","last_unstale":"2024-04-25T00:30:15.790859+0000","last_undegraded":"2024-04-25T00:30:15.790859+0000","last_fullsized":"2024-04-25T00:30:15.790859+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:26.243635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"74'15","reported_seq":253,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.739696+0000","last_change":"2024-04-25T00:01:18.496129+0000","last_active":"2024-04-25T00:30:26.739696+0000","last_peered":"2024-04-25T00:30:26.739696+0000","last_clean":"2024-04-25T00:30:26.739696+0000","last_became_active":"2024-04-25T00:01:18.495415+0000","last_became_peered":"2024-04-25T00:01:18.495415+0000","last_unstale":"2024-04-25T00:30:26.739696+0000","last_undegraded":"2024-04-25T00:30:26.739696+0000","last_fullsized":"2024-04-25T00:30:26.739696+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:38.089411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":354,"num_read_kb":354,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"74'3474","reported_seq":3592,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.668678+0000","last_change":"2024-04-25T00:01:19.509542+0000","last_active":"2024-04-25T00:30:26.668678+0000","last_peered":"2024-04-25T00:30:26.668678+0000","last_clean":"2024-04-25T00:30:26.668678+0000","last_became_active":"2024-04-25T00:01:19.509015+0000","last_became_peered":"2024-04-25T00:01:19.509015+0000","last_unstale":"2024-04-25T00:30:26.668678+0000","last_undegraded":"2024-04-25T00:30:26.668678+0000","last_fullsized":"2024-04-25T00:30:26.668678+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3474,"log_dups_size":0,"ondisk_log_size":3474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:25:20.185355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19390,"num_write_kb":245789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"70'4331","reported_seq":4457,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.806796+0000","last_change":"2024-04-25T00:01:05.365413+0000","last_active":"2024-04-25T00:30:15.806796+0000","last_peered":"2024-04-25T00:30:15.806796+0000","last_clean":"2024-04-25T00:30:15.806796+0000","last_became_active":"2024-04-25T00:01:05.361865+0000","last_became_peered":"2024-04-25T00:01:05.361865+0000","last_unstale":"2024-04-25T00:30:15.806796+0000","last_undegraded":"2024-04-25T00:30:15.806796+0000","last_fullsized":"2024-04-25T00:30:15.806796+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4331,"log_dups_size":0,"ondisk_log_size":4331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:30.228060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22828,"num_write_kb":312693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"74'7","reported_seq":135,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.944255+0000","last_change":"2024-04-25T00:01:04.360804+0000","last_active":"2024-04-25T00:30:15.944255+0000","last_peered":"2024-04-25T00:30:15.944255+0000","last_clean":"2024-04-25T00:30:15.944255+0000","last_became_active":"2024-04-25T00:01:04.360374+0000","last_became_peered":"2024-04-25T00:01:04.360374+0000","last_unstale":"2024-04-25T00:30:15.944255+0000","last_undegraded":"2024-04-25T00:30:15.944255+0000","last_fullsized":"2024-04-25T00:30:15.944255+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"74'5","reported_seq":125,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.739206+0000","last_change":"2024-04-25T00:01:18.480003+0000","last_active":"2024-04-25T00:30:26.739206+0000","last_peered":"2024-04-25T00:30:26.739206+0000","last_clean":"2024-04-25T00:30:26.739206+0000","last_became_active":"2024-04-25T00:01:18.479270+0000","last_became_peered":"2024-04-25T00:01:18.479270+0000","last_unstale":"2024-04-25T00:30:26.739206+0000","last_undegraded":"2024-04-25T00:30:26.739206+0000","last_fullsized":"2024-04-25T00:30:26.739206+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:52:40.942502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"74'5681","reported_seq":5805,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.778355+0000","last_change":"2024-04-25T00:01:19.524463+0000","last_active":"2024-04-25T00:30:26.778355+0000","last_peered":"2024-04-25T00:30:26.778355+0000","last_clean":"2024-04-25T00:30:26.778355+0000","last_became_active":"2024-04-25T00:01:19.524166+0000","last_became_peered":"2024-04-25T00:01:19.524166+0000","last_unstale":"2024-04-25T00:30:26.778355+0000","last_undegraded":"2024-04-25T00:30:26.778355+0000","last_fullsized":"2024-04-25T00:30:26.778355+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":5681,"log_dups_size":0,"ondisk_log_size":5681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:50.351599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":40,"num_write":31879,"num_write_kb":410681,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"74'28","reported_seq":150,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.738030+0000","last_change":"2024-04-25T00:01:18.497096+0000","last_active":"2024-04-25T00:30:26.738030+0000","last_peered":"2024-04-25T00:30:26.738030+0000","last_clean":"2024-04-25T00:30:26.738030+0000","last_became_active":"2024-04-25T00:01:18.496289+0000","last_became_peered":"2024-04-25T00:01:18.496289+0000","last_unstale":"2024-04-25T00:30:26.738030+0000","last_undegraded":"2024-04-25T00:30:26.738030+0000","last_fullsized":"2024-04-25T00:30:26.738030+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:59:40.316831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"70'4773","reported_seq":4900,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.791447+0000","last_change":"2024-04-25T00:01:05.370043+0000","last_active":"2024-04-25T00:30:15.791447+0000","last_peered":"2024-04-25T00:30:15.791447+0000","last_clean":"2024-04-25T00:30:15.791447+0000","last_became_active":"2024-04-25T00:01:05.368080+0000","last_became_peered":"2024-04-25T00:01:05.368080+0000","last_unstale":"2024-04-25T00:30:15.791447+0000","last_undegraded":"2024-04-25T00:30:15.791447+0000","last_fullsized":"2024-04-25T00:30:15.791447+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4773,"log_dups_size":0,"ondisk_log_size":4773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:58:07.610923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":25218,"num_write_kb":330520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"69'2","reported_seq":130,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.804702+0000","last_change":"2024-04-25T00:01:04.364523+0000","last_active":"2024-04-25T00:30:15.804702+0000","last_peered":"2024-04-25T00:30:15.804702+0000","last_clean":"2024-04-25T00:30:15.804702+0000","last_became_active":"2024-04-25T00:01:04.364050+0000","last_became_peered":"2024-04-25T00:01:04.364050+0000","last_unstale":"2024-04-25T00:30:15.804702+0000","last_undegraded":"2024-04-25T00:30:15.804702+0000","last_fullsized":"2024-04-25T00:30:15.804702+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:49.479804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"74'3390","reported_seq":3510,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.735796+0000","last_change":"2024-04-25T00:01:19.491523+0000","last_active":"2024-04-25T00:30:26.735796+0000","last_peered":"2024-04-25T00:30:26.735796+0000","last_clean":"2024-04-25T00:30:26.735796+0000","last_became_active":"2024-04-25T00:01:19.490755+0000","last_became_peered":"2024-04-25T00:01:19.490755+0000","last_unstale":"2024-04-25T00:30:26.735796+0000","last_undegraded":"2024-04-25T00:30:26.735796+0000","last_fullsized":"2024-04-25T00:30:26.735796+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3390,"log_dups_size":0,"ondisk_log_size":3390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:47:19.526845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18979,"num_write_kb":248305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"74'1082","reported_seq":1259,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.777825+0000","last_change":"2024-04-25T00:01:18.472794+0000","last_active":"2024-04-25T00:30:26.777825+0000","last_peered":"2024-04-25T00:30:26.777825+0000","last_clean":"2024-04-25T00:30:26.777825+0000","last_became_active":"2024-04-25T00:01:18.472534+0000","last_became_peered":"2024-04-25T00:01:18.472534+0000","last_unstale":"2024-04-25T00:30:26.777825+0000","last_undegraded":"2024-04-25T00:30:26.777825+0000","last_fullsized":"2024-04-25T00:30:26.777825+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":1082,"log_dups_size":0,"ondisk_log_size":1082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:55:45.843808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":101,"num_read_kb":11812,"num_write":1055,"num_write_kb":4728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"70'4239","reported_seq":4365,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.794924+0000","last_change":"2024-04-25T00:01:05.378867+0000","last_active":"2024-04-25T00:30:15.794924+0000","last_peered":"2024-04-25T00:30:15.794924+0000","last_clean":"2024-04-25T00:30:15.794924+0000","last_became_active":"2024-04-25T00:01:05.378056+0000","last_became_peered":"2024-04-25T00:01:05.378056+0000","last_unstale":"2024-04-25T00:30:15.794924+0000","last_undegraded":"2024-04-25T00:30:15.794924+0000","last_fullsized":"2024-04-25T00:30:15.794924+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4239,"log_dups_size":0,"ondisk_log_size":4239,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:36:10.377757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22560,"num_write_kb":311538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"70'3","reported_seq":131,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.809380+0000","last_change":"2024-04-25T00:01:04.370896+0000","last_active":"2024-04-25T00:30:15.809380+0000","last_peered":"2024-04-25T00:30:15.809380+0000","last_clean":"2024-04-25T00:30:15.809380+0000","last_became_active":"2024-04-25T00:01:04.370273+0000","last_became_peered":"2024-04-25T00:01:04.370273+0000","last_unstale":"2024-04-25T00:30:15.809380+0000","last_undegraded":"2024-04-25T00:30:15.809380+0000","last_fullsized":"2024-04-25T00:30:15.809380+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:49:22.673421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"74'5148","reported_seq":5266,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.723815+0000","last_change":"2024-04-25T00:01:19.521143+0000","last_active":"2024-04-25T00:30:26.723815+0000","last_peered":"2024-04-25T00:30:26.723815+0000","last_clean":"2024-04-25T00:30:26.723815+0000","last_became_active":"2024-04-25T00:01:19.514664+0000","last_became_peered":"2024-04-25T00:01:19.514664+0000","last_unstale":"2024-04-25T00:30:26.723815+0000","last_undegraded":"2024-04-25T00:30:26.723815+0000","last_fullsized":"2024-04-25T00:30:26.723815+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":5148,"log_dups_size":0,"ondisk_log_size":5148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:03:47.440442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":29131,"num_write_kb":338102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"74'249","reported_seq":380,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:27.894626+0000","last_change":"2024-04-25T00:01:04.360254+0000","last_active":"2024-04-25T00:30:27.894626+0000","last_peered":"2024-04-25T00:30:27.894626+0000","last_clean":"2024-04-25T00:30:27.894626+0000","last_became_active":"2024-04-25T00:01:04.359586+0000","last_became_peered":"2024-04-25T00:01:04.359586+0000","last_unstale":"2024-04-25T00:30:27.894626+0000","last_undegraded":"2024-04-25T00:30:27.894626+0000","last_fullsized":"2024-04-25T00:30:27.894626+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:47.192930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2250578,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4308,"num_write":220,"num_write_kb":2323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"69'3397","reported_seq":3522,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.792637+0000","last_change":"2024-04-25T00:01:05.370723+0000","last_active":"2024-04-25T00:30:15.792637+0000","last_peered":"2024-04-25T00:30:15.792637+0000","last_clean":"2024-04-25T00:30:15.792637+0000","last_became_active":"2024-04-25T00:01:05.368970+0000","last_became_peered":"2024-04-25T00:01:05.368970+0000","last_unstale":"2024-04-25T00:30:15.792637+0000","last_undegraded":"2024-04-25T00:30:15.792637+0000","last_fullsized":"2024-04-25T00:30:15.792637+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3397,"log_dups_size":0,"ondisk_log_size":3397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:06:53.743074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17978,"num_write_kb":244099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"74'5","reported_seq":170,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:19.198496+0000","last_change":"2024-04-25T00:01:18.498570+0000","last_active":"2024-04-25T00:30:19.198496+0000","last_peered":"2024-04-25T00:30:19.198496+0000","last_clean":"2024-04-25T00:30:19.198496+0000","last_became_active":"2024-04-25T00:01:18.497745+0000","last_became_peered":"2024-04-25T00:01:18.497745+0000","last_unstale":"2024-04-25T00:30:19.198496+0000","last_undegraded":"2024-04-25T00:30:19.198496+0000","last_fullsized":"2024-04-25T00:30:19.198496+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:26:42.047149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"74'3919","reported_seq":4039,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.527005+0000","last_change":"2024-04-25T00:01:19.514919+0000","last_active":"2024-04-25T00:30:26.527005+0000","last_peered":"2024-04-25T00:30:26.527005+0000","last_clean":"2024-04-25T00:30:26.527005+0000","last_became_active":"2024-04-25T00:01:19.514280+0000","last_became_peered":"2024-04-25T00:01:19.514280+0000","last_unstale":"2024-04-25T00:30:26.527005+0000","last_undegraded":"2024-04-25T00:30:26.527005+0000","last_fullsized":"2024-04-25T00:30:26.527005+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3919,"log_dups_size":0,"ondisk_log_size":3919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:37.975265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":20874,"num_write_kb":280510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"69'752","reported_seq":907,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.801549+0000","last_change":"2024-04-25T00:01:04.365607+0000","last_active":"2024-04-25T00:30:15.801549+0000","last_peered":"2024-04-25T00:30:15.801549+0000","last_clean":"2024-04-25T00:30:15.801549+0000","last_became_active":"2024-04-25T00:01:04.364788+0000","last_became_peered":"2024-04-25T00:01:04.364788+0000","last_unstale":"2024-04-25T00:30:15.801549+0000","last_undegraded":"2024-04-25T00:30:15.801549+0000","last_fullsized":"2024-04-25T00:30:15.801549+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":752,"log_dups_size":0,"ondisk_log_size":752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":28,"num_read_kb":89132,"num_write":742,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"69'3790","reported_seq":3915,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.942896+0000","last_change":"2024-04-25T00:01:05.375142+0000","last_active":"2024-04-25T00:30:15.942896+0000","last_peered":"2024-04-25T00:30:15.942896+0000","last_clean":"2024-04-25T00:30:15.942896+0000","last_became_active":"2024-04-25T00:01:05.373857+0000","last_became_peered":"2024-04-25T00:01:05.373857+0000","last_unstale":"2024-04-25T00:30:15.942896+0000","last_undegraded":"2024-04-25T00:30:15.942896+0000","last_fullsized":"2024-04-25T00:30:15.942896+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3790,"log_dups_size":0,"ondisk_log_size":3790,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:15.879673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19857,"num_write_kb":263976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"74'48","reported_seq":213,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.740146+0000","last_change":"2024-04-25T00:01:18.502786+0000","last_active":"2024-04-25T00:30:26.740146+0000","last_peered":"2024-04-25T00:30:26.740146+0000","last_clean":"2024-04-25T00:30:26.740146+0000","last_became_active":"2024-04-25T00:01:18.501785+0000","last_became_peered":"2024-04-25T00:01:18.501785+0000","last_unstale":"2024-04-25T00:30:26.740146+0000","last_undegraded":"2024-04-25T00:30:26.740146+0000","last_fullsized":"2024-04-25T00:30:26.740146+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:09:31.441044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.26","version":"74'3694","reported_seq":3813,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.735546+0000","last_change":"2024-04-25T00:01:19.511239+0000","last_active":"2024-04-25T00:30:26.735546+0000","last_peered":"2024-04-25T00:30:26.735546+0000","last_clean":"2024-04-25T00:30:26.735546+0000","last_became_active":"2024-04-25T00:01:19.510155+0000","last_became_peered":"2024-04-25T00:01:19.510155+0000","last_unstale":"2024-04-25T00:30:26.735546+0000","last_undegraded":"2024-04-25T00:30:26.735546+0000","last_fullsized":"2024-04-25T00:30:26.735546+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3694,"log_dups_size":0,"ondisk_log_size":3694,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:43:14.452663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20834,"num_write_kb":259761,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.20","version":"70'4043","reported_seq":4171,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.807721+0000","last_change":"2024-04-25T00:01:05.378517+0000","last_active":"2024-04-25T00:30:15.807721+0000","last_peered":"2024-04-25T00:30:15.807721+0000","last_clean":"2024-04-25T00:30:15.807721+0000","last_became_active":"2024-04-25T00:01:05.377742+0000","last_became_peered":"2024-04-25T00:01:05.377742+0000","last_unstale":"2024-04-25T00:30:15.807721+0000","last_undegraded":"2024-04-25T00:30:15.807721+0000","last_fullsized":"2024-04-25T00:30:15.807721+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4043,"log_dups_size":0,"ondisk_log_size":4043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:06.450369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21543,"num_write_kb":291017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"74'5759","reported_seq":5879,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.589637+0000","last_change":"2024-04-25T00:01:19.508999+0000","last_active":"2024-04-25T00:30:26.589637+0000","last_peered":"2024-04-25T00:30:26.589637+0000","last_clean":"2024-04-25T00:30:26.589637+0000","last_became_active":"2024-04-25T00:01:19.507960+0000","last_became_peered":"2024-04-25T00:01:19.507960+0000","last_unstale":"2024-04-25T00:30:26.589637+0000","last_undegraded":"2024-04-25T00:30:26.589637+0000","last_fullsized":"2024-04-25T00:30:26.589637+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":5759,"log_dups_size":0,"ondisk_log_size":5759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:32:40.274908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":31168,"num_write_kb":366885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.21","version":"70'4068","reported_seq":4197,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.803508+0000","last_change":"2024-04-25T00:01:05.372927+0000","last_active":"2024-04-25T00:30:15.803508+0000","last_peered":"2024-04-25T00:30:15.803508+0000","last_clean":"2024-04-25T00:30:15.803508+0000","last_became_active":"2024-04-25T00:01:05.372050+0000","last_became_peered":"2024-04-25T00:01:05.372050+0000","last_unstale":"2024-04-25T00:30:15.803508+0000","last_undegraded":"2024-04-25T00:30:15.803508+0000","last_fullsized":"2024-04-25T00:30:15.803508+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4068,"log_dups_size":0,"ondisk_log_size":4068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:48.202251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21791,"num_write_kb":292243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.24","version":"74'4315","reported_seq":4436,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.724152+0000","last_change":"2024-04-25T00:01:19.531072+0000","last_active":"2024-04-25T00:30:26.724152+0000","last_peered":"2024-04-25T00:30:26.724152+0000","last_clean":"2024-04-25T00:30:26.724152+0000","last_became_active":"2024-04-25T00:01:19.530836+0000","last_became_peered":"2024-04-25T00:01:19.530836+0000","last_unstale":"2024-04-25T00:30:26.724152+0000","last_undegraded":"2024-04-25T00:30:26.724152+0000","last_fullsized":"2024-04-25T00:30:26.724152+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4315,"log_dups_size":0,"ondisk_log_size":4315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:08.508741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":23956,"num_write_kb":309408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"70'4267","reported_seq":4395,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.807204+0000","last_change":"2024-04-25T00:01:05.362960+0000","last_active":"2024-04-25T00:30:15.807204+0000","last_peered":"2024-04-25T00:30:15.807204+0000","last_clean":"2024-04-25T00:30:15.807204+0000","last_became_active":"2024-04-25T00:01:05.361067+0000","last_became_peered":"2024-04-25T00:01:05.361067+0000","last_unstale":"2024-04-25T00:30:15.807204+0000","last_undegraded":"2024-04-25T00:30:15.807204+0000","last_fullsized":"2024-04-25T00:30:15.807204+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4267,"log_dups_size":0,"ondisk_log_size":4267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:24.017263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23473,"num_write_kb":311317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"74'4634","reported_seq":4754,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.737724+0000","last_change":"2024-04-25T00:01:19.506029+0000","last_active":"2024-04-25T00:30:26.737724+0000","last_peered":"2024-04-25T00:30:26.737724+0000","last_clean":"2024-04-25T00:30:26.737724+0000","last_became_active":"2024-04-25T00:01:19.505184+0000","last_became_peered":"2024-04-25T00:01:19.505184+0000","last_unstale":"2024-04-25T00:30:26.737724+0000","last_undegraded":"2024-04-25T00:30:26.737724+0000","last_fullsized":"2024-04-25T00:30:26.737724+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4634,"log_dups_size":0,"ondisk_log_size":4634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:08.508741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":6,"num_write":26896,"num_write_kb":316898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"70'4083","reported_seq":4211,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.801208+0000","last_change":"2024-04-25T00:01:05.358690+0000","last_active":"2024-04-25T00:30:15.801208+0000","last_peered":"2024-04-25T00:30:15.801208+0000","last_clean":"2024-04-25T00:30:15.801208+0000","last_became_active":"2024-04-25T00:01:05.357340+0000","last_became_peered":"2024-04-25T00:01:05.357340+0000","last_unstale":"2024-04-25T00:30:15.801208+0000","last_undegraded":"2024-04-25T00:30:15.801208+0000","last_fullsized":"2024-04-25T00:30:15.801208+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4083,"log_dups_size":0,"ondisk_log_size":4083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:36:55.436661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21935,"num_write_kb":296816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.22","version":"74'5147","reported_seq":5272,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.738948+0000","last_change":"2024-04-25T00:01:19.508691+0000","last_active":"2024-04-25T00:30:26.738948+0000","last_peered":"2024-04-25T00:30:26.738948+0000","last_clean":"2024-04-25T00:30:26.738948+0000","last_became_active":"2024-04-25T00:01:19.508123+0000","last_became_peered":"2024-04-25T00:01:19.508123+0000","last_unstale":"2024-04-25T00:30:26.738948+0000","last_undegraded":"2024-04-25T00:30:26.738948+0000","last_fullsized":"2024-04-25T00:30:26.738948+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":5147,"log_dups_size":0,"ondisk_log_size":5147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:47:19.526845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":43,"num_write":27901,"num_write_kb":341497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"70'3848","reported_seq":3976,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.812551+0000","last_change":"2024-04-25T00:01:05.387057+0000","last_active":"2024-04-25T00:30:15.812551+0000","last_peered":"2024-04-25T00:30:15.812551+0000","last_clean":"2024-04-25T00:30:15.812551+0000","last_became_active":"2024-04-25T00:01:05.386724+0000","last_became_peered":"2024-04-25T00:01:05.386724+0000","last_unstale":"2024-04-25T00:30:15.812551+0000","last_undegraded":"2024-04-25T00:30:15.812551+0000","last_fullsized":"2024-04-25T00:30:15.812551+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3848,"log_dups_size":0,"ondisk_log_size":3848,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:55:09.073286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20848,"num_write_kb":276248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.23","version":"74'4386","reported_seq":4508,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.730888+0000","last_change":"2024-04-25T00:01:19.522760+0000","last_active":"2024-04-25T00:30:26.730888+0000","last_peered":"2024-04-25T00:30:26.730888+0000","last_clean":"2024-04-25T00:30:26.730888+0000","last_became_active":"2024-04-25T00:01:19.522380+0000","last_became_peered":"2024-04-25T00:01:19.522380+0000","last_unstale":"2024-04-25T00:30:26.730888+0000","last_undegraded":"2024-04-25T00:30:26.730888+0000","last_fullsized":"2024-04-25T00:30:26.730888+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4386,"log_dups_size":0,"ondisk_log_size":4386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:55:46.853565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":23395,"num_write_kb":294532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.25","version":"70'4349","reported_seq":4476,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.812426+0000","last_change":"2024-04-25T00:01:05.367296+0000","last_active":"2024-04-25T00:30:15.812426+0000","last_peered":"2024-04-25T00:30:15.812426+0000","last_clean":"2024-04-25T00:30:15.812426+0000","last_became_active":"2024-04-25T00:01:05.363838+0000","last_became_peered":"2024-04-25T00:01:05.363838+0000","last_unstale":"2024-04-25T00:30:15.812426+0000","last_undegraded":"2024-04-25T00:30:15.812426+0000","last_fullsized":"2024-04-25T00:30:15.812426+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4349,"log_dups_size":0,"ondisk_log_size":4349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:43:07.053074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24353,"num_write_kb":306782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"74'4505","reported_seq":4625,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.749021+0000","last_change":"2024-04-25T00:01:19.510902+0000","last_active":"2024-04-25T00:30:26.749021+0000","last_peered":"2024-04-25T00:30:26.749021+0000","last_clean":"2024-04-25T00:30:26.749021+0000","last_became_active":"2024-04-25T00:01:19.510008+0000","last_became_peered":"2024-04-25T00:01:19.510008+0000","last_unstale":"2024-04-25T00:30:26.749021+0000","last_undegraded":"2024-04-25T00:30:26.749021+0000","last_fullsized":"2024-04-25T00:30:26.749021+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4505,"log_dups_size":0,"ondisk_log_size":4505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:40:32.292140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24621,"num_write_kb":315853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"70'3623","reported_seq":3750,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.806689+0000","last_change":"2024-04-25T00:01:05.365761+0000","last_active":"2024-04-25T00:30:15.806689+0000","last_peered":"2024-04-25T00:30:15.806689+0000","last_clean":"2024-04-25T00:30:15.806689+0000","last_became_active":"2024-04-25T00:01:05.361119+0000","last_became_peered":"2024-04-25T00:01:05.361119+0000","last_unstale":"2024-04-25T00:30:15.806689+0000","last_undegraded":"2024-04-25T00:30:15.806689+0000","last_fullsized":"2024-04-25T00:30:15.806689+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3623,"log_dups_size":0,"ondisk_log_size":3623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20289,"num_write_kb":274005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.21","version":"74'4291","reported_seq":4414,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.777806+0000","last_change":"2024-04-25T00:01:19.518534+0000","last_active":"2024-04-25T00:30:26.777806+0000","last_peered":"2024-04-25T00:30:26.777806+0000","last_clean":"2024-04-25T00:30:26.777806+0000","last_became_active":"2024-04-25T00:01:19.518110+0000","last_became_peered":"2024-04-25T00:01:19.518110+0000","last_unstale":"2024-04-25T00:30:26.777806+0000","last_undegraded":"2024-04-25T00:30:26.777806+0000","last_fullsized":"2024-04-25T00:30:26.777806+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4291,"log_dups_size":0,"ondisk_log_size":4291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:05:09.219823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":23037,"num_write_kb":301131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"69'4983","reported_seq":5110,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.799145+0000","last_change":"2024-04-25T00:01:05.373464+0000","last_active":"2024-04-25T00:30:15.799145+0000","last_peered":"2024-04-25T00:30:15.799145+0000","last_clean":"2024-04-25T00:30:15.799145+0000","last_became_active":"2024-04-25T00:01:05.372611+0000","last_became_peered":"2024-04-25T00:01:05.372611+0000","last_unstale":"2024-04-25T00:30:15.799145+0000","last_undegraded":"2024-04-25T00:30:15.799145+0000","last_fullsized":"2024-04-25T00:30:15.799145+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4983,"log_dups_size":0,"ondisk_log_size":4983,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:31.456792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26776,"num_write_kb":349057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2e","version":"74'4567","reported_seq":4690,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.735238+0000","last_change":"2024-04-25T00:01:19.542475+0000","last_active":"2024-04-25T00:30:26.735238+0000","last_peered":"2024-04-25T00:30:26.735238+0000","last_clean":"2024-04-25T00:30:26.735238+0000","last_became_active":"2024-04-25T00:01:19.541877+0000","last_became_peered":"2024-04-25T00:01:19.541877+0000","last_unstale":"2024-04-25T00:30:26.735238+0000","last_undegraded":"2024-04-25T00:30:26.735238+0000","last_fullsized":"2024-04-25T00:30:26.735238+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4567,"log_dups_size":0,"ondisk_log_size":4567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:02:57.373014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":36,"num_write":24879,"num_write_kb":312940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.28","version":"70'4294","reported_seq":4421,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.804148+0000","last_change":"2024-04-25T00:01:05.372906+0000","last_active":"2024-04-25T00:30:15.804148+0000","last_peered":"2024-04-25T00:30:15.804148+0000","last_clean":"2024-04-25T00:30:15.804148+0000","last_became_active":"2024-04-25T00:01:05.372148+0000","last_became_peered":"2024-04-25T00:01:05.372148+0000","last_unstale":"2024-04-25T00:30:15.804148+0000","last_undegraded":"2024-04-25T00:30:15.804148+0000","last_fullsized":"2024-04-25T00:30:15.804148+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4294,"log_dups_size":0,"ondisk_log_size":4294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:43.615895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23157,"num_write_kb":289708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2f","version":"74'5194","reported_seq":5313,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.771951+0000","last_change":"2024-04-25T00:01:19.498515+0000","last_active":"2024-04-25T00:30:26.771951+0000","last_peered":"2024-04-25T00:30:26.771951+0000","last_clean":"2024-04-25T00:30:26.771951+0000","last_became_active":"2024-04-25T00:01:19.496310+0000","last_became_peered":"2024-04-25T00:01:19.496310+0000","last_unstale":"2024-04-25T00:30:26.771951+0000","last_undegraded":"2024-04-25T00:30:26.771951+0000","last_fullsized":"2024-04-25T00:30:26.771951+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":5194,"log_dups_size":0,"ondisk_log_size":5194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:20:06.661558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":28420,"num_write_kb":367977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"69'4077","reported_seq":4204,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.795788+0000","last_change":"2024-04-25T00:01:05.366340+0000","last_active":"2024-04-25T00:30:15.795788+0000","last_peered":"2024-04-25T00:30:15.795788+0000","last_clean":"2024-04-25T00:30:15.795788+0000","last_became_active":"2024-04-25T00:01:05.361952+0000","last_became_peered":"2024-04-25T00:01:05.361952+0000","last_unstale":"2024-04-25T00:30:15.795788+0000","last_undegraded":"2024-04-25T00:30:15.795788+0000","last_fullsized":"2024-04-25T00:30:15.795788+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4077,"log_dups_size":0,"ondisk_log_size":4077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:41.948662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22153,"num_write_kb":297857,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2c","version":"74'3564","reported_seq":3684,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.750933+0000","last_change":"2024-04-25T00:01:19.508257+0000","last_active":"2024-04-25T00:30:26.750933+0000","last_peered":"2024-04-25T00:30:26.750933+0000","last_clean":"2024-04-25T00:30:26.750933+0000","last_became_active":"2024-04-25T00:01:19.507516+0000","last_became_peered":"2024-04-25T00:01:19.507516+0000","last_unstale":"2024-04-25T00:30:26.750933+0000","last_undegraded":"2024-04-25T00:30:26.750933+0000","last_fullsized":"2024-04-25T00:30:26.750933+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3564,"log_dups_size":0,"ondisk_log_size":3564,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:45.121513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":19793,"num_write_kb":247542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"69'4229","reported_seq":4356,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.942181+0000","last_change":"2024-04-25T00:01:05.387457+0000","last_active":"2024-04-25T00:30:15.942181+0000","last_peered":"2024-04-25T00:30:15.942181+0000","last_clean":"2024-04-25T00:30:15.942181+0000","last_became_active":"2024-04-25T00:01:05.386919+0000","last_became_peered":"2024-04-25T00:01:05.386919+0000","last_unstale":"2024-04-25T00:30:15.942181+0000","last_undegraded":"2024-04-25T00:30:15.942181+0000","last_fullsized":"2024-04-25T00:30:15.942181+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4229,"log_dups_size":0,"ondisk_log_size":4229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22430,"num_write_kb":302838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2d","version":"74'4662","reported_seq":4781,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.561243+0000","last_change":"2024-04-25T00:01:19.520648+0000","last_active":"2024-04-25T00:30:26.561243+0000","last_peered":"2024-04-25T00:30:26.561243+0000","last_clean":"2024-04-25T00:30:26.561243+0000","last_became_active":"2024-04-25T00:01:19.519303+0000","last_became_peered":"2024-04-25T00:01:19.519303+0000","last_unstale":"2024-04-25T00:30:26.561243+0000","last_undegraded":"2024-04-25T00:30:26.561243+0000","last_fullsized":"2024-04-25T00:30:26.561243+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4662,"log_dups_size":0,"ondisk_log_size":4662,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:23.993729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":25332,"num_write_kb":310579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2b","version":"70'4434","reported_seq":4562,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.792469+0000","last_change":"2024-04-25T00:01:05.366249+0000","last_active":"2024-04-25T00:30:15.792469+0000","last_peered":"2024-04-25T00:30:15.792469+0000","last_clean":"2024-04-25T00:30:15.792469+0000","last_became_active":"2024-04-25T00:01:05.365548+0000","last_became_peered":"2024-04-25T00:01:05.365548+0000","last_unstale":"2024-04-25T00:30:15.792469+0000","last_undegraded":"2024-04-25T00:30:15.792469+0000","last_fullsized":"2024-04-25T00:30:15.792469+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4434,"log_dups_size":0,"ondisk_log_size":4434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:22:21.143639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":24224,"num_write_kb":326776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"74'4527","reported_seq":4647,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.719005+0000","last_change":"2024-04-25T00:01:19.514659+0000","last_active":"2024-04-25T00:30:26.719005+0000","last_peered":"2024-04-25T00:30:26.719005+0000","last_clean":"2024-04-25T00:30:26.719005+0000","last_became_active":"2024-04-25T00:01:19.514232+0000","last_became_peered":"2024-04-25T00:01:19.514232+0000","last_unstale":"2024-04-25T00:30:26.719005+0000","last_undegraded":"2024-04-25T00:30:26.719005+0000","last_fullsized":"2024-04-25T00:30:26.719005+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4527,"log_dups_size":0,"ondisk_log_size":4527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:45.121513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":25088,"num_write_kb":313806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"70'3677","reported_seq":3789,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.802932+0000","last_change":"2024-04-25T00:01:46.665943+0000","last_active":"2024-04-25T00:30:15.802932+0000","last_peered":"2024-04-25T00:30:15.802932+0000","last_clean":"2024-04-25T00:30:15.802932+0000","last_became_active":"2024-04-25T00:01:46.663874+0000","last_became_peered":"2024-04-25T00:01:46.663874+0000","last_unstale":"2024-04-25T00:30:15.802932+0000","last_undegraded":"2024-04-25T00:30:15.802932+0000","last_fullsized":"2024-04-25T00:30:15.802932+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3677,"log_dups_size":0,"ondisk_log_size":3677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:53:00.633407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20325,"num_write_kb":253590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2b","version":"74'4916","reported_seq":5040,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.774860+0000","last_change":"2024-04-25T00:01:19.503419+0000","last_active":"2024-04-25T00:30:26.774860+0000","last_peered":"2024-04-25T00:30:26.774860+0000","last_clean":"2024-04-25T00:30:26.774860+0000","last_became_active":"2024-04-25T00:01:19.500568+0000","last_became_peered":"2024-04-25T00:01:19.500568+0000","last_unstale":"2024-04-25T00:30:26.774860+0000","last_undegraded":"2024-04-25T00:30:26.774860+0000","last_fullsized":"2024-04-25T00:30:26.774860+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4916,"log_dups_size":0,"ondisk_log_size":4916,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:08.508741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":26841,"num_write_kb":321097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"69'4618","reported_seq":4745,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.814383+0000","last_change":"2024-04-25T00:01:05.370653+0000","last_active":"2024-04-25T00:30:15.814383+0000","last_peered":"2024-04-25T00:30:15.814383+0000","last_clean":"2024-04-25T00:30:15.814383+0000","last_became_active":"2024-04-25T00:01:05.369919+0000","last_became_peered":"2024-04-25T00:01:05.369919+0000","last_unstale":"2024-04-25T00:30:15.814383+0000","last_undegraded":"2024-04-25T00:30:15.814383+0000","last_fullsized":"2024-04-25T00:30:15.814383+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4618,"log_dups_size":0,"ondisk_log_size":4618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:30:08.311326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25225,"num_write_kb":326547,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.28","version":"74'3900","reported_seq":4020,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.755166+0000","last_change":"2024-04-25T00:01:19.540104+0000","last_active":"2024-04-25T00:30:26.755166+0000","last_peered":"2024-04-25T00:30:26.755166+0000","last_clean":"2024-04-25T00:30:26.755166+0000","last_became_active":"2024-04-25T00:01:19.539775+0000","last_became_peered":"2024-04-25T00:01:19.539775+0000","last_unstale":"2024-04-25T00:30:26.755166+0000","last_undegraded":"2024-04-25T00:30:26.755166+0000","last_fullsized":"2024-04-25T00:30:26.755166+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3900,"log_dups_size":0,"ondisk_log_size":3900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:53:14.768520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21234,"num_write_kb":281892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"70'3829","reported_seq":3955,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.804122+0000","last_change":"2024-04-25T00:01:05.367001+0000","last_active":"2024-04-25T00:30:15.804122+0000","last_peered":"2024-04-25T00:30:15.804122+0000","last_clean":"2024-04-25T00:30:15.804122+0000","last_became_active":"2024-04-25T00:01:05.366264+0000","last_became_peered":"2024-04-25T00:01:05.366264+0000","last_unstale":"2024-04-25T00:30:15.804122+0000","last_undegraded":"2024-04-25T00:30:15.804122+0000","last_fullsized":"2024-04-25T00:30:15.804122+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3829,"log_dups_size":0,"ondisk_log_size":3829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19822,"num_write_kb":292385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.29","version":"74'4218","reported_seq":4338,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.734841+0000","last_change":"2024-04-25T00:01:19.497247+0000","last_active":"2024-04-25T00:30:26.734841+0000","last_peered":"2024-04-25T00:30:26.734841+0000","last_clean":"2024-04-25T00:30:26.734841+0000","last_became_active":"2024-04-25T00:01:19.496517+0000","last_became_peered":"2024-04-25T00:01:19.496517+0000","last_unstale":"2024-04-25T00:30:26.734841+0000","last_undegraded":"2024-04-25T00:30:26.734841+0000","last_fullsized":"2024-04-25T00:30:26.734841+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4218,"log_dups_size":0,"ondisk_log_size":4218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:30.129056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":23201,"num_write_kb":293725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2f","version":"70'5230","reported_seq":5357,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.813628+0000","last_change":"2024-04-25T00:01:05.379958+0000","last_active":"2024-04-25T00:30:15.813628+0000","last_peered":"2024-04-25T00:30:15.813628+0000","last_clean":"2024-04-25T00:30:15.813628+0000","last_became_active":"2024-04-25T00:01:05.379090+0000","last_became_peered":"2024-04-25T00:01:05.379090+0000","last_unstale":"2024-04-25T00:30:15.813628+0000","last_undegraded":"2024-04-25T00:30:15.813628+0000","last_fullsized":"2024-04-25T00:30:15.813628+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":5230,"log_dups_size":0,"ondisk_log_size":5230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:59:50.832958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":27866,"num_write_kb":365657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.36","version":"74'4384","reported_seq":4504,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.681328+0000","last_change":"2024-04-25T00:01:19.530176+0000","last_active":"2024-04-25T00:30:26.681328+0000","last_peered":"2024-04-25T00:30:26.681328+0000","last_clean":"2024-04-25T00:30:26.681328+0000","last_became_active":"2024-04-25T00:01:19.529410+0000","last_became_peered":"2024-04-25T00:01:19.529410+0000","last_unstale":"2024-04-25T00:30:26.681328+0000","last_undegraded":"2024-04-25T00:30:26.681328+0000","last_fullsized":"2024-04-25T00:30:26.681328+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4384,"log_dups_size":0,"ondisk_log_size":4384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:40:13.204880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":7,"num_write":24230,"num_write_kb":310193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.30","version":"69'4051","reported_seq":4177,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.803573+0000","last_change":"2024-04-25T00:01:05.370826+0000","last_active":"2024-04-25T00:30:15.803573+0000","last_peered":"2024-04-25T00:30:15.803573+0000","last_clean":"2024-04-25T00:30:15.803573+0000","last_became_active":"2024-04-25T00:01:05.369848+0000","last_became_peered":"2024-04-25T00:01:05.369848+0000","last_unstale":"2024-04-25T00:30:15.803573+0000","last_undegraded":"2024-04-25T00:30:15.803573+0000","last_fullsized":"2024-04-25T00:30:15.803573+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4051,"log_dups_size":0,"ondisk_log_size":4051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:23.870121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21793,"num_write_kb":288101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.37","version":"74'6689","reported_seq":6811,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.769600+0000","last_change":"2024-04-25T00:01:19.511625+0000","last_active":"2024-04-25T00:30:26.769600+0000","last_peered":"2024-04-25T00:30:26.769600+0000","last_clean":"2024-04-25T00:30:26.769600+0000","last_became_active":"2024-04-25T00:01:19.511349+0000","last_became_peered":"2024-04-25T00:01:19.511349+0000","last_unstale":"2024-04-25T00:30:26.769600+0000","last_undegraded":"2024-04-25T00:30:26.769600+0000","last_fullsized":"2024-04-25T00:30:26.769600+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":6689,"log_dups_size":0,"ondisk_log_size":6689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:02:57.373014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":8,"num_write":38367,"num_write_kb":445068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"70'3993","reported_seq":4119,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.795313+0000","last_change":"2024-04-25T00:01:05.390341+0000","last_active":"2024-04-25T00:30:15.795313+0000","last_peered":"2024-04-25T00:30:15.795313+0000","last_clean":"2024-04-25T00:30:15.795313+0000","last_became_active":"2024-04-25T00:01:05.389707+0000","last_became_peered":"2024-04-25T00:01:05.389707+0000","last_unstale":"2024-04-25T00:30:15.795313+0000","last_undegraded":"2024-04-25T00:30:15.795313+0000","last_fullsized":"2024-04-25T00:30:15.795313+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3993,"log_dups_size":0,"ondisk_log_size":3993,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:54:21.138947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21619,"num_write_kb":287362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.34","version":"74'3411","reported_seq":3532,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.695377+0000","last_change":"2024-04-25T00:01:19.513630+0000","last_active":"2024-04-25T00:30:26.695377+0000","last_peered":"2024-04-25T00:30:26.695377+0000","last_clean":"2024-04-25T00:30:26.695377+0000","last_became_active":"2024-04-25T00:01:19.512978+0000","last_became_peered":"2024-04-25T00:01:19.512978+0000","last_unstale":"2024-04-25T00:30:26.695377+0000","last_undegraded":"2024-04-25T00:30:26.695377+0000","last_fullsized":"2024-04-25T00:30:26.695377+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3411,"log_dups_size":0,"ondisk_log_size":3411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:57.751007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":34,"num_write":18670,"num_write_kb":234685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"70'3759","reported_seq":3886,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.809965+0000","last_change":"2024-04-25T00:01:05.378373+0000","last_active":"2024-04-25T00:30:15.809965+0000","last_peered":"2024-04-25T00:30:15.809965+0000","last_clean":"2024-04-25T00:30:15.809965+0000","last_became_active":"2024-04-25T00:01:05.376570+0000","last_became_peered":"2024-04-25T00:01:05.376570+0000","last_unstale":"2024-04-25T00:30:15.809965+0000","last_undegraded":"2024-04-25T00:30:15.809965+0000","last_fullsized":"2024-04-25T00:30:15.809965+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3759,"log_dups_size":0,"ondisk_log_size":3759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:30.228060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20571,"num_write_kb":279348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.35","version":"74'4078","reported_seq":4199,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.645015+0000","last_change":"2024-04-25T00:01:19.516716+0000","last_active":"2024-04-25T00:30:26.645015+0000","last_peered":"2024-04-25T00:30:26.645015+0000","last_clean":"2024-04-25T00:30:26.645015+0000","last_became_active":"2024-04-25T00:01:19.516142+0000","last_became_peered":"2024-04-25T00:01:19.516142+0000","last_unstale":"2024-04-25T00:30:26.645015+0000","last_undegraded":"2024-04-25T00:30:26.645015+0000","last_fullsized":"2024-04-25T00:30:26.645015+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4078,"log_dups_size":0,"ondisk_log_size":4078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:28:18.234546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":22241,"num_write_kb":273608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.33","version":"70'3191","reported_seq":3317,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.810448+0000","last_change":"2024-04-25T00:01:05.381011+0000","last_active":"2024-04-25T00:30:15.810448+0000","last_peered":"2024-04-25T00:30:15.810448+0000","last_clean":"2024-04-25T00:30:15.810448+0000","last_became_active":"2024-04-25T00:01:05.380236+0000","last_became_peered":"2024-04-25T00:01:05.380236+0000","last_unstale":"2024-04-25T00:30:15.810448+0000","last_undegraded":"2024-04-25T00:30:15.810448+0000","last_fullsized":"2024-04-25T00:30:15.810448+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3191,"log_dups_size":0,"ondisk_log_size":3191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:06:19.307378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17526,"num_write_kb":230308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"74'4055","reported_seq":4174,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.756767+0000","last_change":"2024-04-25T00:01:19.529023+0000","last_active":"2024-04-25T00:30:26.756767+0000","last_peered":"2024-04-25T00:30:26.756767+0000","last_clean":"2024-04-25T00:30:26.756767+0000","last_became_active":"2024-04-25T00:01:19.524609+0000","last_became_peered":"2024-04-25T00:01:19.524609+0000","last_unstale":"2024-04-25T00:30:26.756767+0000","last_undegraded":"2024-04-25T00:30:26.756767+0000","last_fullsized":"2024-04-25T00:30:26.756767+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4055,"log_dups_size":0,"ondisk_log_size":4055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:41:19.150476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":21801,"num_write_kb":276048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.34","version":"70'2711","reported_seq":2837,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.792324+0000","last_change":"2024-04-25T00:01:05.385853+0000","last_active":"2024-04-25T00:30:15.792324+0000","last_peered":"2024-04-25T00:30:15.792324+0000","last_clean":"2024-04-25T00:30:15.792324+0000","last_became_active":"2024-04-25T00:01:05.384997+0000","last_became_peered":"2024-04-25T00:01:05.384997+0000","last_unstale":"2024-04-25T00:30:15.792324+0000","last_undegraded":"2024-04-25T00:30:15.792324+0000","last_fullsized":"2024-04-25T00:30:15.792324+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":2711,"log_dups_size":0,"ondisk_log_size":2711,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:36:55.436661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14839,"num_write_kb":207133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.33","version":"74'3434","reported_seq":3556,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.729873+0000","last_change":"2024-04-25T00:01:19.516867+0000","last_active":"2024-04-25T00:30:26.729873+0000","last_peered":"2024-04-25T00:30:26.729873+0000","last_clean":"2024-04-25T00:30:26.729873+0000","last_became_active":"2024-04-25T00:01:19.515764+0000","last_became_peered":"2024-04-25T00:01:19.515764+0000","last_unstale":"2024-04-25T00:30:26.729873+0000","last_undegraded":"2024-04-25T00:30:26.729873+0000","last_fullsized":"2024-04-25T00:30:26.729873+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3434,"log_dups_size":0,"ondisk_log_size":3434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:53:14.768520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":5,"num_write":19347,"num_write_kb":241776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"69'3819","reported_seq":3946,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.790737+0000","last_change":"2024-04-25T00:01:05.389424+0000","last_active":"2024-04-25T00:30:15.790737+0000","last_peered":"2024-04-25T00:30:15.790737+0000","last_clean":"2024-04-25T00:30:15.790737+0000","last_became_active":"2024-04-25T00:01:05.388999+0000","last_became_peered":"2024-04-25T00:01:05.388999+0000","last_unstale":"2024-04-25T00:30:15.790737+0000","last_undegraded":"2024-04-25T00:30:15.790737+0000","last_fullsized":"2024-04-25T00:30:15.790737+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3819,"log_dups_size":0,"ondisk_log_size":3819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:31.456792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":19986,"num_write_kb":272748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.30","version":"74'4335","reported_seq":4453,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.723246+0000","last_change":"2024-04-25T00:01:19.519365+0000","last_active":"2024-04-25T00:30:26.723246+0000","last_peered":"2024-04-25T00:30:26.723246+0000","last_clean":"2024-04-25T00:30:26.723246+0000","last_became_active":"2024-04-25T00:01:19.519085+0000","last_became_peered":"2024-04-25T00:01:19.519085+0000","last_unstale":"2024-04-25T00:30:26.723246+0000","last_undegraded":"2024-04-25T00:30:26.723246+0000","last_fullsized":"2024-04-25T00:30:26.723246+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4335,"log_dups_size":0,"ondisk_log_size":4335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:05:09.219823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23507,"num_write_kb":286989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"69'4866","reported_seq":4992,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.790509+0000","last_change":"2024-04-25T00:01:05.368974+0000","last_active":"2024-04-25T00:30:15.790509+0000","last_peered":"2024-04-25T00:30:15.790509+0000","last_clean":"2024-04-25T00:30:15.790509+0000","last_became_active":"2024-04-25T00:01:05.367893+0000","last_became_peered":"2024-04-25T00:01:05.367893+0000","last_unstale":"2024-04-25T00:30:15.790509+0000","last_undegraded":"2024-04-25T00:30:15.790509+0000","last_fullsized":"2024-04-25T00:30:15.790509+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4866,"log_dups_size":0,"ondisk_log_size":4866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:30:23.557381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26504,"num_write_kb":327449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.31","version":"74'4017","reported_seq":4136,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.738931+0000","last_change":"2024-04-25T00:01:19.513514+0000","last_active":"2024-04-25T00:30:26.738931+0000","last_peered":"2024-04-25T00:30:26.738931+0000","last_clean":"2024-04-25T00:30:26.738931+0000","last_became_active":"2024-04-25T00:01:19.513184+0000","last_became_peered":"2024-04-25T00:01:19.513184+0000","last_unstale":"2024-04-25T00:30:26.738931+0000","last_undegraded":"2024-04-25T00:30:26.738931+0000","last_fullsized":"2024-04-25T00:30:26.738931+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4017,"log_dups_size":0,"ondisk_log_size":4017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:47.859626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":21984,"num_write_kb":288846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"70'5540","reported_seq":5666,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.944163+0000","last_change":"2024-04-25T00:01:05.362303+0000","last_active":"2024-04-25T00:30:15.944163+0000","last_peered":"2024-04-25T00:30:15.944163+0000","last_clean":"2024-04-25T00:30:15.944163+0000","last_became_active":"2024-04-25T00:01:05.361586+0000","last_became_peered":"2024-04-25T00:01:05.361586+0000","last_unstale":"2024-04-25T00:30:15.944163+0000","last_undegraded":"2024-04-25T00:30:15.944163+0000","last_fullsized":"2024-04-25T00:30:15.944163+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":5540,"log_dups_size":0,"ondisk_log_size":5540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:30.228060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30654,"num_write_kb":393416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3e","version":"74'5058","reported_seq":5180,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.759205+0000","last_change":"2024-04-25T00:01:19.515101+0000","last_active":"2024-04-25T00:30:26.759205+0000","last_peered":"2024-04-25T00:30:26.759205+0000","last_clean":"2024-04-25T00:30:26.759205+0000","last_became_active":"2024-04-25T00:01:19.513314+0000","last_became_peered":"2024-04-25T00:01:19.513314+0000","last_unstale":"2024-04-25T00:30:26.759205+0000","last_undegraded":"2024-04-25T00:30:26.759205+0000","last_fullsized":"2024-04-25T00:30:26.759205+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":5058,"log_dups_size":0,"ondisk_log_size":5058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:50.351599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":15,"num_write":27439,"num_write_kb":359791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.38","version":"70'4026","reported_seq":4151,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.793659+0000","last_change":"2024-04-25T00:01:05.390256+0000","last_active":"2024-04-25T00:30:15.793659+0000","last_peered":"2024-04-25T00:30:15.793659+0000","last_clean":"2024-04-25T00:30:15.793659+0000","last_became_active":"2024-04-25T00:01:05.389724+0000","last_became_peered":"2024-04-25T00:01:05.389724+0000","last_unstale":"2024-04-25T00:30:15.793659+0000","last_undegraded":"2024-04-25T00:30:15.793659+0000","last_fullsized":"2024-04-25T00:30:15.793659+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4026,"log_dups_size":0,"ondisk_log_size":4026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:09:18.315688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21352,"num_write_kb":290527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3f","version":"74'4365","reported_seq":4489,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.773739+0000","last_change":"2024-04-25T00:01:19.511493+0000","last_active":"2024-04-25T00:30:26.773739+0000","last_peered":"2024-04-25T00:30:26.773739+0000","last_clean":"2024-04-25T00:30:26.773739+0000","last_became_active":"2024-04-25T00:01:19.508049+0000","last_became_peered":"2024-04-25T00:01:19.508049+0000","last_unstale":"2024-04-25T00:30:26.773739+0000","last_undegraded":"2024-04-25T00:30:26.773739+0000","last_fullsized":"2024-04-25T00:30:26.773739+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4365,"log_dups_size":0,"ondisk_log_size":4365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:55:46.853565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":8,"num_write":24611,"num_write_kb":307761,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"70'3970","reported_seq":4095,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.790756+0000","last_change":"2024-04-25T00:01:05.374766+0000","last_active":"2024-04-25T00:30:15.790756+0000","last_peered":"2024-04-25T00:30:15.790756+0000","last_clean":"2024-04-25T00:30:15.790756+0000","last_became_active":"2024-04-25T00:01:05.369739+0000","last_became_peered":"2024-04-25T00:01:05.369739+0000","last_unstale":"2024-04-25T00:30:15.790756+0000","last_undegraded":"2024-04-25T00:30:15.790756+0000","last_fullsized":"2024-04-25T00:30:15.790756+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3970,"log_dups_size":0,"ondisk_log_size":3970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:09:18.315688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22008,"num_write_kb":289131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3c","version":"74'3966","reported_seq":4087,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.664683+0000","last_change":"2024-04-25T00:01:19.521456+0000","last_active":"2024-04-25T00:30:26.664683+0000","last_peered":"2024-04-25T00:30:26.664683+0000","last_clean":"2024-04-25T00:30:26.664683+0000","last_became_active":"2024-04-25T00:01:19.520787+0000","last_became_peered":"2024-04-25T00:01:19.520787+0000","last_unstale":"2024-04-25T00:30:26.664683+0000","last_undegraded":"2024-04-25T00:30:26.664683+0000","last_fullsized":"2024-04-25T00:30:26.664683+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3966,"log_dups_size":0,"ondisk_log_size":3966,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:25:38.981364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22683,"num_write_kb":271349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3a","version":"70'3294","reported_seq":3419,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.805979+0000","last_change":"2024-04-25T00:01:05.359253+0000","last_active":"2024-04-25T00:30:15.805979+0000","last_peered":"2024-04-25T00:30:15.805979+0000","last_clean":"2024-04-25T00:30:15.805979+0000","last_became_active":"2024-04-25T00:01:05.358477+0000","last_became_peered":"2024-04-25T00:01:05.358477+0000","last_unstale":"2024-04-25T00:30:15.805979+0000","last_undegraded":"2024-04-25T00:30:15.805979+0000","last_fullsized":"2024-04-25T00:30:15.805979+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3294,"log_dups_size":0,"ondisk_log_size":3294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:22.739214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17420,"num_write_kb":225566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3d","version":"74'5153","reported_seq":5274,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.749265+0000","last_change":"2024-04-25T00:01:19.509505+0000","last_active":"2024-04-25T00:30:26.749265+0000","last_peered":"2024-04-25T00:30:26.749265+0000","last_clean":"2024-04-25T00:30:26.749265+0000","last_became_active":"2024-04-25T00:01:19.507812+0000","last_became_peered":"2024-04-25T00:01:19.507812+0000","last_unstale":"2024-04-25T00:30:26.749265+0000","last_undegraded":"2024-04-25T00:30:26.749265+0000","last_fullsized":"2024-04-25T00:30:26.749265+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":5153,"log_dups_size":0,"ondisk_log_size":5153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:23.993729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":28174,"num_write_kb":326190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3b","version":"70'3663","reported_seq":3788,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.792919+0000","last_change":"2024-04-25T00:01:05.386068+0000","last_active":"2024-04-25T00:30:15.792919+0000","last_peered":"2024-04-25T00:30:15.792919+0000","last_clean":"2024-04-25T00:30:15.792919+0000","last_became_active":"2024-04-25T00:01:05.378650+0000","last_became_peered":"2024-04-25T00:01:05.378650+0000","last_unstale":"2024-04-25T00:30:15.792919+0000","last_undegraded":"2024-04-25T00:30:15.792919+0000","last_fullsized":"2024-04-25T00:30:15.792919+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3663,"log_dups_size":0,"ondisk_log_size":3663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:12:52.846250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18938,"num_write_kb":284951,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3a","version":"74'3833","reported_seq":3950,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.611583+0000","last_change":"2024-04-25T00:01:19.514843+0000","last_active":"2024-04-25T00:30:26.611583+0000","last_peered":"2024-04-25T00:30:26.611583+0000","last_clean":"2024-04-25T00:30:26.611583+0000","last_became_active":"2024-04-25T00:01:19.513956+0000","last_became_peered":"2024-04-25T00:01:19.513956+0000","last_unstale":"2024-04-25T00:30:26.611583+0000","last_undegraded":"2024-04-25T00:30:26.611583+0000","last_fullsized":"2024-04-25T00:30:26.611583+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3833,"log_dups_size":0,"ondisk_log_size":3833,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:38.005233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20679,"num_write_kb":271382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"69'3516","reported_seq":3645,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.808655+0000","last_change":"2024-04-25T00:01:05.359716+0000","last_active":"2024-04-25T00:30:15.808655+0000","last_peered":"2024-04-25T00:30:15.808655+0000","last_clean":"2024-04-25T00:30:15.808655+0000","last_became_active":"2024-04-25T00:01:05.358866+0000","last_became_peered":"2024-04-25T00:01:05.358866+0000","last_unstale":"2024-04-25T00:30:15.808655+0000","last_undegraded":"2024-04-25T00:30:15.808655+0000","last_fullsized":"2024-04-25T00:30:15.808655+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3516,"log_dups_size":0,"ondisk_log_size":3516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:50.489125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":19226,"num_write_kb":253281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3b","version":"74'4130","reported_seq":4248,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.756223+0000","last_change":"2024-04-25T00:01:19.499574+0000","last_active":"2024-04-25T00:30:26.756223+0000","last_peered":"2024-04-25T00:30:26.756223+0000","last_clean":"2024-04-25T00:30:26.756223+0000","last_became_active":"2024-04-25T00:01:19.498699+0000","last_became_peered":"2024-04-25T00:01:19.498699+0000","last_unstale":"2024-04-25T00:30:26.756223+0000","last_undegraded":"2024-04-25T00:30:26.756223+0000","last_fullsized":"2024-04-25T00:30:26.756223+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4130,"log_dups_size":0,"ondisk_log_size":4130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:47.859626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":3,"num_write":21749,"num_write_kb":288055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"70'3483","reported_seq":3611,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.798639+0000","last_change":"2024-04-25T00:01:05.371291+0000","last_active":"2024-04-25T00:30:15.798639+0000","last_peered":"2024-04-25T00:30:15.798639+0000","last_clean":"2024-04-25T00:30:15.798639+0000","last_became_active":"2024-04-25T00:01:05.370522+0000","last_became_peered":"2024-04-25T00:01:05.370522+0000","last_unstale":"2024-04-25T00:30:15.798639+0000","last_undegraded":"2024-04-25T00:30:15.798639+0000","last_fullsized":"2024-04-25T00:30:15.798639+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3483,"log_dups_size":0,"ondisk_log_size":3483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:11:27.044893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18326,"num_write_kb":261938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.38","version":"74'4120","reported_seq":4237,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.292105+0000","last_change":"2024-04-25T00:01:19.523798+0000","last_active":"2024-04-25T00:30:26.292105+0000","last_peered":"2024-04-25T00:30:26.292105+0000","last_clean":"2024-04-25T00:30:26.292105+0000","last_became_active":"2024-04-25T00:01:19.523450+0000","last_became_peered":"2024-04-25T00:01:19.523450+0000","last_unstale":"2024-04-25T00:30:26.292105+0000","last_undegraded":"2024-04-25T00:30:26.292105+0000","last_fullsized":"2024-04-25T00:30:26.292105+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4120,"log_dups_size":0,"ondisk_log_size":4120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:40:36.350832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22879,"num_write_kb":288519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3e","version":"70'4991","reported_seq":5119,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.801414+0000","last_change":"2024-04-25T00:01:05.368190+0000","last_active":"2024-04-25T00:30:15.801414+0000","last_peered":"2024-04-25T00:30:15.801414+0000","last_clean":"2024-04-25T00:30:15.801414+0000","last_became_active":"2024-04-25T00:01:05.362606+0000","last_became_peered":"2024-04-25T00:01:05.362606+0000","last_unstale":"2024-04-25T00:30:15.801414+0000","last_undegraded":"2024-04-25T00:30:15.801414+0000","last_fullsized":"2024-04-25T00:30:15.801414+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4991,"log_dups_size":0,"ondisk_log_size":4991,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:12:52.846250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26684,"num_write_kb":361051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.39","version":"74'4850","reported_seq":4967,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.763325+0000","last_change":"2024-04-25T00:01:19.505737+0000","last_active":"2024-04-25T00:30:26.763325+0000","last_peered":"2024-04-25T00:30:26.763325+0000","last_clean":"2024-04-25T00:30:26.763325+0000","last_became_active":"2024-04-25T00:01:19.504991+0000","last_became_peered":"2024-04-25T00:01:19.504991+0000","last_unstale":"2024-04-25T00:30:26.763325+0000","last_undegraded":"2024-04-25T00:30:26.763325+0000","last_fullsized":"2024-04-25T00:30:26.763325+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4850,"log_dups_size":0,"ondisk_log_size":4850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:47:19.526845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26612,"num_write_kb":323915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3f","version":"70'3815","reported_seq":3943,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.791882+0000","last_change":"2024-04-25T00:01:05.389596+0000","last_active":"2024-04-25T00:30:15.791882+0000","last_peered":"2024-04-25T00:30:15.791882+0000","last_clean":"2024-04-25T00:30:15.791882+0000","last_became_active":"2024-04-25T00:01:05.389127+0000","last_became_peered":"2024-04-25T00:01:05.389127+0000","last_unstale":"2024-04-25T00:30:15.791882+0000","last_undegraded":"2024-04-25T00:30:15.791882+0000","last_fullsized":"2024-04-25T00:30:15.791882+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3815,"log_dups_size":0,"ondisk_log_size":3815,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:09.858617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20941,"num_write_kb":280561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":105,"num_read_kb":408,"num_write":1543888,"num_write_kb":19306803,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":280555,"ondisk_log_size":280555,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":19554860,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1733,"num_read_kb":229197,"num_write":5129,"num_write_kb":21277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":39174144,"data_stored":39109720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4705,"ondisk_log_size":4705,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":3,"num_write":1412500,"num_write_kb":18776370,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":261221,"ondisk_log_size":261221,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":19117950,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1590,"num_read_kb":257618,"num_write":5060,"num_write_kb":20770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":38297600,"data_stored":38235900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4699,"ondisk_log_size":4699,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509801,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":674532,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":673470,"kb_avail":93697308,"statfs":{"total":96636764160,"available":95946043392,"internally_reserved":0,"allocated":1077248,"data_stored":617327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":689633739},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509801,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":843472,"kb_used_data":11784,"kb_used_omap":1,"kb_used_meta":831678,"kb_avail":93528368,"statfs":{"total":96636764160,"available":95773048832,"internally_reserved":0,"allocated":12066816,"data_stored":11609303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":851638726},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509800,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":870568,"kb_used_data":18016,"kb_used_omap":1,"kb_used_meta":852542,"kb_avail":93501272,"statfs":{"total":96636764160,"available":95745302528,"internally_reserved":0,"allocated":18448384,"data_stored":17985439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":873003461},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509801,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":922240,"kb_used_data":15544,"kb_used_omap":1,"kb_used_meta":906686,"kb_avail":93449600,"statfs":{"total":96636764160,"available":95692390400,"internally_reserved":0,"allocated":15917056,"data_stored":15447989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":928446919},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509802,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1076840,"kb_used_data":15968,"kb_used_omap":1,"kb_used_meta":1060862,"kb_avail":93295000,"statfs":{"total":96636764160,"available":95534080000,"internally_reserved":0,"allocated":16351232,"data_stored":15888287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1086323141},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509801,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":788556,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":787006,"kb_avail":93583284,"statfs":{"total":96636764160,"available":95829282816,"internally_reserved":0,"allocated":1576960,"data_stored":1119003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":805894597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509801,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1028004,"kb_used_data":3164,"kb_used_omap":1,"kb_used_meta":1024830,"kb_avail":93343836,"statfs":{"total":96636764160,"available":95584088064,"internally_reserved":0,"allocated":3239936,"data_stored":2779219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1049426373},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575209,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":962780,"kb_used_data":17364,"kb_used_omap":1,"kb_used_meta":945406,"kb_avail":93409060,"statfs":{"total":96636764160,"available":95650877440,"internally_reserved":0,"allocated":17780736,"data_stored":17307749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":968096197},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2260992,"data_stored":2250634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8396800,"data_stored":8389280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":10645504,"data_stored":10639768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8404992,"data_stored":8389952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4292608,"data_stored":4283082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":88810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6979584,"data_stored":6970512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4296704,"data_stored":4279726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8400896,"data_stored":8389302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12587008,"data_stored":12583494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":6897664,"data_stored":6886504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T00:30:41.354 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T00:30:41.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.510+0000 7f0c973ac640 1 Processor -- start 2024-04-25T00:30:41.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.510+0000 7f0c973ac640 1 -- start start 2024-04-25T00:30:41.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.510+0000 7f0c973ac640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f0c900a94a0 0x7f0c901538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:41.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.510+0000 7f0c973ac640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0c900a99f0 0x7f0c90153e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:41.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.510+0000 7f0c973ac640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c900c3910 con 0x7f0c90150d20 2024-04-25T00:30:41.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.510+0000 7f0c973ac640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f0c900c16e0 con 0x7f0c900a94a0 2024-04-25T00:30:41.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.510+0000 7f0c973ac640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f0c900c1400 con 0x7f0c900a99f0 2024-04-25T00:30:41.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.510+0000 7f0c95148640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f0c90150d20 0x7f0c900c2a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57418/0 (socket says 172.21.15.116:57418) 2024-04-25T00:30:41.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.510+0000 7f0c95148640 1 -- 172.21.15.116:0/865912061 learned_addr learned my addr 172.21.15.116:0/865912061 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:30:41.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.510+0000 7f0c87fff640 1 -- 172.21.15.116:0/865912061 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1256925017 0 0) 0x7f0c900c3910 con 0x7f0c90150d20 2024-04-25T00:30:41.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.510+0000 7f0c87fff640 1 -- 172.21.15.116:0/865912061 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c7c003530 con 0x7f0c90150d20 2024-04-25T00:30:41.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.510+0000 7f0c94947640 1 --2- 172.21.15.116:0/865912061 >> v2:172.21.15.174:3300/0 conn(0x7f0c900a94a0 0x7f0c901538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:41.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.510+0000 7f0c95949640 1 --2- 172.21.15.116:0/865912061 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0c900a99f0 0x7f0c90153e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:41.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.510+0000 7f0c87fff640 1 -- 172.21.15.116:0/865912061 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2200888180 0 0) 0x7f0c7c003530 con 0x7f0c90150d20 2024-04-25T00:30:41.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.510+0000 7f0c87fff640 1 -- 172.21.15.116:0/865912061 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0c900a99f0 msgr2=0x7f0c90153e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:41.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.510+0000 7f0c87fff640 1 --2- 172.21.15.116:0/865912061 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0c900a99f0 0x7f0c90153e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:41.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.510+0000 7f0c87fff640 1 -- 172.21.15.116:0/865912061 >> v2:172.21.15.174:3300/0 conn(0x7f0c900a94a0 msgr2=0x7f0c901538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:41.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.510+0000 7f0c87fff640 1 --2- 172.21.15.116:0/865912061 >> v2:172.21.15.174:3300/0 conn(0x7f0c900a94a0 0x7f0c901538e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:41.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.510+0000 7f0c87fff640 1 -- 172.21.15.116:0/865912061 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c9006e660 con 0x7f0c90150d20 2024-04-25T00:30:41.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.510+0000 7f0c87fff640 1 -- 172.21.15.116:0/865912061 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f0c8c001680 con 0x7f0c90150d20 2024-04-25T00:30:41.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.514+0000 7f0c87fff640 1 -- 172.21.15.116:0/865912061 >> v1:172.21.15.116:6789/0 conn(0x7f0c90150d20 legacy=0x7f0c900c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:41.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.514+0000 7f0c87fff640 1 --2- 172.21.15.116:0/865912061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0c7c0039c0 0x7f0c7c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:41.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.514+0000 7f0c87fff640 1 --2- 172.21.15.116:0/865912061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c7c004810 0x7f0c7c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:41.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.514+0000 7f0c87fff640 1 --2- 172.21.15.116:0/865912061 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0c900a99f0 0x7f0c7c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:41.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.514+0000 7f0c87fff640 1 -- 172.21.15.116:0/865912061 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0c900c16e0 con 0x7f0c7c004810 2024-04-25T00:30:41.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.514+0000 7f0c87fff640 1 -- 172.21.15.116:0/865912061 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0c900c1400 con 0x7f0c7c0039c0 2024-04-25T00:30:41.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.514+0000 7f0c87fff640 1 -- 172.21.15.116:0/865912061 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f0c7c00ba80 con 0x7f0c900a99f0 2024-04-25T00:30:41.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.514+0000 7f0c94947640 1 --2- 172.21.15.116:0/865912061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c7c004810 0x7f0c7c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:41.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.514+0000 7f0c95148640 1 --2- 172.21.15.116:0/865912061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0c7c0039c0 0x7f0c7c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:41.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.514+0000 7f0c95949640 1 --2- 172.21.15.116:0/865912061 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0c900a99f0 0x7f0c7c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:41.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.514+0000 7f0c94947640 1 -- 172.21.15.116:0/865912061 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0c900a99f0 msgr2=0x7f0c7c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:41.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.514+0000 7f0c94947640 1 --2- 172.21.15.116:0/865912061 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0c900a99f0 0x7f0c7c0051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:41.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.514+0000 7f0c94947640 1 -- 172.21.15.116:0/865912061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0c7c0039c0 msgr2=0x7f0c7c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:41.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.514+0000 7f0c94947640 1 --2- 172.21.15.116:0/865912061 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0c7c0039c0 0x7f0c7c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:41.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.514+0000 7f0c94947640 1 -- 172.21.15.116:0/865912061 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c7c003530 con 0x7f0c7c004810 2024-04-25T00:30:41.520 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.514+0000 7f0c94947640 1 --2- 172.21.15.116:0/865912061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c7c004810 0x7f0c7c004c90 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f0c8000ba00 tx=0x7f0c8001c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44a5f8c746490b89 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:41.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.514+0000 7f0c87fff640 1 -- 172.21.15.116:0/865912061 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c8000b600 con 0x7f0c7c004810 2024-04-25T00:30:41.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.514+0000 7f0c87fff640 1 -- 172.21.15.116:0/865912061 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c800b1070 con 0x7f0c7c004810 2024-04-25T00:30:41.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.514+0000 7f0c87fff640 1 -- 172.21.15.116:0/865912061 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c800ac860 con 0x7f0c7c004810 2024-04-25T00:30:41.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.514+0000 7f0c973ac640 1 -- 172.21.15.116:0/865912061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c7c004810 msgr2=0x7f0c7c004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:41.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.514+0000 7f0c973ac640 1 --2- 172.21.15.116:0/865912061 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c7c004810 0x7f0c7c004c90 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f0c8000ba00 tx=0x7f0c8001c680 comp rx=0 tx=0).stop 2024-04-25T00:30:41.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.514+0000 7f0c95949640 1 -- 172.21.15.116:0/865912061 reap_dead start 2024-04-25T00:30:41.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.514+0000 7f0c973ac640 1 -- 172.21.15.116:0/865912061 shutdown_connections 2024-04-25T00:30:41.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.514+0000 7f0c973ac640 1 -- 172.21.15.116:0/865912061 >> 172.21.15.116:0/865912061 conn(0x7f0c900c3b00 msgr2=0x7f0c900c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:41.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.514+0000 7f0c973ac640 1 -- 172.21.15.116:0/865912061 shutdown_connections 2024-04-25T00:30:41.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.514+0000 7f0c973ac640 1 -- 172.21.15.116:0/865912061 wait complete. 2024-04-25T00:30:41.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.518+0000 7f0c973ac640 1 Processor -- start 2024-04-25T00:30:41.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.518+0000 7f0c973ac640 1 -- start start 2024-04-25T00:30:41.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.518+0000 7f0c973ac640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0c7c004810 0x7f0c90149120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:41.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.518+0000 7f0c973ac640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0c900c6940 0x7f0c9014ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:41.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.518+0000 7f0c973ac640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c900c5ea0 0x7f0c901e9810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:41.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.518+0000 7f0c973ac640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0c900c3120 con 0x7f0c900c5ea0 2024-04-25T00:30:41.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.518+0000 7f0c973ac640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0c9006ec20 con 0x7f0c900c6940 2024-04-25T00:30:41.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.518+0000 7f0c973ac640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f0c900c32a0 con 0x7f0c7c004810 2024-04-25T00:30:41.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.518+0000 7f0c95949640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c900c5ea0 0x7f0c901e9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:41.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.518+0000 7f0c95949640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c900c5ea0 0x7f0c901e9810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36474/0 (socket says 172.21.15.116:36474) 2024-04-25T00:30:41.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.518+0000 7f0c95949640 1 -- 172.21.15.116:0/619370899 learned_addr learned my addr 172.21.15.116:0/619370899 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:41.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.518+0000 7f0c94947640 1 --2- 172.21.15.116:0/619370899 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0c900c6940 0x7f0c9014ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:41.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.518+0000 7f0c95148640 1 --2- 172.21.15.116:0/619370899 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0c7c004810 0x7f0c90149120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:41.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.518+0000 7f0c95949640 1 -- 172.21.15.116:0/619370899 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0c7c004810 msgr2=0x7f0c90149120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:41.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.518+0000 7f0c95949640 1 --2- 172.21.15.116:0/619370899 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0c7c004810 0x7f0c90149120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:41.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.518+0000 7f0c95949640 1 -- 172.21.15.116:0/619370899 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0c900c6940 msgr2=0x7f0c9014ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:41.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.518+0000 7f0c95949640 1 --2- 172.21.15.116:0/619370899 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0c900c6940 0x7f0c9014ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:41.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.518+0000 7f0c95949640 1 -- 172.21.15.116:0/619370899 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c9014d580 con 0x7f0c900c5ea0 2024-04-25T00:30:41.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.518+0000 7f0c95949640 1 --2- 172.21.15.116:0/619370899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c900c5ea0 0x7f0c901e9810 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f0c90070890 tx=0x7f0c88009d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:41.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.518+0000 7f0c85ffb640 1 -- 172.21.15.116:0/619370899 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c88019070 con 0x7f0c900c5ea0 2024-04-25T00:30:41.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.518+0000 7f0c85ffb640 1 -- 172.21.15.116:0/619370899 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c88007cb0 con 0x7f0c900c5ea0 2024-04-25T00:30:41.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.518+0000 7f0c973ac640 1 -- 172.21.15.116:0/619370899 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c9014d860 con 0x7f0c900c5ea0 2024-04-25T00:30:41.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.518+0000 7f0c85ffb640 1 -- 172.21.15.116:0/619370899 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c88007e60 con 0x7f0c900c5ea0 2024-04-25T00:30:41.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.518+0000 7f0c973ac640 1 -- 172.21.15.116:0/619370899 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c9014ab70 con 0x7f0c900c5ea0 2024-04-25T00:30:41.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.518+0000 7f0c973ac640 1 -- 172.21.15.116:0/619370899 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c90068350 con 0x7f0c900c5ea0 2024-04-25T00:30:41.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.518+0000 7f0c85ffb640 1 -- 172.21.15.116:0/619370899 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f0c8801c070 con 0x7f0c900c5ea0 2024-04-25T00:30:41.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.522+0000 7f0c85ffb640 1 --2- 172.21.15.116:0/619370899 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0c740678c0 0x7f0c74069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:41.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.522+0000 7f0c85ffb640 1 -- 172.21.15.116:0/619370899 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7f0c88086920 con 0x7f0c900c5ea0 2024-04-25T00:30:41.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.522+0000 7f0c95148640 1 --2- 172.21.15.116:0/619370899 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0c740678c0 0x7f0c74069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:41.524 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.522+0000 7f0c95148640 1 --2- 172.21.15.116:0/619370899 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0c740678c0 0x7f0c74069d80 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c005360 tx=0x7f0c8c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:41.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.526+0000 7f0c85ffb640 1 -- 172.21.15.116:0/619370899 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f0c88055b20 con 0x7f0c900c5ea0 2024-04-25T00:30:41.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.670+0000 7f0c973ac640 1 -- 172.21.15.116:0/619370899 --> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0c9006c220 con 0x7f0c740678c0 2024-04-25T00:30:41.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.706+0000 7f0c85ffb640 1 -- 172.21.15.116:0/619370899 <== mgr.4105 v2:172.21.15.174:6832/2004448211 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+493789 (secure 0 0 0) 0x7f0c9006c220 con 0x7f0c740678c0 2024-04-25T00:30:41.707 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:30:41.713 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-04-25T00:30:41.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.710+0000 7f0c973ac640 1 -- 172.21.15.116:0/619370899 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0c740678c0 msgr2=0x7f0c74069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:41.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.710+0000 7f0c973ac640 1 --2- 172.21.15.116:0/619370899 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0c740678c0 0x7f0c74069d80 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c005360 tx=0x7f0c8c009040 comp rx=0 tx=0).stop 2024-04-25T00:30:41.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.710+0000 7f0c973ac640 1 -- 172.21.15.116:0/619370899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c900c5ea0 msgr2=0x7f0c901e9810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:41.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.710+0000 7f0c973ac640 1 --2- 172.21.15.116:0/619370899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c900c5ea0 0x7f0c901e9810 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f0c90070890 tx=0x7f0c88009d40 comp rx=0 tx=0).stop 2024-04-25T00:30:41.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.710+0000 7f0c973ac640 1 -- 172.21.15.116:0/619370899 shutdown_connections 2024-04-25T00:30:41.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.710+0000 7f0c973ac640 1 --2- 172.21.15.116:0/619370899 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0c740678c0 0x7f0c74069d80 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:41.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.710+0000 7f0c973ac640 1 --2- 172.21.15.116:0/619370899 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0c900c5ea0 0x7f0c901e9810 unknown :-1 s=CLOSED pgs=1801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:41.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.710+0000 7f0c973ac640 1 --2- 172.21.15.116:0/619370899 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0c900c6940 0x7f0c9014ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:41.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.710+0000 7f0c973ac640 1 --2- 172.21.15.116:0/619370899 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0c7c004810 0x7f0c90149120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:41.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.710+0000 7f0c973ac640 1 -- 172.21.15.116:0/619370899 >> 172.21.15.116:0/619370899 conn(0x7f0c900c3b00 msgr2=0x7f0c9006cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:41.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.710+0000 7f0c973ac640 1 -- 172.21.15.116:0/619370899 shutdown_connections 2024-04-25T00:30:41.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.710+0000 7f0c973ac640 1 -- 172.21.15.116:0/619370899 wait complete. 2024-04-25T00:30:41.731 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":962,"stamp":"2024-04-25T00:30:40.026992+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":39263178,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":3556,"num_read_kb":487326,"num_write":2966736,"num_write_kb":38127636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":551297,"ondisk_log_size":551297,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":7166992,"kb_used_data":84432,"kb_used_omap":12,"kb_used_meta":7082483,"kb_avail":747807728,"statfs":{"total":773094113280,"available":765755113472,"internally_reserved":0,"allocated":86458368,"data_stored":82754316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":7252463153},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-5424637476,"num_objects":-1401,"num_object_clones":0,"num_object_copies":-2802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1401,"num_whiteouts":0,"num_read":3,"num_read_kb":19,"num_write":1414,"num_write_kb":133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005794"},"pg_stats":[{"pgid":"5.18","version":"74'4491","reported_seq":4612,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.766890+0000","last_change":"2024-04-25T00:01:19.508728+0000","last_active":"2024-04-25T00:30:26.766890+0000","last_peered":"2024-04-25T00:30:26.766890+0000","last_clean":"2024-04-25T00:30:26.766890+0000","last_became_active":"2024-04-25T00:01:19.508390+0000","last_became_peered":"2024-04-25T00:01:19.508390+0000","last_unstale":"2024-04-25T00:30:26.766890+0000","last_undegraded":"2024-04-25T00:30:26.766890+0000","last_fullsized":"2024-04-25T00:30:26.766890+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4491,"log_dups_size":0,"ondisk_log_size":4491,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:45.591905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":12,"num_write":23917,"num_write_kb":321110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1f","version":"70'1087","reported_seq":1273,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.794182+0000","last_change":"2024-04-25T00:01:04.356032+0000","last_active":"2024-04-25T00:30:15.794182+0000","last_peered":"2024-04-25T00:30:15.794182+0000","last_clean":"2024-04-25T00:30:15.794182+0000","last_became_active":"2024-04-25T00:01:04.355293+0000","last_became_peered":"2024-04-25T00:01:04.355293+0000","last_unstale":"2024-04-25T00:30:15.794182+0000","last_undegraded":"2024-04-25T00:30:15.794182+0000","last_fullsized":"2024-04-25T00:30:15.794182+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1087,"log_dups_size":0,"ondisk_log_size":1087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:56:16.678132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":16397,"num_write":1061,"num_write_kb":4715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"70'3760","reported_seq":3888,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.801356+0000","last_change":"2024-04-25T00:01:05.367207+0000","last_active":"2024-04-25T00:30:15.801356+0000","last_peered":"2024-04-25T00:30:15.801356+0000","last_clean":"2024-04-25T00:30:15.801356+0000","last_became_active":"2024-04-25T00:01:05.366471+0000","last_became_peered":"2024-04-25T00:01:05.366471+0000","last_unstale":"2024-04-25T00:30:15.801356+0000","last_undegraded":"2024-04-25T00:30:15.801356+0000","last_fullsized":"2024-04-25T00:30:15.801356+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3760,"log_dups_size":0,"ondisk_log_size":3760,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:30:23.557381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21072,"num_write_kb":264901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"74'3","reported_seq":123,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.743279+0000","last_change":"2024-04-25T00:01:18.502364+0000","last_active":"2024-04-25T00:30:26.743279+0000","last_peered":"2024-04-25T00:30:26.743279+0000","last_clean":"2024-04-25T00:30:26.743279+0000","last_became_active":"2024-04-25T00:01:18.501633+0000","last_became_peered":"2024-04-25T00:01:18.501633+0000","last_unstale":"2024-04-25T00:30:26.743279+0000","last_undegraded":"2024-04-25T00:30:26.743279+0000","last_fullsized":"2024-04-25T00:30:26.743279+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:55.074018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"74'5031","reported_seq":5145,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.765069+0000","last_change":"2024-04-25T00:01:46.662019+0000","last_active":"2024-04-25T00:30:26.765069+0000","last_peered":"2024-04-25T00:30:26.765069+0000","last_clean":"2024-04-25T00:30:26.765069+0000","last_became_active":"2024-04-25T00:01:46.661167+0000","last_became_peered":"2024-04-25T00:01:46.661167+0000","last_unstale":"2024-04-25T00:30:26.765069+0000","last_undegraded":"2024-04-25T00:30:26.765069+0000","last_fullsized":"2024-04-25T00:30:26.765069+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":5031,"log_dups_size":0,"ondisk_log_size":5031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:13:34.153117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":22,"num_write":27250,"num_write_kb":338892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"73'28","reported_seq":158,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.802312+0000","last_change":"2024-04-25T00:01:04.361809+0000","last_active":"2024-04-25T00:30:15.802312+0000","last_peered":"2024-04-25T00:30:15.802312+0000","last_clean":"2024-04-25T00:30:15.802312+0000","last_became_active":"2024-04-25T00:01:04.361059+0000","last_became_peered":"2024-04-25T00:01:04.361059+0000","last_unstale":"2024-04-25T00:30:15.802312+0000","last_undegraded":"2024-04-25T00:30:15.802312+0000","last_fullsized":"2024-04-25T00:30:15.802312+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"70'5376","reported_seq":5504,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.799500+0000","last_change":"2024-04-25T00:01:05.370818+0000","last_active":"2024-04-25T00:30:15.799500+0000","last_peered":"2024-04-25T00:30:15.799500+0000","last_clean":"2024-04-25T00:30:15.799500+0000","last_became_active":"2024-04-25T00:01:05.369991+0000","last_became_peered":"2024-04-25T00:01:05.369991+0000","last_unstale":"2024-04-25T00:30:15.799500+0000","last_undegraded":"2024-04-25T00:30:15.799500+0000","last_fullsized":"2024-04-25T00:30:15.799500+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":5376,"log_dups_size":0,"ondisk_log_size":5376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:43.615895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29984,"num_write_kb":386541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"74'2","reported_seq":122,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:23.313841+0000","last_change":"2024-04-25T00:01:18.497682+0000","last_active":"2024-04-25T00:30:23.313841+0000","last_peered":"2024-04-25T00:30:23.313841+0000","last_clean":"2024-04-25T00:30:23.313841+0000","last_became_active":"2024-04-25T00:01:18.497397+0000","last_became_peered":"2024-04-25T00:01:18.497397+0000","last_unstale":"2024-04-25T00:30:23.313841+0000","last_undegraded":"2024-04-25T00:30:23.313841+0000","last_fullsized":"2024-04-25T00:30:23.313841+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:22.983972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"74'4080","reported_seq":4199,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.684096+0000","last_change":"2024-04-25T00:01:19.510591+0000","last_active":"2024-04-25T00:30:26.684096+0000","last_peered":"2024-04-25T00:30:26.684096+0000","last_clean":"2024-04-25T00:30:26.684096+0000","last_became_active":"2024-04-25T00:01:19.510252+0000","last_became_peered":"2024-04-25T00:01:19.510252+0000","last_unstale":"2024-04-25T00:30:26.684096+0000","last_undegraded":"2024-04-25T00:30:26.684096+0000","last_fullsized":"2024-04-25T00:30:26.684096+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4080,"log_dups_size":0,"ondisk_log_size":4080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:38.005233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":21789,"num_write_kb":296332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"74'53","reported_seq":203,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.793824+0000","last_change":"2024-04-25T00:01:04.362030+0000","last_active":"2024-04-25T00:30:15.793824+0000","last_peered":"2024-04-25T00:30:15.793824+0000","last_clean":"2024-04-25T00:30:15.793824+0000","last_became_active":"2024-04-25T00:01:04.361708+0000","last_became_peered":"2024-04-25T00:01:04.361708+0000","last_unstale":"2024-04-25T00:30:15.793824+0000","last_undegraded":"2024-04-25T00:30:15.793824+0000","last_fullsized":"2024-04-25T00:30:15.793824+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:38:33.392719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":69,"num_read_kb":69,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"69'3669","reported_seq":3794,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.942949+0000","last_change":"2024-04-25T00:01:05.387195+0000","last_active":"2024-04-25T00:30:15.942949+0000","last_peered":"2024-04-25T00:30:15.942949+0000","last_clean":"2024-04-25T00:30:15.942949+0000","last_became_active":"2024-04-25T00:01:05.386853+0000","last_became_peered":"2024-04-25T00:01:05.386853+0000","last_unstale":"2024-04-25T00:30:15.942949+0000","last_undegraded":"2024-04-25T00:30:15.942949+0000","last_fullsized":"2024-04-25T00:30:15.942949+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3669,"log_dups_size":0,"ondisk_log_size":3669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:06:19.307378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19904,"num_write_kb":264274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"74'723","reported_seq":869,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.740062+0000","last_change":"2024-04-25T00:01:18.487835+0000","last_active":"2024-04-25T00:30:26.740062+0000","last_peered":"2024-04-25T00:30:26.740062+0000","last_clean":"2024-04-25T00:30:26.740062+0000","last_became_active":"2024-04-25T00:01:18.487571+0000","last_became_peered":"2024-04-25T00:01:18.487571+0000","last_unstale":"2024-04-25T00:30:26.740062+0000","last_undegraded":"2024-04-25T00:30:26.740062+0000","last_fullsized":"2024-04-25T00:30:26.740062+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":723,"log_dups_size":0,"ondisk_log_size":723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:44.582148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":27,"num_read_kb":82727,"num_write":711,"num_write_kb":4310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"74'3586","reported_seq":3704,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.700653+0000","last_change":"2024-04-25T00:01:19.519834+0000","last_active":"2024-04-25T00:30:26.700653+0000","last_peered":"2024-04-25T00:30:26.700653+0000","last_clean":"2024-04-25T00:30:26.700653+0000","last_became_active":"2024-04-25T00:01:19.519470+0000","last_became_peered":"2024-04-25T00:01:19.519470+0000","last_unstale":"2024-04-25T00:30:26.700653+0000","last_undegraded":"2024-04-25T00:30:26.700653+0000","last_fullsized":"2024-04-25T00:30:26.700653+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3586,"log_dups_size":0,"ondisk_log_size":3586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:47.859626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":5,"num_write":18586,"num_write_kb":254663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"70'6","reported_seq":179,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.793266+0000","last_change":"2024-04-25T00:01:04.362832+0000","last_active":"2024-04-25T00:30:15.793266+0000","last_peered":"2024-04-25T00:30:15.793266+0000","last_clean":"2024-04-25T00:30:15.793266+0000","last_became_active":"2024-04-25T00:01:04.362059+0000","last_became_peered":"2024-04-25T00:01:04.362059+0000","last_unstale":"2024-04-25T00:30:15.793266+0000","last_undegraded":"2024-04-25T00:30:15.793266+0000","last_fullsized":"2024-04-25T00:30:15.793266+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:05.441048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"70'4973","reported_seq":5098,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.813402+0000","last_change":"2024-04-25T00:01:05.381120+0000","last_active":"2024-04-25T00:30:15.813402+0000","last_peered":"2024-04-25T00:30:15.813402+0000","last_clean":"2024-04-25T00:30:15.813402+0000","last_became_active":"2024-04-25T00:01:05.378320+0000","last_became_peered":"2024-04-25T00:01:05.378320+0000","last_unstale":"2024-04-25T00:30:15.813402+0000","last_undegraded":"2024-04-25T00:30:15.813402+0000","last_fullsized":"2024-04-25T00:30:15.813402+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4973,"log_dups_size":0,"ondisk_log_size":4973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:34:06.175918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27225,"num_write_kb":350946,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"74'355","reported_seq":476,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:31.515142+0000","last_change":"2024-04-25T00:01:18.490300+0000","last_active":"2024-04-25T00:30:31.515142+0000","last_peered":"2024-04-25T00:30:31.515142+0000","last_clean":"2024-04-25T00:30:31.515142+0000","last_became_active":"2024-04-25T00:01:18.490050+0000","last_became_peered":"2024-04-25T00:01:18.490050+0000","last_unstale":"2024-04-25T00:30:31.515142+0000","last_undegraded":"2024-04-25T00:30:31.515142+0000","last_fullsized":"2024-04-25T00:30:31.515142+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":355,"log_dups_size":0,"ondisk_log_size":355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:13:05.971606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2691528,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":907,"num_write":326,"num_write_kb":2825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"74'4320","reported_seq":4442,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.772385+0000","last_change":"2024-04-25T00:01:19.500026+0000","last_active":"2024-04-25T00:30:26.772385+0000","last_peered":"2024-04-25T00:30:26.772385+0000","last_clean":"2024-04-25T00:30:26.772385+0000","last_became_active":"2024-04-25T00:01:19.499244+0000","last_became_peered":"2024-04-25T00:01:19.499244+0000","last_unstale":"2024-04-25T00:30:26.772385+0000","last_undegraded":"2024-04-25T00:30:26.772385+0000","last_fullsized":"2024-04-25T00:30:26.772385+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4320,"log_dups_size":0,"ondisk_log_size":4320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:40:36.350832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":15,"num_write":23810,"num_write_kb":308534,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"70'4158","reported_seq":4285,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.809005+0000","last_change":"2024-04-25T00:01:05.376984+0000","last_active":"2024-04-25T00:30:15.809005+0000","last_peered":"2024-04-25T00:30:15.809005+0000","last_clean":"2024-04-25T00:30:15.809005+0000","last_became_active":"2024-04-25T00:01:05.376259+0000","last_became_peered":"2024-04-25T00:01:05.376259+0000","last_unstale":"2024-04-25T00:30:15.809005+0000","last_undegraded":"2024-04-25T00:30:15.809005+0000","last_fullsized":"2024-04-25T00:30:15.809005+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4158,"log_dups_size":0,"ondisk_log_size":4158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:13:28.543302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23006,"num_write_kb":301346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"74'1526","reported_seq":1710,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:23.032874+0000","last_change":"2024-04-25T00:01:04.359262+0000","last_active":"2024-04-25T00:30:23.032874+0000","last_peered":"2024-04-25T00:30:23.032874+0000","last_clean":"2024-04-25T00:30:23.032874+0000","last_became_active":"2024-04-25T00:01:04.357971+0000","last_became_peered":"2024-04-25T00:01:04.357971+0000","last_unstale":"2024-04-25T00:30:23.032874+0000","last_undegraded":"2024-04-25T00:30:23.032874+0000","last_fullsized":"2024-04-25T00:30:23.032874+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1526,"log_dups_size":0,"ondisk_log_size":1526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:05.441048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":78,"num_read_kb":52062,"num_write":1499,"num_write_kb":4525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"74'6","reported_seq":150,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.768405+0000","last_change":"2024-04-25T00:01:18.475624+0000","last_active":"2024-04-25T00:30:26.768405+0000","last_peered":"2024-04-25T00:30:26.768405+0000","last_clean":"2024-04-25T00:30:26.768405+0000","last_became_active":"2024-04-25T00:01:18.475149+0000","last_became_peered":"2024-04-25T00:01:18.475149+0000","last_unstale":"2024-04-25T00:30:26.768405+0000","last_undegraded":"2024-04-25T00:30:26.768405+0000","last_fullsized":"2024-04-25T00:30:26.768405+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:40:35.341075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"74'4556","reported_seq":4678,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.774727+0000","last_change":"2024-04-25T00:01:19.531113+0000","last_active":"2024-04-25T00:30:26.774727+0000","last_peered":"2024-04-25T00:30:26.774727+0000","last_clean":"2024-04-25T00:30:26.774727+0000","last_became_active":"2024-04-25T00:01:19.520150+0000","last_became_peered":"2024-04-25T00:01:19.520150+0000","last_unstale":"2024-04-25T00:30:26.774727+0000","last_undegraded":"2024-04-25T00:30:26.774727+0000","last_fullsized":"2024-04-25T00:30:26.774727+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4556,"log_dups_size":0,"ondisk_log_size":4556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:38.152376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":19,"num_write":25634,"num_write_kb":315935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"70'3060","reported_seq":3187,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.803143+0000","last_change":"2024-04-25T00:01:05.362611+0000","last_active":"2024-04-25T00:30:15.803143+0000","last_peered":"2024-04-25T00:30:15.803143+0000","last_clean":"2024-04-25T00:30:15.803143+0000","last_became_active":"2024-04-25T00:01:05.361817+0000","last_became_peered":"2024-04-25T00:01:05.361817+0000","last_unstale":"2024-04-25T00:30:15.803143+0000","last_undegraded":"2024-04-25T00:30:15.803143+0000","last_fullsized":"2024-04-25T00:30:15.803143+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3060,"log_dups_size":0,"ondisk_log_size":3060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:50.489125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16241,"num_write_kb":228970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"70'3","reported_seq":132,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.810550+0000","last_change":"2024-04-25T00:01:04.371935+0000","last_active":"2024-04-25T00:30:15.810550+0000","last_peered":"2024-04-25T00:30:15.810550+0000","last_clean":"2024-04-25T00:30:15.810550+0000","last_became_active":"2024-04-25T00:01:04.371420+0000","last_became_peered":"2024-04-25T00:01:04.371420+0000","last_unstale":"2024-04-25T00:30:15.810550+0000","last_undegraded":"2024-04-25T00:30:15.810550+0000","last_fullsized":"2024-04-25T00:30:15.810550+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"74'5","reported_seq":126,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.747597+0000","last_change":"2024-04-25T00:01:18.488623+0000","last_active":"2024-04-25T00:30:26.747597+0000","last_peered":"2024-04-25T00:30:26.747597+0000","last_clean":"2024-04-25T00:30:26.747597+0000","last_became_active":"2024-04-25T00:01:18.488291+0000","last_became_peered":"2024-04-25T00:01:18.488291+0000","last_unstale":"2024-04-25T00:30:26.747597+0000","last_undegraded":"2024-04-25T00:30:26.747597+0000","last_fullsized":"2024-04-25T00:30:26.747597+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:30:21.436681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"74'3591","reported_seq":3711,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.734674+0000","last_change":"2024-04-25T00:01:19.516731+0000","last_active":"2024-04-25T00:30:26.734674+0000","last_peered":"2024-04-25T00:30:26.734674+0000","last_clean":"2024-04-25T00:30:26.734674+0000","last_became_active":"2024-04-25T00:01:19.516405+0000","last_became_peered":"2024-04-25T00:01:19.516405+0000","last_unstale":"2024-04-25T00:30:26.734674+0000","last_undegraded":"2024-04-25T00:30:26.734674+0000","last_fullsized":"2024-04-25T00:30:26.734674+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3591,"log_dups_size":0,"ondisk_log_size":3591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:44:37.936092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":19584,"num_write_kb":250628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"69'3777","reported_seq":3904,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.790888+0000","last_change":"2024-04-25T00:01:05.355455+0000","last_active":"2024-04-25T00:30:15.790888+0000","last_peered":"2024-04-25T00:30:15.790888+0000","last_clean":"2024-04-25T00:30:15.790888+0000","last_became_active":"2024-04-25T00:01:05.354699+0000","last_became_peered":"2024-04-25T00:01:05.354699+0000","last_unstale":"2024-04-25T00:30:15.790888+0000","last_undegraded":"2024-04-25T00:30:15.790888+0000","last_fullsized":"2024-04-25T00:30:15.790888+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3777,"log_dups_size":0,"ondisk_log_size":3777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:43:58.069271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20993,"num_write_kb":256390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"69'2","reported_seq":131,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.791647+0000","last_change":"2024-04-25T00:01:04.362268+0000","last_active":"2024-04-25T00:30:15.791647+0000","last_peered":"2024-04-25T00:30:15.791647+0000","last_clean":"2024-04-25T00:30:15.791647+0000","last_became_active":"2024-04-25T00:01:04.361622+0000","last_became_peered":"2024-04-25T00:01:04.361622+0000","last_unstale":"2024-04-25T00:30:15.791647+0000","last_undegraded":"2024-04-25T00:30:15.791647+0000","last_fullsized":"2024-04-25T00:30:15.791647+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:58:46.021343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"74'4","reported_seq":147,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.744595+0000","last_change":"2024-04-25T00:01:18.494047+0000","last_active":"2024-04-25T00:30:26.744595+0000","last_peered":"2024-04-25T00:30:26.744595+0000","last_clean":"2024-04-25T00:30:26.744595+0000","last_became_active":"2024-04-25T00:01:18.493614+0000","last_became_peered":"2024-04-25T00:01:18.493614+0000","last_unstale":"2024-04-25T00:30:26.744595+0000","last_undegraded":"2024-04-25T00:30:26.744595+0000","last_fullsized":"2024-04-25T00:30:26.744595+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:37:08.562016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"74'4895","reported_seq":5013,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.674895+0000","last_change":"2024-04-25T00:01:19.505397+0000","last_active":"2024-04-25T00:30:26.674895+0000","last_peered":"2024-04-25T00:30:26.674895+0000","last_clean":"2024-04-25T00:30:26.674895+0000","last_became_active":"2024-04-25T00:01:19.505126+0000","last_became_peered":"2024-04-25T00:01:19.505126+0000","last_unstale":"2024-04-25T00:30:26.674895+0000","last_undegraded":"2024-04-25T00:30:26.674895+0000","last_fullsized":"2024-04-25T00:30:26.674895+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4895,"log_dups_size":0,"ondisk_log_size":4895,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:08:49.640267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26799,"num_write_kb":349350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"69'3934","reported_seq":4061,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.805653+0000","last_change":"2024-04-25T00:01:05.354953+0000","last_active":"2024-04-25T00:30:15.805653+0000","last_peered":"2024-04-25T00:30:15.805653+0000","last_clean":"2024-04-25T00:30:15.805653+0000","last_became_active":"2024-04-25T00:01:05.354118+0000","last_became_peered":"2024-04-25T00:01:05.354118+0000","last_unstale":"2024-04-25T00:30:15.805653+0000","last_undegraded":"2024-04-25T00:30:15.805653+0000","last_fullsized":"2024-04-25T00:30:15.805653+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3934,"log_dups_size":0,"ondisk_log_size":3934,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:09.858617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21059,"num_write_kb":273031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"70'6","reported_seq":135,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.789548+0000","last_change":"2024-04-25T00:01:04.365515+0000","last_active":"2024-04-25T00:30:15.789548+0000","last_peered":"2024-04-25T00:30:15.789548+0000","last_clean":"2024-04-25T00:30:15.789548+0000","last_became_active":"2024-04-25T00:01:04.364942+0000","last_became_peered":"2024-04-25T00:01:04.364942+0000","last_unstale":"2024-04-25T00:30:15.789548+0000","last_undegraded":"2024-04-25T00:30:15.789548+0000","last_fullsized":"2024-04-25T00:30:15.789548+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"74'7","reported_seq":174,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.741808+0000","last_change":"2024-04-25T00:01:18.476207+0000","last_active":"2024-04-25T00:30:26.741808+0000","last_peered":"2024-04-25T00:30:26.741808+0000","last_clean":"2024-04-25T00:30:26.741808+0000","last_became_active":"2024-04-25T00:01:18.475862+0000","last_became_peered":"2024-04-25T00:01:18.475862+0000","last_unstale":"2024-04-25T00:30:26.741808+0000","last_undegraded":"2024-04-25T00:30:26.741808+0000","last_fullsized":"2024-04-25T00:30:26.741808+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:11:40.170249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"74'3090","reported_seq":3210,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.776689+0000","last_change":"2024-04-25T00:01:19.529607+0000","last_active":"2024-04-25T00:30:26.776689+0000","last_peered":"2024-04-25T00:30:26.776689+0000","last_clean":"2024-04-25T00:30:26.776689+0000","last_became_active":"2024-04-25T00:01:19.528509+0000","last_became_peered":"2024-04-25T00:01:19.528509+0000","last_unstale":"2024-04-25T00:30:26.776689+0000","last_undegraded":"2024-04-25T00:30:26.776689+0000","last_fullsized":"2024-04-25T00:30:26.776689+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3090,"log_dups_size":0,"ondisk_log_size":3090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:45:01.068091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17043,"num_write_kb":228060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"70'4336","reported_seq":4463,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.793683+0000","last_change":"2024-04-25T00:01:05.358106+0000","last_active":"2024-04-25T00:30:15.793683+0000","last_peered":"2024-04-25T00:30:15.793683+0000","last_clean":"2024-04-25T00:30:15.793683+0000","last_became_active":"2024-04-25T00:01:05.357135+0000","last_became_peered":"2024-04-25T00:01:05.357135+0000","last_unstale":"2024-04-25T00:30:15.793683+0000","last_undegraded":"2024-04-25T00:30:15.793683+0000","last_fullsized":"2024-04-25T00:30:15.793683+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4336,"log_dups_size":0,"ondisk_log_size":4336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:41:42.228234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23030,"num_write_kb":325887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"71'9","reported_seq":184,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.814467+0000","last_change":"2024-04-25T00:01:04.369917+0000","last_active":"2024-04-25T00:30:15.814467+0000","last_peered":"2024-04-25T00:30:15.814467+0000","last_clean":"2024-04-25T00:30:15.814467+0000","last_became_active":"2024-04-25T00:01:04.369309+0000","last_became_peered":"2024-04-25T00:01:04.369309+0000","last_unstale":"2024-04-25T00:30:15.814467+0000","last_undegraded":"2024-04-25T00:30:15.814467+0000","last_fullsized":"2024-04-25T00:30:15.814467+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:55:08.063965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":8,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"74'51","reported_seq":241,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:28.848602+0000","last_change":"2024-04-25T00:01:18.477189+0000","last_active":"2024-04-25T00:30:28.848602+0000","last_peered":"2024-04-25T00:30:28.848602+0000","last_clean":"2024-04-25T00:30:28.848602+0000","last_became_active":"2024-04-25T00:01:18.476909+0000","last_became_peered":"2024-04-25T00:01:18.476909+0000","last_unstale":"2024-04-25T00:30:28.848602+0000","last_undegraded":"2024-04-25T00:30:28.848602+0000","last_fullsized":"2024-04-25T00:30:28.848602+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:36.995477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84646,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":68,"num_read_kb":112,"num_write":46,"num_write_kb":112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"74'3821","reported_seq":3942,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.704068+0000","last_change":"2024-04-25T00:01:19.517500+0000","last_active":"2024-04-25T00:30:26.704068+0000","last_peered":"2024-04-25T00:30:26.704068+0000","last_clean":"2024-04-25T00:30:26.704068+0000","last_became_active":"2024-04-25T00:01:19.517182+0000","last_became_peered":"2024-04-25T00:01:19.517182+0000","last_unstale":"2024-04-25T00:30:26.704068+0000","last_undegraded":"2024-04-25T00:30:26.704068+0000","last_fullsized":"2024-04-25T00:30:26.704068+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3821,"log_dups_size":0,"ondisk_log_size":3821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:58:49.738233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20929,"num_write_kb":256020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"70'3227","reported_seq":3354,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.793834+0000","last_change":"2024-04-25T00:01:05.364829+0000","last_active":"2024-04-25T00:30:15.793834+0000","last_peered":"2024-04-25T00:30:15.793834+0000","last_clean":"2024-04-25T00:30:15.793834+0000","last_became_active":"2024-04-25T00:01:05.364125+0000","last_became_peered":"2024-04-25T00:01:05.364125+0000","last_unstale":"2024-04-25T00:30:15.793834+0000","last_undegraded":"2024-04-25T00:30:15.793834+0000","last_fullsized":"2024-04-25T00:30:15.793834+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3227,"log_dups_size":0,"ondisk_log_size":3227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:15:45.531760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17808,"num_write_kb":231231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"74'458","reported_seq":610,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:25.796746+0000","last_change":"2024-04-25T00:01:04.361603+0000","last_active":"2024-04-25T00:30:25.796746+0000","last_peered":"2024-04-25T00:30:25.796746+0000","last_clean":"2024-04-25T00:30:25.796746+0000","last_became_active":"2024-04-25T00:01:04.361046+0000","last_became_peered":"2024-04-25T00:01:04.361046+0000","last_unstale":"2024-04-25T00:30:25.796746+0000","last_undegraded":"2024-04-25T00:30:25.796746+0000","last_fullsized":"2024-04-25T00:30:25.796746+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":458,"log_dups_size":0,"ondisk_log_size":458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:35:18.036843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":46,"num_read_kb":63,"num_write":1069,"num_write_kb":360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"74'12","reported_seq":134,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.743234+0000","last_change":"2024-04-25T00:01:18.488352+0000","last_active":"2024-04-25T00:30:26.743234+0000","last_peered":"2024-04-25T00:30:26.743234+0000","last_clean":"2024-04-25T00:30:26.743234+0000","last_became_active":"2024-04-25T00:01:18.487321+0000","last_became_peered":"2024-04-25T00:01:18.487321+0000","last_unstale":"2024-04-25T00:30:26.743234+0000","last_undegraded":"2024-04-25T00:30:26.743234+0000","last_fullsized":"2024-04-25T00:30:26.743234+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:37:22.329655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"74'4221","reported_seq":4341,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.775265+0000","last_change":"2024-04-25T00:01:19.519653+0000","last_active":"2024-04-25T00:30:26.775265+0000","last_peered":"2024-04-25T00:30:26.775265+0000","last_clean":"2024-04-25T00:30:26.775265+0000","last_became_active":"2024-04-25T00:01:19.519052+0000","last_became_peered":"2024-04-25T00:01:19.519052+0000","last_unstale":"2024-04-25T00:30:26.775265+0000","last_undegraded":"2024-04-25T00:30:26.775265+0000","last_fullsized":"2024-04-25T00:30:26.775265+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4221,"log_dups_size":0,"ondisk_log_size":4221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:40:36.350832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22937,"num_write_kb":280510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"70'5174","reported_seq":5302,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.813035+0000","last_change":"2024-04-25T00:01:05.374011+0000","last_active":"2024-04-25T00:30:15.813035+0000","last_peered":"2024-04-25T00:30:15.813035+0000","last_clean":"2024-04-25T00:30:15.813035+0000","last_became_active":"2024-04-25T00:01:05.373276+0000","last_became_peered":"2024-04-25T00:01:05.373276+0000","last_unstale":"2024-04-25T00:30:15.813035+0000","last_undegraded":"2024-04-25T00:30:15.813035+0000","last_fullsized":"2024-04-25T00:30:15.813035+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":5174,"log_dups_size":0,"ondisk_log_size":5174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:40:53.349428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28022,"num_write_kb":362384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"69'2","reported_seq":132,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.814680+0000","last_change":"2024-04-25T00:01:04.365391+0000","last_active":"2024-04-25T00:30:15.814680+0000","last_peered":"2024-04-25T00:30:15.814680+0000","last_clean":"2024-04-25T00:30:15.814680+0000","last_became_active":"2024-04-25T00:01:04.364989+0000","last_became_peered":"2024-04-25T00:01:04.364989+0000","last_unstale":"2024-04-25T00:30:15.814680+0000","last_undegraded":"2024-04-25T00:30:15.814680+0000","last_fullsized":"2024-04-25T00:30:15.814680+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:06:18.298057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"74'3","reported_seq":125,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.737749+0000","last_change":"2024-04-25T00:01:18.491597+0000","last_active":"2024-04-25T00:30:26.737749+0000","last_peered":"2024-04-25T00:30:26.737749+0000","last_clean":"2024-04-25T00:30:26.737749+0000","last_became_active":"2024-04-25T00:01:18.490859+0000","last_became_peered":"2024-04-25T00:01:18.490859+0000","last_unstale":"2024-04-25T00:30:26.737749+0000","last_undegraded":"2024-04-25T00:30:26.737749+0000","last_fullsized":"2024-04-25T00:30:26.737749+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:36.995477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"74'5316","reported_seq":5429,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.548329+0000","last_change":"2024-04-25T00:01:46.667934+0000","last_active":"2024-04-25T00:30:26.548329+0000","last_peered":"2024-04-25T00:30:26.548329+0000","last_clean":"2024-04-25T00:30:26.548329+0000","last_became_active":"2024-04-25T00:01:46.667075+0000","last_became_peered":"2024-04-25T00:01:46.667075+0000","last_unstale":"2024-04-25T00:30:26.548329+0000","last_undegraded":"2024-04-25T00:30:26.548329+0000","last_fullsized":"2024-04-25T00:30:26.548329+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":5316,"log_dups_size":0,"ondisk_log_size":5316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:28:18.234546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29390,"num_write_kb":363802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"70'3961","reported_seq":4089,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.804843+0000","last_change":"2024-04-25T00:01:05.378340+0000","last_active":"2024-04-25T00:30:15.804843+0000","last_peered":"2024-04-25T00:30:15.804843+0000","last_clean":"2024-04-25T00:30:15.804843+0000","last_became_active":"2024-04-25T00:01:05.377593+0000","last_became_peered":"2024-04-25T00:01:05.377593+0000","last_unstale":"2024-04-25T00:30:15.804843+0000","last_undegraded":"2024-04-25T00:30:15.804843+0000","last_fullsized":"2024-04-25T00:30:15.804843+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3961,"log_dups_size":0,"ondisk_log_size":3961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:48.202251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21237,"num_write_kb":269574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"70'2","reported_seq":132,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.799354+0000","last_change":"2024-04-25T00:01:04.366577+0000","last_active":"2024-04-25T00:30:15.799354+0000","last_peered":"2024-04-25T00:30:15.799354+0000","last_clean":"2024-04-25T00:30:15.799354+0000","last_became_active":"2024-04-25T00:01:04.365881+0000","last_became_peered":"2024-04-25T00:01:04.365881+0000","last_unstale":"2024-04-25T00:30:15.799354+0000","last_undegraded":"2024-04-25T00:30:15.799354+0000","last_fullsized":"2024-04-25T00:30:15.799354+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:49:22.673421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'4","reported_seq":126,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.132180+0000","last_change":"2024-04-25T00:01:18.504341+0000","last_active":"2024-04-25T00:30:26.132180+0000","last_peered":"2024-04-25T00:30:26.132180+0000","last_clean":"2024-04-25T00:30:26.132180+0000","last_became_active":"2024-04-25T00:01:18.491498+0000","last_became_peered":"2024-04-25T00:01:18.491498+0000","last_unstale":"2024-04-25T00:30:26.132180+0000","last_undegraded":"2024-04-25T00:30:26.132180+0000","last_fullsized":"2024-04-25T00:30:26.132180+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:56.741250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"74'4156","reported_seq":4283,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.774804+0000","last_change":"2024-04-25T00:01:46.667022+0000","last_active":"2024-04-25T00:30:26.774804+0000","last_peered":"2024-04-25T00:30:26.774804+0000","last_clean":"2024-04-25T00:30:26.774804+0000","last_became_active":"2024-04-25T00:01:46.666224+0000","last_became_peered":"2024-04-25T00:01:46.666224+0000","last_unstale":"2024-04-25T00:30:26.774804+0000","last_undegraded":"2024-04-25T00:30:26.774804+0000","last_fullsized":"2024-04-25T00:30:26.774804+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4156,"log_dups_size":0,"ondisk_log_size":4156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:25:20.185355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":7,"num_write":23008,"num_write_kb":280758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"69'4491","reported_seq":4619,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.793150+0000","last_change":"2024-04-25T00:01:05.368769+0000","last_active":"2024-04-25T00:30:15.793150+0000","last_peered":"2024-04-25T00:30:15.793150+0000","last_clean":"2024-04-25T00:30:15.793150+0000","last_became_active":"2024-04-25T00:01:05.366817+0000","last_became_peered":"2024-04-25T00:01:05.366817+0000","last_unstale":"2024-04-25T00:30:15.793150+0000","last_undegraded":"2024-04-25T00:30:15.793150+0000","last_fullsized":"2024-04-25T00:30:15.793150+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4491,"log_dups_size":0,"ondisk_log_size":4491,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:11:27.044893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24298,"num_write_kb":330898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"69'5","reported_seq":135,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.810345+0000","last_change":"2024-04-25T00:01:04.361900+0000","last_active":"2024-04-25T00:30:15.810345+0000","last_peered":"2024-04-25T00:30:15.810345+0000","last_clean":"2024-04-25T00:30:15.810345+0000","last_became_active":"2024-04-25T00:01:04.361572+0000","last_became_peered":"2024-04-25T00:01:04.361572+0000","last_unstale":"2024-04-25T00:30:15.810345+0000","last_undegraded":"2024-04-25T00:30:15.810345+0000","last_fullsized":"2024-04-25T00:30:15.810345+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:29.218739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"74'2","reported_seq":124,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.744419+0000","last_change":"2024-04-25T00:01:18.504139+0000","last_active":"2024-04-25T00:30:26.744419+0000","last_peered":"2024-04-25T00:30:26.744419+0000","last_clean":"2024-04-25T00:30:26.744419+0000","last_became_active":"2024-04-25T00:01:18.503379+0000","last_became_peered":"2024-04-25T00:01:18.503379+0000","last_unstale":"2024-04-25T00:30:26.744419+0000","last_undegraded":"2024-04-25T00:30:26.744419+0000","last_fullsized":"2024-04-25T00:30:26.744419+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:02:56.363258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"74'5402","reported_seq":5525,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.707463+0000","last_change":"2024-04-25T00:01:19.531756+0000","last_active":"2024-04-25T00:30:26.707463+0000","last_peered":"2024-04-25T00:30:26.707463+0000","last_clean":"2024-04-25T00:30:26.707463+0000","last_became_active":"2024-04-25T00:01:19.531145+0000","last_became_peered":"2024-04-25T00:01:19.531145+0000","last_unstale":"2024-04-25T00:30:26.707463+0000","last_undegraded":"2024-04-25T00:30:26.707463+0000","last_fullsized":"2024-04-25T00:30:26.707463+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":5402,"log_dups_size":0,"ondisk_log_size":5402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:13:42.678415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":13,"num_write":29532,"num_write_kb":356063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"70'3335","reported_seq":3463,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.794572+0000","last_change":"2024-04-25T00:01:05.354557+0000","last_active":"2024-04-25T00:30:15.794572+0000","last_peered":"2024-04-25T00:30:15.794572+0000","last_clean":"2024-04-25T00:30:15.794572+0000","last_became_active":"2024-04-25T00:01:05.353556+0000","last_became_peered":"2024-04-25T00:01:05.353556+0000","last_unstale":"2024-04-25T00:30:15.794572+0000","last_undegraded":"2024-04-25T00:30:15.794572+0000","last_fullsized":"2024-04-25T00:30:15.794572+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3335,"log_dups_size":0,"ondisk_log_size":3335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:24.964055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18377,"num_write_kb":241696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"74'53","reported_seq":252,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:25.820987+0000","last_change":"2024-04-25T00:01:04.372916+0000","last_active":"2024-04-25T00:30:25.820987+0000","last_peered":"2024-04-25T00:30:25.820987+0000","last_clean":"2024-04-25T00:30:25.820987+0000","last_became_active":"2024-04-25T00:01:04.372185+0000","last_became_peered":"2024-04-25T00:01:04.372185+0000","last_unstale":"2024-04-25T00:30:25.820987+0000","last_undegraded":"2024-04-25T00:30:25.820987+0000","last_fullsized":"2024-04-25T00:30:25.820987+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88686,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":69,"num_read_kb":117,"num_write":46,"num_write_kb":117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"74'9","reported_seq":176,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.750624+0000","last_change":"2024-04-25T00:01:18.504614+0000","last_active":"2024-04-25T00:30:26.750624+0000","last_peered":"2024-04-25T00:30:26.750624+0000","last_clean":"2024-04-25T00:30:26.750624+0000","last_became_active":"2024-04-25T00:01:18.501448+0000","last_became_peered":"2024-04-25T00:01:18.501448+0000","last_unstale":"2024-04-25T00:30:26.750624+0000","last_undegraded":"2024-04-25T00:30:26.750624+0000","last_fullsized":"2024-04-25T00:30:26.750624+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:22.983972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":8,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"74'3337","reported_seq":3456,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.763497+0000","last_change":"2024-04-25T00:01:19.517510+0000","last_active":"2024-04-25T00:30:26.763497+0000","last_peered":"2024-04-25T00:30:26.763497+0000","last_clean":"2024-04-25T00:30:26.763497+0000","last_became_active":"2024-04-25T00:01:19.517243+0000","last_became_peered":"2024-04-25T00:01:19.517243+0000","last_unstale":"2024-04-25T00:30:26.763497+0000","last_undegraded":"2024-04-25T00:30:26.763497+0000","last_fullsized":"2024-04-25T00:30:26.763497+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3337,"log_dups_size":0,"ondisk_log_size":3337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:42:38.149502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18533,"num_write_kb":242287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"69'3723","reported_seq":3852,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.809022+0000","last_change":"2024-04-25T00:01:05.370968+0000","last_active":"2024-04-25T00:30:15.809022+0000","last_peered":"2024-04-25T00:30:15.809022+0000","last_clean":"2024-04-25T00:30:15.809022+0000","last_became_active":"2024-04-25T00:01:05.369987+0000","last_became_peered":"2024-04-25T00:01:05.369987+0000","last_unstale":"2024-04-25T00:30:15.809022+0000","last_undegraded":"2024-04-25T00:30:15.809022+0000","last_fullsized":"2024-04-25T00:30:15.809022+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3723,"log_dups_size":0,"ondisk_log_size":3723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20208,"num_write_kb":269376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"73'13","reported_seq":143,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.804977+0000","last_change":"2024-04-25T00:01:04.370226+0000","last_active":"2024-04-25T00:30:15.804977+0000","last_peered":"2024-04-25T00:30:15.804977+0000","last_clean":"2024-04-25T00:30:15.804977+0000","last_became_active":"2024-04-25T00:01:04.369904+0000","last_became_peered":"2024-04-25T00:01:04.369904+0000","last_unstale":"2024-04-25T00:30:15.804977+0000","last_undegraded":"2024-04-25T00:30:15.804977+0000","last_fullsized":"2024-04-25T00:30:15.804977+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:47:25.595275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"74'487","reported_seq":631,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.794202+0000","last_change":"2024-04-25T00:01:18.497156+0000","last_active":"2024-04-25T00:30:26.794202+0000","last_peered":"2024-04-25T00:30:26.794202+0000","last_clean":"2024-04-25T00:30:26.794202+0000","last_became_active":"2024-04-25T00:01:18.495573+0000","last_became_peered":"2024-04-25T00:01:18.495573+0000","last_unstale":"2024-04-25T00:30:26.794202+0000","last_undegraded":"2024-04-25T00:30:26.794202+0000","last_fullsized":"2024-04-25T00:30:26.794202+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":487,"log_dups_size":0,"ondisk_log_size":487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:29.119299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":46,"num_read_kb":69,"num_write":1147,"num_write_kb":382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"74'5079","reported_seq":5200,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.738941+0000","last_change":"2024-04-25T00:01:19.519411+0000","last_active":"2024-04-25T00:30:26.738941+0000","last_peered":"2024-04-25T00:30:26.738941+0000","last_clean":"2024-04-25T00:30:26.738941+0000","last_became_active":"2024-04-25T00:01:19.519020+0000","last_became_peered":"2024-04-25T00:01:19.519020+0000","last_unstale":"2024-04-25T00:30:26.738941+0000","last_undegraded":"2024-04-25T00:30:26.738941+0000","last_fullsized":"2024-04-25T00:30:26.738941+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":5079,"log_dups_size":0,"ondisk_log_size":5079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:50.351599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":14,"num_write":27774,"num_write_kb":353080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"70'3676","reported_seq":3804,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.793841+0000","last_change":"2024-04-25T00:01:05.364939+0000","last_active":"2024-04-25T00:30:15.793841+0000","last_peered":"2024-04-25T00:30:15.793841+0000","last_clean":"2024-04-25T00:30:15.793841+0000","last_became_active":"2024-04-25T00:01:05.364272+0000","last_became_peered":"2024-04-25T00:01:05.364272+0000","last_unstale":"2024-04-25T00:30:15.793841+0000","last_undegraded":"2024-04-25T00:30:15.793841+0000","last_fullsized":"2024-04-25T00:30:15.793841+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3676,"log_dups_size":0,"ondisk_log_size":3676,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:24:59.679908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19925,"num_write_kb":256146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"69'3","reported_seq":133,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.809642+0000","last_change":"2024-04-25T00:01:04.357028+0000","last_active":"2024-04-25T00:30:15.809642+0000","last_peered":"2024-04-25T00:30:15.809642+0000","last_clean":"2024-04-25T00:30:15.809642+0000","last_became_active":"2024-04-25T00:01:04.356644+0000","last_became_peered":"2024-04-25T00:01:04.356644+0000","last_unstale":"2024-04-25T00:30:15.809642+0000","last_undegraded":"2024-04-25T00:30:15.809642+0000","last_fullsized":"2024-04-25T00:30:15.809642+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:21.729893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":286,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.942363+0000","last_change":"2024-04-25T00:00:55.084492+0000","last_active":"2024-04-25T00:30:15.942363+0000","last_peered":"2024-04-25T00:30:15.942363+0000","last_clean":"2024-04-25T00:30:15.942363+0000","last_became_active":"2024-04-25T00:00:55.083493+0000","last_became_peered":"2024-04-25T00:00:55.083493+0000","last_unstale":"2024-04-25T00:30:15.942363+0000","last_undegraded":"2024-04-25T00:30:15.942363+0000","last_fullsized":"2024-04-25T00:30:15.942363+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:00:54.060578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:00:54.060578+0000","last_clean_scrub_stamp":"2024-04-25T00:00:54.060578+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:14:58.472477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"74'2","reported_seq":124,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.137996+0000","last_change":"2024-04-25T00:01:18.476298+0000","last_active":"2024-04-25T00:30:26.137996+0000","last_peered":"2024-04-25T00:30:26.137996+0000","last_clean":"2024-04-25T00:30:26.137996+0000","last_became_active":"2024-04-25T00:01:18.475898+0000","last_became_peered":"2024-04-25T00:01:18.475898+0000","last_unstale":"2024-04-25T00:30:26.137996+0000","last_undegraded":"2024-04-25T00:30:26.137996+0000","last_fullsized":"2024-04-25T00:30:26.137996+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:49.341842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"74'3757","reported_seq":3877,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.774597+0000","last_change":"2024-04-25T00:01:19.518102+0000","last_active":"2024-04-25T00:30:26.774597+0000","last_peered":"2024-04-25T00:30:26.774597+0000","last_clean":"2024-04-25T00:30:26.774597+0000","last_became_active":"2024-04-25T00:01:19.517376+0000","last_became_peered":"2024-04-25T00:01:19.517376+0000","last_unstale":"2024-04-25T00:30:26.774597+0000","last_undegraded":"2024-04-25T00:30:26.774597+0000","last_fullsized":"2024-04-25T00:30:26.774597+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3757,"log_dups_size":0,"ondisk_log_size":3757,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:55:46.853565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":2,"num_write":20920,"num_write_kb":243770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"70'4367","reported_seq":4495,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.813789+0000","last_change":"2024-04-25T00:01:05.376609+0000","last_active":"2024-04-25T00:30:15.813789+0000","last_peered":"2024-04-25T00:30:15.813789+0000","last_clean":"2024-04-25T00:30:15.813789+0000","last_became_active":"2024-04-25T00:01:05.375881+0000","last_became_peered":"2024-04-25T00:01:05.375881+0000","last_unstale":"2024-04-25T00:30:15.813789+0000","last_undegraded":"2024-04-25T00:30:15.813789+0000","last_fullsized":"2024-04-25T00:30:15.813789+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4367,"log_dups_size":0,"ondisk_log_size":4367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:46:05.592233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":2,"num_write":23695,"num_write_kb":308143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"70'6","reported_seq":135,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.813175+0000","last_change":"2024-04-25T00:01:04.368954+0000","last_active":"2024-04-25T00:30:15.813175+0000","last_peered":"2024-04-25T00:30:15.813175+0000","last_clean":"2024-04-25T00:30:15.813175+0000","last_became_active":"2024-04-25T00:01:04.367903+0000","last_became_peered":"2024-04-25T00:01:04.367903+0000","last_unstale":"2024-04-25T00:30:15.813175+0000","last_undegraded":"2024-04-25T00:30:15.813175+0000","last_fullsized":"2024-04-25T00:30:15.813175+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"74'3","reported_seq":124,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.737742+0000","last_change":"2024-04-25T00:01:18.476525+0000","last_active":"2024-04-25T00:30:26.737742+0000","last_peered":"2024-04-25T00:30:26.737742+0000","last_clean":"2024-04-25T00:30:26.737742+0000","last_became_active":"2024-04-25T00:01:18.476247+0000","last_became_peered":"2024-04-25T00:01:18.476247+0000","last_unstale":"2024-04-25T00:30:26.737742+0000","last_undegraded":"2024-04-25T00:30:26.737742+0000","last_fullsized":"2024-04-25T00:30:26.737742+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:30:36.682736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"74'4936","reported_seq":5056,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.737754+0000","last_change":"2024-04-25T00:01:19.514615+0000","last_active":"2024-04-25T00:30:26.737754+0000","last_peered":"2024-04-25T00:30:26.737754+0000","last_clean":"2024-04-25T00:30:26.737754+0000","last_became_active":"2024-04-25T00:01:19.513640+0000","last_became_peered":"2024-04-25T00:01:19.513640+0000","last_unstale":"2024-04-25T00:30:26.737754+0000","last_undegraded":"2024-04-25T00:30:26.737754+0000","last_fullsized":"2024-04-25T00:30:26.737754+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4936,"log_dups_size":0,"ondisk_log_size":4936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:08.620602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":27403,"num_write_kb":306834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"70'4466","reported_seq":4593,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.793110+0000","last_change":"2024-04-25T00:01:05.385670+0000","last_active":"2024-04-25T00:30:15.793110+0000","last_peered":"2024-04-25T00:30:15.793110+0000","last_clean":"2024-04-25T00:30:15.793110+0000","last_became_active":"2024-04-25T00:01:05.384822+0000","last_became_peered":"2024-04-25T00:01:05.384822+0000","last_unstale":"2024-04-25T00:30:15.793110+0000","last_undegraded":"2024-04-25T00:30:15.793110+0000","last_fullsized":"2024-04-25T00:30:15.793110+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4466,"log_dups_size":0,"ondisk_log_size":4466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:24.017263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23831,"num_write_kb":316890,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"70'6","reported_seq":158,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.798998+0000","last_change":"2024-04-25T00:01:04.356627+0000","last_active":"2024-04-25T00:30:15.798998+0000","last_peered":"2024-04-25T00:30:15.798998+0000","last_clean":"2024-04-25T00:30:15.798998+0000","last_became_active":"2024-04-25T00:01:04.354419+0000","last_became_peered":"2024-04-25T00:01:04.354419+0000","last_unstale":"2024-04-25T00:30:15.798998+0000","last_undegraded":"2024-04-25T00:30:15.798998+0000","last_fullsized":"2024-04-25T00:30:15.798998+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:26.243635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'1453","reported_seq":1626,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.754238+0000","last_change":"2024-04-25T00:01:18.490511+0000","last_active":"2024-04-25T00:30:26.754238+0000","last_peered":"2024-04-25T00:30:26.754238+0000","last_clean":"2024-04-25T00:30:26.754238+0000","last_became_active":"2024-04-25T00:01:18.489629+0000","last_became_peered":"2024-04-25T00:01:18.489629+0000","last_unstale":"2024-04-25T00:30:26.754238+0000","last_undegraded":"2024-04-25T00:30:26.754238+0000","last_fullsized":"2024-04-25T00:30:26.754238+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":1453,"log_dups_size":0,"ondisk_log_size":1453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:07.610846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":75,"num_read_kb":41971,"num_write":1427,"num_write_kb":4503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'3796","reported_seq":3915,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.602587+0000","last_change":"2024-04-25T00:01:19.508931+0000","last_active":"2024-04-25T00:30:26.602587+0000","last_peered":"2024-04-25T00:30:26.602587+0000","last_clean":"2024-04-25T00:30:26.602587+0000","last_became_active":"2024-04-25T00:01:19.507867+0000","last_became_peered":"2024-04-25T00:01:19.507867+0000","last_unstale":"2024-04-25T00:30:26.602587+0000","last_undegraded":"2024-04-25T00:30:26.602587+0000","last_fullsized":"2024-04-25T00:30:26.602587+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3796,"log_dups_size":0,"ondisk_log_size":3796,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:38.152376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20867,"num_write_kb":276131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"70'5324","reported_seq":5452,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.792960+0000","last_change":"2024-04-25T00:01:05.378840+0000","last_active":"2024-04-25T00:30:15.792960+0000","last_peered":"2024-04-25T00:30:15.792960+0000","last_clean":"2024-04-25T00:30:15.792960+0000","last_became_active":"2024-04-25T00:01:05.378112+0000","last_became_peered":"2024-04-25T00:01:05.378112+0000","last_unstale":"2024-04-25T00:30:15.792960+0000","last_undegraded":"2024-04-25T00:30:15.792960+0000","last_fullsized":"2024-04-25T00:30:15.792960+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":5324,"log_dups_size":0,"ondisk_log_size":5324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:43.615895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":28190,"num_write_kb":387378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"69'7","reported_seq":182,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.795644+0000","last_change":"2024-04-25T00:01:04.370794+0000","last_active":"2024-04-25T00:30:15.795644+0000","last_peered":"2024-04-25T00:30:15.795644+0000","last_clean":"2024-04-25T00:30:15.795644+0000","last_became_active":"2024-04-25T00:01:04.370099+0000","last_became_peered":"2024-04-25T00:01:04.370099+0000","last_unstale":"2024-04-25T00:30:15.795644+0000","last_undegraded":"2024-04-25T00:30:15.795644+0000","last_fullsized":"2024-04-25T00:30:15.795644+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:42:03.181036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"74'5","reported_seq":126,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.138150+0000","last_change":"2024-04-25T00:01:18.475256+0000","last_active":"2024-04-25T00:30:26.138150+0000","last_peered":"2024-04-25T00:30:26.138150+0000","last_clean":"2024-04-25T00:30:26.138150+0000","last_became_active":"2024-04-25T00:01:18.474952+0000","last_became_peered":"2024-04-25T00:01:18.474952+0000","last_unstale":"2024-04-25T00:30:26.138150+0000","last_undegraded":"2024-04-25T00:30:26.138150+0000","last_fullsized":"2024-04-25T00:30:26.138150+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:46.849870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"74'4048","reported_seq":4167,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.238601+0000","last_change":"2024-04-25T00:01:19.516833+0000","last_active":"2024-04-25T00:30:26.238601+0000","last_peered":"2024-04-25T00:30:26.238601+0000","last_clean":"2024-04-25T00:30:26.238601+0000","last_became_active":"2024-04-25T00:01:19.515926+0000","last_became_peered":"2024-04-25T00:01:19.515926+0000","last_unstale":"2024-04-25T00:30:26.238601+0000","last_undegraded":"2024-04-25T00:30:26.238601+0000","last_fullsized":"2024-04-25T00:30:26.238601+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4048,"log_dups_size":0,"ondisk_log_size":4048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:23:48.855381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23468,"num_write_kb":270462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"70'3273","reported_seq":3401,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.808120+0000","last_change":"2024-04-25T00:01:05.372591+0000","last_active":"2024-04-25T00:30:15.808120+0000","last_peered":"2024-04-25T00:30:15.808120+0000","last_clean":"2024-04-25T00:30:15.808120+0000","last_became_active":"2024-04-25T00:01:05.371204+0000","last_became_peered":"2024-04-25T00:01:05.371204+0000","last_unstale":"2024-04-25T00:30:15.808120+0000","last_undegraded":"2024-04-25T00:30:15.808120+0000","last_fullsized":"2024-04-25T00:30:15.808120+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3273,"log_dups_size":0,"ondisk_log_size":3273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:15.879673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":17647,"num_write_kb":246988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"70'4","reported_seq":156,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.793404+0000","last_change":"2024-04-25T00:01:04.372376+0000","last_active":"2024-04-25T00:30:15.793404+0000","last_peered":"2024-04-25T00:30:15.793404+0000","last_clean":"2024-04-25T00:30:15.793404+0000","last_became_active":"2024-04-25T00:01:04.372040+0000","last_became_peered":"2024-04-25T00:01:04.372040+0000","last_unstale":"2024-04-25T00:30:15.793404+0000","last_undegraded":"2024-04-25T00:30:15.793404+0000","last_fullsized":"2024-04-25T00:30:15.793404+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:21.729893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"74'2","reported_seq":123,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.736656+0000","last_change":"2024-04-25T00:01:18.487548+0000","last_active":"2024-04-25T00:30:26.736656+0000","last_peered":"2024-04-25T00:30:26.736656+0000","last_clean":"2024-04-25T00:30:26.736656+0000","last_became_active":"2024-04-25T00:01:18.487227+0000","last_became_peered":"2024-04-25T00:01:18.487227+0000","last_unstale":"2024-04-25T00:30:26.736656+0000","last_undegraded":"2024-04-25T00:30:26.736656+0000","last_fullsized":"2024-04-25T00:30:26.736656+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:55.074018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"74'4306","reported_seq":4427,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.642530+0000","last_change":"2024-04-25T00:01:19.528574+0000","last_active":"2024-04-25T00:30:26.642530+0000","last_peered":"2024-04-25T00:30:26.642530+0000","last_clean":"2024-04-25T00:30:26.642530+0000","last_became_active":"2024-04-25T00:01:19.525531+0000","last_became_peered":"2024-04-25T00:01:19.525531+0000","last_unstale":"2024-04-25T00:30:26.642530+0000","last_undegraded":"2024-04-25T00:30:26.642530+0000","last_fullsized":"2024-04-25T00:30:26.642530+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4306,"log_dups_size":0,"ondisk_log_size":4306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:47:08.132895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":31,"num_write":22571,"num_write_kb":279217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"70'3794","reported_seq":3920,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.804529+0000","last_change":"2024-04-25T00:01:05.364103+0000","last_active":"2024-04-25T00:30:15.804529+0000","last_peered":"2024-04-25T00:30:15.804529+0000","last_clean":"2024-04-25T00:30:15.804529+0000","last_became_active":"2024-04-25T00:01:05.363320+0000","last_became_peered":"2024-04-25T00:01:05.363320+0000","last_unstale":"2024-04-25T00:30:15.804529+0000","last_undegraded":"2024-04-25T00:30:15.804529+0000","last_fullsized":"2024-04-25T00:30:15.804529+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3794,"log_dups_size":0,"ondisk_log_size":3794,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:31.456792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20831,"num_write_kb":276143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"74'15","reported_seq":257,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.791248+0000","last_change":"2024-04-25T00:01:04.358365+0000","last_active":"2024-04-25T00:30:15.791248+0000","last_peered":"2024-04-25T00:30:15.791248+0000","last_clean":"2024-04-25T00:30:15.791248+0000","last_became_active":"2024-04-25T00:01:04.357571+0000","last_became_peered":"2024-04-25T00:01:04.357571+0000","last_unstale":"2024-04-25T00:30:15.791248+0000","last_undegraded":"2024-04-25T00:30:15.791248+0000","last_fullsized":"2024-04-25T00:30:15.791248+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:50:23.023114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":342,"num_read_kb":342,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"74'3","reported_seq":123,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.747647+0000","last_change":"2024-04-25T00:01:18.497679+0000","last_active":"2024-04-25T00:30:26.747647+0000","last_peered":"2024-04-25T00:30:26.747647+0000","last_clean":"2024-04-25T00:30:26.747647+0000","last_became_active":"2024-04-25T00:01:18.496489+0000","last_became_peered":"2024-04-25T00:01:18.496489+0000","last_unstale":"2024-04-25T00:30:26.747647+0000","last_undegraded":"2024-04-25T00:30:26.747647+0000","last_fullsized":"2024-04-25T00:30:26.747647+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:23:47.845624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"74'4849","reported_seq":4970,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.756589+0000","last_change":"2024-04-25T00:01:19.498082+0000","last_active":"2024-04-25T00:30:26.756589+0000","last_peered":"2024-04-25T00:30:26.756589+0000","last_clean":"2024-04-25T00:30:26.756589+0000","last_became_active":"2024-04-25T00:01:19.496805+0000","last_became_peered":"2024-04-25T00:01:19.496805+0000","last_unstale":"2024-04-25T00:30:26.756589+0000","last_undegraded":"2024-04-25T00:30:26.756589+0000","last_fullsized":"2024-04-25T00:30:26.756589+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4849,"log_dups_size":0,"ondisk_log_size":4849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:57:36.256250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":16,"num_write":26442,"num_write_kb":335585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"70'2974","reported_seq":3100,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.941797+0000","last_change":"2024-04-25T00:01:05.373927+0000","last_active":"2024-04-25T00:30:15.941797+0000","last_peered":"2024-04-25T00:30:15.941797+0000","last_clean":"2024-04-25T00:30:15.941797+0000","last_became_active":"2024-04-25T00:01:05.373077+0000","last_became_peered":"2024-04-25T00:01:05.373077+0000","last_unstale":"2024-04-25T00:30:15.941797+0000","last_undegraded":"2024-04-25T00:30:15.941797+0000","last_fullsized":"2024-04-25T00:30:15.941797+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":2974,"log_dups_size":0,"ondisk_log_size":2974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:50.489125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15895,"num_write_kb":231645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"70'5","reported_seq":133,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.791512+0000","last_change":"2024-04-25T00:01:04.357265+0000","last_active":"2024-04-25T00:30:15.791512+0000","last_peered":"2024-04-25T00:30:15.791512+0000","last_clean":"2024-04-25T00:30:15.791512+0000","last_became_active":"2024-04-25T00:01:04.357004+0000","last_became_peered":"2024-04-25T00:01:04.357004+0000","last_unstale":"2024-04-25T00:30:15.791512+0000","last_undegraded":"2024-04-25T00:30:15.791512+0000","last_fullsized":"2024-04-25T00:30:15.791512+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:47:25.595275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"74'5","reported_seq":125,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.707490+0000","last_change":"2024-04-25T00:01:18.487276+0000","last_active":"2024-04-25T00:30:26.707490+0000","last_peered":"2024-04-25T00:30:26.707490+0000","last_clean":"2024-04-25T00:30:26.707490+0000","last_became_active":"2024-04-25T00:01:18.486279+0000","last_became_peered":"2024-04-25T00:01:18.486279+0000","last_unstale":"2024-04-25T00:30:26.707490+0000","last_undegraded":"2024-04-25T00:30:26.707490+0000","last_fullsized":"2024-04-25T00:30:26.707490+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:17:37.141843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"74'4040","reported_seq":4158,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.630478+0000","last_change":"2024-04-25T00:01:19.516827+0000","last_active":"2024-04-25T00:30:26.630478+0000","last_peered":"2024-04-25T00:30:26.630478+0000","last_clean":"2024-04-25T00:30:26.630478+0000","last_became_active":"2024-04-25T00:01:19.515765+0000","last_became_peered":"2024-04-25T00:01:19.515765+0000","last_unstale":"2024-04-25T00:30:26.630478+0000","last_undegraded":"2024-04-25T00:30:26.630478+0000","last_fullsized":"2024-04-25T00:30:26.630478+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4040,"log_dups_size":0,"ondisk_log_size":4040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:13:34.153117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23288,"num_write_kb":269674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"70'4514","reported_seq":4640,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.806193+0000","last_change":"2024-04-25T00:01:05.379501+0000","last_active":"2024-04-25T00:30:15.806193+0000","last_peered":"2024-04-25T00:30:15.806193+0000","last_clean":"2024-04-25T00:30:15.806193+0000","last_became_active":"2024-04-25T00:01:05.378644+0000","last_became_peered":"2024-04-25T00:01:05.378644+0000","last_unstale":"2024-04-25T00:30:15.806193+0000","last_undegraded":"2024-04-25T00:30:15.806193+0000","last_fullsized":"2024-04-25T00:30:15.806193+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4514,"log_dups_size":0,"ondisk_log_size":4514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:08:35.505154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25321,"num_write_kb":326961,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"70'3","reported_seq":131,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.809820+0000","last_change":"2024-04-25T00:01:04.360987+0000","last_active":"2024-04-25T00:30:15.809820+0000","last_peered":"2024-04-25T00:30:15.809820+0000","last_clean":"2024-04-25T00:30:15.809820+0000","last_became_active":"2024-04-25T00:01:04.360664+0000","last_became_peered":"2024-04-25T00:01:04.360664+0000","last_unstale":"2024-04-25T00:30:15.809820+0000","last_undegraded":"2024-04-25T00:30:15.809820+0000","last_fullsized":"2024-04-25T00:30:15.809820+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:55.283076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"74'334","reported_seq":571,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:19.197817+0000","last_change":"2024-04-25T00:01:18.505577+0000","last_active":"2024-04-25T00:30:19.197817+0000","last_peered":"2024-04-25T00:30:19.197817+0000","last_clean":"2024-04-25T00:30:19.197817+0000","last_became_active":"2024-04-25T00:01:18.504963+0000","last_became_peered":"2024-04-25T00:01:18.504963+0000","last_unstale":"2024-04-25T00:30:19.197817+0000","last_undegraded":"2024-04-25T00:30:19.197817+0000","last_fullsized":"2024-04-25T00:30:19.197817+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":334,"log_dups_size":0,"ondisk_log_size":334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:37.142619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":209,"num_read_kb":90302,"num_write":330,"num_write_kb":4274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"74'4674","reported_seq":4789,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.733748+0000","last_change":"2024-04-25T00:01:46.658442+0000","last_active":"2024-04-25T00:30:26.733748+0000","last_peered":"2024-04-25T00:30:26.733748+0000","last_clean":"2024-04-25T00:30:26.733748+0000","last_became_active":"2024-04-25T00:01:46.657588+0000","last_became_peered":"2024-04-25T00:01:46.657588+0000","last_unstale":"2024-04-25T00:30:26.733748+0000","last_undegraded":"2024-04-25T00:30:26.733748+0000","last_fullsized":"2024-04-25T00:30:26.733748+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4674,"log_dups_size":0,"ondisk_log_size":4674,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:58:49.738233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":16,"num_write":25396,"num_write_kb":335454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"70'3777","reported_seq":3903,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.810481+0000","last_change":"2024-04-25T00:01:05.379102+0000","last_active":"2024-04-25T00:30:15.810481+0000","last_peered":"2024-04-25T00:30:15.810481+0000","last_clean":"2024-04-25T00:30:15.810481+0000","last_became_active":"2024-04-25T00:01:05.377625+0000","last_became_peered":"2024-04-25T00:01:05.377625+0000","last_unstale":"2024-04-25T00:30:15.810481+0000","last_undegraded":"2024-04-25T00:30:15.810481+0000","last_fullsized":"2024-04-25T00:30:15.810481+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3777,"log_dups_size":0,"ondisk_log_size":3777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:15.879673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20730,"num_write_kb":283926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"73'9","reported_seq":183,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.803482+0000","last_change":"2024-04-25T00:01:04.370640+0000","last_active":"2024-04-25T00:30:15.803482+0000","last_peered":"2024-04-25T00:30:15.803482+0000","last_clean":"2024-04-25T00:30:15.803482+0000","last_became_active":"2024-04-25T00:01:04.370031+0000","last_became_peered":"2024-04-25T00:01:04.370031+0000","last_unstale":"2024-04-25T00:30:15.803482+0000","last_undegraded":"2024-04-25T00:30:15.803482+0000","last_fullsized":"2024-04-25T00:30:15.803482+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:26.243635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":184,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"74'31","reported_seq":220,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.781231+0000","last_change":"2024-04-25T00:01:18.473113+0000","last_active":"2024-04-25T00:30:26.781231+0000","last_peered":"2024-04-25T00:30:26.781231+0000","last_clean":"2024-04-25T00:30:26.781231+0000","last_became_active":"2024-04-25T00:01:18.472931+0000","last_became_peered":"2024-04-25T00:01:18.472931+0000","last_unstale":"2024-04-25T00:30:26.781231+0000","last_undegraded":"2024-04-25T00:30:26.781231+0000","last_fullsized":"2024-04-25T00:30:26.781231+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:07.498985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":161,"num_read_kb":161,"num_write":32,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"74'5042","reported_seq":5161,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.730299+0000","last_change":"2024-04-25T00:01:19.533664+0000","last_active":"2024-04-25T00:30:26.730299+0000","last_peered":"2024-04-25T00:30:26.730299+0000","last_clean":"2024-04-25T00:30:26.730299+0000","last_became_active":"2024-04-25T00:01:19.533048+0000","last_became_peered":"2024-04-25T00:01:19.533048+0000","last_unstale":"2024-04-25T00:30:26.730299+0000","last_undegraded":"2024-04-25T00:30:26.730299+0000","last_fullsized":"2024-04-25T00:30:26.730299+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":5042,"log_dups_size":0,"ondisk_log_size":5042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:45:43.141569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":28058,"num_write_kb":350341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"69'3549","reported_seq":3675,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.790799+0000","last_change":"2024-04-25T00:01:05.385827+0000","last_active":"2024-04-25T00:30:15.790799+0000","last_peered":"2024-04-25T00:30:15.790799+0000","last_clean":"2024-04-25T00:30:15.790799+0000","last_became_active":"2024-04-25T00:01:05.385258+0000","last_became_peered":"2024-04-25T00:01:05.385258+0000","last_unstale":"2024-04-25T00:30:15.790799+0000","last_undegraded":"2024-04-25T00:30:15.790799+0000","last_fullsized":"2024-04-25T00:30:15.790799+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3549,"log_dups_size":0,"ondisk_log_size":3549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:23.870121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19806,"num_write_kb":251506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"74'341","reported_seq":585,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:22.899825+0000","last_change":"2024-04-25T00:01:04.365185+0000","last_active":"2024-04-25T00:30:22.899825+0000","last_peered":"2024-04-25T00:30:22.899825+0000","last_clean":"2024-04-25T00:30:22.899825+0000","last_became_active":"2024-04-25T00:01:04.364930+0000","last_became_peered":"2024-04-25T00:01:04.364930+0000","last_unstale":"2024-04-25T00:30:22.899825+0000","last_undegraded":"2024-04-25T00:30:22.899825+0000","last_fullsized":"2024-04-25T00:30:22.899825+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":341,"log_dups_size":0,"ondisk_log_size":341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:45:03.618340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":208,"num_read_kb":94392,"num_write":336,"num_write_kb":4266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"74'3","reported_seq":123,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.747231+0000","last_change":"2024-04-25T00:01:18.496220+0000","last_active":"2024-04-25T00:30:26.747231+0000","last_peered":"2024-04-25T00:30:26.747231+0000","last_clean":"2024-04-25T00:30:26.747231+0000","last_became_active":"2024-04-25T00:01:18.492850+0000","last_became_peered":"2024-04-25T00:01:18.492850+0000","last_unstale":"2024-04-25T00:30:26.747231+0000","last_undegraded":"2024-04-25T00:30:26.747231+0000","last_fullsized":"2024-04-25T00:30:26.747231+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:50:16.922314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"74'4133","reported_seq":4252,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.680551+0000","last_change":"2024-04-25T00:01:19.515725+0000","last_active":"2024-04-25T00:30:26.680551+0000","last_peered":"2024-04-25T00:30:26.680551+0000","last_clean":"2024-04-25T00:30:26.680551+0000","last_became_active":"2024-04-25T00:01:19.509285+0000","last_became_peered":"2024-04-25T00:01:19.509285+0000","last_unstale":"2024-04-25T00:30:26.680551+0000","last_undegraded":"2024-04-25T00:30:26.680551+0000","last_fullsized":"2024-04-25T00:30:26.680551+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4133,"log_dups_size":0,"ondisk_log_size":4133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:25:20.185355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":4,"num_write":22847,"num_write_kb":280058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"70'5270","reported_seq":5396,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.809540+0000","last_change":"2024-04-25T00:01:05.371061+0000","last_active":"2024-04-25T00:30:15.809540+0000","last_peered":"2024-04-25T00:30:15.809540+0000","last_clean":"2024-04-25T00:30:15.809540+0000","last_became_active":"2024-04-25T00:01:05.369950+0000","last_became_peered":"2024-04-25T00:01:05.369950+0000","last_unstale":"2024-04-25T00:30:15.809540+0000","last_undegraded":"2024-04-25T00:30:15.809540+0000","last_fullsized":"2024-04-25T00:30:15.809540+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":5270,"log_dups_size":0,"ondisk_log_size":5270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:02:43.237902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28224,"num_write_kb":363198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"74'31","reported_seq":205,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:25.818373+0000","last_change":"2024-04-25T00:01:04.366503+0000","last_active":"2024-04-25T00:30:25.818373+0000","last_peered":"2024-04-25T00:30:25.818373+0000","last_clean":"2024-04-25T00:30:25.818373+0000","last_became_active":"2024-04-25T00:01:04.365945+0000","last_became_peered":"2024-04-25T00:01:04.365945+0000","last_unstale":"2024-04-25T00:30:25.818373+0000","last_undegraded":"2024-04-25T00:30:25.818373+0000","last_fullsized":"2024-04-25T00:30:25.818373+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":92,"num_write":32,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"74'8","reported_seq":174,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.742489+0000","last_change":"2024-04-25T00:01:18.476456+0000","last_active":"2024-04-25T00:30:26.742489+0000","last_peered":"2024-04-25T00:30:26.742489+0000","last_clean":"2024-04-25T00:30:26.742489+0000","last_became_active":"2024-04-25T00:01:18.475876+0000","last_became_peered":"2024-04-25T00:01:18.475876+0000","last_unstale":"2024-04-25T00:30:26.742489+0000","last_undegraded":"2024-04-25T00:30:26.742489+0000","last_fullsized":"2024-04-25T00:30:26.742489+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:56.741250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":184,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"74'4233","reported_seq":4354,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.727907+0000","last_change":"2024-04-25T00:01:19.515376+0000","last_active":"2024-04-25T00:30:26.727907+0000","last_peered":"2024-04-25T00:30:26.727907+0000","last_clean":"2024-04-25T00:30:26.727907+0000","last_became_active":"2024-04-25T00:01:19.512425+0000","last_became_peered":"2024-04-25T00:01:19.512425+0000","last_unstale":"2024-04-25T00:30:26.727907+0000","last_undegraded":"2024-04-25T00:30:26.727907+0000","last_fullsized":"2024-04-25T00:30:26.727907+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4233,"log_dups_size":0,"ondisk_log_size":4233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:43:16.957273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":24291,"num_write_kb":290168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"70'4413","reported_seq":4539,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.800988+0000","last_change":"2024-04-25T00:01:05.361723+0000","last_active":"2024-04-25T00:30:15.800988+0000","last_peered":"2024-04-25T00:30:15.800988+0000","last_clean":"2024-04-25T00:30:15.800988+0000","last_became_active":"2024-04-25T00:01:05.360956+0000","last_became_peered":"2024-04-25T00:01:05.360956+0000","last_unstale":"2024-04-25T00:30:15.800988+0000","last_undegraded":"2024-04-25T00:30:15.800988+0000","last_fullsized":"2024-04-25T00:30:15.800988+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4413,"log_dups_size":0,"ondisk_log_size":4413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:23.870121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22903,"num_write_kb":321461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"70'3","reported_seq":131,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.790859+0000","last_change":"2024-04-25T00:01:04.362129+0000","last_active":"2024-04-25T00:30:15.790859+0000","last_peered":"2024-04-25T00:30:15.790859+0000","last_clean":"2024-04-25T00:30:15.790859+0000","last_became_active":"2024-04-25T00:01:04.361827+0000","last_became_peered":"2024-04-25T00:01:04.361827+0000","last_unstale":"2024-04-25T00:30:15.790859+0000","last_undegraded":"2024-04-25T00:30:15.790859+0000","last_fullsized":"2024-04-25T00:30:15.790859+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:26.243635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"74'15","reported_seq":253,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.739696+0000","last_change":"2024-04-25T00:01:18.496129+0000","last_active":"2024-04-25T00:30:26.739696+0000","last_peered":"2024-04-25T00:30:26.739696+0000","last_clean":"2024-04-25T00:30:26.739696+0000","last_became_active":"2024-04-25T00:01:18.495415+0000","last_became_peered":"2024-04-25T00:01:18.495415+0000","last_unstale":"2024-04-25T00:30:26.739696+0000","last_undegraded":"2024-04-25T00:30:26.739696+0000","last_fullsized":"2024-04-25T00:30:26.739696+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:38.089411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":354,"num_read_kb":354,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"74'3474","reported_seq":3592,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.668678+0000","last_change":"2024-04-25T00:01:19.509542+0000","last_active":"2024-04-25T00:30:26.668678+0000","last_peered":"2024-04-25T00:30:26.668678+0000","last_clean":"2024-04-25T00:30:26.668678+0000","last_became_active":"2024-04-25T00:01:19.509015+0000","last_became_peered":"2024-04-25T00:01:19.509015+0000","last_unstale":"2024-04-25T00:30:26.668678+0000","last_undegraded":"2024-04-25T00:30:26.668678+0000","last_fullsized":"2024-04-25T00:30:26.668678+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3474,"log_dups_size":0,"ondisk_log_size":3474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:25:20.185355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19390,"num_write_kb":245789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"70'4331","reported_seq":4457,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.806796+0000","last_change":"2024-04-25T00:01:05.365413+0000","last_active":"2024-04-25T00:30:15.806796+0000","last_peered":"2024-04-25T00:30:15.806796+0000","last_clean":"2024-04-25T00:30:15.806796+0000","last_became_active":"2024-04-25T00:01:05.361865+0000","last_became_peered":"2024-04-25T00:01:05.361865+0000","last_unstale":"2024-04-25T00:30:15.806796+0000","last_undegraded":"2024-04-25T00:30:15.806796+0000","last_fullsized":"2024-04-25T00:30:15.806796+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4331,"log_dups_size":0,"ondisk_log_size":4331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:30.228060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22828,"num_write_kb":312693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"74'7","reported_seq":135,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.944255+0000","last_change":"2024-04-25T00:01:04.360804+0000","last_active":"2024-04-25T00:30:15.944255+0000","last_peered":"2024-04-25T00:30:15.944255+0000","last_clean":"2024-04-25T00:30:15.944255+0000","last_became_active":"2024-04-25T00:01:04.360374+0000","last_became_peered":"2024-04-25T00:01:04.360374+0000","last_unstale":"2024-04-25T00:30:15.944255+0000","last_undegraded":"2024-04-25T00:30:15.944255+0000","last_fullsized":"2024-04-25T00:30:15.944255+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"74'5","reported_seq":125,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.739206+0000","last_change":"2024-04-25T00:01:18.480003+0000","last_active":"2024-04-25T00:30:26.739206+0000","last_peered":"2024-04-25T00:30:26.739206+0000","last_clean":"2024-04-25T00:30:26.739206+0000","last_became_active":"2024-04-25T00:01:18.479270+0000","last_became_peered":"2024-04-25T00:01:18.479270+0000","last_unstale":"2024-04-25T00:30:26.739206+0000","last_undegraded":"2024-04-25T00:30:26.739206+0000","last_fullsized":"2024-04-25T00:30:26.739206+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:52:40.942502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"74'5681","reported_seq":5805,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.778355+0000","last_change":"2024-04-25T00:01:19.524463+0000","last_active":"2024-04-25T00:30:26.778355+0000","last_peered":"2024-04-25T00:30:26.778355+0000","last_clean":"2024-04-25T00:30:26.778355+0000","last_became_active":"2024-04-25T00:01:19.524166+0000","last_became_peered":"2024-04-25T00:01:19.524166+0000","last_unstale":"2024-04-25T00:30:26.778355+0000","last_undegraded":"2024-04-25T00:30:26.778355+0000","last_fullsized":"2024-04-25T00:30:26.778355+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":5681,"log_dups_size":0,"ondisk_log_size":5681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:50.351599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":40,"num_write":31879,"num_write_kb":410681,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"74'28","reported_seq":150,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.738030+0000","last_change":"2024-04-25T00:01:18.497096+0000","last_active":"2024-04-25T00:30:26.738030+0000","last_peered":"2024-04-25T00:30:26.738030+0000","last_clean":"2024-04-25T00:30:26.738030+0000","last_became_active":"2024-04-25T00:01:18.496289+0000","last_became_peered":"2024-04-25T00:01:18.496289+0000","last_unstale":"2024-04-25T00:30:26.738030+0000","last_undegraded":"2024-04-25T00:30:26.738030+0000","last_fullsized":"2024-04-25T00:30:26.738030+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:59:40.316831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"70'4773","reported_seq":4900,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.791447+0000","last_change":"2024-04-25T00:01:05.370043+0000","last_active":"2024-04-25T00:30:15.791447+0000","last_peered":"2024-04-25T00:30:15.791447+0000","last_clean":"2024-04-25T00:30:15.791447+0000","last_became_active":"2024-04-25T00:01:05.368080+0000","last_became_peered":"2024-04-25T00:01:05.368080+0000","last_unstale":"2024-04-25T00:30:15.791447+0000","last_undegraded":"2024-04-25T00:30:15.791447+0000","last_fullsized":"2024-04-25T00:30:15.791447+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4773,"log_dups_size":0,"ondisk_log_size":4773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:58:07.610923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":25218,"num_write_kb":330520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"69'2","reported_seq":130,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.804702+0000","last_change":"2024-04-25T00:01:04.364523+0000","last_active":"2024-04-25T00:30:15.804702+0000","last_peered":"2024-04-25T00:30:15.804702+0000","last_clean":"2024-04-25T00:30:15.804702+0000","last_became_active":"2024-04-25T00:01:04.364050+0000","last_became_peered":"2024-04-25T00:01:04.364050+0000","last_unstale":"2024-04-25T00:30:15.804702+0000","last_undegraded":"2024-04-25T00:30:15.804702+0000","last_fullsized":"2024-04-25T00:30:15.804702+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:49.479804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"74'3390","reported_seq":3510,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.735796+0000","last_change":"2024-04-25T00:01:19.491523+0000","last_active":"2024-04-25T00:30:26.735796+0000","last_peered":"2024-04-25T00:30:26.735796+0000","last_clean":"2024-04-25T00:30:26.735796+0000","last_became_active":"2024-04-25T00:01:19.490755+0000","last_became_peered":"2024-04-25T00:01:19.490755+0000","last_unstale":"2024-04-25T00:30:26.735796+0000","last_undegraded":"2024-04-25T00:30:26.735796+0000","last_fullsized":"2024-04-25T00:30:26.735796+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3390,"log_dups_size":0,"ondisk_log_size":3390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:47:19.526845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18979,"num_write_kb":248305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"74'1082","reported_seq":1259,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.777825+0000","last_change":"2024-04-25T00:01:18.472794+0000","last_active":"2024-04-25T00:30:26.777825+0000","last_peered":"2024-04-25T00:30:26.777825+0000","last_clean":"2024-04-25T00:30:26.777825+0000","last_became_active":"2024-04-25T00:01:18.472534+0000","last_became_peered":"2024-04-25T00:01:18.472534+0000","last_unstale":"2024-04-25T00:30:26.777825+0000","last_undegraded":"2024-04-25T00:30:26.777825+0000","last_fullsized":"2024-04-25T00:30:26.777825+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":1082,"log_dups_size":0,"ondisk_log_size":1082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:55:45.843808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":101,"num_read_kb":11812,"num_write":1055,"num_write_kb":4728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"70'4239","reported_seq":4365,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.794924+0000","last_change":"2024-04-25T00:01:05.378867+0000","last_active":"2024-04-25T00:30:15.794924+0000","last_peered":"2024-04-25T00:30:15.794924+0000","last_clean":"2024-04-25T00:30:15.794924+0000","last_became_active":"2024-04-25T00:01:05.378056+0000","last_became_peered":"2024-04-25T00:01:05.378056+0000","last_unstale":"2024-04-25T00:30:15.794924+0000","last_undegraded":"2024-04-25T00:30:15.794924+0000","last_fullsized":"2024-04-25T00:30:15.794924+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4239,"log_dups_size":0,"ondisk_log_size":4239,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:36:10.377757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22560,"num_write_kb":311538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"70'3","reported_seq":131,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.809380+0000","last_change":"2024-04-25T00:01:04.370896+0000","last_active":"2024-04-25T00:30:15.809380+0000","last_peered":"2024-04-25T00:30:15.809380+0000","last_clean":"2024-04-25T00:30:15.809380+0000","last_became_active":"2024-04-25T00:01:04.370273+0000","last_became_peered":"2024-04-25T00:01:04.370273+0000","last_unstale":"2024-04-25T00:30:15.809380+0000","last_undegraded":"2024-04-25T00:30:15.809380+0000","last_fullsized":"2024-04-25T00:30:15.809380+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:49:22.673421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"74'5148","reported_seq":5266,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.723815+0000","last_change":"2024-04-25T00:01:19.521143+0000","last_active":"2024-04-25T00:30:26.723815+0000","last_peered":"2024-04-25T00:30:26.723815+0000","last_clean":"2024-04-25T00:30:26.723815+0000","last_became_active":"2024-04-25T00:01:19.514664+0000","last_became_peered":"2024-04-25T00:01:19.514664+0000","last_unstale":"2024-04-25T00:30:26.723815+0000","last_undegraded":"2024-04-25T00:30:26.723815+0000","last_fullsized":"2024-04-25T00:30:26.723815+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":5148,"log_dups_size":0,"ondisk_log_size":5148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:03:47.440442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":29131,"num_write_kb":338102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"74'249","reported_seq":380,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:27.894626+0000","last_change":"2024-04-25T00:01:04.360254+0000","last_active":"2024-04-25T00:30:27.894626+0000","last_peered":"2024-04-25T00:30:27.894626+0000","last_clean":"2024-04-25T00:30:27.894626+0000","last_became_active":"2024-04-25T00:01:04.359586+0000","last_became_peered":"2024-04-25T00:01:04.359586+0000","last_unstale":"2024-04-25T00:30:27.894626+0000","last_undegraded":"2024-04-25T00:30:27.894626+0000","last_fullsized":"2024-04-25T00:30:27.894626+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:47.192930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2250578,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4308,"num_write":220,"num_write_kb":2323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"69'3397","reported_seq":3522,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.792637+0000","last_change":"2024-04-25T00:01:05.370723+0000","last_active":"2024-04-25T00:30:15.792637+0000","last_peered":"2024-04-25T00:30:15.792637+0000","last_clean":"2024-04-25T00:30:15.792637+0000","last_became_active":"2024-04-25T00:01:05.368970+0000","last_became_peered":"2024-04-25T00:01:05.368970+0000","last_unstale":"2024-04-25T00:30:15.792637+0000","last_undegraded":"2024-04-25T00:30:15.792637+0000","last_fullsized":"2024-04-25T00:30:15.792637+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3397,"log_dups_size":0,"ondisk_log_size":3397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:06:53.743074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17978,"num_write_kb":244099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"74'5","reported_seq":170,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:19.198496+0000","last_change":"2024-04-25T00:01:18.498570+0000","last_active":"2024-04-25T00:30:19.198496+0000","last_peered":"2024-04-25T00:30:19.198496+0000","last_clean":"2024-04-25T00:30:19.198496+0000","last_became_active":"2024-04-25T00:01:18.497745+0000","last_became_peered":"2024-04-25T00:01:18.497745+0000","last_unstale":"2024-04-25T00:30:19.198496+0000","last_undegraded":"2024-04-25T00:30:19.198496+0000","last_fullsized":"2024-04-25T00:30:19.198496+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:26:42.047149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"74'3919","reported_seq":4039,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.527005+0000","last_change":"2024-04-25T00:01:19.514919+0000","last_active":"2024-04-25T00:30:26.527005+0000","last_peered":"2024-04-25T00:30:26.527005+0000","last_clean":"2024-04-25T00:30:26.527005+0000","last_became_active":"2024-04-25T00:01:19.514280+0000","last_became_peered":"2024-04-25T00:01:19.514280+0000","last_unstale":"2024-04-25T00:30:26.527005+0000","last_undegraded":"2024-04-25T00:30:26.527005+0000","last_fullsized":"2024-04-25T00:30:26.527005+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3919,"log_dups_size":0,"ondisk_log_size":3919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:37.975265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":20874,"num_write_kb":280510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"69'752","reported_seq":907,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.801549+0000","last_change":"2024-04-25T00:01:04.365607+0000","last_active":"2024-04-25T00:30:15.801549+0000","last_peered":"2024-04-25T00:30:15.801549+0000","last_clean":"2024-04-25T00:30:15.801549+0000","last_became_active":"2024-04-25T00:01:04.364788+0000","last_became_peered":"2024-04-25T00:01:04.364788+0000","last_unstale":"2024-04-25T00:30:15.801549+0000","last_undegraded":"2024-04-25T00:30:15.801549+0000","last_fullsized":"2024-04-25T00:30:15.801549+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":752,"log_dups_size":0,"ondisk_log_size":752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":28,"num_read_kb":89132,"num_write":742,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"69'3790","reported_seq":3915,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.942896+0000","last_change":"2024-04-25T00:01:05.375142+0000","last_active":"2024-04-25T00:30:15.942896+0000","last_peered":"2024-04-25T00:30:15.942896+0000","last_clean":"2024-04-25T00:30:15.942896+0000","last_became_active":"2024-04-25T00:01:05.373857+0000","last_became_peered":"2024-04-25T00:01:05.373857+0000","last_unstale":"2024-04-25T00:30:15.942896+0000","last_undegraded":"2024-04-25T00:30:15.942896+0000","last_fullsized":"2024-04-25T00:30:15.942896+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3790,"log_dups_size":0,"ondisk_log_size":3790,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:15.879673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19857,"num_write_kb":263976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"74'48","reported_seq":213,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.740146+0000","last_change":"2024-04-25T00:01:18.502786+0000","last_active":"2024-04-25T00:30:26.740146+0000","last_peered":"2024-04-25T00:30:26.740146+0000","last_clean":"2024-04-25T00:30:26.740146+0000","last_became_active":"2024-04-25T00:01:18.501785+0000","last_became_peered":"2024-04-25T00:01:18.501785+0000","last_unstale":"2024-04-25T00:30:26.740146+0000","last_undegraded":"2024-04-25T00:30:26.740146+0000","last_fullsized":"2024-04-25T00:30:26.740146+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:09:31.441044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.26","version":"74'3694","reported_seq":3813,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.735546+0000","last_change":"2024-04-25T00:01:19.511239+0000","last_active":"2024-04-25T00:30:26.735546+0000","last_peered":"2024-04-25T00:30:26.735546+0000","last_clean":"2024-04-25T00:30:26.735546+0000","last_became_active":"2024-04-25T00:01:19.510155+0000","last_became_peered":"2024-04-25T00:01:19.510155+0000","last_unstale":"2024-04-25T00:30:26.735546+0000","last_undegraded":"2024-04-25T00:30:26.735546+0000","last_fullsized":"2024-04-25T00:30:26.735546+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3694,"log_dups_size":0,"ondisk_log_size":3694,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:43:14.452663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20834,"num_write_kb":259761,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.20","version":"70'4043","reported_seq":4171,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.807721+0000","last_change":"2024-04-25T00:01:05.378517+0000","last_active":"2024-04-25T00:30:15.807721+0000","last_peered":"2024-04-25T00:30:15.807721+0000","last_clean":"2024-04-25T00:30:15.807721+0000","last_became_active":"2024-04-25T00:01:05.377742+0000","last_became_peered":"2024-04-25T00:01:05.377742+0000","last_unstale":"2024-04-25T00:30:15.807721+0000","last_undegraded":"2024-04-25T00:30:15.807721+0000","last_fullsized":"2024-04-25T00:30:15.807721+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4043,"log_dups_size":0,"ondisk_log_size":4043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:06.450369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21543,"num_write_kb":291017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"74'5759","reported_seq":5879,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.589637+0000","last_change":"2024-04-25T00:01:19.508999+0000","last_active":"2024-04-25T00:30:26.589637+0000","last_peered":"2024-04-25T00:30:26.589637+0000","last_clean":"2024-04-25T00:30:26.589637+0000","last_became_active":"2024-04-25T00:01:19.507960+0000","last_became_peered":"2024-04-25T00:01:19.507960+0000","last_unstale":"2024-04-25T00:30:26.589637+0000","last_undegraded":"2024-04-25T00:30:26.589637+0000","last_fullsized":"2024-04-25T00:30:26.589637+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":5759,"log_dups_size":0,"ondisk_log_size":5759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:32:40.274908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":31168,"num_write_kb":366885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.21","version":"70'4068","reported_seq":4197,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.803508+0000","last_change":"2024-04-25T00:01:05.372927+0000","last_active":"2024-04-25T00:30:15.803508+0000","last_peered":"2024-04-25T00:30:15.803508+0000","last_clean":"2024-04-25T00:30:15.803508+0000","last_became_active":"2024-04-25T00:01:05.372050+0000","last_became_peered":"2024-04-25T00:01:05.372050+0000","last_unstale":"2024-04-25T00:30:15.803508+0000","last_undegraded":"2024-04-25T00:30:15.803508+0000","last_fullsized":"2024-04-25T00:30:15.803508+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4068,"log_dups_size":0,"ondisk_log_size":4068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:48.202251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21791,"num_write_kb":292243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.24","version":"74'4315","reported_seq":4436,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.724152+0000","last_change":"2024-04-25T00:01:19.531072+0000","last_active":"2024-04-25T00:30:26.724152+0000","last_peered":"2024-04-25T00:30:26.724152+0000","last_clean":"2024-04-25T00:30:26.724152+0000","last_became_active":"2024-04-25T00:01:19.530836+0000","last_became_peered":"2024-04-25T00:01:19.530836+0000","last_unstale":"2024-04-25T00:30:26.724152+0000","last_undegraded":"2024-04-25T00:30:26.724152+0000","last_fullsized":"2024-04-25T00:30:26.724152+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4315,"log_dups_size":0,"ondisk_log_size":4315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:08.508741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":23956,"num_write_kb":309408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"70'4267","reported_seq":4395,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.807204+0000","last_change":"2024-04-25T00:01:05.362960+0000","last_active":"2024-04-25T00:30:15.807204+0000","last_peered":"2024-04-25T00:30:15.807204+0000","last_clean":"2024-04-25T00:30:15.807204+0000","last_became_active":"2024-04-25T00:01:05.361067+0000","last_became_peered":"2024-04-25T00:01:05.361067+0000","last_unstale":"2024-04-25T00:30:15.807204+0000","last_undegraded":"2024-04-25T00:30:15.807204+0000","last_fullsized":"2024-04-25T00:30:15.807204+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4267,"log_dups_size":0,"ondisk_log_size":4267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:24.017263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23473,"num_write_kb":311317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"74'4634","reported_seq":4754,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.737724+0000","last_change":"2024-04-25T00:01:19.506029+0000","last_active":"2024-04-25T00:30:26.737724+0000","last_peered":"2024-04-25T00:30:26.737724+0000","last_clean":"2024-04-25T00:30:26.737724+0000","last_became_active":"2024-04-25T00:01:19.505184+0000","last_became_peered":"2024-04-25T00:01:19.505184+0000","last_unstale":"2024-04-25T00:30:26.737724+0000","last_undegraded":"2024-04-25T00:30:26.737724+0000","last_fullsized":"2024-04-25T00:30:26.737724+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4634,"log_dups_size":0,"ondisk_log_size":4634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:08.508741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":6,"num_write":26896,"num_write_kb":316898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"70'4083","reported_seq":4211,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.801208+0000","last_change":"2024-04-25T00:01:05.358690+0000","last_active":"2024-04-25T00:30:15.801208+0000","last_peered":"2024-04-25T00:30:15.801208+0000","last_clean":"2024-04-25T00:30:15.801208+0000","last_became_active":"2024-04-25T00:01:05.357340+0000","last_became_peered":"2024-04-25T00:01:05.357340+0000","last_unstale":"2024-04-25T00:30:15.801208+0000","last_undegraded":"2024-04-25T00:30:15.801208+0000","last_fullsized":"2024-04-25T00:30:15.801208+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4083,"log_dups_size":0,"ondisk_log_size":4083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:36:55.436661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21935,"num_write_kb":296816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.22","version":"74'5147","reported_seq":5272,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.738948+0000","last_change":"2024-04-25T00:01:19.508691+0000","last_active":"2024-04-25T00:30:26.738948+0000","last_peered":"2024-04-25T00:30:26.738948+0000","last_clean":"2024-04-25T00:30:26.738948+0000","last_became_active":"2024-04-25T00:01:19.508123+0000","last_became_peered":"2024-04-25T00:01:19.508123+0000","last_unstale":"2024-04-25T00:30:26.738948+0000","last_undegraded":"2024-04-25T00:30:26.738948+0000","last_fullsized":"2024-04-25T00:30:26.738948+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":5147,"log_dups_size":0,"ondisk_log_size":5147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:47:19.526845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":43,"num_write":27901,"num_write_kb":341497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"70'3848","reported_seq":3976,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.812551+0000","last_change":"2024-04-25T00:01:05.387057+0000","last_active":"2024-04-25T00:30:15.812551+0000","last_peered":"2024-04-25T00:30:15.812551+0000","last_clean":"2024-04-25T00:30:15.812551+0000","last_became_active":"2024-04-25T00:01:05.386724+0000","last_became_peered":"2024-04-25T00:01:05.386724+0000","last_unstale":"2024-04-25T00:30:15.812551+0000","last_undegraded":"2024-04-25T00:30:15.812551+0000","last_fullsized":"2024-04-25T00:30:15.812551+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3848,"log_dups_size":0,"ondisk_log_size":3848,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:55:09.073286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20848,"num_write_kb":276248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.23","version":"74'4386","reported_seq":4508,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.730888+0000","last_change":"2024-04-25T00:01:19.522760+0000","last_active":"2024-04-25T00:30:26.730888+0000","last_peered":"2024-04-25T00:30:26.730888+0000","last_clean":"2024-04-25T00:30:26.730888+0000","last_became_active":"2024-04-25T00:01:19.522380+0000","last_became_peered":"2024-04-25T00:01:19.522380+0000","last_unstale":"2024-04-25T00:30:26.730888+0000","last_undegraded":"2024-04-25T00:30:26.730888+0000","last_fullsized":"2024-04-25T00:30:26.730888+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4386,"log_dups_size":0,"ondisk_log_size":4386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:55:46.853565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":23395,"num_write_kb":294532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.25","version":"70'4349","reported_seq":4476,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.812426+0000","last_change":"2024-04-25T00:01:05.367296+0000","last_active":"2024-04-25T00:30:15.812426+0000","last_peered":"2024-04-25T00:30:15.812426+0000","last_clean":"2024-04-25T00:30:15.812426+0000","last_became_active":"2024-04-25T00:01:05.363838+0000","last_became_peered":"2024-04-25T00:01:05.363838+0000","last_unstale":"2024-04-25T00:30:15.812426+0000","last_undegraded":"2024-04-25T00:30:15.812426+0000","last_fullsized":"2024-04-25T00:30:15.812426+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4349,"log_dups_size":0,"ondisk_log_size":4349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:43:07.053074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24353,"num_write_kb":306782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"74'4505","reported_seq":4625,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.749021+0000","last_change":"2024-04-25T00:01:19.510902+0000","last_active":"2024-04-25T00:30:26.749021+0000","last_peered":"2024-04-25T00:30:26.749021+0000","last_clean":"2024-04-25T00:30:26.749021+0000","last_became_active":"2024-04-25T00:01:19.510008+0000","last_became_peered":"2024-04-25T00:01:19.510008+0000","last_unstale":"2024-04-25T00:30:26.749021+0000","last_undegraded":"2024-04-25T00:30:26.749021+0000","last_fullsized":"2024-04-25T00:30:26.749021+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4505,"log_dups_size":0,"ondisk_log_size":4505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:40:32.292140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24621,"num_write_kb":315853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"70'3623","reported_seq":3750,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.806689+0000","last_change":"2024-04-25T00:01:05.365761+0000","last_active":"2024-04-25T00:30:15.806689+0000","last_peered":"2024-04-25T00:30:15.806689+0000","last_clean":"2024-04-25T00:30:15.806689+0000","last_became_active":"2024-04-25T00:01:05.361119+0000","last_became_peered":"2024-04-25T00:01:05.361119+0000","last_unstale":"2024-04-25T00:30:15.806689+0000","last_undegraded":"2024-04-25T00:30:15.806689+0000","last_fullsized":"2024-04-25T00:30:15.806689+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3623,"log_dups_size":0,"ondisk_log_size":3623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20289,"num_write_kb":274005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.21","version":"74'4291","reported_seq":4414,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.777806+0000","last_change":"2024-04-25T00:01:19.518534+0000","last_active":"2024-04-25T00:30:26.777806+0000","last_peered":"2024-04-25T00:30:26.777806+0000","last_clean":"2024-04-25T00:30:26.777806+0000","last_became_active":"2024-04-25T00:01:19.518110+0000","last_became_peered":"2024-04-25T00:01:19.518110+0000","last_unstale":"2024-04-25T00:30:26.777806+0000","last_undegraded":"2024-04-25T00:30:26.777806+0000","last_fullsized":"2024-04-25T00:30:26.777806+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4291,"log_dups_size":0,"ondisk_log_size":4291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:05:09.219823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":23037,"num_write_kb":301131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"69'4983","reported_seq":5110,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.799145+0000","last_change":"2024-04-25T00:01:05.373464+0000","last_active":"2024-04-25T00:30:15.799145+0000","last_peered":"2024-04-25T00:30:15.799145+0000","last_clean":"2024-04-25T00:30:15.799145+0000","last_became_active":"2024-04-25T00:01:05.372611+0000","last_became_peered":"2024-04-25T00:01:05.372611+0000","last_unstale":"2024-04-25T00:30:15.799145+0000","last_undegraded":"2024-04-25T00:30:15.799145+0000","last_fullsized":"2024-04-25T00:30:15.799145+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4983,"log_dups_size":0,"ondisk_log_size":4983,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:31.456792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26776,"num_write_kb":349057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2e","version":"74'4567","reported_seq":4690,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.735238+0000","last_change":"2024-04-25T00:01:19.542475+0000","last_active":"2024-04-25T00:30:26.735238+0000","last_peered":"2024-04-25T00:30:26.735238+0000","last_clean":"2024-04-25T00:30:26.735238+0000","last_became_active":"2024-04-25T00:01:19.541877+0000","last_became_peered":"2024-04-25T00:01:19.541877+0000","last_unstale":"2024-04-25T00:30:26.735238+0000","last_undegraded":"2024-04-25T00:30:26.735238+0000","last_fullsized":"2024-04-25T00:30:26.735238+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4567,"log_dups_size":0,"ondisk_log_size":4567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:02:57.373014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":36,"num_write":24879,"num_write_kb":312940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.28","version":"70'4294","reported_seq":4421,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.804148+0000","last_change":"2024-04-25T00:01:05.372906+0000","last_active":"2024-04-25T00:30:15.804148+0000","last_peered":"2024-04-25T00:30:15.804148+0000","last_clean":"2024-04-25T00:30:15.804148+0000","last_became_active":"2024-04-25T00:01:05.372148+0000","last_became_peered":"2024-04-25T00:01:05.372148+0000","last_unstale":"2024-04-25T00:30:15.804148+0000","last_undegraded":"2024-04-25T00:30:15.804148+0000","last_fullsized":"2024-04-25T00:30:15.804148+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4294,"log_dups_size":0,"ondisk_log_size":4294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:43.615895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23157,"num_write_kb":289708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2f","version":"74'5194","reported_seq":5313,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.771951+0000","last_change":"2024-04-25T00:01:19.498515+0000","last_active":"2024-04-25T00:30:26.771951+0000","last_peered":"2024-04-25T00:30:26.771951+0000","last_clean":"2024-04-25T00:30:26.771951+0000","last_became_active":"2024-04-25T00:01:19.496310+0000","last_became_peered":"2024-04-25T00:01:19.496310+0000","last_unstale":"2024-04-25T00:30:26.771951+0000","last_undegraded":"2024-04-25T00:30:26.771951+0000","last_fullsized":"2024-04-25T00:30:26.771951+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":5194,"log_dups_size":0,"ondisk_log_size":5194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:20:06.661558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":28420,"num_write_kb":367977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"69'4077","reported_seq":4204,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.795788+0000","last_change":"2024-04-25T00:01:05.366340+0000","last_active":"2024-04-25T00:30:15.795788+0000","last_peered":"2024-04-25T00:30:15.795788+0000","last_clean":"2024-04-25T00:30:15.795788+0000","last_became_active":"2024-04-25T00:01:05.361952+0000","last_became_peered":"2024-04-25T00:01:05.361952+0000","last_unstale":"2024-04-25T00:30:15.795788+0000","last_undegraded":"2024-04-25T00:30:15.795788+0000","last_fullsized":"2024-04-25T00:30:15.795788+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4077,"log_dups_size":0,"ondisk_log_size":4077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:41.948662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22153,"num_write_kb":297857,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2c","version":"74'3564","reported_seq":3684,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.750933+0000","last_change":"2024-04-25T00:01:19.508257+0000","last_active":"2024-04-25T00:30:26.750933+0000","last_peered":"2024-04-25T00:30:26.750933+0000","last_clean":"2024-04-25T00:30:26.750933+0000","last_became_active":"2024-04-25T00:01:19.507516+0000","last_became_peered":"2024-04-25T00:01:19.507516+0000","last_unstale":"2024-04-25T00:30:26.750933+0000","last_undegraded":"2024-04-25T00:30:26.750933+0000","last_fullsized":"2024-04-25T00:30:26.750933+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3564,"log_dups_size":0,"ondisk_log_size":3564,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:45.121513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":19793,"num_write_kb":247542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"69'4229","reported_seq":4356,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.942181+0000","last_change":"2024-04-25T00:01:05.387457+0000","last_active":"2024-04-25T00:30:15.942181+0000","last_peered":"2024-04-25T00:30:15.942181+0000","last_clean":"2024-04-25T00:30:15.942181+0000","last_became_active":"2024-04-25T00:01:05.386919+0000","last_became_peered":"2024-04-25T00:01:05.386919+0000","last_unstale":"2024-04-25T00:30:15.942181+0000","last_undegraded":"2024-04-25T00:30:15.942181+0000","last_fullsized":"2024-04-25T00:30:15.942181+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4229,"log_dups_size":0,"ondisk_log_size":4229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22430,"num_write_kb":302838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2d","version":"74'4662","reported_seq":4781,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.561243+0000","last_change":"2024-04-25T00:01:19.520648+0000","last_active":"2024-04-25T00:30:26.561243+0000","last_peered":"2024-04-25T00:30:26.561243+0000","last_clean":"2024-04-25T00:30:26.561243+0000","last_became_active":"2024-04-25T00:01:19.519303+0000","last_became_peered":"2024-04-25T00:01:19.519303+0000","last_unstale":"2024-04-25T00:30:26.561243+0000","last_undegraded":"2024-04-25T00:30:26.561243+0000","last_fullsized":"2024-04-25T00:30:26.561243+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4662,"log_dups_size":0,"ondisk_log_size":4662,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:23.993729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":25332,"num_write_kb":310579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2b","version":"70'4434","reported_seq":4562,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.792469+0000","last_change":"2024-04-25T00:01:05.366249+0000","last_active":"2024-04-25T00:30:15.792469+0000","last_peered":"2024-04-25T00:30:15.792469+0000","last_clean":"2024-04-25T00:30:15.792469+0000","last_became_active":"2024-04-25T00:01:05.365548+0000","last_became_peered":"2024-04-25T00:01:05.365548+0000","last_unstale":"2024-04-25T00:30:15.792469+0000","last_undegraded":"2024-04-25T00:30:15.792469+0000","last_fullsized":"2024-04-25T00:30:15.792469+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4434,"log_dups_size":0,"ondisk_log_size":4434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:22:21.143639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":24224,"num_write_kb":326776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"74'4527","reported_seq":4647,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.719005+0000","last_change":"2024-04-25T00:01:19.514659+0000","last_active":"2024-04-25T00:30:26.719005+0000","last_peered":"2024-04-25T00:30:26.719005+0000","last_clean":"2024-04-25T00:30:26.719005+0000","last_became_active":"2024-04-25T00:01:19.514232+0000","last_became_peered":"2024-04-25T00:01:19.514232+0000","last_unstale":"2024-04-25T00:30:26.719005+0000","last_undegraded":"2024-04-25T00:30:26.719005+0000","last_fullsized":"2024-04-25T00:30:26.719005+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4527,"log_dups_size":0,"ondisk_log_size":4527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:45.121513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":25088,"num_write_kb":313806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"70'3677","reported_seq":3789,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.802932+0000","last_change":"2024-04-25T00:01:46.665943+0000","last_active":"2024-04-25T00:30:15.802932+0000","last_peered":"2024-04-25T00:30:15.802932+0000","last_clean":"2024-04-25T00:30:15.802932+0000","last_became_active":"2024-04-25T00:01:46.663874+0000","last_became_peered":"2024-04-25T00:01:46.663874+0000","last_unstale":"2024-04-25T00:30:15.802932+0000","last_undegraded":"2024-04-25T00:30:15.802932+0000","last_fullsized":"2024-04-25T00:30:15.802932+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3677,"log_dups_size":0,"ondisk_log_size":3677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:53:00.633407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20325,"num_write_kb":253590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2b","version":"74'4916","reported_seq":5040,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.774860+0000","last_change":"2024-04-25T00:01:19.503419+0000","last_active":"2024-04-25T00:30:26.774860+0000","last_peered":"2024-04-25T00:30:26.774860+0000","last_clean":"2024-04-25T00:30:26.774860+0000","last_became_active":"2024-04-25T00:01:19.500568+0000","last_became_peered":"2024-04-25T00:01:19.500568+0000","last_unstale":"2024-04-25T00:30:26.774860+0000","last_undegraded":"2024-04-25T00:30:26.774860+0000","last_fullsized":"2024-04-25T00:30:26.774860+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4916,"log_dups_size":0,"ondisk_log_size":4916,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:08.508741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":26841,"num_write_kb":321097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"69'4618","reported_seq":4745,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.814383+0000","last_change":"2024-04-25T00:01:05.370653+0000","last_active":"2024-04-25T00:30:15.814383+0000","last_peered":"2024-04-25T00:30:15.814383+0000","last_clean":"2024-04-25T00:30:15.814383+0000","last_became_active":"2024-04-25T00:01:05.369919+0000","last_became_peered":"2024-04-25T00:01:05.369919+0000","last_unstale":"2024-04-25T00:30:15.814383+0000","last_undegraded":"2024-04-25T00:30:15.814383+0000","last_fullsized":"2024-04-25T00:30:15.814383+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4618,"log_dups_size":0,"ondisk_log_size":4618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:30:08.311326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25225,"num_write_kb":326547,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.28","version":"74'3900","reported_seq":4020,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.755166+0000","last_change":"2024-04-25T00:01:19.540104+0000","last_active":"2024-04-25T00:30:26.755166+0000","last_peered":"2024-04-25T00:30:26.755166+0000","last_clean":"2024-04-25T00:30:26.755166+0000","last_became_active":"2024-04-25T00:01:19.539775+0000","last_became_peered":"2024-04-25T00:01:19.539775+0000","last_unstale":"2024-04-25T00:30:26.755166+0000","last_undegraded":"2024-04-25T00:30:26.755166+0000","last_fullsized":"2024-04-25T00:30:26.755166+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3900,"log_dups_size":0,"ondisk_log_size":3900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:53:14.768520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21234,"num_write_kb":281892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"70'3829","reported_seq":3955,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.804122+0000","last_change":"2024-04-25T00:01:05.367001+0000","last_active":"2024-04-25T00:30:15.804122+0000","last_peered":"2024-04-25T00:30:15.804122+0000","last_clean":"2024-04-25T00:30:15.804122+0000","last_became_active":"2024-04-25T00:01:05.366264+0000","last_became_peered":"2024-04-25T00:01:05.366264+0000","last_unstale":"2024-04-25T00:30:15.804122+0000","last_undegraded":"2024-04-25T00:30:15.804122+0000","last_fullsized":"2024-04-25T00:30:15.804122+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3829,"log_dups_size":0,"ondisk_log_size":3829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19822,"num_write_kb":292385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.29","version":"74'4218","reported_seq":4338,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.734841+0000","last_change":"2024-04-25T00:01:19.497247+0000","last_active":"2024-04-25T00:30:26.734841+0000","last_peered":"2024-04-25T00:30:26.734841+0000","last_clean":"2024-04-25T00:30:26.734841+0000","last_became_active":"2024-04-25T00:01:19.496517+0000","last_became_peered":"2024-04-25T00:01:19.496517+0000","last_unstale":"2024-04-25T00:30:26.734841+0000","last_undegraded":"2024-04-25T00:30:26.734841+0000","last_fullsized":"2024-04-25T00:30:26.734841+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4218,"log_dups_size":0,"ondisk_log_size":4218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:30.129056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":23201,"num_write_kb":293725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2f","version":"70'5230","reported_seq":5357,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.813628+0000","last_change":"2024-04-25T00:01:05.379958+0000","last_active":"2024-04-25T00:30:15.813628+0000","last_peered":"2024-04-25T00:30:15.813628+0000","last_clean":"2024-04-25T00:30:15.813628+0000","last_became_active":"2024-04-25T00:01:05.379090+0000","last_became_peered":"2024-04-25T00:01:05.379090+0000","last_unstale":"2024-04-25T00:30:15.813628+0000","last_undegraded":"2024-04-25T00:30:15.813628+0000","last_fullsized":"2024-04-25T00:30:15.813628+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":5230,"log_dups_size":0,"ondisk_log_size":5230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:59:50.832958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":27866,"num_write_kb":365657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.36","version":"74'4384","reported_seq":4504,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.681328+0000","last_change":"2024-04-25T00:01:19.530176+0000","last_active":"2024-04-25T00:30:26.681328+0000","last_peered":"2024-04-25T00:30:26.681328+0000","last_clean":"2024-04-25T00:30:26.681328+0000","last_became_active":"2024-04-25T00:01:19.529410+0000","last_became_peered":"2024-04-25T00:01:19.529410+0000","last_unstale":"2024-04-25T00:30:26.681328+0000","last_undegraded":"2024-04-25T00:30:26.681328+0000","last_fullsized":"2024-04-25T00:30:26.681328+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4384,"log_dups_size":0,"ondisk_log_size":4384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:40:13.204880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":7,"num_write":24230,"num_write_kb":310193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.30","version":"69'4051","reported_seq":4177,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.803573+0000","last_change":"2024-04-25T00:01:05.370826+0000","last_active":"2024-04-25T00:30:15.803573+0000","last_peered":"2024-04-25T00:30:15.803573+0000","last_clean":"2024-04-25T00:30:15.803573+0000","last_became_active":"2024-04-25T00:01:05.369848+0000","last_became_peered":"2024-04-25T00:01:05.369848+0000","last_unstale":"2024-04-25T00:30:15.803573+0000","last_undegraded":"2024-04-25T00:30:15.803573+0000","last_fullsized":"2024-04-25T00:30:15.803573+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4051,"log_dups_size":0,"ondisk_log_size":4051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:23.870121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21793,"num_write_kb":288101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.37","version":"74'6689","reported_seq":6811,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.769600+0000","last_change":"2024-04-25T00:01:19.511625+0000","last_active":"2024-04-25T00:30:26.769600+0000","last_peered":"2024-04-25T00:30:26.769600+0000","last_clean":"2024-04-25T00:30:26.769600+0000","last_became_active":"2024-04-25T00:01:19.511349+0000","last_became_peered":"2024-04-25T00:01:19.511349+0000","last_unstale":"2024-04-25T00:30:26.769600+0000","last_undegraded":"2024-04-25T00:30:26.769600+0000","last_fullsized":"2024-04-25T00:30:26.769600+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":6689,"log_dups_size":0,"ondisk_log_size":6689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:02:57.373014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":8,"num_write":38367,"num_write_kb":445068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"70'3993","reported_seq":4119,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.795313+0000","last_change":"2024-04-25T00:01:05.390341+0000","last_active":"2024-04-25T00:30:15.795313+0000","last_peered":"2024-04-25T00:30:15.795313+0000","last_clean":"2024-04-25T00:30:15.795313+0000","last_became_active":"2024-04-25T00:01:05.389707+0000","last_became_peered":"2024-04-25T00:01:05.389707+0000","last_unstale":"2024-04-25T00:30:15.795313+0000","last_undegraded":"2024-04-25T00:30:15.795313+0000","last_fullsized":"2024-04-25T00:30:15.795313+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3993,"log_dups_size":0,"ondisk_log_size":3993,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:54:21.138947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21619,"num_write_kb":287362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.34","version":"74'3411","reported_seq":3532,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.695377+0000","last_change":"2024-04-25T00:01:19.513630+0000","last_active":"2024-04-25T00:30:26.695377+0000","last_peered":"2024-04-25T00:30:26.695377+0000","last_clean":"2024-04-25T00:30:26.695377+0000","last_became_active":"2024-04-25T00:01:19.512978+0000","last_became_peered":"2024-04-25T00:01:19.512978+0000","last_unstale":"2024-04-25T00:30:26.695377+0000","last_undegraded":"2024-04-25T00:30:26.695377+0000","last_fullsized":"2024-04-25T00:30:26.695377+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3411,"log_dups_size":0,"ondisk_log_size":3411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:57.751007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":34,"num_write":18670,"num_write_kb":234685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"70'3759","reported_seq":3886,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.809965+0000","last_change":"2024-04-25T00:01:05.378373+0000","last_active":"2024-04-25T00:30:15.809965+0000","last_peered":"2024-04-25T00:30:15.809965+0000","last_clean":"2024-04-25T00:30:15.809965+0000","last_became_active":"2024-04-25T00:01:05.376570+0000","last_became_peered":"2024-04-25T00:01:05.376570+0000","last_unstale":"2024-04-25T00:30:15.809965+0000","last_undegraded":"2024-04-25T00:30:15.809965+0000","last_fullsized":"2024-04-25T00:30:15.809965+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3759,"log_dups_size":0,"ondisk_log_size":3759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:30.228060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20571,"num_write_kb":279348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.35","version":"74'4078","reported_seq":4199,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.645015+0000","last_change":"2024-04-25T00:01:19.516716+0000","last_active":"2024-04-25T00:30:26.645015+0000","last_peered":"2024-04-25T00:30:26.645015+0000","last_clean":"2024-04-25T00:30:26.645015+0000","last_became_active":"2024-04-25T00:01:19.516142+0000","last_became_peered":"2024-04-25T00:01:19.516142+0000","last_unstale":"2024-04-25T00:30:26.645015+0000","last_undegraded":"2024-04-25T00:30:26.645015+0000","last_fullsized":"2024-04-25T00:30:26.645015+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4078,"log_dups_size":0,"ondisk_log_size":4078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:28:18.234546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":22241,"num_write_kb":273608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.33","version":"70'3191","reported_seq":3317,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.810448+0000","last_change":"2024-04-25T00:01:05.381011+0000","last_active":"2024-04-25T00:30:15.810448+0000","last_peered":"2024-04-25T00:30:15.810448+0000","last_clean":"2024-04-25T00:30:15.810448+0000","last_became_active":"2024-04-25T00:01:05.380236+0000","last_became_peered":"2024-04-25T00:01:05.380236+0000","last_unstale":"2024-04-25T00:30:15.810448+0000","last_undegraded":"2024-04-25T00:30:15.810448+0000","last_fullsized":"2024-04-25T00:30:15.810448+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3191,"log_dups_size":0,"ondisk_log_size":3191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:06:19.307378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17526,"num_write_kb":230308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"74'4055","reported_seq":4174,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.756767+0000","last_change":"2024-04-25T00:01:19.529023+0000","last_active":"2024-04-25T00:30:26.756767+0000","last_peered":"2024-04-25T00:30:26.756767+0000","last_clean":"2024-04-25T00:30:26.756767+0000","last_became_active":"2024-04-25T00:01:19.524609+0000","last_became_peered":"2024-04-25T00:01:19.524609+0000","last_unstale":"2024-04-25T00:30:26.756767+0000","last_undegraded":"2024-04-25T00:30:26.756767+0000","last_fullsized":"2024-04-25T00:30:26.756767+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4055,"log_dups_size":0,"ondisk_log_size":4055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:41:19.150476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":21801,"num_write_kb":276048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.34","version":"70'2711","reported_seq":2837,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.792324+0000","last_change":"2024-04-25T00:01:05.385853+0000","last_active":"2024-04-25T00:30:15.792324+0000","last_peered":"2024-04-25T00:30:15.792324+0000","last_clean":"2024-04-25T00:30:15.792324+0000","last_became_active":"2024-04-25T00:01:05.384997+0000","last_became_peered":"2024-04-25T00:01:05.384997+0000","last_unstale":"2024-04-25T00:30:15.792324+0000","last_undegraded":"2024-04-25T00:30:15.792324+0000","last_fullsized":"2024-04-25T00:30:15.792324+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":2711,"log_dups_size":0,"ondisk_log_size":2711,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:36:55.436661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14839,"num_write_kb":207133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.33","version":"74'3434","reported_seq":3556,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.729873+0000","last_change":"2024-04-25T00:01:19.516867+0000","last_active":"2024-04-25T00:30:26.729873+0000","last_peered":"2024-04-25T00:30:26.729873+0000","last_clean":"2024-04-25T00:30:26.729873+0000","last_became_active":"2024-04-25T00:01:19.515764+0000","last_became_peered":"2024-04-25T00:01:19.515764+0000","last_unstale":"2024-04-25T00:30:26.729873+0000","last_undegraded":"2024-04-25T00:30:26.729873+0000","last_fullsized":"2024-04-25T00:30:26.729873+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3434,"log_dups_size":0,"ondisk_log_size":3434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:53:14.768520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":5,"num_write":19347,"num_write_kb":241776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"69'3819","reported_seq":3946,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.790737+0000","last_change":"2024-04-25T00:01:05.389424+0000","last_active":"2024-04-25T00:30:15.790737+0000","last_peered":"2024-04-25T00:30:15.790737+0000","last_clean":"2024-04-25T00:30:15.790737+0000","last_became_active":"2024-04-25T00:01:05.388999+0000","last_became_peered":"2024-04-25T00:01:05.388999+0000","last_unstale":"2024-04-25T00:30:15.790737+0000","last_undegraded":"2024-04-25T00:30:15.790737+0000","last_fullsized":"2024-04-25T00:30:15.790737+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3819,"log_dups_size":0,"ondisk_log_size":3819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:31.456792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":19986,"num_write_kb":272748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.30","version":"74'4335","reported_seq":4453,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.723246+0000","last_change":"2024-04-25T00:01:19.519365+0000","last_active":"2024-04-25T00:30:26.723246+0000","last_peered":"2024-04-25T00:30:26.723246+0000","last_clean":"2024-04-25T00:30:26.723246+0000","last_became_active":"2024-04-25T00:01:19.519085+0000","last_became_peered":"2024-04-25T00:01:19.519085+0000","last_unstale":"2024-04-25T00:30:26.723246+0000","last_undegraded":"2024-04-25T00:30:26.723246+0000","last_fullsized":"2024-04-25T00:30:26.723246+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4335,"log_dups_size":0,"ondisk_log_size":4335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:05:09.219823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23507,"num_write_kb":286989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"69'4866","reported_seq":4992,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.790509+0000","last_change":"2024-04-25T00:01:05.368974+0000","last_active":"2024-04-25T00:30:15.790509+0000","last_peered":"2024-04-25T00:30:15.790509+0000","last_clean":"2024-04-25T00:30:15.790509+0000","last_became_active":"2024-04-25T00:01:05.367893+0000","last_became_peered":"2024-04-25T00:01:05.367893+0000","last_unstale":"2024-04-25T00:30:15.790509+0000","last_undegraded":"2024-04-25T00:30:15.790509+0000","last_fullsized":"2024-04-25T00:30:15.790509+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4866,"log_dups_size":0,"ondisk_log_size":4866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:30:23.557381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26504,"num_write_kb":327449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.31","version":"74'4017","reported_seq":4136,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.738931+0000","last_change":"2024-04-25T00:01:19.513514+0000","last_active":"2024-04-25T00:30:26.738931+0000","last_peered":"2024-04-25T00:30:26.738931+0000","last_clean":"2024-04-25T00:30:26.738931+0000","last_became_active":"2024-04-25T00:01:19.513184+0000","last_became_peered":"2024-04-25T00:01:19.513184+0000","last_unstale":"2024-04-25T00:30:26.738931+0000","last_undegraded":"2024-04-25T00:30:26.738931+0000","last_fullsized":"2024-04-25T00:30:26.738931+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4017,"log_dups_size":0,"ondisk_log_size":4017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:47.859626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":21984,"num_write_kb":288846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"70'5540","reported_seq":5666,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.944163+0000","last_change":"2024-04-25T00:01:05.362303+0000","last_active":"2024-04-25T00:30:15.944163+0000","last_peered":"2024-04-25T00:30:15.944163+0000","last_clean":"2024-04-25T00:30:15.944163+0000","last_became_active":"2024-04-25T00:01:05.361586+0000","last_became_peered":"2024-04-25T00:01:05.361586+0000","last_unstale":"2024-04-25T00:30:15.944163+0000","last_undegraded":"2024-04-25T00:30:15.944163+0000","last_fullsized":"2024-04-25T00:30:15.944163+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":5540,"log_dups_size":0,"ondisk_log_size":5540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:30.228060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30654,"num_write_kb":393416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3e","version":"74'5058","reported_seq":5180,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.759205+0000","last_change":"2024-04-25T00:01:19.515101+0000","last_active":"2024-04-25T00:30:26.759205+0000","last_peered":"2024-04-25T00:30:26.759205+0000","last_clean":"2024-04-25T00:30:26.759205+0000","last_became_active":"2024-04-25T00:01:19.513314+0000","last_became_peered":"2024-04-25T00:01:19.513314+0000","last_unstale":"2024-04-25T00:30:26.759205+0000","last_undegraded":"2024-04-25T00:30:26.759205+0000","last_fullsized":"2024-04-25T00:30:26.759205+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":5058,"log_dups_size":0,"ondisk_log_size":5058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:50.351599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":15,"num_write":27439,"num_write_kb":359791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.38","version":"70'4026","reported_seq":4151,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.793659+0000","last_change":"2024-04-25T00:01:05.390256+0000","last_active":"2024-04-25T00:30:15.793659+0000","last_peered":"2024-04-25T00:30:15.793659+0000","last_clean":"2024-04-25T00:30:15.793659+0000","last_became_active":"2024-04-25T00:01:05.389724+0000","last_became_peered":"2024-04-25T00:01:05.389724+0000","last_unstale":"2024-04-25T00:30:15.793659+0000","last_undegraded":"2024-04-25T00:30:15.793659+0000","last_fullsized":"2024-04-25T00:30:15.793659+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4026,"log_dups_size":0,"ondisk_log_size":4026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:09:18.315688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21352,"num_write_kb":290527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3f","version":"74'4365","reported_seq":4489,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.773739+0000","last_change":"2024-04-25T00:01:19.511493+0000","last_active":"2024-04-25T00:30:26.773739+0000","last_peered":"2024-04-25T00:30:26.773739+0000","last_clean":"2024-04-25T00:30:26.773739+0000","last_became_active":"2024-04-25T00:01:19.508049+0000","last_became_peered":"2024-04-25T00:01:19.508049+0000","last_unstale":"2024-04-25T00:30:26.773739+0000","last_undegraded":"2024-04-25T00:30:26.773739+0000","last_fullsized":"2024-04-25T00:30:26.773739+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4365,"log_dups_size":0,"ondisk_log_size":4365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:55:46.853565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":8,"num_write":24611,"num_write_kb":307761,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"70'3970","reported_seq":4095,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.790756+0000","last_change":"2024-04-25T00:01:05.374766+0000","last_active":"2024-04-25T00:30:15.790756+0000","last_peered":"2024-04-25T00:30:15.790756+0000","last_clean":"2024-04-25T00:30:15.790756+0000","last_became_active":"2024-04-25T00:01:05.369739+0000","last_became_peered":"2024-04-25T00:01:05.369739+0000","last_unstale":"2024-04-25T00:30:15.790756+0000","last_undegraded":"2024-04-25T00:30:15.790756+0000","last_fullsized":"2024-04-25T00:30:15.790756+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3970,"log_dups_size":0,"ondisk_log_size":3970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:09:18.315688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22008,"num_write_kb":289131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3c","version":"74'3966","reported_seq":4087,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.664683+0000","last_change":"2024-04-25T00:01:19.521456+0000","last_active":"2024-04-25T00:30:26.664683+0000","last_peered":"2024-04-25T00:30:26.664683+0000","last_clean":"2024-04-25T00:30:26.664683+0000","last_became_active":"2024-04-25T00:01:19.520787+0000","last_became_peered":"2024-04-25T00:01:19.520787+0000","last_unstale":"2024-04-25T00:30:26.664683+0000","last_undegraded":"2024-04-25T00:30:26.664683+0000","last_fullsized":"2024-04-25T00:30:26.664683+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3966,"log_dups_size":0,"ondisk_log_size":3966,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:25:38.981364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22683,"num_write_kb":271349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3a","version":"70'3294","reported_seq":3419,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.805979+0000","last_change":"2024-04-25T00:01:05.359253+0000","last_active":"2024-04-25T00:30:15.805979+0000","last_peered":"2024-04-25T00:30:15.805979+0000","last_clean":"2024-04-25T00:30:15.805979+0000","last_became_active":"2024-04-25T00:01:05.358477+0000","last_became_peered":"2024-04-25T00:01:05.358477+0000","last_unstale":"2024-04-25T00:30:15.805979+0000","last_undegraded":"2024-04-25T00:30:15.805979+0000","last_fullsized":"2024-04-25T00:30:15.805979+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3294,"log_dups_size":0,"ondisk_log_size":3294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:22.739214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17420,"num_write_kb":225566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3d","version":"74'5153","reported_seq":5274,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.749265+0000","last_change":"2024-04-25T00:01:19.509505+0000","last_active":"2024-04-25T00:30:26.749265+0000","last_peered":"2024-04-25T00:30:26.749265+0000","last_clean":"2024-04-25T00:30:26.749265+0000","last_became_active":"2024-04-25T00:01:19.507812+0000","last_became_peered":"2024-04-25T00:01:19.507812+0000","last_unstale":"2024-04-25T00:30:26.749265+0000","last_undegraded":"2024-04-25T00:30:26.749265+0000","last_fullsized":"2024-04-25T00:30:26.749265+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":5153,"log_dups_size":0,"ondisk_log_size":5153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:23.993729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":28174,"num_write_kb":326190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3b","version":"70'3663","reported_seq":3788,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.792919+0000","last_change":"2024-04-25T00:01:05.386068+0000","last_active":"2024-04-25T00:30:15.792919+0000","last_peered":"2024-04-25T00:30:15.792919+0000","last_clean":"2024-04-25T00:30:15.792919+0000","last_became_active":"2024-04-25T00:01:05.378650+0000","last_became_peered":"2024-04-25T00:01:05.378650+0000","last_unstale":"2024-04-25T00:30:15.792919+0000","last_undegraded":"2024-04-25T00:30:15.792919+0000","last_fullsized":"2024-04-25T00:30:15.792919+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3663,"log_dups_size":0,"ondisk_log_size":3663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:12:52.846250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18938,"num_write_kb":284951,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3a","version":"74'3833","reported_seq":3950,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.611583+0000","last_change":"2024-04-25T00:01:19.514843+0000","last_active":"2024-04-25T00:30:26.611583+0000","last_peered":"2024-04-25T00:30:26.611583+0000","last_clean":"2024-04-25T00:30:26.611583+0000","last_became_active":"2024-04-25T00:01:19.513956+0000","last_became_peered":"2024-04-25T00:01:19.513956+0000","last_unstale":"2024-04-25T00:30:26.611583+0000","last_undegraded":"2024-04-25T00:30:26.611583+0000","last_fullsized":"2024-04-25T00:30:26.611583+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3833,"log_dups_size":0,"ondisk_log_size":3833,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:38.005233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20679,"num_write_kb":271382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"69'3516","reported_seq":3645,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.808655+0000","last_change":"2024-04-25T00:01:05.359716+0000","last_active":"2024-04-25T00:30:15.808655+0000","last_peered":"2024-04-25T00:30:15.808655+0000","last_clean":"2024-04-25T00:30:15.808655+0000","last_became_active":"2024-04-25T00:01:05.358866+0000","last_became_peered":"2024-04-25T00:01:05.358866+0000","last_unstale":"2024-04-25T00:30:15.808655+0000","last_undegraded":"2024-04-25T00:30:15.808655+0000","last_fullsized":"2024-04-25T00:30:15.808655+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3516,"log_dups_size":0,"ondisk_log_size":3516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:50.489125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":19226,"num_write_kb":253281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3b","version":"74'4130","reported_seq":4248,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.756223+0000","last_change":"2024-04-25T00:01:19.499574+0000","last_active":"2024-04-25T00:30:26.756223+0000","last_peered":"2024-04-25T00:30:26.756223+0000","last_clean":"2024-04-25T00:30:26.756223+0000","last_became_active":"2024-04-25T00:01:19.498699+0000","last_became_peered":"2024-04-25T00:01:19.498699+0000","last_unstale":"2024-04-25T00:30:26.756223+0000","last_undegraded":"2024-04-25T00:30:26.756223+0000","last_fullsized":"2024-04-25T00:30:26.756223+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4130,"log_dups_size":0,"ondisk_log_size":4130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:47.859626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":3,"num_write":21749,"num_write_kb":288055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"70'3483","reported_seq":3611,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.798639+0000","last_change":"2024-04-25T00:01:05.371291+0000","last_active":"2024-04-25T00:30:15.798639+0000","last_peered":"2024-04-25T00:30:15.798639+0000","last_clean":"2024-04-25T00:30:15.798639+0000","last_became_active":"2024-04-25T00:01:05.370522+0000","last_became_peered":"2024-04-25T00:01:05.370522+0000","last_unstale":"2024-04-25T00:30:15.798639+0000","last_undegraded":"2024-04-25T00:30:15.798639+0000","last_fullsized":"2024-04-25T00:30:15.798639+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3483,"log_dups_size":0,"ondisk_log_size":3483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:11:27.044893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18326,"num_write_kb":261938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.38","version":"74'4120","reported_seq":4237,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.292105+0000","last_change":"2024-04-25T00:01:19.523798+0000","last_active":"2024-04-25T00:30:26.292105+0000","last_peered":"2024-04-25T00:30:26.292105+0000","last_clean":"2024-04-25T00:30:26.292105+0000","last_became_active":"2024-04-25T00:01:19.523450+0000","last_became_peered":"2024-04-25T00:01:19.523450+0000","last_unstale":"2024-04-25T00:30:26.292105+0000","last_undegraded":"2024-04-25T00:30:26.292105+0000","last_fullsized":"2024-04-25T00:30:26.292105+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4120,"log_dups_size":0,"ondisk_log_size":4120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:40:36.350832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22879,"num_write_kb":288519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3e","version":"70'4991","reported_seq":5119,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.801414+0000","last_change":"2024-04-25T00:01:05.368190+0000","last_active":"2024-04-25T00:30:15.801414+0000","last_peered":"2024-04-25T00:30:15.801414+0000","last_clean":"2024-04-25T00:30:15.801414+0000","last_became_active":"2024-04-25T00:01:05.362606+0000","last_became_peered":"2024-04-25T00:01:05.362606+0000","last_unstale":"2024-04-25T00:30:15.801414+0000","last_undegraded":"2024-04-25T00:30:15.801414+0000","last_fullsized":"2024-04-25T00:30:15.801414+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4991,"log_dups_size":0,"ondisk_log_size":4991,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:12:52.846250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26684,"num_write_kb":361051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.39","version":"74'4850","reported_seq":4967,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.763325+0000","last_change":"2024-04-25T00:01:19.505737+0000","last_active":"2024-04-25T00:30:26.763325+0000","last_peered":"2024-04-25T00:30:26.763325+0000","last_clean":"2024-04-25T00:30:26.763325+0000","last_became_active":"2024-04-25T00:01:19.504991+0000","last_became_peered":"2024-04-25T00:01:19.504991+0000","last_unstale":"2024-04-25T00:30:26.763325+0000","last_undegraded":"2024-04-25T00:30:26.763325+0000","last_fullsized":"2024-04-25T00:30:26.763325+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4850,"log_dups_size":0,"ondisk_log_size":4850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:47:19.526845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26612,"num_write_kb":323915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3f","version":"70'3815","reported_seq":3943,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.791882+0000","last_change":"2024-04-25T00:01:05.389596+0000","last_active":"2024-04-25T00:30:15.791882+0000","last_peered":"2024-04-25T00:30:15.791882+0000","last_clean":"2024-04-25T00:30:15.791882+0000","last_became_active":"2024-04-25T00:01:05.389127+0000","last_became_peered":"2024-04-25T00:01:05.389127+0000","last_unstale":"2024-04-25T00:30:15.791882+0000","last_undegraded":"2024-04-25T00:30:15.791882+0000","last_fullsized":"2024-04-25T00:30:15.791882+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3815,"log_dups_size":0,"ondisk_log_size":3815,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:09.858617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20941,"num_write_kb":280561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":105,"num_read_kb":408,"num_write":1543888,"num_write_kb":19306803,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":280555,"ondisk_log_size":280555,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":19554860,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1733,"num_read_kb":229197,"num_write":5129,"num_write_kb":21277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":39174144,"data_stored":39109720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4705,"ondisk_log_size":4705,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":3,"num_write":1412500,"num_write_kb":18776370,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":261221,"ondisk_log_size":261221,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":19117950,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1590,"num_read_kb":257618,"num_write":5060,"num_write_kb":20770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":38297600,"data_stored":38235900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4699,"ondisk_log_size":4699,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509801,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":674532,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":673470,"kb_avail":93697308,"statfs":{"total":96636764160,"available":95946043392,"internally_reserved":0,"allocated":1077248,"data_stored":617327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":689633739},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509801,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":843472,"kb_used_data":11784,"kb_used_omap":1,"kb_used_meta":831678,"kb_avail":93528368,"statfs":{"total":96636764160,"available":95773048832,"internally_reserved":0,"allocated":12066816,"data_stored":11609303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":851638726},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509800,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":870568,"kb_used_data":18016,"kb_used_omap":1,"kb_used_meta":852542,"kb_avail":93501272,"statfs":{"total":96636764160,"available":95745302528,"internally_reserved":0,"allocated":18448384,"data_stored":17985439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":873003461},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509801,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":922240,"kb_used_data":15544,"kb_used_omap":1,"kb_used_meta":906686,"kb_avail":93449600,"statfs":{"total":96636764160,"available":95692390400,"internally_reserved":0,"allocated":15917056,"data_stored":15447989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":928446919},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509802,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1076840,"kb_used_data":15968,"kb_used_omap":1,"kb_used_meta":1060862,"kb_avail":93295000,"statfs":{"total":96636764160,"available":95534080000,"internally_reserved":0,"allocated":16351232,"data_stored":15888287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1086323141},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509801,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":788556,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":787006,"kb_avail":93583284,"statfs":{"total":96636764160,"available":95829282816,"internally_reserved":0,"allocated":1576960,"data_stored":1119003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":805894597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509801,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1028004,"kb_used_data":3164,"kb_used_omap":1,"kb_used_meta":1024830,"kb_avail":93343836,"statfs":{"total":96636764160,"available":95584088064,"internally_reserved":0,"allocated":3239936,"data_stored":2779219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1049426373},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575209,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":962780,"kb_used_data":17364,"kb_used_omap":1,"kb_used_meta":945406,"kb_avail":93409060,"statfs":{"total":96636764160,"available":95650877440,"internally_reserved":0,"allocated":17780736,"data_stored":17307749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":968096197},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2260992,"data_stored":2250634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8396800,"data_stored":8389280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":10645504,"data_stored":10639768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8404992,"data_stored":8389952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4292608,"data_stored":4283082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":88810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6979584,"data_stored":6970512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4296704,"data_stored":4279726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8400896,"data_stored":8389302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12587008,"data_stored":12583494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":6897664,"data_stored":6886504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T00:30:41.774 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-25T00:30:41.775 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T00:30:41.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.898+0000 7fe43e873640 1 Processor -- start 2024-04-25T00:30:41.898 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.898+0000 7fe43e873640 1 -- start start 2024-04-25T00:30:41.899 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.898+0000 7fe43e873640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe438152d80 0x7fe43806bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:41.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.898+0000 7fe43e873640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4380c9150 con 0x7fe43806c290 2024-04-25T00:30:41.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.898+0000 7fe43e873640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4380c9430 con 0x7fe43806d5f0 2024-04-25T00:30:41.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.898+0000 7fe43e873640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fe4380c9d60 con 0x7fe438152d80 2024-04-25T00:30:41.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.898+0000 7fe437fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fe438152d80 0x7fe43806bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:41.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.898+0000 7fe43ce10640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fe43806c290 0x7fe438157390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57426/0 (socket says 172.21.15.116:57426) 2024-04-25T00:30:41.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.898+0000 7fe43ce10640 1 -- 172.21.15.116:0/3859939796 learned_addr learned my addr 172.21.15.116:0/3859939796 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:30:41.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.898+0000 7fe437fff640 1 -- 172.21.15.116:0/3859939796 >> v1:172.21.15.174:6789/0 conn(0x7fe43806d5f0 legacy=0x7fe438156c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:41.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.898+0000 7fe437fff640 1 -- 172.21.15.116:0/3859939796 >> v1:172.21.15.116:6789/0 conn(0x7fe43806c290 legacy=0x7fe438157390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:41.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.898+0000 7fe437fff640 1 -- 172.21.15.116:0/3859939796 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4380c9710 con 0x7fe438152d80 2024-04-25T00:30:41.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.902+0000 7fe437fff640 1 --2- 172.21.15.116:0/3859939796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe438152d80 0x7fe43806bd50 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7fe428009dc0 tx=0x7fe42801c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ff3af70cf3b72d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:41.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.902+0000 7fe436ffd640 1 -- 172.21.15.116:0/3859939796 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4280b2070 con 0x7fe438152d80 2024-04-25T00:30:41.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.902+0000 7fe436ffd640 1 -- 172.21.15.116:0/3859939796 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4280a57e0 con 0x7fe438152d80 2024-04-25T00:30:41.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.902+0000 7fe436ffd640 1 -- 172.21.15.116:0/3859939796 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4280a5bb0 con 0x7fe438152d80 2024-04-25T00:30:41.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.902+0000 7fe43e873640 1 -- 172.21.15.116:0/3859939796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe438152d80 msgr2=0x7fe43806bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:41.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.902+0000 7fe43e873640 1 --2- 172.21.15.116:0/3859939796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe438152d80 0x7fe43806bd50 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7fe428009dc0 tx=0x7fe42801c800 comp rx=0 tx=0).stop 2024-04-25T00:30:41.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.902+0000 7fe43e873640 1 -- 172.21.15.116:0/3859939796 shutdown_connections 2024-04-25T00:30:41.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.902+0000 7fe43e873640 1 --2- 172.21.15.116:0/3859939796 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe438152d80 0x7fe43806bd50 unknown :-1 s=CLOSED pgs=1803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:41.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.902+0000 7fe43e873640 1 -- 172.21.15.116:0/3859939796 >> 172.21.15.116:0/3859939796 conn(0x7fe4380acdd0 msgr2=0x7fe4380ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:41.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.902+0000 7fe43e873640 1 -- 172.21.15.116:0/3859939796 shutdown_connections 2024-04-25T00:30:41.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.902+0000 7fe43e873640 1 -- 172.21.15.116:0/3859939796 wait complete. 2024-04-25T00:30:41.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.902+0000 7fe43e873640 1 Processor -- start 2024-04-25T00:30:41.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.902+0000 7fe43e873640 1 -- start start 2024-04-25T00:30:41.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.902+0000 7fe43e873640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe43806c290 0x7fe43814fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:41.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.902+0000 7fe43e873640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe43806d5f0 0x7fe4381f1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:41.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.902+0000 7fe43e873640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe438152d80 0x7fe438151700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:41.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.902+0000 7fe43e873640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe438163a60 con 0x7fe438152d80 2024-04-25T00:30:41.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.902+0000 7fe43e873640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe4381638e0 con 0x7fe43806d5f0 2024-04-25T00:30:41.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.902+0000 7fe43e873640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe438163be0 con 0x7fe43806c290 2024-04-25T00:30:41.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.902+0000 7fe43ce10640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe438152d80 0x7fe438151700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:41.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.902+0000 7fe43ce10640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe438152d80 0x7fe438151700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36482/0 (socket says 172.21.15.116:36482) 2024-04-25T00:30:41.907 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.902+0000 7fe43ce10640 1 -- 172.21.15.116:0/354696124 learned_addr learned my addr 172.21.15.116:0/354696124 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:41.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.902+0000 7fe4377fe640 1 --2- 172.21.15.116:0/354696124 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe43806d5f0 0x7fe4381f1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:41.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.902+0000 7fe437fff640 1 --2- 172.21.15.116:0/354696124 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe43806c290 0x7fe43814fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:41.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.902+0000 7fe43ce10640 1 -- 172.21.15.116:0/354696124 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe43806c290 msgr2=0x7fe43814fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:41.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.902+0000 7fe43ce10640 1 --2- 172.21.15.116:0/354696124 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe43806c290 0x7fe43814fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:41.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.902+0000 7fe43ce10640 1 -- 172.21.15.116:0/354696124 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe43806d5f0 msgr2=0x7fe4381f1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:41.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.902+0000 7fe43ce10640 1 --2- 172.21.15.116:0/354696124 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe43806d5f0 0x7fe4381f1fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:41.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.902+0000 7fe43ce10640 1 -- 172.21.15.116:0/354696124 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4380c68c0 con 0x7fe438152d80 2024-04-25T00:30:41.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.902+0000 7fe437fff640 1 --2- 172.21.15.116:0/354696124 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe43806c290 0x7fe43814fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:30:41.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.902+0000 7fe43ce10640 1 --2- 172.21.15.116:0/354696124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe438152d80 0x7fe438151700 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7fe43000c3f0 tx=0x7fe43000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:41.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.902+0000 7fe434ff9640 1 -- 172.21.15.116:0/354696124 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe430017070 con 0x7fe438152d80 2024-04-25T00:30:41.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.906+0000 7fe434ff9640 1 -- 172.21.15.116:0/354696124 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe430005cf0 con 0x7fe438152d80 2024-04-25T00:30:41.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.906+0000 7fe43e873640 1 -- 172.21.15.116:0/354696124 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4380c6ba0 con 0x7fe438152d80 2024-04-25T00:30:41.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.906+0000 7fe434ff9640 1 -- 172.21.15.116:0/354696124 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4300072f0 con 0x7fe438152d80 2024-04-25T00:30:41.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.906+0000 7fe43e873640 1 -- 172.21.15.116:0/354696124 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe43814ffa0 con 0x7fe438152d80 2024-04-25T00:30:41.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.906+0000 7fe43e873640 1 -- 172.21.15.116:0/354696124 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe438068350 con 0x7fe438152d80 2024-04-25T00:30:41.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.906+0000 7fe434ff9640 1 -- 172.21.15.116:0/354696124 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe43001a070 con 0x7fe438152d80 2024-04-25T00:30:41.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.906+0000 7fe434ff9640 1 --2- 172.21.15.116:0/354696124 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe4180678f0 0x7fe418069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:41.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.906+0000 7fe434ff9640 1 -- 172.21.15.116:0/354696124 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7fe430086300 con 0x7fe438152d80 2024-04-25T00:30:41.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.906+0000 7fe437fff640 1 --2- 172.21.15.116:0/354696124 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe4180678f0 0x7fe418069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:41.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.906+0000 7fe437fff640 1 --2- 172.21.15.116:0/354696124 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe4180678f0 0x7fe418069db0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fe4280af000 tx=0x7fe4280a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:41.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:41.910+0000 7fe434ff9640 1 -- 172.21.15.116:0/354696124 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fe430055b50 con 0x7fe438152d80 2024-04-25T00:30:42.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.054+0000 7fe43e873640 1 -- 172.21.15.116:0/354696124 --> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe4380cab60 con 0x7fe4180678f0 2024-04-25T00:30:42.089 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.090+0000 7fe434ff9640 1 -- 172.21.15.116:0/354696124 <== mgr.4105 v2:172.21.15.174:6832/2004448211 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+493805 (secure 0 0 0) 0x7fe4380cab60 con 0x7fe4180678f0 2024-04-25T00:30:42.094 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:30:42.100 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-04-25T00:30:42.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.094+0000 7fe43e873640 1 -- 172.21.15.116:0/354696124 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe4180678f0 msgr2=0x7fe418069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:42.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.094+0000 7fe43e873640 1 --2- 172.21.15.116:0/354696124 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe4180678f0 0x7fe418069db0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fe4280af000 tx=0x7fe4280a7070 comp rx=0 tx=0).stop 2024-04-25T00:30:42.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.094+0000 7fe43e873640 1 -- 172.21.15.116:0/354696124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe438152d80 msgr2=0x7fe438151700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:42.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.094+0000 7fe43e873640 1 --2- 172.21.15.116:0/354696124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe438152d80 0x7fe438151700 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7fe43000c3f0 tx=0x7fe43000c8b0 comp rx=0 tx=0).stop 2024-04-25T00:30:42.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.094+0000 7fe43e873640 1 -- 172.21.15.116:0/354696124 shutdown_connections 2024-04-25T00:30:42.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.094+0000 7fe43e873640 1 --2- 172.21.15.116:0/354696124 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe4180678f0 0x7fe418069db0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:42.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.094+0000 7fe43e873640 1 --2- 172.21.15.116:0/354696124 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe438152d80 0x7fe438151700 unknown :-1 s=CLOSED pgs=1804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:42.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.094+0000 7fe43e873640 1 --2- 172.21.15.116:0/354696124 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe43806d5f0 0x7fe4381f1fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:42.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.094+0000 7fe43e873640 1 --2- 172.21.15.116:0/354696124 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe43806c290 0x7fe43814fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:42.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.094+0000 7fe43e873640 1 -- 172.21.15.116:0/354696124 >> 172.21.15.116:0/354696124 conn(0x7fe4380acdd0 msgr2=0x7fe43814d1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:42.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.094+0000 7fe43e873640 1 -- 172.21.15.116:0/354696124 shutdown_connections 2024-04-25T00:30:42.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.094+0000 7fe43e873640 1 -- 172.21.15.116:0/354696124 wait complete. 2024-04-25T00:30:42.110 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":963,"stamp":"2024-04-25T00:30:42.027981+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":39263178,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":3556,"num_read_kb":487326,"num_write":2966736,"num_write_kb":38127636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":551297,"ondisk_log_size":551297,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":7166992,"kb_used_data":84432,"kb_used_omap":12,"kb_used_meta":7082483,"kb_avail":747807728,"statfs":{"total":773094113280,"available":765755113472,"internally_reserved":0,"allocated":86458368,"data_stored":82754316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":7252463153},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-440752685,"num_objects":-128,"num_object_clones":0,"num_object_copies":-256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-128,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":135,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005848"},"pg_stats":[{"pgid":"5.18","version":"74'4491","reported_seq":4612,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.766890+0000","last_change":"2024-04-25T00:01:19.508728+0000","last_active":"2024-04-25T00:30:26.766890+0000","last_peered":"2024-04-25T00:30:26.766890+0000","last_clean":"2024-04-25T00:30:26.766890+0000","last_became_active":"2024-04-25T00:01:19.508390+0000","last_became_peered":"2024-04-25T00:01:19.508390+0000","last_unstale":"2024-04-25T00:30:26.766890+0000","last_undegraded":"2024-04-25T00:30:26.766890+0000","last_fullsized":"2024-04-25T00:30:26.766890+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4491,"log_dups_size":0,"ondisk_log_size":4491,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:45.591905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":12,"num_write":23917,"num_write_kb":321110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1f","version":"70'1087","reported_seq":1273,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.794182+0000","last_change":"2024-04-25T00:01:04.356032+0000","last_active":"2024-04-25T00:30:15.794182+0000","last_peered":"2024-04-25T00:30:15.794182+0000","last_clean":"2024-04-25T00:30:15.794182+0000","last_became_active":"2024-04-25T00:01:04.355293+0000","last_became_peered":"2024-04-25T00:01:04.355293+0000","last_unstale":"2024-04-25T00:30:15.794182+0000","last_undegraded":"2024-04-25T00:30:15.794182+0000","last_fullsized":"2024-04-25T00:30:15.794182+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1087,"log_dups_size":0,"ondisk_log_size":1087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:56:16.678132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":16397,"num_write":1061,"num_write_kb":4715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"70'3760","reported_seq":3888,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.801356+0000","last_change":"2024-04-25T00:01:05.367207+0000","last_active":"2024-04-25T00:30:15.801356+0000","last_peered":"2024-04-25T00:30:15.801356+0000","last_clean":"2024-04-25T00:30:15.801356+0000","last_became_active":"2024-04-25T00:01:05.366471+0000","last_became_peered":"2024-04-25T00:01:05.366471+0000","last_unstale":"2024-04-25T00:30:15.801356+0000","last_undegraded":"2024-04-25T00:30:15.801356+0000","last_fullsized":"2024-04-25T00:30:15.801356+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3760,"log_dups_size":0,"ondisk_log_size":3760,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:30:23.557381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21072,"num_write_kb":264901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"74'3","reported_seq":123,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.743279+0000","last_change":"2024-04-25T00:01:18.502364+0000","last_active":"2024-04-25T00:30:26.743279+0000","last_peered":"2024-04-25T00:30:26.743279+0000","last_clean":"2024-04-25T00:30:26.743279+0000","last_became_active":"2024-04-25T00:01:18.501633+0000","last_became_peered":"2024-04-25T00:01:18.501633+0000","last_unstale":"2024-04-25T00:30:26.743279+0000","last_undegraded":"2024-04-25T00:30:26.743279+0000","last_fullsized":"2024-04-25T00:30:26.743279+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:55.074018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"74'5031","reported_seq":5145,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.765069+0000","last_change":"2024-04-25T00:01:46.662019+0000","last_active":"2024-04-25T00:30:26.765069+0000","last_peered":"2024-04-25T00:30:26.765069+0000","last_clean":"2024-04-25T00:30:26.765069+0000","last_became_active":"2024-04-25T00:01:46.661167+0000","last_became_peered":"2024-04-25T00:01:46.661167+0000","last_unstale":"2024-04-25T00:30:26.765069+0000","last_undegraded":"2024-04-25T00:30:26.765069+0000","last_fullsized":"2024-04-25T00:30:26.765069+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":5031,"log_dups_size":0,"ondisk_log_size":5031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:13:34.153117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":22,"num_write":27250,"num_write_kb":338892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"73'28","reported_seq":158,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.802312+0000","last_change":"2024-04-25T00:01:04.361809+0000","last_active":"2024-04-25T00:30:15.802312+0000","last_peered":"2024-04-25T00:30:15.802312+0000","last_clean":"2024-04-25T00:30:15.802312+0000","last_became_active":"2024-04-25T00:01:04.361059+0000","last_became_peered":"2024-04-25T00:01:04.361059+0000","last_unstale":"2024-04-25T00:30:15.802312+0000","last_undegraded":"2024-04-25T00:30:15.802312+0000","last_fullsized":"2024-04-25T00:30:15.802312+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"70'5376","reported_seq":5504,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.799500+0000","last_change":"2024-04-25T00:01:05.370818+0000","last_active":"2024-04-25T00:30:15.799500+0000","last_peered":"2024-04-25T00:30:15.799500+0000","last_clean":"2024-04-25T00:30:15.799500+0000","last_became_active":"2024-04-25T00:01:05.369991+0000","last_became_peered":"2024-04-25T00:01:05.369991+0000","last_unstale":"2024-04-25T00:30:15.799500+0000","last_undegraded":"2024-04-25T00:30:15.799500+0000","last_fullsized":"2024-04-25T00:30:15.799500+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":5376,"log_dups_size":0,"ondisk_log_size":5376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:43.615895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29984,"num_write_kb":386541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"74'2","reported_seq":122,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:23.313841+0000","last_change":"2024-04-25T00:01:18.497682+0000","last_active":"2024-04-25T00:30:23.313841+0000","last_peered":"2024-04-25T00:30:23.313841+0000","last_clean":"2024-04-25T00:30:23.313841+0000","last_became_active":"2024-04-25T00:01:18.497397+0000","last_became_peered":"2024-04-25T00:01:18.497397+0000","last_unstale":"2024-04-25T00:30:23.313841+0000","last_undegraded":"2024-04-25T00:30:23.313841+0000","last_fullsized":"2024-04-25T00:30:23.313841+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:22.983972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"74'4080","reported_seq":4199,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.684096+0000","last_change":"2024-04-25T00:01:19.510591+0000","last_active":"2024-04-25T00:30:26.684096+0000","last_peered":"2024-04-25T00:30:26.684096+0000","last_clean":"2024-04-25T00:30:26.684096+0000","last_became_active":"2024-04-25T00:01:19.510252+0000","last_became_peered":"2024-04-25T00:01:19.510252+0000","last_unstale":"2024-04-25T00:30:26.684096+0000","last_undegraded":"2024-04-25T00:30:26.684096+0000","last_fullsized":"2024-04-25T00:30:26.684096+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4080,"log_dups_size":0,"ondisk_log_size":4080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:38.005233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":21789,"num_write_kb":296332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"74'53","reported_seq":203,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.793824+0000","last_change":"2024-04-25T00:01:04.362030+0000","last_active":"2024-04-25T00:30:15.793824+0000","last_peered":"2024-04-25T00:30:15.793824+0000","last_clean":"2024-04-25T00:30:15.793824+0000","last_became_active":"2024-04-25T00:01:04.361708+0000","last_became_peered":"2024-04-25T00:01:04.361708+0000","last_unstale":"2024-04-25T00:30:15.793824+0000","last_undegraded":"2024-04-25T00:30:15.793824+0000","last_fullsized":"2024-04-25T00:30:15.793824+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:38:33.392719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":69,"num_read_kb":69,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"69'3669","reported_seq":3794,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.942949+0000","last_change":"2024-04-25T00:01:05.387195+0000","last_active":"2024-04-25T00:30:15.942949+0000","last_peered":"2024-04-25T00:30:15.942949+0000","last_clean":"2024-04-25T00:30:15.942949+0000","last_became_active":"2024-04-25T00:01:05.386853+0000","last_became_peered":"2024-04-25T00:01:05.386853+0000","last_unstale":"2024-04-25T00:30:15.942949+0000","last_undegraded":"2024-04-25T00:30:15.942949+0000","last_fullsized":"2024-04-25T00:30:15.942949+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3669,"log_dups_size":0,"ondisk_log_size":3669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:06:19.307378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19904,"num_write_kb":264274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"74'723","reported_seq":869,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.740062+0000","last_change":"2024-04-25T00:01:18.487835+0000","last_active":"2024-04-25T00:30:26.740062+0000","last_peered":"2024-04-25T00:30:26.740062+0000","last_clean":"2024-04-25T00:30:26.740062+0000","last_became_active":"2024-04-25T00:01:18.487571+0000","last_became_peered":"2024-04-25T00:01:18.487571+0000","last_unstale":"2024-04-25T00:30:26.740062+0000","last_undegraded":"2024-04-25T00:30:26.740062+0000","last_fullsized":"2024-04-25T00:30:26.740062+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":723,"log_dups_size":0,"ondisk_log_size":723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:44.582148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":27,"num_read_kb":82727,"num_write":711,"num_write_kb":4310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"74'3586","reported_seq":3704,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.700653+0000","last_change":"2024-04-25T00:01:19.519834+0000","last_active":"2024-04-25T00:30:26.700653+0000","last_peered":"2024-04-25T00:30:26.700653+0000","last_clean":"2024-04-25T00:30:26.700653+0000","last_became_active":"2024-04-25T00:01:19.519470+0000","last_became_peered":"2024-04-25T00:01:19.519470+0000","last_unstale":"2024-04-25T00:30:26.700653+0000","last_undegraded":"2024-04-25T00:30:26.700653+0000","last_fullsized":"2024-04-25T00:30:26.700653+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3586,"log_dups_size":0,"ondisk_log_size":3586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:47.859626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":5,"num_write":18586,"num_write_kb":254663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"70'6","reported_seq":179,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.793266+0000","last_change":"2024-04-25T00:01:04.362832+0000","last_active":"2024-04-25T00:30:15.793266+0000","last_peered":"2024-04-25T00:30:15.793266+0000","last_clean":"2024-04-25T00:30:15.793266+0000","last_became_active":"2024-04-25T00:01:04.362059+0000","last_became_peered":"2024-04-25T00:01:04.362059+0000","last_unstale":"2024-04-25T00:30:15.793266+0000","last_undegraded":"2024-04-25T00:30:15.793266+0000","last_fullsized":"2024-04-25T00:30:15.793266+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:05.441048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"70'4973","reported_seq":5098,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.813402+0000","last_change":"2024-04-25T00:01:05.381120+0000","last_active":"2024-04-25T00:30:15.813402+0000","last_peered":"2024-04-25T00:30:15.813402+0000","last_clean":"2024-04-25T00:30:15.813402+0000","last_became_active":"2024-04-25T00:01:05.378320+0000","last_became_peered":"2024-04-25T00:01:05.378320+0000","last_unstale":"2024-04-25T00:30:15.813402+0000","last_undegraded":"2024-04-25T00:30:15.813402+0000","last_fullsized":"2024-04-25T00:30:15.813402+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4973,"log_dups_size":0,"ondisk_log_size":4973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:34:06.175918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27225,"num_write_kb":350946,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"74'355","reported_seq":476,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:31.515142+0000","last_change":"2024-04-25T00:01:18.490300+0000","last_active":"2024-04-25T00:30:31.515142+0000","last_peered":"2024-04-25T00:30:31.515142+0000","last_clean":"2024-04-25T00:30:31.515142+0000","last_became_active":"2024-04-25T00:01:18.490050+0000","last_became_peered":"2024-04-25T00:01:18.490050+0000","last_unstale":"2024-04-25T00:30:31.515142+0000","last_undegraded":"2024-04-25T00:30:31.515142+0000","last_fullsized":"2024-04-25T00:30:31.515142+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":355,"log_dups_size":0,"ondisk_log_size":355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:13:05.971606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2691528,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":907,"num_write":326,"num_write_kb":2825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"74'4320","reported_seq":4442,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.772385+0000","last_change":"2024-04-25T00:01:19.500026+0000","last_active":"2024-04-25T00:30:26.772385+0000","last_peered":"2024-04-25T00:30:26.772385+0000","last_clean":"2024-04-25T00:30:26.772385+0000","last_became_active":"2024-04-25T00:01:19.499244+0000","last_became_peered":"2024-04-25T00:01:19.499244+0000","last_unstale":"2024-04-25T00:30:26.772385+0000","last_undegraded":"2024-04-25T00:30:26.772385+0000","last_fullsized":"2024-04-25T00:30:26.772385+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4320,"log_dups_size":0,"ondisk_log_size":4320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:40:36.350832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":15,"num_write":23810,"num_write_kb":308534,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"70'4158","reported_seq":4285,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.809005+0000","last_change":"2024-04-25T00:01:05.376984+0000","last_active":"2024-04-25T00:30:15.809005+0000","last_peered":"2024-04-25T00:30:15.809005+0000","last_clean":"2024-04-25T00:30:15.809005+0000","last_became_active":"2024-04-25T00:01:05.376259+0000","last_became_peered":"2024-04-25T00:01:05.376259+0000","last_unstale":"2024-04-25T00:30:15.809005+0000","last_undegraded":"2024-04-25T00:30:15.809005+0000","last_fullsized":"2024-04-25T00:30:15.809005+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4158,"log_dups_size":0,"ondisk_log_size":4158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:13:28.543302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23006,"num_write_kb":301346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"74'1526","reported_seq":1710,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:23.032874+0000","last_change":"2024-04-25T00:01:04.359262+0000","last_active":"2024-04-25T00:30:23.032874+0000","last_peered":"2024-04-25T00:30:23.032874+0000","last_clean":"2024-04-25T00:30:23.032874+0000","last_became_active":"2024-04-25T00:01:04.357971+0000","last_became_peered":"2024-04-25T00:01:04.357971+0000","last_unstale":"2024-04-25T00:30:23.032874+0000","last_undegraded":"2024-04-25T00:30:23.032874+0000","last_fullsized":"2024-04-25T00:30:23.032874+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1526,"log_dups_size":0,"ondisk_log_size":1526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:05.441048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":78,"num_read_kb":52062,"num_write":1499,"num_write_kb":4525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"74'6","reported_seq":150,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.768405+0000","last_change":"2024-04-25T00:01:18.475624+0000","last_active":"2024-04-25T00:30:26.768405+0000","last_peered":"2024-04-25T00:30:26.768405+0000","last_clean":"2024-04-25T00:30:26.768405+0000","last_became_active":"2024-04-25T00:01:18.475149+0000","last_became_peered":"2024-04-25T00:01:18.475149+0000","last_unstale":"2024-04-25T00:30:26.768405+0000","last_undegraded":"2024-04-25T00:30:26.768405+0000","last_fullsized":"2024-04-25T00:30:26.768405+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:40:35.341075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"74'4556","reported_seq":4678,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.774727+0000","last_change":"2024-04-25T00:01:19.531113+0000","last_active":"2024-04-25T00:30:26.774727+0000","last_peered":"2024-04-25T00:30:26.774727+0000","last_clean":"2024-04-25T00:30:26.774727+0000","last_became_active":"2024-04-25T00:01:19.520150+0000","last_became_peered":"2024-04-25T00:01:19.520150+0000","last_unstale":"2024-04-25T00:30:26.774727+0000","last_undegraded":"2024-04-25T00:30:26.774727+0000","last_fullsized":"2024-04-25T00:30:26.774727+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4556,"log_dups_size":0,"ondisk_log_size":4556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:38.152376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":19,"num_write":25634,"num_write_kb":315935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"70'3060","reported_seq":3187,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.803143+0000","last_change":"2024-04-25T00:01:05.362611+0000","last_active":"2024-04-25T00:30:15.803143+0000","last_peered":"2024-04-25T00:30:15.803143+0000","last_clean":"2024-04-25T00:30:15.803143+0000","last_became_active":"2024-04-25T00:01:05.361817+0000","last_became_peered":"2024-04-25T00:01:05.361817+0000","last_unstale":"2024-04-25T00:30:15.803143+0000","last_undegraded":"2024-04-25T00:30:15.803143+0000","last_fullsized":"2024-04-25T00:30:15.803143+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3060,"log_dups_size":0,"ondisk_log_size":3060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:50.489125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16241,"num_write_kb":228970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"70'3","reported_seq":132,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.810550+0000","last_change":"2024-04-25T00:01:04.371935+0000","last_active":"2024-04-25T00:30:15.810550+0000","last_peered":"2024-04-25T00:30:15.810550+0000","last_clean":"2024-04-25T00:30:15.810550+0000","last_became_active":"2024-04-25T00:01:04.371420+0000","last_became_peered":"2024-04-25T00:01:04.371420+0000","last_unstale":"2024-04-25T00:30:15.810550+0000","last_undegraded":"2024-04-25T00:30:15.810550+0000","last_fullsized":"2024-04-25T00:30:15.810550+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"74'5","reported_seq":126,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.747597+0000","last_change":"2024-04-25T00:01:18.488623+0000","last_active":"2024-04-25T00:30:26.747597+0000","last_peered":"2024-04-25T00:30:26.747597+0000","last_clean":"2024-04-25T00:30:26.747597+0000","last_became_active":"2024-04-25T00:01:18.488291+0000","last_became_peered":"2024-04-25T00:01:18.488291+0000","last_unstale":"2024-04-25T00:30:26.747597+0000","last_undegraded":"2024-04-25T00:30:26.747597+0000","last_fullsized":"2024-04-25T00:30:26.747597+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:30:21.436681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"74'3591","reported_seq":3711,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.734674+0000","last_change":"2024-04-25T00:01:19.516731+0000","last_active":"2024-04-25T00:30:26.734674+0000","last_peered":"2024-04-25T00:30:26.734674+0000","last_clean":"2024-04-25T00:30:26.734674+0000","last_became_active":"2024-04-25T00:01:19.516405+0000","last_became_peered":"2024-04-25T00:01:19.516405+0000","last_unstale":"2024-04-25T00:30:26.734674+0000","last_undegraded":"2024-04-25T00:30:26.734674+0000","last_fullsized":"2024-04-25T00:30:26.734674+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3591,"log_dups_size":0,"ondisk_log_size":3591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:44:37.936092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":19584,"num_write_kb":250628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"69'3777","reported_seq":3904,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.790888+0000","last_change":"2024-04-25T00:01:05.355455+0000","last_active":"2024-04-25T00:30:15.790888+0000","last_peered":"2024-04-25T00:30:15.790888+0000","last_clean":"2024-04-25T00:30:15.790888+0000","last_became_active":"2024-04-25T00:01:05.354699+0000","last_became_peered":"2024-04-25T00:01:05.354699+0000","last_unstale":"2024-04-25T00:30:15.790888+0000","last_undegraded":"2024-04-25T00:30:15.790888+0000","last_fullsized":"2024-04-25T00:30:15.790888+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3777,"log_dups_size":0,"ondisk_log_size":3777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:43:58.069271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20993,"num_write_kb":256390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"69'2","reported_seq":131,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.791647+0000","last_change":"2024-04-25T00:01:04.362268+0000","last_active":"2024-04-25T00:30:15.791647+0000","last_peered":"2024-04-25T00:30:15.791647+0000","last_clean":"2024-04-25T00:30:15.791647+0000","last_became_active":"2024-04-25T00:01:04.361622+0000","last_became_peered":"2024-04-25T00:01:04.361622+0000","last_unstale":"2024-04-25T00:30:15.791647+0000","last_undegraded":"2024-04-25T00:30:15.791647+0000","last_fullsized":"2024-04-25T00:30:15.791647+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:58:46.021343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"74'4","reported_seq":147,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.744595+0000","last_change":"2024-04-25T00:01:18.494047+0000","last_active":"2024-04-25T00:30:26.744595+0000","last_peered":"2024-04-25T00:30:26.744595+0000","last_clean":"2024-04-25T00:30:26.744595+0000","last_became_active":"2024-04-25T00:01:18.493614+0000","last_became_peered":"2024-04-25T00:01:18.493614+0000","last_unstale":"2024-04-25T00:30:26.744595+0000","last_undegraded":"2024-04-25T00:30:26.744595+0000","last_fullsized":"2024-04-25T00:30:26.744595+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:37:08.562016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"74'4895","reported_seq":5013,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.674895+0000","last_change":"2024-04-25T00:01:19.505397+0000","last_active":"2024-04-25T00:30:26.674895+0000","last_peered":"2024-04-25T00:30:26.674895+0000","last_clean":"2024-04-25T00:30:26.674895+0000","last_became_active":"2024-04-25T00:01:19.505126+0000","last_became_peered":"2024-04-25T00:01:19.505126+0000","last_unstale":"2024-04-25T00:30:26.674895+0000","last_undegraded":"2024-04-25T00:30:26.674895+0000","last_fullsized":"2024-04-25T00:30:26.674895+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4895,"log_dups_size":0,"ondisk_log_size":4895,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:08:49.640267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26799,"num_write_kb":349350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"69'3934","reported_seq":4061,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.805653+0000","last_change":"2024-04-25T00:01:05.354953+0000","last_active":"2024-04-25T00:30:15.805653+0000","last_peered":"2024-04-25T00:30:15.805653+0000","last_clean":"2024-04-25T00:30:15.805653+0000","last_became_active":"2024-04-25T00:01:05.354118+0000","last_became_peered":"2024-04-25T00:01:05.354118+0000","last_unstale":"2024-04-25T00:30:15.805653+0000","last_undegraded":"2024-04-25T00:30:15.805653+0000","last_fullsized":"2024-04-25T00:30:15.805653+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3934,"log_dups_size":0,"ondisk_log_size":3934,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:09.858617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21059,"num_write_kb":273031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"70'6","reported_seq":135,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.789548+0000","last_change":"2024-04-25T00:01:04.365515+0000","last_active":"2024-04-25T00:30:15.789548+0000","last_peered":"2024-04-25T00:30:15.789548+0000","last_clean":"2024-04-25T00:30:15.789548+0000","last_became_active":"2024-04-25T00:01:04.364942+0000","last_became_peered":"2024-04-25T00:01:04.364942+0000","last_unstale":"2024-04-25T00:30:15.789548+0000","last_undegraded":"2024-04-25T00:30:15.789548+0000","last_fullsized":"2024-04-25T00:30:15.789548+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"74'7","reported_seq":174,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.741808+0000","last_change":"2024-04-25T00:01:18.476207+0000","last_active":"2024-04-25T00:30:26.741808+0000","last_peered":"2024-04-25T00:30:26.741808+0000","last_clean":"2024-04-25T00:30:26.741808+0000","last_became_active":"2024-04-25T00:01:18.475862+0000","last_became_peered":"2024-04-25T00:01:18.475862+0000","last_unstale":"2024-04-25T00:30:26.741808+0000","last_undegraded":"2024-04-25T00:30:26.741808+0000","last_fullsized":"2024-04-25T00:30:26.741808+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:11:40.170249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"74'3090","reported_seq":3210,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.776689+0000","last_change":"2024-04-25T00:01:19.529607+0000","last_active":"2024-04-25T00:30:26.776689+0000","last_peered":"2024-04-25T00:30:26.776689+0000","last_clean":"2024-04-25T00:30:26.776689+0000","last_became_active":"2024-04-25T00:01:19.528509+0000","last_became_peered":"2024-04-25T00:01:19.528509+0000","last_unstale":"2024-04-25T00:30:26.776689+0000","last_undegraded":"2024-04-25T00:30:26.776689+0000","last_fullsized":"2024-04-25T00:30:26.776689+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3090,"log_dups_size":0,"ondisk_log_size":3090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:45:01.068091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17043,"num_write_kb":228060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"70'4336","reported_seq":4463,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.793683+0000","last_change":"2024-04-25T00:01:05.358106+0000","last_active":"2024-04-25T00:30:15.793683+0000","last_peered":"2024-04-25T00:30:15.793683+0000","last_clean":"2024-04-25T00:30:15.793683+0000","last_became_active":"2024-04-25T00:01:05.357135+0000","last_became_peered":"2024-04-25T00:01:05.357135+0000","last_unstale":"2024-04-25T00:30:15.793683+0000","last_undegraded":"2024-04-25T00:30:15.793683+0000","last_fullsized":"2024-04-25T00:30:15.793683+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4336,"log_dups_size":0,"ondisk_log_size":4336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:41:42.228234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23030,"num_write_kb":325887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"71'9","reported_seq":184,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.814467+0000","last_change":"2024-04-25T00:01:04.369917+0000","last_active":"2024-04-25T00:30:15.814467+0000","last_peered":"2024-04-25T00:30:15.814467+0000","last_clean":"2024-04-25T00:30:15.814467+0000","last_became_active":"2024-04-25T00:01:04.369309+0000","last_became_peered":"2024-04-25T00:01:04.369309+0000","last_unstale":"2024-04-25T00:30:15.814467+0000","last_undegraded":"2024-04-25T00:30:15.814467+0000","last_fullsized":"2024-04-25T00:30:15.814467+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:55:08.063965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":8,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"74'51","reported_seq":241,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:28.848602+0000","last_change":"2024-04-25T00:01:18.477189+0000","last_active":"2024-04-25T00:30:28.848602+0000","last_peered":"2024-04-25T00:30:28.848602+0000","last_clean":"2024-04-25T00:30:28.848602+0000","last_became_active":"2024-04-25T00:01:18.476909+0000","last_became_peered":"2024-04-25T00:01:18.476909+0000","last_unstale":"2024-04-25T00:30:28.848602+0000","last_undegraded":"2024-04-25T00:30:28.848602+0000","last_fullsized":"2024-04-25T00:30:28.848602+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:36.995477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84646,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":68,"num_read_kb":112,"num_write":46,"num_write_kb":112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"74'3821","reported_seq":3942,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.704068+0000","last_change":"2024-04-25T00:01:19.517500+0000","last_active":"2024-04-25T00:30:26.704068+0000","last_peered":"2024-04-25T00:30:26.704068+0000","last_clean":"2024-04-25T00:30:26.704068+0000","last_became_active":"2024-04-25T00:01:19.517182+0000","last_became_peered":"2024-04-25T00:01:19.517182+0000","last_unstale":"2024-04-25T00:30:26.704068+0000","last_undegraded":"2024-04-25T00:30:26.704068+0000","last_fullsized":"2024-04-25T00:30:26.704068+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3821,"log_dups_size":0,"ondisk_log_size":3821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:58:49.738233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20929,"num_write_kb":256020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"70'3227","reported_seq":3354,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.793834+0000","last_change":"2024-04-25T00:01:05.364829+0000","last_active":"2024-04-25T00:30:15.793834+0000","last_peered":"2024-04-25T00:30:15.793834+0000","last_clean":"2024-04-25T00:30:15.793834+0000","last_became_active":"2024-04-25T00:01:05.364125+0000","last_became_peered":"2024-04-25T00:01:05.364125+0000","last_unstale":"2024-04-25T00:30:15.793834+0000","last_undegraded":"2024-04-25T00:30:15.793834+0000","last_fullsized":"2024-04-25T00:30:15.793834+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3227,"log_dups_size":0,"ondisk_log_size":3227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:15:45.531760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17808,"num_write_kb":231231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"74'458","reported_seq":610,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:25.796746+0000","last_change":"2024-04-25T00:01:04.361603+0000","last_active":"2024-04-25T00:30:25.796746+0000","last_peered":"2024-04-25T00:30:25.796746+0000","last_clean":"2024-04-25T00:30:25.796746+0000","last_became_active":"2024-04-25T00:01:04.361046+0000","last_became_peered":"2024-04-25T00:01:04.361046+0000","last_unstale":"2024-04-25T00:30:25.796746+0000","last_undegraded":"2024-04-25T00:30:25.796746+0000","last_fullsized":"2024-04-25T00:30:25.796746+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":458,"log_dups_size":0,"ondisk_log_size":458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:35:18.036843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":46,"num_read_kb":63,"num_write":1069,"num_write_kb":360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"74'12","reported_seq":134,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.743234+0000","last_change":"2024-04-25T00:01:18.488352+0000","last_active":"2024-04-25T00:30:26.743234+0000","last_peered":"2024-04-25T00:30:26.743234+0000","last_clean":"2024-04-25T00:30:26.743234+0000","last_became_active":"2024-04-25T00:01:18.487321+0000","last_became_peered":"2024-04-25T00:01:18.487321+0000","last_unstale":"2024-04-25T00:30:26.743234+0000","last_undegraded":"2024-04-25T00:30:26.743234+0000","last_fullsized":"2024-04-25T00:30:26.743234+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:37:22.329655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"74'4221","reported_seq":4341,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.775265+0000","last_change":"2024-04-25T00:01:19.519653+0000","last_active":"2024-04-25T00:30:26.775265+0000","last_peered":"2024-04-25T00:30:26.775265+0000","last_clean":"2024-04-25T00:30:26.775265+0000","last_became_active":"2024-04-25T00:01:19.519052+0000","last_became_peered":"2024-04-25T00:01:19.519052+0000","last_unstale":"2024-04-25T00:30:26.775265+0000","last_undegraded":"2024-04-25T00:30:26.775265+0000","last_fullsized":"2024-04-25T00:30:26.775265+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4221,"log_dups_size":0,"ondisk_log_size":4221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:40:36.350832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22937,"num_write_kb":280510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"70'5174","reported_seq":5302,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.813035+0000","last_change":"2024-04-25T00:01:05.374011+0000","last_active":"2024-04-25T00:30:15.813035+0000","last_peered":"2024-04-25T00:30:15.813035+0000","last_clean":"2024-04-25T00:30:15.813035+0000","last_became_active":"2024-04-25T00:01:05.373276+0000","last_became_peered":"2024-04-25T00:01:05.373276+0000","last_unstale":"2024-04-25T00:30:15.813035+0000","last_undegraded":"2024-04-25T00:30:15.813035+0000","last_fullsized":"2024-04-25T00:30:15.813035+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":5174,"log_dups_size":0,"ondisk_log_size":5174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:40:53.349428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28022,"num_write_kb":362384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"69'2","reported_seq":132,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.814680+0000","last_change":"2024-04-25T00:01:04.365391+0000","last_active":"2024-04-25T00:30:15.814680+0000","last_peered":"2024-04-25T00:30:15.814680+0000","last_clean":"2024-04-25T00:30:15.814680+0000","last_became_active":"2024-04-25T00:01:04.364989+0000","last_became_peered":"2024-04-25T00:01:04.364989+0000","last_unstale":"2024-04-25T00:30:15.814680+0000","last_undegraded":"2024-04-25T00:30:15.814680+0000","last_fullsized":"2024-04-25T00:30:15.814680+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:06:18.298057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"74'3","reported_seq":125,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.737749+0000","last_change":"2024-04-25T00:01:18.491597+0000","last_active":"2024-04-25T00:30:26.737749+0000","last_peered":"2024-04-25T00:30:26.737749+0000","last_clean":"2024-04-25T00:30:26.737749+0000","last_became_active":"2024-04-25T00:01:18.490859+0000","last_became_peered":"2024-04-25T00:01:18.490859+0000","last_unstale":"2024-04-25T00:30:26.737749+0000","last_undegraded":"2024-04-25T00:30:26.737749+0000","last_fullsized":"2024-04-25T00:30:26.737749+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:36.995477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"74'5316","reported_seq":5429,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.548329+0000","last_change":"2024-04-25T00:01:46.667934+0000","last_active":"2024-04-25T00:30:26.548329+0000","last_peered":"2024-04-25T00:30:26.548329+0000","last_clean":"2024-04-25T00:30:26.548329+0000","last_became_active":"2024-04-25T00:01:46.667075+0000","last_became_peered":"2024-04-25T00:01:46.667075+0000","last_unstale":"2024-04-25T00:30:26.548329+0000","last_undegraded":"2024-04-25T00:30:26.548329+0000","last_fullsized":"2024-04-25T00:30:26.548329+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":5316,"log_dups_size":0,"ondisk_log_size":5316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:28:18.234546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29390,"num_write_kb":363802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"70'3961","reported_seq":4089,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.804843+0000","last_change":"2024-04-25T00:01:05.378340+0000","last_active":"2024-04-25T00:30:15.804843+0000","last_peered":"2024-04-25T00:30:15.804843+0000","last_clean":"2024-04-25T00:30:15.804843+0000","last_became_active":"2024-04-25T00:01:05.377593+0000","last_became_peered":"2024-04-25T00:01:05.377593+0000","last_unstale":"2024-04-25T00:30:15.804843+0000","last_undegraded":"2024-04-25T00:30:15.804843+0000","last_fullsized":"2024-04-25T00:30:15.804843+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3961,"log_dups_size":0,"ondisk_log_size":3961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:48.202251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21237,"num_write_kb":269574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"70'2","reported_seq":132,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.799354+0000","last_change":"2024-04-25T00:01:04.366577+0000","last_active":"2024-04-25T00:30:15.799354+0000","last_peered":"2024-04-25T00:30:15.799354+0000","last_clean":"2024-04-25T00:30:15.799354+0000","last_became_active":"2024-04-25T00:01:04.365881+0000","last_became_peered":"2024-04-25T00:01:04.365881+0000","last_unstale":"2024-04-25T00:30:15.799354+0000","last_undegraded":"2024-04-25T00:30:15.799354+0000","last_fullsized":"2024-04-25T00:30:15.799354+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:49:22.673421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'4","reported_seq":126,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.132180+0000","last_change":"2024-04-25T00:01:18.504341+0000","last_active":"2024-04-25T00:30:26.132180+0000","last_peered":"2024-04-25T00:30:26.132180+0000","last_clean":"2024-04-25T00:30:26.132180+0000","last_became_active":"2024-04-25T00:01:18.491498+0000","last_became_peered":"2024-04-25T00:01:18.491498+0000","last_unstale":"2024-04-25T00:30:26.132180+0000","last_undegraded":"2024-04-25T00:30:26.132180+0000","last_fullsized":"2024-04-25T00:30:26.132180+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:56.741250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"74'4156","reported_seq":4283,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.774804+0000","last_change":"2024-04-25T00:01:46.667022+0000","last_active":"2024-04-25T00:30:26.774804+0000","last_peered":"2024-04-25T00:30:26.774804+0000","last_clean":"2024-04-25T00:30:26.774804+0000","last_became_active":"2024-04-25T00:01:46.666224+0000","last_became_peered":"2024-04-25T00:01:46.666224+0000","last_unstale":"2024-04-25T00:30:26.774804+0000","last_undegraded":"2024-04-25T00:30:26.774804+0000","last_fullsized":"2024-04-25T00:30:26.774804+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4156,"log_dups_size":0,"ondisk_log_size":4156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:25:20.185355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":7,"num_write":23008,"num_write_kb":280758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"69'4491","reported_seq":4619,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.793150+0000","last_change":"2024-04-25T00:01:05.368769+0000","last_active":"2024-04-25T00:30:15.793150+0000","last_peered":"2024-04-25T00:30:15.793150+0000","last_clean":"2024-04-25T00:30:15.793150+0000","last_became_active":"2024-04-25T00:01:05.366817+0000","last_became_peered":"2024-04-25T00:01:05.366817+0000","last_unstale":"2024-04-25T00:30:15.793150+0000","last_undegraded":"2024-04-25T00:30:15.793150+0000","last_fullsized":"2024-04-25T00:30:15.793150+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4491,"log_dups_size":0,"ondisk_log_size":4491,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:11:27.044893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24298,"num_write_kb":330898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"69'5","reported_seq":135,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.810345+0000","last_change":"2024-04-25T00:01:04.361900+0000","last_active":"2024-04-25T00:30:15.810345+0000","last_peered":"2024-04-25T00:30:15.810345+0000","last_clean":"2024-04-25T00:30:15.810345+0000","last_became_active":"2024-04-25T00:01:04.361572+0000","last_became_peered":"2024-04-25T00:01:04.361572+0000","last_unstale":"2024-04-25T00:30:15.810345+0000","last_undegraded":"2024-04-25T00:30:15.810345+0000","last_fullsized":"2024-04-25T00:30:15.810345+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:29.218739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"74'2","reported_seq":124,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.744419+0000","last_change":"2024-04-25T00:01:18.504139+0000","last_active":"2024-04-25T00:30:26.744419+0000","last_peered":"2024-04-25T00:30:26.744419+0000","last_clean":"2024-04-25T00:30:26.744419+0000","last_became_active":"2024-04-25T00:01:18.503379+0000","last_became_peered":"2024-04-25T00:01:18.503379+0000","last_unstale":"2024-04-25T00:30:26.744419+0000","last_undegraded":"2024-04-25T00:30:26.744419+0000","last_fullsized":"2024-04-25T00:30:26.744419+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:02:56.363258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"74'5402","reported_seq":5525,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.707463+0000","last_change":"2024-04-25T00:01:19.531756+0000","last_active":"2024-04-25T00:30:26.707463+0000","last_peered":"2024-04-25T00:30:26.707463+0000","last_clean":"2024-04-25T00:30:26.707463+0000","last_became_active":"2024-04-25T00:01:19.531145+0000","last_became_peered":"2024-04-25T00:01:19.531145+0000","last_unstale":"2024-04-25T00:30:26.707463+0000","last_undegraded":"2024-04-25T00:30:26.707463+0000","last_fullsized":"2024-04-25T00:30:26.707463+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":5402,"log_dups_size":0,"ondisk_log_size":5402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:13:42.678415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":13,"num_write":29532,"num_write_kb":356063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"70'3335","reported_seq":3463,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.794572+0000","last_change":"2024-04-25T00:01:05.354557+0000","last_active":"2024-04-25T00:30:15.794572+0000","last_peered":"2024-04-25T00:30:15.794572+0000","last_clean":"2024-04-25T00:30:15.794572+0000","last_became_active":"2024-04-25T00:01:05.353556+0000","last_became_peered":"2024-04-25T00:01:05.353556+0000","last_unstale":"2024-04-25T00:30:15.794572+0000","last_undegraded":"2024-04-25T00:30:15.794572+0000","last_fullsized":"2024-04-25T00:30:15.794572+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3335,"log_dups_size":0,"ondisk_log_size":3335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:24.964055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18377,"num_write_kb":241696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"74'53","reported_seq":252,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:25.820987+0000","last_change":"2024-04-25T00:01:04.372916+0000","last_active":"2024-04-25T00:30:25.820987+0000","last_peered":"2024-04-25T00:30:25.820987+0000","last_clean":"2024-04-25T00:30:25.820987+0000","last_became_active":"2024-04-25T00:01:04.372185+0000","last_became_peered":"2024-04-25T00:01:04.372185+0000","last_unstale":"2024-04-25T00:30:25.820987+0000","last_undegraded":"2024-04-25T00:30:25.820987+0000","last_fullsized":"2024-04-25T00:30:25.820987+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88686,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":69,"num_read_kb":117,"num_write":46,"num_write_kb":117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"74'9","reported_seq":176,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.750624+0000","last_change":"2024-04-25T00:01:18.504614+0000","last_active":"2024-04-25T00:30:26.750624+0000","last_peered":"2024-04-25T00:30:26.750624+0000","last_clean":"2024-04-25T00:30:26.750624+0000","last_became_active":"2024-04-25T00:01:18.501448+0000","last_became_peered":"2024-04-25T00:01:18.501448+0000","last_unstale":"2024-04-25T00:30:26.750624+0000","last_undegraded":"2024-04-25T00:30:26.750624+0000","last_fullsized":"2024-04-25T00:30:26.750624+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:22.983972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":8,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"74'3337","reported_seq":3456,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.763497+0000","last_change":"2024-04-25T00:01:19.517510+0000","last_active":"2024-04-25T00:30:26.763497+0000","last_peered":"2024-04-25T00:30:26.763497+0000","last_clean":"2024-04-25T00:30:26.763497+0000","last_became_active":"2024-04-25T00:01:19.517243+0000","last_became_peered":"2024-04-25T00:01:19.517243+0000","last_unstale":"2024-04-25T00:30:26.763497+0000","last_undegraded":"2024-04-25T00:30:26.763497+0000","last_fullsized":"2024-04-25T00:30:26.763497+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3337,"log_dups_size":0,"ondisk_log_size":3337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:42:38.149502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18533,"num_write_kb":242287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"69'3723","reported_seq":3852,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.809022+0000","last_change":"2024-04-25T00:01:05.370968+0000","last_active":"2024-04-25T00:30:15.809022+0000","last_peered":"2024-04-25T00:30:15.809022+0000","last_clean":"2024-04-25T00:30:15.809022+0000","last_became_active":"2024-04-25T00:01:05.369987+0000","last_became_peered":"2024-04-25T00:01:05.369987+0000","last_unstale":"2024-04-25T00:30:15.809022+0000","last_undegraded":"2024-04-25T00:30:15.809022+0000","last_fullsized":"2024-04-25T00:30:15.809022+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3723,"log_dups_size":0,"ondisk_log_size":3723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20208,"num_write_kb":269376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"73'13","reported_seq":143,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.804977+0000","last_change":"2024-04-25T00:01:04.370226+0000","last_active":"2024-04-25T00:30:15.804977+0000","last_peered":"2024-04-25T00:30:15.804977+0000","last_clean":"2024-04-25T00:30:15.804977+0000","last_became_active":"2024-04-25T00:01:04.369904+0000","last_became_peered":"2024-04-25T00:01:04.369904+0000","last_unstale":"2024-04-25T00:30:15.804977+0000","last_undegraded":"2024-04-25T00:30:15.804977+0000","last_fullsized":"2024-04-25T00:30:15.804977+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:47:25.595275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"74'487","reported_seq":631,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.794202+0000","last_change":"2024-04-25T00:01:18.497156+0000","last_active":"2024-04-25T00:30:26.794202+0000","last_peered":"2024-04-25T00:30:26.794202+0000","last_clean":"2024-04-25T00:30:26.794202+0000","last_became_active":"2024-04-25T00:01:18.495573+0000","last_became_peered":"2024-04-25T00:01:18.495573+0000","last_unstale":"2024-04-25T00:30:26.794202+0000","last_undegraded":"2024-04-25T00:30:26.794202+0000","last_fullsized":"2024-04-25T00:30:26.794202+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":487,"log_dups_size":0,"ondisk_log_size":487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:29.119299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":46,"num_read_kb":69,"num_write":1147,"num_write_kb":382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"74'5079","reported_seq":5200,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.738941+0000","last_change":"2024-04-25T00:01:19.519411+0000","last_active":"2024-04-25T00:30:26.738941+0000","last_peered":"2024-04-25T00:30:26.738941+0000","last_clean":"2024-04-25T00:30:26.738941+0000","last_became_active":"2024-04-25T00:01:19.519020+0000","last_became_peered":"2024-04-25T00:01:19.519020+0000","last_unstale":"2024-04-25T00:30:26.738941+0000","last_undegraded":"2024-04-25T00:30:26.738941+0000","last_fullsized":"2024-04-25T00:30:26.738941+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":5079,"log_dups_size":0,"ondisk_log_size":5079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:50.351599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":14,"num_write":27774,"num_write_kb":353080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"70'3676","reported_seq":3804,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.793841+0000","last_change":"2024-04-25T00:01:05.364939+0000","last_active":"2024-04-25T00:30:15.793841+0000","last_peered":"2024-04-25T00:30:15.793841+0000","last_clean":"2024-04-25T00:30:15.793841+0000","last_became_active":"2024-04-25T00:01:05.364272+0000","last_became_peered":"2024-04-25T00:01:05.364272+0000","last_unstale":"2024-04-25T00:30:15.793841+0000","last_undegraded":"2024-04-25T00:30:15.793841+0000","last_fullsized":"2024-04-25T00:30:15.793841+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3676,"log_dups_size":0,"ondisk_log_size":3676,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:24:59.679908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19925,"num_write_kb":256146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"69'3","reported_seq":133,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.809642+0000","last_change":"2024-04-25T00:01:04.357028+0000","last_active":"2024-04-25T00:30:15.809642+0000","last_peered":"2024-04-25T00:30:15.809642+0000","last_clean":"2024-04-25T00:30:15.809642+0000","last_became_active":"2024-04-25T00:01:04.356644+0000","last_became_peered":"2024-04-25T00:01:04.356644+0000","last_unstale":"2024-04-25T00:30:15.809642+0000","last_undegraded":"2024-04-25T00:30:15.809642+0000","last_fullsized":"2024-04-25T00:30:15.809642+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:21.729893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":286,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.942363+0000","last_change":"2024-04-25T00:00:55.084492+0000","last_active":"2024-04-25T00:30:15.942363+0000","last_peered":"2024-04-25T00:30:15.942363+0000","last_clean":"2024-04-25T00:30:15.942363+0000","last_became_active":"2024-04-25T00:00:55.083493+0000","last_became_peered":"2024-04-25T00:00:55.083493+0000","last_unstale":"2024-04-25T00:30:15.942363+0000","last_undegraded":"2024-04-25T00:30:15.942363+0000","last_fullsized":"2024-04-25T00:30:15.942363+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:00:54.060578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:00:54.060578+0000","last_clean_scrub_stamp":"2024-04-25T00:00:54.060578+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:14:58.472477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"74'2","reported_seq":124,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.137996+0000","last_change":"2024-04-25T00:01:18.476298+0000","last_active":"2024-04-25T00:30:26.137996+0000","last_peered":"2024-04-25T00:30:26.137996+0000","last_clean":"2024-04-25T00:30:26.137996+0000","last_became_active":"2024-04-25T00:01:18.475898+0000","last_became_peered":"2024-04-25T00:01:18.475898+0000","last_unstale":"2024-04-25T00:30:26.137996+0000","last_undegraded":"2024-04-25T00:30:26.137996+0000","last_fullsized":"2024-04-25T00:30:26.137996+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:49.341842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"74'3757","reported_seq":3877,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.774597+0000","last_change":"2024-04-25T00:01:19.518102+0000","last_active":"2024-04-25T00:30:26.774597+0000","last_peered":"2024-04-25T00:30:26.774597+0000","last_clean":"2024-04-25T00:30:26.774597+0000","last_became_active":"2024-04-25T00:01:19.517376+0000","last_became_peered":"2024-04-25T00:01:19.517376+0000","last_unstale":"2024-04-25T00:30:26.774597+0000","last_undegraded":"2024-04-25T00:30:26.774597+0000","last_fullsized":"2024-04-25T00:30:26.774597+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3757,"log_dups_size":0,"ondisk_log_size":3757,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:55:46.853565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":2,"num_write":20920,"num_write_kb":243770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"70'4367","reported_seq":4495,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.813789+0000","last_change":"2024-04-25T00:01:05.376609+0000","last_active":"2024-04-25T00:30:15.813789+0000","last_peered":"2024-04-25T00:30:15.813789+0000","last_clean":"2024-04-25T00:30:15.813789+0000","last_became_active":"2024-04-25T00:01:05.375881+0000","last_became_peered":"2024-04-25T00:01:05.375881+0000","last_unstale":"2024-04-25T00:30:15.813789+0000","last_undegraded":"2024-04-25T00:30:15.813789+0000","last_fullsized":"2024-04-25T00:30:15.813789+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4367,"log_dups_size":0,"ondisk_log_size":4367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:46:05.592233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":2,"num_write":23695,"num_write_kb":308143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"70'6","reported_seq":135,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.813175+0000","last_change":"2024-04-25T00:01:04.368954+0000","last_active":"2024-04-25T00:30:15.813175+0000","last_peered":"2024-04-25T00:30:15.813175+0000","last_clean":"2024-04-25T00:30:15.813175+0000","last_became_active":"2024-04-25T00:01:04.367903+0000","last_became_peered":"2024-04-25T00:01:04.367903+0000","last_unstale":"2024-04-25T00:30:15.813175+0000","last_undegraded":"2024-04-25T00:30:15.813175+0000","last_fullsized":"2024-04-25T00:30:15.813175+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"74'3","reported_seq":124,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.737742+0000","last_change":"2024-04-25T00:01:18.476525+0000","last_active":"2024-04-25T00:30:26.737742+0000","last_peered":"2024-04-25T00:30:26.737742+0000","last_clean":"2024-04-25T00:30:26.737742+0000","last_became_active":"2024-04-25T00:01:18.476247+0000","last_became_peered":"2024-04-25T00:01:18.476247+0000","last_unstale":"2024-04-25T00:30:26.737742+0000","last_undegraded":"2024-04-25T00:30:26.737742+0000","last_fullsized":"2024-04-25T00:30:26.737742+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:30:36.682736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"74'4936","reported_seq":5056,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.737754+0000","last_change":"2024-04-25T00:01:19.514615+0000","last_active":"2024-04-25T00:30:26.737754+0000","last_peered":"2024-04-25T00:30:26.737754+0000","last_clean":"2024-04-25T00:30:26.737754+0000","last_became_active":"2024-04-25T00:01:19.513640+0000","last_became_peered":"2024-04-25T00:01:19.513640+0000","last_unstale":"2024-04-25T00:30:26.737754+0000","last_undegraded":"2024-04-25T00:30:26.737754+0000","last_fullsized":"2024-04-25T00:30:26.737754+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4936,"log_dups_size":0,"ondisk_log_size":4936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:08.620602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":27403,"num_write_kb":306834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"70'4466","reported_seq":4593,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.793110+0000","last_change":"2024-04-25T00:01:05.385670+0000","last_active":"2024-04-25T00:30:15.793110+0000","last_peered":"2024-04-25T00:30:15.793110+0000","last_clean":"2024-04-25T00:30:15.793110+0000","last_became_active":"2024-04-25T00:01:05.384822+0000","last_became_peered":"2024-04-25T00:01:05.384822+0000","last_unstale":"2024-04-25T00:30:15.793110+0000","last_undegraded":"2024-04-25T00:30:15.793110+0000","last_fullsized":"2024-04-25T00:30:15.793110+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4466,"log_dups_size":0,"ondisk_log_size":4466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:24.017263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23831,"num_write_kb":316890,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"70'6","reported_seq":158,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.798998+0000","last_change":"2024-04-25T00:01:04.356627+0000","last_active":"2024-04-25T00:30:15.798998+0000","last_peered":"2024-04-25T00:30:15.798998+0000","last_clean":"2024-04-25T00:30:15.798998+0000","last_became_active":"2024-04-25T00:01:04.354419+0000","last_became_peered":"2024-04-25T00:01:04.354419+0000","last_unstale":"2024-04-25T00:30:15.798998+0000","last_undegraded":"2024-04-25T00:30:15.798998+0000","last_fullsized":"2024-04-25T00:30:15.798998+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:26.243635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'1453","reported_seq":1626,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.754238+0000","last_change":"2024-04-25T00:01:18.490511+0000","last_active":"2024-04-25T00:30:26.754238+0000","last_peered":"2024-04-25T00:30:26.754238+0000","last_clean":"2024-04-25T00:30:26.754238+0000","last_became_active":"2024-04-25T00:01:18.489629+0000","last_became_peered":"2024-04-25T00:01:18.489629+0000","last_unstale":"2024-04-25T00:30:26.754238+0000","last_undegraded":"2024-04-25T00:30:26.754238+0000","last_fullsized":"2024-04-25T00:30:26.754238+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":1453,"log_dups_size":0,"ondisk_log_size":1453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:07.610846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":75,"num_read_kb":41971,"num_write":1427,"num_write_kb":4503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'3796","reported_seq":3915,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.602587+0000","last_change":"2024-04-25T00:01:19.508931+0000","last_active":"2024-04-25T00:30:26.602587+0000","last_peered":"2024-04-25T00:30:26.602587+0000","last_clean":"2024-04-25T00:30:26.602587+0000","last_became_active":"2024-04-25T00:01:19.507867+0000","last_became_peered":"2024-04-25T00:01:19.507867+0000","last_unstale":"2024-04-25T00:30:26.602587+0000","last_undegraded":"2024-04-25T00:30:26.602587+0000","last_fullsized":"2024-04-25T00:30:26.602587+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3796,"log_dups_size":0,"ondisk_log_size":3796,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:38.152376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20867,"num_write_kb":276131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"70'5324","reported_seq":5452,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.792960+0000","last_change":"2024-04-25T00:01:05.378840+0000","last_active":"2024-04-25T00:30:15.792960+0000","last_peered":"2024-04-25T00:30:15.792960+0000","last_clean":"2024-04-25T00:30:15.792960+0000","last_became_active":"2024-04-25T00:01:05.378112+0000","last_became_peered":"2024-04-25T00:01:05.378112+0000","last_unstale":"2024-04-25T00:30:15.792960+0000","last_undegraded":"2024-04-25T00:30:15.792960+0000","last_fullsized":"2024-04-25T00:30:15.792960+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":5324,"log_dups_size":0,"ondisk_log_size":5324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:43.615895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":28190,"num_write_kb":387378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"69'7","reported_seq":182,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.795644+0000","last_change":"2024-04-25T00:01:04.370794+0000","last_active":"2024-04-25T00:30:15.795644+0000","last_peered":"2024-04-25T00:30:15.795644+0000","last_clean":"2024-04-25T00:30:15.795644+0000","last_became_active":"2024-04-25T00:01:04.370099+0000","last_became_peered":"2024-04-25T00:01:04.370099+0000","last_unstale":"2024-04-25T00:30:15.795644+0000","last_undegraded":"2024-04-25T00:30:15.795644+0000","last_fullsized":"2024-04-25T00:30:15.795644+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:42:03.181036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"74'5","reported_seq":126,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.138150+0000","last_change":"2024-04-25T00:01:18.475256+0000","last_active":"2024-04-25T00:30:26.138150+0000","last_peered":"2024-04-25T00:30:26.138150+0000","last_clean":"2024-04-25T00:30:26.138150+0000","last_became_active":"2024-04-25T00:01:18.474952+0000","last_became_peered":"2024-04-25T00:01:18.474952+0000","last_unstale":"2024-04-25T00:30:26.138150+0000","last_undegraded":"2024-04-25T00:30:26.138150+0000","last_fullsized":"2024-04-25T00:30:26.138150+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:46.849870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"74'4048","reported_seq":4167,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.238601+0000","last_change":"2024-04-25T00:01:19.516833+0000","last_active":"2024-04-25T00:30:26.238601+0000","last_peered":"2024-04-25T00:30:26.238601+0000","last_clean":"2024-04-25T00:30:26.238601+0000","last_became_active":"2024-04-25T00:01:19.515926+0000","last_became_peered":"2024-04-25T00:01:19.515926+0000","last_unstale":"2024-04-25T00:30:26.238601+0000","last_undegraded":"2024-04-25T00:30:26.238601+0000","last_fullsized":"2024-04-25T00:30:26.238601+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4048,"log_dups_size":0,"ondisk_log_size":4048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:23:48.855381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23468,"num_write_kb":270462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"70'3273","reported_seq":3401,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.808120+0000","last_change":"2024-04-25T00:01:05.372591+0000","last_active":"2024-04-25T00:30:15.808120+0000","last_peered":"2024-04-25T00:30:15.808120+0000","last_clean":"2024-04-25T00:30:15.808120+0000","last_became_active":"2024-04-25T00:01:05.371204+0000","last_became_peered":"2024-04-25T00:01:05.371204+0000","last_unstale":"2024-04-25T00:30:15.808120+0000","last_undegraded":"2024-04-25T00:30:15.808120+0000","last_fullsized":"2024-04-25T00:30:15.808120+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3273,"log_dups_size":0,"ondisk_log_size":3273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:15.879673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":17647,"num_write_kb":246988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"70'4","reported_seq":156,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.793404+0000","last_change":"2024-04-25T00:01:04.372376+0000","last_active":"2024-04-25T00:30:15.793404+0000","last_peered":"2024-04-25T00:30:15.793404+0000","last_clean":"2024-04-25T00:30:15.793404+0000","last_became_active":"2024-04-25T00:01:04.372040+0000","last_became_peered":"2024-04-25T00:01:04.372040+0000","last_unstale":"2024-04-25T00:30:15.793404+0000","last_undegraded":"2024-04-25T00:30:15.793404+0000","last_fullsized":"2024-04-25T00:30:15.793404+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:21.729893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"74'2","reported_seq":123,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.736656+0000","last_change":"2024-04-25T00:01:18.487548+0000","last_active":"2024-04-25T00:30:26.736656+0000","last_peered":"2024-04-25T00:30:26.736656+0000","last_clean":"2024-04-25T00:30:26.736656+0000","last_became_active":"2024-04-25T00:01:18.487227+0000","last_became_peered":"2024-04-25T00:01:18.487227+0000","last_unstale":"2024-04-25T00:30:26.736656+0000","last_undegraded":"2024-04-25T00:30:26.736656+0000","last_fullsized":"2024-04-25T00:30:26.736656+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:55.074018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"74'4306","reported_seq":4427,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.642530+0000","last_change":"2024-04-25T00:01:19.528574+0000","last_active":"2024-04-25T00:30:26.642530+0000","last_peered":"2024-04-25T00:30:26.642530+0000","last_clean":"2024-04-25T00:30:26.642530+0000","last_became_active":"2024-04-25T00:01:19.525531+0000","last_became_peered":"2024-04-25T00:01:19.525531+0000","last_unstale":"2024-04-25T00:30:26.642530+0000","last_undegraded":"2024-04-25T00:30:26.642530+0000","last_fullsized":"2024-04-25T00:30:26.642530+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4306,"log_dups_size":0,"ondisk_log_size":4306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:47:08.132895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":31,"num_write":22571,"num_write_kb":279217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"70'3794","reported_seq":3920,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.804529+0000","last_change":"2024-04-25T00:01:05.364103+0000","last_active":"2024-04-25T00:30:15.804529+0000","last_peered":"2024-04-25T00:30:15.804529+0000","last_clean":"2024-04-25T00:30:15.804529+0000","last_became_active":"2024-04-25T00:01:05.363320+0000","last_became_peered":"2024-04-25T00:01:05.363320+0000","last_unstale":"2024-04-25T00:30:15.804529+0000","last_undegraded":"2024-04-25T00:30:15.804529+0000","last_fullsized":"2024-04-25T00:30:15.804529+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3794,"log_dups_size":0,"ondisk_log_size":3794,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:31.456792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20831,"num_write_kb":276143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"74'15","reported_seq":257,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.791248+0000","last_change":"2024-04-25T00:01:04.358365+0000","last_active":"2024-04-25T00:30:15.791248+0000","last_peered":"2024-04-25T00:30:15.791248+0000","last_clean":"2024-04-25T00:30:15.791248+0000","last_became_active":"2024-04-25T00:01:04.357571+0000","last_became_peered":"2024-04-25T00:01:04.357571+0000","last_unstale":"2024-04-25T00:30:15.791248+0000","last_undegraded":"2024-04-25T00:30:15.791248+0000","last_fullsized":"2024-04-25T00:30:15.791248+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:50:23.023114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":342,"num_read_kb":342,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"74'3","reported_seq":123,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.747647+0000","last_change":"2024-04-25T00:01:18.497679+0000","last_active":"2024-04-25T00:30:26.747647+0000","last_peered":"2024-04-25T00:30:26.747647+0000","last_clean":"2024-04-25T00:30:26.747647+0000","last_became_active":"2024-04-25T00:01:18.496489+0000","last_became_peered":"2024-04-25T00:01:18.496489+0000","last_unstale":"2024-04-25T00:30:26.747647+0000","last_undegraded":"2024-04-25T00:30:26.747647+0000","last_fullsized":"2024-04-25T00:30:26.747647+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:23:47.845624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"74'4849","reported_seq":4970,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.756589+0000","last_change":"2024-04-25T00:01:19.498082+0000","last_active":"2024-04-25T00:30:26.756589+0000","last_peered":"2024-04-25T00:30:26.756589+0000","last_clean":"2024-04-25T00:30:26.756589+0000","last_became_active":"2024-04-25T00:01:19.496805+0000","last_became_peered":"2024-04-25T00:01:19.496805+0000","last_unstale":"2024-04-25T00:30:26.756589+0000","last_undegraded":"2024-04-25T00:30:26.756589+0000","last_fullsized":"2024-04-25T00:30:26.756589+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4849,"log_dups_size":0,"ondisk_log_size":4849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:57:36.256250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":16,"num_write":26442,"num_write_kb":335585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"70'2974","reported_seq":3100,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.941797+0000","last_change":"2024-04-25T00:01:05.373927+0000","last_active":"2024-04-25T00:30:15.941797+0000","last_peered":"2024-04-25T00:30:15.941797+0000","last_clean":"2024-04-25T00:30:15.941797+0000","last_became_active":"2024-04-25T00:01:05.373077+0000","last_became_peered":"2024-04-25T00:01:05.373077+0000","last_unstale":"2024-04-25T00:30:15.941797+0000","last_undegraded":"2024-04-25T00:30:15.941797+0000","last_fullsized":"2024-04-25T00:30:15.941797+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":2974,"log_dups_size":0,"ondisk_log_size":2974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:50.489125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15895,"num_write_kb":231645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"70'5","reported_seq":133,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.791512+0000","last_change":"2024-04-25T00:01:04.357265+0000","last_active":"2024-04-25T00:30:15.791512+0000","last_peered":"2024-04-25T00:30:15.791512+0000","last_clean":"2024-04-25T00:30:15.791512+0000","last_became_active":"2024-04-25T00:01:04.357004+0000","last_became_peered":"2024-04-25T00:01:04.357004+0000","last_unstale":"2024-04-25T00:30:15.791512+0000","last_undegraded":"2024-04-25T00:30:15.791512+0000","last_fullsized":"2024-04-25T00:30:15.791512+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:47:25.595275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"74'5","reported_seq":125,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.707490+0000","last_change":"2024-04-25T00:01:18.487276+0000","last_active":"2024-04-25T00:30:26.707490+0000","last_peered":"2024-04-25T00:30:26.707490+0000","last_clean":"2024-04-25T00:30:26.707490+0000","last_became_active":"2024-04-25T00:01:18.486279+0000","last_became_peered":"2024-04-25T00:01:18.486279+0000","last_unstale":"2024-04-25T00:30:26.707490+0000","last_undegraded":"2024-04-25T00:30:26.707490+0000","last_fullsized":"2024-04-25T00:30:26.707490+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:17:37.141843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"74'4040","reported_seq":4158,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.630478+0000","last_change":"2024-04-25T00:01:19.516827+0000","last_active":"2024-04-25T00:30:26.630478+0000","last_peered":"2024-04-25T00:30:26.630478+0000","last_clean":"2024-04-25T00:30:26.630478+0000","last_became_active":"2024-04-25T00:01:19.515765+0000","last_became_peered":"2024-04-25T00:01:19.515765+0000","last_unstale":"2024-04-25T00:30:26.630478+0000","last_undegraded":"2024-04-25T00:30:26.630478+0000","last_fullsized":"2024-04-25T00:30:26.630478+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4040,"log_dups_size":0,"ondisk_log_size":4040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:13:34.153117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23288,"num_write_kb":269674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"70'4514","reported_seq":4640,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.806193+0000","last_change":"2024-04-25T00:01:05.379501+0000","last_active":"2024-04-25T00:30:15.806193+0000","last_peered":"2024-04-25T00:30:15.806193+0000","last_clean":"2024-04-25T00:30:15.806193+0000","last_became_active":"2024-04-25T00:01:05.378644+0000","last_became_peered":"2024-04-25T00:01:05.378644+0000","last_unstale":"2024-04-25T00:30:15.806193+0000","last_undegraded":"2024-04-25T00:30:15.806193+0000","last_fullsized":"2024-04-25T00:30:15.806193+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4514,"log_dups_size":0,"ondisk_log_size":4514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:08:35.505154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25321,"num_write_kb":326961,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"70'3","reported_seq":131,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.809820+0000","last_change":"2024-04-25T00:01:04.360987+0000","last_active":"2024-04-25T00:30:15.809820+0000","last_peered":"2024-04-25T00:30:15.809820+0000","last_clean":"2024-04-25T00:30:15.809820+0000","last_became_active":"2024-04-25T00:01:04.360664+0000","last_became_peered":"2024-04-25T00:01:04.360664+0000","last_unstale":"2024-04-25T00:30:15.809820+0000","last_undegraded":"2024-04-25T00:30:15.809820+0000","last_fullsized":"2024-04-25T00:30:15.809820+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:55.283076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"74'334","reported_seq":571,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:19.197817+0000","last_change":"2024-04-25T00:01:18.505577+0000","last_active":"2024-04-25T00:30:19.197817+0000","last_peered":"2024-04-25T00:30:19.197817+0000","last_clean":"2024-04-25T00:30:19.197817+0000","last_became_active":"2024-04-25T00:01:18.504963+0000","last_became_peered":"2024-04-25T00:01:18.504963+0000","last_unstale":"2024-04-25T00:30:19.197817+0000","last_undegraded":"2024-04-25T00:30:19.197817+0000","last_fullsized":"2024-04-25T00:30:19.197817+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":334,"log_dups_size":0,"ondisk_log_size":334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:37.142619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":209,"num_read_kb":90302,"num_write":330,"num_write_kb":4274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"74'4674","reported_seq":4789,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.733748+0000","last_change":"2024-04-25T00:01:46.658442+0000","last_active":"2024-04-25T00:30:26.733748+0000","last_peered":"2024-04-25T00:30:26.733748+0000","last_clean":"2024-04-25T00:30:26.733748+0000","last_became_active":"2024-04-25T00:01:46.657588+0000","last_became_peered":"2024-04-25T00:01:46.657588+0000","last_unstale":"2024-04-25T00:30:26.733748+0000","last_undegraded":"2024-04-25T00:30:26.733748+0000","last_fullsized":"2024-04-25T00:30:26.733748+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4674,"log_dups_size":0,"ondisk_log_size":4674,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:58:49.738233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":16,"num_write":25396,"num_write_kb":335454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"70'3777","reported_seq":3903,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.810481+0000","last_change":"2024-04-25T00:01:05.379102+0000","last_active":"2024-04-25T00:30:15.810481+0000","last_peered":"2024-04-25T00:30:15.810481+0000","last_clean":"2024-04-25T00:30:15.810481+0000","last_became_active":"2024-04-25T00:01:05.377625+0000","last_became_peered":"2024-04-25T00:01:05.377625+0000","last_unstale":"2024-04-25T00:30:15.810481+0000","last_undegraded":"2024-04-25T00:30:15.810481+0000","last_fullsized":"2024-04-25T00:30:15.810481+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3777,"log_dups_size":0,"ondisk_log_size":3777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:15.879673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20730,"num_write_kb":283926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"73'9","reported_seq":183,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.803482+0000","last_change":"2024-04-25T00:01:04.370640+0000","last_active":"2024-04-25T00:30:15.803482+0000","last_peered":"2024-04-25T00:30:15.803482+0000","last_clean":"2024-04-25T00:30:15.803482+0000","last_became_active":"2024-04-25T00:01:04.370031+0000","last_became_peered":"2024-04-25T00:01:04.370031+0000","last_unstale":"2024-04-25T00:30:15.803482+0000","last_undegraded":"2024-04-25T00:30:15.803482+0000","last_fullsized":"2024-04-25T00:30:15.803482+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:26.243635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":184,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"74'31","reported_seq":220,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.781231+0000","last_change":"2024-04-25T00:01:18.473113+0000","last_active":"2024-04-25T00:30:26.781231+0000","last_peered":"2024-04-25T00:30:26.781231+0000","last_clean":"2024-04-25T00:30:26.781231+0000","last_became_active":"2024-04-25T00:01:18.472931+0000","last_became_peered":"2024-04-25T00:01:18.472931+0000","last_unstale":"2024-04-25T00:30:26.781231+0000","last_undegraded":"2024-04-25T00:30:26.781231+0000","last_fullsized":"2024-04-25T00:30:26.781231+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:07.498985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":161,"num_read_kb":161,"num_write":32,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"74'5042","reported_seq":5161,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.730299+0000","last_change":"2024-04-25T00:01:19.533664+0000","last_active":"2024-04-25T00:30:26.730299+0000","last_peered":"2024-04-25T00:30:26.730299+0000","last_clean":"2024-04-25T00:30:26.730299+0000","last_became_active":"2024-04-25T00:01:19.533048+0000","last_became_peered":"2024-04-25T00:01:19.533048+0000","last_unstale":"2024-04-25T00:30:26.730299+0000","last_undegraded":"2024-04-25T00:30:26.730299+0000","last_fullsized":"2024-04-25T00:30:26.730299+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":5042,"log_dups_size":0,"ondisk_log_size":5042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:45:43.141569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":28058,"num_write_kb":350341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"69'3549","reported_seq":3675,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.790799+0000","last_change":"2024-04-25T00:01:05.385827+0000","last_active":"2024-04-25T00:30:15.790799+0000","last_peered":"2024-04-25T00:30:15.790799+0000","last_clean":"2024-04-25T00:30:15.790799+0000","last_became_active":"2024-04-25T00:01:05.385258+0000","last_became_peered":"2024-04-25T00:01:05.385258+0000","last_unstale":"2024-04-25T00:30:15.790799+0000","last_undegraded":"2024-04-25T00:30:15.790799+0000","last_fullsized":"2024-04-25T00:30:15.790799+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3549,"log_dups_size":0,"ondisk_log_size":3549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:23.870121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19806,"num_write_kb":251506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"74'341","reported_seq":585,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:22.899825+0000","last_change":"2024-04-25T00:01:04.365185+0000","last_active":"2024-04-25T00:30:22.899825+0000","last_peered":"2024-04-25T00:30:22.899825+0000","last_clean":"2024-04-25T00:30:22.899825+0000","last_became_active":"2024-04-25T00:01:04.364930+0000","last_became_peered":"2024-04-25T00:01:04.364930+0000","last_unstale":"2024-04-25T00:30:22.899825+0000","last_undegraded":"2024-04-25T00:30:22.899825+0000","last_fullsized":"2024-04-25T00:30:22.899825+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":341,"log_dups_size":0,"ondisk_log_size":341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:45:03.618340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":208,"num_read_kb":94392,"num_write":336,"num_write_kb":4266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"74'3","reported_seq":123,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.747231+0000","last_change":"2024-04-25T00:01:18.496220+0000","last_active":"2024-04-25T00:30:26.747231+0000","last_peered":"2024-04-25T00:30:26.747231+0000","last_clean":"2024-04-25T00:30:26.747231+0000","last_became_active":"2024-04-25T00:01:18.492850+0000","last_became_peered":"2024-04-25T00:01:18.492850+0000","last_unstale":"2024-04-25T00:30:26.747231+0000","last_undegraded":"2024-04-25T00:30:26.747231+0000","last_fullsized":"2024-04-25T00:30:26.747231+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:50:16.922314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"74'4133","reported_seq":4252,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.680551+0000","last_change":"2024-04-25T00:01:19.515725+0000","last_active":"2024-04-25T00:30:26.680551+0000","last_peered":"2024-04-25T00:30:26.680551+0000","last_clean":"2024-04-25T00:30:26.680551+0000","last_became_active":"2024-04-25T00:01:19.509285+0000","last_became_peered":"2024-04-25T00:01:19.509285+0000","last_unstale":"2024-04-25T00:30:26.680551+0000","last_undegraded":"2024-04-25T00:30:26.680551+0000","last_fullsized":"2024-04-25T00:30:26.680551+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4133,"log_dups_size":0,"ondisk_log_size":4133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:25:20.185355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":4,"num_write":22847,"num_write_kb":280058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"70'5270","reported_seq":5396,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.809540+0000","last_change":"2024-04-25T00:01:05.371061+0000","last_active":"2024-04-25T00:30:15.809540+0000","last_peered":"2024-04-25T00:30:15.809540+0000","last_clean":"2024-04-25T00:30:15.809540+0000","last_became_active":"2024-04-25T00:01:05.369950+0000","last_became_peered":"2024-04-25T00:01:05.369950+0000","last_unstale":"2024-04-25T00:30:15.809540+0000","last_undegraded":"2024-04-25T00:30:15.809540+0000","last_fullsized":"2024-04-25T00:30:15.809540+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":5270,"log_dups_size":0,"ondisk_log_size":5270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:02:43.237902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28224,"num_write_kb":363198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"74'31","reported_seq":205,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:25.818373+0000","last_change":"2024-04-25T00:01:04.366503+0000","last_active":"2024-04-25T00:30:25.818373+0000","last_peered":"2024-04-25T00:30:25.818373+0000","last_clean":"2024-04-25T00:30:25.818373+0000","last_became_active":"2024-04-25T00:01:04.365945+0000","last_became_peered":"2024-04-25T00:01:04.365945+0000","last_unstale":"2024-04-25T00:30:25.818373+0000","last_undegraded":"2024-04-25T00:30:25.818373+0000","last_fullsized":"2024-04-25T00:30:25.818373+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":92,"num_write":32,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"74'8","reported_seq":174,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.742489+0000","last_change":"2024-04-25T00:01:18.476456+0000","last_active":"2024-04-25T00:30:26.742489+0000","last_peered":"2024-04-25T00:30:26.742489+0000","last_clean":"2024-04-25T00:30:26.742489+0000","last_became_active":"2024-04-25T00:01:18.475876+0000","last_became_peered":"2024-04-25T00:01:18.475876+0000","last_unstale":"2024-04-25T00:30:26.742489+0000","last_undegraded":"2024-04-25T00:30:26.742489+0000","last_fullsized":"2024-04-25T00:30:26.742489+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:56.741250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":184,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"74'4233","reported_seq":4354,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.727907+0000","last_change":"2024-04-25T00:01:19.515376+0000","last_active":"2024-04-25T00:30:26.727907+0000","last_peered":"2024-04-25T00:30:26.727907+0000","last_clean":"2024-04-25T00:30:26.727907+0000","last_became_active":"2024-04-25T00:01:19.512425+0000","last_became_peered":"2024-04-25T00:01:19.512425+0000","last_unstale":"2024-04-25T00:30:26.727907+0000","last_undegraded":"2024-04-25T00:30:26.727907+0000","last_fullsized":"2024-04-25T00:30:26.727907+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4233,"log_dups_size":0,"ondisk_log_size":4233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:43:16.957273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":24291,"num_write_kb":290168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"70'4413","reported_seq":4539,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.800988+0000","last_change":"2024-04-25T00:01:05.361723+0000","last_active":"2024-04-25T00:30:15.800988+0000","last_peered":"2024-04-25T00:30:15.800988+0000","last_clean":"2024-04-25T00:30:15.800988+0000","last_became_active":"2024-04-25T00:01:05.360956+0000","last_became_peered":"2024-04-25T00:01:05.360956+0000","last_unstale":"2024-04-25T00:30:15.800988+0000","last_undegraded":"2024-04-25T00:30:15.800988+0000","last_fullsized":"2024-04-25T00:30:15.800988+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4413,"log_dups_size":0,"ondisk_log_size":4413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:23.870121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22903,"num_write_kb":321461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"70'3","reported_seq":131,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.790859+0000","last_change":"2024-04-25T00:01:04.362129+0000","last_active":"2024-04-25T00:30:15.790859+0000","last_peered":"2024-04-25T00:30:15.790859+0000","last_clean":"2024-04-25T00:30:15.790859+0000","last_became_active":"2024-04-25T00:01:04.361827+0000","last_became_peered":"2024-04-25T00:01:04.361827+0000","last_unstale":"2024-04-25T00:30:15.790859+0000","last_undegraded":"2024-04-25T00:30:15.790859+0000","last_fullsized":"2024-04-25T00:30:15.790859+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:26.243635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"74'15","reported_seq":253,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.739696+0000","last_change":"2024-04-25T00:01:18.496129+0000","last_active":"2024-04-25T00:30:26.739696+0000","last_peered":"2024-04-25T00:30:26.739696+0000","last_clean":"2024-04-25T00:30:26.739696+0000","last_became_active":"2024-04-25T00:01:18.495415+0000","last_became_peered":"2024-04-25T00:01:18.495415+0000","last_unstale":"2024-04-25T00:30:26.739696+0000","last_undegraded":"2024-04-25T00:30:26.739696+0000","last_fullsized":"2024-04-25T00:30:26.739696+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:38.089411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":354,"num_read_kb":354,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"74'3474","reported_seq":3592,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.668678+0000","last_change":"2024-04-25T00:01:19.509542+0000","last_active":"2024-04-25T00:30:26.668678+0000","last_peered":"2024-04-25T00:30:26.668678+0000","last_clean":"2024-04-25T00:30:26.668678+0000","last_became_active":"2024-04-25T00:01:19.509015+0000","last_became_peered":"2024-04-25T00:01:19.509015+0000","last_unstale":"2024-04-25T00:30:26.668678+0000","last_undegraded":"2024-04-25T00:30:26.668678+0000","last_fullsized":"2024-04-25T00:30:26.668678+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3474,"log_dups_size":0,"ondisk_log_size":3474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:25:20.185355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19390,"num_write_kb":245789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"70'4331","reported_seq":4457,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.806796+0000","last_change":"2024-04-25T00:01:05.365413+0000","last_active":"2024-04-25T00:30:15.806796+0000","last_peered":"2024-04-25T00:30:15.806796+0000","last_clean":"2024-04-25T00:30:15.806796+0000","last_became_active":"2024-04-25T00:01:05.361865+0000","last_became_peered":"2024-04-25T00:01:05.361865+0000","last_unstale":"2024-04-25T00:30:15.806796+0000","last_undegraded":"2024-04-25T00:30:15.806796+0000","last_fullsized":"2024-04-25T00:30:15.806796+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4331,"log_dups_size":0,"ondisk_log_size":4331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:30.228060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22828,"num_write_kb":312693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"74'7","reported_seq":135,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.944255+0000","last_change":"2024-04-25T00:01:04.360804+0000","last_active":"2024-04-25T00:30:15.944255+0000","last_peered":"2024-04-25T00:30:15.944255+0000","last_clean":"2024-04-25T00:30:15.944255+0000","last_became_active":"2024-04-25T00:01:04.360374+0000","last_became_peered":"2024-04-25T00:01:04.360374+0000","last_unstale":"2024-04-25T00:30:15.944255+0000","last_undegraded":"2024-04-25T00:30:15.944255+0000","last_fullsized":"2024-04-25T00:30:15.944255+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"74'5","reported_seq":125,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.739206+0000","last_change":"2024-04-25T00:01:18.480003+0000","last_active":"2024-04-25T00:30:26.739206+0000","last_peered":"2024-04-25T00:30:26.739206+0000","last_clean":"2024-04-25T00:30:26.739206+0000","last_became_active":"2024-04-25T00:01:18.479270+0000","last_became_peered":"2024-04-25T00:01:18.479270+0000","last_unstale":"2024-04-25T00:30:26.739206+0000","last_undegraded":"2024-04-25T00:30:26.739206+0000","last_fullsized":"2024-04-25T00:30:26.739206+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:52:40.942502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"74'5681","reported_seq":5805,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.778355+0000","last_change":"2024-04-25T00:01:19.524463+0000","last_active":"2024-04-25T00:30:26.778355+0000","last_peered":"2024-04-25T00:30:26.778355+0000","last_clean":"2024-04-25T00:30:26.778355+0000","last_became_active":"2024-04-25T00:01:19.524166+0000","last_became_peered":"2024-04-25T00:01:19.524166+0000","last_unstale":"2024-04-25T00:30:26.778355+0000","last_undegraded":"2024-04-25T00:30:26.778355+0000","last_fullsized":"2024-04-25T00:30:26.778355+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":5681,"log_dups_size":0,"ondisk_log_size":5681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:50.351599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":40,"num_write":31879,"num_write_kb":410681,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"74'28","reported_seq":150,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.738030+0000","last_change":"2024-04-25T00:01:18.497096+0000","last_active":"2024-04-25T00:30:26.738030+0000","last_peered":"2024-04-25T00:30:26.738030+0000","last_clean":"2024-04-25T00:30:26.738030+0000","last_became_active":"2024-04-25T00:01:18.496289+0000","last_became_peered":"2024-04-25T00:01:18.496289+0000","last_unstale":"2024-04-25T00:30:26.738030+0000","last_undegraded":"2024-04-25T00:30:26.738030+0000","last_fullsized":"2024-04-25T00:30:26.738030+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:59:40.316831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"70'4773","reported_seq":4900,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.791447+0000","last_change":"2024-04-25T00:01:05.370043+0000","last_active":"2024-04-25T00:30:15.791447+0000","last_peered":"2024-04-25T00:30:15.791447+0000","last_clean":"2024-04-25T00:30:15.791447+0000","last_became_active":"2024-04-25T00:01:05.368080+0000","last_became_peered":"2024-04-25T00:01:05.368080+0000","last_unstale":"2024-04-25T00:30:15.791447+0000","last_undegraded":"2024-04-25T00:30:15.791447+0000","last_fullsized":"2024-04-25T00:30:15.791447+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4773,"log_dups_size":0,"ondisk_log_size":4773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:58:07.610923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":25218,"num_write_kb":330520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"69'2","reported_seq":130,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.804702+0000","last_change":"2024-04-25T00:01:04.364523+0000","last_active":"2024-04-25T00:30:15.804702+0000","last_peered":"2024-04-25T00:30:15.804702+0000","last_clean":"2024-04-25T00:30:15.804702+0000","last_became_active":"2024-04-25T00:01:04.364050+0000","last_became_peered":"2024-04-25T00:01:04.364050+0000","last_unstale":"2024-04-25T00:30:15.804702+0000","last_undegraded":"2024-04-25T00:30:15.804702+0000","last_fullsized":"2024-04-25T00:30:15.804702+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:49.479804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"74'3390","reported_seq":3510,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.735796+0000","last_change":"2024-04-25T00:01:19.491523+0000","last_active":"2024-04-25T00:30:26.735796+0000","last_peered":"2024-04-25T00:30:26.735796+0000","last_clean":"2024-04-25T00:30:26.735796+0000","last_became_active":"2024-04-25T00:01:19.490755+0000","last_became_peered":"2024-04-25T00:01:19.490755+0000","last_unstale":"2024-04-25T00:30:26.735796+0000","last_undegraded":"2024-04-25T00:30:26.735796+0000","last_fullsized":"2024-04-25T00:30:26.735796+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3390,"log_dups_size":0,"ondisk_log_size":3390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:47:19.526845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18979,"num_write_kb":248305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"74'1082","reported_seq":1259,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.777825+0000","last_change":"2024-04-25T00:01:18.472794+0000","last_active":"2024-04-25T00:30:26.777825+0000","last_peered":"2024-04-25T00:30:26.777825+0000","last_clean":"2024-04-25T00:30:26.777825+0000","last_became_active":"2024-04-25T00:01:18.472534+0000","last_became_peered":"2024-04-25T00:01:18.472534+0000","last_unstale":"2024-04-25T00:30:26.777825+0000","last_undegraded":"2024-04-25T00:30:26.777825+0000","last_fullsized":"2024-04-25T00:30:26.777825+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":1082,"log_dups_size":0,"ondisk_log_size":1082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:55:45.843808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":101,"num_read_kb":11812,"num_write":1055,"num_write_kb":4728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"70'4239","reported_seq":4365,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.794924+0000","last_change":"2024-04-25T00:01:05.378867+0000","last_active":"2024-04-25T00:30:15.794924+0000","last_peered":"2024-04-25T00:30:15.794924+0000","last_clean":"2024-04-25T00:30:15.794924+0000","last_became_active":"2024-04-25T00:01:05.378056+0000","last_became_peered":"2024-04-25T00:01:05.378056+0000","last_unstale":"2024-04-25T00:30:15.794924+0000","last_undegraded":"2024-04-25T00:30:15.794924+0000","last_fullsized":"2024-04-25T00:30:15.794924+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4239,"log_dups_size":0,"ondisk_log_size":4239,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:36:10.377757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22560,"num_write_kb":311538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"70'3","reported_seq":131,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.809380+0000","last_change":"2024-04-25T00:01:04.370896+0000","last_active":"2024-04-25T00:30:15.809380+0000","last_peered":"2024-04-25T00:30:15.809380+0000","last_clean":"2024-04-25T00:30:15.809380+0000","last_became_active":"2024-04-25T00:01:04.370273+0000","last_became_peered":"2024-04-25T00:01:04.370273+0000","last_unstale":"2024-04-25T00:30:15.809380+0000","last_undegraded":"2024-04-25T00:30:15.809380+0000","last_fullsized":"2024-04-25T00:30:15.809380+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:49:22.673421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"74'5148","reported_seq":5266,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.723815+0000","last_change":"2024-04-25T00:01:19.521143+0000","last_active":"2024-04-25T00:30:26.723815+0000","last_peered":"2024-04-25T00:30:26.723815+0000","last_clean":"2024-04-25T00:30:26.723815+0000","last_became_active":"2024-04-25T00:01:19.514664+0000","last_became_peered":"2024-04-25T00:01:19.514664+0000","last_unstale":"2024-04-25T00:30:26.723815+0000","last_undegraded":"2024-04-25T00:30:26.723815+0000","last_fullsized":"2024-04-25T00:30:26.723815+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":5148,"log_dups_size":0,"ondisk_log_size":5148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:03:47.440442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":29131,"num_write_kb":338102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"74'249","reported_seq":380,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:27.894626+0000","last_change":"2024-04-25T00:01:04.360254+0000","last_active":"2024-04-25T00:30:27.894626+0000","last_peered":"2024-04-25T00:30:27.894626+0000","last_clean":"2024-04-25T00:30:27.894626+0000","last_became_active":"2024-04-25T00:01:04.359586+0000","last_became_peered":"2024-04-25T00:01:04.359586+0000","last_unstale":"2024-04-25T00:30:27.894626+0000","last_undegraded":"2024-04-25T00:30:27.894626+0000","last_fullsized":"2024-04-25T00:30:27.894626+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:47.192930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2250578,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4308,"num_write":220,"num_write_kb":2323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"69'3397","reported_seq":3522,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.792637+0000","last_change":"2024-04-25T00:01:05.370723+0000","last_active":"2024-04-25T00:30:15.792637+0000","last_peered":"2024-04-25T00:30:15.792637+0000","last_clean":"2024-04-25T00:30:15.792637+0000","last_became_active":"2024-04-25T00:01:05.368970+0000","last_became_peered":"2024-04-25T00:01:05.368970+0000","last_unstale":"2024-04-25T00:30:15.792637+0000","last_undegraded":"2024-04-25T00:30:15.792637+0000","last_fullsized":"2024-04-25T00:30:15.792637+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3397,"log_dups_size":0,"ondisk_log_size":3397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:06:53.743074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17978,"num_write_kb":244099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"74'5","reported_seq":170,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:19.198496+0000","last_change":"2024-04-25T00:01:18.498570+0000","last_active":"2024-04-25T00:30:19.198496+0000","last_peered":"2024-04-25T00:30:19.198496+0000","last_clean":"2024-04-25T00:30:19.198496+0000","last_became_active":"2024-04-25T00:01:18.497745+0000","last_became_peered":"2024-04-25T00:01:18.497745+0000","last_unstale":"2024-04-25T00:30:19.198496+0000","last_undegraded":"2024-04-25T00:30:19.198496+0000","last_fullsized":"2024-04-25T00:30:19.198496+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:26:42.047149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"74'3919","reported_seq":4039,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.527005+0000","last_change":"2024-04-25T00:01:19.514919+0000","last_active":"2024-04-25T00:30:26.527005+0000","last_peered":"2024-04-25T00:30:26.527005+0000","last_clean":"2024-04-25T00:30:26.527005+0000","last_became_active":"2024-04-25T00:01:19.514280+0000","last_became_peered":"2024-04-25T00:01:19.514280+0000","last_unstale":"2024-04-25T00:30:26.527005+0000","last_undegraded":"2024-04-25T00:30:26.527005+0000","last_fullsized":"2024-04-25T00:30:26.527005+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3919,"log_dups_size":0,"ondisk_log_size":3919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:37.975265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":20874,"num_write_kb":280510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"69'752","reported_seq":907,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.801549+0000","last_change":"2024-04-25T00:01:04.365607+0000","last_active":"2024-04-25T00:30:15.801549+0000","last_peered":"2024-04-25T00:30:15.801549+0000","last_clean":"2024-04-25T00:30:15.801549+0000","last_became_active":"2024-04-25T00:01:04.364788+0000","last_became_peered":"2024-04-25T00:01:04.364788+0000","last_unstale":"2024-04-25T00:30:15.801549+0000","last_undegraded":"2024-04-25T00:30:15.801549+0000","last_fullsized":"2024-04-25T00:30:15.801549+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":752,"log_dups_size":0,"ondisk_log_size":752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":28,"num_read_kb":89132,"num_write":742,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"69'3790","reported_seq":3915,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.942896+0000","last_change":"2024-04-25T00:01:05.375142+0000","last_active":"2024-04-25T00:30:15.942896+0000","last_peered":"2024-04-25T00:30:15.942896+0000","last_clean":"2024-04-25T00:30:15.942896+0000","last_became_active":"2024-04-25T00:01:05.373857+0000","last_became_peered":"2024-04-25T00:01:05.373857+0000","last_unstale":"2024-04-25T00:30:15.942896+0000","last_undegraded":"2024-04-25T00:30:15.942896+0000","last_fullsized":"2024-04-25T00:30:15.942896+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3790,"log_dups_size":0,"ondisk_log_size":3790,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:15.879673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19857,"num_write_kb":263976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"74'48","reported_seq":213,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.740146+0000","last_change":"2024-04-25T00:01:18.502786+0000","last_active":"2024-04-25T00:30:26.740146+0000","last_peered":"2024-04-25T00:30:26.740146+0000","last_clean":"2024-04-25T00:30:26.740146+0000","last_became_active":"2024-04-25T00:01:18.501785+0000","last_became_peered":"2024-04-25T00:01:18.501785+0000","last_unstale":"2024-04-25T00:30:26.740146+0000","last_undegraded":"2024-04-25T00:30:26.740146+0000","last_fullsized":"2024-04-25T00:30:26.740146+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:09:31.441044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.26","version":"74'3694","reported_seq":3813,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.735546+0000","last_change":"2024-04-25T00:01:19.511239+0000","last_active":"2024-04-25T00:30:26.735546+0000","last_peered":"2024-04-25T00:30:26.735546+0000","last_clean":"2024-04-25T00:30:26.735546+0000","last_became_active":"2024-04-25T00:01:19.510155+0000","last_became_peered":"2024-04-25T00:01:19.510155+0000","last_unstale":"2024-04-25T00:30:26.735546+0000","last_undegraded":"2024-04-25T00:30:26.735546+0000","last_fullsized":"2024-04-25T00:30:26.735546+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3694,"log_dups_size":0,"ondisk_log_size":3694,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:43:14.452663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20834,"num_write_kb":259761,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.20","version":"70'4043","reported_seq":4171,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.807721+0000","last_change":"2024-04-25T00:01:05.378517+0000","last_active":"2024-04-25T00:30:15.807721+0000","last_peered":"2024-04-25T00:30:15.807721+0000","last_clean":"2024-04-25T00:30:15.807721+0000","last_became_active":"2024-04-25T00:01:05.377742+0000","last_became_peered":"2024-04-25T00:01:05.377742+0000","last_unstale":"2024-04-25T00:30:15.807721+0000","last_undegraded":"2024-04-25T00:30:15.807721+0000","last_fullsized":"2024-04-25T00:30:15.807721+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4043,"log_dups_size":0,"ondisk_log_size":4043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:06.450369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21543,"num_write_kb":291017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"74'5759","reported_seq":5879,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.589637+0000","last_change":"2024-04-25T00:01:19.508999+0000","last_active":"2024-04-25T00:30:26.589637+0000","last_peered":"2024-04-25T00:30:26.589637+0000","last_clean":"2024-04-25T00:30:26.589637+0000","last_became_active":"2024-04-25T00:01:19.507960+0000","last_became_peered":"2024-04-25T00:01:19.507960+0000","last_unstale":"2024-04-25T00:30:26.589637+0000","last_undegraded":"2024-04-25T00:30:26.589637+0000","last_fullsized":"2024-04-25T00:30:26.589637+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":5759,"log_dups_size":0,"ondisk_log_size":5759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:32:40.274908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":31168,"num_write_kb":366885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.21","version":"70'4068","reported_seq":4197,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.803508+0000","last_change":"2024-04-25T00:01:05.372927+0000","last_active":"2024-04-25T00:30:15.803508+0000","last_peered":"2024-04-25T00:30:15.803508+0000","last_clean":"2024-04-25T00:30:15.803508+0000","last_became_active":"2024-04-25T00:01:05.372050+0000","last_became_peered":"2024-04-25T00:01:05.372050+0000","last_unstale":"2024-04-25T00:30:15.803508+0000","last_undegraded":"2024-04-25T00:30:15.803508+0000","last_fullsized":"2024-04-25T00:30:15.803508+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4068,"log_dups_size":0,"ondisk_log_size":4068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:48.202251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21791,"num_write_kb":292243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.24","version":"74'4315","reported_seq":4436,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.724152+0000","last_change":"2024-04-25T00:01:19.531072+0000","last_active":"2024-04-25T00:30:26.724152+0000","last_peered":"2024-04-25T00:30:26.724152+0000","last_clean":"2024-04-25T00:30:26.724152+0000","last_became_active":"2024-04-25T00:01:19.530836+0000","last_became_peered":"2024-04-25T00:01:19.530836+0000","last_unstale":"2024-04-25T00:30:26.724152+0000","last_undegraded":"2024-04-25T00:30:26.724152+0000","last_fullsized":"2024-04-25T00:30:26.724152+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4315,"log_dups_size":0,"ondisk_log_size":4315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:08.508741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":23956,"num_write_kb":309408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"70'4267","reported_seq":4395,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.807204+0000","last_change":"2024-04-25T00:01:05.362960+0000","last_active":"2024-04-25T00:30:15.807204+0000","last_peered":"2024-04-25T00:30:15.807204+0000","last_clean":"2024-04-25T00:30:15.807204+0000","last_became_active":"2024-04-25T00:01:05.361067+0000","last_became_peered":"2024-04-25T00:01:05.361067+0000","last_unstale":"2024-04-25T00:30:15.807204+0000","last_undegraded":"2024-04-25T00:30:15.807204+0000","last_fullsized":"2024-04-25T00:30:15.807204+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4267,"log_dups_size":0,"ondisk_log_size":4267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:24.017263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23473,"num_write_kb":311317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"74'4634","reported_seq":4754,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.737724+0000","last_change":"2024-04-25T00:01:19.506029+0000","last_active":"2024-04-25T00:30:26.737724+0000","last_peered":"2024-04-25T00:30:26.737724+0000","last_clean":"2024-04-25T00:30:26.737724+0000","last_became_active":"2024-04-25T00:01:19.505184+0000","last_became_peered":"2024-04-25T00:01:19.505184+0000","last_unstale":"2024-04-25T00:30:26.737724+0000","last_undegraded":"2024-04-25T00:30:26.737724+0000","last_fullsized":"2024-04-25T00:30:26.737724+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4634,"log_dups_size":0,"ondisk_log_size":4634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:08.508741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":6,"num_write":26896,"num_write_kb":316898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"70'4083","reported_seq":4211,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.801208+0000","last_change":"2024-04-25T00:01:05.358690+0000","last_active":"2024-04-25T00:30:15.801208+0000","last_peered":"2024-04-25T00:30:15.801208+0000","last_clean":"2024-04-25T00:30:15.801208+0000","last_became_active":"2024-04-25T00:01:05.357340+0000","last_became_peered":"2024-04-25T00:01:05.357340+0000","last_unstale":"2024-04-25T00:30:15.801208+0000","last_undegraded":"2024-04-25T00:30:15.801208+0000","last_fullsized":"2024-04-25T00:30:15.801208+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4083,"log_dups_size":0,"ondisk_log_size":4083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:36:55.436661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21935,"num_write_kb":296816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.22","version":"74'5147","reported_seq":5272,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.738948+0000","last_change":"2024-04-25T00:01:19.508691+0000","last_active":"2024-04-25T00:30:26.738948+0000","last_peered":"2024-04-25T00:30:26.738948+0000","last_clean":"2024-04-25T00:30:26.738948+0000","last_became_active":"2024-04-25T00:01:19.508123+0000","last_became_peered":"2024-04-25T00:01:19.508123+0000","last_unstale":"2024-04-25T00:30:26.738948+0000","last_undegraded":"2024-04-25T00:30:26.738948+0000","last_fullsized":"2024-04-25T00:30:26.738948+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":5147,"log_dups_size":0,"ondisk_log_size":5147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:47:19.526845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":43,"num_write":27901,"num_write_kb":341497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"70'3848","reported_seq":3976,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.812551+0000","last_change":"2024-04-25T00:01:05.387057+0000","last_active":"2024-04-25T00:30:15.812551+0000","last_peered":"2024-04-25T00:30:15.812551+0000","last_clean":"2024-04-25T00:30:15.812551+0000","last_became_active":"2024-04-25T00:01:05.386724+0000","last_became_peered":"2024-04-25T00:01:05.386724+0000","last_unstale":"2024-04-25T00:30:15.812551+0000","last_undegraded":"2024-04-25T00:30:15.812551+0000","last_fullsized":"2024-04-25T00:30:15.812551+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3848,"log_dups_size":0,"ondisk_log_size":3848,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:55:09.073286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20848,"num_write_kb":276248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.23","version":"74'4386","reported_seq":4508,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.730888+0000","last_change":"2024-04-25T00:01:19.522760+0000","last_active":"2024-04-25T00:30:26.730888+0000","last_peered":"2024-04-25T00:30:26.730888+0000","last_clean":"2024-04-25T00:30:26.730888+0000","last_became_active":"2024-04-25T00:01:19.522380+0000","last_became_peered":"2024-04-25T00:01:19.522380+0000","last_unstale":"2024-04-25T00:30:26.730888+0000","last_undegraded":"2024-04-25T00:30:26.730888+0000","last_fullsized":"2024-04-25T00:30:26.730888+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4386,"log_dups_size":0,"ondisk_log_size":4386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:55:46.853565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":23395,"num_write_kb":294532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.25","version":"70'4349","reported_seq":4476,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.812426+0000","last_change":"2024-04-25T00:01:05.367296+0000","last_active":"2024-04-25T00:30:15.812426+0000","last_peered":"2024-04-25T00:30:15.812426+0000","last_clean":"2024-04-25T00:30:15.812426+0000","last_became_active":"2024-04-25T00:01:05.363838+0000","last_became_peered":"2024-04-25T00:01:05.363838+0000","last_unstale":"2024-04-25T00:30:15.812426+0000","last_undegraded":"2024-04-25T00:30:15.812426+0000","last_fullsized":"2024-04-25T00:30:15.812426+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4349,"log_dups_size":0,"ondisk_log_size":4349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:43:07.053074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24353,"num_write_kb":306782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"74'4505","reported_seq":4625,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.749021+0000","last_change":"2024-04-25T00:01:19.510902+0000","last_active":"2024-04-25T00:30:26.749021+0000","last_peered":"2024-04-25T00:30:26.749021+0000","last_clean":"2024-04-25T00:30:26.749021+0000","last_became_active":"2024-04-25T00:01:19.510008+0000","last_became_peered":"2024-04-25T00:01:19.510008+0000","last_unstale":"2024-04-25T00:30:26.749021+0000","last_undegraded":"2024-04-25T00:30:26.749021+0000","last_fullsized":"2024-04-25T00:30:26.749021+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4505,"log_dups_size":0,"ondisk_log_size":4505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:40:32.292140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24621,"num_write_kb":315853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"70'3623","reported_seq":3750,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.806689+0000","last_change":"2024-04-25T00:01:05.365761+0000","last_active":"2024-04-25T00:30:15.806689+0000","last_peered":"2024-04-25T00:30:15.806689+0000","last_clean":"2024-04-25T00:30:15.806689+0000","last_became_active":"2024-04-25T00:01:05.361119+0000","last_became_peered":"2024-04-25T00:01:05.361119+0000","last_unstale":"2024-04-25T00:30:15.806689+0000","last_undegraded":"2024-04-25T00:30:15.806689+0000","last_fullsized":"2024-04-25T00:30:15.806689+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3623,"log_dups_size":0,"ondisk_log_size":3623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20289,"num_write_kb":274005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.21","version":"74'4291","reported_seq":4414,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.777806+0000","last_change":"2024-04-25T00:01:19.518534+0000","last_active":"2024-04-25T00:30:26.777806+0000","last_peered":"2024-04-25T00:30:26.777806+0000","last_clean":"2024-04-25T00:30:26.777806+0000","last_became_active":"2024-04-25T00:01:19.518110+0000","last_became_peered":"2024-04-25T00:01:19.518110+0000","last_unstale":"2024-04-25T00:30:26.777806+0000","last_undegraded":"2024-04-25T00:30:26.777806+0000","last_fullsized":"2024-04-25T00:30:26.777806+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4291,"log_dups_size":0,"ondisk_log_size":4291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:05:09.219823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":23037,"num_write_kb":301131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"69'4983","reported_seq":5110,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.799145+0000","last_change":"2024-04-25T00:01:05.373464+0000","last_active":"2024-04-25T00:30:15.799145+0000","last_peered":"2024-04-25T00:30:15.799145+0000","last_clean":"2024-04-25T00:30:15.799145+0000","last_became_active":"2024-04-25T00:01:05.372611+0000","last_became_peered":"2024-04-25T00:01:05.372611+0000","last_unstale":"2024-04-25T00:30:15.799145+0000","last_undegraded":"2024-04-25T00:30:15.799145+0000","last_fullsized":"2024-04-25T00:30:15.799145+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4983,"log_dups_size":0,"ondisk_log_size":4983,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:31.456792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26776,"num_write_kb":349057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2e","version":"74'4567","reported_seq":4690,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.735238+0000","last_change":"2024-04-25T00:01:19.542475+0000","last_active":"2024-04-25T00:30:26.735238+0000","last_peered":"2024-04-25T00:30:26.735238+0000","last_clean":"2024-04-25T00:30:26.735238+0000","last_became_active":"2024-04-25T00:01:19.541877+0000","last_became_peered":"2024-04-25T00:01:19.541877+0000","last_unstale":"2024-04-25T00:30:26.735238+0000","last_undegraded":"2024-04-25T00:30:26.735238+0000","last_fullsized":"2024-04-25T00:30:26.735238+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4567,"log_dups_size":0,"ondisk_log_size":4567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:02:57.373014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":36,"num_write":24879,"num_write_kb":312940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.28","version":"70'4294","reported_seq":4421,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.804148+0000","last_change":"2024-04-25T00:01:05.372906+0000","last_active":"2024-04-25T00:30:15.804148+0000","last_peered":"2024-04-25T00:30:15.804148+0000","last_clean":"2024-04-25T00:30:15.804148+0000","last_became_active":"2024-04-25T00:01:05.372148+0000","last_became_peered":"2024-04-25T00:01:05.372148+0000","last_unstale":"2024-04-25T00:30:15.804148+0000","last_undegraded":"2024-04-25T00:30:15.804148+0000","last_fullsized":"2024-04-25T00:30:15.804148+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4294,"log_dups_size":0,"ondisk_log_size":4294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:43.615895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23157,"num_write_kb":289708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2f","version":"74'5194","reported_seq":5313,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.771951+0000","last_change":"2024-04-25T00:01:19.498515+0000","last_active":"2024-04-25T00:30:26.771951+0000","last_peered":"2024-04-25T00:30:26.771951+0000","last_clean":"2024-04-25T00:30:26.771951+0000","last_became_active":"2024-04-25T00:01:19.496310+0000","last_became_peered":"2024-04-25T00:01:19.496310+0000","last_unstale":"2024-04-25T00:30:26.771951+0000","last_undegraded":"2024-04-25T00:30:26.771951+0000","last_fullsized":"2024-04-25T00:30:26.771951+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":5194,"log_dups_size":0,"ondisk_log_size":5194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:20:06.661558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":28420,"num_write_kb":367977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"69'4077","reported_seq":4204,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.795788+0000","last_change":"2024-04-25T00:01:05.366340+0000","last_active":"2024-04-25T00:30:15.795788+0000","last_peered":"2024-04-25T00:30:15.795788+0000","last_clean":"2024-04-25T00:30:15.795788+0000","last_became_active":"2024-04-25T00:01:05.361952+0000","last_became_peered":"2024-04-25T00:01:05.361952+0000","last_unstale":"2024-04-25T00:30:15.795788+0000","last_undegraded":"2024-04-25T00:30:15.795788+0000","last_fullsized":"2024-04-25T00:30:15.795788+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4077,"log_dups_size":0,"ondisk_log_size":4077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:41.948662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22153,"num_write_kb":297857,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2c","version":"74'3564","reported_seq":3684,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.750933+0000","last_change":"2024-04-25T00:01:19.508257+0000","last_active":"2024-04-25T00:30:26.750933+0000","last_peered":"2024-04-25T00:30:26.750933+0000","last_clean":"2024-04-25T00:30:26.750933+0000","last_became_active":"2024-04-25T00:01:19.507516+0000","last_became_peered":"2024-04-25T00:01:19.507516+0000","last_unstale":"2024-04-25T00:30:26.750933+0000","last_undegraded":"2024-04-25T00:30:26.750933+0000","last_fullsized":"2024-04-25T00:30:26.750933+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3564,"log_dups_size":0,"ondisk_log_size":3564,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:45.121513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":19793,"num_write_kb":247542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"69'4229","reported_seq":4356,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.942181+0000","last_change":"2024-04-25T00:01:05.387457+0000","last_active":"2024-04-25T00:30:15.942181+0000","last_peered":"2024-04-25T00:30:15.942181+0000","last_clean":"2024-04-25T00:30:15.942181+0000","last_became_active":"2024-04-25T00:01:05.386919+0000","last_became_peered":"2024-04-25T00:01:05.386919+0000","last_unstale":"2024-04-25T00:30:15.942181+0000","last_undegraded":"2024-04-25T00:30:15.942181+0000","last_fullsized":"2024-04-25T00:30:15.942181+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4229,"log_dups_size":0,"ondisk_log_size":4229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22430,"num_write_kb":302838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2d","version":"74'4662","reported_seq":4781,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.561243+0000","last_change":"2024-04-25T00:01:19.520648+0000","last_active":"2024-04-25T00:30:26.561243+0000","last_peered":"2024-04-25T00:30:26.561243+0000","last_clean":"2024-04-25T00:30:26.561243+0000","last_became_active":"2024-04-25T00:01:19.519303+0000","last_became_peered":"2024-04-25T00:01:19.519303+0000","last_unstale":"2024-04-25T00:30:26.561243+0000","last_undegraded":"2024-04-25T00:30:26.561243+0000","last_fullsized":"2024-04-25T00:30:26.561243+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4662,"log_dups_size":0,"ondisk_log_size":4662,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:23.993729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":25332,"num_write_kb":310579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2b","version":"70'4434","reported_seq":4562,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.792469+0000","last_change":"2024-04-25T00:01:05.366249+0000","last_active":"2024-04-25T00:30:15.792469+0000","last_peered":"2024-04-25T00:30:15.792469+0000","last_clean":"2024-04-25T00:30:15.792469+0000","last_became_active":"2024-04-25T00:01:05.365548+0000","last_became_peered":"2024-04-25T00:01:05.365548+0000","last_unstale":"2024-04-25T00:30:15.792469+0000","last_undegraded":"2024-04-25T00:30:15.792469+0000","last_fullsized":"2024-04-25T00:30:15.792469+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4434,"log_dups_size":0,"ondisk_log_size":4434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:22:21.143639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":24224,"num_write_kb":326776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"74'4527","reported_seq":4647,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.719005+0000","last_change":"2024-04-25T00:01:19.514659+0000","last_active":"2024-04-25T00:30:26.719005+0000","last_peered":"2024-04-25T00:30:26.719005+0000","last_clean":"2024-04-25T00:30:26.719005+0000","last_became_active":"2024-04-25T00:01:19.514232+0000","last_became_peered":"2024-04-25T00:01:19.514232+0000","last_unstale":"2024-04-25T00:30:26.719005+0000","last_undegraded":"2024-04-25T00:30:26.719005+0000","last_fullsized":"2024-04-25T00:30:26.719005+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4527,"log_dups_size":0,"ondisk_log_size":4527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:45.121513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":25088,"num_write_kb":313806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"70'3677","reported_seq":3789,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.802932+0000","last_change":"2024-04-25T00:01:46.665943+0000","last_active":"2024-04-25T00:30:15.802932+0000","last_peered":"2024-04-25T00:30:15.802932+0000","last_clean":"2024-04-25T00:30:15.802932+0000","last_became_active":"2024-04-25T00:01:46.663874+0000","last_became_peered":"2024-04-25T00:01:46.663874+0000","last_unstale":"2024-04-25T00:30:15.802932+0000","last_undegraded":"2024-04-25T00:30:15.802932+0000","last_fullsized":"2024-04-25T00:30:15.802932+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3677,"log_dups_size":0,"ondisk_log_size":3677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:53:00.633407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20325,"num_write_kb":253590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2b","version":"74'4916","reported_seq":5040,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.774860+0000","last_change":"2024-04-25T00:01:19.503419+0000","last_active":"2024-04-25T00:30:26.774860+0000","last_peered":"2024-04-25T00:30:26.774860+0000","last_clean":"2024-04-25T00:30:26.774860+0000","last_became_active":"2024-04-25T00:01:19.500568+0000","last_became_peered":"2024-04-25T00:01:19.500568+0000","last_unstale":"2024-04-25T00:30:26.774860+0000","last_undegraded":"2024-04-25T00:30:26.774860+0000","last_fullsized":"2024-04-25T00:30:26.774860+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4916,"log_dups_size":0,"ondisk_log_size":4916,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:08.508741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":26841,"num_write_kb":321097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"69'4618","reported_seq":4745,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.814383+0000","last_change":"2024-04-25T00:01:05.370653+0000","last_active":"2024-04-25T00:30:15.814383+0000","last_peered":"2024-04-25T00:30:15.814383+0000","last_clean":"2024-04-25T00:30:15.814383+0000","last_became_active":"2024-04-25T00:01:05.369919+0000","last_became_peered":"2024-04-25T00:01:05.369919+0000","last_unstale":"2024-04-25T00:30:15.814383+0000","last_undegraded":"2024-04-25T00:30:15.814383+0000","last_fullsized":"2024-04-25T00:30:15.814383+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4618,"log_dups_size":0,"ondisk_log_size":4618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:30:08.311326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25225,"num_write_kb":326547,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.28","version":"74'3900","reported_seq":4020,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.755166+0000","last_change":"2024-04-25T00:01:19.540104+0000","last_active":"2024-04-25T00:30:26.755166+0000","last_peered":"2024-04-25T00:30:26.755166+0000","last_clean":"2024-04-25T00:30:26.755166+0000","last_became_active":"2024-04-25T00:01:19.539775+0000","last_became_peered":"2024-04-25T00:01:19.539775+0000","last_unstale":"2024-04-25T00:30:26.755166+0000","last_undegraded":"2024-04-25T00:30:26.755166+0000","last_fullsized":"2024-04-25T00:30:26.755166+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3900,"log_dups_size":0,"ondisk_log_size":3900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:53:14.768520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21234,"num_write_kb":281892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"70'3829","reported_seq":3955,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.804122+0000","last_change":"2024-04-25T00:01:05.367001+0000","last_active":"2024-04-25T00:30:15.804122+0000","last_peered":"2024-04-25T00:30:15.804122+0000","last_clean":"2024-04-25T00:30:15.804122+0000","last_became_active":"2024-04-25T00:01:05.366264+0000","last_became_peered":"2024-04-25T00:01:05.366264+0000","last_unstale":"2024-04-25T00:30:15.804122+0000","last_undegraded":"2024-04-25T00:30:15.804122+0000","last_fullsized":"2024-04-25T00:30:15.804122+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3829,"log_dups_size":0,"ondisk_log_size":3829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19822,"num_write_kb":292385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.29","version":"74'4218","reported_seq":4338,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.734841+0000","last_change":"2024-04-25T00:01:19.497247+0000","last_active":"2024-04-25T00:30:26.734841+0000","last_peered":"2024-04-25T00:30:26.734841+0000","last_clean":"2024-04-25T00:30:26.734841+0000","last_became_active":"2024-04-25T00:01:19.496517+0000","last_became_peered":"2024-04-25T00:01:19.496517+0000","last_unstale":"2024-04-25T00:30:26.734841+0000","last_undegraded":"2024-04-25T00:30:26.734841+0000","last_fullsized":"2024-04-25T00:30:26.734841+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4218,"log_dups_size":0,"ondisk_log_size":4218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:30.129056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":23201,"num_write_kb":293725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2f","version":"70'5230","reported_seq":5357,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.813628+0000","last_change":"2024-04-25T00:01:05.379958+0000","last_active":"2024-04-25T00:30:15.813628+0000","last_peered":"2024-04-25T00:30:15.813628+0000","last_clean":"2024-04-25T00:30:15.813628+0000","last_became_active":"2024-04-25T00:01:05.379090+0000","last_became_peered":"2024-04-25T00:01:05.379090+0000","last_unstale":"2024-04-25T00:30:15.813628+0000","last_undegraded":"2024-04-25T00:30:15.813628+0000","last_fullsized":"2024-04-25T00:30:15.813628+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":5230,"log_dups_size":0,"ondisk_log_size":5230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:59:50.832958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":27866,"num_write_kb":365657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.36","version":"74'4384","reported_seq":4504,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.681328+0000","last_change":"2024-04-25T00:01:19.530176+0000","last_active":"2024-04-25T00:30:26.681328+0000","last_peered":"2024-04-25T00:30:26.681328+0000","last_clean":"2024-04-25T00:30:26.681328+0000","last_became_active":"2024-04-25T00:01:19.529410+0000","last_became_peered":"2024-04-25T00:01:19.529410+0000","last_unstale":"2024-04-25T00:30:26.681328+0000","last_undegraded":"2024-04-25T00:30:26.681328+0000","last_fullsized":"2024-04-25T00:30:26.681328+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4384,"log_dups_size":0,"ondisk_log_size":4384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:40:13.204880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":7,"num_write":24230,"num_write_kb":310193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.30","version":"69'4051","reported_seq":4177,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.803573+0000","last_change":"2024-04-25T00:01:05.370826+0000","last_active":"2024-04-25T00:30:15.803573+0000","last_peered":"2024-04-25T00:30:15.803573+0000","last_clean":"2024-04-25T00:30:15.803573+0000","last_became_active":"2024-04-25T00:01:05.369848+0000","last_became_peered":"2024-04-25T00:01:05.369848+0000","last_unstale":"2024-04-25T00:30:15.803573+0000","last_undegraded":"2024-04-25T00:30:15.803573+0000","last_fullsized":"2024-04-25T00:30:15.803573+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4051,"log_dups_size":0,"ondisk_log_size":4051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:23.870121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21793,"num_write_kb":288101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.37","version":"74'6689","reported_seq":6811,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.769600+0000","last_change":"2024-04-25T00:01:19.511625+0000","last_active":"2024-04-25T00:30:26.769600+0000","last_peered":"2024-04-25T00:30:26.769600+0000","last_clean":"2024-04-25T00:30:26.769600+0000","last_became_active":"2024-04-25T00:01:19.511349+0000","last_became_peered":"2024-04-25T00:01:19.511349+0000","last_unstale":"2024-04-25T00:30:26.769600+0000","last_undegraded":"2024-04-25T00:30:26.769600+0000","last_fullsized":"2024-04-25T00:30:26.769600+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":6689,"log_dups_size":0,"ondisk_log_size":6689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:02:57.373014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":8,"num_write":38367,"num_write_kb":445068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"70'3993","reported_seq":4119,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.795313+0000","last_change":"2024-04-25T00:01:05.390341+0000","last_active":"2024-04-25T00:30:15.795313+0000","last_peered":"2024-04-25T00:30:15.795313+0000","last_clean":"2024-04-25T00:30:15.795313+0000","last_became_active":"2024-04-25T00:01:05.389707+0000","last_became_peered":"2024-04-25T00:01:05.389707+0000","last_unstale":"2024-04-25T00:30:15.795313+0000","last_undegraded":"2024-04-25T00:30:15.795313+0000","last_fullsized":"2024-04-25T00:30:15.795313+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3993,"log_dups_size":0,"ondisk_log_size":3993,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:54:21.138947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21619,"num_write_kb":287362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.34","version":"74'3411","reported_seq":3532,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.695377+0000","last_change":"2024-04-25T00:01:19.513630+0000","last_active":"2024-04-25T00:30:26.695377+0000","last_peered":"2024-04-25T00:30:26.695377+0000","last_clean":"2024-04-25T00:30:26.695377+0000","last_became_active":"2024-04-25T00:01:19.512978+0000","last_became_peered":"2024-04-25T00:01:19.512978+0000","last_unstale":"2024-04-25T00:30:26.695377+0000","last_undegraded":"2024-04-25T00:30:26.695377+0000","last_fullsized":"2024-04-25T00:30:26.695377+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3411,"log_dups_size":0,"ondisk_log_size":3411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:57.751007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":34,"num_write":18670,"num_write_kb":234685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"70'3759","reported_seq":3886,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.809965+0000","last_change":"2024-04-25T00:01:05.378373+0000","last_active":"2024-04-25T00:30:15.809965+0000","last_peered":"2024-04-25T00:30:15.809965+0000","last_clean":"2024-04-25T00:30:15.809965+0000","last_became_active":"2024-04-25T00:01:05.376570+0000","last_became_peered":"2024-04-25T00:01:05.376570+0000","last_unstale":"2024-04-25T00:30:15.809965+0000","last_undegraded":"2024-04-25T00:30:15.809965+0000","last_fullsized":"2024-04-25T00:30:15.809965+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3759,"log_dups_size":0,"ondisk_log_size":3759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:30.228060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20571,"num_write_kb":279348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.35","version":"74'4078","reported_seq":4199,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.645015+0000","last_change":"2024-04-25T00:01:19.516716+0000","last_active":"2024-04-25T00:30:26.645015+0000","last_peered":"2024-04-25T00:30:26.645015+0000","last_clean":"2024-04-25T00:30:26.645015+0000","last_became_active":"2024-04-25T00:01:19.516142+0000","last_became_peered":"2024-04-25T00:01:19.516142+0000","last_unstale":"2024-04-25T00:30:26.645015+0000","last_undegraded":"2024-04-25T00:30:26.645015+0000","last_fullsized":"2024-04-25T00:30:26.645015+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4078,"log_dups_size":0,"ondisk_log_size":4078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:28:18.234546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":22241,"num_write_kb":273608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.33","version":"70'3191","reported_seq":3317,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.810448+0000","last_change":"2024-04-25T00:01:05.381011+0000","last_active":"2024-04-25T00:30:15.810448+0000","last_peered":"2024-04-25T00:30:15.810448+0000","last_clean":"2024-04-25T00:30:15.810448+0000","last_became_active":"2024-04-25T00:01:05.380236+0000","last_became_peered":"2024-04-25T00:01:05.380236+0000","last_unstale":"2024-04-25T00:30:15.810448+0000","last_undegraded":"2024-04-25T00:30:15.810448+0000","last_fullsized":"2024-04-25T00:30:15.810448+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3191,"log_dups_size":0,"ondisk_log_size":3191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:06:19.307378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17526,"num_write_kb":230308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"74'4055","reported_seq":4174,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.756767+0000","last_change":"2024-04-25T00:01:19.529023+0000","last_active":"2024-04-25T00:30:26.756767+0000","last_peered":"2024-04-25T00:30:26.756767+0000","last_clean":"2024-04-25T00:30:26.756767+0000","last_became_active":"2024-04-25T00:01:19.524609+0000","last_became_peered":"2024-04-25T00:01:19.524609+0000","last_unstale":"2024-04-25T00:30:26.756767+0000","last_undegraded":"2024-04-25T00:30:26.756767+0000","last_fullsized":"2024-04-25T00:30:26.756767+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4055,"log_dups_size":0,"ondisk_log_size":4055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:41:19.150476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":21801,"num_write_kb":276048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.34","version":"70'2711","reported_seq":2837,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.792324+0000","last_change":"2024-04-25T00:01:05.385853+0000","last_active":"2024-04-25T00:30:15.792324+0000","last_peered":"2024-04-25T00:30:15.792324+0000","last_clean":"2024-04-25T00:30:15.792324+0000","last_became_active":"2024-04-25T00:01:05.384997+0000","last_became_peered":"2024-04-25T00:01:05.384997+0000","last_unstale":"2024-04-25T00:30:15.792324+0000","last_undegraded":"2024-04-25T00:30:15.792324+0000","last_fullsized":"2024-04-25T00:30:15.792324+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":2711,"log_dups_size":0,"ondisk_log_size":2711,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:36:55.436661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14839,"num_write_kb":207133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.33","version":"74'3434","reported_seq":3556,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.729873+0000","last_change":"2024-04-25T00:01:19.516867+0000","last_active":"2024-04-25T00:30:26.729873+0000","last_peered":"2024-04-25T00:30:26.729873+0000","last_clean":"2024-04-25T00:30:26.729873+0000","last_became_active":"2024-04-25T00:01:19.515764+0000","last_became_peered":"2024-04-25T00:01:19.515764+0000","last_unstale":"2024-04-25T00:30:26.729873+0000","last_undegraded":"2024-04-25T00:30:26.729873+0000","last_fullsized":"2024-04-25T00:30:26.729873+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3434,"log_dups_size":0,"ondisk_log_size":3434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:53:14.768520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":5,"num_write":19347,"num_write_kb":241776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"69'3819","reported_seq":3946,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.790737+0000","last_change":"2024-04-25T00:01:05.389424+0000","last_active":"2024-04-25T00:30:15.790737+0000","last_peered":"2024-04-25T00:30:15.790737+0000","last_clean":"2024-04-25T00:30:15.790737+0000","last_became_active":"2024-04-25T00:01:05.388999+0000","last_became_peered":"2024-04-25T00:01:05.388999+0000","last_unstale":"2024-04-25T00:30:15.790737+0000","last_undegraded":"2024-04-25T00:30:15.790737+0000","last_fullsized":"2024-04-25T00:30:15.790737+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3819,"log_dups_size":0,"ondisk_log_size":3819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:31.456792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":19986,"num_write_kb":272748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.30","version":"74'4335","reported_seq":4453,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.723246+0000","last_change":"2024-04-25T00:01:19.519365+0000","last_active":"2024-04-25T00:30:26.723246+0000","last_peered":"2024-04-25T00:30:26.723246+0000","last_clean":"2024-04-25T00:30:26.723246+0000","last_became_active":"2024-04-25T00:01:19.519085+0000","last_became_peered":"2024-04-25T00:01:19.519085+0000","last_unstale":"2024-04-25T00:30:26.723246+0000","last_undegraded":"2024-04-25T00:30:26.723246+0000","last_fullsized":"2024-04-25T00:30:26.723246+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4335,"log_dups_size":0,"ondisk_log_size":4335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:05:09.219823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23507,"num_write_kb":286989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"69'4866","reported_seq":4992,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.790509+0000","last_change":"2024-04-25T00:01:05.368974+0000","last_active":"2024-04-25T00:30:15.790509+0000","last_peered":"2024-04-25T00:30:15.790509+0000","last_clean":"2024-04-25T00:30:15.790509+0000","last_became_active":"2024-04-25T00:01:05.367893+0000","last_became_peered":"2024-04-25T00:01:05.367893+0000","last_unstale":"2024-04-25T00:30:15.790509+0000","last_undegraded":"2024-04-25T00:30:15.790509+0000","last_fullsized":"2024-04-25T00:30:15.790509+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4866,"log_dups_size":0,"ondisk_log_size":4866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:30:23.557381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26504,"num_write_kb":327449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.31","version":"74'4017","reported_seq":4136,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.738931+0000","last_change":"2024-04-25T00:01:19.513514+0000","last_active":"2024-04-25T00:30:26.738931+0000","last_peered":"2024-04-25T00:30:26.738931+0000","last_clean":"2024-04-25T00:30:26.738931+0000","last_became_active":"2024-04-25T00:01:19.513184+0000","last_became_peered":"2024-04-25T00:01:19.513184+0000","last_unstale":"2024-04-25T00:30:26.738931+0000","last_undegraded":"2024-04-25T00:30:26.738931+0000","last_fullsized":"2024-04-25T00:30:26.738931+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4017,"log_dups_size":0,"ondisk_log_size":4017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:47.859626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":21984,"num_write_kb":288846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"70'5540","reported_seq":5666,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.944163+0000","last_change":"2024-04-25T00:01:05.362303+0000","last_active":"2024-04-25T00:30:15.944163+0000","last_peered":"2024-04-25T00:30:15.944163+0000","last_clean":"2024-04-25T00:30:15.944163+0000","last_became_active":"2024-04-25T00:01:05.361586+0000","last_became_peered":"2024-04-25T00:01:05.361586+0000","last_unstale":"2024-04-25T00:30:15.944163+0000","last_undegraded":"2024-04-25T00:30:15.944163+0000","last_fullsized":"2024-04-25T00:30:15.944163+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":5540,"log_dups_size":0,"ondisk_log_size":5540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:30.228060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30654,"num_write_kb":393416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3e","version":"74'5058","reported_seq":5180,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.759205+0000","last_change":"2024-04-25T00:01:19.515101+0000","last_active":"2024-04-25T00:30:26.759205+0000","last_peered":"2024-04-25T00:30:26.759205+0000","last_clean":"2024-04-25T00:30:26.759205+0000","last_became_active":"2024-04-25T00:01:19.513314+0000","last_became_peered":"2024-04-25T00:01:19.513314+0000","last_unstale":"2024-04-25T00:30:26.759205+0000","last_undegraded":"2024-04-25T00:30:26.759205+0000","last_fullsized":"2024-04-25T00:30:26.759205+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":5058,"log_dups_size":0,"ondisk_log_size":5058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:50.351599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":15,"num_write":27439,"num_write_kb":359791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.38","version":"70'4026","reported_seq":4151,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.793659+0000","last_change":"2024-04-25T00:01:05.390256+0000","last_active":"2024-04-25T00:30:15.793659+0000","last_peered":"2024-04-25T00:30:15.793659+0000","last_clean":"2024-04-25T00:30:15.793659+0000","last_became_active":"2024-04-25T00:01:05.389724+0000","last_became_peered":"2024-04-25T00:01:05.389724+0000","last_unstale":"2024-04-25T00:30:15.793659+0000","last_undegraded":"2024-04-25T00:30:15.793659+0000","last_fullsized":"2024-04-25T00:30:15.793659+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4026,"log_dups_size":0,"ondisk_log_size":4026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:09:18.315688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21352,"num_write_kb":290527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3f","version":"74'4365","reported_seq":4489,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.773739+0000","last_change":"2024-04-25T00:01:19.511493+0000","last_active":"2024-04-25T00:30:26.773739+0000","last_peered":"2024-04-25T00:30:26.773739+0000","last_clean":"2024-04-25T00:30:26.773739+0000","last_became_active":"2024-04-25T00:01:19.508049+0000","last_became_peered":"2024-04-25T00:01:19.508049+0000","last_unstale":"2024-04-25T00:30:26.773739+0000","last_undegraded":"2024-04-25T00:30:26.773739+0000","last_fullsized":"2024-04-25T00:30:26.773739+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4365,"log_dups_size":0,"ondisk_log_size":4365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:55:46.853565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":8,"num_write":24611,"num_write_kb":307761,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"70'3970","reported_seq":4095,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.790756+0000","last_change":"2024-04-25T00:01:05.374766+0000","last_active":"2024-04-25T00:30:15.790756+0000","last_peered":"2024-04-25T00:30:15.790756+0000","last_clean":"2024-04-25T00:30:15.790756+0000","last_became_active":"2024-04-25T00:01:05.369739+0000","last_became_peered":"2024-04-25T00:01:05.369739+0000","last_unstale":"2024-04-25T00:30:15.790756+0000","last_undegraded":"2024-04-25T00:30:15.790756+0000","last_fullsized":"2024-04-25T00:30:15.790756+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3970,"log_dups_size":0,"ondisk_log_size":3970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:09:18.315688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22008,"num_write_kb":289131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3c","version":"74'3966","reported_seq":4087,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.664683+0000","last_change":"2024-04-25T00:01:19.521456+0000","last_active":"2024-04-25T00:30:26.664683+0000","last_peered":"2024-04-25T00:30:26.664683+0000","last_clean":"2024-04-25T00:30:26.664683+0000","last_became_active":"2024-04-25T00:01:19.520787+0000","last_became_peered":"2024-04-25T00:01:19.520787+0000","last_unstale":"2024-04-25T00:30:26.664683+0000","last_undegraded":"2024-04-25T00:30:26.664683+0000","last_fullsized":"2024-04-25T00:30:26.664683+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3966,"log_dups_size":0,"ondisk_log_size":3966,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:25:38.981364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22683,"num_write_kb":271349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3a","version":"70'3294","reported_seq":3419,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.805979+0000","last_change":"2024-04-25T00:01:05.359253+0000","last_active":"2024-04-25T00:30:15.805979+0000","last_peered":"2024-04-25T00:30:15.805979+0000","last_clean":"2024-04-25T00:30:15.805979+0000","last_became_active":"2024-04-25T00:01:05.358477+0000","last_became_peered":"2024-04-25T00:01:05.358477+0000","last_unstale":"2024-04-25T00:30:15.805979+0000","last_undegraded":"2024-04-25T00:30:15.805979+0000","last_fullsized":"2024-04-25T00:30:15.805979+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3294,"log_dups_size":0,"ondisk_log_size":3294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:22.739214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17420,"num_write_kb":225566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3d","version":"74'5153","reported_seq":5274,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.749265+0000","last_change":"2024-04-25T00:01:19.509505+0000","last_active":"2024-04-25T00:30:26.749265+0000","last_peered":"2024-04-25T00:30:26.749265+0000","last_clean":"2024-04-25T00:30:26.749265+0000","last_became_active":"2024-04-25T00:01:19.507812+0000","last_became_peered":"2024-04-25T00:01:19.507812+0000","last_unstale":"2024-04-25T00:30:26.749265+0000","last_undegraded":"2024-04-25T00:30:26.749265+0000","last_fullsized":"2024-04-25T00:30:26.749265+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":5153,"log_dups_size":0,"ondisk_log_size":5153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:23.993729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":28174,"num_write_kb":326190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3b","version":"70'3663","reported_seq":3788,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.792919+0000","last_change":"2024-04-25T00:01:05.386068+0000","last_active":"2024-04-25T00:30:15.792919+0000","last_peered":"2024-04-25T00:30:15.792919+0000","last_clean":"2024-04-25T00:30:15.792919+0000","last_became_active":"2024-04-25T00:01:05.378650+0000","last_became_peered":"2024-04-25T00:01:05.378650+0000","last_unstale":"2024-04-25T00:30:15.792919+0000","last_undegraded":"2024-04-25T00:30:15.792919+0000","last_fullsized":"2024-04-25T00:30:15.792919+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3663,"log_dups_size":0,"ondisk_log_size":3663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:12:52.846250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18938,"num_write_kb":284951,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3a","version":"74'3833","reported_seq":3950,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.611583+0000","last_change":"2024-04-25T00:01:19.514843+0000","last_active":"2024-04-25T00:30:26.611583+0000","last_peered":"2024-04-25T00:30:26.611583+0000","last_clean":"2024-04-25T00:30:26.611583+0000","last_became_active":"2024-04-25T00:01:19.513956+0000","last_became_peered":"2024-04-25T00:01:19.513956+0000","last_unstale":"2024-04-25T00:30:26.611583+0000","last_undegraded":"2024-04-25T00:30:26.611583+0000","last_fullsized":"2024-04-25T00:30:26.611583+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3833,"log_dups_size":0,"ondisk_log_size":3833,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:38.005233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20679,"num_write_kb":271382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"69'3516","reported_seq":3645,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.808655+0000","last_change":"2024-04-25T00:01:05.359716+0000","last_active":"2024-04-25T00:30:15.808655+0000","last_peered":"2024-04-25T00:30:15.808655+0000","last_clean":"2024-04-25T00:30:15.808655+0000","last_became_active":"2024-04-25T00:01:05.358866+0000","last_became_peered":"2024-04-25T00:01:05.358866+0000","last_unstale":"2024-04-25T00:30:15.808655+0000","last_undegraded":"2024-04-25T00:30:15.808655+0000","last_fullsized":"2024-04-25T00:30:15.808655+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3516,"log_dups_size":0,"ondisk_log_size":3516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:50.489125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":19226,"num_write_kb":253281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3b","version":"74'4130","reported_seq":4248,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.756223+0000","last_change":"2024-04-25T00:01:19.499574+0000","last_active":"2024-04-25T00:30:26.756223+0000","last_peered":"2024-04-25T00:30:26.756223+0000","last_clean":"2024-04-25T00:30:26.756223+0000","last_became_active":"2024-04-25T00:01:19.498699+0000","last_became_peered":"2024-04-25T00:01:19.498699+0000","last_unstale":"2024-04-25T00:30:26.756223+0000","last_undegraded":"2024-04-25T00:30:26.756223+0000","last_fullsized":"2024-04-25T00:30:26.756223+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4130,"log_dups_size":0,"ondisk_log_size":4130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:47.859626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":3,"num_write":21749,"num_write_kb":288055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"70'3483","reported_seq":3611,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.798639+0000","last_change":"2024-04-25T00:01:05.371291+0000","last_active":"2024-04-25T00:30:15.798639+0000","last_peered":"2024-04-25T00:30:15.798639+0000","last_clean":"2024-04-25T00:30:15.798639+0000","last_became_active":"2024-04-25T00:01:05.370522+0000","last_became_peered":"2024-04-25T00:01:05.370522+0000","last_unstale":"2024-04-25T00:30:15.798639+0000","last_undegraded":"2024-04-25T00:30:15.798639+0000","last_fullsized":"2024-04-25T00:30:15.798639+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3483,"log_dups_size":0,"ondisk_log_size":3483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:11:27.044893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18326,"num_write_kb":261938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.38","version":"74'4120","reported_seq":4237,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.292105+0000","last_change":"2024-04-25T00:01:19.523798+0000","last_active":"2024-04-25T00:30:26.292105+0000","last_peered":"2024-04-25T00:30:26.292105+0000","last_clean":"2024-04-25T00:30:26.292105+0000","last_became_active":"2024-04-25T00:01:19.523450+0000","last_became_peered":"2024-04-25T00:01:19.523450+0000","last_unstale":"2024-04-25T00:30:26.292105+0000","last_undegraded":"2024-04-25T00:30:26.292105+0000","last_fullsized":"2024-04-25T00:30:26.292105+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4120,"log_dups_size":0,"ondisk_log_size":4120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:40:36.350832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22879,"num_write_kb":288519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3e","version":"70'4991","reported_seq":5119,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.801414+0000","last_change":"2024-04-25T00:01:05.368190+0000","last_active":"2024-04-25T00:30:15.801414+0000","last_peered":"2024-04-25T00:30:15.801414+0000","last_clean":"2024-04-25T00:30:15.801414+0000","last_became_active":"2024-04-25T00:01:05.362606+0000","last_became_peered":"2024-04-25T00:01:05.362606+0000","last_unstale":"2024-04-25T00:30:15.801414+0000","last_undegraded":"2024-04-25T00:30:15.801414+0000","last_fullsized":"2024-04-25T00:30:15.801414+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4991,"log_dups_size":0,"ondisk_log_size":4991,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:12:52.846250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26684,"num_write_kb":361051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.39","version":"74'4850","reported_seq":4967,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.763325+0000","last_change":"2024-04-25T00:01:19.505737+0000","last_active":"2024-04-25T00:30:26.763325+0000","last_peered":"2024-04-25T00:30:26.763325+0000","last_clean":"2024-04-25T00:30:26.763325+0000","last_became_active":"2024-04-25T00:01:19.504991+0000","last_became_peered":"2024-04-25T00:01:19.504991+0000","last_unstale":"2024-04-25T00:30:26.763325+0000","last_undegraded":"2024-04-25T00:30:26.763325+0000","last_fullsized":"2024-04-25T00:30:26.763325+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4850,"log_dups_size":0,"ondisk_log_size":4850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:47:19.526845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26612,"num_write_kb":323915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3f","version":"70'3815","reported_seq":3943,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.791882+0000","last_change":"2024-04-25T00:01:05.389596+0000","last_active":"2024-04-25T00:30:15.791882+0000","last_peered":"2024-04-25T00:30:15.791882+0000","last_clean":"2024-04-25T00:30:15.791882+0000","last_became_active":"2024-04-25T00:01:05.389127+0000","last_became_peered":"2024-04-25T00:01:05.389127+0000","last_unstale":"2024-04-25T00:30:15.791882+0000","last_undegraded":"2024-04-25T00:30:15.791882+0000","last_fullsized":"2024-04-25T00:30:15.791882+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3815,"log_dups_size":0,"ondisk_log_size":3815,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:09.858617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20941,"num_write_kb":280561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":105,"num_read_kb":408,"num_write":1543888,"num_write_kb":19306803,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":280555,"ondisk_log_size":280555,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":19554860,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1733,"num_read_kb":229197,"num_write":5129,"num_write_kb":21277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":39174144,"data_stored":39109720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4705,"ondisk_log_size":4705,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":3,"num_write":1412500,"num_write_kb":18776370,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":261221,"ondisk_log_size":261221,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":19117950,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1590,"num_read_kb":257618,"num_write":5060,"num_write_kb":20770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":38297600,"data_stored":38235900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4699,"ondisk_log_size":4699,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509801,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":674532,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":673470,"kb_avail":93697308,"statfs":{"total":96636764160,"available":95946043392,"internally_reserved":0,"allocated":1077248,"data_stored":617327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":689633739},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509801,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":843472,"kb_used_data":11784,"kb_used_omap":1,"kb_used_meta":831678,"kb_avail":93528368,"statfs":{"total":96636764160,"available":95773048832,"internally_reserved":0,"allocated":12066816,"data_stored":11609303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":851638726},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509800,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":870568,"kb_used_data":18016,"kb_used_omap":1,"kb_used_meta":852542,"kb_avail":93501272,"statfs":{"total":96636764160,"available":95745302528,"internally_reserved":0,"allocated":18448384,"data_stored":17985439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":873003461},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509801,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":922240,"kb_used_data":15544,"kb_used_omap":1,"kb_used_meta":906686,"kb_avail":93449600,"statfs":{"total":96636764160,"available":95692390400,"internally_reserved":0,"allocated":15917056,"data_stored":15447989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":928446919},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509802,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1076840,"kb_used_data":15968,"kb_used_omap":1,"kb_used_meta":1060862,"kb_avail":93295000,"statfs":{"total":96636764160,"available":95534080000,"internally_reserved":0,"allocated":16351232,"data_stored":15888287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1086323141},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509801,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":788556,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":787006,"kb_avail":93583284,"statfs":{"total":96636764160,"available":95829282816,"internally_reserved":0,"allocated":1576960,"data_stored":1119003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":805894597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509801,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1028004,"kb_used_data":3164,"kb_used_omap":1,"kb_used_meta":1024830,"kb_avail":93343836,"statfs":{"total":96636764160,"available":95584088064,"internally_reserved":0,"allocated":3239936,"data_stored":2779219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1049426373},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575210,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":962780,"kb_used_data":17364,"kb_used_omap":1,"kb_used_meta":945406,"kb_avail":93409060,"statfs":{"total":96636764160,"available":95650877440,"internally_reserved":0,"allocated":17780736,"data_stored":17307749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":968096197},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2260992,"data_stored":2250634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8396800,"data_stored":8389280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":10645504,"data_stored":10639768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8404992,"data_stored":8389952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4292608,"data_stored":4283082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":88810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6979584,"data_stored":6970512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4296704,"data_stored":4279726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8400896,"data_stored":8389302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12587008,"data_stored":12583494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":6897664,"data_stored":6886504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T00:30:42.117 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T00:30:42.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.270+0000 7f9f25603640 1 Processor -- start 2024-04-25T00:30:42.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.274+0000 7f9f25603640 1 -- start start 2024-04-25T00:30:42.274 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.274+0000 7f9f25603640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9f20150d20 0x7f9f200c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:42.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.274+0000 7f9f25603640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f9f200a94a0 0x7f9f20153ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:42.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.274+0000 7f9f25603640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f200c3910 con 0x7f9f200c2f50 2024-04-25T00:30:42.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.274+0000 7f9f25603640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f9f20154530 con 0x7f9f200a94a0 2024-04-25T00:30:42.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.274+0000 7f9f25603640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f9f200c16e0 con 0x7f9f20150d20 2024-04-25T00:30:42.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.274+0000 7f9f1effd640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9f20150d20 0x7f9f200c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:42.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.274+0000 7f9f1f7fe640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f9f200a94a0 0x7f9f20153ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:42.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.274+0000 7f9f1e7fc640 1 --1- >> v1:172.21.15.174:6789/0 conn(0x7f9f200c2f50 0x7f9f201538e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:37172/0 (socket says 172.21.15.116:37172) 2024-04-25T00:30:42.275 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.274+0000 7f9f1e7fc640 1 -- 172.21.15.116:0/4255173242 learned_addr learned my addr 172.21.15.116:0/4255173242 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:30:42.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.274+0000 7f9f1effd640 1 -- 172.21.15.116:0/4255173242 >> v2:172.21.15.174:3300/0 conn(0x7f9f200a94a0 msgr2=0x7f9f20153ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:42.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.274+0000 7f9f1effd640 1 --2- 172.21.15.116:0/4255173242 >> v2:172.21.15.174:3300/0 conn(0x7f9f200a94a0 0x7f9f20153ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:42.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.274+0000 7f9f1effd640 1 -- 172.21.15.116:0/4255173242 >> v1:172.21.15.174:6789/0 conn(0x7f9f200c2f50 legacy=0x7f9f201538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:42.276 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.274+0000 7f9f1effd640 1 -- 172.21.15.116:0/4255173242 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f200c1400 con 0x7f9f20150d20 2024-04-25T00:30:42.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.278+0000 7f9f1effd640 1 --2- 172.21.15.116:0/4255173242 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9f20150d20 0x7f9f200c2a10 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9f140097c0 tx=0x7f9f1401c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=565c0a590fe7e7e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:42.277 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.278+0000 7f9f1dffb640 1 -- 172.21.15.116:0/4255173242 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f140b1070 con 0x7f9f20150d20 2024-04-25T00:30:42.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.278+0000 7f9f1dffb640 1 -- 172.21.15.116:0/4255173242 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f140ac410 con 0x7f9f20150d20 2024-04-25T00:30:42.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.278+0000 7f9f1dffb640 1 -- 172.21.15.116:0/4255173242 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f140ac7e0 con 0x7f9f20150d20 2024-04-25T00:30:42.278 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.278+0000 7f9f25603640 1 -- 172.21.15.116:0/4255173242 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9f20150d20 msgr2=0x7f9f200c2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:42.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.278+0000 7f9f25603640 1 --2- 172.21.15.116:0/4255173242 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9f20150d20 0x7f9f200c2a10 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9f140097c0 tx=0x7f9f1401c4a0 comp rx=0 tx=0).stop 2024-04-25T00:30:42.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.278+0000 7f9f25603640 1 -- 172.21.15.116:0/4255173242 shutdown_connections 2024-04-25T00:30:42.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.278+0000 7f9f25603640 1 --2- 172.21.15.116:0/4255173242 >> v2:172.21.15.174:3300/0 conn(0x7f9f200a94a0 0x7f9f20153ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:42.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.278+0000 7f9f25603640 1 --2- 172.21.15.116:0/4255173242 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9f20150d20 0x7f9f200c2a10 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:42.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.278+0000 7f9f25603640 1 -- 172.21.15.116:0/4255173242 >> 172.21.15.116:0/4255173242 conn(0x7f9f200c3b00 msgr2=0x7f9f200c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:42.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.278+0000 7f9f25603640 1 -- 172.21.15.116:0/4255173242 shutdown_connections 2024-04-25T00:30:42.279 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.278+0000 7f9f25603640 1 -- 172.21.15.116:0/4255173242 wait complete. 2024-04-25T00:30:42.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.278+0000 7f9f25603640 1 Processor -- start 2024-04-25T00:30:42.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.278+0000 7f9f25603640 1 -- start start 2024-04-25T00:30:42.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.278+0000 7f9f25603640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9f200a94a0 0x7f9f201eeea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:42.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.278+0000 7f9f25603640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f200c2f50 0x7f9f2015c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:42.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.278+0000 7f9f25603640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9f20150d20 0x7f9f20070d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:42.281 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.278+0000 7f9f25603640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f9f201608b0 con 0x7f9f200c2f50 2024-04-25T00:30:42.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.278+0000 7f9f25603640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f9f20160730 con 0x7f9f20150d20 2024-04-25T00:30:42.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.278+0000 7f9f25603640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f9f20160a30 con 0x7f9f200a94a0 2024-04-25T00:30:42.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.278+0000 7f9f1e7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f200c2f50 0x7f9f2015c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:42.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.278+0000 7f9f1e7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f200c2f50 0x7f9f2015c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36484/0 (socket says 172.21.15.116:36484) 2024-04-25T00:30:42.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.278+0000 7f9f1e7fc640 1 -- 172.21.15.116:0/3161636065 learned_addr learned my addr 172.21.15.116:0/3161636065 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:42.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.278+0000 7f9f1f7fe640 1 --2- 172.21.15.116:0/3161636065 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9f20150d20 0x7f9f20070d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:42.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.278+0000 7f9f1effd640 1 --2- 172.21.15.116:0/3161636065 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9f200a94a0 0x7f9f201eeea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:42.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.278+0000 7f9f1e7fc640 1 -- 172.21.15.116:0/3161636065 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9f200a94a0 msgr2=0x7f9f201eeea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:42.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.278+0000 7f9f1e7fc640 1 --2- 172.21.15.116:0/3161636065 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9f200a94a0 0x7f9f201eeea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:42.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.278+0000 7f9f1e7fc640 1 -- 172.21.15.116:0/3161636065 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9f20150d20 msgr2=0x7f9f20070d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:42.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.278+0000 7f9f1e7fc640 1 --2- 172.21.15.116:0/3161636065 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9f20150d20 0x7f9f20070d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:42.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.278+0000 7f9f1e7fc640 1 -- 172.21.15.116:0/3161636065 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f201ee3e0 con 0x7f9f200c2f50 2024-04-25T00:30:42.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.278+0000 7f9f1effd640 1 --2- 172.21.15.116:0/3161636065 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9f200a94a0 0x7f9f201eeea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:30:42.282 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.278+0000 7f9f1e7fc640 1 --2- 172.21.15.116:0/3161636065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f200c2f50 0x7f9f2015c550 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f9f1000c450 tx=0x7f9f1000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:42.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.282+0000 7f9f077fe640 1 -- 172.21.15.116:0/3161636065 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f10017070 con 0x7f9f200c2f50 2024-04-25T00:30:42.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.282+0000 7f9f077fe640 1 -- 172.21.15.116:0/3161636065 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f10005cf0 con 0x7f9f200c2f50 2024-04-25T00:30:42.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.282+0000 7f9f077fe640 1 -- 172.21.15.116:0/3161636065 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f10007290 con 0x7f9f200c2f50 2024-04-25T00:30:42.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.282+0000 7f9f25603640 1 -- 172.21.15.116:0/3161636065 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f201418a0 con 0x7f9f200c2f50 2024-04-25T00:30:42.283 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.282+0000 7f9f25603640 1 -- 172.21.15.116:0/3161636065 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f201ee6a0 con 0x7f9f200c2f50 2024-04-25T00:30:42.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.282+0000 7f9f25603640 1 -- 172.21.15.116:0/3161636065 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f20141dc0 con 0x7f9f200c2f50 2024-04-25T00:30:42.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.282+0000 7f9f077fe640 1 -- 172.21.15.116:0/3161636065 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f9f1001a070 con 0x7f9f200c2f50 2024-04-25T00:30:42.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.282+0000 7f9f077fe640 1 --2- 172.21.15.116:0/3161636065 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9f0c0678f0 0x7f9f0c069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:42.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.282+0000 7f9f077fe640 1 -- 172.21.15.116:0/3161636065 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7f9f10086320 con 0x7f9f200c2f50 2024-04-25T00:30:42.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.282+0000 7f9f1effd640 1 --2- 172.21.15.116:0/3161636065 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9f0c0678f0 0x7f9f0c069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:42.284 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.282+0000 7f9f1effd640 1 --2- 172.21.15.116:0/3161636065 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9f0c0678f0 0x7f9f0c069db0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f9f140ae000 tx=0x7f9f14008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:42.288 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.286+0000 7f9f077fe640 1 -- 172.21.15.116:0/3161636065 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f9f10055ae0 con 0x7f9f200c2f50 2024-04-25T00:30:42.448 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.446+0000 7f9f25603640 1 -- 172.21.15.116:0/3161636065 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9f20068350 con 0x7f9f200c2f50 2024-04-25T00:30:42.452 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.450+0000 7f9f077fe640 1 -- 172.21.15.116:0/3161636065 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v74) ==== 74+0+26655 (secure 0 0 0) 0x7f9f100553e0 con 0x7f9f200c2f50 2024-04-25T00:30:42.452 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:30:42.453 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":74,"fsid":"6d6a1aa7-0478-4c83-8057-05917e81f046","created":"2024-04-25T00:00:40.682493+0000","modified":"2024-04-25T00:30:15.781714+0000","last_up_change":"2024-04-25T00:00:51.822174+0000","last_in_change":"2024-04-25T00:00:42.980473+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T00:00:53.324376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-04-25T00:01:03.197326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-04-25T00:01:03.728618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-04-25T00:01:17.125210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-04-25T00:01:17.821873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8b6ddcb2-1ad9-4167-961c-bd6936bb9060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6809","nonce":3751765682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6811","nonce":3751765682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6815","nonce":3751765682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6813","nonce":3751765682}]},"public_addr":"172.21.15.116:6809/3751765682","cluster_addr":"172.21.15.116:6811/3751765682","heartbeat_back_addr":"172.21.15.116:6815/3751765682","heartbeat_front_addr":"172.21.15.116:6813/3751765682","state":["exists","up"]},{"osd":1,"uuid":"a6431768-994f-41da-8853-886f5687fb91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6801","nonce":377879124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6803","nonce":377879124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6807","nonce":377879124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6805","nonce":377879124}]},"public_addr":"172.21.15.116:6801/377879124","cluster_addr":"172.21.15.116:6803/377879124","heartbeat_back_addr":"172.21.15.116:6807/377879124","heartbeat_front_addr":"172.21.15.116:6805/377879124","state":["exists","up"]},{"osd":2,"uuid":"4f69bd40-5a23-45ff-8693-e1de75e1aa9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6817","nonce":1099282267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6819","nonce":1099282267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6823","nonce":1099282267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6821","nonce":1099282267}]},"public_addr":"172.21.15.116:6817/1099282267","cluster_addr":"172.21.15.116:6819/1099282267","heartbeat_back_addr":"172.21.15.116:6823/1099282267","heartbeat_front_addr":"172.21.15.116:6821/1099282267","state":["exists","up"]},{"osd":3,"uuid":"6a0d5344-8b6f-40d3-918d-0f0a4be688e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6825","nonce":1179942661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6827","nonce":1179942661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6831","nonce":1179942661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6829","nonce":1179942661}]},"public_addr":"172.21.15.116:6825/1179942661","cluster_addr":"172.21.15.116:6827/1179942661","heartbeat_back_addr":"172.21.15.116:6831/1179942661","heartbeat_front_addr":"172.21.15.116:6829/1179942661","state":["exists","up"]},{"osd":4,"uuid":"132e92e4-b825-441e-965a-ca00533fa86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6801","nonce":3132444330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6803","nonce":3132444330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6807","nonce":3132444330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6805","nonce":3132444330}]},"public_addr":"172.21.15.174:6801/3132444330","cluster_addr":"172.21.15.174:6803/3132444330","heartbeat_back_addr":"172.21.15.174:6807/3132444330","heartbeat_front_addr":"172.21.15.174:6805/3132444330","state":["exists","up"]},{"osd":5,"uuid":"a2eeb5b6-1b31-43cc-8306-a0adf3ee15bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6809","nonce":881817164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6811","nonce":881817164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6815","nonce":881817164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6813","nonce":881817164}]},"public_addr":"172.21.15.174:6809/881817164","cluster_addr":"172.21.15.174:6811/881817164","heartbeat_back_addr":"172.21.15.174:6815/881817164","heartbeat_front_addr":"172.21.15.174:6813/881817164","state":["exists","up"]},{"osd":6,"uuid":"588e873a-c841-4ba7-90f2-8956c87a9983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6825","nonce":117574251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6827","nonce":117574251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6831","nonce":117574251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6829","nonce":117574251}]},"public_addr":"172.21.15.174:6825/117574251","cluster_addr":"172.21.15.174:6827/117574251","heartbeat_back_addr":"172.21.15.174:6831/117574251","heartbeat_front_addr":"172.21.15.174:6829/117574251","state":["exists","up"]},{"osd":7,"uuid":"3b90f9b0-8a0e-4883-b088-ca1d4012870c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6817","nonce":2543899196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6819","nonce":2543899196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6823","nonce":2543899196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6821","nonce":2543899196}]},"public_addr":"172.21.15.174:6817/2543899196","cluster_addr":"172.21.15.174:6819/2543899196","heartbeat_back_addr":"172.21.15.174:6823/2543899196","heartbeat_front_addr":"172.21.15.174:6821/2543899196","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.293769+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.310034+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.404865+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.549658+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.192174+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.297266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.493649+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.452309+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2c","mappings":[{"from":0,"to":7}]},{"pgid":"5.3","mappings":[{"from":4,"to":5}]},{"pgid":"5.5","mappings":[{"from":4,"to":5}]},{"pgid":"5.14","mappings":[{"from":4,"to":5}]},{"pgid":"5.19","mappings":[{"from":4,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:6835/3499878473":"2024-04-26T00:30:15.781624+0000","172.21.15.116:6834/3499878473":"2024-04-26T00:30:15.781624+0000","172.21.15.174:6836/2915274073":"2024-04-26T00:29:50.776462+0000","172.21.15.116:6833/3891103804":"2024-04-26T00:28:50.767503+0000","172.21.15.116:6834/2371126056":"2024-04-26T00:27:35.754759+0000","172.21.15.174:6837/349772087":"2024-04-26T00:27:15.748274+0000","172.21.15.174:6836/349772087":"2024-04-26T00:27:15.748274+0000","172.21.15.116:6833/3693049068":"2024-04-26T00:26:45.739776+0000","172.21.15.116:6832/3693049068":"2024-04-26T00:26:45.739776+0000","172.21.15.174:6837/4181466530":"2024-04-26T00:26:15.734586+0000","172.21.15.174:6836/4181466530":"2024-04-26T00:26:15.734586+0000","172.21.15.174:6836/492366509":"2024-04-26T00:24:15.701887+0000","172.21.15.116:6832/2998042514":"2024-04-26T00:23:30.691827+0000","172.21.15.116:6833/663910035":"2024-04-26T00:22:35.677323+0000","172.21.15.116:6832/663910035":"2024-04-26T00:22:35.677323+0000","172.21.15.174:6837/321466844":"2024-04-26T00:22:10.671673+0000","172.21.15.116:6835/980221971":"2024-04-26T00:21:40.665115+0000","172.21.15.174:6837/991183935":"2024-04-26T00:20:50.654151+0000","172.21.15.174:6836/991183935":"2024-04-26T00:20:50.654151+0000","172.21.15.174:6836/321466844":"2024-04-26T00:22:10.671673+0000","172.21.15.116:6834/1358527848":"2024-04-26T00:11:00.511032+0000","172.21.15.116:6832/3841609112":"2024-04-26T00:07:00.448498+0000","172.21.15.116:6835/1670061091":"2024-04-26T00:09:30.490485+0000","172.21.15.116:6834/1971675962":"2024-04-26T00:25:25.721965+0000","172.21.15.116:6835/1233829678":"2024-04-26T00:07:30.454393+0000","172.21.15.116:6835/2371126056":"2024-04-26T00:27:35.754759+0000","172.21.15.174:6834/2883220846":"2024-04-26T00:11:50.521053+0000","172.21.15.116:6834/1233829678":"2024-04-26T00:07:30.454393+0000","172.21.15.174:6837/3180024481":"2024-04-26T00:15:25.567768+0000","172.21.15.116:6835/879531976":"2024-04-26T00:20:10.642563+0000","172.21.15.116:6833/174781672":"2024-04-26T00:02:40.399073+0000","172.21.15.174:6835/4215220222":"2024-04-26T00:15:40.572498+0000","172.21.15.116:6834/221971311":"2024-04-26T00:22:48.898075+0000","172.21.15.174:6836/1281466397":"2024-04-26T00:18:20.613347+0000","172.21.15.116:6832/3891103804":"2024-04-26T00:28:50.767503+0000","172.21.15.116:6835/1358527848":"2024-04-26T00:11:00.511032+0000","172.21.15.116:6835/221971311":"2024-04-26T00:22:48.898075+0000","172.21.15.116:6832/3006671671":"2024-04-26T00:09:30.490406+0000","172.21.15.116:6834/980221971":"2024-04-26T00:21:40.665115+0000","172.21.15.174:6836/1762356055":"2024-04-26T00:03:36.597345+0000","172.21.15.116:6833/3006671671":"2024-04-26T00:09:30.490406+0000","172.21.15.116:6834/1459177921":"2024-04-26T00:03:30.406778+0000","172.21.15.116:6834/462190523":"2024-04-26T00:14:55.560782+0000","172.21.15.116:6834/879531976":"2024-04-26T00:20:10.642563+0000","172.21.15.116:6832/1459177921":"2024-04-26T00:03:30.406778+0000","172.21.15.174:6837/1159002662":"2024-04-26T00:10:20.502218+0000","172.21.15.174:6837/1762356055":"2024-04-26T00:03:36.597345+0000","172.21.15.174:6834/4215220222":"2024-04-26T00:15:40.572498+0000","172.21.15.116:6835/1971675962":"2024-04-26T00:25:25.721965+0000","172.21.15.174:6836/3081407040":"2024-04-26T00:17:25.598516+0000","172.21.15.116:6834/1599708526":"2024-04-26T00:12:20.526742+0000","172.21.15.174:6836/2276529656":"2024-04-26T00:19:05.629048+0000","172.21.15.174:6836/1159002662":"2024-04-26T00:10:20.502218+0000","172.21.15.116:6833/2998042514":"2024-04-26T00:23:30.691827+0000","172.21.15.174:6837/3081407040":"2024-04-26T00:17:25.598516+0000","172.21.15.174:6835/2883220846":"2024-04-26T00:11:50.521053+0000","172.21.15.174:6837/492366509":"2024-04-26T00:24:15.701887+0000","172.21.15.116:6835/2004876235":"2024-04-26T00:04:45.421611+0000","172.21.15.116:6835/174781672":"2024-04-26T00:02:40.399073+0000","172.21.15.116:6833/3633657268":"2024-04-26T00:25:40.724634+0000","172.21.15.116:6832/365773583":"2024-04-26T00:14:58.857970+0000","172.21.15.174:6836/1889314958":"2024-04-26T00:06:30.441408+0000","172.21.15.116:6833/3705654249":"2024-04-26T00:12:45.533219+0000","172.21.15.116:6832/3705654249":"2024-04-26T00:12:45.533219+0000","172.21.15.116:6834/1670061091":"2024-04-26T00:09:30.490485+0000","172.21.15.116:6835/1599708526":"2024-04-26T00:12:20.526742+0000","172.21.15.116:6832/3633657268":"2024-04-26T00:25:40.724634+0000","172.21.15.174:6836/3583791822":"2024-04-26T00:08:00.460587+0000","172.21.15.174:6837/3583791822":"2024-04-26T00:08:00.460587+0000","172.21.15.174:6836/3398690896":"2024-04-26T00:13:30.543729+0000","172.21.15.174:6837/3398690896":"2024-04-26T00:13:30.543729+0000","172.21.15.116:6835/462190523":"2024-04-26T00:14:55.560782+0000","172.21.15.116:6834/3336339376":"2024-04-26T00:18:35.616887+0000","172.21.15.174:6836/3180024481":"2024-04-26T00:15:25.567768+0000","172.21.15.116:6833/365773583":"2024-04-26T00:14:58.857970+0000","172.21.15.116:6835/2012470875":"2024-04-26T00:16:35.584511+0000","172.21.15.116:6833/2004876235":"2024-04-26T00:04:45.421611+0000","172.21.15.174:6837/1281466397":"2024-04-26T00:18:20.613347+0000","172.21.15.116:6832/2340581953":"2024-04-26T00:16:45.587651+0000","172.21.15.174:6837/2915274073":"2024-04-26T00:29:50.776462+0000","172.21.15.116:6834/1649769369":"2024-04-26T00:04:35.418892+0000","172.21.15.116:6832/1649769369":"2024-04-26T00:04:35.418892+0000","172.21.15.116:6833/2340581953":"2024-04-26T00:16:45.587651+0000","172.21.15.174:6837/1889314958":"2024-04-26T00:06:30.441408+0000","172.21.15.116:6832/3645553116":"2024-04-26T00:17:35.601559+0000","172.21.15.116:6833/3645553116":"2024-04-26T00:17:35.601559+0000","172.21.15.116:6833/3841609112":"2024-04-26T00:07:00.448498+0000","172.21.15.116:6834/2012470875":"2024-04-26T00:16:35.584511+0000","172.21.15.116:6835/3336339376":"2024-04-26T00:18:35.616887+0000","172.21.15.174:6837/2276529656":"2024-04-26T00:19:05.629048+0000","172.21.15.116:6832/1655959433":"2024-04-26T00:19:30.633957+0000","172.21.15.116:6833/1655959433":"2024-04-26T00:19:30.633957+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:30:42.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.454+0000 7f9f25603640 1 -- 172.21.15.116:0/3161636065 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9f0c0678f0 msgr2=0x7f9f0c069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:42.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.454+0000 7f9f25603640 1 --2- 172.21.15.116:0/3161636065 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9f0c0678f0 0x7f9f0c069db0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f9f140ae000 tx=0x7f9f14008040 comp rx=0 tx=0).stop 2024-04-25T00:30:42.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.454+0000 7f9f25603640 1 -- 172.21.15.116:0/3161636065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f200c2f50 msgr2=0x7f9f2015c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:42.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.454+0000 7f9f25603640 1 --2- 172.21.15.116:0/3161636065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f200c2f50 0x7f9f2015c550 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f9f1000c450 tx=0x7f9f1000c910 comp rx=0 tx=0).stop 2024-04-25T00:30:42.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.454+0000 7f9f25603640 1 -- 172.21.15.116:0/3161636065 shutdown_connections 2024-04-25T00:30:42.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.454+0000 7f9f25603640 1 --2- 172.21.15.116:0/3161636065 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f9f0c0678f0 0x7f9f0c069db0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:42.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.454+0000 7f9f25603640 1 --2- 172.21.15.116:0/3161636065 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f9f20150d20 0x7f9f20070d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:42.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.454+0000 7f9f25603640 1 --2- 172.21.15.116:0/3161636065 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f9f200c2f50 0x7f9f2015c550 unknown :-1 s=CLOSED pgs=1805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:42.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.454+0000 7f9f25603640 1 --2- 172.21.15.116:0/3161636065 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f9f200a94a0 0x7f9f201eeea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:42.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.454+0000 7f9f25603640 1 -- 172.21.15.116:0/3161636065 >> 172.21.15.116:0/3161636065 conn(0x7f9f200c3b00 msgr2=0x7f9f200acd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:42.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.454+0000 7f9f25603640 1 -- 172.21.15.116:0/3161636065 shutdown_connections 2024-04-25T00:30:42.455 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:42.454+0000 7f9f25603640 1 -- 172.21.15.116:0/3161636065 wait complete. 2024-04-25T00:30:42.613 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:30:42.615 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:30:43.472 INFO:tasks.ceph:Scrubbing osd.0 2024-04-25T00:30:43.472 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-25T00:30:43.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.606+0000 7f848427f640 1 Processor -- start 2024-04-25T00:30:43.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.606+0000 7f848427f640 1 -- start start 2024-04-25T00:30:43.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.606+0000 7f848427f640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f847c14a020 0x7f847c0c24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:43.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.606+0000 7f848427f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f847c0a94a0 0x7f847c0a9930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:43.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.606+0000 7f848427f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f847c0ab2f0 con 0x7f847c153f10 2024-04-25T00:30:43.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.606+0000 7f848427f640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f847c14a400 con 0x7f847c14a020 2024-04-25T00:30:43.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.606+0000 7f848427f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f847c0ac480 con 0x7f847c0a94a0 2024-04-25T00:30:43.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.606+0000 7f848181a640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f847c153f10 0x7f847c0abd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57430/0 (socket says 172.21.15.116:57430) 2024-04-25T00:30:43.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.606+0000 7f848181a640 1 -- 172.21.15.116:0/2828153187 learned_addr learned my addr 172.21.15.116:0/2828153187 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:30:43.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.606+0000 7f848201b640 1 --2- 172.21.15.116:0/2828153187 >> v2:172.21.15.174:3300/0 conn(0x7f847c14a020 0x7f847c0c24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:43.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.606+0000 7f848281c640 1 --2- 172.21.15.116:0/2828153187 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f847c0a94a0 0x7f847c0a9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:43.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.606+0000 7f8481019640 1 -- 172.21.15.116:0/2828153187 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3082553122 0 0) 0x7f846c001120 con 0x7f847c153f10 2024-04-25T00:30:43.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.610+0000 7f8481019640 1 -- 172.21.15.116:0/2828153187 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8468003530 con 0x7f847c153f10 2024-04-25T00:30:43.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.610+0000 7f8481019640 1 -- 172.21.15.116:0/2828153187 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1863328181 0 0) 0x7f8468003530 con 0x7f847c153f10 2024-04-25T00:30:43.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.610+0000 7f8481019640 1 -- 172.21.15.116:0/2828153187 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f847c0a94a0 msgr2=0x7f847c0a9930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:43.611 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.610+0000 7f8481019640 1 --2- 172.21.15.116:0/2828153187 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f847c0a94a0 0x7f847c0a9930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:43.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.610+0000 7f8481019640 1 -- 172.21.15.116:0/2828153187 >> v2:172.21.15.174:3300/0 conn(0x7f847c14a020 msgr2=0x7f847c0c24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:43.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.610+0000 7f8481019640 1 --2- 172.21.15.116:0/2828153187 >> v2:172.21.15.174:3300/0 conn(0x7f847c14a020 0x7f847c0c24c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:43.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.610+0000 7f8481019640 1 -- 172.21.15.116:0/2828153187 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f847c154370 con 0x7f847c153f10 2024-04-25T00:30:43.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.610+0000 7f8481019640 1 -- 172.21.15.116:0/2828153187 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f846c001850 con 0x7f847c153f10 2024-04-25T00:30:43.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.610+0000 7f8481019640 1 -- 172.21.15.116:0/2828153187 >> v1:172.21.15.116:6789/0 conn(0x7f847c153f10 legacy=0x7f847c0abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:43.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.610+0000 7f8481019640 1 --2- 172.21.15.116:0/2828153187 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f84680039c0 0x7f8468003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:43.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.610+0000 7f8481019640 1 --2- 172.21.15.116:0/2828153187 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8468004810 0x7f8468004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:43.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.610+0000 7f8481019640 1 --2- 172.21.15.116:0/2828153187 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f847c0a94a0 0x7f84680051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:43.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.610+0000 7f8481019640 1 -- 172.21.15.116:0/2828153187 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f847c14a400 con 0x7f8468004810 2024-04-25T00:30:43.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.610+0000 7f8481019640 1 -- 172.21.15.116:0/2828153187 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f847c0ac480 con 0x7f84680039c0 2024-04-25T00:30:43.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.610+0000 7f8481019640 1 -- 172.21.15.116:0/2828153187 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f846800b9d0 con 0x7f847c0a94a0 2024-04-25T00:30:43.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.610+0000 7f848181a640 1 --2- 172.21.15.116:0/2828153187 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8468004810 0x7f8468004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:43.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.610+0000 7f848201b640 1 --2- 172.21.15.116:0/2828153187 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f84680039c0 0x7f8468003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:43.612 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.610+0000 7f848281c640 1 --2- 172.21.15.116:0/2828153187 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f847c0a94a0 0x7f84680051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:43.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.610+0000 7f848181a640 1 -- 172.21.15.116:0/2828153187 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f847c0a94a0 msgr2=0x7f84680051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:43.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.610+0000 7f848181a640 1 --2- 172.21.15.116:0/2828153187 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f847c0a94a0 0x7f84680051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:43.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.610+0000 7f848181a640 1 -- 172.21.15.116:0/2828153187 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f84680039c0 msgr2=0x7f8468003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:43.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.610+0000 7f848181a640 1 --2- 172.21.15.116:0/2828153187 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f84680039c0 0x7f8468003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:43.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.610+0000 7f848181a640 1 -- 172.21.15.116:0/2828153187 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8468003530 con 0x7f8468004810 2024-04-25T00:30:43.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.610+0000 7f848201b640 1 --2- 172.21.15.116:0/2828153187 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f84680039c0 0x7f8468003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:30:43.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.610+0000 7f848181a640 1 --2- 172.21.15.116:0/2828153187 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8468004810 0x7f8468004c90 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7f846c00c490 tx=0x7f846c01f400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcde3568cd92fa46 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:43.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.610+0000 7f8481019640 1 -- 172.21.15.116:0/2828153187 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f846c00c0e0 con 0x7f8468004810 2024-04-25T00:30:43.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.610+0000 7f8481019640 1 -- 172.21.15.116:0/2828153187 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f846c005ae0 con 0x7f8468004810 2024-04-25T00:30:43.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.610+0000 7f8481019640 1 -- 172.21.15.116:0/2828153187 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f846c007070 con 0x7f8468004810 2024-04-25T00:30:43.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.614+0000 7f848427f640 1 -- 172.21.15.116:0/2828153187 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8468004810 msgr2=0x7f8468004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:43.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.614+0000 7f848427f640 1 --2- 172.21.15.116:0/2828153187 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8468004810 0x7f8468004c90 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7f846c00c490 tx=0x7f846c01f400 comp rx=0 tx=0).stop 2024-04-25T00:30:43.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.614+0000 7f848281c640 1 -- 172.21.15.116:0/2828153187 reap_dead start 2024-04-25T00:30:43.614 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.614+0000 7f848427f640 1 -- 172.21.15.116:0/2828153187 shutdown_connections 2024-04-25T00:30:43.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.614+0000 7f848427f640 1 -- 172.21.15.116:0/2828153187 >> 172.21.15.116:0/2828153187 conn(0x7f847c06b810 msgr2=0x7f847c0c6f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:43.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.614+0000 7f848427f640 1 -- 172.21.15.116:0/2828153187 shutdown_connections 2024-04-25T00:30:43.615 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.614+0000 7f848427f640 1 -- 172.21.15.116:0/2828153187 wait complete. 2024-04-25T00:30:43.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.614+0000 7f848427f640 1 Processor -- start 2024-04-25T00:30:43.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.614+0000 7f848427f640 1 -- start start 2024-04-25T00:30:43.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.614+0000 7f848427f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8468004810 0x7f847c1e9270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:43.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.614+0000 7f848427f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f847c157330 0x7f847c154950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:43.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.614+0000 7f848427f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f847c1ec7e0 0x7f847c1ecc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:43.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.614+0000 7f848427f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f847c0ab570 con 0x7f847c1ec7e0 2024-04-25T00:30:43.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.614+0000 7f848427f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f847c0c1da0 con 0x7f847c157330 2024-04-25T00:30:43.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.614+0000 7f848427f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f847c0ab850 con 0x7f8468004810 2024-04-25T00:30:43.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.614+0000 7f848281c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f847c1ec7e0 0x7f847c1ecc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:43.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.614+0000 7f848281c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f847c1ec7e0 0x7f847c1ecc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36510/0 (socket says 172.21.15.116:36510) 2024-04-25T00:30:43.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.614+0000 7f848281c640 1 -- 172.21.15.116:0/3893879824 learned_addr learned my addr 172.21.15.116:0/3893879824 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:43.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.614+0000 7f848201b640 1 --2- 172.21.15.116:0/3893879824 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8468004810 0x7f847c1e9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:43.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.614+0000 7f848181a640 1 --2- 172.21.15.116:0/3893879824 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f847c157330 0x7f847c154950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:43.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.614+0000 7f848281c640 1 -- 172.21.15.116:0/3893879824 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8468004810 msgr2=0x7f847c1e9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:43.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.614+0000 7f848281c640 1 --2- 172.21.15.116:0/3893879824 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8468004810 0x7f847c1e9270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:43.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.614+0000 7f848281c640 1 -- 172.21.15.116:0/3893879824 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f847c157330 msgr2=0x7f847c154950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:43.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.614+0000 7f848281c640 1 --2- 172.21.15.116:0/3893879824 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f847c157330 0x7f847c154950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:43.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.614+0000 7f848281c640 1 -- 172.21.15.116:0/3893879824 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f847c156c20 con 0x7f847c1ec7e0 2024-04-25T00:30:43.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.614+0000 7f848281c640 1 --2- 172.21.15.116:0/3893879824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f847c1ec7e0 0x7f847c1ecc60 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f847c0c7a50 tx=0x7f84740115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:43.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.614+0000 7f8472ffd640 1 -- 172.21.15.116:0/3893879824 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8474011db0 con 0x7f847c1ec7e0 2024-04-25T00:30:43.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.614+0000 7f8472ffd640 1 -- 172.21.15.116:0/3893879824 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f847401b070 con 0x7f847c1ec7e0 2024-04-25T00:30:43.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.614+0000 7f848427f640 1 -- 172.21.15.116:0/3893879824 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f847c152a60 con 0x7f847c1ec7e0 2024-04-25T00:30:43.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.614+0000 7f8472ffd640 1 -- 172.21.15.116:0/3893879824 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84740039a0 con 0x7f847c1ec7e0 2024-04-25T00:30:43.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.614+0000 7f848427f640 1 -- 172.21.15.116:0/3893879824 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f847c1518c0 con 0x7f847c1ec7e0 2024-04-25T00:30:43.618 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.618+0000 7f848427f640 1 -- 172.21.15.116:0/3893879824 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f847c068350 con 0x7f847c1ec7e0 2024-04-25T00:30:43.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.618+0000 7f8472ffd640 1 -- 172.21.15.116:0/3893879824 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8474003b40 con 0x7f847c1ec7e0 2024-04-25T00:30:43.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.618+0000 7f8472ffd640 1 --2- 172.21.15.116:0/3893879824 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f84600678f0 0x7f8460069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:43.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.618+0000 7f8472ffd640 1 -- 172.21.15.116:0/3893879824 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7f8474087690 con 0x7f847c1ec7e0 2024-04-25T00:30:43.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.618+0000 7f8472ffd640 1 --2- 172.21.15.116:0/3893879824 >> [v2:172.21.15.116:6808/3751765682,v1:172.21.15.116:6809/3751765682] conn(0x7f84600746e0 0x7f8460076b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:43.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.618+0000 7f8472ffd640 1 -- 172.21.15.116:0/3893879824 --> [v2:172.21.15.116:6808/3751765682,v1:172.21.15.116:6809/3751765682] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8460077250 con 0x7f84600746e0 2024-04-25T00:30:43.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.618+0000 7f8472ffd640 1 -- 172.21.15.116:0/3893879824 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=74) ==== 24+0+0 (secure 0 0 0) 0x7f847408e050 con 0x7f847c1ec7e0 2024-04-25T00:30:43.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.618+0000 7f848201b640 1 --2- 172.21.15.116:0/3893879824 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f84600678f0 0x7f8460069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:43.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.618+0000 7f848181a640 1 --2- 172.21.15.116:0/3893879824 >> [v2:172.21.15.116:6808/3751765682,v1:172.21.15.116:6809/3751765682] conn(0x7f84600746e0 0x7f8460076b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:43.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.618+0000 7f848181a640 1 --2- 172.21.15.116:0/3893879824 >> [v2:172.21.15.116:6808/3751765682,v1:172.21.15.116:6809/3751765682] conn(0x7f84600746e0 0x7f8460076b80 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:43.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.618+0000 7f848201b640 1 --2- 172.21.15.116:0/3893879824 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f84600678f0 0x7f8460069db0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f847c1e0c30 tx=0x7f8478008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:43.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.618+0000 7f8472ffd640 1 -- 172.21.15.116:0/3893879824 <== osd.0 v2:172.21.15.116:6808/3751765682 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8460077250 con 0x7f84600746e0 2024-04-25T00:30:43.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.630+0000 7f848427f640 1 -- 172.21.15.116:0/3893879824 --> [v2:172.21.15.116:6808/3751765682,v1:172.21.15.116:6809/3751765682] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f847c0c7a80 con 0x7f84600746e0 2024-04-25T00:30:43.633 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.630+0000 7f8472ffd640 1 -- 172.21.15.116:0/3893879824 <== osd.0 v2:172.21.15.116:6808/3751765682 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f847c0c7a80 con 0x7f84600746e0 2024-04-25T00:30:43.633 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-25T00:30:43.633 INFO:teuthology.orchestra.run.smithi116.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T00:30:43.633 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-25T00:30:43.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.634+0000 7f848427f640 1 -- 172.21.15.116:0/3893879824 >> [v2:172.21.15.116:6808/3751765682,v1:172.21.15.116:6809/3751765682] conn(0x7f84600746e0 msgr2=0x7f8460076b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:43.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.634+0000 7f848427f640 1 --2- 172.21.15.116:0/3893879824 >> [v2:172.21.15.116:6808/3751765682,v1:172.21.15.116:6809/3751765682] conn(0x7f84600746e0 0x7f8460076b80 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:43.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.634+0000 7f848427f640 1 -- 172.21.15.116:0/3893879824 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f84600678f0 msgr2=0x7f8460069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:43.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.634+0000 7f848427f640 1 --2- 172.21.15.116:0/3893879824 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f84600678f0 0x7f8460069db0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f847c1e0c30 tx=0x7f8478008040 comp rx=0 tx=0).stop 2024-04-25T00:30:43.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.634+0000 7f848427f640 1 -- 172.21.15.116:0/3893879824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f847c1ec7e0 msgr2=0x7f847c1ecc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:43.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.634+0000 7f848427f640 1 --2- 172.21.15.116:0/3893879824 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f847c1ec7e0 0x7f847c1ecc60 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f847c0c7a50 tx=0x7f84740115a0 comp rx=0 tx=0).stop 2024-04-25T00:30:43.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.634+0000 7f848281c640 1 -- 172.21.15.116:0/3893879824 reap_dead start 2024-04-25T00:30:43.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.634+0000 7f848427f640 1 -- 172.21.15.116:0/3893879824 shutdown_connections 2024-04-25T00:30:43.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.634+0000 7f848427f640 1 -- 172.21.15.116:0/3893879824 >> 172.21.15.116:0/3893879824 conn(0x7f847c06b810 msgr2=0x7f847c06d710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:43.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.634+0000 7f848427f640 1 -- 172.21.15.116:0/3893879824 shutdown_connections 2024-04-25T00:30:43.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.634+0000 7f848427f640 1 -- 172.21.15.116:0/3893879824 wait complete. 2024-04-25T00:30:43.649 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-04-25T00:30:43.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.758+0000 7fa0e7d8b640 1 Processor -- start 2024-04-25T00:30:43.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.758+0000 7fa0e7d8b640 1 -- start start 2024-04-25T00:30:43.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.762+0000 7fa0e7d8b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa0e0150d20 0x7fa0e00c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:43.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.762+0000 7fa0e7d8b640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa0e00c2f50 0x7fa0e01538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:43.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.762+0000 7fa0e7d8b640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0e00c3910 con 0x7fa0e00a94a0 2024-04-25T00:30:43.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.762+0000 7fa0e7d8b640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fa0e0154530 con 0x7fa0e00c2f50 2024-04-25T00:30:43.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.762+0000 7fa0e7d8b640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa0e00c16e0 con 0x7fa0e0150d20 2024-04-25T00:30:43.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.762+0000 7fa0e5326640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa0e00c2f50 0x7fa0e01538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:43.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.762+0000 7fa0e5326640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa0e00c2f50 0x7fa0e01538e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36518/0 (socket says 172.21.15.116:36518) 2024-04-25T00:30:43.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.762+0000 7fa0e5326640 1 -- 172.21.15.116:0/3348442002 learned_addr learned my addr 172.21.15.116:0/3348442002 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:43.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.762+0000 7fa0e5b27640 1 --2- 172.21.15.116:0/3348442002 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa0e0150d20 0x7fa0e00c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:43.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.762+0000 7fa0e5326640 1 -- 172.21.15.116:0/3348442002 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa0e0150d20 msgr2=0x7fa0e00c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:43.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.762+0000 7fa0e5326640 1 --2- 172.21.15.116:0/3348442002 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa0e0150d20 0x7fa0e00c2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:43.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.762+0000 7fa0e5326640 1 -- 172.21.15.116:0/3348442002 >> v1:172.21.15.174:6789/0 conn(0x7fa0e00a94a0 legacy=0x7fa0e0153e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:43.764 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.762+0000 7fa0e5326640 1 -- 172.21.15.116:0/3348442002 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0e00c1400 con 0x7fa0e00c2f50 2024-04-25T00:30:43.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.762+0000 7fa0e5326640 1 --2- 172.21.15.116:0/3348442002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0e00c2f50 0x7fa0e01538e0 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7fa0d0009870 tx=0x7fa0d001c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1da37a5db8dfd93a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:43.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.762+0000 7fa0e4b25640 1 -- 172.21.15.116:0/3348442002 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0d00ab5f0 con 0x7fa0e00c2f50 2024-04-25T00:30:43.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.762+0000 7fa0e4b25640 1 -- 172.21.15.116:0/3348442002 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0d00b0070 con 0x7fa0e00c2f50 2024-04-25T00:30:43.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.762+0000 7fa0e4b25640 1 -- 172.21.15.116:0/3348442002 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0d00ab9b0 con 0x7fa0e00c2f50 2024-04-25T00:30:43.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.766+0000 7fa0e7d8b640 1 -- 172.21.15.116:0/3348442002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0e00c2f50 msgr2=0x7fa0e01538e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:43.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.766+0000 7fa0e7d8b640 1 --2- 172.21.15.116:0/3348442002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0e00c2f50 0x7fa0e01538e0 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7fa0d0009870 tx=0x7fa0d001c520 comp rx=0 tx=0).stop 2024-04-25T00:30:43.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.766+0000 7fa0e7d8b640 1 -- 172.21.15.116:0/3348442002 shutdown_connections 2024-04-25T00:30:43.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.766+0000 7fa0e7d8b640 1 --2- 172.21.15.116:0/3348442002 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0e00c2f50 0x7fa0e01538e0 unknown :-1 s=CLOSED pgs=1809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:43.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.766+0000 7fa0e7d8b640 1 --2- 172.21.15.116:0/3348442002 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa0e0150d20 0x7fa0e00c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:43.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.766+0000 7fa0e7d8b640 1 -- 172.21.15.116:0/3348442002 >> 172.21.15.116:0/3348442002 conn(0x7fa0e00c3b00 msgr2=0x7fa0e00c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:43.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.766+0000 7fa0e7d8b640 1 -- 172.21.15.116:0/3348442002 shutdown_connections 2024-04-25T00:30:43.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.766+0000 7fa0e7d8b640 1 -- 172.21.15.116:0/3348442002 wait complete. 2024-04-25T00:30:43.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.766+0000 7fa0e7d8b640 1 Processor -- start 2024-04-25T00:30:43.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.766+0000 7fa0e7d8b640 1 -- start start 2024-04-25T00:30:43.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.766+0000 7fa0e7d8b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0e00a94a0 0x7fa0e00c6570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:43.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.766+0000 7fa0e7d8b640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa0e00c2f50 0x7fa0e01ef5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:43.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.766+0000 7fa0e7d8b640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa0e0150d20 0x7fa0e0140640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:43.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.766+0000 7fa0e7d8b640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa0e01608b0 con 0x7fa0e00a94a0 2024-04-25T00:30:43.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.766+0000 7fa0e7d8b640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa0e0160730 con 0x7fa0e00c2f50 2024-04-25T00:30:43.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.766+0000 7fa0e7d8b640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa0e0160a30 con 0x7fa0e0150d20 2024-04-25T00:30:43.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.766+0000 7fa0e5b27640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0e00a94a0 0x7fa0e00c6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:43.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.766+0000 7fa0e5b27640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0e00a94a0 0x7fa0e00c6570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36522/0 (socket says 172.21.15.116:36522) 2024-04-25T00:30:43.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.766+0000 7fa0e5b27640 1 -- 172.21.15.116:0/3185480269 learned_addr learned my addr 172.21.15.116:0/3185480269 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:43.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.766+0000 7fa0e5326640 1 --2- 172.21.15.116:0/3185480269 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa0e00c2f50 0x7fa0e01ef5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:43.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.766+0000 7fa0e5b27640 1 -- 172.21.15.116:0/3185480269 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa0e0150d20 msgr2=0x7fa0e0140640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:43.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.766+0000 7fa0e6328640 1 --2- 172.21.15.116:0/3185480269 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa0e0150d20 0x7fa0e0140640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:43.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.766+0000 7fa0e5b27640 1 --2- 172.21.15.116:0/3185480269 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa0e0150d20 0x7fa0e0140640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:43.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.766+0000 7fa0e5b27640 1 -- 172.21.15.116:0/3185480269 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa0e00c2f50 msgr2=0x7fa0e01ef5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:43.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.766+0000 7fa0e5b27640 1 --2- 172.21.15.116:0/3185480269 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa0e00c2f50 0x7fa0e01ef5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:43.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.766+0000 7fa0e5b27640 1 -- 172.21.15.116:0/3185480269 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0e0147110 con 0x7fa0e00a94a0 2024-04-25T00:30:43.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.766+0000 7fa0e5326640 1 --2- 172.21.15.116:0/3185480269 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa0e00c2f50 0x7fa0e01ef5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:30:43.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.766+0000 7fa0e5b27640 1 --2- 172.21.15.116:0/3185480269 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0e00a94a0 0x7fa0e00c6570 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc00d800 tx=0x7fa0dc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:43.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.766+0000 7fa0d67fc640 1 -- 172.21.15.116:0/3185480269 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0dc00b6c0 con 0x7fa0e00a94a0 2024-04-25T00:30:43.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.766+0000 7fa0d67fc640 1 -- 172.21.15.116:0/3185480269 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0dc019070 con 0x7fa0e00a94a0 2024-04-25T00:30:43.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.766+0000 7fa0e7d8b640 1 -- 172.21.15.116:0/3185480269 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0e014b0b0 con 0x7fa0e00a94a0 2024-04-25T00:30:43.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.766+0000 7fa0d67fc640 1 -- 172.21.15.116:0/3185480269 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0dc00ba60 con 0x7fa0e00a94a0 2024-04-25T00:30:43.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.766+0000 7fa0e7d8b640 1 -- 172.21.15.116:0/3185480269 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0e01ec4d0 con 0x7fa0e00a94a0 2024-04-25T00:30:43.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.770+0000 7fa0e7d8b640 1 -- 172.21.15.116:0/3185480269 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0e01ec000 con 0x7fa0e00a94a0 2024-04-25T00:30:43.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.770+0000 7fa0d67fc640 1 -- 172.21.15.116:0/3185480269 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa0dc00bc00 con 0x7fa0e00a94a0 2024-04-25T00:30:43.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.770+0000 7fa0d67fc640 1 --2- 172.21.15.116:0/3185480269 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa0c40678c0 0x7fa0c4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:43.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.770+0000 7fa0d67fc640 1 -- 172.21.15.116:0/3185480269 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7fa0dc087440 con 0x7fa0e00a94a0 2024-04-25T00:30:43.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.770+0000 7fa0e5326640 1 --2- 172.21.15.116:0/3185480269 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa0c40678c0 0x7fa0c4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:43.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.770+0000 7fa0e5326640 1 --2- 172.21.15.116:0/3185480269 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa0c40678c0 0x7fa0c4069d80 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fa0d00af000 tx=0x7fa0d0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:43.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.774+0000 7fa0d67fc640 1 -- 172.21.15.116:0/3185480269 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fa0dc056c90 con 0x7fa0e00a94a0 2024-04-25T00:30:43.932 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.930+0000 7fa0e7d8b640 1 -- 172.21.15.116:0/3185480269 --> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7fa0e0068350 con 0x7fa0c40678c0 2024-04-25T00:30:43.933 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.930+0000 7fa0d67fc640 1 -- 172.21.15.116:0/3185480269 <== mgr.4105 v2:172.21.15.174:6832/2004448211 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fa0e0068350 con 0x7fa0c40678c0 2024-04-25T00:30:43.933 INFO:teuthology.orchestra.run.smithi116.stderr:instructed osd(s) 0 to deep-scrub 2024-04-25T00:30:43.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.934+0000 7fa0e7d8b640 1 -- 172.21.15.116:0/3185480269 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa0c40678c0 msgr2=0x7fa0c4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:43.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.934+0000 7fa0e7d8b640 1 --2- 172.21.15.116:0/3185480269 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa0c40678c0 0x7fa0c4069d80 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fa0d00af000 tx=0x7fa0d0008040 comp rx=0 tx=0).stop 2024-04-25T00:30:43.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.934+0000 7fa0e7d8b640 1 -- 172.21.15.116:0/3185480269 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0e00a94a0 msgr2=0x7fa0e00c6570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:43.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.934+0000 7fa0e7d8b640 1 --2- 172.21.15.116:0/3185480269 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0e00a94a0 0x7fa0e00c6570 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc00d800 tx=0x7fa0dc00dcc0 comp rx=0 tx=0).stop 2024-04-25T00:30:43.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.934+0000 7fa0e7d8b640 1 -- 172.21.15.116:0/3185480269 shutdown_connections 2024-04-25T00:30:43.935 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.934+0000 7fa0e7d8b640 1 --2- 172.21.15.116:0/3185480269 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa0c40678c0 0x7fa0c4069d80 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:43.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.934+0000 7fa0e7d8b640 1 --2- 172.21.15.116:0/3185480269 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa0e0150d20 0x7fa0e0140640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:43.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.934+0000 7fa0e7d8b640 1 --2- 172.21.15.116:0/3185480269 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa0e00c2f50 0x7fa0e01ef5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:43.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.934+0000 7fa0e7d8b640 1 --2- 172.21.15.116:0/3185480269 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa0e00a94a0 0x7fa0e00c6570 unknown :-1 s=CLOSED pgs=1810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:43.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.934+0000 7fa0e7d8b640 1 -- 172.21.15.116:0/3185480269 >> 172.21.15.116:0/3185480269 conn(0x7fa0e00c3b00 msgr2=0x7fa0e01539e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:43.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.934+0000 7fa0e7d8b640 1 -- 172.21.15.116:0/3185480269 shutdown_connections 2024-04-25T00:30:43.936 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:43.934+0000 7fa0e7d8b640 1 -- 172.21.15.116:0/3185480269 wait complete. 2024-04-25T00:30:43.951 INFO:tasks.ceph:Scrubbing osd.1 2024-04-25T00:30:43.951 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-25T00:30:44.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.074+0000 7fb995f72640 1 Processor -- start 2024-04-25T00:30:44.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.074+0000 7fb995f72640 1 -- start start 2024-04-25T00:30:44.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.074+0000 7fb995f72640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb99014a020 0x7fb9900c24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:44.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.074+0000 7fb995f72640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9900ab2f0 con 0x7fb990153f10 2024-04-25T00:30:44.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.074+0000 7fb995f72640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb99014a400 con 0x7fb9900a94a0 2024-04-25T00:30:44.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.074+0000 7fb995f72640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fb9900ac480 con 0x7fb99014a020 2024-04-25T00:30:44.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.074+0000 7fb98f7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb99014a020 0x7fb9900c24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:44.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.074+0000 7fb98f7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fb99014a020 0x7fb9900c24c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36528/0 (socket says 172.21.15.116:36528) 2024-04-25T00:30:44.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.074+0000 7fb98f7fe640 1 -- 172.21.15.116:0/7695550 learned_addr learned my addr 172.21.15.116:0/7695550 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:44.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.074+0000 7fb98f7fe640 1 -- 172.21.15.116:0/7695550 >> v1:172.21.15.174:6789/0 conn(0x7fb9900a94a0 legacy=0x7fb99015de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:44.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.074+0000 7fb98f7fe640 1 -- 172.21.15.116:0/7695550 >> v1:172.21.15.116:6789/0 conn(0x7fb990153f10 legacy=0x7fb9900abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:44.076 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.074+0000 7fb98f7fe640 1 -- 172.21.15.116:0/7695550 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb990154370 con 0x7fb99014a020 2024-04-25T00:30:44.077 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.074+0000 7fb98effd640 1 -- 172.21.15.116:0/7695550 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3989041153 0 0) 0x7fb978001120 con 0x7fb990153f10 2024-04-25T00:30:44.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.078+0000 7fb98f7fe640 1 --2- 172.21.15.116:0/7695550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb99014a020 0x7fb9900c24c0 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7fb980009870 tx=0x7fb98001c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33f346f281a1522f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:44.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.078+0000 7fb98effd640 1 -- 172.21.15.116:0/7695550 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9800ab600 con 0x7fb99014a020 2024-04-25T00:30:44.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.078+0000 7fb98effd640 1 -- 172.21.15.116:0/7695550 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9800b0070 con 0x7fb99014a020 2024-04-25T00:30:44.078 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.078+0000 7fb98effd640 1 -- 172.21.15.116:0/7695550 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9800ab9c0 con 0x7fb99014a020 2024-04-25T00:30:44.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.078+0000 7fb995f72640 1 -- 172.21.15.116:0/7695550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb99014a020 msgr2=0x7fb9900c24c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:44.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.078+0000 7fb995f72640 1 --2- 172.21.15.116:0/7695550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb99014a020 0x7fb9900c24c0 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7fb980009870 tx=0x7fb98001c4f0 comp rx=0 tx=0).stop 2024-04-25T00:30:44.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.078+0000 7fb995f72640 1 -- 172.21.15.116:0/7695550 shutdown_connections 2024-04-25T00:30:44.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.078+0000 7fb995f72640 1 --2- 172.21.15.116:0/7695550 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb99014a020 0x7fb9900c24c0 unknown :-1 s=CLOSED pgs=1812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:44.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.078+0000 7fb995f72640 1 -- 172.21.15.116:0/7695550 >> 172.21.15.116:0/7695550 conn(0x7fb99006b810 msgr2=0x7fb9900c6f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:44.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.078+0000 7fb995f72640 1 -- 172.21.15.116:0/7695550 shutdown_connections 2024-04-25T00:30:44.079 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.078+0000 7fb995f72640 1 -- 172.21.15.116:0/7695550 wait complete. 2024-04-25T00:30:44.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.078+0000 7fb995f72640 1 Processor -- start 2024-04-25T00:30:44.080 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.078+0000 7fb995f72640 1 -- start start 2024-04-25T00:30:44.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.078+0000 7fb995f72640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9900a94a0 0x7fb990150e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:44.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.078+0000 7fb995f72640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb99014a020 0x7fb990148d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:44.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.078+0000 7fb995f72640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb990153f10 0x7fb9901efd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:44.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.078+0000 7fb995f72640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fb9900c1560 con 0x7fb99014a020 2024-04-25T00:30:44.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.078+0000 7fb995f72640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fb9900ab850 con 0x7fb9900a94a0 2024-04-25T00:30:44.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.078+0000 7fb995f72640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fb9900c7ca0 con 0x7fb990153f10 2024-04-25T00:30:44.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.078+0000 7fb986ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb99014a020 0x7fb990148d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:44.082 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.078+0000 7fb986ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb99014a020 0x7fb990148d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36542/0 (socket says 172.21.15.116:36542) 2024-04-25T00:30:44.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.078+0000 7fb986ffd640 1 -- 172.21.15.116:0/1558779423 learned_addr learned my addr 172.21.15.116:0/1558779423 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:44.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.082+0000 7fb98f7fe640 1 --2- 172.21.15.116:0/1558779423 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9900a94a0 0x7fb990150e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:44.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.082+0000 7fb98ffff640 1 --2- 172.21.15.116:0/1558779423 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb990153f10 0x7fb9901efd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:44.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.082+0000 7fb986ffd640 1 -- 172.21.15.116:0/1558779423 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb990153f10 msgr2=0x7fb9901efd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:44.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.082+0000 7fb986ffd640 1 --2- 172.21.15.116:0/1558779423 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb990153f10 0x7fb9901efd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:44.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.082+0000 7fb986ffd640 1 -- 172.21.15.116:0/1558779423 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9900a94a0 msgr2=0x7fb990150e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:44.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.082+0000 7fb986ffd640 1 --2- 172.21.15.116:0/1558779423 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fb9900a94a0 0x7fb990150e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:44.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.082+0000 7fb986ffd640 1 -- 172.21.15.116:0/1558779423 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9901532d0 con 0x7fb99014a020 2024-04-25T00:30:44.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.082+0000 7fb98ffff640 1 --2- 172.21.15.116:0/1558779423 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fb990153f10 0x7fb9901efd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:30:44.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.082+0000 7fb986ffd640 1 --2- 172.21.15.116:0/1558779423 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb99014a020 0x7fb990148d30 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7fb97800c450 tx=0x7fb97800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:44.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.082+0000 7fb98cff9640 1 -- 172.21.15.116:0/1558779423 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb978017070 con 0x7fb99014a020 2024-04-25T00:30:44.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.082+0000 7fb98cff9640 1 -- 172.21.15.116:0/1558779423 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb978005480 con 0x7fb99014a020 2024-04-25T00:30:44.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.082+0000 7fb995f72640 1 -- 172.21.15.116:0/1558779423 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9900716e0 con 0x7fb99014a020 2024-04-25T00:30:44.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.082+0000 7fb98cff9640 1 -- 172.21.15.116:0/1558779423 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb978003840 con 0x7fb99014a020 2024-04-25T00:30:44.083 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.082+0000 7fb995f72640 1 -- 172.21.15.116:0/1558779423 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb990071c80 con 0x7fb99014a020 2024-04-25T00:30:44.084 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.082+0000 7fb995f72640 1 -- 172.21.15.116:0/1558779423 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb990068350 con 0x7fb99014a020 2024-04-25T00:30:44.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.082+0000 7fb98cff9640 1 -- 172.21.15.116:0/1558779423 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fb97801a070 con 0x7fb99014a020 2024-04-25T00:30:44.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.082+0000 7fb98cff9640 1 --2- 172.21.15.116:0/1558779423 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb9700678c0 0x7fb970069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:44.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.082+0000 7fb98cff9640 1 -- 172.21.15.116:0/1558779423 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7fb978086930 con 0x7fb99014a020 2024-04-25T00:30:44.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.082+0000 7fb98cff9640 1 --2- 172.21.15.116:0/1558779423 >> [v2:172.21.15.116:6800/377879124,v1:172.21.15.116:6801/377879124] conn(0x7fb9700746c0 0x7fb970076b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:44.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.082+0000 7fb98cff9640 1 -- 172.21.15.116:0/1558779423 --> [v2:172.21.15.116:6800/377879124,v1:172.21.15.116:6801/377879124] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb978020ec0 con 0x7fb9700746c0 2024-04-25T00:30:44.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.082+0000 7fb98cff9640 1 -- 172.21.15.116:0/1558779423 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=74) ==== 24+0+0 (secure 0 0 0) 0x7fb978086e10 con 0x7fb99014a020 2024-04-25T00:30:44.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.082+0000 7fb98f7fe640 1 --2- 172.21.15.116:0/1558779423 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb9700678c0 0x7fb970069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:44.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.082+0000 7fb98ffff640 1 --2- 172.21.15.116:0/1558779423 >> [v2:172.21.15.116:6800/377879124,v1:172.21.15.116:6801/377879124] conn(0x7fb9700746c0 0x7fb970076b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:44.085 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.082+0000 7fb98ffff640 1 --2- 172.21.15.116:0/1558779423 >> [v2:172.21.15.116:6800/377879124,v1:172.21.15.116:6801/377879124] conn(0x7fb9700746c0 0x7fb970076b60 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:44.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.082+0000 7fb98f7fe640 1 --2- 172.21.15.116:0/1558779423 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb9700678c0 0x7fb970069d80 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fb9800af000 tx=0x7fb9800a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:44.086 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.086+0000 7fb98cff9640 1 -- 172.21.15.116:0/1558779423 <== osd.1 v2:172.21.15.116:6800/377879124 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb978020ec0 con 0x7fb9700746c0 2024-04-25T00:30:44.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.098+0000 7fb995f72640 1 -- 172.21.15.116:0/1558779423 --> [v2:172.21.15.116:6800/377879124,v1:172.21.15.116:6801/377879124] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fb990153cf0 con 0x7fb9700746c0 2024-04-25T00:30:44.099 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.098+0000 7fb98cff9640 1 -- 172.21.15.116:0/1558779423 <== osd.1 v2:172.21.15.116:6800/377879124 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fb990153cf0 con 0x7fb9700746c0 2024-04-25T00:30:44.099 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-25T00:30:44.099 INFO:teuthology.orchestra.run.smithi116.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T00:30:44.099 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-25T00:30:44.100 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.098+0000 7fb995f72640 1 -- 172.21.15.116:0/1558779423 >> [v2:172.21.15.116:6800/377879124,v1:172.21.15.116:6801/377879124] conn(0x7fb9700746c0 msgr2=0x7fb970076b60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:44.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.098+0000 7fb995f72640 1 --2- 172.21.15.116:0/1558779423 >> [v2:172.21.15.116:6800/377879124,v1:172.21.15.116:6801/377879124] conn(0x7fb9700746c0 0x7fb970076b60 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:44.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.098+0000 7fb995f72640 1 -- 172.21.15.116:0/1558779423 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb9700678c0 msgr2=0x7fb970069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:44.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.098+0000 7fb995f72640 1 --2- 172.21.15.116:0/1558779423 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fb9700678c0 0x7fb970069d80 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fb9800af000 tx=0x7fb9800a7000 comp rx=0 tx=0).stop 2024-04-25T00:30:44.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.098+0000 7fb995f72640 1 -- 172.21.15.116:0/1558779423 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb99014a020 msgr2=0x7fb990148d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:44.101 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.098+0000 7fb995f72640 1 --2- 172.21.15.116:0/1558779423 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fb99014a020 0x7fb990148d30 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7fb97800c450 tx=0x7fb97800c910 comp rx=0 tx=0).stop 2024-04-25T00:30:44.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.098+0000 7fb98ffff640 1 -- 172.21.15.116:0/1558779423 reap_dead start 2024-04-25T00:30:44.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.098+0000 7fb995f72640 1 -- 172.21.15.116:0/1558779423 shutdown_connections 2024-04-25T00:30:44.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.098+0000 7fb995f72640 1 -- 172.21.15.116:0/1558779423 >> 172.21.15.116:0/1558779423 conn(0x7fb99006b810 msgr2=0x7fb990143b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:44.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.098+0000 7fb995f72640 1 -- 172.21.15.116:0/1558779423 shutdown_connections 2024-04-25T00:30:44.102 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.098+0000 7fb995f72640 1 -- 172.21.15.116:0/1558779423 wait complete. 2024-04-25T00:30:44.115 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-04-25T00:30:44.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.226+0000 7f1861741640 1 Processor -- start 2024-04-25T00:30:44.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.226+0000 7f1861741640 1 -- start start 2024-04-25T00:30:44.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.230+0000 7f1861741640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f185c1562d0 0x7f185c161130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:44.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.230+0000 7f1861741640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f185c073200 con 0x7f185c152e60 2024-04-25T00:30:44.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.230+0000 7f1861741640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f185c158ec0 con 0x7f185c1599a0 2024-04-25T00:30:44.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.230+0000 7f1861741640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f185c153240 con 0x7f185c1562d0 2024-04-25T00:30:44.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.230+0000 7f185affd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f185c152e60 0x7f185c0c36e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57456/0 (socket says 172.21.15.116:57456) 2024-04-25T00:30:44.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.230+0000 7f185affd640 1 -- 172.21.15.116:0/2732285046 learned_addr learned my addr 172.21.15.116:0/2732285046 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:30:44.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.230+0000 7f1859ffb640 1 -- 172.21.15.116:0/2732285046 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2091571802 0 0) 0x7f185c073200 con 0x7f185c152e60 2024-04-25T00:30:44.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.230+0000 7f1859ffb640 1 -- 172.21.15.116:0/2732285046 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1844003530 con 0x7f185c152e60 2024-04-25T00:30:44.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.230+0000 7f185b7fe640 1 --2- 172.21.15.116:0/2732285046 >> v2:172.21.15.174:3300/0 conn(0x7f185c1562d0 0x7f185c161130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:44.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.230+0000 7f1859ffb640 1 -- 172.21.15.116:0/2732285046 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3965437814 0 0) 0x7f1844003530 con 0x7f185c152e60 2024-04-25T00:30:44.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.230+0000 7f1859ffb640 1 -- 172.21.15.116:0/2732285046 >> v2:172.21.15.174:3300/0 conn(0x7f185c1562d0 msgr2=0x7f185c161130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:44.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.230+0000 7f1859ffb640 1 --2- 172.21.15.116:0/2732285046 >> v2:172.21.15.174:3300/0 conn(0x7f185c1562d0 0x7f185c161130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:44.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.230+0000 7f1859ffb640 1 -- 172.21.15.116:0/2732285046 >> v1:172.21.15.174:6789/0 conn(0x7f185c1599a0 legacy=0x7f185c0c5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:44.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.230+0000 7f1859ffb640 1 -- 172.21.15.116:0/2732285046 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f185c1591a0 con 0x7f185c152e60 2024-04-25T00:30:44.232 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.230+0000 7f1859ffb640 1 -- 172.21.15.116:0/2732285046 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f184c001680 con 0x7f185c152e60 2024-04-25T00:30:44.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.230+0000 7f1859ffb640 1 -- 172.21.15.116:0/2732285046 >> v1:172.21.15.116:6789/0 conn(0x7f185c152e60 legacy=0x7f185c0c36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:44.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.230+0000 7f1859ffb640 1 --2- 172.21.15.116:0/2732285046 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f18440039c0 0x7f1844003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:44.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.230+0000 7f1859ffb640 1 --2- 172.21.15.116:0/2732285046 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1844004810 0x7f1844004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:44.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.230+0000 7f1859ffb640 1 --2- 172.21.15.116:0/2732285046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18440052f0 0x7f184400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:44.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.230+0000 7f1859ffb640 1 -- 172.21.15.116:0/2732285046 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f185c153240 con 0x7f18440052f0 2024-04-25T00:30:44.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.230+0000 7f1859ffb640 1 -- 172.21.15.116:0/2732285046 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f184400c1c0 con 0x7f18440039c0 2024-04-25T00:30:44.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.230+0000 7f1859ffb640 1 -- 172.21.15.116:0/2732285046 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f184400c3c0 con 0x7f1844004810 2024-04-25T00:30:44.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.230+0000 7f185b7fe640 1 --2- 172.21.15.116:0/2732285046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18440052f0 0x7f184400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:44.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.230+0000 7f185affd640 1 --2- 172.21.15.116:0/2732285046 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f18440039c0 0x7f1844003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:44.233 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.230+0000 7f185a7fc640 1 --2- 172.21.15.116:0/2732285046 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1844004810 0x7f1844004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:44.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.230+0000 7f185b7fe640 1 -- 172.21.15.116:0/2732285046 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1844004810 msgr2=0x7f1844004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:44.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.230+0000 7f185b7fe640 1 --2- 172.21.15.116:0/2732285046 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1844004810 0x7f1844004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:44.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.230+0000 7f185b7fe640 1 -- 172.21.15.116:0/2732285046 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f18440039c0 msgr2=0x7f1844003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:44.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.230+0000 7f185b7fe640 1 --2- 172.21.15.116:0/2732285046 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f18440039c0 0x7f1844003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:44.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.230+0000 7f185b7fe640 1 -- 172.21.15.116:0/2732285046 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1844003530 con 0x7f18440052f0 2024-04-25T00:30:44.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.230+0000 7f185b7fe640 1 -- 172.21.15.116:0/2732285046 reap_dead start 2024-04-25T00:30:44.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.230+0000 7f185b7fe640 1 --2- 172.21.15.116:0/2732285046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18440052f0 0x7f184400b9d0 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f1854009fc0 tx=0x7f185401ce40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e2fd27272842faf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:44.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.230+0000 7f1859ffb640 1 -- 172.21.15.116:0/2732285046 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1854009bc0 con 0x7f18440052f0 2024-04-25T00:30:44.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.230+0000 7f1859ffb640 1 -- 172.21.15.116:0/2732285046 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18540a5c50 con 0x7f18440052f0 2024-04-25T00:30:44.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.230+0000 7f1859ffb640 1 -- 172.21.15.116:0/2732285046 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18540ad440 con 0x7f18440052f0 2024-04-25T00:30:44.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.234+0000 7f1861741640 1 -- 172.21.15.116:0/2732285046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18440052f0 msgr2=0x7f184400b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:44.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.234+0000 7f1861741640 1 --2- 172.21.15.116:0/2732285046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18440052f0 0x7f184400b9d0 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f1854009fc0 tx=0x7f185401ce40 comp rx=0 tx=0).stop 2024-04-25T00:30:44.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.234+0000 7f1861741640 1 -- 172.21.15.116:0/2732285046 shutdown_connections 2024-04-25T00:30:44.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.234+0000 7f1861741640 1 --2- 172.21.15.116:0/2732285046 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f18440052f0 0x7f184400b9d0 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:44.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.234+0000 7f1861741640 1 -- 172.21.15.116:0/2732285046 >> 172.21.15.116:0/2732285046 conn(0x7f185c06c720 msgr2=0x7f185c06cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:44.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.234+0000 7f1861741640 1 -- 172.21.15.116:0/2732285046 shutdown_connections 2024-04-25T00:30:44.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.234+0000 7f1861741640 1 -- 172.21.15.116:0/2732285046 wait complete. 2024-04-25T00:30:44.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.234+0000 7f1861741640 1 Processor -- start 2024-04-25T00:30:44.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.234+0000 7f1861741640 1 -- start start 2024-04-25T00:30:44.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.234+0000 7f1861741640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f18440052f0 0x7f185c0c6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:44.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.234+0000 7f1861741640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f185c14f180 0x7f185c0ab6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:44.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.234+0000 7f1861741640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f185c0abbf0 0x7f185c14e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:44.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.234+0000 7f1861741640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f185c156720 con 0x7f185c14f180 2024-04-25T00:30:44.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.234+0000 7f1861741640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f185c0734e0 con 0x7f18440052f0 2024-04-25T00:30:44.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.234+0000 7f1861741640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f185c072080 con 0x7f185c0abbf0 2024-04-25T00:30:44.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.234+0000 7f185a7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f185c14f180 0x7f185c0ab6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:44.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.234+0000 7f185a7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f185c14f180 0x7f185c0ab6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36568/0 (socket says 172.21.15.116:36568) 2024-04-25T00:30:44.240 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.234+0000 7f185a7fc640 1 -- 172.21.15.116:0/623926 learned_addr learned my addr 172.21.15.116:0/623926 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:44.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.234+0000 7f185affd640 1 --2- 172.21.15.116:0/623926 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f18440052f0 0x7f185c0c6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:44.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.234+0000 7f185b7fe640 1 --2- 172.21.15.116:0/623926 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f185c0abbf0 0x7f185c14e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:44.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.234+0000 7f185a7fc640 1 -- 172.21.15.116:0/623926 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f185c0abbf0 msgr2=0x7f185c14e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:44.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.234+0000 7f185a7fc640 1 --2- 172.21.15.116:0/623926 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f185c0abbf0 0x7f185c14e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:44.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.234+0000 7f185a7fc640 1 -- 172.21.15.116:0/623926 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f18440052f0 msgr2=0x7f185c0c6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:44.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.234+0000 7f185a7fc640 1 --2- 172.21.15.116:0/623926 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f18440052f0 0x7f185c0c6fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:44.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.234+0000 7f185a7fc640 1 -- 172.21.15.116:0/623926 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f185c0aca00 con 0x7f185c14f180 2024-04-25T00:30:44.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.234+0000 7f185a7fc640 1 --2- 172.21.15.116:0/623926 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f185c14f180 0x7f185c0ab6b0 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f185000e510 tx=0x7f185000e9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:44.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.234+0000 7f18437fe640 1 -- 172.21.15.116:0/623926 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18500052e0 con 0x7f185c14f180 2024-04-25T00:30:44.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.234+0000 7f18437fe640 1 -- 172.21.15.116:0/623926 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1850018070 con 0x7f185c14f180 2024-04-25T00:30:44.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.234+0000 7f1861741640 1 -- 172.21.15.116:0/623926 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f185c1ebda0 con 0x7f185c14f180 2024-04-25T00:30:44.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.234+0000 7f18437fe640 1 -- 172.21.15.116:0/623926 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1850005480 con 0x7f185c14f180 2024-04-25T00:30:44.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.234+0000 7f1861741640 1 -- 172.21.15.116:0/623926 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f185c0acd70 con 0x7f185c14f180 2024-04-25T00:30:44.241 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.238+0000 7f1861741640 1 -- 172.21.15.116:0/623926 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f185c068a60 con 0x7f185c14f180 2024-04-25T00:30:44.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.238+0000 7f18437fe640 1 -- 172.21.15.116:0/623926 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1850003840 con 0x7f185c14f180 2024-04-25T00:30:44.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.238+0000 7f18437fe640 1 --2- 172.21.15.116:0/623926 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f18480678c0 0x7f1848069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:44.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.238+0000 7f18437fe640 1 -- 172.21.15.116:0/623926 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7f1850086b70 con 0x7f185c14f180 2024-04-25T00:30:44.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.238+0000 7f185affd640 1 --2- 172.21.15.116:0/623926 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f18480678c0 0x7f1848069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:44.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.238+0000 7f185affd640 1 --2- 172.21.15.116:0/623926 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f18480678c0 0x7f1848069d80 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f184c001060 tx=0x7f184c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:44.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.242+0000 7f18437fe640 1 -- 172.21.15.116:0/623926 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f18500563c0 con 0x7f185c14f180 2024-04-25T00:30:44.396 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.394+0000 7f1861741640 1 -- 172.21.15.116:0/623926 --> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f185c0abfd0 con 0x7f18480678c0 2024-04-25T00:30:44.397 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.398+0000 7f18437fe640 1 -- 172.21.15.116:0/623926 <== mgr.4105 v2:172.21.15.174:6832/2004448211 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f185c0abfd0 con 0x7f18480678c0 2024-04-25T00:30:44.397 INFO:teuthology.orchestra.run.smithi116.stderr:instructed osd(s) 1 to deep-scrub 2024-04-25T00:30:44.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.398+0000 7f1861741640 1 -- 172.21.15.116:0/623926 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f18480678c0 msgr2=0x7f1848069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:44.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.398+0000 7f1861741640 1 --2- 172.21.15.116:0/623926 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f18480678c0 0x7f1848069d80 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f184c001060 tx=0x7f184c009040 comp rx=0 tx=0).stop 2024-04-25T00:30:44.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.398+0000 7f1861741640 1 -- 172.21.15.116:0/623926 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f185c14f180 msgr2=0x7f185c0ab6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:44.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.398+0000 7f1861741640 1 --2- 172.21.15.116:0/623926 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f185c14f180 0x7f185c0ab6b0 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f185000e510 tx=0x7f185000e9d0 comp rx=0 tx=0).stop 2024-04-25T00:30:44.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.398+0000 7f1861741640 1 -- 172.21.15.116:0/623926 shutdown_connections 2024-04-25T00:30:44.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.398+0000 7f1861741640 1 --2- 172.21.15.116:0/623926 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f18480678c0 0x7f1848069d80 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:44.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.398+0000 7f1861741640 1 --2- 172.21.15.116:0/623926 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f185c0abbf0 0x7f185c14e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:44.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.398+0000 7f1861741640 1 --2- 172.21.15.116:0/623926 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f185c14f180 0x7f185c0ab6b0 unknown :-1 s=CLOSED pgs=1816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:44.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.398+0000 7f1861741640 1 --2- 172.21.15.116:0/623926 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f18440052f0 0x7f185c0c6fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:44.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.398+0000 7f1861741640 1 -- 172.21.15.116:0/623926 >> 172.21.15.116:0/623926 conn(0x7f185c06c720 msgr2=0x7f185c0a94a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:44.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.398+0000 7f1861741640 1 -- 172.21.15.116:0/623926 shutdown_connections 2024-04-25T00:30:44.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.398+0000 7f1861741640 1 -- 172.21.15.116:0/623926 wait complete. 2024-04-25T00:30:44.416 INFO:tasks.ceph:Scrubbing osd.2 2024-04-25T00:30:44.416 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-25T00:30:44.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.538+0000 7f45d9322640 1 Processor -- start 2024-04-25T00:30:44.540 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.538+0000 7f45d9322640 1 -- start start 2024-04-25T00:30:44.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.542+0000 7f45d9322640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f45d414d690 0x7f45d4073090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:44.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.542+0000 7f45d9322640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f45d406ee40 0x7f45d40ad4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:44.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.542+0000 7f45d9322640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45d4156ca0 con 0x7f45d40ada10 2024-04-25T00:30:44.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.542+0000 7f45d9322640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f45d414da70 con 0x7f45d414d690 2024-04-25T00:30:44.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.542+0000 7f45d9322640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f45d40aa540 con 0x7f45d406ee40 2024-04-25T00:30:44.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.542+0000 7f45d2ffd640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f45d414d690 0x7f45d4073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:44.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.542+0000 7f45d27fc640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f45d406ee40 0x7f45d40ad4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:44.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.542+0000 7f45d27fc640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f45d406ee40 0x7f45d40ad4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:33816/0 (socket says 172.21.15.116:33816) 2024-04-25T00:30:44.542 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.542+0000 7f45d27fc640 1 -- 172.21.15.116:0/493659555 learned_addr learned my addr 172.21.15.116:0/493659555 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:44.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.542+0000 7f45d27fc640 1 -- 172.21.15.116:0/493659555 >> v2:172.21.15.174:3300/0 conn(0x7f45d414d690 msgr2=0x7f45d4073090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:44.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.542+0000 7f45d27fc640 1 --2- 172.21.15.116:0/493659555 >> v2:172.21.15.174:3300/0 conn(0x7f45d414d690 0x7f45d4073090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:44.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.542+0000 7f45d27fc640 1 -- 172.21.15.116:0/493659555 >> v1:172.21.15.174:6789/0 conn(0x7f45d40ada10 legacy=0x7f45d41571a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:44.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.542+0000 7f45d27fc640 1 -- 172.21.15.116:0/493659555 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45d40a94a0 con 0x7f45d406ee40 2024-04-25T00:30:44.543 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.542+0000 7f45d2ffd640 1 --2- 172.21.15.116:0/493659555 >> v2:172.21.15.174:3300/0 conn(0x7f45d414d690 0x7f45d4073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:30:44.544 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.542+0000 7f45d27fc640 1 --2- 172.21.15.116:0/493659555 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f45d406ee40 0x7f45d40ad4d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f45c8009870 tx=0x7f45c801c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5d48aff729447c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:44.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.542+0000 7f45d1ffb640 1 -- 172.21.15.116:0/493659555 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c80b1070 con 0x7f45d406ee40 2024-04-25T00:30:44.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.542+0000 7f45d1ffb640 1 -- 172.21.15.116:0/493659555 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45c80ac620 con 0x7f45d406ee40 2024-04-25T00:30:44.545 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.542+0000 7f45d1ffb640 1 -- 172.21.15.116:0/493659555 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c80ac9f0 con 0x7f45d406ee40 2024-04-25T00:30:44.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.546+0000 7f45d9322640 1 -- 172.21.15.116:0/493659555 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f45d406ee40 msgr2=0x7f45d40ad4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:44.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.546+0000 7f45d9322640 1 --2- 172.21.15.116:0/493659555 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f45d406ee40 0x7f45d40ad4d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f45c8009870 tx=0x7f45c801c4d0 comp rx=0 tx=0).stop 2024-04-25T00:30:44.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.546+0000 7f45d9322640 1 -- 172.21.15.116:0/493659555 shutdown_connections 2024-04-25T00:30:44.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.546+0000 7f45d9322640 1 --2- 172.21.15.116:0/493659555 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f45d406ee40 0x7f45d40ad4d0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:44.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.546+0000 7f45d9322640 1 --2- 172.21.15.116:0/493659555 >> v2:172.21.15.174:3300/0 conn(0x7f45d414d690 0x7f45d4073090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:44.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.546+0000 7f45d9322640 1 -- 172.21.15.116:0/493659555 >> 172.21.15.116:0/493659555 conn(0x7f45d40c80a0 msgr2=0x7f45d40c84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:44.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.546+0000 7f45d9322640 1 -- 172.21.15.116:0/493659555 shutdown_connections 2024-04-25T00:30:44.546 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.546+0000 7f45d9322640 1 -- 172.21.15.116:0/493659555 wait complete. 2024-04-25T00:30:44.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.546+0000 7f45d9322640 1 Processor -- start 2024-04-25T00:30:44.547 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.546+0000 7f45d9322640 1 -- start start 2024-04-25T00:30:44.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.546+0000 7f45d9322640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45d406ee40 0x7f45d40c7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:44.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.546+0000 7f45d9322640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45d40ada10 0x7f45d41513a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:44.549 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.546+0000 7f45d9322640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f45d414d690 0x7f45d4155240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:44.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.546+0000 7f45d9322640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f45d4157a30 con 0x7f45d406ee40 2024-04-25T00:30:44.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.546+0000 7f45d9322640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f45d41578b0 con 0x7f45d40ada10 2024-04-25T00:30:44.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.546+0000 7f45d9322640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f45d41606d0 con 0x7f45d414d690 2024-04-25T00:30:44.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.546+0000 7f45d2ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45d406ee40 0x7f45d40c7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:44.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.546+0000 7f45d2ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45d406ee40 0x7f45d40c7470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36570/0 (socket says 172.21.15.116:36570) 2024-04-25T00:30:44.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.546+0000 7f45d2ffd640 1 -- 172.21.15.116:0/3386151785 learned_addr learned my addr 172.21.15.116:0/3386151785 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:44.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.546+0000 7f45d27fc640 1 --2- 172.21.15.116:0/3386151785 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45d40ada10 0x7f45d41513a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:44.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.546+0000 7f45d37fe640 1 --2- 172.21.15.116:0/3386151785 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f45d414d690 0x7f45d4155240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:44.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.546+0000 7f45d2ffd640 1 -- 172.21.15.116:0/3386151785 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f45d414d690 msgr2=0x7f45d4155240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:44.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.546+0000 7f45d2ffd640 1 --2- 172.21.15.116:0/3386151785 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f45d414d690 0x7f45d4155240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:44.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.546+0000 7f45d2ffd640 1 -- 172.21.15.116:0/3386151785 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45d40ada10 msgr2=0x7f45d41513a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:44.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.546+0000 7f45d2ffd640 1 --2- 172.21.15.116:0/3386151785 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f45d40ada10 0x7f45d41513a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:44.550 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.546+0000 7f45d2ffd640 1 -- 172.21.15.116:0/3386151785 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45d40c79b0 con 0x7f45d406ee40 2024-04-25T00:30:44.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.546+0000 7f45d2ffd640 1 --2- 172.21.15.116:0/3386151785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45d406ee40 0x7f45d40c7470 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f45c400e980 tx=0x7f45c400ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:44.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.546+0000 7f45bb7fe640 1 -- 172.21.15.116:0/3386151785 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c4017070 con 0x7f45d406ee40 2024-04-25T00:30:44.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.546+0000 7f45bb7fe640 1 -- 172.21.15.116:0/3386151785 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45c400ca00 con 0x7f45d406ee40 2024-04-25T00:30:44.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.546+0000 7f45d9322640 1 -- 172.21.15.116:0/3386151785 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45d41414c0 con 0x7f45d406ee40 2024-04-25T00:30:44.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.546+0000 7f45bb7fe640 1 -- 172.21.15.116:0/3386151785 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c400cdd0 con 0x7f45d406ee40 2024-04-25T00:30:44.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.546+0000 7f45d9322640 1 -- 172.21.15.116:0/3386151785 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45d4154b50 con 0x7f45d406ee40 2024-04-25T00:30:44.551 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.550+0000 7f45d9322640 1 -- 172.21.15.116:0/3386151785 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f45d4068a60 con 0x7f45d406ee40 2024-04-25T00:30:44.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.550+0000 7f45bb7fe640 1 -- 172.21.15.116:0/3386151785 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f45c401a070 con 0x7f45d406ee40 2024-04-25T00:30:44.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.550+0000 7f45bb7fe640 1 --2- 172.21.15.116:0/3386151785 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f45c00678c0 0x7f45c0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:44.552 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.550+0000 7f45bb7fe640 1 -- 172.21.15.116:0/3386151785 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7f45c40871c0 con 0x7f45d406ee40 2024-04-25T00:30:44.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.550+0000 7f45bb7fe640 1 --2- 172.21.15.116:0/3386151785 >> [v2:172.21.15.116:6816/1099282267,v1:172.21.15.116:6817/1099282267] conn(0x7f45c00746c0 0x7f45c0076b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:44.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.550+0000 7f45bb7fe640 1 -- 172.21.15.116:0/3386151785 --> [v2:172.21.15.116:6816/1099282267,v1:172.21.15.116:6817/1099282267] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f45c0077230 con 0x7f45c00746c0 2024-04-25T00:30:44.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.550+0000 7f45bb7fe640 1 -- 172.21.15.116:0/3386151785 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=74) ==== 24+0+0 (secure 0 0 0) 0x7f45c4087610 con 0x7f45d406ee40 2024-04-25T00:30:44.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.550+0000 7f45d27fc640 1 --2- 172.21.15.116:0/3386151785 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f45c00678c0 0x7f45c0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:44.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.550+0000 7f45d37fe640 1 --2- 172.21.15.116:0/3386151785 >> [v2:172.21.15.116:6816/1099282267,v1:172.21.15.116:6817/1099282267] conn(0x7f45c00746c0 0x7f45c0076b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:44.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.550+0000 7f45d37fe640 1 --2- 172.21.15.116:0/3386151785 >> [v2:172.21.15.116:6816/1099282267,v1:172.21.15.116:6817/1099282267] conn(0x7f45c00746c0 0x7f45c0076b60 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:44.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.550+0000 7f45d27fc640 1 --2- 172.21.15.116:0/3386151785 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f45c00678c0 0x7f45c0069d80 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f45c80ae000 tx=0x7f45c8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:44.553 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.550+0000 7f45bb7fe640 1 -- 172.21.15.116:0/3386151785 <== osd.2 v2:172.21.15.116:6816/1099282267 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f45c0077230 con 0x7f45c00746c0 2024-04-25T00:30:44.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.562+0000 7f45d9322640 1 -- 172.21.15.116:0/3386151785 --> [v2:172.21.15.116:6816/1099282267,v1:172.21.15.116:6817/1099282267] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f45d415dc90 con 0x7f45c00746c0 2024-04-25T00:30:44.565 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.562+0000 7f45bb7fe640 1 -- 172.21.15.116:0/3386151785 <== osd.2 v2:172.21.15.116:6816/1099282267 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f45d415dc90 con 0x7f45c00746c0 2024-04-25T00:30:44.565 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-25T00:30:44.565 INFO:teuthology.orchestra.run.smithi116.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T00:30:44.565 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-25T00:30:44.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.566+0000 7f45d9322640 1 -- 172.21.15.116:0/3386151785 >> [v2:172.21.15.116:6816/1099282267,v1:172.21.15.116:6817/1099282267] conn(0x7f45c00746c0 msgr2=0x7f45c0076b60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:44.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.566+0000 7f45d9322640 1 --2- 172.21.15.116:0/3386151785 >> [v2:172.21.15.116:6816/1099282267,v1:172.21.15.116:6817/1099282267] conn(0x7f45c00746c0 0x7f45c0076b60 crc :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:44.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.566+0000 7f45d9322640 1 -- 172.21.15.116:0/3386151785 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f45c00678c0 msgr2=0x7f45c0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:44.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.566+0000 7f45d9322640 1 --2- 172.21.15.116:0/3386151785 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f45c00678c0 0x7f45c0069d80 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f45c80ae000 tx=0x7f45c8008040 comp rx=0 tx=0).stop 2024-04-25T00:30:44.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.566+0000 7f45d9322640 1 -- 172.21.15.116:0/3386151785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45d406ee40 msgr2=0x7f45d40c7470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:44.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.566+0000 7f45d9322640 1 --2- 172.21.15.116:0/3386151785 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f45d406ee40 0x7f45d40c7470 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f45c400e980 tx=0x7f45c400ee40 comp rx=0 tx=0).stop 2024-04-25T00:30:44.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.566+0000 7f45d37fe640 1 -- 172.21.15.116:0/3386151785 reap_dead start 2024-04-25T00:30:44.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.566+0000 7f45d9322640 1 -- 172.21.15.116:0/3386151785 shutdown_connections 2024-04-25T00:30:44.566 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.566+0000 7f45d9322640 1 -- 172.21.15.116:0/3386151785 >> 172.21.15.116:0/3386151785 conn(0x7f45d40c80a0 msgr2=0x7f45d4153750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:44.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.566+0000 7f45d9322640 1 -- 172.21.15.116:0/3386151785 shutdown_connections 2024-04-25T00:30:44.567 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.566+0000 7f45d9322640 1 -- 172.21.15.116:0/3386151785 wait complete. 2024-04-25T00:30:44.580 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-04-25T00:30:44.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.682+0000 7f49b2aa4640 1 Processor -- start 2024-04-25T00:30:44.682 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.682+0000 7f49b2aa4640 1 -- start start 2024-04-25T00:30:44.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.682+0000 7f49b2aa4640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f49ac0c73e0 0x7f49ac0728b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:44.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.682+0000 7f49b2aa4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f49ac0c29c0 0x7f49ac072df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:44.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.682+0000 7f49b2aa4640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49ac0723d0 con 0x7f49ac157870 2024-04-25T00:30:44.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.682+0000 7f49b2aa4640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f49ac157c50 con 0x7f49ac0c73e0 2024-04-25T00:30:44.684 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.682+0000 7f49b2aa4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f49ac073330 con 0x7f49ac0c29c0 2024-04-25T00:30:44.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.682+0000 7f49b0840640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f49ac157870 0x7f49ac06c220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:57464/0 (socket says 172.21.15.116:57464) 2024-04-25T00:30:44.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.682+0000 7f49b0840640 1 -- 172.21.15.116:0/2192522309 learned_addr learned my addr 172.21.15.116:0/2192522309 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:30:44.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.682+0000 7f49abfff640 1 --2- 172.21.15.116:0/2192522309 >> v2:172.21.15.174:3300/0 conn(0x7f49ac0c73e0 0x7f49ac0728b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:44.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.682+0000 7f49ab7fe640 1 -- 172.21.15.116:0/2192522309 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2003681381 0 0) 0x7f49ac0723d0 con 0x7f49ac157870 2024-04-25T00:30:44.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.682+0000 7f49b1041640 1 --2- 172.21.15.116:0/2192522309 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f49ac0c29c0 0x7f49ac072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:44.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.682+0000 7f49ab7fe640 1 -- 172.21.15.116:0/2192522309 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4994003530 con 0x7f49ac157870 2024-04-25T00:30:44.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.682+0000 7f49ab7fe640 1 -- 172.21.15.116:0/2192522309 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 39403250 0 0) 0x7f4994003530 con 0x7f49ac157870 2024-04-25T00:30:44.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.682+0000 7f49ab7fe640 1 -- 172.21.15.116:0/2192522309 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f49ac0c29c0 msgr2=0x7f49ac072df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:44.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.682+0000 7f49ab7fe640 1 --2- 172.21.15.116:0/2192522309 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f49ac0c29c0 0x7f49ac072df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:44.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.682+0000 7f49ab7fe640 1 -- 172.21.15.116:0/2192522309 >> v2:172.21.15.174:3300/0 conn(0x7f49ac0c73e0 msgr2=0x7f49ac0728b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:44.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.682+0000 7f49ab7fe640 1 --2- 172.21.15.116:0/2192522309 >> v2:172.21.15.174:3300/0 conn(0x7f49ac0c73e0 0x7f49ac0728b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:44.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.682+0000 7f49ab7fe640 1 -- 172.21.15.116:0/2192522309 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49ac156ce0 con 0x7f49ac157870 2024-04-25T00:30:44.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.682+0000 7f49ab7fe640 1 -- 172.21.15.116:0/2192522309 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f499c001a00 con 0x7f49ac157870 2024-04-25T00:30:44.685 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.682+0000 7f49b1041640 1 --2- 172.21.15.116:0/2192522309 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f49ac0c29c0 0x7f49ac072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:30:44.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.682+0000 7f49ab7fe640 1 -- 172.21.15.116:0/2192522309 >> v1:172.21.15.116:6789/0 conn(0x7f49ac157870 legacy=0x7f49ac06c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:44.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.682+0000 7f49ab7fe640 1 --2- 172.21.15.116:0/2192522309 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f49ac0c29c0 0x7f49940039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:44.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.682+0000 7f49ab7fe640 1 --2- 172.21.15.116:0/2192522309 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4994003f00 0x7f4994004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:44.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.682+0000 7f49ab7fe640 1 --2- 172.21.15.116:0/2192522309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4994004d50 0x7f49940051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:44.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.682+0000 7f49ab7fe640 1 -- 172.21.15.116:0/2192522309 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f49ac157c50 con 0x7f4994004d50 2024-04-25T00:30:44.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.682+0000 7f49ab7fe640 1 -- 172.21.15.116:0/2192522309 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f49ac073330 con 0x7f4994003f00 2024-04-25T00:30:44.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.682+0000 7f49ab7fe640 1 -- 172.21.15.116:0/2192522309 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f499400ba00 con 0x7f49ac0c29c0 2024-04-25T00:30:44.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.682+0000 7f49b1041640 1 --2- 172.21.15.116:0/2192522309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4994004d50 0x7f49940051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:44.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.682+0000 7f49b0840640 1 --2- 172.21.15.116:0/2192522309 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f49ac0c29c0 0x7f49940039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:44.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.682+0000 7f49b1041640 1 -- 172.21.15.116:0/2192522309 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f49ac0c29c0 msgr2=0x7f49940039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:44.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.682+0000 7f49b1041640 1 --2- 172.21.15.116:0/2192522309 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f49ac0c29c0 0x7f49940039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:44.686 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.682+0000 7f49b1041640 1 -- 172.21.15.116:0/2192522309 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4994003f00 msgr2=0x7f4994004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:44.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.682+0000 7f49b1041640 1 --2- 172.21.15.116:0/2192522309 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4994003f00 0x7f4994004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:44.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.682+0000 7f49b1041640 1 -- 172.21.15.116:0/2192522309 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4994003530 con 0x7f4994004d50 2024-04-25T00:30:44.687 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.682+0000 7f49b0840640 1 --2- 172.21.15.116:0/2192522309 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f49ac0c29c0 0x7f49940039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:30:44.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.686+0000 7f49b1041640 1 --2- 172.21.15.116:0/2192522309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4994004d50 0x7f49940051d0 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7f49a40098d0 tx=0x7f49a401c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfe9508f5dad8a06 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:44.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.686+0000 7f49ab7fe640 1 -- 172.21.15.116:0/2192522309 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a400f040 con 0x7f4994004d50 2024-04-25T00:30:44.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.686+0000 7f49ab7fe640 1 -- 172.21.15.116:0/2192522309 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49a40a8640 con 0x7f4994004d50 2024-04-25T00:30:44.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.686+0000 7f49ab7fe640 1 -- 172.21.15.116:0/2192522309 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a40a89f0 con 0x7f4994004d50 2024-04-25T00:30:44.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.686+0000 7f49b2aa4640 1 -- 172.21.15.116:0/2192522309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4994004d50 msgr2=0x7f49940051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:44.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.686+0000 7f49b2aa4640 1 --2- 172.21.15.116:0/2192522309 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4994004d50 0x7f49940051d0 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7f49a40098d0 tx=0x7f49a401c5d0 comp rx=0 tx=0).stop 2024-04-25T00:30:44.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.686+0000 7f49b1041640 1 -- 172.21.15.116:0/2192522309 reap_dead start 2024-04-25T00:30:44.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.686+0000 7f49b2aa4640 1 -- 172.21.15.116:0/2192522309 shutdown_connections 2024-04-25T00:30:44.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.686+0000 7f49b2aa4640 1 -- 172.21.15.116:0/2192522309 >> 172.21.15.116:0/2192522309 conn(0x7f49ac0c42e0 msgr2=0x7f49ac0c46f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:44.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.686+0000 7f49b2aa4640 1 -- 172.21.15.116:0/2192522309 shutdown_connections 2024-04-25T00:30:44.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.686+0000 7f49b2aa4640 1 -- 172.21.15.116:0/2192522309 wait complete. 2024-04-25T00:30:44.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.686+0000 7f49b2aa4640 1 Processor -- start 2024-04-25T00:30:44.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.686+0000 7f49b2aa4640 1 -- start start 2024-04-25T00:30:44.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.686+0000 7f49b2aa4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f49ac153eb0 0x7f49ac14df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:44.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.686+0000 7f49b2aa4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49ac1518b0 0x7f49ac14e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:44.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.686+0000 7f49b2aa4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f49ac14feb0 0x7f49ac150330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:44.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.686+0000 7f49b2aa4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f49ac074b80 con 0x7f49ac1518b0 2024-04-25T00:30:44.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.686+0000 7f49b2aa4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f49ac0748a0 con 0x7f49ac14feb0 2024-04-25T00:30:44.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.686+0000 7f49b2aa4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f49ac0c6910 con 0x7f49ac153eb0 2024-04-25T00:30:44.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.686+0000 7f49abfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49ac1518b0 0x7f49ac14e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:44.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.686+0000 7f49abfff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49ac1518b0 0x7f49ac14e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:36592/0 (socket says 172.21.15.116:36592) 2024-04-25T00:30:44.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.686+0000 7f49abfff640 1 -- 172.21.15.116:0/1954165743 learned_addr learned my addr 172.21.15.116:0/1954165743 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:44.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.686+0000 7f49b1041640 1 --2- 172.21.15.116:0/1954165743 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f49ac14feb0 0x7f49ac150330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:44.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.686+0000 7f49b0840640 1 --2- 172.21.15.116:0/1954165743 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f49ac153eb0 0x7f49ac14df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:44.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.686+0000 7f49abfff640 1 -- 172.21.15.116:0/1954165743 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f49ac153eb0 msgr2=0x7f49ac14df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:44.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.686+0000 7f49abfff640 1 --2- 172.21.15.116:0/1954165743 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f49ac153eb0 0x7f49ac14df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:44.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.686+0000 7f49abfff640 1 -- 172.21.15.116:0/1954165743 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f49ac14feb0 msgr2=0x7f49ac150330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:44.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.686+0000 7f49abfff640 1 --2- 172.21.15.116:0/1954165743 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f49ac14feb0 0x7f49ac150330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:44.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.686+0000 7f49abfff640 1 -- 172.21.15.116:0/1954165743 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49ac152710 con 0x7f49ac1518b0 2024-04-25T00:30:44.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.690+0000 7f49abfff640 1 --2- 172.21.15.116:0/1954165743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49ac1518b0 0x7f49ac14e480 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7f49a0009fd0 tx=0x7f49a000ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:44.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.690+0000 7f49a97fa640 1 -- 172.21.15.116:0/1954165743 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a0017070 con 0x7f49ac1518b0 2024-04-25T00:30:44.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.690+0000 7f49a97fa640 1 -- 172.21.15.116:0/1954165743 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49a000c940 con 0x7f49ac1518b0 2024-04-25T00:30:44.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.690+0000 7f49b2aa4640 1 -- 172.21.15.116:0/1954165743 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49ac0ac300 con 0x7f49ac1518b0 2024-04-25T00:30:44.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.690+0000 7f49a97fa640 1 -- 172.21.15.116:0/1954165743 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a000eea0 con 0x7f49ac1518b0 2024-04-25T00:30:44.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.690+0000 7f49b2aa4640 1 -- 172.21.15.116:0/1954165743 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49ac152430 con 0x7f49ac1518b0 2024-04-25T00:30:44.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.690+0000 7f49b2aa4640 1 -- 172.21.15.116:0/1954165743 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49ac14fad0 con 0x7f49ac1518b0 2024-04-25T00:30:44.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.690+0000 7f49a97fa640 1 -- 172.21.15.116:0/1954165743 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f49a001a070 con 0x7f49ac1518b0 2024-04-25T00:30:44.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.690+0000 7f49a97fa640 1 --2- 172.21.15.116:0/1954165743 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f49980678f0 0x7f4998069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:44.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.690+0000 7f49a97fa640 1 -- 172.21.15.116:0/1954165743 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7f49a0086eb0 con 0x7f49ac1518b0 2024-04-25T00:30:44.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.690+0000 7f49b0840640 1 --2- 172.21.15.116:0/1954165743 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f49980678f0 0x7f4998069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:44.693 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.690+0000 7f49b0840640 1 --2- 172.21.15.116:0/1954165743 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f49980678f0 0x7f4998069db0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f499c002240 tx=0x7f499c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:44.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.694+0000 7f49a97fa640 1 -- 172.21.15.116:0/1954165743 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f49ac152710 con 0x7f49ac1518b0 2024-04-25T00:30:44.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.842+0000 7f49b2aa4640 1 -- 172.21.15.116:0/1954165743 --> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f49ac0acd70 con 0x7f49980678f0 2024-04-25T00:30:44.844 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.842+0000 7f49a97fa640 1 -- 172.21.15.116:0/1954165743 <== mgr.4105 v2:172.21.15.174:6832/2004448211 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f49ac0acd70 con 0x7f49980678f0 2024-04-25T00:30:44.844 INFO:teuthology.orchestra.run.smithi116.stderr:instructed osd(s) 2 to deep-scrub 2024-04-25T00:30:44.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.846+0000 7f49b2aa4640 1 -- 172.21.15.116:0/1954165743 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f49980678f0 msgr2=0x7f4998069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:44.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.846+0000 7f49b2aa4640 1 --2- 172.21.15.116:0/1954165743 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f49980678f0 0x7f4998069db0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f499c002240 tx=0x7f499c009040 comp rx=0 tx=0).stop 2024-04-25T00:30:44.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.846+0000 7f49b2aa4640 1 -- 172.21.15.116:0/1954165743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49ac1518b0 msgr2=0x7f49ac14e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:44.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.846+0000 7f49b2aa4640 1 --2- 172.21.15.116:0/1954165743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49ac1518b0 0x7f49ac14e480 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7f49a0009fd0 tx=0x7f49a000ec50 comp rx=0 tx=0).stop 2024-04-25T00:30:44.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.846+0000 7f49b2aa4640 1 -- 172.21.15.116:0/1954165743 shutdown_connections 2024-04-25T00:30:44.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.846+0000 7f49b2aa4640 1 --2- 172.21.15.116:0/1954165743 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f49980678f0 0x7f4998069db0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:44.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.846+0000 7f49b2aa4640 1 --2- 172.21.15.116:0/1954165743 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f49ac14feb0 0x7f49ac150330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:44.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.846+0000 7f49b2aa4640 1 --2- 172.21.15.116:0/1954165743 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f49ac1518b0 0x7f49ac14e480 unknown :-1 s=CLOSED pgs=1820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:44.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.846+0000 7f49b2aa4640 1 --2- 172.21.15.116:0/1954165743 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f49ac153eb0 0x7f49ac14df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:44.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.846+0000 7f49b2aa4640 1 -- 172.21.15.116:0/1954165743 >> 172.21.15.116:0/1954165743 conn(0x7f49ac0c42e0 msgr2=0x7f49ac0c46f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:44.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.846+0000 7f49b2aa4640 1 -- 172.21.15.116:0/1954165743 shutdown_connections 2024-04-25T00:30:44.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.846+0000 7f49b2aa4640 1 -- 172.21.15.116:0/1954165743 wait complete. 2024-04-25T00:30:44.862 INFO:tasks.ceph:Scrubbing osd.3 2024-04-25T00:30:44.862 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-04-25T00:30:44.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.986+0000 7f5988bb9640 1 Processor -- start 2024-04-25T00:30:44.986 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.986+0000 7f5988bb9640 1 -- start start 2024-04-25T00:30:44.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.986+0000 7f5988bb9640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f598414a020 0x7f59840c24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:44.987 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.986+0000 7f5988bb9640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f5984153f10 0x7f59840abd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:44.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.986+0000 7f5988bb9640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59840ab2f0 con 0x7f59840ac2b0 2024-04-25T00:30:44.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.986+0000 7f5988bb9640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f598414a400 con 0x7f598414a020 2024-04-25T00:30:44.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.986+0000 7f5988bb9640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f5984154370 con 0x7f5984153f10 2024-04-25T00:30:44.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.986+0000 7f598259c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f598414a020 0x7f59840c24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:44.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.986+0000 7f598259c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f598414a020 0x7f59840c24c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40486/0 (socket says 172.21.15.116:40486) 2024-04-25T00:30:44.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.986+0000 7f598259c640 1 -- 172.21.15.116:0/960554937 learned_addr learned my addr 172.21.15.116:0/960554937 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:44.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.986+0000 7f5981d9b640 1 --2- 172.21.15.116:0/960554937 >> v2:172.21.15.174:3300/0 conn(0x7f5984153f10 0x7f59840abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:44.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.986+0000 7f598259c640 1 -- 172.21.15.116:0/960554937 >> v2:172.21.15.174:3300/0 conn(0x7f5984153f10 msgr2=0x7f59840abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:44.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.986+0000 7f598259c640 1 --2- 172.21.15.116:0/960554937 >> v2:172.21.15.174:3300/0 conn(0x7f5984153f10 0x7f59840abd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:44.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.986+0000 7f598259c640 1 -- 172.21.15.116:0/960554937 >> v1:172.21.15.174:6789/0 conn(0x7f59840ac2b0 legacy=0x7f59840a94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:44.988 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.986+0000 7f598259c640 1 -- 172.21.15.116:0/960554937 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59840c1ac0 con 0x7f598414a020 2024-04-25T00:30:44.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.990+0000 7f598259c640 1 --2- 172.21.15.116:0/960554937 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f598414a020 0x7f59840c24c0 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7f598406f490 tx=0x7f597401c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91c1453288493926 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:44.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.990+0000 7f598159a640 1 -- 172.21.15.116:0/960554937 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59740b0070 con 0x7f598414a020 2024-04-25T00:30:44.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.990+0000 7f598159a640 1 -- 172.21.15.116:0/960554937 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59740ab600 con 0x7f598414a020 2024-04-25T00:30:44.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.990+0000 7f598159a640 1 -- 172.21.15.116:0/960554937 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59740ab9d0 con 0x7f598414a020 2024-04-25T00:30:44.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.990+0000 7f5988bb9640 1 -- 172.21.15.116:0/960554937 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f598414a020 msgr2=0x7f59840c24c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:44.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.990+0000 7f5988bb9640 1 --2- 172.21.15.116:0/960554937 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f598414a020 0x7f59840c24c0 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7f598406f490 tx=0x7f597401c2d0 comp rx=0 tx=0).stop 2024-04-25T00:30:44.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.990+0000 7f5988bb9640 1 -- 172.21.15.116:0/960554937 shutdown_connections 2024-04-25T00:30:44.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.990+0000 7f5988bb9640 1 --2- 172.21.15.116:0/960554937 >> v2:172.21.15.174:3300/0 conn(0x7f5984153f10 0x7f59840abd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:44.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.990+0000 7f5988bb9640 1 --2- 172.21.15.116:0/960554937 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f598414a020 0x7f59840c24c0 unknown :-1 s=CLOSED pgs=1821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:44.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.990+0000 7f5988bb9640 1 -- 172.21.15.116:0/960554937 >> 172.21.15.116:0/960554937 conn(0x7f598406b810 msgr2=0x7f59840c6f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:44.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.990+0000 7f5988bb9640 1 -- 172.21.15.116:0/960554937 shutdown_connections 2024-04-25T00:30:44.992 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.990+0000 7f5988bb9640 1 -- 172.21.15.116:0/960554937 wait complete. 2024-04-25T00:30:44.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.990+0000 7f5988bb9640 1 Processor -- start 2024-04-25T00:30:44.993 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.994+0000 7f5988bb9640 1 -- start start 2024-04-25T00:30:44.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.994+0000 7f5988bb9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f598414a020 0x7f598415c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:44.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.994+0000 7f5988bb9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5984153f10 0x7f598413c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:44.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.994+0000 7f5988bb9640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f598413cd80 0x7f59841ee7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:44.995 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.994+0000 7f5988bb9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f59840c7ca0 con 0x7f598414a020 2024-04-25T00:30:44.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.994+0000 7f5988bb9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f59840c1560 con 0x7f5984153f10 2024-04-25T00:30:44.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.994+0000 7f5988bb9640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f59840a9bb0 con 0x7f598413cd80 2024-04-25T00:30:44.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.994+0000 7f598259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f598414a020 0x7f598415c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:44.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.994+0000 7f598259c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f598414a020 0x7f598415c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40498/0 (socket says 172.21.15.116:40498) 2024-04-25T00:30:44.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.994+0000 7f598259c640 1 -- 172.21.15.116:0/2318080469 learned_addr learned my addr 172.21.15.116:0/2318080469 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:44.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.994+0000 7f5981d9b640 1 --2- 172.21.15.116:0/2318080469 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5984153f10 0x7f598413c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:44.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.994+0000 7f598259c640 1 -- 172.21.15.116:0/2318080469 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f598413cd80 msgr2=0x7f59841ee7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:44.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.994+0000 7f5982d9d640 1 --2- 172.21.15.116:0/2318080469 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f598413cd80 0x7f59841ee7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:44.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.994+0000 7f598259c640 1 --2- 172.21.15.116:0/2318080469 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f598413cd80 0x7f59841ee7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:44.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.994+0000 7f598259c640 1 -- 172.21.15.116:0/2318080469 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5984153f10 msgr2=0x7f598413c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:44.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.994+0000 7f598259c640 1 --2- 172.21.15.116:0/2318080469 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5984153f10 0x7f598413c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:44.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.994+0000 7f598259c640 1 -- 172.21.15.116:0/2318080469 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59841f26c0 con 0x7f598414a020 2024-04-25T00:30:44.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.994+0000 7f5981d9b640 1 --2- 172.21.15.116:0/2318080469 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5984153f10 0x7f598413c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:30:44.996 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.994+0000 7f5982d9d640 1 --2- 172.21.15.116:0/2318080469 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f598413cd80 0x7f59841ee7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:30:44.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.994+0000 7f598259c640 1 --2- 172.21.15.116:0/2318080469 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f598414a020 0x7f598415c4a0 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f5974019f00 tx=0x7f5974019070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:44.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.994+0000 7f5972ffd640 1 -- 172.21.15.116:0/2318080469 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59740b0040 con 0x7f598414a020 2024-04-25T00:30:44.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.994+0000 7f5972ffd640 1 -- 172.21.15.116:0/2318080469 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59740b9070 con 0x7f598414a020 2024-04-25T00:30:44.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.994+0000 7f5988bb9640 1 -- 172.21.15.116:0/2318080469 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59841f29a0 con 0x7f598414a020 2024-04-25T00:30:44.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.994+0000 7f5972ffd640 1 -- 172.21.15.116:0/2318080469 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59740b4660 con 0x7f598414a020 2024-04-25T00:30:44.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.994+0000 7f5988bb9640 1 -- 172.21.15.116:0/2318080469 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f598415c9e0 con 0x7f598414a020 2024-04-25T00:30:44.997 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.994+0000 7f5970ff9640 1 -- 172.21.15.116:0/2318080469 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f595c001040 con 0x7f598414a020 2024-04-25T00:30:44.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.994+0000 7f5972ffd640 1 -- 172.21.15.116:0/2318080469 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f59740b4800 con 0x7f598414a020 2024-04-25T00:30:44.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.994+0000 7f5972ffd640 1 --2- 172.21.15.116:0/2318080469 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f59640678c0 0x7f5964069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:44.998 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.994+0000 7f5972ffd640 1 -- 172.21.15.116:0/2318080469 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7f5974120c00 con 0x7f598414a020 2024-04-25T00:30:44.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.994+0000 7f5972ffd640 1 --2- 172.21.15.116:0/2318080469 >> [v2:172.21.15.116:6824/1179942661,v1:172.21.15.116:6825/1179942661] conn(0x7f59640746c0 0x7f5964076b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:44.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.994+0000 7f5972ffd640 1 -- 172.21.15.116:0/2318080469 --> [v2:172.21.15.116:6824/1179942661,v1:172.21.15.116:6825/1179942661] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f59740ab600 con 0x7f59640746c0 2024-04-25T00:30:44.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.994+0000 7f5972ffd640 1 -- 172.21.15.116:0/2318080469 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=74) ==== 24+0+0 (secure 0 0 0) 0x7f59741262d0 con 0x7f598414a020 2024-04-25T00:30:44.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.994+0000 7f5981d9b640 1 --2- 172.21.15.116:0/2318080469 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f59640678c0 0x7f5964069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:44.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.994+0000 7f5982d9d640 1 --2- 172.21.15.116:0/2318080469 >> [v2:172.21.15.116:6824/1179942661,v1:172.21.15.116:6825/1179942661] conn(0x7f59640746c0 0x7f5964076b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:44.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.998+0000 7f5982d9d640 1 --2- 172.21.15.116:0/2318080469 >> [v2:172.21.15.116:6824/1179942661,v1:172.21.15.116:6825/1179942661] conn(0x7f59640746c0 0x7f5964076b60 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:44.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.998+0000 7f5981d9b640 1 --2- 172.21.15.116:0/2318080469 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f59640678c0 0x7f5964069d80 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f59840c4670 tx=0x7f5978008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:44.999 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:44.998+0000 7f5972ffd640 1 -- 172.21.15.116:0/2318080469 <== osd.3 v2:172.21.15.116:6824/1179942661 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f59740ab600 con 0x7f59640746c0 2024-04-25T00:30:45.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.010+0000 7f5970ff9640 1 -- 172.21.15.116:0/2318080469 --> [v2:172.21.15.116:6824/1179942661,v1:172.21.15.116:6825/1179942661] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f595c002e40 con 0x7f59640746c0 2024-04-25T00:30:45.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.010+0000 7f5972ffd640 1 -- 172.21.15.116:0/2318080469 <== osd.3 v2:172.21.15.116:6824/1179942661 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f595c002e40 con 0x7f59640746c0 2024-04-25T00:30:45.012 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-25T00:30:45.012 INFO:teuthology.orchestra.run.smithi116.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T00:30:45.012 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-25T00:30:45.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.010+0000 7f5970ff9640 1 -- 172.21.15.116:0/2318080469 >> [v2:172.21.15.116:6824/1179942661,v1:172.21.15.116:6825/1179942661] conn(0x7f59640746c0 msgr2=0x7f5964076b60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:45.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.010+0000 7f5970ff9640 1 --2- 172.21.15.116:0/2318080469 >> [v2:172.21.15.116:6824/1179942661,v1:172.21.15.116:6825/1179942661] conn(0x7f59640746c0 0x7f5964076b60 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:45.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.010+0000 7f5970ff9640 1 -- 172.21.15.116:0/2318080469 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f59640678c0 msgr2=0x7f5964069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:45.013 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.010+0000 7f5970ff9640 1 --2- 172.21.15.116:0/2318080469 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f59640678c0 0x7f5964069d80 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f59840c4670 tx=0x7f5978008040 comp rx=0 tx=0).stop 2024-04-25T00:30:45.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.010+0000 7f5970ff9640 1 -- 172.21.15.116:0/2318080469 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f598414a020 msgr2=0x7f598415c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:45.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.010+0000 7f5970ff9640 1 --2- 172.21.15.116:0/2318080469 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f598414a020 0x7f598415c4a0 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f5974019f00 tx=0x7f5974019070 comp rx=0 tx=0).stop 2024-04-25T00:30:45.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.010+0000 7f5982d9d640 1 -- 172.21.15.116:0/2318080469 reap_dead start 2024-04-25T00:30:45.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.010+0000 7f5970ff9640 1 -- 172.21.15.116:0/2318080469 shutdown_connections 2024-04-25T00:30:45.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.010+0000 7f5970ff9640 1 -- 172.21.15.116:0/2318080469 >> 172.21.15.116:0/2318080469 conn(0x7f598406b810 msgr2=0x7f59840c24c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:45.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.010+0000 7f5970ff9640 1 -- 172.21.15.116:0/2318080469 shutdown_connections 2024-04-25T00:30:45.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.010+0000 7f5970ff9640 1 -- 172.21.15.116:0/2318080469 wait complete. 2024-04-25T00:30:45.028 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-04-25T00:30:45.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.142+0000 7f8b15a33640 1 Processor -- start 2024-04-25T00:30:45.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.142+0000 7f8b15a33640 1 -- start start 2024-04-25T00:30:45.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.142+0000 7f8b15a33640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f8b100a94a0 0x7f8b101563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:45.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.142+0000 7f8b15a33640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f8b10071680 0x7f8b10071ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:45.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.142+0000 7f8b15a33640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b10071220 con 0x7f8b1014c720 2024-04-25T00:30:45.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.142+0000 7f8b15a33640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f8b10153460 con 0x7f8b100a94a0 2024-04-25T00:30:45.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.142+0000 7f8b15a33640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f8b10152420 con 0x7f8b10071680 2024-04-25T00:30:45.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.142+0000 7f8b0effd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f8b1014c720 0x7f8b100c2a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:43666/0 (socket says 172.21.15.116:43666) 2024-04-25T00:30:45.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.142+0000 7f8b0effd640 1 -- 172.21.15.116:0/3569086526 learned_addr learned my addr 172.21.15.116:0/3569086526 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:30:45.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.142+0000 7f8b07fff640 1 --2- 172.21.15.116:0/3569086526 >> v2:172.21.15.116:3300/0 conn(0x7f8b100a94a0 0x7f8b101563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:45.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.142+0000 7f8b0f7fe640 1 --2- 172.21.15.116:0/3569086526 >> v2:172.21.15.174:3300/0 conn(0x7f8b10071680 0x7f8b10071ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:45.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.142+0000 7f8b0e7fc640 1 -- 172.21.15.116:0/3569086526 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3322347301 0 0) 0x7f8b10071220 con 0x7f8b1014c720 2024-04-25T00:30:45.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.142+0000 7f8b0e7fc640 1 -- 172.21.15.116:0/3569086526 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8afc005740 con 0x7f8b1014c720 2024-04-25T00:30:45.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.142+0000 7f8b07fff640 1 -- 172.21.15.116:0/3569086526 >> v2:172.21.15.174:3300/0 conn(0x7f8b10071680 msgr2=0x7f8b10071ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:45.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.142+0000 7f8b07fff640 1 --2- 172.21.15.116:0/3569086526 >> v2:172.21.15.174:3300/0 conn(0x7f8b10071680 0x7f8b10071ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:45.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.142+0000 7f8b07fff640 1 -- 172.21.15.116:0/3569086526 >> v1:172.21.15.116:6789/0 conn(0x7f8b1014c720 legacy=0x7f8b100c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:45.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.142+0000 7f8b07fff640 1 -- 172.21.15.116:0/3569086526 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b1014cb00 con 0x7f8b100a94a0 2024-04-25T00:30:45.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.146+0000 7f8b07fff640 1 --2- 172.21.15.116:0/3569086526 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b100a94a0 0x7f8b101563d0 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f8b100ad370 tx=0x7f8af801c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95a98fb8879159e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:45.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.146+0000 7f8b0e7fc640 1 -- 172.21.15.116:0/3569086526 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8af80b0070 con 0x7f8b100a94a0 2024-04-25T00:30:45.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.146+0000 7f8b0e7fc640 1 -- 172.21.15.116:0/3569086526 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8af80ab600 con 0x7f8b100a94a0 2024-04-25T00:30:45.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.146+0000 7f8b0e7fc640 1 -- 172.21.15.116:0/3569086526 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8af80ab9d0 con 0x7f8b100a94a0 2024-04-25T00:30:45.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.146+0000 7f8b15a33640 1 -- 172.21.15.116:0/3569086526 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b100a94a0 msgr2=0x7f8b101563d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:45.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.146+0000 7f8b15a33640 1 --2- 172.21.15.116:0/3569086526 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b100a94a0 0x7f8b101563d0 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f8b100ad370 tx=0x7f8af801c3c0 comp rx=0 tx=0).stop 2024-04-25T00:30:45.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.146+0000 7f8b15a33640 1 -- 172.21.15.116:0/3569086526 shutdown_connections 2024-04-25T00:30:45.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.146+0000 7f8b15a33640 1 --2- 172.21.15.116:0/3569086526 >> v2:172.21.15.174:3300/0 conn(0x7f8b10071680 0x7f8b10071ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:45.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.146+0000 7f8b15a33640 1 --2- 172.21.15.116:0/3569086526 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b100a94a0 0x7f8b101563d0 unknown :-1 s=CLOSED pgs=1824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:45.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.146+0000 7f8b15a33640 1 -- 172.21.15.116:0/3569086526 >> 172.21.15.116:0/3569086526 conn(0x7f8b100c3b00 msgr2=0x7f8b100c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:45.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.146+0000 7f8b15a33640 1 -- 172.21.15.116:0/3569086526 shutdown_connections 2024-04-25T00:30:45.148 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.146+0000 7f8b15a33640 1 -- 172.21.15.116:0/3569086526 wait complete. 2024-04-25T00:30:45.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.146+0000 7f8b15a33640 1 Processor -- start 2024-04-25T00:30:45.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.146+0000 7f8b15a33640 1 -- start start 2024-04-25T00:30:45.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.150+0000 7f8b15a33640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b10071680 0x7f8b10154b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:45.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.150+0000 7f8b15a33640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8b100a94a0 0x7f8b101ed6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:45.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.150+0000 7f8b15a33640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8b1014c720 0x7f8b101edbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:45.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.150+0000 7f8b15a33640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f8b100c3120 con 0x7f8b10071680 2024-04-25T00:30:45.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.150+0000 7f8b15a33640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f8b10152f40 con 0x7f8b1014c720 2024-04-25T00:30:45.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.150+0000 7f8b15a33640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f8b100c32a0 con 0x7f8b100a94a0 2024-04-25T00:30:45.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.150+0000 7f8b0effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b10071680 0x7f8b10154b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:45.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.150+0000 7f8b0effd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b10071680 0x7f8b10154b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40522/0 (socket says 172.21.15.116:40522) 2024-04-25T00:30:45.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.150+0000 7f8b0effd640 1 -- 172.21.15.116:0/2380535651 learned_addr learned my addr 172.21.15.116:0/2380535651 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:45.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.150+0000 7f8b0f7fe640 1 --2- 172.21.15.116:0/2380535651 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8b1014c720 0x7f8b101edbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:45.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.150+0000 7f8b0effd640 1 -- 172.21.15.116:0/2380535651 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8b100a94a0 msgr2=0x7f8b101ed6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:45.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.150+0000 7f8b0effd640 1 --2- 172.21.15.116:0/2380535651 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8b100a94a0 0x7f8b101ed6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:45.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.150+0000 7f8b0effd640 1 -- 172.21.15.116:0/2380535651 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8b1014c720 msgr2=0x7f8b101edbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:45.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.150+0000 7f8b0effd640 1 --2- 172.21.15.116:0/2380535651 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8b1014c720 0x7f8b101edbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:45.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.150+0000 7f8b0effd640 1 -- 172.21.15.116:0/2380535651 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b100ab8f0 con 0x7f8b10071680 2024-04-25T00:30:45.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.150+0000 7f8b0effd640 1 --2- 172.21.15.116:0/2380535651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b10071680 0x7f8b10154b20 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f8b0000c3f0 tx=0x7f8b0000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:45.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.150+0000 7f8b0cff9640 1 -- 172.21.15.116:0/2380535651 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b00017070 con 0x7f8b10071680 2024-04-25T00:30:45.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.150+0000 7f8b0cff9640 1 -- 172.21.15.116:0/2380535651 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b00005cf0 con 0x7f8b10071680 2024-04-25T00:30:45.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.150+0000 7f8b15a33640 1 -- 172.21.15.116:0/2380535651 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b1014a7d0 con 0x7f8b10071680 2024-04-25T00:30:45.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.150+0000 7f8b0cff9640 1 -- 172.21.15.116:0/2380535651 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b000072f0 con 0x7f8b10071680 2024-04-25T00:30:45.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.150+0000 7f8b15a33640 1 -- 172.21.15.116:0/2380535651 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b1014a3b0 con 0x7f8b10071680 2024-04-25T00:30:45.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.150+0000 7f8b0cff9640 1 -- 172.21.15.116:0/2380535651 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8b0001a070 con 0x7f8b10071680 2024-04-25T00:30:45.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.150+0000 7f8b15a33640 1 -- 172.21.15.116:0/2380535651 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b10068350 con 0x7f8b10071680 2024-04-25T00:30:45.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.150+0000 7f8b0cff9640 1 --2- 172.21.15.116:0/2380535651 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8af00678c0 0x7f8af0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:45.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.150+0000 7f8b0cff9640 1 -- 172.21.15.116:0/2380535651 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7f8b000863f0 con 0x7f8b10071680 2024-04-25T00:30:45.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.150+0000 7f8b07fff640 1 --2- 172.21.15.116:0/2380535651 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8af00678c0 0x7f8af0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:45.154 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.154+0000 7f8b07fff640 1 --2- 172.21.15.116:0/2380535651 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8af00678c0 0x7f8af0069d80 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f8af80ad000 tx=0x7f8af8004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:45.156 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.154+0000 7f8b0cff9640 1 -- 172.21.15.116:0/2380535651 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f8b00055c40 con 0x7f8b10071680 2024-04-25T00:30:45.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.310+0000 7f8b15a33640 1 -- 172.21.15.116:0/2380535651 --> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f8b101ed200 con 0x7f8af00678c0 2024-04-25T00:30:45.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.314+0000 7f8b0cff9640 1 -- 172.21.15.116:0/2380535651 <== mgr.4105 v2:172.21.15.174:6832/2004448211 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f8b101ed200 con 0x7f8af00678c0 2024-04-25T00:30:45.313 INFO:teuthology.orchestra.run.smithi116.stderr:instructed osd(s) 3 to deep-scrub 2024-04-25T00:30:45.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.314+0000 7f8b15a33640 1 -- 172.21.15.116:0/2380535651 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8af00678c0 msgr2=0x7f8af0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:45.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.314+0000 7f8b15a33640 1 --2- 172.21.15.116:0/2380535651 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8af00678c0 0x7f8af0069d80 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f8af80ad000 tx=0x7f8af8004740 comp rx=0 tx=0).stop 2024-04-25T00:30:45.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.314+0000 7f8b15a33640 1 -- 172.21.15.116:0/2380535651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b10071680 msgr2=0x7f8b10154b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:45.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.314+0000 7f8b15a33640 1 --2- 172.21.15.116:0/2380535651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b10071680 0x7f8b10154b20 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7f8b0000c3f0 tx=0x7f8b0000c8b0 comp rx=0 tx=0).stop 2024-04-25T00:30:45.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.314+0000 7f8b15a33640 1 -- 172.21.15.116:0/2380535651 shutdown_connections 2024-04-25T00:30:45.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.314+0000 7f8b15a33640 1 --2- 172.21.15.116:0/2380535651 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f8af00678c0 0x7f8af0069d80 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:45.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.314+0000 7f8b15a33640 1 --2- 172.21.15.116:0/2380535651 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f8b1014c720 0x7f8b101edbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:45.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.314+0000 7f8b15a33640 1 --2- 172.21.15.116:0/2380535651 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f8b100a94a0 0x7f8b101ed6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:45.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.314+0000 7f8b15a33640 1 --2- 172.21.15.116:0/2380535651 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f8b10071680 0x7f8b10154b20 unknown :-1 s=CLOSED pgs=1825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:45.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.314+0000 7f8b15a33640 1 -- 172.21.15.116:0/2380535651 >> 172.21.15.116:0/2380535651 conn(0x7f8b100c3b00 msgr2=0x7f8b100c2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:45.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.314+0000 7f8b15a33640 1 -- 172.21.15.116:0/2380535651 shutdown_connections 2024-04-25T00:30:45.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.314+0000 7f8b15a33640 1 -- 172.21.15.116:0/2380535651 wait complete. 2024-04-25T00:30:45.332 INFO:tasks.ceph:Scrubbing osd.4 2024-04-25T00:30:45.332 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-04-25T00:30:45.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.454+0000 7f223b550640 1 Processor -- start 2024-04-25T00:30:45.454 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.454+0000 7f223b550640 1 -- start start 2024-04-25T00:30:45.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.454+0000 7f223b550640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f22340c0350 0x7f22340c0730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:45.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.454+0000 7f223b550640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f22340c1920 0x7f22340c2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:45.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.454+0000 7f223b550640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22340c6b70 con 0x7f22340c1dd0 2024-04-25T00:30:45.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.454+0000 7f223b550640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f22340c6920 con 0x7f22340c0350 2024-04-25T00:30:45.457 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.454+0000 7f223b550640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f22340c6260 con 0x7f22340c1920 2024-04-25T00:30:45.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.454+0000 7f2238aeb640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f22340c1dd0 0x7f22340c2600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:43676/0 (socket says 172.21.15.116:43676) 2024-04-25T00:30:45.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.454+0000 7f2238aeb640 1 -- 172.21.15.116:0/2932330181 learned_addr learned my addr 172.21.15.116:0/2932330181 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:30:45.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.454+0000 7f222bfff640 1 -- 172.21.15.116:0/2932330181 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1632486535 0 0) 0x7f22340c6b70 con 0x7f22340c1dd0 2024-04-25T00:30:45.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.454+0000 7f222bfff640 1 -- 172.21.15.116:0/2932330181 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2220003530 con 0x7f22340c1dd0 2024-04-25T00:30:45.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.454+0000 7f2239aed640 1 --2- 172.21.15.116:0/2932330181 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f22340c1920 0x7f22340c2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:45.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.454+0000 7f22392ec640 1 --2- 172.21.15.116:0/2932330181 >> v2:172.21.15.174:3300/0 conn(0x7f22340c0350 0x7f22340c0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:45.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.454+0000 7f222bfff640 1 -- 172.21.15.116:0/2932330181 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4293483798 0 0) 0x7f2220003530 con 0x7f22340c1dd0 2024-04-25T00:30:45.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.454+0000 7f222bfff640 1 -- 172.21.15.116:0/2932330181 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f22340c1920 msgr2=0x7f22340c2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:45.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.454+0000 7f222bfff640 1 --2- 172.21.15.116:0/2932330181 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f22340c1920 0x7f22340c2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:45.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.454+0000 7f222bfff640 1 -- 172.21.15.116:0/2932330181 >> v2:172.21.15.174:3300/0 conn(0x7f22340c0350 msgr2=0x7f22340c0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:45.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.454+0000 7f222bfff640 1 --2- 172.21.15.116:0/2932330181 >> v2:172.21.15.174:3300/0 conn(0x7f22340c0350 0x7f22340c0730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:45.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.454+0000 7f222bfff640 1 -- 172.21.15.116:0/2932330181 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22340c6540 con 0x7f22340c1dd0 2024-04-25T00:30:45.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.454+0000 7f222bfff640 1 -- 172.21.15.116:0/2932330181 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f2224001680 con 0x7f22340c1dd0 2024-04-25T00:30:45.458 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.454+0000 7f2239aed640 1 --2- 172.21.15.116:0/2932330181 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f22340c1920 0x7f22340c2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:30:45.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.454+0000 7f222bfff640 1 -- 172.21.15.116:0/2932330181 >> v1:172.21.15.116:6789/0 conn(0x7f22340c1dd0 legacy=0x7f22340c2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:45.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.454+0000 7f222bfff640 1 --2- 172.21.15.116:0/2932330181 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f22340c1920 0x7f22200039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:45.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.454+0000 7f222bfff640 1 --2- 172.21.15.116:0/2932330181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2220003f00 0x7f2220004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:45.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.454+0000 7f222bfff640 1 --2- 172.21.15.116:0/2932330181 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2220004d50 0x7f22200051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:45.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.454+0000 7f222bfff640 1 -- 172.21.15.116:0/2932330181 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f22340c6920 con 0x7f2220003f00 2024-04-25T00:30:45.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.454+0000 7f222bfff640 1 -- 172.21.15.116:0/2932330181 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f22340c6260 con 0x7f2220004d50 2024-04-25T00:30:45.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.454+0000 7f222bfff640 1 -- 172.21.15.116:0/2932330181 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f222000ba80 con 0x7f22340c1920 2024-04-25T00:30:45.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.454+0000 7f2238aeb640 1 --2- 172.21.15.116:0/2932330181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2220003f00 0x7f2220004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:45.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.454+0000 7f22392ec640 1 --2- 172.21.15.116:0/2932330181 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f22340c1920 0x7f22200039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:45.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.454+0000 7f2239aed640 1 --2- 172.21.15.116:0/2932330181 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2220004d50 0x7f22200051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:45.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.454+0000 7f2238aeb640 1 -- 172.21.15.116:0/2932330181 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f22340c1920 msgr2=0x7f22200039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:45.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.454+0000 7f2238aeb640 1 --2- 172.21.15.116:0/2932330181 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f22340c1920 0x7f22200039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:45.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.454+0000 7f2238aeb640 1 -- 172.21.15.116:0/2932330181 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2220004d50 msgr2=0x7f22200051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:45.459 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.454+0000 7f2238aeb640 1 --2- 172.21.15.116:0/2932330181 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2220004d50 0x7f22200051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:45.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.454+0000 7f2238aeb640 1 -- 172.21.15.116:0/2932330181 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2220003530 con 0x7f2220003f00 2024-04-25T00:30:45.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.454+0000 7f2239aed640 1 --2- 172.21.15.116:0/2932330181 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2220004d50 0x7f22200051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:30:45.460 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.454+0000 7f22392ec640 1 --2- 172.21.15.116:0/2932330181 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f22340c1920 0x7f22200039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:30:45.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.458+0000 7f2238aeb640 1 --2- 172.21.15.116:0/2932330181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2220003f00 0x7f2220004810 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f222400c240 tx=0x7f222401cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f9192965e182782 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:45.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.458+0000 7f222bfff640 1 -- 172.21.15.116:0/2932330181 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2224007d60 con 0x7f2220003f00 2024-04-25T00:30:45.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.458+0000 7f222bfff640 1 -- 172.21.15.116:0/2932330181 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22240a9020 con 0x7f2220003f00 2024-04-25T00:30:45.461 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.458+0000 7f222bfff640 1 -- 172.21.15.116:0/2932330181 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2224005b00 con 0x7f2220003f00 2024-04-25T00:30:45.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.458+0000 7f223b550640 1 -- 172.21.15.116:0/2932330181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2220003f00 msgr2=0x7f2220004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:45.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.458+0000 7f223b550640 1 --2- 172.21.15.116:0/2932330181 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2220003f00 0x7f2220004810 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f222400c240 tx=0x7f222401cc60 comp rx=0 tx=0).stop 2024-04-25T00:30:45.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.458+0000 7f2239aed640 1 -- 172.21.15.116:0/2932330181 reap_dead start 2024-04-25T00:30:45.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.458+0000 7f223b550640 1 -- 172.21.15.116:0/2932330181 shutdown_connections 2024-04-25T00:30:45.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.458+0000 7f223b550640 1 -- 172.21.15.116:0/2932330181 >> 172.21.15.116:0/2932330181 conn(0x7f223406b810 msgr2=0x7f22340c6f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:45.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.458+0000 7f223b550640 1 -- 172.21.15.116:0/2932330181 shutdown_connections 2024-04-25T00:30:45.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.458+0000 7f223b550640 1 -- 172.21.15.116:0/2932330181 wait complete. 2024-04-25T00:30:45.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.458+0000 7f223b550640 1 Processor -- start 2024-04-25T00:30:45.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.458+0000 7f223b550640 1 -- start start 2024-04-25T00:30:45.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.462+0000 7f223b550640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2234155d80 0x7f22341eed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:45.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.462+0000 7f223b550640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22341534f0 0x7f22341ef280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:45.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.462+0000 7f223b550640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f223415ba30 0x7f22341f22b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:45.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.462+0000 7f223b550640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2234160ba0 con 0x7f22341534f0 2024-04-25T00:30:45.462 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.462+0000 7f223b550640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2234160a20 con 0x7f223415ba30 2024-04-25T00:30:45.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.462+0000 7f223b550640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f2234160d20 con 0x7f2234155d80 2024-04-25T00:30:45.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.462+0000 7f2238aeb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22341534f0 0x7f22341ef280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:45.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.462+0000 7f2238aeb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22341534f0 0x7f22341ef280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40538/0 (socket says 172.21.15.116:40538) 2024-04-25T00:30:45.463 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.462+0000 7f2238aeb640 1 -- 172.21.15.116:0/1327683041 learned_addr learned my addr 172.21.15.116:0/1327683041 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:45.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.462+0000 7f22392ec640 1 --2- 172.21.15.116:0/1327683041 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2234155d80 0x7f22341eed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:45.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.462+0000 7f2239aed640 1 --2- 172.21.15.116:0/1327683041 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f223415ba30 0x7f22341f22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:45.466 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.462+0000 7f2238aeb640 1 -- 172.21.15.116:0/1327683041 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2234155d80 msgr2=0x7f22341eed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:45.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.462+0000 7f2238aeb640 1 --2- 172.21.15.116:0/1327683041 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2234155d80 0x7f22341eed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:45.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.462+0000 7f2238aeb640 1 -- 172.21.15.116:0/1327683041 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f223415ba30 msgr2=0x7f22341f22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:45.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.462+0000 7f2238aeb640 1 --2- 172.21.15.116:0/1327683041 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f223415ba30 0x7f22341f22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:45.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.462+0000 7f2238aeb640 1 -- 172.21.15.116:0/1327683041 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22341540e0 con 0x7f22341534f0 2024-04-25T00:30:45.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.462+0000 7f2239aed640 1 --2- 172.21.15.116:0/1327683041 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f223415ba30 0x7f22341f22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:30:45.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.462+0000 7f2238aeb640 1 --2- 172.21.15.116:0/1327683041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22341534f0 0x7f22341ef280 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f22240ab040 tx=0x7f2220005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:45.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.462+0000 7f2229ffb640 1 -- 172.21.15.116:0/1327683041 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2224007d60 con 0x7f22341534f0 2024-04-25T00:30:45.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.462+0000 7f2229ffb640 1 -- 172.21.15.116:0/1327683041 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22240b2030 con 0x7f22341534f0 2024-04-25T00:30:45.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.462+0000 7f223b550640 1 -- 172.21.15.116:0/1327683041 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22341543c0 con 0x7f22341534f0 2024-04-25T00:30:45.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.462+0000 7f2229ffb640 1 -- 172.21.15.116:0/1327683041 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2224001ca0 con 0x7f22341534f0 2024-04-25T00:30:45.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.462+0000 7f223b550640 1 -- 172.21.15.116:0/1327683041 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22340aab60 con 0x7f22341534f0 2024-04-25T00:30:45.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.462+0000 7f223b550640 1 -- 172.21.15.116:0/1327683041 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2234153e90 con 0x7f22341534f0 2024-04-25T00:30:45.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.462+0000 7f2229ffb640 1 -- 172.21.15.116:0/1327683041 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f22240a5850 con 0x7f22341534f0 2024-04-25T00:30:45.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.462+0000 7f2229ffb640 1 --2- 172.21.15.116:0/1327683041 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f22180678f0 0x7f2218069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:45.467 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.462+0000 7f2229ffb640 1 -- 172.21.15.116:0/1327683041 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7f2224120200 con 0x7f22341534f0 2024-04-25T00:30:45.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.462+0000 7f2229ffb640 1 --2- 172.21.15.116:0/1327683041 >> [v2:172.21.15.174:6800/3132444330,v1:172.21.15.174:6801/3132444330] conn(0x7f22180746e0 0x7f2218076b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:45.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.462+0000 7f2229ffb640 1 -- 172.21.15.116:0/1327683041 --> [v2:172.21.15.174:6800/3132444330,v1:172.21.15.174:6801/3132444330] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f22240b6e50 con 0x7f22180746e0 2024-04-25T00:30:45.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.462+0000 7f2229ffb640 1 -- 172.21.15.116:0/1327683041 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=74) ==== 24+0+0 (secure 0 0 0) 0x7f2224127050 con 0x7f22341534f0 2024-04-25T00:30:45.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.462+0000 7f22392ec640 1 --2- 172.21.15.116:0/1327683041 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f22180678f0 0x7f2218069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:45.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.462+0000 7f2239aed640 1 --2- 172.21.15.116:0/1327683041 >> [v2:172.21.15.174:6800/3132444330,v1:172.21.15.174:6801/3132444330] conn(0x7f22180746e0 0x7f2218076b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:45.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.466+0000 7f22392ec640 1 --2- 172.21.15.116:0/1327683041 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f22180678f0 0x7f2218069db0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f2234151db0 tx=0x7f2230008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:45.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.466+0000 7f2239aed640 1 --2- 172.21.15.116:0/1327683041 >> [v2:172.21.15.174:6800/3132444330,v1:172.21.15.174:6801/3132444330] conn(0x7f22180746e0 0x7f2218076b80 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:45.468 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.466+0000 7f2229ffb640 1 -- 172.21.15.116:0/1327683041 <== osd.4 v2:172.21.15.174:6800/3132444330 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f22240b6e50 con 0x7f22180746e0 2024-04-25T00:30:45.482 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.482+0000 7f223b550640 1 -- 172.21.15.116:0/1327683041 --> [v2:172.21.15.174:6800/3132444330,v1:172.21.15.174:6801/3132444330] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f22340ab080 con 0x7f22180746e0 2024-04-25T00:30:45.483 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.482+0000 7f2229ffb640 1 -- 172.21.15.116:0/1327683041 <== osd.4 v2:172.21.15.174:6800/3132444330 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f222c009770 con 0x7f22180746e0 2024-04-25T00:30:45.483 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-25T00:30:45.483 INFO:teuthology.orchestra.run.smithi116.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T00:30:45.483 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-25T00:30:45.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.482+0000 7f223b550640 1 -- 172.21.15.116:0/1327683041 >> [v2:172.21.15.174:6800/3132444330,v1:172.21.15.174:6801/3132444330] conn(0x7f22180746e0 msgr2=0x7f2218076b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:45.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.482+0000 7f223b550640 1 --2- 172.21.15.116:0/1327683041 >> [v2:172.21.15.174:6800/3132444330,v1:172.21.15.174:6801/3132444330] conn(0x7f22180746e0 0x7f2218076b80 crc :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:45.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.482+0000 7f223b550640 1 -- 172.21.15.116:0/1327683041 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f22180678f0 msgr2=0x7f2218069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:45.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.482+0000 7f223b550640 1 --2- 172.21.15.116:0/1327683041 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f22180678f0 0x7f2218069db0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f2234151db0 tx=0x7f2230008040 comp rx=0 tx=0).stop 2024-04-25T00:30:45.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.482+0000 7f223b550640 1 -- 172.21.15.116:0/1327683041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22341534f0 msgr2=0x7f22341ef280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:45.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.482+0000 7f223b550640 1 --2- 172.21.15.116:0/1327683041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f22341534f0 0x7f22341ef280 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f22240ab040 tx=0x7f2220005840 comp rx=0 tx=0).stop 2024-04-25T00:30:45.484 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.482+0000 7f2239aed640 1 -- 172.21.15.116:0/1327683041 reap_dead start 2024-04-25T00:30:45.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.482+0000 7f223b550640 1 -- 172.21.15.116:0/1327683041 shutdown_connections 2024-04-25T00:30:45.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.482+0000 7f223b550640 1 -- 172.21.15.116:0/1327683041 >> 172.21.15.116:0/1327683041 conn(0x7f223406b810 msgr2=0x7f2234150f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:45.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.482+0000 7f223b550640 1 -- 172.21.15.116:0/1327683041 shutdown_connections 2024-04-25T00:30:45.485 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.482+0000 7f223b550640 1 -- 172.21.15.116:0/1327683041 wait complete. 2024-04-25T00:30:45.499 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-04-25T00:30:45.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.598+0000 7f101f13a640 1 Processor -- start 2024-04-25T00:30:45.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.598+0000 7f101f13a640 1 -- start start 2024-04-25T00:30:45.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.602+0000 7f101f13a640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f10180a94a0 0x7f10181538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:45.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.602+0000 7f101f13a640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10180c3910 con 0x7f10180a99f0 2024-04-25T00:30:45.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.602+0000 7f101f13a640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1018154530 con 0x7f1018150d20 2024-04-25T00:30:45.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.602+0000 7f101f13a640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f10180c16e0 con 0x7f10180a94a0 2024-04-25T00:30:45.602 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.602+0000 7f1017fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f10180a94a0 0x7f10181538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:45.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.602+0000 7f101d6d7640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f10180a99f0 0x7f1018153e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:43688/0 (socket says 172.21.15.116:43688) 2024-04-25T00:30:45.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.602+0000 7f101d6d7640 1 -- 172.21.15.116:0/807536817 learned_addr learned my addr 172.21.15.116:0/807536817 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:30:45.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.602+0000 7f10177fe640 1 -- 172.21.15.116:0/807536817 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 10742030 0 0) 0x7f10180c3910 con 0x7f10180a99f0 2024-04-25T00:30:45.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.602+0000 7f10177fe640 1 -- 172.21.15.116:0/807536817 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1000003530 con 0x7f10180a99f0 2024-04-25T00:30:45.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.602+0000 7f1017fff640 1 -- 172.21.15.116:0/807536817 >> v1:172.21.15.174:6789/0 conn(0x7f1018150d20 legacy=0x7f10180c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:45.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.602+0000 7f1017fff640 1 -- 172.21.15.116:0/807536817 >> v1:172.21.15.116:6789/0 conn(0x7f10180a99f0 legacy=0x7f1018153e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:45.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.602+0000 7f1017fff640 1 -- 172.21.15.116:0/807536817 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10180c1400 con 0x7f10180a94a0 2024-04-25T00:30:45.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.602+0000 7f1017fff640 1 --2- 172.21.15.116:0/807536817 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10180a94a0 0x7f10181538e0 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f10080097c0 tx=0x7f100801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70f68ae2d0580abc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:45.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.602+0000 7f10177fe640 1 -- 172.21.15.116:0/807536817 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10080b0070 con 0x7f10180a94a0 2024-04-25T00:30:45.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.602+0000 7f10177fe640 1 -- 172.21.15.116:0/807536817 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10080ab600 con 0x7f10180a94a0 2024-04-25T00:30:45.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.602+0000 7f10177fe640 1 -- 172.21.15.116:0/807536817 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10080ab9d0 con 0x7f10180a94a0 2024-04-25T00:30:45.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.602+0000 7f101f13a640 1 -- 172.21.15.116:0/807536817 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10180a94a0 msgr2=0x7f10181538e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:45.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.602+0000 7f101f13a640 1 --2- 172.21.15.116:0/807536817 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10180a94a0 0x7f10181538e0 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f10080097c0 tx=0x7f100801c460 comp rx=0 tx=0).stop 2024-04-25T00:30:45.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.602+0000 7f101f13a640 1 -- 172.21.15.116:0/807536817 shutdown_connections 2024-04-25T00:30:45.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.602+0000 7f101f13a640 1 --2- 172.21.15.116:0/807536817 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10180a94a0 0x7f10181538e0 unknown :-1 s=CLOSED pgs=1830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:45.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.602+0000 7f101f13a640 1 -- 172.21.15.116:0/807536817 >> 172.21.15.116:0/807536817 conn(0x7f10180c3b00 msgr2=0x7f10180c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:45.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.602+0000 7f101f13a640 1 -- 172.21.15.116:0/807536817 shutdown_connections 2024-04-25T00:30:45.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.602+0000 7f101f13a640 1 -- 172.21.15.116:0/807536817 wait complete. 2024-04-25T00:30:45.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.606+0000 7f101f13a640 1 Processor -- start 2024-04-25T00:30:45.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.606+0000 7f101f13a640 1 -- start start 2024-04-25T00:30:45.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.606+0000 7f101f13a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10180a94a0 0x7f10181efce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:45.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.606+0000 7f101f13a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f10180a99f0 0x7f10181ec780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:45.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.606+0000 7f101f13a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1018150d20 0x7f1018136540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:45.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.606+0000 7f101f13a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f10180c32a0 con 0x7f10180a94a0 2024-04-25T00:30:45.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.606+0000 7f101f13a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f10180c3120 con 0x7f1018150d20 2024-04-25T00:30:45.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.606+0000 7f101f13a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f1018161740 con 0x7f10180a99f0 2024-04-25T00:30:45.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.606+0000 7f101ced6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10180a94a0 0x7f10181efce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:45.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.606+0000 7f101ced6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10180a94a0 0x7f10181efce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40556/0 (socket says 172.21.15.116:40556) 2024-04-25T00:30:45.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.606+0000 7f101ced6640 1 -- 172.21.15.116:0/658850812 learned_addr learned my addr 172.21.15.116:0/658850812 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:45.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.606+0000 7f101ced6640 1 -- 172.21.15.116:0/658850812 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f10180a99f0 msgr2=0x7f10181ec780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:45.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.606+0000 7f101ced6640 1 --2- 172.21.15.116:0/658850812 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f10180a99f0 0x7f10181ec780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:45.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.606+0000 7f101ced6640 1 -- 172.21.15.116:0/658850812 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1018150d20 msgr2=0x7f1018136540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:45.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.606+0000 7f101ced6640 1 --2- 172.21.15.116:0/658850812 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1018150d20 0x7f1018136540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:45.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.606+0000 7f101ced6640 1 -- 172.21.15.116:0/658850812 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1018136a80 con 0x7f10180a94a0 2024-04-25T00:30:45.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.606+0000 7f101ced6640 1 --2- 172.21.15.116:0/658850812 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10180a94a0 0x7f10181efce0 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f100c00b410 tx=0x7f100c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:45.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.606+0000 7f10157fa640 1 -- 172.21.15.116:0/658850812 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f100c015030 con 0x7f10180a94a0 2024-04-25T00:30:45.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.606+0000 7f10157fa640 1 -- 172.21.15.116:0/658850812 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f100c004020 con 0x7f10180a94a0 2024-04-25T00:30:45.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.606+0000 7f101f13a640 1 -- 172.21.15.116:0/658850812 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1018141b80 con 0x7f10180a94a0 2024-04-25T00:30:45.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.606+0000 7f10157fa640 1 -- 172.21.15.116:0/658850812 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f100c004410 con 0x7f10180a94a0 2024-04-25T00:30:45.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.606+0000 7f101f13a640 1 -- 172.21.15.116:0/658850812 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10181ef4e0 con 0x7f10180a94a0 2024-04-25T00:30:45.609 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.606+0000 7f101f13a640 1 -- 172.21.15.116:0/658850812 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10181420a0 con 0x7f10180a94a0 2024-04-25T00:30:45.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.606+0000 7f10157fa640 1 -- 172.21.15.116:0/658850812 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f100c018070 con 0x7f10180a94a0 2024-04-25T00:30:45.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.610+0000 7f10157fa640 1 --2- 172.21.15.116:0/658850812 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f10040678c0 0x7f1004069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:45.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.610+0000 7f10157fa640 1 -- 172.21.15.116:0/658850812 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7f100c086b60 con 0x7f10180a94a0 2024-04-25T00:30:45.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.610+0000 7f1017fff640 1 --2- 172.21.15.116:0/658850812 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f10040678c0 0x7f1004069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:45.610 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.610+0000 7f1017fff640 1 --2- 172.21.15.116:0/658850812 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f10040678c0 0x7f1004069d80 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f10080ad000 tx=0x7f1008008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:45.613 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.614+0000 7f10157fa640 1 -- 172.21.15.116:0/658850812 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f10181420a0 con 0x7f10180a94a0 2024-04-25T00:30:45.762 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.762+0000 7f101f13a640 1 -- 172.21.15.116:0/658850812 --> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f10181eccc0 con 0x7f10040678c0 2024-04-25T00:30:45.763 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.762+0000 7f10157fa640 1 -- 172.21.15.116:0/658850812 <== mgr.4105 v2:172.21.15.174:6832/2004448211 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f10080b1040 con 0x7f10040678c0 2024-04-25T00:30:45.763 INFO:teuthology.orchestra.run.smithi116.stderr:instructed osd(s) 4 to deep-scrub 2024-04-25T00:30:45.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.762+0000 7f101f13a640 1 -- 172.21.15.116:0/658850812 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f10040678c0 msgr2=0x7f1004069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:45.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.762+0000 7f101f13a640 1 --2- 172.21.15.116:0/658850812 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f10040678c0 0x7f1004069d80 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f10080ad000 tx=0x7f1008008040 comp rx=0 tx=0).stop 2024-04-25T00:30:45.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.762+0000 7f101f13a640 1 -- 172.21.15.116:0/658850812 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10180a94a0 msgr2=0x7f10181efce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:45.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.762+0000 7f101f13a640 1 --2- 172.21.15.116:0/658850812 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10180a94a0 0x7f10181efce0 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f100c00b410 tx=0x7f100c00b8d0 comp rx=0 tx=0).stop 2024-04-25T00:30:45.765 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.766+0000 7f101f13a640 1 -- 172.21.15.116:0/658850812 shutdown_connections 2024-04-25T00:30:45.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.766+0000 7f101f13a640 1 --2- 172.21.15.116:0/658850812 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f10040678c0 0x7f1004069d80 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:45.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.766+0000 7f101f13a640 1 --2- 172.21.15.116:0/658850812 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1018150d20 0x7f1018136540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:45.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.766+0000 7f101f13a640 1 --2- 172.21.15.116:0/658850812 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f10180a99f0 0x7f10181ec780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:45.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.766+0000 7f101f13a640 1 --2- 172.21.15.116:0/658850812 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f10180a94a0 0x7f10181efce0 unknown :-1 s=CLOSED pgs=1831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:45.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.766+0000 7f101f13a640 1 -- 172.21.15.116:0/658850812 >> 172.21.15.116:0/658850812 conn(0x7f10180c3b00 msgr2=0x7f10181538e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:45.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.766+0000 7f101f13a640 1 -- 172.21.15.116:0/658850812 shutdown_connections 2024-04-25T00:30:45.766 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.766+0000 7f101f13a640 1 -- 172.21.15.116:0/658850812 wait complete. 2024-04-25T00:30:45.780 INFO:tasks.ceph:Scrubbing osd.5 2024-04-25T00:30:45.781 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-04-25T00:30:45.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.898+0000 7f2a67cc6640 1 Processor -- start 2024-04-25T00:30:45.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.898+0000 7f2a67cc6640 1 -- start start 2024-04-25T00:30:45.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.902+0000 7f2a67cc6640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f2a600ac1b0 0x7f2a600718e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:45.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.902+0000 7f2a67cc6640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2a60152920 0x7f2a6015c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:45.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.902+0000 7f2a67cc6640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a600c1f10 con 0x7f2a6014f3d0 2024-04-25T00:30:45.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.902+0000 7f2a67cc6640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f2a600c38a0 con 0x7f2a600ac1b0 2024-04-25T00:30:45.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.902+0000 7f2a67cc6640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f2a60152490 con 0x7f2a60152920 2024-04-25T00:30:45.902 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.902+0000 7f2a65261640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f2a600ac1b0 0x7f2a600718e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:45.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.902+0000 7f2a66263640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2a60152920 0x7f2a6015c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:45.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.902+0000 7f2a65261640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f2a600ac1b0 0x7f2a600718e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:36674/0 (socket says 172.21.15.116:36674) 2024-04-25T00:30:45.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.902+0000 7f2a65261640 1 -- 172.21.15.116:0/1254333039 learned_addr learned my addr 172.21.15.116:0/1254333039 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:45.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.902+0000 7f2a65261640 1 -- 172.21.15.116:0/1254333039 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2a60152920 msgr2=0x7f2a6015c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:45.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.902+0000 7f2a65261640 1 --2- 172.21.15.116:0/1254333039 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2a60152920 0x7f2a6015c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:45.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.902+0000 7f2a65261640 1 -- 172.21.15.116:0/1254333039 >> v1:172.21.15.174:6789/0 conn(0x7f2a6014f3d0 legacy=0x7f2a600c4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:45.903 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.902+0000 7f2a65261640 1 -- 172.21.15.116:0/1254333039 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a60151e80 con 0x7f2a600ac1b0 2024-04-25T00:30:45.904 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.902+0000 7f2a65261640 1 --2- 172.21.15.116:0/1254333039 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a600ac1b0 0x7f2a600718e0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f2a5000aa10 tx=0x7f2a5001d930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6dcf52a3d539884a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:45.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.902+0000 7f2a64a60640 1 -- 172.21.15.116:0/1254333039 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a500b3070 con 0x7f2a600ac1b0 2024-04-25T00:30:45.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.902+0000 7f2a64a60640 1 -- 172.21.15.116:0/1254333039 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a500a6920 con 0x7f2a600ac1b0 2024-04-25T00:30:45.905 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.902+0000 7f2a64a60640 1 -- 172.21.15.116:0/1254333039 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a5000b040 con 0x7f2a600ac1b0 2024-04-25T00:30:45.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.906+0000 7f2a67cc6640 1 -- 172.21.15.116:0/1254333039 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a600ac1b0 msgr2=0x7f2a600718e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:45.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.906+0000 7f2a67cc6640 1 --2- 172.21.15.116:0/1254333039 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a600ac1b0 0x7f2a600718e0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f2a5000aa10 tx=0x7f2a5001d930 comp rx=0 tx=0).stop 2024-04-25T00:30:45.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.906+0000 7f2a67cc6640 1 -- 172.21.15.116:0/1254333039 shutdown_connections 2024-04-25T00:30:45.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.906+0000 7f2a67cc6640 1 --2- 172.21.15.116:0/1254333039 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2a60152920 0x7f2a6015c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:45.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.906+0000 7f2a67cc6640 1 --2- 172.21.15.116:0/1254333039 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a600ac1b0 0x7f2a600718e0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:45.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.906+0000 7f2a67cc6640 1 -- 172.21.15.116:0/1254333039 >> 172.21.15.116:0/1254333039 conn(0x7f2a6006b810 msgr2=0x7f2a600c6f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:45.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.906+0000 7f2a67cc6640 1 -- 172.21.15.116:0/1254333039 shutdown_connections 2024-04-25T00:30:45.906 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.906+0000 7f2a67cc6640 1 -- 172.21.15.116:0/1254333039 wait complete. 2024-04-25T00:30:45.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.906+0000 7f2a67cc6640 1 Processor -- start 2024-04-25T00:30:45.908 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.906+0000 7f2a67cc6640 1 -- start start 2024-04-25T00:30:45.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.906+0000 7f2a67cc6640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2a600ac1b0 0x7f2a60153c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:45.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.906+0000 7f2a67cc6640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a6014f3d0 0x7f2a601414d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:45.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.906+0000 7f2a67cc6640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a60152920 0x7f2a601efbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:45.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.906+0000 7f2a67cc6640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2a6015eaa0 con 0x7f2a60152920 2024-04-25T00:30:45.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.906+0000 7f2a67cc6640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f2a600c5530 con 0x7f2a6014f3d0 2024-04-25T00:30:45.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.906+0000 7f2a67cc6640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f2a6015ec20 con 0x7f2a600ac1b0 2024-04-25T00:30:45.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.906+0000 7f2a66263640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a60152920 0x7f2a601efbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:45.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.906+0000 7f2a66263640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a60152920 0x7f2a601efbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40566/0 (socket says 172.21.15.116:40566) 2024-04-25T00:30:45.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.906+0000 7f2a66263640 1 -- 172.21.15.116:0/825791233 learned_addr learned my addr 172.21.15.116:0/825791233 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:45.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.906+0000 7f2a65261640 1 --2- 172.21.15.116:0/825791233 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a6014f3d0 0x7f2a601414d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:45.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.906+0000 7f2a65a62640 1 --2- 172.21.15.116:0/825791233 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2a600ac1b0 0x7f2a60153c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:45.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.906+0000 7f2a66263640 1 -- 172.21.15.116:0/825791233 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2a600ac1b0 msgr2=0x7f2a60153c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:45.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.906+0000 7f2a66263640 1 --2- 172.21.15.116:0/825791233 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2a600ac1b0 0x7f2a60153c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:45.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.906+0000 7f2a66263640 1 -- 172.21.15.116:0/825791233 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a6014f3d0 msgr2=0x7f2a601414d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:45.909 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.906+0000 7f2a66263640 1 --2- 172.21.15.116:0/825791233 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a6014f3d0 0x7f2a601414d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:45.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.906+0000 7f2a66263640 1 -- 172.21.15.116:0/825791233 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a60141fd0 con 0x7f2a60152920 2024-04-25T00:30:45.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.906+0000 7f2a65261640 1 --2- 172.21.15.116:0/825791233 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2a6014f3d0 0x7f2a601414d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:30:45.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.906+0000 7f2a66263640 1 --2- 172.21.15.116:0/825791233 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a60152920 0x7f2a601efbe0 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f2a5800efc0 tx=0x7f2a5800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:45.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.906+0000 7f2a567fc640 1 -- 172.21.15.116:0/825791233 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a58018020 con 0x7f2a60152920 2024-04-25T00:30:45.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.906+0000 7f2a567fc640 1 -- 172.21.15.116:0/825791233 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a58014480 con 0x7f2a60152920 2024-04-25T00:30:45.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.906+0000 7f2a67cc6640 1 -- 172.21.15.116:0/825791233 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a60154180 con 0x7f2a60152920 2024-04-25T00:30:45.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.906+0000 7f2a567fc640 1 -- 172.21.15.116:0/825791233 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a58014870 con 0x7f2a60152920 2024-04-25T00:30:45.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.906+0000 7f2a67cc6640 1 -- 172.21.15.116:0/825791233 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a60155ed0 con 0x7f2a60152920 2024-04-25T00:30:45.910 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.906+0000 7f2a67cc6640 1 -- 172.21.15.116:0/825791233 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2a60068350 con 0x7f2a60152920 2024-04-25T00:30:45.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.910+0000 7f2a567fc640 1 -- 172.21.15.116:0/825791233 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2a5801b020 con 0x7f2a60152920 2024-04-25T00:30:45.911 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.910+0000 7f2a567fc640 1 --2- 172.21.15.116:0/825791233 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2a440678c0 0x7f2a44069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:45.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.910+0000 7f2a567fc640 1 -- 172.21.15.116:0/825791233 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7f2a58088170 con 0x7f2a60152920 2024-04-25T00:30:45.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.910+0000 7f2a567fc640 1 --2- 172.21.15.116:0/825791233 >> [v2:172.21.15.174:6808/881817164,v1:172.21.15.174:6809/881817164] conn(0x7f2a440746b0 0x7f2a44076b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:45.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.910+0000 7f2a567fc640 1 -- 172.21.15.116:0/825791233 --> [v2:172.21.15.174:6808/881817164,v1:172.21.15.174:6809/881817164] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2a44077220 con 0x7f2a440746b0 2024-04-25T00:30:45.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.910+0000 7f2a567fc640 1 -- 172.21.15.116:0/825791233 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=74) ==== 24+0+0 (secure 0 0 0) 0x7f2a580885c0 con 0x7f2a60152920 2024-04-25T00:30:45.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.910+0000 7f2a65a62640 1 --2- 172.21.15.116:0/825791233 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2a440678c0 0x7f2a44069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:45.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.910+0000 7f2a65261640 1 --2- 172.21.15.116:0/825791233 >> [v2:172.21.15.174:6808/881817164,v1:172.21.15.174:6809/881817164] conn(0x7f2a440746b0 0x7f2a44076b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:45.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.910+0000 7f2a65261640 1 --2- 172.21.15.116:0/825791233 >> [v2:172.21.15.174:6808/881817164,v1:172.21.15.174:6809/881817164] conn(0x7f2a440746b0 0x7f2a44076b50 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:45.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.910+0000 7f2a65a62640 1 --2- 172.21.15.116:0/825791233 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2a440678c0 0x7f2a44069d80 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c000f70 tx=0x7f2a5c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:45.912 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.910+0000 7f2a567fc640 1 -- 172.21.15.116:0/825791233 <== osd.5 v2:172.21.15.174:6808/881817164 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2a44077220 con 0x7f2a440746b0 2024-04-25T00:30:45.924 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.922+0000 7f2a67cc6640 1 -- 172.21.15.116:0/825791233 --> [v2:172.21.15.174:6808/881817164,v1:172.21.15.174:6809/881817164] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f2a600c6d00 con 0x7f2a440746b0 2024-04-25T00:30:45.925 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.926+0000 7f2a567fc640 1 -- 172.21.15.116:0/825791233 <== osd.5 v2:172.21.15.174:6808/881817164 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f2a600c6d00 con 0x7f2a440746b0 2024-04-25T00:30:45.925 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-25T00:30:45.926 INFO:teuthology.orchestra.run.smithi116.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T00:30:45.926 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-25T00:30:45.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.926+0000 7f2a67cc6640 1 -- 172.21.15.116:0/825791233 >> [v2:172.21.15.174:6808/881817164,v1:172.21.15.174:6809/881817164] conn(0x7f2a440746b0 msgr2=0x7f2a44076b50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:45.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.926+0000 7f2a67cc6640 1 --2- 172.21.15.116:0/825791233 >> [v2:172.21.15.174:6808/881817164,v1:172.21.15.174:6809/881817164] conn(0x7f2a440746b0 0x7f2a44076b50 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:45.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.926+0000 7f2a67cc6640 1 -- 172.21.15.116:0/825791233 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2a440678c0 msgr2=0x7f2a44069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:45.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.926+0000 7f2a67cc6640 1 --2- 172.21.15.116:0/825791233 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f2a440678c0 0x7f2a44069d80 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c000f70 tx=0x7f2a5c009040 comp rx=0 tx=0).stop 2024-04-25T00:30:45.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.926+0000 7f2a67cc6640 1 -- 172.21.15.116:0/825791233 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a60152920 msgr2=0x7f2a601efbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:45.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.926+0000 7f2a67cc6640 1 --2- 172.21.15.116:0/825791233 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2a60152920 0x7f2a601efbe0 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f2a5800efc0 tx=0x7f2a5800c490 comp rx=0 tx=0).stop 2024-04-25T00:30:45.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.926+0000 7f2a66263640 1 -- 172.21.15.116:0/825791233 reap_dead start 2024-04-25T00:30:45.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.926+0000 7f2a67cc6640 1 -- 172.21.15.116:0/825791233 shutdown_connections 2024-04-25T00:30:45.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.926+0000 7f2a67cc6640 1 -- 172.21.15.116:0/825791233 >> 172.21.15.116:0/825791233 conn(0x7f2a6006b810 msgr2=0x7f2a60148e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:45.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.926+0000 7f2a67cc6640 1 -- 172.21.15.116:0/825791233 shutdown_connections 2024-04-25T00:30:45.927 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:45.926+0000 7f2a67cc6640 1 -- 172.21.15.116:0/825791233 wait complete. 2024-04-25T00:30:45.941 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-04-25T00:30:46.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.054+0000 7f439c07e640 1 Processor -- start 2024-04-25T00:30:46.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.054+0000 7f439c07e640 1 -- start start 2024-04-25T00:30:46.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.058+0000 7f439c07e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4394150d20 0x7f43940c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:46.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.058+0000 7f439c07e640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f43940a94a0 0x7f4394153ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:46.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.058+0000 7f439c07e640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43940c3910 con 0x7f43940c2f50 2024-04-25T00:30:46.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.058+0000 7f439c07e640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f4394154530 con 0x7f43940a94a0 2024-04-25T00:30:46.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.058+0000 7f439c07e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f43940c16e0 con 0x7f4394150d20 2024-04-25T00:30:46.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.058+0000 7f4399e1a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4394150d20 0x7f43940c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:46.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.058+0000 7f439a61b640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f43940a94a0 0x7f4394153ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:46.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.058+0000 7f4399e1a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4394150d20 0x7f43940c2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:33618/0 (socket says 172.21.15.116:33618) 2024-04-25T00:30:46.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.058+0000 7f4399e1a640 1 -- 172.21.15.116:0/1633207524 learned_addr learned my addr 172.21.15.116:0/1633207524 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:46.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.058+0000 7f439a61b640 1 -- 172.21.15.116:0/1633207524 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4394150d20 msgr2=0x7f43940c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:46.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.058+0000 7f439a61b640 1 --2- 172.21.15.116:0/1633207524 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4394150d20 0x7f43940c2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:46.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.058+0000 7f439a61b640 1 -- 172.21.15.116:0/1633207524 >> v1:172.21.15.174:6789/0 conn(0x7f43940c2f50 legacy=0x7f43941538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:46.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.058+0000 7f439a61b640 1 -- 172.21.15.116:0/1633207524 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43940c1400 con 0x7f43940a94a0 2024-04-25T00:30:46.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.058+0000 7f4399e1a640 1 --2- 172.21.15.116:0/1633207524 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4394150d20 0x7f43940c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:30:46.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.062+0000 7f439a61b640 1 --2- 172.21.15.116:0/1633207524 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43940a94a0 0x7f4394153ff0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f438c009d90 tx=0x7f438c01c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=478abc192c5c7a35 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:46.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.062+0000 7f4398e18640 1 -- 172.21.15.116:0/1633207524 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f438c0b2070 con 0x7f43940a94a0 2024-04-25T00:30:46.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.062+0000 7f4398e18640 1 -- 172.21.15.116:0/1633207524 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f438c0a5940 con 0x7f43940a94a0 2024-04-25T00:30:46.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.062+0000 7f4398e18640 1 -- 172.21.15.116:0/1633207524 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f438c0a5d10 con 0x7f43940a94a0 2024-04-25T00:30:46.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.062+0000 7f439c07e640 1 -- 172.21.15.116:0/1633207524 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43940a94a0 msgr2=0x7f4394153ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:46.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.062+0000 7f439c07e640 1 --2- 172.21.15.116:0/1633207524 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43940a94a0 0x7f4394153ff0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f438c009d90 tx=0x7f438c01c970 comp rx=0 tx=0).stop 2024-04-25T00:30:46.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.062+0000 7f439c07e640 1 -- 172.21.15.116:0/1633207524 shutdown_connections 2024-04-25T00:30:46.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.062+0000 7f439c07e640 1 --2- 172.21.15.116:0/1633207524 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43940a94a0 0x7f4394153ff0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:46.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.062+0000 7f439c07e640 1 --2- 172.21.15.116:0/1633207524 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4394150d20 0x7f43940c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:46.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.062+0000 7f439c07e640 1 -- 172.21.15.116:0/1633207524 >> 172.21.15.116:0/1633207524 conn(0x7f43940c3b00 msgr2=0x7f43940c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:46.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.062+0000 7f439c07e640 1 -- 172.21.15.116:0/1633207524 shutdown_connections 2024-04-25T00:30:46.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.062+0000 7f439c07e640 1 -- 172.21.15.116:0/1633207524 wait complete. 2024-04-25T00:30:46.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.062+0000 7f439c07e640 1 Processor -- start 2024-04-25T00:30:46.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.062+0000 7f439c07e640 1 -- start start 2024-04-25T00:30:46.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.062+0000 7f439c07e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43940a94a0 0x7f43940c0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:46.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.062+0000 7f439c07e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43940c2f50 0x7f4394140760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:46.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.062+0000 7f439c07e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4394150d20 0x7f4394140ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:46.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.062+0000 7f439c07e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f43941608b0 con 0x7f4394150d20 2024-04-25T00:30:46.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.062+0000 7f439c07e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4394160730 con 0x7f43940c2f50 2024-04-25T00:30:46.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.062+0000 7f439c07e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f4394160a30 con 0x7f43940a94a0 2024-04-25T00:30:46.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.062+0000 7f439a61b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4394150d20 0x7f4394140ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:46.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.062+0000 7f439a61b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4394150d20 0x7f4394140ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40572/0 (socket says 172.21.15.116:40572) 2024-04-25T00:30:46.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.062+0000 7f439a61b640 1 -- 172.21.15.116:0/2349637467 learned_addr learned my addr 172.21.15.116:0/2349637467 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:46.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.062+0000 7f4399e1a640 1 --2- 172.21.15.116:0/2349637467 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43940a94a0 0x7f43940c0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:46.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.062+0000 7f4399619640 1 --2- 172.21.15.116:0/2349637467 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43940c2f50 0x7f4394140760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:46.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.062+0000 7f439a61b640 1 -- 172.21.15.116:0/2349637467 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43940a94a0 msgr2=0x7f43940c0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:46.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.062+0000 7f439a61b640 1 --2- 172.21.15.116:0/2349637467 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43940a94a0 0x7f43940c0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:46.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.062+0000 7f439a61b640 1 -- 172.21.15.116:0/2349637467 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43940c2f50 msgr2=0x7f4394140760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:46.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.062+0000 7f439a61b640 1 --2- 172.21.15.116:0/2349637467 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43940c2f50 0x7f4394140760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:46.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.062+0000 7f439a61b640 1 -- 172.21.15.116:0/2349637467 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43941435b0 con 0x7f4394150d20 2024-04-25T00:30:46.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.062+0000 7f4399619640 1 --2- 172.21.15.116:0/2349637467 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43940c2f50 0x7f4394140760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:30:46.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.066+0000 7f439a61b640 1 --2- 172.21.15.116:0/2349637467 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4394150d20 0x7f4394140ca0 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7f438c0b0040 tx=0x7f438c019710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:46.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.066+0000 7f438a7fc640 1 -- 172.21.15.116:0/2349637467 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f438c0b2050 con 0x7f4394150d20 2024-04-25T00:30:46.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.066+0000 7f438a7fc640 1 -- 172.21.15.116:0/2349637467 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f438c0be070 con 0x7f4394150d20 2024-04-25T00:30:46.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.066+0000 7f439c07e640 1 -- 172.21.15.116:0/2349637467 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4394143890 con 0x7f4394150d20 2024-04-25T00:30:46.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.066+0000 7f438a7fc640 1 -- 172.21.15.116:0/2349637467 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f438c0ada90 con 0x7f4394150d20 2024-04-25T00:30:46.067 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.066+0000 7f439c07e640 1 -- 172.21.15.116:0/2349637467 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4394143e50 con 0x7f4394150d20 2024-04-25T00:30:46.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.066+0000 7f439c07e640 1 -- 172.21.15.116:0/2349637467 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4394068350 con 0x7f4394150d20 2024-04-25T00:30:46.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.066+0000 7f438a7fc640 1 -- 172.21.15.116:0/2349637467 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f438c0adc30 con 0x7f4394150d20 2024-04-25T00:30:46.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.066+0000 7f438a7fc640 1 --2- 172.21.15.116:0/2349637467 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f43780678f0 0x7f4378069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:46.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.066+0000 7f438a7fc640 1 -- 172.21.15.116:0/2349637467 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7f438c121fd0 con 0x7f4394150d20 2024-04-25T00:30:46.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.066+0000 7f4399e1a640 1 --2- 172.21.15.116:0/2349637467 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f43780678f0 0x7f4378069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:46.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.066+0000 7f4399e1a640 1 --2- 172.21.15.116:0/2349637467 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f43780678f0 0x7f4378069db0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f4384006cb0 tx=0x7f4384009240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:46.071 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.070+0000 7f438a7fc640 1 -- 172.21.15.116:0/2349637467 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f438c016560 con 0x7f4394150d20 2024-04-25T00:30:46.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.222+0000 7f439c07e640 1 -- 172.21.15.116:0/2349637467 --> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f43940c34f0 con 0x7f43780678f0 2024-04-25T00:30:46.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.226+0000 7f438a7fc640 1 -- 172.21.15.116:0/2349637467 <== mgr.4105 v2:172.21.15.174:6832/2004448211 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f43940c34f0 con 0x7f43780678f0 2024-04-25T00:30:46.225 INFO:teuthology.orchestra.run.smithi116.stderr:instructed osd(s) 5 to deep-scrub 2024-04-25T00:30:46.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.226+0000 7f439c07e640 1 -- 172.21.15.116:0/2349637467 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f43780678f0 msgr2=0x7f4378069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:46.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.226+0000 7f439c07e640 1 --2- 172.21.15.116:0/2349637467 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f43780678f0 0x7f4378069db0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f4384006cb0 tx=0x7f4384009240 comp rx=0 tx=0).stop 2024-04-25T00:30:46.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.226+0000 7f439c07e640 1 -- 172.21.15.116:0/2349637467 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4394150d20 msgr2=0x7f4394140ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:46.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.226+0000 7f439c07e640 1 --2- 172.21.15.116:0/2349637467 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4394150d20 0x7f4394140ca0 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7f438c0b0040 tx=0x7f438c019710 comp rx=0 tx=0).stop 2024-04-25T00:30:46.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.226+0000 7f439c07e640 1 -- 172.21.15.116:0/2349637467 shutdown_connections 2024-04-25T00:30:46.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.226+0000 7f439c07e640 1 --2- 172.21.15.116:0/2349637467 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f43780678f0 0x7f4378069db0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:46.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.226+0000 7f439c07e640 1 --2- 172.21.15.116:0/2349637467 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4394150d20 0x7f4394140ca0 unknown :-1 s=CLOSED pgs=1833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:46.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.226+0000 7f439c07e640 1 --2- 172.21.15.116:0/2349637467 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f43940c2f50 0x7f4394140760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:46.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.226+0000 7f439c07e640 1 --2- 172.21.15.116:0/2349637467 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f43940a94a0 0x7f43940c0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:46.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.226+0000 7f439c07e640 1 -- 172.21.15.116:0/2349637467 >> 172.21.15.116:0/2349637467 conn(0x7f43940c3b00 msgr2=0x7f439406cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:46.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.226+0000 7f439c07e640 1 -- 172.21.15.116:0/2349637467 shutdown_connections 2024-04-25T00:30:46.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.226+0000 7f439c07e640 1 -- 172.21.15.116:0/2349637467 wait complete. 2024-04-25T00:30:46.243 INFO:tasks.ceph:Scrubbing osd.6 2024-04-25T00:30:46.243 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-04-25T00:30:46.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.374+0000 7fc6b71ad640 1 Processor -- start 2024-04-25T00:30:46.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.374+0000 7fc6b71ad640 1 -- start start 2024-04-25T00:30:46.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.374+0000 7fc6b71ad640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fc6b00c6d70 0x7fc6b00c75c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:46.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.374+0000 7fc6b71ad640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fc6b006c5d0 0x7fc6b00c5c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:46.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.374+0000 7fc6b71ad640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6b0156530 con 0x7fc6b0153210 2024-04-25T00:30:46.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.374+0000 7fc6b71ad640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fc6b006bf00 con 0x7fc6b00c6d70 2024-04-25T00:30:46.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.374+0000 7fc6b71ad640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fc6b01567b0 con 0x7fc6b006c5d0 2024-04-25T00:30:46.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.374+0000 7fc6b4f49640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fc6b0153210 0x7fc6b00c8fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:43702/0 (socket says 172.21.15.116:43702) 2024-04-25T00:30:46.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.374+0000 7fc6b4f49640 1 -- 172.21.15.116:0/3977130997 learned_addr learned my addr 172.21.15.116:0/3977130997 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:30:46.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.374+0000 7fc6affff640 1 --2- 172.21.15.116:0/3977130997 >> v2:172.21.15.116:3300/0 conn(0x7fc6b00c6d70 0x7fc6b00c75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:46.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.374+0000 7fc6af7fe640 1 -- 172.21.15.116:0/3977130997 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 11818189 0 0) 0x7fc6a0001120 con 0x7fc6b0153210 2024-04-25T00:30:46.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.374+0000 7fc6af7fe640 1 -- 172.21.15.116:0/3977130997 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc698003530 con 0x7fc6b0153210 2024-04-25T00:30:46.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.374+0000 7fc6affff640 1 -- 172.21.15.116:0/3977130997 >> v2:172.21.15.174:3300/0 conn(0x7fc6b006c5d0 msgr2=0x7fc6b00c5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:46.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.374+0000 7fc6b574a640 1 --2- 172.21.15.116:0/3977130997 >> v2:172.21.15.174:3300/0 conn(0x7fc6b006c5d0 0x7fc6b00c5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:46.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.374+0000 7fc6affff640 1 --2- 172.21.15.116:0/3977130997 >> v2:172.21.15.174:3300/0 conn(0x7fc6b006c5d0 0x7fc6b00c5c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:46.376 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.374+0000 7fc6affff640 1 -- 172.21.15.116:0/3977130997 >> v1:172.21.15.116:6789/0 conn(0x7fc6b0153210 legacy=0x7fc6b00c8fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:46.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.374+0000 7fc6affff640 1 -- 172.21.15.116:0/3977130997 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6b00c96e0 con 0x7fc6b00c6d70 2024-04-25T00:30:46.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.374+0000 7fc6b574a640 1 --2- 172.21.15.116:0/3977130997 >> v2:172.21.15.174:3300/0 conn(0x7fc6b006c5d0 0x7fc6b00c5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:30:46.377 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.378+0000 7fc6affff640 1 --2- 172.21.15.116:0/3977130997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6b00c6d70 0x7fc6b00c75c0 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7fc6b0073560 tx=0x7fc6a401c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dd410d3c3e9c9f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:46.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.378+0000 7fc6af7fe640 1 -- 172.21.15.116:0/3977130997 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6a40b0070 con 0x7fc6b00c6d70 2024-04-25T00:30:46.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.378+0000 7fc6af7fe640 1 -- 172.21.15.116:0/3977130997 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6a40ab600 con 0x7fc6b00c6d70 2024-04-25T00:30:46.378 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.378+0000 7fc6af7fe640 1 -- 172.21.15.116:0/3977130997 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6a40ab9d0 con 0x7fc6b00c6d70 2024-04-25T00:30:46.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.378+0000 7fc6b71ad640 1 -- 172.21.15.116:0/3977130997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6b00c6d70 msgr2=0x7fc6b00c75c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:46.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.378+0000 7fc6b71ad640 1 --2- 172.21.15.116:0/3977130997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6b00c6d70 0x7fc6b00c75c0 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7fc6b0073560 tx=0x7fc6a401c360 comp rx=0 tx=0).stop 2024-04-25T00:30:46.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.378+0000 7fc6b71ad640 1 -- 172.21.15.116:0/3977130997 shutdown_connections 2024-04-25T00:30:46.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.378+0000 7fc6b71ad640 1 --2- 172.21.15.116:0/3977130997 >> v2:172.21.15.174:3300/0 conn(0x7fc6b006c5d0 0x7fc6b00c5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:46.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.378+0000 7fc6b71ad640 1 --2- 172.21.15.116:0/3977130997 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6b00c6d70 0x7fc6b00c75c0 unknown :-1 s=CLOSED pgs=1835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:46.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.378+0000 7fc6b71ad640 1 -- 172.21.15.116:0/3977130997 >> 172.21.15.116:0/3977130997 conn(0x7fc6b00acbc0 msgr2=0x7fc6b00acfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:46.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.378+0000 7fc6b71ad640 1 -- 172.21.15.116:0/3977130997 shutdown_connections 2024-04-25T00:30:46.379 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.378+0000 7fc6b71ad640 1 -- 172.21.15.116:0/3977130997 wait complete. 2024-04-25T00:30:46.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.378+0000 7fc6b71ad640 1 Processor -- start 2024-04-25T00:30:46.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.378+0000 7fc6b71ad640 1 -- start start 2024-04-25T00:30:46.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.378+0000 7fc6b71ad640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6b006c5d0 0x7fc6b00c2400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:46.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.378+0000 7fc6b71ad640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc6b00c6d70 0x7fc6b014a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:46.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.378+0000 7fc6b71ad640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc6b0153210 0x7fc6b014a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:46.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.378+0000 7fc6b71ad640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc6b0162f30 con 0x7fc6b006c5d0 2024-04-25T00:30:46.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.378+0000 7fc6b71ad640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc6b0162db0 con 0x7fc6b00c6d70 2024-04-25T00:30:46.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.378+0000 7fc6b71ad640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fc6b01630b0 con 0x7fc6b0153210 2024-04-25T00:30:46.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.378+0000 7fc6b4f49640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6b006c5d0 0x7fc6b00c2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:46.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.378+0000 7fc6b4f49640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6b006c5d0 0x7fc6b00c2400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40604/0 (socket says 172.21.15.116:40604) 2024-04-25T00:30:46.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.378+0000 7fc6b4f49640 1 -- 172.21.15.116:0/141645649 learned_addr learned my addr 172.21.15.116:0/141645649 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:46.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.378+0000 7fc6b4f49640 1 -- 172.21.15.116:0/141645649 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc6b0153210 msgr2=0x7fc6b014a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:46.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.378+0000 7fc6b4f49640 1 --2- 172.21.15.116:0/141645649 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc6b0153210 0x7fc6b014a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:46.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.378+0000 7fc6b4f49640 1 -- 172.21.15.116:0/141645649 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc6b00c6d70 msgr2=0x7fc6b014a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:46.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.378+0000 7fc6b4f49640 1 --2- 172.21.15.116:0/141645649 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc6b00c6d70 0x7fc6b014a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:46.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.378+0000 7fc6b4f49640 1 -- 172.21.15.116:0/141645649 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6b01ef270 con 0x7fc6b006c5d0 2024-04-25T00:30:46.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.378+0000 7fc6b4f49640 1 --2- 172.21.15.116:0/141645649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6b006c5d0 0x7fc6b00c2400 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7fc6a000c3f0 tx=0x7fc6a000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:46.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.378+0000 7fc6ad7fa640 1 -- 172.21.15.116:0/141645649 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6a0017070 con 0x7fc6b006c5d0 2024-04-25T00:30:46.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.378+0000 7fc6ad7fa640 1 -- 172.21.15.116:0/141645649 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6a0005cf0 con 0x7fc6b006c5d0 2024-04-25T00:30:46.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.378+0000 7fc6b71ad640 1 -- 172.21.15.116:0/141645649 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6b01ef550 con 0x7fc6b006c5d0 2024-04-25T00:30:46.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.378+0000 7fc6ad7fa640 1 -- 172.21.15.116:0/141645649 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6a0007310 con 0x7fc6b006c5d0 2024-04-25T00:30:46.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.378+0000 7fc6b71ad640 1 -- 172.21.15.116:0/141645649 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6b014d8f0 con 0x7fc6b006c5d0 2024-04-25T00:30:46.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.382+0000 7fc6b71ad640 1 -- 172.21.15.116:0/141645649 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc6b014d420 con 0x7fc6b006c5d0 2024-04-25T00:30:46.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.382+0000 7fc6ad7fa640 1 -- 172.21.15.116:0/141645649 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc6a001a070 con 0x7fc6b006c5d0 2024-04-25T00:30:46.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.382+0000 7fc6ad7fa640 1 --2- 172.21.15.116:0/141645649 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc69c0678c0 0x7fc69c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:46.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.382+0000 7fc6ad7fa640 1 -- 172.21.15.116:0/141645649 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7fc6a00863d0 con 0x7fc6b006c5d0 2024-04-25T00:30:46.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.382+0000 7fc6ad7fa640 1 --2- 172.21.15.116:0/141645649 >> [v2:172.21.15.174:6824/117574251,v1:172.21.15.174:6825/117574251] conn(0x7fc69c0746b0 0x7fc69c076b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:46.384 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.382+0000 7fc6ad7fa640 1 -- 172.21.15.116:0/141645649 --> [v2:172.21.15.174:6824/117574251,v1:172.21.15.174:6825/117574251] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc6a0016ec0 con 0x7fc69c0746b0 2024-04-25T00:30:46.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.382+0000 7fc6ad7fa640 1 -- 172.21.15.116:0/141645649 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=74) ==== 24+0+0 (secure 0 0 0) 0x7fc6a008d050 con 0x7fc6b006c5d0 2024-04-25T00:30:46.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.382+0000 7fc6affff640 1 --2- 172.21.15.116:0/141645649 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc69c0678c0 0x7fc69c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:46.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.382+0000 7fc6b574a640 1 --2- 172.21.15.116:0/141645649 >> [v2:172.21.15.174:6824/117574251,v1:172.21.15.174:6825/117574251] conn(0x7fc69c0746b0 0x7fc69c076b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:46.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.382+0000 7fc6b574a640 1 --2- 172.21.15.116:0/141645649 >> [v2:172.21.15.174:6824/117574251,v1:172.21.15.174:6825/117574251] conn(0x7fc69c0746b0 0x7fc69c076b50 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:46.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.382+0000 7fc6affff640 1 --2- 172.21.15.116:0/141645649 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc69c0678c0 0x7fc69c069d80 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fc6a40ad000 tx=0x7fc6a4004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:46.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.386+0000 7fc6ad7fa640 1 -- 172.21.15.116:0/141645649 <== osd.6 v2:172.21.15.174:6824/117574251 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc6a0016ec0 con 0x7fc69c0746b0 2024-04-25T00:30:46.398 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.398+0000 7fc6b71ad640 1 -- 172.21.15.116:0/141645649 --> [v2:172.21.15.174:6824/117574251,v1:172.21.15.174:6825/117574251] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fc6b0068350 con 0x7fc69c0746b0 2024-04-25T00:30:46.399 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.398+0000 7fc6ad7fa640 1 -- 172.21.15.116:0/141645649 <== osd.6 v2:172.21.15.174:6824/117574251 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fc6b0068350 con 0x7fc69c0746b0 2024-04-25T00:30:46.400 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-25T00:30:46.400 INFO:teuthology.orchestra.run.smithi116.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T00:30:46.400 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-25T00:30:46.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.398+0000 7fc6b71ad640 1 -- 172.21.15.116:0/141645649 >> [v2:172.21.15.174:6824/117574251,v1:172.21.15.174:6825/117574251] conn(0x7fc69c0746b0 msgr2=0x7fc69c076b50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:46.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.398+0000 7fc6b71ad640 1 --2- 172.21.15.116:0/141645649 >> [v2:172.21.15.174:6824/117574251,v1:172.21.15.174:6825/117574251] conn(0x7fc69c0746b0 0x7fc69c076b50 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:46.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.398+0000 7fc6b71ad640 1 -- 172.21.15.116:0/141645649 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc69c0678c0 msgr2=0x7fc69c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:46.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.398+0000 7fc6b71ad640 1 --2- 172.21.15.116:0/141645649 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc69c0678c0 0x7fc69c069d80 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fc6a40ad000 tx=0x7fc6a4004740 comp rx=0 tx=0).stop 2024-04-25T00:30:46.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.398+0000 7fc6b71ad640 1 -- 172.21.15.116:0/141645649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6b006c5d0 msgr2=0x7fc6b00c2400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:46.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.398+0000 7fc6b71ad640 1 --2- 172.21.15.116:0/141645649 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc6b006c5d0 0x7fc6b00c2400 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7fc6a000c3f0 tx=0x7fc6a000c8b0 comp rx=0 tx=0).stop 2024-04-25T00:30:46.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.398+0000 7fc6b574a640 1 -- 172.21.15.116:0/141645649 reap_dead start 2024-04-25T00:30:46.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.398+0000 7fc6b71ad640 1 -- 172.21.15.116:0/141645649 shutdown_connections 2024-04-25T00:30:46.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.398+0000 7fc6b71ad640 1 -- 172.21.15.116:0/141645649 >> 172.21.15.116:0/141645649 conn(0x7fc6b00acbc0 msgr2=0x7fc6b00c74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:46.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.398+0000 7fc6b71ad640 1 -- 172.21.15.116:0/141645649 shutdown_connections 2024-04-25T00:30:46.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.398+0000 7fc6b71ad640 1 -- 172.21.15.116:0/141645649 wait complete. 2024-04-25T00:30:46.415 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-04-25T00:30:46.528 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.526+0000 7f4e7463e640 1 Processor -- start 2024-04-25T00:30:46.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.526+0000 7f4e7463e640 1 -- start start 2024-04-25T00:30:46.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.530+0000 7f4e7463e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4e6c14c720 0x7f4e6c1563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:46.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.530+0000 7f4e7463e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4e6c0a94a0 0x7f4e6c0c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:46.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.530+0000 7f4e7463e640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e6c071220 con 0x7f4e6c0c2f50 2024-04-25T00:30:46.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.530+0000 7f4e7463e640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f4e6c153460 con 0x7f4e6c14c720 2024-04-25T00:30:46.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.530+0000 7f4e7463e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f4e6c152420 con 0x7f4e6c0a94a0 2024-04-25T00:30:46.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.530+0000 7f4e723da640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4e6c14c720 0x7f4e6c1563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:46.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.530+0000 7f4e723da640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f4e6c14c720 0x7f4e6c1563d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40612/0 (socket says 172.21.15.116:40612) 2024-04-25T00:30:46.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.530+0000 7f4e723da640 1 -- 172.21.15.116:0/2548825348 learned_addr learned my addr 172.21.15.116:0/2548825348 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:46.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.530+0000 7f4e71bd9640 1 --2- 172.21.15.116:0/2548825348 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4e6c0a94a0 0x7f4e6c0c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:46.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.530+0000 7f4e723da640 1 -- 172.21.15.116:0/2548825348 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4e6c0a94a0 msgr2=0x7f4e6c0c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:46.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.530+0000 7f4e723da640 1 --2- 172.21.15.116:0/2548825348 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4e6c0a94a0 0x7f4e6c0c2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:46.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.530+0000 7f4e723da640 1 -- 172.21.15.116:0/2548825348 >> v1:172.21.15.174:6789/0 conn(0x7f4e6c0c2f50 legacy=0x7f4e6c071680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:46.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.530+0000 7f4e723da640 1 -- 172.21.15.116:0/2548825348 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e6c14cb00 con 0x7f4e6c14c720 2024-04-25T00:30:46.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.530+0000 7f4e71bd9640 1 --2- 172.21.15.116:0/2548825348 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4e6c0a94a0 0x7f4e6c0c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:30:46.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.530+0000 7f4e723da640 1 --2- 172.21.15.116:0/2548825348 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e6c14c720 0x7f4e6c1563d0 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f4e68009870 tx=0x7f4e6801c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f97736281d30333c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:46.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.530+0000 7f4e713d8640 1 -- 172.21.15.116:0/2548825348 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e680aa620 con 0x7f4e6c14c720 2024-04-25T00:30:46.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.530+0000 7f4e713d8640 1 -- 172.21.15.116:0/2548825348 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e680af070 con 0x7f4e6c14c720 2024-04-25T00:30:46.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.530+0000 7f4e713d8640 1 -- 172.21.15.116:0/2548825348 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e680aa9e0 con 0x7f4e6c14c720 2024-04-25T00:30:46.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.530+0000 7f4e7463e640 1 -- 172.21.15.116:0/2548825348 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e6c14c720 msgr2=0x7f4e6c1563d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:46.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.530+0000 7f4e7463e640 1 --2- 172.21.15.116:0/2548825348 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e6c14c720 0x7f4e6c1563d0 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f4e68009870 tx=0x7f4e6801c4a0 comp rx=0 tx=0).stop 2024-04-25T00:30:46.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.530+0000 7f4e7463e640 1 -- 172.21.15.116:0/2548825348 shutdown_connections 2024-04-25T00:30:46.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.530+0000 7f4e7463e640 1 --2- 172.21.15.116:0/2548825348 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4e6c0a94a0 0x7f4e6c0c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:46.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.530+0000 7f4e7463e640 1 --2- 172.21.15.116:0/2548825348 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e6c14c720 0x7f4e6c1563d0 unknown :-1 s=CLOSED pgs=1837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:46.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.530+0000 7f4e7463e640 1 -- 172.21.15.116:0/2548825348 >> 172.21.15.116:0/2548825348 conn(0x7f4e6c0c3b00 msgr2=0x7f4e6c0c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:46.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.530+0000 7f4e7463e640 1 -- 172.21.15.116:0/2548825348 shutdown_connections 2024-04-25T00:30:46.533 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.530+0000 7f4e7463e640 1 -- 172.21.15.116:0/2548825348 wait complete. 2024-04-25T00:30:46.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.534+0000 7f4e7463e640 1 Processor -- start 2024-04-25T00:30:46.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.534+0000 7f4e7463e640 1 -- start start 2024-04-25T00:30:46.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.534+0000 7f4e7463e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e6c0a94a0 0x7f4e6c14ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:46.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.534+0000 7f4e7463e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4e6c0c2f50 0x7f4e6c13fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:46.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.534+0000 7f4e7463e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4e6c14c720 0x7f4e6c1ed830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:46.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.534+0000 7f4e7463e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f4e6c152f40 con 0x7f4e6c0a94a0 2024-04-25T00:30:46.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.534+0000 7f4e7463e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f4e6c152c60 con 0x7f4e6c14c720 2024-04-25T00:30:46.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.534+0000 7f4e7463e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f4e6c06cc50 con 0x7f4e6c0c2f50 2024-04-25T00:30:46.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.534+0000 7f4e723da640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e6c0a94a0 0x7f4e6c14ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:46.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.534+0000 7f4e723da640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e6c0a94a0 0x7f4e6c14ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40624/0 (socket says 172.21.15.116:40624) 2024-04-25T00:30:46.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.534+0000 7f4e723da640 1 -- 172.21.15.116:0/222194569 learned_addr learned my addr 172.21.15.116:0/222194569 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:46.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.534+0000 7f4e723da640 1 -- 172.21.15.116:0/222194569 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4e6c0c2f50 msgr2=0x7f4e6c13fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:46.536 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.534+0000 7f4e723da640 1 --2- 172.21.15.116:0/222194569 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4e6c0c2f50 0x7f4e6c13fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:46.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.534+0000 7f4e72bdb640 1 --2- 172.21.15.116:0/222194569 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4e6c14c720 0x7f4e6c1ed830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:46.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.534+0000 7f4e723da640 1 -- 172.21.15.116:0/222194569 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4e6c14c720 msgr2=0x7f4e6c1ed830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:46.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.534+0000 7f4e723da640 1 --2- 172.21.15.116:0/222194569 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4e6c14c720 0x7f4e6c1ed830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:46.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.534+0000 7f4e723da640 1 -- 172.21.15.116:0/222194569 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e6c13cfc0 con 0x7f4e6c0a94a0 2024-04-25T00:30:46.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.534+0000 7f4e723da640 1 --2- 172.21.15.116:0/222194569 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e6c0a94a0 0x7f4e6c14ab90 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f4e680ac040 tx=0x7f4e6801ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:46.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.534+0000 7f4e62ffd640 1 -- 172.21.15.116:0/222194569 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e680aae10 con 0x7f4e6c0a94a0 2024-04-25T00:30:46.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.534+0000 7f4e62ffd640 1 -- 172.21.15.116:0/222194569 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e680bd3f0 con 0x7f4e6c0a94a0 2024-04-25T00:30:46.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.534+0000 7f4e62ffd640 1 -- 172.21.15.116:0/222194569 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e680bd7e0 con 0x7f4e6c0a94a0 2024-04-25T00:30:46.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.534+0000 7f4e7463e640 1 -- 172.21.15.116:0/222194569 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e6c149b80 con 0x7f4e6c0a94a0 2024-04-25T00:30:46.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.534+0000 7f4e7463e640 1 -- 172.21.15.116:0/222194569 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e6c1554b0 con 0x7f4e6c0a94a0 2024-04-25T00:30:46.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.534+0000 7f4e7463e640 1 -- 172.21.15.116:0/222194569 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e6c068350 con 0x7f4e6c0a94a0 2024-04-25T00:30:46.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.534+0000 7f4e62ffd640 1 -- 172.21.15.116:0/222194569 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f4e680b33f0 con 0x7f4e6c0a94a0 2024-04-25T00:30:46.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.538+0000 7f4e62ffd640 1 --2- 172.21.15.116:0/222194569 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4e500678c0 0x7f4e50069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:46.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.538+0000 7f4e62ffd640 1 -- 172.21.15.116:0/222194569 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7f4e68121480 con 0x7f4e6c0a94a0 2024-04-25T00:30:46.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.538+0000 7f4e71bd9640 1 --2- 172.21.15.116:0/222194569 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4e500678c0 0x7f4e50069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:46.539 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.538+0000 7f4e71bd9640 1 --2- 172.21.15.116:0/222194569 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4e500678c0 0x7f4e50069d80 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c149910 tx=0x7f4e5c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:46.541 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.542+0000 7f4e62ffd640 1 -- 172.21.15.116:0/222194569 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f4e68016150 con 0x7f4e6c0a94a0 2024-04-25T00:30:46.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.702+0000 7f4e7463e640 1 -- 172.21.15.116:0/222194569 --> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7f4e6c13cd70 con 0x7f4e500678c0 2024-04-25T00:30:46.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.702+0000 7f4e62ffd640 1 -- 172.21.15.116:0/222194569 <== mgr.4105 v2:172.21.15.174:6832/2004448211 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f4e6c13cd70 con 0x7f4e500678c0 2024-04-25T00:30:46.704 INFO:teuthology.orchestra.run.smithi116.stderr:instructed osd(s) 6 to deep-scrub 2024-04-25T00:30:46.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.706+0000 7f4e7463e640 1 -- 172.21.15.116:0/222194569 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4e500678c0 msgr2=0x7f4e50069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:46.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.706+0000 7f4e7463e640 1 --2- 172.21.15.116:0/222194569 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4e500678c0 0x7f4e50069d80 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c149910 tx=0x7f4e5c008040 comp rx=0 tx=0).stop 2024-04-25T00:30:46.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.706+0000 7f4e7463e640 1 -- 172.21.15.116:0/222194569 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e6c0a94a0 msgr2=0x7f4e6c14ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:46.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.706+0000 7f4e7463e640 1 --2- 172.21.15.116:0/222194569 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e6c0a94a0 0x7f4e6c14ab90 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f4e680ac040 tx=0x7f4e6801ace0 comp rx=0 tx=0).stop 2024-04-25T00:30:46.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.706+0000 7f4e7463e640 1 -- 172.21.15.116:0/222194569 shutdown_connections 2024-04-25T00:30:46.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.706+0000 7f4e7463e640 1 --2- 172.21.15.116:0/222194569 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f4e500678c0 0x7f4e50069d80 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:46.707 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.706+0000 7f4e7463e640 1 --2- 172.21.15.116:0/222194569 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4e6c14c720 0x7f4e6c1ed830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:46.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.706+0000 7f4e7463e640 1 --2- 172.21.15.116:0/222194569 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f4e6c0c2f50 0x7f4e6c13fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:46.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.706+0000 7f4e7463e640 1 --2- 172.21.15.116:0/222194569 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f4e6c0a94a0 0x7f4e6c14ab90 unknown :-1 s=CLOSED pgs=1838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:46.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.706+0000 7f4e7463e640 1 -- 172.21.15.116:0/222194569 >> 172.21.15.116:0/222194569 conn(0x7f4e6c0c3b00 msgr2=0x7f4e6c06e510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:46.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.706+0000 7f4e7463e640 1 -- 172.21.15.116:0/222194569 shutdown_connections 2024-04-25T00:30:46.708 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.706+0000 7f4e7463e640 1 -- 172.21.15.116:0/222194569 wait complete. 2024-04-25T00:30:46.724 INFO:tasks.ceph:Scrubbing osd.7 2024-04-25T00:30:46.724 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-04-25T00:30:46.846 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.846+0000 7f56c5763640 1 Processor -- start 2024-04-25T00:30:46.847 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.846+0000 7f56c5763640 1 -- start start 2024-04-25T00:30:46.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.846+0000 7f56c5763640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f56c014f0d0 0x7f56c0155bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:46.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.846+0000 7f56c5763640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56c00c3eb0 con 0x7f56c006f600 2024-04-25T00:30:46.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.846+0000 7f56c5763640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56c0156110 con 0x7f56c006fa60 2024-04-25T00:30:46.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.846+0000 7f56c5763640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f56c0151e00 con 0x7f56c014f0d0 2024-04-25T00:30:46.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.846+0000 7f56be7fc640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f56c006f600 0x7f56c0152290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:43712/0 (socket says 172.21.15.116:43712) 2024-04-25T00:30:46.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.846+0000 7f56be7fc640 1 -- 172.21.15.116:0/551201258 learned_addr learned my addr 172.21.15.116:0/551201258 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:30:46.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.846+0000 7f56beffd640 1 --2- 172.21.15.116:0/551201258 >> v2:172.21.15.174:3300/0 conn(0x7f56c014f0d0 0x7f56c0155bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:46.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.846+0000 7f56bdffb640 1 -- 172.21.15.116:0/551201258 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1318541184 0 0) 0x7f56c00c3eb0 con 0x7f56c006f600 2024-04-25T00:30:46.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.846+0000 7f56bdffb640 1 -- 172.21.15.116:0/551201258 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f56ac003530 con 0x7f56c006f600 2024-04-25T00:30:46.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.846+0000 7f56bdffb640 1 -- 172.21.15.116:0/551201258 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 436258100 0 0) 0x7f56ac003530 con 0x7f56c006f600 2024-04-25T00:30:46.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.846+0000 7f56bdffb640 1 -- 172.21.15.116:0/551201258 >> v2:172.21.15.174:3300/0 conn(0x7f56c014f0d0 msgr2=0x7f56c0155bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:46.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.846+0000 7f56bdffb640 1 --2- 172.21.15.116:0/551201258 >> v2:172.21.15.174:3300/0 conn(0x7f56c014f0d0 0x7f56c0155bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:46.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.846+0000 7f56bdffb640 1 -- 172.21.15.116:0/551201258 >> v1:172.21.15.174:6789/0 conn(0x7f56c006fa60 legacy=0x7f56c015c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:46.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.846+0000 7f56bdffb640 1 -- 172.21.15.116:0/551201258 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56c00c3330 con 0x7f56c006f600 2024-04-25T00:30:46.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.846+0000 7f56bdffb640 1 -- 172.21.15.116:0/551201258 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f56a80016a0 con 0x7f56c006f600 2024-04-25T00:30:46.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.846+0000 7f56bdffb640 1 -- 172.21.15.116:0/551201258 >> v1:172.21.15.116:6789/0 conn(0x7f56c006f600 legacy=0x7f56c0152290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:46.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.846+0000 7f56bdffb640 1 --2- 172.21.15.116:0/551201258 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f56ac0039c0 0x7f56ac003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:46.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.846+0000 7f56bdffb640 1 --2- 172.21.15.116:0/551201258 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56ac004810 0x7f56ac004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:46.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.846+0000 7f56bdffb640 1 --2- 172.21.15.116:0/551201258 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56ac0051d0 0x7f56ac00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:46.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.846+0000 7f56bdffb640 1 -- 172.21.15.116:0/551201258 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f56c0151e00 con 0x7f56ac004810 2024-04-25T00:30:46.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.846+0000 7f56bdffb640 1 -- 172.21.15.116:0/551201258 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f56ac00c170 con 0x7f56ac0039c0 2024-04-25T00:30:46.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.846+0000 7f56bdffb640 1 -- 172.21.15.116:0/551201258 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f56ac00c370 con 0x7f56ac0051d0 2024-04-25T00:30:46.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.846+0000 7f56be7fc640 1 --2- 172.21.15.116:0/551201258 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56ac004810 0x7f56ac004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:46.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.846+0000 7f56beffd640 1 --2- 172.21.15.116:0/551201258 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f56ac0039c0 0x7f56ac003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:46.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.846+0000 7f56bf7fe640 1 --2- 172.21.15.116:0/551201258 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56ac0051d0 0x7f56ac00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:46.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.846+0000 7f56be7fc640 1 -- 172.21.15.116:0/551201258 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56ac0051d0 msgr2=0x7f56ac00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:46.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.846+0000 7f56be7fc640 1 --2- 172.21.15.116:0/551201258 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56ac0051d0 0x7f56ac00b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:46.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.846+0000 7f56be7fc640 1 -- 172.21.15.116:0/551201258 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f56ac0039c0 msgr2=0x7f56ac003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:46.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.846+0000 7f56be7fc640 1 --2- 172.21.15.116:0/551201258 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f56ac0039c0 0x7f56ac003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:46.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.846+0000 7f56be7fc640 1 -- 172.21.15.116:0/551201258 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56ac003530 con 0x7f56ac004810 2024-04-25T00:30:46.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.846+0000 7f56bf7fe640 1 -- 172.21.15.116:0/551201258 reap_dead start 2024-04-25T00:30:46.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.850+0000 7f56be7fc640 1 --2- 172.21.15.116:0/551201258 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56ac004810 0x7f56ac004c90 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7f56a800c210 tx=0x7f56a801cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f49db161677a2d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:46.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.850+0000 7f56bdffb640 1 -- 172.21.15.116:0/551201258 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56a8007d40 con 0x7f56ac004810 2024-04-25T00:30:46.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.850+0000 7f56bdffb640 1 -- 172.21.15.116:0/551201258 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56a80a9070 con 0x7f56ac004810 2024-04-25T00:30:46.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.850+0000 7f56bdffb640 1 -- 172.21.15.116:0/551201258 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56a8005d40 con 0x7f56ac004810 2024-04-25T00:30:46.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.850+0000 7f56c5763640 1 -- 172.21.15.116:0/551201258 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56ac004810 msgr2=0x7f56ac004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:46.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.850+0000 7f56c5763640 1 --2- 172.21.15.116:0/551201258 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56ac004810 0x7f56ac004c90 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7f56a800c210 tx=0x7f56a801cc60 comp rx=0 tx=0).stop 2024-04-25T00:30:46.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.850+0000 7f56c5763640 1 -- 172.21.15.116:0/551201258 shutdown_connections 2024-04-25T00:30:46.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.850+0000 7f56c5763640 1 --2- 172.21.15.116:0/551201258 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56ac004810 0x7f56ac004c90 unknown :-1 s=CLOSED pgs=1840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:46.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.850+0000 7f56c5763640 1 -- 172.21.15.116:0/551201258 >> 172.21.15.116:0/551201258 conn(0x7f56c006b210 msgr2=0x7f56c006b620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:46.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.850+0000 7f56c5763640 1 -- 172.21.15.116:0/551201258 shutdown_connections 2024-04-25T00:30:46.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.850+0000 7f56c5763640 1 -- 172.21.15.116:0/551201258 wait complete. 2024-04-25T00:30:46.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.850+0000 7f56c5763640 1 Processor -- start 2024-04-25T00:30:46.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.850+0000 7f56c5763640 1 -- start start 2024-04-25T00:30:46.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.854+0000 7f56c5763640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f56ac004810 0x7f56c00a95a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:46.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.854+0000 7f56c5763640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56c01dd240 0x7f56c00abe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:46.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.854+0000 7f56c5763640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56c014cfb0 0x7f56c00ac370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:46.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.854+0000 7f56c5763640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f56c006c890 con 0x7f56c01dd240 2024-04-25T00:30:46.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.854+0000 7f56c5763640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f56c006fe70 con 0x7f56ac004810 2024-04-25T00:30:46.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.854+0000 7f56c5763640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f56c006ca10 con 0x7f56c014cfb0 2024-04-25T00:30:46.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.854+0000 7f56be7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56c01dd240 0x7f56c00abe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:46.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.854+0000 7f56be7fc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56c01dd240 0x7f56c00abe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40638/0 (socket says 172.21.15.116:40638) 2024-04-25T00:30:46.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.854+0000 7f56be7fc640 1 -- 172.21.15.116:0/2984556277 learned_addr learned my addr 172.21.15.116:0/2984556277 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:46.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.854+0000 7f56beffd640 1 --2- 172.21.15.116:0/2984556277 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f56ac004810 0x7f56c00a95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:46.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.854+0000 7f56bf7fe640 1 --2- 172.21.15.116:0/2984556277 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56c014cfb0 0x7f56c00ac370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:46.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.854+0000 7f56be7fc640 1 -- 172.21.15.116:0/2984556277 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56c014cfb0 msgr2=0x7f56c00ac370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:46.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.854+0000 7f56be7fc640 1 --2- 172.21.15.116:0/2984556277 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56c014cfb0 0x7f56c00ac370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:46.856 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.854+0000 7f56be7fc640 1 -- 172.21.15.116:0/2984556277 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f56ac004810 msgr2=0x7f56c00a95a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:46.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.854+0000 7f56be7fc640 1 --2- 172.21.15.116:0/2984556277 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f56ac004810 0x7f56c00a95a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:46.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.854+0000 7f56be7fc640 1 -- 172.21.15.116:0/2984556277 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56c00ab3c0 con 0x7f56c01dd240 2024-04-25T00:30:46.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.854+0000 7f56be7fc640 1 --2- 172.21.15.116:0/2984556277 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56c01dd240 0x7f56c00abe30 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f56a801c6d0 tx=0x7f56ac005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:46.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.854+0000 7f56b77fe640 1 -- 172.21.15.116:0/2984556277 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56a8007d40 con 0x7f56c01dd240 2024-04-25T00:30:46.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.854+0000 7f56b77fe640 1 -- 172.21.15.116:0/2984556277 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56a8004e50 con 0x7f56c01dd240 2024-04-25T00:30:46.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.854+0000 7f56c5763640 1 -- 172.21.15.116:0/2984556277 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56c01ee3f0 con 0x7f56c01dd240 2024-04-25T00:30:46.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.854+0000 7f56b77fe640 1 -- 172.21.15.116:0/2984556277 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56a8001660 con 0x7f56c01dd240 2024-04-25T00:30:46.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.854+0000 7f56c5763640 1 -- 172.21.15.116:0/2984556277 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56c00ab630 con 0x7f56c01dd240 2024-04-25T00:30:46.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.854+0000 7f56c5763640 1 -- 172.21.15.116:0/2984556277 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f56c0068a60 con 0x7f56c01dd240 2024-04-25T00:30:46.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.854+0000 7f56b77fe640 1 -- 172.21.15.116:0/2984556277 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f56a80b6030 con 0x7f56c01dd240 2024-04-25T00:30:46.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.854+0000 7f56b77fe640 1 --2- 172.21.15.116:0/2984556277 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f56a00678c0 0x7f56a0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:46.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.854+0000 7f56b77fe640 1 -- 172.21.15.116:0/2984556277 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7f56a8120650 con 0x7f56c01dd240 2024-04-25T00:30:46.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.854+0000 7f56b77fe640 1 --2- 172.21.15.116:0/2984556277 >> [v2:172.21.15.174:6816/2543899196,v1:172.21.15.174:6817/2543899196] conn(0x7f56a00746b0 0x7f56a0076b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:46.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.854+0000 7f56b77fe640 1 -- 172.21.15.116:0/2984556277 --> [v2:172.21.15.174:6816/2543899196,v1:172.21.15.174:6817/2543899196] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f56a0077220 con 0x7f56a00746b0 2024-04-25T00:30:46.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.854+0000 7f56b77fe640 1 -- 172.21.15.116:0/2984556277 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_get_version_reply(handle=1 version=74) ==== 24+0+0 (secure 0 0 0) 0x7f56a8120aa0 con 0x7f56c01dd240 2024-04-25T00:30:46.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.854+0000 7f56beffd640 1 --2- 172.21.15.116:0/2984556277 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f56a00678c0 0x7f56a0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:46.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.854+0000 7f56bf7fe640 1 --2- 172.21.15.116:0/2984556277 >> [v2:172.21.15.174:6816/2543899196,v1:172.21.15.174:6817/2543899196] conn(0x7f56a00746b0 0x7f56a0076b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:46.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.854+0000 7f56beffd640 1 --2- 172.21.15.116:0/2984556277 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f56a00678c0 0x7f56a0069d80 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f56b8007520 tx=0x7f56b800b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:46.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.854+0000 7f56bf7fe640 1 --2- 172.21.15.116:0/2984556277 >> [v2:172.21.15.174:6816/2543899196,v1:172.21.15.174:6817/2543899196] conn(0x7f56a00746b0 0x7f56a0076b50 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:46.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.858+0000 7f56b77fe640 1 -- 172.21.15.116:0/2984556277 <== osd.7 v2:172.21.15.174:6816/2543899196 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f56a0077220 con 0x7f56a00746b0 2024-04-25T00:30:46.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.870+0000 7f56c5763640 1 -- 172.21.15.116:0/2984556277 --> [v2:172.21.15.174:6816/2543899196,v1:172.21.15.174:6817/2543899196] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f56c014bc90 con 0x7f56a00746b0 2024-04-25T00:30:46.871 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.870+0000 7f56b77fe640 1 -- 172.21.15.116:0/2984556277 <== osd.7 v2:172.21.15.174:6816/2543899196 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f56c014bc90 con 0x7f56a00746b0 2024-04-25T00:30:46.872 INFO:teuthology.orchestra.run.smithi116.stdout:{ 2024-04-25T00:30:46.872 INFO:teuthology.orchestra.run.smithi116.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T00:30:46.872 INFO:teuthology.orchestra.run.smithi116.stdout:} 2024-04-25T00:30:46.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.870+0000 7f56c5763640 1 -- 172.21.15.116:0/2984556277 >> [v2:172.21.15.174:6816/2543899196,v1:172.21.15.174:6817/2543899196] conn(0x7f56a00746b0 msgr2=0x7f56a0076b50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:46.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.870+0000 7f56c5763640 1 --2- 172.21.15.116:0/2984556277 >> [v2:172.21.15.174:6816/2543899196,v1:172.21.15.174:6817/2543899196] conn(0x7f56a00746b0 0x7f56a0076b50 crc :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:46.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.870+0000 7f56c5763640 1 -- 172.21.15.116:0/2984556277 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f56a00678c0 msgr2=0x7f56a0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:46.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.870+0000 7f56c5763640 1 --2- 172.21.15.116:0/2984556277 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f56a00678c0 0x7f56a0069d80 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f56b8007520 tx=0x7f56b800b040 comp rx=0 tx=0).stop 2024-04-25T00:30:46.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.870+0000 7f56c5763640 1 -- 172.21.15.116:0/2984556277 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56c01dd240 msgr2=0x7f56c00abe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:46.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.870+0000 7f56c5763640 1 --2- 172.21.15.116:0/2984556277 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56c01dd240 0x7f56c00abe30 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f56a801c6d0 tx=0x7f56ac005840 comp rx=0 tx=0).stop 2024-04-25T00:30:46.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.870+0000 7f56bf7fe640 1 -- 172.21.15.116:0/2984556277 reap_dead start 2024-04-25T00:30:46.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.870+0000 7f56c5763640 1 -- 172.21.15.116:0/2984556277 shutdown_connections 2024-04-25T00:30:46.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.870+0000 7f56c5763640 1 -- 172.21.15.116:0/2984556277 >> 172.21.15.116:0/2984556277 conn(0x7f56c006b210 msgr2=0x7f56c0151060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:46.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.870+0000 7f56c5763640 1 -- 172.21.15.116:0/2984556277 shutdown_connections 2024-04-25T00:30:46.873 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.870+0000 7f56c5763640 1 -- 172.21.15.116:0/2984556277 wait complete. 2024-04-25T00:30:46.886 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-04-25T00:30:47.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.998+0000 7f8048391640 1 Processor -- start 2024-04-25T00:30:47.000 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.998+0000 7f8048391640 1 -- start start 2024-04-25T00:30:47.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.998+0000 7f8048391640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f80400a99f0 0x7f8040153ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:47.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.998+0000 7f8048391640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80400c3910 con 0x7f80400a94a0 2024-04-25T00:30:47.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.998+0000 7f8048391640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8040154530 con 0x7f8040150d20 2024-04-25T00:30:47.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:46.998+0000 7f8048391640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f80400c16e0 con 0x7f80400a99f0 2024-04-25T00:30:47.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.002+0000 7f804592c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f80400a94a0 0x7f80401538e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:43726/0 (socket says 172.21.15.116:43726) 2024-04-25T00:30:47.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.002+0000 7f804592c640 1 -- 172.21.15.116:0/3931238721 learned_addr learned my addr 172.21.15.116:0/3931238721 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:30:47.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.002+0000 7f804692e640 1 --2- 172.21.15.116:0/3931238721 >> v2:172.21.15.174:3300/0 conn(0x7f80400a99f0 0x7f8040153ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:47.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.002+0000 7f804512b640 1 -- 172.21.15.116:0/3931238721 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1663893249 0 0) 0x7f80400c3910 con 0x7f80400a94a0 2024-04-25T00:30:47.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.002+0000 7f804512b640 1 -- 172.21.15.116:0/3931238721 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f802c003530 con 0x7f80400a94a0 2024-04-25T00:30:47.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.002+0000 7f804512b640 1 -- 172.21.15.116:0/3931238721 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3885559987 0 0) 0x7f802c003530 con 0x7f80400a94a0 2024-04-25T00:30:47.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.002+0000 7f804512b640 1 -- 172.21.15.116:0/3931238721 >> v2:172.21.15.174:3300/0 conn(0x7f80400a99f0 msgr2=0x7f8040153ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:47.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.002+0000 7f804512b640 1 --2- 172.21.15.116:0/3931238721 >> v2:172.21.15.174:3300/0 conn(0x7f80400a99f0 0x7f8040153ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:47.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.002+0000 7f804512b640 1 -- 172.21.15.116:0/3931238721 >> v1:172.21.15.174:6789/0 conn(0x7f8040150d20 legacy=0x7f80400c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:47.003 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.002+0000 7f804512b640 1 -- 172.21.15.116:0/3931238721 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80400c1400 con 0x7f80400a94a0 2024-04-25T00:30:47.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.002+0000 7f804512b640 1 -- 172.21.15.116:0/3931238721 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f8030001680 con 0x7f80400a94a0 2024-04-25T00:30:47.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.002+0000 7f804512b640 1 -- 172.21.15.116:0/3931238721 >> v1:172.21.15.116:6789/0 conn(0x7f80400a94a0 legacy=0x7f80401538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:47.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.002+0000 7f804512b640 1 --2- 172.21.15.116:0/3931238721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f802c0039c0 0x7f802c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:47.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.002+0000 7f804512b640 1 --2- 172.21.15.116:0/3931238721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f802c004810 0x7f802c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:47.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.002+0000 7f804512b640 1 --2- 172.21.15.116:0/3931238721 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f802c0051d0 0x7f802c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:47.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.002+0000 7f804512b640 1 -- 172.21.15.116:0/3931238721 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f80400c16e0 con 0x7f802c004810 2024-04-25T00:30:47.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.002+0000 7f804512b640 1 -- 172.21.15.116:0/3931238721 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f802c00c070 con 0x7f802c0039c0 2024-04-25T00:30:47.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.002+0000 7f804512b640 1 -- 172.21.15.116:0/3931238721 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f802c00c270 con 0x7f802c0051d0 2024-04-25T00:30:47.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.002+0000 7f804592c640 1 --2- 172.21.15.116:0/3931238721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f802c004810 0x7f802c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:47.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.002+0000 7f804592c640 1 -- 172.21.15.116:0/3931238721 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f802c0051d0 msgr2=0x7f802c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:47.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.002+0000 7f804592c640 1 --2- 172.21.15.116:0/3931238721 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f802c0051d0 0x7f802c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:47.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.002+0000 7f804612d640 1 --2- 172.21.15.116:0/3931238721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f802c0039c0 0x7f802c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:47.004 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.002+0000 7f804592c640 1 -- 172.21.15.116:0/3931238721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f802c0039c0 msgr2=0x7f802c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:47.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.002+0000 7f804592c640 1 --2- 172.21.15.116:0/3931238721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f802c0039c0 0x7f802c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:47.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.002+0000 7f804692e640 1 -- 172.21.15.116:0/3931238721 reap_dead start 2024-04-25T00:30:47.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.002+0000 7f804592c640 1 -- 172.21.15.116:0/3931238721 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f802c003530 con 0x7f802c004810 2024-04-25T00:30:47.005 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.002+0000 7f804612d640 1 --2- 172.21.15.116:0/3931238721 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f802c0039c0 0x7f802c003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:30:47.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.002+0000 7f804592c640 1 --2- 172.21.15.116:0/3931238721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f802c004810 0x7f802c004c90 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f803000c210 tx=0x7f803001f3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b21e45fdfa93f951 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:47.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.002+0000 7f804512b640 1 -- 172.21.15.116:0/3931238721 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8030007d40 con 0x7f802c004810 2024-04-25T00:30:47.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.002+0000 7f804512b640 1 -- 172.21.15.116:0/3931238721 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8030005b00 con 0x7f802c004810 2024-04-25T00:30:47.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.002+0000 7f804512b640 1 -- 172.21.15.116:0/3931238721 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8030007070 con 0x7f802c004810 2024-04-25T00:30:47.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.002+0000 7f8048391640 1 -- 172.21.15.116:0/3931238721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f802c004810 msgr2=0x7f802c004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:47.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.002+0000 7f8048391640 1 --2- 172.21.15.116:0/3931238721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f802c004810 0x7f802c004c90 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f803000c210 tx=0x7f803001f3b0 comp rx=0 tx=0).stop 2024-04-25T00:30:47.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.002+0000 7f8048391640 1 -- 172.21.15.116:0/3931238721 shutdown_connections 2024-04-25T00:30:47.006 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.002+0000 7f8048391640 1 --2- 172.21.15.116:0/3931238721 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f802c004810 0x7f802c004c90 unknown :-1 s=CLOSED pgs=1843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:47.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.002+0000 7f8048391640 1 -- 172.21.15.116:0/3931238721 >> 172.21.15.116:0/3931238721 conn(0x7f80400c3b00 msgr2=0x7f80400c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:47.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.002+0000 7f8048391640 1 -- 172.21.15.116:0/3931238721 shutdown_connections 2024-04-25T00:30:47.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.002+0000 7f8048391640 1 -- 172.21.15.116:0/3931238721 wait complete. 2024-04-25T00:30:47.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.006+0000 7f8048391640 1 Processor -- start 2024-04-25T00:30:47.007 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.006+0000 7f8048391640 1 -- start start 2024-04-25T00:30:47.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.006+0000 7f8048391640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f802c004810 0x7f80401eec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:47.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.006+0000 7f8048391640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80401468f0 0x7f8040145790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:47.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.006+0000 7f8048391640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f80401463c0 0x7f8040145cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:47.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.006+0000 7f8048391640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f804006ec20 con 0x7f80401468f0 2024-04-25T00:30:47.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.006+0000 7f8048391640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f804006e940 con 0x7f80401463c0 2024-04-25T00:30:47.009 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.006+0000 7f8048391640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f80400c2a10 con 0x7f802c004810 2024-04-25T00:30:47.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.006+0000 7f804592c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80401468f0 0x7f8040145790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:47.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.006+0000 7f804592c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80401468f0 0x7f8040145790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40648/0 (socket says 172.21.15.116:40648) 2024-04-25T00:30:47.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.006+0000 7f804592c640 1 -- 172.21.15.116:0/1855104895 learned_addr learned my addr 172.21.15.116:0/1855104895 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:47.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.006+0000 7f804612d640 1 --2- 172.21.15.116:0/1855104895 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f802c004810 0x7f80401eec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:47.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.006+0000 7f804692e640 1 --2- 172.21.15.116:0/1855104895 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f80401463c0 0x7f8040145cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:47.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.006+0000 7f804592c640 1 -- 172.21.15.116:0/1855104895 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f802c004810 msgr2=0x7f80401eec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:47.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.006+0000 7f804592c640 1 --2- 172.21.15.116:0/1855104895 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f802c004810 0x7f80401eec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:47.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.006+0000 7f804592c640 1 -- 172.21.15.116:0/1855104895 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f80401463c0 msgr2=0x7f8040145cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:47.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.006+0000 7f804592c640 1 --2- 172.21.15.116:0/1855104895 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f80401463c0 0x7f8040145cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:47.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.006+0000 7f804592c640 1 -- 172.21.15.116:0/1855104895 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80401ef520 con 0x7f80401468f0 2024-04-25T00:30:47.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.006+0000 7f804592c640 1 --2- 172.21.15.116:0/1855104895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80401468f0 0x7f8040145790 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f8030026020 tx=0x7f802c005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:47.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.006+0000 7f8036ffd640 1 -- 172.21.15.116:0/1855104895 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8030001680 con 0x7f80401468f0 2024-04-25T00:30:47.010 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.006+0000 7f8036ffd640 1 -- 172.21.15.116:0/1855104895 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80300b6070 con 0x7f80401468f0 2024-04-25T00:30:47.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.006+0000 7f8048391640 1 -- 172.21.15.116:0/1855104895 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80401ef800 con 0x7f80401468f0 2024-04-25T00:30:47.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.006+0000 7f8036ffd640 1 -- 172.21.15.116:0/1855104895 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8030004a10 con 0x7f80401468f0 2024-04-25T00:30:47.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.006+0000 7f8048391640 1 -- 172.21.15.116:0/1855104895 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8040154c60 con 0x7f80401468f0 2024-04-25T00:30:47.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.006+0000 7f8048391640 1 -- 172.21.15.116:0/1855104895 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8040068350 con 0x7f80401468f0 2024-04-25T00:30:47.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.010+0000 7f8036ffd640 1 -- 172.21.15.116:0/1855104895 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f8030004bb0 con 0x7f80401468f0 2024-04-25T00:30:47.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.010+0000 7f8036ffd640 1 --2- 172.21.15.116:0/1855104895 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f80240678c0 0x7f8024069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:47.011 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.010+0000 7f8036ffd640 1 -- 172.21.15.116:0/1855104895 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7f8030124510 con 0x7f80401468f0 2024-04-25T00:30:47.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.010+0000 7f804612d640 1 --2- 172.21.15.116:0/1855104895 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f80240678c0 0x7f8024069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:47.012 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.010+0000 7f804612d640 1 --2- 172.21.15.116:0/1855104895 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f80240678c0 0x7f8024069d80 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f804014bd00 tx=0x7f803c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:47.014 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.014+0000 7f8036ffd640 1 -- 172.21.15.116:0/1855104895 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f80300190c0 con 0x7f80401468f0 2024-04-25T00:30:47.174 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.174+0000 7f8048391640 1 -- 172.21.15.116:0/1855104895 --> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7f804006d820 con 0x7f80240678c0 2024-04-25T00:30:47.175 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.174+0000 7f8036ffd640 1 -- 172.21.15.116:0/1855104895 <== mgr.4105 v2:172.21.15.174:6832/2004448211 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f804006d820 con 0x7f80240678c0 2024-04-25T00:30:47.175 INFO:teuthology.orchestra.run.smithi116.stderr:instructed osd(s) 7 to deep-scrub 2024-04-25T00:30:47.177 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.174+0000 7f8048391640 1 -- 172.21.15.116:0/1855104895 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f80240678c0 msgr2=0x7f8024069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:47.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.174+0000 7f8048391640 1 --2- 172.21.15.116:0/1855104895 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f80240678c0 0x7f8024069d80 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f804014bd00 tx=0x7f803c009040 comp rx=0 tx=0).stop 2024-04-25T00:30:47.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.174+0000 7f8048391640 1 -- 172.21.15.116:0/1855104895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80401468f0 msgr2=0x7f8040145790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:47.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.174+0000 7f8048391640 1 --2- 172.21.15.116:0/1855104895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80401468f0 0x7f8040145790 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f8030026020 tx=0x7f802c005840 comp rx=0 tx=0).stop 2024-04-25T00:30:47.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.178+0000 7f8048391640 1 -- 172.21.15.116:0/1855104895 shutdown_connections 2024-04-25T00:30:47.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.178+0000 7f8048391640 1 --2- 172.21.15.116:0/1855104895 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f80240678c0 0x7f8024069d80 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:47.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.178+0000 7f8048391640 1 --2- 172.21.15.116:0/1855104895 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f80401463c0 0x7f8040145cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:47.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.178+0000 7f8048391640 1 --2- 172.21.15.116:0/1855104895 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f80401468f0 0x7f8040145790 unknown :-1 s=CLOSED pgs=1844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:47.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.178+0000 7f8048391640 1 --2- 172.21.15.116:0/1855104895 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f802c004810 0x7f80401eec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:47.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.178+0000 7f8048391640 1 -- 172.21.15.116:0/1855104895 >> 172.21.15.116:0/1855104895 conn(0x7f80400c3b00 msgr2=0x7f80400a9fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:47.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.178+0000 7f8048391640 1 -- 172.21.15.116:0/1855104895 shutdown_connections 2024-04-25T00:30:47.178 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.178+0000 7f8048391640 1 -- 172.21.15.116:0/1855104895 wait complete. 2024-04-25T00:30:47.193 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T00:30:47.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.310+0000 7fa8cd239640 1 Processor -- start 2024-04-25T00:30:47.312 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.310+0000 7fa8cd239640 1 -- start start 2024-04-25T00:30:47.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.310+0000 7fa8cd239640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa8c806d5f0 0x7fa8c8156c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:47.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.310+0000 7fa8cd239640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa8c8155810 0x7fa8c81571c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:47.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.310+0000 7fa8cd239640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8c80c9150 con 0x7fa8c8152d80 2024-04-25T00:30:47.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.310+0000 7fa8cd239640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fa8c80c9d60 con 0x7fa8c806d5f0 2024-04-25T00:30:47.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.310+0000 7fa8cd239640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa8c80c9430 con 0x7fa8c8155810 2024-04-25T00:30:47.313 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.310+0000 7fa8bffff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa8c806d5f0 0x7fa8c8156c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:47.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.310+0000 7fa8bffff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa8c806d5f0 0x7fa8c8156c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40664/0 (socket says 172.21.15.116:40664) 2024-04-25T00:30:47.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.310+0000 7fa8bffff640 1 -- 172.21.15.116:0/1438517495 learned_addr learned my addr 172.21.15.116:0/1438517495 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:47.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.310+0000 7fa8c759e640 1 --2- 172.21.15.116:0/1438517495 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa8c8155810 0x7fa8c81571c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:47.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.314+0000 7fa8bffff640 1 -- 172.21.15.116:0/1438517495 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa8c8155810 msgr2=0x7fa8c81571c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:30:47.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.314+0000 7fa8bffff640 1 --2- 172.21.15.116:0/1438517495 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa8c8155810 0x7fa8c81571c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:47.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.314+0000 7fa8bffff640 1 -- 172.21.15.116:0/1438517495 >> v1:172.21.15.174:6789/0 conn(0x7fa8c8152d80 legacy=0x7fa8c806bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:47.314 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.314+0000 7fa8bffff640 1 -- 172.21.15.116:0/1438517495 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8c80c9710 con 0x7fa8c806d5f0 2024-04-25T00:30:47.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.314+0000 7fa8bffff640 1 --2- 172.21.15.116:0/1438517495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8c806d5f0 0x7fa8c8156c80 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7fa8b0009870 tx=0x7fa8b001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dd28c3ddca0a42a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:47.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.314+0000 7fa8c659c640 1 -- 172.21.15.116:0/1438517495 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b00b1070 con 0x7fa8c806d5f0 2024-04-25T00:30:47.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.314+0000 7fa8c659c640 1 -- 172.21.15.116:0/1438517495 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8b00ac630 con 0x7fa8c806d5f0 2024-04-25T00:30:47.315 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.314+0000 7fa8c659c640 1 -- 172.21.15.116:0/1438517495 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b00a8050 con 0x7fa8c806d5f0 2024-04-25T00:30:47.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.314+0000 7fa8cd239640 1 -- 172.21.15.116:0/1438517495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8c806d5f0 msgr2=0x7fa8c8156c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:47.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.314+0000 7fa8cd239640 1 --2- 172.21.15.116:0/1438517495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8c806d5f0 0x7fa8c8156c80 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7fa8b0009870 tx=0x7fa8b001c4a0 comp rx=0 tx=0).stop 2024-04-25T00:30:47.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.314+0000 7fa8cd239640 1 -- 172.21.15.116:0/1438517495 shutdown_connections 2024-04-25T00:30:47.316 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.314+0000 7fa8cd239640 1 --2- 172.21.15.116:0/1438517495 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa8c8155810 0x7fa8c81571c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:47.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.314+0000 7fa8cd239640 1 --2- 172.21.15.116:0/1438517495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8c806d5f0 0x7fa8c8156c80 unknown :-1 s=CLOSED pgs=1845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:47.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.314+0000 7fa8cd239640 1 -- 172.21.15.116:0/1438517495 >> 172.21.15.116:0/1438517495 conn(0x7fa8c80acdd0 msgr2=0x7fa8c80ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:47.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.314+0000 7fa8cd239640 1 -- 172.21.15.116:0/1438517495 shutdown_connections 2024-04-25T00:30:47.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.314+0000 7fa8cd239640 1 -- 172.21.15.116:0/1438517495 wait complete. 2024-04-25T00:30:47.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.318+0000 7fa8cd239640 1 Processor -- start 2024-04-25T00:30:47.317 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.318+0000 7fa8cd239640 1 -- start start 2024-04-25T00:30:47.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.318+0000 7fa8cd239640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8c806d5f0 0x7fa8c8151f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:47.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.318+0000 7fa8cd239640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8c8152d80 0x7fa8c80adcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:47.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.318+0000 7fa8cd239640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa8c8155810 0x7fa8c814be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:47.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.318+0000 7fa8cd239640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c80caf00 con 0x7fa8c806d5f0 2024-04-25T00:30:47.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.318+0000 7fa8cd239640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c80cad80 con 0x7fa8c8152d80 2024-04-25T00:30:47.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.318+0000 7fa8cd239640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa8c81628d0 con 0x7fa8c8155810 2024-04-25T00:30:47.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.318+0000 7fa8c6d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8c806d5f0 0x7fa8c8151f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:47.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.318+0000 7fa8c6d9d640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8c806d5f0 0x7fa8c8151f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:40680/0 (socket says 172.21.15.116:40680) 2024-04-25T00:30:47.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.318+0000 7fa8c6d9d640 1 -- 172.21.15.116:0/3991139064 learned_addr learned my addr 172.21.15.116:0/3991139064 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:30:47.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.318+0000 7fa8c759e640 1 --2- 172.21.15.116:0/3991139064 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa8c8155810 0x7fa8c814be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:47.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.318+0000 7fa8bffff640 1 --2- 172.21.15.116:0/3991139064 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8c8152d80 0x7fa8c80adcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:47.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.318+0000 7fa8c6d9d640 1 -- 172.21.15.116:0/3991139064 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa8c8155810 msgr2=0x7fa8c814be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:47.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.318+0000 7fa8c6d9d640 1 --2- 172.21.15.116:0/3991139064 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa8c8155810 0x7fa8c814be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:47.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.318+0000 7fa8c6d9d640 1 -- 172.21.15.116:0/3991139064 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8c8152d80 msgr2=0x7fa8c80adcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:47.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.318+0000 7fa8c6d9d640 1 --2- 172.21.15.116:0/3991139064 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8c8152d80 0x7fa8c80adcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:47.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.318+0000 7fa8c6d9d640 1 -- 172.21.15.116:0/3991139064 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8c80b05a0 con 0x7fa8c806d5f0 2024-04-25T00:30:47.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.318+0000 7fa8c6d9d640 1 --2- 172.21.15.116:0/3991139064 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8c806d5f0 0x7fa8c8151f70 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7fa8b800c370 tx=0x7fa8b800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:47.320 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.318+0000 7fa8bf7fe640 1 -- 172.21.15.116:0/3991139064 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b80052d0 con 0x7fa8c806d5f0 2024-04-25T00:30:47.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.318+0000 7fa8bf7fe640 1 -- 172.21.15.116:0/3991139064 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8b8005470 con 0x7fa8c806d5f0 2024-04-25T00:30:47.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.318+0000 7fa8bf7fe640 1 -- 172.21.15.116:0/3991139064 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b8007260 con 0x7fa8c806d5f0 2024-04-25T00:30:47.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.318+0000 7fa8cd239640 1 -- 172.21.15.116:0/3991139064 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8c81ef9e0 con 0x7fa8c806d5f0 2024-04-25T00:30:47.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.318+0000 7fa8cd239640 1 -- 172.21.15.116:0/3991139064 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8c81ef700 con 0x7fa8c806d5f0 2024-04-25T00:30:47.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.318+0000 7fa8cd239640 1 -- 172.21.15.116:0/3991139064 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8c8068350 con 0x7fa8c806d5f0 2024-04-25T00:30:47.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.318+0000 7fa8bf7fe640 1 -- 172.21.15.116:0/3991139064 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa8b8007400 con 0x7fa8c806d5f0 2024-04-25T00:30:47.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.322+0000 7fa8bf7fe640 1 --2- 172.21.15.116:0/3991139064 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa8a80678c0 0x7fa8a8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:30:47.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.322+0000 7fa8bf7fe640 1 -- 172.21.15.116:0/3991139064 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7fa8b8086400 con 0x7fa8c806d5f0 2024-04-25T00:30:47.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.322+0000 7fa8bffff640 1 --2- 172.21.15.116:0/3991139064 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa8a80678c0 0x7fa8a8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:30:47.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.322+0000 7fa8bffff640 1 --2- 172.21.15.116:0/3991139064 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa8a80678c0 0x7fa8a8069d80 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fa8b00ae000 tx=0x7fa8b0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:30:47.325 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.326+0000 7fa8bf7fe640 1 -- 172.21.15.116:0/3991139064 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fa8b8055c50 con 0x7fa8c806d5f0 2024-04-25T00:30:47.493 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.490+0000 7fa8cd239640 1 -- 172.21.15.116:0/3991139064 --> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa8c80759d0 con 0x7fa8a80678c0 2024-04-25T00:30:47.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.526+0000 7fa8bf7fe640 1 -- 172.21.15.116:0/3991139064 <== mgr.4105 v2:172.21.15.174:6832/2004448211 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+492824 (secure 0 0 0) 0x7fa8c80759d0 con 0x7fa8a80678c0 2024-04-25T00:30:47.533 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:30:47.536 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-04-25T00:30:47.537 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.534+0000 7fa8cd239640 1 -- 172.21.15.116:0/3991139064 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa8a80678c0 msgr2=0x7fa8a8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:47.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.534+0000 7fa8cd239640 1 --2- 172.21.15.116:0/3991139064 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa8a80678c0 0x7fa8a8069d80 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fa8b00ae000 tx=0x7fa8b0008040 comp rx=0 tx=0).stop 2024-04-25T00:30:47.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.534+0000 7fa8cd239640 1 -- 172.21.15.116:0/3991139064 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8c806d5f0 msgr2=0x7fa8c8151f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:30:47.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.534+0000 7fa8cd239640 1 --2- 172.21.15.116:0/3991139064 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8c806d5f0 0x7fa8c8151f70 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7fa8b800c370 tx=0x7fa8b800c830 comp rx=0 tx=0).stop 2024-04-25T00:30:47.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.534+0000 7fa8cd239640 1 -- 172.21.15.116:0/3991139064 shutdown_connections 2024-04-25T00:30:47.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.534+0000 7fa8cd239640 1 --2- 172.21.15.116:0/3991139064 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa8a80678c0 0x7fa8a8069d80 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:47.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.534+0000 7fa8cd239640 1 --2- 172.21.15.116:0/3991139064 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa8c8155810 0x7fa8c814be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:47.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.534+0000 7fa8cd239640 1 --2- 172.21.15.116:0/3991139064 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa8c8152d80 0x7fa8c80adcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:47.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.534+0000 7fa8cd239640 1 --2- 172.21.15.116:0/3991139064 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa8c806d5f0 0x7fa8c8151f70 unknown :-1 s=CLOSED pgs=1846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:30:47.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.534+0000 7fa8cd239640 1 -- 172.21.15.116:0/3991139064 >> 172.21.15.116:0/3991139064 conn(0x7fa8c80acdd0 msgr2=0x7fa8c8159380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:30:47.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.534+0000 7fa8cd239640 1 -- 172.21.15.116:0/3991139064 shutdown_connections 2024-04-25T00:30:47.538 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:30:47.534+0000 7fa8cd239640 1 -- 172.21.15.116:0/3991139064 wait complete. 2024-04-25T00:30:47.553 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":965,"stamp":"2024-04-25T00:30:46.029999+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":39263623,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":3556,"num_read_kb":487326,"num_write":2966738,"num_write_kb":38127638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":551299,"ondisk_log_size":551299,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":7166992,"kb_used_data":84432,"kb_used_omap":12,"kb_used_meta":7082483,"kb_avail":747807728,"statfs":{"total":773094113280,"available":765755113472,"internally_reserved":0,"allocated":86458368,"data_stored":82755206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":7252463153},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":445,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006039"},"pg_stats":[{"pgid":"5.18","version":"74'4491","reported_seq":4612,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.766890+0000","last_change":"2024-04-25T00:01:19.508728+0000","last_active":"2024-04-25T00:30:26.766890+0000","last_peered":"2024-04-25T00:30:26.766890+0000","last_clean":"2024-04-25T00:30:26.766890+0000","last_became_active":"2024-04-25T00:01:19.508390+0000","last_became_peered":"2024-04-25T00:01:19.508390+0000","last_unstale":"2024-04-25T00:30:26.766890+0000","last_undegraded":"2024-04-25T00:30:26.766890+0000","last_fullsized":"2024-04-25T00:30:26.766890+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4491,"log_dups_size":0,"ondisk_log_size":4491,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:45.591905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":12,"num_write":23917,"num_write_kb":321110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1f","version":"70'1087","reported_seq":1274,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.794243+0000","last_change":"2024-04-25T00:01:04.356032+0000","last_active":"2024-04-25T00:30:15.794243+0000","last_peered":"2024-04-25T00:30:15.794243+0000","last_clean":"2024-04-25T00:30:15.794243+0000","last_became_active":"2024-04-25T00:01:04.355293+0000","last_became_peered":"2024-04-25T00:01:04.355293+0000","last_unstale":"2024-04-25T00:30:15.794243+0000","last_undegraded":"2024-04-25T00:30:15.794243+0000","last_fullsized":"2024-04-25T00:30:15.794243+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1087,"log_dups_size":0,"ondisk_log_size":1087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":16397,"num_write":1061,"num_write_kb":4715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"70'3760","reported_seq":3888,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.801356+0000","last_change":"2024-04-25T00:01:05.367207+0000","last_active":"2024-04-25T00:30:15.801356+0000","last_peered":"2024-04-25T00:30:15.801356+0000","last_clean":"2024-04-25T00:30:15.801356+0000","last_became_active":"2024-04-25T00:01:05.366471+0000","last_became_peered":"2024-04-25T00:01:05.366471+0000","last_unstale":"2024-04-25T00:30:15.801356+0000","last_undegraded":"2024-04-25T00:30:15.801356+0000","last_fullsized":"2024-04-25T00:30:15.801356+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3760,"log_dups_size":0,"ondisk_log_size":3760,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:30:23.557381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21072,"num_write_kb":264901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"74'3","reported_seq":123,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.743279+0000","last_change":"2024-04-25T00:01:18.502364+0000","last_active":"2024-04-25T00:30:26.743279+0000","last_peered":"2024-04-25T00:30:26.743279+0000","last_clean":"2024-04-25T00:30:26.743279+0000","last_became_active":"2024-04-25T00:01:18.501633+0000","last_became_peered":"2024-04-25T00:01:18.501633+0000","last_unstale":"2024-04-25T00:30:26.743279+0000","last_undegraded":"2024-04-25T00:30:26.743279+0000","last_fullsized":"2024-04-25T00:30:26.743279+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:55.074018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"74'5031","reported_seq":5145,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.765069+0000","last_change":"2024-04-25T00:01:46.662019+0000","last_active":"2024-04-25T00:30:26.765069+0000","last_peered":"2024-04-25T00:30:26.765069+0000","last_clean":"2024-04-25T00:30:26.765069+0000","last_became_active":"2024-04-25T00:01:46.661167+0000","last_became_peered":"2024-04-25T00:01:46.661167+0000","last_unstale":"2024-04-25T00:30:26.765069+0000","last_undegraded":"2024-04-25T00:30:26.765069+0000","last_fullsized":"2024-04-25T00:30:26.765069+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":5031,"log_dups_size":0,"ondisk_log_size":5031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:13:34.153117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":22,"num_write":27250,"num_write_kb":338892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"73'28","reported_seq":158,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.802312+0000","last_change":"2024-04-25T00:01:04.361809+0000","last_active":"2024-04-25T00:30:15.802312+0000","last_peered":"2024-04-25T00:30:15.802312+0000","last_clean":"2024-04-25T00:30:15.802312+0000","last_became_active":"2024-04-25T00:01:04.361059+0000","last_became_peered":"2024-04-25T00:01:04.361059+0000","last_unstale":"2024-04-25T00:30:15.802312+0000","last_undegraded":"2024-04-25T00:30:15.802312+0000","last_fullsized":"2024-04-25T00:30:15.802312+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"70'5376","reported_seq":5504,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.799500+0000","last_change":"2024-04-25T00:01:05.370818+0000","last_active":"2024-04-25T00:30:15.799500+0000","last_peered":"2024-04-25T00:30:15.799500+0000","last_clean":"2024-04-25T00:30:15.799500+0000","last_became_active":"2024-04-25T00:01:05.369991+0000","last_became_peered":"2024-04-25T00:01:05.369991+0000","last_unstale":"2024-04-25T00:30:15.799500+0000","last_undegraded":"2024-04-25T00:30:15.799500+0000","last_fullsized":"2024-04-25T00:30:15.799500+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":5376,"log_dups_size":0,"ondisk_log_size":5376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:43.615895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29984,"num_write_kb":386541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"74'2","reported_seq":122,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:23.313841+0000","last_change":"2024-04-25T00:01:18.497682+0000","last_active":"2024-04-25T00:30:23.313841+0000","last_peered":"2024-04-25T00:30:23.313841+0000","last_clean":"2024-04-25T00:30:23.313841+0000","last_became_active":"2024-04-25T00:01:18.497397+0000","last_became_peered":"2024-04-25T00:01:18.497397+0000","last_unstale":"2024-04-25T00:30:23.313841+0000","last_undegraded":"2024-04-25T00:30:23.313841+0000","last_fullsized":"2024-04-25T00:30:23.313841+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:22.983972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"74'4080","reported_seq":4199,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.684096+0000","last_change":"2024-04-25T00:01:19.510591+0000","last_active":"2024-04-25T00:30:26.684096+0000","last_peered":"2024-04-25T00:30:26.684096+0000","last_clean":"2024-04-25T00:30:26.684096+0000","last_became_active":"2024-04-25T00:01:19.510252+0000","last_became_peered":"2024-04-25T00:01:19.510252+0000","last_unstale":"2024-04-25T00:30:26.684096+0000","last_undegraded":"2024-04-25T00:30:26.684096+0000","last_fullsized":"2024-04-25T00:30:26.684096+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4080,"log_dups_size":0,"ondisk_log_size":4080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:38.005233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":21789,"num_write_kb":296332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"74'53","reported_seq":203,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.793824+0000","last_change":"2024-04-25T00:01:04.362030+0000","last_active":"2024-04-25T00:30:15.793824+0000","last_peered":"2024-04-25T00:30:15.793824+0000","last_clean":"2024-04-25T00:30:15.793824+0000","last_became_active":"2024-04-25T00:01:04.361708+0000","last_became_peered":"2024-04-25T00:01:04.361708+0000","last_unstale":"2024-04-25T00:30:15.793824+0000","last_undegraded":"2024-04-25T00:30:15.793824+0000","last_fullsized":"2024-04-25T00:30:15.793824+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:38:33.392719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":69,"num_read_kb":69,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"69'3669","reported_seq":3794,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.942949+0000","last_change":"2024-04-25T00:01:05.387195+0000","last_active":"2024-04-25T00:30:15.942949+0000","last_peered":"2024-04-25T00:30:15.942949+0000","last_clean":"2024-04-25T00:30:15.942949+0000","last_became_active":"2024-04-25T00:01:05.386853+0000","last_became_peered":"2024-04-25T00:01:05.386853+0000","last_unstale":"2024-04-25T00:30:15.942949+0000","last_undegraded":"2024-04-25T00:30:15.942949+0000","last_fullsized":"2024-04-25T00:30:15.942949+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3669,"log_dups_size":0,"ondisk_log_size":3669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:06:19.307378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19904,"num_write_kb":264274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"74'723","reported_seq":869,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.740062+0000","last_change":"2024-04-25T00:01:18.487835+0000","last_active":"2024-04-25T00:30:26.740062+0000","last_peered":"2024-04-25T00:30:26.740062+0000","last_clean":"2024-04-25T00:30:26.740062+0000","last_became_active":"2024-04-25T00:01:18.487571+0000","last_became_peered":"2024-04-25T00:01:18.487571+0000","last_unstale":"2024-04-25T00:30:26.740062+0000","last_undegraded":"2024-04-25T00:30:26.740062+0000","last_fullsized":"2024-04-25T00:30:26.740062+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":723,"log_dups_size":0,"ondisk_log_size":723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:44.582148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":27,"num_read_kb":82727,"num_write":711,"num_write_kb":4310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"74'3586","reported_seq":3704,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.700653+0000","last_change":"2024-04-25T00:01:19.519834+0000","last_active":"2024-04-25T00:30:26.700653+0000","last_peered":"2024-04-25T00:30:26.700653+0000","last_clean":"2024-04-25T00:30:26.700653+0000","last_became_active":"2024-04-25T00:01:19.519470+0000","last_became_peered":"2024-04-25T00:01:19.519470+0000","last_unstale":"2024-04-25T00:30:26.700653+0000","last_undegraded":"2024-04-25T00:30:26.700653+0000","last_fullsized":"2024-04-25T00:30:26.700653+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3586,"log_dups_size":0,"ondisk_log_size":3586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:47.859626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":5,"num_write":18586,"num_write_kb":254663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"70'6","reported_seq":179,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.793266+0000","last_change":"2024-04-25T00:01:04.362832+0000","last_active":"2024-04-25T00:30:15.793266+0000","last_peered":"2024-04-25T00:30:15.793266+0000","last_clean":"2024-04-25T00:30:15.793266+0000","last_became_active":"2024-04-25T00:01:04.362059+0000","last_became_peered":"2024-04-25T00:01:04.362059+0000","last_unstale":"2024-04-25T00:30:15.793266+0000","last_undegraded":"2024-04-25T00:30:15.793266+0000","last_fullsized":"2024-04-25T00:30:15.793266+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:05.441048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"70'4973","reported_seq":5098,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.813402+0000","last_change":"2024-04-25T00:01:05.381120+0000","last_active":"2024-04-25T00:30:15.813402+0000","last_peered":"2024-04-25T00:30:15.813402+0000","last_clean":"2024-04-25T00:30:15.813402+0000","last_became_active":"2024-04-25T00:01:05.378320+0000","last_became_peered":"2024-04-25T00:01:05.378320+0000","last_unstale":"2024-04-25T00:30:15.813402+0000","last_undegraded":"2024-04-25T00:30:15.813402+0000","last_fullsized":"2024-04-25T00:30:15.813402+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4973,"log_dups_size":0,"ondisk_log_size":4973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:34:06.175918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27225,"num_write_kb":350946,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"74'356","reported_seq":477,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:31.515973+0000","last_change":"2024-04-25T00:01:18.490300+0000","last_active":"2024-04-25T00:30:31.515973+0000","last_peered":"2024-04-25T00:30:31.515973+0000","last_clean":"2024-04-25T00:30:31.515973+0000","last_became_active":"2024-04-25T00:01:18.490050+0000","last_became_peered":"2024-04-25T00:01:18.490050+0000","last_unstale":"2024-04-25T00:30:31.515973+0000","last_undegraded":"2024-04-25T00:30:31.515973+0000","last_fullsized":"2024-04-25T00:30:31.515973+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":356,"log_dups_size":0,"ondisk_log_size":356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:13:05.971606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2691973,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":907,"num_write":327,"num_write_kb":2826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"74'4320","reported_seq":4442,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.772385+0000","last_change":"2024-04-25T00:01:19.500026+0000","last_active":"2024-04-25T00:30:26.772385+0000","last_peered":"2024-04-25T00:30:26.772385+0000","last_clean":"2024-04-25T00:30:26.772385+0000","last_became_active":"2024-04-25T00:01:19.499244+0000","last_became_peered":"2024-04-25T00:01:19.499244+0000","last_unstale":"2024-04-25T00:30:26.772385+0000","last_undegraded":"2024-04-25T00:30:26.772385+0000","last_fullsized":"2024-04-25T00:30:26.772385+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4320,"log_dups_size":0,"ondisk_log_size":4320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:40:36.350832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":15,"num_write":23810,"num_write_kb":308534,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"70'4158","reported_seq":4285,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.809005+0000","last_change":"2024-04-25T00:01:05.376984+0000","last_active":"2024-04-25T00:30:15.809005+0000","last_peered":"2024-04-25T00:30:15.809005+0000","last_clean":"2024-04-25T00:30:15.809005+0000","last_became_active":"2024-04-25T00:01:05.376259+0000","last_became_peered":"2024-04-25T00:01:05.376259+0000","last_unstale":"2024-04-25T00:30:15.809005+0000","last_undegraded":"2024-04-25T00:30:15.809005+0000","last_fullsized":"2024-04-25T00:30:15.809005+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4158,"log_dups_size":0,"ondisk_log_size":4158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:13:28.543302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23006,"num_write_kb":301346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"74'1526","reported_seq":1710,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:23.032874+0000","last_change":"2024-04-25T00:01:04.359262+0000","last_active":"2024-04-25T00:30:23.032874+0000","last_peered":"2024-04-25T00:30:23.032874+0000","last_clean":"2024-04-25T00:30:23.032874+0000","last_became_active":"2024-04-25T00:01:04.357971+0000","last_became_peered":"2024-04-25T00:01:04.357971+0000","last_unstale":"2024-04-25T00:30:23.032874+0000","last_undegraded":"2024-04-25T00:30:23.032874+0000","last_fullsized":"2024-04-25T00:30:23.032874+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1526,"log_dups_size":0,"ondisk_log_size":1526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:05.441048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":78,"num_read_kb":52062,"num_write":1499,"num_write_kb":4525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"74'6","reported_seq":150,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.768405+0000","last_change":"2024-04-25T00:01:18.475624+0000","last_active":"2024-04-25T00:30:26.768405+0000","last_peered":"2024-04-25T00:30:26.768405+0000","last_clean":"2024-04-25T00:30:26.768405+0000","last_became_active":"2024-04-25T00:01:18.475149+0000","last_became_peered":"2024-04-25T00:01:18.475149+0000","last_unstale":"2024-04-25T00:30:26.768405+0000","last_undegraded":"2024-04-25T00:30:26.768405+0000","last_fullsized":"2024-04-25T00:30:26.768405+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:40:35.341075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"74'4556","reported_seq":4678,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.774727+0000","last_change":"2024-04-25T00:01:19.531113+0000","last_active":"2024-04-25T00:30:26.774727+0000","last_peered":"2024-04-25T00:30:26.774727+0000","last_clean":"2024-04-25T00:30:26.774727+0000","last_became_active":"2024-04-25T00:01:19.520150+0000","last_became_peered":"2024-04-25T00:01:19.520150+0000","last_unstale":"2024-04-25T00:30:26.774727+0000","last_undegraded":"2024-04-25T00:30:26.774727+0000","last_fullsized":"2024-04-25T00:30:26.774727+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4556,"log_dups_size":0,"ondisk_log_size":4556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:38.152376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":19,"num_write":25634,"num_write_kb":315935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"70'3060","reported_seq":3187,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.803143+0000","last_change":"2024-04-25T00:01:05.362611+0000","last_active":"2024-04-25T00:30:15.803143+0000","last_peered":"2024-04-25T00:30:15.803143+0000","last_clean":"2024-04-25T00:30:15.803143+0000","last_became_active":"2024-04-25T00:01:05.361817+0000","last_became_peered":"2024-04-25T00:01:05.361817+0000","last_unstale":"2024-04-25T00:30:15.803143+0000","last_undegraded":"2024-04-25T00:30:15.803143+0000","last_fullsized":"2024-04-25T00:30:15.803143+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3060,"log_dups_size":0,"ondisk_log_size":3060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:50.489125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16241,"num_write_kb":228970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"70'3","reported_seq":132,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.810550+0000","last_change":"2024-04-25T00:01:04.371935+0000","last_active":"2024-04-25T00:30:15.810550+0000","last_peered":"2024-04-25T00:30:15.810550+0000","last_clean":"2024-04-25T00:30:15.810550+0000","last_became_active":"2024-04-25T00:01:04.371420+0000","last_became_peered":"2024-04-25T00:01:04.371420+0000","last_unstale":"2024-04-25T00:30:15.810550+0000","last_undegraded":"2024-04-25T00:30:15.810550+0000","last_fullsized":"2024-04-25T00:30:15.810550+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"74'5","reported_seq":126,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.747597+0000","last_change":"2024-04-25T00:01:18.488623+0000","last_active":"2024-04-25T00:30:26.747597+0000","last_peered":"2024-04-25T00:30:26.747597+0000","last_clean":"2024-04-25T00:30:26.747597+0000","last_became_active":"2024-04-25T00:01:18.488291+0000","last_became_peered":"2024-04-25T00:01:18.488291+0000","last_unstale":"2024-04-25T00:30:26.747597+0000","last_undegraded":"2024-04-25T00:30:26.747597+0000","last_fullsized":"2024-04-25T00:30:26.747597+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:30:21.436681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"74'3591","reported_seq":3711,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.734674+0000","last_change":"2024-04-25T00:01:19.516731+0000","last_active":"2024-04-25T00:30:26.734674+0000","last_peered":"2024-04-25T00:30:26.734674+0000","last_clean":"2024-04-25T00:30:26.734674+0000","last_became_active":"2024-04-25T00:01:19.516405+0000","last_became_peered":"2024-04-25T00:01:19.516405+0000","last_unstale":"2024-04-25T00:30:26.734674+0000","last_undegraded":"2024-04-25T00:30:26.734674+0000","last_fullsized":"2024-04-25T00:30:26.734674+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3591,"log_dups_size":0,"ondisk_log_size":3591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:44:37.936092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":19584,"num_write_kb":250628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"69'3777","reported_seq":3905,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.790939+0000","last_change":"2024-04-25T00:01:05.355455+0000","last_active":"2024-04-25T00:30:15.790939+0000","last_peered":"2024-04-25T00:30:15.790939+0000","last_clean":"2024-04-25T00:30:15.790939+0000","last_became_active":"2024-04-25T00:01:05.354699+0000","last_became_peered":"2024-04-25T00:01:05.354699+0000","last_unstale":"2024-04-25T00:30:15.790939+0000","last_undegraded":"2024-04-25T00:30:15.790939+0000","last_fullsized":"2024-04-25T00:30:15.790939+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3777,"log_dups_size":0,"ondisk_log_size":3777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20993,"num_write_kb":256390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"69'2","reported_seq":132,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.791780+0000","last_change":"2024-04-25T00:01:04.362268+0000","last_active":"2024-04-25T00:30:15.791780+0000","last_peered":"2024-04-25T00:30:15.791780+0000","last_clean":"2024-04-25T00:30:15.791780+0000","last_became_active":"2024-04-25T00:01:04.361622+0000","last_became_peered":"2024-04-25T00:01:04.361622+0000","last_unstale":"2024-04-25T00:30:15.791780+0000","last_undegraded":"2024-04-25T00:30:15.791780+0000","last_fullsized":"2024-04-25T00:30:15.791780+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"74'4","reported_seq":148,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.744694+0000","last_change":"2024-04-25T00:01:18.494047+0000","last_active":"2024-04-25T00:30:26.744694+0000","last_peered":"2024-04-25T00:30:26.744694+0000","last_clean":"2024-04-25T00:30:26.744694+0000","last_became_active":"2024-04-25T00:01:18.493614+0000","last_became_peered":"2024-04-25T00:01:18.493614+0000","last_unstale":"2024-04-25T00:30:26.744694+0000","last_undegraded":"2024-04-25T00:30:26.744694+0000","last_fullsized":"2024-04-25T00:30:26.744694+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:37:08.562016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"74'4895","reported_seq":5014,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.739393+0000","last_change":"2024-04-25T00:01:19.505397+0000","last_active":"2024-04-25T00:30:26.739393+0000","last_peered":"2024-04-25T00:30:26.739393+0000","last_clean":"2024-04-25T00:30:26.739393+0000","last_became_active":"2024-04-25T00:01:19.505126+0000","last_became_peered":"2024-04-25T00:01:19.505126+0000","last_unstale":"2024-04-25T00:30:26.739393+0000","last_undegraded":"2024-04-25T00:30:26.739393+0000","last_fullsized":"2024-04-25T00:30:26.739393+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4895,"log_dups_size":0,"ondisk_log_size":4895,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:08:49.640267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26799,"num_write_kb":349350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"69'3934","reported_seq":4061,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.805653+0000","last_change":"2024-04-25T00:01:05.354953+0000","last_active":"2024-04-25T00:30:15.805653+0000","last_peered":"2024-04-25T00:30:15.805653+0000","last_clean":"2024-04-25T00:30:15.805653+0000","last_became_active":"2024-04-25T00:01:05.354118+0000","last_became_peered":"2024-04-25T00:01:05.354118+0000","last_unstale":"2024-04-25T00:30:15.805653+0000","last_undegraded":"2024-04-25T00:30:15.805653+0000","last_fullsized":"2024-04-25T00:30:15.805653+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3934,"log_dups_size":0,"ondisk_log_size":3934,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:09.858617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21059,"num_write_kb":273031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"70'6","reported_seq":135,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.789548+0000","last_change":"2024-04-25T00:01:04.365515+0000","last_active":"2024-04-25T00:30:15.789548+0000","last_peered":"2024-04-25T00:30:15.789548+0000","last_clean":"2024-04-25T00:30:15.789548+0000","last_became_active":"2024-04-25T00:01:04.364942+0000","last_became_peered":"2024-04-25T00:01:04.364942+0000","last_unstale":"2024-04-25T00:30:15.789548+0000","last_undegraded":"2024-04-25T00:30:15.789548+0000","last_fullsized":"2024-04-25T00:30:15.789548+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"74'7","reported_seq":174,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.741808+0000","last_change":"2024-04-25T00:01:18.476207+0000","last_active":"2024-04-25T00:30:26.741808+0000","last_peered":"2024-04-25T00:30:26.741808+0000","last_clean":"2024-04-25T00:30:26.741808+0000","last_became_active":"2024-04-25T00:01:18.475862+0000","last_became_peered":"2024-04-25T00:01:18.475862+0000","last_unstale":"2024-04-25T00:30:26.741808+0000","last_undegraded":"2024-04-25T00:30:26.741808+0000","last_fullsized":"2024-04-25T00:30:26.741808+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:11:40.170249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"74'3090","reported_seq":3211,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.778252+0000","last_change":"2024-04-25T00:01:19.529607+0000","last_active":"2024-04-25T00:30:26.778252+0000","last_peered":"2024-04-25T00:30:26.778252+0000","last_clean":"2024-04-25T00:30:26.778252+0000","last_became_active":"2024-04-25T00:01:19.528509+0000","last_became_peered":"2024-04-25T00:01:19.528509+0000","last_unstale":"2024-04-25T00:30:26.778252+0000","last_undegraded":"2024-04-25T00:30:26.778252+0000","last_fullsized":"2024-04-25T00:30:26.778252+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3090,"log_dups_size":0,"ondisk_log_size":3090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17043,"num_write_kb":228060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"70'4336","reported_seq":4464,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.793819+0000","last_change":"2024-04-25T00:01:05.358106+0000","last_active":"2024-04-25T00:30:15.793819+0000","last_peered":"2024-04-25T00:30:15.793819+0000","last_clean":"2024-04-25T00:30:15.793819+0000","last_became_active":"2024-04-25T00:01:05.357135+0000","last_became_peered":"2024-04-25T00:01:05.357135+0000","last_unstale":"2024-04-25T00:30:15.793819+0000","last_undegraded":"2024-04-25T00:30:15.793819+0000","last_fullsized":"2024-04-25T00:30:15.793819+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4336,"log_dups_size":0,"ondisk_log_size":4336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23030,"num_write_kb":325887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"71'9","reported_seq":184,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.814467+0000","last_change":"2024-04-25T00:01:04.369917+0000","last_active":"2024-04-25T00:30:15.814467+0000","last_peered":"2024-04-25T00:30:15.814467+0000","last_clean":"2024-04-25T00:30:15.814467+0000","last_became_active":"2024-04-25T00:01:04.369309+0000","last_became_peered":"2024-04-25T00:01:04.369309+0000","last_unstale":"2024-04-25T00:30:15.814467+0000","last_undegraded":"2024-04-25T00:30:15.814467+0000","last_fullsized":"2024-04-25T00:30:15.814467+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:55:08.063965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":8,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"74'51","reported_seq":241,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:28.848602+0000","last_change":"2024-04-25T00:01:18.477189+0000","last_active":"2024-04-25T00:30:28.848602+0000","last_peered":"2024-04-25T00:30:28.848602+0000","last_clean":"2024-04-25T00:30:28.848602+0000","last_became_active":"2024-04-25T00:01:18.476909+0000","last_became_peered":"2024-04-25T00:01:18.476909+0000","last_unstale":"2024-04-25T00:30:28.848602+0000","last_undegraded":"2024-04-25T00:30:28.848602+0000","last_fullsized":"2024-04-25T00:30:28.848602+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:36.995477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84646,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":68,"num_read_kb":112,"num_write":46,"num_write_kb":112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"74'3821","reported_seq":3942,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.704068+0000","last_change":"2024-04-25T00:01:19.517500+0000","last_active":"2024-04-25T00:30:26.704068+0000","last_peered":"2024-04-25T00:30:26.704068+0000","last_clean":"2024-04-25T00:30:26.704068+0000","last_became_active":"2024-04-25T00:01:19.517182+0000","last_became_peered":"2024-04-25T00:01:19.517182+0000","last_unstale":"2024-04-25T00:30:26.704068+0000","last_undegraded":"2024-04-25T00:30:26.704068+0000","last_fullsized":"2024-04-25T00:30:26.704068+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3821,"log_dups_size":0,"ondisk_log_size":3821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:58:49.738233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20929,"num_write_kb":256020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"70'3227","reported_seq":3355,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.793885+0000","last_change":"2024-04-25T00:01:05.364829+0000","last_active":"2024-04-25T00:30:15.793885+0000","last_peered":"2024-04-25T00:30:15.793885+0000","last_clean":"2024-04-25T00:30:15.793885+0000","last_became_active":"2024-04-25T00:01:05.364125+0000","last_became_peered":"2024-04-25T00:01:05.364125+0000","last_unstale":"2024-04-25T00:30:15.793885+0000","last_undegraded":"2024-04-25T00:30:15.793885+0000","last_fullsized":"2024-04-25T00:30:15.793885+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3227,"log_dups_size":0,"ondisk_log_size":3227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17808,"num_write_kb":231231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"74'458","reported_seq":611,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:25.818063+0000","last_change":"2024-04-25T00:01:04.361603+0000","last_active":"2024-04-25T00:30:25.818063+0000","last_peered":"2024-04-25T00:30:25.818063+0000","last_clean":"2024-04-25T00:30:25.818063+0000","last_became_active":"2024-04-25T00:01:04.361046+0000","last_became_peered":"2024-04-25T00:01:04.361046+0000","last_unstale":"2024-04-25T00:30:25.818063+0000","last_undegraded":"2024-04-25T00:30:25.818063+0000","last_fullsized":"2024-04-25T00:30:25.818063+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":458,"log_dups_size":0,"ondisk_log_size":458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":46,"num_read_kb":63,"num_write":1069,"num_write_kb":360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"74'12","reported_seq":135,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.748042+0000","last_change":"2024-04-25T00:01:18.488352+0000","last_active":"2024-04-25T00:30:26.748042+0000","last_peered":"2024-04-25T00:30:26.748042+0000","last_clean":"2024-04-25T00:30:26.748042+0000","last_became_active":"2024-04-25T00:01:18.487321+0000","last_became_peered":"2024-04-25T00:01:18.487321+0000","last_unstale":"2024-04-25T00:30:26.748042+0000","last_undegraded":"2024-04-25T00:30:26.748042+0000","last_fullsized":"2024-04-25T00:30:26.748042+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"74'4221","reported_seq":4341,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.775265+0000","last_change":"2024-04-25T00:01:19.519653+0000","last_active":"2024-04-25T00:30:26.775265+0000","last_peered":"2024-04-25T00:30:26.775265+0000","last_clean":"2024-04-25T00:30:26.775265+0000","last_became_active":"2024-04-25T00:01:19.519052+0000","last_became_peered":"2024-04-25T00:01:19.519052+0000","last_unstale":"2024-04-25T00:30:26.775265+0000","last_undegraded":"2024-04-25T00:30:26.775265+0000","last_fullsized":"2024-04-25T00:30:26.775265+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4221,"log_dups_size":0,"ondisk_log_size":4221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:40:36.350832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22937,"num_write_kb":280510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"70'5174","reported_seq":5302,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.813035+0000","last_change":"2024-04-25T00:01:05.374011+0000","last_active":"2024-04-25T00:30:15.813035+0000","last_peered":"2024-04-25T00:30:15.813035+0000","last_clean":"2024-04-25T00:30:15.813035+0000","last_became_active":"2024-04-25T00:01:05.373276+0000","last_became_peered":"2024-04-25T00:01:05.373276+0000","last_unstale":"2024-04-25T00:30:15.813035+0000","last_undegraded":"2024-04-25T00:30:15.813035+0000","last_fullsized":"2024-04-25T00:30:15.813035+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":5174,"log_dups_size":0,"ondisk_log_size":5174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:40:53.349428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28022,"num_write_kb":362384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"69'2","reported_seq":132,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.814680+0000","last_change":"2024-04-25T00:01:04.365391+0000","last_active":"2024-04-25T00:30:15.814680+0000","last_peered":"2024-04-25T00:30:15.814680+0000","last_clean":"2024-04-25T00:30:15.814680+0000","last_became_active":"2024-04-25T00:01:04.364989+0000","last_became_peered":"2024-04-25T00:01:04.364989+0000","last_unstale":"2024-04-25T00:30:15.814680+0000","last_undegraded":"2024-04-25T00:30:15.814680+0000","last_fullsized":"2024-04-25T00:30:15.814680+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:06:18.298057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"74'3","reported_seq":125,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.737749+0000","last_change":"2024-04-25T00:01:18.491597+0000","last_active":"2024-04-25T00:30:26.737749+0000","last_peered":"2024-04-25T00:30:26.737749+0000","last_clean":"2024-04-25T00:30:26.737749+0000","last_became_active":"2024-04-25T00:01:18.490859+0000","last_became_peered":"2024-04-25T00:01:18.490859+0000","last_unstale":"2024-04-25T00:30:26.737749+0000","last_undegraded":"2024-04-25T00:30:26.737749+0000","last_fullsized":"2024-04-25T00:30:26.737749+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:36.995477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"74'5316","reported_seq":5429,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.548329+0000","last_change":"2024-04-25T00:01:46.667934+0000","last_active":"2024-04-25T00:30:26.548329+0000","last_peered":"2024-04-25T00:30:26.548329+0000","last_clean":"2024-04-25T00:30:26.548329+0000","last_became_active":"2024-04-25T00:01:46.667075+0000","last_became_peered":"2024-04-25T00:01:46.667075+0000","last_unstale":"2024-04-25T00:30:26.548329+0000","last_undegraded":"2024-04-25T00:30:26.548329+0000","last_fullsized":"2024-04-25T00:30:26.548329+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":5316,"log_dups_size":0,"ondisk_log_size":5316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:28:18.234546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29390,"num_write_kb":363802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"70'3961","reported_seq":4089,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.804843+0000","last_change":"2024-04-25T00:01:05.378340+0000","last_active":"2024-04-25T00:30:15.804843+0000","last_peered":"2024-04-25T00:30:15.804843+0000","last_clean":"2024-04-25T00:30:15.804843+0000","last_became_active":"2024-04-25T00:01:05.377593+0000","last_became_peered":"2024-04-25T00:01:05.377593+0000","last_unstale":"2024-04-25T00:30:15.804843+0000","last_undegraded":"2024-04-25T00:30:15.804843+0000","last_fullsized":"2024-04-25T00:30:15.804843+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3961,"log_dups_size":0,"ondisk_log_size":3961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:48.202251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21237,"num_write_kb":269574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"70'2","reported_seq":132,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.799354+0000","last_change":"2024-04-25T00:01:04.366577+0000","last_active":"2024-04-25T00:30:15.799354+0000","last_peered":"2024-04-25T00:30:15.799354+0000","last_clean":"2024-04-25T00:30:15.799354+0000","last_became_active":"2024-04-25T00:01:04.365881+0000","last_became_peered":"2024-04-25T00:01:04.365881+0000","last_unstale":"2024-04-25T00:30:15.799354+0000","last_undegraded":"2024-04-25T00:30:15.799354+0000","last_fullsized":"2024-04-25T00:30:15.799354+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:49:22.673421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'4","reported_seq":126,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.132180+0000","last_change":"2024-04-25T00:01:18.504341+0000","last_active":"2024-04-25T00:30:26.132180+0000","last_peered":"2024-04-25T00:30:26.132180+0000","last_clean":"2024-04-25T00:30:26.132180+0000","last_became_active":"2024-04-25T00:01:18.491498+0000","last_became_peered":"2024-04-25T00:01:18.491498+0000","last_unstale":"2024-04-25T00:30:26.132180+0000","last_undegraded":"2024-04-25T00:30:26.132180+0000","last_fullsized":"2024-04-25T00:30:26.132180+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:56.741250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"74'4156","reported_seq":4283,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.774804+0000","last_change":"2024-04-25T00:01:46.667022+0000","last_active":"2024-04-25T00:30:26.774804+0000","last_peered":"2024-04-25T00:30:26.774804+0000","last_clean":"2024-04-25T00:30:26.774804+0000","last_became_active":"2024-04-25T00:01:46.666224+0000","last_became_peered":"2024-04-25T00:01:46.666224+0000","last_unstale":"2024-04-25T00:30:26.774804+0000","last_undegraded":"2024-04-25T00:30:26.774804+0000","last_fullsized":"2024-04-25T00:30:26.774804+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4156,"log_dups_size":0,"ondisk_log_size":4156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:25:20.185355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":7,"num_write":23008,"num_write_kb":280758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"69'4491","reported_seq":4619,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.793150+0000","last_change":"2024-04-25T00:01:05.368769+0000","last_active":"2024-04-25T00:30:15.793150+0000","last_peered":"2024-04-25T00:30:15.793150+0000","last_clean":"2024-04-25T00:30:15.793150+0000","last_became_active":"2024-04-25T00:01:05.366817+0000","last_became_peered":"2024-04-25T00:01:05.366817+0000","last_unstale":"2024-04-25T00:30:15.793150+0000","last_undegraded":"2024-04-25T00:30:15.793150+0000","last_fullsized":"2024-04-25T00:30:15.793150+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4491,"log_dups_size":0,"ondisk_log_size":4491,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:11:27.044893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24298,"num_write_kb":330898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"69'5","reported_seq":135,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.810345+0000","last_change":"2024-04-25T00:01:04.361900+0000","last_active":"2024-04-25T00:30:15.810345+0000","last_peered":"2024-04-25T00:30:15.810345+0000","last_clean":"2024-04-25T00:30:15.810345+0000","last_became_active":"2024-04-25T00:01:04.361572+0000","last_became_peered":"2024-04-25T00:01:04.361572+0000","last_unstale":"2024-04-25T00:30:15.810345+0000","last_undegraded":"2024-04-25T00:30:15.810345+0000","last_fullsized":"2024-04-25T00:30:15.810345+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:29.218739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"74'2","reported_seq":124,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.744419+0000","last_change":"2024-04-25T00:01:18.504139+0000","last_active":"2024-04-25T00:30:26.744419+0000","last_peered":"2024-04-25T00:30:26.744419+0000","last_clean":"2024-04-25T00:30:26.744419+0000","last_became_active":"2024-04-25T00:01:18.503379+0000","last_became_peered":"2024-04-25T00:01:18.503379+0000","last_unstale":"2024-04-25T00:30:26.744419+0000","last_undegraded":"2024-04-25T00:30:26.744419+0000","last_fullsized":"2024-04-25T00:30:26.744419+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:02:56.363258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"74'5402","reported_seq":5525,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.707463+0000","last_change":"2024-04-25T00:01:19.531756+0000","last_active":"2024-04-25T00:30:26.707463+0000","last_peered":"2024-04-25T00:30:26.707463+0000","last_clean":"2024-04-25T00:30:26.707463+0000","last_became_active":"2024-04-25T00:01:19.531145+0000","last_became_peered":"2024-04-25T00:01:19.531145+0000","last_unstale":"2024-04-25T00:30:26.707463+0000","last_undegraded":"2024-04-25T00:30:26.707463+0000","last_fullsized":"2024-04-25T00:30:26.707463+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":5402,"log_dups_size":0,"ondisk_log_size":5402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:13:42.678415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":13,"num_write":29532,"num_write_kb":356063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"70'3335","reported_seq":3463,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.794572+0000","last_change":"2024-04-25T00:01:05.354557+0000","last_active":"2024-04-25T00:30:15.794572+0000","last_peered":"2024-04-25T00:30:15.794572+0000","last_clean":"2024-04-25T00:30:15.794572+0000","last_became_active":"2024-04-25T00:01:05.353556+0000","last_became_peered":"2024-04-25T00:01:05.353556+0000","last_unstale":"2024-04-25T00:30:15.794572+0000","last_undegraded":"2024-04-25T00:30:15.794572+0000","last_fullsized":"2024-04-25T00:30:15.794572+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3335,"log_dups_size":0,"ondisk_log_size":3335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:24.964055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18377,"num_write_kb":241696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"74'53","reported_seq":252,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:25.820987+0000","last_change":"2024-04-25T00:01:04.372916+0000","last_active":"2024-04-25T00:30:25.820987+0000","last_peered":"2024-04-25T00:30:25.820987+0000","last_clean":"2024-04-25T00:30:25.820987+0000","last_became_active":"2024-04-25T00:01:04.372185+0000","last_became_peered":"2024-04-25T00:01:04.372185+0000","last_unstale":"2024-04-25T00:30:25.820987+0000","last_undegraded":"2024-04-25T00:30:25.820987+0000","last_fullsized":"2024-04-25T00:30:25.820987+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88686,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":69,"num_read_kb":117,"num_write":46,"num_write_kb":117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"74'9","reported_seq":176,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.750624+0000","last_change":"2024-04-25T00:01:18.504614+0000","last_active":"2024-04-25T00:30:26.750624+0000","last_peered":"2024-04-25T00:30:26.750624+0000","last_clean":"2024-04-25T00:30:26.750624+0000","last_became_active":"2024-04-25T00:01:18.501448+0000","last_became_peered":"2024-04-25T00:01:18.501448+0000","last_unstale":"2024-04-25T00:30:26.750624+0000","last_undegraded":"2024-04-25T00:30:26.750624+0000","last_fullsized":"2024-04-25T00:30:26.750624+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:22.983972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":8,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"74'3337","reported_seq":3456,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.763497+0000","last_change":"2024-04-25T00:01:19.517510+0000","last_active":"2024-04-25T00:30:26.763497+0000","last_peered":"2024-04-25T00:30:26.763497+0000","last_clean":"2024-04-25T00:30:26.763497+0000","last_became_active":"2024-04-25T00:01:19.517243+0000","last_became_peered":"2024-04-25T00:01:19.517243+0000","last_unstale":"2024-04-25T00:30:26.763497+0000","last_undegraded":"2024-04-25T00:30:26.763497+0000","last_fullsized":"2024-04-25T00:30:26.763497+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3337,"log_dups_size":0,"ondisk_log_size":3337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:42:38.149502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18533,"num_write_kb":242287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"69'3723","reported_seq":3852,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.809022+0000","last_change":"2024-04-25T00:01:05.370968+0000","last_active":"2024-04-25T00:30:15.809022+0000","last_peered":"2024-04-25T00:30:15.809022+0000","last_clean":"2024-04-25T00:30:15.809022+0000","last_became_active":"2024-04-25T00:01:05.369987+0000","last_became_peered":"2024-04-25T00:01:05.369987+0000","last_unstale":"2024-04-25T00:30:15.809022+0000","last_undegraded":"2024-04-25T00:30:15.809022+0000","last_fullsized":"2024-04-25T00:30:15.809022+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3723,"log_dups_size":0,"ondisk_log_size":3723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20208,"num_write_kb":269376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"73'13","reported_seq":143,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.804977+0000","last_change":"2024-04-25T00:01:04.370226+0000","last_active":"2024-04-25T00:30:15.804977+0000","last_peered":"2024-04-25T00:30:15.804977+0000","last_clean":"2024-04-25T00:30:15.804977+0000","last_became_active":"2024-04-25T00:01:04.369904+0000","last_became_peered":"2024-04-25T00:01:04.369904+0000","last_unstale":"2024-04-25T00:30:15.804977+0000","last_undegraded":"2024-04-25T00:30:15.804977+0000","last_fullsized":"2024-04-25T00:30:15.804977+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:47:25.595275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"74'487","reported_seq":631,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.794202+0000","last_change":"2024-04-25T00:01:18.497156+0000","last_active":"2024-04-25T00:30:26.794202+0000","last_peered":"2024-04-25T00:30:26.794202+0000","last_clean":"2024-04-25T00:30:26.794202+0000","last_became_active":"2024-04-25T00:01:18.495573+0000","last_became_peered":"2024-04-25T00:01:18.495573+0000","last_unstale":"2024-04-25T00:30:26.794202+0000","last_undegraded":"2024-04-25T00:30:26.794202+0000","last_fullsized":"2024-04-25T00:30:26.794202+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":487,"log_dups_size":0,"ondisk_log_size":487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:29.119299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":46,"num_read_kb":69,"num_write":1147,"num_write_kb":382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"74'5079","reported_seq":5200,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.738941+0000","last_change":"2024-04-25T00:01:19.519411+0000","last_active":"2024-04-25T00:30:26.738941+0000","last_peered":"2024-04-25T00:30:26.738941+0000","last_clean":"2024-04-25T00:30:26.738941+0000","last_became_active":"2024-04-25T00:01:19.519020+0000","last_became_peered":"2024-04-25T00:01:19.519020+0000","last_unstale":"2024-04-25T00:30:26.738941+0000","last_undegraded":"2024-04-25T00:30:26.738941+0000","last_fullsized":"2024-04-25T00:30:26.738941+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":5079,"log_dups_size":0,"ondisk_log_size":5079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:50.351599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":14,"num_write":27774,"num_write_kb":353080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"70'3676","reported_seq":3805,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.793969+0000","last_change":"2024-04-25T00:01:05.364939+0000","last_active":"2024-04-25T00:30:15.793969+0000","last_peered":"2024-04-25T00:30:15.793969+0000","last_clean":"2024-04-25T00:30:15.793969+0000","last_became_active":"2024-04-25T00:01:05.364272+0000","last_became_peered":"2024-04-25T00:01:05.364272+0000","last_unstale":"2024-04-25T00:30:15.793969+0000","last_undegraded":"2024-04-25T00:30:15.793969+0000","last_fullsized":"2024-04-25T00:30:15.793969+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3676,"log_dups_size":0,"ondisk_log_size":3676,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19925,"num_write_kb":256146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"69'3","reported_seq":133,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.809642+0000","last_change":"2024-04-25T00:01:04.357028+0000","last_active":"2024-04-25T00:30:15.809642+0000","last_peered":"2024-04-25T00:30:15.809642+0000","last_clean":"2024-04-25T00:30:15.809642+0000","last_became_active":"2024-04-25T00:01:04.356644+0000","last_became_peered":"2024-04-25T00:01:04.356644+0000","last_unstale":"2024-04-25T00:30:15.809642+0000","last_undegraded":"2024-04-25T00:30:15.809642+0000","last_fullsized":"2024-04-25T00:30:15.809642+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:21.729893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":286,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.942363+0000","last_change":"2024-04-25T00:00:55.084492+0000","last_active":"2024-04-25T00:30:15.942363+0000","last_peered":"2024-04-25T00:30:15.942363+0000","last_clean":"2024-04-25T00:30:15.942363+0000","last_became_active":"2024-04-25T00:00:55.083493+0000","last_became_peered":"2024-04-25T00:00:55.083493+0000","last_unstale":"2024-04-25T00:30:15.942363+0000","last_undegraded":"2024-04-25T00:30:15.942363+0000","last_fullsized":"2024-04-25T00:30:15.942363+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:00:54.060578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:00:54.060578+0000","last_clean_scrub_stamp":"2024-04-25T00:00:54.060578+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:14:58.472477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"74'2","reported_seq":124,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.137996+0000","last_change":"2024-04-25T00:01:18.476298+0000","last_active":"2024-04-25T00:30:26.137996+0000","last_peered":"2024-04-25T00:30:26.137996+0000","last_clean":"2024-04-25T00:30:26.137996+0000","last_became_active":"2024-04-25T00:01:18.475898+0000","last_became_peered":"2024-04-25T00:01:18.475898+0000","last_unstale":"2024-04-25T00:30:26.137996+0000","last_undegraded":"2024-04-25T00:30:26.137996+0000","last_fullsized":"2024-04-25T00:30:26.137996+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:49.341842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"74'3757","reported_seq":3877,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.774597+0000","last_change":"2024-04-25T00:01:19.518102+0000","last_active":"2024-04-25T00:30:26.774597+0000","last_peered":"2024-04-25T00:30:26.774597+0000","last_clean":"2024-04-25T00:30:26.774597+0000","last_became_active":"2024-04-25T00:01:19.517376+0000","last_became_peered":"2024-04-25T00:01:19.517376+0000","last_unstale":"2024-04-25T00:30:26.774597+0000","last_undegraded":"2024-04-25T00:30:26.774597+0000","last_fullsized":"2024-04-25T00:30:26.774597+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3757,"log_dups_size":0,"ondisk_log_size":3757,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:55:46.853565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":2,"num_write":20920,"num_write_kb":243770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"70'4367","reported_seq":4495,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.813789+0000","last_change":"2024-04-25T00:01:05.376609+0000","last_active":"2024-04-25T00:30:15.813789+0000","last_peered":"2024-04-25T00:30:15.813789+0000","last_clean":"2024-04-25T00:30:15.813789+0000","last_became_active":"2024-04-25T00:01:05.375881+0000","last_became_peered":"2024-04-25T00:01:05.375881+0000","last_unstale":"2024-04-25T00:30:15.813789+0000","last_undegraded":"2024-04-25T00:30:15.813789+0000","last_fullsized":"2024-04-25T00:30:15.813789+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4367,"log_dups_size":0,"ondisk_log_size":4367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:46:05.592233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":2,"num_write":23695,"num_write_kb":308143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"70'6","reported_seq":135,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.813175+0000","last_change":"2024-04-25T00:01:04.368954+0000","last_active":"2024-04-25T00:30:15.813175+0000","last_peered":"2024-04-25T00:30:15.813175+0000","last_clean":"2024-04-25T00:30:15.813175+0000","last_became_active":"2024-04-25T00:01:04.367903+0000","last_became_peered":"2024-04-25T00:01:04.367903+0000","last_unstale":"2024-04-25T00:30:15.813175+0000","last_undegraded":"2024-04-25T00:30:15.813175+0000","last_fullsized":"2024-04-25T00:30:15.813175+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"74'3","reported_seq":124,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.737742+0000","last_change":"2024-04-25T00:01:18.476525+0000","last_active":"2024-04-25T00:30:26.737742+0000","last_peered":"2024-04-25T00:30:26.737742+0000","last_clean":"2024-04-25T00:30:26.737742+0000","last_became_active":"2024-04-25T00:01:18.476247+0000","last_became_peered":"2024-04-25T00:01:18.476247+0000","last_unstale":"2024-04-25T00:30:26.737742+0000","last_undegraded":"2024-04-25T00:30:26.737742+0000","last_fullsized":"2024-04-25T00:30:26.737742+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:30:36.682736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"74'4936","reported_seq":5056,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.737754+0000","last_change":"2024-04-25T00:01:19.514615+0000","last_active":"2024-04-25T00:30:26.737754+0000","last_peered":"2024-04-25T00:30:26.737754+0000","last_clean":"2024-04-25T00:30:26.737754+0000","last_became_active":"2024-04-25T00:01:19.513640+0000","last_became_peered":"2024-04-25T00:01:19.513640+0000","last_unstale":"2024-04-25T00:30:26.737754+0000","last_undegraded":"2024-04-25T00:30:26.737754+0000","last_fullsized":"2024-04-25T00:30:26.737754+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4936,"log_dups_size":0,"ondisk_log_size":4936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:08.620602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":27403,"num_write_kb":306834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"70'4466","reported_seq":4593,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.793110+0000","last_change":"2024-04-25T00:01:05.385670+0000","last_active":"2024-04-25T00:30:15.793110+0000","last_peered":"2024-04-25T00:30:15.793110+0000","last_clean":"2024-04-25T00:30:15.793110+0000","last_became_active":"2024-04-25T00:01:05.384822+0000","last_became_peered":"2024-04-25T00:01:05.384822+0000","last_unstale":"2024-04-25T00:30:15.793110+0000","last_undegraded":"2024-04-25T00:30:15.793110+0000","last_fullsized":"2024-04-25T00:30:15.793110+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4466,"log_dups_size":0,"ondisk_log_size":4466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:24.017263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23831,"num_write_kb":316890,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"70'6","reported_seq":158,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.798998+0000","last_change":"2024-04-25T00:01:04.356627+0000","last_active":"2024-04-25T00:30:15.798998+0000","last_peered":"2024-04-25T00:30:15.798998+0000","last_clean":"2024-04-25T00:30:15.798998+0000","last_became_active":"2024-04-25T00:01:04.354419+0000","last_became_peered":"2024-04-25T00:01:04.354419+0000","last_unstale":"2024-04-25T00:30:15.798998+0000","last_undegraded":"2024-04-25T00:30:15.798998+0000","last_fullsized":"2024-04-25T00:30:15.798998+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:26.243635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'1453","reported_seq":1626,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.754238+0000","last_change":"2024-04-25T00:01:18.490511+0000","last_active":"2024-04-25T00:30:26.754238+0000","last_peered":"2024-04-25T00:30:26.754238+0000","last_clean":"2024-04-25T00:30:26.754238+0000","last_became_active":"2024-04-25T00:01:18.489629+0000","last_became_peered":"2024-04-25T00:01:18.489629+0000","last_unstale":"2024-04-25T00:30:26.754238+0000","last_undegraded":"2024-04-25T00:30:26.754238+0000","last_fullsized":"2024-04-25T00:30:26.754238+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":1453,"log_dups_size":0,"ondisk_log_size":1453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:07.610846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":75,"num_read_kb":41971,"num_write":1427,"num_write_kb":4503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'3796","reported_seq":3915,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.602587+0000","last_change":"2024-04-25T00:01:19.508931+0000","last_active":"2024-04-25T00:30:26.602587+0000","last_peered":"2024-04-25T00:30:26.602587+0000","last_clean":"2024-04-25T00:30:26.602587+0000","last_became_active":"2024-04-25T00:01:19.507867+0000","last_became_peered":"2024-04-25T00:01:19.507867+0000","last_unstale":"2024-04-25T00:30:26.602587+0000","last_undegraded":"2024-04-25T00:30:26.602587+0000","last_fullsized":"2024-04-25T00:30:26.602587+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3796,"log_dups_size":0,"ondisk_log_size":3796,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:38.152376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20867,"num_write_kb":276131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"70'5324","reported_seq":5452,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.792960+0000","last_change":"2024-04-25T00:01:05.378840+0000","last_active":"2024-04-25T00:30:15.792960+0000","last_peered":"2024-04-25T00:30:15.792960+0000","last_clean":"2024-04-25T00:30:15.792960+0000","last_became_active":"2024-04-25T00:01:05.378112+0000","last_became_peered":"2024-04-25T00:01:05.378112+0000","last_unstale":"2024-04-25T00:30:15.792960+0000","last_undegraded":"2024-04-25T00:30:15.792960+0000","last_fullsized":"2024-04-25T00:30:15.792960+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":5324,"log_dups_size":0,"ondisk_log_size":5324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:43.615895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":28190,"num_write_kb":387378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"69'7","reported_seq":183,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:20.770851+0000","last_change":"2024-04-25T00:01:04.370794+0000","last_active":"2024-04-25T00:30:20.770851+0000","last_peered":"2024-04-25T00:30:20.770851+0000","last_clean":"2024-04-25T00:30:20.770851+0000","last_became_active":"2024-04-25T00:01:04.370099+0000","last_became_peered":"2024-04-25T00:01:04.370099+0000","last_unstale":"2024-04-25T00:30:20.770851+0000","last_undegraded":"2024-04-25T00:30:20.770851+0000","last_fullsized":"2024-04-25T00:30:20.770851+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"74'5","reported_seq":126,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.138150+0000","last_change":"2024-04-25T00:01:18.475256+0000","last_active":"2024-04-25T00:30:26.138150+0000","last_peered":"2024-04-25T00:30:26.138150+0000","last_clean":"2024-04-25T00:30:26.138150+0000","last_became_active":"2024-04-25T00:01:18.474952+0000","last_became_peered":"2024-04-25T00:01:18.474952+0000","last_unstale":"2024-04-25T00:30:26.138150+0000","last_undegraded":"2024-04-25T00:30:26.138150+0000","last_fullsized":"2024-04-25T00:30:26.138150+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:46.849870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"74'4048","reported_seq":4167,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.238601+0000","last_change":"2024-04-25T00:01:19.516833+0000","last_active":"2024-04-25T00:30:26.238601+0000","last_peered":"2024-04-25T00:30:26.238601+0000","last_clean":"2024-04-25T00:30:26.238601+0000","last_became_active":"2024-04-25T00:01:19.515926+0000","last_became_peered":"2024-04-25T00:01:19.515926+0000","last_unstale":"2024-04-25T00:30:26.238601+0000","last_undegraded":"2024-04-25T00:30:26.238601+0000","last_fullsized":"2024-04-25T00:30:26.238601+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4048,"log_dups_size":0,"ondisk_log_size":4048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:23:48.855381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23468,"num_write_kb":270462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"70'3273","reported_seq":3401,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.808120+0000","last_change":"2024-04-25T00:01:05.372591+0000","last_active":"2024-04-25T00:30:15.808120+0000","last_peered":"2024-04-25T00:30:15.808120+0000","last_clean":"2024-04-25T00:30:15.808120+0000","last_became_active":"2024-04-25T00:01:05.371204+0000","last_became_peered":"2024-04-25T00:01:05.371204+0000","last_unstale":"2024-04-25T00:30:15.808120+0000","last_undegraded":"2024-04-25T00:30:15.808120+0000","last_fullsized":"2024-04-25T00:30:15.808120+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3273,"log_dups_size":0,"ondisk_log_size":3273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:15.879673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":17647,"num_write_kb":246988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"70'4","reported_seq":156,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.793404+0000","last_change":"2024-04-25T00:01:04.372376+0000","last_active":"2024-04-25T00:30:15.793404+0000","last_peered":"2024-04-25T00:30:15.793404+0000","last_clean":"2024-04-25T00:30:15.793404+0000","last_became_active":"2024-04-25T00:01:04.372040+0000","last_became_peered":"2024-04-25T00:01:04.372040+0000","last_unstale":"2024-04-25T00:30:15.793404+0000","last_undegraded":"2024-04-25T00:30:15.793404+0000","last_fullsized":"2024-04-25T00:30:15.793404+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:21.729893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"74'2","reported_seq":123,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.736656+0000","last_change":"2024-04-25T00:01:18.487548+0000","last_active":"2024-04-25T00:30:26.736656+0000","last_peered":"2024-04-25T00:30:26.736656+0000","last_clean":"2024-04-25T00:30:26.736656+0000","last_became_active":"2024-04-25T00:01:18.487227+0000","last_became_peered":"2024-04-25T00:01:18.487227+0000","last_unstale":"2024-04-25T00:30:26.736656+0000","last_undegraded":"2024-04-25T00:30:26.736656+0000","last_fullsized":"2024-04-25T00:30:26.736656+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:55.074018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"74'4306","reported_seq":4428,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.764052+0000","last_change":"2024-04-25T00:01:19.528574+0000","last_active":"2024-04-25T00:30:26.764052+0000","last_peered":"2024-04-25T00:30:26.764052+0000","last_clean":"2024-04-25T00:30:26.764052+0000","last_became_active":"2024-04-25T00:01:19.525531+0000","last_became_peered":"2024-04-25T00:01:19.525531+0000","last_unstale":"2024-04-25T00:30:26.764052+0000","last_undegraded":"2024-04-25T00:30:26.764052+0000","last_fullsized":"2024-04-25T00:30:26.764052+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4306,"log_dups_size":0,"ondisk_log_size":4306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":31,"num_write":22571,"num_write_kb":279217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"70'3794","reported_seq":3920,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.804529+0000","last_change":"2024-04-25T00:01:05.364103+0000","last_active":"2024-04-25T00:30:15.804529+0000","last_peered":"2024-04-25T00:30:15.804529+0000","last_clean":"2024-04-25T00:30:15.804529+0000","last_became_active":"2024-04-25T00:01:05.363320+0000","last_became_peered":"2024-04-25T00:01:05.363320+0000","last_unstale":"2024-04-25T00:30:15.804529+0000","last_undegraded":"2024-04-25T00:30:15.804529+0000","last_fullsized":"2024-04-25T00:30:15.804529+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3794,"log_dups_size":0,"ondisk_log_size":3794,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:31.456792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20831,"num_write_kb":276143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"74'15","reported_seq":258,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:25.825870+0000","last_change":"2024-04-25T00:01:04.358365+0000","last_active":"2024-04-25T00:30:25.825870+0000","last_peered":"2024-04-25T00:30:25.825870+0000","last_clean":"2024-04-25T00:30:25.825870+0000","last_became_active":"2024-04-25T00:01:04.357571+0000","last_became_peered":"2024-04-25T00:01:04.357571+0000","last_unstale":"2024-04-25T00:30:25.825870+0000","last_undegraded":"2024-04-25T00:30:25.825870+0000","last_fullsized":"2024-04-25T00:30:25.825870+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":342,"num_read_kb":342,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"74'3","reported_seq":123,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.747647+0000","last_change":"2024-04-25T00:01:18.497679+0000","last_active":"2024-04-25T00:30:26.747647+0000","last_peered":"2024-04-25T00:30:26.747647+0000","last_clean":"2024-04-25T00:30:26.747647+0000","last_became_active":"2024-04-25T00:01:18.496489+0000","last_became_peered":"2024-04-25T00:01:18.496489+0000","last_unstale":"2024-04-25T00:30:26.747647+0000","last_undegraded":"2024-04-25T00:30:26.747647+0000","last_fullsized":"2024-04-25T00:30:26.747647+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:23:47.845624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"74'4849","reported_seq":4970,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.756589+0000","last_change":"2024-04-25T00:01:19.498082+0000","last_active":"2024-04-25T00:30:26.756589+0000","last_peered":"2024-04-25T00:30:26.756589+0000","last_clean":"2024-04-25T00:30:26.756589+0000","last_became_active":"2024-04-25T00:01:19.496805+0000","last_became_peered":"2024-04-25T00:01:19.496805+0000","last_unstale":"2024-04-25T00:30:26.756589+0000","last_undegraded":"2024-04-25T00:30:26.756589+0000","last_fullsized":"2024-04-25T00:30:26.756589+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4849,"log_dups_size":0,"ondisk_log_size":4849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:57:36.256250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":16,"num_write":26442,"num_write_kb":335585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"70'2974","reported_seq":3100,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.941797+0000","last_change":"2024-04-25T00:01:05.373927+0000","last_active":"2024-04-25T00:30:15.941797+0000","last_peered":"2024-04-25T00:30:15.941797+0000","last_clean":"2024-04-25T00:30:15.941797+0000","last_became_active":"2024-04-25T00:01:05.373077+0000","last_became_peered":"2024-04-25T00:01:05.373077+0000","last_unstale":"2024-04-25T00:30:15.941797+0000","last_undegraded":"2024-04-25T00:30:15.941797+0000","last_fullsized":"2024-04-25T00:30:15.941797+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":2974,"log_dups_size":0,"ondisk_log_size":2974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:50.489125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15895,"num_write_kb":231645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"70'5","reported_seq":133,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.791512+0000","last_change":"2024-04-25T00:01:04.357265+0000","last_active":"2024-04-25T00:30:15.791512+0000","last_peered":"2024-04-25T00:30:15.791512+0000","last_clean":"2024-04-25T00:30:15.791512+0000","last_became_active":"2024-04-25T00:01:04.357004+0000","last_became_peered":"2024-04-25T00:01:04.357004+0000","last_unstale":"2024-04-25T00:30:15.791512+0000","last_undegraded":"2024-04-25T00:30:15.791512+0000","last_fullsized":"2024-04-25T00:30:15.791512+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:47:25.595275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"74'5","reported_seq":125,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.707490+0000","last_change":"2024-04-25T00:01:18.487276+0000","last_active":"2024-04-25T00:30:26.707490+0000","last_peered":"2024-04-25T00:30:26.707490+0000","last_clean":"2024-04-25T00:30:26.707490+0000","last_became_active":"2024-04-25T00:01:18.486279+0000","last_became_peered":"2024-04-25T00:01:18.486279+0000","last_unstale":"2024-04-25T00:30:26.707490+0000","last_undegraded":"2024-04-25T00:30:26.707490+0000","last_fullsized":"2024-04-25T00:30:26.707490+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:17:37.141843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"74'4040","reported_seq":4158,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.630478+0000","last_change":"2024-04-25T00:01:19.516827+0000","last_active":"2024-04-25T00:30:26.630478+0000","last_peered":"2024-04-25T00:30:26.630478+0000","last_clean":"2024-04-25T00:30:26.630478+0000","last_became_active":"2024-04-25T00:01:19.515765+0000","last_became_peered":"2024-04-25T00:01:19.515765+0000","last_unstale":"2024-04-25T00:30:26.630478+0000","last_undegraded":"2024-04-25T00:30:26.630478+0000","last_fullsized":"2024-04-25T00:30:26.630478+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4040,"log_dups_size":0,"ondisk_log_size":4040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:13:34.153117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23288,"num_write_kb":269674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"70'4514","reported_seq":4640,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.806193+0000","last_change":"2024-04-25T00:01:05.379501+0000","last_active":"2024-04-25T00:30:15.806193+0000","last_peered":"2024-04-25T00:30:15.806193+0000","last_clean":"2024-04-25T00:30:15.806193+0000","last_became_active":"2024-04-25T00:01:05.378644+0000","last_became_peered":"2024-04-25T00:01:05.378644+0000","last_unstale":"2024-04-25T00:30:15.806193+0000","last_undegraded":"2024-04-25T00:30:15.806193+0000","last_fullsized":"2024-04-25T00:30:15.806193+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4514,"log_dups_size":0,"ondisk_log_size":4514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:08:35.505154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25321,"num_write_kb":326961,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"70'3","reported_seq":131,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.809820+0000","last_change":"2024-04-25T00:01:04.360987+0000","last_active":"2024-04-25T00:30:15.809820+0000","last_peered":"2024-04-25T00:30:15.809820+0000","last_clean":"2024-04-25T00:30:15.809820+0000","last_became_active":"2024-04-25T00:01:04.360664+0000","last_became_peered":"2024-04-25T00:01:04.360664+0000","last_unstale":"2024-04-25T00:30:15.809820+0000","last_undegraded":"2024-04-25T00:30:15.809820+0000","last_fullsized":"2024-04-25T00:30:15.809820+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:55.283076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"74'335","reported_seq":572,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:21.801025+0000","last_change":"2024-04-25T00:01:18.505577+0000","last_active":"2024-04-25T00:30:21.801025+0000","last_peered":"2024-04-25T00:30:21.801025+0000","last_clean":"2024-04-25T00:30:21.801025+0000","last_became_active":"2024-04-25T00:01:18.504963+0000","last_became_peered":"2024-04-25T00:01:18.504963+0000","last_unstale":"2024-04-25T00:30:21.801025+0000","last_undegraded":"2024-04-25T00:30:21.801025+0000","last_fullsized":"2024-04-25T00:30:21.801025+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":335,"log_dups_size":0,"ondisk_log_size":335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:37.142619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":209,"num_read_kb":90302,"num_write":331,"num_write_kb":4275,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"74'4674","reported_seq":4789,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.733748+0000","last_change":"2024-04-25T00:01:46.658442+0000","last_active":"2024-04-25T00:30:26.733748+0000","last_peered":"2024-04-25T00:30:26.733748+0000","last_clean":"2024-04-25T00:30:26.733748+0000","last_became_active":"2024-04-25T00:01:46.657588+0000","last_became_peered":"2024-04-25T00:01:46.657588+0000","last_unstale":"2024-04-25T00:30:26.733748+0000","last_undegraded":"2024-04-25T00:30:26.733748+0000","last_fullsized":"2024-04-25T00:30:26.733748+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4674,"log_dups_size":0,"ondisk_log_size":4674,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:58:49.738233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":16,"num_write":25396,"num_write_kb":335454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"70'3777","reported_seq":3903,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.810481+0000","last_change":"2024-04-25T00:01:05.379102+0000","last_active":"2024-04-25T00:30:15.810481+0000","last_peered":"2024-04-25T00:30:15.810481+0000","last_clean":"2024-04-25T00:30:15.810481+0000","last_became_active":"2024-04-25T00:01:05.377625+0000","last_became_peered":"2024-04-25T00:01:05.377625+0000","last_unstale":"2024-04-25T00:30:15.810481+0000","last_undegraded":"2024-04-25T00:30:15.810481+0000","last_fullsized":"2024-04-25T00:30:15.810481+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3777,"log_dups_size":0,"ondisk_log_size":3777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:15.879673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20730,"num_write_kb":283926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"73'9","reported_seq":183,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.803482+0000","last_change":"2024-04-25T00:01:04.370640+0000","last_active":"2024-04-25T00:30:15.803482+0000","last_peered":"2024-04-25T00:30:15.803482+0000","last_clean":"2024-04-25T00:30:15.803482+0000","last_became_active":"2024-04-25T00:01:04.370031+0000","last_became_peered":"2024-04-25T00:01:04.370031+0000","last_unstale":"2024-04-25T00:30:15.803482+0000","last_undegraded":"2024-04-25T00:30:15.803482+0000","last_fullsized":"2024-04-25T00:30:15.803482+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:26.243635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":184,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"74'31","reported_seq":220,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.781231+0000","last_change":"2024-04-25T00:01:18.473113+0000","last_active":"2024-04-25T00:30:26.781231+0000","last_peered":"2024-04-25T00:30:26.781231+0000","last_clean":"2024-04-25T00:30:26.781231+0000","last_became_active":"2024-04-25T00:01:18.472931+0000","last_became_peered":"2024-04-25T00:01:18.472931+0000","last_unstale":"2024-04-25T00:30:26.781231+0000","last_undegraded":"2024-04-25T00:30:26.781231+0000","last_fullsized":"2024-04-25T00:30:26.781231+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:07.498985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":161,"num_read_kb":161,"num_write":32,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"74'5042","reported_seq":5162,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.774023+0000","last_change":"2024-04-25T00:01:19.533664+0000","last_active":"2024-04-25T00:30:26.774023+0000","last_peered":"2024-04-25T00:30:26.774023+0000","last_clean":"2024-04-25T00:30:26.774023+0000","last_became_active":"2024-04-25T00:01:19.533048+0000","last_became_peered":"2024-04-25T00:01:19.533048+0000","last_unstale":"2024-04-25T00:30:26.774023+0000","last_undegraded":"2024-04-25T00:30:26.774023+0000","last_fullsized":"2024-04-25T00:30:26.774023+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":5042,"log_dups_size":0,"ondisk_log_size":5042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":28058,"num_write_kb":350341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"69'3549","reported_seq":3675,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.790799+0000","last_change":"2024-04-25T00:01:05.385827+0000","last_active":"2024-04-25T00:30:15.790799+0000","last_peered":"2024-04-25T00:30:15.790799+0000","last_clean":"2024-04-25T00:30:15.790799+0000","last_became_active":"2024-04-25T00:01:05.385258+0000","last_became_peered":"2024-04-25T00:01:05.385258+0000","last_unstale":"2024-04-25T00:30:15.790799+0000","last_undegraded":"2024-04-25T00:30:15.790799+0000","last_fullsized":"2024-04-25T00:30:15.790799+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3549,"log_dups_size":0,"ondisk_log_size":3549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:23.870121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19806,"num_write_kb":251506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"74'341","reported_seq":585,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:22.899825+0000","last_change":"2024-04-25T00:01:04.365185+0000","last_active":"2024-04-25T00:30:22.899825+0000","last_peered":"2024-04-25T00:30:22.899825+0000","last_clean":"2024-04-25T00:30:22.899825+0000","last_became_active":"2024-04-25T00:01:04.364930+0000","last_became_peered":"2024-04-25T00:01:04.364930+0000","last_unstale":"2024-04-25T00:30:22.899825+0000","last_undegraded":"2024-04-25T00:30:22.899825+0000","last_fullsized":"2024-04-25T00:30:22.899825+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":341,"log_dups_size":0,"ondisk_log_size":341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:45:03.618340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":208,"num_read_kb":94392,"num_write":336,"num_write_kb":4266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"74'3","reported_seq":124,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.751654+0000","last_change":"2024-04-25T00:01:18.496220+0000","last_active":"2024-04-25T00:30:26.751654+0000","last_peered":"2024-04-25T00:30:26.751654+0000","last_clean":"2024-04-25T00:30:26.751654+0000","last_became_active":"2024-04-25T00:01:18.492850+0000","last_became_peered":"2024-04-25T00:01:18.492850+0000","last_unstale":"2024-04-25T00:30:26.751654+0000","last_undegraded":"2024-04-25T00:30:26.751654+0000","last_fullsized":"2024-04-25T00:30:26.751654+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"74'4133","reported_seq":4252,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.680551+0000","last_change":"2024-04-25T00:01:19.515725+0000","last_active":"2024-04-25T00:30:26.680551+0000","last_peered":"2024-04-25T00:30:26.680551+0000","last_clean":"2024-04-25T00:30:26.680551+0000","last_became_active":"2024-04-25T00:01:19.509285+0000","last_became_peered":"2024-04-25T00:01:19.509285+0000","last_unstale":"2024-04-25T00:30:26.680551+0000","last_undegraded":"2024-04-25T00:30:26.680551+0000","last_fullsized":"2024-04-25T00:30:26.680551+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4133,"log_dups_size":0,"ondisk_log_size":4133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:25:20.185355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":4,"num_write":22847,"num_write_kb":280058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"70'5270","reported_seq":5396,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.809540+0000","last_change":"2024-04-25T00:01:05.371061+0000","last_active":"2024-04-25T00:30:15.809540+0000","last_peered":"2024-04-25T00:30:15.809540+0000","last_clean":"2024-04-25T00:30:15.809540+0000","last_became_active":"2024-04-25T00:01:05.369950+0000","last_became_peered":"2024-04-25T00:01:05.369950+0000","last_unstale":"2024-04-25T00:30:15.809540+0000","last_undegraded":"2024-04-25T00:30:15.809540+0000","last_fullsized":"2024-04-25T00:30:15.809540+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":5270,"log_dups_size":0,"ondisk_log_size":5270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:02:43.237902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28224,"num_write_kb":363198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"74'31","reported_seq":205,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:25.818373+0000","last_change":"2024-04-25T00:01:04.366503+0000","last_active":"2024-04-25T00:30:25.818373+0000","last_peered":"2024-04-25T00:30:25.818373+0000","last_clean":"2024-04-25T00:30:25.818373+0000","last_became_active":"2024-04-25T00:01:04.365945+0000","last_became_peered":"2024-04-25T00:01:04.365945+0000","last_unstale":"2024-04-25T00:30:25.818373+0000","last_undegraded":"2024-04-25T00:30:25.818373+0000","last_fullsized":"2024-04-25T00:30:25.818373+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":92,"num_write":32,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"74'8","reported_seq":174,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.742489+0000","last_change":"2024-04-25T00:01:18.476456+0000","last_active":"2024-04-25T00:30:26.742489+0000","last_peered":"2024-04-25T00:30:26.742489+0000","last_clean":"2024-04-25T00:30:26.742489+0000","last_became_active":"2024-04-25T00:01:18.475876+0000","last_became_peered":"2024-04-25T00:01:18.475876+0000","last_unstale":"2024-04-25T00:30:26.742489+0000","last_undegraded":"2024-04-25T00:30:26.742489+0000","last_fullsized":"2024-04-25T00:30:26.742489+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:56.741250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":184,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"74'4233","reported_seq":4354,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.727907+0000","last_change":"2024-04-25T00:01:19.515376+0000","last_active":"2024-04-25T00:30:26.727907+0000","last_peered":"2024-04-25T00:30:26.727907+0000","last_clean":"2024-04-25T00:30:26.727907+0000","last_became_active":"2024-04-25T00:01:19.512425+0000","last_became_peered":"2024-04-25T00:01:19.512425+0000","last_unstale":"2024-04-25T00:30:26.727907+0000","last_undegraded":"2024-04-25T00:30:26.727907+0000","last_fullsized":"2024-04-25T00:30:26.727907+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4233,"log_dups_size":0,"ondisk_log_size":4233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:43:16.957273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":24291,"num_write_kb":290168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"70'4413","reported_seq":4539,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.800988+0000","last_change":"2024-04-25T00:01:05.361723+0000","last_active":"2024-04-25T00:30:15.800988+0000","last_peered":"2024-04-25T00:30:15.800988+0000","last_clean":"2024-04-25T00:30:15.800988+0000","last_became_active":"2024-04-25T00:01:05.360956+0000","last_became_peered":"2024-04-25T00:01:05.360956+0000","last_unstale":"2024-04-25T00:30:15.800988+0000","last_undegraded":"2024-04-25T00:30:15.800988+0000","last_fullsized":"2024-04-25T00:30:15.800988+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4413,"log_dups_size":0,"ondisk_log_size":4413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:23.870121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22903,"num_write_kb":321461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"70'3","reported_seq":131,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.790859+0000","last_change":"2024-04-25T00:01:04.362129+0000","last_active":"2024-04-25T00:30:15.790859+0000","last_peered":"2024-04-25T00:30:15.790859+0000","last_clean":"2024-04-25T00:30:15.790859+0000","last_became_active":"2024-04-25T00:01:04.361827+0000","last_became_peered":"2024-04-25T00:01:04.361827+0000","last_unstale":"2024-04-25T00:30:15.790859+0000","last_undegraded":"2024-04-25T00:30:15.790859+0000","last_fullsized":"2024-04-25T00:30:15.790859+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:32:26.243635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"74'15","reported_seq":253,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.739696+0000","last_change":"2024-04-25T00:01:18.496129+0000","last_active":"2024-04-25T00:30:26.739696+0000","last_peered":"2024-04-25T00:30:26.739696+0000","last_clean":"2024-04-25T00:30:26.739696+0000","last_became_active":"2024-04-25T00:01:18.495415+0000","last_became_peered":"2024-04-25T00:01:18.495415+0000","last_unstale":"2024-04-25T00:30:26.739696+0000","last_undegraded":"2024-04-25T00:30:26.739696+0000","last_fullsized":"2024-04-25T00:30:26.739696+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:02:38.089411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":354,"num_read_kb":354,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"74'3474","reported_seq":3592,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.668678+0000","last_change":"2024-04-25T00:01:19.509542+0000","last_active":"2024-04-25T00:30:26.668678+0000","last_peered":"2024-04-25T00:30:26.668678+0000","last_clean":"2024-04-25T00:30:26.668678+0000","last_became_active":"2024-04-25T00:01:19.509015+0000","last_became_peered":"2024-04-25T00:01:19.509015+0000","last_unstale":"2024-04-25T00:30:26.668678+0000","last_undegraded":"2024-04-25T00:30:26.668678+0000","last_fullsized":"2024-04-25T00:30:26.668678+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3474,"log_dups_size":0,"ondisk_log_size":3474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:25:20.185355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19390,"num_write_kb":245789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"70'4331","reported_seq":4457,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.806796+0000","last_change":"2024-04-25T00:01:05.365413+0000","last_active":"2024-04-25T00:30:15.806796+0000","last_peered":"2024-04-25T00:30:15.806796+0000","last_clean":"2024-04-25T00:30:15.806796+0000","last_became_active":"2024-04-25T00:01:05.361865+0000","last_became_peered":"2024-04-25T00:01:05.361865+0000","last_unstale":"2024-04-25T00:30:15.806796+0000","last_undegraded":"2024-04-25T00:30:15.806796+0000","last_fullsized":"2024-04-25T00:30:15.806796+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4331,"log_dups_size":0,"ondisk_log_size":4331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:30.228060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22828,"num_write_kb":312693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"74'7","reported_seq":135,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.944255+0000","last_change":"2024-04-25T00:01:04.360804+0000","last_active":"2024-04-25T00:30:15.944255+0000","last_peered":"2024-04-25T00:30:15.944255+0000","last_clean":"2024-04-25T00:30:15.944255+0000","last_became_active":"2024-04-25T00:01:04.360374+0000","last_became_peered":"2024-04-25T00:01:04.360374+0000","last_unstale":"2024-04-25T00:30:15.944255+0000","last_undegraded":"2024-04-25T00:30:15.944255+0000","last_fullsized":"2024-04-25T00:30:15.944255+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:56:28.952137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"74'5","reported_seq":126,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.745936+0000","last_change":"2024-04-25T00:01:18.480003+0000","last_active":"2024-04-25T00:30:26.745936+0000","last_peered":"2024-04-25T00:30:26.745936+0000","last_clean":"2024-04-25T00:30:26.745936+0000","last_became_active":"2024-04-25T00:01:18.479270+0000","last_became_peered":"2024-04-25T00:01:18.479270+0000","last_unstale":"2024-04-25T00:30:26.745936+0000","last_undegraded":"2024-04-25T00:30:26.745936+0000","last_fullsized":"2024-04-25T00:30:26.745936+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"74'5681","reported_seq":5805,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.778355+0000","last_change":"2024-04-25T00:01:19.524463+0000","last_active":"2024-04-25T00:30:26.778355+0000","last_peered":"2024-04-25T00:30:26.778355+0000","last_clean":"2024-04-25T00:30:26.778355+0000","last_became_active":"2024-04-25T00:01:19.524166+0000","last_became_peered":"2024-04-25T00:01:19.524166+0000","last_unstale":"2024-04-25T00:30:26.778355+0000","last_undegraded":"2024-04-25T00:30:26.778355+0000","last_fullsized":"2024-04-25T00:30:26.778355+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":5681,"log_dups_size":0,"ondisk_log_size":5681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:50.351599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":40,"num_write":31879,"num_write_kb":410681,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"74'28","reported_seq":151,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.748216+0000","last_change":"2024-04-25T00:01:18.497096+0000","last_active":"2024-04-25T00:30:26.748216+0000","last_peered":"2024-04-25T00:30:26.748216+0000","last_clean":"2024-04-25T00:30:26.748216+0000","last_became_active":"2024-04-25T00:01:18.496289+0000","last_became_peered":"2024-04-25T00:01:18.496289+0000","last_unstale":"2024-04-25T00:30:26.748216+0000","last_undegraded":"2024-04-25T00:30:26.748216+0000","last_fullsized":"2024-04-25T00:30:26.748216+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"70'4773","reported_seq":4901,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.791499+0000","last_change":"2024-04-25T00:01:05.370043+0000","last_active":"2024-04-25T00:30:15.791499+0000","last_peered":"2024-04-25T00:30:15.791499+0000","last_clean":"2024-04-25T00:30:15.791499+0000","last_became_active":"2024-04-25T00:01:05.368080+0000","last_became_peered":"2024-04-25T00:01:05.368080+0000","last_unstale":"2024-04-25T00:30:15.791499+0000","last_undegraded":"2024-04-25T00:30:15.791499+0000","last_fullsized":"2024-04-25T00:30:15.791499+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4773,"log_dups_size":0,"ondisk_log_size":4773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":25218,"num_write_kb":330520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"69'2","reported_seq":130,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.804702+0000","last_change":"2024-04-25T00:01:04.364523+0000","last_active":"2024-04-25T00:30:15.804702+0000","last_peered":"2024-04-25T00:30:15.804702+0000","last_clean":"2024-04-25T00:30:15.804702+0000","last_became_active":"2024-04-25T00:01:04.364050+0000","last_became_peered":"2024-04-25T00:01:04.364050+0000","last_unstale":"2024-04-25T00:30:15.804702+0000","last_undegraded":"2024-04-25T00:30:15.804702+0000","last_fullsized":"2024-04-25T00:30:15.804702+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:49.479804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"74'3390","reported_seq":3510,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.735796+0000","last_change":"2024-04-25T00:01:19.491523+0000","last_active":"2024-04-25T00:30:26.735796+0000","last_peered":"2024-04-25T00:30:26.735796+0000","last_clean":"2024-04-25T00:30:26.735796+0000","last_became_active":"2024-04-25T00:01:19.490755+0000","last_became_peered":"2024-04-25T00:01:19.490755+0000","last_unstale":"2024-04-25T00:30:26.735796+0000","last_undegraded":"2024-04-25T00:30:26.735796+0000","last_fullsized":"2024-04-25T00:30:26.735796+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3390,"log_dups_size":0,"ondisk_log_size":3390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:47:19.526845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18979,"num_write_kb":248305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"74'1082","reported_seq":1259,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.777825+0000","last_change":"2024-04-25T00:01:18.472794+0000","last_active":"2024-04-25T00:30:26.777825+0000","last_peered":"2024-04-25T00:30:26.777825+0000","last_clean":"2024-04-25T00:30:26.777825+0000","last_became_active":"2024-04-25T00:01:18.472534+0000","last_became_peered":"2024-04-25T00:01:18.472534+0000","last_unstale":"2024-04-25T00:30:26.777825+0000","last_undegraded":"2024-04-25T00:30:26.777825+0000","last_fullsized":"2024-04-25T00:30:26.777825+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":1082,"log_dups_size":0,"ondisk_log_size":1082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:55:45.843808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":101,"num_read_kb":11812,"num_write":1055,"num_write_kb":4728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"70'4239","reported_seq":4365,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.794924+0000","last_change":"2024-04-25T00:01:05.378867+0000","last_active":"2024-04-25T00:30:15.794924+0000","last_peered":"2024-04-25T00:30:15.794924+0000","last_clean":"2024-04-25T00:30:15.794924+0000","last_became_active":"2024-04-25T00:01:05.378056+0000","last_became_peered":"2024-04-25T00:01:05.378056+0000","last_unstale":"2024-04-25T00:30:15.794924+0000","last_undegraded":"2024-04-25T00:30:15.794924+0000","last_fullsized":"2024-04-25T00:30:15.794924+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4239,"log_dups_size":0,"ondisk_log_size":4239,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:36:10.377757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22560,"num_write_kb":311538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"70'3","reported_seq":131,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.809380+0000","last_change":"2024-04-25T00:01:04.370896+0000","last_active":"2024-04-25T00:30:15.809380+0000","last_peered":"2024-04-25T00:30:15.809380+0000","last_clean":"2024-04-25T00:30:15.809380+0000","last_became_active":"2024-04-25T00:01:04.370273+0000","last_became_peered":"2024-04-25T00:01:04.370273+0000","last_unstale":"2024-04-25T00:30:15.809380+0000","last_undegraded":"2024-04-25T00:30:15.809380+0000","last_fullsized":"2024-04-25T00:30:15.809380+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:49:22.673421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"74'5148","reported_seq":5267,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.763478+0000","last_change":"2024-04-25T00:01:19.521143+0000","last_active":"2024-04-25T00:30:26.763478+0000","last_peered":"2024-04-25T00:30:26.763478+0000","last_clean":"2024-04-25T00:30:26.763478+0000","last_became_active":"2024-04-25T00:01:19.514664+0000","last_became_peered":"2024-04-25T00:01:19.514664+0000","last_unstale":"2024-04-25T00:30:26.763478+0000","last_undegraded":"2024-04-25T00:30:26.763478+0000","last_fullsized":"2024-04-25T00:30:26.763478+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":5148,"log_dups_size":0,"ondisk_log_size":5148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":29131,"num_write_kb":338102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"74'249","reported_seq":380,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:27.894626+0000","last_change":"2024-04-25T00:01:04.360254+0000","last_active":"2024-04-25T00:30:27.894626+0000","last_peered":"2024-04-25T00:30:27.894626+0000","last_clean":"2024-04-25T00:30:27.894626+0000","last_became_active":"2024-04-25T00:01:04.359586+0000","last_became_peered":"2024-04-25T00:01:04.359586+0000","last_unstale":"2024-04-25T00:30:27.894626+0000","last_undegraded":"2024-04-25T00:30:27.894626+0000","last_fullsized":"2024-04-25T00:30:27.894626+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:47.192930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2250578,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4308,"num_write":220,"num_write_kb":2323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"69'3397","reported_seq":3523,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.792804+0000","last_change":"2024-04-25T00:01:05.370723+0000","last_active":"2024-04-25T00:30:15.792804+0000","last_peered":"2024-04-25T00:30:15.792804+0000","last_clean":"2024-04-25T00:30:15.792804+0000","last_became_active":"2024-04-25T00:01:05.368970+0000","last_became_peered":"2024-04-25T00:01:05.368970+0000","last_unstale":"2024-04-25T00:30:15.792804+0000","last_undegraded":"2024-04-25T00:30:15.792804+0000","last_fullsized":"2024-04-25T00:30:15.792804+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3397,"log_dups_size":0,"ondisk_log_size":3397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17978,"num_write_kb":244099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"74'5","reported_seq":171,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.752918+0000","last_change":"2024-04-25T00:01:18.498570+0000","last_active":"2024-04-25T00:30:26.752918+0000","last_peered":"2024-04-25T00:30:26.752918+0000","last_clean":"2024-04-25T00:30:26.752918+0000","last_became_active":"2024-04-25T00:01:18.497745+0000","last_became_peered":"2024-04-25T00:01:18.497745+0000","last_unstale":"2024-04-25T00:30:26.752918+0000","last_undegraded":"2024-04-25T00:30:26.752918+0000","last_fullsized":"2024-04-25T00:30:26.752918+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"74'3919","reported_seq":4039,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.527005+0000","last_change":"2024-04-25T00:01:19.514919+0000","last_active":"2024-04-25T00:30:26.527005+0000","last_peered":"2024-04-25T00:30:26.527005+0000","last_clean":"2024-04-25T00:30:26.527005+0000","last_became_active":"2024-04-25T00:01:19.514280+0000","last_became_peered":"2024-04-25T00:01:19.514280+0000","last_unstale":"2024-04-25T00:30:26.527005+0000","last_undegraded":"2024-04-25T00:30:26.527005+0000","last_fullsized":"2024-04-25T00:30:26.527005+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3919,"log_dups_size":0,"ondisk_log_size":3919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:31:37.975265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":20874,"num_write_kb":280510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"69'752","reported_seq":907,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.801549+0000","last_change":"2024-04-25T00:01:04.365607+0000","last_active":"2024-04-25T00:30:15.801549+0000","last_peered":"2024-04-25T00:30:15.801549+0000","last_clean":"2024-04-25T00:30:15.801549+0000","last_became_active":"2024-04-25T00:01:04.364788+0000","last_became_peered":"2024-04-25T00:01:04.364788+0000","last_unstale":"2024-04-25T00:30:15.801549+0000","last_undegraded":"2024-04-25T00:30:15.801549+0000","last_fullsized":"2024-04-25T00:30:15.801549+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":752,"log_dups_size":0,"ondisk_log_size":752,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:00:07.831658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":28,"num_read_kb":89132,"num_write":742,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"69'3790","reported_seq":3915,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.942896+0000","last_change":"2024-04-25T00:01:05.375142+0000","last_active":"2024-04-25T00:30:15.942896+0000","last_peered":"2024-04-25T00:30:15.942896+0000","last_clean":"2024-04-25T00:30:15.942896+0000","last_became_active":"2024-04-25T00:01:05.373857+0000","last_became_peered":"2024-04-25T00:01:05.373857+0000","last_unstale":"2024-04-25T00:30:15.942896+0000","last_undegraded":"2024-04-25T00:30:15.942896+0000","last_fullsized":"2024-04-25T00:30:15.942896+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3790,"log_dups_size":0,"ondisk_log_size":3790,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:15.879673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19857,"num_write_kb":263976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"74'48","reported_seq":213,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.740146+0000","last_change":"2024-04-25T00:01:18.502786+0000","last_active":"2024-04-25T00:30:26.740146+0000","last_peered":"2024-04-25T00:30:26.740146+0000","last_clean":"2024-04-25T00:30:26.740146+0000","last_became_active":"2024-04-25T00:01:18.501785+0000","last_became_peered":"2024-04-25T00:01:18.501785+0000","last_unstale":"2024-04-25T00:30:26.740146+0000","last_undegraded":"2024-04-25T00:30:26.740146+0000","last_fullsized":"2024-04-25T00:30:26.740146+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:09:31.441044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.26","version":"74'3694","reported_seq":3814,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.768669+0000","last_change":"2024-04-25T00:01:19.511239+0000","last_active":"2024-04-25T00:30:26.768669+0000","last_peered":"2024-04-25T00:30:26.768669+0000","last_clean":"2024-04-25T00:30:26.768669+0000","last_became_active":"2024-04-25T00:01:19.510155+0000","last_became_peered":"2024-04-25T00:01:19.510155+0000","last_unstale":"2024-04-25T00:30:26.768669+0000","last_undegraded":"2024-04-25T00:30:26.768669+0000","last_fullsized":"2024-04-25T00:30:26.768669+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3694,"log_dups_size":0,"ondisk_log_size":3694,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20834,"num_write_kb":259761,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.20","version":"70'4043","reported_seq":4171,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.807721+0000","last_change":"2024-04-25T00:01:05.378517+0000","last_active":"2024-04-25T00:30:15.807721+0000","last_peered":"2024-04-25T00:30:15.807721+0000","last_clean":"2024-04-25T00:30:15.807721+0000","last_became_active":"2024-04-25T00:01:05.377742+0000","last_became_peered":"2024-04-25T00:01:05.377742+0000","last_unstale":"2024-04-25T00:30:15.807721+0000","last_undegraded":"2024-04-25T00:30:15.807721+0000","last_fullsized":"2024-04-25T00:30:15.807721+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4043,"log_dups_size":0,"ondisk_log_size":4043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:06.450369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21543,"num_write_kb":291017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"74'5759","reported_seq":5880,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.776612+0000","last_change":"2024-04-25T00:01:19.508999+0000","last_active":"2024-04-25T00:30:26.776612+0000","last_peered":"2024-04-25T00:30:26.776612+0000","last_clean":"2024-04-25T00:30:26.776612+0000","last_became_active":"2024-04-25T00:01:19.507960+0000","last_became_peered":"2024-04-25T00:01:19.507960+0000","last_unstale":"2024-04-25T00:30:26.776612+0000","last_undegraded":"2024-04-25T00:30:26.776612+0000","last_fullsized":"2024-04-25T00:30:26.776612+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":5759,"log_dups_size":0,"ondisk_log_size":5759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":31168,"num_write_kb":366885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.21","version":"70'4068","reported_seq":4197,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.803508+0000","last_change":"2024-04-25T00:01:05.372927+0000","last_active":"2024-04-25T00:30:15.803508+0000","last_peered":"2024-04-25T00:30:15.803508+0000","last_clean":"2024-04-25T00:30:15.803508+0000","last_became_active":"2024-04-25T00:01:05.372050+0000","last_became_peered":"2024-04-25T00:01:05.372050+0000","last_unstale":"2024-04-25T00:30:15.803508+0000","last_undegraded":"2024-04-25T00:30:15.803508+0000","last_fullsized":"2024-04-25T00:30:15.803508+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4068,"log_dups_size":0,"ondisk_log_size":4068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:48.202251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21791,"num_write_kb":292243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.24","version":"74'4315","reported_seq":4436,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.724152+0000","last_change":"2024-04-25T00:01:19.531072+0000","last_active":"2024-04-25T00:30:26.724152+0000","last_peered":"2024-04-25T00:30:26.724152+0000","last_clean":"2024-04-25T00:30:26.724152+0000","last_became_active":"2024-04-25T00:01:19.530836+0000","last_became_peered":"2024-04-25T00:01:19.530836+0000","last_unstale":"2024-04-25T00:30:26.724152+0000","last_undegraded":"2024-04-25T00:30:26.724152+0000","last_fullsized":"2024-04-25T00:30:26.724152+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4315,"log_dups_size":0,"ondisk_log_size":4315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:08.508741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":23956,"num_write_kb":309408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"70'4267","reported_seq":4395,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.807204+0000","last_change":"2024-04-25T00:01:05.362960+0000","last_active":"2024-04-25T00:30:15.807204+0000","last_peered":"2024-04-25T00:30:15.807204+0000","last_clean":"2024-04-25T00:30:15.807204+0000","last_became_active":"2024-04-25T00:01:05.361067+0000","last_became_peered":"2024-04-25T00:01:05.361067+0000","last_unstale":"2024-04-25T00:30:15.807204+0000","last_undegraded":"2024-04-25T00:30:15.807204+0000","last_fullsized":"2024-04-25T00:30:15.807204+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4267,"log_dups_size":0,"ondisk_log_size":4267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:14:24.017263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23473,"num_write_kb":311317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"74'4634","reported_seq":4754,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.737724+0000","last_change":"2024-04-25T00:01:19.506029+0000","last_active":"2024-04-25T00:30:26.737724+0000","last_peered":"2024-04-25T00:30:26.737724+0000","last_clean":"2024-04-25T00:30:26.737724+0000","last_became_active":"2024-04-25T00:01:19.505184+0000","last_became_peered":"2024-04-25T00:01:19.505184+0000","last_unstale":"2024-04-25T00:30:26.737724+0000","last_undegraded":"2024-04-25T00:30:26.737724+0000","last_fullsized":"2024-04-25T00:30:26.737724+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4634,"log_dups_size":0,"ondisk_log_size":4634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:08.508741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":6,"num_write":26896,"num_write_kb":316898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"70'4083","reported_seq":4211,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.801208+0000","last_change":"2024-04-25T00:01:05.358690+0000","last_active":"2024-04-25T00:30:15.801208+0000","last_peered":"2024-04-25T00:30:15.801208+0000","last_clean":"2024-04-25T00:30:15.801208+0000","last_became_active":"2024-04-25T00:01:05.357340+0000","last_became_peered":"2024-04-25T00:01:05.357340+0000","last_unstale":"2024-04-25T00:30:15.801208+0000","last_undegraded":"2024-04-25T00:30:15.801208+0000","last_fullsized":"2024-04-25T00:30:15.801208+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4083,"log_dups_size":0,"ondisk_log_size":4083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:36:55.436661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21935,"num_write_kb":296816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.22","version":"74'5147","reported_seq":5272,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.738948+0000","last_change":"2024-04-25T00:01:19.508691+0000","last_active":"2024-04-25T00:30:26.738948+0000","last_peered":"2024-04-25T00:30:26.738948+0000","last_clean":"2024-04-25T00:30:26.738948+0000","last_became_active":"2024-04-25T00:01:19.508123+0000","last_became_peered":"2024-04-25T00:01:19.508123+0000","last_unstale":"2024-04-25T00:30:26.738948+0000","last_undegraded":"2024-04-25T00:30:26.738948+0000","last_fullsized":"2024-04-25T00:30:26.738948+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":5147,"log_dups_size":0,"ondisk_log_size":5147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:47:19.526845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":43,"num_write":27901,"num_write_kb":341497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"70'3848","reported_seq":3976,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.812551+0000","last_change":"2024-04-25T00:01:05.387057+0000","last_active":"2024-04-25T00:30:15.812551+0000","last_peered":"2024-04-25T00:30:15.812551+0000","last_clean":"2024-04-25T00:30:15.812551+0000","last_became_active":"2024-04-25T00:01:05.386724+0000","last_became_peered":"2024-04-25T00:01:05.386724+0000","last_unstale":"2024-04-25T00:30:15.812551+0000","last_undegraded":"2024-04-25T00:30:15.812551+0000","last_fullsized":"2024-04-25T00:30:15.812551+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3848,"log_dups_size":0,"ondisk_log_size":3848,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:55:09.073286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20848,"num_write_kb":276248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.23","version":"74'4386","reported_seq":4508,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.730888+0000","last_change":"2024-04-25T00:01:19.522760+0000","last_active":"2024-04-25T00:30:26.730888+0000","last_peered":"2024-04-25T00:30:26.730888+0000","last_clean":"2024-04-25T00:30:26.730888+0000","last_became_active":"2024-04-25T00:01:19.522380+0000","last_became_peered":"2024-04-25T00:01:19.522380+0000","last_unstale":"2024-04-25T00:30:26.730888+0000","last_undegraded":"2024-04-25T00:30:26.730888+0000","last_fullsized":"2024-04-25T00:30:26.730888+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4386,"log_dups_size":0,"ondisk_log_size":4386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:55:46.853565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":23395,"num_write_kb":294532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.25","version":"70'4349","reported_seq":4476,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.812426+0000","last_change":"2024-04-25T00:01:05.367296+0000","last_active":"2024-04-25T00:30:15.812426+0000","last_peered":"2024-04-25T00:30:15.812426+0000","last_clean":"2024-04-25T00:30:15.812426+0000","last_became_active":"2024-04-25T00:01:05.363838+0000","last_became_peered":"2024-04-25T00:01:05.363838+0000","last_unstale":"2024-04-25T00:30:15.812426+0000","last_undegraded":"2024-04-25T00:30:15.812426+0000","last_fullsized":"2024-04-25T00:30:15.812426+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4349,"log_dups_size":0,"ondisk_log_size":4349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:43:07.053074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24353,"num_write_kb":306782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"74'4505","reported_seq":4625,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.749021+0000","last_change":"2024-04-25T00:01:19.510902+0000","last_active":"2024-04-25T00:30:26.749021+0000","last_peered":"2024-04-25T00:30:26.749021+0000","last_clean":"2024-04-25T00:30:26.749021+0000","last_became_active":"2024-04-25T00:01:19.510008+0000","last_became_peered":"2024-04-25T00:01:19.510008+0000","last_unstale":"2024-04-25T00:30:26.749021+0000","last_undegraded":"2024-04-25T00:30:26.749021+0000","last_fullsized":"2024-04-25T00:30:26.749021+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4505,"log_dups_size":0,"ondisk_log_size":4505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:40:32.292140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24621,"num_write_kb":315853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"70'3623","reported_seq":3750,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.806689+0000","last_change":"2024-04-25T00:01:05.365761+0000","last_active":"2024-04-25T00:30:15.806689+0000","last_peered":"2024-04-25T00:30:15.806689+0000","last_clean":"2024-04-25T00:30:15.806689+0000","last_became_active":"2024-04-25T00:01:05.361119+0000","last_became_peered":"2024-04-25T00:01:05.361119+0000","last_unstale":"2024-04-25T00:30:15.806689+0000","last_undegraded":"2024-04-25T00:30:15.806689+0000","last_fullsized":"2024-04-25T00:30:15.806689+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3623,"log_dups_size":0,"ondisk_log_size":3623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20289,"num_write_kb":274005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.21","version":"74'4291","reported_seq":4414,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.777806+0000","last_change":"2024-04-25T00:01:19.518534+0000","last_active":"2024-04-25T00:30:26.777806+0000","last_peered":"2024-04-25T00:30:26.777806+0000","last_clean":"2024-04-25T00:30:26.777806+0000","last_became_active":"2024-04-25T00:01:19.518110+0000","last_became_peered":"2024-04-25T00:01:19.518110+0000","last_unstale":"2024-04-25T00:30:26.777806+0000","last_undegraded":"2024-04-25T00:30:26.777806+0000","last_fullsized":"2024-04-25T00:30:26.777806+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4291,"log_dups_size":0,"ondisk_log_size":4291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:05:09.219823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":23037,"num_write_kb":301131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"69'4983","reported_seq":5110,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.799145+0000","last_change":"2024-04-25T00:01:05.373464+0000","last_active":"2024-04-25T00:30:15.799145+0000","last_peered":"2024-04-25T00:30:15.799145+0000","last_clean":"2024-04-25T00:30:15.799145+0000","last_became_active":"2024-04-25T00:01:05.372611+0000","last_became_peered":"2024-04-25T00:01:05.372611+0000","last_unstale":"2024-04-25T00:30:15.799145+0000","last_undegraded":"2024-04-25T00:30:15.799145+0000","last_fullsized":"2024-04-25T00:30:15.799145+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4983,"log_dups_size":0,"ondisk_log_size":4983,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:31.456792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26776,"num_write_kb":349057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2e","version":"74'4567","reported_seq":4690,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.735238+0000","last_change":"2024-04-25T00:01:19.542475+0000","last_active":"2024-04-25T00:30:26.735238+0000","last_peered":"2024-04-25T00:30:26.735238+0000","last_clean":"2024-04-25T00:30:26.735238+0000","last_became_active":"2024-04-25T00:01:19.541877+0000","last_became_peered":"2024-04-25T00:01:19.541877+0000","last_unstale":"2024-04-25T00:30:26.735238+0000","last_undegraded":"2024-04-25T00:30:26.735238+0000","last_fullsized":"2024-04-25T00:30:26.735238+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4567,"log_dups_size":0,"ondisk_log_size":4567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:02:57.373014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":36,"num_write":24879,"num_write_kb":312940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.28","version":"70'4294","reported_seq":4421,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.804148+0000","last_change":"2024-04-25T00:01:05.372906+0000","last_active":"2024-04-25T00:30:15.804148+0000","last_peered":"2024-04-25T00:30:15.804148+0000","last_clean":"2024-04-25T00:30:15.804148+0000","last_became_active":"2024-04-25T00:01:05.372148+0000","last_became_peered":"2024-04-25T00:01:05.372148+0000","last_unstale":"2024-04-25T00:30:15.804148+0000","last_undegraded":"2024-04-25T00:30:15.804148+0000","last_fullsized":"2024-04-25T00:30:15.804148+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4294,"log_dups_size":0,"ondisk_log_size":4294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:43.615895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23157,"num_write_kb":289708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2f","version":"74'5194","reported_seq":5313,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.771951+0000","last_change":"2024-04-25T00:01:19.498515+0000","last_active":"2024-04-25T00:30:26.771951+0000","last_peered":"2024-04-25T00:30:26.771951+0000","last_clean":"2024-04-25T00:30:26.771951+0000","last_became_active":"2024-04-25T00:01:19.496310+0000","last_became_peered":"2024-04-25T00:01:19.496310+0000","last_unstale":"2024-04-25T00:30:26.771951+0000","last_undegraded":"2024-04-25T00:30:26.771951+0000","last_fullsized":"2024-04-25T00:30:26.771951+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":5194,"log_dups_size":0,"ondisk_log_size":5194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:20:06.661558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":28420,"num_write_kb":367977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"69'4077","reported_seq":4204,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.795788+0000","last_change":"2024-04-25T00:01:05.366340+0000","last_active":"2024-04-25T00:30:15.795788+0000","last_peered":"2024-04-25T00:30:15.795788+0000","last_clean":"2024-04-25T00:30:15.795788+0000","last_became_active":"2024-04-25T00:01:05.361952+0000","last_became_peered":"2024-04-25T00:01:05.361952+0000","last_unstale":"2024-04-25T00:30:15.795788+0000","last_undegraded":"2024-04-25T00:30:15.795788+0000","last_fullsized":"2024-04-25T00:30:15.795788+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4077,"log_dups_size":0,"ondisk_log_size":4077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:51:41.948662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22153,"num_write_kb":297857,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2c","version":"74'3564","reported_seq":3684,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.750933+0000","last_change":"2024-04-25T00:01:19.508257+0000","last_active":"2024-04-25T00:30:26.750933+0000","last_peered":"2024-04-25T00:30:26.750933+0000","last_clean":"2024-04-25T00:30:26.750933+0000","last_became_active":"2024-04-25T00:01:19.507516+0000","last_became_peered":"2024-04-25T00:01:19.507516+0000","last_unstale":"2024-04-25T00:30:26.750933+0000","last_undegraded":"2024-04-25T00:30:26.750933+0000","last_fullsized":"2024-04-25T00:30:26.750933+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3564,"log_dups_size":0,"ondisk_log_size":3564,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:45.121513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":19793,"num_write_kb":247542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"69'4229","reported_seq":4356,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.942181+0000","last_change":"2024-04-25T00:01:05.387457+0000","last_active":"2024-04-25T00:30:15.942181+0000","last_peered":"2024-04-25T00:30:15.942181+0000","last_clean":"2024-04-25T00:30:15.942181+0000","last_became_active":"2024-04-25T00:01:05.386919+0000","last_became_peered":"2024-04-25T00:01:05.386919+0000","last_unstale":"2024-04-25T00:30:15.942181+0000","last_undegraded":"2024-04-25T00:30:15.942181+0000","last_fullsized":"2024-04-25T00:30:15.942181+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4229,"log_dups_size":0,"ondisk_log_size":4229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22430,"num_write_kb":302838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2d","version":"74'4662","reported_seq":4782,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.708290+0000","last_change":"2024-04-25T00:01:19.520648+0000","last_active":"2024-04-25T00:30:26.708290+0000","last_peered":"2024-04-25T00:30:26.708290+0000","last_clean":"2024-04-25T00:30:26.708290+0000","last_became_active":"2024-04-25T00:01:19.519303+0000","last_became_peered":"2024-04-25T00:01:19.519303+0000","last_unstale":"2024-04-25T00:30:26.708290+0000","last_undegraded":"2024-04-25T00:30:26.708290+0000","last_fullsized":"2024-04-25T00:30:26.708290+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4662,"log_dups_size":0,"ondisk_log_size":4662,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:23.993729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":25332,"num_write_kb":310579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2b","version":"70'4434","reported_seq":4563,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:20.770756+0000","last_change":"2024-04-25T00:01:05.366249+0000","last_active":"2024-04-25T00:30:20.770756+0000","last_peered":"2024-04-25T00:30:20.770756+0000","last_clean":"2024-04-25T00:30:20.770756+0000","last_became_active":"2024-04-25T00:01:05.365548+0000","last_became_peered":"2024-04-25T00:01:05.365548+0000","last_unstale":"2024-04-25T00:30:20.770756+0000","last_undegraded":"2024-04-25T00:30:20.770756+0000","last_fullsized":"2024-04-25T00:30:20.770756+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4434,"log_dups_size":0,"ondisk_log_size":4434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":24224,"num_write_kb":326776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"74'4527","reported_seq":4647,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.719005+0000","last_change":"2024-04-25T00:01:19.514659+0000","last_active":"2024-04-25T00:30:26.719005+0000","last_peered":"2024-04-25T00:30:26.719005+0000","last_clean":"2024-04-25T00:30:26.719005+0000","last_became_active":"2024-04-25T00:01:19.514232+0000","last_became_peered":"2024-04-25T00:01:19.514232+0000","last_unstale":"2024-04-25T00:30:26.719005+0000","last_undegraded":"2024-04-25T00:30:26.719005+0000","last_fullsized":"2024-04-25T00:30:26.719005+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4527,"log_dups_size":0,"ondisk_log_size":4527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:37:45.121513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":25088,"num_write_kb":313806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"70'3677","reported_seq":3789,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.802932+0000","last_change":"2024-04-25T00:01:46.665943+0000","last_active":"2024-04-25T00:30:15.802932+0000","last_peered":"2024-04-25T00:30:15.802932+0000","last_clean":"2024-04-25T00:30:15.802932+0000","last_became_active":"2024-04-25T00:01:46.663874+0000","last_became_peered":"2024-04-25T00:01:46.663874+0000","last_unstale":"2024-04-25T00:30:15.802932+0000","last_undegraded":"2024-04-25T00:30:15.802932+0000","last_fullsized":"2024-04-25T00:30:15.802932+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3677,"log_dups_size":0,"ondisk_log_size":3677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:53:00.633407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20325,"num_write_kb":253590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2b","version":"74'4916","reported_seq":5040,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.774860+0000","last_change":"2024-04-25T00:01:19.503419+0000","last_active":"2024-04-25T00:30:26.774860+0000","last_peered":"2024-04-25T00:30:26.774860+0000","last_clean":"2024-04-25T00:30:26.774860+0000","last_became_active":"2024-04-25T00:01:19.500568+0000","last_became_peered":"2024-04-25T00:01:19.500568+0000","last_unstale":"2024-04-25T00:30:26.774860+0000","last_undegraded":"2024-04-25T00:30:26.774860+0000","last_fullsized":"2024-04-25T00:30:26.774860+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4916,"log_dups_size":0,"ondisk_log_size":4916,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:08.508741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":26841,"num_write_kb":321097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"69'4618","reported_seq":4745,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.814383+0000","last_change":"2024-04-25T00:01:05.370653+0000","last_active":"2024-04-25T00:30:15.814383+0000","last_peered":"2024-04-25T00:30:15.814383+0000","last_clean":"2024-04-25T00:30:15.814383+0000","last_became_active":"2024-04-25T00:01:05.369919+0000","last_became_peered":"2024-04-25T00:01:05.369919+0000","last_unstale":"2024-04-25T00:30:15.814383+0000","last_undegraded":"2024-04-25T00:30:15.814383+0000","last_fullsized":"2024-04-25T00:30:15.814383+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4618,"log_dups_size":0,"ondisk_log_size":4618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:30:08.311326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25225,"num_write_kb":326547,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.28","version":"74'3900","reported_seq":4020,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.755166+0000","last_change":"2024-04-25T00:01:19.540104+0000","last_active":"2024-04-25T00:30:26.755166+0000","last_peered":"2024-04-25T00:30:26.755166+0000","last_clean":"2024-04-25T00:30:26.755166+0000","last_became_active":"2024-04-25T00:01:19.539775+0000","last_became_peered":"2024-04-25T00:01:19.539775+0000","last_unstale":"2024-04-25T00:30:26.755166+0000","last_undegraded":"2024-04-25T00:30:26.755166+0000","last_fullsized":"2024-04-25T00:30:26.755166+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3900,"log_dups_size":0,"ondisk_log_size":3900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:53:14.768520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21234,"num_write_kb":281892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"70'3829","reported_seq":3955,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.804122+0000","last_change":"2024-04-25T00:01:05.367001+0000","last_active":"2024-04-25T00:30:15.804122+0000","last_peered":"2024-04-25T00:30:15.804122+0000","last_clean":"2024-04-25T00:30:15.804122+0000","last_became_active":"2024-04-25T00:01:05.366264+0000","last_became_peered":"2024-04-25T00:01:05.366264+0000","last_unstale":"2024-04-25T00:30:15.804122+0000","last_undegraded":"2024-04-25T00:30:15.804122+0000","last_fullsized":"2024-04-25T00:30:15.804122+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3829,"log_dups_size":0,"ondisk_log_size":3829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:15:56.292397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19822,"num_write_kb":292385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.29","version":"74'4218","reported_seq":4338,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.734841+0000","last_change":"2024-04-25T00:01:19.497247+0000","last_active":"2024-04-25T00:30:26.734841+0000","last_peered":"2024-04-25T00:30:26.734841+0000","last_clean":"2024-04-25T00:30:26.734841+0000","last_became_active":"2024-04-25T00:01:19.496517+0000","last_became_peered":"2024-04-25T00:01:19.496517+0000","last_unstale":"2024-04-25T00:30:26.734841+0000","last_undegraded":"2024-04-25T00:30:26.734841+0000","last_fullsized":"2024-04-25T00:30:26.734841+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4218,"log_dups_size":0,"ondisk_log_size":4218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:32:30.129056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":23201,"num_write_kb":293725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2f","version":"70'5230","reported_seq":5357,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.813628+0000","last_change":"2024-04-25T00:01:05.379958+0000","last_active":"2024-04-25T00:30:15.813628+0000","last_peered":"2024-04-25T00:30:15.813628+0000","last_clean":"2024-04-25T00:30:15.813628+0000","last_became_active":"2024-04-25T00:01:05.379090+0000","last_became_peered":"2024-04-25T00:01:05.379090+0000","last_unstale":"2024-04-25T00:30:15.813628+0000","last_undegraded":"2024-04-25T00:30:15.813628+0000","last_fullsized":"2024-04-25T00:30:15.813628+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":5230,"log_dups_size":0,"ondisk_log_size":5230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:59:50.832958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":27866,"num_write_kb":365657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.36","version":"74'4384","reported_seq":4505,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.728573+0000","last_change":"2024-04-25T00:01:19.530176+0000","last_active":"2024-04-25T00:30:26.728573+0000","last_peered":"2024-04-25T00:30:26.728573+0000","last_clean":"2024-04-25T00:30:26.728573+0000","last_became_active":"2024-04-25T00:01:19.529410+0000","last_became_peered":"2024-04-25T00:01:19.529410+0000","last_unstale":"2024-04-25T00:30:26.728573+0000","last_undegraded":"2024-04-25T00:30:26.728573+0000","last_fullsized":"2024-04-25T00:30:26.728573+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4384,"log_dups_size":0,"ondisk_log_size":4384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":7,"num_write":24230,"num_write_kb":310193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.30","version":"69'4051","reported_seq":4177,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.803573+0000","last_change":"2024-04-25T00:01:05.370826+0000","last_active":"2024-04-25T00:30:15.803573+0000","last_peered":"2024-04-25T00:30:15.803573+0000","last_clean":"2024-04-25T00:30:15.803573+0000","last_became_active":"2024-04-25T00:01:05.369848+0000","last_became_peered":"2024-04-25T00:01:05.369848+0000","last_unstale":"2024-04-25T00:30:15.803573+0000","last_undegraded":"2024-04-25T00:30:15.803573+0000","last_fullsized":"2024-04-25T00:30:15.803573+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4051,"log_dups_size":0,"ondisk_log_size":4051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:23.870121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21793,"num_write_kb":288101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.37","version":"74'6689","reported_seq":6811,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.769600+0000","last_change":"2024-04-25T00:01:19.511625+0000","last_active":"2024-04-25T00:30:26.769600+0000","last_peered":"2024-04-25T00:30:26.769600+0000","last_clean":"2024-04-25T00:30:26.769600+0000","last_became_active":"2024-04-25T00:01:19.511349+0000","last_became_peered":"2024-04-25T00:01:19.511349+0000","last_unstale":"2024-04-25T00:30:26.769600+0000","last_undegraded":"2024-04-25T00:30:26.769600+0000","last_fullsized":"2024-04-25T00:30:26.769600+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":6689,"log_dups_size":0,"ondisk_log_size":6689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:02:57.373014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":8,"num_write":38367,"num_write_kb":445068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"70'3993","reported_seq":4119,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.795313+0000","last_change":"2024-04-25T00:01:05.390341+0000","last_active":"2024-04-25T00:30:15.795313+0000","last_peered":"2024-04-25T00:30:15.795313+0000","last_clean":"2024-04-25T00:30:15.795313+0000","last_became_active":"2024-04-25T00:01:05.389707+0000","last_became_peered":"2024-04-25T00:01:05.389707+0000","last_unstale":"2024-04-25T00:30:15.795313+0000","last_undegraded":"2024-04-25T00:30:15.795313+0000","last_fullsized":"2024-04-25T00:30:15.795313+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3993,"log_dups_size":0,"ondisk_log_size":3993,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:54:21.138947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21619,"num_write_kb":287362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.34","version":"74'3411","reported_seq":3532,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.695377+0000","last_change":"2024-04-25T00:01:19.513630+0000","last_active":"2024-04-25T00:30:26.695377+0000","last_peered":"2024-04-25T00:30:26.695377+0000","last_clean":"2024-04-25T00:30:26.695377+0000","last_became_active":"2024-04-25T00:01:19.512978+0000","last_became_peered":"2024-04-25T00:01:19.512978+0000","last_unstale":"2024-04-25T00:30:26.695377+0000","last_undegraded":"2024-04-25T00:30:26.695377+0000","last_fullsized":"2024-04-25T00:30:26.695377+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3411,"log_dups_size":0,"ondisk_log_size":3411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:57.751007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":34,"num_write":18670,"num_write_kb":234685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"70'3759","reported_seq":3886,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.809965+0000","last_change":"2024-04-25T00:01:05.378373+0000","last_active":"2024-04-25T00:30:15.809965+0000","last_peered":"2024-04-25T00:30:15.809965+0000","last_clean":"2024-04-25T00:30:15.809965+0000","last_became_active":"2024-04-25T00:01:05.376570+0000","last_became_peered":"2024-04-25T00:01:05.376570+0000","last_unstale":"2024-04-25T00:30:15.809965+0000","last_undegraded":"2024-04-25T00:30:15.809965+0000","last_fullsized":"2024-04-25T00:30:15.809965+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3759,"log_dups_size":0,"ondisk_log_size":3759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:30.228060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20571,"num_write_kb":279348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.35","version":"74'4078","reported_seq":4199,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.645015+0000","last_change":"2024-04-25T00:01:19.516716+0000","last_active":"2024-04-25T00:30:26.645015+0000","last_peered":"2024-04-25T00:30:26.645015+0000","last_clean":"2024-04-25T00:30:26.645015+0000","last_became_active":"2024-04-25T00:01:19.516142+0000","last_became_peered":"2024-04-25T00:01:19.516142+0000","last_unstale":"2024-04-25T00:30:26.645015+0000","last_undegraded":"2024-04-25T00:30:26.645015+0000","last_fullsized":"2024-04-25T00:30:26.645015+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4078,"log_dups_size":0,"ondisk_log_size":4078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:28:18.234546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":22241,"num_write_kb":273608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.33","version":"70'3191","reported_seq":3317,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.810448+0000","last_change":"2024-04-25T00:01:05.381011+0000","last_active":"2024-04-25T00:30:15.810448+0000","last_peered":"2024-04-25T00:30:15.810448+0000","last_clean":"2024-04-25T00:30:15.810448+0000","last_became_active":"2024-04-25T00:01:05.380236+0000","last_became_peered":"2024-04-25T00:01:05.380236+0000","last_unstale":"2024-04-25T00:30:15.810448+0000","last_undegraded":"2024-04-25T00:30:15.810448+0000","last_fullsized":"2024-04-25T00:30:15.810448+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3191,"log_dups_size":0,"ondisk_log_size":3191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:06:19.307378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17526,"num_write_kb":230308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"74'4055","reported_seq":4175,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.764545+0000","last_change":"2024-04-25T00:01:19.529023+0000","last_active":"2024-04-25T00:30:26.764545+0000","last_peered":"2024-04-25T00:30:26.764545+0000","last_clean":"2024-04-25T00:30:26.764545+0000","last_became_active":"2024-04-25T00:01:19.524609+0000","last_became_peered":"2024-04-25T00:01:19.524609+0000","last_unstale":"2024-04-25T00:30:26.764545+0000","last_undegraded":"2024-04-25T00:30:26.764545+0000","last_fullsized":"2024-04-25T00:30:26.764545+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4055,"log_dups_size":0,"ondisk_log_size":4055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":21801,"num_write_kb":276048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.34","version":"70'2711","reported_seq":2837,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.792324+0000","last_change":"2024-04-25T00:01:05.385853+0000","last_active":"2024-04-25T00:30:15.792324+0000","last_peered":"2024-04-25T00:30:15.792324+0000","last_clean":"2024-04-25T00:30:15.792324+0000","last_became_active":"2024-04-25T00:01:05.384997+0000","last_became_peered":"2024-04-25T00:01:05.384997+0000","last_unstale":"2024-04-25T00:30:15.792324+0000","last_undegraded":"2024-04-25T00:30:15.792324+0000","last_fullsized":"2024-04-25T00:30:15.792324+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":2711,"log_dups_size":0,"ondisk_log_size":2711,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:36:55.436661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14839,"num_write_kb":207133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.33","version":"74'3434","reported_seq":3556,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.729873+0000","last_change":"2024-04-25T00:01:19.516867+0000","last_active":"2024-04-25T00:30:26.729873+0000","last_peered":"2024-04-25T00:30:26.729873+0000","last_clean":"2024-04-25T00:30:26.729873+0000","last_became_active":"2024-04-25T00:01:19.515764+0000","last_became_peered":"2024-04-25T00:01:19.515764+0000","last_unstale":"2024-04-25T00:30:26.729873+0000","last_undegraded":"2024-04-25T00:30:26.729873+0000","last_fullsized":"2024-04-25T00:30:26.729873+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3434,"log_dups_size":0,"ondisk_log_size":3434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:53:14.768520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":5,"num_write":19347,"num_write_kb":241776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"69'3819","reported_seq":3946,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.790737+0000","last_change":"2024-04-25T00:01:05.389424+0000","last_active":"2024-04-25T00:30:15.790737+0000","last_peered":"2024-04-25T00:30:15.790737+0000","last_clean":"2024-04-25T00:30:15.790737+0000","last_became_active":"2024-04-25T00:01:05.388999+0000","last_became_peered":"2024-04-25T00:01:05.388999+0000","last_unstale":"2024-04-25T00:30:15.790737+0000","last_undegraded":"2024-04-25T00:30:15.790737+0000","last_fullsized":"2024-04-25T00:30:15.790737+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3819,"log_dups_size":0,"ondisk_log_size":3819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:25:31.456792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":19986,"num_write_kb":272748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.30","version":"74'4335","reported_seq":4453,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.723246+0000","last_change":"2024-04-25T00:01:19.519365+0000","last_active":"2024-04-25T00:30:26.723246+0000","last_peered":"2024-04-25T00:30:26.723246+0000","last_clean":"2024-04-25T00:30:26.723246+0000","last_became_active":"2024-04-25T00:01:19.519085+0000","last_became_peered":"2024-04-25T00:01:19.519085+0000","last_unstale":"2024-04-25T00:30:26.723246+0000","last_undegraded":"2024-04-25T00:30:26.723246+0000","last_fullsized":"2024-04-25T00:30:26.723246+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4335,"log_dups_size":0,"ondisk_log_size":4335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:05:09.219823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23507,"num_write_kb":286989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"69'4866","reported_seq":4992,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.790509+0000","last_change":"2024-04-25T00:01:05.368974+0000","last_active":"2024-04-25T00:30:15.790509+0000","last_peered":"2024-04-25T00:30:15.790509+0000","last_clean":"2024-04-25T00:30:15.790509+0000","last_became_active":"2024-04-25T00:01:05.367893+0000","last_became_peered":"2024-04-25T00:01:05.367893+0000","last_unstale":"2024-04-25T00:30:15.790509+0000","last_undegraded":"2024-04-25T00:30:15.790509+0000","last_fullsized":"2024-04-25T00:30:15.790509+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4866,"log_dups_size":0,"ondisk_log_size":4866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:30:23.557381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26504,"num_write_kb":327449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.31","version":"74'4017","reported_seq":4136,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.738931+0000","last_change":"2024-04-25T00:01:19.513514+0000","last_active":"2024-04-25T00:30:26.738931+0000","last_peered":"2024-04-25T00:30:26.738931+0000","last_clean":"2024-04-25T00:30:26.738931+0000","last_became_active":"2024-04-25T00:01:19.513184+0000","last_became_peered":"2024-04-25T00:01:19.513184+0000","last_unstale":"2024-04-25T00:30:26.738931+0000","last_undegraded":"2024-04-25T00:30:26.738931+0000","last_fullsized":"2024-04-25T00:30:26.738931+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4017,"log_dups_size":0,"ondisk_log_size":4017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:47.859626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":21984,"num_write_kb":288846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"70'5540","reported_seq":5666,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.944163+0000","last_change":"2024-04-25T00:01:05.362303+0000","last_active":"2024-04-25T00:30:15.944163+0000","last_peered":"2024-04-25T00:30:15.944163+0000","last_clean":"2024-04-25T00:30:15.944163+0000","last_became_active":"2024-04-25T00:01:05.361586+0000","last_became_peered":"2024-04-25T00:01:05.361586+0000","last_unstale":"2024-04-25T00:30:15.944163+0000","last_undegraded":"2024-04-25T00:30:15.944163+0000","last_fullsized":"2024-04-25T00:30:15.944163+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":5540,"log_dups_size":0,"ondisk_log_size":5540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:30:30.228060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30654,"num_write_kb":393416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3e","version":"74'5058","reported_seq":5180,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.759205+0000","last_change":"2024-04-25T00:01:19.515101+0000","last_active":"2024-04-25T00:30:26.759205+0000","last_peered":"2024-04-25T00:30:26.759205+0000","last_clean":"2024-04-25T00:30:26.759205+0000","last_became_active":"2024-04-25T00:01:19.513314+0000","last_became_peered":"2024-04-25T00:01:19.513314+0000","last_unstale":"2024-04-25T00:30:26.759205+0000","last_undegraded":"2024-04-25T00:30:26.759205+0000","last_fullsized":"2024-04-25T00:30:26.759205+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":5058,"log_dups_size":0,"ondisk_log_size":5058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:48:50.351599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":15,"num_write":27439,"num_write_kb":359791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.38","version":"70'4026","reported_seq":4151,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.793659+0000","last_change":"2024-04-25T00:01:05.390256+0000","last_active":"2024-04-25T00:30:15.793659+0000","last_peered":"2024-04-25T00:30:15.793659+0000","last_clean":"2024-04-25T00:30:15.793659+0000","last_became_active":"2024-04-25T00:01:05.389724+0000","last_became_peered":"2024-04-25T00:01:05.389724+0000","last_unstale":"2024-04-25T00:30:15.793659+0000","last_undegraded":"2024-04-25T00:30:15.793659+0000","last_fullsized":"2024-04-25T00:30:15.793659+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4026,"log_dups_size":0,"ondisk_log_size":4026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:09:18.315688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21352,"num_write_kb":290527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3f","version":"74'4365","reported_seq":4489,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.773739+0000","last_change":"2024-04-25T00:01:19.511493+0000","last_active":"2024-04-25T00:30:26.773739+0000","last_peered":"2024-04-25T00:30:26.773739+0000","last_clean":"2024-04-25T00:30:26.773739+0000","last_became_active":"2024-04-25T00:01:19.508049+0000","last_became_peered":"2024-04-25T00:01:19.508049+0000","last_unstale":"2024-04-25T00:30:26.773739+0000","last_undegraded":"2024-04-25T00:30:26.773739+0000","last_fullsized":"2024-04-25T00:30:26.773739+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4365,"log_dups_size":0,"ondisk_log_size":4365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:55:46.853565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":8,"num_write":24611,"num_write_kb":307761,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"70'3970","reported_seq":4095,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.790756+0000","last_change":"2024-04-25T00:01:05.374766+0000","last_active":"2024-04-25T00:30:15.790756+0000","last_peered":"2024-04-25T00:30:15.790756+0000","last_clean":"2024-04-25T00:30:15.790756+0000","last_became_active":"2024-04-25T00:01:05.369739+0000","last_became_peered":"2024-04-25T00:01:05.369739+0000","last_unstale":"2024-04-25T00:30:15.790756+0000","last_undegraded":"2024-04-25T00:30:15.790756+0000","last_fullsized":"2024-04-25T00:30:15.790756+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3970,"log_dups_size":0,"ondisk_log_size":3970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:09:18.315688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22008,"num_write_kb":289131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3c","version":"74'3966","reported_seq":4088,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.674125+0000","last_change":"2024-04-25T00:01:19.521456+0000","last_active":"2024-04-25T00:30:26.674125+0000","last_peered":"2024-04-25T00:30:26.674125+0000","last_clean":"2024-04-25T00:30:26.674125+0000","last_became_active":"2024-04-25T00:01:19.520787+0000","last_became_peered":"2024-04-25T00:01:19.520787+0000","last_unstale":"2024-04-25T00:30:26.674125+0000","last_undegraded":"2024-04-25T00:30:26.674125+0000","last_fullsized":"2024-04-25T00:30:26.674125+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3966,"log_dups_size":0,"ondisk_log_size":3966,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22683,"num_write_kb":271349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3a","version":"70'3294","reported_seq":3419,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.805979+0000","last_change":"2024-04-25T00:01:05.359253+0000","last_active":"2024-04-25T00:30:15.805979+0000","last_peered":"2024-04-25T00:30:15.805979+0000","last_clean":"2024-04-25T00:30:15.805979+0000","last_became_active":"2024-04-25T00:01:05.358477+0000","last_became_peered":"2024-04-25T00:01:05.358477+0000","last_unstale":"2024-04-25T00:30:15.805979+0000","last_undegraded":"2024-04-25T00:30:15.805979+0000","last_fullsized":"2024-04-25T00:30:15.805979+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3294,"log_dups_size":0,"ondisk_log_size":3294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:32:22.739214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17420,"num_write_kb":225566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3d","version":"74'5153","reported_seq":5274,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.749265+0000","last_change":"2024-04-25T00:01:19.509505+0000","last_active":"2024-04-25T00:30:26.749265+0000","last_peered":"2024-04-25T00:30:26.749265+0000","last_clean":"2024-04-25T00:30:26.749265+0000","last_became_active":"2024-04-25T00:01:19.507812+0000","last_became_peered":"2024-04-25T00:01:19.507812+0000","last_unstale":"2024-04-25T00:30:26.749265+0000","last_undegraded":"2024-04-25T00:30:26.749265+0000","last_fullsized":"2024-04-25T00:30:26.749265+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":5153,"log_dups_size":0,"ondisk_log_size":5153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:23.993729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":28174,"num_write_kb":326190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3b","version":"70'3663","reported_seq":3788,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.792919+0000","last_change":"2024-04-25T00:01:05.386068+0000","last_active":"2024-04-25T00:30:15.792919+0000","last_peered":"2024-04-25T00:30:15.792919+0000","last_clean":"2024-04-25T00:30:15.792919+0000","last_became_active":"2024-04-25T00:01:05.378650+0000","last_became_peered":"2024-04-25T00:01:05.378650+0000","last_unstale":"2024-04-25T00:30:15.792919+0000","last_undegraded":"2024-04-25T00:30:15.792919+0000","last_fullsized":"2024-04-25T00:30:15.792919+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3663,"log_dups_size":0,"ondisk_log_size":3663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:12:52.846250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18938,"num_write_kb":284951,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3a","version":"74'3833","reported_seq":3950,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.611583+0000","last_change":"2024-04-25T00:01:19.514843+0000","last_active":"2024-04-25T00:30:26.611583+0000","last_peered":"2024-04-25T00:30:26.611583+0000","last_clean":"2024-04-25T00:30:26.611583+0000","last_became_active":"2024-04-25T00:01:19.513956+0000","last_became_peered":"2024-04-25T00:01:19.513956+0000","last_unstale":"2024-04-25T00:30:26.611583+0000","last_undegraded":"2024-04-25T00:30:26.611583+0000","last_fullsized":"2024-04-25T00:30:26.611583+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3833,"log_dups_size":0,"ondisk_log_size":3833,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:14:38.005233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20679,"num_write_kb":271382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"69'3516","reported_seq":3645,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.808655+0000","last_change":"2024-04-25T00:01:05.359716+0000","last_active":"2024-04-25T00:30:15.808655+0000","last_peered":"2024-04-25T00:30:15.808655+0000","last_clean":"2024-04-25T00:30:15.808655+0000","last_became_active":"2024-04-25T00:01:05.358866+0000","last_became_peered":"2024-04-25T00:01:05.358866+0000","last_unstale":"2024-04-25T00:30:15.808655+0000","last_undegraded":"2024-04-25T00:30:15.808655+0000","last_fullsized":"2024-04-25T00:30:15.808655+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3516,"log_dups_size":0,"ondisk_log_size":3516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:45:50.489125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":19226,"num_write_kb":253281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3b","version":"74'4130","reported_seq":4248,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.756223+0000","last_change":"2024-04-25T00:01:19.499574+0000","last_active":"2024-04-25T00:30:26.756223+0000","last_peered":"2024-04-25T00:30:26.756223+0000","last_clean":"2024-04-25T00:30:26.756223+0000","last_became_active":"2024-04-25T00:01:19.498699+0000","last_became_peered":"2024-04-25T00:01:19.498699+0000","last_unstale":"2024-04-25T00:30:26.756223+0000","last_undegraded":"2024-04-25T00:30:26.756223+0000","last_fullsized":"2024-04-25T00:30:26.756223+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4130,"log_dups_size":0,"ondisk_log_size":4130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:19:47.859626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":3,"num_write":21749,"num_write_kb":288055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"70'3483","reported_seq":3611,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.798639+0000","last_change":"2024-04-25T00:01:05.371291+0000","last_active":"2024-04-25T00:30:15.798639+0000","last_peered":"2024-04-25T00:30:15.798639+0000","last_clean":"2024-04-25T00:30:15.798639+0000","last_became_active":"2024-04-25T00:01:05.370522+0000","last_became_peered":"2024-04-25T00:01:05.370522+0000","last_unstale":"2024-04-25T00:30:15.798639+0000","last_undegraded":"2024-04-25T00:30:15.798639+0000","last_fullsized":"2024-04-25T00:30:15.798639+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3483,"log_dups_size":0,"ondisk_log_size":3483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:11:27.044893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18326,"num_write_kb":261938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.38","version":"74'4120","reported_seq":4237,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.292105+0000","last_change":"2024-04-25T00:01:19.523798+0000","last_active":"2024-04-25T00:30:26.292105+0000","last_peered":"2024-04-25T00:30:26.292105+0000","last_clean":"2024-04-25T00:30:26.292105+0000","last_became_active":"2024-04-25T00:01:19.523450+0000","last_became_peered":"2024-04-25T00:01:19.523450+0000","last_unstale":"2024-04-25T00:30:26.292105+0000","last_undegraded":"2024-04-25T00:30:26.292105+0000","last_fullsized":"2024-04-25T00:30:26.292105+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4120,"log_dups_size":0,"ondisk_log_size":4120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:40:36.350832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22879,"num_write_kb":288519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3e","version":"70'4991","reported_seq":5119,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.801414+0000","last_change":"2024-04-25T00:01:05.368190+0000","last_active":"2024-04-25T00:30:15.801414+0000","last_peered":"2024-04-25T00:30:15.801414+0000","last_clean":"2024-04-25T00:30:15.801414+0000","last_became_active":"2024-04-25T00:01:05.362606+0000","last_became_peered":"2024-04-25T00:01:05.362606+0000","last_unstale":"2024-04-25T00:30:15.801414+0000","last_undegraded":"2024-04-25T00:30:15.801414+0000","last_fullsized":"2024-04-25T00:30:15.801414+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4991,"log_dups_size":0,"ondisk_log_size":4991,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:12:52.846250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26684,"num_write_kb":361051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.39","version":"74'4850","reported_seq":4967,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.763325+0000","last_change":"2024-04-25T00:01:19.505737+0000","last_active":"2024-04-25T00:30:26.763325+0000","last_peered":"2024-04-25T00:30:26.763325+0000","last_clean":"2024-04-25T00:30:26.763325+0000","last_became_active":"2024-04-25T00:01:19.504991+0000","last_became_peered":"2024-04-25T00:01:19.504991+0000","last_unstale":"2024-04-25T00:30:26.763325+0000","last_undegraded":"2024-04-25T00:30:26.763325+0000","last_fullsized":"2024-04-25T00:30:26.763325+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4850,"log_dups_size":0,"ondisk_log_size":4850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:47:19.526845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26612,"num_write_kb":323915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3f","version":"70'3815","reported_seq":3943,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.791882+0000","last_change":"2024-04-25T00:01:05.389596+0000","last_active":"2024-04-25T00:30:15.791882+0000","last_peered":"2024-04-25T00:30:15.791882+0000","last_clean":"2024-04-25T00:30:15.791882+0000","last_became_active":"2024-04-25T00:01:05.389127+0000","last_became_peered":"2024-04-25T00:01:05.389127+0000","last_unstale":"2024-04-25T00:30:15.791882+0000","last_undegraded":"2024-04-25T00:30:15.791882+0000","last_fullsized":"2024-04-25T00:30:15.791882+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3815,"log_dups_size":0,"ondisk_log_size":3815,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:09.858617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20941,"num_write_kb":280561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":105,"num_read_kb":408,"num_write":1543888,"num_write_kb":19306803,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":280555,"ondisk_log_size":280555,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":19555305,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1733,"num_read_kb":229197,"num_write":5131,"num_write_kb":21279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":39174144,"data_stored":39110610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4707,"ondisk_log_size":4707,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":3,"num_write":1412500,"num_write_kb":18776370,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":261221,"ondisk_log_size":261221,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":19117950,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1590,"num_read_kb":257618,"num_write":5060,"num_write_kb":20770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":38297600,"data_stored":38235900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4699,"ondisk_log_size":4699,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509802,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":674532,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":673470,"kb_avail":93697308,"statfs":{"total":96636764160,"available":95946043392,"internally_reserved":0,"allocated":1077248,"data_stored":617327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":689633739},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509802,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":843472,"kb_used_data":11784,"kb_used_omap":1,"kb_used_meta":831678,"kb_avail":93528368,"statfs":{"total":96636764160,"available":95773048832,"internally_reserved":0,"allocated":12066816,"data_stored":11609748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":851638726},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509801,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":870568,"kb_used_data":18016,"kb_used_omap":1,"kb_used_meta":852542,"kb_avail":93501272,"statfs":{"total":96636764160,"available":95745302528,"internally_reserved":0,"allocated":18448384,"data_stored":17985439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":873003461},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509802,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":922240,"kb_used_data":15544,"kb_used_omap":1,"kb_used_meta":906686,"kb_avail":93449600,"statfs":{"total":96636764160,"available":95692390400,"internally_reserved":0,"allocated":15917056,"data_stored":15447989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":928446919},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509803,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1076840,"kb_used_data":15968,"kb_used_omap":1,"kb_used_meta":1060862,"kb_avail":93295000,"statfs":{"total":96636764160,"available":95534080000,"internally_reserved":0,"allocated":16351232,"data_stored":15888732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1086323141},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509802,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":788556,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":787006,"kb_avail":93583284,"statfs":{"total":96636764160,"available":95829282816,"internally_reserved":0,"allocated":1576960,"data_stored":1119003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":805894597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509802,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1028004,"kb_used_data":3164,"kb_used_omap":1,"kb_used_meta":1024830,"kb_avail":93343836,"statfs":{"total":96636764160,"available":95584088064,"internally_reserved":0,"allocated":3239936,"data_stored":2779219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1049426373},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575211,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":962780,"kb_used_data":17364,"kb_used_omap":1,"kb_used_meta":945406,"kb_avail":93409060,"statfs":{"total":96636764160,"available":95650877440,"internally_reserved":0,"allocated":17780736,"data_stored":17307749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":968096197},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2260992,"data_stored":2250634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8396800,"data_stored":8389280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":10645504,"data_stored":10639768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8404992,"data_stored":8389952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4292608,"data_stored":4283082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":88810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6979584,"data_stored":6970957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4296704,"data_stored":4279726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8400896,"data_stored":8389302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12587008,"data_stored":12583494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":6897664,"data_stored":6886949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T00:30:47.563 INFO:tasks.ceph:pgid 5.18 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.563 INFO:tasks.ceph:pgid 2.1f last_scrub_stamp 2024-04-25T00:01:03.305620+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=3, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.563 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.563 INFO:tasks.ceph:pgid 4.19 last_scrub_stamp 2024-04-25T00:01:17.440296+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.563 INFO:tasks.ceph:pgid 5.19 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.563 INFO:tasks.ceph:pgid 2.1e last_scrub_stamp 2024-04-25T00:01:03.305620+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=3, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.563 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.563 INFO:tasks.ceph:pgid 4.18 last_scrub_stamp 2024-04-25T00:01:17.440296+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.564 INFO:tasks.ceph:pgid 5.1a last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.564 INFO:tasks.ceph:pgid 2.1d last_scrub_stamp 2024-04-25T00:01:03.305620+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=3, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.564 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.564 INFO:tasks.ceph:pgid 4.1b last_scrub_stamp 2024-04-25T00:01:17.440296+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.564 INFO:tasks.ceph:pgid 5.1b last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.564 INFO:tasks.ceph:pgid 2.1c last_scrub_stamp 2024-04-25T00:01:03.305620+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=3, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.564 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.564 INFO:tasks.ceph:pgid 4.1a last_scrub_stamp 2024-04-25T00:01:17.440296+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.564 INFO:tasks.ceph:pgid 5.c last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.564 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.565 INFO:tasks.ceph:pgid 2.b last_scrub_stamp 2024-04-25T00:01:03.305620+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=3, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.565 INFO:tasks.ceph:pgid 4.d last_scrub_stamp 2024-04-25T00:01:17.440296+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.565 INFO:tasks.ceph:pgid 5.d last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.565 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.565 INFO:tasks.ceph:pgid 2.a last_scrub_stamp 2024-04-25T00:01:03.305620+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=3, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.565 INFO:tasks.ceph:pgid 4.c last_scrub_stamp 2024-04-25T00:01:17.440296+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.565 INFO:tasks.ceph:pgid 5.e last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.565 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.565 INFO:tasks.ceph:pgid 2.9 last_scrub_stamp 2024-04-25T00:01:03.305620+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=3, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.566 INFO:tasks.ceph:pgid 4.f last_scrub_stamp 2024-04-25T00:01:17.440296+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.566 INFO:tasks.ceph:pgid 5.f last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.566 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.566 INFO:tasks.ceph:pgid 2.8 last_scrub_stamp 2024-04-25T00:01:03.305620+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=3, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.566 INFO:tasks.ceph:pgid 4.e last_scrub_stamp 2024-04-25T00:01:17.440296+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.566 INFO:tasks.ceph:pgid 5.0 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.566 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.566 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-04-25T00:01:03.305620+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=3, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.566 INFO:tasks.ceph:pgid 4.1 last_scrub_stamp 2024-04-25T00:01:17.440296+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.567 INFO:tasks.ceph:pgid 5.1 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.567 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.567 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-04-25T00:01:03.305620+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=3, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.567 INFO:tasks.ceph:pgid 4.0 last_scrub_stamp 2024-04-25T00:01:17.440296+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.567 INFO:tasks.ceph:pgid 5.2 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.567 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.567 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-04-25T00:01:03.305620+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=3, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.567 INFO:tasks.ceph:pgid 4.3 last_scrub_stamp 2024-04-25T00:01:17.440296+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.567 INFO:tasks.ceph:pgid 5.3 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.568 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.568 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-04-25T00:01:03.305620+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=3, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.568 INFO:tasks.ceph:pgid 4.2 last_scrub_stamp 2024-04-25T00:01:17.440296+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.568 INFO:tasks.ceph:pgid 5.5 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.568 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.568 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-04-25T00:01:03.305620+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=3, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.568 INFO:tasks.ceph:pgid 4.4 last_scrub_stamp 2024-04-25T00:01:17.440296+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.568 INFO:tasks.ceph:pgid 5.6 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.568 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.569 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-04-25T00:01:03.305620+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=3, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.569 INFO:tasks.ceph:pgid 4.7 last_scrub_stamp 2024-04-25T00:01:17.440296+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.569 INFO:tasks.ceph:pgid 5.7 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.569 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.569 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-04-25T00:01:03.305620+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=3, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.569 INFO:tasks.ceph:pgid 4.6 last_scrub_stamp 2024-04-25T00:01:17.440296+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.569 INFO:tasks.ceph:pgid 5.4 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.569 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.569 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-04-25T00:01:03.305620+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=3, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.570 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-25T00:00:54.060578+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=0, tm_sec=54, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.570 INFO:tasks.ceph:pgid 4.5 last_scrub_stamp 2024-04-25T00:01:17.440296+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.570 INFO:tasks.ceph:pgid 5.b last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.570 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.570 INFO:tasks.ceph:pgid 2.c last_scrub_stamp 2024-04-25T00:01:03.305620+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=3, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.570 INFO:tasks.ceph:pgid 4.a last_scrub_stamp 2024-04-25T00:01:17.440296+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.570 INFO:tasks.ceph:pgid 5.a last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.570 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.571 INFO:tasks.ceph:pgid 2.d last_scrub_stamp 2024-04-25T00:01:03.305620+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=3, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.571 INFO:tasks.ceph:pgid 4.b last_scrub_stamp 2024-04-25T00:01:17.440296+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.571 INFO:tasks.ceph:pgid 5.9 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.571 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.571 INFO:tasks.ceph:pgid 2.e last_scrub_stamp 2024-04-25T00:01:03.305620+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=3, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.571 INFO:tasks.ceph:pgid 4.8 last_scrub_stamp 2024-04-25T00:01:17.440296+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.571 INFO:tasks.ceph:pgid 5.8 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.571 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.571 INFO:tasks.ceph:pgid 2.f last_scrub_stamp 2024-04-25T00:01:03.305620+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=3, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.572 INFO:tasks.ceph:pgid 4.9 last_scrub_stamp 2024-04-25T00:01:17.440296+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.572 INFO:tasks.ceph:pgid 5.17 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.572 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.572 INFO:tasks.ceph:pgid 2.10 last_scrub_stamp 2024-04-25T00:01:03.305620+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=3, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.572 INFO:tasks.ceph:pgid 4.16 last_scrub_stamp 2024-04-25T00:01:17.440296+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.572 INFO:tasks.ceph:pgid 5.16 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.572 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.572 INFO:tasks.ceph:pgid 2.11 last_scrub_stamp 2024-04-25T00:01:03.305620+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=3, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.572 INFO:tasks.ceph:pgid 4.17 last_scrub_stamp 2024-04-25T00:01:17.440296+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.573 INFO:tasks.ceph:pgid 5.15 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.573 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.573 INFO:tasks.ceph:pgid 2.12 last_scrub_stamp 2024-04-25T00:01:03.305620+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=3, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.573 INFO:tasks.ceph:pgid 4.14 last_scrub_stamp 2024-04-25T00:01:17.440296+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.573 INFO:tasks.ceph:pgid 5.14 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.573 INFO:tasks.ceph:pgid 3.12 last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.573 INFO:tasks.ceph:pgid 2.13 last_scrub_stamp 2024-04-25T00:01:03.305620+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=3, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.573 INFO:tasks.ceph:pgid 4.15 last_scrub_stamp 2024-04-25T00:01:17.440296+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.573 INFO:tasks.ceph:pgid 5.13 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.574 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.574 INFO:tasks.ceph:pgid 2.14 last_scrub_stamp 2024-04-25T00:01:03.305620+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=3, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.574 INFO:tasks.ceph:pgid 4.12 last_scrub_stamp 2024-04-25T00:01:17.440296+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.574 INFO:tasks.ceph:pgid 5.12 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.574 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.574 INFO:tasks.ceph:pgid 2.15 last_scrub_stamp 2024-04-25T00:01:03.305620+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=3, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.574 INFO:tasks.ceph:pgid 4.13 last_scrub_stamp 2024-04-25T00:01:17.440296+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.575 INFO:tasks.ceph:pgid 5.11 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.575 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.575 INFO:tasks.ceph:pgid 2.16 last_scrub_stamp 2024-04-25T00:01:03.305620+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=3, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.575 INFO:tasks.ceph:pgid 4.10 last_scrub_stamp 2024-04-25T00:01:17.440296+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.575 INFO:tasks.ceph:pgid 5.10 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.575 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.575 INFO:tasks.ceph:pgid 2.17 last_scrub_stamp 2024-04-25T00:01:03.305620+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=3, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.575 INFO:tasks.ceph:pgid 4.11 last_scrub_stamp 2024-04-25T00:01:17.440296+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.576 INFO:tasks.ceph:pgid 5.1f last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.576 INFO:tasks.ceph:pgid 4.1e last_scrub_stamp 2024-04-25T00:01:17.440296+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.576 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.576 INFO:tasks.ceph:pgid 2.18 last_scrub_stamp 2024-04-25T00:01:03.305620+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=3, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.576 INFO:tasks.ceph:pgid 5.1e last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.576 INFO:tasks.ceph:pgid 4.1f last_scrub_stamp 2024-04-25T00:01:17.440296+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.576 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.576 INFO:tasks.ceph:pgid 2.19 last_scrub_stamp 2024-04-25T00:01:03.305620+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=3, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.576 INFO:tasks.ceph:pgid 5.1d last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.576 INFO:tasks.ceph:pgid 2.1a last_scrub_stamp 2024-04-25T00:01:03.305620+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=3, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.577 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.577 INFO:tasks.ceph:pgid 4.1c last_scrub_stamp 2024-04-25T00:01:17.440296+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.577 INFO:tasks.ceph:pgid 5.1c last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.577 INFO:tasks.ceph:pgid 2.1b last_scrub_stamp 2024-04-25T00:01:03.305620+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=3, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.577 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.577 INFO:tasks.ceph:pgid 4.1d last_scrub_stamp 2024-04-25T00:01:17.440296+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.577 INFO:tasks.ceph:pgid 5.26 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.577 INFO:tasks.ceph:pgid 3.20 last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.577 INFO:tasks.ceph:pgid 5.27 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.578 INFO:tasks.ceph:pgid 3.21 last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.578 INFO:tasks.ceph:pgid 5.24 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.578 INFO:tasks.ceph:pgid 3.22 last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.578 INFO:tasks.ceph:pgid 5.25 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.578 INFO:tasks.ceph:pgid 3.23 last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.578 INFO:tasks.ceph:pgid 5.22 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.578 INFO:tasks.ceph:pgid 3.24 last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.578 INFO:tasks.ceph:pgid 5.23 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.578 INFO:tasks.ceph:pgid 3.25 last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.579 INFO:tasks.ceph:pgid 5.20 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.579 INFO:tasks.ceph:pgid 3.26 last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.579 INFO:tasks.ceph:pgid 5.21 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.579 INFO:tasks.ceph:pgid 3.27 last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.579 INFO:tasks.ceph:pgid 5.2e last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.579 INFO:tasks.ceph:pgid 3.28 last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.579 INFO:tasks.ceph:pgid 5.2f last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.579 INFO:tasks.ceph:pgid 3.29 last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.579 INFO:tasks.ceph:pgid 5.2c last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.580 INFO:tasks.ceph:pgid 3.2a last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.580 INFO:tasks.ceph:pgid 5.2d last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.580 INFO:tasks.ceph:pgid 3.2b last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.580 INFO:tasks.ceph:pgid 5.2a last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.580 INFO:tasks.ceph:pgid 3.2c last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.580 INFO:tasks.ceph:pgid 5.2b last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.580 INFO:tasks.ceph:pgid 3.2d last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.580 INFO:tasks.ceph:pgid 5.28 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.580 INFO:tasks.ceph:pgid 3.2e last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.580 INFO:tasks.ceph:pgid 5.29 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.581 INFO:tasks.ceph:pgid 3.2f last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.581 INFO:tasks.ceph:pgid 5.36 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.581 INFO:tasks.ceph:pgid 3.30 last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.581 INFO:tasks.ceph:pgid 5.37 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.581 INFO:tasks.ceph:pgid 3.31 last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.581 INFO:tasks.ceph:pgid 5.34 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.581 INFO:tasks.ceph:pgid 3.32 last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.581 INFO:tasks.ceph:pgid 5.35 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.581 INFO:tasks.ceph:pgid 3.33 last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.581 INFO:tasks.ceph:pgid 5.32 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.582 INFO:tasks.ceph:pgid 3.34 last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.582 INFO:tasks.ceph:pgid 5.33 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.582 INFO:tasks.ceph:pgid 3.35 last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.582 INFO:tasks.ceph:pgid 5.30 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.582 INFO:tasks.ceph:pgid 3.36 last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.582 INFO:tasks.ceph:pgid 5.31 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.582 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.582 INFO:tasks.ceph:pgid 5.3e last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.583 INFO:tasks.ceph:pgid 3.38 last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.583 INFO:tasks.ceph:pgid 5.3f last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.583 INFO:tasks.ceph:pgid 3.39 last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.583 INFO:tasks.ceph:pgid 5.3c last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.583 INFO:tasks.ceph:pgid 3.3a last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.583 INFO:tasks.ceph:pgid 5.3d last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.583 INFO:tasks.ceph:pgid 3.3b last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.583 INFO:tasks.ceph:pgid 5.3a last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.583 INFO:tasks.ceph:pgid 3.3c last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.583 INFO:tasks.ceph:pgid 5.3b last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.584 INFO:tasks.ceph:pgid 3.3d last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.584 INFO:tasks.ceph:pgid 5.38 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.584 INFO:tasks.ceph:pgid 3.3e last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.584 INFO:tasks.ceph:pgid 5.39 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.584 INFO:tasks.ceph:pgid 3.3f last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:30:47.584 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-25T00:31:07.585 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T00:31:07.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.714+0000 7f1746120640 1 Processor -- start 2024-04-25T00:31:07.714 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.714+0000 7f1746120640 1 -- start start 2024-04-25T00:31:07.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.714+0000 7f1746120640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f17400c4300 0x7f17400aa170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:07.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.714+0000 7f1746120640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f17400c5520 0x7f1740156200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:07.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.714+0000 7f1746120640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17401526a0 con 0x7f17400c5100 2024-04-25T00:31:07.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.714+0000 7f1746120640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f1740153460 con 0x7f17400c5520 2024-04-25T00:31:07.715 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.714+0000 7f1746120640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f174006ffd0 con 0x7f17400c4300 2024-04-25T00:31:07.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.714+0000 7f173effd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f17400c5100 0x7f1740155af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:43340/0 (socket says 172.21.15.116:43340) 2024-04-25T00:31:07.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.714+0000 7f173effd640 1 -- 172.21.15.116:0/2345958322 learned_addr learned my addr 172.21.15.116:0/2345958322 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:31:07.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.714+0000 7f173ffff640 1 --2- 172.21.15.116:0/2345958322 >> v2:172.21.15.174:3300/0 conn(0x7f17400c5520 0x7f1740156200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:07.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.714+0000 7f173e7fc640 1 -- 172.21.15.116:0/2345958322 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 872281167 0 0) 0x7f17401526a0 con 0x7f17400c5100 2024-04-25T00:31:07.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.714+0000 7f173e7fc640 1 -- 172.21.15.116:0/2345958322 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f172c003530 con 0x7f17400c5100 2024-04-25T00:31:07.716 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.714+0000 7f173f7fe640 1 --2- 172.21.15.116:0/2345958322 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f17400c4300 0x7f17400aa170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:07.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.714+0000 7f173e7fc640 1 -- 172.21.15.116:0/2345958322 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 615291833 0 0) 0x7f172c003530 con 0x7f17400c5100 2024-04-25T00:31:07.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.714+0000 7f173e7fc640 1 -- 172.21.15.116:0/2345958322 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f17400c4300 msgr2=0x7f17400aa170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:31:07.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.714+0000 7f173e7fc640 1 --2- 172.21.15.116:0/2345958322 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f17400c4300 0x7f17400aa170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:07.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.714+0000 7f173e7fc640 1 -- 172.21.15.116:0/2345958322 >> v2:172.21.15.174:3300/0 conn(0x7f17400c5520 msgr2=0x7f1740156200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:31:07.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.714+0000 7f173e7fc640 1 --2- 172.21.15.116:0/2345958322 >> v2:172.21.15.174:3300/0 conn(0x7f17400c5520 0x7f1740156200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:07.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.714+0000 7f173e7fc640 1 -- 172.21.15.116:0/2345958322 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1740152980 con 0x7f17400c5100 2024-04-25T00:31:07.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.714+0000 7f173e7fc640 1 -- 172.21.15.116:0/2345958322 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f1728001680 con 0x7f17400c5100 2024-04-25T00:31:07.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.714+0000 7f173e7fc640 1 -- 172.21.15.116:0/2345958322 >> v1:172.21.15.116:6789/0 conn(0x7f17400c5100 legacy=0x7f1740155af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:07.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.714+0000 7f173e7fc640 1 --2- 172.21.15.116:0/2345958322 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f17400c4300 0x7f172c0039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:07.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.714+0000 7f173e7fc640 1 --2- 172.21.15.116:0/2345958322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f172c003f40 0x7f172c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:07.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.714+0000 7f173e7fc640 1 --2- 172.21.15.116:0/2345958322 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f172c004d50 0x7f172c0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:07.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.714+0000 7f173e7fc640 1 -- 172.21.15.116:0/2345958322 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1740153460 con 0x7f172c003f40 2024-04-25T00:31:07.718 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.714+0000 7f173e7fc640 1 -- 172.21.15.116:0/2345958322 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f174006ffd0 con 0x7f172c004d50 2024-04-25T00:31:07.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.714+0000 7f173e7fc640 1 -- 172.21.15.116:0/2345958322 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f172c00b9d0 con 0x7f17400c4300 2024-04-25T00:31:07.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.714+0000 7f173effd640 1 --2- 172.21.15.116:0/2345958322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f172c003f40 0x7f172c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:07.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.714+0000 7f173f7fe640 1 --2- 172.21.15.116:0/2345958322 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f17400c4300 0x7f172c0039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:07.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.714+0000 7f173ffff640 1 --2- 172.21.15.116:0/2345958322 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f172c004d50 0x7f172c0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:07.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.714+0000 7f173effd640 1 -- 172.21.15.116:0/2345958322 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f17400c4300 msgr2=0x7f172c0039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:31:07.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.714+0000 7f173effd640 1 --2- 172.21.15.116:0/2345958322 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f17400c4300 0x7f172c0039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:07.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.714+0000 7f173effd640 1 -- 172.21.15.116:0/2345958322 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f172c004d50 msgr2=0x7f172c0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:31:07.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.714+0000 7f173effd640 1 --2- 172.21.15.116:0/2345958322 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f172c004d50 0x7f172c0051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:07.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.714+0000 7f173effd640 1 -- 172.21.15.116:0/2345958322 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f172c003530 con 0x7f172c003f40 2024-04-25T00:31:07.719 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.714+0000 7f173ffff640 1 --2- 172.21.15.116:0/2345958322 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f172c004d50 0x7f172c0051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:31:07.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.718+0000 7f173effd640 1 --2- 172.21.15.116:0/2345958322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f172c003f40 0x7f172c004810 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f172800c240 tx=0x7f172801f3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a953f769e9e37d85 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:07.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.718+0000 7f173e7fc640 1 -- 172.21.15.116:0/2345958322 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1728007d40 con 0x7f172c003f40 2024-04-25T00:31:07.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.718+0000 7f173e7fc640 1 -- 172.21.15.116:0/2345958322 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1728005b00 con 0x7f172c003f40 2024-04-25T00:31:07.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.718+0000 7f173e7fc640 1 -- 172.21.15.116:0/2345958322 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1728007070 con 0x7f172c003f40 2024-04-25T00:31:07.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.718+0000 7f1746120640 1 -- 172.21.15.116:0/2345958322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f172c003f40 msgr2=0x7f172c004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:07.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.718+0000 7f1746120640 1 --2- 172.21.15.116:0/2345958322 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f172c003f40 0x7f172c004810 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f172800c240 tx=0x7f172801f3b0 comp rx=0 tx=0).stop 2024-04-25T00:31:07.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.718+0000 7f173ffff640 1 -- 172.21.15.116:0/2345958322 reap_dead start 2024-04-25T00:31:07.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.718+0000 7f1746120640 1 -- 172.21.15.116:0/2345958322 shutdown_connections 2024-04-25T00:31:07.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.718+0000 7f1746120640 1 -- 172.21.15.116:0/2345958322 >> 172.21.15.116:0/2345958322 conn(0x7f174006c720 msgr2=0x7f174006cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:31:07.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.718+0000 7f1746120640 1 -- 172.21.15.116:0/2345958322 shutdown_connections 2024-04-25T00:31:07.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.718+0000 7f1746120640 1 -- 172.21.15.116:0/2345958322 wait complete. 2024-04-25T00:31:07.721 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.718+0000 7f1746120640 1 Processor -- start 2024-04-25T00:31:07.722 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.718+0000 7f1746120640 1 -- start start 2024-04-25T00:31:07.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.722+0000 7f1746120640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f172c003f40 0x7f1740144ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:07.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.722+0000 7f1746120640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f174015fae0 0x7f1740148880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:07.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.722+0000 7f1746120640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1740148dc0 0x7f17400c1540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:07.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.722+0000 7f1746120640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f17400c4860 con 0x7f172c003f40 2024-04-25T00:31:07.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.722+0000 7f1746120640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f17400c46e0 con 0x7f1740148dc0 2024-04-25T00:31:07.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.722+0000 7f1746120640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f17400aacd0 con 0x7f174015fae0 2024-04-25T00:31:07.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.722+0000 7f173f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f172c003f40 0x7f1740144ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:07.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.722+0000 7f173f7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f172c003f40 0x7f1740144ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:50700/0 (socket says 172.21.15.116:50700) 2024-04-25T00:31:07.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.722+0000 7f173f7fe640 1 -- 172.21.15.116:0/514792345 learned_addr learned my addr 172.21.15.116:0/514792345 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:31:07.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.722+0000 7f173effd640 1 --2- 172.21.15.116:0/514792345 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f174015fae0 0x7f1740148880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:07.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.722+0000 7f173ffff640 1 --2- 172.21.15.116:0/514792345 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1740148dc0 0x7f17400c1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:07.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.722+0000 7f173f7fe640 1 -- 172.21.15.116:0/514792345 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f174015fae0 msgr2=0x7f1740148880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:07.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.722+0000 7f173f7fe640 1 --2- 172.21.15.116:0/514792345 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f174015fae0 0x7f1740148880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:07.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.722+0000 7f173f7fe640 1 -- 172.21.15.116:0/514792345 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1740148dc0 msgr2=0x7f17400c1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:07.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.722+0000 7f173f7fe640 1 --2- 172.21.15.116:0/514792345 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1740148dc0 0x7f17400c1540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:07.724 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.722+0000 7f173f7fe640 1 -- 172.21.15.116:0/514792345 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f174015ef70 con 0x7f172c003f40 2024-04-25T00:31:07.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.722+0000 7f173ffff640 1 --2- 172.21.15.116:0/514792345 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1740148dc0 0x7f17400c1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:31:07.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.722+0000 7f173f7fe640 1 --2- 172.21.15.116:0/514792345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f172c003f40 0x7f1740144ae0 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f1738007f60 tx=0x7f1738007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:07.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.722+0000 7f1737fff640 1 -- 172.21.15.116:0/514792345 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f173801a020 con 0x7f172c003f40 2024-04-25T00:31:07.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.722+0000 7f1737fff640 1 -- 172.21.15.116:0/514792345 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1738016470 con 0x7f172c003f40 2024-04-25T00:31:07.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.722+0000 7f1746120640 1 -- 172.21.15.116:0/514792345 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1740153680 con 0x7f172c003f40 2024-04-25T00:31:07.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.722+0000 7f1737fff640 1 -- 172.21.15.116:0/514792345 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1738016860 con 0x7f172c003f40 2024-04-25T00:31:07.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.722+0000 7f1746120640 1 -- 172.21.15.116:0/514792345 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17401eef30 con 0x7f172c003f40 2024-04-25T00:31:07.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.722+0000 7f1746120640 1 -- 172.21.15.116:0/514792345 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f174015fec0 con 0x7f172c003f40 2024-04-25T00:31:07.725 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.722+0000 7f1737fff640 1 -- 172.21.15.116:0/514792345 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f173801d020 con 0x7f172c003f40 2024-04-25T00:31:07.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.722+0000 7f1737fff640 1 --2- 172.21.15.116:0/514792345 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f17200678c0 0x7f1720069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:07.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.722+0000 7f1737fff640 1 -- 172.21.15.116:0/514792345 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7f1738087e30 con 0x7f172c003f40 2024-04-25T00:31:07.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.722+0000 7f173effd640 1 --2- 172.21.15.116:0/514792345 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f17200678c0 0x7f1720069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:07.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.726+0000 7f173effd640 1 --2- 172.21.15.116:0/514792345 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f17200678c0 0x7f1720069d80 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f17280ae000 tx=0x7f174006f100 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:07.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.726+0000 7f1737fff640 1 -- 172.21.15.116:0/514792345 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f174015fec0 con 0x7f172c003f40 2024-04-25T00:31:07.945 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.946+0000 7f1746120640 1 -- 172.21.15.116:0/514792345 --> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1740068a60 con 0x7f17200678c0 2024-04-25T00:31:07.991 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.990+0000 7f1737fff640 1 -- 172.21.15.116:0/514792345 <== mgr.4105 v2:172.21.15.174:6832/2004448211 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+492772 (secure 0 0 0) 0x7f1728137500 con 0x7f17200678c0 2024-04-25T00:31:07.997 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:31:08.000 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-04-25T00:31:08.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.994+0000 7f1746120640 1 -- 172.21.15.116:0/514792345 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f17200678c0 msgr2=0x7f1720069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:08.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.994+0000 7f1746120640 1 --2- 172.21.15.116:0/514792345 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f17200678c0 0x7f1720069d80 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f17280ae000 tx=0x7f174006f100 comp rx=0 tx=0).stop 2024-04-25T00:31:08.001 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.994+0000 7f1746120640 1 -- 172.21.15.116:0/514792345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f172c003f40 msgr2=0x7f1740144ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:08.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.994+0000 7f1746120640 1 --2- 172.21.15.116:0/514792345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f172c003f40 0x7f1740144ae0 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f1738007f60 tx=0x7f1738007f90 comp rx=0 tx=0).stop 2024-04-25T00:31:08.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.994+0000 7f1746120640 1 -- 172.21.15.116:0/514792345 shutdown_connections 2024-04-25T00:31:08.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.994+0000 7f1746120640 1 --2- 172.21.15.116:0/514792345 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f17200678c0 0x7f1720069d80 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:08.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.994+0000 7f1746120640 1 --2- 172.21.15.116:0/514792345 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1740148dc0 0x7f17400c1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:08.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.994+0000 7f1746120640 1 --2- 172.21.15.116:0/514792345 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f174015fae0 0x7f1740148880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:08.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.994+0000 7f1746120640 1 --2- 172.21.15.116:0/514792345 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f172c003f40 0x7f1740144ae0 unknown :-1 s=CLOSED pgs=1849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:08.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.994+0000 7f1746120640 1 -- 172.21.15.116:0/514792345 >> 172.21.15.116:0/514792345 conn(0x7f174006c720 msgr2=0x7f17400ad160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:31:08.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.994+0000 7f1746120640 1 -- 172.21.15.116:0/514792345 shutdown_connections 2024-04-25T00:31:08.002 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:07.994+0000 7f1746120640 1 -- 172.21.15.116:0/514792345 wait complete. 2024-04-25T00:31:08.014 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":975,"stamp":"2024-04-25T00:31:06.039648+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":39263623,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":3556,"num_read_kb":487326,"num_write":2966738,"num_write_kb":38127638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10000,"num_omap_keys":20,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":551344,"ondisk_log_size":551344,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":7166992,"kb_used_data":84432,"kb_used_omap":12,"kb_used_meta":7082483,"kb_avail":747807728,"statfs":{"total":773094113280,"available":765755113472,"internally_reserved":0,"allocated":86458368,"data_stored":82755206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":7252463153},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006035"},"pg_stats":[{"pgid":"5.18","version":"74'4491","reported_seq":4621,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:54.129321+0000","last_change":"2024-04-25T00:30:54.129382+0000","last_active":"2024-04-25T00:30:54.129321+0000","last_peered":"2024-04-25T00:30:54.129321+0000","last_clean":"2024-04-25T00:30:54.129321+0000","last_became_active":"2024-04-25T00:01:19.508390+0000","last_became_peered":"2024-04-25T00:01:19.508390+0000","last_unstale":"2024-04-25T00:30:54.129321+0000","last_undegraded":"2024-04-25T00:30:54.129321+0000","last_fullsized":"2024-04-25T00:30:54.129321+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4491","last_scrub_stamp":"2024-04-25T00:30:54.129109+0000","last_deep_scrub":"74'4491","last_deep_scrub_stamp":"2024-04-25T00:30:54.129109+0000","last_clean_scrub_stamp":"2024-04-25T00:30:54.129109+0000","objects_scrubbed":0,"log_size":4491,"log_dups_size":0,"ondisk_log_size":4491,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:47:08.927771+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":12,"num_write":23917,"num_write_kb":321110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1f","version":"74'1090","reported_seq":1285,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:56.942373+0000","last_change":"2024-04-25T00:30:56.942455+0000","last_active":"2024-04-25T00:30:56.942373+0000","last_peered":"2024-04-25T00:30:56.942373+0000","last_clean":"2024-04-25T00:30:56.942373+0000","last_became_active":"2024-04-25T00:01:04.355293+0000","last_became_peered":"2024-04-25T00:01:04.355293+0000","last_unstale":"2024-04-25T00:30:56.942373+0000","last_undegraded":"2024-04-25T00:30:56.942373+0000","last_fullsized":"2024-04-25T00:30:56.942373+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1090","last_scrub_stamp":"2024-04-25T00:30:56.942114+0000","last_deep_scrub":"74'1090","last_deep_scrub_stamp":"2024-04-25T00:30:56.942114+0000","last_clean_scrub_stamp":"2024-04-25T00:30:56.942114+0000","objects_scrubbed":3,"log_size":1090,"log_dups_size":0,"ondisk_log_size":1090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:36:32.511817+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":16397,"num_write":1061,"num_write_kb":4715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"70'3760","reported_seq":3897,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:57.662591+0000","last_change":"2024-04-25T00:30:57.662705+0000","last_active":"2024-04-25T00:30:57.662591+0000","last_peered":"2024-04-25T00:30:57.662591+0000","last_clean":"2024-04-25T00:30:57.662591+0000","last_became_active":"2024-04-25T00:01:05.366471+0000","last_became_peered":"2024-04-25T00:01:05.366471+0000","last_unstale":"2024-04-25T00:30:57.662591+0000","last_undegraded":"2024-04-25T00:30:57.662591+0000","last_fullsized":"2024-04-25T00:30:57.662591+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'3760","last_scrub_stamp":"2024-04-25T00:30:57.662289+0000","last_deep_scrub":"70'3760","last_deep_scrub_stamp":"2024-04-25T00:30:57.662289+0000","last_clean_scrub_stamp":"2024-04-25T00:30:57.662289+0000","objects_scrubbed":0,"log_size":3760,"log_dups_size":0,"ondisk_log_size":3760,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:02:40.045444+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21072,"num_write_kb":264901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"74'3","reported_seq":132,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:47.917551+0000","last_change":"2024-04-25T00:30:47.917609+0000","last_active":"2024-04-25T00:30:47.917551+0000","last_peered":"2024-04-25T00:30:47.917551+0000","last_clean":"2024-04-25T00:30:47.917551+0000","last_became_active":"2024-04-25T00:01:18.501633+0000","last_became_peered":"2024-04-25T00:01:18.501633+0000","last_unstale":"2024-04-25T00:30:47.917551+0000","last_undegraded":"2024-04-25T00:30:47.917551+0000","last_fullsized":"2024-04-25T00:30:47.917551+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'3","last_scrub_stamp":"2024-04-25T00:30:47.917353+0000","last_deep_scrub":"74'3","last_deep_scrub_stamp":"2024-04-25T00:30:47.917353+0000","last_clean_scrub_stamp":"2024-04-25T00:30:47.917353+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:33:11.613082+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"74'5031","reported_seq":5147,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:46.227615+0000","last_change":"2024-04-25T00:01:46.662019+0000","last_active":"2024-04-25T00:30:46.227615+0000","last_peered":"2024-04-25T00:30:46.227615+0000","last_clean":"2024-04-25T00:30:46.227615+0000","last_became_active":"2024-04-25T00:01:46.661167+0000","last_became_peered":"2024-04-25T00:01:46.661167+0000","last_unstale":"2024-04-25T00:30:46.227615+0000","last_undegraded":"2024-04-25T00:30:46.227615+0000","last_fullsized":"2024-04-25T00:30:46.227615+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":5031,"log_dups_size":0,"ondisk_log_size":5031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":22,"num_write":27250,"num_write_kb":338892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"73'28","reported_seq":167,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:59.687962+0000","last_change":"2024-04-25T00:30:59.688091+0000","last_active":"2024-04-25T00:30:59.687962+0000","last_peered":"2024-04-25T00:30:59.687962+0000","last_clean":"2024-04-25T00:30:59.687962+0000","last_became_active":"2024-04-25T00:01:04.361059+0000","last_became_peered":"2024-04-25T00:01:04.361059+0000","last_unstale":"2024-04-25T00:30:59.687962+0000","last_undegraded":"2024-04-25T00:30:59.687962+0000","last_fullsized":"2024-04-25T00:30:59.687962+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'28","last_scrub_stamp":"2024-04-25T00:30:59.687701+0000","last_deep_scrub":"73'28","last_deep_scrub_stamp":"2024-04-25T00:30:59.687701+0000","last_clean_scrub_stamp":"2024-04-25T00:30:59.687701+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:55:01.388407+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"70'5376","reported_seq":5513,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:53.537262+0000","last_change":"2024-04-25T00:30:53.537392+0000","last_active":"2024-04-25T00:30:53.537262+0000","last_peered":"2024-04-25T00:30:53.537262+0000","last_clean":"2024-04-25T00:30:53.537262+0000","last_became_active":"2024-04-25T00:01:05.369991+0000","last_became_peered":"2024-04-25T00:01:05.369991+0000","last_unstale":"2024-04-25T00:30:53.537262+0000","last_undegraded":"2024-04-25T00:30:53.537262+0000","last_fullsized":"2024-04-25T00:30:53.537262+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'5376","last_scrub_stamp":"2024-04-25T00:30:53.536989+0000","last_deep_scrub":"70'5376","last_deep_scrub_stamp":"2024-04-25T00:30:53.536989+0000","last_clean_scrub_stamp":"2024-04-25T00:30:53.536989+0000","objects_scrubbed":0,"log_size":5376,"log_dups_size":0,"ondisk_log_size":5376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:28:29.688314+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29984,"num_write_kb":386541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"74'2","reported_seq":131,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:47.445142+0000","last_change":"2024-04-25T00:30:47.445199+0000","last_active":"2024-04-25T00:30:47.445142+0000","last_peered":"2024-04-25T00:30:47.445142+0000","last_clean":"2024-04-25T00:30:47.445142+0000","last_became_active":"2024-04-25T00:01:18.497397+0000","last_became_peered":"2024-04-25T00:01:18.497397+0000","last_unstale":"2024-04-25T00:30:47.445142+0000","last_undegraded":"2024-04-25T00:30:47.445142+0000","last_fullsized":"2024-04-25T00:30:47.445142+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'2","last_scrub_stamp":"2024-04-25T00:30:47.444936+0000","last_deep_scrub":"74'2","last_deep_scrub_stamp":"2024-04-25T00:30:47.444936+0000","last_clean_scrub_stamp":"2024-04-25T00:30:47.444936+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:32:50.758882+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"74'4080","reported_seq":4208,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:57.290235+0000","last_change":"2024-04-25T00:30:57.290275+0000","last_active":"2024-04-25T00:30:57.290235+0000","last_peered":"2024-04-25T00:30:57.290235+0000","last_clean":"2024-04-25T00:30:57.290235+0000","last_became_active":"2024-04-25T00:01:19.510252+0000","last_became_peered":"2024-04-25T00:01:19.510252+0000","last_unstale":"2024-04-25T00:30:57.290235+0000","last_undegraded":"2024-04-25T00:30:57.290235+0000","last_fullsized":"2024-04-25T00:30:57.290235+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4080","last_scrub_stamp":"2024-04-25T00:30:57.290098+0000","last_deep_scrub":"74'4080","last_deep_scrub_stamp":"2024-04-25T00:30:57.290098+0000","last_clean_scrub_stamp":"2024-04-25T00:30:57.290098+0000","objects_scrubbed":0,"log_size":4080,"log_dups_size":0,"ondisk_log_size":4080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:44:28.936028+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":21789,"num_write_kb":296332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"74'54","reported_seq":213,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:53.324062+0000","last_change":"2024-04-25T00:30:53.324192+0000","last_active":"2024-04-25T00:30:53.324062+0000","last_peered":"2024-04-25T00:30:53.324062+0000","last_clean":"2024-04-25T00:30:53.324062+0000","last_became_active":"2024-04-25T00:01:04.361708+0000","last_became_peered":"2024-04-25T00:01:04.361708+0000","last_unstale":"2024-04-25T00:30:53.324062+0000","last_undegraded":"2024-04-25T00:30:53.324062+0000","last_fullsized":"2024-04-25T00:30:53.324062+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'54","last_scrub_stamp":"2024-04-25T00:30:53.323735+0000","last_deep_scrub":"74'54","last_deep_scrub_stamp":"2024-04-25T00:30:53.323735+0000","last_clean_scrub_stamp":"2024-04-25T00:30:53.323735+0000","objects_scrubbed":1,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:26:43.034066+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":69,"num_read_kb":69,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"69'3669","reported_seq":3803,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:50.083758+0000","last_change":"2024-04-25T00:30:50.083862+0000","last_active":"2024-04-25T00:30:50.083758+0000","last_peered":"2024-04-25T00:30:50.083758+0000","last_clean":"2024-04-25T00:30:50.083758+0000","last_became_active":"2024-04-25T00:01:05.386853+0000","last_became_peered":"2024-04-25T00:01:05.386853+0000","last_unstale":"2024-04-25T00:30:50.083758+0000","last_undegraded":"2024-04-25T00:30:50.083758+0000","last_fullsized":"2024-04-25T00:30:50.083758+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'3669","last_scrub_stamp":"2024-04-25T00:30:50.083530+0000","last_deep_scrub":"69'3669","last_deep_scrub_stamp":"2024-04-25T00:30:50.083530+0000","last_clean_scrub_stamp":"2024-04-25T00:30:50.083530+0000","objects_scrubbed":0,"log_size":3669,"log_dups_size":0,"ondisk_log_size":3669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:44:49.651163+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19904,"num_write_kb":264274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"74'724","reported_seq":879,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:46.511926+0000","last_change":"2024-04-25T00:30:46.511981+0000","last_active":"2024-04-25T00:30:46.511926+0000","last_peered":"2024-04-25T00:30:46.511926+0000","last_clean":"2024-04-25T00:30:46.511926+0000","last_became_active":"2024-04-25T00:01:18.487571+0000","last_became_peered":"2024-04-25T00:01:18.487571+0000","last_unstale":"2024-04-25T00:30:46.511926+0000","last_undegraded":"2024-04-25T00:30:46.511926+0000","last_fullsized":"2024-04-25T00:30:46.511926+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'724","last_scrub_stamp":"2024-04-25T00:30:46.511738+0000","last_deep_scrub":"74'724","last_deep_scrub_stamp":"2024-04-25T00:30:46.511738+0000","last_clean_scrub_stamp":"2024-04-25T00:30:46.511738+0000","objects_scrubbed":1,"log_size":724,"log_dups_size":0,"ondisk_log_size":724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:40:44.393760+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":27,"num_read_kb":82727,"num_write":711,"num_write_kb":4310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"74'3586","reported_seq":3705,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.720264+0000","last_change":"2024-04-25T00:01:19.519834+0000","last_active":"2024-04-25T00:30:26.720264+0000","last_peered":"2024-04-25T00:30:26.720264+0000","last_clean":"2024-04-25T00:30:26.720264+0000","last_became_active":"2024-04-25T00:01:19.519470+0000","last_became_peered":"2024-04-25T00:01:19.519470+0000","last_unstale":"2024-04-25T00:30:26.720264+0000","last_undegraded":"2024-04-25T00:30:26.720264+0000","last_fullsized":"2024-04-25T00:30:26.720264+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3586,"log_dups_size":0,"ondisk_log_size":3586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":5,"num_write":18586,"num_write_kb":254663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"74'7","reported_seq":189,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:55.301134+0000","last_change":"2024-04-25T00:30:55.301289+0000","last_active":"2024-04-25T00:30:55.301134+0000","last_peered":"2024-04-25T00:30:55.301134+0000","last_clean":"2024-04-25T00:30:55.301134+0000","last_became_active":"2024-04-25T00:01:04.362059+0000","last_became_peered":"2024-04-25T00:01:04.362059+0000","last_unstale":"2024-04-25T00:30:55.301134+0000","last_undegraded":"2024-04-25T00:30:55.301134+0000","last_fullsized":"2024-04-25T00:30:55.301134+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'7","last_scrub_stamp":"2024-04-25T00:30:55.300906+0000","last_deep_scrub":"74'7","last_deep_scrub_stamp":"2024-04-25T00:30:55.300906+0000","last_clean_scrub_stamp":"2024-04-25T00:30:55.300906+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:18:19.719987+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"70'4973","reported_seq":5107,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:57.204605+0000","last_change":"2024-04-25T00:30:57.204739+0000","last_active":"2024-04-25T00:30:57.204605+0000","last_peered":"2024-04-25T00:30:57.204605+0000","last_clean":"2024-04-25T00:30:57.204605+0000","last_became_active":"2024-04-25T00:01:05.378320+0000","last_became_peered":"2024-04-25T00:01:05.378320+0000","last_unstale":"2024-04-25T00:30:57.204605+0000","last_undegraded":"2024-04-25T00:30:57.204605+0000","last_fullsized":"2024-04-25T00:30:57.204605+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'4973","last_scrub_stamp":"2024-04-25T00:30:57.204327+0000","last_deep_scrub":"70'4973","last_deep_scrub_stamp":"2024-04-25T00:30:57.204327+0000","last_clean_scrub_stamp":"2024-04-25T00:30:57.204327+0000","objects_scrubbed":0,"log_size":4973,"log_dups_size":0,"ondisk_log_size":4973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:23:22.383368+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27225,"num_write_kb":350946,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"74'356","reported_seq":486,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:45.489272+0000","last_change":"2024-04-25T00:30:45.489299+0000","last_active":"2024-04-25T00:30:45.489272+0000","last_peered":"2024-04-25T00:30:45.489272+0000","last_clean":"2024-04-25T00:30:45.489272+0000","last_became_active":"2024-04-25T00:01:18.490050+0000","last_became_peered":"2024-04-25T00:01:18.490050+0000","last_unstale":"2024-04-25T00:30:45.489272+0000","last_undegraded":"2024-04-25T00:30:45.489272+0000","last_fullsized":"2024-04-25T00:30:45.489272+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'356","last_scrub_stamp":"2024-04-25T00:30:45.489171+0000","last_deep_scrub":"74'356","last_deep_scrub_stamp":"2024-04-25T00:30:45.489171+0000","last_clean_scrub_stamp":"2024-04-25T00:30:45.489171+0000","objects_scrubbed":1,"log_size":356,"log_dups_size":0,"ondisk_log_size":356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:47:00.287833+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2691973,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":907,"num_write":327,"num_write_kb":2826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"74'4320","reported_seq":4443,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:38.002989+0000","last_change":"2024-04-25T00:01:19.500026+0000","last_active":"2024-04-25T00:30:38.002989+0000","last_peered":"2024-04-25T00:30:38.002989+0000","last_clean":"2024-04-25T00:30:38.002989+0000","last_became_active":"2024-04-25T00:01:19.499244+0000","last_became_peered":"2024-04-25T00:01:19.499244+0000","last_unstale":"2024-04-25T00:30:38.002989+0000","last_undegraded":"2024-04-25T00:30:38.002989+0000","last_fullsized":"2024-04-25T00:30:38.002989+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4320,"log_dups_size":0,"ondisk_log_size":4320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":15,"num_write":23810,"num_write_kb":308534,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"70'4158","reported_seq":4294,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:51.241517+0000","last_change":"2024-04-25T00:30:51.241583+0000","last_active":"2024-04-25T00:30:51.241517+0000","last_peered":"2024-04-25T00:30:51.241517+0000","last_clean":"2024-04-25T00:30:51.241517+0000","last_became_active":"2024-04-25T00:01:05.376259+0000","last_became_peered":"2024-04-25T00:01:05.376259+0000","last_unstale":"2024-04-25T00:30:51.241517+0000","last_undegraded":"2024-04-25T00:30:51.241517+0000","last_fullsized":"2024-04-25T00:30:51.241517+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'4158","last_scrub_stamp":"2024-04-25T00:30:51.241297+0000","last_deep_scrub":"70'4158","last_deep_scrub_stamp":"2024-04-25T00:30:51.241297+0000","last_clean_scrub_stamp":"2024-04-25T00:30:51.241297+0000","objects_scrubbed":0,"log_size":4158,"log_dups_size":0,"ondisk_log_size":4158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:46:42.723857+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23006,"num_write_kb":301346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"74'1526","reported_seq":1711,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:25.825359+0000","last_change":"2024-04-25T00:01:04.359262+0000","last_active":"2024-04-25T00:30:25.825359+0000","last_peered":"2024-04-25T00:30:25.825359+0000","last_clean":"2024-04-25T00:30:25.825359+0000","last_became_active":"2024-04-25T00:01:04.357971+0000","last_became_peered":"2024-04-25T00:01:04.357971+0000","last_unstale":"2024-04-25T00:30:25.825359+0000","last_undegraded":"2024-04-25T00:30:25.825359+0000","last_fullsized":"2024-04-25T00:30:25.825359+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":1526,"log_dups_size":0,"ondisk_log_size":1526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":78,"num_read_kb":52062,"num_write":1499,"num_write_kb":4525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"74'6","reported_seq":151,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:38.035541+0000","last_change":"2024-04-25T00:01:18.475624+0000","last_active":"2024-04-25T00:30:38.035541+0000","last_peered":"2024-04-25T00:30:38.035541+0000","last_clean":"2024-04-25T00:30:38.035541+0000","last_became_active":"2024-04-25T00:01:18.475149+0000","last_became_peered":"2024-04-25T00:01:18.475149+0000","last_unstale":"2024-04-25T00:30:38.035541+0000","last_undegraded":"2024-04-25T00:30:38.035541+0000","last_fullsized":"2024-04-25T00:30:38.035541+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"74'4556","reported_seq":4687,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:59.782276+0000","last_change":"2024-04-25T00:30:59.782334+0000","last_active":"2024-04-25T00:30:59.782276+0000","last_peered":"2024-04-25T00:30:59.782276+0000","last_clean":"2024-04-25T00:30:59.782276+0000","last_became_active":"2024-04-25T00:01:19.520150+0000","last_became_peered":"2024-04-25T00:01:19.520150+0000","last_unstale":"2024-04-25T00:30:59.782276+0000","last_undegraded":"2024-04-25T00:30:59.782276+0000","last_fullsized":"2024-04-25T00:30:59.782276+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4556","last_scrub_stamp":"2024-04-25T00:30:59.782060+0000","last_deep_scrub":"74'4556","last_deep_scrub_stamp":"2024-04-25T00:30:59.782060+0000","last_clean_scrub_stamp":"2024-04-25T00:30:59.782060+0000","objects_scrubbed":0,"log_size":4556,"log_dups_size":0,"ondisk_log_size":4556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:52:39.891690+0000","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":19,"num_write":25634,"num_write_kb":315935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"70'3060","reported_seq":3196,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:51.843128+0000","last_change":"2024-04-25T00:30:51.843191+0000","last_active":"2024-04-25T00:30:51.843128+0000","last_peered":"2024-04-25T00:30:51.843128+0000","last_clean":"2024-04-25T00:30:51.843128+0000","last_became_active":"2024-04-25T00:01:05.361817+0000","last_became_peered":"2024-04-25T00:01:05.361817+0000","last_unstale":"2024-04-25T00:30:51.843128+0000","last_undegraded":"2024-04-25T00:30:51.843128+0000","last_fullsized":"2024-04-25T00:30:51.843128+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'3060","last_scrub_stamp":"2024-04-25T00:30:51.842888+0000","last_deep_scrub":"70'3060","last_deep_scrub_stamp":"2024-04-25T00:30:51.842888+0000","last_clean_scrub_stamp":"2024-04-25T00:30:51.842888+0000","objects_scrubbed":0,"log_size":3060,"log_dups_size":0,"ondisk_log_size":3060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:18:39.934309+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16241,"num_write_kb":228970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"70'3","reported_seq":141,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:55.515670+0000","last_change":"2024-04-25T00:30:55.515733+0000","last_active":"2024-04-25T00:30:55.515670+0000","last_peered":"2024-04-25T00:30:55.515670+0000","last_clean":"2024-04-25T00:30:55.515670+0000","last_became_active":"2024-04-25T00:01:04.371420+0000","last_became_peered":"2024-04-25T00:01:04.371420+0000","last_unstale":"2024-04-25T00:30:55.515670+0000","last_undegraded":"2024-04-25T00:30:55.515670+0000","last_fullsized":"2024-04-25T00:30:55.515670+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'3","last_scrub_stamp":"2024-04-25T00:30:55.515462+0000","last_deep_scrub":"70'3","last_deep_scrub_stamp":"2024-04-25T00:30:55.515462+0000","last_clean_scrub_stamp":"2024-04-25T00:30:55.515462+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:19:36.914335+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"74'5","reported_seq":135,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:00.319927+0000","last_change":"2024-04-25T00:31:00.319995+0000","last_active":"2024-04-25T00:31:00.319927+0000","last_peered":"2024-04-25T00:31:00.319927+0000","last_clean":"2024-04-25T00:31:00.319927+0000","last_became_active":"2024-04-25T00:01:18.488291+0000","last_became_peered":"2024-04-25T00:01:18.488291+0000","last_unstale":"2024-04-25T00:31:00.319927+0000","last_undegraded":"2024-04-25T00:31:00.319927+0000","last_fullsized":"2024-04-25T00:31:00.319927+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-04-25T00:31:00.319723+0000","last_deep_scrub":"74'5","last_deep_scrub_stamp":"2024-04-25T00:31:00.319723+0000","last_clean_scrub_stamp":"2024-04-25T00:31:00.319723+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:20:08.156713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"74'3591","reported_seq":3712,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:38.035694+0000","last_change":"2024-04-25T00:01:19.516731+0000","last_active":"2024-04-25T00:30:38.035694+0000","last_peered":"2024-04-25T00:30:38.035694+0000","last_clean":"2024-04-25T00:30:38.035694+0000","last_became_active":"2024-04-25T00:01:19.516405+0000","last_became_peered":"2024-04-25T00:01:19.516405+0000","last_unstale":"2024-04-25T00:30:38.035694+0000","last_undegraded":"2024-04-25T00:30:38.035694+0000","last_fullsized":"2024-04-25T00:30:38.035694+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3591,"log_dups_size":0,"ondisk_log_size":3591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":19584,"num_write_kb":250628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"69'3777","reported_seq":3913,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:01.901532+0000","last_change":"2024-04-25T00:31:01.901593+0000","last_active":"2024-04-25T00:31:01.901532+0000","last_peered":"2024-04-25T00:31:01.901532+0000","last_clean":"2024-04-25T00:31:01.901532+0000","last_became_active":"2024-04-25T00:01:05.354699+0000","last_became_peered":"2024-04-25T00:01:05.354699+0000","last_unstale":"2024-04-25T00:31:01.901532+0000","last_undegraded":"2024-04-25T00:31:01.901532+0000","last_fullsized":"2024-04-25T00:31:01.901532+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'3777","last_scrub_stamp":"2024-04-25T00:31:01.901333+0000","last_deep_scrub":"69'3777","last_deep_scrub_stamp":"2024-04-25T00:31:01.901333+0000","last_clean_scrub_stamp":"2024-04-25T00:31:01.901333+0000","objects_scrubbed":0,"log_size":3777,"log_dups_size":0,"ondisk_log_size":3777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:36:47.997136+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20993,"num_write_kb":256390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"69'2","reported_seq":140,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:59.839154+0000","last_change":"2024-04-25T00:30:59.839212+0000","last_active":"2024-04-25T00:30:59.839154+0000","last_peered":"2024-04-25T00:30:59.839154+0000","last_clean":"2024-04-25T00:30:59.839154+0000","last_became_active":"2024-04-25T00:01:04.361622+0000","last_became_peered":"2024-04-25T00:01:04.361622+0000","last_unstale":"2024-04-25T00:30:59.839154+0000","last_undegraded":"2024-04-25T00:30:59.839154+0000","last_fullsized":"2024-04-25T00:30:59.839154+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-04-25T00:30:59.838955+0000","last_deep_scrub":"69'2","last_deep_scrub_stamp":"2024-04-25T00:30:59.838955+0000","last_clean_scrub_stamp":"2024-04-25T00:30:59.838955+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:10:07.461621+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"74'5","reported_seq":158,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:54.971187+0000","last_change":"2024-04-25T00:30:54.971259+0000","last_active":"2024-04-25T00:30:54.971187+0000","last_peered":"2024-04-25T00:30:54.971187+0000","last_clean":"2024-04-25T00:30:54.971187+0000","last_became_active":"2024-04-25T00:01:18.493614+0000","last_became_peered":"2024-04-25T00:01:18.493614+0000","last_unstale":"2024-04-25T00:30:54.971187+0000","last_undegraded":"2024-04-25T00:30:54.971187+0000","last_fullsized":"2024-04-25T00:30:54.971187+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-04-25T00:30:54.970960+0000","last_deep_scrub":"74'5","last_deep_scrub_stamp":"2024-04-25T00:30:54.970960+0000","last_clean_scrub_stamp":"2024-04-25T00:30:54.970960+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:54:38.535896+0000","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"74'4895","reported_seq":5023,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:53.104085+0000","last_change":"2024-04-25T00:30:53.104146+0000","last_active":"2024-04-25T00:30:53.104085+0000","last_peered":"2024-04-25T00:30:53.104085+0000","last_clean":"2024-04-25T00:30:53.104085+0000","last_became_active":"2024-04-25T00:01:19.505126+0000","last_became_peered":"2024-04-25T00:01:19.505126+0000","last_unstale":"2024-04-25T00:30:53.104085+0000","last_undegraded":"2024-04-25T00:30:53.104085+0000","last_fullsized":"2024-04-25T00:30:53.104085+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4895","last_scrub_stamp":"2024-04-25T00:30:53.103877+0000","last_deep_scrub":"74'4895","last_deep_scrub_stamp":"2024-04-25T00:30:53.103877+0000","last_clean_scrub_stamp":"2024-04-25T00:30:53.103877+0000","objects_scrubbed":0,"log_size":4895,"log_dups_size":0,"ondisk_log_size":4895,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:34:00.242731+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26799,"num_write_kb":349350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"69'3934","reported_seq":4070,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:02.251600+0000","last_change":"2024-04-25T00:31:02.251665+0000","last_active":"2024-04-25T00:31:02.251600+0000","last_peered":"2024-04-25T00:31:02.251600+0000","last_clean":"2024-04-25T00:31:02.251600+0000","last_became_active":"2024-04-25T00:01:05.354118+0000","last_became_peered":"2024-04-25T00:01:05.354118+0000","last_unstale":"2024-04-25T00:31:02.251600+0000","last_undegraded":"2024-04-25T00:31:02.251600+0000","last_fullsized":"2024-04-25T00:31:02.251600+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'3934","last_scrub_stamp":"2024-04-25T00:31:02.251375+0000","last_deep_scrub":"69'3934","last_deep_scrub_stamp":"2024-04-25T00:31:02.251375+0000","last_clean_scrub_stamp":"2024-04-25T00:31:02.251375+0000","objects_scrubbed":0,"log_size":3934,"log_dups_size":0,"ondisk_log_size":3934,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:32:16.209225+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21059,"num_write_kb":273031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"70'6","reported_seq":144,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:56.344716+0000","last_change":"2024-04-25T00:30:56.344776+0000","last_active":"2024-04-25T00:30:56.344716+0000","last_peered":"2024-04-25T00:30:56.344716+0000","last_clean":"2024-04-25T00:30:56.344716+0000","last_became_active":"2024-04-25T00:01:04.364942+0000","last_became_peered":"2024-04-25T00:01:04.364942+0000","last_unstale":"2024-04-25T00:30:56.344716+0000","last_undegraded":"2024-04-25T00:30:56.344716+0000","last_fullsized":"2024-04-25T00:30:56.344716+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'6","last_scrub_stamp":"2024-04-25T00:30:56.344504+0000","last_deep_scrub":"70'6","last_deep_scrub_stamp":"2024-04-25T00:30:56.344504+0000","last_clean_scrub_stamp":"2024-04-25T00:30:56.344504+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:02:38.727658+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"74'8","reported_seq":184,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:54.616129+0000","last_change":"2024-04-25T00:30:54.616191+0000","last_active":"2024-04-25T00:30:54.616129+0000","last_peered":"2024-04-25T00:30:54.616129+0000","last_clean":"2024-04-25T00:30:54.616129+0000","last_became_active":"2024-04-25T00:01:18.475862+0000","last_became_peered":"2024-04-25T00:01:18.475862+0000","last_unstale":"2024-04-25T00:30:54.616129+0000","last_undegraded":"2024-04-25T00:30:54.616129+0000","last_fullsized":"2024-04-25T00:30:54.616129+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-04-25T00:30:54.615941+0000","last_deep_scrub":"74'8","last_deep_scrub_stamp":"2024-04-25T00:30:54.615941+0000","last_clean_scrub_stamp":"2024-04-25T00:30:54.615941+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:28:30.767266+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"74'3090","reported_seq":3211,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.778252+0000","last_change":"2024-04-25T00:01:19.529607+0000","last_active":"2024-04-25T00:30:26.778252+0000","last_peered":"2024-04-25T00:30:26.778252+0000","last_clean":"2024-04-25T00:30:26.778252+0000","last_became_active":"2024-04-25T00:01:19.528509+0000","last_became_peered":"2024-04-25T00:01:19.528509+0000","last_unstale":"2024-04-25T00:30:26.778252+0000","last_undegraded":"2024-04-25T00:30:26.778252+0000","last_fullsized":"2024-04-25T00:30:26.778252+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3090,"log_dups_size":0,"ondisk_log_size":3090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17043,"num_write_kb":228060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"70'4336","reported_seq":4472,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:02.910884+0000","last_change":"2024-04-25T00:31:02.910951+0000","last_active":"2024-04-25T00:31:02.910884+0000","last_peered":"2024-04-25T00:31:02.910884+0000","last_clean":"2024-04-25T00:31:02.910884+0000","last_became_active":"2024-04-25T00:01:05.357135+0000","last_became_peered":"2024-04-25T00:01:05.357135+0000","last_unstale":"2024-04-25T00:31:02.910884+0000","last_undegraded":"2024-04-25T00:31:02.910884+0000","last_fullsized":"2024-04-25T00:31:02.910884+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'4336","last_scrub_stamp":"2024-04-25T00:31:02.910672+0000","last_deep_scrub":"70'4336","last_deep_scrub_stamp":"2024-04-25T00:31:02.910672+0000","last_clean_scrub_stamp":"2024-04-25T00:31:02.910672+0000","objects_scrubbed":0,"log_size":4336,"log_dups_size":0,"ondisk_log_size":4336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:01:28.439618+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23030,"num_write_kb":325887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"71'9","reported_seq":185,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.814602+0000","last_change":"2024-04-25T00:01:04.369917+0000","last_active":"2024-04-25T00:30:15.814602+0000","last_peered":"2024-04-25T00:30:15.814602+0000","last_clean":"2024-04-25T00:30:15.814602+0000","last_became_active":"2024-04-25T00:01:04.369309+0000","last_became_peered":"2024-04-25T00:01:04.369309+0000","last_unstale":"2024-04-25T00:30:15.814602+0000","last_undegraded":"2024-04-25T00:30:15.814602+0000","last_fullsized":"2024-04-25T00:30:15.814602+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":8,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"74'51","reported_seq":250,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:48.496620+0000","last_change":"2024-04-25T00:30:48.496682+0000","last_active":"2024-04-25T00:30:48.496620+0000","last_peered":"2024-04-25T00:30:48.496620+0000","last_clean":"2024-04-25T00:30:48.496620+0000","last_became_active":"2024-04-25T00:01:18.476909+0000","last_became_peered":"2024-04-25T00:01:18.476909+0000","last_unstale":"2024-04-25T00:30:48.496620+0000","last_undegraded":"2024-04-25T00:30:48.496620+0000","last_fullsized":"2024-04-25T00:30:48.496620+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'51","last_scrub_stamp":"2024-04-25T00:30:48.496424+0000","last_deep_scrub":"74'51","last_deep_scrub_stamp":"2024-04-25T00:30:48.496424+0000","last_clean_scrub_stamp":"2024-04-25T00:30:48.496424+0000","objects_scrubbed":1,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:01:36.221579+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84646,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":68,"num_read_kb":112,"num_write":46,"num_write_kb":112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"74'3821","reported_seq":3943,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:33.034614+0000","last_change":"2024-04-25T00:01:19.517500+0000","last_active":"2024-04-25T00:30:33.034614+0000","last_peered":"2024-04-25T00:30:33.034614+0000","last_clean":"2024-04-25T00:30:33.034614+0000","last_became_active":"2024-04-25T00:01:19.517182+0000","last_became_peered":"2024-04-25T00:01:19.517182+0000","last_unstale":"2024-04-25T00:30:33.034614+0000","last_undegraded":"2024-04-25T00:30:33.034614+0000","last_fullsized":"2024-04-25T00:30:33.034614+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3821,"log_dups_size":0,"ondisk_log_size":3821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20929,"num_write_kb":256020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"70'3227","reported_seq":3363,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:03.879713+0000","last_change":"2024-04-25T00:31:03.879774+0000","last_active":"2024-04-25T00:31:03.879713+0000","last_peered":"2024-04-25T00:31:03.879713+0000","last_clean":"2024-04-25T00:31:03.879713+0000","last_became_active":"2024-04-25T00:01:05.364125+0000","last_became_peered":"2024-04-25T00:01:05.364125+0000","last_unstale":"2024-04-25T00:31:03.879713+0000","last_undegraded":"2024-04-25T00:31:03.879713+0000","last_fullsized":"2024-04-25T00:31:03.879713+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'3227","last_scrub_stamp":"2024-04-25T00:31:03.879520+0000","last_deep_scrub":"70'3227","last_deep_scrub_stamp":"2024-04-25T00:31:03.879520+0000","last_clean_scrub_stamp":"2024-04-25T00:31:03.879520+0000","objects_scrubbed":0,"log_size":3227,"log_dups_size":0,"ondisk_log_size":3227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:43:58.941733+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17808,"num_write_kb":231231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"74'459","reported_seq":620,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:58.878220+0000","last_change":"2024-04-25T00:30:58.878367+0000","last_active":"2024-04-25T00:30:58.878220+0000","last_peered":"2024-04-25T00:30:58.878220+0000","last_clean":"2024-04-25T00:30:58.878220+0000","last_became_active":"2024-04-25T00:01:04.361046+0000","last_became_peered":"2024-04-25T00:01:04.361046+0000","last_unstale":"2024-04-25T00:30:58.878220+0000","last_undegraded":"2024-04-25T00:30:58.878220+0000","last_fullsized":"2024-04-25T00:30:58.878220+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'459","last_scrub_stamp":"2024-04-25T00:30:58.877990+0000","last_deep_scrub":"74'459","last_deep_scrub_stamp":"2024-04-25T00:30:58.877990+0000","last_clean_scrub_stamp":"2024-04-25T00:30:58.877990+0000","objects_scrubbed":1,"log_size":459,"log_dups_size":0,"ondisk_log_size":459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:33:10.063484+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":46,"num_read_kb":63,"num_write":1069,"num_write_kb":360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"74'12","reported_seq":143,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:46.735683+0000","last_change":"2024-04-25T00:30:46.735741+0000","last_active":"2024-04-25T00:30:46.735683+0000","last_peered":"2024-04-25T00:30:46.735683+0000","last_clean":"2024-04-25T00:30:46.735683+0000","last_became_active":"2024-04-25T00:01:18.487321+0000","last_became_peered":"2024-04-25T00:01:18.487321+0000","last_unstale":"2024-04-25T00:30:46.735683+0000","last_undegraded":"2024-04-25T00:30:46.735683+0000","last_fullsized":"2024-04-25T00:30:46.735683+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-04-25T00:30:46.735452+0000","last_deep_scrub":"74'12","last_deep_scrub_stamp":"2024-04-25T00:30:46.735452+0000","last_clean_scrub_stamp":"2024-04-25T00:30:46.735452+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:21:22.368298+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"74'4221","reported_seq":4342,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:33.622727+0000","last_change":"2024-04-25T00:01:19.519653+0000","last_active":"2024-04-25T00:30:33.622727+0000","last_peered":"2024-04-25T00:30:33.622727+0000","last_clean":"2024-04-25T00:30:33.622727+0000","last_became_active":"2024-04-25T00:01:19.519052+0000","last_became_peered":"2024-04-25T00:01:19.519052+0000","last_unstale":"2024-04-25T00:30:33.622727+0000","last_undegraded":"2024-04-25T00:30:33.622727+0000","last_fullsized":"2024-04-25T00:30:33.622727+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4221,"log_dups_size":0,"ondisk_log_size":4221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22937,"num_write_kb":280510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"70'5174","reported_seq":5311,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:58.516351+0000","last_change":"2024-04-25T00:30:58.516492+0000","last_active":"2024-04-25T00:30:58.516351+0000","last_peered":"2024-04-25T00:30:58.516351+0000","last_clean":"2024-04-25T00:30:58.516351+0000","last_became_active":"2024-04-25T00:01:05.373276+0000","last_became_peered":"2024-04-25T00:01:05.373276+0000","last_unstale":"2024-04-25T00:30:58.516351+0000","last_undegraded":"2024-04-25T00:30:58.516351+0000","last_fullsized":"2024-04-25T00:30:58.516351+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'5174","last_scrub_stamp":"2024-04-25T00:30:58.516111+0000","last_deep_scrub":"70'5174","last_deep_scrub_stamp":"2024-04-25T00:30:58.516111+0000","last_clean_scrub_stamp":"2024-04-25T00:30:58.516111+0000","objects_scrubbed":0,"log_size":5174,"log_dups_size":0,"ondisk_log_size":5174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:41:04.789419+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28022,"num_write_kb":362384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"69'2","reported_seq":133,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:23.032989+0000","last_change":"2024-04-25T00:01:04.365391+0000","last_active":"2024-04-25T00:30:23.032989+0000","last_peered":"2024-04-25T00:30:23.032989+0000","last_clean":"2024-04-25T00:30:23.032989+0000","last_became_active":"2024-04-25T00:01:04.364989+0000","last_became_peered":"2024-04-25T00:01:04.364989+0000","last_unstale":"2024-04-25T00:30:23.032989+0000","last_undegraded":"2024-04-25T00:30:23.032989+0000","last_fullsized":"2024-04-25T00:30:23.032989+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"74'3","reported_seq":134,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:48.514879+0000","last_change":"2024-04-25T00:30:48.514944+0000","last_active":"2024-04-25T00:30:48.514879+0000","last_peered":"2024-04-25T00:30:48.514879+0000","last_clean":"2024-04-25T00:30:48.514879+0000","last_became_active":"2024-04-25T00:01:18.490859+0000","last_became_peered":"2024-04-25T00:01:18.490859+0000","last_unstale":"2024-04-25T00:30:48.514879+0000","last_undegraded":"2024-04-25T00:30:48.514879+0000","last_fullsized":"2024-04-25T00:30:48.514879+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'3","last_scrub_stamp":"2024-04-25T00:30:48.514678+0000","last_deep_scrub":"74'3","last_deep_scrub_stamp":"2024-04-25T00:30:48.514678+0000","last_clean_scrub_stamp":"2024-04-25T00:30:48.514678+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:33:55.653532+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"74'5316","reported_seq":5431,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:46.227559+0000","last_change":"2024-04-25T00:01:46.667934+0000","last_active":"2024-04-25T00:30:46.227559+0000","last_peered":"2024-04-25T00:30:46.227559+0000","last_clean":"2024-04-25T00:30:46.227559+0000","last_became_active":"2024-04-25T00:01:46.667075+0000","last_became_peered":"2024-04-25T00:01:46.667075+0000","last_unstale":"2024-04-25T00:30:46.227559+0000","last_undegraded":"2024-04-25T00:30:46.227559+0000","last_fullsized":"2024-04-25T00:30:46.227559+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":5316,"log_dups_size":0,"ondisk_log_size":5316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29390,"num_write_kb":363802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"70'3961","reported_seq":4098,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:01.721313+0000","last_change":"2024-04-25T00:31:01.721378+0000","last_active":"2024-04-25T00:31:01.721313+0000","last_peered":"2024-04-25T00:31:01.721313+0000","last_clean":"2024-04-25T00:31:01.721313+0000","last_became_active":"2024-04-25T00:01:05.377593+0000","last_became_peered":"2024-04-25T00:01:05.377593+0000","last_unstale":"2024-04-25T00:31:01.721313+0000","last_undegraded":"2024-04-25T00:31:01.721313+0000","last_fullsized":"2024-04-25T00:31:01.721313+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'3961","last_scrub_stamp":"2024-04-25T00:31:01.721093+0000","last_deep_scrub":"70'3961","last_deep_scrub_stamp":"2024-04-25T00:31:01.721093+0000","last_clean_scrub_stamp":"2024-04-25T00:31:01.721093+0000","objects_scrubbed":0,"log_size":3961,"log_dups_size":0,"ondisk_log_size":3961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:15:42.320004+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21237,"num_write_kb":269574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"70'2","reported_seq":141,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:00.728001+0000","last_change":"2024-04-25T00:31:00.728062+0000","last_active":"2024-04-25T00:31:00.728001+0000","last_peered":"2024-04-25T00:31:00.728001+0000","last_clean":"2024-04-25T00:31:00.728001+0000","last_became_active":"2024-04-25T00:01:04.365881+0000","last_became_peered":"2024-04-25T00:01:04.365881+0000","last_unstale":"2024-04-25T00:31:00.728001+0000","last_undegraded":"2024-04-25T00:31:00.728001+0000","last_fullsized":"2024-04-25T00:31:00.728001+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-04-25T00:31:00.727781+0000","last_deep_scrub":"70'2","last_deep_scrub_stamp":"2024-04-25T00:31:00.727781+0000","last_clean_scrub_stamp":"2024-04-25T00:31:00.727781+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:47:19.164558+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'4","reported_seq":135,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:49.524439+0000","last_change":"2024-04-25T00:30:49.524509+0000","last_active":"2024-04-25T00:30:49.524439+0000","last_peered":"2024-04-25T00:30:49.524439+0000","last_clean":"2024-04-25T00:30:49.524439+0000","last_became_active":"2024-04-25T00:01:18.491498+0000","last_became_peered":"2024-04-25T00:01:18.491498+0000","last_unstale":"2024-04-25T00:30:49.524439+0000","last_undegraded":"2024-04-25T00:30:49.524439+0000","last_fullsized":"2024-04-25T00:30:49.524439+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-04-25T00:30:49.524234+0000","last_deep_scrub":"74'4","last_deep_scrub_stamp":"2024-04-25T00:30:49.524234+0000","last_clean_scrub_stamp":"2024-04-25T00:30:49.524234+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:47:04.322897+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"74'4156","reported_seq":4284,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:37.545213+0000","last_change":"2024-04-25T00:01:46.667022+0000","last_active":"2024-04-25T00:30:37.545213+0000","last_peered":"2024-04-25T00:30:37.545213+0000","last_clean":"2024-04-25T00:30:37.545213+0000","last_became_active":"2024-04-25T00:01:46.666224+0000","last_became_peered":"2024-04-25T00:01:46.666224+0000","last_unstale":"2024-04-25T00:30:37.545213+0000","last_undegraded":"2024-04-25T00:30:37.545213+0000","last_fullsized":"2024-04-25T00:30:37.545213+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4156,"log_dups_size":0,"ondisk_log_size":4156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":7,"num_write":23008,"num_write_kb":280758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"69'4491","reported_seq":4628,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:58.442647+0000","last_change":"2024-04-25T00:30:58.442711+0000","last_active":"2024-04-25T00:30:58.442647+0000","last_peered":"2024-04-25T00:30:58.442647+0000","last_clean":"2024-04-25T00:30:58.442647+0000","last_became_active":"2024-04-25T00:01:05.366817+0000","last_became_peered":"2024-04-25T00:01:05.366817+0000","last_unstale":"2024-04-25T00:30:58.442647+0000","last_undegraded":"2024-04-25T00:30:58.442647+0000","last_fullsized":"2024-04-25T00:30:58.442647+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'4491","last_scrub_stamp":"2024-04-25T00:30:58.442421+0000","last_deep_scrub":"69'4491","last_deep_scrub_stamp":"2024-04-25T00:30:58.442421+0000","last_clean_scrub_stamp":"2024-04-25T00:30:58.442421+0000","objects_scrubbed":0,"log_size":4491,"log_dups_size":0,"ondisk_log_size":4491,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:45:13.565273+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24298,"num_write_kb":330898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"69'5","reported_seq":136,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:18.032303+0000","last_change":"2024-04-25T00:01:04.361900+0000","last_active":"2024-04-25T00:30:18.032303+0000","last_peered":"2024-04-25T00:30:18.032303+0000","last_clean":"2024-04-25T00:30:18.032303+0000","last_became_active":"2024-04-25T00:01:04.361572+0000","last_became_peered":"2024-04-25T00:01:04.361572+0000","last_unstale":"2024-04-25T00:30:18.032303+0000","last_undegraded":"2024-04-25T00:30:18.032303+0000","last_fullsized":"2024-04-25T00:30:18.032303+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"74'2","reported_seq":133,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:47.486020+0000","last_change":"2024-04-25T00:30:47.486143+0000","last_active":"2024-04-25T00:30:47.486020+0000","last_peered":"2024-04-25T00:30:47.486020+0000","last_clean":"2024-04-25T00:30:47.486020+0000","last_became_active":"2024-04-25T00:01:18.503379+0000","last_became_peered":"2024-04-25T00:01:18.503379+0000","last_unstale":"2024-04-25T00:30:47.486020+0000","last_undegraded":"2024-04-25T00:30:47.486020+0000","last_fullsized":"2024-04-25T00:30:47.486020+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'2","last_scrub_stamp":"2024-04-25T00:30:47.485766+0000","last_deep_scrub":"74'2","last_deep_scrub_stamp":"2024-04-25T00:30:47.485766+0000","last_clean_scrub_stamp":"2024-04-25T00:30:47.485766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:32:03.027257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"74'5402","reported_seq":5534,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:00.744521+0000","last_change":"2024-04-25T00:31:00.744586+0000","last_active":"2024-04-25T00:31:00.744521+0000","last_peered":"2024-04-25T00:31:00.744521+0000","last_clean":"2024-04-25T00:31:00.744521+0000","last_became_active":"2024-04-25T00:01:19.531145+0000","last_became_peered":"2024-04-25T00:01:19.531145+0000","last_unstale":"2024-04-25T00:31:00.744521+0000","last_undegraded":"2024-04-25T00:31:00.744521+0000","last_fullsized":"2024-04-25T00:31:00.744521+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5402","last_scrub_stamp":"2024-04-25T00:31:00.744322+0000","last_deep_scrub":"74'5402","last_deep_scrub_stamp":"2024-04-25T00:31:00.744322+0000","last_clean_scrub_stamp":"2024-04-25T00:31:00.744322+0000","objects_scrubbed":0,"log_size":5402,"log_dups_size":0,"ondisk_log_size":5402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:19:42.143195+0000","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":13,"num_write":29532,"num_write_kb":356063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"70'3335","reported_seq":3472,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:02.722805+0000","last_change":"2024-04-25T00:31:02.722871+0000","last_active":"2024-04-25T00:31:02.722805+0000","last_peered":"2024-04-25T00:31:02.722805+0000","last_clean":"2024-04-25T00:31:02.722805+0000","last_became_active":"2024-04-25T00:01:05.353556+0000","last_became_peered":"2024-04-25T00:01:05.353556+0000","last_unstale":"2024-04-25T00:31:02.722805+0000","last_undegraded":"2024-04-25T00:31:02.722805+0000","last_fullsized":"2024-04-25T00:31:02.722805+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'3335","last_scrub_stamp":"2024-04-25T00:31:02.722594+0000","last_deep_scrub":"70'3335","last_deep_scrub_stamp":"2024-04-25T00:31:02.722594+0000","last_clean_scrub_stamp":"2024-04-25T00:31:02.722594+0000","objects_scrubbed":0,"log_size":3335,"log_dups_size":0,"ondisk_log_size":3335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:23:27.901635+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18377,"num_write_kb":241696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"74'53","reported_seq":261,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:50.855125+0000","last_change":"2024-04-25T00:30:50.855191+0000","last_active":"2024-04-25T00:30:50.855125+0000","last_peered":"2024-04-25T00:30:50.855125+0000","last_clean":"2024-04-25T00:30:50.855125+0000","last_became_active":"2024-04-25T00:01:04.372185+0000","last_became_peered":"2024-04-25T00:01:04.372185+0000","last_unstale":"2024-04-25T00:30:50.855125+0000","last_undegraded":"2024-04-25T00:30:50.855125+0000","last_fullsized":"2024-04-25T00:30:50.855125+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'53","last_scrub_stamp":"2024-04-25T00:30:50.854930+0000","last_deep_scrub":"74'53","last_deep_scrub_stamp":"2024-04-25T00:30:50.854930+0000","last_clean_scrub_stamp":"2024-04-25T00:30:50.854930+0000","objects_scrubbed":1,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:43:20.976912+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88686,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":69,"num_read_kb":117,"num_write":46,"num_write_kb":117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"74'10","reported_seq":186,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:51.639688+0000","last_change":"2024-04-25T00:30:51.639756+0000","last_active":"2024-04-25T00:30:51.639688+0000","last_peered":"2024-04-25T00:30:51.639688+0000","last_clean":"2024-04-25T00:30:51.639688+0000","last_became_active":"2024-04-25T00:01:18.501448+0000","last_became_peered":"2024-04-25T00:01:18.501448+0000","last_unstale":"2024-04-25T00:30:51.639688+0000","last_undegraded":"2024-04-25T00:30:51.639688+0000","last_fullsized":"2024-04-25T00:30:51.639688+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-04-25T00:30:51.639483+0000","last_deep_scrub":"74'10","last_deep_scrub_stamp":"2024-04-25T00:30:51.639483+0000","last_clean_scrub_stamp":"2024-04-25T00:30:51.639483+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:41:45.674299+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":8,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"74'3337","reported_seq":3457,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.776247+0000","last_change":"2024-04-25T00:01:19.517510+0000","last_active":"2024-04-25T00:30:26.776247+0000","last_peered":"2024-04-25T00:30:26.776247+0000","last_clean":"2024-04-25T00:30:26.776247+0000","last_became_active":"2024-04-25T00:01:19.517243+0000","last_became_peered":"2024-04-25T00:01:19.517243+0000","last_unstale":"2024-04-25T00:30:26.776247+0000","last_undegraded":"2024-04-25T00:30:26.776247+0000","last_fullsized":"2024-04-25T00:30:26.776247+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3337,"log_dups_size":0,"ondisk_log_size":3337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18533,"num_write_kb":242287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"69'3723","reported_seq":3861,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:57.521542+0000","last_change":"2024-04-25T00:30:57.521626+0000","last_active":"2024-04-25T00:30:57.521542+0000","last_peered":"2024-04-25T00:30:57.521542+0000","last_clean":"2024-04-25T00:30:57.521542+0000","last_became_active":"2024-04-25T00:01:05.369987+0000","last_became_peered":"2024-04-25T00:01:05.369987+0000","last_unstale":"2024-04-25T00:30:57.521542+0000","last_undegraded":"2024-04-25T00:30:57.521542+0000","last_fullsized":"2024-04-25T00:30:57.521542+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'3723","last_scrub_stamp":"2024-04-25T00:30:57.521329+0000","last_deep_scrub":"69'3723","last_deep_scrub_stamp":"2024-04-25T00:30:57.521329+0000","last_clean_scrub_stamp":"2024-04-25T00:30:57.521329+0000","objects_scrubbed":0,"log_size":3723,"log_dups_size":0,"ondisk_log_size":3723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:12:05.343805+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20208,"num_write_kb":269376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"73'13","reported_seq":144,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.805104+0000","last_change":"2024-04-25T00:01:04.370226+0000","last_active":"2024-04-25T00:30:15.805104+0000","last_peered":"2024-04-25T00:30:15.805104+0000","last_clean":"2024-04-25T00:30:15.805104+0000","last_became_active":"2024-04-25T00:01:04.369904+0000","last_became_peered":"2024-04-25T00:01:04.369904+0000","last_unstale":"2024-04-25T00:30:15.805104+0000","last_undegraded":"2024-04-25T00:30:15.805104+0000","last_fullsized":"2024-04-25T00:30:15.805104+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"74'487","reported_seq":640,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:54.579270+0000","last_change":"2024-04-25T00:30:54.579389+0000","last_active":"2024-04-25T00:30:54.579270+0000","last_peered":"2024-04-25T00:30:54.579270+0000","last_clean":"2024-04-25T00:30:54.579270+0000","last_became_active":"2024-04-25T00:01:18.495573+0000","last_became_peered":"2024-04-25T00:01:18.495573+0000","last_unstale":"2024-04-25T00:30:54.579270+0000","last_undegraded":"2024-04-25T00:30:54.579270+0000","last_fullsized":"2024-04-25T00:30:54.579270+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'487","last_scrub_stamp":"2024-04-25T00:30:54.579039+0000","last_deep_scrub":"74'487","last_deep_scrub_stamp":"2024-04-25T00:30:54.579039+0000","last_clean_scrub_stamp":"2024-04-25T00:30:54.579039+0000","objects_scrubbed":1,"log_size":487,"log_dups_size":0,"ondisk_log_size":487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:26:01.969169+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":46,"num_read_kb":69,"num_write":1147,"num_write_kb":382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"74'5079","reported_seq":5201,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:33.622823+0000","last_change":"2024-04-25T00:01:19.519411+0000","last_active":"2024-04-25T00:30:33.622823+0000","last_peered":"2024-04-25T00:30:33.622823+0000","last_clean":"2024-04-25T00:30:33.622823+0000","last_became_active":"2024-04-25T00:01:19.519020+0000","last_became_peered":"2024-04-25T00:01:19.519020+0000","last_unstale":"2024-04-25T00:30:33.622823+0000","last_undegraded":"2024-04-25T00:30:33.622823+0000","last_fullsized":"2024-04-25T00:30:33.622823+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":5079,"log_dups_size":0,"ondisk_log_size":5079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":14,"num_write":27774,"num_write_kb":353080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"70'3676","reported_seq":3813,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:55.887299+0000","last_change":"2024-04-25T00:30:55.887422+0000","last_active":"2024-04-25T00:30:55.887299+0000","last_peered":"2024-04-25T00:30:55.887299+0000","last_clean":"2024-04-25T00:30:55.887299+0000","last_became_active":"2024-04-25T00:01:05.364272+0000","last_became_peered":"2024-04-25T00:01:05.364272+0000","last_unstale":"2024-04-25T00:30:55.887299+0000","last_undegraded":"2024-04-25T00:30:55.887299+0000","last_fullsized":"2024-04-25T00:30:55.887299+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'3676","last_scrub_stamp":"2024-04-25T00:30:55.887065+0000","last_deep_scrub":"70'3676","last_deep_scrub_stamp":"2024-04-25T00:30:55.887065+0000","last_clean_scrub_stamp":"2024-04-25T00:30:55.887065+0000","objects_scrubbed":0,"log_size":3676,"log_dups_size":0,"ondisk_log_size":3676,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:24:50.050373+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19925,"num_write_kb":256146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"69'3","reported_seq":134,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:18.032320+0000","last_change":"2024-04-25T00:01:04.357028+0000","last_active":"2024-04-25T00:30:18.032320+0000","last_peered":"2024-04-25T00:30:18.032320+0000","last_clean":"2024-04-25T00:30:18.032320+0000","last_became_active":"2024-04-25T00:01:04.356644+0000","last_became_peered":"2024-04-25T00:01:04.356644+0000","last_unstale":"2024-04-25T00:30:18.032320+0000","last_undegraded":"2024-04-25T00:30:18.032320+0000","last_fullsized":"2024-04-25T00:30:18.032320+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'119","reported_seq":298,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:45.080295+0000","last_change":"2024-04-25T00:30:45.080295+0000","last_active":"2024-04-25T00:30:45.080295+0000","last_peered":"2024-04-25T00:30:45.080295+0000","last_clean":"2024-04-25T00:30:45.080295+0000","last_became_active":"2024-04-25T00:00:55.083493+0000","last_became_peered":"2024-04-25T00:00:55.083493+0000","last_unstale":"2024-04-25T00:30:45.080295+0000","last_undegraded":"2024-04-25T00:30:45.080295+0000","last_fullsized":"2024-04-25T00:30:45.080295+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'119","last_scrub_stamp":"2024-04-25T00:30:45.079892+0000","last_deep_scrub":"74'119","last_deep_scrub_stamp":"2024-04-25T00:30:45.079892+0000","last_clean_scrub_stamp":"2024-04-25T00:30:45.079892+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:23:08.387129+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"74'2","reported_seq":133,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:48.259823+0000","last_change":"2024-04-25T00:30:48.259891+0000","last_active":"2024-04-25T00:30:48.259823+0000","last_peered":"2024-04-25T00:30:48.259823+0000","last_clean":"2024-04-25T00:30:48.259823+0000","last_became_active":"2024-04-25T00:01:18.475898+0000","last_became_peered":"2024-04-25T00:01:18.475898+0000","last_unstale":"2024-04-25T00:30:48.259823+0000","last_undegraded":"2024-04-25T00:30:48.259823+0000","last_fullsized":"2024-04-25T00:30:48.259823+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'2","last_scrub_stamp":"2024-04-25T00:30:48.259598+0000","last_deep_scrub":"74'2","last_deep_scrub_stamp":"2024-04-25T00:30:48.259598+0000","last_clean_scrub_stamp":"2024-04-25T00:30:48.259598+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:11:56.082075+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"74'3757","reported_seq":3878,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:38.002910+0000","last_change":"2024-04-25T00:01:19.518102+0000","last_active":"2024-04-25T00:30:38.002910+0000","last_peered":"2024-04-25T00:30:38.002910+0000","last_clean":"2024-04-25T00:30:38.002910+0000","last_became_active":"2024-04-25T00:01:19.517376+0000","last_became_peered":"2024-04-25T00:01:19.517376+0000","last_unstale":"2024-04-25T00:30:38.002910+0000","last_undegraded":"2024-04-25T00:30:38.002910+0000","last_fullsized":"2024-04-25T00:30:38.002910+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3757,"log_dups_size":0,"ondisk_log_size":3757,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":2,"num_write":20920,"num_write_kb":243770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"70'4367","reported_seq":4504,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:49.280546+0000","last_change":"2024-04-25T00:30:49.280608+0000","last_active":"2024-04-25T00:30:49.280546+0000","last_peered":"2024-04-25T00:30:49.280546+0000","last_clean":"2024-04-25T00:30:49.280546+0000","last_became_active":"2024-04-25T00:01:05.375881+0000","last_became_peered":"2024-04-25T00:01:05.375881+0000","last_unstale":"2024-04-25T00:30:49.280546+0000","last_undegraded":"2024-04-25T00:30:49.280546+0000","last_fullsized":"2024-04-25T00:30:49.280546+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'4367","last_scrub_stamp":"2024-04-25T00:30:49.280321+0000","last_deep_scrub":"70'4367","last_deep_scrub_stamp":"2024-04-25T00:30:49.280321+0000","last_clean_scrub_stamp":"2024-04-25T00:30:49.280321+0000","objects_scrubbed":0,"log_size":4367,"log_dups_size":0,"ondisk_log_size":4367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:40:55.553629+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":2,"num_write":23695,"num_write_kb":308143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"70'6","reported_seq":144,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:53.550425+0000","last_change":"2024-04-25T00:30:53.550490+0000","last_active":"2024-04-25T00:30:53.550425+0000","last_peered":"2024-04-25T00:30:53.550425+0000","last_clean":"2024-04-25T00:30:53.550425+0000","last_became_active":"2024-04-25T00:01:04.367903+0000","last_became_peered":"2024-04-25T00:01:04.367903+0000","last_unstale":"2024-04-25T00:30:53.550425+0000","last_undegraded":"2024-04-25T00:30:53.550425+0000","last_fullsized":"2024-04-25T00:30:53.550425+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'6","last_scrub_stamp":"2024-04-25T00:30:53.550214+0000","last_deep_scrub":"70'6","last_deep_scrub_stamp":"2024-04-25T00:30:53.550214+0000","last_clean_scrub_stamp":"2024-04-25T00:30:53.550214+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:20:11.988688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"74'3","reported_seq":133,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:50.476522+0000","last_change":"2024-04-25T00:30:50.476589+0000","last_active":"2024-04-25T00:30:50.476522+0000","last_peered":"2024-04-25T00:30:50.476522+0000","last_clean":"2024-04-25T00:30:50.476522+0000","last_became_active":"2024-04-25T00:01:18.476247+0000","last_became_peered":"2024-04-25T00:01:18.476247+0000","last_unstale":"2024-04-25T00:30:50.476522+0000","last_undegraded":"2024-04-25T00:30:50.476522+0000","last_fullsized":"2024-04-25T00:30:50.476522+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'3","last_scrub_stamp":"2024-04-25T00:30:50.476304+0000","last_deep_scrub":"74'3","last_deep_scrub_stamp":"2024-04-25T00:30:50.476304+0000","last_clean_scrub_stamp":"2024-04-25T00:30:50.476304+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:37.264544+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"74'4936","reported_seq":5057,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.753308+0000","last_change":"2024-04-25T00:01:19.514615+0000","last_active":"2024-04-25T00:30:26.753308+0000","last_peered":"2024-04-25T00:30:26.753308+0000","last_clean":"2024-04-25T00:30:26.753308+0000","last_became_active":"2024-04-25T00:01:19.513640+0000","last_became_peered":"2024-04-25T00:01:19.513640+0000","last_unstale":"2024-04-25T00:30:26.753308+0000","last_undegraded":"2024-04-25T00:30:26.753308+0000","last_fullsized":"2024-04-25T00:30:26.753308+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4936,"log_dups_size":0,"ondisk_log_size":4936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":27403,"num_write_kb":306834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"70'4466","reported_seq":4602,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:47.353563+0000","last_change":"2024-04-25T00:30:47.353608+0000","last_active":"2024-04-25T00:30:47.353563+0000","last_peered":"2024-04-25T00:30:47.353563+0000","last_clean":"2024-04-25T00:30:47.353563+0000","last_became_active":"2024-04-25T00:01:05.384822+0000","last_became_peered":"2024-04-25T00:01:05.384822+0000","last_unstale":"2024-04-25T00:30:47.353563+0000","last_undegraded":"2024-04-25T00:30:47.353563+0000","last_fullsized":"2024-04-25T00:30:47.353563+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'4466","last_scrub_stamp":"2024-04-25T00:30:47.353447+0000","last_deep_scrub":"70'4466","last_deep_scrub_stamp":"2024-04-25T00:30:47.353447+0000","last_clean_scrub_stamp":"2024-04-25T00:30:47.353447+0000","objects_scrubbed":0,"log_size":4466,"log_dups_size":0,"ondisk_log_size":4466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:44:18.999377+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23831,"num_write_kb":316890,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"74'7","reported_seq":168,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:58.656057+0000","last_change":"2024-04-25T00:30:58.656124+0000","last_active":"2024-04-25T00:30:58.656057+0000","last_peered":"2024-04-25T00:30:58.656057+0000","last_clean":"2024-04-25T00:30:58.656057+0000","last_became_active":"2024-04-25T00:01:04.354419+0000","last_became_peered":"2024-04-25T00:01:04.354419+0000","last_unstale":"2024-04-25T00:30:58.656057+0000","last_undegraded":"2024-04-25T00:30:58.656057+0000","last_fullsized":"2024-04-25T00:30:58.656057+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'7","last_scrub_stamp":"2024-04-25T00:30:58.655859+0000","last_deep_scrub":"74'7","last_deep_scrub_stamp":"2024-04-25T00:30:58.655859+0000","last_clean_scrub_stamp":"2024-04-25T00:30:58.655859+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:18:55.837676+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'1455","reported_seq":1637,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:52.580874+0000","last_change":"2024-04-25T00:30:52.580950+0000","last_active":"2024-04-25T00:30:52.580874+0000","last_peered":"2024-04-25T00:30:52.580874+0000","last_clean":"2024-04-25T00:30:52.580874+0000","last_became_active":"2024-04-25T00:01:18.489629+0000","last_became_peered":"2024-04-25T00:01:18.489629+0000","last_unstale":"2024-04-25T00:30:52.580874+0000","last_undegraded":"2024-04-25T00:30:52.580874+0000","last_fullsized":"2024-04-25T00:30:52.580874+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1455","last_scrub_stamp":"2024-04-25T00:30:52.580648+0000","last_deep_scrub":"74'1455","last_deep_scrub_stamp":"2024-04-25T00:30:52.580648+0000","last_clean_scrub_stamp":"2024-04-25T00:30:52.580648+0000","objects_scrubbed":2,"log_size":1455,"log_dups_size":0,"ondisk_log_size":1455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:06:03.147140+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":75,"num_read_kb":41971,"num_write":1427,"num_write_kb":4503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'3796","reported_seq":3924,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:55.186739+0000","last_change":"2024-04-25T00:30:55.186823+0000","last_active":"2024-04-25T00:30:55.186739+0000","last_peered":"2024-04-25T00:30:55.186739+0000","last_clean":"2024-04-25T00:30:55.186739+0000","last_became_active":"2024-04-25T00:01:19.507867+0000","last_became_peered":"2024-04-25T00:01:19.507867+0000","last_unstale":"2024-04-25T00:30:55.186739+0000","last_undegraded":"2024-04-25T00:30:55.186739+0000","last_fullsized":"2024-04-25T00:30:55.186739+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'3796","last_scrub_stamp":"2024-04-25T00:30:55.186403+0000","last_deep_scrub":"74'3796","last_deep_scrub_stamp":"2024-04-25T00:30:55.186403+0000","last_clean_scrub_stamp":"2024-04-25T00:30:55.186403+0000","objects_scrubbed":0,"log_size":3796,"log_dups_size":0,"ondisk_log_size":3796,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:21:11.806761+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20867,"num_write_kb":276131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"70'5324","reported_seq":5461,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:49.364228+0000","last_change":"2024-04-25T00:30:49.364288+0000","last_active":"2024-04-25T00:30:49.364228+0000","last_peered":"2024-04-25T00:30:49.364228+0000","last_clean":"2024-04-25T00:30:49.364228+0000","last_became_active":"2024-04-25T00:01:05.378112+0000","last_became_peered":"2024-04-25T00:01:05.378112+0000","last_unstale":"2024-04-25T00:30:49.364228+0000","last_undegraded":"2024-04-25T00:30:49.364228+0000","last_fullsized":"2024-04-25T00:30:49.364228+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'5324","last_scrub_stamp":"2024-04-25T00:30:49.364024+0000","last_deep_scrub":"70'5324","last_deep_scrub_stamp":"2024-04-25T00:30:49.364024+0000","last_clean_scrub_stamp":"2024-04-25T00:30:49.364024+0000","objects_scrubbed":0,"log_size":5324,"log_dups_size":0,"ondisk_log_size":5324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:25:16.580358+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":28190,"num_write_kb":387378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"74'8","reported_seq":192,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:00.891145+0000","last_change":"2024-04-25T00:31:00.891211+0000","last_active":"2024-04-25T00:31:00.891145+0000","last_peered":"2024-04-25T00:31:00.891145+0000","last_clean":"2024-04-25T00:31:00.891145+0000","last_became_active":"2024-04-25T00:01:04.370099+0000","last_became_peered":"2024-04-25T00:01:04.370099+0000","last_unstale":"2024-04-25T00:31:00.891145+0000","last_undegraded":"2024-04-25T00:31:00.891145+0000","last_fullsized":"2024-04-25T00:31:00.891145+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-04-25T00:31:00.890959+0000","last_deep_scrub":"74'8","last_deep_scrub_stamp":"2024-04-25T00:31:00.890959+0000","last_clean_scrub_stamp":"2024-04-25T00:31:00.890959+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:55.122756+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"74'5","reported_seq":127,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.742425+0000","last_change":"2024-04-25T00:01:18.475256+0000","last_active":"2024-04-25T00:30:26.742425+0000","last_peered":"2024-04-25T00:30:26.742425+0000","last_clean":"2024-04-25T00:30:26.742425+0000","last_became_active":"2024-04-25T00:01:18.474952+0000","last_became_peered":"2024-04-25T00:01:18.474952+0000","last_unstale":"2024-04-25T00:30:26.742425+0000","last_undegraded":"2024-04-25T00:30:26.742425+0000","last_fullsized":"2024-04-25T00:30:26.742425+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:17.440296+0000","last_clean_scrub_stamp":"2024-04-25T00:01:17.440296+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"74'4048","reported_seq":4168,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.363896+0000","last_change":"2024-04-25T00:01:19.516833+0000","last_active":"2024-04-25T00:30:26.363896+0000","last_peered":"2024-04-25T00:30:26.363896+0000","last_clean":"2024-04-25T00:30:26.363896+0000","last_became_active":"2024-04-25T00:01:19.515926+0000","last_became_peered":"2024-04-25T00:01:19.515926+0000","last_unstale":"2024-04-25T00:30:26.363896+0000","last_undegraded":"2024-04-25T00:30:26.363896+0000","last_fullsized":"2024-04-25T00:30:26.363896+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4048,"log_dups_size":0,"ondisk_log_size":4048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23468,"num_write_kb":270462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"70'3273","reported_seq":3410,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:00.689741+0000","last_change":"2024-04-25T00:31:00.689803+0000","last_active":"2024-04-25T00:31:00.689741+0000","last_peered":"2024-04-25T00:31:00.689741+0000","last_clean":"2024-04-25T00:31:00.689741+0000","last_became_active":"2024-04-25T00:01:05.371204+0000","last_became_peered":"2024-04-25T00:01:05.371204+0000","last_unstale":"2024-04-25T00:31:00.689741+0000","last_undegraded":"2024-04-25T00:31:00.689741+0000","last_fullsized":"2024-04-25T00:31:00.689741+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'3273","last_scrub_stamp":"2024-04-25T00:31:00.689495+0000","last_deep_scrub":"70'3273","last_deep_scrub_stamp":"2024-04-25T00:31:00.689495+0000","last_clean_scrub_stamp":"2024-04-25T00:31:00.689495+0000","objects_scrubbed":0,"log_size":3273,"log_dups_size":0,"ondisk_log_size":3273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:09:14.491669+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":17647,"num_write_kb":246988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"74'5","reported_seq":166,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:57.386592+0000","last_change":"2024-04-25T00:30:57.386653+0000","last_active":"2024-04-25T00:30:57.386592+0000","last_peered":"2024-04-25T00:30:57.386592+0000","last_clean":"2024-04-25T00:30:57.386592+0000","last_became_active":"2024-04-25T00:01:04.372040+0000","last_became_peered":"2024-04-25T00:01:04.372040+0000","last_unstale":"2024-04-25T00:30:57.386592+0000","last_undegraded":"2024-04-25T00:30:57.386592+0000","last_fullsized":"2024-04-25T00:30:57.386592+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-04-25T00:30:57.386400+0000","last_deep_scrub":"74'5","last_deep_scrub_stamp":"2024-04-25T00:30:57.386400+0000","last_clean_scrub_stamp":"2024-04-25T00:30:57.386400+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:45:01.205083+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"74'2","reported_seq":132,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:52.083336+0000","last_change":"2024-04-25T00:30:52.083398+0000","last_active":"2024-04-25T00:30:52.083336+0000","last_peered":"2024-04-25T00:30:52.083336+0000","last_clean":"2024-04-25T00:30:52.083336+0000","last_became_active":"2024-04-25T00:01:18.487227+0000","last_became_peered":"2024-04-25T00:01:18.487227+0000","last_unstale":"2024-04-25T00:30:52.083336+0000","last_undegraded":"2024-04-25T00:30:52.083336+0000","last_fullsized":"2024-04-25T00:30:52.083336+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'2","last_scrub_stamp":"2024-04-25T00:30:52.083145+0000","last_deep_scrub":"74'2","last_deep_scrub_stamp":"2024-04-25T00:30:52.083145+0000","last_clean_scrub_stamp":"2024-04-25T00:30:52.083145+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:32:07.624636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"74'4306","reported_seq":4428,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.764052+0000","last_change":"2024-04-25T00:01:19.528574+0000","last_active":"2024-04-25T00:30:26.764052+0000","last_peered":"2024-04-25T00:30:26.764052+0000","last_clean":"2024-04-25T00:30:26.764052+0000","last_became_active":"2024-04-25T00:01:19.525531+0000","last_became_peered":"2024-04-25T00:01:19.525531+0000","last_unstale":"2024-04-25T00:30:26.764052+0000","last_undegraded":"2024-04-25T00:30:26.764052+0000","last_fullsized":"2024-04-25T00:30:26.764052+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4306,"log_dups_size":0,"ondisk_log_size":4306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":31,"num_write":22571,"num_write_kb":279217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"70'3794","reported_seq":3929,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:58.227884+0000","last_change":"2024-04-25T00:30:58.228007+0000","last_active":"2024-04-25T00:30:58.227884+0000","last_peered":"2024-04-25T00:30:58.227884+0000","last_clean":"2024-04-25T00:30:58.227884+0000","last_became_active":"2024-04-25T00:01:05.363320+0000","last_became_peered":"2024-04-25T00:01:05.363320+0000","last_unstale":"2024-04-25T00:30:58.227884+0000","last_undegraded":"2024-04-25T00:30:58.227884+0000","last_fullsized":"2024-04-25T00:30:58.227884+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'3794","last_scrub_stamp":"2024-04-25T00:30:58.227621+0000","last_deep_scrub":"70'3794","last_deep_scrub_stamp":"2024-04-25T00:30:58.227621+0000","last_clean_scrub_stamp":"2024-04-25T00:30:58.227621+0000","objects_scrubbed":0,"log_size":3794,"log_dups_size":0,"ondisk_log_size":3794,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:19:57.260701+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20831,"num_write_kb":276143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"74'19","reported_seq":270,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:57.927120+0000","last_change":"2024-04-25T00:30:57.927194+0000","last_active":"2024-04-25T00:30:57.927120+0000","last_peered":"2024-04-25T00:30:57.927120+0000","last_clean":"2024-04-25T00:30:57.927120+0000","last_became_active":"2024-04-25T00:01:04.357571+0000","last_became_peered":"2024-04-25T00:01:04.357571+0000","last_unstale":"2024-04-25T00:30:57.927120+0000","last_undegraded":"2024-04-25T00:30:57.927120+0000","last_fullsized":"2024-04-25T00:30:57.927120+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-04-25T00:30:57.926899+0000","last_deep_scrub":"74'19","last_deep_scrub_stamp":"2024-04-25T00:30:57.926899+0000","last_clean_scrub_stamp":"2024-04-25T00:30:57.926899+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:13:19.179171+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":342,"num_read_kb":342,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"74'3","reported_seq":132,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:56.622780+0000","last_change":"2024-04-25T00:30:56.622912+0000","last_active":"2024-04-25T00:30:56.622780+0000","last_peered":"2024-04-25T00:30:56.622780+0000","last_clean":"2024-04-25T00:30:56.622780+0000","last_became_active":"2024-04-25T00:01:18.496489+0000","last_became_peered":"2024-04-25T00:01:18.496489+0000","last_unstale":"2024-04-25T00:30:56.622780+0000","last_undegraded":"2024-04-25T00:30:56.622780+0000","last_fullsized":"2024-04-25T00:30:56.622780+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'3","last_scrub_stamp":"2024-04-25T00:30:56.622511+0000","last_deep_scrub":"74'3","last_deep_scrub_stamp":"2024-04-25T00:30:56.622511+0000","last_clean_scrub_stamp":"2024-04-25T00:30:56.622511+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:18:21.041592+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"74'4849","reported_seq":4979,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:46.353474+0000","last_change":"2024-04-25T00:30:46.353525+0000","last_active":"2024-04-25T00:30:46.353474+0000","last_peered":"2024-04-25T00:30:46.353474+0000","last_clean":"2024-04-25T00:30:46.353474+0000","last_became_active":"2024-04-25T00:01:19.496805+0000","last_became_peered":"2024-04-25T00:01:19.496805+0000","last_unstale":"2024-04-25T00:30:46.353474+0000","last_undegraded":"2024-04-25T00:30:46.353474+0000","last_fullsized":"2024-04-25T00:30:46.353474+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4849","last_scrub_stamp":"2024-04-25T00:30:46.353342+0000","last_deep_scrub":"74'4849","last_deep_scrub_stamp":"2024-04-25T00:30:46.353342+0000","last_clean_scrub_stamp":"2024-04-25T00:30:46.353342+0000","objects_scrubbed":0,"log_size":4849,"log_dups_size":0,"ondisk_log_size":4849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:10:14.560315+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":16,"num_write":26442,"num_write_kb":335585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"70'2974","reported_seq":3109,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:48.040269+0000","last_change":"2024-04-25T00:30:48.040334+0000","last_active":"2024-04-25T00:30:48.040269+0000","last_peered":"2024-04-25T00:30:48.040269+0000","last_clean":"2024-04-25T00:30:48.040269+0000","last_became_active":"2024-04-25T00:01:05.373077+0000","last_became_peered":"2024-04-25T00:01:05.373077+0000","last_unstale":"2024-04-25T00:30:48.040269+0000","last_undegraded":"2024-04-25T00:30:48.040269+0000","last_fullsized":"2024-04-25T00:30:48.040269+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2974","last_scrub_stamp":"2024-04-25T00:30:48.040059+0000","last_deep_scrub":"70'2974","last_deep_scrub_stamp":"2024-04-25T00:30:48.040059+0000","last_clean_scrub_stamp":"2024-04-25T00:30:48.040059+0000","objects_scrubbed":0,"log_size":2974,"log_dups_size":0,"ondisk_log_size":2974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:24:53.452964+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15895,"num_write_kb":231645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"70'5","reported_seq":142,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:52.357346+0000","last_change":"2024-04-25T00:30:52.357411+0000","last_active":"2024-04-25T00:30:52.357346+0000","last_peered":"2024-04-25T00:30:52.357346+0000","last_clean":"2024-04-25T00:30:52.357346+0000","last_became_active":"2024-04-25T00:01:04.357004+0000","last_became_peered":"2024-04-25T00:01:04.357004+0000","last_unstale":"2024-04-25T00:30:52.357346+0000","last_undegraded":"2024-04-25T00:30:52.357346+0000","last_fullsized":"2024-04-25T00:30:52.357346+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'5","last_scrub_stamp":"2024-04-25T00:30:52.357144+0000","last_deep_scrub":"70'5","last_deep_scrub_stamp":"2024-04-25T00:30:52.357144+0000","last_clean_scrub_stamp":"2024-04-25T00:30:52.357144+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:12:00.179621+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"74'5","reported_seq":134,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:50.571384+0000","last_change":"2024-04-25T00:30:50.571444+0000","last_active":"2024-04-25T00:30:50.571384+0000","last_peered":"2024-04-25T00:30:50.571384+0000","last_clean":"2024-04-25T00:30:50.571384+0000","last_became_active":"2024-04-25T00:01:18.486279+0000","last_became_peered":"2024-04-25T00:01:18.486279+0000","last_unstale":"2024-04-25T00:30:50.571384+0000","last_undegraded":"2024-04-25T00:30:50.571384+0000","last_fullsized":"2024-04-25T00:30:50.571384+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-04-25T00:30:50.571181+0000","last_deep_scrub":"74'5","last_deep_scrub_stamp":"2024-04-25T00:30:50.571181+0000","last_clean_scrub_stamp":"2024-04-25T00:30:50.571181+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:25:17.787515+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"74'4040","reported_seq":4159,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.631172+0000","last_change":"2024-04-25T00:01:19.516827+0000","last_active":"2024-04-25T00:30:26.631172+0000","last_peered":"2024-04-25T00:30:26.631172+0000","last_clean":"2024-04-25T00:30:26.631172+0000","last_became_active":"2024-04-25T00:01:19.515765+0000","last_became_peered":"2024-04-25T00:01:19.515765+0000","last_unstale":"2024-04-25T00:30:26.631172+0000","last_undegraded":"2024-04-25T00:30:26.631172+0000","last_fullsized":"2024-04-25T00:30:26.631172+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4040,"log_dups_size":0,"ondisk_log_size":4040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23288,"num_write_kb":269674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"70'4514","reported_seq":4649,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:53.240060+0000","last_change":"2024-04-25T00:30:53.240122+0000","last_active":"2024-04-25T00:30:53.240060+0000","last_peered":"2024-04-25T00:30:53.240060+0000","last_clean":"2024-04-25T00:30:53.240060+0000","last_became_active":"2024-04-25T00:01:05.378644+0000","last_became_peered":"2024-04-25T00:01:05.378644+0000","last_unstale":"2024-04-25T00:30:53.240060+0000","last_undegraded":"2024-04-25T00:30:53.240060+0000","last_fullsized":"2024-04-25T00:30:53.240060+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'4514","last_scrub_stamp":"2024-04-25T00:30:53.239822+0000","last_deep_scrub":"70'4514","last_deep_scrub_stamp":"2024-04-25T00:30:53.239822+0000","last_clean_scrub_stamp":"2024-04-25T00:30:53.239822+0000","objects_scrubbed":0,"log_size":4514,"log_dups_size":0,"ondisk_log_size":4514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:18:50.421639+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25321,"num_write_kb":326961,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"70'3","reported_seq":132,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.809953+0000","last_change":"2024-04-25T00:01:04.360987+0000","last_active":"2024-04-25T00:30:15.809953+0000","last_peered":"2024-04-25T00:30:15.809953+0000","last_clean":"2024-04-25T00:30:15.809953+0000","last_became_active":"2024-04-25T00:01:04.360664+0000","last_became_peered":"2024-04-25T00:01:04.360664+0000","last_unstale":"2024-04-25T00:30:15.809953+0000","last_undegraded":"2024-04-25T00:30:15.809953+0000","last_fullsized":"2024-04-25T00:30:15.809953+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"74'338","reported_seq":584,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:01.552063+0000","last_change":"2024-04-25T00:31:01.552189+0000","last_active":"2024-04-25T00:31:01.552063+0000","last_peered":"2024-04-25T00:31:01.552063+0000","last_clean":"2024-04-25T00:31:01.552063+0000","last_became_active":"2024-04-25T00:01:18.504963+0000","last_became_peered":"2024-04-25T00:01:18.504963+0000","last_unstale":"2024-04-25T00:31:01.552063+0000","last_undegraded":"2024-04-25T00:31:01.552063+0000","last_fullsized":"2024-04-25T00:31:01.552063+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'338","last_scrub_stamp":"2024-04-25T00:31:01.551839+0000","last_deep_scrub":"74'338","last_deep_scrub_stamp":"2024-04-25T00:31:01.551839+0000","last_clean_scrub_stamp":"2024-04-25T00:31:01.551839+0000","objects_scrubbed":4,"log_size":338,"log_dups_size":0,"ondisk_log_size":338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:21:33.109775+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":209,"num_read_kb":90302,"num_write":331,"num_write_kb":4275,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"74'4674","reported_seq":4790,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.774711+0000","last_change":"2024-04-25T00:01:46.658442+0000","last_active":"2024-04-25T00:30:26.774711+0000","last_peered":"2024-04-25T00:30:26.774711+0000","last_clean":"2024-04-25T00:30:26.774711+0000","last_became_active":"2024-04-25T00:01:46.657588+0000","last_became_peered":"2024-04-25T00:01:46.657588+0000","last_unstale":"2024-04-25T00:30:26.774711+0000","last_undegraded":"2024-04-25T00:30:26.774711+0000","last_fullsized":"2024-04-25T00:30:26.774711+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4674,"log_dups_size":0,"ondisk_log_size":4674,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":16,"num_write":25396,"num_write_kb":335454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"70'3777","reported_seq":3912,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:00.549419+0000","last_change":"2024-04-25T00:31:00.549565+0000","last_active":"2024-04-25T00:31:00.549419+0000","last_peered":"2024-04-25T00:31:00.549419+0000","last_clean":"2024-04-25T00:31:00.549419+0000","last_became_active":"2024-04-25T00:01:05.377625+0000","last_became_peered":"2024-04-25T00:01:05.377625+0000","last_unstale":"2024-04-25T00:31:00.549419+0000","last_undegraded":"2024-04-25T00:31:00.549419+0000","last_fullsized":"2024-04-25T00:31:00.549419+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'3777","last_scrub_stamp":"2024-04-25T00:31:00.549184+0000","last_deep_scrub":"70'3777","last_deep_scrub_stamp":"2024-04-25T00:31:00.549184+0000","last_clean_scrub_stamp":"2024-04-25T00:31:00.549184+0000","objects_scrubbed":0,"log_size":3777,"log_dups_size":0,"ondisk_log_size":3777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:46:52.031744+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20730,"num_write_kb":283926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"74'11","reported_seq":194,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:49.875799+0000","last_change":"2024-04-25T00:30:49.875871+0000","last_active":"2024-04-25T00:30:49.875799+0000","last_peered":"2024-04-25T00:30:49.875799+0000","last_clean":"2024-04-25T00:30:49.875799+0000","last_became_active":"2024-04-25T00:01:04.370031+0000","last_became_peered":"2024-04-25T00:01:04.370031+0000","last_unstale":"2024-04-25T00:30:49.875799+0000","last_undegraded":"2024-04-25T00:30:49.875799+0000","last_fullsized":"2024-04-25T00:30:49.875799+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-04-25T00:30:49.875609+0000","last_deep_scrub":"74'11","last_deep_scrub_stamp":"2024-04-25T00:30:49.875609+0000","last_clean_scrub_stamp":"2024-04-25T00:30:49.875609+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:43:52.774202+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":184,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"74'33","reported_seq":231,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:01.203820+0000","last_change":"2024-04-25T00:31:01.203909+0000","last_active":"2024-04-25T00:31:01.203820+0000","last_peered":"2024-04-25T00:31:01.203820+0000","last_clean":"2024-04-25T00:31:01.203820+0000","last_became_active":"2024-04-25T00:01:18.472931+0000","last_became_peered":"2024-04-25T00:01:18.472931+0000","last_unstale":"2024-04-25T00:31:01.203820+0000","last_undegraded":"2024-04-25T00:31:01.203820+0000","last_fullsized":"2024-04-25T00:31:01.203820+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'33","last_scrub_stamp":"2024-04-25T00:31:01.203595+0000","last_deep_scrub":"74'33","last_deep_scrub_stamp":"2024-04-25T00:31:01.203595+0000","last_clean_scrub_stamp":"2024-04-25T00:31:01.203595+0000","objects_scrubbed":2,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:49:21.809560+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":161,"num_read_kb":161,"num_write":32,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"74'5042","reported_seq":5162,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.774023+0000","last_change":"2024-04-25T00:01:19.533664+0000","last_active":"2024-04-25T00:30:26.774023+0000","last_peered":"2024-04-25T00:30:26.774023+0000","last_clean":"2024-04-25T00:30:26.774023+0000","last_became_active":"2024-04-25T00:01:19.533048+0000","last_became_peered":"2024-04-25T00:01:19.533048+0000","last_unstale":"2024-04-25T00:30:26.774023+0000","last_undegraded":"2024-04-25T00:30:26.774023+0000","last_fullsized":"2024-04-25T00:30:26.774023+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":5042,"log_dups_size":0,"ondisk_log_size":5042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":28058,"num_write_kb":350341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"69'3549","reported_seq":3684,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:45.362286+0000","last_change":"2024-04-25T00:30:45.362327+0000","last_active":"2024-04-25T00:30:45.362286+0000","last_peered":"2024-04-25T00:30:45.362286+0000","last_clean":"2024-04-25T00:30:45.362286+0000","last_became_active":"2024-04-25T00:01:05.385258+0000","last_became_peered":"2024-04-25T00:01:05.385258+0000","last_unstale":"2024-04-25T00:30:45.362286+0000","last_undegraded":"2024-04-25T00:30:45.362286+0000","last_fullsized":"2024-04-25T00:30:45.362286+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'3549","last_scrub_stamp":"2024-04-25T00:30:45.362153+0000","last_deep_scrub":"69'3549","last_deep_scrub_stamp":"2024-04-25T00:30:45.362153+0000","last_clean_scrub_stamp":"2024-04-25T00:30:45.362153+0000","objects_scrubbed":0,"log_size":3549,"log_dups_size":0,"ondisk_log_size":3549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:21:01.982510+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19806,"num_write_kb":251506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"74'345","reported_seq":598,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:55.626243+0000","last_change":"2024-04-25T00:30:55.626314+0000","last_active":"2024-04-25T00:30:55.626243+0000","last_peered":"2024-04-25T00:30:55.626243+0000","last_clean":"2024-04-25T00:30:55.626243+0000","last_became_active":"2024-04-25T00:01:04.364930+0000","last_became_peered":"2024-04-25T00:01:04.364930+0000","last_unstale":"2024-04-25T00:30:55.626243+0000","last_undegraded":"2024-04-25T00:30:55.626243+0000","last_fullsized":"2024-04-25T00:30:55.626243+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'345","last_scrub_stamp":"2024-04-25T00:30:55.626025+0000","last_deep_scrub":"74'345","last_deep_scrub_stamp":"2024-04-25T00:30:55.626025+0000","last_clean_scrub_stamp":"2024-04-25T00:30:55.626025+0000","objects_scrubbed":4,"log_size":345,"log_dups_size":0,"ondisk_log_size":345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:41:01.899333+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":208,"num_read_kb":94392,"num_write":336,"num_write_kb":4266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"74'3","reported_seq":132,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:48.766670+0000","last_change":"2024-04-25T00:30:48.766729+0000","last_active":"2024-04-25T00:30:48.766670+0000","last_peered":"2024-04-25T00:30:48.766670+0000","last_clean":"2024-04-25T00:30:48.766670+0000","last_became_active":"2024-04-25T00:01:18.492850+0000","last_became_peered":"2024-04-25T00:01:18.492850+0000","last_unstale":"2024-04-25T00:30:48.766670+0000","last_undegraded":"2024-04-25T00:30:48.766670+0000","last_fullsized":"2024-04-25T00:30:48.766670+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'3","last_scrub_stamp":"2024-04-25T00:30:48.766478+0000","last_deep_scrub":"74'3","last_deep_scrub_stamp":"2024-04-25T00:30:48.766478+0000","last_clean_scrub_stamp":"2024-04-25T00:30:48.766478+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:34:48.730022+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"74'4133","reported_seq":4253,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.774271+0000","last_change":"2024-04-25T00:01:19.515725+0000","last_active":"2024-04-25T00:30:26.774271+0000","last_peered":"2024-04-25T00:30:26.774271+0000","last_clean":"2024-04-25T00:30:26.774271+0000","last_became_active":"2024-04-25T00:01:19.509285+0000","last_became_peered":"2024-04-25T00:01:19.509285+0000","last_unstale":"2024-04-25T00:30:26.774271+0000","last_undegraded":"2024-04-25T00:30:26.774271+0000","last_fullsized":"2024-04-25T00:30:26.774271+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4133,"log_dups_size":0,"ondisk_log_size":4133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":4,"num_write":22847,"num_write_kb":280058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"70'5270","reported_seq":5405,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:54.260418+0000","last_change":"2024-04-25T00:30:54.260483+0000","last_active":"2024-04-25T00:30:54.260418+0000","last_peered":"2024-04-25T00:30:54.260418+0000","last_clean":"2024-04-25T00:30:54.260418+0000","last_became_active":"2024-04-25T00:01:05.369950+0000","last_became_peered":"2024-04-25T00:01:05.369950+0000","last_unstale":"2024-04-25T00:30:54.260418+0000","last_undegraded":"2024-04-25T00:30:54.260418+0000","last_fullsized":"2024-04-25T00:30:54.260418+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'5270","last_scrub_stamp":"2024-04-25T00:30:54.260111+0000","last_deep_scrub":"70'5270","last_deep_scrub_stamp":"2024-04-25T00:30:54.260111+0000","last_clean_scrub_stamp":"2024-04-25T00:30:54.260111+0000","objects_scrubbed":0,"log_size":5270,"log_dups_size":0,"ondisk_log_size":5270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:54:55.960816+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28224,"num_write_kb":363198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"74'32","reported_seq":215,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:48.886523+0000","last_change":"2024-04-25T00:30:48.886594+0000","last_active":"2024-04-25T00:30:48.886523+0000","last_peered":"2024-04-25T00:30:48.886523+0000","last_clean":"2024-04-25T00:30:48.886523+0000","last_became_active":"2024-04-25T00:01:04.365945+0000","last_became_peered":"2024-04-25T00:01:04.365945+0000","last_unstale":"2024-04-25T00:30:48.886523+0000","last_undegraded":"2024-04-25T00:30:48.886523+0000","last_fullsized":"2024-04-25T00:30:48.886523+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'32","last_scrub_stamp":"2024-04-25T00:30:48.886315+0000","last_deep_scrub":"74'32","last_deep_scrub_stamp":"2024-04-25T00:30:48.886315+0000","last_clean_scrub_stamp":"2024-04-25T00:30:48.886315+0000","objects_scrubbed":2,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:24:54.299220+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":92,"num_write":32,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"74'10","reported_seq":185,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:52.611626+0000","last_change":"2024-04-25T00:30:52.611706+0000","last_active":"2024-04-25T00:30:52.611626+0000","last_peered":"2024-04-25T00:30:52.611626+0000","last_clean":"2024-04-25T00:30:52.611626+0000","last_became_active":"2024-04-25T00:01:18.475876+0000","last_became_peered":"2024-04-25T00:01:18.475876+0000","last_unstale":"2024-04-25T00:30:52.611626+0000","last_undegraded":"2024-04-25T00:30:52.611626+0000","last_fullsized":"2024-04-25T00:30:52.611626+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-04-25T00:30:52.611425+0000","last_deep_scrub":"74'10","last_deep_scrub_stamp":"2024-04-25T00:30:52.611425+0000","last_clean_scrub_stamp":"2024-04-25T00:30:52.611425+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:54:56.416784+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":184,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"74'4233","reported_seq":4355,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.776252+0000","last_change":"2024-04-25T00:01:19.515376+0000","last_active":"2024-04-25T00:30:26.776252+0000","last_peered":"2024-04-25T00:30:26.776252+0000","last_clean":"2024-04-25T00:30:26.776252+0000","last_became_active":"2024-04-25T00:01:19.512425+0000","last_became_peered":"2024-04-25T00:01:19.512425+0000","last_unstale":"2024-04-25T00:30:26.776252+0000","last_undegraded":"2024-04-25T00:30:26.776252+0000","last_fullsized":"2024-04-25T00:30:26.776252+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4233,"log_dups_size":0,"ondisk_log_size":4233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":24291,"num_write_kb":290168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"70'4413","reported_seq":4540,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.801053+0000","last_change":"2024-04-25T00:01:05.361723+0000","last_active":"2024-04-25T00:30:15.801053+0000","last_peered":"2024-04-25T00:30:15.801053+0000","last_clean":"2024-04-25T00:30:15.801053+0000","last_became_active":"2024-04-25T00:01:05.360956+0000","last_became_peered":"2024-04-25T00:01:05.360956+0000","last_unstale":"2024-04-25T00:30:15.801053+0000","last_undegraded":"2024-04-25T00:30:15.801053+0000","last_fullsized":"2024-04-25T00:30:15.801053+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4413,"log_dups_size":0,"ondisk_log_size":4413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22903,"num_write_kb":321461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"70'3","reported_seq":140,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:54.287654+0000","last_change":"2024-04-25T00:30:54.287799+0000","last_active":"2024-04-25T00:30:54.287654+0000","last_peered":"2024-04-25T00:30:54.287654+0000","last_clean":"2024-04-25T00:30:54.287654+0000","last_became_active":"2024-04-25T00:01:04.361827+0000","last_became_peered":"2024-04-25T00:01:04.361827+0000","last_unstale":"2024-04-25T00:30:54.287654+0000","last_undegraded":"2024-04-25T00:30:54.287654+0000","last_fullsized":"2024-04-25T00:30:54.287654+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'3","last_scrub_stamp":"2024-04-25T00:30:54.287407+0000","last_deep_scrub":"70'3","last_deep_scrub_stamp":"2024-04-25T00:30:54.287407+0000","last_clean_scrub_stamp":"2024-04-25T00:30:54.287407+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:41:00.560714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"74'19","reported_seq":266,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:55.844441+0000","last_change":"2024-04-25T00:30:55.844512+0000","last_active":"2024-04-25T00:30:55.844441+0000","last_peered":"2024-04-25T00:30:55.844441+0000","last_clean":"2024-04-25T00:30:55.844441+0000","last_became_active":"2024-04-25T00:01:18.495415+0000","last_became_peered":"2024-04-25T00:01:18.495415+0000","last_unstale":"2024-04-25T00:30:55.844441+0000","last_undegraded":"2024-04-25T00:30:55.844441+0000","last_fullsized":"2024-04-25T00:30:55.844441+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-04-25T00:30:55.844234+0000","last_deep_scrub":"74'19","last_deep_scrub_stamp":"2024-04-25T00:30:55.844234+0000","last_clean_scrub_stamp":"2024-04-25T00:30:55.844234+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:21:12.464591+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":354,"num_read_kb":354,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"74'3474","reported_seq":3601,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:51.615755+0000","last_change":"2024-04-25T00:30:51.615816+0000","last_active":"2024-04-25T00:30:51.615755+0000","last_peered":"2024-04-25T00:30:51.615755+0000","last_clean":"2024-04-25T00:30:51.615755+0000","last_became_active":"2024-04-25T00:01:19.509015+0000","last_became_peered":"2024-04-25T00:01:19.509015+0000","last_unstale":"2024-04-25T00:30:51.615755+0000","last_undegraded":"2024-04-25T00:30:51.615755+0000","last_fullsized":"2024-04-25T00:30:51.615755+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'3474","last_scrub_stamp":"2024-04-25T00:30:51.615588+0000","last_deep_scrub":"74'3474","last_deep_scrub_stamp":"2024-04-25T00:30:51.615588+0000","last_clean_scrub_stamp":"2024-04-25T00:30:51.615588+0000","objects_scrubbed":0,"log_size":3474,"log_dups_size":0,"ondisk_log_size":3474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:10:19.822560+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19390,"num_write_kb":245789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"70'4331","reported_seq":4466,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:58.713028+0000","last_change":"2024-04-25T00:30:58.713166+0000","last_active":"2024-04-25T00:30:58.713028+0000","last_peered":"2024-04-25T00:30:58.713028+0000","last_clean":"2024-04-25T00:30:58.713028+0000","last_became_active":"2024-04-25T00:01:05.361865+0000","last_became_peered":"2024-04-25T00:01:05.361865+0000","last_unstale":"2024-04-25T00:30:58.713028+0000","last_undegraded":"2024-04-25T00:30:58.713028+0000","last_fullsized":"2024-04-25T00:30:58.713028+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'4331","last_scrub_stamp":"2024-04-25T00:30:58.712670+0000","last_deep_scrub":"70'4331","last_deep_scrub_stamp":"2024-04-25T00:30:58.712670+0000","last_clean_scrub_stamp":"2024-04-25T00:30:58.712670+0000","objects_scrubbed":0,"log_size":4331,"log_dups_size":0,"ondisk_log_size":4331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:21:30.270607+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22828,"num_write_kb":312693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"74'7","reported_seq":144,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:46.064393+0000","last_change":"2024-04-25T00:30:46.064444+0000","last_active":"2024-04-25T00:30:46.064393+0000","last_peered":"2024-04-25T00:30:46.064393+0000","last_clean":"2024-04-25T00:30:46.064393+0000","last_became_active":"2024-04-25T00:01:04.360374+0000","last_became_peered":"2024-04-25T00:01:04.360374+0000","last_unstale":"2024-04-25T00:30:46.064393+0000","last_undegraded":"2024-04-25T00:30:46.064393+0000","last_fullsized":"2024-04-25T00:30:46.064393+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'7","last_scrub_stamp":"2024-04-25T00:30:46.064254+0000","last_deep_scrub":"74'7","last_deep_scrub_stamp":"2024-04-25T00:30:46.064254+0000","last_clean_scrub_stamp":"2024-04-25T00:30:46.064254+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:08:01.206564+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"74'5","reported_seq":134,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:47.769492+0000","last_change":"2024-04-25T00:30:47.769552+0000","last_active":"2024-04-25T00:30:47.769492+0000","last_peered":"2024-04-25T00:30:47.769492+0000","last_clean":"2024-04-25T00:30:47.769492+0000","last_became_active":"2024-04-25T00:01:18.479270+0000","last_became_peered":"2024-04-25T00:01:18.479270+0000","last_unstale":"2024-04-25T00:30:47.769492+0000","last_undegraded":"2024-04-25T00:30:47.769492+0000","last_fullsized":"2024-04-25T00:30:47.769492+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-04-25T00:30:47.769275+0000","last_deep_scrub":"74'5","last_deep_scrub_stamp":"2024-04-25T00:30:47.769275+0000","last_clean_scrub_stamp":"2024-04-25T00:30:47.769275+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:32:34.378407+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"74'5681","reported_seq":5806,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:33.315419+0000","last_change":"2024-04-25T00:01:19.524463+0000","last_active":"2024-04-25T00:30:33.315419+0000","last_peered":"2024-04-25T00:30:33.315419+0000","last_clean":"2024-04-25T00:30:33.315419+0000","last_became_active":"2024-04-25T00:01:19.524166+0000","last_became_peered":"2024-04-25T00:01:19.524166+0000","last_unstale":"2024-04-25T00:30:33.315419+0000","last_undegraded":"2024-04-25T00:30:33.315419+0000","last_fullsized":"2024-04-25T00:30:33.315419+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":5681,"log_dups_size":0,"ondisk_log_size":5681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":40,"num_write":31879,"num_write_kb":410681,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"74'28","reported_seq":159,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:50.813264+0000","last_change":"2024-04-25T00:30:50.813331+0000","last_active":"2024-04-25T00:30:50.813264+0000","last_peered":"2024-04-25T00:30:50.813264+0000","last_clean":"2024-04-25T00:30:50.813264+0000","last_became_active":"2024-04-25T00:01:18.496289+0000","last_became_peered":"2024-04-25T00:01:18.496289+0000","last_unstale":"2024-04-25T00:30:50.813264+0000","last_undegraded":"2024-04-25T00:30:50.813264+0000","last_fullsized":"2024-04-25T00:30:50.813264+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'28","last_scrub_stamp":"2024-04-25T00:30:50.813048+0000","last_deep_scrub":"74'28","last_deep_scrub_stamp":"2024-04-25T00:30:50.813048+0000","last_clean_scrub_stamp":"2024-04-25T00:30:50.813048+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:06.142467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"70'4773","reported_seq":4909,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:51.855193+0000","last_change":"2024-04-25T00:30:51.855259+0000","last_active":"2024-04-25T00:30:51.855193+0000","last_peered":"2024-04-25T00:30:51.855193+0000","last_clean":"2024-04-25T00:30:51.855193+0000","last_became_active":"2024-04-25T00:01:05.368080+0000","last_became_peered":"2024-04-25T00:01:05.368080+0000","last_unstale":"2024-04-25T00:30:51.855193+0000","last_undegraded":"2024-04-25T00:30:51.855193+0000","last_fullsized":"2024-04-25T00:30:51.855193+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'4773","last_scrub_stamp":"2024-04-25T00:30:51.854990+0000","last_deep_scrub":"70'4773","last_deep_scrub_stamp":"2024-04-25T00:30:51.854990+0000","last_clean_scrub_stamp":"2024-04-25T00:30:51.854990+0000","objects_scrubbed":0,"log_size":4773,"log_dups_size":0,"ondisk_log_size":4773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:34:24.805496+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":25218,"num_write_kb":330520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"69'2","reported_seq":131,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:15.804827+0000","last_change":"2024-04-25T00:01:04.364523+0000","last_active":"2024-04-25T00:30:15.804827+0000","last_peered":"2024-04-25T00:30:15.804827+0000","last_clean":"2024-04-25T00:30:15.804827+0000","last_became_active":"2024-04-25T00:01:04.364050+0000","last_became_peered":"2024-04-25T00:01:04.364050+0000","last_unstale":"2024-04-25T00:30:15.804827+0000","last_undegraded":"2024-04-25T00:30:15.804827+0000","last_fullsized":"2024-04-25T00:30:15.804827+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"74'3390","reported_seq":3519,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:02.663874+0000","last_change":"2024-04-25T00:31:02.663989+0000","last_active":"2024-04-25T00:31:02.663874+0000","last_peered":"2024-04-25T00:31:02.663874+0000","last_clean":"2024-04-25T00:31:02.663874+0000","last_became_active":"2024-04-25T00:01:19.490755+0000","last_became_peered":"2024-04-25T00:01:19.490755+0000","last_unstale":"2024-04-25T00:31:02.663874+0000","last_undegraded":"2024-04-25T00:31:02.663874+0000","last_fullsized":"2024-04-25T00:31:02.663874+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'3390","last_scrub_stamp":"2024-04-25T00:31:02.663640+0000","last_deep_scrub":"74'3390","last_deep_scrub_stamp":"2024-04-25T00:31:02.663640+0000","last_clean_scrub_stamp":"2024-04-25T00:31:02.663640+0000","objects_scrubbed":0,"log_size":3390,"log_dups_size":0,"ondisk_log_size":3390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:45:17.786492+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18979,"num_write_kb":248305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"74'1085","reported_seq":1271,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:59.284516+0000","last_change":"2024-04-25T00:30:59.284587+0000","last_active":"2024-04-25T00:30:59.284516+0000","last_peered":"2024-04-25T00:30:59.284516+0000","last_clean":"2024-04-25T00:30:59.284516+0000","last_became_active":"2024-04-25T00:01:18.472534+0000","last_became_peered":"2024-04-25T00:01:18.472534+0000","last_unstale":"2024-04-25T00:30:59.284516+0000","last_undegraded":"2024-04-25T00:30:59.284516+0000","last_fullsized":"2024-04-25T00:30:59.284516+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1085","last_scrub_stamp":"2024-04-25T00:30:59.284321+0000","last_deep_scrub":"74'1085","last_deep_scrub_stamp":"2024-04-25T00:30:59.284321+0000","last_clean_scrub_stamp":"2024-04-25T00:30:59.284321+0000","objects_scrubbed":3,"log_size":1085,"log_dups_size":0,"ondisk_log_size":1085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:29:03.994337+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":101,"num_read_kb":11812,"num_write":1055,"num_write_kb":4728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"70'4239","reported_seq":4374,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:47.624936+0000","last_change":"2024-04-25T00:30:47.625001+0000","last_active":"2024-04-25T00:30:47.624936+0000","last_peered":"2024-04-25T00:30:47.624936+0000","last_clean":"2024-04-25T00:30:47.624936+0000","last_became_active":"2024-04-25T00:01:05.378056+0000","last_became_peered":"2024-04-25T00:01:05.378056+0000","last_unstale":"2024-04-25T00:30:47.624936+0000","last_undegraded":"2024-04-25T00:30:47.624936+0000","last_fullsized":"2024-04-25T00:30:47.624936+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'4239","last_scrub_stamp":"2024-04-25T00:30:47.624713+0000","last_deep_scrub":"70'4239","last_deep_scrub_stamp":"2024-04-25T00:30:47.624713+0000","last_clean_scrub_stamp":"2024-04-25T00:30:47.624713+0000","objects_scrubbed":0,"log_size":4239,"log_dups_size":0,"ondisk_log_size":4239,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:32:50.938659+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22560,"num_write_kb":311538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"70'3","reported_seq":140,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:54.553929+0000","last_change":"2024-04-25T00:30:54.554053+0000","last_active":"2024-04-25T00:30:54.553929+0000","last_peered":"2024-04-25T00:30:54.553929+0000","last_clean":"2024-04-25T00:30:54.553929+0000","last_became_active":"2024-04-25T00:01:04.370273+0000","last_became_peered":"2024-04-25T00:01:04.370273+0000","last_unstale":"2024-04-25T00:30:54.553929+0000","last_undegraded":"2024-04-25T00:30:54.553929+0000","last_fullsized":"2024-04-25T00:30:54.553929+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'3","last_scrub_stamp":"2024-04-25T00:30:54.553687+0000","last_deep_scrub":"70'3","last_deep_scrub_stamp":"2024-04-25T00:30:54.553687+0000","last_clean_scrub_stamp":"2024-04-25T00:30:54.553687+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:52:34.663317+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"74'5148","reported_seq":5267,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.763478+0000","last_change":"2024-04-25T00:01:19.521143+0000","last_active":"2024-04-25T00:30:26.763478+0000","last_peered":"2024-04-25T00:30:26.763478+0000","last_clean":"2024-04-25T00:30:26.763478+0000","last_became_active":"2024-04-25T00:01:19.514664+0000","last_became_peered":"2024-04-25T00:01:19.514664+0000","last_unstale":"2024-04-25T00:30:26.763478+0000","last_undegraded":"2024-04-25T00:30:26.763478+0000","last_fullsized":"2024-04-25T00:30:26.763478+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":5148,"log_dups_size":0,"ondisk_log_size":5148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":29131,"num_write_kb":338102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"74'249","reported_seq":381,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:29.596615+0000","last_change":"2024-04-25T00:01:04.360254+0000","last_active":"2024-04-25T00:30:29.596615+0000","last_peered":"2024-04-25T00:30:29.596615+0000","last_clean":"2024-04-25T00:30:29.596615+0000","last_became_active":"2024-04-25T00:01:04.359586+0000","last_became_peered":"2024-04-25T00:01:04.359586+0000","last_unstale":"2024-04-25T00:30:29.596615+0000","last_undegraded":"2024-04-25T00:30:29.596615+0000","last_fullsized":"2024-04-25T00:30:29.596615+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:03.305620+0000","last_clean_scrub_stamp":"2024-04-25T00:01:03.305620+0000","objects_scrubbed":0,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2250578,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4308,"num_write":220,"num_write_kb":2323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"69'3397","reported_seq":3532,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:49.790544+0000","last_change":"2024-04-25T00:30:49.790544+0000","last_active":"2024-04-25T00:30:49.790544+0000","last_peered":"2024-04-25T00:30:49.790544+0000","last_clean":"2024-04-25T00:30:49.790544+0000","last_became_active":"2024-04-25T00:01:05.368970+0000","last_became_peered":"2024-04-25T00:01:05.368970+0000","last_unstale":"2024-04-25T00:30:49.790544+0000","last_undegraded":"2024-04-25T00:30:49.790544+0000","last_fullsized":"2024-04-25T00:30:49.790544+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'3397","last_scrub_stamp":"2024-04-25T00:30:49.790273+0000","last_deep_scrub":"69'3397","last_deep_scrub_stamp":"2024-04-25T00:30:49.790273+0000","last_clean_scrub_stamp":"2024-04-25T00:30:49.790273+0000","objects_scrubbed":0,"log_size":3397,"log_dups_size":0,"ondisk_log_size":3397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:48:58.324291+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17978,"num_write_kb":244099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"74'6","reported_seq":180,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:52.817756+0000","last_change":"2024-04-25T00:30:52.817820+0000","last_active":"2024-04-25T00:30:52.817756+0000","last_peered":"2024-04-25T00:30:52.817756+0000","last_clean":"2024-04-25T00:30:52.817756+0000","last_became_active":"2024-04-25T00:01:18.497745+0000","last_became_peered":"2024-04-25T00:01:18.497745+0000","last_unstale":"2024-04-25T00:30:52.817756+0000","last_undegraded":"2024-04-25T00:30:52.817756+0000","last_fullsized":"2024-04-25T00:30:52.817756+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-04-25T00:30:52.817567+0000","last_deep_scrub":"74'6","last_deep_scrub_stamp":"2024-04-25T00:30:52.817567+0000","last_clean_scrub_stamp":"2024-04-25T00:30:52.817567+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:52:59.407401+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"74'3919","reported_seq":4040,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.627773+0000","last_change":"2024-04-25T00:01:19.514919+0000","last_active":"2024-04-25T00:30:26.627773+0000","last_peered":"2024-04-25T00:30:26.627773+0000","last_clean":"2024-04-25T00:30:26.627773+0000","last_became_active":"2024-04-25T00:01:19.514280+0000","last_became_peered":"2024-04-25T00:01:19.514280+0000","last_unstale":"2024-04-25T00:30:26.627773+0000","last_undegraded":"2024-04-25T00:30:26.627773+0000","last_fullsized":"2024-04-25T00:30:26.627773+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3919,"log_dups_size":0,"ondisk_log_size":3919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":20874,"num_write_kb":280510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"74'753","reported_seq":917,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:01.765146+0000","last_change":"2024-04-25T00:31:01.765222+0000","last_active":"2024-04-25T00:31:01.765146+0000","last_peered":"2024-04-25T00:31:01.765146+0000","last_clean":"2024-04-25T00:31:01.765146+0000","last_became_active":"2024-04-25T00:01:04.364788+0000","last_became_peered":"2024-04-25T00:01:04.364788+0000","last_unstale":"2024-04-25T00:31:01.765146+0000","last_undegraded":"2024-04-25T00:31:01.765146+0000","last_fullsized":"2024-04-25T00:31:01.765146+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'753","last_scrub_stamp":"2024-04-25T00:31:01.764916+0000","last_deep_scrub":"74'753","last_deep_scrub_stamp":"2024-04-25T00:31:01.764916+0000","last_clean_scrub_stamp":"2024-04-25T00:31:01.764916+0000","objects_scrubbed":1,"log_size":753,"log_dups_size":0,"ondisk_log_size":753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:20:09.601906+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":28,"num_read_kb":89132,"num_write":742,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"69'3790","reported_seq":3924,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:49.069214+0000","last_change":"2024-04-25T00:30:49.069275+0000","last_active":"2024-04-25T00:30:49.069214+0000","last_peered":"2024-04-25T00:30:49.069214+0000","last_clean":"2024-04-25T00:30:49.069214+0000","last_became_active":"2024-04-25T00:01:05.373857+0000","last_became_peered":"2024-04-25T00:01:05.373857+0000","last_unstale":"2024-04-25T00:30:49.069214+0000","last_undegraded":"2024-04-25T00:30:49.069214+0000","last_fullsized":"2024-04-25T00:30:49.069214+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'3790","last_scrub_stamp":"2024-04-25T00:30:49.068995+0000","last_deep_scrub":"69'3790","last_deep_scrub_stamp":"2024-04-25T00:30:49.068995+0000","last_clean_scrub_stamp":"2024-04-25T00:30:49.068995+0000","objects_scrubbed":0,"log_size":3790,"log_dups_size":0,"ondisk_log_size":3790,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:53:14.653828+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19857,"num_write_kb":263976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"74'49","reported_seq":223,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:56.807593+0000","last_change":"2024-04-25T00:30:56.807741+0000","last_active":"2024-04-25T00:30:56.807593+0000","last_peered":"2024-04-25T00:30:56.807593+0000","last_clean":"2024-04-25T00:30:56.807593+0000","last_became_active":"2024-04-25T00:01:18.501785+0000","last_became_peered":"2024-04-25T00:01:18.501785+0000","last_unstale":"2024-04-25T00:30:56.807593+0000","last_undegraded":"2024-04-25T00:30:56.807593+0000","last_fullsized":"2024-04-25T00:30:56.807593+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'49","last_scrub_stamp":"2024-04-25T00:30:56.807395+0000","last_deep_scrub":"74'49","last_deep_scrub_stamp":"2024-04-25T00:30:56.807395+0000","last_clean_scrub_stamp":"2024-04-25T00:30:56.807395+0000","objects_scrubbed":1,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:41:50.842211+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.26","version":"74'3694","reported_seq":3822,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:05.017336+0000","last_change":"2024-04-25T00:31:05.017367+0000","last_active":"2024-04-25T00:31:05.017336+0000","last_peered":"2024-04-25T00:31:05.017336+0000","last_clean":"2024-04-25T00:31:05.017336+0000","last_became_active":"2024-04-25T00:01:19.510155+0000","last_became_peered":"2024-04-25T00:01:19.510155+0000","last_unstale":"2024-04-25T00:31:05.017336+0000","last_undegraded":"2024-04-25T00:31:05.017336+0000","last_fullsized":"2024-04-25T00:31:05.017336+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'3694","last_scrub_stamp":"2024-04-25T00:31:05.017219+0000","last_deep_scrub":"74'3694","last_deep_scrub_stamp":"2024-04-25T00:31:05.017219+0000","last_clean_scrub_stamp":"2024-04-25T00:31:05.017219+0000","objects_scrubbed":0,"log_size":3694,"log_dups_size":0,"ondisk_log_size":3694,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:02.885084+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20834,"num_write_kb":259761,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.20","version":"70'4043","reported_seq":4180,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:02.742699+0000","last_change":"2024-04-25T00:31:02.742765+0000","last_active":"2024-04-25T00:31:02.742699+0000","last_peered":"2024-04-25T00:31:02.742699+0000","last_clean":"2024-04-25T00:31:02.742699+0000","last_became_active":"2024-04-25T00:01:05.377742+0000","last_became_peered":"2024-04-25T00:01:05.377742+0000","last_unstale":"2024-04-25T00:31:02.742699+0000","last_undegraded":"2024-04-25T00:31:02.742699+0000","last_fullsized":"2024-04-25T00:31:02.742699+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'4043","last_scrub_stamp":"2024-04-25T00:31:02.742482+0000","last_deep_scrub":"70'4043","last_deep_scrub_stamp":"2024-04-25T00:31:02.742482+0000","last_clean_scrub_stamp":"2024-04-25T00:31:02.742482+0000","objects_scrubbed":0,"log_size":4043,"log_dups_size":0,"ondisk_log_size":4043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:18:08.904434+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21543,"num_write_kb":291017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"74'5759","reported_seq":5888,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:54.978481+0000","last_change":"2024-04-25T00:30:54.978525+0000","last_active":"2024-04-25T00:30:54.978481+0000","last_peered":"2024-04-25T00:30:54.978481+0000","last_clean":"2024-04-25T00:30:54.978481+0000","last_became_active":"2024-04-25T00:01:19.507960+0000","last_became_peered":"2024-04-25T00:01:19.507960+0000","last_unstale":"2024-04-25T00:30:54.978481+0000","last_undegraded":"2024-04-25T00:30:54.978481+0000","last_fullsized":"2024-04-25T00:30:54.978481+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5759","last_scrub_stamp":"2024-04-25T00:30:54.978351+0000","last_deep_scrub":"74'5759","last_deep_scrub_stamp":"2024-04-25T00:30:54.978351+0000","last_clean_scrub_stamp":"2024-04-25T00:30:54.978351+0000","objects_scrubbed":0,"log_size":5759,"log_dups_size":0,"ondisk_log_size":5759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:46:55.394949+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":31168,"num_write_kb":366885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.21","version":"70'4068","reported_seq":4206,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:53.848253+0000","last_change":"2024-04-25T00:30:53.848317+0000","last_active":"2024-04-25T00:30:53.848253+0000","last_peered":"2024-04-25T00:30:53.848253+0000","last_clean":"2024-04-25T00:30:53.848253+0000","last_became_active":"2024-04-25T00:01:05.372050+0000","last_became_peered":"2024-04-25T00:01:05.372050+0000","last_unstale":"2024-04-25T00:30:53.848253+0000","last_undegraded":"2024-04-25T00:30:53.848253+0000","last_fullsized":"2024-04-25T00:30:53.848253+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'4068","last_scrub_stamp":"2024-04-25T00:30:53.848045+0000","last_deep_scrub":"70'4068","last_deep_scrub_stamp":"2024-04-25T00:30:53.848045+0000","last_clean_scrub_stamp":"2024-04-25T00:30:53.848045+0000","objects_scrubbed":0,"log_size":4068,"log_dups_size":0,"ondisk_log_size":4068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:34:00.986900+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21791,"num_write_kb":292243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.24","version":"74'4315","reported_seq":4437,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.758116+0000","last_change":"2024-04-25T00:01:19.531072+0000","last_active":"2024-04-25T00:30:26.758116+0000","last_peered":"2024-04-25T00:30:26.758116+0000","last_clean":"2024-04-25T00:30:26.758116+0000","last_became_active":"2024-04-25T00:01:19.530836+0000","last_became_peered":"2024-04-25T00:01:19.530836+0000","last_unstale":"2024-04-25T00:30:26.758116+0000","last_undegraded":"2024-04-25T00:30:26.758116+0000","last_fullsized":"2024-04-25T00:30:26.758116+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4315,"log_dups_size":0,"ondisk_log_size":4315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":23956,"num_write_kb":309408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"70'4267","reported_seq":4404,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:00.236552+0000","last_change":"2024-04-25T00:31:00.236697+0000","last_active":"2024-04-25T00:31:00.236552+0000","last_peered":"2024-04-25T00:31:00.236552+0000","last_clean":"2024-04-25T00:31:00.236552+0000","last_became_active":"2024-04-25T00:01:05.361067+0000","last_became_peered":"2024-04-25T00:01:05.361067+0000","last_unstale":"2024-04-25T00:31:00.236552+0000","last_undegraded":"2024-04-25T00:31:00.236552+0000","last_fullsized":"2024-04-25T00:31:00.236552+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'4267","last_scrub_stamp":"2024-04-25T00:31:00.236260+0000","last_deep_scrub":"70'4267","last_deep_scrub_stamp":"2024-04-25T00:31:00.236260+0000","last_clean_scrub_stamp":"2024-04-25T00:31:00.236260+0000","objects_scrubbed":0,"log_size":4267,"log_dups_size":0,"ondisk_log_size":4267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:53:08.751636+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23473,"num_write_kb":311317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"74'4634","reported_seq":4755,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.737837+0000","last_change":"2024-04-25T00:01:19.506029+0000","last_active":"2024-04-25T00:30:26.737837+0000","last_peered":"2024-04-25T00:30:26.737837+0000","last_clean":"2024-04-25T00:30:26.737837+0000","last_became_active":"2024-04-25T00:01:19.505184+0000","last_became_peered":"2024-04-25T00:01:19.505184+0000","last_unstale":"2024-04-25T00:30:26.737837+0000","last_undegraded":"2024-04-25T00:30:26.737837+0000","last_fullsized":"2024-04-25T00:30:26.737837+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4634,"log_dups_size":0,"ondisk_log_size":4634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":6,"num_write":26896,"num_write_kb":316898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"70'4083","reported_seq":4212,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:23.000311+0000","last_change":"2024-04-25T00:01:05.358690+0000","last_active":"2024-04-25T00:30:23.000311+0000","last_peered":"2024-04-25T00:30:23.000311+0000","last_clean":"2024-04-25T00:30:23.000311+0000","last_became_active":"2024-04-25T00:01:05.357340+0000","last_became_peered":"2024-04-25T00:01:05.357340+0000","last_unstale":"2024-04-25T00:30:23.000311+0000","last_undegraded":"2024-04-25T00:30:23.000311+0000","last_fullsized":"2024-04-25T00:30:23.000311+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4083,"log_dups_size":0,"ondisk_log_size":4083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21935,"num_write_kb":296816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.22","version":"74'5147","reported_seq":5281,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:02.460999+0000","last_change":"2024-04-25T00:31:02.461078+0000","last_active":"2024-04-25T00:31:02.460999+0000","last_peered":"2024-04-25T00:31:02.460999+0000","last_clean":"2024-04-25T00:31:02.460999+0000","last_became_active":"2024-04-25T00:01:19.508123+0000","last_became_peered":"2024-04-25T00:01:19.508123+0000","last_unstale":"2024-04-25T00:31:02.460999+0000","last_undegraded":"2024-04-25T00:31:02.460999+0000","last_fullsized":"2024-04-25T00:31:02.460999+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5147","last_scrub_stamp":"2024-04-25T00:31:02.460752+0000","last_deep_scrub":"74'5147","last_deep_scrub_stamp":"2024-04-25T00:31:02.460752+0000","last_clean_scrub_stamp":"2024-04-25T00:31:02.460752+0000","objects_scrubbed":0,"log_size":5147,"log_dups_size":0,"ondisk_log_size":5147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:07:27.061158+0000","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":43,"num_write":27901,"num_write_kb":341497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"70'3848","reported_seq":3985,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:50.583648+0000","last_change":"2024-04-25T00:30:50.583713+0000","last_active":"2024-04-25T00:30:50.583648+0000","last_peered":"2024-04-25T00:30:50.583648+0000","last_clean":"2024-04-25T00:30:50.583648+0000","last_became_active":"2024-04-25T00:01:05.386724+0000","last_became_peered":"2024-04-25T00:01:05.386724+0000","last_unstale":"2024-04-25T00:30:50.583648+0000","last_undegraded":"2024-04-25T00:30:50.583648+0000","last_fullsized":"2024-04-25T00:30:50.583648+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'3848","last_scrub_stamp":"2024-04-25T00:30:50.583438+0000","last_deep_scrub":"70'3848","last_deep_scrub_stamp":"2024-04-25T00:30:50.583438+0000","last_clean_scrub_stamp":"2024-04-25T00:30:50.583438+0000","objects_scrubbed":0,"log_size":3848,"log_dups_size":0,"ondisk_log_size":3848,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:21:07.203795+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20848,"num_write_kb":276248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.23","version":"74'4386","reported_seq":4509,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:33.622764+0000","last_change":"2024-04-25T00:01:19.522760+0000","last_active":"2024-04-25T00:30:33.622764+0000","last_peered":"2024-04-25T00:30:33.622764+0000","last_clean":"2024-04-25T00:30:33.622764+0000","last_became_active":"2024-04-25T00:01:19.522380+0000","last_became_peered":"2024-04-25T00:01:19.522380+0000","last_unstale":"2024-04-25T00:30:33.622764+0000","last_undegraded":"2024-04-25T00:30:33.622764+0000","last_fullsized":"2024-04-25T00:30:33.622764+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4386,"log_dups_size":0,"ondisk_log_size":4386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":23395,"num_write_kb":294532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.25","version":"70'4349","reported_seq":4485,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:56.535525+0000","last_change":"2024-04-25T00:30:56.535583+0000","last_active":"2024-04-25T00:30:56.535525+0000","last_peered":"2024-04-25T00:30:56.535525+0000","last_clean":"2024-04-25T00:30:56.535525+0000","last_became_active":"2024-04-25T00:01:05.363838+0000","last_became_peered":"2024-04-25T00:01:05.363838+0000","last_unstale":"2024-04-25T00:30:56.535525+0000","last_undegraded":"2024-04-25T00:30:56.535525+0000","last_fullsized":"2024-04-25T00:30:56.535525+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'4349","last_scrub_stamp":"2024-04-25T00:30:56.535318+0000","last_deep_scrub":"70'4349","last_deep_scrub_stamp":"2024-04-25T00:30:56.535318+0000","last_clean_scrub_stamp":"2024-04-25T00:30:56.535318+0000","objects_scrubbed":0,"log_size":4349,"log_dups_size":0,"ondisk_log_size":4349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:06:07.101810+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24353,"num_write_kb":306782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"74'4505","reported_seq":4626,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:33.034557+0000","last_change":"2024-04-25T00:01:19.510902+0000","last_active":"2024-04-25T00:30:33.034557+0000","last_peered":"2024-04-25T00:30:33.034557+0000","last_clean":"2024-04-25T00:30:33.034557+0000","last_became_active":"2024-04-25T00:01:19.510008+0000","last_became_peered":"2024-04-25T00:01:19.510008+0000","last_unstale":"2024-04-25T00:30:33.034557+0000","last_undegraded":"2024-04-25T00:30:33.034557+0000","last_fullsized":"2024-04-25T00:30:33.034557+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4505,"log_dups_size":0,"ondisk_log_size":4505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24621,"num_write_kb":315853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"70'3623","reported_seq":3759,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:59.737559+0000","last_change":"2024-04-25T00:30:59.737624+0000","last_active":"2024-04-25T00:30:59.737559+0000","last_peered":"2024-04-25T00:30:59.737559+0000","last_clean":"2024-04-25T00:30:59.737559+0000","last_became_active":"2024-04-25T00:01:05.361119+0000","last_became_peered":"2024-04-25T00:01:05.361119+0000","last_unstale":"2024-04-25T00:30:59.737559+0000","last_undegraded":"2024-04-25T00:30:59.737559+0000","last_fullsized":"2024-04-25T00:30:59.737559+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'3623","last_scrub_stamp":"2024-04-25T00:30:59.737329+0000","last_deep_scrub":"70'3623","last_deep_scrub_stamp":"2024-04-25T00:30:59.737329+0000","last_clean_scrub_stamp":"2024-04-25T00:30:59.737329+0000","objects_scrubbed":0,"log_size":3623,"log_dups_size":0,"ondisk_log_size":3623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:45:14.860181+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20289,"num_write_kb":274005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.21","version":"74'4291","reported_seq":4415,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:33.315270+0000","last_change":"2024-04-25T00:01:19.518534+0000","last_active":"2024-04-25T00:30:33.315270+0000","last_peered":"2024-04-25T00:30:33.315270+0000","last_clean":"2024-04-25T00:30:33.315270+0000","last_became_active":"2024-04-25T00:01:19.518110+0000","last_became_peered":"2024-04-25T00:01:19.518110+0000","last_unstale":"2024-04-25T00:30:33.315270+0000","last_undegraded":"2024-04-25T00:30:33.315270+0000","last_fullsized":"2024-04-25T00:30:33.315270+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4291,"log_dups_size":0,"ondisk_log_size":4291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":23037,"num_write_kb":301131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"69'4983","reported_seq":5119,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:49.613538+0000","last_change":"2024-04-25T00:30:49.613603+0000","last_active":"2024-04-25T00:30:49.613538+0000","last_peered":"2024-04-25T00:30:49.613538+0000","last_clean":"2024-04-25T00:30:49.613538+0000","last_became_active":"2024-04-25T00:01:05.372611+0000","last_became_peered":"2024-04-25T00:01:05.372611+0000","last_unstale":"2024-04-25T00:30:49.613538+0000","last_undegraded":"2024-04-25T00:30:49.613538+0000","last_fullsized":"2024-04-25T00:30:49.613538+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'4983","last_scrub_stamp":"2024-04-25T00:30:49.613328+0000","last_deep_scrub":"69'4983","last_deep_scrub_stamp":"2024-04-25T00:30:49.613328+0000","last_clean_scrub_stamp":"2024-04-25T00:30:49.613328+0000","objects_scrubbed":0,"log_size":4983,"log_dups_size":0,"ondisk_log_size":4983,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:34:06.955379+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26776,"num_write_kb":349057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2e","version":"74'4567","reported_seq":4699,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:01.752451+0000","last_change":"2024-04-25T00:31:01.752514+0000","last_active":"2024-04-25T00:31:01.752451+0000","last_peered":"2024-04-25T00:31:01.752451+0000","last_clean":"2024-04-25T00:31:01.752451+0000","last_became_active":"2024-04-25T00:01:19.541877+0000","last_became_peered":"2024-04-25T00:01:19.541877+0000","last_unstale":"2024-04-25T00:31:01.752451+0000","last_undegraded":"2024-04-25T00:31:01.752451+0000","last_fullsized":"2024-04-25T00:31:01.752451+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4567","last_scrub_stamp":"2024-04-25T00:31:01.752205+0000","last_deep_scrub":"74'4567","last_deep_scrub_stamp":"2024-04-25T00:31:01.752205+0000","last_clean_scrub_stamp":"2024-04-25T00:31:01.752205+0000","objects_scrubbed":0,"log_size":4567,"log_dups_size":0,"ondisk_log_size":4567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:06:12.318696+0000","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":36,"num_write":24879,"num_write_kb":312940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.28","version":"70'4294","reported_seq":4430,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:50.283328+0000","last_change":"2024-04-25T00:30:50.283393+0000","last_active":"2024-04-25T00:30:50.283328+0000","last_peered":"2024-04-25T00:30:50.283328+0000","last_clean":"2024-04-25T00:30:50.283328+0000","last_became_active":"2024-04-25T00:01:05.372148+0000","last_became_peered":"2024-04-25T00:01:05.372148+0000","last_unstale":"2024-04-25T00:30:50.283328+0000","last_undegraded":"2024-04-25T00:30:50.283328+0000","last_fullsized":"2024-04-25T00:30:50.283328+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'4294","last_scrub_stamp":"2024-04-25T00:30:50.283109+0000","last_deep_scrub":"70'4294","last_deep_scrub_stamp":"2024-04-25T00:30:50.283109+0000","last_clean_scrub_stamp":"2024-04-25T00:30:50.283109+0000","objects_scrubbed":0,"log_size":4294,"log_dups_size":0,"ondisk_log_size":4294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:45.764661+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23157,"num_write_kb":289708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2f","version":"74'5194","reported_seq":5322,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:47.306861+0000","last_change":"2024-04-25T00:30:47.306894+0000","last_active":"2024-04-25T00:30:47.306861+0000","last_peered":"2024-04-25T00:30:47.306861+0000","last_clean":"2024-04-25T00:30:47.306861+0000","last_became_active":"2024-04-25T00:01:19.496310+0000","last_became_peered":"2024-04-25T00:01:19.496310+0000","last_unstale":"2024-04-25T00:30:47.306861+0000","last_undegraded":"2024-04-25T00:30:47.306861+0000","last_fullsized":"2024-04-25T00:30:47.306861+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5194","last_scrub_stamp":"2024-04-25T00:30:47.306735+0000","last_deep_scrub":"74'5194","last_deep_scrub_stamp":"2024-04-25T00:30:47.306735+0000","last_clean_scrub_stamp":"2024-04-25T00:30:47.306735+0000","objects_scrubbed":0,"log_size":5194,"log_dups_size":0,"ondisk_log_size":5194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:05:57.873227+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":28420,"num_write_kb":367977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"69'4077","reported_seq":4205,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:23.000290+0000","last_change":"2024-04-25T00:01:05.366340+0000","last_active":"2024-04-25T00:30:23.000290+0000","last_peered":"2024-04-25T00:30:23.000290+0000","last_clean":"2024-04-25T00:30:23.000290+0000","last_became_active":"2024-04-25T00:01:05.361952+0000","last_became_peered":"2024-04-25T00:01:05.361952+0000","last_unstale":"2024-04-25T00:30:23.000290+0000","last_undegraded":"2024-04-25T00:30:23.000290+0000","last_fullsized":"2024-04-25T00:30:23.000290+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":4077,"log_dups_size":0,"ondisk_log_size":4077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22153,"num_write_kb":297857,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2c","version":"74'3564","reported_seq":3685,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:37.732820+0000","last_change":"2024-04-25T00:01:19.508257+0000","last_active":"2024-04-25T00:30:37.732820+0000","last_peered":"2024-04-25T00:30:37.732820+0000","last_clean":"2024-04-25T00:30:37.732820+0000","last_became_active":"2024-04-25T00:01:19.507516+0000","last_became_peered":"2024-04-25T00:01:19.507516+0000","last_unstale":"2024-04-25T00:30:37.732820+0000","last_undegraded":"2024-04-25T00:30:37.732820+0000","last_fullsized":"2024-04-25T00:30:37.732820+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3564,"log_dups_size":0,"ondisk_log_size":3564,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":19793,"num_write_kb":247542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"69'4229","reported_seq":4365,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:51.097972+0000","last_change":"2024-04-25T00:30:51.098034+0000","last_active":"2024-04-25T00:30:51.097972+0000","last_peered":"2024-04-25T00:30:51.097972+0000","last_clean":"2024-04-25T00:30:51.097972+0000","last_became_active":"2024-04-25T00:01:05.386919+0000","last_became_peered":"2024-04-25T00:01:05.386919+0000","last_unstale":"2024-04-25T00:30:51.097972+0000","last_undegraded":"2024-04-25T00:30:51.097972+0000","last_fullsized":"2024-04-25T00:30:51.097972+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'4229","last_scrub_stamp":"2024-04-25T00:30:51.097711+0000","last_deep_scrub":"69'4229","last_deep_scrub_stamp":"2024-04-25T00:30:51.097711+0000","last_clean_scrub_stamp":"2024-04-25T00:30:51.097711+0000","objects_scrubbed":0,"log_size":4229,"log_dups_size":0,"ondisk_log_size":4229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:18:39.189133+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22430,"num_write_kb":302838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2d","version":"74'4662","reported_seq":4791,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:58.806254+0000","last_change":"2024-04-25T00:30:58.806319+0000","last_active":"2024-04-25T00:30:58.806254+0000","last_peered":"2024-04-25T00:30:58.806254+0000","last_clean":"2024-04-25T00:30:58.806254+0000","last_became_active":"2024-04-25T00:01:19.519303+0000","last_became_peered":"2024-04-25T00:01:19.519303+0000","last_unstale":"2024-04-25T00:30:58.806254+0000","last_undegraded":"2024-04-25T00:30:58.806254+0000","last_fullsized":"2024-04-25T00:30:58.806254+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4662","last_scrub_stamp":"2024-04-25T00:30:58.806038+0000","last_deep_scrub":"74'4662","last_deep_scrub_stamp":"2024-04-25T00:30:58.806038+0000","last_clean_scrub_stamp":"2024-04-25T00:30:58.806038+0000","objects_scrubbed":0,"log_size":4662,"log_dups_size":0,"ondisk_log_size":4662,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:20:17.244512+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":25332,"num_write_kb":310579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2b","version":"70'4434","reported_seq":4571,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:53.864498+0000","last_change":"2024-04-25T00:30:53.864556+0000","last_active":"2024-04-25T00:30:53.864498+0000","last_peered":"2024-04-25T00:30:53.864498+0000","last_clean":"2024-04-25T00:30:53.864498+0000","last_became_active":"2024-04-25T00:01:05.365548+0000","last_became_peered":"2024-04-25T00:01:05.365548+0000","last_unstale":"2024-04-25T00:30:53.864498+0000","last_undegraded":"2024-04-25T00:30:53.864498+0000","last_fullsized":"2024-04-25T00:30:53.864498+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'4434","last_scrub_stamp":"2024-04-25T00:30:53.864303+0000","last_deep_scrub":"70'4434","last_deep_scrub_stamp":"2024-04-25T00:30:53.864303+0000","last_clean_scrub_stamp":"2024-04-25T00:30:53.864303+0000","objects_scrubbed":0,"log_size":4434,"log_dups_size":0,"ondisk_log_size":4434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:49:29.728056+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":24224,"num_write_kb":326776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"74'4527","reported_seq":4648,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:38.316178+0000","last_change":"2024-04-25T00:01:19.514659+0000","last_active":"2024-04-25T00:30:38.316178+0000","last_peered":"2024-04-25T00:30:38.316178+0000","last_clean":"2024-04-25T00:30:38.316178+0000","last_became_active":"2024-04-25T00:01:19.514232+0000","last_became_peered":"2024-04-25T00:01:19.514232+0000","last_unstale":"2024-04-25T00:30:38.316178+0000","last_undegraded":"2024-04-25T00:30:38.316178+0000","last_fullsized":"2024-04-25T00:30:38.316178+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4527,"log_dups_size":0,"ondisk_log_size":4527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":25088,"num_write_kb":313806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"70'3677","reported_seq":3791,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:47.176638+0000","last_change":"2024-04-25T00:01:46.665943+0000","last_active":"2024-04-25T00:30:47.176638+0000","last_peered":"2024-04-25T00:30:47.176638+0000","last_clean":"2024-04-25T00:30:47.176638+0000","last_became_active":"2024-04-25T00:01:46.663874+0000","last_became_peered":"2024-04-25T00:01:46.663874+0000","last_unstale":"2024-04-25T00:30:47.176638+0000","last_undegraded":"2024-04-25T00:30:47.176638+0000","last_fullsized":"2024-04-25T00:30:47.176638+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:04.314940+0000","last_clean_scrub_stamp":"2024-04-25T00:01:04.314940+0000","objects_scrubbed":0,"log_size":3677,"log_dups_size":0,"ondisk_log_size":3677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20325,"num_write_kb":253590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2b","version":"74'4916","reported_seq":5049,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:01.385421+0000","last_change":"2024-04-25T00:31:01.385453+0000","last_active":"2024-04-25T00:31:01.385421+0000","last_peered":"2024-04-25T00:31:01.385421+0000","last_clean":"2024-04-25T00:31:01.385421+0000","last_became_active":"2024-04-25T00:01:19.500568+0000","last_became_peered":"2024-04-25T00:01:19.500568+0000","last_unstale":"2024-04-25T00:31:01.385421+0000","last_undegraded":"2024-04-25T00:31:01.385421+0000","last_fullsized":"2024-04-25T00:31:01.385421+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4916","last_scrub_stamp":"2024-04-25T00:31:01.385297+0000","last_deep_scrub":"74'4916","last_deep_scrub_stamp":"2024-04-25T00:31:01.385297+0000","last_clean_scrub_stamp":"2024-04-25T00:31:01.385297+0000","objects_scrubbed":0,"log_size":4916,"log_dups_size":0,"ondisk_log_size":4916,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:29:13.630138+0000","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":26841,"num_write_kb":321097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"69'4618","reported_seq":4754,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:52.271068+0000","last_change":"2024-04-25T00:30:52.271136+0000","last_active":"2024-04-25T00:30:52.271068+0000","last_peered":"2024-04-25T00:30:52.271068+0000","last_clean":"2024-04-25T00:30:52.271068+0000","last_became_active":"2024-04-25T00:01:05.369919+0000","last_became_peered":"2024-04-25T00:01:05.369919+0000","last_unstale":"2024-04-25T00:30:52.271068+0000","last_undegraded":"2024-04-25T00:30:52.271068+0000","last_fullsized":"2024-04-25T00:30:52.271068+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'4618","last_scrub_stamp":"2024-04-25T00:30:52.270855+0000","last_deep_scrub":"69'4618","last_deep_scrub_stamp":"2024-04-25T00:30:52.270855+0000","last_clean_scrub_stamp":"2024-04-25T00:30:52.270855+0000","objects_scrubbed":0,"log_size":4618,"log_dups_size":0,"ondisk_log_size":4618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:21:23.828791+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25225,"num_write_kb":326547,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.28","version":"74'3900","reported_seq":4021,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.778181+0000","last_change":"2024-04-25T00:01:19.540104+0000","last_active":"2024-04-25T00:30:26.778181+0000","last_peered":"2024-04-25T00:30:26.778181+0000","last_clean":"2024-04-25T00:30:26.778181+0000","last_became_active":"2024-04-25T00:01:19.539775+0000","last_became_peered":"2024-04-25T00:01:19.539775+0000","last_unstale":"2024-04-25T00:30:26.778181+0000","last_undegraded":"2024-04-25T00:30:26.778181+0000","last_fullsized":"2024-04-25T00:30:26.778181+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3900,"log_dups_size":0,"ondisk_log_size":3900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21234,"num_write_kb":281892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"70'3829","reported_seq":3964,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:52.866596+0000","last_change":"2024-04-25T00:30:52.866656+0000","last_active":"2024-04-25T00:30:52.866596+0000","last_peered":"2024-04-25T00:30:52.866596+0000","last_clean":"2024-04-25T00:30:52.866596+0000","last_became_active":"2024-04-25T00:01:05.366264+0000","last_became_peered":"2024-04-25T00:01:05.366264+0000","last_unstale":"2024-04-25T00:30:52.866596+0000","last_undegraded":"2024-04-25T00:30:52.866596+0000","last_fullsized":"2024-04-25T00:30:52.866596+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'3829","last_scrub_stamp":"2024-04-25T00:30:52.866392+0000","last_deep_scrub":"70'3829","last_deep_scrub_stamp":"2024-04-25T00:30:52.866392+0000","last_clean_scrub_stamp":"2024-04-25T00:30:52.866392+0000","objects_scrubbed":0,"log_size":3829,"log_dups_size":0,"ondisk_log_size":3829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:32:08.407884+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19822,"num_write_kb":292385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.29","version":"74'4218","reported_seq":4339,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:37.732745+0000","last_change":"2024-04-25T00:01:19.497247+0000","last_active":"2024-04-25T00:30:37.732745+0000","last_peered":"2024-04-25T00:30:37.732745+0000","last_clean":"2024-04-25T00:30:37.732745+0000","last_became_active":"2024-04-25T00:01:19.496517+0000","last_became_peered":"2024-04-25T00:01:19.496517+0000","last_unstale":"2024-04-25T00:30:37.732745+0000","last_undegraded":"2024-04-25T00:30:37.732745+0000","last_fullsized":"2024-04-25T00:30:37.732745+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4218,"log_dups_size":0,"ondisk_log_size":4218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":23201,"num_write_kb":293725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2f","version":"70'5230","reported_seq":5366,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:55.199981+0000","last_change":"2024-04-25T00:30:55.200044+0000","last_active":"2024-04-25T00:30:55.199981+0000","last_peered":"2024-04-25T00:30:55.199981+0000","last_clean":"2024-04-25T00:30:55.199981+0000","last_became_active":"2024-04-25T00:01:05.379090+0000","last_became_peered":"2024-04-25T00:01:05.379090+0000","last_unstale":"2024-04-25T00:30:55.199981+0000","last_undegraded":"2024-04-25T00:30:55.199981+0000","last_fullsized":"2024-04-25T00:30:55.199981+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'5230","last_scrub_stamp":"2024-04-25T00:30:55.199740+0000","last_deep_scrub":"70'5230","last_deep_scrub_stamp":"2024-04-25T00:30:55.199740+0000","last_clean_scrub_stamp":"2024-04-25T00:30:55.199740+0000","objects_scrubbed":0,"log_size":5230,"log_dups_size":0,"ondisk_log_size":5230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:15:35.798652+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":27866,"num_write_kb":365657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.36","version":"74'4384","reported_seq":4505,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.728573+0000","last_change":"2024-04-25T00:01:19.530176+0000","last_active":"2024-04-25T00:30:26.728573+0000","last_peered":"2024-04-25T00:30:26.728573+0000","last_clean":"2024-04-25T00:30:26.728573+0000","last_became_active":"2024-04-25T00:01:19.529410+0000","last_became_peered":"2024-04-25T00:01:19.529410+0000","last_unstale":"2024-04-25T00:30:26.728573+0000","last_undegraded":"2024-04-25T00:30:26.728573+0000","last_fullsized":"2024-04-25T00:30:26.728573+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4384,"log_dups_size":0,"ondisk_log_size":4384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":7,"num_write":24230,"num_write_kb":310193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.30","version":"69'4051","reported_seq":4186,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:56.212932+0000","last_change":"2024-04-25T00:30:56.213000+0000","last_active":"2024-04-25T00:30:56.212932+0000","last_peered":"2024-04-25T00:30:56.212932+0000","last_clean":"2024-04-25T00:30:56.212932+0000","last_became_active":"2024-04-25T00:01:05.369848+0000","last_became_peered":"2024-04-25T00:01:05.369848+0000","last_unstale":"2024-04-25T00:30:56.212932+0000","last_undegraded":"2024-04-25T00:30:56.212932+0000","last_fullsized":"2024-04-25T00:30:56.212932+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'4051","last_scrub_stamp":"2024-04-25T00:30:56.212707+0000","last_deep_scrub":"69'4051","last_deep_scrub_stamp":"2024-04-25T00:30:56.212707+0000","last_clean_scrub_stamp":"2024-04-25T00:30:56.212707+0000","objects_scrubbed":0,"log_size":4051,"log_dups_size":0,"ondisk_log_size":4051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:18:02.374659+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21793,"num_write_kb":288101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.37","version":"74'6689","reported_seq":6820,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:58.225133+0000","last_change":"2024-04-25T00:30:58.225203+0000","last_active":"2024-04-25T00:30:58.225133+0000","last_peered":"2024-04-25T00:30:58.225133+0000","last_clean":"2024-04-25T00:30:58.225133+0000","last_became_active":"2024-04-25T00:01:19.511349+0000","last_became_peered":"2024-04-25T00:01:19.511349+0000","last_unstale":"2024-04-25T00:30:58.225133+0000","last_undegraded":"2024-04-25T00:30:58.225133+0000","last_fullsized":"2024-04-25T00:30:58.225133+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6689","last_scrub_stamp":"2024-04-25T00:30:58.224895+0000","last_deep_scrub":"74'6689","last_deep_scrub_stamp":"2024-04-25T00:30:58.224895+0000","last_clean_scrub_stamp":"2024-04-25T00:30:58.224895+0000","objects_scrubbed":0,"log_size":6689,"log_dups_size":0,"ondisk_log_size":6689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:20:16.663369+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":8,"num_write":38367,"num_write_kb":445068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"70'3993","reported_seq":4128,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:48.585968+0000","last_change":"2024-04-25T00:30:48.586033+0000","last_active":"2024-04-25T00:30:48.585968+0000","last_peered":"2024-04-25T00:30:48.585968+0000","last_clean":"2024-04-25T00:30:48.585968+0000","last_became_active":"2024-04-25T00:01:05.389707+0000","last_became_peered":"2024-04-25T00:01:05.389707+0000","last_unstale":"2024-04-25T00:30:48.585968+0000","last_undegraded":"2024-04-25T00:30:48.585968+0000","last_fullsized":"2024-04-25T00:30:48.585968+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'3993","last_scrub_stamp":"2024-04-25T00:30:48.585749+0000","last_deep_scrub":"70'3993","last_deep_scrub_stamp":"2024-04-25T00:30:48.585749+0000","last_clean_scrub_stamp":"2024-04-25T00:30:48.585749+0000","objects_scrubbed":0,"log_size":3993,"log_dups_size":0,"ondisk_log_size":3993,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:41:42.620564+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21619,"num_write_kb":287362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.34","version":"74'3411","reported_seq":3541,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:59.195126+0000","last_change":"2024-04-25T00:30:59.195215+0000","last_active":"2024-04-25T00:30:59.195126+0000","last_peered":"2024-04-25T00:30:59.195126+0000","last_clean":"2024-04-25T00:30:59.195126+0000","last_became_active":"2024-04-25T00:01:19.512978+0000","last_became_peered":"2024-04-25T00:01:19.512978+0000","last_unstale":"2024-04-25T00:30:59.195126+0000","last_undegraded":"2024-04-25T00:30:59.195126+0000","last_fullsized":"2024-04-25T00:30:59.195126+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'3411","last_scrub_stamp":"2024-04-25T00:30:59.194889+0000","last_deep_scrub":"74'3411","last_deep_scrub_stamp":"2024-04-25T00:30:59.194889+0000","last_clean_scrub_stamp":"2024-04-25T00:30:59.194889+0000","objects_scrubbed":0,"log_size":3411,"log_dups_size":0,"ondisk_log_size":3411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:52:39.304519+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":34,"num_write":18670,"num_write_kb":234685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"70'3759","reported_seq":3895,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:59.499587+0000","last_change":"2024-04-25T00:30:59.499651+0000","last_active":"2024-04-25T00:30:59.499587+0000","last_peered":"2024-04-25T00:30:59.499587+0000","last_clean":"2024-04-25T00:30:59.499587+0000","last_became_active":"2024-04-25T00:01:05.376570+0000","last_became_peered":"2024-04-25T00:01:05.376570+0000","last_unstale":"2024-04-25T00:30:59.499587+0000","last_undegraded":"2024-04-25T00:30:59.499587+0000","last_fullsized":"2024-04-25T00:30:59.499587+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'3759","last_scrub_stamp":"2024-04-25T00:30:59.499378+0000","last_deep_scrub":"70'3759","last_deep_scrub_stamp":"2024-04-25T00:30:59.499378+0000","last_clean_scrub_stamp":"2024-04-25T00:30:59.499378+0000","objects_scrubbed":0,"log_size":3759,"log_dups_size":0,"ondisk_log_size":3759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:54.980931+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20571,"num_write_kb":279348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.35","version":"74'4078","reported_seq":4200,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.701962+0000","last_change":"2024-04-25T00:01:19.516716+0000","last_active":"2024-04-25T00:30:26.701962+0000","last_peered":"2024-04-25T00:30:26.701962+0000","last_clean":"2024-04-25T00:30:26.701962+0000","last_became_active":"2024-04-25T00:01:19.516142+0000","last_became_peered":"2024-04-25T00:01:19.516142+0000","last_unstale":"2024-04-25T00:30:26.701962+0000","last_undegraded":"2024-04-25T00:30:26.701962+0000","last_fullsized":"2024-04-25T00:30:26.701962+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4078,"log_dups_size":0,"ondisk_log_size":4078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":22241,"num_write_kb":273608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.33","version":"70'3191","reported_seq":3326,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:52.564776+0000","last_change":"2024-04-25T00:30:52.564842+0000","last_active":"2024-04-25T00:30:52.564776+0000","last_peered":"2024-04-25T00:30:52.564776+0000","last_clean":"2024-04-25T00:30:52.564776+0000","last_became_active":"2024-04-25T00:01:05.380236+0000","last_became_peered":"2024-04-25T00:01:05.380236+0000","last_unstale":"2024-04-25T00:30:52.564776+0000","last_undegraded":"2024-04-25T00:30:52.564776+0000","last_fullsized":"2024-04-25T00:30:52.564776+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'3191","last_scrub_stamp":"2024-04-25T00:30:52.564505+0000","last_deep_scrub":"70'3191","last_deep_scrub_stamp":"2024-04-25T00:30:52.564505+0000","last_clean_scrub_stamp":"2024-04-25T00:30:52.564505+0000","objects_scrubbed":0,"log_size":3191,"log_dups_size":0,"ondisk_log_size":3191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:44:24.210435+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17526,"num_write_kb":230308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"74'4055","reported_seq":4175,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.764545+0000","last_change":"2024-04-25T00:01:19.529023+0000","last_active":"2024-04-25T00:30:26.764545+0000","last_peered":"2024-04-25T00:30:26.764545+0000","last_clean":"2024-04-25T00:30:26.764545+0000","last_became_active":"2024-04-25T00:01:19.524609+0000","last_became_peered":"2024-04-25T00:01:19.524609+0000","last_unstale":"2024-04-25T00:30:26.764545+0000","last_undegraded":"2024-04-25T00:30:26.764545+0000","last_fullsized":"2024-04-25T00:30:26.764545+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4055,"log_dups_size":0,"ondisk_log_size":4055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":21801,"num_write_kb":276048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.34","version":"70'2711","reported_seq":2846,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:44.388509+0000","last_change":"2024-04-25T00:30:44.388563+0000","last_active":"2024-04-25T00:30:44.388509+0000","last_peered":"2024-04-25T00:30:44.388509+0000","last_clean":"2024-04-25T00:30:44.388509+0000","last_became_active":"2024-04-25T00:01:05.384997+0000","last_became_peered":"2024-04-25T00:01:05.384997+0000","last_unstale":"2024-04-25T00:30:44.388509+0000","last_undegraded":"2024-04-25T00:30:44.388509+0000","last_fullsized":"2024-04-25T00:30:44.388509+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2711","last_scrub_stamp":"2024-04-25T00:30:44.388310+0000","last_deep_scrub":"70'2711","last_deep_scrub_stamp":"2024-04-25T00:30:44.388310+0000","last_clean_scrub_stamp":"2024-04-25T00:30:44.388310+0000","objects_scrubbed":0,"log_size":2711,"log_dups_size":0,"ondisk_log_size":2711,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:46:59.186973+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14839,"num_write_kb":207133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.33","version":"74'3434","reported_seq":3557,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.730062+0000","last_change":"2024-04-25T00:01:19.516867+0000","last_active":"2024-04-25T00:30:26.730062+0000","last_peered":"2024-04-25T00:30:26.730062+0000","last_clean":"2024-04-25T00:30:26.730062+0000","last_became_active":"2024-04-25T00:01:19.515764+0000","last_became_peered":"2024-04-25T00:01:19.515764+0000","last_unstale":"2024-04-25T00:30:26.730062+0000","last_undegraded":"2024-04-25T00:30:26.730062+0000","last_fullsized":"2024-04-25T00:30:26.730062+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3434,"log_dups_size":0,"ondisk_log_size":3434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":5,"num_write":19347,"num_write_kb":241776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"69'3819","reported_seq":3955,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:48.346026+0000","last_change":"2024-04-25T00:30:48.346092+0000","last_active":"2024-04-25T00:30:48.346026+0000","last_peered":"2024-04-25T00:30:48.346026+0000","last_clean":"2024-04-25T00:30:48.346026+0000","last_became_active":"2024-04-25T00:01:05.388999+0000","last_became_peered":"2024-04-25T00:01:05.388999+0000","last_unstale":"2024-04-25T00:30:48.346026+0000","last_undegraded":"2024-04-25T00:30:48.346026+0000","last_fullsized":"2024-04-25T00:30:48.346026+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'3819","last_scrub_stamp":"2024-04-25T00:30:48.345797+0000","last_deep_scrub":"69'3819","last_deep_scrub_stamp":"2024-04-25T00:30:48.345797+0000","last_clean_scrub_stamp":"2024-04-25T00:30:48.345797+0000","objects_scrubbed":0,"log_size":3819,"log_dups_size":0,"ondisk_log_size":3819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:20:06.784272+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":19986,"num_write_kb":272748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.30","version":"74'4335","reported_seq":4454,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.725060+0000","last_change":"2024-04-25T00:01:19.519365+0000","last_active":"2024-04-25T00:30:26.725060+0000","last_peered":"2024-04-25T00:30:26.725060+0000","last_clean":"2024-04-25T00:30:26.725060+0000","last_became_active":"2024-04-25T00:01:19.519085+0000","last_became_peered":"2024-04-25T00:01:19.519085+0000","last_unstale":"2024-04-25T00:30:26.725060+0000","last_undegraded":"2024-04-25T00:30:26.725060+0000","last_fullsized":"2024-04-25T00:30:26.725060+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4335,"log_dups_size":0,"ondisk_log_size":4335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23507,"num_write_kb":286989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"69'4866","reported_seq":5001,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:59.404114+0000","last_change":"2024-04-25T00:30:59.404177+0000","last_active":"2024-04-25T00:30:59.404114+0000","last_peered":"2024-04-25T00:30:59.404114+0000","last_clean":"2024-04-25T00:30:59.404114+0000","last_became_active":"2024-04-25T00:01:05.367893+0000","last_became_peered":"2024-04-25T00:01:05.367893+0000","last_unstale":"2024-04-25T00:30:59.404114+0000","last_undegraded":"2024-04-25T00:30:59.404114+0000","last_fullsized":"2024-04-25T00:30:59.404114+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'4866","last_scrub_stamp":"2024-04-25T00:30:59.403892+0000","last_deep_scrub":"69'4866","last_deep_scrub_stamp":"2024-04-25T00:30:59.403892+0000","last_clean_scrub_stamp":"2024-04-25T00:30:59.403892+0000","objects_scrubbed":0,"log_size":4866,"log_dups_size":0,"ondisk_log_size":4866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:09:13.206066+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26504,"num_write_kb":327449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.31","version":"74'4017","reported_seq":4145,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:03.444511+0000","last_change":"2024-04-25T00:31:03.444544+0000","last_active":"2024-04-25T00:31:03.444511+0000","last_peered":"2024-04-25T00:31:03.444511+0000","last_clean":"2024-04-25T00:31:03.444511+0000","last_became_active":"2024-04-25T00:01:19.513184+0000","last_became_peered":"2024-04-25T00:01:19.513184+0000","last_unstale":"2024-04-25T00:31:03.444511+0000","last_undegraded":"2024-04-25T00:31:03.444511+0000","last_fullsized":"2024-04-25T00:31:03.444511+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4017","last_scrub_stamp":"2024-04-25T00:31:03.444373+0000","last_deep_scrub":"74'4017","last_deep_scrub_stamp":"2024-04-25T00:31:03.444373+0000","last_clean_scrub_stamp":"2024-04-25T00:31:03.444373+0000","objects_scrubbed":0,"log_size":4017,"log_dups_size":0,"ondisk_log_size":4017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:17:55.839748+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":21984,"num_write_kb":288846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"70'5540","reported_seq":5675,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:47.074024+0000","last_change":"2024-04-25T00:30:47.074090+0000","last_active":"2024-04-25T00:30:47.074024+0000","last_peered":"2024-04-25T00:30:47.074024+0000","last_clean":"2024-04-25T00:30:47.074024+0000","last_became_active":"2024-04-25T00:01:05.361586+0000","last_became_peered":"2024-04-25T00:01:05.361586+0000","last_unstale":"2024-04-25T00:30:47.074024+0000","last_undegraded":"2024-04-25T00:30:47.074024+0000","last_fullsized":"2024-04-25T00:30:47.074024+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'5540","last_scrub_stamp":"2024-04-25T00:30:47.073815+0000","last_deep_scrub":"70'5540","last_deep_scrub_stamp":"2024-04-25T00:30:47.073815+0000","last_clean_scrub_stamp":"2024-04-25T00:30:47.073815+0000","objects_scrubbed":0,"log_size":5540,"log_dups_size":0,"ondisk_log_size":5540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:25:01.256353+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30654,"num_write_kb":393416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3e","version":"74'5058","reported_seq":5181,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:33.002022+0000","last_change":"2024-04-25T00:01:19.515101+0000","last_active":"2024-04-25T00:30:33.002022+0000","last_peered":"2024-04-25T00:30:33.002022+0000","last_clean":"2024-04-25T00:30:33.002022+0000","last_became_active":"2024-04-25T00:01:19.513314+0000","last_became_peered":"2024-04-25T00:01:19.513314+0000","last_unstale":"2024-04-25T00:30:33.002022+0000","last_undegraded":"2024-04-25T00:30:33.002022+0000","last_fullsized":"2024-04-25T00:30:33.002022+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":5058,"log_dups_size":0,"ondisk_log_size":5058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":15,"num_write":27439,"num_write_kb":359791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.38","version":"70'4026","reported_seq":4160,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:46.599739+0000","last_change":"2024-04-25T00:30:46.599801+0000","last_active":"2024-04-25T00:30:46.599739+0000","last_peered":"2024-04-25T00:30:46.599739+0000","last_clean":"2024-04-25T00:30:46.599739+0000","last_became_active":"2024-04-25T00:01:05.389724+0000","last_became_peered":"2024-04-25T00:01:05.389724+0000","last_unstale":"2024-04-25T00:30:46.599739+0000","last_undegraded":"2024-04-25T00:30:46.599739+0000","last_fullsized":"2024-04-25T00:30:46.599739+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'4026","last_scrub_stamp":"2024-04-25T00:30:46.599517+0000","last_deep_scrub":"70'4026","last_deep_scrub_stamp":"2024-04-25T00:30:46.599517+0000","last_clean_scrub_stamp":"2024-04-25T00:30:46.599517+0000","objects_scrubbed":0,"log_size":4026,"log_dups_size":0,"ondisk_log_size":4026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:40:44.481539+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21352,"num_write_kb":290527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3f","version":"74'4365","reported_seq":4498,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:53.632870+0000","last_change":"2024-04-25T00:30:53.632995+0000","last_active":"2024-04-25T00:30:53.632870+0000","last_peered":"2024-04-25T00:30:53.632870+0000","last_clean":"2024-04-25T00:30:53.632870+0000","last_became_active":"2024-04-25T00:01:19.508049+0000","last_became_peered":"2024-04-25T00:01:19.508049+0000","last_unstale":"2024-04-25T00:30:53.632870+0000","last_undegraded":"2024-04-25T00:30:53.632870+0000","last_fullsized":"2024-04-25T00:30:53.632870+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4365","last_scrub_stamp":"2024-04-25T00:30:53.632570+0000","last_deep_scrub":"74'4365","last_deep_scrub_stamp":"2024-04-25T00:30:53.632570+0000","last_clean_scrub_stamp":"2024-04-25T00:30:53.632570+0000","objects_scrubbed":0,"log_size":4365,"log_dups_size":0,"ondisk_log_size":4365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:13:40.444328+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":8,"num_write":24611,"num_write_kb":307761,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"70'3970","reported_seq":4104,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:51.354868+0000","last_change":"2024-04-25T00:30:51.354932+0000","last_active":"2024-04-25T00:30:51.354868+0000","last_peered":"2024-04-25T00:30:51.354868+0000","last_clean":"2024-04-25T00:30:51.354868+0000","last_became_active":"2024-04-25T00:01:05.369739+0000","last_became_peered":"2024-04-25T00:01:05.369739+0000","last_unstale":"2024-04-25T00:30:51.354868+0000","last_undegraded":"2024-04-25T00:30:51.354868+0000","last_fullsized":"2024-04-25T00:30:51.354868+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'3970","last_scrub_stamp":"2024-04-25T00:30:51.354655+0000","last_deep_scrub":"70'3970","last_deep_scrub_stamp":"2024-04-25T00:30:51.354655+0000","last_clean_scrub_stamp":"2024-04-25T00:30:51.354655+0000","objects_scrubbed":0,"log_size":3970,"log_dups_size":0,"ondisk_log_size":3970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:54:55.160015+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22008,"num_write_kb":289131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3c","version":"74'3966","reported_seq":4088,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.674125+0000","last_change":"2024-04-25T00:01:19.521456+0000","last_active":"2024-04-25T00:30:26.674125+0000","last_peered":"2024-04-25T00:30:26.674125+0000","last_clean":"2024-04-25T00:30:26.674125+0000","last_became_active":"2024-04-25T00:01:19.520787+0000","last_became_peered":"2024-04-25T00:01:19.520787+0000","last_unstale":"2024-04-25T00:30:26.674125+0000","last_undegraded":"2024-04-25T00:30:26.674125+0000","last_fullsized":"2024-04-25T00:30:26.674125+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":3966,"log_dups_size":0,"ondisk_log_size":3966,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22683,"num_write_kb":271349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3a","version":"70'3294","reported_seq":3428,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:56.669165+0000","last_change":"2024-04-25T00:30:56.669228+0000","last_active":"2024-04-25T00:30:56.669165+0000","last_peered":"2024-04-25T00:30:56.669165+0000","last_clean":"2024-04-25T00:30:56.669165+0000","last_became_active":"2024-04-25T00:01:05.358477+0000","last_became_peered":"2024-04-25T00:01:05.358477+0000","last_unstale":"2024-04-25T00:30:56.669165+0000","last_undegraded":"2024-04-25T00:30:56.669165+0000","last_fullsized":"2024-04-25T00:30:56.669165+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'3294","last_scrub_stamp":"2024-04-25T00:30:56.668950+0000","last_deep_scrub":"70'3294","last_deep_scrub_stamp":"2024-04-25T00:30:56.668950+0000","last_clean_scrub_stamp":"2024-04-25T00:30:56.668950+0000","objects_scrubbed":0,"log_size":3294,"log_dups_size":0,"ondisk_log_size":3294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:52.150502+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17420,"num_write_kb":225566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3d","version":"74'5153","reported_seq":5283,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:56.232210+0000","last_change":"2024-04-25T00:30:56.232275+0000","last_active":"2024-04-25T00:30:56.232210+0000","last_peered":"2024-04-25T00:30:56.232210+0000","last_clean":"2024-04-25T00:30:56.232210+0000","last_became_active":"2024-04-25T00:01:19.507812+0000","last_became_peered":"2024-04-25T00:01:19.507812+0000","last_unstale":"2024-04-25T00:30:56.232210+0000","last_undegraded":"2024-04-25T00:30:56.232210+0000","last_fullsized":"2024-04-25T00:30:56.232210+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5153","last_scrub_stamp":"2024-04-25T00:30:56.232005+0000","last_deep_scrub":"74'5153","last_deep_scrub_stamp":"2024-04-25T00:30:56.232005+0000","last_clean_scrub_stamp":"2024-04-25T00:30:56.232005+0000","objects_scrubbed":0,"log_size":5153,"log_dups_size":0,"ondisk_log_size":5153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:41:50.266820+0000","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":28174,"num_write_kb":326190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3b","version":"70'3663","reported_seq":3797,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:46.333214+0000","last_change":"2024-04-25T00:30:46.333357+0000","last_active":"2024-04-25T00:30:46.333214+0000","last_peered":"2024-04-25T00:30:46.333214+0000","last_clean":"2024-04-25T00:30:46.333214+0000","last_became_active":"2024-04-25T00:01:05.378650+0000","last_became_peered":"2024-04-25T00:01:05.378650+0000","last_unstale":"2024-04-25T00:30:46.333214+0000","last_undegraded":"2024-04-25T00:30:46.333214+0000","last_fullsized":"2024-04-25T00:30:46.333214+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'3663","last_scrub_stamp":"2024-04-25T00:30:46.332991+0000","last_deep_scrub":"70'3663","last_deep_scrub_stamp":"2024-04-25T00:30:46.332991+0000","last_clean_scrub_stamp":"2024-04-25T00:30:46.332991+0000","objects_scrubbed":0,"log_size":3663,"log_dups_size":0,"ondisk_log_size":3663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:51:25.231176+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18938,"num_write_kb":284951,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3a","version":"74'3833","reported_seq":3959,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:57.844053+0000","last_change":"2024-04-25T00:30:57.844114+0000","last_active":"2024-04-25T00:30:57.844053+0000","last_peered":"2024-04-25T00:30:57.844053+0000","last_clean":"2024-04-25T00:30:57.844053+0000","last_became_active":"2024-04-25T00:01:19.513956+0000","last_became_peered":"2024-04-25T00:01:19.513956+0000","last_unstale":"2024-04-25T00:30:57.844053+0000","last_undegraded":"2024-04-25T00:30:57.844053+0000","last_fullsized":"2024-04-25T00:30:57.844053+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'3833","last_scrub_stamp":"2024-04-25T00:30:57.843830+0000","last_deep_scrub":"74'3833","last_deep_scrub_stamp":"2024-04-25T00:30:57.843830+0000","last_clean_scrub_stamp":"2024-04-25T00:30:57.843830+0000","objects_scrubbed":0,"log_size":3833,"log_dups_size":0,"ondisk_log_size":3833,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:44:29.489760+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20679,"num_write_kb":271382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"69'3516","reported_seq":3654,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:57.662201+0000","last_change":"2024-04-25T00:30:57.662264+0000","last_active":"2024-04-25T00:30:57.662201+0000","last_peered":"2024-04-25T00:30:57.662201+0000","last_clean":"2024-04-25T00:30:57.662201+0000","last_became_active":"2024-04-25T00:01:05.358866+0000","last_became_peered":"2024-04-25T00:01:05.358866+0000","last_unstale":"2024-04-25T00:30:57.662201+0000","last_undegraded":"2024-04-25T00:30:57.662201+0000","last_fullsized":"2024-04-25T00:30:57.662201+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'3516","last_scrub_stamp":"2024-04-25T00:30:57.661979+0000","last_deep_scrub":"69'3516","last_deep_scrub_stamp":"2024-04-25T00:30:57.661979+0000","last_clean_scrub_stamp":"2024-04-25T00:30:57.661979+0000","objects_scrubbed":0,"log_size":3516,"log_dups_size":0,"ondisk_log_size":3516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:46:49.144539+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":19226,"num_write_kb":253281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3b","version":"74'4130","reported_seq":4249,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.773325+0000","last_change":"2024-04-25T00:01:19.499574+0000","last_active":"2024-04-25T00:30:26.773325+0000","last_peered":"2024-04-25T00:30:26.773325+0000","last_clean":"2024-04-25T00:30:26.773325+0000","last_became_active":"2024-04-25T00:01:19.498699+0000","last_became_peered":"2024-04-25T00:01:19.498699+0000","last_unstale":"2024-04-25T00:30:26.773325+0000","last_undegraded":"2024-04-25T00:30:26.773325+0000","last_fullsized":"2024-04-25T00:30:26.773325+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4130,"log_dups_size":0,"ondisk_log_size":4130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":3,"num_write":21749,"num_write_kb":288055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"70'3483","reported_seq":3620,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:51.542203+0000","last_change":"2024-04-25T00:30:51.542272+0000","last_active":"2024-04-25T00:30:51.542203+0000","last_peered":"2024-04-25T00:30:51.542203+0000","last_clean":"2024-04-25T00:30:51.542203+0000","last_became_active":"2024-04-25T00:01:05.370522+0000","last_became_peered":"2024-04-25T00:01:05.370522+0000","last_unstale":"2024-04-25T00:30:51.542203+0000","last_undegraded":"2024-04-25T00:30:51.542203+0000","last_fullsized":"2024-04-25T00:30:51.542203+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'3483","last_scrub_stamp":"2024-04-25T00:30:51.541962+0000","last_deep_scrub":"70'3483","last_deep_scrub_stamp":"2024-04-25T00:30:51.541962+0000","last_clean_scrub_stamp":"2024-04-25T00:30:51.541962+0000","objects_scrubbed":0,"log_size":3483,"log_dups_size":0,"ondisk_log_size":3483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:19:32.940834+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18326,"num_write_kb":261938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.38","version":"74'4120","reported_seq":4238,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:26.738723+0000","last_change":"2024-04-25T00:01:19.523798+0000","last_active":"2024-04-25T00:30:26.738723+0000","last_peered":"2024-04-25T00:30:26.738723+0000","last_clean":"2024-04-25T00:30:26.738723+0000","last_became_active":"2024-04-25T00:01:19.523450+0000","last_became_peered":"2024-04-25T00:01:19.523450+0000","last_unstale":"2024-04-25T00:30:26.738723+0000","last_undegraded":"2024-04-25T00:30:26.738723+0000","last_fullsized":"2024-04-25T00:30:26.738723+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T00:01:18.450053+0000","last_clean_scrub_stamp":"2024-04-25T00:01:18.450053+0000","objects_scrubbed":0,"log_size":4120,"log_dups_size":0,"ondisk_log_size":4120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22879,"num_write_kb":288519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3e","version":"70'4991","reported_seq":5128,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:55.620709+0000","last_change":"2024-04-25T00:30:55.620865+0000","last_active":"2024-04-25T00:30:55.620709+0000","last_peered":"2024-04-25T00:30:55.620709+0000","last_clean":"2024-04-25T00:30:55.620709+0000","last_became_active":"2024-04-25T00:01:05.362606+0000","last_became_peered":"2024-04-25T00:01:05.362606+0000","last_unstale":"2024-04-25T00:30:55.620709+0000","last_undegraded":"2024-04-25T00:30:55.620709+0000","last_fullsized":"2024-04-25T00:30:55.620709+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'4991","last_scrub_stamp":"2024-04-25T00:30:55.620470+0000","last_deep_scrub":"70'4991","last_deep_scrub_stamp":"2024-04-25T00:30:55.620470+0000","last_clean_scrub_stamp":"2024-04-25T00:30:55.620470+0000","objects_scrubbed":0,"log_size":4991,"log_dups_size":0,"ondisk_log_size":4991,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:45:31.561538+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26684,"num_write_kb":361051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.39","version":"74'4850","reported_seq":4976,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:49.562003+0000","last_change":"2024-04-25T00:30:49.562032+0000","last_active":"2024-04-25T00:30:49.562003+0000","last_peered":"2024-04-25T00:30:49.562003+0000","last_clean":"2024-04-25T00:30:49.562003+0000","last_became_active":"2024-04-25T00:01:19.504991+0000","last_became_peered":"2024-04-25T00:01:19.504991+0000","last_unstale":"2024-04-25T00:30:49.562003+0000","last_undegraded":"2024-04-25T00:30:49.562003+0000","last_fullsized":"2024-04-25T00:30:49.562003+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4850","last_scrub_stamp":"2024-04-25T00:30:49.561893+0000","last_deep_scrub":"74'4850","last_deep_scrub_stamp":"2024-04-25T00:30:49.561893+0000","last_clean_scrub_stamp":"2024-04-25T00:30:49.561893+0000","objects_scrubbed":0,"log_size":4850,"log_dups_size":0,"ondisk_log_size":4850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:34:06.903945+0000","scrub_duration":123,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26612,"num_write_kb":323915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3f","version":"70'3815","reported_seq":3952,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:50.372947+0000","last_change":"2024-04-25T00:30:50.373007+0000","last_active":"2024-04-25T00:30:50.372947+0000","last_peered":"2024-04-25T00:30:50.372947+0000","last_clean":"2024-04-25T00:30:50.372947+0000","last_became_active":"2024-04-25T00:01:05.389127+0000","last_became_peered":"2024-04-25T00:01:05.389127+0000","last_unstale":"2024-04-25T00:30:50.372947+0000","last_undegraded":"2024-04-25T00:30:50.372947+0000","last_fullsized":"2024-04-25T00:30:50.372947+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'3815","last_scrub_stamp":"2024-04-25T00:30:50.372743+0000","last_deep_scrub":"70'3815","last_deep_scrub_stamp":"2024-04-25T00:30:50.372743+0000","last_clean_scrub_stamp":"2024-04-25T00:30:50.372743+0000","objects_scrubbed":0,"log_size":3815,"log_dups_size":0,"ondisk_log_size":3815,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:10:18.579715+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20941,"num_write_kb":280561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":105,"num_read_kb":408,"num_write":1543888,"num_write_kb":19306803,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":280555,"ondisk_log_size":280555,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":19555305,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1733,"num_read_kb":229197,"num_write":5131,"num_write_kb":21279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":39174144,"data_stored":39110610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4729,"ondisk_log_size":4729,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":3,"num_write":1412500,"num_write_kb":18776370,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":261221,"ondisk_log_size":261221,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":19117950,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1590,"num_read_kb":257618,"num_write":5060,"num_write_kb":20770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":38297600,"data_stored":38235900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4720,"ondisk_log_size":4720,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509806,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":674532,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":673470,"kb_avail":93697308,"statfs":{"total":96636764160,"available":95946043392,"internally_reserved":0,"allocated":1077248,"data_stored":617327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":689633739},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509806,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":843472,"kb_used_data":11784,"kb_used_omap":1,"kb_used_meta":831678,"kb_avail":93528368,"statfs":{"total":96636764160,"available":95773048832,"internally_reserved":0,"allocated":12066816,"data_stored":11609748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":851638726},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509805,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":870568,"kb_used_data":18016,"kb_used_omap":1,"kb_used_meta":852542,"kb_avail":93501272,"statfs":{"total":96636764160,"available":95745302528,"internally_reserved":0,"allocated":18448384,"data_stored":17985439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":873003461},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509806,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":922240,"kb_used_data":15544,"kb_used_omap":1,"kb_used_meta":906686,"kb_avail":93449600,"statfs":{"total":96636764160,"available":95692390400,"internally_reserved":0,"allocated":15917056,"data_stored":15447989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":928446919},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509807,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1076840,"kb_used_data":15968,"kb_used_omap":1,"kb_used_meta":1060862,"kb_avail":93295000,"statfs":{"total":96636764160,"available":95534080000,"internally_reserved":0,"allocated":16351232,"data_stored":15888732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1086323141},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509806,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":788556,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":787006,"kb_avail":93583284,"statfs":{"total":96636764160,"available":95829282816,"internally_reserved":0,"allocated":1576960,"data_stored":1119003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":805894597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509806,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1028004,"kb_used_data":3164,"kb_used_omap":1,"kb_used_meta":1024830,"kb_avail":93343836,"statfs":{"total":96636764160,"available":95584088064,"internally_reserved":0,"allocated":3239936,"data_stored":2779219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1049426373},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575215,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":962780,"kb_used_data":17364,"kb_used_omap":1,"kb_used_meta":945406,"kb_avail":93409060,"statfs":{"total":96636764160,"available":95650877440,"internally_reserved":0,"allocated":17780736,"data_stored":17307749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":968096197},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2260992,"data_stored":2250634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8396800,"data_stored":8389280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":10645504,"data_stored":10639768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8404992,"data_stored":8389952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4292608,"data_stored":4283082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":88810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6979584,"data_stored":6970957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4296704,"data_stored":4279726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8400896,"data_stored":8389302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12587008,"data_stored":12583494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":6897664,"data_stored":6886949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T00:31:08.022 INFO:tasks.ceph:pgid 5.19 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:31:08.022 INFO:tasks.ceph:pgid 5.1b last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:31:08.022 INFO:tasks.ceph:pgid 5.c last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:31:08.022 INFO:tasks.ceph:pgid 2.b last_scrub_stamp 2024-04-25T00:01:03.305620+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=3, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:31:08.022 INFO:tasks.ceph:pgid 4.d last_scrub_stamp 2024-04-25T00:01:17.440296+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:31:08.023 INFO:tasks.ceph:pgid 5.e last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:31:08.023 INFO:tasks.ceph:pgid 5.0 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:31:08.023 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-04-25T00:01:03.305620+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=3, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:31:08.023 INFO:tasks.ceph:pgid 5.1 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:31:08.023 INFO:tasks.ceph:pgid 5.2 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:31:08.023 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-04-25T00:01:03.305620+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=3, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:31:08.023 INFO:tasks.ceph:pgid 5.3 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:31:08.023 INFO:tasks.ceph:pgid 5.5 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:31:08.024 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-04-25T00:01:03.305620+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=3, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:31:08.024 INFO:tasks.ceph:pgid 5.7 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:31:08.024 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-04-25T00:01:03.305620+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=3, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:31:08.024 INFO:tasks.ceph:pgid 5.4 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:31:08.024 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-04-25T00:01:03.305620+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=3, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:31:08.024 INFO:tasks.ceph:pgid 5.b last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:31:08.024 INFO:tasks.ceph:pgid 5.a last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:31:08.025 INFO:tasks.ceph:pgid 4.8 last_scrub_stamp 2024-04-25T00:01:17.440296+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=17, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:31:08.025 INFO:tasks.ceph:pgid 5.8 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:31:08.025 INFO:tasks.ceph:pgid 5.17 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:31:08.025 INFO:tasks.ceph:pgid 5.15 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:31:08.025 INFO:tasks.ceph:pgid 2.12 last_scrub_stamp 2024-04-25T00:01:03.305620+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=3, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:31:08.025 INFO:tasks.ceph:pgid 5.14 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:31:08.025 INFO:tasks.ceph:pgid 5.13 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:31:08.025 INFO:tasks.ceph:pgid 5.12 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:31:08.026 INFO:tasks.ceph:pgid 5.11 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:31:08.026 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:31:08.026 INFO:tasks.ceph:pgid 5.1f last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:31:08.026 INFO:tasks.ceph:pgid 2.18 last_scrub_stamp 2024-04-25T00:01:03.305620+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=3, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:31:08.026 INFO:tasks.ceph:pgid 5.1d last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:31:08.026 INFO:tasks.ceph:pgid 2.1a last_scrub_stamp 2024-04-25T00:01:03.305620+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=3, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:31:08.026 INFO:tasks.ceph:pgid 5.1c last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:31:08.027 INFO:tasks.ceph:pgid 5.24 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:31:08.027 INFO:tasks.ceph:pgid 5.25 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:31:08.027 INFO:tasks.ceph:pgid 3.23 last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:31:08.027 INFO:tasks.ceph:pgid 5.23 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:31:08.027 INFO:tasks.ceph:pgid 5.20 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:31:08.027 INFO:tasks.ceph:pgid 5.21 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:31:08.027 INFO:tasks.ceph:pgid 3.29 last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:31:08.027 INFO:tasks.ceph:pgid 5.2c last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:31:08.028 INFO:tasks.ceph:pgid 5.2a last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:31:08.028 INFO:tasks.ceph:pgid 3.2c last_scrub_stamp 2024-04-25T00:01:04.314940+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=4, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:31:08.028 INFO:tasks.ceph:pgid 5.28 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:31:08.028 INFO:tasks.ceph:pgid 5.29 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:31:08.028 INFO:tasks.ceph:pgid 5.36 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:31:08.028 INFO:tasks.ceph:pgid 5.35 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:31:08.028 INFO:tasks.ceph:pgid 5.32 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:31:08.028 INFO:tasks.ceph:pgid 5.33 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:31:08.029 INFO:tasks.ceph:pgid 5.30 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:31:08.029 INFO:tasks.ceph:pgid 5.3e last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:31:08.029 INFO:tasks.ceph:pgid 5.3c last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:31:08.029 INFO:tasks.ceph:pgid 5.3b last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:31:08.029 INFO:tasks.ceph:pgid 5.38 last_scrub_stamp 2024-04-25T00:01:18.450053+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=1, tm_sec=18, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=0, tm_min=30, tm_sec=42, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T00:31:08.029 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-25T00:31:12.871 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:31:12.874 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:31:20.117 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:31:20.249 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.250+0000 7f203ae8c640 1 Processor -- start 2024-04-25T00:31:20.250 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.250+0000 7f203ae8c640 1 -- start start 2024-04-25T00:31:20.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.250+0000 7f203ae8c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f20340c3460 0x7f20340a94a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:20.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.250+0000 7f203ae8c640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20340702a0 con 0x7f20340a99e0 2024-04-25T00:31:20.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.250+0000 7f203ae8c640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f203406b210 con 0x7f203414d7e0 2024-04-25T00:31:20.252 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.250+0000 7f203ae8c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f20340c2c30 con 0x7f20340c3460 2024-04-25T00:31:20.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.250+0000 7f2039429640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f20340a99e0 0x7f20341572e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:47680/0 (socket says 172.21.15.116:47680) 2024-04-25T00:31:20.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.250+0000 7f2039429640 1 -- 172.21.15.116:0/1098763189 learned_addr learned my addr 172.21.15.116:0/1098763189 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:31:20.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.250+0000 7f2033fff640 1 --2- 172.21.15.116:0/1098763189 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f20340c3460 0x7f20340a94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:20.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.250+0000 7f20337fe640 1 -- 172.21.15.116:0/1098763189 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4197961915 0 0) 0x7f20340702a0 con 0x7f20340a99e0 2024-04-25T00:31:20.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.250+0000 7f20337fe640 1 -- 172.21.15.116:0/1098763189 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2020003530 con 0x7f20340a99e0 2024-04-25T00:31:20.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.250+0000 7f20337fe640 1 -- 172.21.15.116:0/1098763189 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1607848795 0 0) 0x7f203406b210 con 0x7f203414d7e0 2024-04-25T00:31:20.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.250+0000 7f20337fe640 1 -- 172.21.15.116:0/1098763189 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20340702a0 con 0x7f203414d7e0 2024-04-25T00:31:20.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.250+0000 7f20337fe640 1 -- 172.21.15.116:0/1098763189 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2671064066 0 0) 0x7f2020003530 con 0x7f20340a99e0 2024-04-25T00:31:20.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.250+0000 7f20337fe640 1 -- 172.21.15.116:0/1098763189 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f20340c3460 msgr2=0x7f20340a94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:31:20.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.250+0000 7f20337fe640 1 --2- 172.21.15.116:0/1098763189 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f20340c3460 0x7f20340a94a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:20.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.250+0000 7f20337fe640 1 -- 172.21.15.116:0/1098763189 >> v1:172.21.15.174:6789/0 conn(0x7f203414d7e0 legacy=0x7f20341540b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:20.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.250+0000 7f20337fe640 1 -- 172.21.15.116:0/1098763189 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f203406b4b0 con 0x7f20340a99e0 2024-04-25T00:31:20.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.250+0000 7f20337fe640 1 -- 172.21.15.116:0/1098763189 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f202c001a00 con 0x7f20340a99e0 2024-04-25T00:31:20.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.250+0000 7f20337fe640 1 -- 172.21.15.116:0/1098763189 >> v1:172.21.15.116:6789/0 conn(0x7f20340a99e0 legacy=0x7f20341572e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:20.253 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.250+0000 7f20337fe640 1 --2- 172.21.15.116:0/1098763189 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2020003b80 0x7f2020003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:20.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.250+0000 7f20337fe640 1 --2- 172.21.15.116:0/1098763189 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f20340c3460 0x7f2020004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:20.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.250+0000 7f20337fe640 1 --2- 172.21.15.116:0/1098763189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2020004d50 0x7f20200051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:20.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.250+0000 7f20337fe640 1 -- 172.21.15.116:0/1098763189 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f202000bad0 con 0x7f2020004d50 2024-04-25T00:31:20.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.250+0000 7f20337fe640 1 -- 172.21.15.116:0/1098763189 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f202000bc80 con 0x7f2020003b80 2024-04-25T00:31:20.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.250+0000 7f20337fe640 1 -- 172.21.15.116:0/1098763189 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f202000be80 con 0x7f20340c3460 2024-04-25T00:31:20.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.250+0000 7f2039429640 1 --2- 172.21.15.116:0/1098763189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2020004d50 0x7f20200051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:20.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.250+0000 7f2038c28640 1 --2- 172.21.15.116:0/1098763189 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2020003b80 0x7f2020003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:20.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.250+0000 7f2033fff640 1 --2- 172.21.15.116:0/1098763189 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f20340c3460 0x7f2020004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:20.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.250+0000 7f2039429640 1 -- 172.21.15.116:0/1098763189 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f20340c3460 msgr2=0x7f2020004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:31:20.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.250+0000 7f2039429640 1 --2- 172.21.15.116:0/1098763189 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f20340c3460 0x7f2020004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:20.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.250+0000 7f2039429640 1 -- 172.21.15.116:0/1098763189 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2020003b80 msgr2=0x7f2020003f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:31:20.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.250+0000 7f2039429640 1 --2- 172.21.15.116:0/1098763189 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2020003b80 0x7f2020003f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:20.254 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.250+0000 7f2039429640 1 -- 172.21.15.116:0/1098763189 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2020003530 con 0x7f2020004d50 2024-04-25T00:31:20.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.254+0000 7f2039429640 1 --2- 172.21.15.116:0/1098763189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2020004d50 0x7f20200051d0 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f202c00a820 tx=0x7f202c01d300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f38a4d362b80bf8e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:20.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.254+0000 7f20337fe640 1 -- 172.21.15.116:0/1098763189 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f202c00a450 con 0x7f2020004d50 2024-04-25T00:31:20.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.254+0000 7f20337fe640 1 -- 172.21.15.116:0/1098763189 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f202c0072d0 con 0x7f2020004d50 2024-04-25T00:31:20.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.254+0000 7f20337fe640 1 -- 172.21.15.116:0/1098763189 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f202c007700 con 0x7f2020004d50 2024-04-25T00:31:20.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.254+0000 7f203ae8c640 1 -- 172.21.15.116:0/1098763189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2020004d50 msgr2=0x7f20200051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:20.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.254+0000 7f203ae8c640 1 --2- 172.21.15.116:0/1098763189 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2020004d50 0x7f20200051d0 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f202c00a820 tx=0x7f202c01d300 comp rx=0 tx=0).stop 2024-04-25T00:31:20.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.254+0000 7f2039429640 1 -- 172.21.15.116:0/1098763189 reap_dead start 2024-04-25T00:31:20.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.254+0000 7f203ae8c640 1 -- 172.21.15.116:0/1098763189 shutdown_connections 2024-04-25T00:31:20.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.254+0000 7f203ae8c640 1 -- 172.21.15.116:0/1098763189 >> 172.21.15.116:0/1098763189 conn(0x7f20340c7fa0 msgr2=0x7f20340c83b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:31:20.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.254+0000 7f203ae8c640 1 -- 172.21.15.116:0/1098763189 shutdown_connections 2024-04-25T00:31:20.256 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.254+0000 7f203ae8c640 1 -- 172.21.15.116:0/1098763189 wait complete. 2024-04-25T00:31:20.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.254+0000 7f203ae8c640 1 Processor -- start 2024-04-25T00:31:20.257 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.254+0000 7f203ae8c640 1 -- start start 2024-04-25T00:31:20.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.254+0000 7f203ae8c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2020004d50 0x7f20340c6f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:20.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.254+0000 7f203ae8c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20341513c0 0x7f2034143a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:20.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.254+0000 7f203ae8c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2034151820 0x7f2034143f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:20.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.254+0000 7f203ae8c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2034156dc0 con 0x7f20341513c0 2024-04-25T00:31:20.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.254+0000 7f203ae8c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f203406ba70 con 0x7f2034151820 2024-04-25T00:31:20.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.254+0000 7f203ae8c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f20341579f0 con 0x7f2020004d50 2024-04-25T00:31:20.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.254+0000 7f2033fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20341513c0 0x7f2034143a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:20.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.254+0000 7f2033fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20341513c0 0x7f2034143a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43604/0 (socket says 172.21.15.116:43604) 2024-04-25T00:31:20.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.254+0000 7f2033fff640 1 -- 172.21.15.116:0/2375776971 learned_addr learned my addr 172.21.15.116:0/2375776971 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:31:20.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.254+0000 7f2038c28640 1 --2- 172.21.15.116:0/2375776971 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2020004d50 0x7f20340c6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:20.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.254+0000 7f2039429640 1 --2- 172.21.15.116:0/2375776971 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2034151820 0x7f2034143f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:20.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.258+0000 7f2033fff640 1 -- 172.21.15.116:0/2375776971 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2020004d50 msgr2=0x7f20340c6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:20.259 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.258+0000 7f2033fff640 1 --2- 172.21.15.116:0/2375776971 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2020004d50 0x7f20340c6f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:20.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.258+0000 7f2033fff640 1 -- 172.21.15.116:0/2375776971 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2034151820 msgr2=0x7f2034143f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:20.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.258+0000 7f2033fff640 1 --2- 172.21.15.116:0/2375776971 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2034151820 0x7f2034143f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:20.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.258+0000 7f2033fff640 1 -- 172.21.15.116:0/2375776971 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20340c7990 con 0x7f20341513c0 2024-04-25T00:31:20.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.258+0000 7f2033fff640 1 --2- 172.21.15.116:0/2375776971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20341513c0 0x7f2034143a50 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f20240115e0 tx=0x7f2024011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:20.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.258+0000 7f20317fa640 1 -- 172.21.15.116:0/2375776971 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f202400f8f0 con 0x7f20341513c0 2024-04-25T00:31:20.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.258+0000 7f20317fa640 1 -- 172.21.15.116:0/2375776971 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f202400fa90 con 0x7f20341513c0 2024-04-25T00:31:20.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.258+0000 7f203ae8c640 1 -- 172.21.15.116:0/2375776971 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2034152020 con 0x7f20341513c0 2024-04-25T00:31:20.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.258+0000 7f20317fa640 1 -- 172.21.15.116:0/2375776971 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f202400fe40 con 0x7f20341513c0 2024-04-25T00:31:20.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.258+0000 7f203ae8c640 1 -- 172.21.15.116:0/2375776971 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20341525e0 con 0x7f20341513c0 2024-04-25T00:31:20.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.258+0000 7f203ae8c640 1 -- 172.21.15.116:0/2375776971 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2034068a60 con 0x7f20341513c0 2024-04-25T00:31:20.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.258+0000 7f20317fa640 1 -- 172.21.15.116:0/2375776971 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f2024015460 con 0x7f20341513c0 2024-04-25T00:31:20.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.258+0000 7f20317fa640 1 --2- 172.21.15.116:0/2375776971 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f20140678c0 0x7f2014069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:20.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.258+0000 7f20317fa640 1 -- 172.21.15.116:0/2375776971 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7f2024087950 con 0x7f20341513c0 2024-04-25T00:31:20.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.258+0000 7f2038c28640 1 --2- 172.21.15.116:0/2375776971 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f20140678c0 0x7f2014069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:20.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.258+0000 7f2038c28640 1 --2- 172.21.15.116:0/2375776971 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f20140678c0 0x7f2014069d80 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f201c005630 tx=0x7f201c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:20.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.262+0000 7f20317fa640 1 -- 172.21.15.116:0/2375776971 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f2024019030 con 0x7f20341513c0 2024-04-25T00:31:20.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.498+0000 7f203ae8c640 1 -- 172.21.15.116:0/2375776971 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2034070500 con 0x7f20341513c0 2024-04-25T00:31:20.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.502+0000 7f20317fa640 1 -- 172.21.15.116:0/2375776971 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 242 v242) ==== 95+0+6506 (secure 0 0 0) 0x7f2024057110 con 0x7f20341513c0 2024-04-25T00:31:20.502 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:31:20.502 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":242,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":9027,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/426062493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":426062493},{"type":"v1","addr":"172.21.15.116:6835","nonce":426062493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":242}],"filesystems":[{"mdsmap":{"epoch":236,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:29:54.168353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8847},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8847":{"gid":8847,"name":"c","rank":0,"incarnation":233,"state":"up:active","state_seq":17,"addr":"172.21.15.116:6833/2662028892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2662028892},{"type":"v1","addr":"172.21.15.116:6833","nonce":2662028892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8847,"qdb_cluster":[8847]},"id":1},{"mdsmap":{"epoch":241,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:30:19.111393+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6764},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6764":{"gid":6764,"name":"d","rank":0,"incarnation":238,"state":"up:active","state_seq":8,"addr":"172.21.15.174:6837/3873739116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3873739116},{"type":"v1","addr":"172.21.15.174:6837","nonce":3873739116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6764,"qdb_cluster":[6764]},"id":2}]} 2024-04-25T00:31:20.503 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 242 2024-04-25T00:31:20.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.502+0000 7f203ae8c640 1 -- 172.21.15.116:0/2375776971 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f20140678c0 msgr2=0x7f2014069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:20.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.502+0000 7f203ae8c640 1 --2- 172.21.15.116:0/2375776971 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f20140678c0 0x7f2014069d80 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f201c005630 tx=0x7f201c009040 comp rx=0 tx=0).stop 2024-04-25T00:31:20.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.502+0000 7f203ae8c640 1 -- 172.21.15.116:0/2375776971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20341513c0 msgr2=0x7f2034143a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:20.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.502+0000 7f203ae8c640 1 --2- 172.21.15.116:0/2375776971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20341513c0 0x7f2034143a50 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f20240115e0 tx=0x7f2024011aa0 comp rx=0 tx=0).stop 2024-04-25T00:31:20.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.502+0000 7f203ae8c640 1 -- 172.21.15.116:0/2375776971 shutdown_connections 2024-04-25T00:31:20.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.502+0000 7f203ae8c640 1 --2- 172.21.15.116:0/2375776971 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f20140678c0 0x7f2014069d80 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:20.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.502+0000 7f203ae8c640 1 --2- 172.21.15.116:0/2375776971 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2034151820 0x7f2034143f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:20.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.502+0000 7f203ae8c640 1 --2- 172.21.15.116:0/2375776971 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f20341513c0 0x7f2034143a50 unknown :-1 s=CLOSED pgs=1852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:20.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.502+0000 7f203ae8c640 1 --2- 172.21.15.116:0/2375776971 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f2020004d50 0x7f20340c6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:20.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.502+0000 7f203ae8c640 1 -- 172.21.15.116:0/2375776971 >> 172.21.15.116:0/2375776971 conn(0x7f20340c7fa0 msgr2=0x7f20340c83b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:31:20.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.502+0000 7f203ae8c640 1 -- 172.21.15.116:0/2375776971 shutdown_connections 2024-04-25T00:31:20.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.502+0000 7f203ae8c640 1 -- 172.21.15.116:0/2375776971 wait complete. 2024-04-25T00:31:20.522 INFO:tasks.mds_thrash.fs.[a]:kill mds.c (rank=0) 2024-04-25T00:31:20.522 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-25T00:31:20.522 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:31:20.533 INFO:tasks.ceph.mds.c:Stopped 2024-04-25T00:31:20.533 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:31:20.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.690+0000 7fa9929b5640 1 Processor -- start 2024-04-25T00:31:20.690 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.690+0000 7fa9929b5640 1 -- start start 2024-04-25T00:31:20.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.690+0000 7fa9929b5640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa98c071590 0x7fa98c156180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:20.691 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.690+0000 7fa9929b5640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa98c0a6cc0 con 0x7fa98c1566c0 2024-04-25T00:31:20.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.690+0000 7fa9929b5640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa98c0c6bb0 con 0x7fa98c14fe10 2024-04-25T00:31:20.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.690+0000 7fa9929b5640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fa98c074790 con 0x7fa98c071590 2024-04-25T00:31:20.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.690+0000 7fa98b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa98c071590 0x7fa98c156180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:20.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.690+0000 7fa98b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fa98c071590 0x7fa98c156180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43606/0 (socket says 172.21.15.116:43606) 2024-04-25T00:31:20.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.690+0000 7fa98b7fe640 1 -- 172.21.15.116:0/1874686183 learned_addr learned my addr 172.21.15.116:0/1874686183 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:31:20.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.690+0000 7fa98b7fe640 1 -- 172.21.15.116:0/1874686183 >> v1:172.21.15.174:6789/0 conn(0x7fa98c14fe10 legacy=0x7fa98c06c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:31:20.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.690+0000 7fa98b7fe640 1 -- 172.21.15.116:0/1874686183 >> v1:172.21.15.116:6789/0 conn(0x7fa98c1566c0 legacy=0x7fa98c15cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:20.692 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.690+0000 7fa98b7fe640 1 -- 172.21.15.116:0/1874686183 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa98c0a7150 con 0x7fa98c071590 2024-04-25T00:31:20.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.694+0000 7fa98b7fe640 1 --2- 172.21.15.116:0/1874686183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa98c071590 0x7fa98c156180 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7fa980009870 tx=0x7fa98001c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ef3c9a8f2aa89b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:20.694 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.694+0000 7fa98affd640 1 -- 172.21.15.116:0/1874686183 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9800b0070 con 0x7fa98c071590 2024-04-25T00:31:20.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.694+0000 7fa98affd640 1 -- 172.21.15.116:0/1874686183 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9800ab600 con 0x7fa98c071590 2024-04-25T00:31:20.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.694+0000 7fa98affd640 1 -- 172.21.15.116:0/1874686183 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9800ab9d0 con 0x7fa98c071590 2024-04-25T00:31:20.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.694+0000 7fa9929b5640 1 -- 172.21.15.116:0/1874686183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa98c071590 msgr2=0x7fa98c156180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:20.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.694+0000 7fa9929b5640 1 --2- 172.21.15.116:0/1874686183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa98c071590 0x7fa98c156180 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7fa980009870 tx=0x7fa98001c400 comp rx=0 tx=0).stop 2024-04-25T00:31:20.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.694+0000 7fa9929b5640 1 -- 172.21.15.116:0/1874686183 shutdown_connections 2024-04-25T00:31:20.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.694+0000 7fa9929b5640 1 --2- 172.21.15.116:0/1874686183 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa98c071590 0x7fa98c156180 unknown :-1 s=CLOSED pgs=1854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:20.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.694+0000 7fa9929b5640 1 -- 172.21.15.116:0/1874686183 >> 172.21.15.116:0/1874686183 conn(0x7fa98c0c42e0 msgr2=0x7fa98c0c46f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:31:20.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.694+0000 7fa9929b5640 1 -- 172.21.15.116:0/1874686183 shutdown_connections 2024-04-25T00:31:20.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.694+0000 7fa9929b5640 1 -- 172.21.15.116:0/1874686183 wait complete. 2024-04-25T00:31:20.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.694+0000 7fa9929b5640 1 Processor -- start 2024-04-25T00:31:20.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.694+0000 7fa9929b5640 1 -- start start 2024-04-25T00:31:20.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.694+0000 7fa9929b5640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa98c071590 0x7fa98c148d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:20.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.694+0000 7fa9929b5640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa98c14fe10 0x7fa98c154e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:20.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.694+0000 7fa9929b5640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa98c1566c0 0x7fa98c155390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:20.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.694+0000 7fa9929b5640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fa98c06c930 con 0x7fa98c1566c0 2024-04-25T00:31:20.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.694+0000 7fa9929b5640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fa98c155cf0 con 0x7fa98c071590 2024-04-25T00:31:20.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.694+0000 7fa9929b5640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fa98c06cab0 con 0x7fa98c14fe10 2024-04-25T00:31:20.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.694+0000 7fa990f52640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa98c1566c0 0x7fa98c155390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:20.698 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.694+0000 7fa990f52640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa98c1566c0 0x7fa98c155390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43614/0 (socket says 172.21.15.116:43614) 2024-04-25T00:31:20.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.694+0000 7fa990f52640 1 -- 172.21.15.116:0/1689339679 learned_addr learned my addr 172.21.15.116:0/1689339679 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:31:20.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.694+0000 7fa990f52640 1 -- 172.21.15.116:0/1689339679 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa98c14fe10 msgr2=0x7fa98c154e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:20.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.694+0000 7fa98b7fe640 1 --2- 172.21.15.116:0/1689339679 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa98c14fe10 0x7fa98c154e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:20.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.694+0000 7fa98bfff640 1 --2- 172.21.15.116:0/1689339679 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa98c071590 0x7fa98c148d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:20.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.694+0000 7fa990f52640 1 --2- 172.21.15.116:0/1689339679 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa98c14fe10 0x7fa98c154e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:20.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.694+0000 7fa990f52640 1 -- 172.21.15.116:0/1689339679 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa98c071590 msgr2=0x7fa98c148d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:20.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.694+0000 7fa990f52640 1 --2- 172.21.15.116:0/1689339679 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa98c071590 0x7fa98c148d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:20.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.694+0000 7fa990f52640 1 -- 172.21.15.116:0/1689339679 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa98c1ef220 con 0x7fa98c1566c0 2024-04-25T00:31:20.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.694+0000 7fa990f52640 1 --2- 172.21.15.116:0/1689339679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa98c1566c0 0x7fa98c155390 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7fa98400c730 tx=0x7fa98400cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:20.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.694+0000 7fa988ff9640 1 -- 172.21.15.116:0/1689339679 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa984016020 con 0x7fa98c1566c0 2024-04-25T00:31:20.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.694+0000 7fa988ff9640 1 -- 172.21.15.116:0/1689339679 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa984006d60 con 0x7fa98c1566c0 2024-04-25T00:31:20.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.694+0000 7fa9929b5640 1 -- 172.21.15.116:0/1689339679 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa98c0c10b0 con 0x7fa98c1566c0 2024-04-25T00:31:20.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.694+0000 7fa988ff9640 1 -- 172.21.15.116:0/1689339679 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa984003bc0 con 0x7fa98c1566c0 2024-04-25T00:31:20.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.694+0000 7fa9929b5640 1 -- 172.21.15.116:0/1689339679 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa98c0c1670 con 0x7fa98c1566c0 2024-04-25T00:31:20.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.698+0000 7fa9929b5640 1 -- 172.21.15.116:0/1689339679 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa98c068a60 con 0x7fa98c1566c0 2024-04-25T00:31:20.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.698+0000 7fa988ff9640 1 -- 172.21.15.116:0/1689339679 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fa9840050d0 con 0x7fa98c1566c0 2024-04-25T00:31:20.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.698+0000 7fa988ff9640 1 --2- 172.21.15.116:0/1689339679 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa9780678c0 0x7fa978069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:20.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.698+0000 7fa988ff9640 1 -- 172.21.15.116:0/1689339679 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7fa984086b80 con 0x7fa98c1566c0 2024-04-25T00:31:20.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.698+0000 7fa98bfff640 1 --2- 172.21.15.116:0/1689339679 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa9780678c0 0x7fa978069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:20.701 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.698+0000 7fa98bfff640 1 --2- 172.21.15.116:0/1689339679 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa9780678c0 0x7fa978069d80 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fa97c007fd0 tx=0x7fa97c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:20.703 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.702+0000 7fa988ff9640 1 -- 172.21.15.116:0/1689339679 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fa984020020 con 0x7fa98c1566c0 2024-04-25T00:31:20.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.894+0000 7fa9929b5640 1 -- 172.21.15.116:0/1689339679 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa98c073300 con 0x7fa98c1566c0 2024-04-25T00:31:20.897 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.898+0000 7fa988ff9640 1 -- 172.21.15.116:0/1689339679 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 242 v242) ==== 95+0+6506 (secure 0 0 0) 0x7fa9840079d0 con 0x7fa98c1566c0 2024-04-25T00:31:20.898 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:31:20.898 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":242,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":9027,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/426062493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":426062493},{"type":"v1","addr":"172.21.15.116:6835","nonce":426062493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":242}],"filesystems":[{"mdsmap":{"epoch":236,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:29:54.168353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8847},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8847":{"gid":8847,"name":"c","rank":0,"incarnation":233,"state":"up:active","state_seq":17,"addr":"172.21.15.116:6833/2662028892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2662028892},{"type":"v1","addr":"172.21.15.116:6833","nonce":2662028892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8847,"qdb_cluster":[8847]},"id":1},{"mdsmap":{"epoch":241,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:30:19.111393+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6764},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6764":{"gid":6764,"name":"d","rank":0,"incarnation":238,"state":"up:active","state_seq":8,"addr":"172.21.15.174:6837/3873739116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3873739116},{"type":"v1","addr":"172.21.15.174:6837","nonce":3873739116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6764,"qdb_cluster":[6764]},"id":2}]} 2024-04-25T00:31:20.898 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 242 2024-04-25T00:31:20.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.898+0000 7fa9929b5640 1 -- 172.21.15.116:0/1689339679 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa9780678c0 msgr2=0x7fa978069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:20.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.898+0000 7fa9929b5640 1 --2- 172.21.15.116:0/1689339679 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa9780678c0 0x7fa978069d80 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fa97c007fd0 tx=0x7fa97c009040 comp rx=0 tx=0).stop 2024-04-25T00:31:20.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.898+0000 7fa9929b5640 1 -- 172.21.15.116:0/1689339679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa98c1566c0 msgr2=0x7fa98c155390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:20.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.898+0000 7fa9929b5640 1 --2- 172.21.15.116:0/1689339679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa98c1566c0 0x7fa98c155390 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7fa98400c730 tx=0x7fa98400cb30 comp rx=0 tx=0).stop 2024-04-25T00:31:20.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.898+0000 7fa9929b5640 1 -- 172.21.15.116:0/1689339679 shutdown_connections 2024-04-25T00:31:20.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.898+0000 7fa9929b5640 1 --2- 172.21.15.116:0/1689339679 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fa9780678c0 0x7fa978069d80 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:20.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.898+0000 7fa9929b5640 1 --2- 172.21.15.116:0/1689339679 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fa98c1566c0 0x7fa98c155390 unknown :-1 s=CLOSED pgs=1855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:20.900 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.898+0000 7fa9929b5640 1 --2- 172.21.15.116:0/1689339679 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fa98c14fe10 0x7fa98c154e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:20.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.898+0000 7fa9929b5640 1 --2- 172.21.15.116:0/1689339679 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fa98c071590 0x7fa98c148d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:20.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.898+0000 7fa9929b5640 1 -- 172.21.15.116:0/1689339679 >> 172.21.15.116:0/1689339679 conn(0x7fa98c0c42e0 msgr2=0x7fa98c0c46c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:31:20.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.898+0000 7fa9929b5640 1 -- 172.21.15.116:0/1689339679 shutdown_connections 2024-04-25T00:31:20.901 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:20.898+0000 7fa9929b5640 1 -- 172.21.15.116:0/1689339679 wait complete. 2024-04-25T00:31:20.917 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:31:22.917 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:31:23.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.050+0000 7f3a9ccd8640 1 Processor -- start 2024-04-25T00:31:23.050 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.050+0000 7f3a9ccd8640 1 -- start start 2024-04-25T00:31:23.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.050+0000 7f3a9ccd8640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3a98150d40 0x7f3a98154170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:23.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.050+0000 7f3a9ccd8640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f3a980c4c20 0x7f3a980c6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:23.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.050+0000 7f3a9ccd8640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a98070180 con 0x7f3a98070770 2024-04-25T00:31:23.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.050+0000 7f3a9ccd8640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f3a98071280 con 0x7f3a98150d40 2024-04-25T00:31:23.051 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.050+0000 7f3a9ccd8640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f3a9806ff30 con 0x7f3a980c4c20 2024-04-25T00:31:23.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.050+0000 7f3a9659c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f3a98150d40 0x7f3a98154170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:23.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.050+0000 7f3a96d9d640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f3a98070770 0x7f3a980ab200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:47710/0 (socket says 172.21.15.116:47710) 2024-04-25T00:31:23.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.050+0000 7f3a96d9d640 1 -- 172.21.15.116:0/249574141 learned_addr learned my addr 172.21.15.116:0/249574141 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:31:23.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.050+0000 7f3a9559a640 1 -- 172.21.15.116:0/249574141 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2006096258 0 0) 0x7f3a98070180 con 0x7f3a98070770 2024-04-25T00:31:23.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.050+0000 7f3a95d9b640 1 --2- 172.21.15.116:0/249574141 >> v2:172.21.15.174:3300/0 conn(0x7f3a980c4c20 0x7f3a980c6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:23.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.050+0000 7f3a9559a640 1 -- 172.21.15.116:0/249574141 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3a80003530 con 0x7f3a98070770 2024-04-25T00:31:23.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.050+0000 7f3a9659c640 1 -- 172.21.15.116:0/249574141 >> v2:172.21.15.174:3300/0 conn(0x7f3a980c4c20 msgr2=0x7f3a980c6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:31:23.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.050+0000 7f3a9659c640 1 --2- 172.21.15.116:0/249574141 >> v2:172.21.15.174:3300/0 conn(0x7f3a980c4c20 0x7f3a980c6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:23.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.050+0000 7f3a9659c640 1 -- 172.21.15.116:0/249574141 >> v1:172.21.15.116:6789/0 conn(0x7f3a98070770 legacy=0x7f3a980ab200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:23.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.050+0000 7f3a9659c640 1 -- 172.21.15.116:0/249574141 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a98070460 con 0x7f3a98150d40 2024-04-25T00:31:23.052 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.050+0000 7f3a95d9b640 1 --2- 172.21.15.116:0/249574141 >> v2:172.21.15.174:3300/0 conn(0x7f3a980c4c20 0x7f3a980c6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:31:23.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.050+0000 7f3a9659c640 1 --2- 172.21.15.116:0/249574141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a98150d40 0x7f3a98154170 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f3a980c9610 tx=0x7f3a8801c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=467f71ea1822fa8d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:23.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.054+0000 7f3a9559a640 1 -- 172.21.15.116:0/249574141 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a880b0070 con 0x7f3a98150d40 2024-04-25T00:31:23.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.054+0000 7f3a9559a640 1 -- 172.21.15.116:0/249574141 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a880ab600 con 0x7f3a98150d40 2024-04-25T00:31:23.053 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.054+0000 7f3a9559a640 1 -- 172.21.15.116:0/249574141 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a880ab9b0 con 0x7f3a98150d40 2024-04-25T00:31:23.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.054+0000 7f3a9ccd8640 1 -- 172.21.15.116:0/249574141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a98150d40 msgr2=0x7f3a98154170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:23.054 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.054+0000 7f3a9ccd8640 1 --2- 172.21.15.116:0/249574141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a98150d40 0x7f3a98154170 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f3a980c9610 tx=0x7f3a8801c3f0 comp rx=0 tx=0).stop 2024-04-25T00:31:23.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.054+0000 7f3a9ccd8640 1 -- 172.21.15.116:0/249574141 shutdown_connections 2024-04-25T00:31:23.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.054+0000 7f3a9ccd8640 1 --2- 172.21.15.116:0/249574141 >> v2:172.21.15.174:3300/0 conn(0x7f3a980c4c20 0x7f3a980c6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:23.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.054+0000 7f3a9ccd8640 1 --2- 172.21.15.116:0/249574141 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a98150d40 0x7f3a98154170 unknown :-1 s=CLOSED pgs=1857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:23.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.054+0000 7f3a9ccd8640 1 -- 172.21.15.116:0/249574141 >> 172.21.15.116:0/249574141 conn(0x7f3a9806f0c0 msgr2=0x7f3a9806f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:31:23.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.054+0000 7f3a9ccd8640 1 -- 172.21.15.116:0/249574141 shutdown_connections 2024-04-25T00:31:23.055 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.054+0000 7f3a9ccd8640 1 -- 172.21.15.116:0/249574141 wait complete. 2024-04-25T00:31:23.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.054+0000 7f3a9ccd8640 1 Processor -- start 2024-04-25T00:31:23.056 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.054+0000 7f3a9ccd8640 1 -- start start 2024-04-25T00:31:23.058 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.054+0000 7f3a9ccd8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a98070770 0x7f3a981573a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:23.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.054+0000 7f3a9ccd8640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3a980c4c20 0x7f3a981ec5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:23.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.054+0000 7f3a9ccd8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a98150d40 0x7f3a981ecaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:23.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.054+0000 7f3a9ccd8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3a980aba90 con 0x7f3a98070770 2024-04-25T00:31:23.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.054+0000 7f3a9ccd8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3a980ab910 con 0x7f3a98150d40 2024-04-25T00:31:23.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.054+0000 7f3a9ccd8640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f3a980c7360 con 0x7f3a980c4c20 2024-04-25T00:31:23.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.054+0000 7f3a9659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a98070770 0x7f3a981573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:23.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.054+0000 7f3a9659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a98070770 0x7f3a981573a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:43622/0 (socket says 172.21.15.116:43622) 2024-04-25T00:31:23.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.054+0000 7f3a9659c640 1 -- 172.21.15.116:0/1954629117 learned_addr learned my addr 172.21.15.116:0/1954629117 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:31:23.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.054+0000 7f3a96d9d640 1 --2- 172.21.15.116:0/1954629117 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a98150d40 0x7f3a981ecaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:23.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.054+0000 7f3a9659c640 1 -- 172.21.15.116:0/1954629117 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3a980c4c20 msgr2=0x7f3a981ec5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:23.059 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.054+0000 7f3a95d9b640 1 --2- 172.21.15.116:0/1954629117 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3a980c4c20 0x7f3a981ec5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:23.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.054+0000 7f3a9659c640 1 --2- 172.21.15.116:0/1954629117 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3a980c4c20 0x7f3a981ec5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:23.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.054+0000 7f3a9659c640 1 -- 172.21.15.116:0/1954629117 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a98150d40 msgr2=0x7f3a981ecaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:23.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.054+0000 7f3a9659c640 1 --2- 172.21.15.116:0/1954629117 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a98150d40 0x7f3a981ecaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:23.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.054+0000 7f3a9659c640 1 -- 172.21.15.116:0/1954629117 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a981ef690 con 0x7f3a98070770 2024-04-25T00:31:23.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.054+0000 7f3a96d9d640 1 --2- 172.21.15.116:0/1954629117 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a98150d40 0x7f3a981ecaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:31:23.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.058+0000 7f3a95d9b640 1 --2- 172.21.15.116:0/1954629117 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3a980c4c20 0x7f3a981ec5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:31:23.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.058+0000 7f3a9659c640 1 --2- 172.21.15.116:0/1954629117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a98070770 0x7f3a981573a0 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f3a88019000 tx=0x7f3a88019190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:23.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.058+0000 7f3a86ffd640 1 -- 172.21.15.116:0/1954629117 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a880b9020 con 0x7f3a98070770 2024-04-25T00:31:23.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.058+0000 7f3a86ffd640 1 -- 172.21.15.116:0/1954629117 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a880ab9b0 con 0x7f3a98070770 2024-04-25T00:31:23.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.058+0000 7f3a86ffd640 1 -- 172.21.15.116:0/1954629117 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a880b0040 con 0x7f3a98070770 2024-04-25T00:31:23.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.058+0000 7f3a9ccd8640 1 -- 172.21.15.116:0/1954629117 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a981ef970 con 0x7f3a98070770 2024-04-25T00:31:23.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.058+0000 7f3a9ccd8640 1 -- 172.21.15.116:0/1954629117 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a981eff30 con 0x7f3a98070770 2024-04-25T00:31:23.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.058+0000 7f3a9ccd8640 1 -- 172.21.15.116:0/1954629117 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a98068a60 con 0x7f3a98070770 2024-04-25T00:31:23.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.058+0000 7f3a86ffd640 1 -- 172.21.15.116:0/1954629117 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3a880bc020 con 0x7f3a98070770 2024-04-25T00:31:23.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.058+0000 7f3a86ffd640 1 --2- 172.21.15.116:0/1954629117 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3a780678c0 0x7f3a78069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:23.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.058+0000 7f3a86ffd640 1 -- 172.21.15.116:0/1954629117 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7f3a88120eb0 con 0x7f3a98070770 2024-04-25T00:31:23.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.058+0000 7f3a95d9b640 1 --2- 172.21.15.116:0/1954629117 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3a780678c0 0x7f3a78069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:23.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.058+0000 7f3a95d9b640 1 --2- 172.21.15.116:0/1954629117 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3a780678c0 0x7f3a78069d80 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f3a98154a00 tx=0x7f3a8c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:23.064 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.062+0000 7f3a86ffd640 1 -- 172.21.15.116:0/1954629117 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f3a880c3030 con 0x7f3a98070770 2024-04-25T00:31:23.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.258+0000 7f3a9ccd8640 1 -- 172.21.15.116:0/1954629117 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3a980aa950 con 0x7f3a98070770 2024-04-25T00:31:23.261 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.262+0000 7f3a86ffd640 1 -- 172.21.15.116:0/1954629117 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 242 v242) ==== 95+0+6506 (secure 0 0 0) 0x7f3a88016070 con 0x7f3a98070770 2024-04-25T00:31:23.262 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:31:23.262 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":242,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":9027,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/426062493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":426062493},{"type":"v1","addr":"172.21.15.116:6835","nonce":426062493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":242}],"filesystems":[{"mdsmap":{"epoch":236,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:29:54.168353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8847},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8847":{"gid":8847,"name":"c","rank":0,"incarnation":233,"state":"up:active","state_seq":17,"addr":"172.21.15.116:6833/2662028892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2662028892},{"type":"v1","addr":"172.21.15.116:6833","nonce":2662028892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8847,"qdb_cluster":[8847]},"id":1},{"mdsmap":{"epoch":241,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:30:19.111393+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6764},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6764":{"gid":6764,"name":"d","rank":0,"incarnation":238,"state":"up:active","state_seq":8,"addr":"172.21.15.174:6837/3873739116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3873739116},{"type":"v1","addr":"172.21.15.174:6837","nonce":3873739116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6764,"qdb_cluster":[6764]},"id":2}]} 2024-04-25T00:31:23.262 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 242 2024-04-25T00:31:23.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.262+0000 7f3a9ccd8640 1 -- 172.21.15.116:0/1954629117 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3a780678c0 msgr2=0x7f3a78069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:23.264 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.262+0000 7f3a9ccd8640 1 --2- 172.21.15.116:0/1954629117 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3a780678c0 0x7f3a78069d80 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f3a98154a00 tx=0x7f3a8c008040 comp rx=0 tx=0).stop 2024-04-25T00:31:23.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.262+0000 7f3a9ccd8640 1 -- 172.21.15.116:0/1954629117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a98070770 msgr2=0x7f3a981573a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:23.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.262+0000 7f3a9ccd8640 1 --2- 172.21.15.116:0/1954629117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a98070770 0x7f3a981573a0 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f3a88019000 tx=0x7f3a88019190 comp rx=0 tx=0).stop 2024-04-25T00:31:23.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.262+0000 7f3a9ccd8640 1 -- 172.21.15.116:0/1954629117 shutdown_connections 2024-04-25T00:31:23.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.262+0000 7f3a9ccd8640 1 --2- 172.21.15.116:0/1954629117 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f3a780678c0 0x7f3a78069d80 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:23.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.262+0000 7f3a9ccd8640 1 --2- 172.21.15.116:0/1954629117 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f3a98150d40 0x7f3a981ecaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:23.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.262+0000 7f3a9ccd8640 1 --2- 172.21.15.116:0/1954629117 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3a980c4c20 0x7f3a981ec5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:23.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.262+0000 7f3a9ccd8640 1 --2- 172.21.15.116:0/1954629117 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3a98070770 0x7f3a981573a0 unknown :-1 s=CLOSED pgs=1858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:23.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.262+0000 7f3a9ccd8640 1 -- 172.21.15.116:0/1954629117 >> 172.21.15.116:0/1954629117 conn(0x7f3a9806f0c0 msgr2=0x7f3a980c6380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:31:23.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.262+0000 7f3a9ccd8640 1 -- 172.21.15.116:0/1954629117 shutdown_connections 2024-04-25T00:31:23.265 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:23.262+0000 7f3a9ccd8640 1 -- 172.21.15.116:0/1954629117 wait complete. 2024-04-25T00:31:23.281 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:31:25.282 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:31:25.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.410+0000 7f593e153640 1 Processor -- start 2024-04-25T00:31:25.413 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.410+0000 7f593e153640 1 -- start start 2024-04-25T00:31:25.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.414+0000 7f593e153640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5938150d20 0x7f59380c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:25.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.414+0000 7f593e153640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f59380c2f50 0x7f59381538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:25.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.414+0000 7f593e153640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f59380a94a0 0x7f5938153e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:25.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.414+0000 7f593e153640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f59380c3910 con 0x7f59380a94a0 2024-04-25T00:31:25.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.414+0000 7f593e153640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f59380c16e0 con 0x7f59380c2f50 2024-04-25T00:31:25.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.414+0000 7f593e153640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f59380c1400 con 0x7f5938150d20 2024-04-25T00:31:25.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.414+0000 7f5937fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f59380a94a0 0x7f5938153e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:25.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.414+0000 7f5937fff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f59380a94a0 0x7f5938153e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51016/0 (socket says 172.21.15.116:51016) 2024-04-25T00:31:25.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.414+0000 7f5937fff640 1 -- 172.21.15.116:0/2042811435 learned_addr learned my addr 172.21.15.116:0/2042811435 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:31:25.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.414+0000 7f59377fe640 1 --2- 172.21.15.116:0/2042811435 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5938150d20 0x7f59380c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:25.415 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.414+0000 7f5936ffd640 1 --2- 172.21.15.116:0/2042811435 >> v2:172.21.15.174:3300/0 conn(0x7f59380c2f50 0x7f59381538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:25.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.414+0000 7f5937fff640 1 -- 172.21.15.116:0/2042811435 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5938150d20 msgr2=0x7f59380c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:31:25.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.414+0000 7f5937fff640 1 --2- 172.21.15.116:0/2042811435 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5938150d20 0x7f59380c2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:25.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.414+0000 7f5937fff640 1 -- 172.21.15.116:0/2042811435 >> v2:172.21.15.174:3300/0 conn(0x7f59380c2f50 msgr2=0x7f59381538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:31:25.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.414+0000 7f5937fff640 1 --2- 172.21.15.116:0/2042811435 >> v2:172.21.15.174:3300/0 conn(0x7f59380c2f50 0x7f59381538e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:25.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.414+0000 7f5937fff640 1 -- 172.21.15.116:0/2042811435 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f593806e660 con 0x7f59380a94a0 2024-04-25T00:31:25.416 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.414+0000 7f59377fe640 1 --2- 172.21.15.116:0/2042811435 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5938150d20 0x7f59380c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:31:25.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.414+0000 7f5937fff640 1 --2- 172.21.15.116:0/2042811435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59380a94a0 0x7f5938153e20 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7f5938070620 tx=0x7f592801c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcf05ca0dce49ca8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:25.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.414+0000 7f59367fc640 1 -- 172.21.15.116:0/2042811435 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59280a5740 con 0x7f59380a94a0 2024-04-25T00:31:25.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.414+0000 7f59367fc640 1 -- 172.21.15.116:0/2042811435 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59280a58e0 con 0x7f59380a94a0 2024-04-25T00:31:25.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.414+0000 7f59367fc640 1 -- 172.21.15.116:0/2042811435 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59280a5cb0 con 0x7f59380a94a0 2024-04-25T00:31:25.417 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.418+0000 7f593e153640 1 -- 172.21.15.116:0/2042811435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59380a94a0 msgr2=0x7f5938153e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:25.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.418+0000 7f593e153640 1 --2- 172.21.15.116:0/2042811435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59380a94a0 0x7f5938153e20 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7f5938070620 tx=0x7f592801c9f0 comp rx=0 tx=0).stop 2024-04-25T00:31:25.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.418+0000 7f593e153640 1 -- 172.21.15.116:0/2042811435 shutdown_connections 2024-04-25T00:31:25.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.418+0000 7f593e153640 1 --2- 172.21.15.116:0/2042811435 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f59380a94a0 0x7f5938153e20 unknown :-1 s=CLOSED pgs=1859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:25.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.418+0000 7f593e153640 1 --2- 172.21.15.116:0/2042811435 >> v2:172.21.15.174:3300/0 conn(0x7f59380c2f50 0x7f59381538e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:25.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.418+0000 7f593e153640 1 --2- 172.21.15.116:0/2042811435 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5938150d20 0x7f59380c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:25.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.418+0000 7f593e153640 1 -- 172.21.15.116:0/2042811435 >> 172.21.15.116:0/2042811435 conn(0x7f59380c3b00 msgr2=0x7f59380c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:31:25.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.418+0000 7f593e153640 1 -- 172.21.15.116:0/2042811435 shutdown_connections 2024-04-25T00:31:25.418 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.418+0000 7f593e153640 1 -- 172.21.15.116:0/2042811435 wait complete. 2024-04-25T00:31:25.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.418+0000 7f593e153640 1 Processor -- start 2024-04-25T00:31:25.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.418+0000 7f593e153640 1 -- start start 2024-04-25T00:31:25.421 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.418+0000 7f593e153640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f59380a94a0 0x7f5938152970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:25.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.418+0000 7f593e153640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59380c2f50 0x7f593813ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:25.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.418+0000 7f593e153640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5938150d20 0x7f593813f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:25.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.418+0000 7f593e153640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5938157340 con 0x7f5938150d20 2024-04-25T00:31:25.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.418+0000 7f593e153640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5938154570 con 0x7f59380c2f50 2024-04-25T00:31:25.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.418+0000 7f593e153640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f59381574c0 con 0x7f59380a94a0 2024-04-25T00:31:25.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.418+0000 7f5937fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5938150d20 0x7f593813f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:25.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.418+0000 7f5937fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5938150d20 0x7f593813f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51030/0 (socket says 172.21.15.116:51030) 2024-04-25T00:31:25.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.418+0000 7f5937fff640 1 -- 172.21.15.116:0/558468616 learned_addr learned my addr 172.21.15.116:0/558468616 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:31:25.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.418+0000 7f59377fe640 1 --2- 172.21.15.116:0/558468616 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f59380a94a0 0x7f5938152970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:25.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.418+0000 7f5936ffd640 1 --2- 172.21.15.116:0/558468616 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59380c2f50 0x7f593813ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:25.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.418+0000 7f5937fff640 1 -- 172.21.15.116:0/558468616 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f59380a94a0 msgr2=0x7f5938152970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:25.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.418+0000 7f5937fff640 1 --2- 172.21.15.116:0/558468616 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f59380a94a0 0x7f5938152970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:25.422 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.418+0000 7f5937fff640 1 -- 172.21.15.116:0/558468616 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59380c2f50 msgr2=0x7f593813ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:25.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.418+0000 7f5937fff640 1 --2- 172.21.15.116:0/558468616 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59380c2f50 0x7f593813ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:25.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.418+0000 7f5937fff640 1 -- 172.21.15.116:0/558468616 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5938070f90 con 0x7f5938150d20 2024-04-25T00:31:25.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.418+0000 7f5936ffd640 1 --2- 172.21.15.116:0/558468616 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59380c2f50 0x7f593813ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:31:25.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.418+0000 7f5937fff640 1 --2- 172.21.15.116:0/558468616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5938150d20 0x7f593813f370 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f59280af040 tx=0x7f5928019790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:25.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.418+0000 7f592ffff640 1 -- 172.21.15.116:0/558468616 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59280b8670 con 0x7f5938150d20 2024-04-25T00:31:25.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.418+0000 7f592ffff640 1 -- 172.21.15.116:0/558468616 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59280be070 con 0x7f5938150d20 2024-04-25T00:31:25.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.418+0000 7f593e153640 1 -- 172.21.15.116:0/558468616 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5938071270 con 0x7f5938150d20 2024-04-25T00:31:25.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.418+0000 7f592ffff640 1 -- 172.21.15.116:0/558468616 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59280b8a90 con 0x7f5938150d20 2024-04-25T00:31:25.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.418+0000 7f593e153640 1 -- 172.21.15.116:0/558468616 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5938071830 con 0x7f5938150d20 2024-04-25T00:31:25.423 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.422+0000 7f593e153640 1 -- 172.21.15.116:0/558468616 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5938068350 con 0x7f5938150d20 2024-04-25T00:31:25.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.422+0000 7f592ffff640 1 -- 172.21.15.116:0/558468616 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f59280b8c30 con 0x7f5938150d20 2024-04-25T00:31:25.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.422+0000 7f592ffff640 1 --2- 172.21.15.116:0/558468616 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f59180678c0 0x7f5918069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:25.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.422+0000 7f592ffff640 1 -- 172.21.15.116:0/558468616 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7f5928122180 con 0x7f5938150d20 2024-04-25T00:31:25.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.422+0000 7f59377fe640 1 --2- 172.21.15.116:0/558468616 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f59180678c0 0x7f5918069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:25.424 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.422+0000 7f59377fe640 1 --2- 172.21.15.116:0/558468616 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f59180678c0 0x7f5918069d80 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f5930006fd0 tx=0x7f5930008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:25.427 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.426+0000 7f592ffff640 1 -- 172.21.15.116:0/558468616 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f59280165e0 con 0x7f5938150d20 2024-04-25T00:31:25.616 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.614+0000 7f593e153640 1 -- 172.21.15.116:0/558468616 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f59380c34f0 con 0x7f5938150d20 2024-04-25T00:31:25.617 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.614+0000 7f592ffff640 1 -- 172.21.15.116:0/558468616 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 242 v242) ==== 95+0+6506 (secure 0 0 0) 0x7f59280f13c0 con 0x7f5938150d20 2024-04-25T00:31:25.617 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:31:25.617 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":242,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":9027,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/426062493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":426062493},{"type":"v1","addr":"172.21.15.116:6835","nonce":426062493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":242}],"filesystems":[{"mdsmap":{"epoch":236,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:29:54.168353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8847},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8847":{"gid":8847,"name":"c","rank":0,"incarnation":233,"state":"up:active","state_seq":17,"addr":"172.21.15.116:6833/2662028892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2662028892},{"type":"v1","addr":"172.21.15.116:6833","nonce":2662028892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8847,"qdb_cluster":[8847]},"id":1},{"mdsmap":{"epoch":241,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:30:19.111393+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6764},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6764":{"gid":6764,"name":"d","rank":0,"incarnation":238,"state":"up:active","state_seq":8,"addr":"172.21.15.174:6837/3873739116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3873739116},{"type":"v1","addr":"172.21.15.174:6837","nonce":3873739116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6764,"qdb_cluster":[6764]},"id":2}]} 2024-04-25T00:31:25.618 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 242 2024-04-25T00:31:25.619 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.618+0000 7f593e153640 1 -- 172.21.15.116:0/558468616 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f59180678c0 msgr2=0x7f5918069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:25.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.618+0000 7f593e153640 1 --2- 172.21.15.116:0/558468616 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f59180678c0 0x7f5918069d80 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f5930006fd0 tx=0x7f5930008040 comp rx=0 tx=0).stop 2024-04-25T00:31:25.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.618+0000 7f593e153640 1 -- 172.21.15.116:0/558468616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5938150d20 msgr2=0x7f593813f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:25.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.618+0000 7f593e153640 1 --2- 172.21.15.116:0/558468616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5938150d20 0x7f593813f370 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f59280af040 tx=0x7f5928019790 comp rx=0 tx=0).stop 2024-04-25T00:31:25.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.618+0000 7f593e153640 1 -- 172.21.15.116:0/558468616 shutdown_connections 2024-04-25T00:31:25.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.618+0000 7f593e153640 1 --2- 172.21.15.116:0/558468616 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f59180678c0 0x7f5918069d80 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:25.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.618+0000 7f593e153640 1 --2- 172.21.15.116:0/558468616 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5938150d20 0x7f593813f370 unknown :-1 s=CLOSED pgs=1860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:25.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.618+0000 7f593e153640 1 --2- 172.21.15.116:0/558468616 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f59380c2f50 0x7f593813ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:25.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.618+0000 7f593e153640 1 --2- 172.21.15.116:0/558468616 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f59380a94a0 0x7f5938152970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:25.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.618+0000 7f593e153640 1 -- 172.21.15.116:0/558468616 >> 172.21.15.116:0/558468616 conn(0x7f59380c3b00 msgr2=0x7f5938141530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:31:25.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.618+0000 7f593e153640 1 -- 172.21.15.116:0/558468616 shutdown_connections 2024-04-25T00:31:25.620 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:25.618+0000 7f593e153640 1 -- 172.21.15.116:0/558468616 wait complete. 2024-04-25T00:31:25.636 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:31:27.637 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:31:27.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.766+0000 7f27e809e640 1 Processor -- start 2024-04-25T00:31:27.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.766+0000 7f27e809e640 1 -- start start 2024-04-25T00:31:27.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.770+0000 7f27e809e640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f27e00a94a0 0x7f27e01563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:27.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.770+0000 7f27e809e640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27e0071220 con 0x7f27e014c720 2024-04-25T00:31:27.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.770+0000 7f27e809e640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27e0152420 con 0x7f27e0071680 2024-04-25T00:31:27.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.770+0000 7f27e809e640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f27e0153460 con 0x7f27e00a94a0 2024-04-25T00:31:27.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.770+0000 7f27e5e3a640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f27e014c720 0x7f27e00c2a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:46820/0 (socket says 172.21.15.116:46820) 2024-04-25T00:31:27.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.770+0000 7f27e5e3a640 1 -- 172.21.15.116:0/3562376094 learned_addr learned my addr 172.21.15.116:0/3562376094 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:31:27.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.770+0000 7f27e5639640 1 --2- 172.21.15.116:0/3562376094 >> v2:172.21.15.174:3300/0 conn(0x7f27e00a94a0 0x7f27e01563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:27.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.770+0000 7f27e4e38640 1 -- 172.21.15.116:0/3562376094 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2004543855 0 0) 0x7f27e0071220 con 0x7f27e014c720 2024-04-25T00:31:27.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.770+0000 7f27e4e38640 1 -- 172.21.15.116:0/3562376094 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f27cc003530 con 0x7f27e014c720 2024-04-25T00:31:27.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.770+0000 7f27e4e38640 1 -- 172.21.15.116:0/3562376094 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2681676673 0 0) 0x7f27cc003530 con 0x7f27e014c720 2024-04-25T00:31:27.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.770+0000 7f27e4e38640 1 -- 172.21.15.116:0/3562376094 >> v2:172.21.15.174:3300/0 conn(0x7f27e00a94a0 msgr2=0x7f27e01563d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:31:27.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.770+0000 7f27e4e38640 1 --2- 172.21.15.116:0/3562376094 >> v2:172.21.15.174:3300/0 conn(0x7f27e00a94a0 0x7f27e01563d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:27.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.770+0000 7f27e4e38640 1 -- 172.21.15.116:0/3562376094 >> v1:172.21.15.174:6789/0 conn(0x7f27e0071680 legacy=0x7f27e0071ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:27.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.770+0000 7f27e4e38640 1 -- 172.21.15.116:0/3562376094 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27e00721c0 con 0x7f27e014c720 2024-04-25T00:31:27.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.770+0000 7f27e4e38640 1 -- 172.21.15.116:0/3562376094 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f27dc001680 con 0x7f27e014c720 2024-04-25T00:31:27.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.770+0000 7f27e4e38640 1 -- 172.21.15.116:0/3562376094 >> v1:172.21.15.116:6789/0 conn(0x7f27e014c720 legacy=0x7f27e00c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:27.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.770+0000 7f27e4e38640 1 --2- 172.21.15.116:0/3562376094 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f27cc003a10 0x7f27cc003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:27.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.770+0000 7f27e4e38640 1 --2- 172.21.15.116:0/3562376094 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27cc004810 0x7f27cc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:27.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.770+0000 7f27e4e38640 1 --2- 172.21.15.116:0/3562376094 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f27cc005220 0x7f27cc00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:27.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.770+0000 7f27e4e38640 1 -- 172.21.15.116:0/3562376094 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f27e0153460 con 0x7f27cc004810 2024-04-25T00:31:27.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.770+0000 7f27e4e38640 1 -- 172.21.15.116:0/3562376094 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f27cc00c170 con 0x7f27cc003a10 2024-04-25T00:31:27.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.770+0000 7f27e4e38640 1 -- 172.21.15.116:0/3562376094 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f27cc00c370 con 0x7f27cc005220 2024-04-25T00:31:27.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.770+0000 7f27e5639640 1 --2- 172.21.15.116:0/3562376094 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27cc004810 0x7f27cc004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:27.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.770+0000 7f27e5e3a640 1 --2- 172.21.15.116:0/3562376094 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f27cc003a10 0x7f27cc003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:27.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.770+0000 7f27e663b640 1 --2- 172.21.15.116:0/3562376094 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f27cc005220 0x7f27cc00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:27.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.770+0000 7f27e5639640 1 -- 172.21.15.116:0/3562376094 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f27cc005220 msgr2=0x7f27cc00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:31:27.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.770+0000 7f27e5639640 1 --2- 172.21.15.116:0/3562376094 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f27cc005220 0x7f27cc00b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:27.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.770+0000 7f27e5639640 1 -- 172.21.15.116:0/3562376094 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f27cc003a10 msgr2=0x7f27cc003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:31:27.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.770+0000 7f27e5639640 1 --2- 172.21.15.116:0/3562376094 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f27cc003a10 0x7f27cc003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:27.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.770+0000 7f27e5639640 1 -- 172.21.15.116:0/3562376094 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27cc003530 con 0x7f27cc004810 2024-04-25T00:31:27.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.770+0000 7f27e663b640 1 -- 172.21.15.116:0/3562376094 reap_dead start 2024-04-25T00:31:27.774 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.770+0000 7f27e5e3a640 1 --2- 172.21.15.116:0/3562376094 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f27cc003a10 0x7f27cc003e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:31:27.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.774+0000 7f27e5639640 1 --2- 172.21.15.116:0/3562376094 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27cc004810 0x7f27cc004c90 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7f27d0009ab0 tx=0x7f27d001ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86b90e8d23f2b17e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:27.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.774+0000 7f27e4e38640 1 -- 172.21.15.116:0/3562376094 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d00b2070 con 0x7f27cc004810 2024-04-25T00:31:27.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.774+0000 7f27e4e38640 1 -- 172.21.15.116:0/3562376094 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27d00a5620 con 0x7f27cc004810 2024-04-25T00:31:27.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.774+0000 7f27e4e38640 1 -- 172.21.15.116:0/3562376094 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d00096b0 con 0x7f27cc004810 2024-04-25T00:31:27.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.774+0000 7f27e809e640 1 -- 172.21.15.116:0/3562376094 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27cc004810 msgr2=0x7f27cc004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:27.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.774+0000 7f27e809e640 1 --2- 172.21.15.116:0/3562376094 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27cc004810 0x7f27cc004c90 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7f27d0009ab0 tx=0x7f27d001ca50 comp rx=0 tx=0).stop 2024-04-25T00:31:27.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.774+0000 7f27e809e640 1 -- 172.21.15.116:0/3562376094 shutdown_connections 2024-04-25T00:31:27.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.774+0000 7f27e809e640 1 --2- 172.21.15.116:0/3562376094 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27cc004810 0x7f27cc004c90 unknown :-1 s=CLOSED pgs=1862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:27.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.774+0000 7f27e809e640 1 -- 172.21.15.116:0/3562376094 >> 172.21.15.116:0/3562376094 conn(0x7f27e00c3b00 msgr2=0x7f27e00c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:31:27.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.774+0000 7f27e809e640 1 -- 172.21.15.116:0/3562376094 shutdown_connections 2024-04-25T00:31:27.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.774+0000 7f27e809e640 1 -- 172.21.15.116:0/3562376094 wait complete. 2024-04-25T00:31:27.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.774+0000 7f27e809e640 1 Processor -- start 2024-04-25T00:31:27.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.774+0000 7f27e809e640 1 -- start start 2024-04-25T00:31:27.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.774+0000 7f27e809e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f27cc004810 0x7f27e01ed030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:27.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.774+0000 7f27e809e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f27e01469d0 0x7f27e01474c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:27.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.774+0000 7f27e809e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27e006ceb0 0x7f27e00c5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:27.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.774+0000 7f27e809e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f27e0152980 con 0x7f27e006ceb0 2024-04-25T00:31:27.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.774+0000 7f27e809e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f27e00c3220 con 0x7f27cc004810 2024-04-25T00:31:27.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.774+0000 7f27e809e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f27e0152c60 con 0x7f27e01469d0 2024-04-25T00:31:27.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.774+0000 7f27e663b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27e006ceb0 0x7f27e00c5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:27.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.774+0000 7f27e663b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27e006ceb0 0x7f27e00c5590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51052/0 (socket says 172.21.15.116:51052) 2024-04-25T00:31:27.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.774+0000 7f27e663b640 1 -- 172.21.15.116:0/567650133 learned_addr learned my addr 172.21.15.116:0/567650133 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:31:27.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.774+0000 7f27e5e3a640 1 --2- 172.21.15.116:0/567650133 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f27cc004810 0x7f27e01ed030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:27.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.774+0000 7f27e5639640 1 --2- 172.21.15.116:0/567650133 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f27e01469d0 0x7f27e01474c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:27.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.774+0000 7f27e663b640 1 -- 172.21.15.116:0/567650133 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f27e01469d0 msgr2=0x7f27e01474c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:27.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.774+0000 7f27e663b640 1 --2- 172.21.15.116:0/567650133 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f27e01469d0 0x7f27e01474c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:27.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.774+0000 7f27e663b640 1 -- 172.21.15.116:0/567650133 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f27cc004810 msgr2=0x7f27e01ed030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:27.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.774+0000 7f27e663b640 1 --2- 172.21.15.116:0/567650133 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f27cc004810 0x7f27e01ed030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:27.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.774+0000 7f27e663b640 1 -- 172.21.15.116:0/567650133 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27e00c6730 con 0x7f27e006ceb0 2024-04-25T00:31:27.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.774+0000 7f27e663b640 1 --2- 172.21.15.116:0/567650133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27e006ceb0 0x7f27e00c5590 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f27e00c35f0 tx=0x7f27d800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:27.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.774+0000 7f27d67fc640 1 -- 172.21.15.116:0/567650133 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d8003b80 con 0x7f27e006ceb0 2024-04-25T00:31:27.779 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.774+0000 7f27d67fc640 1 -- 172.21.15.116:0/567650133 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27d8017070 con 0x7f27e006ceb0 2024-04-25T00:31:27.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.774+0000 7f27d67fc640 1 -- 172.21.15.116:0/567650133 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27d8005070 con 0x7f27e006ceb0 2024-04-25T00:31:27.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.778+0000 7f27e809e640 1 -- 172.21.15.116:0/567650133 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27e00c6a10 con 0x7f27e006ceb0 2024-04-25T00:31:27.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.778+0000 7f27e809e640 1 -- 172.21.15.116:0/567650133 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27e00c5070 con 0x7f27e006ceb0 2024-04-25T00:31:27.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.778+0000 7f27e809e640 1 -- 172.21.15.116:0/567650133 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27e00c5270 con 0x7f27e006ceb0 2024-04-25T00:31:27.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.778+0000 7f27d67fc640 1 -- 172.21.15.116:0/567650133 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f27d8005210 con 0x7f27e006ceb0 2024-04-25T00:31:27.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.778+0000 7f27d67fc640 1 --2- 172.21.15.116:0/567650133 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f27c40678c0 0x7f27c4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:27.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.778+0000 7f27d67fc640 1 -- 172.21.15.116:0/567650133 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7f27d8086e70 con 0x7f27e006ceb0 2024-04-25T00:31:27.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.778+0000 7f27e5e3a640 1 --2- 172.21.15.116:0/567650133 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f27c40678c0 0x7f27c4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:27.781 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.778+0000 7f27e5e3a640 1 --2- 172.21.15.116:0/567650133 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f27c40678c0 0x7f27c4069d80 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f27e0050eb0 tx=0x7f27dc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:27.783 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.782+0000 7f27d67fc640 1 -- 172.21.15.116:0/567650133 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f27e00c5270 con 0x7f27e006ceb0 2024-04-25T00:31:27.980 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.978+0000 7f27e809e640 1 -- 172.21.15.116:0/567650133 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f27e0068350 con 0x7f27e006ceb0 2024-04-25T00:31:27.981 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.982+0000 7f27d67fc640 1 -- 172.21.15.116:0/567650133 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 242 v242) ==== 95+0+6506 (secure 0 0 0) 0x7f27e0152420 con 0x7f27e006ceb0 2024-04-25T00:31:27.982 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:31:27.982 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":242,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":9027,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/426062493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":426062493},{"type":"v1","addr":"172.21.15.116:6835","nonce":426062493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":242}],"filesystems":[{"mdsmap":{"epoch":236,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:29:54.168353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8847},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8847":{"gid":8847,"name":"c","rank":0,"incarnation":233,"state":"up:active","state_seq":17,"addr":"172.21.15.116:6833/2662028892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2662028892},{"type":"v1","addr":"172.21.15.116:6833","nonce":2662028892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8847,"qdb_cluster":[8847]},"id":1},{"mdsmap":{"epoch":241,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:30:19.111393+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6764},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6764":{"gid":6764,"name":"d","rank":0,"incarnation":238,"state":"up:active","state_seq":8,"addr":"172.21.15.174:6837/3873739116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3873739116},{"type":"v1","addr":"172.21.15.174:6837","nonce":3873739116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6764,"qdb_cluster":[6764]},"id":2}]} 2024-04-25T00:31:27.982 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 242 2024-04-25T00:31:27.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.982+0000 7f27e809e640 1 -- 172.21.15.116:0/567650133 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f27c40678c0 msgr2=0x7f27c4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:27.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.982+0000 7f27e809e640 1 --2- 172.21.15.116:0/567650133 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f27c40678c0 0x7f27c4069d80 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f27e0050eb0 tx=0x7f27dc009040 comp rx=0 tx=0).stop 2024-04-25T00:31:27.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.982+0000 7f27e809e640 1 -- 172.21.15.116:0/567650133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27e006ceb0 msgr2=0x7f27e00c5590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:27.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.982+0000 7f27e809e640 1 --2- 172.21.15.116:0/567650133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27e006ceb0 0x7f27e00c5590 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f27e00c35f0 tx=0x7f27d800dc00 comp rx=0 tx=0).stop 2024-04-25T00:31:27.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.982+0000 7f27e809e640 1 -- 172.21.15.116:0/567650133 shutdown_connections 2024-04-25T00:31:27.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.982+0000 7f27e809e640 1 --2- 172.21.15.116:0/567650133 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f27c40678c0 0x7f27c4069d80 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:27.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.982+0000 7f27e809e640 1 --2- 172.21.15.116:0/567650133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f27e006ceb0 0x7f27e00c5590 unknown :-1 s=CLOSED pgs=1863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:27.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.982+0000 7f27e809e640 1 --2- 172.21.15.116:0/567650133 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f27e01469d0 0x7f27e01474c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:27.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.982+0000 7f27e809e640 1 --2- 172.21.15.116:0/567650133 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f27cc004810 0x7f27e01ed030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:27.984 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.982+0000 7f27e809e640 1 -- 172.21.15.116:0/567650133 >> 172.21.15.116:0/567650133 conn(0x7f27e00c3b00 msgr2=0x7f27e00c21a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:31:27.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.982+0000 7f27e809e640 1 -- 172.21.15.116:0/567650133 shutdown_connections 2024-04-25T00:31:27.985 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:27.982+0000 7f27e809e640 1 -- 172.21.15.116:0/567650133 wait complete. 2024-04-25T00:31:28.000 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2024-04-25T00:31:28.031 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T00:31:28.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.138+0000 7f567ac44640 1 Processor -- start 2024-04-25T00:31:28.141 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.138+0000 7f567ac44640 1 -- start start 2024-04-25T00:31:28.142 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.142+0000 7f567ac44640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f5674150d20 0x7f56740c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:28.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.142+0000 7f567ac44640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56740a94a0 0x7f5674153ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:28.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.142+0000 7f567ac44640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56740c3910 con 0x7f56740c2f50 2024-04-25T00:31:28.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.142+0000 7f567ac44640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f5674154530 con 0x7f5674150d20 2024-04-25T00:31:28.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.142+0000 7f567ac44640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f56740c16e0 con 0x7f56740a94a0 2024-04-25T00:31:28.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.142+0000 7f56791e1640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56740a94a0 0x7f5674153ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:28.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.142+0000 7f56789e0640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f5674150d20 0x7f56740c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:28.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.142+0000 7f56789e0640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f5674150d20 0x7f56740c2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3300/0 says I am v2:172.21.15.116:39606/0 (socket says 172.21.15.116:39606) 2024-04-25T00:31:28.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.142+0000 7f56789e0640 1 -- 172.21.15.116:0/3203026981 learned_addr learned my addr 172.21.15.116:0/3203026981 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:31:28.143 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.142+0000 7f5673fff640 1 --1- 172.21.15.116:0/3203026981 >> v1:172.21.15.174:6789/0 conn(0x7f56740c2f50 0x7f56741538e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.174:6789/0 says I am v1:172.21.15.116:43198/0 (socket says 172.21.15.116:43198) 2024-04-25T00:31:28.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.142+0000 7f56789e0640 1 -- 172.21.15.116:0/3203026981 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56740a94a0 msgr2=0x7f5674153ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:31:28.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.142+0000 7f56789e0640 1 --2- 172.21.15.116:0/3203026981 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56740a94a0 0x7f5674153ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:28.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.142+0000 7f56789e0640 1 -- 172.21.15.116:0/3203026981 >> v1:172.21.15.174:6789/0 conn(0x7f56740c2f50 legacy=0x7f56741538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:28.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.142+0000 7f56789e0640 1 -- 172.21.15.116:0/3203026981 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56740c1400 con 0x7f5674150d20 2024-04-25T00:31:28.144 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.142+0000 7f56791e1640 1 --2- 172.21.15.116:0/3203026981 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56740a94a0 0x7f5674153ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T00:31:28.145 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.146+0000 7f56789e0640 1 --2- 172.21.15.116:0/3203026981 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5674150d20 0x7f56740c2a10 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f5664009870 tx=0x7f566401c560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7da90be753f6b08f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:28.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.146+0000 7f56737fe640 1 -- 172.21.15.116:0/3203026981 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56640a7070 con 0x7f5674150d20 2024-04-25T00:31:28.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.146+0000 7f56737fe640 1 -- 172.21.15.116:0/3203026981 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56640b0070 con 0x7f5674150d20 2024-04-25T00:31:28.146 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.146+0000 7f56737fe640 1 -- 172.21.15.116:0/3203026981 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56640ab890 con 0x7f5674150d20 2024-04-25T00:31:28.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.146+0000 7f567ac44640 1 -- 172.21.15.116:0/3203026981 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5674150d20 msgr2=0x7f56740c2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:28.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.146+0000 7f567ac44640 1 --2- 172.21.15.116:0/3203026981 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5674150d20 0x7f56740c2a10 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f5664009870 tx=0x7f566401c560 comp rx=0 tx=0).stop 2024-04-25T00:31:28.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.146+0000 7f567ac44640 1 -- 172.21.15.116:0/3203026981 shutdown_connections 2024-04-25T00:31:28.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.146+0000 7f567ac44640 1 --2- 172.21.15.116:0/3203026981 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f56740a94a0 0x7f5674153ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:28.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.146+0000 7f567ac44640 1 --2- 172.21.15.116:0/3203026981 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5674150d20 0x7f56740c2a10 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:28.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.146+0000 7f567ac44640 1 -- 172.21.15.116:0/3203026981 >> 172.21.15.116:0/3203026981 conn(0x7f56740c3b00 msgr2=0x7f56740c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:31:28.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.146+0000 7f567ac44640 1 -- 172.21.15.116:0/3203026981 shutdown_connections 2024-04-25T00:31:28.147 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.146+0000 7f567ac44640 1 -- 172.21.15.116:0/3203026981 wait complete. 2024-04-25T00:31:28.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.146+0000 7f567ac44640 1 Processor -- start 2024-04-25T00:31:28.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.146+0000 7f567ac44640 1 -- start start 2024-04-25T00:31:28.149 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.146+0000 7f567ac44640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f56740a94a0 0x7f5674140740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:28.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.146+0000 7f567ac44640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56740c2f50 0x7f5674141650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:28.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.146+0000 7f567ac44640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5674150d20 0x7f5674148470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:28.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.146+0000 7f567ac44640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f56741608b0 con 0x7f56740c2f50 2024-04-25T00:31:28.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.146+0000 7f567ac44640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5674160730 con 0x7f56740a94a0 2024-04-25T00:31:28.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.146+0000 7f567ac44640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f5674160a30 con 0x7f5674150d20 2024-04-25T00:31:28.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.146+0000 7f5673fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56740c2f50 0x7f5674141650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:28.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.146+0000 7f5673fff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56740c2f50 0x7f5674141650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51056/0 (socket says 172.21.15.116:51056) 2024-04-25T00:31:28.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.146+0000 7f5673fff640 1 -- 172.21.15.116:0/3924327097 learned_addr learned my addr 172.21.15.116:0/3924327097 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:31:28.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.146+0000 7f56789e0640 1 --2- 172.21.15.116:0/3924327097 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f56740a94a0 0x7f5674140740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:28.150 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.146+0000 7f56791e1640 1 --2- 172.21.15.116:0/3924327097 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5674150d20 0x7f5674148470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:28.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.146+0000 7f5673fff640 1 -- 172.21.15.116:0/3924327097 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5674150d20 msgr2=0x7f5674148470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:28.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.146+0000 7f5673fff640 1 --2- 172.21.15.116:0/3924327097 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5674150d20 0x7f5674148470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:28.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.146+0000 7f5673fff640 1 -- 172.21.15.116:0/3924327097 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f56740a94a0 msgr2=0x7f5674140740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:28.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.146+0000 7f5673fff640 1 --2- 172.21.15.116:0/3924327097 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f56740a94a0 0x7f5674140740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:28.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.146+0000 7f5673fff640 1 -- 172.21.15.116:0/3924327097 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5674141dc0 con 0x7f56740c2f50 2024-04-25T00:31:28.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.146+0000 7f56789e0640 1 --2- 172.21.15.116:0/3924327097 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f56740a94a0 0x7f5674140740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:31:28.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.146+0000 7f5673fff640 1 --2- 172.21.15.116:0/3924327097 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56740c2f50 0x7f5674141650 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f566800c450 tx=0x7f566800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:28.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.146+0000 7f56717fa640 1 -- 172.21.15.116:0/3924327097 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5668018070 con 0x7f56740c2f50 2024-04-25T00:31:28.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.150+0000 7f56717fa640 1 -- 172.21.15.116:0/3924327097 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5668007480 con 0x7f56740c2f50 2024-04-25T00:31:28.151 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.150+0000 7f567ac44640 1 -- 172.21.15.116:0/3924327097 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56741437f0 con 0x7f56740c2f50 2024-04-25T00:31:28.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.150+0000 7f56717fa640 1 -- 172.21.15.116:0/3924327097 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56680047f0 con 0x7f56740c2f50 2024-04-25T00:31:28.152 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.150+0000 7f567ac44640 1 -- 172.21.15.116:0/3924327097 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56741489b0 con 0x7f56740c2f50 2024-04-25T00:31:28.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.150+0000 7f567ac44640 1 -- 172.21.15.116:0/3924327097 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5674068350 con 0x7f56740c2f50 2024-04-25T00:31:28.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.150+0000 7f56717fa640 1 -- 172.21.15.116:0/3924327097 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f566801b070 con 0x7f56740c2f50 2024-04-25T00:31:28.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.150+0000 7f56717fa640 1 --2- 172.21.15.116:0/3924327097 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f56600678c0 0x7f5660069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:28.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.150+0000 7f56717fa640 1 -- 172.21.15.116:0/3924327097 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7f5668086270 con 0x7f56740c2f50 2024-04-25T00:31:28.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.150+0000 7f56789e0640 1 --2- 172.21.15.116:0/3924327097 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f56600678c0 0x7f5660069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:28.153 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.150+0000 7f56789e0640 1 --2- 172.21.15.116:0/3924327097 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f56600678c0 0x7f5660069d80 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f56640af000 tx=0x7f56640a7000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:28.155 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.154+0000 7f56717fa640 1 -- 172.21.15.116:0/3924327097 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f5668055a30 con 0x7f56740c2f50 2024-04-25T00:31:28.300 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.298+0000 7f567ac44640 1 -- 172.21.15.116:0/3924327097 --> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f56741420a0 con 0x7f56600678c0 2024-04-25T00:31:28.334 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.334+0000 7f56717fa640 1 -- 172.21.15.116:0/3924327097 <== mgr.4105 v2:172.21.15.174:6832/2004448211 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+495213 (secure 0 0 0) 0x7f56741420a0 con 0x7f56600678c0 2024-04-25T00:31:28.339 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:31:28.343 INFO:teuthology.orchestra.run.smithi116.stderr:dumped all 2024-04-25T00:31:28.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.338+0000 7f567ac44640 1 -- 172.21.15.116:0/3924327097 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f56600678c0 msgr2=0x7f5660069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:28.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.338+0000 7f567ac44640 1 --2- 172.21.15.116:0/3924327097 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f56600678c0 0x7f5660069d80 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f56640af000 tx=0x7f56640a7000 comp rx=0 tx=0).stop 2024-04-25T00:31:28.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.338+0000 7f567ac44640 1 -- 172.21.15.116:0/3924327097 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56740c2f50 msgr2=0x7f5674141650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:28.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.338+0000 7f567ac44640 1 --2- 172.21.15.116:0/3924327097 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56740c2f50 0x7f5674141650 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f566800c450 tx=0x7f566800c910 comp rx=0 tx=0).stop 2024-04-25T00:31:28.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.338+0000 7f567ac44640 1 -- 172.21.15.116:0/3924327097 shutdown_connections 2024-04-25T00:31:28.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.338+0000 7f567ac44640 1 --2- 172.21.15.116:0/3924327097 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f56600678c0 0x7f5660069d80 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:28.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.338+0000 7f567ac44640 1 --2- 172.21.15.116:0/3924327097 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5674150d20 0x7f5674148470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:28.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.338+0000 7f567ac44640 1 --2- 172.21.15.116:0/3924327097 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f56740c2f50 0x7f5674141650 unknown :-1 s=CLOSED pgs=1864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:28.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.338+0000 7f567ac44640 1 --2- 172.21.15.116:0/3924327097 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f56740a94a0 0x7f5674140740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:28.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.338+0000 7f567ac44640 1 -- 172.21.15.116:0/3924327097 >> 172.21.15.116:0/3924327097 conn(0x7f56740c3b00 msgr2=0x7f567415d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:31:28.345 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.338+0000 7f567ac44640 1 -- 172.21.15.116:0/3924327097 shutdown_connections 2024-04-25T00:31:28.346 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.338+0000 7f567ac44640 1 -- 172.21.15.116:0/3924327097 wait complete. 2024-04-25T00:31:28.354 INFO:teuthology.orchestra.run.smithi116.stdout:{"pg_ready":true,"pg_map":{"version":986,"stamp":"2024-04-25T00:31:28.049873+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":39263623,"num_objects":56,"num_object_clones":0,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":3556,"num_read_kb":487326,"num_write":2966738,"num_write_kb":38127638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10000,"num_omap_keys":20,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":551349,"ondisk_log_size":551349,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":386,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":7166992,"kb_used_data":84432,"kb_used_omap":12,"kb_used_meta":7082483,"kb_avail":747807728,"statfs":{"total":773094113280,"available":765755113472,"internally_reserved":0,"allocated":86458368,"data_stored":82755206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":7252463153},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005470"},"pg_stats":[{"pgid":"5.18","version":"74'4491","reported_seq":4621,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:54.129321+0000","last_change":"2024-04-25T00:30:54.129382+0000","last_active":"2024-04-25T00:30:54.129321+0000","last_peered":"2024-04-25T00:30:54.129321+0000","last_clean":"2024-04-25T00:30:54.129321+0000","last_became_active":"2024-04-25T00:01:19.508390+0000","last_became_peered":"2024-04-25T00:01:19.508390+0000","last_unstale":"2024-04-25T00:30:54.129321+0000","last_undegraded":"2024-04-25T00:30:54.129321+0000","last_fullsized":"2024-04-25T00:30:54.129321+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4491","last_scrub_stamp":"2024-04-25T00:30:54.129109+0000","last_deep_scrub":"74'4491","last_deep_scrub_stamp":"2024-04-25T00:30:54.129109+0000","last_clean_scrub_stamp":"2024-04-25T00:30:54.129109+0000","objects_scrubbed":0,"log_size":4491,"log_dups_size":0,"ondisk_log_size":4491,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:47:08.927771+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":12,"num_write":23917,"num_write_kb":321110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1f","version":"74'1090","reported_seq":1285,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:56.942373+0000","last_change":"2024-04-25T00:30:56.942455+0000","last_active":"2024-04-25T00:30:56.942373+0000","last_peered":"2024-04-25T00:30:56.942373+0000","last_clean":"2024-04-25T00:30:56.942373+0000","last_became_active":"2024-04-25T00:01:04.355293+0000","last_became_peered":"2024-04-25T00:01:04.355293+0000","last_unstale":"2024-04-25T00:30:56.942373+0000","last_undegraded":"2024-04-25T00:30:56.942373+0000","last_fullsized":"2024-04-25T00:30:56.942373+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1090","last_scrub_stamp":"2024-04-25T00:30:56.942114+0000","last_deep_scrub":"74'1090","last_deep_scrub_stamp":"2024-04-25T00:30:56.942114+0000","last_clean_scrub_stamp":"2024-04-25T00:30:56.942114+0000","objects_scrubbed":3,"log_size":1090,"log_dups_size":0,"ondisk_log_size":1090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:36:32.511817+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":16397,"num_write":1061,"num_write_kb":4715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"70'3760","reported_seq":3898,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:57.662705+0000","last_change":"2024-04-25T00:30:57.662705+0000","last_active":"2024-04-25T00:30:57.662705+0000","last_peered":"2024-04-25T00:30:57.662705+0000","last_clean":"2024-04-25T00:30:57.662705+0000","last_became_active":"2024-04-25T00:01:05.366471+0000","last_became_peered":"2024-04-25T00:01:05.366471+0000","last_unstale":"2024-04-25T00:30:57.662705+0000","last_undegraded":"2024-04-25T00:30:57.662705+0000","last_fullsized":"2024-04-25T00:30:57.662705+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'3760","last_scrub_stamp":"2024-04-25T00:30:57.662289+0000","last_deep_scrub":"70'3760","last_deep_scrub_stamp":"2024-04-25T00:30:57.662289+0000","last_clean_scrub_stamp":"2024-04-25T00:30:57.662289+0000","objects_scrubbed":0,"log_size":3760,"log_dups_size":0,"ondisk_log_size":3760,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:02:40.045444+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21072,"num_write_kb":264901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.19","version":"74'3","reported_seq":132,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:47.917551+0000","last_change":"2024-04-25T00:30:47.917609+0000","last_active":"2024-04-25T00:30:47.917551+0000","last_peered":"2024-04-25T00:30:47.917551+0000","last_clean":"2024-04-25T00:30:47.917551+0000","last_became_active":"2024-04-25T00:01:18.501633+0000","last_became_peered":"2024-04-25T00:01:18.501633+0000","last_unstale":"2024-04-25T00:30:47.917551+0000","last_undegraded":"2024-04-25T00:30:47.917551+0000","last_fullsized":"2024-04-25T00:30:47.917551+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'3","last_scrub_stamp":"2024-04-25T00:30:47.917353+0000","last_deep_scrub":"74'3","last_deep_scrub_stamp":"2024-04-25T00:30:47.917353+0000","last_clean_scrub_stamp":"2024-04-25T00:30:47.917353+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:33:11.613082+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"74'5031","reported_seq":5155,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:12.751898+0000","last_change":"2024-04-25T00:31:12.751940+0000","last_active":"2024-04-25T00:31:12.751898+0000","last_peered":"2024-04-25T00:31:12.751898+0000","last_clean":"2024-04-25T00:31:12.751898+0000","last_became_active":"2024-04-25T00:01:46.661167+0000","last_became_peered":"2024-04-25T00:01:46.661167+0000","last_unstale":"2024-04-25T00:31:12.751898+0000","last_undegraded":"2024-04-25T00:31:12.751898+0000","last_fullsized":"2024-04-25T00:31:12.751898+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5031","last_scrub_stamp":"2024-04-25T00:31:12.751748+0000","last_deep_scrub":"74'5031","last_deep_scrub_stamp":"2024-04-25T00:31:12.751748+0000","last_clean_scrub_stamp":"2024-04-25T00:31:12.751748+0000","objects_scrubbed":0,"log_size":5031,"log_dups_size":0,"ondisk_log_size":5031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:29:54.310356+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":22,"num_write":27250,"num_write_kb":338892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"73'28","reported_seq":168,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:59.688091+0000","last_change":"2024-04-25T00:30:59.688091+0000","last_active":"2024-04-25T00:30:59.688091+0000","last_peered":"2024-04-25T00:30:59.688091+0000","last_clean":"2024-04-25T00:30:59.688091+0000","last_became_active":"2024-04-25T00:01:04.361059+0000","last_became_peered":"2024-04-25T00:01:04.361059+0000","last_unstale":"2024-04-25T00:30:59.688091+0000","last_undegraded":"2024-04-25T00:30:59.688091+0000","last_fullsized":"2024-04-25T00:30:59.688091+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'28","last_scrub_stamp":"2024-04-25T00:30:59.687701+0000","last_deep_scrub":"73'28","last_deep_scrub_stamp":"2024-04-25T00:30:59.687701+0000","last_clean_scrub_stamp":"2024-04-25T00:30:59.687701+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:55:01.388407+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1f","version":"70'5376","reported_seq":5513,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:53.537262+0000","last_change":"2024-04-25T00:30:53.537392+0000","last_active":"2024-04-25T00:30:53.537262+0000","last_peered":"2024-04-25T00:30:53.537262+0000","last_clean":"2024-04-25T00:30:53.537262+0000","last_became_active":"2024-04-25T00:01:05.369991+0000","last_became_peered":"2024-04-25T00:01:05.369991+0000","last_unstale":"2024-04-25T00:30:53.537262+0000","last_undegraded":"2024-04-25T00:30:53.537262+0000","last_fullsized":"2024-04-25T00:30:53.537262+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'5376","last_scrub_stamp":"2024-04-25T00:30:53.536989+0000","last_deep_scrub":"70'5376","last_deep_scrub_stamp":"2024-04-25T00:30:53.536989+0000","last_clean_scrub_stamp":"2024-04-25T00:30:53.536989+0000","objects_scrubbed":0,"log_size":5376,"log_dups_size":0,"ondisk_log_size":5376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:28:29.688314+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29984,"num_write_kb":386541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"74'2","reported_seq":131,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:47.445142+0000","last_change":"2024-04-25T00:30:47.445199+0000","last_active":"2024-04-25T00:30:47.445142+0000","last_peered":"2024-04-25T00:30:47.445142+0000","last_clean":"2024-04-25T00:30:47.445142+0000","last_became_active":"2024-04-25T00:01:18.497397+0000","last_became_peered":"2024-04-25T00:01:18.497397+0000","last_unstale":"2024-04-25T00:30:47.445142+0000","last_undegraded":"2024-04-25T00:30:47.445142+0000","last_fullsized":"2024-04-25T00:30:47.445142+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'2","last_scrub_stamp":"2024-04-25T00:30:47.444936+0000","last_deep_scrub":"74'2","last_deep_scrub_stamp":"2024-04-25T00:30:47.444936+0000","last_clean_scrub_stamp":"2024-04-25T00:30:47.444936+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:32:50.758882+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"74'4080","reported_seq":4209,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:57.290275+0000","last_change":"2024-04-25T00:30:57.290275+0000","last_active":"2024-04-25T00:30:57.290275+0000","last_peered":"2024-04-25T00:30:57.290275+0000","last_clean":"2024-04-25T00:30:57.290275+0000","last_became_active":"2024-04-25T00:01:19.510252+0000","last_became_peered":"2024-04-25T00:01:19.510252+0000","last_unstale":"2024-04-25T00:30:57.290275+0000","last_undegraded":"2024-04-25T00:30:57.290275+0000","last_fullsized":"2024-04-25T00:30:57.290275+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4080","last_scrub_stamp":"2024-04-25T00:30:57.290098+0000","last_deep_scrub":"74'4080","last_deep_scrub_stamp":"2024-04-25T00:30:57.290098+0000","last_clean_scrub_stamp":"2024-04-25T00:30:57.290098+0000","objects_scrubbed":0,"log_size":4080,"log_dups_size":0,"ondisk_log_size":4080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:44:28.936028+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":21789,"num_write_kb":296332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"74'54","reported_seq":214,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:53.324192+0000","last_change":"2024-04-25T00:30:53.324192+0000","last_active":"2024-04-25T00:30:53.324192+0000","last_peered":"2024-04-25T00:30:53.324192+0000","last_clean":"2024-04-25T00:30:53.324192+0000","last_became_active":"2024-04-25T00:01:04.361708+0000","last_became_peered":"2024-04-25T00:01:04.361708+0000","last_unstale":"2024-04-25T00:30:53.324192+0000","last_undegraded":"2024-04-25T00:30:53.324192+0000","last_fullsized":"2024-04-25T00:30:53.324192+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'54","last_scrub_stamp":"2024-04-25T00:30:53.323735+0000","last_deep_scrub":"74'54","last_deep_scrub_stamp":"2024-04-25T00:30:53.323735+0000","last_clean_scrub_stamp":"2024-04-25T00:30:53.323735+0000","objects_scrubbed":1,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:26:43.034066+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":69,"num_read_kb":69,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"69'3669","reported_seq":3804,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:50.083862+0000","last_change":"2024-04-25T00:30:50.083862+0000","last_active":"2024-04-25T00:30:50.083862+0000","last_peered":"2024-04-25T00:30:50.083862+0000","last_clean":"2024-04-25T00:30:50.083862+0000","last_became_active":"2024-04-25T00:01:05.386853+0000","last_became_peered":"2024-04-25T00:01:05.386853+0000","last_unstale":"2024-04-25T00:30:50.083862+0000","last_undegraded":"2024-04-25T00:30:50.083862+0000","last_fullsized":"2024-04-25T00:30:50.083862+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'3669","last_scrub_stamp":"2024-04-25T00:30:50.083530+0000","last_deep_scrub":"69'3669","last_deep_scrub_stamp":"2024-04-25T00:30:50.083530+0000","last_clean_scrub_stamp":"2024-04-25T00:30:50.083530+0000","objects_scrubbed":0,"log_size":3669,"log_dups_size":0,"ondisk_log_size":3669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:44:49.651163+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19904,"num_write_kb":264274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1b","version":"74'724","reported_seq":880,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:46.511981+0000","last_change":"2024-04-25T00:30:46.511981+0000","last_active":"2024-04-25T00:30:46.511981+0000","last_peered":"2024-04-25T00:30:46.511981+0000","last_clean":"2024-04-25T00:30:46.511981+0000","last_became_active":"2024-04-25T00:01:18.487571+0000","last_became_peered":"2024-04-25T00:01:18.487571+0000","last_unstale":"2024-04-25T00:30:46.511981+0000","last_undegraded":"2024-04-25T00:30:46.511981+0000","last_fullsized":"2024-04-25T00:30:46.511981+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'724","last_scrub_stamp":"2024-04-25T00:30:46.511738+0000","last_deep_scrub":"74'724","last_deep_scrub_stamp":"2024-04-25T00:30:46.511738+0000","last_clean_scrub_stamp":"2024-04-25T00:30:46.511738+0000","objects_scrubbed":1,"log_size":724,"log_dups_size":0,"ondisk_log_size":724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:40:44.393760+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":27,"num_read_kb":82727,"num_write":711,"num_write_kb":4310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1b","version":"74'3586","reported_seq":3714,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:06.873105+0000","last_change":"2024-04-25T00:31:06.873105+0000","last_active":"2024-04-25T00:31:06.873105+0000","last_peered":"2024-04-25T00:31:06.873105+0000","last_clean":"2024-04-25T00:31:06.873105+0000","last_became_active":"2024-04-25T00:01:19.519470+0000","last_became_peered":"2024-04-25T00:01:19.519470+0000","last_unstale":"2024-04-25T00:31:06.873105+0000","last_undegraded":"2024-04-25T00:31:06.873105+0000","last_fullsized":"2024-04-25T00:31:06.873105+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'3586","last_scrub_stamp":"2024-04-25T00:31:06.872963+0000","last_deep_scrub":"74'3586","last_deep_scrub_stamp":"2024-04-25T00:31:06.872963+0000","last_clean_scrub_stamp":"2024-04-25T00:31:06.872963+0000","objects_scrubbed":0,"log_size":3586,"log_dups_size":0,"ondisk_log_size":3586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:53:15.388339+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":5,"num_write":18586,"num_write_kb":254663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1c","version":"74'7","reported_seq":190,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:55.301289+0000","last_change":"2024-04-25T00:30:55.301289+0000","last_active":"2024-04-25T00:30:55.301289+0000","last_peered":"2024-04-25T00:30:55.301289+0000","last_clean":"2024-04-25T00:30:55.301289+0000","last_became_active":"2024-04-25T00:01:04.362059+0000","last_became_peered":"2024-04-25T00:01:04.362059+0000","last_unstale":"2024-04-25T00:30:55.301289+0000","last_undegraded":"2024-04-25T00:30:55.301289+0000","last_fullsized":"2024-04-25T00:30:55.301289+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'7","last_scrub_stamp":"2024-04-25T00:30:55.300906+0000","last_deep_scrub":"74'7","last_deep_scrub_stamp":"2024-04-25T00:30:55.300906+0000","last_clean_scrub_stamp":"2024-04-25T00:30:55.300906+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:18:19.719987+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"70'4973","reported_seq":5108,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:57.204739+0000","last_change":"2024-04-25T00:30:57.204739+0000","last_active":"2024-04-25T00:30:57.204739+0000","last_peered":"2024-04-25T00:30:57.204739+0000","last_clean":"2024-04-25T00:30:57.204739+0000","last_became_active":"2024-04-25T00:01:05.378320+0000","last_became_peered":"2024-04-25T00:01:05.378320+0000","last_unstale":"2024-04-25T00:30:57.204739+0000","last_undegraded":"2024-04-25T00:30:57.204739+0000","last_fullsized":"2024-04-25T00:30:57.204739+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'4973","last_scrub_stamp":"2024-04-25T00:30:57.204327+0000","last_deep_scrub":"70'4973","last_deep_scrub_stamp":"2024-04-25T00:30:57.204327+0000","last_clean_scrub_stamp":"2024-04-25T00:30:57.204327+0000","objects_scrubbed":0,"log_size":4973,"log_dups_size":0,"ondisk_log_size":4973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:23:22.383368+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27225,"num_write_kb":350946,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1a","version":"74'356","reported_seq":487,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:45.489299+0000","last_change":"2024-04-25T00:30:45.489299+0000","last_active":"2024-04-25T00:30:45.489299+0000","last_peered":"2024-04-25T00:30:45.489299+0000","last_clean":"2024-04-25T00:30:45.489299+0000","last_became_active":"2024-04-25T00:01:18.490050+0000","last_became_peered":"2024-04-25T00:01:18.490050+0000","last_unstale":"2024-04-25T00:30:45.489299+0000","last_undegraded":"2024-04-25T00:30:45.489299+0000","last_fullsized":"2024-04-25T00:30:45.489299+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'356","last_scrub_stamp":"2024-04-25T00:30:45.489171+0000","last_deep_scrub":"74'356","last_deep_scrub_stamp":"2024-04-25T00:30:45.489171+0000","last_clean_scrub_stamp":"2024-04-25T00:30:45.489171+0000","objects_scrubbed":1,"log_size":356,"log_dups_size":0,"ondisk_log_size":356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:47:00.287833+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2691973,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":907,"num_write":327,"num_write_kb":2826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"74'4320","reported_seq":4451,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:07.807455+0000","last_change":"2024-04-25T00:31:07.807487+0000","last_active":"2024-04-25T00:31:07.807455+0000","last_peered":"2024-04-25T00:31:07.807455+0000","last_clean":"2024-04-25T00:31:07.807455+0000","last_became_active":"2024-04-25T00:01:19.499244+0000","last_became_peered":"2024-04-25T00:01:19.499244+0000","last_unstale":"2024-04-25T00:31:07.807455+0000","last_undegraded":"2024-04-25T00:31:07.807455+0000","last_fullsized":"2024-04-25T00:31:07.807455+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4320","last_scrub_stamp":"2024-04-25T00:31:07.807329+0000","last_deep_scrub":"74'4320","last_deep_scrub_stamp":"2024-04-25T00:31:07.807329+0000","last_clean_scrub_stamp":"2024-04-25T00:31:07.807329+0000","objects_scrubbed":0,"log_size":4320,"log_dups_size":0,"ondisk_log_size":4320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:34:24.145202+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":15,"num_write":23810,"num_write_kb":308534,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.a","version":"70'4158","reported_seq":4294,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:51.241517+0000","last_change":"2024-04-25T00:30:51.241583+0000","last_active":"2024-04-25T00:30:51.241517+0000","last_peered":"2024-04-25T00:30:51.241517+0000","last_clean":"2024-04-25T00:30:51.241517+0000","last_became_active":"2024-04-25T00:01:05.376259+0000","last_became_peered":"2024-04-25T00:01:05.376259+0000","last_unstale":"2024-04-25T00:30:51.241517+0000","last_undegraded":"2024-04-25T00:30:51.241517+0000","last_fullsized":"2024-04-25T00:30:51.241517+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'4158","last_scrub_stamp":"2024-04-25T00:30:51.241297+0000","last_deep_scrub":"70'4158","last_deep_scrub_stamp":"2024-04-25T00:30:51.241297+0000","last_clean_scrub_stamp":"2024-04-25T00:30:51.241297+0000","objects_scrubbed":0,"log_size":4158,"log_dups_size":0,"ondisk_log_size":4158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:46:42.723857+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23006,"num_write_kb":301346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"74'1528","reported_seq":1721,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:06.185212+0000","last_change":"2024-04-25T00:31:06.185285+0000","last_active":"2024-04-25T00:31:06.185212+0000","last_peered":"2024-04-25T00:31:06.185212+0000","last_clean":"2024-04-25T00:31:06.185212+0000","last_became_active":"2024-04-25T00:01:04.357971+0000","last_became_peered":"2024-04-25T00:01:04.357971+0000","last_unstale":"2024-04-25T00:31:06.185212+0000","last_undegraded":"2024-04-25T00:31:06.185212+0000","last_fullsized":"2024-04-25T00:31:06.185212+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1528","last_scrub_stamp":"2024-04-25T00:31:06.185024+0000","last_deep_scrub":"74'1528","last_deep_scrub_stamp":"2024-04-25T00:31:06.185024+0000","last_clean_scrub_stamp":"2024-04-25T00:31:06.185024+0000","objects_scrubbed":2,"log_size":1528,"log_dups_size":0,"ondisk_log_size":1528,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:20:22.393180+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":78,"num_read_kb":52062,"num_write":1499,"num_write_kb":4525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"74'7","reported_seq":160,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:13.271539+0000","last_change":"2024-04-25T00:31:13.271606+0000","last_active":"2024-04-25T00:31:13.271539+0000","last_peered":"2024-04-25T00:31:13.271539+0000","last_clean":"2024-04-25T00:31:13.271539+0000","last_became_active":"2024-04-25T00:01:18.475149+0000","last_became_peered":"2024-04-25T00:01:18.475149+0000","last_unstale":"2024-04-25T00:31:13.271539+0000","last_undegraded":"2024-04-25T00:31:13.271539+0000","last_fullsized":"2024-04-25T00:31:13.271539+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'7","last_scrub_stamp":"2024-04-25T00:31:13.271338+0000","last_deep_scrub":"74'7","last_deep_scrub_stamp":"2024-04-25T00:31:13.271338+0000","last_clean_scrub_stamp":"2024-04-25T00:31:13.271338+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:14:25.053592+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"74'4556","reported_seq":4687,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:59.782276+0000","last_change":"2024-04-25T00:30:59.782334+0000","last_active":"2024-04-25T00:30:59.782276+0000","last_peered":"2024-04-25T00:30:59.782276+0000","last_clean":"2024-04-25T00:30:59.782276+0000","last_became_active":"2024-04-25T00:01:19.520150+0000","last_became_peered":"2024-04-25T00:01:19.520150+0000","last_unstale":"2024-04-25T00:30:59.782276+0000","last_undegraded":"2024-04-25T00:30:59.782276+0000","last_fullsized":"2024-04-25T00:30:59.782276+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4556","last_scrub_stamp":"2024-04-25T00:30:59.782060+0000","last_deep_scrub":"74'4556","last_deep_scrub_stamp":"2024-04-25T00:30:59.782060+0000","last_clean_scrub_stamp":"2024-04-25T00:30:59.782060+0000","objects_scrubbed":0,"log_size":4556,"log_dups_size":0,"ondisk_log_size":4556,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:52:39.891690+0000","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":19,"num_write":25634,"num_write_kb":315935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"70'3060","reported_seq":3196,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:51.843128+0000","last_change":"2024-04-25T00:30:51.843191+0000","last_active":"2024-04-25T00:30:51.843128+0000","last_peered":"2024-04-25T00:30:51.843128+0000","last_clean":"2024-04-25T00:30:51.843128+0000","last_became_active":"2024-04-25T00:01:05.361817+0000","last_became_peered":"2024-04-25T00:01:05.361817+0000","last_unstale":"2024-04-25T00:30:51.843128+0000","last_undegraded":"2024-04-25T00:30:51.843128+0000","last_fullsized":"2024-04-25T00:30:51.843128+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'3060","last_scrub_stamp":"2024-04-25T00:30:51.842888+0000","last_deep_scrub":"70'3060","last_deep_scrub_stamp":"2024-04-25T00:30:51.842888+0000","last_clean_scrub_stamp":"2024-04-25T00:30:51.842888+0000","objects_scrubbed":0,"log_size":3060,"log_dups_size":0,"ondisk_log_size":3060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:18:39.934309+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16241,"num_write_kb":228970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.a","version":"70'3","reported_seq":141,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:55.515670+0000","last_change":"2024-04-25T00:30:55.515733+0000","last_active":"2024-04-25T00:30:55.515670+0000","last_peered":"2024-04-25T00:30:55.515670+0000","last_clean":"2024-04-25T00:30:55.515670+0000","last_became_active":"2024-04-25T00:01:04.371420+0000","last_became_peered":"2024-04-25T00:01:04.371420+0000","last_unstale":"2024-04-25T00:30:55.515670+0000","last_undegraded":"2024-04-25T00:30:55.515670+0000","last_fullsized":"2024-04-25T00:30:55.515670+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'3","last_scrub_stamp":"2024-04-25T00:30:55.515462+0000","last_deep_scrub":"70'3","last_deep_scrub_stamp":"2024-04-25T00:30:55.515462+0000","last_clean_scrub_stamp":"2024-04-25T00:30:55.515462+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:19:36.914335+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"74'5","reported_seq":135,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:00.319927+0000","last_change":"2024-04-25T00:31:00.319995+0000","last_active":"2024-04-25T00:31:00.319927+0000","last_peered":"2024-04-25T00:31:00.319927+0000","last_clean":"2024-04-25T00:31:00.319927+0000","last_became_active":"2024-04-25T00:01:18.488291+0000","last_became_peered":"2024-04-25T00:01:18.488291+0000","last_unstale":"2024-04-25T00:31:00.319927+0000","last_undegraded":"2024-04-25T00:31:00.319927+0000","last_fullsized":"2024-04-25T00:31:00.319927+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-04-25T00:31:00.319723+0000","last_deep_scrub":"74'5","last_deep_scrub_stamp":"2024-04-25T00:31:00.319723+0000","last_clean_scrub_stamp":"2024-04-25T00:31:00.319723+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:20:08.156713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"74'3591","reported_seq":3720,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:18.457108+0000","last_change":"2024-04-25T00:31:18.457143+0000","last_active":"2024-04-25T00:31:18.457108+0000","last_peered":"2024-04-25T00:31:18.457108+0000","last_clean":"2024-04-25T00:31:18.457108+0000","last_became_active":"2024-04-25T00:01:19.516405+0000","last_became_peered":"2024-04-25T00:01:19.516405+0000","last_unstale":"2024-04-25T00:31:18.457108+0000","last_undegraded":"2024-04-25T00:31:18.457108+0000","last_fullsized":"2024-04-25T00:31:18.457108+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'3591","last_scrub_stamp":"2024-04-25T00:31:18.456966+0000","last_deep_scrub":"74'3591","last_deep_scrub_stamp":"2024-04-25T00:31:18.456966+0000","last_clean_scrub_stamp":"2024-04-25T00:31:18.456966+0000","objects_scrubbed":0,"log_size":3591,"log_dups_size":0,"ondisk_log_size":3591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:17:51.487831+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":19584,"num_write_kb":250628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"69'3777","reported_seq":3914,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:01.901593+0000","last_change":"2024-04-25T00:31:01.901593+0000","last_active":"2024-04-25T00:31:01.901593+0000","last_peered":"2024-04-25T00:31:01.901593+0000","last_clean":"2024-04-25T00:31:01.901593+0000","last_became_active":"2024-04-25T00:01:05.354699+0000","last_became_peered":"2024-04-25T00:01:05.354699+0000","last_unstale":"2024-04-25T00:31:01.901593+0000","last_undegraded":"2024-04-25T00:31:01.901593+0000","last_fullsized":"2024-04-25T00:31:01.901593+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'3777","last_scrub_stamp":"2024-04-25T00:31:01.901333+0000","last_deep_scrub":"69'3777","last_deep_scrub_stamp":"2024-04-25T00:31:01.901333+0000","last_clean_scrub_stamp":"2024-04-25T00:31:01.901333+0000","objects_scrubbed":0,"log_size":3777,"log_dups_size":0,"ondisk_log_size":3777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:36:47.997136+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20993,"num_write_kb":256390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"69'2","reported_seq":141,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:59.839212+0000","last_change":"2024-04-25T00:30:59.839212+0000","last_active":"2024-04-25T00:30:59.839212+0000","last_peered":"2024-04-25T00:30:59.839212+0000","last_clean":"2024-04-25T00:30:59.839212+0000","last_became_active":"2024-04-25T00:01:04.361622+0000","last_became_peered":"2024-04-25T00:01:04.361622+0000","last_unstale":"2024-04-25T00:30:59.839212+0000","last_undegraded":"2024-04-25T00:30:59.839212+0000","last_fullsized":"2024-04-25T00:30:59.839212+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-04-25T00:30:59.838955+0000","last_deep_scrub":"69'2","last_deep_scrub_stamp":"2024-04-25T00:30:59.838955+0000","last_clean_scrub_stamp":"2024-04-25T00:30:59.838955+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:10:07.461621+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"74'5","reported_seq":158,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:54.971187+0000","last_change":"2024-04-25T00:30:54.971259+0000","last_active":"2024-04-25T00:30:54.971187+0000","last_peered":"2024-04-25T00:30:54.971187+0000","last_clean":"2024-04-25T00:30:54.971187+0000","last_became_active":"2024-04-25T00:01:18.493614+0000","last_became_peered":"2024-04-25T00:01:18.493614+0000","last_unstale":"2024-04-25T00:30:54.971187+0000","last_undegraded":"2024-04-25T00:30:54.971187+0000","last_fullsized":"2024-04-25T00:30:54.971187+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-04-25T00:30:54.970960+0000","last_deep_scrub":"74'5","last_deep_scrub_stamp":"2024-04-25T00:30:54.970960+0000","last_clean_scrub_stamp":"2024-04-25T00:30:54.970960+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:54:38.535896+0000","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"74'4895","reported_seq":5023,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:53.104085+0000","last_change":"2024-04-25T00:30:53.104146+0000","last_active":"2024-04-25T00:30:53.104085+0000","last_peered":"2024-04-25T00:30:53.104085+0000","last_clean":"2024-04-25T00:30:53.104085+0000","last_became_active":"2024-04-25T00:01:19.505126+0000","last_became_peered":"2024-04-25T00:01:19.505126+0000","last_unstale":"2024-04-25T00:30:53.104085+0000","last_undegraded":"2024-04-25T00:30:53.104085+0000","last_fullsized":"2024-04-25T00:30:53.104085+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4895","last_scrub_stamp":"2024-04-25T00:30:53.103877+0000","last_deep_scrub":"74'4895","last_deep_scrub_stamp":"2024-04-25T00:30:53.103877+0000","last_clean_scrub_stamp":"2024-04-25T00:30:53.103877+0000","objects_scrubbed":0,"log_size":4895,"log_dups_size":0,"ondisk_log_size":4895,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:34:00.242731+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26799,"num_write_kb":349350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"69'3934","reported_seq":4071,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:02.251665+0000","last_change":"2024-04-25T00:31:02.251665+0000","last_active":"2024-04-25T00:31:02.251665+0000","last_peered":"2024-04-25T00:31:02.251665+0000","last_clean":"2024-04-25T00:31:02.251665+0000","last_became_active":"2024-04-25T00:01:05.354118+0000","last_became_peered":"2024-04-25T00:01:05.354118+0000","last_unstale":"2024-04-25T00:31:02.251665+0000","last_undegraded":"2024-04-25T00:31:02.251665+0000","last_fullsized":"2024-04-25T00:31:02.251665+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'3934","last_scrub_stamp":"2024-04-25T00:31:02.251375+0000","last_deep_scrub":"69'3934","last_deep_scrub_stamp":"2024-04-25T00:31:02.251375+0000","last_clean_scrub_stamp":"2024-04-25T00:31:02.251375+0000","objects_scrubbed":0,"log_size":3934,"log_dups_size":0,"ondisk_log_size":3934,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:32:16.209225+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21059,"num_write_kb":273031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.8","version":"70'6","reported_seq":144,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:56.344716+0000","last_change":"2024-04-25T00:30:56.344776+0000","last_active":"2024-04-25T00:30:56.344716+0000","last_peered":"2024-04-25T00:30:56.344716+0000","last_clean":"2024-04-25T00:30:56.344716+0000","last_became_active":"2024-04-25T00:01:04.364942+0000","last_became_peered":"2024-04-25T00:01:04.364942+0000","last_unstale":"2024-04-25T00:30:56.344716+0000","last_undegraded":"2024-04-25T00:30:56.344716+0000","last_fullsized":"2024-04-25T00:30:56.344716+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'6","last_scrub_stamp":"2024-04-25T00:30:56.344504+0000","last_deep_scrub":"70'6","last_deep_scrub_stamp":"2024-04-25T00:30:56.344504+0000","last_clean_scrub_stamp":"2024-04-25T00:30:56.344504+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:02:38.727658+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"74'8","reported_seq":184,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:54.616129+0000","last_change":"2024-04-25T00:30:54.616191+0000","last_active":"2024-04-25T00:30:54.616129+0000","last_peered":"2024-04-25T00:30:54.616129+0000","last_clean":"2024-04-25T00:30:54.616129+0000","last_became_active":"2024-04-25T00:01:18.475862+0000","last_became_peered":"2024-04-25T00:01:18.475862+0000","last_unstale":"2024-04-25T00:30:54.616129+0000","last_undegraded":"2024-04-25T00:30:54.616129+0000","last_fullsized":"2024-04-25T00:30:54.616129+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-04-25T00:30:54.615941+0000","last_deep_scrub":"74'8","last_deep_scrub_stamp":"2024-04-25T00:30:54.615941+0000","last_clean_scrub_stamp":"2024-04-25T00:30:54.615941+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:28:30.767266+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.0","version":"74'3090","reported_seq":3220,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:10.022710+0000","last_change":"2024-04-25T00:31:10.022710+0000","last_active":"2024-04-25T00:31:10.022710+0000","last_peered":"2024-04-25T00:31:10.022710+0000","last_clean":"2024-04-25T00:31:10.022710+0000","last_became_active":"2024-04-25T00:01:19.528509+0000","last_became_peered":"2024-04-25T00:01:19.528509+0000","last_unstale":"2024-04-25T00:31:10.022710+0000","last_undegraded":"2024-04-25T00:31:10.022710+0000","last_fullsized":"2024-04-25T00:31:10.022710+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'3090","last_scrub_stamp":"2024-04-25T00:31:10.022533+0000","last_deep_scrub":"74'3090","last_deep_scrub_stamp":"2024-04-25T00:31:10.022533+0000","last_clean_scrub_stamp":"2024-04-25T00:31:10.022533+0000","objects_scrubbed":0,"log_size":3090,"log_dups_size":0,"ondisk_log_size":3090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:37:48.745577+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17043,"num_write_kb":228060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"70'4336","reported_seq":4473,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:02.910951+0000","last_change":"2024-04-25T00:31:02.910951+0000","last_active":"2024-04-25T00:31:02.910951+0000","last_peered":"2024-04-25T00:31:02.910951+0000","last_clean":"2024-04-25T00:31:02.910951+0000","last_became_active":"2024-04-25T00:01:05.357135+0000","last_became_peered":"2024-04-25T00:01:05.357135+0000","last_unstale":"2024-04-25T00:31:02.910951+0000","last_undegraded":"2024-04-25T00:31:02.910951+0000","last_fullsized":"2024-04-25T00:31:02.910951+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'4336","last_scrub_stamp":"2024-04-25T00:31:02.910672+0000","last_deep_scrub":"70'4336","last_deep_scrub_stamp":"2024-04-25T00:31:02.910672+0000","last_clean_scrub_stamp":"2024-04-25T00:31:02.910672+0000","objects_scrubbed":0,"log_size":4336,"log_dups_size":0,"ondisk_log_size":4336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:01:28.439618+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23030,"num_write_kb":325887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"74'10","reported_seq":195,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:12.245797+0000","last_change":"2024-04-25T00:31:12.245797+0000","last_active":"2024-04-25T00:31:12.245797+0000","last_peered":"2024-04-25T00:31:12.245797+0000","last_clean":"2024-04-25T00:31:12.245797+0000","last_became_active":"2024-04-25T00:01:04.369309+0000","last_became_peered":"2024-04-25T00:01:04.369309+0000","last_unstale":"2024-04-25T00:31:12.245797+0000","last_undegraded":"2024-04-25T00:31:12.245797+0000","last_fullsized":"2024-04-25T00:31:12.245797+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-04-25T00:31:12.245438+0000","last_deep_scrub":"74'10","last_deep_scrub_stamp":"2024-04-25T00:31:12.245438+0000","last_clean_scrub_stamp":"2024-04-25T00:31:12.245438+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:25:47.381544+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":8,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"74'51","reported_seq":250,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:48.496620+0000","last_change":"2024-04-25T00:30:48.496682+0000","last_active":"2024-04-25T00:30:48.496620+0000","last_peered":"2024-04-25T00:30:48.496620+0000","last_clean":"2024-04-25T00:30:48.496620+0000","last_became_active":"2024-04-25T00:01:18.476909+0000","last_became_peered":"2024-04-25T00:01:18.476909+0000","last_unstale":"2024-04-25T00:30:48.496620+0000","last_undegraded":"2024-04-25T00:30:48.496620+0000","last_fullsized":"2024-04-25T00:30:48.496620+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'51","last_scrub_stamp":"2024-04-25T00:30:48.496424+0000","last_deep_scrub":"74'51","last_deep_scrub_stamp":"2024-04-25T00:30:48.496424+0000","last_clean_scrub_stamp":"2024-04-25T00:30:48.496424+0000","objects_scrubbed":1,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:01:36.221579+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84646,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":68,"num_read_kb":112,"num_write":46,"num_write_kb":112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"74'3821","reported_seq":3951,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:20.490752+0000","last_change":"2024-04-25T00:31:20.490797+0000","last_active":"2024-04-25T00:31:20.490752+0000","last_peered":"2024-04-25T00:31:20.490752+0000","last_clean":"2024-04-25T00:31:20.490752+0000","last_became_active":"2024-04-25T00:01:19.517182+0000","last_became_peered":"2024-04-25T00:01:19.517182+0000","last_unstale":"2024-04-25T00:31:20.490752+0000","last_undegraded":"2024-04-25T00:31:20.490752+0000","last_fullsized":"2024-04-25T00:31:20.490752+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'3821","last_scrub_stamp":"2024-04-25T00:31:20.490616+0000","last_deep_scrub":"74'3821","last_deep_scrub_stamp":"2024-04-25T00:31:20.490616+0000","last_clean_scrub_stamp":"2024-04-25T00:31:20.490616+0000","objects_scrubbed":0,"log_size":3821,"log_dups_size":0,"ondisk_log_size":3821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:59:08.751048+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20929,"num_write_kb":256020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"70'3227","reported_seq":3364,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:03.879774+0000","last_change":"2024-04-25T00:31:03.879774+0000","last_active":"2024-04-25T00:31:03.879774+0000","last_peered":"2024-04-25T00:31:03.879774+0000","last_clean":"2024-04-25T00:31:03.879774+0000","last_became_active":"2024-04-25T00:01:05.364125+0000","last_became_peered":"2024-04-25T00:01:05.364125+0000","last_unstale":"2024-04-25T00:31:03.879774+0000","last_undegraded":"2024-04-25T00:31:03.879774+0000","last_fullsized":"2024-04-25T00:31:03.879774+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'3227","last_scrub_stamp":"2024-04-25T00:31:03.879520+0000","last_deep_scrub":"70'3227","last_deep_scrub_stamp":"2024-04-25T00:31:03.879520+0000","last_clean_scrub_stamp":"2024-04-25T00:31:03.879520+0000","objects_scrubbed":0,"log_size":3227,"log_dups_size":0,"ondisk_log_size":3227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:43:58.941733+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17808,"num_write_kb":231231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"74'459","reported_seq":621,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:58.878367+0000","last_change":"2024-04-25T00:30:58.878367+0000","last_active":"2024-04-25T00:30:58.878367+0000","last_peered":"2024-04-25T00:30:58.878367+0000","last_clean":"2024-04-25T00:30:58.878367+0000","last_became_active":"2024-04-25T00:01:04.361046+0000","last_became_peered":"2024-04-25T00:01:04.361046+0000","last_unstale":"2024-04-25T00:30:58.878367+0000","last_undegraded":"2024-04-25T00:30:58.878367+0000","last_fullsized":"2024-04-25T00:30:58.878367+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'459","last_scrub_stamp":"2024-04-25T00:30:58.877990+0000","last_deep_scrub":"74'459","last_deep_scrub_stamp":"2024-04-25T00:30:58.877990+0000","last_clean_scrub_stamp":"2024-04-25T00:30:58.877990+0000","objects_scrubbed":1,"log_size":459,"log_dups_size":0,"ondisk_log_size":459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:33:10.063484+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":46,"num_read_kb":63,"num_write":1069,"num_write_kb":360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"74'12","reported_seq":144,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:46.735741+0000","last_change":"2024-04-25T00:30:46.735741+0000","last_active":"2024-04-25T00:30:46.735741+0000","last_peered":"2024-04-25T00:30:46.735741+0000","last_clean":"2024-04-25T00:30:46.735741+0000","last_became_active":"2024-04-25T00:01:18.487321+0000","last_became_peered":"2024-04-25T00:01:18.487321+0000","last_unstale":"2024-04-25T00:30:46.735741+0000","last_undegraded":"2024-04-25T00:30:46.735741+0000","last_fullsized":"2024-04-25T00:30:46.735741+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-04-25T00:30:46.735452+0000","last_deep_scrub":"74'12","last_deep_scrub_stamp":"2024-04-25T00:30:46.735452+0000","last_clean_scrub_stamp":"2024-04-25T00:30:46.735452+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:21:22.368298+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2","version":"74'4221","reported_seq":4350,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:06.447236+0000","last_change":"2024-04-25T00:31:06.447264+0000","last_active":"2024-04-25T00:31:06.447236+0000","last_peered":"2024-04-25T00:31:06.447236+0000","last_clean":"2024-04-25T00:31:06.447236+0000","last_became_active":"2024-04-25T00:01:19.519052+0000","last_became_peered":"2024-04-25T00:01:19.519052+0000","last_unstale":"2024-04-25T00:31:06.447236+0000","last_undegraded":"2024-04-25T00:31:06.447236+0000","last_fullsized":"2024-04-25T00:31:06.447236+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4221","last_scrub_stamp":"2024-04-25T00:31:06.447116+0000","last_deep_scrub":"74'4221","last_deep_scrub_stamp":"2024-04-25T00:31:06.447116+0000","last_clean_scrub_stamp":"2024-04-25T00:31:06.447116+0000","objects_scrubbed":0,"log_size":4221,"log_dups_size":0,"ondisk_log_size":4221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:34:22.784989+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22937,"num_write_kb":280510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"70'5174","reported_seq":5311,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:58.516351+0000","last_change":"2024-04-25T00:30:58.516492+0000","last_active":"2024-04-25T00:30:58.516351+0000","last_peered":"2024-04-25T00:30:58.516351+0000","last_clean":"2024-04-25T00:30:58.516351+0000","last_became_active":"2024-04-25T00:01:05.373276+0000","last_became_peered":"2024-04-25T00:01:05.373276+0000","last_unstale":"2024-04-25T00:30:58.516351+0000","last_undegraded":"2024-04-25T00:30:58.516351+0000","last_fullsized":"2024-04-25T00:30:58.516351+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'5174","last_scrub_stamp":"2024-04-25T00:30:58.516111+0000","last_deep_scrub":"70'5174","last_deep_scrub_stamp":"2024-04-25T00:30:58.516111+0000","last_clean_scrub_stamp":"2024-04-25T00:30:58.516111+0000","objects_scrubbed":0,"log_size":5174,"log_dups_size":0,"ondisk_log_size":5174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:41:04.789419+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28022,"num_write_kb":362384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"69'2","reported_seq":141,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:11.203256+0000","last_change":"2024-04-25T00:31:11.203320+0000","last_active":"2024-04-25T00:31:11.203256+0000","last_peered":"2024-04-25T00:31:11.203256+0000","last_clean":"2024-04-25T00:31:11.203256+0000","last_became_active":"2024-04-25T00:01:04.364989+0000","last_became_peered":"2024-04-25T00:01:04.364989+0000","last_unstale":"2024-04-25T00:31:11.203256+0000","last_undegraded":"2024-04-25T00:31:11.203256+0000","last_fullsized":"2024-04-25T00:31:11.203256+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-04-25T00:31:11.203045+0000","last_deep_scrub":"69'2","last_deep_scrub_stamp":"2024-04-25T00:31:11.203045+0000","last_clean_scrub_stamp":"2024-04-25T00:31:11.203045+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:51:56.795716+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"74'3","reported_seq":134,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:48.514879+0000","last_change":"2024-04-25T00:30:48.514944+0000","last_active":"2024-04-25T00:30:48.514879+0000","last_peered":"2024-04-25T00:30:48.514879+0000","last_clean":"2024-04-25T00:30:48.514879+0000","last_became_active":"2024-04-25T00:01:18.490859+0000","last_became_peered":"2024-04-25T00:01:18.490859+0000","last_unstale":"2024-04-25T00:30:48.514879+0000","last_undegraded":"2024-04-25T00:30:48.514879+0000","last_fullsized":"2024-04-25T00:30:48.514879+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'3","last_scrub_stamp":"2024-04-25T00:30:48.514678+0000","last_deep_scrub":"74'3","last_deep_scrub_stamp":"2024-04-25T00:30:48.514678+0000","last_clean_scrub_stamp":"2024-04-25T00:30:48.514678+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:33:55.653532+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3","version":"74'5316","reported_seq":5439,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:13.784839+0000","last_change":"2024-04-25T00:31:13.784962+0000","last_active":"2024-04-25T00:31:13.784839+0000","last_peered":"2024-04-25T00:31:13.784839+0000","last_clean":"2024-04-25T00:31:13.784839+0000","last_became_active":"2024-04-25T00:01:46.667075+0000","last_became_peered":"2024-04-25T00:01:46.667075+0000","last_unstale":"2024-04-25T00:31:13.784839+0000","last_undegraded":"2024-04-25T00:31:13.784839+0000","last_fullsized":"2024-04-25T00:31:13.784839+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5316","last_scrub_stamp":"2024-04-25T00:31:13.784554+0000","last_deep_scrub":"74'5316","last_deep_scrub_stamp":"2024-04-25T00:31:13.784554+0000","last_clean_scrub_stamp":"2024-04-25T00:31:13.784554+0000","objects_scrubbed":0,"log_size":5316,"log_dups_size":0,"ondisk_log_size":5316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:53:57.043867+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29390,"num_write_kb":363802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"70'3961","reported_seq":4098,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:01.721313+0000","last_change":"2024-04-25T00:31:01.721378+0000","last_active":"2024-04-25T00:31:01.721313+0000","last_peered":"2024-04-25T00:31:01.721313+0000","last_clean":"2024-04-25T00:31:01.721313+0000","last_became_active":"2024-04-25T00:01:05.377593+0000","last_became_peered":"2024-04-25T00:01:05.377593+0000","last_unstale":"2024-04-25T00:31:01.721313+0000","last_undegraded":"2024-04-25T00:31:01.721313+0000","last_fullsized":"2024-04-25T00:31:01.721313+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'3961","last_scrub_stamp":"2024-04-25T00:31:01.721093+0000","last_deep_scrub":"70'3961","last_deep_scrub_stamp":"2024-04-25T00:31:01.721093+0000","last_clean_scrub_stamp":"2024-04-25T00:31:01.721093+0000","objects_scrubbed":0,"log_size":3961,"log_dups_size":0,"ondisk_log_size":3961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:15:42.320004+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21237,"num_write_kb":269574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"70'2","reported_seq":141,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:00.728001+0000","last_change":"2024-04-25T00:31:00.728062+0000","last_active":"2024-04-25T00:31:00.728001+0000","last_peered":"2024-04-25T00:31:00.728001+0000","last_clean":"2024-04-25T00:31:00.728001+0000","last_became_active":"2024-04-25T00:01:04.365881+0000","last_became_peered":"2024-04-25T00:01:04.365881+0000","last_unstale":"2024-04-25T00:31:00.728001+0000","last_undegraded":"2024-04-25T00:31:00.728001+0000","last_fullsized":"2024-04-25T00:31:00.728001+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2","last_scrub_stamp":"2024-04-25T00:31:00.727781+0000","last_deep_scrub":"70'2","last_deep_scrub_stamp":"2024-04-25T00:31:00.727781+0000","last_clean_scrub_stamp":"2024-04-25T00:31:00.727781+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:47:19.164558+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'4","reported_seq":135,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:49.524439+0000","last_change":"2024-04-25T00:30:49.524509+0000","last_active":"2024-04-25T00:30:49.524439+0000","last_peered":"2024-04-25T00:30:49.524439+0000","last_clean":"2024-04-25T00:30:49.524439+0000","last_became_active":"2024-04-25T00:01:18.491498+0000","last_became_peered":"2024-04-25T00:01:18.491498+0000","last_unstale":"2024-04-25T00:30:49.524439+0000","last_undegraded":"2024-04-25T00:30:49.524439+0000","last_fullsized":"2024-04-25T00:30:49.524439+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-04-25T00:30:49.524234+0000","last_deep_scrub":"74'4","last_deep_scrub_stamp":"2024-04-25T00:30:49.524234+0000","last_clean_scrub_stamp":"2024-04-25T00:30:49.524234+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:47:04.322897+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.5","version":"74'4156","reported_seq":4292,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:03.709668+0000","last_change":"2024-04-25T00:31:03.709713+0000","last_active":"2024-04-25T00:31:03.709668+0000","last_peered":"2024-04-25T00:31:03.709668+0000","last_clean":"2024-04-25T00:31:03.709668+0000","last_became_active":"2024-04-25T00:01:46.666224+0000","last_became_peered":"2024-04-25T00:01:46.666224+0000","last_unstale":"2024-04-25T00:31:03.709668+0000","last_undegraded":"2024-04-25T00:31:03.709668+0000","last_fullsized":"2024-04-25T00:31:03.709668+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4156","last_scrub_stamp":"2024-04-25T00:31:03.709503+0000","last_deep_scrub":"74'4156","last_deep_scrub_stamp":"2024-04-25T00:31:03.709503+0000","last_clean_scrub_stamp":"2024-04-25T00:31:03.709503+0000","objects_scrubbed":0,"log_size":4156,"log_dups_size":0,"ondisk_log_size":4156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:41:09.982810+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":7,"num_write":23008,"num_write_kb":280758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"69'4491","reported_seq":4628,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:58.442647+0000","last_change":"2024-04-25T00:30:58.442711+0000","last_active":"2024-04-25T00:30:58.442647+0000","last_peered":"2024-04-25T00:30:58.442647+0000","last_clean":"2024-04-25T00:30:58.442647+0000","last_became_active":"2024-04-25T00:01:05.366817+0000","last_became_peered":"2024-04-25T00:01:05.366817+0000","last_unstale":"2024-04-25T00:30:58.442647+0000","last_undegraded":"2024-04-25T00:30:58.442647+0000","last_fullsized":"2024-04-25T00:30:58.442647+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'4491","last_scrub_stamp":"2024-04-25T00:30:58.442421+0000","last_deep_scrub":"69'4491","last_deep_scrub_stamp":"2024-04-25T00:30:58.442421+0000","last_clean_scrub_stamp":"2024-04-25T00:30:58.442421+0000","objects_scrubbed":0,"log_size":4491,"log_dups_size":0,"ondisk_log_size":4491,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:45:13.565273+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24298,"num_write_kb":330898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"69'5","reported_seq":144,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:08.153664+0000","last_change":"2024-04-25T00:31:08.153721+0000","last_active":"2024-04-25T00:31:08.153664+0000","last_peered":"2024-04-25T00:31:08.153664+0000","last_clean":"2024-04-25T00:31:08.153664+0000","last_became_active":"2024-04-25T00:01:04.361572+0000","last_became_peered":"2024-04-25T00:01:04.361572+0000","last_unstale":"2024-04-25T00:31:08.153664+0000","last_undegraded":"2024-04-25T00:31:08.153664+0000","last_fullsized":"2024-04-25T00:31:08.153664+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'5","last_scrub_stamp":"2024-04-25T00:31:08.153471+0000","last_deep_scrub":"69'5","last_deep_scrub_stamp":"2024-04-25T00:31:08.153471+0000","last_clean_scrub_stamp":"2024-04-25T00:31:08.153471+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:10:20.730132+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.4","version":"74'2","reported_seq":133,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:47.486020+0000","last_change":"2024-04-25T00:30:47.486143+0000","last_active":"2024-04-25T00:30:47.486020+0000","last_peered":"2024-04-25T00:30:47.486020+0000","last_clean":"2024-04-25T00:30:47.486020+0000","last_became_active":"2024-04-25T00:01:18.503379+0000","last_became_peered":"2024-04-25T00:01:18.503379+0000","last_unstale":"2024-04-25T00:30:47.486020+0000","last_undegraded":"2024-04-25T00:30:47.486020+0000","last_fullsized":"2024-04-25T00:30:47.486020+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'2","last_scrub_stamp":"2024-04-25T00:30:47.485766+0000","last_deep_scrub":"74'2","last_deep_scrub_stamp":"2024-04-25T00:30:47.485766+0000","last_clean_scrub_stamp":"2024-04-25T00:30:47.485766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:32:03.027257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.6","version":"74'5402","reported_seq":5534,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:00.744521+0000","last_change":"2024-04-25T00:31:00.744586+0000","last_active":"2024-04-25T00:31:00.744521+0000","last_peered":"2024-04-25T00:31:00.744521+0000","last_clean":"2024-04-25T00:31:00.744521+0000","last_became_active":"2024-04-25T00:01:19.531145+0000","last_became_peered":"2024-04-25T00:01:19.531145+0000","last_unstale":"2024-04-25T00:31:00.744521+0000","last_undegraded":"2024-04-25T00:31:00.744521+0000","last_fullsized":"2024-04-25T00:31:00.744521+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5402","last_scrub_stamp":"2024-04-25T00:31:00.744322+0000","last_deep_scrub":"74'5402","last_deep_scrub_stamp":"2024-04-25T00:31:00.744322+0000","last_clean_scrub_stamp":"2024-04-25T00:31:00.744322+0000","objects_scrubbed":0,"log_size":5402,"log_dups_size":0,"ondisk_log_size":5402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:19:42.143195+0000","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":13,"num_write":29532,"num_write_kb":356063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"70'3335","reported_seq":3473,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:02.722871+0000","last_change":"2024-04-25T00:31:02.722871+0000","last_active":"2024-04-25T00:31:02.722871+0000","last_peered":"2024-04-25T00:31:02.722871+0000","last_clean":"2024-04-25T00:31:02.722871+0000","last_became_active":"2024-04-25T00:01:05.353556+0000","last_became_peered":"2024-04-25T00:01:05.353556+0000","last_unstale":"2024-04-25T00:31:02.722871+0000","last_undegraded":"2024-04-25T00:31:02.722871+0000","last_fullsized":"2024-04-25T00:31:02.722871+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'3335","last_scrub_stamp":"2024-04-25T00:31:02.722594+0000","last_deep_scrub":"70'3335","last_deep_scrub_stamp":"2024-04-25T00:31:02.722594+0000","last_clean_scrub_stamp":"2024-04-25T00:31:02.722594+0000","objects_scrubbed":0,"log_size":3335,"log_dups_size":0,"ondisk_log_size":3335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:23:27.901635+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18377,"num_write_kb":241696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"74'53","reported_seq":261,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:50.855125+0000","last_change":"2024-04-25T00:30:50.855191+0000","last_active":"2024-04-25T00:30:50.855125+0000","last_peered":"2024-04-25T00:30:50.855125+0000","last_clean":"2024-04-25T00:30:50.855125+0000","last_became_active":"2024-04-25T00:01:04.372185+0000","last_became_peered":"2024-04-25T00:01:04.372185+0000","last_unstale":"2024-04-25T00:30:50.855125+0000","last_undegraded":"2024-04-25T00:30:50.855125+0000","last_fullsized":"2024-04-25T00:30:50.855125+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'53","last_scrub_stamp":"2024-04-25T00:30:50.854930+0000","last_deep_scrub":"74'53","last_deep_scrub_stamp":"2024-04-25T00:30:50.854930+0000","last_clean_scrub_stamp":"2024-04-25T00:30:50.854930+0000","objects_scrubbed":1,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:43:20.976912+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88686,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":69,"num_read_kb":117,"num_write":46,"num_write_kb":117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"74'10","reported_seq":186,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:51.639688+0000","last_change":"2024-04-25T00:30:51.639756+0000","last_active":"2024-04-25T00:30:51.639688+0000","last_peered":"2024-04-25T00:30:51.639688+0000","last_clean":"2024-04-25T00:30:51.639688+0000","last_became_active":"2024-04-25T00:01:18.501448+0000","last_became_peered":"2024-04-25T00:01:18.501448+0000","last_unstale":"2024-04-25T00:30:51.639688+0000","last_undegraded":"2024-04-25T00:30:51.639688+0000","last_fullsized":"2024-04-25T00:30:51.639688+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-04-25T00:30:51.639483+0000","last_deep_scrub":"74'10","last_deep_scrub_stamp":"2024-04-25T00:30:51.639483+0000","last_clean_scrub_stamp":"2024-04-25T00:30:51.639483+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:41:45.674299+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":8,"num_write_kb":52,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"74'3337","reported_seq":3465,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:19.454800+0000","last_change":"2024-04-25T00:31:19.454838+0000","last_active":"2024-04-25T00:31:19.454800+0000","last_peered":"2024-04-25T00:31:19.454800+0000","last_clean":"2024-04-25T00:31:19.454800+0000","last_became_active":"2024-04-25T00:01:19.517243+0000","last_became_peered":"2024-04-25T00:01:19.517243+0000","last_unstale":"2024-04-25T00:31:19.454800+0000","last_undegraded":"2024-04-25T00:31:19.454800+0000","last_fullsized":"2024-04-25T00:31:19.454800+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'3337","last_scrub_stamp":"2024-04-25T00:31:19.454650+0000","last_deep_scrub":"74'3337","last_deep_scrub_stamp":"2024-04-25T00:31:19.454650+0000","last_clean_scrub_stamp":"2024-04-25T00:31:19.454650+0000","objects_scrubbed":0,"log_size":3337,"log_dups_size":0,"ondisk_log_size":3337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:30:54.078185+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18533,"num_write_kb":242287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"69'3723","reported_seq":3861,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:57.521542+0000","last_change":"2024-04-25T00:30:57.521626+0000","last_active":"2024-04-25T00:30:57.521542+0000","last_peered":"2024-04-25T00:30:57.521542+0000","last_clean":"2024-04-25T00:30:57.521542+0000","last_became_active":"2024-04-25T00:01:05.369987+0000","last_became_peered":"2024-04-25T00:01:05.369987+0000","last_unstale":"2024-04-25T00:30:57.521542+0000","last_undegraded":"2024-04-25T00:30:57.521542+0000","last_fullsized":"2024-04-25T00:30:57.521542+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'3723","last_scrub_stamp":"2024-04-25T00:30:57.521329+0000","last_deep_scrub":"69'3723","last_deep_scrub_stamp":"2024-04-25T00:30:57.521329+0000","last_clean_scrub_stamp":"2024-04-25T00:30:57.521329+0000","objects_scrubbed":0,"log_size":3723,"log_dups_size":0,"ondisk_log_size":3723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:12:05.343805+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20208,"num_write_kb":269376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"73'13","reported_seq":153,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:04.223229+0000","last_change":"2024-04-25T00:31:04.223229+0000","last_active":"2024-04-25T00:31:04.223229+0000","last_peered":"2024-04-25T00:31:04.223229+0000","last_clean":"2024-04-25T00:31:04.223229+0000","last_became_active":"2024-04-25T00:01:04.369904+0000","last_became_peered":"2024-04-25T00:01:04.369904+0000","last_unstale":"2024-04-25T00:31:04.223229+0000","last_undegraded":"2024-04-25T00:31:04.223229+0000","last_fullsized":"2024-04-25T00:31:04.223229+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'13","last_scrub_stamp":"2024-04-25T00:31:04.222946+0000","last_deep_scrub":"73'13","last_deep_scrub_stamp":"2024-04-25T00:31:04.222946+0000","last_clean_scrub_stamp":"2024-04-25T00:31:04.222946+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:59:50.480311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"74'487","reported_seq":640,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:54.579270+0000","last_change":"2024-04-25T00:30:54.579389+0000","last_active":"2024-04-25T00:30:54.579270+0000","last_peered":"2024-04-25T00:30:54.579270+0000","last_clean":"2024-04-25T00:30:54.579270+0000","last_became_active":"2024-04-25T00:01:18.495573+0000","last_became_peered":"2024-04-25T00:01:18.495573+0000","last_unstale":"2024-04-25T00:30:54.579270+0000","last_undegraded":"2024-04-25T00:30:54.579270+0000","last_fullsized":"2024-04-25T00:30:54.579270+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'487","last_scrub_stamp":"2024-04-25T00:30:54.579039+0000","last_deep_scrub":"74'487","last_deep_scrub_stamp":"2024-04-25T00:30:54.579039+0000","last_clean_scrub_stamp":"2024-04-25T00:30:54.579039+0000","objects_scrubbed":1,"log_size":487,"log_dups_size":0,"ondisk_log_size":487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:26:01.969169+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":46,"num_read_kb":69,"num_write":1147,"num_write_kb":382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"74'5079","reported_seq":5209,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:05.416808+0000","last_change":"2024-04-25T00:31:05.416861+0000","last_active":"2024-04-25T00:31:05.416808+0000","last_peered":"2024-04-25T00:31:05.416808+0000","last_clean":"2024-04-25T00:31:05.416808+0000","last_became_active":"2024-04-25T00:01:19.519020+0000","last_became_peered":"2024-04-25T00:01:19.519020+0000","last_unstale":"2024-04-25T00:31:05.416808+0000","last_undegraded":"2024-04-25T00:31:05.416808+0000","last_fullsized":"2024-04-25T00:31:05.416808+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5079","last_scrub_stamp":"2024-04-25T00:31:05.416641+0000","last_deep_scrub":"74'5079","last_deep_scrub_stamp":"2024-04-25T00:31:05.416641+0000","last_clean_scrub_stamp":"2024-04-25T00:31:05.416641+0000","objects_scrubbed":0,"log_size":5079,"log_dups_size":0,"ondisk_log_size":5079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:34:43.223166+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":14,"num_write":27774,"num_write_kb":353080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"70'3676","reported_seq":3813,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:55.887299+0000","last_change":"2024-04-25T00:30:55.887422+0000","last_active":"2024-04-25T00:30:55.887299+0000","last_peered":"2024-04-25T00:30:55.887299+0000","last_clean":"2024-04-25T00:30:55.887299+0000","last_became_active":"2024-04-25T00:01:05.364272+0000","last_became_peered":"2024-04-25T00:01:05.364272+0000","last_unstale":"2024-04-25T00:30:55.887299+0000","last_undegraded":"2024-04-25T00:30:55.887299+0000","last_fullsized":"2024-04-25T00:30:55.887299+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'3676","last_scrub_stamp":"2024-04-25T00:30:55.887065+0000","last_deep_scrub":"70'3676","last_deep_scrub_stamp":"2024-04-25T00:30:55.887065+0000","last_clean_scrub_stamp":"2024-04-25T00:30:55.887065+0000","objects_scrubbed":0,"log_size":3676,"log_dups_size":0,"ondisk_log_size":3676,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:24:50.050373+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19925,"num_write_kb":256146,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"69'3","reported_seq":142,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:05.192707+0000","last_change":"2024-04-25T00:31:05.192765+0000","last_active":"2024-04-25T00:31:05.192707+0000","last_peered":"2024-04-25T00:31:05.192707+0000","last_clean":"2024-04-25T00:31:05.192707+0000","last_became_active":"2024-04-25T00:01:04.356644+0000","last_became_peered":"2024-04-25T00:01:04.356644+0000","last_unstale":"2024-04-25T00:31:05.192707+0000","last_undegraded":"2024-04-25T00:31:05.192707+0000","last_fullsized":"2024-04-25T00:31:05.192707+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'3","last_scrub_stamp":"2024-04-25T00:31:05.192515+0000","last_deep_scrub":"69'3","last_deep_scrub_stamp":"2024-04-25T00:31:05.192515+0000","last_clean_scrub_stamp":"2024-04-25T00:31:05.192515+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:15:42.932459+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"74'119","reported_seq":298,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:45.080295+0000","last_change":"2024-04-25T00:30:45.080295+0000","last_active":"2024-04-25T00:30:45.080295+0000","last_peered":"2024-04-25T00:30:45.080295+0000","last_clean":"2024-04-25T00:30:45.080295+0000","last_became_active":"2024-04-25T00:00:55.083493+0000","last_became_peered":"2024-04-25T00:00:55.083493+0000","last_unstale":"2024-04-25T00:30:45.080295+0000","last_undegraded":"2024-04-25T00:30:45.080295+0000","last_fullsized":"2024-04-25T00:30:45.080295+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'119","last_scrub_stamp":"2024-04-25T00:30:45.079892+0000","last_deep_scrub":"74'119","last_deep_scrub_stamp":"2024-04-25T00:30:45.079892+0000","last_clean_scrub_stamp":"2024-04-25T00:30:45.079892+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:23:08.387129+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"74'2","reported_seq":133,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:48.259823+0000","last_change":"2024-04-25T00:30:48.259891+0000","last_active":"2024-04-25T00:30:48.259823+0000","last_peered":"2024-04-25T00:30:48.259823+0000","last_clean":"2024-04-25T00:30:48.259823+0000","last_became_active":"2024-04-25T00:01:18.475898+0000","last_became_peered":"2024-04-25T00:01:18.475898+0000","last_unstale":"2024-04-25T00:30:48.259823+0000","last_undegraded":"2024-04-25T00:30:48.259823+0000","last_fullsized":"2024-04-25T00:30:48.259823+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'2","last_scrub_stamp":"2024-04-25T00:30:48.259598+0000","last_deep_scrub":"74'2","last_deep_scrub_stamp":"2024-04-25T00:30:48.259598+0000","last_clean_scrub_stamp":"2024-04-25T00:30:48.259598+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:11:56.082075+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"74'3757","reported_seq":3886,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:10.765269+0000","last_change":"2024-04-25T00:31:10.765334+0000","last_active":"2024-04-25T00:31:10.765269+0000","last_peered":"2024-04-25T00:31:10.765269+0000","last_clean":"2024-04-25T00:31:10.765269+0000","last_became_active":"2024-04-25T00:01:19.517376+0000","last_became_peered":"2024-04-25T00:01:19.517376+0000","last_unstale":"2024-04-25T00:31:10.765269+0000","last_undegraded":"2024-04-25T00:31:10.765269+0000","last_fullsized":"2024-04-25T00:31:10.765269+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'3757","last_scrub_stamp":"2024-04-25T00:31:10.765121+0000","last_deep_scrub":"74'3757","last_deep_scrub_stamp":"2024-04-25T00:31:10.765121+0000","last_clean_scrub_stamp":"2024-04-25T00:31:10.765121+0000","objects_scrubbed":0,"log_size":3757,"log_dups_size":0,"ondisk_log_size":3757,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:29:55.415360+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":2,"num_write":20920,"num_write_kb":243770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"70'4367","reported_seq":4504,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:49.280546+0000","last_change":"2024-04-25T00:30:49.280608+0000","last_active":"2024-04-25T00:30:49.280546+0000","last_peered":"2024-04-25T00:30:49.280546+0000","last_clean":"2024-04-25T00:30:49.280546+0000","last_became_active":"2024-04-25T00:01:05.375881+0000","last_became_peered":"2024-04-25T00:01:05.375881+0000","last_unstale":"2024-04-25T00:30:49.280546+0000","last_undegraded":"2024-04-25T00:30:49.280546+0000","last_fullsized":"2024-04-25T00:30:49.280546+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'4367","last_scrub_stamp":"2024-04-25T00:30:49.280321+0000","last_deep_scrub":"70'4367","last_deep_scrub_stamp":"2024-04-25T00:30:49.280321+0000","last_clean_scrub_stamp":"2024-04-25T00:30:49.280321+0000","objects_scrubbed":0,"log_size":4367,"log_dups_size":0,"ondisk_log_size":4367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:40:55.553629+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":2,"num_write":23695,"num_write_kb":308143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"70'6","reported_seq":144,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:53.550425+0000","last_change":"2024-04-25T00:30:53.550490+0000","last_active":"2024-04-25T00:30:53.550425+0000","last_peered":"2024-04-25T00:30:53.550425+0000","last_clean":"2024-04-25T00:30:53.550425+0000","last_became_active":"2024-04-25T00:01:04.367903+0000","last_became_peered":"2024-04-25T00:01:04.367903+0000","last_unstale":"2024-04-25T00:30:53.550425+0000","last_undegraded":"2024-04-25T00:30:53.550425+0000","last_fullsized":"2024-04-25T00:30:53.550425+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'6","last_scrub_stamp":"2024-04-25T00:30:53.550214+0000","last_deep_scrub":"70'6","last_deep_scrub_stamp":"2024-04-25T00:30:53.550214+0000","last_clean_scrub_stamp":"2024-04-25T00:30:53.550214+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:20:11.988688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"74'3","reported_seq":133,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:50.476522+0000","last_change":"2024-04-25T00:30:50.476589+0000","last_active":"2024-04-25T00:30:50.476522+0000","last_peered":"2024-04-25T00:30:50.476522+0000","last_clean":"2024-04-25T00:30:50.476522+0000","last_became_active":"2024-04-25T00:01:18.476247+0000","last_became_peered":"2024-04-25T00:01:18.476247+0000","last_unstale":"2024-04-25T00:30:50.476522+0000","last_undegraded":"2024-04-25T00:30:50.476522+0000","last_fullsized":"2024-04-25T00:30:50.476522+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'3","last_scrub_stamp":"2024-04-25T00:30:50.476304+0000","last_deep_scrub":"74'3","last_deep_scrub_stamp":"2024-04-25T00:30:50.476304+0000","last_clean_scrub_stamp":"2024-04-25T00:30:50.476304+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:21:37.264544+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"74'4936","reported_seq":5065,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:05.684118+0000","last_change":"2024-04-25T00:31:05.684157+0000","last_active":"2024-04-25T00:31:05.684118+0000","last_peered":"2024-04-25T00:31:05.684118+0000","last_clean":"2024-04-25T00:31:05.684118+0000","last_became_active":"2024-04-25T00:01:19.513640+0000","last_became_peered":"2024-04-25T00:01:19.513640+0000","last_unstale":"2024-04-25T00:31:05.684118+0000","last_undegraded":"2024-04-25T00:31:05.684118+0000","last_fullsized":"2024-04-25T00:31:05.684118+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4936","last_scrub_stamp":"2024-04-25T00:31:05.683985+0000","last_deep_scrub":"74'4936","last_deep_scrub_stamp":"2024-04-25T00:31:05.683985+0000","last_clean_scrub_stamp":"2024-04-25T00:31:05.683985+0000","objects_scrubbed":0,"log_size":4936,"log_dups_size":0,"ondisk_log_size":4936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:18:11.845937+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":27403,"num_write_kb":306834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"70'4466","reported_seq":4602,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:47.353563+0000","last_change":"2024-04-25T00:30:47.353608+0000","last_active":"2024-04-25T00:30:47.353563+0000","last_peered":"2024-04-25T00:30:47.353563+0000","last_clean":"2024-04-25T00:30:47.353563+0000","last_became_active":"2024-04-25T00:01:05.384822+0000","last_became_peered":"2024-04-25T00:01:05.384822+0000","last_unstale":"2024-04-25T00:30:47.353563+0000","last_undegraded":"2024-04-25T00:30:47.353563+0000","last_fullsized":"2024-04-25T00:30:47.353563+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'4466","last_scrub_stamp":"2024-04-25T00:30:47.353447+0000","last_deep_scrub":"70'4466","last_deep_scrub_stamp":"2024-04-25T00:30:47.353447+0000","last_clean_scrub_stamp":"2024-04-25T00:30:47.353447+0000","objects_scrubbed":0,"log_size":4466,"log_dups_size":0,"ondisk_log_size":4466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:44:18.999377+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23831,"num_write_kb":316890,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"74'7","reported_seq":168,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:58.656057+0000","last_change":"2024-04-25T00:30:58.656124+0000","last_active":"2024-04-25T00:30:58.656057+0000","last_peered":"2024-04-25T00:30:58.656057+0000","last_clean":"2024-04-25T00:30:58.656057+0000","last_became_active":"2024-04-25T00:01:04.354419+0000","last_became_peered":"2024-04-25T00:01:04.354419+0000","last_unstale":"2024-04-25T00:30:58.656057+0000","last_undegraded":"2024-04-25T00:30:58.656057+0000","last_fullsized":"2024-04-25T00:30:58.656057+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'7","last_scrub_stamp":"2024-04-25T00:30:58.655859+0000","last_deep_scrub":"74'7","last_deep_scrub_stamp":"2024-04-25T00:30:58.655859+0000","last_clean_scrub_stamp":"2024-04-25T00:30:58.655859+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:18:55.837676+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":7,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'1455","reported_seq":1637,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:52.580874+0000","last_change":"2024-04-25T00:30:52.580950+0000","last_active":"2024-04-25T00:30:52.580874+0000","last_peered":"2024-04-25T00:30:52.580874+0000","last_clean":"2024-04-25T00:30:52.580874+0000","last_became_active":"2024-04-25T00:01:18.489629+0000","last_became_peered":"2024-04-25T00:01:18.489629+0000","last_unstale":"2024-04-25T00:30:52.580874+0000","last_undegraded":"2024-04-25T00:30:52.580874+0000","last_fullsized":"2024-04-25T00:30:52.580874+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1455","last_scrub_stamp":"2024-04-25T00:30:52.580648+0000","last_deep_scrub":"74'1455","last_deep_scrub_stamp":"2024-04-25T00:30:52.580648+0000","last_clean_scrub_stamp":"2024-04-25T00:30:52.580648+0000","objects_scrubbed":2,"log_size":1455,"log_dups_size":0,"ondisk_log_size":1455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:06:03.147140+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":75,"num_read_kb":41971,"num_write":1427,"num_write_kb":4503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'3796","reported_seq":3924,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:55.186739+0000","last_change":"2024-04-25T00:30:55.186823+0000","last_active":"2024-04-25T00:30:55.186739+0000","last_peered":"2024-04-25T00:30:55.186739+0000","last_clean":"2024-04-25T00:30:55.186739+0000","last_became_active":"2024-04-25T00:01:19.507867+0000","last_became_peered":"2024-04-25T00:01:19.507867+0000","last_unstale":"2024-04-25T00:30:55.186739+0000","last_undegraded":"2024-04-25T00:30:55.186739+0000","last_fullsized":"2024-04-25T00:30:55.186739+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'3796","last_scrub_stamp":"2024-04-25T00:30:55.186403+0000","last_deep_scrub":"74'3796","last_deep_scrub_stamp":"2024-04-25T00:30:55.186403+0000","last_clean_scrub_stamp":"2024-04-25T00:30:55.186403+0000","objects_scrubbed":0,"log_size":3796,"log_dups_size":0,"ondisk_log_size":3796,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:21:11.806761+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20867,"num_write_kb":276131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"70'5324","reported_seq":5461,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:49.364228+0000","last_change":"2024-04-25T00:30:49.364288+0000","last_active":"2024-04-25T00:30:49.364228+0000","last_peered":"2024-04-25T00:30:49.364228+0000","last_clean":"2024-04-25T00:30:49.364228+0000","last_became_active":"2024-04-25T00:01:05.378112+0000","last_became_peered":"2024-04-25T00:01:05.378112+0000","last_unstale":"2024-04-25T00:30:49.364228+0000","last_undegraded":"2024-04-25T00:30:49.364228+0000","last_fullsized":"2024-04-25T00:30:49.364228+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'5324","last_scrub_stamp":"2024-04-25T00:30:49.364024+0000","last_deep_scrub":"70'5324","last_deep_scrub_stamp":"2024-04-25T00:30:49.364024+0000","last_clean_scrub_stamp":"2024-04-25T00:30:49.364024+0000","objects_scrubbed":0,"log_size":5324,"log_dups_size":0,"ondisk_log_size":5324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:25:16.580358+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":28190,"num_write_kb":387378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.e","version":"74'8","reported_seq":192,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:00.891145+0000","last_change":"2024-04-25T00:31:00.891211+0000","last_active":"2024-04-25T00:31:00.891145+0000","last_peered":"2024-04-25T00:31:00.891145+0000","last_clean":"2024-04-25T00:31:00.891145+0000","last_became_active":"2024-04-25T00:01:04.370099+0000","last_became_peered":"2024-04-25T00:01:04.370099+0000","last_unstale":"2024-04-25T00:31:00.891145+0000","last_undegraded":"2024-04-25T00:31:00.891145+0000","last_fullsized":"2024-04-25T00:31:00.891145+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-04-25T00:31:00.890959+0000","last_deep_scrub":"74'8","last_deep_scrub_stamp":"2024-04-25T00:31:00.890959+0000","last_clean_scrub_stamp":"2024-04-25T00:31:00.890959+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:11:55.122756+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"74'5","reported_seq":135,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:03.235737+0000","last_change":"2024-04-25T00:31:03.235827+0000","last_active":"2024-04-25T00:31:03.235737+0000","last_peered":"2024-04-25T00:31:03.235737+0000","last_clean":"2024-04-25T00:31:03.235737+0000","last_became_active":"2024-04-25T00:01:18.474952+0000","last_became_peered":"2024-04-25T00:01:18.474952+0000","last_unstale":"2024-04-25T00:31:03.235737+0000","last_undegraded":"2024-04-25T00:31:03.235737+0000","last_fullsized":"2024-04-25T00:31:03.235737+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-04-25T00:31:03.235542+0000","last_deep_scrub":"74'5","last_deep_scrub_stamp":"2024-04-25T00:31:03.235542+0000","last_clean_scrub_stamp":"2024-04-25T00:31:03.235542+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:29:26.963526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"74'4048","reported_seq":4176,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:07.621649+0000","last_change":"2024-04-25T00:31:07.621728+0000","last_active":"2024-04-25T00:31:07.621649+0000","last_peered":"2024-04-25T00:31:07.621649+0000","last_clean":"2024-04-25T00:31:07.621649+0000","last_became_active":"2024-04-25T00:01:19.515926+0000","last_became_peered":"2024-04-25T00:01:19.515926+0000","last_unstale":"2024-04-25T00:31:07.621649+0000","last_undegraded":"2024-04-25T00:31:07.621649+0000","last_fullsized":"2024-04-25T00:31:07.621649+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4048","last_scrub_stamp":"2024-04-25T00:31:07.621422+0000","last_deep_scrub":"74'4048","last_deep_scrub_stamp":"2024-04-25T00:31:07.621422+0000","last_clean_scrub_stamp":"2024-04-25T00:31:07.621422+0000","objects_scrubbed":0,"log_size":4048,"log_dups_size":0,"ondisk_log_size":4048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:18:00.016797+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23468,"num_write_kb":270462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.e","version":"70'3273","reported_seq":3410,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:00.689741+0000","last_change":"2024-04-25T00:31:00.689803+0000","last_active":"2024-04-25T00:31:00.689741+0000","last_peered":"2024-04-25T00:31:00.689741+0000","last_clean":"2024-04-25T00:31:00.689741+0000","last_became_active":"2024-04-25T00:01:05.371204+0000","last_became_peered":"2024-04-25T00:01:05.371204+0000","last_unstale":"2024-04-25T00:31:00.689741+0000","last_undegraded":"2024-04-25T00:31:00.689741+0000","last_fullsized":"2024-04-25T00:31:00.689741+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'3273","last_scrub_stamp":"2024-04-25T00:31:00.689495+0000","last_deep_scrub":"70'3273","last_deep_scrub_stamp":"2024-04-25T00:31:00.689495+0000","last_clean_scrub_stamp":"2024-04-25T00:31:00.689495+0000","objects_scrubbed":0,"log_size":3273,"log_dups_size":0,"ondisk_log_size":3273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:09:14.491669+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":17647,"num_write_kb":246988,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"74'5","reported_seq":166,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:57.386592+0000","last_change":"2024-04-25T00:30:57.386653+0000","last_active":"2024-04-25T00:30:57.386592+0000","last_peered":"2024-04-25T00:30:57.386592+0000","last_clean":"2024-04-25T00:30:57.386592+0000","last_became_active":"2024-04-25T00:01:04.372040+0000","last_became_peered":"2024-04-25T00:01:04.372040+0000","last_unstale":"2024-04-25T00:30:57.386592+0000","last_undegraded":"2024-04-25T00:30:57.386592+0000","last_fullsized":"2024-04-25T00:30:57.386592+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-04-25T00:30:57.386400+0000","last_deep_scrub":"74'5","last_deep_scrub_stamp":"2024-04-25T00:30:57.386400+0000","last_clean_scrub_stamp":"2024-04-25T00:30:57.386400+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:45:01.205083+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"74'2","reported_seq":132,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:52.083336+0000","last_change":"2024-04-25T00:30:52.083398+0000","last_active":"2024-04-25T00:30:52.083336+0000","last_peered":"2024-04-25T00:30:52.083336+0000","last_clean":"2024-04-25T00:30:52.083336+0000","last_became_active":"2024-04-25T00:01:18.487227+0000","last_became_peered":"2024-04-25T00:01:18.487227+0000","last_unstale":"2024-04-25T00:30:52.083336+0000","last_undegraded":"2024-04-25T00:30:52.083336+0000","last_fullsized":"2024-04-25T00:30:52.083336+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'2","last_scrub_stamp":"2024-04-25T00:30:52.083145+0000","last_deep_scrub":"74'2","last_deep_scrub_stamp":"2024-04-25T00:30:52.083145+0000","last_clean_scrub_stamp":"2024-04-25T00:30:52.083145+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:32:07.624636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"74'4306","reported_seq":4436,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:07.988880+0000","last_change":"2024-04-25T00:31:07.988948+0000","last_active":"2024-04-25T00:31:07.988880+0000","last_peered":"2024-04-25T00:31:07.988880+0000","last_clean":"2024-04-25T00:31:07.988880+0000","last_became_active":"2024-04-25T00:01:19.525531+0000","last_became_peered":"2024-04-25T00:01:19.525531+0000","last_unstale":"2024-04-25T00:31:07.988880+0000","last_undegraded":"2024-04-25T00:31:07.988880+0000","last_fullsized":"2024-04-25T00:31:07.988880+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4306","last_scrub_stamp":"2024-04-25T00:31:07.988686+0000","last_deep_scrub":"74'4306","last_deep_scrub_stamp":"2024-04-25T00:31:07.988686+0000","last_clean_scrub_stamp":"2024-04-25T00:31:07.988686+0000","objects_scrubbed":0,"log_size":4306,"log_dups_size":0,"ondisk_log_size":4306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:26:19.410395+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":31,"num_write":22571,"num_write_kb":279217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"70'3794","reported_seq":3930,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:58.228007+0000","last_change":"2024-04-25T00:30:58.228007+0000","last_active":"2024-04-25T00:30:58.228007+0000","last_peered":"2024-04-25T00:30:58.228007+0000","last_clean":"2024-04-25T00:30:58.228007+0000","last_became_active":"2024-04-25T00:01:05.363320+0000","last_became_peered":"2024-04-25T00:01:05.363320+0000","last_unstale":"2024-04-25T00:30:58.228007+0000","last_undegraded":"2024-04-25T00:30:58.228007+0000","last_fullsized":"2024-04-25T00:30:58.228007+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'3794","last_scrub_stamp":"2024-04-25T00:30:58.227621+0000","last_deep_scrub":"70'3794","last_deep_scrub_stamp":"2024-04-25T00:30:58.227621+0000","last_clean_scrub_stamp":"2024-04-25T00:30:58.227621+0000","objects_scrubbed":0,"log_size":3794,"log_dups_size":0,"ondisk_log_size":3794,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:19:57.260701+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20831,"num_write_kb":276143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"74'19","reported_seq":271,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:57.927194+0000","last_change":"2024-04-25T00:30:57.927194+0000","last_active":"2024-04-25T00:30:57.927194+0000","last_peered":"2024-04-25T00:30:57.927194+0000","last_clean":"2024-04-25T00:30:57.927194+0000","last_became_active":"2024-04-25T00:01:04.357571+0000","last_became_peered":"2024-04-25T00:01:04.357571+0000","last_unstale":"2024-04-25T00:30:57.927194+0000","last_undegraded":"2024-04-25T00:30:57.927194+0000","last_fullsized":"2024-04-25T00:30:57.927194+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-04-25T00:30:57.926899+0000","last_deep_scrub":"74'19","last_deep_scrub_stamp":"2024-04-25T00:30:57.926899+0000","last_clean_scrub_stamp":"2024-04-25T00:30:57.926899+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:13:19.179171+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":342,"num_read_kb":342,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"74'3","reported_seq":132,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:56.622780+0000","last_change":"2024-04-25T00:30:56.622912+0000","last_active":"2024-04-25T00:30:56.622780+0000","last_peered":"2024-04-25T00:30:56.622780+0000","last_clean":"2024-04-25T00:30:56.622780+0000","last_became_active":"2024-04-25T00:01:18.496489+0000","last_became_peered":"2024-04-25T00:01:18.496489+0000","last_unstale":"2024-04-25T00:30:56.622780+0000","last_undegraded":"2024-04-25T00:30:56.622780+0000","last_fullsized":"2024-04-25T00:30:56.622780+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'3","last_scrub_stamp":"2024-04-25T00:30:56.622511+0000","last_deep_scrub":"74'3","last_deep_scrub_stamp":"2024-04-25T00:30:56.622511+0000","last_clean_scrub_stamp":"2024-04-25T00:30:56.622511+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:18:21.041592+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.16","version":"74'4849","reported_seq":4979,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:46.353474+0000","last_change":"2024-04-25T00:30:46.353525+0000","last_active":"2024-04-25T00:30:46.353474+0000","last_peered":"2024-04-25T00:30:46.353474+0000","last_clean":"2024-04-25T00:30:46.353474+0000","last_became_active":"2024-04-25T00:01:19.496805+0000","last_became_peered":"2024-04-25T00:01:19.496805+0000","last_unstale":"2024-04-25T00:30:46.353474+0000","last_undegraded":"2024-04-25T00:30:46.353474+0000","last_fullsized":"2024-04-25T00:30:46.353474+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4849","last_scrub_stamp":"2024-04-25T00:30:46.353342+0000","last_deep_scrub":"74'4849","last_deep_scrub_stamp":"2024-04-25T00:30:46.353342+0000","last_clean_scrub_stamp":"2024-04-25T00:30:46.353342+0000","objects_scrubbed":0,"log_size":4849,"log_dups_size":0,"ondisk_log_size":4849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:10:14.560315+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":16,"num_write":26442,"num_write_kb":335585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"70'2974","reported_seq":3110,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:48.040334+0000","last_change":"2024-04-25T00:30:48.040334+0000","last_active":"2024-04-25T00:30:48.040334+0000","last_peered":"2024-04-25T00:30:48.040334+0000","last_clean":"2024-04-25T00:30:48.040334+0000","last_became_active":"2024-04-25T00:01:05.373077+0000","last_became_peered":"2024-04-25T00:01:05.373077+0000","last_unstale":"2024-04-25T00:30:48.040334+0000","last_undegraded":"2024-04-25T00:30:48.040334+0000","last_fullsized":"2024-04-25T00:30:48.040334+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2974","last_scrub_stamp":"2024-04-25T00:30:48.040059+0000","last_deep_scrub":"70'2974","last_deep_scrub_stamp":"2024-04-25T00:30:48.040059+0000","last_clean_scrub_stamp":"2024-04-25T00:30:48.040059+0000","objects_scrubbed":0,"log_size":2974,"log_dups_size":0,"ondisk_log_size":2974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:24:53.452964+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15895,"num_write_kb":231645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.11","version":"70'5","reported_seq":142,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:52.357346+0000","last_change":"2024-04-25T00:30:52.357411+0000","last_active":"2024-04-25T00:30:52.357346+0000","last_peered":"2024-04-25T00:30:52.357346+0000","last_clean":"2024-04-25T00:30:52.357346+0000","last_became_active":"2024-04-25T00:01:04.357004+0000","last_became_peered":"2024-04-25T00:01:04.357004+0000","last_unstale":"2024-04-25T00:30:52.357346+0000","last_undegraded":"2024-04-25T00:30:52.357346+0000","last_fullsized":"2024-04-25T00:30:52.357346+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'5","last_scrub_stamp":"2024-04-25T00:30:52.357144+0000","last_deep_scrub":"70'5","last_deep_scrub_stamp":"2024-04-25T00:30:52.357144+0000","last_clean_scrub_stamp":"2024-04-25T00:30:52.357144+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:12:00.179621+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"74'5","reported_seq":134,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:50.571384+0000","last_change":"2024-04-25T00:30:50.571444+0000","last_active":"2024-04-25T00:30:50.571384+0000","last_peered":"2024-04-25T00:30:50.571384+0000","last_clean":"2024-04-25T00:30:50.571384+0000","last_became_active":"2024-04-25T00:01:18.486279+0000","last_became_peered":"2024-04-25T00:01:18.486279+0000","last_unstale":"2024-04-25T00:30:50.571384+0000","last_undegraded":"2024-04-25T00:30:50.571384+0000","last_fullsized":"2024-04-25T00:30:50.571384+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-04-25T00:30:50.571181+0000","last_deep_scrub":"74'5","last_deep_scrub_stamp":"2024-04-25T00:30:50.571181+0000","last_clean_scrub_stamp":"2024-04-25T00:30:50.571181+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:25:17.787515+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"74'4040","reported_seq":4167,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:17.478071+0000","last_change":"2024-04-25T00:31:17.478138+0000","last_active":"2024-04-25T00:31:17.478071+0000","last_peered":"2024-04-25T00:31:17.478071+0000","last_clean":"2024-04-25T00:31:17.478071+0000","last_became_active":"2024-04-25T00:01:19.515765+0000","last_became_peered":"2024-04-25T00:01:19.515765+0000","last_unstale":"2024-04-25T00:31:17.478071+0000","last_undegraded":"2024-04-25T00:31:17.478071+0000","last_fullsized":"2024-04-25T00:31:17.478071+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4040","last_scrub_stamp":"2024-04-25T00:31:17.477875+0000","last_deep_scrub":"74'4040","last_deep_scrub_stamp":"2024-04-25T00:31:17.477875+0000","last_clean_scrub_stamp":"2024-04-25T00:31:17.477875+0000","objects_scrubbed":0,"log_size":4040,"log_dups_size":0,"ondisk_log_size":4040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:10:31.849567+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23288,"num_write_kb":269674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"70'4514","reported_seq":4650,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:53.240122+0000","last_change":"2024-04-25T00:30:53.240122+0000","last_active":"2024-04-25T00:30:53.240122+0000","last_peered":"2024-04-25T00:30:53.240122+0000","last_clean":"2024-04-25T00:30:53.240122+0000","last_became_active":"2024-04-25T00:01:05.378644+0000","last_became_peered":"2024-04-25T00:01:05.378644+0000","last_unstale":"2024-04-25T00:30:53.240122+0000","last_undegraded":"2024-04-25T00:30:53.240122+0000","last_fullsized":"2024-04-25T00:30:53.240122+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'4514","last_scrub_stamp":"2024-04-25T00:30:53.239822+0000","last_deep_scrub":"70'4514","last_deep_scrub_stamp":"2024-04-25T00:30:53.239822+0000","last_clean_scrub_stamp":"2024-04-25T00:30:53.239822+0000","objects_scrubbed":0,"log_size":4514,"log_dups_size":0,"ondisk_log_size":4514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:18:50.421639+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25321,"num_write_kb":326961,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"70'3","reported_seq":141,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:09.193517+0000","last_change":"2024-04-25T00:31:09.193517+0000","last_active":"2024-04-25T00:31:09.193517+0000","last_peered":"2024-04-25T00:31:09.193517+0000","last_clean":"2024-04-25T00:31:09.193517+0000","last_became_active":"2024-04-25T00:01:04.360664+0000","last_became_peered":"2024-04-25T00:01:04.360664+0000","last_unstale":"2024-04-25T00:31:09.193517+0000","last_undegraded":"2024-04-25T00:31:09.193517+0000","last_fullsized":"2024-04-25T00:31:09.193517+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'3","last_scrub_stamp":"2024-04-25T00:31:09.193214+0000","last_deep_scrub":"70'3","last_deep_scrub_stamp":"2024-04-25T00:31:09.193214+0000","last_clean_scrub_stamp":"2024-04-25T00:31:09.193214+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:38:33.051419+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.14","version":"74'338","reported_seq":584,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:01.552063+0000","last_change":"2024-04-25T00:31:01.552189+0000","last_active":"2024-04-25T00:31:01.552063+0000","last_peered":"2024-04-25T00:31:01.552063+0000","last_clean":"2024-04-25T00:31:01.552063+0000","last_became_active":"2024-04-25T00:01:18.504963+0000","last_became_peered":"2024-04-25T00:01:18.504963+0000","last_unstale":"2024-04-25T00:31:01.552063+0000","last_undegraded":"2024-04-25T00:31:01.552063+0000","last_fullsized":"2024-04-25T00:31:01.552063+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'338","last_scrub_stamp":"2024-04-25T00:31:01.551839+0000","last_deep_scrub":"74'338","last_deep_scrub_stamp":"2024-04-25T00:31:01.551839+0000","last_clean_scrub_stamp":"2024-04-25T00:31:01.551839+0000","objects_scrubbed":4,"log_size":338,"log_dups_size":0,"ondisk_log_size":338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:21:33.109775+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":209,"num_read_kb":90302,"num_write":331,"num_write_kb":4275,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.14","version":"74'4674","reported_seq":4798,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:11.774049+0000","last_change":"2024-04-25T00:31:11.774087+0000","last_active":"2024-04-25T00:31:11.774049+0000","last_peered":"2024-04-25T00:31:11.774049+0000","last_clean":"2024-04-25T00:31:11.774049+0000","last_became_active":"2024-04-25T00:01:46.657588+0000","last_became_peered":"2024-04-25T00:01:46.657588+0000","last_unstale":"2024-04-25T00:31:11.774049+0000","last_undegraded":"2024-04-25T00:31:11.774049+0000","last_fullsized":"2024-04-25T00:31:11.774049+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4674","last_scrub_stamp":"2024-04-25T00:31:11.773901+0000","last_deep_scrub":"74'4674","last_deep_scrub_stamp":"2024-04-25T00:31:11.773901+0000","last_clean_scrub_stamp":"2024-04-25T00:31:11.773901+0000","objects_scrubbed":0,"log_size":4674,"log_dups_size":0,"ondisk_log_size":4674,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:44:13.366591+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":16,"num_write":25396,"num_write_kb":335454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"70'3777","reported_seq":3913,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:00.549565+0000","last_change":"2024-04-25T00:31:00.549565+0000","last_active":"2024-04-25T00:31:00.549565+0000","last_peered":"2024-04-25T00:31:00.549565+0000","last_clean":"2024-04-25T00:31:00.549565+0000","last_became_active":"2024-04-25T00:01:05.377625+0000","last_became_peered":"2024-04-25T00:01:05.377625+0000","last_unstale":"2024-04-25T00:31:00.549565+0000","last_undegraded":"2024-04-25T00:31:00.549565+0000","last_fullsized":"2024-04-25T00:31:00.549565+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'3777","last_scrub_stamp":"2024-04-25T00:31:00.549184+0000","last_deep_scrub":"70'3777","last_deep_scrub_stamp":"2024-04-25T00:31:00.549184+0000","last_clean_scrub_stamp":"2024-04-25T00:31:00.549184+0000","objects_scrubbed":0,"log_size":3777,"log_dups_size":0,"ondisk_log_size":3777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:46:52.031744+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20730,"num_write_kb":283926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.13","version":"74'11","reported_seq":195,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:49.875871+0000","last_change":"2024-04-25T00:30:49.875871+0000","last_active":"2024-04-25T00:30:49.875871+0000","last_peered":"2024-04-25T00:30:49.875871+0000","last_clean":"2024-04-25T00:30:49.875871+0000","last_became_active":"2024-04-25T00:01:04.370031+0000","last_became_peered":"2024-04-25T00:01:04.370031+0000","last_unstale":"2024-04-25T00:30:49.875871+0000","last_undegraded":"2024-04-25T00:30:49.875871+0000","last_fullsized":"2024-04-25T00:30:49.875871+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-04-25T00:30:49.875609+0000","last_deep_scrub":"74'11","last_deep_scrub_stamp":"2024-04-25T00:30:49.875609+0000","last_clean_scrub_stamp":"2024-04-25T00:30:49.875609+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:43:52.774202+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":184,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"74'33","reported_seq":231,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:01.203820+0000","last_change":"2024-04-25T00:31:01.203909+0000","last_active":"2024-04-25T00:31:01.203820+0000","last_peered":"2024-04-25T00:31:01.203820+0000","last_clean":"2024-04-25T00:31:01.203820+0000","last_became_active":"2024-04-25T00:01:18.472931+0000","last_became_peered":"2024-04-25T00:01:18.472931+0000","last_unstale":"2024-04-25T00:31:01.203820+0000","last_undegraded":"2024-04-25T00:31:01.203820+0000","last_fullsized":"2024-04-25T00:31:01.203820+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'33","last_scrub_stamp":"2024-04-25T00:31:01.203595+0000","last_deep_scrub":"74'33","last_deep_scrub_stamp":"2024-04-25T00:31:01.203595+0000","last_clean_scrub_stamp":"2024-04-25T00:31:01.203595+0000","objects_scrubbed":2,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:49:21.809560+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":161,"num_read_kb":161,"num_write":32,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"74'5042","reported_seq":5170,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:12.094932+0000","last_change":"2024-04-25T00:31:12.094963+0000","last_active":"2024-04-25T00:31:12.094932+0000","last_peered":"2024-04-25T00:31:12.094932+0000","last_clean":"2024-04-25T00:31:12.094932+0000","last_became_active":"2024-04-25T00:01:19.533048+0000","last_became_peered":"2024-04-25T00:01:19.533048+0000","last_unstale":"2024-04-25T00:31:12.094932+0000","last_undegraded":"2024-04-25T00:31:12.094932+0000","last_fullsized":"2024-04-25T00:31:12.094932+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5042","last_scrub_stamp":"2024-04-25T00:31:12.094811+0000","last_deep_scrub":"74'5042","last_deep_scrub_stamp":"2024-04-25T00:31:12.094811+0000","last_clean_scrub_stamp":"2024-04-25T00:31:12.094811+0000","objects_scrubbed":0,"log_size":5042,"log_dups_size":0,"ondisk_log_size":5042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:44:40.321900+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":28058,"num_write_kb":350341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.15","version":"69'3549","reported_seq":3684,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:45.362286+0000","last_change":"2024-04-25T00:30:45.362327+0000","last_active":"2024-04-25T00:30:45.362286+0000","last_peered":"2024-04-25T00:30:45.362286+0000","last_clean":"2024-04-25T00:30:45.362286+0000","last_became_active":"2024-04-25T00:01:05.385258+0000","last_became_peered":"2024-04-25T00:01:05.385258+0000","last_unstale":"2024-04-25T00:30:45.362286+0000","last_undegraded":"2024-04-25T00:30:45.362286+0000","last_fullsized":"2024-04-25T00:30:45.362286+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'3549","last_scrub_stamp":"2024-04-25T00:30:45.362153+0000","last_deep_scrub":"69'3549","last_deep_scrub_stamp":"2024-04-25T00:30:45.362153+0000","last_clean_scrub_stamp":"2024-04-25T00:30:45.362153+0000","objects_scrubbed":0,"log_size":3549,"log_dups_size":0,"ondisk_log_size":3549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:21:01.982510+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19806,"num_write_kb":251506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"74'345","reported_seq":598,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:55.626243+0000","last_change":"2024-04-25T00:30:55.626314+0000","last_active":"2024-04-25T00:30:55.626243+0000","last_peered":"2024-04-25T00:30:55.626243+0000","last_clean":"2024-04-25T00:30:55.626243+0000","last_became_active":"2024-04-25T00:01:04.364930+0000","last_became_peered":"2024-04-25T00:01:04.364930+0000","last_unstale":"2024-04-25T00:30:55.626243+0000","last_undegraded":"2024-04-25T00:30:55.626243+0000","last_fullsized":"2024-04-25T00:30:55.626243+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'345","last_scrub_stamp":"2024-04-25T00:30:55.626025+0000","last_deep_scrub":"74'345","last_deep_scrub_stamp":"2024-04-25T00:30:55.626025+0000","last_clean_scrub_stamp":"2024-04-25T00:30:55.626025+0000","objects_scrubbed":4,"log_size":345,"log_dups_size":0,"ondisk_log_size":345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:41:01.899333+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":208,"num_read_kb":94392,"num_write":336,"num_write_kb":4266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"74'3","reported_seq":132,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:48.766670+0000","last_change":"2024-04-25T00:30:48.766729+0000","last_active":"2024-04-25T00:30:48.766670+0000","last_peered":"2024-04-25T00:30:48.766670+0000","last_clean":"2024-04-25T00:30:48.766670+0000","last_became_active":"2024-04-25T00:01:18.492850+0000","last_became_peered":"2024-04-25T00:01:18.492850+0000","last_unstale":"2024-04-25T00:30:48.766670+0000","last_undegraded":"2024-04-25T00:30:48.766670+0000","last_fullsized":"2024-04-25T00:30:48.766670+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'3","last_scrub_stamp":"2024-04-25T00:30:48.766478+0000","last_deep_scrub":"74'3","last_deep_scrub_stamp":"2024-04-25T00:30:48.766478+0000","last_clean_scrub_stamp":"2024-04-25T00:30:48.766478+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:34:48.730022+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"74'4133","reported_seq":4261,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:06.647938+0000","last_change":"2024-04-25T00:31:06.648032+0000","last_active":"2024-04-25T00:31:06.647938+0000","last_peered":"2024-04-25T00:31:06.647938+0000","last_clean":"2024-04-25T00:31:06.647938+0000","last_became_active":"2024-04-25T00:01:19.509285+0000","last_became_peered":"2024-04-25T00:01:19.509285+0000","last_unstale":"2024-04-25T00:31:06.647938+0000","last_undegraded":"2024-04-25T00:31:06.647938+0000","last_fullsized":"2024-04-25T00:31:06.647938+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4133","last_scrub_stamp":"2024-04-25T00:31:06.647692+0000","last_deep_scrub":"74'4133","last_deep_scrub_stamp":"2024-04-25T00:31:06.647692+0000","last_clean_scrub_stamp":"2024-04-25T00:31:06.647692+0000","objects_scrubbed":0,"log_size":4133,"log_dups_size":0,"ondisk_log_size":4133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:07:31.248098+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":4,"num_write":22847,"num_write_kb":280058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"70'5270","reported_seq":5405,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:54.260418+0000","last_change":"2024-04-25T00:30:54.260483+0000","last_active":"2024-04-25T00:30:54.260418+0000","last_peered":"2024-04-25T00:30:54.260418+0000","last_clean":"2024-04-25T00:30:54.260418+0000","last_became_active":"2024-04-25T00:01:05.369950+0000","last_became_peered":"2024-04-25T00:01:05.369950+0000","last_unstale":"2024-04-25T00:30:54.260418+0000","last_undegraded":"2024-04-25T00:30:54.260418+0000","last_fullsized":"2024-04-25T00:30:54.260418+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'5270","last_scrub_stamp":"2024-04-25T00:30:54.260111+0000","last_deep_scrub":"70'5270","last_deep_scrub_stamp":"2024-04-25T00:30:54.260111+0000","last_clean_scrub_stamp":"2024-04-25T00:30:54.260111+0000","objects_scrubbed":0,"log_size":5270,"log_dups_size":0,"ondisk_log_size":5270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:54:55.960816+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28224,"num_write_kb":363198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.15","version":"74'32","reported_seq":215,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:48.886523+0000","last_change":"2024-04-25T00:30:48.886594+0000","last_active":"2024-04-25T00:30:48.886523+0000","last_peered":"2024-04-25T00:30:48.886523+0000","last_clean":"2024-04-25T00:30:48.886523+0000","last_became_active":"2024-04-25T00:01:04.365945+0000","last_became_peered":"2024-04-25T00:01:04.365945+0000","last_unstale":"2024-04-25T00:30:48.886523+0000","last_undegraded":"2024-04-25T00:30:48.886523+0000","last_fullsized":"2024-04-25T00:30:48.886523+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'32","last_scrub_stamp":"2024-04-25T00:30:48.886315+0000","last_deep_scrub":"74'32","last_deep_scrub_stamp":"2024-04-25T00:30:48.886315+0000","last_clean_scrub_stamp":"2024-04-25T00:30:48.886315+0000","objects_scrubbed":2,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:24:54.299220+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":92,"num_write":32,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"74'10","reported_seq":185,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:52.611626+0000","last_change":"2024-04-25T00:30:52.611706+0000","last_active":"2024-04-25T00:30:52.611626+0000","last_peered":"2024-04-25T00:30:52.611626+0000","last_clean":"2024-04-25T00:30:52.611626+0000","last_became_active":"2024-04-25T00:01:18.475876+0000","last_became_peered":"2024-04-25T00:01:18.475876+0000","last_unstale":"2024-04-25T00:30:52.611626+0000","last_undegraded":"2024-04-25T00:30:52.611626+0000","last_fullsized":"2024-04-25T00:30:52.611626+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-04-25T00:30:52.611425+0000","last_deep_scrub":"74'10","last_deep_scrub_stamp":"2024-04-25T00:30:52.611425+0000","last_clean_scrub_stamp":"2024-04-25T00:30:52.611425+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:54:56.416784+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":184,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"74'4233","reported_seq":4363,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:15.456671+0000","last_change":"2024-04-25T00:31:15.456729+0000","last_active":"2024-04-25T00:31:15.456671+0000","last_peered":"2024-04-25T00:31:15.456671+0000","last_clean":"2024-04-25T00:31:15.456671+0000","last_became_active":"2024-04-25T00:01:19.512425+0000","last_became_peered":"2024-04-25T00:01:19.512425+0000","last_unstale":"2024-04-25T00:31:15.456671+0000","last_undegraded":"2024-04-25T00:31:15.456671+0000","last_fullsized":"2024-04-25T00:31:15.456671+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4233","last_scrub_stamp":"2024-04-25T00:31:15.456537+0000","last_deep_scrub":"74'4233","last_deep_scrub_stamp":"2024-04-25T00:31:15.456537+0000","last_clean_scrub_stamp":"2024-04-25T00:31:15.456537+0000","objects_scrubbed":0,"log_size":4233,"log_dups_size":0,"ondisk_log_size":4233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:05.549904+0000","scrub_duration":123,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":24291,"num_write_kb":290168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"70'4413","reported_seq":4548,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:04.748878+0000","last_change":"2024-04-25T00:31:04.748941+0000","last_active":"2024-04-25T00:31:04.748878+0000","last_peered":"2024-04-25T00:31:04.748878+0000","last_clean":"2024-04-25T00:31:04.748878+0000","last_became_active":"2024-04-25T00:01:05.360956+0000","last_became_peered":"2024-04-25T00:01:05.360956+0000","last_unstale":"2024-04-25T00:31:04.748878+0000","last_undegraded":"2024-04-25T00:31:04.748878+0000","last_fullsized":"2024-04-25T00:31:04.748878+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'4413","last_scrub_stamp":"2024-04-25T00:31:04.748667+0000","last_deep_scrub":"70'4413","last_deep_scrub_stamp":"2024-04-25T00:31:04.748667+0000","last_clean_scrub_stamp":"2024-04-25T00:31:04.748667+0000","objects_scrubbed":0,"log_size":4413,"log_dups_size":0,"ondisk_log_size":4413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:29:09.458683+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22903,"num_write_kb":321461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"70'3","reported_seq":140,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:54.287654+0000","last_change":"2024-04-25T00:30:54.287799+0000","last_active":"2024-04-25T00:30:54.287654+0000","last_peered":"2024-04-25T00:30:54.287654+0000","last_clean":"2024-04-25T00:30:54.287654+0000","last_became_active":"2024-04-25T00:01:04.361827+0000","last_became_peered":"2024-04-25T00:01:04.361827+0000","last_unstale":"2024-04-25T00:30:54.287654+0000","last_undegraded":"2024-04-25T00:30:54.287654+0000","last_fullsized":"2024-04-25T00:30:54.287654+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'3","last_scrub_stamp":"2024-04-25T00:30:54.287407+0000","last_deep_scrub":"70'3","last_deep_scrub_stamp":"2024-04-25T00:30:54.287407+0000","last_clean_scrub_stamp":"2024-04-25T00:30:54.287407+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:41:00.560714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"74'19","reported_seq":266,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:55.844441+0000","last_change":"2024-04-25T00:30:55.844512+0000","last_active":"2024-04-25T00:30:55.844441+0000","last_peered":"2024-04-25T00:30:55.844441+0000","last_clean":"2024-04-25T00:30:55.844441+0000","last_became_active":"2024-04-25T00:01:18.495415+0000","last_became_peered":"2024-04-25T00:01:18.495415+0000","last_unstale":"2024-04-25T00:30:55.844441+0000","last_undegraded":"2024-04-25T00:30:55.844441+0000","last_fullsized":"2024-04-25T00:30:55.844441+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-04-25T00:30:55.844234+0000","last_deep_scrub":"74'19","last_deep_scrub_stamp":"2024-04-25T00:30:55.844234+0000","last_clean_scrub_stamp":"2024-04-25T00:30:55.844234+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:21:12.464591+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":354,"num_read_kb":354,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"74'3474","reported_seq":3601,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:51.615755+0000","last_change":"2024-04-25T00:30:51.615816+0000","last_active":"2024-04-25T00:30:51.615755+0000","last_peered":"2024-04-25T00:30:51.615755+0000","last_clean":"2024-04-25T00:30:51.615755+0000","last_became_active":"2024-04-25T00:01:19.509015+0000","last_became_peered":"2024-04-25T00:01:19.509015+0000","last_unstale":"2024-04-25T00:30:51.615755+0000","last_undegraded":"2024-04-25T00:30:51.615755+0000","last_fullsized":"2024-04-25T00:30:51.615755+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'3474","last_scrub_stamp":"2024-04-25T00:30:51.615588+0000","last_deep_scrub":"74'3474","last_deep_scrub_stamp":"2024-04-25T00:30:51.615588+0000","last_clean_scrub_stamp":"2024-04-25T00:30:51.615588+0000","objects_scrubbed":0,"log_size":3474,"log_dups_size":0,"ondisk_log_size":3474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:10:19.822560+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19390,"num_write_kb":245789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.16","version":"70'4331","reported_seq":4466,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:58.713028+0000","last_change":"2024-04-25T00:30:58.713166+0000","last_active":"2024-04-25T00:30:58.713028+0000","last_peered":"2024-04-25T00:30:58.713028+0000","last_clean":"2024-04-25T00:30:58.713028+0000","last_became_active":"2024-04-25T00:01:05.361865+0000","last_became_peered":"2024-04-25T00:01:05.361865+0000","last_unstale":"2024-04-25T00:30:58.713028+0000","last_undegraded":"2024-04-25T00:30:58.713028+0000","last_fullsized":"2024-04-25T00:30:58.713028+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'4331","last_scrub_stamp":"2024-04-25T00:30:58.712670+0000","last_deep_scrub":"70'4331","last_deep_scrub_stamp":"2024-04-25T00:30:58.712670+0000","last_clean_scrub_stamp":"2024-04-25T00:30:58.712670+0000","objects_scrubbed":0,"log_size":4331,"log_dups_size":0,"ondisk_log_size":4331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:21:30.270607+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22828,"num_write_kb":312693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.17","version":"74'7","reported_seq":144,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:46.064393+0000","last_change":"2024-04-25T00:30:46.064444+0000","last_active":"2024-04-25T00:30:46.064393+0000","last_peered":"2024-04-25T00:30:46.064393+0000","last_clean":"2024-04-25T00:30:46.064393+0000","last_became_active":"2024-04-25T00:01:04.360374+0000","last_became_peered":"2024-04-25T00:01:04.360374+0000","last_unstale":"2024-04-25T00:30:46.064393+0000","last_undegraded":"2024-04-25T00:30:46.064393+0000","last_fullsized":"2024-04-25T00:30:46.064393+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'7","last_scrub_stamp":"2024-04-25T00:30:46.064254+0000","last_deep_scrub":"74'7","last_deep_scrub_stamp":"2024-04-25T00:30:46.064254+0000","last_clean_scrub_stamp":"2024-04-25T00:30:46.064254+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:08:01.206564+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"74'5","reported_seq":134,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:47.769492+0000","last_change":"2024-04-25T00:30:47.769552+0000","last_active":"2024-04-25T00:30:47.769492+0000","last_peered":"2024-04-25T00:30:47.769492+0000","last_clean":"2024-04-25T00:30:47.769492+0000","last_became_active":"2024-04-25T00:01:18.479270+0000","last_became_peered":"2024-04-25T00:01:18.479270+0000","last_unstale":"2024-04-25T00:30:47.769492+0000","last_undegraded":"2024-04-25T00:30:47.769492+0000","last_fullsized":"2024-04-25T00:30:47.769492+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-04-25T00:30:47.769275+0000","last_deep_scrub":"74'5","last_deep_scrub_stamp":"2024-04-25T00:30:47.769275+0000","last_clean_scrub_stamp":"2024-04-25T00:30:47.769275+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:32:34.378407+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1f","version":"74'5681","reported_seq":5814,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:08.859284+0000","last_change":"2024-04-25T00:31:08.859322+0000","last_active":"2024-04-25T00:31:08.859284+0000","last_peered":"2024-04-25T00:31:08.859284+0000","last_clean":"2024-04-25T00:31:08.859284+0000","last_became_active":"2024-04-25T00:01:19.524166+0000","last_became_peered":"2024-04-25T00:01:19.524166+0000","last_unstale":"2024-04-25T00:31:08.859284+0000","last_undegraded":"2024-04-25T00:31:08.859284+0000","last_fullsized":"2024-04-25T00:31:08.859284+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5681","last_scrub_stamp":"2024-04-25T00:31:08.859147+0000","last_deep_scrub":"74'5681","last_deep_scrub_stamp":"2024-04-25T00:31:08.859147+0000","last_clean_scrub_stamp":"2024-04-25T00:31:08.859147+0000","objects_scrubbed":0,"log_size":5681,"log_dups_size":0,"ondisk_log_size":5681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:32:22.816997+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":40,"num_write":31879,"num_write_kb":410681,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"74'28","reported_seq":160,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:50.813331+0000","last_change":"2024-04-25T00:30:50.813331+0000","last_active":"2024-04-25T00:30:50.813331+0000","last_peered":"2024-04-25T00:30:50.813331+0000","last_clean":"2024-04-25T00:30:50.813331+0000","last_became_active":"2024-04-25T00:01:18.496289+0000","last_became_peered":"2024-04-25T00:01:18.496289+0000","last_unstale":"2024-04-25T00:30:50.813331+0000","last_undegraded":"2024-04-25T00:30:50.813331+0000","last_fullsized":"2024-04-25T00:30:50.813331+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'28","last_scrub_stamp":"2024-04-25T00:30:50.813048+0000","last_deep_scrub":"74'28","last_deep_scrub_stamp":"2024-04-25T00:30:50.813048+0000","last_clean_scrub_stamp":"2024-04-25T00:30:50.813048+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:38:06.142467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"70'4773","reported_seq":4909,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:51.855193+0000","last_change":"2024-04-25T00:30:51.855259+0000","last_active":"2024-04-25T00:30:51.855193+0000","last_peered":"2024-04-25T00:30:51.855193+0000","last_clean":"2024-04-25T00:30:51.855193+0000","last_became_active":"2024-04-25T00:01:05.368080+0000","last_became_peered":"2024-04-25T00:01:05.368080+0000","last_unstale":"2024-04-25T00:30:51.855193+0000","last_undegraded":"2024-04-25T00:30:51.855193+0000","last_fullsized":"2024-04-25T00:30:51.855193+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'4773","last_scrub_stamp":"2024-04-25T00:30:51.854990+0000","last_deep_scrub":"70'4773","last_deep_scrub_stamp":"2024-04-25T00:30:51.854990+0000","last_clean_scrub_stamp":"2024-04-25T00:30:51.854990+0000","objects_scrubbed":0,"log_size":4773,"log_dups_size":0,"ondisk_log_size":4773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:34:24.805496+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":25218,"num_write_kb":330520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.18","version":"69'2","reported_seq":139,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:10.156168+0000","last_change":"2024-04-25T00:31:10.156231+0000","last_active":"2024-04-25T00:31:10.156168+0000","last_peered":"2024-04-25T00:31:10.156168+0000","last_clean":"2024-04-25T00:31:10.156168+0000","last_became_active":"2024-04-25T00:01:04.364050+0000","last_became_peered":"2024-04-25T00:01:04.364050+0000","last_unstale":"2024-04-25T00:31:10.156168+0000","last_undegraded":"2024-04-25T00:31:10.156168+0000","last_fullsized":"2024-04-25T00:31:10.156168+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-04-25T00:31:10.155942+0000","last_deep_scrub":"69'2","last_deep_scrub_stamp":"2024-04-25T00:31:10.155942+0000","last_clean_scrub_stamp":"2024-04-25T00:31:10.155942+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:38:06.085679+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"74'3390","reported_seq":3520,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:02.663989+0000","last_change":"2024-04-25T00:31:02.663989+0000","last_active":"2024-04-25T00:31:02.663989+0000","last_peered":"2024-04-25T00:31:02.663989+0000","last_clean":"2024-04-25T00:31:02.663989+0000","last_became_active":"2024-04-25T00:01:19.490755+0000","last_became_peered":"2024-04-25T00:01:19.490755+0000","last_unstale":"2024-04-25T00:31:02.663989+0000","last_undegraded":"2024-04-25T00:31:02.663989+0000","last_fullsized":"2024-04-25T00:31:02.663989+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'3390","last_scrub_stamp":"2024-04-25T00:31:02.663640+0000","last_deep_scrub":"74'3390","last_deep_scrub_stamp":"2024-04-25T00:31:02.663640+0000","last_clean_scrub_stamp":"2024-04-25T00:31:02.663640+0000","objects_scrubbed":0,"log_size":3390,"log_dups_size":0,"ondisk_log_size":3390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:45:17.786492+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18979,"num_write_kb":248305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"74'1085","reported_seq":1271,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:59.284516+0000","last_change":"2024-04-25T00:30:59.284587+0000","last_active":"2024-04-25T00:30:59.284516+0000","last_peered":"2024-04-25T00:30:59.284516+0000","last_clean":"2024-04-25T00:30:59.284516+0000","last_became_active":"2024-04-25T00:01:18.472534+0000","last_became_peered":"2024-04-25T00:01:18.472534+0000","last_unstale":"2024-04-25T00:30:59.284516+0000","last_undegraded":"2024-04-25T00:30:59.284516+0000","last_fullsized":"2024-04-25T00:30:59.284516+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1085","last_scrub_stamp":"2024-04-25T00:30:59.284321+0000","last_deep_scrub":"74'1085","last_deep_scrub_stamp":"2024-04-25T00:30:59.284321+0000","last_clean_scrub_stamp":"2024-04-25T00:30:59.284321+0000","objects_scrubbed":3,"log_size":1085,"log_dups_size":0,"ondisk_log_size":1085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:29:03.994337+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194886,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":101,"num_read_kb":11812,"num_write":1055,"num_write_kb":4728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"70'4239","reported_seq":4374,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:47.624936+0000","last_change":"2024-04-25T00:30:47.625001+0000","last_active":"2024-04-25T00:30:47.624936+0000","last_peered":"2024-04-25T00:30:47.624936+0000","last_clean":"2024-04-25T00:30:47.624936+0000","last_became_active":"2024-04-25T00:01:05.378056+0000","last_became_peered":"2024-04-25T00:01:05.378056+0000","last_unstale":"2024-04-25T00:30:47.624936+0000","last_undegraded":"2024-04-25T00:30:47.624936+0000","last_fullsized":"2024-04-25T00:30:47.624936+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'4239","last_scrub_stamp":"2024-04-25T00:30:47.624713+0000","last_deep_scrub":"70'4239","last_deep_scrub_stamp":"2024-04-25T00:30:47.624713+0000","last_clean_scrub_stamp":"2024-04-25T00:30:47.624713+0000","objects_scrubbed":0,"log_size":4239,"log_dups_size":0,"ondisk_log_size":4239,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:32:50.938659+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22560,"num_write_kb":311538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"70'3","reported_seq":140,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:54.553929+0000","last_change":"2024-04-25T00:30:54.554053+0000","last_active":"2024-04-25T00:30:54.553929+0000","last_peered":"2024-04-25T00:30:54.553929+0000","last_clean":"2024-04-25T00:30:54.553929+0000","last_became_active":"2024-04-25T00:01:04.370273+0000","last_became_peered":"2024-04-25T00:01:04.370273+0000","last_unstale":"2024-04-25T00:30:54.553929+0000","last_undegraded":"2024-04-25T00:30:54.553929+0000","last_fullsized":"2024-04-25T00:30:54.553929+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'3","last_scrub_stamp":"2024-04-25T00:30:54.553687+0000","last_deep_scrub":"70'3","last_deep_scrub_stamp":"2024-04-25T00:30:54.553687+0000","last_clean_scrub_stamp":"2024-04-25T00:30:54.553687+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:52:34.663317+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"74'5148","reported_seq":5276,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:06.062382+0000","last_change":"2024-04-25T00:31:06.062382+0000","last_active":"2024-04-25T00:31:06.062382+0000","last_peered":"2024-04-25T00:31:06.062382+0000","last_clean":"2024-04-25T00:31:06.062382+0000","last_became_active":"2024-04-25T00:01:19.514664+0000","last_became_peered":"2024-04-25T00:01:19.514664+0000","last_unstale":"2024-04-25T00:31:06.062382+0000","last_undegraded":"2024-04-25T00:31:06.062382+0000","last_fullsized":"2024-04-25T00:31:06.062382+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5148","last_scrub_stamp":"2024-04-25T00:31:06.062224+0000","last_deep_scrub":"74'5148","last_deep_scrub_stamp":"2024-04-25T00:31:06.062224+0000","last_clean_scrub_stamp":"2024-04-25T00:31:06.062224+0000","objects_scrubbed":0,"log_size":5148,"log_dups_size":0,"ondisk_log_size":5148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:42:55.477306+0000","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":29131,"num_write_kb":338102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"74'250","reported_seq":391,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:07.123135+0000","last_change":"2024-04-25T00:31:07.123135+0000","last_active":"2024-04-25T00:31:07.123135+0000","last_peered":"2024-04-25T00:31:07.123135+0000","last_clean":"2024-04-25T00:31:07.123135+0000","last_became_active":"2024-04-25T00:01:04.359586+0000","last_became_peered":"2024-04-25T00:01:04.359586+0000","last_unstale":"2024-04-25T00:31:07.123135+0000","last_undegraded":"2024-04-25T00:31:07.123135+0000","last_fullsized":"2024-04-25T00:31:07.123135+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'250","last_scrub_stamp":"2024-04-25T00:31:07.122875+0000","last_deep_scrub":"74'250","last_deep_scrub_stamp":"2024-04-25T00:31:07.122875+0000","last_clean_scrub_stamp":"2024-04-25T00:31:07.122875+0000","objects_scrubbed":1,"log_size":250,"log_dups_size":0,"ondisk_log_size":250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:32:05.897381+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2250578,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":4308,"num_write":220,"num_write_kb":2323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"69'3397","reported_seq":3532,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:49.790544+0000","last_change":"2024-04-25T00:30:49.790544+0000","last_active":"2024-04-25T00:30:49.790544+0000","last_peered":"2024-04-25T00:30:49.790544+0000","last_clean":"2024-04-25T00:30:49.790544+0000","last_became_active":"2024-04-25T00:01:05.368970+0000","last_became_peered":"2024-04-25T00:01:05.368970+0000","last_unstale":"2024-04-25T00:30:49.790544+0000","last_undegraded":"2024-04-25T00:30:49.790544+0000","last_fullsized":"2024-04-25T00:30:49.790544+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'3397","last_scrub_stamp":"2024-04-25T00:30:49.790273+0000","last_deep_scrub":"69'3397","last_deep_scrub_stamp":"2024-04-25T00:30:49.790273+0000","last_clean_scrub_stamp":"2024-04-25T00:30:49.790273+0000","objects_scrubbed":0,"log_size":3397,"log_dups_size":0,"ondisk_log_size":3397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:48:58.324291+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17978,"num_write_kb":244099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"74'6","reported_seq":181,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:52.817820+0000","last_change":"2024-04-25T00:30:52.817820+0000","last_active":"2024-04-25T00:30:52.817820+0000","last_peered":"2024-04-25T00:30:52.817820+0000","last_clean":"2024-04-25T00:30:52.817820+0000","last_became_active":"2024-04-25T00:01:18.497745+0000","last_became_peered":"2024-04-25T00:01:18.497745+0000","last_unstale":"2024-04-25T00:30:52.817820+0000","last_undegraded":"2024-04-25T00:30:52.817820+0000","last_fullsized":"2024-04-25T00:30:52.817820+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-04-25T00:30:52.817567+0000","last_deep_scrub":"74'6","last_deep_scrub_stamp":"2024-04-25T00:30:52.817567+0000","last_clean_scrub_stamp":"2024-04-25T00:30:52.817567+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:52:59.407401+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"74'3919","reported_seq":4049,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:04.897738+0000","last_change":"2024-04-25T00:31:04.897738+0000","last_active":"2024-04-25T00:31:04.897738+0000","last_peered":"2024-04-25T00:31:04.897738+0000","last_clean":"2024-04-25T00:31:04.897738+0000","last_became_active":"2024-04-25T00:01:19.514280+0000","last_became_peered":"2024-04-25T00:01:19.514280+0000","last_unstale":"2024-04-25T00:31:04.897738+0000","last_undegraded":"2024-04-25T00:31:04.897738+0000","last_fullsized":"2024-04-25T00:31:04.897738+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'3919","last_scrub_stamp":"2024-04-25T00:31:04.897604+0000","last_deep_scrub":"74'3919","last_deep_scrub_stamp":"2024-04-25T00:31:04.897604+0000","last_clean_scrub_stamp":"2024-04-25T00:31:04.897604+0000","objects_scrubbed":0,"log_size":3919,"log_dups_size":0,"ondisk_log_size":3919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:17:57.292980+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":20874,"num_write_kb":280510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"74'753","reported_seq":918,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:01.765222+0000","last_change":"2024-04-25T00:31:01.765222+0000","last_active":"2024-04-25T00:31:01.765222+0000","last_peered":"2024-04-25T00:31:01.765222+0000","last_clean":"2024-04-25T00:31:01.765222+0000","last_became_active":"2024-04-25T00:01:04.364788+0000","last_became_peered":"2024-04-25T00:01:04.364788+0000","last_unstale":"2024-04-25T00:31:01.765222+0000","last_undegraded":"2024-04-25T00:31:01.765222+0000","last_fullsized":"2024-04-25T00:31:01.765222+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'753","last_scrub_stamp":"2024-04-25T00:31:01.764916+0000","last_deep_scrub":"74'753","last_deep_scrub_stamp":"2024-04-25T00:31:01.764916+0000","last_clean_scrub_stamp":"2024-04-25T00:31:01.764916+0000","objects_scrubbed":1,"log_size":753,"log_dups_size":0,"ondisk_log_size":753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:20:09.601906+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":28,"num_read_kb":89132,"num_write":742,"num_write_kb":4321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"69'3790","reported_seq":3925,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:49.069275+0000","last_change":"2024-04-25T00:30:49.069275+0000","last_active":"2024-04-25T00:30:49.069275+0000","last_peered":"2024-04-25T00:30:49.069275+0000","last_clean":"2024-04-25T00:30:49.069275+0000","last_became_active":"2024-04-25T00:01:05.373857+0000","last_became_peered":"2024-04-25T00:01:05.373857+0000","last_unstale":"2024-04-25T00:30:49.069275+0000","last_undegraded":"2024-04-25T00:30:49.069275+0000","last_fullsized":"2024-04-25T00:30:49.069275+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'3790","last_scrub_stamp":"2024-04-25T00:30:49.068995+0000","last_deep_scrub":"69'3790","last_deep_scrub_stamp":"2024-04-25T00:30:49.068995+0000","last_clean_scrub_stamp":"2024-04-25T00:30:49.068995+0000","objects_scrubbed":0,"log_size":3790,"log_dups_size":0,"ondisk_log_size":3790,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:53:14.653828+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19857,"num_write_kb":263976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"74'49","reported_seq":224,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:56.807741+0000","last_change":"2024-04-25T00:30:56.807741+0000","last_active":"2024-04-25T00:30:56.807741+0000","last_peered":"2024-04-25T00:30:56.807741+0000","last_clean":"2024-04-25T00:30:56.807741+0000","last_became_active":"2024-04-25T00:01:18.501785+0000","last_became_peered":"2024-04-25T00:01:18.501785+0000","last_unstale":"2024-04-25T00:30:56.807741+0000","last_undegraded":"2024-04-25T00:30:56.807741+0000","last_fullsized":"2024-04-25T00:30:56.807741+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'49","last_scrub_stamp":"2024-04-25T00:30:56.807395+0000","last_deep_scrub":"74'49","last_deep_scrub_stamp":"2024-04-25T00:30:56.807395+0000","last_clean_scrub_stamp":"2024-04-25T00:30:56.807395+0000","objects_scrubbed":1,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:41:50.842211+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":138,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.26","version":"74'3694","reported_seq":3822,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:05.017336+0000","last_change":"2024-04-25T00:31:05.017367+0000","last_active":"2024-04-25T00:31:05.017336+0000","last_peered":"2024-04-25T00:31:05.017336+0000","last_clean":"2024-04-25T00:31:05.017336+0000","last_became_active":"2024-04-25T00:01:19.510155+0000","last_became_peered":"2024-04-25T00:01:19.510155+0000","last_unstale":"2024-04-25T00:31:05.017336+0000","last_undegraded":"2024-04-25T00:31:05.017336+0000","last_fullsized":"2024-04-25T00:31:05.017336+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'3694","last_scrub_stamp":"2024-04-25T00:31:05.017219+0000","last_deep_scrub":"74'3694","last_deep_scrub_stamp":"2024-04-25T00:31:05.017219+0000","last_clean_scrub_stamp":"2024-04-25T00:31:05.017219+0000","objects_scrubbed":0,"log_size":3694,"log_dups_size":0,"ondisk_log_size":3694,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:57:02.885084+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20834,"num_write_kb":259761,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.20","version":"70'4043","reported_seq":4180,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:02.742699+0000","last_change":"2024-04-25T00:31:02.742765+0000","last_active":"2024-04-25T00:31:02.742699+0000","last_peered":"2024-04-25T00:31:02.742699+0000","last_clean":"2024-04-25T00:31:02.742699+0000","last_became_active":"2024-04-25T00:01:05.377742+0000","last_became_peered":"2024-04-25T00:01:05.377742+0000","last_unstale":"2024-04-25T00:31:02.742699+0000","last_undegraded":"2024-04-25T00:31:02.742699+0000","last_fullsized":"2024-04-25T00:31:02.742699+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'4043","last_scrub_stamp":"2024-04-25T00:31:02.742482+0000","last_deep_scrub":"70'4043","last_deep_scrub_stamp":"2024-04-25T00:31:02.742482+0000","last_clean_scrub_stamp":"2024-04-25T00:31:02.742482+0000","objects_scrubbed":0,"log_size":4043,"log_dups_size":0,"ondisk_log_size":4043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:18:08.904434+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21543,"num_write_kb":291017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.27","version":"74'5759","reported_seq":5888,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:54.978481+0000","last_change":"2024-04-25T00:30:54.978525+0000","last_active":"2024-04-25T00:30:54.978481+0000","last_peered":"2024-04-25T00:30:54.978481+0000","last_clean":"2024-04-25T00:30:54.978481+0000","last_became_active":"2024-04-25T00:01:19.507960+0000","last_became_peered":"2024-04-25T00:01:19.507960+0000","last_unstale":"2024-04-25T00:30:54.978481+0000","last_undegraded":"2024-04-25T00:30:54.978481+0000","last_fullsized":"2024-04-25T00:30:54.978481+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5759","last_scrub_stamp":"2024-04-25T00:30:54.978351+0000","last_deep_scrub":"74'5759","last_deep_scrub_stamp":"2024-04-25T00:30:54.978351+0000","last_clean_scrub_stamp":"2024-04-25T00:30:54.978351+0000","objects_scrubbed":0,"log_size":5759,"log_dups_size":0,"ondisk_log_size":5759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:46:55.394949+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":31168,"num_write_kb":366885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.21","version":"70'4068","reported_seq":4206,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:53.848253+0000","last_change":"2024-04-25T00:30:53.848317+0000","last_active":"2024-04-25T00:30:53.848253+0000","last_peered":"2024-04-25T00:30:53.848253+0000","last_clean":"2024-04-25T00:30:53.848253+0000","last_became_active":"2024-04-25T00:01:05.372050+0000","last_became_peered":"2024-04-25T00:01:05.372050+0000","last_unstale":"2024-04-25T00:30:53.848253+0000","last_undegraded":"2024-04-25T00:30:53.848253+0000","last_fullsized":"2024-04-25T00:30:53.848253+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'4068","last_scrub_stamp":"2024-04-25T00:30:53.848045+0000","last_deep_scrub":"70'4068","last_deep_scrub_stamp":"2024-04-25T00:30:53.848045+0000","last_clean_scrub_stamp":"2024-04-25T00:30:53.848045+0000","objects_scrubbed":0,"log_size":4068,"log_dups_size":0,"ondisk_log_size":4068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:34:00.986900+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21791,"num_write_kb":292243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.24","version":"74'4315","reported_seq":4445,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:09.912147+0000","last_change":"2024-04-25T00:31:09.912209+0000","last_active":"2024-04-25T00:31:09.912147+0000","last_peered":"2024-04-25T00:31:09.912147+0000","last_clean":"2024-04-25T00:31:09.912147+0000","last_became_active":"2024-04-25T00:01:19.530836+0000","last_became_peered":"2024-04-25T00:01:19.530836+0000","last_unstale":"2024-04-25T00:31:09.912147+0000","last_undegraded":"2024-04-25T00:31:09.912147+0000","last_fullsized":"2024-04-25T00:31:09.912147+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4315","last_scrub_stamp":"2024-04-25T00:31:09.911948+0000","last_deep_scrub":"74'4315","last_deep_scrub_stamp":"2024-04-25T00:31:09.911948+0000","last_clean_scrub_stamp":"2024-04-25T00:31:09.911948+0000","objects_scrubbed":0,"log_size":4315,"log_dups_size":0,"ondisk_log_size":4315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:42:30.143085+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":23956,"num_write_kb":309408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"70'4267","reported_seq":4404,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:00.236552+0000","last_change":"2024-04-25T00:31:00.236697+0000","last_active":"2024-04-25T00:31:00.236552+0000","last_peered":"2024-04-25T00:31:00.236552+0000","last_clean":"2024-04-25T00:31:00.236552+0000","last_became_active":"2024-04-25T00:01:05.361067+0000","last_became_peered":"2024-04-25T00:01:05.361067+0000","last_unstale":"2024-04-25T00:31:00.236552+0000","last_undegraded":"2024-04-25T00:31:00.236552+0000","last_fullsized":"2024-04-25T00:31:00.236552+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'4267","last_scrub_stamp":"2024-04-25T00:31:00.236260+0000","last_deep_scrub":"70'4267","last_deep_scrub_stamp":"2024-04-25T00:31:00.236260+0000","last_clean_scrub_stamp":"2024-04-25T00:31:00.236260+0000","objects_scrubbed":0,"log_size":4267,"log_dups_size":0,"ondisk_log_size":4267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:53:08.751636+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23473,"num_write_kb":311317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"74'4634","reported_seq":4763,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:08.859787+0000","last_change":"2024-04-25T00:31:08.859854+0000","last_active":"2024-04-25T00:31:08.859787+0000","last_peered":"2024-04-25T00:31:08.859787+0000","last_clean":"2024-04-25T00:31:08.859787+0000","last_became_active":"2024-04-25T00:01:19.505184+0000","last_became_peered":"2024-04-25T00:01:19.505184+0000","last_unstale":"2024-04-25T00:31:08.859787+0000","last_undegraded":"2024-04-25T00:31:08.859787+0000","last_fullsized":"2024-04-25T00:31:08.859787+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4634","last_scrub_stamp":"2024-04-25T00:31:08.859537+0000","last_deep_scrub":"74'4634","last_deep_scrub_stamp":"2024-04-25T00:31:08.859537+0000","last_clean_scrub_stamp":"2024-04-25T00:31:08.859537+0000","objects_scrubbed":0,"log_size":4634,"log_dups_size":0,"ondisk_log_size":4634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:29:32.587520+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":6,"num_write":26896,"num_write_kb":316898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"70'4083","reported_seq":4220,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:03.701517+0000","last_change":"2024-04-25T00:31:03.701567+0000","last_active":"2024-04-25T00:31:03.701517+0000","last_peered":"2024-04-25T00:31:03.701517+0000","last_clean":"2024-04-25T00:31:03.701517+0000","last_became_active":"2024-04-25T00:01:05.357340+0000","last_became_peered":"2024-04-25T00:01:05.357340+0000","last_unstale":"2024-04-25T00:31:03.701517+0000","last_undegraded":"2024-04-25T00:31:03.701517+0000","last_fullsized":"2024-04-25T00:31:03.701517+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'4083","last_scrub_stamp":"2024-04-25T00:31:03.701336+0000","last_deep_scrub":"70'4083","last_deep_scrub_stamp":"2024-04-25T00:31:03.701336+0000","last_clean_scrub_stamp":"2024-04-25T00:31:03.701336+0000","objects_scrubbed":0,"log_size":4083,"log_dups_size":0,"ondisk_log_size":4083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:20:02.734417+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21935,"num_write_kb":296816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.22","version":"74'5147","reported_seq":5281,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:02.460999+0000","last_change":"2024-04-25T00:31:02.461078+0000","last_active":"2024-04-25T00:31:02.460999+0000","last_peered":"2024-04-25T00:31:02.460999+0000","last_clean":"2024-04-25T00:31:02.460999+0000","last_became_active":"2024-04-25T00:01:19.508123+0000","last_became_peered":"2024-04-25T00:01:19.508123+0000","last_unstale":"2024-04-25T00:31:02.460999+0000","last_undegraded":"2024-04-25T00:31:02.460999+0000","last_fullsized":"2024-04-25T00:31:02.460999+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5147","last_scrub_stamp":"2024-04-25T00:31:02.460752+0000","last_deep_scrub":"74'5147","last_deep_scrub_stamp":"2024-04-25T00:31:02.460752+0000","last_clean_scrub_stamp":"2024-04-25T00:31:02.460752+0000","objects_scrubbed":0,"log_size":5147,"log_dups_size":0,"ondisk_log_size":5147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:07:27.061158+0000","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":43,"num_write":27901,"num_write_kb":341497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"70'3848","reported_seq":3986,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:50.583713+0000","last_change":"2024-04-25T00:30:50.583713+0000","last_active":"2024-04-25T00:30:50.583713+0000","last_peered":"2024-04-25T00:30:50.583713+0000","last_clean":"2024-04-25T00:30:50.583713+0000","last_became_active":"2024-04-25T00:01:05.386724+0000","last_became_peered":"2024-04-25T00:01:05.386724+0000","last_unstale":"2024-04-25T00:30:50.583713+0000","last_undegraded":"2024-04-25T00:30:50.583713+0000","last_fullsized":"2024-04-25T00:30:50.583713+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'3848","last_scrub_stamp":"2024-04-25T00:30:50.583438+0000","last_deep_scrub":"70'3848","last_deep_scrub_stamp":"2024-04-25T00:30:50.583438+0000","last_clean_scrub_stamp":"2024-04-25T00:30:50.583438+0000","objects_scrubbed":0,"log_size":3848,"log_dups_size":0,"ondisk_log_size":3848,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:21:07.203795+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20848,"num_write_kb":276248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.23","version":"74'4386","reported_seq":4517,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:07.486228+0000","last_change":"2024-04-25T00:31:07.486272+0000","last_active":"2024-04-25T00:31:07.486228+0000","last_peered":"2024-04-25T00:31:07.486228+0000","last_clean":"2024-04-25T00:31:07.486228+0000","last_became_active":"2024-04-25T00:01:19.522380+0000","last_became_peered":"2024-04-25T00:01:19.522380+0000","last_unstale":"2024-04-25T00:31:07.486228+0000","last_undegraded":"2024-04-25T00:31:07.486228+0000","last_fullsized":"2024-04-25T00:31:07.486228+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4386","last_scrub_stamp":"2024-04-25T00:31:07.486105+0000","last_deep_scrub":"74'4386","last_deep_scrub_stamp":"2024-04-25T00:31:07.486105+0000","last_clean_scrub_stamp":"2024-04-25T00:31:07.486105+0000","objects_scrubbed":0,"log_size":4386,"log_dups_size":0,"ondisk_log_size":4386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:45:17.802422+0000","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":23395,"num_write_kb":294532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.25","version":"70'4349","reported_seq":4486,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:56.535583+0000","last_change":"2024-04-25T00:30:56.535583+0000","last_active":"2024-04-25T00:30:56.535583+0000","last_peered":"2024-04-25T00:30:56.535583+0000","last_clean":"2024-04-25T00:30:56.535583+0000","last_became_active":"2024-04-25T00:01:05.363838+0000","last_became_peered":"2024-04-25T00:01:05.363838+0000","last_unstale":"2024-04-25T00:30:56.535583+0000","last_undegraded":"2024-04-25T00:30:56.535583+0000","last_fullsized":"2024-04-25T00:30:56.535583+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'4349","last_scrub_stamp":"2024-04-25T00:30:56.535318+0000","last_deep_scrub":"70'4349","last_deep_scrub_stamp":"2024-04-25T00:30:56.535318+0000","last_clean_scrub_stamp":"2024-04-25T00:30:56.535318+0000","objects_scrubbed":0,"log_size":4349,"log_dups_size":0,"ondisk_log_size":4349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:06:07.101810+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24353,"num_write_kb":306782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"74'4505","reported_seq":4634,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:14.400030+0000","last_change":"2024-04-25T00:31:14.400067+0000","last_active":"2024-04-25T00:31:14.400030+0000","last_peered":"2024-04-25T00:31:14.400030+0000","last_clean":"2024-04-25T00:31:14.400030+0000","last_became_active":"2024-04-25T00:01:19.510008+0000","last_became_peered":"2024-04-25T00:01:19.510008+0000","last_unstale":"2024-04-25T00:31:14.400030+0000","last_undegraded":"2024-04-25T00:31:14.400030+0000","last_fullsized":"2024-04-25T00:31:14.400030+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4505","last_scrub_stamp":"2024-04-25T00:31:14.399887+0000","last_deep_scrub":"74'4505","last_deep_scrub_stamp":"2024-04-25T00:31:14.399887+0000","last_clean_scrub_stamp":"2024-04-25T00:31:14.399887+0000","objects_scrubbed":0,"log_size":4505,"log_dups_size":0,"ondisk_log_size":4505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:57:54.582572+0000","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24621,"num_write_kb":315853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"70'3623","reported_seq":3759,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:59.737559+0000","last_change":"2024-04-25T00:30:59.737624+0000","last_active":"2024-04-25T00:30:59.737559+0000","last_peered":"2024-04-25T00:30:59.737559+0000","last_clean":"2024-04-25T00:30:59.737559+0000","last_became_active":"2024-04-25T00:01:05.361119+0000","last_became_peered":"2024-04-25T00:01:05.361119+0000","last_unstale":"2024-04-25T00:30:59.737559+0000","last_undegraded":"2024-04-25T00:30:59.737559+0000","last_fullsized":"2024-04-25T00:30:59.737559+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'3623","last_scrub_stamp":"2024-04-25T00:30:59.737329+0000","last_deep_scrub":"70'3623","last_deep_scrub_stamp":"2024-04-25T00:30:59.737329+0000","last_clean_scrub_stamp":"2024-04-25T00:30:59.737329+0000","objects_scrubbed":0,"log_size":3623,"log_dups_size":0,"ondisk_log_size":3623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:45:14.860181+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20289,"num_write_kb":274005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.21","version":"74'4291","reported_seq":4423,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:05.829564+0000","last_change":"2024-04-25T00:31:05.829596+0000","last_active":"2024-04-25T00:31:05.829564+0000","last_peered":"2024-04-25T00:31:05.829564+0000","last_clean":"2024-04-25T00:31:05.829564+0000","last_became_active":"2024-04-25T00:01:19.518110+0000","last_became_peered":"2024-04-25T00:01:19.518110+0000","last_unstale":"2024-04-25T00:31:05.829564+0000","last_undegraded":"2024-04-25T00:31:05.829564+0000","last_fullsized":"2024-04-25T00:31:05.829564+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4291","last_scrub_stamp":"2024-04-25T00:31:05.829446+0000","last_deep_scrub":"74'4291","last_deep_scrub_stamp":"2024-04-25T00:31:05.829446+0000","last_clean_scrub_stamp":"2024-04-25T00:31:05.829446+0000","objects_scrubbed":0,"log_size":4291,"log_dups_size":0,"ondisk_log_size":4291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:59:33.069499+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":23037,"num_write_kb":301131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"69'4983","reported_seq":5120,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:49.613603+0000","last_change":"2024-04-25T00:30:49.613603+0000","last_active":"2024-04-25T00:30:49.613603+0000","last_peered":"2024-04-25T00:30:49.613603+0000","last_clean":"2024-04-25T00:30:49.613603+0000","last_became_active":"2024-04-25T00:01:05.372611+0000","last_became_peered":"2024-04-25T00:01:05.372611+0000","last_unstale":"2024-04-25T00:30:49.613603+0000","last_undegraded":"2024-04-25T00:30:49.613603+0000","last_fullsized":"2024-04-25T00:30:49.613603+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'4983","last_scrub_stamp":"2024-04-25T00:30:49.613328+0000","last_deep_scrub":"69'4983","last_deep_scrub_stamp":"2024-04-25T00:30:49.613328+0000","last_clean_scrub_stamp":"2024-04-25T00:30:49.613328+0000","objects_scrubbed":0,"log_size":4983,"log_dups_size":0,"ondisk_log_size":4983,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:34:06.955379+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26776,"num_write_kb":349057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2e","version":"74'4567","reported_seq":4699,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:01.752451+0000","last_change":"2024-04-25T00:31:01.752514+0000","last_active":"2024-04-25T00:31:01.752451+0000","last_peered":"2024-04-25T00:31:01.752451+0000","last_clean":"2024-04-25T00:31:01.752451+0000","last_became_active":"2024-04-25T00:01:19.541877+0000","last_became_peered":"2024-04-25T00:01:19.541877+0000","last_unstale":"2024-04-25T00:31:01.752451+0000","last_undegraded":"2024-04-25T00:31:01.752451+0000","last_fullsized":"2024-04-25T00:31:01.752451+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4567","last_scrub_stamp":"2024-04-25T00:31:01.752205+0000","last_deep_scrub":"74'4567","last_deep_scrub_stamp":"2024-04-25T00:31:01.752205+0000","last_clean_scrub_stamp":"2024-04-25T00:31:01.752205+0000","objects_scrubbed":0,"log_size":4567,"log_dups_size":0,"ondisk_log_size":4567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:06:12.318696+0000","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":36,"num_write":24879,"num_write_kb":312940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.28","version":"70'4294","reported_seq":4430,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:50.283328+0000","last_change":"2024-04-25T00:30:50.283393+0000","last_active":"2024-04-25T00:30:50.283328+0000","last_peered":"2024-04-25T00:30:50.283328+0000","last_clean":"2024-04-25T00:30:50.283328+0000","last_became_active":"2024-04-25T00:01:05.372148+0000","last_became_peered":"2024-04-25T00:01:05.372148+0000","last_unstale":"2024-04-25T00:30:50.283328+0000","last_undegraded":"2024-04-25T00:30:50.283328+0000","last_fullsized":"2024-04-25T00:30:50.283328+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'4294","last_scrub_stamp":"2024-04-25T00:30:50.283109+0000","last_deep_scrub":"70'4294","last_deep_scrub_stamp":"2024-04-25T00:30:50.283109+0000","last_clean_scrub_stamp":"2024-04-25T00:30:50.283109+0000","objects_scrubbed":0,"log_size":4294,"log_dups_size":0,"ondisk_log_size":4294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:45.764661+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23157,"num_write_kb":289708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2f","version":"74'5194","reported_seq":5322,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:47.306861+0000","last_change":"2024-04-25T00:30:47.306894+0000","last_active":"2024-04-25T00:30:47.306861+0000","last_peered":"2024-04-25T00:30:47.306861+0000","last_clean":"2024-04-25T00:30:47.306861+0000","last_became_active":"2024-04-25T00:01:19.496310+0000","last_became_peered":"2024-04-25T00:01:19.496310+0000","last_unstale":"2024-04-25T00:30:47.306861+0000","last_undegraded":"2024-04-25T00:30:47.306861+0000","last_fullsized":"2024-04-25T00:30:47.306861+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5194","last_scrub_stamp":"2024-04-25T00:30:47.306735+0000","last_deep_scrub":"74'5194","last_deep_scrub_stamp":"2024-04-25T00:30:47.306735+0000","last_clean_scrub_stamp":"2024-04-25T00:30:47.306735+0000","objects_scrubbed":0,"log_size":5194,"log_dups_size":0,"ondisk_log_size":5194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:05:57.873227+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":28420,"num_write_kb":367977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.29","version":"69'4077","reported_seq":4213,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:05.750467+0000","last_change":"2024-04-25T00:31:05.750602+0000","last_active":"2024-04-25T00:31:05.750467+0000","last_peered":"2024-04-25T00:31:05.750467+0000","last_clean":"2024-04-25T00:31:05.750467+0000","last_became_active":"2024-04-25T00:01:05.361952+0000","last_became_peered":"2024-04-25T00:01:05.361952+0000","last_unstale":"2024-04-25T00:31:05.750467+0000","last_undegraded":"2024-04-25T00:31:05.750467+0000","last_fullsized":"2024-04-25T00:31:05.750467+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'4077","last_scrub_stamp":"2024-04-25T00:31:05.750132+0000","last_deep_scrub":"69'4077","last_deep_scrub_stamp":"2024-04-25T00:31:05.750132+0000","last_clean_scrub_stamp":"2024-04-25T00:31:05.750132+0000","objects_scrubbed":0,"log_size":4077,"log_dups_size":0,"ondisk_log_size":4077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:06:39.544359+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22153,"num_write_kb":297857,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2c","version":"74'3564","reported_seq":3693,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:04.692859+0000","last_change":"2024-04-25T00:31:04.692894+0000","last_active":"2024-04-25T00:31:04.692859+0000","last_peered":"2024-04-25T00:31:04.692859+0000","last_clean":"2024-04-25T00:31:04.692859+0000","last_became_active":"2024-04-25T00:01:19.507516+0000","last_became_peered":"2024-04-25T00:01:19.507516+0000","last_unstale":"2024-04-25T00:31:04.692859+0000","last_undegraded":"2024-04-25T00:31:04.692859+0000","last_fullsized":"2024-04-25T00:31:04.692859+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'3564","last_scrub_stamp":"2024-04-25T00:31:04.692730+0000","last_deep_scrub":"74'3564","last_deep_scrub_stamp":"2024-04-25T00:31:04.692730+0000","last_clean_scrub_stamp":"2024-04-25T00:31:04.692730+0000","objects_scrubbed":0,"log_size":3564,"log_dups_size":0,"ondisk_log_size":3564,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:15:45.291641+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":19793,"num_write_kb":247542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2a","version":"69'4229","reported_seq":4365,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:51.097972+0000","last_change":"2024-04-25T00:30:51.098034+0000","last_active":"2024-04-25T00:30:51.097972+0000","last_peered":"2024-04-25T00:30:51.097972+0000","last_clean":"2024-04-25T00:30:51.097972+0000","last_became_active":"2024-04-25T00:01:05.386919+0000","last_became_peered":"2024-04-25T00:01:05.386919+0000","last_unstale":"2024-04-25T00:30:51.097972+0000","last_undegraded":"2024-04-25T00:30:51.097972+0000","last_fullsized":"2024-04-25T00:30:51.097972+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'4229","last_scrub_stamp":"2024-04-25T00:30:51.097711+0000","last_deep_scrub":"69'4229","last_deep_scrub_stamp":"2024-04-25T00:30:51.097711+0000","last_clean_scrub_stamp":"2024-04-25T00:30:51.097711+0000","objects_scrubbed":0,"log_size":4229,"log_dups_size":0,"ondisk_log_size":4229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:18:39.189133+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22430,"num_write_kb":302838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2d","version":"74'4662","reported_seq":4791,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:58.806254+0000","last_change":"2024-04-25T00:30:58.806319+0000","last_active":"2024-04-25T00:30:58.806254+0000","last_peered":"2024-04-25T00:30:58.806254+0000","last_clean":"2024-04-25T00:30:58.806254+0000","last_became_active":"2024-04-25T00:01:19.519303+0000","last_became_peered":"2024-04-25T00:01:19.519303+0000","last_unstale":"2024-04-25T00:30:58.806254+0000","last_undegraded":"2024-04-25T00:30:58.806254+0000","last_fullsized":"2024-04-25T00:30:58.806254+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4662","last_scrub_stamp":"2024-04-25T00:30:58.806038+0000","last_deep_scrub":"74'4662","last_deep_scrub_stamp":"2024-04-25T00:30:58.806038+0000","last_clean_scrub_stamp":"2024-04-25T00:30:58.806038+0000","objects_scrubbed":0,"log_size":4662,"log_dups_size":0,"ondisk_log_size":4662,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:20:17.244512+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":25332,"num_write_kb":310579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2b","version":"70'4434","reported_seq":4571,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:53.864498+0000","last_change":"2024-04-25T00:30:53.864556+0000","last_active":"2024-04-25T00:30:53.864498+0000","last_peered":"2024-04-25T00:30:53.864498+0000","last_clean":"2024-04-25T00:30:53.864498+0000","last_became_active":"2024-04-25T00:01:05.365548+0000","last_became_peered":"2024-04-25T00:01:05.365548+0000","last_unstale":"2024-04-25T00:30:53.864498+0000","last_undegraded":"2024-04-25T00:30:53.864498+0000","last_fullsized":"2024-04-25T00:30:53.864498+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'4434","last_scrub_stamp":"2024-04-25T00:30:53.864303+0000","last_deep_scrub":"70'4434","last_deep_scrub_stamp":"2024-04-25T00:30:53.864303+0000","last_clean_scrub_stamp":"2024-04-25T00:30:53.864303+0000","objects_scrubbed":0,"log_size":4434,"log_dups_size":0,"ondisk_log_size":4434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:49:29.728056+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":24224,"num_write_kb":326776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"74'4527","reported_seq":4656,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:03.885872+0000","last_change":"2024-04-25T00:31:03.885912+0000","last_active":"2024-04-25T00:31:03.885872+0000","last_peered":"2024-04-25T00:31:03.885872+0000","last_clean":"2024-04-25T00:31:03.885872+0000","last_became_active":"2024-04-25T00:01:19.514232+0000","last_became_peered":"2024-04-25T00:01:19.514232+0000","last_unstale":"2024-04-25T00:31:03.885872+0000","last_undegraded":"2024-04-25T00:31:03.885872+0000","last_fullsized":"2024-04-25T00:31:03.885872+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4527","last_scrub_stamp":"2024-04-25T00:31:03.885755+0000","last_deep_scrub":"74'4527","last_deep_scrub_stamp":"2024-04-25T00:31:03.885755+0000","last_clean_scrub_stamp":"2024-04-25T00:31:03.885755+0000","objects_scrubbed":0,"log_size":4527,"log_dups_size":0,"ondisk_log_size":4527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:07:28.486161+0000","scrub_duration":123,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":25088,"num_write_kb":313806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2c","version":"70'3677","reported_seq":3799,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:02.668938+0000","last_change":"2024-04-25T00:31:02.669056+0000","last_active":"2024-04-25T00:31:02.668938+0000","last_peered":"2024-04-25T00:31:02.668938+0000","last_clean":"2024-04-25T00:31:02.668938+0000","last_became_active":"2024-04-25T00:01:46.663874+0000","last_became_peered":"2024-04-25T00:01:46.663874+0000","last_unstale":"2024-04-25T00:31:02.668938+0000","last_undegraded":"2024-04-25T00:31:02.668938+0000","last_fullsized":"2024-04-25T00:31:02.668938+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'3677","last_scrub_stamp":"2024-04-25T00:31:02.668649+0000","last_deep_scrub":"70'3677","last_deep_scrub_stamp":"2024-04-25T00:31:02.668649+0000","last_clean_scrub_stamp":"2024-04-25T00:31:02.668649+0000","objects_scrubbed":0,"log_size":3677,"log_dups_size":0,"ondisk_log_size":3677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:12:10.491126+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20325,"num_write_kb":253590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2b","version":"74'4916","reported_seq":5049,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:01.385421+0000","last_change":"2024-04-25T00:31:01.385453+0000","last_active":"2024-04-25T00:31:01.385421+0000","last_peered":"2024-04-25T00:31:01.385421+0000","last_clean":"2024-04-25T00:31:01.385421+0000","last_became_active":"2024-04-25T00:01:19.500568+0000","last_became_peered":"2024-04-25T00:01:19.500568+0000","last_unstale":"2024-04-25T00:31:01.385421+0000","last_undegraded":"2024-04-25T00:31:01.385421+0000","last_fullsized":"2024-04-25T00:31:01.385421+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4916","last_scrub_stamp":"2024-04-25T00:31:01.385297+0000","last_deep_scrub":"74'4916","last_deep_scrub_stamp":"2024-04-25T00:31:01.385297+0000","last_clean_scrub_stamp":"2024-04-25T00:31:01.385297+0000","objects_scrubbed":0,"log_size":4916,"log_dups_size":0,"ondisk_log_size":4916,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:29:13.630138+0000","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":26841,"num_write_kb":321097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2d","version":"69'4618","reported_seq":4754,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:52.271068+0000","last_change":"2024-04-25T00:30:52.271136+0000","last_active":"2024-04-25T00:30:52.271068+0000","last_peered":"2024-04-25T00:30:52.271068+0000","last_clean":"2024-04-25T00:30:52.271068+0000","last_became_active":"2024-04-25T00:01:05.369919+0000","last_became_peered":"2024-04-25T00:01:05.369919+0000","last_unstale":"2024-04-25T00:30:52.271068+0000","last_undegraded":"2024-04-25T00:30:52.271068+0000","last_fullsized":"2024-04-25T00:30:52.271068+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'4618","last_scrub_stamp":"2024-04-25T00:30:52.270855+0000","last_deep_scrub":"69'4618","last_deep_scrub_stamp":"2024-04-25T00:30:52.270855+0000","last_clean_scrub_stamp":"2024-04-25T00:30:52.270855+0000","objects_scrubbed":0,"log_size":4618,"log_dups_size":0,"ondisk_log_size":4618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:21:23.828791+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25225,"num_write_kb":326547,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.28","version":"74'3900","reported_seq":4029,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:10.913468+0000","last_change":"2024-04-25T00:31:10.913506+0000","last_active":"2024-04-25T00:31:10.913468+0000","last_peered":"2024-04-25T00:31:10.913468+0000","last_clean":"2024-04-25T00:31:10.913468+0000","last_became_active":"2024-04-25T00:01:19.539775+0000","last_became_peered":"2024-04-25T00:01:19.539775+0000","last_unstale":"2024-04-25T00:31:10.913468+0000","last_undegraded":"2024-04-25T00:31:10.913468+0000","last_fullsized":"2024-04-25T00:31:10.913468+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'3900","last_scrub_stamp":"2024-04-25T00:31:10.913354+0000","last_deep_scrub":"74'3900","last_deep_scrub_stamp":"2024-04-25T00:31:10.913354+0000","last_clean_scrub_stamp":"2024-04-25T00:31:10.913354+0000","objects_scrubbed":0,"log_size":3900,"log_dups_size":0,"ondisk_log_size":3900,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:12:30.122869+0000","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":21234,"num_write_kb":281892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"70'3829","reported_seq":3965,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:52.866656+0000","last_change":"2024-04-25T00:30:52.866656+0000","last_active":"2024-04-25T00:30:52.866656+0000","last_peered":"2024-04-25T00:30:52.866656+0000","last_clean":"2024-04-25T00:30:52.866656+0000","last_became_active":"2024-04-25T00:01:05.366264+0000","last_became_peered":"2024-04-25T00:01:05.366264+0000","last_unstale":"2024-04-25T00:30:52.866656+0000","last_undegraded":"2024-04-25T00:30:52.866656+0000","last_fullsized":"2024-04-25T00:30:52.866656+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'3829","last_scrub_stamp":"2024-04-25T00:30:52.866392+0000","last_deep_scrub":"70'3829","last_deep_scrub_stamp":"2024-04-25T00:30:52.866392+0000","last_clean_scrub_stamp":"2024-04-25T00:30:52.866392+0000","objects_scrubbed":0,"log_size":3829,"log_dups_size":0,"ondisk_log_size":3829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:32:08.407884+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19822,"num_write_kb":292385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.29","version":"74'4218","reported_seq":4347,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:03.706648+0000","last_change":"2024-04-25T00:31:03.706698+0000","last_active":"2024-04-25T00:31:03.706648+0000","last_peered":"2024-04-25T00:31:03.706648+0000","last_clean":"2024-04-25T00:31:03.706648+0000","last_became_active":"2024-04-25T00:01:19.496517+0000","last_became_peered":"2024-04-25T00:01:19.496517+0000","last_unstale":"2024-04-25T00:31:03.706648+0000","last_undegraded":"2024-04-25T00:31:03.706648+0000","last_fullsized":"2024-04-25T00:31:03.706648+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4218","last_scrub_stamp":"2024-04-25T00:31:03.706474+0000","last_deep_scrub":"74'4218","last_deep_scrub_stamp":"2024-04-25T00:31:03.706474+0000","last_clean_scrub_stamp":"2024-04-25T00:31:03.706474+0000","objects_scrubbed":0,"log_size":4218,"log_dups_size":0,"ondisk_log_size":4218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:09:17.508648+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":23201,"num_write_kb":293725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2f","version":"70'5230","reported_seq":5367,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:55.200044+0000","last_change":"2024-04-25T00:30:55.200044+0000","last_active":"2024-04-25T00:30:55.200044+0000","last_peered":"2024-04-25T00:30:55.200044+0000","last_clean":"2024-04-25T00:30:55.200044+0000","last_became_active":"2024-04-25T00:01:05.379090+0000","last_became_peered":"2024-04-25T00:01:05.379090+0000","last_unstale":"2024-04-25T00:30:55.200044+0000","last_undegraded":"2024-04-25T00:30:55.200044+0000","last_fullsized":"2024-04-25T00:30:55.200044+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'5230","last_scrub_stamp":"2024-04-25T00:30:55.199740+0000","last_deep_scrub":"70'5230","last_deep_scrub_stamp":"2024-04-25T00:30:55.199740+0000","last_clean_scrub_stamp":"2024-04-25T00:30:55.199740+0000","objects_scrubbed":0,"log_size":5230,"log_dups_size":0,"ondisk_log_size":5230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:15:35.798652+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":27866,"num_write_kb":365657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.36","version":"74'4384","reported_seq":4513,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:11.092412+0000","last_change":"2024-04-25T00:31:11.092482+0000","last_active":"2024-04-25T00:31:11.092412+0000","last_peered":"2024-04-25T00:31:11.092412+0000","last_clean":"2024-04-25T00:31:11.092412+0000","last_became_active":"2024-04-25T00:01:19.529410+0000","last_became_peered":"2024-04-25T00:01:19.529410+0000","last_unstale":"2024-04-25T00:31:11.092412+0000","last_undegraded":"2024-04-25T00:31:11.092412+0000","last_fullsized":"2024-04-25T00:31:11.092412+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4384","last_scrub_stamp":"2024-04-25T00:31:11.092194+0000","last_deep_scrub":"74'4384","last_deep_scrub_stamp":"2024-04-25T00:31:11.092194+0000","last_clean_scrub_stamp":"2024-04-25T00:31:11.092194+0000","objects_scrubbed":0,"log_size":4384,"log_dups_size":0,"ondisk_log_size":4384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:44:28.141757+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":7,"num_write":24230,"num_write_kb":310193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.30","version":"69'4051","reported_seq":4187,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:56.213000+0000","last_change":"2024-04-25T00:30:56.213000+0000","last_active":"2024-04-25T00:30:56.213000+0000","last_peered":"2024-04-25T00:30:56.213000+0000","last_clean":"2024-04-25T00:30:56.213000+0000","last_became_active":"2024-04-25T00:01:05.369848+0000","last_became_peered":"2024-04-25T00:01:05.369848+0000","last_unstale":"2024-04-25T00:30:56.213000+0000","last_undegraded":"2024-04-25T00:30:56.213000+0000","last_fullsized":"2024-04-25T00:30:56.213000+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'4051","last_scrub_stamp":"2024-04-25T00:30:56.212707+0000","last_deep_scrub":"69'4051","last_deep_scrub_stamp":"2024-04-25T00:30:56.212707+0000","last_clean_scrub_stamp":"2024-04-25T00:30:56.212707+0000","objects_scrubbed":0,"log_size":4051,"log_dups_size":0,"ondisk_log_size":4051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:18:02.374659+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21793,"num_write_kb":288101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.37","version":"74'6689","reported_seq":6820,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:58.225133+0000","last_change":"2024-04-25T00:30:58.225203+0000","last_active":"2024-04-25T00:30:58.225133+0000","last_peered":"2024-04-25T00:30:58.225133+0000","last_clean":"2024-04-25T00:30:58.225133+0000","last_became_active":"2024-04-25T00:01:19.511349+0000","last_became_peered":"2024-04-25T00:01:19.511349+0000","last_unstale":"2024-04-25T00:30:58.225133+0000","last_undegraded":"2024-04-25T00:30:58.225133+0000","last_fullsized":"2024-04-25T00:30:58.225133+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6689","last_scrub_stamp":"2024-04-25T00:30:58.224895+0000","last_deep_scrub":"74'6689","last_deep_scrub_stamp":"2024-04-25T00:30:58.224895+0000","last_clean_scrub_stamp":"2024-04-25T00:30:58.224895+0000","objects_scrubbed":0,"log_size":6689,"log_dups_size":0,"ondisk_log_size":6689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:20:16.663369+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":8,"num_write":38367,"num_write_kb":445068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.31","version":"70'3993","reported_seq":4129,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:48.586033+0000","last_change":"2024-04-25T00:30:48.586033+0000","last_active":"2024-04-25T00:30:48.586033+0000","last_peered":"2024-04-25T00:30:48.586033+0000","last_clean":"2024-04-25T00:30:48.586033+0000","last_became_active":"2024-04-25T00:01:05.389707+0000","last_became_peered":"2024-04-25T00:01:05.389707+0000","last_unstale":"2024-04-25T00:30:48.586033+0000","last_undegraded":"2024-04-25T00:30:48.586033+0000","last_fullsized":"2024-04-25T00:30:48.586033+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'3993","last_scrub_stamp":"2024-04-25T00:30:48.585749+0000","last_deep_scrub":"70'3993","last_deep_scrub_stamp":"2024-04-25T00:30:48.585749+0000","last_clean_scrub_stamp":"2024-04-25T00:30:48.585749+0000","objects_scrubbed":0,"log_size":3993,"log_dups_size":0,"ondisk_log_size":3993,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:41:42.620564+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21619,"num_write_kb":287362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.34","version":"74'3411","reported_seq":3541,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:59.195126+0000","last_change":"2024-04-25T00:30:59.195215+0000","last_active":"2024-04-25T00:30:59.195126+0000","last_peered":"2024-04-25T00:30:59.195126+0000","last_clean":"2024-04-25T00:30:59.195126+0000","last_became_active":"2024-04-25T00:01:19.512978+0000","last_became_peered":"2024-04-25T00:01:19.512978+0000","last_unstale":"2024-04-25T00:30:59.195126+0000","last_undegraded":"2024-04-25T00:30:59.195126+0000","last_fullsized":"2024-04-25T00:30:59.195126+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'3411","last_scrub_stamp":"2024-04-25T00:30:59.194889+0000","last_deep_scrub":"74'3411","last_deep_scrub_stamp":"2024-04-25T00:30:59.194889+0000","last_clean_scrub_stamp":"2024-04-25T00:30:59.194889+0000","objects_scrubbed":0,"log_size":3411,"log_dups_size":0,"ondisk_log_size":3411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:52:39.304519+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":34,"num_write":18670,"num_write_kb":234685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"70'3759","reported_seq":3895,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:59.499587+0000","last_change":"2024-04-25T00:30:59.499651+0000","last_active":"2024-04-25T00:30:59.499587+0000","last_peered":"2024-04-25T00:30:59.499587+0000","last_clean":"2024-04-25T00:30:59.499587+0000","last_became_active":"2024-04-25T00:01:05.376570+0000","last_became_peered":"2024-04-25T00:01:05.376570+0000","last_unstale":"2024-04-25T00:30:59.499587+0000","last_undegraded":"2024-04-25T00:30:59.499587+0000","last_fullsized":"2024-04-25T00:30:59.499587+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'3759","last_scrub_stamp":"2024-04-25T00:30:59.499378+0000","last_deep_scrub":"70'3759","last_deep_scrub_stamp":"2024-04-25T00:30:59.499378+0000","last_clean_scrub_stamp":"2024-04-25T00:30:59.499378+0000","objects_scrubbed":0,"log_size":3759,"log_dups_size":0,"ondisk_log_size":3759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:54.980931+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":20571,"num_write_kb":279348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.35","version":"74'4078","reported_seq":4208,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:16.445448+0000","last_change":"2024-04-25T00:31:16.445492+0000","last_active":"2024-04-25T00:31:16.445448+0000","last_peered":"2024-04-25T00:31:16.445448+0000","last_clean":"2024-04-25T00:31:16.445448+0000","last_became_active":"2024-04-25T00:01:19.516142+0000","last_became_peered":"2024-04-25T00:01:19.516142+0000","last_unstale":"2024-04-25T00:31:16.445448+0000","last_undegraded":"2024-04-25T00:31:16.445448+0000","last_fullsized":"2024-04-25T00:31:16.445448+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4078","last_scrub_stamp":"2024-04-25T00:31:16.445294+0000","last_deep_scrub":"74'4078","last_deep_scrub_stamp":"2024-04-25T00:31:16.445294+0000","last_clean_scrub_stamp":"2024-04-25T00:31:16.445294+0000","objects_scrubbed":0,"log_size":4078,"log_dups_size":0,"ondisk_log_size":4078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:46:59.149829+0000","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":5,"num_write":22241,"num_write_kb":273608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.33","version":"70'3191","reported_seq":3326,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:52.564776+0000","last_change":"2024-04-25T00:30:52.564842+0000","last_active":"2024-04-25T00:30:52.564776+0000","last_peered":"2024-04-25T00:30:52.564776+0000","last_clean":"2024-04-25T00:30:52.564776+0000","last_became_active":"2024-04-25T00:01:05.380236+0000","last_became_peered":"2024-04-25T00:01:05.380236+0000","last_unstale":"2024-04-25T00:30:52.564776+0000","last_undegraded":"2024-04-25T00:30:52.564776+0000","last_fullsized":"2024-04-25T00:30:52.564776+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'3191","last_scrub_stamp":"2024-04-25T00:30:52.564505+0000","last_deep_scrub":"70'3191","last_deep_scrub_stamp":"2024-04-25T00:30:52.564505+0000","last_clean_scrub_stamp":"2024-04-25T00:30:52.564505+0000","objects_scrubbed":0,"log_size":3191,"log_dups_size":0,"ondisk_log_size":3191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:44:24.210435+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17526,"num_write_kb":230308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.32","version":"74'4055","reported_seq":4183,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:08.995088+0000","last_change":"2024-04-25T00:31:08.995158+0000","last_active":"2024-04-25T00:31:08.995088+0000","last_peered":"2024-04-25T00:31:08.995088+0000","last_clean":"2024-04-25T00:31:08.995088+0000","last_became_active":"2024-04-25T00:01:19.524609+0000","last_became_peered":"2024-04-25T00:01:19.524609+0000","last_unstale":"2024-04-25T00:31:08.995088+0000","last_undegraded":"2024-04-25T00:31:08.995088+0000","last_fullsized":"2024-04-25T00:31:08.995088+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4055","last_scrub_stamp":"2024-04-25T00:31:08.994866+0000","last_deep_scrub":"74'4055","last_deep_scrub_stamp":"2024-04-25T00:31:08.994866+0000","last_clean_scrub_stamp":"2024-04-25T00:31:08.994866+0000","objects_scrubbed":0,"log_size":4055,"log_dups_size":0,"ondisk_log_size":4055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:52:34.232298+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":21801,"num_write_kb":276048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.34","version":"70'2711","reported_seq":2846,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:44.388509+0000","last_change":"2024-04-25T00:30:44.388563+0000","last_active":"2024-04-25T00:30:44.388509+0000","last_peered":"2024-04-25T00:30:44.388509+0000","last_clean":"2024-04-25T00:30:44.388509+0000","last_became_active":"2024-04-25T00:01:05.384997+0000","last_became_peered":"2024-04-25T00:01:05.384997+0000","last_unstale":"2024-04-25T00:30:44.388509+0000","last_undegraded":"2024-04-25T00:30:44.388509+0000","last_fullsized":"2024-04-25T00:30:44.388509+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'2711","last_scrub_stamp":"2024-04-25T00:30:44.388310+0000","last_deep_scrub":"70'2711","last_deep_scrub_stamp":"2024-04-25T00:30:44.388310+0000","last_clean_scrub_stamp":"2024-04-25T00:30:44.388310+0000","objects_scrubbed":0,"log_size":2711,"log_dups_size":0,"ondisk_log_size":2711,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:46:59.186973+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14839,"num_write_kb":207133,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.33","version":"74'3434","reported_seq":3565,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:08.606496+0000","last_change":"2024-04-25T00:31:08.606557+0000","last_active":"2024-04-25T00:31:08.606496+0000","last_peered":"2024-04-25T00:31:08.606496+0000","last_clean":"2024-04-25T00:31:08.606496+0000","last_became_active":"2024-04-25T00:01:19.515764+0000","last_became_peered":"2024-04-25T00:01:19.515764+0000","last_unstale":"2024-04-25T00:31:08.606496+0000","last_undegraded":"2024-04-25T00:31:08.606496+0000","last_fullsized":"2024-04-25T00:31:08.606496+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'3434","last_scrub_stamp":"2024-04-25T00:31:08.606255+0000","last_deep_scrub":"74'3434","last_deep_scrub_stamp":"2024-04-25T00:31:08.606255+0000","last_clean_scrub_stamp":"2024-04-25T00:31:08.606255+0000","objects_scrubbed":0,"log_size":3434,"log_dups_size":0,"ondisk_log_size":3434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:59:35.846309+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":5,"num_write":19347,"num_write_kb":241776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"69'3819","reported_seq":3955,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:48.346026+0000","last_change":"2024-04-25T00:30:48.346092+0000","last_active":"2024-04-25T00:30:48.346026+0000","last_peered":"2024-04-25T00:30:48.346026+0000","last_clean":"2024-04-25T00:30:48.346026+0000","last_became_active":"2024-04-25T00:01:05.388999+0000","last_became_peered":"2024-04-25T00:01:05.388999+0000","last_unstale":"2024-04-25T00:30:48.346026+0000","last_undegraded":"2024-04-25T00:30:48.346026+0000","last_fullsized":"2024-04-25T00:30:48.346026+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'3819","last_scrub_stamp":"2024-04-25T00:30:48.345797+0000","last_deep_scrub":"69'3819","last_deep_scrub_stamp":"2024-04-25T00:30:48.345797+0000","last_clean_scrub_stamp":"2024-04-25T00:30:48.345797+0000","objects_scrubbed":0,"log_size":3819,"log_dups_size":0,"ondisk_log_size":3819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:20:06.784272+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":19986,"num_write_kb":272748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.30","version":"74'4335","reported_seq":4462,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:04.415684+0000","last_change":"2024-04-25T00:31:04.415719+0000","last_active":"2024-04-25T00:31:04.415684+0000","last_peered":"2024-04-25T00:31:04.415684+0000","last_clean":"2024-04-25T00:31:04.415684+0000","last_became_active":"2024-04-25T00:01:19.519085+0000","last_became_peered":"2024-04-25T00:01:19.519085+0000","last_unstale":"2024-04-25T00:31:04.415684+0000","last_undegraded":"2024-04-25T00:31:04.415684+0000","last_fullsized":"2024-04-25T00:31:04.415684+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4335","last_scrub_stamp":"2024-04-25T00:31:04.415565+0000","last_deep_scrub":"74'4335","last_deep_scrub_stamp":"2024-04-25T00:31:04.415565+0000","last_clean_scrub_stamp":"2024-04-25T00:31:04.415565+0000","objects_scrubbed":0,"log_size":4335,"log_dups_size":0,"ondisk_log_size":4335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:06:38.209793+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23507,"num_write_kb":286989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.36","version":"69'4866","reported_seq":5001,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:59.404114+0000","last_change":"2024-04-25T00:30:59.404177+0000","last_active":"2024-04-25T00:30:59.404114+0000","last_peered":"2024-04-25T00:30:59.404114+0000","last_clean":"2024-04-25T00:30:59.404114+0000","last_became_active":"2024-04-25T00:01:05.367893+0000","last_became_peered":"2024-04-25T00:01:05.367893+0000","last_unstale":"2024-04-25T00:30:59.404114+0000","last_undegraded":"2024-04-25T00:30:59.404114+0000","last_fullsized":"2024-04-25T00:30:59.404114+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'4866","last_scrub_stamp":"2024-04-25T00:30:59.403892+0000","last_deep_scrub":"69'4866","last_deep_scrub_stamp":"2024-04-25T00:30:59.403892+0000","last_clean_scrub_stamp":"2024-04-25T00:30:59.403892+0000","objects_scrubbed":0,"log_size":4866,"log_dups_size":0,"ondisk_log_size":4866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:09:13.206066+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26504,"num_write_kb":327449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.31","version":"74'4017","reported_seq":4145,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:03.444511+0000","last_change":"2024-04-25T00:31:03.444544+0000","last_active":"2024-04-25T00:31:03.444511+0000","last_peered":"2024-04-25T00:31:03.444511+0000","last_clean":"2024-04-25T00:31:03.444511+0000","last_became_active":"2024-04-25T00:01:19.513184+0000","last_became_peered":"2024-04-25T00:01:19.513184+0000","last_unstale":"2024-04-25T00:31:03.444511+0000","last_undegraded":"2024-04-25T00:31:03.444511+0000","last_fullsized":"2024-04-25T00:31:03.444511+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4017","last_scrub_stamp":"2024-04-25T00:31:03.444373+0000","last_deep_scrub":"74'4017","last_deep_scrub_stamp":"2024-04-25T00:31:03.444373+0000","last_clean_scrub_stamp":"2024-04-25T00:31:03.444373+0000","objects_scrubbed":0,"log_size":4017,"log_dups_size":0,"ondisk_log_size":4017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:17:55.839748+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":21984,"num_write_kb":288846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.37","version":"70'5540","reported_seq":5675,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:47.074024+0000","last_change":"2024-04-25T00:30:47.074090+0000","last_active":"2024-04-25T00:30:47.074024+0000","last_peered":"2024-04-25T00:30:47.074024+0000","last_clean":"2024-04-25T00:30:47.074024+0000","last_became_active":"2024-04-25T00:01:05.361586+0000","last_became_peered":"2024-04-25T00:01:05.361586+0000","last_unstale":"2024-04-25T00:30:47.074024+0000","last_undegraded":"2024-04-25T00:30:47.074024+0000","last_fullsized":"2024-04-25T00:30:47.074024+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'5540","last_scrub_stamp":"2024-04-25T00:30:47.073815+0000","last_deep_scrub":"70'5540","last_deep_scrub_stamp":"2024-04-25T00:30:47.073815+0000","last_clean_scrub_stamp":"2024-04-25T00:30:47.073815+0000","objects_scrubbed":0,"log_size":5540,"log_dups_size":0,"ondisk_log_size":5540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:25:01.256353+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30654,"num_write_kb":393416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3e","version":"74'5058","reported_seq":5189,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:09.828426+0000","last_change":"2024-04-25T00:31:09.828487+0000","last_active":"2024-04-25T00:31:09.828426+0000","last_peered":"2024-04-25T00:31:09.828426+0000","last_clean":"2024-04-25T00:31:09.828426+0000","last_became_active":"2024-04-25T00:01:19.513314+0000","last_became_peered":"2024-04-25T00:01:19.513314+0000","last_unstale":"2024-04-25T00:31:09.828426+0000","last_undegraded":"2024-04-25T00:31:09.828426+0000","last_fullsized":"2024-04-25T00:31:09.828426+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5058","last_scrub_stamp":"2024-04-25T00:31:09.828262+0000","last_deep_scrub":"74'5058","last_deep_scrub_stamp":"2024-04-25T00:31:09.828262+0000","last_clean_scrub_stamp":"2024-04-25T00:31:09.828262+0000","objects_scrubbed":0,"log_size":5058,"log_dups_size":0,"ondisk_log_size":5058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:59:56.085627+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":15,"num_write":27439,"num_write_kb":359791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.38","version":"70'4026","reported_seq":4161,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:46.599801+0000","last_change":"2024-04-25T00:30:46.599801+0000","last_active":"2024-04-25T00:30:46.599801+0000","last_peered":"2024-04-25T00:30:46.599801+0000","last_clean":"2024-04-25T00:30:46.599801+0000","last_became_active":"2024-04-25T00:01:05.389724+0000","last_became_peered":"2024-04-25T00:01:05.389724+0000","last_unstale":"2024-04-25T00:30:46.599801+0000","last_undegraded":"2024-04-25T00:30:46.599801+0000","last_fullsized":"2024-04-25T00:30:46.599801+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'4026","last_scrub_stamp":"2024-04-25T00:30:46.599517+0000","last_deep_scrub":"70'4026","last_deep_scrub_stamp":"2024-04-25T00:30:46.599517+0000","last_clean_scrub_stamp":"2024-04-25T00:30:46.599517+0000","objects_scrubbed":0,"log_size":4026,"log_dups_size":0,"ondisk_log_size":4026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:40:44.481539+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21352,"num_write_kb":290527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3f","version":"74'4365","reported_seq":4498,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:53.632870+0000","last_change":"2024-04-25T00:30:53.632995+0000","last_active":"2024-04-25T00:30:53.632870+0000","last_peered":"2024-04-25T00:30:53.632870+0000","last_clean":"2024-04-25T00:30:53.632870+0000","last_became_active":"2024-04-25T00:01:19.508049+0000","last_became_peered":"2024-04-25T00:01:19.508049+0000","last_unstale":"2024-04-25T00:30:53.632870+0000","last_undegraded":"2024-04-25T00:30:53.632870+0000","last_fullsized":"2024-04-25T00:30:53.632870+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4365","last_scrub_stamp":"2024-04-25T00:30:53.632570+0000","last_deep_scrub":"74'4365","last_deep_scrub_stamp":"2024-04-25T00:30:53.632570+0000","last_clean_scrub_stamp":"2024-04-25T00:30:53.632570+0000","objects_scrubbed":0,"log_size":4365,"log_dups_size":0,"ondisk_log_size":4365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:13:40.444328+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":8,"num_write":24611,"num_write_kb":307761,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"70'3970","reported_seq":4105,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:51.354932+0000","last_change":"2024-04-25T00:30:51.354932+0000","last_active":"2024-04-25T00:30:51.354932+0000","last_peered":"2024-04-25T00:30:51.354932+0000","last_clean":"2024-04-25T00:30:51.354932+0000","last_became_active":"2024-04-25T00:01:05.369739+0000","last_became_peered":"2024-04-25T00:01:05.369739+0000","last_unstale":"2024-04-25T00:30:51.354932+0000","last_undegraded":"2024-04-25T00:30:51.354932+0000","last_fullsized":"2024-04-25T00:30:51.354932+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'3970","last_scrub_stamp":"2024-04-25T00:30:51.354655+0000","last_deep_scrub":"70'3970","last_deep_scrub_stamp":"2024-04-25T00:30:51.354655+0000","last_clean_scrub_stamp":"2024-04-25T00:30:51.354655+0000","objects_scrubbed":0,"log_size":3970,"log_dups_size":0,"ondisk_log_size":3970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:54:55.160015+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22008,"num_write_kb":289131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3c","version":"74'3966","reported_seq":4097,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:07.038408+0000","last_change":"2024-04-25T00:31:07.038408+0000","last_active":"2024-04-25T00:31:07.038408+0000","last_peered":"2024-04-25T00:31:07.038408+0000","last_clean":"2024-04-25T00:31:07.038408+0000","last_became_active":"2024-04-25T00:01:19.520787+0000","last_became_peered":"2024-04-25T00:01:19.520787+0000","last_unstale":"2024-04-25T00:31:07.038408+0000","last_undegraded":"2024-04-25T00:31:07.038408+0000","last_fullsized":"2024-04-25T00:31:07.038408+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'3966","last_scrub_stamp":"2024-04-25T00:31:07.038226+0000","last_deep_scrub":"74'3966","last_deep_scrub_stamp":"2024-04-25T00:31:07.038226+0000","last_clean_scrub_stamp":"2024-04-25T00:31:07.038226+0000","objects_scrubbed":0,"log_size":3966,"log_dups_size":0,"ondisk_log_size":3966,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:00:46.202049+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":22683,"num_write_kb":271349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3a","version":"70'3294","reported_seq":3429,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:56.669228+0000","last_change":"2024-04-25T00:30:56.669228+0000","last_active":"2024-04-25T00:30:56.669228+0000","last_peered":"2024-04-25T00:30:56.669228+0000","last_clean":"2024-04-25T00:30:56.669228+0000","last_became_active":"2024-04-25T00:01:05.358477+0000","last_became_peered":"2024-04-25T00:01:05.358477+0000","last_unstale":"2024-04-25T00:30:56.669228+0000","last_undegraded":"2024-04-25T00:30:56.669228+0000","last_fullsized":"2024-04-25T00:30:56.669228+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'3294","last_scrub_stamp":"2024-04-25T00:30:56.668950+0000","last_deep_scrub":"70'3294","last_deep_scrub_stamp":"2024-04-25T00:30:56.668950+0000","last_clean_scrub_stamp":"2024-04-25T00:30:56.668950+0000","objects_scrubbed":0,"log_size":3294,"log_dups_size":0,"ondisk_log_size":3294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:13:52.150502+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17420,"num_write_kb":225566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3d","version":"74'5153","reported_seq":5283,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:56.232210+0000","last_change":"2024-04-25T00:30:56.232275+0000","last_active":"2024-04-25T00:30:56.232210+0000","last_peered":"2024-04-25T00:30:56.232210+0000","last_clean":"2024-04-25T00:30:56.232210+0000","last_became_active":"2024-04-25T00:01:19.507812+0000","last_became_peered":"2024-04-25T00:01:19.507812+0000","last_unstale":"2024-04-25T00:30:56.232210+0000","last_undegraded":"2024-04-25T00:30:56.232210+0000","last_fullsized":"2024-04-25T00:30:56.232210+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5153","last_scrub_stamp":"2024-04-25T00:30:56.232005+0000","last_deep_scrub":"74'5153","last_deep_scrub_stamp":"2024-04-25T00:30:56.232005+0000","last_clean_scrub_stamp":"2024-04-25T00:30:56.232005+0000","objects_scrubbed":0,"log_size":5153,"log_dups_size":0,"ondisk_log_size":5153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:41:50.266820+0000","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":28174,"num_write_kb":326190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3b","version":"70'3663","reported_seq":3798,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:46.333357+0000","last_change":"2024-04-25T00:30:46.333357+0000","last_active":"2024-04-25T00:30:46.333357+0000","last_peered":"2024-04-25T00:30:46.333357+0000","last_clean":"2024-04-25T00:30:46.333357+0000","last_became_active":"2024-04-25T00:01:05.378650+0000","last_became_peered":"2024-04-25T00:01:05.378650+0000","last_unstale":"2024-04-25T00:30:46.333357+0000","last_undegraded":"2024-04-25T00:30:46.333357+0000","last_fullsized":"2024-04-25T00:30:46.333357+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'3663","last_scrub_stamp":"2024-04-25T00:30:46.332991+0000","last_deep_scrub":"70'3663","last_deep_scrub_stamp":"2024-04-25T00:30:46.332991+0000","last_clean_scrub_stamp":"2024-04-25T00:30:46.332991+0000","objects_scrubbed":0,"log_size":3663,"log_dups_size":0,"ondisk_log_size":3663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:51:25.231176+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18938,"num_write_kb":284951,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3a","version":"74'3833","reported_seq":3960,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:57.844114+0000","last_change":"2024-04-25T00:30:57.844114+0000","last_active":"2024-04-25T00:30:57.844114+0000","last_peered":"2024-04-25T00:30:57.844114+0000","last_clean":"2024-04-25T00:30:57.844114+0000","last_became_active":"2024-04-25T00:01:19.513956+0000","last_became_peered":"2024-04-25T00:01:19.513956+0000","last_unstale":"2024-04-25T00:30:57.844114+0000","last_undegraded":"2024-04-25T00:30:57.844114+0000","last_fullsized":"2024-04-25T00:30:57.844114+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'3833","last_scrub_stamp":"2024-04-25T00:30:57.843830+0000","last_deep_scrub":"74'3833","last_deep_scrub_stamp":"2024-04-25T00:30:57.843830+0000","last_clean_scrub_stamp":"2024-04-25T00:30:57.843830+0000","objects_scrubbed":0,"log_size":3833,"log_dups_size":0,"ondisk_log_size":3833,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:44:29.489760+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20679,"num_write_kb":271382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"69'3516","reported_seq":3655,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:57.662264+0000","last_change":"2024-04-25T00:30:57.662264+0000","last_active":"2024-04-25T00:30:57.662264+0000","last_peered":"2024-04-25T00:30:57.662264+0000","last_clean":"2024-04-25T00:30:57.662264+0000","last_became_active":"2024-04-25T00:01:05.358866+0000","last_became_peered":"2024-04-25T00:01:05.358866+0000","last_unstale":"2024-04-25T00:30:57.662264+0000","last_undegraded":"2024-04-25T00:30:57.662264+0000","last_fullsized":"2024-04-25T00:30:57.662264+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'3516","last_scrub_stamp":"2024-04-25T00:30:57.661979+0000","last_deep_scrub":"69'3516","last_deep_scrub_stamp":"2024-04-25T00:30:57.661979+0000","last_clean_scrub_stamp":"2024-04-25T00:30:57.661979+0000","objects_scrubbed":0,"log_size":3516,"log_dups_size":0,"ondisk_log_size":3516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:46:49.144539+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":19226,"num_write_kb":253281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3b","version":"74'4130","reported_seq":4258,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:06.754737+0000","last_change":"2024-04-25T00:31:06.754737+0000","last_active":"2024-04-25T00:31:06.754737+0000","last_peered":"2024-04-25T00:31:06.754737+0000","last_clean":"2024-04-25T00:31:06.754737+0000","last_became_active":"2024-04-25T00:01:19.498699+0000","last_became_peered":"2024-04-25T00:01:19.498699+0000","last_unstale":"2024-04-25T00:31:06.754737+0000","last_undegraded":"2024-04-25T00:31:06.754737+0000","last_fullsized":"2024-04-25T00:31:06.754737+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4130","last_scrub_stamp":"2024-04-25T00:31:06.754549+0000","last_deep_scrub":"74'4130","last_deep_scrub_stamp":"2024-04-25T00:31:06.754549+0000","last_clean_scrub_stamp":"2024-04-25T00:31:06.754549+0000","objects_scrubbed":0,"log_size":4130,"log_dups_size":0,"ondisk_log_size":4130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:34:44.561074+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":3,"num_write":21749,"num_write_kb":288055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"70'3483","reported_seq":3620,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:51.542203+0000","last_change":"2024-04-25T00:30:51.542272+0000","last_active":"2024-04-25T00:30:51.542203+0000","last_peered":"2024-04-25T00:30:51.542203+0000","last_clean":"2024-04-25T00:30:51.542203+0000","last_became_active":"2024-04-25T00:01:05.370522+0000","last_became_peered":"2024-04-25T00:01:05.370522+0000","last_unstale":"2024-04-25T00:30:51.542203+0000","last_undegraded":"2024-04-25T00:30:51.542203+0000","last_fullsized":"2024-04-25T00:30:51.542203+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'3483","last_scrub_stamp":"2024-04-25T00:30:51.541962+0000","last_deep_scrub":"70'3483","last_deep_scrub_stamp":"2024-04-25T00:30:51.541962+0000","last_clean_scrub_stamp":"2024-04-25T00:30:51.541962+0000","objects_scrubbed":0,"log_size":3483,"log_dups_size":0,"ondisk_log_size":3483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T12:19:32.940834+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18326,"num_write_kb":261938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.38","version":"74'4120","reported_seq":4247,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:31:07.933649+0000","last_change":"2024-04-25T00:31:07.933649+0000","last_active":"2024-04-25T00:31:07.933649+0000","last_peered":"2024-04-25T00:31:07.933649+0000","last_clean":"2024-04-25T00:31:07.933649+0000","last_became_active":"2024-04-25T00:01:19.523450+0000","last_became_peered":"2024-04-25T00:01:19.523450+0000","last_unstale":"2024-04-25T00:31:07.933649+0000","last_undegraded":"2024-04-25T00:31:07.933649+0000","last_fullsized":"2024-04-25T00:31:07.933649+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4120","last_scrub_stamp":"2024-04-25T00:31:07.933472+0000","last_deep_scrub":"74'4120","last_deep_scrub_stamp":"2024-04-25T00:31:07.933472+0000","last_clean_scrub_stamp":"2024-04-25T00:31:07.933472+0000","objects_scrubbed":0,"log_size":4120,"log_dups_size":0,"ondisk_log_size":4120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:49:28.539437+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22879,"num_write_kb":288519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3e","version":"70'4991","reported_seq":5128,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:55.620709+0000","last_change":"2024-04-25T00:30:55.620865+0000","last_active":"2024-04-25T00:30:55.620709+0000","last_peered":"2024-04-25T00:30:55.620709+0000","last_clean":"2024-04-25T00:30:55.620709+0000","last_became_active":"2024-04-25T00:01:05.362606+0000","last_became_peered":"2024-04-25T00:01:05.362606+0000","last_unstale":"2024-04-25T00:30:55.620709+0000","last_undegraded":"2024-04-25T00:30:55.620709+0000","last_fullsized":"2024-04-25T00:30:55.620709+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'4991","last_scrub_stamp":"2024-04-25T00:30:55.620470+0000","last_deep_scrub":"70'4991","last_deep_scrub_stamp":"2024-04-25T00:30:55.620470+0000","last_clean_scrub_stamp":"2024-04-25T00:30:55.620470+0000","objects_scrubbed":0,"log_size":4991,"log_dups_size":0,"ondisk_log_size":4991,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:45:31.561538+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26684,"num_write_kb":361051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.39","version":"74'4850","reported_seq":4977,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:49.562032+0000","last_change":"2024-04-25T00:30:49.562032+0000","last_active":"2024-04-25T00:30:49.562032+0000","last_peered":"2024-04-25T00:30:49.562032+0000","last_clean":"2024-04-25T00:30:49.562032+0000","last_became_active":"2024-04-25T00:01:19.504991+0000","last_became_peered":"2024-04-25T00:01:19.504991+0000","last_unstale":"2024-04-25T00:30:49.562032+0000","last_undegraded":"2024-04-25T00:30:49.562032+0000","last_fullsized":"2024-04-25T00:30:49.562032+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4850","last_scrub_stamp":"2024-04-25T00:30:49.561893+0000","last_deep_scrub":"74'4850","last_deep_scrub_stamp":"2024-04-25T00:30:49.561893+0000","last_clean_scrub_stamp":"2024-04-25T00:30:49.561893+0000","objects_scrubbed":0,"log_size":4850,"log_dups_size":0,"ondisk_log_size":4850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:34:06.903945+0000","scrub_duration":123,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26612,"num_write_kb":323915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3f","version":"70'3815","reported_seq":3952,"reported_epoch":74,"state":"active+clean","last_fresh":"2024-04-25T00:30:50.372947+0000","last_change":"2024-04-25T00:30:50.373007+0000","last_active":"2024-04-25T00:30:50.372947+0000","last_peered":"2024-04-25T00:30:50.372947+0000","last_clean":"2024-04-25T00:30:50.372947+0000","last_became_active":"2024-04-25T00:01:05.389127+0000","last_became_peered":"2024-04-25T00:01:05.389127+0000","last_unstale":"2024-04-25T00:30:50.372947+0000","last_undegraded":"2024-04-25T00:30:50.372947+0000","last_fullsized":"2024-04-25T00:30:50.372947+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"70'3815","last_scrub_stamp":"2024-04-25T00:30:50.372743+0000","last_deep_scrub":"70'3815","last_deep_scrub_stamp":"2024-04-25T00:30:50.372743+0000","last_clean_scrub_stamp":"2024-04-25T00:30:50.372743+0000","objects_scrubbed":0,"log_size":3815,"log_dups_size":0,"ondisk_log_size":3815,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:10:18.579715+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20941,"num_write_kb":280561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":105,"num_read_kb":408,"num_write":1543888,"num_write_kb":19306803,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":280555,"ondisk_log_size":280555,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":19555305,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1733,"num_read_kb":229197,"num_write":5131,"num_write_kb":21279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":39174144,"data_stored":39110610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4730,"ondisk_log_size":4730,"up":64,"acting":64,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":3,"num_write":1412500,"num_write_kb":18776370,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":261221,"ondisk_log_size":261221,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":19117950,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":1590,"num_read_kb":257618,"num_write":5060,"num_write_kb":20770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":38297600,"data_stored":38235900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4724,"ondisk_log_size":4724,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509811,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":674532,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":673470,"kb_avail":93697308,"statfs":{"total":96636764160,"available":95946043392,"internally_reserved":0,"allocated":1077248,"data_stored":617327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":689633739},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":15,"seq":64424509811,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":843472,"kb_used_data":11784,"kb_used_omap":1,"kb_used_meta":831678,"kb_avail":93528368,"statfs":{"total":96636764160,"available":95773048832,"internally_reserved":0,"allocated":12066816,"data_stored":11609748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":851638726},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509810,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":870568,"kb_used_data":18016,"kb_used_omap":1,"kb_used_meta":852542,"kb_avail":93501272,"statfs":{"total":96636764160,"available":95745302528,"internally_reserved":0,"allocated":18448384,"data_stored":17985439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":873003461},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":15,"seq":64424509811,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":922240,"kb_used_data":15544,"kb_used_omap":1,"kb_used_meta":906686,"kb_avail":93449600,"statfs":{"total":96636764160,"available":95692390400,"internally_reserved":0,"allocated":15917056,"data_stored":15447989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":928446919},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":15,"seq":64424509811,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1076840,"kb_used_data":15968,"kb_used_omap":1,"kb_used_meta":1060862,"kb_avail":93295000,"statfs":{"total":96636764160,"available":95534080000,"internally_reserved":0,"allocated":16351232,"data_stored":15888732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1086323141},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509811,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":788556,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":787006,"kb_avail":93583284,"statfs":{"total":96636764160,"available":95829282816,"internally_reserved":0,"allocated":1576960,"data_stored":1119003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":805894597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":15,"seq":64424509810,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1028004,"kb_used_data":3164,"kb_used_omap":1,"kb_used_meta":1024830,"kb_avail":93343836,"statfs":{"total":96636764160,"available":95584088064,"internally_reserved":0,"allocated":3239936,"data_stored":2779219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1049426373},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834575219,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":962780,"kb_used_data":17364,"kb_used_omap":1,"kb_used_meta":945406,"kb_avail":93409060,"statfs":{"total":96636764160,"available":95650877440,"internally_reserved":0,"allocated":17780736,"data_stored":17307749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":968096197},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2260992,"data_stored":2250634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8396800,"data_stored":8389280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":10645504,"data_stored":10639768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8404992,"data_stored":8389952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4292608,"data_stored":4283082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4194304,"data_stored":4194304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":88810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":6979584,"data_stored":6970957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4296704,"data_stored":4279726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8400896,"data_stored":8389302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12587008,"data_stored":12583494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":6897664,"data_stored":6886949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T00:31:28.366 DEBUG:teuthology.orchestra.run.smithi116:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-25T00:31:28.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.518+0000 7fcd5b657640 1 Processor -- start 2024-04-25T00:31:28.521 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.518+0000 7fcd5b657640 1 -- start start 2024-04-25T00:31:28.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.522+0000 7fcd5b657640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcd54157510 0x7fcd5415ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:28.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.522+0000 7fcd5b657640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd540a9cb0 con 0x7fcd540a9630 2024-04-25T00:31:28.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.522+0000 7fcd5b657640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd540a9f90 con 0x7fcd54150d60 2024-04-25T00:31:28.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.522+0000 7fcd5b657640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7fcd540aa270 con 0x7fcd54157510 2024-04-25T00:31:28.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.522+0000 7fcd59bf4640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcd54157510 0x7fcd5415ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:28.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.522+0000 7fcd58bf2640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fcd540a9630 0x7fcd54156e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:46824/0 (socket says 172.21.15.116:46824) 2024-04-25T00:31:28.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.522+0000 7fcd59bf4640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7fcd54157510 0x7fcd5415ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51070/0 (socket says 172.21.15.116:51070) 2024-04-25T00:31:28.522 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.522+0000 7fcd58bf2640 1 -- 172.21.15.116:0/1071880375 learned_addr learned my addr 172.21.15.116:0/1071880375 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:31:28.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.522+0000 7fcd59bf4640 1 -- 172.21.15.116:0/1071880375 >> v1:172.21.15.174:6789/0 conn(0x7fcd54150d60 legacy=0x7fcd540c2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:28.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.522+0000 7fcd59bf4640 1 -- 172.21.15.116:0/1071880375 >> v1:172.21.15.116:6789/0 conn(0x7fcd540a9630 legacy=0x7fcd54156e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:28.523 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.522+0000 7fcd59bf4640 1 -- 172.21.15.116:0/1071880375 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd540aa550 con 0x7fcd54157510 2024-04-25T00:31:28.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.522+0000 7fcd59bf4640 1 --2- 172.21.15.116:0/1071880375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd54157510 0x7fcd5415ef60 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7fcd50009d90 tx=0x7fcd5001c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f941983b1de04583 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:28.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.522+0000 7fcd4bfff640 1 -- 172.21.15.116:0/1071880375 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd500b1070 con 0x7fcd54157510 2024-04-25T00:31:28.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.522+0000 7fcd4bfff640 1 -- 172.21.15.116:0/1071880375 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd540ab7a0 con 0x7fcd54157510 2024-04-25T00:31:28.525 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.522+0000 7fcd4bfff640 1 -- 172.21.15.116:0/1071880375 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd54157b70 con 0x7fcd54157510 2024-04-25T00:31:28.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.526+0000 7fcd5b657640 1 -- 172.21.15.116:0/1071880375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd54157510 msgr2=0x7fcd5415ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:28.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.526+0000 7fcd5b657640 1 --2- 172.21.15.116:0/1071880375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd54157510 0x7fcd5415ef60 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7fcd50009d90 tx=0x7fcd5001c420 comp rx=0 tx=0).stop 2024-04-25T00:31:28.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.526+0000 7fcd5b657640 1 -- 172.21.15.116:0/1071880375 shutdown_connections 2024-04-25T00:31:28.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.526+0000 7fcd5b657640 1 --2- 172.21.15.116:0/1071880375 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd54157510 0x7fcd5415ef60 unknown :-1 s=CLOSED pgs=1866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:28.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.526+0000 7fcd5b657640 1 -- 172.21.15.116:0/1071880375 >> 172.21.15.116:0/1071880375 conn(0x7fcd540c3a20 msgr2=0x7fcd540c3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:31:28.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.526+0000 7fcd5b657640 1 -- 172.21.15.116:0/1071880375 shutdown_connections 2024-04-25T00:31:28.526 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.526+0000 7fcd5b657640 1 -- 172.21.15.116:0/1071880375 wait complete. 2024-04-25T00:31:28.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.526+0000 7fcd5b657640 1 Processor -- start 2024-04-25T00:31:28.527 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.526+0000 7fcd5b657640 1 -- start start 2024-04-25T00:31:28.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.526+0000 7fcd5b657640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcd540a9630 0x7fcd5414cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:28.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.526+0000 7fcd5b657640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcd54150d60 0x7fcd5414f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:28.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.526+0000 7fcd5b657640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd54157510 0x7fcd541427d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:28.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.526+0000 7fcd5b657640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fcd54162660 con 0x7fcd54157510 2024-04-25T00:31:28.529 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.526+0000 7fcd5b657640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fcd541624e0 con 0x7fcd54150d60 2024-04-25T00:31:28.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.526+0000 7fcd5b657640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fcd541627e0 con 0x7fcd540a9630 2024-04-25T00:31:28.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.526+0000 7fcd59bf4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd54157510 0x7fcd541427d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:28.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.526+0000 7fcd59bf4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd54157510 0x7fcd541427d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51080/0 (socket says 172.21.15.116:51080) 2024-04-25T00:31:28.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.526+0000 7fcd59bf4640 1 -- 172.21.15.116:0/1278176673 learned_addr learned my addr 172.21.15.116:0/1278176673 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:31:28.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.526+0000 7fcd593f3640 1 --2- 172.21.15.116:0/1278176673 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcd540a9630 0x7fcd5414cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:28.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.526+0000 7fcd58bf2640 1 --2- 172.21.15.116:0/1278176673 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcd54150d60 0x7fcd5414f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:28.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.526+0000 7fcd59bf4640 1 -- 172.21.15.116:0/1278176673 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcd540a9630 msgr2=0x7fcd5414cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:28.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.526+0000 7fcd59bf4640 1 --2- 172.21.15.116:0/1278176673 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcd540a9630 0x7fcd5414cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:28.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.526+0000 7fcd59bf4640 1 -- 172.21.15.116:0/1278176673 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcd54150d60 msgr2=0x7fcd5414f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:28.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.526+0000 7fcd59bf4640 1 --2- 172.21.15.116:0/1278176673 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcd54150d60 0x7fcd5414f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:28.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.526+0000 7fcd59bf4640 1 -- 172.21.15.116:0/1278176673 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd54136820 con 0x7fcd54157510 2024-04-25T00:31:28.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.526+0000 7fcd593f3640 1 --2- 172.21.15.116:0/1278176673 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcd540a9630 0x7fcd5414cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:31:28.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.526+0000 7fcd59bf4640 1 --2- 172.21.15.116:0/1278176673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd54157510 0x7fcd541427d0 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7fcd5001a260 tx=0x7fcd500194e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:28.530 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.526+0000 7fcd49ffb640 1 -- 172.21.15.116:0/1278176673 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd500b1050 con 0x7fcd54157510 2024-04-25T00:31:28.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.526+0000 7fcd49ffb640 1 -- 172.21.15.116:0/1278176673 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd500bd070 con 0x7fcd54157510 2024-04-25T00:31:28.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.526+0000 7fcd49ffb640 1 -- 172.21.15.116:0/1278176673 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd500ac940 con 0x7fcd54157510 2024-04-25T00:31:28.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.526+0000 7fcd5b657640 1 -- 172.21.15.116:0/1278176673 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd541ecf10 con 0x7fcd54157510 2024-04-25T00:31:28.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.526+0000 7fcd5b657640 1 -- 172.21.15.116:0/1278176673 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd541ed4d0 con 0x7fcd54157510 2024-04-25T00:31:28.531 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.530+0000 7fcd5b657640 1 -- 172.21.15.116:0/1278176673 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd5404f510 con 0x7fcd54157510 2024-04-25T00:31:28.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.530+0000 7fcd49ffb640 1 -- 172.21.15.116:0/1278176673 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fcd500acae0 con 0x7fcd54157510 2024-04-25T00:31:28.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.530+0000 7fcd49ffb640 1 --2- 172.21.15.116:0/1278176673 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcd380678c0 0x7fcd38069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:28.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.530+0000 7fcd49ffb640 1 -- 172.21.15.116:0/1278176673 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7fcd50121320 con 0x7fcd54157510 2024-04-25T00:31:28.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.530+0000 7fcd593f3640 1 --2- 172.21.15.116:0/1278176673 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcd380678c0 0x7fcd38069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:28.532 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.530+0000 7fcd593f3640 1 --2- 172.21.15.116:0/1278176673 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcd380678c0 0x7fcd38069d80 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fcd4c0016f0 tx=0x7fcd4c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:28.534 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.534+0000 7fcd49ffb640 1 -- 172.21.15.116:0/1278176673 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fcd50016480 con 0x7fcd54157510 2024-04-25T00:31:28.695 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.694+0000 7fcd5b657640 1 -- 172.21.15.116:0/1278176673 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7fcd54071db0 con 0x7fcd54157510 2024-04-25T00:31:28.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.698+0000 7fcd49ffb640 1 -- 172.21.15.116:0/1278176673 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v10)=0 v10) ==== 123+0+0 (secure 0 0 0) 0x7fcd500f0b70 con 0x7fcd54157510 2024-04-25T00:31:28.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.698+0000 7fcd49ffb640 1 -- 172.21.15.116:0/1278176673 <== mon.0 v2:172.21.15.116:3300/0 8 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fcd500b2040 con 0x7fcd54157510 2024-04-25T00:31:28.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.702+0000 7fcd5b657640 1 -- 172.21.15.116:0/1278176673 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcd380678c0 msgr2=0x7fcd38069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:28.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.702+0000 7fcd5b657640 1 --2- 172.21.15.116:0/1278176673 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcd380678c0 0x7fcd38069d80 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fcd4c0016f0 tx=0x7fcd4c009040 comp rx=0 tx=0).stop 2024-04-25T00:31:28.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.702+0000 7fcd5b657640 1 -- 172.21.15.116:0/1278176673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd54157510 msgr2=0x7fcd541427d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:28.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.702+0000 7fcd5b657640 1 --2- 172.21.15.116:0/1278176673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd54157510 0x7fcd541427d0 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7fcd5001a260 tx=0x7fcd500194e0 comp rx=0 tx=0).stop 2024-04-25T00:31:28.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.702+0000 7fcd5b657640 1 -- 172.21.15.116:0/1278176673 shutdown_connections 2024-04-25T00:31:28.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.702+0000 7fcd5b657640 1 --2- 172.21.15.116:0/1278176673 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fcd380678c0 0x7fcd38069d80 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:28.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.702+0000 7fcd5b657640 1 --2- 172.21.15.116:0/1278176673 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fcd54157510 0x7fcd541427d0 unknown :-1 s=CLOSED pgs=1867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:28.704 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.702+0000 7fcd5b657640 1 --2- 172.21.15.116:0/1278176673 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fcd54150d60 0x7fcd5414f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:28.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.702+0000 7fcd5b657640 1 --2- 172.21.15.116:0/1278176673 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fcd540a9630 0x7fcd5414cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:28.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.702+0000 7fcd5b657640 1 -- 172.21.15.116:0/1278176673 >> 172.21.15.116:0/1278176673 conn(0x7fcd540c3a20 msgr2=0x7fcd540c2090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:31:28.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.702+0000 7fcd5b657640 1 -- 172.21.15.116:0/1278176673 shutdown_connections 2024-04-25T00:31:28.705 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.702+0000 7fcd5b657640 1 -- 172.21.15.116:0/1278176673 wait complete. 2024-04-25T00:31:28.720 INFO:tasks.cephfs.filesystem:Destroying file system a and related pools 2024-04-25T00:31:28.720 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:31:28.840 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.834+0000 7fe833320640 1 Processor -- start 2024-04-25T00:31:28.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.834+0000 7fe833320640 1 -- start start 2024-04-25T00:31:28.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.838+0000 7fe833320640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe82c0a94a0 0x7fe82c1563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:28.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.838+0000 7fe833320640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fe82c071680 0x7fe82c071ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:28.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.838+0000 7fe833320640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe82c071220 con 0x7fe82c14c720 2024-04-25T00:31:28.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.838+0000 7fe833320640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fe82c153460 con 0x7fe82c071680 2024-04-25T00:31:28.841 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.838+0000 7fe833320640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe82c152420 con 0x7fe82c0a94a0 2024-04-25T00:31:28.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.838+0000 7fe8308bb640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe82c0a94a0 0x7fe82c1563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:28.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.838+0000 7fe8318bd640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fe82c071680 0x7fe82c071ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:28.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.838+0000 7fe8308bb640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe82c0a94a0 0x7fe82c1563d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.174:3301/0 says I am v2:172.21.15.116:36344/0 (socket says 172.21.15.116:36344) 2024-04-25T00:31:28.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.838+0000 7fe8308bb640 1 -- 172.21.15.116:0/2559977165 learned_addr learned my addr 172.21.15.116:0/2559977165 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:31:28.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.838+0000 7fe8308bb640 1 -- 172.21.15.116:0/2559977165 >> v2:172.21.15.174:3300/0 conn(0x7fe82c071680 msgr2=0x7fe82c071ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:31:28.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.838+0000 7fe8308bb640 1 --2- 172.21.15.116:0/2559977165 >> v2:172.21.15.174:3300/0 conn(0x7fe82c071680 0x7fe82c071ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:28.842 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.838+0000 7fe8308bb640 1 -- 172.21.15.116:0/2559977165 >> v1:172.21.15.174:6789/0 conn(0x7fe82c14c720 legacy=0x7fe82c0c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:28.843 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.838+0000 7fe8308bb640 1 -- 172.21.15.116:0/2559977165 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe82c14cb00 con 0x7fe82c0a94a0 2024-04-25T00:31:28.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.842+0000 7fe8308bb640 1 --2- 172.21.15.116:0/2559977165 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe82c0a94a0 0x7fe82c1563d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe81c009870 tx=0x7fe81c01c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8a031f7f7e3b7f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:28.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.842+0000 7fe823fff640 1 -- 172.21.15.116:0/2559977165 <== mon.2 v2:172.21.15.174:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe81c0aa600 con 0x7fe82c0a94a0 2024-04-25T00:31:28.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.842+0000 7fe823fff640 1 -- 172.21.15.116:0/2559977165 <== mon.2 v2:172.21.15.174:3301/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fe81c0af070 con 0x7fe82c0a94a0 2024-04-25T00:31:28.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.842+0000 7fe823fff640 1 -- 172.21.15.116:0/2559977165 <== mon.2 v2:172.21.15.174:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe81c0aa9e0 con 0x7fe82c0a94a0 2024-04-25T00:31:28.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.842+0000 7fe833320640 1 -- 172.21.15.116:0/2559977165 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe82c0a94a0 msgr2=0x7fe82c1563d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:28.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.842+0000 7fe833320640 1 --2- 172.21.15.116:0/2559977165 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe82c0a94a0 0x7fe82c1563d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe81c009870 tx=0x7fe81c01c4a0 comp rx=0 tx=0).stop 2024-04-25T00:31:28.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.842+0000 7fe833320640 1 -- 172.21.15.116:0/2559977165 shutdown_connections 2024-04-25T00:31:28.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.842+0000 7fe833320640 1 --2- 172.21.15.116:0/2559977165 >> v2:172.21.15.174:3300/0 conn(0x7fe82c071680 0x7fe82c071ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:28.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.842+0000 7fe833320640 1 --2- 172.21.15.116:0/2559977165 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe82c0a94a0 0x7fe82c1563d0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:28.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.842+0000 7fe833320640 1 -- 172.21.15.116:0/2559977165 >> 172.21.15.116:0/2559977165 conn(0x7fe82c0c3b00 msgr2=0x7fe82c0c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:31:28.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.842+0000 7fe833320640 1 -- 172.21.15.116:0/2559977165 shutdown_connections 2024-04-25T00:31:28.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.842+0000 7fe833320640 1 -- 172.21.15.116:0/2559977165 wait complete. 2024-04-25T00:31:28.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.842+0000 7fe833320640 1 Processor -- start 2024-04-25T00:31:28.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.842+0000 7fe833320640 1 -- start start 2024-04-25T00:31:28.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.842+0000 7fe833320640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe82c071680 0x7fe82c144dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:28.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.842+0000 7fe833320640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe82c0a94a0 0x7fe82c13f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:28.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.842+0000 7fe833320640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe82c14c720 0x7fe82c13fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:28.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.842+0000 7fe833320640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe82c0c3120 con 0x7fe82c0a94a0 2024-04-25T00:31:28.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.842+0000 7fe833320640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe82c152f40 con 0x7fe82c071680 2024-04-25T00:31:28.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.842+0000 7fe833320640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe82c0c32a0 con 0x7fe82c14c720 2024-04-25T00:31:28.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.842+0000 7fe8308bb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe82c0a94a0 0x7fe82c13f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:28.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.842+0000 7fe8308bb640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe82c0a94a0 0x7fe82c13f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51096/0 (socket says 172.21.15.116:51096) 2024-04-25T00:31:28.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.842+0000 7fe8308bb640 1 -- 172.21.15.116:0/2802330383 learned_addr learned my addr 172.21.15.116:0/2802330383 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:31:28.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.842+0000 7fe8310bc640 1 --2- 172.21.15.116:0/2802330383 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe82c071680 0x7fe82c144dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:28.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.842+0000 7fe8318bd640 1 --2- 172.21.15.116:0/2802330383 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe82c14c720 0x7fe82c13fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:28.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.846+0000 7fe8308bb640 1 -- 172.21.15.116:0/2802330383 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe82c14c720 msgr2=0x7fe82c13fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:28.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.846+0000 7fe8308bb640 1 --2- 172.21.15.116:0/2802330383 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe82c14c720 0x7fe82c13fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:28.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.846+0000 7fe8308bb640 1 -- 172.21.15.116:0/2802330383 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe82c071680 msgr2=0x7fe82c144dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:28.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.846+0000 7fe8308bb640 1 --2- 172.21.15.116:0/2802330383 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe82c071680 0x7fe82c144dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:28.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.846+0000 7fe8308bb640 1 -- 172.21.15.116:0/2802330383 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe82c14b890 con 0x7fe82c0a94a0 2024-04-25T00:31:28.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.846+0000 7fe8308bb640 1 --2- 172.21.15.116:0/2802330383 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe82c0a94a0 0x7fe82c13f850 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7fe81c0ac040 tx=0x7fe81c01ace0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:28.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.846+0000 7fe821ffb640 1 -- 172.21.15.116:0/2802330383 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe81c0aa8d0 con 0x7fe82c0a94a0 2024-04-25T00:31:28.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.846+0000 7fe821ffb640 1 -- 172.21.15.116:0/2802330383 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fe81c0aaa90 con 0x7fe82c0a94a0 2024-04-25T00:31:28.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.846+0000 7fe833320640 1 -- 172.21.15.116:0/2802330383 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe82c1ee0f0 con 0x7fe82c0a94a0 2024-04-25T00:31:28.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.846+0000 7fe821ffb640 1 -- 172.21.15.116:0/2802330383 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe81c0bd3f0 con 0x7fe82c0a94a0 2024-04-25T00:31:28.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.846+0000 7fe833320640 1 -- 172.21.15.116:0/2802330383 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe82c1ee6b0 con 0x7fe82c0a94a0 2024-04-25T00:31:28.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.846+0000 7fe833320640 1 -- 172.21.15.116:0/2802330383 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe82c068350 con 0x7fe82c0a94a0 2024-04-25T00:31:28.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.846+0000 7fe821ffb640 1 -- 172.21.15.116:0/2802330383 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe81c0c4060 con 0x7fe82c0a94a0 2024-04-25T00:31:28.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.846+0000 7fe821ffb640 1 --2- 172.21.15.116:0/2802330383 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe8100678c0 0x7fe810069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:28.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.846+0000 7fe821ffb640 1 -- 172.21.15.116:0/2802330383 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7fe81c1235a0 con 0x7fe82c0a94a0 2024-04-25T00:31:28.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.846+0000 7fe8310bc640 1 --2- 172.21.15.116:0/2802330383 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe8100678c0 0x7fe810069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:28.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.850+0000 7fe8310bc640 1 --2- 172.21.15.116:0/2802330383 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe8100678c0 0x7fe810069d80 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fe824001e00 tx=0x7fe824009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:28.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:28.850+0000 7fe821ffb640 1 -- 172.21.15.116:0/2802330383 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fe81c0b8070 con 0x7fe82c0a94a0 2024-04-25T00:31:29.032 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.030+0000 7fe833320640 1 -- 172.21.15.116:0/2802330383 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe82c0c3640 con 0x7fe82c0a94a0 2024-04-25T00:31:29.034 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.034+0000 7fe821ffb640 1 -- 172.21.15.116:0/2802330383 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 242 v242) ==== 95+0+6506 (secure 0 0 0) 0x7fe81c016150 con 0x7fe82c0a94a0 2024-04-25T00:31:29.034 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:31:29.034 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":242,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":9027,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/426062493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":426062493},{"type":"v1","addr":"172.21.15.116:6835","nonce":426062493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":242}],"filesystems":[{"mdsmap":{"epoch":236,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:29:54.168353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8847},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8847":{"gid":8847,"name":"c","rank":0,"incarnation":233,"state":"up:active","state_seq":17,"addr":"172.21.15.116:6833/2662028892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2662028892},{"type":"v1","addr":"172.21.15.116:6833","nonce":2662028892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8847,"qdb_cluster":[8847]},"id":1},{"mdsmap":{"epoch":241,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:30:19.111393+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6764},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6764":{"gid":6764,"name":"d","rank":0,"incarnation":238,"state":"up:active","state_seq":8,"addr":"172.21.15.174:6837/3873739116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3873739116},{"type":"v1","addr":"172.21.15.174:6837","nonce":3873739116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6764,"qdb_cluster":[6764]},"id":2}]} 2024-04-25T00:31:29.035 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 242 2024-04-25T00:31:29.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.034+0000 7fe833320640 1 -- 172.21.15.116:0/2802330383 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe8100678c0 msgr2=0x7fe810069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:29.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.034+0000 7fe833320640 1 --2- 172.21.15.116:0/2802330383 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe8100678c0 0x7fe810069d80 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fe824001e00 tx=0x7fe824009040 comp rx=0 tx=0).stop 2024-04-25T00:31:29.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.034+0000 7fe833320640 1 -- 172.21.15.116:0/2802330383 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe82c0a94a0 msgr2=0x7fe82c13f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:29.036 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.034+0000 7fe833320640 1 --2- 172.21.15.116:0/2802330383 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe82c0a94a0 0x7fe82c13f850 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7fe81c0ac040 tx=0x7fe81c01ace0 comp rx=0 tx=0).stop 2024-04-25T00:31:29.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.034+0000 7fe833320640 1 -- 172.21.15.116:0/2802330383 shutdown_connections 2024-04-25T00:31:29.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.034+0000 7fe833320640 1 --2- 172.21.15.116:0/2802330383 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe8100678c0 0x7fe810069d80 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:29.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.034+0000 7fe833320640 1 --2- 172.21.15.116:0/2802330383 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe82c14c720 0x7fe82c13fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:29.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.034+0000 7fe833320640 1 --2- 172.21.15.116:0/2802330383 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe82c0a94a0 0x7fe82c13f850 unknown :-1 s=CLOSED pgs=1868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:29.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.034+0000 7fe833320640 1 --2- 172.21.15.116:0/2802330383 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe82c071680 0x7fe82c144dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:29.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.034+0000 7fe833320640 1 -- 172.21.15.116:0/2802330383 >> 172.21.15.116:0/2802330383 conn(0x7fe82c0c3b00 msgr2=0x7fe82c151c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:31:29.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.034+0000 7fe833320640 1 -- 172.21.15.116:0/2802330383 shutdown_connections 2024-04-25T00:31:29.037 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.034+0000 7fe833320640 1 -- 172.21.15.116:0/2802330383 wait complete. 2024-04-25T00:31:29.053 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:31:29.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.170+0000 7faf43abf640 1 Processor -- start 2024-04-25T00:31:29.173 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.170+0000 7faf43abf640 1 -- start start 2024-04-25T00:31:29.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.174+0000 7faf43abf640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faf3c150d20 0x7faf3c0c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:29.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.174+0000 7faf43abf640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7faf3c0c2f50 0x7faf3c1538e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:29.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.174+0000 7faf43abf640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf3c0c3910 con 0x7faf3c0a94a0 2024-04-25T00:31:29.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.174+0000 7faf43abf640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7faf3c154530 con 0x7faf3c0c2f50 2024-04-25T00:31:29.185 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.174+0000 7faf43abf640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7faf3c0c16e0 con 0x7faf3c150d20 2024-04-25T00:31:29.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.174+0000 7faf4205c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7faf3c0a94a0 0x7faf3c153e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:46830/0 (socket says 172.21.15.116:46830) 2024-04-25T00:31:29.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.174+0000 7faf4205c640 1 -- 172.21.15.116:0/193577293 learned_addr learned my addr 172.21.15.116:0/193577293 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:31:29.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.174+0000 7faf4105a640 1 --2- 172.21.15.116:0/193577293 >> v2:172.21.15.174:3300/0 conn(0x7faf3c0c2f50 0x7faf3c1538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:29.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.174+0000 7faf4185b640 1 --2- 172.21.15.116:0/193577293 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faf3c150d20 0x7faf3c0c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:29.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.174+0000 7faf40859640 1 -- 172.21.15.116:0/193577293 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2876027849 0 0) 0x7faf3c0c3910 con 0x7faf3c0a94a0 2024-04-25T00:31:29.186 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.174+0000 7faf40859640 1 -- 172.21.15.116:0/193577293 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faf28003530 con 0x7faf3c0a94a0 2024-04-25T00:31:29.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.174+0000 7faf40859640 1 -- 172.21.15.116:0/193577293 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3668447548 0 0) 0x7faf28003530 con 0x7faf3c0a94a0 2024-04-25T00:31:29.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.174+0000 7faf40859640 1 -- 172.21.15.116:0/193577293 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faf3c150d20 msgr2=0x7faf3c0c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:31:29.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.174+0000 7faf40859640 1 --2- 172.21.15.116:0/193577293 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faf3c150d20 0x7faf3c0c2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:29.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.174+0000 7faf40859640 1 -- 172.21.15.116:0/193577293 >> v2:172.21.15.174:3300/0 conn(0x7faf3c0c2f50 msgr2=0x7faf3c1538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:31:29.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.174+0000 7faf40859640 1 --2- 172.21.15.116:0/193577293 >> v2:172.21.15.174:3300/0 conn(0x7faf3c0c2f50 0x7faf3c1538e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:29.187 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.174+0000 7faf40859640 1 -- 172.21.15.116:0/193577293 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf3c0c1400 con 0x7faf3c0a94a0 2024-04-25T00:31:29.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.174+0000 7faf40859640 1 -- 172.21.15.116:0/193577293 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7faf34001a00 con 0x7faf3c0a94a0 2024-04-25T00:31:29.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.174+0000 7faf40859640 1 -- 172.21.15.116:0/193577293 >> v1:172.21.15.116:6789/0 conn(0x7faf3c0a94a0 legacy=0x7faf3c153e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:29.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.174+0000 7faf40859640 1 --2- 172.21.15.116:0/193577293 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faf3c150d20 0x7faf280039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:29.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.174+0000 7faf40859640 1 --2- 172.21.15.116:0/193577293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf28003f40 0x7faf28004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:29.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.174+0000 7faf40859640 1 --2- 172.21.15.116:0/193577293 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf28004d50 0x7faf280051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:29.188 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.174+0000 7faf40859640 1 -- 172.21.15.116:0/193577293 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7faf3c154530 con 0x7faf28003f40 2024-04-25T00:31:29.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.174+0000 7faf40859640 1 -- 172.21.15.116:0/193577293 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7faf3c0c16e0 con 0x7faf28004d50 2024-04-25T00:31:29.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.174+0000 7faf40859640 1 -- 172.21.15.116:0/193577293 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7faf2800b9d0 con 0x7faf3c150d20 2024-04-25T00:31:29.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.174+0000 7faf4105a640 1 --2- 172.21.15.116:0/193577293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf28003f40 0x7faf28004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:29.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.174+0000 7faf4185b640 1 --2- 172.21.15.116:0/193577293 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faf3c150d20 0x7faf280039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:29.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.174+0000 7faf4205c640 1 --2- 172.21.15.116:0/193577293 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf28004d50 0x7faf280051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:29.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.174+0000 7faf4105a640 1 -- 172.21.15.116:0/193577293 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faf3c150d20 msgr2=0x7faf280039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:31:29.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.174+0000 7faf4105a640 1 --2- 172.21.15.116:0/193577293 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faf3c150d20 0x7faf280039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:29.189 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.174+0000 7faf4105a640 1 -- 172.21.15.116:0/193577293 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf28004d50 msgr2=0x7faf280051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:31:29.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.174+0000 7faf4105a640 1 --2- 172.21.15.116:0/193577293 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf28004d50 0x7faf280051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:29.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.174+0000 7faf4105a640 1 -- 172.21.15.116:0/193577293 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf28003530 con 0x7faf28003f40 2024-04-25T00:31:29.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.174+0000 7faf4105a640 1 --2- 172.21.15.116:0/193577293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf28003f40 0x7faf28004810 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7faf38009ae0 tx=0x7faf3801ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18ec81f4bade4b0a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:29.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.174+0000 7faf40859640 1 -- 172.21.15.116:0/193577293 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf380b2070 con 0x7faf28003f40 2024-04-25T00:31:29.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.174+0000 7faf40859640 1 -- 172.21.15.116:0/193577293 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7faf380a5590 con 0x7faf28003f40 2024-04-25T00:31:29.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.174+0000 7faf40859640 1 -- 172.21.15.116:0/193577293 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf380096b0 con 0x7faf28003f40 2024-04-25T00:31:29.190 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.178+0000 7faf43abf640 1 -- 172.21.15.116:0/193577293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf28003f40 msgr2=0x7faf28004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:29.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.178+0000 7faf43abf640 1 --2- 172.21.15.116:0/193577293 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf28003f40 0x7faf28004810 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7faf38009ae0 tx=0x7faf3801ca50 comp rx=0 tx=0).stop 2024-04-25T00:31:29.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.178+0000 7faf4205c640 1 -- 172.21.15.116:0/193577293 reap_dead start 2024-04-25T00:31:29.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.178+0000 7faf43abf640 1 -- 172.21.15.116:0/193577293 shutdown_connections 2024-04-25T00:31:29.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.178+0000 7faf43abf640 1 -- 172.21.15.116:0/193577293 >> 172.21.15.116:0/193577293 conn(0x7faf3c0c3b00 msgr2=0x7faf3c0c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:31:29.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.178+0000 7faf43abf640 1 -- 172.21.15.116:0/193577293 shutdown_connections 2024-04-25T00:31:29.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.178+0000 7faf43abf640 1 -- 172.21.15.116:0/193577293 wait complete. 2024-04-25T00:31:29.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.178+0000 7faf43abf640 1 Processor -- start 2024-04-25T00:31:29.191 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.178+0000 7faf43abf640 1 -- start start 2024-04-25T00:31:29.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.178+0000 7faf43abf640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf28003f40 0x7faf3c155d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:29.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.178+0000 7faf43abf640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faf3c1442a0 0x7faf3c0c5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:29.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.178+0000 7faf43abf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf3c1eca80 0x7faf3c1ecf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:29.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.178+0000 7faf43abf640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7faf3c06ec20 con 0x7faf3c1eca80 2024-04-25T00:31:29.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.178+0000 7faf43abf640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7faf3c06e940 con 0x7faf28003f40 2024-04-25T00:31:29.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.178+0000 7faf43abf640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7faf3c0a99f0 con 0x7faf3c1442a0 2024-04-25T00:31:29.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.178+0000 7faf4205c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf3c1eca80 0x7faf3c1ecf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:29.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.178+0000 7faf4205c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf3c1eca80 0x7faf3c1ecf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51120/0 (socket says 172.21.15.116:51120) 2024-04-25T00:31:29.192 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.178+0000 7faf4205c640 1 -- 172.21.15.116:0/99770920 learned_addr learned my addr 172.21.15.116:0/99770920 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:31:29.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.178+0000 7faf4185b640 1 --2- 172.21.15.116:0/99770920 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf28003f40 0x7faf3c155d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:29.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.178+0000 7faf4105a640 1 --2- 172.21.15.116:0/99770920 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faf3c1442a0 0x7faf3c0c5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:29.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.178+0000 7faf4205c640 1 -- 172.21.15.116:0/99770920 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faf3c1442a0 msgr2=0x7faf3c0c5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:29.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.178+0000 7faf4205c640 1 --2- 172.21.15.116:0/99770920 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faf3c1442a0 0x7faf3c0c5ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:29.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.178+0000 7faf4205c640 1 -- 172.21.15.116:0/99770920 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf28003f40 msgr2=0x7faf3c155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:29.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.178+0000 7faf4205c640 1 --2- 172.21.15.116:0/99770920 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf28003f40 0x7faf3c155d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:29.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.178+0000 7faf4205c640 1 -- 172.21.15.116:0/99770920 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf3c154bd0 con 0x7faf3c1eca80 2024-04-25T00:31:29.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.178+0000 7faf4105a640 1 --2- 172.21.15.116:0/99770920 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faf3c1442a0 0x7faf3c0c5ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:31:29.193 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.178+0000 7faf4205c640 1 --2- 172.21.15.116:0/99770920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf3c1eca80 0x7faf3c1ecf00 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7faf34000f90 tx=0x7faf34011c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:29.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.178+0000 7faf327fc640 1 -- 172.21.15.116:0/99770920 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf34010040 con 0x7faf3c1eca80 2024-04-25T00:31:29.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.178+0000 7faf327fc640 1 -- 172.21.15.116:0/99770920 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7faf3401a070 con 0x7faf3c1eca80 2024-04-25T00:31:29.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.178+0000 7faf43abf640 1 -- 172.21.15.116:0/99770920 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf3c154eb0 con 0x7faf3c1eca80 2024-04-25T00:31:29.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.178+0000 7faf327fc640 1 -- 172.21.15.116:0/99770920 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf3400fb40 con 0x7faf3c1eca80 2024-04-25T00:31:29.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.178+0000 7faf43abf640 1 -- 172.21.15.116:0/99770920 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf3c0c56a0 con 0x7faf3c1eca80 2024-04-25T00:31:29.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.178+0000 7faf43abf640 1 -- 172.21.15.116:0/99770920 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf3c068350 con 0x7faf3c1eca80 2024-04-25T00:31:29.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.182+0000 7faf327fc640 1 -- 172.21.15.116:0/99770920 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7faf3400fce0 con 0x7faf3c1eca80 2024-04-25T00:31:29.194 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.182+0000 7faf327fc640 1 --2- 172.21.15.116:0/99770920 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faf200678f0 0x7faf20069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:29.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.182+0000 7faf327fc640 1 -- 172.21.15.116:0/99770920 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7faf34089ff0 con 0x7faf3c1eca80 2024-04-25T00:31:29.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.182+0000 7faf4185b640 1 --2- 172.21.15.116:0/99770920 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faf200678f0 0x7faf20069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:29.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.182+0000 7faf4185b640 1 --2- 172.21.15.116:0/99770920 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faf200678f0 0x7faf20069db0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7faf2c009950 tx=0x7faf2c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:29.195 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.186+0000 7faf327fc640 1 -- 172.21.15.116:0/99770920 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7faf340597b0 con 0x7faf3c1eca80 2024-04-25T00:31:29.371 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.370+0000 7faf43abf640 1 -- 172.21.15.116:0/99770920 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faf3c06e200 con 0x7faf3c1eca80 2024-04-25T00:31:29.372 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.370+0000 7faf327fc640 1 -- 172.21.15.116:0/99770920 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 242 v242) ==== 95+0+6506 (secure 0 0 0) 0x7faf340590b0 con 0x7faf3c1eca80 2024-04-25T00:31:29.372 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:31:29.372 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":242,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":9027,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/426062493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":426062493},{"type":"v1","addr":"172.21.15.116:6835","nonce":426062493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":242}],"filesystems":[{"mdsmap":{"epoch":236,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:29:54.168353+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":8847},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8847":{"gid":8847,"name":"c","rank":0,"incarnation":233,"state":"up:active","state_seq":17,"addr":"172.21.15.116:6833/2662028892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6832","nonce":2662028892},{"type":"v1","addr":"172.21.15.116:6833","nonce":2662028892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":8847,"qdb_cluster":[8847]},"id":1},{"mdsmap":{"epoch":241,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:30:19.111393+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6764},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6764":{"gid":6764,"name":"d","rank":0,"incarnation":238,"state":"up:active","state_seq":8,"addr":"172.21.15.174:6837/3873739116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3873739116},{"type":"v1","addr":"172.21.15.174:6837","nonce":3873739116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6764,"qdb_cluster":[6764]},"id":2}]} 2024-04-25T00:31:29.373 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 242 2024-04-25T00:31:29.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.374+0000 7faf43abf640 1 -- 172.21.15.116:0/99770920 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faf200678f0 msgr2=0x7faf20069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:29.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.374+0000 7faf43abf640 1 --2- 172.21.15.116:0/99770920 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faf200678f0 0x7faf20069db0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7faf2c009950 tx=0x7faf2c008040 comp rx=0 tx=0).stop 2024-04-25T00:31:29.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.374+0000 7faf43abf640 1 -- 172.21.15.116:0/99770920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf3c1eca80 msgr2=0x7faf3c1ecf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:29.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.374+0000 7faf43abf640 1 --2- 172.21.15.116:0/99770920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf3c1eca80 0x7faf3c1ecf00 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7faf34000f90 tx=0x7faf34011c20 comp rx=0 tx=0).stop 2024-04-25T00:31:29.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.374+0000 7faf43abf640 1 -- 172.21.15.116:0/99770920 shutdown_connections 2024-04-25T00:31:29.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.374+0000 7faf43abf640 1 --2- 172.21.15.116:0/99770920 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7faf200678f0 0x7faf20069db0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:29.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.374+0000 7faf43abf640 1 --2- 172.21.15.116:0/99770920 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7faf3c1eca80 0x7faf3c1ecf00 unknown :-1 s=CLOSED pgs=1871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:29.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.374+0000 7faf43abf640 1 --2- 172.21.15.116:0/99770920 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7faf3c1442a0 0x7faf3c0c5ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:29.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.374+0000 7faf43abf640 1 --2- 172.21.15.116:0/99770920 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7faf28003f40 0x7faf3c155d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:29.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.374+0000 7faf43abf640 1 -- 172.21.15.116:0/99770920 >> 172.21.15.116:0/99770920 conn(0x7faf3c0c3b00 msgr2=0x7faf3c0ac8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:31:29.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.374+0000 7faf43abf640 1 -- 172.21.15.116:0/99770920 shutdown_connections 2024-04-25T00:31:29.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.374+0000 7faf43abf640 1 -- 172.21.15.116:0/99770920 wait complete. 2024-04-25T00:31:29.390 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T00:31:29.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.506+0000 7fe5b3a42640 1 Processor -- start 2024-04-25T00:31:29.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.506+0000 7fe5b3a42640 1 -- start start 2024-04-25T00:31:29.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.506+0000 7fe5b3a42640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe5ac06fcd0 0x7fe5ac0c1540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:29.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.506+0000 7fe5b3a42640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5ac06bd50 con 0x7fe5ac0730a0 2024-04-25T00:31:29.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.506+0000 7fe5b3a42640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5ac06f990 con 0x7fe5ac0c40e0 2024-04-25T00:31:29.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.506+0000 7fe5b3a42640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe5ac156780 con 0x7fe5ac06fcd0 2024-04-25T00:31:29.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.506+0000 7fe5b0fdd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fe5ac0730a0 0x7fe5ac1525a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:46832/0 (socket says 172.21.15.116:46832) 2024-04-25T00:31:29.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.506+0000 7fe5b0fdd640 1 -- 172.21.15.116:0/267559959 learned_addr learned my addr 172.21.15.116:0/267559959 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:31:29.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.506+0000 7fe5b1fdf640 1 --2- 172.21.15.116:0/267559959 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe5ac06fcd0 0x7fe5ac0c1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:29.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.506+0000 7fe5a3fff640 1 -- 172.21.15.116:0/267559959 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1157079386 0 0) 0x7fe5ac06bd50 con 0x7fe5ac0730a0 2024-04-25T00:31:29.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.506+0000 7fe5a3fff640 1 -- 172.21.15.116:0/267559959 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe598003530 con 0x7fe5ac0730a0 2024-04-25T00:31:29.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.506+0000 7fe5a3fff640 1 -- 172.21.15.116:0/267559959 <== mon.1 v1:172.21.15.174:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2413846273 0 0) 0x7fe5ac06f990 con 0x7fe5ac0c40e0 2024-04-25T00:31:29.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.506+0000 7fe5a3fff640 1 -- 172.21.15.116:0/267559959 --> v1:172.21.15.174:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe5ac06bd50 con 0x7fe5ac0c40e0 2024-04-25T00:31:29.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.506+0000 7fe5a3fff640 1 -- 172.21.15.116:0/267559959 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2766253028 0 0) 0x7fe598003530 con 0x7fe5ac0730a0 2024-04-25T00:31:29.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.506+0000 7fe5a3fff640 1 -- 172.21.15.116:0/267559959 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe5ac06fcd0 msgr2=0x7fe5ac0c1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:31:29.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.506+0000 7fe5a3fff640 1 --2- 172.21.15.116:0/267559959 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe5ac06fcd0 0x7fe5ac0c1540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:29.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.506+0000 7fe5a3fff640 1 -- 172.21.15.116:0/267559959 >> v1:172.21.15.174:6789/0 conn(0x7fe5ac0c40e0 legacy=0x7fe5ac156070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:29.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.506+0000 7fe5a3fff640 1 -- 172.21.15.116:0/267559959 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5ac155b50 con 0x7fe5ac0730a0 2024-04-25T00:31:29.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.506+0000 7fe5a3fff640 1 -- 172.21.15.116:0/267559959 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7fe59c001680 con 0x7fe5ac0730a0 2024-04-25T00:31:29.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.506+0000 7fe5a3fff640 1 -- 172.21.15.116:0/267559959 >> v1:172.21.15.116:6789/0 conn(0x7fe5ac0730a0 legacy=0x7fe5ac1525a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:29.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.506+0000 7fe5a3fff640 1 --2- 172.21.15.116:0/267559959 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe598003b80 0x7fe598003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:29.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.506+0000 7fe5a3fff640 1 --2- 172.21.15.116:0/267559959 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe5ac06fcd0 0x7fe598004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:29.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.506+0000 7fe5a3fff640 1 --2- 172.21.15.116:0/267559959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe598004d50 0x7fe5980051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:29.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.506+0000 7fe5a3fff640 1 -- 172.21.15.116:0/267559959 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe5ac156780 con 0x7fe598004d50 2024-04-25T00:31:29.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.506+0000 7fe5a3fff640 1 -- 172.21.15.116:0/267559959 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe59800bb90 con 0x7fe598003b80 2024-04-25T00:31:29.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.506+0000 7fe5a3fff640 1 -- 172.21.15.116:0/267559959 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe59800bd90 con 0x7fe5ac06fcd0 2024-04-25T00:31:29.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.506+0000 7fe5b1fdf640 1 --2- 172.21.15.116:0/267559959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe598004d50 0x7fe5980051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:29.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.506+0000 7fe5b0fdd640 1 --2- 172.21.15.116:0/267559959 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe5ac06fcd0 0x7fe598004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:29.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.506+0000 7fe5b17de640 1 --2- 172.21.15.116:0/267559959 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe598003b80 0x7fe598003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:29.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.506+0000 7fe5b1fdf640 1 -- 172.21.15.116:0/267559959 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe5ac06fcd0 msgr2=0x7fe598004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:31:29.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.506+0000 7fe5b1fdf640 1 --2- 172.21.15.116:0/267559959 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe5ac06fcd0 0x7fe598004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:29.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.506+0000 7fe5b1fdf640 1 -- 172.21.15.116:0/267559959 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe598003b80 msgr2=0x7fe598003f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:31:29.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.506+0000 7fe5b1fdf640 1 --2- 172.21.15.116:0/267559959 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe598003b80 0x7fe598003f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:29.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.506+0000 7fe5b1fdf640 1 -- 172.21.15.116:0/267559959 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe598003530 con 0x7fe598004d50 2024-04-25T00:31:29.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.506+0000 7fe5b17de640 1 --2- 172.21.15.116:0/267559959 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe598003b80 0x7fe598003f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:31:29.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.510+0000 7fe5b1fdf640 1 --2- 172.21.15.116:0/267559959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe598004d50 0x7fe5980051d0 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7fe5a4009f70 tx=0x7fe5a401cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a51a4d3baf227c3e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:29.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.510+0000 7fe5a3fff640 1 -- 172.21.15.116:0/267559959 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5a40b2070 con 0x7fe598004d50 2024-04-25T00:31:29.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.510+0000 7fe5a3fff640 1 -- 172.21.15.116:0/267559959 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fe5a40a5ad0 con 0x7fe598004d50 2024-04-25T00:31:29.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.510+0000 7fe5a3fff640 1 -- 172.21.15.116:0/267559959 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5a4009bf0 con 0x7fe598004d50 2024-04-25T00:31:29.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.510+0000 7fe5b3a42640 1 -- 172.21.15.116:0/267559959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe598004d50 msgr2=0x7fe5980051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:29.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.510+0000 7fe5b3a42640 1 --2- 172.21.15.116:0/267559959 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe598004d50 0x7fe5980051d0 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7fe5a4009f70 tx=0x7fe5a401cc20 comp rx=0 tx=0).stop 2024-04-25T00:31:29.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.510+0000 7fe5b1fdf640 1 -- 172.21.15.116:0/267559959 reap_dead start 2024-04-25T00:31:29.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.510+0000 7fe5b3a42640 1 -- 172.21.15.116:0/267559959 shutdown_connections 2024-04-25T00:31:29.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.510+0000 7fe5b3a42640 1 -- 172.21.15.116:0/267559959 >> 172.21.15.116:0/267559959 conn(0x7fe5ac0c4c10 msgr2=0x7fe5ac0c5020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:31:29.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.510+0000 7fe5b3a42640 1 -- 172.21.15.116:0/267559959 shutdown_connections 2024-04-25T00:31:29.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.510+0000 7fe5b3a42640 1 -- 172.21.15.116:0/267559959 wait complete. 2024-04-25T00:31:29.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.510+0000 7fe5b3a42640 1 Processor -- start 2024-04-25T00:31:29.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.510+0000 7fe5b3a42640 1 -- start start 2024-04-25T00:31:29.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.510+0000 7fe5b3a42640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe598004d50 0x7fe5ac146ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:29.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.510+0000 7fe5b3a42640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5ac147560 0x7fe5ac146ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:29.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.510+0000 7fe5b3a42640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5ac0abe10 0x7fe5ac14abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:29.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.510+0000 7fe5b3a42640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fe5ac0ad0c0 con 0x7fe5ac0abe10 2024-04-25T00:31:29.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.510+0000 7fe5b3a42640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fe5ac0acf40 con 0x7fe5ac147560 2024-04-25T00:31:29.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.510+0000 7fe5b3a42640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fe5ac152cb0 con 0x7fe598004d50 2024-04-25T00:31:29.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.510+0000 7fe5b1fdf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5ac0abe10 0x7fe5ac14abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:29.515 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.510+0000 7fe5b1fdf640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5ac0abe10 0x7fe5ac14abd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51138/0 (socket says 172.21.15.116:51138) 2024-04-25T00:31:29.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.510+0000 7fe5b1fdf640 1 -- 172.21.15.116:0/2585174515 learned_addr learned my addr 172.21.15.116:0/2585174515 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:31:29.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.510+0000 7fe5b0fdd640 1 --2- 172.21.15.116:0/2585174515 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5ac147560 0x7fe5ac146ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:29.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.510+0000 7fe5b17de640 1 --2- 172.21.15.116:0/2585174515 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe598004d50 0x7fe5ac146ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:29.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.514+0000 7fe5b1fdf640 1 -- 172.21.15.116:0/2585174515 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe598004d50 msgr2=0x7fe5ac146ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:29.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.514+0000 7fe5b1fdf640 1 --2- 172.21.15.116:0/2585174515 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe598004d50 0x7fe5ac146ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:29.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.514+0000 7fe5b1fdf640 1 -- 172.21.15.116:0/2585174515 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5ac147560 msgr2=0x7fe5ac146ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:29.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.514+0000 7fe5b1fdf640 1 --2- 172.21.15.116:0/2585174515 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5ac147560 0x7fe5ac146ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:29.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.514+0000 7fe5b1fdf640 1 -- 172.21.15.116:0/2585174515 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5ac1ef4e0 con 0x7fe5ac0abe10 2024-04-25T00:31:29.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.514+0000 7fe5b17de640 1 --2- 172.21.15.116:0/2585174515 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe598004d50 0x7fe5ac146ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:31:29.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.514+0000 7fe5b1fdf640 1 --2- 172.21.15.116:0/2585174515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5ac0abe10 0x7fe5ac14abd0 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7fe5a4002240 tx=0x7fe5a4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:29.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.514+0000 7fe5a1ffb640 1 -- 172.21.15.116:0/2585174515 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5a40b2040 con 0x7fe5ac0abe10 2024-04-25T00:31:29.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.514+0000 7fe5a1ffb640 1 -- 172.21.15.116:0/2585174515 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fe5a40a58d0 con 0x7fe5ac0abe10 2024-04-25T00:31:29.516 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.514+0000 7fe5b3a42640 1 -- 172.21.15.116:0/2585174515 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5ac1ef7c0 con 0x7fe5ac0abe10 2024-04-25T00:31:29.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.514+0000 7fe5a1ffb640 1 -- 172.21.15.116:0/2585174515 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5a4009bf0 con 0x7fe5ac0abe10 2024-04-25T00:31:29.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.514+0000 7fe5b3a42640 1 -- 172.21.15.116:0/2585174515 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5ac0c6160 con 0x7fe5ac0abe10 2024-04-25T00:31:29.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.514+0000 7fe5b3a42640 1 -- 172.21.15.116:0/2585174515 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5ac068350 con 0x7fe5ac0abe10 2024-04-25T00:31:29.517 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.514+0000 7fe5a1ffb640 1 -- 172.21.15.116:0/2585174515 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fe5a40c0020 con 0x7fe5ac0abe10 2024-04-25T00:31:29.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.514+0000 7fe5a1ffb640 1 --2- 172.21.15.116:0/2585174515 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe5900678f0 0x7fe590069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:29.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.514+0000 7fe5a1ffb640 1 -- 172.21.15.116:0/2585174515 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7fe5a4121590 con 0x7fe5ac0abe10 2024-04-25T00:31:29.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.514+0000 7fe5b17de640 1 --2- 172.21.15.116:0/2585174515 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe5900678f0 0x7fe590069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:29.518 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.514+0000 7fe5b17de640 1 --2- 172.21.15.116:0/2585174515 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe5900678f0 0x7fe590069db0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fe5a8001740 tx=0x7fe5a8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:29.519 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.518+0000 7fe5a1ffb640 1 -- 172.21.15.116:0/2585174515 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fe5a40c7020 con 0x7fe5ac0abe10 2024-04-25T00:31:29.671 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.670+0000 7fe5b3a42640 1 -- 172.21.15.116:0/2585174515 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe5ac072e80 con 0x7fe5ac0abe10 2024-04-25T00:31:29.674 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.674+0000 7fe5a1ffb640 1 -- 172.21.15.116:0/2585174515 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v74) ==== 74+0+26655 (secure 0 0 0) 0x7fe5a4016930 con 0x7fe5ac0abe10 2024-04-25T00:31:29.675 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:31:29.675 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":74,"fsid":"6d6a1aa7-0478-4c83-8057-05917e81f046","created":"2024-04-25T00:00:40.682493+0000","modified":"2024-04-25T00:30:15.781714+0000","last_up_change":"2024-04-25T00:00:51.822174+0000","last_in_change":"2024-04-25T00:00:42.980473+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T00:00:53.324376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"a_metadata","create_time":"2024-04-25T00:01:03.197326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"a_data","create_time":"2024-04-25T00:01:03.728618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"a"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-04-25T00:01:17.125210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-04-25T00:01:17.821873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8b6ddcb2-1ad9-4167-961c-bd6936bb9060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6809","nonce":3751765682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6811","nonce":3751765682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6815","nonce":3751765682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6813","nonce":3751765682}]},"public_addr":"172.21.15.116:6809/3751765682","cluster_addr":"172.21.15.116:6811/3751765682","heartbeat_back_addr":"172.21.15.116:6815/3751765682","heartbeat_front_addr":"172.21.15.116:6813/3751765682","state":["exists","up"]},{"osd":1,"uuid":"a6431768-994f-41da-8853-886f5687fb91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6801","nonce":377879124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6803","nonce":377879124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6807","nonce":377879124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6805","nonce":377879124}]},"public_addr":"172.21.15.116:6801/377879124","cluster_addr":"172.21.15.116:6803/377879124","heartbeat_back_addr":"172.21.15.116:6807/377879124","heartbeat_front_addr":"172.21.15.116:6805/377879124","state":["exists","up"]},{"osd":2,"uuid":"4f69bd40-5a23-45ff-8693-e1de75e1aa9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6817","nonce":1099282267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6819","nonce":1099282267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6823","nonce":1099282267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6821","nonce":1099282267}]},"public_addr":"172.21.15.116:6817/1099282267","cluster_addr":"172.21.15.116:6819/1099282267","heartbeat_back_addr":"172.21.15.116:6823/1099282267","heartbeat_front_addr":"172.21.15.116:6821/1099282267","state":["exists","up"]},{"osd":3,"uuid":"6a0d5344-8b6f-40d3-918d-0f0a4be688e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6825","nonce":1179942661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6827","nonce":1179942661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6831","nonce":1179942661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6829","nonce":1179942661}]},"public_addr":"172.21.15.116:6825/1179942661","cluster_addr":"172.21.15.116:6827/1179942661","heartbeat_back_addr":"172.21.15.116:6831/1179942661","heartbeat_front_addr":"172.21.15.116:6829/1179942661","state":["exists","up"]},{"osd":4,"uuid":"132e92e4-b825-441e-965a-ca00533fa86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6801","nonce":3132444330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6803","nonce":3132444330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6807","nonce":3132444330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6805","nonce":3132444330}]},"public_addr":"172.21.15.174:6801/3132444330","cluster_addr":"172.21.15.174:6803/3132444330","heartbeat_back_addr":"172.21.15.174:6807/3132444330","heartbeat_front_addr":"172.21.15.174:6805/3132444330","state":["exists","up"]},{"osd":5,"uuid":"a2eeb5b6-1b31-43cc-8306-a0adf3ee15bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6809","nonce":881817164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6811","nonce":881817164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6815","nonce":881817164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6813","nonce":881817164}]},"public_addr":"172.21.15.174:6809/881817164","cluster_addr":"172.21.15.174:6811/881817164","heartbeat_back_addr":"172.21.15.174:6815/881817164","heartbeat_front_addr":"172.21.15.174:6813/881817164","state":["exists","up"]},{"osd":6,"uuid":"588e873a-c841-4ba7-90f2-8956c87a9983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6825","nonce":117574251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6827","nonce":117574251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6831","nonce":117574251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6829","nonce":117574251}]},"public_addr":"172.21.15.174:6825/117574251","cluster_addr":"172.21.15.174:6827/117574251","heartbeat_back_addr":"172.21.15.174:6831/117574251","heartbeat_front_addr":"172.21.15.174:6829/117574251","state":["exists","up"]},{"osd":7,"uuid":"3b90f9b0-8a0e-4883-b088-ca1d4012870c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6817","nonce":2543899196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6819","nonce":2543899196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6823","nonce":2543899196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6821","nonce":2543899196}]},"public_addr":"172.21.15.174:6817/2543899196","cluster_addr":"172.21.15.174:6819/2543899196","heartbeat_back_addr":"172.21.15.174:6823/2543899196","heartbeat_front_addr":"172.21.15.174:6821/2543899196","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.293769+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.310034+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.404865+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.549658+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.192174+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.297266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.493649+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.452309+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2c","mappings":[{"from":0,"to":7}]},{"pgid":"5.3","mappings":[{"from":4,"to":5}]},{"pgid":"5.5","mappings":[{"from":4,"to":5}]},{"pgid":"5.14","mappings":[{"from":4,"to":5}]},{"pgid":"5.19","mappings":[{"from":4,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:6835/3499878473":"2024-04-26T00:30:15.781624+0000","172.21.15.116:6834/3499878473":"2024-04-26T00:30:15.781624+0000","172.21.15.174:6836/2915274073":"2024-04-26T00:29:50.776462+0000","172.21.15.116:6833/3891103804":"2024-04-26T00:28:50.767503+0000","172.21.15.116:6834/2371126056":"2024-04-26T00:27:35.754759+0000","172.21.15.174:6837/349772087":"2024-04-26T00:27:15.748274+0000","172.21.15.174:6836/349772087":"2024-04-26T00:27:15.748274+0000","172.21.15.116:6833/3693049068":"2024-04-26T00:26:45.739776+0000","172.21.15.116:6832/3693049068":"2024-04-26T00:26:45.739776+0000","172.21.15.174:6837/4181466530":"2024-04-26T00:26:15.734586+0000","172.21.15.174:6836/4181466530":"2024-04-26T00:26:15.734586+0000","172.21.15.174:6836/492366509":"2024-04-26T00:24:15.701887+0000","172.21.15.116:6832/2998042514":"2024-04-26T00:23:30.691827+0000","172.21.15.116:6833/663910035":"2024-04-26T00:22:35.677323+0000","172.21.15.116:6832/663910035":"2024-04-26T00:22:35.677323+0000","172.21.15.174:6837/321466844":"2024-04-26T00:22:10.671673+0000","172.21.15.116:6835/980221971":"2024-04-26T00:21:40.665115+0000","172.21.15.174:6837/991183935":"2024-04-26T00:20:50.654151+0000","172.21.15.174:6836/991183935":"2024-04-26T00:20:50.654151+0000","172.21.15.174:6836/321466844":"2024-04-26T00:22:10.671673+0000","172.21.15.116:6834/1358527848":"2024-04-26T00:11:00.511032+0000","172.21.15.116:6832/3841609112":"2024-04-26T00:07:00.448498+0000","172.21.15.116:6835/1670061091":"2024-04-26T00:09:30.490485+0000","172.21.15.116:6834/1971675962":"2024-04-26T00:25:25.721965+0000","172.21.15.116:6835/1233829678":"2024-04-26T00:07:30.454393+0000","172.21.15.116:6835/2371126056":"2024-04-26T00:27:35.754759+0000","172.21.15.174:6834/2883220846":"2024-04-26T00:11:50.521053+0000","172.21.15.116:6834/1233829678":"2024-04-26T00:07:30.454393+0000","172.21.15.174:6837/3180024481":"2024-04-26T00:15:25.567768+0000","172.21.15.116:6835/879531976":"2024-04-26T00:20:10.642563+0000","172.21.15.116:6833/174781672":"2024-04-26T00:02:40.399073+0000","172.21.15.174:6835/4215220222":"2024-04-26T00:15:40.572498+0000","172.21.15.116:6834/221971311":"2024-04-26T00:22:48.898075+0000","172.21.15.174:6836/1281466397":"2024-04-26T00:18:20.613347+0000","172.21.15.116:6832/3891103804":"2024-04-26T00:28:50.767503+0000","172.21.15.116:6835/1358527848":"2024-04-26T00:11:00.511032+0000","172.21.15.116:6835/221971311":"2024-04-26T00:22:48.898075+0000","172.21.15.116:6832/3006671671":"2024-04-26T00:09:30.490406+0000","172.21.15.116:6834/980221971":"2024-04-26T00:21:40.665115+0000","172.21.15.174:6836/1762356055":"2024-04-26T00:03:36.597345+0000","172.21.15.116:6833/3006671671":"2024-04-26T00:09:30.490406+0000","172.21.15.116:6834/1459177921":"2024-04-26T00:03:30.406778+0000","172.21.15.116:6834/462190523":"2024-04-26T00:14:55.560782+0000","172.21.15.116:6834/879531976":"2024-04-26T00:20:10.642563+0000","172.21.15.116:6832/1459177921":"2024-04-26T00:03:30.406778+0000","172.21.15.174:6837/1159002662":"2024-04-26T00:10:20.502218+0000","172.21.15.174:6837/1762356055":"2024-04-26T00:03:36.597345+0000","172.21.15.174:6834/4215220222":"2024-04-26T00:15:40.572498+0000","172.21.15.116:6835/1971675962":"2024-04-26T00:25:25.721965+0000","172.21.15.174:6836/3081407040":"2024-04-26T00:17:25.598516+0000","172.21.15.116:6834/1599708526":"2024-04-26T00:12:20.526742+0000","172.21.15.174:6836/2276529656":"2024-04-26T00:19:05.629048+0000","172.21.15.174:6836/1159002662":"2024-04-26T00:10:20.502218+0000","172.21.15.116:6833/2998042514":"2024-04-26T00:23:30.691827+0000","172.21.15.174:6837/3081407040":"2024-04-26T00:17:25.598516+0000","172.21.15.174:6835/2883220846":"2024-04-26T00:11:50.521053+0000","172.21.15.174:6837/492366509":"2024-04-26T00:24:15.701887+0000","172.21.15.116:6835/2004876235":"2024-04-26T00:04:45.421611+0000","172.21.15.116:6835/174781672":"2024-04-26T00:02:40.399073+0000","172.21.15.116:6833/3633657268":"2024-04-26T00:25:40.724634+0000","172.21.15.116:6832/365773583":"2024-04-26T00:14:58.857970+0000","172.21.15.174:6836/1889314958":"2024-04-26T00:06:30.441408+0000","172.21.15.116:6833/3705654249":"2024-04-26T00:12:45.533219+0000","172.21.15.116:6832/3705654249":"2024-04-26T00:12:45.533219+0000","172.21.15.116:6834/1670061091":"2024-04-26T00:09:30.490485+0000","172.21.15.116:6835/1599708526":"2024-04-26T00:12:20.526742+0000","172.21.15.116:6832/3633657268":"2024-04-26T00:25:40.724634+0000","172.21.15.174:6836/3583791822":"2024-04-26T00:08:00.460587+0000","172.21.15.174:6837/3583791822":"2024-04-26T00:08:00.460587+0000","172.21.15.174:6836/3398690896":"2024-04-26T00:13:30.543729+0000","172.21.15.174:6837/3398690896":"2024-04-26T00:13:30.543729+0000","172.21.15.116:6835/462190523":"2024-04-26T00:14:55.560782+0000","172.21.15.116:6834/3336339376":"2024-04-26T00:18:35.616887+0000","172.21.15.174:6836/3180024481":"2024-04-26T00:15:25.567768+0000","172.21.15.116:6833/365773583":"2024-04-26T00:14:58.857970+0000","172.21.15.116:6835/2012470875":"2024-04-26T00:16:35.584511+0000","172.21.15.116:6833/2004876235":"2024-04-26T00:04:45.421611+0000","172.21.15.174:6837/1281466397":"2024-04-26T00:18:20.613347+0000","172.21.15.116:6832/2340581953":"2024-04-26T00:16:45.587651+0000","172.21.15.174:6837/2915274073":"2024-04-26T00:29:50.776462+0000","172.21.15.116:6834/1649769369":"2024-04-26T00:04:35.418892+0000","172.21.15.116:6832/1649769369":"2024-04-26T00:04:35.418892+0000","172.21.15.116:6833/2340581953":"2024-04-26T00:16:45.587651+0000","172.21.15.174:6837/1889314958":"2024-04-26T00:06:30.441408+0000","172.21.15.116:6832/3645553116":"2024-04-26T00:17:35.601559+0000","172.21.15.116:6833/3645553116":"2024-04-26T00:17:35.601559+0000","172.21.15.116:6833/3841609112":"2024-04-26T00:07:00.448498+0000","172.21.15.116:6834/2012470875":"2024-04-26T00:16:35.584511+0000","172.21.15.116:6835/3336339376":"2024-04-26T00:18:35.616887+0000","172.21.15.174:6837/2276529656":"2024-04-26T00:19:05.629048+0000","172.21.15.116:6832/1655959433":"2024-04-26T00:19:30.633957+0000","172.21.15.116:6833/1655959433":"2024-04-26T00:19:30.633957+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:31:29.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.674+0000 7fe5b3a42640 1 -- 172.21.15.116:0/2585174515 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe5900678f0 msgr2=0x7fe590069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:29.677 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.674+0000 7fe5b3a42640 1 --2- 172.21.15.116:0/2585174515 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe5900678f0 0x7fe590069db0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fe5a8001740 tx=0x7fe5a8009040 comp rx=0 tx=0).stop 2024-04-25T00:31:29.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.674+0000 7fe5b3a42640 1 -- 172.21.15.116:0/2585174515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5ac0abe10 msgr2=0x7fe5ac14abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:29.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.674+0000 7fe5b3a42640 1 --2- 172.21.15.116:0/2585174515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5ac0abe10 0x7fe5ac14abd0 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7fe5a4002240 tx=0x7fe5a4009750 comp rx=0 tx=0).stop 2024-04-25T00:31:29.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.678+0000 7fe5b3a42640 1 -- 172.21.15.116:0/2585174515 shutdown_connections 2024-04-25T00:31:29.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.678+0000 7fe5b3a42640 1 --2- 172.21.15.116:0/2585174515 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fe5900678f0 0x7fe590069db0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:29.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.678+0000 7fe5b3a42640 1 --2- 172.21.15.116:0/2585174515 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fe5ac0abe10 0x7fe5ac14abd0 unknown :-1 s=CLOSED pgs=1874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:29.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.678+0000 7fe5b3a42640 1 --2- 172.21.15.116:0/2585174515 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fe5ac147560 0x7fe5ac146ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:29.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.678+0000 7fe5b3a42640 1 --2- 172.21.15.116:0/2585174515 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fe598004d50 0x7fe5ac146ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:29.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.678+0000 7fe5b3a42640 1 -- 172.21.15.116:0/2585174515 >> 172.21.15.116:0/2585174515 conn(0x7fe5ac0c4c10 msgr2=0x7fe5ac154150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:31:29.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.678+0000 7fe5b3a42640 1 -- 172.21.15.116:0/2585174515 shutdown_connections 2024-04-25T00:31:29.678 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.678+0000 7fe5b3a42640 1 -- 172.21.15.116:0/2585174515 wait complete. 2024-04-25T00:31:29.694 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail a 2024-04-25T00:31:29.809 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.810+0000 7f225a94a640 1 Processor -- start 2024-04-25T00:31:29.810 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.810+0000 7f225a94a640 1 -- start start 2024-04-25T00:31:29.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.810+0000 7f225a94a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f225406d5f0 0x7f2254156c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:29.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.810+0000 7f225a94a640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22540c9150 con 0x7f2254152d80 2024-04-25T00:31:29.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.810+0000 7f225a94a640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22540c9430 con 0x7f2254155810 2024-04-25T00:31:29.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.810+0000 7f225a94a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f22540c9d60 con 0x7f225406d5f0 2024-04-25T00:31:29.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.810+0000 7f2253fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f2254152d80 0x7f225406bd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:46834/0 (socket says 172.21.15.116:46834) 2024-04-25T00:31:29.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.810+0000 7f2253fff640 1 -- 172.21.15.116:0/672815932 learned_addr learned my addr 172.21.15.116:0/672815932 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:31:29.812 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.810+0000 7f224bfff640 1 --2- 172.21.15.116:0/672815932 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f225406d5f0 0x7f2254156c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:29.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.810+0000 7f22537fe640 1 -- 172.21.15.116:0/672815932 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4248395583 0 0) 0x7f22540c9150 con 0x7f2254152d80 2024-04-25T00:31:29.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.810+0000 7f22537fe640 1 -- 172.21.15.116:0/672815932 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2234003530 con 0x7f2254152d80 2024-04-25T00:31:29.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.810+0000 7f22537fe640 1 -- 172.21.15.116:0/672815932 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3591603428 0 0) 0x7f2234003530 con 0x7f2254152d80 2024-04-25T00:31:29.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.810+0000 7f22537fe640 1 -- 172.21.15.116:0/672815932 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f225406d5f0 msgr2=0x7f2254156c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:31:29.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.810+0000 7f22537fe640 1 --2- 172.21.15.116:0/672815932 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f225406d5f0 0x7f2254156c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:29.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.810+0000 7f22537fe640 1 -- 172.21.15.116:0/672815932 >> v1:172.21.15.174:6789/0 conn(0x7f2254155810 legacy=0x7f22541571c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:29.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.810+0000 7f22537fe640 1 -- 172.21.15.116:0/672815932 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22540c9710 con 0x7f2254152d80 2024-04-25T00:31:29.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.810+0000 7f22537fe640 1 -- 172.21.15.116:0/672815932 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f2244001680 con 0x7f2254152d80 2024-04-25T00:31:29.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.810+0000 7f22537fe640 1 -- 172.21.15.116:0/672815932 >> v1:172.21.15.116:6789/0 conn(0x7f2254152d80 legacy=0x7f225406bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:29.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.810+0000 7f22537fe640 1 --2- 172.21.15.116:0/672815932 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f225406d5f0 0x7f22340039f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:29.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.810+0000 7f22537fe640 1 --2- 172.21.15.116:0/672815932 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2234003f70 0x7f2234004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:29.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.810+0000 7f22537fe640 1 --2- 172.21.15.116:0/672815932 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2234004d50 0x7f22340051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:29.813 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.810+0000 7f22537fe640 1 -- 172.21.15.116:0/672815932 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f22540c9d60 con 0x7f2234004d50 2024-04-25T00:31:29.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.810+0000 7f22537fe640 1 -- 172.21.15.116:0/672815932 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f223400bc10 con 0x7f2234003f70 2024-04-25T00:31:29.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.810+0000 7f22537fe640 1 -- 172.21.15.116:0/672815932 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f223400be10 con 0x7f225406d5f0 2024-04-25T00:31:29.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.810+0000 7f2258ee7640 1 --2- 172.21.15.116:0/672815932 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2234004d50 0x7f22340051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:29.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.810+0000 7f2253fff640 1 --2- 172.21.15.116:0/672815932 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f225406d5f0 0x7f22340039f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:29.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.810+0000 7f224bfff640 1 --2- 172.21.15.116:0/672815932 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2234003f70 0x7f2234004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:29.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.810+0000 7f2258ee7640 1 -- 172.21.15.116:0/672815932 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f225406d5f0 msgr2=0x7f22340039f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:31:29.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.810+0000 7f2258ee7640 1 --2- 172.21.15.116:0/672815932 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f225406d5f0 0x7f22340039f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:29.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.810+0000 7f2258ee7640 1 -- 172.21.15.116:0/672815932 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2234003f70 msgr2=0x7f2234004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:31:29.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.810+0000 7f2258ee7640 1 --2- 172.21.15.116:0/672815932 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2234003f70 0x7f2234004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:29.814 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.810+0000 7f2258ee7640 1 -- 172.21.15.116:0/672815932 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2234003530 con 0x7f2234004d50 2024-04-25T00:31:29.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.814+0000 7f2258ee7640 1 --2- 172.21.15.116:0/672815932 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2234004d50 0x7f22340051d0 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7f224c001e70 tx=0x7f224c01c750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e09964bf6afac9d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:29.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.814+0000 7f22537fe640 1 -- 172.21.15.116:0/672815932 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f224c001a90 con 0x7f2234004d50 2024-04-25T00:31:29.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.814+0000 7f22537fe640 1 -- 172.21.15.116:0/672815932 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f224c003b80 con 0x7f2234004d50 2024-04-25T00:31:29.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.814+0000 7f22537fe640 1 -- 172.21.15.116:0/672815932 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f224c005080 con 0x7f2234004d50 2024-04-25T00:31:29.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.814+0000 7f225a94a640 1 -- 172.21.15.116:0/672815932 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2234004d50 msgr2=0x7f22340051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:29.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.814+0000 7f225a94a640 1 --2- 172.21.15.116:0/672815932 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f2234004d50 0x7f22340051d0 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7f224c001e70 tx=0x7f224c01c750 comp rx=0 tx=0).stop 2024-04-25T00:31:29.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.814+0000 7f2258ee7640 1 -- 172.21.15.116:0/672815932 reap_dead start 2024-04-25T00:31:29.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.814+0000 7f225a94a640 1 -- 172.21.15.116:0/672815932 shutdown_connections 2024-04-25T00:31:29.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.814+0000 7f225a94a640 1 -- 172.21.15.116:0/672815932 >> 172.21.15.116:0/672815932 conn(0x7f22540acdd0 msgr2=0x7f22540ad1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:31:29.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.814+0000 7f225a94a640 1 -- 172.21.15.116:0/672815932 shutdown_connections 2024-04-25T00:31:29.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.814+0000 7f225a94a640 1 -- 172.21.15.116:0/672815932 wait complete. 2024-04-25T00:31:29.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.814+0000 7f225a94a640 1 Processor -- start 2024-04-25T00:31:29.816 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.814+0000 7f225a94a640 1 -- start start 2024-04-25T00:31:29.818 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.814+0000 7f225a94a640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2234004d50 0x7f22541486f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:29.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.814+0000 7f225a94a640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f22540ac7c0 0x7f22540aaa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:29.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.814+0000 7f225a94a640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f225415fdc0 0x7f22540aafb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:29.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.814+0000 7f225a94a640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f2254153160 con 0x7f225415fdc0 2024-04-25T00:31:29.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.814+0000 7f225a94a640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f22540cad80 con 0x7f2234004d50 2024-04-25T00:31:29.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.814+0000 7f225a94a640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f225415a990 con 0x7f22540ac7c0 2024-04-25T00:31:29.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.814+0000 7f2258ee7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f225415fdc0 0x7f22540aafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:29.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.814+0000 7f2258ee7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f225415fdc0 0x7f22540aafb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51162/0 (socket says 172.21.15.116:51162) 2024-04-25T00:31:29.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.814+0000 7f2258ee7640 1 -- 172.21.15.116:0/1214618741 learned_addr learned my addr 172.21.15.116:0/1214618741 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:31:29.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.814+0000 7f2253fff640 1 --2- 172.21.15.116:0/1214618741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2234004d50 0x7f22541486f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:29.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.814+0000 7f224bfff640 1 --2- 172.21.15.116:0/1214618741 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f22540ac7c0 0x7f22540aaa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:29.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.814+0000 7f2258ee7640 1 -- 172.21.15.116:0/1214618741 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f22540ac7c0 msgr2=0x7f22540aaa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:29.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.814+0000 7f2258ee7640 1 --2- 172.21.15.116:0/1214618741 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f22540ac7c0 0x7f22540aaa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:29.819 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.814+0000 7f2258ee7640 1 -- 172.21.15.116:0/1214618741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2234004d50 msgr2=0x7f22541486f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:29.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.814+0000 7f2258ee7640 1 --2- 172.21.15.116:0/1214618741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2234004d50 0x7f22541486f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:29.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.814+0000 7f2258ee7640 1 -- 172.21.15.116:0/1214618741 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2254145a90 con 0x7f225415fdc0 2024-04-25T00:31:29.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.818+0000 7f2258ee7640 1 --2- 172.21.15.116:0/1214618741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f225415fdc0 0x7f22540aafb0 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f224c098050 tx=0x7f2234005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:29.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.818+0000 7f22517fa640 1 -- 172.21.15.116:0/1214618741 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f224c0bc070 con 0x7f225415fdc0 2024-04-25T00:31:29.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.818+0000 7f22517fa640 1 -- 172.21.15.116:0/1214618741 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f224c005610 con 0x7f225415fdc0 2024-04-25T00:31:29.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.818+0000 7f225a94a640 1 -- 172.21.15.116:0/1214618741 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2254147eb0 con 0x7f225415fdc0 2024-04-25T00:31:29.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.818+0000 7f22517fa640 1 -- 172.21.15.116:0/1214618741 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f224c007dc0 con 0x7f225415fdc0 2024-04-25T00:31:29.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.818+0000 7f225a94a640 1 -- 172.21.15.116:0/1214618741 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22541f2600 con 0x7f225415fdc0 2024-04-25T00:31:29.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.818+0000 7f225a94a640 1 -- 172.21.15.116:0/1214618741 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2254068350 con 0x7f225415fdc0 2024-04-25T00:31:29.820 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.818+0000 7f22517fa640 1 -- 172.21.15.116:0/1214618741 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f224c0b1040 con 0x7f225415fdc0 2024-04-25T00:31:29.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.818+0000 7f22517fa640 1 --2- 172.21.15.116:0/1214618741 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f22380678c0 0x7f2238069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:29.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.818+0000 7f22517fa640 1 -- 172.21.15.116:0/1214618741 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 10622+0+0 (secure 0 0 0) 0x7f224c1209f0 con 0x7f225415fdc0 2024-04-25T00:31:29.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.818+0000 7f2253fff640 1 --2- 172.21.15.116:0/1214618741 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f22380678c0 0x7f2238069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:29.821 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.818+0000 7f2253fff640 1 --2- 172.21.15.116:0/1214618741 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f22380678c0 0x7f2238069d80 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f2244000c10 tx=0x7f2244009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:29.824 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:29.822+0000 7f22517fa640 1 -- 172.21.15.116:0/1214618741 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f224c0ad020 con 0x7f225415fdc0 2024-04-25T00:31:30.001 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:31:30.020 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.018+0000 7f225a94a640 1 -- 172.21.15.116:0/1214618741 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "a"} v 0) -- 0x7f22540cab60 con 0x7f225415fdc0 2024-04-25T00:31:30.022 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.022+0000 7f22517fa640 1 -- 172.21.15.116:0/1214618741 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "a"}]=0 a marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v243) ==== 151+0+0 (secure 0 0 0) 0x7f224c016490 con 0x7f225415fdc0 2024-04-25T00:31:30.022 INFO:teuthology.orchestra.run.smithi116.stderr:a marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-25T00:31:30.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.022+0000 7f225a94a640 1 -- 172.21.15.116:0/1214618741 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f22380678c0 msgr2=0x7f2238069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:30.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.022+0000 7f225a94a640 1 --2- 172.21.15.116:0/1214618741 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f22380678c0 0x7f2238069d80 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f2244000c10 tx=0x7f2244009040 comp rx=0 tx=0).stop 2024-04-25T00:31:30.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.022+0000 7f225a94a640 1 -- 172.21.15.116:0/1214618741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f225415fdc0 msgr2=0x7f22540aafb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:30.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.022+0000 7f225a94a640 1 --2- 172.21.15.116:0/1214618741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f225415fdc0 0x7f22540aafb0 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f224c098050 tx=0x7f2234005840 comp rx=0 tx=0).stop 2024-04-25T00:31:30.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.022+0000 7f225a94a640 1 -- 172.21.15.116:0/1214618741 shutdown_connections 2024-04-25T00:31:30.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.022+0000 7f225a94a640 1 --2- 172.21.15.116:0/1214618741 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f22380678c0 0x7f2238069d80 secure :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f2244000c10 tx=0x7f2244009040 comp rx=0 tx=0).stop 2024-04-25T00:31:30.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.022+0000 7f225a94a640 1 --2- 172.21.15.116:0/1214618741 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f225415fdc0 0x7f22540aafb0 unknown :-1 s=CLOSED pgs=1877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:30.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.022+0000 7f225a94a640 1 --2- 172.21.15.116:0/1214618741 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f22540ac7c0 0x7f22540aaa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:30.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.022+0000 7f225a94a640 1 --2- 172.21.15.116:0/1214618741 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f2234004d50 0x7f22541486f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:30.025 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.022+0000 7f225a94a640 1 -- 172.21.15.116:0/1214618741 >> 172.21.15.116:0/1214618741 conn(0x7f22540acdd0 msgr2=0x7f22541571a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:31:30.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.022+0000 7f225a94a640 1 -- 172.21.15.116:0/1214618741 shutdown_connections 2024-04-25T00:31:30.026 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.022+0000 7f225a94a640 1 -- 172.21.15.116:0/1214618741 wait complete. 2024-04-25T00:31:30.044 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm a --yes-i-really-mean-it 2024-04-25T00:31:30.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.110+0000 7f0a9fd0c640 1 Processor -- start 2024-04-25T00:31:30.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.110+0000 7f0a9fd0c640 1 -- start start 2024-04-25T00:31:30.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.110+0000 7f0a9fd0c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0a98152d50 0x7f0a98159910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:30.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.114+0000 7f0a9fd0c640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0a98073140 0x7f0a980c2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:30.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.114+0000 7f0a9fd0c640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f0a980c37b0 0x7f0a980c5010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:30.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.114+0000 7f0a9fd0c640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f0a981563c0 con 0x7f0a98073140 2024-04-25T00:31:30.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.114+0000 7f0a9fd0c640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f0a98158e30 con 0x7f0a980c37b0 2024-04-25T00:31:30.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.114+0000 7f0a9fd0c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f0a98159110 con 0x7f0a98152d50 2024-04-25T00:31:30.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.114+0000 7f0a9d2a7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0a98073140 0x7f0a980c2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:30.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.114+0000 7f0a9d2a7640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f0a98073140 0x7f0a980c2330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51176/0 (socket says 172.21.15.116:51176) 2024-04-25T00:31:30.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.114+0000 7f0a9d2a7640 1 -- 172.21.15.116:0/4153961717 learned_addr learned my addr 172.21.15.116:0/4153961717 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:31:30.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.114+0000 7f0a9daa8640 1 --2- 172.21.15.116:0/4153961717 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0a98152d50 0x7f0a98159910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:30.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.114+0000 7f0a9e2a9640 1 --2- 172.21.15.116:0/4153961717 >> v2:172.21.15.174:3300/0 conn(0x7f0a980c37b0 0x7f0a980c5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:30.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.114+0000 7f0a9d2a7640 1 -- 172.21.15.116:0/4153961717 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0a98152d50 msgr2=0x7f0a98159910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:31:30.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.114+0000 7f0a9d2a7640 1 --2- 172.21.15.116:0/4153961717 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0a98152d50 0x7f0a98159910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:30.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.114+0000 7f0a9d2a7640 1 -- 172.21.15.116:0/4153961717 >> v2:172.21.15.174:3300/0 conn(0x7f0a980c37b0 msgr2=0x7f0a980c5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:31:30.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.114+0000 7f0a9d2a7640 1 --2- 172.21.15.116:0/4153961717 >> v2:172.21.15.174:3300/0 conn(0x7f0a980c37b0 0x7f0a980c5010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:30.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.114+0000 7f0a9d2a7640 1 -- 172.21.15.116:0/4153961717 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a981593f0 con 0x7f0a98073140 2024-04-25T00:31:30.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.114+0000 7f0a9d2a7640 1 --2- 172.21.15.116:0/4153961717 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a98073140 0x7f0a980c2330 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f0a98159040 tx=0x7f0a9401c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d8d10cb1a61186b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:30.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.114+0000 7f0a9caa6640 1 -- 172.21.15.116:0/4153961717 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a940b0070 con 0x7f0a98073140 2024-04-25T00:31:30.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.114+0000 7f0a9caa6640 1 -- 172.21.15.116:0/4153961717 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f0a940ab600 con 0x7f0a98073140 2024-04-25T00:31:30.116 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.114+0000 7f0a9caa6640 1 -- 172.21.15.116:0/4153961717 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a940ab9f0 con 0x7f0a98073140 2024-04-25T00:31:30.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.114+0000 7f0a9fd0c640 1 -- 172.21.15.116:0/4153961717 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a98073140 msgr2=0x7f0a980c2330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:30.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.114+0000 7f0a9fd0c640 1 --2- 172.21.15.116:0/4153961717 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a98073140 0x7f0a980c2330 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f0a98159040 tx=0x7f0a9401c490 comp rx=0 tx=0).stop 2024-04-25T00:31:30.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.114+0000 7f0a9fd0c640 1 -- 172.21.15.116:0/4153961717 shutdown_connections 2024-04-25T00:31:30.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.114+0000 7f0a9fd0c640 1 --2- 172.21.15.116:0/4153961717 >> v2:172.21.15.174:3300/0 conn(0x7f0a980c37b0 0x7f0a980c5010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:30.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.114+0000 7f0a9fd0c640 1 --2- 172.21.15.116:0/4153961717 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a98073140 0x7f0a980c2330 unknown :-1 s=CLOSED pgs=1878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:30.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.114+0000 7f0a9fd0c640 1 --2- 172.21.15.116:0/4153961717 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0a98152d50 0x7f0a98159910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:30.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.114+0000 7f0a9fd0c640 1 -- 172.21.15.116:0/4153961717 >> 172.21.15.116:0/4153961717 conn(0x7f0a9806c720 msgr2=0x7f0a9806cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:31:30.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.114+0000 7f0a9fd0c640 1 -- 172.21.15.116:0/4153961717 shutdown_connections 2024-04-25T00:31:30.117 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.114+0000 7f0a9fd0c640 1 -- 172.21.15.116:0/4153961717 wait complete. 2024-04-25T00:31:30.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.118+0000 7f0a9fd0c640 1 Processor -- start 2024-04-25T00:31:30.118 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.118+0000 7f0a9fd0c640 1 -- start start 2024-04-25T00:31:30.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.118+0000 7f0a9fd0c640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0a98073140 0x7f0a981ef180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:30.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.118+0000 7f0a9fd0c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a980c37b0 0x7f0a981464e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:30.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.118+0000 7f0a9fd0c640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0a98152d50 0x7f0a98146a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:30.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.118+0000 7f0a9fd0c640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f0a9806bdb0 con 0x7f0a980c37b0 2024-04-25T00:31:30.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.118+0000 7f0a9fd0c640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f0a9806bc30 con 0x7f0a98073140 2024-04-25T00:31:30.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.118+0000 7f0a9fd0c640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f0a9806bf30 con 0x7f0a98152d50 2024-04-25T00:31:30.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.118+0000 7f0a9d2a7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a980c37b0 0x7f0a981464e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:30.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.118+0000 7f0a9d2a7640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a980c37b0 0x7f0a981464e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51190/0 (socket says 172.21.15.116:51190) 2024-04-25T00:31:30.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.118+0000 7f0a9d2a7640 1 -- 172.21.15.116:0/2679973461 learned_addr learned my addr 172.21.15.116:0/2679973461 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:31:30.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.118+0000 7f0a9e2a9640 1 --2- 172.21.15.116:0/2679973461 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0a98152d50 0x7f0a98146a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:30.120 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.118+0000 7f0a9daa8640 1 --2- 172.21.15.116:0/2679973461 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0a98073140 0x7f0a981ef180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:30.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.118+0000 7f0a9d2a7640 1 -- 172.21.15.116:0/2679973461 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0a98152d50 msgr2=0x7f0a98146a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:30.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.118+0000 7f0a9d2a7640 1 --2- 172.21.15.116:0/2679973461 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0a98152d50 0x7f0a98146a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:30.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.118+0000 7f0a9d2a7640 1 -- 172.21.15.116:0/2679973461 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0a98073140 msgr2=0x7f0a981ef180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:30.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.118+0000 7f0a9d2a7640 1 --2- 172.21.15.116:0/2679973461 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0a98073140 0x7f0a981ef180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:30.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.118+0000 7f0a9d2a7640 1 -- 172.21.15.116:0/2679973461 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a98146f60 con 0x7f0a980c37b0 2024-04-25T00:31:30.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.118+0000 7f0a9d2a7640 1 --2- 172.21.15.116:0/2679973461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a980c37b0 0x7f0a981464e0 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7f0a9401ab40 tx=0x7f0a94019230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:30.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.118+0000 7f0a8e7fc640 1 -- 172.21.15.116:0/2679973461 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a940b0040 con 0x7f0a980c37b0 2024-04-25T00:31:30.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.118+0000 7f0a8e7fc640 1 -- 172.21.15.116:0/2679973461 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f0a940b9070 con 0x7f0a980c37b0 2024-04-25T00:31:30.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.118+0000 7f0a9fd0c640 1 -- 172.21.15.116:0/2679973461 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a9814b580 con 0x7f0a980c37b0 2024-04-25T00:31:30.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.118+0000 7f0a8e7fc640 1 -- 172.21.15.116:0/2679973461 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a940b4680 con 0x7f0a980c37b0 2024-04-25T00:31:30.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.118+0000 7f0a9fd0c640 1 -- 172.21.15.116:0/2679973461 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a9814a930 con 0x7f0a980c37b0 2024-04-25T00:31:30.121 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.118+0000 7f0a9fd0c640 1 -- 172.21.15.116:0/2679973461 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a98068a60 con 0x7f0a980c37b0 2024-04-25T00:31:30.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.118+0000 7f0a8e7fc640 1 -- 172.21.15.116:0/2679973461 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f0a940b4820 con 0x7f0a980c37b0 2024-04-25T00:31:30.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.122+0000 7f0a8e7fc640 1 --2- 172.21.15.116:0/2679973461 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0a7c0678c0 0x7f0a7c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:30.122 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.122+0000 7f0a8e7fc640 1 -- 172.21.15.116:0/2679973461 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 10708+0+0 (secure 0 0 0) 0x7f0a94120cf0 con 0x7f0a980c37b0 2024-04-25T00:31:30.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.122+0000 7f0a9daa8640 1 --2- 172.21.15.116:0/2679973461 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0a7c0678c0 0x7f0a7c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:30.123 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.122+0000 7f0a9daa8640 1 --2- 172.21.15.116:0/2679973461 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0a7c0678c0 0x7f0a7c069d80 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f0a98155750 tx=0x7f0a88008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:30.129 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.130+0000 7f0a8e7fc640 1 -- 172.21.15.116:0/2679973461 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f0a94125070 con 0x7f0a980c37b0 2024-04-25T00:31:30.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.202+0000 7f3166bd4640 1 Processor -- start 2024-04-25T00:31:30.203 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.202+0000 7f3166bd4640 1 -- start start 2024-04-25T00:31:30.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.202+0000 7f3166bd4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3160072ff0 0x7f31601531c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:30.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.202+0000 7f3166bd4640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f316006c930 con 0x7f316006e930 2024-04-25T00:31:30.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.202+0000 7f3166bd4640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3160071e90 con 0x7f316006b210 2024-04-25T00:31:30.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.202+0000 7f3166bd4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f316006b680 con 0x7f3160072ff0 2024-04-25T00:31:30.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.202+0000 7f3164970640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f316006e930 0x7f3160152ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:46846/0 (socket says 172.21.15.116:46846) 2024-04-25T00:31:30.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.202+0000 7f3164970640 1 -- 172.21.15.116:0/2519859133 learned_addr learned my addr 172.21.15.116:0/2519859133 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:31:30.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.202+0000 7f315ffff640 1 --2- 172.21.15.116:0/2519859133 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3160072ff0 0x7f31601531c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:30.205 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.202+0000 7f315f7fe640 1 -- 172.21.15.116:0/2519859133 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3846871089 0 0) 0x7f316006c930 con 0x7f316006e930 2024-04-25T00:31:30.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.202+0000 7f315f7fe640 1 -- 172.21.15.116:0/2519859133 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3148003530 con 0x7f316006e930 2024-04-25T00:31:30.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.202+0000 7f315f7fe640 1 -- 172.21.15.116:0/2519859133 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 938656827 0 0) 0x7f3148003530 con 0x7f316006e930 2024-04-25T00:31:30.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.202+0000 7f315f7fe640 1 -- 172.21.15.116:0/2519859133 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3160072ff0 msgr2=0x7f31601531c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:31:30.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.202+0000 7f315f7fe640 1 --2- 172.21.15.116:0/2519859133 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3160072ff0 0x7f31601531c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:30.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.202+0000 7f315f7fe640 1 -- 172.21.15.116:0/2519859133 >> v1:172.21.15.174:6789/0 conn(0x7f316006b210 legacy=0x7f3160158e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:30.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.202+0000 7f315f7fe640 1 -- 172.21.15.116:0/2519859133 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3160072170 con 0x7f316006e930 2024-04-25T00:31:30.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.202+0000 7f315f7fe640 1 -- 172.21.15.116:0/2519859133 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f3158001a20 con 0x7f316006e930 2024-04-25T00:31:30.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.202+0000 7f315f7fe640 1 -- 172.21.15.116:0/2519859133 >> v1:172.21.15.116:6789/0 conn(0x7f316006e930 legacy=0x7f3160152ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:30.206 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.202+0000 7f315f7fe640 1 --2- 172.21.15.116:0/2519859133 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31480039f0 0x7f3148003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:30.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.202+0000 7f315f7fe640 1 --2- 172.21.15.116:0/2519859133 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3160072ff0 0x7f3148004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:30.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.202+0000 7f315f7fe640 1 --2- 172.21.15.116:0/2519859133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3148004d50 0x7f31480051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:30.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.202+0000 7f315f7fe640 1 -- 172.21.15.116:0/2519859133 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f316006b680 con 0x7f3148004d50 2024-04-25T00:31:30.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.202+0000 7f315f7fe640 1 -- 172.21.15.116:0/2519859133 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f314800bb60 con 0x7f31480039f0 2024-04-25T00:31:30.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.202+0000 7f315f7fe640 1 -- 172.21.15.116:0/2519859133 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f314800bd60 con 0x7f3160072ff0 2024-04-25T00:31:30.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.202+0000 7f3165171640 1 --2- 172.21.15.116:0/2519859133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3148004d50 0x7f31480051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:30.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.202+0000 7f3164970640 1 --2- 172.21.15.116:0/2519859133 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31480039f0 0x7f3148003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:30.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.202+0000 7f315ffff640 1 --2- 172.21.15.116:0/2519859133 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3160072ff0 0x7f3148004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:30.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.202+0000 7f3165171640 1 -- 172.21.15.116:0/2519859133 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3160072ff0 msgr2=0x7f3148004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:31:30.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.202+0000 7f3165171640 1 --2- 172.21.15.116:0/2519859133 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3160072ff0 0x7f3148004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:30.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.202+0000 7f3165171640 1 -- 172.21.15.116:0/2519859133 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31480039f0 msgr2=0x7f3148003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:31:30.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.202+0000 7f3165171640 1 --2- 172.21.15.116:0/2519859133 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31480039f0 0x7f3148003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:30.207 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.202+0000 7f3165171640 1 -- 172.21.15.116:0/2519859133 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3148003530 con 0x7f3148004d50 2024-04-25T00:31:30.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.206+0000 7f3165171640 1 --2- 172.21.15.116:0/2519859133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3148004d50 0x7f31480051d0 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f3150001780 tx=0x7f315001c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b935b6e786222096 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:30.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.206+0000 7f315f7fe640 1 -- 172.21.15.116:0/2519859133 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31500013a0 con 0x7f3148004d50 2024-04-25T00:31:30.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.206+0000 7f315f7fe640 1 -- 172.21.15.116:0/2519859133 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f31500a9070 con 0x7f3148004d50 2024-04-25T00:31:30.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.206+0000 7f315f7fe640 1 -- 172.21.15.116:0/2519859133 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3150003600 con 0x7f3148004d50 2024-04-25T00:31:30.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.206+0000 7f3166bd4640 1 -- 172.21.15.116:0/2519859133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3148004d50 msgr2=0x7f31480051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:30.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.206+0000 7f3166bd4640 1 --2- 172.21.15.116:0/2519859133 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f3148004d50 0x7f31480051d0 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f3150001780 tx=0x7f315001c420 comp rx=0 tx=0).stop 2024-04-25T00:31:30.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.206+0000 7f3165171640 1 -- 172.21.15.116:0/2519859133 reap_dead start 2024-04-25T00:31:30.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.206+0000 7f3166bd4640 1 -- 172.21.15.116:0/2519859133 shutdown_connections 2024-04-25T00:31:30.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.206+0000 7f3166bd4640 1 -- 172.21.15.116:0/2519859133 >> 172.21.15.116:0/2519859133 conn(0x7f31600c7a90 msgr2=0x7f31600c7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:31:30.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.206+0000 7f3166bd4640 1 -- 172.21.15.116:0/2519859133 shutdown_connections 2024-04-25T00:31:30.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.206+0000 7f3166bd4640 1 -- 172.21.15.116:0/2519859133 wait complete. 2024-04-25T00:31:30.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.206+0000 7f3166bd4640 1 Processor -- start 2024-04-25T00:31:30.209 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.206+0000 7f3166bd4640 1 -- start start 2024-04-25T00:31:30.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.210+0000 7f3166bd4640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3148004d50 0x7f31601e07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:30.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.210+0000 7f3166bd4640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31600ab9f0 0x7f31601eaf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:30.211 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.210+0000 7f3166bd4640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31601eb440 0x7f3160155930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:30.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.210+0000 7f3166bd4640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f3160072a10 con 0x7f31600ab9f0 2024-04-25T00:31:30.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.210+0000 7f3166bd4640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f3160072730 con 0x7f31601eb440 2024-04-25T00:31:30.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.210+0000 7f3166bd4640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f316006ed10 con 0x7f3148004d50 2024-04-25T00:31:30.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.210+0000 7f315ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31600ab9f0 0x7f31601eaf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:30.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.210+0000 7f315ffff640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31600ab9f0 0x7f31601eaf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51214/0 (socket says 172.21.15.116:51214) 2024-04-25T00:31:30.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.210+0000 7f315ffff640 1 -- 172.21.15.116:0/3790101508 learned_addr learned my addr 172.21.15.116:0/3790101508 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:31:30.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.210+0000 7f315ffff640 1 -- 172.21.15.116:0/3790101508 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3148004d50 msgr2=0x7f31601e07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:30.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.210+0000 7f3165171640 1 --2- 172.21.15.116:0/3790101508 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31601eb440 0x7f3160155930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:30.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.210+0000 7f315ffff640 1 --2- 172.21.15.116:0/3790101508 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3148004d50 0x7f31601e07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:30.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.210+0000 7f315ffff640 1 -- 172.21.15.116:0/3790101508 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31601eb440 msgr2=0x7f3160155930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:30.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.210+0000 7f315ffff640 1 --2- 172.21.15.116:0/3790101508 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31601eb440 0x7f3160155930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:30.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.210+0000 7f315ffff640 1 -- 172.21.15.116:0/3790101508 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31601e0d00 con 0x7f31600ab9f0 2024-04-25T00:31:30.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.210+0000 7f315ffff640 1 --2- 172.21.15.116:0/3790101508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31600ab9f0 0x7f31601eaf00 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f315400f580 tx=0x7f315400fa40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:30.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.210+0000 7f315d7fa640 1 -- 172.21.15.116:0/3790101508 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3154004230 con 0x7f31600ab9f0 2024-04-25T00:31:30.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.210+0000 7f315d7fa640 1 -- 172.21.15.116:0/3790101508 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f31540043d0 con 0x7f31600ab9f0 2024-04-25T00:31:30.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.210+0000 7f3166bd4640 1 -- 172.21.15.116:0/3790101508 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31601ef370 con 0x7f31600ab9f0 2024-04-25T00:31:30.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.210+0000 7f315d7fa640 1 -- 172.21.15.116:0/3790101508 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31540148b0 con 0x7f31600ab9f0 2024-04-25T00:31:30.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.210+0000 7f3166bd4640 1 -- 172.21.15.116:0/3790101508 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31601ef090 con 0x7f31600ab9f0 2024-04-25T00:31:30.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.210+0000 7f3166bd4640 1 -- 172.21.15.116:0/3790101508 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f316004f510 con 0x7f31600ab9f0 2024-04-25T00:31:30.213 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.210+0000 7f315d7fa640 1 -- 172.21.15.116:0/3790101508 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f3154012070 con 0x7f31600ab9f0 2024-04-25T00:31:30.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.210+0000 7f315d7fa640 1 --2- 172.21.15.116:0/3790101508 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f314c0678c0 0x7f314c069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:30.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.210+0000 7f315d7fa640 1 -- 172.21.15.116:0/3790101508 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 10708+0+0 (secure 0 0 0) 0x7f3154087fc0 con 0x7f31600ab9f0 2024-04-25T00:31:30.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.210+0000 7f3164970640 1 --2- 172.21.15.116:0/3790101508 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f314c0678c0 0x7f314c069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:30.214 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.210+0000 7f3164970640 1 --2- 172.21.15.116:0/3790101508 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f314c0678c0 0x7f314c069d80 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f3158001f10 tx=0x7f3158001e80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:30.221 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.218+0000 7f315d7fa640 1 -- 172.21.15.116:0/3790101508 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f31601e0d00 con 0x7f31600ab9f0 2024-04-25T00:31:30.318 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.318+0000 7f0a9fd0c640 1 -- 172.21.15.116:0/2679973461 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0a9806f980 con 0x7f0a980c37b0 2024-04-25T00:31:30.319 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.318+0000 7f0a8e7fc640 1 -- 172.21.15.116:0/2679973461 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 243 v243) ==== 95+0+5661 (secure 0 0 0) 0x7f0a94016110 con 0x7f0a980c37b0 2024-04-25T00:31:30.319 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:31:30.320 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":243,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":9027,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/426062493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":426062493},{"type":"v1","addr":"172.21.15.116:6835","nonce":426062493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":242}],"filesystems":[{"mdsmap":{"epoch":243,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:31:30.022889+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":241,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:30:19.111393+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6764},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6764":{"gid":6764,"name":"d","rank":0,"incarnation":238,"state":"up:active","state_seq":8,"addr":"172.21.15.174:6837/3873739116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3873739116},{"type":"v1","addr":"172.21.15.174:6837","nonce":3873739116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6764,"qdb_cluster":[6764]},"id":2}]} 2024-04-25T00:31:30.320 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 243 2024-04-25T00:31:30.321 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.322+0000 7f0a9fd0c640 1 -- 172.21.15.116:0/2679973461 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0a7c0678c0 msgr2=0x7f0a7c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:30.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.322+0000 7f0a9fd0c640 1 --2- 172.21.15.116:0/2679973461 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0a7c0678c0 0x7f0a7c069d80 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f0a98155750 tx=0x7f0a88008040 comp rx=0 tx=0).stop 2024-04-25T00:31:30.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.322+0000 7f0a9fd0c640 1 -- 172.21.15.116:0/2679973461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a980c37b0 msgr2=0x7f0a981464e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:30.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.322+0000 7f0a9fd0c640 1 --2- 172.21.15.116:0/2679973461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a980c37b0 0x7f0a981464e0 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7f0a9401ab40 tx=0x7f0a94019230 comp rx=0 tx=0).stop 2024-04-25T00:31:30.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.322+0000 7f0a9fd0c640 1 -- 172.21.15.116:0/2679973461 shutdown_connections 2024-04-25T00:31:30.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.322+0000 7f0a9fd0c640 1 --2- 172.21.15.116:0/2679973461 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f0a7c0678c0 0x7f0a7c069d80 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:30.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.322+0000 7f0a9fd0c640 1 --2- 172.21.15.116:0/2679973461 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f0a98152d50 0x7f0a98146a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:30.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.322+0000 7f0a9fd0c640 1 --2- 172.21.15.116:0/2679973461 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f0a980c37b0 0x7f0a981464e0 unknown :-1 s=CLOSED pgs=1879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:30.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.322+0000 7f0a9fd0c640 1 --2- 172.21.15.116:0/2679973461 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f0a98073140 0x7f0a981ef180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:30.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.322+0000 7f0a9fd0c640 1 -- 172.21.15.116:0/2679973461 >> 172.21.15.116:0/2679973461 conn(0x7f0a9806c720 msgr2=0x7f0a9806dec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:31:30.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.322+0000 7f0a9fd0c640 1 -- 172.21.15.116:0/2679973461 shutdown_connections 2024-04-25T00:31:30.322 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.322+0000 7f0a9fd0c640 1 -- 172.21.15.116:0/2679973461 wait complete. 2024-04-25T00:31:30.337 INFO:tasks.mds_thrash.fs.[a]:mds.c down, removed from mdsmap 2024-04-25T00:31:30.337 INFO:tasks.mds_thrash.fs.[a]:waiting for mds cluster to stabilize... 2024-04-25T00:31:30.338 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:31:30.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.402+0000 7f3166bd4640 1 -- 172.21.15.116:0/3790101508 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "a", "yes_i_really_mean_it": true} v 0) -- 0x7f316006bea0 con 0x7f31600ab9f0 2024-04-25T00:31:30.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.498+0000 7f6417742640 1 Processor -- start 2024-04-25T00:31:30.501 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.498+0000 7f6417742640 1 -- start start 2024-04-25T00:31:30.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.502+0000 7f6417742640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6410150d20 0x7f64100c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:30.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.502+0000 7f6417742640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f64100a94a0 0x7f6410153ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:30.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.502+0000 7f6417742640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64100c3910 con 0x7f64100c2f50 2024-04-25T00:31:30.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.502+0000 7f6417742640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f6410154530 con 0x7f64100a94a0 2024-04-25T00:31:30.502 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.502+0000 7f6417742640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f64100c16e0 con 0x7f6410150d20 2024-04-25T00:31:30.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.502+0000 7f6414cdd640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f64100c2f50 0x7f64101538e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:46856/0 (socket says 172.21.15.116:46856) 2024-04-25T00:31:30.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.502+0000 7f6414cdd640 1 -- 172.21.15.116:0/3014481845 learned_addr learned my addr 172.21.15.116:0/3014481845 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:31:30.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.502+0000 7f6415cdf640 1 --2- 172.21.15.116:0/3014481845 >> v2:172.21.15.174:3300/0 conn(0x7f64100a94a0 0x7f6410153ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:30.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.502+0000 7f64154de640 1 --2- 172.21.15.116:0/3014481845 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6410150d20 0x7f64100c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:30.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.502+0000 7f6407fff640 1 -- 172.21.15.116:0/3014481845 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3006037061 0 0) 0x7f64100c3910 con 0x7f64100c2f50 2024-04-25T00:31:30.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.502+0000 7f6407fff640 1 -- 172.21.15.116:0/3014481845 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f63fc003530 con 0x7f64100c2f50 2024-04-25T00:31:30.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.502+0000 7f6407fff640 1 -- 172.21.15.116:0/3014481845 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1248816894 0 0) 0x7f63fc003530 con 0x7f64100c2f50 2024-04-25T00:31:30.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.502+0000 7f6415cdf640 1 -- 172.21.15.116:0/3014481845 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6410150d20 msgr2=0x7f64100c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:31:30.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.502+0000 7f6415cdf640 1 --2- 172.21.15.116:0/3014481845 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6410150d20 0x7f64100c2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:30.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.502+0000 7f6415cdf640 1 -- 172.21.15.116:0/3014481845 >> v1:172.21.15.116:6789/0 conn(0x7f64100c2f50 legacy=0x7f64101538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:30.503 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.502+0000 7f6415cdf640 1 -- 172.21.15.116:0/3014481845 --> v2:172.21.15.174:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64100c1400 con 0x7f64100a94a0 2024-04-25T00:31:30.504 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.502+0000 7f6415cdf640 1 --2- 172.21.15.116:0/3014481845 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64100a94a0 0x7f6410153ff0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f6408009d90 tx=0x7f640801c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d538fc1fcb80ac74 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:30.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.502+0000 7f6407fff640 1 -- 172.21.15.116:0/3014481845 <== mon.1 v2:172.21.15.174:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64080b2070 con 0x7f64100a94a0 2024-04-25T00:31:30.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.502+0000 7f6407fff640 1 -- 172.21.15.116:0/3014481845 <== mon.1 v2:172.21.15.174:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f64080a55a0 con 0x7f64100a94a0 2024-04-25T00:31:30.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.502+0000 7f6407fff640 1 -- 172.21.15.116:0/3014481845 <== mon.1 v2:172.21.15.174:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6400001680 con 0x7f64100a94a0 2024-04-25T00:31:30.505 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.506+0000 7f6417742640 1 -- 172.21.15.116:0/3014481845 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64100a94a0 msgr2=0x7f6410153ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:30.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.506+0000 7f6417742640 1 --2- 172.21.15.116:0/3014481845 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64100a94a0 0x7f6410153ff0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f6408009d90 tx=0x7f640801c890 comp rx=0 tx=0).stop 2024-04-25T00:31:30.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.506+0000 7f6417742640 1 -- 172.21.15.116:0/3014481845 shutdown_connections 2024-04-25T00:31:30.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.506+0000 7f6417742640 1 --2- 172.21.15.116:0/3014481845 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64100a94a0 0x7f6410153ff0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:30.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.506+0000 7f6417742640 1 --2- 172.21.15.116:0/3014481845 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6410150d20 0x7f64100c2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:30.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.506+0000 7f6417742640 1 -- 172.21.15.116:0/3014481845 >> 172.21.15.116:0/3014481845 conn(0x7f64100c3b00 msgr2=0x7f64100c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:31:30.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.506+0000 7f6417742640 1 -- 172.21.15.116:0/3014481845 shutdown_connections 2024-04-25T00:31:30.506 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.506+0000 7f6417742640 1 -- 172.21.15.116:0/3014481845 wait complete. 2024-04-25T00:31:30.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.506+0000 7f6417742640 1 Processor -- start 2024-04-25T00:31:30.508 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.506+0000 7f6417742640 1 -- start start 2024-04-25T00:31:30.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.506+0000 7f6417742640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64100a94a0 0x7f641013f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:30.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.506+0000 7f6417742640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64100c2f50 0x7f641014fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:30.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.506+0000 7f6417742640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6410150d20 0x7f641014f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:30.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.506+0000 7f6417742640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f64101608b0 con 0x7f64100a94a0 2024-04-25T00:31:30.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.506+0000 7f6417742640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6410160730 con 0x7f64100c2f50 2024-04-25T00:31:30.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.506+0000 7f6417742640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6410160a30 con 0x7f6410150d20 2024-04-25T00:31:30.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.506+0000 7f64154de640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64100a94a0 0x7f641013f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:30.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.506+0000 7f64154de640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64100a94a0 0x7f641013f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51218/0 (socket says 172.21.15.116:51218) 2024-04-25T00:31:30.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.506+0000 7f64154de640 1 -- 172.21.15.116:0/2499303111 learned_addr learned my addr 172.21.15.116:0/2499303111 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:31:30.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.506+0000 7f6414cdd640 1 --2- 172.21.15.116:0/2499303111 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64100c2f50 0x7f641014fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:30.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.506+0000 7f6415cdf640 1 --2- 172.21.15.116:0/2499303111 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6410150d20 0x7f641014f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:30.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.506+0000 7f64154de640 1 -- 172.21.15.116:0/2499303111 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6410150d20 msgr2=0x7f641014f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:30.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.506+0000 7f64154de640 1 --2- 172.21.15.116:0/2499303111 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6410150d20 0x7f641014f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:30.509 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.506+0000 7f64154de640 1 -- 172.21.15.116:0/2499303111 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64100c2f50 msgr2=0x7f641014fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:30.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.506+0000 7f64154de640 1 --2- 172.21.15.116:0/2499303111 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64100c2f50 0x7f641014fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:30.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.506+0000 7f64154de640 1 -- 172.21.15.116:0/2499303111 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64101f1a70 con 0x7f64100a94a0 2024-04-25T00:31:30.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.506+0000 7f64154de640 1 --2- 172.21.15.116:0/2499303111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64100a94a0 0x7f641013f370 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f640c00e870 tx=0x7f640c00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:30.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.506+0000 7f6405ffb640 1 -- 172.21.15.116:0/2499303111 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f640c017070 con 0x7f64100a94a0 2024-04-25T00:31:30.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.506+0000 7f6405ffb640 1 -- 172.21.15.116:0/2499303111 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f640c00c940 con 0x7f64100a94a0 2024-04-25T00:31:30.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.506+0000 7f6417742640 1 -- 172.21.15.116:0/2499303111 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64101f1d10 con 0x7f64100a94a0 2024-04-25T00:31:30.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.506+0000 7f6405ffb640 1 -- 172.21.15.116:0/2499303111 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f640c00cd10 con 0x7f64100a94a0 2024-04-25T00:31:30.510 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.506+0000 7f6417742640 1 -- 172.21.15.116:0/2499303111 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64101f22d0 con 0x7f64100a94a0 2024-04-25T00:31:30.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.510+0000 7f6417742640 1 -- 172.21.15.116:0/2499303111 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6410068350 con 0x7f64100a94a0 2024-04-25T00:31:30.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.510+0000 7f6405ffb640 1 -- 172.21.15.116:0/2499303111 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f640c01a070 con 0x7f64100a94a0 2024-04-25T00:31:30.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.510+0000 7f6405ffb640 1 --2- 172.21.15.116:0/2499303111 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f63f40678c0 0x7f63f4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:30.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.510+0000 7f6405ffb640 1 -- 172.21.15.116:0/2499303111 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 10708+0+0 (secure 0 0 0) 0x7f640c086f90 con 0x7f64100a94a0 2024-04-25T00:31:30.511 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.510+0000 7f6414cdd640 1 --2- 172.21.15.116:0/2499303111 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f63f40678c0 0x7f63f4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:30.512 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.510+0000 7f6414cdd640 1 --2- 172.21.15.116:0/2499303111 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f63f40678c0 0x7f63f4069d80 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f64100c6fd0 tx=0x7f6400009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:30.514 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.514+0000 7f6405ffb640 1 -- 172.21.15.116:0/2499303111 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f640c0566f0 con 0x7f64100a94a0 2024-04-25T00:31:30.696 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.694+0000 7f6417742640 1 -- 172.21.15.116:0/2499303111 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f64100c34f0 con 0x7f64100a94a0 2024-04-25T00:31:30.697 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.694+0000 7f6405ffb640 1 -- 172.21.15.116:0/2499303111 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 243 v243) ==== 95+0+5661 (secure 0 0 0) 0x7f640c055ff0 con 0x7f64100a94a0 2024-04-25T00:31:30.697 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:31:30.697 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":243,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":9027,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/426062493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":426062493},{"type":"v1","addr":"172.21.15.116:6835","nonce":426062493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":242}],"filesystems":[{"mdsmap":{"epoch":243,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:04.678737+0000","modified":"2024-04-25T00:31:30.022889+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"a","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":1},{"mdsmap":{"epoch":241,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:30:19.111393+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6764},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6764":{"gid":6764,"name":"d","rank":0,"incarnation":238,"state":"up:active","state_seq":8,"addr":"172.21.15.174:6837/3873739116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3873739116},{"type":"v1","addr":"172.21.15.174:6837","nonce":3873739116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6764,"qdb_cluster":[6764]},"id":2}]} 2024-04-25T00:31:30.697 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 243 2024-04-25T00:31:30.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.698+0000 7f6417742640 1 -- 172.21.15.116:0/2499303111 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f63f40678c0 msgr2=0x7f63f4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:30.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.698+0000 7f6417742640 1 --2- 172.21.15.116:0/2499303111 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f63f40678c0 0x7f63f4069d80 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f64100c6fd0 tx=0x7f6400009040 comp rx=0 tx=0).stop 2024-04-25T00:31:30.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.698+0000 7f6417742640 1 -- 172.21.15.116:0/2499303111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64100a94a0 msgr2=0x7f641013f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:30.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.698+0000 7f6417742640 1 --2- 172.21.15.116:0/2499303111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64100a94a0 0x7f641013f370 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f640c00e870 tx=0x7f640c00ec70 comp rx=0 tx=0).stop 2024-04-25T00:31:30.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.698+0000 7f6417742640 1 -- 172.21.15.116:0/2499303111 shutdown_connections 2024-04-25T00:31:30.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.698+0000 7f6417742640 1 --2- 172.21.15.116:0/2499303111 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f63f40678c0 0x7f63f4069d80 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:30.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.698+0000 7f6417742640 1 --2- 172.21.15.116:0/2499303111 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6410150d20 0x7f641014f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:30.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.698+0000 7f6417742640 1 --2- 172.21.15.116:0/2499303111 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f64100c2f50 0x7f641014fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:30.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.698+0000 7f6417742640 1 --2- 172.21.15.116:0/2499303111 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f64100a94a0 0x7f641013f370 unknown :-1 s=CLOSED pgs=1884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:30.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.698+0000 7f6417742640 1 -- 172.21.15.116:0/2499303111 >> 172.21.15.116:0/2499303111 conn(0x7f64100c3b00 msgr2=0x7f64100acac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:31:30.699 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.698+0000 7f6417742640 1 -- 172.21.15.116:0/2499303111 shutdown_connections 2024-04-25T00:31:30.700 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:30.698+0000 7f6417742640 1 -- 172.21.15.116:0/2499303111 wait complete. 2024-04-25T00:31:30.717 INFO:tasks.mds_thrash.fs.[a]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 243, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:04.678737+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 243, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [ 0 ], "flags": 19, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": false, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "a", "in": [ 0 ], "info": {}, "last_failure": 0, "last_failure_osd_epoch": 75, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2024-04-25T00:31:30.022889+0000", "qdb_cluster": [], "qdb_leader": 0, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": {} } }, { "id": 2, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2024-04-25T00:01:18.825887+0000", "damaged": [], "data_pools": [ 5 ], "enabled": true, "epoch": 241, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "balance_automate": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "b", "in": [ 0 ], "info": { "gid_6764": { "addr": "172.21.15.174:6837/3873739116", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6836", "nonce": 3873739116, "type": "v2" }, { "addr": "172.21.15.174:6837", "nonce": 3873739116, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 6764, "incarnation": 238, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:active", "state_seq": 8 } }, "last_failure": 0, "last_failure_osd_epoch": 74, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 4, "modified": "2024-04-25T00:30:19.111393+0000", "qdb_cluster": [ 6764 ], "qdb_leader": 6764, "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6764 } } } ], "standbys": [ { "addr": "172.21.15.174:6835/4112145782", "addrs": { "addrvec": [ { "addr": "172.21.15.174:6834", "nonce": 4112145782, "type": "v2" }, { "addr": "172.21.15.174:6835", "nonce": 4112145782, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 112, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 5582, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.116:6835/426062493", "addrs": { "addrvec": [ { "addr": "172.21.15.116:6834", "nonce": 426062493, "type": "v2" }, { "addr": "172.21.15.116:6835", "nonce": 426062493, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_12": "quiesce subvolumes", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 242, "export_targets": [], "features": 4540701547738038271, "flags": 0, "gid": 9027, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2024-04-25T00:31:31.057 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.054+0000 7f315d7fa640 1 -- 172.21.15.116:0/3790101508 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "a", "yes_i_really_mean_it": true}]=0 v244) ==== 99+0+0 (secure 0 0 0) 0x7f315400d3e0 con 0x7f31600ab9f0 2024-04-25T00:31:31.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.058+0000 7f3166bd4640 1 -- 172.21.15.116:0/3790101508 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f314c0678c0 msgr2=0x7f314c069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:31.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.058+0000 7f3166bd4640 1 --2- 172.21.15.116:0/3790101508 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f314c0678c0 0x7f314c069d80 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f3158001f10 tx=0x7f3158001e80 comp rx=0 tx=0).stop 2024-04-25T00:31:31.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.058+0000 7f3166bd4640 1 -- 172.21.15.116:0/3790101508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31600ab9f0 msgr2=0x7f31601eaf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:31.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.058+0000 7f3166bd4640 1 --2- 172.21.15.116:0/3790101508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31600ab9f0 0x7f31601eaf00 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f315400f580 tx=0x7f315400fa40 comp rx=0 tx=0).stop 2024-04-25T00:31:31.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.058+0000 7f3166bd4640 1 -- 172.21.15.116:0/3790101508 shutdown_connections 2024-04-25T00:31:31.060 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.058+0000 7f3166bd4640 1 --2- 172.21.15.116:0/3790101508 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f314c0678c0 0x7f314c069d80 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:31.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.058+0000 7f3166bd4640 1 --2- 172.21.15.116:0/3790101508 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f31601eb440 0x7f3160155930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:31.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.058+0000 7f3166bd4640 1 --2- 172.21.15.116:0/3790101508 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f31600ab9f0 0x7f31601eaf00 unknown :-1 s=CLOSED pgs=1882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:31.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.058+0000 7f3166bd4640 1 --2- 172.21.15.116:0/3790101508 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f3148004d50 0x7f31601e07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:31.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.058+0000 7f3166bd4640 1 -- 172.21.15.116:0/3790101508 >> 172.21.15.116:0/3790101508 conn(0x7f31600c7a90 msgr2=0x7f31600c0ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:31:31.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.058+0000 7f3166bd4640 1 -- 172.21.15.116:0/3790101508 shutdown_connections 2024-04-25T00:31:31.061 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.058+0000 7f3166bd4640 1 -- 172.21.15.116:0/3790101508 wait complete. 2024-04-25T00:31:31.081 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm a_metadata a_metadata --yes-i-really-really-mean-it 2024-04-25T00:31:31.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.210+0000 7f7e173a8640 1 Processor -- start 2024-04-25T00:31:31.212 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.210+0000 7f7e173a8640 1 -- start start 2024-04-25T00:31:31.223 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.210+0000 7f7e173a8640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f7e1006da00 0x7f7e1006dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:31.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.210+0000 7f7e173a8640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7e100c7f50 0x7f7e10159bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:31.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.210+0000 7f7e173a8640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e1015d340 con 0x7f7e100715f0 2024-04-25T00:31:31.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.210+0000 7f7e173a8640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f7e100c89f0 con 0x7f7e1006da00 2024-04-25T00:31:31.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.210+0000 7f7e173a8640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f7e1015d670 con 0x7f7e100c7f50 2024-04-25T00:31:31.224 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.210+0000 7f7e15945640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f7e100715f0 0x7f7e1015db00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:46860/0 (socket says 172.21.15.116:46860) 2024-04-25T00:31:31.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.214+0000 7f7e15945640 1 -- 172.21.15.116:0/1471236646 learned_addr learned my addr 172.21.15.116:0/1471236646 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:31:31.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.214+0000 7f7e15144640 1 --2- 172.21.15.116:0/1471236646 >> v2:172.21.15.174:3300/0 conn(0x7f7e1006da00 0x7f7e1006dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:31.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.214+0000 7f7e14943640 1 --2- 172.21.15.116:0/1471236646 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7e100c7f50 0x7f7e10159bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:31.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.214+0000 7f7e07fff640 1 -- 172.21.15.116:0/1471236646 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1992669796 0 0) 0x7f7e1015d340 con 0x7f7e100715f0 2024-04-25T00:31:31.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.214+0000 7f7e07fff640 1 -- 172.21.15.116:0/1471236646 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7dfc003530 con 0x7f7e100715f0 2024-04-25T00:31:31.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.214+0000 7f7e07fff640 1 -- 172.21.15.116:0/1471236646 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 831940748 0 0) 0x7f7dfc003530 con 0x7f7e100715f0 2024-04-25T00:31:31.225 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.214+0000 7f7e07fff640 1 -- 172.21.15.116:0/1471236646 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7e100c7f50 msgr2=0x7f7e10159bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:31:31.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.214+0000 7f7e07fff640 1 --2- 172.21.15.116:0/1471236646 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7e100c7f50 0x7f7e10159bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:31.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.214+0000 7f7e07fff640 1 -- 172.21.15.116:0/1471236646 >> v2:172.21.15.174:3300/0 conn(0x7f7e1006da00 msgr2=0x7f7e1006dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:31:31.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.214+0000 7f7e07fff640 1 --2- 172.21.15.116:0/1471236646 >> v2:172.21.15.174:3300/0 conn(0x7f7e1006da00 0x7f7e1006dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:31.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.214+0000 7f7e07fff640 1 -- 172.21.15.116:0/1471236646 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e100c8430 con 0x7f7e100715f0 2024-04-25T00:31:31.226 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.214+0000 7f7e07fff640 1 -- 172.21.15.116:0/1471236646 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f7e08001a20 con 0x7f7e100715f0 2024-04-25T00:31:31.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.214+0000 7f7e07fff640 1 -- 172.21.15.116:0/1471236646 >> v1:172.21.15.116:6789/0 conn(0x7f7e100715f0 legacy=0x7f7e1015db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:31.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.214+0000 7f7e07fff640 1 --2- 172.21.15.116:0/1471236646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7dfc0039c0 0x7f7dfc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:31.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.214+0000 7f7e07fff640 1 --2- 172.21.15.116:0/1471236646 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7dfc004810 0x7f7dfc004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:31.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.214+0000 7f7e07fff640 1 --2- 172.21.15.116:0/1471236646 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7e100c7f50 0x7f7dfc0051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:31.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.214+0000 7f7e07fff640 1 -- 172.21.15.116:0/1471236646 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7e100c89f0 con 0x7f7dfc0039c0 2024-04-25T00:31:31.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.214+0000 7f7e07fff640 1 -- 172.21.15.116:0/1471236646 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7e1015d670 con 0x7f7dfc004810 2024-04-25T00:31:31.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.214+0000 7f7e07fff640 1 -- 172.21.15.116:0/1471236646 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f7dfc00ba50 con 0x7f7e100c7f50 2024-04-25T00:31:31.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.214+0000 7f7e15144640 1 --2- 172.21.15.116:0/1471236646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7dfc0039c0 0x7f7dfc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:31.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.214+0000 7f7e14943640 1 --2- 172.21.15.116:0/1471236646 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7dfc004810 0x7f7dfc004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:31.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.214+0000 7f7e15945640 1 --2- 172.21.15.116:0/1471236646 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7e100c7f50 0x7f7dfc0051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:31.227 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.214+0000 7f7e15144640 1 -- 172.21.15.116:0/1471236646 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7e100c7f50 msgr2=0x7f7dfc0051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:31:31.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.214+0000 7f7e15144640 1 --2- 172.21.15.116:0/1471236646 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7e100c7f50 0x7f7dfc0051d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:31.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.214+0000 7f7e15144640 1 -- 172.21.15.116:0/1471236646 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7dfc004810 msgr2=0x7f7dfc004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:31:31.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.214+0000 7f7e15144640 1 --2- 172.21.15.116:0/1471236646 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7dfc004810 0x7f7dfc004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:31.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.214+0000 7f7e15144640 1 -- 172.21.15.116:0/1471236646 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7dfc003530 con 0x7f7dfc0039c0 2024-04-25T00:31:31.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.214+0000 7f7e15144640 1 --2- 172.21.15.116:0/1471236646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7dfc0039c0 0x7f7dfc003de0 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c009ae0 tx=0x7f7e0c01ca80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff828971a814e92e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:31.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.214+0000 7f7e07fff640 1 -- 172.21.15.116:0/1471236646 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e0c0b2070 con 0x7f7dfc0039c0 2024-04-25T00:31:31.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.214+0000 7f7e07fff640 1 -- 172.21.15.116:0/1471236646 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f7e0c0a5670 con 0x7f7dfc0039c0 2024-04-25T00:31:31.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.214+0000 7f7e07fff640 1 -- 172.21.15.116:0/1471236646 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e0c0096b0 con 0x7f7dfc0039c0 2024-04-25T00:31:31.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.218+0000 7f7e173a8640 1 -- 172.21.15.116:0/1471236646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7dfc0039c0 msgr2=0x7f7dfc003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:31.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.218+0000 7f7e173a8640 1 --2- 172.21.15.116:0/1471236646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7dfc0039c0 0x7f7dfc003de0 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c009ae0 tx=0x7f7e0c01ca80 comp rx=0 tx=0).stop 2024-04-25T00:31:31.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.218+0000 7f7e15945640 1 -- 172.21.15.116:0/1471236646 reap_dead start 2024-04-25T00:31:31.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.218+0000 7f7e173a8640 1 -- 172.21.15.116:0/1471236646 shutdown_connections 2024-04-25T00:31:31.228 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.218+0000 7f7e173a8640 1 -- 172.21.15.116:0/1471236646 >> 172.21.15.116:0/1471236646 conn(0x7f7e100aacb0 msgr2=0x7f7e100ab0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:31:31.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.218+0000 7f7e173a8640 1 -- 172.21.15.116:0/1471236646 shutdown_connections 2024-04-25T00:31:31.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.218+0000 7f7e173a8640 1 -- 172.21.15.116:0/1471236646 wait complete. 2024-04-25T00:31:31.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.218+0000 7f7e173a8640 1 Processor -- start 2024-04-25T00:31:31.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.218+0000 7f7e173a8640 1 -- start start 2024-04-25T00:31:31.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.218+0000 7f7e173a8640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7dfc0039c0 0x7f7e10156250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:31.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.218+0000 7f7e173a8640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e10070cf0 0x7f7e100c56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:31.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.218+0000 7f7e173a8640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e101f5930 0x7f7e101f5de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:31.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.218+0000 7f7e173a8640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f7e10157710 con 0x7f7e10070cf0 2024-04-25T00:31:31.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.218+0000 7f7e173a8640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f7e10157590 con 0x7f7e101f5930 2024-04-25T00:31:31.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.218+0000 7f7e173a8640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f7e1015e2a0 con 0x7f7dfc0039c0 2024-04-25T00:31:31.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.218+0000 7f7e14943640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e10070cf0 0x7f7e100c56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:31.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.218+0000 7f7e14943640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e10070cf0 0x7f7e100c56d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51240/0 (socket says 172.21.15.116:51240) 2024-04-25T00:31:31.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.218+0000 7f7e14943640 1 -- 172.21.15.116:0/4083035621 learned_addr learned my addr 172.21.15.116:0/4083035621 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:31:31.229 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.218+0000 7f7e15945640 1 --2- 172.21.15.116:0/4083035621 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e101f5930 0x7f7e101f5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:31.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.218+0000 7f7e14943640 1 -- 172.21.15.116:0/4083035621 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7dfc0039c0 msgr2=0x7f7e10156250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:31.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.218+0000 7f7e14943640 1 --2- 172.21.15.116:0/4083035621 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7dfc0039c0 0x7f7e10156250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:31.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.218+0000 7f7e14943640 1 -- 172.21.15.116:0/4083035621 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e101f5930 msgr2=0x7f7e101f5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:31.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.218+0000 7f7e14943640 1 --2- 172.21.15.116:0/4083035621 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e101f5930 0x7f7e101f5de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:31.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.218+0000 7f7e14943640 1 -- 172.21.15.116:0/4083035621 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e10070160 con 0x7f7e10070cf0 2024-04-25T00:31:31.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.218+0000 7f7e14943640 1 --2- 172.21.15.116:0/4083035621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e10070cf0 0x7f7e100c56d0 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7f7e000115e0 tx=0x7f7e00011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:31.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.218+0000 7f7e05ffb640 1 -- 172.21.15.116:0/4083035621 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e0000f8f0 con 0x7f7e10070cf0 2024-04-25T00:31:31.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.218+0000 7f7e05ffb640 1 -- 172.21.15.116:0/4083035621 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f7e0000fa90 con 0x7f7e10070cf0 2024-04-25T00:31:31.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.218+0000 7f7e05ffb640 1 -- 172.21.15.116:0/4083035621 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e0000fe60 con 0x7f7e10070cf0 2024-04-25T00:31:31.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.218+0000 7f7e173a8640 1 -- 172.21.15.116:0/4083035621 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e10160920 con 0x7f7e10070cf0 2024-04-25T00:31:31.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.218+0000 7f7e173a8640 1 -- 172.21.15.116:0/4083035621 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e100ac570 con 0x7f7e10070cf0 2024-04-25T00:31:31.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.218+0000 7f7e173a8640 1 -- 172.21.15.116:0/4083035621 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e10068a60 con 0x7f7e10070cf0 2024-04-25T00:31:31.230 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.222+0000 7f7e05ffb640 1 -- 172.21.15.116:0/4083035621 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f7e00015460 con 0x7f7e10070cf0 2024-04-25T00:31:31.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.222+0000 7f7e05ffb640 1 --2- 172.21.15.116:0/4083035621 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7df40678f0 0x7f7df4069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:31.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.222+0000 7f7e05ffb640 1 -- 172.21.15.116:0/4083035621 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 10708+0+0 (secure 0 0 0) 0x7f7e000879b0 con 0x7f7e10070cf0 2024-04-25T00:31:31.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.222+0000 7f7e15144640 1 --2- 172.21.15.116:0/4083035621 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7df40678f0 0x7f7df4069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:31.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.222+0000 7f7e15144640 1 --2- 172.21.15.116:0/4083035621 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7df40678f0 0x7f7df4069db0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c0af000 tx=0x7f7e0c004740 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:31.231 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.226+0000 7f7e05ffb640 1 -- 172.21.15.116:0/4083035621 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f7e00019030 con 0x7f7e10070cf0 2024-04-25T00:31:31.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:31.386+0000 7f7e173a8640 1 -- 172.21.15.116:0/4083035621 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "a_metadata", "pool2": "a_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f7e1015e4f0 con 0x7f7e10070cf0 2024-04-25T00:31:32.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.066+0000 7f7e05ffb640 1 -- 172.21.15.116:0/4083035621 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "a_metadata", "pool2": "a_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'a_metadata' removed v76) ==== 166+0+0 (secure 0 0 0) 0x7f7e00056ac0 con 0x7f7e10070cf0 2024-04-25T00:31:32.066 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'a_metadata' removed 2024-04-25T00:31:32.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.066+0000 7f7e173a8640 1 -- 172.21.15.116:0/4083035621 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7df40678f0 msgr2=0x7f7df4069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:32.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.066+0000 7f7e173a8640 1 --2- 172.21.15.116:0/4083035621 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7df40678f0 0x7f7df4069db0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c0af000 tx=0x7f7e0c004740 comp rx=0 tx=0).stop 2024-04-25T00:31:32.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.066+0000 7f7e173a8640 1 -- 172.21.15.116:0/4083035621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e10070cf0 msgr2=0x7f7e100c56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:32.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.066+0000 7f7e173a8640 1 --2- 172.21.15.116:0/4083035621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e10070cf0 0x7f7e100c56d0 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7f7e000115e0 tx=0x7f7e00011aa0 comp rx=0 tx=0).stop 2024-04-25T00:31:32.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.066+0000 7f7e173a8640 1 -- 172.21.15.116:0/4083035621 shutdown_connections 2024-04-25T00:31:32.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.066+0000 7f7e173a8640 1 --2- 172.21.15.116:0/4083035621 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f7df40678f0 0x7f7df4069db0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:32.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.066+0000 7f7e173a8640 1 --2- 172.21.15.116:0/4083035621 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f7e101f5930 0x7f7e101f5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:32.069 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.066+0000 7f7e173a8640 1 --2- 172.21.15.116:0/4083035621 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f7e10070cf0 0x7f7e100c56d0 unknown :-1 s=CLOSED pgs=1887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:32.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.066+0000 7f7e173a8640 1 --2- 172.21.15.116:0/4083035621 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f7dfc0039c0 0x7f7e10156250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:32.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.066+0000 7f7e173a8640 1 -- 172.21.15.116:0/4083035621 >> 172.21.15.116:0/4083035621 conn(0x7f7e100aacb0 msgr2=0x7f7e100aa430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:31:32.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.066+0000 7f7e173a8640 1 -- 172.21.15.116:0/4083035621 shutdown_connections 2024-04-25T00:31:32.070 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.066+0000 7f7e173a8640 1 -- 172.21.15.116:0/4083035621 wait complete. 2024-04-25T00:31:32.100 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm a_data a_data --yes-i-really-really-mean-it 2024-04-25T00:31:32.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.234+0000 7efc0cfcc640 1 Processor -- start 2024-04-25T00:31:32.234 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.234+0000 7efc0cfcc640 1 -- start start 2024-04-25T00:31:32.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.234+0000 7efc0cfcc640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7efc081598d0 0x7efc080c6710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:32.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.234+0000 7efc0cfcc640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efc08070280 0x7efc080706f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:32.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.234+0000 7efc0cfcc640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc08158e10 con 0x7efc080c57a0 2024-04-25T00:31:32.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.234+0000 7efc0cfcc640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7efc080c5c30 con 0x7efc081598d0 2024-04-25T00:31:32.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.234+0000 7efc0cfcc640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7efc080c85c0 con 0x7efc08070280 2024-04-25T00:31:32.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.234+0000 7efc05d9b640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7efc080c57a0 0x7efc080c9070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:46866/0 (socket says 172.21.15.116:46866) 2024-04-25T00:31:32.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.234+0000 7efc05d9b640 1 -- 172.21.15.116:0/1356116192 learned_addr learned my addr 172.21.15.116:0/1356116192 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:31:32.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.234+0000 7efc0659c640 1 --2- 172.21.15.116:0/1356116192 >> v2:172.21.15.174:3300/0 conn(0x7efc081598d0 0x7efc080c6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:32.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.234+0000 7efc0559a640 1 -- 172.21.15.116:0/1356116192 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1605939345 0 0) 0x7efc08158e10 con 0x7efc080c57a0 2024-04-25T00:31:32.237 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.234+0000 7efc06d9d640 1 --2- 172.21.15.116:0/1356116192 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efc08070280 0x7efc080706f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:32.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.234+0000 7efc0559a640 1 -- 172.21.15.116:0/1356116192 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efbf4003530 con 0x7efc080c57a0 2024-04-25T00:31:32.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.234+0000 7efc0559a640 1 -- 172.21.15.116:0/1356116192 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 332338427 0 0) 0x7efbf4003530 con 0x7efc080c57a0 2024-04-25T00:31:32.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.234+0000 7efc0559a640 1 -- 172.21.15.116:0/1356116192 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efc08070280 msgr2=0x7efc080706f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:31:32.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.234+0000 7efc0559a640 1 --2- 172.21.15.116:0/1356116192 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efc08070280 0x7efc080706f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:32.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.234+0000 7efc0559a640 1 -- 172.21.15.116:0/1356116192 >> v2:172.21.15.174:3300/0 conn(0x7efc081598d0 msgr2=0x7efc080c6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:31:32.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.234+0000 7efc0559a640 1 --2- 172.21.15.116:0/1356116192 >> v2:172.21.15.174:3300/0 conn(0x7efc081598d0 0x7efc080c6710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:32.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.234+0000 7efc0559a640 1 -- 172.21.15.116:0/1356116192 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc080c88a0 con 0x7efc080c57a0 2024-04-25T00:31:32.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.234+0000 7efc0559a640 1 -- 172.21.15.116:0/1356116192 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7efbf0001680 con 0x7efc080c57a0 2024-04-25T00:31:32.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.234+0000 7efc0559a640 1 -- 172.21.15.116:0/1356116192 >> v1:172.21.15.116:6789/0 conn(0x7efc080c57a0 legacy=0x7efc080c9070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:32.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.234+0000 7efc0559a640 1 --2- 172.21.15.116:0/1356116192 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efc08070280 0x7efbf40039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:32.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.234+0000 7efc0559a640 1 --2- 172.21.15.116:0/1356116192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efbf4003f40 0x7efbf4004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:32.238 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.234+0000 7efc0559a640 1 --2- 172.21.15.116:0/1356116192 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efbf4004d50 0x7efbf40051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:32.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.234+0000 7efc0559a640 1 -- 172.21.15.116:0/1356116192 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7efc080c5c30 con 0x7efbf4003f40 2024-04-25T00:31:32.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.234+0000 7efc0559a640 1 -- 172.21.15.116:0/1356116192 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7efc080c85c0 con 0x7efbf4004d50 2024-04-25T00:31:32.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.234+0000 7efc0559a640 1 -- 172.21.15.116:0/1356116192 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7efbf400ba20 con 0x7efc08070280 2024-04-25T00:31:32.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.234+0000 7efc05d9b640 1 --2- 172.21.15.116:0/1356116192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efbf4003f40 0x7efbf4004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:32.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.234+0000 7efc0659c640 1 --2- 172.21.15.116:0/1356116192 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efc08070280 0x7efbf40039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:32.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.234+0000 7efc05d9b640 1 -- 172.21.15.116:0/1356116192 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efc08070280 msgr2=0x7efbf40039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:31:32.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.234+0000 7efc05d9b640 1 --2- 172.21.15.116:0/1356116192 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efc08070280 0x7efbf40039c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:32.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.234+0000 7efc05d9b640 1 -- 172.21.15.116:0/1356116192 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efbf4004d50 msgr2=0x7efbf40051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:31:32.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.234+0000 7efc05d9b640 1 --2- 172.21.15.116:0/1356116192 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efbf4004d50 0x7efbf40051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:32.239 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.234+0000 7efc05d9b640 1 -- 172.21.15.116:0/1356116192 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efbf4003530 con 0x7efbf4003f40 2024-04-25T00:31:32.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.238+0000 7efc05d9b640 1 --2- 172.21.15.116:0/1356116192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efbf4003f40 0x7efbf4004810 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7efbf000c270 tx=0x7efbf001cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eab591c54217b4d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:32.242 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.238+0000 7efc0559a640 1 -- 172.21.15.116:0/1356116192 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbf0007d40 con 0x7efbf4003f40 2024-04-25T00:31:32.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.238+0000 7efc0559a640 1 -- 172.21.15.116:0/1356116192 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7efbf00a9020 con 0x7efbf4003f40 2024-04-25T00:31:32.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.238+0000 7efc0559a640 1 -- 172.21.15.116:0/1356116192 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbf0005b20 con 0x7efbf4003f40 2024-04-25T00:31:32.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.238+0000 7efc0cfcc640 1 -- 172.21.15.116:0/1356116192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efbf4003f40 msgr2=0x7efbf4004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:32.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.238+0000 7efc0cfcc640 1 --2- 172.21.15.116:0/1356116192 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efbf4003f40 0x7efbf4004810 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7efbf000c270 tx=0x7efbf001cca0 comp rx=0 tx=0).stop 2024-04-25T00:31:32.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.238+0000 7efc06d9d640 1 -- 172.21.15.116:0/1356116192 reap_dead start 2024-04-25T00:31:32.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.238+0000 7efc0cfcc640 1 -- 172.21.15.116:0/1356116192 shutdown_connections 2024-04-25T00:31:32.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.238+0000 7efc0cfcc640 1 -- 172.21.15.116:0/1356116192 >> 172.21.15.116:0/1356116192 conn(0x7efc0806ef80 msgr2=0x7efc0806f390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:31:32.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.238+0000 7efc0cfcc640 1 -- 172.21.15.116:0/1356116192 shutdown_connections 2024-04-25T00:31:32.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.238+0000 7efc0cfcc640 1 -- 172.21.15.116:0/1356116192 wait complete. 2024-04-25T00:31:32.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.238+0000 7efc0cfcc640 1 Processor -- start 2024-04-25T00:31:32.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.238+0000 7efc0cfcc640 1 -- start start 2024-04-25T00:31:32.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.238+0000 7efc0cfcc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efbf4003f40 0x7efc081f22f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:32.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.238+0000 7efc0cfcc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efc0814e870 0x7efc081ec350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:32.243 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.238+0000 7efc0cfcc640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efc08150590 0x7efc081eec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:32.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.238+0000 7efc0cfcc640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7efc080c9900 con 0x7efbf4003f40 2024-04-25T00:31:32.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.238+0000 7efc0cfcc640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7efc080c9780 con 0x7efc0814e870 2024-04-25T00:31:32.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.238+0000 7efc0cfcc640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7efc08160140 con 0x7efc08150590 2024-04-25T00:31:32.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.238+0000 7efc0659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efbf4003f40 0x7efc081f22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:32.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.238+0000 7efc0659c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efbf4003f40 0x7efc081f22f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51270/0 (socket says 172.21.15.116:51270) 2024-04-25T00:31:32.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.238+0000 7efc0659c640 1 -- 172.21.15.116:0/54003712 learned_addr learned my addr 172.21.15.116:0/54003712 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:31:32.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.242+0000 7efc05d9b640 1 --2- 172.21.15.116:0/54003712 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efc0814e870 0x7efc081ec350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:32.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.242+0000 7efc0659c640 1 -- 172.21.15.116:0/54003712 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efc08150590 msgr2=0x7efc081eec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:32.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.242+0000 7efc06d9d640 1 --2- 172.21.15.116:0/54003712 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efc08150590 0x7efc081eec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:32.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.242+0000 7efc0659c640 1 --2- 172.21.15.116:0/54003712 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efc08150590 0x7efc081eec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:32.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.242+0000 7efc0659c640 1 -- 172.21.15.116:0/54003712 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efc0814e870 msgr2=0x7efc081ec350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:32.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.242+0000 7efc0659c640 1 --2- 172.21.15.116:0/54003712 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efc0814e870 0x7efc081ec350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:32.244 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.242+0000 7efc0659c640 1 -- 172.21.15.116:0/54003712 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc0815a470 con 0x7efbf4003f40 2024-04-25T00:31:32.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.242+0000 7efc05d9b640 1 --2- 172.21.15.116:0/54003712 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efc0814e870 0x7efc081ec350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:31:32.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.242+0000 7efc0659c640 1 --2- 172.21.15.116:0/54003712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efbf4003f40 0x7efc081f22f0 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7efc0000f400 tx=0x7efc0000f8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:32.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.242+0000 7efbfeffd640 1 -- 172.21.15.116:0/54003712 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc00016620 con 0x7efbf4003f40 2024-04-25T00:31:32.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.242+0000 7efbfeffd640 1 -- 172.21.15.116:0/54003712 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7efc0001b070 con 0x7efbf4003f40 2024-04-25T00:31:32.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.242+0000 7efbfeffd640 1 -- 172.21.15.116:0/54003712 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc00016a40 con 0x7efbf4003f40 2024-04-25T00:31:32.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.242+0000 7efc0cfcc640 1 -- 172.21.15.116:0/54003712 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc081ee480 con 0x7efbf4003f40 2024-04-25T00:31:32.245 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.242+0000 7efc0cfcc640 1 -- 172.21.15.116:0/54003712 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc0815a750 con 0x7efbf4003f40 2024-04-25T00:31:32.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.242+0000 7efc0cfcc640 1 -- 172.21.15.116:0/54003712 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc08068a60 con 0x7efbf4003f40 2024-04-25T00:31:32.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.242+0000 7efbfeffd640 1 -- 172.21.15.116:0/54003712 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7efc00016be0 con 0x7efbf4003f40 2024-04-25T00:31:32.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.242+0000 7efbfeffd640 1 --2- 172.21.15.116:0/54003712 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efbe80678c0 0x7efbe8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:32.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.242+0000 7efbfeffd640 1 -- 172.21.15.116:0/54003712 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 10280+0+0 (secure 0 0 0) 0x7efc00087df0 con 0x7efbf4003f40 2024-04-25T00:31:32.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.242+0000 7efc05d9b640 1 --2- 172.21.15.116:0/54003712 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efbe80678c0 0x7efbe8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:32.246 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.246+0000 7efc05d9b640 1 --2- 172.21.15.116:0/54003712 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efbe80678c0 0x7efbe8069d80 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7efbf00aa000 tx=0x7efc080c8e60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:32.248 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.246+0000 7efbfeffd640 1 -- 172.21.15.116:0/54003712 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7efc00057670 con 0x7efbf4003f40 2024-04-25T00:31:32.409 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.410+0000 7efc0cfcc640 1 -- 172.21.15.116:0/54003712 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "a_data", "pool2": "a_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7efc080c64f0 con 0x7efbf4003f40 2024-04-25T00:31:32.718 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:31:32.849 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.850+0000 7f5ac84e1640 1 Processor -- start 2024-04-25T00:31:32.850 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.850+0000 7f5ac84e1640 1 -- start start 2024-04-25T00:31:32.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.850+0000 7f5ac84e1640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f5ac014c720 0x7f5ac01563d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:32.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.850+0000 7f5ac84e1640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5ac0071680 0x7f5ac0071ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:32.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.850+0000 7f5ac84e1640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ac0071220 con 0x7f5ac00a94a0 2024-04-25T00:31:32.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.850+0000 7f5ac84e1640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f5ac0153460 con 0x7f5ac0071680 2024-04-25T00:31:32.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.850+0000 7f5ac84e1640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f5ac0152420 con 0x7f5ac014c720 2024-04-25T00:31:32.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.850+0000 7f5ac6a7e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5ac0071680 0x7f5ac0071ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:32.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.850+0000 7f5ac6a7e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f5ac0071680 0x7f5ac0071ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51278/0 (socket says 172.21.15.116:51278) 2024-04-25T00:31:32.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.850+0000 7f5ac6a7e640 1 -- 172.21.15.116:0/1133598512 learned_addr learned my addr 172.21.15.116:0/1133598512 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:31:32.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.850+0000 7f5ac627d640 1 --2- 172.21.15.116:0/1133598512 >> v2:172.21.15.174:3300/0 conn(0x7f5ac014c720 0x7f5ac01563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:32.851 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.850+0000 7f5ac6a7e640 1 -- 172.21.15.116:0/1133598512 >> v2:172.21.15.174:3300/0 conn(0x7f5ac014c720 msgr2=0x7f5ac01563d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:31:32.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.850+0000 7f5ac6a7e640 1 --2- 172.21.15.116:0/1133598512 >> v2:172.21.15.174:3300/0 conn(0x7f5ac014c720 0x7f5ac01563d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:32.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.850+0000 7f5ac6a7e640 1 -- 172.21.15.116:0/1133598512 >> v1:172.21.15.174:6789/0 conn(0x7f5ac00a94a0 legacy=0x7f5ac00c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:31:32.852 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.850+0000 7f5ac6a7e640 1 -- 172.21.15.116:0/1133598512 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ac014cb00 con 0x7f5ac0071680 2024-04-25T00:31:32.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.850+0000 7f5ac6a7e640 1 --2- 172.21.15.116:0/1133598512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ac0071680 0x7f5ac0071ab0 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f5ac0070190 tx=0x7f5ab801c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bc1be21ea4e2304 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:32.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.854+0000 7f5ac527b640 1 -- 172.21.15.116:0/1133598512 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab80a57b0 con 0x7f5ac0071680 2024-04-25T00:31:32.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.854+0000 7f5ac527b640 1 -- 172.21.15.116:0/1133598512 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f5ab80a5950 con 0x7f5ac0071680 2024-04-25T00:31:32.853 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.854+0000 7f5ac527b640 1 -- 172.21.15.116:0/1133598512 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab80a5d40 con 0x7f5ac0071680 2024-04-25T00:31:32.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.854+0000 7f5ac84e1640 1 -- 172.21.15.116:0/1133598512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ac0071680 msgr2=0x7f5ac0071ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:32.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.854+0000 7f5ac84e1640 1 --2- 172.21.15.116:0/1133598512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ac0071680 0x7f5ac0071ab0 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f5ac0070190 tx=0x7f5ab801c8f0 comp rx=0 tx=0).stop 2024-04-25T00:31:32.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.854+0000 7f5ac84e1640 1 -- 172.21.15.116:0/1133598512 shutdown_connections 2024-04-25T00:31:32.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.854+0000 7f5ac84e1640 1 --2- 172.21.15.116:0/1133598512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ac0071680 0x7f5ac0071ab0 unknown :-1 s=CLOSED pgs=1891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:32.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.854+0000 7f5ac84e1640 1 --2- 172.21.15.116:0/1133598512 >> v2:172.21.15.174:3300/0 conn(0x7f5ac014c720 0x7f5ac01563d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:32.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.854+0000 7f5ac84e1640 1 -- 172.21.15.116:0/1133598512 >> 172.21.15.116:0/1133598512 conn(0x7f5ac00c3b00 msgr2=0x7f5ac00c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:31:32.854 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.854+0000 7f5ac84e1640 1 -- 172.21.15.116:0/1133598512 shutdown_connections 2024-04-25T00:31:32.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.854+0000 7f5ac84e1640 1 -- 172.21.15.116:0/1133598512 wait complete. 2024-04-25T00:31:32.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.854+0000 7f5ac84e1640 1 Processor -- start 2024-04-25T00:31:32.855 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.854+0000 7f5ac84e1640 1 -- start start 2024-04-25T00:31:32.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.854+0000 7f5ac84e1640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ac0071680 0x7f5ac00c66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:32.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.854+0000 7f5ac84e1640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5ac00a94a0 0x7f5ac01519b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:32.857 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.854+0000 7f5ac84e1640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ac014c720 0x7f5ac014b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:32.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.854+0000 7f5ac84e1640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac00c3120 con 0x7f5ac014c720 2024-04-25T00:31:32.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.854+0000 7f5ac84e1640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f5ac0152f40 con 0x7f5ac0071680 2024-04-25T00:31:32.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.854+0000 7f5ac84e1640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f5ac00c32a0 con 0x7f5ac00a94a0 2024-04-25T00:31:32.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.854+0000 7f5ac6a7e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ac014c720 0x7f5ac014b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:32.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.854+0000 7f5ac6a7e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ac014c720 0x7f5ac014b140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51286/0 (socket says 172.21.15.116:51286) 2024-04-25T00:31:32.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.854+0000 7f5ac6a7e640 1 -- 172.21.15.116:0/164714256 learned_addr learned my addr 172.21.15.116:0/164714256 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:31:32.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.854+0000 7f5ac627d640 1 --2- 172.21.15.116:0/164714256 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ac0071680 0x7f5ac00c66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:32.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.854+0000 7f5ac6a7e640 1 -- 172.21.15.116:0/164714256 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5ac00a94a0 msgr2=0x7f5ac01519b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:32.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.854+0000 7f5ac5a7c640 1 --2- 172.21.15.116:0/164714256 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5ac00a94a0 0x7f5ac01519b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:32.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.854+0000 7f5ac6a7e640 1 --2- 172.21.15.116:0/164714256 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5ac00a94a0 0x7f5ac01519b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:32.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.854+0000 7f5ac6a7e640 1 -- 172.21.15.116:0/164714256 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ac0071680 msgr2=0x7f5ac00c66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:32.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.854+0000 7f5ac6a7e640 1 --2- 172.21.15.116:0/164714256 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ac0071680 0x7f5ac00c66a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:32.858 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.854+0000 7f5ac6a7e640 1 -- 172.21.15.116:0/164714256 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ac01f2730 con 0x7f5ac014c720 2024-04-25T00:31:32.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.854+0000 7f5ac627d640 1 --2- 172.21.15.116:0/164714256 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ac0071680 0x7f5ac00c66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:31:32.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.854+0000 7f5ac6a7e640 1 --2- 172.21.15.116:0/164714256 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ac014c720 0x7f5ac014b140 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f5ab80af040 tx=0x7f5ab8019690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:32.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.854+0000 7f5ab6ffd640 1 -- 172.21.15.116:0/164714256 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab80bc020 con 0x7f5ac014c720 2024-04-25T00:31:32.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.854+0000 7f5ab6ffd640 1 -- 172.21.15.116:0/164714256 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f5ab80b88a0 con 0x7f5ac014c720 2024-04-25T00:31:32.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.854+0000 7f5ac84e1640 1 -- 172.21.15.116:0/164714256 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ac01f1bb0 con 0x7f5ac014c720 2024-04-25T00:31:32.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.854+0000 7f5ab6ffd640 1 -- 172.21.15.116:0/164714256 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ab80b8cb0 con 0x7f5ac014c720 2024-04-25T00:31:32.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.854+0000 7f5ac84e1640 1 -- 172.21.15.116:0/164714256 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ac01f2170 con 0x7f5ac014c720 2024-04-25T00:31:32.859 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.858+0000 7f5ac84e1640 1 -- 172.21.15.116:0/164714256 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ac0068350 con 0x7f5ac014c720 2024-04-25T00:31:32.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.858+0000 7f5ab6ffd640 1 -- 172.21.15.116:0/164714256 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5ab80bf020 con 0x7f5ac014c720 2024-04-25T00:31:32.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.858+0000 7f5ab6ffd640 1 --2- 172.21.15.116:0/164714256 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5aa40678c0 0x7f5aa4069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:32.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.858+0000 7f5ab6ffd640 1 -- 172.21.15.116:0/164714256 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 10280+0+0 (secure 0 0 0) 0x7f5ab81224f0 con 0x7f5ac014c720 2024-04-25T00:31:32.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.858+0000 7f5ac627d640 1 --2- 172.21.15.116:0/164714256 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5aa40678c0 0x7f5aa4069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:32.860 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.858+0000 7f5ac627d640 1 --2- 172.21.15.116:0/164714256 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5aa40678c0 0x7f5aa4069d80 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f5abc006fd0 tx=0x7f5abc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:32.863 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:32.862+0000 7f5ab6ffd640 1 -- 172.21.15.116:0/164714256 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f5ab80165a0 con 0x7f5ac014c720 2024-04-25T00:31:33.062 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.062+0000 7f5ac84e1640 1 -- 172.21.15.116:0/164714256 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5ac0072070 con 0x7f5ac014c720 2024-04-25T00:31:33.063 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.062+0000 7f5ab6ffd640 1 -- 172.21.15.116:0/164714256 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 244 v244) ==== 95+0+4365 (secure 0 0 0) 0x7f5ab80f1960 con 0x7f5ac014c720 2024-04-25T00:31:33.063 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:31:33.063 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":244,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":9027,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/426062493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":426062493},{"type":"v1","addr":"172.21.15.116:6835","nonce":426062493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":242}],"filesystems":[{"mdsmap":{"epoch":241,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:30:19.111393+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6764},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6764":{"gid":6764,"name":"d","rank":0,"incarnation":238,"state":"up:active","state_seq":8,"addr":"172.21.15.174:6837/3873739116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3873739116},{"type":"v1","addr":"172.21.15.174:6837","nonce":3873739116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6764,"qdb_cluster":[6764]},"id":2}]} 2024-04-25T00:31:33.063 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 244 2024-04-25T00:31:33.065 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.062+0000 7f5ac84e1640 1 -- 172.21.15.116:0/164714256 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5aa40678c0 msgr2=0x7f5aa4069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:33.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.062+0000 7f5ac84e1640 1 --2- 172.21.15.116:0/164714256 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5aa40678c0 0x7f5aa4069d80 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f5abc006fd0 tx=0x7f5abc008040 comp rx=0 tx=0).stop 2024-04-25T00:31:33.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.062+0000 7f5ac84e1640 1 -- 172.21.15.116:0/164714256 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ac014c720 msgr2=0x7f5ac014b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:33.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.062+0000 7f5ac84e1640 1 --2- 172.21.15.116:0/164714256 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ac014c720 0x7f5ac014b140 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f5ab80af040 tx=0x7f5ab8019690 comp rx=0 tx=0).stop 2024-04-25T00:31:33.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.066+0000 7f5ac84e1640 1 -- 172.21.15.116:0/164714256 shutdown_connections 2024-04-25T00:31:33.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.066+0000 7f5ac84e1640 1 --2- 172.21.15.116:0/164714256 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f5aa40678c0 0x7f5aa4069d80 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:33.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.066+0000 7f5ac84e1640 1 --2- 172.21.15.116:0/164714256 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f5ac014c720 0x7f5ac014b140 unknown :-1 s=CLOSED pgs=1892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:33.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.066+0000 7f5ac84e1640 1 --2- 172.21.15.116:0/164714256 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f5ac00a94a0 0x7f5ac01519b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:33.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.066+0000 7f5ac84e1640 1 --2- 172.21.15.116:0/164714256 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f5ac0071680 0x7f5ac00c66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:33.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.066+0000 7f5ac84e1640 1 -- 172.21.15.116:0/164714256 >> 172.21.15.116:0/164714256 conn(0x7f5ac00c3b00 msgr2=0x7f5ac0155cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:31:33.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.066+0000 7f5ac84e1640 1 -- 172.21.15.116:0/164714256 shutdown_connections 2024-04-25T00:31:33.066 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.066+0000 7f5ac84e1640 1 -- 172.21.15.116:0/164714256 wait complete. 2024-04-25T00:31:33.068 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.066+0000 7efbfeffd640 1 -- 172.21.15.116:0/54003712 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "a_data", "pool2": "a_data", "yes_i_really_really_mean_it": true}]=0 pool 'a_data' removed v77) ==== 154+0+0 (secure 0 0 0) 0x7efc00056f70 con 0x7efbf4003f40 2024-04-25T00:31:33.068 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'a_data' removed 2024-04-25T00:31:33.073 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.070+0000 7efc0cfcc640 1 -- 172.21.15.116:0/54003712 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efbe80678c0 msgr2=0x7efbe8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:33.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.070+0000 7efc0cfcc640 1 --2- 172.21.15.116:0/54003712 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efbe80678c0 0x7efbe8069d80 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7efbf00aa000 tx=0x7efc080c8e60 comp rx=0 tx=0).stop 2024-04-25T00:31:33.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.070+0000 7efc0cfcc640 1 -- 172.21.15.116:0/54003712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efbf4003f40 msgr2=0x7efc081f22f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:33.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.070+0000 7efc0cfcc640 1 --2- 172.21.15.116:0/54003712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efbf4003f40 0x7efc081f22f0 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7efc0000f400 tx=0x7efc0000f8c0 comp rx=0 tx=0).stop 2024-04-25T00:31:33.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.074+0000 7efc0cfcc640 1 -- 172.21.15.116:0/54003712 shutdown_connections 2024-04-25T00:31:33.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.074+0000 7efc0cfcc640 1 --2- 172.21.15.116:0/54003712 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7efbe80678c0 0x7efbe8069d80 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:33.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.074+0000 7efc0cfcc640 1 --2- 172.21.15.116:0/54003712 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7efc08150590 0x7efc081eec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:33.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.074+0000 7efc0cfcc640 1 --2- 172.21.15.116:0/54003712 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7efc0814e870 0x7efc081ec350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:33.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.074+0000 7efc0cfcc640 1 --2- 172.21.15.116:0/54003712 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7efbf4003f40 0x7efc081f22f0 unknown :-1 s=CLOSED pgs=1890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:33.074 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.074+0000 7efc0cfcc640 1 -- 172.21.15.116:0/54003712 >> 172.21.15.116:0/54003712 conn(0x7efc0806ef80 msgr2=0x7efc080671c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:31:33.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.074+0000 7efc0cfcc640 1 -- 172.21.15.116:0/54003712 shutdown_connections 2024-04-25T00:31:33.075 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.074+0000 7efc0cfcc640 1 -- 172.21.15.116:0/54003712 wait complete. 2024-04-25T00:31:33.099 INFO:tasks.cephfs.filesystem:Destroying file system b and related pools 2024-04-25T00:31:33.099 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:31:33.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.362+0000 7f6ee4ddc640 1 Processor -- start 2024-04-25T00:31:33.365 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.362+0000 7f6ee4ddc640 1 -- start start 2024-04-25T00:31:33.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.366+0000 7f6ee4ddc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f6ee0150d20 0x7f6ee00c2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:33.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.366+0000 7f6ee4ddc640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6ee00a94a0 0x7f6ee0153ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:33.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.366+0000 7f6ee4ddc640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ee00c3910 con 0x7f6ee00c2f50 2024-04-25T00:31:33.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.366+0000 7f6ed7fff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f6ee00c2f50 0x7f6ee01538e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:46872/0 (socket says 172.21.15.116:46872) 2024-04-25T00:31:33.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.366+0000 7f6ed7fff640 1 -- 172.21.15.116:0/4177070015 learned_addr learned my addr 172.21.15.116:0/4177070015 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:31:33.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.366+0000 7f6ede59c640 1 --2- 172.21.15.116:0/4177070015 >> v2:172.21.15.116:3300/0 conn(0x7f6ee0150d20 0x7f6ee00c2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:33.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.366+0000 7f6ee4ddc640 1 -- 172.21.15.116:0/4177070015 --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f6ee0154530 con 0x7f6ee0150d20 2024-04-25T00:31:33.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.366+0000 7f6ee4ddc640 1 -- 172.21.15.116:0/4177070015 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6ee00c16e0 con 0x7f6ee00a94a0 2024-04-25T00:31:33.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.366+0000 7f6eded9d640 1 --2- 172.21.15.116:0/4177070015 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6ee00a94a0 0x7f6ee0153ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:33.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.366+0000 7f6eddd9b640 1 -- 172.21.15.116:0/4177070015 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2744231716 0 0) 0x7f6ee00c3910 con 0x7f6ee00c2f50 2024-04-25T00:31:33.367 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.366+0000 7f6eddd9b640 1 -- 172.21.15.116:0/4177070015 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ecc003530 con 0x7f6ee00c2f50 2024-04-25T00:31:33.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.366+0000 7f6ede59c640 1 -- 172.21.15.116:0/4177070015 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6ee00a94a0 msgr2=0x7f6ee0153ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:31:33.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.366+0000 7f6ede59c640 1 --2- 172.21.15.116:0/4177070015 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6ee00a94a0 0x7f6ee0153ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:33.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.366+0000 7f6ede59c640 1 -- 172.21.15.116:0/4177070015 >> v1:172.21.15.116:6789/0 conn(0x7f6ee00c2f50 legacy=0x7f6ee01538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:33.368 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.366+0000 7f6ede59c640 1 -- 172.21.15.116:0/4177070015 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ee00c1400 con 0x7f6ee0150d20 2024-04-25T00:31:33.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.374+0000 7f6ede59c640 1 --2- 172.21.15.116:0/4177070015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ee0150d20 0x7f6ee00c2a10 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f6ed00097c0 tx=0x7f6ed001c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4709dd42954d7bd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:33.373 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.374+0000 7f6eddd9b640 1 -- 172.21.15.116:0/4177070015 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ed00b1070 con 0x7f6ee0150d20 2024-04-25T00:31:33.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.374+0000 7f6eddd9b640 1 -- 172.21.15.116:0/4177070015 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f6ed00ac410 con 0x7f6ee0150d20 2024-04-25T00:31:33.374 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.374+0000 7f6eddd9b640 1 -- 172.21.15.116:0/4177070015 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ed00ac800 con 0x7f6ee0150d20 2024-04-25T00:31:33.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.374+0000 7f6ee4ddc640 1 -- 172.21.15.116:0/4177070015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ee0150d20 msgr2=0x7f6ee00c2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:33.375 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.374+0000 7f6ee4ddc640 1 --2- 172.21.15.116:0/4177070015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ee0150d20 0x7f6ee00c2a10 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f6ed00097c0 tx=0x7f6ed001c4a0 comp rx=0 tx=0).stop 2024-04-25T00:31:33.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.382+0000 7f6ee4ddc640 1 -- 172.21.15.116:0/4177070015 shutdown_connections 2024-04-25T00:31:33.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.382+0000 7f6ee4ddc640 1 --2- 172.21.15.116:0/4177070015 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6ee00a94a0 0x7f6ee0153ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:33.381 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.382+0000 7f6ee4ddc640 1 --2- 172.21.15.116:0/4177070015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ee0150d20 0x7f6ee00c2a10 unknown :-1 s=CLOSED pgs=1894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:33.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.382+0000 7f6ee4ddc640 1 -- 172.21.15.116:0/4177070015 >> 172.21.15.116:0/4177070015 conn(0x7f6ee00c3b00 msgr2=0x7f6ee00c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:31:33.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.382+0000 7f6ee4ddc640 1 -- 172.21.15.116:0/4177070015 shutdown_connections 2024-04-25T00:31:33.382 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.382+0000 7f6ee4ddc640 1 -- 172.21.15.116:0/4177070015 wait complete. 2024-04-25T00:31:33.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.382+0000 7f6ee4ddc640 1 Processor -- start 2024-04-25T00:31:33.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.382+0000 7f6ee4ddc640 1 -- start start 2024-04-25T00:31:33.383 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.382+0000 7f6ee4ddc640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ee00a94a0 0x7f6ee014c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:33.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.382+0000 7f6ee4ddc640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6ee00c2f50 0x7f6ee01f1d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:33.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.382+0000 7f6ee4ddc640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6ee0150d20 0x7f6ee01f2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:33.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.382+0000 7f6ee4ddc640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f6ee01608b0 con 0x7f6ee00a94a0 2024-04-25T00:31:33.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.382+0000 7f6ee4ddc640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f6ee0160730 con 0x7f6ee0150d20 2024-04-25T00:31:33.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.382+0000 7f6ee4ddc640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f6ee0160a30 con 0x7f6ee00c2f50 2024-04-25T00:31:33.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.382+0000 7f6ede59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ee00a94a0 0x7f6ee014c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:33.385 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.382+0000 7f6ede59c640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ee00a94a0 0x7f6ee014c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51310/0 (socket says 172.21.15.116:51310) 2024-04-25T00:31:33.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.382+0000 7f6ede59c640 1 -- 172.21.15.116:0/533588074 learned_addr learned my addr 172.21.15.116:0/533588074 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:31:33.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.386+0000 7f6ed7fff640 1 --2- 172.21.15.116:0/533588074 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6ee00c2f50 0x7f6ee01f1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:33.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.386+0000 7f6eded9d640 1 --2- 172.21.15.116:0/533588074 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6ee0150d20 0x7f6ee01f2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:33.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.386+0000 7f6ede59c640 1 -- 172.21.15.116:0/533588074 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6ee00c2f50 msgr2=0x7f6ee01f1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:33.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.386+0000 7f6ede59c640 1 --2- 172.21.15.116:0/533588074 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6ee00c2f50 0x7f6ee01f1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:33.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.386+0000 7f6ede59c640 1 -- 172.21.15.116:0/533588074 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6ee0150d20 msgr2=0x7f6ee01f2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:33.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.386+0000 7f6ede59c640 1 --2- 172.21.15.116:0/533588074 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6ee0150d20 0x7f6ee01f2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:33.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.386+0000 7f6ede59c640 1 -- 172.21.15.116:0/533588074 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ee0067a90 con 0x7f6ee00a94a0 2024-04-25T00:31:33.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.386+0000 7f6eded9d640 1 --2- 172.21.15.116:0/533588074 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6ee0150d20 0x7f6ee01f2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:31:33.386 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.386+0000 7f6ed7fff640 1 --2- 172.21.15.116:0/533588074 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6ee00c2f50 0x7f6ee01f1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:31:33.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.386+0000 7f6ede59c640 1 --2- 172.21.15.116:0/533588074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ee00a94a0 0x7f6ee014c7a0 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f6ed00097c0 tx=0x7f6ed0019240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:33.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.386+0000 7f6ed6ffd640 1 -- 172.21.15.116:0/533588074 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ed00b9020 con 0x7f6ee00a94a0 2024-04-25T00:31:33.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.386+0000 7f6ed6ffd640 1 -- 172.21.15.116:0/533588074 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f6ed00b5410 con 0x7f6ee00a94a0 2024-04-25T00:31:33.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.386+0000 7f6ed6ffd640 1 -- 172.21.15.116:0/533588074 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ed00b1040 con 0x7f6ee00a94a0 2024-04-25T00:31:33.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.386+0000 7f6ee4ddc640 1 -- 172.21.15.116:0/533588074 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ee00c0930 con 0x7f6ee00a94a0 2024-04-25T00:31:33.387 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.386+0000 7f6ee4ddc640 1 -- 172.21.15.116:0/533588074 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ee0072400 con 0x7f6ee00a94a0 2024-04-25T00:31:33.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.386+0000 7f6ed4ff9640 1 -- 172.21.15.116:0/533588074 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6eb80052a0 con 0x7f6ee00a94a0 2024-04-25T00:31:33.389 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.390+0000 7f6ed6ffd640 1 -- 172.21.15.116:0/533588074 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f6ed00bc020 con 0x7f6ee00a94a0 2024-04-25T00:31:33.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.390+0000 7f6ed6ffd640 1 --2- 172.21.15.116:0/533588074 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6ec0067910 0x7f6ec0069dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:33.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.390+0000 7f6ed6ffd640 1 -- 172.21.15.116:0/533588074 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 9847+0+0 (secure 0 0 0) 0x7f6ed0120760 con 0x7f6ee00a94a0 2024-04-25T00:31:33.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.390+0000 7f6ed7fff640 1 --2- 172.21.15.116:0/533588074 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6ec0067910 0x7f6ec0069dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:33.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.390+0000 7f6ed7fff640 1 --2- 172.21.15.116:0/533588074 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6ec0067910 0x7f6ec0069dd0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f6ee00c6fd0 tx=0x7f6ec8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:33.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.398+0000 7f6ed6ffd640 1 -- 172.21.15.116:0/533588074 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f6ed00c3030 con 0x7f6ee00a94a0 2024-04-25T00:31:33.630 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.630+0000 7f6ed4ff9640 1 -- 172.21.15.116:0/533588074 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6eb8005590 con 0x7f6ee00a94a0 2024-04-25T00:31:33.631 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.630+0000 7f6ed6ffd640 1 -- 172.21.15.116:0/533588074 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 244 v244) ==== 95+0+4365 (secure 0 0 0) 0x7f6ed0016090 con 0x7f6ee00a94a0 2024-04-25T00:31:33.632 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:31:33.632 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":244,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":9027,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/426062493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":426062493},{"type":"v1","addr":"172.21.15.116:6835","nonce":426062493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":242}],"filesystems":[{"mdsmap":{"epoch":241,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:30:19.111393+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6764},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6764":{"gid":6764,"name":"d","rank":0,"incarnation":238,"state":"up:active","state_seq":8,"addr":"172.21.15.174:6837/3873739116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3873739116},{"type":"v1","addr":"172.21.15.174:6837","nonce":3873739116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6764,"qdb_cluster":[6764]},"id":2}]} 2024-04-25T00:31:33.632 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 244 2024-04-25T00:31:33.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.634+0000 7f6ed4ff9640 1 -- 172.21.15.116:0/533588074 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6ec0067910 msgr2=0x7f6ec0069dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:33.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.634+0000 7f6ed4ff9640 1 --2- 172.21.15.116:0/533588074 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6ec0067910 0x7f6ec0069dd0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f6ee00c6fd0 tx=0x7f6ec8009040 comp rx=0 tx=0).stop 2024-04-25T00:31:33.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.634+0000 7f6ed4ff9640 1 -- 172.21.15.116:0/533588074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ee00a94a0 msgr2=0x7f6ee014c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:33.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.634+0000 7f6ed4ff9640 1 --2- 172.21.15.116:0/533588074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ee00a94a0 0x7f6ee014c7a0 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f6ed00097c0 tx=0x7f6ed0019240 comp rx=0 tx=0).stop 2024-04-25T00:31:33.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.634+0000 7f6ed4ff9640 1 -- 172.21.15.116:0/533588074 shutdown_connections 2024-04-25T00:31:33.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.634+0000 7f6ed4ff9640 1 --2- 172.21.15.116:0/533588074 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f6ec0067910 0x7f6ec0069dd0 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:33.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.634+0000 7f6ed4ff9640 1 --2- 172.21.15.116:0/533588074 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f6ee0150d20 0x7f6ee01f2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:33.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.634+0000 7f6ed4ff9640 1 --2- 172.21.15.116:0/533588074 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f6ee00c2f50 0x7f6ee01f1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:33.634 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.634+0000 7f6ed4ff9640 1 --2- 172.21.15.116:0/533588074 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f6ee00a94a0 0x7f6ee014c7a0 unknown :-1 s=CLOSED pgs=1895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:33.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.634+0000 7f6ed4ff9640 1 -- 172.21.15.116:0/533588074 >> 172.21.15.116:0/533588074 conn(0x7f6ee00c3b00 msgr2=0x7f6ee006d970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:31:33.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.634+0000 7f6ed4ff9640 1 -- 172.21.15.116:0/533588074 shutdown_connections 2024-04-25T00:31:33.635 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.634+0000 7f6ed4ff9640 1 -- 172.21.15.116:0/533588074 wait complete. 2024-04-25T00:31:33.650 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T00:31:33.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.766+0000 7f01e9953640 1 Processor -- start 2024-04-25T00:31:33.767 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.766+0000 7f01e9953640 1 -- start start 2024-04-25T00:31:33.768 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.766+0000 7f01e9953640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f01e413ed60 0x7f01e41559c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:33.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.766+0000 7f01e9953640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f01e413e810 0x7f01e4155f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:33.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.766+0000 7f01e9953640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01e406d200 con 0x7f01e413e310 2024-04-25T00:31:33.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.766+0000 7f01e9953640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f01e4070d40 con 0x7f01e413ed60 2024-04-25T00:31:33.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.766+0000 7f01e9953640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f01e4071020 con 0x7f01e413e810 2024-04-25T00:31:33.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.766+0000 7f01e27fc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f01e413ed60 0x7f01e41559c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:33.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.766+0000 7f01e27fc640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f01e413ed60 0x7f01e41559c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51326/0 (socket says 172.21.15.116:51326) 2024-04-25T00:31:33.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.766+0000 7f01e27fc640 1 -- 172.21.15.116:0/1970433015 learned_addr learned my addr 172.21.15.116:0/1970433015 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:31:33.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.766+0000 7f01e37fe640 1 --2- 172.21.15.116:0/1970433015 >> v2:172.21.15.174:3300/0 conn(0x7f01e413e810 0x7f01e4155f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:33.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.766+0000 7f01e27fc640 1 -- 172.21.15.116:0/1970433015 >> v2:172.21.15.174:3300/0 conn(0x7f01e413e810 msgr2=0x7f01e4155f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:31:33.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.766+0000 7f01e27fc640 1 --2- 172.21.15.116:0/1970433015 >> v2:172.21.15.174:3300/0 conn(0x7f01e413e810 0x7f01e4155f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:33.769 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.766+0000 7f01e27fc640 1 -- 172.21.15.116:0/1970433015 >> v1:172.21.15.174:6789/0 conn(0x7f01e413e310 legacy=0x7f01e40c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:31:33.770 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.766+0000 7f01e27fc640 1 -- 172.21.15.116:0/1970433015 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01e406cb50 con 0x7f01e413ed60 2024-04-25T00:31:33.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.770+0000 7f01e27fc640 1 --2- 172.21.15.116:0/1970433015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01e413ed60 0x7f01e41559c0 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f01e40c4ba0 tx=0x7f01d801c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45435918fc10d9df server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:33.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.770+0000 7f01e1ffb640 1 -- 172.21.15.116:0/1970433015 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d80a5660 con 0x7f01e413ed60 2024-04-25T00:31:33.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.770+0000 7f01e1ffb640 1 -- 172.21.15.116:0/1970433015 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f01d80a5800 con 0x7f01e413ed60 2024-04-25T00:31:33.771 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.770+0000 7f01e1ffb640 1 -- 172.21.15.116:0/1970433015 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01d80a5bf0 con 0x7f01e413ed60 2024-04-25T00:31:33.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.770+0000 7f01e9953640 1 -- 172.21.15.116:0/1970433015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01e413ed60 msgr2=0x7f01e41559c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:33.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.770+0000 7f01e9953640 1 --2- 172.21.15.116:0/1970433015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01e413ed60 0x7f01e41559c0 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f01e40c4ba0 tx=0x7f01d801c890 comp rx=0 tx=0).stop 2024-04-25T00:31:33.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.770+0000 7f01e9953640 1 -- 172.21.15.116:0/1970433015 shutdown_connections 2024-04-25T00:31:33.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.770+0000 7f01e9953640 1 --2- 172.21.15.116:0/1970433015 >> v2:172.21.15.174:3300/0 conn(0x7f01e413e810 0x7f01e4155f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:33.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.770+0000 7f01e9953640 1 --2- 172.21.15.116:0/1970433015 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01e413ed60 0x7f01e41559c0 unknown :-1 s=CLOSED pgs=1896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:33.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.770+0000 7f01e9953640 1 -- 172.21.15.116:0/1970433015 >> 172.21.15.116:0/1970433015 conn(0x7f01e40c3b00 msgr2=0x7f01e40c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:31:33.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.770+0000 7f01e9953640 1 -- 172.21.15.116:0/1970433015 shutdown_connections 2024-04-25T00:31:33.772 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.770+0000 7f01e9953640 1 -- 172.21.15.116:0/1970433015 wait complete. 2024-04-25T00:31:33.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.770+0000 7f01e9953640 1 Processor -- start 2024-04-25T00:31:33.773 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.770+0000 7f01e9953640 1 -- start start 2024-04-25T00:31:33.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.774+0000 7f01e9953640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01e413e310 0x7f01e412cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:33.775 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.774+0000 7f01e9953640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f01e413e810 0x7f01e4154be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:33.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.774+0000 7f01e9953640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f01e413ed60 0x7f01e41287b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:33.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.774+0000 7f01e9953640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f01e41565c0 con 0x7f01e413e310 2024-04-25T00:31:33.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.774+0000 7f01e9953640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f01e4156440 con 0x7f01e413ed60 2024-04-25T00:31:33.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.774+0000 7f01e9953640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f01e41629f0 con 0x7f01e413e810 2024-04-25T00:31:33.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.774+0000 7f01e2ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01e413e310 0x7f01e412cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:33.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.774+0000 7f01e2ffd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01e413e310 0x7f01e412cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51340/0 (socket says 172.21.15.116:51340) 2024-04-25T00:31:33.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.774+0000 7f01e2ffd640 1 -- 172.21.15.116:0/4071607041 learned_addr learned my addr 172.21.15.116:0/4071607041 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:31:33.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.774+0000 7f01e37fe640 1 --2- 172.21.15.116:0/4071607041 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f01e413ed60 0x7f01e41287b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:33.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.774+0000 7f01e27fc640 1 --2- 172.21.15.116:0/4071607041 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f01e413e810 0x7f01e4154be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:33.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.774+0000 7f01e2ffd640 1 -- 172.21.15.116:0/4071607041 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f01e413e810 msgr2=0x7f01e4154be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:33.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.774+0000 7f01e2ffd640 1 --2- 172.21.15.116:0/4071607041 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f01e413e810 0x7f01e4154be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:33.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.774+0000 7f01e2ffd640 1 -- 172.21.15.116:0/4071607041 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f01e413ed60 msgr2=0x7f01e41287b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:33.776 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.774+0000 7f01e2ffd640 1 --2- 172.21.15.116:0/4071607041 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f01e413ed60 0x7f01e41287b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:33.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.774+0000 7f01e2ffd640 1 -- 172.21.15.116:0/4071607041 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01e412dcf0 con 0x7f01e413e310 2024-04-25T00:31:33.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.774+0000 7f01e2ffd640 1 --2- 172.21.15.116:0/4071607041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01e413e310 0x7f01e412cca0 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f01dc00c480 tx=0x7f01dc00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:33.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.774+0000 7f01cbfff640 1 -- 172.21.15.116:0/4071607041 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01dc012830 con 0x7f01e413e310 2024-04-25T00:31:33.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.774+0000 7f01cbfff640 1 -- 172.21.15.116:0/4071607041 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f01dc017070 con 0x7f01e413e310 2024-04-25T00:31:33.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.774+0000 7f01e9953640 1 -- 172.21.15.116:0/4071607041 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01e412df90 con 0x7f01e413e310 2024-04-25T00:31:33.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.774+0000 7f01cbfff640 1 -- 172.21.15.116:0/4071607041 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01dc012c50 con 0x7f01e413e310 2024-04-25T00:31:33.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.774+0000 7f01e9953640 1 -- 172.21.15.116:0/4071607041 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01e412e550 con 0x7f01e413e310 2024-04-25T00:31:33.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.774+0000 7f01e9953640 1 -- 172.21.15.116:0/4071607041 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01e4068350 con 0x7f01e413e310 2024-04-25T00:31:33.777 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.774+0000 7f01cbfff640 1 -- 172.21.15.116:0/4071607041 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f01dc01c460 con 0x7f01e413e310 2024-04-25T00:31:33.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.774+0000 7f01cbfff640 1 --2- 172.21.15.116:0/4071607041 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f01d00678c0 0x7f01d0069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:33.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.774+0000 7f01cbfff640 1 -- 172.21.15.116:0/4071607041 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 9847+0+0 (secure 0 0 0) 0x7f01dc086bb0 con 0x7f01e413e310 2024-04-25T00:31:33.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.774+0000 7f01e27fc640 1 --2- 172.21.15.116:0/4071607041 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f01d00678c0 0x7f01d0069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:33.778 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.778+0000 7f01e27fc640 1 --2- 172.21.15.116:0/4071607041 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f01d00678c0 0x7f01d0069d80 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f01d80ae000 tx=0x7f01d80a7070 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:33.780 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.778+0000 7f01cbfff640 1 -- 172.21.15.116:0/4071607041 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f01dc056670 con 0x7f01e413e310 2024-04-25T00:31:33.964 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.962+0000 7f01e9953640 1 -- 172.21.15.116:0/4071607041 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f01e40c34f0 con 0x7f01e413e310 2024-04-25T00:31:33.965 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.966+0000 7f01cbfff640 1 -- 172.21.15.116:0/4071607041 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 244 v244) ==== 95+0+4365 (secure 0 0 0) 0x7f01dc055f70 con 0x7f01e413e310 2024-04-25T00:31:33.966 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:31:33.966 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":244,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5582,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.174:6835/4112145782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6834","nonce":4112145782},{"type":"v1","addr":"172.21.15.174:6835","nonce":4112145782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":112},{"gid":9027,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.116:6835/426062493","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6834","nonce":426062493},{"type":"v1","addr":"172.21.15.116:6835","nonce":426062493}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":242}],"filesystems":[{"mdsmap":{"epoch":241,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T00:01:18.825887+0000","modified":"2024-04-25T00:30:19.111393+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6764},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6764":{"gid":6764,"name":"d","rank":0,"incarnation":238,"state":"up:active","state_seq":8,"addr":"172.21.15.174:6837/3873739116","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6836","nonce":3873739116},{"type":"v1","addr":"172.21.15.174:6837","nonce":3873739116}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"b","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6764,"qdb_cluster":[6764]},"id":2}]} 2024-04-25T00:31:33.966 INFO:teuthology.orchestra.run.smithi116.stderr:dumped fsmap epoch 244 2024-04-25T00:31:33.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.966+0000 7f01e9953640 1 -- 172.21.15.116:0/4071607041 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f01d00678c0 msgr2=0x7f01d0069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:33.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.966+0000 7f01e9953640 1 --2- 172.21.15.116:0/4071607041 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f01d00678c0 0x7f01d0069d80 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f01d80ae000 tx=0x7f01d80a7070 comp rx=0 tx=0).stop 2024-04-25T00:31:33.968 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.966+0000 7f01e9953640 1 -- 172.21.15.116:0/4071607041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01e413e310 msgr2=0x7f01e412cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:33.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.966+0000 7f01e9953640 1 --2- 172.21.15.116:0/4071607041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01e413e310 0x7f01e412cca0 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f01dc00c480 tx=0x7f01dc00c940 comp rx=0 tx=0).stop 2024-04-25T00:31:33.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.966+0000 7f01e9953640 1 -- 172.21.15.116:0/4071607041 shutdown_connections 2024-04-25T00:31:33.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.966+0000 7f01e9953640 1 --2- 172.21.15.116:0/4071607041 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f01d00678c0 0x7f01d0069d80 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:33.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.966+0000 7f01e9953640 1 --2- 172.21.15.116:0/4071607041 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f01e413ed60 0x7f01e41287b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:33.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.966+0000 7f01e9953640 1 --2- 172.21.15.116:0/4071607041 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f01e413e810 0x7f01e4154be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:33.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.966+0000 7f01e9953640 1 --2- 172.21.15.116:0/4071607041 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f01e413e310 0x7f01e412cca0 unknown :-1 s=CLOSED pgs=1897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:33.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.966+0000 7f01e9953640 1 -- 172.21.15.116:0/4071607041 >> 172.21.15.116:0/4071607041 conn(0x7f01e40c3b00 msgr2=0x7f01e406e300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:31:33.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.966+0000 7f01e9953640 1 -- 172.21.15.116:0/4071607041 shutdown_connections 2024-04-25T00:31:33.969 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:33.966+0000 7f01e9953640 1 -- 172.21.15.116:0/4071607041 wait complete. 2024-04-25T00:31:33.984 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T00:31:34.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.098+0000 7fc8b1a8f640 1 Processor -- start 2024-04-25T00:31:34.098 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.098+0000 7fc8b1a8f640 1 -- start start 2024-04-25T00:31:34.103 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.098+0000 7fc8b1a8f640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7fc8ac0c1c30 0x7fc8ac0c3060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:34.104 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.098+0000 7fc8b1a8f640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8ac0c22a0 con 0x7fc8ac153170 2024-04-25T00:31:34.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.098+0000 7fc8b1a8f640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8ac0c2580 con 0x7fc8ac06c480 2024-04-25T00:31:34.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.098+0000 7fc8b1a8f640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7fc8ac0c35a0 con 0x7fc8ac0c1c30 2024-04-25T00:31:34.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.098+0000 7fc8abfff640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7fc8ac153170 0x7fc8ac14fa80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:46884/0 (socket says 172.21.15.116:46884) 2024-04-25T00:31:34.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.098+0000 7fc8abfff640 1 -- 172.21.15.116:0/2764605495 learned_addr learned my addr 172.21.15.116:0/2764605495 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:31:34.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.098+0000 7fc8aaffd640 1 --2- 172.21.15.116:0/2764605495 >> v2:172.21.15.174:3300/0 conn(0x7fc8ac0c1c30 0x7fc8ac0c3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:34.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.098+0000 7fc8aa7fc640 1 -- 172.21.15.116:0/2764605495 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 706507039 0 0) 0x7fc8ac0c22a0 con 0x7fc8ac153170 2024-04-25T00:31:34.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.098+0000 7fc8aa7fc640 1 -- 172.21.15.116:0/2764605495 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc894003530 con 0x7fc8ac153170 2024-04-25T00:31:34.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.098+0000 7fc8aa7fc640 1 -- 172.21.15.116:0/2764605495 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3857807655 0 0) 0x7fc894003530 con 0x7fc8ac153170 2024-04-25T00:31:34.106 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.098+0000 7fc8aa7fc640 1 -- 172.21.15.116:0/2764605495 >> v2:172.21.15.174:3300/0 conn(0x7fc8ac0c1c30 msgr2=0x7fc8ac0c3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:31:34.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.098+0000 7fc8aa7fc640 1 --2- 172.21.15.116:0/2764605495 >> v2:172.21.15.174:3300/0 conn(0x7fc8ac0c1c30 0x7fc8ac0c3060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:34.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.098+0000 7fc8aa7fc640 1 -- 172.21.15.116:0/2764605495 >> v1:172.21.15.174:6789/0 conn(0x7fc8ac06c480 legacy=0x7fc8ac071e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:34.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.098+0000 7fc8aa7fc640 1 -- 172.21.15.116:0/2764605495 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8ac0c2860 con 0x7fc8ac153170 2024-04-25T00:31:34.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.098+0000 7fc8aa7fc640 1 -- 172.21.15.116:0/2764605495 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7fc8a4001a00 con 0x7fc8ac153170 2024-04-25T00:31:34.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.098+0000 7fc8aa7fc640 1 -- 172.21.15.116:0/2764605495 >> v1:172.21.15.116:6789/0 conn(0x7fc8ac153170 legacy=0x7fc8ac14fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:34.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.098+0000 7fc8aa7fc640 1 --2- 172.21.15.116:0/2764605495 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc8940039c0 0x7fc894003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:34.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.098+0000 7fc8aa7fc640 1 --2- 172.21.15.116:0/2764605495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc894004810 0x7fc894004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:34.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.098+0000 7fc8aa7fc640 1 --2- 172.21.15.116:0/2764605495 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc8940051d0 0x7fc89400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:34.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.098+0000 7fc8aa7fc640 1 -- 172.21.15.116:0/2764605495 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc8ac0c35a0 con 0x7fc894004810 2024-04-25T00:31:34.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.098+0000 7fc8aa7fc640 1 -- 172.21.15.116:0/2764605495 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc89400c070 con 0x7fc8940039c0 2024-04-25T00:31:34.107 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.098+0000 7fc8aa7fc640 1 -- 172.21.15.116:0/2764605495 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fc89400c270 con 0x7fc8940051d0 2024-04-25T00:31:34.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.098+0000 7fc8aaffd640 1 --2- 172.21.15.116:0/2764605495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc894004810 0x7fc894004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:34.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.098+0000 7fc8aaffd640 1 -- 172.21.15.116:0/2764605495 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc8940051d0 msgr2=0x7fc89400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:31:34.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.098+0000 7fc8aaffd640 1 --2- 172.21.15.116:0/2764605495 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc8940051d0 0x7fc89400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:34.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.098+0000 7fc8aaffd640 1 -- 172.21.15.116:0/2764605495 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc8940039c0 msgr2=0x7fc894003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:31:34.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.098+0000 7fc8ab7fe640 1 --2- 172.21.15.116:0/2764605495 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc8940039c0 0x7fc894003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:34.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.098+0000 7fc8aaffd640 1 --2- 172.21.15.116:0/2764605495 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc8940039c0 0x7fc894003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:34.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.098+0000 7fc8abfff640 1 -- 172.21.15.116:0/2764605495 reap_dead start 2024-04-25T00:31:34.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.098+0000 7fc8aaffd640 1 -- 172.21.15.116:0/2764605495 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc894003530 con 0x7fc894004810 2024-04-25T00:31:34.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.102+0000 7fc8aaffd640 1 --2- 172.21.15.116:0/2764605495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc894004810 0x7fc894004c90 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7fc8a0009ab0 tx=0x7fc8a001ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d203e06530f5f41b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:34.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.102+0000 7fc8aa7fc640 1 -- 172.21.15.116:0/2764605495 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8a00b2070 con 0x7fc894004810 2024-04-25T00:31:34.108 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.102+0000 7fc8aa7fc640 1 -- 172.21.15.116:0/2764605495 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fc8a00a5640 con 0x7fc894004810 2024-04-25T00:31:34.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.102+0000 7fc8aa7fc640 1 -- 172.21.15.116:0/2764605495 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8a00096b0 con 0x7fc894004810 2024-04-25T00:31:34.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.102+0000 7fc8b1a8f640 1 -- 172.21.15.116:0/2764605495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc894004810 msgr2=0x7fc894004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:34.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.102+0000 7fc8b1a8f640 1 --2- 172.21.15.116:0/2764605495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc894004810 0x7fc894004c90 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7fc8a0009ab0 tx=0x7fc8a001ca50 comp rx=0 tx=0).stop 2024-04-25T00:31:34.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.102+0000 7fc8b1a8f640 1 -- 172.21.15.116:0/2764605495 shutdown_connections 2024-04-25T00:31:34.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.102+0000 7fc8b1a8f640 1 --2- 172.21.15.116:0/2764605495 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc894004810 0x7fc894004c90 unknown :-1 s=CLOSED pgs=1899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:34.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.102+0000 7fc8b1a8f640 1 -- 172.21.15.116:0/2764605495 >> 172.21.15.116:0/2764605495 conn(0x7fc8ac0c4270 msgr2=0x7fc8ac0c4680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:31:34.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.102+0000 7fc8b1a8f640 1 -- 172.21.15.116:0/2764605495 shutdown_connections 2024-04-25T00:31:34.109 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.102+0000 7fc8b1a8f640 1 -- 172.21.15.116:0/2764605495 wait complete. 2024-04-25T00:31:34.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.102+0000 7fc8b1a8f640 1 Processor -- start 2024-04-25T00:31:34.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.102+0000 7fc8b1a8f640 1 -- start start 2024-04-25T00:31:34.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.102+0000 7fc8b1a8f640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc894004810 0x7fc8ac0a88c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:34.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.102+0000 7fc8b1a8f640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc8ac0c6570 0x7fc8ac1eb570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:34.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.102+0000 7fc8b1a8f640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc8ac0c11e0 0x7fc8ac1ebab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:34.112 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.102+0000 7fc8b1a8f640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7fc8ac153670 con 0x7fc894004810 2024-04-25T00:31:34.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.102+0000 7fc8b1a8f640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7fc8ac06ca70 con 0x7fc8ac0c6570 2024-04-25T00:31:34.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.102+0000 7fc8b1a8f640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7fc8ac06fbc0 con 0x7fc8ac0c11e0 2024-04-25T00:31:34.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.102+0000 7fc8ab7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc894004810 0x7fc8ac0a88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:34.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.102+0000 7fc8ab7fe640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc894004810 0x7fc8ac0a88c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51362/0 (socket says 172.21.15.116:51362) 2024-04-25T00:31:34.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.102+0000 7fc8ab7fe640 1 -- 172.21.15.116:0/1078889485 learned_addr learned my addr 172.21.15.116:0/1078889485 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:31:34.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.102+0000 7fc8aaffd640 1 --2- 172.21.15.116:0/1078889485 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc8ac0c6570 0x7fc8ac1eb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:34.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.102+0000 7fc8abfff640 1 --2- 172.21.15.116:0/1078889485 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc8ac0c11e0 0x7fc8ac1ebab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:34.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.102+0000 7fc8ab7fe640 1 -- 172.21.15.116:0/1078889485 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc8ac0c11e0 msgr2=0x7fc8ac1ebab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:34.113 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.102+0000 7fc8ab7fe640 1 --2- 172.21.15.116:0/1078889485 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc8ac0c11e0 0x7fc8ac1ebab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:34.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.102+0000 7fc8ab7fe640 1 -- 172.21.15.116:0/1078889485 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc8ac0c6570 msgr2=0x7fc8ac1eb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:34.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.102+0000 7fc8ab7fe640 1 --2- 172.21.15.116:0/1078889485 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc8ac0c6570 0x7fc8ac1eb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:34.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.102+0000 7fc8ab7fe640 1 -- 172.21.15.116:0/1078889485 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8ac1eb0c0 con 0x7fc894004810 2024-04-25T00:31:34.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.106+0000 7fc8ab7fe640 1 --2- 172.21.15.116:0/1078889485 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc894004810 0x7fc8ac0a88c0 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7fc89c00d4a0 tx=0x7fc89c00d960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:34.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.106+0000 7fc893fff640 1 -- 172.21.15.116:0/1078889485 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc89c005480 con 0x7fc894004810 2024-04-25T00:31:34.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.106+0000 7fc893fff640 1 -- 172.21.15.116:0/1078889485 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fc89c017070 con 0x7fc894004810 2024-04-25T00:31:34.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.106+0000 7fc893fff640 1 -- 172.21.15.116:0/1078889485 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc89c003820 con 0x7fc894004810 2024-04-25T00:31:34.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.106+0000 7fc8b1a8f640 1 -- 172.21.15.116:0/1078889485 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8ac14f1c0 con 0x7fc894004810 2024-04-25T00:31:34.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.106+0000 7fc8b1a8f640 1 -- 172.21.15.116:0/1078889485 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8ac0a8e00 con 0x7fc894004810 2024-04-25T00:31:34.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.106+0000 7fc8b1a8f640 1 -- 172.21.15.116:0/1078889485 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8ac068a60 con 0x7fc894004810 2024-04-25T00:31:34.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.106+0000 7fc893fff640 1 -- 172.21.15.116:0/1078889485 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7fc89c0039c0 con 0x7fc894004810 2024-04-25T00:31:34.114 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.106+0000 7fc893fff640 1 --2- 172.21.15.116:0/1078889485 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc8980678f0 0x7fc898069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:34.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.106+0000 7fc893fff640 1 -- 172.21.15.116:0/1078889485 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 9847+0+0 (secure 0 0 0) 0x7fc89c086a10 con 0x7fc894004810 2024-04-25T00:31:34.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.106+0000 7fc8aaffd640 1 --2- 172.21.15.116:0/1078889485 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc8980678f0 0x7fc898069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:34.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.106+0000 7fc8aaffd640 1 --2- 172.21.15.116:0/1078889485 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc8980678f0 0x7fc898069db0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fc8a00af000 tx=0x7fc8a0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:34.115 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.110+0000 7fc893fff640 1 -- 172.21.15.116:0/1078889485 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7fc89c00f2f0 con 0x7fc894004810 2024-04-25T00:31:34.260 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.258+0000 7fc8b1a8f640 1 -- 172.21.15.116:0/1078889485 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc8ac1489e0 con 0x7fc894004810 2024-04-25T00:31:34.263 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.262+0000 7fc893fff640 1 -- 172.21.15.116:0/1078889485 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) ==== 74+0+23042 (secure 0 0 0) 0x7fc89c056560 con 0x7fc894004810 2024-04-25T00:31:34.264 INFO:teuthology.orchestra.run.smithi116.stdout: 2024-04-25T00:31:34.264 INFO:teuthology.orchestra.run.smithi116.stdout:{"epoch":77,"fsid":"6d6a1aa7-0478-4c83-8057-05917e81f046","created":"2024-04-25T00:00:40.682493+0000","modified":"2024-04-25T00:31:33.067956+0000","last_up_change":"2024-04-25T00:00:51.822174+0000","last_in_change":"2024-04-25T00:00:42.980473+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T00:00:53.324376+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"b_metadata","create_time":"2024-04-25T00:01:17.125210+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"b_data","create_time":"2024-04-25T00:01:17.821873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"b"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8b6ddcb2-1ad9-4167-961c-bd6936bb9060","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6808","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6809","nonce":3751765682}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6810","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6811","nonce":3751765682}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6814","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6815","nonce":3751765682}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6812","nonce":3751765682},{"type":"v1","addr":"172.21.15.116:6813","nonce":3751765682}]},"public_addr":"172.21.15.116:6809/3751765682","cluster_addr":"172.21.15.116:6811/3751765682","heartbeat_back_addr":"172.21.15.116:6815/3751765682","heartbeat_front_addr":"172.21.15.116:6813/3751765682","state":["exists","up"]},{"osd":1,"uuid":"a6431768-994f-41da-8853-886f5687fb91","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6800","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6801","nonce":377879124}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6802","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6803","nonce":377879124}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6806","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6807","nonce":377879124}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6804","nonce":377879124},{"type":"v1","addr":"172.21.15.116:6805","nonce":377879124}]},"public_addr":"172.21.15.116:6801/377879124","cluster_addr":"172.21.15.116:6803/377879124","heartbeat_back_addr":"172.21.15.116:6807/377879124","heartbeat_front_addr":"172.21.15.116:6805/377879124","state":["exists","up"]},{"osd":2,"uuid":"4f69bd40-5a23-45ff-8693-e1de75e1aa9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6816","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6817","nonce":1099282267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6818","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6819","nonce":1099282267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6822","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6823","nonce":1099282267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6820","nonce":1099282267},{"type":"v1","addr":"172.21.15.116:6821","nonce":1099282267}]},"public_addr":"172.21.15.116:6817/1099282267","cluster_addr":"172.21.15.116:6819/1099282267","heartbeat_back_addr":"172.21.15.116:6823/1099282267","heartbeat_front_addr":"172.21.15.116:6821/1099282267","state":["exists","up"]},{"osd":3,"uuid":"6a0d5344-8b6f-40d3-918d-0f0a4be688e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6824","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6825","nonce":1179942661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6826","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6827","nonce":1179942661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6830","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6831","nonce":1179942661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.116:6828","nonce":1179942661},{"type":"v1","addr":"172.21.15.116:6829","nonce":1179942661}]},"public_addr":"172.21.15.116:6825/1179942661","cluster_addr":"172.21.15.116:6827/1179942661","heartbeat_back_addr":"172.21.15.116:6831/1179942661","heartbeat_front_addr":"172.21.15.116:6829/1179942661","state":["exists","up"]},{"osd":4,"uuid":"132e92e4-b825-441e-965a-ca00533fa86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6801","nonce":3132444330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6803","nonce":3132444330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6807","nonce":3132444330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":3132444330},{"type":"v1","addr":"172.21.15.174:6805","nonce":3132444330}]},"public_addr":"172.21.15.174:6801/3132444330","cluster_addr":"172.21.15.174:6803/3132444330","heartbeat_back_addr":"172.21.15.174:6807/3132444330","heartbeat_front_addr":"172.21.15.174:6805/3132444330","state":["exists","up"]},{"osd":5,"uuid":"a2eeb5b6-1b31-43cc-8306-a0adf3ee15bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6809","nonce":881817164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6811","nonce":881817164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6815","nonce":881817164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":881817164},{"type":"v1","addr":"172.21.15.174:6813","nonce":881817164}]},"public_addr":"172.21.15.174:6809/881817164","cluster_addr":"172.21.15.174:6811/881817164","heartbeat_back_addr":"172.21.15.174:6815/881817164","heartbeat_front_addr":"172.21.15.174:6813/881817164","state":["exists","up"]},{"osd":6,"uuid":"588e873a-c841-4ba7-90f2-8956c87a9983","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6825","nonce":117574251}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6827","nonce":117574251}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6831","nonce":117574251}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":117574251},{"type":"v1","addr":"172.21.15.174:6829","nonce":117574251}]},"public_addr":"172.21.15.174:6825/117574251","cluster_addr":"172.21.15.174:6827/117574251","heartbeat_back_addr":"172.21.15.174:6831/117574251","heartbeat_front_addr":"172.21.15.174:6829/117574251","state":["exists","up"]},{"osd":7,"uuid":"3b90f9b0-8a0e-4883-b088-ca1d4012870c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6817","nonce":2543899196}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6819","nonce":2543899196}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6823","nonce":2543899196}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":2543899196},{"type":"v1","addr":"172.21.15.174:6821","nonce":2543899196}]},"public_addr":"172.21.15.174:6817/2543899196","cluster_addr":"172.21.15.174:6819/2543899196","heartbeat_back_addr":"172.21.15.174:6823/2543899196","heartbeat_front_addr":"172.21.15.174:6821/2543899196","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.293769+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.310034+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.404865+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.549658+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.192174+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.297266+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.493649+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T00:00:48.452309+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"5.3","mappings":[{"from":4,"to":5}]},{"pgid":"5.5","mappings":[{"from":4,"to":5}]},{"pgid":"5.14","mappings":[{"from":4,"to":5}]},{"pgid":"5.19","mappings":[{"from":4,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.116:6835/3499878473":"2024-04-26T00:30:15.781624+0000","172.21.15.116:6834/3499878473":"2024-04-26T00:30:15.781624+0000","172.21.15.174:6836/2915274073":"2024-04-26T00:29:50.776462+0000","172.21.15.116:6833/3891103804":"2024-04-26T00:28:50.767503+0000","172.21.15.116:6834/2371126056":"2024-04-26T00:27:35.754759+0000","172.21.15.174:6837/349772087":"2024-04-26T00:27:15.748274+0000","172.21.15.174:6836/349772087":"2024-04-26T00:27:15.748274+0000","172.21.15.116:6833/3693049068":"2024-04-26T00:26:45.739776+0000","172.21.15.116:6832/3693049068":"2024-04-26T00:26:45.739776+0000","172.21.15.174:6837/4181466530":"2024-04-26T00:26:15.734586+0000","172.21.15.174:6836/4181466530":"2024-04-26T00:26:15.734586+0000","172.21.15.174:6836/492366509":"2024-04-26T00:24:15.701887+0000","172.21.15.116:6832/2998042514":"2024-04-26T00:23:30.691827+0000","172.21.15.116:6833/663910035":"2024-04-26T00:22:35.677323+0000","172.21.15.116:6832/663910035":"2024-04-26T00:22:35.677323+0000","172.21.15.174:6837/321466844":"2024-04-26T00:22:10.671673+0000","172.21.15.116:6835/980221971":"2024-04-26T00:21:40.665115+0000","172.21.15.174:6837/991183935":"2024-04-26T00:20:50.654151+0000","172.21.15.174:6836/991183935":"2024-04-26T00:20:50.654151+0000","172.21.15.174:6836/321466844":"2024-04-26T00:22:10.671673+0000","172.21.15.116:6834/1358527848":"2024-04-26T00:11:00.511032+0000","172.21.15.116:6832/3841609112":"2024-04-26T00:07:00.448498+0000","172.21.15.116:6835/1670061091":"2024-04-26T00:09:30.490485+0000","172.21.15.116:6834/1971675962":"2024-04-26T00:25:25.721965+0000","172.21.15.116:6835/1233829678":"2024-04-26T00:07:30.454393+0000","172.21.15.116:6835/2371126056":"2024-04-26T00:27:35.754759+0000","172.21.15.174:6834/2883220846":"2024-04-26T00:11:50.521053+0000","172.21.15.116:6834/1233829678":"2024-04-26T00:07:30.454393+0000","172.21.15.116:6833/2662028892":"2024-04-26T00:31:30.022572+0000","172.21.15.174:6837/3180024481":"2024-04-26T00:15:25.567768+0000","172.21.15.116:6835/879531976":"2024-04-26T00:20:10.642563+0000","172.21.15.116:6833/174781672":"2024-04-26T00:02:40.399073+0000","172.21.15.174:6835/4215220222":"2024-04-26T00:15:40.572498+0000","172.21.15.116:6834/221971311":"2024-04-26T00:22:48.898075+0000","172.21.15.174:6836/1281466397":"2024-04-26T00:18:20.613347+0000","172.21.15.116:6832/3891103804":"2024-04-26T00:28:50.767503+0000","172.21.15.116:6835/1358527848":"2024-04-26T00:11:00.511032+0000","172.21.15.116:6835/221971311":"2024-04-26T00:22:48.898075+0000","172.21.15.116:6832/3006671671":"2024-04-26T00:09:30.490406+0000","172.21.15.116:6834/980221971":"2024-04-26T00:21:40.665115+0000","172.21.15.174:6836/1762356055":"2024-04-26T00:03:36.597345+0000","172.21.15.116:6833/3006671671":"2024-04-26T00:09:30.490406+0000","172.21.15.116:6834/1459177921":"2024-04-26T00:03:30.406778+0000","172.21.15.116:6834/462190523":"2024-04-26T00:14:55.560782+0000","172.21.15.116:6834/879531976":"2024-04-26T00:20:10.642563+0000","172.21.15.116:6832/1459177921":"2024-04-26T00:03:30.406778+0000","172.21.15.174:6837/1159002662":"2024-04-26T00:10:20.502218+0000","172.21.15.174:6837/1762356055":"2024-04-26T00:03:36.597345+0000","172.21.15.174:6834/4215220222":"2024-04-26T00:15:40.572498+0000","172.21.15.116:6835/1971675962":"2024-04-26T00:25:25.721965+0000","172.21.15.174:6836/3081407040":"2024-04-26T00:17:25.598516+0000","172.21.15.116:6834/1599708526":"2024-04-26T00:12:20.526742+0000","172.21.15.174:6836/2276529656":"2024-04-26T00:19:05.629048+0000","172.21.15.174:6836/1159002662":"2024-04-26T00:10:20.502218+0000","172.21.15.116:6833/2998042514":"2024-04-26T00:23:30.691827+0000","172.21.15.174:6837/3081407040":"2024-04-26T00:17:25.598516+0000","172.21.15.174:6835/2883220846":"2024-04-26T00:11:50.521053+0000","172.21.15.174:6837/492366509":"2024-04-26T00:24:15.701887+0000","172.21.15.116:6835/2004876235":"2024-04-26T00:04:45.421611+0000","172.21.15.116:6835/174781672":"2024-04-26T00:02:40.399073+0000","172.21.15.116:6833/3633657268":"2024-04-26T00:25:40.724634+0000","172.21.15.116:6832/365773583":"2024-04-26T00:14:58.857970+0000","172.21.15.174:6836/1889314958":"2024-04-26T00:06:30.441408+0000","172.21.15.116:6833/3705654249":"2024-04-26T00:12:45.533219+0000","172.21.15.116:6832/3705654249":"2024-04-26T00:12:45.533219+0000","172.21.15.116:6834/1670061091":"2024-04-26T00:09:30.490485+0000","172.21.15.116:6835/1599708526":"2024-04-26T00:12:20.526742+0000","172.21.15.116:6832/3633657268":"2024-04-26T00:25:40.724634+0000","172.21.15.174:6836/3583791822":"2024-04-26T00:08:00.460587+0000","172.21.15.174:6837/3583791822":"2024-04-26T00:08:00.460587+0000","172.21.15.174:6836/3398690896":"2024-04-26T00:13:30.543729+0000","172.21.15.174:6837/3398690896":"2024-04-26T00:13:30.543729+0000","172.21.15.116:6832/2662028892":"2024-04-26T00:31:30.022572+0000","172.21.15.116:6835/462190523":"2024-04-26T00:14:55.560782+0000","172.21.15.116:6834/3336339376":"2024-04-26T00:18:35.616887+0000","172.21.15.174:6836/3180024481":"2024-04-26T00:15:25.567768+0000","172.21.15.116:6833/365773583":"2024-04-26T00:14:58.857970+0000","172.21.15.116:6835/2012470875":"2024-04-26T00:16:35.584511+0000","172.21.15.116:6833/2004876235":"2024-04-26T00:04:45.421611+0000","172.21.15.174:6837/1281466397":"2024-04-26T00:18:20.613347+0000","172.21.15.116:6832/2340581953":"2024-04-26T00:16:45.587651+0000","172.21.15.174:6837/2915274073":"2024-04-26T00:29:50.776462+0000","172.21.15.116:6834/1649769369":"2024-04-26T00:04:35.418892+0000","172.21.15.116:6832/1649769369":"2024-04-26T00:04:35.418892+0000","172.21.15.116:6833/2340581953":"2024-04-26T00:16:45.587651+0000","172.21.15.174:6837/1889314958":"2024-04-26T00:06:30.441408+0000","172.21.15.116:6832/3645553116":"2024-04-26T00:17:35.601559+0000","172.21.15.116:6833/3645553116":"2024-04-26T00:17:35.601559+0000","172.21.15.116:6833/3841609112":"2024-04-26T00:07:00.448498+0000","172.21.15.116:6834/2012470875":"2024-04-26T00:16:35.584511+0000","172.21.15.116:6835/3336339376":"2024-04-26T00:18:35.616887+0000","172.21.15.174:6837/2276529656":"2024-04-26T00:19:05.629048+0000","172.21.15.116:6832/1655959433":"2024-04-26T00:19:30.633957+0000","172.21.15.116:6833/1655959433":"2024-04-26T00:19:30.633957+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T00:31:34.266 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.266+0000 7fc8b1a8f640 1 -- 172.21.15.116:0/1078889485 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc8980678f0 msgr2=0x7fc898069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:34.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.266+0000 7fc8b1a8f640 1 --2- 172.21.15.116:0/1078889485 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc8980678f0 0x7fc898069db0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fc8a00af000 tx=0x7fc8a0008040 comp rx=0 tx=0).stop 2024-04-25T00:31:34.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.266+0000 7fc8b1a8f640 1 -- 172.21.15.116:0/1078889485 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc894004810 msgr2=0x7fc8ac0a88c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:34.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.266+0000 7fc8b1a8f640 1 --2- 172.21.15.116:0/1078889485 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc894004810 0x7fc8ac0a88c0 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7fc89c00d4a0 tx=0x7fc89c00d960 comp rx=0 tx=0).stop 2024-04-25T00:31:34.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.266+0000 7fc8b1a8f640 1 -- 172.21.15.116:0/1078889485 shutdown_connections 2024-04-25T00:31:34.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.266+0000 7fc8b1a8f640 1 --2- 172.21.15.116:0/1078889485 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7fc8980678f0 0x7fc898069db0 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:34.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.266+0000 7fc8b1a8f640 1 --2- 172.21.15.116:0/1078889485 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7fc8ac0c11e0 0x7fc8ac1ebab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:34.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.266+0000 7fc8b1a8f640 1 --2- 172.21.15.116:0/1078889485 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7fc8ac0c6570 0x7fc8ac1eb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:34.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.266+0000 7fc8b1a8f640 1 --2- 172.21.15.116:0/1078889485 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7fc894004810 0x7fc8ac0a88c0 unknown :-1 s=CLOSED pgs=1900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:34.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.266+0000 7fc8b1a8f640 1 -- 172.21.15.116:0/1078889485 >> 172.21.15.116:0/1078889485 conn(0x7fc8ac0c4270 msgr2=0x7fc8ac152760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:31:34.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.266+0000 7fc8b1a8f640 1 -- 172.21.15.116:0/1078889485 shutdown_connections 2024-04-25T00:31:34.267 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.266+0000 7fc8b1a8f640 1 -- 172.21.15.116:0/1078889485 wait complete. 2024-04-25T00:31:34.282 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail b 2024-04-25T00:31:34.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.386+0000 7ff14148e640 1 Processor -- start 2024-04-25T00:31:34.388 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.386+0000 7ff14148e640 1 -- start start 2024-04-25T00:31:34.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.386+0000 7ff14148e640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff13c13e810 0x7ff13c1560d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:34.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.386+0000 7ff14148e640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff13c06d200 con 0x7ff13c13ed60 2024-04-25T00:31:34.390 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.386+0000 7ff14148e640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff13c156610 con 0x7ff13c13e310 2024-04-25T00:31:34.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.386+0000 7ff14148e640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7ff13c070d40 con 0x7ff13c13e810 2024-04-25T00:31:34.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.386+0000 7ff13b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff13c13e810 0x7ff13c1560d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:34.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.386+0000 7ff13b7fe640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7ff13c13e810 0x7ff13c1560d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51370/0 (socket says 172.21.15.116:51370) 2024-04-25T00:31:34.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.386+0000 7ff13b7fe640 1 -- 172.21.15.116:0/3103228178 learned_addr learned my addr 172.21.15.116:0/3103228178 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:31:34.391 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.386+0000 7ff13b7fe640 1 -- 172.21.15.116:0/3103228178 >> v1:172.21.15.174:6789/0 conn(0x7ff13c13e310 legacy=0x7ff13c0c2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:31:34.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.386+0000 7ff13b7fe640 1 -- 172.21.15.116:0/3103228178 >> v1:172.21.15.116:6789/0 conn(0x7ff13c13ed60 legacy=0x7ff13c1559c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:34.392 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.390+0000 7ff13b7fe640 1 -- 172.21.15.116:0/3103228178 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff13c06cb50 con 0x7ff13c13e810 2024-04-25T00:31:34.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.390+0000 7ff13b7fe640 1 --2- 172.21.15.116:0/3103228178 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff13c13e810 0x7ff13c1560d0 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7ff134009e50 tx=0x7ff13401c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3587697b441f6a68 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:34.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.390+0000 7ff139ffb640 1 -- 172.21.15.116:0/3103228178 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1340a56e0 con 0x7ff13c13e810 2024-04-25T00:31:34.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.390+0000 7ff139ffb640 1 -- 172.21.15.116:0/3103228178 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7ff1340a5880 con 0x7ff13c13e810 2024-04-25T00:31:34.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.390+0000 7ff139ffb640 1 -- 172.21.15.116:0/3103228178 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1340a5c70 con 0x7ff13c13e810 2024-04-25T00:31:34.400 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.390+0000 7ff14148e640 1 -- 172.21.15.116:0/3103228178 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff13c13e810 msgr2=0x7ff13c1560d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:34.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.390+0000 7ff14148e640 1 --2- 172.21.15.116:0/3103228178 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff13c13e810 0x7ff13c1560d0 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7ff134009e50 tx=0x7ff13401c910 comp rx=0 tx=0).stop 2024-04-25T00:31:34.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.390+0000 7ff14148e640 1 -- 172.21.15.116:0/3103228178 shutdown_connections 2024-04-25T00:31:34.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.390+0000 7ff14148e640 1 --2- 172.21.15.116:0/3103228178 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff13c13e810 0x7ff13c1560d0 unknown :-1 s=CLOSED pgs=1902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:34.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.390+0000 7ff14148e640 1 -- 172.21.15.116:0/3103228178 >> 172.21.15.116:0/3103228178 conn(0x7ff13c0c3b00 msgr2=0x7ff13c0c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:31:34.401 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.394+0000 7ff14148e640 1 -- 172.21.15.116:0/3103228178 shutdown_connections 2024-04-25T00:31:34.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.394+0000 7ff14148e640 1 -- 172.21.15.116:0/3103228178 wait complete. 2024-04-25T00:31:34.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.394+0000 7ff14148e640 1 Processor -- start 2024-04-25T00:31:34.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.394+0000 7ff14148e640 1 -- start start 2024-04-25T00:31:34.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.394+0000 7ff14148e640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff13c13e310 0x7ff13c13cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:34.402 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.394+0000 7ff14148e640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff13c13e810 0x7ff13c139c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:34.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.394+0000 7ff14148e640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff13c13ed60 0x7ff13c13a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:34.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.394+0000 7ff14148e640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7ff13c163a00 con 0x7ff13c13e310 2024-04-25T00:31:34.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.394+0000 7ff14148e640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7ff13c071020 con 0x7ff13c13e810 2024-04-25T00:31:34.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.394+0000 7ff14148e640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7ff13c163b80 con 0x7ff13c13ed60 2024-04-25T00:31:34.403 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.394+0000 7ff13affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff13c13e310 0x7ff13c13cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:34.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.394+0000 7ff13affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff13c13e310 0x7ff13c13cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51386/0 (socket says 172.21.15.116:51386) 2024-04-25T00:31:34.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.394+0000 7ff13affd640 1 -- 172.21.15.116:0/2561326882 learned_addr learned my addr 172.21.15.116:0/2561326882 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:31:34.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.394+0000 7ff13a7fc640 1 --2- 172.21.15.116:0/2561326882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff13c13e810 0x7ff13c139c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:34.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.394+0000 7ff13affd640 1 -- 172.21.15.116:0/2561326882 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff13c13ed60 msgr2=0x7ff13c13a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:34.404 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.394+0000 7ff13b7fe640 1 --2- 172.21.15.116:0/2561326882 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff13c13ed60 0x7ff13c13a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:34.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.394+0000 7ff13affd640 1 --2- 172.21.15.116:0/2561326882 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff13c13ed60 0x7ff13c13a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:34.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.394+0000 7ff13affd640 1 -- 172.21.15.116:0/2561326882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff13c13e810 msgr2=0x7ff13c139c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:34.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.394+0000 7ff13affd640 1 --2- 172.21.15.116:0/2561326882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff13c13e810 0x7ff13c139c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:34.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.394+0000 7ff13affd640 1 -- 172.21.15.116:0/2561326882 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff13c0c57a0 con 0x7ff13c13e310 2024-04-25T00:31:34.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.394+0000 7ff13affd640 1 --2- 172.21.15.116:0/2561326882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff13c13e310 0x7ff13c13cd60 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7ff13000c370 tx=0x7ff13000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:34.405 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.394+0000 7ff1237fe640 1 -- 172.21.15.116:0/2561326882 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1300052d0 con 0x7ff13c13e310 2024-04-25T00:31:34.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.394+0000 7ff1237fe640 1 -- 172.21.15.116:0/2561326882 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7ff130005470 con 0x7ff13c13e310 2024-04-25T00:31:34.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.394+0000 7ff1237fe640 1 -- 172.21.15.116:0/2561326882 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff130007280 con 0x7ff13c13e310 2024-04-25T00:31:34.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.394+0000 7ff14148e640 1 -- 172.21.15.116:0/2561326882 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff13c0c5a80 con 0x7ff13c13e310 2024-04-25T00:31:34.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.394+0000 7ff14148e640 1 -- 172.21.15.116:0/2561326882 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff13c0c6570 con 0x7ff13c13e310 2024-04-25T00:31:34.406 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.394+0000 7ff14148e640 1 -- 172.21.15.116:0/2561326882 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff13c068350 con 0x7ff13c13e310 2024-04-25T00:31:34.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.398+0000 7ff1237fe640 1 -- 172.21.15.116:0/2561326882 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7ff130007420 con 0x7ff13c13e310 2024-04-25T00:31:34.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.398+0000 7ff1237fe640 1 --2- 172.21.15.116:0/2561326882 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff1280678c0 0x7ff128069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:34.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.398+0000 7ff1237fe640 1 -- 172.21.15.116:0/2561326882 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 9847+0+0 (secure 0 0 0) 0x7ff130086190 con 0x7ff13c13e310 2024-04-25T00:31:34.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.398+0000 7ff13a7fc640 1 --2- 172.21.15.116:0/2561326882 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff1280678c0 0x7ff128069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:34.407 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.398+0000 7ff13a7fc640 1 --2- 172.21.15.116:0/2561326882 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff1280678c0 0x7ff128069d80 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7ff13c072f50 tx=0x7ff12c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:34.408 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.402+0000 7ff1237fe640 1 -- 172.21.15.116:0/2561326882 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7ff130055c80 con 0x7ff13c13e310 2024-04-25T00:31:34.586 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.586+0000 7ff14148e640 1 -- 172.21.15.116:0/2561326882 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "b"} v 0) -- 0x7ff13c0c34f0 con 0x7ff13c13e310 2024-04-25T00:31:34.590 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.590+0000 7ff1237fe640 1 -- 172.21.15.116:0/2561326882 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "b"}]=0 b marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v245) ==== 151+0+0 (secure 0 0 0) 0x7ff130055580 con 0x7ff13c13e310 2024-04-25T00:31:34.590 INFO:teuthology.orchestra.run.smithi116.stderr:b marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-25T00:31:34.592 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.590+0000 7ff14148e640 1 -- 172.21.15.116:0/2561326882 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff1280678c0 msgr2=0x7ff128069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:34.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.590+0000 7ff14148e640 1 --2- 172.21.15.116:0/2561326882 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff1280678c0 0x7ff128069d80 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7ff13c072f50 tx=0x7ff12c009040 comp rx=0 tx=0).stop 2024-04-25T00:31:34.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.590+0000 7ff14148e640 1 -- 172.21.15.116:0/2561326882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff13c13e310 msgr2=0x7ff13c13cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:34.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.590+0000 7ff14148e640 1 --2- 172.21.15.116:0/2561326882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff13c13e310 0x7ff13c13cd60 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7ff13000c370 tx=0x7ff13000c830 comp rx=0 tx=0).stop 2024-04-25T00:31:34.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.590+0000 7ff14148e640 1 -- 172.21.15.116:0/2561326882 shutdown_connections 2024-04-25T00:31:34.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.590+0000 7ff14148e640 1 --2- 172.21.15.116:0/2561326882 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7ff1280678c0 0x7ff128069d80 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:34.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.590+0000 7ff14148e640 1 --2- 172.21.15.116:0/2561326882 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7ff13c13ed60 0x7ff13c13a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:34.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.590+0000 7ff14148e640 1 --2- 172.21.15.116:0/2561326882 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7ff13c13e810 0x7ff13c139c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:34.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.590+0000 7ff14148e640 1 --2- 172.21.15.116:0/2561326882 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7ff13c13e310 0x7ff13c13cd60 unknown :-1 s=CLOSED pgs=1903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:34.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.590+0000 7ff14148e640 1 -- 172.21.15.116:0/2561326882 >> 172.21.15.116:0/2561326882 conn(0x7ff13c0c3b00 msgr2=0x7ff13c160760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:31:34.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.590+0000 7ff14148e640 1 -- 172.21.15.116:0/2561326882 shutdown_connections 2024-04-25T00:31:34.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.590+0000 7ff14148e640 1 -- 172.21.15.116:0/2561326882 wait complete. 2024-04-25T00:31:34.610 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm b --yes-i-really-mean-it 2024-04-25T00:31:34.648 INFO:tasks.ceph.mds.d.smithi174.stdout:starting mds.d at 2024-04-25T00:31:34.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.722+0000 7f4314fd9640 1 Processor -- start 2024-04-25T00:31:34.723 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.722+0000 7f4314fd9640 1 -- start start 2024-04-25T00:31:34.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.722+0000 7f4314fd9640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f43100a9630 0x7f4310156e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:34.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.722+0000 7f4314fd9640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43100a9cb0 con 0x7f4310150d60 2024-04-25T00:31:34.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.722+0000 7f4314fd9640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43100a9f90 con 0x7f4310157340 2024-04-25T00:31:34.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.722+0000 7f4314fd9640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f43100aa270 con 0x7f43100a9630 2024-04-25T00:31:34.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.722+0000 7f430e59c640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f4310150d60 0x7f43100c2930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:46902/0 (socket says 172.21.15.116:46902) 2024-04-25T00:31:34.726 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.722+0000 7f430e59c640 1 -- 172.21.15.116:0/4146941416 learned_addr learned my addr 172.21.15.116:0/4146941416 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:31:34.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.722+0000 7f430dd9b640 1 --2- 172.21.15.116:0/4146941416 >> v2:172.21.15.174:3300/0 conn(0x7f43100a9630 0x7f4310156e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:34.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.722+0000 7f430d59a640 1 -- 172.21.15.116:0/4146941416 <== mon.0 v1:172.21.15.116:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1427669953 0 0) 0x7f43100a9cb0 con 0x7f4310150d60 2024-04-25T00:31:34.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.722+0000 7f430d59a640 1 -- 172.21.15.116:0/4146941416 --> v1:172.21.15.116:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42f8003530 con 0x7f4310150d60 2024-04-25T00:31:34.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.722+0000 7f430d59a640 1 -- 172.21.15.116:0/4146941416 <== mon.0 v1:172.21.15.116:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1416651115 0 0) 0x7f42f8003530 con 0x7f4310150d60 2024-04-25T00:31:34.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.722+0000 7f430d59a640 1 -- 172.21.15.116:0/4146941416 >> v2:172.21.15.174:3300/0 conn(0x7f43100a9630 msgr2=0x7f4310156e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:31:34.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.722+0000 7f430d59a640 1 --2- 172.21.15.116:0/4146941416 >> v2:172.21.15.174:3300/0 conn(0x7f43100a9630 0x7f4310156e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:34.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.722+0000 7f430d59a640 1 -- 172.21.15.116:0/4146941416 >> v1:172.21.15.174:6789/0 conn(0x7f4310157340 legacy=0x7f431015df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:34.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.722+0000 7f430d59a640 1 -- 172.21.15.116:0/4146941416 --> v1:172.21.15.116:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43100aa550 con 0x7f4310150d60 2024-04-25T00:31:34.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.726+0000 7f430d59a640 1 -- 172.21.15.116:0/4146941416 <== mon.0 v1:172.21.15.116:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 978444654 0 0) 0x7f4300001680 con 0x7f4310150d60 2024-04-25T00:31:34.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.726+0000 7f430d59a640 1 -- 172.21.15.116:0/4146941416 >> v1:172.21.15.116:6789/0 conn(0x7f4310150d60 legacy=0x7f43100c2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:34.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.726+0000 7f430d59a640 1 --2- 172.21.15.116:0/4146941416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42f80039c0 0x7f42f8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:34.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.726+0000 7f430d59a640 1 --2- 172.21.15.116:0/4146941416 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f42f8004810 0x7f42f8004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:34.727 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.726+0000 7f430d59a640 1 --2- 172.21.15.116:0/4146941416 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f42f80052f0 0x7f42f800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:34.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.726+0000 7f430d59a640 1 -- 172.21.15.116:0/4146941416 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f43100aa270 con 0x7f42f80039c0 2024-04-25T00:31:34.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.726+0000 7f430d59a640 1 -- 172.21.15.116:0/4146941416 --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f42f800c1c0 con 0x7f42f8004810 2024-04-25T00:31:34.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.726+0000 7f430d59a640 1 -- 172.21.15.116:0/4146941416 --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f42f800c3c0 con 0x7f42f80052f0 2024-04-25T00:31:34.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.726+0000 7f430e59c640 1 --2- 172.21.15.116:0/4146941416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42f80039c0 0x7f42f8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:34.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.726+0000 7f430ed9d640 1 --2- 172.21.15.116:0/4146941416 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f42f80052f0 0x7f42f800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:34.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.726+0000 7f430e59c640 1 -- 172.21.15.116:0/4146941416 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f42f80052f0 msgr2=0x7f42f800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:31:34.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.726+0000 7f430e59c640 1 --2- 172.21.15.116:0/4146941416 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f42f80052f0 0x7f42f800b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:34.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.726+0000 7f430e59c640 1 -- 172.21.15.116:0/4146941416 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f42f8004810 msgr2=0x7f42f8004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:31:34.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.726+0000 7f430e59c640 1 --2- 172.21.15.116:0/4146941416 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f42f8004810 0x7f42f8004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:34.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.726+0000 7f430e59c640 1 -- 172.21.15.116:0/4146941416 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42f8003530 con 0x7f42f80039c0 2024-04-25T00:31:34.728 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.726+0000 7f430ed9d640 1 -- 172.21.15.116:0/4146941416 reap_dead start 2024-04-25T00:31:34.729 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.726+0000 7f430e59c640 1 --2- 172.21.15.116:0/4146941416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42f80039c0 0x7f42f8003de0 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7f430000c190 tx=0x7f430001cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7c0d4080a40d01c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:34.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.726+0000 7f430d59a640 1 -- 172.21.15.116:0/4146941416 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4300007d00 con 0x7f42f80039c0 2024-04-25T00:31:34.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.726+0000 7f430d59a640 1 -- 172.21.15.116:0/4146941416 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f43000a9020 con 0x7f42f80039c0 2024-04-25T00:31:34.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.726+0000 7f430d59a640 1 -- 172.21.15.116:0/4146941416 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4300005b20 con 0x7f42f80039c0 2024-04-25T00:31:34.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.726+0000 7f4314fd9640 1 -- 172.21.15.116:0/4146941416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42f80039c0 msgr2=0x7f42f8003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:34.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.726+0000 7f4314fd9640 1 --2- 172.21.15.116:0/4146941416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42f80039c0 0x7f42f8003de0 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7f430000c190 tx=0x7f430001cbf0 comp rx=0 tx=0).stop 2024-04-25T00:31:34.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.726+0000 7f4314fd9640 1 -- 172.21.15.116:0/4146941416 shutdown_connections 2024-04-25T00:31:34.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.726+0000 7f4314fd9640 1 --2- 172.21.15.116:0/4146941416 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f42f80039c0 0x7f42f8003de0 unknown :-1 s=CLOSED pgs=1905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:34.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.726+0000 7f4314fd9640 1 -- 172.21.15.116:0/4146941416 >> 172.21.15.116:0/4146941416 conn(0x7f43100c3a20 msgr2=0x7f43100c3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:31:34.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.726+0000 7f4314fd9640 1 -- 172.21.15.116:0/4146941416 shutdown_connections 2024-04-25T00:31:34.730 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.726+0000 7f4314fd9640 1 -- 172.21.15.116:0/4146941416 wait complete. 2024-04-25T00:31:34.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.730+0000 7f4314fd9640 1 Processor -- start 2024-04-25T00:31:34.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.730+0000 7f4314fd9640 1 -- start start 2024-04-25T00:31:34.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.730+0000 7f4314fd9640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f42f80039c0 0x7f43100c05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:34.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.730+0000 7f4314fd9640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f431006da10 0x7f4310140e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:34.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.730+0000 7f4314fd9640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4310141340 0x7f43101ef430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:34.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.730+0000 7f4314fd9640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f43100ab7a0 con 0x7f431006da10 2024-04-25T00:31:34.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.730+0000 7f4314fd9640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f43100721e0 con 0x7f4310141340 2024-04-25T00:31:34.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.730+0000 7f4314fd9640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f4310157b60 con 0x7f42f80039c0 2024-04-25T00:31:34.733 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.730+0000 7f430dd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f431006da10 0x7f4310140e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:34.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.730+0000 7f430dd9b640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f431006da10 0x7f4310140e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:51410/0 (socket says 172.21.15.116:51410) 2024-04-25T00:31:34.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.730+0000 7f430dd9b640 1 -- 172.21.15.116:0/1440794390 learned_addr learned my addr 172.21.15.116:0/1440794390 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:31:34.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.730+0000 7f430ed9d640 1 --2- 172.21.15.116:0/1440794390 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4310141340 0x7f43101ef430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:34.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.730+0000 7f430e59c640 1 --2- 172.21.15.116:0/1440794390 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f42f80039c0 0x7f43100c05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:34.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.730+0000 7f430dd9b640 1 -- 172.21.15.116:0/1440794390 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f42f80039c0 msgr2=0x7f43100c05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:34.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.730+0000 7f430dd9b640 1 --2- 172.21.15.116:0/1440794390 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f42f80039c0 0x7f43100c05b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:34.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.730+0000 7f430dd9b640 1 -- 172.21.15.116:0/1440794390 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4310141340 msgr2=0x7f43101ef430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:34.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.730+0000 7f430dd9b640 1 --2- 172.21.15.116:0/1440794390 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4310141340 0x7f43101ef430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:34.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.730+0000 7f430dd9b640 1 -- 172.21.15.116:0/1440794390 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43101f3190 con 0x7f431006da10 2024-04-25T00:31:34.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.730+0000 7f430dd9b640 1 --2- 172.21.15.116:0/1440794390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f431006da10 0x7f4310140e00 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f430400e770 tx=0x7f430400ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:34.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.730+0000 7f42feffd640 1 -- 172.21.15.116:0/1440794390 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4304017070 con 0x7f431006da10 2024-04-25T00:31:34.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.730+0000 7f42feffd640 1 -- 172.21.15.116:0/1440794390 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f430400ee80 con 0x7f431006da10 2024-04-25T00:31:34.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.730+0000 7f4314fd9640 1 -- 172.21.15.116:0/1440794390 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43101f3470 con 0x7f431006da10 2024-04-25T00:31:34.734 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.730+0000 7f42feffd640 1 -- 172.21.15.116:0/1440794390 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f430400cb00 con 0x7f431006da10 2024-04-25T00:31:34.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.730+0000 7f4314fd9640 1 -- 172.21.15.116:0/1440794390 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43100c0af0 con 0x7f431006da10 2024-04-25T00:31:34.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.730+0000 7f4314fd9640 1 -- 172.21.15.116:0/1440794390 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43101f3690 con 0x7f431006da10 2024-04-25T00:31:34.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.734+0000 7f42feffd640 1 -- 172.21.15.116:0/1440794390 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f430401a070 con 0x7f431006da10 2024-04-25T00:31:34.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.734+0000 7f42feffd640 1 --2- 172.21.15.116:0/1440794390 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f42f00678f0 0x7f42f0069db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:34.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.734+0000 7f42feffd640 1 -- 172.21.15.116:0/1440794390 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 9933+0+0 (secure 0 0 0) 0x7f4304086a40 con 0x7f431006da10 2024-04-25T00:31:34.735 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.734+0000 7f430e59c640 1 --2- 172.21.15.116:0/1440794390 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f42f00678f0 0x7f42f0069db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:34.736 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.734+0000 7f430e59c640 1 --2- 172.21.15.116:0/1440794390 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f42f00678f0 0x7f42f0069db0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f43000aa000 tx=0x7f43100a9e80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:34.738 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.738+0000 7f42feffd640 1 -- 172.21.15.116:0/1440794390 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f43101f3690 con 0x7f431006da10 2024-04-25T00:31:34.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:34.918+0000 7f4314fd9640 1 -- 172.21.15.116:0/1440794390 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "b", "yes_i_really_mean_it": true} v 0) -- 0x7f431004f510 con 0x7f431006da10 2024-04-25T00:31:35.593 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.590+0000 7f42feffd640 1 -- 172.21.15.116:0/1440794390 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "b", "yes_i_really_mean_it": true}]=0 v246) ==== 99+0+0 (secure 0 0 0) 0x7f4304086e20 con 0x7f431006da10 2024-04-25T00:31:35.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.594+0000 7f4314fd9640 1 -- 172.21.15.116:0/1440794390 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f42f00678f0 msgr2=0x7f42f0069db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:35.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.594+0000 7f4314fd9640 1 --2- 172.21.15.116:0/1440794390 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f42f00678f0 0x7f42f0069db0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f43000aa000 tx=0x7f43100a9e80 comp rx=0 tx=0).stop 2024-04-25T00:31:35.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.594+0000 7f4314fd9640 1 -- 172.21.15.116:0/1440794390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f431006da10 msgr2=0x7f4310140e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:35.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.594+0000 7f4314fd9640 1 --2- 172.21.15.116:0/1440794390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f431006da10 0x7f4310140e00 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f430400e770 tx=0x7f430400ec30 comp rx=0 tx=0).stop 2024-04-25T00:31:35.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.594+0000 7f4314fd9640 1 -- 172.21.15.116:0/1440794390 shutdown_connections 2024-04-25T00:31:35.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.594+0000 7f4314fd9640 1 --2- 172.21.15.116:0/1440794390 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f42f00678f0 0x7f42f0069db0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:35.597 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.594+0000 7f4314fd9640 1 --2- 172.21.15.116:0/1440794390 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f4310141340 0x7f43101ef430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:35.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.594+0000 7f4314fd9640 1 --2- 172.21.15.116:0/1440794390 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f431006da10 0x7f4310140e00 unknown :-1 s=CLOSED pgs=1906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:35.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.594+0000 7f4314fd9640 1 --2- 172.21.15.116:0/1440794390 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f42f80039c0 0x7f43100c05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:35.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.594+0000 7f4314fd9640 1 -- 172.21.15.116:0/1440794390 >> 172.21.15.116:0/1440794390 conn(0x7f43100c3a20 msgr2=0x7f4310152d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:31:35.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.594+0000 7f4314fd9640 1 -- 172.21.15.116:0/1440794390 shutdown_connections 2024-04-25T00:31:35.598 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.594+0000 7f4314fd9640 1 -- 172.21.15.116:0/1440794390 wait complete. 2024-04-25T00:31:35.617 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm b_metadata b_metadata --yes-i-really-really-mean-it 2024-04-25T00:31:35.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.738+0000 7f1def966640 1 Processor -- start 2024-04-25T00:31:35.740 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.738+0000 7f1def966640 1 -- start start 2024-04-25T00:31:35.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.738+0000 7f1def966640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1de8153f10 0x7f1de815ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:35.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.738+0000 7f1def966640 1 -- --> v1:172.21.15.116:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1de80ab2f0 con 0x7f1de80c24c0 2024-04-25T00:31:35.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.738+0000 7f1def966640 1 -- --> v1:172.21.15.174:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1de814a400 con 0x7f1de814a020 2024-04-25T00:31:35.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.738+0000 7f1def966640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f1de80ac480 con 0x7f1de8153f10 2024-04-25T00:31:35.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.738+0000 7f1dedf03640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f1de8153f10 0x7f1de815ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:35.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.738+0000 7f1decf01640 1 --1- >> v1:172.21.15.116:6789/0 conn(0x7f1de80c24c0 0x7f1de80a94a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.116:6789/0 says I am v1:172.21.15.116:49208/0 (socket says 172.21.15.116:49208) 2024-04-25T00:31:35.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.738+0000 7f1decf01640 1 -- 172.21.15.116:0/4288539355 learned_addr learned my addr 172.21.15.116:0/4288539355 (peer_addr_for_me v1:172.21.15.116:0/0) 2024-04-25T00:31:35.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.738+0000 7f1dedf03640 1 -- 172.21.15.116:0/4288539355 >> v1:172.21.15.174:6789/0 conn(0x7f1de814a020 legacy=0x7f1de80abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:31:35.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.738+0000 7f1dedf03640 1 -- 172.21.15.116:0/4288539355 >> v1:172.21.15.116:6789/0 conn(0x7f1de80c24c0 legacy=0x7f1de80a94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:35.741 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.738+0000 7f1dedf03640 1 -- 172.21.15.116:0/4288539355 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1de80c1ac0 con 0x7f1de8153f10 2024-04-25T00:31:35.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.742+0000 7f1dedf03640 1 --2- 172.21.15.116:0/4288539355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1de8153f10 0x7f1de815ee70 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f1de0009d90 tx=0x7f1de001c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a029bd15a80f0e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:35.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.742+0000 7f1ddffff640 1 -- 172.21.15.116:0/4288539355 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1de00b2070 con 0x7f1de8153f10 2024-04-25T00:31:35.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.742+0000 7f1ddffff640 1 -- 172.21.15.116:0/4288539355 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f1de00a59f0 con 0x7f1de8153f10 2024-04-25T00:31:35.743 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.742+0000 7f1ddffff640 1 -- 172.21.15.116:0/4288539355 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1de00a5de0 con 0x7f1de8153f10 2024-04-25T00:31:35.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.742+0000 7f1def966640 1 -- 172.21.15.116:0/4288539355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1de8153f10 msgr2=0x7f1de815ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:35.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.742+0000 7f1def966640 1 --2- 172.21.15.116:0/4288539355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1de8153f10 0x7f1de815ee70 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7f1de0009d90 tx=0x7f1de001c930 comp rx=0 tx=0).stop 2024-04-25T00:31:35.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.742+0000 7f1def966640 1 -- 172.21.15.116:0/4288539355 shutdown_connections 2024-04-25T00:31:35.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.742+0000 7f1def966640 1 --2- 172.21.15.116:0/4288539355 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1de8153f10 0x7f1de815ee70 unknown :-1 s=CLOSED pgs=1908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:35.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.742+0000 7f1def966640 1 -- 172.21.15.116:0/4288539355 >> 172.21.15.116:0/4288539355 conn(0x7f1de806b810 msgr2=0x7f1de80c6f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:31:35.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.742+0000 7f1def966640 1 -- 172.21.15.116:0/4288539355 shutdown_connections 2024-04-25T00:31:35.744 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.742+0000 7f1def966640 1 -- 172.21.15.116:0/4288539355 wait complete. 2024-04-25T00:31:35.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.742+0000 7f1def966640 1 Processor -- start 2024-04-25T00:31:35.745 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.742+0000 7f1def966640 1 -- start start 2024-04-25T00:31:35.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.746+0000 7f1def966640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1de80c24c0 0x7f1de814c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:35.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.746+0000 7f1def966640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1de814a020 0x7f1de813cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:35.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.746+0000 7f1def966640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1de8153f10 0x7f1de813d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:35.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.746+0000 7f1def966640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f1de80c1560 con 0x7f1de814a020 2024-04-25T00:31:35.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.746+0000 7f1def966640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f1de8154340 con 0x7f1de8153f10 2024-04-25T00:31:35.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.746+0000 7f1def966640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f1de80c7ca0 con 0x7f1de80c24c0 2024-04-25T00:31:35.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.746+0000 7f1decf01640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1de814a020 0x7f1de813cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:35.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.746+0000 7f1decf01640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1de814a020 0x7f1de813cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:32994/0 (socket says 172.21.15.116:32994) 2024-04-25T00:31:35.747 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.746+0000 7f1decf01640 1 -- 172.21.15.116:0/4276004646 learned_addr learned my addr 172.21.15.116:0/4276004646 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:31:35.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.746+0000 7f1ded702640 1 --2- 172.21.15.116:0/4276004646 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1de80c24c0 0x7f1de814c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:35.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.746+0000 7f1dedf03640 1 --2- 172.21.15.116:0/4276004646 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1de8153f10 0x7f1de813d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:35.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.746+0000 7f1decf01640 1 -- 172.21.15.116:0/4276004646 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1de80c24c0 msgr2=0x7f1de814c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:35.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.746+0000 7f1decf01640 1 --2- 172.21.15.116:0/4276004646 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1de80c24c0 0x7f1de814c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:35.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.746+0000 7f1decf01640 1 -- 172.21.15.116:0/4276004646 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1de8153f10 msgr2=0x7f1de813d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:35.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.746+0000 7f1decf01640 1 --2- 172.21.15.116:0/4276004646 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1de8153f10 0x7f1de813d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:35.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.746+0000 7f1decf01640 1 -- 172.21.15.116:0/4276004646 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1de8139610 con 0x7f1de814a020 2024-04-25T00:31:35.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.746+0000 7f1dedf03640 1 --2- 172.21.15.116:0/4276004646 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1de8153f10 0x7f1de813d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T00:31:35.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.746+0000 7f1decf01640 1 --2- 172.21.15.116:0/4276004646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1de814a020 0x7f1de813cb00 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f1dd800c370 tx=0x7f1dd800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:35.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.746+0000 7f1dddffb640 1 -- 172.21.15.116:0/4276004646 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dd8017070 con 0x7f1de814a020 2024-04-25T00:31:35.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.746+0000 7f1dddffb640 1 -- 172.21.15.116:0/4276004646 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f1dd8005cf0 con 0x7f1de814a020 2024-04-25T00:31:35.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.746+0000 7f1dddffb640 1 -- 172.21.15.116:0/4276004646 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dd8007310 con 0x7f1de814a020 2024-04-25T00:31:35.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.746+0000 7f1def966640 1 -- 172.21.15.116:0/4276004646 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1de813d580 con 0x7f1de814a020 2024-04-25T00:31:35.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.746+0000 7f1def966640 1 -- 172.21.15.116:0/4276004646 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1de81ec2b0 con 0x7f1de814a020 2024-04-25T00:31:35.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.746+0000 7f1def966640 1 -- 172.21.15.116:0/4276004646 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1de8068350 con 0x7f1de814a020 2024-04-25T00:31:35.749 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.746+0000 7f1dddffb640 1 -- 172.21.15.116:0/4276004646 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f1dd801a070 con 0x7f1de814a020 2024-04-25T00:31:35.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.746+0000 7f1dddffb640 1 --2- 172.21.15.116:0/4276004646 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1dcc0678c0 0x7f1dcc069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:35.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.746+0000 7f1dddffb640 1 -- 172.21.15.116:0/4276004646 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 9933+0+0 (secure 0 0 0) 0x7f1dd8086130 con 0x7f1de814a020 2024-04-25T00:31:35.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.746+0000 7f1ded702640 1 --2- 172.21.15.116:0/4276004646 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1dcc0678c0 0x7f1dcc069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:35.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.750+0000 7f1ded702640 1 --2- 172.21.15.116:0/4276004646 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1dcc0678c0 0x7f1dcc069d80 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f1de400a750 tx=0x7f1de4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:35.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.750+0000 7f1dddffb640 1 -- 172.21.15.116:0/4276004646 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f1de813d580 con 0x7f1de814a020 2024-04-25T00:31:35.914 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:35.914+0000 7f1def966640 1 -- 172.21.15.116:0/4276004646 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "b_metadata", "pool2": "b_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f1de814a7b0 con 0x7f1de814a020 2024-04-25T00:31:36.601 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.598+0000 7f1dddffb640 1 -- 172.21.15.116:0/4276004646 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "b_metadata", "pool2": "b_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'b_metadata' removed v79) ==== 166+0+0 (secure 0 0 0) 0x7f1de8139610 con 0x7f1de814a020 2024-04-25T00:31:36.601 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'b_metadata' removed 2024-04-25T00:31:36.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.602+0000 7f1def966640 1 -- 172.21.15.116:0/4276004646 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1dcc0678c0 msgr2=0x7f1dcc069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:36.604 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.602+0000 7f1def966640 1 --2- 172.21.15.116:0/4276004646 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1dcc0678c0 0x7f1dcc069d80 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f1de400a750 tx=0x7f1de4008040 comp rx=0 tx=0).stop 2024-04-25T00:31:36.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.602+0000 7f1def966640 1 -- 172.21.15.116:0/4276004646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1de814a020 msgr2=0x7f1de813cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:36.605 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.602+0000 7f1def966640 1 --2- 172.21.15.116:0/4276004646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1de814a020 0x7f1de813cb00 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f1dd800c370 tx=0x7f1dd800c830 comp rx=0 tx=0).stop 2024-04-25T00:31:36.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.602+0000 7f1def966640 1 -- 172.21.15.116:0/4276004646 shutdown_connections 2024-04-25T00:31:36.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.602+0000 7f1def966640 1 --2- 172.21.15.116:0/4276004646 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f1dcc0678c0 0x7f1dcc069d80 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:36.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.602+0000 7f1def966640 1 --2- 172.21.15.116:0/4276004646 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f1de8153f10 0x7f1de813d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:36.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.602+0000 7f1def966640 1 --2- 172.21.15.116:0/4276004646 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f1de814a020 0x7f1de813cb00 unknown :-1 s=CLOSED pgs=1909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:36.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.602+0000 7f1def966640 1 --2- 172.21.15.116:0/4276004646 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f1de80c24c0 0x7f1de814c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:36.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.602+0000 7f1def966640 1 -- 172.21.15.116:0/4276004646 >> 172.21.15.116:0/4276004646 conn(0x7f1de806b810 msgr2=0x7f1de8146c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:31:36.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.606+0000 7f1def966640 1 -- 172.21.15.116:0/4276004646 shutdown_connections 2024-04-25T00:31:36.606 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.606+0000 7f1def966640 1 -- 172.21.15.116:0/4276004646 wait complete. 2024-04-25T00:31:36.629 DEBUG:teuthology.orchestra.run.smithi116:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm b_data b_data --yes-i-really-really-mean-it 2024-04-25T00:31:36.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.746+0000 7f5211c51640 1 Processor -- start 2024-04-25T00:31:36.748 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.746+0000 7f5211c51640 1 -- start start 2024-04-25T00:31:36.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.746+0000 7f5211c51640 1 --2- >> v2:172.21.15.174:3300/0 conn(0x7f520c14f3d0 0x7f520c0c4e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:36.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.746+0000 7f5211c51640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f520c0ac1b0 0x7f520c0718e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:36.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.746+0000 7f5211c51640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f520c152920 0x7f520c1551e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:36.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.746+0000 7f5211c51640 1 -- --> v2:172.21.15.116:3300/0 -- mon_getmap magic: 0 -- 0x7f520c0c38a0 con 0x7f520c152920 2024-04-25T00:31:36.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.746+0000 7f5211c51640 1 -- --> v2:172.21.15.174:3300/0 -- mon_getmap magic: 0 -- 0x7f520c152490 con 0x7f520c14f3d0 2024-04-25T00:31:36.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.746+0000 7f5211c51640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f520c0c1f10 con 0x7f520c0ac1b0 2024-04-25T00:31:36.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.746+0000 7f520bfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f520c152920 0x7f520c1551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:36.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.746+0000 7f520bfff640 1 --2- >> v2:172.21.15.116:3300/0 conn(0x7f520c152920 0x7f520c1551e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33010/0 (socket says 172.21.15.116:33010) 2024-04-25T00:31:36.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.746+0000 7f520bfff640 1 -- 172.21.15.116:0/3519703029 learned_addr learned my addr 172.21.15.116:0/3519703029 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:31:36.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.750+0000 7f520b7fe640 1 --2- 172.21.15.116:0/3519703029 >> v2:172.21.15.174:3300/0 conn(0x7f520c14f3d0 0x7f520c0c4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:36.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.750+0000 7f520affd640 1 --2- 172.21.15.116:0/3519703029 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f520c0ac1b0 0x7f520c0718e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:36.750 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.750+0000 7f520bfff640 1 -- 172.21.15.116:0/3519703029 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f520c0ac1b0 msgr2=0x7f520c0718e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:31:36.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.750+0000 7f520bfff640 1 --2- 172.21.15.116:0/3519703029 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f520c0ac1b0 0x7f520c0718e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:36.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.750+0000 7f520bfff640 1 -- 172.21.15.116:0/3519703029 >> v2:172.21.15.174:3300/0 conn(0x7f520c14f3d0 msgr2=0x7f520c0c4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T00:31:36.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.750+0000 7f520bfff640 1 --2- 172.21.15.116:0/3519703029 >> v2:172.21.15.174:3300/0 conn(0x7f520c14f3d0 0x7f520c0c4e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:36.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.750+0000 7f520bfff640 1 -- 172.21.15.116:0/3519703029 --> v2:172.21.15.116:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f520c151e80 con 0x7f520c152920 2024-04-25T00:31:36.751 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.750+0000 7f520b7fe640 1 --2- 172.21.15.116:0/3519703029 >> v2:172.21.15.174:3300/0 conn(0x7f520c14f3d0 0x7f520c0c4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T00:31:36.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.750+0000 7f520bfff640 1 --2- 172.21.15.116:0/3519703029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f520c152920 0x7f520c1551e0 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7f520c06c0f0 tx=0x7f520401c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72a572f9cdaae81c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:36.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.750+0000 7f520a7fc640 1 -- 172.21.15.116:0/3519703029 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52040ab5f0 con 0x7f520c152920 2024-04-25T00:31:36.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.750+0000 7f520a7fc640 1 -- 172.21.15.116:0/3519703029 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f52040b0070 con 0x7f520c152920 2024-04-25T00:31:36.752 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.750+0000 7f520a7fc640 1 -- 172.21.15.116:0/3519703029 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52040ab9d0 con 0x7f520c152920 2024-04-25T00:31:36.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.750+0000 7f5211c51640 1 -- 172.21.15.116:0/3519703029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f520c152920 msgr2=0x7f520c1551e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:36.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.750+0000 7f5211c51640 1 --2- 172.21.15.116:0/3519703029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f520c152920 0x7f520c1551e0 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7f520c06c0f0 tx=0x7f520401c520 comp rx=0 tx=0).stop 2024-04-25T00:31:36.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.750+0000 7f5211c51640 1 -- 172.21.15.116:0/3519703029 shutdown_connections 2024-04-25T00:31:36.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.750+0000 7f5211c51640 1 --2- 172.21.15.116:0/3519703029 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f520c152920 0x7f520c1551e0 unknown :-1 s=CLOSED pgs=1910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:36.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.750+0000 7f5211c51640 1 --2- 172.21.15.116:0/3519703029 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f520c0ac1b0 0x7f520c0718e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:36.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.750+0000 7f5211c51640 1 --2- 172.21.15.116:0/3519703029 >> v2:172.21.15.174:3300/0 conn(0x7f520c14f3d0 0x7f520c0c4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:36.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.750+0000 7f5211c51640 1 -- 172.21.15.116:0/3519703029 >> 172.21.15.116:0/3519703029 conn(0x7f520c06b810 msgr2=0x7f520c0c6f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:31:36.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.750+0000 7f5211c51640 1 -- 172.21.15.116:0/3519703029 shutdown_connections 2024-04-25T00:31:36.753 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.750+0000 7f5211c51640 1 -- 172.21.15.116:0/3519703029 wait complete. 2024-04-25T00:31:36.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.754+0000 7f5211c51640 1 Processor -- start 2024-04-25T00:31:36.754 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.754+0000 7f5211c51640 1 -- start start 2024-04-25T00:31:36.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.754+0000 7f5211c51640 1 --2- >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f520c0ac1b0 0x7f520c0aab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:36.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.754+0000 7f5211c51640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f520c14f3d0 0x7f520c14a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:36.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.754+0000 7f5211c51640 1 --2- >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f520c152920 0x7f520c1f2450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:36.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.754+0000 7f5211c51640 1 -- --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_getmap magic: 0 -- 0x7f520c071fa0 con 0x7f520c14f3d0 2024-04-25T00:31:36.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.754+0000 7f5211c51640 1 -- --> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] -- mon_getmap magic: 0 -- 0x7f520c071e20 con 0x7f520c152920 2024-04-25T00:31:36.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.754+0000 7f5211c51640 1 -- --> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] -- mon_getmap magic: 0 -- 0x7f520c072120 con 0x7f520c0ac1b0 2024-04-25T00:31:36.756 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.754+0000 7f520affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f520c14f3d0 0x7f520c14a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:36.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.754+0000 7f520affd640 1 --2- >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f520c14f3d0 0x7f520c14a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.116:3300/0 says I am v2:172.21.15.116:33014/0 (socket says 172.21.15.116:33014) 2024-04-25T00:31:36.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.754+0000 7f520affd640 1 -- 172.21.15.116:0/2033067512 learned_addr learned my addr 172.21.15.116:0/2033067512 (peer_addr_for_me v2:172.21.15.116:0/0) 2024-04-25T00:31:36.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.754+0000 7f520bfff640 1 --2- 172.21.15.116:0/2033067512 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f520c152920 0x7f520c1f2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:36.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.754+0000 7f520b7fe640 1 --2- 172.21.15.116:0/2033067512 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f520c0ac1b0 0x7f520c0aab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:36.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.754+0000 7f520affd640 1 -- 172.21.15.116:0/2033067512 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f520c0ac1b0 msgr2=0x7f520c0aab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:36.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.754+0000 7f520affd640 1 --2- 172.21.15.116:0/2033067512 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f520c0ac1b0 0x7f520c0aab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:36.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.754+0000 7f520affd640 1 -- 172.21.15.116:0/2033067512 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f520c152920 msgr2=0x7f520c1f2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:36.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.754+0000 7f520affd640 1 --2- 172.21.15.116:0/2033067512 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f520c152920 0x7f520c1f2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:36.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.754+0000 7f520affd640 1 -- 172.21.15.116:0/2033067512 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f520c0ab0a0 con 0x7f520c14f3d0 2024-04-25T00:31:36.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.754+0000 7f520affd640 1 --2- 172.21.15.116:0/2033067512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f520c14f3d0 0x7f520c14a750 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7f520000d800 tx=0x7f520000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:36.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.754+0000 7f51f3fff640 1 -- 172.21.15.116:0/2033067512 <== mon.0 v2:172.21.15.116:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52000096a0 con 0x7f520c14f3d0 2024-04-25T00:31:36.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.754+0000 7f51f3fff640 1 -- 172.21.15.116:0/2033067512 <== mon.0 v2:172.21.15.116:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f5200019070 con 0x7f520c14f3d0 2024-04-25T00:31:36.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.754+0000 7f5211c51640 1 -- 172.21.15.116:0/2033067512 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f520c1f2ca0 con 0x7f520c14f3d0 2024-04-25T00:31:36.757 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.754+0000 7f51f3fff640 1 -- 172.21.15.116:0/2033067512 <== mon.0 v2:172.21.15.116:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5200009aa0 con 0x7f520c14f3d0 2024-04-25T00:31:36.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.754+0000 7f5211c51640 1 -- 172.21.15.116:0/2033067512 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f520c1498d0 con 0x7f520c14f3d0 2024-04-25T00:31:36.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.754+0000 7f5211c51640 1 -- 172.21.15.116:0/2033067512 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f520c068350 con 0x7f520c14f3d0 2024-04-25T00:31:36.758 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.754+0000 7f51f3fff640 1 -- 172.21.15.116:0/2033067512 <== mon.0 v2:172.21.15.116:3300/0 4 ==== mgrmap(e 5) ==== 87445+0+0 (secure 0 0 0) 0x7f5200009c40 con 0x7f520c14f3d0 2024-04-25T00:31:36.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.758+0000 7f51f3fff640 1 --2- 172.21.15.116:0/2033067512 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f51f80678c0 0x7f51f8069d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T00:31:36.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.758+0000 7f51f3fff640 1 -- 172.21.15.116:0/2033067512 <== mon.0 v2:172.21.15.116:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 9505+0+0 (secure 0 0 0) 0x7f5200087090 con 0x7f520c14f3d0 2024-04-25T00:31:36.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.758+0000 7f520b7fe640 1 --2- 172.21.15.116:0/2033067512 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f51f80678c0 0x7f51f8069d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T00:31:36.759 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.758+0000 7f520b7fe640 1 --2- 172.21.15.116:0/2033067512 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f51f80678c0 0x7f51f8069d80 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f51fc00acb0 tx=0x7f51fc00c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T00:31:36.761 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.762+0000 7f51f3fff640 1 -- 172.21.15.116:0/2033067512 <== mon.0 v2:172.21.15.116:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+177576 (secure 0 0 0) 0x7f5200056ca0 con 0x7f520c14f3d0 2024-04-25T00:31:36.920 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:36.918+0000 7f5211c51640 1 -- 172.21.15.116:0/2033067512 --> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "b_data", "pool2": "b_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f520c0c6d00 con 0x7f520c14f3d0 2024-04-25T00:31:37.603 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:37.602+0000 7f51f3fff640 1 -- 172.21.15.116:0/2033067512 <== mon.0 v2:172.21.15.116:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "b_data", "pool2": "b_data", "yes_i_really_really_mean_it": true}]=0 pool 'b_data' removed v80) ==== 154+0+0 (secure 0 0 0) 0x7f52000565a0 con 0x7f520c14f3d0 2024-04-25T00:31:37.603 INFO:teuthology.orchestra.run.smithi116.stderr:pool 'b_data' removed 2024-04-25T00:31:37.607 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:37.606+0000 7f5211c51640 1 -- 172.21.15.116:0/2033067512 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f51f80678c0 msgr2=0x7f51f8069d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:37.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:37.606+0000 7f5211c51640 1 --2- 172.21.15.116:0/2033067512 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f51f80678c0 0x7f51f8069d80 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f51fc00acb0 tx=0x7f51fc00c040 comp rx=0 tx=0).stop 2024-04-25T00:31:37.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:37.606+0000 7f5211c51640 1 -- 172.21.15.116:0/2033067512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f520c14f3d0 msgr2=0x7f520c14a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T00:31:37.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:37.606+0000 7f5211c51640 1 --2- 172.21.15.116:0/2033067512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f520c14f3d0 0x7f520c14a750 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7f520000d800 tx=0x7f520000dcc0 comp rx=0 tx=0).stop 2024-04-25T00:31:37.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:37.606+0000 7f5211c51640 1 -- 172.21.15.116:0/2033067512 shutdown_connections 2024-04-25T00:31:37.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:37.606+0000 7f5211c51640 1 --2- 172.21.15.116:0/2033067512 >> [v2:172.21.15.174:6832/2004448211,v1:172.21.15.174:6833/2004448211] conn(0x7f51f80678c0 0x7f51f8069d80 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:37.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:37.606+0000 7f5211c51640 1 --2- 172.21.15.116:0/2033067512 >> [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] conn(0x7f520c152920 0x7f520c1f2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:37.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:37.606+0000 7f5211c51640 1 --2- 172.21.15.116:0/2033067512 >> [v2:172.21.15.116:3300/0,v1:172.21.15.116:6789/0] conn(0x7f520c14f3d0 0x7f520c14a750 unknown :-1 s=CLOSED pgs=1911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:37.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:37.606+0000 7f5211c51640 1 --2- 172.21.15.116:0/2033067512 >> [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] conn(0x7f520c0ac1b0 0x7f520c0aab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T00:31:37.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:37.606+0000 7f5211c51640 1 -- 172.21.15.116:0/2033067512 >> 172.21.15.116:0/2033067512 conn(0x7f520c06b810 msgr2=0x7f520c0c4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T00:31:37.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:37.606+0000 7f5211c51640 1 -- 172.21.15.116:0/2033067512 shutdown_connections 2024-04-25T00:31:37.608 INFO:teuthology.orchestra.run.smithi116.stderr:2024-04-25T00:31:37.606+0000 7f5211c51640 1 -- 172.21.15.116:0/2033067512 wait complete. 2024-04-25T00:31:37.651 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-25T00:31:37.651 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-25T00:31:37.651 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:31:37.662 INFO:tasks.ceph.mds.a:Stopped 2024-04-25T00:31:37.662 ERROR:tasks.ceph.mds.c:tried to stop a non-running daemon 2024-04-25T00:31:37.662 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-25T00:31:37.662 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:31:37.681 INFO:tasks.ceph.mds.b:Stopped 2024-04-25T00:31:37.682 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-25T00:31:37.682 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:31:37.737 INFO:tasks.ceph.mds.d:Stopped 2024-04-25T00:31:37.737 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-25T00:31:37.737 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-25T00:31:37.738 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:31:42.503 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-04-25T00:31:42.924 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:31:42.927 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:31:43.839 INFO:tasks.ceph.osd.0:Stopped 2024-04-25T00:31:43.839 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-25T00:31:43.840 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:31:46.374 INFO:tasks.ceph.mgr.x.smithi174.stderr:2024-04-25T00:31:46.371+0000 7fc0972c6640 -1 calc_pg_upmaps abort due to max <= 0 2024-04-25T00:31:48.612 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-04-25T00:31:48.612 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-04-25T00:31:49.941 INFO:tasks.ceph.osd.1:Stopped 2024-04-25T00:31:49.942 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-25T00:31:49.942 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:31:54.621 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-04-25T00:31:54.621 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-04-25T00:31:56.043 INFO:tasks.ceph.osd.2:Stopped 2024-04-25T00:31:56.044 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-25T00:31:56.044 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:32:00.529 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-04-25T00:32:00.530 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-04-25T00:32:02.146 INFO:tasks.ceph.osd.3:Stopped 2024-04-25T00:32:02.146 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-04-25T00:32:02.147 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:32:06.337 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2024-04-25T00:32:06.337 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-04-25T00:32:08.249 INFO:tasks.ceph.osd.4:Stopped 2024-04-25T00:32:08.249 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-04-25T00:32:08.250 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:32:12.044 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2024-04-25T00:32:12.044 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2024-04-25T00:32:13.035 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:32:13.038 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:32:14.352 INFO:tasks.ceph.osd.5:Stopped 2024-04-25T00:32:14.352 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-04-25T00:32:14.352 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:32:14.447 INFO:tasks.ceph.osd.6:Stopped 2024-04-25T00:32:14.448 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-04-25T00:32:14.448 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:32:17.550 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2024-04-25T00:32:17.550 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2024-04-25T00:32:20.549 INFO:tasks.ceph.osd.7:Stopped 2024-04-25T00:32:20.549 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-25T00:32:20.549 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-04-25T00:32:20.550 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:32:20.585 INFO:tasks.ceph.mgr.y:Stopped 2024-04-25T00:32:20.585 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-25T00:32:20.586 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:32:20.621 INFO:tasks.ceph.mgr.x:Stopped 2024-04-25T00:32:20.622 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-25T00:32:20.622 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-25T00:32:20.622 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:32:20.648 INFO:tasks.ceph.mon.a:Stopped 2024-04-25T00:32:20.648 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-25T00:32:20.648 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:32:20.690 INFO:tasks.ceph.mon.b:Stopped 2024-04-25T00:32:20.691 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-25T00:32:20.691 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T00:32:20.747 INFO:tasks.ceph.mon.c:Stopped 2024-04-25T00:32:20.747 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-25T00:32:20.748 DEBUG:teuthology.orchestra.run.smithi116:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'Replacing daemon mds' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-04-25T00:32:20.901 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi116.front.sepia.ceph.com 2024-04-25T00:32:20.901 DEBUG:teuthology.orchestra.run.smithi116:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-25T00:32:21.194 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi116.front.sepia.ceph.com 2024-04-25T00:32:21.195 DEBUG:teuthology.orchestra.run.smithi116:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-25T00:32:21.473 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi116.front.sepia.ceph.com 2024-04-25T00:32:21.473 DEBUG:teuthology.orchestra.run.smithi116:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-25T00:32:21.686 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi116.front.sepia.ceph.com 2024-04-25T00:32:21.686 DEBUG:teuthology.orchestra.run.smithi116:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-25T00:32:21.914 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi174.front.sepia.ceph.com 2024-04-25T00:32:21.914 DEBUG:teuthology.orchestra.run.smithi174:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-04-25T00:32:22.413 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi174.front.sepia.ceph.com 2024-04-25T00:32:22.413 DEBUG:teuthology.orchestra.run.smithi174:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-04-25T00:32:22.550 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2024-04-25T00:32:22.642 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi174.front.sepia.ceph.com 2024-04-25T00:32:22.642 DEBUG:teuthology.orchestra.run.smithi174:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-04-25T00:32:22.844 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi174.front.sepia.ceph.com 2024-04-25T00:32:22.845 DEBUG:teuthology.orchestra.run.smithi174:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-04-25T00:32:23.037 INFO:tasks.ceph:Archiving mon data... 2024-04-25T00:32:23.038 DEBUG:teuthology.misc:Transferring archived files from smithi116:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672224/data/mon.a.tgz 2024-04-25T00:32:23.038 DEBUG:teuthology.orchestra.run.smithi116:> mktemp 2024-04-25T00:32:23.057 INFO:teuthology.orchestra.run.smithi116.stdout:/tmp/tmp.7l4QghhP2f 2024-04-25T00:32:23.057 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.7l4QghhP2f 2024-04-25T00:32:23.250 DEBUG:teuthology.orchestra.run.smithi116:> sudo chmod 0666 /tmp/tmp.7l4QghhP2f 2024-04-25T00:32:23.336 DEBUG:teuthology.orchestra.remote:smithi116:/tmp/tmp.7l4QghhP2f is 548KB 2024-04-25T00:32:23.399 DEBUG:teuthology.orchestra.run.smithi116:> rm -fr /tmp/tmp.7l4QghhP2f 2024-04-25T00:32:23.404 DEBUG:teuthology.misc:Transferring archived files from smithi174:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672224/data/mon.b.tgz 2024-04-25T00:32:23.404 DEBUG:teuthology.orchestra.run.smithi174:> mktemp 2024-04-25T00:32:23.415 INFO:teuthology.orchestra.run.smithi174.stdout:/tmp/tmp.1SKPyZ7p0U 2024-04-25T00:32:23.415 DEBUG:teuthology.orchestra.run.smithi174:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.1SKPyZ7p0U 2024-04-25T00:32:23.591 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 0666 /tmp/tmp.1SKPyZ7p0U 2024-04-25T00:32:23.683 DEBUG:teuthology.orchestra.remote:smithi174:/tmp/tmp.1SKPyZ7p0U is 561KB 2024-04-25T00:32:23.704 DEBUG:teuthology.orchestra.run.smithi174:> rm -fr /tmp/tmp.1SKPyZ7p0U 2024-04-25T00:32:23.709 DEBUG:teuthology.misc:Transferring archived files from smithi174:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672224/data/mon.c.tgz 2024-04-25T00:32:23.709 DEBUG:teuthology.orchestra.run.smithi174:> mktemp 2024-04-25T00:32:23.757 INFO:teuthology.orchestra.run.smithi174.stdout:/tmp/tmp.T6ZoKQsLwU 2024-04-25T00:32:23.758 DEBUG:teuthology.orchestra.run.smithi174:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.T6ZoKQsLwU 2024-04-25T00:32:23.929 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 0666 /tmp/tmp.T6ZoKQsLwU 2024-04-25T00:32:23.994 DEBUG:teuthology.orchestra.remote:smithi174:/tmp/tmp.T6ZoKQsLwU is 561KB 2024-04-25T00:32:24.055 DEBUG:teuthology.orchestra.run.smithi174:> rm -fr /tmp/tmp.T6ZoKQsLwU 2024-04-25T00:32:24.060 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-25T00:32:24.061 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-25T00:32:24.063 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-25T00:32:24.134 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-25T00:32:24.175 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-25T00:32:24.175 INFO:tasks.ceph:Archiving crash dumps... 2024-04-25T00:32:24.177 DEBUG:teuthology.misc:Transferring archived files from smithi116:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672224/remote/smithi116/crash 2024-04-25T00:32:24.177 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-25T00:32:24.192 DEBUG:teuthology.misc:Transferring archived files from smithi174:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672224/remote/smithi174/crash 2024-04-25T00:32:24.193 DEBUG:teuthology.orchestra.run.smithi174:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-25T00:32:24.228 INFO:tasks.ceph:Shutting down logrotate 2024-04-25T00:32:24.229 DEBUG:teuthology.orchestra.run.smithi116:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:32:24.238 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T00:32:24.319 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-04-25T00:32:24.336 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-04-25T00:32:24.373 INFO:tasks.ceph:Compressing logs... 2024-04-25T00:32:24.373 DEBUG:teuthology.orchestra.run.smithi116:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T00:32:24.383 DEBUG:teuthology.orchestra.run.smithi174:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T00:32:24.409 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74048.log 2024-04-25T00:32:24.410 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47503.log 2024-04-25T00:32:24.410 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.74048.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.74048.log.gz 2024-04-25T00:32:24.410 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36228.log 2024-04-25T00:32:24.411 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49916.log 2024-04-25T00:32:24.411 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.47503.log: /var/log/ceph/ceph-client.admin.36228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46229.log 2024-04-25T00:32:24.412 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.49916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47659.log 2024-04-25T00:32:24.412 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.46229.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49916.log.gz 2024-04-25T00:32:24.413 INFO:teuthology.orchestra.run.smithi116.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47503.log.gz 2024-04-25T00:32:24.413 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82767.log 2024-04-25T00:32:24.413 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46229.log.gz 2024-04-25T00:32:24.413 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.47659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78570.log 2024-04-25T00:32:24.416 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47659.log.gz 2024-04-25T00:32:24.416 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.82767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37102.log 2024-04-25T00:32:24.416 INFO:teuthology.orchestra.run.smithi116.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.82767.log.gz 2024-04-25T00:32:24.416 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.78570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83322.log 2024-04-25T00:32:24.416 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.37102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83985.log 2024-04-25T00:32:24.416 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.78570.log.gz 2024-04-25T00:32:24.416 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.83322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76510.log 2024-04-25T00:32:24.416 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37102.log.gz 2024-04-25T00:32:24.416 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.83985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48010.log 2024-04-25T00:32:24.416 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.83322.log.gz 2024-04-25T00:32:24.416 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.83985.log.gz 2024-04-25T00:32:24.416 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.76510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41522.log 2024-04-25T00:32:24.416 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.48010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49589.log 2024-04-25T00:32:24.417 INFO:teuthology.orchestra.run.smithi116.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.76510.log.gz 2024-04-25T00:32:24.417 INFO:teuthology.orchestra.run.smithi116.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48010.log.gz 2024-04-25T00:32:24.417 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.41522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81285.log 2024-04-25T00:32:24.422 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.49589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55784.log 2024-04-25T00:32:24.422 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49589.log.gz 2024-04-25T00:32:24.422 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.81285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55679.log 2024-04-25T00:32:24.422 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.55784.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.81285.log.gz 2024-04-25T00:32:24.422 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51378.log 2024-04-25T00:32:24.422 INFO:teuthology.orchestra.run.smithi116.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.55784.log.gz 2024-04-25T00:32:24.422 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44666.log 2024-04-25T00:32:24.422 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.55679.log: /var/log/ceph/ceph-client.admin.51378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45904.log 2024-04-25T00:32:24.422 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55679.log.gz 2024-04-25T00:32:24.422 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51378.log.gz 2024-04-25T00:32:24.423 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.44666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34485.log 2024-04-25T00:32:24.423 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.45904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83881.log 2024-04-25T00:32:24.423 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40811.log 2024-04-25T00:32:24.423 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34485.log: /var/log/ceph/ceph-client.admin.83881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40466.log 2024-04-25T00:32:24.423 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.40811.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.83881.log.gz 2024-04-25T00:32:24.423 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77596.log 2024-04-25T00:32:24.423 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36228.log.gz 2024-04-25T00:32:24.423 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.40466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60425.log 2024-04-25T00:32:24.423 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.77596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49049.log 2024-04-25T00:32:24.423 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83751.log 2024-04-25T00:32:24.423 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.60425.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.77596.log.gz 2024-04-25T00:32:24.423 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35680.log 2024-04-25T00:32:24.423 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.49049.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.60425.log.gz 2024-04-25T00:32:24.423 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49049.log.gz 2024-04-25T00:32:24.423 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51108.log 2024-04-25T00:32:24.424 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.83751.log: /var/log/ceph/ceph-client.admin.35680.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.83751.log.gz 2024-04-25T00:32:24.424 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60373.log 2024-04-25T00:32:24.424 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.51108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36826.log 2024-04-25T00:32:24.450 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.33357.log 2024-04-25T00:32:24.450 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-04-25T00:32:24.450 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.1.33357.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-04-25T00:32:24.450 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-04-25T00:32:24.451 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27834.log 2024-04-25T00:32:24.451 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27786.log 2024-04-25T00:32:24.451 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.27834.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-25T00:32:24.451 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-25T00:32:24.451 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.27786.log: /var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27882.log 2024-04-25T00:32:24.451 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-04-25T00:32:24.451 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27762.log 2024-04-25T00:32:24.451 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.27882.log: /var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-25T00:32:24.451 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.27762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27738.log 2024-04-25T00:32:24.451 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-04-25T00:32:24.451 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.27738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27858.log 2024-04-25T00:32:24.451 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-04-25T00:32:24.451 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.27858.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-04-25T00:32:24.451 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27906.log 2024-04-25T00:32:24.451 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27810.log 2024-04-25T00:32:24.452 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.27906.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-04-25T00:32:24.493 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/ceph-client.admin.27810.log: /var/log/ceph/ceph-osd.4.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.27738.log.gz 2024-04-25T00:32:24.493 INFO:teuthology.orchestra.run.smithi174.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.27786.log.gz 2024-04-25T00:32:24.499 INFO:teuthology.orchestra.run.smithi174.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.27810.log.gz 2024-04-25T00:32:24.499 INFO:teuthology.orchestra.run.smithi174.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.27834.log.gz 2024-04-25T00:32:24.499 INFO:teuthology.orchestra.run.smithi174.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.27858.log.gz 2024-04-25T00:32:24.499 INFO:teuthology.orchestra.run.smithi174.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.27882.log.gz 2024-04-25T00:32:24.499 INFO:teuthology.orchestra.run.smithi174.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.27906.log.gz 2024-04-25T00:32:24.503 INFO:teuthology.orchestra.run.smithi174.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.27762.log.gz 2024-04-25T00:32:24.524 INFO:teuthology.orchestra.run.smithi174.stderr: 94.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-25T00:32:24.667 INFO:teuthology.orchestra.run.smithi116.stderr: 87.7%/var/log/ceph/ceph-client.admin.60373.log: -- replaced with /var/log/ceph/ceph-client.admin.51108.log.gz 2024-04-25T00:32:24.667 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83725.log 2024-04-25T00:32:24.668 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.36826.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.60373.log.gz 2024-04-25T00:32:24.668 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36996.log 2024-04-25T00:32:24.668 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.83725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82663.log 2024-04-25T00:32:24.668 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.36996.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.83725.log.gz 2024-04-25T00:32:24.668 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33243.log 2024-04-25T00:32:24.668 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.82663.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-04-25T00:32:24.668 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.82663.log.gz 2024-04-25T00:32:24.668 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.33243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51710.log 2024-04-25T00:32:24.668 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-mgr.y.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44666.log.gz 2024-04-25T00:32:24.668 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43305.log 2024-04-25T00:32:24.668 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.51710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41770.log 2024-04-25T00:32:24.668 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.43305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34477.log 2024-04-25T00:32:24.668 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51710.log.gz 2024-04-25T00:32:24.669 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.41770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43089.log 2024-04-25T00:32:24.669 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51241.log 2024-04-25T00:32:24.669 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.43089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45563.log 2024-04-25T00:32:24.669 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.51241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33526.log 2024-04-25T00:32:24.669 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51241.log.gz 2024-04-25T00:32:24.669 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.45563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46281.log 2024-04-25T00:32:24.669 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41770.log.gz 2024-04-25T00:32:24.669 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.33526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81933.log 2024-04-25T00:32:24.669 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36996.log.gz 2024-04-25T00:32:24.669 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.46281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44134.log 2024-04-25T00:32:24.669 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46281.log.gz 2024-04-25T00:32:24.669 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.81933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49298.log 2024-04-25T00:32:24.669 INFO:teuthology.orchestra.run.smithi116.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.81933.log.gz 2024-04-25T00:32:24.669 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.44134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48330.log 2024-04-25T00:32:24.670 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.49298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35653.log 2024-04-25T00:32:24.670 INFO:teuthology.orchestra.run.smithi116.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49298.log.gz 2024-04-25T00:32:24.670 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.48330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75728.log 2024-04-25T00:32:24.670 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.35653.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48330.log.gz 2024-04-25T00:32:24.670 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47063.log 2024-04-25T00:32:24.670 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.75728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35237.log 2024-04-25T00:32:24.670 INFO:teuthology.orchestra.run.smithi116.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.33243.log.gz 2024-04-25T00:32:24.670 INFO:teuthology.orchestra.run.smithi116.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.75728.log.gz 2024-04-25T00:32:24.670 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.47063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82119.log 2024-04-25T00:32:24.670 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47063.log.gz 2024-04-25T00:32:24.670 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.35237.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42710.log 2024-04-25T00:32:24.670 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.33526.log.gz 2024-04-25T00:32:24.670 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.82119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35810.log 2024-04-25T00:32:24.671 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.82119.log.gz 2024-04-25T00:32:24.671 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.42710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82899.log 2024-04-25T00:32:24.671 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.35810.log: /var/log/ceph/ceph-client.admin.49510.log 2024-04-25T00:32:24.671 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36124.log 2024-04-25T00:32:24.671 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.82899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54215.log 2024-04-25T00:32:24.671 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.49510.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.82899.log.gz 2024-04-25T00:32:24.671 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.36124.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35237.log.gz 2024-04-25T00:32:24.671 INFO:teuthology.orchestra.run.smithi116.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49510.log.gz 2024-04-25T00:32:24.671 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46704.log 2024-04-25T00:32:24.671 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.54215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81137.log 2024-04-25T00:32:24.671 INFO:teuthology.orchestra.run.smithi116.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.54215.log.gz 2024-04-25T00:32:24.671 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.46704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46307.log 2024-04-25T00:32:24.671 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.34485.log.gz 2024-04-25T00:32:24.671 INFO:teuthology.orchestra.run.smithi116.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34477.log.gz 2024-04-25T00:32:24.671 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.81137.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46704.log.gz 2024-04-25T00:32:24.672 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45987.log 2024-04-25T00:32:24.672 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.46307.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.81137.log.gz 2024-04-25T00:32:24.672 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49692.log 2024-04-25T00:32:24.672 INFO:teuthology.orchestra.run.smithi116.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.46307.log.gz 2024-04-25T00:32:24.672 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.45987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82374.log 2024-04-25T00:32:24.672 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49458.log 2024-04-25T00:32:24.672 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.49692.log: /var/log/ceph/ceph-client.admin.82374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44383.log 2024-04-25T00:32:24.672 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49692.log.gz 2024-04-25T00:32:24.672 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.49458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46099.log 2024-04-25T00:32:24.672 INFO:teuthology.orchestra.run.smithi116.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.82374.log.gz 2024-04-25T00:32:24.672 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43089.log.gz 2024-04-25T00:32:24.672 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.44383.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49458.log.gz 2024-04-25T00:32:24.672 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50117.log 2024-04-25T00:32:24.672 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.46099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49272.log 2024-04-25T00:32:24.673 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.50117.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.46099.log.gz 2024-04-25T00:32:24.673 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84405.log 2024-04-25T00:32:24.673 INFO:teuthology.orchestra.run.smithi116.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.43305.log.gz 2024-04-25T00:32:24.673 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50117.log.gz 2024-04-25T00:32:24.673 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.49272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45670.log 2024-04-25T00:32:24.673 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49272.log.gz 2024-04-25T00:32:24.673 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.84405.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44134.log.gz 2024-04-25T00:32:24.673 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.84405.log.gz 2024-04-25T00:32:24.673 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40991.log 2024-04-25T00:32:24.673 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.45670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36592.log 2024-04-25T00:32:24.673 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.40991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50972.log 2024-04-25T00:32:24.673 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.36592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46655.log 2024-04-25T00:32:24.673 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.50972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41549.log 2024-04-25T00:32:24.673 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50972.log.gz 2024-04-25T00:32:24.673 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.46655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34480.log 2024-04-25T00:32:24.674 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40811.log.gz 2024-04-25T00:32:24.674 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45622.log 2024-04-25T00:32:24.674 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.41549.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46655.log.gz 2024-04-25T00:32:24.674 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40496.log 2024-04-25T00:32:24.674 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.45622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46730.log 2024-04-25T00:32:24.674 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.40496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45289.log 2024-04-25T00:32:24.674 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-25T00:32:24.674 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.46730.log: /var/log/ceph/ceph-client.admin.45289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34687.log 2024-04-25T00:32:24.674 INFO:teuthology.orchestra.run.smithi116.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.46730.log.gz 2024-04-25T00:32:24.674 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74130.log 2024-04-25T00:32:24.674 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33394.log 2024-04-25T00:32:24.674 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.74130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34223.log 2024-04-25T00:32:24.674 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.33394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36462.log 2024-04-25T00:32:24.675 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.74130.log.gz 2024-04-25T00:32:24.675 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75549.log 2024-04-25T00:32:24.675 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- 88.4% /var/log/ceph/ceph-client.admin.84297.log 2024-04-25T00:32:24.675 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34480.log.gz 2024-04-25T00:32:24.675 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.36462.log: /var/log/ceph/ceph-client.admin.75549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44079.log 2024-04-25T00:32:24.675 INFO:teuthology.orchestra.run.smithi116.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45670.log.gz 2024-04-25T00:32:24.675 INFO:teuthology.orchestra.run.smithi116.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.33394.log.gz 2024-04-25T00:32:24.675 INFO:teuthology.orchestra.run.smithi116.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.34223.log.gz 2024-04-25T00:32:24.675 INFO:teuthology.orchestra.run.smithi116.stderr: 87.5%/var/log/ceph/ceph-client.admin.84297.log: -- replaced with /var/log/ceph/ceph-client.admin.75549.log.gz 2024-04-25T00:32:24.675 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78823.log 2024-04-25T00:32:24.675 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.44079.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.84297.log.gz 2024-04-25T00:32:24.675 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33998.log 2024-04-25T00:32:24.675 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.78823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45236.log 2024-04-25T00:32:24.675 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.33998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47788.log 2024-04-25T00:32:24.675 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.78823.log.gz 2024-04-25T00:32:24.676 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.45236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48278.log 2024-04-25T00:32:24.676 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.47788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46626.log 2024-04-25T00:32:24.676 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47788.log.gz 2024-04-25T00:32:24.676 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50065.log 2024-04-25T00:32:24.676 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.48278.log: /var/log/ceph/ceph-client.admin.46626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49772.log 2024-04-25T00:32:24.676 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48278.log.gz 2024-04-25T00:32:24.676 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.50065.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46626.log.gz 2024-04-25T00:32:24.676 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48737.log 2024-04-25T00:32:24.676 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.49772.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50065.log.gz 2024-04-25T00:32:24.676 INFO:teuthology.orchestra.run.smithi116.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45289.log.gz 2024-04-25T00:32:24.676 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34249.log 2024-04-25T00:32:24.676 INFO:teuthology.orchestra.run.smithi116.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.49772.log.gz 2024-04-25T00:32:24.676 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.48737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46385.log 2024-04-25T00:32:24.676 INFO:teuthology.orchestra.run.smithi116.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.48737.log.gz 2024-04-25T00:32:24.676 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41831.log 2024-04-25T00:32:24.677 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.46385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84557.log 2024-04-25T00:32:24.677 INFO:teuthology.orchestra.run.smithi116.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.34249.log.gz 2024-04-25T00:32:24.677 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.41831.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46385.log.gz 2024-04-25T00:32:24.677 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49075.log 2024-04-25T00:32:24.677 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.84557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81393.log 2024-04-25T00:32:24.677 INFO:teuthology.orchestra.run.smithi116.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41831.log.gz 2024-04-25T00:32:24.677 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.49075.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.84557.log.gz 2024-04-25T00:32:24.677 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45262.log 2024-04-25T00:32:24.677 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49075.log.gz 2024-04-25T00:32:24.677 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.81393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51215.log 2024-04-25T00:32:24.677 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.45262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36774.log 2024-04-25T00:32:24.677 INFO:teuthology.orchestra.run.smithi116.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.81393.log.gz 2024-04-25T00:32:24.677 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.51215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43564.log 2024-04-25T00:32:24.677 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.36774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60503.log 89.4% 2024-04-25T00:32:24.677 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51215.log.gz 2024-04-25T00:32:24.677 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.43564.log: /var/log/ceph/ceph-client.admin.41995.log 2024-04-25T00:32:24.678 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45262.log.gz 2024-04-25T00:32:24.678 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.60503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41627.log 2024-04-25T00:32:24.678 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.41995.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.60503.log.gz 2024-04-25T00:32:24.678 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56274.log 2024-04-25T00:32:24.678 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48038.log 2024-04-25T00:32:24.678 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.41627.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.33998.log.gz 89.5% 2024-04-25T00:32:24.678 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43564.log.gz 2024-04-25T00:32:24.678 INFO:teuthology.orchestra.run.smithi116.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41995.log.gz 2024-04-25T00:32:24.678 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.56274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84349.log 2024-04-25T00:32:24.678 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.48038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65439.log 2024-04-25T00:32:24.678 INFO:teuthology.orchestra.run.smithi116.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.56274.log.gz 2024-04-25T00:32:24.678 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48038.log.gz 2024-04-25T00:32:24.678 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.84349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35836.log 2024-04-25T00:32:24.678 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.65439.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.84349.log.gz 2024-04-25T00:32:24.679 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44290.log 2024-04-25T00:32:24.679 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.65439.log.gz 2024-04-25T00:32:24.679 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.35836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35341.log 2024-04-25T00:32:24.679 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.44290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41653.log 2024-04-25T00:32:24.679 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.35341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40163.log 2024-04-25T00:32:24.679 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.41653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35185.log 2024-04-25T00:32:24.679 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.40163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81471.log 2024-04-25T00:32:24.679 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.35185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48572.log 2024-04-25T00:32:24.679 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.81471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27679.log 2024-04-25T00:32:24.679 INFO:teuthology.orchestra.run.smithi116.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.81471.log.gz 2024-04-25T00:32:24.679 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.48572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68597.log 2024-04-25T00:32:24.679 INFO:teuthology.orchestra.run.smithi116.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48572.log.gz 2024-04-25T00:32:24.679 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.27679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48709.log 2024-04-25T00:32:24.679 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.68597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76770.log 2024-04-25T00:32:24.679 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.68597.log.gz 2024-04-25T00:32:24.680 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.48709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43356.log 2024-04-25T00:32:24.680 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41653.log.gz 2024-04-25T00:32:24.680 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48709.log.gz 2024-04-25T00:32:24.680 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.76770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50698.log 2024-04-25T00:32:24.680 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.43356.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.76770.log.gz 2024-04-25T00:32:24.680 INFO:teuthology.orchestra.run.smithi116.stderr:gzip 88.5% -5 -- replaced with /var/log/ceph/ceph-client.admin.44079.log.gz --verbose -- 2024-04-25T00:32:24.680 INFO:teuthology.orchestra.run.smithi116.stderr: /var/log/ceph/ceph-client.admin.35523.log 2024-04-25T00:32:24.680 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44290.log.gz 2024-04-25T00:32:24.680 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.50698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34685.log 2024-04-25T00:32:24.680 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44383.log.gz 2024-04-25T00:32:24.680 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.35523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83593.log 2024-04-25T00:32:24.680 INFO:teuthology.orchestra.run.smithi116.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50698.log.gz 2024-04-25T00:32:24.680 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43248.log 2024-04-25T00:32:24.680 INFO:teuthology.orchestra.run.smithi116.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.40163.log.gz 2024-04-25T00:32:24.680 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.83593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47031.log 2024-04-25T00:32:24.681 INFO:teuthology.orchestra.run.smithi116.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.83593.log.gz 2024-04-25T00:32:24.681 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.43248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81367.log 2024-04-25T00:32:24.681 INFO:teuthology.orchestra.run.smithi116.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35185.log.gz 2024-04-25T00:32:24.681 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.47031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45210.log 2024-04-25T00:32:24.681 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47031.log.gz 2024-04-25T00:32:24.681 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.81367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35627.log 2024-04-25T00:32:24.681 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.45210.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.81367.log.gz 2024-04-25T00:32:24.681 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47377.log 2024-04-25T00:32:24.681 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.35627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36150.log 2024-04-25T00:32:24.681 INFO:teuthology.orchestra.run.smithi116.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.27679.log.gz 2024-04-25T00:32:24.681 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.47377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41207.log 2024-04-25T00:32:24.681 INFO:teuthology.orchestra.run.smithi116.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.47377.log.gz 2024-04-25T00:32:24.681 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.36150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46333.log 2024-04-25T00:32:24.681 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.41207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43167.log 2024-04-25T00:32:24.681 INFO:teuthology.orchestra.run.smithi116.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34687.log.gz 2024-04-25T00:32:24.681 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43356.log.gz 2024-04-25T00:32:24.682 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.46333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40785.log 2024-04-25T00:32:24.682 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.43167.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.46333.log.gz 2024-04-25T00:32:24.682 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34685.log.gz 2024-04-25T00:32:24.682 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49324.log 2024-04-25T00:32:24.682 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.40785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47607.log 2024-04-25T00:32:24.682 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.49324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40967.log 2024-04-25T00:32:24.682 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49324.log.gz 2024-04-25T00:32:24.682 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.47607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36332.log 2024-04-25T00:32:24.682 INFO:teuthology.orchestra.run.smithi116.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.43248.log.gz 2024-04-25T00:32:24.682 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.40967.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47607.log.gz 2024-04-25T00:32:24.682 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78861.log 2024-04-25T00:32:24.682 INFO:teuthology.orchestra.run.smithi116.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.41207.log.gz 2024-04-25T00:32:24.682 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.36332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34484.log 2024-04-25T00:32:24.683 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.78861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83191.log 2024-04-25T00:32:24.683 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46073.log 2024-04-25T00:32:24.683 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34484.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.78861.log.gz 2024-04-25T00:32:24.683 INFO:teuthology.orchestra.run.smithi116.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43167.log.gz 2024-04-25T00:32:24.683 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.83191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44833.log 2024-04-25T00:32:24.683 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.46073.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.83191.log.gz 2024-04-25T00:32:24.683 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43697.log 2024-04-25T00:32:24.683 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.44833.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46073.log.gz 2024-04-25T00:32:24.683 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44052.log 2024-04-25T00:32:24.683 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.43697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39520.log 2024-04-25T00:32:24.683 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.44052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84089.log 2024-04-25T00:32:24.683 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.39520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42851.log 2024-04-25T00:32:24.683 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.84089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43996.log 2024-04-25T00:32:24.683 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.42851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34479.log 2024-04-25T00:32:24.684 INFO:teuthology.orchestra.run.smithi116.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.84089.log.gz 2024-04-25T00:32:24.684 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43352.log 2024-04-25T00:32:24.684 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.43996.log: /var/log/ceph/ceph-client.admin.34479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45073.log 2024-04-25T00:32:24.684 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.43352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45867.log 2024-04-25T00:32:24.684 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.45073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45696.log 2024-04-25T00:32:24.684 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77529.log 2024-04-25T00:32:24.684 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.45867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45722.log 2024-04-25T00:32:24.684 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.45696.log: /var/log/ceph/ceph-client.admin.77529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37022.log 2024-04-25T00:32:24.684 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.77529.log.gz 2024-04-25T00:32:24.684 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79814.log 2024-04-25T00:32:24.684 INFO:teuthology.orchestra.run.smithi116.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.40466.log.gz 2024-04-25T00:32:24.684 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.45722.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40496.log.gz 2024-04-25T00:32:24.684 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.37022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34431.log 2024-04-25T00:32:24.684 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.79814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44915.log 2024-04-25T00:32:24.685 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.79814.log.gz 2024-04-25T00:32:24.685 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42047.log 2024-04-25T00:32:24.685 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.44915.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-25T00:32:24.685 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.42047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56108.log 2024-04-25T00:32:24.685 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46125.log/var/log/ceph/ceph-osd.2.log: 2024-04-25T00:32:24.685 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.56108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83400.log 2024-04-25T00:32:24.685 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.46125.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.56108.log.gz 2024-04-25T00:32:24.685 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60716.log 2024-04-25T00:32:24.685 INFO:teuthology.orchestra.run.smithi116.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.45210.log.gz 2024-04-25T00:32:24.685 INFO:teuthology.orchestra.run.smithi116.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45236.log.gz 2024-04-25T00:32:24.685 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.83400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33634.log 2024-04-25T00:32:24.685 INFO:teuthology.orchestra.run.smithi116.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.46125.log.gz 2024-04-25T00:32:24.685 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.60716.log: 89.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.83400.log.gz -5 2024-04-25T00:32:24.685 INFO:teuthology.orchestra.run.smithi116.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41679.log 2024-04-25T00:32:24.685 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.60716.log.gz 2024-04-25T00:32:24.686 INFO:teuthology.orchestra.run.smithi116.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.40785.log.gz 2024-04-25T00:32:24.686 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.33634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49406.log 2024-04-25T00:32:24.686 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.41679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35367.log 2024-04-25T00:32:24.686 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.34484.log.gz 2024-04-25T00:32:24.686 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.49406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40915.log 2024-04-25T00:32:24.686 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.35367.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49406.log.gz 2024-04-25T00:32:24.686 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36410.log 2024-04-25T00:32:24.686 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.40915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36488.log 2024-04-25T00:32:24.686 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44833.log.gz 2024-04-25T00:32:24.686 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.36410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44316.log 2024-04-25T00:32:24.686 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43352.log.gz 2024-04-25T00:32:24.686 INFO:teuthology.orchestra.run.smithi116.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45696.log.gz 2024-04-25T00:32:24.686 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.36488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41153.log 2024-04-25T00:32:24.687 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.44316.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45722.log.gz 2024-04-25T00:32:24.687 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33472.log 2024-04-25T00:32:24.687 INFO:teuthology.orchestra.run.smithi116.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.34431.log.gz 2024-04-25T00:32:24.687 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.41153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45930.log 2024-04-25T00:32:24.687 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.33472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46840.log 2024-04-25T00:32:24.687 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.45930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44770.log 2024-04-25T00:32:24.687 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.46840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47945.log 2024-04-25T00:32:24.687 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.44770.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46840.log.gz 2024-04-25T00:32:24.687 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74076.log 2024-04-25T00:32:24.687 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35862.log 2024-04-25T00:32:24.687 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.47945.log: /var/log/ceph/ceph-client.admin.74076.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34479.log.gz 2024-04-25T00:32:24.687 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47325.log 2024-04-25T00:32:24.687 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47945.log.gz 2024-04-25T00:32:24.687 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.74076.log.gz 2024-04-25T00:32:24.687 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51751.log 2024-04-25T00:32:24.687 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.35862.log: /var/log/ceph/ceph-client.admin.47325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44995.log 2024-04-25T00:32:24.688 INFO:teuthology.orchestra.run.smithi116.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.47325.log.gz 2024-04-25T00:32:24.688 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.51751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35732.log 2024-04-25T00:32:24.688 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51751.log.gz 2024-04-25T00:32:24.688 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.44995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84117.log 2024-04-25T00:32:24.688 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.35732.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36826.log.gz 2024-04-25T00:32:24.688 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84479.log 2024-04-25T00:32:24.688 INFO:teuthology.orchestra.run.smithi116.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.37022.log.gz 2024-04-25T00:32:24.688 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.84117.log: /var/log/ceph/ceph-client.admin.84221.log 2024-04-25T00:32:24.688 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.84479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46044.log 2024-04-25T00:32:24.688 INFO:teuthology.orchestra.run.smithi116.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.84117.log.gz 2024-04-25T00:32:24.688 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.84221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46782.log 2024-04-25T00:32:24.688 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.84479.log.gz 2024-04-25T00:32:24.688 INFO:teuthology.orchestra.run.smithi116.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.84221.log.gz 2024-04-25T00:32:24.688 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.46044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48605.log 2024-04-25T00:32:24.688 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35341.log.gz 89.0% 2024-04-25T00:32:24.689 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46044.log.gz 2024-04-25T00:32:24.689 INFO:teuthology.orchestra.run.smithi116.stderr:gzip/var/log/ceph/ceph-client.admin.46782.log: 89.6% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.43697.log.gz -- 2024-04-25T00:32:24.689 INFO:teuthology.orchestra.run.smithi116.stderr: /var/log/ceph/ceph-client.admin.40541.log 2024-04-25T00:32:24.689 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.48605.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46782.log.gz 2024-04-25T00:32:24.689 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.35367.log.gz 2024-04-25T00:32:24.689 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36514.log 2024-04-25T00:32:24.689 INFO:teuthology.orchestra.run.smithi116.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35523.log.gz 2024-04-25T00:32:24.689 INFO:teuthology.orchestra.run.smithi116.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.48605.log.gz 2024-04-25T00:32:24.689 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.40541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40241.log 2024-04-25T00:32:24.689 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35627.log.gz 2024-04-25T00:32:24.689 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.36514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40293.log 2024-04-25T00:32:24.689 INFO:teuthology.orchestra.run.smithi116.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35653.log.gz 2024-04-25T00:32:24.689 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.40241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50433.log 2024-04-25T00:32:24.689 INFO:teuthology.orchestra.run.smithi116.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.35680.log.gz 2024-04-25T00:32:24.689 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.40293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46415.log 2024-04-25T00:32:24.689 INFO:teuthology.orchestra.run.smithi116.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35810.log.gz 2024-04-25T00:32:24.690 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.50433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34889.log 2024-04-25T00:32:24.690 INFO:teuthology.orchestra.run.smithi116.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.35836.log.gz 2024-04-25T00:32:24.690 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50433.log.gz 2024-04-25T00:32:24.690 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36124.log.gz 2024-04-25T00:32:24.690 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.46415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47711.log 2024-04-25T00:32:24.690 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46415.log.gz 2024-04-25T00:32:24.690 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36150.log.gz 2024-04-25T00:32:24.690 INFO:teuthology.orchestra.run.smithi116.stderr:gzip/var/log/ceph/ceph-client.admin.34889.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.83270.log 2024-04-25T00:32:24.690 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36332.log.gz 2024-04-25T00:32:24.690 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.47711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48683.log 2024-04-25T00:32:24.690 INFO:teuthology.orchestra.run.smithi116.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.36410.log.gz 2024-04-25T00:32:24.690 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.83270.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.47711.log.gz 2024-04-25T00:32:24.690 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36462.log.gz 2024-04-25T00:32:24.690 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48995.log 2024-04-25T00:32:24.691 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36488.log.gz 2024-04-25T00:32:24.691 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81632.log 2024-04-25T00:32:24.691 INFO:teuthology.orchestra.run.smithi116.stderr: 89.2%/var/log/ceph/ceph-client.admin.48683.log: -- replaced with /var/log/ceph/ceph-client.admin.83270.log.gz 2024-04-25T00:32:24.691 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.36592.log.gz 2024-04-25T00:32:24.691 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.48995.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48683.log.gz 2024-04-25T00:32:24.691 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50039.log 2024-04-25T00:32:24.691 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.81632.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48995.log.gz 2024-04-25T00:32:24.691 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50303.log 2024-04-25T00:32:24.691 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.50039.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.81632.log.gz 2024-04-25T00:32:24.691 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36722.log 2024-04-25T00:32:24.691 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36774.log.gz 2024-04-25T00:32:24.691 INFO:teuthology.orchestra.run.smithi116.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.50039.log.gz 2024-04-25T00:32:24.691 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-25T00:32:24.691 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.50303.log: /var/log/ceph/ceph-client.admin.36722.log: 88.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.50303.log.gz 2024-04-25T00:32:24.692 INFO:teuthology.orchestra.run.smithi116.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.62017.log 2024-04-25T00:32:24.692 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48146.log 2024-04-25T00:32:24.692 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.62017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82873.log 2024-04-25T00:32:24.692 INFO:teuthology.orchestra.run.smithi116.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.42047.log.gz 2024-04-25T00:32:24.692 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.62017.log.gz 2024-04-25T00:32:24.692 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49380.log 2024-04-25T00:32:24.692 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.48146.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43996.log.gz 2024-04-25T00:32:24.692 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45337.log 2024-04-25T00:32:24.692 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.82873.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.48146.log.gz 2024-04-25T00:32:24.692 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.49380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39934.log 2024-04-25T00:32:24.692 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.82873.log.gz 2024-04-25T00:32:24.692 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49380.log.gz 2024-04-25T00:32:24.692 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.45337.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44052.log.gz 2024-04-25T00:32:24.692 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44516.log 2024-04-25T00:32:24.693 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.39934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34672.log 2024-04-25T00:32:24.693 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.44516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84063.log 2024-04-25T00:32:24.693 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40705.log 2024-04-25T00:32:24.693 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.84063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50854.log 2024-04-25T00:32:24.693 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.84063.log.gz 2024-04-25T00:32:24.693 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.40705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49620.log 2024-04-25T00:32:24.693 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.50854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42376.log 2024-04-25T00:32:24.693 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.49620.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50854.log.gz 2024-04-25T00:32:24.693 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54194.log 2024-04-25T00:32:24.693 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.42376.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49620.log.gz 2024-04-25T00:32:24.693 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44807.log 2024-04-25T00:32:24.693 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.54194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83907.log 2024-04-25T00:32:24.693 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.44807.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54194.log.gz 2024-04-25T00:32:24.694 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50907.log 2024-04-25T00:32:24.694 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.83907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49021.log 2024-04-25T00:32:24.694 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.83907.log.gz/var/log/ceph/ceph-client.admin.50907.log: 2024-04-25T00:32:24.694 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41744.log 2024-04-25T00:32:24.694 INFO:teuthology.orchestra.run.smithi116.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50907.log.gz 2024-04-25T00:32:24.694 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.49021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73556.log 2024-04-25T00:32:24.694 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.41744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81989.log 2024-04-25T00:32:24.694 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49021.log.gz 2024-04-25T00:32:24.694 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.73556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46155.log 2024-04-25T00:32:24.694 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.73556.log.gz 2024-04-25T00:32:24.694 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.81989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82145.log 2024-04-25T00:32:24.694 INFO:teuthology.orchestra.run.smithi116.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.81989.log.gz 2024-04-25T00:32:24.694 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.46155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81577.log 2024-04-25T00:32:24.695 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.82145.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.46155.log.gz 2024-04-25T00:32:24.695 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42736.log 2024-04-25T00:32:24.695 INFO:teuthology.orchestra.run.smithi116.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.82145.log.gz 2024-04-25T00:32:24.695 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.81577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47115.log 2024-04-25T00:32:24.695 INFO:teuthology.orchestra.run.smithi116.stderr: 89.2%/var/log/ceph/ceph-client.admin.42736.log: -- replaced with /var/log/ceph/ceph-client.admin.81577.log.gz 2024-04-25T00:32:24.695 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66760.log 2024-04-25T00:32:24.695 INFO:teuthology.orchestra.run.smithi116.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44915.log.gz 2024-04-25T00:32:24.695 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.47115.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45867.log.gz 2024-04-25T00:32:24.695 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47873.log 2024-04-25T00:32:24.695 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.66760.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47115.log.gz 2024-04-25T00:32:24.695 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83004.log 2024-04-25T00:32:24.695 INFO:teuthology.orchestra.run.smithi116.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.66760.log.gz 2024-04-25T00:32:24.695 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6%/var/log/ceph/ceph-client.admin.47873.log: -- replaced with /var/log/ceph/ceph-client.admin.42851.log.gz 2024-04-25T00:32:24.695 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84661.log 2024-04-25T00:32:24.696 INFO:teuthology.orchestra.run.smithi116.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.47873.log.gz 2024-04-25T00:32:24.696 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.83004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42021.log 2024-04-25T00:32:24.696 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41679.log.gz 2024-04-25T00:32:24.696 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.83004.log.gz 2024-04-25T00:32:24.696 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.84661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46955.log 2024-04-25T00:32:24.696 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.42021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82478.log 2024-04-25T00:32:24.696 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.84661.log.gz 2024-04-25T00:32:24.696 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.46955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45800.log 2024-04-25T00:32:24.696 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.82478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82322.log 2024-04-25T00:32:24.696 INFO:teuthology.orchestra.run.smithi116.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.46955.log.gz 2024-04-25T00:32:24.696 INFO:teuthology.orchestra.run.smithi116.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.40241.log.gz 2024-04-25T00:32:24.696 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.45800.log: gzip -5 89.3% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.82478.log.gz /var/log/ceph/ceph-client.admin.49801.log 2024-04-25T00:32:24.696 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-04-25T00:32:24.696 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.82322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41601.log 2024-04-25T00:32:24.697 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.49801.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41522.log.gz 2024-04-25T00:32:24.697 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.82322.log.gz 2024-04-25T00:32:24.697 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45021.log 2024-04-25T00:32:24.697 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41549.log.gz 2024-04-25T00:32:24.697 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49801.log.gz 2024-04-25T00:32:24.697 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33790.log 2024-04-25T00:32:24.697 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.41601.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41627.log.gz 2024-04-25T00:32:24.697 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.45021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34474.log 2024-04-25T00:32:24.697 INFO:teuthology.orchestra.run.smithi116.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.41153.log.gz 2024-04-25T00:32:24.697 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.33790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33712.log 2024-04-25T00:32:24.697 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44316.log.gz 2024-04-25T00:32:24.697 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34474.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49827.log 2024-04-25T00:32:24.697 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.33712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78590.log 2024-04-25T00:32:24.697 INFO:teuthology.orchestra.run.smithi116.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45563.log.gz 2024-04-25T00:32:24.697 INFO:teuthology.orchestra.run.smithi116.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.33472.log.gz 2024-04-25T00:32:24.698 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.49827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42454.log 2024-04-25T00:32:24.698 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45496.log 2024-04-25T00:32:24.698 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.78590.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.49827.log.gz 2024-04-25T00:32:24.698 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44770.log.gz 2024-04-25T00:32:24.698 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34889.log.gz 2024-04-25T00:32:24.698 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.42454.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.78590.log.gz 2024-04-25T00:32:24.698 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83541.log 2024-04-25T00:32:24.698 INFO:teuthology.orchestra.run.smithi116.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.35732.log.gz 2024-04-25T00:32:24.698 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65026.log 2024-04-25T00:32:24.698 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.45496.log: /var/log/ceph/ceph-client.admin.83541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81663.log 2024-04-25T00:32:24.698 INFO:teuthology.orchestra.run.smithi116.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.35862.log.gz 2024-04-25T00:32:24.698 INFO:teuthology.orchestra.run.smithi116.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.36514.log.gz 2024-04-25T00:32:24.698 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.83541.log.gz 2024-04-25T00:32:24.698 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.65026.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.36722.log.gz 2024-04-25T00:32:24.699 INFO:teuthology.orchestra.run.smithi116.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.65026.log.gz 2024-04-25T00:32:24.699 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83348.log 2024-04-25T00:32:24.699 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.81663.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.40541.log.gz 2024-04-25T00:32:24.699 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47477.log 2024-04-25T00:32:24.699 INFO:teuthology.orchestra.run.smithi116.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.40293.log.gz 2024-04-25T00:32:24.699 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.83348.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.81663.log.gz 2024-04-25T00:32:24.699 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43115.log 2024-04-25T00:32:24.699 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42376.log.gz 2024-04-25T00:32:24.699 INFO:teuthology.orchestra.run.smithi116.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.45337.log.gz 2024-04-25T00:32:24.699 INFO:teuthology.orchestra.run.smithi116.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.83348.log.gz 2024-04-25T00:32:24.699 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.47477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77423.log 2024-04-25T00:32:24.699 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47477.log.gz 2024-04-25T00:32:24.699 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.43115.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.39934.log.gz 2024-04-25T00:32:24.699 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34884.log 2024-04-25T00:32:24.699 INFO:teuthology.orchestra.run.smithi116.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.42736.log.gz 2024-04-25T00:32:24.700 INFO:teuthology.orchestra.run.smithi116.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40705.log.gz 2024-04-25T00:32:24.700 INFO:teuthology.orchestra.run.smithi116.stderr:gzip/var/log/ceph/ceph-client.admin.77423.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41310.log 2024-04-25T00:32:24.700 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41744.log.gz 2024-04-25T00:32:24.700 INFO:teuthology.orchestra.run.smithi116.stderr: 89.2% 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44807.log.gz 2024-04-25T00:32:24.700 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/ceph-client.admin.77423.log.gz 2024-04-25T00:32:24.700 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81190.log 2024-04-25T00:32:24.700 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34474.log.gz 2024-04-25T00:32:24.700 INFO:teuthology.orchestra.run.smithi116.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34672.log.gz 2024-04-25T00:32:24.700 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.41310.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.42021.log.gz 2024-04-25T00:32:24.700 INFO:teuthology.orchestra.run.smithi116.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.41601.log.gz 2024-04-25T00:32:24.700 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44516.log.gz 2024-04-25T00:32:24.700 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33868.log 2024-04-25T00:32:24.700 INFO:teuthology.orchestra.run.smithi116.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45622.log.gz 2024-04-25T00:32:24.700 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.81190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37048.log 2024-04-25T00:32:24.701 INFO:teuthology.orchestra.run.smithi116.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.45800.log.gz 2024-04-25T00:32:24.701 INFO:teuthology.orchestra.run.smithi116.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45987.log.gz 2024-04-25T00:32:24.701 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.81190.log.gz 2024-04-25T00:32:24.701 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.33868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40863.log 2024-04-25T00:32:24.701 INFO:teuthology.orchestra.run.smithi116.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.33634.log.gz 2024-04-25T00:32:24.701 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.33712.log.gz 2024-04-25T00:32:24.701 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.37048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36306.log 2024-04-25T00:32:24.701 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.40863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50407.log 2024-04-25T00:32:24.701 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.36306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43929.log 2024-04-25T00:32:24.701 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.50407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77475.log 2024-04-25T00:32:24.701 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42454.log.gz 2024-04-25T00:32:24.701 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.43929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42632.log 2024-04-25T00:32:24.701 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50407.log.gz 2024-04-25T00:32:24.701 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.77475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81497.log 2024-04-25T00:32:24.702 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.42632.log: gzip 87.5% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.77475.log.gz -- 2024-04-25T00:32:24.702 INFO:teuthology.orchestra.run.smithi116.stderr: /var/log/ceph/ceph-client.admin.46151.log 2024-04-25T00:32:24.702 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.81497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82978.log 2024-04-25T00:32:24.702 INFO:teuthology.orchestra.run.smithi116.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.81497.log.gz 2024-04-25T00:32:24.702 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.46151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49244.log 2024-04-25T00:32:24.702 INFO:teuthology.orchestra.run.smithi116.stderr:gzip/var/log/ceph/ceph-client.admin.82978.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.43770.log 2024-04-25T00:32:24.702 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46151.log.gz 2024-04-25T00:32:24.702 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.49244.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.82978.log.gz 2024-04-25T00:32:24.702 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81737.log 2024-04-25T00:32:24.702 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.43770.log: gzip 88.7% -5 -- replaced with /var/log/ceph/ceph-client.admin.49244.log.gz --verbose 2024-04-25T00:32:24.702 INFO:teuthology.orchestra.run.smithi116.stderr: -- /var/log/ceph/ceph-client.admin.34193.log 2024-04-25T00:32:24.702 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.81737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33764.log 2024-04-25T00:32:24.702 INFO:teuthology.orchestra.run.smithi116.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.81737.log.gz 2024-04-25T00:32:24.702 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47818.log 2024-04-25T00:32:24.703 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.33764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56134.log 2024-04-25T00:32:24.703 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.47818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82400.log 2024-04-25T00:32:24.703 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.47818.log.gz 2024-04-25T00:32:24.703 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.56134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43300.log 2024-04-25T00:32:24.703 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.82400.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.56134.log.gz 2024-04-25T00:32:24.703 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36970.log 2024-04-25T00:32:24.703 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.43300.log: gzip -5 --verbose 88.9% -- -- replaced with /var/log/ceph/ceph-client.admin.82400.log.gz /var/log/ceph/ceph-client.admin.50604.log 2024-04-25T00:32:24.703 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-04-25T00:32:24.703 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.36970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83108.log 2024-04-25T00:32:24.703 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.50604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44105.log 2024-04-25T00:32:24.703 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.83108.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50604.log.gz 2024-04-25T00:32:24.703 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55171.log 2024-04-25T00:32:24.703 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.83108.log.gz 2024-04-25T00:32:24.704 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.44105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50565.log 2024-04-25T00:32:24.704 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.55171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82348.log 2024-04-25T00:32:24.704 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.50565.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.55171.log.gz 2024-04-25T00:32:24.704 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33842.log 2024-04-25T00:32:24.704 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50565.log.gz 2024-04-25T00:32:24.704 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.82348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48408.log 2024-04-25T00:32:24.704 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.33842.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.82348.log.gz 2024-04-25T00:32:24.704 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84609.log 2024-04-25T00:32:24.704 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.48408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40757.log 2024-04-25T00:32:24.704 INFO:teuthology.orchestra.run.smithi116.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.48408.log.gz 2024-04-25T00:32:24.704 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.84609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37252.log 2024-04-25T00:32:24.704 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.40757.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.84609.log.gz 2024-04-25T00:32:24.704 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34275.log 2024-04-25T00:32:24.704 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.37252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42551.log 2024-04-25T00:32:24.704 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44409.log 2024-04-25T00:32:24.704 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.42551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34327.log 2024-04-25T00:32:24.705 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.44409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56589.log 2024-04-25T00:32:24.705 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47971.log 2024-04-25T00:32:24.705 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.56589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33276.log 2024-04-25T00:32:24.705 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40915.log.gz 2024-04-25T00:32:24.705 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45496.log.gz 2024-04-25T00:32:24.705 INFO:teuthology.orchestra.run.smithi116.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.56589.log.gz 2024-04-25T00:32:24.705 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.47971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40189.log 2024-04-25T00:32:24.705 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43115.log.gz 2024-04-25T00:32:24.705 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.33276.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.47971.log.gz 2024-04-25T00:32:24.705 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- 87.5% /var/log/ceph/ceph-client.admin.48631.log 2024-04-25T00:32:24.705 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33790.log.gz 2024-04-25T00:32:24.705 INFO:teuthology.orchestra.run.smithi116.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34884.log.gz 2024-04-25T00:32:24.705 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48068.log 2024-04-25T00:32:24.705 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.40189.log: gzip/var/log/ceph/ceph-client.admin.48631.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.82951.log 2024-04-25T00:32:24.705 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41310.log.gz 2024-04-25T00:32:24.706 INFO:teuthology.orchestra.run.smithi116.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.33868.log.gz 2024-04-25T00:32:24.706 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.48068.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48631.log.gz 2024-04-25T00:32:24.706 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36280.log 2024-04-25T00:32:24.706 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.82951.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48068.log.gz 2024-04-25T00:32:24.706 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44969.log 2024-04-25T00:32:24.706 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37048.log.gz 2024-04-25T00:32:24.706 INFO:teuthology.orchestra.run.smithi116.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.82951.log.gz 2024-04-25T00:32:24.706 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.36280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43568.log 2024-04-25T00:32:24.706 INFO:teuthology.orchestra.run.smithi116.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36306.log.gz 2024-04-25T00:32:24.706 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45361.log 2024-04-25T00:32:24.706 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.44969.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.43929.log.gz 2024-04-25T00:32:24.706 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42632.log.gz 2024-04-25T00:32:24.706 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.43568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48767.log 2024-04-25T00:32:24.706 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43300.log.gz 2024-04-25T00:32:24.707 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73994.log 2024-04-25T00:32:24.707 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.45361.log: /var/log/ceph/ceph-client.admin.48767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42788.log 2024-04-25T00:32:24.707 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.73994.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.48767.log.gz 2024-04-25T00:32:24.707 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83829.log 2024-04-25T00:32:24.707 INFO:teuthology.orchestra.run.smithi116.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.73994.log.gz 2024-04-25T00:32:24.707 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43196.log 2024-04-25T00:32:24.707 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.42788.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.36970.log.gz 2024-04-25T00:32:24.707 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.83829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77141.log 2024-04-25T00:32:24.707 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.43196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48930.log 2024-04-25T00:32:24.707 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44105.log.gz 2024-04-25T00:32:24.707 INFO:teuthology.orchestra.run.smithi116.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.83829.log.gz 2024-04-25T00:32:24.707 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.77141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40267.log 2024-04-25T00:32:24.707 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.77141.log.gz 2024-04-25T00:32:24.707 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.48930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34050.log 2024-04-25T00:32:24.708 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.40267.log: -- /var/log/ceph/ceph-osd.3.log 2024-04-25T00:32:24.708 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48930.log.gz 2024-04-25T00:32:24.708 INFO:teuthology.orchestra.run.smithi116.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.43770.log.gz 2024-04-25T00:32:24.708 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44611.log 2024-04-25T00:32:24.708 INFO:teuthology.orchestra.run.smithi116.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.33842.log.gz 2024-04-25T00:32:24.708 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34689.log 2024-04-25T00:32:24.708 INFO:teuthology.orchestra.run.smithi116.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.40757.log.gz 2024-04-25T00:32:24.708 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.44611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49890.log 2024-04-25T00:32:24.708 INFO:teuthology.orchestra.run.smithi116.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.34275.log.gz 2024-04-25T00:32:24.708 INFO:teuthology.orchestra.run.smithi116.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.34327.log.gz 2024-04-25T00:32:24.708 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50459.log 2024-04-25T00:32:24.708 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.49890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83619.log 2024-04-25T00:32:24.708 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44409.log.gz 2024-04-25T00:32:24.709 INFO:teuthology.orchestra.run.smithi116.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42551.log.gz 2024-04-25T00:32:24.709 INFO:teuthology.orchestra.run.smithi116.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49890.log.gz 2024-04-25T00:32:24.709 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.50459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.36045.log 2024-04-25T00:32:24.709 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40189.log.gz 2024-04-25T00:32:24.709 INFO:teuthology.orchestra.run.smithi116.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50459.log.gz 2024-04-25T00:32:24.709 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.83619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43516.log 2024-04-25T00:32:24.709 INFO:teuthology.orchestra.run.smithi116.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.36280.log.gz 2024-04-25T00:32:24.709 INFO:teuthology.orchestra.run.smithi116.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.44969.log.gz 2024-04-25T00:32:24.709 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4%/var/log/ceph/ceph-client.0.36045.log: -- replaced with /var/log/ceph/ceph-client.admin.83619.log.gz 2024-04-25T00:32:24.709 INFO:teuthology.orchestra.run.smithi116.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.44995.log.gz 2024-04-25T00:32:24.709 INFO:teuthology.orchestra.run.smithi116.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45021.log.gz 2024-04-25T00:32:24.709 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45073.log.gz 2024-04-25T00:32:24.709 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48471.log 2024-04-25T00:32:24.709 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.43516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43851.log 2024-04-25T00:32:24.710 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42788.log.gz 2024-04-25T00:32:24.710 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.48471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37129.log 2024-04-25T00:32:24.710 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48471.log.gz 2024-04-25T00:32:24.710 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.43851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43456.log 2024-04-25T00:32:24.710 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.37129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67677.log 2024-04-25T00:32:24.710 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.43456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84037.log 2024-04-25T00:32:24.710 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.67677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49746.log 2024-04-25T00:32:24.710 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.84037.log: gzip -5 88.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.67677.log.gz -- 2024-04-25T00:32:24.710 INFO:teuthology.orchestra.run.smithi116.stderr: /var/log/ceph/ceph-client.admin.35471.log 2024-04-25T00:32:24.710 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.49746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40434.log 2024-04-25T00:32:24.710 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.84037.log.gz 2024-04-25T00:32:24.710 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43568.log.gz 2024-04-25T00:32:24.710 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45361.log.gz 2024-04-25T00:32:24.711 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.35471.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43196.log.gz 2024-04-25T00:32:24.711 INFO:teuthology.orchestra.run.smithi116.stderr: 89.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.49746.log.gz 2024-04-25T00:32:24.711 INFO:teuthology.orchestra.run.smithi116.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.51656.log 2024-04-25T00:32:24.711 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40267.log.gz 2024-04-25T00:32:24.711 INFO:teuthology.orchestra.run.smithi116.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34050.log.gz 2024-04-25T00:32:24.711 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.34689.log.gz 2024-04-25T00:32:24.711 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45593.log 2024-04-25T00:32:24.711 INFO:teuthology.orchestra.run.smithi116.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.44611.log.gz 2024-04-25T00:32:24.711 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.40434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36095.log 2024-04-25T00:32:24.711 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.51656.log: /var/log/ceph/ceph-client.admin.45593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80308.log 2024-04-25T00:32:24.711 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51656.log.gz 2024-04-25T00:32:24.711 INFO:teuthology.orchestra.run.smithi116.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.33276.log.gz 2024-04-25T00:32:24.711 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.36095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82821.log 2024-04-25T00:32:24.712 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.80308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41235.log 2024-04-25T00:32:24.712 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43516.log.gz 2024-04-25T00:32:24.712 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.82821.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.80308.log.gz 2024-04-25T00:32:24.712 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42074.log 2024-04-25T00:32:24.712 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.41235.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.82821.log.gz 2024-04-25T00:32:24.712 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43256.log 2024-04-25T00:32:24.712 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.42074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47738.log 2024-04-25T00:32:24.712 INFO:teuthology.orchestra.run.smithi116.stderr:gzip/var/log/ceph/ceph-client.admin.43256.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.34379.log 2024-04-25T00:32:24.712 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.47738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47141.log 2024-04-25T00:32:24.712 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5/var/log/ceph/ceph-client.admin.34379.log: --verbose -- /var/log/ceph/ceph-client.admin.41046.log 2024-04-25T00:32:24.712 INFO:teuthology.orchestra.run.smithi116.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47738.log.gz 2024-04-25T00:32:24.712 INFO:teuthology.orchestra.run.smithi116.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43851.log.gz 2024-04-25T00:32:24.712 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.47141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81844.log 2024-04-25T00:32:24.713 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.34193.log.gz 2024-04-25T00:32:24.713 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.41046.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.47141.log.gz 2024-04-25T00:32:24.713 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48655.log 2024-04-25T00:32:24.713 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.81844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44640.log 2024-04-25T00:32:24.713 INFO:teuthology.orchestra.run.smithi116.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.37129.log.gz 2024-04-25T00:32:24.713 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.48655.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.81844.log.gz 2024-04-25T00:32:24.713 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36436.log 2024-04-25T00:32:24.713 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48655.log.gz 2024-04-25T00:32:24.713 INFO:teuthology.orchestra.run.smithi116.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43456.log.gz 2024-04-25T00:32:24.713 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52090.log 2024-04-25T00:32:24.713 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35471.log.gz 2024-04-25T00:32:24.713 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.44640.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.45593.log.gz 2024-04-25T00:32:24.713 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.36436.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.36095.log.gz 2024-04-25T00:32:24.714 INFO:teuthology.orchestra.run.smithi116.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.41235.log.gz 2024-04-25T00:32:24.714 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42074.log.gz 2024-04-25T00:32:24.714 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43256.log.gz 2024-04-25T00:32:24.714 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56056.log 2024-04-25T00:32:24.714 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.34379.log.gz 2024-04-25T00:32:24.714 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.52090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82404.log 2024-04-25T00:32:24.714 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.56056.log: 88.9% 89.8% -- replaced with /var/log/ceph/ceph-client.admin.52090.log.gz -- replaced with /var/log/ceph/ceph-client.admin.44640.log.gz 2024-04-25T00:32:24.714 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-04-25T00:32:24.714 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74787.log 2024-04-25T00:32:24.714 INFO:teuthology.orchestra.run.smithi116.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.56056.log.gz 2024-04-25T00:32:24.714 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.36436.log.gz 2024-04-25T00:32:24.714 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42933.log 2024-04-25T00:32:24.714 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.82404.log: /var/log/ceph/ceph-client.admin.74787.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83933.log 2024-04-25T00:32:24.714 INFO:teuthology.orchestra.run.smithi116.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.82404.log.gz 2024-04-25T00:32:24.714 INFO:teuthology.orchestra.run.smithi116.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.74787.log.gz 2024-04-25T00:32:24.715 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.42933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44264.log 2024-04-25T00:32:24.715 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.83933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50800.log 2024-04-25T00:32:24.715 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.44264.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.83933.log.gz 2024-04-25T00:32:24.715 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51777.log 2024-04-25T00:32:24.715 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.50800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46929.log 2024-04-25T00:32:24.715 INFO:teuthology.orchestra.run.smithi116.stderr: 87.8%/var/log/ceph/ceph-client.admin.51777.log: -- replaced with /var/log/ceph/ceph-client.admin.50800.log.gz 2024-04-25T00:32:24.715 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51137.log 2024-04-25T00:32:24.715 INFO:teuthology.orchestra.run.smithi116.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.51777.log.gz 2024-04-25T00:32:24.715 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.46929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50539.log 2024-04-25T00:32:24.715 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42933.log.gz 2024-04-25T00:32:24.715 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44264.log.gz 2024-04-25T00:32:24.715 INFO:teuthology.orchestra.run.smithi116.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.46929.log.gz 2024-04-25T00:32:24.715 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48821.log 2024-04-25T00:32:24.715 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.51137.log: /var/log/ceph/ceph-client.admin.50539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74022.log 2024-04-25T00:32:24.716 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51137.log.gz 2024-04-25T00:32:24.716 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.48821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35107.log 2024-04-25T00:32:24.716 INFO:teuthology.orchestra.run.smithi116.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.50539.log.gz 2024-04-25T00:32:24.716 INFO:teuthology.orchestra.run.smithi116.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.48821.log.gz 2024-04-25T00:32:24.716 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.74022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73935.log 2024-04-25T00:32:24.716 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.74022.log.gz 2024-04-25T00:32:24.716 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41705.log 2024-04-25T00:32:24.716 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.35107.log: /var/log/ceph/ceph-client.admin.73935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43404.log 2024-04-25T00:32:24.716 INFO:teuthology.orchestra.run.smithi116.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.73935.log.gz 2024-04-25T00:32:24.716 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.41705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49354.log 2024-04-25T00:32:24.716 INFO:teuthology.orchestra.run.smithi116.stderr: 87.9%/var/log/ceph/ceph-client.admin.43404.log: -- replaced with /var/log/ceph/ceph-client.admin.35107.log.gz 2024-04-25T00:32:24.716 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-25T00:32:24.716 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.49354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43825.log 2024-04-25T00:32:24.716 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49354.log.gz 2024-04-25T00:32:24.716 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose --/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.82199.log 2024-04-25T00:32:24.717 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41705.log.gz 2024-04-25T00:32:24.717 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.43825.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43404.log.gz 2024-04-25T00:32:24.717 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33187.log 2024-04-25T00:32:24.717 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.82199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49484.log 2024-04-25T00:32:24.717 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.33187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83478.log 2024-04-25T00:32:24.717 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.82199.log.gz 2024-04-25T00:32:24.717 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.49484.log: -- /var/log/ceph/ceph-client.admin.45826.log 2024-04-25T00:32:24.717 INFO:teuthology.orchestra.run.smithi116.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.33764.log.gz 2024-04-25T00:32:24.717 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49484.log.gz 2024-04-25T00:32:24.717 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43825.log.gz 2024-04-25T00:32:24.717 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40583.log 2024-04-25T00:32:24.717 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.83478.log: /var/log/ceph/ceph-client.admin.45826.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.33187.log.gz 2024-04-25T00:32:24.717 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83217.log 2024-04-25T00:32:24.717 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.83478.log.gz 2024-04-25T00:32:24.717 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.40583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44692.log 2024-04-25T00:32:24.717 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.83217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41107.log 2024-04-25T00:32:24.717 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.44692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50826.log 2024-04-25T00:32:24.718 INFO:teuthology.orchestra.run.smithi116.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.83217.log.gz 2024-04-25T00:32:24.718 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.41107.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79950.log 2024-04-25T00:32:24.718 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.50826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83959.log 2024-04-25T00:32:24.718 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.79950.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50826.log.gz 2024-04-25T00:32:24.718 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78458.log 2024-04-25T00:32:24.718 INFO:teuthology.orchestra.run.smithi116.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.79950.log.gz 2024-04-25T00:32:24.718 INFO:teuthology.orchestra.run.smithi116.stderr:gzip/var/log/ceph/ceph-client.admin.83959.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.77804.log 2024-04-25T00:32:24.718 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.78458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49987.log 2024-04-25T00:32:24.718 INFO:teuthology.orchestra.run.smithi116.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.83959.log.gz 2024-04-25T00:32:24.718 INFO:teuthology.orchestra.run.smithi116.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45826.log.gz 2024-04-25T00:32:24.718 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.77804.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.78458.log.gz 2024-04-25T00:32:24.718 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37074.log 2024-04-25T00:32:24.718 INFO:teuthology.orchestra.run.smithi116.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.77804.log.gz 2024-04-25T00:32:24.719 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.49987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41051.log 2024-04-25T00:32:24.719 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.37074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82292.log 2024-04-25T00:32:24.719 INFO:teuthology.orchestra.run.smithi116.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.49987.log.gz 2024-04-25T00:32:24.719 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.41051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50195.log 2024-04-25T00:32:24.719 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.82292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44943.log 2024-04-25T00:32:24.719 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.50195.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.82292.log.gz 2024-04-25T00:32:24.719 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41479.log 2024-04-25T00:32:24.719 INFO:teuthology.orchestra.run.smithi116.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50195.log.gz 2024-04-25T00:32:24.719 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44212.log 2024-04-25T00:32:24.719 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.44943.log: /var/log/ceph/ceph-client.admin.41479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55617.log 2024-04-25T00:32:24.719 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.44212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34873.log 2024-04-25T00:32:24.719 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.55617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36696.log 2024-04-25T00:32:24.719 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40356.log 2024-04-25T00:32:24.719 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55617.log.gz 2024-04-25T00:32:24.719 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.36696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47351.log 2024-04-25T00:32:24.719 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.40356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33420.log 2024-04-25T00:32:24.720 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.47351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81066.log 2024-04-25T00:32:24.720 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.33420.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.47351.log.gz 2024-04-25T00:32:24.720 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34686.log 2024-04-25T00:32:24.720 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77020.log 2024-04-25T00:32:24.720 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.81066.log: /var/log/ceph/ceph-client.admin.34686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51604.log 2024-04-25T00:32:24.720 INFO:teuthology.orchestra.run.smithi116.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.81066.log.gz 2024-04-25T00:32:24.720 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48432.log 2024-04-25T00:32:24.720 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.77020.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.37252.log.gz 2024-04-25T00:32:24.720 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.51604.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.40863.log.gz 2024-04-25T00:32:24.720 INFO:teuthology.orchestra.run.smithi116.stderr: 87.8%gzip 88.5% -- replaced with /var/log/ceph/ceph-client.admin.77020.log.gz 2024-04-25T00:32:24.720 INFO:teuthology.orchestra.run.smithi116.stderr: -5 -- replaced with /var/log/ceph/ceph-client.admin.40967.log.gz 2024-04-25T00:32:24.720 INFO:teuthology.orchestra.run.smithi116.stderr: --verbose -- /var/log/ceph/ceph-client.admin.35445.log 2024-04-25T00:32:24.720 INFO:teuthology.orchestra.run.smithi116.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51604.log.gz 2024-04-25T00:32:24.720 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40991.log.gz 2024-04-25T00:32:24.721 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.48432.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41046.log.gz 2024-04-25T00:32:24.721 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77393.log 2024-04-25T00:32:24.721 INFO:teuthology.orchestra.run.smithi116.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.40583.log.gz 2024-04-25T00:32:24.721 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47261.log 2024-04-25T00:32:24.721 INFO:teuthology.orchestra.run.smithi116.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.48432.log.gz 2024-04-25T00:32:24.721 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.35445.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44692.log.gz 2024-04-25T00:32:24.721 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.77393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50485.log 2024-04-25T00:32:24.721 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37074.log.gz 2024-04-25T00:32:24.721 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41051.log.gz 2024-04-25T00:32:24.721 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42710.log.gz 2024-04-25T00:32:24.721 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.77393.log.gz 2024-04-25T00:32:24.721 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.47261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82585.log 2024-04-25T00:32:24.721 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47261.log.gz 2024-04-25T00:32:24.721 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.50485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50511.log 2024-04-25T00:32:24.722 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44943.log.gz 2024-04-25T00:32:24.722 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50485.log.gz 2024-04-25T00:32:24.722 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.82585.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.41479.log.gz 2024-04-25T00:32:24.722 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44212.log.gz 2024-04-25T00:32:24.722 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50773.log 2024-04-25T00:32:24.722 INFO:teuthology.orchestra.run.smithi116.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.82585.log.gz 2024-04-25T00:32:24.722 INFO:teuthology.orchestra.run.smithi116.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34873.log.gz 2024-04-25T00:32:24.722 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.50511.log: 89.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81870.log 2024-04-25T00:32:24.722 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36696.log.gz 2024-04-25T00:32:24.722 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40356.log.gz 2024-04-25T00:32:24.722 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.50773.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50511.log.gz 2024-04-25T00:32:24.722 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50089.log 2024-04-25T00:32:24.722 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.33420.log.gz 2024-04-25T00:32:24.723 INFO:teuthology.orchestra.run.smithi116.stderr:gzip/var/log/ceph/ceph-client.admin.81870.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.77503.log 2024-04-25T00:32:24.723 INFO:teuthology.orchestra.run.smithi116.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50773.log.gz 2024-04-25T00:32:24.723 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34686.log.gz 2024-04-25T00:32:24.723 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35445.log.gz 2024-04-25T00:32:24.723 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.50089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45151.log 2024-04-25T00:32:24.723 INFO:teuthology.orchestra.run.smithi116.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.81870.log.gz 2024-04-25T00:32:24.723 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.77503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48354.log 2024-04-25T00:32:24.723 INFO:teuthology.orchestra.run.smithi116.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50089.log.gz 2024-04-25T00:32:24.723 INFO:teuthology.orchestra.run.smithi116.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.77503.log.gz 2024-04-25T00:32:24.723 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.45151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84399.log 2024-04-25T00:32:24.723 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.48354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43512.log 2024-04-25T00:32:24.723 INFO:teuthology.orchestra.run.smithi116.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.48354.log.gz 2024-04-25T00:32:24.723 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.84399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40215.log 2024-04-25T00:32:24.723 INFO:teuthology.orchestra.run.smithi116.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45151.log.gz 2024-04-25T00:32:24.724 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.43512.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.84399.log.gz 2024-04-25T00:32:24.724 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48172.log 2024-04-25T00:32:24.724 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.40215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76075.log 2024-04-25T00:32:24.724 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.48172.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43512.log.gz 2024-04-25T00:32:24.724 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40609.log 2024-04-25T00:32:24.724 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.76075.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48172.log.gz 2024-04-25T00:32:24.724 INFO:teuthology.orchestra.run.smithi116.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.40215.log.gz 2024-04-25T00:32:24.724 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42907.log 2024-04-25T00:32:24.724 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.76075.log.gz 2024-04-25T00:32:24.724 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.40609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50335.log 2024-04-25T00:32:24.724 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.42907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81341.log 2024-04-25T00:32:24.724 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40609.log.gz 2024-04-25T00:32:24.724 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.50335.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.22845.log 2024-04-25T00:32:24.724 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42907.log.gz 2024-04-25T00:32:24.725 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50335.log.gz 2024-04-25T00:32:24.725 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36618.log 2024-04-25T00:32:24.725 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.81341.log: /var/log/ceph/ceph.tmp-client.admin.22845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83777.log 2024-04-25T00:32:24.725 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.81341.log.gz 2024-04-25T00:32:24.725 INFO:teuthology.orchestra.run.smithi116.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.22845.log.gz 2024-04-25T00:32:24.725 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.36618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50169.log 2024-04-25T00:32:24.725 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.83777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84687.log 2024-04-25T00:32:24.725 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.50169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48495.log 2024-04-25T00:32:24.725 INFO:teuthology.orchestra.run.smithi116.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.83777.log.gz 2024-04-25T00:32:24.725 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.84687.log: gzip -5 89.6% --verbose -- replaced with /var/log/ceph/ceph-client.admin.36618.log.gz -- 2024-04-25T00:32:24.725 INFO:teuthology.orchestra.run.smithi116.stderr: /var/log/ceph/ceph-client.admin.45535.log 2024-04-25T00:32:24.725 INFO:teuthology.orchestra.run.smithi116.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50169.log.gz 2024-04-25T00:32:24.725 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.48495.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.84687.log.gz 2024-04-25T00:32:24.725 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34900.log 2024-04-25T00:32:24.726 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.45535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46411.log 2024-04-25T00:32:24.726 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48495.log.gz 2024-04-25T00:32:24.726 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47685.log 2024-04-25T00:32:24.726 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.46411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48252.log 2024-04-25T00:32:24.726 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.47685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33608.log 2024-04-25T00:32:24.726 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46411.log.gz 2024-04-25T00:32:24.726 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37222.log 2024-04-25T00:32:24.726 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.48252.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47685.log.gz 2024-04-25T00:32:24.726 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.33608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53760.log 2024-04-25T00:32:24.726 INFO:teuthology.orchestra.run.smithi116.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.48252.log.gz 2024-04-25T00:32:24.726 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.37222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82066.log 2024-04-25T00:32:24.726 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81816.log 2024-04-25T00:32:24.726 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.53760.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.34900.log.gz 2024-04-25T00:32:24.726 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.82066.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53760.log.gz 2024-04-25T00:32:24.727 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45535.log.gz 2024-04-25T00:32:24.727 INFO:teuthology.orchestra.run.smithi116.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.82066.log.gz 2024-04-25T00:32:24.727 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83426.log 2024-04-25T00:32:24.727 INFO:teuthology.orchestra.run.smithi116.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.33608.log.gz 2024-04-25T00:32:24.727 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.81816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83673.log 2024-04-25T00:32:24.727 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.83426.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.81816.log.gz 2024-04-25T00:32:24.727 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84195.log 2024-04-25T00:32:24.727 INFO:teuthology.orchestra.run.smithi116.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.83426.log.gz 2024-04-25T00:32:24.727 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.83673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42350.log 2024-04-25T00:32:24.727 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.83673.log.gz 2024-04-25T00:32:24.727 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.84195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34078.log 2024-04-25T00:32:24.727 INFO:teuthology.orchestra.run.smithi116.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.84195.log.gz 2024-04-25T00:32:24.727 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.42350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74156.log 2024-04-25T00:32:24.727 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35263.log 2024-04-25T00:32:24.728 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.74156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35577.log 2024-04-25T00:32:24.728 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.35263.log: gzip 88.8% -- replaced with /var/log/ceph/ceph-client.admin.74156.log.gz -5 2024-04-25T00:32:24.728 INFO:teuthology.orchestra.run.smithi116.stderr: --verbose -- /var/log/ceph/ceph-client.admin.48380.log 2024-04-25T00:32:24.728 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.35577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42684.log 2024-04-25T00:32:24.728 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.48380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36358.log 2024-04-25T00:32:24.728 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.42684.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48380.log.gz 2024-04-25T00:32:24.728 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84247.log 2024-04-25T00:32:24.728 INFO:teuthology.orchestra.run.smithi116.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.42350.log.gz 2024-04-25T00:32:24.728 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.36358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84323.log 2024-04-25T00:32:24.728 INFO:teuthology.orchestra.run.smithi116.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.34078.log.gz 2024-04-25T00:32:24.728 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.84247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42961.log 2024-04-25T00:32:24.728 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35263.log.gz 2024-04-25T00:32:24.728 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.84323.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.84247.log.gz 2024-04-25T00:32:24.728 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42658.log 2024-04-25T00:32:24.728 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.42961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56160.log 2024-04-25T00:32:24.729 INFO:teuthology.orchestra.run.smithi116.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.84323.log.gz 2024-04-25T00:32:24.729 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.42658.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-25T00:32:24.729 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.56160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48847.log 2024-04-25T00:32:24.729 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph.log: gzip 88.8% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.56160.log.gz -- 2024-04-25T00:32:24.729 INFO:teuthology.orchestra.run.smithi116.stderr: /var/log/ceph/ceph-client.admin.43482.log 2024-04-25T00:32:24.729 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.48847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51430.log 2024-04-25T00:32:24.729 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48847.log.gz 2024-04-25T00:32:24.729 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.43482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82018.log 2024-04-25T00:32:24.729 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.37222.log.gz 2024-04-25T00:32:24.729 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.51430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80669.log 2024-04-25T00:32:24.729 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.82018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46545.log 2024-04-25T00:32:24.729 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51430.log.gz 2024-04-25T00:32:24.729 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.80669.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.35577.log.gz 88.9% 2024-04-25T00:32:24.729 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/ceph-client.admin.82018.log.gz 2024-04-25T00:32:24.730 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27921.log 2024-04-25T00:32:24.730 INFO:teuthology.orchestra.run.smithi116.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.42684.log.gz 2024-04-25T00:32:24.730 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48969.log 2024-04-25T00:32:24.730 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.46545.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.80669.log.gz 2024-04-25T00:32:24.730 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.27921.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46545.log.gz 2024-04-25T00:32:24.730 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41017.log 2024-04-25T00:32:24.730 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.48969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35419.log 2024-04-25T00:32:24.730 INFO:teuthology.orchestra.run.smithi116.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.36358.log.gz 2024-04-25T00:32:24.730 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48969.log.gz 2024-04-25T00:32:24.730 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.41017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44186.log 2024-04-25T00:32:24.730 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.35419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40731.log 2024-04-25T00:32:24.730 INFO:teuthology.orchestra.run.smithi116.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42961.log.gz 2024-04-25T00:32:24.730 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43903.log 2024-04-25T00:32:24.730 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.44186.log: /var/log/ceph/ceph-client.admin.40731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81606.log 2024-04-25T00:32:24.731 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42658.log.gz 2024-04-25T00:32:24.731 INFO:teuthology.orchestra.run.smithi116.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41017.log.gz 2024-04-25T00:32:24.731 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.43903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43210.log 2024-04-25T00:32:24.731 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.81606.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35419.log.gz 2024-04-25T00:32:24.731 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33894.log 2024-04-25T00:32:24.731 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.43210.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44186.log.gz 2024-04-25T00:32:24.731 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.81606.log.gz 2024-04-25T00:32:24.731 INFO:teuthology.orchestra.run.smithi116.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40731.log.gz 2024-04-25T00:32:24.731 INFO:teuthology.orchestra.run.smithi116.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.43903.log.gz 2024-04-25T00:32:24.731 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59789.log 2024-04-25T00:32:24.731 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.33894.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43210.log.gz 2024-04-25T00:32:24.731 INFO:teuthology.orchestra.run.smithi116.stderr: 87.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-25T00:32:24.732 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48226.log 2024-04-25T00:32:24.732 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.59789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83374.log 2024-04-25T00:32:24.732 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.48226.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.33894.log.gz 2024-04-25T00:32:24.732 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50251.log 2024-04-25T00:32:24.732 INFO:teuthology.orchestra.run.smithi116.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.59789.log.gz 2024-04-25T00:32:24.732 INFO:teuthology.orchestra.run.smithi116.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.48226.log.gz 2024-04-25T00:32:24.732 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.83374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42402.log 2024-04-25T00:32:24.732 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.50251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51531.log 2024-04-25T00:32:24.732 INFO:teuthology.orchestra.run.smithi116.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.83374.log.gz 2024-04-25T00:32:24.732 INFO:teuthology.orchestra.run.smithi116.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.50251.log.gz 2024-04-25T00:32:24.732 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.42402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49127.log 2024-04-25T00:32:24.732 INFO:teuthology.orchestra.run.smithi116.stderr:gzip/var/log/ceph/ceph-client.admin.51531.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.47289.log 2024-04-25T00:32:24.732 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.49127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37159.log 2024-04-25T00:32:24.732 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.47289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34353.log 2024-04-25T00:32:24.733 INFO:teuthology.orchestra.run.smithi116.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.51531.log.gz 2024-04-25T00:32:24.733 INFO:teuthology.orchestra.run.smithi116.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49127.log.gz 2024-04-25T00:32:24.733 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47289.log.gz 2024-04-25T00:32:24.733 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.37159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34301.log 2024-04-25T00:32:24.733 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49101.log 2024-04-25T00:32:24.733 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34301.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37159.log.gz 2024-04-25T00:32:24.733 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34891.log 2024-04-25T00:32:24.733 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.49101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81790.log 2024-04-25T00:32:24.733 INFO:teuthology.orchestra.run.smithi116.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34353.log.gz 2024-04-25T00:32:24.733 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49101.log.gz 2024-04-25T00:32:24.733 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34301.log.gz 2024-04-25T00:32:24.733 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34891.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65896.log 2024-04-25T00:32:24.733 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.81790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41347.log 2024-04-25T00:32:24.734 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.65896.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.34891.log.gz 2024-04-25T00:32:24.734 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55997.log 2024-04-25T00:32:24.734 INFO:teuthology.orchestra.run.smithi116.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.81790.log.gz 2024-04-25T00:32:24.734 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.65896.log.gz 2024-04-25T00:32:24.734 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.41347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35784.log 2024-04-25T00:32:24.734 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.55997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46359.log 2024-04-25T00:32:24.734 INFO:teuthology.orchestra.run.smithi116.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43482.log.gz 2024-04-25T00:32:24.734 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.35784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40408.log 2024-04-25T00:32:24.734 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55997.log.gz 2024-04-25T00:32:24.734 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.46359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33660.log 2024-04-25T00:32:24.734 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.40408.log: gzip 87.7% -5 -- replaced with /var/log/ceph/ceph-client.admin.46359.log.gz --verbose 2024-04-25T00:32:24.734 INFO:teuthology.orchestra.run.smithi116.stderr: -- /var/log/ceph/ceph-client.admin.40889.log 2024-04-25T00:32:24.734 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.33660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33446.log 2024-04-25T00:32:24.734 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.40889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36176.log 2024-04-25T00:32:24.735 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.33446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41969.log 2024-04-25T00:32:24.735 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.36176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82847.log 2024-04-25T00:32:24.735 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41347.log.gz 2024-04-25T00:32:24.735 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.41969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51026.log 2024-04-25T00:32:24.735 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.82847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46018.log 2024-04-25T00:32:24.735 INFO:teuthology.orchestra.run.smithi116.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.41107.log.gz 2024-04-25T00:32:24.735 INFO:teuthology.orchestra.run.smithi116.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.82847.log.gz 2024-04-25T00:32:24.735 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.51026.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.35784.log.gz 2024-04-25T00:32:24.735 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51056.log 2024-04-25T00:32:24.735 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51026.log.gz 2024-04-25T00:32:24.735 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83803.log 2024-04-25T00:32:24.735 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.46018.log: /var/log/ceph/ceph-client.admin.51056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43743.log 2024-04-25T00:32:24.735 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.33660.log.gz 2024-04-25T00:32:24.735 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46018.log.gz 2024-04-25T00:32:24.735 INFO:teuthology.orchestra.run.smithi116.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51056.log.gz 2024-04-25T00:32:24.736 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.83803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59945.log 2024-04-25T00:32:24.736 INFO:teuthology.orchestra.run.smithi116.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40889.log.gz 2024-04-25T00:32:24.736 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47194.log 2024-04-25T00:32:24.736 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.43743.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.83803.log.gz 2024-04-25T00:32:24.736 INFO:teuthology.orchestra.run.smithi116.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.33446.log.gz 2024-04-25T00:32:24.736 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.59945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41575.log 2024-04-25T00:32:24.736 INFO:teuthology.orchestra.run.smithi116.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.59945.log.gz 2024-04-25T00:32:24.736 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.47194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81445.log 2024-04-25T00:32:24.736 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.41575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51293.log 2024-04-25T00:32:24.736 INFO:teuthology.orchestra.run.smithi116.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.47194.log.gz 2024-04-25T00:32:24.736 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44718.log 2024-04-25T00:32:24.736 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.81445.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.36176.log.gz 2024-04-25T00:32:24.736 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.51293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44160.log 2024-04-25T00:32:24.736 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.81445.log.gz 2024-04-25T00:32:24.737 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.44718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82504.log 2024-04-25T00:32:24.737 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51293.log.gz 2024-04-25T00:32:24.737 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.44160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83452.log 2024-04-25T00:32:24.737 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.82504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42229.log 2024-04-25T00:32:24.737 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.82504.log.gz 2024-04-25T00:32:24.737 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74330.log 2024-04-25T00:32:24.737 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.83452.log: /var/log/ceph/ceph-client.admin.42229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77568.log 2024-04-25T00:32:24.737 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% 89.4% -- replaced with /var/log/ceph/ceph-client.admin.83452.log.gz 2024-04-25T00:32:24.737 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41969.log.gz 2024-04-25T00:32:24.737 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.74330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47007.log 2024-04-25T00:32:24.737 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.77568.log: gzip 88.4% 88.4% -- replaced with /var/log/ceph/ceph-client.admin.74330.log.gz -- replaced with /var/log/ceph/ceph-client.admin.42402.log.gz 2024-04-25T00:32:24.737 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-04-25T00:32:24.737 INFO:teuthology.orchestra.run.smithi116.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.48200.log 2024-04-25T00:32:24.737 INFO:teuthology.orchestra.run.smithi116.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41575.log.gz 2024-04-25T00:32:24.737 INFO:teuthology.orchestra.run.smithi116.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.77568.log.gz 2024-04-25T00:32:24.738 INFO:teuthology.orchestra.run.smithi116.stderr: 89.5%/var/log/ceph/ceph-client.admin.47007.log: -- replaced with /var/log/ceph/ceph-client.admin.44718.log.gz 2024-04-25T00:32:24.738 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44160.log.gz 2024-04-25T00:32:24.738 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83165.log 2024-04-25T00:32:24.738 INFO:teuthology.orchestra.run.smithi116.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.42229.log.gz 2024-04-25T00:32:24.738 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47007.log.gz 2024-04-25T00:32:24.738 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33816.log 2024-04-25T00:32:24.738 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.48200.log: /var/log/ceph/ceph-client.admin.83165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81959.log 2024-04-25T00:32:24.738 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48200.log.gz 2024-04-25T00:32:24.738 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.83165.log.gz 2024-04-25T00:32:24.738 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.33816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56187.log 2024-04-25T00:32:24.738 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.81959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27789.log 2024-04-25T00:32:24.738 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.56187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46515.log 88.7% 2024-04-25T00:32:24.738 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/ceph-client.admin.81959.log.gz 2024-04-25T00:32:24.738 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56187.log.gz 2024-04-25T00:32:24.739 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.27789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45748.log 2024-04-25T00:32:24.739 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.46515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45470.log 2024-04-25T00:32:24.739 INFO:teuthology.orchestra.run.smithi116.stderr: 88.1%/var/log/ceph/ceph-client.admin.45748.log: -- replaced with /var/log/ceph/ceph-client.admin.46515.log.gz 2024-04-25T00:32:24.739 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34691.log 2024-04-25T00:32:24.739 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.45470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47559.log 2024-04-25T00:32:24.739 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.33816.log.gz 2024-04-25T00:32:24.739 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84635.log 2024-04-25T00:32:24.739 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.47559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84505.log 2024-04-25T00:32:24.739 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.84635.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47559.log.gz 2024-04-25T00:32:24.739 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43039.log 2024-04-25T00:32:24.739 INFO:teuthology.orchestra.run.smithi116.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45748.log.gz 2024-04-25T00:32:24.739 INFO:teuthology.orchestra.run.smithi116.stderr: 88.1%/var/log/ceph/ceph-client.admin.84505.log: -- replaced with /var/log/ceph/ceph-client.admin.84635.log.gz 2024-04-25T00:32:24.739 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79488.log 2024-04-25T00:32:24.739 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.43039.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.84505.log.gz 2024-04-25T00:32:24.739 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.45470.log.gz 2024-04-25T00:32:24.740 INFO:teuthology.orchestra.run.smithi116.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.81315.log 2024-04-25T00:32:24.740 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34691.log.gz 2024-04-25T00:32:24.740 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.79488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74579.log 2024-04-25T00:32:24.740 INFO:teuthology.orchestra.run.smithi116.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.79488.log.gz 2024-04-25T00:32:24.740 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.81315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51505.log 2024-04-25T00:32:24.740 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.74579.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.27921.log.gz 2024-04-25T00:32:24.740 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60327.log 2024-04-25T00:32:24.740 INFO:teuthology.orchestra.run.smithi116.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.81315.log.gz 2024-04-25T00:32:24.740 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43039.log.gz 2024-04-25T00:32:24.740 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.51505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81222.log 2024-04-25T00:32:24.740 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.74579.log.gz 2024-04-25T00:32:24.740 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.60327.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.51505.log.gz 2024-04-25T00:32:24.740 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37281.log 2024-04-25T00:32:24.740 INFO:teuthology.orchestra.run.smithi116.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.60327.log.gz 2024-04-25T00:32:24.741 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.81222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40661.log 2024-04-25T00:32:24.741 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.37281.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.81222.log.gz 2024-04-25T00:32:24.741 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51682.log 2024-04-25T00:32:24.741 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.40661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73970.log 2024-04-25T00:32:24.741 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.51682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49179.log 2024-04-25T00:32:24.741 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39433.log 2024-04-25T00:32:24.741 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51682.log.gz 2024-04-25T00:32:24.741 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.73970.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40661.log.gz 2024-04-25T00:32:24.741 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.49179.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34688.log 2024-04-25T00:32:24.741 INFO:teuthology.orchestra.run.smithi116.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.73970.log.gz 2024-04-25T00:32:24.741 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.39433.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49179.log.gz 2024-04-25T00:32:24.741 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36852.log 2024-04-25T00:32:24.741 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46465.log 2024-04-25T00:32:24.742 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.36852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49853.log 2024-04-25T00:32:24.742 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.46465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60399.log 2024-04-25T00:32:24.742 INFO:teuthology.orchestra.run.smithi116.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.46465.log.gz 2024-04-25T00:32:24.742 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.49853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34106.log 2024-04-25T00:32:24.742 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.60399.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49853.log.gz 2024-04-25T00:32:24.742 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35917.log 2024-04-25T00:32:24.742 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34106.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.34688.log.gz 2024-04-25T00:32:24.742 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66345.log 2024-04-25T00:32:24.742 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.60399.log.gz 2024-04-25T00:32:24.742 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.35917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35133.log 2024-04-25T00:32:24.742 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34106.log.gz 2024-04-25T00:32:24.742 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.66345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48549.log 2024-04-25T00:32:24.742 INFO:teuthology.orchestra.run.smithi116.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.66345.log.gz 2024-04-25T00:32:24.742 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.35133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56280.log 2024-04-25T00:32:24.743 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.48549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41798.log 2024-04-25T00:32:24.743 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.56280.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.48549.log.gz 2024-04-25T00:32:24.743 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43743.log.gz 2024-04-25T00:32:24.743 INFO:teuthology.orchestra.run.smithi116.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.56280.log.gz 2024-04-25T00:32:24.743 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81111.log 2024-04-25T00:32:24.743 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.41798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33554.log 2024-04-25T00:32:24.743 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.81111.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-25T00:32:24.743 INFO:teuthology.orchestra.run.smithi116.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.35917.log.gz 2024-04-25T00:32:24.743 INFO:teuthology.orchestra.run.smithi116.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.81111.log.gz 2024-04-25T00:32:24.743 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.33554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53212.log 2024-04-25T00:32:24.743 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33972.log 2024-04-25T00:32:24.743 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.53212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82719.log 2024-04-25T00:32:24.743 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.33972.log: gzip -5 --verbose -- 87.6% /var/log/ceph/ceph-client.admin.37327.log 2024-04-25T00:32:24.743 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53212.log.gz 2024-04-25T00:32:24.744 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.82719.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-04-25T00:32:24.744 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.37327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81549.log 2024-04-25T00:32:24.744 INFO:teuthology.orchestra.run.smithi116.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.27789.log.gz 2024-04-25T00:32:24.744 INFO:teuthology.orchestra.run.smithi116.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.82719.log.gz 2024-04-25T00:32:24.744 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44468.log 2024-04-25T00:32:24.744 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.81549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47764.log 2024-04-25T00:32:24.744 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.44468.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.81549.log.gz 2024-04-25T00:32:24.744 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41103.log 2024-04-25T00:32:24.744 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.47764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56562.log 2024-04-25T00:32:24.744 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.41103.log: gzip -5 89.1% --verbose -- replaced with /var/log/ceph/ceph-client.admin.47764.log.gz -- 2024-04-25T00:32:24.744 INFO:teuthology.orchestra.run.smithi116.stderr: /var/log/ceph/ceph-mds.a.log 2024-04-25T00:32:24.744 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.56562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40137.log 2024-04-25T00:32:24.744 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-mds.a.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.56562.log.gz 2024-04-25T00:32:24.744 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83030.log 2024-04-25T00:32:24.745 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.40137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35706.log 2024-04-25T00:32:24.745 INFO:teuthology.orchestra.run.smithi116.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.35133.log.gz 2024-04-25T00:32:24.745 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.83030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83699.log 2024-04-25T00:32:24.745 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.83030.log.gz 2024-04-25T00:32:24.745 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.35706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50998.log 2024-04-25T00:32:24.745 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.83699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60477.log 2024-04-25T00:32:24.745 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.50998.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.83699.log.gz 2024-04-25T00:32:24.745 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46577.log 2024-04-25T00:32:24.745 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.60477.log: gzip 88.8% -5 -- replaced with /var/log/ceph/ceph-client.admin.50998.log.gz --verbose 2024-04-25T00:32:24.745 INFO:teuthology.orchestra.run.smithi116.stderr: -- /var/log/ceph/ceph-client.admin.34483.log 2024-04-25T00:32:24.745 INFO:teuthology.orchestra.run.smithi116.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.37281.log.gz/var/log/ceph/ceph-client.admin.46577.log: 2024-04-25T00:32:24.745 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- 88.9% /var/log/ceph/ceph-client.admin.84169.log 2024-04-25T00:32:24.745 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/ceph-client.admin.60477.log.gz 2024-04-25T00:32:24.745 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34483.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46577.log.gz 2024-04-25T00:32:24.745 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81417.log 2024-04-25T00:32:24.746 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.84169.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41798.log.gz 2024-04-25T00:32:24.746 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43877.log 2024-04-25T00:32:24.746 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.84169.log.gz 2024-04-25T00:32:24.746 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.81417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45389.log 2024-04-25T00:32:24.746 INFO:teuthology.orchestra.run.smithi116.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.81417.log.gz/var/log/ceph/ceph-client.admin.43877.log: 2024-04-25T00:32:24.746 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51320.log 2024-04-25T00:32:24.746 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.45389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82556.log 2024-04-25T00:32:24.746 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.51320.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.33554.log.gz 2024-04-25T00:32:24.746 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56003.log 2024-04-25T00:32:24.746 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.82556.log: gzip -5 89.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.51320.log.gz -- 2024-04-25T00:32:24.746 INFO:teuthology.orchestra.run.smithi116.stderr: /var/log/ceph/ceph-client.admin.82264.log 2024-04-25T00:32:24.746 INFO:teuthology.orchestra.run.smithi116.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.82556.log.gz 2024-04-25T00:32:24.746 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.56003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83504.log 2024-04-25T00:32:24.746 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.82264.log: gzip 89.0% -- replaced with /var/log/ceph/ceph-client.admin.56003.log.gz 2024-04-25T00:32:24.747 INFO:teuthology.orchestra.run.smithi116.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.34892.log 2024-04-25T00:32:24.747 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.83504.log: 89.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.82264.log.gz -5 2024-04-25T00:32:24.747 INFO:teuthology.orchestra.run.smithi116.stderr: --verbose -- /var/log/ceph/ceph-client.admin.33341.log 2024-04-25T00:32:24.747 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34892.log: gzip 88.3% -5 --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.83504.log.gz /var/log/ceph/ceph-client.admin.48097.log 2024-04-25T00:32:24.747 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-04-25T00:32:24.747 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.33341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49205.log 2024-04-25T00:32:24.747 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.48097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67264.log 2024-04-25T00:32:24.747 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35289.log 2024-04-25T00:32:24.747 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.49205.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48097.log.gz 2024-04-25T00:32:24.747 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.67264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64901.log 2024-04-25T00:32:24.747 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49205.log.gz 2024-04-25T00:32:24.747 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.35289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51348.log 2024-04-25T00:32:24.747 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.67264.log.gz 2024-04-25T00:32:24.748 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.64901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33367.log 2024-04-25T00:32:24.748 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.64901.log.gz 2024-04-25T00:32:24.748 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82795.log 2024-04-25T00:32:24.748 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.51348.log: /var/log/ceph/ceph-client.admin.33367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58313.log 2024-04-25T00:32:24.748 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51348.log.gz 2024-04-25T00:32:24.748 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.82795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43143.log 2024-04-25T00:32:24.748 INFO:teuthology.orchestra.run.smithi116.stderr: 89.3%/var/log/ceph/ceph-client.admin.58313.log: -- replaced with /var/log/ceph/ceph-client.admin.82795.log.gz 2024-04-25T00:32:24.748 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84271.log 2024-04-25T00:32:24.748 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.43143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50143.log 2024-04-25T00:32:24.748 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58313.log.gz 2024-04-25T00:32:24.748 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.84271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47089.log 2024-04-25T00:32:24.748 INFO:teuthology.orchestra.run.smithi116.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.84271.log.gz 2024-04-25T00:32:24.748 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.50143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43063.log 2024-04-25T00:32:24.748 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.47089.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50143.log.gz 2024-04-25T00:32:24.749 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49153.log 2024-04-25T00:32:24.749 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.43063.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.47089.log.gz 2024-04-25T00:32:24.749 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84143.log 2024-04-25T00:32:24.749 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.49153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74104.log 2024-04-25T00:32:24.749 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.84143.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.49153.log.gz 2024-04-25T00:32:24.749 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48521.log 2024-04-25T00:32:24.749 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.84143.log.gz 2024-04-25T00:32:24.749 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.74104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83296.log 2024-04-25T00:32:24.749 INFO:teuthology.orchestra.run.smithi116.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.74104.log.gz 2024-04-25T00:32:24.749 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.48521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50013.log 2024-04-25T00:32:24.749 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.83296.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48521.log.gz 2024-04-25T00:32:24.749 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35497.log 2024-04-25T00:32:24.749 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4%/var/log/ceph/ceph-client.admin.50013.log: -- replaced with /var/log/ceph/ceph-client.admin.83296.log.gz 2024-04-25T00:32:24.749 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51630.log 2024-04-25T00:32:24.750 INFO:teuthology.orchestra.run.smithi116.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50013.log.gz 2024-04-25T00:32:24.750 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.35497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81523.log 2024-04-25T00:32:24.750 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.51630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49720.log 2024-04-25T00:32:24.750 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51630.log.gz 2024-04-25T00:32:24.750 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.81523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36202.log 2024-04-25T00:32:24.750 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.49720.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.81523.log.gz 2024-04-25T00:32:24.750 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81709.log 2024-04-25T00:32:24.750 INFO:teuthology.orchestra.run.smithi116.stderr: 87.7%/var/log/ceph/ceph-client.admin.36202.log: -- replaced with /var/log/ceph/ceph-client.admin.49720.log.gz 2024-04-25T00:32:24.750 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82452.log 2024-04-25T00:32:24.750 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.81709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36930.log 2024-04-25T00:32:24.750 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.82452.log: gzip 87.7% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.81709.log.gz -- 2024-04-25T00:32:24.750 INFO:teuthology.orchestra.run.smithi116.stderr: /var/log/ceph/ceph-client.admin.51163.log 2024-04-25T00:32:24.750 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.36930.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.82452.log.gz 2024-04-25T00:32:24.750 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81259.log 2024-04-25T00:32:24.750 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.51163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42521.log 2024-04-25T00:32:24.751 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.81259.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51163.log.gz 2024-04-25T00:32:24.751 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36384.log 2024-04-25T00:32:24.751 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.81259.log.gz 2024-04-25T00:32:24.751 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.42521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67166.log 2024-04-25T00:32:24.751 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.36384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82637.log 2024-04-25T00:32:24.751 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.67166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48884.log 2024-04-25T00:32:24.751 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.82637.log: gzip -5 88.2% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.45930.log.gz /var/log/ceph/ceph-client.admin.47633.log 2024-04-25T00:32:24.751 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-04-25T00:32:24.751 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% 88.6% -- replaced with /var/log/ceph/ceph-client.admin.67166.log.gz -- replaced with /var/log/ceph/ceph-client.admin.45904.log.gz 2024-04-25T00:32:24.751 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-04-25T00:32:24.751 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.82637.log.gz 2024-04-25T00:32:24.751 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.48884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45047.log 2024-04-25T00:32:24.751 INFO:teuthology.orchestra.run.smithi116.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48884.log.gz 2024-04-25T00:32:24.751 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.47633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45125.log 2024-04-25T00:32:24.752 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.45047.log: gzip 89.5% -- replaced with /var/log/ceph/ceph-client.admin.47633.log.gz -5 2024-04-25T00:32:24.752 INFO:teuthology.orchestra.run.smithi116.stderr: --verbose -- /var/log/ceph/ceph-client.admin.37196.log 2024-04-25T00:32:24.752 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.45125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47555.log 2024-04-25T00:32:24.752 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.37196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41895.log 2024-04-25T00:32:24.752 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.47555.log: /var/log/ceph/ceph-client.admin.46756.log 2024-04-25T00:32:24.752 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.41895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34693.log 2024-04-25T00:32:24.752 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47555.log.gz 2024-04-25T00:32:24.752 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.46756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73402.log 2024-04-25T00:32:24.752 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73830.log 2024-04-25T00:32:24.752 INFO:teuthology.orchestra.run.smithi116.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.46756.log.gz 2024-04-25T00:32:24.752 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52561.log 2024-04-25T00:32:24.752 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.73402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82225.log 2024-04-25T00:32:24.752 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.73830.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.73402.log.gz 2024-04-25T00:32:24.752 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.52561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81164.log 2024-04-25T00:32:24.752 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.73830.log.gz 2024-04-25T00:32:24.753 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52561.log.gz 2024-04-25T00:32:24.753 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.82225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35211.log 2024-04-25T00:32:24.753 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.81164.log: 88.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.82225.log.gz 2024-04-25T00:32:24.753 INFO:teuthology.orchestra.run.smithi116.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.42586.log 2024-04-25T00:32:24.753 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.35211.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.81164.log.gz 2024-04-25T00:32:24.753 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82611.log 2024-04-25T00:32:24.753 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.42586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34024.log 2024-04-25T00:32:24.753 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.82611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83134.log 2024-04-25T00:32:24.753 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36800.log 2024-04-25T00:32:24.753 INFO:teuthology.orchestra.run.smithi116.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.82611.log.gz 2024-04-25T00:32:24.753 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.83134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44889.log 2024-04-25T00:32:24.753 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.36800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47167.log 2024-04-25T00:32:24.753 INFO:teuthology.orchestra.run.smithi116.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.83134.log.gz 2024-04-25T00:32:24.753 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.44889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40319.log 2024-04-25T00:32:24.754 INFO:teuthology.orchestra.run.smithi116.stderr: 89.4%/var/log/ceph/ceph-client.admin.47167.log: -- replaced with /var/log/ceph/ceph-client.admin.33972.log.gz 2024-04-25T00:32:24.754 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33498.log 2024-04-25T00:32:24.754 INFO:teuthology.orchestra.run.smithi116.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.37327.log.gz 2024-04-25T00:32:24.754 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.40319.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47167.log.gz 2024-04-25T00:32:24.754 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47222.log 2024-04-25T00:32:24.754 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.33498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50277.log 2024-04-25T00:32:24.754 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.47222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43955.log 2024-04-25T00:32:24.754 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.50277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44744.log 2024-04-25T00:32:24.754 INFO:teuthology.orchestra.run.smithi116.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47222.log.gz 2024-04-25T00:32:24.754 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.43955.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50277.log.gz 2024-04-25T00:32:24.754 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82689.log 2024-04-25T00:32:24.754 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.44744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81764.log 2024-04-25T00:32:24.754 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.82689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84530.log 2024-04-25T00:32:24.754 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44468.log.gz 2024-04-25T00:32:24.755 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.81764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36878.log 2024-04-25T00:32:24.755 INFO:teuthology.orchestra.run.smithi116.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.82689.log.gz 2024-04-25T00:32:24.755 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.81764.log.gz 2024-04-25T00:32:24.755 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.84530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83855.log 2024-04-25T00:32:24.755 INFO:teuthology.orchestra.run.smithi116.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.41103.log.gz 2024-04-25T00:32:24.755 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.84530.log.gz 2024-04-25T00:32:24.755 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.36878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34901.log 2024-04-25T00:32:24.755 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5%/var/log/ceph/ceph-client.admin.83855.log: -- replaced with /var/log/ceph/ceph-client.admin.34483.log.gz 2024-04-25T00:32:24.755 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84583.log 2024-04-25T00:32:24.755 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34892.log.gz 2024-04-25T00:32:24.755 INFO:teuthology.orchestra.run.smithi116.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.83855.log.gz 2024-04-25T00:32:24.755 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36748.log 2024-04-25T00:32:24.755 INFO:teuthology.orchestra.run.smithi116.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43877.log.gz 2024-04-25T00:32:24.755 INFO:teuthology.orchestra.run.smithi116.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.35289.log.gz 2024-04-25T00:32:24.755 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41433.log 2024-04-25T00:32:24.756 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.84583.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35497.log.gz 2024-04-25T00:32:24.756 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.36748.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.35706.log.gz 2024-04-25T00:32:24.756 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.84583.log.gz 2024-04-25T00:32:24.756 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83567.log 2024-04-25T00:32:24.756 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.41433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35393.log 2024-04-25T00:32:24.756 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.83567.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.36202.log.gz 2024-04-25T00:32:24.756 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46981.log 2024-04-25T00:32:24.756 INFO:teuthology.orchestra.run.smithi116.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36384.log.gz 2024-04-25T00:32:24.756 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.83567.log.gz 2024-04-25T00:32:24.756 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.35393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44439.log 2024-04-25T00:32:24.756 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42521.log.gz 2024-04-25T00:32:24.756 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.46981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50933.log 2024-04-25T00:32:24.756 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.44439.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46981.log.gz 2024-04-25T00:32:24.756 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51267.log 2024-04-25T00:32:24.757 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.50933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36254.log 2024-04-25T00:32:24.757 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.51267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42762.log 2024-04-25T00:32:24.757 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50933.log.gz 2024-04-25T00:32:24.757 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.36254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36904.log 2024-04-25T00:32:24.757 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51267.log.gz 2024-04-25T00:32:24.757 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.42762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44342.log 2024-04-25T00:32:24.757 INFO:teuthology.orchestra.run.smithi116.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.42586.log.gz 2024-04-25T00:32:24.757 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.36904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84375.log 2024-04-25T00:32:24.757 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.44342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84011.log 2024-04-25T00:32:24.757 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.84375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41373.log 2024-04-25T00:32:24.757 INFO:teuthology.orchestra.run.smithi116.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.84375.log.gz 2024-04-25T00:32:24.757 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.84011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47451.log 2024-04-25T00:32:24.757 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60570.log 2024-04-25T00:32:24.757 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.41373.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.84011.log.gz 2024-04-25T00:32:24.757 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.47451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77449.log 2024-04-25T00:32:24.758 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.60570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41399.log 2024-04-25T00:32:24.758 INFO:teuthology.orchestra.run.smithi116.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.47451.log.gz 2024-04-25T00:32:24.758 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.77449.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60570.log.gz 2024-04-25T00:32:24.758 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45444.log 2024-04-25T00:32:24.758 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.41399.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.77449.log.gz 2024-04-25T00:32:24.758 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54709.log 2024-04-25T00:32:24.758 INFO:teuthology.orchestra.run.smithi116.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45125.log.gz 2024-04-25T00:32:24.758 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.45444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40063.log 2024-04-25T00:32:24.758 INFO:teuthology.orchestra.run.smithi116.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.40137.log.gz 2024-04-25T00:32:24.758 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41264.log 2024-04-25T00:32:24.758 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.54709.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.33341.log.gz 2024-04-25T00:32:24.758 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.40063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33738.log 2024-04-25T00:32:24.758 INFO:teuthology.orchestra.run.smithi116.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.54709.log.gz 2024-04-25T00:32:24.758 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34405.log 2024-04-25T00:32:24.759 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.33367.log.gz 2024-04-25T00:32:24.759 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.41264.log: /var/log/ceph/ceph-client.admin.33738.log: gzip -5 --verbose 88.1% -- /var/log/ceph/ceph-client.admin.83056.log 2024-04-25T00:32:24.759 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33498.log.gz 2024-04-25T00:32:24.759 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74405.log 2024-04-25T00:32:24.759 INFO:teuthology.orchestra.run.smithi116.stderr:gzip/var/log/ceph/ceph-client.admin.83056.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.42298.log 2024-04-25T00:32:24.759 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.74405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82173.log 2024-04-25T00:32:24.759 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.83056.log.gz 2024-04-25T00:32:24.759 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.42298.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.74405.log.gz 2024-04-25T00:32:24.759 INFO:teuthology.orchestra.run.smithi116.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.44744.log.gz 2024-04-25T00:32:24.759 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50727.log 2024-04-25T00:32:24.759 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.82173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83082.log 2024-04-25T00:32:24.759 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.50727.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.82173.log.gz 2024-04-25T00:32:24.759 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57877.log 2024-04-25T00:32:24.759 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.34693.log.gz 2024-04-25T00:32:24.760 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.83082.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44889.log.gz 2024-04-25T00:32:24.760 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- 88.8% /var/log/ceph/ceph-client.admin.34470.log 2024-04-25T00:32:24.760 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50727.log.gz 2024-04-25T00:32:24.760 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.83082.log.gz 2024-04-25T00:32:24.760 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.57877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46203.log 2024-04-25T00:32:24.760 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37196.log.gz 2024-04-25T00:32:24.760 INFO:teuthology.orchestra.run.smithi116.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.57877.log.gz 2024-04-25T00:32:24.760 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49536.log 2024-04-25T00:32:24.760 INFO:teuthology.orchestra.run.smithi116.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.43955.log.gz 2024-04-25T00:32:24.760 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.46203.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49432.log 2024-04-25T00:32:24.760 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45389.log.gz 2024-04-25T00:32:24.760 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46203.log.gz 2024-04-25T00:32:24.760 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43668.log 2024-04-25T00:32:24.760 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.49536.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34024.log.gz 2024-04-25T00:32:24.761 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42126.log 2024-04-25T00:32:24.761 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34901.log.gz 2024-04-25T00:32:24.761 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.49432.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.49536.log.gz 2024-04-25T00:32:24.761 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.43668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41183.log 2024-04-25T00:32:24.761 INFO:teuthology.orchestra.run.smithi116.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49432.log.gz 2024-04-25T00:32:24.761 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.42126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47842.log 2024-04-25T00:32:24.761 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.41183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53747.log 2024-04-25T00:32:24.761 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.47842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44238.log 2024-04-25T00:32:24.761 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51082.log 2024-04-25T00:32:24.761 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.53747.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47842.log.gz 2024-04-25T00:32:24.761 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5/var/log/ceph/ceph-client.admin.44238.log: --verbose -- /var/log/ceph/ceph-client.admin.36644.log 2024-04-25T00:32:24.761 INFO:teuthology.orchestra.run.smithi116.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53747.log.gz 2024-04-25T00:32:24.761 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.51082.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44439.log.gz 2024-04-25T00:32:24.761 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40382.log 2024-04-25T00:32:24.762 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51082.log.gz 2024-04-25T00:32:24.762 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.36644.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35393.log.gz 2024-04-25T00:32:24.762 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56250.log 2024-04-25T00:32:24.762 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40635.log 2024-04-25T00:32:24.762 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.40382.log: /var/log/ceph/ceph-client.admin.56250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46866.log 2024-04-25T00:32:24.762 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.40635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35159.log 2024-04-25T00:32:24.762 INFO:teuthology.orchestra.run.smithi116.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.56250.log.gz 2024-04-25T00:32:24.762 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.46866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47919.log 2024-04-25T00:32:24.762 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.35159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83647.log 2024-04-25T00:32:24.762 INFO:teuthology.orchestra.run.smithi116.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.46866.log.gz 2024-04-25T00:32:24.762 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36254.log.gz 2024-04-25T00:32:24.762 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.47919.log: /var/log/ceph/ceph-client.admin.84713.log 2024-04-25T00:32:24.762 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.83647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52708.log 2024-04-25T00:32:24.762 INFO:teuthology.orchestra.run.smithi116.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47919.log.gz 2024-04-25T00:32:24.762 INFO:teuthology.orchestra.run.smithi116.stderr:gzip/var/log/ceph/ceph-client.admin.84713.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.51404.log 2024-04-25T00:32:24.763 INFO:teuthology.orchestra.run.smithi116.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.83647.log.gz 2024-04-25T00:32:24.763 INFO:teuthology.orchestra.run.smithi116.stderr:gzip/var/log/ceph/ceph-client.admin.52708.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.49562.log 2024-04-25T00:32:24.763 INFO:teuthology.orchestra.run.smithi116.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.84713.log.gz 2024-04-25T00:32:24.763 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.51404.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52708.log.gz 2024-04-25T00:32:24.763 INFO:teuthology.orchestra.run.smithi116.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.36748.log.gz 2024-04-25T00:32:24.763 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48793.log 2024-04-25T00:32:24.763 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51404.log.gz 2024-04-25T00:32:24.763 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.49562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51189.log 2024-04-25T00:32:24.763 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.48793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40941.log 2024-04-25T00:32:24.763 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49562.log.gz 2024-04-25T00:32:24.763 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.51189.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48793.log.gz 2024-04-25T00:32:24.763 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35315.log 2024-04-25T00:32:24.763 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51189.log.gz 2024-04-25T00:32:24.763 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50381.log 2024-04-25T00:32:24.764 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.40941.log: /var/log/ceph/ceph-client.admin.35315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35758.log 2024-04-25T00:32:24.764 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.50381.log: gzip 88.8% -5 --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.43143.log.gz /var/log/ceph/ceph-client.admin.50221.log 2024-04-25T00:32:24.764 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-04-25T00:32:24.764 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50381.log.gz 2024-04-25T00:32:24.764 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.35758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83243.log 2024-04-25T00:32:24.764 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45444.log.gz 2024-04-25T00:32:24.764 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41869.log 2024-04-25T00:32:24.764 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.50221.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.33738.log.gz 2024-04-25T00:32:24.764 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.83243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42181.log 2024-04-25T00:32:24.764 INFO:teuthology.orchestra.run.smithi116.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.50221.log.gz 2024-04-25T00:32:24.764 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.41869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45099.log 2024-04-25T00:32:24.764 INFO:teuthology.orchestra.run.smithi116.stderr: 89.2% 88.0% -- replaced with /var/log/ceph/ceph-client.admin.83243.log.gz 2024-04-25T00:32:24.764 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40063.log.gz 2024-04-25T00:32:24.764 INFO:teuthology.orchestra.run.smithi116.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41399.log.gz 2024-04-25T00:32:24.764 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33582.log 2024-04-25T00:32:24.765 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.42181.log: /var/log/ceph/ceph-client.admin.45099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56060.log 2024-04-25T00:32:24.765 INFO:teuthology.orchestra.run.smithi116.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.42762.log.gz 2024-04-25T00:32:24.765 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.33582.log: gzip 87.9% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.34405.log.gz 2024-04-25T00:32:24.765 INFO:teuthology.orchestra.run.smithi116.stderr: -- /var/log/ceph/ceph-client.admin.49666.log 2024-04-25T00:32:24.765 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.56060.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.34470.log.gz 2024-04-25T00:32:24.765 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42152.log 2024-04-25T00:32:24.765 INFO:teuthology.orchestra.run.smithi116.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.56060.log.gz 2024-04-25T00:32:24.765 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.49666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33920.log 2024-04-25T00:32:24.765 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.42152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35081.log 2024-04-25T00:32:24.765 INFO:teuthology.orchestra.run.smithi116.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49666.log.gz 2024-04-25T00:32:24.765 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.33920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51459.log 2024-04-25T00:32:24.765 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.35081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46469.log 2024-04-25T00:32:24.765 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.51459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47529.log 2024-04-25T00:32:24.765 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75116.log 2024-04-25T00:32:24.766 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51459.log.gz 2024-04-25T00:32:24.766 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.46469.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41264.log.gz 2024-04-25T00:32:24.766 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.47529.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42126.log.gz 2024-04-25T00:32:24.766 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84451.log 2024-04-25T00:32:24.766 INFO:teuthology.orchestra.run.smithi116.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46469.log.gz 2024-04-25T00:32:24.766 INFO:teuthology.orchestra.run.smithi116.stderr: 89.0%/var/log/ceph/ceph-client.admin.75116.log: -- replaced with /var/log/ceph/ceph-client.admin.47529.log.gz 2024-04-25T00:32:24.766 INFO:teuthology.orchestra.run.smithi116.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.43668.log.gz 2024-04-25T00:32:24.766 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% 89.3% -- replaced with /var/log/ceph/ceph-client.admin.75116.log.gz 2024-04-25T00:32:24.766 INFO:teuthology.orchestra.run.smithi116.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44342.log.gz 2024-04-25T00:32:24.766 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.84451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48304.log 2024-04-25T00:32:24.766 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.84451.log.gz 2024-04-25T00:32:24.766 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44022.log 2024-04-25T00:32:24.766 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.48304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43797.log 2024-04-25T00:32:24.766 INFO:teuthology.orchestra.run.smithi116.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.36644.log.gz 2024-04-25T00:32:24.767 INFO:teuthology.orchestra.run.smithi116.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.48304.log.gz 2024-04-25T00:32:24.767 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.44022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33686.log 2024-04-25T00:32:24.767 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40635.log.gz 2024-04-25T00:32:24.767 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.43797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36566.log 2024-04-25T00:32:24.767 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.33686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36540.log 2024-04-25T00:32:24.767 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.36566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82925.log 2024-04-25T00:32:24.767 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43620.log 2024-04-25T00:32:24.767 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.36540.log: /var/log/ceph/ceph-client.admin.82925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50630.log 2024-04-25T00:32:24.767 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.43620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44585.log 2024-04-25T00:32:24.767 INFO:teuthology.orchestra.run.smithi116.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.82925.log.gz 2024-04-25T00:32:24.767 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60542.log 2024-04-25T00:32:24.767 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.50630.log: /var/log/ceph/ceph-client.admin.44585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42100.log 2024-04-25T00:32:24.767 INFO:teuthology.orchestra.run.smithi116.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50630.log.gz 2024-04-25T00:32:24.767 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.60542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42428.log 2024-04-25T00:32:24.767 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60542.log.gz 2024-04-25T00:32:24.768 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.42100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33946.log 2024-04-25T00:32:24.768 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.42428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55971.log 2024-04-25T00:32:24.768 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.33946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42814.log 2024-04-25T00:32:24.768 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.55971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42495.log 2024-04-25T00:32:24.768 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.42814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79160.log 2024-04-25T00:32:24.768 INFO:teuthology.orchestra.run.smithi116.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.55971.log.gz 2024-04-25T00:32:24.768 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45416.log 2024-04-25T00:32:24.768 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.42495.log: /var/log/ceph/ceph-client.admin.79160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81896.log 2024-04-25T00:32:24.768 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.45416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60451.log 2024-04-25T00:32:24.768 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.79160.log.gz 2024-04-25T00:32:24.768 INFO:teuthology.orchestra.run.smithi116.stderr:gzip/var/log/ceph/ceph-client.admin.81896.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.50880.log 2024-04-25T00:32:24.768 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.60451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35549.log 2024-04-25T00:32:24.768 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.81896.log.gz 2024-04-25T00:32:24.768 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.50880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42272.log 2024-04-25T00:32:24.769 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.60451.log.gz 2024-04-25T00:32:24.769 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.35549.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50880.log.gz 2024-04-25T00:32:24.769 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42298.log.gz 2024-04-25T00:32:24.769 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40837.log 2024-04-25T00:32:24.769 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.42272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34167.log 2024-04-25T00:32:24.769 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.40837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82530.log 2024-04-25T00:32:24.769 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34893.log 2024-04-25T00:32:24.769 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.82530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40101.log 2024-04-25T00:32:24.769 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43063.log.gz 2024-04-25T00:32:24.769 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.82530.log.gz 2024-04-25T00:32:24.769 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36670.log 2024-04-25T00:32:24.769 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.40101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34141.log 2024-04-25T00:32:24.769 INFO:teuthology.orchestra.run.smithi116.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.41183.log.gz 2024-04-25T00:32:24.769 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.36670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44863.log 2024-04-25T00:32:24.769 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.34141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45960.log 2024-04-25T00:32:24.770 INFO:teuthology.orchestra.run.smithi116.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35159.log.gz 2024-04-25T00:32:24.770 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.44863.log: /var/log/ceph/ceph-client.admin.82093.log 2024-04-25T00:32:24.770 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.45960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43410.log 2024-04-25T00:32:24.770 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42181.log.gz 2024-04-25T00:32:24.770 INFO:teuthology.orchestra.run.smithi116.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42152.log.gz 2024-04-25T00:32:24.770 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45774.log 2024-04-25T00:32:24.770 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.82093.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.35315.log.gz 2024-04-25T00:32:24.770 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42324.log 2024-04-25T00:32:24.770 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35758.log.gz 2024-04-25T00:32:24.770 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.43410.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45099.log.gz 2024-04-25T00:32:24.770 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46892.log 2024-04-25T00:32:24.770 INFO:teuthology.orchestra.run.smithi116.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45047.log.gz/var/log/ceph/ceph-client.admin.42324.log: 2024-04-25T00:32:24.770 INFO:teuthology.orchestra.run.smithi116.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.82093.log.gz 2024-04-25T00:32:24.770 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40941.log.gz 2024-04-25T00:32:24.771 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.45774.log: 88.6%/var/log/ceph/ceph-client.admin.46892.log: -- replaced with /var/log/ceph/ceph-client.admin.33920.log.gz 2024-04-25T00:32:24.771 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46892.log.gz 2024-04-25T00:32:24.771 INFO:teuthology.orchestra.run.smithi116.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.33582.log.gz 2024-04-25T00:32:24.771 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42877.log 2024-04-25T00:32:24.771 INFO:teuthology.orchestra.run.smithi116.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.33686.log.gz 2024-04-25T00:32:24.771 INFO:teuthology.orchestra.run.smithi116.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46255.log 2024-04-25T00:32:24.771 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42428.log.gz 2024-04-25T00:32:24.771 INFO:teuthology.orchestra.run.smithi116.stderr:/var/log/ceph/ceph-client.admin.42877.log: /var/log/ceph/ceph-client.admin.46255.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.46255.log.gz 2024-04-25T00:32:24.771 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35081.log.gz 2024-04-25T00:32:24.771 INFO:teuthology.orchestra.run.smithi116.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.40837.log.gz 2024-04-25T00:32:24.771 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42100.log.gz 2024-04-25T00:32:24.771 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35549.log.gz 2024-04-25T00:32:24.771 INFO:teuthology.orchestra.run.smithi116.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.36566.log.gz 2024-04-25T00:32:24.771 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36540.log.gz 2024-04-25T00:32:24.772 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36670.log.gz 2024-04-25T00:32:24.772 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42272.log.gz 2024-04-25T00:32:24.772 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42495.log.gz 2024-04-25T00:32:24.772 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44022.log.gz 2024-04-25T00:32:24.772 INFO:teuthology.orchestra.run.smithi116.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45960.log.gz 2024-04-25T00:32:24.772 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41869.log.gz 2024-04-25T00:32:24.772 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41895.log.gz 2024-04-25T00:32:24.772 INFO:teuthology.orchestra.run.smithi116.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42814.log.gz 2024-04-25T00:32:24.772 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40319.log.gz 2024-04-25T00:32:24.772 INFO:teuthology.orchestra.run.smithi116.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.40382.log.gz 2024-04-25T00:32:24.772 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41433.log.gz 2024-04-25T00:32:24.772 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43410.log.gz 2024-04-25T00:32:24.772 INFO:teuthology.orchestra.run.smithi116.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.43797.log.gz 2024-04-25T00:32:24.772 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44238.log.gz 2024-04-25T00:32:24.772 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43620.log.gz 2024-04-25T00:32:24.773 INFO:teuthology.orchestra.run.smithi116.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45416.log.gz 2024-04-25T00:32:24.773 INFO:teuthology.orchestra.run.smithi116.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.33946.log.gz 2024-04-25T00:32:24.773 INFO:teuthology.orchestra.run.smithi116.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34141.log.gz 2024-04-25T00:32:24.773 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40101.log.gz 2024-04-25T00:32:24.773 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42877.log.gz 2024-04-25T00:32:24.773 INFO:teuthology.orchestra.run.smithi116.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44585.log.gz 2024-04-25T00:32:24.773 INFO:teuthology.orchestra.run.smithi116.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44863.log.gz 2024-04-25T00:32:24.773 INFO:teuthology.orchestra.run.smithi116.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45774.log.gz 2024-04-25T00:32:24.773 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.34167.log.gz 2024-04-25T00:32:24.773 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34893.log.gz 2024-04-25T00:32:24.773 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42324.log.gz 2024-04-25T00:32:24.774 INFO:teuthology.orchestra.run.smithi116.stderr: 95.0% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-04-25T00:32:24.900 INFO:teuthology.orchestra.run.smithi116.stderr: 88.8% 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36878.log.gz -- replaced with /var/log/ceph/ceph-client.admin.36852.log.gz 2024-04-25T00:32:24.923 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-04-25T00:32:24.923 INFO:teuthology.orchestra.run.smithi116.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.36904.log.gz 2024-04-25T00:32:24.923 INFO:teuthology.orchestra.run.smithi116.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.36930.log.gz 2024-04-25T00:32:24.923 INFO:teuthology.orchestra.run.smithi174.stderr: 92.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-25T00:32:24.924 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36800.log.gz 2024-04-25T00:32:24.924 INFO:teuthology.orchestra.run.smithi116.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.39433.log.gz 2024-04-25T00:32:24.924 INFO:teuthology.orchestra.run.smithi116.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39520.log.gz 2024-04-25T00:32:24.924 INFO:teuthology.orchestra.run.smithi116.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.40408.log.gz 2024-04-25T00:32:24.925 INFO:teuthology.orchestra.run.smithi116.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.40434.log.gz 2024-04-25T00:32:24.925 INFO:teuthology.orchestra.run.smithi116.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35211.log.gz 2024-04-25T00:32:24.925 INFO:teuthology.orchestra.run.smithi116.stderr: 89.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-25T00:32:26.137 INFO:teuthology.orchestra.run.smithi116.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41373.log.gz 2024-04-25T00:32:29.825 INFO:teuthology.orchestra.run.smithi174.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-25T00:32:30.548 INFO:teuthology.orchestra.run.smithi174.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-04-25T00:32:30.867 INFO:teuthology.orchestra.run.smithi174.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-04-25T00:32:42.958 INFO:teuthology.orchestra.run.smithi174.stderr: 94.7% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-04-25T00:32:43.202 INFO:teuthology.orchestra.run.smithi116.stderr: 90.2% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-25T00:32:48.484 INFO:teuthology.orchestra.run.smithi174.stderr: 94.1% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-04-25T00:32:48.525 INFO:teuthology.orchestra.run.smithi174.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-04-25T00:32:57.324 INFO:teuthology.orchestra.run.smithi174.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-04-25T00:33:00.092 INFO:teuthology.orchestra.run.smithi116.stderr: 94.4% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-04-25T00:33:01.478 INFO:teuthology.orchestra.run.smithi116.stderr: 94.3% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-04-25T00:33:03.546 INFO:teuthology.orchestra.run.smithi174.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-04-25T00:33:05.330 INFO:teuthology.orchestra.run.smithi116.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-25T00:33:07.552 INFO:teuthology.orchestra.run.smithi116.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-25T00:33:08.899 INFO:teuthology.orchestra.run.smithi174.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-04-25T00:33:09.103 INFO:teuthology.orchestra.run.smithi116.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-25T00:33:16.268 INFO:teuthology.orchestra.run.smithi116.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-04-25T00:35:20.618 INFO:teuthology.orchestra.run.smithi174.stderr: 96.0% -- replaced with /var/log/ceph/ceph-client.1.33357.log.gz 2024-04-25T00:35:20.619 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-04-25T00:35:20.620 INFO:teuthology.orchestra.run.smithi174.stderr:real 2m56.198s 2024-04-25T00:35:20.620 INFO:teuthology.orchestra.run.smithi174.stderr:user 4m9.916s 2024-04-25T00:35:20.620 INFO:teuthology.orchestra.run.smithi174.stderr:sys 0m11.824s 2024-04-25T00:35:24.054 INFO:teuthology.orchestra.run.smithi116.stderr: 96.2% -- replaced with /var/log/ceph/ceph-client.0.36045.log.gz 2024-04-25T00:35:24.058 INFO:teuthology.orchestra.run.smithi116.stderr: 2024-04-25T00:35:24.058 INFO:teuthology.orchestra.run.smithi116.stderr:real 2m59.671s 2024-04-25T00:35:24.058 INFO:teuthology.orchestra.run.smithi116.stderr:user 4m31.621s 2024-04-25T00:35:24.058 INFO:teuthology.orchestra.run.smithi116.stderr:sys 0m12.582s 2024-04-25T00:35:24.059 INFO:tasks.ceph:Archiving logs... 2024-04-25T00:35:24.059 DEBUG:teuthology.misc:Transferring archived files from smithi116:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672224/remote/smithi116/log 2024-04-25T00:35:24.061 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T00:35:52.510 DEBUG:teuthology.misc:Transferring archived files from smithi174:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672224/remote/smithi174/log 2024-04-25T00:35:52.511 DEBUG:teuthology.orchestra.run.smithi174:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T00:36:19.511 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-25T00:36:19.521 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-25T00:36:19.522 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T00:36:19.525 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T00:36:19.590 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-25T00:36:19.601 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-25T00:36:19.602 DEBUG:teuthology.orchestra.run.smithi116:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T00:36:19.604 DEBUG:teuthology.orchestra.run.smithi174:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T00:36:19.647 INFO:teuthology.orchestra.run.smithi116.stdout: remote refid st t when poll reach delay offset jitter 2024-04-25T00:36:19.647 INFO:teuthology.orchestra.run.smithi116.stdout:============================================================================== 2024-04-25T00:36:19.647 INFO:teuthology.orchestra.run.smithi116.stdout:*hv01.front.sepi 44.4.53.2 2 u 63 64 377 0.081 -2.951 0.114 2024-04-25T00:36:19.647 INFO:teuthology.orchestra.run.smithi116.stdout:+hv02.front.sepi 63.231.80.2 3 u 66 64 377 0.075 -2.017 0.118 2024-04-25T00:36:19.647 INFO:teuthology.orchestra.run.smithi116.stdout:+hv03.front.sepi 74.6.168.72 3 u 8 64 377 0.068 -2.098 0.129 2024-04-25T00:36:19.647 INFO:teuthology.orchestra.run.smithi116.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-04-25T00:36:19.678 INFO:teuthology.orchestra.run.smithi174.stdout: remote refid st t when poll reach delay offset jitter 2024-04-25T00:36:19.678 INFO:teuthology.orchestra.run.smithi174.stdout:============================================================================== 2024-04-25T00:36:19.678 INFO:teuthology.orchestra.run.smithi174.stdout:*hv01.front.sepi 44.4.53.2 2 u 7 64 377 0.082 -1.279 0.055 2024-04-25T00:36:19.678 INFO:teuthology.orchestra.run.smithi174.stdout:+hv02.front.sepi 63.231.80.2 3 u 58 64 377 0.074 -0.501 0.219 2024-04-25T00:36:19.678 INFO:teuthology.orchestra.run.smithi174.stdout:+hv03.front.sepi 74.6.168.72 3 u 1 64 377 0.073 -0.391 0.066 2024-04-25T00:36:19.678 INFO:teuthology.orchestra.run.smithi174.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-04-25T00:36:19.679 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-25T00:36:19.689 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-25T00:36:19.690 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-25T00:36:19.699 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-25T00:36:19.712 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-25T00:36:19.722 INFO:teuthology.task.internal:Duration was 2793.264481 seconds 2024-04-25T00:36:19.722 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-25T00:36:19.732 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-25T00:36:19.732 DEBUG:teuthology.orchestra.run.smithi116:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T00:36:19.735 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T00:36:19.939 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-25T00:36:19.939 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi116.front.sepia.ceph.com 2024-04-25T00:36:19.939 DEBUG:teuthology.orchestra.run.smithi116:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T00:36:19.950 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi174.front.sepia.ceph.com 2024-04-25T00:36:19.951 DEBUG:teuthology.orchestra.run.smithi174:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T00:36:19.969 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-25T00:36:19.970 DEBUG:teuthology.orchestra.run.smithi116:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T00:36:19.999 DEBUG:teuthology.orchestra.run.smithi174:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T00:36:20.053 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-25T00:36:20.053 DEBUG:teuthology.orchestra.run.smithi116:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T00:36:20.056 DEBUG:teuthology.orchestra.run.smithi174:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T00:36:20.230 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-25T00:36:20.242 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-25T00:36:20.242 DEBUG:teuthology.orchestra.run.smithi116:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T00:36:20.275 DEBUG:teuthology.orchestra.run.smithi174:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T00:36:20.296 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-25T00:36:20.306 DEBUG:teuthology.orchestra.run.smithi116:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T00:36:20.319 DEBUG:teuthology.orchestra.run.smithi174:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T00:36:20.343 INFO:teuthology.orchestra.run.smithi116.stdout:kernel.core_pattern = core 2024-04-25T00:36:20.357 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern = core 2024-04-25T00:36:20.386 DEBUG:teuthology.orchestra.run.smithi116:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T00:36:20.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T00:36:20.417 DEBUG:teuthology.orchestra.run.smithi174:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T00:36:20.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T00:36:20.433 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-25T00:36:20.444 INFO:teuthology.task.internal:Transferring archived files... 2024-04-25T00:36:20.446 DEBUG:teuthology.misc:Transferring archived files from smithi116:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672224/remote/smithi116 2024-04-25T00:36:20.446 DEBUG:teuthology.orchestra.run.smithi116:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T00:36:20.483 DEBUG:teuthology.misc:Transferring archived files from smithi174:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672224/remote/smithi174 2024-04-25T00:36:20.483 DEBUG:teuthology.orchestra.run.smithi174:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T00:36:20.504 INFO:teuthology.task.internal:Removing archive directory... 2024-04-25T00:36:20.504 DEBUG:teuthology.orchestra.run.smithi116:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T00:36:20.523 DEBUG:teuthology.orchestra.run.smithi174:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T00:36:20.545 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-25T00:36:20.557 INFO:teuthology.task.internal:Not uploading archives. 2024-04-25T00:36:20.557 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-25T00:36:20.567 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-25T00:36:20.568 DEBUG:teuthology.orchestra.run.smithi116:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T00:36:20.570 DEBUG:teuthology.orchestra.run.smithi174:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T00:36:20.574 INFO:teuthology.orchestra.run.smithi116.stdout: 658640 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 00:36 /home/ubuntu/cephtest 2024-04-25T00:36:20.594 INFO:teuthology.orchestra.run.smithi174.stdout: 658623 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 00:36 /home/ubuntu/cephtest 2024-04-25T00:36:20.595 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-25T00:36:20.606 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-25T00:36:20.631 INFO:teuthology.run:Summary data: description: fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} duration: 2793.2644805908203 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-04-25T00:36:20.631 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T00:36:20.721 INFO:teuthology.run:pass